We've always played it such that you can just use two workers as jokers for food/wood/stone to speed up the game. However, drawing cards still takes a turn.
This is how we were playing it. I'm gonna switch over to the "any conversion multi or single takes a turn" mode. It appears that's the way the game was intended.