You typically won't want to vote every round anyway.
I'm also pretty sure that you start with three cards then immediately draw another three cards on your first turn. So you'll actually have five cards (after playing one) at the end of your first turn.
Ah, that makes sense! So the 3 cards you get during setup are in addition to the 3 you draw on your turn. Although, when do you need to discard down to 5 in respect to the hand limit?