There must have been a moment at the beginning, where we could have said no. Somehow we missed it. Well, we'll know better next time.
We can't tell from the board which side is for 5-6 players and which is for 3-4. There seems to be no indication anywhere.
Our copy was missing the rulebook (which we downloaded here). Does it say which is which in the final rules?
From Bruno Faidutti (one of the designers)'s blog:
Oooops ! We just forgot to label the boards to indicate which ones are for 3-4 players and which ones for 5-6. The shorter tracks, iplain and city, are for 3-4 players, and the longer ones, mountain and palace, for 5-6.