On the Nations: Dynasties page, under the Community Wiki, you'll see a link to an officially sanctioned variant:
To streamline it further, you could use the following rule:
2 players: keep the first 5 buildings, the first 2 of every other category
3 players: keep the first 5 buildings, the first 3 of every other category
4 players: keep the first 6 buildings, the first 3 of every other category
5 players: keep the first 8 buildings, the first 4 of every other category
This adds up to potentially 19/26/27/36 cards, of which only 12/15/18/21 are actually used, i.e. it is still possible that some categories don't appear at all.