The only downside would be the game dragging on longer than might be fun, it's already a long with four players. But if all the players know the cards well maybe it would be fine.
You can get to six players with the Reuse expansion and include the Office. That's three extra cards right there.
Reuse does not add more players