I don't see any reason not to extend it to six teams/players either.
Minor thing: The Rivendell stack will be gone when the 6th player reaches there. But that's just bad luck, right?
But for downtime reasons, i wouldn't play it with more than 4. With 6, i'd rather play with 3 teams of 2 players each. Or one or two 'LotR experts' on their own, and the others in teams.