My scythe... I like to keep it next to where my heart used to be.
The biggest downside is with less player pawns then the less frequent ousting will occur...maybe not a huge deal, but a free roll is a free roll.
I haven't played the game with 2 players yet, only with 3 and 4. So I'll try to be careful about jumping to conclusions. However, it seems very likely to me that this will indeed be the biggest problem with 2 players.
However, I think it would be an easy house rule to combine some of the action spaces. For example, combining the 1/1/2/3 spots into a single location on the sun side, and the 1/1/2/3 spots into a single location on the moon side. Then, combining the 4/5 spots on the sun side with the 4/5 spots on the moon side.
That way you have a total of 4 action spots for 2 players as opposed to 7 action spots for 3 or 4 players. I'll certainly be trying this variant in the future.