It's great with 3! Maybe a little better with 4 (I think some of the boss abilities make it so) but nothing major and it will just shift the balance of each boss a bit.
Boss abilities? Unless you mean the Ship/Port cards you flip...you might be talking about Boss Monsters
Either way your statement still stands. Game plays great with 3,and 4. And the 2 Player variants are somehow more strategic, and make for a different game play, but also very fun!