Everything is relative to perception, and your perception is limited.
The Ginger Ninja
Even if you have 5 investigators, the keeper should probably still get only 4 threat per turn unless you're playing Call of the Wild.
Agree with this. I've played several games with 5 investigators (all in base game), and it really does work fine. Tried 1 game with 6 investigators, and it was an abomination. Don't recommend that at all.