I'm not crazy. My mother had me tested.
The odds of me using 6 characters in a scenario is low. Is there any reason I can't use the Guard Tower with less than 6? Otherwise I doubt I would ever encounter this location.
It's your game, go ahead and change the locations if you want.