The statement below is false.
The statement above is correct.
Is there any reason it can't be removed?
Yes, there is: No effect in the rules would allow you to just remove an activation marker or a routed activation marker, whether you still have units in the area or not.
You can make sure that no routed activation marker is being placed in the first place by, when having to rout units, just destroying them instead (as per page 31 of the rules).
If your movement is due to an event card, activation markers are ignored, too (page 44) - so even if there's a routed activation marker, you can still land and force coexistence battles, you merely won't be able to move your units out of the area for the next couple of rounds.
(Thematically, I imagine that not all of your units are killed just yet, but the imperials are busy slaughtering them while some of them run around while screaming in headless panic and others spend their time fearfully waiting for the inevitable in a corner of the ship.)