While that's true, it also seemingly reduces the benefit to capturing a town. While the waiving of the visit cost is not insignificant, you can still claim victory without it.
Furthermore it raises the questions of why would you desire to conquer towns held by other players, aside from removing the visit benefit from them. If towns were counted towards your final renown score, it would add a clearer strategic benefit to claiming towns and stealing towns from other players, would it not?
It's not intended to be an aggressive sort of game, so it doesn't hugely reward taking over another's town - it even makes it trickier as you need 2 extra attack or diplomacy.
However towards the end of the game we often do take over each other's town - just because we're looking for any town to convert our current attack or diplomacy into points. Towns that have quite a high cost for use (one where you have to exhaust someone as well as pay a coin) are also ripe for taking over earlier.
I'm very glad it plays the way it does - it doesn't feel too sad if someone steals your island, and equally it's not hugely important to try and take over your opponents islands.
There are many aggressive games where it's all about taking over each others land (like Cyclades for instance), but this isn't one of them.