Just to add, while the other player cannot take control over the town, they could take control over another, unoccupied building in the region, as you do not have control over the region for as long as the town citizen remains a rebel (you occupy the town, but do not control it). (see: Multiple players in a region, page 12)
You are right and this is what won him the game. He took my port which gave him enough points to pass me. We played the game right, we just weren't sure if we were at the time.