The rule book states:
"only the Baron, the Mercenaries and the Infantrymen can move through or stay in Villages and Cities they control".
A nice player aide I've been using states this about Cities and Villages:
"May be entered or crossed by Baron, Milady, Infantry or Mercenary if under your control"
I have been using what is posted in the rule book, but I wanted to make sure.