Under the "Placing Buildings" heading of the rules:
Other players' campesinos cannot end their movement in a space occupied by one of your casas. They can move through a space with your casa, but only if you do not also have a campesino there.
So from what I understand, if you have a casa in your hex and one or more campesinos, no one can even enter your hex. You are safe from being driven off.