In particular, there is a special rule that "before a mouse can use an explore action to leave this tile, all mice must be on the same side of the water as the exploring mouse".
IIRC that rule is not always there in all instances where the mice need to cross water. Though I always play it that way. And therefor...
But I am not sure whether this applies when the explore space is in the water.
...I would rule that they do all in fact need to be in the water.
Though that would of course make things more dangerous. The rules don't seem to indicate either way, so you could also do what you suggest, fly over there and explore safely.
It's one of many things left ambiguous in this game. Personally, when in doubt I always take the harder option because I find the game a bit on the easy side.