When I form a federation, I don’t think I can place a satellite on a hex which is adjacent to my existing federation’s planet or satellite.
Correct. This would connect your planets to your existing first federation, rather than form a new one. For most races, this isn't a legal action.
But how about a hex which is adjacent to a planet or a satellite of my opponent’s federation? Can I place my satellite on such a hex or should I place it so that it is at least one hex away from any part of my opponent’s federation?
You can border, and even share space with, an opponent's federation. Multiple players can have a satellite on the same empty space.