can units keep on retreating past zones with 10 friendly units in them?
Yes, retreating units retreat until they find an area with enough room. If they can't find a legal area they are eliminated.
If so what is to stop the russian player from simply bouncing off all the defense 3 areas towards the end of the game
The germans ??
How exactly do the retreating rules work
Hmm, I'm not sure what you mean. I suggest you read rule 9.2 again.