If it was empty, wouldn't the shadow player be able to retreat to it? There is no resistance there.
You can only retreat into a free region. A free region may not contain an unconquered enemy settlement. Therefore, an SP marker in that region is required.
On the other hand, the question really was about retreating into a siege. AFAIK, if it's your Stronghold, whether captured or naturally, you may retreat into a Siege in that Stronghold as the defending army.