You can take my game… when you pry my cold, dead fingers off the board!
This is covered in "Uncommon Situations" on p5 of the version 2 rules:
NY1901 Rules V2
If you have no available Workers at the start of your turn AND you cannot Build (or Demolish) on your turn, then you MUST reclaim one of your Workers from the board and return a corresponding Territory Card from the top of your Character Card back to the bottom of the Future Market, then pass the turn to the next player.
So yes, the worker is "stuck" there for a while at least. But you can get by pretty well with just 3 workers.