My opponent had a 2-cost world and had 2 dice underneath settle. I selected the settle action He didn't want to use any of his settlers, but instead wanted to return them to his cup. I said because the settle action was selected he had to use his settlers to settle the planet.
It may have been possible to abandon the world during the Explore phase, then during the settle phase, the dice would have nowhere to go and would be returned to the cup. If there was a higher value world next, during maintenance phase he would be able to return them to his cup.
But that sequence would depend on him having an Explore action triggered. and of course, the world would be gone.
Alternatively, if this situation (the tile present and 2 settlers upon it) had existed at the end of the previous round, the player could have chosen then to return his settlers to the cup during the Manage Empire: recall step.