In our group, we found it natural to have everybody leave their meeples on the table in the caves. . . before the next card is drawn, we all grab our meeples on the table. Someone counts down to zero, and at that moment, the people who stay leave their meeple and the people who wish to leave pull them off the table.