It's possible for you to remove all of the gems. This way you always need to get at least one key to escape.
Exactly. We stole a yellow gem from one of our other games and add it to the pot. Then we tell everybody you need to roll keys equal to the number of gems, except the yellow gem we cannot place on a spot on the board.