No skeletons are ever removed permanently from the game, they keep coming at you from the graveyards!
Therefore, during setup, you will need exactly one skeleton of each colour/symbol (Green, Purple, Blue and Yellow).
Should you draw another skeleton with the same symbol, send it back to the bag.
Right after setup, your player board should only have four skeletons on it.
Hope this answers your question!
Thanks. I figured as much.