I'm fairly sure the Shards that you put on the Gems come from the unclaimed supply of Shards, not from any player's personal stash of Shards. It's a new setence. The inferred "you" of that sentence is the person who drew the Scheme Twist.
If there are no Shards left in the general supply after making change wherever there are multiple Shards (changing multiples ones to twos or threes or fives), then just ignore that part of the Twist.
Just for general reference, here is what the Scheme says:
Forge the Infinity Gauntlet
Setup: 8 Twists. Always include the Infinity Gems Villain Group.
Twist: Starting to your left and going clockwise, the first player with an Infinity Gem Artifact card in play or in their discard pile chooses one of those Infinity Gems to enter the city. Then put a Shard on each Infinity Gem in the city.
Evil Wins: When 6 Infinity Gem Villains are in the city and/or the Escape Pile.
Evil Wins: When a plyaer controls 4 Infinity Gem Artifacts, that player is corrupted by power. That player wins, Evil wins, and all other players lose.