I think its the scheme Forging the Infinity gauntlet. When a scheme twist comes out, first player to the starting on the left has to place an infinity gem they have in play/discard pile back into the city and add one shard to every infinity gem in the city.
If the Soul Gem has 4 shards on it in a player's play area and its the only infinity gem in play/discard piles. Do the 4 shards go with it back into the city plus one from the arrival of a scheme twist?
For me they will not go, the "shards" on the soul gem are not really shards that could be expended, they are there to show how much atack could be achieved using the soul gem:
Artifact - Whenever you defeat a Villain, put a Shard on Soul Gem from the supply. Once per turn, you get + Attack equal to the number of Shards on Soul Gem.