Theoretically, yes. You would remove 8 metal, 5 wood, 3 gem, and 2 magic as the following is stated in the original rules:
Base Game Rules wrote:
All resources in the game are finite. For example, if you choose a card that would gain you a metal die, but there are currently no metal dice in the stock; you will receive no metal die and cannot retroactively claim a metal die when one becomes available.
The Apprentices rules indicate that players have some leeway on this question:
Apprentices Rules wrote:
Note: these extra dice ... may be included in games with less than 5 players as well. If you prefer game economies where there is never a shortage of a particular type of resource, this is encouraged.
Honestly, we don't worry about it in an effort to just get into the game and keep the table display orderly. Of course, the play style of my group doesn't horde dice and people tend use the Docks often in the rush to craft items first. Plus, I play with my nephews who are both under 11. If I were playing with four serious gamers more often, it would be worth it for me to reset the dice.
Personally, I appreciate the incredible variability in King's Forge and this is just another knob you may adjust in your play style. "Adjust to play group is my recommendation."