I'll be honest - we're throwing science at the wall here to see what sticks.
Dice left over in front of a player are not added to the library at the end of the round, They do not score at all. The rules for the library are clear in terms of which dice are put there. That being said, dice leftover in buildings are added to the library. I disagree. The rules for the library say "Any dice that do not yield any reward when buildings are resolved will also be moved here." It doesn't say this exclusively happens to dice on buildings. It says "any dice". Also, if Karmic_devil says so, it must be true.