I've always seen it as definitive.
The Harbour says: "You may choose to unload a gemstone that you had previously loaded onto a horse or cart, in order to reload a different gemstone."
Road spaces don't allow unloading, saying: "On a road space, you may load all of the gemstones located there that you have the matching empty equipment for."
What makes you think that you can't unload gems on road spaces? To me it seems quite obvious that that rule is a general rule for both roads and harbors.
For harbors the rules says: "Here also you may only load those gemstones that you have a matching empty equipment for."