I don't remember seeing the rule about not placing them in the trade point squares. Can anyone point out where that is stated in the rules?
From the rules here on the geek (near the end):
The huts of the workers…
Are shown on both starting buildings with the totem sculptures. If a starting building is covered, its owner must immediately move it to free fields - This is done directly before he continues his turn. Starting buildings may not occupy trade fields or fields with ones own or opponents’ building permits.