When you lose a building, you lose the points as well. If you expand to the building again, you get the points back.
Yep, just like in the cousin game Kingsburg. Many Kingsburg players count their building points at the end of the game instead of keeping track of it during the game for this reason. However, there are some scenarios that uses current victory points to determine certain things (I think there's one that uses VP instead of dice rolls to determine player order for the placement phase), so you can't get away with not scoring buildings if you're playing that scenario.