My guess is after the initial 4 improvements the rest get put back in the box? There was nothing in the rules about drawing replacement improvements after you build one. Wasn't sure if there was a maximum of 4 per game or not.
I have wondered this as well. As you said, it does not say what exactly to do in the rulebook after an improvement has been built.
But, in Rodney's Watch It Played video, he says to replace a card that is bought with a random card from the improvement deck.
So in our game, we went with Rodney's rule from the video. I am pretty sure Plaid Hat looks over his videos to help catch any last minute rules mistakes, so I feel like they would have corrected him if that rule was incorrect.