The Defender spends the 2 hourglasses and places the one wall after the Invader goes in Phase One (so they may have 3 hourglasses if the Invader got wood). It is listed on page 12 of the English rulebook:
At the start of every turn, the Defender receives 1 Stone wall Component and 2 hourglasses to use after Phase 1 - Invader's Supplies.
And destroyed wall sections go back into the Defender's supply and can be reused later.
Infinite wall resources isn't too bad. They can only place one per turn and already would have 5 turns of extra walls to place even without reusing the destroyed wall sections.
Invader wins have a larger learning curve attached to them. Try starting your assaults earlier to have a better chance. I've started to flip my Phase 1 card as the Invader at the start of Turn 4 to force myself out of the habit of spending too much time building and preparing before attacking. It helped me a lot as far as getting into a better rhythm of when to make my assaults.