The bricks you have collected throughout the game are all counted up towards your score.The bonus of finishing with 18 bricks is a +5 points to this total.
If you attack and they have no defenders then still swap cards with their card counted as zero.