Or is it just the two spaces on either side of the wall...?
I was wondering the same and decided to play it with only the 2 spaces on either side of the wall, but that was just our guess. I'd like an answer on that too.
Also, I've been wondering does anyone actually break walls by just damaging it a bunch? I feel like that would take you a few turns at least and by that time you could have just gone around or something, and if you COULD do that much damage at once you'd probably want to save it for an enemy wizard. I haven't seen all the spells in the game yet but it just seems impractical, no one even bothered shooting at walls just to add one crack to it.