Originaly all Whirlwind did was letting you move Storm Brothers. The extra dice was added to make the Storm Brothers more appealing as a unit, since they are otherwise only expensive barbarians.
So yes I believe it is the intended rule that the extra dice is only added if a Storm Brother show up. This is what make the most thematical sense.
This is my understanding as well.