In addition, players who recall their troops will be missing out on the prayer point generation from the temples during the night phase. Would you want a slow economy of 2PP per night phase, or a whopping 2PP + 2PP + 3PP from temples? That is plenty extra to recruit and move more reinforcements into your temple.
This would depend on how many units were recalled, since the player gets 1 PP for each unit recalled. I guess the counter to this though is that the temples provide free PP, where as the prayer points you get from recalling a unit will be spent in recruitment of the unit. Having to re-recruit these units also wastes an action that could have been better served elsewhere (e.g. getting def buffs from your pyramids to support the units at the temples).