The confusion came from the PoG Guide in the files section, because that one says:
'units defending in an undestroyed fort space add the CF of the fort to their CF sum'
It makes sense not to add it but first time it happened we added it because of that paragraph in the guide. Better read the rule book more carefully
I tend to not use guides for any game unless I have checked it for that reason. Most times I don't bother and don't use them.