Morten Monrad Pedersen
Here's what the rulebook says about the mill (page 18):
Whenever you take the Produce action, the Mill may produce as if it were 1 worker.
and here's a quote from page 25 about factory actions:
Even if the Factory card action does something similar to another action on your Player Mat, they’re completely independent of one another. So you get neither Recruit Ongoing Bonuses, Structure bonuses, nor coins from similar actions on your Player Mat.
Thus it's only for the produce action on the player mat that you gain the mill structure bonus.
EDIT: Fixed quote code.