So the way we play it is that the tokens only generate +1 if the colour of the token has been drawn as a card? Is this correct?
It is not necessary to draw a card in token's colour for the token to give you a bonus. You just need to draw any card that will cause a "successful" activation.
There is an example for this in the rulebook (Example 4: Chapel of Mars).
To simplify that example: Ley's say, a card lets you pray and then gives goods for pink and black cards. If you draw, for example, 1 pink card, it counts as a success and your black token (as well as a pink token) will give you +1 good.