"Pay one mana" could be from any source: crystals in your inventory, mana tokens you have gained somehow that turn, or dice from the Source.
Thanks. The reason why I was not sure is following situation:
The next mana token we had to remove was a white mana. It was my turn and there was a white mana dice in the pool which could really helped my opponent. So I used it to remove the white mana token from the fame board. But for us it was not 100% clear if this would be allowed.