The rings would surly remove the color. Your agent-color idea should work.
(I see no real benefit in using the rings at all. I have a big agent card in front of me to tell me which mini I'm playing.)
The rings match the colors of the cardboard priority markers. Not all agents have a plastic 3D priority marker.