The key part of the rules:
Each player counts the cards he has in each color. The table
below shows the number of plus or minus points a player
scores for the number of cards in each color.
It says nothing about adding colors together.