I haven't actually played this game yet but have it on order and have been studying the rules. However...
Often in the game I will push the die down to a lower number only to have it taken immediately by the next player. Frusterating! They get paid to get what I paid to lower!
Isn't it true that when you exchange a die with another player that you have to exchange a die of the same value as the one you are giving up? In which case exchanging a die is only useful for changing the colours of your home dice and not for lowering the value of them (something which, as you say, you have to pay to do).