What is supposed to happen towards the end of a phase when you end up having to barter? Say there are no more gem cards to be drawn, so this is the final round before final scoring. It is of no benefit to either of you to get gems that would not give you the majority of that color, but you do not want to give away enough gems to lose the majority either. I've played this game a few times, but that problem has always come up, and we usually just settle it by calling it as if 3 had picked the card in question and neither player gets anything, because no one is going to barter anything of real value away.
From the rules:
"Should the player who begins not wish (or be able) to bid any
gems, then the other player may bid a single gem, which the first
player is obligated to accept. Should both players decline to bid, then
the action is not carried out."
Using the above rule we never had a problem. If there's no benefit to the first player, he simply declares so and the other player either passes and the action is canceled, or gives any gem to execute the action.