Everyone play the white dice or not. Everyone done? Fine. Now the active player can choose colored dice or not. If someone decides to change a previous action as a consequence of a subsequent action, well that’s just wrong and this person should not insist on doing this.
Indeed, you should also pay a bit of attention to what others are doing and if they can close a row, then you just need to take some risk by taking X.