I have noticed after several plays that I roll white side very often. I lost many figths and games as well due to this. I started to record rolls and last 5 games I rolled white side in 45-70%. I have experienced bad days in other games of course but not several plays in row. How are your rolls?
You can check out this video to learn how to test the balance of your dice.
Granted it's for a d20, but I don't see why it wouldn't work for a d6.
Then you could at least know if it's a balance issue or possibly something else like a rolling method issue as Patrick suggested.