It will certainly tend to make for a streakier game; for example, it's possible that all the good Star Empire cards will be in the bottom half of the deck that never comes to the surface.
But only you can decide if that potential problem outweighs the task of sorting out the decks.
Flip the problem and you find a solution. Sort the decks once and use a single trade deck for 2,3 or 4 players. I bet it will work well.