It's really up to you. The problem with mixing them all is that you may get all high ones or all low ones instead of a more even distribution. But I believe the design intent is to just mix all of them (including ones you from expansions you aren't using) and draw from that. It won't break anything as you still have to roll the dice for results and a "1" could be more dangerous than a "3" at times.
The answer seems so blurred.