Looking at these Villager cards I have 3 extra:
Do I just mix these in and play with all of them or do I randomly remove 1 traveler and 2 normal villagers?
The official setup is to make sure your starting Town Deck has 10 starter villagers and your Traveler Deck has 4 Infiltrators and 10 positive Travelers.
As long as you're not up against the Demons or Lycanthropes, you can have a slightly easier (both to set up and to win) game by simply throwing everything you don't put in your starting Town into your Traveler deck. If you want the great Villager variety this offers but want to try and maintain the difficulty this can be offset a little by introducing Cultists as necessary to ensure your Traveler deck is at least 4/14 Infiltrators in total.