Saboteur is the screw your neighbor type of game. If you eliminate the inside competition of who gets to the gold first, then you tear away a big chunk of fun. It's every gnome for himself. As a non-saboteur you only help the group to help you get to the gold.
Indeed, we even had a 3-player game once without a saboteur and didn't reach the gold because we screwed eachother to be first and nobody dared to lay down the sixth card. (the difference between the first and other diggers can be huge if you draw a 3 + 1 + 1 + 1 gold)