If you're using the latest rules published here
, on page 11 there is a rule which should be adjusting the number of Map cards and thus - the difficulty, according to the number of players.
This rule increases the chance that you would find the numbered Settlements sooner in the game, if there are less than 4 players.
Otherwise, if you haven't used this rule, the game should be easier for more players, since this increases the chance a beneficial Map Card to be found and used, and since the End of Turn population decrease happens only after everyone has played their turn.
However, playing with only 1 or 2 players may give you more focus and allow you to be more effective. It all depends on your decisions and, well, your luck