swift learner seems to be a strong perk. Is there a reason not to pick it if you have the choice?
Depends on what stat you drew with Intelligence upon leveling up. If you really want a specific perk of that stat or if you have a sweet weapon and that's the exact stat you need for another reroll you may choose to keep that stat instead of trying your chance with Swift Learner.
Early to mid game it is always better to get Swift Learner probably though.