Remember: "Plantation" does NOT equal "Region."
"Building a plantation" is putting 1 tile on the board. Thus, each tile placed is considered; you do not "build a region" which would be putting multiple tiles on the board together.
So, when building a plantation that would join your region to a neutral region, you must first check that your region is at least as large. It's in the rulebook.
Adding this rules quote for supporting reference (my highlight)
You may build a plantation that connects one of your regions to a neutral region of the same color, provided that your region is at least as large (number of tiles) as the neutral region before the merger. If the neutral region is larger merging is off-limits.
(saw the same thing today with an opponent. came here to verify)