No. The sentence means that the barbarians will have the territory if the occupying player abandons it. The barbarians behave just like players' discs and are never removed. When you conquer one of the barbarians' territories, you just stack your disc over theirs.
Thanks for your reply. So, can I replace "take it back" with "remain"?