It's more the rules layout I don't like. It took me a long while just to set up my first game because they have information scattered. I'm used to other board games where they have a list of steps to get you playing. This one required flipping to multiple pages to cross reference information. The rules are very good once you know how to play and just want to look something up.
That's typical of wargame rules. You're right that SE4X and some of their other games could probably ease the barrier to entry by packaging them more like a FF game (I think SE4X would rock Eclipse and Twilight Imperium fans if it were pimped out with a lot of plastic bling instead of cardboard counters), but GMT primarily publishes wargames so their games and rulebooks look like it.