I think you're right. From the Wikipedia entry for impartial game:
In combinatorial game theory, an impartial game is a game in which the allowable moves depend only on the position and not on which of the two players is currently moving, and where the payoffs are symmetric. [...] Games like ZÈRTZ and Chameleon are [...] not impartial.
By "the payoffs are symmetric", it seems they mean that both players have the exact same goal, just as they have the exact same allowable moves. In Nim, for instance, both players want to have the last move.
That's not the case in Yodd, since Black's goal is for the end position to have fewer Black groups than White groups, while White's goal is the opposite. It's similar to ZÉRTZ and Chameleon in that sense.
Another way to look at it is that, despite the pieces being undifferentiated by who can use them, they are still differentiated by whose goal they contribute to.