We played this game for the first time with four players and it was an okay time...
Near the end of the game however, we were running out of cards in the draw deck sorely. We reshuffled several times and ended up drawing cards from a draw pile of a measley two cards!
The brief answer to your question is: Whoever had the lowest self worth didn't end the game.
Did you reduce your family members to 4?
"In a 4 player game, each player chooses 1 character to discard; this keeps the game from running long. To play with 5 players, make a new misfit family from these extra characters."
Assuming you did that correctly,
With 2 cards in the draw pile, I have to guess that there were a total of ~22 cards in the deck and in peoples hands (draw limit modifiers may change this, but without knowing I can only make an estimate)
you have have 4 family members.
If every player has 3 of their 4 family members dead
that leaves 10 untimely deaths in circulation (in hands and draw pile).
There are 11 event cards.
Making a total of 21 cards in circulation, so there may have been 1 or so modifier cards as well.
The question at this point is why didn't someone just kill their last family member and end the game? Whoever had the lowest self worth for their family should have just ended it.
Remember its not about having the lowest self worth. Its about having the lowest self worth on your _dead_ family members. So whoever takes out their last family member first has a whole extra character worth of points. Everyone else will just have fewer dead family members.
So keep you opponents family positive (or neutral) early on to keep them from killing them. Or kill them yourself while they have relatively small negative self worth.
Kill your own characters quickly. Personally I believe any character with -20 to -30 self worth is ready for an untimely death.