Yes. In this condition, you can ignore both retreats.
One wrinkle is if your unit suffers casualties. In that case, the leader has to make a casualty check. If he is killed, you don't get to ignore the flag.
This subtle order will matter once in a great while.
Basically, a unit can ignore flags for the following:
1) Unit Benefit (examples: Ru Line can ignore 1 flag; Fr OG two flags, etc.)
2) Attached Leader (who survives a casualty check)
3) Support (which could could partially come from a lone, adjacent leader)
4) Prussian Iron Will: 1 token/flag
5) Certain types of terrain
Like bonus combat dice, these simply add up.
There is also a Tactician card which allows you to negate flags.