It is any bonus/penalty to toughness at the time you trade it in. Plain and simple. No need to remember anything.
Frankly, the level of book-keeping necessary to use any other approach than using the current Toughness would quite ridiculous. Use the current Toughness. Trust me, over a number of games, it'll all even out (at best!)
That's what I thought, but my co-players disagreed, until I explained that it works both ways.
Thank you for your answers.