The rules say:
Taking (or repaying) Promissory notes does not count as a Trade action
(although you may only do so on the Trade phase).
Each Promissory note you hold at the end of the game reduces your final score by 3 VPs. It’s therefore a good idea to pay them back before the game ends.
Emphasis was mine.
My group plays that you can rotate the tiles during a move, I can't imagine that they're supposed to keep the same orientation.