Yes, we took hats each time they were available. A couple times we used them to complete areas on the carts.
As it was a 2 player game, we used the "The original La Granja" variant. Roll 5 dice once and select from them. I guess we didn't roll hats very much and it seemed to bring the game to a bad end. I guess I won't use that variant any more.
Yeah IMO the basic rule of rerolling each round is better - and much shorter