So, it seems that there are some players that feel like there should be more rules in the game's manual, explaining what can happen and what can not happen during play.
The general rule is the following: Unless it is said EXPLICITLY in the rulebook that you can NOT do something, everything is FAIR play!
This means, that if you want to use your castle to cover another player's castle, then it is allowed!
If you want to gather all gems of a specific type so that other's can not complete their goals, it is allowed!
If you want to not play a specific card (for example a Tavern from your hand), ever, and keep it in your hand to mess other people over, it is allowed!
So, if the rules do not specifically say that you are NOT allowed to do an action, then implicitly it is allowed. Although this game is "small" and looks "simple", it actually comes with a lot of strategy and in-depth mechanics.