There is a rule in OP that governs that.
Adventure Card Guild Guide wrote:
Don’t take actions that may harm another player’s character without that player’s consent.
So, if you are going to play a Goblin with those kinds of powers, you'll have to make sure such actions are ok with the other players. If they aren't, you'll be limited in what you can do or maybe should play another character.
I know everyone I've played with wouldn't mind, so long as you weren't going to kill them or push them to the brink of death. But taking a card from my discard pile or dealing me 1 damage wouldn't be a bid deal to me and everyone I've ever played with. I realize other groups my have different experiences though.
There was some similar discussion on the Paizo forums.