A boardgame geek user, Denny, started a poll a couple of weeks ago to find the most hated things people do during gaming. It's interesting reading since the list is not ordered in the same way that I would and I'm always curious to hear what people care about. Some of these problems are also things that can be encouraged or discouraged by designing a game in a particular way, so thinking about them from that perspective seems like a good idea.
The top of the list mostly consists of criminal behaviours, such as stealing pieces, physically destroying the game or abusing other players. I'm pleased that these things made the top of the list, they deserve to be there, but they're not really something to be addressed from a game design point of view. If these things are problems in your gaming group, now would be an excellent time to seek a new group.
The only item in the top five that isn't illegal is cheating, which over 98% of players rate as unacceptable or borderline. I've got some fairly tolerant ideas about cheating, but they're all dependant on the consent of the group (i.e. a group agrees to the acceptable level of cheating in that group). If a group finds it unacceptable then I'd be just as quick to be rid of any cheats who showed up. I think that there is an extent to which game design helps here. People are far more likely to cheat by making a genuine mistake and failing to correct it than they are to make a deliberate mistake in a cynical attempt to gain an advantage. Similarly people are far more likely to interpret an ambiguous rule one way on their own turn and another way on their opponent's turn than they are to simply try to ignore or subvert the rules. In both cases the effect is the same, the players aren't playing by the same rules and one has an unfair advantage, but it's easier for people to fall into these behaviours. From a design point of view reducing rules ambiguities and making it less frequent for players to be doing complicated calculations in ways that can't be scrutinised by other players are steps that can be taken to prevent this.
Moving down to the middle of the list there are some personal behaviours that a game can't do much about. With the best will in the world I'm not going to try to write rules that require players to shower properly, though if anyone's heard of a game with a rule like "Most aromatically pleasing player moves first" that'd be a boon to passive-aggressive people everywhere.
The mid-list does have a few things designers could get cracking on. Over 90% of people hate it when people quit a game before the end, presumably this happens when people stopping having fun and any designer worth their salt should be interested in dealing with that. It's easy to get caught up in how it feels to win a game or to play it well in the manner that is intended, but it is more important to consider how it feels to lose a game. Most traditional games generate far more losses than wins, so games should be fun to lose. What's going wrong for players who quit? Does the game need to detect an un-winnable state and award a loss more quickly so a new game can start? Is a catch up mechanic needed to get back into the game? Could a secondary objective that doesn't award an outright win but that is still fun to pursue be implemented? The answer will be different for every game and some players will rage quit no matter what, but there's space for design to improve this issue.
Also high on the list is texting continuously (89%) which is part of a larger suite of behaviours associated with not being engaged with the game. A player who starts doing non game activity is a massive pain in the ass, laptops or tablets at the table is a pet hate for me, but different games encourage this behaviour to different degrees. Where a game has a lot of downtime, in which players can do nothing but watch other players take moves that are irrelevant to them, they encourage players to look for their entertainment elsewhere. Conversely when a game involves simultaneous play and high levels of interaction even the worst of distractible players won't get their phone out.
Control freak (84%) is a tough one to work with, since it can mean a lot of things: the organiser who insists everyone sits on their hands during the rules explanation, the alpha player in a cooperative game who dictates everyone's moves, players who insist on executing procedural rules that serve no purpose. All of these things can kill the fun of a game but require different responses from a designer so there's no one thing to look out for.
Jumping down to the stuff that 50-70% of players object to there are a few more problems that are beyond my reach. For instance, I can't imagine the game mechanic that prevents players from farting (Well I can, but I have a very active imagination and some of the things it produces could be considered war crimes, I can't imagine a practical way to do it.) The other features are aspects of problems addressed above, the same solutions that stop people quitting early act to stop them complaining constantly (75%) or not playing to win (58%). The things you do to address some types of control freak also deal with being a know it all (61%).
The behaviours that are acceptable to more than half of people generally aren't enough of a problem to provoke a design response, though perhaps that some people consider them a problem is something that ruins some people's day. There are a few that may be considered borderline acceptable because parts of the problem are considered the game's fault rather than the player's fault, such as analysis paralysis (49%) and arguing about rules (45%).
These problems are both best addressed by matching the players to the game. Some players love games with lots of options that produce interesting decisions, others will freeze up when they play them. Simplifying the game isn't necessarily the best option, as it might harm what makes it enjoyable for the first group, it's more an issue of making sure that the game clearly communicates its level of complexity so that it is played by people who'll get the most out of it. Similarly some games invite rules arguments, Nomic is nothing but an extended rules argument, this problem is more about a mismatch between players and rules. In general designers should strive for clear, concise rules, but if a game is heavily based on the interaction between a large number of parts it needs to communicate this so that it attracts players who enjoy puzzling such things out.
I've heard people argue that these things are not design problems, some players are jerks who will ruin the game for everyone, we shouldn't cater to them. We should focus on making games that are enjoyable for players who will play them in the intended way and to hell with anyone who would engage in these behaviours in a way that ruins the game for other people.
I don't hold with that line of thinking. Of course there is no level of sloppy design that makes a players behaviour cease to be their own behaviour, however I think that we should acknowledge that people are complicated and flawed. It's easy for people to write "If I saw a player behaving this way, I'd throw them out of my game" in the abstract, but when it's a person that you know who may have a great many positive qualities and one annoying thing they do in games it's not something that would really happen. Designers aren't responsible for player's actions, but where it's possible why shouldn't we write games that encourage people to play in ways that make the game more enjoyable for everyone?