Recommend
3 
 Thumb up
 Hide
27 Posts
1 , 2  Next »   | 

BoardGameGeek» Forums » Board Game Design » Board Game Design

Subject: Dealing with feedback: how do you decide when to listen and when to ignore? rss

Your Tags: Add tags
Popular Tags: [View All]
David Goh
Singapore
flag msg tools
designer
badge
Avatar
mbmbmbmbmb
Hello all,

As someone who's still relatively new to the board game design scene, I've found the art of handling feedback to be one of the most challenging aspects of the design process, because of the sheer amount of considerations that come with it. There are so many variables to account for, such as the background of the playtesters, their number of games played and their preferences, as well as your own data collected, perspectives and biases...

So my question to you is this: how do you decide what parts of your playtesters' feedback to listen to, and what parts to ignore?

I'll start out by sharing an article I recently posted on my blog — it details an experience I had on dealing with feedback while I was designing Endogenesis. In it, I cover the parts of the player feedback I chose to listen to, and the other parts that I ignored, as well as the rationale behind doing so.

Here it is: http://endogenesis.cards/blog/2018/05/25/game-design-2-liste...

I hope someone finds it useful, and I would love to hear about your own perspective on this topic.
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Reiji Kobayashi
Japan
flag msg tools
I can't remember the source, but I recall someone stating that the game designer should pay attention to what the playtesters felt (in the case of your example, that the game could be long and boring) and not their suggested improvements.
14 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
David Goh
Singapore
flag msg tools
designer
badge
Avatar
mbmbmbmbmb
That's a good way to do it, and it's definitely what happened in my case!

Personally, I place a lot of value on understanding the first impressions formed by testers, and those tend to be attributed to how one "feels" about the game on an instinctual level — these reactions are the most genuine and usually reveals the a lot.

I think there's value to when testers suggest improvements, even if they can be distracting at times. Especially from playtesters who are veteran boardgamers, as they'd draw from their experience from other games to provide solutions, and sometimes leads to viable options.
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
David Goh
Singapore
flag msg tools
designer
badge
Avatar
mbmbmbmbmb
8Oj4N wrote:
Before I was hitting gym every week I used to listen all the feedback. Now, when they start nagging ''If i were you I would change that, and that'' I just put the hand on their shoulder like Bane in DK3 and ask them:

''Do you feel in charge?''.

But, seriously, first impressions could be very misguiding. Usually after playing 3-4 games with the same person he/she would have a much better formed opinion about it, and that's the time you need to listen.

I should try that See how being physically intimidating affects tester feedback!

Thanks for your input! You're right, and I think I should clarify what I meant by "first impressions." I've been lucky enough where whenever I show my game with new testers, I have them sticking around for a few games. After which, I'd then talk about the game with them and gather their "first impressions" (which, I suppose, isn't really first impressions anymore.) But yes, even then, I hardly make any significant changes to the game based on that feedback... If anything, it's more of a litmus test over whether or not I've achieved certain goals in the latest draft of the design.

 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Warren Fitzpatrick
United States
Ohio
flag msg tools
Avatar
mbmbmbmbmb
I "listen" to everything. I implement one, maybe two changes per iteration. Sometimes, the new iteration has the suggestion; frequently, it does not.

My go-to rule is to keep an ongoing log of suggestions, whether I agree with them or not. If I hear it from one person, and don't see it as a good idea, I still have a record. If another, unrelated person, plays and gives me the same feedback, then it moves from "not a good idea" to "let me consider this more strongly". Then, it's likely to hit the next iteration or two. And if I hear it a third time, I know I'd better make that move to at least try it and see the results.

I also consider the source -

Is the feedback predominately because of their character (trying to buff a character's asymmetrical powers is a frequent thing I see)?

Are they a player of hobby games or are they a casual player?

Do they have a history of designing games? (which means they've done this development thing before)
8 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
James Naylor
United Kingdom
flag msg tools
designer
Avatar
mbmbmbmb
Nice post!

Those are some great answers already, so I am not sure how much I can add, but I'd take it right back to 1st principles.

Ultimately I'm going to assume your main aim is to create a game that your target market will like so much they will buy it. Is that fair?

If so then this is my personal take (with a the big caveat that while I have done many other kinds of design, I am not yet published myself!):

I would say your primary interest in playtesting is in seeing how much your game creates that engagement, not in the feedback at all.

You want to know what, at different points, gave them pleasure and frustrated them. How they got animated at some points and bored at others, what they enjoyed or did not enjoy doing. You are in the great position that your playtesters can provide an account afterwards of their experience that might be hard to get from just observing them (and critical if you are blind testing, of course). They might, as a bonus, directly give you some good ideas too. You might be lucky that they are a hyper-self aware and very seasoned game developer or designer with an excellent track record but you couldn't rely on either of these things for the test to be useful.

So then to sort the wheat from the chaff, I would listen closely and intently to their account of their experience, with the consideration of context which Warren mentions.

But I would disregard their suggestions entirely: unless you decide, totally for yourself, that a change creates the kind of experience you want for the kind of person you are aiming at, better than what you have now.

That was probably no help at all
5 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
f s
Germany
flag msg tools
mbmbmbmbmb
You should have in mind the possibility that they have not actually playtested your game and are telling you stuff that is based on skimming the rules and maybe sorting through the game pieces once. (This is what I have often witnessed with playtesters).

While the first impression is important, what is even more important is how the game works middle and long term. And it is very hard to get anyone to actually play your prototype more than once. (Even with published, fully produced games, many games seem to not make it past one play with the groups that I know).

Also, keep in mind that some people will lie to you about how often they have played your game.

As for the actual feedback: Listening can't hurt. If a game is rejected by a considerable percentage of playtesters, it is probably not worth pursuing further in the present form - regardless of whether they actually played it or not.

I also would recommend listening to ideas for improvements and especially for ideas on how to make the game easier to understand and easier to keep up with.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
David Goh
Singapore
flag msg tools
designer
badge
Avatar
mbmbmbmbmb
Thanks for the great answers!

warrenfitz45: Considering the context in which feedback is given is definitely important. Especially for gamers, their preferences can result in very different types of feedback (e.g. people who love dice games might want more randomness, while MTG fans might prefer if randomness was kept to a minimum)

JamesDN: I agree that for playtest sessions that a designer can personally witness, there's a lot to learn from observing player's reactions and body language to gauge their level of engagement. At the same time, I feel that the physical presence of the designer can sometimes make some testers pressured to act in a certain way (perhaps out of sensitivity to the designer) and therefore be less honest. I suppose that the best of both worlds would be being able to witness test sessions without the testers knowing that you're the designer.

In the example I gave in my blog post, if I were physically present for the playtests where the issue turned up, I believe I would've figured out the cause of the problem much faster. Perhaps I can look into asking my testers to record videos of their sessions? That could be helpful, I think.

Si Fei: I guess I've been pretty lucky with my testers! They're generally personal friends that I trust, who I sometimes lend prototypes to so that they may playtest with their own groups. From experience, I also agree that suggestions on increasing accessibility tend to be more helpful versus say, suggestions on how to balance the game.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Eric Brosius
United States
Needham Heights
Massachusetts
flag msg tools
badge
My favorite 18xx game for six players is two games of 1846 with three players each.
Avatar
mbmbmbmbmb
I've heard Tom Lehmann say what Reiji said above. If your playtesters say that something is bothering them, this is almost surely true. If they tell you what the solution is, they're often wrong.

Tom's interview here is worth listening to:

The Long View: Tom Lehman Interview

And David, remember that if your game is published, all sorts of people will try it -- not only the savvy ones like your group, but also people who don't understand things that you think everyone should understand. And most of them will try it without you around. So you want it to be robust under various play situations.
10 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
f s
Germany
flag msg tools
mbmbmbmbmb
Eric Brosius wrote:
If they tell you what the solution is, they're often wrong.

But so is the designer.
Obviously, the choice ultimately lies with the person who decides (the designer). That does not mean that listening to ideas is a bad idea, especially if they are good ideas.

And then, of course, I do know about those people who will always suggest that it would be totally awesome if you would include their favourite pet whatever. Do not listen to them.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Haberack Shninklebottom
United States
Minnesota
flag msg tools
Hello! Without reading the other replies my opinion is that taking feedback is purely based on how complex the feedback is. For example, if you get two messages about your game, one says --- aspect is overpowered, the other message says ---- aspect can be powerfully used with --- aspect, I would suggest tweaking ----. You can see that one is (dare to say) better.

In short, if you have to choose, go with the tester that takes your game as serious as you do. Got a bad review? Don't despair! Some people will love it some will hate it. But if your bad reviews outnumber your good ones...
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
David Goh
Singapore
flag msg tools
designer
badge
Avatar
mbmbmbmbmb
The Haberack wrote:
But if your bad reviews outnumber your good ones...

...then it's back to the drawing board

I like the idea of taking feedback more seriously if based on its complexity. It's great when playtesters are able to offer feedback that's packaged in a useful manner, but otherwise, it's up to the designer to do a bit of coaxing/investigating to find out more. (e.g. when an aspect is overpowered, why is it overpowered? What was the specific experience that made you feel like it was OP, etc)

Eric, thank you for the link, and for your words of advice. A game that's broken when its designer isn't around to explain how it works is, well.. just broken. (But fixable!)
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Warren Adams
Australia
Mt Lawley
Western Australia
flag msg tools
badge
Avatar
mbmbmbmbmb
Listen to it all.

Then decide what is useful.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
James Naylor
United Kingdom
flag msg tools
designer
Avatar
mbmbmbmb
@David - I think that's a good point you make about it changing behaviour.

But I guess I'm thinking about the stuff that's much harder to fake: let people's eye movements, facial expressions, body language, reflex actions (like picking up their phone), frequency of asking questions etc. Most people I test with (I don't think) are so hyper-aware of themselves they can control this sufficiently to please a designer, even if they verbally suggest that they've had a different level of engagement.

I think video is a really interesting solution to blind-testing. But even without that this thread is making me think about the ways you could try to capture people's own sense of their own engagement more effectively without lapsing into too much analysis of your game (which is the bit I think will be mostly less useful unless your playtesters are really experienced).

A thought provoking thread. Thanks!
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
David Short
msg tools
I think this goes along with what other people are saying but might be a slightly different perspective...

I would at least be aware of what everyone is telling you, especially the issues, and I would, also, read the suggested fixes. Not because that is the direction you will take, but because they can give you insight into what the playtester is actually taking issue with. It might not even be the 'problem' that they stated!

Nervously dipping my toe in the water,

E
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jeff Warrender
United States
Averill Park
New York
flag msg tools
designer
publisher
badge
Avatar
mbmbmbmbmb
I have the good fortune of testing in a designers’ group, and all its members are better designers than I, so I tend to take, or at least attempt, quite a high percentage of the suggestions they make. That said I think the following is generally accurate:

If testers say:

I couldn’t see what I was supposed to be doing
I found [this aspect] frustrating; I couldn’t do what I wanted to be able to
There are too many things going on to keep track of

they're usually right. Listen to them!

If they say:

The rules made no sense
I didn’t feel like my actions determined my outcome

they may be right. Sometimes designers are actually fairly bad at teaching their own games, though. Consider how your rules presentation may cause some of the confusion.

If they say:
[This] seems way over/under powered
Have you tried [this outlandish suggestion]?

they’re probably wrong. If you’ve worked on the game for a while it’s unlikely a first time player understands immediately that major flaws or major changes are evident. Consider these cautiously.
13 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
David Goh
Singapore
flag msg tools
designer
badge
Avatar
mbmbmbmbmb
Jeff: That's a really good breakdown, thank you!

Stuggling to teach my game well is something I definitely identify with... Especially with new testers, I try to work off a checklist of points to cover so that I don't miss out anything.

From your list, I see a trend where the more specific a feedback is — when the tester zones in on a detail in the game, such as a card or character they believe to be overpowered — the more caution is required in considering it. Whereas for feedback that's general, such as their feelings over a game's mechanic, that's usually safe to listen to. This makes a lot of sense!
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Mark Schwab
United States
San Jose
California
flag msg tools
Avatar
Was just listening to the Board Game Design Lab Podcast with JR Honeycutt (starting at 14:06):

JR Honeycutt wrote:
One of the big things that I've learned as a developer, and it hasn't always been like this, but it's one of the takeaaways that I've learned, is to never ignore playtest feedback.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Theodore Karvounis
Greece
Moschato
flag msg tools
designer
badge
Avatar
mbmbmbmbmb
One problem that I have faced and thought a lot about, is when a game is quite heavier than the playtesters. Usually, they can't pay attention to a 15 minutes rules explanation and, as a result, they misinterpret some vital rule. Or they can't process the complicated decisions tree. Ultimately their suggestions are almost useless. Of course, that can be a designer's fault too, maybe he did not explain the rules in a clear way, or the rules themselves are super complicated. Not always though...

Having said that, I have also found that even useless/totally wrong feedback can be helpful. It makes you reevaluate things, it forces you to try to find more ways to improve the rulebook and most importantly to stop thinking that the game is finished. Shaking things a bit many times something useful is revealed.

My 2 cents...
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
marc lecours
Canada
ottawa
ontario
flag msg tools
mbmbmbmb
8odoros wrote:
One problem that I have faced and thought a lot about, is when a game is quite heavier than the playtesters. Usually, they can't pay attention to a 15 minutes rules explanation and, as a result, they misinterpret some vital rule. Or they can't process the complicated decisions tree. Ultimately their suggestions are almost useless. Of course, that can be a designer's fault too, maybe he did not explain the rules in a clear way, or the rules themselves are super complicated. Not always though...


Heavy games with very complex rules add a whole different element to play testing. In addition to testing to make sure the game works, and testing to make sure the game is interesting and testing to make sure the game is balanced, you have to test to make sure the game is reasonably easy to learn. You can't test for other things until the players learn the correct rules. Sometimes you just can't make a game simpler without losing the essence of the game.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Fertessa
United States
Georgia
flag msg tools
badge
Avatar
mbmbmbmbmb
For me, I keep in mind the reason I made my game, who my audience is, and the reaction I want it to get. I made Book of Villainy, because I wanted a game where I could be a villain, and my friends and I could mess with each other and play up being villains. (Kind of how we would get SUPER into our roles as duelists when we played Yu-gi-oh/Duel Monsters). I wanted it to be fun, and easy to pick up by casual to none gamers, where they could start playing within 15 minutes of explaining the game. Most importantly I wanted my game to encourage mischief and laughter.

So when I playtest, it's clear to me how near or far I am from that goal, without my testers having to say anything. I write everything down, but what I really listen to are the suggestions that they get excited about, to see what would feel fun to them. Then I go back and mesh everything together to see if the next iteration can get closer to that feeling.

I'm relatively new to TTG and game design in general, so my knowledge of mechanics and gameplay execution aren't strong enough for me to focus on those and get my game to where I want it to be. I've found that by chasing the fun, my brain has been able to filter the comments from playtesters, as well as my own ideas to get me to the place I need with my game.

As for ignoring, while I do write down every suggestion, no matter how ridiculous, I will ignore the ones that seem arbitrary. In almost a year of playtesting however, I can only recall 2 testers (a couple) whose feedback I didn't put much stock in. Oh wait...and that one guy who didn't play my game, but asked where the chest was on my character, because she was too flat-chested to be a woman. shake

TLDR; I record everything, but I filter based off the reactions I know I'm looking for from playtesters.

1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
David Goh
Singapore
flag msg tools
designer
badge
Avatar
mbmbmbmbmb
8odoros: I find what you said about useless/wrong feedback being helpful to be quite interesting. In the example I spoke about in my blog post, the suggestions in the form of feedback that I received only served to distract me from the real problem, but it did push me to re-visit my older playtest and change logs, and ultimately got me thinking about how it can be a challenge to handle feedback properly. So in a way, it did turn out to be helpful after all, having contributed to my growth as a game designer!

Shintotchi: Thank you for sharing your thoughts! For someone who's relatively new to game design, I think you're starting off on the right foot by chasing the fun. It reminded me a lot of how I make my prototypes too — initially, I'd approach it with the mindset of wanting to just make something that's fun that my friends and I can enjoy. If they love it, great... and if they don't, I'll do something else. And if they think it's something that can be published, then I give it a shot working on it, and expand it to more playtesters to see if others like it too. But at the core, it starts out by just wanting to make something that my friends and I can have fun with while hanging out. It makes for a very stress-free creative process.
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Brandon Vi
msg tools
Avatar
mb
In the Board Game Design podcast I have heard Brian and Jeremy say that the best feedback comes from the feelings players display and not what they tell you. I have found that advice extremely helpful.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
f s
Germany
flag msg tools
mbmbmbmbmb
Soap1 wrote:
In the Board Game Design podcast I have heard Brian and Jeremy say that the best feedback comes from the feelings players display and not what they tell you. I have found that advice extremely helpful.

I find that advice offensive.

It is a misled idea that the game designer somehow knows better what the player would like than the player himself does.

IMHO the truth is that some players will give very helpful suggestions and some will not. It is a bit similar to the answers that you get when you post a discussion topic or a question on BGG.
Following the BGG analogy, you also need to watch out for dynamics. A single vocal (and maybe popular) comment might influence others in their opinion. It is a good idea to try to see through that.
1 
 Thumb up
0.01
 tip
 Hide
  • [+] Dice rolls
Adrian Pillai
France
Paris
flag msg tools
designer
badge
Avatar
mbmbmbmbmb
Si Fei wrote:
Soap1 wrote:
In the Board Game Design podcast I have heard Brian and Jeremy say that the best feedback comes from the feelings players display and not what they tell you. I have found that advice extremely helpful.

I find that advice offensive.

It is a misled idea that the game designer somehow knows better what the player would like than the player himself does.

I'm not offended, but I also don't agree it's the feelings players display that matters, though I'm no psychologist.

I find it more about the feelings players convey. How they convey it might also be a clue - if someone is clear/sure about their feelings, you know the game/problem must have made a very strong impression. And if someone is vague or unsure of their feelings and they struggle to put it in words, then more sleuthing is required.

Some of my best feedback has come from people's vague conveyance of feelings rather than the strongest. The 'there's something off' feeling is much more unsettling than 'this is broken', and can really sink a good piece of work.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
1 , 2  Next »   |