GeekGold Bonus for All Supporters at year's end: 1000!
8,897 Supporters
$15 min for supporter badge & GeekGold bonus
18 Days Left
Support:
Ted Alspach
United States Louisville Tennessee
Preorder One Night Ultimate Werewolf now from www.beziergames.com
One Night Ultimate Werewolf

A little insight into Bezier's Playtests for Colony (this same technique works for other games we have published, like the tiles in Suburbia or Castles of Mad King Ludwig, or the role cards in the One Night Ultimate Werewolf games).
Colony comes with 28 different Variable card types (in addition to the standard 6 Basic card types and each player's 4 starting card types). Of the 28 Variable cards, any combination of 7 can be used in a single game.
The number of Variable cards of each type is equal to the number of players. If you do the math, that's a bunch of combinations. However, that said, I can confidently say that all possible combinations (order is not important here, though it is suggested to display them in a certain order to make them easier to be initially accessible by players) have been tested thoroughly.
We have achieved this feat by providing a complete set of all game components (including, of course, all 28 Variable card types) to a number of remote playtester groups, asking them to choose a random set of variable cards to play with. Through this process we were able to determine if there were any issues with certain combinations of variable cards that were reported to us, and if so, we addressed those issuess, refining the Variable cards and resending complete sets to those playtesting groups until the issues no longer appeared.
During the final round of testing, some playtest sessions were never reported to us, per our instructions.

Gunther Schmidl
Austria Linz OÖ

toulouse wrote: During the final round of testing, some playtest sessions were never reported to us, per our instructions.
"Don't tell us if it doesn't work"?

Ted Alspach
United States Louisville Tennessee
Preorder One Night Ultimate Werewolf now from www.beziergames.com
One Night Ultimate Werewolf

gschmidl wrote: toulouse wrote: During the final round of testing, some playtest sessions were never reported to us, per our instructions. "Don't tell us if it doesn't work"?
No, *that* we needed to know. We didn't want to hear back for all of the play sessions with combos that *did* work. Because if we heard back from all of them, then this method would not work.

Eric Yorkston
United States Fort Worth Texas
I voted for Kodos

Hi Ted,
I'm eagerly awaiting this game, but I think your thoroughness in playtesting combinations may be a bit overstated.
If there are 28 cards and a any 7 can be chosen (and order isn't important), than isn't there 28 Choose 7 combinations? That would be 1,184,040 possible unordered combinations.
I'm also concerned that your methodology wouldn't be certain of testing all combinations without an even larger sample. If players choose a random set of cards to play with, then you are going to need quite a few more playtests than just the total number of combinations. This is due to some players randomly choosing the same set as one already randomly chosen. And this number also gets big really quickly. For example, if you only had 3 sets to choose from, the chances that 3 players randomly choosing different sets is about 22%. (I'm writing this off the cuff, but I think my math is right).
I'm still looking forward to this game and am sure it's going to be balanced!

Nikolas Co
United States New York New York

boredbeyondbelief wrote: If there are 28 cards and a any 7 can be chosen (and order isn't important), than isn't there 28 Choose 7 combinations? That would be 1,184,040 possible unordered combinations. For perspective, assuming the 45min playtime and a different combination each time, 10,000 copies would need to be played ~89 hours each.
boredbeyondbelief wrote: I'm also concerned that your methodology wouldn't be certain of testing all combinations without an even larger sample. Indeed, this is the coupon collector's problem. Good news: if you're fine with a 50% chance of hitting them all, then only ~34,483,173 plays are needed. So, that's ~2,586 hours (~108 days nonstop) of each of the 10,000 copies.
Of course, if you want 90% likelihood then you'd need 172,415,867 plays. (This all just applies the formula from the Wikipedia article with Markov's inequality. I haven't found or derived a tighter bound.)

Ted Alspach
United States Louisville Tennessee
Preorder One Night Ultimate Werewolf now from www.beziergames.com
One Night Ultimate Werewolf

boredbeyondbelief wrote: If there are 28 cards and a any 7 can be chosen (and order isn't important), than isn't there 28 Choose 7 combinations? That would be 1,184,040 possible unordered combinations.
That is the correct math. And they have all been tested.
I'm waiting for someone to jump in and explain why based on my description.

Ted Alspach
United States Louisville Tennessee
Preorder One Night Ultimate Werewolf now from www.beziergames.com
One Night Ultimate Werewolf

NikolasCo wrote: For perspective, assuming the 45min playtime and a different combination each time, 10,000 copies would need to be played ~89 hours each
Yes, conventionally. I used a much better, unconventional methodology that theoretically requires a tiny fraction of that time.

Y P
United States Mississippi

toulouse wrote: boredbeyondbelief wrote: If there are 28 cards and a any 7 can be chosen (and order isn't important), than isn't there 28 Choose 7 combinations? That would be 1,184,040 possible unordered combinations. That is the correct math. And they have all been tested. I'm waiting for someone to jump in and explain why based on my description. Tested programmatically by an application instead of through human test plays?

Ted Alspach
United States Louisville Tennessee
Preorder One Night Ultimate Werewolf now from www.beziergames.com
One Night Ultimate Werewolf

MentatYP wrote: Tested programmatically by an application instead of through human test plays?
Nice guess but no....it would actually be nice to do that, and if you KNEW you were going to an appbased version of the game, you could simulate those plays, but you'd have to program the AI in a way that has it looking to break the game in a huge variety of ways, and doing so would probably reveal the breaks as you were programming.
You'd probably need an AI to program another AI.

Y P
United States Mississippi

toulouse wrote: MentatYP wrote: Tested programmatically by an application instead of through human test plays? Nice guess but no....it would actually be nice to do that, and if you KNEW you were going to an appbased version of the game, you could simulate those plays, but you'd have to program the AI in a way that has it looking to break the game in a huge variety of ways, and doing so would probably reveal the breaks as you were programming. You'd probably need an AI to program another AI. That's what things like neural nets and genetic algorithms are for But yes, you'd essentially have an AI system program the AI to look for the cracks. You'd just need enough processing power and time to test all the possible combos (he says as if such a thing were easy to obtain).

Todd Fast
United States Mountain View California

toulouse wrote: Yes, conventionally. I used a much better, unconventional methodology that theoretically requires a tiny fraction of that time.
Without knowing the content of the 28 cards and how they can be represented mathematically (e.g. are many of them mutually exclusive, or just variations of quantity or degree?), it's hard to imagine a method that would work with the degree of rigor you claim. Extraordinary claims require extraordinary evidence, so I think the burden of proof is on you.
Until then, logic dictates that we assume the game is broken in ~1,184,040 ways.

Nikolas Co
United States New York New York

toulouse wrote: boredbeyondbelief wrote: …That would be 1,184,040 possible unordered combinations. …they have all been tested. I'm waiting for someone to jump in and explain why based on my description. I don't think it's possible without making some assumptions…
If we assume that altering the number of variable cards used is OK for the types of problems you want to find, then you could test smaller sets, or big ones. Therefore, you'd have fewer combinations to test. A very simple approach is to use all 28 variable cards when playing. It won't identify some problems that could arise with a subset of 7, e.g. one card (or a combination of cards) being needed to counter another one.
(This means asking them to "choose a random set" is a vacuous step, of course)
Another approach I thought of is considering some cards to be equivalent, or not interacting with each other. That seems easy to screw up, especially as you modify the cards.

Ted Alspach
United States Louisville Tennessee
Preorder One Night Ultimate Werewolf now from www.beziergames.com
One Night Ultimate Werewolf

Okay, since it's been a day, and no one has jumped on this with the right solution...
Quantum mechanics. This is Schrodinger’s cat but with infinite states instead of binary ones. The key here is that the playtest sessions that were not reported (observed) consist of all possible combination of cards...until they are reported (observed). Only when I am informed of the actual combination does the playtest event have a specific combination that was used. Using this methodology, I could have simply given the game to a single group, and without them reporting the combination that was used, as long as there were no issues, all current possible combinations would be considered having tested (unless someone happened to mention which combination was used).
This is a very efficient method of playtesting...unfortunately, it isn't very practical (though you could probably determine the statistical likelihood of all combinations being errorfree after a certain number of unreported turlyrandom playtests).
All of that said, the game was indeed playtested several hundred times with real combinations of cards, but the focus was on card combinations that were more likely to possibly break the game in a variety of ways. Certain cards (some which were removed from the game during early stage playtesting) were requested to have been added to combinations with a random set of other cards, to ensure that those cards didn't have a detrimental effect on gameplay.

Gunther Schmidl
Austria Linz OÖ

Easier way would have been to quantumcollapse the reality in which all cards *have* been tested, the game is out and sits at the #1 spot of BGG with a 9.99 rating, though...

Ted Alspach
United States Louisville Tennessee
Preorder One Night Ultimate Werewolf now from www.beziergames.com
One Night Ultimate Werewolf

gschmidl wrote: Easier way would have been to quantumcollapse the reality in which all cards *have* been tested, the game is out and sits at the #1 spot of BGG with a 9.99 rating, though...
That's an excellent point. Although that presupposes that the game has already been published, which means there was some sort of split back about a year or so ago. And I'm fond of this particular reality...

Eric Yorkston
United States Fort Worth Texas
I voted for Kodos

Think of all the time and money that could be saved playtesting games this way! You only need to send one copy of the game to one group. As long as you don't hear back from them with any complaints, then the game must be perfect! In fact, completing multiple play tests is just sloppy and irresponsible design as it just increases the likelihood of the quantum manufacturing of unbalanced cards and imperfect games.
That said, in this universe, I don't know why we would need playtesting at all. Obviously, all possible game combinations exist in Mr. Alspach's mind. He has already ran through all possible combinations and only reports those he knows will be successful. (We can assume that the imperfect games are unreported and remain unpublished). The money saved by being able to skip the playtesting step should be passed through to the consumer (or at least to this avid consumer of Mr. Alspach's games).
I guess this is what I get for not assuming quantum mechanics hold above the quantum scale....


