Back in 2016 I read an interesting remark that I stumbled upon again recently. The way I read the remark, it implied that customizable bots are objectively better than non-customizable ones.
On the surface it makes a lot of sense. Customizability should give higher replayability and an experience that’s tailored to your specific preferences, right?
Well, I both agree and disagree with those two latter statements, and I strongly disagree with the claim that customizable bots are objectively better than non-customizable ones.
To me it’s a subjective trade-off between different kinds of play experiences and I’ve always made conscious decisions in favor of less customizability because it helps to achieve the kind of play experience, I want to give the player.
Right or wrong aside, I think that the remark leads to a discussion about several topics that are important not only to bot design but board game design in general.
Before we get started, I'll note that while I mainly talk about bots in this post, then most of it is equally applicable to games.
Customizability for your first play(s)
When I start playing a game with a bot, I don’t understand the game and the bot very well and so customization decisions I make are likely to lead to a significantly worse experience than one that’s crafted by someone who intimately know the system and has gotten a ton of feedback on it.
If I’ve just learned Newtonian physics, then the first catapult I build will be worse than one built by a master catapult builder .
Not built by me. Image credit: Angeless
To make the point by exaggeration: The ultimate customizable bot is the one that isn’t there but is instead fully left for the player to make. I don’t think, though, that no bot is the best “bot” to “include” in a game .
This exaggeration is of course silly, but it establishes that that more customizability isn’t always better than less. So, a claim about the objective superiority of customizability is invalid instead it’s a matter of where the cutoff point is between customizability being better and worse.
Decreasing customizability of Living Card Games
Fantasy Flight Games (FFG) is well-known for their extremely successful Living Card Games (LCGs) that have a high degree of customizability via deck construction done before the game starts. Over the releases of their cooperative LCGs (which are the only ones in print) they’ve reduced the customizability.
They’ve gone from 100% free form in Lord of the Rings (2011), over Arkham Horror’s (2016) that AFAIK have a few specific cards always included for each hero and significant card restrictions based on the hero’s class, to Marvel Champions (2019) where each hero has 15ish out of 40-50 cards that’ll always be in their deck.
It’s interesting to note that on the 2021 People’s Choice Top 200 Solo Games the games are ranked by solo players, the least customizable of them is ranked highest and the most customizable one is ranked lowest least. The latter is still ranked high, though, and there can be many explanations other than customizability for their relative rankings.
2The customizability of Lord of the Rings is so high that many players, me included, get paralyzed or must spend so much time doing it that they stop playing the game or go to websites where skilled players list preconstructed decks that require no customization.
That said even the least customizable one, Marvel Champions, is very customizable but this is alleviated by the fact that the customizability is to a large extent part of the strategy of the game like in deck builders with the deck being constructed before playing instead of during and the customization options were created gradually via 30 expansions with room for years of playtesting.
When I’ve played the game I’ve changed heroes to vary the game experience while the deck construction has been a matter of strategy, which I don’t consider customization but maybe I’m splitting hairs.
To make such highly customizable games balanced, which they’ll never fully be, a ton of playtesting is required and by board game standards FFG is a huge company and the games make tons of money which combine to allow enormous resources for playtesting again and again and again.
In the ideal world all game designers have infinite time and an infinite number of playtesters each spending infinite time but that’s not how the real-world works. FFG with their size and resources is significantly closer to the ideal than me or probably anyone reading this post, and this has a big impact on the feasibility of customization.
I’ll use the Scythe Automa (Automas are my kind of bots) as an example. It had 302 external playtests logged by 32 external playtesters. Let’s say that we in the design team did 198 plays in total
(probably not too far off) that takes us to a nice round 500 playtests by 35 people.
The Scythe Automa playtest results that were logged during the external playtest. A lot of our own results weren’t logged here.
The ~200 plays that were made by us with intimate knowledge of the system tested many aspects of the game in depth, but our testing had severe limitations.
Given that we aren’t superhuman we are limited in what we test because we know how the game and Automa are intended to be played and so there’ll be many things we simply won’t test because we’ll overlook many possible options and situations.
Additionally, there are mechanisms and edge cases we take for granted and therefore don’t notice aren’t covered by the rulebook or aren’t explained well.
Even if the burden of knowledge wasn’t there, we would still be limited by us only being three people with overlaps in our way of thinking about games and Automas and who have unwittingly create groupthink.
This is where the playtesters come in because they aren’t burdened by our knowledge of the game when figuring it out and deciding how to play it and they’re not limited by our groupthink.
So, they’ll check a wider and, in some cases, deeper part of the Automa, game, and strategic space.
500 plays by a combination of designers and players with less knowledge but no preconceptions provide for solid testing of the Automa and strategic space.
But let’s say that our team had made two significantly different Automas for the game that players could choose between. If they’re completely different that takes us down to 250 tests of each. If there were four Automas, we’d go down to 125 for each.
The Scythe Automa is controlled by a deck of cards and customizing that deck would be an obvious way to customize the Automa’s behavior. Let’s say the deck was constructed by the player by selecting of 10 out of 20 cards. That would take us our 125 down to less than 0.0007 tests of each configuration. Which I think we can agree is a less than ideal number .
Based on my experience, 3 designers, 32 playtesters, and 500 playtests is significantly above the average numbers for board game bot development, which means that in many cases the test coverage above would be even lower.
The Scythe Automa components. Image credit: Philip Mazzone
Difficulty levels and game variability
I’m a great believer in having difficulty levels for bots. The Scythe Automa has 4, which turns the less than 0.0007 tests per configuration down to less than 0.0002.
It’s important add that customization options for bots comes on top of the customization of the game, which all things being equal means that they can have significant less customizability than a game or test coverage will become even more of a joke.
Scythe has 25 player mats configurations and 5 Automa factions (one of which will be blocked by the player’s choice of faction). Adding these option takes us below 0.000002. I.e. less than 1 test per 500,000 setup configurations and that’s not counting the setup configurations from the personal goal cards, which takes us into the double-digit millions.
The cost of playtest dilution
If I measure the weight of 500 random Danish men who are 47 years old, then I can get a decent estimate of how much I’m above or below the average weight of that segment while weighing 500 random apes (I’m including humans in that category) of all ages and genders will be much less useful to evaluate my weight compared to 47-year old Danish males.
For balance this means that the less variation in bot configurations there are, the better the bots can be balanced, so that player skill isn’t drowned out by the randomness caused by untested configurations.
Further, the fewer configurations, the better I can make sure that all edge cases are covered. Undiscovered edge cases and groups of bad configurations only encountered in one or two plays can hide game breaks, frustrating play experiences, situations that aren’t covered by the rules, configurations that are boring to play, etc.
An extension of this is that the rulebook won’t be tested as extensively, because theoretical reading just doesn’t stand up to a blind playtester actually trying to figure out what the rulebook says about an edge case.
We won’t be able to test many catapults in depth with this amount of ammo . Image credit: lino9999
It’s not quite that bad
I’m of course exaggerating for effect. It’s rare that everything changes completely when one parameter for a bot is changed and so a test of one configuration is to some extent also a test of one or more other configurations.
Additionally, theorizing can assist somewhat in uncovering potential issues, adjusting balance, and in some cases different difficulty levels are simply differences in scoring offsets with almost no impact on gameplay.
Initial vs. expert customizability
When I’m to learn a rule set, I want it as simple as possible to learn and if anything, initial customizability hinders me instead of being a good thing, because I have no understanding to base my customization on. The latter effect can of course be handled by having beginner setups defined in the rulebook.
The approach that I prefer is instead to have a core bot to learn and then offer customizability to veteran players, so that the barrier of entry is as low as possible, while offering a few customizations later on and this bears out in our Automas (bots), where for example Scythe has a core system that is used at the beginning and which I think that the majority of players stick to, but there are also some slight customization options at the end of the rulebook for use once you’ve learned the game well enough that you can make meaningful predictions of the impact those customizations have on the play experience.
The above-mentioned LCGs also take this approach because they come with a number of preconstructed decks and in the latest one, Marvel Champions, all new heroes come with a preconstructed deck and recently packs with preconstructed decks have been added to Arkham Horror.
Getting out of the way
Customizability leads to more rules and rule changes between games. For games where the customizability has effects in the bot ruleset, then that means that the player has more rules to learn, which for many players will give a worse experience and if the extra rule load is significant, it’ll turn some players completely off the game.
Even if the introductory rule load isn’t higher then customizability will more often than not lead to more game elements changing from one play to the next, which means that it’ll take longer before the player learns everything by heart. That makes the player spend more time and mental bandwidth on running the bot and the vast majority of players prefer to spend their time and mental bandwidth on their own turns rather than on those of the bot. It’s also more likely that the player makes rule mistakes if there are changes all the time.
If you juggle catapults, you’ll get to the point where you can do it effortlessly much faster than if you regularly change the kind of catapult you juggle. OK, that’s a bad example, but you get the point.
Similarly, making a bot as streamlined as possible can lead to a better play experience because it makes the rules easier to remember and thus bot turns will be faster. Streamlined rulesets can be so tight that there are almost no knobs to turn without breaking the system. Our Automa for Red Rising is an example of this. There very little customizability because to allow that we would have been forced to make the bot system significantly more complex and slow to use.[/thing]
Getting out of the way also includes a quick setup. Typically, setup will be slower the more configuration options there are. It is for example much faster to shuffle a deck of 10 cards that control a bot than to pick 10 specific ones out of 20 and then shuffle.
The three mats on the left and the card with a cube represent the main setup configuration options of a game of Scythe against the Automa. Image credit: Garry H
Personally, I’d rather have 5 balanced bot configurations that give great experiences and play smoothly than 1000 where most are badly balanced with just OK play experiences that require more of my mental bandwidth to run.
At the end of the day, though, we’re talking about a lot of subjective trade-offs between variability, balance, ease of play, rule quality, problematic edge cases, play experience etc. There is no right or wrong, just subjective preferences and design goals.
A blog about solitaire games and how to design them. I'm your host, Morten, co-designer of solo modes for Scythe, Gaia Project, Wingspan, Glen More II, and others.
- [+] Dice rolls