A little bit of background: Susuexp is currently conducting an ongoing analysis of the 40,000 game anonymized OCTGN dataset. If you’re a stats nerd, it is definitely a must-watch. The level of detail and types of analyses are fantastic and provide much to think about.
In Susuexp’s most recent episode, The Community, he presented a series of graphs towards the end that suggested for very frequent players, the corporation and runner sides were more or less balanced, and the corporation may even have a slight edge. However, it was commented that the results needed to be taken with a grain of salt; there were not enough data to draw strong conclusions.
For at least two reasons, I was very skeptical of these particular graphs Susuexp presented. I am definitely one of those “very frequent players” on OCTGN. My subjective perception is that, when playing against other regulars, the runner most certainly has the edge -- probably in the area of a 70% win rate. Second, I would have to guess I have played at least 100 games against other regulars in the past couple months, just by myself. Given that other regulars must be playing against each other as well, I have a hard time believing that the data set contains less than 500 games played between OCTGN regulars. 500 datapoints is quite sufficient for answering any sort of simple question like “which side is favored?”
So I decided to do some followup work on the 40,000 game dataset. If you’ve been following Susuexp’s video series, you’ll have noticed he’s doing some fairly complicated time- and order- dependent analyses. This is a good thing -- someone needs to be doing this stuff. However, the (very granular) types of analyses he is performing spread the data a little thin when you want to ask questions like, “what is game balance like for the most experienced players?”
So with that in mind, I am just going to do some very simple plots of corp and runner win rates in games involving “the regulars”.
The 40,000 game dataset contains data from 3858 unique players. Most have only played a single game or two. A couple have played a thousand or more. The mean number of games played by a single player is 22. The median is 4. The 90th percentile is 58 games played. The 95th percentile is 100 games played. The 99th percentile is 250 games played.
I am curious about “what does game balance look like for the most experienced players?” We will call this group “the regulars”, and arbitrarily define the group as everyone in the 99th percentile for games played.
In addition, we are only going to be looking at games that are, at minimum, the regular’s 200th game -- this is simply to make sure we are only looking at games where the regulars are at their “most experienced”. So, say a regular has 250 games under their belt. I am only looking at their 250 - 200 = 50 last games played.
We are going to walk through a few questions. First, let’s ask the question “how do the regulars perform across the field of all players?”. The most important graph is the rightmost one; it contains all the games from all the regulars, combined.
The regulars perform quite well against the field. Their win rate for both corporation and runner is above 50%. However, their win rate as runner is higher (about 68%) than their win rate as corporation (about 57%). The regulars are strong players, and win far more than their fair share of games. This shouldn’t be surprising; these are the players with the most experience playing Android: Netrunner. They do have an easier time winning as runner than corporation, though, which is interesting.
All of the smaller graphs to the left are data for the ten individual most frequent players. Note that the patterns of “the regulars are better than average” and “the regulars win more often as runner than corporation” seem to hold at an individual level.
Ok, let’s move onto a new question: “how do the regulars perform when playing against each other?”. Now we are only going to look at games played between two regulars. Additionally, a game will only be included for analysis if it is both player’s 200th+ game. This leaves us with a very healthy n = 532 games. Again, the most important graph is the rightmost one; it contains all the games from all the regulars, combined.
Corporation win rate is 42%, which means runner win rate is 58%. This is reliably different from a 50/50 split (n = 532; p < 0.001). Just to make sure this difference is not due to “a couple of regulars who are very weird players”, I have again plotted the individual win rates for the ten regulars with the most number of games under their belt. In fact, there does seem to be variability from player to player, and at least three of these players do not follow the overall trend (#3837, #2100, #3088). Note, though, the n’s are fairly low for individual player games.
Taken as a whole group, the runner wins more often than the corporation when the regulars play against each other.
But... but...! Maybe this is just because of that dastardly Gabriel Santiago! Yeah, ok, fair. Let’s ask a third question: “how do the regulars perform when playing against each other, and neither is playing Gabriel Santiago?”
When excluding games involving Gabriel Santiago, we are left with n=368 games. Corporation win rates go up (to 45%), runner win rates go down (to 55%). This is still a marginally reliable effect (two-sided p = 0.08; one-sided p = 0.04). Gabriel is certainly influencing the data, but he alone is not the entire story. There does seem to be a bias in favor of the runner for the regulars. However, contrary to my expectations, it is not in the 60-70% range.
So, where does the “runners will win 60-70% of the time between two experienced players” statement come from? It is certainly not supported by these data, which actually pin the runner win rate somewhere between 55% - 60%.
I think there are two possibilities.
1. Go back to the first set of graphs. When you look at the regulars’ win rate against the entire field of players, their runner win rates are in the 70% range. Perhaps experienced players are having a hard time mentally separating their games against other experienced players from games against everyone else, and are actually providing estimates that are contaminated by their “win rates against everyone”. This would be a very reasonable, non-surprising error for humans to make.
2. Go back to the second set of graphs. Although the overall runner win rate between experienced players is 58%, there are in fact some regulars in the 60+ range. Notably, these also tend to be players with less than 50% win rates as corporation against other regulars. It could just be these are the people who are loudest on the issue.
So, in conclusion: I don’t think it matters how you cut it. Between experienced players, there is a tendency for runners to have the edge. There are certainly exceptions, but they are just that: exceptions. The runner’s favor is certainly not in the 60-70% range, and Gabriel Santiago is certainly influencing the data (to an extent), but at this point I would be very surprised to see someone find a reasonable way to argue to the contrary; the runner is currently favored when regulars play.
Articles on Android: Netrunner strategy, as well as example situations for sharpening your wits on.
09 May 2013
- [+] Dice rolls