Recommend
14 
 Thumb up
 Hide
32 Posts
1 , 2  Next »   | 

Android: Netrunner» Forums » General

Subject: How often is the tournament winner the best player? rss

Your Tags: Add tags
Popular Tags: [View All]
Rus
United States
Mountain View
California
flag msg tools
Avatar
mbmbmbmbmb
I've often wondered how accurate Netrunner tournaments are at identifying the best player. The worlds tournament finally inspired me to do the calculation.

Technical details:
I did a Monte-Carlo simulation (1000 trials) of a Swiss tournament where each player is assigned a random Elo rating with a Gaussian distribution with sigma = 100 (chosen to roughly match the octgn league Elo distributions: http://www.challengeboards.net/boards/standings/4 and http://www.challengeboards.net/boards/standings/1028). Each game between two players is then resolved randomly according to the Elo rating difference between those two players. I only considered prestige and ignored tiebreakers (probably makes only a negligible difference in the final results).

Matlab code below (copy/paste to enlarge):
Np = 16; % Number of players, must be even
sigma = 100;
N_trials = 1000;
N_rounds = 4;

for trial = 1:N_trials

ratings = randn(Np,1)*sigma + 1500;
[ratings_sorted true_ranked] = sort(ratings, 'descend');
[temp true_rank] = sort(true_ranked, 'ascend');
initial_scores = zeros(Np,1);
initial_ranked = 1:Np;

scores = initial_scores;% + 1e-2*(length(initial_scores):-1:1);
ranked = initial_ranked;

for round = 1:N_rounds

for i = 1:Np/2
playerA = ranked(2*i-1);
playerB = ranked(2*i);
Ea = 1./(1 + 10.^((ratings(playerB)-ratings(playerA))/400));
% game1
for games = 1:2 % 2 games in a round
if rand < Ea; % player A wins
scores(playerA) = scores(playerA) + 2;
else % player B wins
scores(playerB) = scores(playerB) + 2;
end
end
end

[sorted_scores ranked] = sort(scores,'descend');

end

final_scores(:,trial) = scores;
[temp rank] = sort(ranked, 'ascend');
final_ranked(:,trial) = ranked;

winner_true_rank(trial) = true_rank(final_ranked(1,trial));

rank_v_true_rank(:,trial) = rank(true_ranked);

end

figure(1)
errorbar(mean(rank_v_true_rank,2), std(rank_v_true_rank'),'*-'); hold on;
plot(1:Np, 1:Np, 'r');
axis equal;
axis([0 Np+1 0 Np+1]);
xlabel('true player rank');
ylabel('tournament rank');
title('Tournament measurement accuracy of player rank')
legend('Measured', 'Ideal');
hold off;

figure(2) % what is the tournament winner's true rank distribution?
N = hist(winner_true_rank,1:30);
bar(1:30, N/sum(N));
axis([0 10 0 0.4]);
xlabel('true player rank')
ylabel('probability of winning the tournament');
title('Probability that the tournament winner is the n-th best player');


Results (see edit at the bottom):
Case I: 16-player, 4 round Swiss tournament (typical for FLGS events):
The next two figures show that the best player only has ~30% chance of winning, which means that the tournament winner is probably not the best player, and is more likely to be in the 2-4 range. Most players' final tournament rank is off by about 4 places from their actual rank. Of course, if the top player is much better than everybody else, then these numbers can change drastically.



CaseII: 160-player, 11 round Swiss tournament. (This is similar to worlds -- except that I'm treating elimination rounds as Swiss rounds. Too lazy to implement actual elimination rounds!):
Here, the top player has a 25% shot at winning, so again the tournament winner is probably not the best player, and is likely to be somewhere in the ~3-10 range. Players on the top and the bottom have their tournament ranks off by ~ 10 from their true ranks, and players in the middle are off by 40!



Edit: when fitting my Elo distributions to OCTGN, I forgot about the Elo measurement bias (see http://www.boardgamegeek.com/article/12444817#12444817). This slightly changes the results.

For the 160-player, 11-round Swiss case, the probability that the tournament winner's true rank is:
1: 21%
2: 13%
3: 8%
4: 6%

For the 16-player, 4-round Swiss, the probability that the tournament winner's true rank is:
1: 26%
2: 18%
3: 12%
4: 9%

The most significant change is that the odds of the top player winning drops. The second-best player is now ~2/3 as likely to win as the top player (used to be 1/2), and the 3-rd best player is now ~1/2 as likely to win as the top player (used to be ~1/3).
8 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
João Almeida
msg tools
Avatar
mbmb
Do you any way to calculate how many rounds the same 2 players should play against each other for us to have a reliable measure of which player is the best one?
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Matt Wilson
msg tools
Avatar
mbmbmbmbmb
Worlds was 6 swiss rounds and 5 single-elim rounds, though . I suspect single-elim rounds are vastly more variable than 5 additional rounds of swiss!

Nevertheless, thanks for running these numbers. Very interesting.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Lynk Fox
msg tools
Avatar
mbmbmbmb
This is roughly true for any tournament though - ELO is a great measure of skill over a large amount of games, but the whole /point/ of a tournament is that sometimes those with lesser elo can defeat those with bigger ones.

Thats the Human Element. Thats the reason you go to a tournament - you might just be lucky enough to win that crucial game.
18 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Scott Rubin
United States
New York
New York
flag msg tools
Avatar
mbmbmbmbmb
This is all well and good, but it is missing one key thing. A tournament is not supposed to determine who is the better player overall. It is there to determine who is the better player at that exact moment in that exact situation. And in doing so, a tournament is 100% accurate. The player who wins it might get completely swept in rematches against the runner up, but they were the best player in the game that mattered.

Netrunner is the Super Bowl, not the World Series.
21 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
David Jackman
United States
Indianapolis
Indiana
flag msg tools
badge
Avatar
mbmbmbmbmb
lynkfox wrote:
This is roughly true for any tournament though - ELO is a great measure of skill over a large amount of games, but the whole /point/ of a tournament is that sometimes those with lesser elo can defeat those with bigger ones.

Thats the Human Element. Thats the reason you go to a tournament - you might just be lucky enough to win that crucial game.


Yep. And I would go so far to say that, if in a 160 player tourney, a 25% chance of the best player winning is huge.

think about it - 1/160th chance of a player winning on average, 1/4 chance of the best player? Thats pretty significant.
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Rob Wiley
United States
Chicago
Illinois
flag msg tools
mb
I've played in ~4 major tournaments and won one of them, so I'd say the best player wins about 25% of the time .
8 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Drake Villareal
United States
California
flag msg tools
Avatar
mbmb
To speak nothing of yer fancy math, it seems like on any day, any of the top 8 could be the winner, possibly even some people who missed the cut barely. I think the best mark of a netrunner player isn't good play, but rather the ability to punish mistakes and poor play by the opponent.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Patrick Jamet
France
Paris
flag msg tools
badge
Avatar
mbmbmbmbmb
Apreche wrote:
This is all well and good, but it is missing one key thing. A tournament is not supposed to determine who is the better player overall. It is there to determine who is the better player at that exact moment in that exact situation.

Sorry. I disagree because there is no "at that exact moment in that exact situation".

It would be (almost) true in the case of a tournament in duplicate, like a tournament of Bridge.

In case you don't know : http://en.wikipedia.org/wiki/Duplicate_bridge

The winner of a Netrunner tournament can only prove that he is a very competent player, nothing more.
3 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Roberta Yang
msg tools
Netrunner tournament rules also skew things heavily in favor of whoever is lucky enough to be able to run first. Combine that with single elimination rounds and your actual probability of finding the "best" player drops significantly. (It's no coincidence that the winner of Gencon ran first in almost all of his single-elimination matches.)
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Matt Wilson
msg tools
Avatar
mbmbmbmbmb
OddCrow19 wrote:
To speak nothing of yer fancy math, it seems like on any day, any of the top 8 could be the winner, possibly even some people who missed the cut barely. I think the best mark of a netrunner player isn't good play, but rather the ability to punish mistakes and poor play by the opponent.


For sure: I believe both #30 and #32 defeated #3 and #1 seeds coming out of swiss .

I was #33, sadly!
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Chris Hinkes
United States
New York
New York
flag msg tools
Avatar
mbmbmbmbmb
Having experienced Worlds this year, I think that the "best player" doesn't exist, or is impossible to determine. With 160 people in the tournament, if you made it to the top 32 after 6 rounds of swiss, you are good enough to win. Once it gets there, every player is really good, so who actually wins is based on all sorts of factors, including luck. One little screw up or moment of bad luck (or good luck for opponent) and you are out. Having played against your opponent one time and lost doesn't mean he is necessarily better than you. Everyone in the top 32 is good. (And probably many people outside of the top 32 who had a string of bad luck.)

salty53 wrote:
Netrunner tournament rules also skew things heavily in favor of whoever is lucky enough to be able to run first. Combine that with single elimination rounds and your actual probability of finding the "best" player drops significantly. (It's no coincidence that the winner of Gencon ran first in almost all of his single-elimination matches.)

Could you elaborate on why you think running first skews things heavily in your favor?
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Alex Rockwell
United States
Lynnwood
Washington
flag msg tools
designer
badge
Avatar
mbmbmbmbmb
Pyjam wrote:

The winner of a Netrunner tournament can only prove that he is a very competent player, nothing more.


Yes, this. Consistently doing well in tournaments is a sign of a strong player. As the OP's math shows, while the 'best player' has the highest chance of winning, the chance that the best player and the winner are the same, for any given tournament, is actually pretty low.
5 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jared Inc.
United States
Saint Paul
Minnesota
flag msg tools
Fueled by the Void
badge
Avatar
mbmbmbmbmb
Thank you for taking these first steps to apply some quantitative rigor to what many have instinctively suspected.

I'd like to emphasize what a few people have touched on. These numbers depend predominately on the number and skill distribution of players in a tournament and little else.

rbelikov wrote:
Each game between two players is then resolved randomly according to the Elo rating difference between those two players.


From my understanding, these values accurately represent the tournament outcome of any 2 player competition where skill can influence the outcome of game, whether that game be Netrunner, Chess, Ping-Pong or something else. So a 16 player Chess tournament with the same number of rounds (and playing 2 games per round) looks the same.

It seems best not to emphasize best player but rather top 8, top 16, and so forth knowing that there is essentially uncertainty in their ranking.

It's also good to see that when I go to a tournament, I have a nonzero chance of winning!

Edited to attribute quote


 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jeremy York
United States
Washington
flag msg tools
mbmb
I did some similar simulations to decide upon the exact format to use with the limited time available for the Washington state regionals, and came to some similar conclusions.

A few comments :

(1) The results depend a lot on the assumptions you make about the probability of which player will win a particular game. We know that a master player can still get destroyed by a terrible mulligan or a couple of lucky R&D accesses, but how likely is it? You have to make assumptions like this to run a simulation, and the results of the simulation will show as much about these assumptions as they do about the format and scoring system. I did multiple runs with a variety of distributions to get a feel for things.

(2) There is something to be said for the best prizes to be for the top 4 or 8 in a bigger tournament. Plugged In and Regionals both did this, with spiffy playmats for the top 8. This led to my simulations being mostly concerned with the question : how many rounds of Swiss does it take for the top 4 players to be likely to make the top 8? That number looks a little better than whether the top player wins, and made me feel good about the format we used.

(3) Because of the randomness of the game, the top player may well get knocked out of single elimination play. That's fine, I think; the tournament experience in a game like this is about who can ride that combination of deck building + play skill + luck to the top. I think that because of this, having the prize structure be very top-heavy is a bad idea. Having the top 1-2 prizes be something worth playing for is a good way to keep the excitement and incentive, but don't over do it.

(4) In addition to the sims, I observed in practice just how random the swiss rankings are in the top-middle of the pack. At the WA regionals, I ran a consolation bracket of the same structure as the top-8 single elimination. A lot of the people who didn't make the finals didn't want to stick around for the consolation, and so the entrants for the consolation went into the middle of the overall field. And in the end, it was people who were seeded towards the bottom who emerged at the top of the consolation bracket, after some very strong and very competitive play. Perhaps it was fatigue, but several of the top seeded folks in the consolation bracket were knocked out early.

We also saw this last point at Worlds, where a high seed after Swiss was no guarantee of surviving the first round of elimination play. Faction and deck choice also comes into play here, with the decks that are good at Swiss (aiming to win every game) may not be the best for elimination (where being able to recklessly rush points, or control a game, become important strategic options that some decks don't have).
3 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Edward K.
United States
flag msg tools
mbmbmbmbmb
To everyone saying "yeah so what and that's fancy numbers but big deal" I think the point of this post was to say that there is quite a "random" element to winning. In any competitive environment you want to have the game based on skill and not "luck."

Obviously this is a card game and if you just so happen to shuffle your deck to counter your opponent or you top deck like a boss then you will will. So luck and randomness is a major factor in playing.

@Apreche: I don't think what you said is accurate at all. At Worlds it is kind of hard to control who gets to sign up but ultimately a tournament is designed to find the best player. Because this is a major commitment and most people will have to fly there and get a hotel, only very serious players will sign up. So this will filter out "casuals" in a way and the best of the best should be playing to determine who is the best (and not just in this particular point).
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Geoff Hollis
Canada
Edmonton
Alberta
flag msg tools
Avatar
rbelikov wrote:
CaseII: 160-player, 11 round Swiss tournament. (This is similar to worlds -- except that I'm treating elimination rounds as Swiss rounds. Too lazy to implement actual elimination rounds!):


Do you think implementing single elimination would increase or decrease variance in rankings? By how much, would you expect it to matter?

My guess is that single elimination adds variance, and a nontrivial amount of it.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Piotr Jekel
Poland
Torun
flag msg tools
Avatar
mbmbmbmbmb
laikal wrote:
OddCrow19 wrote:
To speak nothing of yer fancy math, it seems like on any day, any of the top 8 could be the winner, possibly even some people who missed the cut barely. I think the best mark of a netrunner player isn't good play, but rather the ability to punish mistakes and poor play by the opponent.


For sure: I believe both #30 and #32 defeated #3 and #1 seeds coming out of swiss .

I was #33, sadly!


I have been in favour of swiss-only tournaments (especially for larger groups) for a long time exactly because ot this. You play six or seven rounds of swiss and score the most PP. Most likely, you had some lucky wins and some bad draws along the way as any other player. Luck mostly balances itself.

And than elimination rounds begin. 1 unlucky mulligan or lucky R$D dig for your opponent and you are out. It may happen even to the best player and there is nothing you can do about it.

All your efforts and consistently good play in swiss means nothing as being a higher seed is only a minor advantage. Elimination rounds only favour lower-ranked players by giving them a second chance.

I think that Variance plays too huge a factor in elimination rounds. Maybe a double elimination for losing higher-seeds only would help?

1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Gregory Pettigrew
United States
Massachusetts
flag msg tools
mbmbmb
hollis wrote:
rbelikov wrote:
CaseII: 160-player, 11 round Swiss tournament. (This is similar to worlds -- except that I'm treating elimination rounds as Swiss rounds. Too lazy to implement actual elimination rounds!):


Do you think implementing single elimination would increase or decrease variance in rankings? By how much, would you expect it to matter?

My guess is that single elimination adds variance, and a nontrivial amount of it.


We recently (4 years ago) switched our martial arts tournaments from straight double-elim to something more like Swiss. Variance for the top 16 went *way* down. Single Elimination would add a ton of variance vs. Swiss.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jeremy York
United States
Washington
flag msg tools
mbmb
Long Swiss tournaments get problematic because people begin to drop, and it starts to mess up the tiebreakers and pairings. I don't think it's realistic to expect someone who's a third of the way down the pack to be willing to do more than 5 or 6 rounds. And when those people drop out, the top players who had been paired against them suffer on tiebreaker, and some of the second-tier people will get lucky breaks as they get paired down the field and jump up.

The best format for a really large field, I think, would be a two-day event with both days being Swiss, and the second day being cut to a top 32 or top 16 based on the first day. That second day then gives you more good information about just the top players, with less of the drop-out problems than if you tried to run an 8-round Swiss for the whole field.

However, for anything but Worlds, Gencon, or Origins, a multi-day event isn't very likely to happen.

While FFG's preferred format is to cut to single elimination finals, any canny tournament organizer will mimic that format for bigger events. That's because it's important to give your top players experience with the format that will be used in the big events.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Chris Hinkes
United States
New York
New York
flag msg tools
Avatar
mbmbmbmbmb
hobb3 wrote:
I have been in favour of swiss-only tournaments (especially for larger groups) for a long time exactly because ot this. You play six or seven rounds of swiss and score the most PP. Most likely, you had some lucky wins and some bad draws along the way as any other player. Luck mostly balances itself.


I definitely prefer swiss only tournaments. But for a tournament of 160 people, is six or seven rounds enough? The winner had to play 11 rounds. Would having the top 32 play another five rounds of swiss work, starting from zero prestige? I think that would do a better job of finding the "best player," but perhaps that wouldn't be as exciting for other people to watch?
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Scott Rubin
United States
New York
New York
flag msg tools
Avatar
mbmbmbmbmb
I think it would be slightly less exciting to watch because you are not guaranteed to have this final game where the winner wins and second place is second place.

However, I think it will be MORE exciting for two reasons. One is that you will get to see games with more different decks that are in the second swiss tournament. Even if that Professor/Stronger Together player loses the first round of swiss, you can keep rooting for them.

Secondly, even though there is no finals, in most situations there will be two finals. In the last round of swiss there will be two players at the top table. If either one sweeps, they will be in first place guaranteed. The other player will drop. If they split, then the players at the second table have a chance to pass them.

It's complicated, but I think people like that. It's just like the last week of the NFL where you are doing math to figure out who has to win and who has to lose for your team to make the playoffs. When the results of the second table come in as a sweep for one of the players, now you know watching the top table that the other player will be champ if this game isn't also a sweep, and so forth. All the different multi-table scenarios will create some great tension for spectators.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Andy Palmer
United States
Lebanon
New Hampshire
flag msg tools
Apreche wrote:
This is all well and good, but it is missing one key thing. A tournament is not supposed to determine who is the better player overall. It is there to determine who is the better player at that exact moment in that exact situation. And in doing so, a tournament is 100% accurate.


Sorry this isn't true, or to be more accurate, it is only true in a general sense. In Netrunner, like many other games, you can make a subpar decision that actually strengthens your position. Subpar play will not win you many tournaments in the long run, but it might contribute to you winning the one you are in.

I suppose one could argue that the subpar decision was "correct" in that certain situation. However, taking that perspective renders the concept of a better player completely meaningless.

2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Piotr Jekel
Poland
Torun
flag msg tools
Avatar
mbmbmbmbmb
Zeromus wrote:
hobb3 wrote:
I have been in favour of swiss-only tournaments (especially for larger groups) for a long time exactly because ot this. You play six or seven rounds of swiss and score the most PP. Most likely, you had some lucky wins and some bad draws along the way as any other player. Luck mostly balances itself.


I definitely prefer swiss only tournaments. But for a tournament of 160 people, is six or seven rounds enough? The winner had to play 11 rounds. Would having the top 32 play another five rounds of swiss work, starting from zero prestige? I think that would do a better job of finding the "best player," but perhaps that wouldn't be as exciting for other people to watch?


I really like the idea of 2 swiss mini-tournaments. The first swiss to select the Top 16-32 depending on the number of players and the other one to see who performs best overall. The first-swiss pre-selection tournament could also determine the seeding, so the highest seed would play the lowest-ranked opponent.

It would have worked great in the Polish championships, which were held over the course of two days. There were a few people screwed over by variance.

However, even 1-day events such as the Worlds would fit this format. Instead of 2 people playing 11 games, there would be 16-32 players to show off their skill and endurance.

Plus, table 1 would always have the currently best scoring players.
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Piotr Jekel
Poland
Torun
flag msg tools
Avatar
mbmbmbmbmb
Another advantage: no more stupid games in which your opponent only plays to score x-number of points rather than win. The runner-up from Worlds would have played it differently, if it had not been for the first game score. Matches would be more interesting. Hostile Takeover for the win in the first action... I saw it happen anfew times.

Plus, the WSW from the first swiss tournament could be added and used for tie-breaks.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
1 , 2  Next »   | 
Front Page | Welcome | Contact | Privacy Policy | Terms of Service | Advertise | Support BGG | Feeds RSS
Geekdo, BoardGameGeek, the Geekdo logo, and the BoardGameGeek logo are trademarks of BoardGameGeek, LLC.