Recommend
 
 Thumb up
 Hide
38 Posts
1 , 2  Next »   | 

BoardGameGeek» Forums » Everything Else » OLD BGGBlogs (do not use)

Subject: A Solution for Shill Ratings rss

Your Tags: Add tags
Popular Tags: [View All]
Richard Smith
Canada
Coquitlam
B.C.
flag msg tools
mbmbmbmbmb
Hi Everyone,
I have been thinking about Shill ratings. shake
(A shill rating is one where a person creates a bunch of accounts under fake names and give a game a high rating to lie to people about how good their game is). I think this system will work:

If you are a new user and post a rating it is worth 0/1000 weight in the weighted average. The weight of your rating would be increased using the following algorithm:

+1 pt for each game you rate without comment.
+1 pt for each comment on a game that is ~20 words or more.
+1 pt for each comment in a journal / session report / review / geeklist.
+1 pt for each geek list.
+2 pt for each session report >100 words.
+3 pt for each review.
+10 pt for articles that an admin likes so much that it is flaged as a featured article.

etc. (What ever our admins think are important).

Total points and that is the weight given up to 1000/1000.

Thus those who love games and contribute to our community have more weight than an account that appears, rates a couple games and then vanishes.

Let us see what happens when some ass comes along and tried to promote his or her game by rating it all 10's. He or she create 30 accounts and rates the game straight 10's. Each account has a value of 1 or 2 (depending if they left a comment as well). Game has a 10 rating and shoots up to the top of the list. Some one like DangerMouse (picking a name I remember at random) tries game and says it sucks, gives a rating of 4. DangerMouse would likely have a weight of 500 or more under this system:

Fake accounts: 30 * 10 * 2 pts = 600 pts.
DangerMouse::: 01 * 04 * 500 pt = 2000 pts.

(First #, number of people rating game.)
(Second #, score of rating.)
(Third #, weight score found by system above.)

Final rating: 2600 pt / [ 500 + (2 * 30) ] = 4.643
(If acounts weight goes over 1000 treat it as 1000. Not needed in this case.)

Using this system, shilling becomes almost impossible.

I do not think that the exact algorithm should be published. Just say that regular contributers are given more weight and leave it at that.

The main advantage is it does not take our honored admin's time to enforce, it is done with a simple algorithm.


What do people think?

Warm regards, Rick.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Aaron Potter
United States
Riverside
California
flag msg tools
Avatar
So basically reward long-time users of BGG by giving their votes more weight? I see the reason for your concern, and I understand that this would "solve" the problem of phony fly-by-night accounts being used to pump ratings...
But ultimately, I think it creates more problems than it solves. Among other things, it devalues new users' voices, encourages the submission of empty commentary and data just to improve one's voting weight, and gives entirely too much credence to the establishment here.
I don't think this is a good idea.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Richard Smith
Canada
Coquitlam
B.C.
flag msg tools
mbmbmbmbmb
Very long people get more weight
Hi potterama,
Thanks for the quick reply! Yes long time posters would get more weight, but it levels off after they reach 1000 pts.
So someone who has been active for 1 year, would not have any less weight than someone who has been active for 5 years.

I agree that it would encourage short posts of little merit to force up the rating for people who care about such things, but no system is perfect. That is one reason I suggest that Derk and Aldie keep the exact system secret. If I know I have to write 50 words in a journal comment to get a couple points, then I'll write 50 words. But if I don't know, I think it will be just too much work.

Right now, people admit that they are abusing the system, and I think this would put a stop to it. (I like the ratings and it bugs me that they are being corrupted by selfish people.)

Also, speaking for myself, I would rate games and comment on journals even if I had no weight. Just to be able to help people who care about our obsession.

Thanks!
Rick
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Steve McIlhatton
New Zealand
Palmerston North
flag msg tools
badge
Avatar
mbmbmbmbmb
A nice idea. Not sure if it'd work in practise but I like the general idea.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Kyle Sasser
United States
Tampa
Florida
flag msg tools
Avatar
newgrounds.com has a similar setup. In their portal section, you can vote on what rocks and what sucks. You get 1 point for every vote, with a maximum of 5 points a day (you can vote on as many items as you like, they only count 5 a day).

They then have Levels, say 20 points, 50 point, 100 points, etc, and when you reach a new level your votes get weighted more (Level 1 = 1x, Level 2 = 2x, Level 3 = 3x etc).

It definately has its flaws but it solved the very large problem they were having over there of utter crap getting voted up because of people logging in with 500 different accounts.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
jefF, There are some who call me... DuneKitteh
United States
Wood Dale
Illinois
flag msg tools
Dune Kitty really does love you, despite any asshattery.
badge
Useless rollover, booya! Dune Kitty says, "deal with it."
Avatar
mbmbmbmbmb
And how would this rating weighting actually be applied and physcially differentiate itself when all of them are compared to each other? e.g. 30 "2 point" shill accounts rate it as a 10, one valid "500 point" account rates it as a one - how would this show up differently on a ratings page? Or are you just speaking of weighting only the Average Rating or the Bayesian Average Rating (that'll be a mess), or simply having a third "Weighted Average" option?
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Andrew Faehnle
United States
Cincinnati
OH
flag msg tools
badge
Avatar
mbmbmbmbmb
Standard
My opinion on this subject has always been to prevent users from being able to rate a game until they either

1). Have an avatar, or
2). Have a BGG Supporter badge.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Richard Smith
Canada
Coquitlam
B.C.
flag msg tools
mbmbmbmbmb
Drews ideas
Quote:
My opinion on this subject has always been to prevent users from being able to rate a game until they either

1). Have an avatar, or
2). Have a BGG Supporter badge.


Hi Drew,
If newbies can't rate until they have an avatar I am afraid that might make it harder for people to get in the habit of contributing to the site.

As for a supporter badge, how much does it cost? I am wondering if it is enough to deter a cheater with any sort of advertising budget.

Thanks for your comments! Rick
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Richard Smith
Canada
Coquitlam
B.C.
flag msg tools
mbmbmbmbmb
JefF's idea of a third rated average.
Quote:
... or simply having a third "Weighted Average" option?


Hi everyone, JefF.
I posted a reply to this awhile ago, but it has vanished. Sorry if it ends up being posted twice.

I was thinking of replacing the regular rating but I like your idea better. Showing all three ratings might actually deter shilling because it will be embarrassing if the normal rating is far higher than the geek rating.
laugh

It would also make a new type of geek list: "Games with largest difference between ratings (minimum of 10 geek ratings)".

Warm regards, Rick
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Pierce Ostrander
United States
Albuquerque
New Mexico
flag msg tools
badge
Avatar
mbmbmbmbmb
Must have avatar for your ratings to COUNT. You can post a rating, but it will not be entered into the average unless you have an avatar.

Best & simplist deterrent.

Get ready for the crys of "elitism"!

Foob
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Lyman Hurd
United States
Cupertino
California
flag msg tools
Avatar
mbmbmbmbmb
is it this much of a problem?
I cannot recall ever having seriously considered a game for which this is a problem. Sure, if a game looks intriguing and is highly ranked I will read the reviews, possible browse the rules, etc., and I am unlikely to be impressed by a bunch of carbon copy descriptions that sound as if they were spub out of someone's marketing department. I automatically buy any game that makes it into the top ten.

I know there have been some notable exceptions and in those very few exceptions, I think the admins would be justified in manually wiping out bogus votes but in general I cannot believe shilling affects more than a very small number of ratings and like spam prevention, the people motivated to cirumvent any scheme you come up with, will. I would and I would hate to come up with elaborate schemes for a relatively minor problem, especially any schemes that make newbies feel less welcome thatn currently. I know that people find avater acquisition somewhat daunting and if we project the idea that we do not even want to hear from you until you have one, we might as well say you cannot be here unless you know what TGOO (always hated that term) means!
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
M@tthijs
Netherlands
Venlo
flag msg tools
This user is outstanding in mediocre videogaming
badge
Did you visit my www.kobudovenlo.nl? It has game info
Avatar
mbmbmbmbmb
Get ready for the crys?
Get ready?
I agree! I don't have an avatar. And for being supporter: I'm still waiting for the answer to Rick Smith's mail (2005-03-01 22:44:35)
Since I don't don't have much time to spent on writing articles and game-reviews (actually: I'm rather glad when I am able to play a game), it will take me a long time to get an avatar.
I like being able to rate games (for my own list)
I don't mind not being counted in (yet) any avg.
I do mind when the system gets screwed by sales-^%$$% by using muliple accounts. When I see a game with a high rating, I want to be able to trust it being a game which is highly rated by fellow-games, not the sales-department.

my5cts
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Ray
United States
Carpentersville
Illinois
flag msg tools
badge
Avatar
mbmbmbmbmb
This geeklist should be of interest to the readers of this thread:

http://www.boardgamegeek.com/geeklist.php3?action=view&listi...

edit: now its this URL: Experience Based Ratings (EBR)
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Chester
United States
Temple
Texas
flag msg tools
badge
Avatar
mbmbmbmbmb
Another problem is that even those who've been on the site a long time and built up 'credibility' according to the criteria you listed have been among the abusers. I've seen ratings from those whith avatars and badges where it was obvious that they'd rated a game a '1' that they'd never played (not talking about any Cardchess games here).

I'm thinking of some episodes where several CSW users organized, joined en masse and started rating some of their favorite wargames. Suddenly there were some highly rated wargames. Someone didn't like that and countered by rating some of these games a '1'. I find it extrememly unlikely that person actually played these games. Even if he/she had, a '1' signifies the game is broken...so I don't find that a valid rating in most situations. Under your proposal, the voice of all the experienced wargamers who happened to be new users would have been completely drowned out by one long-time Geek with his/her own agenda. Seems like it will discourage new folks from bothering...
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Don Weed
United States
Clemmons
North Carolina
flag msg tools
badge
Avatar
mbmbmbmbmb
Other fallout
Although I think the suggestion is noteworthy and would solve the shill problem, I was thinking of other things that this may imply.

1)Does this imply that all new members are considered the same status as 'shills'. I agree about the 'elitest' statement. It may deter new blood.
2) Would a members 'ratings worth' change (add more weight) with time when they gradually build up to 1000 points? I can see this being a logistics nightmare. I wouldn't wish that on anybody to monitor.
3) Would or could you re-rate games with more weight once you've established some credibility?
4) Whwn all is said and done would this just encourage more elaborate 'shill'ing? More complete profiles would be harder to detect. I think that there are enough honest Geek users to spot the frauds.

Food for thought but a good suggestion none-the-less.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Chester
United States
Temple
Texas
flag msg tools
badge
Avatar
mbmbmbmbmb
Or how about this example...

Here, Jason's rating would have a huge impact on the overall rating of this game. I would argue unfairly so. (Unfair in the sense that I see no reason his rating is any more valid than those of the others who've rated that game.) Since he didn't leave a comment its hard to know why his single opinion is so harshly counter to the overwhelming voice of the others who've rated and left comments on this game....but you can see how the rating would be changed using the weighted system you suggest.

http://www.boardgamegeek.com/viewcollections.php3?orderby=ra...=
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
The Games Are About Glory
United States
Macedon
New York
flag msg tools
badge
Avatar
mbmbmbmbmb
Quote:
"Jesus, I'd hate to tell you what I think about in the bathroom."


Nice Mallrats reference, Sega Boy.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
John Lopez
United States
Tucson
Arizona
flag msg tools
Get off my lawn!
badge
The explanation: Impossible Triangle + TW (my company initials) = my logo.
Avatar
mbmbmbmbmb
Quote:
Or how about this example...


Actually, I would count that as a perfect example of why system in this vein would be better than the current status quo. There are half a dozen accounts on that game rated 10 that have no other ratings. Either GMT itself or some other group came over to shill the game and this low rating is an attempt to "fix" the problem. Not that I agree with doing so (I find it darn annoying), but people are acting as vigilantes as there is no better solution.

I have actually picked up a few GMT games that have some decent ratings here, but I have to do the math myself to knock out the dozen "10" ratings to realize that this game is less than a 7 by Geek standards, not the 8.67 that the average or even the 7.32 that the Basian worked out. (I will say that the big difference between the two is a huge clue though, so Asian does help).

Bah, I got far more worked up than I wanted to about this *yet again*. Maybe I need some chill pills, but my personal solution still remains the broken record "blacklist" request. Yes, I know I'm supposed to build a "trusted buddy" list, but frankly the Geek is my trusted buddy, minus a few yahoo's, shills and a few people stuck at age 14 "must troll... can't resist damnation" winners. I would far prefer to amplify a few people I really trust and black out the few I don't, leaving the majority of the opinions at the current neutral.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
John Lopez
United States
Tucson
Arizona
flag msg tools
Get off my lawn!
badge
The explanation: Impossible Triangle + TW (my company initials) = my logo.
Avatar
mbmbmbmbmb
s/Asian/Basian/ (I swear I fix that post preview...)
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Robert Martin
United States
Atlanta
Georgia
flag msg tools
badge
Avatar
mbmbmbmbmb
Boring...
This is an old, tired debate. Who really cares? The system seems to work just fine as is. Let's talk about something interesting for a change.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Chester
United States
Temple
Texas
flag msg tools
badge
Avatar
mbmbmbmbmb
John,
I seem to recognize a few of those names from CSW. I suspect that those votes represent real people that really love the game. Why should somone else's uncommented '1' be considered with more weight than those 10s? In fact, that someone would be motivated to join BGG just to rate that game may even indicate MORE strongly that its a good game. They love it that much! Just as valid is the scenario where some wargamer perused the Geek...saw that there was virtually NO commentary on one of his favorite games....invited some other guys he knew loved the game and they all joined the Geek for that purpose.
I don't know which scenario is right...but I don't like the idea of one single user being able to influence the games cumulative rating that much. I don't think ANYONE should be rating for effect. That is, they should not rate to manipulate the cumulative ratings. They should be rating their honest opinion and let the chips fall.
What's wrong with 8 or 10 guys at GMT rating a game they are working on? Who would know the game better? I doubt they'd be producing the game if they didn't think it was any good. I find no basis for the idea that the game designer's opinion is worth less than some random guy that played it (or DIDN'T play it, potentially).
Anyway, your argument is based on a big assumption that those 10s are not 'legit' ratings. I argue that you don't know that.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Richard Smith
Canada
Coquitlam
B.C.
flag msg tools
mbmbmbmbmb
Answers
Quote:
....

1)Does this imply that all new members are considered the same status as 'shills'. I agree about the 'elitest' statement. It may deter new blood.


I'm worried about that as well. However if we go with JefF's idea of just adding a 3rd rating I can't see it being a problem.

Quote:
2) Would a members 'ratings worth' change (add more weight) with time when they gradually build up to 1000 points? I can see this being a logistics nightmare. I wouldn't wish that on anybody to monitor.


Yes, it would, but no human has to monitor it. The query that grabs information from the database would do the addition and then calculate the average. So someone would start with a low weight. But as time goes by and they contribute their voice gradually gets louder.

Quote:
3) Would or could you re-rate games with more weight once you've established some credibility?


Yes, just like now. (Go down to your rating on a game you have already rated and edit it. Hit submit and you have revised your rating.)

Quote:
4) When all is said and done would this just encourage more elaborate 'shill'ing? More complete profiles would be harder to detect. I think that there are enough honest Geek users to spot the frauds.


I do not think so. To build up 1000 points is a very large investiment in time and effort. Most of the people who have done so are big fans of games and have done it as a labor of love. A parasite on the list surely has better things to do, and even if they build one rating up to 1000 points, then are they going to do it 29 more times?

Quote:
Food for thought but a good suggestion none-the-less.


Thanks,
Regards, Rick
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Richard Smith
Canada
Coquitlam
B.C.
flag msg tools
mbmbmbmbmb
Quote:
Another problem is that even those who've been on the site a long time and built up 'credibility' according to the criteria you listed have been among the abusers. ...

I'm thinking of some episodes where several CSW users organized, joined en masse and started rating some of their favorite wargames. Suddenly there were some highly rated wargames. Someone didn't like that and countered by rating some of these games a '1'. ... Under your proposal, the voice of all the experienced wargamers who happened to be new users would have been completely drowned out by one long-time Geek with his/her own agenda. Seems like it will discourage new folks from bothering...


Thanks for your comments! Two points.

With JefF's suggestion the unweighted rating stands as well so you can compaire them. Ones that have a large difference between the two are a red flag that this game needs closer investigation.

Also, right now we have vigelante justice. If you think some one or some group is pumping the ratings you have to play politics and vote as extream as you can to make any difference. Under the weighted system, you can vote honestly and have the same or greater influence. I think that this system would discourage a lot of 10's and 1's.

And if someone brings to Aldie or Derk's notice that some one with a huge weight is lying, their weight can be reset to zero or they could be kicked off. However, I doubt that it would be required. Most of the abuse is by people pumping a game. Under this system it is much harder. Given that, the desire for vigilante votes is less needed.

And if someone who has earned 1000 points wants to rate a game a 1, when it really diserves a 4. In some way, have they not earned it?

Warm regards, Rick
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Richard Smith
Canada
Coquitlam
B.C.
flag msg tools
mbmbmbmbmb
Quote:
Or how about this example...

Here, Jason's rating would have a huge impact on the overall rating of this game.


Very true. No system is perfect. If we keep both ratings as per JefF's idea, it will be easier to identify both positive shills (promoting a game) and negative shills (those who have made strong negative votes.

Rick
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
John Lopez
United States
Tucson
Arizona
flag msg tools
Get off my lawn!
badge
The explanation: Impossible Triangle + TW (my company initials) = my logo.
Avatar
mbmbmbmbmb
Quote:
I argue that you don't know that.

Yes, what you say is very true... but the bad actions of some have caused me to mentally black list those ratings (in fact, pretty much all 9 or 10 ratings without comments or without a geek I recognize and trust attached to them), even if the Geek didn't allow me to do it "auto-magically". See, the problem isn't just that some fans rated the game, but that there is no *balance* to their ratings. And no, I agree that trumped up "1"s are not balance, I hate those as well (and those who do so would be on my black list as well).

The power of the Geek's rating system is I can see a cross section representation of gamers similar to those I currently play with. I have to admit to being a fan of some pretty esoteric games (Star Fleet Battles being one). If I actually rated it according to the geek guidelines it would be a 10 for the endless fascinating that the game provides me. I could wander over to the SFBOnline board and rally up a couple hundred people who would also rate it a 10. What value does that provide to the community? Would SFB be "served" by pumping it into the top 10 as some other games have been be. I have to say that it would do the exact opposite: it might cause someone to attempt to play a game which really should only be played by people who are obsessive like me, putting a bitter taste in peoples mouths (as I have gotten from a few of the GMT games I have purchased based on others feedback. I like wargames, but GMT hasn't really measured up as I expected it to.)

Of course talking about SFB and GMT really highlight perhaps a more underlying problem. Gaming is divided into "factions". There are the Grognards, the CCG fans, the Eurogamers, Miniatures Gamers, etc. I'm pretty broad based, but I know gamers who refuse to admit to the existence of other style of play. That's part of why my SFB rating isn't a 10: my Eurogamer side doesn't rate it as high as my Grognard side. Since the Geek is a Eurogamer game site, I gave it a compromise rating that reflected that.

However, the Geek has grown beyond that small niche and you can see it's impact when you basically get ratings that are "all game style-X games suck" being attached. If you examine peoples ratings you will see that kind of thing: one person heavily dings all CCGs, another rates all chit and counter games "1". Children's games get a sound thrashing around here, making the site fairly useless for looking at games for children under 14.

All that means that ratings are pointless and arbitrary, and yet many obsess over them. (Reference all the "top games by designer" or "alternate stats" lists). Why? Because when a game gets "unfairly" rated because of genre, age or marketing, people who care about the Geek providing meaningful feedback to its users get upset. Since the engine under the geek isn't designed to allow people to specify their preference for different genres, it can't handle accommodating the CCG basher or the anti-children's game ratings any different from those who love the same. I suspect as the "geek" widens its audience, those conflicts will only grow. Which probably means I should just stop worrying and learn to love the bomb. I just convinced myself that the ratings are meaningless and will only become more so... now I can live in peace and harmony. (And get more work done rather than ranting pointlessly.)
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
1 , 2  Next »   | 
Front Page | Welcome | Contact | Privacy Policy | Terms of Service | Advertise | Support BGG | Feeds RSS
Geekdo, BoardGameGeek, the Geekdo logo, and the BoardGameGeek logo are trademarks of BoardGameGeek, LLC.