Recommend
5 
 Thumb up
 Hide
17 Posts

BoardGameGeek» Forums » Gaming Related » General Gaming

Subject: PhD research: internet search and gaming rss

Your Tags: Add tags
Popular Tags: [View All]
Brent Wenerstrom
United States
Riverton
Utah
flag msg tools
Avatar
mbmbmbmbmb
Thank you to all who are reading this. I am a PhD student working in computer science on search engine summaries. I have come up with multiple ways of summarizing search results. However, I need some of you to give me some feedback. I have set up searches on all of the top 25 games here on the Geek. I then gathered top 5 search results from a major search engine. I altered the search summary in some cases. I just need you to see how effective each summary is at helping you predict how much useful content a search result will have.

Search Task

My search task should take you only 5-10 minutes. Any help is appreciated. If you have specific comments about any of the search summaries please mail me. Each user is randomly assigned to method of summary generation. Others will not see the same result summary, though they will see the same result pages. Feedback of all kinds is appreciated. Also, please alert me if anything is broken. I built it from scratch and it resides on my server at school. Thank you.
3 
 Thumb up
 tip
 Hide
  • [+] Dice rolls

Lacombe
Louisiana
msg tools
badge
Suddenly a shot rang out! A door slammed. The maid screamed. Suddenly a pirate ship appeared on the horizon! While millions of people were starving, the king lived in luxury. Meanwhile, on a small farm in Kansas, a boy was growing up.
Avatar
mbmbmbmbmb
Ok, I did it... but, honestly, my answers were based on my familiarity with similar searches in the past and familiarity with general writing styles of some big search engine results for games (BoardGameGeek, Wikipedia, game publisher sites, etc). I was pretty well able to guess what kind of page was represented, even with you masking the URLs and titles. Wikipedia and BGG are especially easy.

Also, I don't think anyone attempts to judge how much relevant information is contained on a page when looking through search summaries, so your basic assumption [that this is an interesting question to ask / answer] seems wrong. Unless this is the whole point of your research, but that wasn't exactly clear. This seems like showing people a book summary and asking them how long it is.

People don't work that way. They pick up the book and go "Oh, it's pretty big". They read summaries to find out how relevant it is, not how much relevant info is there.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Brent Wenerstrom
United States
Riverton
Utah
flag msg tools
Avatar
mbmbmbmbmb
First off. Thank you for completing the survey.

I can see what you are getting at. I struggled with exactly what to ask. Relevancy is such a difficult thing to judge. I have tried the how relevant something is and get back varied results. I would really like to measure how often people are disappointed when they click through to a search result. The simplest, most straightforward I approach that came to mind was that when I am disappointed with a search result, I click through and find far less relevant content than I was expecting. People easily can measure quantity, but it is much more difficult to say that something is less relevant than you thought.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls

Lacombe
Louisiana
msg tools
badge
Suddenly a shot rang out! A door slammed. The maid screamed. Suddenly a pirate ship appeared on the horizon! While millions of people were starving, the king lived in luxury. Meanwhile, on a small farm in Kansas, a boy was growing up.
Avatar
mbmbmbmbmb
stormybriggs wrote:
I can see what you are getting at. I struggled with exactly what to ask. Relevancy is such a difficult thing to judge. I have tried the how relevant something is and get back varied results. I would really like to measure how often people are disappointed when they click through to a search result. The simplest, most straightforward I approach that came to mind was that when I am disappointed with a search result, I click through and find far less relevant content than I was expecting. People easily can measure quantity, but it is much more difficult to say that something is less relevant than you thought.


Sure, but honestly... I don't think altering text search summaries is going to change this experience.

But maybe you're just really really good at writing text summaries. I can't guess the length of a block of text from two sentences.

It's why I've added the "Better Search" Fire-Fox add-on to my browser, which gives a SCREENSHOT summary.

Honestly, I don't know why it's not the default method of presenting search results. It is orders of magnitude of orders of magnitude better.



 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Brent Wenerstrom
United States
Riverton
Utah
flag msg tools
Avatar
mbmbmbmbmb
Excellent point. I have seen a number of research papers look into screen page images for improving shots. One paper in particular pointed out that when we search a second time on the same query, we refind what we found earlier much quicker and more effectively if images are involved.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls

Lacombe
Louisiana
msg tools
badge
Suddenly a shot rang out! A door slammed. The maid screamed. Suddenly a pirate ship appeared on the horizon! While millions of people were starving, the king lived in luxury. Meanwhile, on a small farm in Kansas, a boy was growing up.
Avatar
mbmbmbmbmb
stormybriggs wrote:
Excellent point. I have seen a number of research papers look into screen page images for improving shots. One paper in particular pointed out that when we search a second time on the same query, we refind what we found earlier much quicker and more effectively if images are involved.


Have you tried something akin to a tag cloud instead of actual snippets of text?

You can get across a whole lot more meta-information in the same word-count with tags rather than sentences.

No clue how you'd generate them for a search engine summary, as most sites aren't tagged.

Quote:
I am a PhD student working in computer science on search engine summaries. I have come up with multiple ways of summarizing search results. However, I need some of you to give me some feedback.


or

Quote:
-Search Engines -Summaries -Recall -Preview -Relevant -How much -How relevant -Feedback -PhD -Survey -Survey task -Research -Research paper -Refind -Quick -Effective -Predicting -Disappointed -Click through


Not a perfect example, since this particular page is so small. For huge blocks of text, though, I can imagine this working better. Here, the sentence version is probably better, but it's something to think about.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Netherlands
Enschede
flag msg tools
Avatar
mbmbmbmbmb
Pandemic gave some interesting search results... I answered with the assumption I was looking for information about the board game, not the flash game.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls

Lacombe
Louisiana
msg tools
badge
Suddenly a shot rang out! A door slammed. The maid screamed. Suddenly a pirate ship appeared on the horizon! While millions of people were starving, the king lived in luxury. Meanwhile, on a small farm in Kansas, a boy was growing up.
Avatar
mbmbmbmbmb
Polgara wrote:
Pandemic gave some interesting search results... I answered with the assumption I was looking for information about the board game, not the flash game.


Steam was worse, since there's a video game retailer called Steam.

I did Age Of Steam, Race For The Galaxy, and Steam.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Brent Wenerstrom
United States
Riverton
Utah
flag msg tools
Avatar
mbmbmbmbmb
Using tag clouds was mentioned recently in a 2009 paper (Towards Visual Web Search: Interactive Query Formulation and Search Result Visualization) that gives a few visual ways of presenting search results. Though in this specific paper no user studies were performed. I need to see if the authors (Marian Dork et al. with an umlaut) have now done any user studies using the approaches mentioned.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Scott Nicholson
Canada
Brantford
Ontario
flag msg tools
designer
badge
Avatar
mbmbmbmbmb
stormybriggs wrote:
Relevancy is such a difficult thing to judge.


This is indeed an understatement.

My PhD was in Information Science, and the concept of "What is Relevant" is an issue we have fought with as a field for decades. There are many papers about this concept in the publications of IS. A good summary article is by Schamber - Relevancy and Information Behavior - Annual Review of Information Science and Technology (ARIST), v29 p3-48 1994

Basically, the problem with defining relevance is that it is user-dependent. One of the failings in a lot of computer-science relevance research is that the user viewpoint is removed from the research, and relevance is determined simply by term or meaning match.

As a researcher, I can't look at a search query and an item and say if the item is relevant to that user's search query.

It is dependent upon what the user already knows. The same item that is not relevant for a novice's search may be very relevant for an expert's search.

This isn't the forum to go into this - heck, we had entire courses on the concept of what is relevancy - but it is a challenge with no right answer.
3 
 Thumb up
0.25
 tip
 Hide
  • [+] Dice rolls

Lacombe
Louisiana
msg tools
badge
Suddenly a shot rang out! A door slammed. The maid screamed. Suddenly a pirate ship appeared on the horizon! While millions of people were starving, the king lived in luxury. Meanwhile, on a small farm in Kansas, a boy was growing up.
Avatar
mbmbmbmbmb
snicholson wrote:
This isn't the forum to go into this - heck, we had entire courses on the concept of what is relevancy - but it is a challenge with no right answer.


Or, rather, a challenge with 1,000 right answers that are variously relevant dependent upon user knowledge and needs.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Brent Wenerstrom
United States
Riverton
Utah
flag msg tools
Avatar
mbmbmbmbmb
snicholson wrote:
A good summary article is by Schamber - Relevancy and Information Behavior - Annual Review of Information Science and Technology (ARIST), v29 p3-48 1994


Thank you for the reference. I'll give this a good read over once I find a copy.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jeremy Arcus-Goldberg
United States
Natick
MA
flag msg tools
mbmbmbmb
The links to BGG seemed to be the ones that at first seemed like they would be unhelpful in your search. Of course, when I clicked through, I knew that the BGG link would be the place to find the most information.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Guido Gloor
Switzerland
Ostermundigen
Bern
flag msg tools
The statement below is false.
badge
The statement above is correct.
Avatar
mbmbmbmbmb
tevacircus wrote:
The links to BGG seemed to be the ones that at first seemed like they would be unhelpful in your search. Of course, when I clicked through, I knew that the BGG link would be the place to find the most information.

Aye, this happened to me as well. BGG has most unhelpful summaries, it seems. Knowing the site however, since that's the first line of the search result, I don't even have to look at the summary to know it's relevant, I just skip that part of the site because I know it's irrelevant for my skimming (like all users do, according to pretty much every website usability study out there).
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Brent Wenerstrom
United States
Riverton
Utah
flag msg tools
Avatar
mbmbmbmbmb
haslo wrote:
I don't even have to look at the summary to know it's relevant, I just skip that part of the site because I know it's irrelevant for my skimming (like all users do, according to pretty much every website usability study out there).


It is true that on many searches, users mainly look at the titles, and avoid the summaries. However, research (I'll point you to the references if you are interested) has shown that for some types of searches, the amount of time required to find what you are looking for varies by the quality of the summary and the length of the summary in addition to the title.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Magnus Karlsson
Sweden
Eskilstuna
Unspecified
flag msg tools
badge
Avatar
mbmbmbmbmb
I chose Goa and Shogun which turned out to be bad choices since just doing that and adding "game" would not be a search I would expect to turn out useful and it didn't. Both summaries that actually pointed to their BGG pages talked about variants so I marked them first as expecting less content than they then turned out to have (since they actually pointed to the main entry).

I'm a Swede so if that skews your results feel free to ignore my search results :-).

/Magnus
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Brent Wenerstrom
United States
Riverton
Utah
flag msg tools
Avatar
mbmbmbmbmb
Jern wrote:
I'm a Swede so if that skews your results feel free to ignore my search results :-).


Thank you for the help. All Swedes are welcome (I have many ancestors from Sweden). I won't delete your entry. Here at the Computer Science department I'm in almost all graduate students are international. I've asked several of them to participate (with Computer Science related searches though).
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Front Page | Welcome | Contact | Privacy Policy | Terms of Service | Advertise | Support BGG | Feeds RSS
Geekdo, BoardGameGeek, the Geekdo logo, and the BoardGameGeek logo are trademarks of BoardGameGeek, LLC.