Recommend
2 
 Thumb up
 Hide
21 Posts

BoardGameGeek» Forums » BoardGameGeek Related » BGG Help and How-To

Subject: Analytics on BGG data - some questions rss

Your Tags: Add tags
Popular Tags: [View All]
Ian Pytlarz
United States
West Lafayette
Indiana
flag msg tools
badge
Avatar
mbmbmbmbmb
I've been working in data visualization and analytics professionally for about five or so years now, and I'm interested in using those skills on BGG's data for fun. However, I'm having trouble figuring out how to access the data efficiently.

From what I've gathered from some articles/forum posts is that the BGG data is all stored in XML, and it is using XML as if it were a database. Is this true? I have never heard of someone using XML to store data like this, it would amaze me if this data wasn't in a relational database somewhere. If it IS in a relational database somewhere, is there a way I could more efficiently run queries on it with, say, some SQL tool? I have some fun ideas but unfortunately I don't have the time to make some convoluted script that parses a ton of XML into a usable dataset.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Herb
United States
Georgia
flag msg tools
mbmbmbmbmb
BGG allows users to extract data via XML from the BGG databases.

From tools discussed in the past the actual data is in a relational database. BGG insulates users from SQL since the SQL database itself is the real value of the site.
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Walt
United States
Orange County
California
flag msg tools
Before terraforming Mars, Surviving Mars is required: Paradox Interactive; Steam.
badge
Please contact me about board gaming in Orange County.
Avatar
mbmbmbmbmb
See xmlapi.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Ian Pytlarz
United States
West Lafayette
Indiana
flag msg tools
badge
Avatar
mbmbmbmbmb
Tall_Walt wrote:
See xmlapi.


I've seen this, but it isn't useful for what I would want. I'd prefer to pull in bulk information on the games themselves, and having to do a new request for each and every individual game (so 60k+ requests?) would be far too slow and unwieldy. That's just my first thought though, I've never had to use XML to interact with a relational DB before so maybe there is some way I could code a method into the API to do this.

For instance, as a first pass I was hoping to select all games and their associated mechanics in a big dump, and do a network graph or chord chart to see how different mechanics tend to get clustered, alongside a legitimate cluster analysis. I'd hope to reveal areas of over-saturation or under-saturation, which may be helpful to designers, and in any case would be interesting.

There has to be a good way to get dumps of this data. If there isn't, I presume it has been discussed and I would be interested in joining that discussion. I'll have more time in the summer months if there is a possibility that we could build something.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Herb
United States
Georgia
flag msg tools
mbmbmbmbmb
Ian2400 wrote:
Tall_Walt wrote:
See xmlapi.


I've seen this, but it isn't useful for what I would want. I'd prefer to pull in bulk information on the games themselves, and having to do a new request for each and every individual game (so 60k+ requests?) would be far too slow and unwieldy. That's just my first thought though, I've never had to use XML to interact with a relational DB before so maybe there is some way I could code a method into the API to do this.

For instance, as a first pass I was hoping to select all games and their associated mechanics in a big dump, and do a network graph or chord chart to see how different mechanics tend to get clustered, alongside a legitimate cluster analysis. I'd hope to reveal areas of over-saturation or under-saturation, which may be helpful to designers, and in any case would be interesting.

There has to be a good way to get dumps of this data. If there isn't, I presume it has been discussed and I would be interested in joining that discussion. I'll have more time in the summer months if there is a possibility that we could build something.


soblue Whatever you want to do with the data, don't you realize that by cloning the site that you're stealing the essence of the site? The only thing that BGG has to offer is the data.

You have to accept the fact that BGG doesn't want anyone to suck all of the data from the site. That is why it is hard to extract all the data at one time. If BGG wanted to make it easy they they would offer to give you a copy of the database.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Ian Pytlarz
United States
West Lafayette
Indiana
flag msg tools
badge
Avatar
mbmbmbmbmb
herace wrote:
Ian2400 wrote:
Tall_Walt wrote:
See xmlapi.


I've seen this, but it isn't useful for what I would want. I'd prefer to pull in bulk information on the games themselves, and having to do a new request for each and every individual game (so 60k+ requests?) would be far too slow and unwieldy. That's just my first thought though, I've never had to use XML to interact with a relational DB before so maybe there is some way I could code a method into the API to do this.

For instance, as a first pass I was hoping to select all games and their associated mechanics in a big dump, and do a network graph or chord chart to see how different mechanics tend to get clustered, alongside a legitimate cluster analysis. I'd hope to reveal areas of over-saturation or under-saturation, which may be helpful to designers, and in any case would be interesting.

There has to be a good way to get dumps of this data. If there isn't, I presume it has been discussed and I would be interested in joining that discussion. I'll have more time in the summer months if there is a possibility that we could build something.


soblue Whatever you want to do with the data, don't you realize that by cloning the site that you're stealing the essence of the site? The only thing that BGG has to offer is the data.

You have to accept the fact that BGG doesn't want anyone to suck all of the data from the site. That is why it is hard to extract all the data at one time. If BGG wanted to make it easy they they would offer to give you a copy of the database.


I don't know that I agree with that, the offering is the front-end that allows everyone to interact with the data properly. Even if I had a clone of the games table and wanted to make money/compete with BGG, I don't see how I could really do so without far too much time investment.

Anyway, I just wanted to do some interesting stuff for kicks. If it's going to cause a fuss I'll just find a different data set to mess around with.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
A L D A R O N
United States
Cambridge
Massachusetts
flag msg tools
A L D A R O N
badge
----[---->+<]>++.+++++++++++.--------.---.>-[--->+<]>---.---.-.
Avatar
mb
herace wrote:
Whatever you want to do with the data, don't you realize that by cloning the site that you're stealing the essence of the site? The only thing that BGG has to offer is the data.

If that's the concern its a simple thing to release the data under an agreement that limits the ways in which it can be used. If anything, the risk in such cases falls on the users of the data and not the owners.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Ian Pytlarz
United States
West Lafayette
Indiana
flag msg tools
badge
Avatar
mbmbmbmbmb
Aldaron wrote:
herace wrote:
Whatever you want to do with the data, don't you realize that by cloning the site that you're stealing the essence of the site? The only thing that BGG has to offer is the data.

If that's the concern its a simple thing to release the data under an agreement that limits the ways in which it can be used. If anything, the risk in such cases falls on the users of the data and not the owners.


I'd be willing to do this, it's common practice.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
A L D A R O N
United States
Cambridge
Massachusetts
flag msg tools
A L D A R O N
badge
----[---->+<]>++.+++++++++++.--------.---.>-[--->+<]>---.---.-.
Avatar
mb
Ian2400 wrote:
Aldaron wrote:
herace wrote:
Whatever you want to do with the data, don't you realize that by cloning the site that you're stealing the essence of the site? The only thing that BGG has to offer is the data.

If that's the concern its a simple thing to release the data under an agreement that limits the ways in which it can be used. If anything, the risk in such cases falls on the users of the data and not the owners.

I'd be willing to do this, it's common practice.

Exactly; that's my point. It sounds like 'herace' (and perhaps BGG) doesn't understand that though.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Ian Pytlarz
United States
West Lafayette
Indiana
flag msg tools
badge
Avatar
mbmbmbmbmb
Aldaron wrote:
Ian2400 wrote:
Aldaron wrote:
herace wrote:
Whatever you want to do with the data, don't you realize that by cloning the site that you're stealing the essence of the site? The only thing that BGG has to offer is the data.

If that's the concern its a simple thing to release the data under an agreement that limits the ways in which it can be used. If anything, the risk in such cases falls on the users of the data and not the owners.

I'd be willing to do this, it's common practice.

Exactly; that's my point. It sounds like 'herace' (and perhaps BGG) doesn't understand that though.


I only didn't bring it up previously because it seems like it would be some amount of work for someone at BGG and they have little incentive to do it. That said, I'd be willing to do additional analysis for them in exchange if they are interested (though that may have to wait till summer). It would be nice to help the geek in ways other than donations. Also, any visualizations I did would be interactive and capable of being hosted online here, I'd be using this as a way to cut my teeth on Javascript D3.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
A L D A R O N
United States
Cambridge
Massachusetts
flag msg tools
A L D A R O N
badge
----[---->+<]>++.+++++++++++.--------.---.>-[--->+<]>---.---.-.
Avatar
mb
Ian2400 wrote:
Also, any visualizations I did would be interactive and capable of being hosted online here, I'd be using this as a way to cut my teeth on Javascript D3.

That would be a great addition to the site.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Ian Pytlarz
United States
West Lafayette
Indiana
flag msg tools
badge
Avatar
mbmbmbmbmb
Aldaron wrote:
Ian2400 wrote:
Also, any visualizations I did would be interactive and capable of being hosted online here, I'd be using this as a way to cut my teeth on Javascript D3.

That would be a great addition to the site.


Well, if anyone knows the appropriate contact I'll reach out.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Ian Pytlarz
United States
West Lafayette
Indiana
flag msg tools
badge
Avatar
mbmbmbmbmb
Quick update on this: I reached out to several people and eventually got bumped over to Aldie, who hasn't responded and probably won't. I'm guessing this is dead in the water.

Unfortunate, as just yesterday I started creating some really gorgeous interactive chord graphs...
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Maarten D. de Jong
Netherlands
Zaandam
flag msg tools
badge
Avatar
Ian2400 wrote:
For instance, as a first pass I was hoping to select all games and their associated mechanics in a big dump, and do a network graph or chord chart to see how different mechanics tend to get clustered, alongside a legitimate cluster analysis. I'd hope to reveal areas of over-saturation or under-saturation, which may be helpful to designers, and in any case would be interesting.

It would be interesting if the analysis was based on a robust categorisation of a game's mechanisms. Unfortunately that categorisation is anything but robust to the point where you have to wonder whether it is worthwhile spending time on such a project.

As for your original question: there used to be snapshots of the entire DB available. Usually these were half a year old or so. But from the discussion I see that this functionality has been quietly abandoned and ultimately withdrawn. For now, your only option is using the XML API, and hoping that your requests won't put too big a load on the server for Aldie to tell you to take a hike. Since you don't want to use this API, you are stuck.

Don't feel too bad, though. There have been many suggestions over the years, with people putting in some time to grab data through the XML API for seeding their calculations, some even useful... and in the end it all came to nothing. Aldie is, somewhat unfortunately, more of a cool stuff- rather than a useful stuff-programmer; and the announcement that the much-awaited site redesign was put on hold indefinitely because of the harsh realisation that the code base would be just as crummy as before has certainly put things in an even worse stasis than they already were.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
A L D A R O N
United States
Cambridge
Massachusetts
flag msg tools
A L D A R O N
badge
----[---->+<]>++.+++++++++++.--------.---.>-[--->+<]>---.---.-.
Avatar
mb
cymric wrote:
It would be interesting if the analysis was based on a robust categorisation of a game's mechanisms. Unfortunately that categorisation is anything but robust to the point where you have to wonder whether it is worthwhile spending time on such a project.

You'd be surprised what you can learn through analytics.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Maarten D. de Jong
Netherlands
Zaandam
flag msg tools
badge
Avatar
In my experience the lesson has invariably been 'Next time, do better in the data gathering as it'll make data processing so much easier'.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Ian Pytlarz
United States
West Lafayette
Indiana
flag msg tools
badge
Avatar
mbmbmbmbmb
cymric wrote:
In my experience the lesson has invariably been 'Next time, do better in the data gathering as it'll make data processing so much easier'.


The thing I've learned now that I've been working for a, shall we say, less than well organized (data-wise) company for a few years, is that sunlight is the best disinfectant. If you can pull the data and show where it is messed up, it is so much easier to discuss and fix. That's the area I've learned to specialize in lately, and why I've become so skilled at data visualization.

My prior experience was pure analytics which does rely on having good data, but if the need is more clean-up than analysis then I also have a lot of expertise there as well and am willing to put some time in. I just need access, and I don't have the sort of free time to putz around with an XML API pulling one record at a time.

P.S. I'm hoping to have some testing space available to me in the near future for the javascript work I'm doing, maybe I can share some of what I'm working on when I do and get more people interesting in this. I've noticed that beautiful visualizations tend to get people interested in a topic.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Herb
United States
Georgia
flag msg tools
mbmbmbmbmb
Ian2400 wrote:

Well, if anyone knows the appropriate contact I'll reach out.


The person to pitch a proposal to is Aldie. You have not only to convince him that you'll keep the data private, you also need to convince him that it is worth BGG's time to teach you(or work with you) how to access the data. I'm sure that written documentation is sparse.

Scott Alden
United States
Dallas
Texas
flag msg tools
admin
badge
Aldie's Full of Love!
Avatar
mbmbmbmbmb
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Maarten D. de Jong
Netherlands
Zaandam
flag msg tools
badge
Avatar
Ian2400 wrote:
The thing I've learned now that I've been working for a, shall we say, less than well organized (data-wise) company for a few years, is that sunlight is the best disinfectant. If you can pull the data and show where it is messed up, it is so much easier to discuss and fix. That's the area I've learned to specialize in lately, and why I've become so skilled at data visualization.

The point is that you don't even know what you are visualising in the example you gave, exactly. Yes, the data you pulled. But there is no visual indicator anywhere that that data is complete or correct. There are no outliers to identify. Games don't come with stickers outlining their mechanisms, and in many tens of thousands of cases there are simply too few owners who can be arsed to verify and/or enhance the data's accuracy. There is no way of supplying it yourself save for petitioning the BGG community as a whole.

In somewhat more abstract terms: querying the DB in whatever way will yield a crystal clear answer that game G has mechanisms M1, M2, and M3. No clean-up required. What you don't know is whether M1 really ought to be M4, whether M5 and M6 are missing, and why the hell M3 was included in the first place. You cannot rely automatically on games being related through their families to guesstimate. You cannot rely on expansions, if available. You cannot rely on mechanisms having a correlation of their own—that would in fact be a circular argument for it is precisely this link you are trying to visualise. (It can be done, of course, but the problem would become quite non-linear and thus difficult and time-consuming.) Asking the community introduces a bias for certain mechanisms simply because of their exposure to previous titles: it is not a trained taxonomist who will answer.

So, given all that, do you still want to put in the time to conduct network and cluster analyses?

Quote:
I've noticed that beautiful visualizations tend to get people interested in a topic.

It does.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Ian Pytlarz
United States
West Lafayette
Indiana
flag msg tools
badge
Avatar
mbmbmbmbmb
herace wrote:
Ian2400 wrote:

Well, if anyone knows the appropriate contact I'll reach out.


The person to pitch a proposal to is Aldie. You have not only to convince him that you'll keep the data private, you also need to convince him that it is worth BGG's time to teach you(or work with you) how to access the data. I'm sure that written documentation is sparse.

Scott Alden
United States
Dallas
Texas
flag msg tools
admin
badge
Aldie's Full of Love!
Avatar
mbmbmbmbmb


I already reached out and he didn't respond, so I assume this is pretty much dead in the water.

Quote:
In somewhat more abstract terms: querying the DB in whatever way will yield a crystal clear answer that game G has mechanisms M1, M2, and M3. No clean-up required. What you don't know is whether M1 really ought to be M4, whether M5 and M6 are missing, and why the hell M3 was included in the first place. You cannot rely automatically on games being related through their families to guesstimate. You cannot rely on expansions, if available. You cannot rely on mechanisms having a correlation of their own—that would in fact be a circular argument for it is precisely this link you are trying to visualise. (It can be done, of course, but the problem would become quite non-linear and thus difficult and time-consuming.) Asking the community introduces a bias for certain mechanisms simply because of their exposure to previous titles: it is not a trained taxonomist who will answer.


It's a good point, but as I said we don't know how dirty it is until we look, do we? Perhaps by limiting the query to only games with 100+ owners (or more) we would get more reliable data. Point is, we don't know until we try, and it seems worth taking a look to me. This also is far from the only thing we could do with the data, but until I get a good look at what is available (and, as you said, how clean it is) it's hard to come up with analytics ideas.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Ian Pytlarz
United States
West Lafayette
Indiana
flag msg tools
badge
Avatar
mbmbmbmbmb
So I managed to make the visualization I wanted using data from my job (two birds, one stone). In case anyone is interested in what kinds of visualizations I'm interested in doing, this is getting pretty close to production ready and is a good example of the sort of thing I want to do more often:

https://www.purdue.edu/oir/test/CODOchord/

My only real to-do items are making the reset button look nicer and shortening up the text on the college/department names so I can make the SVG box containing the viz smaller and tighter. Should work fine in everything except IE8 and earlier (though if you are using IE8 or earlier, you should download a real browser!).

If anyone has a good idea for a project involving board game data and any of the visualizations in d3.js, I'm all ears.
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Front Page | Welcome | Contact | Privacy Policy | Terms of Service | Advertise | Support BGG | Feeds RSS
Geekdo, BoardGameGeek, the Geekdo logo, and the BoardGameGeek logo are trademarks of BoardGameGeek, LLC.