Recommend
10 
 Thumb up
 Hide
96 Posts
1 , 2 , 3 , 4  Next »   | 

BoardGameGeek» Forums » BoardGameGeek Related » BGG General

Subject: BGG XML Snapshot Discussion rss

Your Tags: Add tags
Popular Tags: xmlapi [+] XML [+] [View All]
Matthew Gray
United States
Reading
Massachusetts
flag msg tools
admin
Avatar
mbmbmbmbmb
Have you ever wished you could download the entire BGG database? Well, you can't, so stop thinking about it.

But, have you ever wished you could download part of the BGG database? How about a really really useful part? Well, now you can!

You can now download a BGG XML Snapshot, which contains basic details and stats for all of the ranked games in the database.

The intent is to make this periodically updated, but I wanted to get some general community feedback before everything was finished.

Thoughts, comments, praise, questions and suggestions are enthusiastically requested. Criticism is welcome as well, though less enthusiastically.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Michael Alexander
Australia
Canberra
ACT
flag msg tools
beef
badge
sack
Avatar
mbmbmbmbmb
Thanks for the great news, offline browsers here we come

I've run some regular expressions on it to remove all the info and just have game names, been very useful for me to speed up the searching process using SAX techniques

http://beefsack.com/beefcrap/bgg-snapshot-search.zip
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Brent Mair
United States
Roy
Utah
flag msg tools
The Meeple Nation Boardgame Podcast - 30 minutes a week!
badge
BGGCon 2015 or Bust
Avatar
mbmbmbmbmb
Is there a reason why it is gzipped? Some of us are winders users and even though I have a computer job I don't have various mad skills for opening such files.
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Matthew Gray
United States
Reading
Massachusetts
flag msg tools
admin
Avatar
mbmbmbmbmb
Spielguy wrote:
Is there a reason why it is gzipped? Some of us are winders users and even though I have a computer job I don't have various mad skills for opening such files.


Are you serious? It's the year 2006 and Windows doesn't come with a program that can handle gzip compression? I'm not being sarcastic here. Really? I don't use Windows, but this seems insane. Isn't there some sort of bundled unzip program or the like that can handle it?

If not, I look into some sort of alternative, but I'd suggest looking for some sort of operating system that comes with basic tools like "compression".

At worst, you can get cygwin for windows which certainly comes with the right tools.
3 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jeff Michaud
United States
Longwood
Florida
flag msg tools
On-Line Want List Generator - Hopefully Making Math Trades a Little Bit Easier
badge
Captain Kirk, Captain Picard, Captain Sisko, Captain Janeway, Captain Archer
Avatar
mbmbmbmbmb
www.winzip.com
Spielguy wrote:
Is there a reason why it is gzipped? Some of us are winders users and even though I have a computer job I don't have various mad skills for opening such files.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Rick Holzgrafe
United States
San Jose
California
flag msg tools
designer
badge
"the understandably frightening and chimeric semicolon" -- HiveGod
Avatar
mbmbmbmbmb
Matt, this is wonderful. Thank you!

I've posted elsewhere my methods for getting BGG info onto my Palm device; this will make it nearly trivial. I will be ripping your XML down into a CSV file, and loading that into a HanDBase database. HanDBase costs money but it's not real expensive, and comes with desktop software for both Windows and Macs (alas, not Linux). See http://www.ddhsoftware.com/handbase.html

If there's interest, I might eventually post the Java app that I'll be writing to do the conversion to CSV.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Iain Cheyne
United Kingdom
Reading
Berkshire
flag msg tools
badge
Avatar
mb
Screw winzip. It's payware and does nothing special. www.7-zip.org is better.
12 
 Thumb up
0.01
 tip
 Hide
  • [+] Dice rolls
Jeff Michaud
United States
Longwood
Florida
flag msg tools
On-Line Want List Generator - Hopefully Making Math Trades a Little Bit Easier
badge
Captain Kirk, Captain Picard, Captain Sisko, Captain Janeway, Captain Archer
Avatar
mbmbmbmbmb
been using winzip for at least a decade and haven't paid yet... I'm still "evaluating" it
icheyne wrote:
Screw winzip. It's payware and does nothing special. www.7-zip.org is better.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Dave Peters
United States
Belmont
California
flag msg tools
Powers:Coleridge:Milton: Faith...must be, if anything, a clear-eyed recognition of the patterns and tendencies, to be found in every piece of the world's fabric, which are the lineaments of God.
badge
That's Tim Powers' fictional Samuel Coleridge "quoting" John Milton in _The Anubis Gates_.
Avatar
mbmbmbmbmb
Thanks, Matt; a very useful feature.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Brent Mair
United States
Roy
Utah
flag msg tools
The Meeple Nation Boardgame Podcast - 30 minutes a week!
badge
BGGCon 2015 or Bust
Avatar
mbmbmbmbmb
mkgray wrote:


If not, I look into some sort of alternative, but I'd suggest looking for some sort of operating system that comes with basic tools like "compression".


XP comes with a bundled compressor but it doesn't handle this unix formatted compression. What's with all this proprietary Unix stuff anyway?

I will certainly try one of these other programs. Last I tried winzip it didn't open .gz compresed files.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jeff Michaud
United States
Longwood
Florida
flag msg tools
On-Line Want List Generator - Hopefully Making Math Trades a Little Bit Easier
badge
Captain Kirk, Captain Picard, Captain Sisko, Captain Janeway, Captain Archer
Avatar
mbmbmbmbmb
Quote:
Last I tried winzip it didn't open .gz compresed files.

I'm using version 9 from 2 years ago and it reads .gz just fine
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Matthew Gray
United States
Reading
Massachusetts
flag msg tools
admin
Avatar
mbmbmbmbmb
I've got to say I'm really stunned that Windows doesn't come with a compressor that can handle a 15-year-old but still very good, completely patent unencumbered, open-source-implementation available, highly-ubiquitous compression algorithm.

But, despite my shock, it sounds like there are plenty of downloadable options, so I'm going to stick with gzip.

Spielguy wrote:
What's with all this proprietary Unix stuff anyway?


I'm assuming the smily is for acknowledging the irony, but on the off cheance it's not an ironic smile, I'll point out that gzip is the exact opposite of proprietary. (that's a happy smile, not an ironic or smug smile)

Quote:
I will certainly try one of these other programs. Last I tried winzip it didn't open .gz compresed files.


If you do like Unix-like interfaces, Cygwin makes Windows a lot more tolerable and it does include gzip tools. Plus, Cygwin includes perl and ruby and python if you want, which are all useful for mucking with XML files. I'll probably make a YAML version available as well at some point.
3 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Tim K.
United States
Seattle
Washington
flag msg tools
badge
Avatar
mbmbmbmb
As a thrift store junkie it would be extremely useful to have some price history data in there as well. I realize that this is a wish.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Michael Alexander
Australia
Canberra
ACT
flag msg tools
beef
badge
sack
Avatar
mbmbmbmbmb
I use Power Archiver 2006, found it to be my favourite archiving proggy, does gzip, 7z, rar, ace, even opens ISO I reckon it's worth a look for anyone

Going thrifting right now with the handy little PPC prog and this database, and my friend will be using my WAP implementation so we can search games fast the XML API is great
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Matthew Gray
United States
Reading
Massachusetts
flag msg tools
admin
Avatar
mbmbmbmbmb
beefsack wrote:
I use Power Archiver 2006, found it to be my favourite archiving proggy, does gzip, 7z, rar, ace, even opens ISO I reckon it's worth a look for anyone

Going thrifting right now with the handy little PPC prog and this database, and my friend will be using my WAP implementation so we can search games fast the XML API is great


Best of luck! I look forward to hearing stories of gems found or travesties avoided thanks tot he API/snapshot.

My commission, as usual, is 10%.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Simon Hunt
United States
Roseville
California
flag msg tools
designer
badge
Avatar
mbmbmb
Rats,

I looked in the stats data and discovered that "Users Wanting" was not represented. (Nor "Wishlist" data).

I wanted to take a look at games with more people wanting than owning.

 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Adam Skinner
United States
Seven Hills
Ohio
flag msg tools
Avatar
mbmbmbmbmb
Powerarchiver is good. Personally I use ZipGenius now.

I noticed that publisher isn't encapsulated in a container. Same thing with names. Given, I haven't tried to use this, but I figure that if we're going to be converting this xml to objects, it's best to not duplicate things like that.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Tobias Minich
Germany
Unspecified
Unspecified
flag msg tools
mb
rholzgrafe wrote:
Matt, this is wonderful. Thank you!

I've posted elsewhere my methods for getting BGG info onto my Palm device; this will make it nearly trivial. I will be ripping your XML down into a CSV file, and loading that into a HanDBase database. HanDBase costs money but it's not real expensive, and comes with desktop software for both Windows and Macs (alas, not Linux). See http://www.ddhsoftware.com/handbase.html

If there's interest, I might eventually post the Java app that I'll be writing to do the conversion to CSV.


For those who don't want to spend money there is PilotDB (http://pilot-db.sourceforge.net/) and it's java desktop software JPilot-DB (http://www.chickenshick.com/jpilot-db/). The advantage, beside it being free, is that JPilot-DB can handle xml, although I so far haven't tested that feature.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
John Farrell
Australia
Rozelle
New South Wales
flag msg tools
Averagely Inadequate
badge
Buster Keaton from 'Go West'
Avatar
mbmbmbmbmb
How often is the snapshot updated? My stats program downloads stats for each game that it needs every couple of days. If I knew that the snapshot was at most 2 days out of date at any time, I would change it to use that instead.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Michael Leuchtenburg
United States
Cambridge
Massachusetts
flag msg tools
badge
Avatar
mbmbmbmbmb
Friendless wrote:
How often is the snapshot updated? My stats program downloads stats for each game that it needs every couple of days. If I knew that the snapshot was at most 2 days out of date at any time, I would change it to use that instead.


The original link still points to the original snapshot. If you look at the list of older snapshots through the link on the wiki page, you can see the dates of the last update. The most recent update is from the 15th of December, 2006.

Any chance of a cronjob to keep it up to date, Matt?
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Michael Leuchtenburg
United States
Cambridge
Massachusetts
flag msg tools
badge
Avatar
mbmbmbmbmb
Thanks for the update, Matt! Any chance we could get the weight added to the snapshot? I know the XML API now has it, according to this: http://www.boardgamegeek.com/thread/157618
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jeff Michaud
United States
Longwood
Florida
flag msg tools
On-Line Want List Generator - Hopefully Making Math Trades a Little Bit Easier
badge
Captain Kirk, Captain Picard, Captain Sisko, Captain Janeway, Captain Archer
Avatar
mbmbmbmbmb
dyfrgi wrote:
Thanks for the update, Matt! Any chance we could get the weight added to the snapshot? I know the XML API now has it, according to this: http://www.boardgamegeek.com/thread/157618

Or all the stats one would get with the /xmlapi/game api
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Michael Leuchtenburg
United States
Cambridge
Massachusetts
flag msg tools
badge
Avatar
mbmbmbmbmb
JeffyJeff wrote:
dyfrgi wrote:
Thanks for the update, Matt! Any chance we could get the weight added to the snapshot? I know the XML API now has it, according to this: http://www.boardgamegeek.com/thread/157618

Or all the stats one would get with the /xmlapi/game api


Ideally, yes. Then I won't have to write something to maintain my own local database by pulling stats from the XML API - I can just convert the XML dump into a more queryable format. That'd mean less load on the BGG servers, and less work for me (and other people who want to play with stats), so I'm definitely in favor.

People wanting to run stats on words in rating comments will still need to write a scraping script.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Kevin Williams
United States
Davis
California
flag msg tools
mbmbmbmbmb
Any idea when the newest snapshot is coming out? I was hoping to use the weight, mechanics, and category data. If it won't be for a while, then I'll work on pulling it off the API.

Thanks
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
David Spitzley
United States
Belleville
Michigan
flag msg tools
Avatar
mbmbmbmbmb
Would it be possible to add Users Wanting and Users Trading after Owned? If so, then I would happily generate a "Thrifter's File" for download onto PDAs of those who want to avoid buying garbage.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
1 , 2 , 3 , 4  Next »   | 
Front Page | Welcome | Contact | Privacy Policy | Terms of Service | Advertise | Support BGG | Feeds RSS
Geekdo, BoardGameGeek, the Geekdo logo, and the BoardGameGeek logo are trademarks of BoardGameGeek, LLC.