Recommend
58 
 Thumb up
 Hide
54 Posts
1 , 2 , 3  Next »   | 

Geekdo» Forums » News

Subject: Export collections has been updated (XMLAPI developers read this) rss

Your Tags: Add tags
Popular Tags: BGG-API [+] [View All]
Scott Alden
United States
Dallas
Texas
flag msg tools
admin
badge
Aldie's Full of Love!
Avatar
mbmbmb
I have updated the exporting of collections function to take it "out of band". What this means is that when you make a request to export your collection in CSV or using the XML API it will get queued on our backend servers.

Those servers will then generate the file and save it, so that you can download it over and over without having to wait for the generation time again (which can be quite long for large collections).

This also means that API developers need to watch for the HTTP 202 response code when they request a collection that gets queued for generation.

Also, since a lot of the data in a collection for the API is dynamic (based on stats, rankings, etc...) we will hard expire them after 7 days. If you make changes to your collection - the data will be expired and you'll have to wait for the queue to export the collection again.
43 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Tim P.
United States
Thousand Oaks
California
flag msg tools
Visit the Wargame Bootcamp guild
badge
Muppet !
Avatar
mbmbmbmbmb

Yay, well done.

How do we know when the collection is ready for downloading ?

cheers

Tim
6 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
John Farrell
Australia
Rozelle
New South Wales
flag msg tools
Averagely Inadequate
badge
Buster Keaton from 'Go West'
Avatar
mbmbmbmbmb
That suggests that you'll be keeping the collection export around for some time, even if the user has changed their collection details. It would be sensible for people like me to not request it more often than you would change it, so what are the time periods involved?
7 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Todd Snyder
United States
Oregon City
Oregon
flag msg tools
badge
Avatar
mbmbmbmbmb
hmm, I use a vbscript to fetch collection data and save it as a xml file locally, I'll have to figure out what I'll need to change, as I really don't know what a HTTP 202 response code is. Does that have the path to the xml file?
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Chris Comeaux
United States
Birmingham
Alabama
flag msg tools
War Eagle!
badge
Image is from Turandot. It looks just like me.
Avatar
mbmbmbmbmb
Thanks, Aldie! That would explain why I started getting new errors. It's working on your side, but my side wasn't expecting it.

I see you've added a "pubdate" attribute that I assume is the datetime the collection was generated? And if we filter the our request via query string e.g. id), does the entire collection get generated, not just the filtered portion? Thanks again!
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jeff Michaud
United States
Longwood
Florida
flag msg tools
On-Line Want List Generator - Hopefully Making Math Trades a Little Bit Easier
badge
Captain Kirk, Captain Picard, Captain Sisko, Captain Janeway, Captain Archer
Avatar
mbmbmbmbmb
since my users ( OLWLG) tell me when they want me to sync/cache (on my end) their collection data I'll have to check for it and let them know they need to keep retrying since I can't keep my script waiting too long

I assume when you mean "servers will then generate the file and save it, so that you can download it over and over" that you mean a file in some BGG specific format and you will still generate the xml on the fly... based on the bgg xml api arguments requested (such as "modified since", etc)?

any chance of an option to have BGG re-generate the cache for a user if my user requests it... for example, after making lots of collection changes? my users like to have accurate "want"/wishlist/etc as the OLWLG uses that to determine what to display (if user is searching for only those in a math trade) and for what to highlight to bring to users attention
11 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Darth Ed
United States
Maryland
flag msg tools
badge
Avatar
mbmbmbmbmb
Here's hoping http://bggauctions.workingasintended.com will be updated soon.
3 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Scott Alden
United States
Dallas
Texas
flag msg tools
admin
badge
Aldie's Full of Love!
Avatar
mbmbmb
Friendless wrote:
That suggests that you'll be keeping the collection export around for some time, even if the user has changed their collection details. It would be sensible for people like me to not request it more often than you would change it, so what are the time periods involved?

I invalidate all collection data when a user makes a change to their collection. So next time you pull - you'll have to wait for the queue to finish exporting it.
9 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Scott Alden
United States
Dallas
Texas
flag msg tools
admin
badge
Aldie's Full of Love!
Avatar
mbmbmb
ccomeaux wrote:
Thanks, Aldie! That would explain why I started getting new errors. It's working on your side, but my side wasn't expecting it.

I see you've added a "pubdate" attribute that I assume is the datetime the collection was generated? And if we filter the our request via query string e.g. id), does the entire collection get generated, not just the filtered portion? Thanks again!

Yes, pubdate is when the collection data was exported.

Every set of filtered queries also has to go through the queue - we don't assemble from parts, it just generates the file from scratch.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Scott Alden
United States
Dallas
Texas
flag msg tools
admin
badge
Aldie's Full of Love!
Avatar
mbmbmb
JeffyJeff wrote:
since my users ( OLWLG) tell me when they want me to sync/cache (on my end) their collection data I'll have to check for it and let them know they need to keep retrying since I can't keep my script waiting too long

I assume when you mean "servers will then generate the file and save it, so that you can download it over and over" that you mean a file in some BGG specific format and you will still generate the xml on the fly... based on the bgg xml api arguments requested (such as "modified since", etc)?

any chance of an option to have BGG re-generate the cache for a user if my user requests it... for example, after making lots of collection changes? my users like to have accurate "want"/wishlist/etc as the OLWLG uses that to determine what to display (if user is searching for only those in a math trade) and for what to highlight to bring to users attention

Think of it as an XML cache of the user's collection that lasts for 7 days. If they make changes - I invalidate all their caches.
10 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Adam Kranzel
United States
Fremont
California
flag msg tools
badge
Avatar
mbmbmbmbmb
This is great. Thanks Aldie!
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Eddy Sterckx
Belgium
Vilvoorde
flag msg tools
designer
Avatar
mbmbmbmbmb
Just played around a bit with the XML API - when requesting the collection you get an XML file containing


Your request for this collection has been accepted and will be processed. Please try again later for access.


In essence you can just keep requesting the data until you get an xml file which is different from the above. So not much I need to change in my code.


2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Todd Snyder
United States
Oregon City
Oregon
flag msg tools
badge
Avatar
mbmbmbmbmb
eddy_sterckx wrote:
Just played around a bit with the XML API - when requesting the collection you get an XML file containing


Your request for this collection has been accepted and will be processed. Please try again later for access.


In essence you can just keep requesting the data until you get an xml file which is different from the above. So not much I need to change in my code.



ah, I see - yep, stuff was downloaded when I ran it again, so I'll just need to do a prerun, wait a while, then do a real run. Coolie.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Kevin O
Canada
Chatham
Ontario
flag msg tools
Avatar
mbmbmbmb
When I pull a users collection:
http://boardgamegeek.com/xmlapi2/collection?username=iron_fo...

The subtype for all elements is listed as boardgame.

So I do a second pull with:
http://boardgamegeek.com/xmlapi2/collection?username=iron_fo...

Which only returns the expansions, letting me flag which results from the collection are expansions. However, now the subtype flag returns a page with "Oops, something went wrong - try again later."

Was the subtype parameter broken with this change? As a side question, can the subtype be fixed to correctly return the type?

Thanks!

3 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Tim P.
United States
Thousand Oaks
California
flag msg tools
Visit the Wargame Bootcamp guild
badge
Muppet !
Avatar
mbmbmbmbmb
I tried it again, to make sure I did the same thing twice.... and it still did not give me what I used to get.

IIRC, I would 'Filter' the list and modify the 'Columns' being viewed to get the data types I wanted, then I would save the data as a csv. It would include the filtering and the columns that I specified.

This new collection export provides the standard listing data, specifically I am requiring the "Private Info" data as that includes which box the game is stored in.

I had found a way to get the data through the XMLAPI, but that must have stopped working yesterday afternoon, with the following message, so that is being queued too.

Quote:
Your request for this collection has been accepted and will be processed. Please try again later for access.
2 
 Thumb up
0.01
 tip
 Hide
  • [+] Dice rolls
Jack
South Africa
Cape Town
Western Cape
flag msg tools
"No single raindrop believes it is to blame for the flood."
badge
Small talk (noun): insincere questions and dishonest answers.
Avatar
mbmbmbmbmb
oi_you_nutter wrote:

Yay, well done.

How do we know when the collection is ready for downloading ?

cheers

Tim

Will we get a Geekmail?
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Tim P.
United States
Thousand Oaks
California
flag msg tools
Visit the Wargame Bootcamp guild
badge
Muppet !
Avatar
mbmbmbmbmb
IamJacksUsername wrote:
oi_you_nutter wrote:

Yay, well done.

How do we know when the collection is ready for downloading ?

cheers

Tim

Will we get a Geekmail?

Nope, repeat what you did before to export the collection, it will either tell you that the job has been submitted or (once the file has been created) it will download the CSV file.
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Sterling Babcock
United States
Longmont
Colorado
flag msg tools
admin
Winkles is here to help!
badge
Please let me know how I can help you.
Avatar
mbmbmbmbmb
Many, many thanks, Aldie! This is wonderful!
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jeff Michaud
United States
Longwood
Florida
flag msg tools
On-Line Want List Generator - Hopefully Making Math Trades a Little Bit Easier
badge
Captain Kirk, Captain Picard, Captain Sisko, Captain Janeway, Captain Archer
Avatar
mbmbmbmbmb
oi_you_nutter wrote:
Your request for this collection has been ....
can you please please remove the "code" markup around this... it makes the whole page of this thread much wider than even my wide window as the code markup tells bgg to make the styling a hard no-wrap

edit: looks like you did so, thanks!
5 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Todd Snyder
United States
Oregon City
Oregon
flag msg tools
badge
Avatar
mbmbmbmbmb
Aldie wrote:
Those servers will then generate the file and save it, so that you can download it over and over without having to wait for the generation time again (which can be quite long for large collections).

I hate to ask the obvious question here, but have you performance tuned the queries in question? Even with large collections, and even when you have to do multi table joins to get everything involved, you're not talking about that much data in the grand scope of things. My guess is there is a missing index that would greatly help. Not that you wouldn't want to offload this type of thing anyway, but if it's hurting this export process, it's probably hurting other things too. I say this coming from a DBA background with 10+ years of experience.
2 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Scott Alden
United States
Dallas
Texas
flag msg tools
admin
badge
Aldie's Full of Love!
Avatar
mbmbmb
I could probably do an geekmail notification for CSV requests if you are logged in while making the request. I'll see about adding that.
5 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Scott Alden
United States
Dallas
Texas
flag msg tools
admin
badge
Aldie's Full of Love!
Avatar
mbmbmb
Iron_Forge wrote:
When I pull a users collection:
http://boardgamegeek.com/xmlapi2/collection?username=iron_fo...

The subtype for all elements is listed as boardgame.

So I do a second pull with:
http://boardgamegeek.com/xmlapi2/collection?username=iron_fo...

Which only returns the expansions, letting me flag which results from the collection are expansions. However, now the subtype flag returns a page with "Oops, something went wrong - try again later."

Was the subtype parameter broken with this change? As a side question, can the subtype be fixed to correctly return the type?

Thanks!



I'll check to see if it changed.
3 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Scott Alden
United States
Dallas
Texas
flag msg tools
admin
badge
Aldie's Full of Love!
Avatar
mbmbmb
I expanded the subtypes supported, and also added GeekMail notifications for CSV exporting.
9 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Kean Pedersen
Denmark
Taastrup
flag msg tools
Avatar
mbmbmbmb
Ah, that is why my collection importer on boardgameprices.co.uk suddenly started giving errors for my users :) - It has been fixed now, and I think I better subscribe to this forum to ensure getting notified of any changes in the future ;)

But thank you for actually providing an API - not many sites does this! :D
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Tim P.
United States
Thousand Oaks
California
flag msg tools
Visit the Wargame Bootcamp guild
badge
Muppet !
Avatar
mbmbmbmbmb
The geekmail when job complete works great, many thanks !

The current download toolbar look like this
Quote:

Download board games: (all | owned) | Permalink | Current Filters: own=yes [X]


Btw, the url for the all and owned links are the same, i.e:
Quote:


When exporting my collection, I apply Filters and customize the Column views, these are reflected in the arguments to the Permalink link.

By taking the arguments from the Permalink url and applying those to the exportcsv function I was able to generate a csv with the customized columns that I wanted.

How about adding a Download link to the toolbar that also reflects the users customized view choices ?



3 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
1 , 2 , 3  Next »   |