lib.getTracks() Hosed - Returns Dups & Omitting Records

 
  • lib.getTracks() Hosed - Returns Dups & Omitting Records

    Something's up with the either the data access logic, or the data itself. The feed is returning duplicate records throughout (example), and it's also omitting 100's to 1000's of records.

    The records listed below were taken from page #44 & #72 of the getTracks() call against my own account (JustSomeOldJoe). The problem is, the same record exists on "both" pages.

    Likewise, I've run the paginated feed from start to finish and returned with diff record counts across runs. Lib should have about 5912 tracks returned. Set returned ranges from 3900 to 4100 rows (5912 returned, but up to 2000 dups).

    Note this is being run with a working, unchanged process. Example records listed below.

    PAGE #44
    -------------------------------------------------------------------------------
    <track>
    <name>Lucky And Unhappy</name>
    <playcount>1</playcount>
    <tagcount>1</tagcount>
    <mbid></mbid>
    <url>http://www.last.fm/music/Air/_/Lucky+And+Unhappy</url>;
    <duration>270000</duration>
    <streamable fulltrack="0">1</streamable>
    <artist>
    <name>Air</name>
    <mbid>cb67438a-7f50-4f2b-a6f1-2bb2729fd538</mbid>
    <url>http://www.last.fm/music/Air</url>;
    </artist>
    <album>
    <name>10 000 Hz Legend</name>
    <position></position>
    </album>
    <image size="small">http://userserve-ak.last.fm/serve/34s/14377575.jpg</image>;
    <image size="medium">http://userserve-ak.last.fm/serve/64s/14377575.jpg</image>;
    <image size="large">http://userserve-ak.last.fm/serve/126/14377575.jpg</image>;
    <image size="extralarge">http://userserve-ak.last.fm/serve/300x300/14377575.jpg</image>;
    </track>


    Page #72
    ----------------------------------------------------------------------------
    <name>Lucky And Unhappy</name>
    <playcount>1</playcount>
    <tagcount>1</tagcount>
    <mbid></mbid>
    <url>http://www.last.fm/music/Air/_/Lucky+And+Unhappy</url>;
    <duration>270000</duration>
    <streamable fulltrack="0">1</streamable>
    <artist>
    <name>Air</name>
    <mbid>cb67438a-7f50-4f2b-a6f1-2bb2729fd538</mbid>
    <url>http://www.last.fm/music/Air</url>;
    </artist>
    <album>
    <name>10 000 Hz Legend</name>
    <position></position>
    </album>
    <image size="small">http://userserve-ak.last.fm/serve/34s/14377575.jpg</image>;
    <image size="medium">http://userserve-ak.last.fm/serve/64s/14377575.jpg</image>;
    <image size="large">http://userserve-ak.last.fm/serve/126/14377575.jpg</image>;
    <image size="extralarge">http://userserve-ak.last.fm/serve/300x300/14377575.jpg</image>;
    </track>

    • Zescher escribió...
    • Suscriptor
    • 19 Abr 2010, 6:50
    Same issue with library.getArtists

    R.I.P.(Resisto, Insisto, Persisto).
  • Well... that ain't good.

    • Egg21 escribió...
    • Usuario
    • 20 Abr 2010, 10:00
    I am having similar trouble with library.getartists. It seems some artists are missing.

    Getting all 90 pages and dumping them into one file I can search i.e. Queen. Not found. Going to the artist page I see “19 plays in your library” (which is true).

    If you want to find new bands then try joining under 2000 listeners
  • Quick update on this. I've been listening to music and scrobbling since the post (doh!). So, the page numbers referenced will obviously be off (as might the specific examples). I'll take a look and try to update the page num/examples.

    Note that the total number of records returned "is" correct. It's just that many of the records returned are dups of others in place of the correct record (leading to dups and missing records).

    You can verify this by running library.getTracks() against a lib, capturing the returned track count, then taking the results and throwing 'em into a map (hashed on artist/album/track combo). Take your count from the map and compare it to the count against getTracks(). If the counts don't match, you got dups.

    Anyhow... will update examples as soon as I find 'em.

  • UPDATE:

    Instead of searching through a few 100 pages looking for dups (or hammering the back-end with a lib.getTracks() call against a large lib), I decided to write a quick test against a small lib to confirm a "hunch". Lib in question is "JSOJ-Jazz".

    Running library.getTracks() returns 269 records. Mapping those recs keyed by Artist/Album/Track produces 169 unique records. A small subset of the recs returned from lib.getTracks() is listed below.

    Notice the dups. Then, notice the returned rec count from lib.getTracks() (269) vs. the playcount for that acct (269).

    Looks like lib.getTracks() is getting crossed with user.getRecentTracks() on the back end... will write a quick check to confirm.

    Anyhow, the subset of recs returned from library.getTracks() run against JSOJ-Jazz...


    Ahmad Jamal: The Girl Next Door
    Allie Moss: Corner
    Allie Moss: Corner
    Allie Moss: Corner
    Art Blakey & The Jazz Messengers: Are You Real
    Art Blakey & The Jazz Messengers: Are You Real
    Art Blakey & The Jazz Messengers: Are You Real
    Art Blakey & The Jazz Messengers: Blues March
    Art Blakey & The Jazz Messengers: Dat Dere
    Branford Marsalis: Emanon
    Cannonball Adderley: One For Daddy-O
    Cannonball Adderley: One For Daddy-O
    Cannonball Adderley: One For Daddy-O
    Charles Mingus: Bird Calls
    Charles Mingus: Boogie Stop Shuffle
    Chick Corea: Love Castle
    Chick Corea: Love Castle
    Clifford Brown: Yesterdays
    Clifford Brown: Yesterdays
    Grant Green: Django
    Grant Green: Django (Alternate Version)
    Hank Mobley: The Turnaround

  • I've had a poke around to try and work out what's going on here, it certainly looks like a nasty bug. library.getTracks and library.getArtists share a similar code path so it's likely one underlying cause.

    I've managed to get make a simple test case using the JSOJ-Jazz account, the same track (Cannonball Adderley - One For Daddy-O) appears one page 1 and 4.

    http://ws.audioscrobbler.com/2.0/?method=library.gettracks&api_key=b25b959554ed76058ac220b7b2e0a026&user=JSOJ-Jazz&page=1

    http://ws.audioscrobbler.com/2.0/?method=library.gettracks&api_key=b25b959554ed76058ac220b7b2e0a026&user=JSOJ-Jazz&page=4

    I'll file a ticket and we'll look in to it properly soon. Thank you all for the exhaustive testing and bug reporting details, it really helps!

    Editado por dunk el 16 May 2010, 1:37
  • Right, we've tracked the bug down and managed to fix it. It'll be a day or two before the fix goes live.

    The short version is there was a sorting bug for different artists/tracks with the same playcount and the order would vary over multiple (paginated) requests.

    Cheers for the excellent bug reports guys,
    David

    • [Usuario eliminado] escribió...
    • Usuario
    • 22 Abr 2010, 0:08
    underpangs said:
    Right, we've tracked the bug down and managed to fix it. It'll be a day or two before the fix goes live.

    The short version is there was a sorting bug for different artists/tracks with the same playcount and the order would vary over multiple (paginated) requests.

    Cheers for the excellent bug reports guys,
    David
    Hi,

    Noob question - is this just a service fix, or will it also remedy the same problem which was occurring on the site, e.g. as mentioned here?

  • StudleyUK said:
    underpangs said:
    Right, we've tracked the bug down and managed to fix it. It'll be a day or two before the fix goes live.

    The short version is there was a sorting bug for different artists/tracks with the same playcount and the order would vary over multiple (paginated) requests.

    Cheers for the excellent bug reports guys,
    David
    Hi,

    Noob question - is this just a service fix, or will it also remedy the same problem which was occurring on the site, e.g. as mentioned here?


    *skims*

    Same root cause (and code) so it should fix both :)

    • Egg21 escribió...
    • Usuario
    • 23 Abr 2010, 9:55
    Great! Thank you for the bug fix and the answers. You are a hero!

    If you want to find new bands then try joining under 2000 listeners
    • Egg21 escribió...
    • Usuario
    • 23 Abr 2010, 9:59
    underpangs said:
    Right, we've tracked the bug down and managed to fix it. It'll be a day or two before the fix goes live.

    Not yet. The bug is still there.

    If you want to find new bands then try joining under 2000 listeners
  • This bug also applies to library.getArtists, so be sure to fix that one too.


    • Egg21 escribió...
    • Usuario
    • 24 Abr 2010, 20:15
    Bug is still there.

    If you want to find new bands then try joining under 2000 listeners
    • Egg21 escribió...
    • Usuario
    • 4 May 2010, 14:38
    May 4th: Bug is still there.

    If you want to find new bands then try joining under 2000 listeners
    • Egg21 escribió...
    • Usuario
    • 14 May 2010, 19:41
    May 14th: Bug is still there. Wasn't it supposed to have been fixed?

    If you want to find new bands then try joining under 2000 listeners
    • Egg21 escribió...
    • Usuario
    • 21 May 2010, 18:51
    May 21th: Bug still there. Sigh!

    If you want to find new bands then try joining under 2000 listeners
  • Egg21 said:
    May 21th: Bug still there. Sigh!


    Hi,

    Apologies for not seeing the bumps for this topic earlier. I've had a quick look at my test cases but I can't replicate the bug.

    Could you you give some examples?

    Cheers,
    David

    • Egg21 escribió...
    • Usuario
    • 24 May 2010, 19:57
    On calling library.getartists with user=Egg21 just now, I get
    Sunn O))) on all pages 27-36.

    Frank Thorstein on page 49 and 85.

    What concerns me a lot more is the artists I am not getting. Like the 3 doors down, Cold, Hoobastank and Zz top is not on any of the pages.

    If you want to find new bands then try joining under 2000 listeners
    • Egg21 escribió...
    • Usuario
    • 25 May 2010, 8:20
    Some more examples:
    Zloty Dawai on all pages 27-33
    Yeah Yeah Yeahs on all pages 27-34
    The Ropes on all pages 30-37
    Moi Caprice on all pages 27-36

    If you want to find new bands then try joining under 2000 listeners
  • Thanks for the examples, helps me get to the bottom of the problem really quickly, thanks!

    I can only apologise, this was basically the same bug but happening in a different code path, in some cases that I missed that the first time round.

    I'm going through this a second time, identifying and fixing cases where we're using an inconsistent sort. It may be a day or two before this is completely fixed and the live web services updated.

    Again, sorry I didn't catch this first time round and thanks for your patience. I'll bump this thread when the fix is live.

    Update: I've commited a single fix for library.getArtists, that will go out with our next patch deploy. I've also found a number of other places this problem would happen, mostly with library filtering.

    I'm working on a more complete fix, but as it touches a fair bit of code i want to run it through our QA process, which won't be released till mid/end next week.

    - David

    • Egg21 escribió...
    • Usuario
    • 26 May 2010, 9:50
    Thank you very much!

    If you want to find new bands then try joining under 2000 listeners
    • Egg21 escribió...
    • Usuario
    • 28 May 2010, 9:46
    Works now. Great!

    If you want to find new bands then try joining under 2000 listeners
  • No problem.

    The library.getArtists fix is live, but some of the more esoteric API calls (eg library.getAlbums by artist) that have this bug won't be fixed till next week. That's so we can do full testing :)

    - David

Los usuarios anónimos no pueden escribir mensajes. Para participar en los foros inicia sesión o crea una cuenta.