Created by Martin D., one of my ex-colleagues from Last.fm, this is an amazing visualisation of scrobbling activity by year, month, day of the week and even time of day.

They’re manually generated at the moment, with a set for Last.fm staff past and present. The slideshow version is particularly interesting to compare varying peoples listening habits.

Each year of data is arranged in a row and horizontally grouped into 12 blocks, one for each month… Months are organised by an inner grid, where data is arranged in seven columns for the days of week and 24 rows for the hours of day. Weekdays are aggregated so that e.g. all Mondays of a particular month end up in the same column.

Colour is a measure of relative intensity: grey → green → yellow → red. A light grey strip highlights the most active hours of the day across the entire period.

Despite the depth of information it’s stunningly readable and rather interesting. You can pick out real patterns quite easily, as Martin notes;

Frequently people’s graphs are detailed enough to provide a fairly good summary of big life changes. New jobs, busy weekends, holidays, the month when they bought an iPod, or picked up running again, or moved to a different timezone, … I found that showing these graphs to the people portrayed often stimulated interesting conversation about their habits and their choices.

From my graph I was able to spot these behaviours:

• Mid-2005 to 2006 - Finished Uni, got a job and spent the first 6+ months comuting without an iPod after moving to London was still missing a laptop for a long time.
• Late 2007 - An increase in evening listening, a sign of joining the Last.fm team and getting stuck in to startup culture of late nights.
• Early 2008 - Evening listening, but with more separation from daytime listening. I suspect this was nights spent playing albums with my flatmate Ben Ward.
• 2009 - Less evening music, which I think stems from different flat mates, a different flat and starting (an unscrobbled) vinyl collection.
• February 2010 - A quiet month for scrobbling, most of which on holiday in New York, little time for digital music.

I received this email a few months ago, and it made me smile.

Anthony from The Hype Machine here. (http://hypem.com) Good news! We’ve just connected your Hype Machine account to Last.fm using a new secure method recommended by their team (it’s similar to OAuth, if you want to get technical)! To do this, we used the Last.fm username and a scrambled version of the password you’ve given us before. We’ve now deleted the scrambled version of your password from our database for extra security - with this new authentication method, we no longer need it! Have a great week!

Hype Machine is one of the many music sites that allow you to scrobble your listening there to Last.fm. This is done through our Scrobbling protocol, a part of our API, in fact the same way the official Last.fm clients (and even the website) record what you listen to. We’re proud to eat our own dogfood.

Scrobbling is core to Last.fm, each track is a chunk of attention data. It tells us more about what you like, building better recommendations of music, events, etc. The more you scrobble the better it gets, so the more 3rd parties scrobbling the better. “If it doesn’t scrobble it doesn’t count”, as Stefan says.

Having slowly evolved over 8 years the original scrobbling protocol was always a little complex and not very developer friendly. That’s something we’ve tried to improve with Scrobbling 2.0, which is a revolution, rather than evolution of the protocol.

Short history of scrobbling aside, I’m going to talk about how a 3rd party developer can migrate user authentication from the old scrobbling protocol to the new. The two systems use different authentication mechanisms, Submissions 1.3 requires a username + password, while Scrobbling 2.0 uses the same OAuth mechanism as our API.

Having a 3rd party store your Last.fm credentials is not great. Despite the protocol requiring the passport to be hashed (rather plaintext) it still increases the risk of your account being compromised, by a malicious 3rd party or just a careless one that accidentally exposes data (which happens more than anyone would like).

An advantage of OAuth style authentication is that it’s token based, meaning that a 3rd party will never need to ask for a users’ password. To get a token they direct the user to our OAuth endpoint on Last.fm where the user chooses to allow or deny the 3rd party. If allowed then a token is sent back to the 3rd party giving them access+write data for that user. Another advantage is that the user (or Last.fm) can easily revoke access for the 3rd party by deleting that token.

So, all in all OAuth is more secure and offers more control, a “Good Thing”. However, for there is a problem for users of the old srobbling protocol. In order to use Scrobbling 2.0 you need an OAuth token for a user, which is not something they’ll already have. One option would be to send all of their users through the OAuth authentication process and collect each token, but this sucks for everyone. It’s a big pain for the 3d party developers (writing new auth flow, maintaining to scrobbling protocols), the users (getting asked to allow something they’d already allowed) and for Last.fm (probably result in fewer people scrobbling as a result of the fuss). Not ideal.

A few months back one of our partners asked about this problem and we were able to come up with a novel solution that sidestepped that complexity. It relies on two things, 1/ Old-school scrobblers already store usernames and (hashed) passwords, 2/ We offer a second kind of API authentication, which exchanges a hashed username and password for an OAuth token, without direct user interaction.

So, having just said how much better OAuth based authentication is, why do we have offer an auth flow that circumvents it? Basically, user experience. The OAuth flow on mobile generally sucks, often a user won’t be logged in on a mobile device, or a mobile app 3rd party will have difficulty capturing the auth token. Sometimes a level of security has to be sarcraficed to make something more usable. It’s also not all bad, unlike the old scrobbling procotol it doesn’t need to store username and password, only use them once to get an auth token, which is still revokable by the user. It’s not ideal, but it’s an improvement.

Anyway, that authentication method requires the 3rd party to provide the username and password (in a hashed form), which older scrobbling clients already have - no need to ask the user. They can use the existing user credidentials to get an OAuth token via mobile authentication. After that they can store the new token and purge the old and insecure credentials, which is exactly what Anthony described in the opening quote. No fuss for the user, some work for the 3rd party developer, but all of it automatable (and extractable in to a publishable, resuable library).

I’m not going to go in to any actual code, in part because the post is already very long, but also because with the mobile authentication documentation it’s quite straight forward. But here’s the gist of it:

// This is the token format mobile auth expects:
md5(username + md5(password))

// How to generate it from Submissions 1.x auth details
username = scrob1_username
password_hash = scrob1_password_hash

// Generate the token and hit up the mobile auth API method
mobile_auth_token = md5(username + password_hash)

Bosh.

A couple of weeks I made a trip to Music Hackday Boston. The 4th in a series of music hackdays, designed to bring together smart and passionate people to hack on music technology.

I won’t go in to grand detail as there have already been some excellent write-ups from Ben, Anthony and Brian, but it’s by far the best hackday i’ve attended. A perfect combination of smart people, brilliant technology and a great venue, even the wifi worked properly. It’s just a shame I didn’t find much time to hack on anything myself.

Another highlight were the panels. I’m not usually a big fan of panels at hackdays, they tend to distract from the task at hand, but with the wealth of smart in the room it would have been foolish not to have them. There were only 3 but they’re well worth watching and featured people who really know what they’re talking about. Videos for: Starting a Music Business, Music Discovery and The Future of Music.

Hack-wise I was most impressed by Dan Kantor’s playdarTunes. An iTunes like web-interface that you populate by upload your iTunes library file and then play tracks through Playdar (as it’s all local content, in theory). It’s similar in principle to the Playlick player (James introduced me to Dan, in fact) and I can’t help but get excited about this idea of portable music collections. Where not only is the audio portable, but the library itself could come from different sources and is sharable. What I really want is a slick web-based iTunes which lets me select which of my libraries (or friends libraries) I want to browse and listen to.

We’re getting closer and closer to that ideal thanks to technology like Playdar, Scrobbling and HTML5 audio. Since Boston there’s been more talk on the Playdar mailing list about exposing a get library type call for a resolver. This would be handy for local content in projects like playdarTunes, but even more so for larger resolvers/services - Imagine Magnatune providing that kind of library information, you could browse a library of their artists though your choice of interface and just play it. Hot.

I gave a couple of short talks to give an overview of the Last.fm API (panickedly put together on the plane) to help people use it, which was fun. The more interesting was the second one in a smaller room, lots of interesting discussion and questions. One of the common and surprising questions was about scrobbling, “Who’s allowed to do it?”. A lot of people were under the impression it’s a proprietary system, rather than a completely open protocol. Getting more people scrobbling (on Last.fm, on other sites/services, on mobile, etc) is one of our aims for next year, so we have some serious work to do here.

I was a bit disappointed I didn’t get to meet anyone from Libre.fm, who were supposed to be there. I’ve been in touch with them before and was hoping to discuss improving exporting your data from Last.fm. It’s always been possible, but it could definitely be easier.

A big thanks to organisers Dave Haynes, Jon Pierce and Paul Lamere who did an excellent job. Microsoft for providing facilities and Brian Whitman of the Echo Nest for being a gracious host.

I’m really looking forward to Music Hackday Stockholm now.