Wouter de Bie, Team Lead of Data Infrastructure with Spotify, was interviewed by theCUBE co-hosts John Furrier and Dave Vellante, at #BigDataNYC 2013 in New York.

“Big data is changing the landscape of the business front, and there’s still a lot of technical opportunities with it, especially with open source.” noted Furrier. For this segment, the conversation revolved around the data involved in music streaming.

“Talking about data, right from the beginning we knew that data was going to drive this company,” said de Bie. “We knew that we would have a lot of users – that was the goal – and that would create a lot of data. We started thinking and talking about leveraging that data to do smarter business. We’ve invested a lot of time and money into building a true data infrastructure that basically everyone from our company is able to use today.”

Spotify used Hadoop early

Asked to reveal more about the working process within the company, de Bie admitted that the Spotify was one of the earlier adopters of Hadoop because they knew they were going to have a lot of data.

“It all started with reporting towards record labels and license holders and, up to a certain point it was pretty simple. Then our CEO – who is a big visionary in the industry – said that we should leverage that data and start doing analytics. That was the next step: business intelligence, analytics – and from there we’ve taken that to using data in our product: we do recommendations with the machine learning,” explained de Bie.

In that respect, Hadoop was a given. “At that time, and even today, Hadoop is the platform for doing Big Data. We’ve evolved on that, we’ve upgraded our versions and now we’re doing business with Hortonworks and they give us a great platform for exploring and exploiting Hadoop.”

Using data to disrupt the market

Noting that Spotify “has a very disruptive business model,” Vellante wanted de Bie to talk more in depth about that.

“It’s all basically about scale and being able to handle scale,” de Bie started. “We serve more than 20 million tracks, so being able to scale all of our backend systems across different continents, that is the big driver. The growth of the company is the main business driver that falls down onto the IT.”

As for the data conversation, it is pretty similar: “The more users we get, the more features we put into our product. Spotify is known as the iTunes in the cloud, but it’s much more than that. We have radio, discovery of music, playlist and artist pages, we are on a multitude of different platforms – mobile, web, desktop – and with all those features there’s more data. Since we’ve been trying to transform Spotify into a data driven company we’ve seen that data exploding internally, across 3 dimensions:

.
1. product features
2. amount of users that join the service
3. internal usage of data.”

As an IT practitioner, “a ‘data driven company’ means to be able to use data that we have to validate our hypothesis with data. Four years ago Spotify was driven more on a gut feeling. Nowadays we do a lot of AB testing – checking what features work and what features don’t.”

Impact on productivity

“We’re trying to enable everybody within the company to develop based on data, making them learn from what they do. Because they have access to data, they can iterate much quicker.”

Make sure you watch the entire segment to catch de Bie talking about machine learning, collaborative filtering and the infrastructure behind Spotify.