Spotify AB can instantly deliver nearly any song available to hundreds of millions of users around the world. The company has experienced meteoric growth and now offers a huge amount of podcast content as well. Its total monthly average users grew 29% year over year, to 320 million, in the third quarter of this year. Such an accomplishment relies on a significant and complicated infrastructure that must scale with a constantly growing business — a mammoth undertaking, no doubt.

The company’s infrastructure is supported by thousands of microservices running in production. Spotify was an early adopter of Dockerized spaces and, more recently, Kubernetes. With thousands of data pipelines, hundreds of websites and micro-app features, Spotify conducts about 20,000 deployments per day. And speed is just as important today, if not more so. 

“To quote Daniel Burrus, we aim to ‘fail faster,’” said Jai Chakrabarti (pictured), director of engineering, Core Infrastructure at Spotify. “It’s absolutely critical that we move fast and get ideas out in the startup phase, vetted and validated and then moved into the scale-up stage.” In addition, speed is “a great virtue” as the company tests new features and works to “roll out the next thing for the audio network,” he added.

Chakrabarti spoke with Jeff Frick, host of theCUBE, SiliconANGLE Media’s livestreaming studio, during the KubeCon + CloudNativeCon NA event. They discussed Spotify’s philosophy about open source and approach to managing cost, as well as how the company motivates its engineers. (* Disclosure below.)

Solving problems at scale, and motivating engineers with open source

Philosophically, Spotify is a big proponent of open source. “We believe in giving back to the community and that when we come together as a community to solve problems at scale, the end result is much better than if we were to try it alone,” Chakrabarti explained.

In fact, that philosophy is reflected in how the company recruits and retains its engineers. Many engineers want to be active participants in open source and show their work to others outside the organization. But, at the same time they have to get company work done. Recognizing that the opportunity to work in open source is a motivating factor, Spotify allows engineers to devote up to 20% of their time on open-source projects — a high number compared to other companies, according to Chakrabarti.

Spotify has contributed to or invested in a number of open-source projects, such as Envoy, which is used to power Spotify’s perimeter, and Kubernetes, which is used for deployment purposes. Other projects the company has committed to include Cloud Bigtable, Heroic and Backstage.

Backstage acts as a central nervous system that supports about 500 engineering teams, enabling them to interface across the wide landscape of Spotify’s engineering ecosystem.

“If you’re an engineer, you can go into Backstage and manage your services, data pipelines, and micro features,” Chakrabarti explained. “You can see what other teams are doing, what the organizational structure is, as well as get recommendations and insights on your tech health and where you might need to invest more time.” 

Backstage is a one-stop developer portal where Spotify engineers spend the bulk of their time. “We open-sourced it earlier this year and have been absolutely thrilled with the response received so far,” he added.

A challenge that Spotify deals with is how to sustainably manage cloud costs in a way that doesn’t make it a compliance exercise. Engineers love to optimize, but premature optimization — or trying to make things more efficient at a stage when it is too early to do so — can drain resources. With Cost Insights, a product within Backstage, Spotify can strike a good balance between engineers’ love of optimization and making the best use of time.

“If making an investment somewhere isn’t going to move the needle for us, then we let people know that this isn’t worth their time to worry about,” Chakrabarti concluded.

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of the KubeCon + CloudNativeCon NA event. (*Disclosure: The Cloud Native Computing Foundation sponsored this segment of theCUBE. Neither CNCF nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE Media