The rise of Kubernetes came about through a combination of forces that were, in hindsight, quite a long shot.

Amazon Web Services Inc.’s dominance created momentum for cloud-native application development and the need for simpler experiences beyond easily spinning up compute as a service. This wave crashed into innovations from a startup named Docker Inc. and a reluctant open-source benefactor in Google that needed a way to change the game on Amazon.com Inc. in the cloud.

Factor in Red Hat Inc., which needed a path beyond Linux and was just about to opt for an alternative to Kubernetes to power OpenShift. Finally, figure out a governance structure to herd all the cats in the ecosystem so you can win out over other competition and create a de facto standard. Make all that happen and you get the remarkable ascendancy of Kubernetes.

Such was the story unveiled recently in a new two-part documentary series from Honeypot titled simply “Kubernetes.” In this Breaking Analysis, we tap the back stories of this documentary, which explains the improbable events leading to the creation of Kubernetes. We’ll share commentary from early Kubernetes committers and key players who came on theCUBE, SiliconANGLE’s video studio, to piece together how it all happened. Finally, we’ll present new survey data from Enterprise Technology Research on containers.

The story of Kubernetes

Kubernetes Part 1 explains the dynamics of the technology industry in the early part of last decade, the impact that AWS had on the DevOps movement, the importance of Docker, which blazed a new trail for developers, and how Google ultimately landed on the decision to open-source Kubernetes. Part 2 of the series introduces the contribution of Red Hat and how the ecosystem evolved to create a new standard for software development.

The documentary is sponsored by Google, Red Hat and the CNCF. It includes commentary from many of the players who were early committers or key individuals who influenced the development of Kubernetes. Many of these innovators came on theCUBE at various events during the formative years of mainstream containers, including Tim Hockin, Kelsey Hightower, Craig McLuckie, Joe Beda, Brian Grant, Solomon Hykes, Jerry Chen and others. John Furrier and Stu Miniman were at the many shows theCUBE covered and unpacked what was happening at the time. In this post, we’ll share the commentary from the guests they interviewed.

Developer-defined infrastructure

Over the years we’ve used several phrases to describe the programmability of infrastructure. Jerry Chen is a partner at Greylock Ventures. Years ago, he coined the term developer-defined infrastructure or DDI. Chen was head of VMware Inc.’s cloud product group and left the company in 2013 to become a full time investor.

His first investment was Docker. Chen explained on theCUBE what happens when infrastructure is defined in software — how breaking down infrastructure into components that each have an application programming interface allows developers to control infrastructure through code and make applications portable, and how DDI completely disrupted the entire application development process.

Listen to Jerry Chen from 2015 explain DDI and how it changed development in IT.

The concept of DDI or what Wikibon’s David Floyer called software-led infrastructure in 2012, was spawned from the cloud and the obvious opportunity for automation. This ushered in the DevOps movement, which was initially seen as something for startup developers, but over time, as Kelsey Hightower explains, eventually it caught on with the ops crowd.

Listen to Kelsey Hightower at KubeCon 2017 explain the dynamic between dev and ops.

Abstractions on abstractions

What Jerry was explaining is a new abstraction layer that was being built to essentially make infrastructure invisible to developers. Here’s some ETR data that quantifies that trend and visually shows you where we are today.

The chart above shows Net Score or spending momentum on the vertical axis and Market Share, which represents the pervasiveness in the survey. As Jerry Chen and the innovators who created Docker saw, the cloud was becoming prominent and you can see it still has spending velocity that is elevated above that 40% red line – kind of a magic mark of momentum. And of course it’s prominent on the X axis. Cloud is what got this movement going.

Below the 40% line you see the low-level infrastructure, including virtualization. Virtualization is an abstraction layer above servers, storage and networking. Back in 2011, in a VMware briefing with Chad Sakac, we first heard the stated goal the company had to make infrastructure “invisible.” Containers are yet another abstraction that lives above virtualization.

And ironically the spending chart above shows this. The two red arrows point to containers – container orchestration and container platforms – which are the abstraction layers and services above the underlying VMs and hardware.

And you can see the spending momentum containers have in the market along with cloud, artificial intelligence, machine learning and robotic process automation.

How the stars aligned to create Kubernetes

You had the forces that Jerry Chen and Kelsey Hightower described taking shape. The “Kubernetes” documentary does an excellent job explaining how they came together in the words of the insiders at Google, Red Hat and CNCF. The picture below identifies the key elements and we’ll describe their improbable interaction that formed Kubernetes.

In the upper left, you see AWS. We inserted a picture from a post we did right after the first re:Invent in 2012. It was obvious Amazon was the cloud gorilla and had all the momentum in IT. We saw it as an unstoppable wave, as did the protagonists in the documentary.

Solomon Hykes, the founder of Docker (pictured, upper right) saw the need to simplify the packaging of applications for cloud developers. Containers had been around for more than 30 years prior to Docker, initially developed as part of the Unix operating system to isolate certain processes from other application components. But the vast majority of programmers didn’t know or care about containers. They were a low-level construct and hidden from the mainstream.

Docker changed all that. Docker at the time was a 30-person company and had just changed its name from dotcloud. It’s astounding to think that this tiny company changed the history of computing.

Listen to Solomon Hykes answer the question in 2014: “What is a container?”

Google’s need to change the cloud game tips the scales

Imagine you’re at Google in 2013. You’ve built the world’s largest and most impressive computer network with millions of servers around the world. You have great tech and along comes Amazon, an online commerce company. Its entry into the cloud captures the minds and wallets of developers and organizations everywhere. Amazon has all the momentum in IT and your business is built on appropriating consumer data to sell ads. And your motto at the time is “Don’t be evil.”

If you’re an engineer at Google, you want to change the world. You want to develop something meaningful. You don’t want to just follow Amazon – that won’t get you far. So what’s the play? Hence in the diagram above the question mark in the Google cloud logo.  You see Docker emerge and it creates an opportunity. Docker is great because it simplified containers and took them mainstream. But Google sees that you need more. Heck, Docker saw that you needed more, and so did Mesosphere and others.

Why do you need something other than Docker?

So why exactly would you need more than what Docker had created? Craig McLuckie, who was a product manager at Google in 2014, explained to theCUBE host John Furrier the need for another management layer. He shares that what Docker did was create the capability to run a binary anywhere. But there needs to be a way to manage that with a framework that can run anywhere.

Listen to Craig McLuckie discuss the union of Docker and Kubernetes and why there needed to be another layer.

So McLuckie explained why the industry needed something like Kubernetes, but let’s go further back in time. Here’s Google with this amazing cloud infrastructure but no commercial cloud business to compete with AWS – at least not one that was taken seriously at the time. It had compute as a service and could spin up virtual machines easily, but it really wasn’t that exciting in Google’s mind and AWS had nailed that use case. So Google needed a way to increase its relevance.

Google never embraced virtualization to solve its internal challenges

Google had this thing called Google Borg, which is a container management system and scheduler. And Google looked at the problem in cloud, which was not just “How to I spin up a virtual machine?” but rather “How do I run and manage my code?”

Joe Beda, who was with Google at the time, explains to Furrier and Miniman the company’s mindset and how it saw the opportunity.

Listen to Joe Beda explain how Google engineers thought about the problem and how they thought about going beyond VMs as a service.

So Google looked at the market and said, “What can we do to make Google relevant in cloud?” Eric Brewer from Google explained on theCUBE Google’s thought process at the time. Google started the same year as VMware. So it never used virtualization for its own infrastructure. Rather, it used containers and low-level processes to build its cloud network. So it came at the problem with a different mindset.

Listen to Eric Brewer, Google’s vice president of infrastructure, explain the company’s use of containers in its early days.

Now, Brewer gave us the short version of the story. What he reveals in the documentary is that initially his posture was an instant negative reaction to open-sourcing Kubernetes. Google management was skeptical and asked, “Why would we open-source our secret sauce to help our competitors?” But over time and through a series of meetings and discussions, Google’s engineers were able to convince management that open-sourcing Kubernetes was not only good for the industry but also good for Google.

Google management was reluctant to open-source Kubernetes

Tim Hockin and Brian Grant were on the original Kubernetes team and pressed management to persuade them to bless open-sourcing Kubernetes. Hockin explained to Furrier on theCUBE the narrative inside Google at the time.

Listen to Tim Hockin explain how they pitched the idea of open-sourcing Kubernetes to senior Google management.

The open-source strategy became more compelling as they studied the problem because it gave Google a way to neutralize AWS’ advantage: With containers, you could develop on AWS, for example, and then run the applications anywhere — such as on Google’s cloud. So it not only gave developers a path off AWS, if Google could develop a strong service on Google Cloud Platform, it could make money from the play over time.

Linux needed a path to the cloud

Now back to the diagram, which shows a smiling Alex Polvi from CoreOS. His company was acquired by RedHat in 2018. And he saw the need to bring Linux to the cloud. After all, Linux was powering the internet. It was the OS for enterprise apps and to Polvi, it made sense to extend its path.

Linux became the operating system for the data center. It did so by becoming an open standard with consistent APIs. And although there are still other OSes out there, Linux is the dominant production-grade operating system for data center applications.

At an OpenStack event in 2015, Polvi described how he thought a similar standard would emerge in an abstraction layer that treats the data center as a computer, rather than addressing individual machines. And he stated at the time his belief that that layer would standardize around Kubernetes.

Listen to Alex Polvi describe the parallels between the standardization of Linux and Kubernetes.

Red Hat’s future depended on a new path to the cloud

Red Hat had been around since 1993. So it needed a future that took the company to cloud. OpenShift was that future and it had to leverage a container management framework. According to the documentary, Red Hat connected to Google through their respective board members, which led to a discussion among engineers about containers. Google was noncommittal but did expose that it was thinking about doing something and maybe open-sourcing this new project.

But Google’s internal debates and arm-twisting were taking too long, so Red Hat couldn’t wait. It had to move forward with an open-source alternative, so it decided to do what Apple Inc., Airbnb Inc. and Heroku (now owned by Salesforce.com Inc.) were doing and build on an alternative. It was ready to commit to Mesos, which was more sophisticated and much more mature than Kubernetes. But at the last minute, Google came back and said, “Let’s do it.”

Clayton Coleman was the Red Hat architect who started leaning in right away and was one of the first committers outside Google to participate in the project.

Decision time for Kubernetes: Go for scale or simplicity?

At the time, you had competing forces in the market and internally between “do we go with simplicity or do we go for system scale?” Google clearly had the expertise to build large, complex, scalable systems. Platforms such as Docker Swarm and Mesos were tackling more complex use cases and yet the Kubernetes team bet on simplicity first.

Chen Goldberg from Google talked about why the company focused first on simplicity and getting that right before tackling the larger problems. Perhaps is seems obvious today, but with external market forces and internal engineers advocating for different directions, it was an important decision. Goldberg explains that, rather than competing right away with say Mesos or Docker Swarm, which were far more baked, Google made the bet to keep it simple and go for ubiquitous adoption — which obviously turned out to be the right choice.

Listen to Google’s Chen Goldberg explain the rationale of focusing on simplicity first.

Herding the cats

The last piece of the complex puzzle was governance. According to commentary in the documentary, Google had promised to open-source Kubernetes, but when it started to open up to contributors outside Google, the code was still controlled by Google. Developers had to sign Google papers that basically said Google could still do whatever it wanted.

We’ve seen this before when a large company tries to elbow its way into an open-source community and yet still maintain control. Google had to pass the baton to an independent entity, and that’s how CNCF was started. Kubernetes was its first project.

Listen to Chris Aniszczyk describe the mission of CNCF as a neutral organization and how it supports the adoption of cloud-native technologies.

It was quite an amazing path to get to this point. CNCF now has more than 725 members and has become a premier organization hosting more than 100 projects beyond Kubernetes.

Kubernetes’ dominant position in the market

This ETR data above shows container orchestration offerings – the same XY graph as shown before — and you can see where Kubernetes lands. Notwithstanding that Kubernetes is not a company, respondents know they’re doing Kubernetes. They may not know which platform they’re using, but they know it’s Kubernetes.

The ETR taxonomy has two container categories, container platforms and container orchestration offerings. It’s a challenging topic to survey within the ETR sample because Kubernetes is increasingly becoming embedded into cloud platforms and IT pros may not even know which one they’re specifically using.

In the diagram above, we link Red Hat OpenShift and Kubernetes because we really think OpenShift is a dominant revenue player in the market as an increasingly popular platform-as-a-service layer. Yes, you can download Kubernetes and do what you want with it, but if you’re building enterprise apps, you’re going to need support, and that’s where OpenShift comes in.

Tracking container revenue

There’s not much market data on container software revenue, but we did find this data below from Omdia. It shows revenue share for container software players. It’s not clear how that’s defined, but Red Hat has a nearly 50% revenue share with Mirantis (Docker Swarm), VMware and SUSE (Rancher) in the top four.

We found some other data from Informa, but it’s dated. It also had Red Hat as the largest player by revenue, which we believe is accurate — especially because we know the muscle of IBM is behind OpenShift, which is a linchpin of IBM’s application modernization services strategy. IBM can bundle OpenShift into its application modernization services and that gives Red Hat a captive market. We’ve always said that was a key return-on-investment enabler for the Red Hat acquisition and gives IBM an edge in the battle for hybrid cloud.

Kubernetes under the hood

The CNCF recently released its annual survey. A data table from that publication is shown below:

This past CNCF annual survey had 1,800 respondents, as shown above. Note: The data above only includes managed services only. It indicates the following:

• 79% of respondents use certified Kubernetes hosted platforms;
• Amazon Elastic Container Service for Kubernetes was the most prominent at 39%;
• Azure Kubernetes Service followed at 23%; and
• Azure AKS Engine comes in at 17%, with Google Kubernetes Engine falling behind those.

Also note that OpenShift shows up on the right side of the chart. Remember, this represents just managed services, which is a relatively new offering for OpenShift. The majority of Red Hat’s business is self-service, either in the data center or in the cloud. Moreover, many of the cloud service providers offer their Kubernetes services for no charge. This explains why Red Hat is the revenue leader (shown earlier) but may not show as strong in some of the surveys.

Which makes you think back to Google management’s initial concern: Why open-source such a key technology? The premise was that it would level the playing field in cloud. And for sure it has.

But one has to ask: Has that driven the monetization Google was after? You’d have to say probably not. But where would Google be if it hadn’t open-sourced Kubernetes? How relevant would it be in cloud? And despite its distant third position behind AWS and Microsoft, without Kubernetes Google probably wouldn’t be in the conversation.

What’s ahead for Kubernetes?

Let’s wrap with some comments on the state of Kubernetes and thoughts about where the community is headed.

The state of Kubernetes today

No new insight on this first one, but Kubernetes, for all its improbable beginnings, has gone mainstream. In the past year or so, we’re seeing much more maturity and support for stateful workloads and big ecosystem support in this respect, with better security and continued simplification. But Kubernetes is still pretty complex. It’s getting better, but it’s not VMware-level of maturity, for example. Of course not.

Adoption has always been strong for cloud-native companies that start with containers on day one. But we’re seeing many more information technology organizations adopting Kubernetes as it matures, confirming Kelsey Hightower’s earlier commentary.

It’s interesting that Docker set out to be the operating system of the cloud, and Kubernetes has really become that piece of the puzzle. Docker Desktop is where Docker is thriving. It sold off Swarm to Mirantis and has made some tweaks to its licensing model to be able to continue to evolve its business model.

As we said in our 2020 predictions post, we expected then for Kubernetes to become less visible and more embedded into other platforms, and that’s exactly what is happening. But it still has some complexity challenges to overcome. Remember in the early and mid-cycle of VMware adoption? Understanding application performance required people in lab coats to remediate problems and scale the system.

In some ways you’re seeing that dynamic repeat with Kubernetes. To secure, scale up, get the system to perform and recover quickly when something goes wrong require lots of expertise. This is all made more difficult by the rapid pace at which the ecosystem is evolving Kubernetes. But it’s definitely headed in the right direction.

And it presents opportunities for the ecosystem.

More abstractions, multicluster, automation, new workloads

So what’s ahead for Kubernetes?

We would expect further simplification and more abstractions. As Kubernetes improves support for multicluster, it will begin to treat those clusters as a unified group and enable clusters to be managed together. This will create a lot of ecosystem focus on scaling globally and lowering latency, keeping pace with security and recovery. And that will require new services to share resources across clusters. So look for that in the near- to mid-term.

Expect more automation. This will be driven to a large degree by host cloud providers, but also others in the ecosystem. As Kubernetes supports more stateful applications and begins to extend its cluster management, cloud providers will inject as much automation as possible into the system. And ecosystem startups will bring innovations to manage application resources better.

And finally, as these capabilities mature, we would expect to see better support for data-intensive workloads such as artificial intelligence, machine learning and inference. Scheduling with these workloads becomes harder because they are so resource-intensive. And performance management becomes more complex so that will have to evolve. Frankly many of the things the Kubernetes team back-burnered early on, which were seen for example in Docker Swarm or Mesos, will start to enter the Kubernetes scene.

What’s after Kubernetes?

The last thing we’ll ask you to think about is what’s next beyond Kubernetes? You know this isn’t it, right? With serverless and IoT and the edge and new data-heavy workloads, there’s something that will disrupt Kubernetes. In that CNCF survey, nearly 40% of respondents were using serverless and that’s going to keep growing.

How will that change the development model? Amazon Chief Executive Andy Jassy famously said when he was AWS CEO that if they had to start over with Amazon retail, they’d start with serverless. So let’s keep and eye on the horizon to see what’s coming next.

One last thought, a prediction of sorts: The cloud-native community likes to meet. Many of the innovations we’re seeing today were spawned at small meetups or other significant events. This crowd will be some of the first to return to in-person meetings to keep the ideas flowing.

Don’t miss it!