The rise of agentic AI systems is giving Kubernetes, long valued for its reliability and predictability, an unexpected second act.

As AI spending shifts from training models toward serving them and running agents, the open-source community is developing tools to handle the growing volume of inference requests. The orchestration platform was built from the start for dynamic, distributed workloads, which gave fast-growing AI companies a reliable foundation, according to Jonathan Bryce (pictured), executive director of the Cloud Native Computing Foundation.

“The AI boom has sparked a massive investment and a huge build-out of infrastructure overall,” Bryce said. “Having Kubernetes built from the beginning to handle really dynamic distributed workloads as those companies were growing like crazy — they had a really reliable, solid foundation to build on at the same time.”

Bryce spoke with theCUBE’s Rebecca Knight at KubeCon + CloudNativeCon NA, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how CNCF projects can meet rising inference demand, make better use of data center hardware and secure AI agents. (* Disclosure below.)

Inference at scale for agentic AI systems

New models grab headlines with benchmark scores in math and software engineering. But a model has no value until it can be served at high performance, and early chatbots made that look simple because each message triggered a single inference call, Bryce explained. Agents can make repeated inference calls to complete a single task, a shift the CNCF has put on the KubeCon agenda with its new AI inference and agentic track.

“We’re using coding agents to say, ‘Build me a system that does this,’ and it’ll spawn 10 sub-agents,” Bryce said. “Each of those are doing thousands of inference calls over the course of hours and just churning away.”

As inference demand fuels a data center construction boom and raises local concerns about resource use, the llm-d inference engine and Agent Substrate, which offers compact hosting for long-running agents, aim to get more performance out of existing hardware, Bryce said. Security and control pose another challenge because agents can make different choices when given the same task. OpenShell, a security project now under the CNCF’s neutral governance following Nvidia Corp.’s contribution, drew more than 300 contributing companies within months, a milestone which might once have taken two years to reach, he added.

“What I am constantly thinking about is, ‘How do we lower friction to collaboration?'” Bryce said. “How do we enable people to be able to move at the pace that the world and the industry is moving at, and how do we, while doing that, continue to drive that into collaboration around open solutions that can become those base and infrastructure components that we can all build on?”

Stay tuned for the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of KubeCon + CloudNativeCon NA.

(* Disclosure: The Linux Foundation sponsored this segment of theCUBE. Neither The Linux Foundation nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE