Kubernetes has become the go-to platform for distributed training and inference, but what happens to the compute bill when it runs AI agents that spend most of their time waiting?

When an agent’s container or virtual machine stays provisioned, customers pay for the wait as well as the work. To help customers control those costs and make agent workloads more predictable, Google LLC has donated three projects to the Cloud Native Computing Foundation, according to Allan Naim (pictured, left), director of product management, AI infrastructure, at Google Cloud. For Kubernetes itself, the new challenge is freeing up compute between bursts of activity while letting each agent pick up where it left off.

“Act 2 is a few years back; Kubernetes became the de facto for inference and training,” Naim said. “What’s really interesting right now is we’re embarking on this Act 3 … where the typical patterns are very, very different from how Kubernetes was designed.”

Naim and Gari Singh (right), product manager at Google Cloud, spoke with theCUBE’s Rebecca Knight at KubeCon + CloudNativeCon NA, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how Google Cloud is using open-source tools to cut compute costs when AI agents sit idle. (* Disclosure below.)

Open building blocks for distributed training and inference meet agentic AI

Cost presents a serious problem in agentic workloads, with agents sitting idle 90% to 95% of the time waiting on models, test runs or human approvals, Naim explained. Google developed Agent Substrate, an open-source runtime that hosts AI agents in isolated environments on Kubernetes. When an agent is suspended, the runtime saves a snapshot of its memory and local files and frees up its compute for other agents. When work resumes, it restores the agent from that snapshot in an available sandbox.

“It helps from an economic standpoint,” Naim said. “We’ve seen 10x reduction in cost in terms of running Agent Substrate on Kubernetes versus running these agents directly on dedicated virtual machines and containers.”

Security is another concern. In response, Google open-sourced gVisor in 2018, which sits on top of the host kernel to block or audit every system call an agent makes, according to Singh.

“Agents and agent harnesses are designed to solve a problem — and to try to solve a problem — until they can’t solve the problem,” Singh said. “They will find all kinds of clever workarounds. They know everything that’s around. You want to make sure that it didn’t go make some low-level change to [your] low-level security, didn’t make an outbound network call, didn’t try to open a file that it shouldn’t.”

Google is also working with Anyscale Inc. on KubeRay, which runs Anyscale’s Ray platform natively on Kubernetes so platform teams can offer distributed training and inference to many research groups, Singh noted. OpenRL brings that approach to reinforcement learning, which teaches post-trained models guardrails, Naim added.

“In the past, to do reinforcement learning, you needed to hire a dozen research scientists with PhDs,” Naim said. “Our goal was, ‘How can we bring reinforcement learning into the world of platform engineering so that you can run reinforcement learning on your Kubernetes environment and take advantage of it to actually do continuous improvement for your agentic workload?'”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of KubeCon + CloudNativeCon NA:

(* Disclosure: Google Cloud sponsored this segment of theCUBE. Neither Google Cloud nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE