Cloud providers are scaling Kubernetes inference to serve frontier AI models, and the problems they encounter in production are helping shape the open-source projects they rely on.

CoreWeave Inc. sits near the center of that shift as AI agents reshape enterprise infrastructure. The company describes itself as a cloud built from the ground up for AI rather than retrofitted for it. Much of its growth traces back to the open nature of the container orchestration platform and the ecosystem around it, according to Yoshi Tamura (pictured, left), principal product manager at CoreWeave. CoreWeave has now upgraded to Platinum, the Cloud Native Computing Foundation’s highest corporate membership tier, to help shape how Kubernetes handles the large AI workloads its customers run.

“Without [Kubernetes], I don’t think we would’ve been able to get this kind of explosive growth,” Tamura said. “With this upgrade to Platinum, we indicate that we really mean it. We want to give back to the community. We want to be more involved.”

Tamura and Rita Zhang (right), principal software engineer at CoreWeave, spoke with theCUBE’s John Furrier and Rebecca Knight at KubeCon + CloudNativeCon NA, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed why CoreWeave upgraded to CNCF Platinum membership and what it takes to run Kubernetes inference in production. (* Disclosure below.)

Kubernetes inference and the hard work of running AI in production

Open source can get a model into a cluster, but that doesn’t guarantee it will perform at scale. CoreWeave’s inference team brings together infrastructure specialists and machine learning engineers, working with model labs and open-source inference engines to patch engines and runtimes for newly released models, according to Zhang.

“Running in production is hard,” Zhang said. “We work a lot with the open-source community, not just on the inference stack, but also with the model providers. Just because it’s on Hugging Face and you download the weights doesn’t mean it will work per your requirement. This is where we work very closely with the labs and the open-source community — like vLLM, SGLang — to make sure that for day-zero models we actually are able to patch the engines and provide our own kernels and patch the runtime.”

Kubernetes special interest groups rely on users to bring them production problems, such as finding nodes with the right devices and accounting for cluster topology when scheduling large training and inference jobs, Zhang explained. CoreWeave remains smaller than incumbent cloud providers and relies on open source more than many realize, Tamura added. Its work with leading AI model builders now gives it an early look at where frontier workloads are headed.

“We’re in a very unique position,” Tamura said. “Maybe we can actually find a way to kind of bring it back to the community and sort of lead which direction Kubernetes, a larger community, could go.”

Stay tuned for the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of KubeCon + CloudNativeCon NA.

(* Disclosure: CoreWeave sponsored this segment of theCUBE. Neither CoreWeave nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE