Since KubeCon + CloudNativeCon in Paris, advancements in artificial intelligence have moved rapidly. This includes developments tied to Kubernetes AI solutions, plus advancements on bare metal and Linux in big hyperscale-type GPU deployments.

With that in mind, the enterprise has been looking toward Kubernetes in a big way. As that happens, Red Hat Inc. has been looking to respond with Kubernetes AI solutions, according to Sally O’Malley (pictured, right), principal software engineer at Red Hat Inc.

Red Hat’s Sally O’Malley discuses Kubernetes AI solutions with theCUBE.

“What’s happening right now is data scientists have been doing their thing for a decade longer and us software engineers have been doing our thing,” she said. “Data scientists don’t really know a lot about containers and how to deploy things, and I don’t know a lot about data science.”

This past year, the two have come together and are creating solutions now that LLMs have entered the picture, according to O’Malley. That involves some important questions.

“How do we deploy them? How do we scale them? How do we make them accessible to everybody?” she asked. “That’s what we’ve been working on at Red Hat.”

O’Malley and Jeremy Eder (right), distinguished engineer and chief AI/ML platform strategist at Red Hat, spoke with theCUBE Research’s Rob Strechay and Savannah Peterson at KubeCon + CloudNativeCon NA, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed Red Hat’s advancements in Kubernetes and AI collaboration and the impact of open-source on enabling efficient, accessible Kubernetes AI solutions. (* Disclosure below.)

Kubernetes AI solutions come into focus

Since the last KubeCon, it became obvious that the killer workload for the next couple of years was going to be some flavor of inference that requires very particular management of hardware that is new to Kubernetes, according to Eder. With that in mind, Red Hat worked to spin up a couple of working groups.

“Go back eight or nine years, we had similar things in Kubernetes. We did the Resource Management Working Group, which was folded into SIG Node eventually,” Eder said. “Then we also did something called a Network Plumbing Working Group. The idea behind that one was, we understand some really heavy hitter workloads need specialized networks. So, that plumbing is in place and has gotten downstream into product, and that was workload-agnostic.”

Now that there is a particular workload that has sucked all of the oxygen out of the room, it’s possible to go back to some of those principles, according to Eder. Out of that emerged Working Group Serving and Working Group Device Management.

“Each one of those has a different focus. Device Management being, how do we drive efficiency at a density for the utilization of these really exotic hardware, let’s call it?” Eder said. “Maybe, just plainly speaking, it’s expensive.”

One of the projects Red Hat has been collaborating with IBM Corp. is called InstructLab. The project enables a user to bring their data to easily fine-tune and align a smaller model to make it useful for a narrowly focused, domain-specific use case, according to O’Malley.

“I really think that’s where we’re going with LLMs. You were talking about the cost of running them,” O’Malley said. “The smaller models, if we can show that they can be super effective, that’s going to bring the cost down and make it more accessible to everyone.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE Research’s coverage of KubeCon + CloudNativeCon NA:

(* Disclosure: TheCUBE is a paid media partner for the KubeCon + CloudNativeCon NA. Neither Red Hat Inc., the headline sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE