As AI inference demand grows, storage is becoming the pressure point that determines how far GPUs can truly scale.

The pressure is especially acute for enterprises deploying agentic workflows, where massive fleets of agents send constant inference requests to GPU servers. Offloading previously computed attention data from high-bandwidth memory to intelligent storage tiers can significantly improve throughput without expanding GPU footprints, according to Andy Pernsteiner (pictured), field chief technology officer of data infrastructure company Vast Data Inc.

“We’re working with [Nvidia Corp.] on their Dynamo inference engine, specifically around tuning of [key-value] cache as it relates to offloading attention data from the GPU to compute off into a cache that can grow exponentially and allow for inference, not just for users who are using chatbots, but also for agents who are constantly throwing inference requests at GPU servers,” Pernsteiner told theCUBE. “We see a 10X improvement in inference capability out of a single GPU server.”

Pernsteiner spoke with theCUBE’s John Furrier at the Nvidia GTC AI Conference & Expo, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how storage-layer AI inference optimization is reshaping GPU economics and enterprise AI deployment strategies. (* Disclosure below.)

AI inference optimization through intelligent storage tiering

Nvidia poised Dynamo as an open-source, low-latency, modular inference framework for serving generative AI models in distributed environments. Vast Data’s work with Nvidia on KV cache offloading — shifting stored model context out of GPU memory to free up compute capacity — fits directly into that architecture. By moving previously computed session data to storage, GPUs can handle more active sessions instead of repeatedly recalculating the same context, Pernsteiner explained.

“If a GPU isn’t busy having to recalculate previously computed session data, then it can easily service another request and asynchronously fetch that session data,” he said. “That’s the work that we’ve been working with Nvidia on.”

Enterprise organizations face another obstacle in scaling AI inference into production: security. Many remain stuck in pilot phases because roll-your-own retrieval-augmented generation implementations often lack the policy enforcement needed to protect regulated data across business units, according to Pernsteiner. Vast Data has responded by integrating a policy model with Nvidia’s pipeline deployment framework to provide end-to-end security.

“We’ve talked to many customers this week where they’ve been in the pilot phase and what’s stopping them is this security and the ability to scale,” Pernsteiner said. “So, we’re giving them a path to do that.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of Nvidia GTC AI Conference & Expo:

(* Disclosure: Vast Data sponsored this segment of theCUBE. Neither Vast nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE