The shift from model training to agentic inference is forcing a fundamental rethink of how artificial intelligence infrastructure is built and which components carry the most strategic weight. What was once treated as commodity plumbing is now being recognized as the intelligence layer where raw data becomes actionable intelligence.

The rise of sovereign AI deployments and enterprise AI clusters is compounding that pressure, as organizations from hyperscalers to regional governments race to build out high-performance compute stacks that can sustain continuous inference at scale. Storage has quietly moved from afterthought to forethought in those conversations, according to Greg Matson (pictured), senior vice president and head of marketing and products at Solidigm, a trademark of SK Hynix NAND Product Solutions Corp.

“It started a couple of years ago with training, where the need for high-capacity, high-performance storage very adjacent to the GPUs was all of a sudden center stage,” Matson said. “But now at the inference, as we go from last year into this year inference phase into agentic inference, it’s exploding even more. … Storage is actually a whole new storage tier that’s being created to extend the memory for the system. … I almost call it the intelligence layer is now being housed on storage.”

Matson spoke with theCUBE’s John Furrier at RAISE Summit, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how high-capacity SSDs are becoming essential intelligence layer infrastructure for AI inference, token economics and sovereign compute deployments. (* Disclosure below.)

Intelligence layer economics reshape the AI infrastructure stack

As inference workloads scale from single prompts to sprawling agentic sessions, the gap between GPU memory and the data feeding it has become a critical bottleneck. Hyperscalers recognized the problem roughly 12 to 18 months ago and began replacing legacy storage infrastructure — some of it more than a decade old — with high-capacity solid-state drives to keep GPUs fully utilized, Matson noted.

“While the GPU is the most expensive part of your infrastructure, you want that thing to be humming 100% of the time generating tokens,” he said. “And if it’s down, right, waiting for data, then you’re wasting your money on GPUs.”

The economics of token generation make that case concrete. A simple 15-word prompt can generate as many as 40,000 tokens — representing five to 10 gigabytes of context data — and multiplying that across enterprise workforces quickly pushes storage requirements into the petabytes, Matson explained. Solidigm has responded with drives that now reach 122 terabytes of capacity per unit, alongside the industry’s first cold-plate-cooled enterprise SSD designed for fully fanless Nvidia GPU servers — a product built as AI racks shed fans entirely in favor of liquid cooling.

“Everything that’s coming out over the next couple of years, nothing will be air-cooled,” Matson said. “The infrastructure is the data layer. The data layer is the storage layer. That’s the Solidigm layer.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of RAISE Summit:

(* Disclosure: TheCUBE is a paid media partner for the RAISE Summit event. Neither Solidigm, the headline sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE