Inference pushes AI to smaller sites, making the data layer the deciding factor
Inference is moving to the center of enterprise AI spending, and the AI data layer is emerging as the piece that decides whether those investments pay off. As GPU costs climb and component shortages persist, the architecture behind AI is shifting from a handful of massive facilities toward a distributed web of smaller sites.
That shift is putting storage companies with roots in high-performance computing at the center of the AI conversation. Turning data into insight at an acceptable cost now depends on a resilient, secure data engine built for GPU efficiency, according to Alex Bouzari (pictured), co-founder and chief executive officer of DataDirect Networks Inc.
“As inference is off the charts in terms of utilization, the data layer is what makes a difference,” Bouzari said. “Costs are skyrocketing. GPUs are becoming more and more powerful, but also more and more expensive.”
Bouzari spoke with theCUBE Research’s John Furrier and Rebecca Knight at SC26, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how inference, cost pressure and shortages are reshaping AI infrastructure. (* Disclosure below.)
Why the AI data layer is going distributed
The first wave of AI buildouts concentrated GPUs in large, power-hungry facilities. Inference changes that math because it needs to run close to users and their data, Bouzari explained. Communities are also pushing back on giant monolithic data centers, he added.
“Now, what is happening with inference, you need to supplement that with smaller footprints. Let’s call them data centers, 1 megawatt, 2 megawatt, 5 megawatts. And those need to reside where the action is taking place, where the autonomous driving is happening, where physical AI is happening, where the sensor data is,” Bouzari said.
Connecting small pods back to large AI factories requires a different approach to data. DDN is building a software layer that spans many types of hardware, a direction reflected in its recent move to free its EXAScaler file system from DDN’s own hardware. The goal is to squeeze more use out of very expensive gear, according to Bouzari.
“It is a software layer that needs to accommodate the different types of hardware because, look, the costs being so high, people want to be able to maximize the utilization over an extended period of time, just like it was in high-performance computing,” Bouzari said.
The scale of spending has changed dramatically. A couple hundred million dollars once funded the world’s fastest supercomputer but now buys 15 to 20 racks of equipment, Bouzari noted. Shortages of GPUs, memory and solid-state drives are colliding with tighter budget scrutiny.
“On the one hand, you have budgetary constraints and the need to extract as much value as you can from your investment,” Bouzari said. “On the other hand, you have the shortages. And so a data layer that can do a lot more with a lot less infrastructure, the smaller footprint is extremely valuable.”
That thinking extends into the enterprise. DDN’s Enterprise AI HyperPOD packages CPUs, GPUs, networking and storage into a rack that can drop into existing corporate sites with available power. Keeping the pod on-premises also gives companies tighter control over their data, Bouzari explained.
“I think we’re entering an era where we will see a blossoming of smaller footprint pods being deployed in enterprises globally,” Bouzari said.
Bouzari framed the AI data layer as the counterpart to accelerated compute, with Nvidia Corp. supplying the processing and DDN supplying the data. As inference and agents take over, the data side carries more weight, he said: “You need both to make it successful. But the data engine, I think, is becoming increasingly important as inference and agentic is really becoming the predominant way to utilize AI and get value out of it.”
Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of SC26:
(* Disclosure: NYSE Wired sponsored this segment of theCUBE. Neither NYSE Wired nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)
Photo: SiliconANGLE
A message from John Furrier, co-founder of SiliconANGLE:
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
- 15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more
- 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network
Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/
About SiliconANGLE Media
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.