AI’s next build-out: Inference racks that resize themselves for every new model
Memory pooling is fast becoming one of the sharpest tools in AI inference economics, with companies racing to wring more output from every graphics processing unit.
Most AI servers still ship with a fixed GPU-to-memory ratio that holds for the life of the machine. But that is a poor fit when nobody knows what model they will deploy six months from now, according to Sumit Puri (pictured), co-founder and chief technical officer of Liqid Inc.
“Our core fundamental technology is around pooling and sharing the most critical resources in the data center, which are now GPU and memory,” Puri said. “Rather than statically deploying these things inside of a server, our vision is simple. Let’s put pools of resources in the rack, GPUs and memory, and dynamically assign them to any server that needs it based upon the model or the workload that we are looking to deploy.”
Puri spoke with theCUBE Research’s John Furrier and Gemma Allen at SC26, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed pooling GPUs and memory, the tokenomics of AI inference and Liqid as a Service. (* Disclosure below.)
Memory pooling meets the tokenomics test
Liqid measures payoff in tokenomics, with the he yardsticks being tokens per second, tokens per watt and, above all, tokens per dollar. Supply is another pressure point in the AI infrastructure build-out, with memory among its most constrained components. But delivering only the memory each application needs stretches scarce supply further, Puri explained.
“Supply chain is a massive constraint for all of us — memory being a giantly massive constrained thing,” he said. “If we can do things like pool memory, share memory, optimally deliver memory to the server in the right amount of memory as required by the application, we can directly address some of those supply chain issues.”
Liqid is also widening how customers buy memory pooling. At SC26, the company announced Liqid as a Service, a fully managed, single-tenant offering, and it is moving toward preconfigured appliances. The bet is that most enterprises will deploy models rather than build them and therefore require their own capacity, according to Puri.
“The world went to unlimited consumption of minutes, and that’s where we believe tokens are headed,” Puri said. “Everybody’s going to own their own token machine so they can consume these tokens at will.”
Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of SC26:
(* Disclosure: Liqid sponsored this segment of theCUBE. Neither Liqid nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)
Photo: SiliconANGLE
A message from John Furrier, co-founder of SiliconANGLE:
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
- 15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more
- 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network
Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/
About SiliconANGLE Media
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.