Memory has become the scarcest resource in the AI data center, and hyperscalers are rethinking how they hand it out. Composable memory, which pools capacity beyond individual servers and assigns it to workloads on demand, is emerging as one of the most practical answers to that squeeze.

The pressure is visible across AI factories where graphics processing units sit idle waiting on memory and storage. Rather than leaving memory locked inside each server, Microsoft Corp.’s pathfinding team is scouting system-level technologies that can turn it into a shared, dynamically assigned resource across Azure, according to Samir Rajadnya (pictured), principal architect at Microsoft.

“One of the main things that is happening or will happen in the next several years is composable architecture,” Rajadnya said. “The original idea was everything is locked behind a CPU or a GPU. Memory was a servant to the CPU or GPU. Now memory is looked at as another very important resource.”

Rajadnya spoke with theCUBE Research’s John Furrier at the AI Data Pipeline Forum, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how composable memory, faster interconnects and workload-specific design help hyperscalers and enterprises get more from scarce memory. (* Disclosure below.)

Composable memory meets the memory wall

Supply has tightened as AI demand outpaces production, a shift HPCwire reports is rewriting the rules of AI inference. For a cloud running at Azure’s scale, idle capacity is costly. Azure now functions as a utility for companies such as Delta Air Lines Inc. and Walmart Inc., so any new approach also has to stay reliable and secure, according to Rajadnya.

“As you see in the media right now, there is a shortage of memory,” Rajadnya said. “So we have to use that memory more efficiently. And as a part of that, we have to make sure memory goes where the workload demands it. If we strand memory behind a workload who is not using it, we are simply wasting a valuable resource at this time.”

Inference has become one of the largest workloads in the industry, and it runs into two ceilings at once. Asked what he would build with no constraints, Rajadnya described memory that could be attached across long distances without a power or latency penalty.

“AI inference is limited by power and then it is limited by memory, correct?” Rajadnya said. “So if I had a wish list, I would go for a memory that I can connect with the lowest possible power over a large distance and an infinite amount of memory.”

Enterprises face a narrower version of the problem. A cloud provider has to serve every kind of workload, while most companies run a handful of critical applications with known characteristics, which makes their design problem more manageable, Rajadnya noted. His advice is to identify the top two or three workloads and build systems tailored to them.

“It’s all the game of what we call vertical integration right now. Workload, the system, software, silicon, all in one,” Rajadnya said. “And you have to do that even at the enterprise level.”

Making any of this work at scale depends on the links between memory and processors, including the specialized accelerators Rajadnya groups under the label XPU. Once memory leaves the server chassis, the distance it travels decides whether copper is enough or optical connections are required. Microsoft has long contributed designs to the Open Compute Project and is showcasing its AI infrastructure work at this year’s Open Compute Project Global Summit. Innovation in a single component means little unless the whole ecosystem moves with it, according to Rajadnya.

“There is a lot of great innovation happening on the memory component side of things. There is a lot of great innovation happening on the XPU CPU side of things, but how do you connect those two things together with the memory, right?” Rajadnya said. “So that is where the interconnects are very important. Some of the things that are happening in the optical space, they are going to be important. And these things are going to help us to build that composable memory.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of the AI Data Pipeline Forum:

(* Disclosure: ScaleFlux Inc. sponsored this segment of theCUBE. Neither ScaleFlux Inc. nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE