Nvidia Corp. has long been the poster child of the AI boom, but the secret sauce behind modern AI factories isn’t just graphical processing units — it’s the network interconnects that tie thousands of accelerators into a single supercomputer.

Nvidia’s Gilad Shainer (right) discusses AI infrastructure with theCUBE’s John Furrier.

Today’s AI workloads are inherently distributed: Models no longer run on single CPUs or GPUs but across thousands or more accelerators. That shift changes the unit of compute from a single server to the entire data center, which must behave like a single synchronized engine, according to Gilad Shainer (pictured), senior vice president of marketing at Nvidia.

“We need to be very synchronized in the way that we work,” he said. “The same goes here. That connectivity cannot have jitter, which means one GPU cannot get data before another GPU gets the data, because that will delay everything. And this is what networking does. Networking connects compute engines, connects compute ASICs to form a single supercomputer that needs to scale across hundreds of thousands of GPU units.”

Shainer spoke with theCUBE’s John Furrier for theCUBE + NYSE Wired: AI Factories – Data Centers of the Future interview series, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed why networking is the operating system of AI, how co-design and density unlock performance and why innovations like co-packaged optics are as important as the silicon itself.

Network interconnects make distributed computing possible

The shift from GPU reliance to distributed computing changes the unit of compute from a single server to the entire data center, which must behave like a single synchronized engine. Network interconnects are the wiring harness that lets GPU application-specific integrated circuits act together — moving massive volumes of data with extremely low latency and near-zero jitter, so every accelerator receives the same information at the same time, according to Shainer.

“That workload needs to run across multiple GPUs and multiple servers, and we’re not talking about two or three,” he said. “We’re talking about thousands and tens of thousands and hundreds of thousands. Now, the compute engine is not just a GPU ASIC; it is the entire data center. Therefore, you need to have a way to take those GPU ASICs and connect them so they can exchange information between themselves and to form a single unit of computing out of those ASICs.”

Hardware alone can’t deliver results. The real gains come from co-design; software, libraries, telemetry and management must be designed with the hardware and networking in mind. Only a co-designed stack — from model frameworks down to the physical links — yields the tokens-per-second, runtime efficiency and predictable performance AI developers demand, Shainer added.

“At the end of the day, of course, you measure that in the outcome,” he said. “You measure that in tokens per second. You manage that in the productivity of the workers who are running on top of it. It’s not a separate element here; it’s one co-design that actually enables those AI supercomputers.”

Where past data-center design feared density, Nvidia embraces it. Packing more GPU ASICs into a rack reduces reliance on power-hungry optical links and enables more efficient copper-based connections for scale-up workloads, according to Shainer.

“To increase the efficiency there, if for scale-up, we focus on density and using copper,” he said. “On a scale-out [infrastructure], we brought co-packaged optics and we built both Quantum-X InfiniBand with co-packaged optics or Spectrum-X Ethernet Photonics with co-packaged optics because that reduces the amount of energy you need to invest in moving data between those racks and building the most efficient infrastructure for scale-out.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of theCUBE + NYSE Wired: AI Factories – Data Centers of the Future interview series:

Photo: SiliconANGLE