Nvidia Corp. today previewed an upcoming chip, the Rubin CPX, that will power artificial intelligence appliances with 8 exaflops of performance.

AI inference involves two main steps. First, an AI model analyzes the information on which it will draw to answer the user’s prompt. Once the analysis is complete, the algorithm generates its prompt response one token at a time. Today, the two tasks are usually done using the same hardware. 

Nvidia plans to take a different approach with its future AI systems. Instead of performing both steps of the inference workflow using the same graphics card, it plans to assign each step to a different chip. The company calls this approach disaggregated inference.

Nvidia’s upcoming Rubin CPX chip is optimized for the initial, so-called context phase of the two-step inference workflow. The company will use it to power a rack-scale system called the Vera Rubin NVL144 CPX (pictured.) Each appliance will combine 144 Rubin CPX chips with 144 Rubin GPUs, upcoming processors optimized for both phases of the inference workflow. The accelerators will be supported by 36 central processing units.

The company says the upcoming system will provide 8 exaflops of computing capacity. One exaflop corresponds to a quintillion computing operations per second. That’s more than seven times the performance of the top-end GB300 NVL72 appliances currently sold by Nvidia.

Under the hood, the Rubin CPX is based on a monolithic die design with 128 gigabytes of integrated GDDR7 memory. Nvidia also included components optimized to run the attention mechanism of large language models.

An LLM’s attention mechanism enables it to identify and prioritize the most important parts of the text snippet it’s processing. According to Nvidia, the Rubin CPX can perform the task three times faster than its current-generation silicon. “We’ve tripled down on the attention processing,” said Ian Buck, Nvidia’s vice president of hyperscale and high-performance computing.

The executive detailed that video processing workloads will receive a speed boost as well. The Rubin CPX includes hardware-level support for video encoding and decoding. That’s the process of compressing a clip before it’s transmitted over the network to save bandwidth and then restoring the original file.

According to Nvidia, the Rubin CPX will enable AI models to process prompts with one million tokens’ worth of data. That corresponds to tens of thousands of lines of code or one hour of video. In many cases, increasing the amount of data an AI model can consider while generating a prompt response boosts its output quality. 

Nvidia plans to start shipping the Rubin CPX at the end of 2026. 

Image: Nvidia