Fractile Ltd., a British startup that develops artificial intelligence inference chips, has raised $720 million in funding.

Reuters reported today that the Series C round was led by Thrive Capital, Founders Fund, Lightspeed and Redpoint. They were joined by about a dozen others including the U.K. Sovereign AI Fund, the British Business Bank and Alphabet Inc.’s CapitalG investment arm. Fractile is now worth $7 billion.

Fractile was founded in 2022 by AI researcher Walter Goodwin (pictured), who serves as its chief executive. The company reportedly expects its chips to provide up to 100 times better performance than current silicon. Furthermore, Fractile is promising a 90% reduction in energy use. The chips deliver those improvements by removing the need for AI models to move data to off-chip HBM memory.

Large language models generate prompt responses through a workflow that involves numerous steps. When an LLM completes a processing step, it sends the results to an off-chip HBM memory chip. It then retrieves those results to lay the groundwork for the next step. Those repeated data round-trips take a significant amount of time, which slows down inference.

Fractile is not the only AI chip supplier tackling that bottleneck. Cerebras Systems Inc., which went public in May, offers wafer-size accelerators that reduce data movement by keeping information in a large onboard cache. That cache is faster than off-chip HBM modules. The company’s newest processor, the WSE-3T, can hold 44 gigabytes of data. 

Cerebras’ on-chip cache is based on SRAM, a type of memory that was historically added to processors in only small quantities. SRAM cells, the technology’s basic building blocks, have six transistors apiece. Those transistors read and write data faster than the single transistor found in HBM cells. At the same time, SRAM’s speed comes with certain trade-offs: it’s expensive and has limited capacity. 

Fractile is taking a different approach to reducing data movement. Instead of integrating SRAM into its inference chips, the company is stacking standard DRAM memory on top of them. Placing a memory module immediately above a processor’s logic circuits minimizes the distance that data has to cover. That speeds up traffic while lowering the amount of electricity needed to shuffle it back and forth.

Samsung Electronics Co. is developing a similar technology. In August, the company debuted an HBM variant called zHBM that can be stacked atop AI accelerators. It uses tiny copper wires to move data to and from the host chip. Samsung says the technology will be eight times faster than HBM5, a future iteration of HBM more performant than the versions in use today.

Fractile, for its part, estimates that its chips will enable LLM to process several thousand tokens per second. It plans to ship the processors as part of rack-scale systems that will also include custom software. Many AI chipmakers offer custom compilers, software tools that automatically turn customers’ LLMs into a format compatible with their silicon.

Fractile expects to start shipping its chips next year. 

Photo: Lightspeed