Gimlet Labs Inc., a startup that helps developers speed up their inference workloads, has raised $300 million in funding at a $3 billion valuation.

Andreessen Horowitz led the Series B round. Gimlet stated in a blog post today that the firm was joined by Arm Holdings Inc., Samsung Ventures, Microsoft Corp.’s M12 fund and more than a dozen others. The company’s total outside funding now stands at $392 million.

A large language model comprises multiple software modules that have vastly different hardware requirements. For example, some modules make heavier use of the underlying graphics cards’ memory than others. Gimlet has developed a platform that harnesses that phenomenon to speed up inference. 

The company’s software can automatically break up an LLM into modules. From there, it deploys each module on the chip architecture that best aligns with its hardware requirements. For example, Gimlet might send a memory-intensive component to an accelerator with a large amount of onboard RAM.

There are multiple ways to split up an LLM. The most common approach, PD disaggregation, runs the prefill and decode phases of the inference workflow on separate chips. The prefill and decode phases are processing steps in which an LLM interprets prompts and generates a response, respectively.

Gimlet also supports more granular disaggregation approaches. 

Some developers don’t stop at splitting the inference workflow into prefill and decode phases. They take it a step further by diving the decode phase into smaller workflows that can each be assigned to a different chip. In other projects, developers use a lightweight “drafter” model to generate initial prompt responses and a frontier LLM to refine those drafts. Each model runs on a separate chip architecture.

Gimlet’s platform reduces the amount of work required to implement such disaggregation workflows. Additionally, it optimizes each LLM module for the chip architecture on which it’s deployed. The software optimizes model code using a combination of AI agents and a custom compiler.

According to Gimlet, its agents find the best way of adapting LLM code to a chip by exploring multiple design approaches. After identifying a promising approach, the AI runs tests to ensure that it’s correct. The company’s compiler, in turn, applies a mix of generic and chip-specific optimizations to customer models.

Gimlet sells its software in a serverless edition and as a managed service that enterprises can deploy on their own infrastructure. The company says it has received billions of dollars’ worth of customer orders. In March, it stated that its customer base includes one of the world’s largest cloud providers and a top three AI lab.

The funding will help the company grow the infrastructure capacity of its platform’s serverless edition. Gimlet plans to add several hundred megawatts of computing power. Additionally, the company intends to expand into the custom hardware market. Gimlet is developing an inference-optimized server that doesn’t feature a motherboard and can be used outside data centers.

Photo: Unsplash