RadixArk Inc., a startup that provides tools for artificial intelligence developers, has raised $100 million from a group of high-profile backers.

Accel led the seed round with Spark Capital. They were joined by Nvidia Corp.’s NVentures fund, Advanced Micro Devices Inc., Databricks Inc., Broadcom Inc. Chief Executive Hock Tan and others. The Wall Street Journal reported today that RadixArk is now worth $400 million.

The company is building commercial AI development tools based on two open-source projects called SGLang and Miles. Members of the RadixArk team helped develop the former technology prior to the company’s launch. Miles was open-sourced by RadixArk last November.

Software teams can use Miles to streamline reinforcement learning projects. Reinforcement learning is an AI training method that is commonly used to develop large language models. Miles can compress LLMs with one trillion parameters into a format that fits in the memory of a single high-end graphics card, which significantly reduces training costs.

The software also streamlines AI projects in other ways. It includes a so-called asynchronous co-evolutionary framework, MrlX, that can train multiple AI agents at the same time by placing them in the same simulated environment. That arrangement enables the agents to hone their reasoning capabilities by learning from one another.

After developers train an AI with Miles, they can use SGLang to perform inference. The latter project provides building blocks for building inference environments. According to RadixArk, SGLang powers AI clusters that contain more than 400,000 graphics cards in aggregate.

An LLM’s attention mechanism, a module that it uses to interpret user instructions, generates a significant amount of temporary data while processing prompts. That information is stored in a data structure called a KV cache. LLMs usually clear the KV cache after every prompt.

SGLang enables LLMs to reuse some KV cache data across prompts. That avoids the need to generate all the data from scratch after every request, which lowers the associated infrastructure overhead. Furthermore, SGLang speeds up prompt response times in the process. 

The tool’s KV cache reuse feature is complemented by several other performance optimizations. SGLang uses a method called speculative decoding to offload some tasks from an application’s LLM to a lighter, less hardware-intensive model. Additionally, the tool can spread the calculations involved in processing a prompt across chips with different architectures. That approach can significantly boost model performance in some cases. 

The commercial products that RadixArk plans to build using Miles and SGLang will include ”managed infrastructure and tooling.” According to the Journal, the offerings will enable customers to host AI models in the cloud. RadixArk also plans to enhance the open-source versions of SGLang and Miles.

Image: Unsplash