A group of artificial intelligence researchers today launched Inferact Inc., a new startup that will commercialize the open-source vLLM project.

The company is backed by $150 million in seed funding. Andreessen Horowitz and Lightspeed led the round with participation from Databricks Inc.’s venture capital arm, UC Berkeley Chancellor’s Fund and several other backers. Their investment values Inferact at $800 million.

Inferact’s founding team includes computer science professor and Databricks co-founder Ion Stoica. He is currently the director of the University of California at Berkeley’s Sky Computing Lab, which developed the original version of vLLM in 2023. The project’s pool of code contributors has since grown to more than 2,000 developers.

Software teams use vLLM to speed up inference workloads. The tool boosts performance by applying a long list of optimizations to large language models. Many of those optimizations, including a particularly important vLLM feature called PagedAttention, focus on reducing models’ memory use. 

When an LLM receives a prompt, it completes a small portion of the calculations needed to produce an answer and saves the results to a so-called KV cache. It then performs another portion of the calculations, updates the KV cache with the new results and repeats the process until a prompt response is generated. Storing all those results requires a significant amount of memory.

PagedAttention makes it possible to store KV cache data in non-adjacent sections of a server’s RAM. That feature and certain other capabilities significantly reduce memory waste, which lowers LLMs’ hardware consumption. For added measure, vLLM uses a method called quantization to compress AI models’ weights and thereby shrink their memory footprint.

Besides optimizing RAM use, vLLM can also boost inference speeds. LLMs usually generate prompt responses one token at a time. With LLM, developers can configure their models to generate multiple tokens at once to reduce loading times for users.

“We see a future where serving AI becomes effortless,” Inferact co-founder Woosuk Kwon wrote in a blog post. “Today, deploying a frontier model at scale requires a dedicated infrastructure team. Tomorrow, it should be as simple as spinning up a serverless database. The complexity doesn’t disappear; it gets absorbed into the infrastructure we’re building.”

The blog post hints that Inferact plans to launch a paid serverless version of vLLM. Many startups focused on commercializing open-source projects take that route. Usually, managed versions of open-source technologies automate administrative tasks such as provisioning infrastructure and downloading updates.

An Inferact job posting indicates that it plans to equip its software with observability, troubleshooting and disaster recovery features. The listing suggests that the software will run on Kubernetes.

Kwon wrote in today’s blog post that the Inferact team, which includes several core vLLM maintainers, will also enhance the upstream open-source version. The company plans to release new performance optimizations and support for emerging AI model architectures. Additionally, Inferact will enable vLLM to run on more types of data center hardware.

Image: Unsplash