Open-source software giant Red Hat Inc. announced today that it has agreed to acquire Neural Magic Inc., a machine learning startup that optimizes AI models to run whatever hardware is available, including central processing units or graphics processing units.

As AI large language models grow in size and complexity, they also consume more compute power and energy. As a result, businesses have seen the need to build more cost-efficient LLMs designed for efficiency as well as accuracy that take advantage of the hardware they’re deployed on.

Founded in 2018 by Massachusetts Institute of Technology Research Scientist Alex Matveev and Professor Nir Shavit, Neural Magic’s technology allows for the optimization of LLMs to run on off-the-shelf CPUs and GPUs at similar speeds to specialized AI chips. That means a company can fine-tune and optimize AI models to maximize for hardware efficiency across numerous hardware architectures.

“AI workloads need to run wherever customer data lives across the hybrid cloud,” said Matt Hicks, president and chief executive of Red Hat. “This makes flexible, standardized and open platforms and tools a necessity, as they enable organizations to select the environments, resources and architectures that best align with their unique operational and data needs.”

Neural Magic has contributed significantly to an open-source project for model deployment called vLLM, which Red Hat said brought the company to its attention. The project provides a fast, efficient library for serving and deploying AI models optimized for the cloud that supports numerous processor and GPU architectures. The company has also designed a specialized LLM compressor that provides faster inference with vLLM allowing faster delivery of AI models.

The startup also keeps a repository of prebuilt, pre-optimized open-source LLMs ready to deploy with vLLM so businesses can quickly deploy high-performance models at scale.

Red Hat said Neural Magic’s leadership in vLLM will help build on the company’s ability to support LLM deployments for the hybrid cloud with any hardware back end.

This acquisition represents an important move for Red Hat’s open-source AI strategy. The company recently launched Red Hat Enterprise Linux AI, which brings together the open-source Granite family of LLMs developed by IBM Corp. with Red Hat’s InstructLab open-source community-driven model alignment tools, packaged as optimized, bootable server deployments.

“Open source has proven time and again to drive innovation through the power of community collaboration,” said Neural Magic CEO Brian Stevens. “At Neural Magic, we’ve assembled some of the industry’s top talent in AI performance engineering with a singular mission of building open, cross-platform, ultra-efficient LLM serving capabilities.”

Image: SiliconANGLE/Microsoft Designer