Startup Lightning AI Inc. today debuted Thunder, an open-source compiler that rewrites artificial intelligence models to make them easier to train.

The company claims its software can reduce the time it takes to develop AI models by several weeks. That cuts the amount of compute resources needed for machine learning projects, which in turn lowers the associated hardware expenses. Those savings can be significant given the most advanced large language models reportedly cost more than $100 million to train. 

“What we are seeing is that customers aren’t using available GPUs to their full capacity, and are instead throwing more GPUs at the problem,” said Lightning AI Chief Technology Officer Luca Antiga.

New York-based Lightning AI is backed by more than $50 million in funding from Index Ventures and other investors. It provides cloud-based workspaces that software teams can use to build machine learning software. Lightning AI also sells products that make it easier to use PyTorch Lightning, a popular AI development framework created by Chief Executive Officer William Falcon.

The company’s newly debuted Thunder tool is a so-called source-to-source compiler. It can take the code of a neural network and rewrite it into a more hardware-efficient form. According to Lightning AI, the optimizations made by Thunder reduce the amount of infrastructure necessary to train AI models.

Under the hood, the compiler rewrites code using a collection of open-source tools developed by Nvidia Corp. and other companies. All those tools focus on increasing AI models’ efficiency, but they do so in different ways. Thunder can analyze an AI model, identify which open-source optimization tool could enhance it the most and then apply the available improvements.

One of the technologies Thunder uses to reduce neural networks’ hardware usage is a library called cuDNN. Developed by Nvidia, it provides a set of prepackaged software building blocks that developers can use to implement AI models. Those building blocks take the form of kernels, programs optimized to run on graphics cards.

Thunder also makes use of another Nvidia-developed tool called NVFuse. A typical AI model comprises a large number of kernels like those included in the cuDNN library. NVFuse can consolidate multiple kernels into a single program that requires less processing power to run, which reduces AI models’ hardware requirements.

Thunder incorporates a number of other open-source technologies as well, including OpenAI’s Triton toolkit. The toolkit includes, among other components, a compiler designed to speed up AI models. It eases tasks such as parallelizing the computations an AI performs across a graphics card’s cores, which makes it possible to carry out a large number of calculations at once. Performing calculations in parallel is faster than running them one after another.

In an internal evaluation of Thunder’s capabilities, Lightning AI used the compiler to optimize the popular Llama 2 large language model. The company tested the most lightweight version of the LLM, which includes 7 billion parameters. Lightning AI claims training throughput increased by 40%.

The Thunder team is led by Dr. Thomas Viehmann, a deep learning pioneer who contributed early on to PyTorch.

The company has made Thunder available on GitHub under an open-source license. It also offers the compiler as part of Lightning Studios, a commercial cloud platform it debuted in December. The platform provides access to managed development environments that software teams can use to build AI models.

Image: Lightning AI