Goodfire AI, a public benefit corporation and research lab that’s trying to demystify the world of generative artificial intelligence, said today it has closed on $7 million in seed funding to help it develop tools for debugging AI systems.

Today’s round was led by the high-profile venture capital firm Lightspeed Venture Partners and saw the participation of several other prominent backers, including Menlo Ventures, South Park Commons, Work-Bench, Juniper Ventures, Mythos Ventures, Bluebirds Capital and various angel investors.

The startup is trying to unravel the increasing complexity of today’s most powerful generative AI models, which are often described as a kind of “black box,” with mysterious inner workings. This leads to problems such as understanding why an LLM generates certain kinds of responses to prompts. But this understanding is required by many enterprises, which are worried about deploying AI systems that might go astray.

Goodfire believes there’s strong demand for more explainable AI. It cites a McKinsey Co. survey published earlier this year that revealed how 44% of business leaders have experienced negative consequences due to unintended model behavior.

The startup’s approach to AI explainability is known in the industry as “mechanistic interpretability,” which refers to the study of how AI models reason and make decisions. The goal is to understand the inner workings of LLMs at the most granular level.

Goodfire believes it has created the world’s first product that leverages mechanistic interpretability to not only try to understand, but also to edit the behavior of AI models. According to the startup, its tools provide developers with deep insights into the internal processes of LLMs. It also offers controls for developers to try and steer their models’ outputs, in a process it likens to performing “brain surgery.” It’s an approach that Goodfire says can lessen the need for time-consuming trial-and-error-based prompt engineering.

Goodfire’s team is well qualified to pursue this kind of approach. Its co-founder and Chief Executive Eric Ho previously founded the AI job finding and recruitment startup RippleMatch Inc. alongside Chief Technology Officer Dan Balsam, while its third co-founder and Chief Scientist Tom McGrath was formerly a senior researcher at Google LLC’s DeepMind.

In an interview with VentureBeat, Ho explained that Goodfire’s tools enable developers to effectively map their AI models’ brain, similar to how a neuroscientist might use imaging techniques to try and understand what’s going on inside a human brain. “We use interpretability techniques to understand which neurons correspond to different tasks, concepts, and decisions,” he explained.

Once the brain has been mapped, Goodfire then attempts to understand which of those neurons are responsible for any unwanted behaviors in the model. It essentially visualizes the AI’s “brain” to make those troublesome pieces easier to identify.

Finally, developers can use Goodfire’s control systems to “perform surgery” on their models, such as by removing or enhancing a certain feature, in order to correct its behavior. Ho said this is similar to how a neurosurgeon might try to manipulate a specific part of the human brain.

“By doing this, users can improve the capabilities of the model, remove problems and fix bugs,” he stressed. “By making AI models more interpretable and editable, we’re paving the way for safer, more reliable, and more beneficial AI technologies.”

Nnamdi Iregbulem of Lightspeed Venture Partners said interpretability is fast becoming a crucial building block in AI development. He believes that these kinds of model editing tools will soon become a “fundamental primitive” for AI developers, enabling them to interact with their models in new and more flexible ways. “We’re backing Goodfire to lead this critical layer of the AI stack,” he added.

Image: SiliconANGLE/Microsoft Designer