Oumi PBC, a startup building what it calls an “unconditionally open artificial intelligence platform” in collaboration with researchers from top universities, today released an open-source hallucination and bias detection model that it says outperforms some of the leading large language models, using a comparatively small 8 billion parameters.

AI hallucinations refer to instances where an AI model generates false, misleading or nonsensical information that appears plausible but lacks a basis in real data or facts. Although becoming relatively more rare, they have proved to be nearly impossible to eliminate entirely.

Oumi’s HallOumi model uses per-sentence verification, sentence-level citations and human-readable explanations to verify AI-generated summaries against source documents, checking each claim and linking it to relevant context to determine whether the answer is supported by source material.

Because HallOumi requires access to designated source documents, it’s primarily intended for use with retrieval-augmented generation or fine-tuning within an organization.

Trust equation

Trust issues are one of the principal concerns enterprises have about deploying AI models, particularly in customer-facing scenarios. OpenAI LLC’s ChatGPT recently came under fire in Europe for an incident in which the model accused a user of having been convicted of murdering two of his children and several large Wall Street firms warned investors about the risks of hallucinations.

A recent Accenture plc report found that 77% of executives believe the true benefits of AI can only be realized when built on a foundation of trust.

“Hallucinations can occur even in a benign context where you ask for a particular relevant answer according to some chosen truth but the model chooses to ignore or override your source with its own parametric knowledge,” said Oumi co-founder Jeremy Greer.

Errors can also be intentional, he said. For example, DeepSeek, the Chinese LLM that is growing in popularity thanks to its permissive licensing terms, won’t discuss the details of the 1989 Tiananmen Square massacre and praises People’s Republic of China President Xi Jinping’s response to the COVID-19 pandemic.

“It seems that the model’s been trained to ignore context and provide a verbatim biased answer,” Greer said.

Context check

When given context documents and an AI-generated summary, HallOumi goes through every claim in a summary and identifies the relevant context sentences associated with it, an explanation of why a claim is supported or unsupported and a determination of whether or not the statement is supported by the provided context. Results are assigned a score reflecting the probability that a statement is inaccurate.

Statements are color-coded to flag those that likely reflect bias or hallucinations. “It gives you rationale of why it thinks there’s a hallucination to make it easy for you as a human to understand what the problem is,” said Manos Koukoumidis, Oumi’s co-founder and chief executive. The model also provides citations of specific passages in source documents that relate to the questionable output.

“You don’t have to read pages and pages,” Koukoumidis said. “It’s easy for humans to catch the most nuanced hallucinations from a quick glance.”

That involves some additional processing overhead, but the co-founders said the trade-off will be worth it for many enterprises. “It’s slower — it may take several seconds — but it’s more accurate and helps minimize human effort,” Koukoumidis said. “We talked to some of the financial enterprises and asked if they want a small quick model or this. They definitely want the latter.”

Oumi has posted an explanation of how the model works and example outputs. The software will be released under a full open-source license, with Oumi providing customization and support for a fee.

Image: SiliconANGLE/DALL-E