Artificial intelligence agent and assistant platform provider Vectara Inc. today announced the launch of Open RAG Eval, an open-source evaluation framework for retrieval-augmented generation.

RAG is a technique that enhances AI responses by retrieving relevant external data to inform its output. The approach improves accuracy and reduces hallucinations by grounding responses in real-time, trusted information.

Vectara’s new Open RAG Eval framework, developed in conjunction with researchers from the University of Waterloo, allows enterprise users to evaluate response quality for each component and configuration of their RAG systems in order to quickly and consistently optimize the accuracy and reliability of their AI agents and other tools.

Open RAG Eval is designed to determine the accuracy and usefulness of the responses provided to user prompts, depending on the components and configuration of an enterprise RAG stack. The framework assesses response quality according to two major metric categories: retrieval metrics and generation metrics.

By surfacing insights across these two metric categories, Open RAG Eval enables developers to diagnose performance issues at a granular level. For example, low retrieval scores may signal the need for better document chunking or improved search strategies, while weak generation scores might point to suboptimal prompts or the use of an underperforming language model.

The framework is compatible with any RAG pipeline, including Vectara’s own generative AI platform and other custom solutions. Early adopters can use Open RAG Eval to make informed decisions about whether to implement semantic chunking, adjust hybrid search parameters, or refine prompt engineering for better overall results.

Vectara notes that the framework’s development was made possible through its collaboration with Professor Jimmy Lin and his team at the University of Waterloo, who are renowned for their contributions to information retrieval and evaluation benchmarks. Their research foundation helps ensure that Open RAG Eval delivers both scientific rigor and practical utility for enterprise applications.

“AI agents and other systems are becoming increasingly central to how enterprises operate today and how they plan to grow in the future,” said Professor Lin. “In order to capitalize on the promise these technologies offer, organizations need robust evaluation methodologies that combine scientific rigor and practical utility in order to continually assess and optimize their RAG systems.”

Vectara is a venture capital-backed startup that has raised $73.5 million over three rounds, including rounds of $28 million in May 2023 and $25 million last July. Investors in the company include FPV Ventures LP, Race Capital, Alumni Ventures, WVV Capital, Samsung NEXT, Fusion Fund, Green Sands Equity LP and Mack Ventures LP.

Image: Vectara