High-performance open-source vector database Qdrant today announced the launch of BM42, a new pure vector-based hybrid search approach for modern artificial intelligence and retrieval-augmented generation applications.

The new algorithm marks a new generation of text-based keyword search capabilities for RAG and AI applications, which allows enterprise customers to combine the best of both worlds. It enables developers to take the understanding of keyword search and combine it with vector understanding to obtain more accurate results and lower costs.

Andrey Vasnetsov, chief technology officer of Qdrant, told SiliconANGLE in an interview that BM42 builds on a previous algorithm, Best Match 25, or BM25. Introduced in the 1990s, BM25 works by assigning a ranking to documents to determine how relevant they are to a given query. The algorithm calculates a relevance score for each document in a collection based on statistics and normalization.

RAG, or retrieval-augmented generation, is used by AI applications to bring real-time information to AI large language models to reduce errors, such as hallucinations, by fetching facts and knowledge outside the trained-in knowledge of the model. BM42 targets the “retrieval” part of RAG by greatly augmenting the ability to pull in information faster, with smaller, cheaper queries.

Although BM25 worked extremely well for highly generalized sets of documents, Vasnetsov explained, it would begin to break down when the knowledge it was working with became too specialized.

“[Traditional models] have a blind spot, such as rare domains such as legal and medical,” he said. “Or imagine that you have a chemical formula or a very long word that doesn’t match with anything across documents. You will get back a document that doesn’t really match what you need. In this case, the search splits into two different branches. It’s a semantic search with dense vectors and lexical search with traditional algorithms.”

Semantic search uses high-dimensional data and the relationships between the meanings of words to bring back relevant information. Whereas lexical search uses the closest match between keywords to determine how likely two searches are related. These two combine, Vasnetsov explained, into what the company calls “hybrid search.”

Vasnetsov said that, unlike BM25, the new algorithm uses a transformer AI model to produce its search results. Instead of using statistics over a collection of documents, the transformer infers the importance of what parts of the document matter to the search. That can overcome issues of not having enough documents in a set to create enough statistical variance.

It also means that BM42 can be used with any transformer-based AI model without the need to fine-tune it. Or customers could choose to fine-tune that model for their use case, purpose or language. For example, it could be tuned for medical, insurance or finance or set to a specific language such as English, German or Chinese and work just as well for hybrid search. This provides the algorithm a high versatility when used for hybrid search capabilities.

Vasnetsov said that the default query method across many AI application integrations is already split – such as LlamaIndex and LangChain — so it should be easy for many developers simply to switch to BM42. “By switching, our customers will get faster retrieval, less memory overhead and a better experience overall with search,” Vasnetsov added.

Image: geralt/Pixabay