Oath, the subsidiary of Verizon Communications Inc. that was created when the company acquired Yahoo Inc. earlier this year, said today it’s open-sourcing some of its most important internal software for executing web searches and generating recommendations and targeted advertisements.

The software is called Vespa, and Oath said it’s used to tackle the tricky problem of deciding what to show users in response to input such as text typed into a search box. Oath said it actually uses Vespa to power more than 150 applications, including its popular photography website Flickr.com, Yahoo Mail and some aspects of the Yahoo search engine, such as local results, answers to questions and image searches. Vespa also powers Yahoo’s advertising, handling more than 3 billion native ad requests every day.

Oath said Vespa, now available to download on GitHub under an Apache 2.0 open-source license, can easily be added to a whole host of different applications. As such, the company reckons it could well be useful even for big web companies such as Amazon.com Inc., Google LLC, and Facebook Inc. to process requests on different types of data sets.

Yahoo actually created Vespa back in 2005, following its acquisition of a small search company called AlltheWeb in 2003. The software was originally used to power Yahoo’s shopping services, before being put to use in numerous other applications.

According to Jon Bratseth, distinguished architect for Vespa, the software can process and serve up content or ads up to 90,000 times per second with latency of just tens of milliseconds.

“To deliver a search result or a list of recommended articles to a user, you need to find all the items matching the query, determine how good each item is for the particular request using a relevance/recommendation model, organize the matches to remove duplicates, add navigation aids, and then return a response to the user,” Bratseth wrote in a blog post.

Vespa architecture

Bratseth claimed the release is the most important since Yahoo open-sourced the code for its big data software Hadoop back in 2006. Hadoop has since emerged to become the foundational technology for almost all big data software in use today. It also led to the creation of three major tech companies – Cloudera Inc., Hortonworks Inc., and MapR Technologies Inc. — all of which offer sophisticated big data platforms based on Hadoop.

“By releasing Vespa, we are making it easy for anyone to build applications that can compute responses to user requests, over large datasets, at real time and at internet scale — capabilities that up until now have been within reach of only a few large companies,” Bratseth added.

It remains to be seen if Vespa will ever achieve the same success as Hadoop has had in the enterprise, however. At least one analyst urged caution. Holger Mueller, principal analyst and vice president at Constellation Research Inc., said the technology looks “interesting” and could possibly become a “key contribution” to the open-source ecosystem, though it’s too early to tell just yet.

“It follows the long Yahoo tradition on Hadoop and big data all the way back to Pig,” Mueller said. “But the question is, will it find uptake and is Yahoo going to support this, or is this the infamous, not working, ‘punt over the wall’ to open source?”

Image: zuccherolibero/Flickr