The Big Data revolution has changed almost everything in business, but what it hasn’t changed is the need for the right information, in the right hands, at the right time. This is the domain of search engines, where the user’s needs are translated into commands to find data and bring it out from the cold dark of storage. Lucidworks is a player in the search market, building applications using Solr and Spark.

To shed some light on Spark and search, Jeff Frick and George Gilbert, cohosts of theCUBE, from the SiliconANGLE Media team, joined Grant Ingersoll, cofounder and CTO of Lucidworks, at the Spark Summit East 2016 conference.

Trusted filters

“Search permeates every part of your life,” Ingersoll said, as he listed off some search applications like maps and looking up info about the Spark Summit show. Lucidworks, he said, was trying to drive more of that feeling into the enterprise. Toward that goal, the company  has brought in machine learning and natural language processing and has leveraged Spark to constantly update its understanding of data. The more users interact, he said, the better data gets.

Search, Ingersoll said, helps refine the algorithm itself. He then mentioned what he called “the three C’s,” meaning content, collaboration and context. By combining these, search applications can put together a solid ranking of what’s important to the user.

Spark and the stack

Lucidworks has relied on Spark for its top products, creating open-source connectors between Spark and Solr. Ingersoll pointed out that using data locale awareness, they could co-locate Spark workers with Solr nodes. They’ve also been using Spark for tasks from low-level DevOps to machine-learning models.

As for the stack, Ingersoll said that it’s constantly evolving. Spark is a good fit right now, he said, but it’s hard to predict what will happen in a few years. He then mentioned how new tools to help manage distributed programming will be useful in the future.

Watch the full video interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of Spark Summit East 2016. And join in on the conversation by CrowdChatting with theCUBE hosts.

Photo by SiliconANGLE