The big data market will reach $5 billion this year and grow at 50% to 70% CAGR for the next five years says Lucid Imagination CEO Paul Dosher. That will take it over $50 billion. In an interview in the Cube from the recent Hadoop Summit 2012, he said that the market split would be 31% hardware, 44% services, and 25% software, which is why major vendors from HP to Cap Gemini were already becoming interested in what is today a very immature technology.

Big data will also be a major new platform for custom application development, he said. “Companies don’t want off-the-shelf big data analysis applications because they won’t give a company a competitive advantage,” he said. However, the real boom in big data software is probably three-to-five years out. “If you compare it to the database market, the big RDMS revolution started at around 1990, and it wasn’t until later in the 1990s that you saw the applications start taking advantage of that infrastructure.”

That puts Lucid’s May announcement of LucidWorks Big Data, the first fully integrated application development platform built on Hadoop, ahead of the curve. LucidWorks Big Data combines Mahout, R, and Lucene/Solr to provide search, machine learning, recommendation engines, and analytics for structured and unstructured content, with a secure, well document REST API for application integration and administration. It fully integrates Pig, Shef, hBase, and Zoo Keeper scripts for rapid application development.

Lucid then made LucidWorks Big Data available in the cloud as a Search-as-a-Service platform under the name LucidWorks Cloud. Its latest announcement is the inclusion of LucidWorks Cloud in the Microsoft Windows Azure marketplace.

“We think this could be a tremendous catalyst for enterprises,” Dosher said. Instead of having to provision its own hardware and software for Hadoop trials and hire rare Big Data developers, “we give you the complete stack and can have a trial up and running in a couple of days. That allows you to bring a representative sample set of your data, load it easily, and focus on your business applications. Once you have user acceptance and understand and appreciate the value of the technology, you can either move the application in house or scale it out in the cloud.” Or it can support a hybrid architecture, which is becoming the dominant trend in enterprise development.

This is already gaining acceptance, he said. “We have very large companies, household names, and large integration firms starting trails in healthcare, insurance, customer support, telecom, genetic research.”

One of the attractions of Open Source is that “you can take that nugget and make it into whatever diamond you want.” That, he said, allows Lucid’s platform to provide a basis for a wide range of use cases across a horizontal spread of industry segments.

“If Lucid Imagination can become the search provider for all these different cloud platforms,” he said, “It provides a great entry point into our technology that can then scale all the way up to the Big Data environment.

“We come to the market with an application development platform that allows you a more scalable, repeatable, experience and allows you to get a further level of abstraction, above the technology. We believe strongly that coupled with the fast that it’s all integrated, ready to go, the APIs are all available, and it’s in the cloud, it will act as a real catalyst.”