Big data projects are ready to move to the cloud on a large scale, but a host of concerns about regulations, ownership and the structural impediments of moving large amounts of data off-site will limit mass adoption of public cloud platforms, says a new Wikibon report.

The market for big data in the public cloud was $1.1 billion last year, comprising just five percent of all big data revenue. But it will grow to $21.8 billion by 2026, at that point making up 24% of the total market. Still, early adopters should proceed with caution, advises Wikibon Big Data in the Public Cloud Forecast, 2016-2026 (available on Wikibon Premium; subscription required).

Plenty of big companies are already doing some form of big data in the cloud, including Airbnb Inc., Lyft Inc., Yelp Inc. and Splunk Inc., researchers report, based upon a review of approximately 650 case studies profiled on Amazon Web Services LLC’s website. However, the lack of full scale software-as-a-service (SaaS) applications for big data today is limiting growth. Researchers expect that the applications are on the way, however.

Wikibon gathered its data by conducting scores of interviews with big data vendors earlier this year and incorporating extensive user surveys conducted over the past two years. Interviews at trade shows, media reports and other unstructured information was also considered.

Go where the data is

One of the reasons big data in the cloud is so popular in advertising technology, social networks, e-commerce and other customer-facing applications is that the cloud is a good place to merge proprietary data with public data that is already being gathered in the cloud. The cloud is also a good place to experiment with analytics without incurring large capital expenses. The biggest users of cloud-based big data so far have been departments within organizations, which appreciate the rapid deployment advantages.

However, enterprise-scale projects are more complex, and large enterprises may still opt for private cloud solutions from established big data players or may choose to build their own. Still, the cloud has one big benefit in the complex Hadoop ecosystem: simplicity. “Cloud-native products…are designed, built, tested, and operated to work as one integrated set of services,” Wikibon writes.

The more intractable problem for market growth is the complexity and cost of moving large amounts of data to the cloud. Edge-based solutions will emerge over time to moderate this issue and reduce the need to put all data in one place.

Three stages of growth

Wikibon expects big data in the cloud to unfold in three stages. Over the next three years, users will build data lakes that take advantage of the relative simplicity of integrated Hadoop ecosystems as a service. Data lakes comprised 43 percent of big data-related spending in the public cloud last year, Wikibon estimates, but that share will diminish over time.

Researchers call the second stage “maturing intelligent systems of engagement.” These applications will be driven largely by the need to improve the customer experience by taking advantage of data such as e-commerce transactions and click-throughs, which are more efficiently captured in cloud-based systems. Streaming and near-real-time technologies like Apache Spark will be an important factor in making customer interactions more immediate.

Beyond 2022, growth will be driven by “self-tuning systems of intelligence,” including predictive analytics, disaster avoidance and proactive repair. The Internet of things (IoT) will factor heavily in this development, with cloud-based analytics used to effect adjustments and repairs at the edge. The amount of data processed may be smaller, but uses will be more focused and the value will be greater.

By 2026 data lakes will shrink in importance to 25 percent of overall big data spending in the public cloud. By that point systems of engagement will make up nearly half of all spending.

Proactive organizations should get started now by identifying big data patterns that are a natural fit with cloud deployment. Focus on those that exploit data that’s already in the cloud or that take advantage of integrated big data stacks. However, researchers caution users to “avoid the temptation to assume that all big data apps will end up in the cloud. The costs of data movement are real and permanent and will always impact big data architectures.” It’s going to be a big market nonetheless.

Photo by Theerawat Sangprakarn via Flickr CC