A survey of organizations using big data has found that only a tiny minority have achieved transformational business value, while most are still at the tire-kicking stage.

The survey of 5,600 respondents at 429 companies also found that most organizations think they’re more accomplished at using big data than they really are. The study was fielded by AtScale Inc., maker of a business intelligence abstraction platform, in cooperation with a host of big data analytics partners.

It found that only 9 percent of organizations are successfully using big data for new revenue generation. Just over half are still in the experimental phase, focusing on saving money and offloading projects from an enterprise data warehouse. Executive support for big data projects at these organizations is low, as is awareness of  big data’s potential benefits, researchers said. Nearly 40 percent of organizations are in a transition phase, having committed to a big data platform and started to build significant projects.

The good news is that 95 percent of respondents said they’ll do as much or more with big data in the future as they have in the past. Only 17 percent consider big data to be just an experiment. Organizations may be guilty of some irrational exuberance when assessing their own big data expertise, however. Nearly 80 percent of respondents rated their maturity as “medium” or “high,” but the survey authors, citing their own undefined maturity methodology, said the real number is closer to 12 percent.

Silage

The top impediment to success is silos, of both the organizational and data kind. These smokestacks limit self-service access to big data goodness. Forty-two percent of respondents said their business units have self-serve capabilities, meaning that the majority are dependent on the IT organization to respond to their requests. That figure is actually worse than the the 47 percent who reported self-service capabilities last year. However, it should be noted that a decentralized strategy may also be a conscious choice.

Researchers said the culprit for the decline in self-service analytics may actually be users’ quickening shift to the cloud, where 59 percent of respondents say they have deployed big data, up from 53 percent last year. The move may be temporarily disrupting end-users’ ability to get at the data they need. There’s no doubt that cloud is where the action is, though, with 76 percent of respondents saying they will use the cloud for at least some of their big data projects and 40 percent saying cloud is a preferable option to on-premises deployment in general.

A shortage of skills was cited as the No. 1 challenge to big data success for the third year in a row. That’s followed by governance and performance, which both ranked higher on the challenge ladder than they did in previous surveys. On the flip side, big data management fell from from the second-most serious challenge in 2016 to fifth this year.

When it comes to the vendors whom respondents trust most for big data deployments, the stalwarts lap the field with 46.5 percent of respondents rating legacy vendors favorably. That’s followed by Amazon Web Services’ Elastic MapReduce with a 29.5 percent favorable rating and Microsoft’s HDI at 11 percent. Google LLC’s BigQuery is coming fast up the adoption curve, though, with 60 percent of respondents saying they’re investigating it.

One indication of the increasingly strategic view organizations are taking toward big data is the perception of the value of Hadoop. Last year, respondents cited cost savings as the No. 2 reason to adopt the big data platform. This year, that factor fell to fifth place.

Image: Flickr CC