The big data bubble could be about to burst, with Gartner Inc. warning in a new report that many companies are looking to cut back on their investments in such projects.

For many years, big data has been touted as a way for businesses to get one step ahead of their rivals by trawling through the vast amounts of data they accumulate to garner useful nuggets of information and recognize trends that would otherwise remain hidden. But the reality is rather different, Gartner has found. In its latest survey of 199 technology executives, the analyst firm found that many companies have struggled to obtain insights that make a real difference to their bottom line.

That finding comes despite the fact that 48 percent of firms surveyed invested in big data projects in 2016, up 3 percent from a year ago. But the number of companies that are planning to invest in big data in the next two years has fallen by 6 percent, from 31 percent in 2015 to just 25 percent this year.

Another telling statistic is that while roughly three-quarters of companies have invested, or are planning to invest, in big data, the overwhelming majority of those firms remain stuck at the pilot stage. Gartner says just 15 percent reported shifting their big data projects into production, up from 14 percent last year.

That’s a negligible increase, and Gartner says that one of the main reasons big data projects have failed to come together is a lack of proper management, combined with lots of trial and error from companies that just throw systems together willy-nilly.

“Too often, pilots and experiments are built with ad-hoc technologies and infrastructure that are not created with production-level reliability in mind,” Gartner said.

Gartner’s findings jibe with recent analysis from Wikibon Research and other experts, who made similar points at the Hadoop Summit trade show in San Jose last June, saying that larger companies are simply not willing to spend the time and energy cobbling together different systems from dozens of startups. “It’s sort of like putting Frankenstein together,” said George Gilbert, big data analyst at Wikibon.

Because of this apparent failure to get big data projects off the ground, a growing number of companies are now prioritizing other IT initiatives instead. Just 11 percent of those organizations which indicated they have invested in big data described their investments as being “important” or “more important than other IT initiatives,” while 46 percent considered them to be “less important,” Gartner said.

A likely reason for this is that most big data projects do not present a tangible return on investment that companies can determine before they’re launched.

“Another reason could be that the big data initiative is a part of a larger funded initiative,” said Nick Heudecker, research director at Gartner. “This will become more common as the term ‘big data’ fades away, and dealing with larger data sets and multiple data types continues to be the norm. While organizations have understood that big data is not just about a specific technology, they need to avoid thinking about big data as a separate effort.”

The report may appear to contradict another recent report from Forrester Research Inc., which said the big data market is set to grow three times as fast as the overall information technology market. However, the latter is close to flat, and the Forrester report looks ahead to future spending out to the year 2021.

Image credit: markusspiske via pixabay