Big Data might not get as much press as it used to, but a new study by AtScale suggests that it’s rapidly maturing as analytics software and services explode in the cloud.
AtScale, a specialist in business intelligence on Hadoop, last week published its “Big Data Maturity Survey” of 2,550 big data professionals at 1,400 companies across 77 countries, in conjunction with Cloudera Inc., Hortonworks Inc., MapR Technologies Inc., Tableau Software Inc., Cognizant Technology Solutions Corp. and Trifact Inc.
The survey found that almost 70 percent of respondents have had Big Data deployments up and running for over a year, compared with 59 percent a year ago. Of those, 76 percent are using Hadoop now, and 73 percent say they’re using it in production. In addition, 74 percent say they have more than 10 Hadoop nodes up and running, while 20 percent have more than 100 nodes on the go.
“The maturity of respondents in this survey is a key consideration,” Thomas Dinsmore, Big Data analytics industry analyst and author of the book “Disruptive Analytics,” said in a statement. “One in five respondents has more than 100 nodes and 74 percent of them are in production, indicating double-digit growth year-over-year.”
The future of Big Data is cloudy
A more important finding is that most companies now deploy Big Data in the cloud rather than keeping it on-premises. Some 53 percent of respondents said they host some Big Data analytics in the cloud, while 14 percent have all of their Big Data in the cloud.
“There’s been a clear surge in use of big data in the cloud over the last year and what’s perhaps as interesting is the fact that respondents are far more likely to achieve tangible value when their data is in the cloud,” said AtScale cofounder and Chief Technology Officer Matt Baird.
Business Intelligence is booming
AtScale’s study also looked at the primary Big Data workloads companies are performing, and found that Business Intelligence has overtaken extract, transform and load, known as ETL, as the most popular workload. Seventy-five percent of respondents indicated that they were planning to use BI on Big Data in the next year, AtScale said, while 97 percent say they plan on doing more in the next three months.
Surprisingly, though, despite all the hype around Apache Spark, which seems to have displaced Hadoop as the sexiest Big Data tool in the box, not many companies are using it. AtScale’s survey found that 42 percent of companies use Spark for educational reasons, but have no real projects up and running yet. A third of respondents said they’re using Spark in development, but just 25 percent said they were using Spark in both development and production.
Despite that finding, the study does reveal that organizations that have deployed Spark in production are 85 percent more likely to achieve value.