Hadoop has well and truly passed the hype cycle and is now delivering tangible value to the almost half of all organizations that have deployed between 10 and 500 plus nodes, according to a comprehensive new survey on the state of the Big Data framework’s maturity.

The 2015 Hadoop Maturity Survey, conducted by AtScale Inc., a company that provides business users with fast and secure self-service access to Hadoop, Cloudera Inc., MapR Technologies Inc., and Tableau Software Inc., reveals that 49 percent of organizations have already realized tangible value through their deployments, while another 45 percent of Hadoop users say they’re “hopeful” of realizing that value soon.

AtScale says the survey, which was compiled with answers from more than 2,100 participants in North America, Latin America, Europe and Asia, is the broadest and deepest survey on the state of Hadoop thus far, and provides insights into the current and future use of Hadoop and where and how it delivers business value to the companies investing in this foundational big data technology. Unlike other surveys, the 2015 Hadoop Maturity Survey was focused only on those organizations that have already deployed, or are on the verge of deploying Hadoop.

Adoption is growing and accelerating

One of the survey’s key findings comes in direct contrast to other reports claiming to highlight the slow pace of Hadoop’s adoption the enterprise. On the contrary, Hadoop’s adoption is growing and accelerating rapidly among AtScale’s survey participants, with 76 percent of those already using it saying they plan to do more with the Big Data framework in the next three months, compared to just three percent who say they’re planning on doing less. As for those companies that have not yet deployed Hadoop, half of those say they will do in the next 12 months.

Despite the high level of enthusiasm for Hadoop, most users are still in the early stages of their deployments, with 56 percent of respondents having just 25 nodes or less. Nevertheless, a sizeable 30 percent of organizations are making good progress, and already have more than 50 nodes up and running.

More Hadoop = Greater value

Most users are still in the early stages of their deployments, but that suggests there’s plenty more value to come. The survey found that organizations with more advanced deployments are seeing significantly more value than those who’ve just begun. Almost three quarters of organizations with 500 or more nodes claim to be getting value out of Hadoop, compared to just 33 percent of organizations with 10 nodes or less. The survey shows a significant correlation between the amount of nodes deployed and the realization of value, as this graphic illustrates.

Whether or not companies achieve value can depend on the reasons why they’re adopting Hadoop, however. The survey showed that organizations which deploy Hadoop to drive revenues and address scale out needs are far more likely to achieve value than those who use the platform to deploy new applications or cut costs. The driving force behind Hadoop adoption also has an impact, according to the survey. Some 57 percent of organizations which said executive mandate is driving Hadoop have achieved tangible value, compared to 49% of organizations where business is the main driver, and 47% where adoption is driven by IT.

Hadoop at a tipping point?

When the Big Data trend first emerged around five years ago, Hadoop was quickly recognized as the platform of choice for two kinds of workloads – Extract-Transform-Load (ETL, the process of taking data from multiple sources, formatted and uploaded to a database or data warehouse) and data science, which involves querying that data to discover potentially game-changing business insights.

It’s a story that’s reflected in Hadoop’s deployment thus far. The vast majority – some 74 percent of companies – are using Hadoop for ETL, while 62 percent are using it for data science and another 65 percent for business intelligence. However, the survey suggests that business intelligence could soon become the number one reason to use Hadoop. Of those who’ve yet to roll out their deploments, 69 percent said they intend to use it for business Intelligence, compared to 56 percent for data science and 51 percent for ETL.

Maturity by Industry

Hadoop adoption not surprisingly varies quite drastically depending on the industry. The survey reveals that online companies that build their businesses on vast amounts of unstructured data are the most mature adopters. Hadoop is also popular in the consulting, financial services, and
telecommunications industries, but surprisingly has yet to gain much traction in healthcare and financial services.

“This survey is full of surprising insights, at times contradicting results of previous surveys, and provides a very granular view of what people are actually doing with Hadoop,” said Wayne Eckerson, an internationally recognized thought leader in the business intelligence and analytics field, who will be dissecting the survey results at an upcoming webinar on September 22.

“There has been a lot of confusion around the actual state of the Hadoop market,” says Dave Mariani, CEO and founder of AtScale. “With this survey, we aimed to cut through the hype, identify where leading organizations can win with Hadoop, and expose the best practices they employed to gain value faster from this new data platform.”

Image credit: Nuzree via pixabay.com