It’s been over a decade since Google’s engineers first developed MapReduce to process massive datasets using distributed computing. That technology rapidly evolved into the Hadoop platform for processing and storing Big Data, which for years been hyped as the “next big thing” in IT.

Now, it looks like the hype is finally giving way to reality. Or at least that’s what Forrester Research believes. The research company’s just-published 2015 predictions say 2015 will be the year Hadoop finally becomes “a cornerstone of your business technology agenda”.

“The jury is in. Hadoop has been found not guilty of being a hyped-up open source platform. Hadoop has proven real value in any number of use cases,” said Forrester in its Hadoop predictions for 2015.

Forrester says that all of these uses cases are powered by something called “Hadooponomics”, which it describes as “the ability to linearly scale both data storage and data processing,” and is closely tied to pay-per-use “cloudonomics”.

Prioritizing Hadoop

Analysts disagree on the exact state of Hadoop in the enterprise right now, but its fair to say that most organizations have only dabbled with it so far. International Data Corp. said earlier this year that only 32 percent of enterprises had actually deployed Hadoop. Meanwhile, Gartner Inc. said just 30 percent of enterprises have invested in Big Data technology, and that the number that had actually deployed Hadoop was far lower than this. Another report from Wikibon found that 36% of Big Data practitioners had deployed Hadoop, although “of the majority of respondents using Hadoop, 64% are doing so in proof-of-concept environments.”

But Forrester says a number of visionaries are bucking this trend. Its report names Wal-Mart, Fidelity Investments, Sears, Verizon, USAA, Cardinal Health, Wells Fargo, Proctor & Gamble, Cablevision, Nasdaq, AutoTrader, Netflix, and Yelp as “leaders” in Hadoop, with proven, mission-critical deployments.

“The remaining minority of dazed and confused CIOs will make Hadoop a priority for 2015,” Forrester predicts. “Application development and delivery professionals should be ready and waiting with a compelling use case to get started.”

Evolving technology

Forrester’s other Hadoop predictions for 2015 address the Hadoop skills shortage, its evolution beyond analytics, Hadoop cloud deployments and SQL.

SQL – According to Forrester, rapid ANSI-compliant SQL-on-Hadoop options will lead to the creation of “immediate opportunities for Hadoop to become a useful data platform for enterprises” due to their accessibility from existing systems and data-management professionals familiarity with them.

Cloud clusters – Forrester also predicts that “cloud elasticity” will become central to the cost-efficiency of Hadoop operations, as compute, networking and storage needs are rarely in sync. That means that Hadoop will increasingly run workloads in the cloud to meet the demand for ad hoc analytics, although this trend won’t slow down on-premise adoption just yet.

Beyond analytics – Thanks to the YARN management layer, which includes options like Apache Spark, HBase and NoSQL, Hadoop won’t just be used for analytics. Instead, it will evolve into an application platform that runs code in combination with data, effectively leading to a collapse of the application stack. The implication is that developers of middleware, databases, search engines, integration tools, and other software infrastructure components will need to develop versions that can run “natively, inside Hadoop”, says Forrester.

Skills shortage – The most surprising prediction is the Hadoop skills shortage will evaporate by the end of next year. “CIOs won’t have to hire high-priced Hadoop consultants to get projects done,” says Forrester’s report. “Hadoop projects will get done faster because the enterprise’s very own application developers and operations professionals know the data, the integration points, the applications and the business challenges.”

photo credit: Jason A. Samfield via photopin cc