Early in his new report on adoption of Apache Spark in Systems of Intelligence on Wikibon Premium, Wikibon Analyst George Gilbert uses a wonderful analogy to describe the much-maligned performance limitations of the MapReduce processing framework: “If an entire program is like a spreadsheet, each cell has to write to disk in order to pass its results to other cells.”

That’s one of several reasons Spark has generated so much interest and why it may, in Gilbert’s words, “fuel overall big data investment between 2020 and 2022…Spark-based investments will capture six percent of total big data spending, growing to 37 percent by 2022.”

The complexity of the fragmented Hadoop ecosystem is holding back the framework’s potential to revolutionize the way enterprises manage data, Gilbert asserts. Into this maelstrom has come a simpler and faster alternative in the form of Spark, along with broad file system support and simpler interfaces for programmers and administrators alike.

“Simplicity through unification is progressively replacing the mix-and-match flexibility and complexity of specialized engines that grew up in Hadoop to compensate for MapReduce’s shortcomings,” Gilbert notes.

The fact that Spark works in memory makes it orders of magnitude faster than Hadoop. Spark also employs a unified engine that can ingest both batch and streaming data and iterate constantly in memory. That means organizations can move closer to the goal of performing analytics in near-real-time. This would make it possible, for example, for a retailer to deliver customized coupons to shoppers standing at a checkout counter by analyzing both historical data and the contents of a shopping cart. That same real-time iterative capability will make Spark the preferred platform for machine learning applications, in which computers continually run analytics routines over a data set.

“Spark will be able to operate…fast enough that it’ll be possible to connect customer interactions with the transactional applications that recommend or take a decision,” Gilbert predicts. As a result. “by 2026, 59 percent of all big data spending will be tied to Spark or related streaming analytics as enterprises seek to deploy applications that can make decisions on behalf of individuals.”

Spark’s dominance isn’t a foregone conclusion, though. For one thing, Hadoop isn’t standing still, and streaming alternatives like Apache Flink and the still-incubating Apache Apex may offer better stream-processing options. Cloud vendors are also working to create tightly packaged suites of applications that will be easier to use than native Spark.

But regardless of which technology wins out, the shift from batch to real-time analytics is under way, and users should understand the issues and potential. “Doers need to get started early with Spark as a unified streaming analytic engine because each successive application pattern builds on capabilities of the previous one,” Gilbert concludes.

SiliconANGLE photo