

IBM has legitimized Spark and wants to shift the axis of Big Data processing, the way it did the operating system world when it committed to Linux across all its platforms 16 years ago, writes Wikibon Big Data analyst George Gilbert. Back then the competition for Linux was several proprietary flavors of Unix and the Microsoft server platform. Today the competition for Spark is Hadoop.
Spark is a unified engine that can handle real-time, interactive and batch processing, applying machine learning, graph processing, SQL and streaming capabilities (see figure above). This makes it simpler than Hadoop, which is an ecosystem of tools unified around the YARN cluster resource manager on top of HDFS. Each distro has a slightly different set of component products.
Spark shines in applications that involve multiple data environments, such as omni-channel marketing. Spark can use batch machine learning to profile each customer from the historical database, then apply streaming data analysis to update individual profiles in real time as customers browse products. Spark can then use that information to influence the customer’s purchase choices.
Hadoop’s advantage is that each of its individual tools is optimized for a single function, making them more efficient at that particular task than Spark can be. For example, the Presto SQL data warehouse will probably have greater capacity and speed than anything built on Spark, writes Gilbert.
As a result, Hadoop is not going away. However, with IBM’s commitment Spark is no longer an academic project, Gilbert writes, and it may unify many of the processing layers that are separate in Hadoop.
The full Alert and other Big Data analysis is available on the Wikibon Premium Web site.
Graphic ©2016 Wikibon
Support our open free content by sharing and engaging with our content and community.
Where Technology Leaders Connect, Share Intelligence & Create Opportunities
SiliconANGLE Media is a recognized leader in digital media innovation serving innovative audiences and brands, bringing together cutting-edge technology, influential content, strategic insights and real-time audience engagement. As the parent company of SiliconANGLE, theCUBE Network, theCUBE Research, CUBE365, theCUBE AI and theCUBE SuperStudios — such as those established in Silicon Valley and the New York Stock Exchange (NYSE) — SiliconANGLE Media operates at the intersection of media, technology, and AI. .
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a powerful ecosystem of industry-leading digital media brands, with a reach of 15+ million elite tech professionals. The company’s new, proprietary theCUBE AI Video cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.