UPDATED 12:50 EDT / JUNE 19 2015

George Gilbert NEWS

Wikibon view: IBM legitimizing Spark to compete with Hadoop

IBM has legitimized Spark and wants to shift the axis of Big Data processing, the way it did the operating system world when it committed to Linux across all its platforms 16 years ago, writes Wikibon Big Data analyst George Gilbert.  Back then the competition for Linux was several proprietary flavors of Unix and the Microsoft server platform. Today the competition for Spark is Hadoop.

Spark is a unified engine that can handle real-time, interactive and batch processing, applying machine learning, graph processing, SQL and streaming capabilities (see figure above). This makes it simpler than Hadoop, which is an ecosystem of tools unified around the YARN cluster resource manager on top of HDFS. Each distro has a slightly different set of component products.

Spark shines in applications that involve multiple data environments, such as omni-channel marketing. Spark can use batch machine learning to profile each customer from the historical database, then apply streaming data analysis to update individual profiles in real time as customers browse products. Spark can then use that information to influence the customer’s purchase choices.

Hadoop’s advantage is that each of its individual tools is optimized for a single function, making them more efficient at that particular task than Spark can be. For example, the Presto SQL data warehouse will probably have greater capacity and speed than anything built on Spark, writes Gilbert.

As a result, Hadoop is not going away. However, with IBM’s commitment Spark is no longer an academic project, Gilbert writes, and it may unify many of the processing layers that are separate in Hadoop.

The full Alert and other Big Data analysis is available on the Wikibon Premium Web site.

Graphic ©2016 Wikibon


A message from John Furrier, co-founder of SiliconANGLE:

Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.

  • 15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more
  • 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network.
About SiliconANGLE Media
SiliconANGLE Media is a recognized leader in digital media innovation, uniting breakthrough technology, strategic insights and real-time audience engagement. As the parent company of SiliconANGLE, theCUBE Network, theCUBE Research, CUBE365, theCUBE AI and theCUBE SuperStudios — with flagship locations in Silicon Valley and the New York Stock Exchange — SiliconANGLE Media operates at the intersection of media, technology and AI.

Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.