UPDATED 14:02 EST / NOVEMBER 03 2011

spark by Jonas Maaløe Jespersen NEWS

Another Hadoop Alternative: Spark

spark by Jonas Maaløe Jespersen I just published a list of Apache Hadoop alternatives, but here’s another one for the list: Spark. Spark is an distributed in-memory data analytics platform that uses the Scala programming language. IBM claims that Spark should be must faster than Hadoop because it uses in-memory analytics instead of Hadoop’s cluster file system approach. Spark was developed at the UC Berkeley AMP Lab along with Mesos, which is now an Apache Incubator project.

According to a recent paper on Spark from IBM:

Spark is an open source cluster computing environment similar to Hadoop, but it has some useful differences that make it superior in certain workloads—namely, Spark enables in-memory distributed datasets that optimize iterative workloads in addition to interactive queries.

Spark is implemented in the Scala language and uses Scala as its application framework. Unlike Hadoop, Spark and Scala create a tight integration, where Scala can easily manipulate distributed datasets as locally collective objects.

Although Spark was created to support iterative jobs on distributed datasets, it’s actually complementary to Hadoop and can run side by side over the Hadoop file system. This behavior is supported through a third-party clustering framework called Mesos. Spark was developed at the University of California, Berkeley, Algorithms, Machines, and People Lab to build large-scale and low-latency data analytics applications.

Spark is currently in use at Conviva.

Services Angle

Spark is a fresh approach that demonstrates that Hadoop isn’t necessarily the end-all-be-all of big data analytics. There’s quite a bit of room for improvement on Hadoop’s model, whether that’s through Hadoop distributions that add tools to the Hadoop stack or through alternatives like Spark and the others I’ve written about. Most of these tools don’t have the traction that Hadoop has yet, but the market is still open.

Photo by Jonas Maaløe Jespersen


A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU