I just published a list of Apache Hadoop alternatives, but here’s another one for the list: Spark. Spark is an distributed in-memory data analytics platform that uses the Scala programming language. IBM claims that Spark should be must faster than Hadoop because it uses in-memory analytics instead of Hadoop’s cluster file system approach. Spark was developed at the UC Berkeley AMP Lab along with Mesos, which is now an Apache Incubator project.

According to a recent paper on Spark from IBM:

Spark is an open source cluster computing environment similar to Hadoop, but it has some useful differences that make it superior in certain workloads—namely, Spark enables in-memory distributed datasets that optimize iterative workloads in addition to interactive queries.

Spark is implemented in the Scala language and uses Scala as its application framework. Unlike Hadoop, Spark and Scala create a tight integration, where Scala can easily manipulate distributed datasets as locally collective objects.

Although Spark was created to support iterative jobs on distributed datasets, it’s actually complementary to Hadoop and can run side by side over the Hadoop file system. This behavior is supported through a third-party clustering framework called Mesos. Spark was developed at the University of California, Berkeley, Algorithms, Machines, and People Lab to build large-scale and low-latency data analytics applications.

Spark is currently in use at Conviva.

Services Angle

Spark is a fresh approach that demonstrates that Hadoop isn’t necessarily the end-all-be-all of big data analytics. There’s quite a bit of room for improvement on Hadoop’s model, whether that’s through Hadoop distributions that add tools to the Hadoop stack or through alternatives like Spark and the others I’ve written about. Most of these tools don’t have the traction that Hadoop has yet, but the market is still open.

Photo by Jonas Maaløe Jespersen