Databricks Inc., the leading commercial entity behind the Apache Spark, the open source cluster computing framework for Big Data processing, last week dropped a few hints about some of the new features we can expect to see in Spark 2.0.

There’s still no concrete release date for Spark 2.0, although it’s widely expected to be released at some point this year. At last week’s SparkSummit though, Databricks CTO Matel Zaharia, the creator of Spark, showed off a number of slides that hint at the coming evolution of the platform, as first reported by InfoWorld.

According to Zaharia there’s going to be three major changes to Spark in the next edition. First, Zaharia revealed Databricks is planning to deepen the integration of Project Tungsten with Spark in order to resolve one of its key limitations, Java’s memory-handling. The second major evolution will see significant improvements to Spark’s real-time streaming capabilities, while the third change will see the unification of Spark’s APIs into a single API.

It’s also likely that Databricks will want to further the integration of Apache Arrow, the latest Top Level Apache project, into Spark. After all, Arrow promises the Holy Grail of a ten to one hundred-fold speed boost to the already very fast Spark by enabling columnar in-memory analytics, a memory mapping technique that arranges data in columns rather than rows.

Zaharia also spoke a little about Databrick’s new Community Edition, a free version of its platform that’s currently in beta, saying that one of the features allows users to “seamlessly transition their prototypes to production applications on the full Databricks platform.” Clearly then, Databricks is planning to use the free version as a way to tempt customers to try out its subscription-based Databricks Cloud platform.

It remains to be seen how well Databricks is doing from a financial perspective. As a private company it doesn’t report earnings (however, that could soon change) though it claims to have signed up 200 paying customers already. However, Databricks will have to watch its back. Much bigger players like IBM are also pushing their own Spark-as-a-Service offerings, and although Databricks is still seen as the leading entity behind Spark’s development, it may not always remain that way.

Image credit: wilhei via pixabay.com