Databricks makes Spark projects easier to manage


Analytics professionals are increasingly relying on the cloud as part of their work. Databricks Inc. is stepping up its efforts to capitalize on this trend today by launching a new edition of its data processing service that aims to reduce the amount of effort involved in number-crunching projects.

More specifically, the offering is designed to ease the task of setting up and maintaining an analytics environment. One of the most time-consuming chores that Databricks for Data Engineering tackles is optimizing performance. The service is based on a customized version of Apache Spark, the popular analytics engine created by the startup’s founders, that is fine-tuned out of the box for several common deployment scenarios.

Databricks for Data Engineering runs workloads on Amazon Web Services, which allows users to take advantage of the different infrastructure options that AWS offers. Databricks claims that the optimizations included in its service enable analytics teams to explore their information up to 10 times faster than they could otherwise. Meanwhile, a set of complementary automation features can help streamline the day-to-day administrative work that goes on in the background.

The service can automatically provision more AWS resources for a project when infrastructure requirements increase and helps with several other aspects of cluster management as well. Moreover, there are built-in security controls that provide the ability to restrict what parts of a project analysts may access based on their role.

The latter feature should come in particularly handy for companies in regulated industries, which often process sensitive information as part of their data-crunching efforts that needs to be treated with extra care. To attract such firms, Databricks has made its offering compatible with the healthcare industry’s HIPAA regulations and the SOC 2 standard used in the financial sector.

Today’s release is the fruit of an ambitious growth effort that is fueled by a $60 million funding round Databricks announced last December. The investment was led by New Enterprise Associates and saw the participation of Andreessen Horowitz.

Image: Databricks