When the Hadoop project (an open-source software for reliable, scalable, distributed computing) began, it became clear that things were far too complicated. So Databricks, Inc. set out to simplify the Big Data process with the Apache Spark project.

Dave Vellante and George Gilbert, cohosts of theCUBE, from the SiliconANGLE Media team, set down at Spark Summit East 2016 to speak with Ali Ghodsi, CEO of Databricks, to talk about the simplification of Big Data.

The advantages of Cloud 

The implementation of Cloud has made everything a very different game, and Databricks is using this to its advantage with Apache Spark, the largest open-source project in data processing.

“This is one of the beauties of the Cloud,” Ghodsi said. “It’s up and running. It’s elastic. So you can compose those services and products that you need seamlessly.”

For admins, it’s easier because Databricks makes sure software updates are always integrated. The company only has two versions of its software running at all times in the Cloud. And it also releases updates every week. But with non-Cloud settings, things release every few months. “It’s continuously developing features and deploying them to customers,” said Ghodsi.

Big Data problems customers need solved 

For businesses and Big Data, the problem that Databricks is attacking has many pieces. First, companies have already stored data, but the data is siloed and they can’t make decisions on it. Second, there was the problem of unifying the use cases that companies have. Companies want to go further with advanced analytics. They want to cluster and segment the data they have. They also want to do things in real time.

So one of the visions of  Databricks’ Spark service is: How can Spark give businesses something that works end to end? “So unifying these is where we provide value to the customer,” said Ghodsi.

Watch the full video interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of Spark Summit East 2016. And join in on the conversation by CrowdChatting with theCUBE hosts.

https://youtu.be/NKUvui8D8Ic

Photo by SiliconANGLE