The real-time streaming enabled by Apache Spark 2.0, the newest version of the popular engine for Big Data processing, may be the best thing to happen to data so far this year. It’s difficult to think of another breakthrough that is solving so many issues for companies trying to make practical use of data.

But streaming data tools have been around for a while. What makes the new applications novel, says Spark creator and Databricks Inc. Chief Technology Officer Matei Zaharia, is how they integrate the batch and interactive queries that have been done for decades with streaming to provide an “end-to-end” tool that Databricks calls “continuous applications.”

Zaharia spoke recently about the evolution of data tools and how they came to be combined into single solutions. He told George Gilbert, host of theCUBE and an analyst with Wikibon Research, both owned by the same company as SiliconANGLE, about the problem of separate batch and streaming systems producing different results on the same data.

“And then the customer would say, ‘Hey, I was looking at your streaming thing at 5 o’clock, and it said there were 10,000 users on my video, but now you charge me for 11,000 users,'” he said. “‘What’s us up with that?'”

A single solution

He spoke about Databricks’ mission to resolve the disconnect between interactive applications and the need to make sense of real-time streaming data. “We want to design APIs where you can combine these end-to-end pieces,” he said, adding that most companies and organizations desire that consolidation. So for example, streaming data can be integrated into recommendation engines or credit card fraud detection programs to improve their results in real time.

“You’ll be able to use the same sophisticated algorithms you could use on static data and run them on a stream and get the same results, results that make sense,” he said.

Watch the complete video interviews below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of Innovation Day at Databricks.

Photo by SiliconANGLE