

Big data management platform Apache Spark wants to extend its streaming capabilities to serverless application development via DataBricks Inc., a cloud-based data service founded by Spark’s creators. But will this make it harder to wield data at the Internet of Things’ edge?
“I’d like to get a sense for how you optimize Spark deployments in a radically distributed IoT edge environment,” James Kobielus (@jameskobielus) (pictured, center) told George Gilbert (@ggilbert41) (pictured, left) and David Goad (@davidgoad) (pictured, right), co-hosts of theCUBE, SiliconANGLE Media’s mobile livestreaming studio, during this year’s Spark Summit. (* Disclosure below.)
The analysts discussed the benefits and possible challenges of database integration and serverless automation coming into view for Spark.
Spark started out as an offline branch of analytics that users applied to a separate data lake or repository. Now, with streaming and serverless app development, users desire greater mobility, Gilbert stated.
“We want to see it put into production, but to do that, you need more than just what Spark is today. You need a database or key value option,” he said.
Just as database integration may open new terrain for operational apps, serverless app development may constrain Spark users at the IoT edge, according to Kobielus. With serverless applications, users do not know the underlying infrastructure, so how will they monitor the edge?
Indeed, the IoT edge environment can consist of thousands or even millions of end-points, Kobielus explained. “How do you monitor end-to-end in an environment like that and optimize the passing of data and the transfer of the control flow or orchestration across all of those disparate points?” he asked.
“Spark is kind of a heavyweight environment, so you’re probably not going to put it in the boot of your car — or at least not likely any time soon,” Gilbert said.
Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of Spark Summit 2017. (* Disclosure: DataBricks Inc. sponsored this Spark Summit 2017 segment on SiliconANGLE Media’s theCUBE. Neither DataBricks nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)
Support our open free content by sharing and engaging with our content and community.
Where Technology Leaders Connect, Share Intelligence & Create Opportunities
SiliconANGLE Media is a recognized leader in digital media innovation serving innovative audiences and brands, bringing together cutting-edge technology, influential content, strategic insights and real-time audience engagement. As the parent company of SiliconANGLE, theCUBE Network, theCUBE Research, CUBE365, theCUBE AI and theCUBE SuperStudios — such as those established in Silicon Valley and the New York Stock Exchange (NYSE) — SiliconANGLE Media operates at the intersection of media, technology, and AI. .
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a powerful ecosystem of industry-leading digital media brands, with a reach of 15+ million elite tech professionals. The company’s new, proprietary theCUBE AI Video cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.