BIG DATA
BIG DATA
BIG DATA
What’s the big deal with event-by-event streaming in Spark’s 2.2 release?
Apache has announced that its Spark 2.2 release will finally break from near real-time data streaming into true real-time, event-by-event streaming. How is this difference of milliseconds relevant, anyway?
“Your streaming capabilities dictate the class of apps that you’re appropriate for,” George Gilbert (@ggilbert41) (pictured, right) told David Goad (pictured, left), co-hosts of theCUBE, SiliconANGLE Media’s mobile live streaming studio. Gilbert and Goad discussed the announcement during the Spark Summit event in San Francisco, California.
Applications were a focus during the Summit’s keynote earlier today, said Gilbert, who is also head big data and analytics researcher at Wikibon.com.
“Spark started out as … offline analytic preparation of data that was in data lakes, and it’s moving more into the mainstream of production apps,” he said, noting that event streaming helps make predictive machine learning applications possible.
Until now, Spark’s structured streaming “had to manage a cluster; it was working with a query optimizer; and so it would basically batch up events in groups that would go through, like, once every 200 milliseconds to a full second,” he said.
Spark has re-engineered structured streaming in the 2.2 release to the tune of one millisecond latency for event-by-event streaming, Gilbert explained.
Continuous coming attractions
A special onstage presentation during the keynote showed an application make predictions about cars in James Bond movies with streaming event data and machine learning.
The idea behind this is that with streaming data and machine learning in prefect parallel, apps can perform predictive analytics at a faster clip, Gilbert stated.
The implications of this will continue to be parsed throughout the summit. “The big thing is what’s the sweet spot? What type of apps? What are the edge conditions?” he asked.
Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s independent editorial coverage of Spark Summit 2017.
Photo: SiliconANGLE
A message from John Furrier, co-founder of SiliconANGLE:
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
- 15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more
- 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network.
About SiliconANGLE Media
SiliconANGLE Media is a recognized leader in digital media innovation, uniting breakthrough technology, strategic insights and real-time audience engagement. As the parent company of SiliconANGLE, theCUBE Network, theCUBE Research, CUBE365, theCUBE AI and theCUBE SuperStudios — with flagship locations in Silicon Valley and the New York Stock Exchange — SiliconANGLE Media operates at the intersection of media, technology and AI.
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.
