As the Spark Summit East 2016 event continued into its final day, more discussion of Spark’s potential and how to bring it to consumers was shared by attendees. Looking at how to address its competition and compatibility with Hadoop environments, making it more widely accessible, and finding effective ways to enhance revenue generation were all on the table. But, for some, understanding the immediate impact of Spark has been the most engaging item.

Jeff Frick and George Gilbert, cohosts of theCUBE, from the SiliconANGLE Media team, met with Dirk deRoos, data lake product strategist for IBM, to discuss some of the ways in which IBM is putting Spark to work and how its customers have been responding.

Spark and success stories

deRoos, who had attended previous Spark Summit events, touched on how distinct it was from other industry events. “One thing that strikes me about this conference … is how technical it is,” he said. “When you come here, it’s different.” From the inclusion of lines of code in keynote presentations to the willingness of attendees to go into detail when discussing their products and solution tools, the Spark Summit serves as an informative gathering for those already familiar with Spark and for those who are looking to begin integrating it into their products and services.

However, deRoos also noted, “For a lot of the customers that I’ve worked with, [adopting Spark has] been a frustrating journey.” He pointed out that while there have been some significant success stories using the engine, “by and large, they’re fairly limited.”

To deRoos, this is partially a result of the utility’s relative newness, as technicians responsible for integrating it are still learning its ins and outs. He characterized Spark as “special plumbing,” but added that “once we’ve figured out how to use that plumbing, we can do some pretty incredible things for the user.”

Overcoming hype; enhancing understanding

deRoos also saw some of the disconnect between expectations of people looking to adopt Spark, and the reality of it, as part of the “hype cycle,” which most major new tech utilities have to endure. Taking away from the glamor of this hype were the challenges encountered by adopters in meeting issues of governance and regulatory compliance.

As he stated, “Engines aren’t about governance, they’re about how to get work done,” and companies that expect to be able to apply a broad fix without addressing the context into which Spark is being integrated are running into more than they anticipated. deRoos cited a recent case of a bank wanting to fit Spark into a web app, drawn to the hype around the service without understanding how to best utilize it, or that many of its strongest features would be unused in such a situation. For these cases, making customers aware of effective alternatives tends to produce the best results.

However, deRoos was optimistic overall in considering Spark’s further development in the tech landscape. “Where we have seen success stories is where there’s a very specific problem … and people try to do some work at scale. When there’s a business problem tied to that, that’s where we see success. … Where there’s a return, where people can see how does this fit into our bigger business journey.”

Watch the full video interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of Spark Summit East 2016. And join in on the conversation by CrowdChatting with theCUBE hosts.

Photo by SiliconANGLE