As attendees gather for this year’s Spark Summit 2016, many are asking how Spark has been leveraged over the past year to address emerging trends, such as machine learning, expanded data harvesting, improved analytics and more.

Dinesh Nirmal, VP of development for next-gen analytics, Spark, Hadoop and site exec at IBM, met with John Walls and George Gilbert (@ggilbert41), cohosts of theCUBE, from the SiliconANGLE Media team, to discuss these things along with his take on the IBM-customer relationship and the importance of flexibility.

Spark is growing

Nirmal was very happy to see the number of attendees and speakers at this year’s event, mentioning that he felt as though “there’s double the number of people in the keynote.” A major reason for the growth of interest in the event, he felt, was that development and adoption are “taking off” for Spark, along with related technologies.

“Every single customer shop that I’ve gone to has an R-shop,” he said, referring to the increasingly popular programming language that is commonly used for statistical analysis and similar functions. “[R] is getting adopted in a very accelerated space,” he said.

Making it easy for users

Nirmal also noted the flexibility that IBM was striving to enact, saying, “We see that we can give a rich set of libraries that focused developers can use to build new models and have machine learning.”

That open draw for programmers and developers will lead to big things, IBM hopes. “What we envision … let’s make it flexible for the developer … let’s give the developer the choice on what language they want to use,” he said.

Addressing the various goals of statisticians, Nirmal said, IBM is asking: “How can we give the end user, the data scientist, a multitude of choices that they can pick and choose from, rather than just saying [for instance]: ‘You want to develop, use Python.’”

Platform goals

“When we started talking about how [IBM would] build a platform, there’s three things that we looked at,” Nirmal continued. “One, how do we build a scalable open-source-based architecture? … The second piece is, how do we differentiate? … And the last piece is … what we want to do is bring analytics to where your data is.”

This last point, termed as “house calls” for data, along the lines of a doctor making house calls, was one of the ways in which IBM will be more engaged with users looking to fully capitalize on their Spark experience.

“Those are the kinds of things that will get the buzz on top of Spark,” Nirmal said, giving an example of the system applied to machine learning while asking the question of “’how do we real-time score in a feedback loop, so that the model is learning?’”  The answers, he felt, are waiting to be uncovered by the Spark users and developers.

Looking at the future of Spark and these resultant technological abilities, Nirmal was highly positive, saying: “I think there’s tremendous potential.”

Watch the full interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of Spark Summit 2016.

Photo by SiliconANGLE