When looking at next-generation platforms, it’s clear that the key is collaboration. It’s important not to silo the data. The traditional structures “didn’t enable discoveries, didn’t enable personalization that a role might need,” said Nancy Hensley, director of offering management at IBM.

And, according to Dinesh Nirmal, VP of development for next-gen analytics, Spark and Hadoop at IBM, you need a platform where you can create a project that the data scientist can share with the data engineer. “Spark becomes the engine,” he added.

Hensley and Nirmal talked with John Furrier (@furrier) and George Gilbert (@ggilbert41), hosts of theCUBE, from the SiliconANGLE Media team, during Hadoop Summit US in San Jose, CA.

Adding a security and governance layer

The ideal situation would be to have this open data while masking the lineage, the guests said. The user doesn’t need to know where the data is kept or other extraneous details. They just want to know it’s safe and easy to access.

“We’re putting a fabric between external data and who wants to consume that data,” said Hensley.

“What we bring as enterprise vendor is a blanket,” added Nirmal. Their goal is to make sure it’s covered end to end and that the security and governance layer is there.

Commitment to the open-source community

The pair also highlighted IBM’s commitment to the open-source community with such programs as Spark SQL (Apache Spark’s module for working with structured data). In fact, IBM’s Spark Technology Center in San Francisco is solely focused on contributing to the open-source community.

People are beginning to see the value in that, Hensley said. Both guests see that trend continuing in the future, with Hensley suggesting there will be even more collaboration in the future and Nirmal hinting at interesting work on machine learning in the future.

Watch the full interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of the Hadoop Summit US.

Photo by SiliconANGLE