IBM’s support for Apache Spark “throws a huge endorsement to the community and to customers as a way to telegraph what’s next,” said theCUBE cohost John Furrier. At IBM Spark, held in conjunction with Spark Summit, thought leaders got a sense of why a powerhouse such as IBM supports the platform.

“Spark is a game changer,” said Paco Nathan, Databricks training director and data veteran. “It’s allowing more of the business customer inside of an organization to get hands-on with the data and get something useful out of it.”

Investing more in data

With the rise of Spark and, relatedly, Databricks, a solution for data scientists and engineers that will now collaborate with IBM to contribute key machine learning capabilities to Spark, companies now face the decision of whether to invest more in data. Nathan sees less risk in a nine-minute project than the nine-month projects of days past.

“A lot of the Spark adoption is about mitigating risk,” he told theCUBE. “We were cutting back in IT because projects were enormous. With Spark, you can get right into the data, prove what you need to prove and scale it up when it’s time.”

New data science capabilities have proven effective in certain marketing projects. Nathan said the traditional process of taking a sample of data to make a model is a thing of the past.

“The data is too large to really sample effectively,” he said. “If you have a hundred million customers, why not work with the data for a hundred million customers directly and come up with segments — maybe 10, 20 or 100. Spark is opening up the ability to get in and treat the data.”

Transformations in the medical community

Nathan predicts huge transformations in the medical community, particularly cancer research and genomics. These and other industries can expect bottom line savings.

“We need to be smarter about how we use resources, about how we answer to the bottom line,” he said. “Spark makes this a lot easier to do than the tools that we had before.”

Watch the full interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of IBM Spark 2015.