Evolution is a process that takes time — and this is especially true when it comes to the development of new data structures.

As Doug Cutting, chief architect at Cloudera, Inc., puts it, “To make something that’s widely used, you have to cross several thresholds, and sometimes that can take quite awhile.” Cutting hopes to integrate Spark as a universal platform but expects this switchover to be slower going than most technological breakthroughs. His time-frame estimate? Almost a decade.

Turns out that it’s difficult to teach an old dog new tricks even when it comes to software: “People learn new systems, they adopt them and embrace them,” Cutting explained to John Walls and George Gilbert (@ggilbert41), cohosts of theCUBE, from the SiliconANGLE Media team, during Spark Summit 2016 in downtown San Francisco.

“It’s difficult to give up these technologies lightly,” Cutting continued. “You want to pick up things that will last.”

Planes, trains and automobiles

And while adopting Spark in more and more verticals may take time, creating a product with longevity is more important than reinventing the system overnight.

“We’re using data to accomplish more than ever before,” said Cutting. It’s a big change that’s currently altering several industries — from financial institutions fighting fraud to pharmaceutical companies better understanding medications and farmers improving crop management

“It’s planes, trains and automobiles,” said Cutting, “It’s everywhere.” And even if the process takes time, Cutting said it’s to be expected: “These big institutions are bigger ships. It’ll take some time to turn them over.”

Watch the full video interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of Spark Summit 2016.

Photo by SiliconANGLE