It’s hard to appreciate just how large organizations can be until someone has to get all those departments and splintered businesses talking to each other. They need a common platform to share data. Hewlett Packard Enterprise Co., a company currently in a downsizing transition of its own, found that common platform in the open-source data management platform, Apache Spark.

“There’s a lot of organizational dynamics as part of our migration to Spark,” said John Cavanaugh (pictured), research and development master architect at HPE.

Cavanaugh was speaking with David Goad (@davidgoad) and George Gilbert (@ggilbert41), co-hosts of theCUBE, SiliconANGLE’s mobile livestreaming studio, at the Spark Summit 2017 conference in San Francisco. He discussed the benefits of open-source data management frameworks in addressing the fragmented nature of early data systems. (* Disclosure below.)

Herding cats and building platforms

While the benefits of Spark are great, moving to the new technology was not easy for industry giant HPE, according to Cavanaugh. It is a very distributed company, with much autonomy in the various parts, so trying to implement a centralized activity was a challenge. It took some effort to get past the immediate “shields up” response and soothe fears about change.

During Spark Summit, Cavanaugh gave a talk that mentioned the Wild West era of big data. Every department was standing up their own big data systems, he explained. Soon, people realized they had to pull these fragmented setups together. They needed a common platform, and Spark caught on quickly.

In a distributed company, leadership can’t dictate to the businesses. Because of this, HPE made a big push toward cloud-hosted and business-owned. The company knew big data had to be built into their products and services, and at HPE, those were managed by the businesses. “Part of the whole strategy was to empower the businesses,” Cavanaugh said.

Even so, this was not a model where the businesses were set loose to do all sorts of things. HPE shepherded its managed companies to create a virtual community within the company. Through this community, it has been able to share knowledge and strategy. The result has been a more organic movement up through the developers rather than a series of dictates from above, Cavanaugh concluded.

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of Spark Summit 2017. (* Disclosure: DataBricks Inc. sponsored this Spark Summit 2017 segment on SiliconANGLE Media’s theCUBE. Neither DataBricks nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE