There are two absolutes in the tech world today. One, businesses must go to the cloud. Two, data is the lifeblood of a company. Adapting to this new reality requires a whole new style of infrastructure, and a wise choice in strategic partners, according to John Landry (pictured), distinguished technologist for personal systems data science at HP Development Co. LP.

As companies learn these lessons, cloud spending has increased. However, once a company moves to the cloud, the data must still be processed. Apache Spark provides a popular framework for big data processing.

“Spark is great because we needed a programming platform that could scale,” Landry said.

To learn more about how Spark fits into the big picture of big data, Dave Vellante (@dvellante) and George Gilbert (@ggilbert41), co-hosts of theCUBE, SiliconANGLE Media’s mobile live-streaming studio, spoke to Landry during the Spark Summit East 2017 conference in Boston, MA. (*Disclosure below.)

A solid platform for software development

The interview opened with a look at the reasons for choosing Spark. HP is building its software platform in the cloud, Landry explained. That platform includes analytics, and the company wanted to build its analytics on Spark. It’s about software development more than anything else, Landry revealed. HP needed a common platform for their projects.

Internally, HP spends millions to better its products and optimize its business models, Landry stated. It can also take that same data and apply it to its customers. By monitoring usage in the field through big data analytics, HP can tell how its devices are performing. With this information, the company can improve its processes, increasing the quality it delivers, Landy added.

The topic then turned toward how HP moved to the cloud. “We used as many Amazon services as we could,” Landry said. Using these services allowed HP to focus on working with Spark rather than maintaining infrastructure. The flexibility of Spark is a huge part of HP’s success, Landry mentioned.

“We didn’t want to be the experts on managing Spark,” Landry said. So, they turned toward another partner, Databricks Inc.

Databricks and Spark go hand-in-hand, Landry explained. Having Databricks manage that layer saves a ton of time and resources. “We don’t want to spend resources on running Spark; we have better things to do,” Landry said.

Watch the complete video interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of the Spark Summit East 2017 Boston. (*Disclosure: TheCUBE is a media partner at the conference. Neither Databricks nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)  

Photo by SiliconANGLE