Enterprises answering complex big data problems with similarly complex software “solutions” are burying themselves deeper in the weeds, according to David Hseih (pictured), senior vice president of marketing at Qubole Inc.

“We really believe that complexity is just killing big data,” Hseih told Jeff Frick (@JeffFrick) and George Gilbert (@ggilbert41), co-hosts of theCUBE, SiliconANGLE Media’s mobile live streaming studio, during the Data Platforms event in Litchfield Park, Arizona. (* Disclosure below.)

Throwing buckets of hyped-up software at the problem clearly doesn’t cut it, Hseih said, citing research from Gartner Inc. showing a dismal 70 percent failure rate for big data initiatives. “I mean that’s enormous. There’s no segment of the technology business that can survive 70 percent project failure,” he said.

There is hope yet for software solutions, but they must abstract away complexities, not simply rearrange them, according to Hseih. Intelligent automation with machine learning holds promise here, allowing data teams to stop playing defense so much, he added.

“They’re debugging pipelines that are blowing up; they’re getting yelled at because somebody’s report’s too slow or a dashboard didn’t get updated,” he said. Automation takes these tasks off their plate, freeing them to create things that differentiate the business.

The higher-level work may include DataOps, which reworks information technology and end-user relations with both technology and cultural adjustments, Hseih stated.

“There’s a bunch of sort of slightly counterintuitive behaviors, like decoupling your IT team from the end-users and treating the data team as sort of a platform building team, that can unleash amazing efficiencies,” he said. Automation and DataOps solutions such as these rely heavily on cloud infrastructure and services, he added.

Hybrids carry that weight to cloud

When Gilbert asked if some vendors’ on-prem beginnings might constrain them in the cloud, Hseih (chuckling) said indeed they would.

The crux is that cloud infrastructure decouples the storage and compute bound together on-prem, so software and orchestration techniques land in totally different places, he explained.

“Anybody that tells you, ‘Oh well, we have a hybrid system,’ is telling you they have a least common denominator system, because these things are so different,” Hseih concluded.

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s independent editorial coverage of Data Platforms 2017. (* Disclosure: TheCUBE is a paid media partner for Data Platforms 2017. Neither Qubole Inc. nor other sponsors have editorial influence on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE