How much pressure are enterprise data teams facing today? It’s a tricky subject to measure, but one company has given it a try. Ascend.io published the results of a study in June that found that 96% of data professionals are at or over capacity.

Metrics such as these point to a conclusion that data teams are getting caught up in the “muck,” the minutia demanded by an insatiable thirst for data to drive business outcomes. Much as “software ate the world” and spawned the DevOps movement, we are now seeing a similar dynamic play out in the data space.

“We saw this proliferation and expansion of what software roles look like in a company, pulled through this entire new era of DevOps,” said Sean Knapp (pictured, left), founder and chief executive officer of Ascend.io. “We’re finding that same pattern now emerging around data. In the early days, Amazon was described as getting people out of the muck of DevOps, and I think we’re going to do the same thing around getting people out of the muck of data and get them focused on the higher-level aspects.”

Knapp spoke with John Furrier, host of theCUBE, SiliconANGLE Media’s livestreaming studio, during the AWS Startup Showcase: New Breakthroughs in DevOps, Analytics, and Cloud Management Tools event. He was joined in the interview by Jason Robinson (pictured, right), vice president of data science and engineering at Steady Platform Inc., and they discussed the evolution of DataOps, how Steady Platform confronts the challenge of technical debt, and the future role of intelligent systems. (* Disclosure below.)

Faster and safer data

Ascend is a data engineering company that provides a uniform platform to orchestrate and automate a customer’s entire data and analytics pipeline. As a consequence, Knapp and his colleagues are in a prime position to see not simply the evolution of the DevOps world, but the transformation of DataOps as well.

“I think the era of these massive multiyear efforts around data are gone,” Knapp said. “We’ll get into an agile, more nimble methodology where smaller initiatives and smaller efforts are possible by more diverse skillsets across the business. If DevOps basically boiled down to how we help more people write more software faster and safely, DataOps would be very similarly how to enable more people to do more things with data faster and safely.”

One of Ascend’s customers is Steady Platform, a fintech application firm built on the premise that technology can help people steady their financial lives and earn more income. That takes a bedrock of data.

“We have 3 million members actually using this,” Robinson explained. “A lot of what we’re building on the data engineering side is powering insights and analytics that the business stakeholders use every day to run the organization.”

Dealing with technical debt

Robinson’s team is not immune from the pressure of the business identified in Ascend’s survey. The data engineering group at Steady Platform has doubled since the beginning of the year, according to Robinson, yet capacity to meet all of the requests remains a challenge.

Technical debt, the implied cost of additional rework caused by cutting corners, has been an issue in the DevOps world for years. Is DataOps facing the same dilemma?

“The technical debt certainly is there on data,” Robinson said. “You can start cutting corners, you start changing your normalization, and you start taking implicit schema that tend to build into big things. You’re going to make tradeoffs and choices and that debt builds up.”

It is this potential for technical debt in the data space that drives startups like Ascend to build new solutions. The problem, as Knapp sees it, is there is not enough decomposition of key data systems components and data engineering tends to be monolithic in nature. This calls for the greater use of machine intelligence to apply proven techniques in the DataOps world.

“We can improve these things through combinations of intelligent systems,” Knapp said. “It’s things like instantaneous schema detection and validation; it’s things like automated lineage and dependency tracking so you know that when you deploy code what piece of data it affects. It’s pulling a lot of these DevOps principles into the data world.”

Ascend and Steady Platform are on the front lines of the DataOps movement, creating new products and services that will be heavily reliant on effective data processes. It’s challenging work that requires a willingness to think outside the box and an understanding of the end goal.

“You have to be ready to challenge the status quo at every step of that journey,” Robinson said. “If you think through the exact destination you want to be at, that will drive a lot of the decisions as you think backwards to where you started. Thinking through exactly where you want to be is the key.”

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of the AWS Startup Showcase: New Breakthroughs in DevOps, Analytics, and Cloud Management Tools event. (* Disclosure: Ascend.io sponsored this segment of theCUBE. Neither Ascend nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE