Today’s data teams must face the costs of maintaining fragmented solutions with lots of seams between tools. 

They deal with decreased productivity and velocity of data teams, as well as challenges in hiring and retaining data engineering expertise. 

Sean Knapp (pictured), founder and chief executive officer of Ascension Labs Inc., aims to solve these challenges through automation by eliminating repetitive coding tasks that take up valuable time and resources while also eliminating errors that negatively affect data quality. This allows data teams to focus on creating business value with accurate data. Spaces mature and evolve, and the number of repetitive and repeatable tasks becomes less differentiating but far more taxable to the business, according to Knapp.

“As you see those patterns really start to formalize and standardize, it opens up the door to, basically, not have your team have to do all those things anymore … and you can lean more on technology to properly automate and remove the monotony of those tasks and give your teams greater leverage,” he said.

Knapp spoke with theCUBE industry analyst John Walls at the AWS re:Invent 2022 Global Startup Program, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how tech stack automation can free up team resources and make data more useful. (* Disclosure below.)

An aware data automation platform

Data stacks are undergoing an innovation boom. Most of these innovations consist of point solutions for developers building extract, transform and load; orchestrating jobs; or loading or moving data between systems. Using more integrated data automation eliminates seams to address the data problem end to end.

Ascend.io is a data automation platform that connects disparate parts of what data teams do when they create ETL and EOT data pipelines with advanced levels of automation.

“We use advanced levels of automation to make it easier and faster for them to build these complex systems and have their world be a little bit less of a cluster,” Knapp stated. 

The company has developed an engine that knows every piece of code, data, and the links and dependencies between them. Every change in the engine is propagated autonomously through all links, including newly arrived data and changes in code.

Trends to watch for

In early 2023, a survey of 500 chief data officers, data scientists and data architects asked whether they currently or intend to invest in data automation to increase their team’s productivity. Only 3.5% said they currently invest in automation, while 88.5% said they intended to do so in the next 12 months, according to Knapp.

“That stark contrast of how many people have a thing and how many people want that benefit of automation — I think it is incredibly critical as we look to 2023 and beyond,” Knapp said. “[There are] other tools in the marketplace around ETO tools or orchestration tools that some would argue fit in the automation category. And I think what the market is telling us based on that research is that those … don’t qualify as automation — that the market has a larger vision for automation, something that is more metadata-driven, more AI-backed … than what the existing capabilities in the industry today can afford.”

What does the future hold for the 88% who intend to automate? With the large movement toward data platforms, one of the biggest challenges is getting data in so it can be utilized, according to Knapp.

“I think as we start to invest more in this metadata-backed layer that can connect those notions of how do you ingest your data, how do you transform it, how do you orchestrate it, how do you observe it, one of the really compelling parts of this is that metadata does become the new big data itself,” he said. 

To give data teams greater levels of automation and leverage, cloud capabilities need to be able to process large volumes of metadata to deliver on these powerful capabilities.

“I think that’s why this new world that we see of the developer platforms for modern data cloud applications actually benefit from being a cloud-native application themselves,” Knapp said.

In 2024, Ascend.io customers can look forward to deepening integration with data infrastructures such as Snowflake, Databricks, BigQuery and AWS, as well as increased engagement with developers, such as with free tiers, developer events and investment in smart onboarding features. Data teams will be profoundly more productive in delivering data products and analytics to the business, allowing companies to expand their businesses without having to struggle for scarce data, Knapp concluded.

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of the AWS re:Invent 2022 Global Startup Program:

(* Disclosure: Ascension Labs Inc. sponsored this segment of theCUBE. Neither Ascend nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE