Estuary Technologies Inc. has raised $14 million in early-stage funding to try to merge batch and streaming data into a single pipeline so it can simplify how information is fed into artificial intelligence applications.

Today’s Series A round was led by M13 and saw participation from several other investors, who were not named by the company.

The startup is taking aim at something that continues to cause lots of headaches for enterprises as they race to implement AI – namely, data movement. For many businesses, moving data between systems is not an easy thing, because existing tools force them to make an unenviable choice. Either they can utilize batch processes to integrate their data, which are more reliable but far too slow, missing critical changes, or they can adopt real-time streaming systems that are notoriously fragile and expensive, resulting in excessive costs and many broken integrations.

Estuary co-founder and Chief Executive David Yaffe told SiliconANGLE that the problem is not so much about the choice between batch and streaming, but rather the fragmentation that results from having to use both. “They have slow, rigid batch pipelines on the one side, and costly, fragile streaming systems on the other,” he said. “Each AI workload demands a different latency profile, and companies end up gluing together multiple vendors or homegrown tools, which drives up cost, complexity and risk.”

Traditional data pipelines tend to be based on batch processing, and these are generally fine for workloads such as analytics, which only require scheduled data ingestion, Yaffe said. But batch systems aren’t fast enough for many AI applications, which need the freshest, most recent data to be useful.

“It’s at this point that data teams have to bolt on a streaming tool or build an entirely new data pipeline, creating a second integration stack to maintain,” Yaffe explained. “The issue is that companies outgrow their batch pipelines, and end up stuck with multiple vendors and contracts, duplicated transformations and logic, inconsistent data depending on which system they query and more cost and complexity.”

Integrated batch and streaming

Rather than deal with two systems and all of these problems, Estuary gives companies the option of a single, integrated batch and streaming pipeline that delivers data to applications with variable latency, depending on the workload. According to Yaffe, it allows customers to choose between streaming high-frequency updates for mission-critical applications and scheduled batch processing data for analytics and training AI models. “Estuary solves the fragmentation by collapsing the tradeoffs into a single system where latency is a dial, not a choice,” he said.

In other words, the company has built an entirely new kind of data integration platform that merges batch and real-time processing. That way, companies can capture, transform and synchronize their data across every source, be it legacy databases, customer relationship management portals or software-as-a-service applications. Then they can then feed it into different applications at the desired speed — either in batches or in real time.

Yaffe said businesses can use Estuary to replace their existing change data capture, batch and streaming tools. It gives them a single, fully managed platform that does both, with enhanced control over data latency. With Estuary, latency can be controlled like a speed dial, so teams can automatically switch among subsecond, near-real-time or batch processing, depending on the needs of each specific workload. Estuary also improves the reliability of data pipelines, Yaffe said, with desirable features such as exactly-once semantics, deterministic recovery, targeted backfills and flexible deployment.

Other advantages include more predictable costs, with customers able to choose from flat-fee and throughput-based pricing options. According to Yaffe, this can help companies to achieve cost savings of between 40% and 60% on average compared with traditional data integration systems. “Pricing is based on data throughput or else we charge a flat fee,” he said. “We eliminate the opaque monthly active row-based pricing, which lets enterprises accurately forecast spend.”

A hard problem to solve

Constellation Research Inc. analyst Michael Ni said Estuary’s push to unify batch and real-time data pipelines and bundle it together as a managed service mirrors a broader consolidation trend that’s sweeping the big data ecosystem. “It’s a smart bet because data movement is the unsung foundation of enterprise analytics and intelligence, and given AI’s demand for fresh, trustworthy data, enterprises will appreciate being able to trade off latency and cost,” he said.

Ni said Estuary is taking on a lot of competition in this area, as all of the biggest data warehouse players, including Snowflake Inc., Databricks Inc. and Google LLC, as well as integration specialists such as Fivetran Inc., are looking for ways to consolidate pipelines. He said it’s an incredibly challenging problem, since it requires them to provide all of the connectors and the reliability enterprises need, while ensuring data engineers have the level of control and governance they’re used to. In addition, they have to keep things simple.

“The problem Estuary is trying to solve is very hard, and it’s unclear how it is doing this under the covers, since it is making hard tradeoffs that typically require architectural differences,” Ni explained. “Most companies still operate two stacks — one for batch and one for real-time streaming — so it’ll be interesting to find out how it has broken the back of that problem.”

Estuary claims that it’s already serving dozens of customers across industries such as finance, healthcare and logistics, helping them to consolidate their data stacks, modernize their data infrastructure and realize significant savings. With the funding from today’s round, it intends to build on that early success by expanding its product, engineering and go-to-market teams.

“This raise allows us to accelerate toward a future where pipelines simply work, where data moves when and how teams need it, powering both today’s analytics and tomorrow’s AI,” Yaffe promised.

Image: SiliconANGLE/Meta AI