To build data-driven organizations, enterprises have to deal with a plethora of tooling, making data orchestration fundamental.

As the driving force behind open-source workflow management platform Apache Airflow, Astronomer Inc. uses it to author, monitor and schedule data pipelines for optimal data orchestration, according to Paola Peraza Calderon (pictured, left), co-founder and product leader at Astronomer.

“By data orchestration, we mean data pipeline, so generally speaking, getting data from one place to another, transforming it, running it on a schedule, and overall just building a central system that tangibly connects your entire ecosystem of data services,” she said. “We at Astronomer … build products powered by Apache Airflow for data teams and for data practitioners.”

Peraza Calderon and Viraj Parekh (right), co-founder and field chief technology officer of Astronomer, spoke with theCUBE industry analyst Lisa Martin for a CUBE conversation ahead of the AWS Startup Showcase: “AI/ML: Top Startups Building Foundation Model Infrastructure” event. They discussed why data orchestration has become a burning issue in the modern enterprise world and how Astronomer fits into the picture. (* Disclosure below.)

Astro is at the heart of data orchestration

Using its flagship cloud-native data orchestration platform called Astro, Astronomer makes data orchestration a reality, according to Parekh, who said that it makes the life of data practitioners easier.

“What we really believe here at Astronomer is Airflow is the core of how you solve data orchestration for any sort of use case, but it’s not everything,” he said. “It needs a little more. And that’s really where our commercial product, Astro, comes in. We’ve also built the right things around developer tooling for customers to really rely on Astro as the heart of the way they do data orchestration.”

Since Apache Airflow is an open-source tool, it plays an instrumental role in making data orchestration a reality by taking pain points away. It also acts as the foundational layer that brings together different tools and practices, Parekh pointed out.

“Airflow emerged as the open-source standard, and there are companies of all sorts using it,” he said.  “And it’s really used to tie all these tools together. Especially, as that number of tools increases, companies move to hybrid cloud, hybrid multicloud strategies, and so on and so forth.”

Astronomer takes the open-source route

The open-source culture is heavily embedded in Astronomer, according to Peraza Calderon, who said that this has made innovation inevitable.

“We at Astronomers started using Airflow and actually building our products because Airflow is open source, and we were our own customers at the beginning of our company journey,” she said. “The open-source community is at the core of everything we do. We organize meetups, and we sponsor the Airflow Summit. And there’s this sort of baseline community efforts.”

Since data orchestration encompasses automatically organizing data for easier accessibility, processing, and analysis, Astronomer prompts this using tools like Sigma dashboard, according to Peraza Calderon.

“We have data from third-party API tools, like Salesforce and HubSpot, and other ways in which we actually engage with our customers and their behavior,” she pointed out. “Our data team internally at Astronomer uses a bunch of tools to transform and use that data. We use Fivetran, for example, to ingest. We use Snowflake as our data warehouse.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s pre-event coverage of the AWS Startup Showcase: “AI/ML: Top Startups Building Foundation Model Infrastructure” event:

 (* Disclosure: Astronomer Inc. sponsored this segment of theCUBE. Neither Astronomer nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE