Three years ago Zhamak Dehghani (pictured) shook up the data management world with a radical new concept called a data mesh. Now the company she founded is bringing it to market as a product.

Nextdata Technologies Inc. today is introducing Nextdata OS, which it describes a unified data product development and operating platform for building, governing and discovering autonomous data products.

A data product is a structured, reusable asset meant to deliver specific business outcomes or insights by leveraging data. They’re built to be easily understood and consumed by users, persistent and reusable across multiple business contexts, self-governing and discoverable. Examples include personalized recommendation engines, real-time analytics dashboards and predictive models.

Nextdata OS encapsulates data products in software containers, which are lightweight, portable units that package applications and dependencies together to ensure consistency across different computing environments that continually sense their environment and adjust processing and orchestration to respond to updates and changes.

They’re an evolution of data pipelines, which are widely and have become “a hairball” in most large organizations, said Dehghani (pictured).

Nextdata was founded on the principle that traditional centralized data management operations are poorly suited to increasingly complex environment characterized by multiple data types and repositories. Such complexity frustrates efforts to unify data governance and creates bottlenecks that cause delays. The problem is worsened by constant data migration, unmanageable data pipelines and operating budgets consumed by ongoing maintenance and support burdens.

Self-service discovery

Nextdata OS’ self-service features let decentralized domain teams safely build, share and discover autonomous data products, making data management is more scalable and assets discoverable. Autonomous data products combine code and policies to automatically self-govern, self-orchestrate and self-publish.

“Data is shared and discovered through automated mechanisms that search your mesh of interconnected data products,” Dehghani said. “We have a discovery tool that allows you to search with natural language, or you can browse and see relationships between data products.

Organizations don’t have to modify or replace their existing data management repositories or physically move data. The data catalog, which is a centralized inventory of data assets within an organization, is replaced by a publish-and-subscribe mechanism that automatically delivers updates, senses the environment, and actively manages, validates and enforces policies. “As soon as new data is added, the company enforces controls.

“They are domain-first and decentralized, so you might have one data supply chain for marketing and 10 for sales,” she said. “They’re all unified and look and feel the same, but underneath they’re sitting on whatever data stack you have.”

Beyond data catalogs

That’s in contrast to data catalogs, which operate on passive data after it has been generated. “Catalogs crawl around and find out what data there is and try to make meaning out of it,” Dehghani said. “In this model we have active data products that are publishing live information about their health, their data and who’s using it with a tool that listens and can surface new data.”

Autonomous data products can manage an arbitrary variety of data types and formats, orchestrating simple tables, unstructured data or vector databases for analytics, machine learning or generative artificial intelligence training through retrieval-augmented generation.

NextData is providing generative AI assistants to help developers build data products from existing data assets and pipelines, reducing the process from months to hours, it claimed. An extensible, pluggable driver architecture allows data products to adapt to different infrastructure and services.

NextDataOS is currently in beta test at several large enterprises, Dehghani said. General availability is “a few months away.” Pricing hasn’t been set, but the company’s philosophy is that “we want customers to pay for what’s generating value for them and make sure customers have predictability,” she said.

Photo: SiliconANGLE