We examine the challenges organizations grapple with in the complexities of the modern data stack and look toward a sixth data platform, one assembled from modular, multivendor components. Dbt Labs Inc. has positioned itself to be a leader in turning data into business-level models, a central component in the world of metadata management.

Research is by Rob Strechay and George Gilbert, and the video conversation is with Drew Banin, co-founder of dbt Labs.

Our position

We believe that no single data platform will conquer them all. And that organizations will look to continue to break apart the compute or execution layer from the data storage layer in their composable data platform. In the research, we see organizations looking at ways to address their major challenges: costs of compute and storage, number of systems to manage, how to understand what data they have, and how to govern the use of that data. This research note digs into the last two issues around knowing what data you have and who can use it.

The market

We see customers taking notice of metadata and governance platform solutions offered by their existing vendors, such as Informatica. The company saw an increase of 37% year-over-year, to $617 million in cloud subscription annual recurring revenue in the fourth quarter and full-year 2023. We also see emerging platforms like Starburst, with Trino and their emerging governance platform, and dbt Labs, which is becoming the authoring and execution platform for turning data into business-level models, are also seeing growth in their citation-weight net sentiment, which our partner Enterprise Technology Research reports this growth.

Source-conformed to business-conformed data

In this chapter of the Sixth Data Platform, we examine the intricacies of extending dbt’s pipeline or Directed Acyclic Graph or DAG, expanding the metadata that dbt and partners can generate from that DAG, and the nuances of federated data topologies.

In order to understand how data is transformed within dbt’s platform, we need to delve into its journey from raw, source data to refined data products. This transformation is a complex process that involves multiple steps, such as filtering, joining and aggregating data sets. By using SQL and Python models, dbt creates a detailed map of data transformations, the DAG. This map is not just a technical blueprint but also a living document that tracks the lineage of data, and it is version-controlled, which makes it easy for all stakeholders to collaborate and make modifications over time.

What is a data mesh?

The dbt Mesh paradigm emerged as a response to the scholarly definition of the data mesh concept. Originally designed for composability and representing the data mesh, dbt Mesh has expanded to address governance challenges when building composable transformations in decentralized data environments. With the ability to define data contracts within dbt Mesh, diverse teams can efficiently handle data stewardship. This paradigm draws inspiration from established software engineering workflows, such as the use of contracts with APIs. By bringing these workflows into the data sphere, dbt Mesh helps manage complexity and scale collaboration and provides data contracts.

The challenges and solutions

The problem of scaling across data silos is not limited to data platforms only. In fact, these challenges of scalability arise in dbt and other mesh architectures as well, especially when it comes to supporting multiple compute or execution engines. Although dbt can’t do this today, it’s something the company is considering. This may be necessary for customers who serve pipeline processing or BI dashboards using separate execution engines. dbt serves as a logical layer, not an execution engine like Databricks, Snowflake, MongoDB, Oracle or BigQuery. Compatibility with different underlying data platforms is the key feature of dbt. Despite challenges such as compliance and data sovereignty, dbt’s logical orientation makes it well-suited to provide governed, cross-platform data solutions.

Many organizations have to deal with multiple data silos, which can create difficulties in managing metadata. To address this issue, dbt has introduced the Discovery API that offers comprehensive metadata insights. This new feature enables users and partner ISVs to view the lineage of their entire dbt project, which can help improve data quality and recency understanding. Moreover, column-level lineage is now available, which can greatly enhance impact and root cause analyses.

Dbt Lab’s strategic acquisition of Transform, back in February of 2023, brought a new dimension of capabilities, incorporating a semantic layer that promises to streamline the operationalization of metrics at an enterprise scale. The goal of a semantic layer becomes the de facto standard for metric definitions, enabling data analysts and business stakeholders to coalesce around a single, immutable source of truth.

What about governance?

Governance can be a multifaceted and sometimes nebulous concept, being broken down into three pillars: visibility, change management and data quality. It is seen that effective governance hinges on comprehensive visibility into the data’s lifecycle within an organization. This transparency enables a deep understanding of data’s transformation and application, paving the way for informed governance practices.

SQL versus Python

The question of whether semantics will grow in SQL or if it will require Python for more expressivity was top of mind. Part of this debate is surrounded by whether organizations will ensure consistency and reliability when integrating Python models into dbt, given the diverse nature of Python code. While it is acknowledged that Python’s expressive power is a perceived advantage, it was noted that SQL’s position is that of the dominant language of analytics. We believe there will be a continued expansion of SQL’s capabilities, driven by data platforms’ evolution, thus blurring the lines between the functionalities traditionally exclusive to Python.

Our perspective

Our perspective through an ongoing thorough analysis of the evolving landscape of data platforms highlights the shift towards modular, multi-vendor systems exemplified in the Sixth Data platform. Central to this transformation is the metadata management and governance “control plane,” where dbt Labs is positioned to be a pivotal player in turning data into actionable business models and advancing metadata management. This underscores the growing importance of metadata and governance solutions, challenges in scalability and data management, and the strategic integration of technologies like dbt Mesh and the Discovery API.

The acquisition of Transform by dbt Labs, enhancing metric operationalization, and the ongoing debate between SQL and Python’s roles in analytics reflect the dynamic nature of this field. This evolution signifies a future where data platforms are more adaptable, integrated and focused on efficient data governance and enhanced metadata visibility, driving informed decision-making and innovation in data analytics.

Here’s the full interview with Banin:

Image: Adobe Stock