In one of its most significant product updates since going public, Confluent Inc. today introduced a new cloud-based toolkit to help enterprises analyze real-time data from their systems more efficiently. 

The company is calling the toolkit Stream Governance. 

Confluent sells data management products based on the open-source Apache Kafka platform. Kafka is used by companies to move data among their systems. For example, a manufacturer can leverage the platform to transfer maintenance data from factory sensors to a production line troubleshooting tool. Kafka is deployed at more than 80% of the Fortune 100 and many other companies.

Kafka is popular because it can transfer large amounts of data and do so at high speed. The platform is capable of moving hundreds of thousands of data points per second with latency measured in milliseconds. Kafka’s speed makes it useful in scenarios where information needs to be processed quickly, such as when a manufacturer wishes to analyze equipment error alerts as soon as they’re generated to catch technical issues early. 

Stream Governance, the new toolkit Confluent introduced today, extends Kafka with additional features. The company has two main objectives with the offering. One is to reduce the amount of development work involved in real-time data analytics projects. The other is making it easier for enterprises to ensure that their real-time analytics comply with data regulations and internal policies. 

The first component of Stream Governance toolkit is Stream Lineage. It provides administrators with insight into what information is being processed by each application connected to Kafka. Stream Lineage tracks which systems generate the information that an application processes, whether the data was modified en route by one of the company’s analytics workflows and, if so, how.

Stream Lineage can simplify regulatory compliance, according to Confluent. The insights the tool provides into how data travels between systems and whether it’s modified along the way helps administrators check if the information is being processed in compliance with relevant regulations.

Stream Lineage also promises to make developers’ work easier. Before upgrading one of their company’s real-time data analytics systems or deploying a new one, software teams have to ensure that the change won’t cause technical issues. Developers can consult the insights that Stream Lineage provides on how data travels between systems to determine if an update might lead to a malfunction, then adjust their code accordingly should the need arise.

The second major component of Confluent’s Stream Governance toolkit is Stream Catalog. It’s a kind of app store for data streams. 

A stream of data, such as equipment maintenance logs generated by a factory sensor, is known as a topic in the context of Kafka deployments. Multiple applications can “subscribe” to a topic to access the latest data points as they arrive. According to Confluent, Stream Catalog allows a company to organize all its topics in a centralized catalog to help employees easily find data sources they can use in software projects. 

One potential benefit of using Stream Catalog is a reduction in duplicate development work. When two different business units at a company are building applications that subscribe to data from the same system, each unit may create a separate Kafka topic to ingest the data. With Stream Catalog, Kafka topics can be reused across teams to reduce duplicate coding. Stream Catalog also lends itself to sharing other Kafka-related software components besides topics across an organization.

Another new feature Confluent announced today is called Schema Management UI. It complements an existing component of Confluent’s product portfolio known as the Schema Registry.

Confluent created Schema Registry because many enterprise applications can only process information if it’s organized in a specific way. The details of how an application’s information needs to be organized are kept in a file known as a schema. Schema Registry, as the name implies, is a software system that helps companies manage the schemas used by their various applications. 

The newly introduced Schema Management UI aims to make Schema Registry easier to use by adding on a centralized administration interface. The interface enables developers to understand what schemas have been deployed throughout their company, how they’re used and by which applications. Obtaining this information can normally be a challenge in a large enterprise with a lot of applications. 

The introduction of Stream Governance today marks one of Confluent’s most significant product updates since its initial public offering earlier this year. The company raised $825 million in the IPO and saw its shares more than double on their first day of trading.

Image: Confluent