Confluent Inc., a provider of real-time data streaming tools based on Apache Kafka, today is using its London-based Kafka Summit to announce new capabilities in its Confluent Cloud that address data quality issues.

The company is also simplifying the process of building connectors to third-party data sources, making it easier for customers to share streams between organizations, delivering a cloud-based Kafka engine and launching an early-access program for a managed version of the Apache Flink unified stream-processing and batch-processing framework.

Quality conundrum

Citing its own research that found that 72% of information technology leaders say inconsistent integration methods and standards impair their use of streaming data, the new Data Quality Rules feature expands upon Confluent’s Stream Governance suite, which helps simplify complex data relationships, enable self-service data discovery and improve event stream quality.

Data Quality Rules allow schemas stored in Schema Registry to be augmented with rules that validate and constrain the values of individual fields within a data stream, enable customizable follow-up actions to take action on incompatible messages and simplify schema evolution using migration rules to transform messages from one data format to another.

Domain validation is an example of a type of rule that can be used in Schema Registry, Dan Rosanova, head of product management for Confluent Cloud, said in emailed comments. That includes declarative constraints on the domain values of fields, such as requiring age must be a positive integer.

“When using this rule, one can model arbitrary follow-ups by specifying an action to use when the condition is true and an action to use when the condition is false,” Rosanova said. “For example, we might have a rule checking if a Social Security number is nine digits long, and if it fails, sends the record to a special dead letter queue topic for investigation.”

Custom Connectors enable any Kafka connector to run on Confluent Cloud without infrastructure management, the company said. This reduces or eliminates the need for customers to manually provision, upgrade and monitor their own connectors. They can use the cloud service to quickly connect to any data system using their own Kafka Connect plugins without code changes and use logs and metrics to monitor the health of connectors. The option is available on the Amazon Web Services Inc. cloud in select regions.

Previously, “teams would have to self-manage their custom-built or modified connectors, which leaves them with time-consuming, perpetual operational burdens,” Rosanova said. “With Custom Connectors, Confluent takes on critical infrastructure activities like provisioning connector resources and managing Connect clusters. Users simply have to upload each of their custom connector plugins once in the console for anyone in their organization to view and provision additional connector instances from Confluent Cloud.”

Stream Sharing allows data and analytics teams to easily exchange real-time data between Confluent and any Kafka client. The feature provides for authenticated sharing, access management and layered encryption controls as well as enforcement of consistent schemas across users and organizations. JSON, Avro, and Protobuf are all supported for schema evolution.

The Flink early-access program follows Confluent’s January acquisition of Immerok GmbH, a startup that built a platform based on Flink. Select Confluent Cloud customers can test the fully managed service and provide feedback to product and engineering teams. Flink is used to handle large-scale, high-throughput and low-latency data streams.

Photo: Arran Bee Sunny Bank Holiday Weekend/photopin (license)