Confluent Inc. is extending Confluent Cloud, its flagship data streaming platform, with a managed Apache Flink service that will make it easier for customers to analyze their business information. 

The service made its debut today at Current, the company’s annual user conference. Confluent also introduced a number of other new features at the event including an artificial intelligence chatbot and a secure networking tool.

Nasdaq-listed Confluent was founded in 2014 by the creators of Apache Kafka, a popular open-source data streaming platform. Kafka enables companies to quickly move information between their internal systems. A manufacturer, for example, could use the software to stream malfunction alerts from a factory sensor to a cloud-based maintenance application.

Confluent’s flagship product is a managed data streaming platform called Confluent Cloud. It promises to reduce the amount of development work involved in moving data between applications. Confluent Cloud was is still based on Kafka but is now powered by Kora Engine, its Apache Kafka engine built for the cloud that the company describes as faster and more cost-efficient.

Real-time data processing

As part of today’s update, Confluent Cloud is receiving a managed Flink service. Flink is an open-source tool that allows companies to run calculations on large volumes of real-time information. Using the technology, Confluent Cloud customers can not only stream data in real time between their systems but also modify that data while it’s traveling.

The first task to which the new managed Flink service can be applied is data filtering. Using the technology, a company could analyze a stream of customer purchase logs and filter records that contain incorrectly entered information. Flink can also merge multiple data streams into a single stream, which eases processing, as well as enrich the information with records from external sources.

Flink is considered fairly complicated to use. According to Confluent, its new managed Flink service automates much of the manual work involved in deploying and maintaining the software. As a result, processing real-time data should become less complicated for customers.

The service is currently in open preview. On launch, it’s available in a limited number of Amazon Web Services cloud regions. Confluent said the service will become generally available “soon.”

New AI features

A few months from now, Confluent plans to update its managed Flink service with an AI feature powered by OpenAI LP technology. Customers will gain the ability to analyze the information they process with Confluent Cloud using large language models. According to the software maker, the upcoming feature will lend itself to tasks such as extracting the most important records from a data stream and summarizing text.

“Stream processing is critical for identifying and protecting against security risks in real time,” said Vinay Krishna Patnana, an engineering manager at Cisco Systems Inc.’s Meraki unit, which develops Wi-Fi networking equipment. “With Confluent’s fully managed Flink offering, we can access, aggregate and enrich data from IoT sensors, smart cameras and Wi-Fi analytics to swiftly take action on potential threats in real time, such as intrusion detection.”

The Flink service’s OpenAI integration will roll out alongside another new machine learning tool called Confluent AI Assistant. It’s a chatbot aimed at making Confluent Cloud easier to use.

Using natural language prompts, developers can have the chatbot fetch information such as the costs that a data streaming project incurred in the past month. The Confluent AI Assistant is also capable of automating certain technical tasks. A developer could, for example, ask the chatbot to generate code that changes how information travels between two applications.

The Confluent AI Assistant is trained on publicly available information such as the software maker’s product documentation. According to the company, the chatbot can also take into account technical information about customers’ Confluent Cloud deployments. It will become available early next year.

Simplified security and network access

Alongside its upcoming Flink service and AI capabilities, the company introduced two other new tools for Confluent Cloud at its Current event today. Those tools are focused on cybersecurity and developer productivity, respectively.

Companies often implement their AWS deployments as a VPC, an isolated cloud environment that can’t be accessed from the public internet. Confluent is rolling out a tool called Enterprise Cluster that will make it easier for customers to move data between their VPCs and Confluent Cloud. Enterprise Cluster sends data not via public internet infrastructure but rather a private network link operated by AWS, which offers better security.

Topping off today’s roster of product updates is a tool dubbed Data Portal. It allows developers to view centrally all the data streams in their company’s Confluent Cloud environments. According to Confluent, the tool will enable software teams to access the internal data they require for an application project more quickly.

The company detailed the feature additions alongside new technical collaborations with five database makers: MongoDB Inc., Rockset Inc., Pinecone Systems Inc., Weaviate Holding Inc. and Zilliz Inc. The four companies will enable customers to access real-time data streams powered by Confluent Cloud directly from their respective databases. 

The main goal of the partnerships is to simplify joint customers’ AI initiatives. MongoDB, Rockset, Pinecone, Weaviate and Zilliz have all equipped their databases with the ability to store vectors, the mathematical structures in which large language models store their information. To make those language models more useful, companies can now provide them with real-time information from Confluent Cloud data streams. 

Image: Confluent