Startup Streamlio Inc. today announced a major update to the Apache Pulsar publish-and-subscribe messaging platform, which serves as the main rival to the better-known Apache Kafka project.

Publish-and-subscribe messaging systems such as Apache Pulsar are designed to provide a highly scalable way for software applications to communicate with one another.

Pulsar was first developed at Yahoo Inc. by that company’s former engineers Matteo Merli and Joe Francis and donated to the open-source community in 2016. Merli and Francis explained at the time that Pulsar was developed in order to meet the requirements of several Yahoo applications, for which they could not find any existing solution.

“These applications provide real-time services, and need publish-latencies of 5 milliseconds on average and no more than 15ms at the 99th percentile,” they wrote. “At the time we started, we could not find any existing open-source messaging solution that could provide the scale, performance and features Yahoo required to provide messaging as a hosted service, supporting a million topics.”

Merli later left Yahoo to form the startup Streamlio with the intention of commercializing Apache Pulsar, launching a real-time analytics suite that incorporates the messaging system in September. Streamlio’s system also incorporates open-source projects such as Heron and Apache BookKeeper, which combine to create “an enterprise-grade messaging solution optimized for streaming and storage.”

Now, the Apache Pulsar community is updating Pulsar with new analytics capabilities and enhancements under the hood that should boost its performance, scalability and durability.

The headline is a new feature called Pulsar Functions which adds stream-native processing capabilities to Pulsar. It provides developers with a simple way to apply transformations and analytics directly to data as it flows through Pulsar, without any need for external systems or add-ons.

In an interview with SilconANGLE, Merli said the company had seen numerous cases where it would be beneficial to apply processing to data as soon as possible, rather than waiting for it to work its way through the extract, transact and load pipeline, or through a data lake or data warehouse, as is normally the case with analytics. With the addition of Pulsar Functions, Streamlio is now pitching Pulsar as a kind of “Streaming Messaging system” that helps to connect diverse data with users and applications in real-time.

“Most options for that are simply too complicated and heavyweight for organizations, requiring specialized programming skills such as functional programming and requiring deployment and maintenance of additional infrastructure services,” Merli said. “We saw that a stream-native processing capability like Pulsar Functions could address the majority of those use cases without adding any new complexity for the developer or management burdens for operations teams.”

Other new features in Pulsar 2.0 include a new schema registry that helps to simplify the development of data-driven apps, and a new enhancement called Topic Compaction, which helps to improve the performance of apps that use data from Pulsar.

“Pulsar’s advantages make it a great solution for data-driven applications that need to bring streaming data sources into the data and analytics pipeline,” Merli said. “Those include IoT and industrial IoT analytics, security, fraud detection, systems monitoring and online services.”

Image: Markusspiske/Pixabay