It looks like the already overcrowded stream processing camp of the Hadoop ecosystem is about to receive even more competition, and from one of its most important partners no less. Confluent Inc. has revealed the completion of a $24 million funding round this morning that will help augment its hugely popular messaging service with native capabilities for analyzing fast-moving data.

Originally developed at LinkedIn Inc. to help manage the streams of information crisscrossing its business networking service, Apache Kafka acts as a sort of central nervous system for real-time processing clusters that automates the distribution of the various signals ingested for analysis. The main feature setting the project apart from more traditional messaging services is scalability.

That is very much a reflection of the internal operational conditions at LinkedIn, which necessitate delivering data to its destination not only quickly but also efficiently, a requirement that has led to Kafka being able to handle upwards of hundreds of thousands of data points per second using just a few servers. As a result, it’s particularly well-equipped to support the vast amounts of information that flow through a typical production-grade stream processing clusters.

That has helped make Kafka a staple companion to every major real-time analytics engine in the Hadoop ecosystem, including Spark Streaming, Storm and Samza, which was also developed at LinkedIn and is deliberately built to take advantage of its scalable architecture. It’s this reliance that gives so much weight to Confluent’s plans for getting into stream processing.

The fact that Kafka is used so widely means that practically every organization that is processing real-time data in Hadoop today on a meaningful scale has operational professionals trained in using the service. That will make it a much easier for prospective buyers to adopt the upcoming native stream management functionality.

With complexity often cited as one of the main barrier to large-scale analytics in the enterprise, the simplicity of integrated messaging and analytics functionality could provide a competitive advantage for Confluent against the existing alternatives, which are notoriously difficult to implement. And the pitch will be made all the stronger by the fact that the company also intends to add storage capabilities into the mix.

Confluent is banking that the same scalability and efficiently that make Kafka so good at shuffling messages around should also prove helpful in storing those messages for extended periods of time. Of course, it will probably take a while for the startup to bring its planned stream processing bundle up to par with the competition on every level, but the new funding should help hurry that along.

At the same time, the investment – led by Index Ventures with participation from existing backer Benchmark Capital – also raises the pressure on Confluent to monetize the success of Kafka. That makes it more than likely that the upcoming additions will only be available in its commercial distribution of the project.

Photo via Mathias Pastwa