Self-styled open-source-software-as-a-service company Instaclustr Pty Ltd. is adding to its lineup of managed services with the addition of the Apache Kafka publish-subscribe messaging system to its product suite.

Apache Kafka is open-source stream-processing software first developed by LinkedIn Corp. that aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. It’s an extremely popular tool used by thousands of enterprises that want to process streaming data in real-time.

But despite its popularity, Kafka is, like many open-source tools, rather difficult to get up and running, and even when companies manage it, they face a daily struggle to maintain the platform if they lack in-house expertise.

That’s where Instaclustr wants to help with its managed Kafka service. The company reckons it has both the talent and the experience to manage Kafka environments for its customers, boasting more than “20 million node hours under management” following a successful early-access program that saw a handful of its customers use the new service.

Instaclustr already has plenty of experience managing other open-source deployments for its customers, including Apache Cassandra, Apache Spark and Elastic Search, among others. The company said it uses the same robust provisioning and management patterns it provides for those platforms to deliver its Kafka managed service. It adds that its new service is SOC2-certified, which ensures all of the data it manages for its clients will be kept secure.

Apache Kafka is actually the missing piece of the puzzle as far as Instaclustr is concerned, as it aims to build a comprehensive platform of open-source technologies focused on the data layer, Chief Executive Peter Nichol told SiliconANGLE. He explained that the four main categories in this layer include storage, analytics, search and messaging. The first three are already covered with Cassandra, Spark and Elastic Search, he added, so messaging was the last component.

“We chose Kafka because it is the most widely used open source messaging technology, and it is quite early in the adoption cycle,” Nichol said. “The opportunity for Instaclustr is to become the leading managed Kakfa provider, as well as the only company providing a fully managed platform across all main data layer categories.”

Instaclustr said its Kafka managed service guarantees high throughput and availability, zero downtime and also integrations with external stream processing layers such as Spark Streaming. The company also provides built-in monitoring services so its customers can understand their data streams in real-time.

The initial focus will be on selling Kafka to Instaclustr’s 100-plus existing enterprise customers, Nichol said, though he added that the service would be useful for any customers building applications that need to scale and where they have need for a messaging system to optimize the overall architecture.

“The ability to manage all these data layer technologies on one platform provides the added advantage of a single vendor on a highly optimized integrated platform,” Nichol said. “Customers who want to run this themselves will have to find experts and build a team which needs to be able to optimize the applications and architecture, and be available 24x7x365 in case anything goes wrong, which is expensive and takes time to build. By outsourcing this to Instaclustr, they can do it much cheaper.”

Instaclustr’s Apache Kafka integrations (Image: Instaclustr)

Those customers profess to be happy with what Instaclustr is offering.

“We see Apache Kafka as a core capability for our architectural strategy as we scale our business,” said Glen McRae, chief technology officer of Lendi Pty Ltd., an Australian home loan platform that used Instaclustr’s Kafka-as-a-Service during the Early Access Program. “Getting set up with Instaclustr’s Kafka service was easy and significantly accelerated our timelines. Instaclustr consulting services were also instrumental in helping us understand how to properly use Kafka in our architecture.”

The new service, available immediately, can be deployed on a range of public cloud platforms including Amazon Web Services, Google Cloud Platform and Microsoft Azure.

Main image: Instaclustr/Facebook