Startup Streamlio Inc. is betting that organizations are ready for real-time streaming architectures to process their basic data needs, and now it has brought three of the latest open-source technologies to bear on the process.

The company’s new real-time analytics suite incorporates the Apache Pulsar publish-and-subscribe engine with Heron, a real-time, distributed, fault-tolerant stream processing engine originally developed at Twitter Inc. and Apache BookKeeper, a low-latency storage service designed for real-time workloads. The combination is “the only enterprise-grade messaging solution optimized for streaming and storage,” said Streamlio co-founder and Chief Executive Lewis Kaneshiro.

Like many open-source-based companies, Streamlio is betting that organizations would rather purchase a pre-integrated package than deal with the grunt work of mashing together multiple open-source technologies. Kaneshiro, a former data scientist at Shazam Entertainment Ltd. and Wall Street investment firms, teamed up with Heron co-creator Karthik Ramasamy and a group of former Twitter and Yahoo Inc. engineers who were instrumental in the development of the platforms Streamlio uses. They’ve raised $7.5 million in an oversubscribed series A round led by Lightspeed Venture Partners.

The company is betting that enterprises are ready to move large parts of their operations from batch to stream processing as real-time analytics becomes a competitive necessity and trends like the Internet of Things take hold. Ingesting and processing large volumes of streaming data requires specialized software and a higher degree of automation than is typically applied in batch operations. “We believe enterprises are increasingly event-driven with no humans in the loop,” Kaneshiro said. “If the world is moving to real-time and data is continuously in flight, then applications need to be contextual to that environment.”

Up-and-comers

Streamlio chose to place its bets on a clutch of newer and lesser-known open-source technologies. It chose Apache Pulsar rather than the more widely adopted Apache Kafka because “Kafka wasn’t developed to enterprise-grade requirements,” Kaneshiro said. Pulsar development began after Kafka was open-sourced with the aim of addressing those deficiencies. It has been running in production at Yahoo for more than three years, anchoring such applications as mail, finance and sports, according to Kaneshiro.

Heron was developed by Twitter as a successor to Apache Storm for use in real-time analytics. It’s backward-compatible with Storm application program interfaces and has numerous advantages in areas like usability, ease of development and administration.

Apache BookKeeper was adopted because “today’s streaming analytics systems don’t have storage built in,” Ramasamy said. “Ours is tightly integrated, with the underlying data store being BookKeeper.” BookKeeper is a multiserver replicated service designed for reliability. It can tolerate crashes, corrupted data and discarded data using a network of servers that store and replicate log streams for quick recovery. BookKeeper has been the storage layer underpinning enterprise services at Twitter and Yahoo for the past four years, and is particularly well-suited for publish-and-subscribe applications using message brokers such as Pulsar.

In keeping with the common open-source business model, Streamlio will offer a free community version and a paid enterprise version with enhanced security, metadata management and user interface. Pricing will be per node, but  specifics haven’t been determined yet, Kaneshiro said.

Image: Flickr CC