Hortonworks Inc. today added improved support for Apache Kafka to its Hortonworks DataPlane Service, underlining the growing importance of streaming sources to its big-data customers.

Launched about a year ago, DPS is a cloud offering for combining all types of data in multiple data lakes and other data repositories. It’s intended to make it easier for companies to see the entire fabric of data flowing into their Hortonworks environments.

DPS is more of a framework than an application. The company has been adding plug-ins for such functions as lifecycle management, data analytics development and data governance and has said the popularity of the service is one of the factors underlying its recent positive earnings results.

The new Streams Messaging Manager is described as an open source operational monitoring and management tool that provides end-to-end visibility in enterprise Kafka environments. It’s essentially a management console for Kafka streams. “If you’re running multiple streams, this is a control panel where you can see how those streams are behaving,” said Scott Gnau (pictured), Hortonworks’ chief technology officer.

For example, he said, the management tools can be used to integrate Kafka streams with the Apache Atlas metadata management and governance framework for consistent metatagging of data. This enables Kafka data to be intermingled with static data, a useful feature in analytics. “We’re about enabling edge devices, and this makes it easier to help customers gain lineage about their infrastructure,” Gnau said.

Hortonworks said it’s addressing one of the most difficult aspects of managing streaming data, which is understanding where it came from and how it’s been used. Applying Apache Atlas provides “consistent coverage across the lifecycle of data,” Gnau said. “Metadata management has not been very done in the past, and with more data streaming in from devices, that context is important.”

Streams Messaging Manager can also be used to troubleshoot Kafka environments to identify bottlenecks, producer/consumer patterns and traffic flow. It enables filters to be applied to analyze stream dynamics between producers and consumers and gives users complete data lineage across multiple Kafka operations, producers and consumers via data flow visualizations integrated with Apache Atlas.

The company also announced a point release of the Hortonworks DataFlow streaming analytics platform that improves performance and provides better integration with the flagship Hortonworks Data Platform. The result is a single open-source platform that integrates governance, security and management across the entire data lifecycle from the edge to analytics, the company said.

Hortonworks has been high on HDF, saying that it was included in half of the $1 million-plus deals it closed during its fiscal first quarter. In addition to performance and stability improvements, the new release provides more granular control in a multi-tenant environment, support for Apache Hive 3.0. data warehousing and consistent operations, security and data governance across HDF and HDP instances.

Image: SiliconANGLE