MapR Technologies Inc. is tackling Hadoop’s administrative complexity with the announcement today of a new campaign it calls the Spyglass Initiative. It’s also taking steps to make its own product updates simpler to manage.

The company is making a long-term commitment to deliver a series of enhancements to its Converged Data Platform that provide improved visibility into big data deployments with customizable reports and dashboards, along with third-party tool integration via application program interfaces (APIs).

Hadoop clusters can dense become as they grow, making it difficult for administrators to pinpoint bottlenecks and outages. “As you grow to massive scale, traditional monitoring doesn’t  work,” said Anoop Dawar, vice president of product management. “You need tools that scale to thousands of volumes and millions of objects.”

MapR is open-sourcing the technology it’s developing for the initiative, hoping that licensing plus open APIs will encourage customers and third-party vendors to integrate their existing administrative platforms and share custom dashboards with a community. In line with the latter goal, the company is also launching a section of its Converge Community where customers can swap and import each other’s dashboards.

The MapR monitoring architecture uses the open-source Elasticsearch for discovery and its sister software Kibana for log search, OpenTSDB for log management and Grafana for visualization. OpenTSDB is optimized for storing time-series data in Internet of things (IoT) scenarios.

Customizable dashboards enable users to create their own views of the literally hundreds of metrics the system provides and to share those views with others using JSON specification templates. Customers may also choose to use their own log analytics framework if they prefer, Dawar said.

The first phase of the program will cover monitoring of nodes/infrastructure, YARN/MapReduce, cluster space utilization and service daemons.

Simplifying the ecosystem

MapR also said it will begin to decouple delivery of core Converged Data Platform components from other ecosystem projects and put them on different schedules to balance customers interests in the latest and greatest technology with their need for upgrade sanity.

“Ecosystem Packs” are described as an enhanced program for delivering the latest open source project versions while ensuring interoperability. Instead of monthly project updates, customers will henceforth get upgrades to the core platform on a semi-annual basis and updates to related projects quarterly.

Core components, which include the MapR Data Platform, YARN, Hadoop common components, MapR-FS, MapR-DB and MapR-Streams, will undergo more-stringent testing for reliability and compatibility. Ecosystem components (updated quarterly) include Apache Drill, Apache Sqoop2, Apache Hive, Hue, Apache Spark, Apache Flume, Apache Pig, Apache Storm, Apache Oozie, Apache Mahout, Apache Sqoop1, Apache Myriad, Impala and Apache Sentry.

“Today, customers are asked to migrate everything to get that version of Hive they want,” Dawar said. “With this approach you can update the ecosystem pack to get full functionality.” Customers can also upgrade core release without upgrading the ecosystem packs.

Also new in the MapR Platform are performance and JSON enhancements to MapR-DB.  Advanced multi-master JSON replication provides disaster recovery for JSON documents as well as a global view of enterprise-wide data in local deployments. The next phase of optimizations on solid state drives (SSDs) delivers faster parallel processing of data through MapR-DB.

The announced enhancements are included in existing MapR subscription fees.