While the skils shortage is often cited as being one of the biggest obstacles in the way of Hadoop’s widespread enterprise adoption, there are numerous other challenges that must also be overcome. Some of these include being able to support multiple users on a single cluster and leveraging systems that handle diverse workloads. Another challenge involves the allocation of costs and resources that need to be spread across multiple departments.

These are the obstacles that Pepperdata Inc., which specializes in real-time cluster optimization, has set out to overcome with a new solution that’s able to accurately measure the costs of Hadoop workloads running across distributed systems. Pepperdata’s new“Chargeback” feature means IT teams can now see exactly how much capacity each user or workload requires, and accurately determine the costs to each department that shares a centralized, multi-tenant Hadoop platform.

According to Sean Suchter, Co-founder and CEO, Pepperdata has in one fell swoop solved three of the most pressing issues blocking the way to Hadoop’s widespread enterprise adoption – the collective need for predictability, visibility and utilization.

“We’ve reached an inflection point where more Hadoop jobs from various departments are stomping on each other,” Suchter said in a statement. “Organizations can’t count on those systems to run efficiently without the tools to optimize cluster performance at scale. That’s the software gap in core Hadoop that we’re solving for.”

Pepperdata also thinks it can boost the efficiency of shared Hadoop clusters. The company says that wherever Hadoop is deployed, it’s rapidly consuming a larger share of IT budgets. This means it’s of vital importance that Hadoop is made as efficient as possible, to ensure that one department’s heavy usage doesn’t cause a slow down in other departments. To that end, the “detailed visibility” provided by Pepperdata’s Chargeback feature will be critical to those enterprises that want to offer “internal Hadoop-as-a-service”.

This visibility is derived from the Chargeback feature’s ability to track Hadoop usage in real-time for each user, department or workflow.

It’s a capability that Pepperdata says will grow in importance as IT trends towards more multi-tenant deployments. Most enterprises typically begin with small Hadoop clusters of a dozen or so nodes for prototype projects, but as these projects go into production, the clusters can expand into thousands of nodes.

“Organizations need to be able to better manage these environments and measure and cost out the amount of resources consumed,” said Nik Rouda, Senior Analyst at Enterprise Strategy Group. “The level of visibility and granularity that Pepperdata is delivering with its chargeback feature is incredibly valuable for organizations looking to integrate Hadoop into their standard business processes and put in place internal Hadoop as a service offerings.”

Image credit: tookapic via pixabay.com