Microsoft Corp.-owned professional networking site LinkedIn is open-sourcing another new project. Dynamometer is a new tool is focused around stress-testing large Hadoop big-data deployments without using massive amounts of infrastructure.

In a blog post today, LinkedIn senior software engineer Erik Krogen said the company built Dynamometer back in 2015 in response to an issue that cropped up when it attempted to boost performance by adding 500 machines to its Hadoop Distributed File System cluster. Hadoop is a software framework for managing large amounts of data across computer systems.

That deployment initially went well, only for a bug to show up later that caused some jobs targeting the new cluster to shut down. Dynamometer would have prevented that from happening. It’s a tool that can be used to simulate large-scale HDFS clusters and test their performance using just 5 percent of the underlying infrastructure. That helps developers test software at scale while saving on the cost of actually provisioning the machines it runs on.

With Dynamometer, Krogen said, information technology teams can test production workloads and ensure they’ll be able to cope with any changes to their Hadoop clusters. LinkedIn used the tool to test the migration of its clusters to Hadoop version 2.6 from Hadoop 2.3, an operation that required adjusting certain parameters of those clusters to avoid taking a performance hit.

In his blog post, Krogen explained that Dynamometer is designed for those running large-scale Hadoop deployments, as well as those who propose changes to the core Hadoop project and want to ensure any new features won’t hurt performance.

Krogen added that he hopes Dynamometer will eventually become the standard tool for testing new releases of Hadoop, as well as any new code changes that are added between releases.

Image: LinkedIn