Hadoop has gotten itself another big-name contributor in the shape of business-oriented social network LinkedIn Corp., which has just made the LinkedIn Gradle Plugin for Apache Hadoop available on GitHub.

LinkedIn’s Gradle plugin was designed by the company to help it manage Hadoop jobs running on multiple frameworks, and its engineers have used it regularly while building data-driven apps.

LinkedIn engineer Alex Bain explained in a blog post that Gradle has been adopted as the company’s primary build system for the last couple of years.

“With Gradle, developers can easily extend the build system by defining their own plugins,” Bain wrote. “We developed the Hadoop plugin to help our Hadoop application developers more effectively build, test and deploy Hadoop applications. The plugin includes the Hadoop DSL, a domain-specific language for specifying jobs and workflows for Hadoop workflow managers like Azkaban and Apache Oozie.”

Gradle is written in the Groovy programming language, and its embedded DSL offers syntactic code constructs for working with jobs and workflows for Hadoop managers. This makes it possible for developers to use Groovy or Java anywhere throughout the DSL, Bain explained. The DSL simplifies the process of creating Azkaban or Oozie files, and because it’s statically complied, it can also be statically checked, he continued. This ensures developers can catch many of the common problems that crop up with workflow files at build time, as opposed to finding them later when running the actual Hadoop workflow.

LinkedIn is welcoming new contributors to the project, be it bug reports, documentation improvements, pull requests or any other ideas. Those interested can view documentation for the Gradle plugin here.

“The Hadoop Plugin and Hadoop DSL have been embraced as the standard way to develop Hadoop workflows at LinkedIn,” Bain said. “If you are writing Hadoop jobs using Gradle as your build system, you should definitely consider using the Hadoop Plugin. It will save you time and energy in developing your Hadoop workflows.”

Photo Credit: http://heatherbuckley.co.uk via Compfight cc