NEWS
NEWS
NEWS
LinkedIn open-sources its Groovy Gradle plugin for Hadoop
Hadoop has gotten itself another big-name contributor in the shape of business-oriented social network LinkedIn Corp., which has just made the LinkedIn Gradle Plugin for Apache Hadoop available on GitHub.
LinkedIn’s Gradle plugin was designed by the company to help it manage Hadoop jobs running on multiple frameworks, and its engineers have used it regularly while building data-driven apps.
LinkedIn engineer Alex Bain explained in a blog post that Gradle has been adopted as the company’s primary build system for the last couple of years.
“With Gradle, developers can easily extend the build system by defining their own plugins,” Bain wrote. “We developed the Hadoop plugin to help our Hadoop application developers more effectively build, test and deploy Hadoop applications. The plugin includes the Hadoop DSL, a domain-specific language for specifying jobs and workflows for Hadoop workflow managers like Azkaban and Apache Oozie.”
Gradle is written in the Groovy programming language, and its embedded DSL offers syntactic code constructs for working with jobs and workflows for Hadoop managers. This makes it possible for developers to use Groovy or Java anywhere throughout the DSL, Bain explained. The DSL simplifies the process of creating Azkaban or Oozie files, and because it’s statically complied, it can also be statically checked, he continued. This ensures developers can catch many of the common problems that crop up with workflow files at build time, as opposed to finding them later when running the actual Hadoop workflow.
LinkedIn is welcoming new contributors to the project, be it bug reports, documentation improvements, pull requests or any other ideas. Those interested can view documentation for the Gradle plugin here.
“The Hadoop Plugin and Hadoop DSL have been embraced as the standard way to develop Hadoop workflows at LinkedIn,” Bain said. “If you are writing Hadoop jobs using Gradle as your build system, you should definitely consider using the Hadoop Plugin. It will save you time and energy in developing your Hadoop workflows.”
Photo Credit: http://heatherbuckley.co.uk via Compfight cc
A message from John Furrier, co-founder of SiliconANGLE:
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
- 15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more
- 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network.
About SiliconANGLE Media
SiliconANGLE Media is a recognized leader in digital media innovation, uniting breakthrough technology, strategic insights and real-time audience engagement. As the parent company of SiliconANGLE, theCUBE Network, theCUBE Research, CUBE365, theCUBE AI and theCUBE SuperStudios — with flagship locations in Silicon Valley and the New York Stock Exchange — SiliconANGLE Media operates at the intersection of media, technology and AI.
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.
