Hadoop security is fragmented and immature, and until this changes few Hadoop-based projects will move into production, writes Wikibon Big Data Analyst Jeff Kelly. Wikibon’s Big Data Analytics Adoption Survey, 2014-2015, found that security concerns were the second largest obstacle to moving Hadoop into production, just behind backup and recovery issues. As a result, the Big Data field is becoming littered with projects that were abandoned after a successful test phase when the practitioners discovered that they could not provide the data security required for production implementation. If the Hadoop market is to grow, vendors and Apache need to work to together to develop an open security platform that their individual projects can plug into to provide the security and data management services companies need and that, in many cases, regulations require.

The problem goes back to Hadoop’s creation in Yahoo. It was originally conceived as a solution to a single problem: cataloging the World Wide Web for Yahoo search. Since all the data being collected was public and the number of users restricted to a small group inside Yahoo, data security was not an issue. When Hadoop entered the Open Source community it lacked even the most basic security capabilities such as user authentication and access controls. As Hadoop matured and was generalized to handle multiple use cases, some basic security components were added, but no underlying security platform has emerged. As a result these capabilities, developed independently by several Open Source project teams and vendors, are difficult to integrate.

Another problem hampering the creation of a mature security environment is that Hadoop is not a single technology, Kelly writes. Rather it is a collection of components that can be combined in various configurations to meet the specific needs of a use case. Hadoop also presents some new security challenges that were not seen in relational databases and data warehouses. For example, YARN (Yet Another Resource Negotiator) allows a single Hadoop cluster to support multiple applications. These often have different access and security needs, requiring a security platform with greater flexibility than is required for a relational environment.

If Hadoop is to fulfill its potential as a general-purpose Big Data platform in enterprises, it needs a comprehensive security platform based on open standards that can unify the various security tools from vendors and the Open Source community. This will benefit the vendors and practitioners alike. Without it, the Hadoop market will never grow beyond test cases and small, isolated point solutions used by a small groups of practitioners.

Kelly’s full analysis, “New Approaches Required for Comprehensive Hadoop Security”  is available without charge on the new Wikibon Premium site.

photo credit: Markus Wi via photopin cc