UPDATED 12:39 EDT / DECEMBER 15 2015


GCHQ open-sources key snooping tool

While the work of the U.K’s Government Communications Headquarters (GCHQ) is a closely guarded secret known only to select personnel, the internally-developed software powering its intelligence gathering operations is now publicly available for anyone to download. The source code of the aptly-named Gaffer database was released to GitHub yesterday under a free license to make it easier for developers to store complicated hierarchical records that can’t easily fit into traditional schemas.

A relevant example would be phone records, which need to be organized in a format that enables fast access not only to caller identities but also related details such as the date and duration of a particular conversation if they are to be properly analyzed. Maintaining a sufficiently high standard of availability, however, becomes increasingly difficult over time as new information is added and existing entries are changed. Gaffer minimizes clutter by making it possible to neatly group all of the data associated with an individual in a logical construct called a “node” with multiple “edges” each containing a different item.

That way, a GCHQ analyst can quickly view the activities of a suspect or search for details that might link them to co-conspirators without having to waste time manually filtering the information first. Gaffer further improves response times with a unique updating mechanism that makes it possible to add and change edges without nearly as much overhead as other graph databases, a capability that should lend itself particularly well to large deployments with a constant stream of new information to process. It’s one of the two main features that set the project apart, with the other being the fact that all records are kept by default in Apache Accumulo.

The likewise open key-value store is arguably the most secure in its category thanks to a granular authorization mechanism that allows administrators to restrict access all the way down to individual data points. The functionality reflects the strict requirements that the original creators of Accumulo were tasked to address for their employer, the National Security Agency (NSA), which has been investing even more than its U.K counterpart in forging ties with the open-source community. The spy agency most recently saw a team of former staffers who spun off to commercialize the Nifi network orchestration framework, another internal project, join Hortonworks Inc. through an acquisition.

Image via Geralt

A message from John Furrier, co-founder of SiliconANGLE:

Show your support for our mission by joining our Cube Club and Cube Event Community of experts. Join the community that includes Amazon Web Services and Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger and many more luminaries and experts.

Join Our Community 

We are holding our third cloud startup showcase on Sept. 22. Click here to join the free and open Startup Showcase event.

“TheCUBE is part of re:Invent, you know, you guys really are a part of the event and we really appreciate your coming here and I know people appreciate the content you create as well” – Andy Jassy

We really want to hear from you, and we’re looking forward to seeing you at the event and in theCUBE Club.

Click here to join the free and open Startup Showcase event.