UPDATED 15:20 EDT / AUGUST 06 2013

Sqrrl CTO Explains the Secret Behind Accumulo’s Near Real-Time Performance

Accumulo logoAdam Fuchs, the co-creator of Accumulo and the founding CTO of Sqrrl, gave an exclusive whiteboard explanation of his company’s technology on SiliconANGLE’s theCube.

Accumulo is a highly secure, disk-based key-value store that combines Google’s BigTable storage system with innovations Fuchs and his colleagues developed as part of their work for the NSA. It utilizes a data structure known as the log-structured merge tree to rapidly sort randomly ordered key-value pairs using as little disk space as possible.

Fuchs says that the platform partitions incoming data into containers called tablets. Inside these tablets, data is fed into an in-memory map and then replicated onto HDFS to maximize availability. The latter process involves buffering information into sequential streams that are flushed to disk as soon as they “fill up.”

Accumulo merges tablets into a unified stream of key-value pairs in order to make data easily accessible for users. The amount of latency is proportional to the number of tablets, Fuchs adds, but it’s greatly reduced by the major compaction that the platform carries out in the background. This operation integrates data into a globally sorted file that is ready to go through iterator keys.

Iterator keys are operators that can perform versioning and filtering, aggregation and function application. Since these tasks are carried out in the stream, applications don’t have to request datasets multiple times and random I/O is further reduced.

Fuchs concludes the session by highlighting that Accumulo enables users to optimize their deployments depending on how write- and read-intensive their workloads happen to be.

Click the video below for the full presentation.

 


A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU