UPDATED 14:37 EDT / AUGUST 31 2016

NEWS

Facebook’s new compression algorithm could be a game-changer for data storage

Compression is front and center at Facebook’s annual @Scale conference in San Jose. The social networking giant open-sourced a homegrown algorithm called Zstandard at the event today that is described as faster, more efficient and easier to use than many of the most popular alternatives in use today.

One of the rival compression technologies that Facebook singled out in its release announcement is Zlib, a free library originally created over 20 years ago that is employed by Git, PostreSQL and thousands of other applications. The project is based on an algorithm known as DEFLATE that in turn enjoys even broader adoption in the software world. According to the social networking giant, Zstandard sets itself apart by taking advantage of compression techniques and CPU capabilities that weren’t available two decades ago.

Facebook says that its algorithm can provide a slightly higher compression rate than Zlib while working five times faster when shrinking files and twice as fast when decompressing them. Moreover, the algorithm provides the ability to trade data density off for increased performance and vice versa depending on the use case with 22 levels of granularity. As a result, a company that wants to make the most out of its storage equipment could dial compression rates all the way up, while a firm interested in speeding up data management operations could do the opposite and focus on maiming performance. But most of the organizations that will end up adopting the algorithm can be expected to fall somewhere in between.

Zstandard made its debut alongside a new open-source called MyRocks that makes it possible to use Facebook’s speedy RocksDB storage engine with the popular MySQL database. The social networking giant claims that its internal implementation has helped reduce storage requirements buy up to 50 percent in some cases, a massive decrease that should get the technology a lot of attention in the database space.

The code for MyRocks and Zstandard is available immediately on Faceboook’s GitHub page.

Image from Facebook

A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU