International Data Corp. estimates that the total volume of digital information in the world will balloon from 33 zettabytes, or trillion gigabytes, today to 175 zettabytes in 2025. This rapid growth is being felt particularly strongly by cloud providers such as Microsoft Corp., which host not just their own information but also that of countless other organizations.

To reduce the strain on its infrastructure, the company has developed a cutting-edge system for compressing data. Microsoft this morning released the specifications for the system under an open-source project dubbed Zipline.

The company touts its technology as considerably more powerful than the compression software commonly used in the industry today. Kushagra Vaid, the general manager of the Azure Hardware Infrastructure team, used the popular Zlib tool as a reference point in the blog post announcing Zipline.

Zlib is an industry-standard compression library that can be found in the Linux kernel, iOS and other foundational software platforms. Vaid wrote that Zipline provides data compression rates up to twice as high as high those offered by Zlib. Moreover, the system is described as capable of doing so while providing better throughput and lower latency than several other popular compression tools.

In practice, this means that Zipline can shrink workloads to just a fraction of their size. Microsoft claims that the system compresses the storage footprint of application data stored on Azure by as much as 92 percent. Zipline provides even greater reductions for other types of data such as machine-generated logs from connected devices.

Microsoft is open-sourcing the algorithm that the system uses to perform compression, as well as the specifications for the custom hardware on which the algorithm is designed to run. These specifications include the low-level register transfer language in which Zipline expresses data operations.

“Over time, we anticipate Project Zipline compression technology will make its way into several market segments and usage models such as network data processing, smart SSDs, archival systems, cloud appliances, general purpose microprocessor, IoT and edge devices,” Microsoft’s Vaid wrote.

Zipline is not the first component of Azure that the company has contributed to the open-source community. Previously, Microsoft Corp. released the code for an artificial intelligence engine that supports some of the cloud platform’s services. It has also shared the schematics for a homegrown chip called Cerberus that can protect a server’s firmware from tampering attempts.

Photo: Microsoft