Data is being created at an unprecedented speed, posing many infrastructure challenges for businesses of all types and sizes. One primary challenge is effective storage of a data archive — data that is not accessed frequently but must be retained for years or decades for purposes of compliance, auditing, or just in case there is a need for further processing or analysis.

Today’s storage hierarchy will need to evolve to provide more cost-effective solutions to address the “archival avalanche,” according to Fred Moore (pictured), president of consulting firm Horison Information Strategies and author of the paper “Reinventing Archival Storage: the Storage Hierarchy.”

“To do all this with traditional technologies is going to get very painful for a variety of reasons,” he said. “So, the stage is set for a new tier and a new technology to appear in the next five years. Fortunately, I’m actually working with somebody who is after this in a big way and in a different way.”

Moore spoke with Dave Vellante, host of theCUBE, SiliconANGLE Media’s livestreaming studio, for a digital CUBE Conversation. They discussed the challenges of archival storage, the renaissance of tape in this industry, the future hierarchy of storage and new technologies in this area.

Tiered hierarchical structure

The best data storage architecture involves a tiered hierarchical structure balancing performance and cost, according to Moore. It consists of four tiers ranging from ultra-high capacity and low-cost storage at the bottom of the hierarchy to very high levels of performance and functionality at the top.

At the bottom is the archive, which corresponds to more than 60% of all data and can reach 80% or more by 2024, being by far the largest storage class, according to Moore. Given this trajectory, the traditional storage hierarchy paradigm is going to need to disrupt itself, which Moore emphasized in his “Reinventing Archival Storage” paper.

“I think there is some hope here that we can redefine and really add a new tier down at the bottom,” Moore said. “It’s infinite storage. I mean, if you look at major league sports, the World Series and Super Bowl, that data will never be deleted; it’ll be here forever. It’ll be used periodically based on circumstances.”

The growing amount of data in the “archive tier” and in the emerging “deep archive” tier is fueling demand for a new storage paradigm that can drive costs to close to $1.0 per terabyte, according to Moore. “We’re going to have the race to a dollar a terabyte … here in 2025,” he added.

Multilayer photonic storage is promising

In the storage industry’s search for effective ways to keep data, the tape has regained space, especially because of the hyperscale data center markets.

“Those … are so big in terms of footprint and infrastructure that they can no longer keep everything on disk; it’s just economically not possible,” Moore explained. “The energy consumption per disk, the infrastructure costs, the frequency of taking a disc out every three, four or five years just for replacement has made it very difficult to do that. So hyperscale has gone to tape in a big way.”

Although the hyperscale market has done much to bring tape costs down, it still faces another problem, which is the magnitude of data centers. “We’re going to finish the year 2020 with about 570 hyperscale data centers. So it’s going right now around the world,” Moore pointed out. “Each one of these things is 350 to 400,000 square feet.”

Furthermore, the tape industry does not do a very effective job of marketing, and most people are still reluctant to use it, he added. “They have bad memories of tape in the past,” Moore said. “That’s pretty well gone away now … but most people don’t know that.”

In this scenario of need for innovation, the most promising technology for archive storage is a version of a multilayer photonic storage, according to Moore.

“I would say [it’s] sort of like optical but with data center class characteristics, multilayer recording capability on that. Random access, which tape doesn’t have,” he said. “That’s probably the one that you would want to take some look at going forward. The other [technologies] are highly speculative.”

Here’s the complete video interview:

Photo: SiliconANGLE