Traditional approaches to data backup have become outdated for large, scale-out infrastructures. Whether government agencies, cloud service providers, or Internet giants (e.g. Facebook, Twitter), storage has quickly become long overdue for an overhaul. Hyperscale computing is the infrastructure and provisioning needed to effectively scale from a few servers to thousands of servers in a distributed computing environment.

Hyperscale is a growing trend that necessitates a new method of protecting data. Simplified storage approaches based on object stores, combined with erasure coding as a means of protecting large quantities of data will dramatically lower storage costs. A new storage paradigm is forming, and flash will play an increasingly active and important role in how we house metadata and enable real-time analytics to be performed on large data repositories.

A quick data 101: Data today is backed up by making copies. Whether snapshots or physical, data is copied in order to provide recovery options if necessary. Consequently, the sheer amount of data (aka Big Data) is growing and compounding at such a rate that the old way of data storage and recovery simply isn’t feasible anymore. There are 3 key questions to how hyperscale will help with software-led storage:

• How are petabytes or exabytes of data backed up?
• How are petabytes or exabytes of data restored?
• How are petabytes or exabytes of data accessed?

Wikibon’s CTO David Floyer says that at the root of the problem is a simple case of Moore’s Law: as costs decline and capacities rise, elapsed time only becomes more problematic because access times and transfer rates from magnetic media barely improve. As data volume continues to grow at a rate of 2-3x, backup and recovery of data cannot efficiently or effectively scale at the same rate. What is left is increased costs and increased probability of lost data.

Hyperscale computing is proving that traditional methods of reading hyperscale volumes of data from disk to find and analyze patterns are no longer viable. Hyperscale storage requirements are leading the charge for this new medium of software-led storage based on, bluntly put, need. The most important challenge of reading hyperscale volumes of data involve the metadata. How do you create metadata for each object that describes where it is, what it is, when it was stored, and how it is related to other objects? The metadata needs to be quickly accessed too. In order for very high accessibility speeds, it has to be held centrally (mainly in non-volatile memory).

By creating metadata around hyperscale volumes of data where the the object store can be used for multiple purposes, you’ve in essence created a single store. This store can contain data warehouses, backups, and multiple archives. The infrastructure and provisioning of hyperscale computing should improve both the Big Data and Cloud environments. Spinning up to scale faster and easier every step of the way.