Hyper-Scale Databases Require New Approaches to Infrastructure, Data Protection, say Wikibon Analysts

An increasing number of companies are watching their central databases grow through the multi-petabytes towards the exabyte category writes Wikibon Analyst John McArthur in “Hyperscale Storage: Not if, When.” The Wikibon alert headlines the latest Peer Incite newsletter triggered by the January 22 Wikibon Peer Incite meeting on hyperscaling  with Cleversafe VP Russ Kennedy. Databases at this scale have been the province of large government agencies and dominant cloud companies like Facebook, YouTube, and Shutterfly. However, McArthur predicts, increasing numbers of more traditional companies are going to see their data grow to the multi-petabyte range as video, still images, and audio recording become more common parts of business communications.

Databases of this size make traditional RAID and replication data protection strategies impractical. It is just too expensive to maintain multiple copies of a multi-petabyte database. Traditional customized hardware becomes impractically expensive, and with huge numbers of disk spinning, failures become increasingly common. This reality has driven Facebook and other online companies to develop a new IT infrastructure, including storage through the Open Compute Project, he writes.

As a result, writes Wikibon Analyst Scott Lowe, “CIOS must watch hyperscale trends and jump when ready.” “CIOs must maintain and open mind as times change and new opportunities and offerings make their way to market.” Lessons from the hyperscale experience can help CIOs improve their services, decrease costs, increase availability and increase capacity and capability as demands grow, even if those demands do not reach the petabyte range. “In 1999 who would have thought that we could run 100 servers worth of workload on just five physical hosts with room to spare?”

Hyperscale databases demand a new approach to data protection, adds Wikibon CTO David Floyer in Software-led Storage: Hyperscale Storage Requirements. “The only technology available to address these issues is erasure coding, the ability to use compute cycles to split up and transform the data into n slices with the ability to restore it with only m slices (n>m).” These slices can be distributed geographically or locally. Metadata becomes vital in this approach as it is the key to finding the slices and rebuilding the file correctly.

Kennedy contributes a non-technical explanation of what erasure coding and information dispersal, as Cleversafe implements them, are. “Erasure codes use advanced math to create ‘extra data’ that allows a user to need only a subset of data to recreate” the file. Information dispersal eliminates the need of replication for data protection by distributing the data slices among multiple systems across a large geographic area. This allows the files to survive multiple, simultaneous failures or, if the geographic area is large enough, even a regional disaster.

Cleversafe’s information dispersal system uses a REST protocol-based approach, explains Wikibon Analyst Gary McFadden. In such systems, object-based storage has several advantages. However, erasure coding and information dispersal comes with a performance tax. It takes time to reassemble an object. Under ideal conditions this means a millisecond-scale response time, rather than the microsecond response required by high-performance, transactional systems. This means that vendors need to determine the vertical markets in which they wish to compete and examine the requirements of specific applications in these markets before deciding on the architecture on which they will build their systems. And users need to be conscious of the trade-offs when matching an application to a technology.

Facebook provided a deep dive into its hyperscale architecture at the Open Computing Summit Winter 2013 event earlier in January, writes Wikibon’s Stuart Miniman in his examination of “Rack Level Architectures and Hyperscale Operations.” Facebook uses five standardized homogeneous building blocks using rack architectures to construct its infrastructure. Everything is done on massive scale, with each rack holding 20-40 servers.

Facebook calls these servers “vanity free,” meaning they are not a name bran or even a whitebox system but are bare-bones hardware. All services are moved to an independent software layer rather than being built in to the hardware. Facebook considers flash storage critical and is Fusion-io’s largest customer, with disk used to storage of massive amounts of less active data. Its architectural solution, he says, is the “Distributed Rack,” the “opposite of virtualization” that pushes servers as hard as they can go. Hardware failures are handled by shutting down the server and replacing racks when enough of their servers fail.

So what does the the hyperscale architecture eliminate? For a start, maintenance contracts and repair expenses and detailed vendor management. But one of the biggest savings, writes Wikibon Analyst John McArthur, is “Eliminating Backup and Replication in Hyper-Scale Storage.” This is absolutely necessary. Companies simply cannot afford to maintain multiple copies of multi-petabyte databases.

Wikibon Peer Incite meetings and the accompanying newsletter are free to Wikibon members. They are scheduled for Tuesdays, usually at 12:00 ET, and are lead by an expert in or user of an advanced technology designed to meet 21st Century data issues. They are available both as an interactive audio conference and one-way video conference. Interested professionals are invited to register for free membership at the Wikibon site.