The big data era is upon us and virtually every storage announcement will be assessed in this context going forward. Traditionally, the emphasis on data has been about the difficulty of managing explosive growth. The industry in our view is undergoing a ‘bit flip’ where data is becoming a strategic asset.

We’ve heard this story before however the difference this time around is that new technologies actually exist to take advantage of massive amounts of data and turn it into an information asset. The Hadoop movement is an example where organizations are using new techniques to manage massive amounts of data. At Hadoop World 2010 in October, for instance, the average reported database size of attendees was a whopping 115TB.

This is game-changing. Analytics is an obvious example where firms in financial services, health care and government are taking advantage of the ability to process, analyze and act upon data—not just sampling data but entire data sets; in near real time. Beyond analytics, companies are monetizing big data, especially Web firms such as Twitter, Facebook, Zynga and many others.

The other major trend in the context of big data is, “Little Data.” Specifically, the typical home in a developed country contains 4-5 microprocessors and up to 100 microcontrollers. In addition, according to various market research firms, more than one billion cell phones shipped in 2010. The prediction for 2011 is it will be the year of the smart phone with Apple shipping more than 100M iPhones and several vendors bringing sub $100 Android handsets to the market. These smart devices are becoming a connected set of sensors creating huge volumes of data in the form of video, text messages, social gestural data and the like.

Wikibon will be watching these trends closely. Our fundamental premise is that every organization is creating some form of big data and this data will have increasing value to all organizations. Of particular interest to our community are EMC’s recent acquisitions of Greenplum and Isilon, both directly involved in the big data movement. How EMC manages these acquisitions and other moves it makes in this space will determine its position in the big data arms race and also guide many decisions at IT organizations.

There are three relevant points related to big data storage that bear watching:

1. Big data will require infrastructure that is simple, efficient and inexpensive to manage, yet provides access to multiple platforms and data sources simultaneously. As such, virtualization and storage optimization techniques will be required to exploit big data especially where legacy infrastructure is involved;
2. The big data trend will heavily utilize flash across the storage hierarchy to deliver real- or near-real-time performance for users. This will be seen in the disk system and at the memory level of the hierarchy;
3. Real-time, big data analysis will not be done from tape-based systems; increasingly disk-based architectures will be the norm for analyzing massive archives.

Where Does Cloud Fit with Unified Storage?

The industry is going through what Nicholas Carr calls “The Big Switch”—a move from traditional data centers filled with equipment to a more agile, flexible, responsive, self-service set of offerings that have a transparent cost to the end user: cloud computing. Public cloud vendors provide IT services with scalable solutions and dynamic costs similar to an electrical utility. Unlike electricity, there is a significant difference to an IT department related to the security, management, performance and control of an outsourced solution. Traditional data center vendors have created private cloud offerings, which provide many of the same benefits as public clouds through the use of virtualization and optimized, converged infrastructure. The goal is not necessarily to have the same cost as a public cloud, but to be able to provide the business transparency so that comparisons of the requirements and costs can be made allowing for informed decisions that balance risk and reward.

Amazingly, EMC, a storage company, is the central player in this transition due to its acquisition of VMware in 2003. While EMC knew that VMware technology shifted intelligence and dollars away from servers, no one was predicting that hypervisors would become the new OS of the enterprise. Virtualization has had the largest impact on architectural design for servers, network and storage in the last decade. Not only are the individual technologies affected, but the inherent abstraction of virtualization causes collisions of domains which require that customers and vendors break-down the walls between infrastructure silos to allow for the deployment of services rather than just boxes. Unified storage is a natural fit for virtualization.

Specifically, hypervisor deployments have been the rising tide that has lifted both block and file networked storage, with customers installing a mix of NFS, iSCSI and FC/FCoE (a majority of customers are running multiple network protocols). Customers require unified storage to be a flexible, multi-purpose building block that can be used across a broad spectrum of applications (including, but not limited to, virtualized environments). While EMC has offered some type of unified storage option for years (e.g., block behind a NAS head), the release of the VNX family creates an integrated and more competitive offering.

As virtualization has matured, it is moving beyond a tool to increase server utilization rates towards enabling IT to be consumed as a service (ITaaS). For ITaaS to be realized, solutions need to be more than just simplified, they also need to be automated. Simplification and automation touch many aspects of the storage solution including automated tiering of storage (SSD/HDD) where the system self-regulates the resources. On balance, smart storage systems are far more efficient at managing data placement than humans. Every vendor in the industry has been working on simplification of storage management over the past decade, mostly with incremental improvements to existing legacy platforms. One of the more interesting pieces of EMC’s new entry level VNXe solution is that the management of the system was built from the ground up, similar to the startups that emerged last decade. This means that not only could the latest design tools be used, but rather than taking storage language and simplifying it (e.g., simplifying LUN management), the VNXe design starts from the application point of view. This resulted in a solution that not only an IT generalist, but even a business person working with Exchange or SQL could utilize. In short, the VNXe provides an ITaaS solution for the underserved SMB and remote office marketplaces.

How does this Announcement Impact the Data Center as a Whole?

Along with the growth of VMware, EMC has expanded its reach into the data center as a whole. The introduction of the VCE joint venture with Cisco, and the Atmos initiative have given EMC integrated stack offerings that reflect the trends of simplification, unification and integration of former IT components into larger building blocks.

Wikibon has looked at these trends in detail and has modeled their impact on data center budgets. The main conclusion of this research is that simplification, unification and integration are very strong trends within IT, and will have a profound impact on IT budgets. The major impacts of these trends are:

1. The cost of supporting existing traditional applications will be reduced by 39% over five years;
2. These cost savings will go into additional external services, including cloud services;
3. The percentage of budget spend on external services will rise from 14% to 53% over the five year period.

A summary of the transformation of IT budgets is shown in Figure 1. The additional benefits of this trend are:

1. The improvement in time to implement new projects and applications will be significantly reduced;
2. The business risks of new projects will be reduced from later and faster implementation, and the use of external services.

Figure 1: Transformation of IT Budget as a Result of Technology Simplification, Unification & Integration. Source: Wikibon 2011

The availability of simplified stacks will come from large IT vendors who will put together their own integrated stacks, and coalitions of vendors who will join together to create integrated stacks based on simplified components. EMC is simplifying and unifying its products, has key components of the stack with storage and the hypervisor, and is partnering with other IT vendors to bring this vision to market.

Action Item: EMC’s big announcement signals a new period of change in the storage business. While much of the announcement underscores the need to make near-term tactical decisions (e.g., which midrange platform to deploy and how to get the best price), practitioners should use this announcement as an opportunity to begin evolving toward ITaaS, understanding how to leverage big data and where EMC fits in the broader spectrum of infrastructure players required to deliver on this vision.