We are seeing a major transformation in the cloud and data market that is forcing traditional enterprise infrastructures to rethink their “data strategies” or “data warehouse” plans.

This “Big Data” explosion” is being driven by Internet giants such as Google, Yahoo, Facebook, Twitter, Bit.ly, LinkedIn, etc., and startups such as Cloudera, ClickFox, Membase, Karmasphere, and many others. The big data movement is spilling into mainstream enterprises (e.g. BofA, GE, ComScore and the U.S. Army) and is bringing critical organizational questions to CIOs and CTOs related to how best exploit this emerging trend.

This past week our research partners at Wikibon, led by Dave Vellante discussed this themers at during a video theCube broadcast called Wikibon Peer Incite Research Meeting on big data.

Wikibon and their community  presented their key findings and conclusions from a recent study of 40 IT practitioners, data scientists, academics, business leaders and technology “alpha geeks.”

Wikibon presents the finding in their innovative wiki-style briefing package here.

Highlights include:

The Data Warehouse Trend Angle

• Virtually all practitioners indicated that they were I/O constrained. Users are looking to flash storage and columnar compression to ease performance bottlenecks and lower costs by allowing more data to be stored longer.
• While big data and traditional data warehousing are quite different, practitioners indicate that data warehouses are already feeding big data apps, and over time they believe that big data apps will in turn feed traditional data warehouses.

The Oracle Angle

• The single SKU approach of an appliance is alluring because it minimizes complexity, speeds deployment, and simplifies patches. However some practitioners questioned the merits the Exadata single SKU approach because of concerns over accommodating diverse operating systems beyond Linux. In addition, to scale resources in a single SKU model they need to buy a full appliance (for example to scale storage independently of compute). While this can be done through an Ethernet connection off of an appliance, it breaks the single SKU model.
• Exadata is a masterful lock-in strategy by Oracle. It’s appeal is very high because it’s essentially the iPhone of data warehouse appliances. At the same time, Oracle’s high degrees of vertical integration fully expose customers to lock-in.
• As a result of these factors, RYO data warehouse solutions will still command a good portion of the marketplace. These solutions integrate best-of-breed servers (dominated by IBM, HP and Dell) and Storage (dominated by EMC and followed by IBM and HP).
• Exadata and products like it are not well-suited for big data. High degrees of hierarchy and locking make it ideal for traditional DW use cases while big data apps, in contrast, will often thrive with a shared-nothing approach with lots of inherent parallelism (e.g. Greenplum).

The Big Data Angle

• Big data applications can be both transactional (e.g. uploading Facebook pictures) or analytical (e.g. ClickFox, LinkedIn, Bit.ly, etc).
• Big data apps have traditionally been built by techies, but as the trend goes mainstream, business heads are driving ways to monetize information and build data products.
• Big data apps are often also very industry specific – for example focused on geological exploration in energy, genome research, medical research applications to predict disease, predicting terrorist threats, etc.

A Case Study:  Value of Big Data Analytics

• Big data is more real-time in nature than traditional DW applications.
• Traditional DW architectures (e.g. Exadata, Teradata) are not well-suited for big data apps.
• Shared nothing, massively parallel processing, scale out architectures are well-suited for big data apps (but not so much for traditional DW use cases).

My Angle

The megatrends of cloud and mobile data are changing the requirements of what companies need to do in order to store and use data for their business.  The notion of big data and traditional data warehouses are different.  CIOs and business leaders are rethinking their plans and infrastructure going forward.

I see the a new data warehouse infrastructure that will be completely different than what it is now.  This will mean more spending will be required for CIOs to invest in the future.  For startups and new emerging companies like Clickfox, Cloudera, and Nirvanix this presents an opportunity.  For established players like EMC, Data Direct Networks, Hitachi, and IBM it means an opportunity to extend their product leadership.  Those leaders who don’t change and hold on to the past will lose market share fast.  Look no further than what happened to Nokia in the mobile business -from market leader hero to a zero.

Get the full research report  at Wikibon.org.