CIOs are facing a major inflection point in the adoption of Big Data, writes Wikibon Big Data Analyst Jeff Kelly and Chief Analyst and Co-Founder David Vellante in their latest Wikibon Professional Alert. Legacy data warehouses and business intelligence (BI) systems have failed to deliver the promised 360 degree views of customers, predictive analytics, and major productivity enhancements and today are inefficient, wasteful patchworks of myriad technologies and work arounds.

While companies have become dependent on these “data jailhouses”, they are holding those companies back, forcing them to depend on rearward-looking analytics whose value is based on the false assumption that tomorrow will be like last week and next month will be like last year. It blinds them to the oncoming trains of innovation that run over unprepared companies. Meanwhile expensive assets remain under-utilized as technology vendors maneuver to maximize their profits and customer control.

All of this is changing with the advent of Big Data that allows companies to see what their customers and other market influencers are doing when they are not interacting with the company Web site, sales or support staff. It provides the basis for true forward-looking analysis that allows companies to anticipate rather than react to change and grab the initiative rather than constantly playing catchup. It can allow corporate planners to grasp what their customers need rather than what they think they want. As Steve Jobs famously said, if Henry Ford had depended on focus groups, he would have spent his life breeding faster horses. It also shifts the initiative for data creation away from technology vendors and to technology users.

“We believe that Big Data practitioners will create more value than Big Data technology suppliers going forward. In other words, what user do with Big Data will become more important to value creation than vendor activities. Therefore, organizations need to re-assess and re-tool their strategies, data architectures, processes, and skill sets to create differentiable competitive advantage,” Kelly and Vellante write.

This puts CIOs in a difficult position. On the one hand, they have huge investments in existing data warehouse and BI infrastructures. And the vendors of those technologies are reacting to the Big Data wave by providing connectors that allow their existing systems to connect to Hadoop and other Big Data technologies. The problem with this approach, Wikibon believes, is that it depends on a “cumbersome and fundamentally flawed data infrastructure” that cannot process the huge volumes of “machine-generated, social media, geo-location, and countless other types of multi-structured data” fast enough to provide the near-real-time, forward-looking analysis businesses need. Further, “they’re buckling under the added weight and threatening to bring gown laggard enterprises with them.”

The better choice, therefore, is to “go all-in on Big Data” by investing in new, agile, scalable, and service-oriented data management infrastructures. Big Data pioneers such as Facebook, Google, LinkedIn, and Amazon have shown what data-driven organizations can accomplish by harnessing Big Data technologies. In this environment, marginal improvements to existing data management technologies and infrastructures are not enough to thrive or even survive. organizations must embrace the new breed of Big Data technologies including flash storage, Hadoop, interactive data visualizations, and agile application development tools, including Open Source products such as Hadoop and architectures from startups such as Hadapt.

This does not mean that companies should rip-and-replace all their legacy data management tools immediately. CIOs, Kelly and Vellante say, should think like portfolio managers, laying the groundwork toward innovation and growth while mitigating risk factors. The pace of the transition will vary from one organization to another depending on numerous factors. But it is a transition that all CIOs must make if they and their companies are going to survive and thrive in the coming century.

As with all Wikibon research, this paper is available in its entirety on the public Wikibon Web site. IT professionals are invited to register for membership in the Wikibon community. This allows them to comment on research and publish their own Professional Alerts, tips, questions, and relevant white papers. It also subscribes them to invitations to the periodic Peer Incite meetings, at which their peers discuss the solutions they have found to real-world problems, and to the Peer Incite Newsletter, in which Wikibon and outside experts analyze aspects of the subjects discussed in these meetings.