Barely a year after Gartner Inc. predicted that 60 percent of big data projects won’t get past the pilot stage, a new survey from NewVantage Partners LLC paints a considerably rosier picture.

The big data/business innovation advisory firm’s fifth annual Big Data Executive Survey finds that 80 percent of the enterprise-class respondents say their big data investments have been successful and almost half have demonstrable return on investment to show. Less than two percent said their projects are failures.

The respondent base of 50 Fortune 1000 firms was heavily weighted toward financial services, but included giants from the life sciences, media and industrial segments as well. The decision to focus on financial services was intentional, said NewVantage founder and Chief Executive Randy Bean. “Financial services is fundamentally a data business. Big data investment will be driven by what financial services firms do,” he said.

The results indicate that these companies have invested heavily in big data projects, with 37 percent estimating that their total investment has exceeded $100 million over the past five years. They are also buying into the concept of a chief data officer; nearly 56 percent have appointed someone with that title, although many still see the role as defensive, driven by regulatory compliance requirements.

Cost is king

The primary payoff of big data so far has been cost reduction, but a significant plurality of 44 percent say big data has created new avenues for innovation. More than one third have launched new products and services as a result and nearly one-third have seen increased or new revenue. Nearly half say their big data efforts are either highly successful or transformational.

Not that success has exactly been a slam dunk. More than 85 percent of respondents are attempting to create a data-driven culture, but only 37 percent report success. Management understanding, organizational alignment and organizational resistance are the biggest impediments. “Resistance to changing roles, changing processes, non-traditional ways of thinking, and new technologies and approaches all obstacles,” said NewVantage Chief Executive Randy Bean. “It implies new ways of doing things, and it is hard for many people to adapt quickly enough.” Eighteen percent say their companies lack a coherent data strategy. Four in 10 lack an enterprise big data strategy.

Becoming data-driven doesn’t mean making every decision by the numbers, Bean said, but it does mean giving decision-makers access to quantitative metrics that they can balance with intuition, creativity, and human judgement. “It becomes a way of thinking,” he said.

The survey documents a significant shift in attitude toward the role of the chief data officer, which is generally defined as an individual who is charged with creating consistent standards, formats and processes for handling information resources. The 56 percent of organizations that have appointed a CDO is up from just 12 percent in the 2012 survey.

Secure from disruption?

Business disruption, which is often cited as a driving factor in big data adoption, is less of a concern than management magazines and conference organizers would indicate. Less than half of respondents fear that their firm may be at risk of a major disruption in the coming decade. Artificial intelligence/machine learning was cited as the technology area with the greatest disruptive potential, followed by digitization and the internet of things. Cloud computing and blockchain also rate high on the disruption scale.

Does the relatively low concern about disruption indicate executive complacency? Bean said he’s of two minds. “I was actually surprised the fear of disruption was so high, because these are large corporations that tend to be conservative by nature,” he said. “On the other hand, shouldn’t all firms think ‘grow or die?’ It’s a good mindset to have.”

Success with big data clearly demands a strategy and a roadmap, Bean said. The three-step process he recommends begins with documenting the current state of capabilities and uses, then defining a vision of what capabilities are needed to support a vision. The third step is to create a roadmap that gets you there. “Organizations should define small steps – one success at a time,” Bean said. “This enables them to demonstrate business value, establish credibility, gain buy-in and build credibility.”

Image by Pexels via Pixabay