Organizations are upping their investment in big-data and artificial intelligence initiatives, driven by fear of disruption more than a quest for operational efficiency.

That’s according to a just-released NewVantage Partners LLC survey of Fortune 1,000 companies, most of which are in the financial services and healthcare industries.

The survey, which NewVantage has conducted annually since 2011, covers only 64 firms, but they include some of the world’s largest banks, insurance and life sciences companies. More than 97 percent of respondents have C-level titles.

The good news for big-data aficionados is that enterprises are all in. More than 90 percent of the companies surveyed are quickening their pace of investments in big data and AI projects, with an overwhelming 75 percent saying it’s because they fear disruption from data-driven digital competitors. That’s up dramatically from the 47 percent who cited disruption as a motivator just two years ago. Eight in nine said their investment strategy carries some urgency.

“We are in a disruptive time and big data, AI and machine learning are at the center of much of the change,” said Randy Bean, NewVantage’s chief executive and principal author of the study.

What’s not so good is that companies aren’t seeing game-changing results — at least not yet — and most are struggling with organizational and human resistance to new technology. While 62 percent reported measurable results from their investments, a little less than half said they’re competing on data and analytics while only 28 percent said they’ve built a data-driven culture. The 31 percent of executives who said their companies are data-driven is down from the 37 percent who said so in the same survey two years ago.

Several factors are holding organizations back from achieving their big data and AI potential, but the principal issues are cultural. In fact, 95 percent of respondents said the problems relate to people and process while just 5 percent cited technology.

That’s alarming, according to Bean. “That number should be rising sharply.  The fact that it is not is troubling,” he said.

Bean and co-author Tom Davenport write that the implications of these organizational struggles are significant because few firms can afford not to be data-centric in the future. “Companies that fail to develop a data culture are at risk of potential erosion, either over time or suddenly,” Bean said.

Legacy burden

Cultural impediments are indicative of the negative impact of legacy structures during times of rapid change, Bean noted. “Most organizations are still organized along traditional business and technology lines, but data cuts across all lines of business,” he said. “Data is horizontal, but organizations tend to be vertical. They must come to appreciate that new organizational structures are required.”

Efforts by businesses to transform themselves around data often founder for the same reasons, he noted. “People can only digest so much information or assimilate so much in terms of new approaches to traditional business processes,” he said. “That is where transformation efforts are failing.”

The survey also uncovered some disparities in how executives see the role of the emerging position of chief data officer. Although Gartner Inc. has reported that the CDO position is rapidly going mainstream in enterprises, respondents indicated sharp differences in what they believe the position should encompass and who should fill it.

For example, 38 percent said the CDO should be an external change agent, but a nearly equal 32 percent prefer a company insider. Nearly half think the CDO should have primary responsibility for data, but a significant 28 percent said there should be no single point of accountability. And despite the fact that 68 percent of responding organizations have a CDO, nearly one in five executives said the role is interim or unnecessary.

The disparities aren’t surprising for a role that has come practically out of nowhere over the last five years, Bean said. “As I recall, it took the better part of a decade for firms to refine the chief information officer role,” he said. “I expect we will face the same issues.”

Consensus will emerge through a process of trial and error over time. “Just writing up a bunch of policies and procedures is not a formula for success,” Bean said.

Image: Pixabay