In his new book, “Fail Fast, Learn Faster,” Randy Bean uses exhaustive anecdotal evidence to make the case that businesses must become data-driven or risk irrelevance.

Bean, who is the founder of data-focused consultancy NewVantage Partners, has been evangelizing that message for years as a regular contributor to major business publications and in an annual survey that documents the progress that large enterprises are making in creating a data-centric culture.

He recently met with SiliconANGLE to talk about how data-driven culture change is finally sweeping big businesses.

Your most recent survey found that 92% of executives said the principal challenges to becoming data-driven are people and culture rather than technology. Why do these factors continue to be such large impediments?

It’s not easy for people to change their behavior, particularly in organizations that have been around for generations. There have to be incentives for collaboration and cooperation. Many pay lip service to those things, but when it comes to individual behavior it’s a different situation.

Do you see many companies effectively addressing these limitations?

We’re seeing some companies invest in data literacy, but less than half of organizations say they’re competing on data and analytics today and less than one-quarter say they’re fully data-driven. Those numbers have even declined in recent years. But I don’t think that’s necessarily a bad thing. Rather, organizations are becoming more mature and realistic about their data literacy.

Do any organizations stand out to you as being especially effective at using data?

Two are Amazon in retail and Capital One in financial services. The companies that do this best are always looking to improve. When I hear someone say they have data under control, that’s when I begin to worry.

The rise of big data has brought about concepts like “alternative facts” and the use of data for misinformation. Do you believe those trends will continue?

Data has become politicized, and I guess that’s representative of some of the challenges we face on a broader set of issues. History teaches us, however, that things happen in cycles. While the trend toward challenging data is likely to continue for some time, people eventually get worn out and move along.

Does data proficiency make organizations more innovative?

There’s a high correlation between the ability to use data and the ability to disrupt businesses and industries. But the ROI can be difficult to measure, and results play out over years, which is why many organizations abandon data initiatives.

There must be a balance between data and intuition; otherwise, we might as well surrender the world to algorithms. Emotions, intuition and reading the situation are things algorithms don’t always have the nuance to appreciate. And data can have inherent biases as Cathy O’Neil pointed out in her book “Weapons of Math Destruction.” Unconscious bias can go into algorithms and the data can reflect those biases.

How quickly do you believe a consensus is emerging around the role of the chief data officer?

It’s been poor to date; only one-third of organizations report that the role is well-established. Some banks are on the fifth or sixth iteration of the CDO. But I think the CDO role is now established. Over the years our survey has asked about what people envisioned for the future of the CDO role. Until recently, just a quarter of respondents said it was just a temporary role, but that’s a very small number now.

The role is accepted but organizations are still struggling to define it. Half of the organizations in our survey say there’s not one person charged with responsibility for data across the organization. It’s more decentralized in many cases.

You also write a lot about AI. What, in your view, makes an application intelligent?

Throughout my career, AI existed, but there wasn’t enough data to do meaningful analysis. Work was done with samples. Now the computing power makes it possible to look at every transaction you and others have made. That, plus the ability to capture online transactions, has increased the body of data AI has to work with.

Some people have great fears about unleashing AI, but AI has also failed to do some things it had been expected to do. It’s a mixed bag and there will be roles for both machines and people for a long time to come.

Photo: Randy Bean