For the past four years, big data consultancy NewVantage Partners LLC has surveyed a group of executives at large enterprises to determine their strategies surrounding big data. This year’s survey confirms that big data has turned the corner. In fact, 62.5 percent of firms reported that they now have at least one instance of big data in production and 70 percent said big data is now important or critical to their success. NewVantage CEO Randy Bean joined us for a CUBEconversation to discuss the findings and what they mean for users and big data technology providers. Bean founded the boutique consultancy in 2001. He writes a monthly column on big data for the Wall Street Journal, and is a contributor to MIT Sloan Management Review and Harvard Business Review.

This is the final edition of the survey. Why?

We host a series of breakfasts for CIOs from large companies, and about four years ago a CIO from JP Morgan said big data is a new phenomenon and he would like to understand how it is being adopted and what it means for them. We’re not in the survey business, but we had access to a couple of hundred C-level executives, particularly in financial services, so we decided to reach out to them.

The reason this is the final survey is because we realize the big data is being broadly adopted across organizations. It’s not a new phenomenon. We may come back and ask more questions next year, but they’ll be more along the lines of what results companies are seeing.

One of your findings is that more than 62 percent of the firms you surveyed now have at least one instance of big data in production. That raises the question to me of definitions. How are they defining big data?

We provide a definition of big data that we use globally. It can refer to all kinds of data: large, small, legacy, structured, unstructured. We’re referring to new approaches such as Hadoop that have made it possible to be much more agile, to democratize data processes and put data in the hands of business users much more rapidly. It really changes the whole equation. Traditionally, firms had to go through rigorous data engineering that took a lot of time. It wasn’t unusual to hear that it took 15 months and $3-$4 million to get an answer to a simple business question. Big data has changed that.

When a market goes mainstream, priorities change. Big data has been struggling to get a foothold until recently. Now that it’s mainstream, how will things change?

In the early years a lot of organizations were focused on doing proof-of-concepts, pilots, setting up an analytical sandbox, setting up a big data laboratory or center of excellence. They were trying to prove out what were the most likely use cases. Now we’re at a stage where big data is operationalized, but we’re learning that organizations aren’t able to point to the return on their investment. Many organizations can point to having created the big data environment, but very few organizations can point to the return they’re getting. In the future we see that as being the major challenge.

Your survey also found a dramatic increase in the number of people with the title of Chief Data Officer. This role means different things to different people and it lives in different parts of the organization. Are there any trends you found in how this role is being defined?

When we first started the doing this survey in 2012 only about 12 percent of organizations had a Chief Data Officer. Now it’s more than half. But there’s no clear definition of what the role means. Last week I was on the west coast with Charles Schwab, where the chief data officer reports to the chief marketing officer. In many organizations the CDO reports to the CIO, and in other organizations the position reports to the Chief Financial Officer.

There are also CDOs in both offensive and defensive roles. Many CEOs were hired to fulfill regulatory requirements. But at the same time many CDOs have strong analytical capabilities to leverage data as a corporate asset and create new products and capabilities. But much of their time is spent firefighting.

Do you see the potential for the Chief Data Officer to become a more important position than the CIO because the person is the steward of the corporate asset?

It’s unclear how the role will evolve because it’s a new role and it’s also a bit of a target. A few years into having a Chief Data Officer, some organizations are asking exactly what they’re getting for a return. It’s both an opportunity and a challenge.

In that vein, the CIO is typically responsible for technology platforms and the Chief Data Officer for data itself. Your survey found that technology was actually a minor big data issue among the executives you surveyed. Did that surprise you?

It wasn’t entirely surprising for two reasons. First, when NewVantage Partners started 15 years ago we assumed we were going to be 95 percent focused on technology and five percent on other issues. Today we’re focused five percent on technology and 95 percent on organizational issues. There’s a lot of technology that can solve data problems; the bigger issue is what the environment is like. It’s not so much a technology issue from our perspective, it’s an organization, business process and perspective issue.

Another surprising finding was that big data initiatives are currently being driven more by operational efficiency rather than business transformation, customer experience and other more strategic issues.

That’s been a major disappointment for many people because there has been hope that big data will be used for exciting things, but for the majority of financial services and life-sciences companies it’s primarily a migration from mainframes to Hadoop environments. Credit companies that in the past used to look at a subset of data to make credit decisions can now look at 10 years of data instead of one, and do it in a fraction of the time. These aren’t the sexy, consumer-focused applications that a lot of people had hoped for.

Do you think those applications will come? Are companies just looking at the low hanging fruit right now?

Absolutely. They have to provide the initial justification for return on investment, and there are some obvious processes to move from the mainframe to Hadoop. Big data should be looked at as enabling discovery in a rapid fashion. It creates a much more iterative process for an organization; rather than coming up with a hypothesis in advance, they can learn as they go. The metaphor of failing faster and more frequently is what big data enables. But innovative applications have been slower to come.

Gartner last year made some waves when it said that most big data initiatives were still in the starting blocks. Your survey indicates that they’re quite a bit further along. What’s the disconnect?

It’s a glass half-full/glass half-empty issue. There’s been substantial progress over the last three or four years in implementing big data environments, but in terms of creating new products and services, firms are still in their infancy. It’s like the adoption of the Internet; in the early stages using the World Wide Web was remarkable and 10 years down the road there was nothing special about it.

Another surprise in your research was relatively low interest in unstructured data. It seems there was a lot of excitement about tapping into social data, text documents and emails, but not much is happening yet.

It’s not the most important area thus far. A lot of organizations are focused on trying to bring in sources of data they haven’t been able to tap into in the past. For example, there may be thousands of silos within an organization, each with a different database.

Are vendors talking past their customers when they talk about technology rather than business benefits?

Big data projects are driven by the business. Business folks are focused on finding faster answers to their questions. There are a lot of technology solutions that can do the job, so the tech is not that critical at this time.

Watch the full interview (16:33)