The Big Data market is dynamic, with companies going public, others like Hadapt and Revelytix, Inc. being acquired by Teradata Corp., and a lot of technology innovation happening. Large amounts of money have also been changing hands. For example, last year we saw Intel Corp., one of the industry’s biggest Big Data players, invest $740 million into Cloudera, Inc. We also saw MapR Technologies, Inc. complete a $110 million round of financing.

In fact, billions of dollars have been raised in the Big Data space over the last few years, but where exactly has all this money gone and why? These were just a couple of questions answered by Wikibon’s Principal Analyst Jeff Kelly in a presentation entitled “Following the Big Data Money Trail” at last week’s Big DataSV 2015 event at the Fairmont San Jose. In his presentation, Kelly answered the most common questions he gets asked about the Big Data market, which we’ll examine here.

.

How big is the Big Data market?

.

Wikibon determined that, in 2014, about $28 billion in revenue was created by the supply side in the Big Data space. This includes all different areas—from Hadoop and NoSQL to all of the related tools and technologies including data warehousing, data integration and professional services. If we look at the Hadoop and NoSQL slice in particular, Kelly said we’re talking about just over $1 billion dollars in revenue in 2014 going into 2015. That’s a pretty small slice of the overall market but Kelly said that’s where Wikibon is seeing a lot of the innovation happening.

.

Which enterprises are adopting Big Data?

.

The Global 1000—the really big enterprises out there—are adopting Hadoop and some of the other Big Data approaches, Kelly said. Then we have the companies he described as the “born data-driven startups” or the companies “that have Big Data built into their DNA” (such as Uber, Inc., for example) who are adopting Big Data. But we still have a large number of enterprises that are just exploring Big Data but haven’t really taken any steps to adopt it yet.

.

What are Big Data’s early adopters doing with it?

.

On the enterprise side, what Kelly is consistently hearing is that the initial use cases for technologies like Hadoop have been focused more on the cost savings side. He said this includes data warehouse optimization and data warehouse offloading—essentially, the moving of data from more expensive systems to the less expensive Hadoop platform to save money, archive that data, and reduce spend on some of the more expensive technologies.

.

Why are enterprises still struggling with Big Data?

.

Enterprises are still struggling with Big Data because Hadoop, in general, is complex. Plus, Big Data is complex and there’s a lot of different components involved; it’s not a simple, monolithic technology. Kelly said even pioneers on the enterprise side are still struggling in a lot of cases to expand beyond their pilot projects and move to more production-grade deployments. They’re supporting not just cost savings but more revenue-generating applications.

.

How much return on investment (ROI) have companies been seeing?

.

Enterprises who have adopted Big Data technologies have told Wikibon that, on average, they have seen an ROI of about 55 cents on the dollar, which Kelly said is “not great.” Plus, since Wikibon did their Big Data survey (over a year ago), Kelly says that the average ROI has actually gone down because adoption has increased but the struggling with the technology continues. “If they’re getting 50 cents on the dollar, they’re doing pretty well in these early days,” Kelly said of these enterprises. There’s still a lot of complexity involved, he said, and Wikibon is seeing most of the use cases still focused on cost savings and less on revenue generation.

.

What can we learn from previous markets like Business Intelligence (BI) and Data Warehousing (DW)?

.

We saw consolidation in both the BI and DW spaces when Business Objects SA, Cognos, Inc. and Hyperion Solutions Corp. were acquired. Then, in the MPP data warehouse space, Netezza Corp., Aster Data Systems, Inc. and Vertica Systems, Inc. were acquired. Wikibon thinks we are seeing (and will continue to see) similar consolidation in the Big Data space. In the BI and analytics space, Pentaho Corp and Revolution Analytics, Inc. were recently acquired. In the Hadoop space, Hadapt and Revelytix were recently acquired.

Another interesting development in the data warehouse space in particular was the consumption model, Kelly said. The appliance model—the bringing together of hardware and software—really became the standard way for data warehouse technology to be adopted in the enterprise. Kelly doesn’t think that’s going to happen in the Big Data space, at least not to the extent that we’re going to see a lot of “appliances.” But he said we’re definitely seeing a need for a more platform approach that will tie together the different components into something more consumable for the enterprise.

.

How will open source help drive innovation in Big Data?

.

In the BI and DW space, we did not see open source playing a huge role (back in 2007 up through 2011 or so) according to Kelly. What’s really different now is that open source is driving a level of innovation in the Big Data space that was never seen in the BI and DW markets. Nowadays, open source software is not just being accepted by the enterprise, it’s increasingly becoming a requirement. Wikibon thinks this is going to lead to some good outcomes. Kelly explained this by saying, with the open source nature of Big Data generally (and Hadoop specifically), acquisitions will happen but the innovation will continue.

Because, not only are we going to have new technologies being developed by well-funded startups, but a lot of the innovation in the Big Data space is coming from practitioners—from companies like Facebook, Inc. and Netflix, Inc. who are creating technologies internally and then open-sourcing them to the community. This will lead to more startups commercializing these technologies and the cycle of innovation will continue.

.

Where is all the money that’s been raised?

.

In terms of venture capital, Kelly said we’re seeing “really crazy amounts of money being raised.” The three Hadoop pure-play vendors—MapR, Hortonworks, Inc. and Cloudera—together have raised over $1.6 billion. Add to that the NoSQL players and the amount rises to over $2 billion. If you add in all of the other Hadoop players, Kelly said we’re probably talking close to over $3 billion that’s been raised in the Big Data market.

.

Why do we need billions of dollars to enable the Big Data market?

.

Kelly said takes time in the open source ecosystem to build the market. This is separate from the innovation, which is happening fast and furious. But it takes time in the open-source community to actually build an enterprise-grade solution or platform that will be accepted by the mainstream enterprise. You’ve got to come to an agreement on some kind of standard, and you’ve got to understand the different concerns of the enterprise—which is not always the sweet spot for startup companies that are focused on an exciting but cutting-edge technology.

It also takes time to build out distribution and sales channels that startups don’t necessarily have. Startups are really good at innovating stuff; they’re not always great at taking that innovation, packaging it, and delivering it to the enterprise in a way that can be consumed.

The other reason that all of this capital is being raised, Kelly said, is because the Big Data market is not confined to small players or startups. The industry big whales—IBM, Hewlett-Packard Co., EMC and others—are being very aggressive in this market because they see the opportunity. As a result, startups have to compete for mind share with companies that have huge marketing budgets.

.

Will the Big Data market be disruptive to the big players?

.

When the Big Data market first started to evolve, Kelly said people thought it was going to be very disruptive to the big players like IBM and Oracle. “But I would say that that’s actually not the case,” Kelly said. “I think, in fact, the rich are going to get richer. This market is going to be dominated by the big players.”

Kelly said the big players are spending their money on targeted, strategic acquisitions that are going to fill functional gaps with some of the hot new technologies. The big whales are really good at packaging that and delivering it to the enterprise because they have the relationships at the C level; they can actually move this market forward in a way that startups can’t.

.

How will the consolidation within the Big Data market impact startups?

.

Kelly said it’s going to be very difficult for the startups to build long-term, sustainable businesses when you have the big players (like IBM, Pivotal Software, Inc., EMC and Oracle) playing such a big role. Will there be any independent players left in five or 10 years? Kelly thinks that Cloudera and Hortonworks are the two companies best positioned right now to be the ones left standing in five years. They got off to an early start, they have laid the foundation for this entire market (along with companies like MapR). The Hadoop slice of the market is where Kelly said we’re seeing a lot of the innovation. Kelly thinks that a lot of the startups that are focused on what he describes as “a particular tool or some segment of the larger Big Data stack” will get acquired.

That’s not to say that the innovation, the tools and the technologies that these startups are creating are not important because they are. But, as Kelly points out, it all has to fit into the larger, platform approach that mainstream enterprises require. Enterprises don’t want to cobble various components together to enable innovation around analytics or data-driven applications. Kelly said a platform approach also helps from a governance and compliance perspective. When you cobble different things together, it gets more difficult to track things like data lineage and how people are using the different components within the stack.

.

Why does Big Data itself need to fit into the larger IT infrastructure?

.

Kelly said that the tools not only have to fit into the Big Data stack but Big Data itself needs to fit into the larger infrastructure, the larger data management landscape within a mature enterprise. And if you even take a step back from that, he said, that needs to fit into the larger infrastructure and some of the innovation that’s been happening in the cloud space. For all that to happen, it’s going to require some significant innovation.

.

Is this the beginning of the end of the Big Data space?

.

When there’s acquisitions in a space, people tend to think it’s the end of the innovation. But Kelly thinks this is not the end of the Big Data space. Kelly thinks that, in the next phase of Big Data, enterprises are going to get more mature around data governance. “Not just the technology and bringing it into their environments,” Kelly said, “but some of the other, more process-centric, more non-technology challenges associated with Big Data. That’s mainly around data governance, compliance, ethics, more of the process and people and political issues.”

.

Real-time and the Internet of Things (IoT): Big Data’s next phase

.

Kelly said we have moved from some of the early use cases (i.e., cost savings in data warehouses) and using a Hadoop environment to do so. But now you can build data-driven applications that are actually going to focus on revenue generation, as well as other applications that are focused on customer data.

Kelly thinks the next phase of Big Data will be moving towards real-time, near real-time, and the Internet of Things. “I think, clearly, when you look at the amount of data being created by essentially objects that previously did not create data, there’s so much opportunity there and so much innovation that’s going to happen in that space. I think that’s where you’re going to see some of the real excitement.”

.

You can view the video of Jeff Kelly’s presentation in its entirety below.

Also, be sure to catch SiliconANGLE’s archived coverage of Big Data NYC 2015 on our YouTube channel here..

 .

.

Photo credit: Patrick Hoesly via photopin cc

Big Data graphic courtesy of IBM

Photo credit: opensourceway via photopin cc

Photo credit: Damian Gadal via photopin cc

Photo credit: Knee Deep Photography via photopin cc