Venture capital fund Accel Partners, a major investor in Cloudera, Clickview, and Fusion-IO, announced the creation of a $100M Big Data fund at HadoopWorld 2011 in New York City Nov. 8, to foster rapid innovation and development on top of the Hadoop/Big Data platform.

The opportunity to invest early in a new technology platform like big data only comes along once every few years, Ping Li, who will manage the Big Data Fund for Accel, said later that day in an interview in the Cube webcast live on SiliconAngle.tv. “We are already seeing how entrepreneurs are deploying Hadoop in applications that we didn’t think possible,” he says. The fund is dedicated to forming a coherent portfolio of investments around big data to expand Accel’s existing participation in development of the basic platform.

It takes time for a new platform to develop and for entrepreneurs to understand how to build value on top of them, he told Wikibon.org Co-Founder and Chief Analyst David Velante and SiliconAngle Founder and CEO John Furrier. So far the focus has been on the base platform, but the market is already moving up the stack to the business application level.

Cloudera CEO Michael Olson also sees the excitement in the announcement. “Developers will only see the new opportunities as big data becomes engrained in our thinking. If you have a deep understanding of the data source, where the data comes from, and what it can tell you, then you can apply that knowledge to create new opportunities.”

A lot of the opportunities will come from simple ideas applied to large-scale data, he said. “The ability to work with data at scale and deploy these tools on a platform is brand new. Even simple ideas can be transformative.”

To the critics who have said that Hadoop is more of a feature, too simple to be a platform, Li replies that platforms often start that way. The iPhone, Facebook, and VMware, for instance, started small and seemingly simple. “Something like that finds its way into the enterprise as something light-weight, and then suddenly everybody is using it, and it becomes your storage infrastructure in the cloud.”

Olson compares Hadoop today to relational data structures in its early years. “It was a platform that could manage data, but it wasn’t easy to use. Today Hadoop can manage large amounts of unstructured data and companies are already using it in several verticals. But map reduce is not easy to use.”

Then companies like Oracle, Informix, Sybase, and IBM started building on top of it, creating systems and toolsets that were more practical. And after that application companies built business applications such as ERP and CRM that today are basic business tools on top of those tool sets. And the market grew from a tiny beginning to its present huge size.

Hadoop is following a similar path, Olson said, and it will go through the same stages and grow until the big data market rivals the RDBMS market in size. And the presence of Oracle at HadoopWorld this year is one indication of that.

“There is plenty for us to do on the Hadoop platform to make it more scalable and usable,” he says. “And the greenfield opportunities are huge. So this is not a matter of competition but of developing a market with huge potential.”


Watch live video from SiliconANGLE.com on Justin.tv