Yahoo’s core Hadoop contributors have left the company to form their own startup called HortonWorks. HortonWorks has big time financing and support from Benchmark, and the company will be headed by be ex-Yahoo engineer manager Eric Baldeschwieler.  According to my sources HortonWorks raised around $2o million at a very high valuation of over $200 million.

This is one of the big stories at the Hadoop Summit 2011 in Santa Clara, California. Hadoop Summit is where the open source Apache community converge with new industry commercial players to take Hadoop to the next level.

New details are emerging about HortonWorks as we speak.  Sources close to the deal confirm that Yahoo had shopped the deal (although not officially) to EMC and Netapp. According to that source both companies passed on buying or investing in the company. In the end, the white knight was Benchmark Capital who has actively been trying to get into the big data open source market that has been dominated by Cloudera.

Benchmark is a natural fit since they have expertise in open source within their partnership and with Red Hat. Yahoo exec Jay Rossiter denied that they “shopped it” around but did say that they had conversations with many companies. This is code for shopping it around.

Benchmark will not only provide the funding, but will contribute their partner Rob Bearden, who will take on the Chief Operating Officier (COO) role at HortonWorks. Bearden has operating experience from his time as President and Chief Operating Officer (COO) of SpringSource and Before joining SpringSource,OpenSpan, and JBoss.

When pressed for comment on HortonWorks funding amounts and ownership Yahoo’s Rossiter and HortonWorks Baldeschwieler, declined to comment. According to my sources in Silicon Valley, Yahoo will maintain less than 20% ownership in the HortonWorks venture with a warrant kicker to bring it much higher if they chose to exercise that option. So, this is a fully funded and a pure startup, and Yahoo has a direct interest in it’s success.

What is really going on with Yahoo and HortonWorks?

Why is this happening? If Hadoop and Big Data is so huge and disruptive, why is Yahoo in essence spinning out HortonWorks? Why did EMC and Netapp pass?

Answer: Market forces were strong. Yahoo wanted to keep presence in Hadoop given their scale, and Yahoo didn’t want to compete in the software distribution aspect or servicing the open source Hadoop market. Jay Rossiter said:

“Yahoo is absolutely committed to Hadoop. Yahoo will be HortonWorks first customer and work through HortonWorks on the Apache side, Yahoo will leverage what comes back internally at Yahoo. This will not disrupt Yahoo’s current plans in the continued advancement of Hadoop internally for Yahoo’s core media business”.

Another big reason for this move by Yahoo was the potential brain drain of top talent possibly walking out. This was confirmed by a source close to HortonWorks. My source said that one of the big drivers in allowing the new venture HortonWorks to go forward was that Yahoo was having trouble retaining the core team, and was seeing a mass exodus of talent to companies like Cloudera. Yahoo obviously denies this, but the risk of talent loss was real. In talking to Jay and Eric it was very clear to me that Yahoo is pro the startup move verses trying to keep it internal. I found yahoo’s answers to be genuine. I’m a fan of the move of the creation of the new team. Overall a win-win for all involved.

Contrary to public conversations, Yahoo denies that they are doing HortonWorks for cost cutting purposes. Jay Rossiter was hard core on saying that they have a multiple times more engineers in Yahoo contributing to Hadoop than the number of engineers at HortonWorks.

I am very interested in this spin out for many reasons:  First, I’m excited for the Yahoo employees who are now free to do a creative, relevant job where their efforts will be rewarded. Second, the entire ecosystem is very fragile right now and the market demand is growing stronger every day. Third, new services and companies continue to fuel an economic recovery and job creation.

The biggest challenges will be 1) how does HortonWorks try to catch up to Cloudera in a way that doesn’t create “diseconomies of scale”, in trying to take short cuts in developing commercial grade products; and 2) can HortonWorks serve two masters – the Yahoo infrastructure and the community and ultimately the enterprise customer. This will be difficult as Cloudera just extended their already huge lead with announcement of their new enterprise grade management software.

One risk that I mentioned was that of losing opportunity in possibly getting into the private cloud or hybrid cloud market as the IT market goes consumer. In talking to Jay Rossiter the consumerization of IT and hybrid cloud services wasn’t on his radar. However,  e did acknowledge that Yahoo might in the future be perfectly poised to be a major player in the IT enterprise SaaS, PaaS, and IaaS.

I would agree that a big data software play with Yahoo’s mature consumer platform could possibly be a huge win for Yahoo. This isn’t about search. It’s about the consumer experience in IT.

Good luck to HortonWorks and Benchmark as they put the task in the hands of the new class of entrepreneurs.