In their ongoing coverage of the 2012 Hadoop Summit, John Furrier and Jeff Kelly spoke with Arun Murthy, Hortonworks co-founder about Murthy’s personal experience stepping away from Yahoo!, Hadoop business strategies, technology innovations and future directions (see part I of the interview below, and click here for part II).

Furrier inquires about Murthy’s risk in leaving Yahoo! to pursue his own entrepreneurial venture with Hortonworks co-founders. (Although he does note that Murthy’s leap must have felt a bit more secure with backing from the well-respected and indexed investor, Benchmark.) Reflecting on his experience since Yahoo!, Murthy felt enthusiasm for the transition: “As an engineer, you see a whole new set of use cases…from healthcare, to crime protection, to Fraud, [it’s] really exciting to be in this space and make an impact in such a variety of use cases.”

Furrier suggests the next business move for Hortownworks is quite clear: “100% all in on the technology, 100% open source, all money goes towards development. Period.”  He adds: “On the business side, it’s pretty specific, create bulletproof, enterprise-grade quality business deals with the big guys and let that be the pull.”  Murthy agrees, suggesting that while Hadoop does have customers outside the Fortune 500, they believe working with leading firms “is the easiest way for Hadoop to get real use cases and real pull in the enterprise…[so] we can actually take this technology and make it much more prevalent in a significantly smaller time frame.”

Discussing Hortonworks and Hadoop’s growth, Furrier notes that while these companies generate significantly less revenue than heavyweights like IBM and EMC, “the disruption you’re providing in their market is significant.  You guys are creating such a cost reduction of data store and data warehousing that they’re eventually going to need to reposition their products.”

Concerning product development, Murthy explains that there is no need to “re-invent the wheel” as most clients do not want to completely replace their existing software with Hadoop.  Rather, Hadoop will remain open with more integration with existing software.

The panel quickly delves into new capabilities in Hadoop, discussing its two essential components, HDFS and MapReduce. Murthy explains that whereas before, to process data in HDFS, the only option was MapReduce, now Hadoop can be generalized to run on different applications. Murthy illustrates the added value this presents to the client: “You can now take an open MPI application [and] without your knowledge as a developer, start using Hadoop for MPI processing. It’s actually a really useful way for people to get more value out of Hadoop clusters…If we do our job right, there will be zero investment.”

Murthy notes most Hadoop summit attendees are concerned with integrating Hadoop and how to make it better.  Murthy also shares one of “the most mind-blowing” innovations he has heard at the summit is an offering for highly personalized medications, not based on average health categories of height, weight and age, but individual DNA sequences.  He adds: “The fact that we can make such a difference in humans as a whole is insane from where I stand as a technologist.”  For Murthy, Etch Base, S-Core and Yarn are also exciting new technologies still being developed in the Hadoop ecosystem.

Kelly asks about the potential for overlap between Hortonworks and Teradata, two companies offering very similar big data solutions, and partnering together, nevertheless. Earlier Murthy explained that while the media sensationalized rivalry among the brands, they actually all partner together, and it’s understood that working with Apache Hadoop means “you’re going to be a friend of Hortonworks.” In response to Kelly’s question, Murthy echoes his earlier point about partnering with similar brands simply stating: “At the end of the day, it’s a win-win situation for Hortonworks, Teradata and the customer.”

Next year’s priority list for Hadoop, according to Murthy, includes enhancing reliability across multiple interfaces and increasing overall development and engineering investments from industry leaders to make Hadoop 2 a success.  Furrier suggests that Hadoop is no longer a distant second place behind Cloudera.