Merv Adrian, VP of Gartner, was asked by John Furrier and Dave Vellante, theCUBE co-hosts, to recap the evolution of Hadoop, in an interview that took place at our BigDataNYC event in New York this week.

“If Hadoop were a baby, it has gone from why and where to how and who,” said Adrian. “This conference was all about real deployment. Organizations that were kicking the tires are now prepared for the production readiness, Enterprise class features, and more and more governance and security, the issues on the lips and minds of people who were getting ready to make investments. At Gartner we are concerned with some issues that have been inhibitors so far. Our data says 7 percent of the Enterprises have made real investments that have led to a deployment for a Big Data project of some sort – that’s not confined to Hadoop only – but to Big Data of any form that meets the classic Gartner definition of the 3 V’s. It’s an early market, but I think we’re moving into mainstream.”

Next up Adrian mentioned the Gartner hype cycle, “the stage where we get past that early enthusiasm and early adopters hit a wall, for one reason or another. Maybe the product is not good enough, maybe they chose it for the wrong reason, maybe they are not doing it very well. While all of them may be true, we generally move past that stage into what’s called the ‘plateau of productivity’ once balance begins to reassert itself. Hadoop distribution is well into the trough right now, and that’s not a negative; it’s a positive because you want to move from left to right, towards a more balanced and informed deployment.”
“By contrast, one of the big themes here at the conference is SQL, which is just after the starting point, beginning to climb the curve. There’s a lot of enthusiasm now but a lot of people still don’t know that it’s going to be possible to do interactive query,” commented Adrian.

“The big story here is choices. There are many, many alternatives than it has ever been and that’s bewildering for people,” thinks Adrian. Hadoop’s simplicity and ease of use may come to its advantage in such a competitive world.

“Hadoop is a platform on which many other things are going to be constructed and next to many other things are going to be deployed and those choices are going to define how people do data processing.”

Watch the entire segment below to find out Merv Adrian’s take on the ‘rise of the machines.’

“We’re not at the SkyNet moment yet, but when the machines start to tune themselves, things get really interesting,” predicted Adrian.