HP has watched Big Data develop over several years and believes that it will cause major disruption in the storage and systems market, says Manoj Goyal, Senior Director of Data Management Solutions in HP’s Enterprise Group.

As a result, “HP is all in on Big Data,” he said in an interview in The SiliconAngle Cube at HP Discover 2012 (full video below). It is working to develop a complete ecosystem around Hadoop that includes hardware, middleware, software, and services to “make Big Data more accessible to customers.” And, he said, “While HP does not make its M&A plans public, we are always looking for partners and ways to fill in gaps. Over time big things will happen.”

HP, he said, is solving the “Big Data consumption problem. What was Hadoop/Map Reduce has become an ecosystem.” The first step in productizing Big Data was simplifying the underlying infrastructure users need to support their Big Data installations. HP’s Enterprise Group, ESSN, has focused on developing a pre-engineered, factory-integrated, workflow-optimized building-block hardware/middleware cluster that users can plug & play to get value almost immediately.

Meanwhile, Goyal’s group is working with Autonomy and Vertica internally to create a software layer to support Big Data analysis. Some of the results of that collaboration were announced at HP Discover.

“Clearly Vertica is the best place to take the Big Data once it is structured,” he said. So the goal of this part of the project was to develop the capability to transform the data into a structured format that Vertica can analyze.

Meanwhile, HP Labs “started well ahead of any other business units, as you would expect, analyzing the needs of Big Data.” It is developing several innovations to solve important issues. The first of these, HP’s Memrister solid-state storage system, “will make Petabytes of storage behave like DRAM.” This will solve a basic problem, which is that Hadoop is by nature an IO-bound environment. “Making so much of the storage very fast will do radical things to that environment,” Goyal said.

Then HP is working with outside partners like CloudEra to develop the specific applications to run on top of this ecosystem to support user needs.

And HP IT has become the first real-world customer of this technology, working closely with the development groups and applying the ecosystem to HP’s own Web commerce and analytics. This experience is yielding information on how to further improve the system.

HP services is also working closely with the Big Data groups to prepare the programs to help users understand where the Big Data ecosystem can provide the most advantage as well as how to install and use it.

HP also realized early that Big Data goes hand-in-hand with cloud. “We are making sure every solution we come out with is private cloud ready,” Goyal said. “We announced today that we are bringing Autonomy and Vertical to the cloud.” And HP works closely with public cloud providers to ensure that its private cloud systems have elasticity so that they will work seamlessly with public Infrastructure-as-a-Service providers in the hybrid cloud architectures that are growing in popularity in the enterprise market.

“HP tends to make single big steps that are equivalent to 10 smaller steps by other vendors,” he said. “HP is poised to make that giant step to leap forward in big data. We have made the decision that this will be big and that we will lead it.”