About 60 percent of organizations are in the process of shifting resources from their existing data management investments to Hadoop, according to research from Wikibon.org, while 30 percent intend to begin on the journey by the end of the year. For Chris Selland, a former longtime analyst and current vice president of marketing and business development at Hewlett-Packard Co.’s Vertica unit, that is as clear a sign as any that the analytics buzz is finally turning into business reality.

As SiliconANGLE kickoff coverage of this year’s HP Vertica event, we recount Selland’s most recent return to our roving news desk, theCUBE, at HP Discover 2014. Selland shares his unique insight into the state of the market and Vertica’s place in the industry with theCUBE hosts Wikibon co-founder Dave Vellante Jeff Frick. He made clear right off the bat that, from where he’s standing, the Big Data train has already reached full throttle.

“Across the board, everybody is realizing that there is a fundamental shift going on today in how business gets done and how government gets done,” he noted. “It’s all around analytics, it’s all around understanding customers, suppliers and operations.”

Hadoop adoption rising, but commercial apps remain low

Although reflective of the growth happening in the market, the surging interest in Hadoop among enterprises and government agencies should be taken with a grain of salt. Wikibon analysts found that, while adoption of the platform is accelerating, only about a quarter of users have deployed a commercial distribution, which Selland noted confirms another pattern that HP has been seeing recently.

“This is what we hear all the time from customers – is that they really haven’t figured out exactly what they’re going to do with all this technology yet,” he elaborated. “But there’s a desire to upgrade what they have been doing because they’re starting to realize the infrastructure they have in place, the legacy data warehouses, are running out of gas and can’t support all these news data types.”

HP Vertica bridges old and new worlds

The Vertica database is touted as something of a bridge between old and new that can make the transition less painful for CIOs without forcing any compromises along the way. According to Selland, that is the result of a “store, explore and serve” methodology that focuses in equal part on analyzing the data and making the resulting insights accessible to the business users who need them.

A crucial aspect of that, he continued, is holding onto every last bit of data that flows into the corporate network. But in terabyte- and petabyte-scale environments, that is much easier said than done. The answer to the challenge is Hadoop, which Selland highlighted has been tightly integrated into Vertica with the latest “Dragline” release introduced in the end of May.

• Hadoop integration is key

The new version also brought with it a mixed workloads management feature that automatically optimizes the amount of resources used based on the requirements of specific requests and enhanced structured query functionality for Hadoop, among other improvements. Together, the upgrades span the entirety of what Selland calls the analytics continuum.

“So you go across that continuum of ‘store, explore, serve’: you go backwards and say ‘what do I need in real-time, what not necessarily, what do I need to keep around and what’s the stuff in the middle,’ so there are different tiers and different levels of data,” he detailed.

That is true in the most literal sense, Selland explained: the telcos, cable providers and other large organizations that utilize Vertica in their environments have terabytes if not petabytes of data under management, but for the end-user that ultimately benefits from the technology, only the specific subset relevant to them matters. He said that Dragline includes a function that accelerates the delivery of those individual “chunks”, as he put it, without diminishing the ability of corporate customers to manage their fast-growing information troves.

The integration with Hadoop ties directly into that effort to balance experience with scale. The batch processing platform provides a cost-effective environment vast amounts of data, which makes it uniquely important to HP’s plans for Vertica and its broader analytics strategy. Selland detailed that his company’s current approach of partnering with all the major distributors, rather than putting all of its eggs in one basket as Intel and Google have  or developing a distro of its own, is the result of a very deliberate strategy centered squarely on the customer.

“We are working very hard to partner with all… it’s not about being Switzerland but it’s what customers are saying,” he explained. “When we ask ‘should we have our own Hadoop distribution?’ customers don’t want that, because there’s so much experimentation going on, because so many of them are using the free distros. So the neutrality is about being customer-friendly.”

photo credit: Thomas Hawk via photopin cc