Informatica Corp.’s Todd Goldman

Data integration software has become the single most widely used category of solutions in new analytics projects, according to recently published Wikibon research, a trend that Informatica Corp.’s Todd Goldman sees as a natural part of the evolution of Hadoop. He appeared on theCUBE at SiliconANGLE’s recently concluded BigDataNYC 2014 meet-up to explain why that’s the case and to share his perspective on the underlying drivers behind that industry-wide change of focus with hosts Jeff Kelly and Dave Vellante.

Over the last year, “people have gone from sandbox environments where they’re using maybe one or two sources that they can do via hand-coding” to larger scale deployments aggregating data from a multitude of systems, Goldman said. Past a certain threshold, he continued, manually combining different types of information into a consistent and and error-free whole  becomes impractical for even the most well-funded IT organizations. That creates the need for automated tooling such as the kind Informatica sells.

The challenge of data integration and quality management is hardly new, but Goldman explained that the batch processing platform presents a different environment with its own unique set of complexities that require organizations to adjust the way they consolidate and reconcile their information sources.

“If you hand-code everything, it’s very hard to figure that out, and this is where tooling and visualization of these environments comes in,” he elaborated. “This was true with classic data warehouses, too The issue is that the process of prototyping and build took too long.” Hadoop eliminates much of that hassle, freeing up time once spent on working around the limitations of the underlying architecture for the more productive analytics phase.

The Hadoop batch processing framework provides the foundation, but it’s not a silver bullet for removing complexity and data integration problems. For that, Goldman said organizations have to turn to specialized solutions such as Informatica’s, which hide the messy implementation details under an abstraction layer that borrows a page from the Apple playbook of user experience.

In the same way that few Macintosh users want to interact with the underlying Unix operating system, the power of integration tools is in the increased efficiency and accuracy they provide when messy implementation details are out of the way, he said.

That value proposition extends beyond the information being ingested to the metadata describing it. Metadata is a powerful tool for ensuring that the workflow is complete and correct, Goldman noted. “If there a report sitting in MicroStrategy or Tableau or BusinesObjects that the CEO is depending on and I change an attribute that’s feeding it, I’d better make sure that I know what the downstream impact of that is,” he noted.

The trick is to balance governance and new Hadoop projects without compromising one or the other. Goldman said that’s essential to the success of enterprise-scale analytics initiatives. “The [customers] who just do Hadoop and don’t figure out how to operationalize it hit a bump where they have a great project but can’t roll it out to the rest of the organization,” he explained.

Hadoop demands a combination of skills that are scarce in the job market, making it all but impossible for CIOs to hire enough talent to manually scale their Hadoop clusters to the enterprise level.

Abstraction is again the answer, he said. “There are billions of dollars of investment going to hide that complexity, to make it the Unix operating system underneath a graphical interface for data integration, preparation, analytics and building new kinds of applications,” Goldman concluded.

Watch the full interview (18:42)