Over the last decade, a clear trend has emerged. Companies that used to store data without a clear plan for use are now uncovering untapped value through AI-driven data intelligence.

In an industry hurtling toward a new frontier, Databricks Inc. is redefining the possibilities of data and AI, according to Naveen Rao (pictured), vice president of artificial intelligence at Databricks.

Discussing AI-driven data intelligence with Databricks’ Naveen Rao.

“For the last 20 years, we’ve been building data technologies, and the promise had always been, ‘If you log it, you’ll figure out something really amazing to do with it,'” Rao said. “I think now we’re getting there. It took a while, but now I think we’re at a point where we do have some ideas of how to do something interesting like improving your business operationally or even new experiences.”

Rao spoke with theCUBE Research’s John Furrier for theCUBE’s “Cloud AWS re:Invent Coverage,” during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed the transformative landscape of generative AI, data intelligence platforms and the evolving role of developers in building next-generation solutions.

Mapping the industry’s shift to AI-driven data intelligence

The advent of generative AI and data lakehouse architecture has unlocked previously dormant growth channels. Databricks’ innovations, such as the Lakehouse and their data intelligence platform, enable businesses to leverage their unique data to create tailored solutions, improving operations and customer experiences, according to Rao.

“The foundational work goes back to data warehousing where we organize the data, govern it, log it, monitor it and then started adding capabilities with compute on top of it,” he said. “Now we’ve extended that further. We’re building a new foundation with gen AI, and we call this the Data Intelligence Platform. How can I use my data to customize intelligence, and can I build new capabilities that others can’t because I have some unique data?”

By extending foundational data practices such as governance and compute to generative AI, Databricks is empowering enterprises to extract actionable insights from vast data stores. This transition, however, requires robust infrastructure and secure, high-quality pipelines to avoid the pitfalls of past failed projects, Rao noted.

“What we found is something like 85-90% of these gen AI projects never went anywhere,” he said. “It’s not because things didn’t work to some degree, but it’s that the plumbing just wasn’t there.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s “Cloud AWS re:Invent Coverage”:

Photo: SiliconANGLE