Databricks Inc. has emerged as platform tailored for the integration of data-driven artificial intelligence within a unified stack. The 11-year-old company has built its reputation by redefining the separation of storage from compute, a process that has positioned it to become a valuable resource for enterprises seeking to leverage generative AI.

In Databricks’ version of the data lakehouse, storage and compute can come from different vendors, giving customers a wider choice of price/performance capabilities. Databricks’ data lakehouse architecture is designed to facilitate a simpler solution for customers to manage their end-to-end analytics data estate.

“Historically, you’ve had different silos or different tech stacks for your data stack and your AI stack,” said Jason Pohl (pictured, center), senior director of data management for Databricks. “Now these are coming together with the lakehouse, because whenever you want to train your models you want to be able to train them in a distributed fashion over open data sets. Lakehouse has facilitated that.”

Pohl spoke with Shelly Kramer (left), managing director and principal analyst at theCUBE Research, and George Gilbert (right), senior analyst at theCUBE Research, in the latest episode of “The Road to Intelligent Data Apps,” theCUBE’s continuing conversation about the Sixth Data Platform, an emerging framework where the leading vendors are Databricks, Snowflake, AWS, Azure and Google. They discussed the company’s technology model and how customers are adopting new approaches for the implementation of generative AI.

Databricks’ Unity Catalog enables end-to-end governance

In addition to providing storage and compute from different vendors, Databricks also enables a process by which end-to-end data management can be shifted to end-to-end data governance. The company facilitates this through its Unity Catalog, metadata stored in a distributed database and hosted in the Databricks control plane.

“You can register different databases, those could be Postgres, MySQL, Oracle, or even Snowflake, and register those as what we call foreign catalogs within Unity,” Pohl explained. “Unity has context about these datasets and when it goes to generate queries for the data intelligence, it can do so against those. It’s capturing and persisting a whole bunch of metadata.”

Along with governance, Databricks has also taken steps to address the need for speed in model training. Last year, Databricks acquired generative AI startup MosaicML for $1.3 billion, a move designed to open new opportunities for AI training, fine-tuning and deployment.

“If you can train that model faster than somebody else then you can operate these models much cheaper,” Pohl said. “With the technology we have with Mosaic AI, we can train the same model as PyTorch, but we can do it seven or eight times faster.”

Databricks’ customers are leveraging these and other technologies to gain efficiencies in critical work tasks.

“One of our customers is Chevron, and they’ve got all of these people out in the field that are fixing oil rigs and everything else,” Pohl said. “There’s a mountain of instruction manuals and parts manuals and what to do. We’ve helped them create LLMs where they just load that unstructured data, and now the people in the field can just ask a question about what they’ll need.”

Here is the complete conversation, part of the “Road to Intelligent Data Apps” series:

Image: SiliconANGLE