Data lakehouses are emerging as the backbone of enterprise artificial intelligence, promising unified access to structured and unstructured data. But building and managing them is no simple feat.

From ingestion to extraction, enterprises are grappling with complexity — and that’s where storage leaders such as Super Micro Computer Inc. are stepping in. By teaming with partners across compute, open storage and databases, Supermicro is helping streamline ETL pipelines so that data can flow into AI models with speed and precision.

Industry experts from Supermicro, AMD, MinIO and EDB discuss the growth of open storage.

“Our partnerships with everybody, in terms of software defined storage, has been really one of our great strengths,” said Paul McLeod (pictured, back row left), product director, storage systems at Supermicro. “Supermicro has always been software defined. And ever since I’ve joined Supermicro, that’s been my charge … to go after all the partners we can to make sure that we have solutions, whether those are compute solutions or whether those are storage.”

McLeod along with Shiva Gurumurthy (back row right), global partner marketing manager of database and analytics at Advanced Micro Devices Inc.; Brenna Buuck (front row right), developer evangelist, MinIO Inc.; and Simon Lightstone (front row left), director, technology product manager at Enterprise DB Corp., spoke with theCUBE’s Rob Strechay at the Supermicro Open Storage Summit interview series, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed what storage looks like in the AI era and designing effective data lakehouses and data lakes. (* Disclosure below.)

Object storage: The core of data lakehouses

Object storage and open table formats are transforming data management. Modern data lakehouses now have three components: object storage, which offers scalability and cost-efficiency, open table formats such as Apache Iceberg or Delta Lake, and the compute engines.

“In this design, the storage and the compute are separate,” Buuck explained. “That also means that this old concept of selling storage and compute together has sort of dissipated in light of this new stack, and now more and more users are able to just choose whichever compute works best for their workloads. We often see our users adopt more than one.”

MinIO now offers AIStor, an object storage system designed for enterprise-grade AI. Its technology is compatible with EDB, a company which provides services based on Postgres, an open-source relational database. EDB gives users a way to manage their Postgres estate from one screen, making it easier for them to leverage data for Retrieval-Augmented Generation pipelines, according to Lightstone.

“We’re providing the hardware and a partnership with Supermicro to provide a solution, which we call the Sovereign Data and AI Factory,” he said. “That just makes it very easy to use all the vendors that are speaking here, MinIO and so forth … to leverage that storage investment, to give that software layer which allows you to automatically move to different tiers of storage as data gets old.”

EDB Postgres AI also enables the movement of data from the operational databases into the lakehouse, while putting it into a tiered format. This process happens in real-time, ensuring that AI models are querying the most recent data.

“You no longer have this delay where you’re running an ETL process at the end of the day,” Lightstone added. “It’s going there automatically into your [data lakehouses] where it’s in columnar format.”

Embracing open storage solutions

AMD is providing the hardware in this new data landscape. The company recently released a series of new DPU products and is working closely with EDB, MinIO and Supermicro to offer the full suite of storage solutions for AI.

“Storage is the critical enabler of all the data that’s out there,” Gurumurthy said. “When we think of AI, we think a lot about compute, but it’s really the disks and the infrastructure behind it that’s running a lot of data today. Obviously, GPUs do a lot of training these days. But to keep it fed with a lot of data, it’s actually the storage devices that are doing it on a daily basis.”

Everyone on the panel highlighted the importance of sticking to open standards in storage. Even as technology progresses rapidly, open storage solutions — including data lakehouses — mean that customers won’t have to redo their entire architecture every time they need to adapt to a new model or platform, according to McLeod.

“At Supermicro, of course, we’re always looking at the performance of our systems,” he said. “I always tell my customers and my partners and everything, ‘Hey, we deliver you the bandwidth. Now, you have to deliver the throughput. So, basically with your application, the doors are wide open, let’s see what we can do together.’”

Here’s a short clip from our interview, part of SiliconANGLE’s and theCUBE’s coverage of the Supermicro Open Storage Summit:

(* Disclosure: TheCUBE is a paid media partner for the Supermicro Open Storage Summit. Neither Super Micro Computer Inc., the sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE