Despite rapid advances in artificial intelligence, the enterprise world is still dealing with a data pipeline problem.

More than 80% of enterprise data is unstructured, and 99% of this data is dark to AI because there is no easy solution to query it, according to industry experts. Yet organizations are still trying to build AI systems that require consistent, governed access to this information, while dealing with the challenges of data fragmentation and interoperability.

As a result, data lakes and lakehouse architectures are emerging as critical components of the modern AI stack, and key industry players, including Advanced Micro Devices Inc., Super Micro Computer Inc. and MinIO Inc., are working on solutions to address issues in the data pipeline.

“From what we are seeing, enterprise customers today don’t have a big compute problem, but they actually have a data pipeline problem,” said Varun Selvaraj (pictured, center), business development manager of enterprise AI at AMD. “What’s happening is these customers are now sitting with massive amounts of unstructured data where their infrastructure is really not AI scale. Slowly, the shift is kind of changing towards not compute, but more towards data, moving the data across the system. How effectively can you move this data between systems?”

Selvaraj spoke with Rob Strechay for the Supermicro Open Storage Summit interview series, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. He was joined by Albert Tan (left), staff solution architect at Supermicro, and Greg DeMichillie (right), vice president of product and technical marketing at MinIO. They discussed how enterprises should approach lakehouse architectures, the role of open standards, and how data lakes are evolving to support both traditional analytics and modern AI workloads. (* Disclosure below.)

Storage support for the data pipeline

One factor in the data pipeline problem is that AI processing depends on CPU and GPU compute systems that must be constantly supplied with massive amounts of information. This is where storage can play a central role in the design of lakehouse environments, according to DeMichillie.

“Not only do you have to feed a CPU, the storage layer is the foundation for making sure that you’re actually getting value out of one of the most expensive resources you have, which is the GPU,” DeMichillie told theCUBE. “Keeping those productively fed is an essential part of designing an AI lakehouse. To build a modern lakehouse, you really want to eliminate fragmentation. So, you want all of your data in one storage environment, and we’re able to do that up to exabyte scale.”

As data lakes and lakehouses emerge as key components in the modern AI stack, Supermicro and its partners are working with customers to consolidate data while maintaining flexibility and openness. The company works with MinIO and its AIStor platform, which provides native support for Apache Iceberg tables and the Iceberg REST Catalog. Supermicro relies on AMD EPYC systems to balance CPU resources, PCIe lanes and memory across the data layer.

“Open is really focused on real flexibility, choice and quality,” Tan said. “What open means for us is …we’re able to bring those ‘Lego blocks’ together and fit what you need at the time you need it and in the size you need it. Supermicro brings it all together. We do our testing pre-validated so that you know for sure that when you get this in your data center, it’s pre-validated and ready to go.”

Iceberg supports database functionality

The integration of open solutions such as Apache Iceberg has been a significant element in the evolution of data lakes. Iceberg transforms cloud object stores into high-performance, transactional data lakehouses, enabling full database functionality and multi-engine flexibility without vendor lock-in.

“On the data lake side, I think it’s hard to overstate just what a big deal Iceberg has been,” DeMichillie said. “If you think back, it’s not that long ago that every database had its own proprietary format. The idea of an open table format was just bonkers…but we are really seeing now the incredible widespread adoption. That’s enabled us to do really clever things like integrating support for Iceberg into the storage tier.”

The rise of generative AI has led enterprises into a world where systems have to be carefully balanced with the right I/O, memory and bandwidth, along with the correct GPU sizing. This has been a focus of the three partners in developing solutions that provide the necessary level of performance while moving large amounts of data across systems.

“What we are seeing today is these systems are really complex,” Selvaraj explained. “This is where we come in as MinIO, AMD and Supermicro together. We can address performance across the whole solution, whether it is from the software stack or it is the hardware. The data layer doesn’t fail at a single component, but it breaks with the entire system, so it is very important for us to make sure the whole system is balanced, and the data is designed to flow continuously.”

Stay tuned for the complete video, part of SiliconANGLE’s and theCUBE’s coverage of the Supermicro Open Storage Summit interview series.

(* Disclosure: TheCUBE is a paid media partner for the Supermicro Open Storage Summit interview series. Neither Supermicro, the sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE