Google Cloud said today it’s enhancing its cloud storage options to help customers come to grips with the unique demands of artificial intelligence workloads that are notoriously data-intensive.

The three new storage offerings announced today include Parallelstore, Cloud Storage FUSE and Google Cloud NetApp Volumes. They can all be accessed via the Google Cloud Console, providing enterprises with what Google says are more capable and cost-effective data storage solutions for their AI applications.

Parallelstore

According to Google, Parallelstore is a parallel file system for the most demanding AI and high-performance computing applications that rely on graphics processing units. Available now in private preview, Parallelstore is designed to help customers stop wasting precious GPU resources while waiting for their storage to catch up.

Google explained Parallelstore works by keeping GPUs “saturated” with the data required to optimize AI and machine learning training. In this way, it prevents the GPUs from becoming idle and eliminates the wasted expenditure associated with that.

The company said Parallelstore is based on a next-generation distributed asynchronous object storage, or DAOS, architecture that ensures all GPUs within its environment have equal access to storage. It means GPU-based virtual machines can access the data they need instantly, without having to wait. In this way, it’s the most suitable storage option for cloud-based applications that require the highest performance in terms of input/output operations and throughput.

“Parallelstore is a differentiated high-performance solution for when they need to push the limits of I/O patterns, file sizes, latency, and throughput,” Sameet Agarwal, vice president and general manager of Storage, and Sean Derrington, group product manager of Storage, wrote in a co-authored blog post. “For high-performance AI/ML workloads, Parallelstore can be configured to eliminate waste on unnecessary storage so you’re not caught flat-footed with a solution that can’t handle your workload requirements.”

Steve McDowell of NAND Research said the launch of Parallelstore brings a much-needed storage capability to its cloud and will surely be appreciated by the AI community. “Training deep learning models on large datasets, as we do with generative AI and LLMs, requires a high-bandwidth scalable file system, which Google is delivering with Parallelstore,” he told SiliconANGLE. “Beyond delivering this critical functionality to its customers, it also closes a competitive gap with AWS’s FSx for Lustre.”

Cloud Storage FUSE

Cloud Storage FUSE is a new storage offering targeted at AI applications that require file system semantics. Generally available now, it makes it possible for customers to mount and access Google Cloud Storage buckets as local file systems, providing a smoother experience for AI apps that require their own file system to store and access training data, models and checkpoints. It does this cost-effectively, while maintaining the scale, performance and simplicity of Google Cloud Storage, Google said.

McDowell said the process of making object storage look like a local file system has always been extremely cumbersome for enterprises, involving a hodgepodge of different technologies. “Google is really simplifying things,” he explained. “With FUSE as a first-party offering, AI engineers can now more efficiently build solution with easy object storage integration. That’s a powerful enabler and is good not only for AI, but for any analytics stack that uses data stored as objects.”

Agarwal and Derrington outlined four key benefits of Cloud Storage FUSE for AI workloads, with the main one being compatibility, eliminating the need to refactor applications to call cloud-specific application programming interfaces. In addition, Cloud Storage FUSE is integrated with Google’s official Go Cloud Storage client library, and has been validated for the PyTorch and TensorFlow AI frameworks to ensure its reliability.

Moreover, because Cloud Storage FUSE treats Cloud Storage buckets as a local file system, there’s no delay when shifting data to its cloud-based GPUs and tensor processing units, meaning reduced idle time for those resources. Finally, they explained how it’s possible for customers to use pre-built Google ML images to deploy Cloud Storage FUSE as a Linux package within their own environments, meaning it’s a much more portable storage solution.

Google Cloud NetApp Volumes

Finally, Google announced Google Cloud NetApp Volumes for enterprise applications running in the cloud. Available now, it’s aimed at customers that have already architected their apps to run on NetApp storage arrays. It provides them with the high-performance, fully managed storage resources they need to migrate those apps to Google Cloud without making any changes to them, vastly simplifying the process.

Other capabilities include greater scalability, with support for storage volumes of up to 100 terabytes, the ability to implement ONTAP data management for hybrid workloads that run on the cloud and on-premises, and support for both Linux- and Windows-based apps.

“As you adapt to these workloads, we’re trying to make the process as seamless as possible with options tailored to your storage needs,” Agarwal and Derrington said. “With the right storage solution, you can simplify operations, unlock innovation, reduce costs, and position your business to meet the changing needs of your workloads and applications.”

McDowell said this is an important update for Google that closes a competitive gap, as both Amazon Web Services and Microsoft Azure already offer first-party integration with NetApp. “This is good for Google and even better for NetApp, as it really is a tremendous validation for its cloud storage offerings,” he pointed out. “With today’s announcement, NetApp now has native integration with the top three U.S. cloud providers.”

Combined, the three announcements will help Google Cloud to level the playing field with its chief rivals in the cloud. “Google has always been a little behind AWS and Azure in terms of its storage capabilities, but it looks like that is changing,” McDowell added.

Image: Freepik