UPDATED 13:21 EST / NOVEMBER 17 2021


Upgraded MLPerf HPC benchmark helps measure supercomputers’ AI performance

The MLCommons industry group today detailed an upgraded version of MLPerf HPC, its benchmark suite for measuring how fast a supercomputer can train artificial intelligence models.

The group, which is backed by some of the tech industry’s most prominent companies, also shared the results from its latest supercomputer performance contest. The contest was carried out using the new version of the MLPerf HPC benchmark suite that debuted today. Eight supercomputing organizations participated. 

MLCommons is an AI-focused engineering consortium backed by chipmakers such as Nvidia Corp. and Advanced Micro Devices Inc., as well as a long list of other tech firms, including Google LLC. MLCommons is responsible for developing a popular set of benchmark suites used to measure how fast different types of systems can run AI models. 

The benchmark suite that MLCommons updated today, MLPerf HPC, is designed to measure the speed with which a supercomputer can train AI models. MLPerf HPC gives the supercomputing industry a standardized way to compare the machine learning performance of different systems. As a result, the benchmark suite plays an important role in both the supercomputing and AI ecosystems. 

MLPerf HPC 1.0, the latest version of the suite that debuted today, introduces two major improvements. The first is a new benchmark called OpenCatalyst for evaluating how fast supercomputers can carry out certain scientific calculations. The other enhancement, in turn, is a metric for understanding how many AI models a supercomputer can train per minute. 

To take part in a MLPerf HPC performance contest, a participating organization must evaluate its supercomputer’s speed by having the system train AI models specified by MLCommons. Each AI model is optimized for a different set of scientific calculations. By measuring how fast a supercomputer can train different types of neural networks, the MLPerf HPC benchmark suite provides insight into the system’s AI performance.

The new MLPerf HPC 1.0 release expands the number of AI models that can be used to evaluate a supercomputer’s performance by adding in a benchmark called OpenCatalyst. In OpenCatalyst, the task is to train a neural network to predict quantum mechanical properties of atoms in molecules. The calculations in the test make it possible to explore new materials that can be used to build energy storage systems. 

To complete the OpenCatalyst benchmark, a supercomputer must train a specific open-source neural network called DimeNet++. It’s an AI model for analyzing atomic systems that, in MLPerf 1.0, is used to perform quantum mechanical predictions related to energy storage. Test participants must train DimeNet++ with the open-source OC20 dataset, which includes information on 1.2 million molecular phenomena.

“These benchmarks are aimed at measuring the full capabilities of modern supercomputers,” said Murali Emani, the Co-Chair of the MLPerf HPC Working Group. “This iteration of MLPerf HPC will help guide upcoming Exascale systems for emerging machine learning workloads such as AI for science applications.”

Alongside the OpenCatalyst benchmark, the latest MLPerf HPC release adds a new performance metric dubbed weak scaling. According to MLCommons, the metric provides insight into the number of AI models that a supercomputer can train per minute. It does so by measuring the system’s aggregate throughput.

Information on how many neural networks a system can train in a given time frame is useful because supercomputers are often expected to run multiple workloads at the same time. As a result, data on how well a supercomputer can run multiple workloads helps researchers gain a more complete understanding of its performance.

Eight supercomputing organizations tested their systems’ speed in a recent contest that used MLPerf HPC 1.0 with OpenCatalyst and the weak scaling metric. The participants included the Argonne National Laboratory, the Swiss National Supercomputing Centre, Fujitsu Ltd. and Japan’s Institute of Physical and Chemical Research, Helmholtz AI, Lawrence Berkeley National Laboratory, the National Center for Supercomputing Applications, Nvidia Corp., and the Texas Advanced Computing Center.

Nvidia participated with its in-house Selene supercomputer (pictured), which features the chipmaker’s graphics processing units. Several of the other supercomputers in the contest are also equipped with Nvidia silicon. 

According to MLCommons, the best benchmark results were four to seven times better than the performance data logged during the previous contest. Separately, Nvidia noted in a blog post today that systems powered by its chips won four of five tests in the benchmark contest. Nvidia’s own Selene supercomputer secured three top spots in the weak scaling category, which measures how many AI models a system can train per minute, as well as on strong scaling for CosmoFlow and DeepCAM models.

Photo: Nvidia

A message from John Furrier, co-founder of SiliconANGLE:

Show your support for our mission by joining our Cube Club and Cube Event Community of experts. Join the community that includes Amazon Web Services and Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger and many more luminaries and experts.

Join Our Community 

Click here to join the free and open Startup Showcase event.

“TheCUBE is part of re:Invent, you know, you guys really are a part of the event and we really appreciate your coming here and I know people appreciate the content you create as well” – Andy Jassy

We really want to hear from you, and we’re looking forward to seeing you at the event and in theCUBE Club.

Click here to join the free and open Startup Showcase event.