MLCommons, the nonprofit entity that’s best known for its artificial intelligence performance benchmarks, said today it is forming a new MLPerf Client working group.

The aim of the group is to establish benchmarks to measure the capabilities of desktops, laptops and workstations that run AI workloads locally. The new client AI benchmarks will help to answer questions about which laptops and computers will be able to run different generative AI-powered applications and workloads faster, in comparison with other products.

As AI moves from the cloud to run locally on devices, such questions will become more important for consumers. They could make the difference between generating an image in a couple of seconds, or having to wait several minutes.

MLCommons is best known for its cloud computing-based standards, with most of its benchmarks designed to test the performance of processors such as Nvidia Corp.’s graphics processing units and Intel Corp’s rival silicon. It has become the industry gold standard for AI data center benchmarks. Now it wants to achieve the same level of respect with client systems, so it can guide businesses’ and consumers’ purchases.

The organization believes that as the importance of AI grows, the ability to run such workloads locally will become a vital aspect of the computing experience. But each device is different, with literally hundreds of different chips available that all incorporate different AI hardware acceleration capabilities. Moreover, each operating system boasts its own AI-powered features, and device makers and application developers are also adding their own AI features to try and boost creativity.

For the consumer, it’s almost impossible to know which kind of computer will best run the AI capabilities they need. As such, there’s a clear need for a reliable yardstick to measure the AI performance and efficiency of each system.

In a blog post, MLCommons said the new benchmarks will be “scenario-based,” which means they’ll be focused on real-world use cases and “grounded in feedback from the community.”

The first benchmark for client devices is focused on generative AI text-generation. It measures the performance of laptops, desktops and workstations when running Meta Platforms Inc.’s Llama 2. It’s notable that Meta has collaborated closely with Microsoft Corp. and Qualcomm Inc. to optimize Llama 2 for Windows and some mobile devices.

Joining Microsoft and Qualcomm in the MLPerf Client working group are companies that include Advanced Micro Devices Inc., Arm Plc, ASUSTek Computing Inc., Dell Technologies Inc., Intel, Lenovo Group Ltd. and Nvidia.

David Kanter, executive director of MLCommons, said the time is ripe to bring MLPerf to client systems as the importance of AI grows. “Large language models are a natural and exciting starting point for our MLPerf Client working group,” he said. “We look forward to teaming up with our members to bring the excellence of MLPerf into client systems and drive new capabilities for the broader community.”

Jani Joki, Nvidia’s director of performance benchmarking, said the MLPerf benchmarks have served as essential measuring sticks for the advances in machine learning performance and efficiency in the data center. “We look forward to contributing to the creation of benchmarks that will serve a similar role in client systems,” he added.

One notable absentee from the list of collaborators is Apple Inc., and that means it’s unlikely that its MacBook devices will be subject to whatever tests MLCommons is able to come up with. That said, Apple may be forced to participate in the future as it looks increasingly likely that generative AI is here to stay. Assuming that’s the case, the ability to run generative AI on-device will likely be a key consideration in people’s buying decisions.

Given the participation of Qualcomm and Arm in the working group, the new MLCommons client benchmarks may one day be expanded to include smartphones and tablets too.

Image: MLCommons