The world’s top performing artificial intelligence models, including OpenAI’s o3 and 04-mini, Google LLC’s Gemini 2.5 Pro and Gemini 2.5 Flash, Anthropic’s Claude Opus 4, and xAI Corp.’s Grok 4 are set to go head-to-head on the chess board.

The three-day AI chess battle is the first in a series of tournaments set to be hosted by Google’s data science community Kaggle, within a newly developed Game Arena. There, the models will compete against each other in a range of strategic games designed to evaluate their thinking and reasoning capabilities.

Google DeepMind and Kaggle are partnering with Chess.com, the chess app Take Take Take and legendary chess live streamers Levy Rozman and Hikaru Nakamura on the tournament, with the first simulations set to begin tomorrow.

The Kaggle Game Arena is a new AI benchmarking platform that’s designed to test how competitive large language models are in a series of strategic games, including Go and Werewolf. But first up is the AI chess exhibition, which runs Aug. 5-7, with the simulated games livestreamed on Kaggle.com. Hikaru Nakamura will provide commentary on each of the matchups, while Levy Rozman will deliver a daily recap of each day’s battles, complete with analysis, on the GothamChess YouTube channel. The tournament will conclude with a stream of the championship match-up and tournament recap from Magnus Carlsen on the Take Take Take YouTube channel.

Checkmate

There will be eight competitors battling for chess supremacy: Gemini 2.5 Pro, Gemini 2.5 Flash, Claude Opus 4, DeepSeek-R1, Moonshot’s Kimi 2-K2-Instruct, o3, o4-mini and Grok 4. The tournament will be based on a standard, single-elimination bracket format, where the winners of each match will be decided over a best-of-four series of games. Kaggle Game Arena will livestream one round each day, so the first round will involve four matchups of eight models at the quarter-finals stage, followed by two matchups in the semi-final round on day two, and a single, final matchup on day three.

In a blog post, Google outlined a number of rules, saying that the models will be responding to text-based inputs. None of the competing models will be allowed to access any third-party tools, so they can’t just use the Stockfish chess engine to identify the best move in any situation. Instead, they’ll have to think about it themselves.

The models will not be given a list of possible, legal moves, and if one attempts to make such a move, it will be allowed three retries. Should it fail to make a legal move, it will forfeit the game. Moreover, there will be a 60-minute time limit for each move.

The livestream will attempt to show how each of the competing models “reasons” about its next move, and its response to any failed moves.

Besides the tournament, Kaggle will also create a more comprehensive leaderboard that ranks each of the models, based on their performance in hundreds of “behind the scenes” games that won’t be livestreamed. Each model will be pitted against a rival model multiple times, with the matchups being chosen randomly. The idea is that this will allow Kaggle to create a more robust leaderboard that serves as a comprehensive benchmark of each model’s chess playing capabilities.

“While the tournament is a fun way to spectate and learn how different models play chess in the Game Arena environment, the final leaderboard will represent the rigorous benchmark of the models’ capabilities at chess that we maintain over time,” said Kaggle Product Manager Meg Risdal.

Holger Mueller of Constellation Research Inc. said chess is a fun way to evaluate the reasoning capabilities of AI models and believes there will be a lot of interest in the tournament from AI enthusiasts. But he said most people will be aware that just because an AI model kicks ass in chess, that doesn’t necessarily mean it’s suitable for enterprise workloads

“Esports is coming to LLMs, and it will be interesting to see if the major AI developers start training their models to perform better in these kinds of competitions, especially with more games on the way,” Mueller said. “However, the tournament really only holds entertainment value, and while it will provide some interesting insights, the ability to win a chess match is unlikely to sway enterprise executives who are more interested in how it can automate business work.”

Evaluating real-world skills

Google said it’s launching the Kaggle Game Arena because games like chess represent one of the best ways to carry out a robust evaluation of an LLM’s reasoning capabilities.

That’s because games are resilient to what Google calls “saturation,” or in other words, being solved using a standard formula. Chess, Go and other games are hugely complex, and no two matches are ever the same, which means that the difficulty level increases as each competitor improves. The Werewolf game, meanwhile, is able to test essential enterprise skills, such as navigating through incomplete information and balancing collaboration with competition.

In addition, Google says games act like a proxy for real-world skills, testing a model’s capabilities in terms of strategic planning, memory, reasoning, adaptation, deception and “theory of mind,” or the ability to try to predict an opponent’s thoughts. Meanwhile, team games such as Werewolf can help to evaluate each model’s communication and coordination skills.

Kaggle’s new Game Arena will showcase both current and upcoming livestreamed tournaments, and each game will have its own, dedicated page that lists the leaderboards of ranked models, matchup results, and specific details of the open-source game environment and its rules. The leaderboards will update dynamically as each model plays more games, and newer models are added to the rankings.

In future, Kaggle Game Arena will expand to include more complex, multiplayer video games and real-world simulations in order to generate more comprehensive benchmarks that evaluate an expanding array of AI model skills.

Image: DeepMind