An Anthropic PBC researcher has resigned over concerns that artificial intelligence labs are “gambling with our lives.”

Jacob Coxon was part of the company’s AI pretraining team until today. He announced his resignation in a series of X posts that has been viewed millions of times. Coxon wrote that his decision was motivated by concerns over recursive self-improvement, or RSI. That’s a term for a hypothetical future AI with the ability to automatically improve its own capabilities.

“These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,” Coxon wrote. “The people building AI earnestly believe that it could kill us all by the end of the decade.”

Evan Hubinger, Anthropic’s alignment science lead, reaffirmed the latter point in an X post. “Jacob is correct here — we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” he wrote.

No AI lab has announced a working implementation of RSI yet. However, Anthropic and OpenAI Group PBC are already using AI to automate model development tasks. In April, the former company revealed that Claude had completed an AI research experiment with minimal human input. The project focused on mitigating the risks posed by large language models.

Coxon’s resignation drew responses from not only Hubinger but also hundreds of other industry observers. Many expressed alarm about the concerns he raised, while others voiced doubts. Some commentators pointed out that Coxon’s resignation could have implications for Anthropic’s upcoming initial public offering. “If Anthropic is intellectually honest; then this risk better appear on their S-1,” tweeted AI investor and researcher Azeem Azhar. “This will be the best test on whether they do seriously believe that.”

A number of lawmakers also reacted to the news. Vermont Senator Bernie Sanders wrote on X that “I will soon be introducing legislation to ban superintelligence and pause AI development.” Massachusetts representative Lori Trahan, who recently introduced a bill designed to regulate advanced AI models, stated that “it’s past time for Congress to get off the sidelines and do its job.”

Coxon’s resignation comes on the heels of several landmark advances in AI research. 

On Tuesday, OpenAI revealed that an unreleased model had solved one of the most difficult open problems in mathematics. The algorithm found a fault in a set of equations that scientists use to study the motion of fluids. The Navier-Stokes equations, as they’re called, have applications in fields ranging from healthcare to auto design.

Last week, Anthropic used Claude to develop a computer-verifiable version of an important mathematical proof. The AI created it in 11 days instead of the years that the task was expected to take.

Coxon joins a growing number of AI researchers who have expressed concerns about the rapid advance of frontier models. On Sunday, OpenAI Chief Scientist Jakub Pachocki urged the tech industry to slow down AI development. In July, a group of researchers signed an open letter that warned about the risks of self-improving frontier models.

Image: Anthropic