Tokyo-based artificial intelligence startup Sakana AI today unveiled what it says is the first generative AI model in the world designed to conduct scientific research on its own.

The model, called AI Scientist, is the result of a collaboration with scientists from the University of Oxford and the University of British Columbia. Sakana says it has the potential to automate many of the processes involved in scientific discovery.

According to its creators, The AI Scientist is based on large language models that are able to mimic the scientific discovery process, beginning with idea generation, then running experiments, and writing full scientific papers that outline their discoveries.

“We propose and run a fully AI-driven system for automated scientific discovery, applied to machine learning research,” Sakana AI’s team wrote in an academic paper outlining the new model.

The study outlines how The AI Scientist is designed to perform research in the area of machine learning specifically, and is capable of suggesting ideas such as new techniques for transformer-based language models, diffusion models and analyzing learning dynamics.

Sakana AI claims that the AI Scientist can produce a complete research paper at a cost of just $15, based on the computing resources it uses. Moreover, its papers “exceed the acceptance threshold” of a well known machine learning conference, as judged by an automated review process.

“From ideation, writing code, running experiments and summarizing results, to writing entire papers and conducting peer-review, The AI Scientist opens a new era of AI-driven scientific research and accelerated discovery,” the researchers wrote.

In a blog post, Sakana AI’s team said The AI Scientist employs a broad research direction and a starting codebase, such as the open-source code base of older research on GitHub, and starts its own discovery process. It also claimed that the model can follow the procedure of AI researchers and conduct literature searches, experiment planning, figure generation, manuscript reviewing and more. It can also run in an open-ended loop to improve the next generation of ideas, based on the previous generation.

The AI Scientist appears to be a significant breakthrough in generative AI, with its ability to conduct research processes entirely autonomously. It suggests that Sakana AI’s team has achieved a level of creativity and reasoning that’s much closer to that of humans.

The implications of such a system are significant, as it could potentially pave the way for continuous scientific research, overcoming the limitations of human researchers, who need to take coffee breaks, eat lunch and go home at the end of each day. It could lead to more rapid advances in various fields, such as materials science, drug discovery and climate change mitigation.

Limitations

That said, it appears that a lot of work needs to be done before the day when The AI Scientist can compete on a par with genuine human scientists. For now, the model is restricted to only one specific research area, namely AI software development, and there may also be hardware limitations.

In addition, The AI Scientist still needs an existing codebase as a starting point for its research, which calls into question its ability to innovate and conceive new ideas.

Sakana AI’s team also pointed out that The AI Scientist does not have computer vision capabilities, which limits its ability to fix any visual issues with the papers it produces. It can also be prone to AI hallucinations, which means it can sometimes incorrectly implement ideas or make unfair comparisons to baselines, contaminating its results. In addition, they admitted that the model sometimes makes “critical errors when writing and evaluating results,” especially when comparing the magnitude of two numbers.

Holger Mueller, an analyst with Constellation Research Inc., told SiliconANGLE that science is an interesting use case because it lends itself well to generative AI in many ways. He points out that scientists generally follow rigid and standardized methodologies when conducting research, and there is tons of public data available in the academic world.

“The process of validating and challenging insights is highly structured too,” he said. “But anybody trying to create an automated scientist is going to face a lot of scrutiny, and will be pushed to prove that they can deliver value in daily scientific research. Human supervision will be a critical aspect of this, but from the overall characteristics of science, it’s ideally suited to be automated with AI.”

Though it’s clear The AI Scientist demonstrates an impressive ability to innovate on existing ideas, it remains to be seen if the system will ever be able to propose and outline genuinely original and paradigm-changing ideas. AI systems excel at processing vast amounts of data and identifying patterns within it, but replicating the level of human intuition, creativity and judgment that’s required to steer scientific inquiry into meaningful and beneficial outcomes is a much tougher challenge.

Image: SiliconANGLE/Microsoft Designer