OpenAI LP is forming a new research team to tackle the potential dangers that could be posed by future “superintelligent” machine learning models.

OpenAI Chief Scientist Ilya Sutskever and Jan Leike, the startup’s head of alignment, announced the move in a blog post today. Sutskever and Leike will co-lead the new research team.

In their blog post, the two researchers expressed the view that artificial intelligence models exhibiting “superintelligence” could arrive by the end of the decade. “Superintelligence will be the most impactful technology humanity has ever invented,” Sutskever and Leike wrote. “But the vast power of superintelligence could also be very dangerous, and could lead to the disempowerment of humanity or even human extinction.”

Addressing that risk, the researchers continued, will require developing a new approach to supervising AI. Developing such an approach is the focus of the research team OpenAI announced today.

The startup believes that current approaches to preventing AI harms are insufficient because they rely on humans to scrutinize machine learning models. “Humans won’t be able to reliably supervise AI systems much smarter than us,” Sutskever and Leike explained. They believe the answer to the challenge is to develop a “roughly human-level automated alignment researcher” powered by AI.

OpenAI’s newly formed research team will lead the effort. The effort, the startup detailed, will revolve around three main objectives. 

The first priority is to develop a way of training the “automated alignment researcher” OpenAI hopes to build. According to the startup, scientists will have to teach the system to supervise aspects of superintelligence AI models they themselves may not fully understand. “We want to understand and control how our models generalize our oversight to tasks we can’t supervise,” Sutskever and Leike detailed. 

After OpenAI develops its automated alignment researcher, it plans to validate the system’s reliability. That task is the second main priority of the startup’s newly formed research team. According to OpenAI, the team plans to check that the AI model works as intended using two main methods.

The first method will involve searching for so-called robustness issues. That’s the term for harmful output generated by AI models. In the past, OpenAI’s AI robustness research has also prioritized finding user prompts that can lead a neural network to generate harmful output.

The second way OpenAI plans to verify the reliability of its automated alignment researcher is by conducting so-called interpretability research. In some cases, analyzing the input and output of an AI model is insufficient to find potential issues. Interpretability research involves evaluating an AI neural network’s internal components, such as its artificial neurons, to identify malfunctions.

The third priority of OpenAI’s new research team will be “stress testing” the system. The startup plans to train misaligned models with the goal of using them to test its automated alignment researcher’s effectiveness.

“We expect our research priorities will evolve substantially as we learn more about the problem and we’ll likely add entirely new research areas,” Sutskever and Leike added. 

OpenAI detailed that Sutskever, who is not only the startup’s chief scientist but also one of its co-founders, will make the initiative his core research focus. The new research team leading effort comprises Sutskever, Leike and members of the startup’s existing alignment group. It will also include researchers from other OpenAI units as well as “new researchers and engineers.”

The team is set to receive 20% of the compute infrastructure that will be available to OpenAI in the next four years. Since 2019, Microsoft Corp. has reportedly provided the AI startup with billions of dollars’ worth of Azure cloud resources. Last month, CNBC reported that the technology giant may spend billions of dollars more to rent infrastructure for OpenAI from CoreWeave Inc., a venture-backed cloud startup. 

Image: OpenAI