Highlights:

  • The startup affirms that current methods of preventing AI hazards are insufficient as they depend on humans to analyze ML models.
  • 20% of the computational infrastructure that will be made accessible to OpenAI for the next four years will be included in the team.

To address the possible risks posed by future “superintelligent” machine learning models, OpenAI LP is creating a new research team.

Ilya Sutskever, Chief Scientist at OpenAI, and Jan Leike, Head of Alignment at the startup, announced in a blog post to co-lead the new research group.

Both experts put forth their opinion that AI models featuring “superintelligence” could show up by the decade’s end. “Superintelligence will be the most impactful technology humanity has ever invented. But the vast power of superintelligence could also be very dangerous and could lead to the disempowerment of humanity or even human extinction,” Leike and Sutskever reported.

The researchers said that a new strategy for AI supervision must be developed to address the risks. The research team introduced by OpenAI is primarily interested in creating such an approach.

The startup affirms that current methods of preventing AI hazards are not sufficient as they depend on humans to analyze ML models. Sutskever and Leike mentioned, “Humans won’t be able to reliably supervise AI systems much smarter than us.” They think the response to this hurdle is to create a “roughly human-level automated alignment researcher” backed by AI.

OpenAI’s newly established research team will coordinate the project. The startup described three major goals that will drive the effort.

The first step is to figure out how to train the “automated alignment researcher” that OpenAI wants to create. Scientists will have to train the system to supervise elements of superintelligent AI models that they may not completely grasp, the business claims. “We want to understand and control how our models generalize our oversight to tasks we can’t supervise,” Leike and Sutskever stated.

After creating its automated alignment researcher, OpenAI intends to confirm the system’s dependability. The newly formed team at the startup has that goal as its second top priority. According to OpenAI, the team intends to use two primary techniques to ensure the AI model performs as expected.

The first approach entails looking for alleged robustness problems that describe the negative output produced by AI models. Identifying user cues that could cause a neural network to produce detrimental output has been a priority in OpenAI’s research on AI robustness.

The second way employed by OpenAI is to carry out so-called interpretability research to confirm the dependability of its automated alignment researcher. Analyzing an AI model’s input and output can occasionally fall short of identifying possible problems. A study on interpretability examines an AI neural network’s core part, such as its artificial neurons, to spot errors.

The latest research group at OpenAI will keep “stress testing” the system as its third priority. The startup intends to train misaligned models to use them to evaluate the performance of its automated alignment researcher.

“We expect our research priorities will evolve substantially as we learn more about the problem, and we’ll likely add entirely new research areas,” added Sutskever and Leike.

Sutskever, who is not only the startup’s Chief Scientist but also one of its Co-founders, will make the program his primary area of research, according to a statement from OpenAI. Sutskever, Leike, and the startup’s current alignment group comprise the newly formed team that will coordinate the effort. Along with “new researchers and engineers,” it will also feature researchers from existing OpenAI groups.

Moreover, 20% of the compute infrastructure that will be made accessible to OpenAI for the next four years will be included in the team. Microsoft Corp. has reportedly given the AI company Azure cloud resources worth billions of dollars annually since 2019. According to last month’s report, the technology giant may shell out further billions of dollars to hire CoreWeave Inc., a venture-backed cloud firm, to provide infrastructure for OpenAI.