OpenAI has assembled a team of experts named “Preparedness,” aiming to mitigate the “catastrophic” risks associated with AI.
This comes not long after OpenAI comprehensively rebranded itself as an AGI-centric company. The new team is dedicated to evaluating both current and future AI models for risks.
Risks specified by OpenAI individualized persuasion (tailoring messages to precisely what the recipient wants to hear), threats to cybersecurity, autonomous replication, adaptation (AI altering itself independently), and even existential threats such as chemical, biological, and nuclear attacks.
Experts are largely divided over whether AI’s existential risks are hype-driven or practically realistic, though most agree that current problems, such as AI deep fakes, are already tremendously disruptive.
OpenAI’s statement on the matter is clear: “We believe that frontier AI models, which will exceed the capabilities of the most advanced existing models, have the potential to benefit all of humanity. However, they also pose increasingly severe risks.”
Artificial general intelligence – AGI – defines AI models that equal and surpass human cognition across multiple modalities.
OpenAI CEO Sam Altman has made his intention to create such models clear, but the concept of AGI in real terms remains highly speculative.
Altman recently admitted on the Joe Rogan podcast that AI hadn’t developed as he expected. AGI might not occur spontaneously as a ‘singularity’ but slowly over the forthcoming decade or so.
OpenAI on Preparedness
In a recent blog post, OpenAI Safety & Alignment and Community teams highlighted the organization’s ongoing commitment to addressing the full spectrum of safety risks associated with AI.
This move comes after OpenAI, along with other prominent AI labs, voluntarily committed to upholding safety, security, and trust in AI development as part of a collaboration with the White House.
The Preparedness team is led by OpenAI researcher Aleksander Madry. The team will be crucial in assessing capabilities, conducting evaluations, and performing internal red teaming for frontier models, ranging from near-future developments to AGI models.
Additionally, the team will develop and maintain a “Risk-Informed Development Policy (RDP),” outlining an approach to evaluating and monitoring frontier model capabilities, implementing protective measures, and establishing governance structures for accountability and oversight.
OpenAI is also launching the “AI Preparedness Challenge” to uncover areas of AI misuse.
The challenge will reward top submissions with $25,000 in API credits, publish novel ideas and entries, and potentially recruit candidates for the Preparedness team.
