AI development has been swift, meaning regulation needs to catch up, and governance is somewhat of an afterthought.
There are few formal processes for ensuring AI safety, and researchers say it’s down to the industry to change that.
Many AI leaders and figures within the industry have compared AI to nuclear energy. There are many similarities – AI is growing quickly, it poses a potentially existential risk, and it’s complicated by foreign affairs, this time between the emergent AI superpowers of the US and China.
It was decades before nuclear engineers convinced the wider scientific community of nuclear reactor safety. The first reactors were built in the late 40s and early 50s, and it wasn’t until the US Atomic Energy Act of 1954 that some level of regulation surfaced.
The US government was aware of the risks of nuclear power, but the USSR brought their first civic reactor online in mid-1954, so they wasted no time constructing their own. In 1955, Commissioner Willard F. Libby said, “Our great hazard is that this great benefit to mankind will be killed aborning by unnecessary regulation.”
Does anything here sound familiar? It was only last month that OpenAI CEO Sam Altman urged forth an international body for AI safety like the International Atomic Energy Agency (IAEA), founded in 1957. Much like nuclear energy, governments around the world are working out how to preserve AI’s benefits while regulating the risks.
The IAEA was an autonomous component of the UN and now has 174 member states. After 1957, there were relatively few nuclear disasters until Chernobyl.
AI should learn from the history of nuclear safety
Heidy Khlaaf, an engineering director at cyber security consultancy Trail of Bits, used to assess and verify the safety of nuclear plants. She recently told MIT that nuclear power plants require thousands of documents certifying their safety, delving into the tiniest nuances of each individual component.
After Chernobyl, nuclear regulation became extremely strict. Planning, licensing, and building a nuclear reactor can take 10 years or more, partly because the process is so rigorously monitored at every juncture. Khlaaf notes that this has nothing to do with profits – nuclear safety is born from genuine existential risk.
In contrast, AI companies report their models using simple ‘cards’ that essentially list functions.
In addition, the model’s inner workings are often ‘black box,’ meaning decision-making processes are largely closed off to external observers.
Even ChatGPT, contrary to OpenAI’s namesake, is a black box, and AI companies have already established a reputation for being cagey about their models and training data.
Imagine that nuclear reactors were a ‘black box,’ and their developers wouldn’t tell the public how they work?
AI risks analysis needs to be systematic
To mitigate the risks AI leaders like Altman readily admit, companies must go deeper with their traceability strategies. This involves rigorous monitoring from the very start of the AI development process.
Khlaaf says, “You need to have a systematic way to go through the risks. It’s not a scenario where you just go, ‘Oh, this could happen. Let me just write it down.’”
Currently, there is no established process for AI risk assessment, but there have been some attempts to create one.
For instance, DeepMind recently published a technical blog in collaboration with several universities, specifying that models should be evaluated for “extreme risks” before training. The paper puts forward two strategies:
- Evaluating the extent of a model’s ‘dangerous capabilities’ that could be used to threaten security, exert influence, or evade human oversight.
- Evaluating the extent of a model’s chance of causing harm, e.g., whether it risks not behaving as intended.
Toby Shevlane, one of the researchers on the project, said, “Leading AI companies that are pushing forward the frontier have a responsibility to be watchful of emerging issues and spot them early so that we can address them as soon as possible.”
If AI is to learn from nuclear safety, then early regulation is critical, but long-term complacency is lethal. No one knows what an AI version of Chernobyl or Fukushima would look like, and no one wants to find out.
