Nuclear war, bioweapons, runaway AI: How researchers rank risks of smart systems

Artificial intelligence researchers have long warned the technology could wipe out humanity, whether by designing an unstoppable disease, triggering nuclear war, or, as one famous thought experiment imagines, converting the entire planet into paper clip factories. Debate over how seriously to take these doomsday scenarios has intensified since several AI executives called for slowing the technology’s development on safety grounds.

Not everyone is convinced the threat is genuine. Juan Andrés Guerrero-Saade, a researcher at cybersecurity firm SentinelOne who also sits on OpenAI’s Frontier Risk Council, argued that many of these scenarios resemble science fiction more than serious risk analysis, suggesting they appeal to a certain kind of thinking and help frontier labs recruit talent.

A nuclear trigger

One long-standing fear holds that AI could manipulate nations into launching nuclear weapons against each other, with the resulting radioactive fallout devastating life on Earth. A RAND Corp. report concluded that current safeguards built into nuclear command and control systems make this scenario implausible for now, though that could change if AI systems were integrated directly into decision-making chains or gained unauthorized system access.

Most researchers see the real danger differently: not an AI agent acting alone, but nations or groups deliberately using the technology against their adversaries.

Engineered pathogens

AI’s potential to accelerate disease research cuts both ways. Anthropic recently said it had blocked attempts by bad actors to misuse its Claude system for cyberattacks, surveillance and biological weapons research, including one request seeking help making the mosquito-borne chikungunya virus more dangerous through targeted mutations. The same RAND report noted that AI could meaningfully advance the design of synthetic pathogens, though it envisioned humans, not autonomous systems, planning and carrying out any resulting attack.

When goals go wrong

A different category of concern centers on what researchers call misalignment, where an AI system acts without authorization or works around human oversight, not out of malicious intent but simply because people become an obstacle to its programmed goal. Oxford philosopher Nick Bostrom illustrated the idea in 2003 with his now-famous “paper clip maximizer,” in which an AI instructed to produce as many paper clips as possible could eventually consume the planet, and then increasing portions of space, to keep manufacturing more.

Former Anthropic researcher Jacob Coxon, who resigned over concerns that AI companies weren’t developing the technology responsibly, recently acknowledged on social media that he still needs to “work on” explaining these risks more clearly, since he finds it “very difficult to convey specific scenarios.”

A digital takeover

Anthropic CEO Dario Amodei has separately warned that swarms of AI agents could become capable of hijacking large portions of the internet through a botnet within months, potentially inflicting billions of dollars in damage through attacks on electrical grids, water systems, transportation networks or financial institutions. Such an event likely wouldn’t end humanity outright, but could cause significant chaos and loss of life.

Past software failures have already exposed how fragile a digitized world can be when it depends on just a handful of core providers. Even so, some experts consider the idea of AI bots overtaking the internet’s famously fragmented infrastructure to be far-fetched.

Source: https://interestingengineering.com/ai-robotics/ai-doomsday-scenarios-existential-risk-debate