August 14, 2023
How to Prevent an AI Catastrophe
Society Must Get Ready for Very Powerful Artificial Intelligence
In April 2023, a group of academics at Carnegie Mellon University set out to test the chemistry powers of artificial intelligence. To do so, they connected an AI system to a hypothetical laboratory. Then they asked it to produce various substances. With just two words of guidance—“synthesize ibuprofen”—the chemists got the system to identify the steps necessary for laboratory machines to manufacture the painkiller. The AI, as it turned out, knew both the recipe for ibuprofen and how to produce it.
How dangerous is AI? The honest and scary answer is that no one knows.
Unfortunately, the researchers quickly discovered that their AI tool would synthesize chemicals far more dangerous than Advil. The program was happy to craft instruction to produce a World War I–era chemical weapon and a common date-rape drug. It almost agreed to synthesize sarin, the notoriously lethal nerve gas, until it Googled the compound’s dark history. The researchers found this safeguard to be cold comfort. “The search function,” they wrote, “can be easily manipulated by altering the terminology.” AI, the chemists concluded, can make devastating weapons.
Read the full article from Foreign Affairs.
More from CNAS
-
Technology & National Security
Countering the Digital Silk RoadExecutive Summary The year 2025 marks the 10th anniversary of the Digital Silk Road (DSR), China’s effort to strengthen its global ties and influence through technology. In th...
By Vivek Chilukuri & Ruby Scanlon
-
Technology & National Security
Microsoft Announcement Highlights Complicated Relationship Between Big Tech and WarMicrosoft restricted the Israeli military's access to some of its technology after it found that Israel's Defense Ministry was using its services to carry out mass surveillanc...
By Paul Scharre
-
Technology & National Security
Quantum Sensing at Scale: Navigating Commercialization RoadblocksQuantum sensing is racing forward in the lab—but turning prototypes into products still means wrestling with supply chains, certification, and unit economics. In “Quantum Sens...
By Constanza M. Vidal Bustamante
-
Technology & National Security
Constanza Bustamante, Research Fellow at the Center for a New American Security (CNAS) Joins the Superposition Guy’s PodcastConstanza Bustamante, a fellow at the Center for a New American Security (CNAS) is interviewed by Yuval Boger to discuss quantum policy at the nexus of national and economic s...
By Constanza M. Vidal Bustamante