August 14, 2023
How to Prevent an AI Catastrophe
Society Must Get Ready for Very Powerful Artificial Intelligence
In April 2023, a group of academics at Carnegie Mellon University set out to test the chemistry powers of artificial intelligence. To do so, they connected an AI system to a hypothetical laboratory. Then they asked it to produce various substances. With just two words of guidance—“synthesize ibuprofen”—the chemists got the system to identify the steps necessary for laboratory machines to manufacture the painkiller. The AI, as it turned out, knew both the recipe for ibuprofen and how to produce it.
How dangerous is AI? The honest and scary answer is that no one knows.
Unfortunately, the researchers quickly discovered that their AI tool would synthesize chemicals far more dangerous than Advil. The program was happy to craft instruction to produce a World War I–era chemical weapon and a common date-rape drug. It almost agreed to synthesize sarin, the notoriously lethal nerve gas, until it Googled the compound’s dark history. The researchers found this safeguard to be cold comfort. “The search function,” they wrote, “can be easily manipulated by altering the terminology.” AI, the chemists concluded, can make devastating weapons.
Read the full article from Foreign Affairs.
More from CNAS
-
CNAS Insights
Technology & National Security
CNAS Insights | Governing Jailbreak IncidentsIn June 2026, Anthropic publicly released Claude Fable 5, a restricted version of its highly cyber-capable Mythos model. Within days, reports reached U.S. officials that resea...
By Ben Hayum
-
Commentary
Technology & National Security
Closing the Remote Access LoopThis article was originally published in Issues in Science and Technology. As Asad Ramzanali argues in “Why the Cloud Needs Competition” (Issues, Winter 2026), cloud computing...
By Michelle Nie
-
Commentary
Technology & National Security
Losing the War of the FutureThis article was originally published in Foreign Affairs. In its recent campaign against Iran, the United States dominated the skies using its traditional airpower. The U.S. m...
By Paul Scharre
-
Podcast
Technology & National Security
We Need AI Treaties. This Is How We Get ThemIn this episode, Tristan sits down with two experts in this field to discuss the kinds of verification technology we need for AI, the challenges of building it, and the world ...
By Janet Egan & Tim Fist
