August 14, 2023

How to Prevent an AI Catastrophe

Society Must Get Ready for Very Powerful Artificial Intelligence

In April 2023, a group of academics at Carnegie Mellon University set out to test the chemistry powers of artificial intelligence. To do so, they connected an AI system to a hypothetical laboratory. Then they asked it to produce various substances. With just two words of guidance—“synthesize ibuprofen”—the chemists got the system to identify the steps necessary for laboratory machines to manufacture the painkiller. The AI, as it turned out, knew both the recipe for ibuprofen and how to produce it.

How dangerous is AI? The honest and scary answer is that no one knows.

Unfortunately, the researchers quickly discovered that their AI tool would synthesize chemicals far more dangerous than Advil. The program was happy to craft instruction to produce a World War I–era chemical weapon and a common date-rape drug. It almost agreed to synthesize sarin, the notoriously lethal nerve gas, until it Googled the compound’s dark history. The researchers found this safeguard to be cold comfort. “The search function,” they wrote, “can be easily manipulated by altering the terminology.” AI, the chemists concluded, can make devastating weapons.

Read the full article from Foreign Affairs.

  • Commentary
    • April 29, 2025
    Five Objectives to Guide U.S. AI Diffusion

    The Framework for AI Diffusion (the Framework) is an ambitious proposal to shape the global distribution of critical AI capabilities, maintain U.S. AI leadership, and prevent ...

    By Janet Egan & Spencer Michaels

  • Commentary
    • Just Security
    • April 25, 2025
    Shaping the World’s AI Future: How the U.S. and China Compete to Promote Their Digital Visions

    As the United States navigates evolving global AI competition, balancing these elements will be crucial in determining whose AI systems — and by extension, whose approaches, v...

    By Keegan McBride

  • Reports
    • April 24, 2025
    Countering the Digital Silk Road: Brazil

    Project Overview This year marks the 10th anniversary of the Digital Silk Road (DSR), China’s ambitious initiative to shape critical digital infrastructure around the world to...

    By Ruby Scanlon & Bill Drexel

  • Reports
    • April 22, 2025
    Promethean Rivalry

    Executive Summary Just as nuclear weapons revolutionized 20th-century geopolitics, artificial intelligence (AI) is primed to transform 21st-century power dynamics—with world l...

    By Bill Drexel

View All Reports View All Articles & Multimedia