The AGI Moment

Introduction

The age of AI is upon us. What that means is not yet clear.

As general-purpose AI capabilities continue to march toward human-level performance across a range of tasks, many researchers have considered the strategic implications of artificial general intelligence (AGI). While definitions vary, AGI is often characterized as human-level intelligence across a wide range of tasks. While current AI systems are generally not considered to perform at this level yet (although there is some dispute about this), AI performance continues to improve. Views on when AGI might be achieved, if at all, range from present day to never.

A growing set of AGI strategies¹ describe how AGI might affect global power and what various actors, such as private AI developers or major governments, ought to do to best manage the transition to AGI. These strategies share the common assumption that AI continues to advance to roughly human-level abilities and sometimes beyond. Yet the types of futures that researchers envision and their associated recommendations wildly diverge. Proposed strategies range from racing to AGI to Cold War-style deterrence to a global moratorium. These strategies differ not only in their approach, but in what problem they are solving, what future they envision, and the core assumptions they make about AI development. While there are some common concerns, such as great power conflict or misaligned AGI, strategies differ in which risks they prioritize and tradeoffs they are willing to accept.

The aim of this paper is more modest than crafting a compelling future scenario and strategy for managing the transition to AGI. Instead, I will provide a brief overview of the landscape of AGI futures and associated strategies that have been written to date and map these to the core assumptions that lead to a given future scenario. Then I explore key dimensions that could affect how the transition to AGI could unfold, including technological, political, and social factors. This analysis elucidates a few key variables, such as the role of computing power, the rate of proliferation, the speed of adoption, and the effect of automating AI research, that could drive very different outcomes. As we explore this landscape of possibilities, we see that some possible futures are relatively underexplored. Finally, I will highlight actionable steps that companies, governments, and civil society can take today to track important developments that could tell us which future we might be headed toward as well as steps that actors could take to increase or decrease the likelihood of certain outcomes.

My goal in exploring these possibilities is not predictive. I have no idea what the future will bring. Instead, I hope to equip readers to better prepare for an uncertain future by widening the aperture of futures that are considered. Many AGI future scenarios are point predictions or vary along only one or two dimensions. These are incredibly useful exercises. It is also true that the space of possibilities is vast and there remain many that are still unexplored. The future is not only unknown but, if AGI or something approximating it comes to pass, could turn out to be deeply unfamiliar.

Range of AGI Scenarios

AGI strategies vary widely not only in what they aim to accomplish but in what future they envision. One end of the spectrum is neatly summed up by the title of Eliezer Yudkowsky and Nate Soares’ book, If Anyone Builds It, Everyone Dies.² In this scenario, AGI leads to an “intelligence explosion” in which AI accelerates further progress in building more advanced AI. This leads to artificial superintelligence, which vastly outperforms even the most intelligent humans. The core argument behind an intelligence explosion dates to the very dawn of the field of AI, in an article by I.J. Good published in 1965:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion,” and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.³

Yudkowsky, Nick Bostrom, Daniel Kokotajlo, and others have expanded on this core idea with scenarios that are more detailed and updated to account for current developments, but the core features of this scenario remain. AI reaches some critical tipping point in intelligence where it can improve itself, allowing AI to advance of its own accord not only beyond human intelligence but beyond human control. At that point, humanity’s destiny is no longer its own—the machine is in charge. Variations of this scenario include the speed of the “takeoff” to superintelligence, the number of competing AI projects, the end state of an intelligence explosion, and to what extent if at all humanity can steer AI development toward a positive future.

On the other end of the spectrum is the view that AI is a “normal technology” much like other strategically important technologies humanity has developed in the past and that historical patterns of how other technologies have affected global power are likely to be relevant for AI. Michael Horowitz, Jeffrey Ding, and Allan Dafoe have each drawn lessons from other general-purpose technologies, such as electricity or the combustion engine, to understand how AI might affect global power.⁴ Arvind Narayanan and Sayash Kapoor explicitly use the term “normal technology,” arguing that AI should be viewed as a tool, not a “humanlike intelligence,” and that the changes AI brings will unfold over decades.⁵ In this view, AI adoption does not immediately follow development; diffusion across society will take time; and while AI might unleash monumental changes across human society (as the industrial revolution did), humanity has many tools and opportunities to address any harms from AI. Building superintelligent AI does not automatically mean that everyone dies.

These are radically different views of the future of AI, and within each of these are a set of assumptions about how AI might unfold, both as a technology and how it is adopted. The remainder of this paper will explore the key assumptions and variables underpinning these competing worldviews.