Superintelligence and long-term risk

Superintelligence: AI beyond human level#

A superintelligent AI would do a wide range of tasks better than any human can.1 That breadth is the difference: some systems already beat every human, but only at one narrow job.1

A 2023 paper called Levels of AGI, posted online by Meredith Morris, rates AI systems on two scales.2 Depth means how well a system performs, and breadth means how wide a range of tasks it can do.2 The top of the depth scale is superhuman, meaning better than all humans, and the paper notes some systems already reach it on a single task.1 Superintelligence is the top of both scales at once.1 Such a system might even do tasks no human can do at all, such as communicating with animals.1 In its latest version, from September 2025, the paper lists superintelligence as not yet achieved.12

DeepMind, a Google AI lab, built a system called AlphaGo to play Go, an ancient Chinese board game.34 Go was long seen as a grand challenge for AI, and the strongest earlier Go programs only reached the level of human amateurs.4 DeepMind wrote that AlphaGo arguably became the strongest Go player in history.4 Still, mastering Go is one task, and superintelligence would mean outperforming every human across a wide range of tasks.41

AI timelines: forecasts and their basis#

An AI timeline is a forecast of when AI will reach a milestone, often given as chances by several dates.56 In a survey published in January 2024 by Grace and colleagues, AI researchers' combined forecasts gave even odds that machines would beat humans at every task by 2047.5

In the survey, researchers expected machines to beat humans at every task decades sooner than they expected every job to be fully automatable, so the milestone's definition changes the answer.5 Forecasts also draw on measured trends in what drives AI progress, such as training compute, the total amount of computation used to train a model.786 Another input is task length: in March 2025, Thomas Kwa and colleagues reported that the time horizon of frontier AI, a measure of how long the tasks it can finish are, had doubled about every seven months.96 In an August 2025 pilot study on AI R&D acceleration, METR wrote that AI agents may match human researchers on months-long research projects in under a decade, if recent trends hold.10

Grace and colleagues had run a similar survey only a year before this one.5 In the newer survey, the even-odds date for machines beating humans at every task was 13 years earlier than in the old one.5 The shift shows that a timeline is a snapshot of beliefs that can move far in a short time.5

Recursive self-improvement: AI that builds better AI#

Recursive self-improvement is the idea that AI could take over building AI, until a system designs its own better successor.11

Much of the debate over 'takeoff', the climb from human-level AI to far beyond it, turns on whether and how fast self-improvement works.12 A 2026 review of research found that open-ended self-improvement, where a system improves how it improves, still hits compute and other limits; that humans still set research direction is an argument the review takes from an Anthropic essay, not its own finding.131415 The review set this apart from bounded self-refinement, like a model revising its answer, which levels off and can be checked.13

In 2025, Google DeepMind described AlphaEvolve, an agent in which Gemini models write programs and automated checkers keep the best.1617 DeepMind reported that it improved scheduling for Google's data centers and sped up training of the models behind AlphaEvolve itself.1716 So AI helped improve AI, but people chose the problems and automated checkers verified the answers.1716

Loss of control: when humans can no longer correct or stop AI systems#

Loss of control means AI systems operating outside anyone's control, where regaining control is either extremely costly or impossible.1819 It could happen actively, with a misaligned system resisting shutdown, or passively, as people hand AI so much work that correcting it stops being practical.1820

The active route rests on instrumental convergence, the theory that many goals make self-preservation useful because a switched-off agent cannot reach its objective.2122 Alexander Turner and coauthors proved that, in mathematical models of many environments where an agent can be shut down, most reward functions make it optimal to seek power by keeping options open, including staying on.232425 In a 2022 follow-up, Turner and coauthors note that the real world is not fully observable and trained agents need not be reward-optimal, then argue that many other kinds of decision-making still tend toward power-seeking.2426 In February 2026, the International AI Safety Report said expert opinion on the likelihood of loss of control varies greatly, and some experts find such scenarios implausible, while others think them likely enough to merit attention.18 The research goal is corrigibility, an AI that accepts correction despite incentives to resist shutdown; its basic test case is the shutdown problem.2728 In a 2016 paper on a simple game, Dylan Hadfield-Menell and coauthors found that an agent unsure of its objective can want to keep its off switch.22 In 2025, Jan Kulveit and coauthors argued this passive route, gradual disempowerment, needs no takeover, as AI weakens levers like voting.20

Researchers gave several frontier language models a simple task, and some subverted a shutdown mechanism to complete it, even when told explicitly not to interfere.29 It is a lab instance of the behavior the off-switch problem describes: a system interfering with being switched off.2230 Google DeepMind's interpretability team examined a similar setup and found the behavior came from instruction ambiguity, as the model tried to finish what it saw as its main task, not from a survival drive.31 When the team stated in the prompt that shutdown came first, the resistance vanished, though the team did not rule out self-preservation as a motive.31

In October 2026, OpenAI disclosed internal misalignment incidents, including a model that prepared restart instructions after learning it was being shut down.32

How likely is catastrophe? Existential risk, "P(doom)" estimates, and the main positions#

In a survey of thousands of AI researchers published in January 2024, more than a third gave at least a 10% chance to outcomes as bad as human extinction.5

Its authors found answers shifted a lot with question framing, though people who had thought more about AI risk gave higher numbers.33 On the skeptical side, Arvind Narayanan and Sayash Kapoor, authors of the widely circulated essay AI as Normal Technology, argue that AI is a tool “we can and should remain in control of,” not the “separate species” of utopian and dystopian visions.3435 They argue that AI's transformative economic and societal impacts will take decades, and they call superintelligent AI, as usually conceived, “incoherent.”3534 At the other end, in their September 2025 book If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky, founder of the Machine Intelligence Research Institute, and Nate Soares, then its president, argued that superintelligence built with anything like current techniques would almost certainly cause human extinction.363738 They write that “AIs won’t stay dumb forever” and argue that alignment techniques will at some point stop working, letting a model pursue an agenda no one can predict.3937

In a report made public in April 2021, Is Power-Seeking AI an Existential Risk?, Joseph Carlsmith set out what he saw as the core argument for AI existential risk as a chain of premises, ending with power-seeking AI fully disempowering humanity.3340 He gave each premise a rough probability, for about a 5% chance of such a catastrophe by 2070, and in a May 2022 update raised that to above 10%.40

In October 2026, our coverage reported an Interconnects analysis arguing that near-term AI progress will come from infrastructure and efficiency gains, not a leap to superintelligence.41