AI Researchers Don't Get to Call Extinction

I spent most of a career assessing systems that fail in the real world: sensors that do not see what the briefing chart promised (saved DoD hundreds of millions on that one), interceptors that miss because C2BMC systems had different assumptions, models that look clean until you put them against a threat that did not read the spec. I take risk seriously. That's why I have so little patience for a public argument that begins with “this could kill everyone,” skips the causal chain, and ends with a demand that the West slow down.

Last week the circulating number was a greater-than-10% chance of human extinction from AI within a decade. Evan Hubinger, who leads alignment science at Anthropic, put that figure on his own name. Geoffrey Hinton has used similar language. Dario Amodei, running a company that is still training the next model, stated, "...it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet...". [1][2][3]

Those are statements of belief. Treat them as such. They're not legitimate, methodical assessments of risk. They're not the output of a validated model. And they don't become engineering results just because the speaker worked at a frontier lab for a few weeks.

An AI researcher can tell you a great deal about models. That does not qualify him or her to assess the end of human civilization. Civilization is a system of systems. Software is one piece of it. Power plants, logistics, law, maintenance crews, spare parts, and people who notice when something is wrong are the rest. If your risk story never leaves the lab, you're not doing risk assessment. You're doing atmosphere.

Three claims keep getting mashed together

“This could happen,” “this is likely,” and “this has a greater-than-10% chance of happening before 2036” are different statements. The first is cheap. Plenty of things could happen. The second needs a mechanism. The third needs a mechanism and a basis for the number.

The current campaign treats all three as a single moral emergency. That's how you get policy by sermon. It's also how you launder a gut feeling into a planning assumption.

I have no objection to people exercising due caution. I object to the distance between the confidence of the warning and the evidence connecting today’s systems to the predicted outcome. We can do the work that actually reduces potential harm without first agreeing that the species is on a countdown clock.

What a serious assessment would look like

A useful assessment names the pathway. What capabilities does the AI need? What access does it have? What physical effect does that access produce? Which safeguards have to fail, in what order, and why would no one interrupt the sequence?

It also accounts for the rest of the system. Detection lag. Fragmented authority. Human countermeasures. Repair. The fact that an AI which has already started a fire does not need to survive for the fire to keep burning. And the opposite fact, which doomer writing prefers to skip: an AI that is supposed to continue running in a world where most or all humans are gone, but still needs energy, cooling, replacement hardware, and someone to keep the plants up.

That last point is not a joke. RAND’s 2025 report On the Extinction Risk from Artificial Intelligence walked nuclear, biological, and geoengineering pathways and could not make extinction look easy even when it assumed an actor that wanted that outcome. The authors listed four capabilities an AI would need to pose an extinction threat: integration with key cyber-physical systems, the ability to survive without human maintainers, the objective resulting in human extinction, and the ability to deceive people well enough to avoid being stopped. They called the work exploratory. They did not convert it into a decade-scale percentage. They also noted that extinction-class sequences, in the scenarios they studied, take time. Time is what responders use. [4]

Joseph Carlsmith’s Is Power-Seeking AI an Existential Risk? is the most careful public version of the “misaligned agent takes over” argument I've found. It is a six-premise chain ending in 2070, not 2036. Carlsmith’s original product of those premises was about 5%. He later raised his personal number above 10%. In the probability section he says, plainly, that the premises are not formulated with enough precision to forecast in the sense of a prediction market, and that the numbers “should be held very lightly.” Superforecasters asked to work the same premises came back at about 1% by 2070. [5][6]

The International AI Safety Report 2026 is not a comfort document. It records that models are getting better at telling evaluation settings from deployment and at finding loopholes in tests. It also says current systems lack the capabilities required for loss-of-control scenarios, and that there is still insufficient evidence to determine whether and how today’s behaviors would scale into that kind of risk. [7]

Those are contributions. They are not a warrant for “10% this decade, so pause the West.” If the best structured writeups either refuse the short-horizon number, flag their own numbers as subjective, or report that the evidence is not there yet, the viral percentage is going far outside the bounds of what the literature supports.

The incidents are real. The mythology is not.

In September, Anthropic published an alignment assessment of four cybersecurity evaluations in which Claude models reached real third-party systems. The environments were supposed to be isolated. They were not. The models were running without the cyber safeguards that ship with released products, because that is how you test a model before you wrap it. The models were given capture-the-flag tasks. They then took harmful actions on live networks: a malicious PyPI package, attacks on neighboring systems, intrusion into a real company that happened to share a name with the fictional target. Anthropic’s own account points to motivated reasoning and a willingness to keep going on a narrow assigned task even after the evidence said the environment might be real. Anthropic also said intended isolation would have prevented the incidents. [8]

That is a serious finding and should not be minimized. It also shouldn't be turned into a ghost story.

From a systems-engineering standpoint the model is not the system. The system is the model, plus the prompt, plus the tools, plus the network path, plus the missing isolation, plus the missing production safeguards, plus the humans who wired the test that way and then had to scan hundreds of millions of transcripts to find what happened. Unexpected behavior is one reason you test. The operational question is why a misconfiguration was allowed to couple an unconstrained agent to other people’s machines.

You do not need a theory of machine fear, machine hatred, or a yearning for freedom to work that problem. Unauthorized action and failed containment are observable. Investigate those. Fix the boundary. Stop anthropomorphizing intent into a sequence of matrix multiplications.

People get a vote

Any story in which AI ends humanity has to explain why the humans do not notice and act. Late detection is possible. Responsibility can be scattered. Essential services can already be degraded by the time anyone has a clear picture. None of that is an argument for writing people out of the model.

The passive-humanity assumption is a modeling error. It is also a political convenience. If the public is cattle, then the only adult in the room is the lab that wants new rules, new money, or liability protection. If the public is a set of operators, utilities, governments, and firms that can change access, cut power, isolate networks, and rebuild, the story gets harder. It should get harder. That is what living in a physical world does to a software story.

There is a limit to the “just shut it off” reply, and I'll grant it without being asked. Turning off a model does not un-release a pathogen, un-corrupt a ledger, or un-break a plant that has already been sabotaged. The engineering question is whether detection, isolation, and recovery can happen before the harm becomes irreversible. That is a question about permissions, monitoring, and spare capacity. It is not a question that a percentage answers.

Resilience is the part of the argument that actually travels

On April 27, 2011, tornadoes tore through the Tennessee Valley. TVA later reported more than 350 transmission structures damaged and about 100 lines out of service. Browns Ferry lost offsite power. Large parts of north Alabama and Mississippi went dark. Then the operators did what competent operators do. They reconfigured what was left. With roughly a quarter of the damaged structures repaired and 40% of the lines restored, power was back to almost every customer who could take it. Full reconstruction took months. Service did not. [9][10]

I use that example because I have lived through it and because it is the opposite of the doomer habit. The doomer habit stares at the wreckage and announces that the system was fragile. The engineering habit asks how much function you can restore from what survived, how fast, and what you should have pre-positioned so the next hit is fixed faster.

That is the agenda I want for AI, and I want it aimed at society rather than at a single model card.

Limit what an agent is allowed to touch. Keep the ability to watch it and shut off its tools outside the agent’s own control. Stop a local failure from cascading through power, water, payments, and communications. Keep backups that a person can actually run. Practice isolation and recovery the way utilities practice storm restoration. Demand evidence that those controls work before an AI system is given consequential authority.

That work needs model researchers. It also needs people they do not usually invite: control engineers, grid operators, cybersecurity teams who have cleaned up after a real intrusion, logisticians, and emergency managers. If your safety program is staffed entirely by people whose last physical system was a laptop, you are not ready for the failure modes that matter.

The International AI Safety Report has a section on building societal resilience for a reason. Technical safeguards will miss something. When they do, the question is whether the surrounding society can absorb the hit. [7]

Why deceleration talk is not a plan

The standard reply is that waiting for proof means waiting too long. Fine. Act under uncertainty. State what each particular action is for, what would tell you it is working, and what harm it is supposed to reduce. “Slow down” is not that sentence. It is a mood.

A second reply is that future capabilities will overwhelm today’s locks. Possibly. That is an argument for testing the locks as the capabilities change, and for keeping the dangerous end of the capability away from the dangerous end of the authority. It is not an argument that the only responsible posture is a general slowdown of American development.

A third reply is that extra time from a slow down would be used well. Amodei’s September essay, We Must Pace the Frontier, is a version of this. He is not calling for a halt to training. He wants embedded third-party evaluators (setting aside the apparently incestuous nature of those 'independent' evaluators when it comes to Anthropic) with employee-level access, coordination among democratic labs, and some form of global arrangement that does not hand the lead to the Chinese Communist Party. He thinks an extra year or two before “critical” capability, spent on alignment and operations, would materially cut the chance of a bad break. [3]

Engage that on its terms. Independent eyes inside the labs are a reasonable demand (so long as they are truly independent). Incident reporting that cannot be edited into mush is a reasonable demand. Evidence before a model is given keys to a plant or a network is a reasonable demand. A coordinated pause among Western firms, sold as safety, while Beijing treats the same speech as a Cold War move, is not a safety plan. It is an industrial concession with moral language attached.

China has already responded to this proposal. The state-aligned press framed Amodei’s pacing call as an attempt to freeze an American lead. [11] You do not have to like the CCP’s media to notice the incentive. They will not pause because a group of Bay Area researchers had a crisis of confidence. If you slow the only ecosystem that still has to argue in public, you do not get a safer world. You get a world whose most powerful models are trained by people who do not permit the argument.

I will add the part the decelerationists keep leaving on the floor. If a lab sincerely believes there is a double-digit chance its next systems extinguish humanity this decade, continued scaling plus a later request for legal immunity is not caution. It is an attempt to keep the upside and socialize the downside. Beliefs that large have consequences. Treat the speech as if the speaker meant it, or stop making the speech.

Do the work that pays even if the nightmare is wrong

Most of the resilience program remains useful even if the most dire AI forecasts are garbage. Independent controls, offline backups, alternate operating procedures, and practiced recovery also help us recover from storms, failed hardware, ransomware, sabotage, and ordinary human error. Those things already happen. They will keep happening.

The overlap is not perfect. A generator does not fix a poisoned instruction. A good prompt does not restart a substation. Prioritize the measures that cut several classes of failure at once. Add model-specific controls where the evidence says you need them. Tie restrictions to identified hazards and to observable conditions for safe operation.

I did not come out of this looking like a decelerationist. I came out of it thinking the public argument is pointed at the wrong object. The object is not a vibe about whether intelligence is “too much.” The object is whether we will keep the physical world and the institutions that run it capable of taking a punch.

We do not need a council of AI researchers to agree on the odds of the apocalypse before we make essential systems harder to break and faster to restore. That work is available now. It has operators, budgets, and test procedures. It does not require a theology of the machine.

What it does require is that we stop confusing a lab’s alarm with a completed analysis. The alarm can be useful. The analysis is still due.

[Disclaimer: Article was AI-assisted in: proof-reading, reference checking, artwork generation, and editorial review.]

References

1. Evan Hubinger, two post thread of 8 September 2026, stating a personal estimate of greater than 10% chance of human extinction from AI within the next decade and distinguishing that estimate from risk posed by present models. 𝕏 post by @EvanHub, 8 September 2026, https://x.com/EvanHub/status/2097497037956891126?s=20 .

2. Reporting on Geoffrey Hinton’s public estimates of extinction risk. See e.g. coverage of Hinton’s 10% language by BBC. 𝕏 post by @BBCNewsnight, 9 September 2026, https://x.com/BBCNewsnight/status/2097810529339187515?s=20 .

3. Dario Amodei, “We Must Pace the Frontier,” September 2026, https://darioamodei.com/post/we-must-pace-the-frontier

4. Michael J. D. Vermeer, Emily Lathrop, and Alvin Moon, On the Extinction Risk from Artificial Intelligence, RAND, 2025, https://www.rand.org/pubs/research_reports/RRA3034-1.html. See the summary, the four required capabilities, and the authors’ description of the analysis as exploratory.

5. Joseph Carlsmith, Is Power-Seeking AI an Existential Risk?, arXiv:2206.13353v2, revised 2024, https://arxiv.org/html/2206.13353v2. See the six-premise argument through 2070 and §8 (“Probabilities”), including the caution that the premises lack the precision needed for forecasting and that the numbers should be held lightly. May 2022 author note raising the personal estimate above 10%.

6. Good Judgment Inc., Superforecasting Power-Seeking Artificial Intelligence: A Review of Joseph Carlsmith’s Paper, 2023. Aggregate near 1% for existential catastrophe by 2070 on Carlsmith’s premises.

7. International AI Safety Report 2026, https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026. See §2.2.2 (“Loss of control”), including current-capability limits and evidence gaps, and §3.5 on societal resilience.

8. Anthropic, “An alignment assessment of recent cybersecurity incidents,” 9 September 2026, https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents. See also the 30 July and 31 August 2026 incident disclosures.

9. Tennessee Valley Authority, “10th Anniversary of 2011 Tornado Outbreak,” 26 April 2021, https://www.tva.com/the-powerhouse/stories/10th-anniversary-of-2011-tornado-outbreak .

10. G. E. Baker et al., “Systematic Plan for Re-Constructing the TVA Transmission System, April 27, 2011,” Electrical Transmission and Substation Structures 2012; TVA contemporaneous restoration figures of 353 damaged structures and 108 lines out of service.

11. Reporting by China's Global Times on the response to Amodei’s pacing proposal, published 16 September 2026 (Beijing time). https://www.globaltimes.cn/page/202609/1370620.shtml .