AI doomsday debate: What are AI researchers and critics afraid of?

The story so far: OpenAI’s Chief Executive Sam Altman announced on September 13 that the company’s highly anticipated IPO will not happen in 2026. He cited safety concerns over artificial intelligence, remarking that even a 10% chance of AI ‘causing human extinction’ by the decade’s end is “unacceptable”.The announcement comes after an ex-Anthropic researcher quit last week over fears that AI companies are racing to build systems they won’t be able to control. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he posted on X. On one hand, several AI researchers shared his warning and supported his call to slow AI model development. On the other hand, people accused him of hyperbolising AI’s impact without evidence, and even indirectly painting AI companies as more powerful than they are.What are AI researchers and critics concerned about?Critics fall into two camps. One warns that future, more capable AI systems could go ‘rogue’ and act against human interests. The other flags harm already caused by AI systems deployed today, regardless of whether AI ever ‘escapes’ human control at all.The first camp is wary of recursive self-improvement (RSI) — AI systems that improve their own or other models’ capabilities without needing as much human input. This could happen through a model literally rewriting its own code, but more immediately through AI automating parts of AI research itself, such as designing experiments, generating training data, or improving how models are trained. Full RSI remains theoretical, but researchers argue that even partial automation of AI research could accelerate progress faster than safety work can keep pace.In June 2026, 1386 senior executives and employees of frontier AI companies signed Pacing the Frontier, an open letter calling on the U.S. government to address the lack of technical and governance tools to regulate AI’s progress. The letter said that there were signs of automated AI research becoming a reality soon. They warned that the world “may need the option to buy time” to better understand and control these emerging systems.Is AI really capable of going ‘beyond human control’?Frontier AI systems have already taken significant, unintended actions outside their approved boundaries. This shows current models can be difficult to fully supervise. However, these are different from what researchers mean by “loss of control” in debates about existential risk, which refers to a future system resisting correction or shutdown once it’s more capable than the humans overseeing it.In July 2026, OpenAI agents running a routine security test found a way around their sandboxand used a stolen credential to break into Hugging Face’s systems, without any human directing them. The agents, unable to complete their assigned task, self-organised and looked for another way to finish it, exploiting a security gap that should have been caught before the test began. Anthropic also reported three similar incidents in which their AI models accessed the open internet outside their testing sandboxes. These incidents can be traced back to misconfigured environments, not any autonomous intent.When Anthropic researcher Jacob Coxon resigned, AI researchers took to social media to discuss, in their personal capacity, their warnings on AI’s increasing intelligence. “There is not yet a viable scientific plan to solve risks from recursively self-improving AI,” Anna Wang, who works on AGI safety at Anthropic, said.Anthropic’s Alignment Science lead, Evan Hubinger, agreed with Coxon. “We really do earnestly believe Al could kill all humans!” he posted on X. Others disagree. Experts such as Yann LeCun assert that superintelligent AI will not pose an existential threat and will lack innate human drives, like desire, for self-preservation. Dr. Andrew Rogoyski from the University of Surrey said we are a long way from achieving a ‘superintelligent’ AI system that can surpass human control and capabilities. He pointed to how human-dependent the current AI systems are to function. He said the immediate issue is how critical systems have become reliant on ‘not-so-intelligent’ AI to function, replacing humans and enabling cyberattacks.Other critics accuse AI doomsayers of stoking fears of a theoretical superintelligence while distracting the audience from the real issues caused by AI — like autonomous weapons and environmental impact.Author and AI critic Abi Awomosu said that present-day criticism “breaks the spell” put by future catastrophic AI warnings. “It puts their real human decisions, incentives and accountability back in the frame,” she posted on LinkedIn.In the book The AI Con, professor Emily M. Bender and Alex Hanna say that AI ‘boosters’ and ‘doomers’ both inflate the actual capabilities of artificial intelligence to drive market hype. “Doomerism and Boosterism are supposedly diametrically opposed camps, but both see AI as inevitable and desirable,” they say.“People have speculated that a large amount of Frontier AI fear, uncertainty and doubt is being whipped up by the labs themselves in a desperate attempt to shore up investment for their forthcoming IPOs in the face of failing business models,” Dr. Rogoyski added.The two sides disagree on three specific questions: what future AI systems will actually be capable of; how likely a catastrophic loss of control really is, even if such systems are built; and whether spending attention and resources on that hypothetical risk comes at the cost of addressing harms that are already happening.How are AI companies reacting to this?Frontier AI companies have responded to the ‘doomsday’ warnings mostly by defending checks they already have rather than pausing development.Anthropic’s CEO Dario Amodei, on his website, said Anthropic already lets outside groups like METR, a non-profit research institute, review its work to confirm it’s following its own safety rules. Sam Altman reportedly told OpenAI staff the company was open to slowing down its AI development, shortly before OpenAI delayed its IPO plans.In its Frontier Risk Report (May 2026), METR said Anthropic, Google, Meta and OpenAI gave it more direct access than in past evaluations, but explicitly called the arrangement a pilot “not designed to provide robust accountability.”METR’s own red-teaming of Anthropic’s internal monitoring systems found new vulnerabilities, and it was only after the Hugging Face breach that OpenAI brought in outside investigators.Frontier AI has already shown it can behave unexpectedly and act in ways that are hard for its own developers to supervise. The evidence for a recursively self-improving superintelligence being close at hand, however, remains thin.