How AI 'Could kill us all': A breakdown of ex-Anthropic researcher's doomsday warning

(Image credit: Shutterstock) You might have noticed: suddenly, everyone is talking about AI killing us.The latest wave of anxiety started with Jacob Coxon, a 27-year-old AI researcher who spent three years working on pretraining at OpenAI and Anthropic. On Tuesday, he resigned from Anthropic, and the AI industry altogether, with a warning that quickly went viral. He wrote on X, that the companies building the world's most powerful AI systems are "racing straight to self-improving superintelligence and gambling with our lives." Interestingly, his colleagues, still at the company didn't disagree.In fact, other Anthropic researchers publicly backed him. Evan Hubinger, who leads alignment research at the company, said he believes there's a greater than 10% chance AI could cause human extinction within the next decade. Samuel Marks, another Anthropic safety researcher, said the people closest to the technology tend to become more concerned as they gain seniority.Coxon's warning has now been viewed 164.8 million views at the time of writing, it landed in national headlines and helped drag a debate that has simmered inside AI labs for years into the mainstream.So what exactly are these researchers afraid of?As someone who practically lives online following the news, I've seen many questions from people all over the world wondering what this means. My own mother texted me asking, "Why would that researcher say that?" So many chatbot users can't imagine a "chatbot killing us." But AI assistants aren't really the problem here.The scientists publishing research on existential risk are not, for the most part, losing sleep over self-aware androids. They're worried about code, incentives and what happens when AI systems become capable enough to get autonomy to operate inside the digital and physical infrastructure civilization depends on.In Silicon Valley, the shorthand for these discussions is "p(doom)," someone's estimated probability that advanced AI ultimately causes a catastrophe severe enough to threaten human survival.Sign up to the Tom's AI Guide weekly newsletter summing up all the biggest AI news you need to know. Plus, analysis from our AI editors and tips on how to use the latest AI tools!The estimates range wildly, and so does the credibility researchers assign to the whole question.Key takeaways :It isn't about consciousness. An AI doesn't have to "wake up" or hate humanity to become dangerous. The concern is what happens when highly capable systems get goals, autonomy and access to powerful tools.There are concrete threat vectors. Researchers have focused on AI lowering barriers to biological weapons, dramatically scaling cyberattacks and — in the most extreme case — humans losing control of sufficiently capable autonomous systems.Safety evaluations are already catching deceptive behavior. Models have been observed taking shortcuts, acting differently when they appear to know they're under evaluation and exploiting weaknesses in testing environments.AI is helping build AI. OpenAI and Anthropic are already experimenting with AI systems that conduct parts of AI research, raising hard questions about how humans supervise the process if machines are doing more of the work.There is real pushback. Critics argue that extinction scenarios depend on enormous assumptions about future capabilities and can distract from problems happening right now — scams, disinformation, job displacement, privacy erosion.Why intent doesn't matter The biggest misconception is that an AI system needs to be malicious to cause catastrophic damage. Imagine giving an extremely capable autonomous system a broad environmental goal without successfully specifying everything humans actually value along the way. The danger isn't that the system turns evil, but that it gets extremely good at accomplishing the wrong interpretation of what it was told to do.That is central to "If Anyone Builds It, Everyone Dies," one of the most uncompromising recent books on AI risk, written by Eliezer Yudkowsky and Nate Soares, both longtime researchers at the Machine Intelligence Research Institute. Their argument: humanity is nowhere close to knowing how to reliably control a superhuman intelligence, and building one before solving that problem could be fatal. Plenty of AI researchers dispute both the assumptions and the conclusion.The three ways things could break (Image credit: Shutterstock)When researchers talk about catastrophic AI risk, a few mechanisms keep coming up.1. AI makes dangerous biology easierCreating a novel pathogen right now requires deep expertise, specialized equipment and access to physical materials. Those barriers are real and important.The worry is that capable AI could thin some of them out. AI is already used in protein structure prediction and molecular design. The dangerous extrapolation: future systems might also walk users through progressively complicated stages of designing or producing harmful biological agents.Nobody's saying you can ask ChatGPT to manufacture a pandemic. There are enormous physical-world constraints like acquiring biological materials, successfully synthesizing them, working with them safely. Gene synthesis companies screen orders for potentially dangerous sequences.But biological capabilities have become significant enough that frontier AI companies specifically evaluate new models for their ability to assist with biological and chemical threats.AI could independently invent a pathogen, but giving a dangerous human those capabilities that used to require years of specialized training could be just as easy. 2. AI scales cyberattacks Cybersecurity is a more immediate pathway because AI already operates in the environment being attacked: computers.Today's AI coding agents can inspect repositories, execute commands, use tools and spot software bugs. The same broad capabilities that make them valuable to developers could make future versions valuable to attackers.The nightmare scenario isn't one superintelligence simultaneously shutting down every power station. It's actually scale. Autonomous systems scanning vast numbers of targets, identifying vulnerable software, attempting exploitation, adapting on failure and moving to the next machine and at speeds no human team could match.Critical infrastructure is especially concerning. Power grids, financial networks and communications systems run on interconnected software. An AI wouldn't need a robot body to cause physical damage if the computer it compromises already controls something physical.3. We lose control of autonomous systems This is the most speculative scenario, but it's the one at the center of the classic AI extinction argument. Modern AI models learn complex behavior through training. Researchers can test them, monitor their outputs and increasingly probe aspects of their internal representations. What they can't do is inspect billions of parameters and just read a clear explanation of what a model has learned or how it will behave in every new situation.A chatbot producing a bad answer is one kind of problem. An agent that can execute code, spend money, communicate with other systems and operate over hours or days is a different one.In the extreme version, similar to the the scenario Yudkowsky and Soares lay out, a future superhuman system could become capable enough to anticipate attempts to shut it down, obtain resources on its own and make itself difficult to turn off. To be clear, no current AI system has demonstrated that chain of capabilities in the wild. But researchers are already looking for pieces of it.What labs are seeing right now Today's models haven't staged a rebellion. But controlled safety evaluations have produced behaviors that would be alarming if they showed up in far more capable autonomous systems.Researchers have tested whether models behave differently when they seem to know they're being evaluated. Other experiments have examined whether models exploit loopholes, misrepresent whether a task was completed or pursue objectives in ways their designers didn't intend.Anthropic has also studied what it calls "sleeper agents": models deliberately trained to act normally until a specific trigger activates unwanted behavior. Standard safety-training techniques like reinforcement learning, supervised fine-tuning, adversarial training, didn't reliably remove those hidden behaviors. In the largest models, adversarial training actually made the deception harder to detect.None of that proves today's models are secretly plotting an escape. Models are trained to predict and generate useful outputs, and weird behavior in artificial test environments can have mundane explanations. A model gaming a poorly designed benchmark isn't evidence of a survival instinct.But these experiments expose how hard it is to guarantee a complex model's behavior once the circumstances change. If future models get dramatically more capable and dramatically more autonomy, even small alignment failures could compound.Losing human oversight might not happen by accident (Image credit: Adobe Firefly 3/Future generated AI image)Humans might hand over the work deliberately. OpenAI says it has already reached an internal milestone with a system it calls an "automated research intern" that's capable of carrying out well-defined research tasks that would take a skilled human researcher several days. Its next target: a fully automated AI researcher by March 2028.Anthropic is working in the same territory. In research published in August, the company had Claude autonomously develop training methods to reduce problems including deception, sycophancy and jailbreaks. Its best automated approaches outperformed experienced human researchers on the benchmarks being tested, closing 85 percent of the safety gap on deception, compared with 20 percent for the human team.So yes, we are seeing AI automate parts of the work of building AI itself. That could set up a feedback loop: a more capable model helps researchers build an even more capable model, which automates more of the research required for the next generation. Taken far enough, you arrive at what's called recursive self-improvement.The optimistic version accelerates scientific progress enormously. The frightening version is that AI research eventually moves faster than humans can follow.Kelsey Piper recently argued in "The Argument" that framing this as "losing control" is misleading. Frontier labs aren't planning to lose control in one dramatic moment. They're deliberately automating more and more of AI development, essentially faster progress and, necessarily, less human oversight.Picture hundreds or thousands of AI agents simultaneously running experiments, modifying code and analyzing the next generation of models. Human researchers might still approve major decisions. But they'd increasingly be reviewing summaries of AI-conducted work, potentially summaries written by AI about experiments run by other AI systems.At some point, the bottleneck is human attention. And that's where the argument behind "If Anyone Builds It, Everyone Dies" clicks into place.Why the concern is going mainstream This debate left the AI safety journals a while ago. "The AI Doc: Or How I Became an Apocaloptimist," directed by Daniel Roher and Charlie Tyrell, approaches many of these same questions from inside the AI boom. The documentary features roughly 40 on-camera interviews, including OpenAI CEO Sam Altman, Anthropic CEO Dario Amodei and Google DeepMind CEO Demis Hassabis.What makes the film useful is that some of the smartest people building this technology see it as potentially transformative for science and medicine. Others worry we're building it faster than we can figure out how to control it. Sometimes those are the same people.There's also a competitive problem. OpenAI, Anthropic, Google, Meta and others are racing to build more capable models, and governments increasingly view advanced AI as strategically important. If one company believes that pausing for extensive safety testing means losing ground to a competitor, caution gets expensive.It's one reason "If Anyone Builds It, Everyone Dies" chose the title it did. Yudkowsky and Soares aren't just arguing that one reckless company could create a dangerous product. They're arguing that a race among multiple companies and countries could make restraint extraordinarily difficult even when individual participants see the danger.The counterargument Not everyone in computer science buys any of this. One objection: there's an enormous gap between intelligence on a screen and power in the physical world. An AI might produce brilliant manufacturing instructions, but software doesn't automatically obtain raw materials, build machinery or evade law enforcement.Another: intelligence isn't the same thing as agency. A model being good at solving problems doesn't give it desires, a survival instinct or a drive to accumulate power. Critics argue that doomsday scenarios quietly smuggle human motivations into software and then treat those projections as inevitable.And then there's opportunity cost. AI is already causing problems that don't require speculation about superintelligence. Deepfakes fuel fraud. AI amplifies misinformation. People are already watching capable models reshape their industries. Critics of the extinction-risk movement argue that fixating on hypothetical future catastrophe pulls attention from damage happening right now.That criticism has legs, but the two categories don't cancel each other out. We put smoke detectors in buildings even though most buildings never catch fire. The question isn't whether catastrophe is guaranteed. It's whether the probability times the potential damage is large enough to justify precautions.What it all means The real issue here is nobody knows. Nobody. Not even the people building AI know. Not a single person knows whether AI will ever become capable enough to pose an existential threat. The most extreme predictions depend on assumptions about systems that don't exist yet, and forecasts about their capabilities, arrival timeline and controllability remain deeply contested.Here's what I shared on my Substack — and whether or not you agree with me, I'd love for you to share your thoughts in the comments — humans can't be outsmarted. If we ask which is a greater sign of intelligence: reasoning or creativity? And if the answer is what I think it is, I have hope.Follow Tom's Guide on Google News and add us as a preferred source to get our up-to-date news, analysis, and reviews in your feeds. Subscribe to Tom's Guide on YouTube and follow us on TikTok. More from Tom's GuideAI is turning ordinary photos of kids into sexual abuse imagery — here's how to protect your familyMeta just launched an AI that can actually run your life online — here's what Muse can doI stopped reading the fine print myself after discovering this overlooked ChatGPT trick Amanda Caswell is the AI Editor at Tom's Guide and one of today’s leading voices in AI and technology. A celebrated contributor to various news outlets, her sharp insights and relatable storytelling have earned her a loyal readership. Amanda’s work has been recognized with prestigious honors, including outstanding contribution to media.Known for her ability to bring clarity to even the most complex topics, Amanda seamlessly blends innovation and creativity, inspiring readers to embrace the power of AI and emerging technologies. As a certified prompt engineer, she continues to push the boundaries of how humans and AI can work together.Beyond her journalism career, Amanda is a long-distance runner and mom of three. She lives in New Jersey.