Ex-Anthropic researcher Jacob Coxon warns AI could get out of control and end humanity

Humans may lose control of AI. This used to sound like science fiction, but according to former Anthropic researcher Jacob Coxon, it may soon be the reality. He believes that if we don’t take steps to regulate AI development now, humanity may even face the risk of going extinct. Coxon insisted that companies that were building AI today were being “reckless,” with a “break things, fix later” approach that was not safe for a technology as powerful as AI.Speaking at a New York City Council hearing, Jacob Coxon outlined his concerns around the way companies were building AI. “We do not know how to control any AI system yet,” he said. Coxon insisted that while an AI “kill switch” may solve problems in the short-term, it was likely not “enough to ensure safety in the long run.”Given the current state of the industry, the former Anthropic researcher predicted that if we lose control of AI, it may be the end of humanity. “On the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction,” Jacob Coxon said. “From my experience, the companies are being extremely reckless, given the stakes.”ex-Anthropic Jacob Coxon says that on the current path humanity more likely than not loses control of AI"So all the AI companies are trying to build super intelligent AI. This means AIs that are better than humans at every task.""If this goes well, there could be tremendous https://t.co/VQgPiSWKGA pic.twitter.com/zVgITpsCWJ— Fireside Alpha (@firesidealpha) October 5, 2026At the hearing, lawmakers discussed whether to impose safeguards on AI development as people became more vocal about safety concerns. Coxon is one of the biggest critics of the current AI development trajectory. After resigning from Anthropic last month, he wrote on X, “The people building AI earnestly believe that it could kill us all by the end of the decade.” AI companies cannot run like startupsYou may wonder why are AI companies being reckless? After all these companies are making some of the most advanced tech we have ever seen. The answer, Jacob Coxon says, lies with the way these startups work. “The companies run on a startup mindset. Move fast, break things, fix them later,” he explained.Jacob Coxon pointed out that such an approach was fine if a company was making a new photo app, but this was not how AI should be trained. “That works for a photo sharing app. It does not work for building the most powerful technology ever built,” he said. The former researcher also addressed the Hugging Face incident where hundreds of rogue OpenAI agents tried to hack the US company. “The AIs had developed goals no one intended, and carried out what would have been a federal felony if a human had done it,” he explained. According to Coxon, this incident, along with many others, showed that “we don’t know how to prevent AI from developing goals we don’t want.” If the industry does not implement proper safeguards in development, Jacob Coxon believes that such cases may only get worse. “As long as the attitude is to wait for things to break, one day something like this will probably happen again, except the AIs will be much more capable and the outcome much worse,” he added.Though Jacob Coxon accepted that if done right, AI may bring some major benefits to humanity. “If this goes well, there could be tremendous upside. This could transform the economy, it could make huge improvements in science and medicine, and it could make nearly everyone’s life better,” he explained.Apart from Coxon, representatives from OpenAI, Anthropic, Google and Meta appeared at the hearing, with some testifying remotely. Proposals before the council include protections for whistleblowers, including a share of fines or penalties recovered from AI companies following reported violations that would allow people harmed by AI systems to sue developers in some circumstances. The council also discussed third-party validation for some systems before they can be deployed in the city.There is a growing debate in the AI industry about such fears. While OpenAI and Anthropic have pushed for regulation around frontier AI development, Meta CEO Mark Zuckerberg has opposed such a system. Though US AI companies, including Anthropic, Google, Meta, Nvidia, and OpenAI, have signed President Donald Trump’s AI accords – a voluntary non-binding agreement that focusses on safe AI development.- Ends