Worried about the end of humanity thanks to AI? Here’s what’s really happening

Professor of Artificial Intelligence & Society at King's College London This week, a post on X unleashed panic when Jacob Coxon, announcing his resignation from tech company Anthropic, wrote: “The people building AI earnestly believe that it could kill us all by the end of the decade.” That might have been dismissed as fearmongering had it not been quoted by a research lead at the same company, Evan Hubinger, who replied “Jacob is correct here – we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” Alarm ensued. With some relief, Hubinger subsequently added: “To be clear, as we say in our latest Risk Report, I think the risk from present models is low. What I am worried about is superintelligence.” These posts come at a time when we have seen a spate of AI agents – software programmes that carry out tasks independently when directed by humans – break out of their test environments and hack into other websites. That sounds concerning, but it’s also attributable to gaps in security and carelessness on the part of the companies testing them. Should we be worried about these particular claims of a threat to human existence? Fortunately, it might be worth taking the doom-mongering with a pinch of salt. AI companies today are some of the most powerful entities on the planet. Investors have poured vast sums of money into AI development, and the market is dominated by a handful of tech firms, mostly in the US, who need to keep investment flowing. Surprisingly, some leading firms are still losing large amounts of money; it’s the hope of bigger and better AI that attracts the funding. If a company can promise tech so advanced and so powerful that it’s dangerous, that could be viewed as a marketing opportunity. And if the technology was said to be so dangerous that it could wipe us out? Well, that would require money to develop it more safely, or regulation demanding compliance with conditions that can only be met by companies with huge budgets, which could effectively destroy smaller competitors and give Big Tech the market advantage. We’ve had stories about AI and existential threat before. The threats go back decades, as anyone who reads or watches sci-fi will know. But even if we aren’t facing imminent potential extinction, there is still cause for concern. What makes it scary this time is the speed and scale at which development is occurring, and the lack of accountability in place. AI is largely unregulated territory, meaning tech companies are effectively marking their own homework. The EU introduced the AI Act, the world’s first attempt at providing overarching governance of AI at that level. The previous US administration was taking steps towards AI regulation, but this was rolled back by Trump, who views it as stifling innovation. In many countries there have been moves to monitor safety through government bodies without regulatory powers, such as the UK’s AI Security Institute. The UK has no specific AI regulation, though, just a patchwork of existing laws that cover some potentially problematic areas. China was working towards comprehensive national regulation but has since changed to targeted rules and technical standards. There have been calls by the AI research community for international guidance and consensus but so far little is in place. All of this means the hugely powerful cluster of tech companies get to set the agenda. Right now, both Anthropic and OpenAI, the two leading AI companies, are preparing to go public – getting ready to sell shares to the general public for the first time. They are fighting with each other to be on top, and that doesn’t just mean they need the best product. They need to show that they have the most potential, and the dominant AI model. They might also want to look like they take dangers seriously, portraying themselves as the most attuned to potential threats. When researchers post on a social media platform that their companies are serious about existential risk, it might be feeding the hype, but it’s also likely they do believe this, viewing themselves as the Oppenheimers of the modern age. They see the race to develop bigger and better models as something that must be done despite potentially lethal consequences. We don’t need the threat of superintelligence to scare us. That technology is not here yet, nor does it have to be. AI, as machine learning, brings us many things we have come to rely on: recommender systems seen in streaming TV and music, mapping software that gets us from A to B via the easiest and quickest route, and scientific discoveries providing us with incredible improvements to our lives. Generative AI, however – the models that are used to generate text and images, and AI agents that complete tasks on their own – is energy-hungry and prone to error. It demands water and electricity. It is unavoidably biased because it is trained on human-generated data. It is changing how we live our lives, and not always for the best. A recent survey we carried out at King’s College London showed that public sentiment about AI is becoming more negative, and that even those who use it regularly are concerned enough to limit their use. This is echoed in the growing backlash to AI, with widespread angry protests about data centres in the US – so much so that it’s becoming a mid-term election issue there. There’s a reckoning coming. If tech companies tell us that their product threatens our very existence, they shouldn’t be surprised when we decide we don’t want it in our lives. The widespread claims that superintelligence is arriving and that we must get on board or be left behind is a convenient narrative for the tech industry. But the future is not determined. If the tech bros are genuinely concerned there’s a greater than 10 per cent chance they’ll destroy us, maybe it’s time we asked them why they want to continue in the face of those odds.