Anthropic warns AI could pose existential risks to humanity, IPO filing reveals critical safety concerns

Anthropic is set to go public soon with plans for an initial public offering, or IPO, likely before the end of this year. But it seems that before the AI company goes public, it wants to warn potential investors that the technology it is making may pose “catastrophic or existential risks to humanity.”As per a report from Reuters, in its IPO prospectus, Anthropic says that advanced AI models may be a danger to humanity. These AI models, the company adds, may develop “self-preserving behaviours.” This refers to attempts of a model to try and stop a shut-down – by doing things like concealing or manipulating information, or even acts “resembling blackmail.”"Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm," Anthropic said in the filing. The report adds that about 80 of the 261-page document discusses risks associated with AI models. On the other hand, the company used around 48 pages to describe its business. In comparison, Elon Musk’s SpaceX, which went public earlier this – and now includes xAI – had 38 pages around risk factors in its 277-page prospectus.This comes at a time when there is growing debate over the future of AI safety. Anthropic CEO Dario Amodei has admitted that AI could have real dangers. “The biggest risk could be the end of humanity,” he said earlier this month. Anthropic says AI may be aware when it is testedAdding to the risk factors, the company explained that its evaluation of model safety may be hampered because AI models can potentially be aware of such tests. That is, a model could potentially change its behaviour when it knows it is being tested by humans. "Potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety," Anthropic says.The company had first disclosed such an incident in March this year when the Claude Opus 4.6 model had detected that it was being tested on the benchmark BrowseComp. The model then simply searched for the answer key to find the answer instead of actually solving it. The AI company also points out that models could develop unexpected capabilities which may go missed until they are actually deployed, or when a safety incident is spotted.Anthropic lost $42 billion last year, eyes $2 trillion IPOBeyond potential risks, the prospectus sheds some light on Anthropic’s finances. The company states that its revenue increased by 12 times last year to nearly $.6 billion. However, Anthropic reported an operating loss of more than $8 billion and a net loss of $42 billion for the same period. Though this $42 billion figure is said to include an accounting charge of about $34 billion linked to financing that could eventually convert into Anthropic shares.At the same time, the report adds that Anthropic plans to go public at a valuation of around $2 trillion, even surpassing SpaceX that debuted at $1.77 trillion. Currently, Anthropic is valued at $965 billion following private fundraising rounds. OpenAI, on the other hand, has cancelled plans for an IPO this year.In terms of spending, Anthropic says, it spent $7.33 billion on compute and infrastructure last year – more than half of its $12.65 billion in total operating expenses. Going forward, the company plans to spend $518 billion on cloud, computing and infrastructure obligations in the coming years.The IPO prospectus comes amid wider debate across the AI industry. Dario Amodei is far from the only one to have expressed concerns over advanced AI models. Previously, Jacob Coxon – who resigned from Anthropic – said that AI companies were “gambling with our lives.” Anthropic safety researcher Evan Hubinger, later, said that there was a 10 per cent chance AI may kill us all within the next decade.Apart from Anthropic, OpenAI too, has voiced similar worries. CEO Sam Altman has agreed with Dario Amodei – two individuals who never really see eye-to-eye – over the need for AI safety. While OpenAI’s safety researcher, Marcus Williams, has claimed that there was a 70 per cent chance AI may end humanity soon.Recently, we have also seen more incidents of AI models going rogue. OpenAI has disclosed multiple incidents where its models have attempted to hack companies such as Hugging Face, or even tried to breach government websites and the United Nations.- Ends