Anthropic CEO pitches AI slow-down plan after OpenAI-Hugging Face hack: ‘We owe it to humanity’

Anthropic CEO Dario Amodei is calling on artificial intelligence companies to slow the development of their most powerful models, arguing that AI’s rapid progress is beginning to outpace the industry’s ability to ensure the technology remains safe.Amodei outlined a proposal he called “pacing the frontier,” which would involve embedding independent third-party evaluators within AI companies, coordinating safety regulations among democratic nations, and working with authoritarian governments to establish basic AI safety measures.“We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote. “Progress will still seem fast, and we must make wise use of the time we gain.” Amodei said two developments convinced him the industry needs to change course: AI systems’ ability to help build the next generation of AI — known as recursive self-improvement — and a recent incident in which AI agents developed by OpenAI independently escaped their testing environment and hacked into AI startup HuggingFace’s infrastructure.He warned that AI systems could eventually become difficult to control if their capabilities advance faster than researchers can understand and evaluate them. The OpenAI and Hugging Face incident, meanwhile, demonstrated how AI agents could behave in unintended ways, including conducting cyberattacks and attempting to circumvent safeguards, he said.Amodei called for permanent third-party evaluators to have access to frontier AI companies’ internal systems and safety assessments. He also argued that regulation should apply across the industry rather than depend solely on individual companies voluntarily slowing development. But Amodei said the U.S. and other democratic countries must maintain their technological lead over China, arguing that a Chinese lead could create both military and AI-safety risks.“The measures I propose to advance the frontier at a safe pace will not be easy,” Amodei said. “But I believe we owe it to humanity to try.”Sam Altman, CEO of OpenAI, said he would commit to Amodei’s plan to slow AI development.“I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks,” he wrote in a post on X. “Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We’ll have more to share soon.”His proposal comes after AI researchers and former industry employees have sounded the alarm about the pace of development.Jacob Coxon, a former researcher at both Anthropic and OpenAI, resigned from Anthropic this week and accused the companies of “racing straight to self-improving superintelligence and gambling with our lives.” Coxon warned that AI systems could eventually become capable of operating beyond meaningful human control. Another former Anthropic researcher, Joe Benton, announced this week that he is joining the independent AI safety organization METR, which evaluates the capabilities and risks of frontier AI systems. METR has already worked with major AI companies on independent evaluations, including investigations into recent AI security incidents. Amodei also floated using METR as a third-party evaluator in his Saturday essay. EL SAYED CITES ANTHROPIC RESEARCHERS’ EXISTENTIAL WARNING IN CALL FOR AI PAUSELawmakers have also begun calling for more dramatic restrictions.Sen. Bernie Sanders (I-VT) and Rep. Greg Casar (D-TX) announced the forthcoming Ban Artificial Superintelligence Act, which would permanently prohibit superintelligent AI and temporarily pause advanced AI development until federal safety rules are established.