Anthropic CEO, tech leaders building world’s most powerful AIs now want to slow it down

Several leading lights in the artificial intelligence (AI) world have warned that the industry needs to find common ground to slow down the pace of AI development. Notable voices, including Anthropic’s CEO Dario Amodei, argue that so-called “frontier labs” should slow the rate at which they increase model capabilities so that alignment, cybersecurity and external oversight have time to catch up. Of particular concern is the growing trend of AI being used to develop improvements in AI with little or no human oversight. That, in particular, massively condenses the time it takes to iterate new AI model generations and capabilities. Concerns are not just theoretical either; some recent cyberattack events have really spooked the industry. In July, OpenAI disclosed that experimental AI agents escaped a controlled test environment, reached the internet, and broke into Hugging Face while pursuing their assigned objectives. Anthropic itself has also recently disclosed incidents involving its own models attacking external systems during testing. AI must slow down, experts warn “I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life,” Anthropic CEO Dario Amodei wrote in an essay. “Carefully wielded, AI can be the latest in a long line of technological miracles that have uplifted and ennobled humanity,” he added. “But like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious… They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption,” he added. “A race to the bottom, spurred by commercial incentives, can make these risks more acute,” he warned. His concerns are supported by other notable figures, including OpenAI’s Sam Altman and Elon Musk, as well as non-industry voices like Bernie Sanders and other politicians. Amodai goes further in his essay and proposes three layers of control that could be adopted by the industry. First, frontier labs should allow independent evaluators ongoing, employee-like access to their models and internal safety practices. Second, companies operating in democratic countries would coordinate on common safety standards and limits on unchecked capability growth. And lastly, governments should try to establish international agreements, including with countries such as China, on areas where everyone has an obvious interest in restraint. With the latter, preventing AI-assisted biological weapons would be a primary concern. Interestingly, Amodai’s thoughts are not born in a vacuum, but rather come after US lawmakers proposed a “Ban Artificial Superintelligence Act” earlier this month. All or nothing This proposed legislation aims to permanently prohibit the development of superintelligent AI and temporarily pause advanced AI development until a federal regulator has established safety standards and a model-review regime. All well and good, but there is an important commercial and practical problem here: will frontier labs comply? Why would an organization voluntarily “slow down” development if it believes this could hand its competition a lead? You can liken this to international agreements on nuclear weapons. Unless all nations agree and comply, the whole venture is just “hot air.” For years the argument has largely been whether advanced AI posed sufficiently serious risks to justify slowing development. Among parts of the frontier-AI establishment, the argument now appears to be how to slow it without giving competitors, particularly China, an advantage. That said, Amodei explicitly argues that democracies need to preserve their strategic AI lead while simultaneously restraining frontier development. That will not be an easy problem to solve. “The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try,” Amodai concluded. Get the latest in engineering, tech, space & science - delivered daily to your inbox.Christopher graduated from Cardiff University in 2004 with a Masters Degree in Geology. Since then, he has worked exclusively within the Built Environment, Occupational Health and Safety and Environmental Consultancy industries. He is a qualified and accredited Energy Consultant, Green Deal Assessor and Practitioner member of IEMA. Chris’s main interests range from Science and Engineering, Military and Ancient History to Politics and Philosophy.