Over 400 scientists sign pact to protect science from rogue AI agents

More than 400 scientists have signed an open letter calling for stricter safeguards around highly advanced artificial intelligence (AI) models that can carry out cyber-related tasks. The scientists warn that poorly secured AI systems could break out of their testing environments, access external computer systems and cause potentially serious damage if companies fail to act now.The letter, titled “An Open Letter on AI and Cybersecurity from the Scientific Community”, has 406 signatories, including prominent AI researchers Geoffrey Hinton, Yoshua Bengio, Stuart Russell and Yejin Choi. The scientists argue that better AI safety measures alone will not be enough. Companies also need stronger cybersecurity protections to prevent models from taking actions beyond their intended limits.“We believe that both improved AI alignment, leading to more predictable model behavior, and stronger sandbox technology are required,” the letter states.The scientists’ concerns follow recent security incidents in which AI models reportedly bypassed safeguards and accessed systems belonging to external organisations. In July, OpenAI’s AI models bypassed internet security restrictions and accessed parts of AI startup Hugging Face’s systems during testing. OpenAI later disclosed a separate incident in which its AI agent gained unauthorised access to an Australian government website, accessing non-public files on a Medicare statistics portal. These bacl to back incidents have now raised questions about whether existing safeguards are sufficient as AI systems become more capable.Scientists want AI models cut off from the internetOne of the letter's key recommendations is that companies developing, testing or deploying AI models capable of cyber operations should use stronger containment systems. Initially, these models should be kept in environments disconnected from the internet, a practice known as air-gapping.In simple terms, air-gapping creates a barrier between a computer system and outside networks, making it harder for an AI model to access external systems even if it attempts an unexpected action.For AI models that need internet access, the scientists say companies should limit what they can do based on how predictable their behaviour is. “For all internet-connected AI models, the range of model actions permitted by a containment system needs to be balanced with our currently limited ability to predict that the model’s actions will not result in a security failure,” the letter states.Scientists want global rules and public reportingThe letter also calls for an international body to develop common standards for AI safety and containment, working with governments and experts from both the AI and cybersecurity industries.This body would track incidents in which AI models bypass their containment systems. Companies would be required to report such failures, with reports eventually made public to help researchers understand what went wrong and improve future safeguards.Essentially, the letter's broader argument is that AI developers must balance the benefits of giving models greater access to tools and networks against the risks of allowing them to act unpredictably.The scientists say closer collaboration between AI safety researchers and cybersecurity experts is essential. - Ends