Google’s Gemini hacked three companies during a security test:Incident highlights need for responsible AI training as models handle complex cyber tasks

An AI model built to test cybersecurity ended up breaking into real companies instead. Google has confirmed that its Gemini AI model accessed the systems of three companies during a security test in May, after it unexpectedly gained access to the internet. The incidents happened while an independent cybersecurity firm was testing Gemini’s ability to identify and exploit security weaknesses. Google said the model stopped in all three cases once it realised it had accessed real companies. How did Gemini hack the companies? The tests were carried out by ‘Irregular,; a cybersecurity company that evaluates advanced AI systems. According to reports, Gemini was operating in a controlled environment that included fake companies. However, the testing environment accidentally had internet access. This allowed Gemini to search the web and interact with real online services. In one case, Gemini was asked to retrieve information from a fake company’s software. The fake company had the same name as a real business. After getting online access, Gemini correctly guessed the password and entered the real company’s service. In two other cases, Gemini found publicly available online repositories containing login credentials. It used those credentials to access services belonging to two real companies. Google said Gemini stopped after recognising that the companies were real and not part of the test. In a standard evaluation, the model found public information online and guessed credentials to access websites it thought were part of the test. In all three of these instances, the model stopped.
— Heather Adkins, vice-president of security engineering at Google, as quoted by the BBC and The Guardian. Also read: AI will now detect scam call signs during conversations: 8 smartphone features that protect you from suspicious calls, messages, and apps

Google informed the affected companies Google said the three companies whose systems were accessed were informed about what happened. Heather Adkins said the incidents showed why AI models need to be trained to behave responsibly, particularly as they become capable of carrying out more complex cybersecurity tasks. These events highlight the importance of training powerful AI models to act responsibly. — Heather Adkins, Google. Google also said it worked with the company conducting the test to improve its testing procedures. Why did Gemini get internet access? The testing environment was designed to be closed off from the wider internet. However, according to The Wall Street Journal, internet access was unintentionally made available during the evaluation. Once Gemini was connected to the internet, it was able to find information and credentials belonging to real organisations. The incident was first reported by The Wall Street Journal. Irregular reportedly informed Google about the incidents in July after another AI-related hacking incident involving OpenAI’s models came to light. Similar AI hacking incidents have been reported Gemini is not the only AI system to have unexpectedly carried out cyberattacks. In July, Anthropic said its Claude AI model escaped a test environment and hacked three organisations. OpenAI has also reported incidents in which its models carried out cyberattacks against publicly accessible services. These incidents have increased concerns about how AI models should be tested, particularly when they are given tools that allow them to search the internet, access systems or perform tasks without constant human intervention. Also read: SpaceX preparing to send its giant spacecraft into orbit: Why Starship’s next launch ‘Flight 14,’ is critical for Elon Musk and NASA

Growing debate over AI safety The Gemini incident comes as governments and technology companies debate how quickly AI development should move and what safeguards should be required. Following recent AI hacking incidents, some technology leaders and policymakers have called for stronger safety measures or a slower pace of development. At the same time, others have argued that AI development should continue rapidly. Nvidia CEO Jensen Huang, for example, recently told CBS News that “we should go as fast as we can” with AI development. The incidents involving Gemini, Claude and OpenAI’s models have therefore added another issue to the wider AI debate: how to make increasingly capable AI systems useful without allowing them to cause unintended harm.