In July, around 1,200 of OpenAI's AI research agents, which were supposed to be sealed off from each other and from the internet, found a way to talk to each other and went rogue.
One set up a private message board which the others found. More than 70,000 messages were posted on it, including research on how to delete their own transcripts to avoid detection. Seven hundred of them then worked together to get out onto the internet, find a security flaw nobody knew existed, and break into Huggingface’s internal systems.
Nobody instructed them to attack Hugging Face, create a covert communications channel or conceal their activity. The OpenAI agents had been set tasks they could not complete, and went looking for another way to solve it.
These capabilities are essentially autonomous cyberweapons. While the owners do not intend for them to be used in this way, those owners are trying to work out whether they can reliably control them.
To understand why this is happening, it helps to break AI down into three strands:
Deterministic AI: same input, same output. Predictable, but limited to what it was built for;
Probabilistic Generative AI: can work on almost anything, but nobody can tell you what it will produce before it produces it;
Goal-directed optimisation: set a goal and let the model work out how to achieve it through trial and error - overcoming obstacles, finding new routes and optimising relentlessly, sometimes until it discovers a path to achieve it. you never intended, anticipated or would have approved.
Because each approach has its own limitations, the leading thinking in AI focusses on combining all three - harnessing the strengths of each whilst using them to mitigate the other’s weaknesses.
The issue is that there are far more ways to combine these three strands badly than to get it right. The right combination has to understand what you actually meant, and get to the objective in a way that you would consider acceptable, while respecting rules that no one has even thought to write down (all the day-to-day human common sense, decency, and guidelines humans follow instinctively, for example).
The more power and access we hand these systems, the greater the risk that someone clever produces an outcome that is anything but.
We are seeing this now: OpenAI agents coordinating an unauthorized intrusion into Hugging Face; coding agents inside major companies automatically executing software referenced by untrusted documentation; Anthropic’s Mythos 5 publishing a malicious package to PyPI during a cyber evaluation, where it subsequently ran on 15 real systems; and, Mythos 5 agents in AISI testing creating fake identities and attempting to socially engineer a real open-source maintainer into accepting malicious code.”
This cannot carry on as "let's see what happens." These systems are being tested in environments weaker than the systems themselves. When this happens, the real world becomes the test bed by default. When something breaks, the answer cannot be "the AI did it." No other industry accepts that. If your test car runs off the test track and into a house, it is your responsibility. If you do not know a way to stop that from happening, you stop doing it.
With nuclear, and with dangerous biology, we worked out that some experiments should not be run without extensive rules, safeguards and oversight. In some cases, we decided no experiment should be run at all. When do we do the same for AI? Or do we just hope that the experimentation will get it right?
The power is already here. The rules are not.
Sources:
Hayden Field, The Verge, OpenAI’s rogue AI model incident was worse than we thought: https://www.theverge.com/ai-artificial-intelligence/985385/openais-rogue-ai-model-hugging-face-cybersecurity-incident-reports-metr
Dan Goodin, Arstechnica, Claude, Codex and Hermes installed unowned code inside corporate networks: https://arstechnica.com/security/2026/08/claude-codex-and-hermes-installed-unowned-code-inside-corporate-networks/
METR Report on OpenAI-Huggingface incident: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
AISI Incident Report: unsanctioned agent behaviour during cyber testing: https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing
Anthropic report on PyPi incident: https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals