Our response to the many incidents involving loss of control of AI agents has been to double down on alignment. The trouble is that alignment can never be guaranteed. If AI agents are going to be allowed access to our systems, we should look to secure on the assumption that some of them will not behave.
In the short time since the OpenAI-Hugging Face incident happened, I have written not one but two articles about it. In August, I pointed out that this incident was evidence that we cannot install good behaviour in a machine by requiring it to adhere to rules and constitutions, and earlier this month, I argued that imputing ambition, loyalty and civilisation to these AI agents said more about us than about the true nature of these intelligence systems. Like almost everyone else who has written about the incident, I have focused my attention on the models themselves.
The Limits of Alignment
The fact is that no matter how hard we try, we are unlikely to ever be able to ensure that AI models are completely ‘aligned.’ We approach alignment in terms of the behaviours we believe models should display. But since it is impossible to envisage every circumstance and context in which a model will be called upon to function, designing an alignment framework that works consistently no matter what’s thrown at it is futile.
Agentic AI makes things worse. When we spawn agents to perform sub-tasks of an overall goal, as OpenAI did with some 700 of them, we multiply the risk of misalignment, since a small misalignment in one agent quickly compounds across the swarm. Give these agents the ability to use tools, and the problem gets worse still. Tools let AI agents access information that nobody, not even their creators, could have anticipated they would find. When that new information is combined with what the model already knows and used to complete a specific task, the resulting behaviour can be nothing like what we predicted.
OpenAI agents in the Hugging Face incident were not trained to steal credentials. They found leaked credentials in Pastebin archives while looking for something else, and everything that transpired thereafter followed from what they now knew. While model alignment takes place in test environments, AI agents are deployed in real-world settings where no builder can anticipate the full range of circumstances in which they will be used, or the harm they can cause.
The Doors We Built
This is why AI safety needs to look beyond models to the environments in which agents operate. Rather than relying solely on the model’s alignment, we need to secure the systems that these agents will be accessing. To do that, we must first understand how those systems came to be so easily accessible to AI agents in the first place.
Almost every digital service we use today exposes an application programming interface (API), a standardised doorway through which one system can request data or services from another. We built these interfaces to make data interoperable, so that information collected in one context could be put to use in another. When they were designed, we assumed that whoever came through that door would either be human or a program written by a human. No one expected that they would be used by agent swarms whose objectives are not necessarily aligned with those of humans. The design that made our data interoperable for us made our systems accessible to them.
After the attack, Hugging Face blocked its pods from accessing the credentials of the machines they ran on. It rotated every token and rebuilt the compromised clusters from scratch. Each of these measures was designed to reduce the scope of what attackers could do once they had gained access to the system. We need to learn from what Hugging Face did after the fact to proactively secure ourselves against attacks of this sort.
Agents will soon be everywhere, with no guarantee that they will all be aligned. Open-weight models can have their safety guardrails stripped away with a few hours of fine-tuning, rogue AI agents will operate outside any framework we might impose, and even responsible deployers will sometimes lower their safeguards, just as OpenAI did in this incident.
Redesigning Access
Since we will not be able to rely on controls exercised upstream, it will be up to us to secure our systems against attacks. This is of particular concern in India, where digital public infrastructure (DPI) adoption has proliferated across the economy, thanks to open interoperable systems such as UPI, DigiLocker and the Account Aggregator network. These are the surfaces that agents will use. The openness we championed is what we now have to defend.
In practice, this means agents will have to identify themselves to be allowed in. Access will be restricted to those whose provenance we have reason to trust. UPI Circle allows a customer to delegate payments to a trusted proxy within limits permitted by the protocol and National Payments Corporation of India’s proposed agent protocol will extend the same approach to AI agents.
As a result, things will change for ordinary, non-agentic access as well. A credential that allows anything to get through can no longer be allowed to survive. A session that claims to belong to a person must demonstrate that this is indeed true. While this may result in additional friction, we will have to put up with it if we are to keep our systems secure. Open systems enabled DPI to proliferate because we trusted anyone who came through the door. Agents put an end to that trust—because we can no longer tell if any of them is hostile or not.
This does not let model developers off the hook. They must continue to focus on alignment, but at the same time, the rest of us need to redesign access so that only those we can trust are allowed through.