If AI models can continuously train themselves even after release, our current approach to evaluation and testing will no longer be sufficient to keep us safe. We will need to work harder to regulate the products of distributed intelligence and shift liability downstream where it deserves to be.
In June last year, researchers at the Massachusetts Institute of Technology published a paper describing a process by which large language models could generate training notes and use them to update their own weights. A one-billion-parameter model that could not solve a single puzzle in the test set managed to solve over 70% of them by using this technique. In a separate test, the notes a larger model wrote for itself proved better training material than those generated by GPT-4.1.
Today, AI models are trained by adjusting their weights on vast amounts of data until they perform as intended. The weights are then frozen, and the model released. From that point onwards, the model weights remain static—nothing that it encounters, not the documents we give it to read, nor the preferences we ask it to remember, is embedded in the model.
Self-Adjusting Weights
Since a model’s weights will not change once it is released, all our safety evaluations are conducted before that. This is also why regulators treat metrics such as the compute required to train a model or the number of parameters it contains as useful proxies for its capabilities. Once continual learning makes it possible for models to use their inputs to adjust their weights on the fly, all of this will have to change.
A model whose weights continuously change can soon become very different from the model that was released, so any evaluation conducted before release can only tell us what it could do on the day it was tested. Since small continuously trained models could quickly become far more capable than their size would suggest, while remaining well below any threshold that would bring them under scrutiny, our current approach to evaluating risk will no longer be useful.
Today’s frontier models are built like Swiss Army knives, packed with more knowledge and capabilities than any one user will need—because their makers can never know exactly how they will be deployed. Continual learning allows us to build each model with basic capabilities that can be augmented by learning from the environments in which it is used. This will enable far greater efficiency, as each AI can excel only in the domain in which it is used, while more general capabilities irrelevant to its role fade away. It is only a matter of time before AI labs ship such models. All that is holding them back from doing so already is probably concerns around safety.
Continual Learning in Practice
Let’s consider how this would work in a law firm. Today, law firms use general-purpose models to analyse their documents, hoping each time that the harnesses and workflows they have built will result in the correct outcome. A continual learning model, on the other hand, would ingest every opinion the firm writes, every transaction it structures and every correction a partner makes to an associate’s draft by adjusting its weights to reflect them. Eventually, the model will represent the sum total of the judgment of the firm’s best lawyers, becoming better over time at working out what the law permits and where its edges lie than any general-purpose model could.
Any model continuously trained by a law firm could become exceptional at identifying loopholes in the law that criminals could exploit. Similarly, one that is continuously trained in a hospital would be outstanding at identifying treatments that could kill someone without being detected. Since all these nefarious capabilities are acquired post-deployment, our current approach to AI safety would be helpless to protect us from the harms that could result.
Drawing the Line
If this is the direction we are headed, we would do well to think through how these new risks can be mitigated. One side-effect of continual learning seems to be that the more a model’s weights are adjusted post-deployment, the more likely it is to ‘forget’ the safety guardrails embedded in it. This is something model developers will have to address before they release these models to the public. But once they have done all that can be expected of them in that regard, the model builder should be protected from the harms that could result from what users do with it.
This approach is not unlike how we treat all providers of utility services. When electricity was first supplied to households, the wiring regularly gave people shocks, and, when badly laid out, set houses on fire. Yet no one held the power generation company responsible. Over time, the law came to draw the line at the meter and the electricity company’s liability ended there. Everything beyond it—the wiring, the appliances and the uses they were put to—was the responsibility of the user household.
We need to apply the same rules to AI. Developers need to ensure that they have built guardrails that can endure the forgetfulness that is a normal side-effect of continual learning. Having done so, their liability should end at the meter. What happens beyond that as a result of what users choose to continuously train their intelligence on cannot be the responsibility of the provider of an intelligence model.
Deployers of continual learning models must be strictly liable for any resulting harm, regardless of the precautions they took. The liability should be serious enough that no one would use such a model without caution. This is what every government will need to do as AI becomes what Arvind Narayanan and Sayash Kapoor, in their 2025 essay, call a normal technology, and India, as the “use-case capital of the world,” should be the first.