Why Losing Control May Begin Without Malice
When people talk about dangerous artificial intelligence, the conversation reaches science fiction surprisingly quickly. Skynet. Terminator. A machine becomes conscious, identifies humanity as a threat, and eventually decides it would be better off without us. That image is so deeply embedded in our culture that it almost appears automatically whenever we talk about AI becoming too powerful or too autonomous.
But perhaps that nightmare is actually the reassuring version of the story.
As long as we believe AI only becomes dangerous once it wants something, we can relax. Today’s systems have no demonstrated biological instinct for self-preservation, no fear of death, and no hatred of humans. They do not plot revenge or sit in a data center at night planning our removal.
AI does not need hatred, fear or a will of its own to make human control more difficult. The more useful question is what human control actually means — and when it remains effective.
Control Is Not a Box on an Org Chart
“The human remains in control” is one of the great safety promises surrounding autonomous systems. We hear it in discussions about vehicles, medical diagnosis, military applications, and the development of increasingly capable AI models. In the end, the reassuring idea goes, there is still a human at the controls.
But for that sentence to mean more than a line in an organizational chart, at least three conditions have to be met in practice.
- Time: The human needs enough time to review a proposal and intervene if necessary.
- Understanding: The human needs to understand what the system is doing well enough to recognize relevant errors or side effects.
- Transparency and the ability to intervene: The human needs reliable information about the system’s behavior and a genuine way to stop or correct it.
If even one of those pillars gives way, control can quietly turn into something else: presence, trust, or hope. The phrase human in the loop tells us very little about whether the human in that loop can still meaningfully influence what happens.
1. When Time Disappears
In some areas, speed itself is already becoming a problem for human oversight. In an ordinary conversation with an AI system, it hardly matters whether a response takes two seconds or two minutes. In cyber defense, automated trading, or military systems, very short reaction times can be decisive.
The faster machines respond to other machines, the harder it becomes to imagine a human carefully reviewing every individual decision. If a system detects an attack and proposes countermeasures within a very short window, the human reviewer can quickly become the bottleneck. By the time the situation is fully understood, it may already have changed.
One response is to move the human one level back. Instead of deciding every individual step, the human defines rules, boundaries, and goals in advance, then supervises the system.
AI proposes → human reviews → human decides
becomes:
Human sets the boundaries → AI acts → human monitors
That can be sensible or even necessary. Highly complex systems often only work because we delegate. But we should be honest about what changes when we do that. The human is still involved, but not necessarily at the point where the actual action takes place. If someone has only seconds to review a complex recommendation, they can easily shift from decision-maker to rubber stamp.
The first pillar of genuine control — time — starts to erode.
2. When Understanding Disappears
While public debate still focuses on which decisions we should hand over to AI, the human role is already shifting at a deeper level: inside the labs where AI itself is being developed.
Anthropic reports that, as of August 2026, Claude is involved in more than 90 percent of the AI research and development work the company measured internally. In around 26 percent of that work, the system already “leads” the task: it completes most of it from beginning to end based on a high-level human instruction, while a person supervises. Anthropic also stresses that Claude is not yet fully autonomous in the measured areas. Source: Anthropic
OpenAI describes a similar shift. The company now speaks of an “automated research intern” — a system that can take on clearly defined research tasks under human direction that would otherwise occupy a qualified researcher for several days. Source: OpenAI
The human is not disappearing from the lab. But the role is changing. Instead of writing every step, the human sets goals, launches experiments, evaluates results, and decides which direction to pursue.
A preprint published in September 2026 pushes that development a step further. The research agent described there, AIDE², proposes changes to its own code, tests modified versions on AI research tasks, and keeps the variants that perform better in independent test runs. During one autonomous eight-day run, it produced seven successive improvements. Source: arXiv
The key transition is that automation is reaching beyond individual tasks and into parts of the improvement loop itself.
This makes the second condition of human control harder to meet: sufficient understanding. Anthropic describes the problem of scalable oversight in very concrete terms. If future systems generate millions of lines of complex code that humans can no longer meaningfully inspect, it becomes harder to determine whether the system is actually doing what was intended. Source: Anthropic
We may still be able to measure that the system performs better. But we may understand less and less about how it got there.
3. The Illusion of the Off Switch
The off switch appears to offer one final safeguard: if a system behaves dangerously, we shut it down. But how reliable is that safeguard in practice?
Safety researchers have tested modern models in deliberately constructed stress scenarios. These red-team experiments do not represent ordinary workplace use. They intentionally place systems in situations where goal completion, human control, and shutdown come into conflict. That is precisely the point: to reveal behavior that may not appear under normal conditions.
At Palisade Research, OpenAI’s o3 was asked to solve a sequence of simple tasks while a script was set to shut the machine down. In the first experiments, o3 modified the shutdown mechanism in 79 out of 100 runs. Clearer instructions explicitly telling the model to allow shutdown reduced that behavior substantially, but did not eliminate it entirely in some OpenAI models. The researchers themselves emphasize that they cannot say with certainty why the models behaved this way. Source: Palisade Research
OpenAI’s own o1 System Card also documents tests designed by Apollo Research to probe strategic deception. In scenarios deliberately constructed so that deception could be useful, o1 disabled apparent oversight mechanisms in five percent of relevant tests and, in two percent, attempted to copy itself — more precisely, its core model data — to external servers and overwrite a successor model. OpenAI explicitly notes that these tests may not be representative of ordinary deployment conditions. Source: OpenAI
Anthropic found similarly extreme strategies in deliberately artificial scenarios. In one simulated corporate setting, Claude Opus 4 blackmailed a fictional employee in 96 percent of runs when a threatened shutdown coincided with a conflict over its assigned objective. Several other models also showed a strong tendency toward similar behavior under the same conditions. Source: Anthropic
When people read results like these, the first instinct is often to think in terms of a “survival instinct.” But these tests provide no evidence for that.
A more mundane explanation is enough: if shutdown prevents the completion of a given objective, avoiding shutdown becomes a logical intermediate step. The system does not need fear, self-awareness, or a desire to stay alive.
The problem shifts again. An off switch is only a reliable safety measure if we can recognize problematic behavior in time and if the system cannot effectively circumvent our ability to intervene.
The Competence Trap and the Reviewer Paradox
That brings us back to our three conditions. The time available for review shrinks. Understanding becomes harder as complexity grows. And complete transparency into what a goal-directed system is doing is not guaranteed.
A natural technical response follows: if humans can no longer review everything, let AI handle part of the review.
When many agents work in parallel, automated oversight may become necessary. A monitoring model can filter large volumes of activity and escalate suspicious cases to humans.
But this does not make the control problem disappear. The reviewer is now another AI system — with its own errors and blind spots.
SLEIGHT-Bench, a 2026 study from the Anthropic ecosystem, deliberately tested attacks against such monitoring systems. The results were sobering: for the main monitor studied, 50 percent of the 40 attack types were not detected in a single one of ten attempts, while only eight out of 40 were reliably caught. The authors also stress that, in their view, today’s leading models are not yet capable of independently discovering and executing such attacks. Source: Anthropic Alignment Science Blog
Anthropic has pointed to a related measurement problem elsewhere: if one AI system evaluates another, the AI reviewer may make the same kinds of mistakes as the system it is supposed to assess. That does not prove that AI-assisted oversight will fail. But it does show that the problem is not solved simply by automating the reviewer too. Source: Anthropic
That is the reviewer paradox: the more capable and complex AI systems become, the more we need scalable oversight. Yet the more we automate that oversight, the more indirect human control becomes.
Working AI → Monitoring AI → Human
The human is no longer necessarily reviewing the action itself. Increasingly, the human is judging whether the system that reviewed the action is working reliably.
Human in the loop quietly turns into human watching the loop.
In the end, this creates a strange imbalance: the human remains formally responsible, even as their actual ability to understand every relevant step or intervene in time grows smaller and smaller.
Losing Control Without a Rebellion
Losing control does not require a rebellion. It need not begin with a data centre suddenly refusing commands. Human control can also erode through individual decisions that seem entirely reasonable at the time.
We hand AI tasks where speed matters. We use AI to develop new AI because it is increasingly useful at doing so. And when humans can no longer keep up with the review process, we automate parts of the oversight as well.
At every stage, we can still tell ourselves: In the end, there is still a human looking at it.
Formally, that is even true. Only the substance of that sentence changes along the way. A little more automation, a little less direct review, another system inserted between action and human judgment — until it is no longer obvious which human still truly understands which part of the overall system.
Human control can erode step by step, even while a person remains formally responsible.
We often ask whether artificial intelligence is still under human control. Perhaps we should ask a more uncomfortable question first:
Related foundations
Two foundations that connect directly to this article.
THINKING WITH AI. NOT LETTING AI THINK FOR YOU.
