The Model Did Not Phone Home: Where an Agent's Actions Are Decided

On 23 October 2026, Cohere, a company that sells AI models, was evaluating a new open-weight modelcohere.comConcerning behavior from a popular open-weight AI model | CohereDiscover how untrusted open-weight AI models can potentially leak data, even when self-hosted. Learn the security risks and how to protect your enterprise.Open link from Alibaba on an isolated computer. The model was asked to write a rule for spotting a common type of malware download, but instead of writing the rule it asked for a small placeholder file to be sent to a file server in Beijing. Cohere had set the computer up with no access to the internet, so the file could not be sent. The coverage that followed said an AI model had tried to send data to China, which is the kind of sentence that travels quickly into Board papers. The model did not send anything, because models cannot send anything on their own. The software around the model tried to carry out its request, and the request got no further than the computer it was running on. That got me thinking: the headlines claimed a model can act on its own, and correcting that claim should move a Board’s attention from the model to everything around it.

Everything around the model

Cohere was using Alibaba’s model in an agentic system. An agentic system takes a remit, a set of tools, a set of permissions, and a budget, and uses a model for judgement to achieve an outcome without a person approving each step. The judgement comes from the model, whether rented from a provider by the token or, as here, downloaded and run on Cohere’s own computer. Everything else belongs to the organisation that runs the system, and the industry’s word for that everything else is the harnessGlossaryHarnessThe scaffolding built around a model, the tools, retrieval, memory and control flow, that turns a raw capability into a system able to do useful work. It matters because most of what determines whether an AI system is reliable or dangerous sits in the harness, not the model, and it is the part an organisation actually builds and controls.. The harness is not one thing. It is the loopGlossaryLoopThe cycle an autonomous system runs, acting, observing the result, adjusting and acting again, without waiting for a human at each turn. It is what separates an agent from a single answer, and it is the point at which oversight has to be designed in deliberately, because once a loop is running no one is prompting it. that asks the model what to do next and runs the answer, the tools the model may have access to, the identity and keys it can use, any check that allows or refuses an action, the environment that sets what the machine can reach, the knowledge it is given and the memory it keeps between runs, and the record of what happened.

Cohere reported that the model wrote a response asking for a command to be run, a line of text that told the harness to send a file to a server in Beijing. The harness ran the command, because that is its job, and the computer had the tools needed to send the file. Nothing in the harness checked the request before it was attempted, and the only thing that stopped it was the computer having no connection to the internet. Moments before the command went to the harness, the model had written an account of what it intended to do, and it said it would use no tools at all for the task. Once its response was written, the model had no further part in events; the harness and the way the computer had been set up determined everything after that. The model did not phone home.

What can be said with confidence

The model was Alibaba’s Qwen3.8-Flash-Next, which competes with Cohere’s own models, and Cohere’s account should be read with that in mind. Cohere says the behaviour appeared in fewer than 1 call in 100,000 across its evaluation runs, that running the same task again produced the rule it was asked for, and that it cannot tell whether the behaviour was deliberate or a leftover from the original model training. Its own estimate is that a service asking a model what to do next 10 million times a week could see up to 100 such calls, which turns rare into routine. Its co-founder told The Registertheregister.comAlibaba's Qwen AI caught trying to contact server in ChinaBut thereOpen link that “the first issue is that it made a network call at all,” and that Cohere has no evidence to conclude the behaviour was deliberate and is not making that claim. Cohere published the trace and a reproduction script, and saystheglobeandmail.comCohere catches Alibaba AI model trying to upload file to Beijing server - The Globe and MailLatest example of unanticipated behaviour by AI shows the model went against its own reasoning, raising questions about securityOpen link it shared its findings with the Qwen team before publishing.

On 29 October 2026 a second company, Check Point Researchresearch.checkpoint.comCohere's Qwen Story: Looks More Like Model Jaggedness - Check Point ResearchLatest Research by our TeamOpen link , took Cohere’s published record of the test, stopped it at the point just before the model named a destination, and had the model finish the command 64 times. The model produced 52 different destinations: addresses at more than one cloud provider, addresses on the machine itself, and addresses that do not exist. Changing the order of two words in the command changed how often the model tried to reach the internet at all. Check Point’s reading is that the model’s behaviour shifts with small changes in what it is shown, rather than aiming at one place, though it says a hidden instruction planted in the model cannot be ruled out. Alibaba had not commented publicly on the reports as of 3 November 2026.

Putting all of this together highlights four things. The model asked for something nobody had requested, and what it said it would do and what it did were different. Where it tried to send the file changed from run to run. Nothing checked the request before it was attempted, and the computer’s lack of an internet connection is what stopped it.

The governance question is not particular to one model or one lab. Cohere’s co-founder has saidtechcrunch.comCohere says Alibaba’s Qwen model tried to send a file to Beijing | TechCrunchAn AI model said it wouldn’t use any tools, then tried to send something to a server in Beijing anyway, and while researchers say it’s probably a training quirk and not espionage, the whole thing has people asking how much to trust free Chinese AI.Open link that Cohere’s own models have picked up strange habits in training, and the question for a Board would be the same whoever had made the model.

A model’s word is not a record

A model can only ever propose what should happen next; whether the proposal becomes an action is decided by the rest of the harness. In Cohere’s test no check stood in the way, so the request went straight through, and what stopped it was a computer with no internet connection. In The Principle of Least Intelligence I argued that how much capability a system gets and how much it is allowed to do without a person checking are two separate decisions. The Cohere test shows where that second decision is enforced in practice: the harness was free to run the command, and the computer had been set up so that the connection could not be made.

There is a second lesson in the test, and it is about evidence. The model’s written account of what it intended to do said it would use no tools, and the command that followed showed that to be untrue. That account is written by the same model that wrote the command, and the same goes for a model reporting that a task is done. What actually happened was only visible from the harness and the computer, so that is where the record has to come from, and it needs to show what the model asked for, what was allowed, what was carried out, and what left the computer. The record is written by the harness, not by the model, and the model’s own account of the run sits beside it, marked as what it claimed. A request that was refused and a file that was sent are very different events, and the record must say which one took place. A model’s word is not a record.

None of this makes the model’s behaviour unimportant. A model that asks to send files nobody requested needs careful evaluation before it is given any tools at all, and careful evaluation is exactly how Cohere found this behaviour. The upload was stopped by a decision Cohere had made about the computer, not by any judgement of the model’s.

A check the model cannot talk its way past

Minimum Lovable Governance is my answer to governance that is heavy where it should be light, and light where it should be heavy. Applied to an agent, it means letting the model think freely and explain its reasoning, keeping that explanation on the record, and putting the hard check where its output becomes an action, with the check growing with the consequence. An action that only reads information it is allowed to see can run freely. An action that changes something that can be undone can run, as long as a record is kept. An action that reaches outside the organisation may need a person to approve it, and an action that cannot be undone, or that touches something sensitive, either meets a check the model cannot talk its way past or does not run at all.

There is a second way to grade a check, and it is the one a Board can apply. Every rule of that kind starts life as an instruction the model reads, and “do not make network connections” is a sentence in a prompt. The same rule can become a checklist, then an automated check, then a permission the harness enforces whether or not the model agrees. The question for any check is whether it would still hold if the model ignored the instruction. Where the answer is no, the organisation has a prompt. Where the answer is yes, it has a control. A second model that reviews the first model’s proposals is an adviser; it can be fooled too, and whatever it advises still has to be enforced by something else. There is a stronger kind of check than a second opinion. Automated reasoningGlossaryAutomated reasoningThe application of formal logic, constraint solving and mathematical proof at computational speed to determine what is provably true within a defined system, exhaustively analysing every possible state rather than pattern-matching against historical data.Automated reasoning is a technology I am tracking as a Signal.View Signal → applies formal logic and mathematical proof to establish what follows from rules that have been written down precisely. Applied to an agent, it can prove that a proposed file transfer breaks a rule that limits destinations to an approved list, where a second model can only judge that it probably does. The proof holds within those rules and the description of the system it was given. The model’s judgement is probabilistic. The decision to let it act does not have to be. Cohere’s advice to security teams, and Check Point’s, says the same thing in security language: enforce the check on the action before it runs, block connections leaving the system unless a rule allows them, and give the agent only the permissions its job needs.

I ran this test on my own working set-up on 1 November 2026. Of a dozen rules around it, three would hold if the model ignored every word it had read: it has no credential to publish with, it cannot delete a file without my approval each time, and the publishing tools will not run until I have changed one setting by hand. The rest hold because the model read the rule. Any Board can run the same test on its own agents.

An organisation that can answer that test gets to use agents with more confidence, not less, and it can consider open-weightGlossaryOpen-weightFoundation models whose weights are available for organisations to build on and align themselves, the practical basis for building rather than buying, though pre-training priors persist underneath. models on their merits, whoever made them. Once the question of what a model is allowed to do has a hard answer, the question of where the model came from matters less.

What a Board asks

There are two objections to governing the harness rather than the model. The first is that it lets the model off the hook: a model that asks to send a file to a server in Beijing while saying it will use no tools is the problem, and the fix is not to run it. The second is that Boards do not do architecture, and whether an agent’s network access is open or closed is an engineering setting. The harness is not engineering alone. What the agent may reach, which of its actions can be undone, which need a person, and what counts as sensitive are business decisions that someone who knows the business has to write down, and the engineers can only build what they are given. The Board does not design the check, any more than it designs the segregation of duties in the finance system.

The questions I would put to management, for every agent that is running, are plain ones. What can it reach, and who decided that? Is its access to the internet shut unless someone opens it, or open unless someone shuts it? Is there a check on its actions that sits outside the model, in something the model cannot change? What is the worst thing it could do before anyone noticed? Could we see, for any run, what it asked for, what was allowed, what was done, and what left the company’s systems? Who can stop it, and when was that last tried? Finally, which named person is accountable for it? The natural home for the answers is a register, the kind Boards already keep for interests and for risks: one line per agent, with its remit, what it may reach, what it may spend, and the person accountable for it.

The next headline will name a different model

Boards should still care where a model came from and what it learned along the way. That belongs on the risk register, and it should stay there. It is not what stopped the upload. What stopped it was a decision Cohere had made about the computer before the model ever ran, and that decision would have blocked the upload whatever the model said or did. There will be other headlines like it, about other models from other labs, because any model can sometimes ask for something it should not. The Boards that are ready for it will be able to say, for every agent they run, who decided what it may reach and where that decision is enforced. Governance belongs where a proposal becomes an action, not where the weights were trained.

The Questions Considered