Humans on the Loop: Governing What an Agent Does, Before It Does It

On 29 October 2026 the founder of Spawn, a 10-person games company, explainedx.comjacob on X: "How I Built the Most AI-Pilled Company in the World" / XHow I Built the Most AI-Pilled Company in the WorldOpen link how the business now runs on a fleet of permanent AI agentsGlossaryAI agentAn AI system that plans, draws on tools and information, and carries out multi-step tasks within a loop it drives itself, rather than answering a single prompt. Its work is only ever as good as the data and tools it can reach, so the governing question is not how capable the agent is but what you have connected it to.. Each agent is given a part of the company and the processes that come with it, and it makes the decisions and takes the actions that part needs: deploying code, triaging bugs, keeping the systems running. On 4 November 2026, at the FT’s Future of AI Summitai.live.ft.comFuture of AI Summit 2026 | 4-5 November | FTLiveJoin the Future of AI a live event by Financial Times, held in London from 4-5 November 2026. Register for in-person or digital pass. Meet those at the cutting edge of AI innovation and explore how this transformative technology will redefine business in 2026 and beyond.Open link in London, IgniteTech’s chief executive described how his company uses agents to handle bug reports and feature requests: working out what to do with each one, fixing the bug or building the feature, testing it, and shipping it. All of this was happening while he was on stage talking about it.

AI-native is a phrase that turns up a lot in the conversations I have. These two companies are living it: the technology has been designed around what the business needs, and the people now set the direction, set the rules, decide what good looks like, and step in where the agents cannot go. Nobody in either company approves each step, the agents act on their own inside those rules, and the rules are where the limits of their autonomy sit. What makes this possible is what has been built around the model: the harnessGlossaryHarnessThe scaffolding built around a model, the tools, retrieval, memory and control flow, that turns a raw capability into a system able to do useful work. It matters because most of what determines whether an AI system is reliable or dangerous sits in the harness, not the model, and it is the part an organisation actually builds and controls., which I described in The Model Did Not Phone Home as everything around the model that decides whether its proposals become actions.

What an agent is

Harnesses are not new. When a person works directly with a model, the person is the harness: deciding what to ask, whether to believe the reply, what to do with the output, and when to stop. Agentic AI moved those decisions into software: a program asks the model what to do next, acts on the answer, and asks again until the work is done or something tells it to stop. The person who used to sit between the model and the real world has stepped out of the loopGlossaryLoopThe cycle an autonomous system runs, acting, observing the result, adjusting and acting again, without waiting for a human at each turn. It is what separates an agent from a single answer, and it is the point at which oversight has to be designed in deliberately, because once a loop is running no one is prompting it.. On its own, that loop has no rules in it: nothing says what the model may not do, what it may not touch, or when it has to stop, so it cannot be trusted with much. Writing those rules before an agent runs is what turns a loop into something a business can govern.

That is where the harness comes back, as software this time rather than a person. The model supplies the judgement each piece of work needs, reading the request, working out what it means, and deciding what to do about it. The harness carries the judgement people made in advance about the limits of that work: which steps are allowed, what must never be touched, and when to stop and ask. It holds the tools the agent is allowed to use, the credentials that decide which systems it can reach, and the budget it can spend before it must stop. It holds the checks a proposed action has to pass before it happens, and the rule that says when the run ends, when it pauses, and when a person has to be asked. And it keeps the record: what was asked, what was allowed, what was done, and what left the organisation’s systems. The model and the harness together are what an agent is.

Creating the harness is not a job for engineers alone. Some of the harness is code: the loop that calls the model, the connections to its tools, and the gates that refuse an action. But what counts as a good result, what the agent must never touch, and when a person has to be asked are written in plain language by someone who knows the business. Only the rules that have to hold whatever the model says are then turned into checks the harness enforces. That was my point at the FT summit’s private roundtable: building a harness takes governance and domain expertise as much as it takes software. The people who used to sit between the model and the real world are still there, but humans in the loop have become humans on the loop.

The harness also settles who decides what the agent does next. There are three answers. A person can decide every step before the run and write the steps down, so the agent follows a fixed workflow that could be drawn as a flowchart, and the model only does the work inside each step. A person can set the workflow but leave a small model to decide which way each request goes at each point. Or the model can run in a loop, deciding each step itself as it goes, with nothing fixed in advance except the rules in the harness. Who picks the next step is a different question from who says yes to it. The agents on my own machine pick their own next step and still ask me before they do anything that matters, which is a loop with a person in it. At Spawn and IgniteTech the yes has moved into the harness, and a person is asked only when the rules say so. Each of the three gives the machine more of the deciding than the one before, and the first thing a Board needs to know about any agent is where its workflow is fixed, where the model is choosing, and who approved that.

The people who build these systems gave both jobs names in 2026. Harness engineeringarxiv.org[2609.00006] Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven SystemsAbstract page for arXiv paper 2609.00006: Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents – A Source-Code Study of Eleven SystemsOpen link is the design of everything that wraps the model: the limits, the checks, and the record. Graph engineeringGlossaryGraph engineeringThe design of an agent’s workflow as a drawn set of steps and the moves allowed between them: the flowchart the work follows, which engineers call a graph. A newer and still unsettled term from 2026. The graph is the workflow, not a knowledge graph. Where the workflow is fixed in advance, people decided each step before any work arrived, so a Board can ask who designed it and who approved it., a newer and still unsettled term, is the design of the workflow: the steps, and the order the work moves through them. The graph in the name is that flowchart, not a knowledge graphGlossaryKnowledge graphThe working network of typed entities and explicit relationships built to an ontology; the manifestation that lets information be connected, traversed, and reasoned over rather than merely retrieved. It is not the graph of graph engineering, which is the drawn workflow an agent follows.Knowledge graph is a technology I am tracking as a Signal.View Signal →. Each job answers a question a Board will want to ask: who set the limits on this agent, and who designed the steps it follows.

When the model picks the next step

A fixed workflow can be read, reviewed, and tested before it is ever used, and management can say who designed it and who approved it. That stays true when small models make the decisions along the way. A decision modelGlossaryDecision modelA model that does not write text. Given a fixed set of options, it returns a calibrated probability for each, so software can act on the answer without a person reading it; it is built for routing, classification, prioritisation, verification, and workflow control. It cannot return an option it was not given, but it can still return the wrong one, so for a Board the question is the threshold: above what probability does the organisation act without a human, who set it, and what happens below it. reads a request and, instead of an answer in words, returns a score for each of a fixed set of options. The software sends the work to the option with the highest score, provided that score clears a threshold the organisation has set. Two decision models shipped in 2026, from TypeSafe AIdocs.typesafe.aiSystem One - TypeSafe AISystem One models make fast, structured decisions for software. Jev is TypeSafe’s flagship model and the first System One model.Open link and Microsoftcommandline.microsoft.comMicrosoft-Decision-1: Our model for fast decision-makingOur new decision-scoring model delivers top performance in latency and quality on structured decision tasks to outperform both LLMs and other decision models.Open link . A person still sets the options and the threshold, so a Board governs this the way it governs the approval limits in the finance system: who set them, when they were last reviewed, and what happens to the cases that fall outside them. The harder case is when the model picks the next step.

Where the yes has moved into the harness, the model chooses each step and the harness carries it out, and nobody sits between the two. By the time an action happens, the only approval the agent has had is the one written into the rules before the run began. The tools the agent may use, what it may reach and spend, the checks an action must pass, and the rule that stops it are all that stands between a proposal and an action while the loop runs. The industry draws the line in the same place. Anthropic’s guidanceanthropic.comBuilding effective AI agents \ AnthropicPractical guidance from Anthropic on building reliable AI agents, from simple composable patterns to fully autonomous systems.Open link separates workflows, where models and tools follow “predefined code paths”, from agents, where models “dynamically direct their own processes”, and recommends a stopping condition to keep control, such as a limit on how many times the loop may repeat. OpenAI’s guide describes agents as systems that work independently on a user’s behalf, and names too many failed attempts and high-risk actions as the points where control goes back to a person. Both are describing the rules, because in a loop nothing else governs the agent while it runs.

Whether the rules are enough is the next question.

Checks the model cannot argue with

Most of the rules in a harness are written for the model to read. A model can read a rule and still not follow it: if it does, good, and if it does not, the rule was only ever a suggestion. The harness is what makes a rule hold either way, because it checks what the model asks it to do instead of taking the model’s word for it. Some rules can be enforced before the model is even asked. If the agent has not been given a tool or a credential, it cannot use them, and it cannot spend beyond its budget, whatever it decides. Other rules can only be checked once the model has said what it wants to do, because only then is there something to check: that a change passes its tests before it ships, or that a payment matches an invoice before it goes out. And some checks can only happen afterwards, by looking at what was done. For anything that cannot be undone, the check has to come before the action, not after. The research bears this out. A 2025 studyarxiv.org[2503.13657] Why Do Multi-Agent LLM Systems Fail?Abstract page for arXiv paper 2503.13657: Why Do Multi-Agent LLM Systems Fail?Open link led by the University of California, Berkeley, found that failing to check the work is one of the three main ways multi-agent systemsGlossaryMulti-agent systemsSwarms of specialised AI agents that collaborate, compete and produce emergent behaviours, shifting governance from controlling individual models to orchestrating entire agent ecosystems and their interactions. break. Anthropic’s own accountanthropic.comEffective harnesses for long-running agents \ AnthropicHow an initializer agent and an incremental coding agent help the Claude Agent SDK make steady progress across many context windows.Open link of running coding agents for hours reports the agents declaring work finished too early until they were given a written definition of done and told to test against it. A test the agent has to pass is a check the model cannot argue with.

So the Board question is not whether the model follows the rules. It is who writes them, which of them are enforced, and whether the record shows they held. The rules are decisions people made before the run: what the agent may reach and spend, what counts as done, and which actions are checked before they happen rather than afterwards. A Board can ask who made those decisions, and when they were last looked at. Where the answer is that nobody did, the agent is running on suggestions. Repetition is a reason to automate, and a loop that is checked on every pass can be trusted with a great deal. A loop is safe where its work can be checked, not where the task repeats.

What the Board approves

The Board does not write the rules, set the thresholds, or design the checks. Management does all of that and is responsible for it. What the Board needs is evidence that those decisions were made intentionally, by named people, for reasons that can be explained and defended. The Principle of Least Intelligence, a RemakeGlossaryRemakeEvery organisation does four kinds of work: thinking, deciding, creating, and delivering, and AI is remaking all four. Remake is how a Board governs that remaking and how an organisation executes it: establishing what the work is, who does the remaking, who is accountable for it (a gate, not a description; where genuine agency or durable accountability is absent, the remaking does not begin), what the verdict on the work is (Stop, Keep, or Change), and how it is done through the ADAPT motion. The mature successor to the Complete AI Adoption Framework.Remake is a concept I first introduced in “Remake”.View article → principle, asks for the least capability that achieves the outcome. Applied to permission, it asks for the least discretion. A loop earns its place where letting the model choose has a clear benefit and each pass can be checked. A loop where a fixed workflow would do the same job is capability bought and exposure accepted for no gain. How much discretion the agent is given, and on what terms, is what the Board approves.

The questions are plain ones, and a director can ask them of any agent without knowing anything about the model inside it. Which steps were fixed in advance, and which does the model choose? Who wrote the rules, and when were they last reviewed? What can it reach, and what can it spend? Which actions are checked before they happen, and what happens when a check fails? What happens when it cannot finish, and who is told? The answers have to exist before the agent runs. Written afterwards, they only describe what the agent did, not what it was allowed to do. And whatever the answers, the Board stays accountable for what the agent does. As I argued in The Accountability Gap, the decisions and the actions can be handed to the machine and the accountability cannot.

The decision that follows, when it comes, is whether to let the agent do more.

Before the remit grows

Once an agent is running, the next request is usually to let it do more: another tool, a bigger budget, a decision that a person used to make. Each of those changes the rules it runs inside. Some of those changes sit within what the Board has already delegated to management. Where a change goes beyond that, the same questions are asked again, and this time there is a record to read: what the agent did with the discretion it already had. That record is what the Board looks at before the remit grows.

Where the harness itself runs, and who can get at it, is the subject of the next article. None of this makes the model unimportant; it still has to be good enough for the work. An agent is as safe as the checks that tell it when to stop, not as clever as the model inside it.

The Questions Considered