Nothing Has to Leave the Building: The AI Risk That Is Not a Data Breach

This month some of the most sophisticated buyers of AI in the world started limiting what they send to it. The Information reportedtheinformation.comAnthropic Data Fears Prompt Nvidia, Palantir and Booz Allen to Restrict Model Use — The InformationAs paranoia rises over whether Anthropic or OpenAI could learn from their customers’ intellectual property, large firms involved in sensitive corporate work, such as Palantir Technologies, Nvidia and Booz Allen Hamilton, have started demanding new guarantees or reducing or eliminating use of the …Open link on 14 September that Nvidia, Palantir, and Booz Allen Hamilton are among the organisations restricting which work goes to the most capable frontier modelsGlossaryFrontier modelA model at the leading edge of AI capability, typically trained at the largest scale and governed under the tightest jurisdictional controls; availability can be withdrawn by regulation as well as by commercial choice.. The report says Nvidia keeps its sensitive internal work on models of its own, and Booz Allen Hamilton’s chief technology officer gave the reason in plain words: the firm worries that a model “might be learning from some of our code”. What I notice about that worry is that it is not about a data breach, because nothing has to leave the building for a model to learn from the way people work in an organisation.

This should matter to Boards because the organisations they govern pay for the judgement of their best people, and those people now do more of their work inside these tools. Suppliers will commit in a contract not to train their models on a business customer’s prompts and outputs, and I have no reason to doubt them. The worry is about what sits around that commitment: the feedback, usage data, and other signals a tool produces as people work in it. If the tools can learn from those, some of that judgement may be improving a model that is sold to everyone, competitors included, and the Board would have no means of knowing. Most of the Boards I meet have approved the use of a frontier model, and few have been asked to decide which work should go to it and which should stay at home.

One customer asked

Shortly before that report, one customer put the question to his supplier directly. On 8 September OpenAI announcedopenai.comOn the Navier–Stokes Millennium Prize Problem | OpenAIWe’re sharing an AI-generated solution to the Navier–Stokes Millennium Prize Problem, including a writeup and a formal proof in Lean.Open link that a system of around 10,000 cooperating agents had produced a proof for one of the seven Millennium Prize problems in mathematics. An NYU mathematician had spent months on a closely related problem, and had been putting “all our drafts for the whole of this project” into OpenAI’s Codex tool. He asked the company whether its model had been trained on his sessions, or had access to them. He was told the model did not look up user data. On trainingGlossaryTrainingThe process of teaching a model by adjusting its parameters against data; the largest up-front cost in building an AI system., he wrote, “I did not get an answer.” He was careful to add: “I am not accusing anyone of anything.” On the day of the announcement OpenAI’s chief research officer drew the distinction in publicx.comMark Chen on X: "Two things to distinguish: Did any human or agent look at user data as part of the Navier Stokes effort? No. Do we use user feedback and de-identified data to improve ChatGPT and Codex in a holistic way? Yes. And so does every LLM company." / XTwo things to distinguish: Did any human or agent look at user data as part of the Navier Stokes effort? No. Do we use user feedback and de-identified data to improve ChatGPT and Codex in a holistic way? Yes. And so does every LLM company.Open link : “Did any human or agent look at user data as part of the Navier Stokes effort? No. Do we use user feedback and de-identified data to improve ChatGPT and Codex in a holistic way? Yes. And so does every LLM company.”

Two days later OpenAI updated its announcement. It had investigated, and it said his Codex prompts over the preceding two months “could not have influenced the system in any way, including through training.” That is a clear answer, given quickly, and I take it at face value. The point I would draw for a Board would hold for any supplier: the answer could only have come from the supplier itself. A customer cannot look inside a model to see what it has learned, so the customer depends on being told. A Board is in the same position, so the decision it can govern comes earlier: which work goes into these tools in the first place.

Transferring your alpha

One company has given this risk a name. Palantir sells access to these models through its own software, so it has an interest in the argument, and in July it published a paper titled “AI Sovereignty is Your Alpha: How to Avoid Transferring Your Alpha to a Hosted Model Provider.”blog.palantir.comMediumAI Sovereignty is Your Alpha: How to Avoid Transferring Your Alpha to a Hosted Model Provider Use of third party AI model services poses significant risk to your alpha. Without sovereign control over …Open link It defines alpha as “your unique institutional knowledge and tradecraft embodied in data exposed to and generated from your use of AI models”, and it describes three routes by which that alpha can reach a provider: through model weights and services the provider later sells to the market, through “metadata about your patterns of use”, and through prompts that trigger a provider’s safety classifiers and are then used to improve them.

Palantir’s argument is that a model may get better at what an organisation’s best people do, and that the improvement is then on sale to everyone. That is a vendor’s claim rather than an established fact, but it is the same worry Booz Allen Hamilton’s chief technology officer put in his own words. Boards already know how to protect expertise when it leaves through a person. Employment contracts carry confidentiality clauses, non-compete clauses, and garden leave for that purpose. When the same expertise may leave through a tool, those terms were not written for it, because no employee has gone anywhere and nothing has been disclosed in the way a confidentiality clause has in mind. Data-protection law does not help much either, because it is not designed to protect an organisation’s know-how as such. Boards already treat a data breach as a serious exposure, and they should go on doing so. This is a different kind of exposure: a transfer of capability, not a breach of data.

The fair objection

I expect some directors to push back on all of this, and the strongest version of the objection deserves an answer. It says that even if a supplier’s model does improve a little from the way thousands of customers use it, no competitor ever receives a copy of how we work, so where is the harm? My answer is that an advantage is always relative. A competitor does not need a copy of what makes an organisation good, it only needs the gap to narrow, and if a widely sold model gets better at the kind of work that sets an organisation apart, every buyer of that model moves a little closer for the price of a subscription. I cannot tell a Board how large that effect is, and I do not think anyone can, because it cannot be seen from the outside and it cannot easily be undone once it has happened. The same report shows how little even a careful buyer can see. C Spire, a telecoms operator, has contracts that stop its AI suppliers training on its data but let them collect technical usage data, and its chief AI officerGlossaryChief AI officerAn emerging executive role, distinct from Chief Data and Chief Technology Officers, accountable for AI strategy, governance, and value realisation as AI becomes a business capability spanning both data and technology., who was clear that he did not see a red flag, called what is collected “more of a blank spot in our radar”. If the effect turns out to be small, a simple rule about what goes where will have cost very little, and if it does not, the rule is what protected the organisation.

How the buyers are responding

The buyers in the report are responding in two ways. The first is to keep their most sensitive work on models they control, as Nvidia is reported to do. The second concerns the terms. Nvidia’s vice-president of enterprise AI told The Information: “we believe ZDR should be on by default.” ZDR is zero data retention: broadly, an arrangement under which the supplier agrees not to keep prompts or answers once a request is complete, although what it covers varies from one supplier to the next. Palantir’s paper advises its customers to go further. It says supplier agreements “often incorporate online terms that are subject to unilateral change”, and it advises customers to use the master agreement to stop that, and to require notice and written consent before any new model or service is switched on. Both point the same way, towards a protection the supplier cannot change by itself. I made a related point in June, when a directive removed two frontier models from every customer overnight: what an organisation depends on but does not control needs a term, a fallback, or both.

Who owns this risk

This is a question worth asking because the risk does not look like anything already on the register. A data breach belongs to the chief information security officer, supplier contracts belong to the legal and procurement teams, and the terms that protect expertise when people leave belong to human resources. Know-how leaving through a tool is none of these: no system has been breached, no contract has been broken, and no employee has resigned, so the risk can sit adjacent to each function’s existing remit without falling squarely inside any of them. The test for a Board is simple: ask who owns this risk, and see whether one name comes back. A risk that falls between three desks reaches the Board only if the Board asks for it, which is why I think the question has to start there.

What I would ask

A Board does not need to commission anything to get started, because the test is three questions it can ask at its next meeting. The first is which of our work is what makes us better than our competitors. Much of what any organisation does is ordinary, like answering customer queries or writing up an expenses policy, and there is little to lose in sending that to an AI model. The way our traders read a market, our engineers design a system, or our scientists approach a problem is different, because that is what competitors would pay to know. The second question is what do our contracts say about that work, and can the supplier change those terms without asking us. The third is do we have a simple rule for what goes where: which work can go to a frontier model, which should stay on a model we control, and which should not go into an AI tool at all. Under the Six Board Concerns this belongs to Safeguarding Innovation, and I would expect the answers to come back to the Board with a named owner.

The Boards I meet that can answer those questions end up using AI more, not less, because their people know where the line is. Without a rule, organisations tend to end up in one of two places. Some ban the tools outright, which can push people to use them unofficially, while others leave the boundary largely undefined. With a rule, teams can get on with their work, because they know which of it can go to a frontier model and which cannot. Nvidia is the example in that report, because it has not stopped using frontier models at all: it is using them for its less sensitive work and keeping the rest at home.

Where this leaves a Board

The suppliers have sound reasons of their own for keeping some record of how their models are used, not least to detect misuse, and nothing here is an argument against them. What I am arguing for is a Board that knows its own position, and that is easier than it sounds, because an organisation usually knows what makes it better than its competitors, even if it has never had to write it down. The work is in writing it down, and in deciding what it means for the tools people use.

The reason to do that now is that the work is already going in. People are using these tools every day, for ordinary tasks and for the ones that matter most, and if nobody has decided where the line sits, it is being drawn by whoever happens to be at the keyboard. Once a Board has decided what can go in, the question that remains is what protects that work once it is there, and at the moment the only answer I can give is whatever the contract says, for as long as the supplier cannot change it. A term the supplier can withdraw is a courtesy, not a control.

The Questions Considered