Closed Source AI Models are not Trojan Horses

Satya Nadella is right that enterprises should protect their learning loops. But the strongest risk is dependency, not covert training on protected customer prompts

Please see relevant disclosures here.

---

Satya Nadella recently warned that companies buying artificial intelligence may be paying twice: first with money, and then with the proprietary knowledge they must reveal to make the AI useful. In the most lurid retelling, OpenAI and Anthropic are “Trojan horses”: enterprises invite them inside, pour confidential information into their models, and unwittingly teach the labs how their businesses work.

There is an immediate problem with that account. OpenAI and Anthropic tell business customers that they do not train their models on ordinary enterprise inputs and outputs. Google Cloud and Microsoft Azure make comparable commitments. Amazon Bedrock generally promises the same protection against training, although, as we will see, whether content reaches a model provider now depends on the particular model and configuration.

But “not used for training” is not the same as “never leaves the company.” A direct model provider still processes the prompt. Some services retain content for operations or abuse detection. Providers may learn from telemetry, aggregate usage and voluntarily submitted feedback. And even if none of that changes a general model, the customer can become dependent on a vendor that mediates its most important AI workflows.

Those are four different issues: processing, retention, provider learning and model training. Nadella’s essay tends to run them together. The published enterprise terms strongly rebut the claim that protected customer conversations routinely flow into general model training. They do not eliminate the broader problems of disclosure, operational access or dependency.

That makes the Trojan horse the wrong metaphor. The more defensive concern is not secret exfiltration. It is that convenience hardens into an architecture in which the enterprise owns its documents but not the portable machinery through which those documents become intelligence.

What Nadella actually said

Nadella’s essay, The Reverse Information Paradox, begins with Kenneth Arrow’s information paradox: a buyer cannot know what information is worth until the seller reveals it, but then the buyer already possesses it.

AI reverses the direction of exposure. The buyer purchases access to a model but must supply proprietary context to receive a useful answer. The more tailored the answer, the more the buyer must disclose. As employees use the system, they also create what Nadella calls “exhaust”: prompts, tool calls, corrections, evaluations, workflow traces and decisions. A correction by an experienced employee may encode the sort of institutional knowledge that never appears in a company handbook.

Nadella’s language is stronger than a simple warning about lock-in. He says his knowledge “leaks almost imperceptibly” and argues that providers reserve the right to learn from customer usage and interaction data. If that exhaust flows back to the provider, he says, the asymmetry compounds: the seller learns more about the customer, while the customer learns little about how the seller’s system is changing. Providers also commonly restrict customers from using model outputs to train competing models. The combination strikes him as perverse: the lab claims broad rights to learn from the world, then limits the enterprise’s ability to learn from the intelligence it purchased.

His remedy is a hard enterprise “trust boundary.” A company should control its evaluations, memory, traces, feedback, adapted weights and institutional context. It should be able to tune models inside its own tenant, route work among providers and retain its accumulated learning when the underlying “generalist” model changes.

Nadella does not call proprietary models Trojan horses, and he does not tell every enterprise to install an open-source model in its own data center. “Trojan horse” was TechCrunch’s framing; open source was the publication’s inferred subtext. But the headline was not wholly invented: Nadella really does use the language of leakage and one-way learning.

The fair question is therefore not whether he raised a risk. He did. It is what kind of “learning” the contracts actually permit, and whether his proposed tenant boundary eliminates dependency or merely moves it to the cloud platform that supplies the boundary.

What the contracts actually say

The published standard enterprise terms do not support the simple claim that proprietary labs routinely train general models on protected business conversations. Individual order forms can differ, and large customers negotiate, but the defaults are clear.

Under the current OpenAI Services Agreement, the customer retains its rights in inputs, owns outputs, and permits OpenAI to use customer content only to provide the service, comply with law, enforce policies and prevent abuse. OpenAI says it will not use that content to develop or improve its services unless the customer explicitly agrees. Voluntarily submitted feedback is different: the agreement gives OpenAI broad rights to use it.

Anthropic’s terms are more direct: “Anthropic may not train models on Customer Content from Services.” Its privacy documentation says commercial inputs and outputs are not used for model training by default. Explicitly submitted feedback is an exception: a thumbs-up or thumbs-down report can include the associated conversation and may be used for research or training. Enterprise administrators can disable that mechanism.

Google Cloud’s terms say Google will not train or fine-tune AI models on customer data without prior permission. Microsoft says prompts sent to models sold through Azure are not used to train or improve the base models. The models are stateless in the relevant technical sense: prompts do not update the model while it is answering, although stateful features and abuse monitoring systems may separately store data.

Amazon Bedrock requires more careful wording than it once did. AWS says Bedrock inputs and outputs are not used to train Amazon or third-party models. For many models and default configurations, content is not shared with the model provider. But AWS now allows provider-specific retention modes, and certain Anthropic models require customers to opt into sharing prompts and completions with Anthropic for trust and safety. The AWS Service Terms also allow inputs and outputs for certain models to be retained for up to 30 days for abuse detection. Therefore, for Bedrock, the model, feature and retention configuration matter.

The key technical question is what happens after inference. For an employee’s correction to change a general model, the interaction must be logged, retained, selected for a training pipeline and incorporated into a later update. Standard enterprise contracts generally prohibit the final step without consent. A provider can still learn in broader ways, from aggregate usage patterns, safety analysis, customer requests or submitted feedback, and so on, but that is different from training a general model on protected customer content.

If a provider used protected enterprise conversations to improve a general model without permission, that would appear to violate the published standard terms, not exploit their ordinary business model. Contracts establish legal obligations rather than proving flawless implementation, and their remedies may not compensate for the practical damage of a breach. But it matters whether the alleged danger is contractually authorized or contractually forbidden.

There are still real security and governance risks. Abuse monitoring may expose flagged content to review; connections introduce third parties; systems can be misconfigured or breached; and employees using personal accounts may fall under consumer rather than enterprise terms. Shadow AI remains a straightforward route by which confidential information can escape.

Those risks justify controls. They do not establish that the ordinary enterprise product is designed to absorb a customer’s confidential knowledge into a general model. Nor does model openness settle the privacy question. A closed model can sit behind a strong infrastructure boundary. An open weight model can be run with poor access controls, insecure dependencies or external telemetry. “Open” does not mean private, and “closed” does not mean the model developer necessarily receives the traffic.

Nadella’s stronger argument

The contracts nevertheless support an important part of Nadella’s case. OpenAI, Anthropic, and Google restrict customers from using outputs to reverse engineer their services or develop competing models. The exact language and exceptions differ. OpenAI, for example, permits some classifiers and customization of OpenAI models, while Google permits distillation where its service expressly provides the feature. But owning an answer is plainly not the same as owning an unrestricted learning capability.

An enterprise may legally own the output generated for a task while lacking the right, or the technical ability, to use thousands of such outputs to train a replacement model. It may retain its documents while becoming dependent on one provider’s behavior, pricing, safety policies, interfaces and product roadmap.

The defensible version of Nadella’s warning is therefore not “OpenAI will steal your secrets.” It is: “Do not let any model vendor become the only system through which your organization evaluates, corrects and improves machine intelligence.”

For strategically important and durable workloads, companies can store prompts, outputs and evaluations on their side of the boundary; separate retrieval and memory from the model; maintain private benchmarks; negotiate output and fine-tuning rights; and preserve the ability to route work elsewhere. Open weight models strengthen that strategy where their performance is adequate because they provide an additional exit option, even when they are not the best model for every task.

The decisive asset may therefore be neither an open nor a closed model, but the enterprise controlled harness around it: context, memory, tools, evaluations, permissions, and routing logic. Owning that layer can reduce switching costs. It does not make models interchangeable. Models differ in tool use, prompting behavior, context handling, safety policies, latency and output structure, so moving a workflow can still require substantial engineering.

Control of the harness does not abolish lock-in, either. It can move lock-in from the model provider to Microsoft Azure, Amazon Web Services, a data platform or a proprietary model gateway. Nadella’s tenant boundary is a sovereignty argument that also happens to be a good cloud platform sales pitch. The practical goal is not perfect portability but lower switching costs and credible alternatives.

This need not mean operating model servers. Downloading weights is easy; maintaining a secure, evaluated and reliable production service is not. Only a minority of enterprises have the staff, infrastructure, and governance to run an open weight stack independently. Hyperscalers, inference providers and application vendors can operate such models on their behalf. Open weights can therefore constrain proprietary labs even when most enterprises never self-host anything.

And not every workload deserves maximum sovereignty. For commodity drafting or low value internal assistance, accepting SaaS dependency may be economically rational. The case for enterprise control is strongest where the workflow is proprietary, high value, regulated or likely to accumulate durable institutional knowledge.

A threat, but not the one advertised

Open weight models are having a real moment, but current enterprise spending does not suggest the imminent displacement of proprietary providers. Menlo Ventures’ 2025 enterprise AI report estimated open weight models at 11 percent of enterprise model usage, down from 19 percent a year earlier. It also estimated that OpenAI, Anthropic and Google together represented 88 percent of enterprise LLM API usage.

Those figures are suggestive, not a census. Menlo surveyed roughly 500 U.S. enterprise decision makers already using AI, then combined their responses with a market model. Its shares approximate spending from reported production API usage, not a direct measurement of every token processed worldwide. Private deployments may be undercounted. Menlo’s separate finding that 76 percent of enterprise AI use cases were purchased rather than built internally concerns applications, not model hosting, and should not be treated as another measure of open model share.

Adoption can also look broad when the question is whether an organization uses any open model, yet narrow when the question is what share of valuable production work those models perform. A company can test Llama, Qwen, or Mistral, and even run one useful production application, while sending most economically important work to Claude, GPT, or Gemini. “Open weight” is the more accurate description here: accessible weights do not necessarily come with a license that meets conventional definitions of open source software.

The probable future is multimodel. Proprietary frontier systems will handle many difficult, fast-changing and high-value tasks. Open weight models will be attractive for some stable, narrow, high volume, latency-sensitive or highly confidential workloads. Enterprises will seek greater control of the gateway, context, memory and evaluation layer above both, even when outside vendors operate most of the underlying compute.

Where performance is good enough, open weights can establish a lower cost performance floor, improve buyer leverage and give customers a credible exit. They may compress proprietary inference margins and force closed providers to offer better privacy, smaller models, private deployments and more flexible customization. None of that requires open models to win most enterprise revenue.

The terminally online version of the debate asks whether open source will defeat closed source. The more consequential question is which parts of the AI stack an enterprise should control, for which workloads, to preserve its institutional knowledge and freedom of action.

On that narrower question, Nadella is largely right. Enterprises should control enough of the learning loop around strategically important work to preserve a credible exit. But the published terms and available evidence do not justify treating ordinary enterprise access to OpenAI or Anthropic as a Trojan horse designed to absorb every confidential prompt. The stronger danger is that a company allows convenience to harden into dependency, and discovers too late that it owns its data but not a portable way to turn that data into intelligence.

---

If you enjoy this newsletter, consider sharing it with a colleague.

I’m always happy to receive comments, questions, and pushback. If you want to connect with me directly, you can:

• follow me on Twitter,

• connect with me on LinkedIn, or

• send an email to dave [at] davefriedman dot co. (Not .com!)