Is Your AI Training the Competition?

For years, corporate lawyers have obsessed over one question when it comes to AI: Is our data going into someone else’s training set? Most companies feel they’ve solved that. They’ve signed contracts saying the AI provider can’t train on the prompts their employees type in. But that’s no longer the only important question. The real new risk is that companies are losing something more valuable than data: their logic—the specific, hard-won way they solve problems. Call it “AI Sherlocking.” It’s not entirely new, but it has entered its most sophisticated form yet, and it’s happening even to companies with airtight data contracts. In the early 2000s, Apple built a search tool called Sherlock that copied a popular add-on app called Watson, feature for feature, and put the smaller company out of business almost overnight. Ever since, “getting Sherlocked” has meant watching a bigger platform absorb your best ideas and use them against you. The AI version works differently but ends the same way. This article draws on recent enterprise AI security incidents, the fine print in major AI providers’ contracts, and current data on “shadow AI” use inside companies to lay out three specific ways this happens—and what leaders can do about each one. How AI Sherlocking Works Traditional Sherlocking relied on copying a feature you could see: a button, a screen, a specific function. AI Sherlocking is about something you can’t see: the reasoning that produced a good answer. Here’s where most executives get the picture only half right. Standard enterprise AI contracts today typically protect both the input and the output—the prompt your engineer typed and the answer the tool handed back—from being used to train future versions of the model. That’s real progress, and it’s what most legal reviews are focused on. The problem is that the protection has real gaps, and those gaps are exactly where competitive advantage tends to leak. Consider what happens when a senior engineer uses an AI tool to debug a tricky piece of code, or a researcher uses it to work through a scientific problem. The answer that comes back isn’t just the AI being smart. It’s a blend of the tool’s general ability and that employee’s specific expertise, guiding it step by step toward the right answer. That blend is exactly what has the most value, and depending on how it moves through the system, it may not be as protected as the contract implies. If an AI provider learns from the shape of those interactions—through feedback signals, through generalized capability improvements, or through the fact that similar high-quality work is happening constantly across its user base—it can effectively absorb your team’s expertise without ever violating a specific line in your contract. The next company that asks a similar question won’t need your files. The model will already be sharper at your kind of problem, because it learned the shape of that problem from users like you. This isn’t hypothetical. In February 2026, Anthropic disclosed that three Chinese AI labs—DeepSeek, Moonshot AI, and MiniMax—had run more than 16 million queries against its Claude models through roughly 24,000 fraudulent accounts, in a systematic effort to train their own competing models on Claude’s answers. The technique, called “distillation,” works by using a stronger model’s outputs to teach a weaker one; Anthropic identified the campaigns by their unusual query patterns, heavily concentrated on Claude’s most distinctive capabilities like reasoning and coding. That was an adversarial case involving fraud and fake accounts, but it points at the same underlying dynamic in a milder, entirely legal form: The more a model is used to solve a particular kind of problem, the better it gets at that kind of problem. Every major provider has some version of that flywheel because their business depends on it. This is also why the standard corporate playbook for AI security is falling short. Past discussions have emphasized “sandboxing” and related controls—walling off AI tools in secure, isolated environments so sensitive data never touches the open internet. Those are genuinely useful steps, but they solve only one part of the problem. A sandbox stops your data from leaking out. It does nothing to stop the AI provider from learning your team’s problem-solving patterns every time they use the tool inside that sandbox. You can lock the data in a vault and still lose the thing that made the data valuable in the first place: the expertise applied to it. There’s also a simpler, more human version of the same risk: shadow AI. Despite paying for secure, contractually protected enterprise accounts, employees at most companies still occasionally use free, personal AI accounts out of habit, convenience, or to work around restrictions their employer has put on the enterprise version—pasting in a contract to summarize, or a piece of code to debug, as Samsung engineers did in 2023 when they inadvertently uploaded proprietary semiconductor source code to ChatGPT, prompting the company to ban the tool internally. The pattern is remarkably widespread. A 2025 MIT study of enterprise generative AI use found that while only about 40% of companies had purchased an official AI subscription, employees at more than 90% of the surveyed companies were regularly using personal AI tools for work. And IBM’s 2025 “Cost of a Data Breach Report” finds that breaches involving shadow AI cost, on average, roughly $670,000 more than incidents without it, while also disproportionately exposing intellectual property and customer data to attack. Every time an employee reaches for a personal AI account, whatever gets pasted in goes straight into that free tool’s training pipeline, with no enterprise agreement in the way. It’s the same loss of expertise, just through an unlocked side door instead of the system that was supposed to be protecting it. Put these threads together and a bigger shift comes into focus. For the last two decades, the advice for companies has been “data is the new oil”—collect it, protect it, and you have an advantage. That’s no longer enough on its own. Data is increasingly becoming a commodity; there’s a lot of it, and AI models are good at using whatever they’re given, even when it isn’t of the highest quality. What’s actually scarce, and actually valuable, is high-quality data and what a company does with its data—the specific judgment, workflow, and reasoning that turns raw information into a good decision. Call this “logic sovereignty”: ownership not just of your data, but of the thinking you apply to it. That’s exactly what AI Sherlocking goes after, and it happens through three specific channels, each worth understanding on its own because each calls for a different fix. The first is the feedback your own employees give an AI tool every day, often without thinking about it. The second is how the industry as a whole is increasingly using AI-generated content—including the outputs of high-value use—as raw material for training future models. The third is a structural version of the same problem: The more your best people use a platform, the smarter that platform gets for everyone, including your competitors. Let’s take each in turn. Takeaway One: Your Team’s Feedback Is Training Data The first channel is the most common one, and probably the easiest to miss. Every time someone on your team gives an AI tool feedback—a thumbs up or down, “that’s not quite right, try again,” or a manual correction to a draft—they’re doing more than getting a better answer. They’re providing exactly the kind of signal AI companies use to retrain and improve their models. In plain terms: your corrections teach the model. Here’s the part that should concern whoever owns your AI vendor contracts. The standard OpenAI Services Agreement protects your prompts and the model’s raw answers from being used in training, but it treats feedback as a separate category. The agreement states that if a customer provides feedback, OpenAI has the right to use and build on that feedback without restriction and without paying for it. Anthropic’s Commercial Terms of Service for Claude contain a nearly identical clause—a sign that this feedback carve-out is a standard industry contract structure rather than the quirk of a single provider. There’s some ambiguity in how broadly “feedback” gets interpreted—the term is typically defined as feedback about the service itself—but OpenAI’s own documentation is more explicit about what it means in practice: Even when a customer has opted out of training generally, giving a thumbs up or down on a response means the entire conversation associated with that feedback may be used to train future models. And it’s not only automated: The reinforcement learning process behind modern chatbots, first described at scale in OpenAI’s 2022 InstructGPT research, relies on contract labelers reading and ranking model outputs on real user prompts as a core training signal, and the same basic approach is now used across the industry. The people doing that grading are, in effect, learning from your team’s proprietary corrections too. What managers should do: Treat feedback mechanisms as a contract issue, not just a product feature. When negotiating or renewing AI vendor agreements, don’t stop at “no training on our data.” Ask specifically about feedback, ratings, corrections, and interaction logs, and push for language that closes this gap explicitly. In March 2026, Anthropic added the ability to manage user feedback settings on Team or Enterprise plans, so be sure to turn off the Rate Chats function. Note this is a new function and not a tried and proven guarantee. It’s worth briefing your teams too: The moment they correct an AI’s work, they may be teaching a system your competitors also use. Takeaway Two: Your Best Outputs May Become Someone Else’s Training Data The second channel is a matter of supply and demand. High-quality human-generated data—the kind AI companies need to keep improving their models—is running short. Researchers at Epoch AI, a nonprofit that tracks AI training resources, estimate that the stock of publicly available high-quality text data will be substantially exhausted somewhere between 2026 and 2032, depending on how efficiently it is reused. So providers have increasingly turned to a workaround: using their own models’ best outputs as training material for the next version. This is now a standard industry practice, not a rare edge case. Microsoft’s Phi-4 model, released in December 2024, was explicitly trained on roughly 400 billion tokens generated across 50 different synthetic datasets, and Microsoft’s technical report argues that this approach was central to the model’s performance. If your team is using well-contracted enterprise AI, your specific outputs are usually walled off from this pipeline. But there are two ways the risk still touches your company. The first is any use that falls outside that contract—consumer or free-tier accounts, shadow AI use, or older agreements without explicit output protection. Whatever is generated there is fair game for training the next release. The second is subtler: Even when your specific outputs aren’t touched, models trained on the pooled outputs of everyone else solving similar problems still get better at your kind of problem, and that improved capability is then sold to anyone who buys access. As a side note, researchers have warned that models trained too heavily on this kind of recycled AI-generated content can gradually lose accuracy over time, something they call “model collapse.” That’s a real technical headache for AI providers, but it doesn’t change the exposure on your end. What managers should do: For your highest-value AI use cases—the ones tied to genuine competitive advantage—push for deployment options that keep everything in-house: custom models operating in a private cloud environment or an on-premises setup where nothing generated is ever sent back to the provider for any purpose. Reserve the public, general-purpose version of these tools for lower-stakes work, where losing the “secret sauce” isn’t a real concern. Takeaway Three: The Smartest User Makes the Platform Smarter for Everyone The third channel is less about individual interactions and more about a structural pattern. AI platforms improve by learning from their most sophisticated users. If your company is the one pushing an AI tool hardest—using it to solve a genuinely difficult, high-value problem, like optimizing a complex supply chain—you’re also the one teaching that tool how to solve problems like yours, generally. The platform gets better. Then it sells that improved capability to the rest of the market, including your direct competitors. This creates an uncomfortable dynamic: The more innovative your use of an AI tool, the more you may be contributing to your own competitors’ future capabilities. It runs counter to how competitive advantage has traditionally worked. Normally, being the smartest user of a tool is what keeps you ahead of the pack, not what erodes your lead. What managers should do: For your most strategically sensitive work, avoid sending the whole problem to a general-purpose AI tool in one shot. Break the task apart, and send only the pieces that don’t reveal your overall approach, while keeping the part that stitches everything together—your actual method—in-house. A related option is to use a large public AI model to generate rough, general ideas, then do the real fine-tuning on a smaller model your company owns and controls outright, so the valuable final step never leaves the building. . . . None of this means companies should stop using AI. The productivity gains and changes in how work gets done are real and will only grow. But the current approach to protecting competitive advantage is built around the wrong question. Most security reviews still ask, “Is our data safe?” The more important question going forward is, “Who is learning from our success?” Over the next several years, expect this issue to move from a niche concern for AI security teams to a standard boardroom topic, the way data privacy did a decade ago. Companies that keep treating “no training on our prompts” as sufficient protection will find, gradually and then all at once, that their competitive edge has been quietly absorbed into the very tools they use to compete. The organizations that get ahead of this—by rewriting contracts, isolating their highest-value work, and training their people to recognize where the real exposure lies—will be the ones whose expertise stays theirs. There’s a broader stake here too, beyond any single company’s bottom line. If a handful of AI providers end up absorbing the best problem-solving logic from across every industry, competitive advantage could concentrate in fewer hands, and the incentive for any company to keep innovating out loud could shrink. That’s worth watching as a question about how competition works in an AI-driven economy, not just as a matter of individual corporate strategy. For now, the practical task for leaders is narrower and more urgent: know exactly what your AI tools are learning from you, and decide, deliberately, what you’re willing to teach them.

Is Your AI Training the Competition?

Summary.

Partner Center