A Rule Older Than Your Startup
Every company with an AI roadmap eventually holds the same meeting. Someone points out that the company is sitting on years of user data, and that a model trained on it would be better than anything the competition can build. Everyone agrees. Then someone asks whether the privacy policy allows it, and the room goes quiet.
The governing law is Section 5 of the FTC Act, which bans unfair or deceptive practices and doesn't mention models, algorithms or training data anywhere.
The FTC first applied Section 5 to a retroactive privacy policy change in a 2004 case, where it called the change unfair as well as deceptive. Gateway Learning, the company behind Hooked on Phonics, had rewritten its policy so it could share customer data, and applied the new terms to data it had already collected. The case resulted in a simple new principle: the promise you made at the time of data collection continues to apply to that data. If you change the promise and don't get consent, that can be unfair. If you told people you'd notify them of changes and you didn't, that can be deceptive.
The Data-Hungry AI Era
In February 2024 the FTC put that old rule in AI terms. A post on its technology blog said what everyone in the industry already knew: companies building AI have a continuous appetite for more data, and some will be tempted to fix the problem by quietly rewriting their privacy policies. The post warned that adopting more permissive data practices, like using data for AI training, and telling people only through a surreptitious, retroactive change to the terms may be unfair or deceptive.
Read the title closely: "AI (and other) Companies." The post didn't announce a new rule for AI. It reminded AI companies that the old one already covered them. A month earlier, the agency had made a similar point to companies that sell models as a service, telling them to keep the privacy and confidentiality promises they make to their customers.
I find that more useful right now than any new AI statute. There's no threshold to argue about and no definition of "AI system" to slip out of. If you made a promise about how you'd use someone's data, you should assess whether training a model breaks your word.
The Remedy Can Be the Model
Where a fine is imposed, a company might be able to absorb it as a cost of doing business. What gets a founder's attention is the remedy the FTC calls deleting “Affected Work Product,” better known in the marketplace as algorithmic disgorgement: deleting the models built with data the company shouldn't have used.
The best-known example is the FTC’s 2021 case against Everalbum, a company that ran a photo storage app called Ever. The FTC alleged it used its users' photos to build facial recognition technology, including technology it sold to enterprise customers. The settlement required Everalbum to delete the models and algorithms it had developed using those photos and videos.
That wasn't a one-off. The FTC put the same requirement into its settlements with WW International and its Kurbo app in 2022, which also carried a $1.5 million civil penalty. It did the same with Edmodo in 2023, and with Rite Aid at the end of that year. In the Kurbo case, the order reached any models or algorithms built with children's data, even though the complaint didn’t identify any models the company had built. For a company whose main asset is its model, that kind of order could reach the core of the business.
Though it’s unclear whether the current FTC is seeking model deletion against AI companies as readily as its predecessor did, companies should still plan for it. State attorneys general enforce their own unfair-practices laws. Some state privacy laws go further. For example, Colorado's privacy rules require companies to tell consumers about material changes to a privacy notice through the channel they normally use to reach them. If the change amounts to a new secondary use, the company needs consent before using data it collected under the old notice.
Four Checks for Your Data Promises
Risks often manifest in the gaps between multiple documents.
The first is your consumer privacy policy. Many say something like "we use your information to improve our services." Does that cover training a model? Maybe, for some kinds of training. However, I wouldn't bet a model on a sentence written before anyone at the company had an AI roadmap.
The second is your enterprise paper. Business customers often negotiate "no training on customer data" clauses, or sign data processing agreements that let you use their data only on their instructions. Violations of those contract commitments are often discovered during renewal security reviews. If you signed as a processor or service provider, training a general model on customer data can also push you outside that role.
The third is your model vendor's terms. Your privacy policy might promise users their data won't train anyone else's model. You should verify the tier and settings you signed up for with your provider to ensure that your model vendor is not permitted to use your users’ data for training. Your promise is only as good as their default.
The fourth is the "check back for updates" clause. Plenty of policies tell users to check the page for changes. For a material change to how you use data previously collected, that clause doesn't do much to protect you. Be sure to notify users before changing your data practices.
Action Items Ahead of Your Next AI Launch
Begin by pulling every version of your privacy policy and terms going back to when you started collecting the data you want to train on. Each batch of data carries the promises that were in effect when it was collected. It’s important to map out which policy versions covered which datasets.
Next, define "training" for your company. Fine-tuning a model, building a retrieval index, running evaluations and having people review outputs are different uses. Some may fit your existing language and some won't. Write that definition down in plain language so you can use it in your privacy policy.
For new uses of old data, give real notice by email or in the product, and get consent where the change is material. This must be done before you implement your new data practices. If consent isn't practical, train only on data collected after the updated policy took effect.
List every enterprise contract with a no-training clause or an instructions-only data agreement. Then make sure the product can actually honor those customer by customer.
Check your model vendor's default, including the tier, the settings and how long it keeps your inputs. Then make the policy match the truth.
Finally, assign one person to own this work and keep your privacy policy in step with your AI plans. Once you have an AI roadmap, the machine learning team's plans are your privacy policy's change log.
Though the unglamorous work of aligning promises and practices might take a few weeks, these steps are the cheapest insurance an AI company can buy before its first training run.
Sources
- AI (and other) Companies: Quietly Changing Your Terms of Service Could Be Unfair or Deceptive, FTC Technology Blog, February 13, 2024 — the warning about retroactive changes for AI training
- Everalbum settlement announcement, Federal Trade Commission — the order requiring deletion of models built from users' photos and videos
- Algorithmic disgorgement and the FTC's AI enforcement, Mintz, January 2024 — the Everalbum, WW and Kurbo, Edmodo and Rite Aid orders side by side
- The FTC treats using customer data for AI training under a policy that didn't permit it as an unfair or deceptive practice under Section 5 of the FTC Act.
- Companies cannot retroactively apply new, more permissive privacy policies to data collected under older, stricter terms without user consent.
- Algorithmic disgorgement—requiring companies to delete models trained on improperly used data—is an established remedy that can destroy a company's core asset.
- Each dataset carries the legal obligations of whichever privacy policy was in effect when that data was collected, requiring version-by-version compliance mapping.
- Enterprise contracts with no-training clauses and model vendor terms can create additional restrictions beyond your own privacy policy.
- Material changes to data use practices require direct user notice and consent before implementation, not just a passive policy update.
| Core legal principle | The privacy promise in effect when you collected data continues to govern that data even after policy changes. |
|---|---|
| Regulatory basis | FTC Section 5 bans unfair and deceptive practices; quietly using old data for AI training can violate both standards. |
| Algorithmic disgorgement | Regulators can order deletion of models trained on improperly used data, potentially destroying your core asset. |
| Enterprise contracts | Business customers often negotiate no-training clauses or processor agreements that prohibit using their data for general model development. |
| Model vendor settings | Your privacy promises to users can be broken by your AI vendor's defaults if they train on your inputs without your knowledge. |
| Compliance steps | Map every policy version to the data collected under it, define what 'training' means for your company, and get consent before material changes. |
| State law overlay | State attorneys general enforce similar laws, and some states like Colorado require affirmative consent for secondary uses of previously collected data. |
| Practical risk mitigation | Audit all policy versions, enterprise contracts, and vendor terms before any training run, then assign ongoing ownership of policy-AI alignment. |
Can I train an AI model on user data I collected years ago under an old privacy policy?
Only if the privacy policy in effect when you collected that data permitted the specific use of training AI models. Generic language about improving services written before your AI roadmap likely doesn't cover model training, and using the data anyway could violate FTC Act Section 5.
What is algorithmic disgorgement and when does the FTC require it?
Algorithmic disgorgement is the FTC's remedy requiring companies to delete models and algorithms built using data they shouldn't have used. The agency has ordered this in multiple cases since 2021, including Everalbum, WW International, Edmodo, and Rite Aid, making it a standard enforcement tool.
Is updating my privacy policy enough to use existing data for new AI training purposes?
No. For material changes to how you use previously collected data, you must provide direct notice to users and obtain consent before implementing the new practices. Simply posting an updated policy and telling users to check for changes doesn't satisfy FTC requirements.
How do enterprise customer contracts affect my ability to train AI models?
Business customers often negotiate no-training clauses or data processing agreements limiting use to their specific instructions. Training a general model on their data can violate these contracts and push you outside your designated role as a processor or service provider.
What should I check with my AI model vendor before promising users their data won't train other models?
Verify your vendor's specific tier, settings, and data retention policies to confirm they're not permitted to use your inputs for training. Your privacy promises to users are only as reliable as your vendor's actual configuration and defaults.
What practical steps should I take before training a model on existing user data?
Pull every historical version of your privacy policy, map which versions covered which datasets, define what training means for your company, notify users and get consent for new uses of old data, audit enterprise contracts, verify vendor settings, and assign one person to keep privacy policies aligned with AI plans.
- Federated Learning: Great Idea and Expensive RealityMove the model, not the data. Great for privacy, expensive for infrastructure.
- Synthetic Data: The Most Exciting Privacy-Enhancing Technology, and the Most OversoldSynthetic doesn't mean private. Generative models can leak real records—here's what testing catches.
- The Risks of AI in Law: What They Are and How to Manage ThemDiscover the risks of AI in law and learn how to protect client data, avoid common pitfalls, and use AI responsibly in legal work.