Training an AI Chatbot on Customer Data: What Privacy Law Requires

Yes, you can train a chatbot on your own customer conversations, and most businesses already do this without thinking of it as "AI training." But most state privacy laws already treat those chat transcripts as personal information, and the FTC's real enforcement tool for misusing that data is not a fine. It is an order to delete the trained model itself, a remedy called algorithmic disgorgement, already used against a children's health app in 2022 and a national pharmacy chain in 2023. Suvysoft builds chatbots with that exposure designed out from the start, not patched in after a complaint.

Every AI chatbot vendor page covers uptime, response accuracy, and per-message pricing. Almost none of them mention what happens to the conversations your customers have with the bot, or what happens to your business if those conversations were used to improve the model without the right disclosures in place first. We cover the separate question of when a chatbot has to say out loud that it is a chatbot in AI chatbot disclosure law; this post is about what happens to the conversation afterward.

Is training a chatbot on customer conversations actually illegal?

Not by itself. Using your own data to improve your own product is generally legal. What creates exposure is the gap between what your privacy policy says the data is used for and what your chatbot vendor's contract actually allows, plus whether that data included information the person had a legal right to control, delete, or never have collected in the first place. A chat log with a name, an email address, an account number, or a health detail in it is personal information under nearly every state privacy law on the books, whether it came from a form field or a sentence someone typed into a chat window.

What is algorithmic disgorgement, and why is it worse than a fine?

Algorithmic disgorgement is the FTC's practice of ordering a company to delete not just improperly collected data, but any model, algorithm, or "affected work product" built from that data. A fine is a line item. Losing the model itself means the months of tuning, prompt engineering, and real customer conversations that made your chatbot good at its job are gone, and you start over with whatever generic version the vendor ships everyone else.

The FTC first used this remedy against Cambridge Analytica in 2019, ordering it to destroy any algorithms derived from data collected without consent. It is not a one-time novelty. The Commission has used it repeatedly since, and the businesses on the receiving end were not just tech giants.

Which real cases has the FTC actually ordered this in?

Three cases make the pattern concrete, and all three involved companies smaller than the household names most compliance articles reach for.

Everalbum, maker of the Ever photo app, settled with the FTC in January 2021 after quietly turning on facial recognition for users who never opted in. The order required deleting the photos, the face embeddings derived from them, and any facial recognition models or algorithms built using those embeddings. No civil penalty attached, since this was the FTC's Section 5 unfairness authority rather than a statute with a built-in fine, but the company lost the technology it had spent years building.

WW International (formerly Weight Watchers) and its Kurbo subsidiary paid $1.5 million in March 2022 for collecting children's health data through a weight-loss app without parental consent, a COPPA violation. The order required the companies to destroy the data itself and, separately, to "destroy any algorithms derived from the data." Losing the algorithm was a distinct, additional requirement on top of the money.

Rite Aid's December 2023 settlement had no announced civil penalty at all. Instead, Rite Aid was banned from using facial recognition for five years and ordered to delete all data, models, and algorithms derived from a system that had generated thousands of false-positive matches, disproportionately in stores in Black and Asian neighborhoods. The technology, not a check, was the cost.

Does California's privacy law already cover your chatbot's transcripts?

Yes, if you meet CCPA's revenue or data-volume thresholds. Chat transcripts, the inferences your chatbot draws from them, and any customer profile built from repeated conversations all count as personal information once they include an identifier: a name, an email, a phone number, or details tied to one. That means a consumer's right to know, right to delete, and right to opt out of a "sale" or "share" of their data all reach your chatbot's logs the same way they reach a spreadsheet of leads.

Whether feeding those logs into a fine-tuning run counts as a "sale" is a genuinely contested question among privacy attorneys, since CCPA defines sale broadly enough to cover any transfer for valuable consideration, even an internal one that never leaves your company. Nobody has a definitive answer yet. What is not contested is the disclosure obligation: if your privacy policy does not mention that chat data may train or improve an AI system, and it does, that is the kind of undisclosed practice that turns into a Section 5 deceptive-practices claim on its own, independent of whether a "sale" technically occurred.

What does a CCPA violation cost, separate from the FTC's disgorgement remedy?

Real money, just smaller than the disgorgement scenario. The California Privacy Protection Agency's own 2025 penalty update set administrative fines at up to $2,663 per violation, rising to $7,988 per intentional violation or any violation involving a consumer under 16. Those numbers apply per violation, not per company, and a single mishandled deletion request across a few hundred chat sessions adds up fast at either rate.

The CCPA gives a business 45 days to respond to a verified deletion request, extendable once by another 45 days with notice, for a maximum of 90 days. If your chatbot's transcripts are stored with a vendor who cannot locate and delete a specific customer's conversation history inside that window, that is a compliance gap regardless of whether anyone ever asks about AI training at all.

What should your privacy policy actually say before you fine-tune anything?

Name the practice specifically, in plain language, before you start. "We may use anonymized data to improve our services" is the generic line most privacy policies already have, and it is not enough once a chatbot is retaining full conversations that include names and account details. State the actual practice: whether transcripts are used to train or fine-tune a model, whether that model is shared with or hosted by a third-party vendor, how long transcripts are retained, and how a customer can ask to have theirs excluded or deleted. If your chatbot vendor's own terms allow them to use your customers' conversations to improve their product across other customers, that needs to be in your policy too, since it is your name on the disclosure a regulator or plaintiff's attorney will read first.

What should a small business actually do before training on chat logs?

Four things, in order. First, read your chatbot vendor's data terms specifically for the words "train," "improve," or "fine-tune," not just "confidential" or "secure," since a vendor can keep your data private from competitors while still using it to train the underlying model. Second, strip or mask direct identifiers, names, emails, account numbers, before any transcript goes into a training or fine-tuning run, so the underlying content is useful without being a personal-information liability. Third, update your privacy policy to name the specific practice before you start, not after a customer asks. Fourth, set an actual retention limit on raw transcripts instead of keeping every conversation indefinitely by default, since data you no longer have is data nobody can force you to disgorge.

If you want a chatbot built with those boundaries in place from day one rather than retrofitted after a complaint, our website chatbot work is scoped around exactly this: what gets logged, what gets retained, and what a customer can ask to have removed. Our broader AI setup and deployment work covers the same data-governance questions for any AI system connected to customer records, not just a chat widget. Get in touch if you want a second set of eyes on what your current chatbot setup actually does with the conversations it collects.

Frequently asked questions

Can I use my own customer chat logs to train my AI chatbot at all?

Generally yes, using your own data to improve your own product is not itself illegal. The risk is in the gap between what you disclosed and what actually happens to the data, and in whether the logs include information a state privacy law gives the customer a right to access, delete, or opt out of using. Disclosure and retention limits, not the training itself, are what create or avoid exposure.

What is algorithmic disgorgement?

It is an FTC remedy that orders a company to delete not only improperly obtained data, but any model or algorithm built from it. The FTC has used it against Cambridge Analytica, Everalbum, WW/Kurbo, and Rite Aid since 2019. It matters more than a standard fine because it can erase the trained system itself, not just charge a penalty on top of it.

Does CCPA's deletion right reach a model that was already trained on someone's data, or just the raw logs?

The law is clearest on the raw logs: a verified deletion request requires removing the personal information itself within 45 to 90 days. Whether a business must also retrain or discard a model that already learned from that data is a less settled question, which is exactly the gap the FTC's disgorgement remedy exists to close when a regulator, rather than an individual consumer, brings the case.

Do I need to disclose AI training in my privacy policy specifically?

Yes, if your chatbot's conversations are used to train, fine-tune, or improve any AI model, your own or a vendor's. A generic line about using data "to improve our services" written before you added a chatbot likely does not cover this specifically enough to hold up, and an undisclosed practice is the kind of gap that turns into a deceptive-practices claim on its own.

Does this only apply to large companies like Rite Aid, or does a small business face the same exposure?

The same rules apply regardless of size, though enforcement priority tends to follow scale and harm. WW/Kurbo was not a facial-recognition giant, it was a weight-loss app that mishandled children's data, and it still lost both the algorithm and $1.5 million. A five-person shop with a website chatbot is unlikely to draw federal attention over a single vague policy line, but state attorneys general and private plaintiffs' attorneys increasingly bring smaller cases, and the underlying legal exposure does not scale down with company size.

What is the lowest-risk way to improve a chatbot without this exposure?

Mask or strip direct identifiers before any transcript is used for training, set a real retention limit instead of keeping everything indefinitely, and put the specific practice in your privacy policy before you start rather than after a customer asks. None of that requires giving up on improving your chatbot with real conversation data. It requires doing it with the same data hygiene you would want if you had to explain the practice to a regulator on short notice.