The opt-out is the product

The Electronic Privacy Information Center spent some weeks this spring filling in forms. Thirty-eight companies, the usual data brokers and now also OpenAI, Google and Meta, each asked politely whether the user might be removed from whatever it is they do with personal data. The resulting report, published 25 May, is being read as an exposé: AI vendors caught using the same dark patterns as the broker industry, fake forms, buried links, account-creation walls in front of the deletion button.¹ The Indian Express ran it under the headline that AI firms are using "deceptive opt-out tactics" to confuse users.² 9to5Mac went further, "even resort to fake forms to keep selling our data".³

I think the framing is wrong, and I want to show why, because the wrong framing leads to the wrong remedy, and the wrong remedy is what we will get if nobody says so out loud.

The finding, stated precisely. EPIC audited opt-out flows at 38 companies. It catalogued, against a known taxonomy of manipulative design, the friction users encounter when they try to exercise statutory rights to deletion, sale opt-out, or training exclusion. It found that OpenAI's "remove personal information from ChatGPT responses" mechanism is an output filter rather than a deletion of the underlying training corpus or the model weights derived from it.¹ It found that several providers require account creation before a deletion request can be submitted, which is the privacy-policy equivalent of charging admission to the complaints desk. It found, at some firms in the cohort, forms that accept input and do nothing with it. The methodology is design-pattern analysis, not a technical audit of training pipelines, and EPIC is transparent about this.

These are real findings. They are also, on inspection, two quite different findings stapled together, and the stapling is doing most of the rhetorical work.

The fake-form problem is straightforward. A form that accepts a deletion request and silently discards it is deception under any reading of FTC Section 5, and arguably fraud under several state consumer-protection statutes. There is no defensible technical or commercial reason for it. The companies operating these forms know what they are doing. If EPIC has documented specific instances at named companies, and the report indicates it has, the appropriate response is enforcement: a state AG complaint, an FTC investigation, a consent decree. The Colorado AG and the California Privacy Protection Agency have the standing and the budget. This is the part of the report that should produce subpoenas.

The OpenAI output-filter finding is a different animal entirely, and conflating it with the fake-form finding is where the analysis goes off the rails.

What a model actually is

A trained language model is not a database of records. It is a set of weights, billions of floating-point numbers, that encode statistical regularities extracted from a training corpus. Once a piece of text has contributed to the gradient updates that shape those weights, the relationship between the original text and the resulting parameters is not addressable. You cannot point at a weight and say, this one came from Jane Smith's blog post. You cannot remove Jane Smith's contribution without retraining, and retraining a frontier model costs something in the range of tens to hundreds of millions of dollars per run.

This is not a thing OpenAI is hiding behind weasel words. It is the architecture. The research literature on "machine unlearning", the field that exists precisely because this problem is unsolved, is a decade old and has produced approximations, not solutions. Influence functions, gradient-based unlearning, sharded retraining, all of these are active research with known limitations at scale. Nobody in the field claims production-grade per-record deletion from a frontier model is currently feasible.

So when EPIC reports that OpenAI's opt-out is "only an output filter", the report is technically correct and rhetorically loaded. An output filter is what is currently possible. The implication that OpenAI is choosing filtering over deletion, the way a data broker might choose to keep selling a record after a user requests removal, presumes a capability that does not exist. It is the difference between a shop refusing to issue a refund and a shop being unable to un-bake the cake the customer's flour went into.

The shop did not invent this problem. It inherited it from the regulatory instrument, which was written for a world of records and is now being applied to a world of weights.

The regulatory instrument is the problem. Notice-and-choice, tell users what you collect, let them opt out, was the consensus US privacy framework constructed during the browser-cookie decade. It assumed data was a record, that records were addressable, that retrieval and deletion were operationally trivial, and that the friction in the consent flow was therefore the meaningful policy lever. Tighten the friction, the theory went, and you tighten the privacy posture.

This worked badly even on its own terms. The data-broker industry spent fifteen years engineering compliance theatre around it, which is the entire reason EPIC has a taxonomy of opt-out dark patterns to apply. But it worked badly within a tractable problem. Training data is a different problem. Once a corpus is ingested, the relationship between the input record and the output model is many-to-many and lossy. The notice-and-choice instrument has no purchase here. You can regulate the front of the pipeline, what gets ingested, or the back — what the model is allowed to output. You cannot regulate the middle by asking the user to fill in a form.

EPIC, I suspect, knows this perfectly well. The report is structured to be cited in FTC comment letters and state AG complaints; the specific naming of OpenAI, Google and Meta is the move that gives it teeth. But the public framing, AI firms using deceptive tactics like data brokers, collapses two findings into one rhetorical posture, and the conflation will produce an enforcement push that hits the wrong target.

The wrong target

Consider what enforcement under the current frame would actually look like. The FTC, or a state AG, brings a Section 5 unfair-and-deceptive-practices action against OpenAI for misrepresenting the scope of its opt-out. The consent decree requires OpenAI to disclose, more prominently, that the output filter does not delete underlying training data or model weights. OpenAI updates the form. Adds a paragraph. Changes the button label from "Delete my data" to "Filter my data from responses". A privacy-policy redline ships. The substantive position — that the model contains statistical traces of a corpus that included the user's content, and there is no production mechanism to remove those traces — is unchanged.

This is the data-broker playbook in reverse. The brokers got rules that required them to honour opt-outs and built ten years of friction to make opt-outs unfindable. AI vendors get rules that require them to honour opt-outs and disclose, with great clarity, that the opt-out cannot do what the user thinks it does. Same regulatory instrument, same UX layer, no movement on the underlying asset.

The strongest counter-case is that disclosure is the point. If consumers genuinely understand that their data, once ingested into a training corpus, cannot be retrieved, they can make an informed choice about which models to use, which platforms to put their writing on, which terms of service to accept. Transparency is a precondition for any market correction, and notice-and-choice at least clears the informational fog. There is something to this. I will concede that the right disclosure regime is better than the current one, and that EPIC's audit, by making the gap between user expectation and technical reality legible, is doing useful work.

But disclosure was the data-broker remedy for fifteen years, and at the end of it we have the broker industry EPIC documents in the same report. The lesson of those fifteen years is that disclosure-plus-friction is the steady-state equilibrium of notice-and-choice regulation, not its failure mode. Companies disclose, in language calibrated to the median user's tolerance for reading, and then build the friction back in. AI vendors are starting earlier in the cycle and have better lawyers.

Disclosure was the data-broker remedy for fifteen years, and at the end of it we have the data-broker industry.

What would actually work

There are two coherent regulatory instruments for training-data governance, and neither is a deletion form.

The first is statutory data minimisation at ingestion. GDPR Article 5(1)(c) does a version of this for processing generally; no US statute does it for AI training at any scale. A minimisation rule would specify categories of data — children's data, sensitive personal data, copyrighted material absent licence, data scraped in violation of platform terms — that cannot lawfully enter a training corpus in the first place. The enforcement target is the ingestion pipeline, which is auditable and addressable, rather than the model weights, which are not. The downstream effect is that the universe of models legally trainable in the United States narrows, frontier labs build provenance tooling because they have to, and the question of post-hoc deletion becomes mostly moot because the offending data was never ingested.

The second is output-side liability with teeth. If a model produces, on demand, an individual's personal information, the operator is liable for that output regardless of whether the underlying training data can be deleted. This shifts the engineering problem from "unlearn the corpus" to "constrain the inference", which is a tractable problem. Output filtering, the thing EPIC dismisses as inadequate, is actually the right unit of intervention if it is paired with a liability rule that bites when the filter fails. The current regime, in which the filter is voluntary and the liability is diffuse, gets the worst of both.

Neither of these instruments is in the bills currently moving through state legislatures. The Colorado AI Act, which takes effect on 30 June, focuses on high-risk algorithmic decision-making in credit, housing and employment, with transparency and impact-assessment obligations. It is a useful statute but it does not touch training-data governance. California's Delete Act establishes a universal deletion mechanism for data brokers and leaves open the question of whether AI training pipelines qualify as broker operations under the statutory definition. The honest answer is that they do not, structurally, and pretending otherwise will produce litigation rather than deletion.

So the regulatory pipeline is loaded with instruments that address the wrong layer, and the EPIC study, valuable as it is on the fake-form findings, is providing rhetorical fuel for more of the same. The fake forms get fixed. The training-data question does not get asked. Five years from now the audit is repeated and the dark-pattern taxonomy has expanded to include the new disclosures that companies will have added under the consent decrees they signed in 2027.

What I would not do

I would not let OpenAI, Google or Meta off the hook for the operational findings. If a form is fake, that is a fraud problem and should be prosecuted as one. If account creation is required to exercise a statutory right, that is a clear violation of the spirit if not the letter of every state privacy statute that purports to grant the right, and the enforcement is straightforward. The companies named in the report have legal departments capable of fixing these things in a quarter, and the only reason they have not is that the cost-benefit calculation has so far run in their favour. EPIC's contribution is to change that calculation by naming names, and on this it has done its job.

What I would not do is accept the framing that follows. The framing — AI vendors are operating the data-broker playbook, the remedy is more notice-and-choice enforcement — produces a regulatory outcome that polishes the consent UX and leaves the training corpus untouched. The companies are mostly fine with that outcome. They will pay the fines, ship the redlines, and continue.

The harder conversation — that training data, once incorporated, is not addressable; that the only honest remedy is to regulate what gets ingested or what gets emitted, not what gets disclosed — is the conversation EPIC's report could have started and chose not to. I understand why. Brief-writing is a discipline of using the instruments you have. But somebody has to say that the instruments are wrong, and I would rather it be said in a long essay than left for the next audit cycle to discover.

The opt-out is the product. The product is working as designed. That is the finding under the finding, and it is the one worth fixing.

Footnotes

¶