OpenAI Sued Over 'Project Lily' Human Review of ChatGPT

A new federal lawsuit alleges that OpenAI routed real ChatGPT conversations, including chats about health conditions, debt, and legal trouble, to outside contractors who read, summarized, and scored them, without disclosing that a human being might be the one reading. The case, Vredenburgh v. OpenAI OpCo, LLC, was filed September 16, 2026 in the U.S. District Court for the Northern District of California and served on OpenAI on September 21. It centers on an internal program reporters have named 'Project Lily.'

This did not start as a lawsuit. 404 Media broke the story on September 14, 2026, after reviewing internal OpenAI materials, including reviewer instructions, Slack messages, a scoring rubric, and real user submissions. Contractors are recruited through a staffing firm called Crossing Hurdles, paid through the AI-training company Mercor, and reportedly earn more than fifty dollars an hour to read anonymized excerpts of live conversations, summarize what the user was trying to accomplish, and rate multiple chatbot responses on a scale of one to seven. Internal accounts described in the reporting suggest even people inside OpenAI didn't expect users to realize a contractor might be analyzing their conversations.

Two California ChatGPT users brought the suit on behalf of a proposed nationwide class, case number 3:26-cv-10527, assigned to Magistrate Judge Alex G. Tse. OpenAI's response is due October 13, 2026, so none of this has been decided by a court, and OpenAI has not been found liable for anything. This is a different dispute from the New York Times copyright litigation that forced OpenAI to preserve, and later produce, 20 million de-identified chat logs, a separate case with its own timeline. Project Lily has nothing to do with litigation-driven data retention. It is about whether OpenAI told users, up front, that people outside the company would read their actual conversations to grade the model's answers.

Keep two things separate here. That contractors exist, that they are paid to read real conversations, and that the practice happened is independently documented reporting, and OpenAI has not disputed it. Whether that specific practice was inadequately disclosed, and whether it violated California privacy and consumer-protection law, is a legal claim a court has not yet ruled on. A complaint states one side's case. OpenAI is entitled to respond, and courts routinely narrow or dismiss claims long before trial.

What made contractors nervous enough to talk was not the existence of review itself; it was what showed up in the queue. Reporting on the internal materials describes prompts touching therapy-style conversations, professional and legal questions, and in some cases a 'user memories summary' that surfaced details from earlier chats, including location information, none of which the person reviewing that single conversation was supposed to see in isolation. Some of the reviewed material came from users who had explicitly asked ChatGPT, in the conversation itself, to keep something private. Anonymization removed names and account identifiers. It did not remove the substance of what people had written.

What the lawsuit alleges, and what the reporting independently confirms

• According to the complaint, OpenAI's Privacy Policy lists eleven categories of outside companies that receive user data, and none of them is described as a data-labeling, annotation, or human-evaluation vendor.
• Reviewers read full conversations, not isolated fragments, and 404 Media's review of internal materials found that OpenAI's automated privacy filter, meant to strip identifying details before a human sees the text, does not catch everything.
• Reviewers work the queue under job titles like 'AI data reviewer' and 'chatbot evaluator,' picking tasks from a dashboard of real prompts, reading multiple ChatGPT responses per task, and writing a rationale to go with each one-to-seven score.
• The complaint asks a court to require opt-in consent before any conversation goes to an outside reviewer, default the 'Improve the model for everyone' setting to off, add an in-chat warning, and order deletion of reviewer work product tied to reviewed conversations.

Other AI chatbots use human review too, this lawsuit is about disclosure

OpenAI is not the only company that has people look at real chatbot conversations. Reinforcement learning from human feedback, the technique behind why modern chatbots sound conversational instead of robotic, generally needs human judgment somewhere in the loop, and 404 Media's reporting notes that Anthropic has confirmed it also uses human review to improve its models. The legal fight here isn't over whether any human should ever see a chat. It's over whether this specific pipeline, run through an outside staffing firm at this scale, was clearly disclosed before it started.

What the complaint is actually suing over, legally

A complaint has to hang its facts on a legal theory, and reporting on the filing describes eight stacked California claims, including alleged violations of the state's Unfair Competition Law and Consumer Privacy Act, plus common-law and state-constitutional invasion-of-privacy theories, on top of the core argument that OpenAI's disclosures fell short of what state consumer-protection law requires before a company routes someone's private writing to a third party. None of that has been tested in court yet. No class has been certified, no discovery has forced OpenAI to produce its own account of what the Privacy Policy was meant to cover, and a motion to dismiss, if OpenAI files one, could narrow the case considerably before it ever reaches a jury.

What people assume versus what's actually disclosed

OpenAI is not silent on this. Its help center carries an article on how conversation data is used to improve model performance, stating that a limited number of authorized personnel, and 'trusted service providers' bound by confidentiality obligations, may access content for reasons including model improvement, unless a user opts out. That page exists and predates the lawsuit. The dispute is narrower than 'OpenAI never said humans read chats.' It's about whether a help center article, describing access in the abstract, is the same as disclosing that a named contractor workforce, hired through a staffing agency and paid by the hour, would read full conversations as a matter of routine work. The complaint says no, and points to the Privacy Policy's own list of third-party recipients, which it says does not include that category.

Also on MemX

AI & Privacy

OpenAI's Agents Leaked 53 ChatGPT Images, and It Can't Say Whose

11 min read→

AI & Privacy

ChatGPT Ad Trackers Lawsuit: OpenAI Says You Already Agreed

11 min read→

AI & Privacy

ChatGPT's New Sponsored Agents: Ads That Chat Back

11 min read→

What 'private by architecture' actually means

Every AI company's privacy language says some version of 'we protect your data.' That sentence does no real work until you ask a narrower question: is this privacy a promise written into a policy that can be reinterpreted, updated, or simply not cover some new internal program, or is it a property of how the system is actually built? 'Private by architecture' is a claim of the second kind. It describes technical facts about a system, not intentions about how a company plans to behave.

• Encryption at rest: stored content is unreadable without the right keys, not just protected while moving between a device and a server.
• Customer-managed encryption keys (CMEK): the keys that decrypt data are managed per customer, rather than pooled in a way that lets one internal key holder access everyone's content at once.
• Per-user isolation: one account's data is walled off from another's at the storage and processing layer, not only by an access-control policy someone could misconfigure.
• On-device processing where applicable: some analysis happens locally, so raw content doesn't have to leave the device to become searchable.

None of that is end-to-end encryption, and none of it is a zero-knowledge system. Those are different, stronger claims, meaning not even the company running the service can read the content under any circumstance. 'Private by architecture' is a narrower, more honest claim: no undisclosed pipeline routes content to outside contractors as routine business, and the technical safeguards around storage and access are structural rather than promised in a policy document that can quietly fall out of date.

This distinction is exactly what the Project Lily lawsuit turns on. Nobody is arguing that OpenAI's servers were breached or that encryption failed. The allegation is about a business process built on top of properly stored data: real conversations, routed on purpose to people outside the company, for a use the complaint says was never clearly named. A policy sentence about protecting data says nothing about which internal or third-party workflows that data is allowed to flow through next. An architectural claim, encryption at rest, per-user isolation, who is structurally able to reach the content at all, is a narrower promise, and a narrower promise is one that's actually possible to verify and hold a company to.

This is the specific gap MemX (memx.app) is built to sit on the right side of. MemX stores what people actually want remembered: photos, documents, voice notes, WhatsApp messages, made searchable later. That's sensitive material by definition, closer to a health record or a bank statement than a throwaway chat message. MemX describes its privacy posture as private by architecture: encryption at rest, customer-managed encryption keys, and per-user isolation built into the storage layer, with on-device processing where the task allows it. What MemX is not building is a pipeline that routes personal memories to an outside contractor workforce, paid by the hour, to read and score them for model training. That's a structural choice about what the product is for, not a promise sitting in a policy document waiting to be tested by a lawsuit. It doesn't make MemX end-to-end encrypted or zero-knowledge, and MemX doesn't claim to be either. It means the privacy claim rests on what the system is built to do, not on a sentence that has to hold up entirely on its own.

The practical lesson isn't that AI chatbots are unsafe to use. It's that 'we don't share your data' and 'no outsider reads your data' are two different sentences, and it often takes a lawsuit to reveal which one a company was actually making. Reading the specific language, encryption at rest, per-user isolation, who is structurally able to reach your content and under what process, is a better five minutes than trusting the general tone of a privacy page.