Talking to a chatbot feels like chatting with software, but behind every answer are real people who reviewed thousands of conversations to teach it what a good reply looks like. ASeptember 20
Talking to a chatbot feels like chatting with software, but behind every answer are real people who reviewed thousands of conversations to teach it what a good reply looks like. ASeptember 2026 class action filed against OpenAI over an internal programme called Project Lily asks whether ChatGPT users knew that real people might read their chats, and the answer depends heavily on which product they used and which setting they chose.
Project Lily is the internal name for a human-review programme in which contractors were shown real ChatGPT prompts and the responses the system generated, then asked to work out what the user wanted, summarize the task and rate several candidate answers. Reporter Joseph Cox shared details of the programme on X, and the lawsuit coverage has since put the privacy question in front of a court.
This article covers how human feedback entered chatbot training, what reviewers see, which product tiers and settings control it, what the lawsuit alleges, why a market for expert-reviewed data is growing, and how crypto projects propose to verify and pay human reviewers.
When OpenAI introduced ChatGPT in 2022, it explained that the system had been trained with Reinforcement Learning from Human Feedback, or RLHF. Human AI trainers wrote conversations, ranked alternative responses, and supplied the comparisons that trained a reward model, and that reward model then guided further training. Instead of writing a rulebook for every good answer, researchers showed the system which answers people preferred and let it learn the pattern.
This matters because predicting the next word is not the same as knowing which answer a person wants. Two replies can both be factually correct while one is clear and direct and the other is confusing or strangely written, and human feedback is one of the main ways a model learns the difference. OpenAI has also shown that people can be assisted by AI when grading AI, since its2024 CriticGPT research reported that reviewers who used a GPT-4-based critic to check ChatGPT’s code outperformed reviewers working without one 60% of the time.
Diagram of a reinforcement learning feedback loop in which a model produces answers, human reviewers rank them, and a reward model guides further training. Source: IBM
In June 2024, OpenAI reported thatCriticGPT, a GPT-4-based model trained to critique code written by ChatGPT, helped trainers catch bugs, and that a second trainer preferred critiques written by a trainer working with CriticGPT over critiques written by an unassisted trainer more than 60% of the time. The result shows that reviewers already work alongside AI tools, which becomes relevant to the contractor removals described later in this article.
Public images of AI training often involve repetitive work such as drawing boxes around cars, and that work still exists, but frontier chatbots depend more on judgment. A reviewer may have to decide whether a response understood a complicated question, whether it invented information, whether it followed instructions, whether it was appropriately cautious, or whether it was simply agreeing with the user to please them. Project Lily reviewers were trained to flag problems such as excessive sycophancy and anthropomorphism, which means they helped decide not only whether ChatGPT was right but how it should behave.
New from 404 Media: humans are reading ChatGPT conversations
OpenAI has hired an army of contractors who read real ChatGPT users’ chats. I’ve seen internal docs, the review system, and real user prompts. Can contain very personal/sensitive information https://t.co/6s7L7JTEmY
— Joseph Cox (@josephfcox) September 14, 2026
A chatbot conversation feels private because the interface is private, yet OpenAI’sprivacy explanation says that content from users, human trainers, and researchers can be used to develop its models, with consumer content used according to the user’s settings. Reviewers did not receive ChatGPT usernames, and OpenAI says conversations pass through itsPrivacy Filter before contractors see them. The reporting on Project Lily found that personal or sensitive details could still appear in what reviewers received, and some internal screenshots showed context from earlier interactions.
Removing a username does not make a conversation anonymous, because a person can often be identified from what they say. A message that mentions an employer, a profession, an unusual event, a location, or a family situation can point to one person even without a name. OpenAI acknowledges that personal information can appear in training data and says it uses filtering and other safeguards to reduce its processing, but a filter that reduces exposure does not guarantee that every sensitive detail disappears.
Three different things get described as “training on your chats,” and the lawsuit depends on keeping them apart. The first is automated training, in which text is fed through software that adjusts a model’s parameters, and no person reads the conversation. The second is human review, which is what Project Lily describes, where a contractor reads a prompt and a response and writes a rating or critique that becomes training signal. The third is retention for safety purposes, which OpenAI says applies even to Temporary Chats, where a copymay be kept for up to 30 days, although the conversation is not used for training.
OpenAI says the purpose of training on conversations is to develop capabilities such as prediction and reasoning, and that the data is not used to build profiles of individuals, contact them or target them with advertising, according to itsprivacy explanation. That statement addresses the first act and says little about the second. A company does not need a permanent profile of a person for a human reviewer to read that person’s conversation, and a user who switches off model improvement has to rely on the product’s terms to know whether the other two acts still apply. Thecomplaint is built on the gap between what users were told about automated training and what actually happened when people read their chats, which is why the distinction carries legal weight and not only semantic weight.
OpenAI’sdata controls FAQ describes different treatment by product. For consumer ChatGPT, the “Improve the model for everyone” setting controls whether new conversations are used for training, and when it is off, they stay in chat history without being used to train models. Temporary Chats are not used for training while they remain temporary. Business, Enterprise, Edu, and API data are not used for training by default.
Table 1. How OpenAI’s product tiers treat conversations for training
Product Used for training? What controls it Consumer ChatGPT Depends on the user’s setting “Improve the model for everyone” toggle Temporary Chat No, while it remains temporary A copy may be kept up to 30 days for safety Business, Enterprise, Edu No, by default Product terms API No, by default Product termsData from OpenAI’sdata controls FAQ.
The complaint, Vredenburgh v. OpenAI OpCo, wasfiled on September 16, 2026, in the Northern District of California under case number 3:26-cv-10527. It makes eight claims and asks to represent consumer users, with the proposed class excluding Enterprise, Business, Team, Edu, and API customers. A case management conference is set for December 18, 2026, and the allegations have not been tested in court.
Table 2. Vredenburgh v. OpenAI OpCo at a glance
Item Detail Filed September 16, 2026 Court U.S. District Court, Northern District of California Case number 3:26-cv-10527 Claims Eight Proposed class Consumer users, excluding Enterprise, Business, Team, Edu and API customers Next date Case management conference, December 18, 2026Data from thecomplaint on CourtListener.
Project Lily also says something about the economics of AI, since compute is only one input to a frontier model. Snorkel AI announced a $350 million Series E at a $3.5 billion valuation on September 22, 2026, led by Insight Partners and S32, nearly triple its previous valuation of $1.3 billion, and the company works with specialists in fields such as coding, law and medicine to produce training data.
Once models have absorbed most of the open internet, the scarce resource is data that helps them do specialized work reliably, and a doctor grading a medical answer is worth more for some tasks than another billion generic web pages. DeFi Planet has covered the same pressure from the infrastructure side in its look atwhether decentralized AI can compete with OpenAI, Google and Meta.
READ ALSO: Can Decentralized AI Really Compete With OpenAI, Google, and Meta
A September 2026 report said OpenAI training contractors had been removed from projects for using AI themselves during evaluation work, that some projects involved thousands of contractors and that workers were explicitly prohibited from using AI to complete their assignments. The prohibition exists because of how the training signal works. In RLHF, a reward model learns to predict which answers human reviewers prefer, and the chatbot is then tuned toward whatever that reward model scores highly. A reviewer who pastes two candidate answers into another chatbot and copies back its verdict has put a machine’s preference into a pipeline built to capture a person’s preference, and the reward model learns a mixture of the two without any record of which rating came from where.
Researchers have a name for what can follow when machine-made data is fed back into training without controls. A2024 Nature study by Ilia Shumailov and colleagues found that models trained repeatedly on text generated by earlier models lose the rare and varied parts of the original data first and, over successive generations, converge on a narrow and distorted output, a process the authors call model collapse. Reviewer shortcuts are a small version of the same risk, because the ratings that teach a chatbot its manners would increasingly reflect another chatbot’s habits.
The difficulty for AI developers is that assisted review is not always a problem, since OpenAI’s own CriticGPT work deliberately gives reviewers an AI critic, and the difference lies in whether the assistance is designed, disclosed, and measured or done privately by a contractor paid per task. Detecting the second kind is hard at the scale of thousands of workers, which leaves the industry building machines that can do annotators’ work while still needing proof that the annotation it relies on is genuinely human.
The contractor removals describe a problem that several crypto projects were built around, which is proving that a person and not a model produced a given judgment and paying that person in a way that penalizes careless work.
World, the proof-of-human network co-founded by OpenAI chief executive Sam Altman, lets users verify with a selfie, a government ID, or an iris scan at a device called an Orb and then prove to other services that a real, verified human approved an action, and its protocol uses zero-knowledge proofs so that a service receives the confirmation without storing personal data. The project announced on July 24, 2026, an initial $52.5 million close from a one-year locked sale of its WLD token, led by Pantera Capital, and said more than 18 million people had verified at an Orb, while Okta and Vercel were reported in April to be building tools that confirm a real human approved actions taken by AI systems.
Sapien applies a similar idea to the labelling work itself. Its Proof of Quality design, described in the project’stokenomics document, requires contributors to lock SAPIEN tokens as collateral before taking complex tasks, has each completed task reviewed by a contributor with a higher reputation score, and slashes stakes by up to 25% for repeated low-quality work or false validations and by up to 100% for fraud. Rewards are paid by default 80% in stablecoins and 20% in SAPIEN, and the document reports about 1.8 million registered contributors, more than 800,000 active participants, and more than 185 million completed tasks, figures that come from the project itself and have not been independently verified.
Table 3. How Sapien ties task access to stake and quality
Tier Stake required Minimum quality Who validates the work Trainee None Above 90% A Contributor Contributor 250 SAPIEN 95% or higher An Expert Expert 5,000 SAPIEN Above 98% A Master Master 25,000 SAPIEN 99% minimum Another Master or panel for critical tasksData from Sapien’sProof of Quality tokenomics document.
These designs answer the labor side of Project Lily and leave its privacy side untouched, because the lawsuit concerns whether ChatGPT users knew that people might read their conversations, and a staking requirement or an Orb verification for the reviewer does nothing to tell the user. None of the reporting on Project Lily says OpenAI or its vendors use World ID or token-based payment;World has drawn criticism from analysts over security and governance, and Sapien’s top tier requires 25,000 SAPIEN in stake, a barrier that favours contributors with capital.
What the two worlds share is a question about verified, accountable human input into AI, and the crypto projects publish the rules for that input in public documents and smart contracts, which is the kind of disclosure the lawsuit says ChatGPT users did not receive.
INTERESTING: The People Building AI Are Warning Us About AI
The record so far shows that human reviewers read real conversations in at least one OpenAI programme, that a filter sits between the chats and the contractors, and that reporting found personal details could still get through. It does not show how many users were affected, how often sensitive material reached reviewers, or whether a court will find the disclosures inadequate.
The comparison that matters for individuals and companies is between products whose terms say human review may occur, products that train only when a setting allows it, and products that do not train on customer data by default, and crypto networks that stake and verify their reviewers show that such rules can be published in advance, although none of them tells a chatbot user whether a person will read a conversation. The next place where more of that detail will become public is the court docket, with a case management conference set for December 18, 2026.
It is the internal name for a programme in which contractors reviewed real ChatGPT prompts and responses, summarized the user’s task and rated candidate answers, according to the reporting cited in the court complaint.
Can OpenAI employees or contractors read ChatGPT conversations?OpenAI says content from users, human trainers, and researchers can be used to develop its models, and the Project Lily reporting shows contractors were given real conversations. Eligibility depends on the product and setting, as laid out in Table 1 and in OpenAI’s data controls FAQ.
How can ChatGPT training on a conversation be turned off?On consumer ChatGPT, the “Improve the model for everyone” setting in data controls governs training, and Temporary Chat keeps a conversation out of training, though a copy may be retained for up to 30 days for safety, according to OpenAI’s data controls FAQ. Business, Enterprise, Edu and API products do not train on customer data by default.
Does turning off model improvement stop human review?OpenAI’s published controls describe what happens to training, and the Project Lily lawsuit argues that users were not clearly told about human review. Whether a given setting also prevents a person from reading a conversation is a question the complaint raises, and the court has not yet answered.
Who can join the Project Lily class action?The proposed class covers consumer ChatGPT users and excludes Enterprise, Business, Team, Edu and API customers, according to the complaint. No class has been certified.
Can crypto verify that a human labeled AI training data?Proof-of-personhood systems such as World ID are designed to confirm that a real person approved something, and staking systems such as Sapien’s Proof of Quality tie task access and rewards to work quality. Neither is reported to be used in Project Lily, and neither tells a chatbot user whether a conversation will be read.
Disclaimer: This article is intended solely for informational purposes and should not be considered trading or investment advice. Nothing herein should be construed as financial, legal, or tax advice. Trading or investing in cryptocurrencies carries a considerable risk of financial loss. Always conduct due diligence.
Enjoyed this? BookmarkDeFi Planet, explore related topics, and follow us onTwitter,LinkedIn,Facebook,Instagram,Threads, and CoinMarketCap Community for seamless access to high-quality industry insights.
Take control of your crypto portfolio with DEFI PLANET PRO, DeFi Planet’s suite of analytics tools.
The post Human Reviewers Still Train AI Chatbots on Real Conversations, and a Lawsuit Is Testing the Privacy Terms appeared first on DeFi Planet.
