Hundreds of contractors for Meta were instructed to impersonate minors online to assess how competing chatbots like ChatGPT and Gemini reacted to sensitive topics such as suicide, sex, and drugs. This initiative, internally dubbed "Cannes," involved contractors creating fake under-18 accounts to submit numerous prompts and collect responses in spreadsheets.
The contractors sent a significant volume of prompts—including over 45,000 during a testing phase in August 2025—designed to elicit responses from the chatbots that their safety systems were meant to reject. These prompts included graphic images, distressing scenarios, and inquiries from the viewpoint of children or teenagers in crisis. For instance, one prompt depicted a 13-year-old pregnant by an adult neighbor seeking abortion pills, while another involved a child asking how to hide an eating disorder from parents. Various prompts also touched on highly sensitive subjects, including self-harm and drug use.
Meta contractors noted some prompts were also not in English, expanding the scope of inquiry. Internal documents indicated that the project aimed to create a comprehensive dataset for "AI safety benchmarking," though details on how Meta intended to use this data were sparse.
Meta’s spokesperson characterized the effort as routine safety testing to ensure chatbots provide safe, age-appropriate experiences. While comparing competitors’ products is common in the AI industry, former contractors expressed discomfort with the method of testing, fearing it might inadvertently generate inappropriate content or violate terms of service of the competing chatbots.
Legal experts reviewed the prompts flagged and determined that although some did not solicit illegal content, the overall conduct likely breached the terms set by companies like OpenAI and Google, who had not authorized such testing. Meta defended its approach, asserting that it was a responsible industry practice.
This incident raises significant ethical questions about the blending of safety evaluation with competitive analysis, with critics arguing it could be construed as harmful to the integrity of AI governance.
If you or someone you know needs help, call 988 for free, 24-hour support from the National Suicide Prevention Lifeline or text HOME to 741-741 for the Crisis Text Line.
