Hundreds of contractors were engaged by Meta to pose as minors online, testing how rival chatbots like ChatGPT, Gemini, and Character.AI would respond to queries about sensitive issues such as suicide, sexual content, and drug use. This internal project, codenamed Cannes, aimed to collect data on chatbot responses to provocative prompts.
The contractors created fake under-18 profiles, sending various written and visual prompts to these chatbots, which were then logged in spreadsheets. Many of the prompts targeted subjects that existing safety measures in these chatbots were designed to reject. For instance, during a testing round in August 2025, the contractors generated over 45,000 prompts without the knowledge of the chatbot companies involved.
The data collection included prompts that asked about navigating crises, such as a 13-year-old seeking information on terminating a pregnancy or a classmate threatened with a gun. The project also featured numerous disturbing requests, some even asking seemingly inappropriate questions about cannibalistic thoughts or illegal drug procurement.
Meta framed the operation as a form of safety testing, describing it as "comprehensive AI safety benchmarking" crucial for improving system responses. A spokesperson for the company emphasized that testing competitor responses is a standard practice in the tech industry, even denying that it used rival data to improve its own models.
However, the methods employed in the Cannes project raised eyebrows among former contractors and industry experts who expressed concerns about the ethical implications. Some contractors feared they might inadvertently create or retain harmful content, while others questioned the legitimacy of secretly gathering data from competing systems.
Experts, including Rumman Chowdhury, critiqued the project’s structure, highlighting its deviation from standard practices and potential risks of combining safety evaluations with competitive benchmarking. Legal experts reviewed the prompts but concluded that they mostly didn’t cross the line into illegal territory, though they did violate the respective terms of service set by the competitors involved.
As the investigation unfolds, concerns linger about the blending of safety evaluations and competitive practices, suggesting a troubling grey area in AI governance where safety might be misused for competitive advantage.
For immediate support, individuals in need can contact the National Suicide Prevention Lifeline at 988 or text HOME to 741-741 for free 24-hour support.
