Researchers have identified a significant issue with the open-source AI platform Hugging Face, which has become a venue for the creation of nonconsensual deepfakes, particularly sexualized images. A report from the European nonprofit AI Forensics disclosed findings from tests on nine image editing models available on Hugging Face. Shockingly, seven of these models could convert an image of a clothed woman into an explicit one with ease.
To investigate further, the researchers created controlled image editing projects on Hugging Face and tracked over 1,000 image editing prompts over a week. They found that approximately 73% of the submissions were of a sexual nature, and of these, 83% aimed to undress or sexualize the individuals in the images. Alarmingly, about 6.7% of the prompts seemed to target children.
Paul Bouchaud, lead researcher at AI Forensics, stated, “Most of the Spaces tested can be used for generating nonconsensual intimate images, and users are actually using it for these purposes.” The research also revealed many pages promoting nudifying technologies, allowing for the sexualized editing of images, which raises profound ethical concerns.
Although Hugging Face has content policies in place against child sexual abuse and explicit deepfakes created without consent, the response from the platform has been minimal. Reports suggest that numerous nudifying tools are still accessible on the site despite initial efforts to remove problematic content after being contacted by researchers.
Generative AI has advanced remarkably, leading to widespread misuse of nudifying algorithms that often victimize women and girls, with tools allowing people to digitally undress images of individuals for harassment and blackmail. While mainstream AI models have begun to implement safety measures to prevent such misuse, the models on Hugging Face were found without these protections.
Leonie Oehmig, an AI researcher, noted that the prevalence of explicit content in training datasets greatly increases the likelihood of these models producing harmful images. Many generative applications appear innocuous but can facilitate the creation of sexualized images without effective safeguards.
Concerns are rampant regarding the ability of AI models to generate harmful content beyond mere nudity, revealing instances of abusive prompts that involved degrading requests and disrespectful behavior towards women. The implications of this research underscore the urgent need for stronger moderation and regulatory measures across AI platforms to protect individuals from such abuses.
