The children’s commissioner for England is urging the government to ban apps that use artificial intelligence (AI) to create sexually explicit images of children.
Dame Rachel de Souza has called for a total ban on apps that enable “nudification” — where AI edits photos of real people to make them appear naked. She criticized the government for allowing such apps to operate unchecked, leading to severe real-world consequences.
A government spokesperson responded, reaffirming that child sexual abuse material is illegal, and announcing plans to create further offenses for the creation, possession, or distribution of AI tools designed to generate such content.
Deepfakes, which use AI to create fake videos, images, or audio that appear real, are becoming increasingly prevalent. Dame Rachel's report, published Monday, highlighted that this technology disproportionately targets girls and young women, with many apps appearing to focus solely on female bodies.
The report noted that many girls are avoiding posting images or engaging online to reduce the risk of being targeted, much like how they take precautions in the offline world, such as avoiding walking alone at night. Children fear that a classmate, stranger, or even a friend could exploit these technologies, which are often found on popular search engines and social media platforms.
Dame Rachel stated, "The evolution of these tools is happening at such scale and speed that it can be overwhelming to try and grasp the dangers they pose. We cannot allow these AI apps to have such a dangerous hold over children’s lives."
Under the Online Safety Act, it is illegal to share or threaten to share explicit deepfake images. In February, the government announced laws to tackle AI-generated child sexual abuse material, including making it illegal to possess, create, or distribute AI tools for creating such content. However, Dame Rachel argues that these measures don't go far enough and that there should be an outright ban on nudifying apps.
The Internet Watch Foundation (IWF) reported a significant rise in AI-generated child sexual abuse material, confirming 245 cases in 2024, compared to just 51 in 2023 — a 380% increase. Derek Ray-Hill, IWF Interim Chief Executive, noted that these apps are being misused in schools, with the imagery quickly spreading out of control.
A Department for Science, Innovation, and Technology spokesperson stated that creating or distributing child sexual abuse material, including AI-generated images, is "abhorrent and illegal." The UK is the first country to introduce laws specifically targeting AI tools that generate child sexual abuse material, they added.
Dame Rachel also called for the government to:
- Impose legal requirements on developers of generative AI tools to identify and mitigate risks to children.
- Set up a systemic process to remove sexually explicit deepfake images of children from the internet.
- Recognize deepfake sexual abuse as a form of violence against women and girls.
Paul Whiteman, general secretary of the school leaders’ union NAHT, echoed Dame Rachel's concerns, emphasizing that the technology is outpacing the law and the need for urgent review.
Meanwhile, Ofcom's Children's Code, which was finalized last week, mandates that platforms hosting harmful content, such as pornography and material promoting self-harm, take stronger action to protect children. Platforms must implement stricter age verification or face hefty fines.
Dame Rachel criticized the code for prioritizing the business interests of technology companies over children's safety.
Found this article interesting? Follow us on X(Twitter) ,Threads and FaceBook to read more exclusive content we post.
