🔗 Share this article

Tech firms and child protection agencies will receive authority to assess whether artificial intelligence tools can produce child abuse images under new UK legislation.

Substantial Rise in AI-Generated Harmful Content

The announcement coincided with revelations from a safety monitoring body showing that cases of AI-generated CSAM have increased dramatically in the past year, rising from 199 in 2024 to 426 in 2025.

Updated Legal Framework

Under the amendments, the authorities will permit approved AI developers and child safety groups to inspect AI models – the foundational systems for conversational AI and image generators – and verify they have adequate protective measures to stop them from creating images of child sexual abuse.

"Ultimately about preventing abuse before it happens," stated the minister for AI and online safety, adding: "Specialists, under rigorous conditions, can now identify the risk in AI systems promptly."

Addressing Legal Challenges

The changes have been implemented because it is against the law to produce and own CSAM, meaning that AI creators and other parties cannot generate such images as part of a testing process. Until now, authorities had to delay action until AI-generated CSAM was published online before dealing with it.

This legislation is aimed at preventing that issue by enabling to stop the creation of those images at source.

Legal Framework

The changes are being introduced by the authorities as modifications to the criminal justice legislation, which is also establishing a ban on owning, producing or sharing AI systems designed to generate child sexual abuse material.

Real-World Consequences

This recently, the minister visited the London headquarters of a children's helpline and listened to a mock-up call to counsellors involving a report of AI-based abuse. The call depicted a adolescent seeking help after facing extortion using a sexualised AI-generated image of himself, constructed using AI.

"When I hear about young people facing blackmail online, it is a source of extreme anger in me and rightful concern amongst parents," he said.

Alarming Statistics

A leading online safety organization reported that cases of AI-generated abuse content – such as webpages that may contain numerous files – had more than doubled so far this year.

Instances of category A content – the gravest form of abuse – increased from 2,621 visual files to 3,086.

• Female children were overwhelmingly targeted, accounting for 94% of prohibited AI images in 2025
• Portrayals of infants to two-year-olds increased from five in 2024 to 92 in 2025

Sector Reaction

The legislative amendment could "constitute a vital step to ensure AI products are safe before they are released," stated the head of the internet monitoring foundation.

"AI tools have enabled so victims can be victimised all over again with just a few clicks, giving offenders the capability to create potentially limitless quantities of sophisticated, photorealistic exploitative content," she continued. "Material which additionally exploits victims' suffering, and makes children, especially female children, more vulnerable on and off line."

Support Interaction Information

The children's helpline also released information of counselling sessions where AI has been referenced. AI-related harms discussed in the sessions include:

• Employing AI to evaluate weight, physique and appearance
• Chatbots dissuading children from consulting trusted guardians about harm
• Being bullied online with AI-generated content
• Digital blackmail using AI-faked images

During April and September this year, Childline delivered 367 support sessions where AI, conversational AI and related terms were mentioned, four times as many as in the equivalent timeframe last year.

Half of the mentions of AI in the 2025 sessions were related to psychological wellbeing and wellbeing, encompassing using chatbots for support and AI therapeutic apps.