San Francisco City Attorney David Chiu has formally issued a cease-and-desist letter to Meta Platforms, Inc., demanding that the social media giant immediately halt the distribution of paid advertisements featuring AI-generated child sexual abuse material (CSAM). The legal escalation follows investigative reports revealing that Meta’s advertising systems allowed hundreds of ads to run that utilized generative artificial intelligence to create sexually explicit videos of minors. The City Attorney’s Office is seeking a comprehensive explanation regarding the failure of Meta’s automated and manual review processes, which the company claims are designed to prevent such content from ever reaching the public.
The controversy centers on the discovery of more than 350 advertisements that appeared on Facebook, Instagram, and Threads over recent months. According to data compiled by the Tech Transparency Project (TTP) and reported by WIRED, these advertisements utilized still images of real children—including at least one member of a European royal family—and transformed them into short, sexually suggestive videoclips using AI technology. These ads primarily served as promotional funnels for "nudify" applications, which are tools designed to digitally undress individuals or generate nonconsensual intimate imagery (NCII) through machine learning algorithms.
Chronology of Investigative Findings and Meta’s Response
The timeline of these events suggests a persistent and systemic failure within Meta’s content moderation framework. In early August, an initial investigation identified a batch of 53 advertisements containing AI-generated child sexual abuse imagery. Despite this being brought to the company’s attention, the Tech Transparency Project discovered that more than 250 additional ads of a similar nature were permitted to run on Meta’s platforms following the initial report. This indicates that the company’s internal "signals" and moderation tools failed to proactively identify repeat offenders or similar content patterns.
The ads reached a significant audience, with data indicating they were displayed to more than 29,000 accounts across the European Union, the United States, Australia, and India. The repetitive nature of the uploads—where identical ads were submitted multiple times—highlights a loophole in Meta’s ad-scanning technology. While Meta asserts that it maintains a zero-tolerance policy regarding child exploitation, the fact that hundreds of ads bypassed its review systems after a known issue was identified has prompted the San Francisco City Attorney to label current efforts as "woefully inadequate."
Technical Analysis of the Exploitative Advertisements
The advertisements in question represent a sophisticated use of generative AI for malicious purposes. The process typically involves "image-to-video" AI models that can animate static photographs. By targeting images of real minors, the creators of these ads generate content that falls under the legal definition of child sexual abuse material, regardless of whether the video itself is a digital fabrication.
Upon clicking these advertisements, users were directed to external platforms or app stores to download AI-generation software. Some of these apps specifically market themselves as "nudifiers," a category of software that has seen a surge in popularity as generative AI becomes more accessible. These tools pose a significant threat to digital safety and privacy, as they allow users to create sexually explicit content of any individual without their consent, often targeting minors and public figures.
The Legal Demands of the San Francisco City Attorney
In a four-page letter addressed to Meta’s legal counsel, David Chiu raised several critical questions regarding the company’s advertising policies. The City Attorney’s Office is demanding that Meta provide a detailed account of how these advertisements managed to evade both automated and manual moderation. Under Meta’s own stated policies, all advertisements are supposed to undergo a review process prior to distribution. The failure of this process in such a high-stakes context raises concerns about the reliability of the company’s safety infrastructure.
The cease-and-desist letter specifically requests information on:
- The mechanisms by which Meta identifies and escalates CSAM to the National Center for Missing and Exploited Children (NCMEC).
- The protocols for handling repeat offenders who successfully bypass ad filters multiple times.
- The internal discussions and actions taken by Meta’s legal and child-safety teams after the initial August reports.
- The justification for accepting payment for advertisements that clearly violate both federal law and company policy.
Meta has 28 days to respond to the inquiry. The City Attorney’s Office emphasized that the objective is not merely to address past failures but to force Meta to implement structural changes that prevent the exploitation of children through its paid advertising services.
Meta’s Jurisdictional and Policy Defense
In response to the legal pressure, Meta has raised questions regarding the jurisdiction of the San Francisco City Attorney. A spokesperson for the company stated that there is "no indication" the specific ads identified by researchers were displayed to users within San Francisco. Meta argues that if the harm did not occur within the city’s limits, the City Attorney may lack the authority to pursue the matter.
However, Meta’s own Ad Library—a public repository of advertisements—does not provide granular geographic data for the United States, making it difficult for independent researchers or local officials to verify exactly where ads are served. From a legal standpoint, San Francisco may argue that because Meta is headquartered in the San Francisco Bay Area and its advertising operations impact the general public interest, the city has a valid stake in ensuring the company adheres to safety standards.
Meta also noted that the total revenue generated from the 300-plus ads was relatively low, estimated at under $5,000. The company further claimed that many of the ads had been removed prior to being reported by TTP, and that most received fewer than 200 impressions. Despite these defenses, the company’s inability to stop the influx of new, identical ads remains the primary point of contention for regulators.
Broader Implications for the Tech Industry and AI Safety
The situation at Meta is indicative of a broader crisis facing social media platforms as they struggle to keep pace with the rapid evolution of generative AI. The ease with which malicious actors can now produce high-quality, deceptive, and harmful content has outstripped the current capabilities of many automated moderation systems.
This case highlights several critical issues for the tech industry:
- The Profit Incentive vs. Safety: Critics argue that because Meta profits from every ad served, there is a financial disincentive to implement overly restrictive filters that might flag legitimate content. However, when the content involves CSAM, the legal and ethical stakes override commercial interests.
- The "Nudify" App Economy: The proliferation of apps designed to create nonconsensual imagery is a growing sector of the "shadow" AI market. While Meta has previously sued companies involved in these apps, the sheer volume of new developers makes it a "whack-a-mole" problem.
- Regulatory Scrutiny: This action by the San Francisco City Attorney may serve as a precursor to broader state or federal investigations. In the United States, Section 230 of the Communications Decency Act generally protects platforms from liability for user-generated content, but these protections are much narrower when it comes to federal criminal law violations, such as those involving child exploitation. Furthermore, since these are paid advertisements, the platform’s role as a commercial distributor is more pronounced.
Reactions from Digital Safety Advocates
Katie Paul, Director of the Tech Transparency Project, has been vocal about the inadequacy of Meta’s response. She noted that brand-new ads featuring the same children identified in their September 8 report continued to appear on the platform as recently as this week. According to Paul, the scale and persistence of the issue suggest that Meta is not effectively addressing the core problem, even when provided with direct evidence of the violations.
Child safety organizations have also expressed concern that the normalization of AI-generated CSAM could lead to a desensitization of the public or provide a "gateway" for individuals to seek out real-world child abuse material. The National Center for Missing and Exploited Children (NCMEC) continues to see a massive surge in reports related to AI-generated imagery, which complicates the process of identifying and rescuing real victims who may be hidden among millions of digital fabrications.
Future Outlook and Corporate Responsibility
As Meta prepares its response to the City Attorney, the company faces increasing pressure to overhaul its advertising algorithms. The incident serves as a reminder that as AI technology becomes more sophisticated, the responsibility of the platforms that host and monetize that technology must increase proportionally.
The San Francisco City Attorney’s Office has made it clear that "isolated takedown efforts" are no longer sufficient. The demand is for a systemic, proactive approach that prioritizes child safety over advertising throughput. Whether Meta can or will implement such a system remains to be seen, but the legal and public relations consequences of failing to do so are mounting. The outcome of this cease-and-desist demand will likely be watched closely by other tech companies and regulators worldwide as a test case for platform accountability in the age of generative AI.
About the Author
