San Francisco AG presses Apple and Google over AI nudify apps

San Francisco Attorney General David Chiu has demanded that Apple and Google remove 13 AI “nudification” apps from their app stores, according to Wired, escalating legal pressure on the two largest mobile distribution platforms over nonconsensual sexual imagery. The letters accuse the companies of enabling apps that can turn ordinary photos of real people into explicit images, a category that has become a live enforcement problem for app review teams and AI safety systems.

Wired reported that Chiu’s office sent cease-and-desist letters this week asking Google to remove five apps and Apple to remove eight. The apps were not identified publicly, with Wired saying it withheld their names to avoid directing users to them. One app cited in the report had more than 1 million downloads and promoted tools to sexualize images of women or create “free and uncensored” videos, according to Wired.

Chiu’s legal argument centers on California laws barring support for services that create deepfake pornography. In comments to Wired, Chiu said his office was “absolutely horrified” by the spread of nudification tools, which he said are used disproportionately against women and children. He said the images are used to bully, humiliate and threaten women and girls, and can damage reputation, mental health and personal autonomy.

The letters also put a financial frame around the issue. Chiu’s office estimated that Apple and Google have likely collected millions of dollars in fees from these apps, according to Wired. The companies did not disclose any revenue from the flagged apps, and Chiu’s estimate has not been independently verified.

Google says the flagged apps were suspended

Google spokesperson Dan Jackson told Ars Technica that the five apps identified by Chiu had been suspended from Google Play for violating policies against harmful content. Jackson said Google Play bars apps containing sexual content and said the company takes proactive steps to find and remove harmful apps. He also said Google has suspended hundreds of violating apps and restricted related search terms, including “nudify,” in its store.

Apple did not respond to Ars Technica’s request for comment. Wired reported that Apple has removed some apps previously flagged by researchers, but the company has been quieter than Google in responding publicly to demands for tougher enforcement.

The enforcement challenge is getting harder because some apps do not market themselves plainly as nudification products. Wired cited a May preprint paper in which researchers identified 420 apps presented as generic face-swapping tools. The researchers tested 155 of them and found that 70% could be used to sexualize images.

AI app review pressure extends beyond small developers

The scrutiny is not limited to niche apps. Ars Technica reported that xAI recently filed a lawsuit acknowledging that it found instances of Grok-generated child sexual abuse material and other nonconsensual intimate imagery targeting adults. xAI sued the user it says prompted Grok to produce the illegal content, but the episode has raised questions about whether app stores should treat large AI chatbots the same way they treat smaller apps that can generate prohibited sexual imagery.

NBC News reported in April that Apple told senators it had privately threatened to remove Grok from the App Store. The xAI app remains available. Chiu’s letters did not seek Grok’s removal, according to the reporting, but his demand signals broader pressure on app stores to police generative AI outputs, not just app descriptions.

Chiu told Wired he wants Apple and Google to remove the flagged apps and strengthen screening so similar apps do not reach their platforms. He said his office would consider legal options if the companies do not act.

This story draws on original reporting from Ars Technica.