San Francisco City Attorney David Chiu has instructed Apple and Google to remove dozens of "nudify" apps from the App Store and Google Play.

This directive marks a significant escalation in the battle against non-consensual deepfake pornography, as Chiu’s office has warned both technology giants they could face substantial civil penalties under California law for their alleged role in facilitating the distribution and monetization of these harmful applications. The "nudify" apps, powered by artificial intelligence, are designed to digitally alter photographs, creating images that depict clothed individuals as nude, often without their consent. These applications are readily available through Apple’s App Store and Google Play, with both companies processing the payments generated by these services, thereby reportedly profiting from their operation.

The Rise of Deepfake Technology and a Mounting Legal Challenge

The proliferation of generative artificial intelligence has brought with it an unprecedented ability to create realistic but fabricated images and videos, commonly known as deepfakes. While the technology holds immense potential for creative and beneficial applications, it has also become a potent tool for malicious actors, particularly in the creation of non-consensual intimate imagery (NCII). These "nudify" apps epitomize this darker side of AI, enabling users to generate explicit images from ordinary photographs, often targeting women and girls. The ease of access and the anonymity afforded by these apps have exacerbated concerns among lawmakers, victim advocates, and cybersecurity experts worldwide.

Globally, governments and regulatory bodies are grappling with how to address the rapid evolution of deepfake technology and its societal implications. From celebrity deepfakes used in pornography to political disinformation campaigns, the technology presents complex challenges for legal frameworks and content moderation. In the United States, several states, including California, have begun enacting laws specifically targeting deepfakes and NCII. These legislative efforts aim to provide victims with avenues for recourse and to hold platforms accountable for their role in the dissemination of such harmful content.

California’s legal landscape is particularly pertinent to San Francisco’s current action. State law already makes it a criminal offense to knowingly facilitate or recklessly aid in the creation or distribution of non-consensual deepfake pornography. This statute provides a powerful basis for Chiu’s office to challenge Apple and Google, suggesting that their oversight and profit-sharing from these apps constitute such facilitation. Furthermore, a new California law, poised to take effect in 2025, will significantly strengthen victims’ rights by allowing them to pursue civil action against third-party facilitators of non-consensual intimate deepfake content. This forthcoming legislation underscores a growing legal trend to expand liability beyond the primary perpetrator to platforms and intermediaries that enable such abuses. Chiu’s letters to Apple and Google explicitly reference both these existing and impending statutes, indicating a robust legal strategy aimed at compelling comprehensive action.

A Chronology of Mounting Pressure and Legal Scrutiny

The current crackdown by the San Francisco City Attorney’s office is not an isolated event but rather the culmination of a period of increasing scrutiny and public pressure on Apple and Google. The issue of "nudify" apps and their harmful potential has been on the radar of advocacy groups for some time, with specific concerns raised about the platforms’ role in their distribution and monetization.

• Early 2023 – Ongoing Awareness: The letters from Chiu’s office note that Apple and Google have been aware of their involvement in processing payments for these problematic apps for nearly a year. This suggests that internal discussions or external warnings about the nature of these applications and their potential legal ramifications had likely taken place well before the recent formal directive. City Attorney David Chiu himself highlighted the financial aspect, telling Wired that both companies had likely amassed millions of dollars in fees from apps offering "nudify" services, underscoring the perceived profit motive.
• January 2024 – Tech Transparency Project’s First Report: The Tech Transparency Project (TTP), a non-profit watchdog group, published its initial report identifying dozens of "nudify" apps available on both the App Store and Google Play. This report detailed how these applications openly sold non-consensual intimate image generation services, with payments seamlessly processed through Apple and Google’s respective payment systems. The TTP simultaneously sent letters to both companies, alerting them to the findings and urging remedial action.
• April 2024 – Tech Transparency Project’s Follow-Up Report: Despite the initial report and direct communication, many of these apps remained accessible. The TTP released a second, more comprehensive report in April, reiterating its findings and criticizing Apple and Google for their continued facilitation. This report specifically accused both tech giants of not only hosting but also, in some instances, guiding users toward such apps. It characterized Apple and Google as "key players in the spread of AI tools that can turn real people into sexualized images," further amplifying the call for immediate intervention. Again, TTP sent formal letters to the companies, citing their continued presence.
• May 2024 – San Francisco City Attorney’s Formal Intervention: Armed with the detailed findings from the TTP reports and drawing upon California’s legal framework, San Francisco City Attorney David Chiu issued formal letters to Apple and Google. These letters formally instructed the companies to remove the identified "nudify" apps and warned of potential civil penalties for violating state law. The letters specifically cited the prior notices from the Tech Transparency Project as part of the basis for the warning about civil penalties, reinforcing the argument that the companies had ample opportunity to address the issue proactively. The companies were given 28 days to respond to Chiu’s directive, setting a clear deadline for compliance.

This chronological sequence demonstrates a pattern of escalating concern and increasingly direct action, moving from watchdog reports to formal legal threats from a major municipal authority.

Core Allegations and the Legal Nexus of Profit and Responsibility

At the heart of City Attorney David Chiu’s directive lies the explicit accusation that Apple and Google are not merely passive hosts but active participants in the proliferation of non-consensual deepfake pornography. Chiu’s stark statement to TechCrunch — "Apple and Google are profiting from apps that exploit women and girls by creating non-consensual intimate deepfakes" — frames the issue not just as a policy violation but as an ethical and legal failure with significant financial implications. The allegation of "millions of dollars in fees" from these apps directly links their profit models to the harm being perpetrated, making the argument for accountability more compelling.

The legal strategy hinges on California’s statutes concerning the facilitation of such content. The existing law, which criminalizes knowingly facilitating or recklessly aiding in the creation or distribution of non-consensual deepfake pornography, is central to Chiu’s argument. Apple and Google, by hosting these apps, processing their payments, and potentially even promoting them through their app store algorithms, are alleged to be fulfilling the criteria of "facilitating" or "aiding." The "knowingly" or "recklessly" clauses are crucial here; the TTP reports and subsequent letters serve as irrefutable evidence that the companies were put on notice, making it difficult for them to claim ignorance.

Furthermore, the impending 2025 law, which allows victims to pursue civil action against "third-party facilitators," significantly strengthens the long-term legal leverage against platforms. While this law is not yet in effect, its inclusion in Chiu’s letters signals a clear warning about future liability and the evolving legal landscape that will hold tech companies to a higher standard of responsibility for content distributed on their platforms. The act of processing payments is particularly scrutinized. For every in-app purchase or subscription to a "nudify" service, Apple and Google typically take a percentage cut (often 15-30%). This direct financial stake transforms them from mere conduits into active beneficiaries of the alleged illegal activity, strengthening the argument that they are "profiting from exploitation."

The Tech Transparency Project’s reports provided critical documentation, detailing how these apps were not hidden but often explicitly marketed for their "nudifying" capabilities. Some apps even demonstrated the process within their promotional materials. This level of transparency in marketing, coupled with the companies’ review processes for app submissions, further complicates any defense that they were unaware of the apps’ primary function. The TTP’s assertion that Apple and Google "guided users toward such apps" through search results or curated recommendations adds another layer of alleged complicity, suggesting active promotion rather than passive hosting.

Official Responses, Corporate Policies, and Persistent Gaps

Following the City Attorney’s directive, both Apple and Google issued statements outlining their initial actions and reiterating their commitment to platform safety, though their responses revealed varying degrees of transparency and immediate impact.

An Apple spokesperson affirmed to TechCrunch that "nudify apps are not allowed on the App Store under existing policies." This statement underscores that the presence of these apps represented a clear violation of their established guidelines, which restrict content that sexualizes real people without consent. The company confirmed it had already removed three of the apps specifically questioned by Chiu’s office and was in the process of terminating their associated developer accounts. Additionally, Apple indicated it was "in contact with four others that need to fix policy violations or they may be removed," suggesting a broader review process beyond the initially named applications. However, Apple did not provide details on the total number of "nudify" apps removed prior to this specific action, nor did it disclose any revenue figures or download counts associated with these applications. The challenge for Apple, given the sheer volume of apps on its store (millions), lies in proactive enforcement and detecting sophisticated policy circumventions.

Google’s response was similarly immediate regarding the named apps. A Google spokesperson stated that "all five Play Store apps mentioned in Chiu’s letter have been suspended." Google also emphasized its broader efforts, noting, "When violations are reported to us, we investigate and take swift action." The spokesperson added that Google had "suspended hundreds of violating apps and restricting related search terms like ‘nudify’ on our store," indicating a more comprehensive, albeit reactive, approach to the problem. Restricting search terms like "nudify" is a crucial step in preventing users from easily discovering such apps, but it doesn’t address the underlying issue of their presence on the platform or the potential for alternative search terms. Like Apple, Google refrained from sharing specific data on the total revenue processed through these apps or the aggregate number of downloads, leaving a significant information gap regarding the scale of their financial involvement.

A notable commonality in both companies’ responses is their focus on "store-level distribution and payment processing" rather than addressing the "underlying models or websites hosting similar tools outside app stores." This distinction is critical. While removing apps from their stores and ceasing payment processing is a direct and impactful action, it does not dismantle the core technology or the developers behind these AI models. The same AI models could potentially be accessed via web browsers or other less regulated channels, posing a continuous challenge to content moderation efforts.

For users who had purchased or subscribed to any of the affected apps, the termination of developer accounts means they will likely lose access to these services. However, neither company has shared specific details regarding refund policies or procedures for users impacted by these removals, which could lead to consumer dissatisfaction and further complaints.

Developers creating apps that combine photo editing with generative AI features for iOS or Android are now on heightened alert. They are advised to rigorously review the specific policies cited by Apple and Google. Apple’s App Store guidelines already contain clauses prohibiting content that sexualizes real people without consent. Google’s explicit restriction of search terms like "nudify" across the Play Store further signals a zero-tolerance approach to this specific type of content. Apps that generate edited images of identifiable individuals without robust safeguards against non-consensual use are at a significantly higher risk of removal under both platforms’ evolving enforcement regimes. The onus is increasingly on developers to implement ethical AI practices and strong content moderation tools within their applications from the outset.

Broader Impact, Implications, and Future Challenges

The San Francisco City Attorney’s action against Apple and Google carries far-reaching implications, extending beyond the immediate removal of "nudify" apps. It represents a significant development in the ongoing debate about platform accountability, the ethical responsibilities of AI developers, and the protection of individuals in the digital age.

For Users: The most immediate impact for users is the removal of these harmful apps, which reduces the accessibility of tools for creating non-consensual intimate imagery. For victims of non-consensual deepfake imagery, especially in California, the 2025 law offers a powerful new avenue for recourse. The ability to pursue civil action against third-party facilitators like app stores provides a mechanism for financial compensation and a measure of justice, moving beyond mere content removal. This could set a precedent for other jurisdictions considering similar victim-centric legislation. Users are also empowered to play a more active role in policing app stores; both platforms offer direct reporting tools. On iOS, the "Report a Problem" option on each app listing allows users to flag violations, while on Android, the "Flag as inappropriate" option is found under the developer contact section. When reporting, it is crucial to reference the specific policy category of non-consensual intimate imagery generation to ensure proper classification and expedite review.

For Developers: The crackdown signals a tightening of content policies for generative AI applications. Developers must now exercise extreme caution when integrating AI features that can manipulate images of identifiable individuals. The emphasis will shift towards implementing robust ethical safeguards, consent mechanisms, and proactive content filtering to prevent misuse. This could spur innovation in "safety by design" for AI tools, encouraging developers to build protective measures into their algorithms from the ground up, rather than relying solely on post-publication moderation. The risk of app removal and account termination is a powerful disincentive for those who might otherwise push the boundaries of acceptable content.

For Platform Accountability: This action could serve as a significant precedent for other cities, states, and even federal regulators to pursue similar legal challenges against tech giants. It underscores the growing legal consensus that platforms cannot simply claim immunity as neutral conduits for third-party content, particularly when they actively profit from that content and have been made aware of its illegal nature. While the U.S. Communication Decency Act’s Section 230 generally protects platforms from liability for user-generated content, the "knowingly facilitate" or "recklessly aid" clauses in California law, particularly concerning specific types of egregious content like child exploitation or non-consensual deepfakes, represent potential carve-outs or interpretations that could circumvent such protections. The case highlights the persistent challenge of content moderation at scale, forcing tech giants to invest more heavily in AI detection, human review, and proactive policy enforcement.

Societal Implications: The issue touches upon fundamental societal concerns regarding privacy, gender-based violence, and the ethical governance of emerging technologies. The normalization of non-consensual imagery, even if AI-generated, contributes to a culture of exploitation and harms the dignity and safety of individuals, particularly women and girls who are disproportionately targeted. Chiu’s office emphasizes that "While the companies have removed some problematic apps, they still have a responsibility to be proactive and vigilant in preventing sexual abuse," framing the issue as a public safety imperative. This legal action contributes to a broader societal push for greater corporate responsibility in mitigating the harms associated with their products and services.

Future Outlook and Challenges: The 28-day response window given to Apple and Google is critical. While both companies have taken initial steps, neither has confirmed whether they plan to review related apps beyond those specifically named in Chiu’s letters. This lack of commitment to a broader, proactive audit suggests that future legal or regulatory pressure may be necessary to ensure comprehensive compliance. The "cat-and-mouse" game between malicious developers and platform moderators is likely to continue, with developers constantly seeking new ways to circumvent policies. This necessitates continuous investment in advanced AI detection systems that can identify new variants of harmful apps and content. Furthermore, this incident could reignite discussions about the need for federal legislation in the U.S. to create a uniform standard for platform liability regarding harmful AI-generated content, or even international cooperation to address the cross-border nature of digital harm. The role of AI in detecting AI-generated harmful content will become increasingly vital, requiring sophisticated solutions to combat sophisticated abuses.

The actions by the San Francisco City Attorney represent a firm statement that profit cannot overshadow responsibility, and that tech giants will be held accountable for the content they host and monetize, particularly when it enables severe harm.