Skip to main content
Incident intelligence/SS-IR-008CASE FILE OPEN
Symbolic editorial illustration for SS-IR-008SERVANTSTACK // INCIDENT INTELLIGENCEFORENSIC IMAGE // VERIFIED FRAME
SS-IR-008 // INCIDENT REPORTReported

FakeApp / r/deepfakes

A Free Face-Swap Tool Industrializes Non-Consensual Celebrity Porn Before Anyone Approved It

EXECUTIVE BRIEF

In December 2017, an anonymous Reddit user calling himself "deepfakes" used a machine-learning face-swap algorithm, publicly available videos, and a home computer to graft the faces of celebrities onto pornographic footage.

CHATBOTS & LLMsDEEPFAKES & FRAUD
FAILURE CHAINTRACE COMPLETE
  1. 01TRIGGERIn December 2017, an anonymous Reddit user calling himself "deepfakes" used a machine-learning face-swap algorithm,…
  2. 02MACHINE ACTIONAutonomous actor
  3. 03MISSING GATEIdentity verification and dual control
  4. 04IMPACTData security
01 // INCIDENT SUMMARY

The short version

In December 2017, an anonymous Reddit user calling himself "deepfakes" used a machine-learning face-swap algorithm, publicly available videos, and a home computer to graft the faces of celebrities onto pornographic footage.

02 // KEY FACTS

Case telemetry

INCIDENT
SS-IR-008
DATE
February 7, 2018
SYSTEM
FakeApp / r/deepfakes
LOCATION / SCOPE
Global (originated on Reddit)
EVIDENCE
Reported
AI ROLE
Autonomous actor
HARM
Data security
SOURCES
2 cited records
03ENTRY POINT // WHAT HAPPENED

The event

In December 2017, an anonymous Reddit user calling himself "deepfakes" used a machine-learning face-swap algorithm, publicly available videos, and a home computer to graft the faces of celebrities onto pornographic footage. By January 2018 the technique had been packaged into FakeApp, a free desktop tool that let anyone with no technical skill produce the same fakes by selecting a video, downloading a pre-trained face model, and pressing one button. Within weeks FakeApp had been downloaded more than 100,000 times, and the r/deepfakes subreddit had swelled past 90,000 members mass-producing non-consensual sexual videos of Gal Gadot, Daisy Ridley, Emma Watson, Taylor Swift, Scarlett Johansson and others. On February 7, 2018, Reddit banned r/deepfakes and its sister communities for violating its involuntary-pornography policy, joining Twitter, Discord, Imgur, Pornhub and Gfycat, which had all moved to ban the content in the same window. It was the first mainstream deepfake-abuse crisis.

04CAUSAL TRACE // AI'S ACTUAL ROLE

What the machine did

The harm was the model output, generated and distributed with zero human approval gate anywhere in the loop. The neural network performed the face-swap automatically; FakeApp wrapped that capability so the only human "decision" left was clicking a button, and no person ever reviewed, authorized, or signed off on whose face was being placed into whose pornography. The system was built to scale identity-theft-grade fabrication to anyone, at machine speed, with no consent check, no victim notification, and no accountable reviewer between the prompt and the published video. By the time platforms reacted, the tool had already industrialized a kind of abuse that previously required a skilled VFX studio - and there was no one in the pipeline who had ever been asked to say yes.

Autonomous actorAutomation was a causal participant—not a decorative label for the system around it.
05BLAST RADIUS // CONSEQUENCES

Where the failure landed

FakeApp's 100,000-plus downloads and the 90,000-member subreddit turned a fringe technique into an off-the-shelf weapon against real, named women in a matter of weeks, and the videos spread far faster than any single platform could remove them. The episode introduced "deepfake" into the mainstream vocabulary and triggered the first wave of platform bans, research into detection, and eventual legislation (US state non-consensual-deepfake laws, the UK Online Safety Act, and the 2025 federal TAKE IT DOWN Act). But the underlying tooling never went away - it forked, rebranded, and proliferated, seeding a non-consensual synthetic-porn ecosystem that studies have repeatedly found makes up the overwhelming majority of all deepfakes online and now overwhelmingly targets private individuals, not just celebrities.

06 // EVIDENCE STATUS

Reported

Documented in the cited public record. Follow the sources for the precise evidentiary posture.

SOURCE RECORD UPDATED 2026-07-09

07 // SOURCE LEDGER

2 cited records

  1. 01
  2. 02
08CONTROL FAILURE // MISSING GOVERNANCE

Identity verification and dual control

The failure pattern in this case: Unverified identity or synthetic media.

09INTERVENTION POINT // HUMAN IN THE MIDDLE

The moment the path could change

A named reviewer verifies identity through a separate trusted channel before money, access, or public claims can move.

AI PROPOSES→HUMAN OWNS THE DECISION→SYSTEM EXECUTES
10CONTROL DEPLOYMENT // AUTHORITYGATE

Identity verification · dual control

AuthorityGate is an Operational Resilience framework: it treats the generation of a real, identifiable person's likeness in a sensitive context as something a qualified human Subject Matter Expert must validate and authorize - not an action a model quietly performs the instant a button is pressed. A change-validation gate would have required documented, verifiable consent from the depicted individual before any face-swap targeting an identifiable person could render or publish, with a named human reviewer accountable for that authorization. FakeApp's entire design was the inverse: it deliberately stripped every human checkpoint out of the workflow so that no one ever had to approve the use of a victim's face. The failure here was not a clever model - it was a pipeline engineered to ensure that no accountable person ever stood between someone's identity and its weaponization.

RELEVANT KEYSTONE CONTROLHuman-in-the-Loop ValidationHow high-risk actions route to a named subject-matter expert who owns the go or no-go decision.
12 // THE ALTERNATIVE

Autonomy is a design choice.

See the operating model that keeps AI useful while preserving human authority at consequential moments.

Compare AgenticAI and AugmentedAI →