SERVANTSTACK // INCIDENT INTELLIGENCEFORENSIC IMAGE // VERIFIED FRAMEAI CSAM Classifier Falsely Flagged a Father's Medical Photo, Auto-Disabled His Entire Account, and Reported Him to Police
In February 2021, a San Francisco father identified only as Mark photographed his toddler son's swollen groin at a nurse's request, so a doctor could review the images ahead of a video consultation during the pandemic.
- 01TRIGGERIn February 2021, a San Francisco father identified only as Mark photographed his toddler son's swollen groin at a…
- 02MACHINE ACTIONAutonomous actor
- 03MISSING GATERisk-based SME approval before execution
- 04IMPACTHuman welfare
The short version
In February 2021, a San Francisco father identified only as Mark photographed his toddler son's swollen groin at a nurse's request, so a doctor could review the images ahead of a video consultation during the pandemic.
Case telemetry
- INCIDENT
- SS-IR-028
- DATE
- August 21, 2022
- SYSTEM
- LOCATION / SCOPE
- San Francisco, California, United States
- EVIDENCE
- Reported
- AI ROLE
- Autonomous actor
- HARM
- Human welfare
- SOURCES
- 3 cited records
The event
In February 2021, a San Francisco father identified only as Mark photographed his toddler son's swollen groin at a nurse's request, so a doctor could review the images ahead of a video consultation during the pandemic. The doctor diagnosed an infection, prescribed antibiotics, and the boy recovered. Two days after the photos synced to Google Photos, Google's automated system flagged them as child sexual abuse material (CSAM), and locked Mark out of his entire Google account for a "severe violation" that "might be illegal." He lost his email, contacts, photos, and his Google Fi phone number, which cascaded into him losing access to other accounts. Google automatically filed a report; the San Francisco Police Department opened a "child exploitation" investigation that ran through December 2021. Investigators reviewed the medical context and concluded no crime had occurred, fully clearing him. Google denied his appeals twice -- including after he submitted the police report exonerating him -- and refused to restore the account, stating it stood by its decision. The case was reported by The New York Times on August 21, 2022. Google had flagged 287,368 instances of suspected CSAM in the first half of 2021 alone.
What the machine did
The classifier operated as a fully autonomous detect-and-punish pipeline with no human SME in the loop before consequences landed. An image-recognition model flagged the photo and the system auto-disabled the user's entire account and auto-generated a law-enforcement referral -- all at machine speed, with zero human review of context before the account was killed and police were notified. The model had no concept of clinical intent: it could not distinguish a parent documenting a medical symptom at a doctor's request from abuse, because nothing in the pipeline asked. Crucially, the failure was not just the false positive -- it was the absence of any human override gate afterward. Even after the police cleared him and he submitted the exonerating report, Google's appeal process upheld the algorithm's original verdict twice. The automated decision was treated as ground truth that no human reviewer was empowered or willing to reverse.
Where the failure landed
An innocent father was reported to police as a suspected child predator and placed under a months-long criminal investigation for a medical photo his doctor asked him to take. He permanently lost his Google account: years of email, contacts, photos, and his phone number, with downstream lockouts of other services. Despite being formally cleared by the SFPD, Google refused reinstatement and denied two appeals. The case exposed a systemic pattern -- a nearly identical incident hit a Houston father in the same period -- showing the failure mode was structural, not a one-off. It became a landmark example of automated content moderation inflicting life-altering harm with no accountable human able to undo it.
Reported
Documented in the cited public record. Follow the sources for the precise evidentiary posture.
SOURCE RECORD UPDATED 2026-07-09
3 cited records
- 01
- 02Secondary / analysisGizmodo -- Google Flagged Parents' Photos of Sick Children as Sexual Abuse
- 03
Risk-based SME approval before execution
The failure pattern in this case: High-stakes output had no accountable checkpoint.
The moment the path could change
The appropriate subject-matter expert reviews the evidence, exceptions, and affected people before the output becomes action.
Autonomy is a design choice.
See the operating model that keeps AI useful while preserving human authority at consequential moments.
Compare AgenticAI and AugmentedAI →