Skip to main content
Incident intelligence/SS-IR-005CASE FILE OPEN
Symbolic editorial illustration for SS-IR-005SERVANTSTACK // INCIDENT INTELLIGENCEFORENSIC IMAGE // VERIFIED FRAME
SS-IR-005 // INCIDENT REPORTReported

IBM Watson for Oncology

AI Recommends Unsafe Cancer Treatments

EXECUTIVE BRIEF

MD Anderson Cancer Center partnered with IBM Watson to build an AI system for recommending cancer treatments.

HEALTHCARE
FAILURE CHAINTRACE COMPLETE
  1. 01TRIGGERMD Anderson Cancer Center partnered with IBM Watson to build an AI system for recommending cancer treatments.
  2. 02MACHINE ACTIONMaterial contributor
  3. 03MISSING GATERisk-based SME approval before execution
  4. 04IMPACTHuman welfare
01 // INCIDENT SUMMARY

The short version

MD Anderson Cancer Center partnered with IBM Watson to build an AI system for recommending cancer treatments.

02 // KEY FACTS

Case telemetry

INCIDENT
SS-IR-005
DATE
2013-2017
SYSTEM
IBM Watson for Oncology
LOCATION / SCOPE
Houston, Texas, USA
EVIDENCE
Reported
AI ROLE
Material contributor
HARM
Human welfare
SOURCES
1 cited record
03ENTRY POINT // WHAT HAPPENED

The event

MD Anderson Cancer Center partnered with IBM Watson to build an AI system for recommending cancer treatments. After spending $62 million, internal audits revealed Watson for Oncology was recommending treatments that were unsafe and contradicted established oncology guidelines. In one documented case, Watson recommended giving a cancer patient with severe bleeding a drug that would worsen the bleeding. The system was trained primarily on synthetic cases created by a small group of doctors at Memorial Sloan Kettering, not on real patient outcomes.

04CAUSAL TRACE // AI'S ACTUAL ROLE

What the machine did

Watson for Oncology was marketed as an AI that could process millions of medical papers and recommend evidence-based treatments. In reality, the system's recommendations were largely based on the opinions of a handful of MSK physicians encoded as training data. When deployed in hospitals across Asia and Latin America, Watson recommended treatments based on American insurance formularies that were unavailable or inappropriate in those countries.

Material contributorAutomation was a causal participant—not a decorative label for the system around it.
05BLAST RADIUS // CONSEQUENCES

Where the failure landed

$62 million spent by MD Anderson with no usable system delivered. The project was cancelled. Hospitals worldwide that had adopted Watson for Oncology found its recommendations unreliable. STAT News documented multiple instances of unsafe treatment suggestions. IBM eventually sold Watson Health for a fraction of its investment.

06 // EVIDENCE STATUS

Reported

Documented in the cited public record. Follow the sources for the precise evidentiary posture.

SOURCE RECORD UPDATED 2026-07-09

07 // SOURCE LEDGER

1 cited record

  1. 01
08CONTROL FAILURE // MISSING GOVERNANCE

Risk-based SME approval before execution

The failure pattern in this case: High-stakes output had no accountable checkpoint.

09INTERVENTION POINT // HUMAN IN THE MIDDLE

The moment the path could change

The appropriate subject-matter expert reviews the evidence, exceptions, and affected people before the output becomes action.

AI PROPOSES→HUMAN OWNS THE DECISION→SYSTEM EXECUTES
10CONTROL DEPLOYMENT // AUTHORITYGATE

Risk routing · named approval · audit trail

AuthorityGate mandates that AI treatment recommendations in healthcare must be reviewed by the treating oncologist before reaching the patient. The framework also requires validation against established clinical guidelines (NCCN, ASCO) - not just training data from a single institution. An AI that recommends a bleeding-risk drug for a patient who is already bleeding would be caught by any oncologist in seconds.

RELEVANT KEYSTONE CONTROLHuman-in-the-Loop ValidationHow high-risk actions route to a named subject-matter expert who owns the go or no-go decision.
12 // THE ALTERNATIVE

Autonomy is a design choice.

See the operating model that keeps AI useful while preserving human authority at consequential moments.

Compare AgenticAI and AugmentedAI →