SERVANTSTACK // INCIDENT INTELLIGENCEFORENSIC IMAGE // VERIFIED FRAME

SS-IR-017 // INCIDENT REPORTDocumented

Optum

Health-Risk Algorithm Cut Extra-Care Referrals for Black Patients by More Than Half

Q: Which governance control was missing?

Named SME review and decision audit trail. A qualified reviewer tests the basis, context, and disparate impact before the decision reaches a person.

EXECUTIVE BRIEF

BIAS & ACCESS HEALTHCARE

FAILURE CHAINTRACE COMPLETE

01TRIGGEROn October 24, 2019, researchers led by Ziad Obermeyer of UC Berkeley published a study in Science showing that a…
02MACHINE ACTIONDecision system
03MISSING GATENamed SME review and decision audit trail
04IMPACTHuman welfare

01 // INCIDENT SUMMARY

The short version

02 // KEY FACTS

Case telemetry

INCIDENT: SS-IR-017
DATE: October 24, 2019
SYSTEM: Optum
LOCATION / SCOPE: United States
EVIDENCE: Documented
AI ROLE: Decision system
HARM: Human welfare
SOURCES: 3 cited records

03ENTRY POINT // WHAT HAPPENED

The event

On October 24, 2019, researchers led by Ziad Obermeyer of UC Berkeley published a study in Science showing that a widely deployed commercial health-risk algorithm systematically underestimated the medical needs of Black patients. Algorithms of this type were applied to roughly 200 million people across the U.S. health system to decide who gets enrolled in high-touch "care management" programs. Analyzing about 50,000 patient records from a large academic hospital, the team found that at any given risk score Black patients were considerably sicker than white patients: those flagged as highest-risk had 26.3 percent more chronic conditions than white patients with the same score. Correcting the bias would have raised the share of Black patients flagged for extra care from 17.7 percent to 46.5 percent -- meaning more than half of the Black patients who should have qualified were silently screened out. Subsequent reporting identified the algorithm as Optum's Impact Pro.

04CAUSAL TRACE // AI'S ACTUAL ROLE

What the machine did

The algorithm predicted future health-care costs and used that cost figure as a proxy for health need. Because the U.S. system historically spends less on Black patients with identical illness burdens (about $1,800 less per year for equivalent chronic conditions), the model read lower spending as lower need and assigned Black patients lower risk scores. No human SME validated that the chosen target variable -- dollars spent -- was a fair stand-in for the thing the program actually cared about: sickness. The proxy was never independently audited against clinical outcomes by race before deployment, and the algorithm's referral decisions ran at population scale with no human review gate to catch that equally sick patients of different races were being treated unequally.

Decision systemAutomation was a causal participant—not a decorative label for the system around it.

05BLAST RADIUS // CONSEQUENCES

Where the failure landed

Black patients who were measurably sicker were denied enrollment in the extra-care programs they qualified for, deepening existing disparities in access to chronic-disease management. The study estimated the bias cut the number of Black patients identified for additional help by more than half. When the researchers retrained the model to predict illness rather than cost, the racial disparity in chronic conditions at each risk score fell by 84 percent. The finding triggered a New York Department of Financial Services and Department of Health inquiry into Optum and prompted broad scrutiny of cost-as-need proxies across the health-tech industry.

06 // EVIDENCE STATUS

Documented

Supported by a first-party disclosure, technical research, or corroborated reporting cited below.

SOURCE RECORD UPDATED 2026-07-09

07 // SOURCE LEDGER

3 cited records

01
Secondary / analysisObermeyer et al., "Dissecting racial bias in an algorithm used to manage the health of populations," Science (Oct 24, 2019)
02
Secondary / analysisScientific American: "Racial Bias Found in a Major Health Care Risk Algorithm"
03
Secondary / analysisScience News: "Bias in a common health care algorithm disproportionately hurts black patients"

08CONTROL FAILURE // MISSING GOVERNANCE

Named SME review and decision audit trail

The failure pattern in this case: Automated judgment without accountable review.

09INTERVENTION POINT // HUMAN IN THE MIDDLE

The moment the path could change

A qualified reviewer tests the basis, context, and disparate impact before the decision reaches a person.

AI PROPOSES→HUMAN OWNS THE DECISION→SYSTEM EXECUTES

10CONTROL DEPLOYMENT // AUTHORITYGATE

SME routing · decision audit trail

The AuthorityGate Operational Resilience framework requires a human SME validation gate on the target variable and the deployment fairness profile before any population-scoring model is allowed to drive care decisions. A clinical SME would have been required to formally sign off that the label being predicted (health-care cost) is a valid proxy for the operational goal (medical need), and a fairness-validation gate would have demanded stratified outcome testing -- risk score versus actual chronic-condition burden, broken out by race -- as a release-blocking artifact. That subgroup test surfaces a 26.3 percent illness gap at equal scores immediately, so the change never ships. AuthorityGate's change-validation gate also re-runs that disparity check on every retrain, so a model that silently regresses on equity cannot reach production without a human SME explicitly accepting the documented disparity.

RELEVANT KEYSTONE CONTROLHuman-in-the-Loop ValidationHow high-risk actions route to a named subject-matter expert who owns the go or no-go decision.

12 // THE ALTERNATIVE

Autonomy is a design choice.

See the operating model that keeps AI useful while preserving human authority at consequential moments.

Compare AgenticAI and AugmentedAI →