In early November 2019, tech entrepreneur David Heinemeier Hansson (creator of Ruby on Rails) posted a viral thread alleging that the new Apple Card, underwritten by Goldman Sachs, offered him a credit limit roughly 20 times higher than his wife's -- despite the couple filing joint tax returns and…
In early November 2019, tech entrepreneur David Heinemeier Hansson (creator of Ruby on Rails) posted a viral thread alleging that the new Apple Card, underwritten by Goldman Sachs, offered him a credit limit roughly 20 times higher than his wife's -- despite the couple filing joint tax returns and his wife having the higher credit score.
Why it matters
The allegations went viral globally and made the Apple Card the highest-profile AI fairness controversy of its moment.
AI / automation’s role
The credit-limit decision was made by an automated underwriting model with no per-decision human in the loop and, critically, no human-defensible explanation attached to its outputs.
On October 24, 2019, researchers led by Ziad Obermeyer of UC Berkeley published a study in Science showing that a widely deployed commercial health-risk algorithm systematically underestimated the medical needs of Black patients.
On October 24, 2019, researchers led by Ziad Obermeyer of UC Berkeley published a study in Science showing that a widely deployed commercial health-risk algorithm systematically underestimated the medical needs of Black patients.
Why it matters
Black patients who were measurably sicker were denied enrollment in the extra-care programs they qualified for, deepening existing disparities in access to chronic-disease management.
AI / automation’s role
The algorithm predicted future health-care costs and used that cost figure as a proxy for health need.
Michigan's Unemployment Insurance Agency deployed MiDAS (Michigan Integrated Data Automated System), an automated fraud detection system that cross-referenced employer and claimant data to flag discrepancies.
Michigan's Unemployment Insurance Agency deployed MiDAS (Michigan Integrated Data Automated System), an automated fraud detection system that cross-referenced employer and claimant data to flag discrepancies.
Why it matters
40,000+ people falsely accused of fraud. $117 million in wrongful penalty assessments.
AI / automation’s role
MiDAS operated for 22 months with zero human review of fraud determinations.
