Why does AI sometimes treat people unfairly?

The full story

The episode's narration, word for word. Numbers link to the sources below; timestamps play that moment.

The question 0:00

Last time, fakes made to fool people. This time, real systems, and the question of who they work for. In 2017, researchers tested three commercial systems that label faces male or female ¹. For lighter-skinned men, the worst error rate was under one per cent ¹. For darker-skinned women, it reached nearly thirty-five per cent ¹. So why does AI sometimes treat people unfairly?

Three ways in 0:30

In 2020, Australia's Human Rights Commission wrote that bias can come from the data a tool learns from, from its design, or from inequality already in society ². So ask three questions. Which examples did it learn from? What counts as a right answer? And what was it told to aim for?

A résumé engine 0:54

In October 2018, Reuters reported, from five people familiar with it, that from 2014 an Amazon team had built software to score job applicants from one to five stars ³. By 2015, the company realised it wasn't rating candidates for technical jobs in a gender-neutral way ³. Amazon declined to comment on its problems, but said its recruiters never used it to evaluate candidates; the sources said recruiters saw its recommendations but never relied on them alone ³.

It learned the past 1:29

Why would software prefer men? It learned from résumés sent to the company over ten years, and most came from men ³. So, in effect, it taught itself that male candidates were preferable ³. It marked down résumés containing the word women's, as in women's chess club captain, the sources said ³. Engineers edited it to be neutral to such terms, but that was no guarantee it wouldn't find other ways of sorting candidates that could prove discriminatory ³.

Faces from six parliaments 2:01

Joy Buolamwini, at the MIT Media Lab, and Timnit Gebru, then at Microsoft Research, found that two existing benchmark sets of faces were about eighty and eighty-six per cent lighter-skinned ¹. So they built a balanced set: 1,270 members of parliament from three African and three European countries, sorted by a dermatologist's six-point skin-type scale ¹. In April and May 2017, they tested classifiers from Microsoft, IBM and the Chinese company Face Plus Plus ¹.

Who it got wrong 2:43

Every system did better on men than on women, and on lighter faces than darker ones ¹. Each system got under one per cent of lighter-skinned men wrong, but between a fifth and a third of darker-skinned women ¹. This measured one narrow task, applying a gender label, not recognising anyone ¹. The authors cautioned that skin tone alone might not explain it; it may go with features less represented in the training data ¹.

The wrong question 3:12

Sometimes the question is the problem. In 2019, Ziad Obermeyer and colleagues studied a commercial algorithm that US health systems use to choose patients for extra care ⁴. In one large hospital's records from 2013 to 2015, at the same risk score, patients who identified as Black were considerably sicker than those who identified as white ⁴. Correcting that would raise Black patients' share of those picked automatically from about eighteen per cent to forty-seven ⁴.

Cost isn't need 3:47

The algorithm didn't predict how ill someone would be; it predicted their future health-care costs ⁴. Because of unequal access to care, less money was spent on Black patients: at the same number of chronic illnesses, about $1,801 a year less ⁴. So the same need earned a lower score, even though the algorithm left race out entirely ⁴. The authors' lesson: a convenient stand-in for what you really care about can be a big source of bias ⁴.

A score in Florida 4:21

In Broward County, Florida, people arrested and awaiting trial were given a risk score from one to ten by a tool called COMPAS, made by Northpointe, mainly to help decide who could be released before trial ^5,6. It draws on a hundred and thirty-seven questions, and race isn't one of them ⁶.

ProPublica counts the mistakes 4:42

In May 2016, the newsroom ProPublica analysed scores from 2013 and 2014, checking who was arrested again within two years, and counting a medium or high score as higher risk ⁵. Of Black defendants who weren't re-arrested, forty-five per cent had been labelled higher risk; of white defendants, twenty-three per cent ⁵. Of people who were re-arrested, forty-eight per cent of white defendants had been labelled low risk, against twenty-eight per cent of Black defendants ⁵. ProPublica called it biased against Black people ⁶.

Northpointe replies 5:20

In July 2016, Northpointe's researchers, William Dieterich, Christina Mendoza and Tim Brennan, strongly rejected the claim, using the same data ⁷. Look instead, they said, at the people labelled higher risk ⁷. Sixty-three per cent of those Black defendants were re-arrested, and fifty-nine per cent of those white defendants ^5,7. So, they argued, the label meant the same for both groups, and the score ranked people equally well in both ⁷. ProPublica stood by its conclusions: it said it had allowed for the different re-arrest rates, and that counting only high scores widened the gap ⁸.

One table, two questions 6:04

Both sides read the same table, and both did the sums right ^5,7. Split people by what happened, and ask how often each group got the wrong label: that's ProPublica's question ⁵. Split them by label, and ask how often each label came true: that's Northpointe's ⁷. Recomputed from ProPublica's counts, every rate matches what each side reported ^5,7. In this sample, fifty-one per cent of Black defendants and thirty-nine per cent of white defendants were re-arrested within two years ⁷.

You can't have both 6:38

In 2016, Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan proved that three reasonable fairness conditions can't all hold at once ⁹. In 2017, Alexandra Chouldechova showed the same clash for the measures in the COMPAS debate ¹⁰. The only escapes are a tool that predicts perfectly, or groups with the same base rate: the same share of people going on to have the outcome ^9,10.

Two groups of a hundred 7:11

A toy version, with made-up numbers. In group A, fifty of a hundred people will have the outcome; in group B, twenty of a hundred. Build a tool that's equally good for both: it catches seventy per cent of the people who'll have the outcome, and seventy per cent of the people it flags really do. In group A, fifteen of the fifty people without the outcome get flagged anyway: thirty per cent. In group B, it's six of eighty, seven and a half per cent.

Pick one 7:45

Now make the false alarms equal instead, twenty per cent each. A flag now means different things: in group A, seventy-eight per cent of flagged people really have the outcome, and in group B, forty-seven per cent. That's the theorem in miniature: when base rates differ and prediction isn't perfect, you have to choose which kind of fairness to give up ^9,10. The maths shows the trade-off; choosing is a question of values.

Robodebt 8:15

Australia has its own lesson, and it wasn't AI ¹¹. From 2015, the federal Robodebt scheme compared welfare records with yearly income reported to the tax office ¹². Unless a person explained a mismatch, it spread that income evenly across fortnights, often over the whole year, and raised a debt wherever the average exceeded what they could earn before their payment was cut ¹². In July 2023, the Royal Commission was explicit: it did not use AI, but a rigid system of automated business rules ¹¹. It called Robodebt a crude and cruel mechanism, neither fair nor legal ¹².

An average isn't a person 8:59

Here's why averaging fails people with irregular work. Imagine someone who earns two thousand dollars a fortnight for ten fortnights, then goes on benefits, earning nothing, for the other sixteen. Their twenty thousand dollars, spread evenly, becomes about seven hundred and seventy dollars every fortnight, including the sixteen when they earned nothing. Averaging puts about twelve thousand dollars of income into fortnights when they earned nothing. In 2020, the government agreed to refund about 381,000 people ¹².

What Australia did next 9:36

The Commission recommended the government consider a consistent legal framework for automated government decisions, with a path to review and algorithms open to expert scrutiny, plus a body to audit them for fairness and bias ¹². The government accepted both in November 2023, and in March 2026 listed them as ongoing ^13,14. Australia's 2019 AI Ethics Principles said AI shouldn't result in unfair discrimination, but they were voluntary, and in October 2025 were folded into newer guidance ¹⁵.

Audits help 10:13

Can unfairness be fixed? Partly. In August 2018, Deborah Raji and Joy Buolamwini re-tested the three updated face systems ¹⁶. Errors for darker-skinned women had fallen: Microsoft's to about one and a half per cent, Face Plus Plus's to about four, and IBM's to about seventeen ¹⁶. And when the health algorithm was changed to predict a mix of health and cost, one measure of its bias fell by eighty-four per cent ⁴.

Where fixes run out 10:47

Leaving race or gender out isn't enough: Northpointe's co-founder Tim Brennan told ProPublica it's hard to build a score without items correlated with race, like poverty and joblessness, and that leaving them out lowers accuracy ⁶. An audit checks only what it measures ^1,16. And no fix escapes the theorem: someone still chooses which fairness counts ^9,10. Australia's Human Rights Commission adds that algorithmic bias can amount to unlawful discrimination ².

Made by people 11:20

So unfairness can come from a lopsided past, the wrong target, or a fairness someone has to choose. In every case today, the choices that mattered were made by people: which data, which target, which kind of fair. Next time: not how AI judges us, but what it makes. Can AI be creative?

Sources

Every factual claim in the episode is tied to one of these. Spotted an error? Tell us.

1. J. Buolamwini, T. Gebru, "Gender Shades: Intersectional Accuracy Disparities in… — proceedings.mlr.press
2. Australian Human Rights Commission, "Using artificial intelligence to make decisions:… — humanrights.gov.au
3. J. Dastin, "Amazon scraps secret AI recruiting tool that showed bias against women",… — reuters.com
4. Z. Obermeyer, B. Powers, C. Vogeli, S. Mullainathan, "Dissecting racial bias in an… — ftc.gov
5. J. Larson, S. Mattu, L. Kirchner, J. Angwin, "How We Analyzed the COMPAS Recidivism… — propublica.org
6. J. Angwin, J. Larson, S. Mattu, L. Kirchner, "Machine Bias", ProPublica (23 May 2016) — propublica.org
7. W. Dieterich, C. Mendoza, T. Brennan, "COMPAS Risk Scales: Demonstrating Accuracy… — go.volarisgroup.com
8. J. Larson, J. Angwin, "Technical Response to Northpointe", ProPublica (29 July 2016) — propublica.org
9. J. Kleinberg, S. Mullainathan, M. Raghavan, "Inherent Trade-Offs in the Fair… — arxiv.org
10. A. Chouldechova, "Fair prediction with disparate impact: A study of bias in recidivism… — arxiv.org
11. Royal Commission into the Robodebt Scheme, Report, Volume 2, Chapter 17 "Automated… — robodebt.royalcommission.gov.au
12. Royal Commission into the Robodebt Scheme, Report, Volume 1 (Commissioner Catherine… — robodebt.royalcommission.gov.au
13. Australian Government, "Government Response: Royal Commission into the Robodebt… — pmc.gov.au
14. Australian Government, "Robodebt Royal Commission – Implementation Update (March… — pmc.gov.au
15. Department of Industry, Science and Resources, "Australia's AI Ethics Principles"… — industry.gov.au
16. I. D. Raji, J. Buolamwini, "Actionable Auditing: Investigating the Impact of Publicly… — dam-prod.media.mit.edu

Researched and scripted with AI assistance, fact-checked claim by claim, with synthetic narration and diagrams drawn in code. How we make episodes.