About This Unit

New technologies arrive wrapped in the language of neutrality: the algorithm doesn't see race, the model just follows the data, the camera records what's there. This unit examines why automated systems so reliably reproduce — and often amplify and disguise — existing racial hierarchies. Ruha Benjamin names the pattern "the New Jim Code": designs that encode inequity while appearing more objective than the discriminatory systems they replace. Safiya Noble's study of search engines and Joy Buolamwini's audits of facial recognition supply the unit's evidentiary core: commercial systems that sexualized Black girls in search results, and face-analysis products that failed darkest-skinned women at rates approaching fifty percent while classifying white men nearly perfectly.

The unit refuses both of the stories technology likes to tell about itself — savior and slayer — and asks the sociological questions instead: who designs, with what defaults, trained on what pasts, audited by whom, and deployed on whom first? Sasha Costanza-Chock's design justice framework moves the analysis from critique to practice, documenting how communities most affected by harmful systems are leading their redesign. Learners leave the unit able to interrogate any automated system — a risk score, a hiring screen, a camera at the door — as a social institution with a history, rather than a verdict from nowhere.

Learning Objectives

By the end of this unit, learners will be able to:

• Explain how automated systems reproduce racial inequality through training data, design defaults, proxy variables, and deployment choices, using documented cases.
• Define the New Jim Code and analyze why algorithmic discrimination can be harder to detect and contest than its human predecessors.
• Evaluate an algorithmic audit — its method, findings, and the institutional response — using Gender Shades as the model case.
• Apply design justice principles to propose how an existing system would differ if the communities most affected led its design.

Keywords

• Algorithmic bias
• The New Jim Code
• The coded gaze
• Design justice
• Proxy discrimination
• Techno-solutionism
• Algorithmic justice
• Digital redlining

Open Educational Resources

• Benjamin, Ruha. "Is Technology Our Savior — or Our Slayer?" TED Talk, 2023. Watch the talk.

Talk (video, ~12 min, with transcript) · Open access.

In this talk, Ruha Benjamin rejects the two futures most often imagined for technology — Silicon Valley's promise of liberation and Hollywood's vision of machine catastrophe — arguing that both are stories told by the same narrow set of people who control the industries that produce them. Against these, she offers what she calls "ustopias": futures imagined and built from the ground up, in which technology is directed toward ordinary needs — health care, housing, a voice in collective decisions — rather than toward profit or control. She illustrates the alternative through participatory projects, including Barcelona's Decidim platform for democratic input and community initiatives in Atlanta. For the unit, the talk sets the framing question that the rest of the readings pursue in detail: the issue is not whether machines are good or bad in themselves, but who gets to design them, toward whose ends, and who is shut out of that imagining.

• Noble, Safiya Umoja. "Algorithms of Oppression." Databite No. 109, Data & Society Research Institute, 2018. Watch the talk.

Recorded talk (video, ~60 min) · Open access.

A recorded talk in which Safiya Noble presents the central findings of her book on commercial search engines. She shows how Google's search — an advertising platform built to sell attention, not a neutral library — once returned pornography as the top results for the query "Black girls," auto-suggested degrading associations for women and girls of color, and elevated racist sources into positions of apparent authority. Her argument is that the familiar defense, "the algorithm did it," obscures a chain of corporate choices about what ranking is designed to optimize and whose interests it serves. Delivered as a talk, the material preserves Noble's specific examples in full, letting learners see the results she analyzes rather than only read about them. For the unit, the talk establishes search as a racial institution and models a way of working that recurs across the readings: the scholar as a public witness, documenting harm in concrete, demonstrable terms.

• Buolamwini, Joy, and Timnit Gebru. Gender Shades (project site). MIT Media Lab, 2018. Explore the project.

Research project site (interactive, with video and paper) · Open access.

The project site for the audit that forced commercial vendors to revise their claims. Joy Buolamwini and Timnit Gebru tested facial-analysis systems from IBM, Microsoft, and the company Face++ on a benchmark dataset they constructed to be balanced across skin type and gender — correcting for the lighter-skinned, male-skewed datasets on which such systems were typically evaluated. They found that error rates reached as high as 34.7 percent for darker-skinned women while remaining below one percent for lighter-skinned men. The site presents the method, the findings, and the companies' responses in an accessible, interactive form. For the unit, it is a model of how algorithmic bias is established rather than merely asserted: through a replicable test, a transparent benchmark, and published numbers that a vendor can be held to. It gives learners a concrete instance of an audit and prepares the way for the unit's own work evaluating one.

• Angwin, Julia, Jeff Larson, Surya Mattu, and Lauren Kirchner. "Machine Bias." ProPublica, May 2016. Read the investigation.

Investigative data journalism · Open access.

The investigation that brought algorithmic risk scoring into public debate. Examining COMPAS, a proprietary tool that scores criminal defendants by their predicted likelihood of reoffending and is used in bail and sentencing decisions, the ProPublica team found that Black defendants were nearly twice as likely as white defendants to be wrongly flagged as high-risk, while white defendants were more often wrongly rated low-risk. The reporters published their methodology and data openly, which set off a sustained scholarly argument about competing mathematical definitions of fairness — whether a fair tool should equalize error rates across groups or equalize the meaning of a given score, and the proof that it cannot generally do both at once when groups have different underlying rates. For the unit, the piece works on two levels: as evidence of racial disparity in a deployed system, and as the opening of a still-unresolved question about what calling a system "unbiased" could even mean.

Key Scholarship

• Buolamwini, Joy. Unmasking AI: My Mission to Protect What Is Human in a World of Machines. New York: Random House, 2023. Publisher page.

Book.

Buolamwini's first-person account of the work behind the Gender Shades audit and what followed from it. She begins with the moment that gave the unit its key image — a face-tracking system that would not register her dark-skinned face until she put on a white mask — which she names the "coded gaze," the way the assumptions of a system's builders get encoded into who it does and does not see. The book follows the research through congressional testimony, pushback from the companies whose products she tested, and the founding of the Algorithmic Justice League, the organization she built to carry the work forward. Along the way it shows how technical audit, public art, and policy advocacy can reinforce one another. For the unit, it supplies the human and institutional story behind the audit learners examine: how a single finding, rigorously established, can be turned into testimony, organization, and pressure for change.

• Adib-Moghaddam, Arshin. Is Artificial Intelligence Racist? The Ethics of AI and the Future of Humanity. London: Bloomsbury, 2023. Publisher page.

Book.

Adib-Moghaddam places algorithmic racism within a much longer history of classificatory thinking — the pseudo-scientific schemes of racial hierarchy developed over centuries — and argues that artificial intelligence inherits not only biased datasets but the habits of mind that race science established. On this account, AI systems are a new vehicle for an old logic of sorting and ranking human beings, made more powerful by their global reach: systems trained largely within a small number of countries are deployed across very different societies, exporting particular assumptions about difference. Written from outside the United States and in a more philosophical register than much of the field, the book widens the frame beyond the American cases that dominate the literature, asking about AI, ethics, and human futures at a planetary scale. For the unit, it supplies the deep historical and global context for the specific systems the other readings examine, connecting today's automated discrimination to the longer career of scientific racism.

• Costanza-Chock, Sasha. Design Justice: Community-Led Practices to Build the Worlds We Need. Cambridge, MA: MIT Press, 2020. Read the open-access book.

Book · Open access (entire book).

Costanza-Chock opens from a personal scene — the millimeter-wave scanners at airport security, which flag the author's nonbinary body as an anomaly to be searched, a recurring experience they tag #TravelingWhileTrans — to show how "universal" design encodes the assumptions of those it treats as the default user and harms those who fall outside it. From this critique the book turns to construction: it documents design justice, a set of practices in which the communities most affected by a technology lead its design rather than receiving it. Drawing on examples from community-run hackathons to technology cooperatives, Costanza-Chock lays out principles for building systems accountable to the people they affect. For the unit, the book is the pivot from diagnosis to practice — it converts the other readings' critiques into a concrete program for doing things differently — and, released in a free and complete open-access edition, it is among the most readily available of the unit's longer works.

• Benjamin, Ruha. Race After Technology: Abolitionist Tools for the New Jim Code. Cambridge: Polity, 2019. Publisher page.

Book.

Benjamin's book introduces the New Jim Code, the concept at the center of the unit: the way technologies marketed as objective, or even as socially beneficial, can deepen the racial hierarchies they claim to leave behind. She distinguishes several forms this takes — inequity deliberately engineered into a system, discrimination that follows from neglect and default settings, the exposure produced when systems make racialized people hyper-visible, and the "techno-benevolence" of tools sold as fixes for bias that entrench it instead. Her examples range from automatic soap dispensers that fail to register dark skin to predictive policing software, showing the same logic at very different scales. The book closes by turning toward what Benjamin calls abolitionist tools — design and organizing oriented toward justice rather than mere efficiency — which connects the unit to a larger argument running through the project: that technology is a terrain of political struggle, not a neutral verdict handed down from outside society.

• Noble, Safiya Umoja. Algorithms of Oppression: How Search Engines Reinforce Racism. New York: NYU Press, 2018. Publisher page.

Book.

Noble's book was among the first to treat a commercial search engine as a racial institution rather than a neutral tool. She documents how Google's ranking system — an advertising operation optimized for revenue — generated degrading and pornographic results for searches about Black women and girls, misrepresented communities both to themselves and to others, and lent the appearance of authority to racist sources. Her central argument is that what she calls data discrimination is not an occasional glitch but a structural feature of systems built to serve advertisers, and that addressing it is a civil-rights matter requiring public-interest alternatives to purely commercial information platforms. The book's detailed examples — the specific searches and the results they returned — remain among the clearest demonstrations in the field of how an ostensibly neutral system encodes and amplifies racial meaning. For the unit, Noble establishes the analytic that the other readings extend: that the design and business model of a system, not only its data, produce its racial effects.

Case Studies

• The Coded Gaze on Trial: Gender Shades and the Wrongful Arrests

The Gender Shades audit and documented wrongful arrests from facial-recognition misidentification; ACLU case materials · Audit-analysis activity.

A two-part audit-analysis activity. Learners first work through the Gender Shades project — its benchmark dataset, its error-rate findings, and the vendors' responses — to evaluate the audit as a method for establishing bias; they then turn to the consequences in deployment, examining documented cases in which Black Americans were wrongfully arrested after false facial-recognition matches, using the ACLU's case materials. The pairing traces the full chain from training data and uneven error rates through a deployment decision to a person in handcuffs.

• Auditing the Algorithm Yourself: COMPAS and the Fairness Wars

The ProPublica COMPAS investigation and the company's rebuttal; competing fairness definitions · Data-reasoning activity.

A data-reasoning activity built on the ProPublica investigation of the COMPAS risk score and the company's published rebuttal. Working only from the published tables — no coding required — learners examine two competing definitions of fairness (equal false-positive rates across groups versus equal predictive meaning of a score), work through why both cannot hold at once when groups have different base rates, and debate which a court should require. The activity also extends to questions of algorithmic transparency and accountability in public decision-making.

Questions for Reflection

1. On Benjamin: The New Jim Code claims that automated discrimination can be worse than what it replaces precisely because it appears neutral. Explain the mechanism — what does the appearance of objectivity do to a person's ability to contest a decision? Use one concrete system in your answer.
2. On Noble: Noble insists a search engine is an advertising platform, not a library. How does that reframing change responsibility for what results return? Run a search relevant to a community you belong to and analyze what the ranking serves.
3. On Buolamwini: Gender Shades succeeded partly because it was an audit — replicable method, balanced benchmark, published numbers — rather than testimony alone. What does that say about what institutions count as evidence of discrimination, and who bears the burden of producing it?
4. On Costanza-Chock: Take one system you use weekly and answer design justice's opening questions about it: who designed it, who was the default user, who bears its failures, and what would change if the people most harmed led a redesign?
5. On Adib-Moghaddam: Adib-Moghaddam argues that AI inherits not only biased data but the long history of race science — centuries of classifying and ranking human beings. What does it add to your understanding of a biased system to see it as the latest form of an old logic rather than a new technical mistake? And what might his vantage — writing from outside the United States, about systems trained in a few countries and deployed everywhere — reveal that the American cases do not?
6. Synthesis across the unit: "The algorithm is biased because the data is biased" is the standard explanation. Drawing on at least two of the unit's readings, explain what that sentence gets right, what it conveniently omits — choices about objectives, deployment, and profit — and who benefits from the omission.

Further Readings and Resources

• Broussard, Meredith. More Than a Glitch: Confronting Race, Gender, and Ability Bias in Tech. Cambridge, MA: MIT Press, 2023. Publisher page.

Book.

Broussard, a data journalist and researcher, argues that bias in technology is not an accident to be patched but a predictable result of how systems are built. Surveying cases across race, gender, and disability — in areas from medicine to education to policing — she makes the case that technical fixes alone cannot resolve what are fundamentally social problems. A recent and accessible survey, the book extends the unit's analysis to disability and to the limits of "debiasing" as a strategy.

• Coded Bias. Directed by Shalini Kantayya, 2020. Visit the film site.

Documentary film.

A documentary that follows Joy Buolamwini from her discovery of the coded gaze through her research, her testimony before Congress, and the broader movement against unaccountable facial recognition. Built around the work the unit's readings describe, the film puts faces and voices to the audit and its stakes, and widens the lens to surveillance and automated decision-making internationally. It offers a narrative entry point to the unit's central case for viewers who want to see the story unfold.

• Benjamin, Ruha, ed. Captivating Technology: Race, Carceral Technoscience, and Liberatory Imagination in Everyday Life. Durham: Duke University Press, 2019. Publisher page.

Edited volume.

An edited collection that connects surveillance and data technologies to the carceral systems examined elsewhere in the project — policing, prisons, and the broader apparatus of monitoring and control. Bringing together scholars across fields, the volume treats technology as "carceral technoscience," tracing how tools of tracking and prediction extend the reach of punishment, while also gathering work on imagining liberatory alternatives. It deepens the connection between automated systems and the machinery of racialized control.

• Buolamwini, Joy, and Timnit Gebru. "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification." Proceedings of Machine Learning Research 81 (2018): 1–15. Read the paper.

Conference paper (peer-reviewed).

The peer-reviewed paper behind the Gender Shades project, presenting the audit's method and results in full technical detail: the construction of the skin-type-and-gender-balanced benchmark, the testing of three commercial systems, and the disaggregated error rates that revealed the largest failures for darker-skinned women. For learners who want to see exactly how the finding was produced and could be replicated, this is the formal scholarly version of the unit's project-site reading.

• Eubanks, Virginia. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. New York: St. Martin's Press, 2018. Publisher page.

Book.

Eubanks examines how automated systems sort, monitor, and discipline poor people across the American welfare state — algorithmic welfare-eligibility determinations, a coordinated-entry system for allocating homeless services, and a predictive model used in child-welfare investigations. Through close case studies she shows how tools sold as neutral and efficient concentrate scrutiny and penalty on the poor. The book brings the dimension of class and poverty to the unit, showing how automated decision-making reshapes access to basic support.

• O'Neil, Cathy. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. New York: Crown, 2016. Publisher page.

Book.

O'Neil, a mathematician, offers a widely read introduction to the harms of opaque predictive models — what she calls weapons of math destruction: systems that are opaque, deployed at scale, and damaging in their effects, from credit scoring to teacher evaluation to predatory advertising. Written for a general audience, the book is a useful starting point for learners newer to the quantitative side of the unit, naming the recurring features that make a model dangerous.

• Algorithmic Justice League. Visit the site.

Organization.

The organization Joy Buolamwini founded to combine research, art, and policy advocacy against harmful and unaccountable AI. Its site collects ongoing audits, campaigns, and resources, and documents the kind of organized response the unit's readings call for. For the unit, it is a window onto algorithmic-justice work as a continuing practice — what it looks like to move from establishing a harm to building pressure for accountability and change.

• Better Images of AI (image library and guide). Explore the library.

Image library and guide.

An open project that rethinks how artificial intelligence is pictured, offering a freely licensed library of alternatives to the glowing-blue-robot and disembodied-brain images that dominate AI coverage and that subtly shape how the public imagines these systems. The image used for this module's cover — Alyssa Chen's "Facial Recognition," licensed CC BY 4.0 — comes from this library, making the politics of representation a small case learners can see on the page in front of them.