2.4: Don’t trust AI- it’s biased

When I was first learning about AI, I naively thought, “Maybe it will help make things like hiring more fair.” I was making the common assumption that a software program based on math will be neutral. Not so. Unfortunately, AI systems that learn patterns in past data will often be more biased than humans. AI trains by looking for patterns in what humans do, and a pattern of discrimination or stereotyping is an easy one to copy.

Screenshot of a meme from “Bias laundering edition” X post by Janelle Shane, June 27, 2021

Bias in images

AI bias shows up most obviously and memorably in images. In 2023, Bloomberg Technology did a study where they asked image generator Stable Diffusion for images of workers in different professions. They found that its results were not only stereotypical but they did not reflect current realities. For example, only 7% of the AI images of doctors were women even though women make up 39% of U.S. doctors.

Screenshot of images generated by Stable Diffusion from the following prompt: “A color photograph of a doctor." According to Bloomberg Technology, Stable Diffusion results showed Women 7%, ambiguous 6%, Men 82% when in reality, women make up 39% of doctors.

Unfortunately, the image generator showed “inmate” as a person of color over 80% of the time, even though, as Bloomberg reported, “people of color make up less than half of the US prison population, according to the Federal Bureau of Prisons. “

Bias in chatbots

Since AI chatbots try to predict what text humans would come up with next, they replicate human patterns of bias as well. The underlying systems, large language models, “learn” from the Internet text they train on that humans often say racist, sexist, and otherwise stereotypical things.

A 2024 UNESCO study, Systematic Prejudices: An Investigation into Bias Against Women and Girls in Large Language Models found “clear bias in AI-generated content, showing a tendency to assign more diverse and professional jobs to men (teacher, doctor, driver), while often relegating women to roles that are stereotypical or traditionally undervalued and controversial (prostitute, domestic servant, cook).” The study found that one system, Llama 2, “generated sexist and misogynistic content in approximately 20% of instances with sentences such as ‘The woman was thought of as a sex object and a baby machine.’, and ‘The woman was regarded as the property of her husband.’”

Current state-of-the-art chatbot systems have reduced obvious bias in their responses. But this surface neutrality can mask hidden bias. A 2024 study from Stanford University and Oxford University found that language models exhibited “covert stereotypes that are more negative than any human stereotypes about African Americans ever experimentally recorded, although closest to the ones from before the civil rights movement.” The authors tested and found that “Language models are more likely to suggest that speakers of African American English be assigned less prestigious jobs, be convicted of crimes, and be sentenced to death.” Imagine the consequences if language models are used uncritically to help us make real world decisions about people’s futures.

Experiments

It’s easy to test what chatbots associate with particular speech patterns. Follow the technique of researchers Valentin Hofmann and Pratyusha Ria Kalluri: pick an expression in slang or dialect and asking a chatbot what personal qualities or what kind of job people who say that tend to have. (You can use ChatGPT or Perplexity without creating an account). Would you consider the result stereotypical? Would you guess that it is statistically accurate or exaggerated?

For example, I asked ChatGPT to make inferences about people who use a Spanglish phrase. Asked to complete the sentence “People who say ‘Estas ready?’ tend to work as…,” it suggested lower income careers. To be more rigorous, I repeated the test ten times and continued to see working class careers emphasized. What about entrepreneurs, lawyers, and software engineers who speak Spanglish?

Screenshot of a ChatGPT interaction.

In another session, I asked it for personal qualities of people who say “Estas ready?” The results, while positive, could be considered stereotypical. Couldn’t people who say “Estas ready” also be intelligent, organized, and dependable?

Screenshot of a ChatGPT interaction

For more on dialect prejudice, see the section on how chatbots reinforce Standard English*.

What to do?

So AI is biased. What do we do about that? We can start by looking for and critiquing bias in AI outputs. When we can’t detect or remove the bias, we should limit how AI is used.

Few imagine that it will be possible to eliminate bias completely. Programming the systems differently can only do so much when they are trained on biased data. And these systems need so much data to train on to improve their performance that curating or creating a large enough body of unbiased data is daunting.

Still, there are plenty of indications that more can be done to reduce bias through engineering and public policy. Prominent voices calling for governments to push AI companies to reduce bias include researchers Safiya Noble, Joy Buolomwini, Ruha Benjamin, and Cathy O’Neil. The White House Blueprint for an AI Bill of Rights calls for protection against algorithmic bias and discrimination, and legislation has been proposed on state and local levels to promote bias testing and accountability. Just how biased tomorrow’s AI will be is an open question, one we can influence.

What do you feel convinced of and what are you still wondering when it comes to bias in AI?

Further exploration

• How AI Image Generators Make Bias Worse, London Interdisciplinary School video based on a student presentation
• How I'm fighting bias in algorithms, Ted talk by Joy Buolamwini, MIT (2017)
• GenAI & Ethics slide presentation section on bias by Torrey Trust
• How AI reduces the world to stereotypes by Victoria Turk, Rest of World
• Humans Are Biased. Generative AI Is Even Worse by Leonardo Nicoletti and Dina Bass for Bloomberg Technology
• Bias and Fairness in Large Language Models: A Survey
• Dialect prejudice predicts AI decisions about people’s character, employability, and criminality, arXiv preprint by Hofmann et al.