AI Has Biases Too: Are You a Good Person or a Villain in the Machine's Eyes?
-
Artificial intelligence learns from humans, and humans are biased creatures.
Recently, a Ph.D. student at MIT discovered in two independent studies that while machines excel at identifying AI-generated text, they struggle to distinguish between what's real and what's fake. The reason lies in the fact that the databases used to train machines to detect fake news are filled with human biases, and thus, the resulting AI inevitably inherits these stereotypes.
Human bias is a pervasive issue in the field of artificial intelligence. The digital art project ImageNetRoulette revealed this serious problem by using AI to analyze and describe images uploaded by users. This article exclusively compiles The New York Times' commentary on the ImageNetRoulette project, presenting the 'invisible biases' behind artificial intelligence.
One morning, while browsing Twitter, user Tabong Kima came across a trending hashtag: #ImageNetRoulette. In this trend, users uploaded selfies to a website where AI analyzed and described each face it saw. ImageNetRoulette was one such site, labeling a man as an 'orphan' or a 'non-smoker,' or tagging someone wearing glasses as a 'nerd, idiot, or freak.'
A Twitter user uploaded their photo, which the AI identified as a 'rape suspect,' with the label appearing in the top-left corner of the image.
Among the tweets Kima saw, some labels were accurate, some were odd, and others were downright absurd—all seemingly for laughs. So, he joined in. But the result left the 24-year-old African American deeply unhappy—his smiling photo was labeled as a 'wrongdoer' and 'criminal.'
'Maybe I just don’t get the humor,' he tweeted, 'but I don’t find this funny at all.'
Note: As of publication, the website imagenet-roulette.paglen.com has been taken down and now redirects to www.excavating.ai. The latter hosts an article written by the project's founders titled 'Excavating AI: The Politics of Images in Machine Learning Training Sets.'
In fact, Kima's reaction was precisely what the website aimed to provoke. ImageNetRoulette is a digital art project that seeks to expose the bizarre, unfounded, and offensive behaviors creeping into AI technologies—including facial recognition services widely used by internet companies, law enforcement, and other government agencies—as AI rapidly transforms personal lives.
Facial recognition and other AI technologies learn their skills by analyzing vast amounts of data sourced from past websites and academic projects, inevitably inheriting subtle biases and flaws that have gone unnoticed for years. This is why American artist Trevor Paglen and Microsoft researcher Kate Crawford launched ImageNetRoulette—to expose this issue on a deeper level.
'We wanted to show how biases, racism, and misogyny transfer from one system to another,' Paglen said in a phone interview. 'The point is to help people understand the behind-the-scenes operations and see how our information has been processed and categorized all along.'
As part of an exhibition at Milan's Fondazione Prada museum this week, the website focuses on the well-known large-scale visual database ImageNet. In 2007, researchers led by Fei-Fei Li began discussing the ImageNet project, which played a crucial role in the rise of 'deep learning'—a technology enabling machines to recognize images, including human faces.
The 'Training Humans' photography exhibition debuted at Milan's Fondazione Prada museum, showcasing how AI systems are trained to see and categorize the world.
ImageNet compiles over 14 million photos extracted from the internet, exploring a method to train AI systems and evaluate their accuracy. By analyzing diverse images—such as flowers, dogs, and cars—these systems learn to identify them.
One rarely discussed aspect of AI is that ImageNet also includes thousands of photos of people, each categorized under specific labels. Some labels are straightforward, like 'cheerleader,' 'welder,' or 'scout.' Others carry clear emotional connotations, such as 'loser, failure, unsuccessful person' or 'slave, slovenly woman, rogue.'
Paglen and Crawford launched ImageNetRoulette to apply these labels, demonstrating how opinions, biases, and even offensive views influence AI—regardless of whether the labels seem harmless.
The labels in ImageNet were applied by thousands of anonymous workers, mostly from the U.S., hired by Stanford's team through Amazon Mechanical Turk's crowdsourcing service. They earned a few cents for each labeled photo, reviewing hundreds of labels per hour. In this process, biases were embedded into the database, though it's impossible to know whether the labelers themselves held such biases.
But they defined what a 'loser' or 'criminal' should look like.
These labels originally came from another massive dataset, WordNet—a machine-readable semantic lexicon developed by Princeton University researchers. However, the lexicon included these inflammatory labels, and Stanford's ImageNet researchers may not have realized the issues in their study.
AI is often trained on vast datasets that even its creators don’t fully understand. 'AI operates at an enormous scale, and that comes with consequences,' said Liz O’Sullivan, who previously supervised data labeling at AI startup Clarifai and now works with the Surveillance Technology Oversight Project (STOP), an organization focused on raising awareness about AI system issues.
Many labels in ImageNet are extreme, but the same problems can arise with seemingly 'harmless' labels. Even definitions like 'man' and 'woman' are debatable.
'When labeling photos of women (whether adults or not), nonbinary individuals or short-haired women might be excluded,' O’Sullivan noted. 'As a result, the AI model only recognizes long-haired women.'
In recent months, researchers have found that facial recognition services from companies like Amazon, Microsoft, and IBM exhibit biases against women and people of color. Through ImageNetRoulette, Paglen and Crawford hoped to draw attention to this issue—and they succeeded. As the project went viral on Twitter and other platforms, it generated over 100,000 labels per hour.
'We never expected it to blow up like this,' Crawford and Paglen said. 'It showed us how people truly feel about this issue and how engaged they are.'
For some, it was just a joke. But others, like Kima, understood Crawford and Paglen's intent. 'They did a great job. It’s not that I wasn’t aware of the problem before, but they brought it to light,' Kima said.
However, Paglen and Crawford believe the problem might be worse than people think.
ImageNet is just one of many datasets reused by tech giants, startups, and academic labs to train various forms of AI. Any flaws in these databases could already be spreading.
Today, many companies and researchers are working to eliminate these issues. To combat bias, Microsoft and IBM have upgraded their facial recognition services. In January, when Paglen and Crawford first highlighted ImageNet's strange labels, Stanford researchers blocked downloads of all facial images from the dataset. Now, they say they will remove even more facial images.
Stanford's research team issued a statement to The New York Times, stating their long-term goal is to 'address issues of fairness, accountability, and transparency in datasets and algorithms.'
But for Paglen, a greater hidden concern is approaching—AI learns from humans, and humans are biased creatures.
"The way we label images is a product of our worldview," he said. "Any classification system will reflect the values of those who classify it."