This piece is originally published in Russian in 2021.

Bias is a disturbing yet inescapable part of human life. Technology cuts both ways: online communities now offer support for people and groups facing discrimination, but AI algorithms can also absorb and amplify the very prejudices we're trying to overcome. What makes an algorithm a carrier of bias and discriminatory practices — and how can we build fairer systems? Artist and researcher Denis Protopopov explains.

Algorithms scale easily and can automate a vast number of routine, complex, and high-stakes tasks. Their often surprisingly high accuracy creates a sense of objectivity and reliability — which naturally tempts us to hand over decisions that seem to demand a rational approach. Hiring, university admissions, criminal sentencing, medical diagnostics: machines are being asked to do all of this and more. Yet human biases, long embedded in society, can still surface in the decisions of an "objective" algorithm.

Three main factors cause AI to acquire our tendency toward prejudice:

• the input data, which forms the foundation of any AI system,
• the architecture of the algorithms themselves,
• the ability of algorithms to detect patterns invisible to humans.

Input Data: Bias In, Bias Out

The ability to learn from accumulated experience is the central feature — and the core value — of AI algorithms. If we want to train an algorithm to recognize handwriting, we need tens of thousands of examples of different letters. A spam filter requires a large collection of spam emails; a credit-scoring algorithm needs the credit histories of hundreds of thousands of people. A crime-prediction algorithm needs historical data on arrests, offenses, and sentences. Every AI algorithm looks to the past to predict the future.

In its investigation "Machine Bias," ProPublica documented how the automated system COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) was used across several U.S. states. The algorithm estimates the likelihood that a defendant will reoffend, and — as its name suggests — helps assess eligibility for alternative sanctions. Judges in some states actually factor these scores into their final decisions.

Machine Bias

There’s software used across the country to predict future criminals. And it’s biased against blacks.

ProPublicaJulia Angwin, Jeff Larson, Lauren Kirchner, and Surya Mattu

When ProPublica checked the system's predictions against reality, journalists found that COMPAS was biased against Black defendants. The system falsely flagged African American defendants as likely repeat offenders while rating white defendants as less prone to reoffending — predictions that often had no basis in what actually happened.

The algorithm's bias against Black defendants traces back to bias already present in the data on which earlier decisions were made. As Sandra Mason, a law professor at the University of Pennsylvania, has explained: although systems like COMPAS were designed to predict likely crimes, in practice they predict only the fact of arrest. That's because the underlying data contains almost no information about crimes actually committed. In some cases, people were falsely accused; other cases never led to conviction; some arrests were simply mistakes.

The statistics are stark: Black people in the U.S. are arrested far more often than white people, even though the two groups commit crimes at roughly equal rates. Mason cites an ACLU report showing that Black and white Americans use marijuana at comparable levels — yet Black Americans are arrested for it far more often.

AI is used at various stages of legal proceedings well beyond the United States. In China, the technology helps optimize document processing, transcription, and identity verification. In Russia, AI assists with document preparation and generating court orders for debt collection. And Estonia's Ministry of Justice has explored using automated courts to handle small claims under €7,000.

Still, the COMPAS case documented by ProPublica remains arguably the most prominent example of dangerous AI design in legal proceedings. The investigation is cited, for instance, in the European Ethical Charter on the Use of Artificial Intelligence in Judicial Systems, adopted by the Council of Europe in 2018. The Charter highlights the biases that can emerge in AI systems — including the fact that the behavioral history of a group can effectively determine the fate of an individual.

There is also a notable case from Russian practice: the creators of the "Algorithm of Light" project analyzed the texts of criminal cases involving domestic violence. They built a program that could determine, from the text of a murder conviction involving a female victim, whether she had previously been subjected to domestic violence. The team explains how this mechanism helps assess the true scale of the problem: official domestic violence statistics count only people who are legally family members, leaving out unmarried partners, former spouses, and so on. Moreover, those statistics include only criminal cases, not administrative ones.

1,001 Categories

The cases described above touch on just one of the problems that arise from poorly prepared training data — and in just one area of application. Right now, somewhere, datasets are being assembled to train AI systems that will eventually be used for a huge variety of tasks. These systems are impressively good at recognizing objects in images, identifying faces and speech, generating text, and much more.

Here's one example. ImageNet is a massive dataset containing 14 million images across 20,000 categories. Thanks to this dataset, improved computing power, and the right algorithmic architecture, computer vision saw a breakthrough in the early 2010s. This branch of AI aims to build machines that can recognize visual patterns and generate new images. Algorithms can now successfully identify and isolate one or more objects in a picture. It feels like the problem is solved — a computer can tell what it's looking at. Yet researcher Kate Crawford and artist Trevor Paglen argue in their essay that teaching a computer to describe what it sees will always carry ethical and political complications. And once again, the problem comes down to data.

Crawford and Paglen call their method of examining datasets "archaeology." They dig into the contents of databases to understand how they're structured. If we apply this archaeological approach to ImageNet, we find unremarkable categories at first: types of vehicles, fruits, berries, furniture, natural phenomena, and thousands more — all organized hierarchically, with categories nested inside one another.

But among the labels describing people, the essay's authors found entries like: "man," "woman," "father," "bisexual," "Boy Scout," "hermaphrodite," "drug addict," "barber," "alcoholic," "schizophrenic," "weakling," "neuroscientist," "prostitute," "loser," "hillbilly," "mulatto," "Bolshevik," "anti-Semite" — and dozens of others. Each category was matched to specific images. Where did these categories come from? ImageNet's taxonomy is built on WordNet, a word-classification database created in the 1980s. That's where the offensive and subjective labels originated. The categories and images describing people have since been removed from the current version of ImageNet, but older versions still circulate online — and somewhere, algorithms trained on that discriminatory edition are still in use.

This case leads to an important conclusion: datasets assembled for AI, despite their appearance of objectivity, are the product of specific decisions. A limited number of people made those decisions — people who once added categories like "kleptomaniac," "hermaphrodite," and "Bolshevik" and attached images to them. These are just a few dozen out of 20,000 categories, yet they can slip into autonomous vehicle navigation systems or surveillance networks and shape the actions taken on the basis of those technologies.

Beyond problematic categories entering datasets, essential information can also go missing. A clear example: when Amazon launched an AI-based recruiting system in the mid-2010s, a strange pattern emerged — women consistently received lower scores. The problem, once again, was the data. The algorithm had been trained on résumés submitted to Amazon over the previous ten years. The vast majority came from male candidates, who were also hired more often than women. Naturally, the algorithm learned to prefer men. Résumés containing the word "women's" — as in "captain of the women's chess club" or graduates of women's colleges — were downgraded. Amazon ultimately abandoned the system.

Similar problems can arise when AI is used to evaluate university applicants. These systems are introduced precisely to eliminate bias in admissions. But as we've seen, teaching an algorithm to advise and predict means showing it the data it will draw conclusions from — and if bias already exists in admissions, it will be reflected in the results. In New York City colleges, despite a relatively proportional number of applications from Black and white students, the admitted class is overwhelmingly white. Meanwhile, the share of women earning degrees in computer science in the U.S. has only declined — from 37% in the 1980s to 18% in 2016. Patterns like these can easily shape the logic of an algorithm tasked with making future admissions decisions.

The Architecture of AI Systems

The dismal gender statistics among computer science students are reflected in the industry itself: men hold between 77 and 83 percent of technical positions at Apple, Facebook, Microsoft, Google, and General Electric. As Yolande Strengers and Jenny Kennedy note in their book The Smart Wife, it's not only women who are underrepresented among developers and management — Black people and queer people are too. The result is that tech companies build entire AI-powered systems that reproduce and reinforce existing prejudices.

As the book's title suggests, the authors examine the phenomenon of the "smart wife" — their conceptual framing for voice assistants like Alexa, Siri, and Google Home. These are devices and virtual agents that come with stereotypically feminine characteristics by default — designed, in particular, to help with household management. Such assistants are built primarily by male engineers, and men are also the ones who typically initiate the installation of smart home systems. The authors advocate for new approaches to building smart assistants that don't reinforce existing gender hierarchies.

An additional complication stems from a fundamental principle of how AI systems work: they understand only numbers. This makes it especially difficult to handle fluid categories like gender and sexual identity. It also creates a temptation to identify specific physical and behavioral traits that correspond to artificially constructed identities.

The lack of diversity in data also undermines the quality of facial recognition algorithms — particularly when it comes to recognizing Black faces. Various companies have sought solutions. In 2019, IBM released a dataset called "Diversity in Faces," consisting of tens of thousands of photographs of people with different skin tones. However, each image came with data on facial symmetry and skull measurements. Crawford and Paglen argue that this approach doesn't eliminate prejudice — it deepens it. They compare such data to the practice of craniometry — skull measurement — used in the 19th and 20th centuries.

Hidden Patterns

AI systems — particularly in computer vision — have a fascinating capability: they can independently discover features in images that allow them to assign a picture to a specific class. A programmer doesn't need to manually explain to a computer why pictures of cats differ from pictures of dogs. It's enough to assemble two folders — one with cats, the other with dogs — and the algorithm will learn on its own to find distinguishing features and sort images it has never seen before. The features a machine discovers are often incomprehensible to humans, and some we can't even perceive. Stanford professor Michal Kosinski, together with his colleague Yilun Wang, described in a 2017 paper that an algorithm based on convolutional neural networks*could determine a person's sexual identity from their face.

No known objective features allow the human eye to do this. However, the study itself can be criticized for oversimplifying sexual identity, reducing it to "homosexual" and "heterosexual." The very act of such recognition creates pathologization: the status of anomaly is assigned to features that, in reality, have nothing to do with sexual identity. Deploying such a system — despite its limited understanding of the spectrum of identities — could put people in real danger in countries where queer individuals are marginalized.

Another paper, published in the summer of 2021, describes how AI-based systems can identify a patient's ethnicity from medical imaging — X-rays and CT scans — even though humans cannot. The researchers tested possible physiological markers of ethnicity and found that they played no role in the AI's predictions. Moreover, recognition accuracy remained high even with heavily degraded images or when part of the face was obscured. There are already known cases where AI learned to detect fractures on X-rays not by looking at the bone itself, but by identifying which hospital took the image — something it could determine from specific calibration characteristics of the equipment. All of these examples can create serious complications in patient care.

What Can We Do About AI Bias?

Bias in AI algorithms is being actively studied by researchers and practitioners. Various libraries and tools for fair machine learning (Fairness) are being developed, designed to ensure that an algorithm's performance isn't influenced by gender, ethnicity, sexual identity, religion, or other characteristics. Examples include AI Fairness 360 from IBM and Fairlearn. These tools reveal which features may compromise the objectivity of results. However, this approach — essentially isolating individual features — isn't always effective. First, reducing the size of a dataset affects its accuracy. Second, other features may correlate with those most likely to produce bias but can't simply be excluded: information about where someone lives, their marital status, education, and much more. Moreover, the previously mentioned paper by Sandra Mason argues that excluding individual features from data — ethnicity in particular — doesn't solve the problem: ignoring ethnicity means ignoring racism. Awareness of institutional racism is precisely what needs to be taken into account when designing AI systems.

It's unlikely we can eliminate bias from AI systems once and for all — after all, there are no humans without biases either. The least we can do, and the most realistic step, is to include in the development of AI systems the very groups who are least represented in them. That's the goal of initiatives like Black in AI and Queer in AI, which work on educational projects and help their participants find grants.

For many people today, artificial intelligence is nothing more than a black box — one assumed to be completely objective and unbiased. And paradoxically, the more widespread the technology becomes, the less we understand about how it works. Simply recognizing that algorithms and data are subjective and can carry prejudice is already critically important. And understanding how AI works saves the time needed to build more inclusive and fair systems.

Editor's note, 2026:

This article was originally published in Russian in 2021. Since then, the landscape has shifted — but not as much as we might have hoped. The EU adopted the AI Act in 2024, imposing fairness and non-discrimination requirements on high-risk AI systems in hiring, lending, and healthcare. Japan passed its first AI-specific law in 2025. The U.S. issued a White House AI Bill of Rights in 2022, though enforcement remains fragmented and the regulatory direction under the current administration is uncertain. On the technical side, generative AI has introduced new dimensions of bias: studies have shown that large language models recommend lower salaries to women and minority candidates, and image generators amplify racial and gender stereotypes. New York City passed a law requiring bias audits for AI hiring tools — one of the first concrete local regulations. The core argument of this article, however, remains unchanged: the problem isn't in the technology itself, but in the data, design choices, and lack of diversity behind it.