What is AI bias?
Measure the decision and its impact.

Bias can enter before the model makes a prediction.

AI bias describes systematic patterns in how a system represents people, makes predictions, or affects them. A measured difference between groups is important evidence, but the number alone does not tell you what caused it or whether the outcome is fair in its context.

NIST’s guidance treats bias as a socio-technical problem. It identifies systemic factors, statistical choices in data and models, and human factors in design and use. A model can inherit patterns from historical records, use a label that only imperfectly represents the thing a team cares about, or be deployed in a workflow where people treat its score as more certain than it is.

That is why a fairness review asks about the whole decision process: what the system is intended to do, which people it affects, what alternatives they have, and what a mistaken result costs them.

Measure who receives which errors—and what follows.

A model can perform well on average while making more mistakes for a smaller group. Review relevant segments of the test data, including intersections where sample sizes allow, and report uncertainty alongside the measurements. Choose test cases that reflect the real population, language, devices, and conditions of use.

Look past the prediction itself. If a system prioritizes requests, who waits longer when it misses one? If a person can correct an error, is the appeal route visible and usable? NIST recommends realistic tests representative of expected use, and says accuracy can be broken down across data segments. Measurement should be connected to the outcome people experience.

Each measurement answers a different question.

These measures are not interchangeable. In a particular binary prediction setting, Chouldechova showed that when group outcome rates differ and a predictor is imperfect, predictive parity and equal error rates can conflict. That result does not make fairness impossible; it shows why teams must state which errors and impacts matter in their context.

Before choosing a metric, ask whether the target label is a sound measure of the outcome the organization actually values. A mathematically consistent score cannot repair a poorly chosen objective.

Make the assumptions visible before comparing scores.

1. Define the decision. State what the system predicts, who uses the result, and what action follows.
2. Identify the people affected. Include users, people represented in the data, and people who may be affected without directly using the product.
3. Inspect the labels and data. Ask who is missing, how outcomes were recorded, and whether a proxy stands in for a contested or incomplete concept.
4. Choose measures for the task. Explain which harms, errors, or opportunities each metric is intended to capture.
5. Test realistic cases. Report overall and relevant group results, sample sizes, uncertainty, and the test conditions.
6. Review the workflow. Check explanations, human review, appeal paths, and whether people can correct bad inputs or decisions.
7. Monitor after release. Watch for changes in data, users, outcomes, and the way staff rely on the system.

Change the part of the system that creates the harm.

Possible responses include collecting better evidence, revisiting labels, changing the decision threshold, giving people a meaningful review path, or deciding that automation is not suitable for the task. Each intervention creates new trade-offs, so measure its effects and ask the people affected whether the process improved.

Adding examples from an underrepresented group may help with coverage, but more data alone does not guarantee better labels, a sound objective, or a fair workflow. Keep the original test cases, run the revised system against them, and record what changed.

For broader evaluation practices, read how to read an AI benchmark ↗ and how to choose a model for a real task ↗. When an explanation is part of the review, see how to test whether it reflects the model ↗. For the principles behind intent and system behavior, continue to AI alignment ↗.