Training & Alignment

AI Bias and Fairness

By Aditya Kumar Jha, Engineer

AI bias is the tendency of machine learning systems to produce systematically unfair outcomes for certain groups, while fairness is the study of formal criteria, audits, and mitigations used to measure and reduce that harm.

Where bias enters a machine learning system

Bias in machine learning rarely comes from a single line of code. It accumulates across the pipeline, and the same word can mean a statistical property, a social harm, or both at once. The most common entry point is the training data. Historical bias reflects patterns that already exist in the world: if past hiring or lending decisions favored one group, a model trained to imitate those decisions will reproduce the pattern even when the target variable looks neutral. Sampling bias occurs when the data collected does not represent the population the model will serve, so error rates differ sharply across underrepresented groups.

Other sources are more subtle. Label bias arises when the ground truth labels themselves encode human prejudice or proxy a different quantity than intended, for example using rearrest as a stand-in for crime when policing intensity varies by neighborhood. Aggregation bias happens when one model is fit across groups that actually need different models, so a single decision boundary fits the majority well and minorities poorly. Measurement bias enters when a feature is recorded differently across groups.

Finally, deployment introduces feedback loops. A model that under-serves a group can shape future data: predictive policing that sends patrols to certain areas generates more recorded incidents there, which then confirm the model's prior. Because the model influences the world it later learns from, an initially small disparity can compound over time. Treating fairness as a one-time pre-launch check therefore misses harms that only emerge once the system is live.

  • Historical and sampling bias originate in data that misrepresents either the past or the target population.
  • Label bias and measurement bias corrupt the supervision signal itself, not just the feature distribution.
  • Aggregation bias forces one model onto groups whose relationships between features and outcome differ.
  • Deployment feedback loops let a biased model reshape future training data and amplify the original gap.

Formal group-fairness criteria

To move past intuition, researchers defined statistical criteria over a classifier's predictions, a binary outcome Y, and a protected attribute A such as race or gender. Demographic parity, also called statistical parity, requires the positive prediction rate to be equal across groups, independent of the true label. It is simple to state but ignores whether the groups actually differ in the outcome, so it can force equal acceptance even when qualified rates differ.

Equalized odds and equal opportunity were formalized by Hardt, Price, and Srebro in 2016 in Equality of Opportunity in Supervised Learning. Equalized odds requires the prediction to be independent of the protected attribute conditional on the true label, which means both the true positive rate and the false positive rate must match across groups. Equal opportunity relaxes this to only the true positive rate among those who truly qualify, so it equalizes the chance of a correct positive for genuinely positive individuals.

A third family, calibration, asks that among everyone assigned a given risk score, the actual fraction who are positive is the same across groups, so a score of 0.7 means the same thing for everyone. Each criterion captures a defensible notion of fairness, yet they answer different questions. Choosing among them is a value judgment about which kind of error matters most, not a purely technical decision.

Demographic parity: P(Ŷ=1 | A=a) = P(Ŷ=1 | A=b) Equalized odds: P(Ŷ=1 | Y=y, A=a) = P(Ŷ=1 | Y=y, A=b), for y ∈ {0,1} Equal opportunity: P(Ŷ=1 | Y=1, A=a) = P(Ŷ=1 | Y=1, A=b)
Three group-fairness criteria for a predictor Ŷ, true label Y, and protected attribute A. Equalized odds is equal opportunity plus an equal false-positive-rate constraint (y = 0).
  • Demographic parity equalizes the positive rate across groups but ignores the true outcome.
  • Equalized odds equalizes both true positive and false positive rates conditional on the label.
  • Equal opportunity equalizes only the true positive rate, focusing on qualified individuals.
  • Calibration requires a given risk score to carry the same meaning of probability for every group.

Why fairness criteria provably conflict

A central result is that these criteria cannot all hold at once. Kleinberg, Mullainathan, and Raghavan, in Inherent Trade-Offs in the Fair Determination of Risk Scores in 2016, formalized three conditions, including calibration and a balance condition on error across groups, and proved that except in highly constrained special cases no risk score can satisfy them simultaneously. The two degenerate cases are perfect prediction and equal base rates across groups. When the prevalence of the outcome genuinely differs between groups, the conflict is unavoidable.

Alexandra Chouldechova reached a parallel conclusion in Fair prediction with disparate impact: A study of bias in recidivism prediction instruments, also 2016. She showed that a score that is calibrated cannot also equalize false positive and false negative rates when recidivism prevalence differs across groups, producing disparate impact as a mathematical consequence rather than a programming mistake. This is the formal core of the COMPAS dispute.

The practical lesson is that there is no free lunch. A team cannot simply demand a fair model and expect every notion of fairness to be satisfied. Selecting a criterion means accepting that another reasonable criterion will be violated, so the choice should be made explicitly, documented, and tied to the real costs of each error type in the specific application.

  • Kleinberg et al. proved calibration and group-balanced error rates are jointly unsatisfiable unless base rates are equal or prediction is perfect.
  • Chouldechova showed a calibrated recidivism score must have unequal false positive or false negative rates when prevalence differs.
  • The impossibility is driven by differing base rates, not by a flawed model or biased engineer.
  • Picking a fairness criterion is a values decision that necessarily forfeits another reasonable criterion.

Individual fairness and real audited cases

Group criteria are not the only lens. Dwork, Hardt, Pitassi, Reingold, and Zemel introduced individual fairness in Fairness Through Awareness, presented at ITCS 2012, with the principle that similar individuals should be treated similarly. The classifier is constrained so that two people who are close under a task-specific similarity metric receive close predictions. The difficulty is defining that similarity metric, which itself can encode the very bias one hopes to avoid.

Real audits made these abstractions concrete. In Machine Bias, published by ProPublica on May 23, 2016, Angwin, Larson, Mattu, and Kirchner analyzed the COMPAS recidivism tool in Broward County and found Black defendants who did not re-offend were labeled higher risk almost twice as often as white defendants who did not re-offend, 44.9 percent versus 23.5 percent. The tool's vendor countered that the scores were calibrated by race, and both sides were correct, which is exactly the impossibility result in practice.

Two other cases are frequently cited. In 2018 Reuters reported that Amazon scrapped an experimental AI recruiting tool that had learned to penalize resumes containing the word women's, because it was trained on a decade of mostly male resumes. The same year, Buolamwini and Gebru published Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification, finding commercial gender classifiers misclassified darker-skinned women at error rates up to 34.7 percent while lighter-skinned men saw at most 0.8 percent.

  • Individual fairness (Dwork et al., ITCS 2012) requires similar individuals to receive similar predictions under a task similarity metric.
  • ProPublica's 2016 COMPAS analysis found Black defendants mislabeled high-risk at 44.9 percent versus 23.5 percent for white defendants.
  • Amazon scrapped its AI hiring tool in 2018 after it learned to downrank resumes signaling that the candidate was a woman.
  • Gender Shades (2018) found up to 34.7 percent error for darker-skinned women versus 0.8 percent for lighter-skinned men.

Measuring disparity in practice

Before mitigating bias, teams have to measure it, which means computing the relevant rates for each group and comparing them. For equalized odds the quantities of interest are the per-group true positive rate and false positive rate, and the disparity is the gap between the best and worst served group. The open-source Fairlearn library packages this pattern in a MetricFrame object that evaluates any metric across subgroups and reports the maximum difference.

The snippet below computes per-group true positive and false positive rates and their disparities directly from predictions and a sensitive attribute. The same disaggregated approach generalizes to selection rate for demographic parity or to calibration curves. The key discipline is to never report only an aggregate accuracy number, because a model can be highly accurate overall while failing badly for a minority subgroup that is small enough to be invisible in the average.

Disaggregated evaluation also exposes intersectional gaps, where a model performs acceptably on each single attribute but poorly on a combination such as darker-skinned women, which was the precise blind spot Gender Shades surfaced. Reporting metrics sliced by group, and by intersections of groups, is now a baseline expectation for any model used in a consequential decision.

python
from fairlearn.metrics import MetricFrame, true_positive_rate, false_positive_rate

# y_true, y_pred: arrays of 0/1 labels and predictions
# sensitive: array of group labels, e.g. ['A', 'B', 'A', ...]
frame = MetricFrame(
    metrics={'tpr': true_positive_rate, 'fpr': false_positive_rate},
    y_true=y_true,
    y_pred=y_pred,
    sensitive_features=sensitive,
)

print(frame.by_group)          # tpr and fpr for each group
print(frame.difference())      # max - min gap per metric == equalized-odds violation

# A plain-numpy equivalent of the per-group TPR:
# tpr_g = (pred[(group==g) & (y==1)] == 1).mean()
Computing per-group true positive and false positive rates with Fairlearn; .difference() gives the equalized-odds disparity. API verified against Fairlearn 0.14.
  • Equalized-odds auditing compares per-group true positive and false positive rates, not just overall accuracy.
  • Fairlearn's MetricFrame evaluates a metric across subgroups and reports the max-minus-min disparity via .difference().
  • Aggregate accuracy can hide severe failures on small subgroups, so always disaggregate.
  • Intersectional slicing catches combined-attribute gaps that single-attribute metrics miss.

Mitigation, audits, and documentation

Mitigation techniques are grouped by where they act in the pipeline. Pre-processing methods transform the training data, by reweighting examples or learning a representation that removes information about the protected attribute, before any model is fit. In-processing methods change the learning algorithm itself, adding a fairness constraint or penalty to the loss so the optimizer trades a small amount of accuracy for reduced disparity. Post-processing methods adjust a trained model's outputs, for instance setting group-specific thresholds, which is the approach Hardt and colleagues used to achieve equalized odds from an existing predictor.

Large language models introduce bias forms beyond the binary-classifier setting. They can produce stereotype associations, for example linking occupations to a gender, and representational harms where a group is depicted in demeaning or erasing ways. Because generation is open-ended, fairness here is evaluated with benchmark probes and red-teaming rather than a single parity number, and mitigations include data filtering, fine-tuning on balanced examples, and alignment methods that steer outputs away from harmful generalizations.

Technical fixes are incomplete without process. Independent audits, model cards introduced by Mitchell and colleagues in Model Cards for Model Reporting (2018) that report performance disaggregated by group, and datasheets for datasets introduced by Gebru and colleagues (2018) that document how data was collected and its limitations, create accountability. Documentation does not by itself remove bias, but it makes the choices and the residual disparities legible to the people affected and to regulators.

  • Pre-processing reweights or transforms data, in-processing constrains the loss, post-processing adjusts trained outputs.
  • LLM bias appears as stereotype associations and representational harm, audited with probes and red-teaming.
  • Model cards report group-disaggregated performance; datasheets document dataset provenance and limits.
  • Audits and documentation supply accountability that no single metric or algorithm can provide.

Key takeaways

  • Bias enters through historical, sampling, label, aggregation, and measurement effects in data, and through deployment feedback loops that compound over time.
  • Demographic parity, equalized odds, equal opportunity, and calibration each encode a different, defensible notion of fairness.
  • Kleinberg et al. and Chouldechova (both 2016) proved calibration and equal group error rates cannot both hold when base rates differ, so criteria must be chosen, not all satisfied.
  • The COMPAS, Amazon hiring, and Gender Shades cases are verified real examples of group-disaggregated harm.
  • Auditing means comparing per-group rates such as TPR and FPR, never just aggregate accuracy, and slicing by intersections of attributes.
  • Mitigation spans pre-, in-, and post-processing, and is incomplete without audits, model cards, and datasheets for accountability.

Frequently asked questions

Equalized odds requires both the true positive rate and the false positive rate to be equal across protected groups, conditioning on the true label. Equal opportunity is a relaxation that only requires the true positive rate to match, focusing on individuals who genuinely qualify. Both were formalized by Hardt, Price, and Srebro in 2016.
No, not in general. Kleinberg, Mullainathan, and Raghavan (2016) and Chouldechova (2016) proved that calibration and equal group error rates cannot both hold when the base rate of the outcome differs across groups, except in degenerate cases like perfect prediction. Teams therefore must choose which fairness criterion matters most for their application.
ProPublica's 2016 analysis found that COMPAS labeled Black defendants who did not reoffend as high-risk almost twice as often as comparable white defendants, 44.9 versus 23.5 percent. The vendor responded that the scores were calibrated by race. Both claims were mathematically true, which is the impossibility result playing out in a real system.
Compute your metrics separately for each group rather than only in aggregate, since overall accuracy can hide large subgroup failures. For equalized odds, compare per-group true positive and false positive rates; libraries like Fairlearn's MetricFrame report these and the max-minus-min gap. Also slice by intersections of attributes to catch combined-group blind spots.
Beyond classifier-style disparities, LLMs exhibit stereotype associations, such as linking certain occupations to a gender, and representational harms where groups are depicted in demeaning or erasing ways. Because output is open-ended, fairness is assessed with benchmark probes and red-teaming rather than a single parity number, and mitigated through data filtering, balanced fine-tuning, and alignment.
Model cards, introduced by Mitchell et al. in 2018, are short documents that report a model's intended use and performance disaggregated by group. Datasheets for datasets, introduced by Gebru et al. in 2018, document how a dataset was collected, its composition, and its limitations. Both create accountability by making choices and residual disparities transparent.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free