Bias Is a Product Decision Before It Is a Technical One

Most of the harm attributed to biased models is decided upstream: what the system is asked to predict, and what data was available to predict it from.

The standard account is that a model learns bias from its training data. That is true and it is only part of the story. The larger decisions are made before any data is collected.

The choice of target

A model does not predict what you want; it predicts the variable you gave it. A hiring system asked to predict "who was hired before" learns previous hiring patterns, including their prejudices. A health system asked to predict cost as a proxy for need learns that less money has historically been spent on people with less access to care, and concludes they are healthier. In both cases the technique is sound and the question was wrong.

Choosing the target is a product decision, usually made in a meeting, frequently by whoever knows which variable is available.

Absence in the data

Groups under-represented in training data get worse results, and the shortfall is invisible in an aggregate accuracy figure. A system that is 95 percent accurate overall can be far worse for a subgroup that is 3 percent of the sample without moving the headline number. Measuring accuracy per group is the single cheapest thing a team can do, and it is skipped constantly.

Fairness definitions conflict, mathematically

Equal error rates across groups, equal selection rates, and equally calibrated scores cannot generally all hold at once when base rates differ. This is a proven result, not an engineering shortcoming. Someone has to decide which definition applies, and that is a question about values that no amount of technical work answers.

Where to put the effort

Ask what is being predicted and why. Measure per group. Write down which fairness definition was chosen and by whom. Give people affected by a decision a route to challenge it that reaches a person. None of this requires new research and all of it requires someone with authority to care.

Get the next one by email