Aggregate Accuracy Cannot Be the Finish Line for Consequential AI
Updated: Sep 8
A system can look impressive overall and still distribute its failures very differently across human beings.
That is not a philosophical objection to AI. It is a measurement problem.
In NIST’s 2019 Face Recognition Vendor Test, researchers evaluated 189 algorithms from 99 developers using more than 18 million images. NIST found demographic differentials in the majority of the algorithms evaluated. For one-to-one matching, false-positive rates for Asian and African American faces were often 10 to 100 times higher than for Caucasian faces, depending on the algorithm.
The technical report states the broader finding even more plainly: false-positive rates often varied by one or two orders of magnitude, meaning roughly 10x or 100x. NIST also found that different algorithms behaved differently, and some of the most equitable algorithms ranked among the most accurate.
The average can be correct and still hide the problem.
Imagine a consequential AI system reported as 97% accurate.
That number alone does not tell us whether the remaining 3% is randomly distributed. It does not tell us whether one population or circumstance carries far more of those errors. And it does not tell us what the error costs the person receiving it.
A false match that makes someone retry a device login is not the same event as a false match that places someone on a list for additional scrutiny. NIST explicitly distinguishes these consequences.
A metric is not evidence of fairness simply because everyone was included in its denominator.
Accuracy is one question. Distribution is another.
For consequential AI, asking only ‘How accurate is the system?’ is incomplete.
We also need to know who receives the errors, how frequently, in which direction, under what conditions, and what happens to the human being when the machine is wrong.
Those questions matter anywhere automated systems influence hiring, lending, health care, insurance, fraud detection, investigation, visibility, recommendations, access or opportunity.
An aggregate score can describe the machine while concealing radically different experiences of that machine.
This is not an argument against measurement. It is an argument for better measurement.
NIST’s findings matter precisely because they show that disparities are measurable and that algorithm choice matters. ‘Bias is inevitable’ is not an acceptable escape hatch when some systems perform substantially better than others.
For consequential AI, aggregate performance should therefore be evaluated alongside the distribution of error, the conditions producing the error, the direction of the error and the severity of the human consequence.
Only after those things are visible should humans authorize whether a system has earned the authority to influence what happens next.
The standard
Aggregate accuracy cannot be the finish line.
We should not accept a system as responsible merely because its average looks impressive. We have to see the people carrying its mistakes.
Primary evidence
NIST, ‘NIST Study Evaluates Effects of Race, Age, Sex on Face Recognition Software’ (Dec. 19, 2019; updated Feb. 3, 2025): https://www.nist.gov/news-events/news/2019/12/nist-study-evaluates-effects-race-age-sex-face-recognition-software
NIST Interagency Report 8280, ‘Face Recognition Vendor Test Part 3: Demographic Effects’: https://nvlpubs.nist.gov/nistpubs/ir/2019/NIST.IR.8280.pdf
NIST Face Recognition Technology Evaluation, 1:1 Demographic Differentials Summary, last updated July 31, 2026: https://pages.nist.gov/frvt/html/frvt11.html
Frick-E Energy™ examines how human identity, evidence and authority are interpreted by AI and decision systems. Public standards describe the problem and the accountability threshold. Proprietary diagnostic architecture remains private.
PUT THIS TO WORK
If humans, search systems, or AI can build the wrong version of you or your business, Frick-E Identity Engine™ builds the evidence-backed identity layer they should be able to retrieve, verify, and understand. WTFRICK-E™ functions inside that system as diagnostic intelligence, identifying identity drift, missing proof, conflicting signals, and interpretation gaps before they cost trust, consideration, or opportunity.
Comments