The Code of Enlightenment
A Warning Concerning Systems That Score People
The number does not have to be wrong to do damage.
AI-drafted Reasoned position
- Reference
- CoE-0012
- Category
- warning
- Text revision
- 4.0
Imagine a risk score made for one decision. It travels ahead of the person into another office. The number arrives. The conditions under which it was valid do not.
A record of missed payments becomes an account of character. A flag produced for review becomes a reason to refuse. A later correction reaches the source database while an exported copy continues to decide what somebody may do.
The problem is larger than accuracy. An accurate observation has a scope. A person has a future.
When the recipient of a score must justify an override but need not justify acceptance, the default acquires authority. The worker at the counter may disagree and still lack the power to change anything. Naming that worker as the accountable human only relocates the anger.
A consequential score needs its purpose, source, uncertainty, date and route of correction to travel with it. It needs an expiry or an argued reason not to expire. A correction needs to reach the places the error reached.
Not all inputs can be published without exposing somebody else or defeating a legitimate protection. That limit increases the importance of independent review and prompt redress. It does not justify an answer that ends with the system says so.
Assessment can be necessary. A person can still ask which decision is actually being made, what the number is supposed to establish, and who can set it aside. The score is not the person who must live after the assessment ends.