Auditing Human Oversight in Live Decision Making
Most regulatory frameworks require human oversight of AI decisions. The EU AI Act, GDPR Article 22, and employment law all contemplate a human in the loop or human oversight of automated decisions. Yet many organisations deploy AI in ways where human oversight is absent or ineffective. This lesson covers auditing human oversight: how to verify it exists, how to test it is working, and how to identify the common gaps that allow AI to make harmful decisions without challenge.
Human Review Points and Escalation
Start by identifying where human review should happen. In a hiring system, review points might include: manual review of candidates who score near the decision boundary and manual review of all rejected candidates before notification. In a fraud detection system, review points might include: manual investigation of all transactions flagged as high-risk before blocking, or at least review of all blocked transactions before they become final.
Audit for explicit human review requirements by asking: what decisions does the system make automatically without human review? For each automatic decision, has management consciously decided that human review is not needed? That decision should be documented and justifiable. High-risk decisions should have mandatory human review. Low-risk decisions might not. But the decision should be made intentionally, not by accident.
Escalation Procedures for Edge Cases
Humans cannot review every decision when the system makes hundreds of decisions per day. Instead, escalation procedures focus human attention on decisions where the system is uncertain, unusual, or high-stakes. A mortgage AI might escalate decisions for applicants with irregular income patterns. A hiring AI might escalate candidates with potential protected characteristic reasons for rejection.
Audit escalation by asking: what triggers escalation? Are thresholds set (e.g., escalate decisions outside 40th to 60th percentile of scores, where uncertainty is highest)? Is escalation working? Review the past thirty days of escalations. For each escalated decision, confirm it was actually reviewed by a human and the human’s decision was recorded. If escalations exist but are rubber-stamped (humans approve what the AI recommends without genuine review), then human oversight is illusory.
Override and Audit Trail
When a human reviewer overrides an AI recommendation, the override decision should be logged. Did the human accept the AI recommendation or reject it? Why? If humans override the AI frequently, that might mean the AI is not fit for purpose. If humans never override, that might mean escalation is not happening, or escalated cases are being rubber-stamped.
Pull override data from the past month. What percentage of escalated decisions were overridden? For decisions that were overridden, is there a recorded reason? If overrides are large, that is data for the model owner; if overrides are zero, escalation might not be working.
Time to Review and Practical Constraints
In practice, human reviewers have time constraints. A hiring manager might have thirty seconds to review a candidate before making a decision. A fraud analyst might have minutes to investigate a flagged transaction before the customer is denied service. Audit whether review time is realistic and whether reviewers have the information they need to make a good decision. If review materials are incomplete or difficult to understand, then review is not effective even if it is nominally happening.
Right to Explanation and Contestation
Some regulations, particularly GDPR, grant data subjects a right to explanation and contestation. If an AI system rejects someone’s credit application, they have a right to know why and a right to contest the decision. Audit whether the organisation honours this right. When someone requests an explanation, can the organisation provide it? Is the explanation meaningful to someone without technical training, or is it a dump of model parameters? Does the organisation have a process for reconsidering decisions when people contest them?