Litigation Risk From Automated Decisions and Discrimination
AI systems that make decisions about people create litigation risk. If the decision is wrong or is biased against a protected group, the person affected may sue. Understanding this risk is essential for advising clients who deploy AI systems affecting individuals.
The first risk is discrimination. If an AI system systematically favours one protected group over another, it violates discrimination law. In employment law, AI recruiting systems have faced liability for bias against women. In lending, AI credit scoring systems have faced liability for bias against racial minorities. The discrimination can be unlawful even if it is unintentional. The statute on discrimination focuses on outcome, not intent.
UK discrimination law protects people with protected characteristics: age, sex, race, disability, sexual orientation, gender reassignment, marriage and civil partnership, pregnancy and maternity, religion or belief. If an AI system makes employment, credit, housing or other consequential decisions and those decisions are influenced by protected characteristics, that is discrimination.
The challenge is that discrimination is often hidden. An AI system might not explicitly use protected characteristics as input variables. But if the system uses proxy variables that correlate with protected characteristics, the system will discriminate. For example, if a recruiting system uses postal code as a variable and postal codes correlate with race, the system will discriminate based on race even though race is not explicitly in the model.
Courts have recognised this as indirect discrimination. The system need not explicitly reference race. If the outcome of the system is less favourable to people of a particular race, and the outcome was not justified by business necessity, it is discrimination. The employer is liable.
Liability extends to the organisation that deploys the system. If a company uses an AI recruiting system that discriminates, the company is liable even if the system was built by an external vendor. The company has a duty to test the system for bias before deployment and to monitor for bias after deployment.
The second litigation risk is negligence. If an AI system makes a decision that harms a person through negligence, they can sue. Negligence requires duty, breach, causation and loss. The organisation deploying the system owes a duty of care to people affected by its decisions. The breach is failure to verify the system works properly before deployment. Causation is the AI decision that caused harm. Loss is the harm suffered.
If a bank deploys an AI loan approval system without testing it for accuracy, and the system incorrectly denies a loan to a qualified applicant, the applicant might sue for negligence. The negligence is the failure to test. If the applicant can show they would have been approved by a human reviewer, the bank is liable.
The third litigation risk is vicarious liability. If an employee uses an AI tool to make a decision and that decision is wrong, the employer is liable for the employee’s wrongdoing. If an HR manager uses an AI recruiting system to reject a candidate and the rejection is discriminatory, the employer is liable. The employer cannot escape liability by delegating the decision to a tool.
The fourth risk is product liability. If an AI system is sold as a service, the vendor may be liable if the system is defective or if warnings about limitations are inadequate. A vendor that sells an AI recruiting system without disclosing known bias risks may face product liability claims from customers whose use of the system resulted in discrimination.
How do clients manage this risk? The starting point is testing. Before deploying an AI system that affects people, the client should test it thoroughly. For employment, credit, housing and other high-stakes decisions, fairness testing is essential. The system should be tested on representative populations to ensure outcomes are fair across groups.
Documentation is the second step. The client should document what testing was performed, what the results showed and what steps were taken to address bias. If discrimination claims are later filed, this documentation shows that the client was diligent. If the client can show that they tested for bias and corrected problems, their liability exposure is reduced.
Monitoring is the third step. After deployment, the client should monitor the system for bias. If outcome rates start to diverge across groups, that signals bias. The client should be prepared to take action, including removing the system from use if bias cannot be remedied.