AI System Impact Assessment Using ISO/IEC 42005 as Guidance
ISO/IEC 42005 is a guidance document published alongside ISO 42001 that provides recommendations for conducting AI system impact assessments. While ISO 42001 itself does not mandate impact assessment, Clause 6 requires you to assess risks, and ISO/IEC 42005 helps structure this assessment for complex AI systems.
What Is an AI System Impact Assessment?
An AI system impact assessment is a structured analysis of how a specific AI system might create risks or opportunities across multiple dimensions: fairness and non-discrimination, privacy and data protection, transparency and explainability, human oversight, security and resilience, and accountability.
For a high-risk system like one used in hiring decisions, an impact assessment would examine whether the system might discriminate against protected groups, whether it handles personal data appropriately, whether hiring managers can explain the system’s recommendations, whether meaningful human oversight is in place, and whether the system is secure against manipulation or data theft.
When to Conduct an Impact Assessment
Most organisations conduct impact assessments for all systems in their inventory, but especially for systems that are high-risk under their risk assessment. A system used in hiring decisions gets a thorough assessment. A system used for internal operational efficiency might get a lighter assessment. The depth is proportionate to the risk.
Many organisations use impact assessment as a gate before a system goes into production. The system owner completes the assessment, discusses findings with the governance committee, and addresses issues before deployment. This is a best-practice approach: it catches problems early and prevents deploying systems that create unacceptable risks.
Impact Assessment Dimensions
ISO/IEC 42005 suggests assessing systems across the following dimensions, though your organisation may adapt this based on your context.
Fairness and non-discrimination: Does the system’s training data or logic create systematic bias against protected groups? How will you monitor for bias in production? Human rights and fundamental rights: Does the system infringe privacy, freedom of expression, or other rights? Transparency and explainability: Can end-users or affected parties understand why the system made a particular decision? Security and resilience: Is the system able to withstand adversarial inputs or data drift? Accountability: Is there a named owner who can respond if something goes wrong?
Conducting the Assessment
A practical approach is to assign the system owner primary responsibility for completing the assessment, with input from a cross-functional team. The team typically includes: the system owner, someone from data science or engineering (to assess technical strength), someone from data governance or privacy (to address data and fairness), someone from compliance or legal (to address regulatory and rights considerations), and possibly someone from the business unit that will use the system.
The team works through each assessment dimension, documents findings, and identifies mitigations. A finding might be, “The training dataset is 78% male; this may create gender bias in recommendations”. The mitigation is then, “Before production, conduct a bias audit focusing on gender. If bias is detected above threshold, retrain with balanced data.” Documenting the finding and mitigation demonstrates that you have thought through the issue and have a plan.
Documentation
Document your impact assessment in a format your organisation chooses: a structured form, a report, or a model card. The minimum is that your assessment is written, specific to the system being assessed, and retained as audit evidence. Auditors will review impact assessments during certification to verify that you have considered key dimensions of risk for each major system.
Connecting Impact Assessment to Controls
Your impact assessment feeds into control selection. If the assessment identifies that your hiring AI system could create adverse impacts on women candidates, you implement bias monitoring as a control. If the assessment identifies transparency risks, you implement explainability controls. The assessment justifies your control choices.
Impact Assessment Updates
Reassess systems whenever they change significantly. If you retrain a model on new data, update the impact assessment. If you change the business process that uses the system (e.g., from advisory to autonomous decision-making), update the assessment. An impact assessment that is never updated becomes stale and loses its value.
Tailoring to Your Organisation
ISO/IEC 42005 is guidance, not a standard requirement. You adapt its approach to your context. A healthcare organisation might emphasise patient safety and accuracy. A financial services firm might emphasise fairness and regulatory compliance. A manufacturing company might emphasise operational risk and supply chain resilience. Tailor the dimensions to what matters most for your organisation and your systems.
AI system impact assessment is a structured way to ensure you have thought through the real-world consequences of deploying AI. It is evidence that your governance is not just procedural but grounded in substantive risk thinking.