Understanding the Model Life Cycle Well Enough to Audit It

You do not need to train models yourself to audit them, but you do need to understand the life cycle from conception to retirement. The model life cycle has six stages: problem definition and scoping, data preparation and quality, model development and selection, validation and testing, deployment and monitoring, and retirement or replacement. At each stage, specific failures are possible and controls are required. This lesson maps those stages to audit procedures.

Stage 1: Problem Definition and Scoping

This stage happens before any data scientist touches code. A business owner describes the problem they want AI to solve: maybe they want to reduce time-to-hire or catch fraudulent transactions earlier. The team should document the business objective, define success measures and identify constraints (legal, budget, data availability). Audit tests here focus on whether this documentation exists and was reviewed against regulatory requirements.

Stage 2: Data Preparation and Quality

AI systems are only as good as their training data. In this stage, teams select data sources and create features (derived data points) that feed into models. Governance questions include: where did the data come from and is there a legal basis for use? Has consent been obtained if personal data is involved? Audit teams should review data governance decisions and sample data to confirm it matches description. You might discover that data has been used for purposes not covered by original consent.

Stage 3: Model Development and Selection

Data scientists now train multiple models and pick the best one based on evaluation metrics. They split data into training and testing sets, train models, and measure performance. The question for audit is whether testing was rigorous enough and whether it covered the risks that matter. A model might achieve 95 percent accuracy overall but perform badly on a specific subset of users or data types. If that subset includes people with disabilities, protected characteristics, or vulnerable groups, then the model has a fairness failure even though overall accuracy looks good. Audit work here involves understanding what testing was done and whether it included specific checks for protected characteristics and edge cases.

Stage 4: Validation and Testing

Before deployment, mature organisations conduct independent validation by someone not involved in model building. This stage produces model cards, threshold documentation, and test reports. The auditor’s role is to confirm this validation happened, was independent, and was thorough. You should review the model card for completeness and accuracy, check whether fairness testing was done, and confirm that decision thresholds were set with business input, not just technical optimisation.

Stage 5: Deployment and Monitoring

The model moves to production and starts making decisions. Governance here includes access controls, version management and logging of decisions. Drift means the model’s performance degrades over time because the real world has changed in ways the training data did not predict. Audit tests include checking that only authorised changes reach production and that drift detection is working.

Stage 6: Retirement or Replacement

Eventually the system is retired or replaced. Audit questions include whether decisions made by the old system are still reviewable and whether archived data is handled according to retention policies.