Auditing Model Development, Testing and Validation Records

Model development is where technical decisions are made that have lasting consequences. A model trained to optimise the wrong objective, tested in inadequate ways, or validated by people with conflicts of interest will produce harmful decisions at scale. This lesson guides you through audit procedures for development and testing: what to look for, what questions to ask, and what evidence should exist.

Development Records and Decisions

Start by requesting development records. Good development records document: what objective the team was optimising for (accuracy, speed, cost, fairness), what data was used in training and testing, what model architecture was selected and why, what alternative architectures were tried and rejected, and what hyperparameters (tuning settings) were used. If these records are vague or missing, that is a red flag. Development without documented decision-making creates a form of knowledge that lives only in people’s heads and disappears when they leave.

Audit for clarity in development records. First, was the objective clear and aligned with business goals? Many models are optimised for accuracy without considering fairness, interpretability, or cost. A hiring model optimised for “select the best candidate” will reflect biases in historical hiring data. A better objective would be “select qualified candidates with equal fairness across demographic groups.” Second, was any alternative architecture considered or was one approach chosen without comparison? The first model architecture tried is rarely the best; audit should reveal whether alternatives were evaluated. Third, if the model is meant to make decisions about people, were any fairness or bias considerations included in the development objective?

Testing and Evaluation

Request the test or evaluation report. At minimum, this should document: what test data was used (separate from training data), what metrics were measured, what the results were, and whether results met the acceptance criteria. A common audit finding is that test data came from the same source as training data or was too similar, making the test results optimistic and unreliable.

Test whether the testing included evaluation on specific subgroups, particularly protected characteristics like age, gender, race, or disability. A model might achieve 95 percent accuracy overall but perform at 70 percent for a particular demographic group. That is a material fairness failure that overall accuracy metrics hide. Audit should identify whether such subgroup testing happened and what the results showed.

Ask whether the team tested edge cases and failure modes. What happens when the model receives data it has never seen before? What happens when key data sources are unavailable? What happens if the user provides deliberately misleading input? If the team did not think through edge cases, the model is not ready for production.

Independent Validation

The strongest control is independent validation by someone not on the model development team. Request the validation report. Who conducted it? Were they truly independent, or was it done by the model owner’s manager? What did they check? Did they verify the development records, review test results, run their own tests, or just spot-check things?

Look for conflict of interest. If the person who built the model is also responsible for validating it, or if success of the model affects their performance rating, validation is not independent. This often happens in time-pressured projects where the same person develops and validates to move fast. It is a weakness in control, not efficiency.

Model Cards and Documentation

Modern practice requires model cards: single-page documents that summarise what the model does, how it was built, what data it uses, what its limitations are, and who should not use it. A model card is not a technical research paper; it is a practical document for people who need to use or audit the model. Audit for completeness: is the model card present? Does it cover all key areas? Is it accurate?