Sampling Techniques for Model Outputs and Decision Records
When an AI system makes hundreds or thousands of decisions daily, you cannot audit every single one. Instead, internal audit uses sampling: selecting a subset of decisions to examine in detail. This lesson covers practical sampling methods that give you reasonable assurance without auditing everything. Sampling is a core audit skill, and understanding how to design and execute good samples is essential for efficient AI audits.
Population Definition and Sampling Frame
The first step is defining your population clearly. If you are auditing hiring decisions, your population is “all candidates rejected by the hiring AI in the past six months.” Define the boundary precisely: does it include candidates rejected at every stage or only those who failed the initial screening? Does it include withdrawn applications? A poorly defined population means your sample does not represent what you think it does.
Next, create a sampling frame: the complete list of items in your population. If the hiring system records decisions in a database, your sampling frame is the query that pulls all rejections from the past six months. If your sampling frame is incomplete (some decisions are not in the database), then your sample is biased. Audit access to the sampling frame: can you access it directly, or does someone have to provide it to you? If someone else extracts the sample, verify the extraction is correct by spot-checking records.
Sample Size and Confidence Level
How many decisions do you need to sample? Audit standards suggest that for a population of 1,000 or more decisions, a random sample of 50 to 100 is typically sufficient to give you reasonable assurance. For smaller populations, you might sample a higher percentage. If the hiring system made 3,000 decisions in six months, a sample of 75 decisions gives you reasonable confidence. If it made 200 decisions, you might sample 50.
The confidence level depends on what you are testing. If you are checking that decisions are logged, a smaller sample might suffice because the control is binary (logged or not). If you are checking for fairness or bias, you need a larger sample to have confidence in findings.
Stratified and Targeted Sampling
Random sampling works, but stratified sampling can be more efficient. Stratified sampling divides the population into subgroups and samples each subgroup. If the hiring system makes decisions about candidates from different job families, you might stratify by job family to ensure your sample includes decisions for each type of role. This gives you more granular assurance than a purely random sample.
Targeted sampling focuses on decisions most likely to reveal issues. You might oversample decisions from candidates with protected characteristics, or decisions made near the decision boundary (borderline cases). Targeted samples reveal problems faster but do not give you population estimates, so use them for investigating issues, not for forming overall conclusions.
Documenting Your Sampling Approach
Whatever sampling method you choose, document it clearly. Write down: the population definition, the sampling frame, the sample size, the sampling method (random, stratified, or targeted), and the confidence level you are claiming. This documentation allows management to understand your methodology and allows other auditors to replicate your work. It also creates evidence for external auditors or regulators.
What to Check in Sampled Decisions
Once you have your sample, what do you check? Common audit steps include: confirming the decision exists and is recorded, confirming the decision can be traced back to model outputs, confirming any human review that should have happened is logged, and confirming the decision complies with policy. For hiring decisions, you might verify that rejected candidates received notification with an explanation and information about appeal rights. For fraud decisions, you might verify that a human investigation happened before transactions were permanently blocked.