Writing Test Sets That Represent Real Users and Real Failures

A test set for an AI model is not the same as a test set for traditional software. Software test sets are usually small and designed to cover code paths: does this function return the right value for these 20 inputs? AI test sets must be large, diverse and representative of the distribution of real user behaviour in production.

An inadequate test set causes a dangerous problem: evaluation gates pass, deployment happens, and the model fails silently in production for cases that the test set did not cover. If your test set contains 500 examples of customer support questions, but your real production traffic includes unusual edge cases that appear in only 1 in 100,000 messages, your test set will not catch the failure mode.

This is why test set design is a critical responsibility of development teams.

Start by asking: what does real traffic look like?

If you are building a customer support chatbot, analyse your production logs and historical chat transcripts to understand the distribution. How many messages ask about shipping?

How many ask about returns? How many are off-topic or hostile? How many contain typos or use regional slang?

Your test set should roughly match this distribution.

The second step is to intentionally add edge cases and failure modes. Real traffic contains edge cases that are rare but important. A healthcare chatbot must be tested on edge cases like patients describing symptoms that are ambiguous, patients asking about drug interactions that are not in the training data, and patients asking the chatbot to diagnose a serious condition.

You should not wait for a patient to discover that the chatbot fails on these cases in production; you must test them before deployment.

Create a matrix of scenarios that matter for your use case. For a loan assessment system, the matrix might be: – Applicant profile: employed, self-employed, unemployed, student, retired – Loan type: personal loan, mortgage, business loan, credit card – Risk signal: clear approval, borderline case, clear rejection, fraud indicator – Data quality: complete application, missing information, inconsistent information

For each cell of the matrix, create representative examples

An employed person with a clear approval should be straightforward for the model. An unemployed person with a loan application should see whether the model correctly handles the case.

A self-employed person with incomplete income documentation should test whether the model handles missing data appropriately.

The third step is demographic and fairness representation. If your model makes decisions about people, your test set should include a fair representation of demographic groups: age, gender, ethnic background, disability status, if relevant to your domain. This serves two purposes.

First, it enables you to run fairness evaluation gates that measure whether performance differs across groups. Second, it prevents you from accidentally optimising for the majority population and leaving minorities underserved.

For a recruitment AI, your test set should include candidates from different educational backgrounds, career paths and geographic locations. If your training data is predominantly from expensive universities and metropolitan areas, and your test set is not, your model will fail on candidates from other backgrounds that exist in real traffic.

The fourth step is to create labelled examples where the correct answer is known. A test case should have both the input and the expected output. For a customer support chatbot, a test case is a customer message and the expected response category (shipping enquiry, return request, complaint, out-of-scope).

For a loan assessment model, a test case is an applicant’s data and the correct decision (approve, reject, refer to human review).

Labelling can be expensive and slow. Prioritise. For the most sensitive use cases, hire domain experts to label.

For less sensitive cases, use crowdsourcing platforms like Amazon Mechanical Turk or Surge AI. For some use cases, you can use synthetic data: generate test cases programmatically to cover specific scenarios without manual labelling.

Version your test sets. As you discover new failure modes in production, add them to the test set. As you receive feedback that the model performs poorly on certain scenarios, add those scenarios to the test set.

Over time, your test sets become richer and more representative. A test set from six months ago is not good enough for next month’s deployment; refresh and expand it regularly.