Anatomy of an eval: tasks, solvers, scorers

Evaluations of artificial intelligence systems often report conflicting results, even when the same model is tested on the same benchmark. This inconsistency arises not from differences in the model itself, but from variations in how the evaluation is structured. The three-part structure of modern evaluations, tasks, solvers, and scorers, can silently introduce discrepancies if not carefully controlled. Understanding this structure is essential for interpreting benchmark results reliably.

Structure of an Evaluation

The Inspect framework, developed by the UK AI Security Institute in collaboration with Meridian Labs, provides a consistent structure for evaluations. This framework consists of three components: a dataset of tasks, a solver that generates the model’s response, and a scorer that evaluates that response. The framework has accumulated around 2,400 GitHub stars, nearly 7,000 commits, and is at version 0.3.130 as of June 2026. It is actively maintained and used across multiple organisations for AI safety research.

Solvers and Their Impact

The solver component determines how a model interacts with a task. It can be a simple generate() call or a complex multi-turn agent with tool use. Solvers vary widely in design and can significantly affect outcomes. For example, a solver that uses few-shot examples may produce different results than one that uses chain-of-thought prompting. The method of solving a task is a critical part of any evaluation and must be carefully documented.

Scoring Methods and Their Risks

Scorers determine how a model’s output is judged. These can range from exact text matching to model-graded scoring, where a second model evaluates the response. Model-graded scoring, for example, uses a function like model_graded_qa() to assess open-ended answers. While powerful, this method introduces risks: the judge model may have its own biases or failure modes. As such, a score from a model-graded benchmark is only as reliable as the scorer that produced it.

Inspect’s Tools and Ecosystem

The Inspect framework includes tools for inspecting evaluation transcripts, such as a VS Code extension and a web-based viewer. These tools allow researchers to review outputs in detail. The framework also supports external agents like Claude Code and Gemini CLI as solvers, increasing its flexibility. It is written in Python and available via PyPI as inspect-ai. The companion library inspect_evals, developed with Arcadia Impact and the Vector Institute, includes over 200 benchmarks, promoting consistency across research labs.

Why Papers Disagree on Benchmarks

Two studies may report different scores for the same model on the same benchmark, even when using the same evaluation framework. This happens when different solvers or scorers are used. For example, one study might use exact matching, while another uses model grading. The course example demonstrates that such differences are not due to the model’s performance but to the evaluation setup. This is why it is crucial to ask which solver and scorer were used when interpreting benchmark numbers.

What to take away

Evaluations are only as reliable as their structure. The three-part anatomy of tasks, solvers, and scorers can introduce variability that leads to divergent results. Researchers must pay close attention to these components when designing or interpreting benchmarks. The framework’s tools and libraries help standardise this process, but careful documentation remains essential. Always question which scorer and solver were used to produce a benchmark result.

Reference

Sources and further reading

• Inspect documentation
• inspect_ai repository