Running your first evaluation with Inspect

Running evaluations with Inspect requires a clear understanding of setup, execution, and result interpretation. The process is straightforward but demands attention to detail, especially when dealing with agentic evaluations that require sandboxing. This lesson walks through the practical steps of using Inspect to run and interpret evaluations, from initial configuration to debugging results.

Setting up and running evaluations

Setup for Inspect is deliberately unassuming. Users install the core package using pip install inspect-ai and optionally inspect-evals for access to pre-built benchmarks. Evaluations are executed from the command line or via Python against any major model API. The VS Code extension supports authoring and debugging, offering transcript views that help trace model responses and identify issues.

A minimal custom evaluation consists of a Python file that defines a dataset of samples, including inputs and expected targets, a solver such as generate(), and a scorer such as includes() or model_graded_qa(). Inspect automates batching, retries, logging, and provides a results viewer for analysis. This modular structure allows users to compose evaluations with flexibility and precision.

Reproducing published results

To begin evaluating with Inspect, start by reproducing a published benchmark score. Run an inspect-evals benchmark against a model you use and compare your result with the published score. A gap between scores often reveals insights into solver settings, scorer selection, or version differences. For example, a mismatch in scoring may indicate that the model response was misclassified due to a subtle change in the prompting strategy or scorer logic.

Investigating these gaps is more instructive than simply accepting a score. It builds familiarity with how Inspect interprets model outputs and how to adjust components accordingly. This method of validation is essential for ensuring reproducibility and trust in evaluation outcomes.

Agent evaluations and sandboxing

Agent evaluations demand a dedicated sandboxing layer for safety and consistency. Users must define the runtime environment using Docker or Kubernetes, whitelist necessary tools, and configure the three axes of isolation: process, network, and file system. For example, a model agent interacting with a code interpreter must be restricted to prevent unintended access to the host system.

Never run agentic evaluations unsandboxed on machines that handle sensitive data. Misconfigurations in sandboxing can lead to security vulnerabilities or data leakage. A secure environment is non-negotiable when testing agents that interact with external systems or execute code.

Budgeting for large-scale evaluations

When running evaluations across many models or long suites, adopt professional budgeting practices. Use multiple epochs for stochastic models to capture variability, and calculate confidence intervals to assess statistical significance. For instance, a model’s performance might vary by 2.3% across epochs, and this variance must be quantified to draw reliable conclusions.

Implement optstop-style early stopping to manage computational cost when evaluating many models. This method halts evaluation early if performance plateaus, saving time and resources. It ensures that testing is both thorough and efficient, particularly in resource-constrained environments.

Interpreting results and learning from errors

The transcript viewer is the primary tool for interpreting results in Inspect. While scores summarise performance, transcripts explain why a model failed or succeeded on a given sample. Every mis-scored sample is a learning opportunity, whether it reveals a model limitation or a flaw in the scorer logic.

For example, a model might consistently misclassify a specific class of questions, or a scorer might be overly strict in its evaluation. Both scenarios are valuable for refining the evaluation or the model itself. This iterative process of analysis and refinement is central to effective AI evaluation.

What to take away

Running an evaluation in Inspect involves careful setup, precise execution, and critical analysis of results. Reproducing published benchmarks helps validate your configuration and spot inconsistencies. Agent evaluations must be sandboxed to ensure safety and reproducibility. Budgeting practices like epochs and confidence intervals improve reliability. Finally, the transcript viewer is essential for uncovering the root causes of model performance.

Reference

Sources and further reading

• Inspect getting started
• inspect_evals usage docs