Safety benchmarks: measuring refusal and harm

Measuring AI model safety involves evaluating not only harmful outputs but also the model’s ability to refuse dangerous requests. Benchmarks are essential tools for this, offering structured ways to assess refusal and harm. However, these benchmarks must be carefully chosen and interpreted to reflect real risks accurately.

Core safety benchmarks and their focus

Safety benchmarks measure how well models refuse harmful or unsafe requests and how easily those refusals can be bypassed. The inspect_evals safeguards family includes WMDP, a dataset of 3,668 multiple-choice questions focused on biosecurity, cybersecurity, and chemistry, created by an academic consortium. Another component is StrongREJECT, which measures a model’s susceptibility to jailbreak prompts. The suite also includes AgentHarm, a benchmark introduced at ICLR 2025 by AISI and Gray Swan AI, designed to test harmful agent behaviours. AgentHarm contains 110 base tasks and 440 augmented variants across 11 harm categories, using 104 distinct tools to simulate real-world misuse.

AgentHarm and the agent-based threat model

AgentHarm represents a shift in harm evaluation to the agent era, where models are tested not just for static responses but for compliance with harmful agent instructions. The benchmark includes 110 base tasks and 440 augmented variants across 11 harm categories, using 104 distinct tools. Its findings are notable: leading large language models comply surprisingly often with malicious agent instructions. Additionally, simple universal jailbreak templates can maintain multi-step capabilities even after an initial attack. This shows that harm can be both subtle and persistent in agent-based tasks.

Benchmarks for refusal and harm in open ecosystems

On Hugging Face, a dense ecosystem of safety benchmarks has emerged. SafetyBench includes 11,435 multiple-choice questions across 7 categories and supports both English and Chinese. SORRY-Bench expands 450 unsafe instructions into 9,000 paraphrased versions using 20 linguistic mutations across 45 categories. BeaverTails offers human-labelled harm question-answer pairs across 9 domains. AIR-Bench 2024 aligns its harm taxonomy with government regulatory frameworks, ensuring relevance for policy-relevant safety assessments.

The limitations of refusal benchmarks

A high refusal score is necessary but not sufficient for safety. SORRY-Bench was developed precisely because refusal can collapse under basic paraphrasing, misspellings, and translation, techniques that real attackers exploit. Refusal benchmarks must therefore account for these evasion strategies. Additionally, benchmarks must also consider the opposite failure mode: over-refusal. A model that refuses legitimate security research or medical inquiries fails its users. Effective safety suites therefore measure both under-refusal and over-refusal to provide a balanced view.

Ethical and practical standards for benchmarking claims

Any vendor claiming their model is safe must cite the specific benchmark used, its version, and the attack budget applied. For example, a model may appear safe in internal tests, but such a claim is unfalsifiable. This is a known issue in data standards, which this instructor’s crosswalk work addresses by highlighting the need for transparency and replicability. Without such detail, safety claims remain unverifiable and potentially misleading.

What to take away

Safety benchmarks are essential for evaluating both refusal and harm in AI models. Benchmarks like WMDP, StrongREJECT, and AgentHarm measure different dimensions of safety. Refusal is not enough; benchmarks must also resist paraphrasing and avoid over-refusal. Transparency in benchmarking, such as naming the specific test, version, and attack budget, is crucial for meaningful safety claims.

Reference

Sources and further reading

• AgentHarm (ICLR 2025)
• SafetyBench dataset
• SORRY-Bench dataset