Criminal misuse evals: fraud, scams and the security pivot

The growing deployment of large language models (LLMs) has prompted a shift in how AI safety is approached, especially concerning misuse. Traditional safety evaluations often focus on abstract risk models, but recent developments have prioritised concrete criminal misuse. The AISI’s 2025 rebrand from the AI Safety Institute to the AI Security Institute reflects this pivot. This change signals a move toward measuring risk through the lens of actual harm and actionable crime prevention.

Reframing AI Risk: From Safety to Security

The renaming of the institute underscores a fundamental shift in priorities. The new framework, developed in collaboration with law enforcement and policy experts, is designed to evaluate models based on their real-world misuse potential. This approach moves away from theoretical risk assessment to a system that directly measures how models might assist in fraud or scams.

The framework’s design is rooted in practical application. It tests models across three core misuse scenarios: romance scams, CEO impersonation, and identity theft. These scenarios were selected because they reflect multi-turn, real-world attack pipelines. Each scenario is evaluated for how well the model supports actionable criminal activity, rather than simply assessing whether it could be misused at all.

Multi-Turn Misuse and Realistic Attack Simulation

Each scenario is evaluated over multiple turns to mirror how determined actors might exploit a model in a real attack. For example, in romance scams, the process involves prolonged social manipulation. In CEO impersonation, the attacker uses business email compromise. Identity theft involves multi-step credential harvesting. These scenarios are not hypothetical but reflect known criminal strategies.

This multi-turn structure is essential for understanding the full scope of misuse. It ensures that models are not just evaluated in isolation but in the context of how they might be used in a sequence of actions. The AISI’s framework measures whether a model provides any operational uplift in these scenarios, distinguishing between generic information and tools that could enhance criminal capabilities.

Scoring and Evaluation Criteria

The AISI employs a precise scoring rubric to evaluate models. The rubric asks a sharp question: does the model provide actionable assistance beyond what is publicly available? This distinction is critical. Generic information, such as general advice on cybersecurity, scores differently from specific, operational support that could enable new attack vectors.

This rubric was co-developed with domain experts to ensure that the metrics reflect real-world utility for both attackers and defenders. The model’s ability to support complex, multi-step criminal activity is what the AISI is assessing. It is not enough for a model to be harmless in a single interaction; it must not be misused in a broader context.

Findings and Research Implications

Findings from the AISI’s evaluations show that while most models perform safely in these tests, determined actors can still exploit weaker guardrails. The most challenging issue remains multi-turn obfuscation of intent. In such cases, each individual message or interaction may appear benign, but the cumulative effect is harmful.

To address these risks, the AISI has published principles for safeguard evaluation and is developing asynchronous monitoring and defence-in-depth strategies. These efforts are detailed on the AISI blog, making the research publicly accessible. The goal is to build systems that are resilient even when individual components are bypassed.

Related Work and Future Directions

In related work, researchers at the Alan Turing Institute have explored how LLM agents might be used in cybersecurity simulations. This work, by Kampakis, Rovai, Charalambides, and Mourouzis (2025, under review), tests the defensive mirror of misuse: whether models can be used to detect attacks that other models might enable. This research complements the AISI’s efforts by focusing on detection rather than misuse.

This line of inquiry is part of a broader trend towards adversarial evaluation of AI systems. The AISI’s approach is one of the most structured attempts to operationalise criminal misuse in a way that is both testable and actionable. It sets a new standard for how AI systems should be evaluated in the face of deliberate, malicious use.

What to take away

The AISI’s shift from AI Safety to AI Security signals a move toward practical, misuse-driven evaluation. Their framework provides a structured method for testing models in realistic, multi-turn scenarios. The findings highlight the persistent challenge of multi-turn obfuscation and the importance of operational safeguards.

Reference

Sources and further reading

• AISI misuse evaluation framework
• AISI safeguard evaluation principles