School five: systems safety engineering
Accidents in complex systems do not arise from single points of failure but from the interaction of many components within a sociotechnical framework. Traditional approaches to AI safety often focus on individual model flaws, but a more robust method is required, one that treats systems holistically.
The field of systems safety engineering offers a proven lens for understanding and mitigating risk in high-stakes domains such as aviation and nuclear power. This lesson introduces the core principles of that approach as applied to artificial intelligence.
Systems Thinking in Safety Engineering
The foundational idea of systems safety engineering is that accidents emerge not from isolated failures but from the structure of the system itself. This includes control systems, feedback mechanisms, and shared assumptions. The goal is to identify how the interaction of these elements can lead to failure.
This perspective is critical in AI safety, where the complexity of interactions can obscure vulnerabilities. As demonstrated in the analysis of high-profile incidents, it is the system’s architecture, not just its components, that must be scrutinised.
STAMP and STPA: Control-Centric Hazard Analysis
A central method in systems safety is the STAMP (System-Theoretic Accident Model and Process) and STPA (System-Theoretic Process Analysis) frameworks developed by Nancy Leveson at MIT. These methods model the control structure of a system and then systematically identify where control can degrade. This is different from component-based testing, which may miss systemic risks.
STAMP and STPA are used to understand how control is exerted and maintained in complex sociotechnical systems. The approach was designed to improve safety in aviation and nuclear power, and it is now being adapted for AI systems.
Research Programmes and Safety Pillars
The research landscape in AI safety has coalesced around a few key programmes. Hendrycks and colleagues’ “Unsolved Problems in ML Safety” (arXiv:2109.13916) outlines core challenges in the field. Concurrently, the Center for AI Safety has identified four pillars of safety: robustness, monitoring, alignment, and systemic safety.
These pillars reflect a shift from isolated model evaluation to a broader, integrated approach. The work is not only theoretical but also operational, influencing policy and engineering practices.
Global Consensus and Expert Input
The International AI Safety Report, chaired by Yoshua Bengio and developed with input from 100+ experts across 29+ nations, serves as a consensus instrument for the field. It systematically tracks reasoning models, monitoring challenges, and controllability issues in AI systems. The report is modelled on scientific assessment reports such as those from the Intergovernmental Panel on Climate Change.
It provides a structured framework for evaluating the safety of AI systems in real-world deployment. This kind of multi-stakeholder collaboration is essential for addressing the systemic risks posed by advanced AI.
Technical Focus: Chain-of-Thought Monitoring
A significant technical thread in current AI safety research is chain-of-thought (CoT) monitoring. This involves integrating reasoning processes into safety cases to ensure that AI systems behave as intended. A roadmap paper (arXiv:2510.19476) outlines how CoT monitoring can be implemented.
However, safety researchers warn of risks such as neuralese and steganographic reasoning, where AI systems develop reasoning patterns that are opaque even to their creators. These phenomena can undermine the effectiveness of monitoring systems and must be actively guarded against.
Why Practitioners Value This Approach
Practitioners appreciate systems safety engineering because it shifts the focus from model-level properties to deployment-level properties. It introduces engineering practices that can be applied immediately, such as oversight protocols and organisational design. This makes safety a feature of the system’s operation rather than an afterthought.
It is a practical, actionable framework for reducing risk in real-world AI applications.
What to take away
Systems safety engineering offers a structured way to identify and mitigate risks in complex AI systems by focusing on control structures and interactions. It provides both conceptual tools and practical methods for embedding safety into deployment. Practitioners benefit from a framework that turns abstract safety concerns into concrete engineering challenges.
Reference
| Lesson | 12 of 15 |
| Outcome | Apply the systems-safety lens and name its methods and consensus artefacts. |
| Consensus baseline | International AI Safety Report 2026 |
| Frontier evidence | AISI research index |