School three: guaranteed and provably safe AI

Current approaches to AI safety rely heavily on statistical methods and empirical testing. These methods are insufficient for safety-critical applications. A new paradigm is needed, one that ensures safety through mathematical proof rather than probabilistic evidence.

This lesson introduces the provable-safety programme and its core tenets.

Provable Safety: A Mathematical Foundation

Provable safety treats safety as a mathematical property that must be formally verified against a precise world model and specification. This approach is advocated by works such as “Towards Guaranteed Safe AI” (arXiv:2405.06624) and “Provably Safe Systems” (arXiv:2309.01933). These sources argue that only provable safety ensures controllable artificial general intelligence (AGI).

The mathematical framework requires explicit models of the environment and formal definitions of safety criteria.

The Standard Architecture of Provable Safety

The architecture of provable safety follows a consistent triple: world model, specification, and verifier. This structure is common across projects in the field. The differences lie in the level of the world model’s fidelity and the strength of the safety guarantees required.

The world model must capture enough detail to support formal reasoning. The specification defines what constitutes safe behaviour. The verifier confirms that the system adheres to the specification under the model’s assumptions.

This framework is foundational to current research. It is not limited to one line of work but is a guiding principle. The complexity of the model and the strength of the proof determine how closely the system reflects real-world conditions.

The goal is to scale this architecture to systems at the frontier of AI capability.

Current Frontiers and Limitations

The ARIA (AI Research and Innovation) Safeguarded AI initiative is the school’s primary funder. In 2026, ARIA restructured its efforts: TA2 was cancelled, and TA3 was redirected toward formally-verified cybersecurity. This shift marks the current frontier of the school’s capabilities.

Proofs are currently valid at the infrastructure and component layer, not yet for large-scale frontier models. The distinction is critical: while components can be verified, the integration of these components into complex systems remains a challenge.

Enabling Technologies and Progress

Enabling technologies for provable safety are advancing rapidly. Neural theorem provers have reached IMO (International Mathematical Olympiad) medal level with systems like AlphaProof. Open-weight provers such as DeepSeek-Prover are also emerging.

These tools make machine-checked proof generation more accessible and cost-effective than ever before. This development lowers the barrier for integrating formal verification into AI development pipelines.

Research Contributions and Applications

The school’s research programme includes three key areas. First, WorldKernel formalises world models as coupling kernels over admissible possible worlds (arXiv:2606.10934). Second, Deterministic Event-Graph Substrates builds models that support counterfactual reasoning (arXiv:2605.15967).

Third, CIVeX verifies language-agent actions through causal intervention (arXiv:2605.09168). Together, these efforts produce certified world models that serve as the foundation for provable safety in agents.

Relationship to Other Schools of Thought

The provable safety school considers the limitations of standard evaluation methods as a central concern. It argues that contamination, sandbagging, and Goodhart’s law undermine the reliability of empirical benchmarks. This critique forms the intellectual foundation of the school’s approach.

The next course in this series will examine these limitations in depth. This focus on evaluation sets the school apart from others that rely more heavily on empirical validation.

What to take away

Provable safety requires mathematical proof rather than statistical evidence to ensure safety in critical systems. The core architecture relies on a world model, a specification, and a verifier. Current limitations lie in applying these methods to complex systems, though enabling technologies are improving rapidly. The school’s contributions lie in formalising core concepts like world models and causal reasoning. These insights form the basis for a more robust and verifiable approach to AI safety.

Reference

Sources and further reading

• Towards Guaranteed Safe AI (arXiv:2405.06624)
• Provably Safe Systems (arXiv:2309.01933)
• WorldKernel (Rovai, arXiv:2606.10934)