Working the frontier: problems, programmes and a starting kit

This lesson addresses the practical challenge of working on AI safety as a professional engineer or researcher. It offers a structured way to identify open problems, locate funding opportunities, and begin contributing through accessible tools and projects. The focus is on actionable steps that align with real-world constraints and existing technical capabilities.

Key Open Problems in AI Safety

The most impactful research areas in AI safety can be prioritised by their leverage. Specification engineering for value-laden domains is a top concern, as systems must reliably interpret human intent in complex, normative environments. World-model learning with quantified error is another core challenge, involving the ability to learn accurate internal representations of the environment with uncertainty bounds. Proof-cost compression for systems software ensures that safety-critical components can be verified efficiently. Monitorable-by-design agent architectures are necessary to enable real-time oversight of AI systems. Finally, the compositionality problem ensures that guarantees from individual verified components remain valid when integrated into larger systems.

Funding and Research Programmes

Several active programmes offer opportunities for contribution. ARIA’s TA3 stream focuses on formally-verified cybersecurity, while AISI’s Alignment Project identifies priority areas such as information theory, benchmark design, and monitoring among its 11 focus areas. The academic pipeline for safe-and-trusted-AI doctoral centres also provides support for training the next generation of researchers. These funding streams are detailed in the landscape course of this series, which maps the funding ecosystem for safety research.

Accessible Tools for Practical Engagement

Engineers can begin contributing to AI safety using open-source tools immediately. Lean 4 offers a platform for formal verification exercises. SWI-Prolog 10 or clingo, when interfaced with an MCP (Meta-Constraint Programming) layer, can serve as a reasoning engine. Scallop provides a differentiable logic framework for experimentation. A grammar-constrained decoding layer can be added to production outputs for controlled generation. Finally, Soufflé can be used with a real fact base to explore logical inference and knowledge representation.

Research-Adjacent Contributions

Contributions to AI safety need not require a dedicated research lab. Benchmark replication with published confidence intervals is a valuable and publishable effort. Registering assumptions in popular world models allows for better understanding of their limitations. Measuring the falsifiability of knowledge layers in existing systems provides insight into their robustness. These activities are practical, reproducible, and contribute meaningfully to the field.

Example Research Pathway

An illustrative example of a coherent research trajectory is provided by one instructor’s work. This includes five arXiv papers covering agent verification (2605.09168), ontology engineering (2605.09184), event-graph world models (2605.15967), scaling-law measurement (2605.23983), and world-model theory (2606.10934). The instructor maintains open repositories such as tardygrada, open-ontologies, and worldkernel. Contributions span verification-adjacent work in open inference infrastructure. The consistent thread is translating claims into verifiable artefacts.

What to Take Away

AI safety is not limited to academic or institutional research. Engineers can directly contribute through accessible tools, replicable benchmarks, and verifiable systems. The field benefits from structured engagement and a focus on checkable outcomes. A mature AI practice integrates safety at every stage of development and deployment.

Reference

Sources and further reading

• ARIA Safeguarded AI (TA3 cybersecurity)
• The Alignment Project research agenda
• Fabio Rovai, Google Scholar