The Safety Requirements That Land on Engineering Teams
When a development team deploys an AI feature to production, they are not deploying just a model or an algorithm. They are deploying a system that has obligations baked into law, compliance frameworks and contractual liability. Understanding which safety requirements land on engineers, rather than on compliance teams or board members, is the foundation of safe AI delivery.
The EU AI Act imposes direct obligations on providers of AI systems. Article 5 has prohibited practices that came into force on 2 February 2025: the use of AI systems for real-time biometric identification in public spaces, social credit scoring, and subliminal manipulation designed to distort behaviour.
If your application touches any of these, it is illegal to ship regardless of how well engineered it is. A provider cannot delegate this risk to compliance. Developers need to know which use cases are off the table from the start.
For systems classified as high-risk under Annex III of the EU AI Act, the obligations multiply. These include AI systems used in recruitment, employment contract evaluation, redundancy decisions, performance monitoring, and loan approval. Article 50, which came into force on 2 August 2026, requires transparency: users of high-risk systems must be informed that an AI system is being used in a decision that affects them.
This is not an optional disclosure. It is a minimum legal requirement.
General-purpose AI (GPAI) model providers face separate obligations. Since August 2025, providers of GPAI models must ensure systems provide user documentation, implement appropriate risk management measures, and design their systems to comply with EU transparency obligations. The Commission began enforcement on 2 August 2026.
Developers working with or fine-tuning GPAI models must understand that those models carry pre-existing compliance obligations that flow downstream.
ISO/IEC 42001, published in 2023, codifies AI management practices that no law yet requires but auditors and customers increasingly demand. The standard defines controls across the AI lifecycle: information security, data governance, model development, testing, deployment, monitoring and incident management. The first UKAS-accredited certification body to certify organisations against ISO/IEC 42001 was BSI, on 15 January 2026.
When your organisation seeks ISO certification, engineering teams will be asked to produce evidence that controls were designed and executed. Writing that evidence as you build, rather than after deployment, is material cost difference.
What lands on engineers specifically? Threat modelling before code is written. Evaluation gates in continuous integration that must pass before deployment.
Logging that captures model inputs, outputs, user actions and system state so that failures can be reconstructed and audited. Documentation of data sourcing, model training, testing methodology and performance across user cohorts. Guardrails at input and output that reject attempts to extract secrets or cause harm.
Rate limiting and cost controls that prevent malicious wallet exhaustion. Change management and rollback procedures so that a failing model can be recovered within hours.
These are not suggestions or best practices written by vendors. They are regulatory requirements written into law or standard, and auditors will ask to see evidence that they were implemented. A developer who ships an AI feature without these controls is shipping a liability for their organisation.
Understanding these obligations early means you can design systems to satisfy them from the start rather than retrofitting controls after deployment. When you write code for an AI feature, your job is not just to make it work: it is to make it work safely, auditably and legally.
The safety requirements land on engineering teams because engineers are the people closest to the system and best positioned to implement controls. This is not bureaucracy imposed from above; it is responsibility that comes with the power to build systems that affect people’s lives.
As you build AI features, consult the specific rules for your jurisdiction and use case. The EU AI Act applies to systems used in the EU. The UK has its own AI governance landscape. The United States has patchwork state laws and sectoral rules (FCA for financial services, MHRA for medical devices, Ofcom for online safety).
ISO/IEC 42001 is a global standard that applies wherever your organisation operates. Know which rules apply to you, and build evidence that you comply with them from the start, not after the fact. This learning path covers the practical mechanics of compliance: the controls you implement, the tests you run, the logs you keep, and the documentation you produce.
These are not abstract requirements; they are concrete engineering practices that make your systems safer while producing evidence that satisfies auditors and regulators.
Frequently asked questions
How do you build AI features safely?
Safe AI development starts with threat modelling before code is written, then incorporates evaluation gates into CI/CD, structured logging from day one, and complete documentation as you build. Each component (input validation, guardrails, monitoring) should be designed and tested independently, then layered together. Documentation and audit-ready evidence should be captured as you work, not reconstructed after deployment.
What is prompt injection and how do you prevent it?
Prompt injection is when an attacker includes instructions in user input to redirect a model away from its intended task. Prevent it through layered defences: input validation and pattern detection, role-based access control on tools the model can invoke, structured output schemas that constrain what the model can return, and content segmentation that separates system instructions from user data in the prompt.
No single defence is bulletproof; use multiple methods.
What should be logged when an application calls a language model?
Structured logs should capture: the user’s input, the model’s output, the model version and timestamp, execution metrics (latency, token count, cost), user actions after the response, and any errors. Implement data minimisation: log only what is necessary for audit and debugging. Use structured formats (JSON) so logs can be queried programmatically.
Store logs separately from application data and restrict access to authorised personnel.
How do you test an LLM feature before release?
Test sets must be large, diverse and representative of actual production traffic. Include edge cases and failure modes intentionally. Ensure demographic representation so you can run fairness gates.
Label test cases with expected outputs so you can measure correctness. Run automatic evaluation gates in CI that check performance, fairness, safety metrics and cost. Do not rely on a single test set; refresh and expand it as you discover new failure modes in production.
What are AI guardrails?
Guardrails are automated controls that constrain what a model can input, output or invoke. Input guardrails filter or reject suspicious user input. Output guardrails check for harmful content before it reaches the user.
Tool call guardrails prevent the model from invoking tools improperly or without permission. Implement guardrails in layers (input, output, tool level) so that if one is bypassed, others catch attacks. Tune guardrails to balance safety and usability.
How do you stop an AI application leaking personal data?
Implement data minimisation: send to external systems only what is necessary for the stated purpose. Use secret management systems to store credentials outside the model’s reach; never embed secrets in prompts where prompt injection could expose them. Enforce access control on tools the model can invoke.
Encrypt data in transit to model providers. Monitor logs for unusual data access patterns. Implement data retention policies so data is deleted when no longer needed.
What is a model card and what goes in it?
A model card is a document describing the model’s purpose, performance, limitations and risks. Required by Article 50 of the EU AI Act, it must include: model description and training data, performance metrics and fairness testing results, known limitations and risks, mitigations implemented, guardrails and constraints, and guidance on when human review is recommended.
Write model cards as you build; they are living documents updated when the model changes or new issues are discovered.
Do developers have obligations under the EU AI Act?
Yes. Developers are part of the supply chain for AI systems. Article 5 prohibitions on certain practices (real-time biometric identification, social credit scoring, subliminal manipulation) apply regardless of who built the system.
Developers must implement controls required for high-risk systems under Annex III, including testing, guardrails, monitoring, and documentation. ISO/IEC 42001 also imposes direct obligations on anyone involved in the AI lifecycle.
How do you add a kill switch to an AI system?
A kill switch is a flag or configuration that disables the AI feature entirely. Implement it as a feature flag or configuration that can be changed without redeploying code. Design it so it can be activated by an oncall engineer in seconds.
When the kill switch is active, all requests are rejected with a message that the service is temporarily unavailable. Test kill switch procedures regularly (game days) to ensure the process works. Do not wait for a real failure to learn how to recover.