Incident Handling When AI Causes Harm or Distress

Understanding AI-Related Incidents

When artificial intelligence systems cause harm or distress in educational settings, staff must recognise the signs and respond appropriately. An incident might occur when an AI tool provides incorrect academic advice, makes discriminatory recommendations, or generates content that offends students. For example, a learning platform might suggest inappropriate resources to a student with special educational needs, or an automated marking system could misjudge work quality, affecting a student’s confidence and progress.

Incident reporting requires staff to identify whether harm involves physical, emotional, or academic damage. The school’s safeguarding team must assess whether the AI system contributed to the situation. A teacher might notice a student becoming upset after receiving feedback from an AI tutor that was perceived as harsh or unfair.

The incident handler must determine if the AI’s response was inappropriate or if the system malfunctioned. Documentation should capture the specific AI tool involved, the nature of the harm, and any immediate actions taken.

• Record the exact date, time, and location of the incident
• Document the AI system’s output or behaviour that caused concern
• Note any student or staff reactions and emotional responses
• Identify whether the incident was isolated or part of a pattern

Immediate Response and Support

Staff must act quickly when AI systems cause distress or damage. The first step involves addressing any immediate safety concerns. If a student shows signs of emotional upset or academic distress, staff should provide appropriate support through existing safeguarding protocols.

A teacher might notice a student becoming withdrawn after receiving feedback from an AI assessment tool. The immediate response includes speaking with the student privately, offering emotional support, and ensuring they understand that the AI’s feedback was not personal.

Staff should also notify the designated AI safety officer or senior leadership team. The incident must be logged through the school’s established reporting system. In one scenario, a student might express confusion or frustration after an AI chatbot provided inaccurate information about university applications.

The teacher would immediately contact the AI coordinator to report the incident, ensuring the tool is reviewed and potentially suspended until further investigation.

Support measures might include:

• Providing pastoral care through existing student support networks
• Reviewing the AI tool’s settings or content filters
• Offering additional academic support or clarification
• Communicating with parents or guardians when appropriate

Investigation and Long-term Resolution

Following an incident, the school must conduct a thorough investigation to understand what occurred and prevent similar situations. The AI safety team should examine the tool’s outputs, review system logs, and interview staff or students who experienced the issue. For example, if an AI-powered learning platform consistently provided incorrect feedback to students from certain backgrounds, the investigation would focus on whether the tool’s algorithms were biased or misconfigured.

ISO 27001 clause 10.2 requires organisations to identify and address security incidents. Schools must apply similar principles to AI-related problems. The investigation process involves gathering evidence, determining root causes, and implementing corrective actions.

The school might discover that an AI tool was not properly trained on diverse datasets, leading to unfair outcomes. The resolution could involve updating the tool’s training data, adjusting its parameters, or removing it from use.

Long-term solutions focus on preventing recurrence through policy updates and staff training. The school might develop new protocols for AI tool evaluation before implementation. Regular monitoring of AI systems helps identify potential issues early.

Staff training should cover how to recognise AI-related problems and when to escalate concerns. The school’s data protection policy must also address AI usage, ensuring that student data is handled appropriately during investigations.

Documentation of the incident and resolution process forms part of the school’s risk management framework. This record helps demonstrate due diligence in AI implementation and provides evidence of proper incident handling. The school should communicate outcomes to relevant stakeholders while protecting student privacy.

Regular review of AI incident reports helps identify systemic issues that require broader policy attention or technical improvements.