Structured Output, Schema Validation and Refusal Handling

Structured output means defining a schema that constrains what a model can return, then enforcing that schema validation before the output reaches the user. Instead of allowing a language model to return arbitrary text, you specify a JSON or XML structure, and the model must adhere to it.

If the model tries to return something outside the schema, validation fails and an error is generated.

Structured output serves multiple purposes. It makes model output predictable and machine-readable, so downstream systems can reliably parse and act on it. It prevents a model from going off-piste and returning something the application did not design for.

It makes adversarial attacks harder because the attacker must produce output that fits the schema.

Define your schema based on what your application needs

If your customer support chatbot should return an answer and a confidence score, define the schema as {“answer”: string, “confidence”: number (0 to 1)}. The model sees this schema and knows it must return exactly those two fields.

If the model tries to return a longer explanation or metadata, the schema validation layer rejects it.

Most modern language models can be instructed to return structured output. OpenAI’s API supports a modes parameter that enforces JSON schema. Anthropic’s API supports structured output through prompt instructions and validation.

Other vendors provide similar capabilities. If your model provider does not support structured output natively, you can implement it by parsing the model’s output, validating it against the schema, and if validation fails, asking the model to try again with explicit schema requirements in the prompt.

Schema validation is not just for output structure; it is also a security layer. If a prompt injection attack tricks the model into trying to return confidential information, and that information does not fit the defined schema, the validation layer rejects it. An attacker cannot compromise a system by redirecting the model if the model’s output is constrained to a safe schema.

Implement schema validation with clear error messages. When validation fails, you can either fail closed or fail open. Fail closed means returning an error to the user without trying to execute the response.

Fail open means trying to interpret what the model meant and applying a default behaviour. For safety, prefer fail closed unless there is a good reason to fail open.

Refusal handling is about designing what happens when a model correctly refuses a request. If the user asks for instructions on violence and the model refuses, what should the application return to the user? A vague error message?

A helpful explanation? A logging entry for security review? The design of refusal responses matters for both user experience and security.

A well-designed refusal response tells the user that the model cannot help with this request, optionally explains why, and offers an alternative if possible. For example: “I cannot provide instructions for violence. If you are experiencing a crisis, please contact your local emergency services or a crisis helpline.”

This is better than a generic error because it is respectful and helpful. Users are more likely to trust a system that explains why it is refusing them than one that silently rejects requests.

Some applications want to handle refusals differently than errors. A refusal is when the model correctly identifies that a request is out of scope and intentionally does not answer. An error is when the model fails to process a request due to a technical problem.

Logging and alerting should distinguish between them. A high rate of refusals might indicate that users do not understand what the system is designed to do, which is a usability problem worth investigating. A high error rate indicates a technical problem that needs immediate attention.

Measure refusal rates and false refusal rates

True refusals are when the model appropriately refuses harmful requests. False refusals are when the model refuses legitimate requests that it should answer.

A model that refuses everything is useless; one that refuses nothing is unsafe. Find the balance by measuring these rates and tuning the model or prompt.

Implement refusal audit trails. When a model refuses a request, log the request, the reason for refusal, the user, and the timestamp. This is required for compliance and for detecting patterns.

If a user makes 100 refusal requests in an hour, that is a security signal that should trigger investigation.