Build the First Pack: Runbook Domain, Five Seeded Defects

Runbooks are not prose; they are executable state machines disguised as text, and treating them as natural language is the single largest source of operational risk in incident response. The pipeline needs a domain where the cost of a wrong answer is immediate and measurable. Runbooks fit this perfectly. They have strict structural requirements, explicit prerequisites, and a finite vocabulary. If a runbook tells an engineer to restart a service before checking its logs, the system goes down. If it uses a command that does not exist in the current OS version, the engineer wastes time. We do not need an LLM to tell us this. We need a deterministic check. We are building the first pack. This pack targets runbooks. It contains five seeded defects. Each defect represents a distinct violation class. The verifier must catch all five. The output must be a list of violations with precise locations. No scores. No confidence intervals. Just facts.

Defining the Runbook Schema

A runbook is a sequence of steps. Each step has an ID, a description, and a command. Steps have prerequisites. Prerequisites are other steps that must have completed successfully. The runbook has a vocabulary. Only commands in the vocabulary are allowed. The schema is simple. It is not flexible. Flexibility is where errors hide. We define the schema in JSON Schema. The schema enforces the structure. It does not enforce the logic. The logic lives in the pack. The pack is a set of rules. Each rule checks one thing. The rules run in parallel. The output is a union of violations. The schema requires:

• A title.
• A list of steps.
• Each step has an ID, a description, and a command.
• Each step may have a list of prerequisite IDs.
• A global vocabulary list.

This schema is the contract. Any runbook that does not fit this schema is invalid. The verifier rejects it immediately. This is the first layer of defense. It is cheap. It is fast. It catches malformed input.

Seeding the Five Defects

We need a runbook that looks correct but is wrong. We write a runbook for restarting a web server. It has six steps. We inject five defects. Each defect is a different type. The first defect is a missing required element. Step 3 has no command. It only has a description. The schema allows this. The business logic does not. A step without a command is useless. The rule checks: every step must have a command. The violation is: Step 3 is missing a command. Location: steps[2].command. The second defect is a violated prerequisite. Step 4 depends on Step 5. Step 5 comes after Step 4. The rule checks: prerequisites must appear before the current step. The violation is: Step 4 depends on Step 5, which has not run. Location: steps[3].prerequisites. The third defect is a forbidden ordering. Step 2 is “Restart Service”. Step 1 is “Check Logs”. The rule checks: “Restart Service” must not appear before “Check Logs”. This is a domain rule. It is not in the schema. It is in the pack. The violation is: Service restarted before logs checked. Location: steps[1].id. The fourth defect is an out-of-vocabulary term. Step 6 uses the command “sysrestart”. The vocabulary only contains “systemctl restart”. The rule checks: command must be in vocabulary. The violation is: Command not in vocabulary. Location: steps[5].command. The fifth defect is a structural defect. The runbook has two steps with the same ID. Step 1 and Step 4 both have ID “check-logs”. The rule checks: step IDs must be unique. The violation is: Duplicate step ID. Location: steps[3].id. These five defects cover the main failure modes. Missing data. Wrong order. Wrong dependency. Wrong vocabulary. Wrong structure. They are easy to write. They are hard to catch with an LLM. An LLM might miss the duplicate ID. It might not notice the out-of-order dependency. It might hallucinate that “sysrestart” is valid. The verifier does not hallucinate. It checks the rules.

Implementing the Verification Rules

The rules are functions. Each function takes the runbook and returns a list of violations. The violations have a type, a message, and a location. The location is a JSON pointer. It points to the exact field in the runbook. The rule for missing commands is simple. Iterate over steps. If command is null or empty, add violation. The rule for prerequisites is slightly harder. Build a map of step IDs to indices. For each step, check its prerequisites. If a prerequisite ID is not in the map, or if its index is greater than the current step index, add violation. The rule for forbidden ordering is a domain rule. Define a list of forbidden pairs. For each pair, check if the first step appears after the second. The rule for vocabulary is a set lookup. Load the vocabulary. For each command, check if it is in the set. The rule for structural defects is a uniqueness check. Use a set to track seen IDs. If an ID is already in the set, add violation. The output format is critical. It must be machine-readable. It must be human-readable. It must be precise. The violation object has:

• Type: e.g. MISSING_COMMAND.
• Message: e.g. Step 3 is missing a command.
• Location: e.g. steps[2].command.

This format allows the pipeline to aggregate violations. It allows the UI to highlight the error. It allows the engineer to fix the error. It does not require interpretation. It is a fact.

The cost of this approach is maintenance. The rules are fragile. If the schema changes, the rules break. If the domain changes, the rules break. You have to write a rule for every constraint. This is tedious. It is also precise. The tradeoff is clear. You write more code. You get more certainty. The limitation is scope. This pack only checks runbooks. It does not check code. It does not check logs. It does not check metrics. It is a single domain. To verify other domains, you need other packs. The pipeline supports this. The pack is a plugin. You write a new pack. You plug it in. The verifier runs it. The output is the same. The format is the same. The integration is seamless. The working pack is now complete. It has the schema. It has the rules. It has the seeded runbook. It produces the expected violations. It is deterministic. It is fast. It is correct. You have built the first verifier. It is not an LLM. It is code. It is better. The next lesson, Testing the Verifier: Precision First, covers how to measure the quality of these rules.

Apply it to your work