The Pack Contract: rules.yaml, vocabulary.yaml, Fixtures, README
The engine must never import domain logic, or your verifier becomes a fragile monolith that no one else can maintain. The separation between the verification engine and the domain pack is the single most critical architectural decision in this system. If the engine knows about runbooks, incident reports, or code reviews, it is not a general-purpose verifier. It is a bespoke script. The goal is to make the pack a self-contained unit of truth. The engine provides the machinery: loading YAML, running regexes, checking JSON schemas. The pack provides the intent: what constitutes a violation, what vocabulary is allowed, and what examples prove the rules work. This separation allows a domain expert to write a pack without touching Python. It allows a platform engineer to update the engine without breaking the domain rules. It allows legal or compliance teams to review the rules in plain text. The contract between these two halves is defined by four artifacts. These files form the interface. The engine reads them. The pack author writes them. Any deviation from this structure breaks the shareability of the pack.
rules.yaml: The Deterministic Core The rules.yaml file is the heart of the pack. It defines the checks that the engine will run. Each rule is a dictionary with a name, a type, and a configuration. The engine iterates over this list. It applies each rule to the input data. It collects violations. It returns a structured report. The rule types are limited. This is a feature, not a bug. If you allow arbitrary code execution in rules, you have defeated the purpose of a deterministic verifier. The supported types are:
Regex: A pattern that must not appear in the text. Used for banned phrases, PII leaks, or structural violations.
Schema: A JSON schema that the output must conform to. Used for ensuring structured outputs have the correct fields.
Count: A constraint on the number of occurrences of a pattern. Used for limiting list lengths or checking for duplicate entries.
Presence: A check that a specific string or pattern exists. Used for mandatory disclaimers or headers.
Each rule must have a unique ID. This ID is used in the violation report. It allows the user to trace a violation back to the specific rule that triggered it. The rule configuration must be serialisable to YAML. If it requires complex logic, it belongs in the engine, not the pack. Consider a rule that checks for banned words in a runbook. The rule might look like this:
The engine compiles this regex. It scans the runbook text. If it finds a match, it records a violation. The violation includes the rule ID, the message, the severity, and the location of the match. This is deterministic. It does not depend on the mood of an LLM. It does not depend on the version of the model. It depends only on the text and the pattern. The limitation here is expressiveness. You cannot write a rule that checks for subtle semantic errors. You cannot write a rule that checks for tone. You can only check for patterns. This is the tradeoff. You gain determinism and shareability. You lose the ability to catch nuanced errors. This is why the pack needs fixtures. The fixtures prove that the rules catch the errors you care about.
vocabulary.yaml: The Shared Lexicon Domain experts use jargon. The engine does not. The vocabulary.yaml file maps domain terms to their definitions. This serves two purposes. First, it documents the domain for the pack author. Second, it provides a source of truth for the rules. For example, a runbook pack might define the term “incident” as “an unplanned interruption to a service.” The rules can then reference this definition. If a rule checks for the presence of the word “incident,” it can link to this definition. This makes the rules more transparent. It allows the reviewer to understand why a rule exists. The vocabulary file is also useful for normalisation. If the domain uses multiple terms for the same concept, the vocabulary file can map them to a single canonical term. The rules can then check for the canonical term. This reduces the number of rules needed. It makes the pack easier to maintain. The vocabulary file is not just a dictionary. It is a contract. It defines the terms that the pack uses. If a rule uses a term that is not in the vocabulary, it is invalid. The engine should reject such rules. This ensures that the pack is self-consistent. It prevents the pack author from introducing new terms without documenting them.
Fixtures: The Proof of Correctness Rules are abstract. Fixtures are concrete. The pack must include a set of good and bad fixtures. These are sample inputs that the engine will run against the rules. The good fixtures should pass all rules. The bad fixtures should trigger specific violations. The fixtures are stored in a fixtures directory. Each fixture is a file. The filename indicates whether it is good or bad. For example, good_runbook.yaml and bad_runbook.yaml. The engine loads these files. It runs the rules against them. It checks that the good fixtures produce no violations. It checks that the bad fixtures produce the expected violations. This is the test suite for the pack. It proves that the rules work as intended. It catches regressions. If a rule is changed, the fixtures will fail if the change breaks the expected behaviour. This allows the pack author to iterate on the rules with confidence. The fixtures also serve as documentation. They show examples of good and bad outputs. This helps new users understand the domain. It provides concrete examples of what the verifier is checking for. This is often more useful than reading the rules themselves. The limitation of fixtures is coverage. It is impossible to write fixtures for every possible input. The pack author must choose representative examples. This requires domain expertise. The pack author must understand the common failure modes. They must write fixtures that catch these failures. This is not automated. It is a manual process. It requires care and attention.
README: The Human Interface The README.md file is the entry point for the pack. It explains the domain. It lists the rules. It describes the fixtures. It provides instructions for using the pack. The README is not just documentation. It is part of the contract. It defines the expectations for the pack. It tells the user what the pack does. It tells the user what the pack does not do. It sets the scope. The README should be written for the domain expert. It should use the vocabulary defined in vocabulary.yaml. It should reference the rules by ID. It should explain the rationale for each rule. This makes the pack transparent. It allows the reviewer to understand the intent behind the rules. The README should also include a section on limitations. It should list the types of errors that the pack does not catch. This manages expectations. It prevents users from relying on the pack for things it cannot do. This is crucial for maintaining trust in the system. This spec defines the interface between the engine and the pack. It ensures that the pack is shareable, reviewable, and testable. It separates the domain logic from the verification machinery. This separation is the foundation of the system. Without it, the verifier becomes a fragile monolith. With it, the verifier becomes a robust platform. The next lesson, “Build the First Pack: Runbook Domain, Five Seeded Defects,” will apply this contract. You will write the first pack. You will define the rules. You will write the fixtures. You will prove that the pack works.
