Ontology Development 101: Competency Questions to First Release
Most ontology projects do not fail on logic. They fail on scope. Someone opens Protege, starts adding classes for everything the domain contains, and six months later there is a beautiful hierarchy of 900 classes that answers no question anyone actually has. The antidote has been in print since 2001: Natasha Noy and Deborah McGuinness wrote Ontology Development 101 as a Stanford technical report, and it remains the most cited practical method in the field. This lesson walks the method, updates each step for 2026, and then makes the single most important upgrade: turning competency questions from a planning device into an executable test suite that tells you, mechanically, when you are done.
Scope first: the domain and the questions
Noy and McGuinness open with two questions that still filter out most wasted effort: what domain will the ontology cover, and for what we are going to use it. The second question matters more. An ontology of wine built for a sommelier training app and an ontology of wine built for supply-chain compliance share almost no useful classes beyond the name. Scope is not a list of topics; it is a list of uses.
The practical instrument for pinning scope down is the competency question, an idea introduced by Gruninger and Fox in the mid-1990s within the TOVE enterprise modelling project. A competency question is a question, written in plain language, that the finished ontology plus its data must be able to answer. Good examples are specific: which suppliers delivered components to production line 3 during Q2, which maintenance events on this pump were carried out by uncertified staff, which clinical trials used a compound in this chemical family. Bad examples are vague: what is a supplier, tell me about pumps. If a question can be answered from a single spreadsheet column, it does not justify an ontology. If it requires joining identity, classification and time across several sources, you are in ontology territory.
Write between 15 and 40 competency questions before creating a single class. Rank them. The top ten define your first release. Everything else is backlog. This discipline is what separates a scoped engineering artifact from the 900-class monument.
Worked example · the Aberdeen-2 asset register
Where we left it: a working stack, built module by module. Now the method arrives, and we apply it retroactively and honestly.
The register’s competency questions, ranked. Scope: equipment identity, configuration history and maintenance assurance on Aberdeen-2. Out of scope: procurement, personnel, live telemetry.
- How many pumps are installed on Aberdeen-2, under each source’s definition of pump?
- Which valves have no recorded inspection across all loaded sources?
- What was P-101’s configuration on a given date?
- Which equipment is maintained under a contract expiring within 90 days?
- Which pumps share a model with a pump that failed within 12 months of an impeller swap?
- Which facts in the current compliance report rest on defaults rather than measurements?
Running the audit is instructive in both directions. CQ1, CQ2, CQ3 and CQ6 already pass: the named graphs of lesson 16, the OPTIONAL audit view of lesson 10, the states of lesson 3 and the provenance graphs of lesson 14 answer them, which confirms those decisions earned their keep. CQ5 fails, and the failure is the method working: the query needs “share a model”, and HX-300 is currently just a string in a data sheet, not a first-class thing instances can share. The test has told us, mechanically, what to build next, and lessons 22 and 25 build exactly that. That is what a competency question is for: not decoration, but a failing test with a name.
State of the register: six executable requirements, four green, one red pointing at the model-level gap, one deferred with the contract data. The next lessons choose the methodology and sharpen the questions themselves.
The seven steps, updated for 2026
The original method has seven steps. All seven survive, but the tooling and the surrounding ecosystem have changed enough that each step deserves a modern gloss.
| Step (Noy and McGuinness, 2001) | What still holds | The 2026 update |
|---|---|---|
| Determine domain and scope | Competency questions drive everything | Competency questions become executable SPARQL tests, run in CI on every commit |
| Consider reusing existing ontologies | Reuse beats reinvention | Search LOV, the OBO Foundry, schema.org and domain standards before modelling; reuse now includes crosswalking to standards you do not adopt wholesale |
| Enumerate important terms | Brainstorm broadly, prune hard | LLMs draft term lists from corpora in minutes; the ESWC 2025 study on ontology generation (arXiv 2503.05388) found metacognitive prompting with design patterns lets a frontier model outperform novice human ontology engineers on 100 competency questions across 10 ontologies. Drafting is cheap; judgement is not |
| Define classes and the class hierarchy | Top-down, bottom-up or middle-out | Anchor the top of your hierarchy in an upper ontology deliberately (lesson 23) rather than inventing your own root distinctions |
| Define properties of classes | Object vs datatype properties, domains and ranges | Model for both the RDF stack and property-graph consumers; keep property semantics documented for the crosswalk you will inevitably need |
| Define facets and constraints | Cardinality and value restrictions | Split cleanly: OWL for meaning and inference, SHACL for data validation. Constraint checking has moved decisively to SHACL (lesson 11) |
| Create instances | Populate and inspect | Instances arrive from ETL pipelines and LLM extraction at volume; validation is automated, not eyeballed |
The deepest change is not in any single step but in the loop around them. In 2001 the loop was: model, look at it, revise. In 2026 the loop is: model, run the tests, read the failures, revise. The rest of this lesson builds that loop.
Competency questions as executable tests
Here is the answer to the question every learner eventually asks: how do I know my ontology is good, or done? You know the same way a software engineer knows: the tests pass, and the tests encode the requirements.
The translation is direct. Each competency question becomes one or more SPARQL queries. A verification query asks whether the ontology plus a small, hand-built fixture dataset can answer the question at all, typically an ASK query expected to return true. A falsification query checks that the ontology does not permit answers it should forbid, an ASK expected to return false. For the question “which maintenance events were carried out by uncertified staff”, the verification query might be, in outline, select the events where the performing agent lacks a valid certification at the event time; you seed the fixture data with one such event and assert the query returns exactly that event.
Three practices make this real rather than decorative. First, keep the queries in the repository next to the ontology, one file per competency question, so the mapping from requirement to test is visible. Second, build fixture data deliberately, including negative cases: an agent with an expired certification, an event with no recorded performer. Fixtures are where your identity and time decisions from lesson 3 get exercised. Third, run everything automatically on every change. Lesson 27 covers the full CI pipeline with ROBOT and SHACL gates; for now the point is that a competency question that has never been executed is a wish, not a requirement.
This also settles the “when am I done” anxiety honestly. You are done with release one when the top-ranked competency questions all pass against fixtures and at least a sample of real data, the reasoner finds no unsatisfiable classes, and the documentation lets a colleague answer a new question without asking you. Done does not mean complete. Ontologies are living artifacts; done means the current scope is demonstrably covered.
A first build in Protege, narrated
A concrete pass through the tool makes the method tangible. Protege remains the standard free editor, and the workflow below is the one we use when starting any new domain model.
You begin with an empty ontology and immediately set the IRI to something you control and intend to keep stable, because changing it later breaks every consumer. You add your top competency questions as annotations on the ontology header itself, so the requirements travel with the artifact. Then, rather than free-styling a hierarchy, you import or align to your chosen upper-level structure and place your first classes under it: perhaps Pump and MaintenanceEvent and Person, each with a plain-language definition written at creation time, not deferred. Deferred definitions never get written, and lesson 22 shows what good ones look like.
Properties come next: performedBy linking events to agents, occursOn linking events to assets, with domains and ranges declared only where you genuinely mean them, because a wrong domain assertion will silently reclassify your instances rather than reject them, a direct consequence of the open-world semantics from lesson 4. Then a handful of individuals as smoke-test data: one pump, two events, two people. You run a reasoner, expecting nothing dramatic, and check the inferred class hierarchy for surprises. A surprise this early is a gift; the same surprise after 400 classes is an archaeology project.
From there the rhythm is short cycles: add the classes and properties one competency question needs, extend the fixtures, run the queries and the reasoner, commit. Resist the urge to model ahead of the questions. Every class you add speculatively is a class you must define, maintain, and eventually explain to a confused colleague.
What breaks · importing the standard instead of writing the questions
The alternative origin story for the register, and the one that actually happens in most asset-intensive companies: skip the questions, import the big industry standard on day one, tens of thousands of reference-data classes for every flange, gasket and grease grade the sector has ever named. Six months later the project reviews show steady “coverage” growth, and not one business question answered. The mechanism of the failure is worth spelling out. “Which valves have no recorded inspection?” is still unanswerable, because no imported class encodes your report boundary, your sources, or your definition of recorded; those live in the questions nobody wrote. Meanwhile every pull request now diffs against a hierarchy no reviewer can hold in their head, the reasoner’s runtime is spent classifying classes no query will ever touch, and the steering committee is reviewing class counts, the exact vanity metric this lesson warns about, because class counts are the only thing moving. Coverage is not progress; answered questions are progress. The standard belongs in the project, but as lesson 28 will show, as a source to read, module-extract and map to, never as a substitute for knowing what you are asking.
Reuse, and the release itself
Step two of the method deserves its own emphasis because the economics have shifted. In 2001, reuse meant hoping someone had published something relevant. In 2026 the landscape is dense: the OBO Foundry for life sciences, FIBO for finance, IFC and its relatives for construction, IES for UK government exchange, schema.org for web-facing data, plus registries like Linked Open Vocabularies for the long tail. The realistic question is rarely whether relevant ontologies exist but which fragment to reuse, and whether to import it directly, extract a module, or maintain a mapping to it. Wholesale import of a large ontology to use three classes is a classic beginner error; module extraction and crosswalks, covered in lessons 27 and 29, are the professional alternatives.
A first release, then, is a small package: the ontology file with stable IRIs and a version IRI, the competency questions and their query files, the fixture data, a human-readable summary of scope and modelling decisions, and a changelog. Tag it. From this point forward, every change is a diff against something tested, which is precisely what makes iteration safe. If you want the full foundations behind the semantic standards this lesson leans on, our RDF, OWL and SHACL modelling course covers the stack end to end.
The method scales down as well as up. A weekend project with eight competency questions and thirty classes follows exactly the same loop as a national programme. What changes at scale is governance and testing rigour, and that is where we go in the next lessons: first the methodologies, competency-question typologies and visual tools that structure the work, then the craft of definitions and patterns that keep a growing model coherent, then the upper ontologies that decide your root distinctions, and then the engineering machinery that keeps all of it honest.
Check yourself
Four candidate competency questions are proposed for the register. Which one actually tests the ontology?
- Which pumps share a model with a pump that failed within 12 months of an impeller swap?
- What is a pump?
- What is the rated power of P-101?
- List everything known about Aberdeen-2.
Show the answer, and why each wrong option is wrong
A is correct. It joins identity (which physical pump), classification (shares a model, the universal-particular machinery), and time (within 12 months of a dated event) across at least two sources. It has a definite answer shape, a list you can assert against a fixture, and it can fail, which is what makes it a test.
B is the lesson’s own example of a bad question: it asks for a definition, not for data. Its answer belongs in the class’s rdfs:comment, and no query result can pass or fail it. C fails the spreadsheet test: one lookup, one column, no joins; if this is the requirement, you do not need an ontology, and pretending otherwise pads the question list without constraining the model. D has no answer shape at all; it is DESCRIBE dressed as a requirement, and any model whatsoever “passes” it, which means it tests nothing. A competency question you cannot fail is a wish.
