Limits, biases and how this analysis could be wrong

This lesson outlines the key caveats and limitations of the measurements and assumptions made in the course’s analysis of space object cataloguing. It highlights how specific technical choices, data constraints, and modelling decisions can affect the validity and generalisability of results. The goal is to make explicit the conditions under which the analysis is valid, and where further work is needed.

Witness reliability and stale data

The analysis relies on witness evidence where operator-reported status is treated as ground truth. This introduces a dependency on the timeliness and accuracy of status codes, which can become outdated. In practice, this is mitigated by the scale of witness sets, thousands of witnesses can survive realistic error rates. However, this dependence on stale data is a critical limitation that must be acknowledged.

Arbitrary regime boundaries and sensitivity to thresholds

Key boundaries such as the 2,000 km altitude, 5-degree inclination box, and graveyard perigee are conventions rather than hard limits. The repository explicitly reports these values to allow for recalibration by users. For instance, the geostationary conflation rate shifts by only fractions of a point across reasonable thresholds, but the conclusion does not hinge on any single boundary value. This demonstrates the sensitivity of results to arbitrary choices in data framing.

Limited catalogue coverage and data gaps

The space catalogue used in the analysis is a single, curated source. It contains gaps such as 16,489 unknown statuses, and small debris is not detectable. This single-source approach cannot identify systematic errors within the source itself. Cross-catalogue validation methods, such as those used in DISCOS, are required to assess the robustness of findings and to detect inconsistencies across datasets.

Ontological limitations in measurement

The analysis was conducted using one specific ontology: SSAO. Its near-zero refutation power is consistent with patterns seen in other industrial vocabularies. However, other space-related ontologies and profiles exist, and they have not been evaluated for this study. A full measurement of those alternatives would require similar effort, around thirty lines of code or more, and is necessary before any conclusions can be generalised.

Matcher failure modes and future validation

The “mutants” used in the analysis were designed to model failure modes of matchers, not to be sampled from a live system. Actual lexical, embedding, and LLM-based matchers have not yet been evaluated on the same test pair. The next step involves running these full matchers and scoring outputs through both channels. The experimental harness is designed to accept arbitrary candidate sets, enabling scalable testing of these systems.

Meta-lesson: The importance of honest error accounting

The core meta-lesson, consistent across this course series, is that any analysis that does not explicitly list its potential ways of being wrong is not a rigorous analysis. The assumption ledger, far from being a weak point, is the entire point of the work. The integrity of the analysis is not undermined by listing these limitations, it is strengthened by it.

Reference

Sources and further reading

• Assumption notes in the rule scripts

Checkpoint: test your judgment