Home Architecture

Architecture

Cairn is built like a data pipeline, not a dashboard. Every figure on this site is the output of one fixed chain: an official source, pinned by hash, transformed by reviewed code, into a versioned warehouse the site reads from. This is the structural view — where each stage lives in the repo. For the provenance and limitations of individual figures, see Methodology & sources →.

The pipeline

1 · Official sources
CBS StatLine 85669NED · euets.info · EEA Union Registry
Public, official datasets. Nothing is hand-entered or estimated.
↓
2 · Ingestion
ingestion/*.py
Each run writes a new, versioned raw path. Raw data is immutable — never overwritten.
↓
3 · Manifest
sources/cbs|euets|eea/manifest.yml
Each release is pinned by sha256 in an append-only manifest. A data change without a manifest change is impossible.
↓
4 · Transform (dbt)
transform/ · seed: sector_mapping_cbs
Mappings are code. The CBS category → NACE mapping is a reviewed seed, so its numeric impact shows up in a CI benchmark diff.
↓
5 · Warehouse
cairn.duckdb (R2-pinned)
Materialised by scripts/materialize_warehouse.py. The single source the exports and the site read from.
↓
6 · Exports
scripts/export_esrs_e1.py · export_mart.py
The ESRS E1 bundle: CSV plus a meta.json audit trail (source pin, methodology commit, warehouse version, SHA256 of the CSV).
↓
7 · Site
site/ (Evidence)
Reads only the warehouse. This site.

The CI rail

Running alongside every stage, not as an afterthought:

  • A weekly job re-verifies the pinned raw files against their manifest hashes — a mismatch is an integrity alarm, not flakiness.
  • A reconciliation test checks the ETS total against the EEA Union Registry aggregate; both derive from the EUTL and match to ~0.02%.
  • scripts/verify_reproducibility.py confirms the warehouse rebuilds identically from the pinned sources.
  • scripts/benchmark_diff.py surfaces the numeric impact of any mapping change in the pull request that makes it.

The guarantee

Because each stage consumes only a hash-pinned input from the stage before it, no number can drift from its source without a manifest change and a failing CI build. That property — not the benchmark itself — is what separates Cairn from a dashboard.

See Methodology & sources → for per-figure provenance, or the repository for the code behind each stage.