geointel/docs/accuracy-program/05-metric-framework.md

# GeoIntel Accuracy Improvement Program — metric framework

- Status: Phase 1 measurement contract
- Scope: Belgium and the Belgian North Sea, with task-specific claims only
- Evidence root: `artifacts/evidence/accuracy/P1/`
- Last updated: 2026-08-01

## 1. Purpose

This document defines how GeoIntel must measure, compare and communicate
accuracy. It is a measurement contract, not an assertion that any current model
meets a release threshold.

The framework has five non-negotiable goals:

1. every reported number is reproducible from checksum-bound inputs;
2. model-selection data stays separate from protected release evidence;
3. aggregate scores cannot hide a failing region or context;
4. missing, undefined or insufficient evidence fails closed;
5. claims never exceed the evaluated task, class, geography, source edition or
   operating configuration.

This framework applies to learned detection and segmentation outputs and to
deterministic spatial QA where a candidate layer is compared with a reference.
It does not turn authoritative GIS functions into machine-learning tasks.

## 2. Current baseline versus future acceptance thresholds

### 2.1 Baselines that exist now

GeoIntel currently has a deterministic QA/QC code-regression baseline in
`fixtures/golden/golden_qa_benchmarks.json`. At IoU `0.5`, its frozen scenarios
include:

| Scenario | Precision | Recall | F1 | Mean matched IoU |
|---|---:|---:|---:|---:|
| partial match | 0.5 | 0.5 | 0.5 | 0.8339768339761133 |
| perfect match | 1.0 | 1.0 | 1.0 | 1.0 |
| no overlap | 0.0 | 0.0 | undefined (`null`) | undefined (`null`) |
| multipolygon match | 1.0 | 1.0 | 1.0 | 1.0 |

These values prove regression stability of the fixture matcher. They do **not**
measure production-model accuracy, Belgium-wide generalisation, source quality
or human acceptability.

The runtime GPU/model smoke evidence proves only that a particular asset can be
loaded and invoked under the recorded configuration. It is not an accuracy
benchmark. Historical calibration reports and persisted `Metric` rows are
evidence candidates only when their complete lineage, split role, evaluator
version and inputs can be reproduced.

Several scripts contain historical or provisional numeric defaults, including
the gate defaults in `scripts/assess_belgium_building_training_iteration.py`.
Those implementation defaults are not automatically approved Accuracy Program
release thresholds. Reports using them must label the gate source and version
as `legacy` or `provisional`; they cannot be relabelled as a Phase 1 contract.

At the Phase 1 evidence cut, a canonical release-grade national model-quality
baseline is therefore `not_established`. No existing number may be presented as
a Belgium-wide acceptance result until the data, split, evaluator and review
requirements below are satisfied.

### 2.2 Thresholds that must be frozen later

All new promotion floors, ceilings, non-inferiority margins, sample minima,
confidence levels and latency budgets are `TBD`. They must be proposed from a
reviewed incumbent baseline and operational requirements, approved and
versioned **before** the protected test set is opened.

The freeze record must identify:

- task, class ontology and claim scope;
- dataset and split-manifest hashes;
- incumbent and candidate model hashes;
- exact preprocessing, tile, overlap, confidence and NMS configuration;
- evaluator version and object-matching policy;
- metric, aggregation level, direction and threshold;
- required strata and minimum evaluable support;
- confidence-interval or paired-comparison rule;
- runtime hardware and latency measurement protocol;
- approver, timestamp and immutable gate-definition hash.

Seeing test results can never justify relaxing a threshold. A changed threshold
creates a new, independently approved evaluation protocol and requires evidence
that remains independent of the change.

## 3. Evaluation unit and identity contract

Every metric record must be traceable to the following identity tuple:

```text
task + class_ontology_version + model_sha256 + dataset_manifest_sha256
+ split_manifest_sha256 + evaluator_version + inference_config_sha256
+ reference_source_version + imagery_source_version + run_id
```

The evidence record must also contain the Git commit, container image digest,
Python/PyTorch/CUDA/driver versions, random seed, deterministic-mode flags,
hardware, timestamps and command arguments. A metric without this tuple is
`unverifiable`, not an approximate success.

All geometries must be validated and transformed to a declared common CRS
before matching. Distance, area and boundary metrics require a documented
metric/equal-area CRS appropriate to the AOI. CRS assumptions, geometry repair,
clipping and ignored boundary zones must be counted and reported.

## 4. Error taxonomy

Each reviewed error receives one primary code and any applicable secondary
codes. Counts must be available by sample, AOI, region and context. Free text
may explain an error but cannot replace a code.

### 4.1 Data and lineage errors (`D-*`)

- `D-MISSING-PROVENANCE`: missing source, edition, acquisition time, licence,
  checksum or transformation lineage.
- `D-STALE-SOURCE`: imagery or reference is outside its declared validity
  period.
- `D-TEMPORAL-MISMATCH`: the labelled object and image do not represent the
  same observable time.
- `D-CRS-MISSING` / `D-CRS-WRONG`: CRS is absent, misdeclared or transformed
  incorrectly.
- `D-ALIGNMENT`: systematic or local image/reference displacement.
- `D-INVALID-GEOMETRY`: empty, corrupt, self-intersecting or otherwise invalid
  geometry, including an unrecorded repair.
- `D-RESOLUTION`: object is below the declared resolvable pixel/ground size or
  input resolution differs from the model contract.
- `D-LICENCE-SCOPE`: source use or redistribution cannot be proven.
- `D-CACHE-STALE`: cached content does not match the requested source/version.

### 4.2 Label and ontology errors (`L-*`)

- `L-MISSING`: visible in-scope object has no label.
- `L-SPURIOUS`: label has no visible in-scope object.
- `L-CLASS`: wrong canonical class or source-to-canonical mapping.
- `L-EXTENT`: polygon/box extent is materially wrong.
- `L-INSTANCE-SPLIT`: one real instance is labelled as multiple instances.
- `L-INSTANCE-MERGE`: distinct instances are merged.
- `L-AMBIGUOUS`: imagery/reference cannot support a defensible decision.
- `L-EXCLUSION`: excluded ontology item was included, or a valid item was
  excluded.
- `L-REVIEW-MISSING`: required human review or adjudication is absent.

### 4.3 Split and duplication errors (`S-*`)

- `S-EXACT-DUPLICATE`: byte/content-equivalent image, label or derived tile
  crosses split boundaries.
- `S-NEAR-DUPLICATE`: materially identical view or transformed derivative
  crosses split boundaries.
- `S-SPATIAL-OVERLAP`: AOIs, tile context or instances overlap across protected
  split groups.
- `S-TEMPORAL-FAMILY`: repeated acquisitions of the same site leak site identity
  or labels across splits without an approved temporal generalisation design.
- `S-SOURCE-FAMILY`: provider mosaics or parent rasters are divided in a way
  that leaks shared context.
- `S-PROTECTED-USE`: calibration, test or background-test evidence influenced
  fitting, sampling, augmentation, architecture, threshold or NMS selection.
- `S-MANIFEST-MUTATION`: an evaluated split or its role changed after freeze.

Any `S-*` error invalidates the affected comparative evaluation until a new
independent split is frozen.

### 4.4 Model-output errors (`M-*`)

- `M-FP-BACKGROUND`: detection on a pure-background sample.
- `M-FP-CONFUSER`: detection on a named hard-negative/confuser.
- `M-FP-DUPLICATE`: multiple retained predictions for one reference instance.
- `M-FP-LOCALISATION`: predicted instance overlaps a reference but misses the
  frozen match criterion.
- `M-FN-MISSED`: resolvable in-scope reference instance is absent.
- `M-FN-SUPPRESSED`: valid instance is removed by confidence, NMS, containment
  or post-processing.
- `M-CLASS`: prediction is assigned to the wrong class.
- `M-BOUNDARY`: segmentation boundary is materially displaced despite object
  detection.
- `M-AREA-BIAS`: predicted geometry has systematic area over- or
  under-estimation.
- `M-MISCALIBRATED`: confidence does not correspond to empirical correctness.
- `M-OOD`: sample is outside the declared training/evaluation domain.

False positives and false negatives must retain evidence links to prediction
and reference IDs, source tiles and review decisions.

### 4.5 Pipeline and claim errors (`P-*`, `C-*`)

- `P-FALLBACK`: mock, heuristic, alternate source or stale cache replaced the
  declared path.
- `P-NONDETERMINISTIC`: rerun differences exceed the frozen reproducibility
  policy without explanation.
- `P-PARTIAL`: missing tile/chunk/output was silently omitted.
- `P-UNIT`: metres, degrees, pixels, CRS units or resolution were confused.
- `P-RUNTIME`: OOM, timeout, truncation or worker failure changed the evaluated
  output.
- `C-OVERCLAIM`: wording exceeds evaluated task/scope/data.
- `C-MISSING-DENOMINATOR`: a rate is published without counts/support.
- `C-UNSUPPORTED-AGGREGATE`: an aggregate hides missing or failed strata.
- `C-UNTRACEABLE`: result cannot be tied to the identity tuple in section 3.

## 5. Object-level metrics

### 5.1 Matching contract

Metrics are meaningless without a matching policy. Every report must specify:

- supported geometry types and class matching;
- IoU thresholds;
- confidence ordering;
- one-to-one matching algorithm and tie-breaking;
- boundary clipping/ignore policy;
- treatment of difficult, ambiguous and below-resolution labels;
- duplicate-suppression configuration.

The current `QaService` performs one-to-one greedy polygon matching at a
configurable IoU threshold (default `0.5`) and conditions mean IoU on matched
objects. This is a deterministic spatial QA baseline when input order and
evaluator version are fixed, but it is not automatically COCO AP.

For detector PR/AP evaluation, predictions must be sorted by confidence and
matched one-to-one at each IoU threshold using a frozen COCO-compatible policy.
For fixed-threshold spatial comparison, a separately named matcher may use a
deterministic maximum-weight one-to-one assignment. Results from different
matchers must never be combined under the same metric key.

### 5.2 Counts and rates

For an evaluable class and stratum:

```text
precision = TP / (TP + FP)
recall    = TP / (TP + FN)
F1        = 2 * precision * recall / (precision + recall)
IoU       = intersection_area / union_area
```

Always publish `TP`, `FP`, `FN`, prediction count and reference count beside the
rates. Undefined denominators produce `null`, never zero and never pass. In
particular:

- no predictions makes precision undefined;
- no references makes recall undefined;
- zero precision plus zero recall keeps F1 undefined under the current QA
  service semantics;
- pure-background performance is evaluated with explicit FP/background metrics,
  not fabricated precision or recall;
- mean matched IoU is conditional on matches and must not be used alone because
  it excludes FP and FN.

### 5.3 Precision-recall and average precision

Report the full precision-recall curve and its confidence thresholds. Required
detector summaries are:

- `AP50`: area under the interpolated PR curve at IoU 0.50;
- `AP50-95`: mean AP across IoU 0.50 through 0.95 in steps of 0.05;
- per-class AP and macro AP for multi-class tasks;
- recall at the frozen operating point;
- precision at the frozen operating point.

AP must be calculated before choosing a production confidence threshold. A
single-threshold F1 value cannot be called AP. AP values from a training
framework are accepted only when evaluator version, input set, class mapping
and settings are recorded and independently reproducible.

### 5.4 Localisation and geometry

Report the distribution, not only the mean, of matched IoU: count, median,
quartiles and lower-tail quantiles. Segmentation candidates additionally need
pixel/geometry IoU, Dice, boundary distance/score, area bias and topology-error
counts under a frozen geometry protocol. Metric thresholds for those measures
remain `TBD` until the segmentation label contract exists.

Object results must be stratified by predeclared size bands derived from ground
area and/or source-image pixels. Band boundaries are `TBD` and must be frozen
from resolution and ontology rules before candidate results are inspected.

## 6. Tile-level metrics

Tile metrics expose operational failure modes that object-level micro-averages
can hide:

- tile count, evaluable tile count and excluded-tile reasons;
- positive/empty tile counts;
- fraction of pure-empty tiles with one or more predictions;
- FP count per pure-empty tile and per square kilometre;
- fraction of positive tiles with at least one FN;
- per-tile precision, recall and F1 where defined;
- detection/reference count error per tile;
- duplicate predictions created at tile overlaps;
- tile processing latency, throughput and peak memory;
- tile-level error/uncertainty score for review routing.

Report macro distributions across tiles as diagnostics. Do not average undefined
tile rates into zero, and do not let a large number of easy empty tiles dominate
the primary object metric.

## 7. AOI-level metrics

The AOI is the primary unit for paired operational comparison. For each AOI,
publish:

- TP, FP, FN, precision, recall, F1 and matched-IoU distribution;
- reference and predicted object counts and signed/absolute count error;
- reference and predicted geometry area and signed/absolute area bias when
  geometry outputs support it;
- pure-background/context-negative errors;
- excluded area/objects and reasons;
- end-to-end runtime, failed/retried tiles and completeness;
- taxonomy counts and links to visual evidence.

Candidate-versus-incumbent deltas must be paired on the exact same AOIs,
references and inference contract. Macro AOI results give each AOI equal weight;
micro results pool TP/FP/FN. Both are reported and explicitly named. The primary
aggregation and any non-inferiority margin remain `TBD` until frozen in the
gate definition.

## 8. Region-, context- and portfolio-level metrics

At minimum, building-model evidence must report Flanders, Wallonia and Brussels
separately. The Belgian North Sea is not an implicit building-detector claim;
offshore tasks need their own ontology and evidence.

Required stratification dimensions, where applicable, are:

- region and provider/source edition;
- urban, suburban, ribbon development, rural/village, farm, industrial and
  other frozen context families;
- dense/occluded/touching-object contexts;
- pure background and named hard-negative families;
- object-size band and input ground resolution;
- imagery period/season and reference-to-imagery time difference;
- AOI boundary/interior and tile-edge/interior;
- native class/source subtype and label-review state;
- normal domain versus declared OOD challenge set.

For every required stratum report support, micro metrics, macro AOI metrics and
uncertainty intervals. Also report the worst evaluable required stratum. A
portfolio aggregate is valid only if every mandatory stratum has sufficient
predeclared support; otherwise its state is `not_evaluable`.

Intersectional slices should be added where error evidence warrants them, but
post-hoc slices are diagnostic and cannot replace the frozen primary analysis.
Small groups stay visible with an `insufficient_support` label; they are not
silently pooled into a better-performing group.

## 9. Confidence calibration

Model confidence is not accuracy and is not a complete uncertainty estimate.
Calibration is evaluated on the frozen calibration split after one-to-one
matching and before protected-test access.

### 9.1 Expected calibration error

For prediction bins `B_m` frozen before evaluation:

```text
ECE = sum_m (|B_m| / n) * |accuracy(B_m) - confidence(B_m)|
```

The binning method, boundaries, empty-bin handling and minimum observations must
be recorded. Report a reliability diagram and per-bin counts. A predefined
fixed-bin ECE is the comparable metric; adaptive-bin ECE may be reported only as
a labelled diagnostic. ECE is paired with recall because missed references have
no prediction confidence and are invisible to prediction-only ECE.

### 9.2 Brier score

For each scored prediction, define `y=1` only when it is the retained match under
the frozen policy, otherwise `y=0`:

```text
Brier = mean((confidence - y)^2)
```

Report Brier overall and by required stratum. The exact construction of the
prediction set must be frozen; Brier does not replace FN/recall reporting.

Calibration fitting may use only calibration data. If temperature scaling,
isotonic regression or another mapping is used, its fitted parameters and code
version become part of the immutable inference configuration. Test data may
evaluate that mapping but may not refit it.

## 10. Latency, capacity and reliability

Accuracy promotion also requires a reproducible operational profile on the
declared server GPU. Record separately:

- cold-start/model-load time;
- warm model-only inference time per tile;
- preprocessing, inference, post-processing and persistence time;
- end-to-end AOI latency excluding and including queue wait;
- median, p95, p99 and maximum latency with sample counts;
- tiles/second and square kilometres/minute at the recorded resolution;
- peak allocated/reserved VRAM and host RAM;
- batch size, tile size, overlap, worker concurrency and retry count;
- OOM, timeout, truncation, partial-output and failed-job rates.

Warm-up count, timing clock, synchronisation method, hardware, driver and
background load must be fixed. A local CPU run and an RTX 4080 SUPER CUDA run
are different baselines. Latency budgets are `TBD`; current smoke timings may be
reported as observations only.

## 11. Uncertainty and selective review

Uncertainty evidence may include confidence, entropy/margin where meaningful,
test-time-augmentation disagreement, ensemble variance, spatial disagreement
and OOD scores. Every method must state what variation it measures; confidence
alone may not be labelled epistemic uncertainty.

For any abstain/review policy, report:

- retained coverage versus error/risk curve;
- error rate in auto-accepted, human-review and rejected bands;
- review volume per AOI/region/context;
- area under the risk-coverage curve as a diagnostic;
- failure cases where high-confidence predictions are wrong;
- threshold source and calibration-only selection evidence.

Uncertainty thresholds are `TBD`. Uncertainty may route work to a human; it must
not create self-training labels or silently suppress output.

## 12. Strict split and deduplication rules

### 12.1 Split roles

- `train`: fitting and training-time augmentation only;
- `val`: epoch/checkpoint/architecture selection only;
- `calibration`: confidence, tiling, overlap, NMS and calibration mapping only;
- `test`: sealed, one-time promotion evidence for a frozen candidate family;
- `background-test`: sealed, difficult and pure-background release evidence.

No test or background-test result may influence fitting, sampling weights,
label policy, architecture, hyperparameters, post-processing or gate values. If
it does, that split is retired from independent testing, the candidate becomes
a new family and a new untouched protected split is required.

### 12.2 Grouping before splitting

Deduplication and grouping happen before split assignment. The atomic group must
keep together:

- a parent AOI plus every derived/overlapping tile;
- all tiles sharing an object instance;
- exact image/label hashes and transformed copies;
- perceptual near-duplicate families;
- repeated views whose shared spatial context exceeds the declared independence
  buffer;
- source-raster/mosaic or temporal families when they would leak scene identity.

Train, val, calibration, test and background-test AOIs must be spatially
disjoint, including a buffer larger than the maximum tile context used by any
candidate. The buffer value and near-duplicate thresholds are `TBD` but must be
frozen before corpus generation, not chosen after metric inspection.

### 12.3 Required leakage audits

The frozen split evidence must contain:

- exact SHA-256 duplicate matrix for raw image, processed image and label;
- perceptual-image and label-geometry near-duplicate audit;
- buffered AOI/tile intersection audit in a metric CRS;
- shared native feature/source identifier audit;
- parent raster, acquisition and temporal-family audit;
- tile-versus-manifest role consistency audit;
- counts and explicit disposition for every collision.

All cross-split collisions must be zero unless a written experimental design
defines a non-independence group and assigns that whole group to one split. A
leakage audit that did not run, timed out or lacks inputs is a failed gate.

## 13. Human-review metrics

Automated geometry checks do not replace representative human review. Corpus
labels and model outputs need separate review records.

### 13.1 Label-corpus review

Report:

- planned, rendered, reviewed and adjudicated sample/object counts;
- completion rate by region, context, provider, size band, split and label
  source;
- contact-sheet/render failures and unreadable samples;
- prevalence of every `D-*` and `L-*` finding;
- accepted, corrected, excluded and ambiguous counts;
- single-review and independent double-review counts;
- raw reviewer agreement, class-wise agreement and an approved chance-corrected
  agreement statistic such as Cohen's kappa or Krippendorff's alpha;
- disagreement/adjudication rate and unresolved critical findings.

Sampling quotas, double-review fraction, agreement statistic and acceptance
thresholds are `TBD` and must be frozen before reviewers see model results.
Protected test labels should be reviewed independently of candidate predictions.

### 13.2 Output-error review

For each candidate, use a frozen stratified sample that includes TP, FP, FN,
low-confidence, high-confidence, tile-edge, dense, temporal-mismatch and OOD
cases. Reviewers record taxonomy codes and severity, not only approve/reject.
Report reviewer agreement, adjudication, severe-error prevalence and error
distribution by stratum.

Review UI/version, imagery/reference layers, zoom/scale and reviewer identity or
pseudonymous ID must be retained. A generated contact sheet with zero completed
review decisions is `unreviewed`, not passed.

## 14. Statistical reporting

All rates include numerator, denominator and support. Confidence intervals use
a predeclared method appropriate to the unit: object-level bootstrap must not
pretend correlated tiles are independent. Prefer cluster bootstrap or paired
analysis at AOI/group level for model comparisons.

Report:

- point estimate and interval;
- number of independent AOIs/groups;
- micro and macro aggregation;
- paired candidate-minus-incumbent delta by AOI;
- worst required stratum and its support;
- missing/excluded evidence and reason;
- multiplicity policy for any formal multi-stratum hypothesis claims.

The confidence level, bootstrap seed/replicates and non-inferiority margins are
`TBD` until the gate specification is frozen. Descriptive diagnostics must not
be presented as confirmatory release evidence.

## 15. Acceptance-gate semantics

Every gate returns exactly one state:

- `pass`: complete evidence meets the frozen rule;
- `fail`: complete evidence violates the frozen rule;
- `not_evaluable`: evidence is missing, invalid, leaked or below frozen support.

`not_evaluable` blocks promotion exactly like `fail`; it is never coerced to
zero, ignored or averaged away.

A promotion decision is a logical AND across all mandatory gate families:

1. identity, provenance, licence and immutable manifest;
2. zero unresolved split leakage and duplicate-family violations;
3. automated data/label/geometry integrity;
4. completed representative human review and adjudication;
5. frozen object/AOI/region/context accuracy gates;
6. pure-background and hard-negative gates;
7. calibration and uncertainty/review-policy gates;
8. paired incumbent comparison and reproducibility rerun;
9. GPU latency, capacity and failure-rate gates;
10. model card, scope limitation, rollback and live shadow evidence.

No weighted composite score may compensate for a failed mandatory gate. The
aggregate and every mandatory region/context gate must pass independently.
Candidate selection and operating-point selection use calibration evidence
only. The protected test is evaluated once with the frozen configuration. A
rerun is allowed only to reproduce the same immutable computation or to resolve
a proven execution failure without inspecting/using partial results.

Gate output must include machine-readable rule IDs, observed values, expected
direction/range, support, evidence paths and hashes. Manual override cannot
change a metric result; an exceptional operational decision must remain a
separate signed record and cannot widen the accuracy claim.

## 16. No-claim rules

GeoIntel must not claim that a model is “fully trained”, “validated”,
“production accurate”, “Belgium-wide” or equivalent when any of the following
is true:

- only model-file presence, load success, a GPU smoke or output existence was
  proven;
- results come from train/val data, an unfrozen calibration set or leaked test
  evidence;
- the dataset, model, evaluator or inference configuration lacks hashes;
- required region/context support or representative human review is missing;
- a metric is undefined/null, a job is partial or a fallback/mock/heuristic was
  used;
- only aggregate performance is shown while a required stratum is absent or
  failing;
- precision/recall/AP is reported without match policy and TP/FP/FN/support;
- confidence is presented as probability of correctness without calibration
  evidence;
- a deterministic fixture benchmark is presented as production-model quality;
- the claim is extrapolated to another class, source, imagery period,
  resolution, region, segmentation task or deterministic GIS function;
- test results were used to choose the next training data or configuration.

Permitted wording must be evidence-bounded, for example: “candidate X achieved
the recorded metrics on frozen portfolio Y under configuration Z.” It must name
limitations and may not imply untested generalisation.

## 17. Required machine-readable outputs

Each future baseline or candidate evaluation must retain, without overwriting
earlier evidence:

```text
evaluation-contract.json
input-manifest.json
split-and-leakage-audit.json
object-metrics.json
tile-metrics.json
aoi-metrics.json
stratified-metrics.json
calibration-metrics.json
latency-and-reliability.json
human-review-summary.json
error-taxonomy.json
candidate-vs-incumbent.json
acceptance-gates.json
evidence-manifest.json
```

`evidence-manifest.json` must SHA-256 hash every retained input and output. Any
amendment is additive, versioned and linked to its predecessor. Raw records
must remain available so every aggregate can be recomputed.

## 18. Phase 2 metric implementation sequence

1. freeze evaluator schemas, taxonomy codes and undefined-value semantics;
2. implement strict group split and exact/near-duplicate audits;
3. validate CRS/alignment/label lineage and complete human corpus review;
4. freeze an incumbent portfolio and run a reproducible baseline twice;
5. derive and approve numeric thresholds from reviewed baseline distributions
   and operational requirements without protected-test access;
6. calibrate candidate operating points on calibration data only;
7. execute the one-time protected test and background-test gates;
8. publish scope-bounded evidence or retain `not_configured`/current limited
   scope when any gate is not evaluable.

Until those steps pass, this framework authorises measurement and remediation,
not a widened accuracy claim.