geointel/docs/accuracy-program/05-metric-framework.md

GeoIntel Accuracy Improvement Program — metric framework

• Status: Phase 1 measurement contract
• Scope: Belgium and the Belgian North Sea, with task-specific claims only
• Evidence root: artifacts/evidence/accuracy/P1/
• Last updated: 2026-08-01

1. Purpose

This document defines how GeoIntel must measure, compare and communicate accuracy. It is a measurement contract, not an assertion that any current model meets a release threshold.

The framework has five non-negotiable goals:

1. every reported number is reproducible from checksum-bound inputs;
2. model-selection data stays separate from protected release evidence;
3. aggregate scores cannot hide a failing region or context;
4. missing, undefined or insufficient evidence fails closed;
5. claims never exceed the evaluated task, class, geography, source edition or operating configuration.

This framework applies to learned detection and segmentation outputs and to deterministic spatial QA where a candidate layer is compared with a reference. It does not turn authoritative GIS functions into machine-learning tasks.

2. Current baseline versus future acceptance thresholds

2.1 Baselines that exist now

GeoIntel currently has a deterministic QA/QC code-regression baseline in fixtures/golden/golden_qa_benchmarks.json. At IoU 0.5, its frozen scenarios include:

Scenario	Precision	Recall	F1	Mean matched IoU
partial match	0.5	0.5	0.5	0.8339768339761133
perfect match	1.0	1.0	1.0	1.0
no overlap	0.0	0.0	undefined (null)	undefined (null)
multipolygon match	1.0	1.0	1.0	1.0

These values prove regression stability of the fixture matcher. They do not measure production-model accuracy, Belgium-wide generalisation, source quality or human acceptability.

The runtime GPU/model smoke evidence proves only that a particular asset can be loaded and invoked under the recorded configuration. It is not an accuracy benchmark. Historical calibration reports and persisted Metric rows are evidence candidates only when their complete lineage, split role, evaluator version and inputs can be reproduced.

Several scripts contain historical or provisional numeric defaults, including the gate defaults in scripts/assess_belgium_building_training_iteration.py. Those implementation defaults are not automatically approved Accuracy Program release thresholds. Reports using them must label the gate source and version as legacy or provisional; they cannot be relabelled as a Phase 1 contract.

At the Phase 1 evidence cut, a canonical release-grade national model-quality baseline is therefore not_established. No existing number may be presented as a Belgium-wide acceptance result until the data, split, evaluator and review requirements below are satisfied.

2.2 Thresholds that must be frozen later

All new promotion floors, ceilings, non-inferiority margins, sample minima, confidence levels and latency budgets are TBD. They must be proposed from a reviewed incumbent baseline and operational requirements, approved and versioned before the protected test set is opened.

The freeze record must identify:

• task, class ontology and claim scope;
• dataset and split-manifest hashes;
• incumbent and candidate model hashes;
• exact preprocessing, tile, overlap, confidence and NMS configuration;
• evaluator version and object-matching policy;
• metric, aggregation level, direction and threshold;
• required strata and minimum evaluable support;
• confidence-interval or paired-comparison rule;
• runtime hardware and latency measurement protocol;
• approver, timestamp and immutable gate-definition hash.

Seeing test results can never justify relaxing a threshold. A changed threshold creates a new, independently approved evaluation protocol and requires evidence that remains independent of the change.

3. Evaluation unit and identity contract

Every metric record must be traceable to the following identity tuple:

task + class_ontology_version + model_sha256 + dataset_manifest_sha256
+ split_manifest_sha256 + evaluator_version + inference_config_sha256
+ reference_source_version + imagery_source_version + run_id

The evidence record must also contain the Git commit, container image digest, Python/PyTorch/CUDA/driver versions, random seed, deterministic-mode flags, hardware, timestamps and command arguments. A metric without this tuple is unverifiable, not an approximate success.

All geometries must be validated and transformed to a declared common CRS before matching. Distance, area and boundary metrics require a documented metric/equal-area CRS appropriate to the AOI. CRS assumptions, geometry repair, clipping and ignored boundary zones must be counted and reported.

4. Error taxonomy

Each reviewed error receives one primary code and any applicable secondary codes. Counts must be available by sample, AOI, region and context. Free text may explain an error but cannot replace a code.

4.1 Data and lineage errors (D-*)

• D-MISSING-PROVENANCE: missing source, edition, acquisition time, licence, checksum or transformation lineage.
• D-STALE-SOURCE: imagery or reference is outside its declared validity period.
• D-TEMPORAL-MISMATCH: the labelled object and image do not represent the same observable time.
• D-CRS-MISSING / D-CRS-WRONG: CRS is absent, misdeclared or transformed incorrectly.
• D-ALIGNMENT: systematic or local image/reference displacement.
• D-INVALID-GEOMETRY: empty, corrupt, self-intersecting or otherwise invalid geometry, including an unrecorded repair.
• D-RESOLUTION: object is below the declared resolvable pixel/ground size or input resolution differs from the model contract.
• D-LICENCE-SCOPE: source use or redistribution cannot be proven.
• D-CACHE-STALE: cached content does not match the requested source/version.

4.2 Label and ontology errors (L-*)

• L-MISSING: visible in-scope object has no label.
• L-SPURIOUS: label has no visible in-scope object.
• L-CLASS: wrong canonical class or source-to-canonical mapping.
• L-EXTENT: polygon/box extent is materially wrong.
• L-INSTANCE-SPLIT: one real instance is labelled as multiple instances.
• L-INSTANCE-MERGE: distinct instances are merged.
• L-AMBIGUOUS: imagery/reference cannot support a defensible decision.
• L-EXCLUSION: excluded ontology item was included, or a valid item was excluded.
• L-REVIEW-MISSING: required human review or adjudication is absent.

4.3 Split and duplication errors (S-*)

• S-EXACT-DUPLICATE: byte/content-equivalent image, label or derived tile crosses split boundaries.
• S-NEAR-DUPLICATE: materially identical view or transformed derivative crosses split boundaries.
• S-SPATIAL-OVERLAP: AOIs, tile context or instances overlap across protected split groups.
• S-TEMPORAL-FAMILY: repeated acquisitions of the same site leak site identity or labels across splits without an approved temporal generalisation design.
• S-SOURCE-FAMILY: provider mosaics or parent rasters are divided in a way that leaks shared context.
• S-PROTECTED-USE: calibration, test or background-test evidence influenced fitting, sampling, augmentation, architecture, threshold or NMS selection.
• S-MANIFEST-MUTATION: an evaluated split or its role changed after freeze.

Any S-* error invalidates the affected comparative evaluation until a new independent split is frozen.

4.4 Model-output errors (M-*)

• M-FP-BACKGROUND: detection on a pure-background sample.
• M-FP-CONFUSER: detection on a named hard-negative/confuser.
• M-FP-DUPLICATE: multiple retained predictions for one reference instance.
• M-FP-LOCALISATION: predicted instance overlaps a reference but misses the frozen match criterion.
• M-FN-MISSED: resolvable in-scope reference instance is absent.
• M-FN-SUPPRESSED: valid instance is removed by confidence, NMS, containment or post-processing.
• M-CLASS: prediction is assigned to the wrong class.
• M-BOUNDARY: segmentation boundary is materially displaced despite object detection.
• M-AREA-BIAS: predicted geometry has systematic area over- or under-estimation.
• M-MISCALIBRATED: confidence does not correspond to empirical correctness.
• M-OOD: sample is outside the declared training/evaluation domain.

False positives and false negatives must retain evidence links to prediction and reference IDs, source tiles and review decisions.

4.5 Pipeline and claim errors (P-*, C-*)

• P-FALLBACK: mock, heuristic, alternate source or stale cache replaced the declared path.
• P-NONDETERMINISTIC: rerun differences exceed the frozen reproducibility policy without explanation.
• P-PARTIAL: missing tile/chunk/output was silently omitted.
• P-UNIT: metres, degrees, pixels, CRS units or resolution were confused.
• P-RUNTIME: OOM, timeout, truncation or worker failure changed the evaluated output.
• C-OVERCLAIM: wording exceeds evaluated task/scope/data.
• C-MISSING-DENOMINATOR: a rate is published without counts/support.
• C-UNSUPPORTED-AGGREGATE: an aggregate hides missing or failed strata.
• C-UNTRACEABLE: result cannot be tied to the identity tuple in section 3.

5. Object-level metrics

5.1 Matching contract

Metrics are meaningless without a matching policy. Every report must specify:

• supported geometry types and class matching;
• IoU thresholds;
• confidence ordering;
• one-to-one matching algorithm and tie-breaking;
• boundary clipping/ignore policy;
• treatment of difficult, ambiguous and below-resolution labels;
• duplicate-suppression configuration.

The current QaService performs one-to-one greedy polygon matching at a configurable IoU threshold (default 0.5) and conditions mean IoU on matched objects. This is a deterministic spatial QA baseline when input order and evaluator version are fixed, but it is not automatically COCO AP.

For detector PR/AP evaluation, predictions must be sorted by confidence and matched one-to-one at each IoU threshold using a frozen COCO-compatible policy. For fixed-threshold spatial comparison, a separately named matcher may use a deterministic maximum-weight one-to-one assignment. Results from different matchers must never be combined under the same metric key.

5.2 Counts and rates

For an evaluable class and stratum:

precision = TP / (TP + FP)
recall    = TP / (TP + FN)
F1        = 2 * precision * recall / (precision + recall)
IoU       = intersection_area / union_area

Always publish TP, FP, FN, prediction count and reference count beside the rates. Undefined denominators produce null, never zero and never pass. In particular:

• no predictions makes precision undefined;
• no references makes recall undefined;
• zero precision plus zero recall keeps F1 undefined under the current QA service semantics;
• pure-background performance is evaluated with explicit FP/background metrics, not fabricated precision or recall;
• mean matched IoU is conditional on matches and must not be used alone because it excludes FP and FN.

5.3 Precision-recall and average precision

Report the full precision-recall curve and its confidence thresholds. Required detector summaries are:

• AP50: area under the interpolated PR curve at IoU 0.50;
• AP50-95: mean AP across IoU 0.50 through 0.95 in steps of 0.05;
• per-class AP and macro AP for multi-class tasks;
• recall at the frozen operating point;
• precision at the frozen operating point.

AP must be calculated before choosing a production confidence threshold. A single-threshold F1 value cannot be called AP. AP values from a training framework are accepted only when evaluator version, input set, class mapping and settings are recorded and independently reproducible.

5.4 Localisation and geometry

Report the distribution, not only the mean, of matched IoU: count, median, quartiles and lower-tail quantiles. Segmentation candidates additionally need pixel/geometry IoU, Dice, boundary distance/score, area bias and topology-error counts under a frozen geometry protocol. Metric thresholds for those measures remain TBD until the segmentation label contract exists.

Object results must be stratified by predeclared size bands derived from ground area and/or source-image pixels. Band boundaries are TBD and must be frozen from resolution and ontology rules before candidate results are inspected.

6. Tile-level metrics

Tile metrics expose operational failure modes that object-level micro-averages can hide:

• tile count, evaluable tile count and excluded-tile reasons;
• positive/empty tile counts;
• fraction of pure-empty tiles with one or more predictions;
• FP count per pure-empty tile and per square kilometre;
• fraction of positive tiles with at least one FN;
• per-tile precision, recall and F1 where defined;
• detection/reference count error per tile;
• duplicate predictions created at tile overlaps;
• tile processing latency, throughput and peak memory;
• tile-level error/uncertainty score for review routing.

Report macro distributions across tiles as diagnostics. Do not average undefined tile rates into zero, and do not let a large number of easy empty tiles dominate the primary object metric.

7. AOI-level metrics

The AOI is the primary unit for paired operational comparison. For each AOI, publish:

• TP, FP, FN, precision, recall, F1 and matched-IoU distribution;
• reference and predicted object counts and signed/absolute count error;
• reference and predicted geometry area and signed/absolute area bias when geometry outputs support it;
• pure-background/context-negative errors;
• excluded area/objects and reasons;
• end-to-end runtime, failed/retried tiles and completeness;
• taxonomy counts and links to visual evidence.

Candidate-versus-incumbent deltas must be paired on the exact same AOIs, references and inference contract. Macro AOI results give each AOI equal weight; micro results pool TP/FP/FN. Both are reported and explicitly named. The primary aggregation and any non-inferiority margin remain TBD until frozen in the gate definition.

8. Region-, context- and portfolio-level metrics

At minimum, building-model evidence must report Flanders, Wallonia and Brussels separately. The Belgian North Sea is not an implicit building-detector claim; offshore tasks need their own ontology and evidence.

Required stratification dimensions, where applicable, are:

• region and provider/source edition;
• urban, suburban, ribbon development, rural/village, farm, industrial and other frozen context families;
• dense/occluded/touching-object contexts;
• pure background and named hard-negative families;
• object-size band and input ground resolution;
• imagery period/season and reference-to-imagery time difference;
• AOI boundary/interior and tile-edge/interior;
• native class/source subtype and label-review state;
• normal domain versus declared OOD challenge set.

For every required stratum report support, micro metrics, macro AOI metrics and uncertainty intervals. Also report the worst evaluable required stratum. A portfolio aggregate is valid only if every mandatory stratum has sufficient predeclared support; otherwise its state is not_evaluable.

Intersectional slices should be added where error evidence warrants them, but post-hoc slices are diagnostic and cannot replace the frozen primary analysis. Small groups stay visible with an insufficient_support label; they are not silently pooled into a better-performing group.

9. Confidence calibration

Model confidence is not accuracy and is not a complete uncertainty estimate. Calibration is evaluated on the frozen calibration split after one-to-one matching and before protected-test access.

9.1 Expected calibration error

For prediction bins B_m frozen before evaluation:

ECE = sum_m (|B_m| / n) * |accuracy(B_m) - confidence(B_m)|

The binning method, boundaries, empty-bin handling and minimum observations must be recorded. Report a reliability diagram and per-bin counts. A predefined fixed-bin ECE is the comparable metric; adaptive-bin ECE may be reported only as a labelled diagnostic. ECE is paired with recall because missed references have no prediction confidence and are invisible to prediction-only ECE.

9.2 Brier score

For each scored prediction, define y=1 only when it is the retained match under the frozen policy, otherwise y=0:

Brier = mean((confidence - y)^2)

Report Brier overall and by required stratum. The exact construction of the prediction set must be frozen; Brier does not replace FN/recall reporting.

Calibration fitting may use only calibration data. If temperature scaling, isotonic regression or another mapping is used, its fitted parameters and code version become part of the immutable inference configuration. Test data may evaluate that mapping but may not refit it.

10. Latency, capacity and reliability

Accuracy promotion also requires a reproducible operational profile on the declared server GPU. Record separately:

• cold-start/model-load time;
• warm model-only inference time per tile;
• preprocessing, inference, post-processing and persistence time;
• end-to-end AOI latency excluding and including queue wait;
• median, p95, p99 and maximum latency with sample counts;
• tiles/second and square kilometres/minute at the recorded resolution;
• peak allocated/reserved VRAM and host RAM;
• batch size, tile size, overlap, worker concurrency and retry count;
• OOM, timeout, truncation, partial-output and failed-job rates.

Warm-up count, timing clock, synchronisation method, hardware, driver and background load must be fixed. A local CPU run and an RTX 4080 SUPER CUDA run are different baselines. Latency budgets are TBD; current smoke timings may be reported as observations only.

11. Uncertainty and selective review

Uncertainty evidence may include confidence, entropy/margin where meaningful, test-time-augmentation disagreement, ensemble variance, spatial disagreement and OOD scores. Every method must state what variation it measures; confidence alone may not be labelled epistemic uncertainty.

For any abstain/review policy, report:

• retained coverage versus error/risk curve;
• error rate in auto-accepted, human-review and rejected bands;
• review volume per AOI/region/context;
• area under the risk-coverage curve as a diagnostic;
• failure cases where high-confidence predictions are wrong;
• threshold source and calibration-only selection evidence.

Uncertainty thresholds are TBD. Uncertainty may route work to a human; it must not create self-training labels or silently suppress output.

12. Strict split and deduplication rules

12.1 Split roles

• train: fitting and training-time augmentation only;
• val: epoch/checkpoint/architecture selection only;
• calibration: confidence, tiling, overlap, NMS and calibration mapping only;
• test: sealed, one-time promotion evidence for a frozen candidate family;
• background-test: sealed, difficult and pure-background release evidence.

No test or background-test result may influence fitting, sampling weights, label policy, architecture, hyperparameters, post-processing or gate values. If it does, that split is retired from independent testing, the candidate becomes a new family and a new untouched protected split is required.

12.2 Grouping before splitting

Deduplication and grouping happen before split assignment. The atomic group must keep together:

• a parent AOI plus every derived/overlapping tile;
• all tiles sharing an object instance;
• exact image/label hashes and transformed copies;
• perceptual near-duplicate families;
• repeated views whose shared spatial context exceeds the declared independence buffer;
• source-raster/mosaic or temporal families when they would leak scene identity.

Train, val, calibration, test and background-test AOIs must be spatially disjoint, including a buffer larger than the maximum tile context used by any candidate. The buffer value and near-duplicate thresholds are TBD but must be frozen before corpus generation, not chosen after metric inspection.

12.3 Required leakage audits

The frozen split evidence must contain:

• exact SHA-256 duplicate matrix for raw image, processed image and label;
• perceptual-image and label-geometry near-duplicate audit;
• buffered AOI/tile intersection audit in a metric CRS;
• shared native feature/source identifier audit;
• parent raster, acquisition and temporal-family audit;
• tile-versus-manifest role consistency audit;
• counts and explicit disposition for every collision.

All cross-split collisions must be zero unless a written experimental design defines a non-independence group and assigns that whole group to one split. A leakage audit that did not run, timed out or lacks inputs is a failed gate.

13. Human-review metrics

Automated geometry checks do not replace representative human review. Corpus labels and model outputs need separate review records.

13.1 Label-corpus review

Report:

• planned, rendered, reviewed and adjudicated sample/object counts;
• completion rate by region, context, provider, size band, split and label source;
• contact-sheet/render failures and unreadable samples;
• prevalence of every D-* and L-* finding;
• accepted, corrected, excluded and ambiguous counts;
• single-review and independent double-review counts;
• raw reviewer agreement, class-wise agreement and an approved chance-corrected agreement statistic such as Cohen's kappa or Krippendorff's alpha;
• disagreement/adjudication rate and unresolved critical findings.

Sampling quotas, double-review fraction, agreement statistic and acceptance thresholds are TBD and must be frozen before reviewers see model results. Protected test labels should be reviewed independently of candidate predictions.

13.2 Output-error review

For each candidate, use a frozen stratified sample that includes TP, FP, FN, low-confidence, high-confidence, tile-edge, dense, temporal-mismatch and OOD cases. Reviewers record taxonomy codes and severity, not only approve/reject. Report reviewer agreement, adjudication, severe-error prevalence and error distribution by stratum.

Review UI/version, imagery/reference layers, zoom/scale and reviewer identity or pseudonymous ID must be retained. A generated contact sheet with zero completed review decisions is unreviewed, not passed.

14. Statistical reporting

All rates include numerator, denominator and support. Confidence intervals use a predeclared method appropriate to the unit: object-level bootstrap must not pretend correlated tiles are independent. Prefer cluster bootstrap or paired analysis at AOI/group level for model comparisons.

Report:

• point estimate and interval;
• number of independent AOIs/groups;
• micro and macro aggregation;
• paired candidate-minus-incumbent delta by AOI;
• worst required stratum and its support;
• missing/excluded evidence and reason;
• multiplicity policy for any formal multi-stratum hypothesis claims.

The confidence level, bootstrap seed/replicates and non-inferiority margins are TBD until the gate specification is frozen. Descriptive diagnostics must not be presented as confirmatory release evidence.

15. Acceptance-gate semantics

Every gate returns exactly one state:

• pass: complete evidence meets the frozen rule;
• fail: complete evidence violates the frozen rule;
• not_evaluable: evidence is missing, invalid, leaked or below frozen support.

not_evaluable blocks promotion exactly like fail; it is never coerced to zero, ignored or averaged away.

A promotion decision is a logical AND across all mandatory gate families:

1. identity, provenance, licence and immutable manifest;
2. zero unresolved split leakage and duplicate-family violations;
3. automated data/label/geometry integrity;
4. completed representative human review and adjudication;
5. frozen object/AOI/region/context accuracy gates;
6. pure-background and hard-negative gates;
7. calibration and uncertainty/review-policy gates;
8. paired incumbent comparison and reproducibility rerun;
9. GPU latency, capacity and failure-rate gates;
10. model card, scope limitation, rollback and live shadow evidence.

No weighted composite score may compensate for a failed mandatory gate. The aggregate and every mandatory region/context gate must pass independently. Candidate selection and operating-point selection use calibration evidence only. The protected test is evaluated once with the frozen configuration. A rerun is allowed only to reproduce the same immutable computation or to resolve a proven execution failure without inspecting/using partial results.

Gate output must include machine-readable rule IDs, observed values, expected direction/range, support, evidence paths and hashes. Manual override cannot change a metric result; an exceptional operational decision must remain a separate signed record and cannot widen the accuracy claim.

16. No-claim rules

GeoIntel must not claim that a model is “fully trained”, “validated”, “production accurate”, “Belgium-wide” or equivalent when any of the following is true:

• only model-file presence, load success, a GPU smoke or output existence was proven;
• results come from train/val data, an unfrozen calibration set or leaked test evidence;
• the dataset, model, evaluator or inference configuration lacks hashes;
• required region/context support or representative human review is missing;
• a metric is undefined/null, a job is partial or a fallback/mock/heuristic was used;
• only aggregate performance is shown while a required stratum is absent or failing;
• precision/recall/AP is reported without match policy and TP/FP/FN/support;
• confidence is presented as probability of correctness without calibration evidence;
• a deterministic fixture benchmark is presented as production-model quality;
• the claim is extrapolated to another class, source, imagery period, resolution, region, segmentation task or deterministic GIS function;
• test results were used to choose the next training data or configuration.

Permitted wording must be evidence-bounded, for example: “candidate X achieved the recorded metrics on frozen portfolio Y under configuration Z.” It must name limitations and may not imply untested generalisation.

17. Required machine-readable outputs

Each future baseline or candidate evaluation must retain, without overwriting earlier evidence:

evaluation-contract.json
input-manifest.json
split-and-leakage-audit.json
object-metrics.json
tile-metrics.json
aoi-metrics.json
stratified-metrics.json
calibration-metrics.json
latency-and-reliability.json
human-review-summary.json
error-taxonomy.json
candidate-vs-incumbent.json
acceptance-gates.json
evidence-manifest.json

evidence-manifest.json must SHA-256 hash every retained input and output. Any amendment is additive, versioned and linked to its predecessor. Raw records must remain available so every aggregate can be recomputed.

18. Phase 2 metric implementation sequence

1. freeze evaluator schemas, taxonomy codes and undefined-value semantics;
2. implement strict group split and exact/near-duplicate audits;
3. validate CRS/alignment/label lineage and complete human corpus review;
4. freeze an incumbent portfolio and run a reproducible baseline twice;
5. derive and approve numeric thresholds from reviewed baseline distributions and operational requirements without protected-test access;
6. calibrate candidate operating points on calibration data only;
7. execute the one-time protected test and background-test gates;
8. publish scope-bounded evidence or retain not_configured/current limited scope when any gate is not evaluable.

Until those steps pass, this framework authorises measurement and remediation, not a widened accuracy claim.