geointel/docs/QA_QC_SPECIFICATION.md

GeoIntel Kempen — QA/QC Specification v1.0

QA/QC is a core showcase feature. It must compare model-derived geospatial outputs against reference data such as GRB or user-supplied labels.

Primary use case

Compare AI building detections or segmentations against GRB building footprints.

Inputs

Prediction layer

• Polygons converted from object detections or segmentation masks.
• Required fields: id, class_name, confidence, geometry, analysis_run_id.

Reference layer

• GRB building polygons, OSM buildings, or user annotation polygons.
• Required fields: id, class_name or canonical class, geometry, source.

Geometry preparation

1. Reproject both layers to the metric CRS.
2. Repair invalid geometries.
3. Clip both layers to the analysis area.
4. Optionally simplify only for visualization, not for metric calculation.
5. Filter classes if class-specific QA is requested.

Matching strategy

Default matching is polygon IoU-based matching.

IoU

IoU = area(intersection(prediction, reference)) / area(union(prediction, reference))

Match threshold

Default: IoU >= 0.50.

Additional thresholds for reporting:

• lenient: 0.30
• standard: 0.50
• strict: 0.75

Matching algorithm

1. Build spatial index.
2. For every prediction, find intersecting reference candidates.
3. Calculate IoU for candidates.
4. Select the highest IoU candidate.
5. Enforce one-to-one matching: one reference can match only one prediction.
6. Resolve conflicts by highest IoU, then highest confidence.

Classification of outcomes

Outcome	Definition
True Positive	prediction matched to a reference above threshold
False Positive	prediction not matched to any reference
False Negative	reference not matched by any prediction
Low-IoU Match	prediction overlaps reference but below selected threshold
Class Mismatch	geometry match but class differs

Metrics

precision = TP / (TP + FP)
recall = TP / (TP + FN)
f1 = 2 * precision * recall / (precision + recall)
mean_iou = average IoU of matched pairs

Also calculate:

• false positive count
• false negative count
• low confidence false positives
• confidence distribution by outcome
• area-weighted recall

Outputs

QA summary JSON

{
  "threshold": 0.5,
  "true_positive_count": 120,
  "false_positive_count": 8,
  "false_negative_count": 13,
  "precision": 0.9375,
  "recall": 0.9022,
  "f1": 0.9195,
  "mean_iou": 0.72,
  "class_name": "building"
}

QA layers

• qa_true_positives
• qa_false_positives
• qa_false_negatives
• qa_low_iou_matches
• qa_match_lines connecting prediction centroids to reference centroids

QA dashboard cards

• Precision
• Recall
• F1
• Mean IoU
• False positives
• False negatives
• Top issue areas

UI requirements

The QA/QC page must show:

1. Reference layer selector.
2. Prediction layer selector.
3. Class filter.
4. IoU threshold slider.
5. Run QA button.
6. Metrics cards.
7. Map with color-coded TP/FP/FN layers.
8. Table of individual findings.
9. Export buttons for GeoJSON and CSV.

V1 implementation target

V1 must support polygon-vs-polygon QA for building detections against GRB or OSM building footprints. Raster mask QA and pixel-level IoU can be added later.

Sprint 12 golden QA/QC benchmark

Sprint 12 adds a deterministic benchmark package for regression detection:

• fixtures/golden/reference_buildings.geojson
• fixtures/golden/predicted_buildings.geojson
• fixtures/golden/expected_qa_metrics.json
• scripts/run_golden_qa_benchmark.py

The benchmark uses two reference building polygons and two candidate building polygons:

• one candidate polygon matches one reference polygon at IoU 0.8339768339761133;
• one candidate polygon is a false positive;
• one reference polygon is a false negative.

Expected baseline at IoU threshold 0.5:

{
  "matches": 1,
  "false_positive_count": 1,
  "false_negative_count": 1,
  "precision": 0.5,
  "recall": 0.5,
  "f1": 0.5,
  "mean_iou": 0.8339768339761133
}

Run from the repository root:

python scripts/run_golden_qa_benchmark.py

The script uses the existing QaService and QualityService paths. It fails if metrics drift outside the documented tolerance and verifies that one QualityCheck plus metric rows for precision, recall, F1, mean IoU, false positives and false negatives are produced.

This benchmark is explicit fixture/demo data only. It does not require live providers, real AI models, Docker or PostGIS.