geointel/docs/QA_QC_SPECIFICATION.md

# GeoIntel Kempen — QA/QC Specification v1.0

QA/QC is a core showcase feature. It must compare model-derived geospatial outputs against reference data such as GRB or user-supplied labels.

## Primary use case

Compare AI building detections or segmentations against GRB building footprints.

## Inputs

### Prediction layer

- Polygons converted from object detections or segmentation masks.
- Required fields: `id`, `class_name`, `confidence`, `geometry`, `analysis_run_id`.

### Reference layer

- GRB building polygons, OSM buildings, or user annotation polygons.
- Required fields: `id`, `class_name` or canonical class, `geometry`, `source`.

## Geometry preparation

1. Reproject both layers to the metric CRS.
2. Repair invalid geometries.
3. Clip both layers to the analysis area.
4. Optionally simplify only for visualization, not for metric calculation.
5. Filter classes if class-specific QA is requested.

## Matching strategy

Default matching is polygon IoU-based matching.

### IoU

```text
IoU = area(intersection(prediction, reference)) / area(union(prediction, reference))
```

### Match threshold

Default: `IoU >= 0.50`.

Additional thresholds for reporting:

- lenient: `0.30`
- standard: `0.50`
- strict: `0.75`

### Matching algorithm

1. Build spatial index.
2. For every prediction, find intersecting reference candidates.
3. Calculate IoU for candidates.
4. Select the highest IoU candidate.
5. Enforce one-to-one matching: one reference can match only one prediction.
6. Resolve conflicts by highest IoU, then highest confidence.

## Classification of outcomes

| Outcome | Definition |
|---|---|
| True Positive | prediction matched to a reference above threshold |
| False Positive | prediction not matched to any reference |
| False Negative | reference not matched by any prediction |
| Low-IoU Match | prediction overlaps reference but below selected threshold |
| Class Mismatch | geometry match but class differs |

## Metrics

```text
precision = TP / (TP + FP)
recall = TP / (TP + FN)
f1 = 2 * precision * recall / (precision + recall)
mean_iou = average IoU of matched pairs
```

Also calculate:

- false positive count
- false negative count
- low confidence false positives
- confidence distribution by outcome
- area-weighted recall

## Outputs

### QA summary JSON

```json
{
  "threshold": 0.5,
  "true_positive_count": 120,
  "false_positive_count": 8,
  "false_negative_count": 13,
  "precision": 0.9375,
  "recall": 0.9022,
  "f1": 0.9195,
  "mean_iou": 0.72,
  "class_name": "building"
}
```

### QA layers

- `qa_true_positives`
- `qa_false_positives`
- `qa_false_negatives`
- `qa_low_iou_matches`
- `qa_match_lines` connecting prediction centroids to reference centroids

### QA dashboard cards

- Precision
- Recall
- F1
- Mean IoU
- False positives
- False negatives
- Top issue areas

## UI requirements

The QA/QC page must show:

1. Reference layer selector.
2. Prediction layer selector.
3. Class filter.
4. IoU threshold slider.
5. Run QA button.
6. Metrics cards.
7. Map with color-coded TP/FP/FN layers.
8. Table of individual findings.
9. Export buttons for GeoJSON and CSV.

## V1 implementation target

V1 must support polygon-vs-polygon QA for building detections against GRB or OSM building footprints. Raster mask QA and pixel-level IoU can be added later.

## Sprint 12 golden QA/QC benchmark

Sprint 12 adds a deterministic benchmark package for regression detection:

- `fixtures/golden/reference_buildings.geojson`
- `fixtures/golden/predicted_buildings.geojson`
- `fixtures/golden/expected_qa_metrics.json`
- `scripts/run_golden_qa_benchmark.py`

The benchmark uses two reference building polygons and two candidate building polygons:

- one candidate polygon matches one reference polygon at IoU `0.8339768339761133`;
- one candidate polygon is a false positive;
- one reference polygon is a false negative.

Expected baseline at IoU threshold `0.5`:

```json
{
  "matches": 1,
  "false_positive_count": 1,
  "false_negative_count": 1,
  "precision": 0.5,
  "recall": 0.5,
  "f1": 0.5,
  "mean_iou": 0.8339768339761133
}
```

Run from the repository root:

```bash
python scripts/run_golden_qa_benchmark.py
```

The script uses the existing `QaService` and `QualityService` paths. It fails if metrics drift outside the documented tolerance and verifies that one `QualityCheck` plus metric rows for precision, recall, F1, mean IoU, false positives and false negatives are produced.

This benchmark is explicit fixture/demo data only. It does not require live providers, real AI models, Docker or PostGIS.