docs(accuracy): refresh governed scan and training evidence
This commit is contained in:
Jens
@@ -12970,3 +12970,40 @@ Open:
|
||||
- Kept the historical evidence boundary explicit: this runtime receipt does not
|
||||
invent human review, a historical training commit/container, protected-test
|
||||
independence, national validity or a new promotion decision.
|
||||
|
||||
## 2026-08-30 - Accuracy-status en fail-closed modelactivatie
|
||||
|
||||
- Herhaalde de lokale Phase-3-scan deterministisch als
|
||||
`p3-eb67185e61107cc7`: 1013 bestanden onderzocht, nul overgeslagen en drie
|
||||
gedeclareerde externe productiegrenzen, met content-hash
|
||||
`f4ea193d2bdd96a5b391cf7d9d93685cc58e98742c40e4513bcbc434907e720a`.
|
||||
- Corrigeerde de statusgrens naar `local_complete_production_inventory_pending`;
|
||||
Tower-volumes en productie-PostGIS zijn nog niet als totale productie-
|
||||
inventaris gereconcilieerd.
|
||||
- Retireerde de stale tracked `geointel/`-mirror met immutable Git-recovery en
|
||||
verplaatste 207 niet-getrackte runtimebestanden intact naar een genegeerde
|
||||
recoverymap; productiegegevens en modelgewichten zijn niet verwijderd.
|
||||
- Legde onafhankelijke AI-visuele triage vast met `human_reviewer=false`.
|
||||
Referentiegeometrie, dense/nested labels, tile-edge-regels en regionale
|
||||
hard-negative dekking blokkeren corpusacceptatie, training en promotie.
|
||||
- Versterkte de activatiegrens: alleen een exact overeenkomende candidate key,
|
||||
model-SHA-256 en benchmarkmanifest-SHA-256 in een geslaagd governed Phase-4/5
|
||||
release-gaterapport kunnen activatie autoriseren. Een legacy diagnostisch
|
||||
promotierapport is onvoldoende.
|
||||
- De taakgeschikte officiële bron blijft doorslaggevend voor productresultaten.
|
||||
Deze remediatie levert geen menselijke review, nationale validatie,
|
||||
modelpromotie of 100%-accuratieclaim.
|
||||
|
||||
## 2026-08-30 - Fail-closed release-, backup- en rollbackpad
|
||||
|
||||
- Retireerde het aparte, niet-gegate Unraid-workflowpad; productie-deploy hangt
|
||||
nu uitsluitend af van quality-, dependency- en AI-containergates.
|
||||
- Bond controller, OCI-label, SBOM, Trivy-evidence en draaiende container aan
|
||||
het exacte full-SHA en lokale Docker image-ID.
|
||||
- Maakte storage/models byte-compleet, symlink-veilig en checksum-geverifieerd;
|
||||
latere snapshots hardlinken alleen gelijke backup-naar-backup bytes.
|
||||
- Bewees databaseherstel vóór productiecutover in een tijdelijke database,
|
||||
behield de pre-restore database en bond same-revision rollback aan het
|
||||
image-ID uit de specifieke pre-deploybackup.
|
||||
- Maakte cleanup-quarantine en herstel interruption-safe en no-clobber, en
|
||||
behield HTTPS-forwarding uitsluitend voor interne vertrouwde proxies.
|
||||
|
||||
@@ -1,5 +1,24 @@
|
||||
# GeoIntel TODO
|
||||
|
||||
## Accuracy-remediatie (2026-08-30)
|
||||
|
||||
- [x] Herhaal de lokale Phase-3-scan deterministisch: scan
|
||||
`p3-eb67185e61107cc7` onderzocht 1013 bestanden, sloeg er nul over en
|
||||
registreerde drie externe productiegrenzen expliciet; content-hash
|
||||
`f4ea193d2bdd96a5b391cf7d9d93685cc58e98742c40e4513bcbc434907e720a`.
|
||||
- [ ] Inventariseer de drie gedeclareerde Tower-/productiegrenzen gecontroleerd
|
||||
en reconcileer servervolumes en productie-PostGIS; de lokale scan is geen
|
||||
totale productie-inventaris.
|
||||
- [x] Leg de onafhankelijke AI-visuele hercontrole vast met
|
||||
`human_reviewer=false`; corpusacceptatie, training en promotie blijven
|
||||
geblokkeerd.
|
||||
- [x] Laat modelactivatie fail-closed vereisen dat candidate key, model-SHA-256
|
||||
en benchmarkmanifest-SHA-256 exact overeenkomen met een geslaagd governed
|
||||
Phase-4/5 release-gaterapport.
|
||||
- [ ] Herstel de vastgestelde labelsemantiek, dense/nested conflicten,
|
||||
tile-edge-regels en regionale hard-negative dekking; verkrijg daarna
|
||||
representatieve menselijke acceptatie vóór een volgende governed training.
|
||||
|
||||
## CI-onderhoud (2026-08-29)
|
||||
|
||||
- [x] Vervang de Gitea-incompatibele TruffleHog-actionwrapper.
|
||||
|
||||
@@ -224,6 +224,22 @@ produce map geometry or location claims.
|
||||
| DHMV | Contract requires product key, DTM/DSM semantics, EPSG:31370, native/analysis resolution, one-band numeric units, nodata, TAW vertical reference and 2013-2015 period for the currently configured DHMV II source. It is elevation evidence, not building truth. |
|
||||
| Bathymetry | Vertical datum/reference plane, depth versus bed-elevation semantics, survey epoch and uncertainty are mandatory. No water volume is inferred. |
|
||||
|
||||
|
||||
### 5.4 Inference tile-manifest contract
|
||||
|
||||
Raster tiling emits `geointel.raster.tile-manifest@2.0.0`. The manifest binds
|
||||
the source Dataset id/checksum/size, the latest DatasetVersion identity and
|
||||
checksum, registry/snapshot identities and snapshot checksum, data-contract
|
||||
identity, source version and—when present—the Area identity and geometry hash.
|
||||
Each tile has an explicit CRS/bounds, byte size and SHA-256.
|
||||
|
||||
Configured detection and segmentation validate that exact binding against the
|
||||
requested live Dataset before model loading. They recompute every tile hash,
|
||||
normalize the tile union to EPSG:4326 and require it to remain inside the
|
||||
manifest and persisted Dataset extents and intersect the bound Area. Missing,
|
||||
legacy or changed evidence is not inferred from filenames: it fails closed with
|
||||
a typed dataset, provenance, tile-integrity or spatial-scope error.
|
||||
|
||||
## 6. Vector data contract
|
||||
|
||||
### 6.1 Required vector fields
|
||||
|
||||
@@ -6,21 +6,30 @@ De reproduceerbare scan `scripts/run_accuracy_phase3_full_data_scan.py` is read-
|
||||
|
||||
| Telling | Aantal |
|
||||
| --- | ---: |
|
||||
| Onderzochte bestanden | 295 |
|
||||
| Onderzochte bestanden | 1013 |
|
||||
| Overgeslagen bestanden | 0 |
|
||||
| Onbereikbare scope-items | 3 |
|
||||
| Geïnventariseerd totaal | 298 |
|
||||
| Reconciliatie | groen (`295 + 0 + 3 = 298`) |
|
||||
| Geïnventariseerd totaal | 1016 |
|
||||
| Reconciliatie | groen (`1013 + 0 + 3 = 1016`) |
|
||||
|
||||
De drie expliciet onbereikbare items zijn de niet-gemounte Tower-corpora, gemounte modelvolumes en een productie-PostGIS/API-endpoint. Elk staat als `unreachable` in het volledige manifest met concrete reden; er is geen stilzwijgende skip.
|
||||
De drie expliciet onbereikbare items zijn grenzen van deze lokale
|
||||
Windows-scan: de niet lokaal gemounte Tower-corpora, modelvolumes en het
|
||||
productie-PostGIS/API-endpoint. Elk staat als `unreachable` in het volledige
|
||||
manifest met concrete reden; er is geen stilzwijgende skip. Dit betekent niet
|
||||
dat de server zelf onbereikbaar is. Een afzonderlijke gecontroleerde
|
||||
productiescan moet die remote scopes beoordelen voordat een nationale
|
||||
volledigheidsclaim mogelijk is.
|
||||
|
||||
De scan leverde 172 anomalieën op. Dat is geen geldigheidsclaim: 163 items zijn daarom in de logische quarantine-manifest opgenomen. De scanner promoot geen enkel foutief item tot ground truth en voert geen automatische reparatie uit. De huidige bevindingen zijn:
|
||||
De scan leverde 711 anomalieën op. Dat is geen geldigheidsclaim: 682 items
|
||||
zijn daarom in het logische quarantainemanifest opgenomen. De scanner promoot
|
||||
geen enkel foutief item tot ground truth en voert geen automatische reparatie
|
||||
uit. De huidige bevindingen zijn:
|
||||
|
||||
| Categorie | Aantal | Standaardactie |
|
||||
| --- | ---: | --- |
|
||||
| Onleesbare/corrupte GeoTIFF | 96 | quarantine |
|
||||
| Afgeleide artefacten zonder expliciete lineage | 58 | quarantine |
|
||||
| Bronlabel zonder declared freshness | 9 | requires review |
|
||||
| Onleesbare/corrupte GeoTIFF | 96 | unavailable or unreadable |
|
||||
| Afgeleide artefacten zonder expliciete lineage | 577 | quarantine |
|
||||
| Bronlabel zonder declared freshness | 29 | requires review |
|
||||
| Manifest met nul-bounding-box | 6 | quarantine |
|
||||
| Niet-bereikbare externe scope | 3 | unavailable or unreadable |
|
||||
|
||||
@@ -43,7 +52,7 @@ Er zijn vijf vaste ernstniveaus (`blocker`, `critical`, `major`, `minor`, `infor
|
||||
|
||||
De scanner bouwt eerst een gesorteerde inventory-hash. Per batch wordt `scan-checkpoint.json` atomair bijgewerkt. Met dezelfde inventory en `--resume` worden onveranderde records hergebruikt; gewijzigde records worden opnieuw gelezen. De inhoudshash sluit alleen uitvoertimestamps uit. Twee opeenvolgende runs gaven dezelfde scan-ID en content-hash:
|
||||
|
||||
`p3-46ee3f8d3a2dc52b` / `1a219362c6cb2ac00489625f1a9b36e2fd4809ab58ee55ab1f67c3cc34773f3e`
|
||||
`p3-eb67185e61107cc7` / `f4ea193d2bdd96a5b391cf7d9d93685cc58e98742c40e4513bcbc434907e720a`
|
||||
|
||||
## Bewijsartefacten
|
||||
|
||||
@@ -69,9 +78,26 @@ De fixturetest controleert geldige en foutieve GeoJSON, corrupte rasterinput, ex
|
||||
|
||||
## Fasebeslissing
|
||||
|
||||
Fase 3 is `done` voor de afgebakende projectomgeving: alle 295 veilige lokale bestanden zijn verwerkt en de drie niet-bereikbare externe grenzen zijn expliciet geregistreerd. Fase 4 is `ready` voor verdere beoordeling van de gevonden quarantine- en leakage-signalen. Dit is geen modelpromotie- of nationale kwaliteitsclaim; de inhoudelijke anomalieën moeten eerst door de volgende fase worden hersteld of menselijk beoordeeld.
|
||||
De lokale Fase-3-scan is volledig voor de afgebakende projectomgeving: alle
|
||||
1013 veilige lokale bestanden zijn verwerkt en de drie niet-gemounte grenzen
|
||||
zijn expliciet geregistreerd. Fase 3 blijft voor de totale operationele
|
||||
omgeving `production_inventory_pending` totdat de servervolumes en database
|
||||
in een gecontroleerde productiescan zijn gereconcilieerd. Fase 4 kan de lokale
|
||||
quarantine- en leakage-signalen al beoordelen. Dit is geen modelpromotie- of
|
||||
nationale kwaliteitsclaim; de inhoudelijke anomalieën moeten eerst worden
|
||||
hersteld of representatief menselijk beoordeeld.
|
||||
|
||||
> **Historische handoff:** `ready` beschrijft het overdrachtsmoment direct na
|
||||
> de Phase-3-scan. Het Phase-4-harnas is inmiddels geïmplementeerd, maar de
|
||||
> actuele productbenchmark is nog `not_evaluable`; de normatieve actuele
|
||||
> status staat in `docs/accuracy-program/status.json`.
|
||||
|
||||
## Accuracy-handoff na de scan
|
||||
|
||||
De onafhankelijke AI-visuele triage van 30 augustus is vastgelegd in
|
||||
`artifacts/evidence/accuracy/model-training/20260830-independent-ai-visual-review.json`.
|
||||
Daarin blijven `human_reviewer=false`, corpusacceptatie, training en promotie
|
||||
expliciet geblokkeerd. Een model kan bovendien pas worden geactiveerd wanneer
|
||||
de exacte candidate key en model-SHA-256 overeenkomen met een geslaagd governed
|
||||
Phase-4/5 release-gaterapport dat ook de benchmarkmanifest-SHA-256 bindt. De
|
||||
taakgeschikte officiële bron blijft doorslaggevend voor productresultaten.
|
||||
|
||||
@@ -2,11 +2,12 @@
|
||||
"schema_version": 1,
|
||||
"program": "GeoIntel Accuracy Improvement Program",
|
||||
"phase": "P4",
|
||||
"generated_at": "2026-08-02T04:40:51+02:00",
|
||||
"generated_at": "2026-08-30T05:58:29+02:00",
|
||||
"scope": {
|
||||
"product": "Belgium and the Belgian North Sea",
|
||||
"active_building_model_claim": "Mol/Kempen only, operator review required",
|
||||
"national_building_validation": false
|
||||
"national_building_validation": false,
|
||||
"result_authority_policy": "Task-appropriate official sources remain decisive for product results; AI output is a reviewable proposal unless a governed task-specific release gate proves otherwise."
|
||||
},
|
||||
"baseline": {
|
||||
"branch": "codex/geointel-accuracy-program",
|
||||
@@ -34,7 +35,8 @@
|
||||
"promotion_allowed": false,
|
||||
"scope_widening_allowed": false,
|
||||
"training_allowed_now": false,
|
||||
"training_unlock_gate": "P2-08 after P2-00 through P2-07 have passed",
|
||||
"training_unlock_gate": "A corrected checksum-frozen corpus, independent splits, representative human acceptance and every applicable pre-registered training gate must pass before a new governed run.",
|
||||
"activation_gate": "Activation requires the exact candidate key and model SHA-256 bound to a passing governed Phase-4/5 release-gate report and its benchmark-manifest SHA-256; a legacy diagnostic promotion report alone has no activation authority.",
|
||||
"critical_risk_count": 8,
|
||||
"high_risk_count": 17,
|
||||
"medium_risk_count": 4
|
||||
@@ -80,23 +82,24 @@
|
||||
]
|
||||
},
|
||||
"phase3": {
|
||||
"status": "done",
|
||||
"meaning": "All 295 safe local files in the configured GeoIntel roots were scanned read-only; three known external boundaries were explicitly recorded as unreachable.",
|
||||
"status": "local_complete_production_inventory_pending",
|
||||
"meaning": "All 1013 safe local files in the configured GeoIntel roots were scanned read-only; three declared external production boundaries were recorded explicitly and still require a controlled server-side inventory.",
|
||||
"scanner": "scripts/run_accuracy_phase3_full_data_scan.py",
|
||||
"scanner_version": "3.0.3",
|
||||
"scan_id": "p3-46ee3f8d3a2dc52b",
|
||||
"scanner_version": "3.1.0",
|
||||
"scan_id": "p3-eb67185e61107cc7",
|
||||
"evidence_root": "artifacts/evidence/accuracy/P3",
|
||||
"content_hash": "1a219362c6cb2ac00489625f1a9b36e2fd4809ab58ee55ab1f67c3cc34773f3e",
|
||||
"content_hash": "f4ea193d2bdd96a5b391cf7d9d93685cc58e98742c40e4513bcbc434907e720a",
|
||||
"reconciliation": {
|
||||
"examined": 295,
|
||||
"examined": 1013,
|
||||
"skipped": 0,
|
||||
"unreachable": 3,
|
||||
"inventory_total": 298,
|
||||
"inventory_total": 1016,
|
||||
"reconciles": true
|
||||
},
|
||||
"anomaly_count": 172,
|
||||
"quarantine_item_count": 163,
|
||||
"anomaly_count": 711,
|
||||
"quarantine_item_count": 682,
|
||||
"does_not_mean": [
|
||||
"the total production inventory is complete",
|
||||
"all anomalies are repaired",
|
||||
"AOI split independence is proven",
|
||||
"GRB ground truth is locally available",
|
||||
@@ -256,21 +259,31 @@
|
||||
"promotion_evidence": false
|
||||
},
|
||||
"protected_test_isolation": false,
|
||||
"human_label_acceptance": false
|
||||
"human_label_acceptance": false,
|
||||
"latest_independent_ai_visual_review": {
|
||||
"review_id": "20260830-independent-ai-visual-review",
|
||||
"evidence": "artifacts/evidence/accuracy/model-training/20260830-independent-ai-visual-review.json",
|
||||
"human_reviewer": false,
|
||||
"corpus_accepted": false,
|
||||
"training_gate": "blocked",
|
||||
"promotion_allowed": false,
|
||||
"claim_boundary": "Independent AI visual inspection is triage evidence only; it is not representative human acceptance, protected-test evidence or an accuracy measurement."
|
||||
}
|
||||
},
|
||||
"verification": {
|
||||
"backend_full_suite": {
|
||||
"status": "failed",
|
||||
"passed": 1282,
|
||||
"failed": 17,
|
||||
"duration_seconds": 116.62,
|
||||
"classification": "The canonical backend test import boundary now collects. Sixteen remaining failures are historical source-text assertions for changed UI/deployment/README behavior; one full Windows run also hit an intermittent WSL-backed bash.exe host failure in a shell-wrapper syntax test. This still prevents a whole-suite-green claim."
|
||||
"status": "passed",
|
||||
"passed": 1702,
|
||||
"skipped": 1,
|
||||
"failed": 0,
|
||||
"duration_seconds": 279.63,
|
||||
"classification": "The complete canonical backend suite passes with deprecations promoted to errors. The single skip is the Windows-host symlink fixture; the symlink refusal path remains covered by static and Linux-targeted release checks."
|
||||
},
|
||||
"backend_ci_entrypoint": {
|
||||
"status": "collected_with_failures",
|
||||
"collected": 1299,
|
||||
"result": "1282 passed, 17 failed",
|
||||
"note": "The former scripts.render_operator_polygon_label_qa collection failure is resolved by the canonical backend-test import boundary."
|
||||
"status": "passed",
|
||||
"collected": 1703,
|
||||
"result": "1702 passed, 1 skipped",
|
||||
"note": "The canonical backend test import boundary, deployment contracts, guest analysis flow and governed raster handoff all collect and pass."
|
||||
},
|
||||
"phase1_tooling_tests": {
|
||||
"status": "passed",
|
||||
@@ -280,14 +293,14 @@
|
||||
"status": "passed"
|
||||
},
|
||||
"repository_ruff": {
|
||||
"status": "failed",
|
||||
"finding_count": 95,
|
||||
"note": "P2-changed Python paths pass their scoped Ruff check; repository-wide remediation remains an explicit P2-01 gate."
|
||||
"status": "passed",
|
||||
"finding_count": 0,
|
||||
"note": "Repository-wide Ruff validation passes across backend, scripts and tests without policy weakening."
|
||||
},
|
||||
"frontend_unit": {
|
||||
"status": "passed",
|
||||
"test_files": 16,
|
||||
"tests": 51,
|
||||
"test_files": 40,
|
||||
"tests": 170,
|
||||
"command": "npm run test:unit"
|
||||
},
|
||||
"frontend_typecheck": {
|
||||
@@ -302,13 +315,13 @@
|
||||
},
|
||||
"openapi_contract": {
|
||||
"status": "passed",
|
||||
"implemented_routes": 147,
|
||||
"explicit_non_envelope_endpoints": 10
|
||||
"implemented_routes": 155,
|
||||
"explicit_non_envelope_endpoints": 12
|
||||
},
|
||||
"alembic": {
|
||||
"status": "passed_offline_and_disposable_postgis",
|
||||
"heads": [
|
||||
"202608010001"
|
||||
"202608230001"
|
||||
],
|
||||
"offline_upgrade_rendered": true,
|
||||
"offline_downgrade_rendered": true,
|
||||
@@ -390,10 +403,12 @@
|
||||
"docs/accuracy-program/09-full-data-scan.md",
|
||||
"docs/accuracy-program/10-evaluation-protocol.md",
|
||||
"docs/accuracy-program/11-baseline-benchmark.md",
|
||||
"docs/accuracy-program/12-release-gates.md"
|
||||
"docs/accuracy-program/12-release-gates.md",
|
||||
"docs/accuracy-program/13-nested-mirror-retirement.md"
|
||||
],
|
||||
"evidence_root": "artifacts/evidence/accuracy/P1",
|
||||
"evidence_manifest": "artifacts/evidence/accuracy/P1/evidence-manifest.json",
|
||||
"latest_verification_evidence_manifest": "artifacts/evidence/accuracy/P1/evidence-manifest-20260830-verification.json",
|
||||
"phase2_evidence_root": "artifacts/evidence/accuracy/P2",
|
||||
"phase2_evidence_manifest": "artifacts/evidence/accuracy/P2/evidence-manifest.json",
|
||||
"phase3_evidence_root": "artifacts/evidence/accuracy/P3",
|
||||
|
||||
Reference in New Issue
Block a user