docs(accuracy): publish phase 4 benchmark evidence

2026-08-02 05:10:05 +02:00
parent 70fb4b94e5
commit f41392a415
34 changed files with 25672 additions and 8 deletions
@@ -12391,3 +12391,56 @@ Open:
  **ready** for remediation/review. This does not unlock training, promotion,
  national validation or production release; those gates remain governed by
  the execution contract and the incomplete Phase 2 gates.
+
+## 2026-08-02 - Accuracy Improvement Program Phase 4 evaluation harness
+
+### Implemented and verified scope
+
+- Added a deterministic evaluator for object detection, building-footprint
+  segmentation, categorical raster interpretation, vector comparison, change
+  detection, terrain/height interpretation and geospatial data validation. It
+  retains exact references, pre-/post-filter predictions, config and lineage,
+  and reports task metrics, Wilson intervals, calibration, abstention,
+  subgroup support and a stable failure taxonomy.
+- Added a six-role split generator for `train`, `val`, `calibration`, `test`,
+  `background-test` and sealed `challenge`, with exact/near-duplicate,
+  identity, acquisition, object, geometry and 2-km spatial leakage gates.
+  Product mode requires strict P3/provenance records and rehashes four physical
+  asset types; protected roles are rejected by the training firewall.
+- Added a fail-closed governed product-baseline validator. It recomputes all
+  seven task families in-process and verifies current active-model bytes,
+  CUDA/GPU identity, authority, human review, leakage, vault and all thirteen
+  subgroup dimensions. Loose status fields cannot turn a productgate green.
+- Fixed an evidence-ledger self-reference in workflow v2.0.1: only the
+  gate-relevant `/runtime/active_model` projection from `status.json` is
+  canonically hashed. Updating timestamps, Phase-4/5 bookkeeping, documents or
+  evidence IDs therefore leaves the run fingerprint stable; changing the
+  active model changes it.
+
+### Evidence and verification
+
+- Normative run `p4-2.0.1-9677d0ef37db82bcf39b` is bound to source commit
+  `70fb4b94e5cb7c248beec5a936ce186f38cc183c`. Its canonical benchmark hash is
+  `0ea5ab07f46c509a7a24943b31d5e9bfd6368920609227bc47613e18e52c4642` and
+  its evidence-manifest file hash is
+  `fdc15a95ee2a0754dfa169f4b41036e084b8d8909afc37fd8ea68ee6b9210f98`.
+- The targeted Phase-4 suite passed 60 tests. The broader contracts,
+  provenance, Phase-3/4, golden-QA, runtime-model and migration suite passed
+  103 tests with the installed Git Bash provider; the Windows Store WSL stub
+  was unavailable and is not a code failure. The focused migration/provenance
+  subset passed 29 tests and Alembic has one head, `202608010001`.
+- Ruff, Ruff format check, compileall and diff check passed on all Phase-4
+  paths. No Python mypy/pyright typecheck is configured in this repository.
+- Repeated workflow execution produced the same run ID and byte-identical
+  immutable bundle. `--allow-product-blocked` returned 0; standard fail-closed
+  execution returned 2.
+
+### Decision
+
+- The local harness is `pass`, but the governed product benchmark is `fail`
+  because Phase-3 leakage remains `attention`. Active model execution,
+  authority coverage, representative review, independent product splits,
+  physical vault isolation and thirteen-dimension subgroup support remain
+  `not_evaluable` without governed evidence. Phase 4 remains **in progress**,
+  Phase 5 remains **not ready**, and promotion/training feedback from protected
+  data is not authorized.