audit: establish accuracy phase 1 baseline
This commit is contained in:
Jens
@@ -0,0 +1,189 @@
|
||||
# Phase 1 execution contract
|
||||
|
||||
- Status: **executed**
|
||||
- Audit date: 2026-08-01
|
||||
- Audit branch: `codex/geointel-accuracy-program`
|
||||
- Audited baseline commit: `0c019bb22f816db1e4b7a68379bcad08924d9a21`
|
||||
- Product scope: Belgium and the Belgian North Sea
|
||||
- Release conclusion: **blocked**
|
||||
- Phase 2 conclusion: **safe to start as controlled remediation, not as training or promotion**
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This contract turns the accuracy program into an evidence-bearing engineering
|
||||
program. Phase 1 establishes what exists, what actually runs, where data and
|
||||
model results originate, which claims are supportable, and which defects must
|
||||
be corrected before any new training or release promotion.
|
||||
|
||||
Phase 1 is complete when the required inventory, lineage, baseline, risk,
|
||||
metric, roadmap, machine-readable status and retained evidence exist and can be
|
||||
reproduced. Completion of Phase 1 does **not** mean that GeoIntel is accurate,
|
||||
nationally validated, release-ready, or fully trained.
|
||||
|
||||
## 2. Non-negotiable rules
|
||||
|
||||
1. No quality, coverage, authority, calibration or national-readiness claim may
|
||||
be stronger than the retained evidence.
|
||||
2. A green unit test that uses a fixture, mock, placeholder or source-text
|
||||
assertion is not evidence of live geospatial or model quality.
|
||||
3. Protected test data is never used for training, calibration, failure-driven
|
||||
sampling, threshold selection or stopping decisions.
|
||||
4. Dataset, label, tile, checkpoint, runtime and evaluation identities are
|
||||
immutable hashes, not filenames or mutable display names.
|
||||
5. Geometry entering persistence is validated, assigned an explicit source CRS,
|
||||
transformed to the canonical CRS and checked for valid domain and units.
|
||||
6. Official or authoritative status is server-attested; caller-controlled source
|
||||
names can never establish authority.
|
||||
7. Failed, unavailable and `not_configured` states remain explicit. A fallback
|
||||
cannot silently be reported as a successful primary result.
|
||||
8. Human review is only complete when reviewer identity, decision, timestamp,
|
||||
sampled object/AOI identity and reviewed artifact hash are persisted.
|
||||
9. Phase 1 does not delete, rewrite or promote datasets, checkpoints, caches,
|
||||
outputs, migrations or server state. The GPU smoke is read-only and persists
|
||||
no detections.
|
||||
10. User-owned untracked files, including `.codex-artifacts/`, server
|
||||
`.env.bak.*`, `.forgeflow/`, `storage/evaluation/` and
|
||||
`storage/source-cache/`, remain untouched.
|
||||
|
||||
## 3. Evidence vocabulary
|
||||
|
||||
Every statement in this program uses one of these classes:
|
||||
|
||||
| Class | Meaning |
|
||||
|---|---|
|
||||
| observed | Directly read from the repository, database, storage, runtime or retained artifact |
|
||||
| reproduced | Deterministic read-only reproducer demonstrated the behavior |
|
||||
| tested | An executable test or build gate ran and its result is retained or recorded |
|
||||
| inferred | A conclusion drawn from observed evidence; it must be labelled as an inference |
|
||||
| planned | Not implemented or not yet evidenced |
|
||||
| unavailable | The required system or artifact could not be inspected; reason is stated |
|
||||
| blocked | A required gate cannot pass without remediation |
|
||||
|
||||
The words “implemented”, “configured”, “available”, “validated” and “promoted”
|
||||
are not synonyms:
|
||||
|
||||
- **implemented** means code exists;
|
||||
- **configured** means a runtime selects it;
|
||||
- **available** means dependencies/assets can be loaded;
|
||||
- **validated** requires a registered evaluation and acceptance decision;
|
||||
- **promoted** requires every release gate and an auditable activation record.
|
||||
|
||||
## 4. Authorized Phase 1 mutations
|
||||
|
||||
The only authorized repository mutations are:
|
||||
|
||||
- audit collectors and deterministic reproducer scripts;
|
||||
- focused tests for those collectors;
|
||||
- `docs/accuracy-program/`;
|
||||
- retained evidence under `artifacts/evidence/accuracy/P1/`;
|
||||
- required execution-log and TODO updates;
|
||||
- a selective `.gitignore` exception that tracks only the Phase 1 evidence
|
||||
tree while keeping other generated `artifacts/` ignored.
|
||||
|
||||
No product behavior, model checkpoint, dataset, label or production database row
|
||||
is changed in Phase 1.
|
||||
|
||||
## 5. Executed investigation surface
|
||||
|
||||
The audit covered:
|
||||
|
||||
- root and nested tracked source trees, Git state and ignored/untracked state;
|
||||
- backend API routes, schemas, services, ORM entities and Alembic chain;
|
||||
- frontend TypeScript, tests, typecheck and production build;
|
||||
- CI/readiness scripts, container definitions and Tower deployment identity;
|
||||
- Postgres/PostGIS table counts, statuses, lineage gaps and geometry integrity;
|
||||
- mounted storage paths referenced by the database;
|
||||
- active NVIDIA/PyTorch/Ultralytics runtime and active model hash;
|
||||
- one existing Geel raster tile through the production YOLO adapter on
|
||||
`cuda:0`;
|
||||
- available Belgian building corpus, split, label, tiling, calibration,
|
||||
checkpoint and human-review evidence;
|
||||
- mocks, fixtures, fallbacks, placeholders and documentation drift.
|
||||
|
||||
Detailed outcomes live in documents 01 through 06 and in the P1 evidence tree.
|
||||
|
||||
## 6. Reproducibility entry points
|
||||
|
||||
Run from the repository root:
|
||||
|
||||
```powershell
|
||||
python scripts/run_accuracy_phase1_baseline.py --output-dir artifacts/evidence/accuracy/P1
|
||||
python scripts/reproduce_accuracy_phase1_findings.py
|
||||
python -m pytest tests/test_accuracy_phase1_baseline.py -q -p no:cacheprovider
|
||||
```
|
||||
|
||||
The runtime/database collector is intentionally executed inside the deployed
|
||||
application container and uses read-only SQL with a statement timeout:
|
||||
|
||||
```text
|
||||
scripts/collect_accuracy_phase1_runtime.py
|
||||
```
|
||||
|
||||
The GPU collector is streamed into the deployed container and uses the
|
||||
production adapter with an existing checksummed model, manifest and tile:
|
||||
|
||||
```text
|
||||
scripts/collect_accuracy_phase1_inference_smoke.py
|
||||
```
|
||||
|
||||
Its retained JSON includes the model, manifest and tile SHA-256 values, package
|
||||
versions, CUDA device, seed, configuration, raster CRS and raw output summary.
|
||||
It explicitly forbids deriving an accuracy or release claim from one smoke.
|
||||
|
||||
## 7. Proven release blockers
|
||||
|
||||
Phase 1 reproduced or observed all of the following:
|
||||
|
||||
- cross-theme coverage contamination can incorrectly return
|
||||
`operational`;
|
||||
- a nominal metre buffer is applied as degrees;
|
||||
- Lambert coordinates can be persisted unchanged under SRID 4326;
|
||||
- caller-controlled source identity can be elevated to authoritative coverage;
|
||||
- YOLO validation scope is bypassable through a mutable Area-name substring;
|
||||
- legal coverage identity changes when an Area display name changes;
|
||||
- Area PATCH silently ignores a documented geometry field;
|
||||
- four persisted Geel detections contain Lambert-domain coordinates while their
|
||||
geometry column is labelled SRID 4326;
|
||||
- the training loop contains a future protected-test feedback path;
|
||||
- V56 has zero accepted human AOI reviews out of 180;
|
||||
- split independence, pure-background coverage, temporal label alignment and
|
||||
unique-object evaluation are insufficient;
|
||||
- no candidate has protected-test, background-test and promotion evidence;
|
||||
- the real CI pytest work directory fails during collection, the full root suite
|
||||
has 17 stale assertion failures, Ruff has 112 findings and frontend lint is
|
||||
not configured;
|
||||
- the production AI image is not built or exercised by CI;
|
||||
- a tracked nested source mirror contains 1,153 files, 68 of which differ from
|
||||
root, creating source-of-truth ambiguity.
|
||||
|
||||
## 8. Decision gates
|
||||
|
||||
| Gate | Phase 1 decision |
|
||||
|---|---|
|
||||
| Runtime CUDA and active model load | pass |
|
||||
| One production-adapter GPU inference | pass, runtime-only evidence |
|
||||
| Database/storage reachability | pass for inspected references |
|
||||
| Geometry/CRS integrity | fail |
|
||||
| Authoritative-source integrity | fail |
|
||||
| Coverage correctness | fail |
|
||||
| Strict protected-test isolation | fail in code design |
|
||||
| Human label acceptance | fail |
|
||||
| Split independence and negatives | fail |
|
||||
| Reproducible national metric baseline | absent |
|
||||
| Backend release gate | fail |
|
||||
| Lint and frontend lint | fail / absent |
|
||||
| National model promotion | blocked |
|
||||
| Start test-first Phase 2 remediation | allowed |
|
||||
|
||||
## 9. Stop and escalation conditions
|
||||
|
||||
Training and model promotion remain prohibited until Phase 2 has closed the
|
||||
CRS, authority, coverage and protected-test isolation blockers and a frozen,
|
||||
human-approved corpus exists. If a required source snapshot, reviewer decision,
|
||||
immutable hash or independent holdout cannot be produced, the corresponding
|
||||
claim remains blocked; it is never replaced by synthetic success.
|
||||
|
||||
The protected test may be opened exactly once for a pre-registered candidate
|
||||
after all calibration gates pass. A failure after that opening creates a new
|
||||
model-development cycle and requires a newly governed protected set; its errors
|
||||
must not feed back into the same training lineage.
|
||||
Reference in New Issue
Block a user