190 lines
8.5 KiB
Markdown
190 lines
8.5 KiB
Markdown
# Phase 1 execution contract
|
|
|
|
- Status: **executed**
|
|
- Audit date: 2026-08-01
|
|
- Audit branch: `codex/geointel-accuracy-program`
|
|
- Audited baseline commit: `0c019bb22f816db1e4b7a68379bcad08924d9a21`
|
|
- Product scope: Belgium and the Belgian North Sea
|
|
- Release conclusion: **blocked**
|
|
- Phase 2 conclusion: **safe to start as controlled remediation, not as training or promotion**
|
|
|
|
## 1. Purpose
|
|
|
|
This contract turns the accuracy program into an evidence-bearing engineering
|
|
program. Phase 1 establishes what exists, what actually runs, where data and
|
|
model results originate, which claims are supportable, and which defects must
|
|
be corrected before any new training or release promotion.
|
|
|
|
Phase 1 is complete when the required inventory, lineage, baseline, risk,
|
|
metric, roadmap, machine-readable status and retained evidence exist and can be
|
|
reproduced. Completion of Phase 1 does **not** mean that GeoIntel is accurate,
|
|
nationally validated, release-ready, or fully trained.
|
|
|
|
## 2. Non-negotiable rules
|
|
|
|
1. No quality, coverage, authority, calibration or national-readiness claim may
|
|
be stronger than the retained evidence.
|
|
2. A green unit test that uses a fixture, mock, placeholder or source-text
|
|
assertion is not evidence of live geospatial or model quality.
|
|
3. Protected test data is never used for training, calibration, failure-driven
|
|
sampling, threshold selection or stopping decisions.
|
|
4. Dataset, label, tile, checkpoint, runtime and evaluation identities are
|
|
immutable hashes, not filenames or mutable display names.
|
|
5. Geometry entering persistence is validated, assigned an explicit source CRS,
|
|
transformed to the canonical CRS and checked for valid domain and units.
|
|
6. Official or authoritative status is server-attested; caller-controlled source
|
|
names can never establish authority.
|
|
7. Failed, unavailable and `not_configured` states remain explicit. A fallback
|
|
cannot silently be reported as a successful primary result.
|
|
8. Human review is only complete when reviewer identity, decision, timestamp,
|
|
sampled object/AOI identity and reviewed artifact hash are persisted.
|
|
9. Phase 1 does not delete, rewrite or promote datasets, checkpoints, caches,
|
|
outputs, migrations or server state. The GPU smoke is read-only and persists
|
|
no detections.
|
|
10. User-owned untracked files, including `.codex-artifacts/`, server
|
|
`.env.bak.*`, `.forgeflow/`, `storage/evaluation/` and
|
|
`storage/source-cache/`, remain untouched.
|
|
|
|
## 3. Evidence vocabulary
|
|
|
|
Every statement in this program uses one of these classes:
|
|
|
|
| Class | Meaning |
|
|
|---|---|
|
|
| observed | Directly read from the repository, database, storage, runtime or retained artifact |
|
|
| reproduced | Deterministic read-only reproducer demonstrated the behavior |
|
|
| tested | An executable test or build gate ran and its result is retained or recorded |
|
|
| inferred | A conclusion drawn from observed evidence; it must be labelled as an inference |
|
|
| planned | Not implemented or not yet evidenced |
|
|
| unavailable | The required system or artifact could not be inspected; reason is stated |
|
|
| blocked | A required gate cannot pass without remediation |
|
|
|
|
The words “implemented”, “configured”, “available”, “validated” and “promoted”
|
|
are not synonyms:
|
|
|
|
- **implemented** means code exists;
|
|
- **configured** means a runtime selects it;
|
|
- **available** means dependencies/assets can be loaded;
|
|
- **validated** requires a registered evaluation and acceptance decision;
|
|
- **promoted** requires every release gate and an auditable activation record.
|
|
|
|
## 4. Authorized Phase 1 mutations
|
|
|
|
The only authorized repository mutations are:
|
|
|
|
- audit collectors and deterministic reproducer scripts;
|
|
- focused tests for those collectors;
|
|
- `docs/accuracy-program/`;
|
|
- retained evidence under `artifacts/evidence/accuracy/P1/`;
|
|
- required execution-log and TODO updates;
|
|
- a selective `.gitignore` exception that tracks only the Phase 1 evidence
|
|
tree while keeping other generated `artifacts/` ignored.
|
|
|
|
No product behavior, model checkpoint, dataset, label or production database row
|
|
is changed in Phase 1.
|
|
|
|
## 5. Executed investigation surface
|
|
|
|
The audit covered:
|
|
|
|
- root and nested tracked source trees, Git state and ignored/untracked state;
|
|
- backend API routes, schemas, services, ORM entities and Alembic chain;
|
|
- frontend TypeScript, tests, typecheck and production build;
|
|
- CI/readiness scripts, container definitions and Tower deployment identity;
|
|
- Postgres/PostGIS table counts, statuses, lineage gaps and geometry integrity;
|
|
- mounted storage paths referenced by the database;
|
|
- active NVIDIA/PyTorch/Ultralytics runtime and active model hash;
|
|
- one existing Geel raster tile through the production YOLO adapter on
|
|
`cuda:0`;
|
|
- available Belgian building corpus, split, label, tiling, calibration,
|
|
checkpoint and human-review evidence;
|
|
- mocks, fixtures, fallbacks, placeholders and documentation drift.
|
|
|
|
Detailed outcomes live in documents 01 through 06 and in the P1 evidence tree.
|
|
|
|
## 6. Reproducibility entry points
|
|
|
|
Run from the repository root:
|
|
|
|
```powershell
|
|
python scripts/run_accuracy_phase1_baseline.py --output-dir artifacts/evidence/accuracy/P1
|
|
python scripts/reproduce_accuracy_phase1_findings.py
|
|
python -m pytest tests/test_accuracy_phase1_baseline.py -q -p no:cacheprovider
|
|
```
|
|
|
|
The runtime/database collector is intentionally executed inside the deployed
|
|
application container and uses read-only SQL with a statement timeout:
|
|
|
|
```text
|
|
scripts/collect_accuracy_phase1_runtime.py
|
|
```
|
|
|
|
The GPU collector is streamed into the deployed container and uses the
|
|
production adapter with an existing checksummed model, manifest and tile:
|
|
|
|
```text
|
|
scripts/collect_accuracy_phase1_inference_smoke.py
|
|
```
|
|
|
|
Its retained JSON includes the model, manifest and tile SHA-256 values, package
|
|
versions, CUDA device, seed, configuration, raster CRS and raw output summary.
|
|
It explicitly forbids deriving an accuracy or release claim from one smoke.
|
|
|
|
## 7. Proven release blockers
|
|
|
|
Phase 1 reproduced or observed all of the following:
|
|
|
|
- cross-theme coverage contamination can incorrectly return
|
|
`operational`;
|
|
- a nominal metre buffer is applied as degrees;
|
|
- Lambert coordinates can be persisted unchanged under SRID 4326;
|
|
- caller-controlled source identity can be elevated to authoritative coverage;
|
|
- YOLO validation scope is bypassable through a mutable Area-name substring;
|
|
- legal coverage identity changes when an Area display name changes;
|
|
- Area PATCH silently ignores a documented geometry field;
|
|
- four persisted Geel detections contain Lambert-domain coordinates while their
|
|
geometry column is labelled SRID 4326;
|
|
- the training loop contains a future protected-test feedback path;
|
|
- V56 has zero accepted human AOI reviews out of 180;
|
|
- split independence, pure-background coverage, temporal label alignment and
|
|
unique-object evaluation are insufficient;
|
|
- no candidate has protected-test, background-test and promotion evidence;
|
|
- the real CI pytest work directory fails during collection, the full root suite
|
|
has 17 stale assertion failures, Ruff has 112 findings and frontend lint is
|
|
not configured;
|
|
- the production AI image is not built or exercised by CI;
|
|
- a tracked nested source mirror contains 1,153 files, 68 of which differ from
|
|
root, creating source-of-truth ambiguity.
|
|
|
|
## 8. Decision gates
|
|
|
|
| Gate | Phase 1 decision |
|
|
|---|---|
|
|
| Runtime CUDA and active model load | pass |
|
|
| One production-adapter GPU inference | pass, runtime-only evidence |
|
|
| Database/storage reachability | pass for inspected references |
|
|
| Geometry/CRS integrity | fail |
|
|
| Authoritative-source integrity | fail |
|
|
| Coverage correctness | fail |
|
|
| Strict protected-test isolation | fail in code design |
|
|
| Human label acceptance | fail |
|
|
| Split independence and negatives | fail |
|
|
| Reproducible national metric baseline | absent |
|
|
| Backend release gate | fail |
|
|
| Lint and frontend lint | fail / absent |
|
|
| National model promotion | blocked |
|
|
| Start test-first Phase 2 remediation | allowed |
|
|
|
|
## 9. Stop and escalation conditions
|
|
|
|
Training and model promotion remain prohibited until Phase 2 has closed the
|
|
CRS, authority, coverage and protected-test isolation blockers and a frozen,
|
|
human-approved corpus exists. If a required source snapshot, reviewer decision,
|
|
immutable hash or independent holdout cannot be produced, the corresponding
|
|
claim remains blocked; it is never replaced by synthetic success.
|
|
|
|
The protected test may be opened exactly once for a pre-registered candidate
|
|
after all calibration gates pass. A failure after that opening creates a new
|
|
model-development cycle and requires a newly governed protected set; its errors
|
|
must not feed back into the same training lineage.
|