Gate completed checkpoints through training loop
GeoIntel release gates / Compile, test, contracts and builds (push) Canceled after 0s
GeoIntel release gates / Python and npm vulnerability policy (push) Canceled after 0s
GeoIntel release gates / GIS image, SBOM and container scan (push) Canceled after 0s

This commit is contained in:
Jens
2026-07-29 15:37:08 +02:00
parent b5155c702f
commit 1ce291d652
5 changed files with 56 additions and 7 deletions
+5
View File
@@ -96,6 +96,11 @@ The checkpointed orchestrator invokes this builder after every rejected
iteration, stores its checksum in `training-loop-state.json`, and uses the
resulting dataset YAML for the next checkpoint. A restart resumes both the
candidate weights and that exact failure-driven training input.
An already completed out-of-band checkpoint enters the same contract with
`--evaluate-initial-model`: the first iteration skips fitting, copies and
hashes the checkpoint, and begins at calibration. A rejection then follows
the identical failure-driven CUDA path and cannot open protected test evidence
early.
The orchestrator refuses to start unless every automated frozen-dataset gate
passes and the corpus contains zero blank/low-variance positive tiles. The
audit status may remain `needs_human_review` while training and objective
+6 -1
View File
@@ -11777,6 +11777,11 @@ Deployment evidence:
leak-free failure-driven sampler automatically, records its evidence
checksum and next dataset YAML in `training-loop-state.json`, and resumes
both the candidate weights and exact sampling input after interruption.
- Added `--evaluate-initial-model` for completed checkpoints such as v37. It
skips redundant fitting only for the first iteration, copies and hashes the
supplied weights, runs calibration first, and rejoins the same automatic
sampling/training path after rejection. Protected evidence remains closed
until calibration passes.
- Confirmed v37 epoch 1 completed on CUDA with validation precision `0.601`,
recall `0.455`, mAP50 `0.474` and mAP50-95 `0.205`; the run remains inactive
and these internal-validation metrics are not release evidence.
@@ -11786,7 +11791,7 @@ Verified in this pass:
- `py -3 -m pytest -q backend/tests/test_belgium_training_loop.py backend/tests/test_belgium_training_iteration_assessment.py backend/tests/test_belgium_training_portfolio.py`
(`12 passed`).
- `py -3 -m pytest -q backend/tests/test_belgium_training_loop.py backend/tests/test_failure_driven_yolo_sampling.py backend/tests/test_belgium_training_iteration_assessment.py`
(`15 passed`).
(`16 passed` after adding the completed-checkpoint entry contract).
Open:
+1
View File
@@ -973,6 +973,7 @@ This file now starts with the current implementation status. Older preparation/b
- [x] Exclude GRB/PICC features created after the corresponding dated imagery period while retaining auditable rejection evidence.
- [x] Allow the objective CUDA loop to consume an automatically clean `needs_human_review` corpus while keeping final human sign-off as a separate, mandatory promotion gate.
- [x] Persist checksummed train-only failure-driven sampling after every rejected loop iteration and resume the next checkpoint from that exact dataset YAML.
- [x] Add a guarded calibration-first entry point for completed checkpoints so v37 and future externally interrupted runs can rejoin the automated loop without redundant retraining.
- [x] Evaluate the completed v36 YOLO11x checkpoint calibration-first on the rotated v30 holdouts; reject it before opening test/background because the regional calibration gate failed.
- [ ] Finish and assess the leak-free v37 YOLO11x failure-driven CUDA iteration; open test/background evidence only if every calibration gate passes.