Supervise checkpointed Tower training
GeoIntel release gates / Compile, test, contracts and builds (push) Canceled after 0s
GeoIntel release gates / Python and npm vulnerability policy (push) Canceled after 0s
GeoIntel release gates / GIS image, SBOM and container scan (push) Canceled after 0s

This commit is contained in:
Jens
2026-07-29 15:43:26 +02:00
parent 1ce291d652
commit d5bf9c102e
4 changed files with 139 additions and 0 deletions
+14
View File
@@ -11782,6 +11782,18 @@ Deployment evidence:
supplied weights, runs calibration first, and rejoins the same automatic
sampling/training path after rejection. Protected evidence remains closed
until calibration passes.
- The all-in-one container was externally recreated after v37 epoch 1. Both
456 MB checkpoints remained intact and training resumed from `last.pt` on
the RTX 4080 instead of restarting the experiment.
- Added and activated a host-side YOLO supervisor. It requires the exact run
marker, a valid checkpoint larger than 1 MB, a running target container and
absence of `results.png` before issuing a bounded resume. It exits on a
completed training artifact or a missing/incomplete checkpoint.
- The first live probe exposed that `docker top -eo args` is rejected by the
daemon and could misclassify an active Python-launched YOLO process. Two
transient duplicate resume processes were detected and terminated before
another epoch completed. Detection now uses `docker top -eo pid,args`; a
live one-shot check returned `monitoring` with exactly one GPU process.
- Confirmed v37 epoch 1 completed on CUDA with validation precision `0.601`,
recall `0.455`, mAP50 `0.474` and mAP50-95 `0.205`; the run remains inactive
and these internal-validation metrics are not release evidence.
@@ -11792,6 +11804,8 @@ Verified in this pass:
(`12 passed`).
- `py -3 -m pytest -q backend/tests/test_belgium_training_loop.py backend/tests/test_failure_driven_yolo_sampling.py backend/tests/test_belgium_training_iteration_assessment.py`
(`16 passed` after adding the completed-checkpoint entry contract).
- `py -3 -m pytest -q backend/tests/test_yolo_training_supervisor.py backend/tests/test_belgium_training_loop.py`
(`11 passed`), plus a live supervisor one-shot and single-process GPU audit.
Open:
+1
View File
@@ -974,6 +974,7 @@ This file now starts with the current implementation status. Older preparation/b
- [x] Allow the objective CUDA loop to consume an automatically clean `needs_human_review` corpus while keeping final human sign-off as a separate, mandatory promotion gate.
- [x] Persist checksummed train-only failure-driven sampling after every rejected loop iteration and resume the next checkpoint from that exact dataset YAML.
- [x] Add a guarded calibration-first entry point for completed checkpoints so v37 and future externally interrupted runs can rejoin the automated loop without redundant retraining.
- [x] Add and activate a host-side, exact-run-marker supervisor that resumes the v37 CUDA checkpoint after container recreation without launching concurrent trainers.
- [x] Evaluate the completed v36 YOLO11x checkpoint calibration-first on the rotated v30 holdouts; reject it before opening test/background because the regional calibration gate failed.
- [ ] Finish and assess the leak-free v37 YOLO11x failure-driven CUDA iteration; open test/background evidence only if every calibration gate passes.