Handoff completed training into closed loop
This commit is contained in:
@@ -101,6 +101,11 @@ An already completed out-of-band checkpoint enters the same contract with
|
||||
hashes the checkpoint, and begins at calibration. A rejection then follows
|
||||
the identical failure-driven CUDA path and cannot open protected test evidence
|
||||
early.
|
||||
Tower's v37 supervisor binds the completed `results.png` artifact to a
|
||||
versioned JSON argv list. The handoff starts the orchestrator detached exactly
|
||||
once; shell strings are not accepted. Subsequent iterations retain the frozen
|
||||
180-degree aerial rotation, vertical/horizontal flip, scale and translation
|
||||
parameters rather than silently reverting to generic augmentation defaults.
|
||||
The orchestrator refuses to start unless every automated frozen-dataset gate
|
||||
passes and the corpus contains zero blank/low-variance positive tiles. The
|
||||
audit status may remain `needs_human_review` while training and objective
|
||||
|
||||
@@ -11794,6 +11794,14 @@ Deployment evidence:
|
||||
transient duplicate resume processes were detected and terminated before
|
||||
another epoch completed. Detection now uses `docker top -eo pid,args`; a
|
||||
live one-shot check returned `monitoring` with exactly one GPU process.
|
||||
- Extended the supervisor with a one-time, JSON-list-only completion handoff.
|
||||
The active v37 supervisor now starts the calibration-first closed loop when
|
||||
`results.png` appears. The bound command permits up to 20 iterations, keeps
|
||||
protected test/background closed until calibration passes, and preserves
|
||||
the exact AdamW, 180-degree rotation, flip, scale and translation contract.
|
||||
- Added those aerial augmentation parameters to the orchestrator CLI and
|
||||
training command, preventing later failure-driven checkpoints from silently
|
||||
reverting to generic orientation assumptions.
|
||||
- Confirmed v37 epoch 1 completed on CUDA with validation precision `0.601`,
|
||||
recall `0.455`, mAP50 `0.474` and mAP50-95 `0.205`; the run remains inactive
|
||||
and these internal-validation metrics are not release evidence.
|
||||
|
||||
@@ -975,6 +975,7 @@ This file now starts with the current implementation status. Older preparation/b
|
||||
- [x] Persist checksummed train-only failure-driven sampling after every rejected loop iteration and resume the next checkpoint from that exact dataset YAML.
|
||||
- [x] Add a guarded calibration-first entry point for completed checkpoints so v37 and future externally interrupted runs can rejoin the automated loop without redundant retraining.
|
||||
- [x] Add and activate a host-side, exact-run-marker supervisor that resumes the v37 CUDA checkpoint after container recreation without launching concurrent trainers.
|
||||
- [x] Bind v37 completion to a one-shot JSON command that enters calibration-first evaluation and up to 20 automatic failure-driven CUDA iterations with the frozen aerial augmentation contract.
|
||||
- [x] Evaluate the completed v36 YOLO11x checkpoint calibration-first on the rotated v30 holdouts; reject it before opening test/background because the regional calibration gate failed.
|
||||
- [ ] Finish and assess the leak-free v37 YOLO11x failure-driven CUDA iteration; open test/background evidence only if every calibration gate passes.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user