Recover active training loop checkpoints

2026-07-29 16:32:01 +02:00
parent 18387448de
commit e73468318f
7 changed files with 174 additions and 6 deletions
@@ -106,6 +106,13 @@ versioned JSON argv list. The handoff starts the orchestrator detached exactly
 once; shell strings are not accepted. Subsequent iterations retain the frozen
 180-degree aerial rotation, vertical/horizontal flip, scale and translation
 parameters rather than silently reverting to generic augmentation defaults.
+Before fitting a state-pending iteration, the orchestrator checks its canonical
+run directory for `weights/last.pt`. If present, it resumes that exact CUDA
+checkpoint instead of restarting from the prior candidate. A host-side parent
+supervisor monitors the exact loop-process marker and persisted loop state; on
+container recreation it restores the current orchestrator script and launches
+the versioned JSON argv command. Invalid state and bounded launch exhaustion
+fail closed.
 The orchestrator refuses to start unless every automated frozen-dataset gate
 passes and the corpus contains zero blank/low-variance positive tiles. The
 audit status may remain `needs_human_review` while training and objective