Automate failure-driven training continuation
This commit is contained in:
@@ -92,6 +92,10 @@ recall are repeated, while true negative train tiles are repeated when a
|
||||
regional precision gate or the pure-background gate fails. Calibration, test,
|
||||
background-test and validation AOIs are excluded by their frozen corpus split;
|
||||
the generated evidence records that no protected sample entered training.
|
||||
The checkpointed orchestrator invokes this builder after every rejected
|
||||
iteration, stores its checksum in `training-loop-state.json`, and uses the
|
||||
resulting dataset YAML for the next checkpoint. A restart resumes both the
|
||||
candidate weights and that exact failure-driven training input.
|
||||
The orchestrator refuses to start unless every automated frozen-dataset gate
|
||||
passes and the corpus contains zero blank/low-variance positive tiles. The
|
||||
audit status may remain `needs_human_review` while training and objective
|
||||
|
||||
@@ -11773,11 +11773,20 @@ Deployment evidence:
|
||||
immutable, automated failures are empty, spatial leakage is `ok`, and blank
|
||||
positive-tile count is zero. Human sign-off remains a separate mandatory
|
||||
final promotion gate.
|
||||
- Closed the next orchestration gap: a rejected iteration now invokes the
|
||||
leak-free failure-driven sampler automatically, records its evidence
|
||||
checksum and next dataset YAML in `training-loop-state.json`, and resumes
|
||||
both the candidate weights and exact sampling input after interruption.
|
||||
- Confirmed v37 epoch 1 completed on CUDA with validation precision `0.601`,
|
||||
recall `0.455`, mAP50 `0.474` and mAP50-95 `0.205`; the run remains inactive
|
||||
and these internal-validation metrics are not release evidence.
|
||||
|
||||
Verified in this pass:
|
||||
|
||||
- `py -3 -m pytest -q backend/tests/test_belgium_training_loop.py backend/tests/test_belgium_training_iteration_assessment.py backend/tests/test_belgium_training_portfolio.py`
|
||||
(`12 passed`).
|
||||
- `py -3 -m pytest -q backend/tests/test_belgium_training_loop.py backend/tests/test_failure_driven_yolo_sampling.py backend/tests/test_belgium_training_iteration_assessment.py`
|
||||
(`15 passed`).
|
||||
|
||||
Open:
|
||||
|
||||
|
||||
@@ -972,6 +972,7 @@ This file now starts with the current implementation status. Older preparation/b
|
||||
- [x] Reject positive labels over blank/no-data imagery and replace partial SPW 2024 coverage with the complete dated SPW 2023 campaign.
|
||||
- [x] Exclude GRB/PICC features created after the corresponding dated imagery period while retaining auditable rejection evidence.
|
||||
- [x] Allow the objective CUDA loop to consume an automatically clean `needs_human_review` corpus while keeping final human sign-off as a separate, mandatory promotion gate.
|
||||
- [x] Persist checksummed train-only failure-driven sampling after every rejected loop iteration and resume the next checkpoint from that exact dataset YAML.
|
||||
- [x] Evaluate the completed v36 YOLO11x checkpoint calibration-first on the rotated v30 holdouts; reject it before opening test/background because the regional calibration gate failed.
|
||||
- [ ] Finish and assess the leak-free v37 YOLO11x failure-driven CUDA iteration; open test/background evidence only if every calibration gate passes.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user