diff --git a/CHANGELOG.md b/CHANGELOG.md index 104a9976..83e82711 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1283,3 +1283,12 @@ Added: - Segmentation now distinguishes configured model state from UI-runnable state and blocks the explicit test/demo-only fixture segmenter in the normal run form. - Added regression coverage for AI Lab action guardrails and compact guardrail styling. - No API contracts, migrations, product capabilities, live provider fetching or AI/model dependency changes were introduced. + +## Operator YOLOv8s hard-negative benchmark (2026-07-08) + +- Trained a Tower-local YOLOv8s hard-negative building detector from the existing operator tile dataset. +- Published the trained runtime artifact as `geointel-building-yolov8s-hardneg160r4e50-pt` in the live model asset catalog without adding application download behavior. +- Reused persisted dense QA and hard-negative benchmark runs through the existing live API. +- Observed dense QA F1 scores up to `0.6380` and safest current threshold behavior around `0.25`. +- Kept the model inactive by default because the `kasterlee_bos` hard-negative sample still produced 10 detections at threshold `0.25`. +- No repository code, API contracts, migrations, product behavior, provider fetching or AI dependency strategy changed in this benchmark pass. diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index c7c77608..c4d1cde8 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -5000,6 +5000,55 @@ Limitations: Next recommended pass: - Continue with V1 usability work that reduces operator confusion without expanding frozen product scope. +## Operator YOLOv8s hard-negative model benchmark (2026-07-08) + +Changed: +- No repository code, API contract, migration, UI or application behavior was changed in this pass. +- Downloaded the official Ultralytics YOLOv8s base model manually as an operator/runtime asset on Tower at `/mnt/user/appdata/geointel/models/yolov8s.pt`. +- Trained a local hard-negative building detector on Tower from the existing exported operator tile dataset `yolo-building-tile-hardneg160r4`. +- Produced the trained runtime model artifact `/mnt/user/appdata/geointel/models/geointel-building-yolov8s-hardneg160r4e50.pt`, mounted in the container as `/app/models/geointel-building-yolov8s-hardneg160r4e50.pt`. +- Verified the live model catalog exposes the trained model as `geointel-building-yolov8s-hardneg160r4e50-pt` with SHA256 `9bf71ad4742048ac77f07060b677bacd9757b8d310497fcada334d543e320d19`, size `22473194`, `will_download_models=false` and `active=false`. +- Reused existing persisted dense QA and hard-negative benchmark runs through the live API; no external provider fetching and no fake data were introduced. + +Training evidence: +- Base model: `/app/models/yolov8s.pt`. +- Base model SHA256: `1f47a78bf100391c2a140b7ac73a1caae18c32779be7d310658112f7ac9aa78a`. +- Training run: `/app/storage/training/operator-yolo/geointel-building-yolov8s-hardneg160r4e50`. +- Best checkpoint copied to: `/app/models/geointel-building-yolov8s-hardneg160r4e50.pt`. +- Final validation from Ultralytics: precision `0.449`, recall `0.404`, mAP50 `0.322`, mAP50-95 `0.110`. + +Live benchmark evidence: +- Created 24 persisted rescored detection runs with `model_asset_id=geointel-building-yolov8s-hardneg160r4e50-pt`. +- Dense persisted QA matrix results at IoU `0.1`: + - Four dense benchmark scenes saturated at 300 detections across thresholds `0.05`, `0.15` and `0.25`. + - Dense F1 scores observed: `0.6380`, `0.5627`, `0.5247`, `0.4678`. + - Dense precision ranged from `0.7167` to `0.8600`; recall ranged from `0.3247` to `0.5749`. + - Mean IoU ranged from `0.3962` to `0.4465`. + - The sparse/forest scene improved as the threshold increased: `49` detections and F1 `0.1429` at `0.05`; `18` detections and F1 `0.3200` at `0.15`; `10` detections and F1 `0.4706` at `0.25`. +- Hard-negative detection counts: + - `kasterlee_bos`: `49` at threshold `0.05`, `18` at `0.15`, `10` at `0.25`. + - `lommel_heide`: `0` at thresholds `0.05`, `0.15` and `0.25`. + - `postel_bos`: `1` at threshold `0.05`, `0` at `0.15` and `0.25`. + +Assessment: +- The YOLOv8s hard-negative model is a materially better evaluation candidate than the previous tiny smoke models and is useful in dense building scenes. +- The model should not be made the silent default yet because `kasterlee_bos` still produces 10 hard-negative detections at threshold `0.25`. +- Threshold `0.25` is the safest observed operating point for the current candidate. +- The next product step should be model-selection and threshold workflow hardening before any operator-facing default activation. + +Operational issue: +- Tower root SSH became unavailable after training. The key `widefrog_unraid_deploy` was offered, but the server returned `Permission denied (publickey)`. +- This blocks Codex-driven host-side deploy/sync until the Tower root `authorized_keys` entry is restored. +- Live HTTP API verification on `http://192.168.10.150:1202` remained reachable. + +Open: +- Restore root SSH access on Tower for `widefrog_unraid_deploy`. +- Add an operator-facing model selection/activation flow or documented command that can promote a chosen local model deliberately, with visible SHA256 and threshold guidance. +- Add more negative/background AOIs and a threshold calibration benchmark before activating this model by default. + +Next recommended pass: +- Implement V1 model-catalog hardening: show available local model assets, make active model/threshold explicit, and prevent accidental silent default activation. + ## Sprint 117 Safe local YOLO model activation (2026-07-06) Changed: diff --git a/docs/TODO.md b/docs/TODO.md index 3496ced9..65991cb0 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -415,5 +415,9 @@ This file now starts with the current implementation status. Older preparation/b - [x] Train/evaluate a small local GeoIntel building-detector smoke from the current operator samples and reject it because QA/QC did not improve. - [x] Add operator-only tile-level YOLO dataset export with overlapping windows and deterministic negative tile retention. - [x] Run tile-level training on Tower and accept/reject the resulting local model through the persisted QA/QC matrix. +- [x] Train/evaluate a YOLOv8s hard-negative local building-detector candidate on Tower and keep it inactive because hard-negative false positives remain. +- [ ] Restore Tower root SSH access for the `widefrog_unraid_deploy` key so Codex can deploy/sync again from the repo. +- [ ] Add an operator-facing local model catalog/activation workflow with SHA256, active model status and explicit threshold guidance. +- [ ] Add threshold calibration UX so detection runs do not silently rely on an unsafe default confidence. - [ ] Add more AOIs after the tile-level baseline so the next local model attempt is not limited to Geel/Mol/Turnhout. - [ ] Add negative/background AOIs so the next tile dataset is not all positive tiles. diff --git a/scripts/README.md b/scripts/README.md index abd81a6a..c56140c3 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -486,6 +486,19 @@ files are present without `--model-file`. It writes only `GEOINTEL_INSTALL_AI=true`, `YOLO_ENABLED=true`, `YOLO_MODELS_DIR=/app/models` and the mounted `YOLO_MODEL_PATH`. +Tower-local model evaluation status: + +- `geointel-building-yolov8s-hardneg160r4e50.pt` is available as an evaluated + local runtime artifact after the hard-negative YOLOv8s training pass. +- The live catalog id is `geointel-building-yolov8s-hardneg160r4e50-pt`. +- The model SHA256 is + `9bf71ad4742048ac77f07060b677bacd9757b8d310497fcada334d543e320d19`. +- The current safest observed threshold is `0.25`, but the model remains an + evaluation candidate because one hard-negative forest sample still produced + false detections at that threshold. +- Do not silently activate this model as a default. Apply it only as an explicit + operator choice until the model catalog/threshold workflow is hardened. + Clean old offline demo export artifacts without touching uploaded source data: ```bash