Add operator hard-negative detection matrix
This commit is contained in:
@@ -492,6 +492,24 @@ The combined `multi_sample_quality_summary.json` reports per-sample and overall
|
||||
best configurations. It is an operator benchmarking command, not a backend API
|
||||
or provider import path.
|
||||
|
||||
Before promoting any local model as a default, also run the hard-negative
|
||||
matrix against the documented background candidates:
|
||||
|
||||
```bash
|
||||
OPERATOR_SAMPLE_MANIFEST_PATH=storage/operator-data/operator_samples_manifest.json \
|
||||
OPERATOR_BACKGROUND_SAMPLE_SLUGS="postel_bos lommel_heide kasterlee_bos" \
|
||||
QUALITY_MODEL_ASSET_IDS="geointel-building-yolov8n-expanded160e50-pt geointel-building-yolov8n-tile30-pt yolov8s-building-segmentation-pt" \
|
||||
QUALITY_TILE_SIZES="640" \
|
||||
QUALITY_TILE_OVERLAPS="64" \
|
||||
QUALITY_THRESHOLDS="0.25 0.15 0.05" \
|
||||
bash scripts/run_operator_hard_negative_detection_matrix.sh http://192.168.10.150:1202
|
||||
```
|
||||
|
||||
This path uploads only background rasters, runs configured-YOLO detection and
|
||||
counts detections as false-positive pressure. It does not upload reference
|
||||
vectors or run QA/QC, so it cannot produce fake precision/recall metrics for
|
||||
empty background AOIs.
|
||||
|
||||
To inspect the evidence behind a calibration run, export the persisted QA
|
||||
evidence bundle:
|
||||
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def test_operator_hard_negative_matrix_scores_background_samples_without_qa() -> None:
|
||||
script_path = ROOT / "scripts" / "run_operator_hard_negative_detection_matrix.sh"
|
||||
readiness = (ROOT / "scripts" / "run_readiness_check.sh").read_text(encoding="utf-8")
|
||||
|
||||
assert script_path.exists()
|
||||
script = script_path.read_text(encoding="utf-8")
|
||||
|
||||
assert "bash -n scripts/run_operator_hard_negative_detection_matrix.sh" in readiness
|
||||
assert "OPERATOR_SAMPLE_MANIFEST_PATH" in script
|
||||
assert "OPERATOR_BACKGROUND_SAMPLE_SLUGS" in script
|
||||
assert "background_candidate" in script
|
||||
assert "allow_empty_reference" in script
|
||||
assert "/api/v1/detection/run" in script
|
||||
assert "/api/v1/detection/runs/${analysis_run_id}/detections" in script
|
||||
assert "hard_negative_matrix_summary.json" in script
|
||||
assert "false_positive_pressure" in script
|
||||
assert "best_by_lowest_pressure" in script
|
||||
assert "REAL_REFERENCE_VECTOR_PATH" not in script
|
||||
assert "/qa/reference" not in script
|
||||
assert "fixture_mode" not in script
|
||||
assert "demo/workflow" not in script
|
||||
Reference in New Issue
Block a user