diff --git a/CHANGELOG.md b/CHANGELOG.md index dc250fbf..573e6e2c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ - Fixed the operator YOLO training wrapper so the all-in-one runtime defaults to `/opt/geointel/venv/bin/python` when present, while still falling back to `python3` for local shells. - Raised the Nginx upload limit to `250m` in both the compose frontend proxy and Unraid all-in-one proxy after a live 1024px GeoTIFF QA upload hit `413 Request Entity Too Large`. +- Raised Nginx proxy read/send timeouts to `600s` after a long low-threshold persisted YOLO/QA run hit `504 Gateway Timeout`. - Prepared a larger Tower operator sample manifest at `/app/storage/operator-data/operator-samples-1024` using explicit `1024x1024` rasters and doubled AOI half-size. - Exported and audited `/app/storage/operator-data/yolo-building-aoi1024-visible025`: 144 tiles, 117 positive tiles, 27 negative tiles, 15,079 labels and `min_label_visible_ratio=0.25`; audit remains `needs_attention` because median label area is still below gate. - Trained inactive local model asset `geointel-building-yolov8s-aoi1024visible025e50-pt` from the AOI1024 dataset. Ultralytics validation ended at approximately precision `0.275`, recall `0.331`, mAP50 `0.188` and mAP50-95 `0.0716`. diff --git a/backend/tests/test_docker_runtime_config.py b/backend/tests/test_docker_runtime_config.py index bd544b7e..4486761e 100644 --- a/backend/tests/test_docker_runtime_config.py +++ b/backend/tests/test_docker_runtime_config.py @@ -126,6 +126,15 @@ def test_nginx_runtime_allows_real_gis_upload_payloads() -> None: assert "client_max_body_size 250m;" in all_in_one_nginx +def test_nginx_runtime_allows_long_ai_and_qa_requests() -> None: + frontend_nginx = (ROOT / "frontend" / "nginx.conf").read_text(encoding="utf-8") + all_in_one_nginx = (ROOT / "deploy" / "unraid" / "nginx-all-in-one.conf").read_text(encoding="utf-8") + + for config in (frontend_nginx, all_in_one_nginx): + assert "proxy_read_timeout 600s;" in config + assert "proxy_send_timeout 600s;" in config + + def test_compose_does_not_publish_postgis_on_default_host_port() -> None: compose = (ROOT / "docker-compose.yml").read_text(encoding="utf-8") diff --git a/deploy/unraid/nginx-all-in-one.conf b/deploy/unraid/nginx-all-in-one.conf index 80864302..44994a91 100644 --- a/deploy/unraid/nginx-all-in-one.conf +++ b/deploy/unraid/nginx-all-in-one.conf @@ -2,6 +2,8 @@ server { listen 80; server_name _; client_max_body_size 250m; + proxy_read_timeout 600s; + proxy_send_timeout 600s; root /usr/share/nginx/html; index index.html; diff --git a/docs/CODEX_EXECUTION_LOG.md b/docs/CODEX_EXECUTION_LOG.md index dcd2eb69..5fafa142 100644 --- a/docs/CODEX_EXECUTION_LOG.md +++ b/docs/CODEX_EXECUTION_LOG.md @@ -5944,6 +5944,7 @@ Open: - Fixed `scripts/train_operator_yolo_detector.sh` so the all-in-one image uses `/opt/geointel/venv/bin/python` by default when that AI venv exists. Explicit `PYTHON_BIN` still wins, and local shells still fall back to `python3`. - Raised the Nginx request body limit to `250m` in both `frontend/nginx.conf` and `deploy/unraid/nginx-all-in-one.conf` after the live 1024px GeoTIFF upload path returned `413 Request Entity Too Large`. +- Raised Nginx proxy read/send timeouts to `600s` after the live low-threshold persisted YOLO/QA path reached `504 Gateway Timeout`. - Kept the change runtime-only: no API contract, persistence model, migration, model-download behavior or default model selection changed. ## What was tested @@ -5952,6 +5953,8 @@ Open: - `python -m pytest backend\tests\test_sprint129_operator_yolo_training_dataset.py -q` - Red/green TDD guard for real GIS upload payload support: - `python -m pytest backend\tests\test_docker_runtime_config.py::test_nginx_runtime_allows_real_gis_upload_payloads -q` +- Red/green TDD guard for long AI/QA proxy requests: + - `python -m pytest backend\tests\test_docker_runtime_config.py::test_nginx_runtime_allows_long_ai_and_qa_requests -q` - `bash -n scripts/train_operator_yolo_detector.sh` - Tower live model training on `/app/storage/operator-data/yolo-building-aoi1024-visible025`: - output model: `/app/models/geointel-building-yolov8s-aoi1024visible025e50.pt` diff --git a/docs/TODO.md b/docs/TODO.md index 1eb96c76..47b543d9 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -466,6 +466,7 @@ This file now starts with the current implementation status. Older preparation/b - [x] Fix YOLO preflight CLI so it respects Tower `.env` runtime configuration. - [x] Train a new inactive AOI1024 YOLOv8s candidate with visible-label filtering. - [x] Fix the all-in-one/compose Nginx upload limit after live 1024px GeoTIFF uploads hit `413 Request Entity Too Large`. +- [x] Fix the all-in-one/compose Nginx proxy timeout after low-threshold persisted YOLO/QA runs hit `504 Gateway Timeout`. - [ ] Rerun persisted QA/QC matrix for `geointel-building-yolov8s-aoi1024visible025e50-pt` after redeploying the upload-limit fix. - [ ] Add GRB paging or smaller dense AOI sampling before trusting 1000-feature-capped dense reference exports as full ground truth. - [ ] Keep every local YOLO candidate inactive until positive-AOI and hard-negative promotion reports recommend default activation. diff --git a/frontend/nginx.conf b/frontend/nginx.conf index f874f105..02996a8a 100644 --- a/frontend/nginx.conf +++ b/frontend/nginx.conf @@ -2,6 +2,8 @@ server { listen 80; server_name _; client_max_body_size 250m; + proxy_read_timeout 600s; + proxy_send_timeout 600s; root /usr/share/nginx/html; index index.html;