fix: restore GeoIntel and upgrade Atlas workbench
This commit is contained in:
Jens
@@ -1,139 +1,292 @@
|
||||
# DockDeck
|
||||
# GeoIntel Belgium and the Belgian North Sea
|
||||
|
||||
DockDeck is a calm, premium personal start dashboard for Unraid. Its normal Unraid profile is one branded application container, reads container inventory through the existing AppOps GET API and enriches it with sanitized local WebUI metadata.
|
||||
GeoIntel is a map-first GeoAI Workbench for Belgium and the Belgian North Sea.
|
||||
It combines governed official-source coverage, raster/vector processing,
|
||||
historical comparison, computer vision, QA/QC and geospatial exports.
|
||||
|
||||
## What the MVP includes
|
||||
Mol and the Kempen remain deep regression and model-validation references. The
|
||||
release scope is all of Belgium plus legally labelled Belgian maritime zones;
|
||||
source coverage remains explicit per theme and jurisdiction.
|
||||
|
||||
- automatic read-only container discovery with new apps hidden by default;
|
||||
- live online/offline status and 30-second visibility-aware polling;
|
||||
- editable app name, icon fallback, category, order, local URL and external URL fallback;
|
||||
- configurable categories, favorites and overview/category navigation;
|
||||
- fast local search with a Google fallback, always opening in a new tab;
|
||||
- Porcelain, Midnight and Harbor themes, four accents, three density and content-width profiles, and selectable backgrounds;
|
||||
- responsive layouts from compact mobile screens through tablets, desktops and ultrawide displays;
|
||||
- optional dashboard hero and status badges, plus editable category and favorite icons;
|
||||
- up to eight configurable service widgets with independent order, compact/standard/wide layouts, custom labels, metric order, per-metric value/gauge/progress presentation, primary metric, any-signal warning thresholds and refresh cadence;
|
||||
- line, area or bar charts with a selectable app/provider series, legend and 10–60 in-memory samples;
|
||||
- a default native Unraid widget with configurable statistics and charts for Docker inventory, array, parity, disk/pool, temperature and optional UPS status;
|
||||
- rich GET-only providers for AdGuard, Plex, Immich, Home Assistant, Sonarr/Radarr/Lidarr, Ollama, Glances, Tautulli, Paperless, Gitea, Jellyfin, Seerr, Prowlarr, Authentik, NPM, Portainer, Grafana, Prometheus and Nextcloud;
|
||||
- optional GET-only metrics bridges for Deluge, qBittorrent, JDownloader, Tdarr, Bazarr and Vaultwarden;
|
||||
- a searchable Widget Studio that explains available metrics, recommends useful panels and activates up to eight widgets;
|
||||
- persistent per-app metric visibility, order, presentation and card limits, with a tailored profile for every discovered app even before its optional data endpoint is connected;
|
||||
- native Audiobookshelf, Netdata and PeerTube readers plus an optional GET-only JSON endpoint for every custom or niche app;
|
||||
- an editable Tower name and Unraid WebUI URL that defaults to port `5000`;
|
||||
- masked provider credential forms in Settings, with immediate read-only connection testing and no secret echo;
|
||||
- automatic first-party favorite favicons with a resilient local monogram fallback;
|
||||
- a useful desktop sidebar with time, deck health and quick service links;
|
||||
- SQLite persistence plus versioned JSON export/import;
|
||||
- reusable layout presets with independent import/export, direct dashboard edit mode and secret-free diagnostics;
|
||||
- an installable PWA shell that explicitly excludes runtime API data from caching;
|
||||
- optional read-only Nginx Proxy Manager host matching;
|
||||
- one branded, read-only DockDeck container with a true transparent PNG Unraid-list icon, WebUI metadata and no Docker-socket mount.
|
||||
GeoIntel is not a generic dashboard or chatbot. The core product is:
|
||||
|
||||
## Local development
|
||||
> data → processing → geospatial output → QA/QC → export
|
||||
|
||||
Requirements: Node.js 22 or newer and npm 10 or newer.
|
||||
## Current milestone
|
||||
|
||||
**v1.0.0 - Belgium/North Sea release**
|
||||
|
||||
The canonical release controls are:
|
||||
|
||||
- `docs/00-start/START_HERE.md`
|
||||
- `docs/RC_SCOPE_FREEZE_BELGIUM_NORTH_SEA.md`
|
||||
- `docs/RC_ROADMAP_BELGIUM_NORTH_SEA.md`
|
||||
- `docs/RELEASE_RUNBOOK.md`
|
||||
- `docs/KNOWN_LIMITATIONS.md`
|
||||
- `docs/DEFINITION_OF_DONE.md`
|
||||
|
||||
Older milestone and sprint handoff files remain historical evidence. They do
|
||||
not override the active national/maritime scope freeze or RC roadmap.
|
||||
|
||||
## Core V1 vertical slice
|
||||
|
||||
The first implementation target is:
|
||||
|
||||
1. Project + Area creation.
|
||||
2. Dataset registration/upload and metadata extraction.
|
||||
3. Reference building layer loading.
|
||||
4. Predicted detection layer loading/import.
|
||||
5. QA/QC matching against reference polygons.
|
||||
6. Metrics and false positive/false negative outputs.
|
||||
7. GeoJSON export.
|
||||
8. Minimal map/workbench UI.
|
||||
|
||||
## Primary stack
|
||||
|
||||
- Frontend: React, TypeScript, MapLibre GL, Deck.gl, Tailwind.
|
||||
- Backend: FastAPI, Python.
|
||||
- Database: PostgreSQL + PostGIS.
|
||||
- GIS processing: GeoPandas, Shapely, Rasterio, PyProj, GDAL.
|
||||
- AI: PyTorch, Ultralytics YOLO, SAM-compatible architecture.
|
||||
- Jobs: Redis + RQ.
|
||||
- Storage: local filesystem first, MinIO-compatible later.
|
||||
|
||||
## Codex instructions
|
||||
|
||||
Codex must start with:
|
||||
|
||||
1. `docs/00-start/START_HERE.md`
|
||||
2. `prompts/codex/M11_ARCHITECT_MASTER_PROMPT.md`
|
||||
|
||||
Then follow the build order in:
|
||||
|
||||
- `docs/build/BUILD_ORDER_DEPENDENCY_GRAPH.md`
|
||||
- `docs/build/CODEX_OPERATING_SYSTEM.md`
|
||||
|
||||
Before every implementation pass, run available preflight/smoke scripts where applicable.
|
||||
|
||||
## Repo principle
|
||||
|
||||
This is a documentation-driven engineering repo. The documentation is not decorative; it is the control system for autonomous implementation.
|
||||
|
||||
## Fastest Day 1 command path
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
npm install
|
||||
npm run dev
|
||||
make readiness
|
||||
```
|
||||
|
||||
Open `http://127.0.0.1:5173`. The example environment enables deterministic discovery fixtures. Runtime data is written to `./data/dockdeck.db`.
|
||||
## Unraid / Tower deployment
|
||||
|
||||
Quality commands:
|
||||
GeoIntel runs on Unraid as an all-in-one DockerMan-native container. The container embeds PostGIS, runs the FastAPI backend internally, and serves the frontend through nginx on one editable web port.
|
||||
|
||||
Unraid template assets live in:
|
||||
|
||||
- `deploy/unraid/geointel.env.example`
|
||||
- `deploy/unraid/geointel-unraid-template.xml`
|
||||
- `deploy/unraid/geointel-icon.svg`
|
||||
- `deploy/unraid/geointel-icon.png`
|
||||
- `docker-compose.unraid.yml`
|
||||
|
||||
Copy the Unraid env template to `.env` in the checkout and edit ports/paths there:
|
||||
|
||||
```bash
|
||||
npm run format:check
|
||||
npm run lint
|
||||
cd /mnt/user/appdata/geointel
|
||||
cp deploy/unraid/geointel.env.example .env
|
||||
nano .env
|
||||
docker build -f deploy/unraid/Dockerfile.all-in-one -t geointel-all-in-one:latest .
|
||||
bash deploy/unraid/run-dockerman-container.sh
|
||||
```
|
||||
|
||||
Common editable values:
|
||||
|
||||
```env
|
||||
GEOINTEL_FRONTEND_PORT=1202
|
||||
GEOINTEL_STORAGE_PATH=/mnt/user/appdata/geointel/storage
|
||||
GEOINTEL_POSTGIS_DATA_PATH=/mnt/user/appdata/geointel/postgres-data
|
||||
```
|
||||
|
||||
The backend and PostGIS ports are intentionally not exposed to the LAN in the all-in-one runtime. See `deploy/unraid/README.md` for full setup, port-change and cleanup notes.
|
||||
|
||||
On Tower/Unraid, `scripts/deploy_tower.ps1` and `scripts/deploy_tower.sh` validate the Compose reference but build with plain `docker build`, then automatically install the editable DockerMan template as `/boot/config/plugins/dockerMan/templates-user/my-geointel.xml`, install the PNG icon as `/boot/config/plugins/dockerMan/images/geointel-icon.png`, remove any old Compose-owned `geointel` container and start the final container with DockerMan labels.
|
||||
|
||||
## Sprint 2 quick start
|
||||
|
||||
- Update dependencies:
|
||||
|
||||
```bash
|
||||
python -m pip install -e backend/.[dev]
|
||||
cd frontend && npm install
|
||||
```
|
||||
|
||||
- Run full readiness checks (with no scope expansion):
|
||||
|
||||
```bash
|
||||
python -m compileall backend/app
|
||||
cd backend && python -m pytest
|
||||
cd ../frontend && npm run typecheck && npm run build
|
||||
bash scripts/run_readiness_check.sh
|
||||
```
|
||||
|
||||
- Raster workflow validation command (backend only):
|
||||
|
||||
```bash
|
||||
bash scripts/smoke_backend_import.sh
|
||||
cd backend && python -c "from app.main import app; print(app.title)"
|
||||
```
|
||||
|
||||
If `rasterio` is not installed, raster metadata endpoints return `RASTER_PROCESSING_UNAVAILABLE` and the frontend displays the
|
||||
state as failed until the dependency is added.
|
||||
|
||||
## Sprint 4 raster foundation
|
||||
|
||||
- Raster operations now support:
|
||||
- raster metadata extraction,
|
||||
- raster preview generation,
|
||||
- raster clip by area (with provenance on derived datasets),
|
||||
- raster tile generation with manifest output.
|
||||
- Raster services are dependency-aware:
|
||||
- if `rasterio` is unavailable, endpoints return `RASTER_PROCESSING_UNAVAILABLE`.
|
||||
- if preview dependencies (`numpy`, `pillow`) are unavailable, preview generation is unavailable with a clear error.
|
||||
- Enable raster stack explicitly when needed:
|
||||
|
||||
```bash
|
||||
cd backend && python -m pip install -e .[dev,raster]
|
||||
```
|
||||
|
||||
## Sprint 5 raster analytics hardening
|
||||
|
||||
- Added raster band statistics (min/max/mean/std, nodata ratio/count, valid pixel count, dtype, optional histograms).
|
||||
- Added raster reproject workflow with CRS validation and provenance persistence.
|
||||
- Extended tile manifest expectations (`tile_set_id`, `tile_size`, `overlap`, `bounds`, `source_raster_id`, `tile_paths`, `tile_server`).
|
||||
- Clarified raster operation availability in frontend/backend docs (`RASTER_PROCESSING_UNAVAILABLE` and invalid-CRS cases).
|
||||
|
||||
- Raster workflow command set (where available):
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
python -m pip install -e .[dev,raster]
|
||||
python -m pytest
|
||||
cd ../frontend
|
||||
npm run typecheck
|
||||
npm test
|
||||
npm run test:e2e
|
||||
npm run build
|
||||
```
|
||||
|
||||
## Unraid / Docker Compose
|
||||
Then give Codex the prompt in:
|
||||
|
||||
Create an `.env` beside `docker-compose.yml`:
|
||||
- `prompts/codex/final/DAY_1_MASTER_PROMPT.md`
|
||||
|
||||
```dotenv
|
||||
DOCKDECK_BIND_ADDRESS=192.168.10.150
|
||||
DOCKDECK_PORT=1218
|
||||
DOCKDECK_EGRESS_SUBNET=172.31.240.16/28
|
||||
APPOPS_URL=http://192.168.10.150:1216
|
||||
UNRAID_TEMPLATES_DIR=/mnt/user/appdata/dockdeck/unraid-templates
|
||||
UNRAID_HOST=192.168.10.150
|
||||
UNRAID_URL=http://192.168.10.150:5000
|
||||
```
|
||||
|
||||
For the current Unraid host, `deploy/unraid.env.example` contains this non-secret profile and can be copied to `.env`. It pins a small egress subnet because this Unraid host has exhausted its automatic address pools; change it if the host network layout changes.
|
||||
## M13 Codex optimization
|
||||
|
||||
Unraid keeps its source templates root-only and templates can contain sensitive values. Create DockDeck's sanitized metadata mirror before starting Compose; the helper copies only `Name`, `WebUI` and `Icon`:
|
||||
For the first serious Codex build run, use:
|
||||
|
||||
- `prompts/codex/m13/DAY_1_OPTIMIZED_MASTER_PROMPT.md`
|
||||
|
||||
Codex should also use the relevant reusable skill under `skills/` for each implementation pass. Validate the optimization assets with:
|
||||
|
||||
```bash
|
||||
python3 deploy/sanitize-unraid-templates.py \
|
||||
/boot/config/plugins/dockerMan/templates-user \
|
||||
/mnt/user/appdata/dockdeck/unraid-templates
|
||||
make m13
|
||||
```
|
||||
|
||||
Then run. The base Compose file creates exactly one container named `DockDeck`:
|
||||
The full readiness path remains:
|
||||
|
||||
```bash
|
||||
make readiness
|
||||
```
|
||||
|
||||
|
||||
## M14 Build Launch
|
||||
|
||||
For the first serious implementation run, use:
|
||||
|
||||
- `docs/40-build-launch/SPRINT_1_SCOPE_FREEZE.md`
|
||||
- `docs/40-build-launch/BUILD_SUCCESS_DEFINITION.md`
|
||||
- `docs/40-build-launch/CODEX_STOP_RULES.md`
|
||||
- `prompts/codex/m14/CODEX_FIRST_DAY_MASTER_PROMPT.md`
|
||||
|
||||
Validate launch assets with:
|
||||
|
||||
```bash
|
||||
make m14
|
||||
```
|
||||
|
||||
Full readiness remains:
|
||||
|
||||
```bash
|
||||
make readiness
|
||||
```
|
||||
|
||||
## Sprint 1 execution (Sprint 1 only)
|
||||
|
||||
From a clean machine:
|
||||
|
||||
```bash
|
||||
cd backend && python -m pip install -e .[dev]
|
||||
cd ..
|
||||
make backend-install
|
||||
make frontend-install
|
||||
make readiness
|
||||
```
|
||||
|
||||
Copy `.env.example` to `.env` only when you want local overrides. Docker Compose has safe defaults for the local PostGIS/backend/frontend stack and does not require a root `.env` file to exist.
|
||||
|
||||
With Docker Compose, open the workbench at `http://localhost:1202`.
|
||||
|
||||
The Docker frontend is served by nginx and proxies `/api` and `/health` to the backend container, so browser clients should use the frontend URL only, for example `http://192.168.10.150:1202` on a LAN host.
|
||||
|
||||
Runtime containers include healthchecks for PostGIS, backend and frontend. After
|
||||
startup, inspect them with:
|
||||
|
||||
```bash
|
||||
docker compose up -d --build
|
||||
docker compose ps
|
||||
sudo sh deploy/refresh-unraid-icon.sh
|
||||
```
|
||||
|
||||
The final helper validates that the configured `<Icon>` URL returns a real PNG and replaces both DockDeck-specific caches used by Unraid DockerMan: the persistent cache and the active RAM cache. Run it after changing the icon or recreating the container. It never changes Docker state or touches another container's files. An explicit URL can be supplied as the second argument when no user template exists.
|
||||
|
||||
On a host without AppOps, use the optional hardened fallback overlay. Only that additional proxy sees the socket and it disables POST and all unused API families:
|
||||
Verify the browser-facing API proxy after rebuilding Docker images:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.socket-proxy.yml up -d --build
|
||||
bash scripts/verify_browser_runtime.sh http://localhost:1202 http://localhost:8000/health
|
||||
```
|
||||
|
||||
By default Compose publishes DockDeck only on `127.0.0.1:1218`. For this Unraid host, set `DOCKDECK_BIND_ADDRESS=192.168.10.150` so the trusted LAN can reach `http://192.168.10.150:1218`. DockDeck has no built-in authentication; do not publish it directly to the internet.
|
||||
Verify the Docker GIS runtime after rebuilding the backend image:
|
||||
|
||||
Persistent configuration lives in the named volume `dockdeck_data`. Normal preferences use `/data/dockdeck.db`; credentials entered through Settings use the permission-restricted `/data/integrations.env`. Protect volume backups because that environment file contains plaintext secrets. The JSON export never includes it.
|
||||
```bash
|
||||
bash scripts/verify_gis_runtime.sh http://localhost:1202
|
||||
```
|
||||
|
||||
## Environment variables
|
||||
On the LAN host use the published browser URL, for example:
|
||||
|
||||
| Variable | Purpose |
|
||||
| ------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
|
||||
| `DATABASE_PATH` | SQLite file path; defaults to `./data/dockdeck.db` locally and `/data/dockdeck.db` in Docker. |
|
||||
| `INTEGRATION_ENV_PATH` | Permission-restricted environment file used for credentials entered in Settings. |
|
||||
| `DOCKDECK_BIND_ADDRESS`, `DOCKDECK_PORT` | Host interface and port published by Compose; safely default to `127.0.0.1:1218`. |
|
||||
| `DOCKDECK_EGRESS_SUBNET` | Explicit small subnet for the app's outbound network. |
|
||||
| `APPOPS_URL` | Existing AppOps origin used for GET-only inventory and measured CPU/memory status. |
|
||||
| `DOCKER_PROXY_URL` | Optional internal URL used only by the socket-proxy fallback overlay. |
|
||||
| `UNRAID_TEMPLATES_PATH` | Read-only XML template mount inside the app. |
|
||||
| `UNRAID_HOST` | Host used to resolve `[IP]` in WebUI templates; defaults to `tower.local`. |
|
||||
| `UNRAID_URL` | Initial Tower/Unraid link; Settings can override it and a missing port defaults to `5000`. |
|
||||
| `UNRAID_RUNTIME_PATH`, `UNRAID_HWMON_PATH` | Container paths for minimum read-only Unraid runtime and hardware-monitor mounts. |
|
||||
| `UNRAID_UPS_STATUS_PATH` | Optional read-only APC/UPS status file inside the app container. |
|
||||
| `MOCK_DISCOVERY` | Enables deterministic local/test fixtures; never enable for production discovery. |
|
||||
| `NPM_URL`, `NPM_TOKEN` | Optional server-only Nginx Proxy Manager read credential. |
|
||||
| `NPM_USERNAME`, `NPM_PASSWORD` | Optional server-only alternative for requesting an NPM session token. |
|
||||
| `ADGUARD_URL`, `ADGUARD_USERNAME`, `ADGUARD_PASSWORD` | Optional AdGuard Home statistics for its selected widget. |
|
||||
| `PLEX_URL`, `PLEX_TOKEN` | Optional Plex sessions, transcodes and library count. |
|
||||
| `IMMICH_URL`, `IMMICH_API_KEY` | Optional Immich photo, video and storage totals. |
|
||||
| `HOME_ASSISTANT_URL`, `HOME_ASSISTANT_TOKEN` | Optional Home Assistant automation, light and availability totals. |
|
||||
| `OLLAMA_URL`, `GLANCES_URL` | Optional overrides; their discovered local app URLs provide zero-config read-only metrics. |
|
||||
| `TAUTULLI_URL`, `TAUTULLI_API_KEY` | Optional Plex activity, transcode and bandwidth metrics. |
|
||||
| `PAPERLESS_URL`, `PAPERLESS_TOKEN` | Optional Paperless document, inbox and archive-size metrics. |
|
||||
| `AUDIOBOOKSHELF_URL`, `AUDIOBOOKSHELF_TOKEN` | Optional library, book, podcast and item totals. |
|
||||
| `NETDATA_URL`, `PEERTUBE_URL` | Optional discovered-URL overrides for their native GET-only readers. |
|
||||
| `SONARR_*`, `RADARR_*`, `LIDARR_*` | Optional provider URL and API key pairs for queue and version data. |
|
||||
| `GITEA_WIDGET_*`, `JELLYFIN_*`, `SEERR_*`, `PROWLARR_*` | Optional server-side URL/token pairs for native GET-only metrics. |
|
||||
| `AUTHENTIK_*`, `PORTAINER_*`, `GRAFANA_*` | Optional server-side URL/token pairs for administrative summary metrics. |
|
||||
| `PROMETHEUS_URL`, `NEXTCLOUD_*`, `NPM_*` | Optional read-only monitoring, cloud and proxy metrics. |
|
||||
| `*_METRICS_URL`, `*_METRICS_TOKEN` | Optional GET-only bridge for providers whose native API would require POST-based control. |
|
||||
| `CUSTOM_WIDGET_<APP>_URL`, `CUSTOM_WIDGET_<APP>_TOKEN` | Settings-managed GET-only JSON metrics endpoint and optional bearer token for any other app. |
|
||||
| `GITEA_URL`, `GITEA_OWNER`, `GITEA_TOKEN` | Used only by `npm run gitea:init` to create/push the private repository. |
|
||||
```bash
|
||||
bash scripts/verify_gis_runtime.sh http://192.168.10.150:1202
|
||||
```
|
||||
|
||||
Provider secrets are never stored in SQLite, JSON exports, logs or tracked files and are never returned to the browser. Values entered in Settings are persisted as environment assignments in `INTEGRATION_ENV_PATH` with mode `0600`; use DockDeck only on a trusted LAN or behind authenticated HTTPS, and protect raw volume backups.
|
||||
Load the explicit offline demo workflow:
|
||||
|
||||
## Repository initialization
|
||||
```bash
|
||||
curl -X POST http://192.168.10.150:1202/api/v1/demo/workflow
|
||||
```
|
||||
|
||||
After a passing quality gate and with the three Gitea variables exported, run `npm run gitea:init`. The script refuses an unexpected existing remote, creates only a private repository, keeps the token out of Git configuration, and pushes `main` through a temporary authorization header.
|
||||
If `/api/v1/projects` returns frontend HTML instead of a JSON envelope, rebuild
|
||||
and restart the frontend container.
|
||||
|
||||
Useful direct verification commands:
|
||||
|
||||
```bash
|
||||
python -m compileall backend/app
|
||||
cd backend && python -c "from app.main import app; print(app.title)"
|
||||
python -m pytest
|
||||
cd ../frontend && npm run typecheck
|
||||
cd ../frontend && npm run build
|
||||
docker compose config
|
||||
bash scripts/run_readiness_check.sh
|
||||
```
|
||||
|
||||
If `make` or `docker` are unavailable in your shell, run the equivalent script entrypoints directly:
|
||||
|
||||
```bash
|
||||
bash scripts/backend_install.sh
|
||||
bash scripts/backend_test.sh
|
||||
bash scripts/frontend_install.sh
|
||||
bash scripts/frontend_typecheck.sh
|
||||
bash scripts/frontend_build.sh
|
||||
bash scripts/run_readiness_check.sh
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user