Operationalize RC10 data retention

2026-07-18 06:58:10 +02:00
parent bf867f8f4f
commit 7b96037853
22 changed files with 1306 additions and 2 deletions
@@ -0,0 +1,145 @@
+# Data Operations and Retention Runbook
+
+## Purpose
+
+This runbook governs persistent GeoIntel data for Belgium and the Belgian
+North Sea. It covers raw, normalized, derived, export, AI and release-evidence
+artifacts. No command in this runbook schedules itself and no source is
+downloaded implicitly.
+
+## Lifecycle classes
+
+| Class | Paths | Retention |
+|---|---|---|
+| Raw and normalized source | `originals/`, `uploads/`, `operator-data/` | Protected; replace no official edition in place |
+| Derived and cache | `derived/`, `rasters/derived/`, `previews/`, `tiles/`, `masks/` | Eligible only when old and not referenced by persisted provenance |
+| Exports | `exports/` | Retain persisted exports; unreferenced old files may be candidates |
+| AI models | `/app/models` and `storage/models/` | Protected; activation and promotion evidence govern removal |
+| Immutable evidence | `release-evidence/`, `operator-evidence/` | Never a cleanup candidate |
+| Unknown | every unclassified path | Protected by default |
+
+Dataset, DatasetVersion, Export, Detection, Segmentation, Job and AnalysisRun
+paths and nested provenance are treated as references. Failed work remains
+audit provenance. Project archival remains reversible and does not physically
+delete its datasets or files.
+
+## Read-only audit
+
+Run the complete local storage, database-reference, disk-pressure and
+national/regional/maritime source-family report:
+
+```bash
+docker exec geointel python /app/scripts/audit_data_operations.py \
+  --minimum-age-days 7 \
+  --output /app/storage/release-evidence/rc-current/data-operations.json
+```
+
+The command:
+
+- inventories storage without following symlinks;
+- reports bytes and file counts per lifecycle class;
+- reports free disk bytes and percentage before new acquisition;
+- reports missing persisted path references;
+- identifies only old, unreferenced files inside the explicit cleanup
+  allowlist;
+- summarizes national, regional and maritime source names, versions and latest
+  import dates;
+- reports old failed Jobs and AnalysisRuns without deleting them.
+
+Use `--fail-on-pressure warning` for a stricter acquisition preflight and
+`--fail-on-missing-reference` for release gates. `critical` disk pressure
+blocks acquisition by default. The thresholds are 10 GiB or 5% free for
+critical and 25 GiB or 10% free for warning.
+
+The existing project-level freshness policy remains authoritative:
+
+```bash
+docker exec geointel python /app/scripts/audit_source_freshness.py \
+  --project-id <belgium-and-north-sea-project-id> \
+  --api-url http://127.0.0.1/api/v1 \
+  --json \
+  --output /app/storage/release-evidence/rc-current/source-freshness.json
+```
+
+Together, these reports cover the national workbench, regional source
+families and Belgian North Sea datasets without performing an external
+refresh. Catalog probes remain separate, explicit and read-only.
+
+## Cleanup dry run
+
+```bash
+docker exec geointel python /app/scripts/cleanup_storage_artifacts.py \
+  --minimum-age-days 7 \
+  --max-delete 25
+```
+
+This is always a dry run unless `--apply` is present. Review every candidate.
+Release evidence, operator evidence, official source material, uploads,
+models, unknown paths and every persisted path reference are excluded.
+
+Technical benchmark projects can be archived reversibly:
+
+```bash
+docker exec geointel python /app/scripts/archive_technical_projects.py --show-names
+```
+
+The normal project lifecycle archive path does not need destructive
+confirmation because it changes only `status=archived` and preserves all
+data.
+
+## Destructive apply gate
+
+First create a fresh backup with a SHA-256 storage inventory on the host:
+
+```bash
+bash scripts/backup_release_state.sh \
+  --container geointel \
+  --release-id rc10-before-cleanup-$(date -u +%Y%m%dT%H%M%SZ) \
+  --output-root /mnt/user/appdata/geointel/backups \
+  --storage-path /mnt/user/appdata/geointel/storage \
+  --models-path /mnt/user/appdata/geointel/models \
+  --inventory-mode sha256
+```
+
+Verify it read-only:
+
+```bash
+bash scripts/verify_release_backup.sh \
+  --backup-dir /mnt/user/appdata/geointel/backups/<release-id> \
+  --container geointel
+```
+
+The Unraid runtime mounts `GEOINTEL_BACKUPS_PATH` read-only at `/app/backups`.
+Only after reviewing the dry run may an operator execute:
+
+```bash
+docker exec geointel python /app/scripts/cleanup_storage_artifacts.py \
+  --minimum-age-days 7 \
+  --max-delete <reviewed-candidate-count> \
+  --backup-dir /app/backups/<release-id> \
+  --backup-max-age-hours 24 \
+  --confirm DELETE_STORAGE_ARTIFACTS \
+  --apply
+```
+
+The command re-runs the audit immediately before deletion. It refuses the
+operation when the exact token is absent, the candidate count exceeds the
+operator limit, the backup is stale/incomplete, checksums differ, the storage
+inventory is not SHA-256, or the path is outside the cleanup allowlist.
+
+The older demo-export cleanup has the same gate and uses confirmation token
+`DELETE_DEMO_EXPORTS`.
+
+## Failure meaning
+
+- `disk_pressure=critical`: stop new source acquisition and resolve capacity.
+- `missing_referenced_path_count>0`: persisted provenance points to absent
+  files; investigate before cleanup or release.
+- checksum or backup-age failure: create and verify a new backup.
+- candidate count above `--max-delete`: keep dry-run mode and review; never
+  increase the limit blindly.
+- symlink listed as skipped: inspect manually; the audit will not traverse or
+  delete it.
+
+No automatic cron or background cleanup is installed. Operators may schedule
+the read-only audit externally, but destructive commands must remain manual.