Initial public release

docs/DATA_OPERATIONS_RUNBOOK.md

@@ -0,0 +1,172 @@
+# Data Operations and Retention Runbook
+
+## Purpose
+
+This runbook governs persistent GeoIntel data for Belgium and the Belgian
+North Sea. It covers raw, normalized, derived, export, AI and release-evidence
+artifacts. No command in this runbook schedules itself and no source is
+downloaded implicitly.
+
+## Lifecycle classes
+
+| Class | Paths | Retention |
+|---|---|---|
+| Raw and normalized source | `originals/`, `uploads/`, `operator-data/` | Protected; replace no official edition in place |
+| Derived and cache | `derived/`, `rasters/derived/`, `previews/`, `tiles/`, `masks/` | Eligible only when old and not referenced by persisted provenance |
+| Exports | `exports/` | Retain persisted exports; unreferenced old files may be candidates |
+| AI models | `/app/models` and `storage/models/` | Protected; activation and promotion evidence govern removal |
+| Immutable evidence | `release-evidence/`, `operator-evidence/` | Never a cleanup candidate |
+| Unknown | every unclassified path | Protected by default |
+
+Dataset, DatasetVersion, Export, Detection, Segmentation, Job and AnalysisRun
+paths and nested provenance are treated as references. Failed work remains
+audit provenance. Project archival remains reversible and does not physically
+delete its datasets or files.
+
+## Read-only audit
+
+Run the complete local storage, database-reference, disk-pressure and
+national/regional/maritime source-family report:
+
+```bash
+docker exec geointel python /app/scripts/audit_data_operations.py \
+  --minimum-age-days 7 \
+  --output /app/storage/release-evidence/rc-current/data-operations.json
+```
+
+The command:
+
+- inventories storage without following symlinks;
+- reports bytes and file counts per lifecycle class;
+- reports free disk bytes and percentage before new acquisition;
+- reports missing persisted path references;
+- identifies only old, unreferenced files inside the explicit cleanup
+  allowlist;
+- summarizes national, regional and maritime source names, versions and latest
+  import dates;
+- reports old failed Jobs and AnalysisRuns without deleting them.
+
+Use `--fail-on-pressure warning` for a stricter acquisition preflight and
+`--fail-on-missing-reference` for release gates. `critical` disk pressure
+blocks acquisition by default. Critical means less than 10 GiB free. Warning
+means less than 50 GiB free, or less than 2% and less than 250 GiB free. The
+combined percentage/absolute rule avoids falsely blocking a large Unraid
+array that still has hundreds of GiB available.
+
+The existing project-level freshness policy remains authoritative:
+
+```bash
+docker exec geointel python /app/scripts/audit_source_freshness.py \
+  --project-id <belgium-and-north-sea-project-id> \
+  --api-url http://127.0.0.1/api/v1 \
+  --json \
+  --output /app/storage/release-evidence/rc-current/source-freshness.json
+```
+
+Together, these reports cover the national workbench, regional source
+families and Belgian North Sea datasets without performing an external
+refresh. Catalog probes remain separate, explicit and read-only.
+
+## Cleanup dry run
+
+```bash
+docker exec geointel python /app/scripts/cleanup_storage_artifacts.py \
+  --minimum-age-days 7 \
+  --max-delete 25
+```
+
+This is always a dry run unless `--apply` is present. Review every candidate.
+Release evidence, operator evidence, official source material, uploads,
+models, unknown paths and every persisted path reference are excluded.
+
+Technical benchmark projects can be archived reversibly:
+
+```bash
+docker exec geointel python /app/scripts/archive_technical_projects.py --show-names
+```
+
+The normal project lifecycle archive path does not need destructive
+confirmation because it changes only `status=archived` and preserves all
+data.
+
+## Recoverable quarantine gate
+
+First create a fresh backup with a byte-complete SHA-256 storage snapshot on the host:
+
+```bash
+bash scripts/backup_release_state.sh \
+  --container geointel \
+  --release-id rc10-before-cleanup-$(date -u +%Y%m%dT%H%M%SZ) \
+  --output-root /mnt/user/appdata/geointel/backups \
+  --storage-path /mnt/user/appdata/geointel/storage \
+  --models-path /mnt/user/appdata/geointel/models \
+  --inventory-mode sha256
+```
+
+Verify it read-only:
+
+```bash
+bash scripts/verify_release_backup.sh \
+  --backup-dir /mnt/user/appdata/geointel/backups/<release-id> \
+  --container geointel
+```
+
+The Unraid runtime mounts `GEOINTEL_BACKUPS_PATH` read-only at `/app/backups`.
+Only after reviewing the dry run may an operator move the exact candidates to
+a protected quarantine:
+
+```bash
+docker exec geointel python /app/scripts/cleanup_storage_artifacts.py \
+  --minimum-age-days 7 \
+  --max-delete <reviewed-candidate-count> \
+  --backup-dir /app/backups/<release-id> \
+  --backup-max-age-hours 24 \
+  --confirm QUARANTINE_STORAGE_ARTIFACTS \
+  --apply
+```
+
+The command re-runs the audit immediately before quarantine. It refuses the
+operation when the exact token is absent, the candidate count exceeds the
+operator limit, the backup is stale/incomplete, snapshot checksums differ, or
+the path is outside the cleanup allowlist.
+Every moved byte is hashed and retained below
+`operator-evidence/cleanup-quarantine/<operation-id>/files/`; an atomic
+manifest records its original path, quarantine path, size, checksum and
+backup identity. Cleanup uses a backup-to-quarantine hard-link state machine:
+the manifest records `planned`, `linked` and `quarantined` transitions so each
+crash window can be reconciled without losing the retained inode.
+
+Restore a reviewed quarantine without overwriting any path:
+
+```bash
+docker exec geointel python /app/scripts/restore_storage_quarantine.py \
+  --storage-root /app/storage \
+  --manifest /app/storage/operator-evidence/cleanup-quarantine/<operation-id>/manifest.json \
+  --confirm RESTORE_QUARANTINED_ARTIFACTS
+```
+
+Restore validates every retained checksum and never replaces an existing
+destination. It uses an exclusive hard link and reconciles both-file and
+one-file interruption states before updating the manifest, so an interrupted
+restore can be resumed and verified.
+
+The older demo-export cleanup has the same gate and uses confirmation token
+`DELETE_DEMO_EXPORTS`.
+
+## Failure meaning
+
+- `disk_pressure=critical`: stop new source acquisition and resolve capacity.
+- `missing_referenced_path_count>0`: persisted provenance points to absent
+  files; investigate before cleanup or release.
+- `missing_manifest_artifact_count>0`: a retained operator manifest names an
+  absent non-authoritative intermediate. The direct database artifacts remain
+  the release gate; retain this warning as provenance debt and do not delete
+  the protected manifest or its other evidence.
+- checksum or backup-age failure: create and verify a new backup.
+- candidate count above `--max-delete`: keep dry-run mode and review; never
+  increase the limit blindly.
+- symlink listed as skipped: inspect manually; the audit will not traverse or
+  delete it.
+
+No automatic cron or background cleanup is installed. Operators may schedule
+the read-only audit externally, but destructive commands must remain manual.