geointel/docs/DATA_OPERATIONS_RUNBOOK.md

# Data Operations and Retention Runbook

## Purpose

This runbook governs persistent GeoIntel data for Belgium and the Belgian
North Sea. It covers raw, normalized, derived, export, AI and release-evidence
artifacts. No command in this runbook schedules itself and no source is
downloaded implicitly.

## Lifecycle classes

| Class | Paths | Retention |
|---|---|---|
| Raw and normalized source | `originals/`, `uploads/`, `operator-data/` | Protected; replace no official edition in place |
| Derived and cache | `derived/`, `rasters/derived/`, `previews/`, `tiles/`, `masks/` | Eligible only when old and not referenced by persisted provenance |
| Exports | `exports/` | Retain persisted exports; unreferenced old files may be candidates |
| AI models | `/app/models` and `storage/models/` | Protected; activation and promotion evidence govern removal |
| Immutable evidence | `release-evidence/`, `operator-evidence/` | Never a cleanup candidate |
| Unknown | every unclassified path | Protected by default |

Dataset, DatasetVersion, Export, Detection, Segmentation, Job and AnalysisRun
paths and nested provenance are treated as references. Failed work remains
audit provenance. Project archival remains reversible and does not physically
delete its datasets or files.

## Read-only audit

Run the complete local storage, database-reference, disk-pressure and
national/regional/maritime source-family report:

```bash
docker exec geointel python /app/scripts/audit_data_operations.py \
  --minimum-age-days 7 \
  --output /app/storage/release-evidence/rc-current/data-operations.json
```

The command:

- inventories storage without following symlinks;
- reports bytes and file counts per lifecycle class;
- reports free disk bytes and percentage before new acquisition;
- reports missing persisted path references;
- identifies only old, unreferenced files inside the explicit cleanup
  allowlist;
- summarizes national, regional and maritime source names, versions and latest
  import dates;
- reports old failed Jobs and AnalysisRuns without deleting them.

Use `--fail-on-pressure warning` for a stricter acquisition preflight and
`--fail-on-missing-reference` for release gates. `critical` disk pressure
blocks acquisition by default. Critical means less than 10 GiB free. Warning
means less than 50 GiB free, or less than 2% and less than 250 GiB free. The
combined percentage/absolute rule avoids falsely blocking a large Unraid
array that still has hundreds of GiB available.

The existing project-level freshness policy remains authoritative:

```bash
docker exec geointel python /app/scripts/audit_source_freshness.py \
  --project-id <belgium-and-north-sea-project-id> \
  --api-url http://127.0.0.1/api/v1 \
  --json \
  --output /app/storage/release-evidence/rc-current/source-freshness.json
```

Together, these reports cover the national workbench, regional source
families and Belgian North Sea datasets without performing an external
refresh. Catalog probes remain separate, explicit and read-only.

## Cleanup dry run

```bash
docker exec geointel python /app/scripts/cleanup_storage_artifacts.py \
  --minimum-age-days 7 \
  --max-delete 25
```

This is always a dry run unless `--apply` is present. Review every candidate.
Release evidence, operator evidence, official source material, uploads,
models, unknown paths and every persisted path reference are excluded.

Technical benchmark projects can be archived reversibly:

```bash
docker exec geointel python /app/scripts/archive_technical_projects.py --show-names
```

The normal project lifecycle archive path does not need destructive
confirmation because it changes only `status=archived` and preserves all
data.

## Destructive apply gate

First create a fresh backup with a SHA-256 storage inventory on the host:

```bash
bash scripts/backup_release_state.sh \
  --container geointel \
  --release-id rc10-before-cleanup-$(date -u +%Y%m%dT%H%M%SZ) \
  --output-root /mnt/user/appdata/geointel/backups \
  --storage-path /mnt/user/appdata/geointel/storage \
  --models-path /mnt/user/appdata/geointel/models \
  --inventory-mode sha256
```

Verify it read-only:

```bash
bash scripts/verify_release_backup.sh \
  --backup-dir /mnt/user/appdata/geointel/backups/<release-id> \
  --container geointel
```

The Unraid runtime mounts `GEOINTEL_BACKUPS_PATH` read-only at `/app/backups`.
Only after reviewing the dry run may an operator execute:

```bash
docker exec geointel python /app/scripts/cleanup_storage_artifacts.py \
  --minimum-age-days 7 \
  --max-delete <reviewed-candidate-count> \
  --backup-dir /app/backups/<release-id> \
  --backup-max-age-hours 24 \
  --confirm DELETE_STORAGE_ARTIFACTS \
  --apply
```

The command re-runs the audit immediately before deletion. It refuses the
operation when the exact token is absent, the candidate count exceeds the
operator limit, the backup is stale/incomplete, checksums differ, the storage
inventory is not SHA-256, or the path is outside the cleanup allowlist.

The older demo-export cleanup has the same gate and uses confirmation token
`DELETE_DEMO_EXPORTS`.

## Failure meaning

- `disk_pressure=critical`: stop new source acquisition and resolve capacity.
- `missing_referenced_path_count>0`: persisted provenance points to absent
  files; investigate before cleanup or release.
- `missing_manifest_artifact_count>0`: a retained operator manifest names an
  absent non-authoritative intermediate. The direct database artifacts remain
  the release gate; retain this warning as provenance debt and do not delete
  the protected manifest or its other evidence.
- checksum or backup-age failure: create and verify a new backup.
- candidate count above `--max-delete`: keep dry-run mode and review; never
  increase the limit blindly.
- symlink listed as skipped: inspect manually; the audit will not traverse or
  delete it.

No automatic cron or background cleanup is installed. Operators may schedule
the read-only audit externally, but destructive commands must remain manual.