Data Operations and Retention Runbook

Purpose

This runbook governs persistent GeoIntel data for Belgium and the Belgian North Sea. It covers raw, normalized, derived, export, AI and release-evidence artifacts. No command in this runbook schedules itself and no source is downloaded implicitly.

Lifecycle classes

Class	Paths	Retention
Raw and normalized source	`originals/`, `uploads/`, `operator-data/`	Protected; replace no official edition in place
Derived and cache	`derived/`, `rasters/derived/`, `previews/`, `tiles/`, `masks/`	Eligible only when old and not referenced by persisted provenance
Exports	`exports/`	Retain persisted exports; unreferenced old files may be candidates
AI models	`/app/models` and `storage/models/`	Protected; activation and promotion evidence govern removal
Immutable evidence	`release-evidence/`, `operator-evidence/`	Never a cleanup candidate
Unknown	every unclassified path	Protected by default

Dataset, DatasetVersion, Export, Detection, Segmentation, Job and AnalysisRun paths and nested provenance are treated as references. Failed work remains audit provenance. Project archival remains reversible and does not physically delete its datasets or files.

Read-only audit

Run the complete local storage, database-reference, disk-pressure and national/regional/maritime source-family report:

docker exec geointel python /app/scripts/audit_data_operations.py \
  --minimum-age-days 7 \
  --output /app/storage/release-evidence/rc-current/data-operations.json

The command:

inventories storage without following symlinks;
reports bytes and file counts per lifecycle class;
reports free disk bytes and percentage before new acquisition;
reports missing persisted path references;
identifies only old, unreferenced files inside the explicit cleanup allowlist;
summarizes national, regional and maritime source names, versions and latest import dates;
reports old failed Jobs and AnalysisRuns without deleting them.

Use --fail-on-pressure warning for a stricter acquisition preflight and --fail-on-missing-reference for release gates. critical disk pressure blocks acquisition by default. Critical means less than 10 GiB free. Warning means less than 50 GiB free, or less than 2% and less than 250 GiB free. The combined percentage/absolute rule avoids falsely blocking a large Unraid array that still has hundreds of GiB available.

The existing project-level freshness policy remains authoritative:

docker exec geointel python /app/scripts/audit_source_freshness.py \
  --project-id <belgium-and-north-sea-project-id> \
  --api-url http://127.0.0.1/api/v1 \
  --json \
  --output /app/storage/release-evidence/rc-current/source-freshness.json

Together, these reports cover the national workbench, regional source families and Belgian North Sea datasets without performing an external refresh. Catalog probes remain separate, explicit and read-only.

Cleanup dry run

docker exec geointel python /app/scripts/cleanup_storage_artifacts.py \
  --minimum-age-days 7 \
  --max-delete 25

This is always a dry run unless --apply is present. Review every candidate. Release evidence, operator evidence, official source material, uploads, models, unknown paths and every persisted path reference are excluded.

Technical benchmark projects can be archived reversibly:

docker exec geointel python /app/scripts/archive_technical_projects.py --show-names

The normal project lifecycle archive path does not need destructive confirmation because it changes only status=archived and preserves all data.

Recoverable quarantine gate

First create a fresh backup with a byte-complete SHA-256 storage snapshot on the host:

bash scripts/backup_release_state.sh \
  --container geointel \
  --release-id rc10-before-cleanup-$(date -u +%Y%m%dT%H%M%SZ) \
  --output-root /mnt/user/appdata/geointel/backups \
  --storage-path /mnt/user/appdata/geointel/storage \
  --models-path /mnt/user/appdata/geointel/models \
  --inventory-mode sha256

Verify it read-only:

bash scripts/verify_release_backup.sh \
  --backup-dir /mnt/user/appdata/geointel/backups/<release-id> \
  --container geointel

The Unraid runtime mounts GEOINTEL_BACKUPS_PATH read-only at /app/backups. Only after reviewing the dry run may an operator move the exact candidates to a protected quarantine:

docker exec geointel python /app/scripts/cleanup_storage_artifacts.py \
  --minimum-age-days 7 \
  --max-delete <reviewed-candidate-count> \
  --backup-dir /app/backups/<release-id> \
  --backup-max-age-hours 24 \
  --confirm QUARANTINE_STORAGE_ARTIFACTS \
  --apply

The command re-runs the audit immediately before quarantine. It refuses the operation when the exact token is absent, the candidate count exceeds the operator limit, the backup is stale/incomplete, snapshot checksums differ, or the path is outside the cleanup allowlist. Every moved byte is hashed and retained below operator-evidence/cleanup-quarantine/<operation-id>/files/; an atomic manifest records its original path, quarantine path, size, checksum and backup identity. Cleanup uses a backup-to-quarantine hard-link state machine: the manifest records planned, linked and quarantined transitions so each crash window can be reconciled without losing the retained inode.

Restore a reviewed quarantine without overwriting any path:

docker exec geointel python /app/scripts/restore_storage_quarantine.py \
  --storage-root /app/storage \
  --manifest /app/storage/operator-evidence/cleanup-quarantine/<operation-id>/manifest.json \
  --confirm RESTORE_QUARANTINED_ARTIFACTS

Restore validates every retained checksum and never replaces an existing destination. It uses an exclusive hard link and reconciles both-file and one-file interruption states before updating the manifest, so an interrupted restore can be resumed and verified.

The older demo-export cleanup has the same gate and uses confirmation token DELETE_DEMO_EXPORTS.

Failure meaning

disk_pressure=critical: stop new source acquisition and resolve capacity.
missing_referenced_path_count>0: persisted provenance points to absent files; investigate before cleanup or release.
missing_manifest_artifact_count>0: a retained operator manifest names an absent non-authoritative intermediate. The direct database artifacts remain the release gate; retain this warning as provenance debt and do not delete the protected manifest or its other evidence.
checksum or backup-age failure: create and verify a new backup.
candidate count above --max-delete: keep dry-run mode and review; never increase the limit blindly.
symlink listed as skipped: inspect manually; the audit will not traverse or delete it.

No automatic cron or background cleanup is installed. Operators may schedule the read-only audit externally, but destructive commands must remain manual.

6.9 KiB Raw Blame History