184 lines
4.5 KiB
Markdown
184 lines
4.5 KiB
Markdown
# GeoIntel Kempen — Storage Architecture v1.0
|
|
|
|
GeoIntel stores metadata in PostgreSQL/PostGIS and binary/geospatial files on filesystem storage or object storage.
|
|
|
|
## Principles
|
|
|
|
- Database stores metadata, relationships and vector geometries.
|
|
- Filesystem/object storage stores original rasters, derived rasters, tiles, masks, reports and model artifacts.
|
|
- Every stored file must have a dataset/export/model record in the database.
|
|
- Never store large raster binary data directly in regular application tables in V1.
|
|
|
|
## Root storage layout
|
|
|
|
```text
|
|
storage/
|
|
uploads/
|
|
{project_id}/
|
|
rasters/
|
|
vectors/
|
|
lidar/
|
|
rasters/
|
|
derived/
|
|
{project_id}/{dataset_id}/
|
|
tiles/
|
|
{project_id}/{dataset_id}/{tile_set_id}/
|
|
masks/
|
|
{project_id}/{analysis_run_id}/
|
|
previews/
|
|
{project_id}/{dataset_id}/
|
|
exports/
|
|
{project_id}/
|
|
geojson/
|
|
csv/
|
|
reports/
|
|
coco/
|
|
yolo/
|
|
models/
|
|
detection/
|
|
segmentation/
|
|
training-runs/
|
|
cache/
|
|
grb/
|
|
osm/
|
|
sentinel/
|
|
dhmv/
|
|
```
|
|
|
|
## Upload policy
|
|
|
|
When a file is uploaded:
|
|
|
|
1. Save original file unchanged.
|
|
2. Compute checksum.
|
|
3. Extract metadata.
|
|
4. Create dataset record.
|
|
5. Create preview if applicable.
|
|
|
|
Required file metadata:
|
|
|
|
```yaml
|
|
path
|
|
original_filename
|
|
mime_type
|
|
size_bytes
|
|
checksum_sha256
|
|
created_at
|
|
storage_backend
|
|
```
|
|
|
|
## Derived data policy
|
|
|
|
Derived files must record:
|
|
|
|
- source dataset id(s)
|
|
- analysis run id or processing job id
|
|
- processing parameters
|
|
- software component version
|
|
- created_at
|
|
|
|
## Segmentation mask artifacts
|
|
|
|
Sprint 9 stores segmentation masks as filesystem artifacts and segmentation polygons as authoritative PostGIS records.
|
|
|
|
Default mask path convention:
|
|
|
|
```text
|
|
storage/masks/{project_id}/{analysis_run_id}/tile_{tile_index}/mask_{segmentation_id}.png
|
|
```
|
|
|
|
Optional run manifest convention:
|
|
|
|
```text
|
|
storage/masks/{project_id}/{analysis_run_id}/manifest.json
|
|
```
|
|
|
|
Mask files are provenance/debug artifacts. QA, map display and GeoJSON output must use persisted `segmentations.geometry` rather than mask files.
|
|
|
|
## Cleanup policy
|
|
|
|
Do not delete originals automatically. Derived outputs may be cleaned through explicit cache management.
|
|
|
|
## Official temporal source artifacts
|
|
|
|
Waterinfo raw station layers, timeseries responses and checksum manifests live
|
|
under `storage/operator-data/waterinfo/<scope>/`. These are immutable source
|
|
evidence; queryable annual Point snapshots are normal Dataset/vector_feature
|
|
records. Bounded orthophotos are normal raster Dataset files. Their WMS URL,
|
|
product/layer, request/spatial hash, temporal validity and limitations are held
|
|
in source/provenance metadata. Browser PNG rendering is derived on request and
|
|
does not replace the stored GeoTIFF.
|
|
|
|
Offline demo export artifacts can be inspected and cleaned with:
|
|
|
|
```bash
|
|
python scripts/cleanup_demo_artifacts.py
|
|
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --export-type project_report_html
|
|
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --max-delete 100 --apply
|
|
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py
|
|
```
|
|
|
|
The script is dry-run by default, targets only the explicit
|
|
`GeoIntel Demo - Building QA` project unless an exact `--project-name` is
|
|
provided, keeps the newest export artifacts per matching project and refuses to
|
|
delete files outside `STORAGE_ROOT`. It cleans `exports` records/files only; it
|
|
does not remove original uploads, vector features, QA/QC rows, projects, areas,
|
|
tiles, rasters or masks. `--max-delete` defaults to 25 and blocks oversized
|
|
apply runs until the operator increases the cap after reviewing dry-run output.
|
|
Repeat `--export-type` to restrict cleanup to selected artifact kinds.
|
|
|
|
Live runtime validation for this maintenance path is available as a dry-run
|
|
smoke:
|
|
|
|
```bash
|
|
bash scripts/verify_demo_cleanup_dry_run.sh
|
|
CLEANUP_MODE=container CLEANUP_CONTAINER=geointel bash scripts/verify_demo_cleanup_dry_run.sh
|
|
```
|
|
|
|
The smoke never passes `--apply`. It fails if the cleanup summary is not a
|
|
dry-run, if any export/file deletion is reported, or if the dry-run candidate
|
|
fields are missing.
|
|
|
|
## Model storage
|
|
|
|
Model artifacts live under:
|
|
|
|
```text
|
|
storage/models/
|
|
```
|
|
|
|
Database model registry records:
|
|
|
|
```yaml
|
|
model_id
|
|
name
|
|
task_type
|
|
framework
|
|
path
|
|
classes_json
|
|
version
|
|
created_at
|
|
metrics_json
|
|
```
|
|
|
|
## Exports
|
|
|
|
Every export is reproducible and linked to project/analysis.
|
|
|
|
Export record fields:
|
|
|
|
```yaml
|
|
id
|
|
project_id
|
|
analysis_run_id
|
|
export_type
|
|
path
|
|
format
|
|
created_at
|
|
parameters_json
|
|
```
|
|
|
|
## Local development default
|
|
|
|
Use local filesystem paths. Keep MinIO/object storage as future extension.
|