159 lines
3.3 KiB
Markdown
159 lines
3.3 KiB
Markdown
# GeoIntel Kempen — Storage Architecture v1.0
|
|
|
|
GeoIntel stores metadata in PostgreSQL/PostGIS and binary/geospatial files on filesystem storage or object storage.
|
|
|
|
## Principles
|
|
|
|
- Database stores metadata, relationships and vector geometries.
|
|
- Filesystem/object storage stores original rasters, derived rasters, tiles, masks, reports and model artifacts.
|
|
- Every stored file must have a dataset/export/model record in the database.
|
|
- Never store large raster binary data directly in regular application tables in V1.
|
|
|
|
## Root storage layout
|
|
|
|
```text
|
|
storage/
|
|
uploads/
|
|
{project_id}/
|
|
rasters/
|
|
vectors/
|
|
lidar/
|
|
rasters/
|
|
derived/
|
|
{project_id}/{dataset_id}/
|
|
tiles/
|
|
{project_id}/{dataset_id}/{tile_set_id}/
|
|
masks/
|
|
{project_id}/{analysis_run_id}/
|
|
previews/
|
|
{project_id}/{dataset_id}/
|
|
exports/
|
|
{project_id}/
|
|
geojson/
|
|
csv/
|
|
reports/
|
|
coco/
|
|
yolo/
|
|
models/
|
|
detection/
|
|
segmentation/
|
|
training-runs/
|
|
cache/
|
|
grb/
|
|
osm/
|
|
sentinel/
|
|
dhmv/
|
|
```
|
|
|
|
## Upload policy
|
|
|
|
When a file is uploaded:
|
|
|
|
1. Save original file unchanged.
|
|
2. Compute checksum.
|
|
3. Extract metadata.
|
|
4. Create dataset record.
|
|
5. Create preview if applicable.
|
|
|
|
Required file metadata:
|
|
|
|
```yaml
|
|
path
|
|
original_filename
|
|
mime_type
|
|
size_bytes
|
|
checksum_sha256
|
|
created_at
|
|
storage_backend
|
|
```
|
|
|
|
## Derived data policy
|
|
|
|
Derived files must record:
|
|
|
|
- source dataset id(s)
|
|
- analysis run id or processing job id
|
|
- processing parameters
|
|
- software component version
|
|
- created_at
|
|
|
|
## Segmentation mask artifacts
|
|
|
|
Sprint 9 stores segmentation masks as filesystem artifacts and segmentation polygons as authoritative PostGIS records.
|
|
|
|
Default mask path convention:
|
|
|
|
```text
|
|
storage/masks/{project_id}/{analysis_run_id}/tile_{tile_index}/mask_{segmentation_id}.png
|
|
```
|
|
|
|
Optional run manifest convention:
|
|
|
|
```text
|
|
storage/masks/{project_id}/{analysis_run_id}/manifest.json
|
|
```
|
|
|
|
Mask files are provenance/debug artifacts. QA, map display and GeoJSON output must use persisted `segmentations.geometry` rather than mask files.
|
|
|
|
## Cleanup policy
|
|
|
|
Do not delete originals automatically. Derived outputs may be cleaned through explicit cache management.
|
|
|
|
Offline demo export artifacts can be inspected and cleaned with:
|
|
|
|
```bash
|
|
python scripts/cleanup_demo_artifacts.py
|
|
python scripts/cleanup_demo_artifacts.py --keep-latest 3 --apply
|
|
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py
|
|
```
|
|
|
|
The script is dry-run by default, targets only the explicit
|
|
`GeoIntel Demo - Building QA` project unless an exact `--project-name` is
|
|
provided, keeps the newest export artifacts per matching project and refuses to
|
|
delete files outside `STORAGE_ROOT`. It cleans `exports` records/files only; it
|
|
does not remove original uploads, vector features, QA/QC rows, projects, areas,
|
|
tiles, rasters or masks.
|
|
|
|
## Model storage
|
|
|
|
Model artifacts live under:
|
|
|
|
```text
|
|
storage/models/
|
|
```
|
|
|
|
Database model registry records:
|
|
|
|
```yaml
|
|
model_id
|
|
name
|
|
task_type
|
|
framework
|
|
path
|
|
classes_json
|
|
version
|
|
created_at
|
|
metrics_json
|
|
```
|
|
|
|
## Exports
|
|
|
|
Every export is reproducible and linked to project/analysis.
|
|
|
|
Export record fields:
|
|
|
|
```yaml
|
|
id
|
|
project_id
|
|
analysis_run_id
|
|
export_type
|
|
path
|
|
format
|
|
created_at
|
|
parameters_json
|
|
```
|
|
|
|
## Local development default
|
|
|
|
Use local filesystem paths. Keep MinIO/object storage as future extension.
|