229 lines
6.6 KiB
Markdown
229 lines
6.6 KiB
Markdown
# GeoIntel Kempen — Storage Architecture v1.0
|
|
|
|
GeoIntel stores metadata in PostgreSQL/PostGIS and binary/geospatial files on filesystem storage or object storage.
|
|
|
|
## Principles
|
|
|
|
- Database stores metadata, relationships and vector geometries.
|
|
- Filesystem/object storage stores original rasters, derived rasters, tiles, masks, reports and model artifacts.
|
|
- Every stored file must have a dataset/export/model record in the database.
|
|
- Never store large raster binary data directly in regular application tables in V1.
|
|
|
|
## Root storage layout
|
|
|
|
```text
|
|
storage/
|
|
uploads/
|
|
{project_id}/
|
|
rasters/
|
|
vectors/
|
|
lidar/
|
|
rasters/
|
|
derived/
|
|
{project_id}/{dataset_id}/
|
|
tiles/
|
|
{project_id}/{dataset_id}/{tile_set_id}/
|
|
masks/
|
|
{project_id}/{analysis_run_id}/
|
|
previews/
|
|
{project_id}/{dataset_id}/
|
|
exports/
|
|
{project_id}/
|
|
geojson/
|
|
csv/
|
|
reports/
|
|
coco/
|
|
yolo/
|
|
models/
|
|
detection/
|
|
segmentation/
|
|
training-runs/
|
|
cache/
|
|
grb/
|
|
osm/
|
|
sentinel/
|
|
dhmv/
|
|
```
|
|
|
|
## Upload policy
|
|
|
|
When a file is uploaded:
|
|
|
|
1. Save original file unchanged.
|
|
2. Compute checksum.
|
|
3. Extract metadata.
|
|
4. Create dataset record.
|
|
5. Create preview if applicable.
|
|
|
|
Required file metadata:
|
|
|
|
```yaml
|
|
path
|
|
original_filename
|
|
mime_type
|
|
size_bytes
|
|
checksum_sha256
|
|
created_at
|
|
storage_backend
|
|
```
|
|
|
|
## Derived data policy
|
|
|
|
Derived files must record:
|
|
|
|
- source dataset id(s)
|
|
- analysis run id or processing job id
|
|
- processing parameters
|
|
- software component version
|
|
- created_at
|
|
|
|
## Segmentation mask artifacts
|
|
|
|
Sprint 9 stores segmentation masks as filesystem artifacts and segmentation polygons as authoritative PostGIS records.
|
|
|
|
Default mask path convention:
|
|
|
|
```text
|
|
storage/masks/{project_id}/{analysis_run_id}/tile_{tile_index}/mask_{segmentation_id}.png
|
|
```
|
|
|
|
Optional run manifest convention:
|
|
|
|
```text
|
|
storage/masks/{project_id}/{analysis_run_id}/manifest.json
|
|
```
|
|
|
|
Mask files are provenance/debug artifacts. QA, map display and GeoJSON output must use persisted `segmentations.geometry` rather than mask files.
|
|
|
|
## Cleanup policy
|
|
|
|
Do not delete originals automatically. Derived outputs may be cleaned through explicit cache management.
|
|
|
|
## Official temporal source artifacts
|
|
|
|
Waterinfo raw station layers, timeseries responses and checksum manifests live
|
|
under `storage/operator-data/waterinfo/<scope>/`. These are immutable source
|
|
evidence; queryable annual Point snapshots are normal Dataset/vector_feature
|
|
records. Bounded orthophotos are normal raster Dataset files. Their WMS URL,
|
|
product/layer, request/spatial hash, temporal validity and limitations are held
|
|
in source/provenance metadata. Browser PNG rendering is derived on request and
|
|
does not replace the stored GeoTIFF.
|
|
|
|
BWK/Natura 2000 evidence lives under
|
|
`storage/operator-evidence/bwk-natura2000-2025/mol/`. The `raw/` directory
|
|
contains immutable WFS pages; the adjacent manifest records their URLs,
|
|
checksums, boundary checksum, page/feature counts, clipping diagnostics and
|
|
official class totals. The final GeoJSON is the retained source artifact while
|
|
`datasets` and `vector_features` remain the canonical queryable PostGIS state.
|
|
|
|
Definitive agricultural-use parcel evidence lives under:
|
|
|
|
```text
|
|
storage/operator-evidence/agricultural-use-parcels/{scope}/{year}/
|
|
agpa_{year}_*_public.zip
|
|
agricultural_use_parcels_{year}_{scope}.geojson
|
|
agricultural_use_parcels_{year}_crop_codes.json
|
|
agricultural_use_parcels_{year}_{scope}.manifest.json
|
|
```
|
|
|
|
The official ZIP is immutable source evidence and is never removed by normal
|
|
cache cleanup. The extracted GeoPackage is temporary to avoid retaining a
|
|
second full source copy. The normalized GeoJSON is the canonical upload
|
|
artifact; Dataset and vector_feature rows remain the queryable PostGIS state.
|
|
The manifest binds source, crop-code list and upload artifact checksums. A
|
|
checksum conflict with an existing annual Dataset fails closed.
|
|
|
|
Buildings and Addresses Register snapshot evidence lives under:
|
|
|
|
```text
|
|
storage/operator-evidence/buildings-addresses-register/mol/{observed-date}/
|
|
raw/
|
|
buildings_page_*.json
|
|
building_units_page_*.json
|
|
addresses_page_*.json
|
|
buildings_addresses_register.geojson
|
|
buildings_addresses_register.manifest.json
|
|
```
|
|
|
|
Raw pages contain the unmodified official response and are retained only as
|
|
checksummed operator evidence. They can contain address labels and must never
|
|
be served as a map/API artifact. The normalized GeoJSON deliberately contains
|
|
one polygon per register building with lifecycle state, aggregate relation
|
|
counts and classified GRB reconciliation only. It enters PostGIS exclusively
|
|
through DatasetService and ordinary `vector_features`; the operator never
|
|
writes database rows directly. The manifest binds all raw pages, the normalized
|
|
artifact, exact Area boundary and every GRB partition used for reconciliation.
|
|
|
|
Offline demo export artifacts can be inspected and cleaned with:
|
|
|
|
```bash
|
|
python scripts/cleanup_demo_artifacts.py
|
|
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --export-type project_report_html
|
|
python scripts/cleanup_demo_artifacts.py --keep-latest 10 --max-delete 100 --apply
|
|
docker compose exec -T backend python scripts/cleanup_demo_artifacts.py
|
|
```
|
|
|
|
The script is dry-run by default, targets only the explicit
|
|
`GeoIntel Demo - Building QA` project unless an exact `--project-name` is
|
|
provided, keeps the newest export artifacts per matching project and refuses to
|
|
delete files outside `STORAGE_ROOT`. It cleans `exports` records/files only; it
|
|
does not remove original uploads, vector features, QA/QC rows, projects, areas,
|
|
tiles, rasters or masks. `--max-delete` defaults to 25 and blocks oversized
|
|
apply runs until the operator increases the cap after reviewing dry-run output.
|
|
Repeat `--export-type` to restrict cleanup to selected artifact kinds.
|
|
|
|
Live runtime validation for this maintenance path is available as a dry-run
|
|
smoke:
|
|
|
|
```bash
|
|
bash scripts/verify_demo_cleanup_dry_run.sh
|
|
CLEANUP_MODE=container CLEANUP_CONTAINER=geointel bash scripts/verify_demo_cleanup_dry_run.sh
|
|
```
|
|
|
|
The smoke never passes `--apply`. It fails if the cleanup summary is not a
|
|
dry-run, if any export/file deletion is reported, or if the dry-run candidate
|
|
fields are missing.
|
|
|
|
## Model storage
|
|
|
|
Model artifacts live under:
|
|
|
|
```text
|
|
storage/models/
|
|
```
|
|
|
|
Database model registry records:
|
|
|
|
```yaml
|
|
model_id
|
|
name
|
|
task_type
|
|
framework
|
|
path
|
|
classes_json
|
|
version
|
|
created_at
|
|
metrics_json
|
|
```
|
|
|
|
## Exports
|
|
|
|
Every export is reproducible and linked to project/analysis.
|
|
|
|
Export record fields:
|
|
|
|
```yaml
|
|
id
|
|
project_id
|
|
analysis_run_id
|
|
export_type
|
|
path
|
|
format
|
|
created_at
|
|
parameters_json
|
|
```
|
|
|
|
## Local development default
|
|
|
|
Use local filesystem paths. Keep MinIO/object storage as future extension.
|