geointel/docs/accuracy-program/08-data-contracts.md

# GeoIntel Accuracy Improvement Program — 08 Versieerbare datacontracten

- Status: Fase-2 contractnorm met een geïmplementeerde fail-closed
  validatiebibliotheek; bestaande DatasetVersions, artefacten en
  modelbestanden worden niet stilzwijgend betrouwbaar verklaard
- Scope: raster-, vector-, label- en modeldata, inclusief afgeleide datasets
  en uitvoerbare inference-inputs
- Samenhang: 05-metric-framework, 07-source-authority-matrix,
  docs/API_CONTRACTS.md en docs/DATABASE_IMPLEMENTATION_PLAN.md

## 1. Doel en harde regel

Een GeoIntel-resultaat is alleen herleidbaar wanneer alle gebruikte data een
versieerbaar contract heeft, de concrete artefacten checksummed zijn, de
validatie-uitkomst is bewaard en ieder derived artifact zijn parents en
transformaties kent.

Voor een nieuwe ingest geldt daarom, met de actuele P2-statuswoorden:

    ontvangen -> registered/configured -> ingested + contractvalidatie -> eligible of quarantined

`ready` is hoogstens een bestaande processingstatus, geen trust- of
consumptiestatus. Alleen een record met complete P2-evidence kan via de
consumption gate operationeel worden gebruikt. Er bestaat geen consumptiepad
van een ontvangen of legacy `ready` record naar training,
productie-inference, primaire QA, coverage authority of normale export dat een
gefaalde, ontbrekende of onbekende contractcontrole stilzwijgend overslaat. Een
incomplete legacyrecord blijft zichtbaar als `lineage_incomplete` en krijgt
geen gefingeerde metadata.

## 2. Huidige startsituatie versus Fase-2-doel

De generieke exact-versioned contracten zijn geïmplementeerd in
`backend/app/services/data_contract_validation.py`:
`geointel.vector.geojson@1.0.0`, `geointel.raster.geotiff@1.0.0`,
`geointel.label.yolo@1.0.0` (historisch, geen lege labels),
`geointel.label.yolo@1.1.0` (actueel, expliciete `pure_background`-modus) en
`geointel.model.pytorch@1.0.0`. De ingest- en derived-paden bewaren de
reportvelden. De validator levert een fail-closed report en
`DataQuarantineService` bepaalt de beslissing; `SourceRegistryService` en de
database-quarantinetrigger leggen de persistente quarantine en propagatie vast,
terwijl `DatasetConsumptionGate` operationele consumptie blokkeert. Deze
generieke contracten vervangen geen
toekomstige, strengere productprofielen voor GRB, PICC, UrbIS of
sensor-/modelversies.

| Onderwerp | Aangetroffen op 2026-08-01 | Fase-2 contractdoel |
|---|---|---|
| Datasetbasis | Dataset en DatasetVersion bevatten storage path, checksum, CRS, bounds, resolution/bands JSON, source/provenance JSON en temporele velden. De Phase-2-migratie voegt contract, registry/snapshot, validation, lineage, quarantine en idempotencyvelden toe. | Iedere nieuwe DatasetVersion krijgt contract-id/-versie, registry/source-snapshotbinding, validation report met contractfingerprint en lineage edges. |
| Uploadvalidatie | Generic upload en server-owned vector/rasteracquisities gaan via dezelfde registry/snapshot/contractflow; een upload blijft `manual` ongeacht caller text. | Generic metadata blijft descriptief; source authority, policy en training/production eligibility komen uitsluitend uit server-owned contracts. |
| Specialistische acquisitie | GRB, DHMV en andere adapterpaden voeren bron-specifieke checks uit en gebruiken de centrale registry/contract engine. | Elke adapter levert machineleesbare pass/fail/quarantine evidence; product-specifieke semantische profiles blijven afzonderlijke uitbreidingen. |
| Vector-CRS | Het beoogde model bewaart EPSG:4326, maar Phase 1 reproduceerde een non-4326 relabeling en vier historische detection geometrieën buiten het 4326-domein. | Bron-CRS is verplicht; transform wordt expliciet gelogd; relabeling zonder transform is verboden; legacy is quarantined of traceerbaar gerepareerd. |
| Versions | DatasetVersion en checksums bestaan, met unieke dataset/version. | Een inhoudelijke mutatie maakt een nieuwe snapshot/datasetversion met parent edge en reportfingerprint; de ingest-API wijzigt geen geaccepteerde evidence in place. |
| Labels/modellen | De Phase-1-training- en auditbestanden ontbeerden governed manifest-/review-/split-/runtimecontract. P2 voegt versioned YOLO-labelcontracten, release-/asset-/label-sidecars en een PyTorch-runtime-sidecar toe; historische bestanden blijven legacy. | Labels en modellen zijn first-class contractartefacten met hashes, split/firewall, review en runtime lineage. |
| Quarantine | `DatasetQuarantine`, een validation report en een consumption gate bestaan nu. | Quarantine heeft eigen status, reden, evidence en toegangsblokkade; gefaalde input kan nergens operationeel worden geconsumeerd. |

De bestaande velden zijn waardevol als legacy evidence, maar hun aanwezigheid
bewijst geen source authority, semantic validity of release eligibility. Dit
document voorkomt dat metadata-aanwezigheid met contractacceptatie wordt
verward.

## 3. Universeel contractenvelope

Elke valideerbare entiteit gebruikt een versieerbaar envelope. De fysieke
opslag kan relationeel, GeoJSON, GeoTIFF, manifest-JSON of modelbestand zijn.
De tabel hieronder is de normatieve envelope; de huidige relationele P2-kolommen
en `validation_report_json` realiseren de kernvelden. Nog niet getypeerde
snapshotvelden worden niet als aanwezig of groen verondersteld (zie 07 §5.3).

| Veldgroep | Verplicht veld of regel |
|---|---|
| identity | contract_id, contract_version, entity_kind, immutable entity_id, contract/schema fingerprint en waar van toepassing manifest_sha256 |
| source authority | source_registry_id, source_snapshot_id, authority_class, trust_status en allowed_use; de server-owned registrymetadata bevat de seeding/definition-revisie |
| artifact identity | storage URI/path, media type, byte count, sha256, original filename/object key en retained_at |
| time | observed_at, published_at, fetched_at/imported_at, valid_from, valid_to en precise status voor missing/not_applicable/unknown_with_reason |
| spatial | source_crs, canonical_crs, transformation identifier, bounds in source and canonical CRS, units, spatial resolution/scale, vertical datum/unit if applicable |
| coverage | legal/coverage zone IDs, geometry/bbox checksum, exact AOI/partition identity and coverage completeness/truncation status |
| semantic schema | expected geometry/band/class/attribute contract, ontology version, source feature-id strategy and known limitations |
| validation | validation_status, validator version/code commit, started/finished timestamps, checks run, issue list, warning list and report hash |
| lineage | ordered parent identities/hashes, transformation name/version/config hash, child artifact hashes and derivation purpose |
| lifecycle | SourceSnapshot: registered/configured/not_configured/available/ingested/failed/quarantined/legacy_unverified; Dataset contract: not_validated/passed/failed plus quarantine reason/evidence |

De envelope zelf is canonical JSON. De canonical bytes en ieder gerefereerd
input-/outputartefact zijn SHA-256-bound in een evidence manifest. Filename,
mutable display name and current configuration are never identity substitutes.

Voor een relationele `Dataset` of `DatasetVersion` die `passed` wordt, dwingt
de P2-migratie bovendien een niet-leeg, bijpassend validation report af: exact
contract en versie, registry/snapshot, report- en contractfingerprint,
`provenance_status=complete`, geldige lineage-status,
`quarantine_status=not_quarantined` en een canonieke SHA-256 die exact met de
gebonden `SourceSnapshot` overeenkomt. Die evidencevelden kunnen niet worden
vervangen terwijl het record `passed` blijft; een inhoudelijke correctie is een
nieuwe versie of eerst een expliciete invalidatie/quarantaine. De
databaseregel bewaakt die structurele binding en immutable evidencevelden, maar
herberekent geen contract- of reporthash en valideert geen bronsemantiek. De
applicatievalidator en `DatasetConsumptionGate` bewaken de inhoudelijke
artefactcontrole en consumptie; een JSON-vorm op zichzelf is nooit een
accuracy- of authority-claim.

### 3.1 Version rules

1. Contract major version changes only when semantics or required validation
   change incompatibly.
2. New optional fields may use a minor version only when their absence cannot
   relax a required gate.
3. A validation rule may become stricter only with a new contract version; old
   accepted artifacts retain their historical report and are not silently
   relabelled.
4. A re-export, clip, reprojection, normalization, tile build, label repair,
   calibration or model conversion is a derived artifact with a new checksum
   and lineage edge.
5. A source snapshot is immutable in its source identity/checksum semantics.
   Refresh creates a new snapshot and new DatasetVersion/derived graph; it
   never overwrites accepted evidence in place. Registry policy is
   server-owned; callers never supply an authority or policy version.

### 3.2 Required use policies

The allowed_use field is a server computed set. Its relevant values are:

| Use | Minimum state |
|---|---|
| map_context | accepted contextual/authoritative/corroborative dataset with attribution and spatial contract |
| qa_reference | accepted authoritative source for the exact task/zone/version, or a documented corroborative comparison role |
| training_input | accepted source/derived dataset plus label, temporal, review, split and leakage gates |
| calibration_input | training requirements plus frozen calibration split role |
| protected_test | sealed contract with protected access policy; inaccessible to train/val/calibration processes |
| production_inference_input | accepted raster/tile contract and model compatibility contract |
| export | accepted governed record with complete or explicitly not-applicable lineage; quarantined of lineage-incomplete historic records are not normally exportable |

No client-provided request field can add qa_reference, training_input or
production_inference_input to this set.

## 4. Common validation policy

### 4.1 Mandatory gates

Every ingest validates the following where applicable:

| Gate | Required behaviour | Failure disposition |
|---|---|---|
| source attestation | Registry source and snapshot are chosen by the server-side adapter or an explicit review action. | quarantine with SOURCE_UNATTESTED or SOURCE_POLICY_MISMATCH |
| checksum | Persisted content hash equals the observed or generated artifact hash and matches the manifest. | quarantine with CHECKSUM_MISMATCH |
| file safety | File type, byte size, decompression/container constraints and parser safety match the contract. | reject or quarantine before product persistence |
| CRS | Source CRS is declared and parseable; transform to canonical CRS is explicit and recorded. | quarantine with CRS_MISSING, CRS_INVALID or CRS_TRANSFORM_FAILED |
| bounding box | Bounds are finite, nonempty, in source CRS and consistent after transform with legal/domain scope. | quarantine with BBOX_INVALID or BBOX_OUT_OF_SCOPE |
| units/resolution | Unit, pixel/ground resolution, scale and vertical datum match source/product contract. | quarantine with UNIT_MISMATCH or RESOLUTION_MISMATCH |
| time/freshness | Required source version, observed/valid period, fetched time and freshness policy are satisfied or explicitly classified. | quarantine with TEMPORAL_METADATA_MISSING or FRESHNESS_REVIEW_REQUIRED |
| semantic schema | Geometry/bands/attributes/classes/ontology/feature IDs meet the contract. | quarantine with SCHEMA_MISMATCH |
| lineage | Every parent artifact/version/transformation is resolvable and checksummed. | quarantine with LINEAGE_MISSING or PARENT_INVALID |
| topology/integrity | Geometry/raster/label/model-specific validity is green. | quarantine with the specific integrity code |

At a consumption boundary a concrete snapshot must be `current` or explicitly
`not_applicable`. `unknown`, `review_required`, `due` and `stale` are not
eligible for production inference, primary reference validation, derived
processing, authoritative coverage or normal export. This guard is separate
from source-family classification: an authoritative registry entry never
overrides an unsuitable snapshot.

Warnings never change a failed required gate to accepted. A warning may retain
accepted only when the exact contract explicitly says that it is non-blocking,
the warning is visible, and the allowed_use policy excludes inappropriate uses.

### 4.2 CRS, units and geometry rules

- Canonical persisted vector/result geometry is EPSG:4326 only after an
  explicit source-to-canonical transform. Setting SRID to 4326 on coordinates
  in EPSG:31370, EPSG:3812 or any other CRS is prohibited.
- Every metric distance, buffer, area, length and topology tolerance uses an
  approved metric CRS or a documented geodesic operation. In the Belgian land
  scope, EPSG:31370 is the default candidate when suitable; an operation must
  record the chosen CRS and unit.
- The Belgian North Sea keeps horizontal CRS and vertical datum separate.
  TAW, LAT, mDNG, NAP and depth/elevation semantics never convert implicitly.
- Bbox checks validate coordinate order, finite values, positive extent,
  source CRS domain, transformed canonical domain and declared coverage zone.
- Geometry repair is only allowed under a named repair policy. The original
  hash, repair method, before/after validity and changed feature count remain
  in provenance. A repair that changes type, dimension or exceeds the
  tolerance is quarantined for review.

## 5. Raster data contract

### 5.1 Required raster fields

| Group | Required contract content |
|---|---|
| artifact | GeoTIFF/COG or approved format, sha256, byte size, parser/driver result, compression/nodata and immutable storage path |
| spatial | source CRS, affine transform, width/height, source and canonical bounds, pixel size x/y and unit, pixel interpretation and resampling method |
| bands | band count/order, data type, colour interpretation or physical unit, nodata, scale/offset, valid pixel statistics and allowed ranges |
| source/time | source registry/snapshot, product/collection/scene/tile identity, source version, acquisition/observation period and fetched/imported time |
| coverage | requested AOI/partition hash, clipping/mosaic inputs, complete/truncated status, coverage percentage and edge/padding policy |
| lineage | original response/archive hash, normalization/clipping/mosaic operation and configuration hash, every parent raster hash |
| compatibility | task-specific accepted bands, resolution range, colour/channel arrangement and temporal suitability for label/model task |

The parser must reject a non-georeferenced analytical raster unless the
contract explicitly marks it image-space only. Image-space-only data cannot
produce map geometry or location claims.

### 5.2 Raster-specific checks

1. CRS, affine transform, width, height, bounds and resolution must agree
   numerically within a contract tolerance.
2. The raster must have the exact expected band count/data type/nodata policy
   for the source product; unknown values are not silently filled.
3. Resolution cannot be represented as finer than the source's native
   resolution. Derived resampling records source and analysis resolution
   separately.
4. Mosaic/clip outputs declare every parent tile, overlap/nodata merge rule,
   exact AOI mask and output checksum.
5. Acquisition time is separate from fetch time. A rolling latest orthophoto
   with unknown per-pixel observation time remains unknown_per_pixel.
6. A raster used for labels or metrics records the reference-to-imagery time
   delta and its policy decision. This closes the temporal source of
   D-TEMPORAL-MISMATCH without inventing a date.

### 5.3 Source-specific requirements

| Product family | Additional contract rule |
|---|---|
| Official orthophoto | Product key, provider, native resolution, colour mode, imagery edition/flight-date precision, coverage zone and licence are mandatory. It is imagery/context, not reference truth. |
| Sentinel-2 | Contract must declare processing level, tile/granule, acquisition time, band/resampling method, cloud/nodata mask and native ground resolution. It remains contextual and cannot satisfy a building label/reference contract. |
| DHMV | Contract requires product key, DTM/DSM semantics, EPSG:31370, native/analysis resolution, one-band numeric units, nodata, TAW vertical reference and 2013-2015 period for the currently configured DHMV II source. It is elevation evidence, not building truth. |
| Bathymetry | Vertical datum/reference plane, depth versus bed-elevation semantics, survey epoch and uncertainty are mandatory. No water volume is inferred. |


### 5.4 Inference tile-manifest contract

Raster tiling emits `geointel.raster.tile-manifest@2.0.0`. The manifest binds
the source Dataset id/checksum/size, the latest DatasetVersion identity and
checksum, registry/snapshot identities and snapshot checksum, data-contract
identity, source version and—when present—the Area identity and geometry hash.
Each tile has an explicit CRS/bounds, byte size and SHA-256.

Configured detection and segmentation validate that exact binding against the
requested live Dataset before model loading. They recompute every tile hash,
normalize the tile union to EPSG:4326 and require it to remain inside the
manifest and persisted Dataset extents and intersect the bound Area. Missing,
legacy or changed evidence is not inferred from filenames: it fails closed with
a typed dataset, provenance, tile-integrity or spatial-scope error.

## 6. Vector data contract

### 6.1 Required vector fields

| Group | Required contract content |
|---|---|
| artifact | GeoJSON/approved vector artifact hash, feature count, media type, source/canonical CRS, original artifact retained path |
| geometry | allowed geometry types/dimensions, empty/valid counts, repair policy, source and canonical bbox, geometry precision/tolerance |
| identity | source feature-id attribute/namespace, uniqueness policy, stable-identity flag and collision report |
| attributes | contract version, required/optional fields, type/domain/null policy, canonical mapping and retained native properties policy |
| authority/time | registry source/snapshot, product/collection, zone, source version, observation/validity/fetched time, licence/attribution |
| topology | per-product topology rules, overlap/gap/containment/line connectivity policy and known exceptions |
| lineage | input artifact hashes, clip/filter/transform configuration, source feature counts and derived output feature mapping |

### 6.2 Vector-specific checks

1. De parser accepteert een feature collection met een expliciete CRS of de
   RFC 7946-standaard EPSG:4326. De ingestservice transformeert een niet-4326
   bron expliciet vóór canonical persistence; `build_vector_ingest_input`
   valideert daarna de getransformeerde feature collection en haar
   transformation evidence. Coördinaten buiten het Belgische/Noordzee-domein,
   een ongeldige CRS of een Lambertlaag die als 4326 wordt aangeboden, worden
   gequarantined; veldvorm alleen is nooit voldoende om een andere CRS te
   raden.
   De normale `Dataset.storage_path` en zijn checksum verwijzen daarna naar
   exact die canonieke EPSG:4326-bytes. Originele niet-canonieke bronbytes
   blijven uitsluitend als checksummed provenance-evidence buiten het
   consumptiepad bewaard; vectorbewerkingen weigeren een CRS- of
   storage-checksum-mismatch fail-closed.
2. Every feature has a nonempty supported geometry. Invalid features are
   quarantined individually or cause the dataset to quarantine according to
   an all-or-nothing policy recorded in the contract.
3. Source geometry is transformed before canonical PostGIS persistence.
   Post-transform coordinates must pass canonical domain/bbox checks.
4. Source feature IDs are unique whenever object lineage/temporal comparison
   needs them. Missing/duplicate/unexpected IDs set identity_stable=false and
   prevent object-level change claims.
5. Required attributes are type-checked before any source-to-canonical mapping.
   Unknown native fields may be retained in properties JSON but can never
   satisfy a required canonical attribute by name coincidence.
6. Topology rules are task-specific. A self-intersection is always invalid;
   overlapping building footprints may be a source semantic/review event
   rather than an automatic geometric repair. The contract states which.
7. Any clipping, buffering, intersection or dissolve creates a derived
   contract with parent hashes, metric CRS, units, tolerance, analysis run and
   queryable feature-persistence count. A derived Dataset cannot be accepted
   while its artifact and vectorfeatures diverge.

### 6.3 Building reference profiles

| Profile | Required fields / checks | Authority rule |
|---|---|---|
| GRB GBG | approved GRB collection, official feature ID, Polygon/MultiPolygon, governed request/response/artifact hashes, exact coverage, source snapshot/version, temporal limitation and EPSG:4326 output transform | Primary authoritative footprint reference in Flanders only when 07-source-authority-matrix conditions hold |
| Buildings Register | building object/version/status, snapshot date, bounded area, privacy-minimized output, raw-page hashes, register-to-GRB reconciliation and exact source URLs | Authoritative for its register semantics; corroborative for footprint reconciliation, never automatic GRB replacement |
| PICC buildings | allowlisted provider product, stable product identity, authorised Walloon coverage, feature ID, native CRS transform, edition/time and geometry contract | Authoritative only for approved Walloon building task/version |
| UrbIS buildings | allowlisted product, Brussels coverage, provider-native identity, native CRS transform, edition/time and geometry contract | Authoritative only for approved Brussels building task/version |
| OSM buildings | OSM extract/version/timestamp/licence, geometry and tag mapping if intentionally ingested | Contextual/corroborative only; no automatic truth, labels or coverage authority |

## 7. Label data contract

Labels are not just text files. Every label has a physical image/tile,
reference/source geometry and a training-policy decision.

### 7.1 Required label-manifest fields

| Group | Required content |
|---|---|
| identity | corpus_id/version, label artifact sha256, image/tile sha256, sample ID, tile-manifest sha256 and ontology/class-map version |
| spatial/image | image CRS/affine/bounds/resolution, tile size/stride/overlap, pixel coordinate convention and image orientation |
| reference | source registry/snapshot, native source feature IDs, geometry/reference hash, source/reference version and license eligibility |
| time | imagery observed/acquisition time, label/reference observed time, delta calculation, temporal policy/reason code |
| semantics | class, inclusion/exclusion rule, difficult/ambiguous/occluded flag, geometry-to-box conversion policy and minimum resolvable size |
| review | review state, reviewer/adjudicator ID, decision/reason code, contact-sheet/overlay hash and decision timestamp |
| split | train/val/calibration/test/background-test role, split-manifest hash, parent AOI/group ID and protected flag |
| eligibility | accepted, excluded, quarantined or uncertain; allowed training/evaluation use and all reasons |

### 7.2 Label validation

- Every label references an image/tile that exists and has exactly the recorded
  checksum. A label cannot migrate across tiles by filename matching.
- Coordinates are finite, in the declared pixel convention and within image
  bounds. Boxes/polygons must have positive area, supported class and
  deterministic normalization.
- The label's native source feature ID, source snapshot, geometry transform
  and image/reference time relation are mandatory for trainable labels.
- Below-resolution, post-imagery, dense/occluded, extreme-aspect and
  uncertain labels require an explicit disposition. Unknown is not accepted.
- A reviewed label preserves all earlier review decisions; repair produces a
  new label release and links the superseded artifact.
- Test- en background-testlabels zijn logisch beschermd door manifest-,
  release- en trainingsgates. Ze mogen niet aan samplers, augmentation,
  calibration, thresholdselectie of training worden aangeboden. Fysieke
  read-only opslag en afzonderlijke credentials mogen pas worden geclaimd
  wanneer daarvoor afzonderlijke P2-05-evidence bestaat.

`geointel.label.yolo@1.0.0` is historisch en laat geen lege labelbestanden
toe. `geointel.label.yolo@1.1.0` onderscheidt exact twee modi: `objects` voor
een niet-leeg YOLO-bestand en `pure_background` voor een leeg bestand. Een
leeg bestand wordt dus nooit automatisch als negatief voorbeeld geïnterpreteerd.
Voor `pure_background` zijn naast de normale labelchecks minstens verplicht:

- de exacte `sample_slug` en een toegestane `train`- of `val`-split;
- raster- en referencedataset-ID, checksums, registry en concrete snapshot
  via de upstream lineage;
- `review_decision=accepted`, reviewer-ID, tijdstip met tijdzone en hash van
  het menselijke reviewartefact/contact sheet.

Objectlabels met `pure_background` en lege labels zonder deze evidence falen
gesloten. Test- en achtergrond-testrollen worden niet toegelaten als deze
trainbare `pure_background`-modus.

Bij het maken van een trainingsrelease wordt ieder label opnieuw gevalideerd
en gebonden in `<dataset.yaml>.geointel-training-label-contracts.json`. Die
immutable sidecar bevat per label de checksum, sample, mode, exact
contract/fingerprint en de tellingen voor objecten/achtergronden; de release
en release-freeze binden vervolgens de sidecarhash. Voor het aanmaken,
verifiëren en elke initial/retry/resume-trainingsstap wordt de bevroren corpus
opnieuw tegen de live Dataset-status gecontroleerd. Een later gequarantined,
ontbrekende of niet-meer-eligible raster/referencepartner herroept de release
en blokkeert training, ook wanneer de freeze zelf ooit groen was.

The current V56 evidence has automated tile syntax checks but zero accepted
human AOI reviews. It therefore remains a legacy candidate corpus, not an
accepted label release under this contract.

Grayscale-transformaties, SAM-labelverfijningen, rotated holdouts en
proposal-crops controleren hun upstream release vóór verwerking, maar erven die
release niet als nieuwe trainability. Hun output is expliciet
`training_eligible=false` totdat een nieuwe governed corpus-, review-,
contract- en releasecyclus de nieuwe bytes heeft geaccepteerd. Dit voorkomt dat
een cosmetische of modelgestuurde afleiding stilzwijgend in een bestaande
PyTorch-trainingsrelease terechtkomt.

## 8. Model data contract

### 8.1 Required model-manifest fields

| Group | Required content |
|---|---|
| identity | model_id, model_version, weights sha256/size, format, architecture and class ontology version |
| origin | base weights hash/source, training code commit, training image/container digest, dependency lock/SBOM and license |
| data | corpus manifest sha256, label release sha256, split manifest sha256, training/validation/calibration roles and data access policy |
| config | preprocessing/tile/normalization/augmentation config hash, input dimensions/bands/colour order, threshold/NMS/max detections and seed |
| runtime | Python/PyTorch/CUDA/Ultralytics/driver versions, required device, GPU identity, deterministic flags and known nondeterminism |
| evidence | training run manifest/checkpoints/logs/curves, evaluator version, calibration and protected-test policy, model card scope polygon/zone IDs |
| lifecycle | candidate/unpromoted/validated/promoted/retired status, promotion report/rollback link and explicit supported/not_configured task scope |

### 8.2 Model validation

1. Weights hash, architecture and class map must load under the recorded
   runtime. Load success is runtime evidence only.
2. Input raster/tile contract must match bands, colour order, resolution,
   preprocessing and image geometry expected by the model.
3. Model scope is a server-owned geometry/zone policy bound to the model
   manifest, never a mutable Area display name.
4. Training reads only its allowed split roles. A protected credential/path,
   protected sample identifier or protected assessment causes a hard failure.
5. A detection/segmentation run persists the exact model manifest hash, model
   hash, inference config hash, tile manifest hash, dataset version,
   transform, seed, container/runtime/hardware and code commit before
   status=success can be emitted.
6. Legacy model runs with missing values remain lineage_incomplete. Current
   active configuration must not be copied backwards as an invented version.

### 8.3 Runtime sidecar gate for local PyTorch weights

Configured local YOLO and SAM weights are **not runnable** merely because a
`.pt` file exists or is listed as an active model asset. Before
`load_model()` is called, GeoIntel requires a neighbouring immutable sidecar:

    <weights-file>.geointel-model.json

The sidecar must validate exactly as `geointel.model.pytorch@1.0.0` and binds
the following at minimum:

- a fixed sidecar schema version, configured model ID and task type;
- the SHA-256 of the physical weights and the source-snapshot checksum of the
  same bytes;
- `model_format`, framework, non-empty class mapping and source/model version;
- source registry/snapshot UUIDs, a training-manifest checksum and non-empty
  upstream asset/checksum plus transformation lineage;
- a timezone-aware import timestamp; and
- `metadata.runtime_manifest_sha256`: a SHA-256 of canonical sidecar JSON with
  only that self-referential field omitted.

Missing, malformed, wrong-contract, version/task/framework-mismatched or
tampered sidecars make the capability `contract_incomplete`; detection and
segmentation do not load the adapter or weights. Validated sidecar evidence is
copied into the analysis run/job and into each configured detection or
segmentation provenance payload. This is a byte-binding and structural
lineage gate, not a promotion or accuracy claim: a model still needs its
separate training, calibration, protected-test and scope evidence before it
can be called validated or nationally suitable.

For production inference the structural sidecar check is insufficient on its
own. Immediately before adapter loading, GeoIntel resolves both UUIDs against
the database and requires the `model` registry entry, a matching immutable
snapshot, safe configured/current and ingested/current states, exact
model/sidecar/snapshot checksum and source-version equality, and no active
quarantine record. A missing, detached, mismatched, stale or quarantined
database record fails before `load_model()`; catalogue and preflight discovery
remain explicitly non-production structural checks.

The fixture detector/segmenter and placeholders remain explicit test-only
paths. They require `fixture_mode=true` where execution is allowed and cannot
be made production-ready by a sidecar or request parameter.

The Phase-1 active model has an asset hash and a successful GPU smoke, but its
historical detection runs lack the complete RunManifest fields and it is not
nationally validated. It cannot meet this model contract retroactively without
original evidence.

## 9. Derived data and lineage graph

### 9.1 Required lineage edge

Each derived entity records:

    parent entity id + parent manifest/hash + transformation id/version
    + configuration hash + code commit + runtime/container identity
    + output entity id + output hash + validation report hash

Examples include source raster to clipped raster, raster to tiles, source
vector to normalized vector, vector to label, labels to corpus, corpus to
checkpoint, checkpoint to inference run, detections to QA and QA to export.

The graph must be acyclic. A derived artifact cannot cite itself or a mutable
“latest” path as its only parent. Parent availability, hash and contract state
are checked recursively to the configured evidence boundary.

### 9.2 Idempotency and resume

An ingest/derivation idempotency key is the canonical combination of:
source snapshot or parent hashes, contract version, transformation/config hash,
requested AOI/partition hash and destination semantic role.

Repeated requests with the same key return the existing accepted artifact only
after rechecking its manifest/checksum/validation state. Interrupted work
resumes from staged checkpoint evidence or creates a new attempt record; it
does not invent success, duplicate a visible DatasetVersion or bypass
validation. A partial result is marked failed or quarantined and never reused
as a complete parent.

An identical replay may bind a second project-scoped ingest key to the same
immutable source snapshot. It preserves the snapshot's first recorded
`fetched_at`; all other immutable snapshot evidence, including checksum and
CRS, must still be equal. A changed fetch time alone therefore cannot rewrite
provenance or cause a duplicate snapshot.

## 10. Safe quarantine flow

### 10.1 Policy

Quarantine is preservation, not deletion. De huidige P2-implementatie bewaart
in een `DatasetQuarantine`-record de target-FK's, stage, één reason code,
details JSON, optioneel artifactpad/checksum, status en resolutietimestamps.
Het validation report blijft bij `Dataset`/`DatasetVersion`; lineage wordt via
de lineage-edges en de snapshotbinding herleid. `SourceRegistryService` en de
database-trigger maken het target, gedeelde snapshotpartners en downstream
lineage non-consumable voor training, productie-inference, normale QA en
automatische export. Een fysieke move/copy naar een afzonderlijke
access-controlled quarantine-storage is nog geen gerealiseerd
opslagmechanisme; die ontbrekende fysieke isolatie mag niet als bewijs worden
geclaimd. Tot die uitbreiding houdt de logische quarantine gate de verplichte
operationele veiligheidsgrens in stand.

| Event | Required action |
|---|---|
| parser/file failure | Retain minimal evidence and report; quarantine when artifact preservation is safe, otherwise reject with captured hash/error metadata |
| CRS/bounds/unit failure | Quarantine dataset/feature/run with source coordinates and validator report; never relabel CRS |
| source/trust failure | Route user/manual data to manual/untrusted or quarantine; do not materialize authoritative coverage |
| missing version/time/license | Quarantine when policy requires the field; source freshness shows review_required, not current |
| label-review/split failure | Quarantine label/corpus release and deny training/calibration/test use |
| model/run lineage failure | Mark lineage_incomplete or quarantine candidate output; deny validation/promotion |
| derived persistence mismatch | Roll back the transactional visible result when possible; retain failed attempt evidence and do not emit ready/success |

### 10.2 Quarantine dispositions

De gerealiseerde P2-statuswaarden zijn `quarantined`, `released` en
`rejected`, met de bewaartechnische velden hierboven. Een `released` quarantine
maakt de oorspronkelijke dataset niet opnieuw operationeel: een governed
re-ingest of nieuwe versie blijft vereist. Een rijkere disposition-taxonomie
met severity, secundaire reason codes, validatorversie, reviewer action,
`pending_review`, `remediated_as_new_version` en
`accepted_for_limited_historical_view_only` is vervolgwerk en mag nog niet als
gerealiseerde recordstructuur worden voorgesteld. Een reviewer kan nooit een
authoritative-, protected-split- of source-identity-gate in-place opheffen.

## 11. Contract test matrix

The implementation must retain tests for valid and deliberately invalid
fixtures. At a minimum:

| Dataset kind | Green fixture | Red fixture / expected outcome |
|---|---|---|
| governed GRB vector | bounded GBG fixture with registry attestation, valid CRS, unique IDs and request/response hashes | user upload claiming source_name=grb; it remains manual/untrusted or quarantined |
| vector CRS | EPSG:31370 or EPSG:3812 source transformed to EPSG:4326 with correct bounds | Lambert coordinates persisted as 4326 or source CRS absent; quarantine |
| vector geometry/topology | valid source polygons and explicit valid repair evidence | self-intersection, empty geometry, invalid dimensionality, duplicate required source ID or unsupported attribute type |
| raster/DHMV | one-band EPSG:31370, expected nodata/resolution/TAW metadata | wrong band count, unit, vertical reference, resolution, affine/bounds or checksum |
| Sentinel-2 | deliberately contextual fixture with full sensor/tile/time metadata | attempt to use it as building reference/ground truth; policy failure |
| label | bounded objectlabels of expliciet reviewed `pure_background` labels linked to image/hash/source/review/split manifest | out-of-bounds box, missing source feature, unmarked empty label, post-imagery unknown, unreviewed required sample or protected-role leakage |
| model/run | checksummed model with complete manifest and matching tile contract | model/run missing model hash, tile-manifest hash, GPU/runtime identity or scope policy; no success/promotion |
| lineage | one parent-to-child transformation graph with reproducible output hash | missing parent, changed parent checksum, cycle, partial operation or output count mismatch |
| quarantine | failed record remains non-consumable and visible with reason | attempt to train, infer, QA-as-truth, cover or export a quarantined artifact; hard denial |

Tests include unit, service, API, migration and live PostGIS coverage where
the contract relies on database constraints. A test that only asserts source
text is insufficient; it must exercise the actual validation and state
transition.

## 12. Fase-2 completion gates for data contracts

The source/provenance part of Fase 2 is green only when:

1. every new accessible DatasetVersion, raster, vector, label release and
   model/run type is routed through an explicit contract;
2. every new imported dataset has a server-attested registry/snapshot binding,
   checksum, validation report and lineage identity;
3. generic uploads cannot acquire authoritative source identity from caller
   text;
4. invalid CRS, geometry, bbox, unit, resolution, checksum, freshness,
   attribute, topology and lineage fixtures all fail closed;
5. quarantine is tested and has zero path into training, production inference,
   primary QA, coverage authority or normal export;
6. migrations preserve existing records, classify legacy gaps and do not
   silently backfill uncertain facts;
7. an evidence manifest retains test, migration and source/contract audit
   outputs under artifacts/evidence/accuracy/P2.

Until all gates are green, Fase 2 is in progress, Fase 3 is not ready and
training/promotion remains blocked by the Phase-1 execution contract.

> **Historische handoff:** deze zin legt de P2-gate vast zoals die bij oplevering
> van dit document gold. Fase 3 is nadien binnen de afgebakende lokale scope
> uitgevoerd. De actuele fasebeslissing staat uitsluitend in
> `docs/accuracy-program/status.json`.