Files
geointel/docs/KNOWN_LIMITATIONS.md
T
JensandClaude Opus 5 c8d32a4801 serve a promoted model at the post-processing it was gated on
scripts/evaluate_belgium_building_candidate.py freezes its post-processing
before the protected test — NMS IoU and a containment threshold selected during
calibration, defaulting to 1.0. The runtime applied a hardcoded 0.85, so a
promoted candidate was served under suppression its evaluation never measured
and dropped detections the gate had counted. Neither report showed the
difference. That constant was mine, added without noticing the evaluation
pipeline already had a tuned value for the same concept.

Containment is now configuration, recorded on every run beside the duplicate
IoU threshold, so an operator can serve a candidate at the value it was gated
at and afterwards see which value produced a given score.

Two runs that post-processed differently produced different candidate sets from
the same model output, so the comparison endpoint refuses to rank them. Runs
recorded before those values were persisted carry none, and absence is not
treated as a difference.

The remaining gap is deliberate and documented rather than closed: the gate
scores the model on its protected test set, the workbench scores the whole
pipeline including coverage clipping and the tile-edge filter. A promoted
candidate will not reproduce its gate F1 exactly, and pretending otherwise
would be the worse answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 21:15:05 +02:00

7.7 KiB

GeoIntel v1.0.0 Known Limitations

Release position

These limitations are explicit, bounded and non-deceptive. None is a hidden CRITICAL or HIGH release defect. Coverage and capability responses remain the runtime source of truth.

  • The access gate intentionally supports one environment-configured operator account. There is no registration, password-recovery email, organisation management or multi-user database. Optional guest access is a short-lived, read-only, demo-project-scoped role; it is not tenant isolation and must be used only on a dedicated demo instance without private data. Operator password rotation remains a configuration action followed by a restart.

Source coverage

  • National administrative land and maritime scope is operational from persisted NGI/RBINS editions. Detailed themes are federated by jurisdiction; an operational Flemish source does not imply equivalent Walloon or Brussels coverage.
  • Land cover remains source-correct by jurisdiction: WALOUS 2018/2020/2023 covers Wallonia, Flemish products cover Flanders and UrbIS Land Cover covers Brussels. GeoIntel does not relabel WALOUS as a national source or silently merge incompatible regional legends.
  • Buildings, population, terrain, imagery, nature, agriculture, soil and flood themes may report partial, not_configured or unsupported outside the materialized source partitions. The UI and exports retain that state.
  • Belgian North Sea planning/reporting boundaries are materialized. Bounded strict-TLS MDK WCS acquisition is implemented but stays disabled until the operator enables MDK_BATHYMETRY_ACQUISITION_ENABLED with a coverage id that the live readiness probe advertises. Until then the theme reports not_configured; VHA profile observations are not presented as a seabed model or water volume.
  • Official endpoints can be temporarily unavailable. Bounded acquisition fails closed and never substitutes fixture or fabricated production data.

Selection semantics

  • Coverage resolution currently accepts an EPSG:4326 rectangle. A rectangle around an irregular municipality or region can include land across a border or adjacent sea, so outside_supported_scope can be true even when the underlying named Area itself is wholly valid.
  • Cross-region and land/sea selections stay split by legal/source zone. Semantically incompatible metrics are not merged.
  • Raster analysis is cell-based. A selection smaller than one source cell is answered over the cells it touches, so the analysed area is larger than the drawn area; the response says so rather than returning zeros. Sub-cell precision is not available from a raster source.
  • The object count in a vector selection counts whole features that touch the selection, while area and length metrics clip to it. The response reports how many features the edge cuts; it does not attempt a fractional object count.
  • Flood-hazard percentages are shares of the modelled area. Where the VMM model does not cover the selection, that is reported as missing coverage and never as an absence of risk.
  • Change detection compares whole objects and does not clip them to the selection, so a footprint crossing the selection edge is compared in full. That is deliberate: clipping one snapshot and not the other would report the selection edge itself as a change. The count of such objects is reported.
  • The change-detection GeoJSON is capped by preview_limit (changes first). The counts always describe the whole selection; the map does not.
  • An export states its own completeness in geointel_provenance. A capped export is still a valid, usable file — it simply no longer implies it holds everything the selection contains.
  • Bounded acquisition refuses a redirect that leaves the configured origin, and refuses any private, loopback or link-local destination. An official endpoint that legitimately moves to a new host therefore fails closed until the operator updates the configured URL, which is the intended trade: bytes from an unexpected host must never be persisted under an official provenance.

Historical analysis

  • Historical comparison requires compatible source editions, units, coverage and observation periods. Sources without a governed series correctly return no comparable time series.
  • The RC10 storage audit reports 224 unavailable historical manifest intermediates from prior operator runs. There are zero missing current database artifact references; the historical paths remain visible as provenance warnings and are protected from cleanup inference.

AI

  • The configured local YOLO model is opt-in, building-focused and bounded by its documented operator evidence. It is not claimed to be an optimally trained general model for all Belgian objects or themes. The capability API exposes this as nationally_validated=false, validated_regions and an explicit validation scope; operator review remains required.
  • PyTorch and Ultralytics are present only in the AI image. No model weights auto-download. A missing local model reports unavailable.
  • Local YOLO-seg and SAM segmentation are implemented through the ultralytics interface but stay not_configured until the operator points YOLO_SEG_MODEL_PATH/SAM_MODEL_PATH to existing local weights and enables them explicitly. GeoIntel never downloads segmentation weights automatically; fixture segmentation remains explicit-only.
  • The training gate and the workbench measure different things, deliberately. scripts/evaluate_belgium_building_candidate.py scores the model on frozen post-processing over its protected test set; the workbench scores the whole production pipeline, which additionally clips to the persisted inference coverage and drops boxes truncated by an interior tile edge. A promoted candidate will therefore not reproduce its gate F1 exactly in the workbench. What must match is the post-processing: YOLO_CONTAINMENT_NMS_THRESHOLD and YOLO_DUPLICATE_IOU_THRESHOLD are recorded with every run, and two runs that used different values are reported as not comparable.
  • Detection QA reports both a strict footprint IoU and an envelope diagnostic. For an axis-aligned box detector the strict figure has a ceiling below 1 on rotated or non-rectangular buildings; the response states which geometry mode applies rather than silently attributing that gap to detection quality.
  • Segmentation inference still predicts one tile per call. Batching is implemented for detection only, because the SAM adapter is prompt-driven and does not share the detector's batch semantics.
  • Tile-edge truncated boxes are dropped in favour of the overlapping neighbour's complete view. On a manifest generated with overlap=0 there is no such neighbour, so an object on a seam is lost rather than duplicated. Generate tiles with an overlap larger than the biggest expected object.

Operations

  • Detection and segmentation runs can be queued through POST /detection/run-async and executed by the in-process analysis worker, so tiled GPU inference no longer blocks an HTTP request. This is a job table in PostgreSQL polled by the API process, not a distributed durable queue: there is no cross-host distribution and no automatic retry. The synchronous POST /detection/run remains available for small runs. Interrupted work is marked failed on restart and must be retried explicitly.
  • Other long GIS work still uses the synchronous job abstraction.
  • GeoIntel remains a controlled single-operator product. The optional guest demo adds bounded presentation access only; multi-user authorization and tenant isolation remain outside the product scope.
  • Cleanup remains manual, dry-run-first and confirmation-gated. No automatic retention schedule is installed.