Detection and segmentation run listings returned every run a project had ever
produced. Runs accumulate with every analysis while the panel only ever draws
the recent ones, so the response grew without bound for no benefit.
Both take limit and offset now and report total, limit, offset and truncated,
matching the result listings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
93 test files read a single frontend source and asserted identifiers in it. The
MapWorkspace split showed what that costs: 24 tests went red for a move that
changed no behaviour at all. A contract belongs to the feature — a container,
its hooks, its domain layer — not to whichever file currently holds it.
232 read sites now resolve through read_feature(). The distinction that makes
this safe is direction: a *positive* contract ("this is wired") may widen,
because the identifier must still exist somewhere in the feature; a *negative*
one ("this component performs no transport") is a statement about one file, and
widening it would quietly weaken the check. The 73 single-file reads that
remain are exactly those, and a guard now enforces the rule for new tests.
Verified rather than assumed: of the 732 migrated positive assertions, 644 still
match exactly one module — as specific as before — and the other 86 already
spanned a container and its hook by nature. Two apparent misses are an artefact
of the checking regex reading an escaped newline literally.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Segmentation reuses the detection suppressor but passed only the IoU threshold,
so it silently fell back to the hardcoded containment constant while detection
had just been given a configured one. Tuning containment for a promoted model
would have changed detection behaviour and left segmentation on the old value —
the same drift, one commit later.
Masks and boxes overlap differently, so segmentation carries its own setting
rather than borrowing the detector's, and records both thresholds on the run as
detection does.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
scripts/evaluate_belgium_building_candidate.py freezes its post-processing
before the protected test — NMS IoU and a containment threshold selected during
calibration, defaulting to 1.0. The runtime applied a hardcoded 0.85, so a
promoted candidate was served under suppression its evaluation never measured
and dropped detections the gate had counted. Neither report showed the
difference. That constant was mine, added without noticing the evaluation
pipeline already had a tuned value for the same concept.
Containment is now configuration, recorded on every run beside the duplicate
IoU threshold, so an operator can serve a candidate at the value it was gated
at and afterwards see which value produced a given score.
Two runs that post-processed differently produced different candidate sets from
the same model output, so the comparison endpoint refuses to rank them. Runs
recorded before those values were persisted carry none, and absence is not
treated as a difference.
The remaining gap is deliberate and documented rather than closed: the gate
scores the model on its protected test set, the workbench scores the whole
pipeline including coverage clipping and the tile-edge filter. A promoted
candidate will not reproduce its gate F1 exactly, and pretending otherwise
would be the worse answer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Detection QA reports a precision/recall curve, average precision and a
calibration sweep; segmentation QA reported a single operating point. Both rank
their outputs by confidence, so the same view applies, and the asymmetry meant
the two panels answered different questions about comparable runs — an
inconsistency introduced when detection gained the curve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The review vocabulary already separates a model error from a reference gap,
because the product's position is that official footprints are not
automatically perfect ground truth. Those verdicts were only counted. An
operator who inspected forty false positives and established that twelve are
buildings the reference simply lacks still saw a precision counting all forty
against the model — a number they had personally disproved, on the panel where
they disproved it.
Applying the verdicts gives an adjudicated score reported next to the raw one,
so nothing is quietly improved. Not being able to judge is not evidence in the
model's favour, so uncertain and obscured verdicts keep counting, as does a
decision from a later release that this runtime does not recognise.
Because part of the evidence is usually still unreviewed, the honest form is an
interval rather than a single corrected number: pessimistic assumes every
unreviewed finding is a model error, optimistic assumes none is, and the
headline equals the pessimistic reading so a partly reviewed check never
presents as a settled one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The workbench ranks model variants by a stored F1, each measured at that
variant's own confidence threshold. A conservatively calibrated detector then
looks worse than a liberal one without detecting anything differently: the
number says as much about the threshold as about the model. POST
/detection/runs/compare ranks on average precision instead, which describes the
whole ranking a model produced, and keeps each run's own-threshold F1 visible
next to it so the difference between the two readings is auditable.
Comparability comes before the ranking. Runs over different source rasters,
scored against different references, without a proven inference footprint, or
covering a different evaluated population are not alternatives to one another,
and no metric makes them so. The report names which of those applies and still
returns the numbers — they are simply not a ranking.
Each run is scored through the same QA path the workbench uses, so a comparison
and the persisted quality checks cannot drift apart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Threshold calibration ran the model over every tile once per threshold — three
GPU passes to compare 0.50, 0.25 and 0.15 on a hundred-tile raster. The answer
is already in a single run at the lowest value: detections above a higher cut
are a subset of it, and duplicate suppression walks candidates in descending
confidence, so a lower-confidence box can never displace a higher-confidence
one. The kept set above any cut is identical whichever threshold the run used,
which is what makes one pass sufficient rather than merely cheaper.
QA now takes calibration_thresholds and reads each operating point off the same
precision/recall walk it already performs, marking the F1-optimal cut. The lab
runs inference once and fills its table from the sweep.
The contract test asserted the per-threshold loop by name, pinning the waste it
was meant to describe. It now states what calibration owes an operator: a row
per requested threshold, from one run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Extracted for its error paths, not its size. Every step can fail or return
nothing, and each outcome has to leave the operator with a status that says
where the chain stopped — a workflow reporting "afgerond" after a step produced
nothing tells them a result exists when it does not. Inline in the component,
none of that was exercised; it now has fourteen tests covering each stopping
point, both rejection kinds, the reuse path and the preconditions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MapWorkspace held five near-identical useMemo blocks deciding which raster
image the map draws under the active theme — terrain, flood depth, thematic
raster, WALOUS land cover and bathymetry. Each filtered partitions by source
name, read bbox_epsg4326 and assembled the same overlay shape, so the parts
that genuinely differ per theme were buried in the repetition.
One builder makes the rule testable and leaves only the source, the label and
the opacity varying. A raster whose bounds are unusable is now skipped rather
than drawn from a partial bbox: an overlay in the wrong place is worse than no
overlay.
The legend asked "are these thematic or WALOUS overlays" by inspecting two of
the five lists. That is a property of the source, so it says so directly.
Two contract tests needed fixing rather than repointing. One asserted
`"api" not in source.lower()`, which the new hook name useMapImageOverlays
matches inside "useM-api-mageOverlays" — as would rapid, capital or therapy.
The contract is that this component talks to no API client, so it now says
that.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MapWorkspace.tsx opened with ~590 lines of theme catalogue, dataset matching
and label formatting above a 3.200-line component. None of it is React, all of
it is independently testable, and both render paths read from it, so it belongs
beside the pure helpers that already live in mapWorkspaceUtils.
The contract tests that read MapWorkspace.tsx would have gone red for a move
that changes no behaviour at all — 24 of them. That is the brittleness the
frontend_contract helper exists to remove, so it gains read_map_workspace():
the workspace is one feature spread over several modules, and a contract
belongs to the feature rather than to whichever file currently holds it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tile_manifest_path arrives in the detection and segmentation request and was
read straight off disk, and a manifest entry may name an absolute tile path.
That makes an API field an unbounded reference to the host filesystem, and it
contradicts the rule the persistence model rests on: only a governed,
runtime-produced artifact may be consumed, and a file outside the storage root
is not one.
Both the manifest and every tile it names now resolve under STORAGE_ROOT.
Resolution happens before the comparison, so ".." cannot climb out and a
sibling that merely shares a name prefix does not pass.
GEOINTEL_ALLOW_EXTERNAL_ARTIFACT_PATHS opts out for provisioning workflows that
stage tiles before ingest.
The check honours the Settings the caller is operating under rather than the
process-wide ones, because every analysis path already threads its own.
The affected tests write manifests into tmp_path, so they now declare tmp_path
as the storage root — which is what a deployment does, and makes the fixtures
more honest than they were.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two defects of the same kind: work that is supposed to be bounded is not.
The analysis worker selected queued jobs and then set them to running in a
second statement. A restarted process overlapping the previous one, or a second
replica, could both select the same row and both start tiled GPU inference on
it — duplicate analysis runs and double the GPU load. The AOI worker beside it
already claims with FOR UPDATE SKIP LOCKED; this uses a conditional update,
which is the same guarantee in one statement. run_once now reports jobs it
actually claimed rather than jobs it looked at.
urlopen follows redirects, so although every acquisition URL is built from
settings and cannot be steered by a request payload, a misconfigured or
compromised upstream could send the runtime to the loopback interface, to
another container on the compose network, or to a cloud metadata endpoint — and
the bytes would then be persisted under an official provenance. That is exactly
the substitution the product forbids. All eight fetch sites now open through a
guard that refuses private, loopback and link-local destinations (resolving the
host first, so a DNS name cannot hide one) and refuses a redirect that leaves
the configured origin or downgrades from HTTPS.
The guard is proven by calling the services' own fetch paths, not by grepping
for the call: every existing acquisition test injects an opener, which bypasses
it by design.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
/detection/runs/{id}/detections and its GeoJSON sibling returned every
persisted detection, as did the segmentation equivalents. A regional run holds
tens of thousands, and these are the endpoints the results table and the map
overlay call after every run.
They now take limit and offset, default to 2.000, and report total, limit,
offset and truncated so the complete population stays visible while what is
transferred does not. The GeoJSON responses carry the same window in a
geointel_result_window foreign member.
Rows are ordered by confidence, so a capped overlay draws the strongest
detections rather than an arbitrary slice, and the lab says how many of how
many are being shown rather than silently presenting a page as the whole run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
evidence_geojson emitted one feature per false positive, one per false negative
and two per match, with no limit. A regional check of 40k detections against 45k
reference footprints produced well over a hundred thousand features in a single
response, plus one warning string per unresolvable identifier. The endpoint the
entire review workflow depends on therefore failed exactly where review matters
most.
What to draw is now decided before any geometry is fetched, so the query work is
proportional to the result rather than to the size of the check — previously
130k geometries were resolved through an IN clause holding every identifier in
the check, to then discard most of them.
The budget is split between misses and false positives in proportion to their
populations with at least one of each, rather than by strict priority, which
would mean a check with 50.000 misses and three false positives never showed
one. Confirmations fill what remains, and a match is kept or dropped as a pair
because half a match is not reviewable evidence.
limit_evidence and evidence_role_counts are removed: plan_evidence supersedes
them, and helpers kept alive only by their own tests read like a contract.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The temporal timeline summarises every snapshot in a series. Each summary now
also counts how many features the selection edge cuts, but a timeline point
renders values only, so that was one database round trip per snapshot for a
disclosure nobody sees. Make it opt-out and have the timeline opt out.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The vector selection export caps its features and recorded truncated on the
export record. The file said nothing: an operator downloads
mol-selection.geojson, opens 250 buildings in QGIS where the workbench said
1.400, and has nothing to tell them the difference. For a product whose promise
is that an export is a reproducible result, a file that looks complete and is
not is the sharpest possible violation of it.
Every exported FeatureCollection now carries a geointel_provenance foreign
member — RFC 7946 requires parsers to ignore unknown members, so QGIS and
ogr2ogr are unaffected, and the detection export already used the same
convention for its trust classification. It names the source, the dataset and
its edition, the selection, and whether the file is complete; completeness is
derived from the counts as well as the flag, so a caller that forgets the flag
cannot produce a file claiming to hold everything.
Applied uniformly to the selection, dataset, detection and segmentation
exports, which previously disclosed three different amounts of nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change detection was the one analysis that ignored the selection entirely. It
compared two datasets in full, loaded every feature of both into Python with no
spatial predicate, and — with include_unchanged defaulting to true — returned a
FeatureCollection holding both datasets. For a regional building layer that is
the wrong answer to "what changed here" and a response no browser should be
asked to hold.
It now accepts bbox and area_id, resolved the way every other analysis resolves
them, and loads through an indexed ST_Intersects predicate.
Features are deliberately not clipped to the selection. A change class
describes a whole object: comparing a clipped earlier footprint against an
unclipped later one would report the selection edge itself as a change. Objects
the edge crosses are compared in full and counted in a warning.
The returned geometry is capped by preview_limit, spending that budget on
modified, added and removed before unchanged, while every count still describes
the whole selection.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
19 tests were failing on main. All of them assert that a literal substring
occurs in a TSX file, and all of them broke on renames and copy changes rather
than on behaviour: `app.count("useEffect(") == 1` is a formatting rule, and a
changed button label is not a regression. 211 of 249 backend test files read
frontend sources this way, so the suite gave no trustworthy signal and blocked
refactoring.
tests/frontend_contract.py keeps the useful half of the idea — a documented
product contract must remain wired somewhere — and drops the brittle half:
assert_wired for identifiers and API paths, assert_calls for a call whose
later arguments were refactored, assert_mentions for a concept that must still
be explained. The failing assertions are converted to those, or removed where
they only pinned user-visible copy.
test_frontend_contract_test_style.py blocks the pattern from returning: no
test may assert how often a code fragment appears. Counting list values or
network calls is unaffected.
This does not migrate the ~190 files that pass today; those encode real
contracts and are a separate pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
disclosure from data
Change detection had only added/removed/unchanged, so a building extended by
an annexe dropped below the IoU threshold and was reported twice: once as
removed and once as added. That hides exactly the category a change-detection
product exists to show and inflates both counts. A "modified" class now covers
the band between the modified floor and the unchanged threshold.
Matching also ran as a full cross product with no spatial index, unlike the QA
matcher beside it: two municipal building layers meant hundreds of millions of
geometry intersections. It uses an STRtree and considers larger footprints
first, so a big footprint is not left over after a small neighbour claimed its
counterpart.
The assistant guaranteed honesty about estimated values by rewriting the
model's sentences with regular expressions, which only fires when it
recognises the phrasing the model happened to produce. estimate_disclosures
derives the same statement from the metric metadata, so it holds regardless of
how the answer was worded. The prose substitution stays as a second layer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four ways a selection produced a confident number about a different area than
the operator drew:
Flood hazard divided the inundated cells by every cell in the drawn rectangle,
including cells the VMM raster does not model at all. A selection reaching
past the modelled extent therefore reported a diluted risk share, turning
missing data into an implied absence of risk. Terrain, bathymetry and thematic
raster already divided by valid cells; flood hazard was the outlier. It now
reports the three populations separately, states model coverage next to the
drawn area, and returns a null fraction rather than a zero when nothing was
modelled.
geometry_mask selects a cell when its centre falls inside the geometry, so a
rectangle smaller than one cell — or one landing between four centres —
selected nothing and the analysis returned zeros indistinguishable on screen
from "we looked and there is nothing here". On a 100 m population raster a
40 m rectangle over a city block reported no inhabitants. Selection now falls
back to the touched cells and says that it did, since the answer then covers
more ground than was requested. rasterio.mask applies the same centre rule
when cropping, so that call is widened too; the cells that count are still
decided by the centre rule wherever it selects anything.
The object count treated any feature touching the selection as whole, while
intersection_area clipped it — two headline numbers on one panel describing
different populations. The count stays whole-feature, which is what "objecten"
means to an operator, but now reports how many the edge cuts and is marked an
estimate when it does. The area_weighted_sum branch reuses that same count
instead of issuing its own near-identical query.
Partitioned selection de-duplicated the count on source_feature_id but
returned the raw rows, so a building on a municipal boundary was counted once
and drawn twice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tile handling produced results that were wrong before any model quality
question arose:
- orthophoto tiles reached the model through PIL convert("RGB"), which
truncates the high byte of a 16-bit product and treats a 4-band RGB+NIR
tile's infrared channel as colour. Tiles are now read with rasterio, the
visible bands are chosen explicitly, and values are percentile-stretched
across all three bands together so hue is preserved;
- an object wider than the tile overlap was truncated by both tiles into two
boxes that barely intersect, so IoU suppression kept both: two false
positives and one missed footprint per seam building. Suppression now also
compares overlap against the smaller box, and boxes cut by an interior tile
edge are dropped in favour of the neighbouring tile's complete view;
- georeferencing fell back to an assumed EPSG:4326 when a manifest carried no
CRS, producing geometry that renders plausibly in the wrong place. QA
already refused such a tile; inference now fails closed too.
Segmentation QA scored candidates against every reference feature in the
dataset, so every building outside the inferred tiles counted as a false
negative. It now applies the same persisted tile coverage that detection QA
has always used, including the indexed ST_Intersects prefilter.
Duplicate suppression uses an STRtree instead of the O(n^2) scan, tiles are
predicted in batches of YOLO_BATCH_SIZE (a setting that existed but was never
read), and detection/segmentation runs can be queued through /run-async for a
polling background worker rather than holding an HTTP worker thread for
minutes of GPU work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The greedy IoU matcher gave a reference to whichever candidate was offered
first. Row order decided that, and every detection in a run shares one
transaction timestamp, so ordering by created_at left the assignment
undefined: the same QA run over the same data produced different mean IoU,
and the geometry shown to a reviewer as a false positive could be the better
of two detections. Candidates are now ranked by confidence with feature
identity as tiebreaker, which is also the COCO/PASCAL rule.
A single precision/recall/F1 triple describes one operating point, so two
models cannot be compared from it: a conservatively calibrated model looks
worse at a low confidence cut and better at a high one without detecting
anything differently. DetectionMetricsService adds the full curve, average
precision and the threshold where F1 actually peaks.
Also:
- report the population the metrics were computed over, so
matches + false_positives equals candidate_feature_count even under an
area filter; raw dataset totals move to the _raw fields;
- state whether candidates are axis-aligned boxes or footprint polygons.
A box can never reach IoU 1 against a rotated building, so the strict
score has a ceiling that has nothing to do with detection quality.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
