The recommended detection profile showed operators precision 0.614, recall
0.606 and F1 0.607. Those three numbers appear nowhere in this repository
except the file that publishes them and a test that pinned them as literal
strings. The only recorded evaluation of that model at that operating point —
tile 512, overlap 64, threshold 0.15, the exact key its promotion report
recommended — reported 0.590, 0.577 and 0.582. The published figures were about
two and a half points more flattering than anything that was measured, on the
profile labelled "aanbevolen", and the test made sure nobody would correct
them. They now carry the measured values.
Worse in kind: the conservative profile reported "gemeten achtergrondfouten 0".
Its nine-sample hard-negative matrix at threshold 0.35 recorded 198 background
detections with 55 in the worst sample. The one number that tells an operator
whether a high-precision model invents buildings on empty terrain said zero
where the evidence said 55.
Those zeros are not simply wrong everywhere, which is why the fix is not just a
number. The other two profiles genuinely produced zero — against a strict
pure-empty gate of three samples, a different and much weaker test than the
nine-sample hard-negative matrix. Printing 0, 0 and 55 side by side invites a
comparison the evidence does not support, so each profile now states its gate,
its background sample count and the evaluation behind its figures, and the
panel shows them.
A test refuses any published figure that does not appear in the evidence
record, with a negative control so it cannot pass by matching nothing. Pinning
the numbers as literal strings is what let an unsourced precision survive; that
assertion is gone.
Also ignoring .codex-artifacts/ — ~300 MB of the rejected SAM2 and
edge-alignment trials plus a deploy bundle. Kept on disk, out of the repository.
No credentials in it; the two token scripts generate from settings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An ArcGIS layer without supportsPagination accepts resultOffset and ignores it,
answering every page with the first one. The VHA profile reader advanced its
offset by the page length and stopped at the announced count, so for a count
that is a multiple of the page size it collected N copies of page one — and its
completeness check, len(features) == candidate_count, passed. Four announced
records became four stored records, two of them duplicates, filed under an
official provenance. That is the substitution bounded acquisition exists to
prevent, arriving through the front door.
The reader now refuses a record it already collected. It fails rather than
silently dropping the duplicate: a provider that cannot page is a provider
whose count proves nothing, so a smaller-but-clean result would still be
unverifiable. Its watercourse-name loop was worse — a bare `while True` that
ended only when the provider stopped setting exceededTransferLimit, with names
deduplicated by code so a stuck provider produced no visible change while the
requests continued. It now refuses a repeated page body, and both loops have
the page budget the sibling readers already had.
Those siblings turned out to be fine. GRB and official vector already refuse a
repeated page URL, bound the page count, and deduplicate on feature identity —
but none of it had a test, so none of it was known to work. Exercised now,
including the case where distinct URLs defeat the loop check and the budget is
the only backstop. A duplicate across two genuinely different pages is kept
once rather than failing, because a cursor over a changing table produces that
legitimately.
Also: _bash_path fell back to the raw path whenever wslpath failed, except on
timeout, which propagated and reddened the suite when starting WSL took more
than ten seconds under load.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two problems of the same shape: information about *why* something failed being
replaced by something vaguer.
get_dataset_geojson wrapped the JSON parse, the metadata read, the CRS
resolution and the canonicalisation in one try and reported all of it as
"Stored dataset is not valid JSON" with a 500. An operator whose dataset had an
unusable CRS was sent to inspect a file that parses perfectly well, and the
canonicaliser's own AppError — with its code and its status — never reached
them. Only the parse is now inside that handler; everything after it keeps the
error it raised, and a genuine bug becomes a distinct 500 rather than a
mislabelled client error. A guard finds the same shape elsewhere: catching
Exception around a call into another component and relabelling what it
reported. Wrapping one's own private helper stays legitimate and the guard
says so.
The redirect policy was split without anyone saying so. Two acquisition
services rejected every redirect through a hand-rolled opener, while eight
allowed a same-origin one through the shared guard — and only the latter
checked where the response came from. Both live in the guard now, and the
strict path uses the rejecting handler rather than the guard's after-the-fact
check: objecting to response.url means urllib already opened the connection and
read the body, which for a metadata endpoint is the whole attack. That was a
weakening I introduced in this same commit's first draft.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
scripts/evaluate_belgium_building_candidate.py freezes its post-processing
before the protected test — NMS IoU and a containment threshold selected during
calibration, defaulting to 1.0. The runtime applied a hardcoded 0.85, so a
promoted candidate was served under suppression its evaluation never measured
and dropped detections the gate had counted. Neither report showed the
difference. That constant was mine, added without noticing the evaluation
pipeline already had a tuned value for the same concept.
Containment is now configuration, recorded on every run beside the duplicate
IoU threshold, so an operator can serve a candidate at the value it was gated
at and afterwards see which value produced a given score.
Two runs that post-processed differently produced different candidate sets from
the same model output, so the comparison endpoint refuses to rank them. Runs
recorded before those values were persisted carry none, and absence is not
treated as a difference.
The remaining gap is deliberate and documented rather than closed: the gate
scores the model on its protected test set, the workbench scores the whole
pipeline including coverage clipping and the tile-edge filter. A promoted
candidate will not reproduce its gate F1 exactly, and pretending otherwise
would be the worse answer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Detection QA reports a precision/recall curve, average precision and a
calibration sweep; segmentation QA reported a single operating point. Both rank
their outputs by confidence, so the same view applies, and the asymmetry meant
the two panels answered different questions about comparable runs — an
inconsistency introduced when detection gained the curve.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The review vocabulary already separates a model error from a reference gap,
because the product's position is that official footprints are not
automatically perfect ground truth. Those verdicts were only counted. An
operator who inspected forty false positives and established that twelve are
buildings the reference simply lacks still saw a precision counting all forty
against the model — a number they had personally disproved, on the panel where
they disproved it.
Applying the verdicts gives an adjudicated score reported next to the raw one,
so nothing is quietly improved. Not being able to judge is not evidence in the
model's favour, so uncertain and obscured verdicts keep counting, as does a
decision from a later release that this runtime does not recognise.
Because part of the evidence is usually still unreviewed, the honest form is an
interval rather than a single corrected number: pessimistic assumes every
unreviewed finding is a model error, optimistic assumes none is, and the
headline equals the pessimistic reading so a partly reviewed check never
presents as a settled one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The workbench ranks model variants by a stored F1, each measured at that
variant's own confidence threshold. A conservatively calibrated detector then
looks worse than a liberal one without detecting anything differently: the
number says as much about the threshold as about the model. POST
/detection/runs/compare ranks on average precision instead, which describes the
whole ranking a model produced, and keeps each run's own-threshold F1 visible
next to it so the difference between the two readings is auditable.
Comparability comes before the ranking. Runs over different source rasters,
scored against different references, without a proven inference footprint, or
covering a different evaluated population are not alternatives to one another,
and no metric makes them so. The report names which of those applies and still
returns the numbers — they are simply not a ranking.
Each run is scored through the same QA path the workbench uses, so a comparison
and the persisted quality checks cannot drift apart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Threshold calibration ran the model over every tile once per threshold — three
GPU passes to compare 0.50, 0.25 and 0.15 on a hundred-tile raster. The answer
is already in a single run at the lowest value: detections above a higher cut
are a subset of it, and duplicate suppression walks candidates in descending
confidence, so a lower-confidence box can never displace a higher-confidence
one. The kept set above any cut is identical whichever threshold the run used,
which is what makes one pass sufficient rather than merely cheaper.
QA now takes calibration_thresholds and reads each operating point off the same
precision/recall walk it already performs, marking the F1-optimal cut. The lab
runs inference once and fills its table from the sweep.
The contract test asserted the per-threshold loop by name, pinning the waste it
was meant to describe. It now states what calibration owes an operator: a row
per requested threshold, from one run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tile_manifest_path arrives in the detection and segmentation request and was
read straight off disk, and a manifest entry may name an absolute tile path.
That makes an API field an unbounded reference to the host filesystem, and it
contradicts the rule the persistence model rests on: only a governed,
runtime-produced artifact may be consumed, and a file outside the storage root
is not one.
Both the manifest and every tile it names now resolve under STORAGE_ROOT.
Resolution happens before the comparison, so ".." cannot climb out and a
sibling that merely shares a name prefix does not pass.
GEOINTEL_ALLOW_EXTERNAL_ARTIFACT_PATHS opts out for provisioning workflows that
stage tiles before ingest.
The check honours the Settings the caller is operating under rather than the
process-wide ones, because every analysis path already threads its own.
The affected tests write manifests into tmp_path, so they now declare tmp_path
as the storage root — which is what a deployment does, and makes the fixtures
more honest than they were.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two defects of the same kind: work that is supposed to be bounded is not.
The analysis worker selected queued jobs and then set them to running in a
second statement. A restarted process overlapping the previous one, or a second
replica, could both select the same row and both start tiled GPU inference on
it — duplicate analysis runs and double the GPU load. The AOI worker beside it
already claims with FOR UPDATE SKIP LOCKED; this uses a conditional update,
which is the same guarantee in one statement. run_once now reports jobs it
actually claimed rather than jobs it looked at.
urlopen follows redirects, so although every acquisition URL is built from
settings and cannot be steered by a request payload, a misconfigured or
compromised upstream could send the runtime to the loopback interface, to
another container on the compose network, or to a cloud metadata endpoint — and
the bytes would then be persisted under an official provenance. That is exactly
the substitution the product forbids. All eight fetch sites now open through a
guard that refuses private, loopback and link-local destinations (resolving the
host first, so a DNS name cannot hide one) and refuses a redirect that leaves
the configured origin or downgrades from HTTPS.
The guard is proven by calling the services' own fetch paths, not by grepping
for the call: every existing acquisition test injects an opener, which bypasses
it by design.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
/detection/runs/{id}/detections and its GeoJSON sibling returned every
persisted detection, as did the segmentation equivalents. A regional run holds
tens of thousands, and these are the endpoints the results table and the map
overlay call after every run.
They now take limit and offset, default to 2.000, and report total, limit,
offset and truncated so the complete population stays visible while what is
transferred does not. The GeoJSON responses carry the same window in a
geointel_result_window foreign member.
Rows are ordered by confidence, so a capped overlay draws the strongest
detections rather than an arbitrary slice, and the lab says how many of how
many are being shown rather than silently presenting a page as the whole run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
evidence_geojson emitted one feature per false positive, one per false negative
and two per match, with no limit. A regional check of 40k detections against 45k
reference footprints produced well over a hundred thousand features in a single
response, plus one warning string per unresolvable identifier. The endpoint the
entire review workflow depends on therefore failed exactly where review matters
most.
What to draw is now decided before any geometry is fetched, so the query work is
proportional to the result rather than to the size of the check — previously
130k geometries were resolved through an IN clause holding every identifier in
the check, to then discard most of them.
The budget is split between misses and false positives in proportion to their
populations with at least one of each, rather than by strict priority, which
would mean a check with 50.000 misses and three false positives never showed
one. Confirmations fill what remains, and a match is kept or dropped as a pair
because half a match is not reviewable evidence.
limit_evidence and evidence_role_counts are removed: plan_evidence supersedes
them, and helpers kept alive only by their own tests read like a contract.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The README's architecture diagram and stack list promised Redis + RQ. There is
no such thing in this codebase: redis_url is a settings field nothing reads,
rq is not a dependency, and the only worker is an asyncio poller over a
Postgres job table. Describe what exists.
API_CONTRACTS and KNOWN_LIMITATIONS gain the rules a reader needs to interpret
a result: deterministic confidence-ranked matching, the precision/recall curve,
the three flood-hazard cell populations, the sub-cell selection fallback, the
whole-feature object count next to clipped area metrics, and the required
manifest CRS. The new limitations are stated as limits rather than buried:
raster analysis has no sub-cell precision, and the object count is not
fractional.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Jens
Jens