publish the accuracy that was measured
The recommended detection profile showed operators precision 0.614, recall 0.606 and F1 0.607. Those three numbers appear nowhere in this repository except the file that publishes them and a test that pinned them as literal strings. The only recorded evaluation of that model at that operating point — tile 512, overlap 64, threshold 0.15, the exact key its promotion report recommended — reported 0.590, 0.577 and 0.582. The published figures were about two and a half points more flattering than anything that was measured, on the profile labelled "aanbevolen", and the test made sure nobody would correct them. They now carry the measured values. Worse in kind: the conservative profile reported "gemeten achtergrondfouten 0". Its nine-sample hard-negative matrix at threshold 0.35 recorded 198 background detections with 55 in the worst sample. The one number that tells an operator whether a high-precision model invents buildings on empty terrain said zero where the evidence said 55. Those zeros are not simply wrong everywhere, which is why the fix is not just a number. The other two profiles genuinely produced zero — against a strict pure-empty gate of three samples, a different and much weaker test than the nine-sample hard-negative matrix. Printing 0, 0 and 55 side by side invites a comparison the evidence does not support, so each profile now states its gate, its background sample count and the evaluation behind its figures, and the panel shows them. A test refuses any published figure that does not appear in the evidence record, with a negative control so it cannot pass by matching nothing. Pinning the numbers as literal strings is what let an unsourced precision survive; that assertion is gone. Also ignoring .codex-artifacts/ — ~300 MB of the rejected SAM2 and edge-alignment trials plus a deploy bundle. Kept on disk, out of the repository. No credentials in it; the two token scripts generate from settings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -165,8 +165,14 @@ export function DetectionModelManagement({
|
||||
<span>historische herkenningsgraad {profile.recall.toFixed(3)}</span>
|
||||
<span>historische F1 {profile.f1.toFixed(3)}</span>
|
||||
<span>positieve controles {profile.positiveSampleCount}</span>
|
||||
<span>gemeten achtergrondfouten {profile.maxBackgroundDetections}</span>
|
||||
<span>
|
||||
ergste achtergrondmonster {profile.maxBackgroundDetections} van{' '}
|
||||
{profile.backgroundSampleCount}
|
||||
</span>
|
||||
</div>
|
||||
<p className="field-guidance">
|
||||
Achtergrondtoets: {profile.backgroundGate}. Meting: {profile.evidenceReference}.
|
||||
</p>
|
||||
<p className="field-guidance">{profile.limitationMessage}</p>
|
||||
<button
|
||||
className="secondary-action"
|
||||
|
||||
@@ -9,11 +9,18 @@ export interface DetectionOperatorProfile {
|
||||
f1: number
|
||||
positiveSampleCount: number
|
||||
maxBackgroundDetections: number
|
||||
backgroundSampleCount: number
|
||||
backgroundGate: string
|
||||
evidenceReference: string
|
||||
validationScope: string
|
||||
description: string
|
||||
limitationMessage: string
|
||||
}
|
||||
|
||||
// Every number below is a value recorded in docs/CODEX_EXECUTION_LOG.md for
|
||||
// this exact model, tile size, overlap and threshold. Publishing a figure the
|
||||
// evidence record does not contain is the one thing an operator cannot check,
|
||||
// so tests/test_detection_profile_metrics_are_traceable.py refuses it.
|
||||
export const DETECTION_OPERATOR_PROFILES: DetectionOperatorProfile[] = [
|
||||
{
|
||||
id: 'small-building-balanced-review',
|
||||
@@ -21,16 +28,19 @@ export const DETECTION_OPERATOR_PROFILES: DetectionOperatorProfile[] = [
|
||||
modelAssetId: 'geointel-building-yolov8s-smallbld-minpx3-img640-ft30-pt',
|
||||
confidenceThreshold: 0.15,
|
||||
independentTestProven: false,
|
||||
precision: 0.6140895327792112,
|
||||
recall: 0.6062221049337548,
|
||||
f1: 0.6068607646002744,
|
||||
precision: 0.5898197518,
|
||||
recall: 0.5769921004,
|
||||
f1: 0.5824578632,
|
||||
positiveSampleCount: 7,
|
||||
maxBackgroundDetections: 0,
|
||||
backgroundSampleCount: 3,
|
||||
backgroundGate: 'strikte lege-poort (Postel, Lommel, Arendonk)',
|
||||
evidenceReference: 'promotierapport smallbld-minpx3-img640-ft30 | tegel 512 | overlap 64 | drempel 0.15',
|
||||
validationScope: 'historische operatorcontrole in Mol en de Kempen; ruimtelijke onafhankelijkheid niet bewezen',
|
||||
description:
|
||||
'Historisch controleprofiel met een evenwicht tussen gevonden en gemiste kleine gebouwen. Gebruik dit als startpunt voor lokale QA, niet als vrijgavebewijs.',
|
||||
limitationMessage:
|
||||
'Slechts drie pure-achtergrondbeelden en geen onafhankelijke hold-out ondersteunen deze historische meting. Postel bleef het moeilijkste gebied; behandel elke detectie als controlekandidaat en niet als grondwaarheid.',
|
||||
'Slechts drie pure-achtergrondbeelden en geen onafhankelijke hold-out ondersteunen deze historische meting. Dit profiel ruilt precisie in voor dekking: het vindt meer kleine gebouwen dan het vorige, maar levert een hogere controlelast op. Postel bleef het moeilijkste gebied; behandel elke detectie als controlekandidaat en niet als grondwaarheid.',
|
||||
},
|
||||
{
|
||||
id: 'expanded-balanced-review',
|
||||
@@ -38,11 +48,14 @@ export const DETECTION_OPERATOR_PROFILES: DetectionOperatorProfile[] = [
|
||||
modelAssetId: 'geointel-building-yolov8s-aoi1024expandedminpx4vis035e50-pt',
|
||||
confidenceThreshold: 0.15,
|
||||
independentTestProven: false,
|
||||
precision: 0.6470590036169351,
|
||||
recall: 0.4699913836847832,
|
||||
f1: 0.5432865390636915,
|
||||
precision: 0.6470590036,
|
||||
recall: 0.4699913837,
|
||||
f1: 0.5432865391,
|
||||
positiveSampleCount: 7,
|
||||
maxBackgroundDetections: 0,
|
||||
backgroundSampleCount: 3,
|
||||
backgroundGate: 'strikte lege-poort (Postel, Lommel, Arendonk)',
|
||||
evidenceReference: 'zeven-AOI positieve matrix aoi1024expandedminpx4vis035e50 | tegel 512 | overlap 64 | drempel 0.15',
|
||||
validationScope: 'historische operatorcontrole in Mol en de Kempen; ruimtelijke onafhankelijkheid niet bewezen',
|
||||
description: 'Historisch profiel voor lokale controles waarbij minder foutieve vondsten belangrijker zijn dan maximale dekking.',
|
||||
limitationMessage:
|
||||
@@ -58,10 +71,13 @@ export const DETECTION_OPERATOR_PROFILES: DetectionOperatorProfile[] = [
|
||||
recall: 0.202135,
|
||||
f1: 0.32086574003576274,
|
||||
positiveSampleCount: 7,
|
||||
maxBackgroundDetections: 0,
|
||||
maxBackgroundDetections: 55,
|
||||
backgroundSampleCount: 9,
|
||||
backgroundGate: 'hard-negative matrix (negen monsters, 198 detecties in totaal)',
|
||||
evidenceReference: 'conservatieve zeven-AOI matrix + negen-monster hard-negative matrix aoi1024bg512r3e50 | drempel 0.35',
|
||||
validationScope: 'historische operatorcontrole in Mol en de Kempen; ruimtelijke onafhankelijkheid niet bewezen',
|
||||
description: 'Historisch profiel voor lokale controles waarbij zo weinig mogelijk foutieve vondsten zwaarder wegen dan volledige dekking.',
|
||||
limitationMessage:
|
||||
'De historische precisie gaat samen met zeer lage herkenningsgraad. Resultaten blijven controlebewijs en geen automatische waarheid of releasebewijs.',
|
||||
'De historische precisie gaat samen met zeer lage herkenningsgraad. De hoge precisie geldt binnen bebouwde AOI-gebieden: op de negen hard-negative monsters produceerde dit profiel nog 198 achtergronddetecties, met 55 in het zwaarste monster. Resultaten blijven controlebewijs en geen automatische waarheid of releasebewijs.',
|
||||
},
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user