claim analysis jobs atomically and keep acquisition on its official host

Two defects of the same kind: work that is supposed to be bounded is not.

The analysis worker selected queued jobs and then set them to running in a
second statement. A restarted process overlapping the previous one, or a second
replica, could both select the same row and both start tiled GPU inference on
it — duplicate analysis runs and double the GPU load. The AOI worker beside it
already claims with FOR UPDATE SKIP LOCKED; this uses a conditional update,
which is the same guarantee in one statement. run_once now reports jobs it
actually claimed rather than jobs it looked at.

urlopen follows redirects, so although every acquisition URL is built from
settings and cannot be steered by a request payload, a misconfigured or
compromised upstream could send the runtime to the loopback interface, to
another container on the compose network, or to a cloud metadata endpoint — and
the bytes would then be persisted under an official provenance. That is exactly
the substitution the product forbids. All eight fetch sites now open through a
guard that refuses private, loopback and link-local destinations (resolving the
host first, so a DNS name cannot hide one) and refuses a redirect that leaves
the configured origin or downgrades from HTTPS.

The guard is proven by calling the services' own fetch paths, not by grepping
for the call: every existing acquisition test injects an opener, which bypasses
it by design.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Jens
2026-08-22 15:59:01 +02:00
co-authored by Claude Opus 5
parent 5b3839dc89
commit 16dedeb670
14 changed files with 493 additions and 16 deletions
+22 -8
View File
@@ -65,16 +65,26 @@ class AnalysisJobWorker:
return SegmentationService.run_segmentation(**common)
@staticmethod
def _claim(db, job: Job) -> None:
"""Take the job out of the queue before doing any work on it.
def claim(db, job: Job) -> bool:
"""Take the job out of the queue, atomically. Returns whether we won.
Without this the next poll would pick the same row up again while the
first execution is still running on the GPU.
Selecting and then updating in a second statement lets two workers —
a restarted process overlapping the previous one, or a second replica —
both start tiled GPU inference on the same row. The conditional update
makes exactly one caller see a row count of 1; the AOI worker beside
this one already claims with FOR UPDATE SKIP LOCKED for the same reason.
"""
job.status = "running"
db.add(job)
claimed = (
db.query(Job)
.filter(Job.id == job.id, Job.status == "queued")
.update({Job.status: "running"}, synchronize_session=False)
)
db.commit()
if not claimed:
return False
job.status = "running"
return True
@staticmethod
def _finalize(db, job: Job, result: Any) -> None:
@@ -132,9 +142,13 @@ class AnalysisJobWorker:
)
if job.job_type in AnalysisJobWorker.HANDLED_JOB_TYPES and job.status == "queued"
]
claimed_count = 0
for job in rows:
if not AnalysisJobWorker.claim(session, job):
# Another worker took it between the select and the claim.
continue
claimed_count += 1
try:
AnalysisJobWorker._claim(session, job)
result = AnalysisJobWorker._dispatch(session, job)
AnalysisJobWorker._finalize(session, job, result)
except Exception as exc:
@@ -142,7 +156,7 @@ class AnalysisJobWorker:
message = getattr(exc, "message", None) or str(exc) or "Unexpected analysis job failure"
AnalysisJobWorker._mark_failed(session, job, code=str(code), message=str(message))
logger.exception("Analysis job failed job_id=%s job_type=%s", job.id, job.job_type)
return len(rows)
return claimed_count
finally:
if owns_session:
session.close()