claim analysis jobs atomically and keep acquisition on its official host
Two defects of the same kind: work that is supposed to be bounded is not. The analysis worker selected queued jobs and then set them to running in a second statement. A restarted process overlapping the previous one, or a second replica, could both select the same row and both start tiled GPU inference on it — duplicate analysis runs and double the GPU load. The AOI worker beside it already claims with FOR UPDATE SKIP LOCKED; this uses a conditional update, which is the same guarantee in one statement. run_once now reports jobs it actually claimed rather than jobs it looked at. urlopen follows redirects, so although every acquisition URL is built from settings and cannot be steered by a request payload, a misconfigured or compromised upstream could send the runtime to the loopback interface, to another container on the compose network, or to a cloud metadata endpoint — and the bytes would then be persisted under an official provenance. That is exactly the substitution the product forbids. All eight fetch sites now open through a guard that refuses private, loopback and link-local destinations (resolving the host first, so a DNS name cannot hide one) and refuses a redirect that leaves the configured origin or downgrades from HTTPS. The guard is proven by calling the services' own fetch paths, not by grepping for the call: every existing acquisition test injects an opener, which bypasses it by design. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -65,16 +65,26 @@ class AnalysisJobWorker:
|
||||
return SegmentationService.run_segmentation(**common)
|
||||
|
||||
@staticmethod
|
||||
def _claim(db, job: Job) -> None:
|
||||
"""Take the job out of the queue before doing any work on it.
|
||||
def claim(db, job: Job) -> bool:
|
||||
"""Take the job out of the queue, atomically. Returns whether we won.
|
||||
|
||||
Without this the next poll would pick the same row up again while the
|
||||
first execution is still running on the GPU.
|
||||
Selecting and then updating in a second statement lets two workers —
|
||||
a restarted process overlapping the previous one, or a second replica —
|
||||
both start tiled GPU inference on the same row. The conditional update
|
||||
makes exactly one caller see a row count of 1; the AOI worker beside
|
||||
this one already claims with FOR UPDATE SKIP LOCKED for the same reason.
|
||||
"""
|
||||
|
||||
job.status = "running"
|
||||
db.add(job)
|
||||
claimed = (
|
||||
db.query(Job)
|
||||
.filter(Job.id == job.id, Job.status == "queued")
|
||||
.update({Job.status: "running"}, synchronize_session=False)
|
||||
)
|
||||
db.commit()
|
||||
if not claimed:
|
||||
return False
|
||||
job.status = "running"
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def _finalize(db, job: Job, result: Any) -> None:
|
||||
@@ -132,9 +142,13 @@ class AnalysisJobWorker:
|
||||
)
|
||||
if job.job_type in AnalysisJobWorker.HANDLED_JOB_TYPES and job.status == "queued"
|
||||
]
|
||||
claimed_count = 0
|
||||
for job in rows:
|
||||
if not AnalysisJobWorker.claim(session, job):
|
||||
# Another worker took it between the select and the claim.
|
||||
continue
|
||||
claimed_count += 1
|
||||
try:
|
||||
AnalysisJobWorker._claim(session, job)
|
||||
result = AnalysisJobWorker._dispatch(session, job)
|
||||
AnalysisJobWorker._finalize(session, job, result)
|
||||
except Exception as exc:
|
||||
@@ -142,7 +156,7 @@ class AnalysisJobWorker:
|
||||
message = getattr(exc, "message", None) or str(exc) or "Unexpected analysis job failure"
|
||||
AnalysisJobWorker._mark_failed(session, job, code=str(code), message=str(message))
|
||||
logger.exception("Analysis job failed job_id=%s job_type=%s", job.id, job.job_type)
|
||||
return len(rows)
|
||||
return claimed_count
|
||||
finally:
|
||||
if owns_session:
|
||||
session.close()
|
||||
|
||||
Reference in New Issue
Block a user