geointel/docs/QUEUE_ARCHITECTURE.md

Queue Architecture

Decision

Use Redis + RQ for V1 background jobs.

Why

• Simple to run locally.
• Easy to understand.
• Good enough for raster processing and AI inference jobs.
• Avoids Celery complexity in the first implementation.

Queue names

• default for lightweight jobs.
• processing for raster/vector processing.
• ai for detection and segmentation.
• exports for GeoJSON/report exports.

Job lifecycle

1. API validates request.
2. API creates analysis_run with status queued.
3. API enqueues job with analysis_run_id.
4. Worker sets status running.
5. Worker writes artifacts and metrics.
6. Worker sets status completed or failed.
7. Frontend polls analysis run endpoint.

Failure handling

Failures must store:

• error code,
• error message,
• stack trace in internal logs only,
• user-safe explanation.

No silent failures

A failed job must be visible in the UI and queryable through the API.