31 lines
1.7 KiB
Markdown
31 lines
1.7 KiB
Markdown
# ADR 0022 — Bounded serving poll and opaque experiment routes
|
|
|
|
Status: accepted
|
|
|
|
## Context
|
|
|
|
M6 showed a warm Gateway p50 of 1144.46 ms although the model inference itself was much shorter.
|
|
The worker polled for jobs and completions with fixed one-second sleeps, and a project evaluator had
|
|
no capability-first way to address a non-production profile without learning a concrete deployment.
|
|
|
|
## Decision
|
|
|
|
- Job claims use server-side bounded long polling. The wait is capped at five seconds and the worker
|
|
renews it at one-second intervals, so shutdown and liveness remain responsive.
|
|
- Gateway timing is decomposed into validation, resolution, scheduling, payload persistence,
|
|
dispatch, completion transport, result retrieval and worker preprocess/inference/serialization.
|
|
Only bounded summaries are retained; request content is never trace data.
|
|
- A LAB_READY candidate may receive an opaque `capability-experiment` route. Authentication remains
|
|
scoped to `rag.embedding@1`; the caller neither supplies nor receives a model path.
|
|
- Experiment deployments are non-production, use an execution approval rather than a production
|
|
approval, have their own EmbeddingSpace and cannot change the stable route.
|
|
- On a single GPU, the scheduler unloads an inactive resident model before loading another candidate.
|
|
It refuses to switch while requests are active and continues to include unmanaged external VRAM
|
|
in the budget.
|
|
|
|
## Consequences
|
|
|
|
Warm serving no longer inherits a polling floor. Candidate evaluation remains capability-first and
|
|
auditable, while the one-GPU implementation avoids unsafe simultaneous loads. A candidate route is
|
|
not a canary and provides no production-cutover operation.
|