102 lines
4.7 KiB
Markdown
102 lines
4.7 KiB
Markdown
# Soak testing
|
|
|
|
## What a soak is for
|
|
|
|
A short run proves a path works. A soak proves it keeps working: that memory does not climb, that
|
|
leases are released, that queues drain, that retention keeps row growth bounded, and that latency
|
|
does not degrade as caches warm and evict.
|
|
|
|
```sh
|
|
python scripts/m16_soak.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --minutes 45 --secret <gateway-secret> --report soak.json
|
|
```
|
|
|
|
The database URL is mandatory and must use the non-owner runtime role. The harness contains no
|
|
embedded credential or owner-role fallback.
|
|
|
|
## Workload shape
|
|
|
|
Three gateway workers drive the real capability path — `rag.embedding@1` through the authenticated
|
|
gateway to GPU Node — mixing three phases:
|
|
|
|
```text
|
|
burst 3-6 requests back to back, 50-400 ms apart
|
|
steady single requests, 1.5-5 s apart
|
|
idle 8-25 s of nothing
|
|
```
|
|
|
|
The mix matters. A soak that only hammers never exercises keep-warm and unload, so it never
|
|
observes a cold start. A soak that only idles never exercises admission control, so it never
|
|
observes a capacity rejection. Batch sizes vary from one to four inputs so payload handling varies
|
|
too.
|
|
|
|
Two read workers poll registry, projects, hardware, health, operations, alerts, lifecycle,
|
|
migrations and recovery. One background worker drives the operational work the platform already
|
|
performs for itself — capacity collection, SLO evaluation, alert evaluation — at a bounded rate.
|
|
|
|
## What is measured
|
|
|
|
```text
|
|
per kind requests, succeeded, capacity rejected, internal errors, status distribution
|
|
latency p50, p95, p99, mean, max
|
|
gateway cold starts, warm invocations, queue p95, inference p50 and p95, failure codes
|
|
resources container memory, CPU, PIDs and restart counts, before and after
|
|
database row growth per table, connection count
|
|
runtime active leases, serving jobs in flight, residency allocations
|
|
GPU used, free and utilisation per accelerator, plus scheduler pressure
|
|
invariants the full set, at every progress interval
|
|
```
|
|
|
|
Capacity rejections are counted separately from internal errors. A 429 or a `QUEUE_FULL` is the
|
|
scheduler refusing unsafe work, which is the platform behaving correctly; counting it as a failure
|
|
would reward a system that over-admits.
|
|
|
|
## Stopping early
|
|
|
|
The soak aborts and reports partial evidence if an invariant is violated at any progress interval.
|
|
Partial evidence honestly labelled is worth more than a full run whose result is already known to
|
|
be invalid.
|
|
|
|
It should also be aborted manually on uncontrolled host pressure, degradation of an external
|
|
workload, repeated OOM, data corruption or any unsafe production mutation.
|
|
|
|
## Reading the result
|
|
|
|
```text
|
|
queue drained queued, running and active leases all zero at the end,
|
|
unless a resident is intentionally kept warm
|
|
memory a flat or sawtooth profile is healthy; monotonic growth is investigated
|
|
row growth bounded by retention and aggregation, not linear in request count
|
|
restart counts unchanged, unless a restart was deliberately injected
|
|
```
|
|
|
|
## Honesty rules
|
|
|
|
Report the duration actually achieved. A 45-minute soak is a 45-minute soak; it is not evidence
|
|
about a week.
|
|
|
|
If a chaos scenario or an infrastructure change overlapped the run, say so and say which numbers it
|
|
affected, rather than presenting the mixed result as a clean baseline. The M16 evidence includes one
|
|
soak that was discarded for exactly this reason after container hardening restarted its datastores
|
|
mid-run.
|
|
|
|
The disposable gateway client the soak uses is revoked afterwards, and the fixtures it created are
|
|
removed.
|
|
|
|
## Measuring latency honestly
|
|
|
|
The soak reports percentiles, so the harness must first prove that it is not the thing being
|
|
measured. Before any worker starts it times a plain TCP connect to the target and refuses to run if
|
|
the median exceeds `MAX_CONNECT_FLOOR_MS` (100 ms), printing the floor it measured and recording it
|
|
in the report as `client_connect_floor_ms`.
|
|
|
|
This exists because the first M16 soak reported a p50 near two seconds on every route including
|
|
`/api/v1/health/live`, with only a few milliseconds of spread. A constant with no tail is never queueing.
|
|
The cause was the harness addressing the API as `localhost`, which resolves to `::1` first on this
|
|
host; each request waited out the IPv6 connect timeout before falling back to IPv4, adding a fixed
|
|
~2,045 ms of client cost to every sample. Direct measurement, both returning 200: 2,043.3 ms median
|
|
to `http://localhost:8000/api/v1/health/live` against 4.7 ms to
|
|
`http://127.0.0.1:8000/api/v1/health/live`.
|
|
|
|
Address the platform by an address that resolves directly. Both M16 harnesses default to
|
|
`http://127.0.0.1:8000` for this reason, not to `localhost`.
|