3.3 KiB
Credential threat tests
The three credential planes
ModelForge issues three kinds of secret and they never substitute for one another.
operator API key X-ModelForge-Admin-Token admin routes, /metrics, recovery, lifecycle
node credential Authorization: Bearer agent publication only
capability secret Authorization: Bearer one named capability, through the gateway
A capability secret on an operator route returns 401. An operator route with no key configured
returns 503 rather than falling open. The invariant capability_clients_are_not_operators asserts
no client ever carries an operator or node scope in its capability list.
Storage
Only hashes are stored. ServiceCredential has no column that could hold a plaintext secret, and
the stored hash is verified to be the SHA-256 of the issued value. A prefix is retained for operator
identification and is bounded so it cannot reconstruct the secret.
Live checks confirmed no operator key, backup encryption key, node credential, PGPASSWORD or
database password appears in the API logs, in any audit event, in the OpenAPI document, or in any
recovery API response.
Revocation
Revocation is permanent and immediate. A revoked credential is refused on every subsequent attempt,
including after the node re-enrols with a new one — enrolment revokes rather than deletes, so the
revocation survives in history. The invariant revoked_credentials_stay_revoked asserts no revoked
credential records a use later than its revocation.
Rotation
Rotation leaves exactly one usable secret. Two simultaneously valid secrets is an unbounded window rather than a graceful cutover, so the old one is revoked as the new one is issued and the partial unique index enforces it.
Races
Revocation racing authentication was tested with sixteen concurrent threads. Whichever order the race resolves in the revoked credential is never accepted, and re-checking afterwards still refuses it.
A single-use enrolment token was stormed with sixty concurrent attempts. Exactly one succeeded,
fifty-nine were refused, and one node identity with one active credential exists afterwards. This
covers a real M15 defect where the used_at check was read-then-write and two agent threads could
burn one token on two identities for the same hardware.
Burst behaviour
Sixty invalid credentials produced sixty identical 401s with no amplification, and the valid credential still worked afterwards. Refusal is cheap and constant: there is no lockout to trigger and no differential response to measure. Mean refusal latency was 12.75 ms with a 58.09 ms maximum.
Node identity
A recovered node keeps its persisted identity, so losing a credential does not lose its accelerators, storage roots or history. A node whose identity file is also gone enrols as a new node — ModelForge genuinely cannot distinguish it from different hardware — and the operator disables the superseded row rather than letting two nodes claim the same machine.
Diagnostics
A failed publication now names the status, method and path, never the response body or the
credential. Before M16 the agent logged only control-plane publication failed; retrying, which
cannot distinguish a revoked credential from a DNS failure — and had been doing so for two days
while pointed at a placeholder URL.