70 lines
3.3 KiB
Markdown
70 lines
3.3 KiB
Markdown
# Credential threat tests
|
|
|
|
## The three credential planes
|
|
|
|
ModelForge issues three kinds of secret and they never substitute for one another.
|
|
|
|
```text
|
|
operator API key X-ModelForge-Admin-Token admin routes, /metrics, recovery, lifecycle
|
|
node credential Authorization: Bearer agent publication only
|
|
capability secret Authorization: Bearer one named capability, through the gateway
|
|
```
|
|
|
|
A capability secret on an operator route returns 401. An operator route with no key configured
|
|
returns 503 rather than falling open. The invariant `capability_clients_are_not_operators` asserts
|
|
no client ever carries an operator or node scope in its capability list.
|
|
|
|
## Storage
|
|
|
|
Only hashes are stored. `ServiceCredential` has no column that could hold a plaintext secret, and
|
|
the stored hash is verified to be the SHA-256 of the issued value. A prefix is retained for operator
|
|
identification and is bounded so it cannot reconstruct the secret.
|
|
|
|
Live checks confirmed no operator key, backup encryption key, node credential, `PGPASSWORD` or
|
|
database password appears in the API logs, in any audit event, in the OpenAPI document, or in any
|
|
recovery API response.
|
|
|
|
## Revocation
|
|
|
|
Revocation is permanent and immediate. A revoked credential is refused on every subsequent attempt,
|
|
including after the node re-enrols with a new one — enrolment revokes rather than deletes, so the
|
|
revocation survives in history. The invariant `revoked_credentials_stay_revoked` asserts no revoked
|
|
credential records a use later than its revocation.
|
|
|
|
## Rotation
|
|
|
|
Rotation leaves exactly one usable secret. Two simultaneously valid secrets is an unbounded window
|
|
rather than a graceful cutover, so the old one is revoked as the new one is issued and the partial
|
|
unique index enforces it.
|
|
|
|
## Races
|
|
|
|
Revocation racing authentication was tested with sixteen concurrent threads. Whichever order the
|
|
race resolves in the revoked credential is never accepted, and re-checking afterwards still refuses
|
|
it.
|
|
|
|
A single-use enrolment token was stormed with sixty concurrent attempts. Exactly one succeeded,
|
|
fifty-nine were refused, and one node identity with one active credential exists afterwards. This
|
|
covers a real M15 defect where the `used_at` check was read-then-write and two agent threads could
|
|
burn one token on two identities for the same hardware.
|
|
|
|
## Burst behaviour
|
|
|
|
Sixty invalid credentials produced sixty identical 401s with no amplification, and the valid
|
|
credential still worked afterwards. Refusal is cheap and constant: there is no lockout to trigger
|
|
and no differential response to measure. Mean refusal latency was 12.75 ms with a 58.09 ms maximum.
|
|
|
|
## Node identity
|
|
|
|
A recovered node keeps its persisted identity, so losing a credential does not lose its
|
|
accelerators, storage roots or history. A node whose identity file is also gone enrols as a new
|
|
node — ModelForge genuinely cannot distinguish it from different hardware — and the operator
|
|
disables the superseded row rather than letting two nodes claim the same machine.
|
|
|
|
## Diagnostics
|
|
|
|
A failed publication now names the status, method and path, never the response body or the
|
|
credential. Before M16 the agent logged only `control-plane publication failed; retrying`, which
|
|
cannot distinguish a revoked credential from a DNS failure — and had been doing so for two days
|
|
while pointed at a placeholder URL.
|