# Credential threat tests

## The three credential planes

ModelForge issues three kinds of secret and they never substitute for one another.

```text
operator API key    X-ModelForge-Admin-Token   admin routes, /metrics, recovery, lifecycle
node credential     Authorization: Bearer      agent publication only
capability secret   Authorization: Bearer      one named capability, through the gateway
```

A capability secret on an operator route returns 401. An operator route with no key configured
returns 503 rather than falling open. The invariant `capability_clients_are_not_operators` asserts
no client ever carries an operator or node scope in its capability list.

## Storage

Only hashes are stored. `ServiceCredential` has no column that could hold a plaintext secret, and
the stored hash is verified to be the SHA-256 of the issued value. A prefix is retained for operator
identification and is bounded so it cannot reconstruct the secret.

Live checks confirmed no operator key, backup encryption key, node credential, `PGPASSWORD` or
database password appears in the API logs, in any audit event, in the OpenAPI document, or in any
recovery API response.

## Revocation

Revocation is permanent and immediate. A revoked credential is refused on every subsequent attempt,
including after the node re-enrols with a new one — enrolment revokes rather than deletes, so the
revocation survives in history. The invariant `revoked_credentials_stay_revoked` asserts no revoked
credential records a use later than its revocation.

## Rotation

Rotation leaves exactly one usable secret. Two simultaneously valid secrets is an unbounded window
rather than a graceful cutover, so the old one is revoked as the new one is issued and the partial
unique index enforces it.

## Races

Revocation racing authentication was tested with sixteen concurrent threads. Whichever order the
race resolves in the revoked credential is never accepted, and re-checking afterwards still refuses
it.

A single-use enrolment token was stormed with sixty concurrent attempts. Exactly one succeeded,
fifty-nine were refused, and one node identity with one active credential exists afterwards. This
covers a real M15 defect where the `used_at` check was read-then-write and two agent threads could
burn one token on two identities for the same hardware.

## Burst behaviour

Sixty invalid credentials produced sixty identical 401s with no amplification, and the valid
credential still worked afterwards. Refusal is cheap and constant: there is no lockout to trigger
and no differential response to measure. Mean refusal latency was 12.75 ms with a 58.09 ms maximum.

## Node identity

A recovered node keeps its persisted identity, so losing a credential does not lose its
accelerators, storage roots or history. A node whose identity file is also gone enrols as a new
node — ModelForge genuinely cannot distinguish it from different hardware — and the operator
disables the superseded row rather than letting two nodes claim the same machine.

## Diagnostics

A failed publication now names the status, method and path, never the response body or the
credential. Before M16 the agent logged only `control-plane publication failed; retrying`, which
cannot distinguish a revoked credential from a DNS failure — and had been doing so for two days
while pointed at a placeholder URL.