Initial public ModelForge release

docs/security/ADVERSARIAL_API_TESTING.md

@@ -0,0 +1,74 @@
+# Adversarial API testing
+
+Every request in this suite goes to the application under test. Nothing here probes a host service,
+scans a network, or reaches anything ModelForge does not own.
+
+## What is sent
+
+```text
+malformed bodies      empty, truncated, non-JSON, bare scalars, NaN, Infinity, null bytes,
+                      200-deep nesting
+wrong types           integers, nulls, arrays, objects and booleans where a string is required
+oversized input       100 KB identifiers, 5 MB bodies, 5 KB enum values
+mass assignment       state, restore_eligible, verified_at, encryption_key, id
+hostile strings       traversal, encoded traversal, absolute and UNC paths, file:// and
+                      link-local URLs, SQL, template and JNDI expressions, shell metacharacters,
+                      script and image XSS payloads, bidi overrides, astral-plane and emoji runs
+identifiers           malformed UUIDs, SQL appended to a path segment, 500-character values
+query bounds          zero, negative, out-of-range, non-numeric and injected limits
+headers               CR/LF and null-byte injection into a correlation id
+credentials           none, empty, wrong scheme, wrong case, oversized, null-byte, injected
+```
+
+## What must happen
+
+A rejected value and a stored-but-inert value are both acceptable. A 5xx is not: it means input
+reached somewhere it should not have.
+
+```text
+400 / 422   malformed or wrongly typed, with a typed error envelope
+403         authenticated but out of scope
+404         well-formed identifier that does not exist
+413 / 422   oversized
+never 5xx   for any input in the matrix
+```
+
+Every error response carries `code`, `message` and `correlation_id`, and nothing else: no
+traceback, no SQL, no driver name, no DSN, no file path, no credential. This is asserted directly,
+including with the database stopped — a dependency failure is exactly when a framework is most
+likely to leak its internals.
+
+## Scope isolation
+
+The capability a credential names is matched exactly. Case changes, appended whitespace, a second
+capability in the same string and zero-width characters do not widen it. A contract version is part
+of the identity: a client scoped to `rag.embedding@1` cannot reach `rag.embedding@2`.
+
+A client bound to a project may only serve that project's binding, even when its capability list
+would otherwise allow more, and a deprecated binding stops serving immediately.
+
+## No enumeration oracle
+
+An unknown credential, a revoked one, a near-miss differing in the last character and an empty one
+all produce the same status and the same code. A caller cannot learn whether a guess was close.
+
+## Mass assignment
+
+Request models are `extra="forbid"`, so a caller cannot write a field the contract never offered.
+Attempting to set `state`, `restore_eligible`, `verified_at` or an `id` is a validation error rather
+than a silently ignored field — silently ignoring is how these become exploitable later.
+
+## Path handling
+
+Backup identities become directory names, so the identity pattern refuses traversal outright rather
+than relying on later sanitisation. Restore destinations are parsed and identifier-validated:
+non-PostgreSQL schemes, SQL appended to a database name, an empty database and a null byte are all
+refused. The M15 allowlisted-root resolution remains the second layer.
+
+## A defect this suite found
+
+Python's JSON parser accepts `NaN` and `Infinity`; its serialiser rejects them. The validation
+handler echoed the rejected value straight back into the error response, so a body containing one
+made the error response itself fail to serialise — turning a 422 into a server error, on input that
+needs no credential to send. Rejected values are now rendered safely and truncated, since rejected
+input is unbounded by definition.

docs/security/CREDENTIAL_THREAT_TESTS.md

@@ -0,0 +1,69 @@
+# Credential threat tests
+
+## The three credential planes
+
+ModelForge issues three kinds of secret and they never substitute for one another.
+
+```text
+operator API key    X-ModelForge-Admin-Token   admin routes, /metrics, recovery, lifecycle
+node credential     Authorization: Bearer      agent publication only
+capability secret   Authorization: Bearer      one named capability, through the gateway
+```
+
+A capability secret on an operator route returns 401. An operator route with no key configured
+returns 503 rather than falling open. The invariant `capability_clients_are_not_operators` asserts
+no client ever carries an operator or node scope in its capability list.
+
+## Storage
+
+Only hashes are stored. `ServiceCredential` has no column that could hold a plaintext secret, and
+the stored hash is verified to be the SHA-256 of the issued value. A prefix is retained for operator
+identification and is bounded so it cannot reconstruct the secret.
+
+Live checks confirmed no operator key, backup encryption key, node credential, `PGPASSWORD` or
+database password appears in the API logs, in any audit event, in the OpenAPI document, or in any
+recovery API response.
+
+## Revocation
+
+Revocation is permanent and immediate. A revoked credential is refused on every subsequent attempt,
+including after the node re-enrols with a new one — enrolment revokes rather than deletes, so the
+revocation survives in history. The invariant `revoked_credentials_stay_revoked` asserts no revoked
+credential records a use later than its revocation.
+
+## Rotation
+
+Rotation leaves exactly one usable secret. Two simultaneously valid secrets is an unbounded window
+rather than a graceful cutover, so the old one is revoked as the new one is issued and the partial
+unique index enforces it.
+
+## Races
+
+Revocation racing authentication was tested with sixteen concurrent threads. Whichever order the
+race resolves in the revoked credential is never accepted, and re-checking afterwards still refuses
+it.
+
+A single-use enrolment token was stormed with sixty concurrent attempts. Exactly one succeeded,
+fifty-nine were refused, and one node identity with one active credential exists afterwards. This
+covers a real M15 defect where the `used_at` check was read-then-write and two agent threads could
+burn one token on two identities for the same hardware.
+
+## Burst behaviour
+
+Sixty invalid credentials produced sixty identical 401s with no amplification, and the valid
+credential still worked afterwards. Refusal is cheap and constant: there is no lockout to trigger
+and no differential response to measure. Mean refusal latency was 12.75 ms with a 58.09 ms maximum.
+
+## Node identity
+
+A recovered node keeps its persisted identity, so losing a credential does not lose its
+accelerators, storage roots or history. A node whose identity file is also gone enrols as a new
+node — ModelForge genuinely cannot distinguish it from different hardware — and the operator
+disables the superseded row rather than letting two nodes claim the same machine.
+
+## Diagnostics
+
+A failed publication now names the status, method and path, never the response body or the
+credential. Before M16 the agent logged only `control-plane publication failed; retrying`, which
+cannot distinguish a revoked credential from a DNS failure — and had been doing so for two days
+while pointed at a placeholder URL.

docs/security/EXECUTION_APPROVALS.md

@@ -0,0 +1,13 @@
+# Execution Approvals
+
+M4 introduces `lab_execution` approval for one exact ArtifactSet. Approval is allowed only when all
+required members are present, verified and non-quarantined and when static evidence reports neither
+pickle/executable serialization nor required remote repository code.
+
+The approval stores approver, reason, timestamp, optional expiry and a canonical fingerprint of the
+artifact/security facts. It becomes stale on expiry, revocation or evidence change. Approval of a
+model name, another ArtifactSet or an older revision cannot authorize execution.
+
+LAB approval is not production approval. It authorizes only an isolated technical runtime probe.
+The M4 API exposes no production-approval or activation route, and deployment candidates produced by
+successful probes are hard constrained to `channel=lab`, `status=lab_ready`, `production=false`.

docs/security/MODEL_SUPPLY_CHAIN_POLICY.md

@@ -0,0 +1,39 @@
+# Model Supply-Chain Policy
+
+## Policy defaults
+
+- Downloaded model repositories are untrusted.
+- Exact revisions must be resolved and pinned.
+- Safe tensor formats are preferred where supported.
+- Pickle-derived artifacts require stricter handling and may be prohibited by policy.
+- `trust_remote_code=false` by default.
+- Upstream security scanner results are evidence, not absolute proof of safety.
+- License metadata must be captured and evaluated before production approval.
+- Inference workers receive approved artifacts read-only.
+- Internet egress is disabled for inference workers by default.
+- Tokens used for model acquisition remain downloader/control-plane secrets only.
+
+## Candidate admission sequence
+
+1. register upstream source;
+2. snapshot upstream metadata;
+3. resolve exact revision;
+4. download to quarantine/cache;
+5. inventory all files;
+6. calculate/record digests;
+7. collect upstream scanner/security metadata;
+8. evaluate serialization/custom-code requirements;
+9. evaluate license/gating restrictions;
+10. run local static/security checks;
+11. mark verified or blocked;
+12. only verified artifacts can enter runtime compatibility testing.
+
+## Remote code exception
+
+If an artifact requires custom repository code:
+
+- default state: production blocked;
+- optional lab execution requires explicit operator approval;
+- sandbox restrictions must be stricter than normal approved models;
+- exception reason and review evidence are audited;
+- promotion to stable requires a separate production security decision.

docs/security/QUARANTINE_AND_VERIFICATION.md

@@ -0,0 +1,23 @@
+# Quarantine and Verification
+
+Every selected file first lands below `<approved-root>/.quarantine/<job-id>`. Repository paths are
+normalized as relative POSIX paths, traversal and absolute paths are rejected, resolved paths must
+remain beneath the root, and symlink roots/files are blocked. The agent runs non-root with a
+read-only container filesystem, dropped capabilities and narrow data mounts.
+
+Verification never imports or deserializes model content. It checks regular-file size, streams
+SHA-256 in bounded chunks, compares available upstream LFS SHA-256, parses only the bounded JSON
+header of Safetensors, inspects small JSON configuration for `auto_map`, and blocks known
+pickle/executable serialization and repository Python. `trust_remote_code=false` is the invariant.
+Hub scanner signals remain separate upstream evidence.
+
+All required files must pass before the job-scoped directory is atomically renamed to
+`repositories/<repo>/<exact-sha>`. Only then does the agent report completion. The control plane
+validates the report against the immutable plan and transactionally creates `ModelArtifact`,
+`ArtifactLocation`, `ArtifactInspection`, and `ArtifactSetMember` records. The set becomes
+`availability=local`, `completeness=complete`, and `status=verified`; its security state is
+`static_checks_passed_unapproved`, not supply-chain approved.
+
+Failed or cancelled work never creates verified DB records. Retained partials remain job-scoped for
+safe retry; cleanup is an explicit lifecycle operation. A manifest beside promoted bytes records
+job, repository, commit and file digests for crash reconciliation.

docs/security/SERVICE_AUTHENTICATION.md

@@ -0,0 +1,12 @@
+# Service Authentication
+
+Service clients are separate from Node Agent identities and operator administration. Creation
+returns an `mfsvc_…` bearer secret once. Only SHA-256, a non-secret prefix, timestamps, expiry,
+status, rate/concurrency policy and explicit capability scopes are stored. Secrets are absent from
+audit events and ordinary logs.
+
+Gateway authentication rejects missing, malformed, unknown, revoked or expired credentials,
+disabled clients and missing capability scope. Each client has a sliding one-minute request limit
+and a concurrent-request limit. Credential creation and revocation are audited. The frontend keeps
+the one-time value in memory only and never reloads plaintext.
+

docs/security/SUPPLY_CHAIN_RELEASE_GATE.md

@@ -0,0 +1,102 @@
+# Supply-chain release gate
+
+## What is inventoried
+
+```text
+Python packages    read from the built images, not from a manifest
+Node packages      read from the lock file, which is what `npm ci` installs
+base images        pinned per component
+CI actions         pinned by commit SHA
+runtime images     pinned by digest
+```
+
+Reading the Python inventory from the image rather than the manifest matters: a manifest states
+intent, an image states what shipped. A build that silently resolved a different transitive version
+is exactly the case an SBOM exists to catch.
+
+## SBOM
+
+`scripts/m16_sbom.py` produces CycloneDX 1.5 at `docs/security/sbom/modelforge-cyclonedx.json`.
+It lists components and versions with package URLs, not dependency contents: a release inventory
+must be readable and diffable, so no binaries are stored.
+
+Each component carries the image it belongs to, so a finding can be traced to what actually ships
+it rather than to the repository as a whole.
+
+## Image provenance
+
+`docs/security/sbom/image-provenance.json` binds each built image to its image id, repository
+digest where one exists, creation timestamp and the exact source commit. Provenance is what makes
+the SBOM verifiable rather than merely present: without it, an inventory describes an image nobody
+can identify again.
+
+## CVE triage
+
+Findings are classified, and the classification is evidence rather than a label:
+
+```text
+EXPLOITABLE           reachable from an untrusted input in this deployment
+NOT_REACHABLE         the vulnerable path is not executed by ModelForge
+DEV_ONLY              present only in a development or test dependency
+FALSE_POSITIVE        the advisory does not apply to this version or configuration
+ACCEPTED_TEMPORARY    accepted with an owner, a reason and a review date
+MUST_FIX              blocks the release
+```
+
+"Zero CVEs" is never claimed without the scan output that supports it, and an aggregate count never
+replaces the per-finding record for a HIGH or CRITICAL.
+
+## Dependency pinning
+
+Base images are pinned to a major or minor tag so they continue to receive security patches on a
+compatible line; the runtime worker is pinned by digest because its CUDA and PyTorch versions are
+part of the reproducibility contract for a benchmark. CI actions are pinned by commit SHA.
+
+No dependency is declared as `latest`. A lock file makes a build reproducible in practice, but a
+floating specifier means the next `npm install` can pull a different major version — or a
+compromised release — without anyone deciding to.
+
+## Container posture
+
+```text
+non-root            api, node-agent, web; postgres and redis drop to their own users
+no privileged mode  none
+no Docker socket    none
+capabilities        dropped entirely on api, node-agent and web
+no-new-privileges   api, node-agent, web
+read-only rootfs    node-agent, web; api writes only to its mounted data volumes
+healthchecks        api, postgres, redis, web
+```
+
+## Network boundaries
+
+```text
+PostgreSQL   loopback only by default
+Redis        loopback only by default
+console      loopback only by default
+API          published deliberately; every admin route is operator-authenticated
+runtime      no public port; reached only through the gateway
+Node Agent   outbound only; no inbound listener
+```
+
+PostgreSQL and Redis defaulting to every interface was the most serious finding of the M16 review:
+the control-plane database, holding provenance, credential hashes and the audit trail, was reachable
+from the LAN behind a development password. Both now bind to loopback, overridable per deployment.
+
+## Secrets
+
+`gitleaks` runs over the full history, not the working tree. A finding is investigated rather than
+suppressed; the single allowlist entry names one exact string and says why it is not a credential.
+
+Generated artefacts are checked too: logs, backup manifests, audit exports and the OpenAPI document
+must contain no operator key, encryption key, node credential or database password.
+
+## Cryptography
+
+Backup encryption is AES-256-GCM from `cryptography`. ModelForge designs no primitive of its own;
+`BackupCipher` is an adapter boundary so a KMS or HSM can replace the local key.
+
+The M16 review verified, and static tests now enforce, that a fresh nonce is drawn inside the chunk
+loop rather than once per file, that each chunk is authenticated with associated data binding it to
+its key id and index, and that a decryption failure removes both the partial file and the
+destination so no readable plaintext survives a wrong key.

docs/security/THREAT_MODEL.md

@@ -0,0 +1,300 @@
+# Threat Model
+
+## M9 binary modality boundary
+
+Document, image and audio content introduces decompression, oversized-payload and privacy risks. M9 applies decoded-byte, batch, page, pixel and duration limits; validates canonical base64 and media types; denies runtime network access; forbids remote repository code; keeps content in the transient payload store; and excludes payloads from logs, audit and metrics. Specialized workers expose typed jobs only and cannot execute generic shell commands.
+
+## M10 scheduler boundary
+
+The scheduler exposes typed dry-run, managed residency-policy, drain and unload operations only.
+Ordinary clients cannot choose a node, model or eviction target. External GPU processes are
+observation-only and have no controls. Placement history retains bounded numeric shape/provenance
+metadata and never payload content. Database accelerator locks and expiring typed leases prevent
+concurrent overcommit; uncertain telemetry fails closed. Runtime isolation remains non-root,
+outbound-only and offline with read-only artifacts, no Docker socket, SSH material, Hub token,
+generic shell or cloud fallback.
+
+## M11 project-consumer boundary
+
+Operational clients bind to one non-deprecated project contract and one capability scope. The server,
+not the payload, assigns interactive/background priority. Rotation atomically invalidates the prior
+hash-only credential. Consumers cannot select a deployment, model, node, runtime or worker. Image,
+audio and document bytes remain transient and are excluded from audit, metrics and logs; adapters
+enforce decoded-size/dimension/duration bounds and do not fall back to cloud services. OCR discovery
+retains quarantine, safe-serialization, license and `trust_remote_code=false` gates.
+
+## M6 additions
+
+- Capability-scope escalation is denied by hashed service credentials and fixed client scopes.
+- Bulk-priority escalation is denied because priority comes from server-side identity state.
+- Cross-space vector reuse is denied by response, profile, payload and collection space checks.
+- Evaluation accepts typed IDs/scores only; no arbitrary code or document-body upload is executed.
+- Shadow targeting exists only in operator evaluation composition and is absent from public search
+  parameters and production alias mutation.
+
+## Assets
+
+- host and GPU compute resources;
+- project data processed by local inference;
+- Hugging Face/access tokens;
+- model artifacts and derived artifacts;
+- project credentials to ModelForge;
+- benchmark datasets that may contain sensitive project material;
+- production deployment and routing state;
+- audit history.
+
+## Trust boundaries
+
+1. Public internet / model hub → downloader
+2. Downloader → quarantine storage
+3. Quarantine → approved artifact store
+4. Control plane → inference workers
+5. Project → ModelForge gateway
+6. Browser/operator → control-plane write API
+7. Remote compute node agent → central control-plane agent API
+
+## Primary threats
+
+### Malicious model repository
+
+Potential arbitrary code, pickle payloads, custom Python modules, dependency confusion or misleading metadata.
+
+Mitigations:
+
+- treat repository as untrusted;
+- prefer safe serialization;
+- inspect file inventory;
+- hash artifacts;
+- quarantine before approval;
+- default `trust_remote_code=false`;
+- no production exception without explicit approval and evidence;
+- isolated runtime with no secrets and no unnecessary network.
+
+### Compromised runtime endpoint
+
+Direct runtime exposure may bypass ModelForge authorization or surface runtime control endpoints.
+
+Mitigations:
+
+- private runtime network;
+- only gateway can invoke workers;
+- reverse proxy/API boundary;
+- runtime endpoints not bound to public/LAN interfaces unnecessarily.
+
+### Model artifact substitution
+
+Mitigations:
+
+- pin exact revision;
+- resolve to commit SHA;
+- store digests;
+- verify before use;
+- immutable approved artifacts.
+
+### Unauthorized promotion/deletion
+
+Mitigations:
+
+- explicit privileged actions;
+- dependency checks;
+- append-only audit;
+- confirmation for destructive cleanup;
+- rollback retention policy.
+
+### Data exfiltration
+
+Mitigations:
+
+- inference workers default to no egress;
+- no cloud fallback without explicit separate capability/policy;
+- no Hugging Face token in inference workers;
+- redact secrets from logs;
+- benchmark datasets have controlled access.
+
+### GPU denial of service
+
+Mitigations:
+
+- workload priority;
+- GPU lease scheduler;
+- queue bounds;
+- memory envelopes;
+- OOM recovery;
+- production workloads preempt/deny benchmarks as appropriate.
+
+### Rogue or compromised compute node agent
+
+Threats include stolen enrollment secrets, cross-node publication, replayed telemetry, fake client
+timestamps, secret leakage and use of the agent as a remote-execution foothold.
+
+Mitigations:
+
+- short-lived single-use enrollment tokens and node credentials are random and stored hash-only;
+- enrollment is rate-limited and all create/use/revoke/disable transitions are audited;
+- every report is bound to the credential's node identity and an explicit protocol version;
+- persistent sequences reject replay/reordering and central receive time determines liveness;
+- disabled/revoked nodes fail closed;
+- remote deployments require HTTPS with certificate validation;
+- the agent is non-root, outbound-only, exposes no port, mounts no Docker socket and implements no
+  generic command-execution surface;
+- UI/list APIs never redisplay enrollment or node credentials.
+
+### Untrusted runtime execution
+
+M4 permits a narrowly scoped model load only after exact-set LAB approval. The runtime worker is a
+non-root static container with read-only root/artifact filesystems, no published port, Docker socket,
+SSH key, project secret or host-root mount. It accepts typed leases only, re-hashes every file before
+load, uses local-files-only/offline loading, enforces `trust_remote_code=false`, blocks IP network
+connections during load/inference, and exits the child process before verifying VRAM reclamation.
+LAB_READY evidence never creates a production route.
+
+## Initial negative tests
+
+- attempt direct runtime access from outside private runtime network;
+- try promoting a quarantined artifact;
+- try production-running a remote-code-required artifact without approval;
+- attempt deleting an artifact referenced by stable deployment;
+- attempt switching an embedding capability without migration;
+- enqueue large benchmark during production saturation;
+- force worker crash/OOM and verify controlled recovery;
+- verify no runtime container receives HF token or project secrets.
+- reuse an enrollment token and use one node credential for another identity;
+- publish malformed, incompatible, stale and replayed agent reports;
+- disable/revoke a node and verify further publication is denied;
+- inspect logs and list APIs for credential leakage.
+
+## M0 implementation mapping
+
+- `PolicyDefaults` rejects unsafe changes to remote code, exact revision, digest, egress, runtime
+  exposure, automatic promotion, rollback retention and benchmark priority defaults.
+- Runtime profile contracts default remote code and network egress to false and fingerprint the full
+  launch configuration.
+- Artifact provenance validates exact hexadecimal revision and SHA-256 identities; derived artifacts
+  require conversion lineage.
+- Promotion guards reject unverified artifacts, missing evidence/approval/rollback and embedding
+  changes without a ready migration.
+- The M4 runtime worker receives only its node credential through a read-only state mount. It has no
+  hub token, project secret, Docker socket or public port; Compose and GPU Node inspection verify this.
+- HTTP requests receive correlation IDs and structured request logs; persistence defines hash-chain-
+  ready, append-only audit events. Authentication/authorization arrives with the gateway and must
+  precede any production write API.
+
+## M12 lifecycle controls
+
+All lifecycle reads and writes require the operator/admin credential; capability-client, node and
+runtime credentials are denied. Approval evidence stores identifiers and bounded status facts, never
+secrets or model/project content. CAS, idempotency, single-stable uniqueness and execution-time cleanup
+rechecks limit promotion/rollback/delete races. Lifecycle cannot stop or reclaim VRAM from Ollama,
+Plex or Tdarr, invoke arbitrary shell, mount Docker control or expose a worker. Runtime remains offline
+with `trust_remote_code=false`.
+
+## M13 migration controls
+
+All migration endpoints require the operator/admin credential. Plans accept identifiers, hashes,
+counts, policy facts and a closed adapter operation enum; they accept no code, SQL, shell command,
+secret, vector, query or document body. The control plane has no SSH/Docker-socket migration surface.
+External alias changes remain adapter-owned and must report exact identity and health before database
+commit. Generations, CAS, idempotency, one-active-production constraints and stale-source checks
+limit replay/races. Failed rollback or ambiguous restart truth enters manual intervention. Active and
+rollback-retained migrations block cleanup, and external Ollama/Plex/Tdarr remain observation-only.
+
+## M14 observability controls
+
+Operations and `/metrics` require the operator credential and are not public Compose ports.
+Metric definitions reject high-cardinality/sensitive labels and exports contain no prompt, query,
+document, image, audio, vector, filename, digest, request ID, secret or raw error. Label values are
+escaped and bounded. Policy/rule changes, acknowledgement and maintenance windows are audited, while
+individual samples are not. The poller owns no SSH, Docker socket, runtime command or external GPU
+control. Persistence failure degrades monitoring only; it cannot trigger fallback inference,
+promotion, rollback, deletion, process termination or serving failure.
+
+## M16 chaos and release-gate controls
+
+M16 validated the boundaries above under fault and adversarial load, and turned each conclusion into
+a build-enforced property.
+
+- PostgreSQL, Redis and the operator console bind to loopback by default. Before M16 the
+  control-plane database was published on every interface behind a development password: a probe
+  from the host's LAN address reached it and enumerated all 121 tables, including provenance,
+  credential hashes and the audit trail. A test fails if that binding returns.
+- The API, Node Agent and console containers drop all capabilities and set `no-new-privileges`; the
+  Node Agent and console additionally run read-only. The console serves a static build from an
+  unprivileged nginx rather than a development server running as root.
+- No product route can inject a fault, run a command or open a shell. Faults are injected only from
+  the container runtime and from existing rehearsal seams. Subprocess use is allowlisted to the
+  recovery plane, argv is always a list, and static tests enforce both.
+- Fifteen executable invariants cover duplicate production state, duplicate node identity, stale
+  leases, mixed embedding spaces, unevidenced commits, restores from unverified backups, revoked
+  credential reuse, operator scopes on capability clients, unsafe promoted artifacts, orphaned work,
+  hidden promotion and audit-chain integrity. They are read-only and never touch the observability
+  database, so they remain usable when monitoring is what failed.
+- Capability scope matching is exact: case, whitespace, a second capability in the same string and
+  zero-width characters do not widen it, and a contract version is part of the identity. A client
+  bound to a project cannot serve another project's binding.
+- Unknown, revoked, near-miss and empty credentials return an identical status and code, so there is
+  no enumeration oracle. Sixty invalid attempts produced sixty identical refusals with no
+  amplification and no effect on the valid credential.
+- A single-use enrolment token survives a sixty-way concurrent storm with exactly one winner, and
+  revocation racing authentication never yields an accepted revoked credential.
+- Adversarial input across malformed bodies, wrong types, oversized payloads, mass assignment,
+  traversal, injection, XSS and header injection produces typed 4xx responses and never a server
+  error. Error responses carry no traceback, SQL, driver name, DSN, path or credential, verified
+  including with the database stopped.
+- A body containing `NaN` or `Infinity` previously crashed the validation error handler, turning a
+  422 into a server error on unauthenticated-shaped input. Rejected values are now rendered safely
+  and truncated.
+- Backup encryption remains AES-256-GCM from `cryptography` with a fresh nonce per chunk, associated
+  data binding each chunk to its key id and index, and fail-closed decryption that leaves no
+  plaintext. Static tests assert all three.
+- Supply chain: a CycloneDX SBOM read from the built images, image provenance bound to the source
+  commit, zero dependency advisories after upgrading the installer, no floating dependency
+  specifier, no privileged mode, no Docker socket, and Gitleaks clean over full history.
+
+## M15 recovery controls
+
+Backups and restores are operator-authenticated admin routes; there is no public recovery surface
+and no default administrator password anywhere in ModelForge.
+
+- Control-plane backups are AES-256-GCM encrypted at rest using `cryptography`; ModelForge designs
+  no cryptography of its own and the cipher sits behind an adapter so a KMS/HSM can replace the
+  local key. A wrong key fails closed and removes both the partial file and the destination, so no
+  readable plaintext is ever left behind.
+- The backup encryption key, the operator API key and the Hugging Face token are never written into
+  a backup, manifest, log or audit event. The configuration manifest names them and their recovery
+  class only. Live checks confirmed none of them, nor any node credential or database password,
+  appears in 328 KB of API logs or in any audit detail.
+- Credentials are stored as hashes. A restore reinstates which credentials existed and which were
+  revoked but cannot return plaintext; consumers rotate. Revocation survives re-enrolment.
+- One enrolment token mints exactly one node identity. The single-use claim is an atomic conditional
+  update, closing a race in which two concurrent agent threads could create two compute-node rows
+  for the same hardware, and the agent serialises its own enrolment.
+- Every backup and restore path resolves through an allowlisted root with traversal, drive-qualified
+  component, home-expansion and symlink-escape rejection. Archive handling validates every member
+  before extraction and refuses links, devices, absolute or traversing names, more than 10,000
+  members or more than 64 GiB declared.
+- Restore destinations are parsed, identifier-validated and compared against the running control
+  plane's own database; a self-targeting or non-PostgreSQL destination is refused. Production
+  targets need an explicit disaster-recovery mode and an explicit deployment flag, and destructive
+  replacement lives in the operator CLI rather than a remote call. No recovery route executes
+  arbitrary shell input.
+- Restore responses expose only a redacted DSN; `RestorePlanResponse` has no field carrying a
+  password, and the published OpenAPI document is asserted against that.
+- Rehydration is not a security bypass: a recovered artifact re-enters quarantine, is re-hashed
+  against provenance, is re-checked against the security policy, keeps `trust_remote_code=false`
+  and does not inherit its predecessor's approval. A digest mismatch fails the recovery.
+- Semantic fingerprints exclude every column whose name carries secret material, so a recovery
+  comparison can be exported without leaking credential hashes.
+- Recovery reconciliation clears current truth rather than resurrecting it, and the restore gate
+  refuses `READY` while a stale lease, telemetry row or inventory run survives.
+
+## M5 serving additions
+
+- Gateway bearer secrets are one-time, scoped, hashed at rest, revocable and distinct from operator
+  and node credentials.
+- Request text is held transiently in Redis and is absent from PostgreSQL, audit and structured logs.
+- Capability resolution is allow-listed; clients cannot submit a model path, runtime or arbitrary
+  capability.
+- Database accelerator locks, bounded queues, leases, expiry and a 1 GiB reserve limit resource
+  exhaustion/races. Unmanaged GPU processes are capacity, never ModelForge kill targets.
+- The worker retains M4 offline flags, `trust_remote_code=false`, read-only artifacts, dropped
+  capabilities and no public port.

docs/security/sbom/image-provenance.json

@@ -0,0 +1,33 @@
+{
+  "generated_at": "2026-08-27T20:55:00.513442+00:00",
+  "source_commit": "2fc04ba97ee81f112d73efd22629a728afd1bdc7",
+  "images": [
+    {
+      "image": "modelforge-api",
+      "image_id": "sha256:78d780febc6bb07b71d8b39570c8ccbc1b5cf9bd62470b0b26cf40441c189459",
+      "repo_digest": "modelforge-api@sha256:78d780febc6bb07b71d8b39570c8ccbc1b5cf9bd62470b0b26cf40441c189459",
+      "created": "2026-08-27T20:53:32.407519964Z",
+      "base_image": null,
+      "source_commit": "2fc04ba97ee81f112d73efd22629a728afd1bdc7",
+      "source_repository": "https://git.example.com/example/modelforge.git"
+    },
+    {
+      "image": "modelforge-node-agent",
+      "image_id": "sha256:b7eb7eba5fbcae2d0c7acb9f165944d543aa0419a55abc22daffe652d617e7ed",
+      "repo_digest": "modelforge-node-agent@sha256:b7eb7eba5fbcae2d0c7acb9f165944d543aa0419a55abc22daffe652d617e7ed",
+      "created": "2026-08-27T20:54:24.191748869Z",
+      "base_image": null,
+      "source_commit": "2fc04ba97ee81f112d73efd22629a728afd1bdc7",
+      "source_repository": "https://git.example.com/example/modelforge.git"
+    },
+    {
+      "image": "modelforge-web",
+      "image_id": "sha256:dcf39544827eeafd682cd2be1b6b9811fd118847e9e184adbeae20e0cd148244",
+      "repo_digest": "modelforge-web@sha256:dcf39544827eeafd682cd2be1b6b9811fd118847e9e184adbeae20e0cd148244",
+      "created": "2026-08-27T20:53:47.061648758Z",
+      "base_image": null,
+      "source_commit": "2fc04ba97ee81f112d73efd22629a728afd1bdc7",
+      "source_repository": "https://git.example.com/example/modelforge.git"
+    }
+  ]
+}