Jens/ModelForge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial public ModelForge release

Browse Source
This commit is contained in:
Jens
2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations/CHAOS_TESTING.md
+78
View File
@@ -0,0 +1,78 @@
# Chaos testing
## Fault matrix
Every subsystem, the failure modes it can present, how far the damage reaches, and the invariant
that must survive.
| Subsystem | Failure mode | Class | Blast radius | Recovery | Alert | Invariant |
| --- | --- | --- | --- | --- | --- | --- |
| Control Plane | process crash / restart | TRANSIENT | in-flight API requests | startup reconciliation | none by design | no authoritative drift |
| Control Plane | restart during backup | TRANSIENT | that backup only | interrupted backup fails closed | `BACKUP_FAILED` | `restore_requires_verified_backup` |
| PostgreSQL | unavailable | DEPENDENCY | all persistence | pool reconnects, no restart needed | `PLATFORM_API` SLO | `audit_chain_intact` |
| PostgreSQL | deadlock / serialisation failure | CONCURRENCY | one transaction | bounded retry | none | no lost update |
| PostgreSQL | latency | CAPACITY | request queueing | timeouts, typed errors | SLO burn | no connection explosion |
| Redis | unavailable | DEPENDENCY | queues, transient payloads | queues start empty, leases reconciled | dependency health | no duplicate serving job |
| Node Agent | network loss | DEPENDENCY | that node's capacity | heartbeat stale → offline → reconnect | `NODE_OFFLINE` | `single_node_identity` |
| Node Agent | credential revoked | AUTH | that node | re-enrol, identity preserved | `NODE_OFFLINE` | `revoked_credentials_stay_revoked` |
| Node Agent | restart storm | TRANSIENT | that node | same identity, no duplicate inventory | none | `single_active_node_credential` |
| Runtime Worker | crash during load/invoke/unload | TRANSIENT | one request | typed failure, generation fencing | `RUNTIME_CRASH_LOOP` | `no_orphan_serving_work` |
| Gateway | invalid credential burst | AUTH | none | identical refusal | none | no enumeration signal |
| Gateway | malformed or hostile input | SECURITY | none | typed 4xx | none | no execution, no 5xx |
| Scheduler | GPU pressure | CAPACITY | admission | reject or queue | `GPU_PRESSURE` | `no_stale_gpu_lease` |
| Scheduler | external VRAM spike | CAPACITY | admission | schedulable headroom shrinks | `CAPABILITY_CAPACITY` | external workloads untouched |
| GPU | telemetry stale | DEPENDENCY | placement decisions | admission blocked, never guessed | `GPU_PRESSURE` | no stale snapshot as truth |
| Artifact storage | location missing | CORRUPTION | one artifact set | exact-revision rehydration | `ARTIFACT_INTEGRITY` | `no_unsafe_artifact_promoted` |
| Artifact storage | disk low / full | CAPACITY | acquisition | capacity guard refuses first | `STORAGE_LOW` | no partial promotion |
| Backup storage | unwritable / absent | CAPACITY | that backup | typed `DESTINATION_UNAVAILABLE` | `BACKUP_FAILED` | serving unaffected |
| Backup storage | payload corrupted | CORRUPTION | that backup | verification refuses | `BACKUP_VERIFICATION_FAILED` | `restore_requires_verified_backup` |
| Lifecycle | operation interrupted | TRANSIENT | one LAB subject | rollback from immutable snapshot, exactly once | `ROLLBACK_FAILURE` | `lifecycle_commit_has_evidence` |
| Migration | cutover interrupted | CORRUPTION | one plan | never auto-resolved; operator reconciles | `MIGRATION_FAILURE` | `cutover_commit_has_validation` |
| Migration | backfill worker failure | TRANSIENT | one plan | checkpoint preserves completed work | `MIGRATION_FAILURE` | no mixed embedding space |
| Observability | persistence unavailable | DEPENDENCY | monitoring only | degraded gauge, serving continues | `modelforge_observability_degraded` | safety never depends on the alert store |
| Recovery | restore interrupted | TRANSIENT | that restore | manual intervention, resumable | `RESTORE_FAILED` | `restore_requires_verified_backup` |
| Credentials | rotation under load | AUTH | one client | old refused, new accepted | none | one usable secret |
| External | Hugging Face unavailable | DEPENDENCY | acquisition only | `ARTIFACT_REHYDRATION_BLOCKED` | none | no fallback source |
| External | ExampleRAG / ExampleVision down | EXTERNAL | none | not owned | none | never written to |
## Failure classes
```text
TRANSIENT recoverable by retry or restart; state converges
PERSISTENT requires operator action; state is safe but stuck
CORRUPTION integrity is in question; the platform must refuse rather than proceed
CAPACITY the safe answer is to reject work, not to risk exhaustion
DEPENDENCY something ModelForge does not own is unavailable
AUTH a credential is missing, wrong, revoked or out of scope
SECURITY hostile input or an attempt to cross a boundary
CONCURRENCY two operations race; exactly one must win
OPERATOR a human action that must be gated, audited and reversible
```
A capacity rejection is not a failure. The scheduler refusing unsafe work is the system behaving
correctly, and the soak counts those separately from internal errors for exactly that reason.
## Invariants
Fifteen properties, each a query the platform runs against its own authoritative state. They are
read-only, never call an external system, and never depend on the observability database — so they
stay usable exactly when monitoring is the thing that failed.
```sh
python scripts/m16_invariants.py --label "before change"
```
See `docs/security/CHAOS_SECURITY_GATE.md` for the full list and what each one protects.
## Running the suite
```sh
python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --all --report chaos.json
python scripts/m16_soak.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --minutes 45 --secret <secret>
python scripts/m16_invariants.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL"
```
Chaos scenarios and the soak may be run together deliberately — a backup taken while the soak is
generating real traffic is the M16 "backup under load" evidence — but a soak whose numbers are
meant to characterise steady state should not overlap an injected fault. When they do overlap, say
so in the report rather than presenting the mixed result as a clean baseline.
docs/operations/FAILURE_AND_HEALTH_MODEL.md
+43
View File
@@ -0,0 +1,43 @@
# Failure and Health Model
## Health layers
### Process health
Is the worker/control-plane process running?
### Runtime health
Can the adapter communicate with the worker and does its runtime/version match the deployment profile?
### Hardware health
Was the node inventory collected, and are previously observed accelerators still reconcilable?
Remote nodes additionally report `online`, `stale`, `offline` or `disabled`. These states derive
from central receive time, not the node clock. Stale/offline nodes are retained with last-good
inventory; they are visibly unavailable for future scheduling rather than silently deleted.
### Accelerator health
Is the accelerator present and observable? Unsupported optional metrics are not health failures.
### Model health
Can the deployed model load and execute a known smoke inference?
### Capability health
Can the current routing/policy for a capability satisfy its contract?
### Project health
Are all required project capabilities available within acceptable SLO/policy?
## Degradation examples
`rag.reranking` may allow a project-specific policy to continue retrieval without second-stage reranking.
`rag.embedding` may be configured as a hard dependency and fail requests rather than generate inconsistent vectors via an unapproved fallback.
Fallback semantics are part of capability/project policy, not implicit behavior in runtime code.
docs/operations/FAULT_INJECTION.md
+83
View File
@@ -0,0 +1,83 @@
# Fault injection
## Where faults come from
ModelForge has no chaos endpoint and no way to run a command. A control plane that can be told to
break itself is a control plane an attacker can tell to break itself, so every fault is injected
from outside the product:
```text
container runtime stop, start, restart, network disconnect, filesystem permissions
rehearsal seams the M12 lifecycle pause, the M15 restore interruption
real credentials revoked, rotated, malformed, wrongly scoped
real load the soak harness, running the actual capability path
```
A static test asserts the product exposes no `/chaos`, `/shell`, `/exec`, `/command` or `/debug`
route segment, and that subprocess use stays allowlisted to the recovery plane.
## The shape every scenario has
```text
preflight → baseline → inject → observe → assert → recover → verify invariants → cleanup
```
`preflight` refuses to start if an invariant is already violated: injecting a fault into a broken
platform measures nothing. `verify invariants` re-runs the full set afterwards, so a scenario that
survives its fault but corrupts state is reported as `INVARIANT_VIOLATED` rather than passing.
A scenario that cannot restore the platform reports `RECOVERY_FAILED` rather than leaving the
environment dirty and calling itself done.
## Running it
```sh
python scripts/m16_chaos.py --list
python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --scenario redis-outage --seed 20260827
python scripts/m16_chaos.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --all --report chaos.json
```
The database URL is required for executable scenarios and must name the non-owner runtime role;
the harness has no embedded or owner-credential fallback. `--list` does not connect to PostgreSQL.
Every run records its scenario id, seed, target, expected fault, observed result, recovery, cleanup
outcome and the invariant results before and after. The seed is logged even where behaviour is
deterministic, so a surprising result can be reproduced exactly.
## Scenarios
| Key | Class | Subsystem | Expected |
| --- | --- | --- | --- |
| `redis-outage` | DEPENDENCY | Redis | control plane stays up, replays nothing |
| `postgres-restart` | DEPENDENCY | PostgreSQL | fails safely, recovers its pool, no duplicate work |
| `node-agent-disconnect` | DEPENDENCY | Node Agent | heartbeat stops, liveness degrades, one identity survives |
| `control-plane-restart-storm` | TRANSIENT | Control Plane | idempotent reconciliation, no authoritative drift |
| `invalid-auth-burst` | AUTH | Control Plane | identical refusal, no amplification, no enumeration signal |
| `backup-destination-unavailable` | CAPACITY | Backup storage | typed failure, serving unaffected |
## Safety rules
Disposable targets only. The node-disconnect scenario enrols a throwaway agent rather than
isolating GPU Node: GPU Node reaches the control plane over the LAN rather than the Compose network, so
disconnecting a container would not interrupt it anyway — and isolating the production node to
prove a point is exactly what a release gate must not do.
Nothing in the harness touches ExampleRAG, ExampleVision, Ollama, Plex or Tdarr. Their VRAM counts as
external pressure and is observed; their processes are never stopped, paused, throttled or
reconfigured.
The harness deletes its own fixtures. `backup-destination-unavailable` removes the `chaos-*` backup
sets it created; `node-agent-disconnect` tears down its agent, revokes its credential and disables
the node it enrolled.
## When a scenario finds something
Both defects the harness found in M16 were real product defects rather than harness noise: an
unwritable backup root escaping as an unhandled error, and a Node Agent logging a failure with no
diagnostic. Each was fixed with a regression test before the scenario was re-run.
One scenario was itself wrong first: the node-disconnect measured GPU Node while disconnecting an
unrelated container, and reported that the heartbeat kept advancing during the "outage". That
contradiction is why the harness records what it observed rather than only whether it passed.
See `docs/operations/CHAOS_TESTING.md` for the fault matrix and
`docs/security/CHAOS_SECURITY_GATE.md` for the invariants.
docs/operations/GPU_LEASES.md
+17
View File
@@ -0,0 +1,17 @@
# GPU Leases
`serving_gpu_leases` is the database coordination boundary for GPU capacity. A lease binds the
request, deployment, node and accelerator to reserved bytes, priority, owner, server-created
timestamps and an expiry. States are pending, granted, active, releasing, released, expired and
failed.
The scheduler locks the accelerator row before accounting and granting. This prevents two control
plane processes from independently spending the same capacity. Production and interactive are
serving priorities; background and benchmark remain defined but cannot pre-empt production in M5.
Expired leases are reaped and their unfinished request is failed with `LEASE_TIMEOUT`.
M10 distinguishes the lease purpose (`RESIDENCY_RESERVATION`, `REQUEST_EXECUTION`, or
`LOAD_TRANSITION`), records priority, expiry and a 64-bit worker generation, and subtracts only the
unmaterialized portion from physical headroom. Startup reconciliation expires orphan reservations,
preserves exact worker-reported residency and prevents duplicate adoption. The accelerator row lock
remains authoritative when two heavy admissions race.
docs/operations/GPU_SCHEDULER.md
+116
View File
@@ -0,0 +1,116 @@
# GPU Scheduler Design
## M6 reindex QoS
The database claim order remains production, interactive, background, benchmark. ExampleRAG shadow
jobs are stamped `background` from their scoped identity. External Ollama/Tdarr VRAM remains
observed capacity pressure and is never managed by ModelForge.
## M10 operational objective
Admit capability requests on one finite GPU without letting ModelForge or experimental work disrupt
production or unmanaged workloads.
## Priority classes
1. production
2. interactive
3. background
4. benchmark
5. maintenance
Priority is not identical to preemption support. Some runtimes/jobs cannot be safely interrupted and must be drained instead.
## Residency policies
- `always_warm`
- `keep_warm` with TTL
- `load_on_demand`
- `exclusive` and `lab_only` remain future policy values, not M5 serving choices.
## Resource leases
Inference begins only when the scheduler issues a lease for required accelerator resources.
Lease evaluation considers:
- target accelerator compatibility;
- latest total NVML usage, including unmanaged processes;
- reserved safety margin;
- deployment resource envelope;
- current leases;
- queue priority;
- model load/sleep/unload cost;
- production eligibility and exact measured compatibility.
M5 never silently falls back to CPU, another model, a workstation or cloud inference.
The pre-M10 budget was:
```text
total VRAM - external usage - ModelForge residency - active leases - 1 GiB reserve
```
Rows are aggregated once per accelerator. M10's authoritative invariant, dynamic reserve,
multi-resident placement, hysteresis and typed failure behavior are specified in
`docs/architecture/ADVANCED_GPU_SCHEDULER.md`. GPU Node is eligible; the DESKTOP workstation is explicitly
rejected because `production_eligible=false`. The accelerator row is locked while a lease is granted.
## Resource envelope
Memory is not a single static number. Record measured profiles across:
- context length;
- concurrency;
- batch size;
- idle vs peak VRAM;
- runtime version/configuration.
## M5 embedding decision
```text
Request: rag.embedding@1
Deployment: stable and cold
Envelope: 1,216,348,160 process-reserved bytes
Policy: KEEP_WARM 901s, concurrency 1, queue 16
Decision: lock GPU Node GPU, account unmanaged VRAM and reserve, grant a lease, load once, run a
functional health check, invoke, release the request lease and retain residency.
```
## Failure classes
- MODEL_LOAD_FAILED
- GPU_OOM
- RUNTIME_CRASH
- TIMEOUT
- HEALTHCHECK_FAILED
- INVALID_OUTPUT
- ARTIFACT_CORRUPT
- CAPABILITY_UNAVAILABLE
- DRIVER_ERROR
- SCHEDULER_STATE_STALE
- INSUFFICIENT_SCHEDULABLE_VRAM
- EXTERNAL_GPU_PRESSURE
- RESIDENCY_CONFLICT
- EVICTION_NOT_ALLOWED
- DEADLINE_CANNOT_BE_MET
- LEASE_CONFLICT
- GPU_MEMORY_NOT_RECLAIMED
- QUEUE_FULL
- LEASE_TIMEOUT
Failures must produce structured events. No silent retry loop may hide prolonged capability degradation.
## M14 operational telemetry
The Operations plane samples the existing NVML/scheduler truth once per minute. A retained snapshot
separates total, NVML-observed, ModelForge resident, active leased, reserve, external and schedulable
bytes and carries observed/received freshness. `GPU_PRESSURE` uses the existing hysteresis state;
`CAPABILITY_CAPACITY` compares a measured production ResourceEnvelope with known headroom. Neither
alert authorizes control of Ollama, Plex, Tdarr or another external GPU process. See
`docs/architecture/CAPACITY_TRENDS.md` and `docs/operations/RUNBOOK_GPU_PRESSURE.md`.
Runtime adapters own model load, crash and invalid-output detection. The scheduler owns VRAM
reservation, OOM recovery and lease release. The gateway owns bounded retries and only contract-
approved fallback/degradation. The control plane blocks corrupt artifacts and records state changes.
docs/operations/MODEL_STORAGE.md
+66
View File
@@ -0,0 +1,66 @@
# Model Storage
## Configuration model
Registry storage is explicit and node-scoped. `StorageRoot` references a `ComputeNode`, has a host
path and purpose, and stores write/capacity observations plus reserve policy. It is intentionally
separate from `StorageVolumeState`: the latter is agent telemetry, while a storage root is an
operator-approved placement target.
Every placement check is fail-closed:
- unknown capacity blocks placement;
- non-writable roots block placement;
- usable bytes are `free - max(reserve_bytes, capacity * reserve_percent)`;
- a request crossing that reserve becomes `capacity_blocked`.
No default silently assumes infinite storage.
## GPU Node validation — 2026-08-25
The Unraid array reported 54,983,962,927,104 bytes total, 53,206,636,449,792 used and
1,777,326,477,312 available (97% used). It is explicitly unsuitable for model placement.
The cache-backed `appdata` filesystem reported approximately 1.438 TB total and 535 GB free (63%
used). ModelForge created this dedicated root:
`/mnt/cache/appdata/modelforge-node-agent/artifacts/model-registry`
Ownership is UID/GID 999 with mode 0750. A write/create/remove probe from the existing non-root,
read-only, capability-dropped node-agent container passed. The registered root is
`5774fef2-8cd4-44bd-a244-5b5b20c02811` on GPU Node node
`c3fab19c-d7f7-4dde-80ac-2edf565ad61d`.
The stored observation is 1,438,282,285,056 bytes capacity and 534,938,255,360 bytes free, with a
50 GiB absolute reserve and 10% percentage reserve. Effective zero-request usable capacity was
391,110,026,855 bytes. The array-unsuitable decision, backing filesystem, write-probe outcome and
`download_performed=false` are retained in validation details.
## M12 cleanup boundary
CleanupPlan is a dry-run over node-local ArtifactLocations and all known control-plane dependencies.
Execution rechecks the dependency digest, requires operator confirmation and records physical removal
only after the typed storage action succeeds. Removing a GPU Node location cannot mark the artifact
globally missing while another verified node location exists. Metadata archive and byte removal are
separate operations; rollback-retained artifacts are not eligible.
M2 did not download or move any model artifact. M3 maps the host root to
`/data/artifacts/model-registry` inside the GPU Node agent. Planning and control-plane execution checks
use the registered observation; the agent rechecks the actual filesystem immediately before bytes.
There is no fallback to `/mnt/user` or another storage root.
After the M3 acceptance artifact was promoted, GPU Node reported 1,440,503,169,024 bytes capacity and
536,169,676,800 bytes free. The effective reserve is 144,050,316,902 bytes (10%, stricter than
50 GiB), leaving 392,119,359,898 bytes usable. Status remains `ready`. The accepted artifact set
uses 1,207,470,234 bytes plus a 4,256-byte reconciliation manifest.
## M15 artifact recovery
Artifacts are classified for recovery rather than treated uniformly. A set with an exact upstream
repository and commit is `REHYDRATABLE` and is protected by a manifest recording every file's
SHA-256 and size; recovery redownloads that exact revision through quarantine. A set without both is
`NON_REHYDRATABLE` and needs a payload backup — the platform will not pretend it can be redownloaded.
Recovery targets a disposable storage root, never the only usable copy of a production artifact. A
recovered artifact re-enters the supply chain unapproved. See
`docs/architecture/ARTIFACT_RECOVERY.md` and `docs/operations/RUNBOOK_ARTIFACT_LOSS.md`.
docs/operations/RC_ACCEPTANCE.md
+36
View File
@@ -0,0 +1,36 @@
# Isolated public-candidate acceptance
The Gitea workflow `.gitea/workflows/public-candidate-acceptance.yml` moves the expensive release
proof off a developer workstation and onto a Docker-capable server runner. It is manual-only and
accepts an exact canonical commit SHA. It does not invoke `unraid-deploy.yml`, the deploy controller
or the production Compose project.
The job creates a curated public export, validates its content manifest, scans it with Gitleaks,
then creates a temporary parentless Git commit so release provenance describes the exact exported
tree. From that tree it:
1. builds API, Console, Node Agent and Runtime Worker images with OCI identity;
2. produces the release manifest, checksums, CycloneDX inventory and image provenance;
3. scans every exact image ID with checksum-pinned Trivy and rejects ambiguous reports or any
HIGH/CRITICAL finding;
4. starts API, Console, PostgreSQL and Redis with the production overlay under a unique
`modelforge-rc-*` Compose project;
5. uses newly generated secrets, loopback-only ephemeral host ports and project-scoped volumes;
6. proves liveness, readiness, version identity, Console serving and both sides of the operator
authentication boundary; and
7. always removes the temporary containers, networks, volumes and four candidate image tags. An
independent `if: always()` workflow cleanup repeats the exact project-label cleanup after a
failure, cancellation or timeout; it never runs a host-wide Docker prune.
No Node Agent or Runtime Worker is started by this clean-install gate. Consequently it creates no
compute identity and cannot interact with the production GPU node. A later GPU qualification must
enrol a disposable `rc-canary` identity with `production_eligible=false` and retain that separate
evidence.
After the workflow exists on the default branch, dispatch it directly and enter the exact pushed
commit. Before merge, dispatch the existing **Managed validation** workflow on the candidate branch
with profile `build`; that profile calls the same acceptance workflow with `gitea.sha`. Use the
actual intended Console API origin for a publishable artifact; the managed-branch route uses
`https://modelforge.example.test` and therefore produces non-publishable infrastructure rehearsal
artifacts. Evidence is retained for 90 days. A failed scan or install is evidence of a blocker, not
permission to bypass the gate.
docs/operations/RESIDENCY_MANAGER.md
+13
View File
@@ -0,0 +1,13 @@
# Residency Manager
Residency is distinct from a request lease. M10's full multi-resident identity and recovery contract
is specified in `docs/architecture/MULTI_CAPABILITY_RESIDENCY.md`. The state machine is cold → loading → warm/busy →
draining/unloading → cold, with explicit failed states. A cold request creates exactly one serialized
load job; later requests reuse the resident adapter at its measured concurrency of one.
Policies are `always_warm`, `keep_warm`, `load_on_demand` and `lab_only`. The production embedding deployment
uses `keep_warm` with a 901-second TTL. Unload is allowed only without active leases. Model-owned
VRAM reclaim is measured with the PyTorch CUDA allocator; NVML total memory remains the independent
external-pressure signal. Workers report the complete deployment inventory and generation.
Exact immutable identities can be adopted after a control-plane restart; absent identities are
reconciled cold and unknown GPU processes remain external.
docs/operations/RESTART_RECOVERY.md
+82
View File
@@ -0,0 +1,82 @@
# Restart and crash recovery
## What each restart reconciles
The control plane reconciles on every start. Reconciliation is idempotent by design, so a restart
storm converges rather than accumulating.
```text
lifecycle incomplete operations roll back from their immutable snapshot, exactly once
migration interrupted cutovers are reported, never auto-resolved
recovery interrupted backups fail closed; interrupted restores need operator review
serving stale leases and residency are reconciled against live node state
observability current alert state is re-derived; history is retained
```
Migration is deliberately the exception. External alias truth cannot be inferred after a crash, so
an interrupted cutover stays in its intermediate stage and is reported as requiring reconciliation
through the typed adapter. Guessing here is how a platform ends up serving two embedding spaces.
## Restart storms
Four consecutive control-plane restarts under load produced no authoritative drift: models,
revisions, artifacts, artifact sets, projects, bindings, capability deployments, lifecycle
operations, migration plans and cutovers, compute nodes, node credentials, service clients and
backup sets were all identical before and after, and the registry answered on every start.
Audit events are expected to grow — reconciliation is itself an audited action — so the drift check
excludes them and the `audit_chain_intact` invariant covers their integrity instead.
## Node Agent restarts
A restarting agent keeps its persisted identity file, so it re-attaches to the same node with the
same accelerators, storage roots and history. It does not re-enrol unless it has lost its
credential, and enrolment revokes the previous credential rather than adding a second.
The publication and artifact loops share an enrolment lock. Without it they could enrol
concurrently from separate threads and burn a single-use token on two identities for the same
hardware.
## Database restarts
The control plane fails safely while PostgreSQL is unavailable: liveness stays up because it does
not touch the database, and any route that needs persistence returns an error rather than inventing
state. The connection pool recovers on its own — no control-plane restart is required — and no
duplicate audit sequence appears.
An error raised while the database is down carries no DSN, driver name, SQL or traceback. That is
verified directly, because a dependency failure is when a framework is most likely to leak
internals.
## Redis restarts
Redis holds queues and transient payloads, never authoritative state. An outage leaves the control
plane serving, and the return produces no duplicate serving job: idempotency keys are unique in the
database, not in the queue.
## Runtime Worker crashes
A worker crash during load, inference or unload produces a typed failure for the request, fences the
worker generation so a late result from a dead generation cannot be committed, and reconciles the
lease and residency. A crash loop raises `RUNTIME_CRASH_LOOP`.
## Interrupted backups and restores
A backup interrupted mid-write is moved to `FAILED` with `MANIFEST_INCOMPLETE` on the next start and
can never become restore eligible. A restore interrupted mid-phase becomes
`MANUAL_INTERVENTION_REQUIRED` rather than resuming silently; its journal records the phase it
reached.
A backup also cannot contain a record of itself as complete, since the dump is taken while its own
`BackupSet` row is still `CREATING`. A restored control plane therefore consistently reports the
backup it came from as not restore eligible.
## Checking after a restart
```sh
python scripts/m16_invariants.py --label "after restart"
curl -s localhost:8000/api/v1/health/ready
```
Fifteen invariants holding, readiness healthy, and no orphaned lease or queued job is what "the
restart reconciled" means. See `docs/operations/CHAOS_TESTING.md` for the fault matrix.
docs/operations/RUNBOOK_ARTIFACT_LOSS.md
+48
View File
@@ -0,0 +1,48 @@
# Runbook: artifact loss
## Trigger
An artifact location is missing, corrupt, or unavailable after a storage failure or a control-plane
restore.
## Diagnose
Establish the recovery class first, because it decides whether recovery is possible at all:
| Class | Recovery |
| --- | --- |
| `REHYDRATABLE` | exact upstream repository and commit exist; redownload |
| `NON_REHYDRATABLE` | no upstream identity; only a payload backup can recover it |
| `DERIVED` | rebuild from exact source artifacts, preserving lineage |
| `LOCAL_ONLY` | payload backup only |
`artifacts.json` inside the backup lists every artifact set with its class, upstream repository,
exact commit SHA and per-file SHA-256.
## Act and recover
1. Register a disposable storage root for the recovery. Never rehydrate over the only usable copy of
a production artifact.
2. Create the recovery operation (`POST /api/v1/admin/recovery/artifact-recoveries`) and confirm it
reports `REHYDRATABLE` with the exact commit you expect. If it reports `NON_REHYDRATABLE`, stop:
this artifact needs a payload restore, not a download.
3. Create a download plan for that artifact set against the disposable root, approve it and execute
it. The Node Agent downloads into quarantine, hashes every file and promotes atomically.
4. Record the observed evidence against the recovery operation.
Rehydration is never done from a floating revision. If the upstream is unreachable, record the
operation as `BLOCKED` with `ARTIFACT_REHYDRATION_BLOCKED` and say so — do not report the platform
as recovered while an artifact it depends on is still missing.
## Security
A rehydrated artifact is new material. It re-enters quarantine, is re-hashed against provenance and
re-checked against the security policy, and it carries `static_checks_passed_unapproved` rather than
inheriting its predecessor's approval. `trust_remote_code` stays `false`. A digest mismatch fails
the recovery with `ARTIFACT_HASH_MISMATCH` even if the download reported success.
## Confirm
Every expected file present with a matching digest, byte counts equal to provenance, the artifact
locations `verified`, lineage intact for derived artifacts, and a runtime probe or capability smoke
against the recovered set before it is used for anything.
docs/operations/RUNBOOK_BACKUP_FAILURE.md
+43
View File
@@ -0,0 +1,43 @@
# Runbook: backup failure or stale backup
## Trigger
`BACKUP_FAILED`, `BACKUP_VERIFICATION_FAILED`, `BACKUP_STALE` or `RECOVERY_READINESS_DEGRADED`.
## Diagnose
Read the backup set's `failure_code` and the `verification` block, which records exactly how far
verification reached before it stopped.
| Failure code | Meaning |
| --- | --- |
| `MANIFEST_HASH_MISMATCH` | the manifest no longer matches its recorded identity |
| `MANIFEST_INCOMPLETE` | objects and journal disagree, no database payload, or the backup was interrupted |
| `HASH_MISMATCH` | a payload object no longer matches the manifest |
| `PAYLOAD_MISSING` | an object is absent from the allowlisted root |
| `DECRYPTION_FAILED` | the configured key cannot authenticate the payload |
| `ENCRYPTION_KEY_UNAVAILABLE` | the policy requires encryption and no key is configured |
| `INSUFFICIENT_CAPACITY` | the destination lacked headroom; nothing was written |
| `BACKUP_TOOL_UNAVAILABLE` | `pg_dump` is missing from the control-plane image |
| `CONCURRENT_OPERATION` | another backup or restore is in flight |
`BACKUP_STALE` means no *verified* backup exists inside the policy window. A `CREATED` backup that
was never verified does not count and never will.
## Act and recover
Do not delete the failed set: it is evidence, and retention protects the last verified backups
independently of it. Fix the cause — restore storage headroom, supply the encryption key, replace
damaged media — then take a new backup and verify it. Verification is idempotent and safe to repeat.
If verification fails on a backup that previously verified, treat the destination as suspect. Take a
fresh backup immediately, verify it, and only then investigate the older one; a platform with one
damaged backup and no newer verified backup is unprotected.
Never mark a backup restore eligible by editing the database. Only successful verification does that.
## Confirm
`GET /api/v1/admin/recovery/dashboard` shows a `latest_verified_backup_id`, an age inside
`backup_staleness_threshold_seconds`, `stale_backup: false` and no unprotected assets. The
`BACKUP_STALE` and `RECOVERY_READINESS_DEGRADED` alerts resolve on the next evaluation.
docs/operations/RUNBOOK_CONTROL_PLANE_LOSS.md
+46
View File
@@ -0,0 +1,46 @@
# Runbook: control-plane loss
## Trigger
The ModelForge control plane is unrecoverable: host lost, image lost, configuration lost, or all
three.
## Rebuild
1. Read the DR bundle for the chosen backup:
`python -m modelforge_api.cli.dr bundle --backup-id <id>`. It names the ModelForge commit,
repository and reference, the backup identity and manifest hash, the encryption key requirement,
the host paths, the artifact recovery plan, the node enrolment requirement and the external
dependencies.
2. Clone the canonical repository and check out the exact commit the backup records. ModelForge does
not back up its own Git host; that is platform operations.
3. Provision PostgreSQL of a compatible major version and an empty database.
4. Supply configuration and secrets. There is no default administrator password anywhere in
ModelForge. `MODELFORGE_OPERATOR_API_KEY` and `MODELFORGE_BACKUP_ENCRYPTION_KEY` are operator
inputs; the encryption key is never written into a backup and a restore cannot proceed without it.
5. Restore the database — see `RUNBOOK_DATABASE_RESTORE.md`.
6. Start the control plane against the restored database. Do not let it run migrations it has not
restored; the restore already brought the schema to head.
7. Re-enrol or reconnect the node — see `RUNBOOK_NODE_LOSS.md`.
8. Rehydrate missing artifacts — see `RUNBOOK_ARTIFACT_LOSS.md`.
## Break-glass credentials
If the operator API key is lost, the recovered deployment starts with none configured and every
admin route returns 503 rather than falling open. Set a new key in deployment configuration and
restart. Project and node credentials restore as hashes only: consumers that lost their plaintext
need a new credential issued, not a recovered one.
## What the rebuilt plane will and will not know
It knows all restored history: provenance, lifecycle, migrations, approvals, audit, observability
history, project bindings and node identities. It does not know current truth — telemetry, leases,
residency and inventory are empty until a live agent reports. Restored firing alerts are suppressed
so the rebuilt plane re-derives which alerts are actually firing now.
## Confirm
Liveness and readiness are healthy, the OpenAPI document publishes, an unauthenticated admin call
returns 401 and an authenticated one returns 200, the registry and project views serve restored
truth, and the recovery dashboard honestly reports that the *new* deployment has no verified backup
of its own yet. Take one immediately.
docs/operations/RUNBOOK_DATABASE_RESTORE.md
+51
View File
@@ -0,0 +1,51 @@
# Runbook: control-plane database restore
## Trigger
Loss, corruption or suspected divergence of the ModelForge PostgreSQL database, or a scheduled
disaster-recovery rehearsal.
## Preconditions
A `VERIFIED` backup set, the backup encryption key, a PostgreSQL server of a compatible major
version, and an empty destination database. A restore may never target the database the running
control plane is using; that is refused with `DESTINATION_NOT_ISOLATED`.
## Act
1. Verify the backup again — `POST /api/v1/admin/recovery/backups/{id}/verify`. Never restore from
a set you have not just verified.
2. Create a restore plan with the destination, artifact strategy, secret strategy and node
strategy. Use `VALIDATION` into an isolated environment unless this is a real disaster.
3. Run the preflight and read all eleven checks. Do not proceed on a failure; each failure code
names a real precondition.
4. Start the restore and advance it. The journal records every phase with its evidence and
duration.
5. Read the validation result: table, constraint and index counts, the fingerprint diff, the
residual current-truth rows, the measured RPO and RTO, and the `READY` gate.
For a destructive replacement, use the operator CLI rather than the API:
```sh
python -m modelforge_api.cli.dr verify-backup --backup-id <id>
python -m modelforge_api.cli.dr plan-restore --backup-id <id> --target-url <dsn> \
--target-label <label> --mode DISASTER_RECOVERY --target-environment PRODUCTION --reason "..."
python -m modelforge_api.cli.dr validate-restore --plan-id <uuid>
python -m modelforge_api.cli.dr run-restore --plan-id <uuid> --reason "..."
```
## If it fails
`DESTINATION_NOT_EMPTY` on a retry is correct behaviour, not an obstacle: the previous attempt left
data. Drop and recreate the destination database, then retry. Do not merge into a partially
restored database.
`MANUAL_INTERVENTION_REQUIRED` means the restore reached a state that must not be resolved
automatically. Read the journal, decide, and record the decision.
## Confirm
The operation reaches `READY` with every gate true, including `current_truth_reconciled`. Compare
the fingerprint diff against the expected differences: the recovery reconciliation audit event, and
whatever the live source wrote after the backup point. Anything else must be explained before the
restore is accepted.
docs/operations/RUNBOOK_FULL_DR.md
+62
View File
@@ -0,0 +1,62 @@
# Runbook: full disaster recovery
The complete procedure when ModelForge must be rebuilt from a backup. Each step has its own runbook;
this one gives the order and the gates between them.
## Before you start
You need a `VERIFIED` backup set, the backup encryption key, access to the canonical Git repository,
a PostgreSQL server of a compatible major version, and the operator API key you intend to use
afterwards. Without the encryption key the restore cannot proceed, and no part of ModelForge can
supply it — it is deliberately never written into a backup.
Run `python -m modelforge_api.cli.dr bundle --backup-id <id>` first. Everything below is listed
there: the ModelForge commit and repository, the backup identity and manifest hash, the encryption
key requirement, the host paths, the artifact recovery plan, the node enrolment requirement and the
external dependencies.
## Order
1. **Verify the backup.** See `RUNBOOK_BACKUP_FAILURE.md` if it does not verify. Do not continue
with an unverified backup; nothing downstream can compensate.
2. **Rebuild the control plane.** `RUNBOOK_CONTROL_PLANE_LOSS.md`. Exact commit, provisioned
database, supplied secrets.
3. **Restore the database.** `RUNBOOK_DATABASE_RESTORE.md`. Empty destination, eleven preflight
checks, journalled phases, `READY` gate.
4. **Reconcile.** The restore clears current truth, suppresses restored firing alerts and reports
in-flight lifecycle operations and migration cutovers by id and stage. Read that list. Lifecycle
operations are rolled back conservatively from their immutable snapshots on the first start.
Migration cutovers are *not* auto-resolved — external alias truth cannot be inferred after a
crash — and must be reconciled through the typed adapter.
5. **Recover the node.** `RUNBOOK_NODE_LOSS.md`. Until a live agent reports, the platform has node
identities but no telemetry, and that is correct.
6. **Recover artifacts.** `RUNBOOK_ARTIFACT_LOSS.md`. Rehydratable sets first; non-rehydratable sets
need their payload backup.
7. **Validate.** Health, OpenAPI, operator auth, registry, lifecycle, migration, observability,
scheduler reconciliation, capability smoke.
8. **Take a new backup immediately** and verify it. A recovered deployment has no recovery point of
its own until it does, and the dashboard will say so.
## Recovery is complete only when
```text
database validated schema at head
API healthy scheduler reconciled against re-measured truth
node status understood critical artifact coverage known
lifecycle and migration state valid audit history available
no unknown corruption a new verified backup exists
```
## What recovery must not touch
ExampleRAG's Qdrant collections and production alias, ExampleVision's recognizer and
`VISION_AUTO_ACCEPT_ENABLED`, external Ollama, Plex and Tdarr workloads, and any GPU work ModelForge
does not own. ModelForge records these as external dependencies and writes to none of them. If a
recovery procedure appears to require changing one of them, stop and escalate — it is the wrong
procedure.
## Declaring completion
Record the measured RPO and RTO with the phase breakdown, the fingerprint comparison and its
explained differences, and the artifact and node recovery evidence. An operator declares recovery
complete; the platform reports measurements and deliberately refuses to declare it for you.
docs/operations/RUNBOOK_GATEWAY_ERRORS.md
+18
View File
@@ -0,0 +1,18 @@
# Runbook: gateway errors or capability unavailable
## Trigger
`GATEWAY_ERROR_RATE`, `CAPABILITY_UNAVAILABLE`, a breached gateway SLO or fast burn.
## Diagnose
Check valid population, sample minimum, short/long burn, capability, priority and bounded failure
class. Separate authentication/rate-limit rejection from runtime, queue, scheduler and node failure.
Use correlation IDs in structured logs; metrics contain no request content.
## Act and recover
Acknowledge without marking the SLO healthy. Stop unsafe LAB traffic if it is the source; do not
change production routing, model identity or project binding without M12 authority. Confirm a later
rolling-window evaluation, normal request outcomes and explicit alert resolution. For
`rag.embedding@1`, verify the ExampleRAG Ollama baseline remains unchanged.
docs/operations/RUNBOOK_GPU_PRESSURE.md
+19
View File
@@ -0,0 +1,19 @@
# Runbook: GPU pressure
## Trigger
`GPU_PRESSURE`, `CAPABILITY_CAPACITY`, stale capacity, or sustained `HIGH`/`CRITICAL` scheduler
pressure.
## Diagnose
Compare NVML observed, ModelForge resident, leases, reserve, external and schedulable bytes. Confirm
freshness and inspect active placement/queue decisions. Ollama, Plex and Tdarr are external capacity;
ModelForge must not kill, pause or reconfigure them.
## Act and recover
Pause optional LAB/benchmark submissions through existing policy where justified. Let bounded QoS,
queueing and managed-idle eviction operate. Never manufacture an OOM for diagnosis. Resolve only
after fresh telemetry shows recovered headroom/pressure and queues drain without orphan leases.
Escalate unexplained VRAM drift, repeated crash/OOM or managed residency that does not reclaim.
docs/operations/RUNBOOK_MIGRATION_FAILURE.md
+18
View File
@@ -0,0 +1,18 @@
# Runbook: migration or rollback failure
## Trigger
`MIGRATION_FAILURE`, `ROLLBACK_FAILURE`, `FAILED` or `MANUAL_INTERVENTION_REQUIRED`.
## Diagnose
Review immutable plan/generation, adapter fingerprint, source/target identities, latest checkpoint,
validation snapshot, cutover journal, external-state observation and rollback retention. Never guess
external alias truth after a crash.
## Act and recover
Pause further plan actions and acknowledge the alert. Resume only idempotent backfill after source
and generation validation. Reconcile ambiguous cutover through the typed adapter. A rollback failure
is CRITICAL and requires explicit operator intervention; do not mutate production aliases manually
without preserving evidence. Recovery must retain the alert/timeline and prove exact final identity.
docs/operations/RUNBOOK_NODE_DECOMMISSION.md
+52
View File
@@ -0,0 +1,52 @@
# Runbook: permanently decommission a compute node
Use this only when hardware is permanently leaving ModelForge. For a temporary outage, lost
credential or rebuild, use `RUNBOOK_NODE_LOSS.md`. Never use GPU Node or another production node for a
rehearsal; enroll a disposable node identity.
## Prepare
1. Drain and retire every deployment, job, request, probe, lease and residency on the node.
2. Move any sole artifact copy and finish recovery, lifecycle and migration operations.
3. Stop the Node Agent and wait until liveness is `offline` or `stale`.
4. Record the change owner and a durable reason. Obtain the operator API key; project credentials
and node credentials are intentionally insufficient.
## Preview
In Console, open Compute Nodes, select the node and use **Preview decommission** in the danger zone.
Alternatively:
```text
POST /api/v1/admin/hardware/nodes/{node_id}/decommission/preview
X-ModelForge-Admin-Token: <operator key>
```
Resolve every returned blocker. Do not attempt to bypass a blocker in the database. Confirm the
preview names the intended persisted identity and lists both cleanup and retained history.
## Execute
Type the displayed persisted identity, hostname or display name exactly. Supply the preview's
`node_generation` and `dependency_digest`, a fresh idempotency key, operator and reason to:
```text
POST /api/v1/admin/hardware/nodes/{node_id}/decommission
```
A 409 means state changed or a blocker remains: generate a fresh preview. Repeating a completed
request is safe and returns the existing operation; it does not create another audit event.
## Confirm
- node status and liveness are `decommissioned`, all eligibility is false and current inventory is
empty;
- all old node credentials return 401 and rotation/ordinary re-enrollment are refused;
- scheduler/current telemetry rows are absent, accelerators are decommissioned and storage roots
are unavailable/read-only;
- historical inventory, jobs, probes, deployments and audit evidence remain queryable;
- exactly one `NODE_DECOMMISSIONED` audit event and one operation record exist;
- the `decommissioned_nodes_are_terminal` invariant holds.
There is no undo button. A later physical host must enroll with a new persisted identity unless a
future explicit, audited recovery lifecycle is implemented.
docs/operations/RUNBOOK_NODE_LOSS.md
+54
View File
@@ -0,0 +1,54 @@
# Runbook: node loss or credential loss
## Trigger
A compute node stops publishing, its credential is lost or believed stolen, or a node must be
rebuilt after a control-plane restore.
This runbook is for recoverable loss. If hardware is permanently leaving service, use
`RUNBOOK_NODE_DECOMMISSION.md`; do not disable or delete the row by hand.
## Diagnose
Distinguish the three failures, because they have different recoveries:
| Symptom | Meaning |
| --- | --- |
| heartbeats 401 | the credential is revoked or does not match a stored hash |
| heartbeats 403 | the credential authenticates but is not authorised for that node |
| no requests at all | the agent is down, or cannot reach the control plane |
Check `node_credentials` for the node: how many are active, how many revoked, and when the active
one was last used. Only hashes are stored; the plaintext cannot be read back from anywhere.
## Act and recover
**The node still has its credential.** Nothing to do. After a database restore the restored hash
matches and the node reconnects on its own.
**The credential is lost.** Issue a fresh enrolment token
(`POST /api/v1/admin/node-enrollments`) and restart the agent with it. The agent keeps its persisted
identity file, so it re-enrols onto the same node id with its accelerators, storage roots and
history intact. Enrolment revokes the previous credential automatically.
**The credential is believed stolen.** Revoke it first
(`DELETE /api/v1/admin/hardware/nodes/{node_id}/credential`), then re-enrol. Revocation is
permanent: the old credential stays refused after re-enrolment.
**The identity file is also lost.** The agent enrols as a new node, because ModelForge genuinely
cannot tell it apart from different hardware. If the old identity is permanently superseded, use
the audited decommission preview/execute flow rather than deleting or hand-editing it, and confirm
only one enabled node claims that hostname.
One enrolment token mints exactly one node identity. If two nodes appear from one token, the control
plane predates the M15 atomic-claim fix.
## Confirm
Exactly one enabled node for that hardware, one active credential with the old ones revoked,
heartbeats returning 200, inventory and telemetry re-collected, the scheduler reconciled against
freshly measured NVML state rather than a restored snapshot, and a capability smoke against the
recovered node.
Never revoke a production node credential merely to rehearse this. Use a disposable agent identity;
`docker-compose.node-recovery.yml` exists for exactly that.
docs/operations/RUNBOOK_NODE_OFFLINE.md
+21
View File
@@ -0,0 +1,21 @@
# Runbook: production node offline or stale
## Trigger
`NODE_OFFLINE` is pending/firing, or GPU Node freshness is `STALE`/over 90 seconds.
## Diagnose
1. Confirm API and Operations view timestamps; do not infer current health from an old snapshot.
2. Inspect Node Agent liveness, last server-received heartbeat/telemetry and credential state.
3. Check network/TLS reachability from GPU Node outward and the Node Agent service logs.
4. Review the incident for capability alerts correlated as `DOWNSTREAM`.
5. Verify ExampleRAG's external Ollama production baseline independently.
## Act and recover
Acknowledge with an operator reason. Restart the Node Agent only through the established host
runbook and only after checking unrelated workloads; never stop the GPU Node host. Do not promote a
fallback model. Recovery requires new server-received telemetry, online liveness, resolved alerts
and a retained timeline. Escalate if the credential is rejected, inventory identity changed or
freshness oscillates.
docs/operations/RUNBOOK_RUNTIME_FAILURE.md
+19
View File
@@ -0,0 +1,19 @@
# Runbook: runtime failure or crash loop
## Trigger
`RUNTIME_CRASH_LOOP`, capability-unavailable alerts, failed load/invoke/unload/health jobs or
runtime-success SLO degradation.
## Diagnose
Inspect bounded failure code, exact deployment/artifact/runtime identities, worker health, queue,
leases, residency and NVML freshness. Confirm offline execution and `trust_remote_code=false` remain
in effect. Distinguish model failure from node/GPU pressure.
## Act and recover
Do not send shell commands through ModelForge or expose the worker. Use the established isolated
runtime probe/LAB path for reproduction. Production rollback requires an approved M12 plan; M14 does
not perform it. Recovery requires successful typed health/invoke evidence, no orphan lease and an
explicit resolved alert.
docs/operations/RUNBOOK_STORAGE_LOW.md
+19
View File
@@ -0,0 +1,19 @@
# Runbook: model storage low
## Trigger
`STORAGE_LOW` fires because usable free bytes after absolute/percentage reserve are below policy or
unknown.
## Diagnose
Confirm the registered root, observed time, writable status, total/free bytes and reserve. Inspect
active artifact locations, acquisition quarantine, rollback retention and migration dependencies.
Never treat an unavailable capacity reading as healthy.
## Act and recover
Use dependency-aware M12 cleanup planning and explicit confirmation. Never issue ad-hoc recursive
deletion or remove retained rollback artifacts. Reclaim only reviewed candidates, then collect a
fresh storage snapshot and confirm the alert resolves. Escalate filesystem errors, divergent agent
paths or repeated unexplained growth.
docs/operations/RUNTIME_PROBES.md
+40
View File
@@ -0,0 +1,40 @@
# Runtime Probes
## Operator flow
1. Review the exact ArtifactSet and static assessment.
2. Record exact-set LAB execution approval.
3. Queue the typed fixed-input probe for a selected node.
4. Follow preparing/load/health/ready/unload progress.
5. Inspect technical output and measured resource evidence.
The sole M4 embedding input is `ModelForge runtime compatibility probe`. It is retained only as a
SHA-256 in runtime results. This is a technical smoke probe, not a benchmark or quality evaluation.
## Measurements
NVML samples are captured before load, after load, around inference and after child-process exit.
Stored evidence includes absolute used/free VRAM, delta above baseline, peak, GPU utilization,
temperature, power, load time, inference latency, batch and maximum sequence length. Reclamation is
checked against the measured baseline plus a bounded tolerance.
## Failure and cancellation
Failures are typed (`MODEL_LOAD_FAILED`, `GPU_OOM`, `RUNTIME_CRASH`, `TIMEOUT`, `INVALID_OUTPUT`,
`HEALTHCHECK_FAILED`, `OFFLINE_LOAD_VIOLATION`, `GPU_MEMORY_NOT_RECLAIMED`, and related gate
codes). Unexpected exception text is normalized at the child boundary. Cancellation terminates the
child, runs reclamation checks, reports `cancelled`, and never creates a candidate.
## Offline guarantee
The image sets Hugging Face/Transformers offline variables. All loads use the exact local path,
`local_files_only=true` and `trust_remote_code=false`; an in-process socket guard rejects IP network
connections while model loading and inference run. No duplicate model download is permitted.
## Logs and worker identity
Every claimed attempt receives a bounded `runtime-worker://<node>/<probe>/attempt/<n>` log
reference. The probe, node, lease attempt and child-process lifecycle together identify the worker
without exposing a general process-control API. Verbose runtime output is not stored in the probe
row; technical input is represented by its hash, credentials are redacted, and audit events remain
separate from diagnostic logs.
docs/operations/SERVER_AGENT_DEPLOYMENT.md
+115
View File
@@ -0,0 +1,115 @@
# Remote Server Agent Deployment
## Prerequisites
The remote server needs Docker Engine with Compose v2, an NVIDIA driver, NVIDIA Container Toolkit,
and outbound HTTPS access to the central ModelForge API. It needs no inbound SSH path from
ModelForge, no public agent port, no Docker socket mount and no privileged container.
The container deliberately uses a persisted UUID because host `/etc/machine-id` is not mounted.
Hostname/OS/CPU/RAM describe the agent's Linux execution environment; without container resource
limits this normally represents the server, while configured CPU/memory limits represent the
capacity actually available to future workloads. Storage facts cover only the named ModelForge data
volumes. This avoids unsafe host-root mounts and keeps the semantics operationally honest.
The Compose service drops all Linux capabilities, enables `no-new-privileges`, uses a read-only root
filesystem and opens only a bounded in-memory `/tmp`. It publishes no port. Set
`MODELFORGE_AGENT_HOSTNAME` to the real compute-node hostname. On Unraid or another appliance, set
the four `MODELFORGE_AGENT_*_VOLUME` variables to narrow, dedicated absolute bind paths when state
must survive Docker image/volume recreation; never point one at the host root.
Expose the control-plane API through a trusted HTTPS reverse proxy. Use an internal CA by installing
that CA in the agent image/host trust store; do not disable TLS verification outside an isolated
local smoke test.
For a private CA that should be trusted only by ModelForge, combine the base file with
`docker-compose.node-agent.private-ca.yml`. Set `MODELFORGE_AGENT_CA_CERT_PATH` to the public CA
certificate, plus `MODELFORGE_AGENT_CONTROL_PLANE_HOST_ALIAS` and
`MODELFORGE_AGENT_CONTROL_PLANE_HOST_ADDRESS` when private DNS is unavailable. The overlay mounts
only that public certificate read-only and scopes the host mapping to the agent container.
## 1. Configure the control plane
Set a high-entropy `MODELFORGE_OPERATOR_API_KEY`, keep `MODELFORGE_HARDWARE_REFRESH_ON_STARTUP=false`,
and start the base stack. Create an enrollment token from **Compute nodes → Enroll node**, or:
```bash
curl -sS -X POST https://modelforge.example/api/v1/admin/node-enrollments \
-H "X-ModelForge-Admin-Token: $MODELFORGE_OPERATOR_API_KEY" \
-H 'Content-Type: application/json' \
-d '{"expires_in_seconds":900,"display_name":"server01","role":"primary-inference","labels":{"site":"office"},"production_eligible":false,"lab_eligible":true,"benchmark_eligible":true}'
```
The response shows the enrollment secret once. Leave `production_eligible` false until the actual
node has passed hardware, identity, recovery and security acceptance. Eligibility is scheduler
metadata; setting it true does not launch a runtime or bypass later promotion evidence.
## 2. Start the agent on the GPU server
Copy/clone this repository to the server. Create `.env` without committing it:
```dotenv
MODELFORGE_AGENT_CONTROL_PLANE_URL=https://modelforge.example
MODELFORGE_AGENT_ENROLLMENT_TOKEN=<one-time-token>
MODELFORGE_AGENT_DISPLAY_NAME=Server 01
MODELFORGE_AGENT_HOSTNAME=server01
MODELFORGE_AGENT_TLS_VERIFY=true
MODELFORGE_AGENT_STATE_VOLUME=/mnt/user/appdata/modelforge-node-agent/state
MODELFORGE_AGENT_HF_CACHE_VOLUME=/mnt/cache/appdata/modelforge-node-agent/hf-cache
MODELFORGE_AGENT_ARTIFACT_VOLUME=/mnt/cache/appdata/modelforge-node-agent/artifacts
MODELFORGE_AGENT_QUARANTINE_VOLUME=/mnt/cache/appdata/modelforge-node-agent/quarantine
MODELFORGE_HF_TOKEN=<optional-secret-for-gated-repositories>
```
Then run:
```bash
docker compose -f docker-compose.node-agent.yml up -d --build
docker compose -f docker-compose.node-agent.yml logs --tail=100 node-agent
```
With the private-CA overlay, use both files for every lifecycle command:
```bash
docker compose -f docker-compose.node-agent.yml \
-f docker-compose.node-agent.private-ca.yml up -d --build
```
After the node appears online, remove `MODELFORGE_AGENT_ENROLLMENT_TOKEN` from `.env` and recreate
the service. The node-scoped credential persists in `node-agent-state`:
```bash
docker compose -f docker-compose.node-agent.yml up -d --force-recreate node-agent
```
Do not delete the state volume during ordinary upgrades; doing so discards identity and credential.
Use the UI/API to revoke the old credential before deliberately re-enrolling.
For M3, register the host cache path as a storage root and map it to
`/data/artifacts/model-registry`. The host path and container path are deliberately separate facts.
Never substitute `/mnt/user` when the cache capacity guard fails. The agent advertises
`artifact.acquire.v1`, polls one typed job at a time, and retains job-scoped partials below the
approved root for bounded retry. Keep the optional Hub token only in secret environment/config;
do not place it in Compose files or logs.
## 3. Verify and operate
Confirm the UI shows the expected hostname, agent/protocol version, heartbeat age, CPU/RAM/storage,
GPU UUID/model/VRAM/driver and measured telemetry. Restart the agent and confirm node and accelerator
UUIDs remain unchanged. Stop it long enough to observe `stale` then `offline`, restart it and confirm
`online` plus a return audit event.
For temporary outages the agent retains its identity/credential and retries with bounded exponential
backoff. Rotate with the authenticated admin rotation endpoint, securely replace
`/data/state/node-credential` with its one-time response, and recreate the agent. Revoke without a
replacement when a credential may be compromised. Disable a node to make it ineligible and immediately
reject further publications. Upgrades are ordinary image
rebuild/recreate operations; protocol-incompatible agents are visibly rejected instead of silently
downgraded.
## RTX 4080 acceptance checklist
On the actual target server, capture: `nvidia-smi` model/UUID/driver output, node-agent container
status, `/api/v1/hardware/nodes/{id}` output with secrets removed, a restart identity comparison,
stale/offline/recovery evidence, and UI screenshots. The release verdict may not claim RTX 4080
validation until these observations exist.
docs/operations/SOAK_TESTING.md
+101
View File
@@ -0,0 +1,101 @@
# Soak testing
## What a soak is for
A short run proves a path works. A soak proves it keeps working: that memory does not climb, that
leases are released, that queues drain, that retention keeps row growth bounded, and that latency
does not degrade as caches warm and evict.
```sh
python scripts/m16_soak.py --database-url "$MODELFORGE_RUNTIME_DATABASE_URL" --minutes 45 --secret <gateway-secret> --report soak.json
```
The database URL is mandatory and must use the non-owner runtime role. The harness contains no
embedded credential or owner-role fallback.
## Workload shape
Three gateway workers drive the real capability path — `rag.embedding@1` through the authenticated
gateway to GPU Node — mixing three phases:
```text
burst 3-6 requests back to back, 50-400 ms apart
steady single requests, 1.5-5 s apart
idle 8-25 s of nothing
```
The mix matters. A soak that only hammers never exercises keep-warm and unload, so it never
observes a cold start. A soak that only idles never exercises admission control, so it never
observes a capacity rejection. Batch sizes vary from one to four inputs so payload handling varies
too.
Two read workers poll registry, projects, hardware, health, operations, alerts, lifecycle,
migrations and recovery. One background worker drives the operational work the platform already
performs for itself — capacity collection, SLO evaluation, alert evaluation — at a bounded rate.
## What is measured
```text
per kind requests, succeeded, capacity rejected, internal errors, status distribution
latency p50, p95, p99, mean, max
gateway cold starts, warm invocations, queue p95, inference p50 and p95, failure codes
resources container memory, CPU, PIDs and restart counts, before and after
database row growth per table, connection count
runtime active leases, serving jobs in flight, residency allocations
GPU used, free and utilisation per accelerator, plus scheduler pressure
invariants the full set, at every progress interval
```
Capacity rejections are counted separately from internal errors. A 429 or a `QUEUE_FULL` is the
scheduler refusing unsafe work, which is the platform behaving correctly; counting it as a failure
would reward a system that over-admits.
## Stopping early
The soak aborts and reports partial evidence if an invariant is violated at any progress interval.
Partial evidence honestly labelled is worth more than a full run whose result is already known to
be invalid.
It should also be aborted manually on uncontrolled host pressure, degradation of an external
workload, repeated OOM, data corruption or any unsafe production mutation.
## Reading the result
```text
queue drained queued, running and active leases all zero at the end,
unless a resident is intentionally kept warm
memory a flat or sawtooth profile is healthy; monotonic growth is investigated
row growth bounded by retention and aggregation, not linear in request count
restart counts unchanged, unless a restart was deliberately injected
```
## Honesty rules
Report the duration actually achieved. A 45-minute soak is a 45-minute soak; it is not evidence
about a week.
If a chaos scenario or an infrastructure change overlapped the run, say so and say which numbers it
affected, rather than presenting the mixed result as a clean baseline. The M16 evidence includes one
soak that was discarded for exactly this reason after container hardening restarted its datastores
mid-run.
The disposable gateway client the soak uses is revoked afterwards, and the fixtures it created are
removed.
## Measuring latency honestly
The soak reports percentiles, so the harness must first prove that it is not the thing being
measured. Before any worker starts it times a plain TCP connect to the target and refuses to run if
the median exceeds `MAX_CONNECT_FLOOR_MS` (100 ms), printing the floor it measured and recording it
in the report as `client_connect_floor_ms`.
This exists because the first M16 soak reported a p50 near two seconds on every route including
`/api/v1/health/live`, with only a few milliseconds of spread. A constant with no tail is never queueing.
The cause was the harness addressing the API as `localhost`, which resolves to `::1` first on this
host; each request waited out the IPv6 connect timeout before falling back to IPv4, adding a fixed
~2,045 ms of client cost to every sample. Direct measurement, both returning 200: 2,043.3 ms median
to `http://localhost:8000/api/v1/health/live` against 4.7 ms to
`http://127.0.0.1:8000/api/v1/health/live`.
Address the platform by an address that resolves directly. Both M16 harnesses default to
`http://127.0.0.1:8000` for this reason, not to `localhost`.