Jens/ModelForge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial public ModelForge release

Browse Source
This commit is contained in:
Jens
2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CAPABILITIES.md
+83
View File
@@ -0,0 +1,83 @@
# Capabilities
A capability is the stable interface a project binds to. `rag.embedding@1` is a promise about input
and output shape and about semantics — not about which model, node or file satisfies it.
## Why this exists
Bind an application to a model and you have coupled it to a decision you will want to change: a
better model, a different quantisation, a different node. Every one of those becomes a
coordinated change across every consumer.
Bind it to a capability and the substitution is ModelForge's problem. That is why
[SYSTEM_ARCHITECTURE.md](architecture/SYSTEM_ARCHITECTURE.md) establishes that applications bind to capabilities and never to model
paths, and why nothing in the platform couples project code to a Hugging Face repository name.
## The pieces
| Concept | What it is |
| --- | --- |
| **Capability** | A named kind of work: `rag.embedding`, `document.ocr`, `vision.embedding` |
| **Capability contract** | A version of that capability with a fixed input and output schema |
| **Capability deployment** | A specific artifact set and runtime profile serving a contract |
| **Project binding** | A project's declaration that it uses a contract, on a channel |
| **Service client** | An application's credential, scoped to specific capabilities |
A contract has at most one `stable` deployment at a time. That is enforced by a partial unique index
in the database *and* asserted by a platform invariant — two layers, because the invariant is there
for the case where the guard has been bypassed.
## Contracts shipped with v1.0.0
| Capability | Version |
| --- | --- |
| `rag.embedding` | 1 |
| `rag.reranking` | 1 |
| `document.ocr` | 1 |
| `vision.embedding` | 1 |
| `speech.transcription` | 1 |
| `assistant.general` | 1 |
| `assistant.vision` | 1 |
A contract being defined in a manifest does **not** mean anything is serving it. A deployment
requires a model artifact acquired, verified and promoted through the lifecycle. A newly installed
control plane has contracts available to register and no deployments — see
[FIRST_RUN.md](FIRST_RUN.md).
## Channels
`stable`, `experimental` and the production flag are separate ideas. A project can bind to
`experimental` for a capability while another binds to `stable` for the same one, and a deployment
being production is an explicit, approved, audited state — `no_hidden_auto_promotion` asserts that
no production deployment exists without a recorded production approval.
## Upgrade classes
Every contract declares what a change to it means:
| Class | Meaning |
| --- | --- |
| compatible | The replacement can serve existing consumers unchanged |
| `requires_reindex` | The embedding space changed; existing vectors are not comparable |
`requires_reindex` is the one that cannot be waved through. A project binding must declare that it
supports reindex migration before it can bind such a contract, and the migration engine will not
swap an alias underneath data that has not been reindexed.
## Invoking
See [PROJECT_INTEGRATION.md](PROJECT_INTEGRATION.md).
## What a capability guarantees
- the input and output schema for its version;
- that only an approved, verified artifact is behind it;
- that a refusal is named — `NO_ELIGIBLE_NODE`, `CAPACITY_CONSTRAINED`, `CAPABILITY_NOT_AUTHORIZED` —
rather than an answer built on something stale.
## What it does not guarantee
- which model answered, or that the same model answers next time;
- that a deployment exists at all, if nothing has been promoted;
- latency: that depends on the node, the model and current pressure. The gateway reports queue and
inference time per request so you can see which is which.
docs/COMPATIBILITY.md
+114
View File
@@ -0,0 +1,114 @@
# Compatibility matrix — v1.2.1 plus unreleased RC hardening
Nothing here is implied. The executable contract lives in
`backend/src/modelforge_api/domain/release.py`; startup and release tests enforce it.
> RC development note: the unreleased product branch now targets additive schema
> `20260830_0024`, accepts `20260828_0022` and auth-only `20260830_0023` as upgrade sources, and refuses to serve until the
> audit-chain migration completes. Deployed stable v1.2.1 remains frozen on `20260828_0022`; the
> released matrix below continues to describe that production artifact until the RC is tagged.
> Selecting and stamping the post-0024 product version is an explicit pending release tranche; the
> temporary repository `1.2.1` constant must not be presented as a released schema-0024 artifact.
## Product and database
| Contract | Value |
| --- | --- |
| Version | `1.2.1` |
| Channel | `stable` |
| Version source | repository `VERSION` |
| Runtime schema | `20260830_0023` |
| Direct upgrade-source schema | `20260827_0021`, `20260828_0022` |
| Minimum direct upgrade | `v1.0.0` |
| PostgreSQL | 16+; Compose ships 17 |
| SQLite | unit tests only; **NOT SUPPORTED FOR PRODUCTION** |
The hardened candidate serves only on schema 0023. The upgrade tool accepts 0021 or the production
0022 head, applies the intervening revisions, validates all existing node credential scopes, and
reaches 0023 before the API starts. PostgreSQL is the only production database contract; SQLite is
used only to exercise DDL mechanics in isolated tests.
Compatibility answers remain typed and fail closed:
| Answer | Meaning | Operator action |
| --- | --- | --- |
| `COMPATIBLE` | This component understands the observed version | none |
| `TOO_OLD` | The observed version predates the supported range | upgrade it |
| `TOO_NEW` | The observed version belongs to a newer release | upgrade this component |
| `UNKNOWN` | Identity could not be established | investigate; it is refused |
## Agent protocol
| Contract | Value |
| --- | --- |
| Current protocol | 1 |
| Accepted protocols | 1 |
The Node Agent and Runtime Worker remain protocol-compatible with v1.0.0. Release images should
still be upgraded together so their product and provenance identity is v1.2.1.
## Upgrade matrix
| From | Schema | To | Supported |
| --- | --- | --- | --- |
| `v1.0.0` | `20260827_0021` | current candidate / `20260830_0023` | yes; backup and rehearsal required |
| `v1.1.0` | `20260828_0022` | current candidate / `20260830_0023` | yes; scope validation migration |
| `v1.1.1` | `20260828_0022` | current candidate / `20260830_0023` | yes; scope validation migration |
| `v1.2.0` | `20260828_0022` | current candidate / `20260830_0023` | yes; scope validation migration |
| `v1.2.1` | `20260828_0022` | current candidate / `20260830_0023` | yes; scope validation migration |
| current candidate | `20260830_0023` | current candidate | yes; already current |
| older than `v1.0.0` | any | `v1.2.1` | no direct upgrade |
Because 0022 introduces terminal node and decommission-operation state, application-only rollback
to v1.0.0 is unsupported. Restore the verified pre-upgrade backup. Alembic downgrade may be used in
isolated verification to prove DDL mechanics, but it is not the production recovery promise.
## Runtime dependencies
| Component | Version |
| --- | --- |
| PostgreSQL | >= 16 |
| Redis | >= 7 |
| Docker Engine | >= 24 |
| Docker Compose | >= 2 |
| Python host tooling | >= 3.12 |
| NVIDIA container runtime | required on GPU nodes |
The Node Agent image requires a glibc-compatible NVIDIA Container Toolkit/driver injection path.
GPU Compose projections explicitly fail closed when NVML inventory or telemetry is unavailable.
CPU-only agents do not require NVML.
## Why this is a MINOR release
Node Decommission adds operator APIs, a Console workflow and an additive schema revision while
preserving existing capability routes and agent protocol 1. That is a backward-compatible feature
addition and therefore v1.1.0 under SemVer, not v1.0.1 and not v2.0.0.
## Why v1.2.1 is a PATCH release
v1.2.1 changes no contract, no schema, no protocol and no behaviour an application can observe. It
corrects how release artifacts are built and stamped: the console's API origin becomes a required
release input rather than an accidental default, and the Node Agent projection stamps the release
identity it was already expected to carry. Correcting packaging without changing the product is a
PATCH under SemVer.
The console image is the one artifact whose *behaviour* differs, because the previous images could
not reach a non-localhost API at all. That is a defect being repaired, not a capability being added.
## Why v1.2.0 is a MINOR release
v1.2.0 adds substantive, backward-compatible product capability at the console layer: a command
center, a domain-oriented information architecture, a design system, a responsive mobile console and
a consistent guarded-action and accessibility vocabulary. No API contract, domain model, runtime
behavior or schema changed, and no existing route was removed or moved. Adding capability without
breaking a contract is a MINOR release under SemVer — not a patch, and not a major.
The minimum direct upgrade stays `v1.0.0`. v1.2.0 changes nothing about the upgrade surface, so
narrowing the supported source range would withdraw a working, tested path for no engineering
reason.
## Why v1.1.1 is a PATCH release
v1.1.1 changes only the Node Agent runtime packaging and local startup validation needed to restore
the existing NVIDIA-node contract. It adds no API, database migration, agent protocol field,
scheduler contract, or operator capability.
docs/CONFIGURATION.md
+175
View File
@@ -0,0 +1,175 @@
# Configuration reference
Generated from the typed settings by `scripts/generate_configuration_docs.py` for ITWorx ModelForge 1.2.1. Every setting the control plane reads appears here; a setting added without documentation fails the build.
All settings are environment variables with the `MODELFORGE_` prefix, read from the process environment or from `.env`.
## Required in production
`MODELFORGE_ENV=production` turns on fail-closed startup validation. With it set, the control plane refuses to start unless each of these is present and sound:
- `MODELFORGE_DATABASE_URL` — SQLAlchemy URL for the API's non-owner modelforge_runtime role. It may read the audit trail and execute the canonical append function, but cannot mutate audit tables directly.
- `MODELFORGE_REDIS_URL` — Redis URL for transient request payloads and queues.
- `MODELFORGE_OPERATOR_API_KEY` — Operator API key guarding every admin route. Generate at least 32 random characters; ModelForge never mints one for you.
- `MODELFORGE_BACKUP_ENCRYPTION_KEY` — Base64 AES-256 key for backup encryption. Without it no backup can be produced, and without the same key no backup can be restored — store it outside this deployment.
Production additionally refuses: a well-known development database password, a wildcard CORS origin, remote model code execution, an unwritable storage root, an unsupported schema revision, a PostgreSQL major below 16, and three policy combinations that cannot all hold at once.
## Generating secrets
ModelForge never mints its own credentials — a platform that generates its own admin secret has no way to tell you it did. Generate them yourself and store them outside the deployment:
```bash
# Operator API key (at least 32 characters)
python -c "import secrets; print(secrets.token_urlsafe(48))"
# Backup encryption key (base64 AES-256)
python -c "import base64, os; print(base64.b64encode(os.urandom(32)).decode())"
# Database password
python -c "import secrets; print(secrets.token_urlsafe(32))"
```
Losing the backup encryption key makes every existing backup unrecoverable. It is the one value that must be stored somewhere the deployment cannot take down with it.
## Sensitivity
5 settings are credentials. They are never logged, never written to a release artefact and never echoed in an error response:
- `MODELFORGE_DATABASE_URL`
- `MODELFORGE_MIGRATION_DATABASE_URL`
- `MODELFORGE_OPERATOR_API_KEY`
- `MODELFORGE_HF_TOKEN`
- `MODELFORGE_BACKUP_ENCRYPTION_KEY`
## Every setting
| Variable | Type | Default | Required | Sensitivity | Description |
| --- | --- | --- | --- | --- | --- |
| `MODELFORGE_ENV` | `development`, `test`, `production` | `development` | no | public | Deployment profile. 'production' turns on every fail-closed startup rule; 'development' and 'test' report the same problems without refusing to start. |
| `MODELFORGE_API_HOST` | string | `0.0.0.0` | no | public | Interface the API binds inside its container. Leave at 0.0.0.0. |
| `MODELFORGE_API_PORT` | integer | `8000` | no | public | Port the API listens on inside its container. |
| `MODELFORGE_CONTROL_PLANE_MAX_PAYLOAD_BYTES` | integer | `1048576` | no | public | Pre-parser request-body limit for public and operator control-plane routes. |
| `MODELFORGE_SERVICE_NAME` | string | `modelforge-api` | no | public | Name this process reports in logs and audit events. |
| `MODELFORGE_LOG_LEVEL` | string | `INFO` | no | public | Structured log level: DEBUG, INFO, WARNING or ERROR. |
| `MODELFORGE_DATABASE_URL` | string | — | yes | secret | SQLAlchemy URL for the API's non-owner modelforge_runtime role. It may read the audit trail and execute the canonical append function, but cannot mutate audit tables directly. |
| `MODELFORGE_MIGRATION_DATABASE_URL` | secret, optional | — | no | secret | SQLAlchemy URL for the non-superuser modelforge schema-owner role. Set only in the one-shot migration process; production API startup refuses when this secret is present. |
| `MODELFORGE_REDIS_URL` | string | `redis://localhost:6379/0` | yes | internal | Redis URL for transient request payloads and queues. |
| `MODELFORGE_CORS_ORIGINS` | string | `http://localhost:3000` | no | public | Comma-separated exact origins allowed to call the API from a browser. A wildcard is refused in production because requests are credentialed. |
| `MODELFORGE_OPERATOR_API_KEY` | secret, optional | — | yes | secret | Operator API key guarding every admin route. Generate at least 32 random characters; ModelForge never mints one for you. |
| `MODELFORGE_HF_TOKEN` | secret, optional | — | no | secret | Optional Hugging Face token, used only for acquiring gated repositories. It is never passed to a runtime and never leaves the control plane. |
| `MODELFORGE_BACKUP_ENCRYPTION_KEY` | secret, optional | — | yes | secret | Base64 AES-256 key for backup encryption. Without it no backup can be produced, and without the same key no backup can be restored — store it outside this deployment. |
| `MODELFORGE_BACKUP_ENCRYPTION_KEY_ID` | string | `modelforge-backup-key-1` | no | public | Identifier recorded in each backup manifest so a restore can name the key it needs. |
| `MODELFORGE_HF_HOME` | string | `/data/hf-cache` | no | public | Hugging Face cache root inside the container. |
| `MODELFORGE_ARTIFACT_ROOT` | string | `/data/artifacts` | no | public | Verified model artifact root. Must exist and be writable. |
| `MODELFORGE_QUARANTINE_ROOT` | string | `/data/quarantine` | no | public | Where acquired artifacts are held until their checks pass. |
| `MODELFORGE_RUNTIME_ARTIFACT_ROOT` | string | `/models/model-registry` | no | public | Artifact root as a runtime worker sees it on a compute node. |
| `MODELFORGE_CONFIG_ROOT` | path | `PydanticUndefined` | no | public | Directory holding the capability, project and policy manifests. |
| `MODELFORGE_BACKUP_ROOT` | path | `/data/backups` | no | public | Backup destination. Must exist and be writable, or backups fail closed. |
| `MODELFORGE_BACKUP_RESTORE_ROOT` | path | `/data/restore` | no | public | Working directory a restore stages into before it commits. |
| `MODELFORGE_ALEMBIC_DIRECTORY` | path, optional | — | no | public | Override for the migration directory. Leave empty in a container. |
| `MODELFORGE_HF_TIMEOUT_SECONDS` | number | `30` | no | public | Per-request timeout for Hugging Face metadata calls. |
| `MODELFORGE_HF_SNAPSHOT_TTL_SECONDS` | integer | `3600` | no | public | How long a resolved upstream snapshot stays cached. |
| `MODELFORGE_ALLOW_REMOTE_CODE` | boolean | `false` | no | public | Whether model repositories may execute their own Python. Always false in production; startup refuses any other value there. |
| `MODELFORGE_ENABLE_GPU_TELEMETRY` | boolean | `true` | no | public | Collect GPU telemetry on this host. |
| `MODELFORGE_HARDWARE_REFRESH_ON_STARTUP` | boolean | `false` | no | public | Run a hardware inventory pass when the process starts. |
| `MODELFORGE_HARDWARE_POLL_INTERVAL_SECONDS` | integer | `30` | no | public | Interval between hardware inventory passes. |
| `MODELFORGE_NODE_IDENTITY` | string, optional | — | no | internal | Explicit node identity. Leave empty to use the persisted file. |
| `MODELFORGE_NODE_IDENTITY_MODE` | `auto`, `persisted` | `auto` | no | public | 'persisted' keeps a node's identity across restarts; 'auto' derives it. |
| `MODELFORGE_NODE_IDENTITY_FILE` | path | `/data/state/node-id` | no | public | Where a persisted node identity is stored. |
| `MODELFORGE_NODE_STALE_AFTER_SECONDS` | integer | `30` | no | public | Silence after which a node is considered stale. |
| `MODELFORGE_NODE_OFFLINE_AFTER_SECONDS` | integer | `90` | no | public | Silence after which a node is considered offline. Must exceed the stale threshold. |
| `MODELFORGE_LIVENESS_POLL_INTERVAL_SECONDS` | integer | `5` | no | public | How often node liveness is re-evaluated. |
| `MODELFORGE_AGENT_MAX_CLOCK_SKEW_SECONDS` | integer | `300` | no | public | Clock skew tolerated on an agent report before refusal. |
| `MODELFORGE_NODE_AGENT_MAX_PAYLOAD_BYTES` | integer | `4194304` | no | public | Pre-parser request-body limit for enrollment and authenticated Node Agent reports. |
| `MODELFORGE_NODE_LIVENESS_MONITOR_ENABLED` | boolean | `false` | no | public | Run the node liveness monitor in this process. |
| `MODELFORGE_AGENT_PROTOCOL_VERSION` | integer | `1` | no | public | Agent protocol version this control plane speaks. |
| `MODELFORGE_GATEWAY_MAX_BATCH_SIZE` | integer | `8` | no | public | Maximum inputs accepted in a single capability invocation. |
| `MODELFORGE_GATEWAY_MAX_INPUT_CHARACTERS` | integer | `8192` | no | public | Maximum characters per input item. |
| `MODELFORGE_GATEWAY_MAX_PAYLOAD_BYTES` | integer | `65536` | no | public | Maximum accepted request body size. |
| `MODELFORGE_GATEWAY_REQUEST_TIMEOUT_SECONDS` | integer | `45` | no | public | Total time a capability invocation may take. Must exceed the queue timeout. |
| `MODELFORGE_GATEWAY_QUEUE_TIMEOUT_SECONDS` | integer | `30` | no | public | How long a request may wait for capacity before rejection. |
| `MODELFORGE_SERVING_JOB_LEASE_SECONDS` | integer | `120` | no | public | Lease held by a serving job before it is reclaimed. |
| `MODELFORGE_SERVING_PAYLOAD_TTL_SECONDS` | integer | `120` | no | public | How long a request payload survives in Redis. |
| `MODELFORGE_SCHEDULER_SAFETY_RESERVE_BYTES` | integer | `1073741824` | no | public | VRAM never offered to a placement, as an absolute floor. |
| `MODELFORGE_SCHEDULER_SAFETY_RESERVE_PERCENTAGE` | number | `0.05` | no | public | VRAM never offered to a placement, as a fraction. Half a device leaves nothing schedulable. |
| `MODELFORGE_SCHEDULER_RUNTIME_MARGIN_BYTES` | integer | `268435456` | no | public | Headroom reserved for runtime overhead per node. |
| `MODELFORGE_SCHEDULER_DEPLOYMENT_MARGIN_BYTES` | integer | `134217728` | no | public | Absolute headroom added to each deployment estimate. |
| `MODELFORGE_SCHEDULER_DEPLOYMENT_MARGIN_PERCENTAGE` | number | `0.1` | no | public | Proportional headroom added to each estimate. |
| `MODELFORGE_SCHEDULER_GLOBAL_QUEUE_LIMIT` | integer | `128` | no | public | Queued requests accepted before capacity rejection begins. |
| `MODELFORGE_SCHEDULER_TELEMETRY_STALE_SECONDS` | integer | `90` | no | public | Telemetry age past which admission is blocked rather than extrapolated. |
| `MODELFORGE_SCHEDULER_PRESSURE_STABLE_SECONDS` | integer | `30` | no | public | How long pressure must hold before the state changes. |
| `MODELFORGE_SCHEDULER_EVICTION_COOLDOWN_SECONDS` | integer | `60` | no | public | Minimum interval between evictions on a node. |
| `MODELFORGE_SCHEDULER_PLACEMENT_HISTORY_LIMIT` | integer | `500` | no | public | Placement decisions retained for inspection. |
| `MODELFORGE_REGISTRY_SEED_ON_STARTUP` | boolean | `false` | no | public | Seed the candidate and project registries from manifests. |
| `MODELFORGE_SERVING_RECONCILIATION_ENABLED` | boolean | `false` | no | public | Reconcile abandoned serving work in this process. |
| `MODELFORGE_SERVING_RECONCILIATION_INTERVAL_SECONDS` | integer | `5` | no | public | Interval between serving reconciliation passes. |
| `MODELFORGE_LIFECYCLE_RECONCILIATION_ENABLED` | boolean | `false` | no | public | Roll back incomplete lifecycle operations at startup. |
| `MODELFORGE_MIGRATION_RECONCILIATION_ENABLED` | boolean | `false` | no | public | Report interrupted migration cutovers at startup. They are never auto-resolved: external alias truth cannot be inferred after a crash. |
| `MODELFORGE_OBSERVABILITY_MONITOR_ENABLED` | boolean | `false` | no | public | Run SLO and alert evaluation in this process. |
| `MODELFORGE_OBSERVABILITY_POLL_INTERVAL_SECONDS` | integer | `60` | no | public | Interval between observability evaluation passes. |
| `MODELFORGE_RECOVERY_RECONCILIATION_ENABLED` | boolean | `false` | no | public | Reconcile interrupted backups and restores at startup. |
| `MODELFORGE_BACKUP_PG_DUMP_PATH` | string | `pg_dump` | no | public | pg_dump executable. Must match the server major version. |
| `MODELFORGE_BACKUP_PG_RESTORE_PATH` | string | `pg_restore` | no | public | pg_restore executable. |
| `MODELFORGE_BACKUP_PSQL_PATH` | string | `psql` | no | public | psql executable. |
| `MODELFORGE_BACKUP_COMMAND_TIMEOUT_SECONDS` | integer | `1800` | no | public | Timeout for a dump or restore command. |
| `MODELFORGE_BACKUP_STALE_AFTER_SECONDS` | integer | `93600` | no | public | Age past which the newest verified backup raises BACKUP_STALE. |
| `MODELFORGE_BACKUP_MINIMUM_FREE_BYTES` | integer | `1073741824` | no | public | Free space below which a backup refuses to start. |
| `MODELFORGE_BACKUP_CAPACITY_HEADROOM_RATIO` | number | `3.0` | no | public | Required free space as a multiple of the estimated size. |
| `MODELFORGE_RESTORE_ALLOW_PRODUCTION_TARGET` | boolean | `false` | no | public | Whether a restore may overwrite the live database. Keep false outside a rehearsal. |
| `MODELFORGE_BUILD_COMMIT` | string, optional | — | no | public | Source commit stamped into the image at build time. Never set by hand. |
| `MODELFORGE_BUILD_TIMESTAMP` | string, optional | — | no | public | Build time stamped into the image. Never set by hand. |
| `MODELFORGE_BUILD_IMAGE_DIGEST` | string, optional | — | no | public | Image digest recorded at deployment. Never set by hand. |
## Deployment variables
Read by Compose, the Node Agent and the Runtime Worker rather than by the control-plane process. An operator still has to set them, so they are documented here too.
| Variable | Required | Sensitivity | Description |
| --- | --- | --- | --- |
| `MODELFORGE_POSTGRES_BIND` | no | public | Host address the control-plane database is published on. Defaults to 127.0.0.1; publishing it more widely exposes provenance, credential hashes and the audit trail. |
| `MODELFORGE_REDIS_BIND` | no | public | Host address Redis is published on. Defaults to 127.0.0.1. |
| `MODELFORGE_API_BIND` | no | public | Host address the API is published on. Defaults to 0.0.0.0 deliberately: the console and compute nodes need it, and every admin route is operator-authenticated. |
| `MODELFORGE_WEB_BIND` | no | public | Host address the console is published on. Defaults to 127.0.0.1. |
| `MODELFORGE_POSTGRES_PORT` | no | public | Host port the database is published on. Defaults to 5432. |
| `MODELFORGE_REDIS_PORT` | no | public | Host port Redis is published on. Defaults to 6379. |
| `MODELFORGE_API_PUBLISHED_PORT` | no | public | Host port the API is published on. Defaults to 8000. |
| `MODELFORGE_WEB_PORT` | no | public | Host port the console is published on. Defaults to 3000. |
| `MODELFORGE_DR_POSTGRES_BIND` | no | public | Host address for the DR rehearsal database. Loopback only. |
| `MODELFORGE_DR_API_BIND` | no | public | Host address for the DR rehearsal API. Loopback only. |
| `VITE_API_BASE_URL` | no | public | API base URL compiled into the console. Vite inlines it at build time, so changing it requires rebuilding the console image, not restarting it. |
| `MODELFORGE_POSTGRES_DB` | no | public | Production database name. Required by the production overlay. |
| `MODELFORGE_POSTGRES_ADMIN_USER` | no | internal | Bootstrap/admin role used only by PostgreSQL provisioning; defaults to postgres. |
| `MODELFORGE_POSTGRES_ADMIN_PASSWORD` | yes | secret | Bootstrap/admin password; never passed to the migration or API container. |
| `MODELFORGE_MIGRATION_DB_PASSWORD` | yes | secret | Raw password supplied to provisioning for the non-superuser modelforge owner role. |
| `MODELFORGE_RUNTIME_DB_PASSWORD` | yes | secret | Raw password supplied to provisioning for the non-owner modelforge_runtime role. |
| `MODELFORGE_RUNTIME_DATABASE_URL` | yes | secret | Non-owner runtime-role URL passed only to the API container. |
| `MODELFORGE_VERSION` | no | public | Exact version tag applied to built images and required when the production overlay is not given explicit API and web image references. |
| `MODELFORGE_COMMIT` | no | public | Source commit stamped into images at build time. |
| `MODELFORGE_BUILT_AT` | no | public | Build timestamp stamped into images. |
| `MODELFORGE_API_IMAGE` | no | public | Exact tag or digest the production overlay runs for the API; never use latest. |
| `MODELFORGE_WEB_IMAGE` | no | public | Exact tag or digest the production overlay runs for the console; never use latest. |
| `MODELFORGE_NODE_AGENT_IMAGE` | no | public | Exact release tag or digest for the standalone Node Agent. The local-build fallback is named local and never resolves to latest. |
| `MODELFORGE_API_IMAGE_DIGEST` | no | public | Digest recorded as the running API build identity. |
| `MODELFORGE_BACKUP_VOLUME` | no | public | Volume or bind path backing the backup root. |
| `MODELFORGE_RESTORE_VOLUME` | no | public | Volume or bind path backing the restore staging root. |
| `MODELFORGE_AGENT_STATE_VOLUME` | no | public | Volume or bind path holding the agent's persisted identity. |
| `MODELFORGE_AGENT_HF_CACHE_VOLUME` | no | public | Volume or bind path for the agent's Hugging Face cache. |
| `MODELFORGE_AGENT_ARTIFACT_VOLUME` | no | public | Volume or bind path for verified artifacts on a node. |
| `MODELFORGE_AGENT_QUARANTINE_VOLUME` | no | public | Volume or bind path for the node's quarantine area. |
| `MODELFORGE_AGENT_CONTROL_PLANE_URL` | no | public | URL the agent reports to. Outbound only; the control plane never dials a node. |
| `MODELFORGE_AGENT_ENROLLMENT_TOKEN` | no | secret | Single-use enrolment token. Consumed atomically: a storm against one token produces exactly one identity. |
| `MODELFORGE_AGENT_HOSTNAME` | no | public | Hostname the agent enrols under. |
| `MODELFORGE_AGENT_ACCELERATOR_MODE` | no | public | Accelerator contract: nvidia fails closed unless NVML inventory and telemetry are valid; cpu permits a legitimate CPU-only node; auto requires NVIDIA when injected devices are observed. Canonical GPU Compose deployments set nvidia explicitly. |
| `MODELFORGE_AGENT_TLS_VERIFY` | no | public | Whether the agent verifies the control plane's certificate. True wherever TLS is real. |
| `MODELFORGE_AGENT_CONTROL_PLANE_HOST_ADDRESS` | no | internal | Host address mapped for a private-CA deployment. |
| `MODELFORGE_AGENT_CA_CERT_PATH` | no | public | Path to the private CA certificate the agent trusts. |
| `MODELFORGE_RUNTIME_WORKER_ARTIFACT_ROOT` | no | public | Artifact root as the runtime worker sees it. |
| `MODELFORGE_RUNTIME_WORKER_POLL_INTERVAL_SECONDS` | no | public | Worker poll interval, in seconds. |
| `MODELFORGE_SOURCE_COMMIT` | no | public | Source commit reported by the deployment. |
| `MODELFORGE_SOURCE_REFERENCE` | no | public | Git reference reported by the deployment. |
| `MODELFORGE_SOURCE_REPOSITORY` | no | public | Repository URL reported by the deployment. |
## Restarts
Every setting is read at process start. Changing any of them requires restarting the control plane; none is re-read from the environment while the process is running.
docs/FIRST_RUN.md
+113
View File
@@ -0,0 +1,113 @@
# First run
You have a healthy control plane. Nothing is serving yet, and that is correct — this page is how you
get from there to a capability an application can call.
## What you have after installation
| | |
| --- | --- |
| Projects | seeded from the shipped manifests |
| Capability contracts | **none** |
| Capability deployments | **none** |
| Model candidates | seeded as a discovery list, none approved |
| Policies | lifecycle, migration, observability and recovery defaults present |
| Compute nodes | none until you enrol one |
Contracts and deployments are absent on purpose. A contract is lifecycle-owned, and a deployment
requires a model artifact that has been acquired, verified and promoted. ModelForge will not
fabricate either from a manifest, so the bindings that are waiting for a contract are *reported*
rather than invented:
```text
SYNCED project registry 8 binding(s) waiting for a contract
```
## 1. Enrol a compute node
Nothing can serve without a node. See [NODE_AGENT.md](NODE_AGENT.md) for the full procedure. In
short:
```bash
curl -s -X POST -H "X-ModelForge-Admin-Token: $KEY" -H "Content-Type: application/json" \
-d '{"display_name":"GPU Node","expires_in_seconds":900}' \
http://127.0.0.1:8000/api/v1/admin/node-enrollments
```
The token it returns is **single use**. A concurrent storm against one token produces exactly one
identity and one credential — that is enforced by an atomic claim, not by timing.
Give the token to the agent on the GPU host and start it. The node appears with liveness `online`.
## 2. Discover a model candidate
The Discover workspace lists the candidates in `config/models/initial-candidates.yaml`. Nothing in
that file is approved by being listed — it is a starting point for verification.
## 3. Acquire and verify the artifact
Acquisition downloads into quarantine, records a per-file SHA-256, and runs the supply-chain checks
before anything is promoted to the verified artifact root. `trust_remote_code` is never enabled;
production refuses to start if it is.
An artifact that fails verification stays in quarantine. That is the system working.
## 4. Register a capability contract and promote a deployment
Capabilities are the interface applications bind to. A contract fixes the input and output schema
for a version; a deployment is a specific artifact set and runtime profile serving that contract.
Promotion runs through the lifecycle: plan, approval, execution, and an immutable snapshot that lets
an interrupted operation roll back exactly once. `no_hidden_auto_promotion` asserts that no
production deployment exists without a recorded production approval — there is no path that quietly
promotes something.
See [MODEL_LIFECYCLE.md](product/MODEL_LIFECYCLE.md).
## 5. Create a service client for the application
```bash
curl -s -X POST -H "X-ModelForge-Admin-Token: $KEY" -H "Content-Type: application/json" \
-d '{"name":"examplerag-production","allowed_capabilities":["rag.embedding@1"],
"project_key":"examplerag","integration_environment":"production",
"purpose":"ExampleRAG production embedding"}' \
http://127.0.0.1:8000/api/v1/admin/service-clients
```
The response carries the credential **once**. Only a SHA-256 hash is stored, with a bounded prefix
for identification. There is no way to read it back — rotate if you lose it.
## 6. Call the capability
```bash
curl -s -X POST -H "Authorization: Bearer $CREDENTIAL" -H "Content-Type: application/json" \
-d '{"input":["the text to embed"]}' \
http://127.0.0.1:8000/api/v1/capabilities/rag.embedding@1/invoke
```
The application names a **capability**, never a model, a node or a file path. That is the whole point
of the platform: you can replace what is behind `rag.embedding@1` without the caller knowing, and the
caller cannot accidentally depend on which model answered.
See [PROJECT_INTEGRATION.md](PROJECT_INTEGRATION.md).
## What you should see when it is working
| Signal | Healthy value |
| --- | --- |
| `/api/v1/health/ready` | 200 |
| Node liveness | `online` |
| `/api/v1/capabilities` | your contract listed |
| Operations overview | no firing alerts |
| Recovery dashboard | a recent verified backup, no unprotected assets |
## Common first-run outcomes
| What you see | What it means |
| --- | --- |
| `503 NO_ELIGIBLE_NODE` | No online node can serve this capability. Check node liveness first. |
| `CAPACITY_CONSTRAINED` | A node exists but has no headroom. This is a correct refusal, not an error. |
| `401 CAPABILITY_NOT_AUTHORIZED` | The credential is missing, revoked, expired, or not scoped to this capability. |
| `8 binding(s) waiting for a contract` | Expected before you have promoted anything. |
More in [TROUBLESHOOTING.md](TROUBLESHOOTING.md).
docs/INSTALLATION.md
+171
View File
@@ -0,0 +1,171 @@
# Installing ITWorx ModelForge v1.2.1
This is the whole installation. If you have to do something that is not written here, that is a
defect — please report it rather than working around it.
The steps below are release-gated against a fresh project with isolated volumes, newly generated
test secrets and an empty PostgreSQL database.
## 1. What you need
| Requirement | Minimum | Notes |
| --- | --- | --- |
| Docker Engine | 24 | `docker version` |
| Docker Compose | 2.x | `docker compose version` |
| PostgreSQL | 17 (16 accepted) | Provided by the deployment; you do not install it separately |
| Redis | 7 | Provided by the deployment |
| Disk | 50 GiB free, more for model artifacts | Model weights dominate; plan for the models you intend to run |
| Python | 3.12 | Only to run the preflight, bootstrap and upgrade tools from the host |
A GPU is **not** required on the control-plane host. It is required on any compute node that will
serve a capability, along with the NVIDIA driver and the NVIDIA container runtime. See
[NODE_AGENT.md](NODE_AGENT.md).
Networking: the control plane never dials a compute node. Nodes report outbound to the control
plane, so a node needs to reach the API but the API needs no route back.
PostgreSQL 16+ is the production database contract. SQLite is an isolated unit-test backend, not a
supported installation or migration target.
## 2. Check the host first
```bash
python scripts/preflight.py
```
It reports the Docker and Compose versions, the NVIDIA driver if present, whether the ports the
deployment publishes are free, and free disk. Exit status is 0 when the host can run ModelForge.
Nothing is changed by this command.
## 3. Generate your secrets
ModelForge never mints its own credentials. A platform that generates its own admin secret has no
way to tell you it did.
```bash
python -c "import secrets; print(secrets.token_urlsafe(48))" # operator API key
python -c "import base64, os; print(base64.b64encode(os.urandom(32)).decode())" # backup key
python -c "import secrets; print(secrets.token_urlsafe(32))" # PostgreSQL bootstrap admin
python -c "import secrets; print(secrets.token_urlsafe(32))" # migration/schema owner
python -c "import secrets; print(secrets.token_urlsafe(32))" # API runtime role
```
**Store the backup encryption key outside this deployment.** Losing it makes every existing backup
unrecoverable. It is the one value that must survive the machine it protects.
## 4. Configure
```bash
cp .env.example .env
```
Fill in at least these. Production refuses to start without them:
```ini
MODELFORGE_POSTGRES_DB=modelforge
MODELFORGE_POSTGRES_ADMIN_USER=postgres
MODELFORGE_POSTGRES_ADMIN_PASSWORD=<generated-admin-secret>
MODELFORGE_MIGRATION_DB_PASSWORD=<generated-owner-secret>
MODELFORGE_RUNTIME_DB_PASSWORD=<generated-runtime-secret>
MODELFORGE_MIGRATION_DATABASE_URL=postgresql+psycopg://modelforge:<URL-encoded-owner-secret>@postgres:5432/modelforge
MODELFORGE_RUNTIME_DATABASE_URL=postgresql+psycopg://modelforge_runtime:<URL-encoded-runtime-secret>@postgres:5432/modelforge
MODELFORGE_OPERATOR_API_KEY=<generated, at least 32 characters>
MODELFORGE_BACKUP_ENCRYPTION_KEY=<generated base64 AES-256 key>
MODELFORGE_CORS_ORIGINS=https://modelforge.example.internal
VITE_API_BASE_URL=https://modelforge.example.internal:8000
MODELFORGE_VERSION=1.2.1
MODELFORGE_API_IMAGE=modelforge-api:1.2.1
MODELFORGE_WEB_IMAGE=modelforge-web:1.2.1
```
Every setting is documented in [CONFIGURATION.md](CONFIGURATION.md), which is generated from the
typed settings — it cannot drift from what the code actually reads.
To move a port, set `MODELFORGE_API_PUBLISHED_PORT`, `MODELFORGE_WEB_PORT`,
`MODELFORGE_POSTGRES_PORT` or `MODELFORGE_REDIS_PORT`. You never have to edit a Compose file.
Then re-run the preflight with the production rules applied:
```bash
python scripts/preflight.py --production
```
## 5. Deploy
With the verified release images already loaded or pulled, deploy them without rebuilding:
```bash
docker compose -f docker-compose.yml -f docker-compose.production.yml up -d --no-build
```
`docker-compose.yml` on its own is a **development** deployment, but it still requires distinct
database secrets; none is embedded in the repository. It leaves the environment at `development`,
which switches off production-only startup rules. The production overlay sets
`MODELFORGE_ENV=production` and takes each secret through
`${VAR:?message}`, so Compose refuses to render the project at all if one is missing. If you see a
message like
```text
required variable MODELFORGE_OPERATOR_API_KEY is missing a value
```
that is the deployment refusing before it starts a single container.
For an explicitly local source build, keep `MODELFORGE_VERSION`, `MODELFORGE_COMMIT` and
`MODELFORGE_BUILT_AT` pinned and use `up -d --build`. The production projection retains this build
path, but never derives a deployable image from a floating `latest` tag.
To stamp build identity into the images so a running container can say where it came from:
```bash
MODELFORGE_VERSION=1.2.1 \
MODELFORGE_COMMIT=$(git rev-parse HEAD) \
MODELFORGE_BUILT_AT=$(date -u +%FT%TZ) \
docker compose -f docker-compose.yml -f docker-compose.production.yml up -d --build
```
## 6. Verify
```bash
curl -s http://127.0.0.1:8000/api/v1/health/live # 200
curl -s http://127.0.0.1:8000/api/v1/health/ready # 200
curl -s http://127.0.0.1:8000/api/v1/version # version, commit, compatibility
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/api/v1/admin/recovery/dashboard # 401
curl -s -o /dev/null -w '%{http_code}\n' -H "X-ModelForge-Admin-Token: $KEY" \
http://127.0.0.1:8000/api/v1/admin/recovery/dashboard # 200
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/ # 200
```
The 401 matters as much as the 200: it is the operator boundary refusing an unauthenticated caller.
The `migrate` container runs Alembic once with the non-superuser `modelforge` schema-owner URL and
must complete before the API starts. The API receives only the `modelforge_runtime` URL; it cannot
read the owner or bootstrap secret. Migration 0024 grants that runtime role SELECT on the audit
tables and EXECUTE on the canonical append function while revoking direct audit mutation and DDL.
Production startup re-reads the PostgreSQL catalog and refuses any drift in that boundary.
To run bootstrap tooling yourself, or to check what a clean database would do:
```bash
python scripts/bootstrap.py --database-url postgresql+psycopg://user:pass@127.0.0.1:5432/modelforge
```
`bootstrap.py` is safe to run twice. Every step either creates what is missing or confirms what is
already there and says which. Running it repeatedly on an installed system changes nothing — the
release rehearsal ran it three times and all ten authoritative counts were identical.
## 7. Next
- [FIRST_RUN.md](FIRST_RUN.md) — what to do with a running control plane
- [NODE_AGENT.md](NODE_AGENT.md) — enrol a GPU compute node
- [OPERATIONS.md](OPERATIONS.md) — runbooks, health model, backups
- [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — the states you will actually meet
## What a fresh install does not include
A newly installed control plane has **no capability contracts and no deployments**. That is
deliberate: contracts are lifecycle-owned and a deployment requires a model artifact that has been
acquired, verified and promoted. The projects and policies from the shipped manifests are seeded,
and the bindings that wait for a contract are reported rather than fabricated. See
[FIRST_RUN.md](FIRST_RUN.md).
docs/NODE_AGENT.md
+148
View File
@@ -0,0 +1,148 @@
# Compute nodes and the Node Agent
A compute node is a GPU host that serves capabilities. The Node Agent runs there, reports inventory
and telemetry, and claims work.
**The control plane never dials a node.** The agent is outbound-only: it needs to reach the API, and
the API needs no route back. That is what makes a node behind NAT, on a home network, or on a
different site work without inbound exposure.
## Requirements on the node
| | |
| --- | --- |
| NVIDIA driver | matching your GPU |
| NVIDIA container runtime | required for the Runtime Worker |
| Docker | 24+ |
| Reachability | outbound HTTPS to the control plane |
| Storage | artifact root, quarantine and agent state on durable storage |
## Enrolling
1. An operator mints a single-use token:
```bash
curl -s -X POST -H "X-ModelForge-Admin-Token: $KEY" -H "Content-Type: application/json" \
-d '{"display_name":"GPU Node","expires_in_seconds":900}' \
http://127.0.0.1:8000/api/v1/admin/node-enrollments
```
2. Configure the agent on the node:
```ini
MODELFORGE_AGENT_CONTROL_PLANE_URL=https://modelforge.example.internal:8000
MODELFORGE_AGENT_ENROLLMENT_TOKEN=<the token>
MODELFORGE_AGENT_HOSTNAME=gpu_node
MODELFORGE_AGENT_ACCELERATOR_MODE=nvidia
MODELFORGE_AGENT_TLS_VERIFY=true
MODELFORGE_AGENT_STATE_VOLUME=/mnt/user/appdata/modelforge/agent-state
```
3. Start it:
```bash
export MODELFORGE_NODE_AGENT_IMAGE=modelforge-node-agent:1.1.1
docker compose -f docker-compose.yml -f docker-compose.node-agent.yml up -d node-agent
```
For production, set `MODELFORGE_NODE_AGENT_IMAGE` to the exact release tag or registry digest. The
Compose file retains `build:` for local source builds, but its fallback is explicitly `local` and
never the floating `latest` tag.
The token is **single use and atomically claimed**. Sixty concurrent attempts against one token
produce exactly one identity and one active credential — that is enforced by a conditional update
whose row count decides the winner, not by timing.
Verified on a genuinely fresh control plane before release: the node appeared once, `online`, with
exactly one active credential.
## NVIDIA startup contract
The v1.1.1 Node Agent runs on a digest-pinned Debian/glibc base. NVIDIA Container Toolkit injects
the host's driver-facing `libnvidia-ml.so.1`, dynamic loader bindings and `/dev/nvidia*` devices;
the image deliberately does not bundle a userspace driver library. The earlier v1.1.0 Alpine/musl
image could import the pure-Python `pynvml` package but could not relocate GPU Node's glibc-linked
NVML library, producing `NVMLError_LibraryNotFound`.
`MODELFORGE_AGENT_ACCELERATOR_MODE` has three values:
- `nvidia`: require NVML, at least one real GPU, and matching telemetry; canonical GPU Compose
deployments use this value.
- `cpu`: permit a legitimate CPU-only node without NVML.
- `auto`: require NVML only when NVIDIA devices or `NVIDIA_VISIBLE_DEVICES` are observed.
The NVIDIA preflight runs before enrollment. A missing library, empty device result, or missing
telemetry exits with code 3 and one of `NVIDIA_NVML_UNAVAILABLE`, `NVIDIA_DEVICE_NOT_FOUND`, or
`NVIDIA_TELEMETRY_UNAVAILABLE`; no empty-success inventory or fabricated telemetry is sent. A
bounded diagnostic check that never enrolls is available as:
```bash
docker run --rm --gpus all --network none \
-e MODELFORGE_AGENT_ACCELERATOR_MODE=nvidia \
modelforge-node-agent:1.1.1 modelforge-node-agent --preflight
```
## Identity
A node's identity is persisted and survives restarts. If a node reappears as a *new* identity, its
state volume is not durable — fix that before anything else, because duplicate identities are the
one thing the platform cannot untangle for you.
`MODELFORGE_AGENT_IDENTITY_MODE=persisted` keeps the identity in
`MODELFORGE_AGENT_STATE_VOLUME`. On Unraid, point that at `appdata`, not at a container-local path.
## Liveness
`online → stale → offline` on the configured thresholds
(`MODELFORGE_NODE_STALE_AFTER_SECONDS`, `MODELFORGE_NODE_OFFLINE_AFTER_SECONDS`; offline must exceed
stale, and startup refuses a configuration where it does not).
No work is assigned to a node that is not online. A disconnected agent stops publishing and the
control plane says so rather than assuming the node is fine — verified by disconnecting a disposable
agent for 100 seconds and confirming its heartbeat genuinely stopped advancing.
## Private CA
If the control plane is behind a private certificate authority:
```bash
docker compose -f docker-compose.yml \
-f docker-compose.node-agent.yml \
-f docker-compose.node-agent.private-ca.yml up -d node-agent
```
with `MODELFORGE_AGENT_CA_CERT_PATH` and `MODELFORGE_AGENT_CONTROL_PLANE_HOST_ADDRESS` set. Do not
turn off `MODELFORGE_AGENT_TLS_VERIFY` to work around a certificate problem — that replaces a
certificate error with a silent one.
## The Runtime Worker
The worker executes the model. It is a separate service and a separate image, pinned by digest,
built on the CUDA runtime base:
```bash
docker compose -f docker-compose.yml -f docker-compose.runtime-worker.yml up -d runtime-worker
```
It reports its own protocol version, and the same four compatibility answers apply.
## Diagnosing an agent that will not report
A failed publication names the status, the method and the path — never the response body, which can
echo what the agent was publishing, and never the credential:
```text
ConnectError on POST /api/v1/agent/heartbeat (backoff 1s…16s)
```
That distinguishes a revoked credential from a DNS failure. If the cause is not obvious, check that
`MODELFORGE_AGENT_CONTROL_PLANE_URL` is a real address and not the example placeholder.
## Compatibility
The control plane speaks agent protocol version 1 and accepts version 1. An agent reporting anything
else is told which of `TOO_OLD`, `TOO_NEW` or `UNKNOWN` applies and what to do. There is no silent
mismatch. See [COMPATIBILITY.md](COMPATIBILITY.md).
v1.1.1 keeps protocol 1, so v1.0.0 and v1.1.0 agents remain protocol-compatible. NVIDIA nodes
should run v1.1.1 to obtain the glibc/NVML fix and fail-closed startup contract.
docs/OPERATIONS.md
+99
View File
@@ -0,0 +1,99 @@
# Operations
The index an operator works from. Everything here already exists in the repository; this page is the
map, not a copy.
## Daily signals
| Where | What it tells you |
| --- | --- |
| `/api/v1/health/live` | The process is up. It touches nothing else, so it stays 200 when a dependency is down. |
| `/api/v1/health/ready` | The manifest set is loadable. |
| `/api/v1/version` | Which build is answering, and what it is compatible with. |
| Operations workspace | Capability health, capacity, SLO evaluations, active alerts. |
| Recovery dashboard | Newest verified backup, unprotected assets, recovery readiness. |
## The health model
`liveness`, `readiness`, `OBSERVABILITY_DEGRADED` and `CAPACITY_CONSTRAINED` each mean something
specific and different — see
[FAILURE_AND_HEALTH_MODEL.md](operations/FAILURE_AND_HEALTH_MODEL.md).
The one that surprises people: **a capacity rejection is a correct outcome.** The scheduler refusing
work it cannot place safely is the system working, not an incident.
## Runbooks
| Situation | Runbook |
| --- | --- |
| A node stops reporting | [RUNBOOK_NODE_OFFLINE.md](operations/RUNBOOK_NODE_OFFLINE.md) |
| A node is gone for good | [RUNBOOK_NODE_LOSS.md](operations/RUNBOOK_NODE_LOSS.md) |
| GPU pressure | [RUNBOOK_GPU_PRESSURE.md](operations/RUNBOOK_GPU_PRESSURE.md) |
| Gateway errors | [RUNBOOK_GATEWAY_ERRORS.md](operations/RUNBOOK_GATEWAY_ERRORS.md) |
| A runtime fails | [RUNBOOK_RUNTIME_FAILURE.md](operations/RUNBOOK_RUNTIME_FAILURE.md) |
| A backup fails | [RUNBOOK_BACKUP_FAILURE.md](operations/RUNBOOK_BACKUP_FAILURE.md) |
| Restoring the database | [RUNBOOK_DATABASE_RESTORE.md](operations/RUNBOOK_DATABASE_RESTORE.md) |
| Losing the control plane | [RUNBOOK_CONTROL_PLANE_LOSS.md](operations/RUNBOOK_CONTROL_PLANE_LOSS.md) |
| Full disaster recovery | [RUNBOOK_FULL_DR.md](operations/RUNBOOK_FULL_DR.md) |
| An artifact is lost | [RUNBOOK_ARTIFACT_LOSS.md](operations/RUNBOOK_ARTIFACT_LOSS.md) |
| A migration cutover is stuck | [RUNBOOK_MIGRATION_FAILURE.md](operations/RUNBOOK_MIGRATION_FAILURE.md) |
| Storage is low | [RUNBOOK_STORAGE_LOW.md](operations/RUNBOOK_STORAGE_LOW.md) |
## Backups and recovery
Backup classification, the immutable manifest, encryption and measured RPO/RTO are in
[BACKUP_AND_RECOVERY.md](architecture/BACKUP_AND_RECOVERY.md).
Three properties worth knowing before you need them:
- a backup interrupted mid-write can **never** become restore eligible — it is reconciled to
`FAILED` / `MANIFEST_INCOMPLETE` on the next start;
- an unwritable destination fails closed with `DESTINATION_UNAVAILABLE` rather than producing a
partial recovery point;
- **point-in-time recovery is `NOT_SUPPORTED`** in v1. Verified snapshot restore is what exists.
Losing `MODELFORGE_BACKUP_ENCRYPTION_KEY` makes every existing backup unrecoverable. Store it
outside the deployment it protects.
## Lifecycle and migration
- [MODEL_LIFECYCLE.md](product/MODEL_LIFECYCLE.md) — discover, verify, evaluate, approve, promote,
canary, roll back, deprecate.
- [MIGRATION_RECOVERY.md](architecture/MIGRATION_RECOVERY.md) — why an interrupted cutover is
reported and never auto-resolved.
Lifecycle **resolves** after an interruption; migration **reports**; backups **fail closed**. Each
subsystem declares which of the three it does — see
[ADR-0045](architecture/adr/0045-chaos-safe-reconciliation-semantics.md).
## Observability
[OBSERVABILITY.md](architecture/OBSERVABILITY.md). Monitoring is never a safety dependency: when
observability persistence is unavailable, serving and recovery continue on their own authoritative
state.
## Verifying the platform is sound
```bash
python scripts/m16_invariants.py --database-url postgresql+psycopg://user:pass@host:5432/modelforge
```
Fifteen read-only checks over authoritative state. They never write, never call an external system,
and deliberately never read the observability tables — so they can answer "is state still sound?"
precisely when monitoring is the thing that failed.
## Chaos and soak
[CHAOS_TESTING.md](operations/CHAOS_TESTING.md), [SOAK_TESTING.md](operations/SOAK_TESTING.md),
[FAULT_INJECTION.md](operations/FAULT_INJECTION.md),
[RESTART_RECOVERY.md](operations/RESTART_RECOVERY.md).
## Deployment
- [INSTALLATION.md](INSTALLATION.md) — clean install
- [UPGRADE.md](UPGRADE.md) — upgrading, and rolling back
- [UNRAID_DEPLOYMENT.md](UNRAID_DEPLOYMENT.md) — the Unraid/GPU Node deployment specifically
- [NODE_AGENT.md](NODE_AGENT.md) — enrolling and running a compute node
- [CONFIGURATION.md](CONFIGURATION.md) — every setting, generated from the code
- [RELEASE_NOTES_v1.0.0.md](RELEASE_NOTES_v1.0.0.md) — release behavior and operator-facing changes from v1.0.0
production cutover: what was measured, and what was left as a documented limitation
docs/PROJECT_INTEGRATION.md
+111
View File
@@ -0,0 +1,111 @@
# Integrating an application with ModelForge
The contract an application depends on is a **capability**, never a model, a node, a file path or a
Hugging Face repository name. That is the whole design: what serves `rag.embedding@1` can be
replaced without the caller knowing, and the caller cannot accidentally couple to which model
answered.
## What your application needs to know
Three things, and nothing else:
```text
base URL https://modelforge.example.internal:8000
credential the bearer token issued for your service client
capability key rag.embedding@1
```
If your integration needs a model name, a revision, a node hostname or an artifact path, something
has gone wrong — those are ModelForge's business, not yours.
## Getting a credential
An operator creates a service client scoped to your project and the capabilities you may call:
```bash
curl -s -X POST -H "X-ModelForge-Admin-Token: $KEY" -H "Content-Type: application/json" \
-d '{"name":"examplerag-production",
"allowed_capabilities":["rag.embedding@1"],
"project_key":"examplerag",
"integration_environment":"production",
"purpose":"ExampleRAG production embedding"}' \
http://127.0.0.1:8000/api/v1/admin/service-clients
```
The credential is returned **once**. Only a SHA-256 hash is stored, with a short prefix for
identification, so it cannot be read back — rotate if you lose it.
`integration_environment` is one of `production`, `shadow`, `evaluation` or `lab`.
## Calling a capability
```http
POST /api/v1/capabilities/rag.embedding@1/invoke
Authorization: Bearer mf_...
Content-Type: application/json
{"input": ["first text", "second text"]}
```
A successful response carries the result and an `execution` block describing what actually happened
— queue time, inference time, whether the deployment was already warm. Use it for your own
observability; do not branch on it.
## Errors your integration must handle
| Status | Code | Meaning | What to do |
| --- | --- | --- | --- |
| 401 | `CAPABILITY_NOT_AUTHORIZED` | Missing, invalid, revoked, expired or out-of-scope credential | Do not retry; get a valid credential |
| 429 | rate limited | Your client's requests-per-minute or concurrency was exceeded | Back off |
| 503 | `NO_ELIGIBLE_NODE` | No online node can serve this capability | Retry with backoff; this is usually transient |
| 503 | `CAPACITY_CONSTRAINED` | A node exists but has no headroom | Retry with backoff; **this is a correct refusal, not an outage** |
| 422 | validation | Your request body does not match the contract | Fix the request |
Every error response is exactly `code`, `message` and `correlation_id`. Log the correlation id — it
is how an operator finds your request in the control-plane logs.
## Scope is exact
- `rag.embedding@1` does not grant `rag.embedding@2`;
- a Vision client cannot reach ASR, and an ASR client cannot reach Vision;
- case changes, appended whitespace, a second capability in the same string and zero-width
characters do not widen a scope;
- a project-bound client cannot serve another project's binding;
- a deprecated binding stops serving immediately.
## Contract versions and migration
A capability contract version fixes the input and output schema. A change that keeps them compatible
happens behind the same version. A change that does not gets a new version, and your application
moves when it is ready.
The case that needs your attention is `REQUIRES_REINDEX`: the embedding space changed, so vectors
produced by the old deployment are not comparable with the new one. ModelForge will not swap an
alias underneath data that has not been reindexed, and your project binding has to declare that it
supports reindex migration before such a contract can be bound. See
[MIGRATION_RECOVERY.md](architecture/MIGRATION_RECOVERY.md).
## Batching and limits
| Limit | Default | Setting |
| --- | --- | --- |
| Inputs per invocation | 8 | `MODELFORGE_GATEWAY_MAX_BATCH_SIZE` |
| Characters per input | 8192 | `MODELFORGE_GATEWAY_MAX_INPUT_CHARACTERS` |
| Request body | 64 KiB | `MODELFORGE_GATEWAY_MAX_PAYLOAD_BYTES` |
| Total request time | 45 s | `MODELFORGE_GATEWAY_REQUEST_TIMEOUT_SECONDS` |
| Queue wait before rejection | 30 s | `MODELFORGE_GATEWAY_QUEUE_TIMEOUT_SECONDS` |
## Idempotency
Supply an idempotency key when a retry must not produce a second execution. Keys are unique in the
database rather than in a queue, so a retry storm converges on one row regardless of which component
retried — 24 concurrent requests on one key produce exactly one.
## What ModelForge will never do to your application
- serve a model you did not approve through the lifecycle;
- swap an embedding space underneath you without a reindex;
- fall back to a different artifact source when the recorded one is unavailable;
- return an answer built on telemetry it knows is stale.
Each of those is a refusal with a name instead.
docs/PUBLICATION.md
+42
View File
@@ -0,0 +1,42 @@
# Public source boundary
## Publication decision
Do not make this canonical repository public in place. Its history contains real private network
addresses, deployment hostnames, storage paths, integration names and production verification
evidence. Those facts are useful to operators but unnecessary for users and increase disclosure
risk.
Create a new, parentless public repository from `scripts/export-public-source.mjs`. The exporter
selects reviewed source and documentation from `public-source.allowlist`, replaces private example
identifiers with stable synthetic ones, rejects remaining private indicators and writes a content
manifest. The canonical repository and all of its Git history remain private.
## Release checklist
1. Verify the canonical AGPL-3.0 `LICENSE` text and SPDX metadata are present.
2. Confirm `security@itworx.tech` is monitored for private vulnerability reports.
3. Generate the export into a new directory outside this repository.
4. Run `node scripts/validate-public-source.mjs` from the generated directory, then run Gitleaks
and all component tests there. The validator recomputes every declared SHA-256 digest and byte
count, rejects undeclared files, links, unsafe paths and case-insensitive path collisions.
5. Dispatch `public-candidate-acceptance.yml` for the exact canonical commit. It renders from the
clean export, builds and scans all four images, and clean-installs an isolated production-shaped
Compose project without touching production. See `docs/operations/RC_ACCEPTANCE.md`.
6. Review generated `PUBLIC_SOURCE_MANIFEST.json` and the final diff.
7. Create a new repository with one parentless initial commit; never push canonical refs or tags.
8. Configure protected branches. Pull requests from forks must not receive internal secrets or run
automatically on persistent private runners. The exporter therefore removes the automatic
`pull_request` trigger from the public copy of `managed-validation.yml` and leaves only explicit
owner dispatch. Keep public branch protection independent of private-runner status checks until
a public-safe CI runner is configured.
## License decision
The owner selected **AGPL-3.0-or-later** on 2026-09-01. ModelForge is a network-accessible,
self-hosted application; the AGPL keeps modifications made available over a network available to
those users and protects the platform from closed hosted forks.
The license covers ModelForge source and distributions. Downloaded model repositories, weights,
datasets and other third-party artifacts retain their own upstream terms. The exporter verifies the
approved canonical license digest before creating a public candidate.
docs/RELEASE_NOTES_v1.0.0.md
+134
View File
@@ -0,0 +1,134 @@
# ITWorx ModelForge v1.0.0
The first stable release. ModelForge is a self-hosted control plane for running AI models on your
own hardware: it decides what may run, where it runs, and what your applications are allowed to
depend on.
## What it does
**Capability-first routing.** Applications bind to a capability — `rag.embedding@1` — and never to a
model, a node, a file path or a Hugging Face repository name. What serves a capability can be
replaced without any consumer knowing, and no consumer can accidentally couple to which model
answered.
**A model lifecycle that refuses to guess.** Discover, acquire into quarantine, verify with a
per-file SHA-256, evaluate, approve, promote, canary, roll back, deprecate. `trust_remote_code` is
never enabled, and production refuses to start if it is. No production deployment exists without a
recorded production approval.
**GPU scheduling that admits what it does not know.** The scheduler places work against measured
capacity and refuses when there is no headroom rather than risking an out-of-memory failure that
would take down work already running. VRAM held by processes ModelForge does not manage counts as
external pressure and is never reclaimed. Stale telemetry blocks admission instead of being
extrapolated.
**Outbound-only compute nodes.** A node reaches the control plane; the control plane never dials a
node. Enrolment is single use and atomically claimed.
**Embedding-space migration that will not corrupt a corpus.** A contract that changes the embedding
space is `REQUIRES_REINDEX`, and the migration engine will not swap an alias underneath data that
has not been reindexed.
**Backup and recovery with measured RPO.** Encrypted backups with immutable manifests, verified
restore, artifact rehydration, node re-enrolment and control-plane rebuild.
**Operational observability.** SLOs, alerts with real state transitions, capacity history, and a
health model in which monitoring is never a safety dependency.
## System requirements
| | |
| --- | --- |
| Docker Engine | 24+ |
| Docker Compose | 2+ |
| PostgreSQL | 16+ (17 ships with the deployment) |
| Redis | 7+ |
| Disk | 50 GiB, plus room for model artifacts |
| GPU | Required on compute nodes only, with the NVIDIA container runtime |
## Installing
[docs/INSTALLATION.md](INSTALLATION.md). A verified clean install took 124 seconds from
`up -d --build` to a healthy API and console.
The production deployment is one command, and it is not the same as the development one:
```bash
docker compose -f docker-compose.yml -f docker-compose.production.yml up -d --build
```
## Upgrading
Supported from `m16-baseline`. [docs/UPGRADE.md](UPGRADE.md).
**v1.0.0 introduces no database migration.** The schema head is unchanged at `20260827_0021`, so
rolling back needs no database restore — verified, with the previous application version healthy
against the same database in 19 seconds.
## Security posture
- Datastores bind to loopback by default; only the API is published, and every admin route is
operator-authenticated.
- Production fails closed at startup on a missing or weak operator key, a development database
password, remote code execution, a missing backup encryption key, a wildcard CORS origin, an
unwritable storage root, or an unsupported schema revision.
- The console ships a Content-Security-Policy with `default-src 'none'`, no `unsafe-inline` or
`unsafe-eval` in the script directive, and inline style permitted only as an attribute — plus six
further security headers on the entry document and on hashed assets.
- All images run unprivileged, drop all capabilities and set `no-new-privileges`. No container
mounts the Docker socket. The console and Node Agent run read-only.
- Backups are AES-256-GCM with a fresh nonce per chunk and fail-closed decryption.
- Credentials are stored as SHA-256 hashes with a bounded prefix and cannot be read back.
## Known limitations
These are real and are stated rather than discovered later.
- **Point-in-time recovery is `NOT_SUPPORTED`.** Verified snapshot restore with a measured RPO is
what exists; continuous WAL recovery is out of scope for v1.
- **Upgrades require downtime.** The control plane is stopped while the new version starts. Rolling
upgrade is not supported and is not claimed.
- **No console workspace for Model Lab or Benchmarks.** Both have platform support; neither has an
operator view in v1.
- **The capacity-rejection and PostgreSQL deadlock-retry paths are covered by deterministic tests,
not by production load.** No soak reached the concurrency that would exercise them naturally.
- **Storage and inode exhaustion are covered by isolated tests, not by live injection**, deliberately:
filling a real volume to prove a guard is not a reasonable thing to do to a production host.
## Deferred project validation
Not ModelForge platform defects — external work that has not been completed:
- ExampleVision Vision owner-photo validation;
- OCR has no production candidate;
- ASR natural-speech validation;
- ExampleRAG production smoke, pending that service running.
## Documentation
| | |
| --- | --- |
| [INSTALLATION.md](INSTALLATION.md) | Clean install |
| [FIRST_RUN.md](FIRST_RUN.md) | From a running control plane to a serving capability |
| [UPGRADE.md](UPGRADE.md) | Upgrading and rolling back |
| [CONFIGURATION.md](CONFIGURATION.md) | Every setting, generated from the code |
| [COMPATIBILITY.md](COMPATIBILITY.md) | Version, schema and protocol ranges |
| [NODE_AGENT.md](NODE_AGENT.md) | Enrolling and running a compute node |
| [UNRAID_DEPLOYMENT.md](UNRAID_DEPLOYMENT.md) | The Unraid deployment specifically |
| [CAPABILITIES.md](CAPABILITIES.md) | What a capability is and why |
| [PROJECT_INTEGRATION.md](PROJECT_INTEGRATION.md) | Integrating an application |
| [OPERATIONS.md](OPERATIONS.md) | Runbooks, health model, backups |
| [TROUBLESHOOTING.md](TROUBLESHOOTING.md) | The states you will actually meet |
| [SECURITY.md](SECURITY.md) | Threat model and security posture |
## Upgrade notes for existing M16 deployments
1. Adopting the production profile requires secrets M16 did not: an operator API key of at least 32
characters, a backup encryption key, a non-development database password, and an exact CORS
origin. The deployment refuses to render without them.
2. The console no longer displays development milestone labels.
3. `GET /api/v1/system` no longer returns `milestone`; it returns `release_channel`.
4. `GET /api/v1/version` is new and unauthenticated.
5. Host ports are configurable without editing a Compose file.
6. If you build images yourself, pass `MODELFORGE_VERSION`, `MODELFORGE_COMMIT` and
`MODELFORGE_BUILT_AT` so the running container can report where it came from.
docs/RELEASE_NOTES_v1.1.0.md
+64
View File
@@ -0,0 +1,64 @@
# ITWorx ModelForge v1.1.0
v1.1.0 is a backward-compatible MINOR release. It adds a first-class operator capability, new API
surface, a new Console workflow and the additive schema migration `20260827_0021` →
`20260828_0022`; it does not break agent protocol 1 or capability-first application bindings.
## Highlights
### Audited Node Decommission
- Dry-run preview names the exact node, typed blockers, dependencies, cleanup and retained history.
- Execute is operator-only, transactional, generation/digest guarded and idempotent.
- Active credentials are revoked; current telemetry, scheduler and disposable inventory truth are
removed; the node remains as a terminal tombstone.
- Old credentials receive 401, old persisted identities cannot ordinarily re-enrol, and the
scheduler excludes terminal nodes.
- Historical deployments, jobs, probes, capacity/provenance and exactly one
`NODE_DECOMMISSIONED` audit event remain available.
### Console and API reliability
- Shared design tokens, light/dark theme persistence, grouped navigation, page-specific headers and
a mobile drawer.
- Keyboard-visible focus, accessible navigation and explicit failure/recovery styling.
- Typed API error envelopes and correlation IDs survive client handling; successful empty responses
no longer fail JSON decoding.
### Packaging and database
- Production Node Agent deployments can pin `modelforge-node-agent:1.1.0` or an immutable digest;
local builds remain possible and no `latest` fallback is used.
- All published images use digest-pinned Alpine bases, apply signed package security updates and
run as their dedicated non-root runtime users. Python build tooling is removed after dependency
consistency verification.
- Fresh installs migrate an empty supported PostgreSQL database through the full chain to
`20260828_0022`.
- Direct upgrades from v1.0.0/schema `20260827_0021` require a verified backup and migrate to
`20260828_0022`. Application-only rollback is not supported after that schema change; restore the
verified pre-upgrade backup.
Eight stale/test node records were manually removed before this capability existed. That SQL is
historical evidence, not a supported workflow; v1.1.0 replaces it with the audited API and Console
operation.
## Compatibility
| Contract | v1.1.0 |
| --- | --- |
| Channel | stable |
| Runtime schema | `20260828_0022` |
| Direct upgrade source | v1.0.0 / `20260827_0021` |
| PostgreSQL | 16+ (17 shipped) |
| Agent protocol | 1 |
SQLite is used by isolated unit tests only and is not a supported production database.
## Known limitations
- Point-in-time recovery is `NOT_SUPPORTED`; verified snapshot restore is supported.
- The historical M13/M15 LAB cutover journal remains open because Qdrant external truth is not
observable. It references no production deployment, active gateway request or scheduler plan.
- Project-specific Vision, OCR, ASR and ExampleRAG runtime validation remains independently governed
and is not silently promoted by this release.
- `VISION_AUTO_ACCEPT_ENABLED=false` remains the ExampleVision production safety setting.
docs/RELEASE_NOTES_v1.1.1.md
+39
View File
@@ -0,0 +1,39 @@
# ITWorx ModelForge v1.1.1
v1.1.1 is a production fix-forward patch for the v1.1.0 Node Agent NVIDIA blocker. The v1.1.0
API, web application and schema `20260828_0022` remain compatible and unchanged in contract.
## Fixed
- The Node Agent no longer uses Alpine/musl. It uses a digest-pinned Debian trixie-slim/glibc
runtime compatible with the host NVIDIA libraries injected by NVIDIA Container Toolkit.
- NVIDIA-configured nodes validate NVML, non-empty accelerator inventory and matching GPU
telemetry before enrollment or publication. Failure is typed and fail-closed.
## Production impact
The v1.1.0 API, web application and database were valid, but its published Alpine/musl Node Agent
could not load GPU Node's glibc-linked `libnvidia-ml.so.1`. Production therefore retained the known-good
v1.0.0 Debian Node Agent until this fix-forward release.
## Compatibility
| Contract | v1.1.1 |
| --- | --- |
| Database schema | `20260828_0022` (no migration from v1.1.0) |
| Agent protocol | `1` |
| API | v1.1.x compatible |
| Node identity / credential | preserved; no re-enrollment |
## Security and packaging
The runtime is multi-stage, non-root UID 100/GID 101, read-only under Compose, capability-free and
`no-new-privileges`. Build tooling and pip/setuptools are absent from the final image. NVIDIA driver
libraries are not bundled; the host runtime injects its compatible driver userspace.
## Known limitations
- Unmanaged GPU workloads can legitimately cause `EXTERNAL_GPU_PRESSURE`. ModelForge reports and
respects that pressure; it does not terminate external workloads.
- Point-in-time database recovery remains `NOT_SUPPORTED`; verified snapshot restore is the
production recovery model.
docs/RELEASE_NOTES_v1.2.0.md
+104
View File
@@ -0,0 +1,104 @@
# ITWorx ModelForge v1.2.0
v1.2.0 is the Premium Operator Console release. It delivers the M18 operator experience — a live
command center, a domain-oriented information architecture, a refined design system and a
consistent interaction vocabulary — and closes the three accessibility and form-semantics gaps that
M18's own final audit found.
It changes no API contract, no domain model, no runtime behavior and no database schema. An
operator upgrading from v1.1.1 replaces application images and nothing else.
## Added
- **Production command center.** A dashboard built only from backend-authoritative evidence:
platform posture, production state, GPU Node compute, schedulable headroom and capability health,
each stating an unknown as unknown rather than as zero.
- **Quick jump.** A modal command palette (Ctrl+K) over all 13 workspaces, matching on workspace
name, domain and keywords, with direct Enter navigation.
- **Evidence timeline.** Recent control-plane activity on the dashboard, linking each entry to the
workspace where it can be verified.
- **Split-pane Registry.** Upstream facts, local governance, exact revisions, artifact state and
provenance in one traceable detail view.
- **Guarded destructive actions.** A single console-wide safety dialog that shows identity,
dependency posture and permanent impact, and keeps its confirm control disabled until the operator
explicitly acknowledges the consequence.
## Changed
- **Information architecture.** Workspaces are grouped by operator mental model — Command, Model
supply, Serving, Infrastructure, Assurance, Projects — while every existing route hash is
preserved. No route moved and no placeholder route was introduced.
- **Design system.** One refined token layer: a calm navy surface ramp in dark mode, clean cool
neutrals in light mode, semantic status roles that always include text, a 4px spacing grid,
explicit focus rings and Geist-first system typography with durable fallbacks.
- **Responsive console.** Every route is reachable and usable from 1440 px down to a 390 px mobile
viewport through a modal navigation drawer, with no horizontal overflow at any tested width.
- No runtime font download, chart library or UI framework was introduced.
## Fixed
- **The ARIA tabs pattern is now complete across all six tabbed workspaces** — Registry,
Capabilities, Operations, Recovery, Lifecycle and Migrations. Every tab carries a stable `id` and
`aria-controls`, and every panel is a real `role="tabpanel"` naming its tab through
`aria-labelledby`. Assistive technology can now relate a tab to the region it controls.
- **Roving keyboard movement now works on every tablist.** Lifecycle, Migrations and Recovery
previously had tab roles without roving `tabindex`, an accessible tablist name, or
Left/Right/Home/End handling. All six now share one implementation.
- **The last native `window.confirm` is gone.** Capability residency unload is guarded by the same
safety dialog as every other destructive action, showing channel, residency state, active
requests, the VRAM that will be released and the cold-load cost the next request will pay.
- **Operator token fields have real form semantics.** The Operations, Recovery, Lifecycle and
Migrations credential inputs now sit inside a real `<form>` with a stable `id`, a `name` and an
explicit `<label for>`, resolving the browser's "password field is not contained in a form" and
"form field element should have an id or name attribute" issues.
## Accessibility
- A skip link targets the main workspace landmark, which is programmatically focusable.
- Quick jump and the mobile drawer are modal dialogs with focus containment, Escape dismissal and
focus restoration to the control that opened them.
- Status is never encoded by colour alone.
- `prefers-reduced-motion: reduce` collapses animation and transition durations; `forced-colors:
active` re-expresses focus and active navigation in system colours.
- Essential mobile controls and every navigation row meet a 44 px target.
## Security
The operator credential remains request-scoped. It is held in browser memory for the duration of the
request and is written to no `localStorage`, no `sessionStorage`, no cookie and no URL. Each token
form prevents its own default submission, so no credential is ever serialized into a query string.
`autocomplete="off"` is retained deliberately on every operator-token field. `current-password` was
considered and rejected: it invites a browser password manager to persist and sync a
high-privilege control-plane credential, which contradicts the non-persistence guarantee the console
makes to the operator. The gap being fixed here was tooling semantics — form membership and stable
identifiers — not autofill participation.
## Compatibility
| Contract | v1.2.0 |
| --- | --- |
| Version | `1.2.0` |
| Channel | `stable` |
| Database schema | `20260828_0022` — unchanged from v1.1.0/v1.1.1, no migration |
| Agent protocol | `1`, accepts `1` |
| Minimum upgrade source | `v1.0.0` |
| Minimum PostgreSQL major | `16` |
The Node Agent's functional behavior is unchanged from the NVIDIA-certified v1.1.1 implementation;
it is rebuilt and relabelled only because `VERSION` is the single source of truth for every
packaged manifest. No node re-enrollment is required and node identity and credentials are
preserved.
## Upgrade
Application-only. Schema is `20260828_0022` before and after, so no migration runs.
## Known limitations
- Unmanaged GPU workloads can legitimately cause `EXTERNAL_GPU_PRESSURE`. ModelForge reports and
respects that pressure; it does not terminate external workloads.
- Point-in-time database recovery remains `NOT_SUPPORTED`; verified snapshot restore is the
production recovery model.
- The console has no committed visual-regression baseline. Visual acceptance is performed by a
human-reviewed browser pass, so future styling drift is not caught automatically.
docs/RELEASE_NOTES_v1.2.1.md
+93
View File
@@ -0,0 +1,93 @@
# ITWorx ModelForge v1.2.1
A packaging and provenance patch over v1.2.0. No new capability, no UX change, no API contract
change, no schema change and no runtime-behaviour change. Schema stays `20260828_0022` and no
migration runs.
v1.2.0 is functionally healthy and remains historical truth, including the fact that its production
acceptance found two defects after publication. v1.2.1 fixes them forward rather than reissuing
v1.2.0.
Both defects share a shape worth naming: **the source was correct and the artifact was wrong.** No
amount of unit testing, linting or label inspection could see either, because both only exist once
an image has been built.
## Fixed
### The published console could not reach its own API
Vite inlines `VITE_API_BASE_URL` into the bundle at build time. `scripts/release_build.py` never
passed it, so every release image since v1.1.0 compiled the Dockerfile's development default —
`http://localhost:8000` — into an immutable artifact. The nginx CSP is derived from the same build
argument, so `connect-src` hardcoded the same wrong origin: the console asked the wrong address, and
its own policy forbade the right one.
In production this presented as a console that loaded perfectly and displayed nothing, with
CORS errors visible only in a browser devtools console.
A release build now takes the origin as an explicit input and refuses to package without it:
```bash
python scripts/release_build.py --output dist \
--public-api-origin https://modelforge.example.com
# or: MODELFORGE_PUBLIC_API_ORIGIN=https://modelforge.example.com
```
The value must be an absolute `http(s)` origin with no path, query or fragment. A missing or
malformed origin fails the build before any image is built.
### The Node Agent's identity could drift from its tag
`docker-compose.node-agent.yml` declared a `build:` block with no `args:`. A Compose-built agent
therefore fell back to version `0.0.0` with empty `revision` and `created` labels, while still being
tagged from whatever `MODELFORGE_VERSION` happened to be. Production ran an image tagged `1.1.1`
whose contents reported `1.2.0` and whose OCI provenance was blank.
The projection now passes `MODELFORGE_VERSION`, `MODELFORGE_COMMIT` and `MODELFORGE_BUILT_AT`,
exactly as the API and console projections already did. The Node Agent's certified behaviour is
unchanged: same Debian/glibc base, NVML preflight, fail-closed NVIDIA validation, non-root runtime,
`cap_drop: ALL`, `no-new-privileges`, protocol 1 and persisted-identity semantics.
## Added
`scripts/release_image_acceptance.py` reads the **built** console image rather than the source that
produced it: the origins compiled into its JavaScript, the `connect-src` in its rendered nginx
policy, and its OCI labels. The release build runs it before packaging and refuses to publish an
image that cannot reach the API it was built for.
It was validated against the exact image that broke production — `modelforge-web:1.1.0` fails three
of its checks — and against the corrected image, which passes all thirteen.
Regression coverage was added for both defects, including proof that each new test fails when its
fix is reverted.
## Compatibility
| Contract | v1.2.1 |
| --- | --- |
| Version | `1.2.1` |
| Channel | `stable` |
| Database schema | `20260828_0022` — unchanged, no migration |
| Agent protocol | `1`, accepts `1` |
| Minimum direct upgrade | `v1.0.0` |
| Minimum PostgreSQL major | `16` |
## Upgrade
Application-only from any v1.2.x. Replace the API, console and Node Agent images with the published
v1.2.1 artifacts. Schema is `20260828_0022` before and after.
Operators upgrading from v1.1.0 or the original v1.2.0 images should note that the console image is
the one that actually changes behaviour here: the previous images cannot reach a non-localhost API,
regardless of how the deployment is configured, because the origin is compiled in.
## Known limitations
- The console's API origin remains a **build-time** property. A single published console image is
therefore bound to one API origin; deployments behind a different origin need an image built for
it. Making this a runtime property is a larger change than a patch release should carry.
- Node Agent HIGH/CRITICAL findings from its digest-pinned Debian base are unchanged from v1.1.1 and
v1.2.0, and none has an upstream fix available.
- Unmanaged GPU workloads can legitimately cause `EXTERNAL_GPU_PRESSURE`; ModelForge observes but
never reclaims them.
- Point-in-time database recovery remains `NOT_SUPPORTED`.
docs/RELEASE_NOTES_v1.2.2.md
+46
View File
@@ -0,0 +1,46 @@
# ITWorx ModelForge v1.2.2
A security and public-packaging patch over v1.2.1. The API contract and agent protocol remain
unchanged. Runtime Worker dependency versions change and therefore require fresh LAB compatibility
evidence before promotion; no production capability is changed automatically.
## Security
- The Runtime Worker now upgrades its digest-pinned Ubuntu base during the image build.
- Build-only Linux kernel headers are removed after dependency installation instead of remaining in
the running Runtime Worker image.
- Pillow, protobuf, SentencePiece, Sentence Transformers, Transformers and inherited Python
security dependencies are pinned to reviewed fixed versions.
- The public-candidate acceptance gate builds and scans all four images, binds each Trivy report to
its exact image ID and fails on every fixable or newly observed HIGH/CRITICAL vulnerability.
- The Node Agent's upstream-unfixed Debian findings are non-blocking only through an exact reviewed
CVE/package/version/severity baseline. Any drift requires a new review.
- PostgreSQL provisioning revokes the default public temporary-table privilege before granting the
separate migration-owner and runtime roles; the runtime startup attestation fails closed if that
boundary is weakened.
## Reliability
- Schema `20260830_0024` now passes PostgreSQL percent syntax through Psycopg without interpreting
`%rowtype` or `%I` as client placeholders.
- Server acceptance supports a sibling Docker daemon without weakening production Compose: the
exact bootstrap SQL and reviewed configuration are transferred through the Docker API, and HTTP
contracts are probed inside the isolated candidate network.
## Compatibility
| Contract | v1.2.2 |
| --- | --- |
| Version | `1.2.2` |
| Channel | `stable` |
| Database schema | `20260830_0024` |
| Agent protocol | `1`, accepts `1` |
| Minimum direct upgrade | `v1.0.0` |
| Minimum PostgreSQL major | `16` |
## Upgrade classification
The control plane and Node Agent change is `transparent`. The Runtime Worker inference dependency
update is `behavioral`: exact LAB probes, output-shape checks and unload/reclaim evidence are
required before an existing runtime profile can be promoted. Embedding-space identities and project
indexes are not reused or changed automatically.
docs/SECURITY.md
+87
View File
@@ -0,0 +1,87 @@
# Security
The operator-facing summary. The reasoning behind each control is in
[THREAT_MODEL.md](security/THREAT_MODEL.md); the supply-chain rules are in
[MODEL_SUPPLY_CHAIN_POLICY.md](security/MODEL_SUPPLY_CHAIN_POLICY.md).
## Trust boundaries
| Boundary | Rule |
| --- | --- |
| Operator ↔ control plane | Every admin route requires the operator API key |
| Application ↔ control plane | A service credential scoped to specific capabilities |
| Control plane ↔ compute node | The node authenticates outbound; the control plane never dials a node |
| Control plane ↔ upstream model source | Acquisition only, into quarantine, verified before use |
An operator credential and a capability credential are different things and cannot substitute for
each other — `capability_clients_are_not_operators` is asserted as a platform invariant.
## Model code is never executed
`trust_remote_code` is `false` and production **refuses to start** if it is anything else. A model
repository cannot ship Python that ModelForge will run. Artifacts are verified per file by SHA-256
against the recorded upstream identity before leaving quarantine, and an artifact that fails stays
there.
## Secrets
| Secret | Stored as | Notes |
| --- | --- | --- |
| Operator API key | configuration only | never persisted by the platform |
| Service credential | SHA-256 hash plus a short prefix | cannot be read back; rotate if lost |
| Node credential | SHA-256 hash | single-use enrolment, atomically claimed |
| Backup encryption key | configuration only | **store it outside this deployment** |
| Hugging Face token | configuration only | acquisition only; never passed to a runtime |
ModelForge never generates its own credentials. A platform that mints its own admin secret has no
way to tell you it did. Generation procedures are in [CONFIGURATION.md](CONFIGURATION.md).
Revocation is permanent, expiry is enforced, a disabled client cannot serve with a valid secret, and
rotation leaves exactly one usable secret.
## Network exposure
By default only the API is published. PostgreSQL, Redis and the console bind to loopback, and a test
fails if any Compose projection publishes a datastore more widely — including the disaster-recovery
projection, which was found doing exactly that during the v1 gate.
## Container hardening
No privileged containers. No container mounts the Docker socket. No host network or PID namespace.
The API, Node Agent and console drop all capabilities and set `no-new-privileges`; the Node Agent and
console run read-only root filesystems. The console serves a static build from an unprivileged
nginx — never a development server.
## Console security headers
Content-Security-Policy with `default-src 'none'`, `script-src 'self'` and no `unsafe-inline` or
`unsafe-eval`; inline style is permitted only as an attribute via `style-src-attr`, which covers the
dynamic width bars the console uses while still blocking an injected `<style>` element. `connect-src`
is derived at build time from the same API base URL compiled into the bundle, so the policy cannot
drift from the origin the bundle calls.
Also: `X-Content-Type-Options`, `X-Frame-Options`, `Referrer-Policy`,
`Cross-Origin-Opener-Policy`, `Cross-Origin-Resource-Policy` and `Permissions-Policy` — on the entry
document **and** on hashed assets.
## Errors
Every error response is exactly `code`, `message` and `correlation_id`. A 500 body is exactly
`Internal Server Error`: no DSN, no driver name, no SQL, no path, no traceback.
## Backups
AES-256-GCM from a reviewed library, a fresh nonce per chunk, associated data binding each chunk to
its key id and index, and fail-closed decryption that removes both the partial file and the
destination. No encryption key appears in any manifest or audit event.
## Supply chain
Dependencies are locked; no floating specifier is permitted and a test enforces it. Python
dependencies are audited **inside the built images** rather than against a manifest, and the
frontend with `npm audit`. Each release publishes a CycloneDX SBOM bound to the image digests and
the source commit.
## Reporting a vulnerability
Report privately to the repository owner. Do not open a public issue.
docs/TROUBLESHOOTING.md
+190
View File
@@ -0,0 +1,190 @@
# Troubleshooting
Operational states you will actually meet, what each one means, and what to do. Most of these are
the platform refusing to guess — a refusal with a name is a better outcome than a plausible answer
built on stale information.
## The control plane will not start
### It exits with code 3 and a list of settings
```text
ModelForge refused to start: 7 configuration problem(s) must be resolved.
- [MISSING_REQUIRED_SETTING] MODELFORGE_OPERATOR_API_KEY: production requires an operator API key
- [INSECURE_PRODUCTION_SETTING] MODELFORGE_DATABASE_URL: the database password is a well-known
development value
...
```
This is startup validation, and it is deliberate. Every problem names the setting at fault. Fix them
and start again; see [CONFIGURATION.md](CONFIGURATION.md).
It only refuses in `production`. In development the same problems are logged and the process starts,
so you find out at startup rather than at first use.
### `INCOMPATIBLE_DATABASE`
The schema revision in front of the application is not one this release supports, or PostgreSQL is
older than major 16. The message says which of `TOO_OLD`, `TOO_NEW` or `UNKNOWN` applies and what to
do. See [COMPATIBILITY.md](COMPATIBILITY.md).
### Compose refuses to render the project
```text
required variable MODELFORGE_OPERATOR_API_KEY is missing a value
```
The production overlay takes secrets through `${VAR:?message}`, so a missing value fails before a
single container starts.
### `FileNotFoundError` on a manifest
The `config/` directory is not mounted, or is mounted empty. The deployment mounts `./config` from
the directory you run Compose in.
## Serving
### `503 NO_ELIGIBLE_NODE`
No online node can serve this capability. In order:
1. Is a node enrolled and `enabled`?
2. Is its liveness `online`, not `stale` or `offline`?
3. Does it have the accelerator the runtime profile requires?
A node goes `online → stale → offline` on the configured thresholds. Nothing is assigned to a node
that is not online — the platform will not send work to a node it cannot currently see.
See [RUNBOOK_NODE_OFFLINE.md](operations/RUNBOOK_NODE_OFFLINE.md).
### `CAPACITY_CONSTRAINED`
A node exists but has no headroom. **This is a correct outcome, not a failure.** The scheduler
refuses rather than risking an out-of-memory failure that would take down work already running.
VRAM held by processes ModelForge does not manage — Ollama, Plex, Tdarr, anything else on the host —
counts as external pressure. ModelForge observes it and never reclaims it: it will not stop, pause,
throttle or reconfigure a workload it does not own.
See [RUNBOOK_GPU_PRESSURE.md](operations/RUNBOOK_GPU_PRESSURE.md).
### `401 CAPABILITY_NOT_AUTHORIZED`
The credential is missing, invalid, revoked, expired, or not scoped to this capability. Status and
error code are identical across those cases so there is no enumeration oracle; the human-readable
message does distinguish them, which only helps a caller who already holds the secret.
Scope is exact. `rag.embedding@1` does not grant `rag.embedding@2`, appended whitespace does not
widen a scope, and a project-bound client cannot serve another project's binding.
### Stale telemetry blocks admission
Rather than extrapolating from an old reading, the scheduler refuses. A reading it does not have is
not a reading it can guess.
## Nodes
### `NVIDIA_NVML_UNAVAILABLE`
The Node Agent is configured for NVIDIA but cannot initialize NVML. It exits with code 3 before
enrollment, heartbeat, inventory, or telemetry publication. Confirm the host driver works with
`nvidia-smi`, NVIDIA Container Toolkit is installed, the container receives `/dev/nvidia*`, and the
runtime injects `libnvidia-ml.so.1`. Do not copy a driver library into the image: it must match the
host kernel driver. The v1.1.1 release image is Debian/glibc specifically for this injection path.
`NVIDIA_DEVICE_NOT_FOUND` means NVML initialized but enumerated no device;
`NVIDIA_TELEMETRY_UNAVAILABLE` means inventory exists but a listed GPU has no matching telemetry.
Use `MODELFORGE_AGENT_ACCELERATOR_MODE=cpu` only for a genuinely CPU-only node.
In `auto` mode, an inherited non-empty `NVIDIA_VISIBLE_DEVICES` is affirmative GPU evidence; clear
it or select `cpu` explicitly on a CPU-only deployment.
### `NODE_STALE` then `NODE_OFFLINE`
The agent has stopped reporting. The agent is outbound-only, so look at the agent side first: is the
process running, can it reach the control-plane URL, is its credential still valid?
A failed publication names the status, method and path — never the response body and never the
credential. If you see repeated failures with no cause, check that the agent's control-plane URL is
not still a placeholder.
### The node reappears with a new identity
It should not. Identity is persisted and enrolment is single use. If it does, the persisted identity
file is not on durable storage — check `MODELFORGE_AGENT_STATE_VOLUME`.
## Artifacts
### `ARTIFACT_REHYDRATION_BLOCKED`
An artifact cannot be re-acquired from its recorded upstream identity. ModelForge will not fall back
to a different source: an artifact that is not the one recorded is not the artifact.
See [RUNBOOK_ARTIFACT_LOSS.md](operations/RUNBOOK_ARTIFACT_LOSS.md).
### An artifact is stuck in quarantine
It failed a supply-chain check. That is the check working. Quarantine is not a staging area to be
emptied — see [MODEL_SUPPLY_CHAIN_POLICY.md](security/MODEL_SUPPLY_CHAIN_POLICY.md).
## Backups and recovery
### `BACKUP_STALE`
The newest verified backup is older than the configured threshold. Take one, and find out why the
schedule did not.
### `BACKUP_FAILED` with `DESTINATION_UNAVAILABLE`
The backup root is missing, full or not writable. The backup failed closed and produced nothing —
there is no partial recovery point to mistake for a real one.
### A backup is `FAILED` with `MANIFEST_INCOMPLETE`
It was interrupted before completion and can never become restore eligible. Take a new one.
### `RESTORE_FAILED`
A restore stopped mid-phase. It is never silently resumed; it needs operator review.
See [RUNBOOK_DATABASE_RESTORE.md](operations/RUNBOOK_DATABASE_RESTORE.md).
### Point-in-time recovery
`NOT_SUPPORTED` in v1. Verified snapshot restore with a measured RPO is what the platform offers.
Continuous WAL recovery is deliberately out of scope.
## Migrations
### A cutover sits in an intermediate stage
Migration is the one subsystem that deliberately does not resolve itself after an interruption.
External alias truth cannot be inferred from a crash, so the cutover is reported and waits for an
operator working through the typed adapter. An upgrade does not resolve it either.
See [RUNBOOK_MIGRATION_FAILURE.md](operations/RUNBOOK_MIGRATION_FAILURE.md).
### `REQUIRES_REINDEX`
The embedding space changed. Vectors from the old space are not comparable with the new one, and no
amount of care makes them so. The migration engine will not swap an alias underneath data that has
not been reindexed.
## Observability
### `OBSERVABILITY_DEGRADED`
Monitoring persistence is unavailable. Serving, scheduling and recovery continue on their own
authoritative state — monitoring is never a safety dependency. The platform invariants deliberately
read none of the observability tables, so "is state still sound?" can be answered precisely when
monitoring is the thing that failed.
### `DEFERRED_EXTERNAL_VALIDATION`
A validation that depends on a service outside ModelForge could not run. It is recorded as deferred,
never as a pass.
## Getting more detail
Every error response carries `code`, `message` and `correlation_id` and nothing else. Search the
logs for the correlation id. A 500 response body is exactly `Internal Server Error` — no DSN, no
driver name, no SQL, no path, no traceback.
docs/UNRAID_DEPLOYMENT.md
+68
View File
@@ -0,0 +1,68 @@
# Deploying on Unraid
The reference deployment: an Unraid server ("GPU Node") acting as the GPU compute node, with the
control plane either on the same host or elsewhere.
For the general procedure see [INSTALLATION.md](INSTALLATION.md). This page is only what is
different about Unraid.
## Paths
Unraid's container-local storage does not survive an update. Everything ModelForge must keep goes on
`appdata` or an array share:
| What | Where | Why |
| --- | --- | --- |
| Agent identity | `/mnt/user/appdata/modelforge/agent-state` | A node that loses its identity re-enrols as a different node |
| Verified artifacts | `/mnt/user/appdata/modelforge/artifacts` | Large, and re-acquisition is expensive |
| Quarantine | `/mnt/user/appdata/modelforge/quarantine` | Holds artifacts mid-verification |
| Hugging Face cache | `/mnt/user/appdata/modelforge/hf-cache` | Avoids repeated downloads |
| Backups | an array share, ideally a different disk from the database | A backup on the disk it protects is not a backup |
Set them through `MODELFORGE_AGENT_STATE_VOLUME`, `MODELFORGE_AGENT_ARTIFACT_VOLUME`,
`MODELFORGE_AGENT_QUARANTINE_VOLUME`, `MODELFORGE_AGENT_HF_CACHE_VOLUME` and
`MODELFORGE_BACKUP_VOLUME` — all of them accept an absolute bind path as well as a volume name.
**Do not point the backup destination at the same disk as the PostgreSQL data directory.**
## GPU
The NVIDIA plugin and the NVIDIA container runtime must both be present. The Runtime Worker needs
device access:
```bash
docker compose -f docker-compose.yml -f docker-compose.gpu.yml \
-f docker-compose.runtime-worker.yml up -d runtime-worker
```
## External GPU workloads
An Unraid box usually runs other things that want VRAM — Ollama, Plex transcoding, Tdarr. ModelForge
**observes** that usage and counts it as external pressure in its admission decisions. It never
reclaims it: it will not stop, pause, throttle or reconfigure a workload it does not own, and it has
no mechanism to do so.
The practical consequence: if Plex starts a transcode, ModelForge may begin refusing placements with
`CAPACITY_CONSTRAINED`. That is the intended behaviour. Size your GPU for the sum of what you run, or
accept that the two compete.
## Networking
The agent is outbound-only, so GPU Node needs to reach the control plane and the control plane needs no
route to GPU Node. If the control plane runs elsewhere behind a private CA, use the private-CA overlay
rather than disabling TLS verification — see [NODE_AGENT.md](NODE_AGENT.md).
## Updates
An Unraid OS update restarts containers. Nothing here depends on container-local state, so a node
comes back with the same identity — provided the state volume is on `appdata` as above.
Set `MODELFORGE_NODE_AGENT_IMAGE=modelforge-node-agent:1.1.1` (or the published digest) before a
release deployment. Compose keeps the optional local `build:` path, but production should run the
immutable released image and verify its OCI revision after recreation.
## Scheduled backups
M15 delivered scheduled backups; enabling them on GPU Node is an operator action, not something the
release does for you. `scripts/modelforge_scheduled_backup.sh` is the entry point. Verify one
execution after enabling, and check the recovery dashboard shows a recent verified backup.
docs/UPGRADE.md
+191
View File
@@ -0,0 +1,191 @@
# Upgrading to the current ModelForge RC candidate
> RC schema note: the repository's temporary `1.2.1` version has **not** been released with
> schema `20260830_0024`. Version selection/stamping is a separate pending release tranche. The
> following preflight is the exact contract for rehearsing an existing stable v1.2.1/schema-0022
> database before this candidate can be released; it must not be read as a READY announcement.
## Required v1.2.1/schema-0022 role provisioning before migration 0024
Schema 0024 changes the production authority boundary. The API may no longer run as the table owner
or the bootstrap superuser. Provisioning is deliberately not hidden inside Alembic because an
application migration must not mint or retain an administrator credential.
1. Take and verify the pre-upgrade backup, then stop `api`, `web`, and every older writer. This is a
downtime migration; the 0024 migration also takes an advisory lock and `ACCESS EXCLUSIVE` on
`audit_events`, but the stopped-application contract prevents an old process from resuming with
its former owner credential.
2. Generate three distinct URL-safe secrets: retained PostgreSQL administrator, non-superuser
`modelforge` migration/schema owner, and NOINHERIT `modelforge_runtime` API role. Set
`MODELFORGE_POSTGRES_ADMIN_USER` to a new retained name such as `modelforge_admin`; set the three
password variables and both owner/runtime SQLAlchemy URLs documented in `CONFIGURATION.md`.
3. Recreate only the PostgreSQL container so those values are in its environment. It may report
unhealthy until the new admin role exists; that is expected during this bounded downtime. While
still connecting with the existing v1.2.1 `modelforge` bootstrap superuser, run:
```bash
docker compose exec -T postgres psql -v ON_ERROR_STOP=1 \
-U modelforge -d "$MODELFORGE_POSTGRES_DB" \
< deploy/postgres/provision-existing-1.2.1.sql
```
The script first creates the retained administrator, then preserves the existing `modelforge`
role OID/table ownership while demoting it to a non-superuser, creates the distinct non-owner
runtime role, and purges stale memberships that would still permit `SET ROLE`. Passwords are
read from process environment; none is in the SQL file.
4. Run only the one-shot migration service. Its environment contains
`MODELFORGE_MIGRATION_DATABASE_URL` but not the runtime/admin URL:
```bash
docker compose -f docker-compose.yml -f docker-compose.production.yml run --rm migrate
```
Migration 0024 fails before schema mutation unless both roles exist, `modelforge` is the current
directly authenticated non-superuser owner, `modelforge_runtime` is NOINHERIT/non-admin, and
neither application role has an explicit role membership.
The refusal explicitly says to run this v1.2.1-to-schema-0024 provisioning step.
5. Start the API only after Alembic reaches 0024. Production startup checks live catalog truth:
runtime role/ownership/admin flags, table/schema/database privileges, canonical function
owner/SECURITY DEFINER/fixed search path/grants and exact body digest, PUBLIC execution, four
exact origin-mode protection-trigger shapes, role memberships, and every unexpected executable
non-system function. Any drift stops startup.
The live PostgreSQL privilege/function exercise is a managed-runner release gate. SQLite exercises
only the defense-in-depth application Engine hooks and is never production proof.
Supported direct schema sources for this unreleased candidate are `20260827_0021`,
`20260828_0022`, and `20260830_0023`; the target is `20260830_0024`. Older releases require an
intermediate upgrade to v1.0.0.
## What this upgrade actually changes
The migration chain first adds the 0022 terminal-node/decommission state when needed, then 0023
validates every existing node-enrollment and node-publisher scope before adding exact database
constraints. Any row outside `node.enroll` or `node.publish` aborts the upgrade before constraint
DDL; it is never silently rewritten.
That has a consequence worth stating plainly: **rolling back across these schema revisions needs
the verified pre-upgrade database restore.** Older applications do not claim compatibility with
the newer schema.
## 1. Preflight
```bash
python scripts/upgrade.py --database-url postgresql+psycopg://user:pass@host:5432/modelforge --plan
```
It changes nothing and reports:
| Check | Blocks the upgrade |
| --- | --- |
| PostgreSQL major version | yes |
| Current schema revision is one this release supports | yes |
| The schema is complete, not half-migrated | yes |
| A verified backup exists | yes (overridable) |
| That backup's age | no, advisory |
| Serving work in flight | no, advisory |
| Enabled nodes speak a supported agent protocol | yes |
| A migration cutover sitting in an intermediate stage | no, advisory |
A refusal looks like this and is the tool working:
```text
BLOCK current schema 20990101_0099 is TOO_NEW; upgrade the control plane, which is older
than the component reporting to it
BLOCK verified backup no verified backup exists; an upgrade without a recovery point cannot be
undone if the schema turns out to be irreversible
BLOCK agent protocol enabled nodes speak [2]; this release supports [1]; incompatible: [2]
refusing to upgrade: 3 blocker(s)
```
## 2. Take a backup, and verify it
The preflight refuses without one. If you have taken a backup by other means, `--allow-without-backup`
proceeds — that override is explicit and recorded in the report, never a default.
See [OPERATIONS.md](OPERATIONS.md#backups-and-recovery).
## 3. Stop the application
```bash
docker compose -f docker-compose.yml -f docker-compose.production.yml stop api web
```
Leave PostgreSQL and Redis running. **This upgrade requires downtime**: the control plane is stopped
while the new version starts. Rolling, zero-downtime upgrade is not supported in v1 and is not
claimed. Record the measured downtime for your deployment.
## 4. Migrate
```bash
python scripts/upgrade.py --database-url postgresql+psycopg://user:pass@host:5432/modelforge --apply
```
For the current candidate this reports `schema migrated: <source> -> 20260830_0024`. Runtime
compatibility begins only after the scope validation, audit-chain checkpoint and PostgreSQL role
boundary reach `0024`.
## 5. Start the new version
```bash
MODELFORGE_VERSION=1.2.1 \
MODELFORGE_API_IMAGE=modelforge-api:1.2.1 \
MODELFORGE_WEB_IMAGE=modelforge-web:1.2.1 \
docker compose -f docker-compose.yml -f docker-compose.production.yml up -d --no-build api web
```
Pin `MODELFORGE_API_IMAGE` and `MODELFORGE_WEB_IMAGE` to the exact release digests from the release
manifest when the images are available through a registry. A local source build is an explicit
alternative for development or rehearsal only: pass `MODELFORGE_COMMIT`, `MODELFORGE_BUILT_AT` and
`--build` yourself. Never overwrite a release tag with such a local build.
## 6. Verify
```bash
curl -s http://127.0.0.1:8000/api/v1/version # expect 1.2.1 and your commit
curl -s http://127.0.0.1:8000/api/v1/health/ready
```
Then compare the pre/post authoritative-state fingerprint: models, revisions, artifact sets,
runtime profiles, projects, bindings, deployments, lifecycle and migration state, recovery state,
policies, SLO state, node identities and credential hashes. Only the expected 0022 structures,
0023 scope constraints, 0024 audit boundary and legitimate reconciliation/audit activity may differ.
Two changes are expected and are not damage: the liveness monitor writes a `NODE_BECAME_OFFLINE`
event for any node not reporting yet, and the alert engine records the transitions that follow. Both
are the platform observing its new surroundings correctly.
## If the upgrade fails
| Symptom | What it means | What to do |
| --- | --- | --- |
| Preflight blocks | The upgrade never started; nothing changed | Fix what it names and re-run |
| The API exits during startup | Configuration was refused before any request was served; the exit code is 3 and the log lists every problem | Fix the settings it names and start again |
| The API starts but is unhealthy | Something beyond configuration | Roll back, then investigate with the deployment up |
### Rolling back across schemas 0022, 0023 and 0024
Application rollback alone is not enough. Restore the verified backup taken in step 2, then start
the previous application version — see
[RUNBOOK_DATABASE_RESTORE.md](operations/RUNBOOK_DATABASE_RESTORE.md). The 0022 → 0021 Alembic
downgrade is retained and release-tested to prove DDL mechanics on an isolated, tombstone-free
copy; 0023 → 0022 only removes the two scope constraints. A 0024 downgrade is refused after v2 audit
events exist because rewriting accepted audit history would destroy the evidence the boundary was
designed to preserve. These DDL paths are not the production rollback promise;
the 0022 → 0021 step removes terminal node tombstones and
decommission-operation evidence. Production rollback therefore restores the verified pre-upgrade
backup instead of discarding history created after the upgrade.
## Agent compatibility
The control plane speaks agent protocol version 1 and accepts version 1. An agent reporting anything
else is answered explicitly — `TOO_OLD`, `TOO_NEW` or `UNKNOWN` — and told what to do, rather than
being silently accepted. The preflight blocks the upgrade when an enabled node speaks a protocol
this release does not support, so you find out before the control plane restarts rather than after.
Node Agents do not need to be upgraded in lockstep because protocol 1 is unchanged. Pin and deploy
the matching Node Agent image to align package and OCI provenance. Upgrading only the Node Agent
does not require re-enrollment; the control plane applies the fixed-scope migration before serving.
The image deliberately preserves runtime UID 100 and GID 101, so the existing durable agent-state
volume retains its ownership contract.
docs/architecture/ADVANCED_GPU_SCHEDULER.md
+61
View File
@@ -0,0 +1,61 @@
# Advanced GPU Scheduler
M10 turns the single-GPU admission boundary into one deterministic planner used by dry-runs and
real requests. The planner consumes current NVML use, managed residency, only the unmaterialized
part of active leases, authoritative envelopes, health, queue state, priority and versioned policy.
## Accounting invariant
NVML is the physical truth and already includes loaded ModelForge models:
```text
physical_used = max(observed_nvml_used, attributed_managed_resident)
unmanaged_external = max(0, observed_nvml_used - attributed_managed_resident)
future_lease = max(0, reserved - materialized)
schedulable = max(0, total - physical_used - future_leases - dynamic_reserve)
```
This deliberately does not subtract residency twice. Attribution is `KNOWN`, `ESTIMATED` or
`UNKNOWN`; missing or stale telemetry is `UNKNOWN` and has zero schedulable capacity. The invariant
delta is exposed for diagnosis without permitting negative external use.
The active `m10-v1` policy uses the maximum of 1 GiB, 5% total VRAM and a 256 MiB runtime margin.
Deployment admission uses measured peak plus the maximum of 128 MiB or 10% measured peak. These
values are one versioned policy, not dispersed constants.
## PlacementPlan
Every plan records capability, node and accelerator, eligible state, residents, unmanaged use,
reserve, requirement, priority, evictions, before/after headroom, reason codes and a deterministic
fingerprint. Verdicts are `ADMIT`, `ADMIT_AFTER_EVICTION`, `QUEUE`, `REJECT_CAPACITY`,
`REJECT_HEALTH` and `REJECT_POLICY`. Recent history is bounded to 500 records.
Pressure worsens immediately through `NORMAL`, `ELEVATED`, `HIGH`, `CRITICAL`. Recovery requires a
separate threshold and 30 seconds of stable evidence. At HIGH/ELEVATED, new background/LAB loads
wait; at CRITICAL, new admissions fail with `EXTERNAL_GPU_PRESSURE`. ModelForge never changes an
unmanaged process.
## Operations and security
Operator controls are limited to dry-run, policy revision/LAB pause, managed residency policy,
drain and idle unload. There is no shell, Docker socket, SSH or GPU-PID control. Ordinary clients
select only a capability. Scheduler evidence contains bounded shapes and hashes, never content.
M11 validates the existing scheduler rather than adding a project shortcut. ExampleVision reference
index requests use a background-bound client, while card and speech requests use interactive-bound
clients. During a 40-reference indexing continuation, ten interactive vision requests all completed.
Vision and ASR also ran under observed external GPU pressure without OOM; final unload returned
managed residency and leases to zero.
## M16 pressure behaviour
Under GPU pressure the scheduler rejects or queues rather than risking an out-of-memory failure. A
capacity rejection is a correct outcome, not a failure, and the soak counts those separately from
internal errors for exactly that reason.
External VRAM is measured as pressure and never reclaimed: Ollama, Plex, Tdarr and any other
unmanaged process are observed, never stopped, paused, throttled or reconfigured.
Stale telemetry blocks admission rather than being extrapolated, and `no_stale_gpu_lease` asserts
no expired lease is still presented as a live reservation.
docs/architecture/ADVISOR_EVIDENCE.md
+37
View File
@@ -0,0 +1,37 @@
# Advisor evidence
## M9 capability recommendations
`/api/v1/capability-advisor` evaluates each contract independently. No active deployment yields `REQUIRES_MORE_EVIDENCE`; a proven LAB deployment without a typed suite yields `LAB_READY`; completed evidence satisfying every declared threshold yields `PROMOTION_ELIGIBLE`; failed cases, failed runs or threshold regressions yield `BLOCKED`. These states are advisory and never trigger promotion. Retrieval's comparison-specific `KEEP_CURRENT` policy remains intact.
Advisor recommendations are deterministic outputs of a versioned persisted policy. Project-local
evaluation is evidence class A and outranks generic, external and vendor evidence. Critical
regressions, supply-chain failures and runtime-fit failures are hard gates; latency remains a distinct
operational dimension.
Each recommendation preserves metric deltas, critical regressions, resource evidence, migration
impact, blockers, rationale, confidence and policy version. Dismissal is supported, but Advisor has no
approve or promote endpoint. M7's real ExampleRAG recommendations legitimately resolve to
`KEEP_CURRENT` or `REQUIRES_MORE_EVIDENCE`.
M10 lets Advisor consume scheduler readiness, authoritative ResourceEnvelope freshness and measured
co-residency as operational evidence. It may explain that a candidate is technically eligible but
not proven for sustained coexistence. Scheduler evidence never overrides quality, supply-chain,
migration or approval gates and never triggers an automatic promotion.
M11 adds immutable project-fit evidence. It is scoped to one project binding and cannot make a global
“best model” claim. Advisor exposes engineering integration separately from project production fit:
ExampleVision vision engineering is `PASS`, while project fit remains `REQUIRES_MORE_EVIDENCE`, production
validation is `DEFERRED_EXTERNAL_VALIDATION` and production action is `NONE`. Its speech flow is
`KEEP_LAB` because four TTS cases do not establish microphone quality. Only reviewed `OWNER_PHOTO`
evidence may satisfy the ExampleVision production gate; public physical captures and catalog references
retain their weaker typed class. Blockers or critical failures make `PROMOTION_ELIGIBLE` invalid, and
the referenced deployment must implement the binding's exact contract.
M12 consumes Advisor output only as evidence. Advisor may recommend `PROMOTION_ELIGIBLE`, but only
the lifecycle engine can evaluate the active policy revision, create an approval snapshot and permit
an immutable plan. There is no Advisor-to-promotion shortcut.
M13 may reference an Advisor/project-fit verdict in an immutable validation snapshot. Advisor still
cannot create, start, validate or cut over a migration; missing or adverse project evidence remains a
separate production blocker even when an isolated technical rehearsal passes.
docs/architecture/ADVISOR_PIPELINE_EVIDENCE.md
+15
View File
@@ -0,0 +1,15 @@
# Advisor Pipeline Evidence
Advisor candidates are explicitly typed as `embedding_deployment` or `retrieval_pipeline`.
Embedding candidates reference completed evaluation runs; pipeline candidates reference immutable
pipeline identities whose latest completed fixed-pool runs provide control/candidate metrics.
Hard gates remain deterministic: incomplete/non-comparable evidence, critical regression, security
or license block, runtime/GPU-fit failure, unsupported migration and latency policy override
aggregate gains. Unknown evidence stays unknown.
Pipeline recommendations may return
`KEEP_CURRENT_EMBEDDING_ADD_RERANKER_CANDIDATE`, but M8's two real reranking runs returned
`KEEP_CURRENT`. The evaluated E5 embedding also returned `KEEP_CURRENT`: its aggregate retrieval
metrics improved, but three critical regressions, unapproved license state, a required reindex and
the p95 latency policy block promotion. Recommendations never promote or cut over automatically.
docs/architecture/ALERTING.md
+48
View File
@@ -0,0 +1,48 @@
# Deduplicated alerting
`AlertRuleRevision` is an immutable, fingerprinted rule containing signal/SLO, condition, pending
duration, severity, bounded labels, cooldown and recovery condition. Changes create a revision.
## State machine
```text
clear → PENDING → FIRING → ACKNOWLEDGED
│ │ │
└──────────┴────────────┴→ RESOLVED
bad + maintenance window → SUPPRESSED
SUPPRESSED + window ended + still bad → PENDING
RESOLVED + cooldown elapsed + recurrence → PENDING/FIRING
```
Acknowledgement records actor/reason but cannot alter the signal or SLO. Resolution is an explicit
history event; alerts do not disappear. The deterministic fingerprint uses rule key, subject type
and subject reference, so every polling cycle updates one logical alert rather than creating spam.
Severities are `INFO`, `WARNING` and `CRITICAL`. Initial rules cover production node liveness, GPU
pressure, storage reserve, gateway errors, stable capability availability, runtime crash loops,
queue saturation, migration/manual-intervention failure, rollback failure, artifact integrity and
production capability envelope/headroom.
Maintenance windows have bounded matchers (`rule_key`, `alert_type`, `subject_ref`), start/end,
reason and creator. Creation is audited and suppression remains visible. M14 delivers persistent
records, API and UI; it has no mandatory Slack/email dependency and performs no automatic restart,
kill, promotion, rollback or deletion.
## M15 recovery rules
Five deduplicated rules extend the same lifecycle to recovery readiness:
```text
BACKUP_STALE WARNING no verified backup inside the policy window
BACKUP_FAILED CRITICAL a backup failed within the window
BACKUP_VERIFICATION_FAILED CRITICAL a backup failed specifically on hash or manifest verification
RESTORE_FAILED CRITICAL a restore failed or needs manual intervention
RECOVERY_READINESS_DEGRADED WARNING an authoritative asset is unprotected
```
They read the same authoritative records the recovery dashboard reads, so an alert and the dashboard
can never disagree. A `CREATED` backup that was never verified does not clear `BACKUP_STALE`.
Live evidence: `BACKUP_STALE` and `RECOVERY_READINESS_DEGRADED` fired while no verified backup
existed and resolved once one did, each as a single alert with an occurrence count rather than one
alert per evaluation.
docs/architecture/API_BLUEPRINT.md
+114
View File
@@ -0,0 +1,114 @@
# API Blueprint
## M5 inference boundary
- `POST /api/v1/capabilities/rag.embedding@1/invoke` — native capability response.
- `POST /v1/embeddings` — documented OpenAI embedding subset using alias `rag.embedding`.
- `GET /api/v1/capability-deployments` and `GET /api/v1/scheduler` — public operational state.
- `/api/v1/admin/service-clients`, production approval/promotion, request history and lifecycle
actions require the operator credential.
- `/api/v1/agent/serving-*` is node-credential-only and never project-facing.
All inference failures use typed codes. `X-Correlation-ID` and the response request ID support
end-to-end tracing without retaining request content.
This document defines intended API surfaces, not frozen implementation details.
## Control-plane API
Operator/UI-facing resource surfaces:
```text
GET /api/v1/system
GET /api/v1/health/live
GET /api/v1/health/ready
GET /api/v1/hardware/nodes
GET /api/v1/hardware
GET /api/v1/hardware/nodes/{id}
GET /api/v1/hardware/accelerators
GET /api/v1/hardware/accelerators/{id}
POST /api/v1/hardware/refresh
POST /api/v1/admin/node-enrollments
GET /api/v1/admin/node-enrollments
DELETE /api/v1/admin/node-enrollments/{id}
PATCH /api/v1/admin/hardware/nodes/{id}
DELETE /api/v1/admin/hardware/nodes/{id}/credential
POST /api/v1/admin/hardware/nodes/{id}/credential/rotate
POST /api/v1/admin/hardware/nodes/{id}/decommission/preview
POST /api/v1/admin/hardware/nodes/{id}/decommission
GET /api/v1/models
GET /api/v1/models/{id}
GET /api/v1/models/{id}/revisions
POST /api/v1/models/discover
POST /api/v1/models/{id}/download
GET /api/v1/artifacts
GET /api/v1/deployments
POST /api/v1/deployments
POST /api/v1/deployments/{id}/validate
POST /api/v1/deployments/{id}/promote
POST /api/v1/deployments/{id}/rollback
GET /api/v1/capabilities
GET /api/v1/projects
GET /api/v1/projects/{id}/bindings
GET /api/v1/benchmarks/suites
POST /api/v1/benchmarks/runs
GET /api/v1/recommendations
GET /api/v1/jobs
GET /api/v1/security/findings
GET /api/v1/audit/events
```
Remote compute-node agent surface (protocol v1):
```text
POST /api/v1/agent/enroll
POST /api/v1/agent/heartbeat
PUT /api/v1/agent/inventory
PUT /api/v1/agent/telemetry
```
Agent endpoints accept no arbitrary command or runtime-launch payload. See `NODE_AGENT_PROTOCOL.md`.
All lifecycle-changing operations should become asynchronous jobs once work can exceed a normal request duration.
## Project-facing capability gateway
Native surface:
```text
POST /api/v1/capabilities/{capability-key}
```
Project identity must be authenticated. Capability version/channel may be selected only within policy.
## Compatibility facades
Where semantically safe, provide OpenAI-compatible or other standard interfaces so existing projects can redirect their base URL to ModelForge without coupling to a runtime worker.
A compatibility facade must still route through project authentication, capability resolution, scheduler policy, telemetry and audit.
## Error model
Normalized errors should identify the layer without exposing secrets:
- `capability_unavailable`
- `deployment_unhealthy`
- `scheduler_capacity_exhausted`
- `request_timeout`
- `invalid_capability_input`
- `policy_denied`
- `migration_required`
- `project_not_authorized`
Runtime-specific stack traces remain internal.
Request bodies are rejected before parsing with `request_body_too_large` (HTTP 413) when their
declared or streamed byte boundary is exceeded. The receiver also allows at most 32 consecutive
empty request events, 128 empty events in total and 4096 body events; exceeding any event/progress
budget returns exactly one `request_body_progress_exhausted` error (HTTP 400) and stops receiving.
Finite empty frames below those limits, normal chunks and disconnect events retain normal ASGI
semantics.
Node-decommission conflicts use `node_decommission_blocked`, `decommission_preview_stale`,
`node_generation_conflict` or `decommission_confirmation_mismatch`, with blocker details in the
normal correlation-ID error envelope. There is no force parameter.
docs/architecture/APPROVAL_POLICIES.md
+25
View File
@@ -0,0 +1,25 @@
# Approval Policies
`LifecyclePolicyRevision` is immutable apart from its active selector. Editing requirements creates
a new revision; existing requests retain their original revision and evidence fingerprint.
M12 bootstraps three policies:
- `lab-promotion`: exact identity, verified integrity, non-blocked security/license, proven runtime,
current scheduler readiness and rollback target;
- `capability-production`: exact production execution approval, completed evaluation, approved
security/license, scheduler evidence revision, zero critical regressions and rollback target;
- `project-production`: all capability requirements plus eligible project fit and satisfied project
production validation.
An `ApprovalRequest` freezes declared fields and database-resolved facts. Exact references include
ArtifactSet, ModelRevision, RuntimeProfile, RuntimeProbe, worker image digest, production execution
approval, evaluation runs, ProjectFitEvidence, ResourceEnvelope, embedding/pipeline identity,
migration and rollback target. Claims are checked against resolved relationships; green labels
cannot substitute for records.
Requests are `PENDING`, `APPROVED`, `REJECTED`, `BLOCKED`, `EXPIRED`, `STALE` or `REVOKED`.
Expiration is evaluated before use. Re-resolving evidence detects changed probes, profiles, fit,
approvals and other facts and makes the request stale. A deferred ExampleVision production gate produces
`REQUIRED_EXTERNAL_VALIDATION_NOT_SATISFIED`; security, license and critical regression gates cannot
be manually waved through by the default production policies.
docs/architecture/ARTIFACT_ACQUISITION.md
+57
View File
@@ -0,0 +1,57 @@
# Artifact Acquisition
## Flow
```text
Hub metadata -> exact revision -> artifact set -> immutable plan -> explicit approval
-> node-scoped lease -> quarantine download -> static verification -> atomic promotion
-> artifact/location registration
```
`DownloadPlan` captures the artifact set, exact SHA, selected paths and expected checksums/sizes,
target node/root, total bytes, capacity result, security constraints, expiry and a canonical
idempotency key. Creation is a dry run with no bytes. Approval is a separate audited action and
execution rejects a merely planned or stale plan.
The planner prefers complete advertised Safetensors weights plus configuration/index/tokenizer
support and excludes duplicate pickle/GGUF weights. If only risky pickle-style weights remain, the
risk is explicit and plan approval fails. No format conversion, quantization or loader execution
occurs.
## Placement and ownership
Jobs are claimed only by their `compute_node_id` through node-scoped bearer authentication. A lease
token binds progress/completion/failure calls to one claim. PostgreSQL row locking and one job per
plan enforce single-node concurrency and replay/idempotency boundaries. The outbound agent loop
processes one acquisition at a time and has no shell, generic command, Docker socket or inbound
listener.
Capacity is checked from the registered root during planning and execution, then checked again from
the real target filesystem by the agent before download. Failure never falls back to another root.
GPU Node uses `/data/artifacts/model-registry`, mapped to the cache-backed host root; the Unraid array
is not a placement candidate.
## Retry, cancellation, and recovery
Transient failures have at most three automatic claims. Partial files use `.part`, byte ranges and
durable flushes. A complete partial is verified without a new request. A failed job can receive an
explicit audited operator retry; its progress epoch resets but historical attempts remain. Cancel
is observed at progress boundaries and cannot promote an artifact.
Expired leases make work reclaimable after agent/control-plane interruption. If promotion succeeded
but the database response was lost, the exact-revision manifest makes promotion idempotent; the
same plan/job can reconcile. A verified existing plan returns the existing completed job rather
than downloading again.
## API
- `POST /api/v1/download-plans`
- `GET /api/v1/download-plans/{id}`
- `POST /api/v1/download-plans/{id}/approve`
- `POST /api/v1/download-plans/{id}/execute`
- `GET /api/v1/artifact-jobs[/{id}]`
- `POST /api/v1/artifact-jobs/{id}/cancel`
- `POST /api/v1/artifact-jobs/{id}/retry`
Agent routes are separately authenticated and expose only claim, typed progress, completion and
failure operations.
docs/architecture/ARTIFACT_PROVENANCE.md
+65
View File
@@ -0,0 +1,65 @@
# Artifact Provenance
## Content identity
An artifact is identified by its parent exact revision and SHA-256 digest. Filename, storage root
and path are descriptive/location data and cannot change identity. One artifact can therefore have
zero, one or many physical `ArtifactLocation` records across compute nodes and storage roots.
Expected digest and byte size are registered before a local file is trusted. Artifact state is one
of `remote`, `local`, `verifying`, `verified`, `missing`, `corrupt`, `quarantined` or `archived`.
## Verification boundary
M2 verification performs only static file operations:
1. resolve a relative path beneath the configured storage root and reject path escape;
2. require a regular file;
3. stream bytes in 1 MiB chunks into SHA-256 while counting bytes;
4. compare both digest and size;
5. persist observed digest/size/time, status and an audit event.
It never deserializes weights, imports repository Python, executes model code or contacts a model
hub. Missing files become `missing`; digest or size mismatch becomes `corrupt`; only a complete match
becomes `verified`. Failure keeps the artifact quarantined.
## Derived lineage
`DerivedArtifactSource` stores an ordered N-source relation. Every source ID and hash is captured.
The output also records transformation type, tool, tool version, complete configuration,
environment snapshot, output digest and byte size. M2 registers this provenance but performs no
conversion or quantization.
Derived content fields are immutable after registration. Source edges make source-artifact deletion
return HTTP 409. This preserves the ability to reconstruct why an output exists even when it has no
current physical location.
## Audit actions
M2 emits append-only hash-chained events for model creation/update/deprecation/deletion, revision
registration, artifact registration and verification, derived registration, storage-root validation,
capacity checks and blocked deletion attempts.
## M3 acquisition chain
M3 persists the complete chain from model and exact revision through upstream snapshot/file,
artifact set, immutable approved plan, target-node job/attempt, quarantine path, static inspection,
local SHA-256, verified artifact and physical location. Artifact-set membership records required
files explicitly. `static_checks_passed_unapproved` means local integrity evidence only and cannot
be confused with a later supply-chain or runtime approval.
M4 extends the chain without mutating artifact identity: exact-set approval fingerprints reviewed
security facts, static assessments snapshot artifact/hardware/runtime facts, probes record the same
ArtifactSet/revision/profile/node plus measured runtime facts, and candidates refer to the successful
probe. Runtime image digests and environment fingerprints are evidence, not artifact digests.
The promoted `.modelforge-manifest.json` is reconciliation evidence beside the bytes. The database
remains operational truth; a retry can reconcile an exact promoted directory after an interrupted
completion report without changing content identity.
## M12 lifecycle evidence
Approval snapshots add exact policy, ArtifactSet, revision, runtime/probe/image, evaluation, project
fit, scheduler, migration and rollback references without copying inference content. Cleanup retains
this chain even when one binary location is later removed: the prior digest, size, location status,
actor, plan and event remain queryable.
docs/architecture/ARTIFACT_RECOVERY.md
+72
View File
@@ -0,0 +1,72 @@
# Artifact recovery
ModelForge holds several gigabytes of model weights. Copying them into every backup would be
expensive and pointless for artifacts that are reproducible from an exact upstream identity, and
dangerous to assume for artifacts that are not. Artifacts are therefore classified, not treated
uniformly.
```text
REHYDRATABLE exact upstream repository + commit SHA + per-file SHA-256 exist
NON_REHYDRATABLE no exact upstream identity; requires a payload backup
DERIVED produced locally from exact source artifacts; lineage must survive a rebuild
LOCAL_ONLY never had an upstream; payload backup only
```
## Manifest rather than payload
For rehydratable artifact sets the backup stores `artifacts.json`: per set, the artifact set id,
revision id, recovery class, upstream repository, exact commit SHA, file count, total bytes and
every file with its filename, SHA-256, size, type, serialization format and security and licence
status. A live backup recorded 24 artifact sets covering 8,100,376,532 rehydratable bytes in a
39,276-byte manifest.
## Rehydration
Recovery reuses the existing acquisition plane rather than inventing a second download path:
```text
ModelRevision + exact commit SHA + expected file list + expected SHA-256
→ download plan against a disposable storage root
→ Node Agent downloads into quarantine
→ hash verification against provenance
→ atomic promotion
```
A floating revision is never used. `_revision_identity` resolves the repository and commit from the
upstream snapshot recorded for that revision, falling back to the model's upstream source; if
neither yields both, the set is classified `NON_REHYDRATABLE` and the recovery operation is created
already `BLOCKED` with `ARTIFACT_NOT_REHYDRATABLE` rather than attempting a download.
## Recovery is not a security bypass
A rehydrated artifact re-enters the supply chain as new material. It is written to quarantine,
hashed, checked against the security policy and promoted only on success, and it carries
`static_checks_passed_unapproved` — not the approval its predecessor held. `trust_remote_code`
stays `false`. A restore never launders an artifact into an approved state.
## Upstream unavailable
If the upstream cannot be reached the operation is recorded `BLOCKED` with
`ARTIFACT_REHYDRATION_BLOCKED` and zero bytes recovered. Recovery does not report the platform as
fully recovered while an artifact it depends on is still missing.
## Hash mismatch
`record_artifact_recovery` compares every verified file against the expected digest from
provenance. A mismatch downgrades the operation to `FAILED` with `ARTIFACT_HASH_MISMATCH` even when
the caller asserted success — the digest decides, not the reporter.
## Lineage
Each operation stores the artifact set id and variant, the upstream snapshot id, the model and
revision ids, the upstream revision and resolved commit, and the derived-artifact links with their
source digests. A rebuilt derived artifact keeps its path back to the exact source artifacts and
transformation identity; recovery never resets lineage.
## Disposable locations
Rehearsals rehydrate into a disposable storage root, never over the only usable copy. Live evidence
from disaster rehearsal C: `microsoft/trocr-small-printed` at commit
`04e994ab854b0089d4929f48c2b4dbe2ce78a340`, 7 files and 247,200,667 bytes recovered into
`m15-recovery-scratch` in 24.2 seconds, all seven digests matching provenance, while the 75
production artifact locations remained untouched.
docs/architecture/ATOMIC_CUTOVER.md
+15
View File
@@ -0,0 +1,15 @@
# Atomic cutover
Cutover is a durable journal around an external operation, not a claimed distributed transaction:
```text
PREPARING -> LOCKING -> SWITCHING -> VERIFYING -> COMMITTING -> COMMITTED
`-> ROLLING_BACK
```
Preparation performs a CAS/version check, exact observed-source check, approval recheck, current
validation check and immutable rollback snapshot. The adapter then uses its safest atomic primitive
(Qdrant alias update for ExampleRAG). ModelForge records `CUTOVER_COMMITTED` only after the adapter
reports the expected external target plus bounded read-path/capability/smoke health. A stale alias is
`STALE_SOURCE`; unhealthy post-switch truth enters rollback. Production execution additionally
requires the matching approved M12 `requires_reindex` promotion plan.
docs/architecture/BACKUP_AND_RECOVERY.md
+131
View File
@@ -0,0 +1,131 @@
# Backup and recovery
M15 makes recovery a first-class control-plane concern with versioned policy, immutable backup
identity and measured outcomes. Nothing in this plane is allowed to look successful without
evidence.
## RecoveryPolicy
`RecoveryPolicyRevision` is an immutable, fingerprinted revision per asset class carrying backup
method, retention, minimum verified backups, RPO and RTO targets, restore verification depth,
encryption requirement, external-dependency and rehydration flags, secret class and rationale.
Superseding a policy creates revision *n+1* and deactivates its predecessor; recovery rules are
never scattered inline.
Seven policies ship by default: `control-plane.database`, `artifacts.rehydratable`,
`artifacts.non-rehydratable`, `runtime.ephemeral`, `external.projects`, `secrets.credentials` and
`configuration.source-controlled`.
## BackupSet
A `BackupSet` is immutable recovery identity. Beyond the backup id and state it records the
ModelForge version and commit, the canonical repository and reference, the Alembic revision, an
environment fingerprint, the PostgreSQL server identity and backup-tool version, the destination
root, payload and manifest paths with their SHA-256, the included and excluded asset classes, the
payload size, encryption algorithm and key id, verification evidence, milestone and legal hold.
Identity fields are sealed by `immutable_at`. A backup is built across several flushes — planned,
written, then stamped — so the guard permits exactly the sealing flush and rejects every later
change to identity. Lifecycle fields (state, verification, expiry) stay writable.
```text
PLANNED → CREATING → CREATED → VERIFYING → VERIFIED
↘ ↘
FAILED FAILED
VERIFIED/CREATED/FAILED → EXPIRED → DELETED
```
Only `VERIFIED` is restore eligible. A `CREATED` backup that has never been verified cannot become
a restore plan, and neither can a `FAILED` one.
## Manifest
Every backup writes `manifest.json` listing each object with its logical asset type, relative path,
size, SHA-256, source generation, schema version and dependency references. The manifest itself is
hashed into the backup set, so tampering with an entry is caught before any payload is read. The
same entries are journaled as `backup_manifest_entries` rows, and verification requires the file
and the journal to agree.
The manifest contains no secret material. `configuration.json` names the secrets a recovery will
need — operator API key, backup encryption key, Hugging Face token — and their recovery class,
never their values.
## Verification
`verify_backup` is the gate between "a file exists" and "this can be restored":
1. the manifest file hash matches the recorded manifest identity, or `MANIFEST_HASH_MISMATCH`;
2. the manifest objects match the journaled entries and include a control-plane database payload,
or `MANIFEST_INCOMPLETE`;
3. every object exists at its allowlisted path with the recorded size and hash, or `HASH_MISMATCH`
/ `PAYLOAD_MISSING`;
4. the payload decrypts under the configured key and matches its recorded plaintext hash, or
`DECRYPTION_FAILED` / `ENCRYPTION_KEY_UNAVAILABLE`;
5. `pg_restore --list` reads the archive structure.
Only then does the set become `VERIFIED`.
## Encryption
Backups of authoritative control-plane state are protected at rest with AES-256-GCM from
`cryptography`, framed in bounded chunks so a multi-gigabyte dump never has to be held in memory.
ModelForge designs no cryptography of its own; `BackupCipher` is an adapter boundary so a KMS or
HSM provider can replace the local key without touching the recovery service.
A wrong key fails closed: decryption raises before anything is renamed into place, and both the
partial file and the destination are removed, so no readable plaintext is ever left behind. The key
lives only in deployment configuration and is never written into a backup, a manifest, a log or an
audit event.
## Consistency under load
The dump is `pg_dump --format=custom --serializable-deferrable`, a supported online-consistent
logical backup taken in one snapshot, so concurrent control-plane writes cannot tear it. Copying a
live data directory is never done.
Honesty about the window matters more than a clean-looking number: a semantic fingerprint is taken
immediately before and immediately after the dump, and the backup records which subject groups were
stable across it and which were being written. A live rehearsal under continuous capacity and SLO
writes reported 13 of 15 groups stable, with `identity`, `observability` and `serving` active.
## Retention
`apply_retention` expires aged backups but never leaves the platform without a recovery point. The
newest *n* verified backups (`minimum_verified_backups`, default 2) are protected regardless of
age, milestone and legal-hold backups are never expired, and retention refuses to run at all while
a restore is in flight.
## Capacity
Backups check headroom before writing rather than discovering it by filling a disk. The estimate
separates bytes that must be copied from bytes protected by manifest alone, and an insufficient
destination fails the backup with `INSUFFICIENT_CAPACITY` instead of producing a truncated payload.
## Interruption
A process that dies mid-backup leaves a `CREATING` row. Startup reconciliation moves it to `FAILED`
with `MANIFEST_INCOMPLETE`, so an interrupted backup can never be mistaken for a usable one. This
is not theoretical: the first live M15 backup attempt failed mid-write and was reconciled exactly
this way on the next start.
A backup also cannot contain a record of *itself* as complete — the dump is taken while its own
`BackupSet` row is still `CREATING` — so a restored control plane consistently reports the backup
it was restored from as not restore eligible.
See ADR-0042, `docs/architecture/POSTGRESQL_BACKUP_RESTORE.md` and
`docs/architecture/DISASTER_RECOVERY.md`.
## M16 recovery under fault
A backup taken while a mixed capability soak is running verifies normally: the dump is a single
consistent snapshot, and the write-window report names which subject groups were being written
rather than claiming a quiet system. A live run recorded 11 of 15 groups stable with `identity`,
`scheduler_history`, `serving` and `traffic_history` active.
Restarting the control plane mid-backup leaves the set `FAILED` with `MANIFEST_INCOMPLETE` on the
next start; it can never become restore eligible.
An unwritable or absent backup destination now fails closed with `DESTINATION_UNAVAILABLE` rather
than escaping as an unhandled error. Before M16 it returned an untyped server error and left the
set stuck in `CREATING`, so `BACKUP_FAILED` never fired.
docs/architecture/CANARY_AND_TRAFFIC_SHIFT.md
+14
View File
@@ -0,0 +1,14 @@
# Canary and Traffic Shift
M12 canaries are bounded evidence producers, not automatic promotion mechanisms. A successful run
ends in `READY_FOR_PROMOTION_REVIEW`; an operator must explicitly commit the plan.
Request-level canary is valid only for compatible stateless capability contracts and can be scoped
to explicit clients/projects, request count and traffic percentage. Gates include error rate, p95,
capability health, scheduler capacity, critical project failures, external pressure and worker
health. Health, error, latency, critical regression or capacity failure triggers automatic rollback.
Embedding spaces are never percentage-mixed inside an index. A `requires_reindex` plan accepts only
`SHADOW`, `DUAL_READ` or `EVALUATION_ROUTE` semantics against a separate index/space. The current
ExampleRAG alias is outside all M12 rehearsal targets. ExampleVision Vision cannot start a production
canary while owner-photo production validation is deferred; only an isolated LAB rehearsal is valid.
docs/architecture/CAPABILITY_CONTRACTS.md
+20
View File
@@ -0,0 +1,20 @@
# Operational Capability Contracts
## M6 project consumption
`rag.embedding@1` remains model-agnostic. Its native response includes the immutable
`embedding_space_id`, dimension and normalization contract. Consumers validate all three. Workload
priority is identity-owned scheduler policy and is not accepted from the invocation payload.
`rag.embedding@1` accepts one non-empty string or a batch of at most eight strings. Each string is
bounded by the configured character limit and the whole request by the gateway payload limit. It
returns finite, normalized 1024-dimensional vectors, an immutable embedding-space ID, request ID,
usage and cold/queue/load/inference/total timing metadata.
The contract is backend-authoritative. Incompatible dimension, normalization, tokenization or API
schema changes require a new capability version. An underlying deployment may change within v1
only when the output contract remains compatible; embedding-space identity still changes and has
`requires_reindex` migration semantics.
The operational SLO boundary is a bounded 16-request deployment queue, one measured concurrent
inference, and explicit timeout/backpressure failures. Input retention is disabled by default.
docs/architecture/CAPABILITY_DEPLOYMENTS.md
+24
View File
@@ -0,0 +1,24 @@
# Capability Deployments
## M9 shared-GPU admission
OCR, visual embedding and ASR deployments use the existing lease/residency scheduler. Manifest resource classes explain expected pressure, but an exact measured `CapabilityResourceEnvelope`, external GPU observation and fixed reserve determine admission. Heavy work fails with `INSUFFICIENT_SCHEDULABLE_VRAM`; ModelForge never evicts unmanaged Ollama, Tdarr or Plex workloads.
A CapabilityDeployment is the immutable production binding between CapabilityContract,
DeploymentCandidate, production approval, exact ArtifactSet, RuntimeProfile, ComputeNode,
accelerator, EmbeddingSpace and measured ResourceEnvelope. Lifecycle states are stable, draining,
deprecated and archived; LAB_READY remains a separate non-production candidate state.
Promotion rejects missing or stale runtime evidence, non-verified artifacts, non-production nodes,
incomplete supply-chain review and mismatched deployment facts. The active M5 deployment is
`420f11de-4906-4751-bfd4-923849fc7d9d`, pinned to candidate
`c6824408-84d1-45bf-b431-a3893e57fd52` and worker image
`sha256:e707953d1cc6fe3e66e4571ea1381f37e5a88bc71a31dac896b98bccf56dbac0`.
## M12 lifecycle overlay
Lifecycle subjects reference deployments rather than replacing them. A partial unique database
index prevents two `production=true, status=stable` rows for one capability contract. Promotion
execution additionally uses subject-version compare-and-swap and idempotency keys. Platform-stable
does not grant a project production binding; that change requires its own project policy and fit
evidence.
docs/architecture/CAPABILITY_EVALUATION_TYPES.md
+19
View File
@@ -0,0 +1,19 @@
# Capability evaluation types
M9 adds immutable `CapabilityEvaluationSuite` and `CapabilityEvaluationRun` records alongside the retrieval-specific M6/M7 framework. A suite binds one exact capability contract, evaluation type, dataset revision, fixture digests, ground truth, metric definitions, thresholds and critical-case flags. A run binds the exact deployment and environment fingerprint and stores case/aggregate evidence without payload content.
Metric families are deliberately separate:
| Type | Metrics |
| --- | --- |
| retrieval | Recall@5/10, MRR, nDCG@10 |
| ocr | CER, WER, text/field/layout accuracy, latency, VRAM |
| visual-retrieval | Recall@1/5/10, MRR, latency, VRAM |
| asr | WER, real-time factor, latency, VRAM |
| tts | latency, real-time factor, VRAM |
| generation | latency, VRAM; later task-specific contracts |
| llm | task success, structured-output validity, latency, VRAM |
The API rejects a metric from another family, mismatched capability/deployment pairs, incomplete case sets, duplicate suite evidence and non-finite aggregate values. There is no universal composite score.
`/api/v1/capability-advisor` deterministically returns `REQUIRES_MORE_EVIDENCE`, `LAB_READY`, `PROMOTION_ELIGIBLE`, or `BLOCKED` from deployment, run, case and threshold evidence. It never promotes automatically. Existing retrieval comparisons retain their `KEEP_CURRENT` semantics.
docs/architecture/CAPABILITY_GATEWAY.md
+31
View File
@@ -0,0 +1,31 @@
# Capability Gateway
## M9 routes
The gateway exposes model-neutral `document.ocr@1`, `vision.embedding@1`, and `speech.transcription@1` routes. Each has modality-specific validation, scoped service-client authorization and transient content handling. The stable text embedding route remains exactly 1024-dimensional; opaque LAB experiment routes may return a different declared `EmbeddingSpace` dimension without weakening the stable contract.
## Background consumers
Service clients carry a fixed workload priority used on Gateway request records, GPU leases and
serving jobs. ExampleRAG shadow work uses `background`; callers cannot supply or escalate this field.
Inputs remain digest-only in durable records and transient in Redis.
The ModelForge Gateway is the only project-facing inference boundary. A client requests the
versioned `rag.embedding@1` contract; it cannot provide an artifact path, runtime adapter or
concrete upstream model. Resolution is capability → stable CapabilityDeployment → RuntimeProfile
→ exact ArtifactSet → eligible ComputeNode.
The native endpoint is `POST /api/v1/capabilities/rag.embedding@1/invoke`. The supported
OpenAI-compatible subset is `POST /v1/embeddings` with logical model alias `rag.embedding`.
Authentication, scope/rate checks, bounded queueing, a database-backed GPU lease, residency and
the outbound worker job all execute before a response is returned. Typed failures never fall back
to cloud inference.
M10 places the shared cross-capability scheduler between authenticated resolution and the worker
job. A caller still selects only a capability; node, accelerator, resident eviction and priority are
server-authoritative. Capability health remains distinct from temporary scheduling readiness such
as `READY_RESIDENT`, `READY_ON_DEMAND`, `WAITING_FOR_CAPACITY`, `POLICY_BLOCKED`,
`NODE_UNAVAILABLE`, and `RUNTIME_UNHEALTHY`.
Request bodies live only in Redis with a short TTL while queued/executing. PostgreSQL retains a
request ID, input hash, byte/count metadata, timings and failure class, never the input text.
docs/architecture/CAPABILITY_TAXONOMY.md
+28
View File
@@ -0,0 +1,28 @@
# Capability taxonomy
ModelForge is capability-first: consumers bind to a versioned contract, never to a model or runtime. M9 defines seven estate categories: `TEXT`, `RAG`, `DOCUMENT`, `VISION`, `AUDIO`, `GENERATION`, and `ASSISTANTS`.
Every manifest declares purpose, input/output modalities, stability, resource class, consumers, privacy class, evaluation type, and bounded payload limits. Runtime state is derived from registry relations and is not confused with declared stability.
| Capability | Category | Stability | Evaluation | Resource class |
| --- | --- | --- | --- | --- |
| `rag.embedding@1` | RAG | stable | retrieval | HEAVY |
| `rag.reranking@1` | RAG | experimental | retrieval | MEDIUM |
| `document.ocr@1` | DOCUMENT | experimental | ocr | HEAVY |
| `vision.embedding@1` | VISION | experimental | visual-retrieval | HEAVY |
| `speech.transcription@1` | AUDIO | experimental | asr | HEAVY |
| assistant manifests | ASSISTANTS | planned | llm | EXCLUSIVE_GPU |
`stable`, `experimental`, `blocked`, and `planned` describe governance intent. Operational states such as no deployment, LAB-ready, resident, and blocked are evidence-derived and independently visible through `/api/v1/capability-estate`.
M10 adds scheduling readiness without changing taxonomy or promotion state. A healthy capability can
be `WAITING_FOR_CAPACITY`, and a cold proven deployment can be `READY_ON_DEMAND`; neither condition
means the capability is broken. Vision and ASR remain LAB promotion-eligible and are not promoted by
co-residency evidence.
Project readiness is planning metadata, not an integration claim: ExampleRAG maps to embeddings, reranking and future OCR; ExampleVision to vision embeddings and future OCR/vision assistance; ExampleOps to future general assistance and OCR; Portfolio Assistant to embeddings and a future local assistant.
M11 adds operational identity without changing capability stability. ExampleVision is an active shadow
consumer of vision embeddings and an active LAB consumer of speech transcription. Its OCR relation is
still planned/blocked. `PROMOTION_ELIGIBLE`, `KEEP_LAB` and `REQUIRES_MORE_EVIDENCE` are evidence
states, not new taxonomy values or automatic lifecycle transitions.
docs/architecture/CAPACITY_TRENDS.md
+32
View File
@@ -0,0 +1,32 @@
# Capacity snapshots and trends
The M14 minute poll samples existing Node Agent/NVML, scheduler, residency, lease and storage truth.
It does not sample raw NVML at high frequency. Each enabled node/accelerator snapshot stores:
- observed and received timestamps, freshness and availability;
- GPU total, observed, external, ModelForge resident, leased, reserve and schedulable bytes;
- scheduler pressure state;
- total/available system RAM;
- total/free storage.
External GPU use is `max(0, NVML observed - measured ModelForge residency)`. Schedulable VRAM uses
the existing scheduler invariant and never grants authority to manage external processes.
## Aggregation and retention
Raw minute snapshots are retained for seven days. Older rows are aggregated into hourly buckets and
retained for 90 days; cleanup is an application operation, not manual SQL. Each aggregate preserves
sample count, schedulable min/max/average and storage-free minimum. Both tiers are bounded.
Trend responses calculate schedulable min/max/average/p50/p95, external min/max, storage min/max
and GPU pressure frequency. Fewer than three snapshots produce `INSUFFICIENT_DATA`; a latest stale
snapshot produces `STALE`.
Storage forecast requires at least 12 samples spanning at least one hour. Otherwise forecast status
is `INSUFFICIENT_DATA`. With sufficient evidence it reports observed bytes/day and, only for a
negative slope, a bounded days-to-current-zero estimate. It is a trend signal, not an automated
cleanup decision.
Capability headroom compares a measured `CapabilityResourceEnvelope` with known persisted node
capacity. A sustained mismatch becomes a warning; unknown/stale capacity is not treated as a
confident fit and no unlimited-scaling claim is made.
docs/architecture/CROSS_PROJECT_OBSERVABILITY.md
+16
View File
@@ -0,0 +1,16 @@
# Cross-project observability
M11 makes real consumer use visible without exposing payload content. For each active project client,
the Projects API/UI reports project, exact capability, environment, state, purpose, request volume,
last use, terminal error count, p50/p95 latency, deployment identity, measured resource requirement
and latest project-fit recommendation/evidence ID.
“Why Installed” now joins active, non-deprecated project bindings and immutable project-fit evidence
to each installed deployment. Duplicate bindings are projected once. SigLIP2 and Whisper consequently
show ExampleVision as an active shadow/LAB consumer and block destructive deletion through the existing
deployment/project dependency checks.
Gateway telemetry remains bounded: correlation and request identity, capability, priority, payload
shape, lifecycle state and timings are retained. Images, audio, document bytes, transcriptions and
recognition results are not observability dimensions.
docs/architecture/DEPENDENCY_AWARE_CLEANUP.md
+27
View File
@@ -0,0 +1,27 @@
# Dependency-Aware Cleanup
Cleanup is plan-first and dry-run by default. A `LifecycleCleanupPlan` records the exact object,
dependency graph, digest, reclaimable bytes, retention state, blockers and proposed action.
Artifact-set checks cover active locations, derived artifacts, candidates, profiles, execution and
production approvals, capability deployments/routes, project bindings, evaluation runs, embedding
spaces/migrations, lifecycle evidence and retention. Location checks distinguish one node-local copy
from remaining copies. Database foreign keys use `RESTRICT`; there are no hidden destructive
cascades.
Execution requires an operator credential and explicit confirmation, re-resolves the graph and
compares its digest. Changed dependencies make the plan `STALE`; active dependencies or retention
return typed 409 conflicts. ModelForge records a location removal only after the typed storage action
confirms physical completion. Retrying after a crash reconciles the same plan; removal records retain
prior status, digest, size, actor and time. Model, revision, hashes, evaluation, deployments and audit
are never erased by binary-location cleanup.
M13 dependency discovery also reports `MIGRATION_PLAN_SOURCE` and `MIGRATION_PLAN_TARGET` when an
artifact set appears in an active plan identity or a retained rollback snapshot. The execution-time
digest recheck prevents cleanup racing a newly planned migration.
## Node lifecycle extension
Node decommission applies the same fail-closed principle to compute infrastructure. Its preview
enumerates blockers, current truth to remove and provenance to retain; execute repeats that graph
under transaction locks and has no force override. See `NODE_DECOMMISSION.md`.
docs/architecture/DISASTER_RECOVERY.md
+115
View File
@@ -0,0 +1,115 @@
# Disaster recovery
## Canonical restore flow
```text
fresh host / control plane
→ checkout the exact ModelForge commit recorded in the BackupSet
→ provision PostgreSQL of a compatible major version
→ verify the BackupSet (manifest hash, payload hashes, decryption, archive structure)
→ restore the database into an empty destination
→ run only the forward migrations this build knows
→ supply secrets; rotate what cannot be restored
→ start the control plane against the restored database
→ reconcile: clear current truth, suppress restored alerts, surface in-flight operations
→ register or re-enrol the node
→ rehydrate missing artifacts from their exact revisions
→ validate, compare fingerprints, measure RPO and RTO
→ operator declares recovery complete
```
## RestorePlan
`RestorePlan` is first class: backup set, mode, target environment and label, redacted database
destination, artifact strategy, secret strategy, node strategy, expected ModelForge version,
preflight evidence and validation requirements. Plans are idempotent — the same backup, mode,
destination and label return the same plan rather than a second one.
```text
VALIDATION isolated rehearsal; never a production environment
REPLACEMENT an operator-declared maintenance target
DISASTER_RECOVERY the only mode permitted to target production, and only when explicitly enabled
```
## Preflight
Eleven checks run before any byte moves, and again at execution time:
```text
backup_verified payload_hashes schema_compatibility
encryption_key destination_isolation destination_reachable
destination_empty postgres_compatibility disk_space
source_control_revision external_dependencies
```
A failing preflight moves the plan to `PREFLIGHT_FAILED` and `start` is refused with
`restore_preflight_required`.
## RestoreOperation
The journal is durable and append-only. Each phase writes a `RestoreOperationEvent` with its
evidence, and phase durations accumulate into the measured RTO.
```text
PLANNED → PREFLIGHT → RESTORING_DATABASE → RESTORING_CONFIGURATION
→ REHYDRATING_ARTIFACTS → RECONCILING → VALIDATING → READY
↘
FAILED | MANUAL_INTERVENTION_REQUIRED
```
Retry resumes the journalled operation rather than creating a second one. A completed database
phase is never replayed onto live data: the destination-empty guard turns a retry into
`DESTINATION_NOT_EMPTY` instead of a merge. A restore interrupted mid-phase is moved to
`MANUAL_INTERVENTION_REQUIRED` by startup reconciliation, never silently resumed.
## Reconciliation
Reconciliation runs exactly once per restore operation, keyed by a marker audit event carrying that
operation's id. Matching on the action alone would be wrong: a backup legitimately contains the
reconciliation events of *earlier* restores, and an unscoped check made every later restore believe
it had already reconciled.
Reconciliation clears every current-truth table, suppresses restored firing alerts so a stale alert
is not presented as present-day truth while its history is retained, and reports in-flight lifecycle
operations and migration cutovers by id and stage with the typed outcome
`RECOVERY_RECONCILIATION_REQUIRED`. It never guesses external state.
## In-flight operations
A backup taken while a lifecycle promotion or a migration cutover is mid-flight restores that
operation intact, with its idempotency key, expected version and generation. On startup the
lifecycle reconciler conservatively rolls the incomplete operation back from its immutable snapshot
exactly once; a second start changes nothing. Migration cutovers are deliberately *not* auto-
resolved, because external alias truth cannot be inferred after a crash — they stay in their
intermediate stage and are reported as requiring reconciliation.
## RPO and RTO
Both are measured, never assumed.
RPO is the gap between the newest audit event in the restored database and the newest in the source
at validation time; the restore point is the backup's completion timestamp. RTO is the sum of the
measured phase durations.
## Rehearsal discipline
A rehearsal that reuses the live database volume is not a rehearsal. `docker-compose.dr.yml`
provisions a separate PostgreSQL 17 with its own volume and a second control plane bound to it, and
every rehearsal starts from a destroyed and recreated volume. The production database, ExampleRAG's
Qdrant, ExampleVision's recognizer and external GPU workloads are never written to.
## Rehearsal catalogue
| | Scenario | Outcome |
| --- | --- | --- |
| A | Database loss | full restore into a fresh PostgreSQL, measured RTO/RPO |
| B | Corrupt backup | payload and manifest tampering both fail closed |
| C | Artifact loss | exact-revision rehydration into a disposable location |
| D | Node credential loss | revoke, re-enrol, identity preserved |
| E | Control-plane rebuild | isolated plane serving restored truth |
| F | In-flight lifecycle | reconciled exactly once |
| G | In-flight migration | surfaced, never auto-resolved |
| H | Observability history | history restored, current state re-derived |
See `docs/quality/M15_VERIFICATION.md` for the measured results and
`docs/operations/RUNBOOK_FULL_DR.md` for the operator procedure.
docs/architecture/DOCUMENT_OCR.md
+9
View File
@@ -0,0 +1,9 @@
# document.ocr@1
`document.ocr@1` accepts one bounded PNG or JPEG page. The M9 baseline limit is one page, 16 MiB decoded content and at most 4096×4096 pixels in the runtime. PDFs must be split into page representations by a trusted local boundary before invocation.
The model-neutral response contains ordered pages, blocks, extracted text and optional confidence. Unsupported layout/table semantics are not fabricated. The TrOCR baseline specializes in printed line text and is therefore a technical LAB baseline, not a production document-understanding claim.
The preferred PaddleOCR-VL candidate is independently tracked. Custom repository/runtime code is a hard blocker under the default policy. The safe baseline uses an exact, verified TrOCR safetensors artifact through `transformers_trocr` with offline `AutoProcessor` and `VisionEncoderDecoderModel` loading.
OCR suites declare only meaningful ground truth and typed metrics: CER, WER, text/field/layout correctness where labels exist, latency and peak VRAM. Raw fixtures and extracted content are not persisted in the database.
docs/architecture/DOMAIN_MODEL.md
+168
View File
@@ -0,0 +1,168 @@
# Domain Model
The implementation-ready field, cardinality, mutability, deletion and cross-plane contracts are
normative in `M0_FOUNDATION_CONTRACTS.md`. The SQLAlchemy mapping and Alembic baseline are the
executable persistence form of this model.
## Core entities
### Model
Logical upstream model family/reference. Does not imply a downloadable or runnable artifact.
Important fields:
- stable internal ID
- display name
- upstream provider/source
- model family
- modalities/tasks
- parameter metadata
- upstream license metadata
### ModelRevision
An immutable upstream revision, preferably resolved to a concrete commit SHA.
Fields include:
- model ID
- upstream revision name supplied by discovery
- resolved commit SHA
- discovery timestamp
- upstream metadata snapshot
### ModelArtifact
Concrete downloaded files associated with a revision.
- storage location
- content digests
- serialization format
- size
- security state
- license state
- provenance
### DerivedArtifact
A locally generated quantization/conversion/merge derived from one immutable source artifact.
Examples:
- AWQ 4-bit
- GGUF Q4_K_M
- GGUF Q5_K_M
Derived artifacts must maintain lineage to their source.
### RuntimeProfile
The reproducible execution configuration:
- runtime type and version/image digest
- artifact variant
- quantization interpretation
- context policy
- concurrency/batch settings
- GPU-memory policy
- launch arguments
- environment constraints
### Deployment
Runnable combination of revision + artifact + runtime profile assigned to one or more capabilities/channels.
### Capability
Logical function requested by projects, such as `rag.embedding`, `vision.embedding` or `speech.transcription`.
### CapabilityContract
Versioned semantics for a capability:
- accepted input/output schema
- quality metrics
- required modalities/languages
- SLOs
- upgrade class
- migration compatibility
- fallback semantics
### Project
ModelForge consumer, e.g. ExampleRAG or ExampleVision.
### ProjectBinding
Maps a project to a capability contract and deployment channel, plus priorities, benchmark suites and migration support.
### ComputeNode
Machine capable of hosting one or more inference workers.
It has stable identity distinct from hostname, an observation source, agent protocol/version and
capabilities, central liveness state, heartbeat/inventory/telemetry timestamps, role/labels and
explicit production/lab/benchmark eligibility. Enrollment and node credentials are separate
hash-only security records; they are not node identity.
### Accelerator
GPU/accelerator attached to a compute node.
### ResourceEnvelope
Measured runtime behavior for a deployment under a test profile:
- context length
- concurrency
- cold/warm load time
- idle VRAM
- peak VRAM
- TTFT
- throughput
- p50/p95/p99 latency
- power/temperature
- OOM/failure behavior
### BenchmarkSuite
Versioned benchmark cases and scoring rules.
### BenchmarkRun
Immutable execution record tying a suite revision to a deployment and environment fingerprint.
### Experiment
Human-driven or automatic A/B comparison. May graduate individual cases into a benchmark suite.
### Recommendation
Advisor output with structured evidence, blockers, regressions, migration cost and confidence.
### Promotion
Audited action changing deployment channel state.
### Migration
Controlled process required when an upgrade changes persistent derived data or output schemas.
### AuditEvent
Append-only record of security-sensitive and operationally meaningful actions.
Audit events include actor, action, resource, outcome, correlation ID, UTC occurrence time, structured
details and previous/current event hashes. Update and delete are rejected by the persistence layer.
## Key invariants
1. Approved model revisions and artifacts are immutable.
2. Deployments reference immutable revision/artifact identities.
3. A project binds to a capability contract, not a path or model repository name.
4. Every stable deployment has local validation and benchmark evidence; M0 defines no production waiver.
5. `requires_reindex` changes must create a migration; direct promotion is invalid.
6. Deletion cannot proceed while a live binding, rollback target, migration, benchmark evidence, or lineage dependency requires the artifact.
7. Storage URIs are mutable locators; revision SHA and artifact digests are immutable identities.
8. Promotion and migration are distinct records and a promotion never implies completed data migration.
docs/architecture/EMBEDDING_MIGRATIONS.md
+22
View File
@@ -0,0 +1,22 @@
# Embedding migrations
An `EmbeddingMigration` names source/target spaces, source collection, distinct shadow collection,
pinned corpus revision and background execution bounds. Supported lifecycle:
```text
PLANNED -> PREFLIGHT -> BACKFILLING -> VALIDATING -> READY_FOR_EVALUATION
-> FAILED / CANCELLED
READY_FOR_EVALUATION -> EVALUATED -> PROMOTION_ELIGIBLE / NOT_ELIGIBLE
```
Preflight must prove current health, authoritative corpus availability, scoped Gateway access,
target space/dimension/distance, Qdrant reachability, storage and collection non-conflict before a
vector write. ExampleRAG checkpoints after successful Qdrant batches only. Resume validates and skips
matching point fingerprints; mismatches fail closed. Cancellation is observed between batches and
retains the shadow for inspection/resume. Validation checks count, missing/duplicate/malformed and
fingerprint evidence plus source-alias equality. No M6 endpoint performs alias cutover.
M13 retains this M6 project-owned backfill mechanism and adds the generic control-plane layer in
`MIGRATION_ENGINE.md`: immutable lifecycle intent, richer durable checkpoints, versioned validation,
shadow evidence, external-truth cutover/rollback and recovery. The old endpoint still cannot cut over;
there is no bypass from an M6 migration record to production.
docs/architecture/EMBEDDING_MODEL_COMPARISON.md
+11
View File
@@ -0,0 +1,11 @@
# Embedding model comparison
`ModelComparison` keeps raw project quality, latency, resource, critical-case and blocker evidence
separate. Evaluated rows reference immutable EvaluationRuns; blocked rows retain unknown metrics.
Comparability requires the same suite revision and retrieval configuration. No weighted score is
invented and no comparison can approve or promote a deployment.
The first live ExampleRAG comparison covers the nomic baseline, Qwen3-Embedding-0.6B raw,
Qwen3-Embedding-0.6B retrieval-aware, and blocked Qwen3-Embedding-4B and BGE-M3 candidates. See
`docs/quality/M7_VERIFICATION.md` for values and candidate blockers.
docs/architecture/EMBEDDING_SPACE_IDENTITY.md
+20
View File
@@ -0,0 +1,20 @@
# Embedding Space Identity
## Consumer enforcement
ExampleRAG stores `embedding_space_id` on the ModelForge profile, collection identity and every new
shadow payload. The adapter rejects a vector response whose space differs even when dimensions are
equal. The current production index is recorded as `legacy-observed` until complete upstream
revision provenance exists.
An EmbeddingSpace identifies the vector geometry produced by an exact ArtifactSet,
RuntimeProfile fingerprint and CapabilityContract. Dimension and normalization are facts, not the
identity: two 1024-dimensional deployments can still be incompatible spaces.
The identity digest is immutable and the M5 production space is
`326075d3-c75f-402e-b1b4-4d243c6a9ca2`. Every change is classified `requires_reindex`; existing
vectors must never be silently reused under another identity.
M13 freezes the target `EmbeddingSpace` and source space reference in its immutable plan. A reindex
plan must name a separate shadow target; changing deployment, runtime, preprocessing, dimension,
normalization, query/document semantics or distance identity requires a new plan and validation.
docs/architecture/ERROR_BUDGETS.md
+25
View File
@@ -0,0 +1,25 @@
# Error budgets
Error budgets apply only to `PRODUCTION` SLO policies.
For a window with population `N`, objective `O` and bad count `B`:
```text
allowed_bad = (1 - O) × N
consumed_bad = B
remaining = max(0, allowed_bad - B)
burn_rate = (B / N) / (1 - O)
```
The long burn rate uses the complete rolling policy window. The short burn rate uses the last five
minutes, bounded by the policy window. Latency bad events are completed requests over the exact
policy threshold; success-ratio bad events are valid terminal failures.
An operational capability can be serving while consuming budget. The UI therefore shows SLO state,
observed value and budget separately. `remaining=0` is not rewritten as a capability outage, and an
acknowledged alert does not restore budget.
Windows roll naturally as old events leave the population. A new evaluation is an immutable
snapshot, so recovery is demonstrated by later snapshots rather than rewriting history. LAB and
BACKGROUND evaluations have `allowed_bad`, `consumed_bad`, `remaining` and burn rates unset by
default.
docs/architecture/EVALUATION_FRAMEWORK.md
+13
View File
@@ -0,0 +1,13 @@
# Evaluation framework
ModelForge persists generic `EvaluationSuite`, immutable `EvaluationSuiteRevision`, bounded
`EvaluationCase`, `EvaluationRun`, per-case result and `EvaluationComparison` records. A typed
project adapter executes retrieval; ModelForge never executes user-supplied code.
A revision digest covers cases, labels, critical flags, metrics, top-K, retrieval settings and
thresholds. Editing any of these requires a new revision. Runs pin suite/dataset and corpus
revisions, target index/space, retrieval-config digest and full local environment identity.
Comparability is `comparable` only when project, suite revision, corpus revision and retrieval
configuration match. Eligibility remains distinct from promotion and is blocked by incompatible
runs, excess errors, a configured Recall@10 regression or critical regressions.
docs/architecture/EXPANDED_MODEL_DISCOVERY.md
+19
View File
@@ -0,0 +1,19 @@
# Expanded Safe Model Discovery
M8 discovery uses the official Hugging Face provider for live metadata, file inventory and exact
commit resolution. Popularity is a discovery signal only. Each shortlisted candidate records safe
artifact availability, byte size, license capture, runtime expectations, storage/GPU fit and a
typed outcome. Blocked candidates retain unknown quality rather than receiving a synthetic zero.
The bounded shortlist was:
| Candidate | Exact revision | Safe evidence | Outcome |
| --- | --- | --- | --- |
| Qwen3-Embedding-4B | `5cf2132abc99cad020ac570b19d031efec650f2b` | Safetensors, Apache-2.0 metadata, 8,059,503,129 bytes | `discovery_gpu_fit_blocked` |
| BAAI/bge-m3 | `5617a9f61b028005a4858fdac845db406aefb181` | Required weights/heads include `.bin`/`.pt` pickle-risk files | `discovery_security_blocked` |
| multilingual-e5-large-instruct | `274baa43b0e13e37fafa6428dbc7938e62e5c439` | 15 selected safe paths, 11 unique blobs, 1,164,135,840 bytes | fully evaluated |
| multilingual-e5-base | `d128750597153bb5987e10b1c3493a34e5a4502a` | safe Safetensors variant, 1,156,508,130 bytes | preflight only; not selected |
Only E5-large-instruct was downloaded because it passed preflight and was the strongest justified
bounded experiment. E5-base was not downloaded after that choice. Qwen 4B was not forced into VRAM;
BGE-M3 policy was not weakened.
docs/architecture/GPU_PRESSURE_AND_HYSTERESIS.md
+22
View File
@@ -0,0 +1,22 @@
# GPU Pressure and Hysteresis
M10 derives memory pressure from schedulable headroom after physical use, unmaterialized leases and
the dynamic reserve. Temperature and utilization are health signals, not memory-pressure aliases.
The states are `NORMAL`, `ELEVATED`, `HIGH`, and `CRITICAL`. Worsening pressure applies immediately.
Recovery crosses a lower threshold and must remain stable for 30 seconds; a cooldown prevents an
immediate new speculative load. This asymmetric state machine prevents external-use oscillation
from repeatedly loading and unloading models.
- `NORMAL`: normal placement and residency policy.
- `ELEVATED`: no new speculative heavy residency.
- `HIGH`: background and LAB heavy loads queue or reject.
- `CRITICAL`: no new admission; eligible idle ModelForge residency may drain.
External GPU memory is always unmanaged. ModelForge observes it, recalculates headroom and protects
its own admission boundary. It exposes no PID, container, Docker or process-control operation and
never kills, pauses, restarts or reconfigures external workloads.
Missing/stale telemetry or uncertain attribution becomes conservative placement evidence. The same
pure hysteresis transition function is used in service decisions and deterministic oscillation,
sustained-pressure, recovery and cooldown tests.
docs/architecture/HARDWARE_PLANE.md
+81
View File
@@ -0,0 +1,81 @@
# Hardware Plane Architecture
## Boundary
M1 observes hardware through `SystemHostCollector` and `NvidiaNvmlCollector`; M1.5 runs those same
typed collectors in an outbound compute-node agent and publishes observations to the central API.
API routes never call NVML or SQLAlchemy directly; `HardwareInventoryService` reconciles typed
observations through `HardwareRepository`. No shell parsing, Docker socket, privileged mode, arbitrary
host mount, inbound agent service, scheduler or inference runtime is introduced. The central API host
is not implicitly a compute node when startup refresh is disabled.
## Identity
A ComputeNode has an internal UUID and a stable identity key. Direct host execution uses a UUIDv5
derived from the operating-system machine ID. Containers use a UUID stored on the `modelforge-state`
volume, preventing image/container IDs or hostnames from becoming identity. An explicit configured ID
has highest priority. Hostname and display name are mutable metadata, never identity.
NVIDIA accelerators reconcile by `(compute_node_id, GPU UUID)`. Device index, PCI bus ID and display
name are observed facts and can change without creating a duplicate. A successful enumeration that no
longer includes a known UUID marks it `missing`; an incomplete/unavailable NVML scan never does.
## Availability semantics
Every optional fact is `known`, `unknown`, `unsupported`, `unavailable` or `temporarily_failed`.
Only `known` observations carry values. Zero is a valid measured value and is never substituted for a
missing metric. One unsupported metric does not invalidate the accelerator.
## Inventory and telemetry
Inventory contains identity and relatively stable facts: OS/kernel/architecture, CPU topology, total
RAM, GPU UUID/model/PCI/VRAM/compute capability, driver/CUDA-driver capability and MIG status.
Latest-state telemetry contains available RAM, relevant storage use, GPU VRAM use, utilization,
temperature, power, clocks, fan and performance state. Latest telemetry is upserted; M1 deliberately
creates no unbounded history. A later observability adapter may export samples without schema rebuild.
## Reconciliation and audit
Refresh is process-serialized and idempotent. It creates or updates one node, upserts storage/current
telemetry, rediscovers accelerators by UUID and preserves missing devices. Meaningful changes emit
`NODE_DISCOVERED`, `ACCELERATOR_DISCOVERED`, `ACCELERATOR_MISSING` or `HARDWARE_CHANGED` audit events.
Telemetry polling does not generate audit noise. Failed runs emit `INVENTORY_FAILED`, are persisted,
and leave the last successful inventory intact.
## Fingerprint
The deterministic SHA-256 fingerprint includes node identity, OS/version/kernel/architecture and the
sorted accelerator UUID/model/PCI/total VRAM/compute capability/driver/CUDA-driver facts. Driver or
relevant hardware changes produce a new fingerprint. Available RAM, storage use, utilization,
temperature, power, clocks and all other transient telemetry are excluded.
## Health
- no inventory: system process healthy, hardware inventory `pending`;
- host observed with zero NVIDIA devices: node `active`, accelerator capability absent;
- NVML unavailable with no prior GPU: node stays active, accelerator inventory is `degraded` with an
explicit unavailable reason;
- incomplete NVML scan: last good GPU state remains, inventory is degraded;
- GPU absent from a successful enumeration: accelerator `missing` and audited;
- unsupported temperature/power/fan: metric is unsupported, accelerator remains active;
- host collection failure: run failed, last good inventory retained, refresh returns normalized 503.
## Real GPU validation
Run `POST /api/v1/hardware/refresh` twice, then compare `/nodes` and `/accelerators` identities and
inspect typed metrics. The optional direct diagnostic is `nvidia-smi --query-gpu=...`; NVML remains
the implementation source. For containers, start with the GPU overlay and verify the NVIDIA Container
Toolkit exposes the intended device. Never infer RTX 4080 availability from the host description.
Remote reports retain agent observation time and central receive time. Persistent per-stream
sequences reject replay/reordering, while liveness uses received heartbeats to tolerate clock skew.
For enrollment, protocol and remote deployment details, see `NODE_AGENT_PROTOCOL.md` and
`../operations/SERVER_AGENT_DEPLOYMENT.md`.
## Permanent decommission
A permanently retired node becomes a terminal tombstone; it is never deleted. Preview and execute
are generation- and dependency-bound, active work blocks the transition, credentials are revoked,
current scheduler/telemetry truth is cleared, and historical evidence remains. Inventory refresh
and agent publication cannot revive the identity. See `NODE_DECOMMISSION.md` and
`../operations/RUNBOOK_NODE_DECOMMISSION.md`.
docs/architecture/HUGGING_FACE_INTEGRATION.md
+44
View File
@@ -0,0 +1,44 @@
# Hugging Face Integration
## Boundary
M3 integrates the Hub through `HuggingFaceProvider`. Domain services do not call
`huggingface_hub` directly and ModelForge exposes no arbitrary Hub proxy. The official `HfApi`
client provides search and `model_info(..., files_metadata=True, securityStatus=True)` metadata.
HTML scraping is not used.
The control plane retrieves metadata only. Artifact bytes are fetched by the authenticated,
node-scoped artifact worker on the selected compute node. `MODELFORGE_HF_TOKEN` is optional secret
configuration, is never persisted or returned by the API, and is passed only to the API and node
agent—not to future runtimes.
## Facts and interpretation
Search results and snapshots keep reported repository ID, author, task, library, tags, popularity,
access state, card data, scanner data, file tree and source timestamps as upstream facts. Local
candidate matching, approval, hardware compatibility and intended capability remain ModelForge
interpretation. Missing values stay absent/unknown.
Scanner results are stored as upstream evidence with `evidence_only=true`; they are never converted
into a ModelForge approval. License declarations are captured as unreviewed evidence and no legal
conclusion is inferred.
## Revision and access semantics
A requested ref such as `main` is resolved to the exact Hub commit SHA before a `ModelRevision` or
artifact set is created. A later movement of `main` creates a new immutable revision. Existing
revisions and approved plans retain their original SHA.
Access is classified as `public`, `gated`, `private`, inaccessible/not-found, or provider
unavailable. Metadata errors are normalized without exposing credentials. Planning a gated/private
snapshot without configured credentials fails closed as access-required behavior.
## API
- `POST /api/v1/discovery/search`
- `POST /api/v1/models/{id}/refresh-upstream`
- `GET /api/v1/models/{id}/upstream`
- `GET /api/v1/revisions/{id}/artifact-sets`
The adapter is covered by offline fakes/mocks; the M3 acceptance also refreshed all 15 real
candidates against the live Hub.
docs/architecture/LATENCY_PROFILING.md
+13
View File
@@ -0,0 +1,13 @@
# Latency profiling
ModelForge records bounded, content-free timing evidence for capability invocations. Local spans use
monotonic clocks; cross-process totals are correlated by request ID and are not calculated by
subtracting wall-clock timestamps.
The M7 live profile showed that one-second worker polling dominated the M6 path. Bounded server-side
long polling (maximum five seconds), a one-second worker wait and a short retry path reduced warm
Gateway p50 from 1144.46 ms to 194.66 ms across 20 requests. Median contributors after the change
were dispatch 49.18 ms, response transport 47.62 ms, inference 38.14 ms, scheduler 30.13 ms, payload
persistence 9.43 ms, resolution 5.05 ms and result retrieval 3.80 ms. Correctness and security gates
remain in the request path. Detailed evidence is in `docs/quality/M7_VERIFICATION.md`.
docs/architecture/LIFECYCLE_AUDIT.md
+113
View File
@@ -0,0 +1,113 @@
# Lifecycle Audit
Lifecycle audit is append-only and change-oriented. Events include approval request/decision/stale,
plan creation/approval/start, canary start/observation/abort/review, commit, rollback completion or
failure, deprecation/archive, retention and cleanup planning/execution.
Each event stores:
- object type/reference and from/to state;
- requester, approver, executor or system actor role;
- policy revision and exact evidence references;
- human reason, timestamp and correlation/change ID;
- bounded operational metadata such as counts, latency, trigger and fingerprints.
It explicitly excludes model inputs, OCR/ASR/Vision content, project payloads, bearer credentials,
tokens and sensitive host paths. SQLAlchemy prevents update/delete of lifecycle events, while normal
database foreign-key restrictions preserve their referenced provenance.
## Audit-chain integrity
`AuditEvent` has an explicit hash format. Legacy `v1` rows retain their exact historical canonical
JSON encoding: correlation id, actor type/id, action, resource type/id, outcome, bounded details and
previous-event hash. The `20260830_0024` migration validates that complete chain before changing any
schema, marks those rows `v1`, and records the first permitted `v2` sequence. It never rewrites an
approved event or silently repairs an invalid hash. A versioned legacy-prefix seal commits every
legacy sequence, UUID, normalized UTC timestamp and event hash, making the two fields that were not
inside the old hash tamper-evident after cutover.
New `v2` events are assigned their UUID and UTC/microsecond timestamp before the append call. In
production, the database-owned canonical function builds the exact payload text, calculates its
SHA-256, inserts that text and advances the checkpoint atomically. SQLite tests use the same stored
payload contract in Python. Both identity values and the `v2` format identifier join the payload.
Sequence must be exactly `1..n`; the first previous hash is null and every later link is
the immediately preceding canonical event hash. The executable verifier recomputes the v1/v2 hash
appropriate to each sequence and the legacy-prefix seal. It also requires every event UUID and
sequence to be unique, independent of whatever constraints a restored schema currently exposes.
The durable `audit_chain_heads` singleton records event count, final sequence/hash, current format,
v2 cutover and legacy-prefix count/seal. It therefore detects middle gaps as well as deletion of the
tail or the entire event table. SQLAlchemy ORM, Core, aliased/annotated targets, legacy bulk and
textual DML for both audit tables fail closed at the ordinary application `Session` boundary. That
includes raw `exec_driver_sql` on a `Session`-owned connection; its lexer ignores comments and
string/dollar literals before classifying schema-qualified quoted targets. The sole persistence
operation accepts one semantic event, constructs the exact insert and head compare-and-set itself,
and exposes no reusable token, execution option, session marker or connection marker. The guard
admits only those exact active statement objects and operation types.
The SQLAlchemy layer also registers every Engine built by the application, guards direct
`engine.begin()`/`session.bind.begin()` connections, and rolls back protected DML introduced by a
later statement-rewrite listener. It defaults raw DDL and procedural calls that could hide audit
mutation to deny. This remains defense in depth: it is not a claim that hostile same-process Python
cannot remove a listener or open a new DBAPI connection.
The production authority is PostgreSQL. `modelforge_runtime` is a NOINHERIT non-owner with SELECT
on both audit tables and EXECUTE only on
`modelforge_audit.append_event_v2(...)`. Direct INSERT/UPDATE/DELETE/TRUNCATE, trigger/DDL creation,
all SET ROLE-capable memberships and every other non-system function are unavailable. The
canonical function is owned by the non-superuser `modelforge` migration role, fixes
`search_path=pg_catalog`, takes the shared advisory lock and checkpoint row lock, validates the
checkpoint and retained tail, constructs/hashes exact bytes, inserts, and advances the head by CAS.
Owner-check triggers protect row mutation and TRUNCATE as a second database layer. Production
startup reads catalog truth and refuses an owner/superuser runtime, role membership, forbidden
grants, a changed function-body digest, wrong owner/SECURITY/search-path attributes, PUBLIC
execution, a non-origin or wrong-event trigger, schema/database CREATE, or any unexpected
executable non-system function. The API container receives no
migration or bootstrap credential. A database owner/superuser remains a privileged administrative
boundary; corruption tests use an independently created admin Engine and strict gates detect it.
Migration verifies and seals the complete immutable legacy prefix. Recovery and the executable
`audit_chain_intact` invariant remain strict full-chain gates. A normal append does not repeat that
linear proof: under the writer lock it validates the checkpoint shape, the two-row retained tail,
the tail payload/hash/link and the checkpoint count, then inserts one event and advances the head by
compare-and-set. That path is O(1) in retained history instead of cumulative O(n²). A database
superuser could still edit a non-tail middle row without making the next append fail; this is not
claimed otherwise. The strict invariant and every recovery READY transition detect that privileged
tamper. PostgreSQL uses a shared transaction-scoped advisory lock; SQLite test databases use an
engine-scoped process lock held until commit or rollback.
Restore reconciliation is the only writer aimed at another database. It reads audit rows in bounded
ordered pages and strictly verifies events plus checkpoint before consulting an `ALREADY_APPLIED`
marker, immediately before appending a marker, and independently before a restore can become
`READY`. The marker calls the same SECURITY DEFINER canonical function, which advances event plus
head atomically under the same advisory lock. Semantic-fingerprint allowances cannot
turn an audit-chain verification failure into a passing restore.
A production-shaped legacy database containing the pre-RC direct recovery marker with a random
`event_hash` is intentionally a migration blocker: 0024 fails before adding its columns or checkpoint.
RC-QUAL-05 must discover and surface that state during rehearsal. The migration does not bless,
rewrite or discard it. Downgrade to 0022 is allowed only before the first v2 event; afterwards it
would require rewriting immutable history and is refused.
On PostgreSQL, migration 0024 takes the runtime advisory key and an `ACCESS EXCLUSIVE` lock on
`audit_events` before validation. The table lock also serializes older 0022 writers that do not know
the advisory key, and remains held through format marking and checkpoint seed. SQLite uses a
test-only no-op write to acquire its writer transaction before validation. A live PostgreSQL
concurrent-upgrade exercise remains a managed-runner gate; the unit contract asserts both locks.
`AuditContext` is the integration seam for a verified principal and request correlation id. Legacy
services retain their current actor labels until the operator authentication boundary supplies that
context; the hash-chain implementation does not fabricate principal identity. Consequently this
integrity change alone does **not** claim that authenticated principal/correlation integration is
complete.
`MigrationEvent` follows the same append-only/content-free rule and adds plan generation plus exact
source/target identity. Per-batch audit is bounded to one summary event per completed configured
batch; errors are capped and vectors, bodies, queries and secrets are excluded.
## Node lifecycle extension
Permanent compute-node removal writes one `NodeDecommissionOperation` and one
`NODE_DECOMMISSIONED` event in the same transaction as the tombstone and credential revocation.
The event retains operator, reason, timestamp, previous state, cleanup counts and credential result.
See `NODE_DECOMMISSION.md`.
docs/architecture/LIFECYCLE_STATE_MODEL.md
+57
View File
@@ -0,0 +1,57 @@
# Lifecycle State Model
M12 adds a lifecycle control layer without collapsing the existing domain identities. `Model`,
`ModelRevision`, `ModelArtifact`, `ArtifactSet`, `RuntimeProfile`, `RuntimeProbe`,
`DeploymentCandidate`, `CapabilityDeployment` and `ProjectBinding` remain separate records.
## Independent dimensions
Readiness is expressed as independent evidence, never inferred from one status:
| Dimension | Values used by lifecycle evidence |
| --- | --- |
| Integrity | `UNKNOWN`, `VERIFIED`, `CORRUPT` |
| Security | `UNREVIEWED`, `APPROVED`, `BLOCKED` |
| Runtime | `UNPROBED`, `PROVEN`, `INCOMPATIBLE` |
| Evaluation | `NOT_EVALUATED`, `EVALUATED`, `REGRESSED`, `PROMOTION_ELIGIBLE` |
| Project fit | `UNKNOWN`, `REQUIRES_MORE_EVIDENCE`, `ELIGIBLE`, `BLOCKED`, `DEFERRED_EXTERNAL_VALIDATION`, `KEEP_LAB` |
| Deployment | `CANDIDATE`, `LAB_READY`, `PROMOTION_ELIGIBLE`, `CANARY`, `LAB_STABLE`, `STABLE`, `DRAINING`, `DEPRECATED`, `ARCHIVED` |
`LAB_READY`, engineering `PASS`, Advisor `PROMOTION_ELIGIBLE` and an approved request are not
synonyms for production. Production is an explicit environment on a lifecycle subject and requires
an evidence-bound approval, immutable plan, rollback snapshot and separate execution.
## Transition graph and concurrency
The typed graph in `domain/lifecycle_contracts.py` rejects shortcuts. Production follows
`STABLE → DRAINING → DEPRECATED → ARCHIVED`; restoration from deprecated requires a new explicit
promotion path, not mutation. LAB supports bounded canary and a separate `LAB_STABLE` state.
Every subject has a monotonically increasing `version`. Execution compares the operator's expected
version before journaling a change. The database also permits only one production/stable deployment
per capability contract. Conflicts return typed 409 responses; retries use a unique idempotency key.
Every transition creates an append-only `LifecycleEvent` containing object, before/after state,
actor and role, policy revision, evidence IDs, reason, time and change ID. No inference payload or
secret is stored.
M13 adds an independent migration state graph. It does not collapse backfill or external cutover
truth into deployment lifecycle state; production cutover remains subordinate to a current M12
approval and matching approved `requires_reindex` promotion plan.
## M14 operational integration
Lifecycle failures and rollback failures feed bounded operational metrics/alerts from the existing
operation journal. Alert acknowledgement is not lifecycle approval, promotion authority or rollback
authority. M14 never executes lifecycle remediation; all production changes still require the M12
evidence-bound policy, immutable plan and explicit actor separation.
## M16 reconciliation under fault
An incomplete lifecycle operation is rolled back conservatively from its immutable snapshot on the
next control-plane start, exactly once. Restart storms do not accumulate: four consecutive restarts
under load produced no authoritative drift.
The `lifecycle_commit_has_evidence` and `no_hidden_auto_promotion` invariants assert that no
operation reaches `COMMITTED` without its plan, approver and executor, and that no production
deployment exists without a recorded production approval.
docs/architecture/M0_FOUNDATION_CONTRACTS.md
+101
View File
@@ -0,0 +1,101 @@
# M0 Foundation Contracts
This document is the implementation-level boundary for M0. The executable forms live in
`domain/contracts.py`, `domain/lifecycle.py`, `persistence/models.py`, the Alembic baseline,
and validated manifests under `config/`.
## Product and trust boundaries
| Plane | Owns | Must not own |
|---|---|---|
| Control plane | registry, policy, provenance, lifecycle decisions, audit, evidence | inference execution or arbitrary hub code |
| Gateway | project authentication, capability/channel resolution, QoS, normalized errors | direct artifact selection by projects |
| Runtime plane | adapter-governed load/probe/infer/unload against approved read-only artifacts | public endpoints, project/HF secrets, Docker socket, default egress |
| Hardware plane | node/accelerator identity, inventory and measured telemetry | lifecycle or routing decisions |
| Hugging Face boundary | discovery metadata and quarantined acquisition by exact revision | approval, deployment or trust decisions |
| Project boundary | declares versioned capabilities, SLO, priority, fallback and migration support | model repository names, paths or runtime endpoints |
| Runtime-engine boundary | implements the internal adapter contract | leaking engine-specific behavior into capability contracts |
ModelForge is not a cloud-AI broker, Kubernetes platform, training cluster, generic chatbot,
or public multi-user inference service. There is no implicit cloud fallback.
## Identity, cardinality and lifecycle
- IDs are UUIDs in persistence; manifest keys are stable human-readable identities.
- `Model 1—N ModelRevision 1—N ModelArtifact 1—N DerivedArtifact` preserves source lineage.
- A `Deployment` references exactly one revision, exactly one source-or-derived artifact and an
immutable runtime profile. Composite foreign keys prevent the artifact revision from diverging.
- `Capability 1—N CapabilityContract`; project bindings reference a contract version and channel,
never a model.
- `ComputeNode 1—N Accelerator`; leases and envelopes remain node-ready while M1/M6 optimize for one GPU.
- Benchmark runs, promotions, migrations and lineage references use restrictive deletion. Lifecycle
removal is deprecation followed by dependency-checked archival; it is never cascade cleanup.
- Approved revisions, verified artifacts, runtime profiles, benchmark runs, promotions and audit events
are immutable records. Mutable labels and operational status are separate fields.
- All timestamps are timezone-aware UTC database timestamps.
## Capability and project contract
Capability contracts version JSON input/output schemas, modalities, language support, streaming,
structured output, vector dimensionality/normalization semantics, SLOs, quality metrics, upgrade
class, fallback, privacy and resource requirements. Vector-producing contracts with unproven
cross-deployment compatibility are rejected unless classified `requires_reindex`.
Project manifests select a capability contract version plus stable/candidate/experimental channel,
priority, optionality, explicit fallback, migration support, project SLO and benchmark suites.
The strict schemas reject unknown fields, including attempts to add a concrete model ID.
## Artifact proof
The reproducibility chain is:
`upstream repository + resolved commit SHA → filename/type + SHA-256 → optional source digest +
derivation tool/version/arguments → runtime-profile fingerprint → deployment → benchmark run`.
Storage locations are locators, not identities. The digest and immutable lineage provide identity.
A derived artifact is invalid without its source digest and conversion tool/version.
## Runtime adapter contract
An adapter validates a runtime profile, probes layered health, loads only under a scheduler lease and
unloads by deployment identity. Profiles capture runtime type/version, image digest, artifact digest,
quantization, context, concurrency, launch arguments, environment constraints and security toggles.
The control plane depends on this protocol rather than vLLM, Transformers, Diffusers or llama.cpp.
Concrete adapters belong to M5.
## Scheduler contract
Lease requests carry deployment, priority, residency (`always_warm`, TTL-based `keep_warm`,
`load_on_demand`, `exclusive`, `lab_only`), measured resource envelope, warm TTL, exclusivity and
explicit CPU-fallback permission. Envelopes are scoped by accelerator kind, context, concurrency and
batch size and include idle/peak VRAM plus safety margin. A lease moves through requested/acquired/
released state. Failure must release reservations and emit a structured event. M6 implements policy;
M0 does not pretend to schedule work.
## Benchmark comparability
A run binds deployment, revision SHA, artifact digest, runtime/version/image, launch arguments,
suite and dataset revisions, CUDA, driver, accelerator identity, context, concurrency, seed,
timestamps and a deterministic environment digest. Runtime, driver/CUDA, accelerator or launch
profile changes make cross-environment comparisons non-comparable. Historical evidence is retained
with a stale reason; it is never rewritten.
## Promotion and migration
Promotion is a human-approved routing/lifecycle action backed by verified artifact state, local and
project benchmark IDs and a rollback deployment. Migration is separate. For embeddings, promotion is
blocked until a migration is `ready` after: current index → distinct shadow index → backfill →
validation → shadow queries/benchmarks → atomic cutover. The prior index and deployment remain until
the recorded rollback-retention deadline.
## Failure ownership and health
The normalized codes are `MODEL_LOAD_FAILED`, `GPU_OOM`, `RUNTIME_CRASH`, `TIMEOUT`,
`INVALID_OUTPUT`, `HEALTHCHECK_FAILED`, `ARTIFACT_CORRUPT`, `DRIVER_ERROR` and
`CAPABILITY_UNAVAILABLE`. Runtime adapters own load/crash/output faults; the scheduler owns capacity,
OOM reservation and lease recovery; the gateway owns timeout and approved fallback/degrade behavior;
the control plane owns artifact corruption, deployment blocking and audit; an operator owns explicit
security exceptions and hard production promotion decisions. Retries must be bounded and observable.
Health is reported separately for process, runtime, model smoke inference, capability contract/routing,
and required project capabilities. Process liveness alone cannot make a deployment healthy.
docs/architecture/M7_MODEL_COMPARISON.md
+21
View File
@@ -0,0 +1,21 @@
# M7 Model Comparison and Advisor Evidence
M7 adds candidate evaluation without changing the production capability binding.
`CapabilityExperimentRoute` is an opaque, authenticated lab route to one non-production
`CapabilityDeployment`. It preserves the `rag.embedding@1` contract, applies RuntimeProfile-owned
query/document semantics, and writes a distinct EmbeddingSpace. Input semantics are part of the
space identity: changing a query instruction therefore requires a new space and full reindex.
`ModelComparison` references a project, capability contract, immutable suite revision and current
run. Evaluated candidates reference their EvaluationRun; blocked candidates preserve their exact
blocker and unknown metrics. `AdvisorRecommendation` contains structured deltas, critical
regressions, resources, migration, security, policy and evidence confidence. Recommendations are
advisory and dismissible but never executable.
The ExampleRAG M7 comparison uses dense embeddings only, unchanged sparse encoding/RRF, unchanged
filters, candidate limit 40, top 10 and no reranker. This isolates the embedding variable.
The latency trace API is admin-only and returns at most 500 summary records. Input content and input
digests are not exposed. Gateway database retention remains bounded by operator retrieval limits;
M7 does not introduce an unbounded span store.
docs/architecture/MIGRATION_ENGINE.md
+26
View File
@@ -0,0 +1,26 @@
# Migration engine
M13 promotes stateful capability change to a first-class, immutable `MigrationPlan`. A plan binds an
exact project binding, capability contract, source and target identities, adapter fingerprint,
validation-policy revision, M12 approval, rollback target, execution bounds and environment
fingerprint. Approval facts are rechecked immediately before cutover.
The guarded state graph is implemented in `migration_contracts.py`. Every transition adds a bounded,
append-only `MigrationEvent` containing identities, actor, policy, generation, reason and evidence
references. Mutable progress uses database compare-and-swap on `version`; adapter results also carry a
64-bit generation. Production permits at most one non-terminal migration per project/capability.
Core never runs migration code from payloads. An adapter contract is a fingerprinted set of known
operations. M13's first concrete operational contract is `examplerag.qdrant-reindex`; ExampleRAG owns
enumeration, deterministic chunk identities, Gateway calls, Qdrant writes and alias inspection. Core
does not assume Qdrant and stores no vectors, document bodies, queries or secrets.
The earlier M6 `EmbeddingMigration` remains evaluation/backfill evidence. M13 layers lifecycle,
cutover, rollback and reconciliation above that proven project-owned mechanism instead of replacing
or weakening it.
## M14 operational integration
Migration plan states, validation/manual-intervention failures and rollback failures are observed
from the M13 journal. `migration.production.reliability` evaluates terminal correctness over seven
days and `MIGRATION_FAILURE`/`ROLLBACK_FAILURE` preserve an explicit lifecycle. Monitoring never
changes a plan generation, resumes backfill or mutates an external alias.
docs/architecture/MIGRATION_RECOVERY.md
+24
View File
@@ -0,0 +1,24 @@
# Migration recovery
Batch checkpoints and generations make preflight/backfill/validation/shadow restart-safe. Matching
complete batches are skipped; incomplete attempts resume from the durable cursor. Old workers cannot
write into a new generation.
Cutover stages are persisted before external work. Startup reports non-terminal cutover journals but
does not guess external truth. A registered adapter/operator inspection submits the observed target,
fingerprint and health to reconciliation. Exact healthy target commits, exact healthy source records
rollback, and ambiguous truth becomes `MANUAL_INTERVENTION_REQUIRED`. Orphan shadows are retained
and surfaced for later explicit cleanup; never auto-deleted.
## M16 reconciliation semantics
Migration is the one subsystem that deliberately does not resolve itself after an interruption.
External alias truth cannot be inferred from a crash, so an interrupted cutover stays in its
intermediate stage, is reported as requiring reconciliation, and waits for an operator working
through the typed adapter.
That is the difference from lifecycle, which resolves: an incomplete lifecycle operation rolls back
from its immutable snapshot exactly once, and a second control-plane start changes nothing.
See ADR-0045 for the resolve / report / fail-closed contract each subsystem declares.
docs/architecture/MIGRATION_ROLLBACK.md
+10
View File
@@ -0,0 +1,10 @@
# Migration rollback
Before switching, M13 snapshots the exact external target, source index/space, capability contract,
project binding, configuration version, environment fingerprint and generation. The source remains
retained for at least 30 days.
Rollback invokes only the adapter's typed inverse operation and verifies exact prior external
identity, source reachability, capability and project read path. Only then is `ROLLED_BACK` recorded;
elapsed time is measured. Unproven restoration becomes `MANUAL_INTERVENTION_REQUIRED`, never false
success. Cleanup treats an active migration or unexpired migration rollback snapshot as blocking.
docs/architecture/MIGRATION_VALIDATION.md
+12
View File
@@ -0,0 +1,12 @@
# Migration validation
`MigrationValidationPolicyRevision` is immutable. Production revision 1 requires full completeness,
zero failures, comparable evaluation, zero critical regressions, no latency regression, project fit,
external validation and approved security. The isolated-LAB revision can permit a technical cutover
for mechanics rehearsal, but never converts missing project-fit/external evidence into production
eligibility.
Snapshots freeze expected/actual/missing/duplicate/malformed counts, vector finiteness and dimension,
content hashes, space, index schema, distance metric, payload integrity, evaluation IDs,
comparability, regressions, latency, project-fit, external validation and security. They bind the
plan and target fingerprints. Target/plan changes make the snapshot stale; no bypass endpoint exists.
docs/architecture/MODEL_INSTALLATION_RATIONALE.md
+16
View File
@@ -0,0 +1,16 @@
# Model installation rationale
The Models view and `GET /api/v1/models/installation-rationale` answer “Why is this installed?” from registry relations rather than prose.
For each model the projection reports local artifact bytes, capability and contract version, deployment and channel, production/LAB status, declared project consumers, evaluation-run identifiers, last gateway use, deletion eligibility and explicit blockers.
Deletion remains an explicit operator action. Deployment, project, evaluation and provenance dependencies block destructive lifecycle changes. Models with only remote registry metadata are clearly distinguished from installed artifacts. ModelForge can recommend cleanup candidates but never removes artifacts automatically.
M11 adds real active project consumers and immutable project-fit evidence IDs. Only non-deprecated
bindings are listed, duplicate project identities are collapsed, and an active scoped client makes the
operational dependency explicit. SigLIP2 and Whisper are therefore installed for ExampleVision shadow/LAB
flows even though neither is production-promoted.
M12 cleanup dry-runs extend “Why Installed” with active deployment, project, evaluation, migration,
rollback-retention and approval/plan evidence dependencies. Reclaimable bytes are displayed separately
from blocked bytes; rationale and provenance survive any later location removal.
docs/architecture/MODEL_REGISTRY.md
+77
View File
@@ -0,0 +1,77 @@
# Operational Model Registry
## M9 installation rationale
Installed state is derived from verified artifact locations. The installation-rationale projection joins models to exact revisions/artifact sets, RuntimeProfiles, deployments, capability contracts, project bindings, gateway use and evaluation evidence. This powers deletion blockers and the Models “Why is this installed?” panel without treating operator prose as evidence.
## Scope
M2 turns the registry into PostgreSQL-backed operational state without starting discovery,
download, runtime, scheduler, benchmark, advisor or project-production binding work. The initial
15 candidates are materialized from the validated seed manifest and remain unverified candidates.
M3 extends this foundation with `UpstreamSnapshot`, `UpstreamFile`, `ArtifactSet`, `DownloadPlan`,
`ArtifactJob`, `ArtifactJobAttempt`, `ArtifactInspection`, and `ArtifactSetMember`. Metadata refresh
does not change candidate verification. Only a complete promoted set creates verified artifacts and
locations; the model lifecycle still remains `candidate` pending later approval.
M4 references, but never collapses, those identities through RuntimeEnvironment, RuntimeProfile,
RuntimeCompatibilityAssessment, ExecutionApproval, RuntimeProbe and DeploymentCandidate. A verified
ArtifactSet remains only a statement about local bytes until the later gates succeed.
## Domain boundaries
| Entity | Identity | Mutable operator data | Immutable provenance |
|---|---|---|---|
| `Model` | local UUID, stable key, unique upstream source | display name, description, `local_metadata`, `interpretation_metadata`, lifecycle | none until represented by a revision |
| `ModelRevision` | model + exact resolved commit SHA | deprecation/archive timestamps | upstream revision label, resolved SHA, metadata snapshot |
| `ModelArtifact` | revision + SHA-256 | lifecycle/verification state | filename, type, serialization, digest, byte size |
| `DerivedArtifact` | output SHA-256 | lifecycle/verification state | ordered source hashes, transformation, tool/version, configuration, environment, output hash |
| `ArtifactLocation` | storage root + relative path | observed presence/hash/size | never used as content identity |
| `UpstreamSnapshot` | observation UUID | staleness by new observation | provider/repository/resolved SHA and captured evidence |
| `ArtifactSet` | revision + variant key | availability/completeness/security state | selected files and immutable selection time |
| `DownloadPlan` | UUID/canonical idempotency hash | lifecycle status only | exact SHA, file payload, target and preflights |
| `ArtifactJob` | one per plan | lease/progress/result | target ownership and plan relation |
Upstream facts, locally entered governance metadata and local interpretation are separate JSON
documents. Seed-derived capability/role/runtime hints are marked `unverified_seed_claim`; they are
not promoted into upstream facts. License data defaults to `unknown` with no invented SPDX ID or
usage right.
## Service boundary
FastAPI routes call `RegistryService`; domain orchestration calls `RegistryRepository`; only the
repository and service own SQLAlchemy access. The UI never derives registry truth from the YAML
manifest. Compose enables the idempotent startup seed, while tests and explicit maintenance code can
call the same seed operation directly.
## API
- `GET/POST /api/v1/models`
- `GET/PATCH/DELETE /api/v1/models/{id}`
- `POST /api/v1/models/{id}/deprecate`
- `POST /api/v1/models/{id}/archive`
- `GET/POST /api/v1/models/{id}/revisions`
- `DELETE /api/v1/revisions/{id}`
- `GET/POST /api/v1/revisions/{id}/artifacts`
- `DELETE /api/v1/artifacts/{id}` and `POST /api/v1/artifacts/{id}/verify`
- `GET /api/v1/revisions/{id}/derived-artifacts`, `POST /api/v1/derived-artifacts`
- `DELETE /api/v1/derived-artifacts/{id}`
- `GET/POST /api/v1/storage-roots` plus observation/capacity/delete endpoints
- M3 discovery, upstream, artifact-set, download-plan and artifact-job endpoints documented in
`ARTIFACT_ACQUISITION.md`
List endpoints are typed and paginated. Models support search plus lifecycle/source-type filters.
Invalid UUIDs and payloads use the normalized error envelope.
## Lifecycle and deletion
Deprecation is a first-class, auditable operator action. Physical deletion first queries dependent
revisions, artifacts, lineage edges, locations and runtime digest references. A dependency returns
HTTP 409 with exact resource type, ID and relation; no cascade is used to erase provenance.
## Seed contract
The operation inserts missing candidates by stable key/source and updates seed-owned display/source
interpretation only. It never overwrites `local_metadata`. The M2 production seed result was 15
inserts followed by 0 inserts on the immediate repeat run.
docs/architecture/MULTIMODAL_CAPABILITIES.md
+11
View File
@@ -0,0 +1,11 @@
# Multimodal capability plane
M9 extends the existing gateway, scheduler, agent protocol and runtime-worker abstraction with typed OCR, visual embedding and speech-transcription jobs. The control flow remains:
`scoped client → gateway validation → deployment resolution → GPU lease → Redis transient payload → typed node job → offline runtime → typed result → payload deletion`.
Binary content is canonical base64 at the HTTP boundary and is immediately placed in the transient payload store. Database rows, audit events, metrics and structured logs contain only identifiers, sizes, timings and evidence digests. There is no worker-direct consumer route and no cloud fallback.
Specialized adapters use local-only Transformers loaders with `trust_remote_code=false`, denied network access, read-only mounted artifacts, bounded inputs, and GPU resource measurement. A model requiring repository code or unsafe serialization fails before execution.
New capabilities are LAB by default. A successful probe proves executability; a completed modality-specific suite provides evaluation evidence; neither action automatically promotes a deployment.
docs/architecture/MULTI_CAPABILITY_RESIDENCY.md
+24
View File
@@ -0,0 +1,24 @@
# Multi-Capability Residency
The GPU Node runtime worker owns a bounded map of resident deployments instead of one implicit slot.
Load, invoke, health, drain and unload always address a deployment UUID. Loading one deployment no
longer unloads another when the placement plan proves capacity.
States are `COLD`, `LOAD_QUEUED`, `LOADING`, `WARM_IDLE`, `BUSY`, `DRAINING`, `UNLOADING` and
`FAILED` at the control-plane boundary; the wire protocol uses their lower-case runtime forms.
Policies are `ALWAYS_WARM`, `KEEP_WARM`, `LOAD_ON_DEMAND` and `LAB_ONLY`.
Eviction considers only managed idle residency. Active requests and `ALWAYS_WARM`/pinned models are
ineligible. Priority and policy precede idle age and measured reload cost. Each selected eviction
has a typed reason and expected/actual reclaimed byte evidence. A failed reclaim becomes
`GPU_MEMORY_NOT_RECLAIMED` and stays conservative.
The worker reports a full resident inventory with deployment, ArtifactSet, RuntimeProfile, runtime
environment fingerprint, upstream revision and worker generation. Startup reconciliation adopts a
resident model only when immutable identity and node ownership match exactly. Missing inventory is
reconciled cold. Any other GPU process remains unmanaged external usage.
Co-residency evidence is pairwise and reports `PROVEN_SAFE`, `EXPECTED_SAFE`, `NOT_SAFE` or
`UNKNOWN`, with individual and combined envelope bytes. `PROVEN_SAFE` is reserved for measured live
evidence; arithmetic alone is never promoted beyond `EXPECTED_SAFE`.
docs/architecture/NODE_AGENT_PROTOCOL.md
+100
View File
@@ -0,0 +1,100 @@
# Node Agent Protocol v1
## Purpose and boundary
M1.5 adds an outbound-only agent transport between a central ModelForge control plane and one or
more compute nodes. The agent and adjacent M4 runtime worker cannot launch arbitrary commands,
expose a shell, access the Docker socket, promote models, or serve production inference.
Protocol version `1` supports `hardware.inventory`, `hardware.telemetry`, and the additive typed
capabilities `artifact.acquire.v1`, `runtime.probe.v1`, `runtime.health.v1` and
`runtime.unload.v1`. The server rejects a
different protocol version explicitly; capabilities are recorded for later additive negotiation.
## Endpoints
| Method | Route | Authentication | Semantics |
|---|---|---|---|
| POST | `/api/v1/agent/enroll` | one-time enrollment token | create/bind node and return its credential once |
| POST | `/api/v1/agent/heartbeat` | node bearer credential | agent/version/capability liveness |
| PUT | `/api/v1/agent/inventory` | node bearer credential | stable host/GPU inventory reconciliation |
| PUT | `/api/v1/agent/telemetry` | node bearer credential | latest measured RAM/storage/GPU telemetry |
| GET | `/api/v1/agent/artifact-jobs/next` | node bearer credential | claim one job owned by this node |
| POST | `/api/v1/agent/artifact-jobs/{id}/progress` | node bearer + lease | bounded progress/cancel handshake |
| POST | `/api/v1/agent/artifact-jobs/{id}/complete` | node bearer + lease | verified inventory and promotion report |
| POST | `/api/v1/agent/artifact-jobs/{id}/fail` | node bearer + lease | typed failure and bounded retry request |
| GET | `/api/v1/agent/runtime-probes/next` | node bearer credential | claim one exact-node typed probe |
| POST | `/api/v1/agent/runtime-probes/{id}/progress` | node bearer + lease | lifecycle progress/cancel handshake |
| POST | `/api/v1/agent/runtime-probes/{id}/complete` | node bearer + lease | layered health, inference and measured evidence |
| POST | `/api/v1/agent/runtime-probes/{id}/fail` | node bearer + lease | normalized typed runtime failure |
Artifact leases contain only repository, exact commit, selected files, expected sizes/checksums,
approved target root and reserve policy. Credentials from one node cannot claim or update another
node's work. There is no arbitrary command field.
Runtime leases similarly contain only exact server-resolved artifact paths and digests, immutable
RuntimeProfile/RuntimeEnvironment identity and the fixed technical probe input. A static adjacent
worker executes them; neither the protocol nor Node Agent provides general process/container control.
M5 adds `deployment.load.v1`, `deployment.invoke.v1`, `deployment.health.v1`,
`deployment.drain.v1` and `deployment.unload.v1`. The adjacent runtime worker polls authenticated
typed jobs outbound over the same private-CA boundary. Raw project traffic never reaches the node,
and the Node Agent credential is never accepted as a service-client credential.
Administrative token creation, revocation, node metadata changes and credential revocation live
under `/api/v1/admin`. They require `X-ModelForge-Admin-Token`; the API fails closed when
`MODELFORGE_OPERATOR_API_KEY` is unset.
`POST /api/v1/admin/hardware/nodes/{id}/credential/rotate` atomically revokes the previous
credential and returns its replacement once. Install the replacement in the protected agent state
file before restarting it. Rotation and revocation are separate audited operations.
Permanent decommission is a separate terminal lifecycle. Its preview and execute endpoints are
operator-only. Completion revokes every node credential, and ordinary enrollment, rotation,
management, authentication, heartbeat, inventory and telemetry all refuse the tombstoned identity.
An outage or lost credential must use the recovery runbook instead of decommission.
## Identity, time and ordering
- A node credential is scoped to exactly one `ComputeNode` UUID.
- Stable node identity comes from the persisted agent state volume, not hostname or container ID.
- GPUs reconcile by `(node_id, GPU UUID)`; hostname, device index and PCI address are mutable facts.
- Each inventory and telemetry stream has a persistent monotonically increasing sequence number.
- The control plane stores both agent `observed_at` and server `received_at`.
- Replayed sequence numbers are rejected. A higher sequence with an older observation timestamp
advances the stream cursor but does not replace last-good state; the reason remains visible until
the node clock catches up. Observations beyond the configurable future-skew limit (default five
minutes) are handled the same way, preventing one bad clock from poisoning latest state.
Server-received time drives liveness, so clock skew cannot keep a dead node online.
- An accelerator is marked missing only after a successful complete inventory omits its UUID. An
unavailable/incomplete NVML collection preserves last-good state.
## Liveness
Enabled nodes are `online`, `stale`, or `offline` using server-received heartbeat time. Defaults are
30 and 90 seconds and are configurable. Disabled nodes always report `disabled` and their agent
credential is rejected. Offline/stale transitions, return online, enrollment, revocation and
meaningful inventory changes are audited; high-rate telemetry does not flood the audit log.
Terminal nodes report `decommissioned`; liveness reconciliation never changes that state.
## Error contract
Malformed payloads return validation errors. Missing/invalid/revoked credentials return 401;
credential/node ownership mismatches and disabled nodes return 403; unknown resources return 404;
incompatible protocol versions, replayed samples and conflicting node identities return 409. The
normal API correlation ID remains present for operational tracing.
## M16 disconnect and diagnostics
A disconnected agent stops publishing; liveness moves online → stale → offline on the configured
thresholds and no work is assigned to it. Reconnection resumes publication on the same persisted
identity, and a live rehearsal confirmed the heartbeat genuinely stops rather than merely ageing.
Enrolment is single-use and atomically claimed. A sixty-way concurrent storm against one token
produced exactly one identity and one active credential.
A failed publication names the status, method and path — never the response body, which can echo
what the agent was publishing, and never the credential. Before M16 the agent logged only
`control-plane publication failed; retrying`, which cannot tell an operator whether the cause is a
revoked credential or a DNS failure.
docs/architecture/NODE_DECOMMISSION.md
+49
View File
@@ -0,0 +1,49 @@
# Node decommission architecture
## Contract
Node decommission is a terminal, audited lifecycle transition for a `ComputeNode`; it is not row
deletion and not a recovery shortcut. The node UUID, persisted identity, historical inventory,
artifact jobs, runtime evidence, deployments, serving history, capacity history and audit records
remain addressable. Only rebuildable current truth is removed or made unschedulable.
The operator first calls the preview endpoint. The preview returns the node generation, a
deterministic SHA-256 dependency digest, blockers, current-state cleanup counts and retained
provenance counts. Execute requires the same generation and digest, a unique idempotency key, an
operator identity, a reason of at least ten characters, and an exact typed match of the persisted
identity, hostname or display name. There is no force or cancel-dependencies flag.
## Fail-closed dependency graph
Execution refuses an online node and any non-terminal production/LAB deployment, experiment route,
download plan, artifact or serving job, gateway request, runtime probe, residency, GPU lease,
production approval, lifecycle approval/operation, migration/cutover, recovery operation or
identity-referencing enrollment. It also refuses removal of the last known live artifact copy and
the last production-eligible node while production deployments exist.
Unknown status values are treated as active because blocker queries exclude an explicit terminal
set. PostgreSQL execution locks every dependency and cleanup table in the transaction, re-runs the
preview, then claims the node generation with a conditional update. A changed generation or digest
returns 409 and requires a fresh preview.
## Terminal mutation
One transaction revokes every active node credential, clears host/storage/accelerator latest-state
telemetry and scheduler accelerator state, removes cold residency truth, marks accelerators
decommissioned, makes storage roots unavailable and read-only, marks their artifact locations
unreachable, removes eligibility and current inventory/capabilities, and writes the node tombstone.
It also writes one immutable `NodeDecommissionOperation` and one `NODE_DECOMMISSIONED` audit event
containing the reason, operator, timestamp, prior state, cleanup counts and credential result.
A uniqueness constraint on node and idempotency key plus the transaction makes retries deterministic
and the audit exactly once. Normal enrollment, metadata update, credential rotation, authentication,
agent publication, hardware reconciliation and scheduler placement all reject or exclude a
decommissioned identity. An explicit future recovery lifecycle would require a separate audited
design; none exists today.
## Invariant
`decommissioned_nodes_are_terminal` verifies every tombstone is disabled, in decommissioned
status/liveness, ineligible, without active credentials, active work, residencies, leases, runtime
probes, scheduler/current telemetry or current inventory. The invariant is read-only and runs with
the existing platform invariant suite.
docs/architecture/NODE_RECOVERY.md
+77
View File
@@ -0,0 +1,77 @@
# Node recovery
A compute node is two separate things during recovery: a persisted hardware identity, which is
authoritative history, and a credential, which is secret material that hashes cannot reconstruct.
## Identity
`ComputeNode.key` is the agent's persisted identity, written to the node's state volume and reused
across restarts. A recovered node that still has its identity file keeps its node id, its
accelerators, its storage roots and its history. Losing the credential does not lose the identity.
A node whose identity file is also gone enrols as a new node. That is correct — ModelForge cannot
distinguish it from different hardware — and the operator disables the superseded row rather than
letting two nodes claim the same machine.
## Single-use enrolment
One enrolment token mints exactly one node identity. The claim is a conditional update whose row
count decides:
```sql
update node_enrollments set used_at = now()
where id = :id and used_at is null
```
This closed a real defect found by the M15 node-recovery rehearsal: the previous check read
`used_at` and wrote it later, and two agent threads racing into `ensure_enrolled` burned one token
on two compute-node rows for the same hardware. The Node Agent's publication and artifact loops now
also share an enrolment lock, so it issues one enrol request rather than two.
## Credential recovery
```text
RESTORABLE_SECRET restored from backup and immediately usable
ROTATABLE_SECRET hash restores as authoritative state; the consumer needs a new secret
NON_EXPORTABLE_SECRET never written to a backup at all
```
Node credentials are `ROTATABLE_SECRET`. Only `secret_hash` is stored, so a restore reinstates
which credentials existed and which were revoked, but cannot hand the plaintext back to a node that
lost it. Scenarios:
| Scenario | Recovery |
| --- | --- |
| database restored, node still holds its credential | the restored hash matches; the node reconnects |
| credential lost on the node | operator issues a fresh enrolment token; the node re-enrols and keeps its identity |
| credential believed stolen | operator revokes it; the node re-enrols |
| node hardware replaced | new identity, new enrolment; the old row is disabled |
Revocation is permanent. A revoked credential stays refused after re-enrolment: the partial unique
index `uq_active_node_credential` allows one active credential per node, and enrolment revokes the
previous one rather than deleting it, so the revocation survives in history.
## Recovery sequence
```text
node unavailable
→ liveness reflects offline/stale; capability health degrades honestly
→ operator issues an enrolment token
→ agent re-enrols, keeping its persisted identity
→ inventory and telemetry are re-collected
→ scheduler reconciles against re-measured NVML truth
→ capability smoke
```
Nothing about the node's current state is restored from a backup. After a control-plane restore the
node rows exist with their identities and history, but telemetry, leases and residency are empty
until a live agent reports again.
## Live evidence
Disaster rehearsal D used a disposable Node Agent identity; the production GPU Node credential was
never revoked. The disposable node enrolled once (one enrol request, zero 403s), its credential was
revoked and refused three times consecutively with 401, its credential file was deleted, and it
re-enrolled on a fresh token as node `41a7530c-7750-47c9-92d7-78302ee664fd` — the same persisted
identity, with one active and one revoked credential, one enabled rehearsal node and one distinct
hostname. The old credential still returned 401 afterwards.
docs/architecture/OBSERVABILITY.md
+90
View File
@@ -0,0 +1,90 @@
# Observability architecture
## Boundary
M14 observes the existing control-plane truth. It does not introduce a second GPU, node, scheduler
or runtime collector and it never becomes a serving dependency. NVML values remain owned by the
Node Agent; gateway requests, placements, leases, residency, jobs, evaluations, lifecycle and
migrations remain owned by their existing durable journals.
Telemetry has one of five explicit kinds:
- `COUNTER`: monotonic outcomes, such as requests or operations;
- `GAUGE`: current bounded state, such as queue depth or bytes;
- `HISTOGRAM`: distributions with fixed buckets, such as request duration;
- `EVENT`: durable transitions with actor and time;
- `STATE`: a typed current classification such as `STALE` or `FIRING`.
## Metric contract
Names are lowercase Prometheus identifiers prefixed with `modelforge_`; counters end in `_total`,
durations in `_seconds` and storage values in `_bytes`. `/metrics` is operator-authenticated and is
not published by Compose. It includes process metrics even when historical storage is unavailable.
| Plane | Authoritative source | Export examples |
| --- | --- | --- |
| API | in-process middleware | `modelforge_api_requests_total`, request-duration histogram |
| Gateway | `gateway_requests` | request outcomes, request and queue duration histograms |
| Scheduler | placements, jobs, capacity snapshots | decisions, queue depth, schedulable/external bytes |
| Runtime | `serving_jobs` | operation outcomes |
| Node/GPU | Node Agent receive time and NVML latest | freshness, RAM, utilization, temperature, power |
| Storage | registered storage roots and node volumes | free bytes and retained capacity snapshots |
| Acquisition | artifact jobs and registry state | job states and integrity alerts |
| Evaluation | evaluation runs | run states |
| Lifecycle | lifecycle operations | operation stages and rollback failures |
| Migration | migration plans/cutovers | states and failure alerts |
## Cardinality and content
Permitted labels are bounded operational classifications: capability, status, priority, route
class, node display name, device index, verdict, operation, state and availability. Label values are
single-line, escaped and limited to 128 characters. The registry rejects request IDs, UUIDs,
digests, query/prompt fields, filenames, raw errors and other high-cardinality or sensitive keys.
Metrics never contain prompts, query/document text, images, audio, vectors, artifact paths or raw
payloads. Request correlation remains in structured logs and bounded database records, not metric
labels.
## Time and failure semantics
Capacity snapshots carry `observed_at`, `received_at`, calculated freshness and `KNOWN`, `STALE` or
`UNAVAILABLE`. Central receive time determines liveness. Stale data is historical evidence, never
current capacity truth.
The minute-level polling service catches persistence and collection failures, rolls back its own
transaction and sets `modelforge_observability_degraded=1`. It does not alter gateway, scheduler or
runtime decisions. The metrics endpoint falls back to the in-memory process registry on database
failure. This is `OBSERVABILITY_DEGRADED`, not inference failure.
## Access and audit
All operations and metrics routes use the operator credential. Policy/rule revisions,
acknowledgements and maintenance windows are audited. Individual metric samples are deliberately not
audited. No notification or external-process-control adapter is part of M14.
## M15 recovery observability
Observability history is authoritative and restores with the database: SLO policy revisions and
evaluations, alert rule revisions, alerts, alert history, maintenance windows, incidents and
capacity snapshots and aggregates are all fingerprinted subjects.
Current alert state is not. Recovery reconciliation suppresses restored `PENDING`/`FIRING` alerts so
a rebuilt control plane re-derives which alerts are firing now, while their history is retained. In
the live rehearsal three restored firing alerts were suppressed and the rebuilt plane raised its own
after re-evaluating current state, with 187 restored capacity snapshots plus the ones it measured
itself.
Recovery readiness is an operational signal rather than a new production SLO family: M15 adds no SLO
policies, only the five recovery alert rules.
## M16 observability under fault
Monitoring is never a safety dependency. When observability persistence is unavailable, the
in-memory `modelforge_observability_degraded` gauge rises, the operations overview reports
`OBSERVABILITY_DEGRADED`, and serving, scheduling and recovery continue on their own authoritative
state. The M16 invariants deliberately read none of the observability tables, so the question "is
state still sound?" can be answered precisely when monitoring is the thing that failed.
Alerts raised during a chaos scenario are real alerts and must be resolved or acknowledged with
their true cause. The release gate reports any alert left pending or firing at the end of a run.
docs/architecture/OCR_MODEL_SELECTION.md
+43
View File
@@ -0,0 +1,43 @@
# OCR model selection
M11 used the ModelForge Hugging Face provider for a bounded TrOCR shortlist; no web scraping or
mass-download was used.
| Candidate | Exact revision | Outcome |
| --- | --- | --- |
| `microsoft/trocr-small-handwritten` | `b4648cfa171985a6745f37ddd637e98c0da958ac` | task mismatch; not selected |
| `microsoft/trocr-base-printed` | `93450be3f1ed40a930690d951ef3932687cc1892` | deep preflight; blocked |
| `microsoft/trocr-large-printed` | `9ff792d8e7c22061f2ee67e1ed2246b1f9ef1e98` | larger duplicate direction; not downloaded |
| `microsoft/trocr-base-handwritten` | `eaacaf452b06415df8f10bb6fad3a4c11e609406` | task mismatch despite clearer license metadata |
The selected base-printed revision exposed a safe eight-file Safetensors plan of 1,334,746,034 bytes;
the parallel 1,333,508,485-byte PyTorch pickle weight was correctly excluded. Execution nevertheless
stopped before download: upstream scanning was incomplete and license metadata was absent/unknown.
The immutable discovery assessment is `license_blocked`. No security gate was weakened to manufacture
quality evidence.
The existing `microsoft/trocr-small-printed` candidate remains quality-blocked at CER 0.721246 and WER
0.976190. PaddleOCR-VL remains blocked because its exact revision requires custom code;
`trust_remote_code=false` remains mandatory. Final state is therefore:
```text
document.ocr@1: PLATFORM_READY / NO_PRODUCTION_MODEL
```
## Closure refresh — 2026-08-26
The official Hugging Face API was queried again for the bounded shortlist. The exact revisions did
not change:
- `microsoft/trocr-small-handwritten@b4648cfa171985a6745f37ddd637e98c0da958ac` still has no
declared license and exposes one pickle weight with no Safetensors weight;
- `microsoft/trocr-base-printed@93450be3f1ed40a930690d951ef3932687cc1892` still has no
declared license and exposes both Safetensors and pickle weights, with no affirmative repository
security status returned;
- `microsoft/trocr-large-printed@9ff792d8e7c22061f2ee67e1ed2246b1f9ef1e98` still has no
declared license and exposes both weight formats;
- `microsoft/trocr-base-handwritten@eaacaf452b06415df8f10bb6fad3a4c11e609406` still declares
MIT and exposes both weight formats, but remains a task mismatch.
This refresh does not satisfy the license/scanner gate for the printed candidate. No candidate was
downloaded or executed, and the final state remains `PLATFORM_READY / NO_PRODUCTION_MODEL`.
docs/architecture/OPERATIONAL_INCIDENTS.md
+19
View File
@@ -0,0 +1,19 @@
# Operational incidents
M14 uses a lightweight `OperationalIncident`; it is not an ITSM replacement. An incident groups
alerts only when an evidenced shared subject exists.
Correlation values are:
- `LIKELY_ROOT`: the alert directly reports the shared resource failure, such as GPU Node offline;
- `DOWNSTREAM`: a capability failure includes the same node identity;
- `RELATED`: shared timeline evidence without a directional claim;
- `UNKNOWN`: no justified causal direction.
A firing `NODE_OFFLINE` alert opens one node incident. Capability-unavailable alerts that carry that
node ID may join it as downstream. Other alerts stay independent rather than claiming causality.
When every grouped alert resolves, the incident resolves and appends a timeline event.
The bounded timeline stores alert state changes, relation, summary, operator acknowledgement and
recovery timestamps. It never stores prompts, vectors, images, audio, document text or raw payloads.
Incidents explain degradation; they authorize no remediation.
docs/architecture/PLACEMENT_EXPLAINABILITY.md
+20
View File
@@ -0,0 +1,20 @@
# Placement Explainability
`PlacementPlan` is the first-class explanation for both a dry-run and a real request. Both paths call
the same planner. A plan records the requested capability/deployment, node and accelerator,
eligibility, priority, current residents, external use, reserve, authoritative envelope requirement,
headroom before and after, selected managed evictions, queue impact, verdict and typed reasons.
Verdicts are `ADMIT`, `ADMIT_AFTER_EVICTION`, `QUEUE`, `REJECT_CAPACITY`, `REJECT_HEALTH`, and
`REJECT_POLICY`. Reason codes include `INSUFFICIENT_SCHEDULABLE_VRAM`, `EXTERNAL_GPU_PRESSURE`,
`RESIDENCY_CONFLICT`, `EVICTION_NOT_ALLOWED`, `DEADLINE_CANNOT_BE_MET`, `LEASE_CONFLICT`,
`GPU_MEMORY_NOT_RECLAIMED`, and `SCHEDULER_STATE_STALE`.
The operator dry-run answers “what if I invoke this capability now?” without loading a model. The UI
shows the actual budget, pressure, residency, possible eviction, expected reclaimed bytes, cold-load
cost and final reasons. Recent placement history is capped at 500 records. It contains safe request
shape metadata only—never text, image, audio or client-supplied placement commands.
Co-residency arithmetic yields at most `EXPECTED_SAFE`. `PROVEN_SAFE` requires two exact residents
on the same worker generation, KNOWN attribution and a measured combined footprint. Promotion and
scheduling readiness remain separate lifecycle decisions.
docs/architecture/POKEVAULT_VISION_INTEGRATION.md
+36
View File
@@ -0,0 +1,36 @@
# ExampleVision vision integration
ExampleVision retains its production recognizer. The M11 shadow path is:
```text
normalized card image -> vision.embedding@1 -> Gateway -> scheduler -> GPU Node
-> exact 768-dimensional visual space -> pgvector nearest neighbours
-> existing metadata/fusion boundary -> AMBIGUOUS | NO_MATCH | UNAVAILABLE
```
The consumer adapter contains no SigLIP, revision, runtime, worker, GPU Node or artifact identity. It
validates bounded images and the exact visual space/dimension returned by the capability contract.
Images are transient and absent from audit, gateway telemetry and ordinary logs.
References and queries use `examplevision-card-rgb-pad-224-v1`: orientation/geometry normalization is
reused from ExampleVision, then the full RGB card is aspect-preserving padded to 224 x 224. Reference
rows retain canonical printing ID, source-image content hash and visual-space identity. A separate
768-dimensional pgvector schema prevents reuse of ExampleVision's existing 1000-dimensional MobileNet
vectors. Index runs are cursor-based, idempotent, bounded to batches of one through four and may be
resumed or activated only with one consistent space.
Live M11 indexing processed 80 real catalog reference images in two resumable 40-image invocations,
20 successful background Gateway requests and zero failures. The first ten batches took about 4.6
seconds. Ten concurrent interactive technical queries completed while background indexing was active.
These queries used a catalog-artwork derivative, not an owner photograph, so they prove routing,
retrieval and QoS but do not constitute the required real-photo project-fit evaluation.
No confidence threshold was fabricated. Until reviewed owner-photo ground truth exists, the shadow
recognizer cannot emit `CONFIDENT`; the recorded recommendation is `REQUIRES_MORE_EVIDENCE`.
ADR-0031 separates this production-fit limitation from engineering closure. The capability-first
integration and complete reference index may pass M11 while owner-photo validation remains
`DEFERRED_EXTERNAL_VALIDATION`. Future production cutover is guarded by
`POKEVAULT_VISION_PRODUCTION_VALIDATION`; only independently reviewed `OWNER_PHOTO` evidence can
satisfy it. `PUBLIC_PHYSICAL_CAPTURE` and `CATALOG_REFERENCE` evidence remain explicitly weaker and
cannot enable auto-accept.
docs/architecture/POSTGRESQL_BACKUP_RESTORE.md
+111
View File
@@ -0,0 +1,111 @@
# PostgreSQL backup and restore
## Method
ModelForge owns its own backup rather than delegating to a filesystem snapshot:
```text
pg_dump --format=custom --compress=6 --no-owner --no-privileges --serializable-deferrable
```
Custom format restores schema, data, sequences, constraints, indexes and extension dependencies
through `pg_restore`, and `--serializable-deferrable` takes the whole dump in one snapshot that
cannot see a partially applied transaction. Copying `/var/lib/postgresql/data` from a running
server is never done.
The control-plane image ships `pg_dump`, `pg_restore` and `psql` pinned to the same major version
as the server (PostgreSQL 17), installed from Alpine's signed 3.23 repository. The release build
upgrades repository packages before adding that client and verifies the resulting image.
## Recorded identity
Every backup records the server version and version number, the major version, the database name,
the installed extensions, the backup tool version, the backup method and a redacted connection
string, alongside the Alembic revision. A restore preflight compares source and destination major
versions and refuses `POSTGRES_VERSION_INCOMPATIBLE`.
Historical M15 live evidence: server `PostgreSQL 17.11`, `server_version_num` 170011, extensions
`plpgsql 1.0`, tool `pg_dump (PostgreSQL) 17.11 (Debian 17.11-1.pgdg13+2)`, schema
`20260827_0021`. The v1.1 Alpine release image independently reports `pg_dump (PostgreSQL) 17.11`.
## Point-in-time recovery
```text
PITR: NOT_SUPPORTED
```
M15 implements verified logical snapshots. WAL archiving and continuous point-in-time recovery are
deliberately out of scope for v1. The value is exported as `POINT_IN_TIME_SUPPORT` and rendered on
the recovery dashboard so no operator has to infer it. The practical consequence is a finite RPO
bounded by the backup interval, and the platform measures it rather than claiming zero.
## Restore
A restore runs against a genuinely separate database. `_same_database` compares host, port and
database name, and a plan targeting the control plane's own database is refused at creation with
`DESTINATION_NOT_ISOLATED`. Production targets additionally require
`MODELFORGE_RESTORE_ALLOW_PRODUCTION_TARGET` and a `DISASTER_RECOVERY` mode.
```text
pg_restore --no-owner --no-privileges --exit-on-error --single-transaction
```
`--single-transaction` means a failed restore leaves nothing behind. The destination must be empty:
`DESTINATION_NOT_EMPTY` is returned rather than merging into existing data, which is what makes a
retry safe.
## Schema compatibility
The build walks its own Alembic chain by `down_revision` — filename order is not migration order —
and compares the backup's revision against it:
| Backup revision | Result |
| --- | --- |
| equals head | restored as is |
| known, older than head | restored, then migrated forward to head |
| unknown to this build | `SCHEMA_TOO_NEW`, fail closed |
| build ships no chain | `SCHEMA_UNKNOWN`, fail closed |
| backup records none | preflight fails |
Restoring a newer backup onto older code is never silently downgraded. The discovery resolves both
the source checkout and the installed image layout; a packaging error that hides the chain produces
`SCHEMA_UNKNOWN` rather than an accidental pass.
## Validation
After the restore the destination is measured, not assumed: table, constraint, index and sequence
counts, the Alembic head, the audit high-water mark, a bounded semantic fingerprint and its diff
against the fingerprint captured at backup time, and the residual row count of every current-truth
table.
The `READY` gate requires all of:
```text
database_validated restored table and constraint counts meet the plan's requirement
current_truth_reconciled no stale lease, telemetry or inventory row survives
schema_current the destination is at this build's Alembic head
audit_available the audit trail restored with a non-zero sequence
audit_chain_intact every event/hash/link and the durable checkpoint verify strictly
fingerprint_compatible current m15.2, or the exact schema-0022 m15.1 migration rule
no_unknown_corruption every fingerprint difference is explained
```
`READY` always performs a fresh strict target audit verification. A resumed operation cannot reuse
the presence of a journaled `VALIDATING` or `READY` duration as that proof. Audit rows are read with
a stable `(sequence, id)` cursor, so a corrupt duplicate sequence split at the 500-row page boundary
is observed and rejected rather than skipped. A separate aggregate identity check requires total
rows, distinct UUIDs and distinct sequences to agree before the chain can pass; this also catches a
duplicate cursor tuple after constraints were removed from a tampered restored schema.
Live evidence from disaster rehearsal A: 121 tables, 1,759 constraints, 590 indexes, audit
high-water mark 838, 108 of 112 fingerprinted tables byte-identical, and four explained
differences.
## Identity columns
`audit_events.sequence` is `GENERATED BY DEFAULT AS IDENTITY`, but `AuditWriter` always supplies the
value explicitly, so the underlying sequence is never advanced. The dump carries its `SEQUENCE SET`
and the restore reproduces the source state exactly — an insert that omits `sequence` collides
identically on both the source and the restored database. This is pre-existing M0 behaviour
faithfully preserved by the restore, not a recovery defect, and it is recorded here because a
future milestone that starts relying on the identity default must fix it in the audit writer first.
docs/architecture/PROJECT_BINDINGS.md
+25
View File
@@ -0,0 +1,25 @@
# Project Bindings
## Operational evidence
M6 links the ExampleRAG `rag.embedding@1` binding to scoped service authentication, isolated embedding
migrations and evaluation suites. A binding still does not select a concrete model or node.
Projects bind to versioned capabilities, never deployments, model names, runtime engines or paths.
M11 permits a service identity to reference exactly one real binding and records its integration
environment and purpose. Authentication revalidates that the binding is not deprecated and that its
contract still equals the client's sole scope. Background and interactive work use separate identities
so callers cannot self-promote priority. See ADR-0030.
Deployment promotion, placement and embedding-space migration remain control-plane concerns. A
project binding may pin a capability version and embedding-space expectation, but cannot override
the stable route or address a worker directly.
ExampleVision has separate operational bindings/clients for background vision indexing, interactive
vision queries and interactive speech transcription. OCR remains planning-only until a safe model and
project evidence exist.
M12 treats a project production binding as an independent lifecycle subject. A platform capability
may remain stable while the project request is blocked. The default project-production policy needs
eligible fit and satisfied external validation; ExampleVision Vision therefore remains blocked and ASR
remains LAB without mutating their engineering integration evidence.
docs/architecture/PROJECT_CAPABILITY_INTEGRATIONS.md
+19
View File
@@ -0,0 +1,19 @@
# Project capability integrations
M11 distinguishes planning metadata from an operational consumer. An operational integration exists
only when an active `ServiceClient` is bound to one non-deprecated `ProjectBinding`. The binding fixes
the project and capability contract; the client adds an environment, purpose, server-owned QoS class
and a one-time credential. Consumers still know only the ModelForge base URL, credential and logical
capability key.
`GET /api/v1/project-integrations` projects that identity together with gateway usage, latest project-
fit evidence, deployment and measured resource envelope. Request volume counts terminal gateway
requests (`completed` and `error`), latency is derived from those requests, and content is never
included. `POST /api/v1/admin/service-clients/{id}/credential/rotate` replaces a credential atomically;
the prior secret immediately fails authentication.
M11 operational consumers are ExampleVision shadow/evaluation clients for `vision.embedding@1` and
`speech.transcription@1`. The separate vision indexing and interactive identities enforce background
versus interactive QoS without accepting a priority from the request body. Planning-only OCR bindings
remain visibly distinct from active consumers.
docs/architecture/PROJECT_FIT_EVALUATION.md
+27
View File
@@ -0,0 +1,27 @@
# Project-fit evaluation
`ProjectFitEvidence` is an immutable, digest-deduplicated statement about one project binding and,
optionally, one capability deployment. It records environment, suite/dataset identity, case count,
metrics, critical failures, blockers, recommendation, rationale, evidence fingerprint and one typed
visual evidence class: `OWNER_PHOTO`, `PUBLIC_PHYSICAL_CAPTURE` or `CATALOG_REFERENCE`. The deployment
must implement the binding's exact capability contract.
Allowed recommendations are `PROMOTION_ELIGIBLE`, `KEEP_LAB`, `BLOCKED` and
`REQUIRES_MORE_EVIDENCE`. ModelForge rejects a production or promotion-eligible claim when blockers
or critical failures are present, and it never turns evidence into a promotion action.
Engineering integration, project production fit, production validation and production action are
separate fields. `POKEVAULT_VISION_PRODUCTION_VALIDATION` can become satisfied only from reviewed
`OWNER_PHOTO` evidence. Public physical captures may support separately labelled robustness or
printing evaluation; catalog references support only index/embedding/route integrity. Neither may be
upgraded to owner-photo evidence. Production action is `NONE` in M11.
M11 records:
- ExampleVision vision engineering: `PASS`; 21,775/21,775 available references, zero indexing failures,
scoped shadow routing and QoS proven.
- ExampleVision vision production fit: `REQUIRES_MORE_EVIDENCE`; owner-photo validation is
`DEFERRED_EXTERNAL_VALIDATION`, zero reviewed owner-photo cases, production action `NONE`.
- ExampleVision voice query: `KEEP_LAB`; four NL/EN TTS acceptance cases, all successful, with a small
non-microphone dataset and measured word error rate of 0.380952.
- OCR: no project-fit claim, because no safe, clearly licensed candidate reached execution.
docs/architecture/PROJECT_INTEGRATIONS.md
+20
View File
@@ -0,0 +1,20 @@
# Project integrations
Projects consume versioned capabilities through scoped service identities. A project integration
contains a `Project`, `ProjectBinding`, optional `ProjectEvaluationBinding`, service client and
project-owned data plane. It never contains a concrete model path, runtime, node or artifact
selector.
ExampleRAG is the first operational integration. Its production retrieval remains `CURRENT`; M6 adds
an operator-only `SHADOW` target. Both targets are server-owned identifiers, not public arbitrary
collection parameters. Text is transient in the Gateway payload store and never copied into
ModelForge audit or metric labels.
Service clients have a fixed scheduler workload priority. The ExampleRAG reindex identity is scoped to
`rag.embedding@1` and `background`, so the consumer cannot escalate bulk work to production.
M13 registers `examplerag.qdrant-reindex` as the first typed stateful migration contract. ExampleRAG owns
chunk enumeration/hashes, Gateway embedding calls, Qdrant writes and alias inspection; ModelForge
owns immutable intent, approval, policy, progress evidence and external-truth journaling. Isolated
LAB aliases never grant the production client a new target. ExampleVision remains a blocked dry-run
candidate while its external production validation is deferred.
docs/architecture/PROMOTION_PLANS.md
+23
View File
@@ -0,0 +1,23 @@
# Promotion Plans
Approval establishes permission to propose a change. It does not activate anything.
`LifecyclePromotionPlan` binds the approval and subject to the exact candidate, desired state,
rollback target, project consumers, affected embedding/pipeline/runtime/storage identities,
migration class, canary/drain strategy, health gates and abort conditions. Its impact projection
lists projects, clients, indexes, retrieval pipelines, runtime deployments, artifacts and scheduler
envelope. Approval freezes these fields; changed intent requires a new plan.
The supported classes are `transparent`, `behavioral`, `requires_reindex` and `schema_breaking`.
Behavioral changes require evaluation/canary evidence. `requires_reindex` requires a ready isolated
migration and shadow/dual-read/evaluation-only route; M12 records but does not automatically execute
the cutover. Schema-breaking plans require a manual runbook and are likewise not auto-executed.
Execution creates a `LifecycleOperation` journal and immutable rollback snapshot transactionally.
Stages are `PREPARING`, `ACTIVATING`, `VERIFYING`, `COMMITTED`, `ROLLING_BACK`, `ROLLED_BACK` and
`FAILED`. Requester, approver and executor are stored separately. A repeated idempotency key returns
the original operation.
For M13 production reindex, the migration plan references this exact approved promotion plan and
approval fingerprint. M12 still refuses to execute reindex as a direct deployment swap; the typed
migration cutover journal is the only supported execution path.
docs/architecture/RAGCORE_INTEGRATION.md
+21
View File
@@ -0,0 +1,21 @@
# ExampleRAG integration
```text
ExampleRAG operator workflow
-> scoped ExampleRAG service credential
-> POST /api/v1/capabilities/rag.embedding@1/invoke
-> Gateway authentication and rate limits
-> GPU scheduler and lease
-> outbound GPU Node node agent
-> typed offline runtime worker
-> vector plus embedding_space_id
-> isolated ExampleRAG shadow collection
```
ExampleRAG configuration names only the ModelForge origin, credential-file boundary and
`rag.embedding@1`. The adapter rejects redirects, public/cloud hosts, invalid credentials,
non-finite vectors, wrong dimensions, non-normalized output and embedding-space mismatch. Safe
transient statuses are retried with bounded backoff. There is no cloud or direct-worker fallback.
The credential is read for every batch, allowing create → secret-file replacement → validate →
revoke rotation without logging or persisting the secret in either repository.
docs/architecture/RECOVERY_ASSET_CLASSIFICATION.md
+65
View File
@@ -0,0 +1,65 @@
# Recovery asset classification
Recovery starts with an inventory, not with a backup job. Every persisted state source in
ModelForge is classified once, and the classification decides whether the state is copied,
rebuilt, discarded, or explicitly disowned.
## Classes
```text
AUTHORITATIVE exists nowhere else; must be copied and provably restorable
REBUILDABLE reproducible from an exact identity; a manifest is enough
EPHEMERAL describes the present, not history; must be re-measured after recovery
EXTERNAL owned by another project; ModelForge records the dependency and never writes
SECRET credential material; hashes restore, plaintext does not
```
A class is not a label on a document: `RecoveryPolicyRevision` refuses incoherent combinations.
Authoritative state cannot declare `NOT_BACKED_UP` or `MANIFEST_ONLY`, and cannot omit an RPO.
Ephemeral state cannot claim a payload backup. External state must be marked as an external
dependency. Secret state must name a secret recovery class. Only rebuildable state may be
rehydrated instead of copied.
## The inventory
`recovery_asset_records` holds one row per state source with its owner, location, backup method,
restore method, rebuild method, RPO, dependencies and readiness. The seeded inventory is:
| Asset | Class | Backup | Recovery |
| --- | --- | --- | --- |
| `postgres.modelforge` | AUTHORITATIVE | `pg_dump --format=custom` | `pg_restore` into a fresh database |
| `artifacts.derived` | AUTHORITATIVE | file copy | payload restore preserving derived lineage |
| `artifacts.huggingface` | REBUILDABLE | manifest only | exact-revision redownload, quarantine, verify, promote |
| `configuration.repository` | REBUILDABLE | source-control reference | checkout of the recorded commit |
| `backup.destination` | REBUILDABLE | source-control reference | operator-provided storage |
| `redis.queues` | EPHEMERAL | not backed up | queues start empty; leases reconciled |
| `runtime.residency` | EPHEMERAL | not backed up | cleared on restore, re-measured from live GPU Node state |
| `node.gpu_node` | SECRET | hashes only | restored hash stays valid, or revoke and re-enrol |
| `credentials.project` | SECRET | hashes only | rotate; plaintext is unrecoverable |
| `credentials.operator` | SECRET | never in a backup | operator-supplied break-glass |
| `external.examplerag` | EXTERNAL | not backed up | owned by ExampleRAG |
| `external.examplevision` | EXTERNAL | not backed up | owned by ExampleVision |
| `external.huggingface` | EXTERNAL | not backed up | upstream availability only |
| `external.gitea` | EXTERNAL | not backed up | platform operations |
## History versus current truth
The same distinction runs through the database itself. `recovery_fingerprint.py` partitions every
persisted table into fingerprinted history and `CURRENT_TRUTH_TABLES`, and a test asserts the two
sets together cover the entire schema — a new milestone cannot add a table that is silently
neither.
Current truth is `host_telemetry_latest`, `accelerator_telemetry_latest`, `storage_volume_states`,
`hardware_inventory_runs`, `scheduler_accelerator_states`, `residency_allocations`,
`serving_gpu_leases` and `gpu_leases`. These are cleared during recovery reconciliation and
re-derived by a running control plane. Restoring them would present a stale NVML reading or a dead
GPU lease as a present-day fact, and the restore gate refuses to reach `READY` while any of them
still holds a row.
## Readiness
Each asset reports one of `PROTECTED`, `REHYDRATABLE`, `ROTATION_REQUIRED`,
`EXTERNAL_DEPENDENCY` or `UNPROTECTED`. Authoritative assets are reported `UNPROTECTED` whenever no
verified backup exists, regardless of what the policy claims: readiness is measured, not declared.
See `docs/architecture/BACKUP_AND_RECOVERY.md` for the backup contract and ADR-0041.
docs/architecture/RECOVERY_COMPATIBILITY.md
+77
View File
@@ -0,0 +1,77 @@
# Recovery compatibility
M17 will use this matrix for upgrades; M15 establishes it and proves the fail-closed directions.
## Backup schema versus ModelForge build
| Backup Alembic revision | Restoring build | Result |
| --- | --- | --- |
| equals the build's head | same or newer build | restored as is |
| older, present in the build's chain | newer build | restored, then migrated forward to head |
| absent from the build's chain | older build | `SCHEMA_TOO_NEW` — refused |
| recorded as none | any build | preflight fails; the backup is not restore eligible |
| any | build shipping no Alembic chain | `SCHEMA_UNKNOWN` — refused |
The chain is walked by `down_revision`, not by filename order, so a revision that sorts earlier but
descends later is still placed correctly. Discovery resolves the source checkout, the installed
image and an explicitly configured `MODELFORGE_ALEMBIC_DIRECTORY`.
A newer backup is never silently downgraded onto older code. Restoring an older backup onto a newer
build runs the forward migrations that build ships and nothing else.
## PostgreSQL
| Source major | Destination major | Result |
| --- | --- | --- |
| 17 | 17 | supported |
| 17 | 18+ | permitted; preflight records both majors |
| 17 | 16 or lower | `POSTGRES_VERSION_INCOMPATIBLE` |
The control-plane image pins its client tools to the server major version. `pg_dump` from an older
major cannot read a newer server, which is why the tools ship with the control plane rather than
being inherited from the host.
## Manifest schema
`manifest_schema_version` is `m15.1`. Verification requires the manifest file and the journaled
`backup_manifest_entries` to agree exactly, so a manifest written by a different schema version is
rejected rather than partially interpreted.
## Semantic fingerprint
`FINGERPRINT_VERSION` is `m15.2`; `m15.2` adds the schema-0024 `audit_chain_heads` singleton to the
audit group. Current schema-0024 fingerprints are comparable only when both sides are `m15.2` and
both contain exactly one complete checkpoint row. Both fingerprints must enumerate the complete
authoritative table contract: a missing expected table, an unexpected table, an absent/sparse
entry or a malformed count/digest is incompatible. After reconciliation, every non-current-truth
table must retain its exact count and digest except the enumerated mutations: one canonical audit
marker, the corresponding singleton-head advance, and same-count suppression changes in
`operational_alerts`.
Every fingerprint also records the exact physical table-name set, including current-truth tables
whose rows are intentionally excluded. Dropping `host_telemetry_latest`, adding an attacker table,
or omitting any authoritative table is therefore incompatible. Table digests and group/overall
digests use typed, length-prefixed cells and rows; separators, nulls and different scalar types
cannot alias. A table exceeding the 250,000-row cap is marked `BOUNDED`; that is useful diagnostic
evidence but can never authorize READY because it is not a complete count-and-digest proof.
There is one explicit cross-version rule: a backup whose recorded Alembic revision is exactly
`20260828_0022` and whose fingerprint is `m15.1` may be compared with a restored/migrated `m15.2`
fingerprint when the old side has no checkpoint and the new side has exactly one. Strict audit-chain
verification must pass independently. The transition still permits exactly one canonical audit
marker and no other digest/count change (apart from same-count alert suppression). No other absent
or introduced table, fingerprint-version pair or schema revision receives that allowance.
Auth migration `20260830_0023` adds constraints only, so it introduces no physical-table exception
inside this schema-0022-to-0024 comparison.
## Encryption
`MFBK1` frames the encrypted payload with the key id it was sealed under. A payload sealed by a
different key id is still authenticated by GCM and fails closed. The algorithm is recorded per
backup set, so introducing a second algorithm later does not invalidate existing backups.
## Recovery contract
Recovery API responses never carry a DSN with its password, a credential hash, or an enrolment
token. `RestorePlanResponse` exposes `database_destination_redacted` and has no
`database_destination` field at all, which is asserted against the published OpenAPI document.
docs/architecture/REINDEX_MIGRATIONS.md
+15
View File
@@ -0,0 +1,15 @@
# Reindex migrations
`REQUIRES_REINDEX` always names distinct source and shadow targets and a distinct target
`EmbeddingSpace`. Direct deployment swaps and same-index vector mixing fail validation.
Preflight proves the frozen source fingerprint/count, target absence, space and capability health,
credential, storage, scheduler capacity, adapter availability, rollback retention, evaluation suite
and current approval. Backfill priority is `BACKGROUND`; batches, concurrency and in-flight work are
bounded. Each checkpoint records cursor, generation, deterministic item/result fingerprints,
completed/retryable/permanent counts, bounded errors and duration.
A retry replaces the prior attempt's counters rather than double-counting them. Completed matching
batches are idempotent. Old versions/generations and changed item fingerprints fail closed. A batch
is complete only after finite shape/space validation, committed destination write and matching
content hashes. Cancellation and pause preserve the shadow for retention-aware resume or cleanup.
docs/architecture/RERANKING_CAPABILITY.md
+17
View File
@@ -0,0 +1,17 @@
# Reranking Capability
`rag.reranking@1` is a first-class capability, never a public model or worker endpoint. A caller
submits one query, 1–40 bounded `{id,text}` documents and `top_n` 1–10. The response contains only
stable IDs, finite scores and ranks. ModelForge resolves the experiment deployment, acquires a GPU
lease and dispatches to the private GPU Node runtime worker.
Query and document text exist only in the bounded Redis payload exchange and worker memory. They
are deleted after completion or timeout and never enter PostgreSQL, audit payloads, metrics or
request history. IDs, timings, counts and failure codes may be persisted. There is no network
egress, cloud fallback or `trust_remote_code` path.
The M8 implementation supports the Qwen3 causal-LM reranker adapter with immutable runtime-profile,
artifact-set, exact-revision and probe evidence. Typed failures include `RERANKER_NOT_AVAILABLE`,
`RERANK_REQUEST_TOO_LARGE`, queue/capacity failures and invalid/non-finite output.
The M8 deployment is experiment-only. No stable route or production promotion was created.
docs/architecture/RERANKING_EVALUATION.md
+10
View File
@@ -0,0 +1,10 @@
# Reranking evaluation
M7 completed embedding comparison before considering reranking. No embedding candidate passed the
critical-regression gate, and live external GPU use left a constrained single-GPU window. Adding a
reranker and a second runtime variable would therefore not strengthen the M7 embedding decision.
Reranking is deferred. A later study must use `rag.reranking@1`, freeze the same initial candidate
pool, bound transient chunk-text input, keep inference offline, compare before/after quality and
latency, and create no production cutover automatically.
docs/architecture/RESOURCE_ACCOUNTING.md
+26
View File
@@ -0,0 +1,26 @@
# GPU Resource Accounting
M10 treats NVML as physical truth while retaining ModelForge attribution as a separate logical
ledger. A resident model is already present in `observed_used_vram_bytes` and must therefore never
be subtracted a second time.
```text
physical_used = max(observed_nvml_used, attributed_managed_resident)
unmanaged_external = max(0, observed_nvml_used - attributed_managed_resident)
future_lease = max(0, reserved_bytes - materialized_bytes)
schedulable = max(0, total - physical_used - future_leases - dynamic_reserve)
```
Every term is non-negative. The exposed invariant delta is
`observed - managed_resident - unmanaged_external`; it diagnoses sampling or attribution drift but
is never converted into capacity. `KNOWN`, `ESTIMATED`, and `UNKNOWN` describe attribution quality.
Unknown or stale telemetry fails closed with zero schedulable capacity; it never means zero use.
The versioned `m10-v1` reserve is the maximum of 1 GiB, 5% of total VRAM, and a 256 MiB runtime
margin. A deployment requires its measured peak plus the maximum of 128 MiB or 10% of that peak.
Only unmaterialized leases reduce future headroom. Accelerator-row locking makes the database the
coordination boundary for concurrent admissions.
Resource envelopes remain bound to the exact artifact, runtime profile, accelerator and environment
fingerprint. A stale or mismatched heavy envelope cannot authorize placement. Persistent history is
bounded placement evidence, not a one-second telemetry archive.
docs/architecture/RESOURCE_CLASSES.md
+14
View File
@@ -0,0 +1,14 @@
# Resource classes
Resource classes provide an operator-facing scheduling expectation; measured `CapabilityResourceEnvelope` evidence remains authoritative.
| Class | Meaning |
| --- | --- |
| LIGHT | Usually CPU or a small GPU footprint; may coexist when measured budgets allow. |
| MEDIUM | Material GPU residency; serialized when current envelopes require it. |
| HEAVY | Requires an explicit measured envelope and conservative shared-GPU admission. |
| EXCLUSIVE_GPU | Intended to occupy the schedulable GPU budget alone. |
Admission uses current accelerator telemetry minus external/unmanaged use, active ModelForge residents and leases, and the fixed safety reserve. It rejects unsafe work with `INSUFFICIENT_SCHEDULABLE_VRAM` instead of attempting an OOM. The label never overrides the byte budget.
GPU Node remains a single shared NVIDIA GPU. Ollama, Tdarr and Plex are observed as external pressure only; ModelForge does not stop, restart or reconfigure them. Load, invoke and unload jobs share the existing database-backed lease and residency coordinator, so OCR, Whisper, image generation and assistants cannot bypass contention controls.
docs/architecture/RETENTION_AND_DEPRECATION.md
+18
View File
@@ -0,0 +1,18 @@
# Retention and Deprecation
Deprecation removes an object from ordinary selection; it does not delete bytes or history.
Supersession is an explicit reference on the lifecycle subject. Archive means no active deployment
and hidden normal selection while identities, hashes, plans, evaluation and audit remain queryable.
`RetentionPolicyRevision` is versioned and immutable. The default `standard-rollback` revision
retains the previous stable target for at least 30 days after commit. A retention record distinguishes
`ACTIVE`, `ROLLBACK_RETAINED`, `DEPRECATED`, `ARCHIVABLE` and `ARCHIVED`; legal hold and unexpired
rollback state block cleanup. Tests use controlled timestamps to evaluate expiry without waiting.
Physical removal is always a separate cleanup operation. Archiving metadata does not imply removing
model weights, and deleting one node-local location does not mark an artifact globally missing while
another verified location remains.
M13 captures a migration rollback snapshot before external switch and applies the plan's minimum
30-day retention. Active migrations and unexpired rollback snapshots are cleanup blockers; orphan
shadow targets become explicit later cleanup candidates and are never auto-deleted.
docs/architecture/RETRIEVAL_AWARE_EMBEDDINGS.md
+11
View File
@@ -0,0 +1,11 @@
# Retrieval-aware embeddings
Query/document intent is an explicit `input_type` on the generic `rag.embedding@1` request. Concrete
instructions remain in the immutable RuntimeProfile; clients never receive model paths or prompt
syntax. Qwen3-Embedding-0.6B uses the exact-revision upstream query instruction and an empty document
prefix. The instruction source and revision are part of EmbeddingSpace identity.
Changing instruction, pooling, normalization, dimension or another behavior-affecting setting creates
a new RuntimeProfile and EmbeddingSpace and requires a separate reindex. The M6 raw profile and space
remain immutable.
docs/architecture/RETRIEVAL_CANDIDATE_POOLS.md
+13
View File
@@ -0,0 +1,13 @@
# Retrieval Candidate Pools
A `RetrievalCandidatePool` freezes the output of first-stage retrieval before reranking. Its
identity binds the project, immutable suite case, source embedding space/index, corpus revision,
retrieval configuration digest and ordered candidate IDs/scores/content hashes. Text is excluded.
M8 uses a top-40 ceiling. Thirty-two cases contain 40 unique in-scope candidates; one scoped case
contains 39. ModelForge records the actual count and never pads the pool with out-of-scope data.
Control and reranked results must reference the same pool IDs. Corpus/config/pool mismatch is a
typed comparability failure.
The pool is immutable after creation. Reranking therefore cannot benefit from a second dense or
sparse retrieval run, and it cannot recover a relevant item absent from the frozen pool.
docs/architecture/RETRIEVAL_PIPELINE_IDENTITY.md
+16
View File
@@ -0,0 +1,16 @@
# Retrieval Pipeline Identity
A retrieval pipeline is behaviorally identified by:
- embedding-space identity;
- sparse and fusion configuration digests;
- optional reranker deployment and reranker configuration digest;
- candidate-k and output-k semantics.
The canonical identity digest covers every field. An embedding-only control and the same embedding
plus reranker are distinct pipelines even when dimensions and source collection are equal. A
reranker profile/configuration change also creates a new identity. Reranker addition is classified
`behavioral`; an embedding-space change remains `requires_reindex`.
Pipeline comparisons consume immutable fixed-pool runs. They do not overwrite embedding evaluation
runs and cannot be compared as though they were embedding deployments.
docs/architecture/ROLLBACK.md
+16
View File
@@ -0,0 +1,16 @@
# Rollback
Every executable plan names a rollback target before execution. The operation captures an immutable
snapshot of subject identity/state/version, environment, affected identities and rollback reference.
The same pattern can represent deployment, project binding, runtime/config and future index-alias
restoration without mutating provenance records.
Rollback is explicit or triggered by canary/restart reconciliation. It restores the exact prior
snapshot and records elapsed wall-clock time from rollback start until the prior state is restored;
the result is evidence, not an invented SLO. Retrying an already rolled-back operation is
idempotent.
If the snapshot/target is unavailable, the operation becomes `FAILED` with
`ROLLBACK_FAILED` and the subject becomes `MANUAL_INTERVENTION_REQUIRED`. ModelForge never reports
stable after incomplete restoration. Startup reconciliation conservatively rolls back operations
left in `PREPARING`, `ACTIVATING` or `ROLLING_BACK`.
docs/architecture/RUNTIME_COMPATIBILITY.md
+36
View File
@@ -0,0 +1,36 @@
# Runtime Compatibility
## Evidence model
`RuntimeCompatibilityAssessment` records the exact ArtifactSet, RuntimeProfile, ComputeNode,
adapter/version, artifact and hardware snapshots, blockers, warnings, approvals, static result and a
canonical environment fingerprint. It distinguishes runtime-declared support from locally proven
support. Static analysis never claims a model was loaded.
## Common checks
- complete, locally available and integrity-verified exact ArtifactSet;
- no quarantined/corrupt member, unapproved pickle or remote-code requirement;
- all registered artifact locations belong to the target node;
- online, enabled and lab-eligible node advertising `runtime.probe.v1`;
- GPU presence for CUDA-required profiles, compute capability, driver/CUDA compatibility and VRAM;
- environment package, modality and serialization-format declarations;
- required model configuration/tokenizer/processor files and shard-index presence;
- requested quantization evidence.
Resource estimates are stored as `kind=estimated`; only a real probe produces `kind=measured`.
## Runtime-specific inspectors
- SentenceTransformers requires safe Transformers files plus `modules.json` and pooling config.
- Transformers requires config, tokenizer and Safetensors weights.
- vLLM uses Transformers-format checks but remains `requires_probe` until proven for the task.
- llama.cpp requires a local GGUF variant; Safetensors alone yields `NO_GGUF_ARTIFACT_VARIANT`.
- Diffusers requires a pipeline `model_index.json`; an embedding repository is blocked.
- custom runners never receive an implicit compatibility claim.
## Staleness
Every read recomputes the canonical fingerprint from current immutable profile/environment identity,
artifact evidence and hardware facts. Driver, CUDA, GPU, node state, artifact security/integrity,
image or profile changes make prior evidence stale. A stale assessment cannot queue a probe.
docs/architecture/RUNTIME_PLANE.md
+80
View File
@@ -0,0 +1,80 @@
# Runtime Plane
## M9 multimodal extension
The runtime plane now has specialized `transformers_trocr`, `transformers_siglip2`, and `transformers_whisper` adapters. They use exact local artifacts, `local_files_only`, `trust_remote_code=false`, denied network access and typed modality payloads. Runtime environments declare supported modality/model types; RuntimeProfiles bind preprocessing/audio semantics; probes and serving record the same measured resource evidence and unload behavior as embedding/reranking workers.
## Scope
M4 turns verified local bytes into reproducible technical execution evidence. M5 consumes only that
evidence through typed deployment load/invoke/health/drain/unload jobs. It does not expose an
inference service, schedule workloads, bind projects, benchmark quality, or activate production.
```text
ArtifactSet -> CompatibilityAssessment -> RuntimeProfile -> RuntimeProbe -> DeploymentCandidate
```
Each arrow is a separately persisted gate. `DeploymentCandidate.status=lab_ready` and
`production=false` are the terminal M4 state; a separate exact M5 production approval and
CapabilityDeployment are required before routing.
The runtime worker remains outbound-only and private. M10 replaces its historical single resident
slot with a bounded map keyed by deployment UUID. Load, invoke, health, drain and unload target one
exact deployment and do not implicitly clear other proven residents. The worker reports a complete
typed inventory including ArtifactSet, RuntimeProfile, environment fingerprint and 64-bit worker
generation. CUDA allocator measurements prove process-owned model allocation/reclaim; NVML remains
the total-GPU/external-pressure source.
Control-plane restart adopts only an exact inventory match. Worker absence reconciles database state
to cold; an exact worker inventory may be adopted without duplicate load. Unknown GPU processes are
never adopted and remain unmanaged external usage.
## Boundaries
- The control plane owns evidence, approvals, typed leases, state transitions and audit events.
- A static Compose runtime worker on the selected node polls outbound with that node's credential.
- The worker receives a fixed probe type, exact registered relative artifact location, manifest,
immutable profile and environment identity. It never receives a shell command.
- A child process owns the ML runtime and exits after the probe, making process death the final
unload boundary. The parent remains responsive to cancellation and reports NVML evidence.
- Artifacts mount read-only; root filesystem is read-only; `/tmp` is the only writable runtime
filesystem; the worker is non-root, has no published ports, Docker socket, SSH material or host-root
mount, and drops Linux capabilities.
## Adapter lifecycle
`RuntimeAdapter` defines `inspect_support`, `prepare`, `start`, `probe`, `health`, `stop`, `unload`
and `collect_runtime_facts`. M4 executes only the SentenceTransformers adapter. Static inspectors
exist for Transformers, vLLM, llama.cpp, Diffusers and custom runners. Unsupported executable
adapters fail with `RUNTIME_INCOMPATIBLE` rather than falling through to arbitrary code.
## Lifecycle and recovery
Probes move through `queued`, `preparing`, `loading`, `healthchecking`, `ready`, `unloading`, then
`completed`, `failed` or `cancelled`. Node-scoped leases are hashed at rest and expire. An expired
active lease becomes claimable by the same node, increments `attempt_count`, and gets a new token.
No active probe is represented as a deployment candidate.
## Health
The worker reports distinct process, runtime, model and capability layers. M4 requires healthy
process/runtime/model plus a finite non-empty vector, offline-local evidence and successful VRAM
reclamation. Capability health remains `not_routed_in_m4` by design.
## M14 runtime observability
Runtime operational truth remains the typed `ServingJob`, health result, GPU lease and measured
residency records. M14 exports bounded operation/status counters, evaluates load/invoke/unload/health
success, detects a bounded crash loop and correlates capability degradation without sending commands
to workers. Observability persistence failure cannot block or silently reroute inference. See
`docs/architecture/OBSERVABILITY.md` and `docs/operations/RUNBOOK_RUNTIME_FAILURE.md`.
## M16 fault behaviour
A Runtime Worker failure during load, inference or unload produces a typed failure for the request,
fences the worker generation so a late result from a dead generation cannot be committed, and
reconciles the lease and residency. A repeated crash raises `RUNTIME_CRASH_LOOP`.
The `no_orphan_serving_work` invariant asserts that no queued or leased job outlives its lease and
that no residency allocation sits on a disabled node, which is what "the worker failure was
reconciled" has to mean in practice.
docs/architecture/RUNTIME_PROFILES.md
+15
View File
@@ -0,0 +1,15 @@
# Runtime Profiles
A RuntimeEnvironment identifies installed executable software: adapter, exact runtime and package
versions, Python/CUDA versions, supported architectures/formats/modalities and container repository
plus digest. It is immutable after registration.
A RuntimeProfile binds one exact ArtifactSet to one RuntimeEnvironment and execution configuration:
dtype, quantization, modality, sequence length, batch, concurrency, device/GPU-memory policies,
non-secret environment values, launch parameters and layered health contract. Its canonical
fingerprint makes identical creates idempotent. Any behavior-changing update is a new profile,
never an in-place mutation.
`trust_remote_code=false` and `network_egress=false` are literal schema constraints. Environment
variable names containing token, secret, password, key or credential are rejected. Runtime Profiles
contain configuration only; node credentials and hub/project secrets are not part of the object.
docs/architecture/SCHEDULER_QOS.md
+19
View File
@@ -0,0 +1,19 @@
# Scheduler QoS
The fixed safety order is production, interactive, background, LAB. Database row locking on the
accelerator remains authoritative for capacity, while serving jobs use row-lock/skip-locked claims.
Five-minute bounded aging moves old background/LAB work forward without overtaking production.
Backpressure is enforced at three boundaries: per-client concurrency/rate, per-capability queue and
global queue. A full boundary returns `QUEUE_FULL`; queues never grow without limit. Cold requests
whose measured load time exceeds an explicit remaining deadline return `DEADLINE_CANNOT_BE_MET`.
Normal scheduling never interrupts active inference. A higher-priority request may evict only an
idle, lower/equal-priority managed residency or wait. Batch metadata is represented without making
unsafe batching promises. Stored request shape is limited to count, bounded size bucket and
modality-relevant numeric characteristics; text, image and audio content is not scheduler evidence.
The admin metrics surface uses bounded names and histories: admissions, rejections, queue seconds,
evictions, resident/leased/external/schedulable bytes, pressure, cold loads, warm hits, unloads and
reclaim failures. Request IDs are not metric labels and ordinary placements are not audit events.
docs/architecture/SCHEMA_BREAKING_MIGRATIONS.md
+9
View File
@@ -0,0 +1,9 @@
# Schema-breaking migrations
M13 models schema-breaking intent, typed adapter steps and an explicit irreversible marker. It does
not execute user SQL, Python, shell or paths. Production planning requires the stronger M12 approval
boundary and explicit rollback/irreversibility facts; automatic cutover is denied.
Known future adapters may model expand, dual-compatible operation, migrate, verify, cutover and
contract. Each step must be compiled into trusted application code and versioned in the adapter
fingerprint. M13 deliberately makes no generic rollback claim for irreversible work.
docs/architecture/SERVICE_LEVEL_INDICATORS.md
+37
View File
@@ -0,0 +1,37 @@
# Service-level indicators
An SLI is a first-class typed definition of a service or capability, its measurement, valid
population, success condition and default window. An SLI does not contain an objective; objectives
belong to versioned SLO policies.
## Core definitions
| Key | Population | Successful observation |
| --- | --- | --- |
| `platform.api.availability` | control-plane and health requests since process start | 2xx/3xx |
| `gateway.availability` | authenticated gateway requests in window | completed |
| `gateway.latency.p95` | completed gateway requests with latency | p95 below policy threshold |
| `scheduler.correctness` | persisted placement plans | typed admission, queue or refusal verdict |
| `node.freshness` | enabled production nodes | heartbeat age at most 90 seconds |
| `runtime.success` | terminal load/invoke/unload/health jobs | completed |
| `migration.reliability` | terminal migration plans | not failed/manual intervention |
Capability definitions cover RAG success and latency plus Vision and Speech LAB success. The latter
remain explicitly LAB. OCR has no quality or production SLO because M9/M11 quality and external
evidence gates remain unsatisfied.
Project-facing health is derived only for actual scoped service clients and bindings. It reports
project, capability, environment, client state, request/error counts and last use. ExampleRAG's
production Ollama baseline is external to ModelForge and is not relabelled as a ModelForge
production SLI. ExampleVision Vision shadow and ASR remain LAB.
## Evaluation semantics
The evaluator selects only events inside the policy rolling window. It persists population, good
and bad counts, observed value, freshness, policy identity and bounded source evidence. Latency
evidence retains p50/p95/p99, while the initial policy evaluates p95.
Possible states are `HEALTHY`, `AT_RISK`, `BREACHED`, `INSUFFICIENT_DATA`, `STALE` and `DISABLED`.
The minimum population is evaluated before health. No samples or too few samples can therefore
never become green. Control-plane health, capability health and project integration health remain
separate dimensions.
docs/architecture/SHADOW_AND_DUAL_READ.md
+9
View File
@@ -0,0 +1,9 @@
# Shadow and dual read
Shadow targets are separate storage identities and are exposed only to evaluation/admin or an
explicit project shadow route. Ordinary production consumers continue using the current binding.
Compatible adapters may execute one request against current and shadow, but persist only result IDs,
metrics, latency, errors and evidence references—not content.
Shadow completion records a `MigrationShadowSession`. Success reaches `READY_FOR_CUTOVER`; it never
switches traffic. An isolated technical verdict and project-promotion verdict remain distinct.
docs/architecture/SLO_POLICIES.md
+24
View File
@@ -0,0 +1,24 @@
# SLO policies
SLO policies are immutable revisions. Changing objective, threshold, window, population minimum,
severity or environment deactivates the prior revision and creates a new fingerprinted revision.
Historic evaluations keep the exact prior policy ID.
## Initial active policies
| Key | Environment | Objective/threshold | Window | Minimum | Rationale |
| --- | --- | --- | ---: | ---: | --- |
| `platform.api.production` | PRODUCTION | 99% | 24h | 20 | local control-plane availability |
| `gateway.production` | PRODUCTION | 99% | 24h | 20 | valid capability routing |
| `rag.embedding.production.success` | PRODUCTION | 99% | 24h | 20 | established stable capability |
| `rag.embedding.production.latency` | PRODUCTION | p95 ≤ 1000 ms | 24h | 20 | measured warm/shared-GPU envelope, not a quality gate |
| `gpu_node.production.freshness` | PRODUCTION | 99%, max age 90 s | 1h | 1 | existing 30/90 receive-time contract |
| `scheduler.production.correctness` | PRODUCTION | 99.5% | 24h | 20 | typed refusal is valid; internal failure is not |
| `migration.production.reliability` | PRODUCTION | 99% | 7d | 1 | failure/manual intervention consumes budget |
| `vision.embedding.lab` | LAB | 95% | 24h | 5 | operational LAB signal only |
| `speech.transcription.lab` | LAB | 95% | 24h | 5 | operational LAB signal only |
Thresholds use the established node liveness contract and M5–M13 measured behavior. They are not
selected after a failure merely to make the platform green. LAB policy evaluation conveys operating
evidence without production readiness or an availability budget. `BACKGROUND` is supported by the
contract for migration/benchmark work but has no initial production objective.
docs/architecture/SPEECH_CONSUMER_INTEGRATION.md
+21
View File
@@ -0,0 +1,21 @@
# Speech consumer integration
ExampleVision spoken card search is the first operational consumer of `speech.transcription@1`. It is a
small, useful local flow adjacent to the scanner and does not require persisted audio. The dedicated
interactive client is bound only to the speech contract; ExampleVision knows no Whisper model, revision,
runtime, worker or node.
```text
ExampleVision WAV upload -> transient consumer buffer -> ModelForge Gateway -> M10 scheduler
-> GPU Node offline speech runtime -> text -> ExampleVision response
```
The consumer rejects unsupported or oversized input, removes temporary audio after the call, exposes
typed unavailable/auth/capacity failures and has no cloud fallback. ModelForge telemetry stores
dimensions, timing and state, never audio.
Live acceptance used four representative NL/EN Windows TTS samples. All four completed through the
consumer and Gateway with zero errors; consumer p50/p95 latency was 319.622/393.442 ms and the small
suite's normalized WER was 0.380952. This proves the integration but not natural-microphone quality,
so the project recommendation remains `KEEP_LAB`.

Some files were not shown because too many files have changed in this diff Show More