Jens/ModelForge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial public ModelForge release

Browse Source
This commit is contained in:
Jens
2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CONFIGURATION.md
+175
View File
@@ -0,0 +1,175 @@
# Configuration reference
Generated from the typed settings by `scripts/generate_configuration_docs.py` for ITWorx ModelForge 1.2.1. Every setting the control plane reads appears here; a setting added without documentation fails the build.
All settings are environment variables with the `MODELFORGE_` prefix, read from the process environment or from `.env`.
## Required in production
`MODELFORGE_ENV=production` turns on fail-closed startup validation. With it set, the control plane refuses to start unless each of these is present and sound:
- `MODELFORGE_DATABASE_URL` — SQLAlchemy URL for the API's non-owner modelforge_runtime role. It may read the audit trail and execute the canonical append function, but cannot mutate audit tables directly.
- `MODELFORGE_REDIS_URL` — Redis URL for transient request payloads and queues.
- `MODELFORGE_OPERATOR_API_KEY` — Operator API key guarding every admin route. Generate at least 32 random characters; ModelForge never mints one for you.
- `MODELFORGE_BACKUP_ENCRYPTION_KEY` — Base64 AES-256 key for backup encryption. Without it no backup can be produced, and without the same key no backup can be restored — store it outside this deployment.
Production additionally refuses: a well-known development database password, a wildcard CORS origin, remote model code execution, an unwritable storage root, an unsupported schema revision, a PostgreSQL major below 16, and three policy combinations that cannot all hold at once.
## Generating secrets
ModelForge never mints its own credentials — a platform that generates its own admin secret has no way to tell you it did. Generate them yourself and store them outside the deployment:
```bash
# Operator API key (at least 32 characters)
python -c "import secrets; print(secrets.token_urlsafe(48))"
# Backup encryption key (base64 AES-256)
python -c "import base64, os; print(base64.b64encode(os.urandom(32)).decode())"
# Database password
python -c "import secrets; print(secrets.token_urlsafe(32))"
```
Losing the backup encryption key makes every existing backup unrecoverable. It is the one value that must be stored somewhere the deployment cannot take down with it.
## Sensitivity
5 settings are credentials. They are never logged, never written to a release artefact and never echoed in an error response:
- `MODELFORGE_DATABASE_URL`
- `MODELFORGE_MIGRATION_DATABASE_URL`
- `MODELFORGE_OPERATOR_API_KEY`
- `MODELFORGE_HF_TOKEN`
- `MODELFORGE_BACKUP_ENCRYPTION_KEY`
## Every setting
| Variable | Type | Default | Required | Sensitivity | Description |
| --- | --- | --- | --- | --- | --- |
| `MODELFORGE_ENV` | `development`, `test`, `production` | `development` | no | public | Deployment profile. 'production' turns on every fail-closed startup rule; 'development' and 'test' report the same problems without refusing to start. |
| `MODELFORGE_API_HOST` | string | `0.0.0.0` | no | public | Interface the API binds inside its container. Leave at 0.0.0.0. |
| `MODELFORGE_API_PORT` | integer | `8000` | no | public | Port the API listens on inside its container. |
| `MODELFORGE_CONTROL_PLANE_MAX_PAYLOAD_BYTES` | integer | `1048576` | no | public | Pre-parser request-body limit for public and operator control-plane routes. |
| `MODELFORGE_SERVICE_NAME` | string | `modelforge-api` | no | public | Name this process reports in logs and audit events. |
| `MODELFORGE_LOG_LEVEL` | string | `INFO` | no | public | Structured log level: DEBUG, INFO, WARNING or ERROR. |
| `MODELFORGE_DATABASE_URL` | string | — | yes | secret | SQLAlchemy URL for the API's non-owner modelforge_runtime role. It may read the audit trail and execute the canonical append function, but cannot mutate audit tables directly. |
| `MODELFORGE_MIGRATION_DATABASE_URL` | secret, optional | — | no | secret | SQLAlchemy URL for the non-superuser modelforge schema-owner role. Set only in the one-shot migration process; production API startup refuses when this secret is present. |
| `MODELFORGE_REDIS_URL` | string | `redis://localhost:6379/0` | yes | internal | Redis URL for transient request payloads and queues. |
| `MODELFORGE_CORS_ORIGINS` | string | `http://localhost:3000` | no | public | Comma-separated exact origins allowed to call the API from a browser. A wildcard is refused in production because requests are credentialed. |
| `MODELFORGE_OPERATOR_API_KEY` | secret, optional | — | yes | secret | Operator API key guarding every admin route. Generate at least 32 random characters; ModelForge never mints one for you. |
| `MODELFORGE_HF_TOKEN` | secret, optional | — | no | secret | Optional Hugging Face token, used only for acquiring gated repositories. It is never passed to a runtime and never leaves the control plane. |
| `MODELFORGE_BACKUP_ENCRYPTION_KEY` | secret, optional | — | yes | secret | Base64 AES-256 key for backup encryption. Without it no backup can be produced, and without the same key no backup can be restored — store it outside this deployment. |
| `MODELFORGE_BACKUP_ENCRYPTION_KEY_ID` | string | `modelforge-backup-key-1` | no | public | Identifier recorded in each backup manifest so a restore can name the key it needs. |
| `MODELFORGE_HF_HOME` | string | `/data/hf-cache` | no | public | Hugging Face cache root inside the container. |
| `MODELFORGE_ARTIFACT_ROOT` | string | `/data/artifacts` | no | public | Verified model artifact root. Must exist and be writable. |
| `MODELFORGE_QUARANTINE_ROOT` | string | `/data/quarantine` | no | public | Where acquired artifacts are held until their checks pass. |
| `MODELFORGE_RUNTIME_ARTIFACT_ROOT` | string | `/models/model-registry` | no | public | Artifact root as a runtime worker sees it on a compute node. |
| `MODELFORGE_CONFIG_ROOT` | path | `PydanticUndefined` | no | public | Directory holding the capability, project and policy manifests. |
| `MODELFORGE_BACKUP_ROOT` | path | `/data/backups` | no | public | Backup destination. Must exist and be writable, or backups fail closed. |
| `MODELFORGE_BACKUP_RESTORE_ROOT` | path | `/data/restore` | no | public | Working directory a restore stages into before it commits. |
| `MODELFORGE_ALEMBIC_DIRECTORY` | path, optional | — | no | public | Override for the migration directory. Leave empty in a container. |
| `MODELFORGE_HF_TIMEOUT_SECONDS` | number | `30` | no | public | Per-request timeout for Hugging Face metadata calls. |
| `MODELFORGE_HF_SNAPSHOT_TTL_SECONDS` | integer | `3600` | no | public | How long a resolved upstream snapshot stays cached. |
| `MODELFORGE_ALLOW_REMOTE_CODE` | boolean | `false` | no | public | Whether model repositories may execute their own Python. Always false in production; startup refuses any other value there. |
| `MODELFORGE_ENABLE_GPU_TELEMETRY` | boolean | `true` | no | public | Collect GPU telemetry on this host. |
| `MODELFORGE_HARDWARE_REFRESH_ON_STARTUP` | boolean | `false` | no | public | Run a hardware inventory pass when the process starts. |
| `MODELFORGE_HARDWARE_POLL_INTERVAL_SECONDS` | integer | `30` | no | public | Interval between hardware inventory passes. |
| `MODELFORGE_NODE_IDENTITY` | string, optional | — | no | internal | Explicit node identity. Leave empty to use the persisted file. |
| `MODELFORGE_NODE_IDENTITY_MODE` | `auto`, `persisted` | `auto` | no | public | 'persisted' keeps a node's identity across restarts; 'auto' derives it. |
| `MODELFORGE_NODE_IDENTITY_FILE` | path | `/data/state/node-id` | no | public | Where a persisted node identity is stored. |
| `MODELFORGE_NODE_STALE_AFTER_SECONDS` | integer | `30` | no | public | Silence after which a node is considered stale. |
| `MODELFORGE_NODE_OFFLINE_AFTER_SECONDS` | integer | `90` | no | public | Silence after which a node is considered offline. Must exceed the stale threshold. |
| `MODELFORGE_LIVENESS_POLL_INTERVAL_SECONDS` | integer | `5` | no | public | How often node liveness is re-evaluated. |
| `MODELFORGE_AGENT_MAX_CLOCK_SKEW_SECONDS` | integer | `300` | no | public | Clock skew tolerated on an agent report before refusal. |
| `MODELFORGE_NODE_AGENT_MAX_PAYLOAD_BYTES` | integer | `4194304` | no | public | Pre-parser request-body limit for enrollment and authenticated Node Agent reports. |
| `MODELFORGE_NODE_LIVENESS_MONITOR_ENABLED` | boolean | `false` | no | public | Run the node liveness monitor in this process. |
| `MODELFORGE_AGENT_PROTOCOL_VERSION` | integer | `1` | no | public | Agent protocol version this control plane speaks. |
| `MODELFORGE_GATEWAY_MAX_BATCH_SIZE` | integer | `8` | no | public | Maximum inputs accepted in a single capability invocation. |
| `MODELFORGE_GATEWAY_MAX_INPUT_CHARACTERS` | integer | `8192` | no | public | Maximum characters per input item. |
| `MODELFORGE_GATEWAY_MAX_PAYLOAD_BYTES` | integer | `65536` | no | public | Maximum accepted request body size. |
| `MODELFORGE_GATEWAY_REQUEST_TIMEOUT_SECONDS` | integer | `45` | no | public | Total time a capability invocation may take. Must exceed the queue timeout. |
| `MODELFORGE_GATEWAY_QUEUE_TIMEOUT_SECONDS` | integer | `30` | no | public | How long a request may wait for capacity before rejection. |
| `MODELFORGE_SERVING_JOB_LEASE_SECONDS` | integer | `120` | no | public | Lease held by a serving job before it is reclaimed. |
| `MODELFORGE_SERVING_PAYLOAD_TTL_SECONDS` | integer | `120` | no | public | How long a request payload survives in Redis. |
| `MODELFORGE_SCHEDULER_SAFETY_RESERVE_BYTES` | integer | `1073741824` | no | public | VRAM never offered to a placement, as an absolute floor. |
| `MODELFORGE_SCHEDULER_SAFETY_RESERVE_PERCENTAGE` | number | `0.05` | no | public | VRAM never offered to a placement, as a fraction. Half a device leaves nothing schedulable. |
| `MODELFORGE_SCHEDULER_RUNTIME_MARGIN_BYTES` | integer | `268435456` | no | public | Headroom reserved for runtime overhead per node. |
| `MODELFORGE_SCHEDULER_DEPLOYMENT_MARGIN_BYTES` | integer | `134217728` | no | public | Absolute headroom added to each deployment estimate. |
| `MODELFORGE_SCHEDULER_DEPLOYMENT_MARGIN_PERCENTAGE` | number | `0.1` | no | public | Proportional headroom added to each estimate. |
| `MODELFORGE_SCHEDULER_GLOBAL_QUEUE_LIMIT` | integer | `128` | no | public | Queued requests accepted before capacity rejection begins. |
| `MODELFORGE_SCHEDULER_TELEMETRY_STALE_SECONDS` | integer | `90` | no | public | Telemetry age past which admission is blocked rather than extrapolated. |
| `MODELFORGE_SCHEDULER_PRESSURE_STABLE_SECONDS` | integer | `30` | no | public | How long pressure must hold before the state changes. |
| `MODELFORGE_SCHEDULER_EVICTION_COOLDOWN_SECONDS` | integer | `60` | no | public | Minimum interval between evictions on a node. |
| `MODELFORGE_SCHEDULER_PLACEMENT_HISTORY_LIMIT` | integer | `500` | no | public | Placement decisions retained for inspection. |
| `MODELFORGE_REGISTRY_SEED_ON_STARTUP` | boolean | `false` | no | public | Seed the candidate and project registries from manifests. |
| `MODELFORGE_SERVING_RECONCILIATION_ENABLED` | boolean | `false` | no | public | Reconcile abandoned serving work in this process. |
| `MODELFORGE_SERVING_RECONCILIATION_INTERVAL_SECONDS` | integer | `5` | no | public | Interval between serving reconciliation passes. |
| `MODELFORGE_LIFECYCLE_RECONCILIATION_ENABLED` | boolean | `false` | no | public | Roll back incomplete lifecycle operations at startup. |
| `MODELFORGE_MIGRATION_RECONCILIATION_ENABLED` | boolean | `false` | no | public | Report interrupted migration cutovers at startup. They are never auto-resolved: external alias truth cannot be inferred after a crash. |
| `MODELFORGE_OBSERVABILITY_MONITOR_ENABLED` | boolean | `false` | no | public | Run SLO and alert evaluation in this process. |
| `MODELFORGE_OBSERVABILITY_POLL_INTERVAL_SECONDS` | integer | `60` | no | public | Interval between observability evaluation passes. |
| `MODELFORGE_RECOVERY_RECONCILIATION_ENABLED` | boolean | `false` | no | public | Reconcile interrupted backups and restores at startup. |
| `MODELFORGE_BACKUP_PG_DUMP_PATH` | string | `pg_dump` | no | public | pg_dump executable. Must match the server major version. |
| `MODELFORGE_BACKUP_PG_RESTORE_PATH` | string | `pg_restore` | no | public | pg_restore executable. |
| `MODELFORGE_BACKUP_PSQL_PATH` | string | `psql` | no | public | psql executable. |
| `MODELFORGE_BACKUP_COMMAND_TIMEOUT_SECONDS` | integer | `1800` | no | public | Timeout for a dump or restore command. |
| `MODELFORGE_BACKUP_STALE_AFTER_SECONDS` | integer | `93600` | no | public | Age past which the newest verified backup raises BACKUP_STALE. |
| `MODELFORGE_BACKUP_MINIMUM_FREE_BYTES` | integer | `1073741824` | no | public | Free space below which a backup refuses to start. |
| `MODELFORGE_BACKUP_CAPACITY_HEADROOM_RATIO` | number | `3.0` | no | public | Required free space as a multiple of the estimated size. |
| `MODELFORGE_RESTORE_ALLOW_PRODUCTION_TARGET` | boolean | `false` | no | public | Whether a restore may overwrite the live database. Keep false outside a rehearsal. |
| `MODELFORGE_BUILD_COMMIT` | string, optional | — | no | public | Source commit stamped into the image at build time. Never set by hand. |
| `MODELFORGE_BUILD_TIMESTAMP` | string, optional | — | no | public | Build time stamped into the image. Never set by hand. |
| `MODELFORGE_BUILD_IMAGE_DIGEST` | string, optional | — | no | public | Image digest recorded at deployment. Never set by hand. |
## Deployment variables
Read by Compose, the Node Agent and the Runtime Worker rather than by the control-plane process. An operator still has to set them, so they are documented here too.
| Variable | Required | Sensitivity | Description |
| --- | --- | --- | --- |
| `MODELFORGE_POSTGRES_BIND` | no | public | Host address the control-plane database is published on. Defaults to 127.0.0.1; publishing it more widely exposes provenance, credential hashes and the audit trail. |
| `MODELFORGE_REDIS_BIND` | no | public | Host address Redis is published on. Defaults to 127.0.0.1. |
| `MODELFORGE_API_BIND` | no | public | Host address the API is published on. Defaults to 0.0.0.0 deliberately: the console and compute nodes need it, and every admin route is operator-authenticated. |
| `MODELFORGE_WEB_BIND` | no | public | Host address the console is published on. Defaults to 127.0.0.1. |
| `MODELFORGE_POSTGRES_PORT` | no | public | Host port the database is published on. Defaults to 5432. |
| `MODELFORGE_REDIS_PORT` | no | public | Host port Redis is published on. Defaults to 6379. |
| `MODELFORGE_API_PUBLISHED_PORT` | no | public | Host port the API is published on. Defaults to 8000. |
| `MODELFORGE_WEB_PORT` | no | public | Host port the console is published on. Defaults to 3000. |
| `MODELFORGE_DR_POSTGRES_BIND` | no | public | Host address for the DR rehearsal database. Loopback only. |
| `MODELFORGE_DR_API_BIND` | no | public | Host address for the DR rehearsal API. Loopback only. |
| `VITE_API_BASE_URL` | no | public | API base URL compiled into the console. Vite inlines it at build time, so changing it requires rebuilding the console image, not restarting it. |
| `MODELFORGE_POSTGRES_DB` | no | public | Production database name. Required by the production overlay. |
| `MODELFORGE_POSTGRES_ADMIN_USER` | no | internal | Bootstrap/admin role used only by PostgreSQL provisioning; defaults to postgres. |
| `MODELFORGE_POSTGRES_ADMIN_PASSWORD` | yes | secret | Bootstrap/admin password; never passed to the migration or API container. |
| `MODELFORGE_MIGRATION_DB_PASSWORD` | yes | secret | Raw password supplied to provisioning for the non-superuser modelforge owner role. |
| `MODELFORGE_RUNTIME_DB_PASSWORD` | yes | secret | Raw password supplied to provisioning for the non-owner modelforge_runtime role. |
| `MODELFORGE_RUNTIME_DATABASE_URL` | yes | secret | Non-owner runtime-role URL passed only to the API container. |
| `MODELFORGE_VERSION` | no | public | Exact version tag applied to built images and required when the production overlay is not given explicit API and web image references. |
| `MODELFORGE_COMMIT` | no | public | Source commit stamped into images at build time. |
| `MODELFORGE_BUILT_AT` | no | public | Build timestamp stamped into images. |
| `MODELFORGE_API_IMAGE` | no | public | Exact tag or digest the production overlay runs for the API; never use latest. |
| `MODELFORGE_WEB_IMAGE` | no | public | Exact tag or digest the production overlay runs for the console; never use latest. |
| `MODELFORGE_NODE_AGENT_IMAGE` | no | public | Exact release tag or digest for the standalone Node Agent. The local-build fallback is named local and never resolves to latest. |
| `MODELFORGE_API_IMAGE_DIGEST` | no | public | Digest recorded as the running API build identity. |
| `MODELFORGE_BACKUP_VOLUME` | no | public | Volume or bind path backing the backup root. |
| `MODELFORGE_RESTORE_VOLUME` | no | public | Volume or bind path backing the restore staging root. |
| `MODELFORGE_AGENT_STATE_VOLUME` | no | public | Volume or bind path holding the agent's persisted identity. |
| `MODELFORGE_AGENT_HF_CACHE_VOLUME` | no | public | Volume or bind path for the agent's Hugging Face cache. |
| `MODELFORGE_AGENT_ARTIFACT_VOLUME` | no | public | Volume or bind path for verified artifacts on a node. |
| `MODELFORGE_AGENT_QUARANTINE_VOLUME` | no | public | Volume or bind path for the node's quarantine area. |
| `MODELFORGE_AGENT_CONTROL_PLANE_URL` | no | public | URL the agent reports to. Outbound only; the control plane never dials a node. |
| `MODELFORGE_AGENT_ENROLLMENT_TOKEN` | no | secret | Single-use enrolment token. Consumed atomically: a storm against one token produces exactly one identity. |
| `MODELFORGE_AGENT_HOSTNAME` | no | public | Hostname the agent enrols under. |
| `MODELFORGE_AGENT_ACCELERATOR_MODE` | no | public | Accelerator contract: nvidia fails closed unless NVML inventory and telemetry are valid; cpu permits a legitimate CPU-only node; auto requires NVIDIA when injected devices are observed. Canonical GPU Compose deployments set nvidia explicitly. |
| `MODELFORGE_AGENT_TLS_VERIFY` | no | public | Whether the agent verifies the control plane's certificate. True wherever TLS is real. |
| `MODELFORGE_AGENT_CONTROL_PLANE_HOST_ADDRESS` | no | internal | Host address mapped for a private-CA deployment. |
| `MODELFORGE_AGENT_CA_CERT_PATH` | no | public | Path to the private CA certificate the agent trusts. |
| `MODELFORGE_RUNTIME_WORKER_ARTIFACT_ROOT` | no | public | Artifact root as the runtime worker sees it. |
| `MODELFORGE_RUNTIME_WORKER_POLL_INTERVAL_SECONDS` | no | public | Worker poll interval, in seconds. |
| `MODELFORGE_SOURCE_COMMIT` | no | public | Source commit reported by the deployment. |
| `MODELFORGE_SOURCE_REFERENCE` | no | public | Git reference reported by the deployment. |
| `MODELFORGE_SOURCE_REPOSITORY` | no | public | Repository URL reported by the deployment. |
## Restarts
Every setting is read at process start. Changing any of them requires restarting the control plane; none is re-read from the environment while the process is running.