Initial public ModelForge release
This commit is contained in:
Jens
@@ -0,0 +1,61 @@
|
||||
# Advanced GPU Scheduler
|
||||
|
||||
M10 turns the single-GPU admission boundary into one deterministic planner used by dry-runs and
|
||||
real requests. The planner consumes current NVML use, managed residency, only the unmaterialized
|
||||
part of active leases, authoritative envelopes, health, queue state, priority and versioned policy.
|
||||
|
||||
## Accounting invariant
|
||||
|
||||
NVML is the physical truth and already includes loaded ModelForge models:
|
||||
|
||||
```text
|
||||
physical_used = max(observed_nvml_used, attributed_managed_resident)
|
||||
unmanaged_external = max(0, observed_nvml_used - attributed_managed_resident)
|
||||
future_lease = max(0, reserved - materialized)
|
||||
schedulable = max(0, total - physical_used - future_leases - dynamic_reserve)
|
||||
```
|
||||
|
||||
This deliberately does not subtract residency twice. Attribution is `KNOWN`, `ESTIMATED` or
|
||||
`UNKNOWN`; missing or stale telemetry is `UNKNOWN` and has zero schedulable capacity. The invariant
|
||||
delta is exposed for diagnosis without permitting negative external use.
|
||||
|
||||
The active `m10-v1` policy uses the maximum of 1 GiB, 5% total VRAM and a 256 MiB runtime margin.
|
||||
Deployment admission uses measured peak plus the maximum of 128 MiB or 10% measured peak. These
|
||||
values are one versioned policy, not dispersed constants.
|
||||
|
||||
## PlacementPlan
|
||||
|
||||
Every plan records capability, node and accelerator, eligible state, residents, unmanaged use,
|
||||
reserve, requirement, priority, evictions, before/after headroom, reason codes and a deterministic
|
||||
fingerprint. Verdicts are `ADMIT`, `ADMIT_AFTER_EVICTION`, `QUEUE`, `REJECT_CAPACITY`,
|
||||
`REJECT_HEALTH` and `REJECT_POLICY`. Recent history is bounded to 500 records.
|
||||
|
||||
Pressure worsens immediately through `NORMAL`, `ELEVATED`, `HIGH`, `CRITICAL`. Recovery requires a
|
||||
separate threshold and 30 seconds of stable evidence. At HIGH/ELEVATED, new background/LAB loads
|
||||
wait; at CRITICAL, new admissions fail with `EXTERNAL_GPU_PRESSURE`. ModelForge never changes an
|
||||
unmanaged process.
|
||||
|
||||
## Operations and security
|
||||
|
||||
Operator controls are limited to dry-run, policy revision/LAB pause, managed residency policy,
|
||||
drain and idle unload. There is no shell, Docker socket, SSH or GPU-PID control. Ordinary clients
|
||||
select only a capability. Scheduler evidence contains bounded shapes and hashes, never content.
|
||||
|
||||
M11 validates the existing scheduler rather than adding a project shortcut. ExampleVision reference
|
||||
index requests use a background-bound client, while card and speech requests use interactive-bound
|
||||
clients. During a 40-reference indexing continuation, ten interactive vision requests all completed.
|
||||
Vision and ASR also ran under observed external GPU pressure without OOM; final unload returned
|
||||
managed residency and leases to zero.
|
||||
|
||||
|
||||
## M16 pressure behaviour
|
||||
|
||||
Under GPU pressure the scheduler rejects or queues rather than risking an out-of-memory failure. A
|
||||
capacity rejection is a correct outcome, not a failure, and the soak counts those separately from
|
||||
internal errors for exactly that reason.
|
||||
|
||||
External VRAM is measured as pressure and never reclaimed: Ollama, Plex, Tdarr and any other
|
||||
unmanaged process are observed, never stopped, paused, throttled or reconfigured.
|
||||
|
||||
Stale telemetry blocks admission rather than being extrapolated, and `no_stale_gpu_lease` asserts
|
||||
no expired lease is still presented as a live reservation.
|
||||
@@ -0,0 +1,37 @@
|
||||
# Advisor evidence
|
||||
|
||||
## M9 capability recommendations
|
||||
|
||||
`/api/v1/capability-advisor` evaluates each contract independently. No active deployment yields `REQUIRES_MORE_EVIDENCE`; a proven LAB deployment without a typed suite yields `LAB_READY`; completed evidence satisfying every declared threshold yields `PROMOTION_ELIGIBLE`; failed cases, failed runs or threshold regressions yield `BLOCKED`. These states are advisory and never trigger promotion. Retrieval's comparison-specific `KEEP_CURRENT` policy remains intact.
|
||||
|
||||
Advisor recommendations are deterministic outputs of a versioned persisted policy. Project-local
|
||||
evaluation is evidence class A and outranks generic, external and vendor evidence. Critical
|
||||
regressions, supply-chain failures and runtime-fit failures are hard gates; latency remains a distinct
|
||||
operational dimension.
|
||||
|
||||
Each recommendation preserves metric deltas, critical regressions, resource evidence, migration
|
||||
impact, blockers, rationale, confidence and policy version. Dismissal is supported, but Advisor has no
|
||||
approve or promote endpoint. M7's real ExampleRAG recommendations legitimately resolve to
|
||||
`KEEP_CURRENT` or `REQUIRES_MORE_EVIDENCE`.
|
||||
|
||||
M10 lets Advisor consume scheduler readiness, authoritative ResourceEnvelope freshness and measured
|
||||
co-residency as operational evidence. It may explain that a candidate is technically eligible but
|
||||
not proven for sustained coexistence. Scheduler evidence never overrides quality, supply-chain,
|
||||
migration or approval gates and never triggers an automatic promotion.
|
||||
|
||||
M11 adds immutable project-fit evidence. It is scoped to one project binding and cannot make a global
|
||||
“best model” claim. Advisor exposes engineering integration separately from project production fit:
|
||||
ExampleVision vision engineering is `PASS`, while project fit remains `REQUIRES_MORE_EVIDENCE`, production
|
||||
validation is `DEFERRED_EXTERNAL_VALIDATION` and production action is `NONE`. Its speech flow is
|
||||
`KEEP_LAB` because four TTS cases do not establish microphone quality. Only reviewed `OWNER_PHOTO`
|
||||
evidence may satisfy the ExampleVision production gate; public physical captures and catalog references
|
||||
retain their weaker typed class. Blockers or critical failures make `PROMOTION_ELIGIBLE` invalid, and
|
||||
the referenced deployment must implement the binding's exact contract.
|
||||
|
||||
M12 consumes Advisor output only as evidence. Advisor may recommend `PROMOTION_ELIGIBLE`, but only
|
||||
the lifecycle engine can evaluate the active policy revision, create an approval snapshot and permit
|
||||
an immutable plan. There is no Advisor-to-promotion shortcut.
|
||||
|
||||
M13 may reference an Advisor/project-fit verdict in an immutable validation snapshot. Advisor still
|
||||
cannot create, start, validate or cut over a migration; missing or adverse project evidence remains a
|
||||
separate production blocker even when an isolated technical rehearsal passes.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Advisor Pipeline Evidence
|
||||
|
||||
Advisor candidates are explicitly typed as `embedding_deployment` or `retrieval_pipeline`.
|
||||
Embedding candidates reference completed evaluation runs; pipeline candidates reference immutable
|
||||
pipeline identities whose latest completed fixed-pool runs provide control/candidate metrics.
|
||||
|
||||
Hard gates remain deterministic: incomplete/non-comparable evidence, critical regression, security
|
||||
or license block, runtime/GPU-fit failure, unsupported migration and latency policy override
|
||||
aggregate gains. Unknown evidence stays unknown.
|
||||
|
||||
Pipeline recommendations may return
|
||||
`KEEP_CURRENT_EMBEDDING_ADD_RERANKER_CANDIDATE`, but M8's two real reranking runs returned
|
||||
`KEEP_CURRENT`. The evaluated E5 embedding also returned `KEEP_CURRENT`: its aggregate retrieval
|
||||
metrics improved, but three critical regressions, unapproved license state, a required reindex and
|
||||
the p95 latency policy block promotion. Recommendations never promote or cut over automatically.
|
||||
@@ -0,0 +1,48 @@
|
||||
# Deduplicated alerting
|
||||
|
||||
`AlertRuleRevision` is an immutable, fingerprinted rule containing signal/SLO, condition, pending
|
||||
duration, severity, bounded labels, cooldown and recovery condition. Changes create a revision.
|
||||
|
||||
## State machine
|
||||
|
||||
```text
|
||||
clear → PENDING → FIRING → ACKNOWLEDGED
|
||||
│ │ │
|
||||
└──────────┴────────────┴→ RESOLVED
|
||||
bad + maintenance window → SUPPRESSED
|
||||
SUPPRESSED + window ended + still bad → PENDING
|
||||
RESOLVED + cooldown elapsed + recurrence → PENDING/FIRING
|
||||
```
|
||||
|
||||
Acknowledgement records actor/reason but cannot alter the signal or SLO. Resolution is an explicit
|
||||
history event; alerts do not disappear. The deterministic fingerprint uses rule key, subject type
|
||||
and subject reference, so every polling cycle updates one logical alert rather than creating spam.
|
||||
|
||||
Severities are `INFO`, `WARNING` and `CRITICAL`. Initial rules cover production node liveness, GPU
|
||||
pressure, storage reserve, gateway errors, stable capability availability, runtime crash loops,
|
||||
queue saturation, migration/manual-intervention failure, rollback failure, artifact integrity and
|
||||
production capability envelope/headroom.
|
||||
|
||||
Maintenance windows have bounded matchers (`rule_key`, `alert_type`, `subject_ref`), start/end,
|
||||
reason and creator. Creation is audited and suppression remains visible. M14 delivers persistent
|
||||
records, API and UI; it has no mandatory Slack/email dependency and performs no automatic restart,
|
||||
kill, promotion, rollback or deletion.
|
||||
|
||||
## M15 recovery rules
|
||||
|
||||
Five deduplicated rules extend the same lifecycle to recovery readiness:
|
||||
|
||||
```text
|
||||
BACKUP_STALE WARNING no verified backup inside the policy window
|
||||
BACKUP_FAILED CRITICAL a backup failed within the window
|
||||
BACKUP_VERIFICATION_FAILED CRITICAL a backup failed specifically on hash or manifest verification
|
||||
RESTORE_FAILED CRITICAL a restore failed or needs manual intervention
|
||||
RECOVERY_READINESS_DEGRADED WARNING an authoritative asset is unprotected
|
||||
```
|
||||
|
||||
They read the same authoritative records the recovery dashboard reads, so an alert and the dashboard
|
||||
can never disagree. A `CREATED` backup that was never verified does not clear `BACKUP_STALE`.
|
||||
|
||||
Live evidence: `BACKUP_STALE` and `RECOVERY_READINESS_DEGRADED` fired while no verified backup
|
||||
existed and resolved once one did, each as a single alert with an occurrence count rather than one
|
||||
alert per evaluation.
|
||||
@@ -0,0 +1,114 @@
|
||||
# API Blueprint
|
||||
|
||||
## M5 inference boundary
|
||||
|
||||
- `POST /api/v1/capabilities/rag.embedding@1/invoke` — native capability response.
|
||||
- `POST /v1/embeddings` — documented OpenAI embedding subset using alias `rag.embedding`.
|
||||
- `GET /api/v1/capability-deployments` and `GET /api/v1/scheduler` — public operational state.
|
||||
- `/api/v1/admin/service-clients`, production approval/promotion, request history and lifecycle
|
||||
actions require the operator credential.
|
||||
- `/api/v1/agent/serving-*` is node-credential-only and never project-facing.
|
||||
|
||||
All inference failures use typed codes. `X-Correlation-ID` and the response request ID support
|
||||
end-to-end tracing without retaining request content.
|
||||
|
||||
This document defines intended API surfaces, not frozen implementation details.
|
||||
|
||||
## Control-plane API
|
||||
|
||||
Operator/UI-facing resource surfaces:
|
||||
|
||||
```text
|
||||
GET /api/v1/system
|
||||
GET /api/v1/health/live
|
||||
GET /api/v1/health/ready
|
||||
GET /api/v1/hardware/nodes
|
||||
GET /api/v1/hardware
|
||||
GET /api/v1/hardware/nodes/{id}
|
||||
GET /api/v1/hardware/accelerators
|
||||
GET /api/v1/hardware/accelerators/{id}
|
||||
POST /api/v1/hardware/refresh
|
||||
POST /api/v1/admin/node-enrollments
|
||||
GET /api/v1/admin/node-enrollments
|
||||
DELETE /api/v1/admin/node-enrollments/{id}
|
||||
PATCH /api/v1/admin/hardware/nodes/{id}
|
||||
DELETE /api/v1/admin/hardware/nodes/{id}/credential
|
||||
POST /api/v1/admin/hardware/nodes/{id}/credential/rotate
|
||||
POST /api/v1/admin/hardware/nodes/{id}/decommission/preview
|
||||
POST /api/v1/admin/hardware/nodes/{id}/decommission
|
||||
GET /api/v1/models
|
||||
GET /api/v1/models/{id}
|
||||
GET /api/v1/models/{id}/revisions
|
||||
POST /api/v1/models/discover
|
||||
POST /api/v1/models/{id}/download
|
||||
GET /api/v1/artifacts
|
||||
GET /api/v1/deployments
|
||||
POST /api/v1/deployments
|
||||
POST /api/v1/deployments/{id}/validate
|
||||
POST /api/v1/deployments/{id}/promote
|
||||
POST /api/v1/deployments/{id}/rollback
|
||||
GET /api/v1/capabilities
|
||||
GET /api/v1/projects
|
||||
GET /api/v1/projects/{id}/bindings
|
||||
GET /api/v1/benchmarks/suites
|
||||
POST /api/v1/benchmarks/runs
|
||||
GET /api/v1/recommendations
|
||||
GET /api/v1/jobs
|
||||
GET /api/v1/security/findings
|
||||
GET /api/v1/audit/events
|
||||
```
|
||||
|
||||
Remote compute-node agent surface (protocol v1):
|
||||
|
||||
```text
|
||||
POST /api/v1/agent/enroll
|
||||
POST /api/v1/agent/heartbeat
|
||||
PUT /api/v1/agent/inventory
|
||||
PUT /api/v1/agent/telemetry
|
||||
```
|
||||
|
||||
Agent endpoints accept no arbitrary command or runtime-launch payload. See `NODE_AGENT_PROTOCOL.md`.
|
||||
|
||||
All lifecycle-changing operations should become asynchronous jobs once work can exceed a normal request duration.
|
||||
|
||||
## Project-facing capability gateway
|
||||
|
||||
Native surface:
|
||||
|
||||
```text
|
||||
POST /api/v1/capabilities/{capability-key}
|
||||
```
|
||||
|
||||
Project identity must be authenticated. Capability version/channel may be selected only within policy.
|
||||
|
||||
## Compatibility facades
|
||||
|
||||
Where semantically safe, provide OpenAI-compatible or other standard interfaces so existing projects can redirect their base URL to ModelForge without coupling to a runtime worker.
|
||||
|
||||
A compatibility facade must still route through project authentication, capability resolution, scheduler policy, telemetry and audit.
|
||||
|
||||
## Error model
|
||||
|
||||
Normalized errors should identify the layer without exposing secrets:
|
||||
|
||||
- `capability_unavailable`
|
||||
- `deployment_unhealthy`
|
||||
- `scheduler_capacity_exhausted`
|
||||
- `request_timeout`
|
||||
- `invalid_capability_input`
|
||||
- `policy_denied`
|
||||
- `migration_required`
|
||||
- `project_not_authorized`
|
||||
|
||||
Runtime-specific stack traces remain internal.
|
||||
|
||||
Request bodies are rejected before parsing with `request_body_too_large` (HTTP 413) when their
|
||||
declared or streamed byte boundary is exceeded. The receiver also allows at most 32 consecutive
|
||||
empty request events, 128 empty events in total and 4096 body events; exceeding any event/progress
|
||||
budget returns exactly one `request_body_progress_exhausted` error (HTTP 400) and stops receiving.
|
||||
Finite empty frames below those limits, normal chunks and disconnect events retain normal ASGI
|
||||
semantics.
|
||||
|
||||
Node-decommission conflicts use `node_decommission_blocked`, `decommission_preview_stale`,
|
||||
`node_generation_conflict` or `decommission_confirmation_mismatch`, with blocker details in the
|
||||
normal correlation-ID error envelope. There is no force parameter.
|
||||
@@ -0,0 +1,25 @@
|
||||
# Approval Policies
|
||||
|
||||
`LifecyclePolicyRevision` is immutable apart from its active selector. Editing requirements creates
|
||||
a new revision; existing requests retain their original revision and evidence fingerprint.
|
||||
|
||||
M12 bootstraps three policies:
|
||||
|
||||
- `lab-promotion`: exact identity, verified integrity, non-blocked security/license, proven runtime,
|
||||
current scheduler readiness and rollback target;
|
||||
- `capability-production`: exact production execution approval, completed evaluation, approved
|
||||
security/license, scheduler evidence revision, zero critical regressions and rollback target;
|
||||
- `project-production`: all capability requirements plus eligible project fit and satisfied project
|
||||
production validation.
|
||||
|
||||
An `ApprovalRequest` freezes declared fields and database-resolved facts. Exact references include
|
||||
ArtifactSet, ModelRevision, RuntimeProfile, RuntimeProbe, worker image digest, production execution
|
||||
approval, evaluation runs, ProjectFitEvidence, ResourceEnvelope, embedding/pipeline identity,
|
||||
migration and rollback target. Claims are checked against resolved relationships; green labels
|
||||
cannot substitute for records.
|
||||
|
||||
Requests are `PENDING`, `APPROVED`, `REJECTED`, `BLOCKED`, `EXPIRED`, `STALE` or `REVOKED`.
|
||||
Expiration is evaluated before use. Re-resolving evidence detects changed probes, profiles, fit,
|
||||
approvals and other facts and makes the request stale. A deferred ExampleVision production gate produces
|
||||
`REQUIRED_EXTERNAL_VALIDATION_NOT_SATISFIED`; security, license and critical regression gates cannot
|
||||
be manually waved through by the default production policies.
|
||||
@@ -0,0 +1,57 @@
|
||||
# Artifact Acquisition
|
||||
|
||||
## Flow
|
||||
|
||||
```text
|
||||
Hub metadata -> exact revision -> artifact set -> immutable plan -> explicit approval
|
||||
-> node-scoped lease -> quarantine download -> static verification -> atomic promotion
|
||||
-> artifact/location registration
|
||||
```
|
||||
|
||||
`DownloadPlan` captures the artifact set, exact SHA, selected paths and expected checksums/sizes,
|
||||
target node/root, total bytes, capacity result, security constraints, expiry and a canonical
|
||||
idempotency key. Creation is a dry run with no bytes. Approval is a separate audited action and
|
||||
execution rejects a merely planned or stale plan.
|
||||
|
||||
The planner prefers complete advertised Safetensors weights plus configuration/index/tokenizer
|
||||
support and excludes duplicate pickle/GGUF weights. If only risky pickle-style weights remain, the
|
||||
risk is explicit and plan approval fails. No format conversion, quantization or loader execution
|
||||
occurs.
|
||||
|
||||
## Placement and ownership
|
||||
|
||||
Jobs are claimed only by their `compute_node_id` through node-scoped bearer authentication. A lease
|
||||
token binds progress/completion/failure calls to one claim. PostgreSQL row locking and one job per
|
||||
plan enforce single-node concurrency and replay/idempotency boundaries. The outbound agent loop
|
||||
processes one acquisition at a time and has no shell, generic command, Docker socket or inbound
|
||||
listener.
|
||||
|
||||
Capacity is checked from the registered root during planning and execution, then checked again from
|
||||
the real target filesystem by the agent before download. Failure never falls back to another root.
|
||||
GPU Node uses `/data/artifacts/model-registry`, mapped to the cache-backed host root; the Unraid array
|
||||
is not a placement candidate.
|
||||
|
||||
## Retry, cancellation, and recovery
|
||||
|
||||
Transient failures have at most three automatic claims. Partial files use `.part`, byte ranges and
|
||||
durable flushes. A complete partial is verified without a new request. A failed job can receive an
|
||||
explicit audited operator retry; its progress epoch resets but historical attempts remain. Cancel
|
||||
is observed at progress boundaries and cannot promote an artifact.
|
||||
|
||||
Expired leases make work reclaimable after agent/control-plane interruption. If promotion succeeded
|
||||
but the database response was lost, the exact-revision manifest makes promotion idempotent; the
|
||||
same plan/job can reconcile. A verified existing plan returns the existing completed job rather
|
||||
than downloading again.
|
||||
|
||||
## API
|
||||
|
||||
- `POST /api/v1/download-plans`
|
||||
- `GET /api/v1/download-plans/{id}`
|
||||
- `POST /api/v1/download-plans/{id}/approve`
|
||||
- `POST /api/v1/download-plans/{id}/execute`
|
||||
- `GET /api/v1/artifact-jobs[/{id}]`
|
||||
- `POST /api/v1/artifact-jobs/{id}/cancel`
|
||||
- `POST /api/v1/artifact-jobs/{id}/retry`
|
||||
|
||||
Agent routes are separately authenticated and expose only claim, typed progress, completion and
|
||||
failure operations.
|
||||
@@ -0,0 +1,65 @@
|
||||
# Artifact Provenance
|
||||
|
||||
## Content identity
|
||||
|
||||
An artifact is identified by its parent exact revision and SHA-256 digest. Filename, storage root
|
||||
and path are descriptive/location data and cannot change identity. One artifact can therefore have
|
||||
zero, one or many physical `ArtifactLocation` records across compute nodes and storage roots.
|
||||
|
||||
Expected digest and byte size are registered before a local file is trusted. Artifact state is one
|
||||
of `remote`, `local`, `verifying`, `verified`, `missing`, `corrupt`, `quarantined` or `archived`.
|
||||
|
||||
## Verification boundary
|
||||
|
||||
M2 verification performs only static file operations:
|
||||
|
||||
1. resolve a relative path beneath the configured storage root and reject path escape;
|
||||
2. require a regular file;
|
||||
3. stream bytes in 1 MiB chunks into SHA-256 while counting bytes;
|
||||
4. compare both digest and size;
|
||||
5. persist observed digest/size/time, status and an audit event.
|
||||
|
||||
It never deserializes weights, imports repository Python, executes model code or contacts a model
|
||||
hub. Missing files become `missing`; digest or size mismatch becomes `corrupt`; only a complete match
|
||||
becomes `verified`. Failure keeps the artifact quarantined.
|
||||
|
||||
## Derived lineage
|
||||
|
||||
`DerivedArtifactSource` stores an ordered N-source relation. Every source ID and hash is captured.
|
||||
The output also records transformation type, tool, tool version, complete configuration,
|
||||
environment snapshot, output digest and byte size. M2 registers this provenance but performs no
|
||||
conversion or quantization.
|
||||
|
||||
Derived content fields are immutable after registration. Source edges make source-artifact deletion
|
||||
return HTTP 409. This preserves the ability to reconstruct why an output exists even when it has no
|
||||
current physical location.
|
||||
|
||||
## Audit actions
|
||||
|
||||
M2 emits append-only hash-chained events for model creation/update/deprecation/deletion, revision
|
||||
registration, artifact registration and verification, derived registration, storage-root validation,
|
||||
capacity checks and blocked deletion attempts.
|
||||
|
||||
## M3 acquisition chain
|
||||
|
||||
M3 persists the complete chain from model and exact revision through upstream snapshot/file,
|
||||
artifact set, immutable approved plan, target-node job/attempt, quarantine path, static inspection,
|
||||
local SHA-256, verified artifact and physical location. Artifact-set membership records required
|
||||
files explicitly. `static_checks_passed_unapproved` means local integrity evidence only and cannot
|
||||
be confused with a later supply-chain or runtime approval.
|
||||
|
||||
M4 extends the chain without mutating artifact identity: exact-set approval fingerprints reviewed
|
||||
security facts, static assessments snapshot artifact/hardware/runtime facts, probes record the same
|
||||
ArtifactSet/revision/profile/node plus measured runtime facts, and candidates refer to the successful
|
||||
probe. Runtime image digests and environment fingerprints are evidence, not artifact digests.
|
||||
|
||||
The promoted `.modelforge-manifest.json` is reconciliation evidence beside the bytes. The database
|
||||
remains operational truth; a retry can reconcile an exact promoted directory after an interrupted
|
||||
completion report without changing content identity.
|
||||
|
||||
## M12 lifecycle evidence
|
||||
|
||||
Approval snapshots add exact policy, ArtifactSet, revision, runtime/probe/image, evaluation, project
|
||||
fit, scheduler, migration and rollback references without copying inference content. Cleanup retains
|
||||
this chain even when one binary location is later removed: the prior digest, size, location status,
|
||||
actor, plan and event remain queryable.
|
||||
@@ -0,0 +1,72 @@
|
||||
# Artifact recovery
|
||||
|
||||
ModelForge holds several gigabytes of model weights. Copying them into every backup would be
|
||||
expensive and pointless for artifacts that are reproducible from an exact upstream identity, and
|
||||
dangerous to assume for artifacts that are not. Artifacts are therefore classified, not treated
|
||||
uniformly.
|
||||
|
||||
```text
|
||||
REHYDRATABLE exact upstream repository + commit SHA + per-file SHA-256 exist
|
||||
NON_REHYDRATABLE no exact upstream identity; requires a payload backup
|
||||
DERIVED produced locally from exact source artifacts; lineage must survive a rebuild
|
||||
LOCAL_ONLY never had an upstream; payload backup only
|
||||
```
|
||||
|
||||
## Manifest rather than payload
|
||||
|
||||
For rehydratable artifact sets the backup stores `artifacts.json`: per set, the artifact set id,
|
||||
revision id, recovery class, upstream repository, exact commit SHA, file count, total bytes and
|
||||
every file with its filename, SHA-256, size, type, serialization format and security and licence
|
||||
status. A live backup recorded 24 artifact sets covering 8,100,376,532 rehydratable bytes in a
|
||||
39,276-byte manifest.
|
||||
|
||||
## Rehydration
|
||||
|
||||
Recovery reuses the existing acquisition plane rather than inventing a second download path:
|
||||
|
||||
```text
|
||||
ModelRevision + exact commit SHA + expected file list + expected SHA-256
|
||||
→ download plan against a disposable storage root
|
||||
→ Node Agent downloads into quarantine
|
||||
→ hash verification against provenance
|
||||
→ atomic promotion
|
||||
```
|
||||
|
||||
A floating revision is never used. `_revision_identity` resolves the repository and commit from the
|
||||
upstream snapshot recorded for that revision, falling back to the model's upstream source; if
|
||||
neither yields both, the set is classified `NON_REHYDRATABLE` and the recovery operation is created
|
||||
already `BLOCKED` with `ARTIFACT_NOT_REHYDRATABLE` rather than attempting a download.
|
||||
|
||||
## Recovery is not a security bypass
|
||||
|
||||
A rehydrated artifact re-enters the supply chain as new material. It is written to quarantine,
|
||||
hashed, checked against the security policy and promoted only on success, and it carries
|
||||
`static_checks_passed_unapproved` — not the approval its predecessor held. `trust_remote_code`
|
||||
stays `false`. A restore never launders an artifact into an approved state.
|
||||
|
||||
## Upstream unavailable
|
||||
|
||||
If the upstream cannot be reached the operation is recorded `BLOCKED` with
|
||||
`ARTIFACT_REHYDRATION_BLOCKED` and zero bytes recovered. Recovery does not report the platform as
|
||||
fully recovered while an artifact it depends on is still missing.
|
||||
|
||||
## Hash mismatch
|
||||
|
||||
`record_artifact_recovery` compares every verified file against the expected digest from
|
||||
provenance. A mismatch downgrades the operation to `FAILED` with `ARTIFACT_HASH_MISMATCH` even when
|
||||
the caller asserted success — the digest decides, not the reporter.
|
||||
|
||||
## Lineage
|
||||
|
||||
Each operation stores the artifact set id and variant, the upstream snapshot id, the model and
|
||||
revision ids, the upstream revision and resolved commit, and the derived-artifact links with their
|
||||
source digests. A rebuilt derived artifact keeps its path back to the exact source artifacts and
|
||||
transformation identity; recovery never resets lineage.
|
||||
|
||||
## Disposable locations
|
||||
|
||||
Rehearsals rehydrate into a disposable storage root, never over the only usable copy. Live evidence
|
||||
from disaster rehearsal C: `microsoft/trocr-small-printed` at commit
|
||||
`04e994ab854b0089d4929f48c2b4dbe2ce78a340`, 7 files and 247,200,667 bytes recovered into
|
||||
`m15-recovery-scratch` in 24.2 seconds, all seven digests matching provenance, while the 75
|
||||
production artifact locations remained untouched.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Atomic cutover
|
||||
|
||||
Cutover is a durable journal around an external operation, not a claimed distributed transaction:
|
||||
|
||||
```text
|
||||
PREPARING -> LOCKING -> SWITCHING -> VERIFYING -> COMMITTING -> COMMITTED
|
||||
`-> ROLLING_BACK
|
||||
```
|
||||
|
||||
Preparation performs a CAS/version check, exact observed-source check, approval recheck, current
|
||||
validation check and immutable rollback snapshot. The adapter then uses its safest atomic primitive
|
||||
(Qdrant alias update for ExampleRAG). ModelForge records `CUTOVER_COMMITTED` only after the adapter
|
||||
reports the expected external target plus bounded read-path/capability/smoke health. A stale alias is
|
||||
`STALE_SOURCE`; unhealthy post-switch truth enters rollback. Production execution additionally
|
||||
requires the matching approved M12 `requires_reindex` promotion plan.
|
||||
@@ -0,0 +1,131 @@
|
||||
# Backup and recovery
|
||||
|
||||
M15 makes recovery a first-class control-plane concern with versioned policy, immutable backup
|
||||
identity and measured outcomes. Nothing in this plane is allowed to look successful without
|
||||
evidence.
|
||||
|
||||
## RecoveryPolicy
|
||||
|
||||
`RecoveryPolicyRevision` is an immutable, fingerprinted revision per asset class carrying backup
|
||||
method, retention, minimum verified backups, RPO and RTO targets, restore verification depth,
|
||||
encryption requirement, external-dependency and rehydration flags, secret class and rationale.
|
||||
Superseding a policy creates revision *n+1* and deactivates its predecessor; recovery rules are
|
||||
never scattered inline.
|
||||
|
||||
Seven policies ship by default: `control-plane.database`, `artifacts.rehydratable`,
|
||||
`artifacts.non-rehydratable`, `runtime.ephemeral`, `external.projects`, `secrets.credentials` and
|
||||
`configuration.source-controlled`.
|
||||
|
||||
## BackupSet
|
||||
|
||||
A `BackupSet` is immutable recovery identity. Beyond the backup id and state it records the
|
||||
ModelForge version and commit, the canonical repository and reference, the Alembic revision, an
|
||||
environment fingerprint, the PostgreSQL server identity and backup-tool version, the destination
|
||||
root, payload and manifest paths with their SHA-256, the included and excluded asset classes, the
|
||||
payload size, encryption algorithm and key id, verification evidence, milestone and legal hold.
|
||||
|
||||
Identity fields are sealed by `immutable_at`. A backup is built across several flushes — planned,
|
||||
written, then stamped — so the guard permits exactly the sealing flush and rejects every later
|
||||
change to identity. Lifecycle fields (state, verification, expiry) stay writable.
|
||||
|
||||
```text
|
||||
PLANNED → CREATING → CREATED → VERIFYING → VERIFIED
|
||||
↘ ↘
|
||||
FAILED FAILED
|
||||
VERIFIED/CREATED/FAILED → EXPIRED → DELETED
|
||||
```
|
||||
|
||||
Only `VERIFIED` is restore eligible. A `CREATED` backup that has never been verified cannot become
|
||||
a restore plan, and neither can a `FAILED` one.
|
||||
|
||||
## Manifest
|
||||
|
||||
Every backup writes `manifest.json` listing each object with its logical asset type, relative path,
|
||||
size, SHA-256, source generation, schema version and dependency references. The manifest itself is
|
||||
hashed into the backup set, so tampering with an entry is caught before any payload is read. The
|
||||
same entries are journaled as `backup_manifest_entries` rows, and verification requires the file
|
||||
and the journal to agree.
|
||||
|
||||
The manifest contains no secret material. `configuration.json` names the secrets a recovery will
|
||||
need — operator API key, backup encryption key, Hugging Face token — and their recovery class,
|
||||
never their values.
|
||||
|
||||
## Verification
|
||||
|
||||
`verify_backup` is the gate between "a file exists" and "this can be restored":
|
||||
|
||||
1. the manifest file hash matches the recorded manifest identity, or `MANIFEST_HASH_MISMATCH`;
|
||||
2. the manifest objects match the journaled entries and include a control-plane database payload,
|
||||
or `MANIFEST_INCOMPLETE`;
|
||||
3. every object exists at its allowlisted path with the recorded size and hash, or `HASH_MISMATCH`
|
||||
/ `PAYLOAD_MISSING`;
|
||||
4. the payload decrypts under the configured key and matches its recorded plaintext hash, or
|
||||
`DECRYPTION_FAILED` / `ENCRYPTION_KEY_UNAVAILABLE`;
|
||||
5. `pg_restore --list` reads the archive structure.
|
||||
|
||||
Only then does the set become `VERIFIED`.
|
||||
|
||||
## Encryption
|
||||
|
||||
Backups of authoritative control-plane state are protected at rest with AES-256-GCM from
|
||||
`cryptography`, framed in bounded chunks so a multi-gigabyte dump never has to be held in memory.
|
||||
ModelForge designs no cryptography of its own; `BackupCipher` is an adapter boundary so a KMS or
|
||||
HSM provider can replace the local key without touching the recovery service.
|
||||
|
||||
A wrong key fails closed: decryption raises before anything is renamed into place, and both the
|
||||
partial file and the destination are removed, so no readable plaintext is ever left behind. The key
|
||||
lives only in deployment configuration and is never written into a backup, a manifest, a log or an
|
||||
audit event.
|
||||
|
||||
## Consistency under load
|
||||
|
||||
The dump is `pg_dump --format=custom --serializable-deferrable`, a supported online-consistent
|
||||
logical backup taken in one snapshot, so concurrent control-plane writes cannot tear it. Copying a
|
||||
live data directory is never done.
|
||||
|
||||
Honesty about the window matters more than a clean-looking number: a semantic fingerprint is taken
|
||||
immediately before and immediately after the dump, and the backup records which subject groups were
|
||||
stable across it and which were being written. A live rehearsal under continuous capacity and SLO
|
||||
writes reported 13 of 15 groups stable, with `identity`, `observability` and `serving` active.
|
||||
|
||||
## Retention
|
||||
|
||||
`apply_retention` expires aged backups but never leaves the platform without a recovery point. The
|
||||
newest *n* verified backups (`minimum_verified_backups`, default 2) are protected regardless of
|
||||
age, milestone and legal-hold backups are never expired, and retention refuses to run at all while
|
||||
a restore is in flight.
|
||||
|
||||
## Capacity
|
||||
|
||||
Backups check headroom before writing rather than discovering it by filling a disk. The estimate
|
||||
separates bytes that must be copied from bytes protected by manifest alone, and an insufficient
|
||||
destination fails the backup with `INSUFFICIENT_CAPACITY` instead of producing a truncated payload.
|
||||
|
||||
## Interruption
|
||||
|
||||
A process that dies mid-backup leaves a `CREATING` row. Startup reconciliation moves it to `FAILED`
|
||||
with `MANIFEST_INCOMPLETE`, so an interrupted backup can never be mistaken for a usable one. This
|
||||
is not theoretical: the first live M15 backup attempt failed mid-write and was reconciled exactly
|
||||
this way on the next start.
|
||||
|
||||
A backup also cannot contain a record of *itself* as complete — the dump is taken while its own
|
||||
`BackupSet` row is still `CREATING` — so a restored control plane consistently reports the backup
|
||||
it was restored from as not restore eligible.
|
||||
|
||||
See ADR-0042, `docs/architecture/POSTGRESQL_BACKUP_RESTORE.md` and
|
||||
`docs/architecture/DISASTER_RECOVERY.md`.
|
||||
|
||||
|
||||
## M16 recovery under fault
|
||||
|
||||
A backup taken while a mixed capability soak is running verifies normally: the dump is a single
|
||||
consistent snapshot, and the write-window report names which subject groups were being written
|
||||
rather than claiming a quiet system. A live run recorded 11 of 15 groups stable with `identity`,
|
||||
`scheduler_history`, `serving` and `traffic_history` active.
|
||||
|
||||
Restarting the control plane mid-backup leaves the set `FAILED` with `MANIFEST_INCOMPLETE` on the
|
||||
next start; it can never become restore eligible.
|
||||
|
||||
An unwritable or absent backup destination now fails closed with `DESTINATION_UNAVAILABLE` rather
|
||||
than escaping as an unhandled error. Before M16 it returned an untyped server error and left the
|
||||
set stuck in `CREATING`, so `BACKUP_FAILED` never fired.
|
||||
@@ -0,0 +1,14 @@
|
||||
# Canary and Traffic Shift
|
||||
|
||||
M12 canaries are bounded evidence producers, not automatic promotion mechanisms. A successful run
|
||||
ends in `READY_FOR_PROMOTION_REVIEW`; an operator must explicitly commit the plan.
|
||||
|
||||
Request-level canary is valid only for compatible stateless capability contracts and can be scoped
|
||||
to explicit clients/projects, request count and traffic percentage. Gates include error rate, p95,
|
||||
capability health, scheduler capacity, critical project failures, external pressure and worker
|
||||
health. Health, error, latency, critical regression or capacity failure triggers automatic rollback.
|
||||
|
||||
Embedding spaces are never percentage-mixed inside an index. A `requires_reindex` plan accepts only
|
||||
`SHADOW`, `DUAL_READ` or `EVALUATION_ROUTE` semantics against a separate index/space. The current
|
||||
ExampleRAG alias is outside all M12 rehearsal targets. ExampleVision Vision cannot start a production
|
||||
canary while owner-photo production validation is deferred; only an isolated LAB rehearsal is valid.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Operational Capability Contracts
|
||||
|
||||
## M6 project consumption
|
||||
|
||||
`rag.embedding@1` remains model-agnostic. Its native response includes the immutable
|
||||
`embedding_space_id`, dimension and normalization contract. Consumers validate all three. Workload
|
||||
priority is identity-owned scheduler policy and is not accepted from the invocation payload.
|
||||
|
||||
`rag.embedding@1` accepts one non-empty string or a batch of at most eight strings. Each string is
|
||||
bounded by the configured character limit and the whole request by the gateway payload limit. It
|
||||
returns finite, normalized 1024-dimensional vectors, an immutable embedding-space ID, request ID,
|
||||
usage and cold/queue/load/inference/total timing metadata.
|
||||
|
||||
The contract is backend-authoritative. Incompatible dimension, normalization, tokenization or API
|
||||
schema changes require a new capability version. An underlying deployment may change within v1
|
||||
only when the output contract remains compatible; embedding-space identity still changes and has
|
||||
`requires_reindex` migration semantics.
|
||||
|
||||
The operational SLO boundary is a bounded 16-request deployment queue, one measured concurrent
|
||||
inference, and explicit timeout/backpressure failures. Input retention is disabled by default.
|
||||
@@ -0,0 +1,24 @@
|
||||
# Capability Deployments
|
||||
|
||||
## M9 shared-GPU admission
|
||||
|
||||
OCR, visual embedding and ASR deployments use the existing lease/residency scheduler. Manifest resource classes explain expected pressure, but an exact measured `CapabilityResourceEnvelope`, external GPU observation and fixed reserve determine admission. Heavy work fails with `INSUFFICIENT_SCHEDULABLE_VRAM`; ModelForge never evicts unmanaged Ollama, Tdarr or Plex workloads.
|
||||
|
||||
A CapabilityDeployment is the immutable production binding between CapabilityContract,
|
||||
DeploymentCandidate, production approval, exact ArtifactSet, RuntimeProfile, ComputeNode,
|
||||
accelerator, EmbeddingSpace and measured ResourceEnvelope. Lifecycle states are stable, draining,
|
||||
deprecated and archived; LAB_READY remains a separate non-production candidate state.
|
||||
|
||||
Promotion rejects missing or stale runtime evidence, non-verified artifacts, non-production nodes,
|
||||
incomplete supply-chain review and mismatched deployment facts. The active M5 deployment is
|
||||
`420f11de-4906-4751-bfd4-923849fc7d9d`, pinned to candidate
|
||||
`c6824408-84d1-45bf-b431-a3893e57fd52` and worker image
|
||||
`sha256:e707953d1cc6fe3e66e4571ea1381f37e5a88bc71a31dac896b98bccf56dbac0`.
|
||||
|
||||
## M12 lifecycle overlay
|
||||
|
||||
Lifecycle subjects reference deployments rather than replacing them. A partial unique database
|
||||
index prevents two `production=true, status=stable` rows for one capability contract. Promotion
|
||||
execution additionally uses subject-version compare-and-swap and idempotency keys. Platform-stable
|
||||
does not grant a project production binding; that change requires its own project policy and fit
|
||||
evidence.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Capability evaluation types
|
||||
|
||||
M9 adds immutable `CapabilityEvaluationSuite` and `CapabilityEvaluationRun` records alongside the retrieval-specific M6/M7 framework. A suite binds one exact capability contract, evaluation type, dataset revision, fixture digests, ground truth, metric definitions, thresholds and critical-case flags. A run binds the exact deployment and environment fingerprint and stores case/aggregate evidence without payload content.
|
||||
|
||||
Metric families are deliberately separate:
|
||||
|
||||
| Type | Metrics |
|
||||
| --- | --- |
|
||||
| retrieval | Recall@5/10, MRR, nDCG@10 |
|
||||
| ocr | CER, WER, text/field/layout accuracy, latency, VRAM |
|
||||
| visual-retrieval | Recall@1/5/10, MRR, latency, VRAM |
|
||||
| asr | WER, real-time factor, latency, VRAM |
|
||||
| tts | latency, real-time factor, VRAM |
|
||||
| generation | latency, VRAM; later task-specific contracts |
|
||||
| llm | task success, structured-output validity, latency, VRAM |
|
||||
|
||||
The API rejects a metric from another family, mismatched capability/deployment pairs, incomplete case sets, duplicate suite evidence and non-finite aggregate values. There is no universal composite score.
|
||||
|
||||
`/api/v1/capability-advisor` deterministically returns `REQUIRES_MORE_EVIDENCE`, `LAB_READY`, `PROMOTION_ELIGIBLE`, or `BLOCKED` from deployment, run, case and threshold evidence. It never promotes automatically. Existing retrieval comparisons retain their `KEEP_CURRENT` semantics.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Capability Gateway
|
||||
|
||||
## M9 routes
|
||||
|
||||
The gateway exposes model-neutral `document.ocr@1`, `vision.embedding@1`, and `speech.transcription@1` routes. Each has modality-specific validation, scoped service-client authorization and transient content handling. The stable text embedding route remains exactly 1024-dimensional; opaque LAB experiment routes may return a different declared `EmbeddingSpace` dimension without weakening the stable contract.
|
||||
|
||||
## Background consumers
|
||||
|
||||
Service clients carry a fixed workload priority used on Gateway request records, GPU leases and
|
||||
serving jobs. ExampleRAG shadow work uses `background`; callers cannot supply or escalate this field.
|
||||
Inputs remain digest-only in durable records and transient in Redis.
|
||||
|
||||
The ModelForge Gateway is the only project-facing inference boundary. A client requests the
|
||||
versioned `rag.embedding@1` contract; it cannot provide an artifact path, runtime adapter or
|
||||
concrete upstream model. Resolution is capability → stable CapabilityDeployment → RuntimeProfile
|
||||
→ exact ArtifactSet → eligible ComputeNode.
|
||||
|
||||
The native endpoint is `POST /api/v1/capabilities/rag.embedding@1/invoke`. The supported
|
||||
OpenAI-compatible subset is `POST /v1/embeddings` with logical model alias `rag.embedding`.
|
||||
Authentication, scope/rate checks, bounded queueing, a database-backed GPU lease, residency and
|
||||
the outbound worker job all execute before a response is returned. Typed failures never fall back
|
||||
to cloud inference.
|
||||
|
||||
M10 places the shared cross-capability scheduler between authenticated resolution and the worker
|
||||
job. A caller still selects only a capability; node, accelerator, resident eviction and priority are
|
||||
server-authoritative. Capability health remains distinct from temporary scheduling readiness such
|
||||
as `READY_RESIDENT`, `READY_ON_DEMAND`, `WAITING_FOR_CAPACITY`, `POLICY_BLOCKED`,
|
||||
`NODE_UNAVAILABLE`, and `RUNTIME_UNHEALTHY`.
|
||||
|
||||
Request bodies live only in Redis with a short TTL while queued/executing. PostgreSQL retains a
|
||||
request ID, input hash, byte/count metadata, timings and failure class, never the input text.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Capability taxonomy
|
||||
|
||||
ModelForge is capability-first: consumers bind to a versioned contract, never to a model or runtime. M9 defines seven estate categories: `TEXT`, `RAG`, `DOCUMENT`, `VISION`, `AUDIO`, `GENERATION`, and `ASSISTANTS`.
|
||||
|
||||
Every manifest declares purpose, input/output modalities, stability, resource class, consumers, privacy class, evaluation type, and bounded payload limits. Runtime state is derived from registry relations and is not confused with declared stability.
|
||||
|
||||
| Capability | Category | Stability | Evaluation | Resource class |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `rag.embedding@1` | RAG | stable | retrieval | HEAVY |
|
||||
| `rag.reranking@1` | RAG | experimental | retrieval | MEDIUM |
|
||||
| `document.ocr@1` | DOCUMENT | experimental | ocr | HEAVY |
|
||||
| `vision.embedding@1` | VISION | experimental | visual-retrieval | HEAVY |
|
||||
| `speech.transcription@1` | AUDIO | experimental | asr | HEAVY |
|
||||
| assistant manifests | ASSISTANTS | planned | llm | EXCLUSIVE_GPU |
|
||||
|
||||
`stable`, `experimental`, `blocked`, and `planned` describe governance intent. Operational states such as no deployment, LAB-ready, resident, and blocked are evidence-derived and independently visible through `/api/v1/capability-estate`.
|
||||
|
||||
M10 adds scheduling readiness without changing taxonomy or promotion state. A healthy capability can
|
||||
be `WAITING_FOR_CAPACITY`, and a cold proven deployment can be `READY_ON_DEMAND`; neither condition
|
||||
means the capability is broken. Vision and ASR remain LAB promotion-eligible and are not promoted by
|
||||
co-residency evidence.
|
||||
|
||||
Project readiness is planning metadata, not an integration claim: ExampleRAG maps to embeddings, reranking and future OCR; ExampleVision to vision embeddings and future OCR/vision assistance; ExampleOps to future general assistance and OCR; Portfolio Assistant to embeddings and a future local assistant.
|
||||
|
||||
M11 adds operational identity without changing capability stability. ExampleVision is an active shadow
|
||||
consumer of vision embeddings and an active LAB consumer of speech transcription. Its OCR relation is
|
||||
still planned/blocked. `PROMOTION_ELIGIBLE`, `KEEP_LAB` and `REQUIRES_MORE_EVIDENCE` are evidence
|
||||
states, not new taxonomy values or automatic lifecycle transitions.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Capacity snapshots and trends
|
||||
|
||||
The M14 minute poll samples existing Node Agent/NVML, scheduler, residency, lease and storage truth.
|
||||
It does not sample raw NVML at high frequency. Each enabled node/accelerator snapshot stores:
|
||||
|
||||
- observed and received timestamps, freshness and availability;
|
||||
- GPU total, observed, external, ModelForge resident, leased, reserve and schedulable bytes;
|
||||
- scheduler pressure state;
|
||||
- total/available system RAM;
|
||||
- total/free storage.
|
||||
|
||||
External GPU use is `max(0, NVML observed - measured ModelForge residency)`. Schedulable VRAM uses
|
||||
the existing scheduler invariant and never grants authority to manage external processes.
|
||||
|
||||
## Aggregation and retention
|
||||
|
||||
Raw minute snapshots are retained for seven days. Older rows are aggregated into hourly buckets and
|
||||
retained for 90 days; cleanup is an application operation, not manual SQL. Each aggregate preserves
|
||||
sample count, schedulable min/max/average and storage-free minimum. Both tiers are bounded.
|
||||
|
||||
Trend responses calculate schedulable min/max/average/p50/p95, external min/max, storage min/max
|
||||
and GPU pressure frequency. Fewer than three snapshots produce `INSUFFICIENT_DATA`; a latest stale
|
||||
snapshot produces `STALE`.
|
||||
|
||||
Storage forecast requires at least 12 samples spanning at least one hour. Otherwise forecast status
|
||||
is `INSUFFICIENT_DATA`. With sufficient evidence it reports observed bytes/day and, only for a
|
||||
negative slope, a bounded days-to-current-zero estimate. It is a trend signal, not an automated
|
||||
cleanup decision.
|
||||
|
||||
Capability headroom compares a measured `CapabilityResourceEnvelope` with known persisted node
|
||||
capacity. A sustained mismatch becomes a warning; unknown/stale capacity is not treated as a
|
||||
confident fit and no unlimited-scaling claim is made.
|
||||
@@ -0,0 +1,16 @@
|
||||
# Cross-project observability
|
||||
|
||||
M11 makes real consumer use visible without exposing payload content. For each active project client,
|
||||
the Projects API/UI reports project, exact capability, environment, state, purpose, request volume,
|
||||
last use, terminal error count, p50/p95 latency, deployment identity, measured resource requirement
|
||||
and latest project-fit recommendation/evidence ID.
|
||||
|
||||
“Why Installed” now joins active, non-deprecated project bindings and immutable project-fit evidence
|
||||
to each installed deployment. Duplicate bindings are projected once. SigLIP2 and Whisper consequently
|
||||
show ExampleVision as an active shadow/LAB consumer and block destructive deletion through the existing
|
||||
deployment/project dependency checks.
|
||||
|
||||
Gateway telemetry remains bounded: correlation and request identity, capability, priority, payload
|
||||
shape, lifecycle state and timings are retained. Images, audio, document bytes, transcriptions and
|
||||
recognition results are not observability dimensions.
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
# Dependency-Aware Cleanup
|
||||
|
||||
Cleanup is plan-first and dry-run by default. A `LifecycleCleanupPlan` records the exact object,
|
||||
dependency graph, digest, reclaimable bytes, retention state, blockers and proposed action.
|
||||
|
||||
Artifact-set checks cover active locations, derived artifacts, candidates, profiles, execution and
|
||||
production approvals, capability deployments/routes, project bindings, evaluation runs, embedding
|
||||
spaces/migrations, lifecycle evidence and retention. Location checks distinguish one node-local copy
|
||||
from remaining copies. Database foreign keys use `RESTRICT`; there are no hidden destructive
|
||||
cascades.
|
||||
|
||||
Execution requires an operator credential and explicit confirmation, re-resolves the graph and
|
||||
compares its digest. Changed dependencies make the plan `STALE`; active dependencies or retention
|
||||
return typed 409 conflicts. ModelForge records a location removal only after the typed storage action
|
||||
confirms physical completion. Retrying after a crash reconciles the same plan; removal records retain
|
||||
prior status, digest, size, actor and time. Model, revision, hashes, evaluation, deployments and audit
|
||||
are never erased by binary-location cleanup.
|
||||
|
||||
M13 dependency discovery also reports `MIGRATION_PLAN_SOURCE` and `MIGRATION_PLAN_TARGET` when an
|
||||
artifact set appears in an active plan identity or a retained rollback snapshot. The execution-time
|
||||
digest recheck prevents cleanup racing a newly planned migration.
|
||||
|
||||
## Node lifecycle extension
|
||||
|
||||
Node decommission applies the same fail-closed principle to compute infrastructure. Its preview
|
||||
enumerates blockers, current truth to remove and provenance to retain; execute repeats that graph
|
||||
under transaction locks and has no force override. See `NODE_DECOMMISSION.md`.
|
||||
@@ -0,0 +1,115 @@
|
||||
# Disaster recovery
|
||||
|
||||
## Canonical restore flow
|
||||
|
||||
```text
|
||||
fresh host / control plane
|
||||
→ checkout the exact ModelForge commit recorded in the BackupSet
|
||||
→ provision PostgreSQL of a compatible major version
|
||||
→ verify the BackupSet (manifest hash, payload hashes, decryption, archive structure)
|
||||
→ restore the database into an empty destination
|
||||
→ run only the forward migrations this build knows
|
||||
→ supply secrets; rotate what cannot be restored
|
||||
→ start the control plane against the restored database
|
||||
→ reconcile: clear current truth, suppress restored alerts, surface in-flight operations
|
||||
→ register or re-enrol the node
|
||||
→ rehydrate missing artifacts from their exact revisions
|
||||
→ validate, compare fingerprints, measure RPO and RTO
|
||||
→ operator declares recovery complete
|
||||
```
|
||||
|
||||
## RestorePlan
|
||||
|
||||
`RestorePlan` is first class: backup set, mode, target environment and label, redacted database
|
||||
destination, artifact strategy, secret strategy, node strategy, expected ModelForge version,
|
||||
preflight evidence and validation requirements. Plans are idempotent — the same backup, mode,
|
||||
destination and label return the same plan rather than a second one.
|
||||
|
||||
```text
|
||||
VALIDATION isolated rehearsal; never a production environment
|
||||
REPLACEMENT an operator-declared maintenance target
|
||||
DISASTER_RECOVERY the only mode permitted to target production, and only when explicitly enabled
|
||||
```
|
||||
|
||||
## Preflight
|
||||
|
||||
Eleven checks run before any byte moves, and again at execution time:
|
||||
|
||||
```text
|
||||
backup_verified payload_hashes schema_compatibility
|
||||
encryption_key destination_isolation destination_reachable
|
||||
destination_empty postgres_compatibility disk_space
|
||||
source_control_revision external_dependencies
|
||||
```
|
||||
|
||||
A failing preflight moves the plan to `PREFLIGHT_FAILED` and `start` is refused with
|
||||
`restore_preflight_required`.
|
||||
|
||||
## RestoreOperation
|
||||
|
||||
The journal is durable and append-only. Each phase writes a `RestoreOperationEvent` with its
|
||||
evidence, and phase durations accumulate into the measured RTO.
|
||||
|
||||
```text
|
||||
PLANNED → PREFLIGHT → RESTORING_DATABASE → RESTORING_CONFIGURATION
|
||||
→ REHYDRATING_ARTIFACTS → RECONCILING → VALIDATING → READY
|
||||
↘
|
||||
FAILED | MANUAL_INTERVENTION_REQUIRED
|
||||
```
|
||||
|
||||
Retry resumes the journalled operation rather than creating a second one. A completed database
|
||||
phase is never replayed onto live data: the destination-empty guard turns a retry into
|
||||
`DESTINATION_NOT_EMPTY` instead of a merge. A restore interrupted mid-phase is moved to
|
||||
`MANUAL_INTERVENTION_REQUIRED` by startup reconciliation, never silently resumed.
|
||||
|
||||
## Reconciliation
|
||||
|
||||
Reconciliation runs exactly once per restore operation, keyed by a marker audit event carrying that
|
||||
operation's id. Matching on the action alone would be wrong: a backup legitimately contains the
|
||||
reconciliation events of *earlier* restores, and an unscoped check made every later restore believe
|
||||
it had already reconciled.
|
||||
|
||||
Reconciliation clears every current-truth table, suppresses restored firing alerts so a stale alert
|
||||
is not presented as present-day truth while its history is retained, and reports in-flight lifecycle
|
||||
operations and migration cutovers by id and stage with the typed outcome
|
||||
`RECOVERY_RECONCILIATION_REQUIRED`. It never guesses external state.
|
||||
|
||||
## In-flight operations
|
||||
|
||||
A backup taken while a lifecycle promotion or a migration cutover is mid-flight restores that
|
||||
operation intact, with its idempotency key, expected version and generation. On startup the
|
||||
lifecycle reconciler conservatively rolls the incomplete operation back from its immutable snapshot
|
||||
exactly once; a second start changes nothing. Migration cutovers are deliberately *not* auto-
|
||||
resolved, because external alias truth cannot be inferred after a crash — they stay in their
|
||||
intermediate stage and are reported as requiring reconciliation.
|
||||
|
||||
## RPO and RTO
|
||||
|
||||
Both are measured, never assumed.
|
||||
|
||||
RPO is the gap between the newest audit event in the restored database and the newest in the source
|
||||
at validation time; the restore point is the backup's completion timestamp. RTO is the sum of the
|
||||
measured phase durations.
|
||||
|
||||
## Rehearsal discipline
|
||||
|
||||
A rehearsal that reuses the live database volume is not a rehearsal. `docker-compose.dr.yml`
|
||||
provisions a separate PostgreSQL 17 with its own volume and a second control plane bound to it, and
|
||||
every rehearsal starts from a destroyed and recreated volume. The production database, ExampleRAG's
|
||||
Qdrant, ExampleVision's recognizer and external GPU workloads are never written to.
|
||||
|
||||
## Rehearsal catalogue
|
||||
|
||||
| | Scenario | Outcome |
|
||||
| --- | --- | --- |
|
||||
| A | Database loss | full restore into a fresh PostgreSQL, measured RTO/RPO |
|
||||
| B | Corrupt backup | payload and manifest tampering both fail closed |
|
||||
| C | Artifact loss | exact-revision rehydration into a disposable location |
|
||||
| D | Node credential loss | revoke, re-enrol, identity preserved |
|
||||
| E | Control-plane rebuild | isolated plane serving restored truth |
|
||||
| F | In-flight lifecycle | reconciled exactly once |
|
||||
| G | In-flight migration | surfaced, never auto-resolved |
|
||||
| H | Observability history | history restored, current state re-derived |
|
||||
|
||||
See `docs/quality/M15_VERIFICATION.md` for the measured results and
|
||||
`docs/operations/RUNBOOK_FULL_DR.md` for the operator procedure.
|
||||
@@ -0,0 +1,9 @@
|
||||
# document.ocr@1
|
||||
|
||||
`document.ocr@1` accepts one bounded PNG or JPEG page. The M9 baseline limit is one page, 16 MiB decoded content and at most 4096×4096 pixels in the runtime. PDFs must be split into page representations by a trusted local boundary before invocation.
|
||||
|
||||
The model-neutral response contains ordered pages, blocks, extracted text and optional confidence. Unsupported layout/table semantics are not fabricated. The TrOCR baseline specializes in printed line text and is therefore a technical LAB baseline, not a production document-understanding claim.
|
||||
|
||||
The preferred PaddleOCR-VL candidate is independently tracked. Custom repository/runtime code is a hard blocker under the default policy. The safe baseline uses an exact, verified TrOCR safetensors artifact through `transformers_trocr` with offline `AutoProcessor` and `VisionEncoderDecoderModel` loading.
|
||||
|
||||
OCR suites declare only meaningful ground truth and typed metrics: CER, WER, text/field/layout correctness where labels exist, latency and peak VRAM. Raw fixtures and extracted content are not persisted in the database.
|
||||
@@ -0,0 +1,168 @@
|
||||
# Domain Model
|
||||
|
||||
The implementation-ready field, cardinality, mutability, deletion and cross-plane contracts are
|
||||
normative in `M0_FOUNDATION_CONTRACTS.md`. The SQLAlchemy mapping and Alembic baseline are the
|
||||
executable persistence form of this model.
|
||||
|
||||
## Core entities
|
||||
|
||||
### Model
|
||||
|
||||
Logical upstream model family/reference. Does not imply a downloadable or runnable artifact.
|
||||
|
||||
Important fields:
|
||||
|
||||
- stable internal ID
|
||||
- display name
|
||||
- upstream provider/source
|
||||
- model family
|
||||
- modalities/tasks
|
||||
- parameter metadata
|
||||
- upstream license metadata
|
||||
|
||||
### ModelRevision
|
||||
|
||||
An immutable upstream revision, preferably resolved to a concrete commit SHA.
|
||||
|
||||
Fields include:
|
||||
|
||||
- model ID
|
||||
- upstream revision name supplied by discovery
|
||||
- resolved commit SHA
|
||||
- discovery timestamp
|
||||
- upstream metadata snapshot
|
||||
|
||||
### ModelArtifact
|
||||
|
||||
Concrete downloaded files associated with a revision.
|
||||
|
||||
- storage location
|
||||
- content digests
|
||||
- serialization format
|
||||
- size
|
||||
- security state
|
||||
- license state
|
||||
- provenance
|
||||
|
||||
### DerivedArtifact
|
||||
|
||||
A locally generated quantization/conversion/merge derived from one immutable source artifact.
|
||||
|
||||
Examples:
|
||||
|
||||
- AWQ 4-bit
|
||||
- GGUF Q4_K_M
|
||||
- GGUF Q5_K_M
|
||||
|
||||
Derived artifacts must maintain lineage to their source.
|
||||
|
||||
### RuntimeProfile
|
||||
|
||||
The reproducible execution configuration:
|
||||
|
||||
- runtime type and version/image digest
|
||||
- artifact variant
|
||||
- quantization interpretation
|
||||
- context policy
|
||||
- concurrency/batch settings
|
||||
- GPU-memory policy
|
||||
- launch arguments
|
||||
- environment constraints
|
||||
|
||||
### Deployment
|
||||
|
||||
Runnable combination of revision + artifact + runtime profile assigned to one or more capabilities/channels.
|
||||
|
||||
### Capability
|
||||
|
||||
Logical function requested by projects, such as `rag.embedding`, `vision.embedding` or `speech.transcription`.
|
||||
|
||||
### CapabilityContract
|
||||
|
||||
Versioned semantics for a capability:
|
||||
|
||||
- accepted input/output schema
|
||||
- quality metrics
|
||||
- required modalities/languages
|
||||
- SLOs
|
||||
- upgrade class
|
||||
- migration compatibility
|
||||
- fallback semantics
|
||||
|
||||
### Project
|
||||
|
||||
ModelForge consumer, e.g. ExampleRAG or ExampleVision.
|
||||
|
||||
### ProjectBinding
|
||||
|
||||
Maps a project to a capability contract and deployment channel, plus priorities, benchmark suites and migration support.
|
||||
|
||||
### ComputeNode
|
||||
|
||||
Machine capable of hosting one or more inference workers.
|
||||
|
||||
It has stable identity distinct from hostname, an observation source, agent protocol/version and
|
||||
capabilities, central liveness state, heartbeat/inventory/telemetry timestamps, role/labels and
|
||||
explicit production/lab/benchmark eligibility. Enrollment and node credentials are separate
|
||||
hash-only security records; they are not node identity.
|
||||
|
||||
### Accelerator
|
||||
|
||||
GPU/accelerator attached to a compute node.
|
||||
|
||||
### ResourceEnvelope
|
||||
|
||||
Measured runtime behavior for a deployment under a test profile:
|
||||
|
||||
- context length
|
||||
- concurrency
|
||||
- cold/warm load time
|
||||
- idle VRAM
|
||||
- peak VRAM
|
||||
- TTFT
|
||||
- throughput
|
||||
- p50/p95/p99 latency
|
||||
- power/temperature
|
||||
- OOM/failure behavior
|
||||
|
||||
### BenchmarkSuite
|
||||
|
||||
Versioned benchmark cases and scoring rules.
|
||||
|
||||
### BenchmarkRun
|
||||
|
||||
Immutable execution record tying a suite revision to a deployment and environment fingerprint.
|
||||
|
||||
### Experiment
|
||||
|
||||
Human-driven or automatic A/B comparison. May graduate individual cases into a benchmark suite.
|
||||
|
||||
### Recommendation
|
||||
|
||||
Advisor output with structured evidence, blockers, regressions, migration cost and confidence.
|
||||
|
||||
### Promotion
|
||||
|
||||
Audited action changing deployment channel state.
|
||||
|
||||
### Migration
|
||||
|
||||
Controlled process required when an upgrade changes persistent derived data or output schemas.
|
||||
|
||||
### AuditEvent
|
||||
|
||||
Append-only record of security-sensitive and operationally meaningful actions.
|
||||
|
||||
Audit events include actor, action, resource, outcome, correlation ID, UTC occurrence time, structured
|
||||
details and previous/current event hashes. Update and delete are rejected by the persistence layer.
|
||||
|
||||
## Key invariants
|
||||
|
||||
1. Approved model revisions and artifacts are immutable.
|
||||
2. Deployments reference immutable revision/artifact identities.
|
||||
3. A project binds to a capability contract, not a path or model repository name.
|
||||
4. Every stable deployment has local validation and benchmark evidence; M0 defines no production waiver.
|
||||
5. `requires_reindex` changes must create a migration; direct promotion is invalid.
|
||||
6. Deletion cannot proceed while a live binding, rollback target, migration, benchmark evidence, or lineage dependency requires the artifact.
|
||||
7. Storage URIs are mutable locators; revision SHA and artifact digests are immutable identities.
|
||||
8. Promotion and migration are distinct records and a promotion never implies completed data migration.
|
||||
@@ -0,0 +1,22 @@
|
||||
# Embedding migrations
|
||||
|
||||
An `EmbeddingMigration` names source/target spaces, source collection, distinct shadow collection,
|
||||
pinned corpus revision and background execution bounds. Supported lifecycle:
|
||||
|
||||
```text
|
||||
PLANNED -> PREFLIGHT -> BACKFILLING -> VALIDATING -> READY_FOR_EVALUATION
|
||||
-> FAILED / CANCELLED
|
||||
READY_FOR_EVALUATION -> EVALUATED -> PROMOTION_ELIGIBLE / NOT_ELIGIBLE
|
||||
```
|
||||
|
||||
Preflight must prove current health, authoritative corpus availability, scoped Gateway access,
|
||||
target space/dimension/distance, Qdrant reachability, storage and collection non-conflict before a
|
||||
vector write. ExampleRAG checkpoints after successful Qdrant batches only. Resume validates and skips
|
||||
matching point fingerprints; mismatches fail closed. Cancellation is observed between batches and
|
||||
retains the shadow for inspection/resume. Validation checks count, missing/duplicate/malformed and
|
||||
fingerprint evidence plus source-alias equality. No M6 endpoint performs alias cutover.
|
||||
|
||||
M13 retains this M6 project-owned backfill mechanism and adds the generic control-plane layer in
|
||||
`MIGRATION_ENGINE.md`: immutable lifecycle intent, richer durable checkpoints, versioned validation,
|
||||
shadow evidence, external-truth cutover/rollback and recovery. The old endpoint still cannot cut over;
|
||||
there is no bypass from an M6 migration record to production.
|
||||
@@ -0,0 +1,11 @@
|
||||
# Embedding model comparison
|
||||
|
||||
`ModelComparison` keeps raw project quality, latency, resource, critical-case and blocker evidence
|
||||
separate. Evaluated rows reference immutable EvaluationRuns; blocked rows retain unknown metrics.
|
||||
Comparability requires the same suite revision and retrieval configuration. No weighted score is
|
||||
invented and no comparison can approve or promote a deployment.
|
||||
|
||||
The first live ExampleRAG comparison covers the nomic baseline, Qwen3-Embedding-0.6B raw,
|
||||
Qwen3-Embedding-0.6B retrieval-aware, and blocked Qwen3-Embedding-4B and BGE-M3 candidates. See
|
||||
`docs/quality/M7_VERIFICATION.md` for values and candidate blockers.
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
# Embedding Space Identity
|
||||
|
||||
## Consumer enforcement
|
||||
|
||||
ExampleRAG stores `embedding_space_id` on the ModelForge profile, collection identity and every new
|
||||
shadow payload. The adapter rejects a vector response whose space differs even when dimensions are
|
||||
equal. The current production index is recorded as `legacy-observed` until complete upstream
|
||||
revision provenance exists.
|
||||
|
||||
An EmbeddingSpace identifies the vector geometry produced by an exact ArtifactSet,
|
||||
RuntimeProfile fingerprint and CapabilityContract. Dimension and normalization are facts, not the
|
||||
identity: two 1024-dimensional deployments can still be incompatible spaces.
|
||||
|
||||
The identity digest is immutable and the M5 production space is
|
||||
`326075d3-c75f-402e-b1b4-4d243c6a9ca2`. Every change is classified `requires_reindex`; existing
|
||||
vectors must never be silently reused under another identity.
|
||||
|
||||
M13 freezes the target `EmbeddingSpace` and source space reference in its immutable plan. A reindex
|
||||
plan must name a separate shadow target; changing deployment, runtime, preprocessing, dimension,
|
||||
normalization, query/document semantics or distance identity requires a new plan and validation.
|
||||
@@ -0,0 +1,25 @@
|
||||
# Error budgets
|
||||
|
||||
Error budgets apply only to `PRODUCTION` SLO policies.
|
||||
|
||||
For a window with population `N`, objective `O` and bad count `B`:
|
||||
|
||||
```text
|
||||
allowed_bad = (1 - O) × N
|
||||
consumed_bad = B
|
||||
remaining = max(0, allowed_bad - B)
|
||||
burn_rate = (B / N) / (1 - O)
|
||||
```
|
||||
|
||||
The long burn rate uses the complete rolling policy window. The short burn rate uses the last five
|
||||
minutes, bounded by the policy window. Latency bad events are completed requests over the exact
|
||||
policy threshold; success-ratio bad events are valid terminal failures.
|
||||
|
||||
An operational capability can be serving while consuming budget. The UI therefore shows SLO state,
|
||||
observed value and budget separately. `remaining=0` is not rewritten as a capability outage, and an
|
||||
acknowledged alert does not restore budget.
|
||||
|
||||
Windows roll naturally as old events leave the population. A new evaluation is an immutable
|
||||
snapshot, so recovery is demonstrated by later snapshots rather than rewriting history. LAB and
|
||||
BACKGROUND evaluations have `allowed_bad`, `consumed_bad`, `remaining` and burn rates unset by
|
||||
default.
|
||||
@@ -0,0 +1,13 @@
|
||||
# Evaluation framework
|
||||
|
||||
ModelForge persists generic `EvaluationSuite`, immutable `EvaluationSuiteRevision`, bounded
|
||||
`EvaluationCase`, `EvaluationRun`, per-case result and `EvaluationComparison` records. A typed
|
||||
project adapter executes retrieval; ModelForge never executes user-supplied code.
|
||||
|
||||
A revision digest covers cases, labels, critical flags, metrics, top-K, retrieval settings and
|
||||
thresholds. Editing any of these requires a new revision. Runs pin suite/dataset and corpus
|
||||
revisions, target index/space, retrieval-config digest and full local environment identity.
|
||||
|
||||
Comparability is `comparable` only when project, suite revision, corpus revision and retrieval
|
||||
configuration match. Eligibility remains distinct from promotion and is blocked by incompatible
|
||||
runs, excess errors, a configured Recall@10 regression or critical regressions.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Expanded Safe Model Discovery
|
||||
|
||||
M8 discovery uses the official Hugging Face provider for live metadata, file inventory and exact
|
||||
commit resolution. Popularity is a discovery signal only. Each shortlisted candidate records safe
|
||||
artifact availability, byte size, license capture, runtime expectations, storage/GPU fit and a
|
||||
typed outcome. Blocked candidates retain unknown quality rather than receiving a synthetic zero.
|
||||
|
||||
The bounded shortlist was:
|
||||
|
||||
| Candidate | Exact revision | Safe evidence | Outcome |
|
||||
| --- | --- | --- | --- |
|
||||
| Qwen3-Embedding-4B | `5cf2132abc99cad020ac570b19d031efec650f2b` | Safetensors, Apache-2.0 metadata, 8,059,503,129 bytes | `discovery_gpu_fit_blocked` |
|
||||
| BAAI/bge-m3 | `5617a9f61b028005a4858fdac845db406aefb181` | Required weights/heads include `.bin`/`.pt` pickle-risk files | `discovery_security_blocked` |
|
||||
| multilingual-e5-large-instruct | `274baa43b0e13e37fafa6428dbc7938e62e5c439` | 15 selected safe paths, 11 unique blobs, 1,164,135,840 bytes | fully evaluated |
|
||||
| multilingual-e5-base | `d128750597153bb5987e10b1c3493a34e5a4502a` | safe Safetensors variant, 1,156,508,130 bytes | preflight only; not selected |
|
||||
|
||||
Only E5-large-instruct was downloaded because it passed preflight and was the strongest justified
|
||||
bounded experiment. E5-base was not downloaded after that choice. Qwen 4B was not forced into VRAM;
|
||||
BGE-M3 policy was not weakened.
|
||||
@@ -0,0 +1,22 @@
|
||||
# GPU Pressure and Hysteresis
|
||||
|
||||
M10 derives memory pressure from schedulable headroom after physical use, unmaterialized leases and
|
||||
the dynamic reserve. Temperature and utilization are health signals, not memory-pressure aliases.
|
||||
|
||||
The states are `NORMAL`, `ELEVATED`, `HIGH`, and `CRITICAL`. Worsening pressure applies immediately.
|
||||
Recovery crosses a lower threshold and must remain stable for 30 seconds; a cooldown prevents an
|
||||
immediate new speculative load. This asymmetric state machine prevents external-use oscillation
|
||||
from repeatedly loading and unloading models.
|
||||
|
||||
- `NORMAL`: normal placement and residency policy.
|
||||
- `ELEVATED`: no new speculative heavy residency.
|
||||
- `HIGH`: background and LAB heavy loads queue or reject.
|
||||
- `CRITICAL`: no new admission; eligible idle ModelForge residency may drain.
|
||||
|
||||
External GPU memory is always unmanaged. ModelForge observes it, recalculates headroom and protects
|
||||
its own admission boundary. It exposes no PID, container, Docker or process-control operation and
|
||||
never kills, pauses, restarts or reconfigures external workloads.
|
||||
|
||||
Missing/stale telemetry or uncertain attribution becomes conservative placement evidence. The same
|
||||
pure hysteresis transition function is used in service decisions and deterministic oscillation,
|
||||
sustained-pressure, recovery and cooldown tests.
|
||||
@@ -0,0 +1,81 @@
|
||||
# Hardware Plane Architecture
|
||||
|
||||
## Boundary
|
||||
|
||||
M1 observes hardware through `SystemHostCollector` and `NvidiaNvmlCollector`; M1.5 runs those same
|
||||
typed collectors in an outbound compute-node agent and publishes observations to the central API.
|
||||
API routes never call NVML or SQLAlchemy directly; `HardwareInventoryService` reconciles typed
|
||||
observations through `HardwareRepository`. No shell parsing, Docker socket, privileged mode, arbitrary
|
||||
host mount, inbound agent service, scheduler or inference runtime is introduced. The central API host
|
||||
is not implicitly a compute node when startup refresh is disabled.
|
||||
|
||||
## Identity
|
||||
|
||||
A ComputeNode has an internal UUID and a stable identity key. Direct host execution uses a UUIDv5
|
||||
derived from the operating-system machine ID. Containers use a UUID stored on the `modelforge-state`
|
||||
volume, preventing image/container IDs or hostnames from becoming identity. An explicit configured ID
|
||||
has highest priority. Hostname and display name are mutable metadata, never identity.
|
||||
|
||||
NVIDIA accelerators reconcile by `(compute_node_id, GPU UUID)`. Device index, PCI bus ID and display
|
||||
name are observed facts and can change without creating a duplicate. A successful enumeration that no
|
||||
longer includes a known UUID marks it `missing`; an incomplete/unavailable NVML scan never does.
|
||||
|
||||
## Availability semantics
|
||||
|
||||
Every optional fact is `known`, `unknown`, `unsupported`, `unavailable` or `temporarily_failed`.
|
||||
Only `known` observations carry values. Zero is a valid measured value and is never substituted for a
|
||||
missing metric. One unsupported metric does not invalidate the accelerator.
|
||||
|
||||
## Inventory and telemetry
|
||||
|
||||
Inventory contains identity and relatively stable facts: OS/kernel/architecture, CPU topology, total
|
||||
RAM, GPU UUID/model/PCI/VRAM/compute capability, driver/CUDA-driver capability and MIG status.
|
||||
Latest-state telemetry contains available RAM, relevant storage use, GPU VRAM use, utilization,
|
||||
temperature, power, clocks, fan and performance state. Latest telemetry is upserted; M1 deliberately
|
||||
creates no unbounded history. A later observability adapter may export samples without schema rebuild.
|
||||
|
||||
## Reconciliation and audit
|
||||
|
||||
Refresh is process-serialized and idempotent. It creates or updates one node, upserts storage/current
|
||||
telemetry, rediscovers accelerators by UUID and preserves missing devices. Meaningful changes emit
|
||||
`NODE_DISCOVERED`, `ACCELERATOR_DISCOVERED`, `ACCELERATOR_MISSING` or `HARDWARE_CHANGED` audit events.
|
||||
Telemetry polling does not generate audit noise. Failed runs emit `INVENTORY_FAILED`, are persisted,
|
||||
and leave the last successful inventory intact.
|
||||
|
||||
## Fingerprint
|
||||
|
||||
The deterministic SHA-256 fingerprint includes node identity, OS/version/kernel/architecture and the
|
||||
sorted accelerator UUID/model/PCI/total VRAM/compute capability/driver/CUDA-driver facts. Driver or
|
||||
relevant hardware changes produce a new fingerprint. Available RAM, storage use, utilization,
|
||||
temperature, power, clocks and all other transient telemetry are excluded.
|
||||
|
||||
## Health
|
||||
|
||||
- no inventory: system process healthy, hardware inventory `pending`;
|
||||
- host observed with zero NVIDIA devices: node `active`, accelerator capability absent;
|
||||
- NVML unavailable with no prior GPU: node stays active, accelerator inventory is `degraded` with an
|
||||
explicit unavailable reason;
|
||||
- incomplete NVML scan: last good GPU state remains, inventory is degraded;
|
||||
- GPU absent from a successful enumeration: accelerator `missing` and audited;
|
||||
- unsupported temperature/power/fan: metric is unsupported, accelerator remains active;
|
||||
- host collection failure: run failed, last good inventory retained, refresh returns normalized 503.
|
||||
|
||||
## Real GPU validation
|
||||
|
||||
Run `POST /api/v1/hardware/refresh` twice, then compare `/nodes` and `/accelerators` identities and
|
||||
inspect typed metrics. The optional direct diagnostic is `nvidia-smi --query-gpu=...`; NVML remains
|
||||
the implementation source. For containers, start with the GPU overlay and verify the NVIDIA Container
|
||||
Toolkit exposes the intended device. Never infer RTX 4080 availability from the host description.
|
||||
|
||||
Remote reports retain agent observation time and central receive time. Persistent per-stream
|
||||
sequences reject replay/reordering, while liveness uses received heartbeats to tolerate clock skew.
|
||||
For enrollment, protocol and remote deployment details, see `NODE_AGENT_PROTOCOL.md` and
|
||||
`../operations/SERVER_AGENT_DEPLOYMENT.md`.
|
||||
|
||||
## Permanent decommission
|
||||
|
||||
A permanently retired node becomes a terminal tombstone; it is never deleted. Preview and execute
|
||||
are generation- and dependency-bound, active work blocks the transition, credentials are revoked,
|
||||
current scheduler/telemetry truth is cleared, and historical evidence remains. Inventory refresh
|
||||
and agent publication cannot revive the identity. See `NODE_DECOMMISSION.md` and
|
||||
`../operations/RUNBOOK_NODE_DECOMMISSION.md`.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Hugging Face Integration
|
||||
|
||||
## Boundary
|
||||
|
||||
M3 integrates the Hub through `HuggingFaceProvider`. Domain services do not call
|
||||
`huggingface_hub` directly and ModelForge exposes no arbitrary Hub proxy. The official `HfApi`
|
||||
client provides search and `model_info(..., files_metadata=True, securityStatus=True)` metadata.
|
||||
HTML scraping is not used.
|
||||
|
||||
The control plane retrieves metadata only. Artifact bytes are fetched by the authenticated,
|
||||
node-scoped artifact worker on the selected compute node. `MODELFORGE_HF_TOKEN` is optional secret
|
||||
configuration, is never persisted or returned by the API, and is passed only to the API and node
|
||||
agent—not to future runtimes.
|
||||
|
||||
## Facts and interpretation
|
||||
|
||||
Search results and snapshots keep reported repository ID, author, task, library, tags, popularity,
|
||||
access state, card data, scanner data, file tree and source timestamps as upstream facts. Local
|
||||
candidate matching, approval, hardware compatibility and intended capability remain ModelForge
|
||||
interpretation. Missing values stay absent/unknown.
|
||||
|
||||
Scanner results are stored as upstream evidence with `evidence_only=true`; they are never converted
|
||||
into a ModelForge approval. License declarations are captured as unreviewed evidence and no legal
|
||||
conclusion is inferred.
|
||||
|
||||
## Revision and access semantics
|
||||
|
||||
A requested ref such as `main` is resolved to the exact Hub commit SHA before a `ModelRevision` or
|
||||
artifact set is created. A later movement of `main` creates a new immutable revision. Existing
|
||||
revisions and approved plans retain their original SHA.
|
||||
|
||||
Access is classified as `public`, `gated`, `private`, inaccessible/not-found, or provider
|
||||
unavailable. Metadata errors are normalized without exposing credentials. Planning a gated/private
|
||||
snapshot without configured credentials fails closed as access-required behavior.
|
||||
|
||||
## API
|
||||
|
||||
- `POST /api/v1/discovery/search`
|
||||
- `POST /api/v1/models/{id}/refresh-upstream`
|
||||
- `GET /api/v1/models/{id}/upstream`
|
||||
- `GET /api/v1/revisions/{id}/artifact-sets`
|
||||
|
||||
The adapter is covered by offline fakes/mocks; the M3 acceptance also refreshed all 15 real
|
||||
candidates against the live Hub.
|
||||
@@ -0,0 +1,13 @@
|
||||
# Latency profiling
|
||||
|
||||
ModelForge records bounded, content-free timing evidence for capability invocations. Local spans use
|
||||
monotonic clocks; cross-process totals are correlated by request ID and are not calculated by
|
||||
subtracting wall-clock timestamps.
|
||||
|
||||
The M7 live profile showed that one-second worker polling dominated the M6 path. Bounded server-side
|
||||
long polling (maximum five seconds), a one-second worker wait and a short retry path reduced warm
|
||||
Gateway p50 from 1144.46 ms to 194.66 ms across 20 requests. Median contributors after the change
|
||||
were dispatch 49.18 ms, response transport 47.62 ms, inference 38.14 ms, scheduler 30.13 ms, payload
|
||||
persistence 9.43 ms, resolution 5.05 ms and result retrieval 3.80 ms. Correctness and security gates
|
||||
remain in the request path. Detailed evidence is in `docs/quality/M7_VERIFICATION.md`.
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
# Lifecycle Audit
|
||||
|
||||
Lifecycle audit is append-only and change-oriented. Events include approval request/decision/stale,
|
||||
plan creation/approval/start, canary start/observation/abort/review, commit, rollback completion or
|
||||
failure, deprecation/archive, retention and cleanup planning/execution.
|
||||
|
||||
Each event stores:
|
||||
|
||||
- object type/reference and from/to state;
|
||||
- requester, approver, executor or system actor role;
|
||||
- policy revision and exact evidence references;
|
||||
- human reason, timestamp and correlation/change ID;
|
||||
- bounded operational metadata such as counts, latency, trigger and fingerprints.
|
||||
|
||||
It explicitly excludes model inputs, OCR/ASR/Vision content, project payloads, bearer credentials,
|
||||
tokens and sensitive host paths. SQLAlchemy prevents update/delete of lifecycle events, while normal
|
||||
database foreign-key restrictions preserve their referenced provenance.
|
||||
|
||||
## Audit-chain integrity
|
||||
|
||||
`AuditEvent` has an explicit hash format. Legacy `v1` rows retain their exact historical canonical
|
||||
JSON encoding: correlation id, actor type/id, action, resource type/id, outcome, bounded details and
|
||||
previous-event hash. The `20260830_0024` migration validates that complete chain before changing any
|
||||
schema, marks those rows `v1`, and records the first permitted `v2` sequence. It never rewrites an
|
||||
approved event or silently repairs an invalid hash. A versioned legacy-prefix seal commits every
|
||||
legacy sequence, UUID, normalized UTC timestamp and event hash, making the two fields that were not
|
||||
inside the old hash tamper-evident after cutover.
|
||||
|
||||
New `v2` events are assigned their UUID and UTC/microsecond timestamp before the append call. In
|
||||
production, the database-owned canonical function builds the exact payload text, calculates its
|
||||
SHA-256, inserts that text and advances the checkpoint atomically. SQLite tests use the same stored
|
||||
payload contract in Python. Both identity values and the `v2` format identifier join the payload.
|
||||
Sequence must be exactly `1..n`; the first previous hash is null and every later link is
|
||||
the immediately preceding canonical event hash. The executable verifier recomputes the v1/v2 hash
|
||||
appropriate to each sequence and the legacy-prefix seal. It also requires every event UUID and
|
||||
sequence to be unique, independent of whatever constraints a restored schema currently exposes.
|
||||
|
||||
The durable `audit_chain_heads` singleton records event count, final sequence/hash, current format,
|
||||
v2 cutover and legacy-prefix count/seal. It therefore detects middle gaps as well as deletion of the
|
||||
tail or the entire event table. SQLAlchemy ORM, Core, aliased/annotated targets, legacy bulk and
|
||||
textual DML for both audit tables fail closed at the ordinary application `Session` boundary. That
|
||||
includes raw `exec_driver_sql` on a `Session`-owned connection; its lexer ignores comments and
|
||||
string/dollar literals before classifying schema-qualified quoted targets. The sole persistence
|
||||
operation accepts one semantic event, constructs the exact insert and head compare-and-set itself,
|
||||
and exposes no reusable token, execution option, session marker or connection marker. The guard
|
||||
admits only those exact active statement objects and operation types.
|
||||
|
||||
The SQLAlchemy layer also registers every Engine built by the application, guards direct
|
||||
`engine.begin()`/`session.bind.begin()` connections, and rolls back protected DML introduced by a
|
||||
later statement-rewrite listener. It defaults raw DDL and procedural calls that could hide audit
|
||||
mutation to deny. This remains defense in depth: it is not a claim that hostile same-process Python
|
||||
cannot remove a listener or open a new DBAPI connection.
|
||||
|
||||
The production authority is PostgreSQL. `modelforge_runtime` is a NOINHERIT non-owner with SELECT
|
||||
on both audit tables and EXECUTE only on
|
||||
`modelforge_audit.append_event_v2(...)`. Direct INSERT/UPDATE/DELETE/TRUNCATE, trigger/DDL creation,
|
||||
all SET ROLE-capable memberships and every other non-system function are unavailable. The
|
||||
canonical function is owned by the non-superuser `modelforge` migration role, fixes
|
||||
`search_path=pg_catalog`, takes the shared advisory lock and checkpoint row lock, validates the
|
||||
checkpoint and retained tail, constructs/hashes exact bytes, inserts, and advances the head by CAS.
|
||||
Owner-check triggers protect row mutation and TRUNCATE as a second database layer. Production
|
||||
startup reads catalog truth and refuses an owner/superuser runtime, role membership, forbidden
|
||||
grants, a changed function-body digest, wrong owner/SECURITY/search-path attributes, PUBLIC
|
||||
execution, a non-origin or wrong-event trigger, schema/database CREATE, or any unexpected
|
||||
executable non-system function. The API container receives no
|
||||
migration or bootstrap credential. A database owner/superuser remains a privileged administrative
|
||||
boundary; corruption tests use an independently created admin Engine and strict gates detect it.
|
||||
|
||||
Migration verifies and seals the complete immutable legacy prefix. Recovery and the executable
|
||||
`audit_chain_intact` invariant remain strict full-chain gates. A normal append does not repeat that
|
||||
linear proof: under the writer lock it validates the checkpoint shape, the two-row retained tail,
|
||||
the tail payload/hash/link and the checkpoint count, then inserts one event and advances the head by
|
||||
compare-and-set. That path is O(1) in retained history instead of cumulative O(n²). A database
|
||||
superuser could still edit a non-tail middle row without making the next append fail; this is not
|
||||
claimed otherwise. The strict invariant and every recovery READY transition detect that privileged
|
||||
tamper. PostgreSQL uses a shared transaction-scoped advisory lock; SQLite test databases use an
|
||||
engine-scoped process lock held until commit or rollback.
|
||||
|
||||
Restore reconciliation is the only writer aimed at another database. It reads audit rows in bounded
|
||||
ordered pages and strictly verifies events plus checkpoint before consulting an `ALREADY_APPLIED`
|
||||
marker, immediately before appending a marker, and independently before a restore can become
|
||||
`READY`. The marker calls the same SECURITY DEFINER canonical function, which advances event plus
|
||||
head atomically under the same advisory lock. Semantic-fingerprint allowances cannot
|
||||
turn an audit-chain verification failure into a passing restore.
|
||||
|
||||
A production-shaped legacy database containing the pre-RC direct recovery marker with a random
|
||||
`event_hash` is intentionally a migration blocker: 0024 fails before adding its columns or checkpoint.
|
||||
RC-QUAL-05 must discover and surface that state during rehearsal. The migration does not bless,
|
||||
rewrite or discard it. Downgrade to 0022 is allowed only before the first v2 event; afterwards it
|
||||
would require rewriting immutable history and is refused.
|
||||
|
||||
On PostgreSQL, migration 0024 takes the runtime advisory key and an `ACCESS EXCLUSIVE` lock on
|
||||
`audit_events` before validation. The table lock also serializes older 0022 writers that do not know
|
||||
the advisory key, and remains held through format marking and checkpoint seed. SQLite uses a
|
||||
test-only no-op write to acquire its writer transaction before validation. A live PostgreSQL
|
||||
concurrent-upgrade exercise remains a managed-runner gate; the unit contract asserts both locks.
|
||||
|
||||
`AuditContext` is the integration seam for a verified principal and request correlation id. Legacy
|
||||
services retain their current actor labels until the operator authentication boundary supplies that
|
||||
context; the hash-chain implementation does not fabricate principal identity. Consequently this
|
||||
integrity change alone does **not** claim that authenticated principal/correlation integration is
|
||||
complete.
|
||||
|
||||
`MigrationEvent` follows the same append-only/content-free rule and adds plan generation plus exact
|
||||
source/target identity. Per-batch audit is bounded to one summary event per completed configured
|
||||
batch; errors are capped and vectors, bodies, queries and secrets are excluded.
|
||||
|
||||
## Node lifecycle extension
|
||||
|
||||
Permanent compute-node removal writes one `NodeDecommissionOperation` and one
|
||||
`NODE_DECOMMISSIONED` event in the same transaction as the tombstone and credential revocation.
|
||||
The event retains operator, reason, timestamp, previous state, cleanup counts and credential result.
|
||||
See `NODE_DECOMMISSION.md`.
|
||||
@@ -0,0 +1,57 @@
|
||||
# Lifecycle State Model
|
||||
|
||||
M12 adds a lifecycle control layer without collapsing the existing domain identities. `Model`,
|
||||
`ModelRevision`, `ModelArtifact`, `ArtifactSet`, `RuntimeProfile`, `RuntimeProbe`,
|
||||
`DeploymentCandidate`, `CapabilityDeployment` and `ProjectBinding` remain separate records.
|
||||
|
||||
## Independent dimensions
|
||||
|
||||
Readiness is expressed as independent evidence, never inferred from one status:
|
||||
|
||||
| Dimension | Values used by lifecycle evidence |
|
||||
| --- | --- |
|
||||
| Integrity | `UNKNOWN`, `VERIFIED`, `CORRUPT` |
|
||||
| Security | `UNREVIEWED`, `APPROVED`, `BLOCKED` |
|
||||
| Runtime | `UNPROBED`, `PROVEN`, `INCOMPATIBLE` |
|
||||
| Evaluation | `NOT_EVALUATED`, `EVALUATED`, `REGRESSED`, `PROMOTION_ELIGIBLE` |
|
||||
| Project fit | `UNKNOWN`, `REQUIRES_MORE_EVIDENCE`, `ELIGIBLE`, `BLOCKED`, `DEFERRED_EXTERNAL_VALIDATION`, `KEEP_LAB` |
|
||||
| Deployment | `CANDIDATE`, `LAB_READY`, `PROMOTION_ELIGIBLE`, `CANARY`, `LAB_STABLE`, `STABLE`, `DRAINING`, `DEPRECATED`, `ARCHIVED` |
|
||||
|
||||
`LAB_READY`, engineering `PASS`, Advisor `PROMOTION_ELIGIBLE` and an approved request are not
|
||||
synonyms for production. Production is an explicit environment on a lifecycle subject and requires
|
||||
an evidence-bound approval, immutable plan, rollback snapshot and separate execution.
|
||||
|
||||
## Transition graph and concurrency
|
||||
|
||||
The typed graph in `domain/lifecycle_contracts.py` rejects shortcuts. Production follows
|
||||
`STABLE → DRAINING → DEPRECATED → ARCHIVED`; restoration from deprecated requires a new explicit
|
||||
promotion path, not mutation. LAB supports bounded canary and a separate `LAB_STABLE` state.
|
||||
|
||||
Every subject has a monotonically increasing `version`. Execution compares the operator's expected
|
||||
version before journaling a change. The database also permits only one production/stable deployment
|
||||
per capability contract. Conflicts return typed 409 responses; retries use a unique idempotency key.
|
||||
|
||||
Every transition creates an append-only `LifecycleEvent` containing object, before/after state,
|
||||
actor and role, policy revision, evidence IDs, reason, time and change ID. No inference payload or
|
||||
secret is stored.
|
||||
|
||||
M13 adds an independent migration state graph. It does not collapse backfill or external cutover
|
||||
truth into deployment lifecycle state; production cutover remains subordinate to a current M12
|
||||
approval and matching approved `requires_reindex` promotion plan.
|
||||
## M14 operational integration
|
||||
|
||||
Lifecycle failures and rollback failures feed bounded operational metrics/alerts from the existing
|
||||
operation journal. Alert acknowledgement is not lifecycle approval, promotion authority or rollback
|
||||
authority. M14 never executes lifecycle remediation; all production changes still require the M12
|
||||
evidence-bound policy, immutable plan and explicit actor separation.
|
||||
|
||||
|
||||
## M16 reconciliation under fault
|
||||
|
||||
An incomplete lifecycle operation is rolled back conservatively from its immutable snapshot on the
|
||||
next control-plane start, exactly once. Restart storms do not accumulate: four consecutive restarts
|
||||
under load produced no authoritative drift.
|
||||
|
||||
The `lifecycle_commit_has_evidence` and `no_hidden_auto_promotion` invariants assert that no
|
||||
operation reaches `COMMITTED` without its plan, approver and executor, and that no production
|
||||
deployment exists without a recorded production approval.
|
||||
@@ -0,0 +1,101 @@
|
||||
# M0 Foundation Contracts
|
||||
|
||||
This document is the implementation-level boundary for M0. The executable forms live in
|
||||
`domain/contracts.py`, `domain/lifecycle.py`, `persistence/models.py`, the Alembic baseline,
|
||||
and validated manifests under `config/`.
|
||||
|
||||
## Product and trust boundaries
|
||||
|
||||
| Plane | Owns | Must not own |
|
||||
|---|---|---|
|
||||
| Control plane | registry, policy, provenance, lifecycle decisions, audit, evidence | inference execution or arbitrary hub code |
|
||||
| Gateway | project authentication, capability/channel resolution, QoS, normalized errors | direct artifact selection by projects |
|
||||
| Runtime plane | adapter-governed load/probe/infer/unload against approved read-only artifacts | public endpoints, project/HF secrets, Docker socket, default egress |
|
||||
| Hardware plane | node/accelerator identity, inventory and measured telemetry | lifecycle or routing decisions |
|
||||
| Hugging Face boundary | discovery metadata and quarantined acquisition by exact revision | approval, deployment or trust decisions |
|
||||
| Project boundary | declares versioned capabilities, SLO, priority, fallback and migration support | model repository names, paths or runtime endpoints |
|
||||
| Runtime-engine boundary | implements the internal adapter contract | leaking engine-specific behavior into capability contracts |
|
||||
|
||||
ModelForge is not a cloud-AI broker, Kubernetes platform, training cluster, generic chatbot,
|
||||
or public multi-user inference service. There is no implicit cloud fallback.
|
||||
|
||||
## Identity, cardinality and lifecycle
|
||||
|
||||
- IDs are UUIDs in persistence; manifest keys are stable human-readable identities.
|
||||
- `Model 1—N ModelRevision 1—N ModelArtifact 1—N DerivedArtifact` preserves source lineage.
|
||||
- A `Deployment` references exactly one revision, exactly one source-or-derived artifact and an
|
||||
immutable runtime profile. Composite foreign keys prevent the artifact revision from diverging.
|
||||
- `Capability 1—N CapabilityContract`; project bindings reference a contract version and channel,
|
||||
never a model.
|
||||
- `ComputeNode 1—N Accelerator`; leases and envelopes remain node-ready while M1/M6 optimize for one GPU.
|
||||
- Benchmark runs, promotions, migrations and lineage references use restrictive deletion. Lifecycle
|
||||
removal is deprecation followed by dependency-checked archival; it is never cascade cleanup.
|
||||
- Approved revisions, verified artifacts, runtime profiles, benchmark runs, promotions and audit events
|
||||
are immutable records. Mutable labels and operational status are separate fields.
|
||||
- All timestamps are timezone-aware UTC database timestamps.
|
||||
|
||||
## Capability and project contract
|
||||
|
||||
Capability contracts version JSON input/output schemas, modalities, language support, streaming,
|
||||
structured output, vector dimensionality/normalization semantics, SLOs, quality metrics, upgrade
|
||||
class, fallback, privacy and resource requirements. Vector-producing contracts with unproven
|
||||
cross-deployment compatibility are rejected unless classified `requires_reindex`.
|
||||
|
||||
Project manifests select a capability contract version plus stable/candidate/experimental channel,
|
||||
priority, optionality, explicit fallback, migration support, project SLO and benchmark suites.
|
||||
The strict schemas reject unknown fields, including attempts to add a concrete model ID.
|
||||
|
||||
## Artifact proof
|
||||
|
||||
The reproducibility chain is:
|
||||
|
||||
`upstream repository + resolved commit SHA → filename/type + SHA-256 → optional source digest +
|
||||
derivation tool/version/arguments → runtime-profile fingerprint → deployment → benchmark run`.
|
||||
|
||||
Storage locations are locators, not identities. The digest and immutable lineage provide identity.
|
||||
A derived artifact is invalid without its source digest and conversion tool/version.
|
||||
|
||||
## Runtime adapter contract
|
||||
|
||||
An adapter validates a runtime profile, probes layered health, loads only under a scheduler lease and
|
||||
unloads by deployment identity. Profiles capture runtime type/version, image digest, artifact digest,
|
||||
quantization, context, concurrency, launch arguments, environment constraints and security toggles.
|
||||
The control plane depends on this protocol rather than vLLM, Transformers, Diffusers or llama.cpp.
|
||||
Concrete adapters belong to M5.
|
||||
|
||||
## Scheduler contract
|
||||
|
||||
Lease requests carry deployment, priority, residency (`always_warm`, TTL-based `keep_warm`,
|
||||
`load_on_demand`, `exclusive`, `lab_only`), measured resource envelope, warm TTL, exclusivity and
|
||||
explicit CPU-fallback permission. Envelopes are scoped by accelerator kind, context, concurrency and
|
||||
batch size and include idle/peak VRAM plus safety margin. A lease moves through requested/acquired/
|
||||
released state. Failure must release reservations and emit a structured event. M6 implements policy;
|
||||
M0 does not pretend to schedule work.
|
||||
|
||||
## Benchmark comparability
|
||||
|
||||
A run binds deployment, revision SHA, artifact digest, runtime/version/image, launch arguments,
|
||||
suite and dataset revisions, CUDA, driver, accelerator identity, context, concurrency, seed,
|
||||
timestamps and a deterministic environment digest. Runtime, driver/CUDA, accelerator or launch
|
||||
profile changes make cross-environment comparisons non-comparable. Historical evidence is retained
|
||||
with a stale reason; it is never rewritten.
|
||||
|
||||
## Promotion and migration
|
||||
|
||||
Promotion is a human-approved routing/lifecycle action backed by verified artifact state, local and
|
||||
project benchmark IDs and a rollback deployment. Migration is separate. For embeddings, promotion is
|
||||
blocked until a migration is `ready` after: current index → distinct shadow index → backfill →
|
||||
validation → shadow queries/benchmarks → atomic cutover. The prior index and deployment remain until
|
||||
the recorded rollback-retention deadline.
|
||||
|
||||
## Failure ownership and health
|
||||
|
||||
The normalized codes are `MODEL_LOAD_FAILED`, `GPU_OOM`, `RUNTIME_CRASH`, `TIMEOUT`,
|
||||
`INVALID_OUTPUT`, `HEALTHCHECK_FAILED`, `ARTIFACT_CORRUPT`, `DRIVER_ERROR` and
|
||||
`CAPABILITY_UNAVAILABLE`. Runtime adapters own load/crash/output faults; the scheduler owns capacity,
|
||||
OOM reservation and lease recovery; the gateway owns timeout and approved fallback/degrade behavior;
|
||||
the control plane owns artifact corruption, deployment blocking and audit; an operator owns explicit
|
||||
security exceptions and hard production promotion decisions. Retries must be bounded and observable.
|
||||
|
||||
Health is reported separately for process, runtime, model smoke inference, capability contract/routing,
|
||||
and required project capabilities. Process liveness alone cannot make a deployment healthy.
|
||||
@@ -0,0 +1,21 @@
|
||||
# M7 Model Comparison and Advisor Evidence
|
||||
|
||||
M7 adds candidate evaluation without changing the production capability binding.
|
||||
|
||||
`CapabilityExperimentRoute` is an opaque, authenticated lab route to one non-production
|
||||
`CapabilityDeployment`. It preserves the `rag.embedding@1` contract, applies RuntimeProfile-owned
|
||||
query/document semantics, and writes a distinct EmbeddingSpace. Input semantics are part of the
|
||||
space identity: changing a query instruction therefore requires a new space and full reindex.
|
||||
|
||||
`ModelComparison` references a project, capability contract, immutable suite revision and current
|
||||
run. Evaluated candidates reference their EvaluationRun; blocked candidates preserve their exact
|
||||
blocker and unknown metrics. `AdvisorRecommendation` contains structured deltas, critical
|
||||
regressions, resources, migration, security, policy and evidence confidence. Recommendations are
|
||||
advisory and dismissible but never executable.
|
||||
|
||||
The ExampleRAG M7 comparison uses dense embeddings only, unchanged sparse encoding/RRF, unchanged
|
||||
filters, candidate limit 40, top 10 and no reranker. This isolates the embedding variable.
|
||||
|
||||
The latency trace API is admin-only and returns at most 500 summary records. Input content and input
|
||||
digests are not exposed. Gateway database retention remains bounded by operator retrieval limits;
|
||||
M7 does not introduce an unbounded span store.
|
||||
@@ -0,0 +1,26 @@
|
||||
# Migration engine
|
||||
|
||||
M13 promotes stateful capability change to a first-class, immutable `MigrationPlan`. A plan binds an
|
||||
exact project binding, capability contract, source and target identities, adapter fingerprint,
|
||||
validation-policy revision, M12 approval, rollback target, execution bounds and environment
|
||||
fingerprint. Approval facts are rechecked immediately before cutover.
|
||||
|
||||
The guarded state graph is implemented in `migration_contracts.py`. Every transition adds a bounded,
|
||||
append-only `MigrationEvent` containing identities, actor, policy, generation, reason and evidence
|
||||
references. Mutable progress uses database compare-and-swap on `version`; adapter results also carry a
|
||||
64-bit generation. Production permits at most one non-terminal migration per project/capability.
|
||||
|
||||
Core never runs migration code from payloads. An adapter contract is a fingerprinted set of known
|
||||
operations. M13's first concrete operational contract is `examplerag.qdrant-reindex`; ExampleRAG owns
|
||||
enumeration, deterministic chunk identities, Gateway calls, Qdrant writes and alias inspection. Core
|
||||
does not assume Qdrant and stores no vectors, document bodies, queries or secrets.
|
||||
|
||||
The earlier M6 `EmbeddingMigration` remains evaluation/backfill evidence. M13 layers lifecycle,
|
||||
cutover, rollback and reconciliation above that proven project-owned mechanism instead of replacing
|
||||
or weakening it.
|
||||
## M14 operational integration
|
||||
|
||||
Migration plan states, validation/manual-intervention failures and rollback failures are observed
|
||||
from the M13 journal. `migration.production.reliability` evaluates terminal correctness over seven
|
||||
days and `MIGRATION_FAILURE`/`ROLLBACK_FAILURE` preserve an explicit lifecycle. Monitoring never
|
||||
changes a plan generation, resumes backfill or mutates an external alias.
|
||||
@@ -0,0 +1,24 @@
|
||||
# Migration recovery
|
||||
|
||||
Batch checkpoints and generations make preflight/backfill/validation/shadow restart-safe. Matching
|
||||
complete batches are skipped; incomplete attempts resume from the durable cursor. Old workers cannot
|
||||
write into a new generation.
|
||||
|
||||
Cutover stages are persisted before external work. Startup reports non-terminal cutover journals but
|
||||
does not guess external truth. A registered adapter/operator inspection submits the observed target,
|
||||
fingerprint and health to reconciliation. Exact healthy target commits, exact healthy source records
|
||||
rollback, and ambiguous truth becomes `MANUAL_INTERVENTION_REQUIRED`. Orphan shadows are retained
|
||||
and surfaced for later explicit cleanup; never auto-deleted.
|
||||
|
||||
|
||||
## M16 reconciliation semantics
|
||||
|
||||
Migration is the one subsystem that deliberately does not resolve itself after an interruption.
|
||||
External alias truth cannot be inferred from a crash, so an interrupted cutover stays in its
|
||||
intermediate stage, is reported as requiring reconciliation, and waits for an operator working
|
||||
through the typed adapter.
|
||||
|
||||
That is the difference from lifecycle, which resolves: an incomplete lifecycle operation rolls back
|
||||
from its immutable snapshot exactly once, and a second control-plane start changes nothing.
|
||||
|
||||
See ADR-0045 for the resolve / report / fail-closed contract each subsystem declares.
|
||||
@@ -0,0 +1,10 @@
|
||||
# Migration rollback
|
||||
|
||||
Before switching, M13 snapshots the exact external target, source index/space, capability contract,
|
||||
project binding, configuration version, environment fingerprint and generation. The source remains
|
||||
retained for at least 30 days.
|
||||
|
||||
Rollback invokes only the adapter's typed inverse operation and verifies exact prior external
|
||||
identity, source reachability, capability and project read path. Only then is `ROLLED_BACK` recorded;
|
||||
elapsed time is measured. Unproven restoration becomes `MANUAL_INTERVENTION_REQUIRED`, never false
|
||||
success. Cleanup treats an active migration or unexpired migration rollback snapshot as blocking.
|
||||
@@ -0,0 +1,12 @@
|
||||
# Migration validation
|
||||
|
||||
`MigrationValidationPolicyRevision` is immutable. Production revision 1 requires full completeness,
|
||||
zero failures, comparable evaluation, zero critical regressions, no latency regression, project fit,
|
||||
external validation and approved security. The isolated-LAB revision can permit a technical cutover
|
||||
for mechanics rehearsal, but never converts missing project-fit/external evidence into production
|
||||
eligibility.
|
||||
|
||||
Snapshots freeze expected/actual/missing/duplicate/malformed counts, vector finiteness and dimension,
|
||||
content hashes, space, index schema, distance metric, payload integrity, evaluation IDs,
|
||||
comparability, regressions, latency, project-fit, external validation and security. They bind the
|
||||
plan and target fingerprints. Target/plan changes make the snapshot stale; no bypass endpoint exists.
|
||||
@@ -0,0 +1,16 @@
|
||||
# Model installation rationale
|
||||
|
||||
The Models view and `GET /api/v1/models/installation-rationale` answer “Why is this installed?” from registry relations rather than prose.
|
||||
|
||||
For each model the projection reports local artifact bytes, capability and contract version, deployment and channel, production/LAB status, declared project consumers, evaluation-run identifiers, last gateway use, deletion eligibility and explicit blockers.
|
||||
|
||||
Deletion remains an explicit operator action. Deployment, project, evaluation and provenance dependencies block destructive lifecycle changes. Models with only remote registry metadata are clearly distinguished from installed artifacts. ModelForge can recommend cleanup candidates but never removes artifacts automatically.
|
||||
|
||||
M11 adds real active project consumers and immutable project-fit evidence IDs. Only non-deprecated
|
||||
bindings are listed, duplicate project identities are collapsed, and an active scoped client makes the
|
||||
operational dependency explicit. SigLIP2 and Whisper are therefore installed for ExampleVision shadow/LAB
|
||||
flows even though neither is production-promoted.
|
||||
|
||||
M12 cleanup dry-runs extend “Why Installed” with active deployment, project, evaluation, migration,
|
||||
rollback-retention and approval/plan evidence dependencies. Reclaimable bytes are displayed separately
|
||||
from blocked bytes; rationale and provenance survive any later location removal.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Operational Model Registry
|
||||
|
||||
## M9 installation rationale
|
||||
|
||||
Installed state is derived from verified artifact locations. The installation-rationale projection joins models to exact revisions/artifact sets, RuntimeProfiles, deployments, capability contracts, project bindings, gateway use and evaluation evidence. This powers deletion blockers and the Models “Why is this installed?” panel without treating operator prose as evidence.
|
||||
|
||||
## Scope
|
||||
|
||||
M2 turns the registry into PostgreSQL-backed operational state without starting discovery,
|
||||
download, runtime, scheduler, benchmark, advisor or project-production binding work. The initial
|
||||
15 candidates are materialized from the validated seed manifest and remain unverified candidates.
|
||||
|
||||
M3 extends this foundation with `UpstreamSnapshot`, `UpstreamFile`, `ArtifactSet`, `DownloadPlan`,
|
||||
`ArtifactJob`, `ArtifactJobAttempt`, `ArtifactInspection`, and `ArtifactSetMember`. Metadata refresh
|
||||
does not change candidate verification. Only a complete promoted set creates verified artifacts and
|
||||
locations; the model lifecycle still remains `candidate` pending later approval.
|
||||
|
||||
M4 references, but never collapses, those identities through RuntimeEnvironment, RuntimeProfile,
|
||||
RuntimeCompatibilityAssessment, ExecutionApproval, RuntimeProbe and DeploymentCandidate. A verified
|
||||
ArtifactSet remains only a statement about local bytes until the later gates succeed.
|
||||
|
||||
## Domain boundaries
|
||||
|
||||
| Entity | Identity | Mutable operator data | Immutable provenance |
|
||||
|---|---|---|---|
|
||||
| `Model` | local UUID, stable key, unique upstream source | display name, description, `local_metadata`, `interpretation_metadata`, lifecycle | none until represented by a revision |
|
||||
| `ModelRevision` | model + exact resolved commit SHA | deprecation/archive timestamps | upstream revision label, resolved SHA, metadata snapshot |
|
||||
| `ModelArtifact` | revision + SHA-256 | lifecycle/verification state | filename, type, serialization, digest, byte size |
|
||||
| `DerivedArtifact` | output SHA-256 | lifecycle/verification state | ordered source hashes, transformation, tool/version, configuration, environment, output hash |
|
||||
| `ArtifactLocation` | storage root + relative path | observed presence/hash/size | never used as content identity |
|
||||
| `UpstreamSnapshot` | observation UUID | staleness by new observation | provider/repository/resolved SHA and captured evidence |
|
||||
| `ArtifactSet` | revision + variant key | availability/completeness/security state | selected files and immutable selection time |
|
||||
| `DownloadPlan` | UUID/canonical idempotency hash | lifecycle status only | exact SHA, file payload, target and preflights |
|
||||
| `ArtifactJob` | one per plan | lease/progress/result | target ownership and plan relation |
|
||||
|
||||
Upstream facts, locally entered governance metadata and local interpretation are separate JSON
|
||||
documents. Seed-derived capability/role/runtime hints are marked `unverified_seed_claim`; they are
|
||||
not promoted into upstream facts. License data defaults to `unknown` with no invented SPDX ID or
|
||||
usage right.
|
||||
|
||||
## Service boundary
|
||||
|
||||
FastAPI routes call `RegistryService`; domain orchestration calls `RegistryRepository`; only the
|
||||
repository and service own SQLAlchemy access. The UI never derives registry truth from the YAML
|
||||
manifest. Compose enables the idempotent startup seed, while tests and explicit maintenance code can
|
||||
call the same seed operation directly.
|
||||
|
||||
## API
|
||||
|
||||
- `GET/POST /api/v1/models`
|
||||
- `GET/PATCH/DELETE /api/v1/models/{id}`
|
||||
- `POST /api/v1/models/{id}/deprecate`
|
||||
- `POST /api/v1/models/{id}/archive`
|
||||
- `GET/POST /api/v1/models/{id}/revisions`
|
||||
- `DELETE /api/v1/revisions/{id}`
|
||||
- `GET/POST /api/v1/revisions/{id}/artifacts`
|
||||
- `DELETE /api/v1/artifacts/{id}` and `POST /api/v1/artifacts/{id}/verify`
|
||||
- `GET /api/v1/revisions/{id}/derived-artifacts`, `POST /api/v1/derived-artifacts`
|
||||
- `DELETE /api/v1/derived-artifacts/{id}`
|
||||
- `GET/POST /api/v1/storage-roots` plus observation/capacity/delete endpoints
|
||||
- M3 discovery, upstream, artifact-set, download-plan and artifact-job endpoints documented in
|
||||
`ARTIFACT_ACQUISITION.md`
|
||||
|
||||
List endpoints are typed and paginated. Models support search plus lifecycle/source-type filters.
|
||||
Invalid UUIDs and payloads use the normalized error envelope.
|
||||
|
||||
## Lifecycle and deletion
|
||||
|
||||
Deprecation is a first-class, auditable operator action. Physical deletion first queries dependent
|
||||
revisions, artifacts, lineage edges, locations and runtime digest references. A dependency returns
|
||||
HTTP 409 with exact resource type, ID and relation; no cascade is used to erase provenance.
|
||||
|
||||
## Seed contract
|
||||
|
||||
The operation inserts missing candidates by stable key/source and updates seed-owned display/source
|
||||
interpretation only. It never overwrites `local_metadata`. The M2 production seed result was 15
|
||||
inserts followed by 0 inserts on the immediate repeat run.
|
||||
@@ -0,0 +1,11 @@
|
||||
# Multimodal capability plane
|
||||
|
||||
M9 extends the existing gateway, scheduler, agent protocol and runtime-worker abstraction with typed OCR, visual embedding and speech-transcription jobs. The control flow remains:
|
||||
|
||||
`scoped client → gateway validation → deployment resolution → GPU lease → Redis transient payload → typed node job → offline runtime → typed result → payload deletion`.
|
||||
|
||||
Binary content is canonical base64 at the HTTP boundary and is immediately placed in the transient payload store. Database rows, audit events, metrics and structured logs contain only identifiers, sizes, timings and evidence digests. There is no worker-direct consumer route and no cloud fallback.
|
||||
|
||||
Specialized adapters use local-only Transformers loaders with `trust_remote_code=false`, denied network access, read-only mounted artifacts, bounded inputs, and GPU resource measurement. A model requiring repository code or unsafe serialization fails before execution.
|
||||
|
||||
New capabilities are LAB by default. A successful probe proves executability; a completed modality-specific suite provides evaluation evidence; neither action automatically promotes a deployment.
|
||||
@@ -0,0 +1,24 @@
|
||||
# Multi-Capability Residency
|
||||
|
||||
The GPU Node runtime worker owns a bounded map of resident deployments instead of one implicit slot.
|
||||
Load, invoke, health, drain and unload always address a deployment UUID. Loading one deployment no
|
||||
longer unloads another when the placement plan proves capacity.
|
||||
|
||||
States are `COLD`, `LOAD_QUEUED`, `LOADING`, `WARM_IDLE`, `BUSY`, `DRAINING`, `UNLOADING` and
|
||||
`FAILED` at the control-plane boundary; the wire protocol uses their lower-case runtime forms.
|
||||
Policies are `ALWAYS_WARM`, `KEEP_WARM`, `LOAD_ON_DEMAND` and `LAB_ONLY`.
|
||||
|
||||
Eviction considers only managed idle residency. Active requests and `ALWAYS_WARM`/pinned models are
|
||||
ineligible. Priority and policy precede idle age and measured reload cost. Each selected eviction
|
||||
has a typed reason and expected/actual reclaimed byte evidence. A failed reclaim becomes
|
||||
`GPU_MEMORY_NOT_RECLAIMED` and stays conservative.
|
||||
|
||||
The worker reports a full resident inventory with deployment, ArtifactSet, RuntimeProfile, runtime
|
||||
environment fingerprint, upstream revision and worker generation. Startup reconciliation adopts a
|
||||
resident model only when immutable identity and node ownership match exactly. Missing inventory is
|
||||
reconciled cold. Any other GPU process remains unmanaged external usage.
|
||||
|
||||
Co-residency evidence is pairwise and reports `PROVEN_SAFE`, `EXPECTED_SAFE`, `NOT_SAFE` or
|
||||
`UNKNOWN`, with individual and combined envelope bytes. `PROVEN_SAFE` is reserved for measured live
|
||||
evidence; arithmetic alone is never promoted beyond `EXPECTED_SAFE`.
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
# Node Agent Protocol v1
|
||||
|
||||
## Purpose and boundary
|
||||
|
||||
M1.5 adds an outbound-only agent transport between a central ModelForge control plane and one or
|
||||
more compute nodes. The agent and adjacent M4 runtime worker cannot launch arbitrary commands,
|
||||
expose a shell, access the Docker socket, promote models, or serve production inference.
|
||||
|
||||
Protocol version `1` supports `hardware.inventory`, `hardware.telemetry`, and the additive typed
|
||||
capabilities `artifact.acquire.v1`, `runtime.probe.v1`, `runtime.health.v1` and
|
||||
`runtime.unload.v1`. The server rejects a
|
||||
different protocol version explicitly; capabilities are recorded for later additive negotiation.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Method | Route | Authentication | Semantics |
|
||||
|---|---|---|---|
|
||||
| POST | `/api/v1/agent/enroll` | one-time enrollment token | create/bind node and return its credential once |
|
||||
| POST | `/api/v1/agent/heartbeat` | node bearer credential | agent/version/capability liveness |
|
||||
| PUT | `/api/v1/agent/inventory` | node bearer credential | stable host/GPU inventory reconciliation |
|
||||
| PUT | `/api/v1/agent/telemetry` | node bearer credential | latest measured RAM/storage/GPU telemetry |
|
||||
| GET | `/api/v1/agent/artifact-jobs/next` | node bearer credential | claim one job owned by this node |
|
||||
| POST | `/api/v1/agent/artifact-jobs/{id}/progress` | node bearer + lease | bounded progress/cancel handshake |
|
||||
| POST | `/api/v1/agent/artifact-jobs/{id}/complete` | node bearer + lease | verified inventory and promotion report |
|
||||
| POST | `/api/v1/agent/artifact-jobs/{id}/fail` | node bearer + lease | typed failure and bounded retry request |
|
||||
| GET | `/api/v1/agent/runtime-probes/next` | node bearer credential | claim one exact-node typed probe |
|
||||
| POST | `/api/v1/agent/runtime-probes/{id}/progress` | node bearer + lease | lifecycle progress/cancel handshake |
|
||||
| POST | `/api/v1/agent/runtime-probes/{id}/complete` | node bearer + lease | layered health, inference and measured evidence |
|
||||
| POST | `/api/v1/agent/runtime-probes/{id}/fail` | node bearer + lease | normalized typed runtime failure |
|
||||
|
||||
Artifact leases contain only repository, exact commit, selected files, expected sizes/checksums,
|
||||
approved target root and reserve policy. Credentials from one node cannot claim or update another
|
||||
node's work. There is no arbitrary command field.
|
||||
|
||||
Runtime leases similarly contain only exact server-resolved artifact paths and digests, immutable
|
||||
RuntimeProfile/RuntimeEnvironment identity and the fixed technical probe input. A static adjacent
|
||||
worker executes them; neither the protocol nor Node Agent provides general process/container control.
|
||||
|
||||
M5 adds `deployment.load.v1`, `deployment.invoke.v1`, `deployment.health.v1`,
|
||||
`deployment.drain.v1` and `deployment.unload.v1`. The adjacent runtime worker polls authenticated
|
||||
typed jobs outbound over the same private-CA boundary. Raw project traffic never reaches the node,
|
||||
and the Node Agent credential is never accepted as a service-client credential.
|
||||
|
||||
Administrative token creation, revocation, node metadata changes and credential revocation live
|
||||
under `/api/v1/admin`. They require `X-ModelForge-Admin-Token`; the API fails closed when
|
||||
`MODELFORGE_OPERATOR_API_KEY` is unset.
|
||||
|
||||
`POST /api/v1/admin/hardware/nodes/{id}/credential/rotate` atomically revokes the previous
|
||||
credential and returns its replacement once. Install the replacement in the protected agent state
|
||||
file before restarting it. Rotation and revocation are separate audited operations.
|
||||
|
||||
Permanent decommission is a separate terminal lifecycle. Its preview and execute endpoints are
|
||||
operator-only. Completion revokes every node credential, and ordinary enrollment, rotation,
|
||||
management, authentication, heartbeat, inventory and telemetry all refuse the tombstoned identity.
|
||||
An outage or lost credential must use the recovery runbook instead of decommission.
|
||||
|
||||
## Identity, time and ordering
|
||||
|
||||
- A node credential is scoped to exactly one `ComputeNode` UUID.
|
||||
- Stable node identity comes from the persisted agent state volume, not hostname or container ID.
|
||||
- GPUs reconcile by `(node_id, GPU UUID)`; hostname, device index and PCI address are mutable facts.
|
||||
- Each inventory and telemetry stream has a persistent monotonically increasing sequence number.
|
||||
- The control plane stores both agent `observed_at` and server `received_at`.
|
||||
- Replayed sequence numbers are rejected. A higher sequence with an older observation timestamp
|
||||
advances the stream cursor but does not replace last-good state; the reason remains visible until
|
||||
the node clock catches up. Observations beyond the configurable future-skew limit (default five
|
||||
minutes) are handled the same way, preventing one bad clock from poisoning latest state.
|
||||
Server-received time drives liveness, so clock skew cannot keep a dead node online.
|
||||
- An accelerator is marked missing only after a successful complete inventory omits its UUID. An
|
||||
unavailable/incomplete NVML collection preserves last-good state.
|
||||
|
||||
## Liveness
|
||||
|
||||
Enabled nodes are `online`, `stale`, or `offline` using server-received heartbeat time. Defaults are
|
||||
30 and 90 seconds and are configurable. Disabled nodes always report `disabled` and their agent
|
||||
credential is rejected. Offline/stale transitions, return online, enrollment, revocation and
|
||||
meaningful inventory changes are audited; high-rate telemetry does not flood the audit log.
|
||||
Terminal nodes report `decommissioned`; liveness reconciliation never changes that state.
|
||||
|
||||
## Error contract
|
||||
|
||||
Malformed payloads return validation errors. Missing/invalid/revoked credentials return 401;
|
||||
credential/node ownership mismatches and disabled nodes return 403; unknown resources return 404;
|
||||
incompatible protocol versions, replayed samples and conflicting node identities return 409. The
|
||||
normal API correlation ID remains present for operational tracing.
|
||||
|
||||
|
||||
## M16 disconnect and diagnostics
|
||||
|
||||
A disconnected agent stops publishing; liveness moves online → stale → offline on the configured
|
||||
thresholds and no work is assigned to it. Reconnection resumes publication on the same persisted
|
||||
identity, and a live rehearsal confirmed the heartbeat genuinely stops rather than merely ageing.
|
||||
|
||||
Enrolment is single-use and atomically claimed. A sixty-way concurrent storm against one token
|
||||
produced exactly one identity and one active credential.
|
||||
|
||||
A failed publication names the status, method and path — never the response body, which can echo
|
||||
what the agent was publishing, and never the credential. Before M16 the agent logged only
|
||||
`control-plane publication failed; retrying`, which cannot tell an operator whether the cause is a
|
||||
revoked credential or a DNS failure.
|
||||
@@ -0,0 +1,49 @@
|
||||
# Node decommission architecture
|
||||
|
||||
## Contract
|
||||
|
||||
Node decommission is a terminal, audited lifecycle transition for a `ComputeNode`; it is not row
|
||||
deletion and not a recovery shortcut. The node UUID, persisted identity, historical inventory,
|
||||
artifact jobs, runtime evidence, deployments, serving history, capacity history and audit records
|
||||
remain addressable. Only rebuildable current truth is removed or made unschedulable.
|
||||
|
||||
The operator first calls the preview endpoint. The preview returns the node generation, a
|
||||
deterministic SHA-256 dependency digest, blockers, current-state cleanup counts and retained
|
||||
provenance counts. Execute requires the same generation and digest, a unique idempotency key, an
|
||||
operator identity, a reason of at least ten characters, and an exact typed match of the persisted
|
||||
identity, hostname or display name. There is no force or cancel-dependencies flag.
|
||||
|
||||
## Fail-closed dependency graph
|
||||
|
||||
Execution refuses an online node and any non-terminal production/LAB deployment, experiment route,
|
||||
download plan, artifact or serving job, gateway request, runtime probe, residency, GPU lease,
|
||||
production approval, lifecycle approval/operation, migration/cutover, recovery operation or
|
||||
identity-referencing enrollment. It also refuses removal of the last known live artifact copy and
|
||||
the last production-eligible node while production deployments exist.
|
||||
|
||||
Unknown status values are treated as active because blocker queries exclude an explicit terminal
|
||||
set. PostgreSQL execution locks every dependency and cleanup table in the transaction, re-runs the
|
||||
preview, then claims the node generation with a conditional update. A changed generation or digest
|
||||
returns 409 and requires a fresh preview.
|
||||
|
||||
## Terminal mutation
|
||||
|
||||
One transaction revokes every active node credential, clears host/storage/accelerator latest-state
|
||||
telemetry and scheduler accelerator state, removes cold residency truth, marks accelerators
|
||||
decommissioned, makes storage roots unavailable and read-only, marks their artifact locations
|
||||
unreachable, removes eligibility and current inventory/capabilities, and writes the node tombstone.
|
||||
It also writes one immutable `NodeDecommissionOperation` and one `NODE_DECOMMISSIONED` audit event
|
||||
containing the reason, operator, timestamp, prior state, cleanup counts and credential result.
|
||||
|
||||
A uniqueness constraint on node and idempotency key plus the transaction makes retries deterministic
|
||||
and the audit exactly once. Normal enrollment, metadata update, credential rotation, authentication,
|
||||
agent publication, hardware reconciliation and scheduler placement all reject or exclude a
|
||||
decommissioned identity. An explicit future recovery lifecycle would require a separate audited
|
||||
design; none exists today.
|
||||
|
||||
## Invariant
|
||||
|
||||
`decommissioned_nodes_are_terminal` verifies every tombstone is disabled, in decommissioned
|
||||
status/liveness, ineligible, without active credentials, active work, residencies, leases, runtime
|
||||
probes, scheduler/current telemetry or current inventory. The invariant is read-only and runs with
|
||||
the existing platform invariant suite.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Node recovery
|
||||
|
||||
A compute node is two separate things during recovery: a persisted hardware identity, which is
|
||||
authoritative history, and a credential, which is secret material that hashes cannot reconstruct.
|
||||
|
||||
## Identity
|
||||
|
||||
`ComputeNode.key` is the agent's persisted identity, written to the node's state volume and reused
|
||||
across restarts. A recovered node that still has its identity file keeps its node id, its
|
||||
accelerators, its storage roots and its history. Losing the credential does not lose the identity.
|
||||
|
||||
A node whose identity file is also gone enrols as a new node. That is correct — ModelForge cannot
|
||||
distinguish it from different hardware — and the operator disables the superseded row rather than
|
||||
letting two nodes claim the same machine.
|
||||
|
||||
## Single-use enrolment
|
||||
|
||||
One enrolment token mints exactly one node identity. The claim is a conditional update whose row
|
||||
count decides:
|
||||
|
||||
```sql
|
||||
update node_enrollments set used_at = now()
|
||||
where id = :id and used_at is null
|
||||
```
|
||||
|
||||
This closed a real defect found by the M15 node-recovery rehearsal: the previous check read
|
||||
`used_at` and wrote it later, and two agent threads racing into `ensure_enrolled` burned one token
|
||||
on two compute-node rows for the same hardware. The Node Agent's publication and artifact loops now
|
||||
also share an enrolment lock, so it issues one enrol request rather than two.
|
||||
|
||||
## Credential recovery
|
||||
|
||||
```text
|
||||
RESTORABLE_SECRET restored from backup and immediately usable
|
||||
ROTATABLE_SECRET hash restores as authoritative state; the consumer needs a new secret
|
||||
NON_EXPORTABLE_SECRET never written to a backup at all
|
||||
```
|
||||
|
||||
Node credentials are `ROTATABLE_SECRET`. Only `secret_hash` is stored, so a restore reinstates
|
||||
which credentials existed and which were revoked, but cannot hand the plaintext back to a node that
|
||||
lost it. Scenarios:
|
||||
|
||||
| Scenario | Recovery |
|
||||
| --- | --- |
|
||||
| database restored, node still holds its credential | the restored hash matches; the node reconnects |
|
||||
| credential lost on the node | operator issues a fresh enrolment token; the node re-enrols and keeps its identity |
|
||||
| credential believed stolen | operator revokes it; the node re-enrols |
|
||||
| node hardware replaced | new identity, new enrolment; the old row is disabled |
|
||||
|
||||
Revocation is permanent. A revoked credential stays refused after re-enrolment: the partial unique
|
||||
index `uq_active_node_credential` allows one active credential per node, and enrolment revokes the
|
||||
previous one rather than deleting it, so the revocation survives in history.
|
||||
|
||||
## Recovery sequence
|
||||
|
||||
```text
|
||||
node unavailable
|
||||
→ liveness reflects offline/stale; capability health degrades honestly
|
||||
→ operator issues an enrolment token
|
||||
→ agent re-enrols, keeping its persisted identity
|
||||
→ inventory and telemetry are re-collected
|
||||
→ scheduler reconciles against re-measured NVML truth
|
||||
→ capability smoke
|
||||
```
|
||||
|
||||
Nothing about the node's current state is restored from a backup. After a control-plane restore the
|
||||
node rows exist with their identities and history, but telemetry, leases and residency are empty
|
||||
until a live agent reports again.
|
||||
|
||||
## Live evidence
|
||||
|
||||
Disaster rehearsal D used a disposable Node Agent identity; the production GPU Node credential was
|
||||
never revoked. The disposable node enrolled once (one enrol request, zero 403s), its credential was
|
||||
revoked and refused three times consecutively with 401, its credential file was deleted, and it
|
||||
re-enrolled on a fresh token as node `41a7530c-7750-47c9-92d7-78302ee664fd` — the same persisted
|
||||
identity, with one active and one revoked credential, one enabled rehearsal node and one distinct
|
||||
hostname. The old credential still returned 401 afterwards.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Observability architecture
|
||||
|
||||
## Boundary
|
||||
|
||||
M14 observes the existing control-plane truth. It does not introduce a second GPU, node, scheduler
|
||||
or runtime collector and it never becomes a serving dependency. NVML values remain owned by the
|
||||
Node Agent; gateway requests, placements, leases, residency, jobs, evaluations, lifecycle and
|
||||
migrations remain owned by their existing durable journals.
|
||||
|
||||
Telemetry has one of five explicit kinds:
|
||||
|
||||
- `COUNTER`: monotonic outcomes, such as requests or operations;
|
||||
- `GAUGE`: current bounded state, such as queue depth or bytes;
|
||||
- `HISTOGRAM`: distributions with fixed buckets, such as request duration;
|
||||
- `EVENT`: durable transitions with actor and time;
|
||||
- `STATE`: a typed current classification such as `STALE` or `FIRING`.
|
||||
|
||||
## Metric contract
|
||||
|
||||
Names are lowercase Prometheus identifiers prefixed with `modelforge_`; counters end in `_total`,
|
||||
durations in `_seconds` and storage values in `_bytes`. `/metrics` is operator-authenticated and is
|
||||
not published by Compose. It includes process metrics even when historical storage is unavailable.
|
||||
|
||||
| Plane | Authoritative source | Export examples |
|
||||
| --- | --- | --- |
|
||||
| API | in-process middleware | `modelforge_api_requests_total`, request-duration histogram |
|
||||
| Gateway | `gateway_requests` | request outcomes, request and queue duration histograms |
|
||||
| Scheduler | placements, jobs, capacity snapshots | decisions, queue depth, schedulable/external bytes |
|
||||
| Runtime | `serving_jobs` | operation outcomes |
|
||||
| Node/GPU | Node Agent receive time and NVML latest | freshness, RAM, utilization, temperature, power |
|
||||
| Storage | registered storage roots and node volumes | free bytes and retained capacity snapshots |
|
||||
| Acquisition | artifact jobs and registry state | job states and integrity alerts |
|
||||
| Evaluation | evaluation runs | run states |
|
||||
| Lifecycle | lifecycle operations | operation stages and rollback failures |
|
||||
| Migration | migration plans/cutovers | states and failure alerts |
|
||||
|
||||
## Cardinality and content
|
||||
|
||||
Permitted labels are bounded operational classifications: capability, status, priority, route
|
||||
class, node display name, device index, verdict, operation, state and availability. Label values are
|
||||
single-line, escaped and limited to 128 characters. The registry rejects request IDs, UUIDs,
|
||||
digests, query/prompt fields, filenames, raw errors and other high-cardinality or sensitive keys.
|
||||
|
||||
Metrics never contain prompts, query/document text, images, audio, vectors, artifact paths or raw
|
||||
payloads. Request correlation remains in structured logs and bounded database records, not metric
|
||||
labels.
|
||||
|
||||
## Time and failure semantics
|
||||
|
||||
Capacity snapshots carry `observed_at`, `received_at`, calculated freshness and `KNOWN`, `STALE` or
|
||||
`UNAVAILABLE`. Central receive time determines liveness. Stale data is historical evidence, never
|
||||
current capacity truth.
|
||||
|
||||
The minute-level polling service catches persistence and collection failures, rolls back its own
|
||||
transaction and sets `modelforge_observability_degraded=1`. It does not alter gateway, scheduler or
|
||||
runtime decisions. The metrics endpoint falls back to the in-memory process registry on database
|
||||
failure. This is `OBSERVABILITY_DEGRADED`, not inference failure.
|
||||
|
||||
## Access and audit
|
||||
|
||||
All operations and metrics routes use the operator credential. Policy/rule revisions,
|
||||
acknowledgements and maintenance windows are audited. Individual metric samples are deliberately not
|
||||
audited. No notification or external-process-control adapter is part of M14.
|
||||
|
||||
## M15 recovery observability
|
||||
|
||||
Observability history is authoritative and restores with the database: SLO policy revisions and
|
||||
evaluations, alert rule revisions, alerts, alert history, maintenance windows, incidents and
|
||||
capacity snapshots and aggregates are all fingerprinted subjects.
|
||||
|
||||
Current alert state is not. Recovery reconciliation suppresses restored `PENDING`/`FIRING` alerts so
|
||||
a rebuilt control plane re-derives which alerts are firing now, while their history is retained. In
|
||||
the live rehearsal three restored firing alerts were suppressed and the rebuilt plane raised its own
|
||||
after re-evaluating current state, with 187 restored capacity snapshots plus the ones it measured
|
||||
itself.
|
||||
|
||||
Recovery readiness is an operational signal rather than a new production SLO family: M15 adds no SLO
|
||||
policies, only the five recovery alert rules.
|
||||
|
||||
|
||||
## M16 observability under fault
|
||||
|
||||
Monitoring is never a safety dependency. When observability persistence is unavailable, the
|
||||
in-memory `modelforge_observability_degraded` gauge rises, the operations overview reports
|
||||
`OBSERVABILITY_DEGRADED`, and serving, scheduling and recovery continue on their own authoritative
|
||||
state. The M16 invariants deliberately read none of the observability tables, so the question "is
|
||||
state still sound?" can be answered precisely when monitoring is the thing that failed.
|
||||
|
||||
Alerts raised during a chaos scenario are real alerts and must be resolved or acknowledged with
|
||||
their true cause. The release gate reports any alert left pending or firing at the end of a run.
|
||||
@@ -0,0 +1,43 @@
|
||||
# OCR model selection
|
||||
|
||||
M11 used the ModelForge Hugging Face provider for a bounded TrOCR shortlist; no web scraping or
|
||||
mass-download was used.
|
||||
|
||||
| Candidate | Exact revision | Outcome |
|
||||
| --- | --- | --- |
|
||||
| `microsoft/trocr-small-handwritten` | `b4648cfa171985a6745f37ddd637e98c0da958ac` | task mismatch; not selected |
|
||||
| `microsoft/trocr-base-printed` | `93450be3f1ed40a930690d951ef3932687cc1892` | deep preflight; blocked |
|
||||
| `microsoft/trocr-large-printed` | `9ff792d8e7c22061f2ee67e1ed2246b1f9ef1e98` | larger duplicate direction; not downloaded |
|
||||
| `microsoft/trocr-base-handwritten` | `eaacaf452b06415df8f10bb6fad3a4c11e609406` | task mismatch despite clearer license metadata |
|
||||
|
||||
The selected base-printed revision exposed a safe eight-file Safetensors plan of 1,334,746,034 bytes;
|
||||
the parallel 1,333,508,485-byte PyTorch pickle weight was correctly excluded. Execution nevertheless
|
||||
stopped before download: upstream scanning was incomplete and license metadata was absent/unknown.
|
||||
The immutable discovery assessment is `license_blocked`. No security gate was weakened to manufacture
|
||||
quality evidence.
|
||||
|
||||
The existing `microsoft/trocr-small-printed` candidate remains quality-blocked at CER 0.721246 and WER
|
||||
0.976190. PaddleOCR-VL remains blocked because its exact revision requires custom code;
|
||||
`trust_remote_code=false` remains mandatory. Final state is therefore:
|
||||
|
||||
```text
|
||||
document.ocr@1: PLATFORM_READY / NO_PRODUCTION_MODEL
|
||||
```
|
||||
|
||||
## Closure refresh — 2026-08-26
|
||||
|
||||
The official Hugging Face API was queried again for the bounded shortlist. The exact revisions did
|
||||
not change:
|
||||
|
||||
- `microsoft/trocr-small-handwritten@b4648cfa171985a6745f37ddd637e98c0da958ac` still has no
|
||||
declared license and exposes one pickle weight with no Safetensors weight;
|
||||
- `microsoft/trocr-base-printed@93450be3f1ed40a930690d951ef3932687cc1892` still has no
|
||||
declared license and exposes both Safetensors and pickle weights, with no affirmative repository
|
||||
security status returned;
|
||||
- `microsoft/trocr-large-printed@9ff792d8e7c22061f2ee67e1ed2246b1f9ef1e98` still has no
|
||||
declared license and exposes both weight formats;
|
||||
- `microsoft/trocr-base-handwritten@eaacaf452b06415df8f10bb6fad3a4c11e609406` still declares
|
||||
MIT and exposes both weight formats, but remains a task mismatch.
|
||||
|
||||
This refresh does not satisfy the license/scanner gate for the printed candidate. No candidate was
|
||||
downloaded or executed, and the final state remains `PLATFORM_READY / NO_PRODUCTION_MODEL`.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Operational incidents
|
||||
|
||||
M14 uses a lightweight `OperationalIncident`; it is not an ITSM replacement. An incident groups
|
||||
alerts only when an evidenced shared subject exists.
|
||||
|
||||
Correlation values are:
|
||||
|
||||
- `LIKELY_ROOT`: the alert directly reports the shared resource failure, such as GPU Node offline;
|
||||
- `DOWNSTREAM`: a capability failure includes the same node identity;
|
||||
- `RELATED`: shared timeline evidence without a directional claim;
|
||||
- `UNKNOWN`: no justified causal direction.
|
||||
|
||||
A firing `NODE_OFFLINE` alert opens one node incident. Capability-unavailable alerts that carry that
|
||||
node ID may join it as downstream. Other alerts stay independent rather than claiming causality.
|
||||
When every grouped alert resolves, the incident resolves and appends a timeline event.
|
||||
|
||||
The bounded timeline stores alert state changes, relation, summary, operator acknowledgement and
|
||||
recovery timestamps. It never stores prompts, vectors, images, audio, document text or raw payloads.
|
||||
Incidents explain degradation; they authorize no remediation.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Placement Explainability
|
||||
|
||||
`PlacementPlan` is the first-class explanation for both a dry-run and a real request. Both paths call
|
||||
the same planner. A plan records the requested capability/deployment, node and accelerator,
|
||||
eligibility, priority, current residents, external use, reserve, authoritative envelope requirement,
|
||||
headroom before and after, selected managed evictions, queue impact, verdict and typed reasons.
|
||||
|
||||
Verdicts are `ADMIT`, `ADMIT_AFTER_EVICTION`, `QUEUE`, `REJECT_CAPACITY`, `REJECT_HEALTH`, and
|
||||
`REJECT_POLICY`. Reason codes include `INSUFFICIENT_SCHEDULABLE_VRAM`, `EXTERNAL_GPU_PRESSURE`,
|
||||
`RESIDENCY_CONFLICT`, `EVICTION_NOT_ALLOWED`, `DEADLINE_CANNOT_BE_MET`, `LEASE_CONFLICT`,
|
||||
`GPU_MEMORY_NOT_RECLAIMED`, and `SCHEDULER_STATE_STALE`.
|
||||
|
||||
The operator dry-run answers “what if I invoke this capability now?” without loading a model. The UI
|
||||
shows the actual budget, pressure, residency, possible eviction, expected reclaimed bytes, cold-load
|
||||
cost and final reasons. Recent placement history is capped at 500 records. It contains safe request
|
||||
shape metadata only—never text, image, audio or client-supplied placement commands.
|
||||
|
||||
Co-residency arithmetic yields at most `EXPECTED_SAFE`. `PROVEN_SAFE` requires two exact residents
|
||||
on the same worker generation, KNOWN attribution and a measured combined footprint. Promotion and
|
||||
scheduling readiness remain separate lifecycle decisions.
|
||||
@@ -0,0 +1,36 @@
|
||||
# ExampleVision vision integration
|
||||
|
||||
ExampleVision retains its production recognizer. The M11 shadow path is:
|
||||
|
||||
```text
|
||||
normalized card image -> vision.embedding@1 -> Gateway -> scheduler -> GPU Node
|
||||
-> exact 768-dimensional visual space -> pgvector nearest neighbours
|
||||
-> existing metadata/fusion boundary -> AMBIGUOUS | NO_MATCH | UNAVAILABLE
|
||||
```
|
||||
|
||||
The consumer adapter contains no SigLIP, revision, runtime, worker, GPU Node or artifact identity. It
|
||||
validates bounded images and the exact visual space/dimension returned by the capability contract.
|
||||
Images are transient and absent from audit, gateway telemetry and ordinary logs.
|
||||
|
||||
References and queries use `examplevision-card-rgb-pad-224-v1`: orientation/geometry normalization is
|
||||
reused from ExampleVision, then the full RGB card is aspect-preserving padded to 224 x 224. Reference
|
||||
rows retain canonical printing ID, source-image content hash and visual-space identity. A separate
|
||||
768-dimensional pgvector schema prevents reuse of ExampleVision's existing 1000-dimensional MobileNet
|
||||
vectors. Index runs are cursor-based, idempotent, bounded to batches of one through four and may be
|
||||
resumed or activated only with one consistent space.
|
||||
|
||||
Live M11 indexing processed 80 real catalog reference images in two resumable 40-image invocations,
|
||||
20 successful background Gateway requests and zero failures. The first ten batches took about 4.6
|
||||
seconds. Ten concurrent interactive technical queries completed while background indexing was active.
|
||||
These queries used a catalog-artwork derivative, not an owner photograph, so they prove routing,
|
||||
retrieval and QoS but do not constitute the required real-photo project-fit evaluation.
|
||||
|
||||
No confidence threshold was fabricated. Until reviewed owner-photo ground truth exists, the shadow
|
||||
recognizer cannot emit `CONFIDENT`; the recorded recommendation is `REQUIRES_MORE_EVIDENCE`.
|
||||
|
||||
ADR-0031 separates this production-fit limitation from engineering closure. The capability-first
|
||||
integration and complete reference index may pass M11 while owner-photo validation remains
|
||||
`DEFERRED_EXTERNAL_VALIDATION`. Future production cutover is guarded by
|
||||
`POKEVAULT_VISION_PRODUCTION_VALIDATION`; only independently reviewed `OWNER_PHOTO` evidence can
|
||||
satisfy it. `PUBLIC_PHYSICAL_CAPTURE` and `CATALOG_REFERENCE` evidence remain explicitly weaker and
|
||||
cannot enable auto-accept.
|
||||
@@ -0,0 +1,111 @@
|
||||
# PostgreSQL backup and restore
|
||||
|
||||
## Method
|
||||
|
||||
ModelForge owns its own backup rather than delegating to a filesystem snapshot:
|
||||
|
||||
```text
|
||||
pg_dump --format=custom --compress=6 --no-owner --no-privileges --serializable-deferrable
|
||||
```
|
||||
|
||||
Custom format restores schema, data, sequences, constraints, indexes and extension dependencies
|
||||
through `pg_restore`, and `--serializable-deferrable` takes the whole dump in one snapshot that
|
||||
cannot see a partially applied transaction. Copying `/var/lib/postgresql/data` from a running
|
||||
server is never done.
|
||||
|
||||
The control-plane image ships `pg_dump`, `pg_restore` and `psql` pinned to the same major version
|
||||
as the server (PostgreSQL 17), installed from Alpine's signed 3.23 repository. The release build
|
||||
upgrades repository packages before adding that client and verifies the resulting image.
|
||||
|
||||
## Recorded identity
|
||||
|
||||
Every backup records the server version and version number, the major version, the database name,
|
||||
the installed extensions, the backup tool version, the backup method and a redacted connection
|
||||
string, alongside the Alembic revision. A restore preflight compares source and destination major
|
||||
versions and refuses `POSTGRES_VERSION_INCOMPATIBLE`.
|
||||
|
||||
Historical M15 live evidence: server `PostgreSQL 17.11`, `server_version_num` 170011, extensions
|
||||
`plpgsql 1.0`, tool `pg_dump (PostgreSQL) 17.11 (Debian 17.11-1.pgdg13+2)`, schema
|
||||
`20260827_0021`. The v1.1 Alpine release image independently reports `pg_dump (PostgreSQL) 17.11`.
|
||||
|
||||
## Point-in-time recovery
|
||||
|
||||
```text
|
||||
PITR: NOT_SUPPORTED
|
||||
```
|
||||
|
||||
M15 implements verified logical snapshots. WAL archiving and continuous point-in-time recovery are
|
||||
deliberately out of scope for v1. The value is exported as `POINT_IN_TIME_SUPPORT` and rendered on
|
||||
the recovery dashboard so no operator has to infer it. The practical consequence is a finite RPO
|
||||
bounded by the backup interval, and the platform measures it rather than claiming zero.
|
||||
|
||||
## Restore
|
||||
|
||||
A restore runs against a genuinely separate database. `_same_database` compares host, port and
|
||||
database name, and a plan targeting the control plane's own database is refused at creation with
|
||||
`DESTINATION_NOT_ISOLATED`. Production targets additionally require
|
||||
`MODELFORGE_RESTORE_ALLOW_PRODUCTION_TARGET` and a `DISASTER_RECOVERY` mode.
|
||||
|
||||
```text
|
||||
pg_restore --no-owner --no-privileges --exit-on-error --single-transaction
|
||||
```
|
||||
|
||||
`--single-transaction` means a failed restore leaves nothing behind. The destination must be empty:
|
||||
`DESTINATION_NOT_EMPTY` is returned rather than merging into existing data, which is what makes a
|
||||
retry safe.
|
||||
|
||||
## Schema compatibility
|
||||
|
||||
The build walks its own Alembic chain by `down_revision` — filename order is not migration order —
|
||||
and compares the backup's revision against it:
|
||||
|
||||
| Backup revision | Result |
|
||||
| --- | --- |
|
||||
| equals head | restored as is |
|
||||
| known, older than head | restored, then migrated forward to head |
|
||||
| unknown to this build | `SCHEMA_TOO_NEW`, fail closed |
|
||||
| build ships no chain | `SCHEMA_UNKNOWN`, fail closed |
|
||||
| backup records none | preflight fails |
|
||||
|
||||
Restoring a newer backup onto older code is never silently downgraded. The discovery resolves both
|
||||
the source checkout and the installed image layout; a packaging error that hides the chain produces
|
||||
`SCHEMA_UNKNOWN` rather than an accidental pass.
|
||||
|
||||
## Validation
|
||||
|
||||
After the restore the destination is measured, not assumed: table, constraint, index and sequence
|
||||
counts, the Alembic head, the audit high-water mark, a bounded semantic fingerprint and its diff
|
||||
against the fingerprint captured at backup time, and the residual row count of every current-truth
|
||||
table.
|
||||
|
||||
The `READY` gate requires all of:
|
||||
|
||||
```text
|
||||
database_validated restored table and constraint counts meet the plan's requirement
|
||||
current_truth_reconciled no stale lease, telemetry or inventory row survives
|
||||
schema_current the destination is at this build's Alembic head
|
||||
audit_available the audit trail restored with a non-zero sequence
|
||||
audit_chain_intact every event/hash/link and the durable checkpoint verify strictly
|
||||
fingerprint_compatible current m15.2, or the exact schema-0022 m15.1 migration rule
|
||||
no_unknown_corruption every fingerprint difference is explained
|
||||
```
|
||||
|
||||
`READY` always performs a fresh strict target audit verification. A resumed operation cannot reuse
|
||||
the presence of a journaled `VALIDATING` or `READY` duration as that proof. Audit rows are read with
|
||||
a stable `(sequence, id)` cursor, so a corrupt duplicate sequence split at the 500-row page boundary
|
||||
is observed and rejected rather than skipped. A separate aggregate identity check requires total
|
||||
rows, distinct UUIDs and distinct sequences to agree before the chain can pass; this also catches a
|
||||
duplicate cursor tuple after constraints were removed from a tampered restored schema.
|
||||
|
||||
Live evidence from disaster rehearsal A: 121 tables, 1,759 constraints, 590 indexes, audit
|
||||
high-water mark 838, 108 of 112 fingerprinted tables byte-identical, and four explained
|
||||
differences.
|
||||
|
||||
## Identity columns
|
||||
|
||||
`audit_events.sequence` is `GENERATED BY DEFAULT AS IDENTITY`, but `AuditWriter` always supplies the
|
||||
value explicitly, so the underlying sequence is never advanced. The dump carries its `SEQUENCE SET`
|
||||
and the restore reproduces the source state exactly — an insert that omits `sequence` collides
|
||||
identically on both the source and the restored database. This is pre-existing M0 behaviour
|
||||
faithfully preserved by the restore, not a recovery defect, and it is recorded here because a
|
||||
future milestone that starts relying on the identity default must fix it in the audit writer first.
|
||||
@@ -0,0 +1,25 @@
|
||||
# Project Bindings
|
||||
|
||||
## Operational evidence
|
||||
|
||||
M6 links the ExampleRAG `rag.embedding@1` binding to scoped service authentication, isolated embedding
|
||||
migrations and evaluation suites. A binding still does not select a concrete model or node.
|
||||
|
||||
Projects bind to versioned capabilities, never deployments, model names, runtime engines or paths.
|
||||
M11 permits a service identity to reference exactly one real binding and records its integration
|
||||
environment and purpose. Authentication revalidates that the binding is not deprecated and that its
|
||||
contract still equals the client's sole scope. Background and interactive work use separate identities
|
||||
so callers cannot self-promote priority. See ADR-0030.
|
||||
|
||||
Deployment promotion, placement and embedding-space migration remain control-plane concerns. A
|
||||
project binding may pin a capability version and embedding-space expectation, but cannot override
|
||||
the stable route or address a worker directly.
|
||||
|
||||
ExampleVision has separate operational bindings/clients for background vision indexing, interactive
|
||||
vision queries and interactive speech transcription. OCR remains planning-only until a safe model and
|
||||
project evidence exist.
|
||||
|
||||
M12 treats a project production binding as an independent lifecycle subject. A platform capability
|
||||
may remain stable while the project request is blocked. The default project-production policy needs
|
||||
eligible fit and satisfied external validation; ExampleVision Vision therefore remains blocked and ASR
|
||||
remains LAB without mutating their engineering integration evidence.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Project capability integrations
|
||||
|
||||
M11 distinguishes planning metadata from an operational consumer. An operational integration exists
|
||||
only when an active `ServiceClient` is bound to one non-deprecated `ProjectBinding`. The binding fixes
|
||||
the project and capability contract; the client adds an environment, purpose, server-owned QoS class
|
||||
and a one-time credential. Consumers still know only the ModelForge base URL, credential and logical
|
||||
capability key.
|
||||
|
||||
`GET /api/v1/project-integrations` projects that identity together with gateway usage, latest project-
|
||||
fit evidence, deployment and measured resource envelope. Request volume counts terminal gateway
|
||||
requests (`completed` and `error`), latency is derived from those requests, and content is never
|
||||
included. `POST /api/v1/admin/service-clients/{id}/credential/rotate` replaces a credential atomically;
|
||||
the prior secret immediately fails authentication.
|
||||
|
||||
M11 operational consumers are ExampleVision shadow/evaluation clients for `vision.embedding@1` and
|
||||
`speech.transcription@1`. The separate vision indexing and interactive identities enforce background
|
||||
versus interactive QoS without accepting a priority from the request body. Planning-only OCR bindings
|
||||
remain visibly distinct from active consumers.
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
# Project-fit evaluation
|
||||
|
||||
`ProjectFitEvidence` is an immutable, digest-deduplicated statement about one project binding and,
|
||||
optionally, one capability deployment. It records environment, suite/dataset identity, case count,
|
||||
metrics, critical failures, blockers, recommendation, rationale, evidence fingerprint and one typed
|
||||
visual evidence class: `OWNER_PHOTO`, `PUBLIC_PHYSICAL_CAPTURE` or `CATALOG_REFERENCE`. The deployment
|
||||
must implement the binding's exact capability contract.
|
||||
|
||||
Allowed recommendations are `PROMOTION_ELIGIBLE`, `KEEP_LAB`, `BLOCKED` and
|
||||
`REQUIRES_MORE_EVIDENCE`. ModelForge rejects a production or promotion-eligible claim when blockers
|
||||
or critical failures are present, and it never turns evidence into a promotion action.
|
||||
|
||||
Engineering integration, project production fit, production validation and production action are
|
||||
separate fields. `POKEVAULT_VISION_PRODUCTION_VALIDATION` can become satisfied only from reviewed
|
||||
`OWNER_PHOTO` evidence. Public physical captures may support separately labelled robustness or
|
||||
printing evaluation; catalog references support only index/embedding/route integrity. Neither may be
|
||||
upgraded to owner-photo evidence. Production action is `NONE` in M11.
|
||||
|
||||
M11 records:
|
||||
|
||||
- ExampleVision vision engineering: `PASS`; 21,775/21,775 available references, zero indexing failures,
|
||||
scoped shadow routing and QoS proven.
|
||||
- ExampleVision vision production fit: `REQUIRES_MORE_EVIDENCE`; owner-photo validation is
|
||||
`DEFERRED_EXTERNAL_VALIDATION`, zero reviewed owner-photo cases, production action `NONE`.
|
||||
- ExampleVision voice query: `KEEP_LAB`; four NL/EN TTS acceptance cases, all successful, with a small
|
||||
non-microphone dataset and measured word error rate of 0.380952.
|
||||
- OCR: no project-fit claim, because no safe, clearly licensed candidate reached execution.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Project integrations
|
||||
|
||||
Projects consume versioned capabilities through scoped service identities. A project integration
|
||||
contains a `Project`, `ProjectBinding`, optional `ProjectEvaluationBinding`, service client and
|
||||
project-owned data plane. It never contains a concrete model path, runtime, node or artifact
|
||||
selector.
|
||||
|
||||
ExampleRAG is the first operational integration. Its production retrieval remains `CURRENT`; M6 adds
|
||||
an operator-only `SHADOW` target. Both targets are server-owned identifiers, not public arbitrary
|
||||
collection parameters. Text is transient in the Gateway payload store and never copied into
|
||||
ModelForge audit or metric labels.
|
||||
|
||||
Service clients have a fixed scheduler workload priority. The ExampleRAG reindex identity is scoped to
|
||||
`rag.embedding@1` and `background`, so the consumer cannot escalate bulk work to production.
|
||||
|
||||
M13 registers `examplerag.qdrant-reindex` as the first typed stateful migration contract. ExampleRAG owns
|
||||
chunk enumeration/hashes, Gateway embedding calls, Qdrant writes and alias inspection; ModelForge
|
||||
owns immutable intent, approval, policy, progress evidence and external-truth journaling. Isolated
|
||||
LAB aliases never grant the production client a new target. ExampleVision remains a blocked dry-run
|
||||
candidate while its external production validation is deferred.
|
||||
@@ -0,0 +1,23 @@
|
||||
# Promotion Plans
|
||||
|
||||
Approval establishes permission to propose a change. It does not activate anything.
|
||||
|
||||
`LifecyclePromotionPlan` binds the approval and subject to the exact candidate, desired state,
|
||||
rollback target, project consumers, affected embedding/pipeline/runtime/storage identities,
|
||||
migration class, canary/drain strategy, health gates and abort conditions. Its impact projection
|
||||
lists projects, clients, indexes, retrieval pipelines, runtime deployments, artifacts and scheduler
|
||||
envelope. Approval freezes these fields; changed intent requires a new plan.
|
||||
|
||||
The supported classes are `transparent`, `behavioral`, `requires_reindex` and `schema_breaking`.
|
||||
Behavioral changes require evaluation/canary evidence. `requires_reindex` requires a ready isolated
|
||||
migration and shadow/dual-read/evaluation-only route; M12 records but does not automatically execute
|
||||
the cutover. Schema-breaking plans require a manual runbook and are likewise not auto-executed.
|
||||
|
||||
Execution creates a `LifecycleOperation` journal and immutable rollback snapshot transactionally.
|
||||
Stages are `PREPARING`, `ACTIVATING`, `VERIFYING`, `COMMITTED`, `ROLLING_BACK`, `ROLLED_BACK` and
|
||||
`FAILED`. Requester, approver and executor are stored separately. A repeated idempotency key returns
|
||||
the original operation.
|
||||
|
||||
For M13 production reindex, the migration plan references this exact approved promotion plan and
|
||||
approval fingerprint. M12 still refuses to execute reindex as a direct deployment swap; the typed
|
||||
migration cutover journal is the only supported execution path.
|
||||
@@ -0,0 +1,21 @@
|
||||
# ExampleRAG integration
|
||||
|
||||
```text
|
||||
ExampleRAG operator workflow
|
||||
-> scoped ExampleRAG service credential
|
||||
-> POST /api/v1/capabilities/rag.embedding@1/invoke
|
||||
-> Gateway authentication and rate limits
|
||||
-> GPU scheduler and lease
|
||||
-> outbound GPU Node node agent
|
||||
-> typed offline runtime worker
|
||||
-> vector plus embedding_space_id
|
||||
-> isolated ExampleRAG shadow collection
|
||||
```
|
||||
|
||||
ExampleRAG configuration names only the ModelForge origin, credential-file boundary and
|
||||
`rag.embedding@1`. The adapter rejects redirects, public/cloud hosts, invalid credentials,
|
||||
non-finite vectors, wrong dimensions, non-normalized output and embedding-space mismatch. Safe
|
||||
transient statuses are retried with bounded backoff. There is no cloud or direct-worker fallback.
|
||||
|
||||
The credential is read for every batch, allowing create → secret-file replacement → validate →
|
||||
revoke rotation without logging or persisting the secret in either repository.
|
||||
@@ -0,0 +1,65 @@
|
||||
# Recovery asset classification
|
||||
|
||||
Recovery starts with an inventory, not with a backup job. Every persisted state source in
|
||||
ModelForge is classified once, and the classification decides whether the state is copied,
|
||||
rebuilt, discarded, or explicitly disowned.
|
||||
|
||||
## Classes
|
||||
|
||||
```text
|
||||
AUTHORITATIVE exists nowhere else; must be copied and provably restorable
|
||||
REBUILDABLE reproducible from an exact identity; a manifest is enough
|
||||
EPHEMERAL describes the present, not history; must be re-measured after recovery
|
||||
EXTERNAL owned by another project; ModelForge records the dependency and never writes
|
||||
SECRET credential material; hashes restore, plaintext does not
|
||||
```
|
||||
|
||||
A class is not a label on a document: `RecoveryPolicyRevision` refuses incoherent combinations.
|
||||
Authoritative state cannot declare `NOT_BACKED_UP` or `MANIFEST_ONLY`, and cannot omit an RPO.
|
||||
Ephemeral state cannot claim a payload backup. External state must be marked as an external
|
||||
dependency. Secret state must name a secret recovery class. Only rebuildable state may be
|
||||
rehydrated instead of copied.
|
||||
|
||||
## The inventory
|
||||
|
||||
`recovery_asset_records` holds one row per state source with its owner, location, backup method,
|
||||
restore method, rebuild method, RPO, dependencies and readiness. The seeded inventory is:
|
||||
|
||||
| Asset | Class | Backup | Recovery |
|
||||
| --- | --- | --- | --- |
|
||||
| `postgres.modelforge` | AUTHORITATIVE | `pg_dump --format=custom` | `pg_restore` into a fresh database |
|
||||
| `artifacts.derived` | AUTHORITATIVE | file copy | payload restore preserving derived lineage |
|
||||
| `artifacts.huggingface` | REBUILDABLE | manifest only | exact-revision redownload, quarantine, verify, promote |
|
||||
| `configuration.repository` | REBUILDABLE | source-control reference | checkout of the recorded commit |
|
||||
| `backup.destination` | REBUILDABLE | source-control reference | operator-provided storage |
|
||||
| `redis.queues` | EPHEMERAL | not backed up | queues start empty; leases reconciled |
|
||||
| `runtime.residency` | EPHEMERAL | not backed up | cleared on restore, re-measured from live GPU Node state |
|
||||
| `node.gpu_node` | SECRET | hashes only | restored hash stays valid, or revoke and re-enrol |
|
||||
| `credentials.project` | SECRET | hashes only | rotate; plaintext is unrecoverable |
|
||||
| `credentials.operator` | SECRET | never in a backup | operator-supplied break-glass |
|
||||
| `external.examplerag` | EXTERNAL | not backed up | owned by ExampleRAG |
|
||||
| `external.examplevision` | EXTERNAL | not backed up | owned by ExampleVision |
|
||||
| `external.huggingface` | EXTERNAL | not backed up | upstream availability only |
|
||||
| `external.gitea` | EXTERNAL | not backed up | platform operations |
|
||||
|
||||
## History versus current truth
|
||||
|
||||
The same distinction runs through the database itself. `recovery_fingerprint.py` partitions every
|
||||
persisted table into fingerprinted history and `CURRENT_TRUTH_TABLES`, and a test asserts the two
|
||||
sets together cover the entire schema — a new milestone cannot add a table that is silently
|
||||
neither.
|
||||
|
||||
Current truth is `host_telemetry_latest`, `accelerator_telemetry_latest`, `storage_volume_states`,
|
||||
`hardware_inventory_runs`, `scheduler_accelerator_states`, `residency_allocations`,
|
||||
`serving_gpu_leases` and `gpu_leases`. These are cleared during recovery reconciliation and
|
||||
re-derived by a running control plane. Restoring them would present a stale NVML reading or a dead
|
||||
GPU lease as a present-day fact, and the restore gate refuses to reach `READY` while any of them
|
||||
still holds a row.
|
||||
|
||||
## Readiness
|
||||
|
||||
Each asset reports one of `PROTECTED`, `REHYDRATABLE`, `ROTATION_REQUIRED`,
|
||||
`EXTERNAL_DEPENDENCY` or `UNPROTECTED`. Authoritative assets are reported `UNPROTECTED` whenever no
|
||||
verified backup exists, regardless of what the policy claims: readiness is measured, not declared.
|
||||
|
||||
See `docs/architecture/BACKUP_AND_RECOVERY.md` for the backup contract and ADR-0041.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Recovery compatibility
|
||||
|
||||
M17 will use this matrix for upgrades; M15 establishes it and proves the fail-closed directions.
|
||||
|
||||
## Backup schema versus ModelForge build
|
||||
|
||||
| Backup Alembic revision | Restoring build | Result |
|
||||
| --- | --- | --- |
|
||||
| equals the build's head | same or newer build | restored as is |
|
||||
| older, present in the build's chain | newer build | restored, then migrated forward to head |
|
||||
| absent from the build's chain | older build | `SCHEMA_TOO_NEW` — refused |
|
||||
| recorded as none | any build | preflight fails; the backup is not restore eligible |
|
||||
| any | build shipping no Alembic chain | `SCHEMA_UNKNOWN` — refused |
|
||||
|
||||
The chain is walked by `down_revision`, not by filename order, so a revision that sorts earlier but
|
||||
descends later is still placed correctly. Discovery resolves the source checkout, the installed
|
||||
image and an explicitly configured `MODELFORGE_ALEMBIC_DIRECTORY`.
|
||||
|
||||
A newer backup is never silently downgraded onto older code. Restoring an older backup onto a newer
|
||||
build runs the forward migrations that build ships and nothing else.
|
||||
|
||||
## PostgreSQL
|
||||
|
||||
| Source major | Destination major | Result |
|
||||
| --- | --- | --- |
|
||||
| 17 | 17 | supported |
|
||||
| 17 | 18+ | permitted; preflight records both majors |
|
||||
| 17 | 16 or lower | `POSTGRES_VERSION_INCOMPATIBLE` |
|
||||
|
||||
The control-plane image pins its client tools to the server major version. `pg_dump` from an older
|
||||
major cannot read a newer server, which is why the tools ship with the control plane rather than
|
||||
being inherited from the host.
|
||||
|
||||
## Manifest schema
|
||||
|
||||
`manifest_schema_version` is `m15.1`. Verification requires the manifest file and the journaled
|
||||
`backup_manifest_entries` to agree exactly, so a manifest written by a different schema version is
|
||||
rejected rather than partially interpreted.
|
||||
|
||||
## Semantic fingerprint
|
||||
|
||||
`FINGERPRINT_VERSION` is `m15.2`; `m15.2` adds the schema-0024 `audit_chain_heads` singleton to the
|
||||
audit group. Current schema-0024 fingerprints are comparable only when both sides are `m15.2` and
|
||||
both contain exactly one complete checkpoint row. Both fingerprints must enumerate the complete
|
||||
authoritative table contract: a missing expected table, an unexpected table, an absent/sparse
|
||||
entry or a malformed count/digest is incompatible. After reconciliation, every non-current-truth
|
||||
table must retain its exact count and digest except the enumerated mutations: one canonical audit
|
||||
marker, the corresponding singleton-head advance, and same-count suppression changes in
|
||||
`operational_alerts`.
|
||||
|
||||
Every fingerprint also records the exact physical table-name set, including current-truth tables
|
||||
whose rows are intentionally excluded. Dropping `host_telemetry_latest`, adding an attacker table,
|
||||
or omitting any authoritative table is therefore incompatible. Table digests and group/overall
|
||||
digests use typed, length-prefixed cells and rows; separators, nulls and different scalar types
|
||||
cannot alias. A table exceeding the 250,000-row cap is marked `BOUNDED`; that is useful diagnostic
|
||||
evidence but can never authorize READY because it is not a complete count-and-digest proof.
|
||||
|
||||
There is one explicit cross-version rule: a backup whose recorded Alembic revision is exactly
|
||||
`20260828_0022` and whose fingerprint is `m15.1` may be compared with a restored/migrated `m15.2`
|
||||
fingerprint when the old side has no checkpoint and the new side has exactly one. Strict audit-chain
|
||||
verification must pass independently. The transition still permits exactly one canonical audit
|
||||
marker and no other digest/count change (apart from same-count alert suppression). No other absent
|
||||
or introduced table, fingerprint-version pair or schema revision receives that allowance.
|
||||
Auth migration `20260830_0023` adds constraints only, so it introduces no physical-table exception
|
||||
inside this schema-0022-to-0024 comparison.
|
||||
|
||||
## Encryption
|
||||
|
||||
`MFBK1` frames the encrypted payload with the key id it was sealed under. A payload sealed by a
|
||||
different key id is still authenticated by GCM and fails closed. The algorithm is recorded per
|
||||
backup set, so introducing a second algorithm later does not invalidate existing backups.
|
||||
|
||||
## Recovery contract
|
||||
|
||||
Recovery API responses never carry a DSN with its password, a credential hash, or an enrolment
|
||||
token. `RestorePlanResponse` exposes `database_destination_redacted` and has no
|
||||
`database_destination` field at all, which is asserted against the published OpenAPI document.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Reindex migrations
|
||||
|
||||
`REQUIRES_REINDEX` always names distinct source and shadow targets and a distinct target
|
||||
`EmbeddingSpace`. Direct deployment swaps and same-index vector mixing fail validation.
|
||||
|
||||
Preflight proves the frozen source fingerprint/count, target absence, space and capability health,
|
||||
credential, storage, scheduler capacity, adapter availability, rollback retention, evaluation suite
|
||||
and current approval. Backfill priority is `BACKGROUND`; batches, concurrency and in-flight work are
|
||||
bounded. Each checkpoint records cursor, generation, deterministic item/result fingerprints,
|
||||
completed/retryable/permanent counts, bounded errors and duration.
|
||||
|
||||
A retry replaces the prior attempt's counters rather than double-counting them. Completed matching
|
||||
batches are idempotent. Old versions/generations and changed item fingerprints fail closed. A batch
|
||||
is complete only after finite shape/space validation, committed destination write and matching
|
||||
content hashes. Cancellation and pause preserve the shadow for retention-aware resume or cleanup.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Reranking Capability
|
||||
|
||||
`rag.reranking@1` is a first-class capability, never a public model or worker endpoint. A caller
|
||||
submits one query, 1–40 bounded `{id,text}` documents and `top_n` 1–10. The response contains only
|
||||
stable IDs, finite scores and ranks. ModelForge resolves the experiment deployment, acquires a GPU
|
||||
lease and dispatches to the private GPU Node runtime worker.
|
||||
|
||||
Query and document text exist only in the bounded Redis payload exchange and worker memory. They
|
||||
are deleted after completion or timeout and never enter PostgreSQL, audit payloads, metrics or
|
||||
request history. IDs, timings, counts and failure codes may be persisted. There is no network
|
||||
egress, cloud fallback or `trust_remote_code` path.
|
||||
|
||||
The M8 implementation supports the Qwen3 causal-LM reranker adapter with immutable runtime-profile,
|
||||
artifact-set, exact-revision and probe evidence. Typed failures include `RERANKER_NOT_AVAILABLE`,
|
||||
`RERANK_REQUEST_TOO_LARGE`, queue/capacity failures and invalid/non-finite output.
|
||||
|
||||
The M8 deployment is experiment-only. No stable route or production promotion was created.
|
||||
@@ -0,0 +1,10 @@
|
||||
# Reranking evaluation
|
||||
|
||||
M7 completed embedding comparison before considering reranking. No embedding candidate passed the
|
||||
critical-regression gate, and live external GPU use left a constrained single-GPU window. Adding a
|
||||
reranker and a second runtime variable would therefore not strengthen the M7 embedding decision.
|
||||
|
||||
Reranking is deferred. A later study must use `rag.reranking@1`, freeze the same initial candidate
|
||||
pool, bound transient chunk-text input, keep inference offline, compare before/after quality and
|
||||
latency, and create no production cutover automatically.
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
# GPU Resource Accounting
|
||||
|
||||
M10 treats NVML as physical truth while retaining ModelForge attribution as a separate logical
|
||||
ledger. A resident model is already present in `observed_used_vram_bytes` and must therefore never
|
||||
be subtracted a second time.
|
||||
|
||||
```text
|
||||
physical_used = max(observed_nvml_used, attributed_managed_resident)
|
||||
unmanaged_external = max(0, observed_nvml_used - attributed_managed_resident)
|
||||
future_lease = max(0, reserved_bytes - materialized_bytes)
|
||||
schedulable = max(0, total - physical_used - future_leases - dynamic_reserve)
|
||||
```
|
||||
|
||||
Every term is non-negative. The exposed invariant delta is
|
||||
`observed - managed_resident - unmanaged_external`; it diagnoses sampling or attribution drift but
|
||||
is never converted into capacity. `KNOWN`, `ESTIMATED`, and `UNKNOWN` describe attribution quality.
|
||||
Unknown or stale telemetry fails closed with zero schedulable capacity; it never means zero use.
|
||||
|
||||
The versioned `m10-v1` reserve is the maximum of 1 GiB, 5% of total VRAM, and a 256 MiB runtime
|
||||
margin. A deployment requires its measured peak plus the maximum of 128 MiB or 10% of that peak.
|
||||
Only unmaterialized leases reduce future headroom. Accelerator-row locking makes the database the
|
||||
coordination boundary for concurrent admissions.
|
||||
|
||||
Resource envelopes remain bound to the exact artifact, runtime profile, accelerator and environment
|
||||
fingerprint. A stale or mismatched heavy envelope cannot authorize placement. Persistent history is
|
||||
bounded placement evidence, not a one-second telemetry archive.
|
||||
@@ -0,0 +1,14 @@
|
||||
# Resource classes
|
||||
|
||||
Resource classes provide an operator-facing scheduling expectation; measured `CapabilityResourceEnvelope` evidence remains authoritative.
|
||||
|
||||
| Class | Meaning |
|
||||
| --- | --- |
|
||||
| LIGHT | Usually CPU or a small GPU footprint; may coexist when measured budgets allow. |
|
||||
| MEDIUM | Material GPU residency; serialized when current envelopes require it. |
|
||||
| HEAVY | Requires an explicit measured envelope and conservative shared-GPU admission. |
|
||||
| EXCLUSIVE_GPU | Intended to occupy the schedulable GPU budget alone. |
|
||||
|
||||
Admission uses current accelerator telemetry minus external/unmanaged use, active ModelForge residents and leases, and the fixed safety reserve. It rejects unsafe work with `INSUFFICIENT_SCHEDULABLE_VRAM` instead of attempting an OOM. The label never overrides the byte budget.
|
||||
|
||||
GPU Node remains a single shared NVIDIA GPU. Ollama, Tdarr and Plex are observed as external pressure only; ModelForge does not stop, restart or reconfigure them. Load, invoke and unload jobs share the existing database-backed lease and residency coordinator, so OCR, Whisper, image generation and assistants cannot bypass contention controls.
|
||||
@@ -0,0 +1,18 @@
|
||||
# Retention and Deprecation
|
||||
|
||||
Deprecation removes an object from ordinary selection; it does not delete bytes or history.
|
||||
Supersession is an explicit reference on the lifecycle subject. Archive means no active deployment
|
||||
and hidden normal selection while identities, hashes, plans, evaluation and audit remain queryable.
|
||||
|
||||
`RetentionPolicyRevision` is versioned and immutable. The default `standard-rollback` revision
|
||||
retains the previous stable target for at least 30 days after commit. A retention record distinguishes
|
||||
`ACTIVE`, `ROLLBACK_RETAINED`, `DEPRECATED`, `ARCHIVABLE` and `ARCHIVED`; legal hold and unexpired
|
||||
rollback state block cleanup. Tests use controlled timestamps to evaluate expiry without waiting.
|
||||
|
||||
Physical removal is always a separate cleanup operation. Archiving metadata does not imply removing
|
||||
model weights, and deleting one node-local location does not mark an artifact globally missing while
|
||||
another verified location remains.
|
||||
|
||||
M13 captures a migration rollback snapshot before external switch and applies the plan's minimum
|
||||
30-day retention. Active migrations and unexpired rollback snapshots are cleanup blockers; orphan
|
||||
shadow targets become explicit later cleanup candidates and are never auto-deleted.
|
||||
@@ -0,0 +1,11 @@
|
||||
# Retrieval-aware embeddings
|
||||
|
||||
Query/document intent is an explicit `input_type` on the generic `rag.embedding@1` request. Concrete
|
||||
instructions remain in the immutable RuntimeProfile; clients never receive model paths or prompt
|
||||
syntax. Qwen3-Embedding-0.6B uses the exact-revision upstream query instruction and an empty document
|
||||
prefix. The instruction source and revision are part of EmbeddingSpace identity.
|
||||
|
||||
Changing instruction, pooling, normalization, dimension or another behavior-affecting setting creates
|
||||
a new RuntimeProfile and EmbeddingSpace and requires a separate reindex. The M6 raw profile and space
|
||||
remain immutable.
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
# Retrieval Candidate Pools
|
||||
|
||||
A `RetrievalCandidatePool` freezes the output of first-stage retrieval before reranking. Its
|
||||
identity binds the project, immutable suite case, source embedding space/index, corpus revision,
|
||||
retrieval configuration digest and ordered candidate IDs/scores/content hashes. Text is excluded.
|
||||
|
||||
M8 uses a top-40 ceiling. Thirty-two cases contain 40 unique in-scope candidates; one scoped case
|
||||
contains 39. ModelForge records the actual count and never pads the pool with out-of-scope data.
|
||||
Control and reranked results must reference the same pool IDs. Corpus/config/pool mismatch is a
|
||||
typed comparability failure.
|
||||
|
||||
The pool is immutable after creation. Reranking therefore cannot benefit from a second dense or
|
||||
sparse retrieval run, and it cannot recover a relevant item absent from the frozen pool.
|
||||
@@ -0,0 +1,16 @@
|
||||
# Retrieval Pipeline Identity
|
||||
|
||||
A retrieval pipeline is behaviorally identified by:
|
||||
|
||||
- embedding-space identity;
|
||||
- sparse and fusion configuration digests;
|
||||
- optional reranker deployment and reranker configuration digest;
|
||||
- candidate-k and output-k semantics.
|
||||
|
||||
The canonical identity digest covers every field. An embedding-only control and the same embedding
|
||||
plus reranker are distinct pipelines even when dimensions and source collection are equal. A
|
||||
reranker profile/configuration change also creates a new identity. Reranker addition is classified
|
||||
`behavioral`; an embedding-space change remains `requires_reindex`.
|
||||
|
||||
Pipeline comparisons consume immutable fixed-pool runs. They do not overwrite embedding evaluation
|
||||
runs and cannot be compared as though they were embedding deployments.
|
||||
@@ -0,0 +1,16 @@
|
||||
# Rollback
|
||||
|
||||
Every executable plan names a rollback target before execution. The operation captures an immutable
|
||||
snapshot of subject identity/state/version, environment, affected identities and rollback reference.
|
||||
The same pattern can represent deployment, project binding, runtime/config and future index-alias
|
||||
restoration without mutating provenance records.
|
||||
|
||||
Rollback is explicit or triggered by canary/restart reconciliation. It restores the exact prior
|
||||
snapshot and records elapsed wall-clock time from rollback start until the prior state is restored;
|
||||
the result is evidence, not an invented SLO. Retrying an already rolled-back operation is
|
||||
idempotent.
|
||||
|
||||
If the snapshot/target is unavailable, the operation becomes `FAILED` with
|
||||
`ROLLBACK_FAILED` and the subject becomes `MANUAL_INTERVENTION_REQUIRED`. ModelForge never reports
|
||||
stable after incomplete restoration. Startup reconciliation conservatively rolls back operations
|
||||
left in `PREPARING`, `ACTIVATING` or `ROLLING_BACK`.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Runtime Compatibility
|
||||
|
||||
## Evidence model
|
||||
|
||||
`RuntimeCompatibilityAssessment` records the exact ArtifactSet, RuntimeProfile, ComputeNode,
|
||||
adapter/version, artifact and hardware snapshots, blockers, warnings, approvals, static result and a
|
||||
canonical environment fingerprint. It distinguishes runtime-declared support from locally proven
|
||||
support. Static analysis never claims a model was loaded.
|
||||
|
||||
## Common checks
|
||||
|
||||
- complete, locally available and integrity-verified exact ArtifactSet;
|
||||
- no quarantined/corrupt member, unapproved pickle or remote-code requirement;
|
||||
- all registered artifact locations belong to the target node;
|
||||
- online, enabled and lab-eligible node advertising `runtime.probe.v1`;
|
||||
- GPU presence for CUDA-required profiles, compute capability, driver/CUDA compatibility and VRAM;
|
||||
- environment package, modality and serialization-format declarations;
|
||||
- required model configuration/tokenizer/processor files and shard-index presence;
|
||||
- requested quantization evidence.
|
||||
|
||||
Resource estimates are stored as `kind=estimated`; only a real probe produces `kind=measured`.
|
||||
|
||||
## Runtime-specific inspectors
|
||||
|
||||
- SentenceTransformers requires safe Transformers files plus `modules.json` and pooling config.
|
||||
- Transformers requires config, tokenizer and Safetensors weights.
|
||||
- vLLM uses Transformers-format checks but remains `requires_probe` until proven for the task.
|
||||
- llama.cpp requires a local GGUF variant; Safetensors alone yields `NO_GGUF_ARTIFACT_VARIANT`.
|
||||
- Diffusers requires a pipeline `model_index.json`; an embedding repository is blocked.
|
||||
- custom runners never receive an implicit compatibility claim.
|
||||
|
||||
## Staleness
|
||||
|
||||
Every read recomputes the canonical fingerprint from current immutable profile/environment identity,
|
||||
artifact evidence and hardware facts. Driver, CUDA, GPU, node state, artifact security/integrity,
|
||||
image or profile changes make prior evidence stale. A stale assessment cannot queue a probe.
|
||||
@@ -0,0 +1,80 @@
|
||||
# Runtime Plane
|
||||
|
||||
## M9 multimodal extension
|
||||
|
||||
The runtime plane now has specialized `transformers_trocr`, `transformers_siglip2`, and `transformers_whisper` adapters. They use exact local artifacts, `local_files_only`, `trust_remote_code=false`, denied network access and typed modality payloads. Runtime environments declare supported modality/model types; RuntimeProfiles bind preprocessing/audio semantics; probes and serving record the same measured resource evidence and unload behavior as embedding/reranking workers.
|
||||
|
||||
## Scope
|
||||
|
||||
M4 turns verified local bytes into reproducible technical execution evidence. M5 consumes only that
|
||||
evidence through typed deployment load/invoke/health/drain/unload jobs. It does not expose an
|
||||
inference service, schedule workloads, bind projects, benchmark quality, or activate production.
|
||||
|
||||
```text
|
||||
ArtifactSet -> CompatibilityAssessment -> RuntimeProfile -> RuntimeProbe -> DeploymentCandidate
|
||||
```
|
||||
|
||||
Each arrow is a separately persisted gate. `DeploymentCandidate.status=lab_ready` and
|
||||
`production=false` are the terminal M4 state; a separate exact M5 production approval and
|
||||
CapabilityDeployment are required before routing.
|
||||
|
||||
The runtime worker remains outbound-only and private. M10 replaces its historical single resident
|
||||
slot with a bounded map keyed by deployment UUID. Load, invoke, health, drain and unload target one
|
||||
exact deployment and do not implicitly clear other proven residents. The worker reports a complete
|
||||
typed inventory including ArtifactSet, RuntimeProfile, environment fingerprint and 64-bit worker
|
||||
generation. CUDA allocator measurements prove process-owned model allocation/reclaim; NVML remains
|
||||
the total-GPU/external-pressure source.
|
||||
|
||||
Control-plane restart adopts only an exact inventory match. Worker absence reconciles database state
|
||||
to cold; an exact worker inventory may be adopted without duplicate load. Unknown GPU processes are
|
||||
never adopted and remain unmanaged external usage.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- The control plane owns evidence, approvals, typed leases, state transitions and audit events.
|
||||
- A static Compose runtime worker on the selected node polls outbound with that node's credential.
|
||||
- The worker receives a fixed probe type, exact registered relative artifact location, manifest,
|
||||
immutable profile and environment identity. It never receives a shell command.
|
||||
- A child process owns the ML runtime and exits after the probe, making process death the final
|
||||
unload boundary. The parent remains responsive to cancellation and reports NVML evidence.
|
||||
- Artifacts mount read-only; root filesystem is read-only; `/tmp` is the only writable runtime
|
||||
filesystem; the worker is non-root, has no published ports, Docker socket, SSH material or host-root
|
||||
mount, and drops Linux capabilities.
|
||||
|
||||
## Adapter lifecycle
|
||||
|
||||
`RuntimeAdapter` defines `inspect_support`, `prepare`, `start`, `probe`, `health`, `stop`, `unload`
|
||||
and `collect_runtime_facts`. M4 executes only the SentenceTransformers adapter. Static inspectors
|
||||
exist for Transformers, vLLM, llama.cpp, Diffusers and custom runners. Unsupported executable
|
||||
adapters fail with `RUNTIME_INCOMPATIBLE` rather than falling through to arbitrary code.
|
||||
|
||||
## Lifecycle and recovery
|
||||
|
||||
Probes move through `queued`, `preparing`, `loading`, `healthchecking`, `ready`, `unloading`, then
|
||||
`completed`, `failed` or `cancelled`. Node-scoped leases are hashed at rest and expire. An expired
|
||||
active lease becomes claimable by the same node, increments `attempt_count`, and gets a new token.
|
||||
No active probe is represented as a deployment candidate.
|
||||
|
||||
## Health
|
||||
|
||||
The worker reports distinct process, runtime, model and capability layers. M4 requires healthy
|
||||
process/runtime/model plus a finite non-empty vector, offline-local evidence and successful VRAM
|
||||
reclamation. Capability health remains `not_routed_in_m4` by design.
|
||||
## M14 runtime observability
|
||||
|
||||
Runtime operational truth remains the typed `ServingJob`, health result, GPU lease and measured
|
||||
residency records. M14 exports bounded operation/status counters, evaluates load/invoke/unload/health
|
||||
success, detects a bounded crash loop and correlates capability degradation without sending commands
|
||||
to workers. Observability persistence failure cannot block or silently reroute inference. See
|
||||
`docs/architecture/OBSERVABILITY.md` and `docs/operations/RUNBOOK_RUNTIME_FAILURE.md`.
|
||||
|
||||
|
||||
## M16 fault behaviour
|
||||
|
||||
A Runtime Worker failure during load, inference or unload produces a typed failure for the request,
|
||||
fences the worker generation so a late result from a dead generation cannot be committed, and
|
||||
reconciles the lease and residency. A repeated crash raises `RUNTIME_CRASH_LOOP`.
|
||||
|
||||
The `no_orphan_serving_work` invariant asserts that no queued or leased job outlives its lease and
|
||||
that no residency allocation sits on a disabled node, which is what "the worker failure was
|
||||
reconciled" has to mean in practice.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Runtime Profiles
|
||||
|
||||
A RuntimeEnvironment identifies installed executable software: adapter, exact runtime and package
|
||||
versions, Python/CUDA versions, supported architectures/formats/modalities and container repository
|
||||
plus digest. It is immutable after registration.
|
||||
|
||||
A RuntimeProfile binds one exact ArtifactSet to one RuntimeEnvironment and execution configuration:
|
||||
dtype, quantization, modality, sequence length, batch, concurrency, device/GPU-memory policies,
|
||||
non-secret environment values, launch parameters and layered health contract. Its canonical
|
||||
fingerprint makes identical creates idempotent. Any behavior-changing update is a new profile,
|
||||
never an in-place mutation.
|
||||
|
||||
`trust_remote_code=false` and `network_egress=false` are literal schema constraints. Environment
|
||||
variable names containing token, secret, password, key or credential are rejected. Runtime Profiles
|
||||
contain configuration only; node credentials and hub/project secrets are not part of the object.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Scheduler QoS
|
||||
|
||||
The fixed safety order is production, interactive, background, LAB. Database row locking on the
|
||||
accelerator remains authoritative for capacity, while serving jobs use row-lock/skip-locked claims.
|
||||
Five-minute bounded aging moves old background/LAB work forward without overtaking production.
|
||||
|
||||
Backpressure is enforced at three boundaries: per-client concurrency/rate, per-capability queue and
|
||||
global queue. A full boundary returns `QUEUE_FULL`; queues never grow without limit. Cold requests
|
||||
whose measured load time exceeds an explicit remaining deadline return `DEADLINE_CANNOT_BE_MET`.
|
||||
|
||||
Normal scheduling never interrupts active inference. A higher-priority request may evict only an
|
||||
idle, lower/equal-priority managed residency or wait. Batch metadata is represented without making
|
||||
unsafe batching promises. Stored request shape is limited to count, bounded size bucket and
|
||||
modality-relevant numeric characteristics; text, image and audio content is not scheduler evidence.
|
||||
|
||||
The admin metrics surface uses bounded names and histories: admissions, rejections, queue seconds,
|
||||
evictions, resident/leased/external/schedulable bytes, pressure, cold loads, warm hits, unloads and
|
||||
reclaim failures. Request IDs are not metric labels and ordinary placements are not audit events.
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
# Schema-breaking migrations
|
||||
|
||||
M13 models schema-breaking intent, typed adapter steps and an explicit irreversible marker. It does
|
||||
not execute user SQL, Python, shell or paths. Production planning requires the stronger M12 approval
|
||||
boundary and explicit rollback/irreversibility facts; automatic cutover is denied.
|
||||
|
||||
Known future adapters may model expand, dual-compatible operation, migrate, verify, cutover and
|
||||
contract. Each step must be compiled into trusted application code and versioned in the adapter
|
||||
fingerprint. M13 deliberately makes no generic rollback claim for irreversible work.
|
||||
@@ -0,0 +1,37 @@
|
||||
# Service-level indicators
|
||||
|
||||
An SLI is a first-class typed definition of a service or capability, its measurement, valid
|
||||
population, success condition and default window. An SLI does not contain an objective; objectives
|
||||
belong to versioned SLO policies.
|
||||
|
||||
## Core definitions
|
||||
|
||||
| Key | Population | Successful observation |
|
||||
| --- | --- | --- |
|
||||
| `platform.api.availability` | control-plane and health requests since process start | 2xx/3xx |
|
||||
| `gateway.availability` | authenticated gateway requests in window | completed |
|
||||
| `gateway.latency.p95` | completed gateway requests with latency | p95 below policy threshold |
|
||||
| `scheduler.correctness` | persisted placement plans | typed admission, queue or refusal verdict |
|
||||
| `node.freshness` | enabled production nodes | heartbeat age at most 90 seconds |
|
||||
| `runtime.success` | terminal load/invoke/unload/health jobs | completed |
|
||||
| `migration.reliability` | terminal migration plans | not failed/manual intervention |
|
||||
|
||||
Capability definitions cover RAG success and latency plus Vision and Speech LAB success. The latter
|
||||
remain explicitly LAB. OCR has no quality or production SLO because M9/M11 quality and external
|
||||
evidence gates remain unsatisfied.
|
||||
|
||||
Project-facing health is derived only for actual scoped service clients and bindings. It reports
|
||||
project, capability, environment, client state, request/error counts and last use. ExampleRAG's
|
||||
production Ollama baseline is external to ModelForge and is not relabelled as a ModelForge
|
||||
production SLI. ExampleVision Vision shadow and ASR remain LAB.
|
||||
|
||||
## Evaluation semantics
|
||||
|
||||
The evaluator selects only events inside the policy rolling window. It persists population, good
|
||||
and bad counts, observed value, freshness, policy identity and bounded source evidence. Latency
|
||||
evidence retains p50/p95/p99, while the initial policy evaluates p95.
|
||||
|
||||
Possible states are `HEALTHY`, `AT_RISK`, `BREACHED`, `INSUFFICIENT_DATA`, `STALE` and `DISABLED`.
|
||||
The minimum population is evaluated before health. No samples or too few samples can therefore
|
||||
never become green. Control-plane health, capability health and project integration health remain
|
||||
separate dimensions.
|
||||
@@ -0,0 +1,9 @@
|
||||
# Shadow and dual read
|
||||
|
||||
Shadow targets are separate storage identities and are exposed only to evaluation/admin or an
|
||||
explicit project shadow route. Ordinary production consumers continue using the current binding.
|
||||
Compatible adapters may execute one request against current and shadow, but persist only result IDs,
|
||||
metrics, latency, errors and evidence references—not content.
|
||||
|
||||
Shadow completion records a `MigrationShadowSession`. Success reaches `READY_FOR_CUTOVER`; it never
|
||||
switches traffic. An isolated technical verdict and project-promotion verdict remain distinct.
|
||||
@@ -0,0 +1,24 @@
|
||||
# SLO policies
|
||||
|
||||
SLO policies are immutable revisions. Changing objective, threshold, window, population minimum,
|
||||
severity or environment deactivates the prior revision and creates a new fingerprinted revision.
|
||||
Historic evaluations keep the exact prior policy ID.
|
||||
|
||||
## Initial active policies
|
||||
|
||||
| Key | Environment | Objective/threshold | Window | Minimum | Rationale |
|
||||
| --- | --- | --- | ---: | ---: | --- |
|
||||
| `platform.api.production` | PRODUCTION | 99% | 24h | 20 | local control-plane availability |
|
||||
| `gateway.production` | PRODUCTION | 99% | 24h | 20 | valid capability routing |
|
||||
| `rag.embedding.production.success` | PRODUCTION | 99% | 24h | 20 | established stable capability |
|
||||
| `rag.embedding.production.latency` | PRODUCTION | p95 ≤ 1000 ms | 24h | 20 | measured warm/shared-GPU envelope, not a quality gate |
|
||||
| `gpu_node.production.freshness` | PRODUCTION | 99%, max age 90 s | 1h | 1 | existing 30/90 receive-time contract |
|
||||
| `scheduler.production.correctness` | PRODUCTION | 99.5% | 24h | 20 | typed refusal is valid; internal failure is not |
|
||||
| `migration.production.reliability` | PRODUCTION | 99% | 7d | 1 | failure/manual intervention consumes budget |
|
||||
| `vision.embedding.lab` | LAB | 95% | 24h | 5 | operational LAB signal only |
|
||||
| `speech.transcription.lab` | LAB | 95% | 24h | 5 | operational LAB signal only |
|
||||
|
||||
Thresholds use the established node liveness contract and M5–M13 measured behavior. They are not
|
||||
selected after a failure merely to make the platform green. LAB policy evaluation conveys operating
|
||||
evidence without production readiness or an availability budget. `BACKGROUND` is supported by the
|
||||
contract for migration/benchmark work but has no initial production objective.
|
||||
@@ -0,0 +1,21 @@
|
||||
# Speech consumer integration
|
||||
|
||||
ExampleVision spoken card search is the first operational consumer of `speech.transcription@1`. It is a
|
||||
small, useful local flow adjacent to the scanner and does not require persisted audio. The dedicated
|
||||
interactive client is bound only to the speech contract; ExampleVision knows no Whisper model, revision,
|
||||
runtime, worker or node.
|
||||
|
||||
```text
|
||||
ExampleVision WAV upload -> transient consumer buffer -> ModelForge Gateway -> M10 scheduler
|
||||
-> GPU Node offline speech runtime -> text -> ExampleVision response
|
||||
```
|
||||
|
||||
The consumer rejects unsupported or oversized input, removes temporary audio after the call, exposes
|
||||
typed unavailable/auth/capacity failures and has no cloud fallback. ModelForge telemetry stores
|
||||
dimensions, timing and state, never audio.
|
||||
|
||||
Live acceptance used four representative NL/EN Windows TTS samples. All four completed through the
|
||||
consumer and Gateway with zero errors; consumer p50/p95 latency was 319.622/393.442 ms and the small
|
||||
suite's normalized WER was 0.380952. This proves the integration but not natural-microphone quality,
|
||||
so the project recommendation remains `KEEP_LAB`.
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
# speech.transcription@1
|
||||
|
||||
`speech.transcription@1` accepts canonical base64 PCM WAV, at most 16 MiB and 120 seconds. Language may be explicit or automatic. The response contains transcription, optional detected language, ordered timestamp segments where supported, duration and processing metadata.
|
||||
|
||||
Audio is transient: it is excluded from database records, audit events, metrics and logs; it is deleted from Redis after completion; inference is offline; and no cloud fallback exists.
|
||||
|
||||
The M9 runtime adapter uses the exact verified Whisper Large v3 Turbo artifact with `AutoModelForSpeechSeq2Seq`. WAV decoding uses the Python standard library, bounded PCM validation and deterministic resampling. Repository code and runtime network egress remain disabled.
|
||||
|
||||
ASR evaluation stores fixture digests and ground truth rather than audio. Typed metrics are WER, latency, real-time factor and peak VRAM. Dutch and English are required where locally generated or licensed fixtures are available; absent fixtures are reported rather than synthesized into false evidence.
|
||||
@@ -0,0 +1,93 @@
|
||||
# System Architecture
|
||||
|
||||
## 1. Product boundary
|
||||
|
||||
ModelForge is the control plane for local AI inference infrastructure. It is responsible for model discovery, provenance, verification, lifecycle, runtime configuration, resource scheduling, project bindings, capability routing, experiments, benchmarks, upgrade advice and auditability.
|
||||
|
||||
It is **not** a model-training cluster, a public AI cloud, a generic chat application, or an orchestration abstraction that hides all operational details.
|
||||
|
||||
## 2. Logical layers
|
||||
|
||||
### Web UI
|
||||
|
||||
Operator-facing control surface. It presents state and evidence but must not invent state independently of the API.
|
||||
|
||||
### Control Plane API
|
||||
|
||||
Canonical source of truth for registry state, projects, capability contracts, runtime profiles, benchmark evidence, recommendations, lifecycle actions and audit events.
|
||||
|
||||
### AI Gateway
|
||||
|
||||
Stable project-facing surface. Resolves project identity + capability + channel to a concrete deployment and enforces policy, queueing and observability.
|
||||
|
||||
### Scheduler
|
||||
|
||||
Controls finite accelerator resources. It handles VRAM leases, residency, load/sleep/unload, priority classes, compatibility constraints and fallback policy.
|
||||
|
||||
### Runtime adapters
|
||||
|
||||
Each runtime implements a stable internal contract. Initial adapters:
|
||||
|
||||
- vLLM — LLM/VLM/embedding/reranking/ASR where appropriate
|
||||
- Transformers — specialized Hugging Face inference
|
||||
- Diffusers — image generation/editing
|
||||
- llama.cpp — GGUF and low-memory fallback
|
||||
|
||||
### Artifact pipeline
|
||||
|
||||
Separates internet-facing acquisition from approved inference artifacts. It manages upstream source, revision resolution, content hashes, quarantine, verification and derived artifacts.
|
||||
|
||||
### Hardware plane
|
||||
|
||||
Collects accelerator and node information. A `ComputeNode` has zero or more `Accelerator` objects.
|
||||
Each compute server runs an outbound-only, authenticated observer agent; the central control plane is
|
||||
not itself a node unless explicitly configured as one. The initial product still optimizes scheduling
|
||||
for one NVIDIA GPU while identity, health and persistence support multiple nodes.
|
||||
|
||||
## 3. Request flow
|
||||
|
||||
```text
|
||||
Project
|
||||
→ ModelForge Gateway
|
||||
→ authenticate project
|
||||
→ resolve capability contract
|
||||
→ resolve deployment channel
|
||||
→ policy evaluation
|
||||
→ scheduler resource lease
|
||||
→ runtime adapter
|
||||
→ inference worker
|
||||
→ telemetry + audit
|
||||
→ response
|
||||
```
|
||||
|
||||
## 4. Control flow for candidate upgrades
|
||||
|
||||
```text
|
||||
Hub discovery
|
||||
→ candidate registration
|
||||
→ metadata + license analysis
|
||||
→ download
|
||||
→ quarantine
|
||||
→ artifact verification
|
||||
→ compatibility probe
|
||||
→ generic local benchmark
|
||||
→ project-specific benchmark
|
||||
→ advisor evaluation
|
||||
→ operator approval
|
||||
→ canary or migration
|
||||
→ promotion
|
||||
→ rollback retention
|
||||
```
|
||||
|
||||
## 5. Required isolation boundaries
|
||||
|
||||
- Project clients never call model workers directly.
|
||||
- Model runtimes do not receive Hugging Face tokens or application secrets.
|
||||
- Download/quarantine components may have internet access; inference workers should not by default.
|
||||
- Artifact mounts are read-only in inference workers.
|
||||
- Control-plane write actions are auditable.
|
||||
- Lab-only remote-code exceptions cannot silently become production deployments.
|
||||
|
||||
## 6. Nonfunctional targets
|
||||
|
||||
The initial product should remain understandable and operable by one person on a home infrastructure server. Prefer explicitness over distributed-system complexity. Every abstraction must justify itself through future-safe model lifecycle, not architectural fashion.
|
||||
@@ -0,0 +1,9 @@
|
||||
# vision.embedding@1
|
||||
|
||||
`vision.embedding@1` accepts a bounded batch of image or text items. Each item has exactly one modality. Images are PNG/JPEG, base64 encoded at the gateway boundary and constrained to 4096×4096 pixels in the runtime.
|
||||
|
||||
The response exposes normalized vectors, their dimension, and an immutable `EmbeddingSpace` identifier. Equal dimensions never imply compatible spaces. Identity binds the exact artifact revision and digest, RuntimeProfile fingerprint, processor, input image size, normalization and pooling/projection semantics.
|
||||
|
||||
The preferred SigLIP2 SO400M candidate is retained as discovery evidence but is not forced into a shared 16 GiB GPU envelope. The M9 safe baseline is SigLIP2 Base patch16/224, loaded offline by `transformers_siglip2`.
|
||||
|
||||
Evaluation uses a deterministic, locally generated image/text retrieval set and typed Recall@K, MRR, latency and peak-VRAM evidence. Fixtures never leave the local environment.
|
||||
@@ -0,0 +1,24 @@
|
||||
# ADR-0001: Capability-first routing
|
||||
|
||||
**Status:** Accepted
|
||||
|
||||
## Context
|
||||
|
||||
Directly binding projects to model IDs creates pervasive coupling. Model upgrades then require application code/config changes and encourage accidental use of incompatible model revisions.
|
||||
|
||||
## Decision
|
||||
|
||||
Projects bind to versioned logical capabilities. The ModelForge gateway resolves the capability, project identity, channel and policy to a concrete deployment.
|
||||
|
||||
## Consequences
|
||||
|
||||
Positive:
|
||||
|
||||
- model upgrades become infrastructure lifecycle operations;
|
||||
- project code remains stable;
|
||||
- benchmark and migration policy can attach to capabilities;
|
||||
- fallbacks can be expressed centrally.
|
||||
|
||||
Tradeoff:
|
||||
|
||||
- ModelForge must define precise capability contracts rather than a generic pass-through API only.
|
||||
@@ -0,0 +1,11 @@
|
||||
# ADR-0002: Separate model, artifact and deployment concepts
|
||||
|
||||
**Status:** Accepted
|
||||
|
||||
## Decision
|
||||
|
||||
Represent upstream logical models, immutable revisions, concrete/downloaded artifacts, derived artifacts, runtime profiles and deployments as distinct entities.
|
||||
|
||||
## Rationale
|
||||
|
||||
Performance and compatibility depend on much more than a model name. Quantization, runtime, context settings, launch parameters and exact revision can materially change memory use, speed and output.
|
||||
@@ -0,0 +1,15 @@
|
||||
# ADR-0003: Single-GPU first, multi-node ready
|
||||
|
||||
**Status:** Accepted
|
||||
|
||||
## Decision
|
||||
|
||||
Build the initial scheduler for one local NVIDIA GPU using Docker Compose and NVML, while keeping `ComputeNode` and `Accelerator` abstractions in the domain model.
|
||||
|
||||
## Rejected initially
|
||||
|
||||
- Kubernetes
|
||||
- distributed inference scheduling
|
||||
- multi-tenant cloud architecture
|
||||
|
||||
These add operational cost without solving the first product problem.
|
||||
@@ -0,0 +1,18 @@
|
||||
# ADR-0004: Embedding-space changes require explicit migration
|
||||
|
||||
**Status:** Accepted
|
||||
|
||||
## Decision
|
||||
|
||||
A deployment change for a vector-producing capability is `requires_reindex` unless compatibility has been explicitly proven by the capability contract.
|
||||
|
||||
## Required migration shape
|
||||
|
||||
1. create a shadow index/version;
|
||||
2. backfill using the candidate embedding deployment;
|
||||
3. validate corpus and counts;
|
||||
4. execute project-specific retrieval benchmarks/shadow queries;
|
||||
5. promote the new index atomically;
|
||||
6. retain old index and deployment for rollback until retention expires.
|
||||
|
||||
Matching vector dimension alone is never evidence of semantic compatibility.
|
||||
@@ -0,0 +1,26 @@
|
||||
# ADR-0005: Validated manifests and relational M0 foundation
|
||||
|
||||
**Status:** Accepted
|
||||
|
||||
## Context
|
||||
|
||||
The starter duplicated candidate metadata in Python and YAML and had no persistent schema. Design
|
||||
invariants therefore could not be checked before M1/M2 work.
|
||||
|
||||
## Decision
|
||||
|
||||
Version-controlled YAML is the read-only M0 seed source and is parsed through strict Pydantic schemas.
|
||||
PostgreSQL is the future operational source of truth using distinct SQLAlchemy entities and an Alembic
|
||||
baseline. API routes consume a registry service rather than raw YAML or module constants. Audit events
|
||||
are append-only and hash-chain ready. Canonical endpoints use `/api/v1`.
|
||||
|
||||
GPU allocation is an optional Compose overlay; the base development stack is CPU-compatible. Redis is
|
||||
profiled until a queue-owning milestone needs it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Invalid capability/project/security manifests fail readiness and tests.
|
||||
- The starter no longer presents seed metadata as a deployment.
|
||||
- M1 can persist hardware without reshaping model lifecycle entities.
|
||||
- Later schema changes require explicit Alembic operations; the metadata-driven creation is restricted
|
||||
to the reviewed initial baseline revision.
|
||||
@@ -0,0 +1,23 @@
|
||||
# ADR-0006: Stable hardware identity and typed observations
|
||||
|
||||
**Status:** Accepted
|
||||
|
||||
## Context
|
||||
|
||||
Hostnames, GPU indexes and current telemetry are not stable identities. Containers can also expose an
|
||||
image-local machine ID that is identical on different hosts. Missing metrics cannot safely be stored
|
||||
as zero or ambiguous nulls.
|
||||
|
||||
## Decision
|
||||
|
||||
Direct nodes derive a privacy-preserving UUIDv5 from the OS machine ID; container deployments use a
|
||||
persisted UUID volume or explicit configured identity. GPU UUID is the accelerator identity within a
|
||||
node. Optional observations pair nullable values with `known`, `unknown`, `unsupported`, `unavailable`
|
||||
or `temporarily_failed`. Inventory and latest telemetry persist separately. Reconciliation preserves
|
||||
temporarily absent hardware.
|
||||
|
||||
## Consequences
|
||||
|
||||
Repeated scans are idempotent, container restarts preserve node identity, and absence is not confused
|
||||
with a measured zero. Moving the same GPU between nodes is represented honestly as a different node-
|
||||
accelerator relation. M1 stores only latest telemetry; history/export belongs to observability work.
|
||||
@@ -0,0 +1,29 @@
|
||||
# ADR-0007: Outbound authenticated compute-node agent
|
||||
|
||||
- Status: accepted
|
||||
- Date: 2026-08-25
|
||||
|
||||
## Context
|
||||
|
||||
The central control plane must manage hardware on separate GPU servers without conflating the API
|
||||
host with the compute node. SSH orchestration, Docker socket exposure and inbound node services widen
|
||||
the trust boundary and do not provide a stable reporting protocol.
|
||||
|
||||
## Decision
|
||||
|
||||
Run a lightweight unprivileged agent on each compute node. It initiates HTTPS requests to a
|
||||
versioned central API, reports typed inventory/telemetry and heartbeats, and persists its identity,
|
||||
credential and stream sequences. Enrollment secrets are random, short-lived, single-use and stored
|
||||
only as SHA-256 hashes. Issued credentials are random, node-scoped, hash-only at rest, revocable and
|
||||
never returned by list APIs. Administrative onboarding uses a separately configured operator secret.
|
||||
|
||||
The control plane uses server-received heartbeat time for liveness and retains agent observation time
|
||||
for provenance. Node roles, labels and eligibility are operator-managed metadata; inventory never
|
||||
grants production eligibility automatically.
|
||||
|
||||
## Consequences
|
||||
|
||||
The compute server needs only outbound API reachability. Lost agent state requires explicit
|
||||
revocation/re-enrollment. HTTPS termination and CA trust are deployment responsibilities. Protocol
|
||||
evolution must remain explicit and test older/replayed/incompatible reports. Runtime execution is
|
||||
intentionally absent until M5/M6 define adapters and scheduling.
|
||||
@@ -0,0 +1,31 @@
|
||||
# ADR-0008: Separate content identity from node-scoped storage location
|
||||
|
||||
## Status
|
||||
|
||||
Accepted — 2026-08-25
|
||||
|
||||
## Context
|
||||
|
||||
The M0 schema stored one `storage_uri` directly on an artifact and one source directly on a derived
|
||||
artifact. Operational registry use requires replicas, missing locations, remote-only metadata and
|
||||
transformations with multiple inputs. Treating a path as identity would make moves look like new
|
||||
artifacts and would weaken provenance.
|
||||
|
||||
## Decision
|
||||
|
||||
Artifact content identity is the exact revision plus SHA-256. `ArtifactLocation` associates content
|
||||
with a node-scoped `StorageRoot` and relative path. A derived output has ordered N-source lineage with
|
||||
captured source hashes. Storage capacity is observed and placement fails closed when unknown or when
|
||||
reserve policy would be crossed.
|
||||
|
||||
Legacy `storage_uri` and primary-source columns remain nullable migration compatibility fields in M2;
|
||||
new services never use them as identity.
|
||||
|
||||
## Consequences
|
||||
|
||||
- replicas and moves do not change content identity;
|
||||
- missing/corrupt state is location-aware and auditable;
|
||||
- delete checks can name exact provenance/location dependencies;
|
||||
- remote compute storage can be configured without exposing workers publicly;
|
||||
- later download/runtime milestones can consume this boundary without redesigning the registry;
|
||||
- the additional joins are deliberate and preferable to provenance ambiguity.
|
||||
@@ -0,0 +1,17 @@
|
||||
# ADR-0009: Node-local artifact acquisition
|
||||
|
||||
## Status
|
||||
|
||||
Accepted — 2026-08-25
|
||||
|
||||
## Decision
|
||||
|
||||
The control plane owns Hub metadata, immutable plans and job state. The selected compute node's
|
||||
outbound agent downloads and verifies bytes against its locally mounted approved storage root.
|
||||
Artifact operations are a closed typed protocol, not shell/SSH/command execution.
|
||||
|
||||
## Consequences
|
||||
|
||||
Capacity and path safety are evaluated where the bytes live, GPU Node needs no inbound service, and
|
||||
credentials/jobs remain node-scoped. SSH remains deployment/debug tooling outside the ModelForge
|
||||
runtime protocol.
|
||||
@@ -0,0 +1,17 @@
|
||||
# ADR-0010: Immutable, explicitly approved download plans
|
||||
|
||||
## Status
|
||||
|
||||
Accepted — 2026-08-25
|
||||
|
||||
## Decision
|
||||
|
||||
A download is derived from an immutable exact commit and selected file inventory. Its canonical
|
||||
payload, target, expected bytes/checksums, preflights, expiry and idempotency hash cannot change.
|
||||
Plan creation and approval are separate actions. Mutable refs and stale/unapproved plans cannot
|
||||
execute.
|
||||
|
||||
## Consequences
|
||||
|
||||
Operators review the precise transfer before bytes move; retries reuse the same intent; upstream
|
||||
movement requires a new revision and plan; duplicate execution resolves to one job.
|
||||
@@ -0,0 +1,18 @@
|
||||
# ADR-0011: Set-level quarantine and atomic promotion
|
||||
|
||||
## Status
|
||||
|
||||
Accepted — 2026-08-25
|
||||
|
||||
## Decision
|
||||
|
||||
All required files for an artifact set are downloaded to a job-scoped directory inside the target
|
||||
filesystem. Static verification precedes one atomic directory rename to the exact-revision store.
|
||||
The database registers artifacts only after that promotion report and records static integrity as
|
||||
distinct from later security/runtime approval.
|
||||
|
||||
## Consequences
|
||||
|
||||
Partial sets cannot appear locally verified, cross-filesystem partial moves are avoided, retries
|
||||
can reuse durable partials, and a sidecar manifest supports reconciliation if the file move and DB
|
||||
transaction are interrupted.
|
||||
@@ -0,0 +1,15 @@
|
||||
# ADR-0012: Typed static runtime worker boundary
|
||||
|
||||
Status: accepted (M4)
|
||||
|
||||
## Decision
|
||||
|
||||
Run heavy ML dependencies in a separately built, static Compose worker. It polls outbound with a
|
||||
node credential, accepts only the `runtime.probe.v1` schema, mounts exact artifacts read-only and
|
||||
uses a child process per probe. Do not expose Docker, shell, SSH, generic Python or public runtime
|
||||
control surfaces.
|
||||
|
||||
## Consequences
|
||||
|
||||
Runtime images are larger but independently pinned. Node Agent remains lightweight. Adding an
|
||||
adapter requires a typed implementation and image revision; remote payloads cannot invent launches.
|
||||
@@ -0,0 +1,14 @@
|
||||
# ADR-0013: Exact-ArtifactSet LAB execution approval
|
||||
|
||||
Status: accepted (M4)
|
||||
|
||||
## Decision
|
||||
|
||||
Separate integrity verification from permission to execute. Store LAB approval against one exact
|
||||
ArtifactSet and fingerprint the evidence reviewed. Expiry, revocation or evidence changes invalidate
|
||||
it. Do not infer production approval.
|
||||
|
||||
## Consequences
|
||||
|
||||
Operators make the first code/data execution boundary explicit and auditable. New revisions and
|
||||
artifact variants require separate review even when the model name is unchanged.
|
||||
@@ -0,0 +1,14 @@
|
||||
# ADR-0014: Static versus locally proven runtime support
|
||||
|
||||
Status: accepted (M4)
|
||||
|
||||
## Decision
|
||||
|
||||
Represent static compatibility and successful local execution as different evidence. Runtime
|
||||
declarations can establish format/architecture expectations; only a completed probe can establish
|
||||
local support for an exact environment/profile/node/artifact fingerprint.
|
||||
|
||||
## Consequences
|
||||
|
||||
Unknown or incomplete evidence remains `unknown`/`requires_probe`; no optimistic compatibility
|
||||
fallback exists. Driver, image, runtime, artifact or hardware changes make evidence stale.
|
||||
@@ -0,0 +1,8 @@
|
||||
# ADR-0015 — Capability-first serving gateway
|
||||
|
||||
Status: accepted (M5)
|
||||
|
||||
Projects call a central authenticated Gateway with a versioned capability. Concrete model/runtime
|
||||
selection is server-side. This preserves upgrade isolation, prevents public worker exposure and
|
||||
provides one QoS, audit and privacy boundary. No cloud or direct-worker fallback is allowed.
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
# ADR-0016 — Database-backed GPU lease coordination
|
||||
|
||||
Status: accepted (M5)
|
||||
|
||||
GPU admission uses persisted leases and a locked accelerator row. Correctness therefore does not
|
||||
depend on one API process or an in-memory mutex. Server timestamps, expiry and reconciliation make
|
||||
crashed-request reservations recoverable.
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user