Initial public ModelForge release

2026-09-01 21:30:16 +02:00
commit 7082ab955a
490 changed files with 104252 additions and 0 deletions
@@ -0,0 +1,57 @@
+# Artifact Acquisition
+
+## Flow
+
+```text
+Hub metadata -> exact revision -> artifact set -> immutable plan -> explicit approval
+    -> node-scoped lease -> quarantine download -> static verification -> atomic promotion
+    -> artifact/location registration
+```
+
+`DownloadPlan` captures the artifact set, exact SHA, selected paths and expected checksums/sizes,
+target node/root, total bytes, capacity result, security constraints, expiry and a canonical
+idempotency key. Creation is a dry run with no bytes. Approval is a separate audited action and
+execution rejects a merely planned or stale plan.
+
+The planner prefers complete advertised Safetensors weights plus configuration/index/tokenizer
+support and excludes duplicate pickle/GGUF weights. If only risky pickle-style weights remain, the
+risk is explicit and plan approval fails. No format conversion, quantization or loader execution
+occurs.
+
+## Placement and ownership
+
+Jobs are claimed only by their `compute_node_id` through node-scoped bearer authentication. A lease
+token binds progress/completion/failure calls to one claim. PostgreSQL row locking and one job per
+plan enforce single-node concurrency and replay/idempotency boundaries. The outbound agent loop
+processes one acquisition at a time and has no shell, generic command, Docker socket or inbound
+listener.
+
+Capacity is checked from the registered root during planning and execution, then checked again from
+the real target filesystem by the agent before download. Failure never falls back to another root.
+GPU Node uses `/data/artifacts/model-registry`, mapped to the cache-backed host root; the Unraid array
+is not a placement candidate.
+
+## Retry, cancellation, and recovery
+
+Transient failures have at most three automatic claims. Partial files use `.part`, byte ranges and
+durable flushes. A complete partial is verified without a new request. A failed job can receive an
+explicit audited operator retry; its progress epoch resets but historical attempts remain. Cancel
+is observed at progress boundaries and cannot promote an artifact.
+
+Expired leases make work reclaimable after agent/control-plane interruption. If promotion succeeded
+but the database response was lost, the exact-revision manifest makes promotion idempotent; the
+same plan/job can reconcile. A verified existing plan returns the existing completed job rather
+than downloading again.
+
+## API
+
+- `POST /api/v1/download-plans`
+- `GET /api/v1/download-plans/{id}`
+- `POST /api/v1/download-plans/{id}/approve`
+- `POST /api/v1/download-plans/{id}/execute`
+- `GET /api/v1/artifact-jobs[/{id}]`
+- `POST /api/v1/artifact-jobs/{id}/cancel`
+- `POST /api/v1/artifact-jobs/{id}/retry`
+
+Agent routes are separately authenticated and expose only claim, typed progress, completion and
+failure operations.