Initial public ModelForge release
This commit is contained in:
Jens
@@ -0,0 +1,100 @@
|
||||
# Node Agent Protocol v1
|
||||
|
||||
## Purpose and boundary
|
||||
|
||||
M1.5 adds an outbound-only agent transport between a central ModelForge control plane and one or
|
||||
more compute nodes. The agent and adjacent M4 runtime worker cannot launch arbitrary commands,
|
||||
expose a shell, access the Docker socket, promote models, or serve production inference.
|
||||
|
||||
Protocol version `1` supports `hardware.inventory`, `hardware.telemetry`, and the additive typed
|
||||
capabilities `artifact.acquire.v1`, `runtime.probe.v1`, `runtime.health.v1` and
|
||||
`runtime.unload.v1`. The server rejects a
|
||||
different protocol version explicitly; capabilities are recorded for later additive negotiation.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Method | Route | Authentication | Semantics |
|
||||
|---|---|---|---|
|
||||
| POST | `/api/v1/agent/enroll` | one-time enrollment token | create/bind node and return its credential once |
|
||||
| POST | `/api/v1/agent/heartbeat` | node bearer credential | agent/version/capability liveness |
|
||||
| PUT | `/api/v1/agent/inventory` | node bearer credential | stable host/GPU inventory reconciliation |
|
||||
| PUT | `/api/v1/agent/telemetry` | node bearer credential | latest measured RAM/storage/GPU telemetry |
|
||||
| GET | `/api/v1/agent/artifact-jobs/next` | node bearer credential | claim one job owned by this node |
|
||||
| POST | `/api/v1/agent/artifact-jobs/{id}/progress` | node bearer + lease | bounded progress/cancel handshake |
|
||||
| POST | `/api/v1/agent/artifact-jobs/{id}/complete` | node bearer + lease | verified inventory and promotion report |
|
||||
| POST | `/api/v1/agent/artifact-jobs/{id}/fail` | node bearer + lease | typed failure and bounded retry request |
|
||||
| GET | `/api/v1/agent/runtime-probes/next` | node bearer credential | claim one exact-node typed probe |
|
||||
| POST | `/api/v1/agent/runtime-probes/{id}/progress` | node bearer + lease | lifecycle progress/cancel handshake |
|
||||
| POST | `/api/v1/agent/runtime-probes/{id}/complete` | node bearer + lease | layered health, inference and measured evidence |
|
||||
| POST | `/api/v1/agent/runtime-probes/{id}/fail` | node bearer + lease | normalized typed runtime failure |
|
||||
|
||||
Artifact leases contain only repository, exact commit, selected files, expected sizes/checksums,
|
||||
approved target root and reserve policy. Credentials from one node cannot claim or update another
|
||||
node's work. There is no arbitrary command field.
|
||||
|
||||
Runtime leases similarly contain only exact server-resolved artifact paths and digests, immutable
|
||||
RuntimeProfile/RuntimeEnvironment identity and the fixed technical probe input. A static adjacent
|
||||
worker executes them; neither the protocol nor Node Agent provides general process/container control.
|
||||
|
||||
M5 adds `deployment.load.v1`, `deployment.invoke.v1`, `deployment.health.v1`,
|
||||
`deployment.drain.v1` and `deployment.unload.v1`. The adjacent runtime worker polls authenticated
|
||||
typed jobs outbound over the same private-CA boundary. Raw project traffic never reaches the node,
|
||||
and the Node Agent credential is never accepted as a service-client credential.
|
||||
|
||||
Administrative token creation, revocation, node metadata changes and credential revocation live
|
||||
under `/api/v1/admin`. They require `X-ModelForge-Admin-Token`; the API fails closed when
|
||||
`MODELFORGE_OPERATOR_API_KEY` is unset.
|
||||
|
||||
`POST /api/v1/admin/hardware/nodes/{id}/credential/rotate` atomically revokes the previous
|
||||
credential and returns its replacement once. Install the replacement in the protected agent state
|
||||
file before restarting it. Rotation and revocation are separate audited operations.
|
||||
|
||||
Permanent decommission is a separate terminal lifecycle. Its preview and execute endpoints are
|
||||
operator-only. Completion revokes every node credential, and ordinary enrollment, rotation,
|
||||
management, authentication, heartbeat, inventory and telemetry all refuse the tombstoned identity.
|
||||
An outage or lost credential must use the recovery runbook instead of decommission.
|
||||
|
||||
## Identity, time and ordering
|
||||
|
||||
- A node credential is scoped to exactly one `ComputeNode` UUID.
|
||||
- Stable node identity comes from the persisted agent state volume, not hostname or container ID.
|
||||
- GPUs reconcile by `(node_id, GPU UUID)`; hostname, device index and PCI address are mutable facts.
|
||||
- Each inventory and telemetry stream has a persistent monotonically increasing sequence number.
|
||||
- The control plane stores both agent `observed_at` and server `received_at`.
|
||||
- Replayed sequence numbers are rejected. A higher sequence with an older observation timestamp
|
||||
advances the stream cursor but does not replace last-good state; the reason remains visible until
|
||||
the node clock catches up. Observations beyond the configurable future-skew limit (default five
|
||||
minutes) are handled the same way, preventing one bad clock from poisoning latest state.
|
||||
Server-received time drives liveness, so clock skew cannot keep a dead node online.
|
||||
- An accelerator is marked missing only after a successful complete inventory omits its UUID. An
|
||||
unavailable/incomplete NVML collection preserves last-good state.
|
||||
|
||||
## Liveness
|
||||
|
||||
Enabled nodes are `online`, `stale`, or `offline` using server-received heartbeat time. Defaults are
|
||||
30 and 90 seconds and are configurable. Disabled nodes always report `disabled` and their agent
|
||||
credential is rejected. Offline/stale transitions, return online, enrollment, revocation and
|
||||
meaningful inventory changes are audited; high-rate telemetry does not flood the audit log.
|
||||
Terminal nodes report `decommissioned`; liveness reconciliation never changes that state.
|
||||
|
||||
## Error contract
|
||||
|
||||
Malformed payloads return validation errors. Missing/invalid/revoked credentials return 401;
|
||||
credential/node ownership mismatches and disabled nodes return 403; unknown resources return 404;
|
||||
incompatible protocol versions, replayed samples and conflicting node identities return 409. The
|
||||
normal API correlation ID remains present for operational tracing.
|
||||
|
||||
|
||||
## M16 disconnect and diagnostics
|
||||
|
||||
A disconnected agent stops publishing; liveness moves online → stale → offline on the configured
|
||||
thresholds and no work is assigned to it. Reconnection resumes publication on the same persisted
|
||||
identity, and a live rehearsal confirmed the heartbeat genuinely stops rather than merely ageing.
|
||||
|
||||
Enrolment is single-use and atomically claimed. A sixty-way concurrent storm against one token
|
||||
produced exactly one identity and one active credential.
|
||||
|
||||
A failed publication names the status, method and path — never the response body, which can echo
|
||||
what the agent was publishing, and never the credential. Before M16 the agent logged only
|
||||
`control-plane publication failed; retrying`, which cannot tell an operator whether the cause is a
|
||||
revoked credential or a DNS failure.
|
||||
Reference in New Issue
Block a user