101 lines
6.5 KiB
Markdown
101 lines
6.5 KiB
Markdown
# Node Agent Protocol v1
|
|
|
|
## Purpose and boundary
|
|
|
|
M1.5 adds an outbound-only agent transport between a central ModelForge control plane and one or
|
|
more compute nodes. The agent and adjacent M4 runtime worker cannot launch arbitrary commands,
|
|
expose a shell, access the Docker socket, promote models, or serve production inference.
|
|
|
|
Protocol version `1` supports `hardware.inventory`, `hardware.telemetry`, and the additive typed
|
|
capabilities `artifact.acquire.v1`, `runtime.probe.v1`, `runtime.health.v1` and
|
|
`runtime.unload.v1`. The server rejects a
|
|
different protocol version explicitly; capabilities are recorded for later additive negotiation.
|
|
|
|
## Endpoints
|
|
|
|
| Method | Route | Authentication | Semantics |
|
|
|---|---|---|---|
|
|
| POST | `/api/v1/agent/enroll` | one-time enrollment token | create/bind node and return its credential once |
|
|
| POST | `/api/v1/agent/heartbeat` | node bearer credential | agent/version/capability liveness |
|
|
| PUT | `/api/v1/agent/inventory` | node bearer credential | stable host/GPU inventory reconciliation |
|
|
| PUT | `/api/v1/agent/telemetry` | node bearer credential | latest measured RAM/storage/GPU telemetry |
|
|
| GET | `/api/v1/agent/artifact-jobs/next` | node bearer credential | claim one job owned by this node |
|
|
| POST | `/api/v1/agent/artifact-jobs/{id}/progress` | node bearer + lease | bounded progress/cancel handshake |
|
|
| POST | `/api/v1/agent/artifact-jobs/{id}/complete` | node bearer + lease | verified inventory and promotion report |
|
|
| POST | `/api/v1/agent/artifact-jobs/{id}/fail` | node bearer + lease | typed failure and bounded retry request |
|
|
| GET | `/api/v1/agent/runtime-probes/next` | node bearer credential | claim one exact-node typed probe |
|
|
| POST | `/api/v1/agent/runtime-probes/{id}/progress` | node bearer + lease | lifecycle progress/cancel handshake |
|
|
| POST | `/api/v1/agent/runtime-probes/{id}/complete` | node bearer + lease | layered health, inference and measured evidence |
|
|
| POST | `/api/v1/agent/runtime-probes/{id}/fail` | node bearer + lease | normalized typed runtime failure |
|
|
|
|
Artifact leases contain only repository, exact commit, selected files, expected sizes/checksums,
|
|
approved target root and reserve policy. Credentials from one node cannot claim or update another
|
|
node's work. There is no arbitrary command field.
|
|
|
|
Runtime leases similarly contain only exact server-resolved artifact paths and digests, immutable
|
|
RuntimeProfile/RuntimeEnvironment identity and the fixed technical probe input. A static adjacent
|
|
worker executes them; neither the protocol nor Node Agent provides general process/container control.
|
|
|
|
M5 adds `deployment.load.v1`, `deployment.invoke.v1`, `deployment.health.v1`,
|
|
`deployment.drain.v1` and `deployment.unload.v1`. The adjacent runtime worker polls authenticated
|
|
typed jobs outbound over the same private-CA boundary. Raw project traffic never reaches the node,
|
|
and the Node Agent credential is never accepted as a service-client credential.
|
|
|
|
Administrative token creation, revocation, node metadata changes and credential revocation live
|
|
under `/api/v1/admin`. They require `X-ModelForge-Admin-Token`; the API fails closed when
|
|
`MODELFORGE_OPERATOR_API_KEY` is unset.
|
|
|
|
`POST /api/v1/admin/hardware/nodes/{id}/credential/rotate` atomically revokes the previous
|
|
credential and returns its replacement once. Install the replacement in the protected agent state
|
|
file before restarting it. Rotation and revocation are separate audited operations.
|
|
|
|
Permanent decommission is a separate terminal lifecycle. Its preview and execute endpoints are
|
|
operator-only. Completion revokes every node credential, and ordinary enrollment, rotation,
|
|
management, authentication, heartbeat, inventory and telemetry all refuse the tombstoned identity.
|
|
An outage or lost credential must use the recovery runbook instead of decommission.
|
|
|
|
## Identity, time and ordering
|
|
|
|
- A node credential is scoped to exactly one `ComputeNode` UUID.
|
|
- Stable node identity comes from the persisted agent state volume, not hostname or container ID.
|
|
- GPUs reconcile by `(node_id, GPU UUID)`; hostname, device index and PCI address are mutable facts.
|
|
- Each inventory and telemetry stream has a persistent monotonically increasing sequence number.
|
|
- The control plane stores both agent `observed_at` and server `received_at`.
|
|
- Replayed sequence numbers are rejected. A higher sequence with an older observation timestamp
|
|
advances the stream cursor but does not replace last-good state; the reason remains visible until
|
|
the node clock catches up. Observations beyond the configurable future-skew limit (default five
|
|
minutes) are handled the same way, preventing one bad clock from poisoning latest state.
|
|
Server-received time drives liveness, so clock skew cannot keep a dead node online.
|
|
- An accelerator is marked missing only after a successful complete inventory omits its UUID. An
|
|
unavailable/incomplete NVML collection preserves last-good state.
|
|
|
|
## Liveness
|
|
|
|
Enabled nodes are `online`, `stale`, or `offline` using server-received heartbeat time. Defaults are
|
|
30 and 90 seconds and are configurable. Disabled nodes always report `disabled` and their agent
|
|
credential is rejected. Offline/stale transitions, return online, enrollment, revocation and
|
|
meaningful inventory changes are audited; high-rate telemetry does not flood the audit log.
|
|
Terminal nodes report `decommissioned`; liveness reconciliation never changes that state.
|
|
|
|
## Error contract
|
|
|
|
Malformed payloads return validation errors. Missing/invalid/revoked credentials return 401;
|
|
credential/node ownership mismatches and disabled nodes return 403; unknown resources return 404;
|
|
incompatible protocol versions, replayed samples and conflicting node identities return 409. The
|
|
normal API correlation ID remains present for operational tracing.
|
|
|
|
|
|
## M16 disconnect and diagnostics
|
|
|
|
A disconnected agent stops publishing; liveness moves online → stale → offline on the configured
|
|
thresholds and no work is assigned to it. Reconnection resumes publication on the same persisted
|
|
identity, and a live rehearsal confirmed the heartbeat genuinely stops rather than merely ageing.
|
|
|
|
Enrolment is single-use and atomically claimed. A sixty-way concurrent storm against one token
|
|
produced exactly one identity and one active credential.
|
|
|
|
A failed publication names the status, method and path — never the response body, which can echo
|
|
what the agent was publishing, and never the credential. Before M16 the agent logged only
|
|
`control-plane publication failed; retrying`, which cannot tell an operator whether the cause is a
|
|
revoked credential or a DNS failure.
|