ModelForge/docs/architecture/NODE_AGENT_PROTOCOL.md

Node Agent Protocol v1

Purpose and boundary

M1.5 adds an outbound-only agent transport between a central ModelForge control plane and one or more compute nodes. The agent and adjacent M4 runtime worker cannot launch arbitrary commands, expose a shell, access the Docker socket, promote models, or serve production inference.

Protocol version 1 supports hardware.inventory, hardware.telemetry, and the additive typed capabilities artifact.acquire.v1, runtime.probe.v1, runtime.health.v1 and runtime.unload.v1. The server rejects a different protocol version explicitly; capabilities are recorded for later additive negotiation.

Endpoints

Method	Route	Authentication	Semantics
POST	/api/v1/agent/enroll	one-time enrollment token	create/bind node and return its credential once
POST	/api/v1/agent/heartbeat	node bearer credential	agent/version/capability liveness
PUT	/api/v1/agent/inventory	node bearer credential	stable host/GPU inventory reconciliation
PUT	/api/v1/agent/telemetry	node bearer credential	latest measured RAM/storage/GPU telemetry
GET	/api/v1/agent/artifact-jobs/next	node bearer credential	claim one job owned by this node
POST	/api/v1/agent/artifact-jobs/{id}/progress	node bearer + lease	bounded progress/cancel handshake
POST	/api/v1/agent/artifact-jobs/{id}/complete	node bearer + lease	verified inventory and promotion report
POST	/api/v1/agent/artifact-jobs/{id}/fail	node bearer + lease	typed failure and bounded retry request
GET	/api/v1/agent/runtime-probes/next	node bearer credential	claim one exact-node typed probe
POST	/api/v1/agent/runtime-probes/{id}/progress	node bearer + lease	lifecycle progress/cancel handshake
POST	/api/v1/agent/runtime-probes/{id}/complete	node bearer + lease	layered health, inference and measured evidence
POST	/api/v1/agent/runtime-probes/{id}/fail	node bearer + lease	normalized typed runtime failure

Artifact leases contain only repository, exact commit, selected files, expected sizes/checksums, approved target root and reserve policy. Credentials from one node cannot claim or update another node's work. There is no arbitrary command field.

Runtime leases similarly contain only exact server-resolved artifact paths and digests, immutable RuntimeProfile/RuntimeEnvironment identity and the fixed technical probe input. A static adjacent worker executes them; neither the protocol nor Node Agent provides general process/container control.

M5 adds deployment.load.v1, deployment.invoke.v1, deployment.health.v1, deployment.drain.v1 and deployment.unload.v1. The adjacent runtime worker polls authenticated typed jobs outbound over the same private-CA boundary. Raw project traffic never reaches the node, and the Node Agent credential is never accepted as a service-client credential.

Administrative token creation, revocation, node metadata changes and credential revocation live under /api/v1/admin. They require X-ModelForge-Admin-Token; the API fails closed when MODELFORGE_OPERATOR_API_KEY is unset.

POST /api/v1/admin/hardware/nodes/{id}/credential/rotate atomically revokes the previous credential and returns its replacement once. Install the replacement in the protected agent state file before restarting it. Rotation and revocation are separate audited operations.

Permanent decommission is a separate terminal lifecycle. Its preview and execute endpoints are operator-only. Completion revokes every node credential, and ordinary enrollment, rotation, management, authentication, heartbeat, inventory and telemetry all refuse the tombstoned identity. An outage or lost credential must use the recovery runbook instead of decommission.

Identity, time and ordering

• A node credential is scoped to exactly one ComputeNode UUID.
• Stable node identity comes from the persisted agent state volume, not hostname or container ID.
• GPUs reconcile by (node_id, GPU UUID); hostname, device index and PCI address are mutable facts.
• Each inventory and telemetry stream has a persistent monotonically increasing sequence number.
• The control plane stores both agent observed_at and server received_at.
• Replayed sequence numbers are rejected. A higher sequence with an older observation timestamp advances the stream cursor but does not replace last-good state; the reason remains visible until the node clock catches up. Observations beyond the configurable future-skew limit (default five minutes) are handled the same way, preventing one bad clock from poisoning latest state. Server-received time drives liveness, so clock skew cannot keep a dead node online.
• An accelerator is marked missing only after a successful complete inventory omits its UUID. An unavailable/incomplete NVML collection preserves last-good state.

Liveness

Enabled nodes are online, stale, or offline using server-received heartbeat time. Defaults are 30 and 90 seconds and are configurable. Disabled nodes always report disabled and their agent credential is rejected. Offline/stale transitions, return online, enrollment, revocation and meaningful inventory changes are audited; high-rate telemetry does not flood the audit log. Terminal nodes report decommissioned; liveness reconciliation never changes that state.

Error contract

Malformed payloads return validation errors. Missing/invalid/revoked credentials return 401; credential/node ownership mismatches and disabled nodes return 403; unknown resources return 404; incompatible protocol versions, replayed samples and conflicting node identities return 409. The normal API correlation ID remains present for operational tracing.

M16 disconnect and diagnostics

A disconnected agent stops publishing; liveness moves online → stale → offline on the configured thresholds and no work is assigned to it. Reconnection resumes publication on the same persisted identity, and a live rehearsal confirmed the heartbeat genuinely stops rather than merely ageing.

Enrolment is single-use and atomically claimed. A sixty-way concurrent storm against one token produced exactly one identity and one active credential.

A failed publication names the status, method and path — never the response body, which can echo what the agent was publishing, and never the credential. Before M16 the agent logged only control-plane publication failed; retrying, which cannot tell an operator whether the cause is a revoked credential or a DNS failure.