# Remote Server Agent Deployment

## Prerequisites

The remote server needs Docker Engine with Compose v2, an NVIDIA driver, NVIDIA Container Toolkit,
and outbound HTTPS access to the central ModelForge API. It needs no inbound SSH path from
ModelForge, no public agent port, no Docker socket mount and no privileged container.

The container deliberately uses a persisted UUID because host `/etc/machine-id` is not mounted.
Hostname/OS/CPU/RAM describe the agent's Linux execution environment; without container resource
limits this normally represents the server, while configured CPU/memory limits represent the
capacity actually available to future workloads. Storage facts cover only the named ModelForge data
volumes. This avoids unsafe host-root mounts and keeps the semantics operationally honest.

The Compose service drops all Linux capabilities, enables `no-new-privileges`, uses a read-only root
filesystem and opens only a bounded in-memory `/tmp`. It publishes no port. Set
`MODELFORGE_AGENT_HOSTNAME` to the real compute-node hostname. On Unraid or another appliance, set
the four `MODELFORGE_AGENT_*_VOLUME` variables to narrow, dedicated absolute bind paths when state
must survive Docker image/volume recreation; never point one at the host root.

Expose the control-plane API through a trusted HTTPS reverse proxy. Use an internal CA by installing
that CA in the agent image/host trust store; do not disable TLS verification outside an isolated
local smoke test.

For a private CA that should be trusted only by ModelForge, combine the base file with
`docker-compose.node-agent.private-ca.yml`. Set `MODELFORGE_AGENT_CA_CERT_PATH` to the public CA
certificate, plus `MODELFORGE_AGENT_CONTROL_PLANE_HOST_ALIAS` and
`MODELFORGE_AGENT_CONTROL_PLANE_HOST_ADDRESS` when private DNS is unavailable. The overlay mounts
only that public certificate read-only and scopes the host mapping to the agent container.

## 1. Configure the control plane

Set a high-entropy `MODELFORGE_OPERATOR_API_KEY`, keep `MODELFORGE_HARDWARE_REFRESH_ON_STARTUP=false`,
and start the base stack. Create an enrollment token from **Compute nodes → Enroll node**, or:

```bash
curl -sS -X POST https://modelforge.example/api/v1/admin/node-enrollments \
  -H "X-ModelForge-Admin-Token: $MODELFORGE_OPERATOR_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"expires_in_seconds":900,"display_name":"server01","role":"primary-inference","labels":{"site":"office"},"production_eligible":false,"lab_eligible":true,"benchmark_eligible":true}'
```

The response shows the enrollment secret once. Leave `production_eligible` false until the actual
node has passed hardware, identity, recovery and security acceptance. Eligibility is scheduler
metadata; setting it true does not launch a runtime or bypass later promotion evidence.

## 2. Start the agent on the GPU server

Copy/clone this repository to the server. Create `.env` without committing it:

```dotenv
MODELFORGE_AGENT_CONTROL_PLANE_URL=https://modelforge.example
MODELFORGE_AGENT_ENROLLMENT_TOKEN=<one-time-token>
MODELFORGE_AGENT_DISPLAY_NAME=Server 01
MODELFORGE_AGENT_HOSTNAME=server01
MODELFORGE_AGENT_TLS_VERIFY=true
MODELFORGE_AGENT_STATE_VOLUME=/mnt/user/appdata/modelforge-node-agent/state
MODELFORGE_AGENT_HF_CACHE_VOLUME=/mnt/cache/appdata/modelforge-node-agent/hf-cache
MODELFORGE_AGENT_ARTIFACT_VOLUME=/mnt/cache/appdata/modelforge-node-agent/artifacts
MODELFORGE_AGENT_QUARANTINE_VOLUME=/mnt/cache/appdata/modelforge-node-agent/quarantine
MODELFORGE_HF_TOKEN=<optional-secret-for-gated-repositories>
```

Then run:

```bash
docker compose -f docker-compose.node-agent.yml up -d --build
docker compose -f docker-compose.node-agent.yml logs --tail=100 node-agent
```

With the private-CA overlay, use both files for every lifecycle command:

```bash
docker compose -f docker-compose.node-agent.yml \
  -f docker-compose.node-agent.private-ca.yml up -d --build
```

After the node appears online, remove `MODELFORGE_AGENT_ENROLLMENT_TOKEN` from `.env` and recreate
the service. The node-scoped credential persists in `node-agent-state`:

```bash
docker compose -f docker-compose.node-agent.yml up -d --force-recreate node-agent
```

Do not delete the state volume during ordinary upgrades; doing so discards identity and credential.
Use the UI/API to revoke the old credential before deliberately re-enrolling.

For M3, register the host cache path as a storage root and map it to
`/data/artifacts/model-registry`. The host path and container path are deliberately separate facts.
Never substitute `/mnt/user` when the cache capacity guard fails. The agent advertises
`artifact.acquire.v1`, polls one typed job at a time, and retains job-scoped partials below the
approved root for bounded retry. Keep the optional Hub token only in secret environment/config;
do not place it in Compose files or logs.

## 3. Verify and operate

Confirm the UI shows the expected hostname, agent/protocol version, heartbeat age, CPU/RAM/storage,
GPU UUID/model/VRAM/driver and measured telemetry. Restart the agent and confirm node and accelerator
UUIDs remain unchanged. Stop it long enough to observe `stale` then `offline`, restart it and confirm
`online` plus a return audit event.

For temporary outages the agent retains its identity/credential and retries with bounded exponential
backoff. Rotate with the authenticated admin rotation endpoint, securely replace
`/data/state/node-credential` with its one-time response, and recreate the agent. Revoke without a
replacement when a credential may be compromised. Disable a node to make it ineligible and immediately
reject further publications. Upgrades are ordinary image
rebuild/recreate operations; protocol-incompatible agents are visibly rejected instead of silently
downgraded.

## RTX 4080 acceptance checklist

On the actual target server, capture: `nvidia-smi` model/UUID/driver output, node-agent container
status, `/api/v1/hardware/nodes/{id}` output with secrets removed, a restart identity comparison,
stale/offline/recovery evidence, and UI screenshots. The release verdict may not claim RTX 4080
validation until these observations exist.