116 lines
6.0 KiB
Markdown
116 lines
6.0 KiB
Markdown
# Remote Server Agent Deployment
|
|
|
|
## Prerequisites
|
|
|
|
The remote server needs Docker Engine with Compose v2, an NVIDIA driver, NVIDIA Container Toolkit,
|
|
and outbound HTTPS access to the central ModelForge API. It needs no inbound SSH path from
|
|
ModelForge, no public agent port, no Docker socket mount and no privileged container.
|
|
|
|
The container deliberately uses a persisted UUID because host `/etc/machine-id` is not mounted.
|
|
Hostname/OS/CPU/RAM describe the agent's Linux execution environment; without container resource
|
|
limits this normally represents the server, while configured CPU/memory limits represent the
|
|
capacity actually available to future workloads. Storage facts cover only the named ModelForge data
|
|
volumes. This avoids unsafe host-root mounts and keeps the semantics operationally honest.
|
|
|
|
The Compose service drops all Linux capabilities, enables `no-new-privileges`, uses a read-only root
|
|
filesystem and opens only a bounded in-memory `/tmp`. It publishes no port. Set
|
|
`MODELFORGE_AGENT_HOSTNAME` to the real compute-node hostname. On Unraid or another appliance, set
|
|
the four `MODELFORGE_AGENT_*_VOLUME` variables to narrow, dedicated absolute bind paths when state
|
|
must survive Docker image/volume recreation; never point one at the host root.
|
|
|
|
Expose the control-plane API through a trusted HTTPS reverse proxy. Use an internal CA by installing
|
|
that CA in the agent image/host trust store; do not disable TLS verification outside an isolated
|
|
local smoke test.
|
|
|
|
For a private CA that should be trusted only by ModelForge, combine the base file with
|
|
`docker-compose.node-agent.private-ca.yml`. Set `MODELFORGE_AGENT_CA_CERT_PATH` to the public CA
|
|
certificate, plus `MODELFORGE_AGENT_CONTROL_PLANE_HOST_ALIAS` and
|
|
`MODELFORGE_AGENT_CONTROL_PLANE_HOST_ADDRESS` when private DNS is unavailable. The overlay mounts
|
|
only that public certificate read-only and scopes the host mapping to the agent container.
|
|
|
|
## 1. Configure the control plane
|
|
|
|
Set a high-entropy `MODELFORGE_OPERATOR_API_KEY`, keep `MODELFORGE_HARDWARE_REFRESH_ON_STARTUP=false`,
|
|
and start the base stack. Create an enrollment token from **Compute nodes → Enroll node**, or:
|
|
|
|
```bash
|
|
curl -sS -X POST https://modelforge.example/api/v1/admin/node-enrollments \
|
|
-H "X-ModelForge-Admin-Token: $MODELFORGE_OPERATOR_API_KEY" \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"expires_in_seconds":900,"display_name":"server01","role":"primary-inference","labels":{"site":"office"},"production_eligible":false,"lab_eligible":true,"benchmark_eligible":true}'
|
|
```
|
|
|
|
The response shows the enrollment secret once. Leave `production_eligible` false until the actual
|
|
node has passed hardware, identity, recovery and security acceptance. Eligibility is scheduler
|
|
metadata; setting it true does not launch a runtime or bypass later promotion evidence.
|
|
|
|
## 2. Start the agent on the GPU server
|
|
|
|
Copy/clone this repository to the server. Create `.env` without committing it:
|
|
|
|
```dotenv
|
|
MODELFORGE_AGENT_CONTROL_PLANE_URL=https://modelforge.example
|
|
MODELFORGE_AGENT_ENROLLMENT_TOKEN=<one-time-token>
|
|
MODELFORGE_AGENT_DISPLAY_NAME=Server 01
|
|
MODELFORGE_AGENT_HOSTNAME=server01
|
|
MODELFORGE_AGENT_TLS_VERIFY=true
|
|
MODELFORGE_AGENT_STATE_VOLUME=/mnt/user/appdata/modelforge-node-agent/state
|
|
MODELFORGE_AGENT_HF_CACHE_VOLUME=/mnt/cache/appdata/modelforge-node-agent/hf-cache
|
|
MODELFORGE_AGENT_ARTIFACT_VOLUME=/mnt/cache/appdata/modelforge-node-agent/artifacts
|
|
MODELFORGE_AGENT_QUARANTINE_VOLUME=/mnt/cache/appdata/modelforge-node-agent/quarantine
|
|
MODELFORGE_HF_TOKEN=<optional-secret-for-gated-repositories>
|
|
```
|
|
|
|
Then run:
|
|
|
|
```bash
|
|
docker compose -f docker-compose.node-agent.yml up -d --build
|
|
docker compose -f docker-compose.node-agent.yml logs --tail=100 node-agent
|
|
```
|
|
|
|
With the private-CA overlay, use both files for every lifecycle command:
|
|
|
|
```bash
|
|
docker compose -f docker-compose.node-agent.yml \
|
|
-f docker-compose.node-agent.private-ca.yml up -d --build
|
|
```
|
|
|
|
After the node appears online, remove `MODELFORGE_AGENT_ENROLLMENT_TOKEN` from `.env` and recreate
|
|
the service. The node-scoped credential persists in `node-agent-state`:
|
|
|
|
```bash
|
|
docker compose -f docker-compose.node-agent.yml up -d --force-recreate node-agent
|
|
```
|
|
|
|
Do not delete the state volume during ordinary upgrades; doing so discards identity and credential.
|
|
Use the UI/API to revoke the old credential before deliberately re-enrolling.
|
|
|
|
For M3, register the host cache path as a storage root and map it to
|
|
`/data/artifacts/model-registry`. The host path and container path are deliberately separate facts.
|
|
Never substitute `/mnt/user` when the cache capacity guard fails. The agent advertises
|
|
`artifact.acquire.v1`, polls one typed job at a time, and retains job-scoped partials below the
|
|
approved root for bounded retry. Keep the optional Hub token only in secret environment/config;
|
|
do not place it in Compose files or logs.
|
|
|
|
## 3. Verify and operate
|
|
|
|
Confirm the UI shows the expected hostname, agent/protocol version, heartbeat age, CPU/RAM/storage,
|
|
GPU UUID/model/VRAM/driver and measured telemetry. Restart the agent and confirm node and accelerator
|
|
UUIDs remain unchanged. Stop it long enough to observe `stale` then `offline`, restart it and confirm
|
|
`online` plus a return audit event.
|
|
|
|
For temporary outages the agent retains its identity/credential and retries with bounded exponential
|
|
backoff. Rotate with the authenticated admin rotation endpoint, securely replace
|
|
`/data/state/node-credential` with its one-time response, and recreate the agent. Revoke without a
|
|
replacement when a credential may be compromised. Disable a node to make it ineligible and immediately
|
|
reject further publications. Upgrades are ordinary image
|
|
rebuild/recreate operations; protocol-incompatible agents are visibly rejected instead of silently
|
|
downgraded.
|
|
|
|
## RTX 4080 acceptance checklist
|
|
|
|
On the actual target server, capture: `nvidia-smi` model/UUID/driver output, node-agent container
|
|
status, `/api/v1/hardware/nodes/{id}` output with secrets removed, a restart identity comparison,
|
|
stale/offline/recovery evidence, and UI screenshots. The release verdict may not claim RTX 4080
|
|
validation until these observations exist.
|