System Architecture

1. Product boundary

ModelForge is the control plane for local AI inference infrastructure. It is responsible for model discovery, provenance, verification, lifecycle, runtime configuration, resource scheduling, project bindings, capability routing, experiments, benchmarks, upgrade advice and auditability.

It is not a model-training cluster, a public AI cloud, a generic chat application, or an orchestration abstraction that hides all operational details.

2. Logical layers

Web UI

Operator-facing control surface. It presents state and evidence but must not invent state independently of the API.

Control Plane API

Canonical source of truth for registry state, projects, capability contracts, runtime profiles, benchmark evidence, recommendations, lifecycle actions and audit events.

AI Gateway

Stable project-facing surface. Resolves project identity + capability + channel to a concrete deployment and enforces policy, queueing and observability.

Scheduler

Controls finite accelerator resources. It handles VRAM leases, residency, load/sleep/unload, priority classes, compatibility constraints and fallback policy.

Runtime adapters

Each runtime implements a stable internal contract. Initial adapters:

vLLM — LLM/VLM/embedding/reranking/ASR where appropriate
Transformers — specialized Hugging Face inference
Diffusers — image generation/editing
llama.cpp — GGUF and low-memory fallback

Artifact pipeline

Separates internet-facing acquisition from approved inference artifacts. It manages upstream source, revision resolution, content hashes, quarantine, verification and derived artifacts.

Hardware plane

Collects accelerator and node information. A ComputeNode has zero or more Accelerator objects. Each compute server runs an outbound-only, authenticated observer agent; the central control plane is not itself a node unless explicitly configured as one. The initial product still optimizes scheduling for one NVIDIA GPU while identity, health and persistence support multiple nodes.

3. Request flow

Project
  → ModelForge Gateway
  → authenticate project
  → resolve capability contract
  → resolve deployment channel
  → policy evaluation
  → scheduler resource lease
  → runtime adapter
  → inference worker
  → telemetry + audit
  → response

4. Control flow for candidate upgrades

Hub discovery
  → candidate registration
  → metadata + license analysis
  → download
  → quarantine
  → artifact verification
  → compatibility probe
  → generic local benchmark
  → project-specific benchmark
  → advisor evaluation
  → operator approval
  → canary or migration
  → promotion
  → rollback retention

5. Required isolation boundaries

Project clients never call model workers directly.
Model runtimes do not receive Hugging Face tokens or application secrets.
Download/quarantine components may have internet access; inference workers should not by default.
Artifact mounts are read-only in inference workers.
Control-plane write actions are auditable.
Lab-only remote-code exceptions cannot silently become production deployments.

6. Nonfunctional targets

The initial product should remain understandable and operable by one person on a home infrastructure server. Prefer explicitness over distributed-system complexity. Every abstraction must justify itself through future-safe model lifecycle, not architectural fashion.

3.4 KiB Raw Blame History