94 lines
3.4 KiB
Markdown
94 lines
3.4 KiB
Markdown
# System Architecture
|
|
|
|
## 1. Product boundary
|
|
|
|
ModelForge is the control plane for local AI inference infrastructure. It is responsible for model discovery, provenance, verification, lifecycle, runtime configuration, resource scheduling, project bindings, capability routing, experiments, benchmarks, upgrade advice and auditability.
|
|
|
|
It is **not** a model-training cluster, a public AI cloud, a generic chat application, or an orchestration abstraction that hides all operational details.
|
|
|
|
## 2. Logical layers
|
|
|
|
### Web UI
|
|
|
|
Operator-facing control surface. It presents state and evidence but must not invent state independently of the API.
|
|
|
|
### Control Plane API
|
|
|
|
Canonical source of truth for registry state, projects, capability contracts, runtime profiles, benchmark evidence, recommendations, lifecycle actions and audit events.
|
|
|
|
### AI Gateway
|
|
|
|
Stable project-facing surface. Resolves project identity + capability + channel to a concrete deployment and enforces policy, queueing and observability.
|
|
|
|
### Scheduler
|
|
|
|
Controls finite accelerator resources. It handles VRAM leases, residency, load/sleep/unload, priority classes, compatibility constraints and fallback policy.
|
|
|
|
### Runtime adapters
|
|
|
|
Each runtime implements a stable internal contract. Initial adapters:
|
|
|
|
- vLLM — LLM/VLM/embedding/reranking/ASR where appropriate
|
|
- Transformers — specialized Hugging Face inference
|
|
- Diffusers — image generation/editing
|
|
- llama.cpp — GGUF and low-memory fallback
|
|
|
|
### Artifact pipeline
|
|
|
|
Separates internet-facing acquisition from approved inference artifacts. It manages upstream source, revision resolution, content hashes, quarantine, verification and derived artifacts.
|
|
|
|
### Hardware plane
|
|
|
|
Collects accelerator and node information. A `ComputeNode` has zero or more `Accelerator` objects.
|
|
Each compute server runs an outbound-only, authenticated observer agent; the central control plane is
|
|
not itself a node unless explicitly configured as one. The initial product still optimizes scheduling
|
|
for one NVIDIA GPU while identity, health and persistence support multiple nodes.
|
|
|
|
## 3. Request flow
|
|
|
|
```text
|
|
Project
|
|
→ ModelForge Gateway
|
|
→ authenticate project
|
|
→ resolve capability contract
|
|
→ resolve deployment channel
|
|
→ policy evaluation
|
|
→ scheduler resource lease
|
|
→ runtime adapter
|
|
→ inference worker
|
|
→ telemetry + audit
|
|
→ response
|
|
```
|
|
|
|
## 4. Control flow for candidate upgrades
|
|
|
|
```text
|
|
Hub discovery
|
|
→ candidate registration
|
|
→ metadata + license analysis
|
|
→ download
|
|
→ quarantine
|
|
→ artifact verification
|
|
→ compatibility probe
|
|
→ generic local benchmark
|
|
→ project-specific benchmark
|
|
→ advisor evaluation
|
|
→ operator approval
|
|
→ canary or migration
|
|
→ promotion
|
|
→ rollback retention
|
|
```
|
|
|
|
## 5. Required isolation boundaries
|
|
|
|
- Project clients never call model workers directly.
|
|
- Model runtimes do not receive Hugging Face tokens or application secrets.
|
|
- Download/quarantine components may have internet access; inference workers should not by default.
|
|
- Artifact mounts are read-only in inference workers.
|
|
- Control-plane write actions are auditable.
|
|
- Lab-only remote-code exceptions cannot silently become production deployments.
|
|
|
|
## 6. Nonfunctional targets
|
|
|
|
The initial product should remain understandable and operable by one person on a home infrastructure server. Prefer explicitness over distributed-system complexity. Every abstraction must justify itself through future-safe model lifecycle, not architectural fashion.
|