2.0 KiB
Capability Gateway
M9 routes
The gateway exposes model-neutral document.ocr@1, vision.embedding@1, and speech.transcription@1 routes. Each has modality-specific validation, scoped service-client authorization and transient content handling. The stable text embedding route remains exactly 1024-dimensional; opaque LAB experiment routes may return a different declared EmbeddingSpace dimension without weakening the stable contract.
Background consumers
Service clients carry a fixed workload priority used on Gateway request records, GPU leases and
serving jobs. ExampleRAG shadow work uses background; callers cannot supply or escalate this field.
Inputs remain digest-only in durable records and transient in Redis.
The ModelForge Gateway is the only project-facing inference boundary. A client requests the
versioned rag.embedding@1 contract; it cannot provide an artifact path, runtime adapter or
concrete upstream model. Resolution is capability → stable CapabilityDeployment → RuntimeProfile
→ exact ArtifactSet → eligible ComputeNode.
The native endpoint is POST /api/v1/capabilities/rag.embedding@1/invoke. The supported
OpenAI-compatible subset is POST /v1/embeddings with logical model alias rag.embedding.
Authentication, scope/rate checks, bounded queueing, a database-backed GPU lease, residency and
the outbound worker job all execute before a response is returned. Typed failures never fall back
to cloud inference.
M10 places the shared cross-capability scheduler between authenticated resolution and the worker
job. A caller still selects only a capability; node, accelerator, resident eviction and priority are
server-authoritative. Capability health remains distinct from temporary scheduling readiness such
as READY_RESIDENT, READY_ON_DEMAND, WAITING_FOR_CAPACITY, POLICY_BLOCKED,
NODE_UNAVAILABLE, and RUNTIME_UNHEALTHY.
Request bodies live only in Redis with a short TTL while queued/executing. PostgreSQL retains a request ID, input hash, byte/count metadata, timings and failure class, never the input text.