ModelForge/CHANGELOG.md

Changelog

Operator- and user-facing changes. Internal refactoring, test additions and documentation-only commits are not listed unless they change what an operator sees or does.

Unreleased

Security

• Runtime Worker packaging now upgrades the digest-pinned Ubuntu base before installation and pins the inference stack plus inherited Python security dependencies to reviewed fixed versions. The public-candidate gate scans the resulting fourth image instead of treating the private worker as an unexamined exception.
• The Runtime Worker removes unneeded kernel-development headers after installation, eliminating their upstream-unfixed vulnerability surface without removing required runtime libraries.
• Public candidate scans distinguish fixable or newly observed HIGH/CRITICAL findings from an exact reviewed upstream-unfixed baseline. A changed CVE, package, installed version, severity or newly available fix fails closed.
• Node enrollment and publisher credentials now require their exact fixed scopes at both service and database boundaries. Schema 20260830_0023 validates every existing row before adding the constraints and refuses the upgrade when malformed legacy scope data exists.
• Request-body streaming now has finite ASGI event/progress budgets in addition to byte limits; progressless frame floods receive the typed request_body_progress_exhausted response.
• PostgreSQL role provisioning now removes the database's default public temporary-table grant; production startup continues to attest that the runtime role cannot create or use it.

Fixed

• Fresh schema 20260830_0024 installs no longer let Psycopg misread PostgreSQL %rowtype and %I syntax as client placeholders.
• Public-candidate acceptance now works across the Gitea sibling Docker daemon while retaining isolated volumes, private workers, exact HTTP contract probes and deterministic cleanup.

v1.2.1 — 2026-08-30

A packaging and provenance patch. No feature, API, schema or runtime-behaviour change; schema stays 20260828_0022 and no migration runs. It exists because v1.2.0's production acceptance found two defects that every existing gate was blind to — both in the artifacts rather than the source.

Fixed

• The published console could not reach its own API. Vite inlines VITE_API_BASE_URL at build time, but the release build never passed it, so every release image since v1.1.0 compiled the Dockerfile's development default http://localhost:8000 into an immutable bundle — and the nginx CSP, derived from the same argument, hardcoded the same wrong origin. A release build now requires an explicit --public-api-origin (or MODELFORGE_PUBLIC_API_ORIGIN) and refuses to package without one.
• The Node Agent's identity could drift from its tag. docker-compose.node-agent.yml declared a build with no arguments, so a Compose-built agent carried version 0.0.0 and empty revision/created labels while still being tagged from MODELFORGE_VERSION. Production ran an image tagged 1.1.1 whose contents were 1.2.0. The projection now passes the release identity, as the API and console projections already did.

Added

• scripts/release_image_acceptance.py: a gate that inspects the built console image — its compiled bundle, its rendered CSP and its OCI labels — rather than the source that produced it. The release build runs it before packaging and refuses to publish an image that cannot reach the API it was built for. Verified against the exact image that broke production.
• Regression coverage for both defects: the release build fails closed without an origin, the origin reaches the console build, the CSP names the same origin and is never widened to a wildcard, every release image declares complete OCI labels, and every Compose projection that builds a published image passes the release identity.

Operations

• The release manifest now records the console's compiled-in API origin, so an operator can see it without unpacking the image.
• Local development is unchanged: docker compose up still defaults to http://localhost:8000. Fail-closed applies to the release path only.

v1.2.0 — 2026-08-29

The Premium Operator Console release. No API contract, domain model, runtime behavior or database schema changed; schema stays 20260828_0022 and no migration runs.

Added

• A production command center built only from backend-authoritative evidence, stating unknowns as unknown rather than as zero.
• Quick jump (Ctrl+K): a modal command palette over all 13 workspaces, matching name, domain and keywords.
• An evidence timeline of recent control-plane activity, each entry linking to the workspace where it can be verified.
• A split-pane Registry detail view covering upstream facts, local governance, revisions, artifacts and provenance.
• One console-wide guarded destructive-action dialog showing identity, dependency posture and permanent impact before an explicit acknowledgement unlocks the confirm control.

Changed

• Workspaces are grouped by operator mental model (Command, Model supply, Serving, Infrastructure, Assurance, Projects) while every existing route hash is preserved.
• One refined design token layer for light and dark, with semantic status roles that always include text and explicit focus rings.
• Every route is usable from 1440 px down to a 390 px mobile viewport through a modal navigation drawer, with no horizontal overflow at any tested width.

Fixed

• The ARIA tabs pattern is complete across all six tabbed workspaces: every tab carries a stable id and aria-controls, and every panel is a real role="tabpanel" naming its tab back.
• Lifecycle, Migrations and Recovery tablists gained roving tabindex, an accessible tablist name and Left/Right/Home/End keyboard movement, which they previously lacked entirely.
• Capability residency unload no longer uses a native window.confirm; it uses the console safety dialog and shows the VRAM released and the cold-load cost the next request will pay.
• Operator token fields in Operations, Recovery, Lifecycle and Migrations now sit in a real form with a stable id, a name and an explicit <label for>.

Accessibility

• Skip link to a programmatically focusable main landmark.
• Quick jump and the mobile drawer contain focus, dismiss on Escape and restore focus to the control that opened them.
• Status is never encoded by colour alone.
• prefers-reduced-motion collapses animation and transition durations; forced-colors: active re-expresses focus and active navigation in system colours.
• Essential mobile controls and navigation rows meet a 44 px target.

UX

• Warnings state the observed condition and name the workspace where an operator can resolve it.
• Destructive registry and residency actions present impact before they present a confirm control.
• Surfaces use compact rows, quiet borders and a restrained status palette rather than decorative effect.

Security / Operations

• The operator credential stays request-scoped: no localStorage, no sessionStorage, no cookie, no URL. Every token form prevents its own default submission so nothing is serialized into a query string.
• autocomplete="off" is retained deliberately on operator-token fields rather than moving to current-password, which would invite a password manager to persist and sync a high-privilege control-plane credential.
• The Node Agent's functional behavior is unchanged from the NVIDIA-certified v1.1.1 implementation; it is rebuilt only because VERSION is the single source of truth for packaged manifests. No re-enrollment is required.

Known limitations

• The console has no committed visual-regression baseline; visual acceptance is a human-reviewed browser pass.
• Point-in-time database recovery remains NOT_SUPPORTED.

v1.1.1 — 2026-08-29

Fixed

• Replaced the Node Agent's Alpine/musl runtime with a digest-pinned Debian/glibc runtime so the NVIDIA Container Toolkit can inject and load GPU Node's glibc-linked NVML driver library.
• NVIDIA-configured agents now validate real NVML inventory and matching telemetry before enrollment or publication. Missing NVML, an empty device list, or missing GPU telemetry exits with a typed diagnostic instead of publishing an apparently healthy zero-GPU node.

Security / Operations

• The Node Agent remains UID 100/GID 101, read-only, capability-free and no-new-privileges; its multi-stage image contains no compiler or package installer.
• CPU-only nodes remain supported through the explicit cpu accelerator mode. Canonical GPU Compose projections set nvidia and fail closed.

Known limitations

• External workloads can legitimately trigger EXTERNAL_GPU_PRESSURE; ModelForge observes but does not terminate those workloads.

v1.1.0 — 2026-08-28

Added

• Operator-only compute-node decommission preview and execute APIs, with exhaustive fail-closed blockers, generation/digest concurrency checks, idempotent exactly-once audit and retained provenance.
• A Console danger-zone flow that explains blockers, cleanup and preserved history before requiring a reason, operator and exact typed node identity.
• A terminal node invariant and runbook. Decommissioned identities cannot ordinarily re-enroll, rotate credentials, publish agent state, refresh inventory or return to scheduler placement.

Changed

• The runtime schema target is 20260828_0022. Schema 20260827_0021 remains a supported upgrade source, not a runtime-compatible schema; rollback requires the verified pre-upgrade backup because downgrade migrations are intentionally unsupported.
• The Console now uses shared design tokens, persistent light/dark themes, grouped responsive navigation, page-specific headers and keyboard-visible focus.

Fixed

• API clients preserve typed error envelopes and correlation IDs, and accept successful empty HTTP responses without attempting to decode JSON.
• Failure and recovery states retain meaningful visual styling in both themes.
• The Node Agent Compose projection accepts an exact release image tag or digest while retaining an explicitly local source-build workflow; it never falls back to latest.

Security / Operations

• Decommission revokes active node credentials, prevents old identities from resurrecting, removes disposable current state and preserves historical provenance and an exactly-once audit event.
• Release tooling binds versioned images, CycloneDX SBOM, provenance and checksums to an exact source commit without mutating floating latest tags.
• Published image bases are digest-pinned and security-updated; Python runtime images remove pip and setuptools after dependency verification, and all three components retain dedicated non-root runtime users.

Known limitations

• Point-in-time recovery remains NOT_SUPPORTED; verified snapshot restore is the recovery model.
• SQLite remains an isolated unit-test backend and is not supported for production.
• One historical M13/M15 LAB cutover remains explicitly unreconciled because external Qdrant truth is unavailable; it has no production deployment, gateway or scheduler dependency.

v1.0.0 — 2026-08-27

The first stable release. See docs/RELEASE_NOTES_v1.0.0.md for the full description.

Added

• One authoritative product version, 1.0.0, with every packaged manifest checked against it.
• GET /api/v1/version — build identity and compatibility, unauthenticated.
• Declared compatibility ranges for the database schema and the agent protocol, answered as COMPATIBLE, TOO_OLD, TOO_NEW or UNKNOWN — never a silent pass.
• Fail-closed startup validation in production: missing or weak operator key, development database password, remote code execution, missing backup encryption key, wildcard CORS origin, invalid URL, unwritable storage root, incompatible schema or PostgreSQL major, and impossible policy combinations.
• A production deployment overlay that refuses to render without its secrets.
• scripts/preflight.py, scripts/bootstrap.py, scripts/upgrade.py and scripts/release_build.py.
• A Content-Security-Policy on the operator console.
• OCI image labels and build identity arguments on all three images.
• Configurable host ports for every published service.
• Operator documentation: installation, first run, upgrade, configuration, compatibility, node agent, Unraid deployment, capabilities, project integration, operations and troubleshooting.

Changed

• GET /api/v1/system reports release_channel instead of milestone.
• The console no longer displays development milestone labels.
• .env.example and the configuration reference are generated from the typed settings.
• The Runtime Worker's inference stack moved to an optional inference extra, so lint, typecheck and its tests no longer require the full PyTorch stack.

Fixed

• A clean clone could not start. A models/ ignore rule intended for model weights also matched config/models/, so the candidate registry manifest was never in the repository.
• Migrations ran against the wrong database. The Alembic environment discarded an explicitly supplied URL and used the configured one, so an upgrade rehearsal aimed at an isolated copy would have migrated the deployment's own database.
• The disaster-recovery projection published its PostgreSQL to the LAN, reachable as a superuser with development credentials.
• The readiness probe re-read every manifest on every call — 77 file reads and 528 ms per probe.
• The Node Agent pinned an old control-plane version, making its image unbuildable at release.
• The console rendered a "not implemented" panel over the delivered Recovery workspace, because the placeholder list had drifted from the navigation.
• Managed validation reported success without running any test suite.

Known limitations

Point-in-time recovery is NOT_SUPPORTED. Upgrades require downtime. Model Lab and Benchmarks have no console workspace. See the release notes for the full list.