ModelForge/CHANGELOG.md

# Changelog

Operator- and user-facing changes. Internal refactoring, test additions and documentation-only
commits are not listed unless they change what an operator sees or does.

## Unreleased

### Security

- Runtime Worker packaging now upgrades the digest-pinned Ubuntu base before installation and pins
  the inference stack plus inherited Python security dependencies to reviewed fixed versions. The
  public-candidate gate scans the resulting fourth image instead of treating the private worker as
  an unexamined exception.
- The Runtime Worker removes unneeded kernel-development headers after installation, eliminating
  their upstream-unfixed vulnerability surface without removing required runtime libraries.
- Public candidate scans distinguish fixable or newly observed HIGH/CRITICAL findings from an
  exact reviewed upstream-unfixed baseline. A changed CVE, package, installed version, severity or
  newly available fix fails closed.
- Node enrollment and publisher credentials now require their exact fixed scopes at both service
  and database boundaries. Schema `20260830_0023` validates every existing row before adding the
  constraints and refuses the upgrade when malformed legacy scope data exists.
- Request-body streaming now has finite ASGI event/progress budgets in addition to byte limits;
  progressless frame floods receive the typed `request_body_progress_exhausted` response.
- PostgreSQL role provisioning now removes the database's default public temporary-table grant;
  production startup continues to attest that the runtime role cannot create or use it.

### Fixed

- Fresh schema `20260830_0024` installs no longer let Psycopg misread PostgreSQL `%rowtype` and
  `%I` syntax as client placeholders.
- Public-candidate acceptance now works across the Gitea sibling Docker daemon while retaining
  isolated volumes, private workers, exact HTTP contract probes and deterministic cleanup.

## v1.2.1 — 2026-08-30

A packaging and provenance patch. No feature, API, schema or runtime-behaviour change; schema stays
`20260828_0022` and no migration runs. It exists because v1.2.0's production acceptance found two
defects that every existing gate was blind to — both in the artifacts rather than the source.

### Fixed

- **The published console could not reach its own API.** Vite inlines `VITE_API_BASE_URL` at build
  time, but the release build never passed it, so every release image since v1.1.0 compiled the
  Dockerfile's development default `http://localhost:8000` into an immutable bundle — and the nginx
  CSP, derived from the same argument, hardcoded the same wrong origin. A release build now requires
  an explicit `--public-api-origin` (or `MODELFORGE_PUBLIC_API_ORIGIN`) and refuses to package
  without one.
- **The Node Agent's identity could drift from its tag.** `docker-compose.node-agent.yml` declared a
  build with no arguments, so a Compose-built agent carried version `0.0.0` and empty
  revision/created labels while still being tagged from `MODELFORGE_VERSION`. Production ran an
  image tagged `1.1.1` whose contents were `1.2.0`. The projection now passes the release identity,
  as the API and console projections already did.

### Added

- `scripts/release_image_acceptance.py`: a gate that inspects the **built** console image — its
  compiled bundle, its rendered CSP and its OCI labels — rather than the source that produced it.
  The release build runs it before packaging and refuses to publish an image that cannot reach the
  API it was built for. Verified against the exact image that broke production.
- Regression coverage for both defects: the release build fails closed without an origin, the origin
  reaches the console build, the CSP names the same origin and is never widened to a wildcard, every
  release image declares complete OCI labels, and every Compose projection that builds a published
  image passes the release identity.

### Operations

- The release manifest now records the console's compiled-in API origin, so an operator can see it
  without unpacking the image.
- Local development is unchanged: `docker compose up` still defaults to `http://localhost:8000`.
  Fail-closed applies to the release path only.

## v1.2.0 — 2026-08-29

The Premium Operator Console release. No API contract, domain model, runtime behavior or database
schema changed; schema stays `20260828_0022` and no migration runs.

### Added

- A production command center built only from backend-authoritative evidence, stating unknowns as
  unknown rather than as zero.
- Quick jump (Ctrl+K): a modal command palette over all 13 workspaces, matching name, domain and
  keywords.
- An evidence timeline of recent control-plane activity, each entry linking to the workspace where
  it can be verified.
- A split-pane Registry detail view covering upstream facts, local governance, revisions, artifacts
  and provenance.
- One console-wide guarded destructive-action dialog showing identity, dependency posture and
  permanent impact before an explicit acknowledgement unlocks the confirm control.

### Changed

- Workspaces are grouped by operator mental model (Command, Model supply, Serving, Infrastructure,
  Assurance, Projects) while every existing route hash is preserved.
- One refined design token layer for light and dark, with semantic status roles that always include
  text and explicit focus rings.
- Every route is usable from 1440 px down to a 390 px mobile viewport through a modal navigation
  drawer, with no horizontal overflow at any tested width.

### Fixed

- The ARIA tabs pattern is complete across all six tabbed workspaces: every tab carries a stable
  `id` and `aria-controls`, and every panel is a real `role="tabpanel"` naming its tab back.
- Lifecycle, Migrations and Recovery tablists gained roving `tabindex`, an accessible tablist name
  and Left/Right/Home/End keyboard movement, which they previously lacked entirely.
- Capability residency unload no longer uses a native `window.confirm`; it uses the console safety
  dialog and shows the VRAM released and the cold-load cost the next request will pay.
- Operator token fields in Operations, Recovery, Lifecycle and Migrations now sit in a real form
  with a stable `id`, a `name` and an explicit `<label for>`.

### Accessibility

- Skip link to a programmatically focusable main landmark.
- Quick jump and the mobile drawer contain focus, dismiss on Escape and restore focus to the control
  that opened them.
- Status is never encoded by colour alone.
- `prefers-reduced-motion` collapses animation and transition durations; `forced-colors: active`
  re-expresses focus and active navigation in system colours.
- Essential mobile controls and navigation rows meet a 44 px target.

### UX

- Warnings state the observed condition and name the workspace where an operator can resolve it.
- Destructive registry and residency actions present impact before they present a confirm control.
- Surfaces use compact rows, quiet borders and a restrained status palette rather than decorative
  effect.

### Security / Operations

- The operator credential stays request-scoped: no `localStorage`, no `sessionStorage`, no cookie,
  no URL. Every token form prevents its own default submission so nothing is serialized into a query
  string.
- `autocomplete="off"` is retained deliberately on operator-token fields rather than moving to
  `current-password`, which would invite a password manager to persist and sync a high-privilege
  control-plane credential.
- The Node Agent's functional behavior is unchanged from the NVIDIA-certified v1.1.1
  implementation; it is rebuilt only because `VERSION` is the single source of truth for packaged
  manifests. No re-enrollment is required.

### Known limitations

- The console has no committed visual-regression baseline; visual acceptance is a human-reviewed
  browser pass.
- Point-in-time database recovery remains `NOT_SUPPORTED`.

## v1.1.1 — 2026-08-29

### Fixed

- Replaced the Node Agent's Alpine/musl runtime with a digest-pinned Debian/glibc runtime so the
  NVIDIA Container Toolkit can inject and load GPU Node's glibc-linked NVML driver library.
- NVIDIA-configured agents now validate real NVML inventory and matching telemetry before
  enrollment or publication. Missing NVML, an empty device list, or missing GPU telemetry exits
  with a typed diagnostic instead of publishing an apparently healthy zero-GPU node.

### Security / Operations

- The Node Agent remains UID 100/GID 101, read-only, capability-free and
  `no-new-privileges`; its multi-stage image contains no compiler or package installer.
- CPU-only nodes remain supported through the explicit `cpu` accelerator mode. Canonical GPU
  Compose projections set `nvidia` and fail closed.

### Known limitations

- External workloads can legitimately trigger `EXTERNAL_GPU_PRESSURE`; ModelForge observes but
  does not terminate those workloads.

## v1.1.0 — 2026-08-28

### Added

- Operator-only compute-node decommission preview and execute APIs, with exhaustive fail-closed
  blockers, generation/digest concurrency checks, idempotent exactly-once audit and retained
  provenance.
- A Console danger-zone flow that explains blockers, cleanup and preserved history before requiring
  a reason, operator and exact typed node identity.
- A terminal node invariant and runbook. Decommissioned identities cannot ordinarily re-enroll,
  rotate credentials, publish agent state, refresh inventory or return to scheduler placement.

### Changed

- The runtime schema target is `20260828_0022`. Schema `20260827_0021` remains a supported
  upgrade source, not a runtime-compatible schema; rollback requires the verified pre-upgrade
  backup because downgrade migrations are intentionally unsupported.
- The Console now uses shared design tokens, persistent light/dark themes, grouped responsive
  navigation, page-specific headers and keyboard-visible focus.

### Fixed

- API clients preserve typed error envelopes and correlation IDs, and accept successful empty HTTP
  responses without attempting to decode JSON.
- Failure and recovery states retain meaningful visual styling in both themes.
- The Node Agent Compose projection accepts an exact release image tag or digest while retaining an
  explicitly local source-build workflow; it never falls back to `latest`.

### Security / Operations

- Decommission revokes active node credentials, prevents old identities from resurrecting, removes
  disposable current state and preserves historical provenance and an exactly-once audit event.
- Release tooling binds versioned images, CycloneDX SBOM, provenance and checksums to an exact source
  commit without mutating floating `latest` tags.
- Published image bases are digest-pinned and security-updated; Python runtime images remove pip and
  setuptools after dependency verification, and all three components retain dedicated non-root
  runtime users.

### Known limitations

- Point-in-time recovery remains `NOT_SUPPORTED`; verified snapshot restore is the recovery model.
- SQLite remains an isolated unit-test backend and is not supported for production.
- One historical M13/M15 LAB cutover remains explicitly unreconciled because external Qdrant truth
  is unavailable; it has no production deployment, gateway or scheduler dependency.

## v1.0.0 — 2026-08-27

The first stable release. See
[docs/RELEASE_NOTES_v1.0.0.md](docs/RELEASE_NOTES_v1.0.0.md) for the full description.

### Added

- One authoritative product version, `1.0.0`, with every packaged manifest checked against it.
- `GET /api/v1/version` — build identity and compatibility, unauthenticated.
- Declared compatibility ranges for the database schema and the agent protocol, answered as
  `COMPATIBLE`, `TOO_OLD`, `TOO_NEW` or `UNKNOWN` — never a silent pass.
- Fail-closed startup validation in production: missing or weak operator key, development database
  password, remote code execution, missing backup encryption key, wildcard CORS origin, invalid URL,
  unwritable storage root, incompatible schema or PostgreSQL major, and impossible policy
  combinations.
- A production deployment overlay that refuses to render without its secrets.
- `scripts/preflight.py`, `scripts/bootstrap.py`, `scripts/upgrade.py` and
  `scripts/release_build.py`.
- A Content-Security-Policy on the operator console.
- OCI image labels and build identity arguments on all three images.
- Configurable host ports for every published service.
- Operator documentation: installation, first run, upgrade, configuration, compatibility, node
  agent, Unraid deployment, capabilities, project integration, operations and troubleshooting.

### Changed

- `GET /api/v1/system` reports `release_channel` instead of `milestone`.
- The console no longer displays development milestone labels.
- `.env.example` and the configuration reference are generated from the typed settings.
- The Runtime Worker's inference stack moved to an optional `inference` extra, so lint, typecheck
  and its tests no longer require the full PyTorch stack.

### Fixed

- **A clean clone could not start.** A `models/` ignore rule intended for model weights also matched
  `config/models/`, so the candidate registry manifest was never in the repository.
- **Migrations ran against the wrong database.** The Alembic environment discarded an explicitly
  supplied URL and used the configured one, so an upgrade rehearsal aimed at an isolated copy would
  have migrated the deployment's own database.
- **The disaster-recovery projection published its PostgreSQL to the LAN**, reachable as a superuser
  with development credentials.
- **The readiness probe re-read every manifest on every call** — 77 file reads and 528 ms per probe.
- **The Node Agent pinned an old control-plane version**, making its image unbuildable at release.
- **The console rendered a "not implemented" panel over the delivered Recovery workspace**, because
  the placeholder list had drifted from the navigation.
- **Managed validation reported success without running any test suite.**

### Known limitations

Point-in-time recovery is `NOT_SUPPORTED`. Upgrades require downtime. Model Lab and Benchmarks have
no console workspace. See the release notes for the full list.