commit 7082ab955a939a10eca9cf72fdcf72a5db7e7804 Author: Jens Date: Tue Sep 1 21:30:16 2026 +0200 Initial public ModelForge release diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..a22e22e --- /dev/null +++ b/.dockerignore @@ -0,0 +1,13 @@ +.git +.env +**/__pycache__ +**/.pytest_cache +**/.mypy_cache +**/.ruff_cache +**/*.pyc +frontend/node_modules +frontend/dist +frontend/tsconfig.tsbuildinfo +backend/tests +node-agent/tests +docs diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..34c752e --- /dev/null +++ b/.env.example @@ -0,0 +1,342 @@ +# ITWorx ModelForge 1.2.1 — configuration example +# +# Generated by scripts/generate_configuration_docs.py. Do not edit by hand. +# Copy to .env and fill in the values marked REQUIRED. See docs/CONFIGURATION.md. +# +# Secrets are intentionally empty here. This file is committed to the repository, so a +# real value placed in it would be published with the release. + +# Deployment profile. 'production' turns on every fail-closed startup rule; 'development' and 'test' report the same problems without refusing to start. +MODELFORGE_ENV=development + +# Interface the API binds inside its container. Leave at 0.0.0.0. +MODELFORGE_API_HOST=0.0.0.0 + +# Port the API listens on inside its container. +MODELFORGE_API_PORT=8000 + +# Pre-parser request-body limit for public and operator control-plane routes. +MODELFORGE_CONTROL_PLANE_MAX_PAYLOAD_BYTES=1048576 + +# Name this process reports in logs and audit events. +MODELFORGE_SERVICE_NAME=modelforge-api + +# Structured log level: DEBUG, INFO, WARNING or ERROR. +MODELFORGE_LOG_LEVEL=INFO + +# SQLAlchemy URL for the API's non-owner modelforge_runtime role. It may read the audit trail and execute the canonical append function, but cannot mutate audit tables directly. (REQUIRED in production) +MODELFORGE_DATABASE_URL= + +# SQLAlchemy URL for the non-superuser modelforge schema-owner role. Set only in the one-shot migration process; production API startup refuses when this secret is present. +MODELFORGE_MIGRATION_DATABASE_URL= + +# Redis URL for transient request payloads and queues. (REQUIRED in production) +MODELFORGE_REDIS_URL=redis://localhost:6379/0 + +# Comma-separated exact origins allowed to call the API from a browser. A wildcard is refused in production because requests are credentialed. +MODELFORGE_CORS_ORIGINS=http://localhost:3000 + +# Operator API key guarding every admin route. Generate at least 32 random characters; ModelForge never mints one for you. (REQUIRED in production) +MODELFORGE_OPERATOR_API_KEY= + +# Optional Hugging Face token, used only for acquiring gated repositories. It is never passed to a runtime and never leaves the control plane. +MODELFORGE_HF_TOKEN= + +# Base64 AES-256 key for backup encryption. Without it no backup can be produced, and without the same key no backup can be restored — store it outside this deployment. (REQUIRED in production) +MODELFORGE_BACKUP_ENCRYPTION_KEY= + +# Identifier recorded in each backup manifest so a restore can name the key it needs. +MODELFORGE_BACKUP_ENCRYPTION_KEY_ID=modelforge-backup-key-1 + +# Hugging Face cache root inside the container. +MODELFORGE_HF_HOME=/data/hf-cache + +# Verified model artifact root. Must exist and be writable. +MODELFORGE_ARTIFACT_ROOT=/data/artifacts + +# Where acquired artifacts are held until their checks pass. +MODELFORGE_QUARANTINE_ROOT=/data/quarantine + +# Artifact root as a runtime worker sees it on a compute node. +MODELFORGE_RUNTIME_ARTIFACT_ROOT=/models/model-registry + +# Backup destination. Must exist and be writable, or backups fail closed. +MODELFORGE_BACKUP_ROOT=/data/backups + +# Working directory a restore stages into before it commits. +MODELFORGE_BACKUP_RESTORE_ROOT=/data/restore + +# Per-request timeout for Hugging Face metadata calls. +MODELFORGE_HF_TIMEOUT_SECONDS=30 + +# How long a resolved upstream snapshot stays cached. +MODELFORGE_HF_SNAPSHOT_TTL_SECONDS=3600 + +# Whether model repositories may execute their own Python. Always false in production; startup refuses any other value there. +MODELFORGE_ALLOW_REMOTE_CODE=false + +# Collect GPU telemetry on this host. +MODELFORGE_ENABLE_GPU_TELEMETRY=true + +# Run a hardware inventory pass when the process starts. +MODELFORGE_HARDWARE_REFRESH_ON_STARTUP=false + +# Interval between hardware inventory passes. +MODELFORGE_HARDWARE_POLL_INTERVAL_SECONDS=30 + +# Explicit node identity. Leave empty to use the persisted file. +MODELFORGE_NODE_IDENTITY= + +# 'persisted' keeps a node's identity across restarts; 'auto' derives it. +MODELFORGE_NODE_IDENTITY_MODE=auto + +# Where a persisted node identity is stored. +MODELFORGE_NODE_IDENTITY_FILE=/data/state/node-id + +# Silence after which a node is considered stale. +MODELFORGE_NODE_STALE_AFTER_SECONDS=30 + +# Silence after which a node is considered offline. Must exceed the stale threshold. +MODELFORGE_NODE_OFFLINE_AFTER_SECONDS=90 + +# How often node liveness is re-evaluated. +MODELFORGE_LIVENESS_POLL_INTERVAL_SECONDS=5 + +# Clock skew tolerated on an agent report before refusal. +MODELFORGE_AGENT_MAX_CLOCK_SKEW_SECONDS=300 + +# Pre-parser request-body limit for enrollment and authenticated Node Agent reports. +MODELFORGE_NODE_AGENT_MAX_PAYLOAD_BYTES=4194304 + +# Run the node liveness monitor in this process. +MODELFORGE_NODE_LIVENESS_MONITOR_ENABLED=false + +# Maximum inputs accepted in a single capability invocation. +MODELFORGE_GATEWAY_MAX_BATCH_SIZE=8 + +# Maximum characters per input item. +MODELFORGE_GATEWAY_MAX_INPUT_CHARACTERS=8192 + +# Maximum accepted request body size. +MODELFORGE_GATEWAY_MAX_PAYLOAD_BYTES=65536 + +# Total time a capability invocation may take. Must exceed the queue timeout. +MODELFORGE_GATEWAY_REQUEST_TIMEOUT_SECONDS=45 + +# How long a request may wait for capacity before rejection. +MODELFORGE_GATEWAY_QUEUE_TIMEOUT_SECONDS=30 + +# Lease held by a serving job before it is reclaimed. +MODELFORGE_SERVING_JOB_LEASE_SECONDS=120 + +# How long a request payload survives in Redis. +MODELFORGE_SERVING_PAYLOAD_TTL_SECONDS=120 + +# VRAM never offered to a placement, as an absolute floor. +MODELFORGE_SCHEDULER_SAFETY_RESERVE_BYTES=1073741824 + +# VRAM never offered to a placement, as a fraction. Half a device leaves nothing schedulable. +MODELFORGE_SCHEDULER_SAFETY_RESERVE_PERCENTAGE=0.05 + +# Headroom reserved for runtime overhead per node. +MODELFORGE_SCHEDULER_RUNTIME_MARGIN_BYTES=268435456 + +# Absolute headroom added to each deployment estimate. +MODELFORGE_SCHEDULER_DEPLOYMENT_MARGIN_BYTES=134217728 + +# Proportional headroom added to each estimate. +MODELFORGE_SCHEDULER_DEPLOYMENT_MARGIN_PERCENTAGE=0.1 + +# Queued requests accepted before capacity rejection begins. +MODELFORGE_SCHEDULER_GLOBAL_QUEUE_LIMIT=128 + +# Telemetry age past which admission is blocked rather than extrapolated. +MODELFORGE_SCHEDULER_TELEMETRY_STALE_SECONDS=90 + +# How long pressure must hold before the state changes. +MODELFORGE_SCHEDULER_PRESSURE_STABLE_SECONDS=30 + +# Minimum interval between evictions on a node. +MODELFORGE_SCHEDULER_EVICTION_COOLDOWN_SECONDS=60 + +# Placement decisions retained for inspection. +MODELFORGE_SCHEDULER_PLACEMENT_HISTORY_LIMIT=500 + +# Seed the candidate and project registries from manifests. +MODELFORGE_REGISTRY_SEED_ON_STARTUP=false + +# Reconcile abandoned serving work in this process. +MODELFORGE_SERVING_RECONCILIATION_ENABLED=false + +# Interval between serving reconciliation passes. +MODELFORGE_SERVING_RECONCILIATION_INTERVAL_SECONDS=5 + +# Roll back incomplete lifecycle operations at startup. +MODELFORGE_LIFECYCLE_RECONCILIATION_ENABLED=false + +# Report interrupted migration cutovers at startup. They are never auto-resolved: external alias truth cannot be inferred after a crash. +MODELFORGE_MIGRATION_RECONCILIATION_ENABLED=false + +# Run SLO and alert evaluation in this process. +MODELFORGE_OBSERVABILITY_MONITOR_ENABLED=false + +# Interval between observability evaluation passes. +MODELFORGE_OBSERVABILITY_POLL_INTERVAL_SECONDS=60 + +# Reconcile interrupted backups and restores at startup. +MODELFORGE_RECOVERY_RECONCILIATION_ENABLED=false + +# pg_dump executable. Must match the server major version. +MODELFORGE_BACKUP_PG_DUMP_PATH=pg_dump + +# pg_restore executable. +MODELFORGE_BACKUP_PG_RESTORE_PATH=pg_restore + +# psql executable. +MODELFORGE_BACKUP_PSQL_PATH=psql + +# Timeout for a dump or restore command. +MODELFORGE_BACKUP_COMMAND_TIMEOUT_SECONDS=1800 + +# Age past which the newest verified backup raises BACKUP_STALE. +MODELFORGE_BACKUP_STALE_AFTER_SECONDS=93600 + +# Free space below which a backup refuses to start. +MODELFORGE_BACKUP_MINIMUM_FREE_BYTES=1073741824 + +# Required free space as a multiple of the estimated size. +MODELFORGE_BACKUP_CAPACITY_HEADROOM_RATIO=3.0 + +# Whether a restore may overwrite the live database. Keep false outside a rehearsal. +MODELFORGE_RESTORE_ALLOW_PRODUCTION_TARGET=false + +# -------------------------------------------------------------------------- +# Deployment variables. Read by Compose, the Node Agent and the Runtime Worker +# rather than by the control-plane process - an operator still has to set them. +# -------------------------------------------------------------------------- + +# Host address the control-plane database is published on. Defaults to 127.0.0.1; publishing it more widely exposes provenance, credential hashes and the audit trail. +MODELFORGE_POSTGRES_BIND= + +# Host address Redis is published on. Defaults to 127.0.0.1. +MODELFORGE_REDIS_BIND= + +# Host address the API is published on. Defaults to 0.0.0.0 deliberately: the console and compute nodes need it, and every admin route is operator-authenticated. +MODELFORGE_API_BIND= + +# Host address the console is published on. Defaults to 127.0.0.1. +MODELFORGE_WEB_BIND= + +# Host port the database is published on. Defaults to 5432. +MODELFORGE_POSTGRES_PORT= + +# Host port Redis is published on. Defaults to 6379. +MODELFORGE_REDIS_PORT= + +# Host port the API is published on. Defaults to 8000. +MODELFORGE_API_PUBLISHED_PORT= + +# Host port the console is published on. Defaults to 3000. +MODELFORGE_WEB_PORT= + +# Host address for the DR rehearsal database. Loopback only. +MODELFORGE_DR_POSTGRES_BIND= + +# Host address for the DR rehearsal API. Loopback only. +MODELFORGE_DR_API_BIND= + +# API base URL compiled into the console. Vite inlines it at build time, so changing it requires rebuilding the console image, not restarting it. +VITE_API_BASE_URL= + +# Production database name. Required by the production overlay. +MODELFORGE_POSTGRES_DB= + +# Bootstrap/admin role used only by PostgreSQL provisioning; defaults to postgres. +MODELFORGE_POSTGRES_ADMIN_USER= + +# Bootstrap/admin password; never passed to the migration or API container. (REQUIRED in production) +MODELFORGE_POSTGRES_ADMIN_PASSWORD= + +# Raw password supplied to provisioning for the non-superuser modelforge owner role. (REQUIRED in production) +MODELFORGE_MIGRATION_DB_PASSWORD= + +# Raw password supplied to provisioning for the non-owner modelforge_runtime role. (REQUIRED in production) +MODELFORGE_RUNTIME_DB_PASSWORD= + +# Non-owner runtime-role URL passed only to the API container. (REQUIRED in production) +MODELFORGE_RUNTIME_DATABASE_URL= + +# Exact version tag applied to built images and required when the production overlay is not given explicit API and web image references. +MODELFORGE_VERSION= + +# Source commit stamped into images at build time. +MODELFORGE_COMMIT= + +# Build timestamp stamped into images. +MODELFORGE_BUILT_AT= + +# Exact tag or digest the production overlay runs for the API; never use latest. +MODELFORGE_API_IMAGE= + +# Exact tag or digest the production overlay runs for the console; never use latest. +MODELFORGE_WEB_IMAGE= + +# Exact release tag or digest for the standalone Node Agent. The local-build fallback is named local and never resolves to latest. +MODELFORGE_NODE_AGENT_IMAGE= + +# Digest recorded as the running API build identity. +MODELFORGE_API_IMAGE_DIGEST= + +# Volume or bind path backing the backup root. +MODELFORGE_BACKUP_VOLUME= + +# Volume or bind path backing the restore staging root. +MODELFORGE_RESTORE_VOLUME= + +# Volume or bind path holding the agent's persisted identity. +MODELFORGE_AGENT_STATE_VOLUME= + +# Volume or bind path for the agent's Hugging Face cache. +MODELFORGE_AGENT_HF_CACHE_VOLUME= + +# Volume or bind path for verified artifacts on a node. +MODELFORGE_AGENT_ARTIFACT_VOLUME= + +# Volume or bind path for the node's quarantine area. +MODELFORGE_AGENT_QUARANTINE_VOLUME= + +# URL the agent reports to. Outbound only; the control plane never dials a node. +MODELFORGE_AGENT_CONTROL_PLANE_URL= + +# Single-use enrolment token. Consumed atomically: a storm against one token produces exactly one identity. +MODELFORGE_AGENT_ENROLLMENT_TOKEN= + +# Hostname the agent enrols under. +MODELFORGE_AGENT_HOSTNAME= + +# Accelerator contract: nvidia fails closed unless NVML inventory and telemetry are valid; cpu permits a legitimate CPU-only node; auto requires NVIDIA when injected devices are observed. Canonical GPU Compose deployments set nvidia explicitly. +MODELFORGE_AGENT_ACCELERATOR_MODE= + +# Whether the agent verifies the control plane's certificate. True wherever TLS is real. +MODELFORGE_AGENT_TLS_VERIFY= + +# Host address mapped for a private-CA deployment. +MODELFORGE_AGENT_CONTROL_PLANE_HOST_ADDRESS= + +# Path to the private CA certificate the agent trusts. +MODELFORGE_AGENT_CA_CERT_PATH= + +# Artifact root as the runtime worker sees it. +MODELFORGE_RUNTIME_WORKER_ARTIFACT_ROOT= + +# Worker poll interval, in seconds. +MODELFORGE_RUNTIME_WORKER_POLL_INTERVAL_SECONDS= + +# Source commit reported by the deployment. +MODELFORGE_SOURCE_COMMIT= + +# Git reference reported by the deployment. +MODELFORGE_SOURCE_REFERENCE= + +# Repository URL reported by the deployment. +MODELFORGE_SOURCE_REPOSITORY= diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..0f2dfef --- /dev/null +++ b/.gitattributes @@ -0,0 +1,12 @@ +* text=auto eol=lf + +*.bat text eol=crlf +*.cmd text eol=crlf +*.ps1 text eol=crlf +*.png binary +*.jpg binary +*.jpeg binary +*.gif binary +*.ico binary +*.woff2 binary + diff --git a/.gitea/tests/test_production_deploy_policy.py b/.gitea/tests/test_production_deploy_policy.py new file mode 100644 index 0000000..faac2a7 --- /dev/null +++ b/.gitea/tests/test_production_deploy_policy.py @@ -0,0 +1,215 @@ +"""Fail-closed checks for the production delivery workflow. + +This test intentionally uses only the Python standard library so it can run before +project dependencies are installed. The small helpers parse indentation-delimited +YAML blocks that are relevant to the policy; they are not a general YAML parser. +""" + +from __future__ import annotations + +import re +import subprocess +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[2] +DEPLOY_WORKFLOW = REPOSITORY_ROOT / ".gitea" / "workflows" / "unraid-deploy.yml" +VALIDATION_WORKFLOW = REPOSITORY_ROOT / ".gitea" / "workflows" / "managed-validation.yml" + + +def _indented_block(text: str, header: str) -> str: + """Return a YAML block beginning at an exact header line.""" + + lines = text.splitlines() + try: + start = lines.index(header) + except ValueError as exc: + raise AssertionError(f"Missing YAML header: {header!r}") from exc + indent = len(header) - len(header.lstrip()) + end = len(lines) + for index in range(start + 1, len(lines)): + line = lines[index] + if not line.strip() or line.lstrip().startswith("#"): + continue + current_indent = len(line) - len(line.lstrip()) + if current_indent <= indent: + end = index + break + return "\n".join(lines[start:end]) + + +def _named_step(text: str, name: str) -> str: + return _indented_block(text, f" - name: {name}") + + +def _run_script(step: str) -> str: + """Extract one inline or literal run body without normalizing shell content.""" + + lines = step.splitlines() + if " run: |" not in lines: + inline = next((line for line in lines if line.startswith(" run: ")), None) + if inline is None: + raise AssertionError("Named step does not contain a run command") + return inline.removeprefix(" run: ") + "\n" + start = lines.index(" run: |") + 1 + script_lines = lines[start:] + if any(line and not line.startswith(" ") for line in script_lines): + raise AssertionError("Unexpected indentation in literal run block") + return "\n".join(line[10:] if line else "" for line in script_lines) + "\n" + + +class ProductionDeployPolicyTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.deploy = DEPLOY_WORKFLOW.read_text(encoding="utf-8") + cls.validation = VALIDATION_WORKFLOW.read_text(encoding="utf-8") + + def test_deployment_has_no_automatic_trigger(self) -> None: + trigger = _indented_block(self.deploy, "on:") + trigger_keys = re.findall(r"^ ([a-z_]+):", trigger, flags=re.MULTILINE) + self.assertEqual(trigger_keys, ["workflow_dispatch"]) + self.assertNotRegex(trigger, r"(?m)^\s+push:") + + def test_dispatch_inputs_are_required_and_fail_safe(self) -> None: + inputs = _indented_block(self.deploy, " inputs:") + input_names = re.findall(r"^ ([a-z_]+):", inputs, flags=re.MULTILINE) + self.assertEqual(input_names, ["release_tag", "release_commit", "action"]) + + for input_name in ("release_tag", "release_commit"): + block = _indented_block(self.deploy, f" {input_name}:") + self.assertRegex(block, r"(?m)^ required: true$") + self.assertRegex(block, r"(?m)^ type: string$") + + action = _indented_block(self.deploy, " action:") + self.assertRegex(action, r"(?m)^ required: true$") + self.assertRegex(action, r"(?m)^ default: VERIFY_ONLY$") + self.assertRegex(action, r"(?m)^ type: choice$") + options = re.findall(r"^ - (\S+)$", action, flags=re.MULTILINE) + self.assertEqual(options, ["VERIFY_ONLY", "DEPLOY_STABLE_TO_PRODUCTION"]) + + def test_inputs_are_validated_before_checkout(self) -> None: + validation = _named_step(self.deploy, "Validate immutable dispatch inputs") + checkout = _named_step(self.deploy, "Check out the exact release commit with full history") + self.assertLess(self.deploy.index(validation), self.deploy.index(checkout)) + self.assertIn('"refs/heads/master"', validation) + self.assertIn( + r"^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$", + validation, + ) + self.assertIn(r"^[0-9a-f]{40}$", validation) + self.assertIn("VERIFY_ONLY|DEPLOY_STABLE_TO_PRODUCTION", validation) + + tag_pattern = re.search(r'RELEASE_TAG\}" =~ (\^.*\$) \]\]', validation) + commit_pattern = re.search(r'RELEASE_COMMIT\}" =~ (\^.*\$) \]\]', validation) + self.assertIsNotNone(tag_pattern) + self.assertIsNotNone(commit_pattern) + assert tag_pattern is not None and commit_pattern is not None + for valid_tag in ("v0.0.0", "v1.2.1", "v10.20.30"): + self.assertRegex(valid_tag, tag_pattern.group(1)) + for invalid_tag in ("1.2.1", "v01.2.1", "v1.2.1-rc.1", "v1.2.1+build", "master"): + self.assertNotRegex(invalid_tag, tag_pattern.group(1)) + self.assertRegex("0" * 40, commit_pattern.group(1)) + for invalid_commit in ("0" * 39, "A" * 40, "g" * 40, "master"): + self.assertNotRegex(invalid_commit, commit_pattern.group(1)) + + def test_checkout_is_pinned_to_the_validated_commit_with_full_history(self) -> None: + checkout = _named_step(self.deploy, "Check out the exact release commit with full history") + self.assertRegex(checkout, r"actions/checkout@[0-9a-f]{40}") + self.assertIn("ref: ${{ inputs.release_commit }}", checkout) + self.assertIn("fetch-depth: 0", checkout) + self.assertIn("persist-credentials: true", checkout) + + def test_release_provenance_is_verified_and_exported(self) -> None: + provenance = _named_step(self.deploy, "Verify stable release provenance") + required_fragments = ( + "git fetch --force --no-recurse-submodules origin", + 'git cat-file -t "${tag_ref}"', + 'git rev-parse "${tag_ref}^{tag}"', + 'git rev-parse "${tag_ref}^{commit}"', + "git rev-parse HEAD", + "< VERSION", + "git merge-base --is-ancestor", + "refs/remotes/origin/master", + "DEPLOY_COMMIT=%s", + '>> "${GITHUB_ENV}"', + ) + for fragment in required_fragments: + with self.subTest(fragment=fragment): + self.assertIn(fragment, provenance) + self.assertNotIn("GITHUB_SHA", self.deploy) + + def test_all_production_shell_blocks_parse(self) -> None: + names = ( + "Validate immutable dispatch inputs", + "Verify stable release provenance", + "Verification-only result", + "Deploy verified stable release to production", + ) + for name in names: + script = _run_script(_named_step(self.deploy, name)) + result = subprocess.run( + ["bash", "-n"], + input=script.encode("utf-8"), + capture_output=True, + check=False, + ) + with self.subTest(step=name): + self.assertEqual(result.returncode, 0, result.stderr.decode("utf-8", "replace")) + + def test_deployment_requires_the_explicit_action_and_verified_commit(self) -> None: + deploy = _named_step(self.deploy, "Deploy verified stable release to production") + self.assertIn("inputs.action == 'DEPLOY_STABLE_TO_PRODUCTION'", deploy) + self.assertIn('test -n "${DEPLOY_COMMIT:-}"', deploy) + self.assertIn('"${GITHUB_REPOSITORY}" "${DEPLOY_COMMIT}"', deploy) + self.assertEqual(self.deploy.count("/opt/gitea-deploy/deploy.py deploy"), 1) + + verify_only = _named_step(self.deploy, "Verification-only result") + self.assertIn("inputs.action == 'VERIFY_ONLY'", verify_only) + + def test_managed_validation_runs_policy_before_profile_work(self) -> None: + policy_step = _named_step(self.validation, "Production delivery policy") + self.assertIn("python3 -m unittest discover -s .gitea/tests", policy_step) + self.assertNotRegex(policy_step, r"(?m)^ if:") + self.assertLess( + self.validation.index(policy_step), + self.validation.index(" - name: Validate the requested profile"), + ) + + def test_security_profile_runs_fail_closed_scanners(self) -> None: + step = _named_step(self.validation, "Security — secrets and vulnerable dependencies") + self.assertIn("env.PROFILE == 'security'", step) + self.assertIn("env.PROFILE == 'full'", step) + script = _run_script(step) + self.assertIn("gitleaks git .", script) + self.assertIn("pip_audit --strict --skip-editable", script) + self.assertIn("npm audit --audit-level=high --omit=dev", script) + self.assertNotRegex(script, r"(?m)(?:\|\|\s*true|continue-on-error)") + + toolchain = _named_step(self.validation, "Toolchain") + toolchain_script = _run_script(toolchain) + self.assertIn("command -v gitleaks", toolchain_script) + self.assertIn("exit 1", toolchain_script) + + def test_compose_validation_renders_the_production_projection(self) -> None: + step = _named_step(self.validation, "Compose projections") + script = _run_script(step) + self.assertIn( + "for overlay in backup dr gpu node-agent node-recovery production runtime-worker", + script, + ) + for variable in ( + "MODELFORGE_POSTGRES_ADMIN_PASSWORD", + "MODELFORGE_MIGRATION_DATABASE_URL", + "MODELFORGE_RUNTIME_DATABASE_URL", + "MODELFORGE_OPERATOR_API_KEY", + "MODELFORGE_BACKUP_ENCRYPTION_KEY", + "MODELFORGE_CORS_ORIGINS", + "VITE_API_BASE_URL", + ): + with self.subTest(variable=variable): + self.assertIn(f"export {variable}=", script) + + +if __name__ == "__main__": + unittest.main() diff --git a/.gitea/tests/test_public_source_export.py b/.gitea/tests/test_public_source_export.py new file mode 100644 index 0000000..39b3e0f --- /dev/null +++ b/.gitea/tests/test_public_source_export.py @@ -0,0 +1,199 @@ +"""Fail-closed integration tests for the parentless public source boundary.""" + +from __future__ import annotations + +import subprocess +import tempfile +import unittest +from pathlib import Path + + +REPOSITORY_ROOT = Path(__file__).resolve().parents[2] +EXPORTER = REPOSITORY_ROOT / "scripts" / "export-public-source.mjs" +VALIDATOR = REPOSITORY_ROOT / "scripts" / "validate-public-source.mjs" + + +class PublicSourceExportTests(unittest.TestCase): + def _repository( + self, + root: Path, + *, + license_present: bool = True, + managed_validation_workflow: str | None = None, + ) -> Path: + repository = root / "private-source" + repository.mkdir() + files: dict[str, str | bytes] = { + "README.md": "# Product\n", + "SECURITY.md": "# Security\n", + "CONTRIBUTING.md": "# Contributing\n", + "docker-compose.yml": "services: {}\n", + "VERSION": "1.2.1\n", + "backend/pyproject.toml": "[project]\nname = 'fixture-api'\n", + "frontend/package.json": '{"name":"fixture-web"}\n', + "node-agent/pyproject.toml": "[project]\nname = 'fixture-agent'\n", + "runtime-worker/pyproject.toml": "[project]\nname = 'fixture-worker'\n", + } + if license_present: + files["LICENSE"] = (REPOSITORY_ROOT / "LICENSE").read_bytes() + if managed_validation_workflow is not None: + files[".gitea/workflows/managed-validation.yml"] = managed_validation_workflow + for relative_path, contents in files.items(): + destination = repository / relative_path + destination.parent.mkdir(parents=True, exist_ok=True) + if isinstance(contents, bytes): + destination.write_bytes(contents) + else: + destination.write_text(contents, encoding="utf-8") + (repository / "public-source.allowlist").write_text( + "\n".join(sorted(files)) + "\n", encoding="utf-8" + ) + subprocess.run(["git", "init", "-q"], cwd=repository, check=True) + subprocess.run(["git", "add", "."], cwd=repository, check=True) + subprocess.run( + [ + "git", + "-c", + "user.name=ModelForge test", + "-c", + "user.email=test@example.invalid", + "commit", + "-qm", + "fixture", + ], + cwd=repository, + check=True, + ) + return repository + + def _export(self, repository: Path, output: Path) -> subprocess.CompletedProcess[str]: + return subprocess.run( + ["node", str(EXPORTER), "--repository", str(repository), "--output", str(output)], + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + check=False, + ) + + def _validate(self, output: Path) -> subprocess.CompletedProcess[str]: + return subprocess.run( + ["node", str(VALIDATOR)], + cwd=output, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + check=False, + ) + + def test_export_requires_an_explicit_tracked_license(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + repository = self._repository(root, license_present=False) + result = self._export(repository, root / "public") + + self.assertNotEqual(result.returncode, 0) + self.assertIn("canonical LICENSE is missing", result.stderr) + + def test_manifest_detects_content_tampering_and_unexpected_files(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + output = root / "public" + result = self._export(self._repository(root), output) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + validation = self._validate(output) + self.assertEqual(validation.returncode, 0, validation.stdout + validation.stderr) + + original_readme = (output / "README.md").read_bytes() + (output / "README.md").write_text("tampered\n", encoding="utf-8") + tampered = self._validate(output) + self.assertNotEqual(tampered.returncode, 0) + self.assertIn("Manifest", tampered.stderr) + + (output / "README.md").write_bytes(original_readme) + (output / "not-reviewed.txt").write_text("extra\n", encoding="utf-8") + unexpected = self._validate(output) + self.assertNotEqual(unexpected.returncode, 0) + self.assertIn("Unexpected public source files", unexpected.stderr) + + def test_dirty_source_tree_cannot_claim_the_head_revision(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + repository = self._repository(root) + (repository / "README.md").write_text("changed after commit\n", encoding="utf-8") + result = self._export(repository, root / "public") + + self.assertNotEqual(result.returncode, 0) + self.assertIn("commit the exact source tree", result.stderr) + + def test_sanitized_destination_collisions_fail_before_writing(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + repository = self._repository(root) + private_name = "Tow" + "er.txt" + public_name = "GPU " + "Node.txt" + (repository / private_name).write_text("one\n", encoding="utf-8") + (repository / public_name).write_text("two\n", encoding="utf-8") + with (repository / "public-source.allowlist").open("a", encoding="utf-8") as allowlist: + allowlist.write(f"{private_name}\n{public_name}\n") + subprocess.run(["git", "add", "."], cwd=repository, check=True) + subprocess.run( + [ + "git", + "-c", + "user.name=ModelForge test", + "-c", + "user.email=test@example.invalid", + "commit", + "-qm", + "collision fixture", + ], + cwd=repository, + check=True, + ) + result = self._export(repository, root / "public") + + self.assertNotEqual(result.returncode, 0) + self.assertIn("finding", result.stderr) + + def test_public_managed_validation_requires_explicit_owner_dispatch(self) -> None: + workflow = "name: Managed validation\n\non:\n pull_request:\n workflow_dispatch:\n" + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + repository = self._repository( + root, + managed_validation_workflow=workflow, + ) + output = root / "public" + result = self._export(repository, output) + + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + canonical = ( + repository / ".gitea/workflows/managed-validation.yml" + ).read_text(encoding="utf-8") + exported = ( + output / ".gitea/workflows/managed-validation.yml" + ).read_text(encoding="utf-8") + + self.assertIn(" pull_request:\n", canonical) + self.assertNotIn(" pull_request:\n", exported) + self.assertIn(" workflow_dispatch:\n", exported) + self.assertIn("fork PRs never reach private runners", exported) + + def test_changed_managed_validation_trigger_fails_closed(self) -> None: + workflow = "name: Managed validation\n\non:\n workflow_dispatch:\n" + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + repository = self._repository( + root, + managed_validation_workflow=workflow, + ) + result = self._export(repository, root / "public") + + self.assertNotEqual(result.returncode, 0) + self.assertIn("has no expected pull_request trigger", result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/.gitea/tests/test_rc_acceptance_policy.py b/.gitea/tests/test_rc_acceptance_policy.py new file mode 100644 index 0000000..650388a --- /dev/null +++ b/.gitea/tests/test_rc_acceptance_policy.py @@ -0,0 +1,280 @@ +"""Structural safety checks for public-candidate server acceptance.""" + +from __future__ import annotations + +import subprocess +import importlib.util +import json +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +WORKFLOW = ROOT / ".gitea" / "workflows" / "public-candidate-acceptance.yml" +MANAGED_WORKFLOW = ROOT / ".gitea" / "workflows" / "managed-validation.yml" +ACCEPTANCE = ROOT / "scripts" / "rc_server_acceptance.py" +TRIVY_VALIDATOR = ROOT / "scripts" / "validate_trivy_report.py" +RUNTIME_DOCKERFILE = ROOT / "runtime-worker" / "Dockerfile" +RUNTIME_PYPROJECT = ROOT / "runtime-worker" / "pyproject.toml" + + +class RcAcceptancePolicyTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.workflow = WORKFLOW.read_text(encoding="utf-8") + cls.managed = MANAGED_WORKFLOW.read_text(encoding="utf-8") + cls.acceptance = ACCEPTANCE.read_text(encoding="utf-8") + cls.trivy = TRIVY_VALIDATOR.read_text(encoding="utf-8") + cls.runtime_dockerfile = RUNTIME_DOCKERFILE.read_text(encoding="utf-8") + cls.runtime_pyproject = RUNTIME_PYPROJECT.read_text(encoding="utf-8") + + def test_workflow_is_manual_and_never_calls_production_deployment(self) -> None: + self.assertIn("workflow_dispatch:", self.workflow) + self.assertNotIn("\n push:", self.workflow) + self.assertNotIn("pull_request:", self.workflow) + self.assertNotIn("deploy.py", self.workflow) + self.assertNotIn("DEPLOY_STABLE_TO_PRODUCTION", self.workflow) + + def test_exact_source_and_docker_are_required(self) -> None: + self.assertIn(r"^[0-9a-f]{40}$", self.workflow) + self.assertIn("ref: ${{ inputs.source_commit }}", self.workflow) + self.assertIn("docker version", self.workflow) + self.assertNotIn("SKIPPED: no docker", self.workflow) + + def test_security_tools_are_downloaded_with_exact_checksums(self) -> None: + self.assertIn("GITLEAKS_VERSION: 8.30.1", self.workflow) + self.assertIn( + "GITLEAKS_SHA256: 551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb", + self.workflow, + ) + self.assertIn('printf \'GITLEAKS=%s\\n\'', self.workflow) + self.assertIn('\"${GITLEAKS}\" dir \"${public_source}\"', self.workflow) + self.assertIn("sha256sum --check --strict", self.workflow) + self.assertNotIn("command -v gitleaks", self.workflow) + + def test_unmerged_branch_can_use_the_existing_managed_dispatch_entry(self) -> None: + self.assertIn("workflow_call:", self.workflow) + self.assertIn("uses: ./.gitea/workflows/public-candidate-acceptance.yml", self.managed) + self.assertIn("inputs.profile == 'build'", self.managed) + self.assertIn("source_commit: ${{ gitea.sha }}", self.managed) + + def test_all_four_images_are_built_and_scanned(self) -> None: + for image in ( + "modelforge-api", + "modelforge-web", + "modelforge-node-agent", + "modelforge-runtime-worker", + ): + self.assertIn(image, self.acceptance) + self.assertIn('"Metadata"', self.trivy) + self.assertIn('metadata.get("ImageID") != expected_image_id', self.trivy) + self.assertIn("reject_duplicate_keys", self.trivy) + self.assertIn("fixable_high_critical + unreviewed_unfixed_high_critical", self.trivy) + self.assertIn("public-candidate-unfixed-vulnerabilities.json", self.acceptance) + + def test_runtime_image_applies_security_updates_and_exact_inference_pins(self) -> None: + self.assertIn("apt-get upgrade --yes", self.runtime_dockerfile) + self.assertIn("apt-get purge --yes linux-libc-dev", self.runtime_dockerfile) + self.assertNotIn("apt-get autoremove", self.runtime_dockerfile) + self.assertIn("pip check", self.runtime_dockerfile) + for dependency in ( + "Pillow==12.3.0", + "protobuf==5.29.6", + "sentencepiece==0.2.2", + "sentence-transformers==6.0.1", + "transformers==5.16.1", + "urllib3==2.7.0", + ): + self.assertIn(dependency, self.runtime_pyproject) + + def test_compose_project_is_isolated_and_always_removed(self) -> None: + self.assertIn('project = f"modelforge-rc-', self.acceptance) + self.assertIn('"MODELFORGE_API_BIND": "127.0.0.1"', self.acceptance) + self.assertIn('"MODELFORGE_API_PUBLISHED_PORT": "0"', self.acceptance) + self.assertIn('"production_changed": False', self.acceptance) + self.assertIn('"compute_identity_created": False', self.acceptance) + self.assertIn('"down", "--volumes", "--remove-orphans"', self.acceptance) + self.assertIn("atexit.register(cleanup_candidate_images", self.acceptance) + self.assertIn('"docker", "image", "rm", "--force", tag', self.acceptance) + self.assertNotIn("docker-compose.runtime-worker.yml", self.acceptance) + self.assertNotIn("docker-compose.node-agent.yml", self.acceptance) + + def test_clean_install_bootstraps_roles_across_a_remote_docker_daemon(self) -> None: + self.assertIn("def provision_database_roles(", self.acceptance) + self.assertIn('run("docker", "cp", str(bootstrap)', self.acceptance) + self.assertIn('"psql",', self.acceptance) + self.assertIn('"--file",', self.acceptance) + self.assertIn( + '"--wait", "postgres", "redis", env=env', self.acceptance + ) + self.assertLess( + self.acceptance.index("provision_database_roles(compose, env, database)"), + self.acceptance.index( + '"--wait", "api", "web", env=env' + ), + ) + + def test_clean_install_seeds_config_across_a_remote_docker_daemon(self) -> None: + self.assertIn("def provision_config_volume(project: str)", self.acceptance) + self.assertIn('f"{project}_acceptance-config"', self.acceptance) + self.assertIn('f"com.docker.compose.project={project}"', self.acceptance) + self.assertIn('run("docker", "cp", f"{ROOT / \'config\'}/."', self.acceptance) + self.assertIn( + 'run("docker", "container", "rm", "--force", seed, check=False)', + self.acceptance, + ) + self.assertIn('" read_only: true\\n"', self.acceptance) + self.assertLess( + self.acceptance.index("provision_config_volume(project)"), + self.acceptance.index('"--wait", "api", "web", env=env'), + ) + + def test_http_contract_runs_inside_the_isolated_compose_network(self) -> None: + self.assertIn("def container_http_status(", self.acceptance) + self.assertIn("def wait_for_container_status(", self.acceptance) + self.assertIn('headers["X-ModelForge-Admin-Token"]', self.acceptance) + self.assertIn('os.environ["MODELFORGE_OPERATOR_API_KEY"]', self.acceptance) + self.assertIn('wait_for_container_status(api_container, "http://web:3000/", 200)', self.acceptance) + self.assertNotIn("urllib.request.urlopen(request, timeout=timeout)", self.acceptance) + + def test_workflow_has_an_always_run_exact_project_cleanup_fallback(self) -> None: + self.assertIn("- name: Always remove acceptance Docker resources", self.workflow) + cleanup = self.workflow.split( + " - name: Always remove acceptance Docker resources", 1 + )[1].split(" - name: Upload acceptance evidence", 1)[0] + self.assertIn("if: always()", cleanup) + self.assertIn('label=com.docker.compose.project=${project}', cleanup) + self.assertIn("docker container rm --force", cleanup) + self.assertIn("docker volume rm --force", cleanup) + self.assertIn("docker network rm", cleanup) + self.assertIn("docker image rm --force", cleanup) + self.assertNotIn("docker system prune", cleanup) + + def test_python_entrypoints_compile(self) -> None: + for path in (ACCEPTANCE, TRIVY_VALIDATOR): + result = subprocess.run( + ["python", "-m", "py_compile", str(path)], + capture_output=True, + check=False, + ) + with self.subTest(path=path.name): + self.assertEqual(result.returncode, 0, result.stderr.decode("utf-8", "replace")) + + def test_trivy_validator_binds_image_and_rejects_blockers(self) -> None: + spec = importlib.util.spec_from_file_location("trivy_report", TRIVY_VALIDATOR) + assert spec and spec.loader + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + image_id = "sha256:" + "a" * 64 + report = { + "ArtifactName": image_id, + "ArtifactType": "container_image", + "Metadata": {"ImageID": image_id}, + "Results": [ + { + "Target": "debian", + "Class": "os-pkgs", + "Type": "debian", + "Vulnerabilities": [ + { + "VulnerabilityID": "CVE-2099-9999", + "PkgName": "blocked-package", + "InstalledVersion": "1.0.0", + "FixedVersion": "1.0.1", + "Severity": "HIGH", + } + ], + } + ], + } + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "report.json" + path.write_text(json.dumps(report), encoding="utf-8") + result = module.validate(path, image_id) + self.assertEqual(result["release_blockers"], 1) + with self.assertRaisesRegex(ValueError, "does not match"): + module.validate(path, "sha256:" + "b" * 64) + + def test_trivy_validator_rejects_duplicate_keys_and_empty_coverage(self) -> None: + spec = importlib.util.spec_from_file_location("trivy_report_negative", TRIVY_VALIDATOR) + assert spec and spec.loader + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + image_id = "sha256:" + "a" * 64 + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "report.json" + path.write_text('{"ArtifactType":"container_image","ArtifactType":"container_image"}', encoding="utf-8") + with self.assertRaisesRegex(ValueError, "duplicate JSON key"): + module.validate(path, image_id) + path.write_text( + json.dumps( + { + "ArtifactType": "container_image", + "Metadata": {"ImageID": image_id}, + "Results": [], + } + ), + encoding="utf-8", + ) + with self.assertRaisesRegex(ValueError, "no package result coverage"): + module.validate(path, image_id) + + def test_trivy_validator_allows_only_exact_reviewed_upstream_unfixed_findings(self) -> None: + spec = importlib.util.spec_from_file_location("trivy_report_reviewed", TRIVY_VALIDATOR) + assert spec and spec.loader + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + image_id = "sha256:" + "a" * 64 + finding = { + "VulnerabilityID": "CVE-2099-0001", + "PkgName": "example-package", + "InstalledVersion": "1.0.0", + "FixedVersion": "", + "Severity": "HIGH", + } + report = { + "ArtifactType": "container_image", + "Metadata": {"ImageID": image_id}, + "Results": [ + { + "Target": "debian", + "Class": "os-pkgs", + "Vulnerabilities": [finding], + } + ], + } + baseline = { + "schema_version": 1, + "vulnerabilities": [ + { + "vulnerability_id": "CVE-2099-0001", + "package": "example-package", + "installed_version": "1.0.0", + "severity": "HIGH", + } + ], + } + with tempfile.TemporaryDirectory() as directory: + report_path = Path(directory) / "report.json" + baseline_path = Path(directory) / "baseline.json" + report_path.write_text(json.dumps(report), encoding="utf-8") + baseline_path.write_text(json.dumps(baseline), encoding="utf-8") + summary = module.validate(report_path, image_id, baseline_path) + self.assertEqual(summary["reviewed_unfixed_high_critical"], 1) + self.assertEqual(summary["release_blockers"], 0) + + finding["InstalledVersion"] = "1.0.1" + report_path.write_text(json.dumps(report), encoding="utf-8") + self.assertEqual(module.validate(report_path, image_id, baseline_path)["release_blockers"], 1) + + finding["InstalledVersion"] = "1.0.0" + finding["FixedVersion"] = "1.0.2" + report_path.write_text(json.dumps(report), encoding="utf-8") + summary = module.validate(report_path, image_id, baseline_path) + self.assertEqual(summary["fixable_high_critical"], 1) + self.assertEqual(summary["release_blockers"], 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/.gitea/workflows/managed-validation.yml b/.gitea/workflows/managed-validation.yml new file mode 100644 index 0000000..b9f8b7e --- /dev/null +++ b/.gitea/workflows/managed-validation.yml @@ -0,0 +1,292 @@ +name: Managed validation + +# The gate on protected master. Before v1 this keyed off manifests at the repository root, and this +# repository has none — pyproject.toml, package.json and the lock files all live in backend/, +# frontend/, node-agent/ and runtime-worker/. The "full" profile therefore completed in nine seconds +# having run a whitespace check, a merge-marker scan and py_compile, and no test suite at all, while +# reporting success to a branch protection rule that required it. +# +# It now targets the component roots explicitly, and refuses to report success when a component that +# should have run tests ran none. A gate that passes because it found nothing to do is worse than no +# gate: it produces the paperwork of validation without the fact of it. + +on: + # Public exports require explicit owner dispatch; fork PRs never reach private runners. + workflow_dispatch: + inputs: + profile: + description: Allowlisted validation profile + required: true + default: full + type: choice + options: [test, lint, typecheck, build, security, full] + +permissions: + contents: read + +concurrency: + group: managed-validation-${{ gitea.repository }}-${{ gitea.ref }} + cancel-in-progress: true + +jobs: + full: + name: full + runs-on: ubuntu-latest + timeout-minutes: 45 + env: + PROFILE: ${{ inputs.profile || 'full' }} + # An explicit interpreter path rather than PATH manipulation: how a runner propagates PATH + # between steps varies, and a validation gate should not depend on that detail. + VENV: /tmp/modelforge-validation-venv + steps: + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + + - name: Production delivery policy + shell: bash + run: python3 -m unittest discover -s .gitea/tests -p 'test_*.py' -v + + - name: Validate the requested profile + shell: bash + run: | + set -euo pipefail + case "${PROFILE}" in + test|lint|typecheck|build|security|full) ;; + *) echo "Profile is not allowlisted: ${PROFILE}" >&2; exit 2 ;; + esac + echo "profile=${PROFILE}" + echo "commit=$(git rev-parse HEAD)" + + - name: Repository hygiene + shell: bash + run: | + set -euo pipefail + git diff --check + if git grep -nE '^(<<<<<<< |=======$|>>>>>>> )' -- . ':!*.lock' ':!*.patch'; then + echo "Unresolved merge markers detected" >&2 + exit 1 + fi + echo "hygiene: clean" + + - name: Prepare the report directory + shell: bash + run: mkdir -p reports + + # Deliberately not using setup-python/setup-node: this runs on a self-hosted runner whose + # image already carries both, and the previous workflow depended on that too. What changes is + # that a missing toolchain now stops the run instead of quietly reducing what gets validated. + - name: Toolchain + shell: bash + run: | + set -euo pipefail + command -v python3 >/dev/null || { echo "python3 is not on PATH" >&2; exit 1; } + command -v node >/dev/null || { echo "node is not on PATH" >&2; exit 1; } + command -v npm >/dev/null || { echo "npm is not on PATH" >&2; exit 1; } + echo " python $(python3 --version)" + echo " node $(node --version)" + echo " npm $(npm --version)" + python3 -m venv "${VENV}" + "${VENV}/bin/python" -m pip install --disable-pip-version-check --quiet --upgrade pip + if [[ "${PROFILE}" == security || "${PROFILE}" == full ]]; then + command -v gitleaks >/dev/null || { + echo "gitleaks is required for the security profile" >&2 + exit 1 + } + "${VENV}/bin/python" -m pip install \ + --disable-pip-version-check --quiet 'pip-audit==2.10.0' + fi + echo " venv $("${VENV}/bin/python" --version) at ${VENV}" + + - name: Backend — install, lint, typecheck, test + id: backend + shell: bash + working-directory: backend + run: | + set -euo pipefail + "${VENV}/bin/python" -m pip install --disable-pip-version-check --quiet -e '.[dev]' + if [[ "${PROFILE}" == lint || "${PROFILE}" == full ]]; then + "${VENV}/bin/python" -m ruff check src tests + fi + if [[ "${PROFILE}" == typecheck || "${PROFILE}" == full ]]; then + "${VENV}/bin/python" -m mypy src + fi + if [[ "${PROFILE}" == test || "${PROFILE}" == full ]]; then + "${VENV}/bin/python" -m pytest -q --no-header --junitxml=../reports/backend.xml + fi + + - name: Node Agent — install, lint, typecheck, test + id: node_agent + shell: bash + working-directory: node-agent + run: | + set -euo pipefail + "${VENV}/bin/python" -m pip install --disable-pip-version-check --quiet -e '.[dev]' + if [[ "${PROFILE}" == lint || "${PROFILE}" == full ]]; then + "${VENV}/bin/python" -m ruff check src + fi + if [[ "${PROFILE}" == typecheck || "${PROFILE}" == full ]]; then + "${VENV}/bin/python" -m mypy src + fi + if [[ "${PROFILE}" == test || "${PROFILE}" == full ]]; then + "${VENV}/bin/python" -m pytest -q --no-header --junitxml=../reports/node-agent.xml + fi + + - name: Runtime Worker — install, lint, typecheck, test + id: runtime_worker + shell: bash + working-directory: runtime-worker + run: | + set -euo pipefail + "${VENV}/bin/python" -m pip install --disable-pip-version-check --quiet -e '.[dev]' + if [[ "${PROFILE}" == lint || "${PROFILE}" == full ]]; then + "${VENV}/bin/python" -m ruff check src + fi + if [[ "${PROFILE}" == typecheck || "${PROFILE}" == full ]]; then + "${VENV}/bin/python" -m mypy src + fi + if [[ "${PROFILE}" == test || "${PROFILE}" == full ]]; then + "${VENV}/bin/python" -m pytest -q --no-header --junitxml=../reports/runtime-worker.xml + fi + + - name: Console — install frozen, typecheck, test, build + id: console + shell: bash + working-directory: frontend + run: | + set -euo pipefail + npm ci + if [[ "${PROFILE}" == typecheck || "${PROFILE}" == full ]]; then npx tsc --noEmit; fi + if [[ "${PROFILE}" == test || "${PROFILE}" == full ]]; then + npx vitest run --reporter=junit --outputFile=../reports/frontend.xml + fi + if [[ "${PROFILE}" == build || "${PROFILE}" == full ]]; then npm run build; fi + + - name: Security — secrets and vulnerable dependencies + if: ${{ env.PROFILE == 'full' || env.PROFILE == 'security' }} + shell: bash + run: | + set -euo pipefail + gitleaks git . --no-banner --redact --report-format json \ + --report-path reports/gitleaks.json + "${VENV}/bin/python" -m pip_audit --strict --skip-editable --desc=on \ + --format=json --output=reports/python-audit.json + cd frontend + npm audit --audit-level=high --omit=dev --json > ../reports/npm-audit.json + + - name: Compose projections + if: ${{ env.PROFILE == 'full' || env.PROFILE == 'build' }} + shell: bash + run: | + set -euo pipefail + if ! command -v docker >/dev/null; then + echo "SKIPPED: no docker CLI on this runner; compose projections are validated by the" + echo "local release gate instead. This step never reports a pass it did not earn." + exit 0 + fi + # Compose interpolation needs values, not live credentials. These fixed validation-only + # strings never reach a service and ensure the fail-closed production projection is + # actually parsed on every full/build run. + export MODELFORGE_POSTGRES_DB=modelforge_validation + export MODELFORGE_POSTGRES_ADMIN_PASSWORD=validation-admin-only + export MODELFORGE_MIGRATION_DB_PASSWORD=validation-owner-only + export MODELFORGE_RUNTIME_DB_PASSWORD=validation-runtime-only + export MODELFORGE_MIGRATION_DATABASE_URL=postgresql+psycopg://modelforge:validation-owner-only@postgres:5432/modelforge_validation + export MODELFORGE_RUNTIME_DATABASE_URL=postgresql+psycopg://modelforge_runtime:validation-runtime-only@postgres:5432/modelforge_validation + export MODELFORGE_OPERATOR_API_KEY=validation-operator-key-32-characters # gitleaks:allow — synthetic Compose interpolation only + export MODELFORGE_BACKUP_ENCRYPTION_KEY=dmFsaWRhdGlvbi1vbmx5LWtleS0zMi1ieXRlcw== # gitleaks:allow — base64 of a public validation-only string + export MODELFORGE_CORS_ORIGINS=https://modelforge.example.test + export VITE_API_BASE_URL=https://modelforge.example.test + export MODELFORGE_VERSION=1.2.1 + export MODELFORGE_AGENT_CONTROL_PLANE_HOST_ADDRESS=127.0.0.1 + export MODELFORGE_AGENT_CA_CERT_PATH=./config/ca.crt + failures=0 + check() { + if docker compose "$@" config -q >/dev/null 2>&1; then + echo " OK $*" + else + echo " FAIL $*"; failures=$((failures + 1)) + fi + } + check -f docker-compose.yml + for overlay in backup dr gpu node-agent node-recovery production runtime-worker; do + check -f docker-compose.yml -f "docker-compose.${overlay}.yml" + done + check -f docker-compose.yml -f docker-compose.node-agent.yml \ + -f docker-compose.node-agent.private-ca.yml + check -f docker-compose.yml -f docker-compose.runtime-worker.yml \ + -f docker-compose.runtime-worker.private-ca.yml + [[ "${failures}" -eq 0 ]] || { echo "${failures} projection(s) invalid" >&2; exit 1; } + + - name: Configuration documentation is current + if: ${{ env.PROFILE == 'full' }} + shell: bash + run: | + "${VENV}/bin/python" scripts/generate_configuration_docs.py --check + + # The rule that makes the rest of this meaningful. Every component above declares a manifest, + # so every component must have reported a test count. Zero tests where tests were expected is + # a failure, not a pass — that is exactly how the previous workflow reported success. + - name: Refuse a validation that silently ran no tests + if: ${{ env.PROFILE == 'full' || env.PROFILE == 'test' }} + shell: bash + run: | + set -euo pipefail + "${VENV}/bin/python" - <<'PY' + import sys + import xml.etree.ElementTree as ET + from pathlib import Path + + expected = { + "backend.xml": "backend", + "node-agent.xml": "node-agent", + "runtime-worker.xml": "runtime-worker", + "frontend.xml": "frontend", + } + reports = Path("reports") + failures = [] + total = 0 + for filename, component in expected.items(): + path = reports / filename + if not path.is_file(): + failures.append(f"{component}: no test report was produced") + continue + root = ET.parse(path).getroot() + suites = [root] if root.tag == "testsuite" else list(root.iter("testsuite")) + tests = sum(int(suite.get("tests", 0)) for suite in suites) + errors = sum(int(suite.get("errors", 0)) for suite in suites) + failed = sum(int(suite.get("failures", 0)) for suite in suites) + skipped = sum(int(suite.get("skipped", 0)) for suite in suites) + executed = tests - skipped + total += tests + print(f" {component:16} {tests:5} tests, {skipped} skipped, " + f"{failed} failed, {errors} errors") + if executed <= 0: + failures.append(f"{component}: {tests} tests collected, {executed} executed") + if failed or errors: + failures.append(f"{component}: {failed} failed, {errors} errors") + print(f" {'TOTAL':16} {total:5} tests") + if failures: + print("\nManaged validation refused:", file=sys.stderr) + for failure in failures: + print(f" - {failure}", file=sys.stderr) + sys.exit(1) + PY + + - name: Validation summary + if: always() + shell: bash + run: | + echo "commit: $(git rev-parse HEAD)" + echo "profile: ${PROFILE}" + ls -l reports/ 2>/dev/null || echo "no reports directory" + + # Gitea only lists dispatchable workflow files from the default branch. Reusing the dedicated + # workflow from the existing managed entry point lets an unmerged branch prove its exact public + # export on the server. Ordinary PR validation and every profile except an explicit `build` + # dispatch remain unchanged. + public_candidate_acceptance: + name: Public candidate server acceptance + if: ${{ gitea.event_name == 'workflow_dispatch' && inputs.profile == 'build' }} + uses: ./.gitea/workflows/public-candidate-acceptance.yml + with: + source_commit: ${{ gitea.sha }} + public_api_origin: https://modelforge.example.test diff --git a/.gitea/workflows/public-candidate-acceptance.yml b/.gitea/workflows/public-candidate-acceptance.yml new file mode 100644 index 0000000..4b526b2 --- /dev/null +++ b/.gitea/workflows/public-candidate-acceptance.yml @@ -0,0 +1,213 @@ +name: Public candidate server acceptance + +# Builds only an exact, curated public-source commit in a disposable Compose namespace. This job +# never calls the Unraid deploy controller and cannot select the production deployment action. +on: + workflow_dispatch: + inputs: + source_commit: + description: Exact canonical 40-character commit SHA to export and validate + required: true + type: string + public_api_origin: + description: Bare API origin compiled into the candidate Console image + required: true + default: https://modelforge.example.test + type: string + workflow_call: + inputs: + source_commit: + description: Exact canonical 40-character commit SHA to export and validate + required: true + type: string + public_api_origin: + description: Bare API origin compiled into the candidate Console image + required: true + type: string + +permissions: + contents: read + +concurrency: + group: public-candidate-acceptance-${{ inputs.source_commit }} + cancel-in-progress: false + +jobs: + acceptance: + name: Four images and isolated clean install + runs-on: ubuntu-latest + timeout-minutes: 240 + env: + GITLEAKS_VERSION: 8.30.1 + GITLEAKS_SHA256: 551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb + TRIVY_VERSION: 0.74.0 + TRIVY_SHA256: 2ae6fe3ee734b7fdf11335663e18c75ea12dccc76062f09f164a3b0f8be4371a + steps: + - name: Validate immutable acceptance inputs + shell: bash + env: + SOURCE_COMMIT: ${{ inputs.source_commit }} + PUBLIC_API_ORIGIN: ${{ inputs.public_api_origin }} + run: | + set -euo pipefail + [[ "${SOURCE_COMMIT}" =~ ^[0-9a-f]{40}$ ]] || { + echo "source_commit must be an exact lowercase commit SHA" >&2; exit 2; + } + python3 - "${PUBLIC_API_ORIGIN}" <<'PY' + import sys + from urllib.parse import urlparse + parsed = urlparse(sys.argv[1]) + if parsed.scheme not in {"http", "https"} or not parsed.netloc: + raise SystemExit("public_api_origin must be an absolute HTTP(S) origin") + if parsed.path or parsed.params or parsed.query or parsed.fragment: + raise SystemExit("public_api_origin must be a bare origin without a path") + PY + + - name: Check out the exact canonical source + uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + with: + ref: ${{ inputs.source_commit }} + fetch-depth: 0 + persist-credentials: false + + - name: Verify checkout and runner isolation toolchain + shell: bash + env: + SOURCE_COMMIT: ${{ inputs.source_commit }} + run: | + set -euo pipefail + [[ "$(git rev-parse HEAD)" == "${SOURCE_COMMIT}" ]] + [[ -z "$(git status --porcelain)" ]] + command -v python3 >/dev/null + command -v node >/dev/null + command -v docker >/dev/null + docker version + docker compose version + python3 -m unittest discover -s .gitea/tests -p 'test_*.py' -v + + - name: Install checksum-pinned acceptance tools + shell: bash + run: | + set -euo pipefail + tool_root="$(mktemp -d /tmp/modelforge-acceptance-tools.XXXXXXXX)" + trivy_archive="${tool_root}/trivy.tar.gz" + curl --fail --location --show-error --retry 3 --retry-all-errors \ + --output "${trivy_archive}" \ + "https://github.com/aquasecurity/trivy/releases/download/v${TRIVY_VERSION}/trivy_${TRIVY_VERSION}_Linux-64bit.tar.gz" + printf '%s %s\n' "${TRIVY_SHA256}" "${trivy_archive}" \ + | sha256sum --check --strict + tar --extract --gzip --file "${trivy_archive}" \ + --directory "${tool_root}" trivy + chmod 0755 "${tool_root}/trivy" + + gitleaks_archive="${tool_root}/gitleaks.tar.gz" + curl --fail --location --show-error --retry 3 --retry-all-errors \ + --output "${gitleaks_archive}" \ + "https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz" + printf '%s %s\n' "${GITLEAKS_SHA256}" "${gitleaks_archive}" \ + | sha256sum --check --strict + tar --extract --gzip --file "${gitleaks_archive}" \ + --directory "${tool_root}" gitleaks + chmod 0755 "${tool_root}/gitleaks" + + printf 'TRIVY=%s\n' "${tool_root}/trivy" >> "${GITHUB_ENV}" + printf 'GITLEAKS=%s\n' "${tool_root}/gitleaks" >> "${GITHUB_ENV}" + + - name: Export and validate the curated public source + shell: bash + env: + SOURCE_COMMIT: ${{ inputs.source_commit }} + run: | + set -euo pipefail + public_source="$(mktemp -d /tmp/modelforge-public-parent.XXXXXXXX)/candidate" + reports="$(mktemp -d /tmp/modelforge-public-reports.XXXXXXXX)" + node scripts/export-public-source.mjs \ + --output "${public_source}" --report "${reports}/export-report.json" + (cd "${public_source}" && node scripts/validate-public-source.mjs) + "${GITLEAKS}" dir "${public_source}" --no-banner --redact \ + --report-format json --report-path "${reports}/gitleaks.json" + source_date="$(git show -s --format=%cI "${SOURCE_COMMIT}")" + git -C "${public_source}" init --initial-branch=main + git -C "${public_source}" config user.name "ModelForge acceptance" + git -C "${public_source}" config user.email "acceptance@modelforge.invalid" + git -C "${public_source}" add --all + GIT_AUTHOR_DATE="${source_date}" GIT_COMMITTER_DATE="${source_date}" \ + git -C "${public_source}" commit -m "Public candidate from ${SOURCE_COMMIT}" + printf 'PUBLIC_SOURCE=%s\n' "${public_source}" >> "${GITHUB_ENV}" + printf 'ACCEPTANCE_REPORTS=%s\n' "${reports}" >> "${GITHUB_ENV}" + + - name: Build, scan and clean-install the public candidate + shell: bash + env: + PUBLIC_API_ORIGIN: ${{ inputs.public_api_origin }} + run: | + set -euo pipefail + cd "${PUBLIC_SOURCE}" + python3 scripts/rc_server_acceptance.py \ + --public-api-origin "${PUBLIC_API_ORIGIN}" \ + --trivy "${TRIVY}" \ + --output acceptance-evidence \ + --project-suffix "${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" + + - name: Always remove acceptance Docker resources + if: always() + shell: bash + run: | + set +e + cleanup_failed=0 + suffix="$(printf '%s' "${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" \ + | tr '[:upper:]' '[:lower:]' | tr -cd '[:alnum:]' | tail -c 24)" + project="modelforge-rc-${suffix}" + for kind in container volume network; do + while IFS= read -r resource; do + [[ -n "${resource}" ]] || continue + case "${kind}" in + container) docker container rm --force "${resource}" || cleanup_failed=1 ;; + volume) docker volume rm --force "${resource}" || cleanup_failed=1 ;; + network) docker network rm "${resource}" || cleanup_failed=1 ;; + esac + done < <(docker "${kind}" ls --quiet \ + --filter "label=com.docker.compose.project=${project}" 2>/dev/null) + done + + if [[ -n "${PUBLIC_SOURCE:-}" && -d "${PUBLIC_SOURCE}/.git" ]]; then + candidate_commit="$(git -C "${PUBLIC_SOURCE}" rev-parse HEAD 2>/dev/null)" + version="$(tr -d '\r\n' < "${PUBLIC_SOURCE}/VERSION" 2>/dev/null)" + for image in modelforge-api modelforge-web modelforge-node-agent \ + modelforge-runtime-worker; do + tag="${image}:${version}" + revision="$(docker inspect --format \ + '{{index .Config.Labels "org.opencontainers.image.revision"}}' \ + "${tag}" 2>/dev/null)" + if [[ -n "${candidate_commit}" && "${revision}" == "${candidate_commit}" ]]; then + docker image rm --force "${tag}" || cleanup_failed=1 + fi + done + fi + leftovers="$(docker container ls --all --quiet \ + --filter "label=com.docker.compose.project=${project}" 2>/dev/null)" + [[ -z "${leftovers}" ]] || { + echo "Acceptance containers remain after cleanup: ${leftovers}" >&2 + exit 1 + } + [[ "${cleanup_failed}" == 0 ]] || { + echo "One or more exact acceptance resources could not be removed" >&2 + exit 1 + } + + - name: Upload acceptance evidence + if: always() + uses: https://gitea.com/actions/gitea-upload-artifact@81f940d004763f986ba3582c007fd842dd5cb0d7 + with: + name: public-candidate-${{ gitea.run_id }}-${{ gitea.run_attempt }}-${{ inputs.source_commit }} + path: | + ${{ env.ACCEPTANCE_REPORTS }}/export-report.json + ${{ env.ACCEPTANCE_REPORTS }}/gitleaks.json + ${{ env.PUBLIC_SOURCE }}/PUBLIC_SOURCE_EXPORT.md + ${{ env.PUBLIC_SOURCE }}/PUBLIC_SOURCE_MANIFEST.json + ${{ env.PUBLIC_SOURCE }}/acceptance-evidence/*.json + ${{ env.PUBLIC_SOURCE }}/acceptance-evidence/*.txt + ${{ env.PUBLIC_SOURCE }}/acceptance-evidence/release/*.json + ${{ env.PUBLIC_SOURCE }}/acceptance-evidence/release/*SHA256SUMS + if-no-files-found: warn + retention-days: 90 diff --git a/.gitea/workflows/unraid-deploy.yml b/.gitea/workflows/unraid-deploy.yml new file mode 100644 index 0000000..ee2a351 --- /dev/null +++ b/.gitea/workflows/unraid-deploy.yml @@ -0,0 +1,132 @@ +name: Unraid stable release deployment + +on: + workflow_dispatch: + inputs: + release_tag: + description: Annotated stable release tag (for example, v1.2.1) + required: true + type: string + release_commit: + description: Exact lowercase 40-character commit SHA referenced by the tag + required: true + type: string + action: + description: Verify provenance only, or deploy the verified stable release + required: true + default: VERIFY_ONLY + type: choice + options: + - VERIFY_ONLY + - DEPLOY_STABLE_TO_PRODUCTION + +concurrency: + group: unraid-production-itworx-modelforge + cancel-in-progress: false + +permissions: + contents: read + +jobs: + deploy: + name: Verify and optionally deploy a stable release + runs-on: unraid-deploy + timeout-minutes: 180 + steps: + - name: Validate immutable dispatch inputs + shell: bash + env: + RELEASE_TAG: ${{ inputs.release_tag }} + RELEASE_COMMIT: ${{ inputs.release_commit }} + DEPLOY_ACTION: ${{ inputs.action }} + run: | + set -euo pipefail + + if [[ "${GITHUB_REF:-}" != "refs/heads/master" ]]; then + echo "Stable production deployment must be dispatched from refs/heads/master" >&2 + exit 2 + fi + if [[ ! "${RELEASE_TAG}" =~ ^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$ ]]; then + echo "release_tag must be a strict stable SemVer tag such as v1.2.1" >&2 + exit 2 + fi + if [[ ! "${RELEASE_COMMIT}" =~ ^[0-9a-f]{40}$ ]]; then + echo "release_commit must be an exact lowercase 40-character SHA" >&2 + exit 2 + fi + case "${DEPLOY_ACTION}" in + VERIFY_ONLY|DEPLOY_STABLE_TO_PRODUCTION) ;; + *) echo "action is not allowlisted: ${DEPLOY_ACTION}" >&2; exit 2 ;; + esac + + - name: Check out the exact release commit with full history + uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + with: + ref: ${{ inputs.release_commit }} + fetch-depth: 0 + # The following provenance step performs an authenticated exact-ref fetch. + persist-credentials: true + + - name: Verify stable release provenance + shell: bash + env: + RELEASE_TAG: ${{ inputs.release_tag }} + RELEASE_COMMIT: ${{ inputs.release_commit }} + run: | + set -euo pipefail + + git fetch --force --no-recurse-submodules origin \ + "refs/tags/${RELEASE_TAG}:refs/tags/${RELEASE_TAG}" \ + "refs/heads/master:refs/remotes/origin/master" + + tag_ref="refs/tags/${RELEASE_TAG}" + if [[ "$(git cat-file -t "${tag_ref}")" != "tag" ]]; then + echo "${RELEASE_TAG} must be an annotated tag" >&2 + exit 1 + fi + + tag_object="$(git rev-parse "${tag_ref}^{tag}")" + tag_commit="$(git rev-parse "${tag_ref}^{commit}")" + head_commit="$(git rev-parse HEAD)" + if [[ "${tag_commit}" != "${RELEASE_COMMIT}" ]]; then + echo "Tag commit ${tag_commit} does not match release_commit ${RELEASE_COMMIT}" >&2 + exit 1 + fi + if [[ "${head_commit}" != "${RELEASE_COMMIT}" ]]; then + echo "Checked-out HEAD ${head_commit} does not match release_commit ${RELEASE_COMMIT}" >&2 + exit 1 + fi + + if [[ ! -f VERSION ]]; then + echo "VERSION is missing at the release commit" >&2 + exit 1 + fi + version="$(tr -d '\r\n' < VERSION)" + expected_version="${RELEASE_TAG#v}" + if [[ "${version}" != "${expected_version}" ]]; then + echo "VERSION ${version} does not match release tag ${RELEASE_TAG}" >&2 + exit 1 + fi + + if ! git merge-base --is-ancestor "${RELEASE_COMMIT}" refs/remotes/origin/master; then + echo "Release commit ${RELEASE_COMMIT} is not an ancestor of origin/master" >&2 + exit 1 + fi + + printf 'DEPLOY_COMMIT=%s\n' "${RELEASE_COMMIT}" >> "${GITHUB_ENV}" + echo "Verified annotated ${RELEASE_TAG} object ${tag_object} at immutable commit ${RELEASE_COMMIT}" + + - name: Verification-only result + if: ${{ inputs.action == 'VERIFY_ONLY' }} + shell: bash + run: echo "Stable release provenance verified; production was not changed." + + - name: Deploy verified stable release to production + if: ${{ inputs.action == 'DEPLOY_STABLE_TO_PRODUCTION' }} + shell: bash + run: | + set -euo pipefail + test -n "${DEPLOY_COMMIT:-}" + docker exec gitea-deploy-control \ + /opt/gitea-deploy/deploy.py deploy \ + "${GITHUB_REPOSITORY}" "${DEPLOY_COMMIT}" diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..eb78e70 --- /dev/null +++ b/.gitignore @@ -0,0 +1,123 @@ +# Python environments, bytecode, analysis and test output +__pycache__/ +*.py[cod] +*$py.class +.venv/ +venv/ +env/ +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ +.hypothesis/ +.tox/ +.nox/ +.coverage +.coverage.* +coverage.xml +htmlcov/ +*.egg-info/ +.eggs/ + +# Node/frontend dependencies, caches and build output +node_modules/ +dist/ +build/ +.vite/ +.turbo/ +.parcel-cache/ +.npm/ +.yarn/ +.pnpm-store/ +.playwright-mcp/ +playwright-report/ +test-results/ +*.tsbuildinfo +npm-debug.log* +yarn-debug.log* +yarn-error.log* +pnpm-debug.log* + +# Local environment and credentials. Safe examples are explicitly allowed. +.env +.env.* +!.env.example +!.env.*.example +# Credential *material*, not every filename that mentions credentials. The bare substring globs +# that used to live here matched source: backend/tests/test_credential_security_m16.py — the entire +# M16 credential security suite, 35 tests — and docs/security/CREDENTIAL_THREAT_TESTS.md were +# silently excluded from the repository while being reported as delivered. +*.credentials +*.secret +*.token +credentials.json +credentials.yaml +credentials.yml +secrets.json +secrets.yaml +secrets.yml +*.pem +*.key +*.p12 +*.pfx +id_rsa* +id_ed25519* + +# IDE, OS and local browser/session state +.vscode/ +.idea/ +*.suo +*.user +*.userosscache +*.sln.docstates +.DS_Store +Thumbs.db +Desktop.ini +*.stackdump + +# ModelForge runtime, container and generated evidence state +.data/ +.state/ +state/ +data/ +# Model weights, not configuration. This rule used to be a bare `models/`, which also matched +# config/models/ and silently excluded the candidate registry manifest the control plane +# needs at startup — a clean clone of the repository could not start. +/models/ +model-artifacts/ +hf-cache/ +huggingface/ +.cache/huggingface/ +artifacts/ +.quarantine/ +quarantine/ +downloads/ +container-state/ +postgres-data/ +redis-data/ +runtime-traces/ +traces/ +benchmarks/results/ +benchmarks/generated/ +*.sqlite +*.sqlite3 +*.db +*.db-shm +*.db-wal + +# Model weights and derived checkpoints are never source artifacts. +*.safetensors +*.gguf +*.bin +*.pt +*.pth +*.ckpt +*.onnx + +# Logs and temporary output +*.log +logs/ +tmp/ +temp/ +*.tmp +*.bak +dist-*/ diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..354c1c9 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,262 @@ +# Changelog + +Operator- and user-facing changes. Internal refactoring, test additions and documentation-only +commits are not listed unless they change what an operator sees or does. + +## Unreleased + +### Security + +- Runtime Worker packaging now upgrades the digest-pinned Ubuntu base before installation and pins + the inference stack plus inherited Python security dependencies to reviewed fixed versions. The + public-candidate gate scans the resulting fourth image instead of treating the private worker as + an unexamined exception. +- The Runtime Worker removes unneeded kernel-development headers after installation, eliminating + their upstream-unfixed vulnerability surface without removing required runtime libraries. +- Public candidate scans distinguish fixable or newly observed HIGH/CRITICAL findings from an + exact reviewed upstream-unfixed baseline. A changed CVE, package, installed version, severity or + newly available fix fails closed. +- Node enrollment and publisher credentials now require their exact fixed scopes at both service + and database boundaries. Schema `20260830_0023` validates every existing row before adding the + constraints and refuses the upgrade when malformed legacy scope data exists. +- Request-body streaming now has finite ASGI event/progress budgets in addition to byte limits; + progressless frame floods receive the typed `request_body_progress_exhausted` response. +- PostgreSQL role provisioning now removes the database's default public temporary-table grant; + production startup continues to attest that the runtime role cannot create or use it. + +### Fixed + +- Fresh schema `20260830_0024` installs no longer let Psycopg misread PostgreSQL `%rowtype` and + `%I` syntax as client placeholders. +- Public-candidate acceptance now works across the Gitea sibling Docker daemon while retaining + isolated volumes, private workers, exact HTTP contract probes and deterministic cleanup. + +## v1.2.1 — 2026-08-30 + +A packaging and provenance patch. No feature, API, schema or runtime-behaviour change; schema stays +`20260828_0022` and no migration runs. It exists because v1.2.0's production acceptance found two +defects that every existing gate was blind to — both in the artifacts rather than the source. + +### Fixed + +- **The published console could not reach its own API.** Vite inlines `VITE_API_BASE_URL` at build + time, but the release build never passed it, so every release image since v1.1.0 compiled the + Dockerfile's development default `http://localhost:8000` into an immutable bundle — and the nginx + CSP, derived from the same argument, hardcoded the same wrong origin. A release build now requires + an explicit `--public-api-origin` (or `MODELFORGE_PUBLIC_API_ORIGIN`) and refuses to package + without one. +- **The Node Agent's identity could drift from its tag.** `docker-compose.node-agent.yml` declared a + build with no arguments, so a Compose-built agent carried version `0.0.0` and empty + revision/created labels while still being tagged from `MODELFORGE_VERSION`. Production ran an + image tagged `1.1.1` whose contents were `1.2.0`. The projection now passes the release identity, + as the API and console projections already did. + +### Added + +- `scripts/release_image_acceptance.py`: a gate that inspects the **built** console image — its + compiled bundle, its rendered CSP and its OCI labels — rather than the source that produced it. + The release build runs it before packaging and refuses to publish an image that cannot reach the + API it was built for. Verified against the exact image that broke production. +- Regression coverage for both defects: the release build fails closed without an origin, the origin + reaches the console build, the CSP names the same origin and is never widened to a wildcard, every + release image declares complete OCI labels, and every Compose projection that builds a published + image passes the release identity. + +### Operations + +- The release manifest now records the console's compiled-in API origin, so an operator can see it + without unpacking the image. +- Local development is unchanged: `docker compose up` still defaults to `http://localhost:8000`. + Fail-closed applies to the release path only. + +## v1.2.0 — 2026-08-29 + +The Premium Operator Console release. No API contract, domain model, runtime behavior or database +schema changed; schema stays `20260828_0022` and no migration runs. + +### Added + +- A production command center built only from backend-authoritative evidence, stating unknowns as + unknown rather than as zero. +- Quick jump (Ctrl+K): a modal command palette over all 13 workspaces, matching name, domain and + keywords. +- An evidence timeline of recent control-plane activity, each entry linking to the workspace where + it can be verified. +- A split-pane Registry detail view covering upstream facts, local governance, revisions, artifacts + and provenance. +- One console-wide guarded destructive-action dialog showing identity, dependency posture and + permanent impact before an explicit acknowledgement unlocks the confirm control. + +### Changed + +- Workspaces are grouped by operator mental model (Command, Model supply, Serving, Infrastructure, + Assurance, Projects) while every existing route hash is preserved. +- One refined design token layer for light and dark, with semantic status roles that always include + text and explicit focus rings. +- Every route is usable from 1440 px down to a 390 px mobile viewport through a modal navigation + drawer, with no horizontal overflow at any tested width. + +### Fixed + +- The ARIA tabs pattern is complete across all six tabbed workspaces: every tab carries a stable + `id` and `aria-controls`, and every panel is a real `role="tabpanel"` naming its tab back. +- Lifecycle, Migrations and Recovery tablists gained roving `tabindex`, an accessible tablist name + and Left/Right/Home/End keyboard movement, which they previously lacked entirely. +- Capability residency unload no longer uses a native `window.confirm`; it uses the console safety + dialog and shows the VRAM released and the cold-load cost the next request will pay. +- Operator token fields in Operations, Recovery, Lifecycle and Migrations now sit in a real form + with a stable `id`, a `name` and an explicit `