Initial public ModelForge release

docs/architecture/LIFECYCLE_AUDIT.md

@@ -0,0 +1,113 @@
+# Lifecycle Audit
+
+Lifecycle audit is append-only and change-oriented. Events include approval request/decision/stale,
+plan creation/approval/start, canary start/observation/abort/review, commit, rollback completion or
+failure, deprecation/archive, retention and cleanup planning/execution.
+
+Each event stores:
+
+- object type/reference and from/to state;
+- requester, approver, executor or system actor role;
+- policy revision and exact evidence references;
+- human reason, timestamp and correlation/change ID;
+- bounded operational metadata such as counts, latency, trigger and fingerprints.
+
+It explicitly excludes model inputs, OCR/ASR/Vision content, project payloads, bearer credentials,
+tokens and sensitive host paths. SQLAlchemy prevents update/delete of lifecycle events, while normal
+database foreign-key restrictions preserve their referenced provenance.
+
+## Audit-chain integrity
+
+`AuditEvent` has an explicit hash format. Legacy `v1` rows retain their exact historical canonical
+JSON encoding: correlation id, actor type/id, action, resource type/id, outcome, bounded details and
+previous-event hash. The `20260830_0024` migration validates that complete chain before changing any
+schema, marks those rows `v1`, and records the first permitted `v2` sequence. It never rewrites an
+approved event or silently repairs an invalid hash. A versioned legacy-prefix seal commits every
+legacy sequence, UUID, normalized UTC timestamp and event hash, making the two fields that were not
+inside the old hash tamper-evident after cutover.
+
+New `v2` events are assigned their UUID and UTC/microsecond timestamp before the append call. In
+production, the database-owned canonical function builds the exact payload text, calculates its
+SHA-256, inserts that text and advances the checkpoint atomically. SQLite tests use the same stored
+payload contract in Python. Both identity values and the `v2` format identifier join the payload.
+Sequence must be exactly `1..n`; the first previous hash is null and every later link is
+the immediately preceding canonical event hash. The executable verifier recomputes the v1/v2 hash
+appropriate to each sequence and the legacy-prefix seal. It also requires every event UUID and
+sequence to be unique, independent of whatever constraints a restored schema currently exposes.
+
+The durable `audit_chain_heads` singleton records event count, final sequence/hash, current format,
+v2 cutover and legacy-prefix count/seal. It therefore detects middle gaps as well as deletion of the
+tail or the entire event table. SQLAlchemy ORM, Core, aliased/annotated targets, legacy bulk and
+textual DML for both audit tables fail closed at the ordinary application `Session` boundary. That
+includes raw `exec_driver_sql` on a `Session`-owned connection; its lexer ignores comments and
+string/dollar literals before classifying schema-qualified quoted targets. The sole persistence
+operation accepts one semantic event, constructs the exact insert and head compare-and-set itself,
+and exposes no reusable token, execution option, session marker or connection marker. The guard
+admits only those exact active statement objects and operation types.
+
+The SQLAlchemy layer also registers every Engine built by the application, guards direct
+`engine.begin()`/`session.bind.begin()` connections, and rolls back protected DML introduced by a
+later statement-rewrite listener. It defaults raw DDL and procedural calls that could hide audit
+mutation to deny. This remains defense in depth: it is not a claim that hostile same-process Python
+cannot remove a listener or open a new DBAPI connection.
+
+The production authority is PostgreSQL. `modelforge_runtime` is a NOINHERIT non-owner with SELECT
+on both audit tables and EXECUTE only on
+`modelforge_audit.append_event_v2(...)`. Direct INSERT/UPDATE/DELETE/TRUNCATE, trigger/DDL creation,
+all SET ROLE-capable memberships and every other non-system function are unavailable. The
+canonical function is owned by the non-superuser `modelforge` migration role, fixes
+`search_path=pg_catalog`, takes the shared advisory lock and checkpoint row lock, validates the
+checkpoint and retained tail, constructs/hashes exact bytes, inserts, and advances the head by CAS.
+Owner-check triggers protect row mutation and TRUNCATE as a second database layer. Production
+startup reads catalog truth and refuses an owner/superuser runtime, role membership, forbidden
+grants, a changed function-body digest, wrong owner/SECURITY/search-path attributes, PUBLIC
+execution, a non-origin or wrong-event trigger, schema/database CREATE, or any unexpected
+executable non-system function. The API container receives no
+migration or bootstrap credential. A database owner/superuser remains a privileged administrative
+boundary; corruption tests use an independently created admin Engine and strict gates detect it.
+
+Migration verifies and seals the complete immutable legacy prefix. Recovery and the executable
+`audit_chain_intact` invariant remain strict full-chain gates. A normal append does not repeat that
+linear proof: under the writer lock it validates the checkpoint shape, the two-row retained tail,
+the tail payload/hash/link and the checkpoint count, then inserts one event and advances the head by
+compare-and-set. That path is O(1) in retained history instead of cumulative O(n²). A database
+superuser could still edit a non-tail middle row without making the next append fail; this is not
+claimed otherwise. The strict invariant and every recovery READY transition detect that privileged
+tamper. PostgreSQL uses a shared transaction-scoped advisory lock; SQLite test databases use an
+engine-scoped process lock held until commit or rollback.
+
+Restore reconciliation is the only writer aimed at another database. It reads audit rows in bounded
+ordered pages and strictly verifies events plus checkpoint before consulting an `ALREADY_APPLIED`
+marker, immediately before appending a marker, and independently before a restore can become
+`READY`. The marker calls the same SECURITY DEFINER canonical function, which advances event plus
+head atomically under the same advisory lock. Semantic-fingerprint allowances cannot
+turn an audit-chain verification failure into a passing restore.
+
+A production-shaped legacy database containing the pre-RC direct recovery marker with a random
+`event_hash` is intentionally a migration blocker: 0024 fails before adding its columns or checkpoint.
+RC-QUAL-05 must discover and surface that state during rehearsal. The migration does not bless,
+rewrite or discard it. Downgrade to 0022 is allowed only before the first v2 event; afterwards it
+would require rewriting immutable history and is refused.
+
+On PostgreSQL, migration 0024 takes the runtime advisory key and an `ACCESS EXCLUSIVE` lock on
+`audit_events` before validation. The table lock also serializes older 0022 writers that do not know
+the advisory key, and remains held through format marking and checkpoint seed. SQLite uses a
+test-only no-op write to acquire its writer transaction before validation. A live PostgreSQL
+concurrent-upgrade exercise remains a managed-runner gate; the unit contract asserts both locks.
+
+`AuditContext` is the integration seam for a verified principal and request correlation id. Legacy
+services retain their current actor labels until the operator authentication boundary supplies that
+context; the hash-chain implementation does not fabricate principal identity. Consequently this
+integrity change alone does **not** claim that authenticated principal/correlation integration is
+complete.
+
+`MigrationEvent` follows the same append-only/content-free rule and adds plan generation plus exact
+source/target identity. Per-batch audit is bounded to one summary event per completed configured
+batch; errors are capped and vectors, bodies, queries and secrets are excluded.
+
+## Node lifecycle extension
+
+Permanent compute-node removal writes one `NodeDecommissionOperation` and one
+`NODE_DECOMMISSIONED` event in the same transaction as the tombstone and credential revocation.
+The event retains operator, reason, timestamp, previous state, cleanup counts and credential result.
+See `NODE_DECOMMISSION.md`.