ModelForge/docs/architecture/LIFECYCLE_AUDIT.md

Lifecycle Audit

Lifecycle audit is append-only and change-oriented. Events include approval request/decision/stale, plan creation/approval/start, canary start/observation/abort/review, commit, rollback completion or failure, deprecation/archive, retention and cleanup planning/execution.

Each event stores:

• object type/reference and from/to state;
• requester, approver, executor or system actor role;
• policy revision and exact evidence references;
• human reason, timestamp and correlation/change ID;
• bounded operational metadata such as counts, latency, trigger and fingerprints.

It explicitly excludes model inputs, OCR/ASR/Vision content, project payloads, bearer credentials, tokens and sensitive host paths. SQLAlchemy prevents update/delete of lifecycle events, while normal database foreign-key restrictions preserve their referenced provenance.

Audit-chain integrity

AuditEvent has an explicit hash format. Legacy v1 rows retain their exact historical canonical JSON encoding: correlation id, actor type/id, action, resource type/id, outcome, bounded details and previous-event hash. The 20260830_0024 migration validates that complete chain before changing any schema, marks those rows v1, and records the first permitted v2 sequence. It never rewrites an approved event or silently repairs an invalid hash. A versioned legacy-prefix seal commits every legacy sequence, UUID, normalized UTC timestamp and event hash, making the two fields that were not inside the old hash tamper-evident after cutover.

New v2 events are assigned their UUID and UTC/microsecond timestamp before the append call. In production, the database-owned canonical function builds the exact payload text, calculates its SHA-256, inserts that text and advances the checkpoint atomically. SQLite tests use the same stored payload contract in Python. Both identity values and the v2 format identifier join the payload. Sequence must be exactly 1..n; the first previous hash is null and every later link is the immediately preceding canonical event hash. The executable verifier recomputes the v1/v2 hash appropriate to each sequence and the legacy-prefix seal. It also requires every event UUID and sequence to be unique, independent of whatever constraints a restored schema currently exposes.

The durable audit_chain_heads singleton records event count, final sequence/hash, current format, v2 cutover and legacy-prefix count/seal. It therefore detects middle gaps as well as deletion of the tail or the entire event table. SQLAlchemy ORM, Core, aliased/annotated targets, legacy bulk and textual DML for both audit tables fail closed at the ordinary application Session boundary. That includes raw exec_driver_sql on a Session-owned connection; its lexer ignores comments and string/dollar literals before classifying schema-qualified quoted targets. The sole persistence operation accepts one semantic event, constructs the exact insert and head compare-and-set itself, and exposes no reusable token, execution option, session marker or connection marker. The guard admits only those exact active statement objects and operation types.

The SQLAlchemy layer also registers every Engine built by the application, guards direct engine.begin()/session.bind.begin() connections, and rolls back protected DML introduced by a later statement-rewrite listener. It defaults raw DDL and procedural calls that could hide audit mutation to deny. This remains defense in depth: it is not a claim that hostile same-process Python cannot remove a listener or open a new DBAPI connection.

The production authority is PostgreSQL. modelforge_runtime is a NOINHERIT non-owner with SELECT on both audit tables and EXECUTE only on modelforge_audit.append_event_v2(...). Direct INSERT/UPDATE/DELETE/TRUNCATE, trigger/DDL creation, all SET ROLE-capable memberships and every other non-system function are unavailable. The canonical function is owned by the non-superuser modelforge migration role, fixes search_path=pg_catalog, takes the shared advisory lock and checkpoint row lock, validates the checkpoint and retained tail, constructs/hashes exact bytes, inserts, and advances the head by CAS. Owner-check triggers protect row mutation and TRUNCATE as a second database layer. Production startup reads catalog truth and refuses an owner/superuser runtime, role membership, forbidden grants, a changed function-body digest, wrong owner/SECURITY/search-path attributes, PUBLIC execution, a non-origin or wrong-event trigger, schema/database CREATE, or any unexpected executable non-system function. The API container receives no migration or bootstrap credential. A database owner/superuser remains a privileged administrative boundary; corruption tests use an independently created admin Engine and strict gates detect it.

Migration verifies and seals the complete immutable legacy prefix. Recovery and the executable audit_chain_intact invariant remain strict full-chain gates. A normal append does not repeat that linear proof: under the writer lock it validates the checkpoint shape, the two-row retained tail, the tail payload/hash/link and the checkpoint count, then inserts one event and advances the head by compare-and-set. That path is O(1) in retained history instead of cumulative O(n²). A database superuser could still edit a non-tail middle row without making the next append fail; this is not claimed otherwise. The strict invariant and every recovery READY transition detect that privileged tamper. PostgreSQL uses a shared transaction-scoped advisory lock; SQLite test databases use an engine-scoped process lock held until commit or rollback.

Restore reconciliation is the only writer aimed at another database. It reads audit rows in bounded ordered pages and strictly verifies events plus checkpoint before consulting an ALREADY_APPLIED marker, immediately before appending a marker, and independently before a restore can become READY. The marker calls the same SECURITY DEFINER canonical function, which advances event plus head atomically under the same advisory lock. Semantic-fingerprint allowances cannot turn an audit-chain verification failure into a passing restore.

A production-shaped legacy database containing the pre-RC direct recovery marker with a random event_hash is intentionally a migration blocker: 0024 fails before adding its columns or checkpoint. RC-QUAL-05 must discover and surface that state during rehearsal. The migration does not bless, rewrite or discard it. Downgrade to 0022 is allowed only before the first v2 event; afterwards it would require rewriting immutable history and is refused.

On PostgreSQL, migration 0024 takes the runtime advisory key and an ACCESS EXCLUSIVE lock on audit_events before validation. The table lock also serializes older 0022 writers that do not know the advisory key, and remains held through format marking and checkpoint seed. SQLite uses a test-only no-op write to acquire its writer transaction before validation. A live PostgreSQL concurrent-upgrade exercise remains a managed-runner gate; the unit contract asserts both locks.

AuditContext is the integration seam for a verified principal and request correlation id. Legacy services retain their current actor labels until the operator authentication boundary supplies that context; the hash-chain implementation does not fabricate principal identity. Consequently this integrity change alone does not claim that authenticated principal/correlation integration is complete.

MigrationEvent follows the same append-only/content-free rule and adds plan generation plus exact source/target identity. Per-batch audit is bounded to one summary event per completed configured batch; errors are capped and vectors, bodies, queries and secrets are excluded.

Node lifecycle extension

Permanent compute-node removal writes one NodeDecommissionOperation and one NODE_DECOMMISSIONED event in the same transaction as the tombstone and credential revocation. The event retains operator, reason, timestamp, previous state, cleanup counts and credential result. See NODE_DECOMMISSION.md.