114 lines
8.0 KiB
Markdown
114 lines
8.0 KiB
Markdown
# Lifecycle Audit
|
|
|
|
Lifecycle audit is append-only and change-oriented. Events include approval request/decision/stale,
|
|
plan creation/approval/start, canary start/observation/abort/review, commit, rollback completion or
|
|
failure, deprecation/archive, retention and cleanup planning/execution.
|
|
|
|
Each event stores:
|
|
|
|
- object type/reference and from/to state;
|
|
- requester, approver, executor or system actor role;
|
|
- policy revision and exact evidence references;
|
|
- human reason, timestamp and correlation/change ID;
|
|
- bounded operational metadata such as counts, latency, trigger and fingerprints.
|
|
|
|
It explicitly excludes model inputs, OCR/ASR/Vision content, project payloads, bearer credentials,
|
|
tokens and sensitive host paths. SQLAlchemy prevents update/delete of lifecycle events, while normal
|
|
database foreign-key restrictions preserve their referenced provenance.
|
|
|
|
## Audit-chain integrity
|
|
|
|
`AuditEvent` has an explicit hash format. Legacy `v1` rows retain their exact historical canonical
|
|
JSON encoding: correlation id, actor type/id, action, resource type/id, outcome, bounded details and
|
|
previous-event hash. The `20260830_0024` migration validates that complete chain before changing any
|
|
schema, marks those rows `v1`, and records the first permitted `v2` sequence. It never rewrites an
|
|
approved event or silently repairs an invalid hash. A versioned legacy-prefix seal commits every
|
|
legacy sequence, UUID, normalized UTC timestamp and event hash, making the two fields that were not
|
|
inside the old hash tamper-evident after cutover.
|
|
|
|
New `v2` events are assigned their UUID and UTC/microsecond timestamp before the append call. In
|
|
production, the database-owned canonical function builds the exact payload text, calculates its
|
|
SHA-256, inserts that text and advances the checkpoint atomically. SQLite tests use the same stored
|
|
payload contract in Python. Both identity values and the `v2` format identifier join the payload.
|
|
Sequence must be exactly `1..n`; the first previous hash is null and every later link is
|
|
the immediately preceding canonical event hash. The executable verifier recomputes the v1/v2 hash
|
|
appropriate to each sequence and the legacy-prefix seal. It also requires every event UUID and
|
|
sequence to be unique, independent of whatever constraints a restored schema currently exposes.
|
|
|
|
The durable `audit_chain_heads` singleton records event count, final sequence/hash, current format,
|
|
v2 cutover and legacy-prefix count/seal. It therefore detects middle gaps as well as deletion of the
|
|
tail or the entire event table. SQLAlchemy ORM, Core, aliased/annotated targets, legacy bulk and
|
|
textual DML for both audit tables fail closed at the ordinary application `Session` boundary. That
|
|
includes raw `exec_driver_sql` on a `Session`-owned connection; its lexer ignores comments and
|
|
string/dollar literals before classifying schema-qualified quoted targets. The sole persistence
|
|
operation accepts one semantic event, constructs the exact insert and head compare-and-set itself,
|
|
and exposes no reusable token, execution option, session marker or connection marker. The guard
|
|
admits only those exact active statement objects and operation types.
|
|
|
|
The SQLAlchemy layer also registers every Engine built by the application, guards direct
|
|
`engine.begin()`/`session.bind.begin()` connections, and rolls back protected DML introduced by a
|
|
later statement-rewrite listener. It defaults raw DDL and procedural calls that could hide audit
|
|
mutation to deny. This remains defense in depth: it is not a claim that hostile same-process Python
|
|
cannot remove a listener or open a new DBAPI connection.
|
|
|
|
The production authority is PostgreSQL. `modelforge_runtime` is a NOINHERIT non-owner with SELECT
|
|
on both audit tables and EXECUTE only on
|
|
`modelforge_audit.append_event_v2(...)`. Direct INSERT/UPDATE/DELETE/TRUNCATE, trigger/DDL creation,
|
|
all SET ROLE-capable memberships and every other non-system function are unavailable. The
|
|
canonical function is owned by the non-superuser `modelforge` migration role, fixes
|
|
`search_path=pg_catalog`, takes the shared advisory lock and checkpoint row lock, validates the
|
|
checkpoint and retained tail, constructs/hashes exact bytes, inserts, and advances the head by CAS.
|
|
Owner-check triggers protect row mutation and TRUNCATE as a second database layer. Production
|
|
startup reads catalog truth and refuses an owner/superuser runtime, role membership, forbidden
|
|
grants, a changed function-body digest, wrong owner/SECURITY/search-path attributes, PUBLIC
|
|
execution, a non-origin or wrong-event trigger, schema/database CREATE, or any unexpected
|
|
executable non-system function. The API container receives no
|
|
migration or bootstrap credential. A database owner/superuser remains a privileged administrative
|
|
boundary; corruption tests use an independently created admin Engine and strict gates detect it.
|
|
|
|
Migration verifies and seals the complete immutable legacy prefix. Recovery and the executable
|
|
`audit_chain_intact` invariant remain strict full-chain gates. A normal append does not repeat that
|
|
linear proof: under the writer lock it validates the checkpoint shape, the two-row retained tail,
|
|
the tail payload/hash/link and the checkpoint count, then inserts one event and advances the head by
|
|
compare-and-set. That path is O(1) in retained history instead of cumulative O(n²). A database
|
|
superuser could still edit a non-tail middle row without making the next append fail; this is not
|
|
claimed otherwise. The strict invariant and every recovery READY transition detect that privileged
|
|
tamper. PostgreSQL uses a shared transaction-scoped advisory lock; SQLite test databases use an
|
|
engine-scoped process lock held until commit or rollback.
|
|
|
|
Restore reconciliation is the only writer aimed at another database. It reads audit rows in bounded
|
|
ordered pages and strictly verifies events plus checkpoint before consulting an `ALREADY_APPLIED`
|
|
marker, immediately before appending a marker, and independently before a restore can become
|
|
`READY`. The marker calls the same SECURITY DEFINER canonical function, which advances event plus
|
|
head atomically under the same advisory lock. Semantic-fingerprint allowances cannot
|
|
turn an audit-chain verification failure into a passing restore.
|
|
|
|
A production-shaped legacy database containing the pre-RC direct recovery marker with a random
|
|
`event_hash` is intentionally a migration blocker: 0024 fails before adding its columns or checkpoint.
|
|
RC-QUAL-05 must discover and surface that state during rehearsal. The migration does not bless,
|
|
rewrite or discard it. Downgrade to 0022 is allowed only before the first v2 event; afterwards it
|
|
would require rewriting immutable history and is refused.
|
|
|
|
On PostgreSQL, migration 0024 takes the runtime advisory key and an `ACCESS EXCLUSIVE` lock on
|
|
`audit_events` before validation. The table lock also serializes older 0022 writers that do not know
|
|
the advisory key, and remains held through format marking and checkpoint seed. SQLite uses a
|
|
test-only no-op write to acquire its writer transaction before validation. A live PostgreSQL
|
|
concurrent-upgrade exercise remains a managed-runner gate; the unit contract asserts both locks.
|
|
|
|
`AuditContext` is the integration seam for a verified principal and request correlation id. Legacy
|
|
services retain their current actor labels until the operator authentication boundary supplies that
|
|
context; the hash-chain implementation does not fabricate principal identity. Consequently this
|
|
integrity change alone does **not** claim that authenticated principal/correlation integration is
|
|
complete.
|
|
|
|
`MigrationEvent` follows the same append-only/content-free rule and adds plan generation plus exact
|
|
source/target identity. Per-batch audit is bounded to one summary event per completed configured
|
|
batch; errors are capped and vectors, bodies, queries and secrets are excluded.
|
|
|
|
## Node lifecycle extension
|
|
|
|
Permanent compute-node removal writes one `NodeDecommissionOperation` and one
|
|
`NODE_DECOMMISSIONED` event in the same transaction as the tombstone and credential revocation.
|
|
The event retains operator, reason, timestamp, previous state, cleanup counts and credential result.
|
|
See `NODE_DECOMMISSION.md`.
|