Publish DevRunbook source

docs/17-observability-operations.md

@@ -0,0 +1,116 @@
+# 17 — Observability and operations
+
+## Health model
+
+### Liveness
+
+Process event loop is responsive. It should not fail solely because PostgreSQL or Gitea is temporarily unavailable.
+
+### Readiness
+
+Required components:
+
+- PostgreSQL reachable;
+- schema migration compatible;
+- required storage directory writable;
+- critical configuration valid;
+- encryption key available for configured integration secrets.
+
+Optional Gitea integrations do not make the entire application unready.
+
+## Structured logs
+
+Common fields:
+
+- timestamp;
+- level;
+- service role;
+- request/job ID;
+- user/workspace ID where safe;
+- route or job type;
+- duration;
+- outcome;
+- safe error code;
+- playbook/run/integration IDs when relevant.
+
+Use redaction middleware and unit tests for logger serialization.
+
+## Metrics
+
+Suggested metrics:
+
+- HTTP request count/latency/errors;
+- composition preview count/latency;
+- generated runs by type and outcome;
+- prompt-lint findings by rule;
+- package imports and failures;
+- worker queue depth, age and retries;
+- artifact generation size/latency;
+- Gitea request count/latency/error class;
+- repository snapshot duration and evidence volume;
+- database connection pool state;
+- active sessions;
+- storage usage.
+
+Metrics must avoid high-cardinality raw repository names or user text.
+
+## Audit events
+
+Security and governance events:
+
+- account login/logout/password reset;
+- integration create/test/rotate/delete;
+- repository connect/disconnect;
+- playbook import/publish/deprecate/delete draft;
+- generated run creation;
+- artifact download where policy requires;
+- retention or data deletion;
+- admin setting change;
+- failed authorization attempt at a meaningful boundary.
+
+Audit events contain safe metadata and are append-only at application level.
+
+## Job operations
+
+Admin view shows:
+
+- queued/running/failed/completed;
+- job type;
+- age and duration;
+- current stage/progress;
+- attempt count;
+- safe error;
+- retryability;
+- related repository, integration or artifact;
+- manual retry/cancel where safe.
+
+A worker uses leases so abandoned jobs can recover after process failure. Non-idempotent work must use transactional outbox or explicit idempotency state.
+
+## Alerts
+
+Self-hosted default avoids external alert dependency. Provide health endpoint and logs suitable for existing monitoring.
+
+Recommended alert conditions:
+
+- readiness failing longer than threshold;
+- worker queue oldest age excessive;
+- repeated package import failures;
+- artifact storage nearly full;
+- database migration mismatch;
+- integration token decryption failure;
+- repeated authentication failures;
+- backup not completed according to external schedule.
+
+## Support bundle
+
+Future or release-candidate operator action can create a privacy-safe diagnostic ZIP containing:
+
+- application and schema versions;
+- redacted configuration summary;
+- component health;
+- recent safe job errors;
+- migration status;
+- storage capacity summary;
+- optional logs after explicit review.
+
+It must exclude tokens, cookies, complete prompts, repository source and user email by default.