DevRunbook-Public/docs/17-observability-operations.md

17 — Observability and operations

Health model

Liveness

Process event loop is responsive. It should not fail solely because PostgreSQL or Gitea is temporarily unavailable.

Readiness

Required components:

• PostgreSQL reachable;
• schema migration compatible;
• required storage directory writable;
• critical configuration valid;
• encryption key available for configured integration secrets.

Optional Gitea integrations do not make the entire application unready.

Structured logs

Common fields:

• timestamp;
• level;
• service role;
• request/job ID;
• user/workspace ID where safe;
• route or job type;
• duration;
• outcome;
• safe error code;
• playbook/run/integration IDs when relevant.

Use redaction middleware and unit tests for logger serialization.

Metrics

Suggested metrics:

• HTTP request count/latency/errors;
• composition preview count/latency;
• generated runs by type and outcome;
• prompt-lint findings by rule;
• package imports and failures;
• worker queue depth, age and retries;
• artifact generation size/latency;
• Gitea request count/latency/error class;
• repository snapshot duration and evidence volume;
• database connection pool state;
• active sessions;
• storage usage.

Metrics must avoid high-cardinality raw repository names or user text.

Audit events

Security and governance events:

• account login/logout/password reset;
• integration create/test/rotate/delete;
• repository connect/disconnect;
• playbook import/publish/deprecate/delete draft;
• generated run creation;
• artifact download where policy requires;
• retention or data deletion;
• admin setting change;
• failed authorization attempt at a meaningful boundary.

Audit events contain safe metadata and are append-only at application level.

Job operations

Admin view shows:

• queued/running/failed/completed;
• job type;
• age and duration;
• current stage/progress;
• attempt count;
• safe error;
• retryability;
• related repository, integration or artifact;
• manual retry/cancel where safe.

A worker uses leases so abandoned jobs can recover after process failure. Non-idempotent work must use transactional outbox or explicit idempotency state.

Alerts

Self-hosted default avoids external alert dependency. Provide health endpoint and logs suitable for existing monitoring.

Recommended alert conditions:

• readiness failing longer than threshold;
• worker queue oldest age excessive;
• repeated package import failures;
• artifact storage nearly full;
• database migration mismatch;
• integration token decryption failure;
• repeated authentication failures;
• backup not completed according to external schedule.

Support bundle

Future or release-candidate operator action can create a privacy-safe diagnostic ZIP containing:

• application and schema versions;
• redacted configuration summary;
• component health;
• recent safe job errors;
• migration status;
• storage capacity summary;
• optional logs after explicit review.

It must exclude tokens, cookies, complete prompts, repository source and user email by default.