# 17 — Observability and operations

## Health model

### Liveness

Process event loop is responsive. It should not fail solely because PostgreSQL or Gitea is temporarily unavailable.

### Readiness

Required components:

- PostgreSQL reachable;
- schema migration compatible;
- required storage directory writable;
- critical configuration valid;
- encryption key available for configured integration secrets.

Optional Gitea integrations do not make the entire application unready.

## Structured logs

Common fields:

- timestamp;
- level;
- service role;
- request/job ID;
- user/workspace ID where safe;
- route or job type;
- duration;
- outcome;
- safe error code;
- playbook/run/integration IDs when relevant.

Use redaction middleware and unit tests for logger serialization.

## Metrics

Suggested metrics:

- HTTP request count/latency/errors;
- composition preview count/latency;
- generated runs by type and outcome;
- prompt-lint findings by rule;
- package imports and failures;
- worker queue depth, age and retries;
- artifact generation size/latency;
- Gitea request count/latency/error class;
- repository snapshot duration and evidence volume;
- database connection pool state;
- active sessions;
- storage usage.

Metrics must avoid high-cardinality raw repository names or user text.

## Audit events

Security and governance events:

- account login/logout/password reset;
- integration create/test/rotate/delete;
- repository connect/disconnect;
- playbook import/publish/deprecate/delete draft;
- generated run creation;
- artifact download where policy requires;
- retention or data deletion;
- admin setting change;
- failed authorization attempt at a meaningful boundary.

Audit events contain safe metadata and are append-only at application level.

## Job operations

Admin view shows:

- queued/running/failed/completed;
- job type;
- age and duration;
- current stage/progress;
- attempt count;
- safe error;
- retryability;
- related repository, integration or artifact;
- manual retry/cancel where safe.

A worker uses leases so abandoned jobs can recover after process failure. Non-idempotent work must use transactional outbox or explicit idempotency state.

## Alerts

Self-hosted default avoids external alert dependency. Provide health endpoint and logs suitable for existing monitoring.

Recommended alert conditions:

- readiness failing longer than threshold;
- worker queue oldest age excessive;
- repeated package import failures;
- artifact storage nearly full;
- database migration mismatch;
- integration token decryption failure;
- repeated authentication failures;
- backup not completed according to external schedule.

## Support bundle

Future or release-candidate operator action can create a privacy-safe diagnostic ZIP containing:

- application and schema versions;
- redacted configuration summary;
- component health;
- recent safe job errors;
- migration status;
- storage capacity summary;
- optional logs after explicit review.

It must exclude tokens, cookies, complete prompts, repository source and user email by default.