117 lines
3.0 KiB
Markdown
117 lines
3.0 KiB
Markdown
# 17 — Observability and operations
|
|
|
|
## Health model
|
|
|
|
### Liveness
|
|
|
|
Process event loop is responsive. It should not fail solely because PostgreSQL or Gitea is temporarily unavailable.
|
|
|
|
### Readiness
|
|
|
|
Required components:
|
|
|
|
- PostgreSQL reachable;
|
|
- schema migration compatible;
|
|
- required storage directory writable;
|
|
- critical configuration valid;
|
|
- encryption key available for configured integration secrets.
|
|
|
|
Optional Gitea integrations do not make the entire application unready.
|
|
|
|
## Structured logs
|
|
|
|
Common fields:
|
|
|
|
- timestamp;
|
|
- level;
|
|
- service role;
|
|
- request/job ID;
|
|
- user/workspace ID where safe;
|
|
- route or job type;
|
|
- duration;
|
|
- outcome;
|
|
- safe error code;
|
|
- playbook/run/integration IDs when relevant.
|
|
|
|
Use redaction middleware and unit tests for logger serialization.
|
|
|
|
## Metrics
|
|
|
|
Suggested metrics:
|
|
|
|
- HTTP request count/latency/errors;
|
|
- composition preview count/latency;
|
|
- generated runs by type and outcome;
|
|
- prompt-lint findings by rule;
|
|
- package imports and failures;
|
|
- worker queue depth, age and retries;
|
|
- artifact generation size/latency;
|
|
- Gitea request count/latency/error class;
|
|
- repository snapshot duration and evidence volume;
|
|
- database connection pool state;
|
|
- active sessions;
|
|
- storage usage.
|
|
|
|
Metrics must avoid high-cardinality raw repository names or user text.
|
|
|
|
## Audit events
|
|
|
|
Security and governance events:
|
|
|
|
- account login/logout/password reset;
|
|
- integration create/test/rotate/delete;
|
|
- repository connect/disconnect;
|
|
- playbook import/publish/deprecate/delete draft;
|
|
- generated run creation;
|
|
- artifact download where policy requires;
|
|
- retention or data deletion;
|
|
- admin setting change;
|
|
- failed authorization attempt at a meaningful boundary.
|
|
|
|
Audit events contain safe metadata and are append-only at application level.
|
|
|
|
## Job operations
|
|
|
|
Admin view shows:
|
|
|
|
- queued/running/failed/completed;
|
|
- job type;
|
|
- age and duration;
|
|
- current stage/progress;
|
|
- attempt count;
|
|
- safe error;
|
|
- retryability;
|
|
- related repository, integration or artifact;
|
|
- manual retry/cancel where safe.
|
|
|
|
A worker uses leases so abandoned jobs can recover after process failure. Non-idempotent work must use transactional outbox or explicit idempotency state.
|
|
|
|
## Alerts
|
|
|
|
Self-hosted default avoids external alert dependency. Provide health endpoint and logs suitable for existing monitoring.
|
|
|
|
Recommended alert conditions:
|
|
|
|
- readiness failing longer than threshold;
|
|
- worker queue oldest age excessive;
|
|
- repeated package import failures;
|
|
- artifact storage nearly full;
|
|
- database migration mismatch;
|
|
- integration token decryption failure;
|
|
- repeated authentication failures;
|
|
- backup not completed according to external schedule.
|
|
|
|
## Support bundle
|
|
|
|
Future or release-candidate operator action can create a privacy-safe diagnostic ZIP containing:
|
|
|
|
- application and schema versions;
|
|
- redacted configuration summary;
|
|
- component health;
|
|
- recent safe job errors;
|
|
- migration status;
|
|
- storage capacity summary;
|
|
- optional logs after explicit review.
|
|
|
|
It must exclude tokens, cookies, complete prompts, repository source and user email by default.
|