This commit is contained in:
ITWorx Pulse release export
@@ -0,0 +1,185 @@
|
||||
# Alerting and incident architecture
|
||||
|
||||
## 1. Concepts
|
||||
|
||||
- **Rule:** versioned definition of a condition.
|
||||
- **Alert instance:** stable rule + entity/label fingerprint.
|
||||
- **Occurrence:** immutable state transition/evaluation history.
|
||||
- **Incident:** grouped operational problem containing related alerts/entities/events.
|
||||
- **Silence:** user-defined temporary notification/state suppression.
|
||||
- **Maintenance window:** scheduled policy for selected entities.
|
||||
- **Inhibition/suppression:** dependency-aware prevention of downstream noise.
|
||||
|
||||
## 2. State machine
|
||||
|
||||
Primary states:
|
||||
|
||||
```text
|
||||
inactive -> pending -> firing -> acknowledged -> resolved
|
||||
```
|
||||
|
||||
Orthogonal/contextual states:
|
||||
- unknown;
|
||||
- silenced;
|
||||
- suppressed;
|
||||
- maintenance.
|
||||
|
||||
Acknowledgement does not mean resolved. A firing alert may be acknowledged.
|
||||
|
||||
## 3. Evaluation
|
||||
|
||||
Every rule defines:
|
||||
- schedule/evaluation interval;
|
||||
- semantic query/event/status input;
|
||||
- scope selector;
|
||||
- comparator/condition;
|
||||
- pending duration;
|
||||
- recovery duration;
|
||||
- hysteresis;
|
||||
- severity;
|
||||
- message template;
|
||||
- grouping/suppression metadata;
|
||||
- unknown behavior.
|
||||
|
||||
Evaluation is idempotent and transactional.
|
||||
|
||||
## 4. Unknown behavior
|
||||
|
||||
When required data is stale/unavailable:
|
||||
- do not auto-resolve a firing alert;
|
||||
- transition/display unknown according to rule policy;
|
||||
- retain last known value/time;
|
||||
- emit source-health context;
|
||||
- avoid repeated notifications for the same unknown episode.
|
||||
|
||||
## 5. Hysteresis example
|
||||
|
||||
Disk temperature:
|
||||
- enter pending above 50°C;
|
||||
- fire after 5 minutes;
|
||||
- remain firing until below 46°C for 5 minutes;
|
||||
- cooldown duplicate notifications.
|
||||
|
||||
Both thresholds and durations are visible in rule UI.
|
||||
|
||||
## 6. Deduplication/grouping
|
||||
|
||||
Fingerprint is based on:
|
||||
- rule ID/version behavior;
|
||||
- stable entity ID;
|
||||
- bounded grouping labels.
|
||||
|
||||
Notifications group by:
|
||||
- incident;
|
||||
- host/application;
|
||||
- severity/time window;
|
||||
- configured labels.
|
||||
|
||||
## 7. Dependency suppression
|
||||
|
||||
Examples:
|
||||
- host unreachable suppresses container/service unreachable alerts;
|
||||
- DNS failure suppresses dependent service resolution alerts;
|
||||
- reverse proxy outage groups external endpoint failures while internal probes remain separate;
|
||||
- Prometheus unavailable suppresses metric-rule noise and creates a monitoring-source incident.
|
||||
|
||||
Suppressed alerts remain inspectable.
|
||||
|
||||
Pulse uses bounded, deterministic cause keys for suppression (`host.unreachable`, `dns.failure`, `source.unavailable`). A downstream alert is suppressed only when the configured cause is active and confirmed or high-confidence; the downstream signal, fingerprint and suppression reason remain inspectable. Group keys use rule identity, severity and configured stable labels with bounded cardinality. Duplicate evaluation signals are collapsed by instance and evaluation key before grouping.
|
||||
|
||||
## 8. Maintenance and silences
|
||||
|
||||
Maintenance:
|
||||
- scheduled;
|
||||
- selector-based;
|
||||
- audited;
|
||||
- optionally changes visual status to maintenance;
|
||||
- prevents configured notifications without deleting evidence.
|
||||
|
||||
Silence:
|
||||
- explicit reason;
|
||||
- owner;
|
||||
- expiry required;
|
||||
- matchers bounded/previewable;
|
||||
- audited.
|
||||
|
||||
## 9. Incident correlation
|
||||
|
||||
Deterministic rules first:
|
||||
- shared parent dependency;
|
||||
- same application/host;
|
||||
- close start time;
|
||||
- known causal hierarchy;
|
||||
- common event.
|
||||
|
||||
Heuristic correlation may be added later, but must expose confidence and rationale.
|
||||
|
||||
Incident severity is derived from:
|
||||
- highest unsuppressed alert;
|
||||
- criticality of affected entities;
|
||||
- duration/scope;
|
||||
- explicit operator override.
|
||||
|
||||
## 10. Notifications
|
||||
|
||||
Channels may include email/webhook/other selected integrations.
|
||||
|
||||
Requirements:
|
||||
- encrypted secret references;
|
||||
- idempotency;
|
||||
- retry with bounded exponential backoff;
|
||||
- delivery audit;
|
||||
- templates with safe escaping;
|
||||
- grouping and cooldown;
|
||||
- recovery notification;
|
||||
- test action;
|
||||
- channel failure health.
|
||||
|
||||
## 11. Default rules
|
||||
|
||||
Provide conservative baseline rules for:
|
||||
- monitoring source stale/down;
|
||||
- host resource/temperature;
|
||||
- container stop/health/restart loop;
|
||||
- array/pool/disk/SMART/capacity;
|
||||
- service availability/latency/TLS;
|
||||
- UPS battery/on-battery when available.
|
||||
|
||||
Defaults are versioned, visible and editable. Avoid hard thresholds where hardware-specific policy is needed; use sensible presets with onboarding confirmation.
|
||||
|
||||
M8-06 implementation note: silences and maintenance windows use explicit bounded matchers (rule IDs, entity IDs, entity types, severities, and exact labels). Preview returns deterministic matching instance IDs. Expiry is mandatory, state transitions are retained, and revocation or expiry never deletes alert occurrences. Maintenance state is exposed as scheduled, active, expired, or revoked.
|
||||
|
||||
M8-07 operation semantics: acknowledgement is an operator state annotation, not resolution. Acknowledgement and unacknowledgement are persisted as immutable occurrence events, use optimistic revisions, and are idempotent by evaluation key. Evaluator transitions remain authoritative for resolved state.
|
||||
|
||||
### M8-08 delivery semantics
|
||||
|
||||
The notification framework uses the alert lifecycle event type (`firing`, `recovery`, or `unknown`) as the outbox contract. Enqueue is idempotent by caller-supplied key. Workers claim due rows with a bounded lease and PostgreSQL row locking, write a `delivering` audit record, and complete the same attempt transactionally. Failures are redacted, retried with bounded exponential delay, and terminally marked failed after ten attempts. A recovery event is delivered through the same path.
|
||||
|
||||
The production transport is a bounded HTTPS webhook configured with `PULSE_NOTIFICATION_WEBHOOK_URL`, `PULSE_NOTIFICATION_WEBHOOK_TOKEN`, and an optional 1–30 second `PULSE_NOTIFICATION_WEBHOOK_TIMEOUT`. The worker reconciles one system-owned webhook channel at startup. PostgreSQL stores only the URL, timeout, and an opaque runtime secret reference; the bearer credential never enters channel JSON, delivery audit, API responses, or logs. Requests contain the lifecycle payload and repeat the stable outbox idempotency key in both the body and `Idempotency-Key` header. Receivers must use that key to collapse a replay when delivery succeeded but acknowledgement or local completion was interrupted. Redirects, URL credentials, query strings, fragments, unbounded timeouts, and non-HTTPS production endpoints are rejected.
|
||||
|
||||
Channel test actions are in-memory only, do not enqueue an outbox record, and are rate limited per actor/channel pair.
|
||||
### M8-09 correlation behavior
|
||||
|
||||
Correlation starts with deterministic keys: a primary dependency outage groups the dependency alert and downstream alerts sharing its parent; otherwise stable application, host, or entity keys are used. Groups are sorted by alert ID and correlation output is stable across input ordering. Every candidate stores the method and confidence used; the model does not present heuristic correlation as proven causation. Manual alert association is persisted separately, audited at the API boundary, and protected from automatic overwrite.
|
||||
|
||||
### M8-10 incident timeline and operator context
|
||||
|
||||
Incident detail presents correlated alerts and operator notes in a deterministic UTC timeline. Ownership and notes are auditable metadata only. The UI explicitly states that correlation is an evidence-backed indication rather than proven causality and that confidence describes the correlation rule, not certainty of root cause. External workflow linking is a disabled placeholder; Pulse remains observational and does not remediate infrastructure.
|
||||
|
||||
|
||||
### M8-11 default rule seeding
|
||||
|
||||
The implementation-owned embedded default bundle is schema-versioned and validated against the semantic metric catalog before startup reconciliation. Seeding is additive and idempotent: a missing default is inserted, while an existing rule ID—including a customized version—is preserved and counted as existing. Event-based restart-loop rules explicitly support `count` aggregation. Duplicate evaluation signals are collapsed before deterministic grouping; dependency causes remain inspectable while downstream notifications are suppressed.
|
||||
|
||||
## Default storage-pool rule — evaluability fix (2026-08-17)
|
||||
|
||||
The seeded default "Opslagpool bijna vol" originally bound `storage.pool.utilization`, whose query
|
||||
template requires a `{{pool}}` value; the alert evaluator issues one instant query per rule without a pool
|
||||
scope and does not fan out over `groupBy`, so the rule could never evaluate
|
||||
(`PROMQL_BINDING_VALUE_REQUIRED (pool)`). It now uses `storage.pool.utilization.maximum` (placeholder-free:
|
||||
the highest utilisation across all pools) so the default fires when any pool crosses the threshold; per-pool
|
||||
rules can still be created with a `poolId` scope. The seed refreshes implementation-owned defaults that are
|
||||
still at revision 1 (never edited by an operator) through the normal versioned update path
|
||||
(`alertdefaults.Seed` → `RuleUpdater`), and never touches edited rules. The `pulse_storage_pool_*` series must
|
||||
be provided by the environment — for Unraid via the recording rules in
|
||||
`deploy/prometheus.pulse-storage-pool.rules.yaml`.
|
||||
@@ -0,0 +1,371 @@
|
||||
# API contract
|
||||
|
||||
## 1. General rules
|
||||
|
||||
- Base path: `/api/v1`.
|
||||
- JSON request/response.
|
||||
- OpenAPI is generated/validated in CI.
|
||||
- UTC RFC3339 timestamps.
|
||||
- UUID resource IDs.
|
||||
- Cursor pagination for mutable/large collections.
|
||||
- `ETag` or explicit version for optimistic concurrency.
|
||||
- Problem details style errors with safe message, code and correlation ID.
|
||||
- Authentication required except health and OIDC bootstrap endpoints.
|
||||
- Authorization is server-side for every resource/action.
|
||||
- All list/query inputs are bounded.
|
||||
|
||||
## 2. Error shape
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "https://pulse.local/problems/query-limit",
|
||||
"title": "Query limit exceeded",
|
||||
"status": 422,
|
||||
"code": "QUERY_POINT_LIMIT",
|
||||
"detail": "Reduce the time range or increase the step.",
|
||||
"correlationId": "01...",
|
||||
"fields": {
|
||||
"range": "..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
No stack trace or secret-bearing upstream response.
|
||||
|
||||
## 3. Authoritative route inventory
|
||||
|
||||
`specs/api-routes.json` is the machine-readable source for this table. `python tools/check_api_contract.py` verifies that every router registration is represented here, every implementation file exists, and this table has neither missing nor invented method/path combinations. Paths outside `/api/v1` are limited to health, OIDC bootstrap, development-only mock login, and logout.
|
||||
|
||||
| Route | Access | Implementation |
|
||||
|---|---|---|
|
||||
| `GET /healthz` | public | `internal/service/health.go` |
|
||||
| `GET /readyz` | public | `internal/service/health.go` |
|
||||
| `GET /auth/login` | public-oidc | `internal/authapi/handler.go` |
|
||||
| `GET /auth/callback` | public-oidc | `internal/authapi/handler.go` |
|
||||
| `GET /auth/test-login` | development-only | `cmd/api/main.go` |
|
||||
| `POST /session/logout` | session | `cmd/api/main.go` |
|
||||
| `GET /api/v1/system/status` | view | `internal/systemstatusapi/handler.go` |
|
||||
| `GET /api/v1/system/diagnostics` | operate | `internal/systemstatusapi/handler.go` |
|
||||
| `GET /api/v1/system/metrics` | operate | `internal/observability/metrics.go` |
|
||||
| `GET /api/v1/system/backups` | admin | `internal/backupapi/handler.go` |
|
||||
| `POST /api/v1/system/backups` | admin | `internal/backupapi/handler.go` |
|
||||
| `GET /api/v1/onboarding` | view | `internal/onboardingapi/handler.go` |
|
||||
| `POST /api/v1/onboarding` | admin | `internal/onboardingapi/handler.go` |
|
||||
| `GET /api/v1/widgets/catalog` | view | `internal/widgetapi/handler.go` |
|
||||
| `POST /api/v1/widgets/preview` | edit | `internal/widgetapi/handler.go` |
|
||||
| `GET /api/v1/metrics/catalog` | view | `internal/metricsapi/handler.go` |
|
||||
| `POST /api/v1/metrics/query` | view | `internal/metricquery/handler.go` |
|
||||
| `POST /api/v1/metrics/query-range` | view | `internal/metricquery/handler.go` |
|
||||
| `POST /api/v1/metrics/inspect` | operate | `internal/metricquery/handler.go` |
|
||||
| `GET /api/v1/live` | view-websocket | `internal/live/live.go` |
|
||||
| `GET /api/v1/dashboards` | view | `internal/dashboardapi/handler.go` |
|
||||
| `POST /api/v1/dashboards` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `GET /api/v1/dashboards/{id}` | view | `internal/dashboardapi/handler.go` |
|
||||
| `PATCH /api/v1/dashboards/{id}` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `DELETE /api/v1/dashboards/{id}` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `PUT /api/v1/dashboards/{id}/document` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `POST /api/v1/dashboards/{id}/preview` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `POST /api/v1/dashboards/{id}/clone` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `GET /api/v1/dashboards/{id}/versions` | view | `internal/dashboardapi/handler.go` |
|
||||
| `GET /api/v1/dashboards/{id}/versions/{version}` | view | `internal/dashboardapi/handler.go` |
|
||||
| `POST /api/v1/dashboards/{id}/restore` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `POST /api/v1/dashboards/{id}/restore/{version}` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `GET /api/v1/entities` | view | `internal/inventoryapi/handler.go` |
|
||||
| `GET /api/v1/entities/{id}` | view | `internal/inventoryapi/handler.go` |
|
||||
| `GET /api/v1/entities/{id}/relations` | view | `internal/inventoryapi/handler.go` |
|
||||
| `GET /api/v1/host` | view | `internal/hostapi/handler.go` |
|
||||
| `GET /api/v1/processes` | view | `internal/processapi/handler.go` |
|
||||
| `GET /api/v1/containers` | view | `internal/containerapi/handler.go` |
|
||||
| `GET /api/v1/containers/{id}` | view | `internal/containerapi/handler.go` |
|
||||
| `GET /api/v1/applications` | view | `internal/applicationapi/handler.go` |
|
||||
| `GET /api/v1/events` | view | `internal/eventapi/handler.go` |
|
||||
| `GET /api/v1/applications/{id}` | view | `internal/applicationapi/handler.go` |
|
||||
| `GET /api/v1/array` | view | `internal/arrayapi/handler.go` |
|
||||
| `GET /api/v1/disks` | view | `internal/diskapi/handler.go` |
|
||||
| `GET /api/v1/disks/{id}` | view | `internal/diskapi/handler.go` |
|
||||
| `GET /api/v1/pools` | view | `internal/poolapi/handler.go` |
|
||||
| `GET /api/v1/pools/{id}` | view | `internal/poolapi/handler.go` |
|
||||
| `GET /api/v1/shares` | view | `internal/shareapi/handler.go` |
|
||||
| `GET /api/v1/shares/{id}` | view | `internal/shareapi/handler.go` |
|
||||
| `GET /api/v1/forecasts` | view | `internal/forecastapi/handler.go` |
|
||||
| `GET /api/v1/services` | view | `internal/serviceapi/handler.go` |
|
||||
| `GET /api/v1/services/{id}` | view | `internal/serviceapi/handler.go` |
|
||||
| `GET /api/v1/services/{id}/history` | view | `internal/serviceapi/handler.go` |
|
||||
| `GET /api/v1/services/{id}/dependencies` | view | `internal/serviceapi/handler.go` |
|
||||
| `GET /api/v1/topology` | view | `internal/serviceapi/handler.go` |
|
||||
| `GET /api/v1/network` | view | `internal/networkapi/handler.go` |
|
||||
| `GET /api/v1/reverse-proxy` | view | `internal/reverseproxyapi/handler.go` |
|
||||
| `GET /api/v1/alert-rules` | view | `internal/alertapi/handler.go` |
|
||||
| `POST /api/v1/alert-rules` | edit | `internal/alertapi/handler.go` |
|
||||
| `GET /api/v1/alert-rules/{id}` | view | `internal/alertapi/handler.go` |
|
||||
| `PUT /api/v1/alert-rules/{id}` | edit | `internal/alertapi/handler.go` |
|
||||
| `GET /api/v1/alert-rules/{id}/versions` | view | `internal/alertapi/handler.go` |
|
||||
| `POST /api/v1/alert-rules/{id}/test` | edit | `internal/alertapi/handler.go` |
|
||||
| `POST /api/v1/alert-rules/{id}/enable` | edit | `internal/alertapi/handler.go` |
|
||||
| `POST /api/v1/alert-rules/{id}/disable` | edit | `internal/alertapi/handler.go` |
|
||||
| `GET /api/v1/alerts` | view | `internal/alertopsapi/handler.go` |
|
||||
| `GET /api/v1/alerts/{id}` | view | `internal/alertopsapi/handler.go` |
|
||||
| `POST /api/v1/alerts/{id}/acknowledge` | operate | `internal/alertopsapi/handler.go` |
|
||||
| `POST /api/v1/alerts/{id}/unacknowledge` | operate | `internal/alertopsapi/handler.go` |
|
||||
| `GET /api/v1/alert-silences` | view | `internal/alertcontrolapi/handler.go` |
|
||||
| `POST /api/v1/alert-silences` | operate | `internal/alertcontrolapi/handler.go` |
|
||||
| `POST /api/v1/alert-silences/preview` | view | `internal/alertcontrolapi/handler.go` |
|
||||
| `POST /api/v1/alert-silences/{id}/revoke` | operate | `internal/alertcontrolapi/handler.go` |
|
||||
| `GET /api/v1/maintenance-windows` | view | `internal/alertcontrolapi/handler.go` |
|
||||
| `POST /api/v1/maintenance-windows` | operate | `internal/alertcontrolapi/handler.go` |
|
||||
| `POST /api/v1/maintenance-windows/preview` | view | `internal/alertcontrolapi/handler.go` |
|
||||
| `POST /api/v1/maintenance-windows/{id}/revoke` | operate | `internal/alertcontrolapi/handler.go` |
|
||||
| `GET /api/v1/incidents` | view | `internal/incidentapi/handler.go` |
|
||||
| `GET /api/v1/incidents/{id}` | view | `internal/incidentapi/handler.go` |
|
||||
| `PATCH /api/v1/incidents/{id}` | operate | `internal/incidentapi/handler.go` |
|
||||
| `GET /api/v1/incidents/{id}/notes` | view | `internal/incidentapi/handler.go` |
|
||||
| `POST /api/v1/incidents/{id}/notes` | operate | `internal/incidentapi/handler.go` |
|
||||
| `POST /api/v1/incidents/{id}/alerts/{alertId}` | operate | `internal/incidentapi/handler.go` |
|
||||
| `DELETE /api/v1/incidents/{id}/alerts/{alertId}` | operate | `internal/incidentapi/handler.go` |
|
||||
|
||||
## 4. Endpoint behavior
|
||||
|
||||
### Session/user
|
||||
|
||||
OIDC login and callback are the only public authentication routes. The browser session is server-side and intentionally has no introspection or preference endpoint in v1. Logout clears the opaque session cookie. Sessions use a bounded sliding idle lifetime: a successful authenticated request renews the HttpOnly cookie after half of the idle TTL has elapsed, while a separate absolute TTL always requires a new OIDC login. The defaults are eight hours idle and seven days absolute; production rejects an absolute TTL below 24 hours because an authenticated wallboard is a supported 24-hour workload. The mock test-login route returns `404` unless both the environment is non-production and mock authentication is configured.
|
||||
|
||||
### Health
|
||||
|
||||
Liveness is served at `/healthz`; dependency-aware readiness is served at `/readyz`. Detailed authenticated state, safe diagnostics, internal metrics and backup operations live below `/api/v1/system/*` with the permissions in the route inventory.
|
||||
|
||||
### Dashboards
|
||||
|
||||
Save accepts a complete validated dashboard document plus expected revision. Metadata update, archive, preview, clone, version reads and restore routes are listed above. Restore accepts the version in the path or request body and always creates a new current version. Import/export are UI-side validated document transfers in v1 and are not API routes. Partial widget edits may exist internally, but the user save is atomic.
|
||||
|
||||
### Widget catalog
|
||||
|
||||
The catalog is built from the same validated registry used by dashboard documents. Preview is edit-protected, bounded, does not persist data and applies the same widget validation as a saved document.
|
||||
|
||||
### Metrics
|
||||
|
||||
`POST /api/v1/metrics/inspect` accepts the same bounded semantic query request as the range endpoint, requires operate permission, and returns the resolved semantic metric, generated approved PromQL, estimated cost and applied series/point limits. It never executes the source query. The `inspector` field on a metric response is omitted for viewers and is safe/redacted for operators. Arbitrary label enumeration is deliberately absent in v1; callers use the allowlisted semantic catalog.
|
||||
|
||||
Semantic query example:
|
||||
|
||||
```json
|
||||
{
|
||||
"metric": "container.cpu.utilization",
|
||||
"scope": {"containerId": "uuid"},
|
||||
"range": {"from": "...", "to": "...", "stepSeconds": 15},
|
||||
"aggregation": "avg",
|
||||
"groupBy": ["container"],
|
||||
"maxSeries": 20,
|
||||
"maxPoints": 4000
|
||||
}
|
||||
```
|
||||
|
||||
### Host
|
||||
|
||||
GET /host — authenticated viewer, bounded current host snapshot
|
||||
|
||||
The response contains host identity, uptime/boot time, CPU aggregate and bounded per-core values, load averages, memory totals/percentages, bounded filesystems/inodes, network counters/errors/drops, time synchronization and source provenance. Optional hardware contains capability states, stable temperature/fan identities and bounded GPU values. Missing optional capabilities are disabled rather than errors. source.freshness=stale or unavailable telemetry makes status.state=unknown; the endpoint never returns fabricated healthy values.
|
||||
### Inventory
|
||||
|
||||
Entity list, detail and relation reads expose the persisted inventory. Discovery scheduling and datasource configuration remain worker/runtime configuration, not public API operations in v1. No endpoint mutates host/container state.
|
||||
|
||||
`GET /api/v1/entities` accepts bounded `limit` (1–100), opaque `after`, `q`, `type`, `status` and `order=asc|desc` parameters. The stable keyset is effective display name plus entity ID. Each summary exposes effective display/status values, fact/override/relation/source counts and stale-fact count; manual overrides therefore remain visible in both filtering and presentation.
|
||||
|
||||
`GET /api/v1/entities/{id}` returns aliases, all source-owned facts, overrides, effective values and incoming/outgoing relation peers. Effective values select a manual/system override before the best non-stale discovered fact while retaining every fact and its source, observation time, confidence and validity. Stale facts, tombstoned peers and absent relations are explicit states. `GET /api/v1/entities/{id}/relations` returns the same deterministic relation projection only. All three endpoints are authenticated, read-only and return safe problem details without database errors.
|
||||
|
||||
### Processes
|
||||
|
||||
GET /processes?limit=25&sort=cpu&q=...&container=...&after=... — authenticated viewer, bounded read-only process page
|
||||
|
||||
The response exposes only PID, normalized process name, state, runtime, CPU/memory counters and optional known container association. Command-line arguments, environment, working directory and control actions are not part of the contract. Supported sort modes are cpu and memory; bounded name/container filters are applied before the cursor so `total` and pagination remain truthful.
|
||||
### Containers
|
||||
|
||||
GET /containers?limit=25&q=...&state=...&health=...&sort=name&after=... — authenticated viewer, bounded read-only container page
|
||||
GET /containers/{id} — authenticated viewer, bounded read-only container detail
|
||||
|
||||
The response preserves runtime state separately from health, includes uptime, restart/exit counters, image and digest, resource counters, ports, volumes, networks, project and bounded labels where available. Search, runtime-state and health filters are applied before cursor pagination; supported deterministic sort modes are name, CPU, memory and state. intentionalStop is explicit and must not be inferred as healthy. Source freshness and provenance are retained; unavailable sources return an Unknown snapshot. There are no start, stop, restart, delete or exec endpoints.
|
||||
### Applications
|
||||
|
||||
GET /applications — authenticated viewer, bounded application snapshot
|
||||
GET /applications/{id} — authenticated viewer, one application with components and contributing reasons
|
||||
|
||||
Applications are derived from discovered components and persisted user overrides. Aggregation keeps critical and optional component roles explicit, treats a service-down component as unhealthy even when its container is running, and returns reason codes with component IDs. No application endpoint changes infrastructure state.
|
||||
### Events
|
||||
|
||||
There is no standalone event route yet. Bounded events are projected in their owning domains (for example service history, alert occurrences and incident notes). The contract does not advertise an unimplemented aggregate event store.
|
||||
|
||||
### Alerts
|
||||
|
||||
```text
|
||||
GET /alerts
|
||||
GET /alerts/{id}
|
||||
POST /alerts/{id}/acknowledge
|
||||
POST /alerts/{id}/unacknowledge
|
||||
POST /alerts/{id}/silence
|
||||
DELETE /alerts/{id}/silence
|
||||
|
||||
GET /alert-rules
|
||||
POST /alert-rules
|
||||
GET /alert-rules/{id}
|
||||
GET /alert-rules/{id}/versions
|
||||
PUT /alert-rules/{id}
|
||||
POST /alert-rules/{id}/test
|
||||
POST /alert-rules/{id}/enable
|
||||
POST /alert-rules/{id}/disable
|
||||
```
|
||||
|
||||
### Alert-rule contract (M8-01)
|
||||
|
||||
Alert-rule documents conform to `specs/alert-rule.schema.json`. The condition references only a semantic metric name from the server-side metric catalog; arbitrary PromQL, query templates, interpolation, and unbounded scope values are rejected. Reads require `view`; create, update, test, enable and disable require `edit`.
|
||||
|
||||
`PUT /alert-rules/{id}` requires the current revision in `If-Match` (or the `revision` query parameter). A stale revision returns `409 REVISION_CONFLICT`. Every accepted document change creates an immutable version; `GET /alert-rules/{id}/versions` returns versions newest first. Enabling and disabling increment the revision and create an audit event.
|
||||
|
||||
`POST /alert-rules/{id}/test` accepts an optional rule document and bounded sample value. It evaluates in memory and returns a preview only; it does not create versions, mutate enabled state, evaluate live sources, or write audit events. A rule condition may define a lower (for high-threshold rules) or higher (for low-threshold rules) `recoveryThreshold`; pending and resolve durations are evaluated at UTC observation timestamps. `cooldownSeconds` suppresses duplicate firing notifications after recovery without deleting occurrence history.
|
||||
|
||||
### Incidents
|
||||
|
||||
Incidents are created by the correlation pipeline in v1. The API exposes bounded list/detail, ownership update, note reads/writes and explicit alert association/disassociation. There is no manual incident-creation route.
|
||||
|
||||
### Probes/services
|
||||
|
||||
Service list, detail, history, dependency and topology routes are read-only. Probe configuration is owned by the worker/runtime boundary in v1: service detail may expose a bounded safe probe summary, but targets, credentials and mutation/test endpoints are deliberately absent. This prevents the observability API from becoming a general network-request primitive.
|
||||
|
||||
### Maintenance
|
||||
|
||||
Silences and maintenance windows support bounded list, create, preview and revision-checked revoke operations. Records are immutable evidence after creation; revoke replaces generic update/delete semantics.
|
||||
|
||||
### Operations
|
||||
|
||||
Administrative backup list/create is exposed at `/api/v1/system/backups`. Safe diagnostics and internal metrics use `/api/v1/system/diagnostics` and `/api/v1/system/metrics`. Restore, backup verification and general audit-log browsing remain CLI/operations procedures until their controlled workflows are implemented; they are not advertised API routes.
|
||||
|
||||
## 5. WebSocket
|
||||
|
||||
Endpoint:
|
||||
|
||||
```text
|
||||
GET /api/v1/live
|
||||
```
|
||||
|
||||
Requirements:
|
||||
- authenticated before upgrade;
|
||||
- same-origin/allowed-origin check;
|
||||
- role-aware subscription authorization;
|
||||
- message size/rate limits;
|
||||
- heartbeat and idle timeout;
|
||||
- per-session subscription/series limits;
|
||||
- sequence numbers and resync;
|
||||
- explicit unsubscribe;
|
||||
- no arbitrary backend URL/query.
|
||||
|
||||
Message schemas are in `specs/live-message.schema.json`.
|
||||
|
||||
## 6. Idempotency
|
||||
|
||||
Use idempotency keys for:
|
||||
- acknowledgement;
|
||||
- incident note/create where retries matter;
|
||||
- notification delivery;
|
||||
- backup start;
|
||||
- manual discovery.
|
||||
|
||||
## 7. Compatibility
|
||||
|
||||
- Breaking changes require `/api/v2` or a documented migration.
|
||||
- Schema versions are explicit in dashboard/export/live messages.
|
||||
- Old dashboard versions are migrated through tested deterministic migrations.
|
||||
|
||||
### Array and parity
|
||||
|
||||
`GET /array` is an authenticated viewer endpoint for one bounded, read-only array snapshot. The response contains source freshness/provenance, operational/degraded/missing/unknown state, parity presence/state/errors, bounded data/parity members, current check progress/speed/errors and bounded check history. Stale or unavailable storage source data is `Unknown`; there are no array start, stop, check or correct endpoints.
|
||||
### Disks
|
||||
|
||||
`GET /disks?limit=...` and `GET /disks/{id}` are authenticated viewer endpoints for bounded, read-only disk inventory and detail. Responses preserve source freshness/provenance and expose canonical stable source IDs, role, availability state, model, privacy-aware serial display, filesystem, size, used/free/utilization, separate `capacitySeverity` and `thermalSeverity`, and optional inode capacity. Missing-disk history remains visible; raw serials are never returned. Exact duplicate source observations are idempotently collapsed, while conflicting observations for one canonical ID fail closed. No format, mount, unmount, remove or repair endpoint exists.
|
||||
### SMART detail
|
||||
|
||||
Disk detail may include a bounded SMART object with capability state (`available` or `unknown`), generic overall result, freshness timestamp, explicit critical attributes/reasons and self-test result/age. Pending, reallocated, offline-uncorrectable, CRC and wear signals remain separate attributes; a generic `passed` result does not suppress an attention reason. SMART has no write or self-test trigger endpoint.
|
||||
### Disk telemetry
|
||||
|
||||
Disk detail may include bounded performance samples (read/write bytes per second, IOPS and latency), temperature state/value and optional spin capability. Unsupported latency or spin is explicit rather than fabricated; history is bounded and deterministically ordered. Temperature status applies warning/critical/recovery hysteresis, while live/history values remain read-only.
|
||||
|
||||
### Pools
|
||||
|
||||
`GET /pools?limit=...` and `GET /pools/{id}` are authenticated viewer endpoints for bounded, read-only cache/Btrfs/ZFS pool snapshots and detail. Responses preserve source freshness/provenance and expose filesystem, device-health state, usable/used/free capacity, separate `capacitySeverity`, profile/redundancy, deduplicated bounded members, filesystem errors, scrub state/history and explicit capability states. Capacity pressure never rewrites device health. Stale pool data maps every pool state and severity to `Unknown`; no scrub, repair, balance, mount or pool-control endpoint exists.
|
||||
### Shares
|
||||
|
||||
`GET /shares?limit=...` and `GET /shares/{id}` are authenticated viewer endpoints for bounded, read-only share usage and growth. Responses expose configured storage policy, source-owned size observations, cache/pool placements, bounded growth history and scan-plan cost metadata. Size scans are rate-limited/cached through the normalized scan plan; stale or unavailable observations are `Unknown`. No path, filename, directory listing or file-content field is returned.
|
||||
|
||||
### Storage map and heatmap views
|
||||
|
||||
The storage view composes authenticated read-only array, pool and disk snapshots in the browser. Array membership and disk telemetry with the same canonical physical ID form one disk node; a pool aggregate remains a separate meaningful role. Every node states availability/device health, capacity and temperature independently, while the strongest trustworthy severity determines its visual accent. Map nodes retain entity links and text/icon state; the temperature heatmap is bounded and always includes a table/text alternative. No new control endpoint or raw source credential is exposed.
|
||||
### Capacity forecasts
|
||||
|
||||
`GET /api/v1/forecasts` is an authenticated, read-only endpoint. Each bounded assessment exposes the entity identity, method, historical window, point count, confidence and reason. `qualifiedCount` counts only medium/high forecasts and is intentionally distinct from `items.length`, which may include insufficient or stale assessments. A projected date is only returned for qualified (`medium` or `high`) median-rate forecasts; disabled policy, insufficient data, bulk-import detection, irregular intervals and unknown/reached capacity remain explicit states without a false date. When no real capacity entity exists, `items` is empty and the top-level `reason` explains the Unknown state; the API never synthesizes a blank `none`/0-byte entity. The response contains no write or capacity-management operation.
|
||||
### M7 service and probe model
|
||||
|
||||
Service and probe configuration is versioned and archive-oriented. Services, endpoints and probes expose revisions for optimistic concurrency; active names are unique per parent while archived configurations remain available for historical result retention. Probe results and certificate observations are append-only by probe/service and UTC observation time. Service dependencies retain source and confidence, and service permissions map existing Pulse roles to view/operate/edit/admin without storing probe credentials.
|
||||
### M7 probe target safety
|
||||
|
||||
Probe target policy is evaluated server-side before every request and again for redirects. HTTP/HTTPS GET/HEAD are the default methods; request headers, response headers/body, timeout and redirect count are bounded. Loopback, link-local, metadata, multicast and unspecified addresses are blocked, and private/LAN targets require an explicit CIDR allowlist. DNS answers are revalidated at dial time so a later blocked answer cannot be used for DNS rebinding. Policy changes emit audit events with redacted bounded policy metadata.
|
||||
### M7 probe execution
|
||||
|
||||
Probe execution supports HTTP/HTTPS, TCP, DNS and TLS within the server-side network policy. HTTP status and bounded keyword/JSON assertions are represented as `up`, `down`, `degraded` or `unknown`; redirect following is opt-in and each hop is revalidated. TLS results retain bounded expiry, issuer, subject and hostname-validity facts. ICMP is capability-aware and returns `Unknown` with an `unsupported` class when the runtime cannot provide it. Probe configuration may contain only a secret reference; credentials are never returned by the API or included in result errors.
|
||||
### M7 service status and history
|
||||
|
||||
`GET /api/v1/services?limit=...` and `GET /api/v1/services/{id}` are authenticated, read-only service projections. `GET /api/v1/services/{id}/history?limit=...` returns bounded probe history. The projection derives service state from probe results independently of container state, preserves last success/failure and latency, calculates bounded availability, and maps absent or stale probe data to `unknown`. Results, histories and transition events are deterministically ordered and capped. Service detail may include a bounded, read-only probe configuration summary (id, name, type, interval, timeout, enabled state and safe TLS/redirect flags); target payloads, content assertions, secret references and credentials are never returned.
|
||||
|
||||
Service snapshots expose separate `capabilityState` (`available`, `unavailable`, `unsupported`) and `configurationState` (`configured`, `not_configured`, `unknown`). A catalog without services is `available/not_configured`; an unreadable repository is `unavailable/unknown`. Per-service reasons distinguish no configured probe, disabled probes, pending results and stale results. These states are also propagated into topology and each network-health scope so an empty configuration is never presented as a generic source failure.
|
||||
|
||||
|
||||
`GET /api/v1/services/{id}/dependencies?limit=...` is an authenticated, read-only bounded relation projection. Dependencies are sorted deterministically and expose source, confidence and confirmed/inferred state; dependency writes remain server-side and audit-backed.
|
||||
|
||||
`GET /api/v1/topology?limit=...` is an authenticated, read-only graph projection bounded to at most 100 nodes and 200 edges by default. Nodes expose service status and whether they are present in the current snapshot; missing nodes remain explicit `unknown`. Edges combine active service dependencies with active `depends_on`, `backs` and `exposes` inventory relations when both inventory entities map to active services. Every edge preserves relation type, source, confidence and confirmed/inferred state, is sorted deterministically, and never asserts causality. The endpoint has no write operation.
|
||||
|
||||
`GET /api/v1/network` is an authenticated, read-only network health projection. It separates internal interface health, gateway, DNS and internet scopes; stale/unavailable host telemetry maps interfaces and internal health to `unknown`, while independent internet or DNS failures remain independently visible. Interface RX/TX bytes, errors and drops are bounded and certificates are exposed as safe observed facts without probe credentials.
|
||||
|
||||
GET /api/v1/reverse-proxy is an authenticated, read-only projection of optional reverse-proxy host mappings. The connector reports a versioned capability and source ownership, returns only bounded host/URL-to-service mappings, and exposes a disabled or unavailable state when no connector is configured. Connector credentials remain external references and are never serialized. Duplicate host mappings with different targets are rejected unless a user-confirmed override explicitly selects the effective service; no proxy configuration mutation endpoint exists. Topology may represent enabled mappings as reverse_proxy nodes with exposes edges that retain source and confirmed/inferred provenance.
|
||||
|
||||
### Alert silences and maintenance windows (M8-06)
|
||||
|
||||
~~~text
|
||||
GET /alert-silences
|
||||
POST /alert-silences
|
||||
POST /alert-silences/preview
|
||||
POST /alert-silences/{id}/revoke
|
||||
GET /maintenance-windows
|
||||
POST /maintenance-windows
|
||||
POST /maintenance-windows/preview
|
||||
POST /maintenance-windows/{id}/revoke
|
||||
~~~
|
||||
|
||||
Reads and matcher previews require view; create and revoke require operate. Silences and maintenance windows require a non-empty reason, bounded matcher and selector fields, and an expiry or end time. Revocation uses If-Match or the revision query parameter. Controls never delete alert history. Every create and revoke is audited; expiry is idempotently marked by the API expiry job.
|
||||
|
||||
|
||||
### Alert operations (M8-07)
|
||||
|
||||
GET /alerts returns a bounded deterministic list of non-inactive alert instances; the state query filter is allow-listed. GET /alerts/{id} returns rule/entity metadata, current state, revision and bounded occurrence history.
|
||||
|
||||
POST /alerts/{id}/acknowledge and POST /alerts/{id}/unacknowledge require operate permission, If-Match or a positive revision query parameter, and a bounded Idempotency-Key or evaluationKey. A replay with the same key is idempotent even when the original revision is stale. Acknowledge changes firing or pending to acknowledged; unacknowledge changes acknowledged back to firing. Resolved state remains a persisted evaluator result and is not rewritten by a read or duplicate operation. Accepted and idempotent operations are audited.
|
||||
|
||||
### Notification delivery contract
|
||||
|
||||
Notification delivery is an internal worker contract in M8-08. Public alert operations create no direct channel side effect: an event is first represented by a bounded outbox record with an idempotency key. Workers claim and complete records transactionally, and delivery history remains queryable for audit. Channel test actions are explicitly bounded and rate limited; no secret value is returned by the channel or delivery model.
|
||||
### M8-09 incident contract
|
||||
|
||||
`GET /incidents` and `GET /incidents/{id}` are bounded viewer operations. `POST /incidents/{id}/alerts/{alertId}` and `DELETE /incidents/{id}/alerts/{alertId}` require operate permission and create audit events. Manual association requests require bounded rationale and confidence; the response exposes whether the association was idempotent. Incident operations remain observational and do not remediate alerts, services, or infrastructure.
|
||||
|
||||
### M8-10 incident detail, ownership and notes
|
||||
|
||||
`GET /incidents/{id}` returns bounded incident detail, associations and ordered notes. `PATCH /incidents/{id}` requires operate permission and updates only the owner user ID with the current positive revision and returns a conflict for stale writes. `GET /incidents/{id}/notes` is viewer-readable; `POST /incidents/{id}/notes` requires operate permission and stores the authenticated actor as author. Note bodies are bounded plain text: tag-shaped markup is removed, whitespace is normalized, and empty or oversized values are rejected. Incident UI communicates correlation confidence and rationale as uncertainty-aware evidence; Pulse provides no remediation action.
|
||||
|
||||
## Onboarding
|
||||
|
||||
`GET /api/v1/onboarding` is an authenticated viewer endpoint. It returns only bounded readiness states for the database, Prometheus, read-only Unraid source, authentication, alert defaults and the default dashboard. It never returns datasource URLs, API tokens, OIDC client secrets or other secret-bearing configuration.
|
||||
|
||||
`POST /api/v1/onboarding` requires the administrator permission and accepts only `dashboard` and `rules` choices with values `default` or `skip`. Completion is additive and resumable: it records progress in `system_settings`, installs the validated default dashboard only when the stable slug is absent, and preserves existing dashboards and alert rules. It exposes no host, Docker, storage, database-control or remediation operation.
|
||||
|
||||
## Authentication bootstrap
|
||||
|
||||
`GET /auth/login` and `GET /auth/callback` are the only unauthenticated non-health endpoints. They are browser redirect endpoints outside `/api/v1`; they return `302` rather than JSON, except for a problem-details `405` on a non-`GET` method.
|
||||
|
||||
`GET /auth/login` starts the Authentik OIDC authorization code flow with state, nonce and PKCE `S256`. State, nonce, PKCE verifier and the requested post-login path are stored server-side in a bounded, TTL-expired flow store; the browser receives only an opaque flow identifier in a short-lived `HttpOnly`, `SameSite=Lax` cookie that is `Secure` in production. The optional `redirect` query parameter is accepted only as an in-app path with a single leading slash: protocol-relative, absolute, backslash-prefixed, control-character and oversized values fall back to `/`. The response redirects to the provider authorization URL.
|
||||
|
||||
`GET /auth/callback` consumes the flow identifier exactly once, deleting the stored flow before validating anything. It compares state in constant time, exchanges the code with the PKCE verifier, verifies the ID token issuer, audience, signature and nonce, maps the configured role claim to a Pulse role and only then issues the session cookie and redirects to the validated post-login path. An unknown, replayed or expired flow, a state or nonce mismatch, a provider `error` response, a failed exchange, a missing ID token, an unmapped role and a failed audit or session write all issue no session and redirect to a fixed in-app error route with one of a closed set of reason codes (`invalid_request`, `expired`, `denied`, `provider_unavailable`, `not_authorized`, `unavailable`). Provider-supplied error text, authorization codes, tokens and PKCE verifiers are never reflected into a response or logged.
|
||||
@@ -0,0 +1,371 @@
|
||||
# Data model
|
||||
|
||||
## 1. Principles
|
||||
|
||||
- Stable internal UUIDs; external IDs are source-scoped aliases.
|
||||
- Source facts and user overrides are stored separately.
|
||||
- Soft deletion/tombstones preserve history and reconciliation.
|
||||
- Important configuration is versioned.
|
||||
- Alert state changes are transactional.
|
||||
- Time is stored in UTC; locale/time zone applied at presentation.
|
||||
- JSONB is used for bounded extensible attributes, not as a substitute for core relational structure.
|
||||
|
||||
## 2. Identity and access
|
||||
|
||||
### `users`
|
||||
- id
|
||||
- external_subject
|
||||
- display_name
|
||||
- email
|
||||
- status
|
||||
- created_at / updated_at / last_login_at
|
||||
|
||||
### `roles`, `user_roles`
|
||||
Roles: viewer, operator, editor, administrator.
|
||||
|
||||
### `sessions` or external session metadata
|
||||
Only if required by the chosen auth implementation. Avoid storing OIDC tokens unnecessarily.
|
||||
|
||||
## 3. Datasources
|
||||
|
||||
### `data_sources`
|
||||
- id
|
||||
- type
|
||||
- name
|
||||
- enabled
|
||||
- configuration reference
|
||||
- capability document
|
||||
- health_state
|
||||
- last_success_at
|
||||
- last_error_code/message_redacted
|
||||
- freshness policy
|
||||
- created/updated
|
||||
|
||||
### `collectors`
|
||||
- id
|
||||
- datasource_id
|
||||
- kind
|
||||
- version
|
||||
- heartbeat
|
||||
- capabilities
|
||||
- status
|
||||
|
||||
## 4. Inventory
|
||||
|
||||
### `entities`
|
||||
- id UUID
|
||||
- entity_type
|
||||
- canonical_name
|
||||
- display_name
|
||||
- status
|
||||
- status_reasons JSONB
|
||||
- first_seen_at
|
||||
- last_seen_at
|
||||
- tombstoned_at
|
||||
- attributes JSONB
|
||||
|
||||
### `entity_aliases`
|
||||
- entity_id
|
||||
- source_id
|
||||
- external_type
|
||||
- external_id
|
||||
- unique(source_id, external_type, external_id)
|
||||
|
||||
### Container identity and recreation
|
||||
|
||||
Container runtime IDs are source-scoped aliases, not logical identity by themselves. A live runtime ID always matches its exact source/runtime alias. When a runtime ID changes, the same logical entity may be reused only when both the source-scoped compose project and compose service are present and uniquely identify one prior active alias. A container name, image, or digest alone is insufficient evidence and creates a new logical entity. The old runtime alias remains in history as inactive, the new alias points to the same logical entity, and recreation events keep that logical entity as events.entity_id while retaining previous/current runtime IDs in bounded attributes.
|
||||
### `entity_facts`
|
||||
- entity_id
|
||||
- field_name
|
||||
- source_id
|
||||
- value JSONB
|
||||
- observed_at
|
||||
- confidence
|
||||
- valid_until
|
||||
|
||||
### `entity_overrides`
|
||||
- entity_id
|
||||
- field_name
|
||||
- value JSONB
|
||||
- user_id
|
||||
- updated_at
|
||||
|
||||
### `entity_relations`
|
||||
- id
|
||||
- source_entity_id
|
||||
- relation_type
|
||||
- target_entity_id
|
||||
- source_id
|
||||
- confidence
|
||||
- confirmed
|
||||
- first_seen/last_seen/tombstoned
|
||||
|
||||
### applications
|
||||
|
||||
- logical application ID and display name
|
||||
- discovered component references
|
||||
- critical/optional component policy
|
||||
- aggregate status and bounded reason list
|
||||
- user override metadata kept separate from discovery grouping
|
||||
## 5. Metrics catalog
|
||||
|
||||
### `metric_definitions`
|
||||
- id
|
||||
- semantic_name unique
|
||||
- version
|
||||
- description
|
||||
- unit
|
||||
- value_kind
|
||||
- source_kind
|
||||
- query_template
|
||||
- label contract
|
||||
- limits
|
||||
- allowed visualizations
|
||||
- default transformations
|
||||
- status policy reference
|
||||
- enabled
|
||||
|
||||
### `metric_bindings`
|
||||
Maps semantic metric to datasource/exporter/capability variants.
|
||||
|
||||
Metric samples remain in Prometheus.
|
||||
|
||||
## 6. Dashboards
|
||||
|
||||
### `dashboards`
|
||||
- id
|
||||
- slug
|
||||
- name
|
||||
- description
|
||||
- owner_user_id nullable
|
||||
- scope personal/shared/system
|
||||
- archived_at
|
||||
- current_version_id
|
||||
- revision (optimistic concurrency token)
|
||||
- created/updated
|
||||
|
||||
### `dashboard_versions`
|
||||
- id
|
||||
- dashboard_id
|
||||
- version_number
|
||||
- schema_version
|
||||
- document JSONB
|
||||
- change_summary
|
||||
- created_by
|
||||
- created_at
|
||||
|
||||
Dashboard versions are immutable. Dashboard writes lock the current row, compare revision, and commit the new version/current pointer atomically. The API stores only bounded revision metadata in audit diffs; full dashboard documents are not copied into audit events.
|
||||
|
||||
A dashboard version contains variables, widget instances, behavior and layout documents. Save as one atomic version to avoid partial layout/config updates.
|
||||
|
||||
Optional normalized indexes may extract widget type/entity references for search/impact analysis.
|
||||
|
||||
## 7. Events
|
||||
|
||||
### `events`
|
||||
- id
|
||||
- event_type
|
||||
- severity
|
||||
- entity_id nullable
|
||||
- source_id
|
||||
- occurred_at
|
||||
- received_at
|
||||
- dedup_key
|
||||
- summary
|
||||
- attributes JSONB bounded/redacted
|
||||
- correlation_id
|
||||
- unique(source_id, dedup_key, occurred_at bucket) as appropriate
|
||||
|
||||
## 8. Alerts
|
||||
|
||||
### `alert_rules`
|
||||
- id
|
||||
- name
|
||||
- enabled
|
||||
- severity
|
||||
- evaluator type/config
|
||||
- scope selector
|
||||
- pending duration
|
||||
- resolve duration
|
||||
- cooldown duration for repeated firing notifications
|
||||
- hysteresis config
|
||||
- grouping labels
|
||||
- suppression/dependency policy
|
||||
- current_version_id
|
||||
- revision (optimistic concurrency token)
|
||||
- created/updated
|
||||
|
||||
### `alert_rule_versions`
|
||||
Immutable rule documents and audit metadata.
|
||||
|
||||
### `alert_instances`
|
||||
Stable entity/rule combination:
|
||||
- id
|
||||
- rule_id
|
||||
- fingerprint
|
||||
- entity_id
|
||||
- current_state
|
||||
- active_since
|
||||
- last_evaluated_at
|
||||
- last_value
|
||||
- reason
|
||||
- acknowledged_by/at
|
||||
- cooldown_until
|
||||
- silenced_until
|
||||
- version for optimistic concurrency
|
||||
|
||||
### `alert_occurrences`
|
||||
Immutable transitions/evaluation outcomes relevant to history.
|
||||
|
||||
## 9. Incidents
|
||||
|
||||
### `incidents`
|
||||
- id
|
||||
- title
|
||||
- summary
|
||||
- severity
|
||||
- status
|
||||
- started_at
|
||||
- resolved_at
|
||||
- owner_user_id
|
||||
- correlation_method
|
||||
- confidence
|
||||
- created/updated
|
||||
- version
|
||||
|
||||
### `incident_alerts`, `incident_entities`
|
||||
Many-to-many links with rationale.
|
||||
|
||||
### `incident_notes`
|
||||
- incident_id
|
||||
- author
|
||||
- body sanitized
|
||||
- created_at
|
||||
|
||||
## 10. Maintenance and notifications
|
||||
|
||||
### `maintenance_windows`
|
||||
- id
|
||||
- name
|
||||
- selector
|
||||
- start/end or recurrence
|
||||
- suppress notifications/state policy
|
||||
- creator/audit
|
||||
|
||||
### `notification_channels`
|
||||
Encrypted secret references and non-secret config.
|
||||
|
||||
### `notification_deliveries`
|
||||
- occurrence/incident
|
||||
- channel
|
||||
- status
|
||||
- attempts
|
||||
- last_error_redacted
|
||||
- timestamps
|
||||
- idempotency_key
|
||||
|
||||
## 11. Audit
|
||||
|
||||
### `audit_events`
|
||||
- actor/user/service
|
||||
- action
|
||||
- resource type/id
|
||||
- result
|
||||
- occurred_at
|
||||
- correlation_id
|
||||
- source IP/session metadata where appropriate
|
||||
- before/after bounded diff with secret fields excluded
|
||||
|
||||
## 12. Operations
|
||||
|
||||
### `job_runs`
|
||||
- job type/key
|
||||
- scheduled/started/completed
|
||||
- status
|
||||
- counts
|
||||
- error code
|
||||
- correlation ID
|
||||
- lease owner and lease expiry for coordinated worker execution
|
||||
|
||||
### `schema_migrations`
|
||||
Managed by migration tool.
|
||||
|
||||
### `system_settings`
|
||||
Typed/versioned settings; no plaintext secrets.
|
||||
|
||||
## 13. Concurrency
|
||||
|
||||
Use optimistic concurrency for:
|
||||
- dashboard save;
|
||||
- alert acknowledgement/state;
|
||||
- incident edit;
|
||||
- settings.
|
||||
|
||||
Return a conflict response with current version rather than silently overwriting.
|
||||
|
||||
## 14. Retention
|
||||
|
||||
Implement partitioning/cleanup when measurements justify it.
|
||||
|
||||
Baseline:
|
||||
- inventory history/tombstones: enough for reconciliation and events;
|
||||
- events: 1 year configurable;
|
||||
- alert occurrences: 2 years configurable;
|
||||
- incidents: retained until explicit policy;
|
||||
- audit: at least 1 year configurable;
|
||||
- job runs: shorter operational retention;
|
||||
- dashboard versions: at least 180 days or fixed count plus protected versions.
|
||||
|
||||
### arrays and parity checks
|
||||
|
||||
An array snapshot preserves source-scoped provenance and UTC observed/received timestamps. Array members retain stable source IDs, role, state and bounded capacity/I/O counters. Parity state is separate from array state, and current/history parity checks retain progress, byte-per-second speed, error count and timestamps. Missing, disabled or emulated members are represented explicitly; stale source data maps to Unknown. Array transition events retain the array entity ID and bounded state/error attributes without exposing operational controls.
|
||||
### disks and capacity
|
||||
|
||||
Disk identity is source-scoped and stable from the provider ID. A disk retains role, state, model, privacy-aware serial display, filesystem and bounded size/used/free/inode metrics. Used values above capacity are rejected; free space and utilization are derived from validated bytes, and inode utilization is derived separately. Missing disks remain in current/history projections instead of being dropped, while raw serials are excluded from API output.
|
||||
### SMART facts
|
||||
|
||||
SMART is an optional disk capability with its own observed timestamp and freshness state. Normalization preserves generic overall result, bounded mapped attributes, critical/reason status and self-test result/age. Unavailable or stale SMART is Unknown and never Healthy; critical pending, reallocated, offline-uncorrectable, CRC or wear attributes remain visible even when the vendor overall result is passed.
|
||||
### disk telemetry facts
|
||||
|
||||
Disk performance history is bounded and timestamped for live/history consumers. Temperature observations retain source time and policy-derived status; recovery does not clear attention until the configured recovery threshold is crossed. Latency/spin capabilities explicitly report unsupported when the source cannot provide them.
|
||||
|
||||
### pools and scrub facts
|
||||
|
||||
Pool identity is stable within its source and retains filesystem, usable/used/free bytes, profile/redundancy, bounded member state and filesystem error facts. Btrfs and ZFS capabilities are explicit and conditional; unsupported fields are never fabricated. Scrub current state and bounded history retain progress, bytes checked, errors and UTC timestamps. Stale source data maps pool state to Unknown while preserving provenance. Transition events identify degraded/faulted/recovered pools and scrub failures without exposing controls.
|
||||
### shares and growth facts
|
||||
|
||||
Share identity is source-scoped and retains configured allocation/cache policy, source-owned used-size timestamp/state, bounded placement by pool and chronological growth points. Growth deltas and daily rates are derived from UTC observations, including negative changes, without exposing paths or file content. `ScanPlan` caps due size refreshes per run and records deferred work plus cache TTL so expensive scans remain bounded and observable.
|
||||
|
||||
### storage visualization projections
|
||||
|
||||
Storage map nodes are bounded projections of array members, pools and disks with entity links, kind, explicit state and detail text. Temperature heatmap points retain disk identity, UTC observed time, value and text status; visual cells never replace the accessible table alternative.
|
||||
### capacity forecasts
|
||||
|
||||
A forecast is a bounded projection over UTC usage observations. Its policy records enabled state, maximum window, minimum points and method. Median daily growth is used for linear projection; bulk-import/outlier and irregular-interval safeguards lower confidence and suppress `daysToCapacity`/`projectedAt`. Method, window, point count, confidence and reason are always retained so the UI cannot present an unexplained precise date.
|
||||
|
||||
`capacity_samples` preserves bounded historical capacity observations for shares, pools and disks without storing file contents. Agent snapshot writes and their samples commit in one PostgreSQL transaction. A six-hour UTC bucket and primary key on `(entity_kind, entity_id, source_id, sampled_at)` make repeated discovery runs idempotent; a delayed retry may not replace a newer observation in the same bucket. The history index `(entity_kind, entity_id, sampled_at DESC)` supports the actual bounded forecast query. Forecast reads use at most 512 points inside the configured window and combine persisted share usage with the current canonical pool capacity. Insufficient, stale or unavailable histories remain assessments with `confidence=none`; only medium/high results count as qualified forecasts.
|
||||
### services and probes
|
||||
|
||||
`services`, `service_endpoints` and `probes` represent reachable capabilities independently of container state. Configuration rows use `revision`, `updated_at` and `archived_at`; active names are unique per parent, while archived rows remain to preserve history. `probe_results` stores immutable timestamped outcomes, and `service_certificates` stores observed certificate facts. `service_dependencies` preserves source, confidence and confirmation state. `service_permissions` maps database roles to service permissions; secret references are identifiers only and never plaintext credentials.
|
||||
### service status projections
|
||||
|
||||
Service status is derived from immutable `probe_results`, not from container state. The bounded projection retains current state/reason, last result/success/failure timestamps, latest latency, availability sample counts and bounded history. A result older than the configured freshness policy becomes `unknown`; it is never silently treated as healthy. State transitions produce deterministic service events with service identity, from/to state and reason.
|
||||
M8-06 persistence adds alert_silences and maintenance_windows with bounded reason and name fields, JSON matcher or selector, UTC start and end timestamps, explicit active/expired/revoked status, creator and owner provenance, revocation and expiry timestamps, revision, expiry indexes, and deterministic listing indexes. These tables are additive and do not alter M1 tables or alert occurrence history.
|
||||
|
||||
M8-07 extends alert_occurrences with an immutable unacknowledge event type and adds an acknowledged-state index. Alert list/detail projections join rule and optional entity metadata while preserving instance revision and occurrence history.
|
||||
|
||||
### M8-08 notification persistence
|
||||
|
||||
- `notification_channels` stores only a bounded non-secret configuration and a reference to an external secret provider; the secret value is never part of the domain object or query result.
|
||||
- `notification_outbox` has a unique idempotency key, bounded event content, a lease-aware status, bounded attempts and deterministic due ordering.
|
||||
- `notification_deliveries` records each claimed attempt with a unique `(outbox_id, attempt)` key. Claim, delivery audit creation, completion and retry state changes are transactionally coordinated.
|
||||
- Channel updates use optimistic revisions. Outbox rows retain recovery events and are not deleted after successful delivery, preserving auditability.
|
||||
### M8-09 incident persistence
|
||||
|
||||
- `incidents` stores a bounded correlation key, title/summary, derived severity, lifecycle status, start/resolution timestamps, rationale method, confidence, owner and optimistic revision. Only one unresolved incident may exist for a correlation key.
|
||||
- `incident_alerts` preserves per-alert rationale/confidence and distinguishes deterministic correlation from manual association. A manual association is never overwritten by a later correlation upsert.
|
||||
- `incident_entities` keeps the affected entity set with bounded rationale/confidence and restricts entity deletion while incident evidence references it.
|
||||
|
||||
### M8-10 incident notes and ownership
|
||||
|
||||
`incident_notes` is bounded operator context attached to an incident. Bodies are normalized to plain text, stripped of tag-shaped markup and capped at 2,000 characters before persistence; notes are ordered by UTC creation time and stable ID. Notes cascade with their incident and never affect alert/entity evidence. Incident ownership is metadata updated with an optimistic revision, so stale UI writes become an explicit conflict.
|
||||
@@ -0,0 +1,248 @@
|
||||
# Security threat model
|
||||
|
||||
## 1. Assets
|
||||
|
||||
- Unraid host and storage.
|
||||
- Docker/container metadata and internal topology.
|
||||
- Prometheus metrics and labels.
|
||||
- Service URLs and availability data.
|
||||
- OIDC identities/roles.
|
||||
- Pulse configuration, alerts, incidents and audit.
|
||||
- Notification/probe credentials.
|
||||
- Database backups.
|
||||
- Server access path used by Codex during deployment.
|
||||
|
||||
## 2. Trust boundaries
|
||||
|
||||
- Browser to Pulse.
|
||||
- Pulse to Authentik.
|
||||
- Pulse to PostgreSQL.
|
||||
- Pulse to Prometheus.
|
||||
- Pulse agent to host/Unraid/Docker.
|
||||
- Probe worker to network targets.
|
||||
- Notification worker to external channels.
|
||||
- Codex workspace to production server.
|
||||
|
||||
## 3. Primary threats and controls
|
||||
|
||||
### M0 discovered deployment posture
|
||||
|
||||
The target host already runs Nginx Proxy Manager, Authentik, Grafana, and multiple Docker/Compose projects. Pulse treats all of them as external protected resources. Existing containers with broad privileges, including any Docker socket access, are not reused as a Pulse pattern. Pulse must be isolated on dedicated resources, integrate with the proxy and OIDC provider additively, and remain server-side for all Unraid/Prometheus access.
|
||||
|
||||
M0 did not identify a local Prometheus service; this is an explicit datasource uncertainty, not permission to substitute an unreviewed source. Missing or stale telemetry must map to `Unknown`. No hostname, port, network, volume, or path is trusted until the deployment task rechecks ownership and conflicts.
|
||||
|
||||
### Unrestricted Docker/host control
|
||||
|
||||
Threat: compromise of web/API leads to host root-equivalent access.
|
||||
|
||||
Controls:
|
||||
- no unrestricted socket in web/API;
|
||||
- separate agent/proxy;
|
||||
- endpoint allowlist;
|
||||
- read-only capability contract;
|
||||
- non-root API;
|
||||
- network separation;
|
||||
- architecture test of compose/mounts;
|
||||
- no mutation API in v1.
|
||||
|
||||
### SSRF from service probes
|
||||
|
||||
Threat: user config probes metadata, loopback, admin services or redirects.
|
||||
|
||||
Controls:
|
||||
- role restriction;
|
||||
- scheme/port allowlist;
|
||||
- DNS resolution validation before and after redirect;
|
||||
- block metadata/link-local/unspecified by default;
|
||||
- configurable LAN allowlist;
|
||||
- response size/time limits;
|
||||
- no arbitrary methods/body;
|
||||
- redacted logging;
|
||||
- tests for DNS rebinding/redirect escape.
|
||||
|
||||
### Query abuse
|
||||
|
||||
Threat: expensive or injection-like Prometheus queries cause outage or expose labels.
|
||||
|
||||
Controls:
|
||||
- semantic query templates;
|
||||
- bounded scope/range/series/points;
|
||||
- server-side parameterization/escaping;
|
||||
- timeout/concurrency/rate limit;
|
||||
- advanced raw query separate permission;
|
||||
- audit and query cost metrics.
|
||||
|
||||
|
||||
### Optional hardware capabilities
|
||||
|
||||
Hardware sensors and GPU telemetry are optional read-only capabilities. The adapter accepts only normalized bounded snapshots from an approved source; absent support is disabled and unsupported support remains inspectable without being treated as a host failure. API/web never gains device, mount, namespace or Docker-socket access for these values.
|
||||
### Authentication/authorization bypass
|
||||
|
||||
Controls:
|
||||
- standards-based OIDC validation;
|
||||
- issuer/audience/nonce/state/PKCE;
|
||||
- secure cookies;
|
||||
- server-side RBAC;
|
||||
- WebSocket auth/origin/subscription auth;
|
||||
- CSRF protection where cookies are used;
|
||||
- role matrix tests;
|
||||
- break-glass disabled by default.
|
||||
|
||||
### XSS and dashboard import
|
||||
|
||||
Controls:
|
||||
- no arbitrary HTML/JS widgets;
|
||||
- sanitize Markdown;
|
||||
- schema validation;
|
||||
- safe chart labels/tooltips;
|
||||
- CSP;
|
||||
- escaped event/upstream text;
|
||||
- import size and complexity limits.
|
||||
|
||||
### Secret leakage
|
||||
|
||||
Controls:
|
||||
- external secret injection;
|
||||
- encrypted-at-rest channel/probe references;
|
||||
- redaction middleware;
|
||||
- no env dumps;
|
||||
- evidence policy;
|
||||
- secret scan;
|
||||
- diagnostic bundle allowlist;
|
||||
- backups exclude plaintext or are encrypted/secured.
|
||||
|
||||
### Supply chain
|
||||
|
||||
Controls:
|
||||
- lockfiles;
|
||||
- minimal maintained dependencies;
|
||||
- provenance/SBOM where feasible;
|
||||
- vulnerability scanning;
|
||||
- pinned base images;
|
||||
- non-root runtime;
|
||||
- update policy;
|
||||
- build in CI/clean environment.
|
||||
|
||||
### Database compromise/data integrity
|
||||
|
||||
Controls:
|
||||
- isolated network;
|
||||
- dedicated credentials;
|
||||
- TLS when remote;
|
||||
- least privilege;
|
||||
- migrations/transactions;
|
||||
- backup/restore;
|
||||
- input validation;
|
||||
- audit;
|
||||
- no exposed database port unless controlled testing override.
|
||||
|
||||
### Live/WebSocket abuse
|
||||
|
||||
Controls:
|
||||
- authentication before upgrade;
|
||||
- opaque HttpOnly sessions with an eight-hour sliding idle limit and a finite,
|
||||
operator-bounded absolute limit; renewal never exposes OIDC tokens to the
|
||||
browser, while a revocable session context propagates through HTTP upgrades
|
||||
so logout, absolute expiry and request/server cancellation also close an
|
||||
already established socket and release its subscriptions;
|
||||
- origin policy;
|
||||
- message/rate/size limits;
|
||||
- max subscriptions/series;
|
||||
- idle timeout/heartbeat;
|
||||
- bounded send queue and slow-client eviction;
|
||||
- no secret data in messages.
|
||||
|
||||
### Alert/notification abuse
|
||||
|
||||
Controls:
|
||||
- RBAC and audit;
|
||||
- versioned rules;
|
||||
- safe templates;
|
||||
- channel test rate limits;
|
||||
- idempotency;
|
||||
- recipient allowlist/policy;
|
||||
- no secret values in notification body.
|
||||
|
||||
### Deployment mistakes
|
||||
|
||||
Controls:
|
||||
- discovery and port/network/volume conflict checks;
|
||||
- backup touched configs;
|
||||
- isolated compose project;
|
||||
- offline validation;
|
||||
- health/smoke tests;
|
||||
- rollback;
|
||||
- no prune/delete/unrelated modifications;
|
||||
- production evidence.
|
||||
|
||||
## 4. Security headers
|
||||
|
||||
At minimum:
|
||||
- Content-Security-Policy;
|
||||
- frame restrictions;
|
||||
- nosniff;
|
||||
- strict referrer policy;
|
||||
- permissions policy;
|
||||
- HSTS when HTTPS deployment is stable;
|
||||
- secure/same-site/httpOnly cookies.
|
||||
|
||||
## 5. Container hardening
|
||||
|
||||
Where compatible:
|
||||
- non-root;
|
||||
- read-only root filesystem;
|
||||
- tmpfs for temporary paths;
|
||||
- drop all capabilities, add only required;
|
||||
- no-new-privileges;
|
||||
- seccomp/default profile;
|
||||
- resource limits;
|
||||
- explicit networks;
|
||||
- no public database/collector ports;
|
||||
- healthchecks;
|
||||
- immutable image digest in production record.
|
||||
|
||||
The agent may need narrow exceptions; document and test each.
|
||||
|
||||
### Deployment hardening pass (2026-08-04)
|
||||
|
||||
- Resource limits are set on all six `deploy/compose.yaml` services via the
|
||||
non-swarm `cpus`/`mem_limit`/`mem_reservation`/`memswap_limit` keys (the
|
||||
project runs plain `docker compose up`, not swarm); sizing rationale is
|
||||
inline in that file against `docs/architecture/SYSTEM_ARCHITECTURE.md` §7.
|
||||
- `pulse-postgres` now runs `read_only: true` with tmpfs for `/tmp` and
|
||||
`/var/run/postgresql`; all six services are now read-only-root. This
|
||||
closes the previously undocumented exception; see
|
||||
`docs/operations/DEPLOYMENT_UNRAID.md` §6/§8 for the required smoke test.
|
||||
- Immutable image digests are enforced by `deploy/verify-image-digests.sh`
|
||||
and CI. Every external registry image in `deploy/*.Dockerfile` is pinned to
|
||||
a verified digest; only Docker's built-in `scratch` rootfs is exempt because
|
||||
it has no registry manifest. See `deploy/IMAGE_DIGESTS.md` for the ledger.
|
||||
- `pulse-worker`/`pulse-agent` healthchecks now verify a heartbeat file's
|
||||
freshness instead of `kill -0 1`, and self-restart the container on
|
||||
staleness (`docker compose up` does not restart on "unhealthy" status
|
||||
alone). Contract for the Go runtime:
|
||||
`docs/operations/WORKER_AGENT_HEALTHCHECK_CONTRACT.md`.
|
||||
- `deploy/nginx.conf` now sends `Strict-Transport-Security` from
|
||||
`pulse-web` as defence-in-depth (§4), verified not to conflict with
|
||||
TLS terminating at Nginx Proxy Manager per ADR-0010.
|
||||
|
||||
## 6. Security acceptance
|
||||
|
||||
Required:
|
||||
- threat model review at M0 and M9;
|
||||
- SAST/dependency/image/secret scans;
|
||||
- auth/RBAC matrix tests;
|
||||
- SSRF suite;
|
||||
- WebSocket security suite;
|
||||
- dashboard import/XSS suite;
|
||||
- query limit/validation suite;
|
||||
- compose privilege/mount test;
|
||||
- backup secret inspection;
|
||||
- production exposure scan from permitted network.
|
||||
|
||||
### M7-02 implementation
|
||||
|
||||
The probe policy validates scheme, host and port before resolution, rejects loopback/link-local/metadata/multicast/unspecified addresses, and requires explicit CIDR permission for private LAN targets. The safe client repeats resolution at dial time and revalidates redirect destinations. Only bounded GET/HEAD requests with an allowlisted header set are accepted; responses are size/time limited. Policy changes are audit events containing no credentials or full request data.
|
||||
### M8-08 notification controls
|
||||
|
||||
Notification channel persistence accepts only secret references and rejects sensitive configuration keys such as token, password, secret, and authorization. Delivery bodies and subjects are bounded. Sender errors are newline-normalized, length-bounded, and redacted before persistence; bearer credentials are removed as a complete value. Test sends are rate limited and never write delivery records.
|
||||
@@ -0,0 +1,240 @@
|
||||
# System architecture
|
||||
|
||||
## 1. Context
|
||||
|
||||
ITWorx Pulse sits between the browser/operator and existing telemetry/inventory sources. It provides no operational write path to Unraid or Docker in v1.
|
||||
|
||||
### M0 environment binding
|
||||
|
||||
The discovered target is one Unraid 7.2.2 host with the native Unraid GraphQL API online, existing Nginx Proxy Manager and Authentik services, Grafana and host telemetry endpoints, and no local Prometheus container/listener identified during read-only discovery. Pulse therefore remains an additive, isolated Compose project. Host port, DNS name, appdata path, proxy record, and Docker network names are intentionally runtime discovery outputs and are not hardcoded in this architecture document.
|
||||
|
||||
The API/web trust boundary is unchanged by discovery: neither service receives a Docker socket or host filesystem privilege. The agent uses an allowlisted read-only Unraid API capability path. A separately configured Prometheus-compatible endpoint is queried only server-side; if it is unavailable or stale, the datasource and dependent status are `Unknown`.
|
||||
|
||||
```text
|
||||
Browser
|
||||
|
|
||||
| HTTPS / OIDC / REST / WebSocket
|
||||
v
|
||||
Pulse Web + API
|
||||
| \
|
||||
| \ PostgreSQL
|
||||
| config, inventory, events,
|
||||
| alerts, incidents, audit
|
||||
|
|
||||
+--> Prometheus-compatible source
|
||||
| historical metrics and range queries
|
||||
|
|
||||
+--> Pulse Worker
|
||||
| discovery, reconciliation, alert evaluation,
|
||||
| probes, retention, notifications
|
||||
|
|
||||
+--> Pulse Agent / constrained adapters
|
||||
read-only Unraid, host, storage and container facts
|
||||
```
|
||||
|
||||
## 2. Deployable units
|
||||
|
||||
### `pulse-web`
|
||||
|
||||
Preferred outcome: static React application served by a minimal web server or the API gateway. It has no secrets beyond public OIDC configuration and no direct infrastructure access.
|
||||
|
||||
### `pulse-api`
|
||||
|
||||
Responsibilities:
|
||||
- authenticated REST API;
|
||||
- WebSocket authentication and subscriptions;
|
||||
- dashboard/config CRUD;
|
||||
- inventory and event reads;
|
||||
- query planning and limits;
|
||||
- alert/incident user actions;
|
||||
- audit;
|
||||
- health/readiness.
|
||||
|
||||
No Docker socket and no host filesystem privilege.
|
||||
|
||||
### `pulse-worker`
|
||||
|
||||
Responsibilities:
|
||||
- scheduled discovery/reconciliation;
|
||||
- alert rule evaluation;
|
||||
- service probes;
|
||||
- incident correlation;
|
||||
- notification delivery;
|
||||
- retention/cleanup;
|
||||
- periodic self-checks.
|
||||
|
||||
Jobs are idempotent and database-coordinated.
|
||||
|
||||
### `pulse-agent`
|
||||
|
||||
Responsibilities:
|
||||
- host/Unraid/storage/container discovery that cannot be safely obtained through existing APIs/exporters;
|
||||
- normalized event/metric/status collection;
|
||||
- capability reporting.
|
||||
|
||||
The agent has the minimum read-only access required. It does not expose a general shell or mutation endpoint.
|
||||
|
||||
The API, worker and agent may share one Go module and image with separate commands while retaining runtime privilege separation.
|
||||
|
||||
### PostgreSQL
|
||||
|
||||
Stores:
|
||||
- users/external identities and roles;
|
||||
- settings and secrets references;
|
||||
- datasource metadata/health;
|
||||
- inventory and relationships;
|
||||
- dashboards/versions/widgets/layouts;
|
||||
- events;
|
||||
- alert rules/state/occurrences;
|
||||
- incidents/notes;
|
||||
- audit;
|
||||
- job coordination and migrations.
|
||||
|
||||
It does not duplicate full Prometheus time-series data.
|
||||
|
||||
### Prometheus-compatible source
|
||||
|
||||
Provides:
|
||||
- metric metadata;
|
||||
- instant/range query;
|
||||
- historical retention;
|
||||
- existing exporter scrape state.
|
||||
|
||||
Pulse queries it through a constrained server-side adapter. Browser access is forbidden.
|
||||
|
||||
## 3. Logical modules
|
||||
|
||||
```text
|
||||
identity
|
||||
authorization
|
||||
configuration
|
||||
datasources
|
||||
inventory
|
||||
metrics
|
||||
dashboards
|
||||
live
|
||||
events
|
||||
alerts
|
||||
incidents
|
||||
probes
|
||||
notifications
|
||||
audit
|
||||
operations
|
||||
```
|
||||
|
||||
Each module defines domain types and interfaces. HTTP/database/Prometheus/Unraid implementations are adapters.
|
||||
|
||||
## 4. Data flows
|
||||
|
||||
### Dashboard load
|
||||
|
||||
1. Browser requests dashboard definition.
|
||||
2. API authorizes user and returns validated config.
|
||||
3. Browser sends a batched semantic historical query.
|
||||
4. API query planner validates bounds, translates to PromQL and deduplicates.
|
||||
5. Prometheus returns series.
|
||||
6. API normalizes units/metadata and returns bounded points.
|
||||
7. Browser opens one WebSocket and subscribes to visible live streams.
|
||||
8. Live broker coalesces equivalent subscriptions.
|
||||
|
||||
### Discovery
|
||||
|
||||
1. Worker requests capability/inventory snapshots from adapters/agent.
|
||||
2. Payloads are validated and source-stamped.
|
||||
3. Reconciler maps stable identities and relationships.
|
||||
4. User overrides remain separate.
|
||||
5. Changes create normalized events.
|
||||
6. Datasource freshness/status is updated.
|
||||
|
||||
### Alert evaluation
|
||||
|
||||
1. Worker selects due rules with a coordination lock.
|
||||
2. Rule queries semantic metric/event/status inputs.
|
||||
3. State machine applies pending/hysteresis/cooldown.
|
||||
4. Occurrence/event/audit records persist transactionally.
|
||||
5. Suppression/grouping is calculated.
|
||||
6. Incident correlation and notifications run.
|
||||
7. UI receives a live state update.
|
||||
|
||||
## 5. Trust boundaries
|
||||
|
||||
1. Browser <-> web/API.
|
||||
2. Pulse runtime <-> OIDC provider.
|
||||
3. API/worker <-> PostgreSQL.
|
||||
4. API/worker <-> Prometheus.
|
||||
5. Agent/adapters <-> host/Unraid/Docker.
|
||||
6. Probe worker <-> configured network targets.
|
||||
7. Notification worker <-> external channels.
|
||||
|
||||
Each boundary requires authentication/authorization, timeouts, validation, redaction and least privilege.
|
||||
|
||||
## 6. Availability model
|
||||
|
||||
Pulse may run as a single instance in v1. It must recover from restarts without losing configuration or alert history.
|
||||
|
||||
- Web/API failure: external health check detects.
|
||||
- Worker failure: heartbeats and dead-man alert.
|
||||
- Agent failure: datasource unknown/stale; no false green.
|
||||
- Prometheus failure: historical/live metrics unknown; inventory remains available.
|
||||
- Database failure: API not ready; no in-memory claim of health.
|
||||
- OIDC failure: existing sessions follow policy; break-glass recovery remains controlled.
|
||||
|
||||
## 7. Scaling limits
|
||||
|
||||
Target:
|
||||
- one host;
|
||||
- 150 containers;
|
||||
- 40 disks;
|
||||
- 300 service probes;
|
||||
- 2,500 active dashboard series;
|
||||
- 10 concurrent users;
|
||||
- long-running wallboard.
|
||||
|
||||
The architecture must bound:
|
||||
- Prometheus concurrency;
|
||||
- series/points;
|
||||
- browser ring buffers;
|
||||
- WebSocket subscriptions;
|
||||
- event payloads;
|
||||
- probe concurrency/response size;
|
||||
- inventory snapshots;
|
||||
- audit retention.
|
||||
|
||||
## 8. Configuration
|
||||
|
||||
Configuration layers:
|
||||
|
||||
1. secure runtime environment/secrets;
|
||||
2. validated application config;
|
||||
3. database-managed settings;
|
||||
4. user/dashboard preferences.
|
||||
|
||||
Startup fails clearly for invalid mandatory config. Optional integrations report disabled/unavailable capabilities.
|
||||
|
||||
## 9. Extensibility
|
||||
|
||||
Connectors implement a capability interface:
|
||||
|
||||
```text
|
||||
discover
|
||||
health
|
||||
inventory
|
||||
metrics bindings
|
||||
events
|
||||
capabilities
|
||||
```
|
||||
|
||||
No connector receives arbitrary access to core storage or bypasses authorization. Version connector contracts.
|
||||
|
||||
## 10. Architecture fitness tests
|
||||
|
||||
Automated tests must enforce:
|
||||
- no Docker socket mount on web/API service;
|
||||
- no mutation operation in public v1 API;
|
||||
- migrations present for schema changes;
|
||||
- OpenAPI/schema compatibility;
|
||||
- package/module dependency direction;
|
||||
- bounded query defaults;
|
||||
- stale -> unknown mapping;
|
||||
- non-root container configuration;
|
||||
- secrets absent from image/repo.
|
||||
@@ -0,0 +1,168 @@
|
||||
# Telemetry and query engine
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Pulse exposes a stable semantic metric model while retaining Prometheus as the v1 time-series source.
|
||||
|
||||
Users/widgets request:
|
||||
|
||||
```text
|
||||
container.cpu.utilization
|
||||
storage.disk.temperature
|
||||
service.response_time
|
||||
```
|
||||
|
||||
The backend maps these names to source-specific PromQL templates and label contracts.
|
||||
|
||||
## 2. Metric definition
|
||||
|
||||
A metric definition includes:
|
||||
|
||||
- semantic name and version;
|
||||
- description and unit;
|
||||
- gauge/counter/state kind;
|
||||
- required capabilities;
|
||||
- query template;
|
||||
- allowed labels/grouping;
|
||||
- default aggregation;
|
||||
- valid transformations;
|
||||
- default interval;
|
||||
- max range/series/points;
|
||||
- freshness;
|
||||
- visualizations;
|
||||
- status/threshold hints;
|
||||
- cardinality budget.
|
||||
|
||||
See `specs/metric-definition.schema.json`.
|
||||
|
||||
## 3. Query planning
|
||||
|
||||
Pipeline:
|
||||
|
||||
1. authenticate and authorize;
|
||||
2. validate semantic metric and scope;
|
||||
3. resolve entity aliases/source binding;
|
||||
4. clamp/validate range, step, series and points;
|
||||
5. select binding based on source capabilities;
|
||||
6. generate parameterized PromQL from templates;
|
||||
7. deduplicate equivalent queries;
|
||||
8. execute with timeout/concurrency budget;
|
||||
9. normalize labels, units and missing values;
|
||||
10. downsample if needed;
|
||||
11. return provenance, warnings and freshness.
|
||||
|
||||
Raw PromQL is an advanced feature, disabled by default, separately authorized and constrained.
|
||||
|
||||
## 4. Query budgets
|
||||
|
||||
Budgets are configurable but must exist:
|
||||
|
||||
- max range;
|
||||
- max series;
|
||||
- max samples/points;
|
||||
- max query string/template expansion;
|
||||
- timeout;
|
||||
- per-user/per-dashboard concurrency;
|
||||
- global Prometheus concurrency;
|
||||
- cacheable result size;
|
||||
- live subscription count;
|
||||
- label enumeration limits.
|
||||
|
||||
Return a clear limit error rather than overloading Prometheus.
|
||||
|
||||
## 5. Caching
|
||||
|
||||
Cache:
|
||||
- metric catalog/capabilities;
|
||||
- short historical range results;
|
||||
- label/metadata results;
|
||||
- repeated dashboard query plans.
|
||||
|
||||
Do not cache:
|
||||
- authorization decisions beyond safe session scope;
|
||||
- live current status beyond its freshness policy;
|
||||
- secret-bearing errors.
|
||||
|
||||
Cache keys include source, tenant/server scope, semantic query, normalized range and permissions where relevant.
|
||||
|
||||
Redis is not required for a single instance. Use bounded in-memory cache and/or PostgreSQL only when appropriate.
|
||||
|
||||
## 6. Live engine
|
||||
|
||||
### Fast lane
|
||||
|
||||
- visible high-frequency widgets subscribe at 1–5 seconds;
|
||||
- backend polls/queries compatible batches or receives collector samples;
|
||||
- equivalent subscriptions share upstream work;
|
||||
- samples are appended with sequence numbers;
|
||||
- browser retains a bounded ring buffer;
|
||||
- historical data is not fully refetched per sample.
|
||||
|
||||
### Durable lane
|
||||
|
||||
Prometheus scrape/history remains durable. Pulse does not persist every live sample into PostgreSQL.
|
||||
|
||||
### Adaptive behavior
|
||||
|
||||
- out-of-view widgets reduce frequency;
|
||||
- background tabs reduce frequency;
|
||||
- paused dashboards unsubscribe;
|
||||
- wallboards remain active with bounded buffers;
|
||||
- slow clients receive coalesced latest samples;
|
||||
- sequence gaps trigger bounded resync.
|
||||
|
||||
## 7. Browser chart architecture
|
||||
|
||||
- historical query initializes chart;
|
||||
- live samples append outside global React state where practical;
|
||||
- series count and point count are capped;
|
||||
- old points are evicted;
|
||||
- chart resources are disposed on unmount;
|
||||
- ResizeObserver/visibility changes are debounced;
|
||||
- tooltip/legend state does not duplicate large arrays;
|
||||
- long wallboard test measures heap and subscription count.
|
||||
|
||||
## 8. Staleness and unknown
|
||||
|
||||
Each response includes:
|
||||
- source timestamp;
|
||||
- received timestamp;
|
||||
- freshness state;
|
||||
- warnings.
|
||||
|
||||
If a required source is stale/unavailable:
|
||||
- current value is marked stale or omitted;
|
||||
- status becomes unknown according to policy;
|
||||
- previous value may be displayed with age;
|
||||
- alerts can enter unknown rather than resolve.
|
||||
|
||||
## 9. Host snapshot binding
|
||||
|
||||
Host detail reads use the explicit read-only host contract. A source adapter returns bounded normalized values with UTC observed/received timestamps and capability version. The adapter sorts filesystem/interface collections deterministically, validates percentage/byte units and rejects payloads over configured collection limits. The API/web boundary has no host or Docker privilege; when no approved source is connected, the result is an explicit unknown snapshot. Load and memory status reasons are returned as text so high load is distinguishable from stale or unavailable telemetry.
|
||||
## 9. Transformations
|
||||
|
||||
Supported through typed operations:
|
||||
- rate;
|
||||
- increase/delta;
|
||||
- average/min/max/sum;
|
||||
- quantile;
|
||||
- percentage;
|
||||
- top/bottom N;
|
||||
- unit conversion;
|
||||
- compare previous period;
|
||||
- fill policy;
|
||||
- status mapping.
|
||||
|
||||
Transform order is explicit and validated. Avoid silently mixing counter rates and gauges.
|
||||
|
||||
|
||||
Optional hardware follows the same boundary: declared thermal, fan and GPU capabilities are independently enabled, unsupported, unavailable or disabled. Missing capability is not an adapter failure. Sensor IDs are source-scoped and stable from external ID/name, collections are bounded and sorted, and thermal reasons are returned as text. No API/web host privilege, device mount or unrestricted socket is needed.
|
||||
## 10. Cardinality controls
|
||||
|
||||
- semantic definitions allow only known labels;
|
||||
- unbounded label values are excluded or normalized;
|
||||
- inventory IDs map to stable bounded labels;
|
||||
- dashboards cannot group by arbitrary label by default;
|
||||
- source cardinality health is monitored;
|
||||
- query inspector shows series count and limits.
|
||||
- query inspector is operate-permissioned, exposes only planner-approved generated PromQL and redacts sensitive-looking text; it does not execute an unrestricted query.
|
||||
@@ -0,0 +1,19 @@
|
||||
# ADR 0001 — Pulse v1 is operationally read-only
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
Pulse may observe, query, configure its own monitoring behavior and create user records such as acknowledgements. It may not mutate Unraid, Docker, storage, containers, host processes or network configuration.
|
||||
|
||||
## Rationale
|
||||
|
||||
Monitoring and management have different security and failure domains. A compromised dashboard must not become a host control plane. Remediation belongs in a separately controlled AppOps workflow.
|
||||
|
||||
## Consequences
|
||||
|
||||
- No start/stop/restart/delete APIs.
|
||||
- No process kill.
|
||||
- No array/pool operations.
|
||||
- Optional links may open an external management workflow.
|
||||
- Tests enforce absence of mutation routes and dangerous agent capabilities.
|
||||
@@ -0,0 +1,17 @@
|
||||
# ADR 0002 — Go services and React/TypeScript web application
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
Use Go for API, worker and agent commands. Use React + TypeScript for the web application, with Vite as the default build tool unless discovery proves a stronger need for server-rendered Next.js behavior.
|
||||
|
||||
## Rationale
|
||||
|
||||
The product is a highly interactive authenticated dashboard rather than a public content site. Go provides efficient long-lived connections, concurrency and small deployable binaries. React has mature dashboard/grid/chart ecosystems. A static frontend reduces production runtime complexity.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Shared contracts are generated or validated from schemas.
|
||||
- API/worker/agent can share modules but deploy with separate privileges.
|
||||
- An ADR is required to add a Node server runtime to production.
|
||||
@@ -0,0 +1,17 @@
|
||||
# ADR 0003 — Existing Prometheus-compatible source remains v1 metrics history
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
Use the existing Prometheus-compatible endpoint for instant/range queries and retention in v1. Do not introduce VictoriaMetrics or another time-series database until measurements show a concrete retention, performance or reliability requirement.
|
||||
|
||||
## Rationale
|
||||
|
||||
Avoid duplicate infrastructure and migration risk. Pulse differentiates through semantic queries, UX, inventory, alerts and incidents.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Query planner and limits protect the source.
|
||||
- Pulse stores no full metric history in PostgreSQL.
|
||||
- Long-term storage remains a future measured decision.
|
||||
@@ -0,0 +1,18 @@
|
||||
# ADR 0004 — PostgreSQL stores Pulse domain state
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
Use PostgreSQL for configuration, inventory, relationships, dashboards, events, alert state/history, incidents, audit and job coordination.
|
||||
|
||||
## Rationale
|
||||
|
||||
The data is relational, transactional and queryable. PostgreSQL supports JSONB for bounded extensibility and reliable migrations/backups.
|
||||
|
||||
## Consequences
|
||||
|
||||
- All schema changes use migrations.
|
||||
- Production database is private.
|
||||
- Backup/restore is a release gate.
|
||||
- Metric samples remain outside PostgreSQL.
|
||||
@@ -0,0 +1,17 @@
|
||||
# ADR 0005 — No unrestricted Docker socket in API/web
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
The web and API containers never mount the unrestricted Docker socket. Docker/Unraid facts come from the official Unraid API, existing exporters, Portainer read-only endpoints, or a separate constrained agent/socket proxy.
|
||||
|
||||
## Rationale
|
||||
|
||||
Docker daemon access is effectively host control. Read-only filesystem mount flags do not create a read-only Docker API.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Collector capability endpoints are allowlisted.
|
||||
- Agent runtime is separated and audited.
|
||||
- Architecture tests inspect compose mounts and API routes.
|
||||
@@ -0,0 +1,17 @@
|
||||
# ADR 0006 — REST plus WebSocket
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
Use versioned REST for requests/configuration/history and one authenticated WebSocket per browser session for bounded live updates.
|
||||
|
||||
## Rationale
|
||||
|
||||
REST is clear for CRUD/query operations. WebSocket supports low-latency subscriptions and server-side coalescing without repeated polling of full datasets.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Messages use explicit schema/version/sequence.
|
||||
- Reconnect and resync are required.
|
||||
- Subscription, rate, size and buffer limits are mandatory.
|
||||
@@ -0,0 +1,17 @@
|
||||
# ADR 0007 — Dutch default, localization-ready architecture
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
Ship user-facing v1 flows in Dutch (`nl-BE`) and use localization keys/contracts so English can be added without rewriting components.
|
||||
|
||||
## Rationale
|
||||
|
||||
The primary user is Dutch-speaking, while code and technical contracts benefit from stable English identifiers.
|
||||
|
||||
## Consequences
|
||||
|
||||
- No hardcoded scattered UI copy.
|
||||
- Tests cover missing translation keys.
|
||||
- Dates/numbers use locale and Europe/Brussels display defaults while storage remains UTC.
|
||||
@@ -0,0 +1,18 @@
|
||||
# ADR 0008 — Stale or missing required telemetry is Unknown
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
When required telemetry is unavailable beyond its freshness policy, Pulse reports Unknown rather than retaining or inferring Healthy.
|
||||
|
||||
## Rationale
|
||||
|
||||
False green status is more dangerous than explicit uncertainty.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Responses carry freshness metadata.
|
||||
- Alert rules define unknown behavior.
|
||||
- UI shows last known value only with age.
|
||||
- Tests inject stale/missing sources throughout the product.
|
||||
@@ -0,0 +1,42 @@
|
||||
# ADR 0009 — Upstream and dependency baseline after M0 research
|
||||
|
||||
**Status:** Accepted for M1 planning; exact package versions remain pinned during M1 implementation.
|
||||
|
||||
## Context
|
||||
|
||||
M0 must verify current upstream capabilities and avoid selecting unsupported or abandoned dependencies. The target Unraid host is 7.2.2 with its native Unraid API online. The local development host has Node.js 24.18.1 and pnpm 10.33.0 but no Go toolchain.
|
||||
|
||||
## Decision
|
||||
|
||||
- Use the native Unraid GraphQL API as the first inventory adapter. Unraid 7.2+ includes the API in the OS, and programmatic access supports API keys, cookies, and SSO/OIDC. Pulse uses a least-privilege read-only API key or equivalent controlled identity; it must not use mutation capabilities.
|
||||
- Keep Prometheus-compatible history as an external server-side datasource. Pulse uses the stable `/api/v1/query` and `/api/v1/query_range` APIs through bounded semantic queries; the browser never contacts Prometheus directly.
|
||||
- Use Authentik OIDC/OAuth2 as the production identity integration. The server performs authorization-code exchange and token validation; public/browser flows use PKCE where applicable. Per-provider issuer/discovery is the default because authentik documents it as the recommended issuer mode.
|
||||
- Implement the frontend with React + TypeScript + Vite, using the current React documentation baseline (19.2) and Vite's supported Node requirement. Use GridStack as the dashboard-grid candidate; retain a measured-equivalent escape hatch. Do not commit exact package versions until M1 lockfile/bootstrap work.
|
||||
- Prefer Go's standard library for the initial backend transport and pin a currently supported Go release in M1. The official Go release policy supports a major release until two newer majors exist. The baseline is maintained at Go 1.26.6 after the M13 release image gate identified fixed standard-library findings in 1.26.5.
|
||||
|
||||
## License and support record
|
||||
|
||||
| Component | Upstream license observed | Support/compatibility note |
|
||||
|---|---|---|
|
||||
| Go toolchain | BSD-style (official Go distribution) | Use an official currently supported release; Go is not installed locally yet. |
|
||||
| React | MIT | Official React docs list 19.2 as latest major baseline. |
|
||||
| Vite | MIT | Official docs require Node 20.19+ or 22.12+; local Node 24.18.1 satisfies the documented floor. |
|
||||
| GridStack | MIT | Candidate only; verify package release and transitive dependencies during M1. |
|
||||
| Prometheus API/source | Apache 2.0 | External source; Pulse does not redistribute Prometheus in the application image. |
|
||||
| authentik integration | MIT core with documented directory/component exceptions | Pulse integrates with the deployed provider; it does not embed or redistribute authentik. |
|
||||
|
||||
## Consequences
|
||||
|
||||
- M1 must provision Go, generate lockfiles, run license/dependency checks, and pin exact versions from official release metadata.
|
||||
- Prometheus location remains unresolved in the actual Unraid environment: no local 9090 listener/container was found. Datasource onboarding must support an explicitly configured external endpoint and report Unknown when absent/stale.
|
||||
- The frontend runtime remains a static React/Vite artifact served behind the API/reverse proxy, preserving the no-infrastructure-access browser boundary.
|
||||
|
||||
## Sources accessed 2026-08-01
|
||||
|
||||
- [Unraid API overview](https://docs.unraid.net/API/) and [Unraid API usage](https://docs.unraid.net/API/how-to-use-the-api/)
|
||||
- [Go release history](https://go.dev/doc/devel/release)
|
||||
- [Prometheus HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/) and [PromQL basics](https://prometheus.io/docs/prometheus/latest/querying/basics/)
|
||||
- [authentik OAuth 2.0/OIDC provider](https://docs.goauthentik.io/add-secure-apps/providers/oauth2)
|
||||
- [React versions](https://react.dev/versions) and [React reference](https://react.dev/reference/react)
|
||||
- [Vite getting started and compatibility](https://vite.dev/guide/)
|
||||
- [React MIT license](https://raw.githubusercontent.com/react/react/main/LICENSE), [Vite MIT license](https://raw.githubusercontent.com/vitejs/vite/main/LICENSE), [GridStack MIT license](https://raw.githubusercontent.com/gridstack/gridstack.js/master/LICENSE), [Prometheus Apache 2.0 license](https://raw.githubusercontent.com/prometheus/prometheus/main/LICENSE), and [authentik license](https://raw.githubusercontent.com/goauthentik/authentik/main/LICENSE)
|
||||
Reference in New Issue
Block a user