This commit is contained in:
ITWorx Pulse release export
@@ -0,0 +1,46 @@
|
||||
# Public deployment guide
|
||||
|
||||
This guide describes the portable deployment contract. It intentionally contains no private hostnames, addresses, paths, identity-provider groups, or production evidence.
|
||||
|
||||
## Supported topology
|
||||
|
||||
Run the base Compose file together with the production overlay:
|
||||
|
||||
```bash
|
||||
docker compose -f deploy/compose.yaml -f deploy/compose.prod.yaml config
|
||||
docker compose -f deploy/compose.yaml -f deploy/compose.prod.yaml up -d
|
||||
```
|
||||
|
||||
Only `pulse-web` is published. The API, PostgreSQL, worker, agent, and migration job stay on explicit internal networks. Put a TLS reverse proxy in front of the web edge.
|
||||
|
||||
## Required configuration
|
||||
|
||||
Copy `.env.example` to a protected runtime location outside Git and replace every example value. Production startup requires:
|
||||
|
||||
- `PULSE_POSTGRES_PASSWORD` and `PULSE_DATABASE_URL` for a dedicated database;
|
||||
- `PULSE_PUBLIC_URL` with the external HTTPS origin;
|
||||
- OIDC issuer, client ID, client secret, redirect URL, groups claim, and explicit claim-to-role mapping;
|
||||
- a pre-created `PULSE_BACKUP_DIR` writable only by the API runtime identity;
|
||||
- `PULSE_UNRAID_URL`, a least-privilege read-only token, a public CA certificate, and matching `PULSE_UNRAID_HOST_NAME`/`PULSE_UNRAID_HOST_GATEWAY` when Unraid API collection is enabled;
|
||||
- an approved Prometheus-compatible endpoint when historical metrics are required.
|
||||
|
||||
Never place secrets in Compose literals, command history, logs, screenshots, or repository evidence.
|
||||
|
||||
## Preflight
|
||||
|
||||
Before starting:
|
||||
|
||||
1. confirm the selected host port is unused;
|
||||
2. verify project, network, volume, and backup paths do not overlap another application;
|
||||
3. render Compose and review every mount, published port, capability, user, and secret recipient;
|
||||
4. run `deploy/verify-image-digests.sh`;
|
||||
5. take checksummed backups of every external configuration that will change;
|
||||
6. record the current image, schema, proxy configuration, and rollback procedure.
|
||||
|
||||
## Acceptance
|
||||
|
||||
Require all services to become healthy, run migrations explicitly, verify `/healthz` and `/readyz`, complete an HTTPS/OIDC login, verify WebSocket reconnect, check datasource freshness, restart the stack, and repeat the smoke tests. Run `scripts/production-smoke.ps1` against the actual public origin to check headers, redirects, anonymous exposure, and readiness behavior.
|
||||
|
||||
## Rollback
|
||||
|
||||
Stop only the Pulse project being deployed. Restore the previous proxy configuration and exact image/configuration. Restore PostgreSQL only from a verified backup when forward recovery is not safe. Never prune Docker or delete unknown networks, volumes, containers, or host data.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Public source boundary
|
||||
|
||||
The canonical engineering repository combines product source with private planning, prompt, packaging, and operational evidence. Do not make that history public directly. Create a parentless candidate with `scripts/export-public-source.mjs` and validate that candidate before publication.
|
||||
|
||||
A public source set may include reviewed code under `cmd/`, `internal/`, `apps/`, stable configuration schemas, synthetic fixtures, durable user documentation, and reproducible build scripts selected by `public-source.allowlist`.
|
||||
|
||||
Exclude from public source/release archives unless individually reviewed:
|
||||
|
||||
- `.agents/`, `.codex/`, prompt packs and operator-specific instructions;
|
||||
- `artifacts/`, generated evidence, checksums and package reports not produced by the tagged release job;
|
||||
- private deployment configuration, host inventories and telemetry;
|
||||
- transient planning/state documents and local test output;
|
||||
- credentials, `.env` files, private keys and production data.
|
||||
|
||||
The exporter blocks when `LICENSE` is missing and scans the rendered candidate for private topology, machine-local paths, denied directories, symlinks, and oversized files. It writes a content manifest so the published tree can be reviewed independently of private Git history.
|
||||
|
||||
First-party source code selected for the public source set is licensed under **AGPL-3.0-or-later**; see the root `LICENSE`. Third-party components and assets retain their own terms and must remain covered by the repository's dependency and notice documentation.
|
||||
|
||||
Before any visibility change, also scan every private ref and object. A clean current export does not sanitize the canonical repository history.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Repository and Release Boundary
|
||||
|
||||
The canonical private repository contains both product source and intentional engineering context. These categories have different publication and packaging rules.
|
||||
|
||||
## Product source
|
||||
|
||||
Application code, contracts, migrations, deployment templates, tests and public product documentation may enter a reviewed source release when they contain only generic configuration and redistributable assets.
|
||||
|
||||
## Development-only context
|
||||
|
||||
`.agents/`, `.codex/`, `planning/`, `MASTER_PROMPT.txt`, detailed audit state and milestone evidence support private development. They remain tracked for continuity but are excluded from `git archive` release output. Machine-local Codex execution policy is not tracked.
|
||||
|
||||
## Runtime/private data
|
||||
|
||||
Telemetry, host inventories, dashboards, alerts, incidents, backups, credentials, OIDC state, local databases and generated heavy evidence must remain outside source control.
|
||||
|
||||
## Licensing
|
||||
|
||||
First-party source code in the repository is licensed under **AGPL-3.0-or-later**; see the root `LICENSE`. Third-party components and assets retain their own terms. A future public release must still use the reviewed parentless export and pass its publication gates; the canonical private history is not itself a publication artifact.
|
||||
@@ -0,0 +1,185 @@
|
||||
# Alerting and incident architecture
|
||||
|
||||
## 1. Concepts
|
||||
|
||||
- **Rule:** versioned definition of a condition.
|
||||
- **Alert instance:** stable rule + entity/label fingerprint.
|
||||
- **Occurrence:** immutable state transition/evaluation history.
|
||||
- **Incident:** grouped operational problem containing related alerts/entities/events.
|
||||
- **Silence:** user-defined temporary notification/state suppression.
|
||||
- **Maintenance window:** scheduled policy for selected entities.
|
||||
- **Inhibition/suppression:** dependency-aware prevention of downstream noise.
|
||||
|
||||
## 2. State machine
|
||||
|
||||
Primary states:
|
||||
|
||||
```text
|
||||
inactive -> pending -> firing -> acknowledged -> resolved
|
||||
```
|
||||
|
||||
Orthogonal/contextual states:
|
||||
- unknown;
|
||||
- silenced;
|
||||
- suppressed;
|
||||
- maintenance.
|
||||
|
||||
Acknowledgement does not mean resolved. A firing alert may be acknowledged.
|
||||
|
||||
## 3. Evaluation
|
||||
|
||||
Every rule defines:
|
||||
- schedule/evaluation interval;
|
||||
- semantic query/event/status input;
|
||||
- scope selector;
|
||||
- comparator/condition;
|
||||
- pending duration;
|
||||
- recovery duration;
|
||||
- hysteresis;
|
||||
- severity;
|
||||
- message template;
|
||||
- grouping/suppression metadata;
|
||||
- unknown behavior.
|
||||
|
||||
Evaluation is idempotent and transactional.
|
||||
|
||||
## 4. Unknown behavior
|
||||
|
||||
When required data is stale/unavailable:
|
||||
- do not auto-resolve a firing alert;
|
||||
- transition/display unknown according to rule policy;
|
||||
- retain last known value/time;
|
||||
- emit source-health context;
|
||||
- avoid repeated notifications for the same unknown episode.
|
||||
|
||||
## 5. Hysteresis example
|
||||
|
||||
Disk temperature:
|
||||
- enter pending above 50°C;
|
||||
- fire after 5 minutes;
|
||||
- remain firing until below 46°C for 5 minutes;
|
||||
- cooldown duplicate notifications.
|
||||
|
||||
Both thresholds and durations are visible in rule UI.
|
||||
|
||||
## 6. Deduplication/grouping
|
||||
|
||||
Fingerprint is based on:
|
||||
- rule ID/version behavior;
|
||||
- stable entity ID;
|
||||
- bounded grouping labels.
|
||||
|
||||
Notifications group by:
|
||||
- incident;
|
||||
- host/application;
|
||||
- severity/time window;
|
||||
- configured labels.
|
||||
|
||||
## 7. Dependency suppression
|
||||
|
||||
Examples:
|
||||
- host unreachable suppresses container/service unreachable alerts;
|
||||
- DNS failure suppresses dependent service resolution alerts;
|
||||
- reverse proxy outage groups external endpoint failures while internal probes remain separate;
|
||||
- Prometheus unavailable suppresses metric-rule noise and creates a monitoring-source incident.
|
||||
|
||||
Suppressed alerts remain inspectable.
|
||||
|
||||
Pulse uses bounded, deterministic cause keys for suppression (`host.unreachable`, `dns.failure`, `source.unavailable`). A downstream alert is suppressed only when the configured cause is active and confirmed or high-confidence; the downstream signal, fingerprint and suppression reason remain inspectable. Group keys use rule identity, severity and configured stable labels with bounded cardinality. Duplicate evaluation signals are collapsed by instance and evaluation key before grouping.
|
||||
|
||||
## 8. Maintenance and silences
|
||||
|
||||
Maintenance:
|
||||
- scheduled;
|
||||
- selector-based;
|
||||
- audited;
|
||||
- optionally changes visual status to maintenance;
|
||||
- prevents configured notifications without deleting evidence.
|
||||
|
||||
Silence:
|
||||
- explicit reason;
|
||||
- owner;
|
||||
- expiry required;
|
||||
- matchers bounded/previewable;
|
||||
- audited.
|
||||
|
||||
## 9. Incident correlation
|
||||
|
||||
Deterministic rules first:
|
||||
- shared parent dependency;
|
||||
- same application/host;
|
||||
- close start time;
|
||||
- known causal hierarchy;
|
||||
- common event.
|
||||
|
||||
Heuristic correlation may be added later, but must expose confidence and rationale.
|
||||
|
||||
Incident severity is derived from:
|
||||
- highest unsuppressed alert;
|
||||
- criticality of affected entities;
|
||||
- duration/scope;
|
||||
- explicit operator override.
|
||||
|
||||
## 10. Notifications
|
||||
|
||||
Channels may include email/webhook/other selected integrations.
|
||||
|
||||
Requirements:
|
||||
- encrypted secret references;
|
||||
- idempotency;
|
||||
- retry with bounded exponential backoff;
|
||||
- delivery audit;
|
||||
- templates with safe escaping;
|
||||
- grouping and cooldown;
|
||||
- recovery notification;
|
||||
- test action;
|
||||
- channel failure health.
|
||||
|
||||
## 11. Default rules
|
||||
|
||||
Provide conservative baseline rules for:
|
||||
- monitoring source stale/down;
|
||||
- host resource/temperature;
|
||||
- container stop/health/restart loop;
|
||||
- array/pool/disk/SMART/capacity;
|
||||
- service availability/latency/TLS;
|
||||
- UPS battery/on-battery when available.
|
||||
|
||||
Defaults are versioned, visible and editable. Avoid hard thresholds where hardware-specific policy is needed; use sensible presets with onboarding confirmation.
|
||||
|
||||
M8-06 implementation note: silences and maintenance windows use explicit bounded matchers (rule IDs, entity IDs, entity types, severities, and exact labels). Preview returns deterministic matching instance IDs. Expiry is mandatory, state transitions are retained, and revocation or expiry never deletes alert occurrences. Maintenance state is exposed as scheduled, active, expired, or revoked.
|
||||
|
||||
M8-07 operation semantics: acknowledgement is an operator state annotation, not resolution. Acknowledgement and unacknowledgement are persisted as immutable occurrence events, use optimistic revisions, and are idempotent by evaluation key. Evaluator transitions remain authoritative for resolved state.
|
||||
|
||||
### M8-08 delivery semantics
|
||||
|
||||
The notification framework uses the alert lifecycle event type (`firing`, `recovery`, or `unknown`) as the outbox contract. Enqueue is idempotent by caller-supplied key. Workers claim due rows with a bounded lease and PostgreSQL row locking, write a `delivering` audit record, and complete the same attempt transactionally. Failures are redacted, retried with bounded exponential delay, and terminally marked failed after ten attempts. A recovery event is delivered through the same path.
|
||||
|
||||
The production transport is a bounded HTTPS webhook configured with `PULSE_NOTIFICATION_WEBHOOK_URL`, `PULSE_NOTIFICATION_WEBHOOK_TOKEN`, and an optional 1–30 second `PULSE_NOTIFICATION_WEBHOOK_TIMEOUT`. The worker reconciles one system-owned webhook channel at startup. PostgreSQL stores only the URL, timeout, and an opaque runtime secret reference; the bearer credential never enters channel JSON, delivery audit, API responses, or logs. Requests contain the lifecycle payload and repeat the stable outbox idempotency key in both the body and `Idempotency-Key` header. Receivers must use that key to collapse a replay when delivery succeeded but acknowledgement or local completion was interrupted. Redirects, URL credentials, query strings, fragments, unbounded timeouts, and non-HTTPS production endpoints are rejected.
|
||||
|
||||
Channel test actions are in-memory only, do not enqueue an outbox record, and are rate limited per actor/channel pair.
|
||||
### M8-09 correlation behavior
|
||||
|
||||
Correlation starts with deterministic keys: a primary dependency outage groups the dependency alert and downstream alerts sharing its parent; otherwise stable application, host, or entity keys are used. Groups are sorted by alert ID and correlation output is stable across input ordering. Every candidate stores the method and confidence used; the model does not present heuristic correlation as proven causation. Manual alert association is persisted separately, audited at the API boundary, and protected from automatic overwrite.
|
||||
|
||||
### M8-10 incident timeline and operator context
|
||||
|
||||
Incident detail presents correlated alerts and operator notes in a deterministic UTC timeline. Ownership and notes are auditable metadata only. The UI explicitly states that correlation is an evidence-backed indication rather than proven causality and that confidence describes the correlation rule, not certainty of root cause. External workflow linking is a disabled placeholder; Pulse remains observational and does not remediate infrastructure.
|
||||
|
||||
|
||||
### M8-11 default rule seeding
|
||||
|
||||
The implementation-owned embedded default bundle is schema-versioned and validated against the semantic metric catalog before startup reconciliation. Seeding is additive and idempotent: a missing default is inserted, while an existing rule ID—including a customized version—is preserved and counted as existing. Event-based restart-loop rules explicitly support `count` aggregation. Duplicate evaluation signals are collapsed before deterministic grouping; dependency causes remain inspectable while downstream notifications are suppressed.
|
||||
|
||||
## Default storage-pool rule — evaluability fix (2026-08-17)
|
||||
|
||||
The seeded default "Opslagpool bijna vol" originally bound `storage.pool.utilization`, whose query
|
||||
template requires a `{{pool}}` value; the alert evaluator issues one instant query per rule without a pool
|
||||
scope and does not fan out over `groupBy`, so the rule could never evaluate
|
||||
(`PROMQL_BINDING_VALUE_REQUIRED (pool)`). It now uses `storage.pool.utilization.maximum` (placeholder-free:
|
||||
the highest utilisation across all pools) so the default fires when any pool crosses the threshold; per-pool
|
||||
rules can still be created with a `poolId` scope. The seed refreshes implementation-owned defaults that are
|
||||
still at revision 1 (never edited by an operator) through the normal versioned update path
|
||||
(`alertdefaults.Seed` → `RuleUpdater`), and never touches edited rules. The `pulse_storage_pool_*` series must
|
||||
be provided by the environment — for Unraid via the recording rules in
|
||||
`deploy/prometheus.pulse-storage-pool.rules.yaml`.
|
||||
@@ -0,0 +1,371 @@
|
||||
# API contract
|
||||
|
||||
## 1. General rules
|
||||
|
||||
- Base path: `/api/v1`.
|
||||
- JSON request/response.
|
||||
- OpenAPI is generated/validated in CI.
|
||||
- UTC RFC3339 timestamps.
|
||||
- UUID resource IDs.
|
||||
- Cursor pagination for mutable/large collections.
|
||||
- `ETag` or explicit version for optimistic concurrency.
|
||||
- Problem details style errors with safe message, code and correlation ID.
|
||||
- Authentication required except health and OIDC bootstrap endpoints.
|
||||
- Authorization is server-side for every resource/action.
|
||||
- All list/query inputs are bounded.
|
||||
|
||||
## 2. Error shape
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "https://pulse.local/problems/query-limit",
|
||||
"title": "Query limit exceeded",
|
||||
"status": 422,
|
||||
"code": "QUERY_POINT_LIMIT",
|
||||
"detail": "Reduce the time range or increase the step.",
|
||||
"correlationId": "01...",
|
||||
"fields": {
|
||||
"range": "..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
No stack trace or secret-bearing upstream response.
|
||||
|
||||
## 3. Authoritative route inventory
|
||||
|
||||
`specs/api-routes.json` is the machine-readable source for this table. `python tools/check_api_contract.py` verifies that every router registration is represented here, every implementation file exists, and this table has neither missing nor invented method/path combinations. Paths outside `/api/v1` are limited to health, OIDC bootstrap, development-only mock login, and logout.
|
||||
|
||||
| Route | Access | Implementation |
|
||||
|---|---|---|
|
||||
| `GET /healthz` | public | `internal/service/health.go` |
|
||||
| `GET /readyz` | public | `internal/service/health.go` |
|
||||
| `GET /auth/login` | public-oidc | `internal/authapi/handler.go` |
|
||||
| `GET /auth/callback` | public-oidc | `internal/authapi/handler.go` |
|
||||
| `GET /auth/test-login` | development-only | `cmd/api/main.go` |
|
||||
| `POST /session/logout` | session | `cmd/api/main.go` |
|
||||
| `GET /api/v1/system/status` | view | `internal/systemstatusapi/handler.go` |
|
||||
| `GET /api/v1/system/diagnostics` | operate | `internal/systemstatusapi/handler.go` |
|
||||
| `GET /api/v1/system/metrics` | operate | `internal/observability/metrics.go` |
|
||||
| `GET /api/v1/system/backups` | admin | `internal/backupapi/handler.go` |
|
||||
| `POST /api/v1/system/backups` | admin | `internal/backupapi/handler.go` |
|
||||
| `GET /api/v1/onboarding` | view | `internal/onboardingapi/handler.go` |
|
||||
| `POST /api/v1/onboarding` | admin | `internal/onboardingapi/handler.go` |
|
||||
| `GET /api/v1/widgets/catalog` | view | `internal/widgetapi/handler.go` |
|
||||
| `POST /api/v1/widgets/preview` | edit | `internal/widgetapi/handler.go` |
|
||||
| `GET /api/v1/metrics/catalog` | view | `internal/metricsapi/handler.go` |
|
||||
| `POST /api/v1/metrics/query` | view | `internal/metricquery/handler.go` |
|
||||
| `POST /api/v1/metrics/query-range` | view | `internal/metricquery/handler.go` |
|
||||
| `POST /api/v1/metrics/inspect` | operate | `internal/metricquery/handler.go` |
|
||||
| `GET /api/v1/live` | view-websocket | `internal/live/live.go` |
|
||||
| `GET /api/v1/dashboards` | view | `internal/dashboardapi/handler.go` |
|
||||
| `POST /api/v1/dashboards` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `GET /api/v1/dashboards/{id}` | view | `internal/dashboardapi/handler.go` |
|
||||
| `PATCH /api/v1/dashboards/{id}` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `DELETE /api/v1/dashboards/{id}` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `PUT /api/v1/dashboards/{id}/document` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `POST /api/v1/dashboards/{id}/preview` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `POST /api/v1/dashboards/{id}/clone` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `GET /api/v1/dashboards/{id}/versions` | view | `internal/dashboardapi/handler.go` |
|
||||
| `GET /api/v1/dashboards/{id}/versions/{version}` | view | `internal/dashboardapi/handler.go` |
|
||||
| `POST /api/v1/dashboards/{id}/restore` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `POST /api/v1/dashboards/{id}/restore/{version}` | edit | `internal/dashboardapi/handler.go` |
|
||||
| `GET /api/v1/entities` | view | `internal/inventoryapi/handler.go` |
|
||||
| `GET /api/v1/entities/{id}` | view | `internal/inventoryapi/handler.go` |
|
||||
| `GET /api/v1/entities/{id}/relations` | view | `internal/inventoryapi/handler.go` |
|
||||
| `GET /api/v1/host` | view | `internal/hostapi/handler.go` |
|
||||
| `GET /api/v1/processes` | view | `internal/processapi/handler.go` |
|
||||
| `GET /api/v1/containers` | view | `internal/containerapi/handler.go` |
|
||||
| `GET /api/v1/containers/{id}` | view | `internal/containerapi/handler.go` |
|
||||
| `GET /api/v1/applications` | view | `internal/applicationapi/handler.go` |
|
||||
| `GET /api/v1/events` | view | `internal/eventapi/handler.go` |
|
||||
| `GET /api/v1/applications/{id}` | view | `internal/applicationapi/handler.go` |
|
||||
| `GET /api/v1/array` | view | `internal/arrayapi/handler.go` |
|
||||
| `GET /api/v1/disks` | view | `internal/diskapi/handler.go` |
|
||||
| `GET /api/v1/disks/{id}` | view | `internal/diskapi/handler.go` |
|
||||
| `GET /api/v1/pools` | view | `internal/poolapi/handler.go` |
|
||||
| `GET /api/v1/pools/{id}` | view | `internal/poolapi/handler.go` |
|
||||
| `GET /api/v1/shares` | view | `internal/shareapi/handler.go` |
|
||||
| `GET /api/v1/shares/{id}` | view | `internal/shareapi/handler.go` |
|
||||
| `GET /api/v1/forecasts` | view | `internal/forecastapi/handler.go` |
|
||||
| `GET /api/v1/services` | view | `internal/serviceapi/handler.go` |
|
||||
| `GET /api/v1/services/{id}` | view | `internal/serviceapi/handler.go` |
|
||||
| `GET /api/v1/services/{id}/history` | view | `internal/serviceapi/handler.go` |
|
||||
| `GET /api/v1/services/{id}/dependencies` | view | `internal/serviceapi/handler.go` |
|
||||
| `GET /api/v1/topology` | view | `internal/serviceapi/handler.go` |
|
||||
| `GET /api/v1/network` | view | `internal/networkapi/handler.go` |
|
||||
| `GET /api/v1/reverse-proxy` | view | `internal/reverseproxyapi/handler.go` |
|
||||
| `GET /api/v1/alert-rules` | view | `internal/alertapi/handler.go` |
|
||||
| `POST /api/v1/alert-rules` | edit | `internal/alertapi/handler.go` |
|
||||
| `GET /api/v1/alert-rules/{id}` | view | `internal/alertapi/handler.go` |
|
||||
| `PUT /api/v1/alert-rules/{id}` | edit | `internal/alertapi/handler.go` |
|
||||
| `GET /api/v1/alert-rules/{id}/versions` | view | `internal/alertapi/handler.go` |
|
||||
| `POST /api/v1/alert-rules/{id}/test` | edit | `internal/alertapi/handler.go` |
|
||||
| `POST /api/v1/alert-rules/{id}/enable` | edit | `internal/alertapi/handler.go` |
|
||||
| `POST /api/v1/alert-rules/{id}/disable` | edit | `internal/alertapi/handler.go` |
|
||||
| `GET /api/v1/alerts` | view | `internal/alertopsapi/handler.go` |
|
||||
| `GET /api/v1/alerts/{id}` | view | `internal/alertopsapi/handler.go` |
|
||||
| `POST /api/v1/alerts/{id}/acknowledge` | operate | `internal/alertopsapi/handler.go` |
|
||||
| `POST /api/v1/alerts/{id}/unacknowledge` | operate | `internal/alertopsapi/handler.go` |
|
||||
| `GET /api/v1/alert-silences` | view | `internal/alertcontrolapi/handler.go` |
|
||||
| `POST /api/v1/alert-silences` | operate | `internal/alertcontrolapi/handler.go` |
|
||||
| `POST /api/v1/alert-silences/preview` | view | `internal/alertcontrolapi/handler.go` |
|
||||
| `POST /api/v1/alert-silences/{id}/revoke` | operate | `internal/alertcontrolapi/handler.go` |
|
||||
| `GET /api/v1/maintenance-windows` | view | `internal/alertcontrolapi/handler.go` |
|
||||
| `POST /api/v1/maintenance-windows` | operate | `internal/alertcontrolapi/handler.go` |
|
||||
| `POST /api/v1/maintenance-windows/preview` | view | `internal/alertcontrolapi/handler.go` |
|
||||
| `POST /api/v1/maintenance-windows/{id}/revoke` | operate | `internal/alertcontrolapi/handler.go` |
|
||||
| `GET /api/v1/incidents` | view | `internal/incidentapi/handler.go` |
|
||||
| `GET /api/v1/incidents/{id}` | view | `internal/incidentapi/handler.go` |
|
||||
| `PATCH /api/v1/incidents/{id}` | operate | `internal/incidentapi/handler.go` |
|
||||
| `GET /api/v1/incidents/{id}/notes` | view | `internal/incidentapi/handler.go` |
|
||||
| `POST /api/v1/incidents/{id}/notes` | operate | `internal/incidentapi/handler.go` |
|
||||
| `POST /api/v1/incidents/{id}/alerts/{alertId}` | operate | `internal/incidentapi/handler.go` |
|
||||
| `DELETE /api/v1/incidents/{id}/alerts/{alertId}` | operate | `internal/incidentapi/handler.go` |
|
||||
|
||||
## 4. Endpoint behavior
|
||||
|
||||
### Session/user
|
||||
|
||||
OIDC login and callback are the only public authentication routes. The browser session is server-side and intentionally has no introspection or preference endpoint in v1. Logout clears the opaque session cookie. Sessions use a bounded sliding idle lifetime: a successful authenticated request renews the HttpOnly cookie after half of the idle TTL has elapsed, while a separate absolute TTL always requires a new OIDC login. The defaults are eight hours idle and seven days absolute; production rejects an absolute TTL below 24 hours because an authenticated wallboard is a supported 24-hour workload. The mock test-login route returns `404` unless both the environment is non-production and mock authentication is configured.
|
||||
|
||||
### Health
|
||||
|
||||
Liveness is served at `/healthz`; dependency-aware readiness is served at `/readyz`. Detailed authenticated state, safe diagnostics, internal metrics and backup operations live below `/api/v1/system/*` with the permissions in the route inventory.
|
||||
|
||||
### Dashboards
|
||||
|
||||
Save accepts a complete validated dashboard document plus expected revision. Metadata update, archive, preview, clone, version reads and restore routes are listed above. Restore accepts the version in the path or request body and always creates a new current version. Import/export are UI-side validated document transfers in v1 and are not API routes. Partial widget edits may exist internally, but the user save is atomic.
|
||||
|
||||
### Widget catalog
|
||||
|
||||
The catalog is built from the same validated registry used by dashboard documents. Preview is edit-protected, bounded, does not persist data and applies the same widget validation as a saved document.
|
||||
|
||||
### Metrics
|
||||
|
||||
`POST /api/v1/metrics/inspect` accepts the same bounded semantic query request as the range endpoint, requires operate permission, and returns the resolved semantic metric, generated approved PromQL, estimated cost and applied series/point limits. It never executes the source query. The `inspector` field on a metric response is omitted for viewers and is safe/redacted for operators. Arbitrary label enumeration is deliberately absent in v1; callers use the allowlisted semantic catalog.
|
||||
|
||||
Semantic query example:
|
||||
|
||||
```json
|
||||
{
|
||||
"metric": "container.cpu.utilization",
|
||||
"scope": {"containerId": "uuid"},
|
||||
"range": {"from": "...", "to": "...", "stepSeconds": 15},
|
||||
"aggregation": "avg",
|
||||
"groupBy": ["container"],
|
||||
"maxSeries": 20,
|
||||
"maxPoints": 4000
|
||||
}
|
||||
```
|
||||
|
||||
### Host
|
||||
|
||||
GET /host — authenticated viewer, bounded current host snapshot
|
||||
|
||||
The response contains host identity, uptime/boot time, CPU aggregate and bounded per-core values, load averages, memory totals/percentages, bounded filesystems/inodes, network counters/errors/drops, time synchronization and source provenance. Optional hardware contains capability states, stable temperature/fan identities and bounded GPU values. Missing optional capabilities are disabled rather than errors. source.freshness=stale or unavailable telemetry makes status.state=unknown; the endpoint never returns fabricated healthy values.
|
||||
### Inventory
|
||||
|
||||
Entity list, detail and relation reads expose the persisted inventory. Discovery scheduling and datasource configuration remain worker/runtime configuration, not public API operations in v1. No endpoint mutates host/container state.
|
||||
|
||||
`GET /api/v1/entities` accepts bounded `limit` (1–100), opaque `after`, `q`, `type`, `status` and `order=asc|desc` parameters. The stable keyset is effective display name plus entity ID. Each summary exposes effective display/status values, fact/override/relation/source counts and stale-fact count; manual overrides therefore remain visible in both filtering and presentation.
|
||||
|
||||
`GET /api/v1/entities/{id}` returns aliases, all source-owned facts, overrides, effective values and incoming/outgoing relation peers. Effective values select a manual/system override before the best non-stale discovered fact while retaining every fact and its source, observation time, confidence and validity. Stale facts, tombstoned peers and absent relations are explicit states. `GET /api/v1/entities/{id}/relations` returns the same deterministic relation projection only. All three endpoints are authenticated, read-only and return safe problem details without database errors.
|
||||
|
||||
### Processes
|
||||
|
||||
GET /processes?limit=25&sort=cpu&q=...&container=...&after=... — authenticated viewer, bounded read-only process page
|
||||
|
||||
The response exposes only PID, normalized process name, state, runtime, CPU/memory counters and optional known container association. Command-line arguments, environment, working directory and control actions are not part of the contract. Supported sort modes are cpu and memory; bounded name/container filters are applied before the cursor so `total` and pagination remain truthful.
|
||||
### Containers
|
||||
|
||||
GET /containers?limit=25&q=...&state=...&health=...&sort=name&after=... — authenticated viewer, bounded read-only container page
|
||||
GET /containers/{id} — authenticated viewer, bounded read-only container detail
|
||||
|
||||
The response preserves runtime state separately from health, includes uptime, restart/exit counters, image and digest, resource counters, ports, volumes, networks, project and bounded labels where available. Search, runtime-state and health filters are applied before cursor pagination; supported deterministic sort modes are name, CPU, memory and state. intentionalStop is explicit and must not be inferred as healthy. Source freshness and provenance are retained; unavailable sources return an Unknown snapshot. There are no start, stop, restart, delete or exec endpoints.
|
||||
### Applications
|
||||
|
||||
GET /applications — authenticated viewer, bounded application snapshot
|
||||
GET /applications/{id} — authenticated viewer, one application with components and contributing reasons
|
||||
|
||||
Applications are derived from discovered components and persisted user overrides. Aggregation keeps critical and optional component roles explicit, treats a service-down component as unhealthy even when its container is running, and returns reason codes with component IDs. No application endpoint changes infrastructure state.
|
||||
### Events
|
||||
|
||||
There is no standalone event route yet. Bounded events are projected in their owning domains (for example service history, alert occurrences and incident notes). The contract does not advertise an unimplemented aggregate event store.
|
||||
|
||||
### Alerts
|
||||
|
||||
```text
|
||||
GET /alerts
|
||||
GET /alerts/{id}
|
||||
POST /alerts/{id}/acknowledge
|
||||
POST /alerts/{id}/unacknowledge
|
||||
POST /alerts/{id}/silence
|
||||
DELETE /alerts/{id}/silence
|
||||
|
||||
GET /alert-rules
|
||||
POST /alert-rules
|
||||
GET /alert-rules/{id}
|
||||
GET /alert-rules/{id}/versions
|
||||
PUT /alert-rules/{id}
|
||||
POST /alert-rules/{id}/test
|
||||
POST /alert-rules/{id}/enable
|
||||
POST /alert-rules/{id}/disable
|
||||
```
|
||||
|
||||
### Alert-rule contract (M8-01)
|
||||
|
||||
Alert-rule documents conform to `specs/alert-rule.schema.json`. The condition references only a semantic metric name from the server-side metric catalog; arbitrary PromQL, query templates, interpolation, and unbounded scope values are rejected. Reads require `view`; create, update, test, enable and disable require `edit`.
|
||||
|
||||
`PUT /alert-rules/{id}` requires the current revision in `If-Match` (or the `revision` query parameter). A stale revision returns `409 REVISION_CONFLICT`. Every accepted document change creates an immutable version; `GET /alert-rules/{id}/versions` returns versions newest first. Enabling and disabling increment the revision and create an audit event.
|
||||
|
||||
`POST /alert-rules/{id}/test` accepts an optional rule document and bounded sample value. It evaluates in memory and returns a preview only; it does not create versions, mutate enabled state, evaluate live sources, or write audit events. A rule condition may define a lower (for high-threshold rules) or higher (for low-threshold rules) `recoveryThreshold`; pending and resolve durations are evaluated at UTC observation timestamps. `cooldownSeconds` suppresses duplicate firing notifications after recovery without deleting occurrence history.
|
||||
|
||||
### Incidents
|
||||
|
||||
Incidents are created by the correlation pipeline in v1. The API exposes bounded list/detail, ownership update, note reads/writes and explicit alert association/disassociation. There is no manual incident-creation route.
|
||||
|
||||
### Probes/services
|
||||
|
||||
Service list, detail, history, dependency and topology routes are read-only. Probe configuration is owned by the worker/runtime boundary in v1: service detail may expose a bounded safe probe summary, but targets, credentials and mutation/test endpoints are deliberately absent. This prevents the observability API from becoming a general network-request primitive.
|
||||
|
||||
### Maintenance
|
||||
|
||||
Silences and maintenance windows support bounded list, create, preview and revision-checked revoke operations. Records are immutable evidence after creation; revoke replaces generic update/delete semantics.
|
||||
|
||||
### Operations
|
||||
|
||||
Administrative backup list/create is exposed at `/api/v1/system/backups`. Safe diagnostics and internal metrics use `/api/v1/system/diagnostics` and `/api/v1/system/metrics`. Restore, backup verification and general audit-log browsing remain CLI/operations procedures until their controlled workflows are implemented; they are not advertised API routes.
|
||||
|
||||
## 5. WebSocket
|
||||
|
||||
Endpoint:
|
||||
|
||||
```text
|
||||
GET /api/v1/live
|
||||
```
|
||||
|
||||
Requirements:
|
||||
- authenticated before upgrade;
|
||||
- same-origin/allowed-origin check;
|
||||
- role-aware subscription authorization;
|
||||
- message size/rate limits;
|
||||
- heartbeat and idle timeout;
|
||||
- per-session subscription/series limits;
|
||||
- sequence numbers and resync;
|
||||
- explicit unsubscribe;
|
||||
- no arbitrary backend URL/query.
|
||||
|
||||
Message schemas are in `specs/live-message.schema.json`.
|
||||
|
||||
## 6. Idempotency
|
||||
|
||||
Use idempotency keys for:
|
||||
- acknowledgement;
|
||||
- incident note/create where retries matter;
|
||||
- notification delivery;
|
||||
- backup start;
|
||||
- manual discovery.
|
||||
|
||||
## 7. Compatibility
|
||||
|
||||
- Breaking changes require `/api/v2` or a documented migration.
|
||||
- Schema versions are explicit in dashboard/export/live messages.
|
||||
- Old dashboard versions are migrated through tested deterministic migrations.
|
||||
|
||||
### Array and parity
|
||||
|
||||
`GET /array` is an authenticated viewer endpoint for one bounded, read-only array snapshot. The response contains source freshness/provenance, operational/degraded/missing/unknown state, parity presence/state/errors, bounded data/parity members, current check progress/speed/errors and bounded check history. Stale or unavailable storage source data is `Unknown`; there are no array start, stop, check or correct endpoints.
|
||||
### Disks
|
||||
|
||||
`GET /disks?limit=...` and `GET /disks/{id}` are authenticated viewer endpoints for bounded, read-only disk inventory and detail. Responses preserve source freshness/provenance and expose canonical stable source IDs, role, availability state, model, privacy-aware serial display, filesystem, size, used/free/utilization, separate `capacitySeverity` and `thermalSeverity`, and optional inode capacity. Missing-disk history remains visible; raw serials are never returned. Exact duplicate source observations are idempotently collapsed, while conflicting observations for one canonical ID fail closed. No format, mount, unmount, remove or repair endpoint exists.
|
||||
### SMART detail
|
||||
|
||||
Disk detail may include a bounded SMART object with capability state (`available` or `unknown`), generic overall result, freshness timestamp, explicit critical attributes/reasons and self-test result/age. Pending, reallocated, offline-uncorrectable, CRC and wear signals remain separate attributes; a generic `passed` result does not suppress an attention reason. SMART has no write or self-test trigger endpoint.
|
||||
### Disk telemetry
|
||||
|
||||
Disk detail may include bounded performance samples (read/write bytes per second, IOPS and latency), temperature state/value and optional spin capability. Unsupported latency or spin is explicit rather than fabricated; history is bounded and deterministically ordered. Temperature status applies warning/critical/recovery hysteresis, while live/history values remain read-only.
|
||||
|
||||
### Pools
|
||||
|
||||
`GET /pools?limit=...` and `GET /pools/{id}` are authenticated viewer endpoints for bounded, read-only cache/Btrfs/ZFS pool snapshots and detail. Responses preserve source freshness/provenance and expose filesystem, device-health state, usable/used/free capacity, separate `capacitySeverity`, profile/redundancy, deduplicated bounded members, filesystem errors, scrub state/history and explicit capability states. Capacity pressure never rewrites device health. Stale pool data maps every pool state and severity to `Unknown`; no scrub, repair, balance, mount or pool-control endpoint exists.
|
||||
### Shares
|
||||
|
||||
`GET /shares?limit=...` and `GET /shares/{id}` are authenticated viewer endpoints for bounded, read-only share usage and growth. Responses expose configured storage policy, source-owned size observations, cache/pool placements, bounded growth history and scan-plan cost metadata. Size scans are rate-limited/cached through the normalized scan plan; stale or unavailable observations are `Unknown`. No path, filename, directory listing or file-content field is returned.
|
||||
|
||||
### Storage map and heatmap views
|
||||
|
||||
The storage view composes authenticated read-only array, pool and disk snapshots in the browser. Array membership and disk telemetry with the same canonical physical ID form one disk node; a pool aggregate remains a separate meaningful role. Every node states availability/device health, capacity and temperature independently, while the strongest trustworthy severity determines its visual accent. Map nodes retain entity links and text/icon state; the temperature heatmap is bounded and always includes a table/text alternative. No new control endpoint or raw source credential is exposed.
|
||||
### Capacity forecasts
|
||||
|
||||
`GET /api/v1/forecasts` is an authenticated, read-only endpoint. Each bounded assessment exposes the entity identity, method, historical window, point count, confidence and reason. `qualifiedCount` counts only medium/high forecasts and is intentionally distinct from `items.length`, which may include insufficient or stale assessments. A projected date is only returned for qualified (`medium` or `high`) median-rate forecasts; disabled policy, insufficient data, bulk-import detection, irregular intervals and unknown/reached capacity remain explicit states without a false date. When no real capacity entity exists, `items` is empty and the top-level `reason` explains the Unknown state; the API never synthesizes a blank `none`/0-byte entity. The response contains no write or capacity-management operation.
|
||||
### M7 service and probe model
|
||||
|
||||
Service and probe configuration is versioned and archive-oriented. Services, endpoints and probes expose revisions for optimistic concurrency; active names are unique per parent while archived configurations remain available for historical result retention. Probe results and certificate observations are append-only by probe/service and UTC observation time. Service dependencies retain source and confidence, and service permissions map existing Pulse roles to view/operate/edit/admin without storing probe credentials.
|
||||
### M7 probe target safety
|
||||
|
||||
Probe target policy is evaluated server-side before every request and again for redirects. HTTP/HTTPS GET/HEAD are the default methods; request headers, response headers/body, timeout and redirect count are bounded. Loopback, link-local, metadata, multicast and unspecified addresses are blocked, and private/LAN targets require an explicit CIDR allowlist. DNS answers are revalidated at dial time so a later blocked answer cannot be used for DNS rebinding. Policy changes emit audit events with redacted bounded policy metadata.
|
||||
### M7 probe execution
|
||||
|
||||
Probe execution supports HTTP/HTTPS, TCP, DNS and TLS within the server-side network policy. HTTP status and bounded keyword/JSON assertions are represented as `up`, `down`, `degraded` or `unknown`; redirect following is opt-in and each hop is revalidated. TLS results retain bounded expiry, issuer, subject and hostname-validity facts. ICMP is capability-aware and returns `Unknown` with an `unsupported` class when the runtime cannot provide it. Probe configuration may contain only a secret reference; credentials are never returned by the API or included in result errors.
|
||||
### M7 service status and history
|
||||
|
||||
`GET /api/v1/services?limit=...` and `GET /api/v1/services/{id}` are authenticated, read-only service projections. `GET /api/v1/services/{id}/history?limit=...` returns bounded probe history. The projection derives service state from probe results independently of container state, preserves last success/failure and latency, calculates bounded availability, and maps absent or stale probe data to `unknown`. Results, histories and transition events are deterministically ordered and capped. Service detail may include a bounded, read-only probe configuration summary (id, name, type, interval, timeout, enabled state and safe TLS/redirect flags); target payloads, content assertions, secret references and credentials are never returned.
|
||||
|
||||
Service snapshots expose separate `capabilityState` (`available`, `unavailable`, `unsupported`) and `configurationState` (`configured`, `not_configured`, `unknown`). A catalog without services is `available/not_configured`; an unreadable repository is `unavailable/unknown`. Per-service reasons distinguish no configured probe, disabled probes, pending results and stale results. These states are also propagated into topology and each network-health scope so an empty configuration is never presented as a generic source failure.
|
||||
|
||||
|
||||
`GET /api/v1/services/{id}/dependencies?limit=...` is an authenticated, read-only bounded relation projection. Dependencies are sorted deterministically and expose source, confidence and confirmed/inferred state; dependency writes remain server-side and audit-backed.
|
||||
|
||||
`GET /api/v1/topology?limit=...` is an authenticated, read-only graph projection bounded to at most 100 nodes and 200 edges by default. Nodes expose service status and whether they are present in the current snapshot; missing nodes remain explicit `unknown`. Edges combine active service dependencies with active `depends_on`, `backs` and `exposes` inventory relations when both inventory entities map to active services. Every edge preserves relation type, source, confidence and confirmed/inferred state, is sorted deterministically, and never asserts causality. The endpoint has no write operation.
|
||||
|
||||
`GET /api/v1/network` is an authenticated, read-only network health projection. It separates internal interface health, gateway, DNS and internet scopes; stale/unavailable host telemetry maps interfaces and internal health to `unknown`, while independent internet or DNS failures remain independently visible. Interface RX/TX bytes, errors and drops are bounded and certificates are exposed as safe observed facts without probe credentials.
|
||||
|
||||
GET /api/v1/reverse-proxy is an authenticated, read-only projection of optional reverse-proxy host mappings. The connector reports a versioned capability and source ownership, returns only bounded host/URL-to-service mappings, and exposes a disabled or unavailable state when no connector is configured. Connector credentials remain external references and are never serialized. Duplicate host mappings with different targets are rejected unless a user-confirmed override explicitly selects the effective service; no proxy configuration mutation endpoint exists. Topology may represent enabled mappings as reverse_proxy nodes with exposes edges that retain source and confirmed/inferred provenance.
|
||||
|
||||
### Alert silences and maintenance windows (M8-06)
|
||||
|
||||
~~~text
|
||||
GET /alert-silences
|
||||
POST /alert-silences
|
||||
POST /alert-silences/preview
|
||||
POST /alert-silences/{id}/revoke
|
||||
GET /maintenance-windows
|
||||
POST /maintenance-windows
|
||||
POST /maintenance-windows/preview
|
||||
POST /maintenance-windows/{id}/revoke
|
||||
~~~
|
||||
|
||||
Reads and matcher previews require view; create and revoke require operate. Silences and maintenance windows require a non-empty reason, bounded matcher and selector fields, and an expiry or end time. Revocation uses If-Match or the revision query parameter. Controls never delete alert history. Every create and revoke is audited; expiry is idempotently marked by the API expiry job.
|
||||
|
||||
|
||||
### Alert operations (M8-07)
|
||||
|
||||
GET /alerts returns a bounded deterministic list of non-inactive alert instances; the state query filter is allow-listed. GET /alerts/{id} returns rule/entity metadata, current state, revision and bounded occurrence history.
|
||||
|
||||
POST /alerts/{id}/acknowledge and POST /alerts/{id}/unacknowledge require operate permission, If-Match or a positive revision query parameter, and a bounded Idempotency-Key or evaluationKey. A replay with the same key is idempotent even when the original revision is stale. Acknowledge changes firing or pending to acknowledged; unacknowledge changes acknowledged back to firing. Resolved state remains a persisted evaluator result and is not rewritten by a read or duplicate operation. Accepted and idempotent operations are audited.
|
||||
|
||||
### Notification delivery contract
|
||||
|
||||
Notification delivery is an internal worker contract in M8-08. Public alert operations create no direct channel side effect: an event is first represented by a bounded outbox record with an idempotency key. Workers claim and complete records transactionally, and delivery history remains queryable for audit. Channel test actions are explicitly bounded and rate limited; no secret value is returned by the channel or delivery model.
|
||||
### M8-09 incident contract
|
||||
|
||||
`GET /incidents` and `GET /incidents/{id}` are bounded viewer operations. `POST /incidents/{id}/alerts/{alertId}` and `DELETE /incidents/{id}/alerts/{alertId}` require operate permission and create audit events. Manual association requests require bounded rationale and confidence; the response exposes whether the association was idempotent. Incident operations remain observational and do not remediate alerts, services, or infrastructure.
|
||||
|
||||
### M8-10 incident detail, ownership and notes
|
||||
|
||||
`GET /incidents/{id}` returns bounded incident detail, associations and ordered notes. `PATCH /incidents/{id}` requires operate permission and updates only the owner user ID with the current positive revision and returns a conflict for stale writes. `GET /incidents/{id}/notes` is viewer-readable; `POST /incidents/{id}/notes` requires operate permission and stores the authenticated actor as author. Note bodies are bounded plain text: tag-shaped markup is removed, whitespace is normalized, and empty or oversized values are rejected. Incident UI communicates correlation confidence and rationale as uncertainty-aware evidence; Pulse provides no remediation action.
|
||||
|
||||
## Onboarding
|
||||
|
||||
`GET /api/v1/onboarding` is an authenticated viewer endpoint. It returns only bounded readiness states for the database, Prometheus, read-only Unraid source, authentication, alert defaults and the default dashboard. It never returns datasource URLs, API tokens, OIDC client secrets or other secret-bearing configuration.
|
||||
|
||||
`POST /api/v1/onboarding` requires the administrator permission and accepts only `dashboard` and `rules` choices with values `default` or `skip`. Completion is additive and resumable: it records progress in `system_settings`, installs the validated default dashboard only when the stable slug is absent, and preserves existing dashboards and alert rules. It exposes no host, Docker, storage, database-control or remediation operation.
|
||||
|
||||
## Authentication bootstrap
|
||||
|
||||
`GET /auth/login` and `GET /auth/callback` are the only unauthenticated non-health endpoints. They are browser redirect endpoints outside `/api/v1`; they return `302` rather than JSON, except for a problem-details `405` on a non-`GET` method.
|
||||
|
||||
`GET /auth/login` starts the Authentik OIDC authorization code flow with state, nonce and PKCE `S256`. State, nonce, PKCE verifier and the requested post-login path are stored server-side in a bounded, TTL-expired flow store; the browser receives only an opaque flow identifier in a short-lived `HttpOnly`, `SameSite=Lax` cookie that is `Secure` in production. The optional `redirect` query parameter is accepted only as an in-app path with a single leading slash: protocol-relative, absolute, backslash-prefixed, control-character and oversized values fall back to `/`. The response redirects to the provider authorization URL.
|
||||
|
||||
`GET /auth/callback` consumes the flow identifier exactly once, deleting the stored flow before validating anything. It compares state in constant time, exchanges the code with the PKCE verifier, verifies the ID token issuer, audience, signature and nonce, maps the configured role claim to a Pulse role and only then issues the session cookie and redirects to the validated post-login path. An unknown, replayed or expired flow, a state or nonce mismatch, a provider `error` response, a failed exchange, a missing ID token, an unmapped role and a failed audit or session write all issue no session and redirect to a fixed in-app error route with one of a closed set of reason codes (`invalid_request`, `expired`, `denied`, `provider_unavailable`, `not_authorized`, `unavailable`). Provider-supplied error text, authorization codes, tokens and PKCE verifiers are never reflected into a response or logged.
|
||||
@@ -0,0 +1,371 @@
|
||||
# Data model
|
||||
|
||||
## 1. Principles
|
||||
|
||||
- Stable internal UUIDs; external IDs are source-scoped aliases.
|
||||
- Source facts and user overrides are stored separately.
|
||||
- Soft deletion/tombstones preserve history and reconciliation.
|
||||
- Important configuration is versioned.
|
||||
- Alert state changes are transactional.
|
||||
- Time is stored in UTC; locale/time zone applied at presentation.
|
||||
- JSONB is used for bounded extensible attributes, not as a substitute for core relational structure.
|
||||
|
||||
## 2. Identity and access
|
||||
|
||||
### `users`
|
||||
- id
|
||||
- external_subject
|
||||
- display_name
|
||||
- email
|
||||
- status
|
||||
- created_at / updated_at / last_login_at
|
||||
|
||||
### `roles`, `user_roles`
|
||||
Roles: viewer, operator, editor, administrator.
|
||||
|
||||
### `sessions` or external session metadata
|
||||
Only if required by the chosen auth implementation. Avoid storing OIDC tokens unnecessarily.
|
||||
|
||||
## 3. Datasources
|
||||
|
||||
### `data_sources`
|
||||
- id
|
||||
- type
|
||||
- name
|
||||
- enabled
|
||||
- configuration reference
|
||||
- capability document
|
||||
- health_state
|
||||
- last_success_at
|
||||
- last_error_code/message_redacted
|
||||
- freshness policy
|
||||
- created/updated
|
||||
|
||||
### `collectors`
|
||||
- id
|
||||
- datasource_id
|
||||
- kind
|
||||
- version
|
||||
- heartbeat
|
||||
- capabilities
|
||||
- status
|
||||
|
||||
## 4. Inventory
|
||||
|
||||
### `entities`
|
||||
- id UUID
|
||||
- entity_type
|
||||
- canonical_name
|
||||
- display_name
|
||||
- status
|
||||
- status_reasons JSONB
|
||||
- first_seen_at
|
||||
- last_seen_at
|
||||
- tombstoned_at
|
||||
- attributes JSONB
|
||||
|
||||
### `entity_aliases`
|
||||
- entity_id
|
||||
- source_id
|
||||
- external_type
|
||||
- external_id
|
||||
- unique(source_id, external_type, external_id)
|
||||
|
||||
### Container identity and recreation
|
||||
|
||||
Container runtime IDs are source-scoped aliases, not logical identity by themselves. A live runtime ID always matches its exact source/runtime alias. When a runtime ID changes, the same logical entity may be reused only when both the source-scoped compose project and compose service are present and uniquely identify one prior active alias. A container name, image, or digest alone is insufficient evidence and creates a new logical entity. The old runtime alias remains in history as inactive, the new alias points to the same logical entity, and recreation events keep that logical entity as events.entity_id while retaining previous/current runtime IDs in bounded attributes.
|
||||
### `entity_facts`
|
||||
- entity_id
|
||||
- field_name
|
||||
- source_id
|
||||
- value JSONB
|
||||
- observed_at
|
||||
- confidence
|
||||
- valid_until
|
||||
|
||||
### `entity_overrides`
|
||||
- entity_id
|
||||
- field_name
|
||||
- value JSONB
|
||||
- user_id
|
||||
- updated_at
|
||||
|
||||
### `entity_relations`
|
||||
- id
|
||||
- source_entity_id
|
||||
- relation_type
|
||||
- target_entity_id
|
||||
- source_id
|
||||
- confidence
|
||||
- confirmed
|
||||
- first_seen/last_seen/tombstoned
|
||||
|
||||
### applications
|
||||
|
||||
- logical application ID and display name
|
||||
- discovered component references
|
||||
- critical/optional component policy
|
||||
- aggregate status and bounded reason list
|
||||
- user override metadata kept separate from discovery grouping
|
||||
## 5. Metrics catalog
|
||||
|
||||
### `metric_definitions`
|
||||
- id
|
||||
- semantic_name unique
|
||||
- version
|
||||
- description
|
||||
- unit
|
||||
- value_kind
|
||||
- source_kind
|
||||
- query_template
|
||||
- label contract
|
||||
- limits
|
||||
- allowed visualizations
|
||||
- default transformations
|
||||
- status policy reference
|
||||
- enabled
|
||||
|
||||
### `metric_bindings`
|
||||
Maps semantic metric to datasource/exporter/capability variants.
|
||||
|
||||
Metric samples remain in Prometheus.
|
||||
|
||||
## 6. Dashboards
|
||||
|
||||
### `dashboards`
|
||||
- id
|
||||
- slug
|
||||
- name
|
||||
- description
|
||||
- owner_user_id nullable
|
||||
- scope personal/shared/system
|
||||
- archived_at
|
||||
- current_version_id
|
||||
- revision (optimistic concurrency token)
|
||||
- created/updated
|
||||
|
||||
### `dashboard_versions`
|
||||
- id
|
||||
- dashboard_id
|
||||
- version_number
|
||||
- schema_version
|
||||
- document JSONB
|
||||
- change_summary
|
||||
- created_by
|
||||
- created_at
|
||||
|
||||
Dashboard versions are immutable. Dashboard writes lock the current row, compare revision, and commit the new version/current pointer atomically. The API stores only bounded revision metadata in audit diffs; full dashboard documents are not copied into audit events.
|
||||
|
||||
A dashboard version contains variables, widget instances, behavior and layout documents. Save as one atomic version to avoid partial layout/config updates.
|
||||
|
||||
Optional normalized indexes may extract widget type/entity references for search/impact analysis.
|
||||
|
||||
## 7. Events
|
||||
|
||||
### `events`
|
||||
- id
|
||||
- event_type
|
||||
- severity
|
||||
- entity_id nullable
|
||||
- source_id
|
||||
- occurred_at
|
||||
- received_at
|
||||
- dedup_key
|
||||
- summary
|
||||
- attributes JSONB bounded/redacted
|
||||
- correlation_id
|
||||
- unique(source_id, dedup_key, occurred_at bucket) as appropriate
|
||||
|
||||
## 8. Alerts
|
||||
|
||||
### `alert_rules`
|
||||
- id
|
||||
- name
|
||||
- enabled
|
||||
- severity
|
||||
- evaluator type/config
|
||||
- scope selector
|
||||
- pending duration
|
||||
- resolve duration
|
||||
- cooldown duration for repeated firing notifications
|
||||
- hysteresis config
|
||||
- grouping labels
|
||||
- suppression/dependency policy
|
||||
- current_version_id
|
||||
- revision (optimistic concurrency token)
|
||||
- created/updated
|
||||
|
||||
### `alert_rule_versions`
|
||||
Immutable rule documents and audit metadata.
|
||||
|
||||
### `alert_instances`
|
||||
Stable entity/rule combination:
|
||||
- id
|
||||
- rule_id
|
||||
- fingerprint
|
||||
- entity_id
|
||||
- current_state
|
||||
- active_since
|
||||
- last_evaluated_at
|
||||
- last_value
|
||||
- reason
|
||||
- acknowledged_by/at
|
||||
- cooldown_until
|
||||
- silenced_until
|
||||
- version for optimistic concurrency
|
||||
|
||||
### `alert_occurrences`
|
||||
Immutable transitions/evaluation outcomes relevant to history.
|
||||
|
||||
## 9. Incidents
|
||||
|
||||
### `incidents`
|
||||
- id
|
||||
- title
|
||||
- summary
|
||||
- severity
|
||||
- status
|
||||
- started_at
|
||||
- resolved_at
|
||||
- owner_user_id
|
||||
- correlation_method
|
||||
- confidence
|
||||
- created/updated
|
||||
- version
|
||||
|
||||
### `incident_alerts`, `incident_entities`
|
||||
Many-to-many links with rationale.
|
||||
|
||||
### `incident_notes`
|
||||
- incident_id
|
||||
- author
|
||||
- body sanitized
|
||||
- created_at
|
||||
|
||||
## 10. Maintenance and notifications
|
||||
|
||||
### `maintenance_windows`
|
||||
- id
|
||||
- name
|
||||
- selector
|
||||
- start/end or recurrence
|
||||
- suppress notifications/state policy
|
||||
- creator/audit
|
||||
|
||||
### `notification_channels`
|
||||
Encrypted secret references and non-secret config.
|
||||
|
||||
### `notification_deliveries`
|
||||
- occurrence/incident
|
||||
- channel
|
||||
- status
|
||||
- attempts
|
||||
- last_error_redacted
|
||||
- timestamps
|
||||
- idempotency_key
|
||||
|
||||
## 11. Audit
|
||||
|
||||
### `audit_events`
|
||||
- actor/user/service
|
||||
- action
|
||||
- resource type/id
|
||||
- result
|
||||
- occurred_at
|
||||
- correlation_id
|
||||
- source IP/session metadata where appropriate
|
||||
- before/after bounded diff with secret fields excluded
|
||||
|
||||
## 12. Operations
|
||||
|
||||
### `job_runs`
|
||||
- job type/key
|
||||
- scheduled/started/completed
|
||||
- status
|
||||
- counts
|
||||
- error code
|
||||
- correlation ID
|
||||
- lease owner and lease expiry for coordinated worker execution
|
||||
|
||||
### `schema_migrations`
|
||||
Managed by migration tool.
|
||||
|
||||
### `system_settings`
|
||||
Typed/versioned settings; no plaintext secrets.
|
||||
|
||||
## 13. Concurrency
|
||||
|
||||
Use optimistic concurrency for:
|
||||
- dashboard save;
|
||||
- alert acknowledgement/state;
|
||||
- incident edit;
|
||||
- settings.
|
||||
|
||||
Return a conflict response with current version rather than silently overwriting.
|
||||
|
||||
## 14. Retention
|
||||
|
||||
Implement partitioning/cleanup when measurements justify it.
|
||||
|
||||
Baseline:
|
||||
- inventory history/tombstones: enough for reconciliation and events;
|
||||
- events: 1 year configurable;
|
||||
- alert occurrences: 2 years configurable;
|
||||
- incidents: retained until explicit policy;
|
||||
- audit: at least 1 year configurable;
|
||||
- job runs: shorter operational retention;
|
||||
- dashboard versions: at least 180 days or fixed count plus protected versions.
|
||||
|
||||
### arrays and parity checks
|
||||
|
||||
An array snapshot preserves source-scoped provenance and UTC observed/received timestamps. Array members retain stable source IDs, role, state and bounded capacity/I/O counters. Parity state is separate from array state, and current/history parity checks retain progress, byte-per-second speed, error count and timestamps. Missing, disabled or emulated members are represented explicitly; stale source data maps to Unknown. Array transition events retain the array entity ID and bounded state/error attributes without exposing operational controls.
|
||||
### disks and capacity
|
||||
|
||||
Disk identity is source-scoped and stable from the provider ID. A disk retains role, state, model, privacy-aware serial display, filesystem and bounded size/used/free/inode metrics. Used values above capacity are rejected; free space and utilization are derived from validated bytes, and inode utilization is derived separately. Missing disks remain in current/history projections instead of being dropped, while raw serials are excluded from API output.
|
||||
### SMART facts
|
||||
|
||||
SMART is an optional disk capability with its own observed timestamp and freshness state. Normalization preserves generic overall result, bounded mapped attributes, critical/reason status and self-test result/age. Unavailable or stale SMART is Unknown and never Healthy; critical pending, reallocated, offline-uncorrectable, CRC or wear attributes remain visible even when the vendor overall result is passed.
|
||||
### disk telemetry facts
|
||||
|
||||
Disk performance history is bounded and timestamped for live/history consumers. Temperature observations retain source time and policy-derived status; recovery does not clear attention until the configured recovery threshold is crossed. Latency/spin capabilities explicitly report unsupported when the source cannot provide them.
|
||||
|
||||
### pools and scrub facts
|
||||
|
||||
Pool identity is stable within its source and retains filesystem, usable/used/free bytes, profile/redundancy, bounded member state and filesystem error facts. Btrfs and ZFS capabilities are explicit and conditional; unsupported fields are never fabricated. Scrub current state and bounded history retain progress, bytes checked, errors and UTC timestamps. Stale source data maps pool state to Unknown while preserving provenance. Transition events identify degraded/faulted/recovered pools and scrub failures without exposing controls.
|
||||
### shares and growth facts
|
||||
|
||||
Share identity is source-scoped and retains configured allocation/cache policy, source-owned used-size timestamp/state, bounded placement by pool and chronological growth points. Growth deltas and daily rates are derived from UTC observations, including negative changes, without exposing paths or file content. `ScanPlan` caps due size refreshes per run and records deferred work plus cache TTL so expensive scans remain bounded and observable.
|
||||
|
||||
### storage visualization projections
|
||||
|
||||
Storage map nodes are bounded projections of array members, pools and disks with entity links, kind, explicit state and detail text. Temperature heatmap points retain disk identity, UTC observed time, value and text status; visual cells never replace the accessible table alternative.
|
||||
### capacity forecasts
|
||||
|
||||
A forecast is a bounded projection over UTC usage observations. Its policy records enabled state, maximum window, minimum points and method. Median daily growth is used for linear projection; bulk-import/outlier and irregular-interval safeguards lower confidence and suppress `daysToCapacity`/`projectedAt`. Method, window, point count, confidence and reason are always retained so the UI cannot present an unexplained precise date.
|
||||
|
||||
`capacity_samples` preserves bounded historical capacity observations for shares, pools and disks without storing file contents. Agent snapshot writes and their samples commit in one PostgreSQL transaction. A six-hour UTC bucket and primary key on `(entity_kind, entity_id, source_id, sampled_at)` make repeated discovery runs idempotent; a delayed retry may not replace a newer observation in the same bucket. The history index `(entity_kind, entity_id, sampled_at DESC)` supports the actual bounded forecast query. Forecast reads use at most 512 points inside the configured window and combine persisted share usage with the current canonical pool capacity. Insufficient, stale or unavailable histories remain assessments with `confidence=none`; only medium/high results count as qualified forecasts.
|
||||
### services and probes
|
||||
|
||||
`services`, `service_endpoints` and `probes` represent reachable capabilities independently of container state. Configuration rows use `revision`, `updated_at` and `archived_at`; active names are unique per parent, while archived rows remain to preserve history. `probe_results` stores immutable timestamped outcomes, and `service_certificates` stores observed certificate facts. `service_dependencies` preserves source, confidence and confirmation state. `service_permissions` maps database roles to service permissions; secret references are identifiers only and never plaintext credentials.
|
||||
### service status projections
|
||||
|
||||
Service status is derived from immutable `probe_results`, not from container state. The bounded projection retains current state/reason, last result/success/failure timestamps, latest latency, availability sample counts and bounded history. A result older than the configured freshness policy becomes `unknown`; it is never silently treated as healthy. State transitions produce deterministic service events with service identity, from/to state and reason.
|
||||
M8-06 persistence adds alert_silences and maintenance_windows with bounded reason and name fields, JSON matcher or selector, UTC start and end timestamps, explicit active/expired/revoked status, creator and owner provenance, revocation and expiry timestamps, revision, expiry indexes, and deterministic listing indexes. These tables are additive and do not alter M1 tables or alert occurrence history.
|
||||
|
||||
M8-07 extends alert_occurrences with an immutable unacknowledge event type and adds an acknowledged-state index. Alert list/detail projections join rule and optional entity metadata while preserving instance revision and occurrence history.
|
||||
|
||||
### M8-08 notification persistence
|
||||
|
||||
- `notification_channels` stores only a bounded non-secret configuration and a reference to an external secret provider; the secret value is never part of the domain object or query result.
|
||||
- `notification_outbox` has a unique idempotency key, bounded event content, a lease-aware status, bounded attempts and deterministic due ordering.
|
||||
- `notification_deliveries` records each claimed attempt with a unique `(outbox_id, attempt)` key. Claim, delivery audit creation, completion and retry state changes are transactionally coordinated.
|
||||
- Channel updates use optimistic revisions. Outbox rows retain recovery events and are not deleted after successful delivery, preserving auditability.
|
||||
### M8-09 incident persistence
|
||||
|
||||
- `incidents` stores a bounded correlation key, title/summary, derived severity, lifecycle status, start/resolution timestamps, rationale method, confidence, owner and optimistic revision. Only one unresolved incident may exist for a correlation key.
|
||||
- `incident_alerts` preserves per-alert rationale/confidence and distinguishes deterministic correlation from manual association. A manual association is never overwritten by a later correlation upsert.
|
||||
- `incident_entities` keeps the affected entity set with bounded rationale/confidence and restricts entity deletion while incident evidence references it.
|
||||
|
||||
### M8-10 incident notes and ownership
|
||||
|
||||
`incident_notes` is bounded operator context attached to an incident. Bodies are normalized to plain text, stripped of tag-shaped markup and capped at 2,000 characters before persistence; notes are ordered by UTC creation time and stable ID. Notes cascade with their incident and never affect alert/entity evidence. Incident ownership is metadata updated with an optimistic revision, so stale UI writes become an explicit conflict.
|
||||
@@ -0,0 +1,248 @@
|
||||
# Security threat model
|
||||
|
||||
## 1. Assets
|
||||
|
||||
- Unraid host and storage.
|
||||
- Docker/container metadata and internal topology.
|
||||
- Prometheus metrics and labels.
|
||||
- Service URLs and availability data.
|
||||
- OIDC identities/roles.
|
||||
- Pulse configuration, alerts, incidents and audit.
|
||||
- Notification/probe credentials.
|
||||
- Database backups.
|
||||
- Server access path used by Codex during deployment.
|
||||
|
||||
## 2. Trust boundaries
|
||||
|
||||
- Browser to Pulse.
|
||||
- Pulse to Authentik.
|
||||
- Pulse to PostgreSQL.
|
||||
- Pulse to Prometheus.
|
||||
- Pulse agent to host/Unraid/Docker.
|
||||
- Probe worker to network targets.
|
||||
- Notification worker to external channels.
|
||||
- Codex workspace to production server.
|
||||
|
||||
## 3. Primary threats and controls
|
||||
|
||||
### M0 discovered deployment posture
|
||||
|
||||
The target host already runs Nginx Proxy Manager, Authentik, Grafana, and multiple Docker/Compose projects. Pulse treats all of them as external protected resources. Existing containers with broad privileges, including any Docker socket access, are not reused as a Pulse pattern. Pulse must be isolated on dedicated resources, integrate with the proxy and OIDC provider additively, and remain server-side for all Unraid/Prometheus access.
|
||||
|
||||
M0 did not identify a local Prometheus service; this is an explicit datasource uncertainty, not permission to substitute an unreviewed source. Missing or stale telemetry must map to `Unknown`. No hostname, port, network, volume, or path is trusted until the deployment task rechecks ownership and conflicts.
|
||||
|
||||
### Unrestricted Docker/host control
|
||||
|
||||
Threat: compromise of web/API leads to host root-equivalent access.
|
||||
|
||||
Controls:
|
||||
- no unrestricted socket in web/API;
|
||||
- separate agent/proxy;
|
||||
- endpoint allowlist;
|
||||
- read-only capability contract;
|
||||
- non-root API;
|
||||
- network separation;
|
||||
- architecture test of compose/mounts;
|
||||
- no mutation API in v1.
|
||||
|
||||
### SSRF from service probes
|
||||
|
||||
Threat: user config probes metadata, loopback, admin services or redirects.
|
||||
|
||||
Controls:
|
||||
- role restriction;
|
||||
- scheme/port allowlist;
|
||||
- DNS resolution validation before and after redirect;
|
||||
- block metadata/link-local/unspecified by default;
|
||||
- configurable LAN allowlist;
|
||||
- response size/time limits;
|
||||
- no arbitrary methods/body;
|
||||
- redacted logging;
|
||||
- tests for DNS rebinding/redirect escape.
|
||||
|
||||
### Query abuse
|
||||
|
||||
Threat: expensive or injection-like Prometheus queries cause outage or expose labels.
|
||||
|
||||
Controls:
|
||||
- semantic query templates;
|
||||
- bounded scope/range/series/points;
|
||||
- server-side parameterization/escaping;
|
||||
- timeout/concurrency/rate limit;
|
||||
- advanced raw query separate permission;
|
||||
- audit and query cost metrics.
|
||||
|
||||
|
||||
### Optional hardware capabilities
|
||||
|
||||
Hardware sensors and GPU telemetry are optional read-only capabilities. The adapter accepts only normalized bounded snapshots from an approved source; absent support is disabled and unsupported support remains inspectable without being treated as a host failure. API/web never gains device, mount, namespace or Docker-socket access for these values.
|
||||
### Authentication/authorization bypass
|
||||
|
||||
Controls:
|
||||
- standards-based OIDC validation;
|
||||
- issuer/audience/nonce/state/PKCE;
|
||||
- secure cookies;
|
||||
- server-side RBAC;
|
||||
- WebSocket auth/origin/subscription auth;
|
||||
- CSRF protection where cookies are used;
|
||||
- role matrix tests;
|
||||
- break-glass disabled by default.
|
||||
|
||||
### XSS and dashboard import
|
||||
|
||||
Controls:
|
||||
- no arbitrary HTML/JS widgets;
|
||||
- sanitize Markdown;
|
||||
- schema validation;
|
||||
- safe chart labels/tooltips;
|
||||
- CSP;
|
||||
- escaped event/upstream text;
|
||||
- import size and complexity limits.
|
||||
|
||||
### Secret leakage
|
||||
|
||||
Controls:
|
||||
- external secret injection;
|
||||
- encrypted-at-rest channel/probe references;
|
||||
- redaction middleware;
|
||||
- no env dumps;
|
||||
- evidence policy;
|
||||
- secret scan;
|
||||
- diagnostic bundle allowlist;
|
||||
- backups exclude plaintext or are encrypted/secured.
|
||||
|
||||
### Supply chain
|
||||
|
||||
Controls:
|
||||
- lockfiles;
|
||||
- minimal maintained dependencies;
|
||||
- provenance/SBOM where feasible;
|
||||
- vulnerability scanning;
|
||||
- pinned base images;
|
||||
- non-root runtime;
|
||||
- update policy;
|
||||
- build in CI/clean environment.
|
||||
|
||||
### Database compromise/data integrity
|
||||
|
||||
Controls:
|
||||
- isolated network;
|
||||
- dedicated credentials;
|
||||
- TLS when remote;
|
||||
- least privilege;
|
||||
- migrations/transactions;
|
||||
- backup/restore;
|
||||
- input validation;
|
||||
- audit;
|
||||
- no exposed database port unless controlled testing override.
|
||||
|
||||
### Live/WebSocket abuse
|
||||
|
||||
Controls:
|
||||
- authentication before upgrade;
|
||||
- opaque HttpOnly sessions with an eight-hour sliding idle limit and a finite,
|
||||
operator-bounded absolute limit; renewal never exposes OIDC tokens to the
|
||||
browser, while a revocable session context propagates through HTTP upgrades
|
||||
so logout, absolute expiry and request/server cancellation also close an
|
||||
already established socket and release its subscriptions;
|
||||
- origin policy;
|
||||
- message/rate/size limits;
|
||||
- max subscriptions/series;
|
||||
- idle timeout/heartbeat;
|
||||
- bounded send queue and slow-client eviction;
|
||||
- no secret data in messages.
|
||||
|
||||
### Alert/notification abuse
|
||||
|
||||
Controls:
|
||||
- RBAC and audit;
|
||||
- versioned rules;
|
||||
- safe templates;
|
||||
- channel test rate limits;
|
||||
- idempotency;
|
||||
- recipient allowlist/policy;
|
||||
- no secret values in notification body.
|
||||
|
||||
### Deployment mistakes
|
||||
|
||||
Controls:
|
||||
- discovery and port/network/volume conflict checks;
|
||||
- backup touched configs;
|
||||
- isolated compose project;
|
||||
- offline validation;
|
||||
- health/smoke tests;
|
||||
- rollback;
|
||||
- no prune/delete/unrelated modifications;
|
||||
- production evidence.
|
||||
|
||||
## 4. Security headers
|
||||
|
||||
At minimum:
|
||||
- Content-Security-Policy;
|
||||
- frame restrictions;
|
||||
- nosniff;
|
||||
- strict referrer policy;
|
||||
- permissions policy;
|
||||
- HSTS when HTTPS deployment is stable;
|
||||
- secure/same-site/httpOnly cookies.
|
||||
|
||||
## 5. Container hardening
|
||||
|
||||
Where compatible:
|
||||
- non-root;
|
||||
- read-only root filesystem;
|
||||
- tmpfs for temporary paths;
|
||||
- drop all capabilities, add only required;
|
||||
- no-new-privileges;
|
||||
- seccomp/default profile;
|
||||
- resource limits;
|
||||
- explicit networks;
|
||||
- no public database/collector ports;
|
||||
- healthchecks;
|
||||
- immutable image digest in production record.
|
||||
|
||||
The agent may need narrow exceptions; document and test each.
|
||||
|
||||
### Deployment hardening pass (2026-08-04)
|
||||
|
||||
- Resource limits are set on all six `deploy/compose.yaml` services via the
|
||||
non-swarm `cpus`/`mem_limit`/`mem_reservation`/`memswap_limit` keys (the
|
||||
project runs plain `docker compose up`, not swarm); sizing rationale is
|
||||
inline in that file against `docs/architecture/SYSTEM_ARCHITECTURE.md` §7.
|
||||
- `pulse-postgres` now runs `read_only: true` with tmpfs for `/tmp` and
|
||||
`/var/run/postgresql`; all six services are now read-only-root. This
|
||||
closes the previously undocumented exception; see
|
||||
`docs/operations/DEPLOYMENT_UNRAID.md` §6/§8 for the required smoke test.
|
||||
- Immutable image digests are enforced by `deploy/verify-image-digests.sh`
|
||||
and CI. Every external registry image in `deploy/*.Dockerfile` is pinned to
|
||||
a verified digest; only Docker's built-in `scratch` rootfs is exempt because
|
||||
it has no registry manifest. See `deploy/IMAGE_DIGESTS.md` for the ledger.
|
||||
- `pulse-worker`/`pulse-agent` healthchecks now verify a heartbeat file's
|
||||
freshness instead of `kill -0 1`, and self-restart the container on
|
||||
staleness (`docker compose up` does not restart on "unhealthy" status
|
||||
alone). Contract for the Go runtime:
|
||||
`docs/operations/WORKER_AGENT_HEALTHCHECK_CONTRACT.md`.
|
||||
- `deploy/nginx.conf` now sends `Strict-Transport-Security` from
|
||||
`pulse-web` as defence-in-depth (§4), verified not to conflict with
|
||||
TLS terminating at Nginx Proxy Manager per ADR-0010.
|
||||
|
||||
## 6. Security acceptance
|
||||
|
||||
Required:
|
||||
- threat model review at M0 and M9;
|
||||
- SAST/dependency/image/secret scans;
|
||||
- auth/RBAC matrix tests;
|
||||
- SSRF suite;
|
||||
- WebSocket security suite;
|
||||
- dashboard import/XSS suite;
|
||||
- query limit/validation suite;
|
||||
- compose privilege/mount test;
|
||||
- backup secret inspection;
|
||||
- production exposure scan from permitted network.
|
||||
|
||||
### M7-02 implementation
|
||||
|
||||
The probe policy validates scheme, host and port before resolution, rejects loopback/link-local/metadata/multicast/unspecified addresses, and requires explicit CIDR permission for private LAN targets. The safe client repeats resolution at dial time and revalidates redirect destinations. Only bounded GET/HEAD requests with an allowlisted header set are accepted; responses are size/time limited. Policy changes are audit events containing no credentials or full request data.
|
||||
### M8-08 notification controls
|
||||
|
||||
Notification channel persistence accepts only secret references and rejects sensitive configuration keys such as token, password, secret, and authorization. Delivery bodies and subjects are bounded. Sender errors are newline-normalized, length-bounded, and redacted before persistence; bearer credentials are removed as a complete value. Test sends are rate limited and never write delivery records.
|
||||
@@ -0,0 +1,240 @@
|
||||
# System architecture
|
||||
|
||||
## 1. Context
|
||||
|
||||
ITWorx Pulse sits between the browser/operator and existing telemetry/inventory sources. It provides no operational write path to Unraid or Docker in v1.
|
||||
|
||||
### M0 environment binding
|
||||
|
||||
The discovered target is one Unraid 7.2.2 host with the native Unraid GraphQL API online, existing Nginx Proxy Manager and Authentik services, Grafana and host telemetry endpoints, and no local Prometheus container/listener identified during read-only discovery. Pulse therefore remains an additive, isolated Compose project. Host port, DNS name, appdata path, proxy record, and Docker network names are intentionally runtime discovery outputs and are not hardcoded in this architecture document.
|
||||
|
||||
The API/web trust boundary is unchanged by discovery: neither service receives a Docker socket or host filesystem privilege. The agent uses an allowlisted read-only Unraid API capability path. A separately configured Prometheus-compatible endpoint is queried only server-side; if it is unavailable or stale, the datasource and dependent status are `Unknown`.
|
||||
|
||||
```text
|
||||
Browser
|
||||
|
|
||||
| HTTPS / OIDC / REST / WebSocket
|
||||
v
|
||||
Pulse Web + API
|
||||
| \
|
||||
| \ PostgreSQL
|
||||
| config, inventory, events,
|
||||
| alerts, incidents, audit
|
||||
|
|
||||
+--> Prometheus-compatible source
|
||||
| historical metrics and range queries
|
||||
|
|
||||
+--> Pulse Worker
|
||||
| discovery, reconciliation, alert evaluation,
|
||||
| probes, retention, notifications
|
||||
|
|
||||
+--> Pulse Agent / constrained adapters
|
||||
read-only Unraid, host, storage and container facts
|
||||
```
|
||||
|
||||
## 2. Deployable units
|
||||
|
||||
### `pulse-web`
|
||||
|
||||
Preferred outcome: static React application served by a minimal web server or the API gateway. It has no secrets beyond public OIDC configuration and no direct infrastructure access.
|
||||
|
||||
### `pulse-api`
|
||||
|
||||
Responsibilities:
|
||||
- authenticated REST API;
|
||||
- WebSocket authentication and subscriptions;
|
||||
- dashboard/config CRUD;
|
||||
- inventory and event reads;
|
||||
- query planning and limits;
|
||||
- alert/incident user actions;
|
||||
- audit;
|
||||
- health/readiness.
|
||||
|
||||
No Docker socket and no host filesystem privilege.
|
||||
|
||||
### `pulse-worker`
|
||||
|
||||
Responsibilities:
|
||||
- scheduled discovery/reconciliation;
|
||||
- alert rule evaluation;
|
||||
- service probes;
|
||||
- incident correlation;
|
||||
- notification delivery;
|
||||
- retention/cleanup;
|
||||
- periodic self-checks.
|
||||
|
||||
Jobs are idempotent and database-coordinated.
|
||||
|
||||
### `pulse-agent`
|
||||
|
||||
Responsibilities:
|
||||
- host/Unraid/storage/container discovery that cannot be safely obtained through existing APIs/exporters;
|
||||
- normalized event/metric/status collection;
|
||||
- capability reporting.
|
||||
|
||||
The agent has the minimum read-only access required. It does not expose a general shell or mutation endpoint.
|
||||
|
||||
The API, worker and agent may share one Go module and image with separate commands while retaining runtime privilege separation.
|
||||
|
||||
### PostgreSQL
|
||||
|
||||
Stores:
|
||||
- users/external identities and roles;
|
||||
- settings and secrets references;
|
||||
- datasource metadata/health;
|
||||
- inventory and relationships;
|
||||
- dashboards/versions/widgets/layouts;
|
||||
- events;
|
||||
- alert rules/state/occurrences;
|
||||
- incidents/notes;
|
||||
- audit;
|
||||
- job coordination and migrations.
|
||||
|
||||
It does not duplicate full Prometheus time-series data.
|
||||
|
||||
### Prometheus-compatible source
|
||||
|
||||
Provides:
|
||||
- metric metadata;
|
||||
- instant/range query;
|
||||
- historical retention;
|
||||
- existing exporter scrape state.
|
||||
|
||||
Pulse queries it through a constrained server-side adapter. Browser access is forbidden.
|
||||
|
||||
## 3. Logical modules
|
||||
|
||||
```text
|
||||
identity
|
||||
authorization
|
||||
configuration
|
||||
datasources
|
||||
inventory
|
||||
metrics
|
||||
dashboards
|
||||
live
|
||||
events
|
||||
alerts
|
||||
incidents
|
||||
probes
|
||||
notifications
|
||||
audit
|
||||
operations
|
||||
```
|
||||
|
||||
Each module defines domain types and interfaces. HTTP/database/Prometheus/Unraid implementations are adapters.
|
||||
|
||||
## 4. Data flows
|
||||
|
||||
### Dashboard load
|
||||
|
||||
1. Browser requests dashboard definition.
|
||||
2. API authorizes user and returns validated config.
|
||||
3. Browser sends a batched semantic historical query.
|
||||
4. API query planner validates bounds, translates to PromQL and deduplicates.
|
||||
5. Prometheus returns series.
|
||||
6. API normalizes units/metadata and returns bounded points.
|
||||
7. Browser opens one WebSocket and subscribes to visible live streams.
|
||||
8. Live broker coalesces equivalent subscriptions.
|
||||
|
||||
### Discovery
|
||||
|
||||
1. Worker requests capability/inventory snapshots from adapters/agent.
|
||||
2. Payloads are validated and source-stamped.
|
||||
3. Reconciler maps stable identities and relationships.
|
||||
4. User overrides remain separate.
|
||||
5. Changes create normalized events.
|
||||
6. Datasource freshness/status is updated.
|
||||
|
||||
### Alert evaluation
|
||||
|
||||
1. Worker selects due rules with a coordination lock.
|
||||
2. Rule queries semantic metric/event/status inputs.
|
||||
3. State machine applies pending/hysteresis/cooldown.
|
||||
4. Occurrence/event/audit records persist transactionally.
|
||||
5. Suppression/grouping is calculated.
|
||||
6. Incident correlation and notifications run.
|
||||
7. UI receives a live state update.
|
||||
|
||||
## 5. Trust boundaries
|
||||
|
||||
1. Browser <-> web/API.
|
||||
2. Pulse runtime <-> OIDC provider.
|
||||
3. API/worker <-> PostgreSQL.
|
||||
4. API/worker <-> Prometheus.
|
||||
5. Agent/adapters <-> host/Unraid/Docker.
|
||||
6. Probe worker <-> configured network targets.
|
||||
7. Notification worker <-> external channels.
|
||||
|
||||
Each boundary requires authentication/authorization, timeouts, validation, redaction and least privilege.
|
||||
|
||||
## 6. Availability model
|
||||
|
||||
Pulse may run as a single instance in v1. It must recover from restarts without losing configuration or alert history.
|
||||
|
||||
- Web/API failure: external health check detects.
|
||||
- Worker failure: heartbeats and dead-man alert.
|
||||
- Agent failure: datasource unknown/stale; no false green.
|
||||
- Prometheus failure: historical/live metrics unknown; inventory remains available.
|
||||
- Database failure: API not ready; no in-memory claim of health.
|
||||
- OIDC failure: existing sessions follow policy; break-glass recovery remains controlled.
|
||||
|
||||
## 7. Scaling limits
|
||||
|
||||
Target:
|
||||
- one host;
|
||||
- 150 containers;
|
||||
- 40 disks;
|
||||
- 300 service probes;
|
||||
- 2,500 active dashboard series;
|
||||
- 10 concurrent users;
|
||||
- long-running wallboard.
|
||||
|
||||
The architecture must bound:
|
||||
- Prometheus concurrency;
|
||||
- series/points;
|
||||
- browser ring buffers;
|
||||
- WebSocket subscriptions;
|
||||
- event payloads;
|
||||
- probe concurrency/response size;
|
||||
- inventory snapshots;
|
||||
- audit retention.
|
||||
|
||||
## 8. Configuration
|
||||
|
||||
Configuration layers:
|
||||
|
||||
1. secure runtime environment/secrets;
|
||||
2. validated application config;
|
||||
3. database-managed settings;
|
||||
4. user/dashboard preferences.
|
||||
|
||||
Startup fails clearly for invalid mandatory config. Optional integrations report disabled/unavailable capabilities.
|
||||
|
||||
## 9. Extensibility
|
||||
|
||||
Connectors implement a capability interface:
|
||||
|
||||
```text
|
||||
discover
|
||||
health
|
||||
inventory
|
||||
metrics bindings
|
||||
events
|
||||
capabilities
|
||||
```
|
||||
|
||||
No connector receives arbitrary access to core storage or bypasses authorization. Version connector contracts.
|
||||
|
||||
## 10. Architecture fitness tests
|
||||
|
||||
Automated tests must enforce:
|
||||
- no Docker socket mount on web/API service;
|
||||
- no mutation operation in public v1 API;
|
||||
- migrations present for schema changes;
|
||||
- OpenAPI/schema compatibility;
|
||||
- package/module dependency direction;
|
||||
- bounded query defaults;
|
||||
- stale -> unknown mapping;
|
||||
- non-root container configuration;
|
||||
- secrets absent from image/repo.
|
||||
@@ -0,0 +1,168 @@
|
||||
# Telemetry and query engine
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Pulse exposes a stable semantic metric model while retaining Prometheus as the v1 time-series source.
|
||||
|
||||
Users/widgets request:
|
||||
|
||||
```text
|
||||
container.cpu.utilization
|
||||
storage.disk.temperature
|
||||
service.response_time
|
||||
```
|
||||
|
||||
The backend maps these names to source-specific PromQL templates and label contracts.
|
||||
|
||||
## 2. Metric definition
|
||||
|
||||
A metric definition includes:
|
||||
|
||||
- semantic name and version;
|
||||
- description and unit;
|
||||
- gauge/counter/state kind;
|
||||
- required capabilities;
|
||||
- query template;
|
||||
- allowed labels/grouping;
|
||||
- default aggregation;
|
||||
- valid transformations;
|
||||
- default interval;
|
||||
- max range/series/points;
|
||||
- freshness;
|
||||
- visualizations;
|
||||
- status/threshold hints;
|
||||
- cardinality budget.
|
||||
|
||||
See `specs/metric-definition.schema.json`.
|
||||
|
||||
## 3. Query planning
|
||||
|
||||
Pipeline:
|
||||
|
||||
1. authenticate and authorize;
|
||||
2. validate semantic metric and scope;
|
||||
3. resolve entity aliases/source binding;
|
||||
4. clamp/validate range, step, series and points;
|
||||
5. select binding based on source capabilities;
|
||||
6. generate parameterized PromQL from templates;
|
||||
7. deduplicate equivalent queries;
|
||||
8. execute with timeout/concurrency budget;
|
||||
9. normalize labels, units and missing values;
|
||||
10. downsample if needed;
|
||||
11. return provenance, warnings and freshness.
|
||||
|
||||
Raw PromQL is an advanced feature, disabled by default, separately authorized and constrained.
|
||||
|
||||
## 4. Query budgets
|
||||
|
||||
Budgets are configurable but must exist:
|
||||
|
||||
- max range;
|
||||
- max series;
|
||||
- max samples/points;
|
||||
- max query string/template expansion;
|
||||
- timeout;
|
||||
- per-user/per-dashboard concurrency;
|
||||
- global Prometheus concurrency;
|
||||
- cacheable result size;
|
||||
- live subscription count;
|
||||
- label enumeration limits.
|
||||
|
||||
Return a clear limit error rather than overloading Prometheus.
|
||||
|
||||
## 5. Caching
|
||||
|
||||
Cache:
|
||||
- metric catalog/capabilities;
|
||||
- short historical range results;
|
||||
- label/metadata results;
|
||||
- repeated dashboard query plans.
|
||||
|
||||
Do not cache:
|
||||
- authorization decisions beyond safe session scope;
|
||||
- live current status beyond its freshness policy;
|
||||
- secret-bearing errors.
|
||||
|
||||
Cache keys include source, tenant/server scope, semantic query, normalized range and permissions where relevant.
|
||||
|
||||
Redis is not required for a single instance. Use bounded in-memory cache and/or PostgreSQL only when appropriate.
|
||||
|
||||
## 6. Live engine
|
||||
|
||||
### Fast lane
|
||||
|
||||
- visible high-frequency widgets subscribe at 1–5 seconds;
|
||||
- backend polls/queries compatible batches or receives collector samples;
|
||||
- equivalent subscriptions share upstream work;
|
||||
- samples are appended with sequence numbers;
|
||||
- browser retains a bounded ring buffer;
|
||||
- historical data is not fully refetched per sample.
|
||||
|
||||
### Durable lane
|
||||
|
||||
Prometheus scrape/history remains durable. Pulse does not persist every live sample into PostgreSQL.
|
||||
|
||||
### Adaptive behavior
|
||||
|
||||
- out-of-view widgets reduce frequency;
|
||||
- background tabs reduce frequency;
|
||||
- paused dashboards unsubscribe;
|
||||
- wallboards remain active with bounded buffers;
|
||||
- slow clients receive coalesced latest samples;
|
||||
- sequence gaps trigger bounded resync.
|
||||
|
||||
## 7. Browser chart architecture
|
||||
|
||||
- historical query initializes chart;
|
||||
- live samples append outside global React state where practical;
|
||||
- series count and point count are capped;
|
||||
- old points are evicted;
|
||||
- chart resources are disposed on unmount;
|
||||
- ResizeObserver/visibility changes are debounced;
|
||||
- tooltip/legend state does not duplicate large arrays;
|
||||
- long wallboard test measures heap and subscription count.
|
||||
|
||||
## 8. Staleness and unknown
|
||||
|
||||
Each response includes:
|
||||
- source timestamp;
|
||||
- received timestamp;
|
||||
- freshness state;
|
||||
- warnings.
|
||||
|
||||
If a required source is stale/unavailable:
|
||||
- current value is marked stale or omitted;
|
||||
- status becomes unknown according to policy;
|
||||
- previous value may be displayed with age;
|
||||
- alerts can enter unknown rather than resolve.
|
||||
|
||||
## 9. Host snapshot binding
|
||||
|
||||
Host detail reads use the explicit read-only host contract. A source adapter returns bounded normalized values with UTC observed/received timestamps and capability version. The adapter sorts filesystem/interface collections deterministically, validates percentage/byte units and rejects payloads over configured collection limits. The API/web boundary has no host or Docker privilege; when no approved source is connected, the result is an explicit unknown snapshot. Load and memory status reasons are returned as text so high load is distinguishable from stale or unavailable telemetry.
|
||||
## 9. Transformations
|
||||
|
||||
Supported through typed operations:
|
||||
- rate;
|
||||
- increase/delta;
|
||||
- average/min/max/sum;
|
||||
- quantile;
|
||||
- percentage;
|
||||
- top/bottom N;
|
||||
- unit conversion;
|
||||
- compare previous period;
|
||||
- fill policy;
|
||||
- status mapping.
|
||||
|
||||
Transform order is explicit and validated. Avoid silently mixing counter rates and gauges.
|
||||
|
||||
|
||||
Optional hardware follows the same boundary: declared thermal, fan and GPU capabilities are independently enabled, unsupported, unavailable or disabled. Missing capability is not an adapter failure. Sensor IDs are source-scoped and stable from external ID/name, collections are bounded and sorted, and thermal reasons are returned as text. No API/web host privilege, device mount or unrestricted socket is needed.
|
||||
## 10. Cardinality controls
|
||||
|
||||
- semantic definitions allow only known labels;
|
||||
- unbounded label values are excluded or normalized;
|
||||
- inventory IDs map to stable bounded labels;
|
||||
- dashboards cannot group by arbitrary label by default;
|
||||
- source cardinality health is monitored;
|
||||
- query inspector shows series count and limits.
|
||||
- query inspector is operate-permissioned, exposes only planner-approved generated PromQL and redacts sensitive-looking text; it does not execute an unrestricted query.
|
||||
@@ -0,0 +1,19 @@
|
||||
# ADR 0001 — Pulse v1 is operationally read-only
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
Pulse may observe, query, configure its own monitoring behavior and create user records such as acknowledgements. It may not mutate Unraid, Docker, storage, containers, host processes or network configuration.
|
||||
|
||||
## Rationale
|
||||
|
||||
Monitoring and management have different security and failure domains. A compromised dashboard must not become a host control plane. Remediation belongs in a separately controlled AppOps workflow.
|
||||
|
||||
## Consequences
|
||||
|
||||
- No start/stop/restart/delete APIs.
|
||||
- No process kill.
|
||||
- No array/pool operations.
|
||||
- Optional links may open an external management workflow.
|
||||
- Tests enforce absence of mutation routes and dangerous agent capabilities.
|
||||
@@ -0,0 +1,17 @@
|
||||
# ADR 0002 — Go services and React/TypeScript web application
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
Use Go for API, worker and agent commands. Use React + TypeScript for the web application, with Vite as the default build tool unless discovery proves a stronger need for server-rendered Next.js behavior.
|
||||
|
||||
## Rationale
|
||||
|
||||
The product is a highly interactive authenticated dashboard rather than a public content site. Go provides efficient long-lived connections, concurrency and small deployable binaries. React has mature dashboard/grid/chart ecosystems. A static frontend reduces production runtime complexity.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Shared contracts are generated or validated from schemas.
|
||||
- API/worker/agent can share modules but deploy with separate privileges.
|
||||
- An ADR is required to add a Node server runtime to production.
|
||||
@@ -0,0 +1,17 @@
|
||||
# ADR 0003 — Existing Prometheus-compatible source remains v1 metrics history
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
Use the existing Prometheus-compatible endpoint for instant/range queries and retention in v1. Do not introduce VictoriaMetrics or another time-series database until measurements show a concrete retention, performance or reliability requirement.
|
||||
|
||||
## Rationale
|
||||
|
||||
Avoid duplicate infrastructure and migration risk. Pulse differentiates through semantic queries, UX, inventory, alerts and incidents.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Query planner and limits protect the source.
|
||||
- Pulse stores no full metric history in PostgreSQL.
|
||||
- Long-term storage remains a future measured decision.
|
||||
@@ -0,0 +1,18 @@
|
||||
# ADR 0004 — PostgreSQL stores Pulse domain state
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
Use PostgreSQL for configuration, inventory, relationships, dashboards, events, alert state/history, incidents, audit and job coordination.
|
||||
|
||||
## Rationale
|
||||
|
||||
The data is relational, transactional and queryable. PostgreSQL supports JSONB for bounded extensibility and reliable migrations/backups.
|
||||
|
||||
## Consequences
|
||||
|
||||
- All schema changes use migrations.
|
||||
- Production database is private.
|
||||
- Backup/restore is a release gate.
|
||||
- Metric samples remain outside PostgreSQL.
|
||||
@@ -0,0 +1,17 @@
|
||||
# ADR 0005 — No unrestricted Docker socket in API/web
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
The web and API containers never mount the unrestricted Docker socket. Docker/Unraid facts come from the official Unraid API, existing exporters, Portainer read-only endpoints, or a separate constrained agent/socket proxy.
|
||||
|
||||
## Rationale
|
||||
|
||||
Docker daemon access is effectively host control. Read-only filesystem mount flags do not create a read-only Docker API.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Collector capability endpoints are allowlisted.
|
||||
- Agent runtime is separated and audited.
|
||||
- Architecture tests inspect compose mounts and API routes.
|
||||
@@ -0,0 +1,17 @@
|
||||
# ADR 0006 — REST plus WebSocket
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
Use versioned REST for requests/configuration/history and one authenticated WebSocket per browser session for bounded live updates.
|
||||
|
||||
## Rationale
|
||||
|
||||
REST is clear for CRUD/query operations. WebSocket supports low-latency subscriptions and server-side coalescing without repeated polling of full datasets.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Messages use explicit schema/version/sequence.
|
||||
- Reconnect and resync are required.
|
||||
- Subscription, rate, size and buffer limits are mandatory.
|
||||
@@ -0,0 +1,17 @@
|
||||
# ADR 0007 — Dutch default, localization-ready architecture
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
Ship user-facing v1 flows in Dutch (`nl-BE`) and use localization keys/contracts so English can be added without rewriting components.
|
||||
|
||||
## Rationale
|
||||
|
||||
The primary user is Dutch-speaking, while code and technical contracts benefit from stable English identifiers.
|
||||
|
||||
## Consequences
|
||||
|
||||
- No hardcoded scattered UI copy.
|
||||
- Tests cover missing translation keys.
|
||||
- Dates/numbers use locale and Europe/Brussels display defaults while storage remains UTC.
|
||||
@@ -0,0 +1,18 @@
|
||||
# ADR 0008 — Stale or missing required telemetry is Unknown
|
||||
|
||||
**Status:** Accepted baseline
|
||||
|
||||
## Decision
|
||||
|
||||
When required telemetry is unavailable beyond its freshness policy, Pulse reports Unknown rather than retaining or inferring Healthy.
|
||||
|
||||
## Rationale
|
||||
|
||||
False green status is more dangerous than explicit uncertainty.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Responses carry freshness metadata.
|
||||
- Alert rules define unknown behavior.
|
||||
- UI shows last known value only with age.
|
||||
- Tests inject stale/missing sources throughout the product.
|
||||
@@ -0,0 +1,42 @@
|
||||
# ADR 0009 — Upstream and dependency baseline after M0 research
|
||||
|
||||
**Status:** Accepted for M1 planning; exact package versions remain pinned during M1 implementation.
|
||||
|
||||
## Context
|
||||
|
||||
M0 must verify current upstream capabilities and avoid selecting unsupported or abandoned dependencies. The target Unraid host is 7.2.2 with its native Unraid API online. The local development host has Node.js 24.18.1 and pnpm 10.33.0 but no Go toolchain.
|
||||
|
||||
## Decision
|
||||
|
||||
- Use the native Unraid GraphQL API as the first inventory adapter. Unraid 7.2+ includes the API in the OS, and programmatic access supports API keys, cookies, and SSO/OIDC. Pulse uses a least-privilege read-only API key or equivalent controlled identity; it must not use mutation capabilities.
|
||||
- Keep Prometheus-compatible history as an external server-side datasource. Pulse uses the stable `/api/v1/query` and `/api/v1/query_range` APIs through bounded semantic queries; the browser never contacts Prometheus directly.
|
||||
- Use Authentik OIDC/OAuth2 as the production identity integration. The server performs authorization-code exchange and token validation; public/browser flows use PKCE where applicable. Per-provider issuer/discovery is the default because authentik documents it as the recommended issuer mode.
|
||||
- Implement the frontend with React + TypeScript + Vite, using the current React documentation baseline (19.2) and Vite's supported Node requirement. Use GridStack as the dashboard-grid candidate; retain a measured-equivalent escape hatch. Do not commit exact package versions until M1 lockfile/bootstrap work.
|
||||
- Prefer Go's standard library for the initial backend transport and pin a currently supported Go release in M1. The official Go release policy supports a major release until two newer majors exist. The baseline is maintained at Go 1.26.6 after the M13 release image gate identified fixed standard-library findings in 1.26.5.
|
||||
|
||||
## License and support record
|
||||
|
||||
| Component | Upstream license observed | Support/compatibility note |
|
||||
|---|---|---|
|
||||
| Go toolchain | BSD-style (official Go distribution) | Use an official currently supported release; Go is not installed locally yet. |
|
||||
| React | MIT | Official React docs list 19.2 as latest major baseline. |
|
||||
| Vite | MIT | Official docs require Node 20.19+ or 22.12+; local Node 24.18.1 satisfies the documented floor. |
|
||||
| GridStack | MIT | Candidate only; verify package release and transitive dependencies during M1. |
|
||||
| Prometheus API/source | Apache 2.0 | External source; Pulse does not redistribute Prometheus in the application image. |
|
||||
| authentik integration | MIT core with documented directory/component exceptions | Pulse integrates with the deployed provider; it does not embed or redistribute authentik. |
|
||||
|
||||
## Consequences
|
||||
|
||||
- M1 must provision Go, generate lockfiles, run license/dependency checks, and pin exact versions from official release metadata.
|
||||
- Prometheus location remains unresolved in the actual Unraid environment: no local 9090 listener/container was found. Datasource onboarding must support an explicitly configured external endpoint and report Unknown when absent/stale.
|
||||
- The frontend runtime remains a static React/Vite artifact served behind the API/reverse proxy, preserving the no-infrastructure-access browser boundary.
|
||||
|
||||
## Sources accessed 2026-08-01
|
||||
|
||||
- [Unraid API overview](https://docs.unraid.net/API/) and [Unraid API usage](https://docs.unraid.net/API/how-to-use-the-api/)
|
||||
- [Go release history](https://go.dev/doc/devel/release)
|
||||
- [Prometheus HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/) and [PromQL basics](https://prometheus.io/docs/prometheus/latest/querying/basics/)
|
||||
- [authentik OAuth 2.0/OIDC provider](https://docs.goauthentik.io/add-secure-apps/providers/oauth2)
|
||||
- [React versions](https://react.dev/versions) and [React reference](https://react.dev/reference/react)
|
||||
- [Vite getting started and compatibility](https://vite.dev/guide/)
|
||||
- [React MIT license](https://raw.githubusercontent.com/react/react/main/LICENSE), [Vite MIT license](https://raw.githubusercontent.com/vitejs/vite/main/LICENSE), [GridStack MIT license](https://raw.githubusercontent.com/gridstack/gridstack.js/master/LICENSE), [Prometheus Apache 2.0 license](https://raw.githubusercontent.com/prometheus/prometheus/main/LICENSE), and [authentik license](https://raw.githubusercontent.com/goauthentik/authentik/main/LICENSE)
|
||||
@@ -0,0 +1,95 @@
|
||||
# Backend and worker standards
|
||||
|
||||
## Domain boundaries
|
||||
|
||||
Core packages define:
|
||||
- entities and reconciliation;
|
||||
- metrics query model;
|
||||
- dashboard validation/versioning;
|
||||
- events;
|
||||
- alert state machine;
|
||||
- incidents;
|
||||
- authorization policies.
|
||||
|
||||
Adapters implement:
|
||||
- PostgreSQL;
|
||||
- Prometheus;
|
||||
- Unraid;
|
||||
- Docker/constrained collector;
|
||||
- OIDC;
|
||||
- probes;
|
||||
- notifications.
|
||||
|
||||
Domain packages must not import HTTP handlers or concrete adapters.
|
||||
|
||||
## HTTP
|
||||
|
||||
- Route groups map to modules.
|
||||
- Middleware order is explicit.
|
||||
- Request body, query and path validation.
|
||||
- Body size limits.
|
||||
- Timeouts and cancellation.
|
||||
- Stable error mapping.
|
||||
- Structured request log with redaction.
|
||||
- Health/live and readiness separated.
|
||||
- No mutation route for host/Docker/storage.
|
||||
|
||||
## Worker
|
||||
|
||||
- Jobs have stable keys and schedules.
|
||||
- Database lock/lease prevents duplicate execution.
|
||||
- Work is idempotent.
|
||||
- Retries are bounded with jitter.
|
||||
- Poison/repeated failures become visible system status.
|
||||
- Shutdown waits for bounded graceful completion.
|
||||
- Job result counts and duration recorded.
|
||||
|
||||
## Prometheus adapter
|
||||
|
||||
- Server-side only.
|
||||
- Timeout and concurrency semaphore.
|
||||
- Semantic templates only by default.
|
||||
- Parse upstream warnings and staleness.
|
||||
- Protect metadata/label endpoints.
|
||||
- Normalize errors to stable codes.
|
||||
- Instrument query duration/series/points/cache.
|
||||
|
||||
## Inventory reconciliation
|
||||
|
||||
- Stable source alias mapping.
|
||||
- Snapshot can be repeated safely.
|
||||
- Missing item becomes tombstoned after source-specific policy, not immediately deleted.
|
||||
- User overrides remain separate.
|
||||
- Relation source/confidence retained.
|
||||
- Changes create deduplicated events.
|
||||
- Source failure does not tombstone all entities.
|
||||
|
||||
## Alerts
|
||||
|
||||
- Deterministic state machine.
|
||||
- Evaluation transaction/locking.
|
||||
- Clock abstracted in tests.
|
||||
- Unknown data explicit.
|
||||
- Rule versions immutable.
|
||||
- Notification side effects use outbox/idempotency pattern or equivalent.
|
||||
- Acknowledgement does not mutate underlying firing condition.
|
||||
|
||||
## Database
|
||||
|
||||
- Context-aware queries.
|
||||
- Parameterized SQL.
|
||||
- Transaction boundaries documented.
|
||||
- Pool limits.
|
||||
- Repository methods return domain types/errors.
|
||||
- Integration tests against real PostgreSQL.
|
||||
- No production reliance on SQLite semantics.
|
||||
|
||||
## Agent
|
||||
|
||||
- Read-only capability list.
|
||||
- No generic command execution endpoint.
|
||||
- Authenticated/mutually trusted channel if remote.
|
||||
- Bounded payloads.
|
||||
- Version/capability negotiation.
|
||||
- Local cache only where safe.
|
||||
- Every elevated mount/capability justified and tested.
|
||||
@@ -0,0 +1,36 @@
|
||||
# CI pipeline target
|
||||
|
||||
Codex must implement CI-equivalent commands locally even when no hosted CI is connected.
|
||||
|
||||
Recommended stages:
|
||||
|
||||
1. repository/planning validation;
|
||||
2. formatting;
|
||||
3. lint/static analysis;
|
||||
4. generated contract drift;
|
||||
5. wiring/reachability gate (`python tools/check_wiring.py`): every production package must have at least one non-test importer reachable from a binary (`cmd/api`, `cmd/worker`, `cmd/agent`, `cmd/migrate`), or be listed in `tools/wiring_allowlist.json` with a reason and a tracking task id; a task may not be marked done while its deliverable is unreachable;
|
||||
6. frontend type/unit (`pnpm test` runs Vitest + Testing Library, including ADR-0008 status invariants);
|
||||
7. Go unit/race where suitable;
|
||||
8. integration with PostgreSQL/fake sources;
|
||||
9. frontend build;
|
||||
10. API/OpenAPI/schema compatibility;
|
||||
11. Playwright smoke (`pnpm test:e2e`, desktop/mobile/wallboard Chromium projects);
|
||||
12. accessibility (axe-core in the Playwright smoke; serious or critical violations fail);
|
||||
13. dependency/license/secret scan;
|
||||
14. container build and image scan;
|
||||
15. compose/config validation;
|
||||
16. evidence summary.
|
||||
|
||||
The executable Gitea Actions definition is `.gitea/workflows/ci.yml`. The local
|
||||
equivalent is `scripts/verify.ps1`; install Chromium once with
|
||||
`pnpm exec playwright install chromium` before its browser stage. Browser tests
|
||||
mock only the versioned API boundary and exercise the real built React routes.
|
||||
|
||||
Release pipeline additionally:
|
||||
- clean checkout;
|
||||
- full E2E;
|
||||
- performance subset;
|
||||
- migration/backup/restore;
|
||||
- SBOM/provenance where feasible;
|
||||
- immutable image digest record;
|
||||
- deployment smoke and rollback proof.
|
||||
@@ -0,0 +1,35 @@
|
||||
# M1 dependency record
|
||||
|
||||
Research and install date: 2026-08-01
|
||||
|
||||
These exact versions are pinned in `package.json`/`apps/web/package.json` and `pnpm-lock.yaml`. Go is installed in the developer's user-local toolchain cache at Go 1.26.6 and is declared in `go.mod`/`go.work`; it is not vendored into the repository.
|
||||
|
||||
| Dependency | Version | Purpose | License | Alternative/decision |
|
||||
|---|---|---|---|---|
|
||||
| Go toolchain | 1.26.6 | API, worker, agent | BSD-style | Official current supported patch; required by the M13 release image gate to remove fixed Go standard-library High findings. |
|
||||
| React | 19.2.8 | Web UI | MIT | React is required by the architecture; framework/server rendering is unnecessary for the static dashboard shell. |
|
||||
| React DOM | 19.2.8 | Browser renderer | MIT | Pinned with React. |
|
||||
| Vite | 8.2.0 | TypeScript web build/dev server | MIT | Chosen over a heavier framework because the architecture calls for a static React app and Vite supports the available Node runtime. |
|
||||
| `@vitejs/plugin-react` | 6.0.5 | Vite React transform | MIT | Official Vite ecosystem plugin; pinned with Vite 8. |
|
||||
| TypeScript | 7.0.2 | Strict web type checking | Apache-2.0 | Required for the React + TypeScript architecture. |
|
||||
| React type declarations | 19.2.18 / 19.2.4 | Compile-time types | MIT | Pinned to the installed React major. |
|
||||
| `github.com/jackc/pgx/v5` | 5.10.0 | PostgreSQL connection pool and parameterized access | MIT | Selected for native context-aware pooling and PostgreSQL support; pinned after module/license review. |
|
||||
| `github.com/coreos/go-oidc/v3` | 3.20.0 | OIDC discovery, issuer/audience/JWK-backed ID-token verification | Apache-2.0 | Uses maintained standards-oriented verifier; server-side only, pinned after module/license review. |
|
||||
| `golang.org/x/oauth2` | 0.36.0 | Authorization-code exchange and PKCE request parameters | BSD-style | Official Go OAuth2 client primitives; pinned and kept behind the auth adapter. |
|
||||
|
||||
No charting, grid, or HTTP-router dependency is added yet. Those material choices require the relevant task's primary-source/security/license review and measurement. `pnpm-lock.yaml` records registry integrity data; Go module checksums are recorded in `go.sum`.
|
||||
|
||||
## Verification record
|
||||
|
||||
- `go version`: `go1.26.6 windows/amd64`.
|
||||
- `pnpm install --frozen-lockfile`: pass with pnpm 10.33.0.
|
||||
- Vite's official compatibility floor is Node 20.19+ or 22.12+; the local Node 24.18.1 satisfies it.
|
||||
- `go test ./...`, `go vet ./...`, TypeScript typecheck, Vite build, and repository bootstrap/test/lint scripts pass.
|
||||
- `go mod verify`: pass; pgx v5.10.0 and transitive modules are checksum-verified.
|
||||
- pgx v5.10.0 module metadata points to the upstream `github.com/jackc/pgx` repository; the cached module includes an MIT license.
|
||||
- go-oidc v3.20.0 and oauth2 v0.36.0 are checksum-verified; the cached go-oidc module includes an Apache-2.0 license and oauth2 is maintained under the Go project license.
|
||||
- Releasegate 2026-08-21 pins transitieve builddependency `nanoid` op 3.3.18 via een beperkte pnpm-override. Dit sluit GHSA-2v37-7h3g-55p8 in Vite -> PostCSS; `pnpm audit --audit-level high` en Trivy met developmentdependencies rapporteren daarna nul High/Critical-bevindingen. Nanoid blijft uitsluitend onderdeel van de MIT-gelicentieerde buildketen en wordt niet aan de browserruntime toegevoegd.
|
||||
|
||||
## Upgrade/removal path
|
||||
|
||||
Update package manifests and lockfile together, rerun the foundation scripts plus the affected milestone gate, review changelogs/security advisories, and record any compatibility or bundle/runtime impact. Removing Vite/React is an architectural change requiring an ADR; removing a foundation tool requires replacement commands and clean-room evidence.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Dependency policy
|
||||
|
||||
A production dependency is accepted only when it:
|
||||
|
||||
- solves a real requirement better than a small maintained implementation;
|
||||
- is actively maintained;
|
||||
- has a compatible license;
|
||||
- has no unresolved unacceptable security issue;
|
||||
- supports the selected runtime/browser versions;
|
||||
- has clear upgrade and removal paths;
|
||||
- does not require excessive privilege or bundle size.
|
||||
|
||||
## Selection record
|
||||
|
||||
For material dependencies record:
|
||||
- package and version range;
|
||||
- purpose;
|
||||
- alternatives considered;
|
||||
- maintenance/release activity;
|
||||
- license;
|
||||
- security check;
|
||||
- bundle/image/runtime impact;
|
||||
- locking strategy.
|
||||
|
||||
This may be an ADR or a dependency manifest note.
|
||||
|
||||
## Default choices to validate during M0/M1
|
||||
|
||||
- React + TypeScript + Vite.
|
||||
- GridStack for layout.
|
||||
- uPlot for high-volume time series.
|
||||
- ECharts for complex visualizations.
|
||||
- TanStack Query for server state.
|
||||
- Accessible UI primitives/component library.
|
||||
- Go HTTP/router, OIDC/JWT and PostgreSQL libraries selected from maintained options.
|
||||
- PostgreSQL migration tool with explicit CLI and rollback strategy.
|
||||
- Playwright and automated accessibility tooling.
|
||||
- Testcontainers for integration tests.
|
||||
|
||||
These are defaults, not permission to install blindly. Verify current supported versions and compatibility.
|
||||
|
||||
## Rules
|
||||
|
||||
- Commit lockfiles.
|
||||
- Prefer exact image tags/digests in production records.
|
||||
- Avoid duplicate libraries for the same concern.
|
||||
- Do not use abandonware because an example already uses it.
|
||||
- Do not add Redis, Kafka, Elasticsearch or a second metrics database without measured need and ADR.
|
||||
- Remove unused dependencies immediately.
|
||||
- Run dependency/license/vulnerability checks at milestones and release.
|
||||
|
||||
|
||||
## M4-06 WebSocket selection
|
||||
|
||||
- Package/version: github.com/coder/websocket v1.8.15, pinned in go.mod and go.sum.
|
||||
- Purpose: RFC6455 server upgrade, context-aware reads/writes, ping/pong and bounded frame reads for the authenticated live endpoint.
|
||||
- Alternatives considered: hand-rolled RFC6455 handling was rejected because it increases protocol and security risk; gorilla/websocket was not needed for this narrow API; golang.org/x/net/websocket is deprecated.
|
||||
- Maintenance/security: current upstream release was resolved locally on 2026-08-01; the module has zero transitive dependencies and the source license is permissive MIT.
|
||||
- Runtime impact: server-only dependency, no browser bundle or privilege change; SetReadLimit and write deadlines enforce the endpoint budget.
|
||||
@@ -0,0 +1,125 @@
|
||||
# Engineering standards
|
||||
|
||||
## Repository shape
|
||||
|
||||
Target:
|
||||
|
||||
```text
|
||||
apps/
|
||||
web/
|
||||
api/
|
||||
worker/
|
||||
agent/
|
||||
packages/
|
||||
contracts/
|
||||
ui/
|
||||
test-fixtures/
|
||||
internal/
|
||||
domain modules or Go internal packages
|
||||
config/
|
||||
deploy/
|
||||
tests/
|
||||
docs/
|
||||
artifacts/evidence/
|
||||
```
|
||||
|
||||
Codex may refine the shape through an ADR, but privilege boundaries and clear ownership must remain.
|
||||
|
||||
## General
|
||||
|
||||
- Optimize for correctness, observability and maintainability.
|
||||
- Keep changes vertically complete.
|
||||
- Validate external inputs at boundaries.
|
||||
- Avoid global mutable state.
|
||||
- Use deterministic IDs/fingerprints where required.
|
||||
- Use UTC internally.
|
||||
- Add correlation IDs to request/job/event paths.
|
||||
- Preserve error causes and add context.
|
||||
- Never log secrets.
|
||||
- Use feature flags only when they have an owner, default and removal plan.
|
||||
|
||||
## Git and commits
|
||||
|
||||
- Focused commits aligned to task IDs.
|
||||
- Commit message format: `<TASK_ID>: <imperative summary>`.
|
||||
- Do not rewrite shared history.
|
||||
- No force push or destructive reset.
|
||||
- Keep generated evidence out of commits only when too large; summaries remain.
|
||||
- Tag releases only after final acceptance.
|
||||
|
||||
## Go
|
||||
|
||||
- Current supported stable Go version selected during M0 and recorded.
|
||||
- `go fmt`, `go vet`, static analysis and tests required.
|
||||
- Context propagated through I/O boundaries.
|
||||
- Errors wrapped with operation context.
|
||||
- Interfaces defined near consumers; avoid interface proliferation.
|
||||
- Goroutines have ownership, cancellation and bounded lifetime.
|
||||
- Worker jobs are idempotent.
|
||||
- SQL is parameterized; transactions explicit.
|
||||
- Migrations are forward/recovery tested.
|
||||
- HTTP handlers contain no core domain logic.
|
||||
|
||||
## TypeScript/React
|
||||
|
||||
- Strict TypeScript.
|
||||
- No `any` except narrow justified boundary adapters.
|
||||
- Runtime validation for external JSON.
|
||||
- Components separate data orchestration from presentation.
|
||||
- Server state uses a deliberate query/cache layer.
|
||||
- Live chart buffers do not live in broad global state.
|
||||
- Effects are cancellable and cleanup subscriptions.
|
||||
- Accessible semantic HTML first.
|
||||
- All user copy goes through localization.
|
||||
- Avoid giant components and prop drilling; centralize domain-specific hooks appropriately.
|
||||
|
||||
## API and contracts
|
||||
|
||||
- OpenAPI/JSON Schema is validated in CI.
|
||||
- Breaking changes are versioned.
|
||||
- Generated types are reproducible.
|
||||
- Error codes are stable.
|
||||
- Pagination, filtering and sorting are bounded.
|
||||
- Every endpoint has authz tests.
|
||||
- WebSocket messages are schema validated.
|
||||
|
||||
## Database
|
||||
|
||||
- Explicit migrations, no startup auto-mutation outside migration command.
|
||||
- Indexes justified by access path.
|
||||
- Constraints enforce invariants where practical.
|
||||
- JSONB payloads have size/schema limits.
|
||||
- Optimistic concurrency for user-edited versioned resources.
|
||||
- Timeouts and connection pool limits.
|
||||
- Test upgrade, restart, backup and restore.
|
||||
|
||||
## Configuration
|
||||
|
||||
- `.env.example` documents non-secret values.
|
||||
- Startup validates configuration and reports all invalid fields.
|
||||
- Secrets use secret files/runtime injection when possible.
|
||||
- No environment-specific values embedded in images.
|
||||
- Production and test compose overrides are separate.
|
||||
- Feature capability detection is visible in UI/system status.
|
||||
|
||||
## Observability
|
||||
|
||||
Pulse emits:
|
||||
- structured logs;
|
||||
- internal metrics;
|
||||
- health/readiness;
|
||||
- job status;
|
||||
- trace/correlation IDs;
|
||||
- redacted upstream error classes.
|
||||
|
||||
Avoid recursive monitoring dependence: an external dead-man check must detect total Pulse failure.
|
||||
|
||||
## Documentation
|
||||
|
||||
Behavioral changes update:
|
||||
- relevant specification;
|
||||
- API/schema;
|
||||
- runbook if operational;
|
||||
- evidence;
|
||||
- current state;
|
||||
- ADR when architectural.
|
||||
@@ -0,0 +1,92 @@
|
||||
# Frontend standards
|
||||
|
||||
## Architecture
|
||||
|
||||
Recommended modules:
|
||||
|
||||
```text
|
||||
app-shell
|
||||
auth
|
||||
routing
|
||||
i18n
|
||||
design-system
|
||||
dashboards
|
||||
widgets
|
||||
metrics
|
||||
inventory
|
||||
alerts
|
||||
incidents
|
||||
events
|
||||
settings
|
||||
operations
|
||||
```
|
||||
|
||||
Feature modules own routes, queries, views and tests. Shared UI remains domain-neutral.
|
||||
|
||||
## State
|
||||
|
||||
- URL state for shareable filters/time range when appropriate.
|
||||
- Server state via TanStack Query or measured equivalent.
|
||||
- Editor draft state isolated from saved dashboard state.
|
||||
- WebSocket subscriptions managed by one client/service.
|
||||
- Chart samples in bounded local stores/ring buffers.
|
||||
- Avoid duplicating server state across stores.
|
||||
|
||||
## Dashboard editor
|
||||
|
||||
- Save complete version atomically.
|
||||
- Use explicit edit session/draft.
|
||||
- Detect optimistic concurrency conflict.
|
||||
- Undo/redo operates on normalized editor commands or bounded snapshots.
|
||||
- Breakpoint layouts validated before save.
|
||||
- Keyboard move/resize and screen-reader labels.
|
||||
- Prevent accidental drag from chart interactions.
|
||||
|
||||
## Charts
|
||||
|
||||
- Lazy-load heavy chart implementations.
|
||||
- Initialize history once per query.
|
||||
- Append live data efficiently.
|
||||
- Cap points/series.
|
||||
- Dispose observers/listeners/instances.
|
||||
- Pause offscreen/background work.
|
||||
- Provide textual summary/table alternative.
|
||||
- Use consistent units, timestamps, legend and tooltip behavior.
|
||||
- Status thresholds do not overwrite data meaning.
|
||||
|
||||
## Tables/lists
|
||||
|
||||
- Cursor/server pagination for large datasets.
|
||||
- Virtualization for large rendered collections.
|
||||
- Stable row keys.
|
||||
- Accessible sorting/filtering labels.
|
||||
- Preserve filter state sensibly.
|
||||
- Loading and empty states are distinct.
|
||||
|
||||
## Error handling
|
||||
|
||||
- Route-level boundary.
|
||||
- Component/query errors show safe, actionable messages.
|
||||
- Correlation ID exposed for diagnostics.
|
||||
- Retry only when safe and bounded.
|
||||
- Authentication expiry has a clean flow.
|
||||
- Partial datasource failure does not blank the entire app.
|
||||
|
||||
## Styling
|
||||
|
||||
- Use design tokens; no scattered literal colors/spacing.
|
||||
- Consistent card padding and grid gaps.
|
||||
- Limited elevation.
|
||||
- Status colors only for status.
|
||||
- Respect reduced motion.
|
||||
- Avoid oversized decorative headers that reduce information space.
|
||||
- Desktop and mobile screenshots/visual regression for key routes.
|
||||
|
||||
## Testing
|
||||
|
||||
- Unit tests for formatting/transforms/editor reducers.
|
||||
- Component tests for states and accessibility.
|
||||
- Playwright for user journeys.
|
||||
- Axe or equivalent on all core routes and viewports.
|
||||
- Real browser verification for drag/resize, WebSocket reconnect and wallboard.
|
||||
- Leak/soak instrumentation for chart/subscription lifecycle.
|
||||
@@ -0,0 +1,73 @@
|
||||
# Performance budgets
|
||||
|
||||
Budgets are acceptance targets measured in the documented test environment.
|
||||
|
||||
## Target scale
|
||||
|
||||
- 1 Unraid host;
|
||||
- 150 containers;
|
||||
- 40 disks;
|
||||
- 300 service probes;
|
||||
- 2,500 active series across an intensive dashboard set;
|
||||
- 10 concurrent authenticated users;
|
||||
- 1 wallboard open for at least 24 hours.
|
||||
|
||||
Container ingestion retains bounded headroom up to 250 records so a host that
|
||||
briefly grows beyond the 150-container performance target remains observable.
|
||||
The 150-container fixture remains the required latency and UI acceptance scale;
|
||||
the additional headroom is a safety boundary, not a higher performance claim.
|
||||
|
||||
## Browser
|
||||
|
||||
| Metric | Target |
|
||||
|---|---:|
|
||||
| First meaningful overview on LAN, warm service | < 2.0 s |
|
||||
| Main route interaction ready | < 3.0 s |
|
||||
| Live sample visual delay | < 2.5 s at 2 s interval |
|
||||
| Drag/resize frame behavior | no sustained visible jank |
|
||||
| 24 h wallboard heap | bounded; no monotonic leak |
|
||||
| Active subscriptions after navigation | returns to expected baseline |
|
||||
| Large table scroll | responsive with virtualization |
|
||||
|
||||
Record browser, hardware and network.
|
||||
|
||||
## API
|
||||
|
||||
| Metric | Target |
|
||||
|---|---:|
|
||||
| P95 cached/config API | < 250 ms |
|
||||
| P95 24 h bounded range query | < 750 ms excluding unavailable upstream |
|
||||
| P95 inventory list | < 500 ms at target scale |
|
||||
| WebSocket reconnect | automatic within 10 s under normal recovery |
|
||||
| Error response | bounded and correlated |
|
||||
|
||||
## Resource envelope
|
||||
|
||||
Initial production goals, to validate:
|
||||
- API/worker/agent combined idle memory should remain reasonable for Unraid;
|
||||
- CPU near idle outside query/evaluation bursts;
|
||||
- database growth predictable under retention;
|
||||
- no unbounded goroutines, queues, caches or event payloads.
|
||||
|
||||
Do not invent a pass. Record actual values and refine budgets through an ADR if hardware/source constraints provide evidence.
|
||||
|
||||
## Query limits
|
||||
|
||||
- max series and points per request;
|
||||
- max concurrent upstream requests;
|
||||
- step adjusted to viewport/time range;
|
||||
- heavy query rejection with guidance;
|
||||
- metadata/label enumeration bounded.
|
||||
|
||||
## Tests
|
||||
|
||||
- frontend bundle analysis;
|
||||
- Lighthouse or equivalent where meaningful;
|
||||
- scripted dashboard load;
|
||||
- WebSocket fan-out/load;
|
||||
- Prometheus slow/error injection;
|
||||
- real wallboard soak of at least 17 hours under the explicit M10-14
|
||||
product-owner duration decision;
|
||||
- worker/probe concurrency;
|
||||
- database query plans for large lists;
|
||||
- restart/recovery under load.
|
||||
@@ -0,0 +1,121 @@
|
||||
# Quality gates
|
||||
|
||||
## Per task
|
||||
|
||||
- Deliverables exist.
|
||||
- Acceptance checks pass.
|
||||
- Formatting/lint/type checks for changed code pass.
|
||||
- Relevant unit/integration/browser tests pass.
|
||||
- Diff review complete.
|
||||
- Docs/contracts updated.
|
||||
- Evidence summary complete.
|
||||
- No introduced secret or critical security issue.
|
||||
- A production Go package has at least one non-test importer reachable from a binary (`cmd/api`, `cmd/worker`, `cmd/agent`, `cmd/migrate`), verified by `python tools/check_wiring.py`; a task may not be marked done while its deliverable is unreachable, unless it is allowlisted in `tools/wiring_allowlist.json` with a reason and a tracking task id.
|
||||
- State updated via `projectctl`.
|
||||
|
||||
## Per milestone
|
||||
|
||||
- Every milestone task done.
|
||||
- Full milestone test set passes.
|
||||
- Architecture drift review.
|
||||
- Dependency/security/license check.
|
||||
- TODO/FIXME/skipped-test/debug scan.
|
||||
- Wiring/reachability scan (`python tools/check_wiring.py`): no package this milestone claims to deliver is unreachable and unallowlisted.
|
||||
- Migration/restart behavior where relevant.
|
||||
- UX/accessibility check for user-visible milestones.
|
||||
- Performance check for hot paths.
|
||||
- Milestone evidence index.
|
||||
- `python tools/projectctl.py gate <MILESTONE>` passes.
|
||||
|
||||
## M0 gate
|
||||
|
||||
- Repository/tooling/server discovery recorded.
|
||||
- Existing services/ports/networks/volumes/monitoring sources inventoried.
|
||||
- Backups/rollback plan for touched configs.
|
||||
- Current versions/capabilities verified from primary sources.
|
||||
- Architecture/security baseline reviewed.
|
||||
- M1-M9 plan adjusted to facts.
|
||||
- No destructive production change.
|
||||
|
||||
## M1 gate
|
||||
|
||||
- Clean local build.
|
||||
- API/web/worker/database start and health.
|
||||
- Migrations empty/restart/repeat.
|
||||
- Auth/RBAC skeleton and audit.
|
||||
- CI-equivalent checks.
|
||||
- No secrets.
|
||||
- Compose isolation/hardening baseline.
|
||||
|
||||
## M2 gate
|
||||
|
||||
- Prometheus and Unraid/mock adapters.
|
||||
- Inventory entities/relations/source ownership.
|
||||
- Discovery idempotency and source failure safety.
|
||||
- Datasource health/freshness.
|
||||
- API and UI inventory.
|
||||
- Target-scale reconciliation test.
|
||||
|
||||
## M3 gate
|
||||
|
||||
- Full dashboard CRUD/versioning/import/export.
|
||||
- Grid edit and per-breakpoint layouts.
|
||||
- Widget catalog/config states.
|
||||
- Undo/redo/restore/concurrency.
|
||||
- Desktop/mobile accessibility.
|
||||
- Browser persistence/reload proof.
|
||||
|
||||
## M4 gate
|
||||
|
||||
- Semantic metrics and bounded query planner.
|
||||
- Historical/live charts.
|
||||
- WebSocket auth, dedup, backpressure, reconnect.
|
||||
- Stale/unknown.
|
||||
- Performance/load and leak baseline.
|
||||
- Query security tests.
|
||||
|
||||
## M5 gate
|
||||
|
||||
- Host/process/container/application coverage.
|
||||
- Restart loop and application aggregation.
|
||||
- Events/detail pages/top-N/status.
|
||||
- Failure scenarios and scale.
|
||||
|
||||
## M6 gate
|
||||
|
||||
- Array/disks/SMART/pools/shares/capacity.
|
||||
- Read-only safety.
|
||||
- Storage stale/unknown and alerts inputs.
|
||||
- Simulated degradation.
|
||||
- No real destructive test.
|
||||
|
||||
## M7 gate
|
||||
|
||||
- Probe engine and SSRF controls.
|
||||
- TLS/DNS/network/service history.
|
||||
- Dependencies/topology.
|
||||
- Container-running/service-down detection.
|
||||
- Suppression inputs.
|
||||
|
||||
## M8 gate
|
||||
|
||||
- Rule versions/state machine/hysteresis.
|
||||
- Unknown/silence/maintenance/suppression.
|
||||
- Notifications audit.
|
||||
- Incident grouping/timeline/notes.
|
||||
- Alert storm scenario produces expected grouping.
|
||||
- Concurrency/restart tests.
|
||||
|
||||
## M9/final gate
|
||||
|
||||
- Complete Dutch UX, mobile and wallboard.
|
||||
- Accessibility and performance budgets.
|
||||
- Security hardening/scans.
|
||||
- Real wallboard soak of at least 17 hours. This supersedes the original
|
||||
24-hour duration only through the explicit product-owner decision recorded
|
||||
for M10-14 on 2026-08-11; all other continuity and performance budgets remain.
|
||||
- Backup/restore and upgrade/rollback.
|
||||
- Clean-room install.
|
||||
- Production deployment, restart and smoke.
|
||||
- Final requirement/evidence matrix.
|
||||
- Runbook and current state accurate.
|
||||
@@ -0,0 +1,137 @@
|
||||
# Test strategy
|
||||
|
||||
## Test pyramid and evidence
|
||||
|
||||
Tests prove behavior at the cheapest reliable level, but critical workflows require end-to-end proof.
|
||||
|
||||
## 1. Unit tests
|
||||
|
||||
Required for:
|
||||
- status aggregation;
|
||||
- freshness/staleness;
|
||||
- metric transformations and units;
|
||||
- query limit calculations;
|
||||
- inventory identity/reconciliation;
|
||||
- dashboard validation/migration/editor reducer;
|
||||
- alert state machine, hysteresis and suppression;
|
||||
- incident correlation rules;
|
||||
- authorization policy;
|
||||
- probe target validation/SSRF rules;
|
||||
- formatting/localization.
|
||||
|
||||
Use deterministic clocks and fixtures.
|
||||
|
||||
## 2. Contract tests
|
||||
|
||||
For:
|
||||
- Prometheus responses/errors/warnings;
|
||||
- Unraid API capabilities and payload variants;
|
||||
- agent protocol;
|
||||
- OIDC claims;
|
||||
- notification connectors;
|
||||
- the isolated real-stack gate (`scripts/integration-smoke.ps1`) for collector → PostgreSQL → API → UI and alert → webhook delivery;
|
||||
- dashboard/live/event schemas.
|
||||
|
||||
Captured fixtures must be redacted and versioned.
|
||||
|
||||
## 3. Integration tests
|
||||
|
||||
Use real PostgreSQL through Testcontainers or equivalent.
|
||||
|
||||
Cover:
|
||||
- migrations from empty and prior versions;
|
||||
- transaction/concurrency;
|
||||
- optimistic locking;
|
||||
- discovery/reconciliation;
|
||||
- alert evaluation/outbox;
|
||||
- backup/restore;
|
||||
- API authorization;
|
||||
- WebSocket persistence/reconnect interactions where practical.
|
||||
|
||||
Prometheus and Unraid can use deterministic simulators/fake servers, plus optional non-destructive live contract checks.
|
||||
|
||||
## 4. End-to-end browser tests
|
||||
|
||||
Playwright core flows:
|
||||
- login/session;
|
||||
- overview healthy/degraded/unknown;
|
||||
- dashboard create/edit/drag/resize/config/save/reload;
|
||||
- version restore and conflict;
|
||||
- time range and cross-filter;
|
||||
- entity drill-down;
|
||||
- alert acknowledge/silence;
|
||||
- incident view/note;
|
||||
- mobile navigation;
|
||||
- wallboard reconnect;
|
||||
- permission differences;
|
||||
- source failure and recovery.
|
||||
|
||||
Run desktop and mobile viewports. Add a wallboard viewport.
|
||||
|
||||
## 5. Accessibility
|
||||
|
||||
Automated checks on every core route/state:
|
||||
- desktop and mobile;
|
||||
- keyboard flow;
|
||||
- focus after modal/drawer/drag alternative;
|
||||
- status without color;
|
||||
- chart summary/alternative;
|
||||
- reduced motion.
|
||||
|
||||
Manual spot checks for screen-reader naming and dashboard editor keyboard behavior.
|
||||
|
||||
## 6. Performance/load/soak
|
||||
|
||||
- API benchmarks and P95 load tests.
|
||||
- WebSocket clients/subscriptions/fan-out.
|
||||
- Query dedup/cache behavior.
|
||||
- 24-hour wallboard heap/subscription/resource soak.
|
||||
- 150 container/40 disk/300 probe fixture scale.
|
||||
- slow Prometheus/database/agent recovery.
|
||||
- frontend rendering with maximum supported widgets.
|
||||
|
||||
## 7. Security
|
||||
|
||||
- RBAC matrix per endpoint and WebSocket message.
|
||||
- OIDC state/nonce/issuer/audience.
|
||||
- CSRF/cookie/origin.
|
||||
- SSRF, redirect, DNS rebinding, metadata targets.
|
||||
- XSS through names/events/import/Markdown.
|
||||
- query template and raw query limits.
|
||||
- rate/body/message limits.
|
||||
- secret scan and diagnostic redaction.
|
||||
- dependency/image/static scan.
|
||||
- compose mounts/capabilities/network exposure.
|
||||
- backup contents.
|
||||
|
||||
## 8. Failure simulation
|
||||
|
||||
Use `fixtures/scenarios/`:
|
||||
- stale Prometheus;
|
||||
- source disconnect;
|
||||
- CPU saturation;
|
||||
- memory pressure/OOM;
|
||||
- container restart loop;
|
||||
- service down while container runs;
|
||||
- disk temperature;
|
||||
- SMART warning;
|
||||
- cache/pool pressure;
|
||||
- array degradation fixture;
|
||||
- DNS/gateway failure;
|
||||
- UPS on battery;
|
||||
- WebSocket slow client/reconnect;
|
||||
- database restart.
|
||||
|
||||
Never induce destructive real faults.
|
||||
|
||||
## 9. Gate behavior
|
||||
|
||||
A failing required test:
|
||||
- keeps task/milestone incomplete;
|
||||
- is diagnosed and repaired;
|
||||
- may be quarantined only for a proven external nondeterministic issue, with owner, expiry and alternate evidence;
|
||||
- is never simply deleted or skipped.
|
||||
|
||||
## 10. Clean-room
|
||||
|
||||
Final release is built/deployed from a clean checkout using documented inputs, no developer `.env`, caches or untracked files. Migrations, seed/default dashboards, auth config, health, smoke and restart are verified.
|
||||
@@ -0,0 +1,128 @@
|
||||
# Backup and restore
|
||||
|
||||
## Backup contents
|
||||
|
||||
- PostgreSQL logical backup or selected physical method with documented compatibility;
|
||||
- Pulse configuration;
|
||||
- dashboard versions;
|
||||
- alert rules and maintenance windows;
|
||||
- inventory/events/incidents/audit according to policy;
|
||||
- non-secret datasource/channel configuration;
|
||||
- schema/application version;
|
||||
- image digests;
|
||||
- checksums and manifest.
|
||||
|
||||
Secrets:
|
||||
- are backed up only through an explicitly secured secret-store procedure;
|
||||
- never appear plaintext in portable exports;
|
||||
- are documented as required reattachment steps if excluded.
|
||||
|
||||
## Backup behavior
|
||||
|
||||
- on-demand administrator action;
|
||||
- scheduled optional job;
|
||||
- unique immutable backup ID;
|
||||
- temporary file + atomic finalize;
|
||||
- checksum;
|
||||
- size and duration;
|
||||
- success/failure audit;
|
||||
- retention;
|
||||
- destination health;
|
||||
- no backup only inside the source database volume.
|
||||
|
||||
## Verification
|
||||
|
||||
A backup is not trusted until:
|
||||
- checksum passes;
|
||||
- archive/format opens;
|
||||
- manifest/schema version valid;
|
||||
- secret scan/redaction policy passes;
|
||||
- restore test has succeeded for the release.
|
||||
|
||||
## Restore test
|
||||
|
||||
Use an empty isolated environment:
|
||||
|
||||
1. deploy compatible database/application version;
|
||||
2. restore backup;
|
||||
3. run migrations if expected;
|
||||
4. start services;
|
||||
5. authenticate;
|
||||
6. verify dashboards/rules/inventory/events/incidents;
|
||||
7. query metrics datasource through restored config;
|
||||
8. run smoke tests;
|
||||
9. compare key counts/checksums;
|
||||
10. document credential reattachment.
|
||||
|
||||
Do not overwrite production during routine validation.
|
||||
|
||||
## Upgrade rollback
|
||||
|
||||
Before schema upgrade:
|
||||
- create verified backup;
|
||||
- record current migration version and image digests;
|
||||
- test upgrade from prior release fixture;
|
||||
- test supported downgrade or forward-recovery strategy;
|
||||
- document compatibility window.
|
||||
|
||||
If down migration is unsafe, use restore/forward-fix and state this explicitly.
|
||||
|
||||
### Supported M9-10 recovery path
|
||||
|
||||
Pulse migrations are forward-only. Do not manually remove rows from
|
||||
`schema_migrations`, run ad-hoc down SQL, or point an older image at a schema
|
||||
whose compatibility has not been proven. The supported recovery path after a
|
||||
failed upgrade is:
|
||||
|
||||
1. Freeze only the Pulse compose project and preserve the migration/API logs.
|
||||
2. Record the image digests, `schema_migrations` rows and the verified backup
|
||||
ID before making another attempt.
|
||||
3. Correct the release/configuration fault, use the same or a newer compatible
|
||||
Pulse image, and re-run `cmd/migrate`. Each migration is committed with its
|
||||
schema-migration record in one transaction and repeat runs are safe.
|
||||
4. Restart the Pulse API/worker/agent, then verify health, schema migration
|
||||
count, dashboard/rule current-version links and the operational smoke
|
||||
checks.
|
||||
5. If forward recovery cannot be completed, build a new empty isolated target,
|
||||
restore the last verified portable backup, reattach excluded secrets, run
|
||||
compatible migrations and validate it before any production cutover.
|
||||
|
||||
The old image/configuration is a rollback point only when its documented
|
||||
schema compatibility is satisfied. M9-10 verified the baseline `61cb2a4` to
|
||||
`11e3496` transition, a repeat migration and a PostgreSQL transaction fault
|
||||
rollback. It does not claim that arbitrary future down-migrations are safe.
|
||||
|
||||
## Recovery objectives
|
||||
|
||||
Codex must measure and document practical RPO/RTO for the environment. Do not invent guarantees.
|
||||
|
||||
Baseline intent:
|
||||
- configuration/incident loss limited by backup schedule;
|
||||
- restore procedure executable by the operator;
|
||||
- no dependency on the failed Pulse API to access backup instructions.
|
||||
|
||||
## M9-09 portable backup
|
||||
|
||||
Configure a protected, operator-owned destination with `PULSE_BACKUP_DIR` and an optional retention count with `PULSE_BACKUP_RETENTION` (default 5, maximum 100). The API exposes the following administrator-only endpoint:
|
||||
|
||||
- `GET /api/v1/system/backups` lists verified backup archives.
|
||||
- `POST /api/v1/system/backups` creates a temporary archive, writes it atomically, emits a sidecar SHA-256 checksum and applies retention.
|
||||
|
||||
Format version 2 of the archive is a ZIP containing a manifest and deterministic JSONL entries for the approved non-secret persistence tables. Durable discovery identity in `container_aliases` is included so a restored worker does not assign new entity identities to unchanged containers. Runtime configuration, `system_settings`, notification channel configuration, secret references, notification delivery state and bounded `agent_snapshots` and `capacity_samples` runtime telemetry are excluded and must be reattached or republished through the approved procedure. The exporter rejects sensitive JSON keys rather than writing them. The manifest records the migration count, table row counts and per-entry SHA-256 values. Restore first verifies the archive and sidecar, then only restores into a database where every backed-up table is empty, inside one transaction. Current dashboard/rule version foreign keys are restored after their version rows exist. Version 1 archives predate durable container-alias coverage and are deliberately rejected instead of being accepted as complete recovery points.
|
||||
|
||||
The PostgreSQL clean-room test classifies every migrated application table as
|
||||
either portable or deliberately excluded and validates every configured export
|
||||
and restore column against the migrated schema. CI sets
|
||||
`PULSE_REQUIRE_BACKUP_INTEGRATION=true` and provisions separate source and
|
||||
restore databases, so a missing DSN or a newly unclassified table fails rather
|
||||
than silently skipping this release gate.
|
||||
|
||||
A reproducible DSN-gated test is:
|
||||
|
||||
```text
|
||||
$env:PULSE_TEST_DATABASE_URL = 'postgresql://...'
|
||||
$env:PULSE_TEST_RESTORE_DATABASE_URL = 'postgresql://...'
|
||||
go test ./internal/backup -run TestPostgreSQLBackupRestoreCleanRoom -count=1 -v
|
||||
```
|
||||
|
||||
Do not put actual DSNs, passwords, secret-store values or backup bytes in the repository, evidence or shell history. The restore test uses an isolated empty environment and never overwrites production.
|
||||
@@ -0,0 +1,61 @@
|
||||
# Development setup
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Git;
|
||||
- Go 1.26.6;
|
||||
- Node.js 24 or newer and pnpm 10.33 or newer;
|
||||
- Python 3 with the packages in `requirements-dev.txt`;
|
||||
- PowerShell 7;
|
||||
- Docker Engine and Docker Compose;
|
||||
- Chromium for Playwright end-to-end tests.
|
||||
|
||||
## Bootstrap
|
||||
|
||||
From a clean checkout:
|
||||
|
||||
```powershell
|
||||
Copy-Item .env.example .env
|
||||
pwsh -NoProfile -File scripts/bootstrap.ps1
|
||||
```
|
||||
|
||||
The example environment uses mock authentication and local-only credentials. Never reuse production credentials, backups, dashboards, telemetry, or host inventories.
|
||||
|
||||
## Run the development stack
|
||||
|
||||
```powershell
|
||||
docker compose -f deploy/compose.yaml -f deploy/compose.dev.yaml up --build
|
||||
```
|
||||
|
||||
The development override publishes the web, API, and PostgreSQL ports for local testing. Treat those ports as development-only and use a host firewall when working on an untrusted network.
|
||||
|
||||
Stop only this stack with:
|
||||
|
||||
```powershell
|
||||
docker compose -f deploy/compose.yaml -f deploy/compose.dev.yaml down --volumes
|
||||
```
|
||||
|
||||
## Validation
|
||||
|
||||
For a public source checkout:
|
||||
|
||||
```powershell
|
||||
pwsh -NoProfile -File scripts/public-verify.ps1
|
||||
```
|
||||
|
||||
The optional real source-to-browser smoke uses an isolated Compose project and tears it down after the run:
|
||||
|
||||
```powershell
|
||||
pwsh -NoProfile -File scripts/integration-smoke.ps1
|
||||
```
|
||||
|
||||
PostgreSQL integration tests use explicit process-local test DSNs. Never point them at a database that contains data you need to keep.
|
||||
|
||||
## Production separation
|
||||
|
||||
- production uses OIDC and rejects mock authentication;
|
||||
- the base/production profiles do not publish PostgreSQL or the API;
|
||||
- runtime secrets remain outside Git;
|
||||
- fixtures are synthetic;
|
||||
- the agent remains read-only and receives no Docker socket;
|
||||
- production changes follow [`../PUBLIC_DEPLOYMENT.md`](../PUBLIC_DEPLOYMENT.md), not the development procedure.
|
||||
@@ -0,0 +1,87 @@
|
||||
# Observability of Pulse
|
||||
|
||||
Pulse must expose its own health without creating a circular single point of truth.
|
||||
|
||||
## Internal metrics
|
||||
|
||||
At minimum:
|
||||
- API request count/duration/error by bounded route code;
|
||||
- active WebSocket clients/subscriptions;
|
||||
- dropped/coalesced live samples;
|
||||
- Prometheus query duration/errors/series/points/cache;
|
||||
- discovery runs and reconciliation counts;
|
||||
- worker job duration/failure/lag;
|
||||
- alert evaluation duration/state transitions;
|
||||
- notification attempts/failures;
|
||||
- probe queue/concurrency/results;
|
||||
- database pool/queries/migration;
|
||||
- agent heartbeat/capabilities/errors;
|
||||
- backup success/age;
|
||||
- process CPU/memory/goroutines.
|
||||
|
||||
Avoid high-cardinality path/user/entity labels.
|
||||
|
||||
## Health endpoints
|
||||
|
||||
- liveness: process loop alive, no dependency checks that cause restart storms;
|
||||
- readiness: mandatory dependencies usable;
|
||||
- detailed system status: authenticated, shows component state/freshness.
|
||||
|
||||
## Logs
|
||||
|
||||
Structured production logs:
|
||||
- timestamp;
|
||||
- level;
|
||||
- service/version;
|
||||
- correlation/request/job ID;
|
||||
- operation;
|
||||
- safe entity/resource IDs;
|
||||
- error code and wrapped message;
|
||||
- duration/result.
|
||||
|
||||
Redact:
|
||||
- tokens/cookies/authorization;
|
||||
- passwords/connection secrets;
|
||||
- sensitive headers;
|
||||
- full upstream bodies;
|
||||
- private keys.
|
||||
|
||||
## External dead-man
|
||||
|
||||
An independent existing monitor or simple external check should detect:
|
||||
- Pulse HTTPS unavailable;
|
||||
- Pulse heartbeat absent;
|
||||
- backup too old if supported.
|
||||
|
||||
Do not rely solely on Pulse to alert that Pulse is completely down.
|
||||
|
||||
## System status page
|
||||
|
||||
Shows:
|
||||
- build/version/commit;
|
||||
- database;
|
||||
- Prometheus;
|
||||
- Unraid/agent;
|
||||
- OIDC;
|
||||
- worker heartbeats;
|
||||
- query health;
|
||||
- probes;
|
||||
- notifications;
|
||||
- backup age;
|
||||
- active clients;
|
||||
- storage use;
|
||||
- last successful critical jobs.
|
||||
|
||||
No secret/config values.
|
||||
|
||||
## M9-08 operating contract
|
||||
|
||||
The public liveness contract is GET /healthz and returns exactly ok with HTTP 200 when the Pulse process is serving requests. It deliberately does not query PostgreSQL or upstream datasources, so an independent monitor can distinguish a dead Pulse process from a dependency outage. GET /readyz is the dependency-aware readiness contract and returns HTTP 503 while the database is unavailable.
|
||||
|
||||
The authenticated status surface is GET /api/v1/system/status and is available to viewers. It reports bounded component state, explicit Unknown/Disabled reasons, backup age (Unknown until a verified backup exists), and source lag (Unknown until a source reports an observed timestamp). It never reports unsampled sources as Healthy. Operators with the operate permission can read /api/v1/system/diagnostics and /api/v1/system/metrics; these routes are not public monitoring endpoints.
|
||||
|
||||
Configure an independent monitor outside Pulse to run every 60 seconds with two consecutive failures required for alerting:
|
||||
|
||||
python tools/deadman_check.py https://pulse.example.invalid/healthz --timeout 5
|
||||
|
||||
The script uses only the Python standard library, refuses embedded credentials, bounds the timeout to 1-30 seconds, accepts only HTTP 200 with the exact ok body, and exits non-zero for transport, timeout, redirect-result, or HTTP failures. Keep the monitor and its alerting path independent of the Pulse API and database.
|
||||
@@ -0,0 +1,118 @@
|
||||
# Worker/agent healthcheck contract
|
||||
|
||||
Applies to `pulse-worker` (`cmd/worker`) and `pulse-agent` (`cmd/agent`). Both
|
||||
are currently foundation stubs (`cmd/worker/main.go`, `cmd/agent/main.go`)
|
||||
that will gain real scheduling loops. This document is the contract the Go
|
||||
implementation must satisfy so the compose-side healthcheck already wired in
|
||||
`deploy/compose.yaml` reports true liveness instead of "process exists".
|
||||
|
||||
## Why not an HTTP `/healthz`
|
||||
|
||||
`pulse-api` and `pulse-web` already expose a network port, so an HTTP
|
||||
`/healthz` costs nothing extra there. `pulse-worker` and `pulse-agent` sit on
|
||||
`pulse-internal` only and have no other reason to bind a port; opening one
|
||||
solely for a healthcheck would be an unnecessary internal attack surface
|
||||
(see `apps/agent/AGENTS.md`: "No generic shell/exec/file-write endpoint" and
|
||||
the general least-privilege posture of ADR-0005/ADR-0010). A heartbeat file
|
||||
on the already-mounted `tmpfs` `/tmp` needs no new port, no new capability,
|
||||
and no new listener.
|
||||
|
||||
## What the compose/image side already does
|
||||
|
||||
- `deploy/pulse-entrypoint.sh` is the container `ENTRYPOINT`. It writes the
|
||||
current unix time to `/tmp/.pulse-started-at` and then `exec`s the real
|
||||
binary (PID 1 becomes the Go process). This exists only so the healthcheck
|
||||
script can grant a startup grace period independent of Docker's
|
||||
`start_period`, which suppresses unhealthy *status* but does not stop the
|
||||
`HEALTHCHECK CMD` (or its side effects) from running during that window.
|
||||
- `deploy/healthcheck-heartbeat.sh` is the `HEALTHCHECK CMD` (also wired
|
||||
explicitly in `deploy/compose.yaml`, `interval=15s timeout=5s
|
||||
start_period=20s retries=3`). It checks the mtime of the heartbeat file
|
||||
described below. If the file is missing for longer than
|
||||
`PULSE_HEARTBEAT_MAX_AGE_SECONDS` (default **45s**) after container start,
|
||||
or exists but its mtime is older than that same threshold, the script logs
|
||||
why and sends `SIGKILL` to PID 1. `docker compose up` (non-swarm) does not
|
||||
restart a container merely because it is reported "unhealthy" — killing
|
||||
PID 1 turns that into a real container exit, which `restart:
|
||||
unless-stopped` then recovers automatically. Same-UID `SIGKILL` needs no
|
||||
Linux capability, so this works under `cap_drop: [ALL]`.
|
||||
|
||||
You do not need to change either script or the Dockerfiles to implement this
|
||||
contract — only the points below, inside `cmd/worker` / `cmd/agent` and
|
||||
whatever internal packages they call.
|
||||
|
||||
## What the Go side must implement
|
||||
|
||||
1. **Path.** Write the heartbeat to the file named by the
|
||||
`PULSE_HEARTBEAT_FILE` environment variable if set, otherwise
|
||||
`/tmp/healthy`. That path is on the container's `tmpfs` `/tmp` mount
|
||||
(already present in `deploy/compose.yaml` for both services), so no new
|
||||
volume or mount is needed.
|
||||
2. **Cadence tied to real progress, not a free-running ticker.** Update the
|
||||
heartbeat only after the main scheduling loop completes an iteration (a
|
||||
scheduler tick, a due-job scan, a lease renewal, a completed
|
||||
discovery/capability-collection pass — whatever the loop's unit of work
|
||||
is for that binary). Do **not** heartbeat from an independent goroutine
|
||||
that ticks on a timer regardless of whether the main loop is stuck — that
|
||||
would silently defeat the whole point of this contract (the exact failure
|
||||
mode `kill -0 1` had).
|
||||
3. **Interval.** The loop must complete an iteration, and therefore heartbeat,
|
||||
at least every **10 seconds** even when there is no work to do (an empty
|
||||
poll is still a completed iteration). This gives a comfortable margin
|
||||
under the 45s staleness threshold enforced by
|
||||
`deploy/healthcheck-heartbeat.sh`, tolerating a couple of missed/slow
|
||||
cycles before the container is killed.
|
||||
4. **First heartbeat.** Write one heartbeat immediately after startup
|
||||
(config loaded, DB reachable) and before blocking on the first real unit
|
||||
of work, so a slow-but-healthy cold start is not mistaken for a hang.
|
||||
5. **Bounded blocking calls.** Every blocking call inside the loop iteration
|
||||
(DB queries, Unraid/agent-protocol calls, probe dials, notification
|
||||
sends) must use a `context` with a timeout well under 10s, and
|
||||
long-running job execution must happen in a separate goroutine so the
|
||||
scheduler loop itself stays responsive. This is also required by
|
||||
`apps/worker/AGENTS.md` ("Every job is idempotent, cancellable, bounded
|
||||
and observable") independent of this contract — the two requirements
|
||||
reinforce each other: if a call isn't bounded, the loop stalls, the
|
||||
heartbeat goes stale, and the container is killed and restarted,
|
||||
correctly surfacing the hang instead of hiding it.
|
||||
6. **Write semantics.** A simple truncate-and-write (or an `os.Chtimes` touch
|
||||
if content does not need to change) is sufficient; the healthcheck reads
|
||||
only the file's mtime, not its contents, so no locking or atomic
|
||||
rename is required for correctness. For operator debuggability, write the
|
||||
current UTC time in RFC 3339 (e.g. `2026-08-04T12:00:03Z\n`) as the file
|
||||
content so `docker exec <container> cat /tmp/healthy` is meaningful.
|
||||
7. **Failure handling.** If writing the heartbeat file itself fails (e.g.
|
||||
tmpfs write error), log it at error level and continue the loop — do not
|
||||
crash the process for a heartbeat I/O failure alone. A sustained write
|
||||
failure will naturally show up as staleness and get caught by the
|
||||
healthcheck.
|
||||
8. **Shutdown.** No special handling is required on `SIGTERM`/graceful
|
||||
shutdown; leaving the heartbeat file in place is fine since the container
|
||||
is stopping anyway (`service.WaitForStop` already handles the
|
||||
signal-driven shutdown path in both `cmd/worker/main.go` and
|
||||
`cmd/agent/main.go`).
|
||||
|
||||
## Tunables
|
||||
|
||||
| Env var | Default | Set where | Meaning |
|
||||
|---|---|---|---|
|
||||
| `PULSE_HEARTBEAT_FILE` | `/tmp/healthy` | worker/agent process env (compose) | Path the Go process writes and the healthcheck script reads. |
|
||||
| `PULSE_HEARTBEAT_MAX_AGE_SECONDS` | `45` | healthcheck script env (compose, if overridden) | Staleness threshold before the healthcheck force-restarts the container. |
|
||||
| `PULSE_STARTED_AT_FILE` | `/tmp/.pulse-started-at` | entrypoint/healthcheck script env (compose, if overridden) | Where the entrypoint records container start time, used only for startup grace. |
|
||||
|
||||
None of these are currently set as explicit environment variables in
|
||||
`deploy/compose.yaml` (the defaults baked into the scripts are used); add
|
||||
them there if the recommended values above ever need to change per
|
||||
deployment.
|
||||
|
||||
## Reviewer double-check
|
||||
|
||||
Self-terminating a container from inside its own healthcheck is an unusual
|
||||
pattern. It was chosen because `docker compose up` (non-swarm, confirmed by
|
||||
`docs/operations/DEVELOPMENT_SETUP.md`) does not auto-restart merely
|
||||
"unhealthy" containers, and an external auto-heal watcher would need Docker
|
||||
socket access, which ADR-0005 forbids host-wide. If that trade-off is
|
||||
unacceptable, the alternative is to drop the `kill -9 1` and rely on the
|
||||
Unraid dashboard surfacing "unhealthy" status for manual operator action —
|
||||
but that leaves the original "never restarts" finding only partially fixed
|
||||
(visible, not self-healing).
|
||||
@@ -0,0 +1,94 @@
|
||||
# Host, container and application monitoring requirements
|
||||
|
||||
## Host
|
||||
|
||||
Metrics/state:
|
||||
- uptime and boot time;
|
||||
- CPU total/per core, load, frequency where available;
|
||||
- iowait, interrupts and context switches;
|
||||
- memory used/available/cache/swap and pressure;
|
||||
- filesystem usage/inodes;
|
||||
- network per interface, errors/drops;
|
||||
- process count and OOM events;
|
||||
- temperatures/fans/sensors where supported;
|
||||
- GPU usage/memory/temperature where supported;
|
||||
- time synchronization health.
|
||||
|
||||
Host status must distinguish:
|
||||
- high but expected load;
|
||||
- sustained saturation;
|
||||
- missing collector;
|
||||
- stale metrics;
|
||||
- thermal risk;
|
||||
- resource exhaustion.
|
||||
|
||||
## Process explorer
|
||||
|
||||
Read-only, bounded view:
|
||||
- top CPU;
|
||||
- top memory;
|
||||
- PID, name, state, runtime;
|
||||
- container association when known;
|
||||
- process tree on demand.
|
||||
|
||||
No process termination. Process command-line arguments, environment and working directory are not shown by default; process rows are bounded and read-only.
|
||||
|
||||
## Containers
|
||||
|
||||
Inventory:
|
||||
- ID/name;
|
||||
- image/tag/digest when available;
|
||||
- state and health;
|
||||
- uptime/start/stop;
|
||||
- restart count and exit code;
|
||||
- CPU/memory;
|
||||
- network;
|
||||
- block I/O;
|
||||
- ports;
|
||||
- volumes/networks;
|
||||
- stack/project and labels.
|
||||
|
||||
Behavior:
|
||||
- preserve runtime state separately from health;
|
||||
- expose intentional stop explicitly so a stopped container is not silently treated as healthy;
|
||||
- retain source freshness and provenance;
|
||||
- expose metric and lifecycle availability explicitly; absent collector fields render as Unknown rather than numeric zero;
|
||||
- bound container count and detail collections for predictable response size;
|
||||
- detect restart loops over a rolling window;
|
||||
- distinguish intentionally stopped/disabled;
|
||||
- preserve history across container recreation through stable identity mapping;
|
||||
- never infer application health solely from Docker `running`.
|
||||
|
||||
### Container identity and recreation
|
||||
|
||||
Runtime IDs are source-scoped aliases. A changed runtime ID maps to the same logical container only when the source, compose project and compose service form a unique stable key. Reusing a name without that evidence creates a new logical entity; ambiguous stable keys also refuse to merge. Historical aliases remain visible to reconciliation, and lifecycle events retain the logical entity ID while recording the prior and current runtime IDs.
|
||||
## Lifecycle events and instability
|
||||
|
||||
Container transitions are normalized into bounded lifecycle events for state, health, restart and intentional-stop changes. Duplicate source/dedup/time events are discarded. A restart loop requires the current container state to be running and at least the configured number of restarts inside the bounded window; a stopped container is not labeled as looping. Ranked resource consumers use deterministic ID/label tie-breaks, and status grids retain text reasons alongside status indicators.
|
||||
## Applications
|
||||
|
||||
Applications group containers and services into a policy-driven status. Critical component failures degrade the application; critical unknown state remains Unknown. Optional component failures are included in the reasons list and degrade the aggregate without being labeled critical. A service-down state degrades the component even when its container runtime is still running. User overrides for application names and component criticality are applied after discovery and survive repeated discovery projections.
|
||||
|
||||
Applications group one or more containers and services.
|
||||
|
||||
Fields:
|
||||
- friendly name;
|
||||
- critical/optional components;
|
||||
- dependencies;
|
||||
- public/internal URLs;
|
||||
- owner/category/tags;
|
||||
- aggregate status reasons.
|
||||
|
||||
Aggregate status is policy-driven and tested. An optional background worker may fail without making the complete application critical, while a database or main endpoint failure normally degrades it.
|
||||
|
||||
## Events
|
||||
|
||||
Generate normalized events for:
|
||||
- container start/stop/restart/die/health change;
|
||||
- image or configuration change;
|
||||
- application status change;
|
||||
- datasource loss/recovery;
|
||||
- host reboot;
|
||||
- OOM and thermal events.
|
||||
|
||||
Deduplicate noisy repeated events.
|
||||
@@ -0,0 +1,238 @@
|
||||
# ITWorx Pulse — Product requirements
|
||||
|
||||
## 1. Product statement
|
||||
|
||||
ITWorx Pulse is a self-hosted operational observability platform for a personal/professional Unraid environment. It combines infrastructure, container, storage, network and service health in a configurable dashboard that is easier to understand and operate than a collection of raw monitoring tools.
|
||||
|
||||
Pulse is not a thin Grafana theme. It owns:
|
||||
|
||||
- inventory and relationships;
|
||||
- semantic metrics;
|
||||
- dashboard composition;
|
||||
- status reasoning;
|
||||
- alert lifecycle;
|
||||
- incident grouping;
|
||||
- user experience;
|
||||
- operational documentation and audit.
|
||||
|
||||
Prometheus-compatible systems remain the primary source of time-series history in v1.
|
||||
|
||||
## 2. Primary user
|
||||
|
||||
The primary user is a technically experienced server owner/operator who:
|
||||
|
||||
- manages an Unraid host with many Docker applications;
|
||||
- wants one reliable status surface;
|
||||
- needs detailed drill-down without constant PromQL work;
|
||||
- uses desktop, mobile and a possible wallboard;
|
||||
- values safe automation and read-only monitoring;
|
||||
- may later link incidents to a separate management platform.
|
||||
|
||||
The architecture supports additional viewers/operators but v1 does not need multi-tenant SaaS behavior.
|
||||
|
||||
## 3. Goals
|
||||
|
||||
1. Determine within seconds whether the environment is healthy.
|
||||
2. Explain every degraded/critical/unknown state.
|
||||
3. Allow dashboards to be composed without code.
|
||||
4. Show smooth live behavior and useful history.
|
||||
5. Detect container, storage and service failure independently.
|
||||
6. Prevent alert storms and group related failures.
|
||||
7. Remain safe when telemetry is stale or unavailable.
|
||||
8. Deploy without disrupting existing Unraid services.
|
||||
9. Provide reproducible evidence for operation, backup and recovery.
|
||||
10. Be portfolio-quality in visual, technical and operational terms.
|
||||
|
||||
## 4. Non-goals for v1
|
||||
|
||||
- General log aggregation/search platform.
|
||||
- Kubernetes monitoring.
|
||||
- Multi-organization SaaS.
|
||||
- Automatic remediation.
|
||||
- Container lifecycle management.
|
||||
- Array/storage write operations.
|
||||
- Public unauthenticated monitoring.
|
||||
- AI-required diagnosis.
|
||||
- Unlimited third-party plugin marketplace.
|
||||
- Replacing Prometheus or Grafana without measured need.
|
||||
- Monitoring arbitrary internet targets without an allowlist.
|
||||
|
||||
## 5. Functional capabilities
|
||||
|
||||
### 5.1 Authentication and access
|
||||
|
||||
- Authentik-compatible OIDC login.
|
||||
- Viewer, Operator, Editor and Administrator roles.
|
||||
- Disabled-by-default local break-glass recovery.
|
||||
- Session expiry and logout.
|
||||
- Audit for security/configuration changes.
|
||||
- Wallboard uses a constrained read-only mode or authenticated session.
|
||||
|
||||
### 5.2 Overview
|
||||
|
||||
The default overview shows:
|
||||
|
||||
- global status and explanation;
|
||||
- active incidents/alerts;
|
||||
- uptime, CPU, memory and load;
|
||||
- array and pool status/capacity;
|
||||
- network traffic;
|
||||
- container/application health;
|
||||
- service reachability and latency;
|
||||
- UPS when available;
|
||||
- recent events;
|
||||
- freshness of each source.
|
||||
|
||||
### 5.3 Dashboards
|
||||
|
||||
Users can:
|
||||
|
||||
- create, clone, rename, archive and delete dashboards;
|
||||
- add widgets from a catalog;
|
||||
- drag, resize, lock, duplicate, hide and remove widgets;
|
||||
- configure data, visualization, thresholds and behavior;
|
||||
- preview and edit desktop/tablet/mobile/wallboard layouts;
|
||||
- use variables and cross-filtering;
|
||||
- undo/redo during editing;
|
||||
- save versioned revisions and restore previous versions;
|
||||
- import/export validated JSON;
|
||||
- use system templates;
|
||||
- open widgets fullscreen and drill into entities.
|
||||
|
||||
### 5.4 Inventory and relationships
|
||||
|
||||
Pulse discovers and maintains:
|
||||
|
||||
- host;
|
||||
- hardware and interfaces;
|
||||
- array, pools, disks, filesystems and shares;
|
||||
- containers, images, stacks, volumes and networks;
|
||||
- applications and services;
|
||||
- endpoints and certificates;
|
||||
- data sources, collectors and probes;
|
||||
- dependency relationships.
|
||||
|
||||
Every reconciled field retains source ownership and optional user override.
|
||||
|
||||
### 5.5 Metrics and charts
|
||||
|
||||
- Historical range queries.
|
||||
- Live updates with bounded memory.
|
||||
- Semantic metric catalog.
|
||||
- Units, transformations and aggregation.
|
||||
- Zoom, hover, pause, time range and compare.
|
||||
- Stale/unknown representation.
|
||||
- Query limits and authorization.
|
||||
- CSV/image export where appropriate.
|
||||
- Event annotations.
|
||||
|
||||
### 5.6 Alerts and incidents
|
||||
|
||||
- Versioned alert rules.
|
||||
- Pending, firing, acknowledged and resolved lifecycle.
|
||||
- Unknown, silenced, suppressed and maintenance states.
|
||||
- Hysteresis, cooldown, grouping and dependency suppression.
|
||||
- Notification channels with delivery audit.
|
||||
- Incident creation/grouping from related alerts.
|
||||
- Timeline, notes, ownership and status.
|
||||
- Suggested relationships labeled as uncertain unless proven.
|
||||
|
||||
### 5.7 Operations
|
||||
|
||||
- Datasource health page.
|
||||
- Self-monitoring page.
|
||||
- Backup/restore.
|
||||
- Config export.
|
||||
- Safe deployment and rollback.
|
||||
- Maintenance windows.
|
||||
- Diagnostic bundle with secret redaction.
|
||||
- Clear runbook.
|
||||
|
||||
## 6. Core user journeys
|
||||
|
||||
### Journey A — Morning health check
|
||||
|
||||
1. User opens Pulse.
|
||||
2. Global state loads within the performance budget.
|
||||
3. The page shows Operational or a specific explained deviation.
|
||||
4. User opens an affected component from the status explanation.
|
||||
5. Related graph/event context is visible without constructing a query.
|
||||
|
||||
### Journey B — Customize overview
|
||||
|
||||
1. User enters Edit mode.
|
||||
2. Adds a network chart.
|
||||
3. Resizes and positions it.
|
||||
4. Configures the interface and time range.
|
||||
5. Previews mobile layout.
|
||||
6. Saves.
|
||||
7. Reload preserves all layouts and version history.
|
||||
|
||||
### Journey C — Diagnose service failure
|
||||
|
||||
1. Application container remains running.
|
||||
2. HTTP probe returns failure.
|
||||
3. Service becomes Degraded.
|
||||
4. Alert transitions pending -> firing after its duration.
|
||||
5. Related container/network events appear.
|
||||
6. User acknowledges the incident.
|
||||
7. Recovery resolves the alert and records duration.
|
||||
|
||||
### Journey D — Storage risk
|
||||
|
||||
1. SMART/temperature or pool capacity crosses policy.
|
||||
2. Pulse explains which disk/pool and why.
|
||||
3. User sees current values and history.
|
||||
4. No destructive storage action is offered.
|
||||
5. Optional link opens an external operational workflow later.
|
||||
|
||||
### Journey E — Mobile incident view
|
||||
|
||||
1. User opens Pulse on mobile.
|
||||
2. Active incident is immediately accessible.
|
||||
3. Summary, affected entities, latest values and acknowledgement fit the mobile flow.
|
||||
4. Complex dashboard editing is not required.
|
||||
|
||||
## 7. Status model
|
||||
|
||||
Top-level and entity status:
|
||||
|
||||
- `operational`
|
||||
- `attention`
|
||||
- `degraded`
|
||||
- `critical`
|
||||
- `unknown`
|
||||
- `maintenance`
|
||||
- `disabled`
|
||||
|
||||
Rules:
|
||||
|
||||
- `unknown` outranks a false `operational` claim when required data is stale.
|
||||
- Top-level status includes a list of contributing reasons.
|
||||
- Optional/non-critical component failures do not automatically become critical.
|
||||
- Status aggregation is deterministic and tested.
|
||||
- Color is never the only representation.
|
||||
|
||||
## 8. Acceptance-level non-functional requirements
|
||||
|
||||
- Clean build/deploy from a new checkout.
|
||||
- Responsive at target scale.
|
||||
- At least 17 hours of real wallboard soak without unbounded growth; the
|
||||
product owner explicitly replaced the original 24-hour duration for M10-14
|
||||
on 2026-08-11 while retaining every other performance budget.
|
||||
- Graceful source disconnect/reconnect.
|
||||
- No unrestricted Docker socket in API/web.
|
||||
- Least privilege and non-root runtime.
|
||||
- Automated accessibility checks plus keyboard flows.
|
||||
- Backup/restore and upgrade/rollback.
|
||||
- Secrets absent from repository, logs and evidence.
|
||||
- Traceable requirement-to-test-to-evidence mapping.
|
||||
|
||||
## 9. Success metrics
|
||||
|
||||
- Time to first meaningful overview: under the defined performance budget.
|
||||
- Time to identify a simulated primary fault: less than two minutes in usability validation.
|
||||
- Zero false-green results in stale-source tests.
|
||||
- All baseline failure scenarios detected with expected status and alert behavior.
|
||||
- Dashboard configuration persists and restores across browser/server restart.
|
||||
- Production deployment survives planned restart with healthy state and preserved configuration.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Requirements index
|
||||
|
||||
The machine-readable/portable traceability matrix is `planning/requirements-matrix.csv`.
|
||||
|
||||
- **PRD-001** — Global health is visible and explained — tasks: `M2-09;M5-07;M6-09;M7-09;M8-10;M9-13` — verification: E2E + final acceptance
|
||||
- **PRD-002** — Stale required telemetry is Unknown, never Healthy — tasks: `M2-07;M4-11;M5-09;M6-09;M8-03;M9-13` — verification: Unit + scenario + E2E
|
||||
- **PRD-003** — Dashboard CRUD and version history — tasks: `M3-01;M3-03;M3-11` — verification: API + integration + E2E
|
||||
- **PRD-004** — Drag, resize, lock, duplicate, hide and remove widgets — tasks: `M3-05;M3-11` — verification: Playwright + accessibility
|
||||
- **PRD-005** — Per-viewport desktop/tablet/mobile/wallboard layouts — tasks: `M3-07;M9-03;M9-04` — verification: E2E + visual
|
||||
- **PRD-006** — Undo/redo, atomic save and conflict recovery — tasks: `M3-08;M3-11` — verification: Unit + integration + E2E
|
||||
- **PRD-007** — Dashboard variables and cross-filtering — tasks: `M3-09;M3-11` — verification: E2E
|
||||
- **PRD-008** — Validated dashboard import/export/templates — tasks: `M3-10;M3-11` — verification: Schema/security/E2E
|
||||
- **PRD-009** — Semantic bounded metrics query layer — tasks: `M4-01;M4-02;M4-03;M4-04` — verification: Unit + contract + load
|
||||
- **PRD-010** — Live charts use bounded efficient subscriptions — tasks: `M4-06;M4-07;M4-08;M4-09;M4-12` — verification: Protocol + load + soak
|
||||
- **PRD-011** — Host compute/memory/network monitoring — tasks: `M5-01;M5-07;M5-09` — verification: Contract + E2E
|
||||
- **PRD-012** — Optional sensors/GPU degrade gracefully — tasks: `M5-02;M5-09` — verification: Capability tests
|
||||
- **PRD-013** — Read-only bounded process explorer — tasks: `M5-03;M5-09` — verification: Security + UI
|
||||
- **PRD-014** — Container inventory/resource/health monitoring — tasks: `M5-04;M5-05;M5-07;M5-09` — verification: Scale + E2E
|
||||
- **PRD-015** — Application grouping independent of container running — tasks: `M5-06;M5-08;M7-05` — verification: Scenario + E2E
|
||||
- **PRD-016** — Array/parity monitoring without controls — tasks: `M6-01;M6-09` — verification: Scenario + architecture test
|
||||
- **PRD-017** — Disk capacity, temperature, performance and SMART — tasks: `M6-02;M6-03;M6-04;M6-09` — verification: Fixtures + E2E
|
||||
- **PRD-018** — Pool/filesystem/scrub monitoring — tasks: `M6-05;M6-09` — verification: Fixtures + contract
|
||||
- **PRD-019** — Shares and capacity forecasting — tasks: `M6-06;M6-08;M6-10` — verification: Unit + performance + UI
|
||||
- **PRD-020** — Service probes and availability/latency — tasks: `M7-01;M7-03;M7-04;M7-05;M7-06` — verification: Security + load + E2E
|
||||
- **PRD-021** — Probe SSRF and target safety — tasks: `M7-02;M7-11;M9-07` — verification: Negative security tests
|
||||
- **PRD-022** — Dependencies and topology with confidence — tasks: `M7-07;M7-08;M7-11` — verification: Unit + UI
|
||||
- **PRD-023** — Network/DNS/gateway/TLS health are distinct — tasks: `M7-09;M7-11` — verification: Scenarios + E2E
|
||||
- **PRD-024** — Alert state lifecycle and persistence — tasks: `M8-01;M8-02;M8-03;M8-04;M8-11` — verification: State machine + restart
|
||||
- **PRD-025** — Alert grouping, suppression, silence and maintenance — tasks: `M8-05;M8-06;M8-11` — verification: Storm scenarios + E2E
|
||||
- **PRD-026** — Acknowledgement and audit — tasks: `M8-07;M8-12` — verification: RBAC + concurrency
|
||||
- **PRD-027** — Notification delivery is idempotent and audited — tasks: `M8-08;M8-11` — verification: Integration + restart
|
||||
- **PRD-028** — Incidents group related alerts with rationale/confidence — tasks: `M8-09;M8-10;M8-11` — verification: Scenario + E2E
|
||||
- **PRD-029** — Authentik OIDC and RBAC — tasks: `M1-06;M9-07;M9-12` — verification: Auth matrix + production smoke
|
||||
- **PRD-030** — Pulse v1 has no host/Docker/storage mutation path — tasks: `M0-06;M1-08;M5-09;M6-09;M9-07` — verification: Architecture/security scan
|
||||
- **PRD-031** — Backup, restore, upgrade and rollback — tasks: `M9-09;M9-10;M9-13` — verification: Clean restore + migration
|
||||
- **PRD-032** — Clean-room build and deployment — tasks: `M9-11;M9-12;M9-13` — verification: Clean checkout + production smoke
|
||||
- **PRD-033** — Dutch default UI and accessible core workflows — tasks: `M1-03;M9-02;M9-05` — verification: Localization + axe + keyboard
|
||||
- **PRD-034** — Mobile and wallboard are first-class — tasks: `M3-07;M9-03;M9-04;M9-06` — verification: E2E + 24h soak
|
||||
- **PRD-035** — Pulse self-observability and independent failure detection — tasks: `M9-08;M9-13` — verification: System status + external smoke
|
||||
- **NFR-001** — Target scale 150 containers, 40 disks, 300 probes — tasks: `M2-10;M5-09;M6-10;M7-11;M9-06` — verification: Load/scale
|
||||
- **NFR-002** — No unbounded browser/server resource growth — tasks: `M4-12;M9-06` — verification: Load + 24h soak
|
||||
- **NFR-003** — Secrets absent from repo/logs/evidence/images — tasks: `M1-05;M1-07;M9-07;M9-09` — verification: Secret scans
|
||||
- **NFR-004** — Non-root least-privilege container deployment — tasks: `M1-08;M9-07;M9-12` — verification: Compose/image inspection
|
||||
- **NFR-005** — Requirement-to-test-to-evidence traceability — tasks: `M0-08;M1-10;M2-10;M3-11;M4-12;M5-10;M6-10;M7-11;M8-12;M9-13` — verification: Evidence indexes + final matrix
|
||||
@@ -0,0 +1,83 @@
|
||||
# Service and network monitoring requirements
|
||||
|
||||
## Service model
|
||||
|
||||
A service represents a reachable capability independently of container state.
|
||||
|
||||
Types:
|
||||
- HTTP/HTTPS;
|
||||
- TCP;
|
||||
- DNS;
|
||||
- ICMP where permitted;
|
||||
- TLS certificate;
|
||||
- JSON response;
|
||||
- keyword/content assertion.
|
||||
|
||||
A service belongs to an application/entity and may declare dependencies.
|
||||
|
||||
## Probe configuration
|
||||
|
||||
- target URL/host/port;
|
||||
- interval and timeout;
|
||||
- expected status codes;
|
||||
- redirect policy;
|
||||
- TLS verification;
|
||||
- optional JSONPath/keyword assertion;
|
||||
- allowed source/agent;
|
||||
- maintenance schedule;
|
||||
- labels/category;
|
||||
- secret reference for authenticated probes.
|
||||
|
||||
Never store plaintext probe credentials in database exports or evidence.
|
||||
|
||||
## SSRF and target safety
|
||||
|
||||
The probe engine must:
|
||||
- validate scheme and port;
|
||||
- resolve and re-check DNS;
|
||||
- block link-local, metadata and unspecified addresses;
|
||||
- apply an explicit network/target allowlist;
|
||||
- prevent redirect escape to disallowed targets;
|
||||
- limit response size;
|
||||
- avoid executing returned content;
|
||||
- restrict custom headers and methods;
|
||||
- log redacted destinations.
|
||||
|
||||
LAN/private targets may be intentionally allowed through administrator configuration.
|
||||
|
||||
## Results
|
||||
|
||||
Store/derive:
|
||||
- current state;
|
||||
- response time;
|
||||
- status/error class;
|
||||
- availability windows;
|
||||
- last success/failure;
|
||||
- incident history;
|
||||
- TLS expiry/issuer/hostname validity;
|
||||
- DNS resolution timing when relevant.
|
||||
|
||||
## Network
|
||||
|
||||
Monitor:
|
||||
- interface throughput;
|
||||
- packet errors/drops;
|
||||
- gateway reachability;
|
||||
- DNS latency/failure;
|
||||
- internet reachability using configured targets;
|
||||
- internal service dependency failures;
|
||||
- certificate expiry.
|
||||
|
||||
Avoid conflating internet failure with server failure. Dependency suppression should group downstream service alerts.
|
||||
|
||||
## Topology
|
||||
|
||||
Relationships may come from:
|
||||
- container labels;
|
||||
- compose project;
|
||||
- reverse proxy connector;
|
||||
- service configuration;
|
||||
- observed dependency declarations;
|
||||
- user confirmation.
|
||||
|
||||
Source/confidence must be retained. Inferred edges are visibly different from confirmed edges.
|
||||
@@ -0,0 +1,95 @@
|
||||
# Storage monitoring requirements
|
||||
|
||||
Storage is a first-class domain with strict read-only behavior.
|
||||
|
||||
## Array
|
||||
|
||||
Display:
|
||||
- current array state;
|
||||
- parity presence/state;
|
||||
- data disk membership;
|
||||
- missing/disabled/emulated disks;
|
||||
- read/write activity;
|
||||
- parity check state, progress, speed, errors and history;
|
||||
- last/next check when available.
|
||||
|
||||
No start/stop/check/correct action is offered in v1.
|
||||
|
||||
## Disks
|
||||
|
||||
For every disk:
|
||||
- stable identity and role;
|
||||
- model/serial with privacy-aware display;
|
||||
- size and filesystem;
|
||||
- used/free/inodes;
|
||||
- temperature and trend;
|
||||
- read/write throughput, IOPS and latency when available;
|
||||
- spin state when available;
|
||||
- SMART overall and selected attributes;
|
||||
- self-test age/result;
|
||||
- reallocated, pending and offline-uncorrectable sectors;
|
||||
- CRC/interface errors;
|
||||
- SSD wear/percentage used where relevant.
|
||||
|
||||
Rules must avoid declaring a disk healthy when SMART data is unavailable/stale.
|
||||
|
||||
## Pools
|
||||
|
||||
Support cache and ZFS/Btrfs pool concepts discovered in the environment:
|
||||
|
||||
- members;
|
||||
- usable/used/free;
|
||||
- profile/redundancy;
|
||||
- degraded/faulted/offline member state;
|
||||
- scrub status/results;
|
||||
- filesystem errors;
|
||||
- write/read performance;
|
||||
- SSD wear;
|
||||
- mover-related signals when available.
|
||||
|
||||
Do not assume all pools use the same filesystem.
|
||||
|
||||
## Shares
|
||||
|
||||
- name and configured policy;
|
||||
- used size and growth;
|
||||
- participating storage;
|
||||
- cache/pool relationship;
|
||||
- recent growth;
|
||||
- forecast to thresholds;
|
||||
- unavailable/stale state.
|
||||
|
||||
Potentially expensive size calculation must be rate-limited/cached and optional.
|
||||
|
||||
## Forecasting
|
||||
|
||||
Capacity forecast:
|
||||
- uses configurable historical window;
|
||||
- reports method and confidence/uncertainty;
|
||||
- handles irregular bulk imports;
|
||||
- never presents a precise date without qualification;
|
||||
- can be disabled when insufficient history exists.
|
||||
|
||||
## Baseline alert classes
|
||||
|
||||
- array degraded/missing disk;
|
||||
- parity errors;
|
||||
- SMART critical attribute;
|
||||
- temperature sustained above policy;
|
||||
- pool degraded/faulted;
|
||||
- capacity thresholds;
|
||||
- read-only filesystem;
|
||||
- scrub/check errors;
|
||||
- stale storage data.
|
||||
|
||||
Availability, device health, capacity pressure and thermal pressure are separate
|
||||
signals. An online device can therefore be capacity-critical or thermally
|
||||
critical without being presented as offline, and a stale observation makes all
|
||||
current signal severities Unknown. Pulse uses the stable source device ID as
|
||||
the canonical physical identity. Repeated identical observations are
|
||||
idempotent; conflicting roles or facts for one physical ID fail closed instead
|
||||
of producing duplicate or contradictory topology nodes.
|
||||
|
||||
## Safety tests
|
||||
|
||||
Storage fault behavior is tested through fixtures/simulator or read-only captured data. Never trigger real SMART damage, pool degradation, disk removal or array operations.
|
||||
@@ -0,0 +1,308 @@
|
||||
# UX specification
|
||||
|
||||
## 1. Experience principles
|
||||
|
||||
- **Explain before exposing raw detail.**
|
||||
- **Dense, not cramped.**
|
||||
- **Live, not distracting.**
|
||||
- **Configurable, not chaotic.**
|
||||
- **Safe, not action-heavy.**
|
||||
- **Unknown is visible.**
|
||||
- **Desktop, mobile and wallboard are designed separately.**
|
||||
|
||||
## 2. Information architecture
|
||||
|
||||
```text
|
||||
Overview
|
||||
Dashboards
|
||||
|
||||
Infrastructure
|
||||
Host
|
||||
Hardware
|
||||
Storage
|
||||
Network
|
||||
UPS
|
||||
|
||||
Containers
|
||||
Applications
|
||||
Services
|
||||
|
||||
Alerts
|
||||
Incidents
|
||||
Events
|
||||
|
||||
Explore
|
||||
Metrics
|
||||
Compare
|
||||
History
|
||||
|
||||
Wallboards
|
||||
|
||||
Settings
|
||||
Data sources
|
||||
Discovery
|
||||
Alerting
|
||||
Notifications
|
||||
Retention
|
||||
Users & access
|
||||
Backups
|
||||
System
|
||||
```
|
||||
|
||||
Navigation may collapse responsively. Route names and labels are localized.
|
||||
|
||||
## 3. Global shell
|
||||
|
||||
### Header
|
||||
|
||||
- current server/environment;
|
||||
- global time range;
|
||||
- live/pause control;
|
||||
- refresh/freshness state;
|
||||
- active filters;
|
||||
- alert count;
|
||||
- user menu.
|
||||
|
||||
### Source freshness
|
||||
|
||||
A compact indicator exposes:
|
||||
|
||||
- healthy;
|
||||
- delayed;
|
||||
- stale;
|
||||
- unavailable.
|
||||
|
||||
Opening it lists each datasource and last successful sample.
|
||||
|
||||
### Global time control
|
||||
|
||||
Presets:
|
||||
|
||||
- Live;
|
||||
- 15 minutes;
|
||||
- 1 hour;
|
||||
- 6 hours;
|
||||
- 24 hours;
|
||||
- 7 days;
|
||||
- 30 days;
|
||||
- custom.
|
||||
|
||||
A widget may override the dashboard range, but the override is clearly indicated.
|
||||
|
||||
## 4. Overview
|
||||
|
||||
### Healthy state
|
||||
|
||||
Prioritize trends, capacity and service summary.
|
||||
|
||||
### Degraded state
|
||||
|
||||
A problem summary appears before normal cards:
|
||||
|
||||
```text
|
||||
Degraded
|
||||
Cachepool is 91% full.
|
||||
Plex restarted 4 times in 10 minutes.
|
||||
Prometheus storage metrics are 3 minutes stale.
|
||||
```
|
||||
|
||||
Each reason links to its entity and relevant timeframe.
|
||||
|
||||
### Layout
|
||||
|
||||
The system template must remain useful before any editing:
|
||||
|
||||
- status summary;
|
||||
- active incidents;
|
||||
- CPU/load;
|
||||
- memory;
|
||||
- array/pools;
|
||||
- network;
|
||||
- container/application state;
|
||||
- service matrix;
|
||||
- recent events.
|
||||
|
||||
## 5. Dashboard composer
|
||||
|
||||
### Modes
|
||||
|
||||
**View mode**
|
||||
- no drag handles;
|
||||
- widgets fixed;
|
||||
- normal chart interactions;
|
||||
- fast rendering.
|
||||
|
||||
**Edit mode**
|
||||
- visible grid and selection outlines;
|
||||
- drag/resize handles;
|
||||
- widget library;
|
||||
- undo/redo;
|
||||
- save/exit;
|
||||
- breakpoint preview.
|
||||
|
||||
Unsaved changes are clearly indicated. Navigation away prompts only when changes would be lost.
|
||||
|
||||
### Grid
|
||||
|
||||
Recommended logical columns:
|
||||
|
||||
- large desktop/wallboard: 24;
|
||||
- desktop: 18;
|
||||
- tablet: 8;
|
||||
- mobile: 1.
|
||||
|
||||
Widgets define minimum and preferred sizes. Collision handling and snapping are predictable.
|
||||
|
||||
### Widget configuration drawer
|
||||
|
||||
Tabs:
|
||||
|
||||
1. Data
|
||||
2. Visualization
|
||||
3. Thresholds
|
||||
4. Behavior
|
||||
5. Links
|
||||
6. Advanced
|
||||
|
||||
Changes preview live but persist only through the editor save transaction.
|
||||
|
||||
### Editing interactions
|
||||
|
||||
- keyboard move/resize alternative;
|
||||
- duplicate;
|
||||
- lock/unlock;
|
||||
- hide per viewport;
|
||||
- copy to another dashboard;
|
||||
- reset to template;
|
||||
- restore version;
|
||||
- inspect validation errors before save.
|
||||
|
||||
## 6. Entity pages
|
||||
|
||||
Common structure:
|
||||
|
||||
1. identity, status and reason;
|
||||
2. key values and freshness;
|
||||
3. primary charts;
|
||||
4. relationships/dependencies;
|
||||
5. events;
|
||||
6. alerts/incidents;
|
||||
7. technical metadata.
|
||||
|
||||
Deep technical payloads are collapsed by default.
|
||||
|
||||
## 7. Alerts and incidents
|
||||
|
||||
### Alert list
|
||||
|
||||
Supports filters for:
|
||||
- state;
|
||||
- severity;
|
||||
- entity;
|
||||
- source;
|
||||
- acknowledged;
|
||||
- maintenance;
|
||||
- time.
|
||||
|
||||
Each row shows:
|
||||
- concise rule name;
|
||||
- affected entity;
|
||||
- current value/reason;
|
||||
- duration;
|
||||
- state;
|
||||
- incident relationship.
|
||||
|
||||
### Incident page
|
||||
|
||||
- plain-language summary;
|
||||
- severity and lifecycle;
|
||||
- affected entities;
|
||||
- contributing alerts;
|
||||
- timeline;
|
||||
- selected charts;
|
||||
- acknowledgement/notes;
|
||||
- uncertain relationship labels;
|
||||
- external workflow link when configured.
|
||||
|
||||
## 8. Mobile
|
||||
|
||||
Primary mobile navigation:
|
||||
|
||||
- Overview;
|
||||
- Incidents;
|
||||
- Containers;
|
||||
- Storage;
|
||||
- Services;
|
||||
- More.
|
||||
|
||||
Mobile priorities:
|
||||
- global status;
|
||||
- active incident;
|
||||
- acknowledgement;
|
||||
- essential entity facts;
|
||||
- readable charts with reduced series;
|
||||
- touch-sized controls.
|
||||
|
||||
Dashboard editing on mobile:
|
||||
- reorder;
|
||||
- show/hide;
|
||||
- simple config;
|
||||
- no precision freeform grid placement.
|
||||
|
||||
## 9. Wallboard
|
||||
|
||||
- fullscreen;
|
||||
- no standard navigation;
|
||||
- large readable status;
|
||||
- page rotation;
|
||||
- reconnect indicator;
|
||||
- last update/freshness;
|
||||
- optional kiosk-safe token/session;
|
||||
- no edit controls;
|
||||
- subtle movement strategy to reduce static burn-in risk;
|
||||
- survives days without reload.
|
||||
|
||||
## 10. Loading, empty and failure states
|
||||
|
||||
Every data surface defines:
|
||||
|
||||
- initial loading;
|
||||
- incremental live update;
|
||||
- empty because no entities exist;
|
||||
- empty because filters exclude all;
|
||||
- stale;
|
||||
- unavailable;
|
||||
- unauthorized;
|
||||
- query limit exceeded;
|
||||
- partial data.
|
||||
|
||||
Skeletons must not imply exact values. Previous values may remain visible only with an explicit stale age.
|
||||
|
||||
## 11. Copy
|
||||
|
||||
Default user-facing copy is natural Dutch:
|
||||
|
||||
Good:
|
||||
- `Geen recente gegevens`
|
||||
- `Container herstart herhaaldelijk`
|
||||
- `Cachepool bijna vol`
|
||||
- `Laatst succesvol bijgewerkt om 13:42`
|
||||
|
||||
Avoid:
|
||||
- raw metric names;
|
||||
- unexplained `NaN`;
|
||||
- only `Error 500`;
|
||||
- invented root-cause certainty;
|
||||
- alarmist language for attention-level conditions.
|
||||
|
||||
## 12. Accessibility
|
||||
|
||||
- WCAG 2.2 AA target for core workflows.
|
||||
- Full keyboard navigation.
|
||||
- Visible focus.
|
||||
- Text/icon plus color for status.
|
||||
- Reduced-motion support.
|
||||
- Charts have accessible summaries/table alternatives.
|
||||
- Drag interactions have non-pointer alternatives.
|
||||
- Contrast is validated in all themes/states.
|
||||
- Live regions announce important state changes without constant noise.
|
||||
@@ -0,0 +1,179 @@
|
||||
# Widget catalog
|
||||
|
||||
Every widget implements a common contract:
|
||||
|
||||
- identity and type;
|
||||
- data query;
|
||||
- transformations;
|
||||
- visualization settings;
|
||||
- thresholds/status mapping;
|
||||
- behavior and links;
|
||||
- layout per breakpoint;
|
||||
- loading/empty/error/stale states;
|
||||
- permission requirements;
|
||||
- export capability.
|
||||
|
||||
## 1. Stat card
|
||||
|
||||
Use for one current value.
|
||||
|
||||
Features:
|
||||
- label and value;
|
||||
- unit/format;
|
||||
- state;
|
||||
- sparkline;
|
||||
- trend against prior period;
|
||||
- min/max/average tooltip;
|
||||
- freshness.
|
||||
|
||||
Examples: CPU, RAM, uptime, active containers, free capacity.
|
||||
|
||||
## 2. Time series
|
||||
|
||||
Features:
|
||||
- one or more bounded series;
|
||||
- line/area;
|
||||
- hover crosshair;
|
||||
- zoom/pan;
|
||||
- pause/live;
|
||||
- event annotations;
|
||||
- min/max/average;
|
||||
- legend and series toggles;
|
||||
- downsampling;
|
||||
- compare period;
|
||||
- export.
|
||||
|
||||
## 3. Gauge
|
||||
|
||||
Use only where a meaningful min/max and thresholds exist.
|
||||
|
||||
Examples:
|
||||
- capacity;
|
||||
- temperature;
|
||||
- UPS charge;
|
||||
- memory limit.
|
||||
|
||||
Avoid gauges for values with no stable range.
|
||||
|
||||
## 4. Ranked list
|
||||
|
||||
- top/bottom N;
|
||||
- current value;
|
||||
- mini trend;
|
||||
- click-to-filter;
|
||||
- deterministic ID/label tie handling;
|
||||
- tie handling;
|
||||
- "other" aggregation where appropriate.
|
||||
|
||||
Examples: top container CPU, disk latency, service response time.
|
||||
|
||||
## 5. Status grid
|
||||
|
||||
The status grid renders bounded text/icon states and supports keyboard click-to-filter for the selected entity.
|
||||
|
||||
Compact tiles for many entities.
|
||||
|
||||
Required:
|
||||
- status icon/text;
|
||||
- entity name;
|
||||
- optional key value;
|
||||
- age/freshness;
|
||||
- grouping and filtering;
|
||||
- accessible list alternative.
|
||||
|
||||
## 6. Table
|
||||
|
||||
- server-side pagination/filtering where large;
|
||||
- sort and column selector;
|
||||
- pinned identifying columns;
|
||||
- export;
|
||||
- row drill-down;
|
||||
- virtualized rendering;
|
||||
- explicit stale fields.
|
||||
|
||||
## 7. Heatmap
|
||||
|
||||
Examples:
|
||||
- CPU by hour/day;
|
||||
- disk temperature history;
|
||||
- service latency distribution.
|
||||
|
||||
Must provide a textual/table summary.
|
||||
|
||||
## 8. Event timeline
|
||||
|
||||
- chronological events;
|
||||
- severity/type filters;
|
||||
- entity grouping;
|
||||
- chart annotation synchronization;
|
||||
- expandable attributes;
|
||||
- no secret/raw sensitive payloads.
|
||||
|
||||
## 9. Storage map
|
||||
|
||||
Visual model of:
|
||||
- parity;
|
||||
- array disks;
|
||||
- pools;
|
||||
- members;
|
||||
- filesystems;
|
||||
- capacity;
|
||||
- temperature;
|
||||
- SMART state.
|
||||
|
||||
It must remain usable without relying on physical slot colors alone.
|
||||
|
||||
## 10. Topology
|
||||
|
||||
Nodes:
|
||||
- applications;
|
||||
- containers;
|
||||
- services;
|
||||
- endpoints;
|
||||
- dependencies;
|
||||
- reverse proxy;
|
||||
- database/storage.
|
||||
|
||||
Edges may show status, latency or traffic when reliable. Unknown relationships are visually distinct from discovered/confirmed ones.
|
||||
|
||||
## 11. Service matrix
|
||||
|
||||
For endpoints/services:
|
||||
- status;
|
||||
- latency;
|
||||
- uptime percentage;
|
||||
- last incident;
|
||||
- TLS expiry;
|
||||
- maintenance.
|
||||
|
||||
## 12. Alert/incident summary
|
||||
|
||||
- active counts by severity/state;
|
||||
- newest/highest priority;
|
||||
- acknowledgement state;
|
||||
- drill-down;
|
||||
- no hidden critical item due to pagination.
|
||||
|
||||
## 13. Text/annotation
|
||||
|
||||
Markdown subset with sanitization.
|
||||
|
||||
Use for:
|
||||
- dashboard instructions;
|
||||
- maintenance note;
|
||||
- runbook link;
|
||||
- scope explanation.
|
||||
|
||||
No arbitrary script/HTML.
|
||||
|
||||
## 14. Query inspector
|
||||
|
||||
Advanced and permission-controlled:
|
||||
- semantic query;
|
||||
- generated PromQL;
|
||||
- execution timing;
|
||||
- returned series/points;
|
||||
- limits applied;
|
||||
- copy for troubleshooting.
|
||||
|
||||
It is not enabled as an unrestricted public query console.
|
||||
Reference in New Issue
Block a user