ServiceMap: update tests and backend/app (+7 areas)
Public changes compared with the previous published export: 89 files changed. Affected areas: - backend/app: 21 files - dependencies: 5 files - deployment and CI: 6 files - documentation: 2 files - frontend: 3 files - frontend/src: 20 files - project files: 1 file - tests: 30 files - unraid: 1 file Changed files: - Updated: ".env.example" - Updated: "Dockerfile.unraid" - Updated: "README.md" - Updated: "backend/Dockerfile" - Added: "backend/app/alerts.py" - Updated: "backend/app/auth.py" - Added: "backend/app/backups.py" - Updated: "backend/app/config_io.py" - Updated: "backend/app/demo.py" - Updated: "backend/app/discovery.py" - Updated: "backend/app/discovery_store.py" - Updated: "backend/app/docker_discovery.py" - Updated: "backend/app/events.py" - Updated: "backend/app/evidence.py" - Updated: "backend/app/health.py" - Updated: "backend/app/incidents.py" - Added: "backend/app/loops.py" - Updated: "backend/app/main.py" - Updated: "backend/app/models.py" - Updated: "backend/app/observation.py" - Updated: "backend/app/probes.py" - Added: "backend/app/response_cache.py" - Updated: "backend/app/scheduler.py" - Updated: "backend/app/semantic.py" - Updated: "backend/app/store.py" - Updated: "backend/pyproject.toml" - Updated: "backend/tests/test_api.py" - Updated: "backend/tests/test_config_io.py" - Updated: "backend/tests/test_demo.py" - Updated: "backend/tests/test_discovery_v120.py" - Updated: "backend/tests/test_health.py" - Updated: "backend/tests/test_probes.py" - Updated: "backend/tests/test_scheduler.py" - Updated: "backend/tests/test_v130_probe_reachability.py" - Updated: "backend/tests/test_v160_evidence_changes.py" - Added: "backend/tests/test_v163_alerts.py" - Added: "backend/tests/test_v163_backups.py" - Added: "backend/tests/test_v163_data_integrity.py" - Added: "backend/tests/test_v163_efficiency.py" - Added: "backend/tests/test_v163_health_performance.py" - Added: "backend/tests/test_v163_loops.py" - Added: "backend/tests/test_v163_management.py" - Added: "backend/tests/test_v163_realtime.py" - Added: "backend/tests/test_v163_security.py" - Added: "backend/tests/test_v163_semantic_etag.py" - Added: "compose.demo.yaml" - Updated: "compose.yaml" - Added: "docker-compose.broker.yml" - Updated: "docker-compose.yml" - Updated: "docs/ARCHITECTURE.md" - Added: "frontend/e2e/a11y.spec.ts" - Updated: "frontend/e2e/demo.spec.ts" - Updated: "frontend/index.html" - Updated: "frontend/nginx.conf" - Updated: "frontend/package-lock.json" - Updated: "frontend/package.json" - Updated: "frontend/playwright.config.ts" - Updated: "frontend/src/App.tsx" - Updated: "frontend/src/LivingEdge.tsx" - Added: "frontend/src/activityView.test.ts" - Added: "frontend/src/activityView.ts" - Updated: "frontend/src/api.ts" - Added: "frontend/src/apiAuth.test.ts" - Added: "frontend/src/components/ActionsMenu.tsx" - Added: "frontend/src/components/ConfirmDialog.test.tsx" - Added: "frontend/src/components/ConfirmDialog.tsx" - Added: "frontend/src/components/StatusIcon.tsx" - Updated: "frontend/src/demo/DemoWalkthrough.test.ts" - Updated: "frontend/src/demo/DemoWalkthrough.tsx" - Added: "frontend/src/elk-worker-url.d.ts" - Added: "frontend/src/labels.ts" - Updated: "frontend/src/main.tsx" - Added: "frontend/src/realtime/refreshPlan.test.ts" - Added: "frontend/src/realtime/refreshPlan.ts" - Updated: "frontend/src/realtime/useRealtimeEvents.ts" - Added: "frontend/src/routing/urlState.test.ts" - Added: "frontend/src/routing/urlState.ts" - Updated: "frontend/src/styles.css" - Added: "frontend/src/topology/GraphListView.test.tsx" - Added: "frontend/src/topology/GraphListView.tsx" - Updated: "frontend/src/topology/ServiceNode.tsx" - Updated: "frontend/src/topology/TopologyCommandBar.tsx" - Added: "frontend/src/topology/semanticGraph.test.ts" - Updated: "frontend/src/topology/semanticGraph.ts" - Added: "frontend/src/topologyLayout.test.ts" - Updated: "frontend/src/topologyLayout.ts" - Updated: "package-lock.json" - Updated: "package.json" - Updated: "unraid/my-servicemap.xml" Export provenance/manifests refreshed; excluded from the change summary. Publish curated ServiceMap source from af46f684e468dffd3a5190e28d3b572d4c26bfd7
This commit is contained in:
1 parent
d7319250e8
commit
adc2c70c21
91 files changed
+9205
-1691
No files matched your search
+34
-6
@@ -35,9 +35,11 @@ v1.6.1 adds no schema migration. Its health-only observation loop is distinct fr
|
||||
|
||||
v1.6.2 keeps the same schema and evidence scopes. An AppOps `warning` observation is degraded application evidence; it cannot resolve a component. Evidence freshness is recomputed when persisted claims are read, so an expired claim cannot appear fresh in the raw inspector/API payload. The current MCP Hub Docker read broker cannot replace the production socket provider yet: its registration covers two unrelated projects, its ServiceMap-scoped inventory is empty, and its short-lived HMAC service-token contract rejects a raw bearer secret. Cutover requires a dedicated ServiceMap read identity, full bounded inventory parity and an explicit network path; broker mode remains fail-closed with no socket fallback.
|
||||
|
||||
The HTTP discovery routes accept only the Docker host and AppOps URL selected in deployment configuration. A request body cannot turn the container into a general-purpose client for another Docker daemon or internal HTTP endpoint. The private-LAN deployment currently keeps OIDC disabled because it has no dedicated browser client or tested owner login path; mutation APIs therefore depend on the LAN perimeter. Broker mode must be deployed without the legacy Docker socket bind even though the current socket-mode Compose file retains it.
|
||||
The HTTP discovery routes accept only the Docker host and AppOps URL selected in deployment configuration. A request body cannot turn the container into a general-purpose client for another Docker daemon or internal HTTP endpoint. The private-LAN deployment keeps OIDC disabled because it has no dedicated browser client or tested owner login path. Since v1.6.3 it runs in `token` mode, so mutation APIs need the admin token instead of depending on the LAN perimeter. Broker mode must be deployed without the legacy Docker socket bind even though the current socket-mode Compose file retains it.
|
||||
|
||||
The v1.4 semantic projection is deliberately not a parallel datastore. `semantic.py` derives Domain → Application → Component from canonical services, relations, runtime/monitoring state and incident analysis. Logical relations are grouped at query time so one graph edge retains all independent evidence records and their first/last-seen provenance. v1.6.1 resolves application health separately: fresh aggregate AppOps evidence can answer an application but never a component; direct component failures override weak aggregate health and conflicting claims remain visible. Semantic application identity uses locked operator grouping, Compose provenance and stable source references before a display name, preventing generic provider labels from collapsing unrelated services. Bulk evidence lookup serves the complete semantic projection in one bounded SQL query rather than an application-by-application scan.
|
||||
The v1.4 semantic projection is deliberately not a parallel datastore. `semantic.py` derives Domain → Application → Component from canonical services, relations, runtime/monitoring state and incident analysis. Logical relations are grouped at query time so one graph edge retains all independent evidence records and their first/last-seen provenance. v1.6.1 resolves application health separately: fresh aggregate AppOps evidence can answer an application but never a component; direct component failures override weak aggregate health and conflicting claims remain visible. Semantic application identity uses locked operator grouping, Compose provenance and stable source references before a display name, preventing generic provider labels from collapsing unrelated services. Bulk evidence lookup serves the complete semantic projection in one bounded SQL query rather than an application-by-application scan. v1.6.3 matches direct and linked claims through two separately indexed lookups (a `UNION`) and applies the per-service limit with a window function, because an `OR` across the link join forced a full `health_evidence` scan. Direct, dependency and effective health for incidents, the overview and service detail all come from one bulk read and one in-memory traversal (`health.topology_health`), never a per-node recursive SQLite walk. Snapshot discovery context (sources and relation evidence) is also read in bulk.
|
||||
|
||||
`GET /api/v1/topology/semantic` sends a strong `ETag` computed over the content without `generated_at`, together with `Cache-Control: no-cache` and `X-ServiceMap-Generated-At`. A client that sends `If-None-Match` while nothing has changed receives `304 Not Modified` without a body. Browsers revalidate on their own, and integrations such as ITWorx Forge send the header explicitly. The serialized response is also cached for at most 10 seconds. The cache is dropped as soon as any event is published or any evidence crosses a fresh→stale→expired boundary. The age cap bounds staleness for the few write paths that publish no event, such as discovery relation evidence; the ETag is always derived from content, so it is correct regardless.
|
||||
|
||||
The service record stores descriptive metadata, group/environment/tags, criticality, source reference, application, runtime state/health, monitoring state, probe source/reachability/confidence, enablement, maintenance state, probe policy and state-machine counters. Probe rows remain immutable and retain `trigger` and bounded JSON `metadata` fields. `HealthEvidence` retains provider, subject, scope, confidence, observation/receipt/expiry timestamps, metadata and supersession. Application-scoped claims may be linked to component inspectors but cannot resolve a component's direct health.
|
||||
|
||||
@@ -86,11 +88,20 @@ Per-service interval, timeout and retry settings override the global defaults. S
|
||||
|
||||
Incident timeline entries are events tied to an incident. Raw probe rows are retained independently under the configured probe retention; non-incident events use `SERVICEMAP_EVENT_RETENTION_DAYS` (default 90). Incident lifecycle events are retained with the incident record.
|
||||
|
||||
A maintenance task runs 30 seconds after startup and then every `SERVICEMAP_MAINTENANCE_INTERVAL_SECONDS` (default 3600). It applies probe and event retention independently of scheduled probing, and deletes superseded health evidence older than `SERVICEMAP_EVIDENCE_RETENTION_DAYS` (default 7; `0` disables). Only inactive rows are deleted. A row becomes inactive only when a newer claim from the same provider and subject replaces it, so every subject keeps its current claim and health resolution is unchanged.
|
||||
|
||||
## SSE and frontend refresh
|
||||
|
||||
The durable SQLite event journal is the source of truth. A small thread-safe bounded queue is allocated per SSE client and receives notifications for probe completion, state changes, topology changes and incident lifecycle changes. Slow clients lose only transient wakeups, not durable history. The stream emits keepalives and unregisters queues on disconnect/shutdown.
|
||||
The durable SQLite event journal is the source of truth. Each SSE client gets a bounded `asyncio.Queue` (size 100) bound to the event loop. A parked stream therefore holds no thread, and publishers on worker threads deliver to it through `call_soon_threadsafe`. Clients receive notifications for probe completion, state changes, health-evidence and provider changes, topology changes and incident lifecycle changes. A slow client loses only transient wakeups, never durable history. The stream starts with a `retry:` hint, sends a keepalive every 15 seconds, and unregisters its queue on disconnect or shutdown. A health-evidence update counts as material only when status, confidence, summary or stable metadata change; per-probe bookkeeping (`probe_id`, `latency_ms`, `trigger`) is stored but publishes nothing.
|
||||
|
||||
React subscribes with SSE, debounces event refreshes and reconnects with backoff. It enables bounded fallback refresh only while the stream is unavailable; a healthy quiet stream does not trigger periodic full reloads. Authenticated clients use the same SSE protocol through `fetch` with a bearer header. The frontend never invents monitoring state optimistically; every node, counter and inspector value comes from the API.
|
||||
React maps each event type to the smallest set of data slices it can affect:
|
||||
|
||||
- core reads: services, incidents, overview, events, demo state and the semantic topology;
|
||||
- discovery reads: only on `topology.changed`;
|
||||
- provider status: only on `provider.status_changed`;
|
||||
- the selected service's history: only for events about that service.
|
||||
|
||||
A single-flight scheduler merges bursts (250 ms debounce, 2 s maximum wait), so refreshes never overlap and never apply out of order. Reconnects use exponential backoff with jitter. The backoff resets only after a connection has been stable for 30 seconds, and every reopen after a gap triggers one full resync. A hidden tab keeps its stream for 60 seconds before it disconnects. The fallback poll runs only while the stream is down, and the UI shows a live, connecting or paused state. Authenticated clients use the same SSE protocol through `fetch` with a bearer header. The frontend never invents monitoring state optimistically; every node, counter and inspector value comes from the API.
|
||||
|
||||
## Production discovery and reconciliation
|
||||
|
||||
@@ -114,6 +125,23 @@ The Northstar demo is built from deterministic UUIDs and synthetic probe records
|
||||
|
||||
The Unraid target uses `/data` on the named `servicemap_data` volume, port mapping `1253:8000`, a healthcheck on `/health`, `no-new-privileges`, dropped capabilities, a read-only root filesystem and bounded `/tmp` tmpfs. Managed validation gates exact-SHA autoredeploy. Socket discovery remains a high-trust Docker API boundary even with a read-only bind. The opt-in broker provider requires a dedicated short-lived ServiceMap token, registration-scoped read capability and shared network, and never falls back to the socket. AppOps access is read-only enrichment.
|
||||
|
||||
## Optional authorization
|
||||
## Authorization
|
||||
|
||||
`SERVICEMAP_AUTH_MODE=disabled` preserves the deployed v1.5 behavior. In `oidc` mode FastAPI validates issuer/audience/signed bearer JWTs against OIDC discovery and JWKS; it maps explicit roles or configured Authentik groups to viewer, operator and admin. `/health` remains anonymous for deployment recovery, while APIs fail closed when OIDC configuration is incomplete. No client credentials, issuer secrets or tokens are stored in ServiceMap's SQLite database or frontend configuration.
|
||||
`SERVICEMAP_AUTH_MODE` selects one of three modes:
|
||||
|
||||
- **`token`** is the default since v1.6.3. Reads stay anonymous, so integrations such as ITWorx Forge and monitors keep working. Every mutating request needs `Authorization: Bearer <admin token>`.
|
||||
- On first start ServiceMap writes a random token with mode 0600 to `admin-token` next to the database, or to `SERVICEMAP_ADMIN_TOKEN_FILE` when that is set.
|
||||
- Only the file's location is logged, never the token itself.
|
||||
- The comparison is constant-time.
|
||||
- The UI asks for the token once per browser tab, checks it with `POST /api/v1/auth/verify` and keeps it in `sessionStorage`. A 401 clears the stored token and asks again.
|
||||
- **`disabled`** preserves the v1.5 behavior.
|
||||
- **`oidc`** is described below.
|
||||
|
||||
Two guards apply in every mode:
|
||||
|
||||
- A mutating request is refused when its `Origin` hostname differs from the public host (`X-Forwarded-Host`, otherwise `Host`), when the `Origin` is `null`, or when the browser marks it `Sec-Fetch-Site: cross-site`. Clients that send none of these headers, such as curl and integrations, are unaffected.
|
||||
- The optional `SERVICEMAP_ALLOWED_HOSTS` list refuses every other `Host`, which guards against DNS rebinding. Loopback is always allowed for the healthcheck.
|
||||
|
||||
Every response carries `nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: no-referrer` and a strict CSP. Request bodies are limited to 4 MiB, a config import to 2 MB, and YAML anchors and aliases are refused.
|
||||
|
||||
In `oidc` mode the role mapping also applies: a configuration export and the demo routes need the admin role. In `oidc` mode FastAPI validates issuer/audience/signed bearer JWTs against OIDC discovery and JWKS; it maps explicit roles or configured Authentik groups to viewer, operator and admin. `/health` remains anonymous for deployment recovery, while APIs fail closed when OIDC configuration is incomplete. No client credentials, issuer secrets or tokens are stored in ServiceMap's SQLite database or frontend configuration.
|
||||
Reference in new issue
Block a user