Files
ServiceMap-Public/docs/ARCHITECTURE.md
NuklearRabbit adc2c70c21 ServiceMap: update tests and backend/app (+7 areas)
Public changes compared with the previous published export:
89 files changed.

Affected areas:
- backend/app: 21 files
- dependencies: 5 files
- deployment and CI: 6 files
- documentation: 2 files
- frontend: 3 files
- frontend/src: 20 files
- project files: 1 file
- tests: 30 files
- unraid: 1 file

Changed files:
- Updated: ".env.example"
- Updated: "Dockerfile.unraid"
- Updated: "README.md"
- Updated: "backend/Dockerfile"
- Added: "backend/app/alerts.py"
- Updated: "backend/app/auth.py"
- Added: "backend/app/backups.py"
- Updated: "backend/app/config_io.py"
- Updated: "backend/app/demo.py"
- Updated: "backend/app/discovery.py"
- Updated: "backend/app/discovery_store.py"
- Updated: "backend/app/docker_discovery.py"
- Updated: "backend/app/events.py"
- Updated: "backend/app/evidence.py"
- Updated: "backend/app/health.py"
- Updated: "backend/app/incidents.py"
- Added: "backend/app/loops.py"
- Updated: "backend/app/main.py"
- Updated: "backend/app/models.py"
- Updated: "backend/app/observation.py"
- Updated: "backend/app/probes.py"
- Added: "backend/app/response_cache.py"
- Updated: "backend/app/scheduler.py"
- Updated: "backend/app/semantic.py"
- Updated: "backend/app/store.py"
- Updated: "backend/pyproject.toml"
- Updated: "backend/tests/test_api.py"
- Updated: "backend/tests/test_config_io.py"
- Updated: "backend/tests/test_demo.py"
- Updated: "backend/tests/test_discovery_v120.py"
- Updated: "backend/tests/test_health.py"
- Updated: "backend/tests/test_probes.py"
- Updated: "backend/tests/test_scheduler.py"
- Updated: "backend/tests/test_v130_probe_reachability.py"
- Updated: "backend/tests/test_v160_evidence_changes.py"
- Added: "backend/tests/test_v163_alerts.py"
- Added: "backend/tests/test_v163_backups.py"
- Added: "backend/tests/test_v163_data_integrity.py"
- Added: "backend/tests/test_v163_efficiency.py"
- Added: "backend/tests/test_v163_health_performance.py"
- Added: "backend/tests/test_v163_loops.py"
- Added: "backend/tests/test_v163_management.py"
- Added: "backend/tests/test_v163_realtime.py"
- Added: "backend/tests/test_v163_security.py"
- Added: "backend/tests/test_v163_semantic_etag.py"
- Added: "compose.demo.yaml"
- Updated: "compose.yaml"
- Added: "docker-compose.broker.yml"
- Updated: "docker-compose.yml"
- Updated: "docs/ARCHITECTURE.md"
- Added: "frontend/e2e/a11y.spec.ts"
- Updated: "frontend/e2e/demo.spec.ts"
- Updated: "frontend/index.html"
- Updated: "frontend/nginx.conf"
- Updated: "frontend/package-lock.json"
- Updated: "frontend/package.json"
- Updated: "frontend/playwright.config.ts"
- Updated: "frontend/src/App.tsx"
- Updated: "frontend/src/LivingEdge.tsx"
- Added: "frontend/src/activityView.test.ts"
- Added: "frontend/src/activityView.ts"
- Updated: "frontend/src/api.ts"
- Added: "frontend/src/apiAuth.test.ts"
- Added: "frontend/src/components/ActionsMenu.tsx"
- Added: "frontend/src/components/ConfirmDialog.test.tsx"
- Added: "frontend/src/components/ConfirmDialog.tsx"
- Added: "frontend/src/components/StatusIcon.tsx"
- Updated: "frontend/src/demo/DemoWalkthrough.test.ts"
- Updated: "frontend/src/demo/DemoWalkthrough.tsx"
- Added: "frontend/src/elk-worker-url.d.ts"
- Added: "frontend/src/labels.ts"
- Updated: "frontend/src/main.tsx"
- Added: "frontend/src/realtime/refreshPlan.test.ts"
- Added: "frontend/src/realtime/refreshPlan.ts"
- Updated: "frontend/src/realtime/useRealtimeEvents.ts"
- Added: "frontend/src/routing/urlState.test.ts"
- Added: "frontend/src/routing/urlState.ts"
- Updated: "frontend/src/styles.css"
- Added: "frontend/src/topology/GraphListView.test.tsx"
- Added: "frontend/src/topology/GraphListView.tsx"
- Updated: "frontend/src/topology/ServiceNode.tsx"
- Updated: "frontend/src/topology/TopologyCommandBar.tsx"
- Added: "frontend/src/topology/semanticGraph.test.ts"
- Updated: "frontend/src/topology/semanticGraph.ts"
- Added: "frontend/src/topologyLayout.test.ts"
- Updated: "frontend/src/topologyLayout.ts"
- Updated: "package-lock.json"
- Updated: "package.json"
- Updated: "unraid/my-servicemap.xml"

Export provenance/manifests refreshed; excluded from the change summary.

Publish curated ServiceMap source from af46f684e468dffd3a5190e28d3b572d4c26bfd7
2026-11-20 13:51:46 +01:00

18 KiB

ServiceMap architecture

Runtime shape

ServiceMap remains deliberately small: one FastAPI process, one React single-page application and one SQLite database. Compose serves React through Nginx; the Unraid image bundles the same frontend and serves it from FastAPI.

Browser
  │  REST + SSE
  ▼
React workspaces
  │
  ▼
FastAPI
  ├── service inventory and dependency validation
  ├── probe engine (HTTP, TCP, DNS, TLS)
  ├── state machine and flapping observation
  ├── health-evidence resolution and provider health
  ├── incident correlation and event journal
  ├── SSE fan-out bus
  ├── atomic config import/export
  ├── Docker/AppOps discovery and reconciliation
  ├── evidence-backed canonical topology overrides
  └── optional per-service scheduler
       │
       ▼
     SQLite on /data

Data ownership and migration

Store owns SQLite access and opens short-lived connections with WAL mode, a 5-second busy timeout and foreign keys enabled. Startup creates missing discovery tables and adds versioned metadata in place. v1.6 adds health evidence, evidence-to-service links and provider-status rows; the one-time migration projects only legitimate existing latest probe/runtime facts with their original timestamps. Existing rows receive safe defaults; the migration never deletes or rewrites topology data.

v1.6.1 adds no schema migration. Its health-only observation loop is distinct from discovery: every configured cadence it reads Docker runtime/healthcheck facts and AppOps application inventory, renews only source-scoped evidence/provider status, and skips demo state. It does not create services, suggestions, relationship evidence, snapshots, probe endpoints or operator overrides. The shared observation TTL is at least 150 seconds and three normal cycles, so a delayed provider cycle ages evidence rather than immediately turning the topology unknown. Provider failures are isolated; they update that provider's availability only and leave other current evidence intact.

v1.6.2 keeps the same schema and evidence scopes. An AppOps warning observation is degraded application evidence; it cannot resolve a component. Evidence freshness is recomputed when persisted claims are read, so an expired claim cannot appear fresh in the raw inspector/API payload. The current MCP Hub Docker read broker cannot replace the production socket provider yet: its registration covers two unrelated projects, its ServiceMap-scoped inventory is empty, and its short-lived HMAC service-token contract rejects a raw bearer secret. Cutover requires a dedicated ServiceMap read identity, full bounded inventory parity and an explicit network path; broker mode remains fail-closed with no socket fallback.

The HTTP discovery routes accept only the Docker host and AppOps URL selected in deployment configuration. A request body cannot turn the container into a general-purpose client for another Docker daemon or internal HTTP endpoint. The private-LAN deployment keeps OIDC disabled because it has no dedicated browser client or tested owner login path. Since v1.6.3 it runs in token mode, so mutation APIs need the admin token instead of depending on the LAN perimeter. Broker mode must be deployed without the legacy Docker socket bind even though the current socket-mode Compose file retains it.

The v1.4 semantic projection is deliberately not a parallel datastore. semantic.py derives Domain → Application → Component from canonical services, relations, runtime/monitoring state and incident analysis. Logical relations are grouped at query time so one graph edge retains all independent evidence records and their first/last-seen provenance. v1.6.1 resolves application health separately: fresh aggregate AppOps evidence can answer an application but never a component; direct component failures override weak aggregate health and conflicting claims remain visible. Semantic application identity uses locked operator grouping, Compose provenance and stable source references before a display name, preventing generic provider labels from collapsing unrelated services. Bulk evidence lookup serves the complete semantic projection in one bounded SQL query rather than an application-by-application scan. v1.6.3 matches direct and linked claims through two separately indexed lookups (a UNION) and applies the per-service limit with a window function, because an OR across the link join forced a full health_evidence scan. Direct, dependency and effective health for incidents, the overview and service detail all come from one bulk read and one in-memory traversal (health.topology_health), never a per-node recursive SQLite walk. Snapshot discovery context (sources and relation evidence) is also read in bulk.

GET /api/v1/topology/semantic sends a strong ETag computed over the content without generated_at, together with Cache-Control: no-cache and X-ServiceMap-Generated-At. A client that sends If-None-Match while nothing has changed receives 304 Not Modified without a body. Browsers revalidate on their own, and integrations such as ITWorx Forge send the header explicitly. The serialized response is also cached for at most 10 seconds. The cache is dropped as soon as any event is published or any evidence crosses a fresh→stale→expired boundary. The age cap bounds staleness for the few write paths that publish no event, such as discovery relation evidence; the ETag is always derived from content, so it is correct regardless.

The service record stores descriptive metadata, group/environment/tags, criticality, source reference, application, runtime state/health, monitoring state, probe source/reachability/confidence, enablement, maintenance state, probe policy and state-machine counters. Probe rows remain immutable and retain trigger and bounded JSON metadata fields. HealthEvidence retains provider, subject, scope, confidence, observation/receipt/expiry timestamps, metadata and supersession. Application-scoped claims may be linked to component inspectors but cannot resolve a component's direct health.

Indexes cover service/time probe history, global probe time, event time/service/incident and incident status/root lookups. Availability is calculated with SQL aggregates over a bounded time window rather than loading all rows into Python. Evidence timestamp renewals retain a stable row and do not publish SSE; material provider status, evidence status, confidence, summary or metadata changes do.

Health state and dependency semantics

The direct state machine is backend-owned, then exposed through an evidence policy:

unknown ──success──> healthy
healthy ──failure threshold──> degraded/down
down/degraded ──success #1──> recovering
recovering ──recovery threshold──> healthy
any monitored state ──maintenance──> maintenance

The first failure can be held at the prior/unknown state when a failure threshold is greater than one. Repeated state changes in a bounded 30-minute query set the flapping observation and its count; flapping is not a competing health state.

Effective health is calculated recursively from direct state and dependencies. A down dependency makes a healthy dependent degraded; the stored probe observation is never rewritten. Cycles and missing dependencies are rejected before writes.

Maintenance and disabled monitoring do not produce a new incident. A running Docker component without a verified endpoint or Docker healthcheck is unmonitored, not down; this state cannot fabricate a root cause. Docker unhealthy and stopped/dead runtime observations are valid direct-down evidence without a synthetic network probe. The resolver selects fresh component-scoped evidence deterministically, exposes conflicts rather than collapsing them and excludes stale/expired claims. AppOps inventory health is application-scoped provenance, not a component probe.

Probe engine

probes.py uses bounded async operations and one asyncio lock per service. Scheduler, probe-all and manual probe calls therefore coalesce at the service boundary rather than running duplicate network checks concurrently.

  • HTTP follows redirects, supports expected status ranges and records status/latency.
  • TCP opens and closes a bounded connection.
  • DNS resolves a hostname, returns at most 20 unique addresses and reports no-answer/timeout failures.
  • TLS performs a direct hostname-verified handshake, reports expiry metadata and applies a configurable warning threshold (SERVICEMAP_TLS_WARNING_DAYS, default 30).

Targets are explicitly typed. HTTP accepts only http/https URLs; TCP/TLS require host and port; DNS requires a host and does not accept arbitrary URI schemes. There is no file:// or generic URL fetcher.

Per-service interval, timeout and retry settings override the global defaults. Scheduled observations are marked scheduled; UI-triggered observations are marked manual.

Incidents and events

incidents.py deterministically finds a down service with no effectively-down dependency, then traverses the reverse graph for blast radius. sync_incidents() reconciles candidates with the durable journal:

  • one active incident per root service;
  • repeated probe cycles update impact instead of opening duplicates;
  • acknowledgement changes open to acknowledged and persists a timestamp;
  • recovery changes the record to resolved and retains resolved_at;
  • state transitions, flapping, incident lifecycle and maintenance changes are journaled.

Incident timeline entries are events tied to an incident. Raw probe rows are retained independently under the configured probe retention; non-incident events use SERVICEMAP_EVENT_RETENTION_DAYS (default 90). Incident lifecycle events are retained with the incident record.

A maintenance task runs 30 seconds after startup and then every SERVICEMAP_MAINTENANCE_INTERVAL_SECONDS (default 3600). It applies probe and event retention independently of scheduled probing, and deletes superseded health evidence older than SERVICEMAP_EVIDENCE_RETENTION_DAYS (default 7; 0 disables). Only inactive rows are deleted. A row becomes inactive only when a newer claim from the same provider and subject replaces it, so every subject keeps its current claim and health resolution is unchanged.

SSE and frontend refresh

The durable SQLite event journal is the source of truth. Each SSE client gets a bounded asyncio.Queue (size 100) bound to the event loop. A parked stream therefore holds no thread, and publishers on worker threads deliver to it through call_soon_threadsafe. Clients receive notifications for probe completion, state changes, health-evidence and provider changes, topology changes and incident lifecycle changes. A slow client loses only transient wakeups, never durable history. The stream starts with a retry: hint, sends a keepalive every 15 seconds, and unregisters its queue on disconnect or shutdown. A health-evidence update counts as material only when status, confidence, summary or stable metadata change; per-probe bookkeeping (probe_id, latency_ms, trigger) is stored but publishes nothing.

React maps each event type to the smallest set of data slices it can affect:

  • core reads: services, incidents, overview, events, demo state and the semantic topology;
  • discovery reads: only on topology.changed;
  • provider status: only on provider.status_changed;
  • the selected service's history: only for events about that service.

A single-flight scheduler merges bursts (250 ms debounce, 2 s maximum wait), so refreshes never overlap and never apply out of order. Reconnects use exponential backoff with jitter. The backoff resets only after a connection has been stable for 30 seconds, and every reopen after a gap triggers one full resync. A hidden tab keeps its stream for 60 seconds before it disconnects. The fallback poll runs only while the stream is down, and the UI shows a live, connecting or paused state. Authenticated clients use the same SSE protocol through fetch with a bearer header. The frontend never invents monitoring state optimistically; every node, counter and inspector value comes from the API.

Production discovery and reconciliation

DiscoveryStore owns durable runs, source records, observations, relation evidence, suggestions, overrides, topology-change events and compact canonical topology snapshots. DiscoveryManager reads the configured Docker endpoint and optional read-only AppOps inventory, then separates observed candidates from inferred relationships and canonical services. Candidates include stable Compose/name identity, image, state, health, labels, networks, ports, Compose metadata, healthcheck presence and sanitized endpoint-host evidence. ServiceMap labels (servicemap.*), Compose depends_on, safe environment hostnames and AppOps mappings produce explainable suggestions. Numeric AppOps discovery_confidence gates classification strength, while its app health remains scoped provenance. Shared networks are low-confidence review hints only.

Discovery runs do not mutate canonical topology. Reconciliation updates existing observed records and only imports new services or accepts high-confidence dependency relationships when the user explicitly requests it. Compose references resolve only inside the same project; environment references use project/network/global-unique scope and ambiguous matches remain review suggestions. Manual overrides and locked edges are preserved across later runs. Missing Docker identities move through missing, stale and retired without deleting probe history. Health propagation uses canonical dependency edges only; evidence-only or rejected relationships cannot affect health.

Published ports become active candidates only when SERVICEMAP_PROBE_HOST is explicitly configured. Internal Docker names are retained as evidence with internal_only reachability, never sent to the network probe engine by default. The reachability-repair endpoint provides a dry-run report before it disables unsafe legacy candidates or selects a verified published endpoint; it does not delete history and resolves false incidents with an endpoint-superseded event reason.

After reconciliation, a revision-hashed snapshot is captured only when canonical services or dependency edges change. The Changes workspace compares two snapshots and makes a service/relationship change actionable without auto-applying anything. The production UI starts with the canonical topology, exposes discovery diffs and evidence, supports individual accept/reject decisions, and offers an explicit edit mode for pinned node layout and manual dependency edges. Hidden/ignored lifecycle records remain available through the include_hidden API for reconciliation and audit context. ELK route sections are passed to LivingEdge, so its directed SVG path follows the actual orthogonal layout.

Configuration

Config v1 is still accepted. Config v2 serializes the service metadata and policy fields by human-readable service name. Parsing validates endpoint fields, duplicate names, duplicate/self edges and document-contained cycles. Import then validates the projected merge or replace graph and commits it in one BEGIN IMMEDIATE transaction.

Demo safety

The Northstar demo is built from deterministic UUIDs and synthetic probe records. Entering demo stores JSON snapshots of services, probes, events, incidents, discovery evidence, health evidence and provider state in the durable demo_sessions row. Reset replaces only the synthetic workspace while preserving the original snapshot. Exit restores the snapshot and deletes the session. Scheduled and manual live probes and discovery reconciliation are blocked while the demo is active.

Deployment trust boundary

The Unraid target uses /data on the named servicemap_data volume, port mapping 1253:8000, a healthcheck on /health, no-new-privileges, dropped capabilities, a read-only root filesystem and bounded /tmp tmpfs. Managed validation gates exact-SHA autoredeploy. Socket discovery remains a high-trust Docker API boundary even with a read-only bind. The opt-in broker provider requires a dedicated short-lived ServiceMap token, registration-scoped read capability and shared network, and never falls back to the socket. AppOps access is read-only enrichment.

Authorization

SERVICEMAP_AUTH_MODE selects one of three modes:

  • token is the default since v1.6.3. Reads stay anonymous, so integrations such as ITWorx Forge and monitors keep working. Every mutating request needs Authorization: Bearer <admin token>.
    • On first start ServiceMap writes a random token with mode 0600 to admin-token next to the database, or to SERVICEMAP_ADMIN_TOKEN_FILE when that is set.
    • Only the file's location is logged, never the token itself.
    • The comparison is constant-time.
    • The UI asks for the token once per browser tab, checks it with POST /api/v1/auth/verify and keeps it in sessionStorage. A 401 clears the stored token and asks again.
  • disabled preserves the v1.5 behavior.
  • oidc is described below.

Two guards apply in every mode:

  • A mutating request is refused when its Origin hostname differs from the public host (X-Forwarded-Host, otherwise Host), when the Origin is null, or when the browser marks it Sec-Fetch-Site: cross-site. Clients that send none of these headers, such as curl and integrations, are unaffected.
  • The optional SERVICEMAP_ALLOWED_HOSTS list refuses every other Host, which guards against DNS rebinding. Loopback is always allowed for the healthcheck.

Every response carries nosniff, X-Frame-Options: DENY, Referrer-Policy: no-referrer and a strict CSP. Request bodies are limited to 4 MiB, a config import to 2 MB, and YAML anchors and aliases are refused.

In oidc mode the role mapping also applies: a configuration export and the demo routes need the admin role. In oidc mode FastAPI validates issuer/audience/signed bearer JWTs against OIDC discovery and JWKS; it maps explicit roles or configured Authentik groups to viewer, operator and admin. /health remains anonymous for deployment recovery, while APIs fail closed when OIDC configuration is incomplete. No client credentials, issuer secrets or tokens are stored in ServiceMap's SQLite database or frontend configuration.