Files
ServiceMap-Public/docs/ARCHITECTURE.md
NuklearRabbit 41a8eee259 ServiceMap: update dependencies and backend/app (+5 areas)
Public changes compared with the previous published export:
16 files changed.

Affected areas:
- backend/app: 3 files
- dependencies: 5 files
- deployment and CI: 2 files
- documentation: 2 files
- frontend/src: 1 file
- project files: 1 file
- tests: 2 files

Changed files:
- Updated: ".env.example"
- Updated: "Dockerfile.unraid"
- Updated: "README.md"
- Updated: "backend/app/incidents.py"
- Updated: "backend/app/main.py"
- Updated: "backend/app/store.py"
- Updated: "backend/pyproject.toml"
- Updated: "backend/tests/test_docker_provider.py"
- Updated: "backend/tests/test_v163_health_performance.py"
- Updated: "docker-compose.broker.yml"
- Updated: "docs/ARCHITECTURE.md"
- Updated: "frontend/package-lock.json"
- Updated: "frontend/package.json"
- Updated: "frontend/src/App.tsx"
- Updated: "package-lock.json"
- Updated: "package.json"

Export provenance/manifests refreshed; excluded from the change summary.

Publish curated ServiceMap source from 8c297f756ff84aaa917ae9c48535e835f8c09ec7
2026-11-20 16:01:36 +01:00

148 lines
18 KiB
Markdown

# ServiceMap architecture
## Runtime shape
ServiceMap remains deliberately small: one FastAPI process, one React single-page application and one SQLite database. Compose serves React through Nginx; the Unraid image bundles the same frontend and serves it from FastAPI.
```text
Browser
│ REST + SSE
▼
React workspaces
│
▼
FastAPI
├── service inventory and dependency validation
├── probe engine (HTTP, TCP, DNS, TLS)
├── state machine and flapping observation
├── health-evidence resolution and provider health
├── incident correlation and event journal
├── SSE fan-out bus
├── atomic config import/export
├── Docker/AppOps discovery and reconciliation
├── evidence-backed canonical topology overrides
└── optional per-service scheduler
│
▼
SQLite on /data
```
## Data ownership and migration
`Store` owns SQLite access and opens short-lived connections with WAL mode, a 5-second busy timeout and foreign keys enabled. Startup creates missing discovery tables and adds versioned metadata in place. v1.6 adds health evidence, evidence-to-service links and provider-status rows; the one-time migration projects only legitimate existing latest probe/runtime facts with their original timestamps. Existing rows receive safe defaults; the migration never deletes or rewrites topology data.
v1.6.1 adds no schema migration. Its health-only observation loop is distinct from discovery: every configured cadence it reads Docker runtime/healthcheck facts and AppOps application inventory, renews only source-scoped evidence/provider status, and skips demo state. It does not create services, suggestions, relationship evidence, snapshots, probe endpoints or operator overrides. The shared observation TTL is at least 150 seconds and three normal cycles, so a delayed provider cycle ages evidence rather than immediately turning the topology unknown. Provider failures are isolated; they update that provider's availability only and leave other current evidence intact.
v1.6.2 keeps the same schema and evidence scopes. An AppOps `warning` observation is degraded application evidence; it cannot resolve a component. Evidence freshness is recomputed when persisted claims are read, so an expired claim cannot appear fresh in the raw inspector/API payload. The current MCP Hub Docker read broker cannot replace the production socket provider yet: its registration covers two unrelated projects, its ServiceMap-scoped inventory is empty, and its short-lived HMAC service-token contract rejects a raw bearer secret. Cutover requires a dedicated ServiceMap read identity, full bounded inventory parity and an explicit network path; broker mode remains fail-closed with no socket fallback.
The HTTP discovery routes accept only the Docker host and AppOps URL selected in deployment configuration. A request body cannot turn the container into a general-purpose client for another Docker daemon or internal HTTP endpoint. The private-LAN deployment keeps OIDC disabled because it has no dedicated browser client or tested owner login path. Since v1.6.3 it runs in `token` mode, so mutation APIs need the admin token instead of depending on the LAN perimeter. The broker Compose uses a dedicated read-only token directory and never mounts the Docker socket. A scoped issuer can atomically replace the token in that directory. Each new inventory client reads the current file. Activation still requires a dedicated Hub identity, matching inventory coverage and a healthy rollback path.
The v1.4 semantic projection is deliberately not a parallel datastore. `semantic.py` derives Domain → Application → Component from canonical services, relations, runtime/monitoring state and incident analysis. Logical relations are grouped at query time so one graph edge retains all independent evidence records and their first/last-seen provenance. v1.6.1 resolves application health separately: fresh aggregate AppOps evidence can answer an application but never a component; direct component failures override weak aggregate health and conflicting claims remain visible. Semantic application identity uses locked operator grouping, Compose provenance and stable source references before a display name, preventing generic provider labels from collapsing unrelated services. Bulk evidence lookup serves the complete semantic projection in one bounded SQL query rather than an application-by-application scan. v1.6.3 matches direct and linked claims through two separately indexed lookups (a `UNION`) and applies the per-service limit with a window function, because an `OR` across the link join forced a full `health_evidence` scan. Direct, dependency and effective health for incidents, the overview and service detail all come from one bulk read and one in-memory traversal (`health.topology_health`), never a per-node recursive SQLite walk. Snapshot discovery context (sources and relation evidence) is also read in bulk.
`GET /api/v1/topology/semantic` sends a strong `ETag` computed over the content without `generated_at`, together with `Cache-Control: no-cache` and `X-ServiceMap-Generated-At`. A client that sends `If-None-Match` while nothing has changed receives `304 Not Modified` without a body. Browsers revalidate on their own, and integrations such as ITWorx Forge send the header explicitly. The serialized response is also cached for at most 10 seconds. The cache is dropped as soon as any event is published or any evidence crosses a fresh→stale→expired boundary. The age cap bounds staleness for the few write paths that publish no event, such as discovery relation evidence; the ETag is always derived from content, so it is correct regardless.
The service record stores descriptive metadata, group/environment/tags, criticality, source reference, application, runtime state/health, monitoring state, probe source/reachability/confidence, enablement, maintenance state, probe policy and state-machine counters. Probe rows remain immutable and retain `trigger` and bounded JSON `metadata` fields. `HealthEvidence` retains provider, subject, scope, confidence, observation/receipt/expiry timestamps, metadata and supersession. Application-scoped claims may be linked to component inspectors but cannot resolve a component's direct health.
Indexes cover service/time probe history, global probe time, event time/service/incident and incident status/root lookups. Availability is calculated with SQL aggregates over a bounded time window rather than loading all rows into Python. Evidence timestamp renewals retain a stable row and do not publish SSE; material provider status, evidence status, confidence, summary or metadata changes do.
## Health state and dependency semantics
The direct state machine is backend-owned, then exposed through an evidence policy:
```text
unknown ──success──> healthy
healthy ──failure threshold──> degraded/down
down/degraded ──success #1──> recovering
recovering ──recovery threshold──> healthy
any monitored state ──maintenance──> maintenance
```
The first failure can be held at the prior/unknown state when a failure threshold is greater than one. Repeated state changes in a bounded 30-minute query set the `flapping` observation and its count; flapping is not a competing health state.
Effective health is calculated recursively from direct state and dependencies. A down dependency makes a healthy dependent degraded; the stored probe observation is never rewritten. Cycles and missing dependencies are rejected before writes.
Maintenance and disabled monitoring do not produce a new incident. A running Docker component without a verified endpoint or Docker healthcheck is `unmonitored`, not `down`; this state cannot fabricate a root cause. Docker `unhealthy` and stopped/dead runtime observations are valid direct-down evidence without a synthetic network probe. The resolver selects fresh component-scoped evidence deterministically, exposes conflicts rather than collapsing them and excludes stale/expired claims. AppOps inventory health is application-scoped provenance, not a component probe.
## Probe engine
`probes.py` uses bounded async operations and one asyncio lock per service. Scheduler, probe-all and manual probe calls therefore coalesce at the service boundary rather than running duplicate network checks concurrently.
- HTTP follows redirects, supports expected status ranges and records status/latency.
- TCP opens and closes a bounded connection.
- DNS resolves a hostname, returns at most 20 unique addresses and reports no-answer/timeout failures.
- TLS performs a direct hostname-verified handshake, reports expiry metadata and applies a configurable warning threshold (`SERVICEMAP_TLS_WARNING_DAYS`, default 30).
Targets are explicitly typed. HTTP accepts only `http`/`https` URLs; TCP/TLS require host and port; DNS requires a host and does not accept arbitrary URI schemes. There is no `file://` or generic URL fetcher.
Per-service interval, timeout and retry settings override the global defaults. Scheduled observations are marked `scheduled`; UI-triggered observations are marked `manual`.
## Incidents and events
`incidents.py` deterministically finds a down service with no effectively-down dependency, then traverses the reverse graph for blast radius. `sync_incidents()` reconciles candidates with the durable journal:
- one active incident per root service;
- repeated probe cycles update impact instead of opening duplicates;
- acknowledgement changes `open` to `acknowledged` and persists a timestamp;
- recovery changes the record to `resolved` and retains `resolved_at`;
- state transitions, flapping, incident lifecycle and maintenance changes are journaled.
Incident timeline entries are events tied to an incident. Raw probe rows are retained independently under the configured probe retention; non-incident events use `SERVICEMAP_EVENT_RETENTION_DAYS` (default 90). Incident lifecycle events are retained with the incident record.
A maintenance task runs 30 seconds after startup and then every `SERVICEMAP_MAINTENANCE_INTERVAL_SECONDS` (default 3600). It applies probe and event retention independently of scheduled probing, and deletes superseded health evidence older than `SERVICEMAP_EVIDENCE_RETENTION_DAYS` (default 7; `0` disables). Only inactive rows are deleted. A row becomes inactive only when a newer claim from the same provider and subject replaces it, so every subject keeps its current claim and health resolution is unchanged.
## SSE and frontend refresh
The durable SQLite event journal is the source of truth. Each SSE client gets a bounded `asyncio.Queue` (size 100) bound to the event loop. A parked stream therefore holds no thread, and publishers on worker threads deliver to it through `call_soon_threadsafe`. Clients receive notifications for probe completion, state changes, health-evidence and provider changes, topology changes and incident lifecycle changes. A slow client loses only transient wakeups, never durable history. The stream starts with a `retry:` hint, sends a keepalive every 15 seconds, and unregisters its queue on disconnect or shutdown. A health-evidence update counts as material only when status, confidence, summary or stable metadata change; per-probe bookkeeping (`probe_id`, `latency_ms`, `trigger`) is stored but publishes nothing.
React maps each event type to the smallest set of data slices it can affect:
- core reads: services, incidents, overview, events, demo state and the semantic topology;
- discovery reads: only on `topology.changed`;
- provider status: only on `provider.status_changed`;
- the selected service's history: only for events about that service.
A single-flight scheduler merges bursts (250 ms debounce, 2 s maximum wait), so refreshes never overlap and never apply out of order. Reconnects use exponential backoff with jitter. The backoff resets only after a connection has been stable for 30 seconds, and every reopen after a gap triggers one full resync. A hidden tab keeps its stream for 60 seconds before it disconnects. The fallback poll runs only while the stream is down, and the UI shows a live, connecting or paused state. Authenticated clients use the same SSE protocol through `fetch` with a bearer header. The frontend never invents monitoring state optimistically; every node, counter and inspector value comes from the API.
## Production discovery and reconciliation
`DiscoveryStore` owns durable runs, source records, observations, relation evidence, suggestions, overrides, topology-change events and compact canonical topology snapshots. `DiscoveryManager` reads the configured Docker endpoint and optional read-only AppOps inventory, then separates observed candidates from inferred relationships and canonical services. Candidates include stable Compose/name identity, image, state, health, labels, networks, ports, Compose metadata, healthcheck presence and sanitized endpoint-host evidence. ServiceMap labels (`servicemap.*`), Compose `depends_on`, safe environment hostnames and AppOps mappings produce explainable suggestions. Numeric AppOps `discovery_confidence` gates classification strength, while its app health remains scoped provenance. Shared networks are low-confidence review hints only.
Discovery runs do not mutate canonical topology. Reconciliation updates existing observed records and only imports new services or accepts high-confidence dependency relationships when the user explicitly requests it. Compose references resolve only inside the same project; environment references use project/network/global-unique scope and ambiguous matches remain review suggestions. Manual overrides and locked edges are preserved across later runs. Missing Docker identities move through `missing`, `stale` and `retired` without deleting probe history. Health propagation uses canonical dependency edges only; evidence-only or rejected relationships cannot affect health.
Published ports become active candidates only when `SERVICEMAP_PROBE_HOST` is explicitly configured. Internal Docker names are retained as evidence with `internal_only` reachability, never sent to the network probe engine by default. The reachability-repair endpoint provides a dry-run report before it disables unsafe legacy candidates or selects a verified published endpoint; it does not delete history and resolves false incidents with an endpoint-superseded event reason.
After reconciliation, a revision-hashed snapshot is captured only when canonical services or dependency edges change. The Changes workspace compares two snapshots and makes a service/relationship change actionable without auto-applying anything. The production UI starts with the canonical topology, exposes discovery diffs and evidence, supports individual accept/reject decisions, and offers an explicit edit mode for pinned node layout and manual dependency edges. Hidden/ignored lifecycle records remain available through the `include_hidden` API for reconciliation and audit context. ELK route sections are passed to `LivingEdge`, so its directed SVG path follows the actual orthogonal layout.
## Configuration
Config v1 is still accepted. Config v2 serializes the service metadata and policy fields by human-readable service name. Parsing validates endpoint fields, duplicate names, duplicate/self edges and document-contained cycles. Import then validates the projected merge or replace graph and commits it in one `BEGIN IMMEDIATE` transaction.
## Demo safety
The Northstar demo is built from deterministic UUIDs and synthetic probe records. Entering demo stores JSON snapshots of services, probes, events, incidents, discovery evidence, health evidence and provider state in the durable `demo_sessions` row. Reset replaces only the synthetic workspace while preserving the original snapshot. Exit restores the snapshot and deletes the session. Scheduled and manual live probes and discovery reconciliation are blocked while the demo is active.
## Deployment trust boundary
The Unraid target uses `/data` on the named `servicemap_data` volume, port mapping `1253:8000`, a healthcheck on `/health`, `no-new-privileges`, dropped capabilities, a read-only root filesystem and bounded `/tmp` tmpfs. Managed validation gates exact-SHA autoredeploy. Socket discovery remains a high-trust Docker API boundary even with a read-only bind. The opt-in broker provider requires a dedicated short-lived ServiceMap token, registration-scoped read capability and shared network, and never falls back to the socket. AppOps access is read-only enrichment.
## Authorization
`SERVICEMAP_AUTH_MODE` selects one of three modes:
- **`token`** is the default since v1.6.3. Reads stay anonymous, so integrations such as ITWorx Forge and monitors keep working. Every mutating request needs `Authorization: Bearer <admin token>`.
- On first start ServiceMap writes a random token with mode 0600 to `admin-token` next to the database, or to `SERVICEMAP_ADMIN_TOKEN_FILE` when that is set.
- Only the file's location is logged, never the token itself.
- The comparison is constant-time.
- The UI asks for the token once per browser tab, checks it with `POST /api/v1/auth/verify` and keeps it in `sessionStorage`. A 401 clears the stored token and asks again.
- **`disabled`** preserves the v1.5 behavior.
- **`oidc`** is described below.
Two guards apply in every mode:
- A mutating request is refused when its `Origin` hostname differs from the public host (`X-Forwarded-Host`, otherwise `Host`), when the `Origin` is `null`, or when the browser marks it `Sec-Fetch-Site: cross-site`. Clients that send none of these headers, such as curl and integrations, are unaffected.
- The optional `SERVICEMAP_ALLOWED_HOSTS` list refuses every other `Host`, which guards against DNS rebinding. Loopback is always allowed for the healthcheck.
Every response carries `nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: no-referrer` and a strict CSP. Request bodies are limited to 4 MiB, a config import to 2 MB, and YAML anchors and aliases are refused.
In `oidc` mode the role mapping also applies: a configuration export and the demo routes need the admin role. In `oidc` mode FastAPI validates issuer/audience/signed bearer JWTs against OIDC discovery and JWKS; it maps explicit roles or configured Authentik groups to viewer, operator and admin. `/health` remains anonymous for deployment recovery, while APIs fail closed when OIDC configuration is incomplete. No client credentials, issuer secrets or tokens are stored in ServiceMap's SQLite database or frontend configuration.