ITWorx-Pulse-Public/docs/architecture/DATA_MODEL.md

# Data model

## 1. Principles

- Stable internal UUIDs; external IDs are source-scoped aliases.
- Source facts and user overrides are stored separately.
- Soft deletion/tombstones preserve history and reconciliation.
- Important configuration is versioned.
- Alert state changes are transactional.
- Time is stored in UTC; locale/time zone applied at presentation.
- JSONB is used for bounded extensible attributes, not as a substitute for core relational structure.

## 2. Identity and access

### `users`
- id
- external_subject
- display_name
- email
- status
- created_at / updated_at / last_login_at

### `roles`, `user_roles`
Roles: viewer, operator, editor, administrator.

### `sessions` or external session metadata
Only if required by the chosen auth implementation. Avoid storing OIDC tokens unnecessarily.

## 3. Datasources

### `data_sources`
- id
- type
- name
- enabled
- configuration reference
- capability document
- health_state
- last_success_at
- last_error_code/message_redacted
- freshness policy
- created/updated

### `collectors`
- id
- datasource_id
- kind
- version
- heartbeat
- capabilities
- status

## 4. Inventory

### `entities`
- id UUID
- entity_type
- canonical_name
- display_name
- status
- status_reasons JSONB
- first_seen_at
- last_seen_at
- tombstoned_at
- attributes JSONB

### `entity_aliases`
- entity_id
- source_id
- external_type
- external_id
- unique(source_id, external_type, external_id)

### Container identity and recreation

Container runtime IDs are source-scoped aliases, not logical identity by themselves. A live runtime ID always matches its exact source/runtime alias. When a runtime ID changes, the same logical entity may be reused only when both the source-scoped compose project and compose service are present and uniquely identify one prior active alias. A container name, image, or digest alone is insufficient evidence and creates a new logical entity. The old runtime alias remains in history as inactive, the new alias points to the same logical entity, and recreation events keep that logical entity as events.entity_id while retaining previous/current runtime IDs in bounded attributes.
### `entity_facts`
- entity_id
- field_name
- source_id
- value JSONB
- observed_at
- confidence
- valid_until

### `entity_overrides`
- entity_id
- field_name
- value JSONB
- user_id
- updated_at

### `entity_relations`
- id
- source_entity_id
- relation_type
- target_entity_id
- source_id
- confidence
- confirmed
- first_seen/last_seen/tombstoned

### applications

- logical application ID and display name
- discovered component references
- critical/optional component policy
- aggregate status and bounded reason list
- user override metadata kept separate from discovery grouping
## 5. Metrics catalog

### `metric_definitions`
- id
- semantic_name unique
- version
- description
- unit
- value_kind
- source_kind
- query_template
- label contract
- limits
- allowed visualizations
- default transformations
- status policy reference
- enabled

### `metric_bindings`
Maps semantic metric to datasource/exporter/capability variants.

Metric samples remain in Prometheus.

## 6. Dashboards

### `dashboards`
- id
- slug
- name
- description
- owner_user_id nullable
- scope personal/shared/system
- archived_at
- current_version_id
- revision (optimistic concurrency token)
- created/updated

### `dashboard_versions`
- id
- dashboard_id
- version_number
- schema_version
- document JSONB
- change_summary
- created_by
- created_at

Dashboard versions are immutable. Dashboard writes lock the current row, compare revision, and commit the new version/current pointer atomically. The API stores only bounded revision metadata in audit diffs; full dashboard documents are not copied into audit events.

A dashboard version contains variables, widget instances, behavior and layout documents. Save as one atomic version to avoid partial layout/config updates.

Optional normalized indexes may extract widget type/entity references for search/impact analysis.

## 7. Events

### `events`
- id
- event_type
- severity
- entity_id nullable
- source_id
- occurred_at
- received_at
- dedup_key
- summary
- attributes JSONB bounded/redacted
- correlation_id
- unique(source_id, dedup_key, occurred_at bucket) as appropriate

## 8. Alerts

### `alert_rules`
- id
- name
- enabled
- severity
- evaluator type/config
- scope selector
- pending duration
- resolve duration
- cooldown duration for repeated firing notifications
- hysteresis config
- grouping labels
- suppression/dependency policy
- current_version_id
- revision (optimistic concurrency token)
- created/updated

### `alert_rule_versions`
Immutable rule documents and audit metadata.

### `alert_instances`
Stable entity/rule combination:
- id
- rule_id
- fingerprint
- entity_id
- current_state
- active_since
- last_evaluated_at
- last_value
- reason
- acknowledged_by/at
- cooldown_until
- silenced_until
- version for optimistic concurrency

### `alert_occurrences`
Immutable transitions/evaluation outcomes relevant to history.

## 9. Incidents

### `incidents`
- id
- title
- summary
- severity
- status
- started_at
- resolved_at
- owner_user_id
- correlation_method
- confidence
- created/updated
- version

### `incident_alerts`, `incident_entities`
Many-to-many links with rationale.

### `incident_notes`
- incident_id
- author
- body sanitized
- created_at

## 10. Maintenance and notifications

### `maintenance_windows`
- id
- name
- selector
- start/end or recurrence
- suppress notifications/state policy
- creator/audit

### `notification_channels`
Encrypted secret references and non-secret config.

### `notification_deliveries`
- occurrence/incident
- channel
- status
- attempts
- last_error_redacted
- timestamps
- idempotency_key

## 11. Audit

### `audit_events`
- actor/user/service
- action
- resource type/id
- result
- occurred_at
- correlation_id
- source IP/session metadata where appropriate
- before/after bounded diff with secret fields excluded

## 12. Operations

### `job_runs`
- job type/key
- scheduled/started/completed
- status
- counts
- error code
- correlation ID
- lease owner and lease expiry for coordinated worker execution

### `schema_migrations`
Managed by migration tool.

### `system_settings`
Typed/versioned settings; no plaintext secrets.

## 13. Concurrency

Use optimistic concurrency for:
- dashboard save;
- alert acknowledgement/state;
- incident edit;
- settings.

Return a conflict response with current version rather than silently overwriting.

## 14. Retention

Implement partitioning/cleanup when measurements justify it.

Baseline:
- inventory history/tombstones: enough for reconciliation and events;
- events: 1 year configurable;
- alert occurrences: 2 years configurable;
- incidents: retained until explicit policy;
- audit: at least 1 year configurable;
- job runs: shorter operational retention;
- dashboard versions: at least 180 days or fixed count plus protected versions.

### arrays and parity checks

An array snapshot preserves source-scoped provenance and UTC observed/received timestamps. Array members retain stable source IDs, role, state and bounded capacity/I/O counters. Parity state is separate from array state, and current/history parity checks retain progress, byte-per-second speed, error count and timestamps. Missing, disabled or emulated members are represented explicitly; stale source data maps to Unknown. Array transition events retain the array entity ID and bounded state/error attributes without exposing operational controls.
### disks and capacity

Disk identity is source-scoped and stable from the provider ID. A disk retains role, state, model, privacy-aware serial display, filesystem and bounded size/used/free/inode metrics. Used values above capacity are rejected; free space and utilization are derived from validated bytes, and inode utilization is derived separately. Missing disks remain in current/history projections instead of being dropped, while raw serials are excluded from API output.
### SMART facts

SMART is an optional disk capability with its own observed timestamp and freshness state. Normalization preserves generic overall result, bounded mapped attributes, critical/reason status and self-test result/age. Unavailable or stale SMART is Unknown and never Healthy; critical pending, reallocated, offline-uncorrectable, CRC or wear attributes remain visible even when the vendor overall result is passed.
### disk telemetry facts

Disk performance history is bounded and timestamped for live/history consumers. Temperature observations retain source time and policy-derived status; recovery does not clear attention until the configured recovery threshold is crossed. Latency/spin capabilities explicitly report unsupported when the source cannot provide them.

### pools and scrub facts

Pool identity is stable within its source and retains filesystem, usable/used/free bytes, profile/redundancy, bounded member state and filesystem error facts. Btrfs and ZFS capabilities are explicit and conditional; unsupported fields are never fabricated. Scrub current state and bounded history retain progress, bytes checked, errors and UTC timestamps. Stale source data maps pool state to Unknown while preserving provenance. Transition events identify degraded/faulted/recovered pools and scrub failures without exposing controls.
### shares and growth facts

Share identity is source-scoped and retains configured allocation/cache policy, source-owned used-size timestamp/state, bounded placement by pool and chronological growth points. Growth deltas and daily rates are derived from UTC observations, including negative changes, without exposing paths or file content. `ScanPlan` caps due size refreshes per run and records deferred work plus cache TTL so expensive scans remain bounded and observable.

### storage visualization projections

Storage map nodes are bounded projections of array members, pools and disks with entity links, kind, explicit state and detail text. Temperature heatmap points retain disk identity, UTC observed time, value and text status; visual cells never replace the accessible table alternative.
### capacity forecasts

A forecast is a bounded projection over UTC usage observations. Its policy records enabled state, maximum window, minimum points and method. Median daily growth is used for linear projection; bulk-import/outlier and irregular-interval safeguards lower confidence and suppress `daysToCapacity`/`projectedAt`. Method, window, point count, confidence and reason are always retained so the UI cannot present an unexplained precise date.

`capacity_samples` preserves bounded historical capacity observations for shares, pools and disks without storing file contents. Agent snapshot writes and their samples commit in one PostgreSQL transaction. A six-hour UTC bucket and primary key on `(entity_kind, entity_id, source_id, sampled_at)` make repeated discovery runs idempotent; a delayed retry may not replace a newer observation in the same bucket. The history index `(entity_kind, entity_id, sampled_at DESC)` supports the actual bounded forecast query. Forecast reads use at most 512 points inside the configured window and combine persisted share usage with the current canonical pool capacity. Insufficient, stale or unavailable histories remain assessments with `confidence=none`; only medium/high results count as qualified forecasts.
### services and probes

`services`, `service_endpoints` and `probes` represent reachable capabilities independently of container state. Configuration rows use `revision`, `updated_at` and `archived_at`; active names are unique per parent, while archived rows remain to preserve history. `probe_results` stores immutable timestamped outcomes, and `service_certificates` stores observed certificate facts. `service_dependencies` preserves source, confidence and confirmation state. `service_permissions` maps database roles to service permissions; secret references are identifiers only and never plaintext credentials.
### service status projections

Service status is derived from immutable `probe_results`, not from container state. The bounded projection retains current state/reason, last result/success/failure timestamps, latest latency, availability sample counts and bounded history. A result older than the configured freshness policy becomes `unknown`; it is never silently treated as healthy. State transitions produce deterministic service events with service identity, from/to state and reason.
M8-06 persistence adds alert_silences and maintenance_windows with bounded reason and name fields, JSON matcher or selector, UTC start and end timestamps, explicit active/expired/revoked status, creator and owner provenance, revocation and expiry timestamps, revision, expiry indexes, and deterministic listing indexes. These tables are additive and do not alter M1 tables or alert occurrence history.

M8-07 extends alert_occurrences with an immutable unacknowledge event type and adds an acknowledged-state index. Alert list/detail projections join rule and optional entity metadata while preserving instance revision and occurrence history.

### M8-08 notification persistence

- `notification_channels` stores only a bounded non-secret configuration and a reference to an external secret provider; the secret value is never part of the domain object or query result.
- `notification_outbox` has a unique idempotency key, bounded event content, a lease-aware status, bounded attempts and deterministic due ordering.
- `notification_deliveries` records each claimed attempt with a unique `(outbox_id, attempt)` key. Claim, delivery audit creation, completion and retry state changes are transactionally coordinated.
- Channel updates use optimistic revisions. Outbox rows retain recovery events and are not deleted after successful delivery, preserving auditability.
### M8-09 incident persistence

- `incidents` stores a bounded correlation key, title/summary, derived severity, lifecycle status, start/resolution timestamps, rationale method, confidence, owner and optimistic revision. Only one unresolved incident may exist for a correlation key.
- `incident_alerts` preserves per-alert rationale/confidence and distinguishes deterministic correlation from manual association. A manual association is never overwritten by a later correlation upsert.
- `incident_entities` keeps the affected entity set with bounded rationale/confidence and restricts entity deletion while incident evidence references it.

### M8-10 incident notes and ownership

`incident_notes` is bounded operator context attached to an incident. Bodies are normalized to plain text, stripped of tag-shaped markup and capped at 2,000 characters before persistence; notes are ordered by UTC creation time and stable ID. Notes cascade with their incident and never affect alert/entity evidence. Incident ownership is metadata updated with an optimistic revision, so stale UI writes become an explicit conflict.