ITWorx-Pulse-Public/docs/architecture/DATA_MODEL.md

Data model

1. Principles

• Stable internal UUIDs; external IDs are source-scoped aliases.
• Source facts and user overrides are stored separately.
• Soft deletion/tombstones preserve history and reconciliation.
• Important configuration is versioned.
• Alert state changes are transactional.
• Time is stored in UTC; locale/time zone applied at presentation.
• JSONB is used for bounded extensible attributes, not as a substitute for core relational structure.

2. Identity and access

users

• id
• external_subject
• display_name
• email
• status
• created_at / updated_at / last_login_at

roles, user_roles

Roles: viewer, operator, editor, administrator.

sessions or external session metadata

Only if required by the chosen auth implementation. Avoid storing OIDC tokens unnecessarily.

3. Datasources

data_sources

• id
• type
• name
• enabled
• configuration reference
• capability document
• health_state
• last_success_at
• last_error_code/message_redacted
• freshness policy
• created/updated

collectors

• id
• datasource_id
• kind
• version
• heartbeat
• capabilities
• status

4. Inventory

entities

• id UUID
• entity_type
• canonical_name
• display_name
• status
• status_reasons JSONB
• first_seen_at
• last_seen_at
• tombstoned_at
• attributes JSONB

entity_aliases

• entity_id
• source_id
• external_type
• external_id
• unique(source_id, external_type, external_id)

Container identity and recreation

Container runtime IDs are source-scoped aliases, not logical identity by themselves. A live runtime ID always matches its exact source/runtime alias. When a runtime ID changes, the same logical entity may be reused only when both the source-scoped compose project and compose service are present and uniquely identify one prior active alias. A container name, image, or digest alone is insufficient evidence and creates a new logical entity. The old runtime alias remains in history as inactive, the new alias points to the same logical entity, and recreation events keep that logical entity as events.entity_id while retaining previous/current runtime IDs in bounded attributes.

entity_facts

• entity_id
• field_name
• source_id
• value JSONB
• observed_at
• confidence
• valid_until

entity_overrides

• entity_id
• field_name
• value JSONB
• user_id
• updated_at

entity_relations

• id
• source_entity_id
• relation_type
• target_entity_id
• source_id
• confidence
• confirmed
• first_seen/last_seen/tombstoned

applications

• logical application ID and display name
• discovered component references
• critical/optional component policy
• aggregate status and bounded reason list
• user override metadata kept separate from discovery grouping

5. Metrics catalog

metric_definitions

• id
• semantic_name unique
• version
• description
• unit
• value_kind
• source_kind
• query_template
• label contract
• limits
• allowed visualizations
• default transformations
• status policy reference
• enabled

metric_bindings

Maps semantic metric to datasource/exporter/capability variants.

Metric samples remain in Prometheus.

6. Dashboards

dashboards

• id
• slug
• name
• description
• owner_user_id nullable
• scope personal/shared/system
• archived_at
• current_version_id
• revision (optimistic concurrency token)
• created/updated

dashboard_versions

• id
• dashboard_id
• version_number
• schema_version
• document JSONB
• change_summary
• created_by
• created_at

Dashboard versions are immutable. Dashboard writes lock the current row, compare revision, and commit the new version/current pointer atomically. The API stores only bounded revision metadata in audit diffs; full dashboard documents are not copied into audit events.

A dashboard version contains variables, widget instances, behavior and layout documents. Save as one atomic version to avoid partial layout/config updates.

Optional normalized indexes may extract widget type/entity references for search/impact analysis.

7. Events

events

• id
• event_type
• severity
• entity_id nullable
• source_id
• occurred_at
• received_at
• dedup_key
• summary
• attributes JSONB bounded/redacted
• correlation_id
• unique(source_id, dedup_key, occurred_at bucket) as appropriate

8. Alerts

alert_rules

• id
• name
• enabled
• severity
• evaluator type/config
• scope selector
• pending duration
• resolve duration
• cooldown duration for repeated firing notifications
• hysteresis config
• grouping labels
• suppression/dependency policy
• current_version_id
• revision (optimistic concurrency token)
• created/updated

alert_rule_versions

Immutable rule documents and audit metadata.

alert_instances

Stable entity/rule combination:

• id
• rule_id
• fingerprint
• entity_id
• current_state
• active_since
• last_evaluated_at
• last_value
• reason
• acknowledged_by/at
• cooldown_until
• silenced_until
• version for optimistic concurrency

alert_occurrences

Immutable transitions/evaluation outcomes relevant to history.

9. Incidents

incidents

• id
• title
• summary
• severity
• status
• started_at
• resolved_at
• owner_user_id
• correlation_method
• confidence
• created/updated
• version

incident_alerts, incident_entities

Many-to-many links with rationale.

incident_notes

• incident_id
• author
• body sanitized
• created_at

10. Maintenance and notifications

maintenance_windows

• id
• name
• selector
• start/end or recurrence
• suppress notifications/state policy
• creator/audit

notification_channels

Encrypted secret references and non-secret config.

notification_deliveries

• occurrence/incident
• channel
• status
• attempts
• last_error_redacted
• timestamps
• idempotency_key

11. Audit

audit_events

• actor/user/service
• action
• resource type/id
• result
• occurred_at
• correlation_id
• source IP/session metadata where appropriate
• before/after bounded diff with secret fields excluded

12. Operations

job_runs

• job type/key
• scheduled/started/completed
• status
• counts
• error code
• correlation ID
• lease owner and lease expiry for coordinated worker execution

schema_migrations

Managed by migration tool.

system_settings

Typed/versioned settings; no plaintext secrets.

13. Concurrency

Use optimistic concurrency for:

• dashboard save;
• alert acknowledgement/state;
• incident edit;
• settings.

Return a conflict response with current version rather than silently overwriting.

14. Retention

Implement partitioning/cleanup when measurements justify it.

Baseline:

• inventory history/tombstones: enough for reconciliation and events;
• events: 1 year configurable;
• alert occurrences: 2 years configurable;
• incidents: retained until explicit policy;
• audit: at least 1 year configurable;
• job runs: shorter operational retention;
• dashboard versions: at least 180 days or fixed count plus protected versions.

arrays and parity checks

An array snapshot preserves source-scoped provenance and UTC observed/received timestamps. Array members retain stable source IDs, role, state and bounded capacity/I/O counters. Parity state is separate from array state, and current/history parity checks retain progress, byte-per-second speed, error count and timestamps. Missing, disabled or emulated members are represented explicitly; stale source data maps to Unknown. Array transition events retain the array entity ID and bounded state/error attributes without exposing operational controls.

disks and capacity

Disk identity is source-scoped and stable from the provider ID. A disk retains role, state, model, privacy-aware serial display, filesystem and bounded size/used/free/inode metrics. Used values above capacity are rejected; free space and utilization are derived from validated bytes, and inode utilization is derived separately. Missing disks remain in current/history projections instead of being dropped, while raw serials are excluded from API output.

SMART facts

SMART is an optional disk capability with its own observed timestamp and freshness state. Normalization preserves generic overall result, bounded mapped attributes, critical/reason status and self-test result/age. Unavailable or stale SMART is Unknown and never Healthy; critical pending, reallocated, offline-uncorrectable, CRC or wear attributes remain visible even when the vendor overall result is passed.

disk telemetry facts

Disk performance history is bounded and timestamped for live/history consumers. Temperature observations retain source time and policy-derived status; recovery does not clear attention until the configured recovery threshold is crossed. Latency/spin capabilities explicitly report unsupported when the source cannot provide them.

pools and scrub facts

Pool identity is stable within its source and retains filesystem, usable/used/free bytes, profile/redundancy, bounded member state and filesystem error facts. Btrfs and ZFS capabilities are explicit and conditional; unsupported fields are never fabricated. Scrub current state and bounded history retain progress, bytes checked, errors and UTC timestamps. Stale source data maps pool state to Unknown while preserving provenance. Transition events identify degraded/faulted/recovered pools and scrub failures without exposing controls.

shares and growth facts

Share identity is source-scoped and retains configured allocation/cache policy, source-owned used-size timestamp/state, bounded placement by pool and chronological growth points. Growth deltas and daily rates are derived from UTC observations, including negative changes, without exposing paths or file content. ScanPlan caps due size refreshes per run and records deferred work plus cache TTL so expensive scans remain bounded and observable.

storage visualization projections

Storage map nodes are bounded projections of array members, pools and disks with entity links, kind, explicit state and detail text. Temperature heatmap points retain disk identity, UTC observed time, value and text status; visual cells never replace the accessible table alternative.

capacity forecasts

A forecast is a bounded projection over UTC usage observations. Its policy records enabled state, maximum window, minimum points and method. Median daily growth is used for linear projection; bulk-import/outlier and irregular-interval safeguards lower confidence and suppress daysToCapacity/projectedAt. Method, window, point count, confidence and reason are always retained so the UI cannot present an unexplained precise date.

capacity_samples preserves bounded historical capacity observations for shares, pools and disks without storing file contents. Agent snapshot writes and their samples commit in one PostgreSQL transaction. A six-hour UTC bucket and primary key on (entity_kind, entity_id, source_id, sampled_at) make repeated discovery runs idempotent; a delayed retry may not replace a newer observation in the same bucket. The history index (entity_kind, entity_id, sampled_at DESC) supports the actual bounded forecast query. Forecast reads use at most 512 points inside the configured window and combine persisted share usage with the current canonical pool capacity. Insufficient, stale or unavailable histories remain assessments with confidence=none; only medium/high results count as qualified forecasts.

services and probes

services, service_endpoints and probes represent reachable capabilities independently of container state. Configuration rows use revision, updated_at and archived_at; active names are unique per parent, while archived rows remain to preserve history. probe_results stores immutable timestamped outcomes, and service_certificates stores observed certificate facts. service_dependencies preserves source, confidence and confirmation state. service_permissions maps database roles to service permissions; secret references are identifiers only and never plaintext credentials.

service status projections

Service status is derived from immutable probe_results, not from container state. The bounded projection retains current state/reason, last result/success/failure timestamps, latest latency, availability sample counts and bounded history. A result older than the configured freshness policy becomes unknown; it is never silently treated as healthy. State transitions produce deterministic service events with service identity, from/to state and reason. M8-06 persistence adds alert_silences and maintenance_windows with bounded reason and name fields, JSON matcher or selector, UTC start and end timestamps, explicit active/expired/revoked status, creator and owner provenance, revocation and expiry timestamps, revision, expiry indexes, and deterministic listing indexes. These tables are additive and do not alter M1 tables or alert occurrence history.

M8-07 extends alert_occurrences with an immutable unacknowledge event type and adds an acknowledged-state index. Alert list/detail projections join rule and optional entity metadata while preserving instance revision and occurrence history.

M8-08 notification persistence

• notification_channels stores only a bounded non-secret configuration and a reference to an external secret provider; the secret value is never part of the domain object or query result.
• notification_outbox has a unique idempotency key, bounded event content, a lease-aware status, bounded attempts and deterministic due ordering.
• notification_deliveries records each claimed attempt with a unique (outbox_id, attempt) key. Claim, delivery audit creation, completion and retry state changes are transactionally coordinated.
• Channel updates use optimistic revisions. Outbox rows retain recovery events and are not deleted after successful delivery, preserving auditability.

M8-09 incident persistence

• incidents stores a bounded correlation key, title/summary, derived severity, lifecycle status, start/resolution timestamps, rationale method, confidence, owner and optimistic revision. Only one unresolved incident may exist for a correlation key.
• incident_alerts preserves per-alert rationale/confidence and distinguishes deterministic correlation from manual association. A manual association is never overwritten by a later correlation upsert.
• incident_entities keeps the affected entity set with bounded rationale/confidence and restricts entity deletion while incident evidence references it.

M8-10 incident notes and ownership

incident_notes is bounded operator context attached to an incident. Bodies are normalized to plain text, stripped of tag-shaped markup and capped at 2,000 characters before persistence; notes are ordered by UTC creation time and stable ID. Notes cascade with their incident and never affect alert/entity evidence. Incident ownership is metadata updated with an optimistic revision, so stale UI writes become an explicit conflict.