ITWorx-Pulse-Public/docs/architecture/TELEMETRY_AND_QUERY_ENGINE.md

Telemetry and query engine

1. Purpose

Pulse exposes a stable semantic metric model while retaining Prometheus as the v1 time-series source.

Users/widgets request:

container.cpu.utilization
storage.disk.temperature
service.response_time

The backend maps these names to source-specific PromQL templates and label contracts.

2. Metric definition

A metric definition includes:

• semantic name and version;
• description and unit;
• gauge/counter/state kind;
• required capabilities;
• query template;
• allowed labels/grouping;
• default aggregation;
• valid transformations;
• default interval;
• max range/series/points;
• freshness;
• visualizations;
• status/threshold hints;
• cardinality budget.

See specs/metric-definition.schema.json.

3. Query planning

Pipeline:

1. authenticate and authorize;
2. validate semantic metric and scope;
3. resolve entity aliases/source binding;
4. clamp/validate range, step, series and points;
5. select binding based on source capabilities;
6. generate parameterized PromQL from templates;
7. deduplicate equivalent queries;
8. execute with timeout/concurrency budget;
9. normalize labels, units and missing values;
10. downsample if needed;
11. return provenance, warnings and freshness.

Raw PromQL is an advanced feature, disabled by default, separately authorized and constrained.

4. Query budgets

Budgets are configurable but must exist:

• max range;
• max series;
• max samples/points;
• max query string/template expansion;
• timeout;
• per-user/per-dashboard concurrency;
• global Prometheus concurrency;
• cacheable result size;
• live subscription count;
• label enumeration limits.

Return a clear limit error rather than overloading Prometheus.

5. Caching

Cache:

• metric catalog/capabilities;
• short historical range results;
• label/metadata results;
• repeated dashboard query plans.

Do not cache:

• authorization decisions beyond safe session scope;
• live current status beyond its freshness policy;
• secret-bearing errors.

Cache keys include source, tenant/server scope, semantic query, normalized range and permissions where relevant.

Redis is not required for a single instance. Use bounded in-memory cache and/or PostgreSQL only when appropriate.

6. Live engine

Fast lane

• visible high-frequency widgets subscribe at 1–5 seconds;
• backend polls/queries compatible batches or receives collector samples;
• equivalent subscriptions share upstream work;
• samples are appended with sequence numbers;
• browser retains a bounded ring buffer;
• historical data is not fully refetched per sample.

Durable lane

Prometheus scrape/history remains durable. Pulse does not persist every live sample into PostgreSQL.

Adaptive behavior

• out-of-view widgets reduce frequency;
• background tabs reduce frequency;
• paused dashboards unsubscribe;
• wallboards remain active with bounded buffers;
• slow clients receive coalesced latest samples;
• sequence gaps trigger bounded resync.

7. Browser chart architecture

• historical query initializes chart;
• live samples append outside global React state where practical;
• series count and point count are capped;
• old points are evicted;
• chart resources are disposed on unmount;
• ResizeObserver/visibility changes are debounced;
• tooltip/legend state does not duplicate large arrays;
• long wallboard test measures heap and subscription count.

8. Staleness and unknown

Each response includes:

• source timestamp;
• received timestamp;
• freshness state;
• warnings.

If a required source is stale/unavailable:

• current value is marked stale or omitted;
• status becomes unknown according to policy;
• previous value may be displayed with age;
• alerts can enter unknown rather than resolve.

9. Host snapshot binding

Host detail reads use the explicit read-only host contract. A source adapter returns bounded normalized values with UTC observed/received timestamps and capability version. The adapter sorts filesystem/interface collections deterministically, validates percentage/byte units and rejects payloads over configured collection limits. The API/web boundary has no host or Docker privilege; when no approved source is connected, the result is an explicit unknown snapshot. Load and memory status reasons are returned as text so high load is distinguishable from stale or unavailable telemetry.

9. Transformations

Supported through typed operations:

• rate;
• increase/delta;
• average/min/max/sum;
• quantile;
• percentage;
• top/bottom N;
• unit conversion;
• compare previous period;
• fill policy;
• status mapping.

Transform order is explicit and validated. Avoid silently mixing counter rates and gauges.

Optional hardware follows the same boundary: declared thermal, fan and GPU capabilities are independently enabled, unsupported, unavailable or disabled. Missing capability is not an adapter failure. Sensor IDs are source-scoped and stable from external ID/name, collections are bounded and sorted, and thermal reasons are returned as text. No API/web host privilege, device mount or unrestricted socket is needed.

10. Cardinality controls

• semantic definitions allow only known labels;
• unbounded label values are excluded or normalized;
• inventory IDs map to stable bounded labels;
• dashboards cannot group by arbitrary label by default;
• source cardinality health is monitored;
• query inspector shows series count and limits.
• query inspector is operate-permissioned, exposes only planner-approved generated PromQL and redacts sensitive-looking text; it does not execute an unrestricted query.