RepoPulse: update backend/app and dependencies (+3 areas)
Public changes compared with the previous published export: 26 files changed. Affected areas: - backend/app: 11 files - dependencies: 5 files - deployment and CI: 3 files - documentation: 2 files - tests: 5 files Changed files: - Updated: "Dockerfile.unraid" - Updated: "README.md" - Updated: "backend/Dockerfile" - Updated: "backend/app/acquisition.py" - Updated: "backend/app/architecture.py" - Updated: "backend/app/git_analysis.py" - Updated: "backend/app/gitea.py" - Updated: "backend/app/jobs.py" - Updated: "backend/app/main.py" - Added: "backend/app/processes.py" - Updated: "backend/app/rules.py" - Updated: "backend/app/scanner.py" - Updated: "backend/app/signals.py" - Updated: "backend/app/store.py" - Updated: "backend/pyproject.toml" - Added: "backend/tests/test_api_responsiveness.py" - Added: "backend/tests/test_processes.py" - Updated: "backend/tests/test_scanner.py" - Updated: "backend/tests/test_signals_v11.py" - Updated: "backend/tests/test_store.py" - Updated: "docker-compose.yml" - Updated: "docs/ARCHITECTURE.md" - Updated: "frontend/package-lock.json" - Updated: "frontend/package.json" - Updated: "package-lock.json" - Updated: "package.json" Export provenance/manifests refreshed; excluded from the change summary. Publish curated RepoPulse source from a251f826238908b156485299061da76482780ed2
This commit is contained in:
1 parent
23a4a721f1
commit
61ae49f9d0
28 files changed
+1083
-408
No files matched your search
+10
-4
@@ -49,6 +49,8 @@ The AppOps-managed runtime candidate is defined separately in `.itworx/deploymen
|
||||
|
||||
The scanner prefers `git ls-files` for a top-level worktree so ignored/untracked build artifacts do not distort results. An empty Git worktree remains empty rather than silently falling back. Non-Git directories use a filesystem traversal that excludes common generated/vendor directories and applies bounded root/nested `.gitignore` rules. Text files larger than the configured internal read threshold are not parsed as source, and binary files are skipped.
|
||||
|
||||
The scanner processes inventory in one streaming pass and holds the full text of only one file at a time. It retains small derived values per file (LOC, debt counts, unresolved import references) plus the texts of the few CI workflow, `package.json` and `pyproject.toml` files needed for test signals, so peak scan memory no longer grows with the total text budget. Each Python file is parsed once for both function heuristics and imports. `text_bytes_scanned` counts the bytes actually read from disk.
|
||||
|
||||
Every report includes `ScanCoverage`: bounded file, per-file-text, total-text, finding and credential-finding limits plus the observed skipped counts. A limit produces a deterministic partial result and an analysis-coverage warning rather than an unqualified completeness claim or an artificial blocker.
|
||||
|
||||
## Signals
|
||||
@@ -69,7 +71,7 @@ Each finding is enriched by a backend rule definition with confidence, why-it-ma
|
||||
|
||||
## Architecture graph
|
||||
|
||||
Python imports are parsed using the standard-library AST. Local JavaScript and TypeScript relative imports, bounded `baseUrl`/`paths` aliases from `tsconfig.json`/`jsconfig.json`, and simple local workspace package imports are resolved against known source files. Only local edges are retained; this remains heuristic source analysis, not a TypeScript compiler. The displayed graph is bounded to 160 nodes and 600 edges to keep large repositories responsive; the response explicitly indicates truncation, reports total discovered nodes/edges and distinguishes visible coupling from total coupling used in hotspot ranking. Extensionless, index and `.js`-specifier-to-TypeScript resolution are supported.
|
||||
Python imports are parsed using the standard-library AST. Local JavaScript and TypeScript relative imports, bounded `baseUrl`/`paths` aliases from `tsconfig.json`/`jsconfig.json`, and simple local workspace package imports are resolved against known source files. Import targets are resolved lexically against the known, already safety-checked file set rather than through filesystem resolution; a specifier escaping the scan root never yields an edge. Only local edges are retained; this remains heuristic source analysis, not a TypeScript compiler. The displayed graph is bounded to 160 nodes and 600 edges to keep large repositories responsive; the response explicitly indicates truncation, reports total discovered nodes/edges and distinguishes visible coupling from total coupling used in hotspot ranking. Extensionless, index and `.js`-specifier-to-TypeScript resolution are supported.
|
||||
|
||||
## Hotspot model
|
||||
|
||||
@@ -81,7 +83,7 @@ Category scores start at 100 and are reduced only by findings with an explicit d
|
||||
|
||||
## History and comparison
|
||||
|
||||
Every report is serialized as a snapshot in SQLite using WAL. Lightweight scan summary columns (readiness, counts, primary signal, deltas and triage priority) drive catalog CTE queries without deserializing full reports. Comparison calculates metric/category deltas and added/resolved findings against the chronological previous scan of the same repository identity. Findings carry stable semantic fingerprints so line movement does not appear as a new risk. Trend endpoints return bounded chronological snapshots for UI visualization. By default, up to 200 snapshots are retained per repository, the latest 30 remain detailed, older retained payloads compact to summary evidence, and snapshots older than 365 days are removed. v1 path history is backfilled into a local catalog identity and is never deleted solely for migration.
|
||||
Every report is serialized as a snapshot in SQLite using WAL. Lightweight scan summary columns (readiness, counts, primary signal, deltas and triage priority) drive catalog CTE queries without deserializing full reports. Comparison calculates metric/category deltas and added/resolved findings against the chronological previous scan of the same repository identity. Findings carry stable semantic fingerprints so line movement does not appear as a new risk. Trend and scan-history endpoints return bounded chronological snapshots computed in SQLite from summary columns and JSON extraction, without deserializing full reports; a compacted snapshot reports its retained active-finding summary counts instead of a misleading zero. By default, up to 200 snapshots are retained per repository, the latest 30 remain detailed, older retained payloads compact to summary evidence, and snapshots older than 365 days are removed. v1 path history is backfilled into a local catalog identity and is never deleted solely for migration.
|
||||
|
||||
## Reporting
|
||||
|
||||
@@ -89,7 +91,11 @@ RepoPulse can export the complete JSON report or a standalone HTML report. HTML
|
||||
|
||||
## Job state and failure isolation
|
||||
|
||||
The persisted queue records repository, trigger, requested/actual revision, phase, progress, attempt, timestamps, resulting scan and sanitized error information. Claims use `BEGIN IMMEDIATE` plus a status guard, so two processes cannot own the same queued job. They move through `queued → acquiring_source → scanning → analyzing → persisting → completed` or an isolated `failed` outcome. A heartbeat renews each active lease; only a crashed worker’s expired lease returns work to `queued` for recovery. Older unstarted revisions become `superseded`; explicit cancellation states remain representable. One repository failure does not terminate the worker loop or affect other queued repositories.
|
||||
The persisted queue records repository, trigger, requested/actual revision, phase, progress, attempt, timestamps, resulting scan and sanitized error information. Claims use `BEGIN IMMEDIATE` plus a status guard, so two processes cannot own the same queued job. They move through `queued → acquiring_source → scanning → analyzing → persisting → completed` or an isolated `failed` outcome. The idle scheduler writes its liveness heartbeat at most every `REPOPULSE_WORKER_HEARTBEAT_SECONDS` (default 5) instead of on every poll; claims still re-check maintenance inside their own transaction. An idle worker backs off exponentially from the 0.5 s poll to `REPOPULSE_WORKER_IDLE_POLL_SECONDS` (default 5), and in-process enqueues (API, webhook) wake it immediately. A claim first runs a read-only probe for queued work or expired leases, so an idle queue never takes the SQLite write lock. After each job the worker runs a garbage collection and, on glibc, `malloc_trim` so the long-running process returns a large scan's memory to the OS. A heartbeat renews each active lease; only a crashed worker’s expired lease returns work to `queued` for recovery. Older unstarted revisions become `superseded`; explicit cancellation states remain representable. One repository failure does not terminate the worker loop or affect other queued repositories.
|
||||
|
||||
## Child processes
|
||||
|
||||
Every Git subprocess runs in its own process group with a hard timeout; on timeout the whole group is killed and reaped, so no descendant (for example `git-remote-https` or `index-pack`) is orphaned. Git automatic housekeeping is forced to the foreground (`gc.autoDetach=false`, `maintenance.autoDetach=false`) so it stays within that bound. The container images run `tini` as PID 1, which reaps any remaining orphan; previously uvicorn ran as PID 1 and accumulated zombie `git` processes.
|
||||
|
||||
## Operator maintenance mode
|
||||
|
||||
@@ -97,4 +103,4 @@ The `app_state` key `maintenance_mode` durably records whether operator maintena
|
||||
|
||||
## Operational configuration
|
||||
|
||||
`REPOPULSE_GITEA_URL` and `REPOPULSE_GITEA_TOKEN` enable the provider. `REPOPULSE_GITEA_SYNC_INTERVAL_SECONDS`, `REPOPULSE_GITEA_BRANCH_WORKERS`, `REPOPULSE_MAX_QUEUED_SCANS`, `REPOPULSE_JOB_LEASE_SECONDS`, scan-budget and retention variables, and `REPOPULSE_REPOSITORY_CACHE` bound reconciliation, queue depth, active leases, analysis and cache placement. The durable Gitea runner deliberately has one worker; `/api/v1/meta` exposes requested versus actual capacity. `REPOPULSE_GITEA_WEBHOOK_SECRET` enables signed default-branch push intake; non-push, non-default-branch, invalid and branch-deletion events are acknowledged without scheduling work. `REPOPULSE_OPERATOR_TOKEN` protects mutating endpoints when an authenticated proxy perimeter is not the selected boundary. `/api/v1/meta` exposes repository state counts, queue depth, storage/retention metrics, recovery status, worker heartbeat, connector URL (without token) and last sync result.
|
||||
`REPOPULSE_GITEA_URL` and `REPOPULSE_GITEA_TOKEN` enable the provider. `REPOPULSE_GITEA_SYNC_INTERVAL_SECONDS`, `REPOPULSE_GITEA_BRANCH_WORKERS`, `REPOPULSE_MAX_QUEUED_SCANS`, `REPOPULSE_JOB_LEASE_SECONDS`, scan-budget and retention variables, and `REPOPULSE_REPOSITORY_CACHE` bound reconciliation, queue depth, active leases, analysis and cache placement. The durable Gitea runner deliberately has one worker; `/api/v1/meta` exposes requested versus actual capacity. `REPOPULSE_GITEA_WEBHOOK_SECRET` enables signed default-branch push intake; non-push, non-default-branch, invalid and branch-deletion events are acknowledged without scheduling work. `REPOPULSE_OPERATOR_TOKEN` protects mutating endpoints when an authenticated proxy perimeter is not the selected boundary. `/api/v1/meta` exposes repository state counts over the complete catalog (no silent cap), queue depth, storage/retention metrics (the mirror-cache walk runs in the background at most every 5 minutes, so the endpoint never waits on it after the first call), recovery status, worker heartbeat, connector URL (without token) and last sync result.
|
||||
Reference in new issue
Block a user