Skip to main content

​
ADR-0007: Durable scan pipeline and stage cache keys

Status: Accepted · 2026-07-14

​
Decision

Job hierarchy: scan_campaigns → scan_partitions → work_units → stage_attempts, with immutable requested parameters, mutable checkpoint state, leases with heartbeats, visibility timeouts, retry classes (transient|throttled|auth| unsupported|corrupt|policy_denied|permanent), and dead-letter rows with safe diagnostics. Pipeline stages (versioned independently): observe → metadata → permissions → labels → content(hash+extract) → features → inference → project(graph/search) → risk → review Stage cache key = (tenantId, assetId, sourceVersionId, stageName, stageImplVersion, configVersion, inputArtifactHashes). A stage first checks for a valid cached artifact under its key and records cache_hit explicitly, so tests can assert no content fetch/hash/extraction occurred for:
  • unchanged content + new model release → features reused, inference re-run;
  • permission-only change → content/features reused, exposure/risk recomputed;
  • rename/move → content features reused, location/context recomputed.
Pause stops new leasing and checkpoints at stage boundaries; resume re-leases from checkpoints; cancel marks drain semantics explicitly. Idempotency: re-running a stage with the same logical key is a no-op (unique index enforced).

​
Consequences

  • The invalidation matrix from the reference architecture is encoded in the per-stage cache-key composition in scan/pipeline.ts (observe/permissions/ labels key on sourceVersion; content/features on contentVersion + algorithm versions + tenant config version; inference additionally on the ensemble release), exercised by the “incremental efficiency (invalidation matrix)” tests in scan/scan.test.ts.
  • Progress reporting uses known/estimated work counts, never fabricated percentages.