> ## Documentation Index
> Fetch the complete documentation index at: https://docs.findmydata.io/llms.txt
> Use this file to discover all available pages before exploring further.

# ADR 0007 scan pipeline and stage caching

# ADR-0007: Durable scan pipeline and stage cache keys

**Status:** Accepted · 2026-07-14

## Decision

Job hierarchy: `scan_campaigns → scan_partitions → work_units → stage_attempts`,
with immutable requested parameters, mutable checkpoint state, leases with
heartbeats, visibility timeouts, retry classes (`transient|throttled|auth|
unsupported|corrupt|policy_denied|permanent`), and dead-letter rows with safe
diagnostics.

Pipeline stages (versioned independently):

`observe → metadata → permissions → labels → content(hash+extract) → features →
inference → project(graph/search) → risk → review`

**Stage cache key** = `(tenantId, assetId, sourceVersionId, stageName,
stageImplVersion, configVersion, inputArtifactHashes)`. A stage first checks for a
valid cached artifact under its key and records `cache_hit` explicitly, so tests can
assert *no content fetch/hash/extraction occurred* for:

* unchanged content + new model release → features reused, inference re-run;
* permission-only change → content/features reused, exposure/risk recomputed;
* rename/move → content features reused, location/context recomputed.

Pause stops new leasing and checkpoints at stage boundaries; resume re-leases from
checkpoints; cancel marks drain semantics explicitly. Idempotency: re-running a
stage with the same logical key is a no-op (unique index enforced).

## Consequences

* The invalidation matrix from the reference architecture is encoded in the
  per-stage cache-key composition in `scan/pipeline.ts` (observe/permissions/
  labels key on `sourceVersion`; content/features on `contentVersion` +
  algorithm versions + tenant config version; inference additionally on the
  ensemble release), exercised by the "incremental efficiency (invalidation
  matrix)" tests in `scan/scan.test.ts`.
* Progress reporting uses known/estimated work counts, never fabricated percentages.