> ## Documentation Index
> Fetch the complete documentation index at: https://docs.findmydata.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Operations guide

# Operations guide

This guide is for platform operators and SREs running Find My Data (release 26.7.15.0, Bun 1.3.14, 196 tests passing). It covers the process topology, the durable work queue, monitoring endpoints and metric semantics, change-notification operations, scan operations, data/backup operations, scale triggers, and log management. Statements about live (Microsoft Graph) behavior are marked with their actual validation status: the read-only Graph path was validated against the live CDX tenant ([ADR-0013](adr/ADR-0013-live-graph-connector.md), [CDX runbook](cdx-test-runbook.md)); several capabilities are built and mock-validated but feature-gated for live use — this doc never describes a gated capability as simply working live.

Related: [architecture](architecture.md), [capacity model](capacity-model.md), [threat model](threat-model.md), [permissions manifest](permissions-manifest.md), [roadmap](roadmap.md).

## 1. Process topology

| Process                      | Start command                                               | What it runs                                                                                                                                          |
| ---------------------------- | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| API + embedded worker        | `bun run dev` (dev) / `bun run --cwd packages/server start` | HTTP API, plus the durable worker loop embedded in-process (`startWorkerLoop` in `packages/server/src/main.ts`), owner id `embedded-<pid>`            |
| Standalone worker (optional) | `bun run worker`                                            | Same worker code path (`packages/server/src/worker-main.ts`), owner id `worker-<pid>`, against the same SQLite queue. Stops cleanly on SIGINT/SIGTERM |

Both processes open the same database file, so standalone workers must run on the same host/volume as the API (single-node SQLite; see §7 for when this stops being appropriate).

**Worker loop mechanics** (`packages/server/src/scan/worker.ts`): the loop ticks every 500 ms by default, drains up to 20 work units per tick, heartbeats its scanner registration every tick, and every 20 ticks (\~10 s) runs the fleet sweep plus, in the API process only, the notification maintenance sweep (subscription renewal, reconcile stamping, expiry). Note: `worker-main.ts` does **not** pass a `periodicSweep`, so change-notification subscription renewal runs only while the API process is up.

### Scanner fleet semantics

Every worker loop registers itself as a scanner (`packages/server/src/scan/fleet.ts`): workload id, display name, host (pid), advertised capabilities (default `["m365"]`), and version (`scanner-0.1.0`). Registration is an upsert — restarting with the same id re-activates the row.

Health is derived from heartbeat age, using constants in `fleet.ts`:

| Status     | Condition                                                 | Effect                                                                                                                                                                             |
| ---------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `active`   | heartbeat within 30 s (`HEARTBEAT_STALE_MS = 30_000`)     | leases work normally                                                                                                                                                               |
| `stale`    | no heartbeat for > 30 s                                   | flagged in fleet view; still owns its leases                                                                                                                                       |
| `offline`  | no heartbeat for > 90 s (`HEARTBEAT_OFFLINE_MS = 90_000`) | **work takeover**: its `leased` work units are reset to `pending` (lease owner cleared) so healthy scanners pick them up immediately, without waiting for each lease's 60 s expiry |
| `draining` | set via `drainScanner`                                    | heartbeats no longer re-activate it                                                                                                                                                |

The sweep (`sweepFleet`) runs from every worker's loop (\~every 10 s), inside a transaction. Duplicate-processing safety does not depend on the sweep: the queue's own lease expiry (§2) recovers abandoned work, and all handlers are idempotent. Fleet membership, per-scanner processed/failed counts, and current lease counts are visible in `GET /api/ops/overview` (`fleet` array).

Honesty note: the fleet is built and tested for in-process and same-host standalone workers. Container/K8s packaging and signed work envelopes are roadmap items ([roadmap](roadmap.md)), not built.

## 2. The durable queue

`packages/server/src/scan/queue.ts` implements a durable at-least-once queue over SQLite (WorkQueue role, [ADR-0002](adr/ADR-0002-modular-monolith-and-worker.md), [ADR-0003](adr/ADR-0003-persistence-single-sqlite-behind-role-interfaces.md)). Consumers are idempotent via logical keys and stage-cache keys ([ADR-0007](adr/ADR-0007-scan-pipeline-and-stage-caching.md)).

### Work unit lifecycle

Work unit kinds: `inventory_page`, `delta_run`, `process_asset`, `rescore_asset`.

| Status      | Meaning                                                                                                                                                                                                                                                                                                                                                           |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pending`   | enqueued, runnable once `run_after` passes                                                                                                                                                                                                                                                                                                                        |
| `leased`    | held by a worker; lease is 60 s (`LEASE_MS = 60_000`). The queue exposes a lease-extension heartbeat, but current handlers do not call it mid-unit — a unit running past its lease becomes re-leasable (idempotent handlers make that safe). A unit whose lease expired is re-leasable by any worker (abandoned-worker recovery); each lease increments `attempt` |
| `completed` | handler succeeded                                                                                                                                                                                                                                                                                                                                                 |
| `dead`      | terminal failure; a `dead_letters` row was written                                                                                                                                                                                                                                                                                                                |
| `cancelled` | campaign was cancelled while the unit was still pending                                                                                                                                                                                                                                                                                                           |
| `failed`    | reserved in the schema (`migrations/0002_source_and_jobs.sql`) but **never written by current code** — failures either return to `pending` (retry) or go `dead`                                                                                                                                                                                                   |

Enqueue is idempotent on `(tenant_id, campaign_id, logical_key)`; duplicate enqueues are no-ops. Default `max_attempts` is 4.

### Retry classes

Failure handling (`fail()` in `queue.ts`) is driven by the retry class (`RETRY_CLASSES` in `packages/shared/src/domain-enums.ts`): `transient`, `throttled`, `auth`, `unsupported`, `corrupt`, `policy_denied`, `permanent`.

| Class                                                  | Behavior                                                              |
| ------------------------------------------------------ | --------------------------------------------------------------------- |
| `unsupported`, `corrupt`, `policy_denied`, `permanent` | terminal → `dead` + dead letter immediately                           |
| `throttled`                                            | retry with linear backoff: `500 ms × 8 × attempt` (4 s, 8 s, 12 s, …) |
| `transient`, `auth`, anything else                     | retry with exponential backoff: `500 ms × 2^(attempt-1)`              |
| any class at `attempt ≥ max_attempts`                  | terminal → `dead`                                                     |

The worker's error classifier (`classifyError` in `worker.ts`) currently emits only `transient`, `permanent`, and `unsupported` from thrown handler errors; the other classes are part of the queue contract and used by the schema/tests. A worker that lost its lease cannot dead-letter the unit (the terminal update is guarded by `lease_owner` + `status='leased'`), so takeover never double-records.

### Dead letters: inspect and replay

Dead letters carry only the retry class and a `safe_diagnostic` (charset-sanitized, length-bounded — never file names, paths, or content).

* **Inspect (tenant-wide):** `GET /api/ops/overview` → `deadLetters` (most recent 50). Requires the `operations.read` capability.
* **Inspect (per campaign):** `GET /api/scans/:id` → `deadLetters` plus per-stage outcome counts.
* **Replay:** there is **no one-click replay endpoint** (not built). The operational replay path is to start a new campaign for the affected scope (`POST /api/scans` with `{"scopeId": "...", "kind": "inventory" | "delta"}`): logical keys are per-campaign so the work re-enqueues, and stage-cache hits make re-processing of unchanged assets cheap (no content re-fetch). The dead unit itself stays `dead` as a permanent record. If the failure was `unsupported`/`corrupt`, replaying will dead-letter again until the underlying cause (e.g. extractor support) changes.

### Campaign pause / resume / cancel

`POST /api/scans/:id/pause|resume|cancel` (capabilities `scan.job.pause` / `scan.job.resume` / `scan.job.cancel`; see `packages/server/src/scan/routes.ts` and `campaigns.ts`):

* **pause** (`running → paused`): leasing joins on campaign status, so paused campaigns simply stop handing out work; in-flight leased units run to completion untouched.
* **resume** (`paused → running`): re-enables leasing. If everything finished while paused, the campaign is completed immediately instead of stranding in `running`.
* **cancel** (`running|paused|pending → cancelled`): marks remaining `pending` units `cancelled`; leased in-flight units finish their current attempt.

All transitions are audited (category `scan`).

## 3. Monitoring

### `/api/health` and `/api/ready`

`GET /api/health` returns `{"status":"ok"}` (liveness only). `GET /api/ready` (`packages/server/src/routes/health.ts`) checks the DB (`503` + `status: "degraded"` on failure) and reports **honest integration modes**. A capability is `live` only when its credential *references resolve to actual values in the environment* — a configured-but-unresolvable secret reference is `blocked`, never `live`, and never a `disabled` that implies an operator turned it off.

| Mode       | Meaning                                                                    |
| ---------- | -------------------------------------------------------------------------- |
| `mock`     | deterministic in-process implementation; fully exercised path              |
| `live`     | feature enabled **and** all required credentials resolve                   |
| `blocked`  | wanted live but a prerequisite is missing (flag, secret value, public URL) |
| `disabled` | feature flag is off                                                        |

Integration keys and their rules:

| Key                                      | Rule                                                                                                                                                                                       |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `connector`                              | `mock` when `FMD_CONNECTOR_MODE=mock`; else `live` iff Graph tenant/client id + resolved client secret, else `blocked`                                                                     |
| `aiProvider`                             | `mock`, or `live` iff the Azure OpenAI (endpoint + deployment + resolved key) or Anthropic (resolved key) config resolves, else `blocked`                                                  |
| `entraSignIn`                            | `blocked` when `FMD_AUTH_MODE=entra` (the OIDC code flow is a documented seam, **not implemented**), `disabled` otherwise — never `live` in this release                                   |
| `purviewLabelRead` / `purviewLabelWrite` | `live` iff the feature flag is on and Graph (and for write, the separate remediation identity) resolves; otherwise `blocked` (these two report `blocked` even when the flag is simply off) |
| `changeNotifications`                    | `mock` in mock connector mode; `live` requires the flag, resolved Graph credentials, **and** `FMD_WEBHOOK_PUBLIC_URL`; `disabled` without the flag, else `blocked`                         |
| `teamsConnector` / `exchangeConnector`   | `mock` in mock mode; else `disabled` without the flag, `live`/`blocked` by Graph credential resolution                                                                                     |

### `/api/ops/overview`

Requires `operations.read` (`packages/server/src/routes/read-models.ts`). Payload:

* `queue` — work-unit counts by status for the caller's tenant.
* `deadLetters` — most recent 50 dead letters (id, retry class, safe diagnostic, timestamp).
* `recentLogs` — last 100 entries of the process log ring buffer, filtered to the caller's tenant (or tenant-agnostic system events).
* `fleet` — deployment-level scanner fleet: status, heartbeat age, processed/failed counts, current leases.

### `/api/ops/metrics`

Requires `operations.read`. Payload:

* `counters` — monotonic in-process counters (reset on process restart).
* `histograms` — per-series `{name, count, p50, p95, max}` over the last 2,000 samples (bounded ring; max 500 series — `packages/server/src/kernel/metrics.ts`).
* `derived` — computed from the DB at request time (these reflect **all** processes): `queuePending`, `queueLeased`, `deadLetters`, `oldestPendingAgeMs`, `oldestDeltaCursorAgeMs` (null when no valid delta cursor exists), `contentStates` (current asset versions by content state).

**Per-process caveat:** counters and histograms are a process-global in-memory registry. The API serves *its own* process's counters — work executed by a standalone `bun run worker` process increments that process's registry, which is not merged into the API's `/api/ops/metrics`. Use the DB-derived gauges and `/api/ops/overview` fleet counts for cross-process truth.

### Metric families

| Family                                                                                                           | Type      | Meaning                                                                                                                                                                                                                                        | Healthy                                                                                | Concerning                                                                                         |
| ---------------------------------------------------------------------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `stage_latency_ms:<stage>`                                                                                       | histogram | Pipeline stage wall time; stages: `observe`, `permissions`, `labels`, `content`, `features` (the `inference` stage records to `stage_attempts` — visible as per-campaign stage outcomes in `GET /api/scans/:id` — not to the metrics registry) | ms-scale p50 for cache-hit-heavy stages; `content` dominated by connector fetch        | p95 growing over time (DB contention, provider slowness)                                           |
| `stage_outcome:<stage>:<outcome>`                                                                                | counter   | Outcomes `computed` and `cache_hit` (the schema also reserves `skipped`/`failed`, not emitted by current pipeline code)                                                                                                                        | high `cache_hit` share on delta/rename-only re-scans (the invalidation matrix working) | `computed` for `content` on changes that should be metadata-only                                   |
| `extraction:<format>:<state>`                                                                                    | counter   | Text extraction results by sniffed format; states `available`, `partial`, `unsupported`, `failed`                                                                                                                                              | `available` dominating for supported formats                                           | growing `unsupported`/`failed` for formats you expect to support                                   |
| `graph_throttled:429` / `graph_throttled:503`                                                                    | counter   | Live Graph throttling responses; the client honors `Retry-After` (capped 30 s, 5 attempts)                                                                                                                                                     | zero in mock mode; occasional in live scans                                            | sustained growth — reduce concurrency/scan pacing; pairs with `throttled_retries_exhausted` errors |
| `work_completed:<kind>`                                                                                          | counter   | Completed work units by kind                                                                                                                                                                                                                   | steadily increasing during campaigns                                                   | flat while `queuePending > 0` → workers not running/leasing                                        |
| `work_failed:<retryClass>`                                                                                       | counter   | Failed attempts by retry class                                                                                                                                                                                                                 | small `transient` counts                                                               | any `permanent`; `transient` growing faster than completions                                       |
| `notification_receive_lag_ms`                                                                                    | histogram | Source event time → webhook receipt                                                                                                                                                                                                            | seconds                                                                                | minutes → provider delay or receiver downtime (gaps also arrive as `missed` lifecycle events)      |
| `notification_reconcile_lag_ms`                                                                                  | histogram | Notification receipt → reconcile delta campaign completed (end-to-end freshness)                                                                                                                                                               | seconds to low minutes                                                                 | sustained growth → queue backlog or paused campaigns                                               |
| `change_notification:<outcome>`                                                                                  | counter   | `reconcile_enqueued`, `reconcile_skipped`, `deduplicated`, `rejected:unknown_subscription`, `rejected:client_state_mismatch`                                                                                                                   | mostly enqueued/deduplicated                                                           | any `rejected:client_state_mismatch` — probing or misconfiguration; investigate                    |
| `change_subscription_created:<mode>` / `change_subscription_renewed:<mode>` / `change_subscription_renew_failed` | counter   | Subscription lifecycle activity by mode (`mock`/`live`)                                                                                                                                                                                        | periodic renewals                                                                      | any `renew_failed` — the subscription flips to `error` and stops receiving hints                   |
| `change_lifecycle:<event>`                                                                                       | counter   | Graph lifecycle events (`reauthorizationRequired`, `missed`, `subscriptionRemoved`)                                                                                                                                                            | rare                                                                                   | frequent `missed` → notification gaps; each forces a delta reconcile                               |

Derived-gauge rules of thumb: `oldestPendingAgeMs` should stay near zero when workers are healthy — growth with `queueLeased ≈ 0` means no worker is leasing (process down, or all campaigns paused). `oldestDeltaCursorAgeMs` is the age of the least-recently-updated valid delta cursor — i.e. how stale your *stalest* covered scope is; a delta run on that scope resets it. `deadLetters` is cumulative — alert on increase, then inspect via `/api/ops/overview`.

## 4. Change-notification operations

Implementation: `packages/server/src/notifications/subscriptions.ts` and `notifications/routes.ts`. Design invariant: a notification is a *hint*, never trusted content — receipt triggers reconciliation through the existing delta feed.

**Status honesty:** the full path (subscription lifecycle, webhook receiver with clientState HMAC, dedup, delta reconcile, freshness metrics) is built and mock-validated; the dev simulator drove a real read-only delta reconcile against the live CDX tenant. Live Graph subscription *registration* is feature-gated (`FMD_FEATURE_CHANGE_NOTIFICATIONS=true` + `FMD_WEBHOOK_PUBLIC_URL`) and has not been exercised against Graph's webhook delivery; creation without those prerequisites fails with an explicit blocked error rather than pretending.

### Subscription lifecycle

Constants in `subscriptions.ts`:

* `SUBSCRIPTION_TTL_MS = 60 min` — subscriptions are created/renewed with a short 60-minute expiry.
* `RENEW_WHEN_WITHIN_MS = 20 min` — the renewal sweep renews any `active`/`expiring` subscription with less than 20 minutes remaining.
* `DEDUP_WINDOW_MS = 10 s` — repeat hints for the same subscription+changeType inside 10 s are recorded as `deduplicated` and do not enqueue another reconcile.

Subscription statuses: `active`, `expiring` (flagged for renewal, e.g. by a `reauthorizationRequired` lifecycle event), `expired` (renewal never rescued it — set by the expiry sweep), `error` (a renewal attempt failed; `last_error = 'renew_failed'`), `removed` (Graph reported `subscriptionRemoved`).

The renewal sweep (`renewDueSubscriptions`), reconcile stamping (`markReconciledNotifications`), and expiry sweep (`sweepExpiredSubscriptions`) run every \~10 s from the **API process's embedded worker only** (see §1). A manual trigger exists: `POST /api/admin/notifications/renew` (capability `connector.configure`). Live renewals PATCH Graph; mock renewals extend locally.

### Receiver and lifecycle webhooks

* `POST /api/webhooks/graph/notifications` — unauthenticated (Graph posts here). Handles the validation-token handshake, then per item verifies the per-subscription `clientState` HMAC in constant time before any side effect; always returns 202 fast. Valid notifications reconcile the scope by creating a delta campaign (outcome `reconcile_enqueued`), or record `reconcile_skipped` with a reason (`scan_already_active`, `no_delta_cursor`, `no_scope`) — both benign, the running/next scan observes the change.
* `POST /api/webhooks/graph/lifecycle` — same trust model. `reauthorizationRequired` → status `expiring` (renew sweep picks it up); `missed` → forces a delta reconcile (a gap in the stream); `subscriptionRemoved` → status `removed`.

### Operating it

* Subscribe a scope: `POST /api/admin/notifications/subscribe` `{"scopeId": "..."}` (capability `connector.configure`).
* Inspect: `GET /api/admin/notifications/subscriptions` and `GET /api/admin/notifications/recent` (capability `operations.read`) — per-notification outcome, source (`live`/`simulated`), and `reconcileLagMs`.
* Dev-only end-to-end drill without a public endpoint: `POST /api/dev/notifications/simulate` `{"scopeId": "..."}` (development env only).
* Freshness: watch `notification_receive_lag_ms` and `notification_reconcile_lag_ms` (§3).

## 5. Scan operations

### Campaign kinds

| Kind        | Trigger                                                                                                                                           | What it does                                                                                                                                                                                                            |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `inventory` | `POST /api/scans` `{"kind":"inventory"}`                                                                                                          | Paged full walk (`inventory_page` units, `page:0`, `page:1`, …); upserts containers, enqueues one `process_asset` unit per item; on the final page stores the delta cursor and sets scope `coverage_state = 'baseline'` |
| `delta`     | `POST /api/scans` `{"kind":"delta"}`, or automatically by a change notification                                                                   | Single `delta_run` unit against the stored delta cursor; requires a prior inventory (validation error otherwise); on success sets `coverage_state = 'delta'`                                                            |
| `rescore`   | model-release promote/rollback or the rescore endpoint (`POST /api/models/rescore`; implementation `packages/server/src/intelligence/rescore.ts`) | Tenant-wide, no scope: one `rescore_asset` unit per active asset                                                                                                                                                        |

Only one campaign per scope may be active (`pending`/`running`/`paused`) at a time — concurrent creation gets a conflict error, which is also how notification-driven reconciles dedupe against a running scan.

### Delta 410 resync

When the provider invalidates a delta token (live Graph returns HTTP 410; the connector maps it to `resync_required` — `packages/server/src/connectors/graph/connector.ts`), the worker (`handleDeltaRun` in `worker.ts`):

1. marks the delta cursor `resync_required` and the scope `coverage_state = 'stale'`;
2. enqueues a fresh baseline walk **inside the same campaign** (`inventory_page` with logical key `resync:page:0`);
3. logs `delta_resync_required` (warn).

Because content/feature stages key on `contentVersion`, unchanged assets resolve as stage-cache hits during the re-baseline — a resync costs a metadata re-walk, not duplicate content fetches or re-classification (tested).

### Cache invalidation matrix (summary)

Providers expose two version signals (`packages/server/src/connectors/types.ts`): `sourceVersion` (eTag equivalent — changes on *any* item change: rename, permissions, label, content) and `contentVersion` (cTag equivalent — changes *only* when file content changes). Stage-cache keys ([ADR-0007](adr/ADR-0007-scan-pipeline-and-stage-caching.md), `packages/server/src/scan/pipeline.ts`) implement the matrix:

| Stage                              | Keyed on                                                                                                                                          | Consequence                                                                                                                                                  |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `observe`, `permissions`, `labels` | `sourceVersion` (eTag)                                                                                                                            | any change re-observes metadata/permissions/labels                                                                                                           |
| `content`, `features`              | `contentVersion` (cTag) + extractor/fingerprint/keyphrase impl versions                                                                           | rename-, permission-, or label-only changes are cache hits: **no content fetch, hash, or extraction**; fingerprints are re-associated to the new version row |
| `inference`                        | `contentVersion` + active ensemble release + a context hash (permission set, label state, container, expected domain, approved-taxonomy versions) | a model release re-scores retained features with **zero source I/O**; permission/label-only changes also re-run inference (from retained features)           |

### Re-score without refetch

A `rescore` campaign bumps (or reuses) the ensemble release and enqueues `rescore_asset` units that run inference from retained features only — the rescore path has no connector dependency at all (structurally source-free; the acceptance test asserts zero content fetches). This was validated on live CDX data: re-scoring 174 assets took \~1.4 s vs \~151 s for the original content pass (see [CDX runbook](cdx-test-runbook.md)).

## 6. Data operations

### Storage layout

* **Database:** single SQLite file, `FMD_DB_PATH` (default `./data/fmd.db`, i.e. `packages/server/data/fmd.db` in dev). Opened with `PRAGMA journal_mode = WAL`, `foreign_keys = ON`, `busy_timeout = 5000` (`packages/server/src/kernel/db.ts`). WAL means you will also see `fmd.db-wal` and `fmd.db-shm` beside it.
* **Artifacts (object store):** `FMD_ARTIFACT_DIR` (default `./data/artifacts`), files at `<artifactDir>/<tenantId>/<artifactId>` with metadata (content hash, purpose flags, retention class) in the `artifacts` table (`packages/server/src/kernel/objectstore.ts`). The DB and the artifacts dir reference each other — treat them as one backup unit.

### Backup

There is **no automated backup tooling built** — backup is a file-level operation you schedule yourself:

1. Checkpoint the WAL so the main file is current: `sqlite3 packages/server/data/fmd.db "PRAGMA wal_checkpoint(TRUNCATE);"` (any SQLite client works).
2. Copy `fmd.db` (plus `-wal`/`-shm` if present) **and** the entire artifacts directory together.
3. For a strictly consistent point-in-time copy, quiesce writers first (stop the API and any standalone workers, or pause campaigns and wait for `queueLeased = 0`); with WAL, a copy under live writes risks tearing between the DB and the artifacts dir even when the DB copy itself is fine.

Restore = put both back and start the process; migrations are applied idempotently at boot. Dev reset: `rm -rf packages/server/data && bun run seed`.

### Projections

The search projection (`search_documents` + FTS) and graph projection (`graph_nodes`/`graph_edges`) are rebuildable by design from the system of record ([ADR-0003](adr/ADR-0003-persistence-single-sqlite-behind-role-interfaces.md); `packages/server/src/intelligence/projections.ts`). However, **no standalone rebuild CLI exists** in this release: projections are (re)written by the inference stage, so the practical regeneration path today is a tenant-wide `rescore` campaign (repopulates projection rows for all active assets, without source I/O). Treat a dedicated rebuild command as not built.

## 7. Scale triggers

Full analysis in the [capacity model](capacity-model.md) (preliminary — slice-scale measurements plus extrapolations; nothing beyond the loadgen harness is validated). Operationally:

* The single-writer SQLite deployment is comfortable at slice scale and plausible to roughly **1–5M assets on one strong node**. The first contention symptom is queue lease/heartbeat writes competing with observation inserts (both serialize on the one writer, `busy_timeout` 5 s).
* At 1M assets the data is only \~8–15 GB — bytes are not the constraint; the writer and bootstrap wall-clock are.
* 500M-asset numbers in the capacity model are explicitly unvalidated extrapolations.

Migration triggers per role interface (measure, don't guess — capacity model §6; each role moves independently behind its interface per [ADR-0003](adr/ADR-0003-persistence-single-sqlite-behind-role-interfaces.md)):

| Role                      | Trigger                                                                                    | Target                                                       |
| ------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------ |
| System of record          | writer contention between worker and API; \~1–5M assets; need online backup/replication/HA | Azure Database for PostgreSQL, partitioned by tenant + scope |
| Object/evidence artifacts | artifact bytes beyond tens of GB; lifecycle/envelope-encryption requirements               | Azure Blob + lifecycle policies                              |
| Search projection         | FTS5 rebuild exceeds maintenance windows; concurrent query load                            | Azure AI Search / OpenSearch (rebuildable, watermarked)      |
| Graph projection          | bounded-neighborhood p95 > 3 s despite indexes                                             | dedicated graph store only if traversal demands it           |
| Work queue                | lease/heartbeat contention; multiple worker **nodes**; campaigns beyond \~10⁶ units        | Azure Service Bus / Storage Queues                           |
| Fingerprint/vector index  | band-row growth (16 rows per fingerprinted asset); live embeddings                         | pgvector / dedicated ANN; pack band rows                     |

## 8. Log management

The logger (`packages/server/src/kernel/log.ts`) emits **structured JSON lines to stdout**, one object per event: `{ts, level, event, ...fields}`.

**Redaction guarantee (structural, not best-effort):** only allowlisted field keys are emitted (ids, enums, and operational fields such as `correlationId`, `tenantId`, `assetId`, `campaignId`, `workUnitId`, `stage`, `retryClass`, `httpStatus`, `durationMs`, `sizeBand`, …); string values must match an opaque-identifier charset (`^[A-Za-z0-9_.:/-]{1,120}$`) or they are replaced with `[redacted]`; unknown keys are dropped and counted in `_droppedFields`. Default logs therefore never contain names, UPNs/emails, file names, paths, URLs, titles, excerpts, prompts, tokens, hashes, or embeddings — and this is verified by an end-to-end redaction scan in the test suite. Exact sizes are logged as bands (`lt10k` … `gte10m`) where exactness is unneeded. The same allowlist discipline applies to metric names (`kernel/metrics.ts`).

**Correlation IDs:** every request gets one — taken from the `x-fmd-correlation` header if the caller supplies it, otherwise generated (`packages/server/src/kernel/http.ts`). It is attached to request/audit log entries, propagated into the auth context (and from there into audit events), and returned in error responses (`{"error": "internal_error", "correlationId": ...}`), so an operator can join a user-reported error to logs and audit without any content leaking between them.

**Ring buffer:** the last 500 entries are kept in memory per process for the Operations view; `GET /api/ops/overview` returns the last 100, filtered to the caller's tenant. There is no built-in log shipping — collect stdout with your platform's log pipeline (the Azure IaC in `infra/main.bicep` provisions Log Analytics for this).