Operations guide

This guide is for platform operators and SREs running Find My Data (release 26.7.15.0, Bun 1.3.14, 196 tests passing). It covers the process topology, the durable work queue, monitoring endpoints and metric semantics, change-notification operations, scan operations, data/backup operations, scale triggers, and log management. Statements about live (Microsoft Graph) behavior are marked with their actual validation status: the read-only Graph path was validated against the live CDX tenant (ADR-0013, CDX runbook); several capabilities are built and mock-validated but feature-gated for live use — this doc never describes a gated capability as simply working live. Related: architecture, capacity model, threat model, permissions manifest, roadmap.

1. Process topology

Process	Start command	What it runs
API + embedded worker	`bun run dev` (dev) / `bun run --cwd packages/server start`	HTTP API, plus the durable worker loop embedded in-process (`startWorkerLoop` in `packages/server/src/main.ts`), owner id `embedded-<pid>`
Standalone worker (optional)	`bun run worker`	Same worker code path (`packages/server/src/worker-main.ts`), owner id `worker-<pid>`, against the same SQLite queue. Stops cleanly on SIGINT/SIGTERM

Both processes open the same database file, so standalone workers must run on the same host/volume as the API (single-node SQLite; see §7 for when this stops being appropriate). Worker loop mechanics (packages/server/src/scan/worker.ts): the loop ticks every 500 ms by default, drains up to 20 work units per tick, heartbeats its scanner registration every tick, and every 20 ticks (~10 s) runs the fleet sweep plus, in the API process only, the notification maintenance sweep (subscription renewal, reconcile stamping, expiry). Note: worker-main.ts does not pass a periodicSweep, so change-notification subscription renewal runs only while the API process is up.

Scanner fleet semantics

Every worker loop registers itself as a scanner (packages/server/src/scan/fleet.ts): workload id, display name, host (pid), advertised capabilities (default ["m365"]), and version (scanner-0.1.0). Registration is an upsert — restarting with the same id re-activates the row. Health is derived from heartbeat age, using constants in fleet.ts:

Status	Condition	Effect
`active`	heartbeat within 30 s (`HEARTBEAT_STALE_MS = 30_000`)	leases work normally
`stale`	no heartbeat for > 30 s	flagged in fleet view; still owns its leases
`offline`	no heartbeat for > 90 s (`HEARTBEAT_OFFLINE_MS = 90_000`)	work takeover: its `leased` work units are reset to `pending` (lease owner cleared) so healthy scanners pick them up immediately, without waiting for each lease’s 60 s expiry
`draining`	set via `drainScanner`	heartbeats no longer re-activate it

The sweep (sweepFleet) runs from every worker’s loop (~every 10 s), inside a transaction. Duplicate-processing safety does not depend on the sweep: the queue’s own lease expiry (§2) recovers abandoned work, and all handlers are idempotent. Fleet membership, per-scanner processed/failed counts, and current lease counts are visible in GET /api/ops/overview (fleet array). Honesty note: the fleet is built and tested for in-process and same-host standalone workers. Container/K8s packaging and signed work envelopes are roadmap items (roadmap), not built.

2. The durable queue

packages/server/src/scan/queue.ts implements a durable at-least-once queue over SQLite (WorkQueue role, ADR-0002, ADR-0003). Consumers are idempotent via logical keys and stage-cache keys (ADR-0007).

Work unit lifecycle

Work unit kinds: inventory_page, delta_run, process_asset, rescore_asset.

Status	Meaning
`pending`	enqueued, runnable once `run_after` passes
`leased`	held by a worker; lease is 60 s (`LEASE_MS = 60_000`). The queue exposes a lease-extension heartbeat, but current handlers do not call it mid-unit — a unit running past its lease becomes re-leasable (idempotent handlers make that safe). A unit whose lease expired is re-leasable by any worker (abandoned-worker recovery); each lease increments `attempt`
`completed`	handler succeeded
`dead`	terminal failure; a `dead_letters` row was written
`cancelled`	campaign was cancelled while the unit was still pending
`failed`	reserved in the schema (`migrations/0002_source_and_jobs.sql`) but never written by current code — failures either return to `pending` (retry) or go `dead`

Enqueue is idempotent on (tenant_id, campaign_id, logical_key); duplicate enqueues are no-ops. Default max_attempts is 4.

Retry classes

Failure handling (fail() in queue.ts) is driven by the retry class (RETRY_CLASSES in packages/shared/src/domain-enums.ts): transient, throttled, auth, unsupported, corrupt, policy_denied, permanent.

Class	Behavior
`unsupported`, `corrupt`, `policy_denied`, `permanent`	terminal → `dead` + dead letter immediately
`throttled`	retry with linear backoff: `500 ms × 8 × attempt` (4 s, 8 s, 12 s, …)
`transient`, `auth`, anything else	retry with exponential backoff: `500 ms × 2^(attempt-1)`
any class at `attempt ≥ max_attempts`	terminal → `dead`

The worker’s error classifier (classifyError in worker.ts) currently emits only transient, permanent, and unsupported from thrown handler errors; the other classes are part of the queue contract and used by the schema/tests. A worker that lost its lease cannot dead-letter the unit (the terminal update is guarded by lease_owner + status='leased'), so takeover never double-records.

Dead letters: inspect and replay

Dead letters carry only the retry class and a safe_diagnostic (charset-sanitized, length-bounded — never file names, paths, or content).

Inspect (tenant-wide): GET /api/ops/overview → deadLetters (most recent 50). Requires the operations.read capability.
Inspect (per campaign): GET /api/scans/:id → deadLetters plus per-stage outcome counts.
Replay: there is no one-click replay endpoint (not built). The operational replay path is to start a new campaign for the affected scope (POST /api/scans with {"scopeId": "...", "kind": "inventory" | "delta"}): logical keys are per-campaign so the work re-enqueues, and stage-cache hits make re-processing of unchanged assets cheap (no content re-fetch). The dead unit itself stays dead as a permanent record. If the failure was unsupported/corrupt, replaying will dead-letter again until the underlying cause (e.g. extractor support) changes.

Campaign pause / resume / cancel

POST /api/scans/:id/pause|resume|cancel (capabilities scan.job.pause / scan.job.resume / scan.job.cancel; see packages/server/src/scan/routes.ts and campaigns.ts):

pause (running → paused): leasing joins on campaign status, so paused campaigns simply stop handing out work; in-flight leased units run to completion untouched.
resume (paused → running): re-enables leasing. If everything finished while paused, the campaign is completed immediately instead of stranding in running.
cancel (running|paused|pending → cancelled): marks remaining pending units cancelled; leased in-flight units finish their current attempt.

All transitions are audited (category scan).

3. Monitoring

`/api/health` and `/api/ready`

GET /api/health returns {"status":"ok"} (liveness only). GET /api/ready (packages/server/src/routes/health.ts) checks the DB (503 + status: "degraded" on failure) and reports honest integration modes. A capability is live only when its credential references resolve to actual values in the environment — a configured-but-unresolvable secret reference is blocked, never live, and never a disabled that implies an operator turned it off.

Mode	Meaning
`mock`	deterministic in-process implementation; fully exercised path
`live`	feature enabled and all required credentials resolve
`blocked`	wanted live but a prerequisite is missing (flag, secret value, public URL)
`disabled`	feature flag is off

Integration keys and their rules:

Key	Rule
`connector`	`mock` when `FMD_CONNECTOR_MODE=mock`; else `live` iff Graph tenant/client id + resolved client secret, else `blocked`
`aiProvider`	`mock`, or `live` iff the Azure OpenAI (endpoint + deployment + resolved key) or Anthropic (resolved key) config resolves, else `blocked`
`entraSignIn`	`blocked` when `FMD_AUTH_MODE=entra` (the OIDC code flow is a documented seam, not implemented), `disabled` otherwise — never `live` in this release
`purviewLabelRead` / `purviewLabelWrite`	`live` iff the feature flag is on and Graph (and for write, the separate remediation identity) resolves; otherwise `blocked` (these two report `blocked` even when the flag is simply off)
`changeNotifications`	`mock` in mock connector mode; `live` requires the flag, resolved Graph credentials, and `FMD_WEBHOOK_PUBLIC_URL`; `disabled` without the flag, else `blocked`
`teamsConnector` / `exchangeConnector`	`mock` in mock mode; else `disabled` without the flag, `live`/`blocked` by Graph credential resolution

`/api/ops/overview`

Requires operations.read (packages/server/src/routes/read-models.ts). Payload:

queue — work-unit counts by status for the caller’s tenant.
deadLetters — most recent 50 dead letters (id, retry class, safe diagnostic, timestamp).
recentLogs — last 100 entries of the process log ring buffer, filtered to the caller’s tenant (or tenant-agnostic system events).
fleet — deployment-level scanner fleet: status, heartbeat age, processed/failed counts, current leases.

`/api/ops/metrics`

Requires operations.read. Payload:

counters — monotonic in-process counters (reset on process restart).
histograms — per-series {name, count, p50, p95, max} over the last 2,000 samples (bounded ring; max 500 series — packages/server/src/kernel/metrics.ts).
derived — computed from the DB at request time (these reflect all processes): queuePending, queueLeased, deadLetters, oldestPendingAgeMs, oldestDeltaCursorAgeMs (null when no valid delta cursor exists), contentStates (current asset versions by content state).

Per-process caveat: counters and histograms are a process-global in-memory registry. The API serves its own process’s counters — work executed by a standalone bun run worker process increments that process’s registry, which is not merged into the API’s /api/ops/metrics. Use the DB-derived gauges and /api/ops/overview fleet counts for cross-process truth.

Metric families

Family	Type	Meaning	Healthy	Concerning
`stage_latency_ms:<stage>`	histogram	Pipeline stage wall time; stages: `observe`, `permissions`, `labels`, `content`, `features` (the `inference` stage records to `stage_attempts` — visible as per-campaign stage outcomes in `GET /api/scans/:id` — not to the metrics registry)	ms-scale p50 for cache-hit-heavy stages; `content` dominated by connector fetch	p95 growing over time (DB contention, provider slowness)
`stage_outcome:<stage>:<outcome>`	counter	Outcomes `computed` and `cache_hit` (the schema also reserves `skipped`/`failed`, not emitted by current pipeline code)	high `cache_hit` share on delta/rename-only re-scans (the invalidation matrix working)	`computed` for `content` on changes that should be metadata-only
`extraction:<format>:<state>`	counter	Text extraction results by sniffed format; states `available`, `partial`, `unsupported`, `failed`	`available` dominating for supported formats	growing `unsupported`/`failed` for formats you expect to support
`graph_throttled:429` / `graph_throttled:503`	counter	Live Graph throttling responses; the client honors `Retry-After` (capped 30 s, 5 attempts)	zero in mock mode; occasional in live scans	sustained growth — reduce concurrency/scan pacing; pairs with `throttled_retries_exhausted` errors
`work_completed:<kind>`	counter	Completed work units by kind	steadily increasing during campaigns	flat while `queuePending > 0` → workers not running/leasing
`work_failed:<retryClass>`	counter	Failed attempts by retry class	small `transient` counts	any `permanent`; `transient` growing faster than completions
`notification_receive_lag_ms`	histogram	Source event time → webhook receipt	seconds	minutes → provider delay or receiver downtime (gaps also arrive as `missed` lifecycle events)
`notification_reconcile_lag_ms`	histogram	Notification receipt → reconcile delta campaign completed (end-to-end freshness)	seconds to low minutes	sustained growth → queue backlog or paused campaigns
`change_notification:<outcome>`	counter	`reconcile_enqueued`, `reconcile_skipped`, `deduplicated`, `rejected:unknown_subscription`, `rejected:client_state_mismatch`	mostly enqueued/deduplicated	any `rejected:client_state_mismatch` — probing or misconfiguration; investigate
`change_subscription_created:<mode>` / `change_subscription_renewed:<mode>` / `change_subscription_renew_failed`	counter	Subscription lifecycle activity by mode (`mock`/`live`)	periodic renewals	any `renew_failed` — the subscription flips to `error` and stops receiving hints
`change_lifecycle:<event>`	counter	Graph lifecycle events (`reauthorizationRequired`, `missed`, `subscriptionRemoved`)	rare	frequent `missed` → notification gaps; each forces a delta reconcile

Derived-gauge rules of thumb: oldestPendingAgeMs should stay near zero when workers are healthy — growth with queueLeased ≈ 0 means no worker is leasing (process down, or all campaigns paused). oldestDeltaCursorAgeMs is the age of the least-recently-updated valid delta cursor — i.e. how stale your stalest covered scope is; a delta run on that scope resets it. deadLetters is cumulative — alert on increase, then inspect via /api/ops/overview.

4. Change-notification operations

Implementation: packages/server/src/notifications/subscriptions.ts and notifications/routes.ts. Design invariant: a notification is a hint, never trusted content — receipt triggers reconciliation through the existing delta feed. Status honesty: the full path (subscription lifecycle, webhook receiver with clientState HMAC, dedup, delta reconcile, freshness metrics) is built and mock-validated; the dev simulator drove a real read-only delta reconcile against the live CDX tenant. Live Graph subscription registration is feature-gated (FMD_FEATURE_CHANGE_NOTIFICATIONS=true + FMD_WEBHOOK_PUBLIC_URL) and has not been exercised against Graph’s webhook delivery; creation without those prerequisites fails with an explicit blocked error rather than pretending.

Subscription lifecycle

Constants in subscriptions.ts:

SUBSCRIPTION_TTL_MS = 60 min — subscriptions are created/renewed with a short 60-minute expiry.
RENEW_WHEN_WITHIN_MS = 20 min — the renewal sweep renews any active/expiring subscription with less than 20 minutes remaining.
DEDUP_WINDOW_MS = 10 s — repeat hints for the same subscription+changeType inside 10 s are recorded as deduplicated and do not enqueue another reconcile.

Subscription statuses: active, expiring (flagged for renewal, e.g. by a reauthorizationRequired lifecycle event), expired (renewal never rescued it — set by the expiry sweep), error (a renewal attempt failed; last_error = 'renew_failed'), removed (Graph reported subscriptionRemoved). The renewal sweep (renewDueSubscriptions), reconcile stamping (markReconciledNotifications), and expiry sweep (sweepExpiredSubscriptions) run every ~10 s from the API process’s embedded worker only (see §1). A manual trigger exists: POST /api/admin/notifications/renew (capability connector.configure). Live renewals PATCH Graph; mock renewals extend locally.

Receiver and lifecycle webhooks

POST /api/webhooks/graph/notifications — unauthenticated (Graph posts here). Handles the validation-token handshake, then per item verifies the per-subscription clientState HMAC in constant time before any side effect; always returns 202 fast. Valid notifications reconcile the scope by creating a delta campaign (outcome reconcile_enqueued), or record reconcile_skipped with a reason (scan_already_active, no_delta_cursor, no_scope) — both benign, the running/next scan observes the change.
POST /api/webhooks/graph/lifecycle — same trust model. reauthorizationRequired → status expiring (renew sweep picks it up); missed → forces a delta reconcile (a gap in the stream); subscriptionRemoved → status removed.

Operating it

Subscribe a scope: POST /api/admin/notifications/subscribe {"scopeId": "..."} (capability connector.configure).
Inspect: GET /api/admin/notifications/subscriptions and GET /api/admin/notifications/recent (capability operations.read) — per-notification outcome, source (live/simulated), and reconcileLagMs.
Dev-only end-to-end drill without a public endpoint: POST /api/dev/notifications/simulate {"scopeId": "..."} (development env only).
Freshness: watch notification_receive_lag_ms and notification_reconcile_lag_ms (§3).

5. Scan operations

Campaign kinds

Kind	Trigger	What it does
`inventory`	`POST /api/scans` `{"kind":"inventory"}`	Paged full walk (`inventory_page` units, `page:0`, `page:1`, …); upserts containers, enqueues one `process_asset` unit per item; on the final page stores the delta cursor and sets scope `coverage_state = 'baseline'`
`delta`	`POST /api/scans` `{"kind":"delta"}`, or automatically by a change notification	Single `delta_run` unit against the stored delta cursor; requires a prior inventory (validation error otherwise); on success sets `coverage_state = 'delta'`
`rescore`	model-release promote/rollback or the rescore endpoint (`POST /api/models/rescore`; implementation `packages/server/src/intelligence/rescore.ts`)	Tenant-wide, no scope: one `rescore_asset` unit per active asset

Only one campaign per scope may be active (pending/running/paused) at a time — concurrent creation gets a conflict error, which is also how notification-driven reconciles dedupe against a running scan.

Delta 410 resync

When the provider invalidates a delta token (live Graph returns HTTP 410; the connector maps it to resync_required — packages/server/src/connectors/graph/connector.ts), the worker (handleDeltaRun in worker.ts):

marks the delta cursor resync_required and the scope coverage_state = 'stale';
enqueues a fresh baseline walk inside the same campaign (inventory_page with logical key resync:page:0);
logs delta_resync_required (warn).

Because content/feature stages key on contentVersion, unchanged assets resolve as stage-cache hits during the re-baseline — a resync costs a metadata re-walk, not duplicate content fetches or re-classification (tested).

Cache invalidation matrix (summary)

Providers expose two version signals (packages/server/src/connectors/types.ts): sourceVersion (eTag equivalent — changes on any item change: rename, permissions, label, content) and contentVersion (cTag equivalent — changes only when file content changes). Stage-cache keys (ADR-0007, packages/server/src/scan/pipeline.ts) implement the matrix:

Stage	Keyed on	Consequence
`observe`, `permissions`, `labels`	`sourceVersion` (eTag)	any change re-observes metadata/permissions/labels
`content`, `features`	`contentVersion` (cTag) + extractor/fingerprint/keyphrase impl versions	rename-, permission-, or label-only changes are cache hits: no content fetch, hash, or extraction; fingerprints are re-associated to the new version row
`inference`	`contentVersion` + active ensemble release + a context hash (permission set, label state, container, expected domain, approved-taxonomy versions)	a model release re-scores retained features with zero source I/O; permission/label-only changes also re-run inference (from retained features)

Re-score without refetch

A rescore campaign bumps (or reuses) the ensemble release and enqueues rescore_asset units that run inference from retained features only — the rescore path has no connector dependency at all (structurally source-free; the acceptance test asserts zero content fetches). This was validated on live CDX data: re-scoring 174 assets took ~1.4 s vs ~151 s for the original content pass (see CDX runbook).

6. Data operations

Storage layout

Database: single SQLite file, FMD_DB_PATH (default ./data/fmd.db, i.e. packages/server/data/fmd.db in dev). Opened with PRAGMA journal_mode = WAL, foreign_keys = ON, busy_timeout = 5000 (packages/server/src/kernel/db.ts). WAL means you will also see fmd.db-wal and fmd.db-shm beside it.
Artifacts (object store): FMD_ARTIFACT_DIR (default ./data/artifacts), files at <artifactDir>/<tenantId>/<artifactId> with metadata (content hash, purpose flags, retention class) in the artifacts table (packages/server/src/kernel/objectstore.ts). The DB and the artifacts dir reference each other — treat them as one backup unit.

Backup

There is no automated backup tooling built — backup is a file-level operation you schedule yourself:

Checkpoint the WAL so the main file is current: sqlite3 packages/server/data/fmd.db "PRAGMA wal_checkpoint(TRUNCATE);" (any SQLite client works).
Copy fmd.db (plus -wal/-shm if present) and the entire artifacts directory together.
For a strictly consistent point-in-time copy, quiesce writers first (stop the API and any standalone workers, or pause campaigns and wait for queueLeased = 0); with WAL, a copy under live writes risks tearing between the DB and the artifacts dir even when the DB copy itself is fine.

Restore = put both back and start the process; migrations are applied idempotently at boot. Dev reset: rm -rf packages/server/data && bun run seed.

Projections

The search projection (search_documents + FTS) and graph projection (graph_nodes/graph_edges) are rebuildable by design from the system of record (ADR-0003; packages/server/src/intelligence/projections.ts). However, no standalone rebuild CLI exists in this release: projections are (re)written by the inference stage, so the practical regeneration path today is a tenant-wide rescore campaign (repopulates projection rows for all active assets, without source I/O). Treat a dedicated rebuild command as not built.

7. Scale triggers

Full analysis in the capacity model (preliminary — slice-scale measurements plus extrapolations; nothing beyond the loadgen harness is validated). Operationally:

The single-writer SQLite deployment is comfortable at slice scale and plausible to roughly 1–5M assets on one strong node. The first contention symptom is queue lease/heartbeat writes competing with observation inserts (both serialize on the one writer, busy_timeout 5 s).
At 1M assets the data is only ~8–15 GB — bytes are not the constraint; the writer and bootstrap wall-clock are.
500M-asset numbers in the capacity model are explicitly unvalidated extrapolations.

Migration triggers per role interface (measure, don’t guess — capacity model §6; each role moves independently behind its interface per ADR-0003):

Role	Trigger	Target
System of record	writer contention between worker and API; ~1–5M assets; need online backup/replication/HA	Azure Database for PostgreSQL, partitioned by tenant + scope
Object/evidence artifacts	artifact bytes beyond tens of GB; lifecycle/envelope-encryption requirements	Azure Blob + lifecycle policies
Search projection	FTS5 rebuild exceeds maintenance windows; concurrent query load	Azure AI Search / OpenSearch (rebuildable, watermarked)
Graph projection	bounded-neighborhood p95 > 3 s despite indexes	dedicated graph store only if traversal demands it
Work queue	lease/heartbeat contention; multiple worker nodes; campaigns beyond ~10⁶ units	Azure Service Bus / Storage Queues
Fingerprint/vector index	band-row growth (16 rows per fingerprinted asset); live embeddings	pgvector / dedicated ANN; pack band rows

8. Log management

The logger (packages/server/src/kernel/log.ts) emits structured JSON lines to stdout, one object per event: {ts, level, event, ...fields}. Redaction guarantee (structural, not best-effort): only allowlisted field keys are emitted (ids, enums, and operational fields such as correlationId, tenantId, assetId, campaignId, workUnitId, stage, retryClass, httpStatus, durationMs, sizeBand, …); string values must match an opaque-identifier charset (^[A-Za-z0-9_.:/-]{1,120}$) or they are replaced with [redacted]; unknown keys are dropped and counted in _droppedFields. Default logs therefore never contain names, UPNs/emails, file names, paths, URLs, titles, excerpts, prompts, tokens, hashes, or embeddings — and this is verified by an end-to-end redaction scan in the test suite. Exact sizes are logged as bands (lt10k … gte10m) where exactness is unneeded. The same allowlist discipline applies to metric names (kernel/metrics.ts). Correlation IDs: every request gets one — taken from the x-fmd-correlation header if the caller supplies it, otherwise generated (packages/server/src/kernel/http.ts). It is attached to request/audit log entries, propagated into the auth context (and from there into audit events), and returned in error responses ({"error": "internal_error", "correlationId": ...}), so an operator can join a user-reported error to logs and audit without any content leaking between them. Ring buffer: the last 500 entries are kept in memory per process for the Operations view; GET /api/ops/overview returns the last 100, filtered to the caller’s tenant. There is no built-in log shipping — collect stdout with your platform’s log pipeline (the Azure IaC in infra/main.bicep provisions Log Analytics for this).

​Operations guide

​1. Process topology

​Scanner fleet semantics

​2. The durable queue

​Work unit lifecycle

​Retry classes

​Dead letters: inspect and replay

​Campaign pause / resume / cancel

​3. Monitoring

​/api/health and /api/ready

​/api/ops/overview

​/api/ops/metrics

​Metric families

​4. Change-notification operations

​Subscription lifecycle

​Receiver and lifecycle webhooks

​Operating it

​5. Scan operations

​Campaign kinds

​Delta 410 resync

​Cache invalidation matrix (summary)

​Re-score without refetch

​6. Data operations

​Storage layout

​Backup

​Projections

​7. Scale triggers

​8. Log management