Troubleshooting

This guide is for operators and developers diagnosing problems in a running Find My Data deployment (or a local dev instance). Each entry follows symptom → cause → resolution and is grounded in the actual error strings and behaviors in the code. Honesty note: Find My Data never silently falls back or overstates its state — a capability that is not fully configured fails with an explicit, actionable error (blocked_capability, unknown, unsupported, blocked), so many of the “errors” below are the product working as designed. Current release: 26.7.20.0, Bun 1.3.14. Contents: Error envelope · Boot · Auth/session · Connectors · Scans · Change notifications · Teams/Exchange · Model releases · UI · Where to look

Reading API errors

All typed errors are mapped centrally (packages/server/src/kernel/errors.ts, handler in packages/server/src/kernel/http.ts). The JSON body is { "error": <code>, "message": <detail> }.

HTTP	`error` code	Meaning
400	`validation_failed`	Bad input, or a precondition like a missing delta cursor. Zod failures include an `issues` array.
401	`authentication_required`	No active session.
403	`not_permitted`	Authenticated but missing the capability. Also written to the audit trail as `authorization_denied`.
403	`csrf_token_invalid`	Mutating request without the session’s CSRF header (see below).
404	`not_found`	Missing — or exists but belongs to another tenant/domain. Existence never leaks across scopes, so a 404 on an ID you “know” exists usually means a scoping problem, not data loss.
409	`conflict`	Valid request, wrong state (e.g. campaign already active, release not evaluated).
422	`blocked_capability`	Feature is honestly blocked: flag off or credentials unresolvable. The message names the exact env vars to set. Never a silent mock fallback (ADR-0012).
500	`internal_error`	Unhandled error. The body includes a `correlationId` — use it to find the matching log line.

Boot and configuration

Server refuses to start: `FATAL: FMD_AUTH_MODE=dev is not permitted when FMD_ENV=production`

Cause: Production boot with dev identity mode. This is a deliberate security invariant in buildConfig() (packages/server/src/kernel/config.ts), covered by tests — the process must not boot, full message: FATAL: FMD_AUTH_MODE=dev is not permitted when FMD_ENV=production. Configure FMD_AUTH_MODE=entra with a valid Entra application.
Resolution: Set FMD_AUTH_MODE=entra. The Entra OIDC code flow is built (ADR-0035) and live-validated; the login endpoints return 501 not_configured only until FMD_ENTRA_TENANT_ID, FMD_ENTRA_CLIENT_ID, FMD_ENTRA_CLIENT_SECRET_REF and FMD_ENTRA_REDIRECT_URI resolve (see Auth below and ADR-0004). Do not work around the boot refusal.

Server refuses to start: `FATAL: default FMD_SESSION_SECRET is not permitted when FMD_ENV=production.`

Cause: FMD_SESSION_SECRET is unset (or set to the dev default dev-only-change-me) with FMD_ENV=production.
Resolution: Set a strong unique FMD_SESSION_SECRET. It signs session cookies and derives change-notification clientState HMACs — rotating it invalidates all sessions and live subscriptions’ clientState verification. Note the schema also requires a minimum length of 8 characters; a shorter value fails config parsing (ZodError) at boot in any environment.

Migrations fail or the schema looks stale

Symptoms: bun run migrate throws a SQLite error; or the server errors on a missing table/column.
Cause: Migrations live in packages/server/migrations/ (0001_identity_governance.sql … 0025_attestations.sql) and are applied in sorted filename order, each inside its own transaction, recorded in the schema_migrations table (packages/server/src/kernel/db.ts). A failing file rolls back only itself; earlier files stay applied.
Resolution:
- Run bun run migrate (root script → bun src/kernel/migrate-cli.ts in packages/server). Output is either Database is up to date. or one applied <file> line per migration.
- Check SELECT name FROM schema_migrations against the directory listing to find the first unapplied file; the SQLite error message identifies what failed in that file.
- database is locked errors: the DB is opened with WAL mode and busy_timeout = 5000; stop the API/worker processes that hold the file before migrating. Single-writer SQLite is a known constraint of the default backend (ADR-0003, scale triggers in capacity model); deployments needing more headroom can run the optional PostgreSQL 16 backend (FMD_DB_BACKEND=postgres + FMD_DB_URL), where the same migrations apply and lock contention behaves differently.
- Never hand-edit schema_migrations to “skip” a failed migration.

Authentication and sessions

401 `authentication_required` on every API call

Cause: No valid session cookie (fmd_session), an expired/revoked session, or — in production — a dev-mode session: kernel/http.ts rejects sessions with auth_mode = 'dev' at request time even if the cookie is otherwise valid (second layer behind the boot refusal).
Resolution: Sign in again. In dev: POST /api/auth/dev/login with a seeded principalId (unknown IDs return 400 Unknown dev persona). Confirm which mode the server runs via GET /api/auth/mode or the x-fmd-auth-mode response header.

403 `{"error":"csrf_token_invalid"}` on POST/PUT/PATCH/DELETE

Cause: Every mutating request from an authenticated session must carry the session’s CSRF token in the x-fmd-csrf header; mismatch or absence returns this error (kernel/http.ts). The web UI fetches the token from GET /api/auth/me (csrfToken field) and attaches it automatically.
Resolution: For scripts/curl: call /api/auth/me first and send x-fmd-csrf: <csrfToken>. If the UI hits this, the session was likely re-issued in another tab — reload so the app refreshes its token. Note: the unauthenticated Graph webhook receivers are exempt (no session, so no CSRF check).

`GET /api/auth/dev/personas` returns 404

Cause: Intentional. The persona list and POST /api/auth/dev/login only exist when FMD_ENV=development and FMD_AUTH_MODE=dev; anything else gets NotFoundError so the endpoint is not even revealed (packages/server/src/identity/auth-routes.ts).
Resolution: Nothing to fix in production — this is the correct behavior. In dev, check both env vars.

Cause: The Entra OIDC flow is built (ADR-0035); 501 means the app registration is not configured. The response message is verbatim Entra OIDC requires FMD_ENTRA_TENANT_ID, FMD_ENTRA_CLIENT_ID, FMD_ENTRA_CLIENT_SECRET_REF and FMD_ENTRA_REDIRECT_URI. See docs/permissions-manifest.md. (In dev auth mode the endpoint 404s.)
Resolution: Set FMD_ENTRA_TENANT_ID, FMD_ENTRA_CLIENT_ID, FMD_ENTRA_CLIENT_SECRET_REF (a reference to another env var holding the secret) and FMD_ENTRA_REDIRECT_URI, then retry — the endpoint 302s to the real Entra authorize endpoint. See the permissions manifest.

Connectors (Microsoft Graph and other sources)

The registry never falls back to mock silently: FMD_CONNECTOR_MODE=graph with missing configuration is an explicit 422 blocked_capability (packages/server/src/connectors/registry.ts). The Graph read connector is built and validated read-only against a live Microsoft 365 tenant (ADR-0013); mock remains the default/CI path. The same posture applies to every other source family — Google Drive, Box, Azure Blob/Files, on-prem SMB shares, and SQL databases — each is gated by its own feature flag and configuration, and a missing piece is an explicit blocked_capability naming the exact env vars (see the source-family table below).

422 `blocked_capability` — which message means what

Message (verbatim)	Cause	Resolution
`Live Microsoft Graph connector requires FMD_GRAPH_TENANT_ID, FMD_GRAPH_CLIENT_ID and FMD_GRAPH_CLIENT_SECRET_REF. See docs/permissions-manifest.md.`	One of the three Graph env vars unset in graph mode.	Set them. `FMD_GRAPH_CLIENT_SECRET_REF` is the name of another env var holding the secret, not the secret itself.
`Live Graph connector: the client-secret reference <ref> does not resolve to a value in the environment.`	The ref is set but the env var it names is empty/absent (e.g. Key Vault reference not mounted).	Ensure the referenced variable is present in the process environment. `GET /api/ready` reports this state as `blocked` — never `live`.
`Live Teams connector requires FMD_FEATURE_TEAMS_CONNECTOR=true and ChannelMessage.Read.All consent. See docs/permissions-manifest.md.`	Teams feature flag off in graph mode.	See Teams and Exchange.
`Live Teams connector requires FMD_GRAPH_TENANT_ID, FMD_GRAPH_CLIENT_ID and a resolvable FMD_GRAPH_CLIENT_SECRET_REF.`	Flag on, Graph credentials incomplete/unresolvable.	As row 1/2.
`Live Exchange connector requires FMD_FEATURE_EXCHANGE_CONNECTOR=true and Mail.Read consent scoped to selected mailboxes. See docs/permissions-manifest.md.`	Exchange feature flag off in graph mode.	See Teams and Exchange.
`Live Exchange connector requires FMD_GRAPH_TENANT_ID, FMD_GRAPH_CLIENT_ID and a resolvable FMD_GRAPH_CLIENT_SECRET_REF.`	Flag on, credentials incomplete/unresolvable.	As row 1/2.

422 `blocked_capability` — other source families

The non-Microsoft source families follow the same pattern: flag off → a message naming the flag and its configuration; flag on but credentials/references unresolvable → a follow-up message naming exactly what failed to resolve. All are thrown by the same registry (packages/server/src/connectors/registry.ts) and reported honestly on GET /api/ready:

Message (verbatim)	Cause	Resolution
`Live Google Drive connector requires FMD_FEATURE_GDRIVE_CONNECTOR=true, FMD_GDRIVE_DRIVES, and a service-account key in the env var named by FMD_GDRIVE_SA_KEY_REF. See docs/permissions-manifest.md.`	Google Drive feature flag off in live mode.	Enable the flag, list the shared drives, set the key ref. Follow-up messages pinpoint a ref that names no env var, a key value that is not valid JSON, or a key missing `client_email`/`private_key`.
`Live Box connector requires FMD_FEATURE_BOX_CONNECTOR=true, FMD_BOX_FOLDERS, and Box app credentials (FMD_BOX_CLIENT_ID / FMD_BOX_CLIENT_SECRET_REF / FMD_BOX_SUBJECT_ID). See docs/permissions-manifest.md.`	Box feature flag off in live mode.	Enable the flag and set the three app credentials. Follow-up messages name the missing credential vars or an unresolvable client-secret reference.
`Live Azure Storage connector requires FMD_FEATURE_AZURE_STORAGE_CONNECTOR=true plus Storage Blob Data Reader (and Storage File Data Privileged Reader for shares) RBAC on the selected resources. See docs/permissions-manifest.md.`	Azure Blob/Files feature flag off in live mode.	Enable the flag and grant the RBAC roles. Identity defaults to the Graph app registration; `Live Azure Storage connector requires an identity: FMD_AZURE_TENANT_ID/FMD_AZURE_CLIENT_ID/FMD_AZURE_CLIENT_SECRET_REF (or the FMD_GRAPH_* equivalents as fallback)` means neither is complete, and an unresolvable secret reference gets its own message.
`Live SMB connector requires FMD_FEATURE_SMB_CONNECTOR=true and mounted share roots in FMD_SMB_SHARES. See docs/permissions-manifest.md.`	SMB feature flag off in live mode.	Enable the flag and mount the share roots on the deployment host — the mount’s credential is the read identity; no app secret is held.
`Live SQL connector requires FMD_FEATURE_SQL_CONNECTOR=true, FMD_SQL_DATABASES, and a connection map in the env var named by FMD_SQL_CONNECTIONS_REF. See docs/permissions-manifest.md.`	SQL feature flag off in live mode.	Enable the flag, list the databases, set the connections ref. Follow-up messages pinpoint a ref that names no env var, a value that is not valid JSON, or a selected database with no connection string in the map (`Live SQL connector: no connection string for '<server/db>'.`).

Connectivity probe fails (`POST /api/admin/connector/test`)

The probe (capability connector.test; Setup page in the UI) acquires an app-only token and reads one site. It returns ok: false with a safe, low-cardinality detail; it never returns a 500 and never leaks a secret, tenant name, or path:

`detail`	Cause	Resolution
`authenticated_sites_reachable`	Healthy (with `ok: true`).	—
`authenticated_no_sites_visible`	Token works but the app can see zero sites.	Grant the app read access to sites (see permissions manifest).
`connector_not_configured`	The registry threw `BlockedError` (missing config, see table above).	Fix env vars.
`graph_token_request_failed_<status>_<status>`	Token endpoint rejected the client credentials (e.g. 400/401: bad secret, wrong tenant, deleted app registration).	Verify tenant ID, client ID, secret validity/expiry.
`graph_<graphErrorCode>_<status>`	Authenticated but the sites read failed; the Graph error code and HTTP status are embedded.	Act on the embedded code (403 → consent missing).
`probe_failed`	Non-Graph failure inside the probe (e.g. network/DNS on the token or sites call).	Check egress to `login.microsoftonline.com` and `graph.microsoft.com`.
`probe_error`	The probe call itself threw unexpectedly at the route level (not a Graph error, not missing config).	Check the server logs via the correlation ID.

Every probe is audited (connector_probe in the config category) with mode, outcome, and detail.

Scans slow down or stall under Graph throttling

Symptom: Live scans crawl; metrics show graph_throttled:429 / graph_throttled:503 counters climbing (GET /api/ops/metrics, Operations page).
Cause: The Graph client honors Retry-After on 429/503 with up to 5 attempts per request, sleeping the server-specified delay capped at 30s (packages/server/src/connectors/graph/client.ts). This is normal backpressure, not a fault.
When it becomes a fault: After 5 throttled attempts the client throws GraphError("throttled_retries_exhausted", 429). Inside a scan this fails the work unit; the worker retries it with queue-level backoff, and only persistent exhaustion dead-letters it.
Resolution: Usually wait — throttling is per-app/per-tenant on Microsoft’s side. If chronic, reduce concurrent scan campaigns and review the tenant’s Graph usage.

Delta scan triggers a full resync (`delta_resync_required` in logs)

Symptom: A delta campaign suddenly does inventory-scale work; log event delta_resync_required; the scope’s cursor status flips to resync_required and its coverage_state to stale.
Cause: Graph returned 410 Gone for the stored delta token — the connector maps this to { kind: "resync_required" } (connectors/graph/connector.ts), and the worker re-baselines inside the same campaign by enqueuing a resync:page:0 inventory unit (packages/server/src/scan/worker.ts).
Resolution: None needed — this is the designed self-heal (ADR-0007). Unchanged assets resolve as stage-cache hits, so no duplicate logical work is done (tested). If resyncs recur constantly, the delta cursor is being invalidated upstream (e.g. drive restructuring).

Scans and the work queue

400 `No delta cursor for this scope yet — run an inventory scan first`

Cause: POST /api/scans with kind: "delta" for a scope that has never completed an inventory scan — a delta cursor only exists after inventory reaches its terminal deltaLink (packages/server/src/scan/campaigns.ts).
Resolution: Run an inventory campaign for the scope first. (This same condition is why change notifications can report reconcile_skipped / no_delta_cursor — see below.)

409 `A campaign is already active for this scope`

Cause: A campaign in state pending, running, or paused already exists for the scope; one active campaign per scope by design.
Resolution: Wait for it, or cancel it (POST /api/scans/:id/cancel). If it looks permanently stuck, check the queue (below) — a stuck lease resolves itself. Related conflict: Cannot <pause|resume|cancel> a campaign in state <status> means the transition isn’t legal from the current state (pause only from running, resume only from paused, cancel from running/paused/pending).

Campaign appears stuck (work units leased, no progress)

Cause & self-healing: The queue is at-least-once with 60-second leases (LEASE_MS in packages/server/src/scan/queue.ts). A crashed worker’s units become leasable again when the lease expires. Additionally, the fleet sweep (packages/server/src/scan/fleet.ts) marks a scanner stale after 30s without heartbeat and offline after 90s, at which point all its in-flight leases are released immediately for takeover by healthy scanners (tested).
Resolution:
- Check the fleet on the Operations page or GET /api/ops/overview (fleet array: status, last heartbeat, leased_now). An offline scanner with released work is already handled.
- If no scanner is running at all, start one: the API process embeds a worker loop, or run a standalone worker with bun run worker.
- A paused campaign hands out no new leases but does not touch in-flight units; resuming a campaign whose last unit finished while paused completes it immediately rather than stranding it in running.

Dead letters: triage by retry class

Failed work units are retried with backoff until max_attempts (default 4), except terminal classes which dead-letter immediately (packages/server/src/scan/queue.ts). Dead letters carry only a retry_class and a sanitized safe_diagnostic (no names, paths, or content). View them in GET /api/ops/overview (deadLetters, latest 50), per-campaign in GET /api/scans/:id, or the Operations page.

`retry_class`	Terminal?	Typical cause	Triage
`transient`	retried (dead only after max attempts)	Network blips, temporary Graph/DB errors	If it dead-lettered, the fault persisted through 4 attempts — check connectivity and the diagnostic slug.
`throttled`	retried, longer linear backoff	Sustained 429/503	See throttling; reduce concurrency.
`auth`	retried	Token/credential failure mid-scan	Check secret expiry and the connector probe.
`unsupported`	yes	An error whose message matches `unsupported\|format` (e.g. `unsupported work kind`). Note: a file format the extractor can’t handle does not fail the unit — the unit completes with `content_state = 'unsupported'` (see extraction states below).	Usually a work-unit kind/format mismatch, not content.
`corrupt`	yes	Unreadable/garbled source bytes	Inspect the source file.
`policy_denied`	yes	Denied by policy	Review policy configuration.
`permanent`	yes	Missing scope/cursor/object (`classifyError` maps messages matching `cursor\|not found\|missing\|invalid`, plus selection/authorization refusals matching `not_in_current_selection\|blocked\|escapes_root` — e.g. `delta cursor missing`, `campaign scope missing`, or a source de-selected mid-scan)	Usually a deleted or reconfigured source; re-run inventory.

Honesty note: the scan worker’s classifier (classifyError in packages/server/src/scan/worker.ts) currently assigns only transient, unsupported, and permanent. The throttled, auth, corrupt, and policy_denied classes are defined in the queue’s retry policy (RETRY_CLASSES in packages/shared/src/domain-enums.ts) and behave as described above if assigned, but no scan path emits them yet — sustained throttling or credential failures surface today as transient.

Extraction: “unsupported” vs “failed” vs “inaccessible”

These are distinct, honest observable states on asset_versions.content_state — never a silent skip (packages/server/src/intelligence/extract.ts, packages/server/src/scan/pipeline.ts; per-format counters in metric extraction:<format>:<state>):

not_fetched — content stage hasn’t run for this version yet.
available / partial — extracted (partial = truncated at output bounds).
unsupported — the connector delivered bytes but no extractor claims the format (e.g. scanned/CID PDFs, true binaries). Honest capability limit; OCR is roadmap (roadmap).
failed — an extractor should have handled it but errored, or the file exceeded size bounds (connector-side maxBytes, or extractor input limits). Worth investigating, unlike unsupported.
inaccessible — the source returned 403/404 for content; the app can see the item but not read it. Check item-level permissions.

Change notifications

Built end-to-end (subscription lifecycle, unauthenticated receiver with clientState HMAC, dedup, delta reconcile) and validated end-to-end via the dev simulator; live Graph subscriptions additionally require a public HTTPS webhook URL and the feature flag, and are honestly disabled (flag off) or blocked (flag on, URL/credentials unresolvable) until then (packages/server/src/notifications/subscriptions.ts, GET /api/ready → changeNotifications).

Subscription creation returns 422 `blocked_capability`

Message (verbatim)	Resolution
`Change notifications require FMD_FEATURE_CHANGE_NOTIFICATIONS=true. See docs/permissions-manifest.md.`	Enable the flag (live-mode scopes only; mock scopes subscribe locally without it).
`Change notifications require FMD_WEBHOOK_PUBLIC_URL (a public HTTPS endpoint Graph can reach) to register a subscription.`	Set `FMD_WEBHOOK_PUBLIC_URL`. Graph must be able to POST to `<url>/api/webhooks/graph/notifications` and answer the validation-token handshake — a localhost or firewalled URL will fail at Graph’s validation step.
`Live change notifications require the Microsoft Graph connector.`	The configured live connector isn’t the Graph connector; check `FMD_CONNECTOR_MODE` and Graph config.

Without a public endpoint, use the development simulator (POST /api/dev/notifications/simulate, dev-only, 404 elsewhere) which drives the identical receiver + reconcile path with a forced mock subscription.

Webhook returns 202 but nothing happens

POST /api/webhooks/graph/notifications always returns 202 fast regardless of per-item outcome (Graph requires it), so a 202 proves delivery, not acceptance. Check the per-notification outcome in GET /api/admin/notifications/recent and the metrics:

change_notification:rejected:client_state_mismatch climbing, no notification rows: the clientState HMAC didn’t verify. By design nothing is persisted on mismatch (an unauthenticated caller must not amplify writes); the metric is the only trace. Cause: FMD_SESSION_SECRET changed after the subscription was created (the HMAC derives from it) — a Graph subscription registered under the old secret keeps posting the old clientState. Fix: recreate subscriptions after any secret rotation.
change_notification:rejected:unknown_subscription: Graph posted for a subscription ID this database doesn’t know — typically leftovers from a wiped DB. Delete the orphan subscription in Graph or let it expire.
deduplicated: a same-subscription hint arrived within the 10s debounce window; the earlier one already triggered reconcile. Normal.
Lifecycle events with wrong/absent clientState are dropped silently (packages/server/src/notifications/routes.ts) — also by design.

Notification accepted but `reconcile_skipped`

The stored reject_reason on the notification row explains it — both common cases are benign (ingestNotification):

Reason	Meaning	Action
`no_delta_cursor`	Scope has never completed inventory, so there is no delta feed to reconcile through.	Run an inventory scan; subsequent notifications will reconcile.
`scan_already_active`	A campaign is already running for the scope; the running scan will observe the change.	None.
`no_scope`	Subscription row has no linked scope.	Recreate the subscription for a real scope.

Subscriptions expire or show `status = 'error'`

Live subscriptions are short-TTL (60 min) and renewed by the periodic sweep when under 20 min remain. A failed renewal sets status = 'error', last_error = 'renew_failed' (metric change_subscription_renew_failed); expired ones are swept to expired. Graph lifecycle events are handled: reauthorizationRequired → marked expiring for the next renew, missed → a forced delta reconcile, subscriptionRemoved → marked removed.
Resolution: trigger POST /api/admin/notifications/renew, or re-subscribe (POST /api/admin/notifications/subscribe returns the existing subscription unless it is removed/error, in which case it re-registers). Confirm the API process is running — its embedded worker loop hosts the renewal sweep (packages/server/src/main.ts); the standalone bun run worker does not run it.

Teams and Exchange connectors

Both are built and mock-validated; the live Graph paths are feature-gated and have not been validated live (see the STATUS honest feature list). Sync endpoints: POST /api/teams/sync, POST /api/exchange/sync (capability connector.configure); read views require operations.read because messages are not yet mapped to business-domain ownership (deny-by-default).

422 `blocked_capability` on sync

The four gating messages are listed verbatim in the connector table. Flags: FMD_FEATURE_TEAMS_CONNECTOR, FMD_FEATURE_EXCHANGE_CONNECTOR. Exchange additionally reads only mailboxes listed in FMD_EXCHANGE_MAILBOXES (comma-separated, least-privilege).

Cause: Teams channel-message read requires ChannelMessage.Read.All — a protected/metered Graph API needing Microsoft approval beyond baseline read; Exchange requires Mail.Read (scope it with an application access policy). Without consent, Graph returns 403; the adapters surface it honestly rather than fabricating data (packages/server/src/connectors/teams/graph.ts, .../exchange/graph.ts).
What you’ll see: a hard 403 on the message-listing call aborts the sync — at the API surface this is a 500 internal_error with a correlationId (a GraphError is not a typed app error). Two spots degrade instead of failing: a 403/404 on a Teams channel’s filesFolder just leaves the backing folder unknown, and unreadable Exchange attachments return an empty attachment list.
Resolution: Complete the consent per the permissions manifest, then re-run sync. Verify honest state first in GET /api/ready → teamsConnector / exchangeConnector: mock, disabled (flag off), blocked (flag on, credentials unresolvable), or live.

Model releases

The governed release lifecycle (packages/server/src/models/pipeline.ts; UI page /models; API under /api/models/) has been validated end-to-end on live tenant data. Every gate below is intentional:

Error (verbatim message)	HTTP	Why	Resolution
`A release must be evaluated before approval`	409	Only `evaluated` releases can be approved — no unmeasured promotion.	`POST /api/models/releases/:id/evaluate` first.
`Separation of duties: a release must be approved by someone other than its proposer`	403 `not_permitted`	SoD: approver ≠ proposer, always (covered by tests).	Have a second `model.release.approve` holder approve. Not configurable.
`Dataset snapshot has no eval-split members`	400	The cluster-aware split produced zero eval members — snapshot too small or single-cluster. (With ≥2 clusters the split auto-balances so neither side is empty.)	Approve more dataset candidates spanning more distinct content, snapshot again.
`Release has no dataset snapshot to evaluate against`	400	Release proposed without `datasetSnapshotId`.	Propose again referencing a snapshot.
`No approved dataset candidates to snapshot — curate candidates first`	400	Snapshot creation with zero approved candidates.	Curate on the Models page (`.../dataset-candidates/:id/decide`).
`A dataset snapshot with that name already exists`	409	Snapshot names are unique per tenant (the name salts the deterministic split).	Pick a new name.
`Only an approved release can be promoted` / `Only a promoted release can be rolled back`	409	State-machine ordering: proposed → evaluated → approved → promoted (→ rolled_back).	Follow the sequence; check `GET /api/models/governed-releases` for current status.

Web UI

Nav sections are missing / page is blank for a user

Not a bug — the sidebar is capability-composed: the SURFACES registry in packages/web/src/App.tsx drives both nav visibility and the route gates, and the API enforces the same capabilities server-side (deny-by-default; a URL typed directly still gets 403/404):

Nav item	Visible when
Home	always (signed in)
My Domain group — Landscape, Review, Findings, Actions, Analyst (with the domain switcher and `/d/<id>` scoped URLs)	user owns ≥1 business domain (`ownedDomains` from `/api/auth/me`)
Approvals (tenant group)	any of `action.approve` / `governance.taxonomy.approve` / `model.release.approve`
Actions ledger (tenant group)	`audit.read`
All findings (tenant group)	tenant-wide `finding.read`
Governance (tenant group)	`governance.taxonomy.propose`, tenant-wide `governance.policy.read`, or `governance.owner.assign`
Model releases (tenant group)	`model.dataset.curate`, `model.release.propose`, or `model.release.approve`
Connections (`/setup`, platform group)	`connector.configure` or `tenant.profile.write`
Operations (platform group)	`operations.read`
Audit trail (platform group)	`audit.read`

Deliberate exclusions (see packages/shared/src/capabilities.ts): platform_admin has no evidence/excerpt/review access, and domain_owner capabilities are domain-scoped, not tenant-wide. So “admin can’t see review items” is the design, not a permissions bug. Shown whenever the session’s auth mode is dev. It means anyone with network access can sign in as any seeded persona — fine locally, never acceptable for real data. Production cannot show it: dev auth is refused at boot and dev sessions rejected per-request. If the persona picker itself is missing at login, the server is not in dev mode (the login page checks GET /api/auth/mode and only shows the picker for dev); an empty persona list in dev mode means an unseeded database — run bun run seed.

Where to look (diagnostics)

Correlation IDs. Every request gets one (or honors a client-supplied x-fmd-correlation header). It appears in the http_request log line for every request, in every audit event, and in 500 bodies. Given a user-reported error with a correlationId, grep the server logs or the Operations page’s recent-log panel for it.
GET /api/ops/overview (capability operations.read; Operations page): work-queue counts by status, latest 50 dead letters (retry class + safe diagnostic), scanner fleet health, and recentLogs — the last 100 in-process log entries, filtered to your tenant (plus tenant-agnostic system events; other tenants’ entries are never shown). Logs are allowlist-redacting by design: no names, paths, URLs, or content phrases (tested end-to-end) — expect slugs and IDs, and pivot on the correlation ID for detail.
GET /api/ops/metrics: counters/histograms (stage latency, extraction:<format>:<state>, graph_throttled:*, change_notification:*, work outcomes) plus DB-derived gauges (queue depth, oldest pending age, delta-cursor lag, content states).
GET /api/ready: honest per-integration mode — mock / live / blocked / disabled. blocked means the capability cannot go live yet: for the connector, AI provider, Teams/Exchange, and change notifications it is “enabled but credentials (or webhook URL) don’t resolve”; the Purview label rows also report blocked when their feature flag is simply off, and entraSignIn reports disabled in dev auth mode, blocked when FMD_AUTH_MODE=entra but the Entra app-registration env vars don’t resolve, and live once they do. If you expected live, start here.
GET /api/health: liveness only ({"status":"ok"}).
Audit trail (GET /api/audit, capability audit.read; Audit page): hash-chained per-tenant events, filterable by category. Authorization denials land here as authorization_denied with the missing capability in policy_decision — the fastest way to answer “why was this user blocked”. Scan, config, model, and action lifecycles are all audited with correlation IDs.
Per-campaign detail: GET /api/scans/:id returns work-unit counts by status, stage outcomes (computed / cache_hit / skipped / failed per stage), and that campaign’s dead letters.

Find My Data release 26.7.20.0 · Bun 1.3.14.

Get started

Run & operate

Security & trust

Reference

Licensing & releases

Reading API errors

Boot and configuration

Server refuses to start: `FATAL: FMD_AUTH_MODE=dev is not permitted when FMD_ENV=production`

Server refuses to start: `FATAL: default FMD_SESSION_SECRET is not permitted when FMD_ENV=production.`

Migrations fail or the schema looks stale

Authentication and sessions

401 `authentication_required` on every API call

403 `{"error":"csrf_token_invalid"}` on POST/PUT/PATCH/DELETE

`GET /api/auth/dev/personas` returns 404

Connectors (Microsoft Graph and other sources)

422 `blocked_capability` — which message means what

422 `blocked_capability` — other source families

Connectivity probe fails (`POST /api/admin/connector/test`)

Scans slow down or stall under Graph throttling

Delta scan triggers a full resync (`delta_resync_required` in logs)

Scans and the work queue

400 `No delta cursor for this scope yet — run an inventory scan first`

409 `A campaign is already active for this scope`

Campaign appears stuck (work units leased, no progress)

Dead letters: triage by retry class

Extraction: “unsupported” vs “failed” vs “inaccessible”

Change notifications

Subscription creation returns 422 `blocked_capability`

Webhook returns 202 but nothing happens

Notification accepted but `reconcile_skipped`

Subscriptions expire or show `status = 'error'`

Teams and Exchange connectors

422 `blocked_capability` on sync

Model releases

Web UI

Nav sections are missing / page is blank for a user

Yellow banner: “Development identities — not for production. Signed in as a seeded fixture persona.”

Where to look (diagnostics)

​Reading API errors

​Boot and configuration

​Server refuses to start: FATAL: FMD_AUTH_MODE=dev is not permitted when FMD_ENV=production

​Server refuses to start: FATAL: default FMD_SESSION_SECRET is not permitted when FMD_ENV=production.

​Migrations fail or the schema looks stale

​Authentication and sessions

​401 authentication_required on every API call

​403 {"error":"csrf_token_invalid"} on POST/PUT/PATCH/DELETE

​GET /api/auth/dev/personas returns 404

​GET /api/auth/entra/login returns 501 not_configured

​Connectors (Microsoft Graph and other sources)

​422 blocked_capability — which message means what

​422 blocked_capability — other source families

​Connectivity probe fails (POST /api/admin/connector/test)

​Scans slow down or stall under Graph throttling

​Delta scan triggers a full resync (delta_resync_required in logs)

​Scans and the work queue

​400 No delta cursor for this scope yet — run an inventory scan first

​409 A campaign is already active for this scope

​Campaign appears stuck (work units leased, no progress)

​Dead letters: triage by retry class

​Extraction: “unsupported” vs “failed” vs “inaccessible”

​Change notifications

​Subscription creation returns 422 blocked_capability

​Webhook returns 202 but nothing happens

​Notification accepted but reconcile_skipped

​Subscriptions expire or show status = 'error'

​Teams and Exchange connectors

​422 blocked_capability on sync

​Sync fails with a Graph 403 (protected-API consent)

​Model releases

​Web UI

​Nav sections are missing / page is blank for a user

​Yellow banner: “Development identities — not for production. Signed in as a seeded fixture persona.”

​Where to look (diagnostics)

Reading API errors

Boot and configuration

Server refuses to start: `FATAL: FMD_AUTH_MODE=dev is not permitted when FMD_ENV=production`

Server refuses to start: `FATAL: default FMD_SESSION_SECRET is not permitted when FMD_ENV=production.`

Migrations fail or the schema looks stale

Authentication and sessions

401 `authentication_required` on every API call

403 `{"error":"csrf_token_invalid"}` on POST/PUT/PATCH/DELETE

`GET /api/auth/dev/personas` returns 404

`GET /api/auth/entra/login` returns 501 `not_configured`

Connectors (Microsoft Graph and other sources)

422 `blocked_capability` — which message means what

422 `blocked_capability` — other source families

Connectivity probe fails (`POST /api/admin/connector/test`)

Scans slow down or stall under Graph throttling

Delta scan triggers a full resync (`delta_resync_required` in logs)

Scans and the work queue

400 `No delta cursor for this scope yet — run an inventory scan first`

409 `A campaign is already active for this scope`

Campaign appears stuck (work units leased, no progress)

Dead letters: triage by retry class

Extraction: “unsupported” vs “failed” vs “inaccessible”

Change notifications

Subscription creation returns 422 `blocked_capability`

Webhook returns 202 but nothing happens

Notification accepted but `reconcile_skipped`

Subscriptions expire or show `status = 'error'`

Teams and Exchange connectors

422 `blocked_capability` on sync

Sync fails with a Graph 403 (protected-API consent)

Model releases

Web UI

Nav sections are missing / page is blank for a user

Yellow banner: “Development identities — not for production. Signed in as a seeded fixture persona.”

Where to look (diagnostics)