Data catalog & handling

This document is for privacy/compliance reviewers and data-governance stakeholders evaluating Find My Data. It catalogs every category of data the platform acquires from customer sources, derives from content, retains, and — equally important — what it never stores, with sensitivity, retention behavior, and access level for each. Every table name, column, constant, and env var below was verified against the code at release 26.7.15.0 (Bun 1.3.14 runtime, 196 passing tests). Companion documents: the threat model (trust boundaries and residual risks), ADR-0011 (retention and evidence profile), ADR-0003 (persistence model), and the permissions manifest (what the product asks Microsoft for).

How to read this catalog

Validation states. Find My Data never overstates its state. Each category below is marked with one of:

Live-validated (CDX) — exercised against the live Contoso CDX demo tenant, read-only (ADR-0013, docs/cdx-test-runbook.md §0).
Mock-validated, live gated — built and tested against the deterministic mock M365 fixture; the live path exists but is behind an explicit feature flag and has not been run against a real tenant.
Not built — recorded intent only; called out explicitly.

Storage locations. All structured data lives in one SQLite database (FMD_DB_PATH, default ./data/fmd.db) behind role interfaces (ADR-0003). Content-derived blobs live in the artifact store (FMD_ARTIFACT_DIR, default ./data/artifacts) at <FMD_ARTIFACT_DIR>/<tenantId>/<artifactId>, indexed by the artifacts table. Every tenant-scoped business table carries a NOT NULL tenant_id; the deliberate exception is the deployment-level scanners table (§7), which is infrastructure, not tenant data. Retention. Retention durations are not enforced in this build. Artifacts carry a retention_class column (default standard) and ADR-0011 records per-class TTLs as a follow-up, but no scheduled deletion sweep exists (threat model §4.3). Where a row below says “until superseded,” the schema keeps temporal history (is_current, valid_from/valid_to, supersession chains) rather than deleting.

1. Evidence access levels

Evidence rows carry an explicit access classification, and reads are gated by distinct capabilities so that “can see that an asset exists” never implies “can read content excerpts.”

Level	`evidence.access_classification`	Capability required	Scope
Metadata	`metadata`	`asset.metadata.read`	domain (for `domain_owner`)
Summary	`evidence_summary`	`asset.evidence.summary.read`	domain
Excerpt	`evidence_excerpt`	`asset.evidence.excerpt.read`	domain

Enforcement facts (all tested):

The capability list is defined in packages/shared/src/capabilities.ts; platform_admin deliberately holds no asset.evidence.*, review.*, or analyst.* capabilities.
A caller with summary access but not excerpt access receives evidence with excerpts: null — never a lesser-redacted fallback (packages/server/src/review/service.ts).
Every evidence read is audited under category evidence_access, with distinct actions for reads that included excerpts (candidate_evidence_with_excerpts_read) vs. reads that did not.
Artifact-store reads are additionally purpose-checked: ObjectStore.get requires a declared purpose against the artifact’s content-use flags (use_flags_json, §3) and throws when the artifact does not permit it.

2. Data-handling profiles

tenants.data_handling_profile accepts three values (migration 0001). The exact profile notes shipped in packages/server/src/admin/settings.ts:

Profile	Note (verbatim from code)
`metadata_only`	”Only metadata and derived signals are retained; no excerpts or extracted text.”
`minimized_evidence`	”Minimal bounded excerpts and detector evidence are retained; extracted text is discarded after scoring.”
`enhanced_evidence`	”Bounded extracted text is retained to accelerate re-scoring; requires a documented data-handling deviation (ADR-0011).”

Honest implementation state: only minimized_evidence (the default) drives actual pipeline behavior, and the current build deviates from its own profile note in one documented way: bounded extracted text (≤ 20,000 characters) is retained, because source-free re-scoring on detector releases requires a text representation. This deviation, its rationale, and its follow-ups (envelope encryption, per-class TTLs, a true drop-text option) are recorded in ADR-0011 and the threat model §4.3. The metadata_only and enhanced_evidence values are schema-ready configuration only — validated, audited, and cache-invalidating (a profile change bumps the tenant config_version, which participates in the evidence-bearing stage cache keys — content, features, inference — so the next scan/rescore re-derives those stages; tested in scan.test.ts and admin.test.ts), but they do not yet change what the pipeline persists. A reviewer should treat the platform’s actual retention posture as: minimized evidence, plus bounded extracted text as the ADR-0011 deviation.

3. Catalog: content-derived data (highest sensitivity)

Everything in this section is derived from customer file content. Per the handoff security model, derivative data is still sensitive — none of these tables is “safe” merely because it holds no full text.

Data	Where	Contents & bounds	Access	Retention
Bounded extracted text	`artifacts` (kind `extracted_text_bounded`) + blob in `FMD_ARTIFACT_DIR`	Extracted text capped at 20,000 chars (`DEFAULT_LIMITS.maxOutputChars`, `packages/server/src/intelligence/extract.ts`); classification `sensitive_derivative`; use flags `classification_inference` + `owner_review` only	Purpose-checked `ObjectStore.get`; not readable for search/analyst/evaluation/training/troubleshooting purposes	Indefinite in this build (ADR-0011 deviation; TTL sweep not built)
Redacted excerpts	`feature_artifacts` (kind `excerpts`), surfaced via review	≤ 240 chars each, ≤ 3 per asset version (`EXCERPT_MAX_CHARS`, `EXCERPT_MAX_COUNT`, `packages/server/src/scan/pipeline.ts`); digit-masked (SSN patterns, `$` amounts, dates) before storage	`asset.evidence.excerpt.read` (domain-scoped), audited	Until superseded by a new content version
Keyphrases, language, detector hits	`feature_artifacts` (kinds `keyphrases`, `language`, `detector_hits`, `structure`)	Derived features keyed by `algo_version` + `config_version` + `input_hash`; small features inline (`value_json`), large ones in the artifact store	Summary level	Until superseded; rows are keyed by `algo_version`/`config_version`, so a version change stops matching old rows (the schema reserves an `incompatible` status; no code sets it yet)
Exact & similarity fingerprints	`fingerprints`, `fingerprint_bands`	`sha256` over `raw_bytes` and `normalized_text`; `simhash64` with a banded index for near-duplicate candidate retrieval	Summary level (fingerprint values are never logged)	Until superseded
Embeddings	`asset_embeddings`, `embedding_buckets`	Model `fmd-hash-256-v1`, dim 256, L2-normalized vectors stored inline as JSON + cosine-LSH signature (24 bits, 8×3-bit bands) (`packages/server/src/intelligence/embeddings.ts`). Default embedding is a deterministic feature-hash — no content leaves the process. Provider embeddings (Azure OpenAI) are config-gated and unexercised	Owner-scoped semantic-neighbors API	One row per (asset version, model); replaced on content change
Evidence records	`evidence`	Kind, direction, weight, bounded safe values (`value_json`), provenance, optional artifact link, `access_classification`	Per-row access level (§1)	Append-style; linked from assertions
Search projection	`search_documents`, `search_fts` (FTS5: title, keyphrases, excerpt)	Rebuildable index over already-bounded derived text; domain id is a security facet	Domain-scoped search	Rebuildable — deletable and reconstructible from source tables
Graph projection	`graph_nodes` (`label_safe`), `graph_edges` (typed, with required `reason`)	Display text explicitly bounded to the owner evidence level	Owner-scoped	Rebuildable

Validation state: extraction, fingerprints, keyphrases, detector hits, excerpts, and embeddings are live-validated (CDX) — 172/174 real files extracted, 172 embeddings indexed, re-score-without-refetch proven live. Azure OpenAI/Anthropic AI-provider paths are config-gated and unexercised; the deterministic mock is the only exercised provider.

4. Catalog: source observations (metadata acquired from connectors)

Data	Where	Contents	Sensitivity	Retention
Asset identity & versions	`physical_assets`, `asset_versions`	Stable external IDs, provider eTag/cTag, size, MIME type, source timestamps, honest `content_state` (`not_fetched`/`available`/`inaccessible`/`unsupported`/`partial`/`failed`)	Moderate (existence + shape, no content)	Version history retained; `is_current` flag
Names, paths, locations	`asset_locations`, `source_containers`, `source_scopes`	File name, path, container hierarchy, site/drive display names and URLs	Moderate–high (names and paths are sensitive per the handoff; never logged)	Temporal history (`valid_from`/`valid_to`) — renames preserve identity and history
Permission observations	`permission_sets` (deduplicated by digest), `permission_grants`, `asset_permission_obs`	Normalized grants: principal references, role, inheritance, link scope, broad-audience categories (e.g. `tenant_wide_group`, `anonymous_link`) with `principal_count` — not unbounded per-user expansion. Every observation carries `completeness` (`complete`/`partial`/`not_observable`): “no broad share observed” is never presented as “none exists”	Moderate	Observations superseded, sets deduplicated tenant-wide
Label observations	`asset_label_obs`, `sensitivity_labels` (catalog)	Label GUID + state (`labeled`/`unlabeled`/`unknown`/`unsupported`/`inaccessible`) and acquisition method; label catalog with GUID/name/priority	Low–moderate	Until superseded
Activity aggregates	`activity_aggregates`	Windowed read/edit/share counts, last-activity time, editor principal ids; `source_state` honestly reports `not_enabled` when the signal is unavailable	Moderate	Aggregate rows retained per window
Directory identities	`principals`, `group_memberships`	Display name, email, title, department, external directory id; membership rows carry `completeness`	Moderate–high (directory PII)	Lifecycle status, not deletion
Connector cursors	`source_cursors`	Opaque provider delta/inventory tokens, stored verbatim (plaintext) with checksum	Moderate — honest limitation: ADR-0008 documents opaque+encrypted persistence for live delta links as future work	Replaced on advance/invalidation

Validation state: the read-only Graph inventory/delta/permission/label-observation path is live-validated (CDX): 34 drives discovered, 174 assets via driveItem delta, deduplicated permission sets, labels honestly unknown under a read-only app. The mock connector remains the default and CI path (FMD_CONNECTOR_MODE=mock).

5. Catalog: messaging (Teams / Exchange)

Raw message bodies are never persisted. Both connectors persist the same bounded shape (packages/server/src/teams/service.ts, packages/server/src/exchange/service.ts):

Preview: whitespace-normalized body, passed through redactSensitivePreview, truncated to 200 chars (BODY_PREVIEW_MAX).
Redaction (packages/server/src/intelligence/redact.ts) masks, in order: US SSNs (123-45-6789 / space-separated) → [redacted-ssn]; card-like 13–16 digit runs (optionally space/dash grouped) → [redacted-card]; bare digit runs of 9+ → [redacted-number]. Detector signals are kept separately; only raw values are masked.
Body hash: sha256 of the full body (body_hash) for change detection — non-reversible.
Detector signals: detector_hits_json holds only {detectorKey, informationTypeName, count} (SafeDetectorHit) — no matched text.

Data	Where	Notes
Teams topology	`teams_teams`, `teams_channels`	Team/channel names, visibility, backing SharePoint drive reference
Teams messages	`teams_messages`	Sender external id + display name, threading ids, preview/hash/signals as above
Teams attachments	`teams_attachment_refs`	Canonicalized to the backing SharePoint driveItem — a channel file never becomes a duplicate asset
Exchange mailboxes/folders	`exchange_mailboxes`, `exchange_folders`	Selected mailboxes only (`FMD_EXCHANGE_MAILBOXES`); `access_scope` records the least-privilege posture (`application_access_policy`, never silently `tenant_wide`)
Exchange messages	`exchange_messages`	Subject, sender address/name, importance (stored as metadata, not redacted), plus preview/hash/signals as above
Exchange attachments	`exchange_attachments`	File name, size, content type, `sha256` content/normalized-text hashes; a match to a SharePoint asset is recorded as similarity (`match_method`: `identical_content`/`identical_text`/`near_duplicate`/`filename_size`), never merged identity — an email attachment is a copy

Access: Teams/Exchange messages are not yet mapped to business-domain ownership, so their read views are admin/audit-scoped, deny-by-default (STATUS.md known limitation). Validation state: mock-validated, live gated. Both connectors are built and tested against fixtures (including a regression test that the preview redaction holds). Live paths require FMD_FEATURE_TEAMS_CONNECTOR (+ ChannelMessage.Read.All, a protected API) and FMD_FEATURE_EXCHANGE_CONNECTOR (+ Mail.Read scoped by application access policy) and have not been run against a real tenant.

6. Catalog: governance and human-decision records

Data	Where	Contents	Notes
Org profile & policy documents	`org_profiles`, `policy_documents`, `policy_sections`	Customer-authored policy full text (section bodies) with citations; uploaded deliberately by a governance admin	Customer-provided governance content, not scanned content
Taxonomy	`business_domains`, `information_types`, `classifications`, `sensitive_categories`, `data_elements`, mapping tables	Names, definitions, examples/counterexamples; provenance (`ModelInferred` vs. human) and draft→approved lifecycle	AI proposals never become active without human approval
Ownership & roles	`ownership_assignments`, `role_assignments`	Principal↔domain assignments with validity windows and attestation timestamps	Drives domain-scoped authorization
Assertions & review	`assertions`, `review_candidates`, `review_decisions`	Classification candidates with confidence, provenance, evidence links; owner decisions (six kinds) with optional free-text `note`. Decisions create linked `OwnerConfirmed` assertions — the original inference is never destroyed	Supersession chains retained by design
Dataset candidates & snapshots	`dataset_candidates`, `dataset_snapshots`, `dataset_members`	Snapshot rows hold references (candidate/asset/information-type ids, label, train/eval split, cluster key) plus counts — not content. `consent_scope` defaults to `tenant_internal`; `vendor_shareable` requires an explicit customer choice (migration `0008`)	No artifact in this build carries `training`/`evaluation` use flags (threat model §4.4)
Model releases	`model_releases`, `release_evaluations`, `detector_releases`, `ensemble_releases`	Proposed/base config JSON, rationale, separation of duties (`decided_by` must differ from `proposed_by`), aggregate evaluation counts (TP/FP/TN/FN, precision/recall/F1) only	Live-validated (CDX): the full curate→snapshot→evaluate→approve→promote loop ran on live data
Analyst queries	`analyst_queries`	The user’s question text, validated plan JSON, server-injected scope, result row references, coverage, and `answer_text`	Stored for reproducibility; scoped + audited. `analyst.export` capability exists in the namespace but no export route is implemented
AI generation runs	`generation_runs`	Task, provider, model, template version, and an input hash only	Raw prompts/responses are never persisted

7. Catalog: operational data

Data	Where	Contents	Safety property
Work queue	`scan_campaigns`, `work_units`, `outbox_events`	Campaign parameters, idempotency keys, lease/retry state; `progress_total_est` is `NULL` when unknown, never fabricated	Params are structured references, not content
Stage attempts & cache	`stage_attempts`, `stage_cache`	Cache keys, outcomes, impl/config versions; `result_json` holds “safe summary/result references only” (schema comment, migration `0002`)	Cache keys include tenant + config version
Dead letters	`dead_letters`	`safe_diagnostic` is an explicitly redacted string — no paths, names, or content	Truncated safe strings by contract
Scanner fleet	`scanners`	Workload identity, pseudonymous host id, capabilities, version, heartbeat, processed/failed counts	Deployment-level, not tenant-scoped
Metrics	In-process only (`packages/server/src/kernel/metrics.ts`)	Counters/gauges/histograms with allowlist-shaped names (`^[a-z0-9_.:-]{1,80}$`), bounded cardinality (500 series, 2,000 samples/series)	Never per-tenant business data; no free-form content; not persisted
Change notifications	`change_subscriptions`, `change_notifications`	Subscription resource, lifecycle, expiry; received notifications with dedup key, freshness timestamps, and outcome. `clientState` is never stored raw — only its `sha256` (`client_state_hash`); the receiver re-derives the expected value with HMAC-SHA256 and compares in constant time before any side effect. Error fields hold safe categories only	A notification is a hint that triggers delta reconciliation — never trusted content. Mock/simulator-validated; live Graph subscriptions gated on `FMD_FEATURE_CHANGE_NOTIFICATIONS` + `FMD_WEBHOOK_PUBLIC_URL`
Sessions	`sessions`	Session id, CSRF token, expiry, revocation; cookie value is HMAC-signed	8-hour expiry; dev-mode sessions rejected in production
Control-plane state	`control_plane_state`	`telemetry_enabled` toggle (customer opt-out) and cached non-secret last-valid entitlement JSON for the offline grace window	Licensing/health only by design

8. Catalog: audit

audit_events (migration 0004) is the platform’s tamper-evident record:

Append-only with a per-tenant SHA-256 hash chain (prev_hash → hash, genesis-anchored); verifyAuditChain recomputes the chain and reports the first broken link (packages/server/src/kernel/audit.ts, tested).
Categories: auth, config, governance, scan, evidence_access, review, analyst, action, model, operations.
details_safe_json is restricted to structured references (ids, versions, outcomes, booleans) by the appendAudit contract and typed input — never excerpts, paths, prompts, or names. Honest nuance: unlike the logger (§10), this is a caller contract backed by tests, not a runtime key allowlist.
Read access requires the audit.read capability (held by auditor).
Limit (threat model T15): the chain detects tampering but does not prevent it against an attacker with direct DB write access; external anchoring is a documented follow-up.

Retention: indefinite, append-only. No audit export route exists yet.

9. What is never stored / never leaves the boundary

Item	Mechanism	Where verified
Full source originals	Raw bytes are processed in memory only, bounded to 5 MB per fetch (`CONTENT_MAX_BYTES`, `packages/server/src/scan/pipeline.ts`); never written to disk	Threat model §2, §4.2
Plaintext credentials, tokens, API keys	Env-ref indirection: config stores env-var names, never values (`FMD_GRAPH_CLIENT_SECRET_REF`, `FMD_GRAPH_REMEDIATION_CLIENT_SECRET_REF`, `FMD_AZURE_OPENAI_API_KEY_REF`, `FMD_ANTHROPIC_API_KEY_REF`, `FMD_ENTITLEMENT_SECRET_REF`); `connector_instances.secret_ref` is an opaque reference; the settings API reduces secrets to booleans (`configured`/`resolvable`)	`packages/server/src/kernel/config.ts`, `packages/server/src/admin/settings.ts`
Webhook `clientState`	Derived per subscription via HMAC-SHA256, never stored raw — only `sha256` for audit (`client_state_hash`)	`packages/server/src/notifications/subscriptions.ts`, migration `0007`
Raw message bodies	200-char redacted preview + `sha256` body hash + detector signals only (§5)	`teams/service.ts`, `exchange/service.ts`
Raw AI prompts/responses	`generation_runs` records an input hash only	Migration `0001`
Unbounded permission expansion	Broad audiences stay categorical with counts (§4)	Migration `0002`
Content in telemetry	Strict allowlist — see §11. Additionally, no vendor Control endpoint exists: the payload is built, previewable, and opt-outable, but nothing is transmitted anywhere in this build	`packages/server/src/controlplane/telemetry.ts`, STATUS.md
Content in logs	Default-deny allowlist logger — see §10	`packages/server/src/kernel/log.ts`
Real customer/tenant data in the repo	All fixtures are synthetic (fictional Meridian Grove Holdings)	Threat model T13

10. Log redaction policy

The logger (packages/server/src/kernel/log.ts) enforces redaction structurally, not best-effort:

Only field keys on the SAFE_KEYS allowlist are emitted (correlation/tenant/principal/work-unit ids, stage, status, outcome, error category, HTTP route/method/status, duration, counts, sizeBand, etc.). Unknown keys are dropped and counted (_droppedFields).
String values must match a conservative opaque-token pattern (^[A-Za-z0-9_.:/-]{1,120}$) or are replaced with [redacted] — so free-text content (excerpts, prompts, multi-word names or titles) cannot pass even under an allowlisted key. The pattern alone would admit a single-token value such as a URL or hex hash; the end-to-end scan test below is what verifies those never appear in default logs.
Sizes are logged as bands (lt10k … gte10m), not exact values, via sizeBand.
A ring buffer of the last 500 already-sanitized entries backs the operations UI.
An end-to-end log-scan test asserts that default logs contain no names, paths, URLs, phrases, GUIDs, or hashes (packages/server/src/testing/acceptance.test.ts; unit tests in packages/server/src/kernel/log.test.ts; ADR-0011).

Known residual (threat model T10): HTTP error responses include AppError messages, and runtime crash output to stderr is outside the logger’s control.

11. Fleet telemetry (strict allowlist)

Defined in packages/server/src/controlplane/telemetry.ts (fleet-telemetry-v1), covering licensing + fleet health only. The Zod schema is .strict() — any field not on this list fails validation, so the payload fails closed rather than over-sharing: schemaVersion, deploymentId (a non-reversible sha256 of the internal tenant id, truncated to 24 hex chars — never the tenant name or id), productVersion, collectedAt, entitlementId, entitlementState, scannerCount, activeScannerCount, queueDepthBand (empty/low/moderate/high), deadLetterBand (none/low/elevated), scaleBand (small/medium/large/xlarge — raw asset counts are never reported), telemetryEnabled. A SENSITIVE_FIELD_DENYLIST (name, path, excerpt, content, domain, informationtype, asset, email, principal, tenantid, secret, token, body, subject, count) backs a CI tripwire test so a future edit adding e.g. assetCount fails the build. scannerCount/activeScannerCount are the only deliberately allowed *count fields (operational, not content). Customer controls: control_plane_state.telemetry_enabled is opt-outable at any time, and the UI shows a transparency preview of the exact payload. Honest state: the payload builder and controls are built and tested; no receiving Control endpoint exists, so no telemetry leaves the machine in this release.

12. Retention summary and open items

Class	Behavior in 26.7.15.0
Source observations	Temporal history retained (supersession, `is_current`, validity windows); no deletion job
Content-derived artifacts	Retained with `retention_class` recorded; TTL sweeps not built (ADR-0011 follow-up)
Evidence excerpts / extracted text	Bounded (≤ 240 chars ×3; ≤ 20,000 chars) and purpose-flagged; encrypted at rest only by OS/disk — envelope encryption not built (ADR-0011 follow-up)
Projections (graph, search)	Rebuildable; safe to drop and regenerate
Audit	Append-only, indefinite, hash-chained
Messaging previews/hashes	Retained per message row; superseded content updates detector signals in place
Scratch data	Not applicable in this build: content is processed in memory (5 MB fetch bound); the `data/scratch/` mechanism described in ADR-0011 becomes mandatory when sandboxed rich-format extraction is added (threat model §4.3, T6)

Open retention items a reviewer should track: retention durations and sweeps (not built), envelope encryption of evidence artifacts (not built), a true text-dropping metadata_only profile (config-only today), and audit chain external anchoring (not built). Each is recorded in ADR-0011, the threat model, and the roadmap.

​Data catalog & handling

​How to read this catalog

​1. Evidence access levels

​2. Data-handling profiles

​3. Catalog: content-derived data (highest sensitivity)

​4. Catalog: source observations (metadata acquired from connectors)

​5. Catalog: messaging (Teams / Exchange)

​6. Catalog: governance and human-decision records

​7. Catalog: operational data

​8. Catalog: audit

​9. What is never stored / never leaves the boundary

​10. Log redaction policy

​11. Fleet telemetry (strict allowlist)

​12. Retention summary and open items