Skip to main content

CDX tenant test runbook

Status of this document: plan + a real run record (see §0). The mock connector remains the default and the CI-validated path (ADR-0012). In a follow-up session the live Microsoft Graph connector was implemented and validated read-only against the live Contoso CDX tenant — see the run record in §0. The rest of this runbook is the executable plan (identities, sites, change sequence, label mapping, cleanup) for a fuller pilot including writes; only the read-only inventory path has actually been run. Related documents:
  • docs/adr/ADR-0013-live-graph-connector.md — the live connector design and what it does.
  • docs/adr/ADR-0012-cdx-strategy.md — why fixtures first, live CDX deferred.
  • docs/adr/ADR-0008-connector-contract-and-mock-m365.md — connector contract and gating.
  • docs/permissions-manifest.md — every Graph permission requested and its caveats.
  • find_my_data_handoff/FIND_MY_DATA_MVP_DELIVERY_PLAN.md §5 — CDX operating rules and scenario.
  • .env.example — the configuration surface referenced throughout.

0. Actual live run record (read-only)

Tenant: Contoso CDX, m365x93338811.onmicrosoft.com, tenant ID 0c39387e-65ed-409c-aba3-d0513fef2d30 (Microsoft Entra ID P2; 34 users, 37 groups). What was created in the tenant (the only change made):
  • One Entra app registration FMD-CDX-Reader (client ID 944ab70e-35f9-4864-b8e4-576655cb0647), application Graph permissions Sites.Read.All + Files.Read.All, admin consent granted, one client secret (expires 2027-01-10; description fmd-cdx-local-readonly). The secret lives only in a gitignored packages/server/.env. No SharePoint content, groups, sites, or labels were created or modified — the connector is read-only.
Cleanup: delete the FMD-CDX-Reader app registration (Entra → App registrations → FMD-CDX-Reader → Delete) to invalidate the secret when finished. Nothing else to undo. How it was run:
Results (read-only scan of the two PurviewPlus document libraries): Bug found and fixed via live data: the ensemble was letting generic evidence (a PII pattern, “a byte-identical duplicate exists”, location context) push a file over the candidate threshold for every information type, so duplicated PII-bearing files were tagged as background-check and compensation and finance at once (60+ spurious candidates). Fixed by requiring type-discriminating evidence (a curated phrase, filename signal, confirmed-similarity, or container-vocabulary match) before a type becomes a candidate; generic signals now only adjust confidence. Result: 4 defensible candidates instead of 60+. Not yet run against a live tenant: change-sequence deltas (edit/rename/permission/label/ delete), Purview label read/write, Teams/Exchange, and any write action. The sections below are the plan for those.

1. Current state: what the mock fixture already validates

The mock connector (packages/server/src/connectors/mock/) models a CDX-shaped tenant (“Meridian Grove”): two SharePoint drives plus one OneDrive, 15 fixture files, folders, permissions, sensitivity labels, and a mutation API that produces authoritative delta feeds with Graph semantics (eTag-like sourceVersion vs. cTag-like contentVersion, opaque delta cursors, simulated 410 Gone resync). The delivery plan’s §5.2 change sequence maps to passing tests as follows: Because the mock exercises the same Connector contract (packages/server/src/connectors/types.ts) the live adapter must implement, the CDX session re-runs the same assertions against real Graph behavior rather than inventing a new test plan. Honest gap to close before §5 can execute. The live path is gated, not hidden: with FMD_CONNECTOR_MODE=graph, ConnectorRegistry.getForInstance (packages/server/src/connectors/registry.ts) raises an explicit BlockedError — first if credentials are missing, and currently even when they are present, because the concrete Graph adapter implementation has not been validated against any tenant. The Graph-shaped contract exists (delta cursors preserved opaquely, eTag/cTag distinction, permission completeness, 410 resync signal); the next session must finish and wire the GraphM365Connector implementation behind that contract before running §5. Do not remove the gate by faking success — the blocked status surfacing in /api/health and the UI is deliberate (ADR-0012: no fake live integration).

2. Prerequisites and inspection (inspect before creating)

Operating rules (from the delivery plan §5.1): inspect existing tenant state first, reuse existing licenses/labels when sufficient, prefix everything created with FMD-CDX, use synthetic fictional people and content only, and record every created/modified resource in the §7 results log.

2.1 Licenses and roles to verify

2.2 Pre-creation inventory

Record answers in the §7 results log before creating anything:
  1. List existing groups matching FMD-CDX* (a previous session may have left resources): Get-MgGroup -Filter "startsWith(displayName,'FMD-CDX')".
  2. List existing sites matching the prefix: Get-MgSite -Search "FMD-CDX".
  3. Export the existing label catalog (§6). Decide reuse vs. create; prefer reuse.
  4. Note the tenant’s default domain and tenant ID — into local .env only, never this file.

3. App registrations (two identities, least privilege)

Per the security handoff §7.1, the read/scan identity and the remediation identity are separate app registrations so a scanner compromise does not inherit write capability, and each can be rotated/revoked independently. The server config already models this split (FMD_GRAPH_CLIENT_ID vs. FMD_GRAPH_REMEDIATION_CLIENT_ID in .env.example).

3.1 fmd-cdx-reader (scan/read identity)

Application (app-only) permissions to request: Known open items to verify in-tenant and record (§7): whether driveItem: extractSensitivityLabels and permission enumeration work under Sites.Selected alone, or force a broader grant (Files.Read.All). The security handoff §7.2 requires documenting any endpoint that forces a broader grant. OneDrive personal sites are not covered by Sites.Selected; either scope the pilot to the two SharePoint sites (recommended first pass, mirroring the fixture’s two SP drives) or record the broader permission that personal-site access would require and get explicit sign-off before requesting it. Creation (Microsoft Graph PowerShell):
Assign the Graph application permissions in the Entra admin center (App registrations → fmd-cdx-reader → API permissions → Microsoft Graph → Application permissions), then click Grant admin consent. Record the exact permission set granted in §7 and in docs/permissions-manifest.md.

3.2 fmd-cdx-remediation (label write identity)

assignSensitivityLabel is protected/metered/asynchronous (reference architecture § “Label remediation”): it needs enrollment for the protected API, possibly a linked Azure subscription for metering, and justification for downgrade/removal. Until that enablement is confirmed, live label write stays behind FMD_FEATURE_PURVIEW_LABEL_WRITE=false and the action service reports blocked with the exact prerequisite (packages/server/src/actions/service.ts). The mock path already proves the full draft → approve → execute-once → verify state machine.

3.3 Sites.Selected per-site grants

After both sites exist (§4.2), grant each app to each site. Two equivalent routes: Graph REST (works with Graph CLI mgc or Invoke-MgGraphRequest):
PnP PowerShell alternative:
Grants to create (record each grant ID in §7 for cleanup): Verify with GET /sites/{siteId}/permissions before proceeding; display the grant status in the results log (the product requirement is that consent and per-resource grant status are visible, not assumed).

4. Resource creation plan (all FMD-CDX-prefixed)

4.1 Groups and identities

4.2 Sites

Optional, second pass only: FMD-CDX HR Operations Team with a channel backed by the HR site, to confirm the backing SharePoint file canonicalizes rather than duplicates.

4.3 Synthetic file plan (mirrors the fixture corpus)

Create fictional content only. Target 8–15 background-check-like documents plus the adjacent/negative set below — the fixture corpus is the template, so the same detectors, duplicate clusters, and findings can be compared 1:1. Synthetic-marker rule. Every file carries an obvious synthetic marker — e.g. custom document property FMD-Synthetic: true and a footer line FMD-CDX synthetic test document, fictional persons only — placed in document properties or the footer, never in the body phrases the classifiers key on. The system must not be able to “cheat” by detecting the marker, and a human finding the file must immediately see it is fake. ROT limitation. Old created/modified timestamps generally cannot be fabricated through Microsoft 365 without altering audit history — do not try. Model aged content with old dates inside the document text and record the limitation in §7, as the fixture does with its 2018/2019 items.

5. The validation change sequence (live)

5.1 Point FMD at the tenant

Local .env (never committed; .gitignore already excludes .env* except .env.example):
The *_SECRET_REF values are references, not secrets; the resolver reads the actual value from the local secret store. Expect /api/health to report the connector as live and, at the current state of the code, expect the registry’s explicit blocked error until the GraphM365Connector implementation is finished and wired (§1). That is the first work item of the CDX session, not a surprise.

5.2 Execute the sequence

Run each step, then a delta scan, and record outcomes against the same assertions the mock tests make (§1 table). Steps 2–6 can be performed through the SharePoint UI or Graph calls; record which was used. Where live behavior diverges from mock semantics (e.g. Graph eTag/cTag quirks, permission enumeration completeness under Sites.Selected), record the divergence in §7 — the contract’s completeness field exists precisely because “no broad share observed” is not “no broad share exists.”

6. Label mapping: replace fixture GUIDs with tenant GUIDs

The build ships a fixture label catalog (packages/server/src/fixtures/labels.ts, catalog_source = 'fixture', synthetic ms_tenant_id = 8f3f6a2e-0000-4000-a000-cdxfixture000): These GUIDs are meaningless in a real tenant. Because the Graph label-listing API is still beta (ADR-0008), the supported production-adjacent path is Get-Label export/import:
  1. Connect-IPPSSession (Security & Compliance PowerShell, requires a compliance role).
  2. Get-Label | Select-Object DisplayName, Guid, Priority | Export-Csv fmd-cdx-labels.csv -NoTypeInformation — write the CSV to a local, gitignored location only.
  3. Map each fixture label name to the closest existing tenant label (reuse before create; most CDX tenants already publish a General/Confidential/Highly Confidential set). Record the mapping in §7.
  4. Import into sensitivity_labels with catalog_source = 'get_label_import', the real ms_tenant_id, and the real GUIDs. The slice implements only the fixture seeding function; the CSV import is a designed path (ADR-0008) the CDX session implements as a small script or one-off SQL against the same columns (id, tenant_id, ms_tenant_id, label_guid, name, description, priority, active, catalog_source, synced_at).
  5. Re-check every place a fixture GUID was planted (file plan §4.3, any drafted actions) against the imported catalog before running sequence step 6.
The real tenant GUIDs live in the local database and local notes only — they are tenant-identifying configuration and stay out of the repo per the redaction rules below.

7. Known blockers and where to record results

Blockers already known from the handoff and this build (delivery plan §11; reference architecture; ADR-0012): Where results go. For every attempt, record: endpoint, HTTP status / error category, permissions in effect, setup attempted, and outcome — with secrets and tenant identifiers redacted. Write results to:
  • a results log appended to this file (add a dated ## Results — <date> section listing every resource created/modified, every grant, and every blocker outcome);
  • STATUS.md → “Mocked / feature-gated” and “Blocked” sections (flip entries as they clear);
  • docs/permissions-manifest.md → per-permission caveat updates.

8. Cleanup

Delete in this order (reverse of creation, so nothing orphans a grant or leaves an app with standing access). Check each item against the §7 results log rather than this plan — clean up what was actually created.
  1. Any FMD-drafted actions left pending in the local database (cancel; local only).
  2. Sites.Selected permission grants on both sites, both apps (DELETE /sites/{siteId}/permissions/{permissionId} or Revoke-PnPAzureADAppSitePermission).
  3. All synthetic files in both document libraries (then empty both site recycle bins so no synthetic screening-like content lingers).
  4. Optional Team FMD-CDX HR Operations (if created).
  5. Sites: FMD-CDX HR Authoritative, FMD-CDX Project Collaboration (delete, then remove from the SharePoint admin deleted-sites list).
  6. Groups: FMD-CDX HR Owners, FMD-CDX HR Staff, FMD-CDX General Staff (do not delete a pre-existing broad group that was reused — remove only what was created).
  7. Sensitivity labels: none, if existing labels were reused as §6 recommends; delete only labels this session created (unlikely and discouraged).
  8. App registrations: delete client secrets first, then fmd-cdx-reader and fmd-cdx-remediation registrations and their service principals.
  9. Local machine: remove tenant values from .env, delete the exported label CSV and any local secret-store entries created for this exercise.
Record the completed cleanup (with anything intentionally left behind and why) in the §7 results log.

Redaction rules

  • Never in this repo: tenant IDs, client IDs, site URLs containing the tenant name, secrets, tokens, certificates, real label GUIDs, or synthetic-but-tenant-hosted content.
  • Tenant IDs, client IDs, and other non-secret configuration live in the local .env (gitignored) only.
  • Credentials/tokens/certificates live in the local secret store, referenced from .env via the opaque *_REF variables; the redacting logger never emits them (log fields are allowlisted, ADR-0011).
  • Results recorded in §7 use placeholders (<tenant>, <SITE_ID>) exactly as this document does.