CDX tenant test runbook
Status of this document: plan + a real run record (see §0). The mock connector remains the default and the CI-validated path (ADR-0012). In a follow-up session the live Microsoft Graph connector was implemented and validated read-only against the live Contoso CDX tenant — see the run record in §0. The rest of this runbook is the executable plan (identities, sites, change sequence, label mapping, cleanup) for a fuller pilot including writes; only the read-only inventory path has actually been run. Related documents:docs/adr/ADR-0013-live-graph-connector.md— the live connector design and what it does.docs/adr/ADR-0012-cdx-strategy.md— why fixtures first, live CDX deferred.docs/adr/ADR-0008-connector-contract-and-mock-m365.md— connector contract and gating.docs/permissions-manifest.md— every Graph permission requested and its caveats.find_my_data_handoff/FIND_MY_DATA_MVP_DELIVERY_PLAN.md§5 — CDX operating rules and scenario..env.example— the configuration surface referenced throughout.
0. Actual live run record (read-only)
Tenant: Contoso CDX,m365x93338811.onmicrosoft.com, tenant ID
0c39387e-65ed-409c-aba3-d0513fef2d30 (Microsoft Entra ID P2; 34 users, 37 groups).
What was created in the tenant (the only change made):
- One Entra app registration
FMD-CDX-Reader(client ID944ab70e-35f9-4864-b8e4-576655cb0647), application Graph permissionsSites.Read.All+Files.Read.All, admin consent granted, one client secret (expires 2027-01-10; descriptionfmd-cdx-local-readonly). The secret lives only in a gitignoredpackages/server/.env. No SharePoint content, groups, sites, or labels were created or modified — the connector is read-only.
FMD-CDX-Reader app registration (Entra → App registrations →
FMD-CDX-Reader → Delete) to invalidate the secret when finished. Nothing else to undo.
How it was run:
PurviewPlus document libraries):
Bug found and fixed via live data: the ensemble was letting generic evidence (a PII
pattern, “a byte-identical duplicate exists”, location context) push a file over the
candidate threshold for every information type, so duplicated PII-bearing files were tagged
as background-check and compensation and finance at once (60+ spurious candidates). Fixed
by requiring type-discriminating evidence (a curated phrase, filename signal,
confirmed-similarity, or container-vocabulary match) before a type becomes a candidate;
generic signals now only adjust confidence. Result: 4 defensible candidates instead of 60+.
Not yet run against a live tenant: change-sequence deltas (edit/rename/permission/label/
delete), Purview label read/write, Teams/Exchange, and any write action. The sections below
are the plan for those.
1. Current state: what the mock fixture already validates
The mock connector (packages/server/src/connectors/mock/) models a CDX-shaped tenant
(“Meridian Grove”): two SharePoint drives plus one OneDrive, 15 fixture files, folders,
permissions, sensitivity labels, and a mutation API that produces authoritative delta
feeds with Graph semantics (eTag-like sourceVersion vs. cTag-like contentVersion,
opaque delta cursors, simulated 410 Gone resync).
The delivery plan’s §5.2 change sequence maps to passing tests as follows:
Because the mock exercises the same
Connector contract
(packages/server/src/connectors/types.ts) the live adapter must implement, the CDX
session re-runs the same assertions against real Graph behavior rather than inventing
a new test plan.
Honest gap to close before §5 can execute. The live path is gated, not hidden: with
FMD_CONNECTOR_MODE=graph, ConnectorRegistry.getForInstance
(packages/server/src/connectors/registry.ts) raises an explicit BlockedError — first
if credentials are missing, and currently even when they are present, because the concrete
Graph adapter implementation has not been validated against any tenant. The Graph-shaped
contract exists (delta cursors preserved opaquely, eTag/cTag distinction, permission
completeness, 410 resync signal); the next session must finish and wire the
GraphM365Connector implementation behind that contract before running §5. Do not remove
the gate by faking success — the blocked status surfacing in /api/health and the UI is
deliberate (ADR-0012: no fake live integration).
2. Prerequisites and inspection (inspect before creating)
Operating rules (from the delivery plan §5.1): inspect existing tenant state first, reuse existing licenses/labels when sufficient, prefix everything created withFMD-CDX, use
synthetic fictional people and content only, and record every created/modified resource in
the §7 results log.
2.1 Licenses and roles to verify
2.2 Pre-creation inventory
Record answers in the §7 results log before creating anything:- List existing groups matching
FMD-CDX*(a previous session may have left resources):Get-MgGroup -Filter "startsWith(displayName,'FMD-CDX')". - List existing sites matching the prefix:
Get-MgSite -Search "FMD-CDX". - Export the existing label catalog (§6). Decide reuse vs. create; prefer reuse.
- Note the tenant’s default domain and tenant ID — into local
.envonly, never this file.
3. App registrations (two identities, least privilege)
Per the security handoff §7.1, the read/scan identity and the remediation identity are separate app registrations so a scanner compromise does not inherit write capability, and each can be rotated/revoked independently. The server config already models this split (FMD_GRAPH_CLIENT_ID vs. FMD_GRAPH_REMEDIATION_CLIENT_ID in .env.example).
3.1 fmd-cdx-reader (scan/read identity)
Application (app-only) permissions to request:
Known open items to verify in-tenant and record (§7): whether
driveItem: extractSensitivityLabels and permission enumeration work under Sites.Selected
alone, or force a broader grant (Files.Read.All). The security handoff §7.2 requires
documenting any endpoint that forces a broader grant. OneDrive personal sites are not
covered by Sites.Selected; either scope the pilot to the two SharePoint sites (recommended
first pass, mirroring the fixture’s two SP drives) or record the broader permission that
personal-site access would require and get explicit sign-off before requesting it.
Creation (Microsoft Graph PowerShell):
docs/permissions-manifest.md.
3.2 fmd-cdx-remediation (label write identity)
assignSensitivityLabel is protected/metered/asynchronous (reference architecture §
“Label remediation”): it needs enrollment for the protected API, possibly a linked Azure
subscription for metering, and justification for downgrade/removal. Until that enablement
is confirmed, live label write stays behind FMD_FEATURE_PURVIEW_LABEL_WRITE=false and the
action service reports blocked with the exact prerequisite
(packages/server/src/actions/service.ts). The mock path already proves the full
draft → approve → execute-once → verify state machine.
3.3 Sites.Selected per-site grants
After both sites exist (§4.2), grant each app to each site. Two equivalent routes:
Graph REST (works with Graph CLI mgc or Invoke-MgGraphRequest):
Verify with
GET /sites/{siteId}/permissions before proceeding; display the grant status
in the results log (the product requirement is that consent and per-resource grant status
are visible, not assumed).
4. Resource creation plan (all FMD-CDX-prefixed)
4.1 Groups and identities
4.2 Sites
Optional, second pass only:
FMD-CDX HR Operations Team with a channel backed by the HR
site, to confirm the backing SharePoint file canonicalizes rather than duplicates.
4.3 Synthetic file plan (mirrors the fixture corpus)
Create fictional content only. Target 8–15 background-check-like documents plus the adjacent/negative set below — the fixture corpus is the template, so the same detectors, duplicate clusters, and findings can be compared 1:1.
Synthetic-marker rule. Every file carries an obvious synthetic marker — e.g. custom
document property
FMD-Synthetic: true and a footer line FMD-CDX synthetic test document, fictional persons only — placed in document properties or the footer, never in the body
phrases the classifiers key on. The system must not be able to “cheat” by detecting the
marker, and a human finding the file must immediately see it is fake.
ROT limitation. Old created/modified timestamps generally cannot be fabricated through
Microsoft 365 without altering audit history — do not try. Model aged content with old
dates inside the document text and record the limitation in §7, as the fixture does with
its 2018/2019 items.
5. The validation change sequence (live)
5.1 Point FMD at the tenant
Local.env (never committed; .gitignore already excludes .env* except .env.example):
*_SECRET_REF values are references, not secrets; the resolver reads the actual value
from the local secret store. Expect /api/health to report the connector as live and, at
the current state of the code, expect the registry’s explicit blocked error until the
GraphM365Connector implementation is finished and wired (§1). That is the first work item
of the CDX session, not a surprise.
5.2 Execute the sequence
Run each step, then a delta scan, and record outcomes against the same assertions the mock tests make (§1 table). Steps 2–6 can be performed through the SharePoint UI or Graph calls; record which was used.
Where live behavior diverges from mock semantics (e.g. Graph eTag/cTag quirks, permission
enumeration completeness under
Sites.Selected), record the divergence in §7 — the
contract’s completeness field exists precisely because “no broad share observed” is not
“no broad share exists.”
6. Label mapping: replace fixture GUIDs with tenant GUIDs
The build ships a fixture label catalog (packages/server/src/fixtures/labels.ts, catalog_source = 'fixture', synthetic
ms_tenant_id = 8f3f6a2e-0000-4000-a000-cdxfixture000):
These GUIDs are meaningless in a real tenant. Because the Graph label-listing API is still
beta (ADR-0008), the supported production-adjacent path is
Get-Label export/import:
Connect-IPPSSession(Security & Compliance PowerShell, requires a compliance role).Get-Label | Select-Object DisplayName, Guid, Priority | Export-Csv fmd-cdx-labels.csv -NoTypeInformation— write the CSV to a local, gitignored location only.- Map each fixture label name to the closest existing tenant label (reuse before create; most CDX tenants already publish a General/Confidential/Highly Confidential set). Record the mapping in §7.
- Import into
sensitivity_labelswithcatalog_source = 'get_label_import', the realms_tenant_id, and the real GUIDs. The slice implements only the fixture seeding function; the CSV import is a designed path (ADR-0008) the CDX session implements as a small script or one-off SQL against the same columns (id, tenant_id, ms_tenant_id, label_guid, name, description, priority, active, catalog_source, synced_at). - Re-check every place a fixture GUID was planted (file plan §4.3, any drafted actions) against the imported catalog before running sequence step 6.
7. Known blockers and where to record results
Blockers already known from the handoff and this build (delivery plan §11; reference architecture; ADR-0012):
Where results go. For every attempt, record: endpoint, HTTP status / error category,
permissions in effect, setup attempted, and outcome — with secrets and tenant identifiers
redacted. Write results to:
- a results log appended to this file (add a dated
## Results — <date>section listing every resource created/modified, every grant, and every blocker outcome); STATUS.md→ “Mocked / feature-gated” and “Blocked” sections (flip entries as they clear);docs/permissions-manifest.md→ per-permission caveat updates.
8. Cleanup
Delete in this order (reverse of creation, so nothing orphans a grant or leaves an app with standing access). Check each item against the §7 results log rather than this plan — clean up what was actually created.- Any FMD-drafted actions left pending in the local database (cancel; local only).
Sites.Selectedpermission grants on both sites, both apps (DELETE /sites/{siteId}/permissions/{permissionId}orRevoke-PnPAzureADAppSitePermission).- All synthetic files in both document libraries (then empty both site recycle bins so no synthetic screening-like content lingers).
- Optional Team
FMD-CDX HR Operations(if created). - Sites:
FMD-CDX HR Authoritative,FMD-CDX Project Collaboration(delete, then remove from the SharePoint admin deleted-sites list). - Groups:
FMD-CDX HR Owners,FMD-CDX HR Staff,FMD-CDX General Staff(do not delete a pre-existing broad group that was reused — remove only what was created). - Sensitivity labels: none, if existing labels were reused as §6 recommends; delete only labels this session created (unlikely and discouraged).
- App registrations: delete client secrets first, then
fmd-cdx-readerandfmd-cdx-remediationregistrations and their service principals. - Local machine: remove tenant values from
.env, delete the exported label CSV and any local secret-store entries created for this exercise.
Redaction rules
- Never in this repo: tenant IDs, client IDs, site URLs containing the tenant name, secrets, tokens, certificates, real label GUIDs, or synthetic-but-tenant-hosted content.
- Tenant IDs, client IDs, and other non-secret configuration live in the local
.env(gitignored) only. - Credentials/tokens/certificates live in the local secret store, referenced from
.envvia the opaque*_REFvariables; the redacting logger never emits them (log fields are allowlisted, ADR-0011). - Results recorded in §7 use placeholders (
<tenant>,<SITE_ID>) exactly as this document does.