> ## Documentation Index > Fetch the complete documentation index at: https://docs.findmydata.io/llms.txt > Use this file to discover all available pages before exploring further. # Capacity model # Capacity model (preliminary) **Status:** preliminary estimate, 2026-07-14. The design ceiling from the reference architecture (§14.1) is \~100,000 directory identities and \~500 million file-like assets per organization. **The prototype has NOT validated this scale.** Everything below is arithmetic over the actual slice schema plus small slice-scale measurements on one developer machine. Numbers at 1M+ assets are extrapolations; numbers at 500M are naive linear extrapolations recorded so they can be falsified by the validation ladder (§7), not claims. Related: ADR-0003 (single SQLite behind role interfaces), ADR-0007 (stage caching), ADR-0011 (minimized-evidence retention), ADR-0012 (CDX strategy), handoff reference architecture §14. ## 1. Model variables Per handoff ARCH §14.1. Baselines are assumptions to be replaced by pilot measurements; none of them has been observed in a real tenant. | Symbol | Variable | Baseline assumption | Source | | -------- | ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | `A` | asset count (file-like) | 1M and 500M scenarios below | design ceiling | | `B_meta` | canonical metadata bytes/asset (asset + current version + location + permission observation, incl. indexes) | \~1.2 KB | measured proxy, §8 | | `B_fp` | fingerprint bytes per content-retrieved asset (2× sha256 + simhash64 + 16 band rows) | \~5.5 KB | measured proxy, §8 | | `B_feat` | feature + evidence + assertion bytes per scored asset | \~1.5–4 KB | schema estimate, §2 | | `B_text` | retained bounded extracted text per retrieved asset (object store) | \~5 KB typical, 20 KB cap | `DEFAULT_LIMITS.maxOutputChars` = 20,000 | | `c` | daily change rate (fraction of assets with a new `sourceVersion`/day) | 2%/day | assumption; measure in pilot | | `r` | content-retrieval percentage (assets whose content is fetched, hashed, extracted) | 30% | assumption; policy-dependent | | `d` | permission-set dedup ratio (permission observations per distinct set) | 10:1 to 100:1 | measured 7:1–23:1 on synthetic data, §8; inherited site permissions in real tenants should dedup far harder | | `m` | model calls per content-changed item | 0 (mock, exercised path) / 1–2 (live ensemble, gated) | ADR-0005 | | `q` | concurrent interactive queries (domain owners on aggregates + bounded neighborhoods) | tens | product target | Byte figures include SQLite row and index overhead as measured, not just payload. ### What one asset costs in the actual schema From `packages/server/migrations/0002` and `0003`, one fully processed asset writes: | Row(s) | Table | When | Approx. effective bytes (row + index share) | | ----------------- | -------------------------------------------------------------- | --------------------------------------------- | ---------------------------------------------- | | 1 | `physical_assets` | always | \~300 B | | 1 per version | `asset_versions` | always; +1 per change | \~400 B | | 1 per move/rename | `asset_locations` | always; history kept (`valid_to`) | \~450 B (path strings dominate) | | 1 per observation | `asset_permission_obs` | always; superseded rows kept (`is_current=0`) | \~350 B | | amortized | `permission_sets` + `permission_grants` | only for distinct sets (dedup by digest) | \~(300 + 150·grants)/`d` per asset | | 3 | `fingerprints` | content-retrieved only (`r`) | \~450 B each | | 16 | `fingerprint_bands` | content-retrieved only | \~120 B each (\~1.9 KB total) | | \~3 | `feature_artifacts` (keyphrases, language, structure/excerpts) | content-retrieved only | \~0.3–1 KB each incl. `value_json` | | 2–6 | `evidence` | scored assets | \~350 B each | | 1–3 | `assertions` | scored assets | \~400 B each (explanation text) | | \~5 | `stage_attempts` | every pipeline pass | \~150 B each — telemetry, TTL-swept (ADR-0011) | | \~4 | `stage_cache` | every pipeline pass | \~150 B each | | 1 artifact | object store (`extracted_text_bounded`) | content-retrieved with supported MIME | `B_text` | Not modeled per asset: audit events (per governed action, \~400 B hash-chained row), activity aggregates (per asset per window, `not_enabled` in the slice), messages (out of slice scope). ## 2. Daily steady-state load Per day, after bootstrap: * changed items: `c · A` → new `asset_versions` + permission/label observations (\~1 KB/item retained history before retention sweeps); * content-changed subset (assume \~⅓ of changes touch content, i.e. `cTag` changes): new fingerprints + features for `~0.33 · c · A · r`... in practice the stage-cache keys (ADR-0007) guarantee rename/permission/label-only changes cost **no** content fetch, hash, or extraction — this is tested in the slice; * model calls: `m ×` content-changed items. At 500M assets, 2%/day, \~⅓ content changes, `m`=1–2 → **3–7M model calls/day**. This, not storage, is the dominant live operating cost and must be bounded by per-tenant budgets and workload priorities (ARCH §14.2). The exercised slice path uses the deterministic mock (`m`=0); Azure OpenAI/Anthropic adapters exist but are config-gated. ## 3. Worked example: 1M assets Assumptions: `r`=30%, `d`≥10:1, one full scan pass of telemetry retained, `B_text`≈5 KB typical. | Store role (ADR-0003) | Contents | Estimate | | ----------------------------------------- | ------------------------------------------------------------------- | ---------------------------------------- | | Relational system of record | canonical metadata `A·B_meta` | \~1.2 GB | | Relational (fingerprints) | `r·A·B_fp` (band rows dominate) | \~1.7 GB | | Relational (features/evidence/assertions) | `r·A·B_feat` + light rows for the rest | \~1–2 GB | | Relational (stage telemetry, pre-sweep) | \~9 rows × \~150 B × `A` | \~1.4 GB (TTL-swept) | | Object store | `r·A·B_text` | \~1.5 GB typical (6 GB at the 20 KB cap) | | Search projection | FTS over names/paths/keyphrases | \~0.3–0.5 GB | | Graph projection | asset/container/principal/cluster nodes + edges | \~0.5–1 GB | | Queue | transient; peak = one inventory campaign ≈ `A` work units × \~300 B | \~0.3 GB peak | | Audit | per governed action, not per asset | negligible at rest | **Total: roughly 8–15 GB.** Bytes are not the problem at 1M assets. The constraint is the single SQLite writer (§5) and bootstrap wall-clock, not storage. ## 4. Worked example: 500M assets (naive linear extrapolation) Explicitly unvalidated; recorded to size the target architecture, not to claim it. | Store role | Estimate | Notes | | ----------------------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | Relational canonical metadata | \~600 GB | requires partitioning (tenant, scope) and read replicas | | Fingerprints | \~800 GB, **\~2.9B band rows** | 16 band rows/asset is untenable as a plain relational table at this scale; pack bands or move to a dedicated index (§6) | | Features/evidence/assertions | \~500–800 GB | retention classes matter; ADR-0011 minimized profile assumed | | Object store (bounded text) | \~750 GB typical, \~3 TB at cap | Blob with lifecycle policies | | Search projection | \~150–250 GB | rebuildable from system of record | | Daily change (2%/day) | 10M changed items/day; \~4–10 GB/day retained history; 3–7M model calls/day | budgets required | | Bootstrap | 500M observations | dominated by provider throttling, not store writes — ARCH §14.2 points to Data Connect-style bulk paths for M365 at this size | **Total order: \~2.5–5 TB across stores.** Feasible for Postgres + Blob + a search service with partitioning; the hard problems are bootstrap throughput against provider rate limits, model-call budget, permission-set dedup holding up in hostile tenants (unique direct shares defeat `d`), and index growth — all of which are exactly what the validation ladder (§7) must measure before any 500M claim. ## 5. Why single-writer SQLite tops out (\~1–5M assets, single node) The slice intentionally runs one SQLite file (WAL) behind role interfaces (ADR-0003). Where that stops working: * **One writer.** WAL allows concurrent readers but exactly one writer. The API process, scan worker, projections, audit appends, and queue lease/heartbeat updates all serialize on the same writer (`busy_timeout` 5 s). Loadgen's \~220k rows/s (§8) is tight batched transactions with no readers; the real pipeline's effective sustained rate is far lower and degrades as interactive load appears. * **Queue on the same writer.** Work-unit leasing and heartbeats compete with observation inserts — the first visible symptom of contention. * **Operational limits, not size limits.** 1M assets ≈ 3–15 GB is comfortable for SQLite; \~5M ≈ 15–60 GB is still storable. But there is no online replication, no HA, backup is whole-file, checkpoint pauses grow with write volume, and query p95s hold only while the working set fits one node's page cache. * **No horizontal read scaling** for many concurrent domain-owner queries. So: comfortable at slice scale, plausible to low single-digit millions of assets on one strong node with careful batching, structurally wrong beyond that. ## 6. Migration triggers per role interface (ADR-0003) Each role moves independently; the code already speaks the role interface, so substitution is additive. | Role | Interface | Trigger (measure, don't guess) | Target | | ------------------------------ | -------------------- | -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | Transactional system of record | module repositories | writer contention between worker and API; asset count approaching \~1–5M; need for online backup/replication/HA | Azure Database for PostgreSQL; partition `asset_*` tables by tenant + scope | | Object/evidence artifacts | `ObjectStore` | artifact bytes beyond tens of GB; retention/lifecycle policies; envelope encryption requirement (ADR-0011 follow-up) | Azure Blob + lifecycle policies | | Search projection | `SearchIndex` | FTS5 rebuild time exceeds maintenance windows; faceting + concurrent query load | Azure AI Search / OpenSearch (rebuildable from system of record, watermarked) | | Graph projection | `GraphRepository` | bounded-neighborhood p95 > 3 s despite indexes | dedicated graph store only if traversal performance demands (per ADR-0003, not by default) | | Event/work substrate | `WorkQueue` + outbox | lease/heartbeat writes competing with observations; multiple worker nodes; campaigns beyond \~10⁶ work units | Azure Service Bus / Storage Queues | | Vector/fingerprint candidates | `FingerprintIndex` | band-row count (16 per fingerprinted asset; \~2.9B rows at the 500M/30% point); embeddings going live | pgvector / dedicated ANN; pack band rows | Cross-cutting at any migration point: keep every row tenant-scoped (already enforced), keep projections rebuildable with watermarks, and keep stage telemetry (`stage_attempts`) on a TTL or in an operational log store rather than the system of record. ## 7. Staged validation ladder (ARCH §14.3) | Stage | What | Status in this run | | ----- | ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1 | Deterministic local fixtures; hundreds–thousands of synthetic assets | **Done.** 15-file mock M365 fixture exercised by the full pipeline + tests; loadgen harness runs at 5k–20k synthetic assets (§8) | | 2 | CDX pilot over bounded sites/drives with controlled mutations | **Not done this run.** Live Graph adapter is config-gated and unvalidated against any tenant; setup runbook exists (ADR-0012, `docs/cdx-test-runbook.md`) | | 3 | Load harness at 1M canonical metadata observations, no content | **Harness built** (`packages/server/src/loadgen/loadgen-cli.ts`, `bun run loadgen 1000000`); not yet executed at 1M | | 4 | 10M+ persistence/index/queue projection tests from generated events | Future; requires the Postgres/queue targets from §6 to be meaningful | | 5 | Production pilot with measured API throttling and change rates | Future | | 6 | Progressive scale and failure testing before publishing any 500M claim | Future | ## 8. First measurements (slice scale, this dev machine) Harness: `bun run loadgen [assetCount]` — isolated `packages/server/data/loadgen.db`, fresh migrations, same INSERT patterns as the pipeline's observe/permissions stages (including `upsertPermissionSet` dedup, imported from the pipeline itself), fingerprint clusters for \~30% of assets, no network, no content bytes. Machine: developer laptop (Apple M4, 16 GB, macOS, Bun 1.3.14). Numbers below are **slice-scale harness measurements**, not a scale validation. They measure raw batched insert cost and one bounded query shape — no connector latency, no extraction/inference stages, no concurrent readers or writers. | Metric | 5,000 assets | 20,000 assets | | -------------------------------------------------------------------------- | ------------------------------- | --------------------------------- | | Rows written | 48,253 | 193,677 | | Wall clock (inserts) | 0.2 s | 0.9 s | | Rows/sec (batched, single writer, no readers) | \~244,000 | \~219,000 | | Assets/sec | \~25,000 | \~23,000 | | DB file size (after WAL checkpoint) | 13.2 MiB | 49.5 MiB | | Bytes/asset on disk (canonical metadata + 30% fingerprints, incl. indexes) | 2,771 | 2,597 | | Permission-set dedup | 668 distinct / 5,000 obs (≈7:1) | 885 distinct / 20,000 obs (≈23:1) | | "Current assets in container" p95 (50 random containers, LIMIT 200) | 0.24 ms | 0.48 ms | Observations: * Measured \~2.6–2.8 KB/asset is the basis for `B_meta` ≈ 1.2 KB and `B_fp` ≈ 5.5 KB in §1 (4 canonical rows per asset + 19 fingerprint rows for 30% of assets; averages \~290 B/row including index overhead). * Dedup ratio improves with asset count as expected (the distinct-set pool saturates); real-tenant inherited permissions should dedup much harder, and hostile cases (per-item unique shares) much worse. Measure in stage 2. * Index discipline is load-bearing: the container-listing query ran at \~14 ms p95 until the `asset_versions` join was tenant-scoped so it could use the partial index `idx_asset_versions_current (tenant_id, asset_id) WHERE is_current = 1`; with tenant scoping it is \~0.2–0.5 ms. Every query must carry `tenant_id` — the repository convention (ADR-0003) is also the performance-correct one. * Bytes/asset here excludes features, evidence, assertions, stage telemetry, and object-store text — the loadgen number is a floor, not the full per-asset cost modeled in §1.