Skip to main content
Status: preliminary estimate, 2026-07-14. The design ceiling from the reference architecture is ~100,000 directory identities and ~500 million file-like assets per organization. The platform has NOT been validated at this scale. Everything below is arithmetic over the actual schema plus small-scale measurements on one developer machine. Numbers at 1M+ assets are extrapolations; numbers at 500M are naive linear extrapolations recorded so they can be falsified by the validation ladder (§7), not claims. Related: ADR-0003 (single SQLite behind role interfaces), ADR-0007 (stage caching), ADR-0011 (minimized-evidence retention), ADR-0012 (CDX strategy), and the architecture reference.

​
1. Model variables

Baselines are assumptions to be replaced by pilot measurements; none of them has been observed in a real tenant. Byte figures include SQLite row and index overhead as measured, not just payload.

​
What one asset costs in the actual schema

From packages/server/migrations/0002 and 0003, one fully processed asset writes: Not modeled per asset: audit events (per governed action, ~400 B hash-chained row), activity aggregates (per asset per window, not_enabled in the current release), and messages (not modeled here).

​
2. Daily steady-state load

Per day, after bootstrap:
  • changed items: c · A → new asset_versions + permission/label observations (~1 KB/item retained history before retention sweeps);
  • content-changed subset (assume ~⅓ of changes touch content, i.e. cTag changes): new fingerprints + features for ~0.33 · c · A · r… in practice the stage-cache keys (ADR-0007) guarantee rename/permission/label-only changes cost no content fetch, hash, or extraction — this is covered by tests;
  • model calls: m × content-changed items. At 500M assets, 2%/day, ~⅓ content changes, m=1–2 → 3–7M model calls/day. This, not storage, is the dominant live operating cost and must be bounded by per-tenant budgets and workload priorities. The default exercised path uses the deterministic mock (m=0); Azure OpenAI/Anthropic adapters exist but are config-gated.
  • Azure Storage scopes only (full_listing delta, ADR-0030): each change scan re-lists the scope’s full namespace (one REST page per 5,000 blobs) and diffs it against one indexed scan of the scope’s active assets — O(N_scope) per delta run, but pure listing + DB read; only actual adds/updates/deletes become work units. Graph scopes are unaffected (server-side delta). Size blob-scope delta cadence to namespace size: a 1M-blob container costs ~200 list requests and a single-digit-seconds diff per change scan.
  • SMB scopes (same full_listing mechanism, ADR-0031): a change scan is a full directory walk (readdir+lstat per entry) over the mount — I/O-bound on the share/network, so size delta cadence to tree size and expect a CIFS-mounted walk to be slower than a local one. Change detection keys on mtime:size; a same-second, same-size in-place rewrite is not detected until either changes (filesystem-metadata limit, stated in ADR-0031).
  • SQL scopes (structured, ADR-0032): a table is an asset, so a database is sized by its table count, not its row count — inventory is one catalog listing plus a stats query per table, and each table profiled is 1 + min(columns, 100) bounded read queries (one sample query per column). Cost scales with tables × columns, not rows, and is trivial next to file corpora. Change detection keys on modify_date:rowCount (exact count), so any schema change or INSERT/DELETE re-profiles. The residual is a pure in-place UPDATE (no DDL, no row-count change): because SQL’s modify_date is DDL-only and the content stage is content-addressed, an ordinary rescan will NOT re-read such a table — it re-profiles only on a schema change, a row-count move, or a deployment upgrade that bumps the extraction/config version (ADR-0032). Plan structured-freshness expectations around that limit.

​
3. Worked example: 1M assets

Assumptions: r=30%, d≥10:1, one full scan pass of telemetry retained, B_text≈5 KB typical. Total: roughly 8–15 GB. Bytes are not the problem at 1M assets. The constraint is the single SQLite writer (§5) and bootstrap wall-clock, not storage.

​
4. Worked example: 500M assets (naive linear extrapolation)

Explicitly unvalidated; recorded to size the target architecture, not to claim it. Total order: ~2.5–5 TB across stores. Feasible for Postgres + Blob + a search service with partitioning; the hard problems are bootstrap throughput against provider rate limits, model-call budget, permission-set dedup holding up in hostile tenants (unique direct shares defeat d), and index growth — all of which are exactly what the validation ladder (§7) must measure before any 500M claim.

​
5. Why single-writer SQLite tops out (~1–5M assets, single node)

The default deployment intentionally runs one SQLite file (WAL) behind role interfaces (ADR-0003). Where that stops working:
  • One writer. WAL allows concurrent readers but exactly one writer. The API process, scan worker, projections, audit appends, and queue lease/heartbeat updates all serialize on the same writer (busy_timeout 5 s). Loadgen’s ~220k rows/s (§8) is tight batched transactions with no readers; the real pipeline’s effective sustained rate is far lower and degrades as interactive load appears.
  • Queue on the same writer. Work-unit leasing and heartbeats compete with observation inserts — the first visible symptom of contention.
  • Operational limits, not size limits. 1M assets ≈ 3–15 GB is comfortable for SQLite; ~5M ≈ 15–60 GB is still storable. But there is no online replication, no HA, backup is whole-file, checkpoint pauses grow with write volume, and query p95s hold only while the working set fits one node’s page cache.
  • No horizontal read scaling for many concurrent domain-owner queries.
So: comfortable at small scale, plausible to low single-digit millions of assets on one strong node with careful batching, structurally wrong beyond that.

​
6. Migration triggers per role interface (ADR-0003)

Each role moves independently; the code already speaks the role interface, so substitution is additive. Cross-cutting at any migration point: keep every row tenant-scoped (already enforced), keep projections rebuildable with watermarks, and keep stage telemetry (stage_attempts) on a TTL or in an operational log store rather than the system of record.

​
7. Staged validation ladder

​
8. First measurements (small scale, single developer machine)

Harness: bun run loadgen [assetCount] — isolated packages/server/data/loadgen.db, fresh migrations, same INSERT patterns as the pipeline’s observe/permissions stages (including upsertPermissionSet dedup, imported from the pipeline itself), fingerprint clusters for ~30% of assets, no network, no content bytes. Machine: developer laptop (Apple M4, 16 GB, macOS, Bun 1.3.14). Numbers below are small-scale harness measurements, not a scale validation. They measure raw batched insert cost and one bounded query shape — no connector latency, no extraction/inference stages, no concurrent readers or writers. Observations:
  • Measured ~2.6–2.8 KB/asset is the basis for B_meta ≈ 1.2 KB and B_fp ≈ 5.5 KB in §1 (4 canonical rows per asset + 19 fingerprint rows for 30% of assets; averages ~290 B/row including index overhead).
  • Dedup ratio improves with asset count as expected (the distinct-set pool saturates); real-tenant inherited permissions should dedup much harder, and hostile cases (per-item unique shares) much worse. Measure in stage 2.
  • Index discipline is load-bearing: the container-listing query ran at ~14 ms p95 until the asset_versions join was tenant-scoped so it could use the partial index idx_asset_versions_current (tenant_id, asset_id) WHERE is_current = 1; with tenant scoping it is ~0.2–0.5 ms. Every query must carry tenant_id — the repository convention (ADR-0003) is also the performance-correct one.
  • Bytes/asset here excludes features, evidence, assertions, stage telemetry, and object-store text — the loadgen number is a floor, not the full per-asset cost modeled in §1.

​
9. Phase D measurements — multi-million on Postgres (2026-07-18)

The load harness is now dual-backend (FMD_DB_BACKEND=postgres + FMD_DB_URL) with multi-row batched inserts, in-memory permission dedup, and the FINAL Phase D schema (packed band0..3 columns, promoted hot-facet columns, and — behind --embeddings — pgvector). Measured on a 4-CPU / 5.77 GiB dev VM (Postgres 16
  • pgvector 0.8.5, Bun 1.3.14); Postgres was tuned lean (shared_buffers=128MB) to survive the write-heavy load on the small VM, which makes cache-sensitive query p95s pessimistic vs a production-sized buffer pool.
Observations:
  • The at-scale run caught a real regression. D3’s packed-band retrieval, as a 4-way WHERE band0=? OR band1=? OR …, made SQLite’s planner scan every simhash64 row — 2.1 s p95 at 1M. Rewritten as a UNION of four single-band probes (each hits its own partial idx_fp_band* index on both engines) it is 20 ms (†, 100×), recall-identical. Postgres was already fast via BitmapOr. Small-scale tests never showed this — the load test is exactly what surfaced it.
  • Dedup keeps improving with scale (274:1 → 327:1), as §2 assumed.
  • 10M is a box limit here, not a product limit. A single-process 10M synthetic run OOMs the 5.77 GiB VM at ~2.6M (PG backend killed by the OOM killer under sustained write pressure). Per-asset cost (~1.95 KB) and ingest rate (~7.1k/s) are flat from 100k→2.6M, so the storage/ingest envelope extrapolates linearly to 10M+; a true 10M+ run needs a box with more RAM. This is called out honestly rather than claimed.
  • ‡ container-listing / facet p95s are inflated by the deliberately tiny shared_buffers (the 3.7 GB working set does not fit a 128 MB cache) and, for the facet filter, a benchmark query that omits the domain-id predicate the real /api/assets path carries; the band-lookup and pgvector numbers use the shipped query shapes and reflect the indexes doing their job.