Allocation query performance validation¶
This supplies reproducible evidence for ledger item 9 in STATUS.md, subject to the acceptance contract. It does not enable allocation, change an environment database or define a new latency SLA. Representative deployment qualification remains distinct from a synthetic run on a shared development host.
Smallest useful measurement slice¶
AllocationQueryPerformanceTests is an opt-in test in
src/libs/project-management/SyRF.ProjectManagement.Mongo.Data.Tests. It exercises the actual
repository next-study, assigned-study page and administrator progress methods, and separately
measures derived-plan construction. No production application code or query design changes
are included in this measurement slice.
The test creates a new isolated MongoDB 8 Testcontainers instance. It accepts no database
connection string and does not read deployment connection settings. It skips before creating
the fixture unless explicitly enabled; its fixture is disposed in finally and Testcontainers
resource reaping remains enabled. No shared Docker cleanup or environment database operation
is part of the command.
Reproducible commands¶
Run from this PR's worktree root, not the main checkout. Docker and the repository's .NET 10 toolchain are required. Ordinary test runs skip this benchmark; existing CI routing is unchanged.
SYRF_ALLOCATION_PERF=1 \
SYRF_ALLOCATION_PERF_STUDIES=10000 \
SYRF_ALLOCATION_PERF_REVIEWERS=10 \
SYRF_ALLOCATION_PERF_ITERATIONS=10 \
dotnet test src/libs/project-management/SyRF.ProjectManagement.Mongo.Data.Tests/SyRF.ProjectManagement.Mongo.Data.Tests.csproj \
--configuration Release --filter FullyQualifiedName~AllocationQueryPerformanceTests \
-m:2 -v:quiet --logger 'console;verbosity=detailed'
Use the same command with 100,000 studies and 50 reviewers for the larger case. For a quick
instrumentation check use 100 studies, two reviewers and three iterations. Inputs are bounded:
100–100,000 studies, 2–50 reviewers and 3–30 timed iterations. These are harness resource bounds,
not newly imposed production feature limits. Run cases sequentially to avoid self-contention.
Opt-in runs read the assembly's actual MSBuild configuration and reject every configuration
other than Release (including Debug, Local, Docker and missing metadata) before starting MongoDB, so development-build timings cannot
accidentally be published as representative application performance.
Corpus and assertions¶
- Deterministic study/reviewer/stage identifiers, two reviews per study and valid equal shares (including deterministic remainder basis points).
- The requested number of primary-project studies plus 10% separate-project noise. Each study carries a 4,096-character compressible synthetic abstract. This is document-size ballast, not a claim to reproduce production annotation payloads or compression ratios.
- Saved work on 35% of studies: 30% have one completed or incomplete ordinary session and 5% have two completed sessions at capacity. Five percent are excluded; five percent have a reservation. Owner/status sequences are decoupled so the measured reviewer has both completed and incomplete work and reservations. These percentages describe the primary-project corpus.
- The actual repository index initializer creates the current configured study index catalogue; the result prints index names/keys. The harness does not invent an alternative index design.
- Independently computed assigned/completed/in-progress totals must match every administrator row. The measured reviewer's counts must match the administrator row and its page must contain at most 25 studies. Existing allocation Mongo regression tests remain the broader semantic authority for failure, exclusion, reservation, legacy and work-in-progress-limit behavior.
- Initial cases have no maximum in-progress limit; the final named limit case sets it to one, asserts no new work is available and permits only incomplete saved-work links.
Metrics and interpretation¶
Each operation emits JSON timing samples, first-call time, warm median, nearest-rank p95/max and mean process allocation bytes. The median averages the middle pair for even sample counts. There is one first call and one additional untimed warm-up before the timed samples.
The first call is not cold-storage latency: seeding has already warmed database/OS caches, and driver/JIT warm-up may be included. Ten samples provide descriptive measurements, not a stable tail-latency estimate or an SLA result. Process allocation counts include background driver/runtime work; Mongo resident memory is a process snapshot, not per-query peak memory.
Database profiling is disabled for timed samples and enabled only for a separate diagnostic execution in the disposable database. Its plan summary, examined document/key counts, response bytes and diagnostic duration are recorded. Entries are matched to the exact collection and execution time window; truncated large command bodies cannot accidentally match an earlier query. Missing or ambiguous diagnostics fail the run instead of reporting stale evidence.
Reported version, processor count, corpus size, data/index bytes and index catalogue accompany the results. Record the Git commit, host/container constraints and concurrent host activity alongside the captured log. Repository-only timings exclude HTTP/authentication, configuration validation, network latency and browser rendering; plan construction is reported separately.
Shared-host observations — 2026-09-05¶
The corrected Release harness passed its 10,000-study / 10-reviewer case against repository
commit d14f66c046e7bed4f5ddc0d2cc3a27b0e38742ff. The captured
raw observations include all samples, exact index
definitions and the harness SHA-256. Earlier Debug/instrumentation runs are discarded.
PR review subsequently hardened only the pre-fixture build-configuration guard to inspect
actual assembly metadata rather than infer Release from the absence of DEBUG. The captured
runs were explicitly built with --configuration Release; their original harness hashes are
retained. Corpus generation, measured operations and timing/diagnostic code are unchanged.
The run used .NET 10.0.0 and MongoDB 8.0.28 on Juniper (Intel Xeon Platinum 8260, 24 cores / 48 logical processors), with no benchmark-specific CPU/memory cap. Concurrent development and CI were active; observed host load averages were approximately 65/58/40. This is a noisy shared host, not a controlled production-equivalent environment. The corpus occupied 63,679,990 data bytes and 2,695,168 index bytes, including the separate-project noise.
| Operation | Warm median (ms) | Sample p95/max (ms) | Diagnostic documents examined |
|---|---|---|---|
| Derived plan construction | 33.7 | 42.2 | Not a database query |
| Legacy next study | 85.6 | 105.6 | 10,000 |
| Allocated next study | 53.6 | 74.0 | 1,988 |
| Reviewer page and counts | 321.9 | 369.6 | 10,000 |
| Administrator progress | 1,423.6 | 1,777.1 | 10,000 |
| Stale reviewer page | 235.5 | 244.2 | 10,000 |
| Next study with empty buckets | 78.2 | 107.7 | 10,000 |
| Reviewer at in-progress limit | 258.0 | 292.8 | 10,000 |
The allocated next-study query used the project/bucket index. Reviewer/admin reads and the empty-bucket query scanned the project's study documents. Plan construction allocated about 30.8 MB per sample in the test process; this is not retained memory. These observations identify admin aggregation, empty-work scans and derived-plan allocation as follow-up candidates, not proof that a particular optimization or replacement storage model is necessary.
The 100,000-study / 50-reviewer case also passed against the same source and harness; its raw observations retain all ten timed samples. It occupied 636,909,890 data bytes and 24,813,568 index bytes. Shared-host load increased during this run (an observed snapshot reached approximately 121/86/66), so the two cases are not a controlled scaling comparison.
| Operation | Warm median (ms) | Sample p95/max (ms) | Diagnostic documents examined |
|---|---|---|---|
| Derived plan construction | 381.2 | 538.9 | Not a database query |
| Legacy next study | 1,223.5 | 1,362.0 | 100,000 |
| Allocated next study | 81.1 | 93.4 | 3,997 |
| Reviewer page and counts | 2,879.5 | 3,395.5 | 100,000 |
| Administrator progress | 56,427.1 | 87,934.6 | 100,000 |
| Stale reviewer page | 3,443.9 | 3,651.0 | 100,000 |
| Next study with empty buckets | 562.8 | 667.3 | 100,000 |
| Reviewer at in-progress limit | 2,543.7 | 3,560.4 | 100,000 |
Administrator progress is a material rollout concern: its separate profiled execution took 98 seconds and scanned the entire project. Plan construction allocated approximately 125 MB per sample in the test process. These are observations under substantial contention, not production predictions. There were no test failures or Mongo query errors in either case.
The local go/no-go conclusion is insufficient evidence for production activation. Repeat under production-like resources with an agreed applicable latency budget; investigate the administrator aggregation first if these costs persist. Preserve the count/eligibility regressions when changing queries. Empty-work short-circuiting and derived-plan memoization are subsequent measured candidates, not silently implemented optimizations.
Budget and correction — 2026-09-07¶
The repository owner set the applicable budget on 2026-09-07: administrator allocation-progress
read warm p95 <= 2,000 ms at 100,000 studies / 50 reviewers, measured on this harness profile.
Reviewer-facing my-studies paging must not regress and is reported at the same profiles. This is
the first declared numeric budget for this read; the 2026-09-05 section above deliberately declared
none.
Diagnosis¶
GetAllocationProgressAsync built the whole project's study stream once and then added four to
five $facet branches per reviewer (assigned, completed, inProgress, available, and
limitInProgress when the in-progress limit applies). Every branch is an independent
$match + $count over the entire facet input, so 50 reviewers meant 200-250 full re-evaluations
of the project's studies inside one aggregation. The profiled plan (IXSCAN { ProjectId: 1 },
100,000 documents examined) hides this: docsExamined counts the single index scan and fetch, not
the per-branch re-processing, which is why the recorded evidence showed a normal-looking plan
against a 98-second query.
The arithmetic matches the measurements. The reviewer page runs the same prefix with six branches and took 2,084.6 ms warm median at 100,000 studies; the administrator read with 250 branches took 50,901.2 ms in the same run - about 200 ms per additional branch either way.
Change¶
The administrator read now classifies each study once in a single $project (its bucket's
owning reviewers, whether it is excluded, whether the stage is already sufficiently allocated, and
the reviewers holding ordinary sessions or slot reservations), expands it only to the reviewers it
can possibly affect, and $groups by reviewer. The number of passes over the corpus no longer
depends on the reviewer count.
Semantics are unchanged. Every constant in the new expressions is produced by the same serializer
the rendered query filters use (legacy GUID binary subtype, enum representation, nullable agreement
ratio), legacy tallies keep the TotalAllocatedSessionCount -> NumberOfCandidateSessions fallback
that the atomic capacity guard uses, a stale or out-of-domain WorkloadShareBucket still resolves to
no owner, and the existing regression tests assert that every administrator row equals the same
reviewer's own page counts. The single-reviewer my-studies page keeps its existing facet pipeline:
its branch count is already fixed, and a measured 10,000-study comparison showed the grouped shape
was slower there.
Method¶
Both branches were measured on the same host on 2026-09-07, back to back, with the harness,
fixture generator and commands unchanged from the 2026-09-05 protocol above. Baseline runs used a
detached worktree at ddf9371dde4a8088c289f1ae2ccf9204271a9681 (the branch point); corrected runs
used c63b432dbb65ee9ff4370f502a8fb10f7cf76ca0. The harness file is byte-identical in both
(SHA-256 19e1f6a3...). Each run was started only while the one-minute load average was below the
host's 48 logical processors; the observed load averages are recorded in each evidence file. The
host remained shared with concurrent development and CI, so these are still shared-host
observations, not environment qualification.
Raw observations, including all ten timed samples per operation, the exact index catalogue and the profiler entries: baseline 10k/10, grouped 10k/10, baseline 100k/50, grouped 100k/50. The 2026-09-05 files are retained unchanged.
Before and after¶
100,000 studies / 50 reviewers (warm median / warm p95, ms):
| Operation | Before | After | Change |
|---|---|---|---|
| Administrator progress | 50,901.2 / 56,100.1 | 2,843.7 / 3,475.7 | 17.9x / 16.1x faster |
| Reviewer page and counts | 2,084.6 / 2,639.1 | 2,022.8 / 2,103.5 | no regression |
| Stale reviewer page | 1,941.7 / 2,036.8 | 2,348.5 / 3,047.1 | within run-to-run noise |
| Reviewer at in-progress limit | 2,105.1 / 2,828.8 | 2,114.2 / 2,964.8 | unchanged |
| Allocated next study | 75.4 / 91.1 | 76.0 / 91.8 | unchanged |
| Legacy next study | 742.8 / 994.0 | 635.2 / 958.5 | unchanged |
| Next study, empty buckets | 401.4 / 411.7 | 417.5 / 439.2 | unchanged |
| Derived plan construction | 153.0 / 164.6 | 145.8 / 199.3 | unchanged (not a query) |
10,000 studies / 10 reviewers (warm median / warm p95, ms):
| Operation | Before | After | Change |
|---|---|---|---|
| Administrator progress | 1,145.9 / 1,260.5 | 340.6 / 457.1 | 3.4x / 2.8x faster |
| Reviewer page and counts | 245.5 / 290.3 | 268.7 / 313.3 | within run-to-run noise |
| Stale reviewer page | 229.8 / 248.3 | 265.0 / 341.0 | within run-to-run noise |
| Reviewer at in-progress limit | 317.6 / 387.3 | 257.5 / 288.0 | unchanged |
First call versus warm, and the separately profiled diagnostic execution, for the administrator read (the first call is not cold storage latency - seeding has already warmed the caches):
| Profile | First call (ms) | Diagnostic query (ms) | Plan | Documents / keys examined | Response bytes | Mean process allocation |
|---|---|---|---|---|---|---|
| 100k/50 before | 56,492.1 | 51,278 | IXSCAN { ProjectId: 1 } |
100,000 / 100,000 | 13,554 | 21.9 MB |
| 100k/50 after | 3,510.2 | 2,928 | IXSCAN { ProjectId: 1 } |
100,000 / 100,000 | 5,936 | 1.8 MB |
| 10k/10 before | 1,276.3 | 1,067 | IXSCAN { ProjectId: 1 } |
10,000 / 10,000 | 2,834 | 8.1 MB |
| 10k/10 after | 735.8 | 350 | IXSCAN { ProjectId: 1 } |
10,000 / 10,000 | 1,296 | 1.7 MB |
The plan and examined counts are unchanged by design: the correction removes repeated in-pipeline work, not the index selection. The client-side allocation per administrator call falls by about 12x at 100,000 studies because the driver no longer materializes 250 facet result arrays.
Result against the budget: not met¶
Warm p95 at 100,000 studies / 50 reviewers is 3,475.7 ms against a 2,000 ms budget - 1,475.7 ms over, or 1.74x the budget. Warm median (2,843.7 ms) is also above it. The budget is not lowered and no threshold is declared passed. Reviewer paging did not regress at either profile.
Remaining risk and the next step¶
The residual cost is per-document, not per-reviewer. Per-study cost is about 26 microseconds at
both 10,000 and 100,000 studies, so the read is now linear in corpus size and flat in reviewer
count, but it still fetches every study document in the project. A plain indexed find over the same
corpus (next-empty-buckets, 417.5 ms for 100,000 documents) puts the scan floor at about 4
microseconds per document, so roughly 2.4 of the 2.8 seconds is aggregation-expression evaluation
over 100,000 fetched documents. Removing pipeline stages does not help: an intermediate variant with
one extra $project measured 2,940.6 / 3,418.5 ms, statistically the same.
The next step, and the only one measured evidence supports, is to stop fetching documents that cannot change the answer:
- Take the bucket-derived part of
assignedandavailablefrom a covered per-bucket count on the existing{ ProjectId: 1, WorkloadShareBucket: 1 }index, with no document fetch. - Restrict the full classification to studies that can deviate from that baseline - those that are excluded, or that hold a tally, session or slot reservation for the stage - and correct the histogram with it.
Step 2 needs the $or to be an index union. The activity arm is already covered by the existing
{ ProjectId: 1, "ExtractionInfo.SessionTallies.StageId": 1, ... } index; the exclusion arm would
need a new pmStudy index on { ProjectId: 1, "ScreeningInfo.InclusionInfo.SufficientlyExcluded": 1 }.
On this fixture that subset is roughly 40-45% of the corpus, which projects to about 1.5 s and would
meet the budget - but it degrades back towards the present cost for a project in which every study
has been reviewed, and it adds a production index build on the largest collection plus count
reconciliation across two reads. That is a separately reviewed change, not an extension of this one.
Until it lands, the administrator progress read is bounded and no longer scales with reviewer count, but the 100,000-study / 50-reviewer profile remains outside the declared budget, and this remains local diagnostic evidence on a shared host rather than environment qualification or activation authority.
Release boundary¶
The synthetic measurement is reproducible local evidence, not completion of production-scale qualification or permission to enable allocation. No latency budget has been declared passed.
Before environment activation, repeat with an appropriately representative study/reviewer/ saved-session distribution and production-like resource/index conditions. Compare allocated and legacy next-study behavior, and include assigned/admin reads, stale and empty work cases. The source handoff specifies no numeric latency budget, so do not declare a fabricated SLA passed. Record applicable existing service budgets or the remaining budget decision, any query timeouts/resource failures, and a rollout go/no-go conclusion supported by measurements.
Memoization or query changes should respond to measured bottlenecks. Materialized allocation plans, a new dashboard and history-aware rebalancing are not prerequisites for collecting this evidence and are not silently included in this PR. Keep the feature flag default-off while required lifecycle, authorization or rollout evidence remains outstanding.