diff --git a/.scratch/poll-pool-optimization/readmix.md b/.scratch/poll-pool-optimization/readmix.md index cc7eb79d..9e78ec5d 100644 --- a/.scratch/poll-pool-optimization/readmix.md +++ b/.scratch/poll-pool-optimization/readmix.md @@ -124,3 +124,30 @@ Median allocated bytes 2,297,159,936 → 2,095,265,576 (-8.8%). Post-GC heap bas Final Go gate: 68 known failures, no new failures. The first gate captured an incorrect legacy fixture expecting a newly inserted run to lack persisted capture directories; fixed the test to explicitly simulate a historical missing field, then reran the complete gate successfully. Tests cover name fallback, capture precedence, duplicated/missing run IDs, legacy artifact fallback, isolated Mongo projection with a 1 MiB run log, and full record availability. No new index, migration, long-lived cache, resource change, push, PR or deployment. Remaining work: query execution-plan/scan measurements on representative large history, complete failure/recovery coverage, and the planned long three-phase capacity comparison. Current release/capacity status stays pending. + +## Mine author-history filtering, 2026-10-29 + +Local explain probe: `POLL_POOL_EXPLAIN=1 go test ./internal/store -run ^TestHumanPoolExplainLocal$ -v -count=1`. Hardcoded loopback Mongo, unique synthetic database dropped on exit, 40,000 rows and existing registered indexes only. QC dual stages, both acceptance stages, available/mine/reviewed, all/single-project scopes. Reports under `local-data/test-results/pool-explain/` omit query bodies/customer content. + +The prepackage all-project mine query examined 4,000 records for an empty result; applying the existing HTTP mine author rule examined zero. QC raw candidate counts differ by design: the legacy query returns authors' ended rows before the HTTP filter; only end-to-end/post-filter equivalence establishes correctness. Some first-page QC explain times did not improve; therefore do not infer universal speed from individual plans or force a hint. No indexes added. + +The new server-only MinePoolOnly applies `(assigned to actor) OR (authored by actor AND quality_review AND ready/submitted)` before pagination. It leaves AnnotationView active/submitted and other ActorUserID uses unchanged. API permission checks, self-review prohibition and cursor ordering remain. Raw fetches can finish earlier because irrelevant history is removed, but visible task sets and the returned-item cursor contract are retained. Memory/Mongo stage/status/ownership/project/view matrices and seven-row page traversal match legacy post-filter results. API differential fixture: 121 candidates → 1 with byte-identical response. Full Go gate has 68 known failures, no new ones; race tests passed. + +Read-mix comparison: `34905aac` → `ca58c258`, both summary mode, initialized authorizations, --run-payload-kib 256 --author-history 1000. Twenty review/acceptance actors each have 1,000 ended authored records (20,000 added history rows), same fixtures per variant. Three alternating-order pairs, 5s warmup + 30s measurement, output `local-data/test-results/mine-author-history`. Each side performed 2,565 total requests, equivalent task sets, zero HTTP errors and stable writes. + +Median of three per-round p95 values (ms): + +| Endpoint | Before | After | Reduction | +| --- | ---: | ---: | ---: | +| Quality all-project mine | 1483.006 | 287.423 | 80.6% | +| Quality single-project mine | 301.730 | 213.263 | 29.3% | +| Legacy acceptance all-project mine (empty) | 1118.602 | 65.794 | 94.1% | +| Final acceptance all-project mine | 1139.727 | 214.825 | 81.2% | +| Final acceptance single-project mine | 265.390 | 160.989 | 39.3% | +| Quality all-project available | 1145.282 | 363.263 | 68.3% | +| Final acceptance all-project available | 885.284 | 305.377 | 65.5% | +| Annotation buffer | 376.121 | 238.267 | 36.7% | + +Available-pool and buffer code did not change in this step; their improvement reflects reduced contention from mine/history reads in this mixed workload. Do not claim their standalone query was rewritten. Mongo replies ~141.316 MB→106.097 MB (-24.9%); commands 10,124→9,788 (-3.3%). Median allocated bytes ~2.500 GB→2.089 GB (-16.4%); post-GC heap baseline 4,698,520 / 4,689,208 / 4,681,336 bytes and candidate 4,664,648 / 4,692,496 / 4,692,680, effectively unchanged. + +This history-heavy short fixture now exceeds the 50% latency improvement target for all-project available/mine, with p95 below 2s. It is NOT the planned full capacity acceptance: no three 5-minute/30-minute phases, Pod CPU/throttling/RSS, ingress/bandwidth, media, submission/recovery workload, or live index verification. No production writes, index migration, resource changes, push, PR or deployment. Release/capacity acceptance remains pending. diff --git a/.scratch/poll-pool-optimization/work.md b/.scratch/poll-pool-optimization/work.md index ca11ec2e..1de0761f 100644 --- a/.scratch/poll-pool-optimization/work.md +++ b/.scratch/poll-pool-optimization/work.md @@ -88,3 +88,9 @@ Risk: mixed-version response compatibility, permissions, in-flight workflow reco - Added server-only MinePoolOnly, enabled only by human-items pool=mine. Dedicated AnnotationView queries and all other actor queries retain their original behavior. Existing API permission/visibility checks remain in place after query filtering. - HTTP before/after comparison reads 121 versus 1 candidates with identical response. A first fixture used a stage absent from its four-stage batch; corrected to the V5.8 stage definitions before accepting evidence. Memory/Mongo matrix covers all task states/ownership variants, project/stage scope, dedicated views and multi-page cursor traversal. Targeted regressions passed. - Added explicit --author-history fixture option to the local read-mix runner; existing zero-history comparisons remain reproducible. Final Go gate and same-fixture performance results pending. + +### Mine/history validation result + +- Full Go baseline gate: no new failures (68 known). Race-enabled mine/pool/pagination tests passed, including local Mongo equivalence. +- Three short same-fixture 81-actor comparisons with 20,000 additional ended author records: each version 2,565 requests, task sets equal, zero errors/stable writes. Median p95 quality/all/mine 1483.006→287.423ms (-80.6%); final-acceptance/all/mine 1139.727→214.825ms (-81.2%). Mixed-workload available pools also improved by 68.3% / 65.5%. No sustained post-GC heap growth; allocations decreased. +- Mongo replies -24.9%, commands -3.3%. Detailed query-plan evidence, scope and limitations are in readmix.md. This satisfies the short history fixture's latency thresholds, not formal long-run or production capacity acceptance. No new index, push, PR, deployment or resource change.