23 KiB
Raw Permalink Blame History

Reduce polling cost and reviewer pool latency

Goal: Reduce polling payloads and pool latency without changing workflow or permissions. Agent: Codex Base: codex/reviewer-entry-guards @ 80bdcee9, deployed version selected by user; not yet in origin/dev. Scope: continuation and work-buffer polling, review pool display reads, compatibility and load validation. Out of scope: production configuration, migrations and release without separate authorization. Risk: mixed-version response compatibility, permissions, in-flight workflow recovery and database load.

2026-10-28 additional read optimization

  • Buffer ledger validation now deduplicates active batch IDs and fetches only scope, workflow and stage fields in one Mongo query. Missing references are cached only within the ledger attempt; exceptional slots use the same projected fallback. Capacity and CAS logic are unchanged.
  • Display enrichment skips full annotation-task reads when capture/payload has already resolved the name. Legacy task fallback and request-local reuse remain covered.
  • Local isolated Mongo regression passed, including deduplicated query count, foreign-organization rejection, missing batch handling, stable poll ledger version and capacity update.
  • Targeted API regressions passed: continuation, annotation buffer, display cache and legacy display fallback. The varied deterministic fixture keeps the same task count; legacy/summary gzip sizes were 12,579/331 bytes (97.4% reduction). This is synthetic response evidence, not production capacity acceptance.
  • Remaining: frontend summary integration and task projection, broader display batching, exact pool query measurements and the approved 81-user repeated soak comparison. No push, PR or deployment performed.

2026-10-28 summary polling integration

  • The annotation page now polls view=summary into an independent snapshot type. Legacy server responses and capacity PUT responses are explicitly converted. The 3-second interval is unchanged.
  • Existing full records retain their subject/draft payload; newly observed task IDs are hydrated once for sidebar compatibility. Explicit work entry still loads a full task. Scope/epoch/mutation fences are rechecked after hydration.
  • Added an explicit store summary method with an inclusion projection for state, ownership, slice phase and processing status. Claim reservation and default/PUT paths retain full behavior. Existing name-cache misses deliberately use original item inputs for historical nested capture names; no new cross-request cache was added.
  • Verification: 27 targeted frontend tests passed (entry, reconciliation, refill and mixed-version conversion); TypeScript/architecture passed; isolated Mongo summary/ledger tests and targeted API tests passed. Stable polls retain ledger versions and exclude large payload and processing-job input.
  • Go baseline gate passed with 68 known failures, no additions. Frontend gate still fails two source guards: review-target-interaction.test.mjs :: source guard: resident editor binds the selected target with a tier label and scope-cycle-inline-editor.test.mjs :: source guard: resident inline editor promotes into edit state on focus. Both reproduce unchanged at HEAD 8cdd2d79 in a separate archived baseline. The baseline full gate also had a temporary-copy local-stack/version failure and a hung devboot test that was terminated; these are not claimed as passes. Initial sandbox frontend run also needed termination of devboot before rerunning outside the sandbox.
  • Global format check reports four untouched files: ReviewHistorySummary.tsx, WorkforceStatisticsPanel.tsx, useAnnotationOutputData.ts, useReviewContinuation.ts. Modified frontend files were formatted; git diff --check passes. No baseline files were altered to hide failures.
  • Pending: browser end-to-end validation of this integration, batch/projected display fallback refinement, actual pool p95 and the specified repeated 81-user soak. This is not a release/capacity acceptance. No push, PR or deployment.

2026-10-28 browser and concurrent HTTP verification

  • Added node tests/browser/annotation-buffer-summary.mjs: real application in Edge/Vite, every API intercepted with synthetic data and external hosts blocked. Passed real 3-second summary polling without repeated full-item hydration, old-server full-response compatibility, failed poll/reconnect, manual task selection retained after polling, and an unsaved action-description draft retained across two polls. No unexpected allocation/submission or page errors. Media playback and actual backend submission are outside this fixture.
  • node tests/browser/review-continuation-errors.mjs: all 12 quality/acceptance compact/legacy renew, error, empty and control-loss cases passed.
  • Extended TestAnnotationBufferMongoHTTPConcurrencyAndReceipt: mixed summary/legacy GETs alongside 16 concurrent claims per round, five rounds in a unique local Mongo database. Capacity remains two; settled summary/full state is equivalent; interrupted receipt recovery and a freshly constructed handler retain results. Passed normally and with Go -race.
  • -race also passed TestMongoAnnotationWorkBufferPollIsReadOnly and TestReviewContinuationCompactRenewKeepsLegacyContract. Targeted review read-count/permission-revocation, display and continuation regressions passed.
  • Browser artifacts: local-data/test-results/annotation-buffer-summary-browser/{result.json,passed.png,page.txt}. Initial test attempts exposed an assertion before React settled and an incorrect child-element click target; these were corrected to await the state and click the actual action block. Final run completed without source changes/HMR during execution.
  • No full 81-user mixed-role harness was found in the existing scripts/tests. These small synthetic concurrent tests are functional evidence only, not pool p95, production bandwidth or CPU/memory capacity proof. The baseline/stage1/stage2 dataset, workload runner and three repeated 5-minute-warmup/30-minute-steady comparisons remain required. Existing global frontend/format gate debt from the preceding entry remains unresolved. No release or production mutation.

2026-10-28 isolated 81-actor read-mix comparison

  • Added local-only opt-in Go load harness and a Python baseline/candidate runner. Two projects, three suppliers, 60 annotation + 12 quality + 8 acceptance + 1 admin; stage-correct nonempty pool fixtures and administrator records; identical gzip settings, task-ID equivalence and zero stable-write checks. The runner archives the baseline into a temporary directory, never changes branches, and preserves output directories.
  • Three short pairs completed: 5s warmup/30s measurement, 855 requests per variant per round, 2,565 per variant total; zero HTTP errors/writes and matching returned task sets in every pair.
  • Synthetic buffer gzip fell 99.2%; Mongo commands fell 18.7%, reply bytes 18.8%. Median of per-round p95: buffer 444.725→463.224ms (no improvement demonstrated), quality all-project available 610.761→549.923ms, prepackage acceptance all-project available 608.618→491.782ms. Neither pool reached the 50% relative reduction goal. Details and interpretation limits are in readmix.md.
  • This is a handler/read-mix smoke comparison, not capacity acceptance: no network ingress/auth transport, continuation renew, video, submission writes, production-size history, long resource stability or three-phase 30-minute soak. No Pod/bandwidth/configuration change, push or deployment.

2026-10-28 member lookup diagnosis and scoped read

  • Added test-only per-endpoint/collection/command Mongo counters and optional CPU profile. The legacy fixture showed 1,200 member-list reads (28.7 MB) for 600 buffer requests; supplier authorization seed derivation accounted for one of the two lists per request.
  • Corrected interpretation: previous v2 load runs did not initialize supplier-project authorizations; normal Mongo migration does. Those are retained as legacy-path evidence, not normal deployed-state proof. The comparison runner now supports both states and defaults to initialized.
  • Independent waste exists in loadAnnotationMembershipsForUser: it loaded a complete member list and immediately filtered by user ID. Added ListProjectMembersForUser retaining organization compatibility, all matching rows and ordering; account, supplier and grant checks remain unchanged and fresh.
  • Regression first failed with broad=2/scoped=0, then passed with scoped reads and immediate disabled-account rejection. Memory/Mongo equivalence tests preserve historical duplicate rows, project/organization boundaries and empty results. Buffer/review permission regression passed. No indexes, migrations or cross-request permission cache added.

Scoped-member validation result

  • Go gate completed with no new failures (68 known). Both initialized and legacy authorization pairs returned equivalent task sets with zero HTTP errors/stable writes.
  • Corrected a second comparison confounder: the runner originally forced baseline legacy/full mode. Added --baseline-mode summary and repeated the initialized comparison with both versions using summaries.
  • In that isolated pair, 855 requests per side; buffer member reply bytes 14,336,400→247,110 (98.3% reduction), total Mongo reply volume 178.92→162.02 MB, command count unchanged at 9,380. Buffer p95 348.547→386.892ms; do not claim latency improvement from this pair. Details in readmix.md, output member-scope-summary-only.
  • Remaining latency/capacity acceptance is unresolved. No production changes, push or PR.

2026-10-28 pool candidate payload reads

  • Reproduced a three-row page reading 3,246,610 bytes of task payload from a 120-row synthetic backlog. Added an internal candidate projection retaining all metadata while omitting subject payload and processing input; final wire responses keep the existing list projection.
  • Extracted the existing visibility/claim checks without changing their rules. Filter metadata first, bulk hydrate only rows that can reach this page, preserve raw-page order/cursors/fetch limits. Compare all candidate metadata rather than only version; changed/missing selected rows trigger a fresh original full-page read before final filtering. No cross-request cache or index.
  • Targeted API tests passed: payload-read bound, legacy-strategy response equality, assignment/project changes without version bump, version changes, pagination, reviewer/self-review/acceptance/history regressions. Memory and local isolated Mongo candidate fields/pagination equivalence passed. Full Go gate and controlled performance comparison pending at this commit.

Candidate projection validation

  • Full Go gate: no new failures (68 known). Three same-mode 81-actor short pairs: 2,565 requests each, matching task sets, zero HTTP errors/stable writes. Mongo reply volume -34.8%; commands +2.2%; median per-round pool p95 improved 13.4% (quality all-project) and 26.3% (final acceptance all-project). Still below the 50% relative target; production/long-soak acceptance pending.
  • Added post-GC memory diagnostics outside measurement. One additional pair showed post-GC heap 4.62→4.34 MB and reduced allocations; end-of-run heap alone was GC-phase sensitive. Evidence and limitations in readmix.md. No expansion of deployment authorization.
  • Final go test -race ./internal/api ./internal/store -run ^TestHumanPool passed, with isolated Mongo projection tests enabled.

2026-10-29 run display metadata projection

  • Found a coverage gap in the earlier read-mix fixture: human items had no RunID, so it did not exercise full execution-history reads in display enrichment. A regression first failed because an already-resolved capture still loaded the complete run.
  • Added independent HumanRunDisplay records and batch ID projection (names, task ID, persisted capture directories only), primed once for the returned human-item page. Preserve request-local missing-ID caching and the exact precedence of capture record → item metadata → annotation task → run. Only legacy run-directory fallback fetches full execution data; no authority checks rely on display metadata.
  • Targeted display/pool/pagination tests pass, including batch deduplication, missing runs and explicit missing historical header fields. Current UpsertRun auto-populates capture directories, so the old-record test deliberately removes that header field rather than assuming a newly inserted record is legacy. Memory/local Mongo projection equivalence passed with a 1 MiB execution log.
  • Added opt-in --run-payload-kib to the local comparison runner; zero preserves the earlier fixture, nonzero adds identical shared run records and task links to both variants. Full gate and performance evidence pending. No production mutation, index, resource, push or deployment change.

Run display validation

  • Final Go baseline gate passed with no new failures (68 known). Three local 81-actor short comparisons using identical 256 KiB run-history fixtures passed task-set equivalence and zero errors/stable writes.
  • Run queries 408→204, run reply volume ~107.219 MB→0.080 MB; total Mongo reply bytes -50.2%. Median per-round p95 improved 23.8% for quality/all and 31.4% for final acceptance/all. Allocations decreased; post-GC heap showed no sustained increase. Details/limitations in readmix.md.
  • This result does not establish production capacity or the planned 50% latency target. No push/deployment or resource change.
  • Final race-enabled display/run-projection/pool tests passed in API and store, with isolated local Mongo enabled.

2026-10-29 mine author-history query pushdown

  • Added an opt-in loopback-only explain diagnostic with 40,000 synthetic rows, current registered indexes, QC dual-stage and both acceptance stages, all/single-project available/mine/reviewed queries. Reports only plan stages/index names/counts/times, not document bodies or credentials.
  • Found mine queries loading authored historical records the HTTP handler always discards. Empty prepackage mine examined 4,000 documents in the fixture. Narrowing the author arm to ready/submitted quality_review preserves the existing rule; assigned tasks remain unrestricted by this added condition. No new index or hint.
  • Added server-only MinePoolOnly, enabled only by human-items pool=mine. Dedicated AnnotationView queries and all other actor queries retain their original behavior. Existing API permission/visibility checks remain in place after query filtering.
  • HTTP before/after comparison reads 121 versus 1 candidates with identical response. A first fixture used a stage absent from its four-stage batch; corrected to the V5.8 stage definitions before accepting evidence. Memory/Mongo matrix covers all task states/ownership variants, project/stage scope, dedicated views and multi-page cursor traversal. Targeted regressions passed.
  • Added explicit --author-history fixture option to the local read-mix runner; existing zero-history comparisons remain reproducible. Final Go gate and same-fixture performance results pending.

Mine/history validation result

  • Full Go baseline gate: no new failures (68 known). Race-enabled mine/pool/pagination tests passed, including local Mongo equivalence.
  • Three short same-fixture 81-actor comparisons with 20,000 additional ended author records: each version 2,565 requests, task sets equal, zero errors/stable writes. Median p95 quality/all/mine 1483.006→287.423ms (-80.6%); final-acceptance/all/mine 1139.727→214.825ms (-81.2%). Mixed-workload available pools also improved by 68.3% / 65.5%. No sustained post-GC heap growth; allocations decreased.
  • Mongo replies -24.9%, commands -3.3%. Detailed query-plan evidence, scope and limitations are in readmix.md. This satisfies the short history fixture's latency thresholds, not formal long-run or production capacity acceptance. No new index, push, PR, deployment or resource change.

2026-10-29 long-run validation preparation

  • Extended the read-mix runner to archive/pin every variant (including candidate), optionally include stage one, rotate execution order, checkpoint each run, validate completed evidence on --resume, preserve failed artifacts, and exclude concurrent runners with a filesystem lock. Harness hash, exact commits, timing and fixture parameters are part of the resume manifest.
  • Added 15-second resource JSONL and final time series: completed/retained requests, errors, heap/GC/goroutines and OS process CPU time. CPU includes the handler and in-process load client/compression, not Mongo or Pod throttling; heap includes retained benchmark samples. Warmup errors fail the run. These are not Pod capacity metrics.
  • Added failure-path HTTP regressions for candidate, detail and display read timeouts: no partial task success, identical retry/fresh-handler response, unchanged persisted task status/version/assignee. Passed under -race. Local Mongo buffer concurrent claim/receipt recovery, compact/legacy continuation and stale-page regressions passed. Full Go gate: 68 known failures, no new failures.
  • Three-version tooling pilot (80bdcee9 / a2eac93e / 3fe37a92), 1s warmup + 3s measurement passed; completed-run resume reused all three reports. Four Python tool tests cover invalid/stale evidence, atomic checkpoints and output locks. A race-enabled resource pilot crossed 15s sampling boundaries with zero errors. These are tooling checks, not performance comparisons.
  • Found a long-run fixture defect before starting: previously ended authored rows were accepted but unreleased, so the real 60s release sweeper could process them. Fixture v3 now creates dispatched synthetic partitions and links the historical rows; the sweeper remains enabled. A 1s/75s check with 20k history rows and 256 KiB run logs completed 2,095 requests with zero errors/stable writes. Do not conflate the v2 short evidence with v3 long-run data.
  • Next run: 80bdcee9 legacy, a2eac93e stage-one summary, pinned final summary; 300s warmup/1800s steady, three rounds, same v3 data/resources/compression. Approximately 5h15m excluding setup. This completes the long local read portion only; renew/media/submission/ingress and Pod resource acceptance remain outside this harness.

First long-run attempt and diagnostic replay

  • poll-pool-long-v3 stopped during baseline 80bdcee9 warmup: two non-200 responses after roughly three/four minutes. The warmup guard correctly rejected the run before accepting steady-state evidence. Its original telemetry did not retain endpoint/status details, so neither root cause nor whether these were timeouts is established. Preserve this failure; do not describe it as fixed or discard it from acceptance history.
  • Added capped, sanitized failure metadata (synthetic actor, endpoint, HTTP status, elapsed time, known error class and code; no response bodies/queries/credentials). Replayed both baseline and candidate for 1s warmup + 260s steady with the same v3 fixture: each 7,090 requests, zero errors/writes and matching task sets. Failure did not reproduce; original cause remains unresolved. Output long-run-error-diagnosis.
  • Added a completed-run analyzer that separates local-read thresholds from capacity acceptance, checks every available-pool scope per round, and reports CPU/heap ratios. Full actor/endpoint schedule validation now rejects duplicates/missing requests even if totals match. Five runner tests and two analyzer tests pass. Short or partial runs can never be labeled long-run acceptance; capacity_accepted always remains false for this harness.
  • Repeat long comparison with diagnostic harness pinned in a new output directory; the failed attempt's manifest and artifacts remain intact. Application behavior is unchanged by these test/tool additions.

Instrumented long run: first baseline complete

  • Baseline round one completed 1,800 steady seconds with 48,750 requests, zero HTTP errors and zero Mongo writes. Available-pool p95 ranges 491.447–822.886 ms. Stage-one round one is running; no candidate long-run comparison is available yet. This does not resolve the original unclassified warmup failures.
  • Closed an analysis-only evidence gap: loading reports now reuses the runner's strict revision/mode/fixture/timing/request/resource validation before computing acceptance metrics. Tests reject reports from another commit or configuration and missing resource evidence. All eight Python runner/analyzer tests passed. No application or active load harness changes.

2026-10-29 post-release 50 Mbps latency diagnosis and repair

  • Public HTTP read probes reproduced workforce waiting before first byte: QC 2.50–2.85s/~550 compressed bytes; admin 15.97–18.90s/~8.8KB; annotator 4.31s then 28ms. Download about 2–3ms. This identifies response-preparation delay, not egress-body saturation for these calls. API pods sampled at 3–10m CPU/51–188MiB; a snapshot is not throttling history.
  • Corrected probe fidelity: initial QC available probes omitted frontend pluginId and timed out at 30s. Those do not represent the actual page request. Complete frontend pool scope (annotation-qa, dual QC stages, limit 50) returned in 6.72s; mine in 0.163s. Actual default/trace and reviewed calls can differ; do not claim every task request is measured.
  • Read-only Mongo plans: existing sorted pool indexes are present. Complete plugin-scoped candidate plan scanned/returned 100 documents in 618ms; omitted-plugin diagnostic scanned 3,939 with blocking sort (~10s). Do not prescribe a missing pool index from that broader query.
  • Confirmed pool waste: API fetches up to six raw pages before permission filtering; other-project/supplier rows can exhaust that cap and leave visible tasks after its cursor. Added server-only QC project/supplier necessary-condition filters for annotation-qa available pool. Membership scope is request-local, legacy blank supplier remains, shared acceptance paths are excluded, and final role/supplier/self-review checks remain. No cross-request permission cache. Multi-page traversal regression checks final visible tasks, even though fewer invisible rows changes where a page boundary falls.
  • Read-only execution of the repaired GET handler against current data returned 17 tasks/119,034 raw response bytes in 735ms. This is local handler plus Mongo forwarding, not deployed public HTTP acceptance. The old first page could be empty while hasMore=true; do not claim equal first-page counts on that production sample.
  • Workforce loaded every slice revision of joined tasks. New calculator-derived version set applies canonical submission deduplication, skips valid frozen durations, and retains legacy quality-approved-version acceptance fallback. Mongo reads these task/version pairs in bounded groups of 200. A local 1,000-extra-history fixture reads one relevant version with unchanged output; matrix preserves Memory/Mongo reports for author/QC/acceptance/org and details scopes.
  • Real-data read-only stats diagnostic completed with 357 required slice versions. Earlier broad read exceeded its 120s diagnostic context. The forwarding channel transfers slowly; its ~99s completed time is not public HTTP timing and cannot establish a dashboard latency percentage. Submission, attribution/revision/task/run history still incurs reads; reviewOperations currently has only id index and showed ~2s initial query. Further index work, if needed, requires a separately reviewed production plan; none created here.
  • Targeted tests and local isolated Mongo parity passed; full Go gate: 68 failures all in existing baseline (70 known), no new failures. Race-enabled added scope/statistics tests passed, including isolated Mongo and >six-round cursor traversal. No production mutations, index creation, push, PR update or deployment for these repairs.