4.3 KiB
Root optimization of task queries and statistics
Goal: Remove broad-pool search timeouts and make task lists/counts and reporting reads bounded, with equivalent legacy data and live authorization results.
Agent: Codex
Base: origin/dev @ 5375074a, fast-forwarded to deployed 4170c74b before changes to preserve PR #75 fixes; no main-checkout branch change.
Scope: Query summaries/search planning, explicit list summaries and scoped counts, reliable derived-data maintenance/backfill, incremental reporting/snapshot reads, regression/performance evidence.
Out of scope: Production data/index writes, push, PR and deployment until separately authorized; unrelated local work; permission relaxation, truncated history or changed search semantics.
Risk: Derived-state freshness and recovery, mixed clients, old payload search, authoritative claim permission, ordering/cursors, dates and supplier/source deduplication. Query speed must meet representative isolated concurrent p95 <= 2 seconds before release preparation; functional checks alone are insufficient.
Implementation and compatibility
- Source-owned
poolSearchfields are stamped atomically with HumanItem/Task/Project/Run/Capture writes after supplier scrubbing. Search narrows candidates before the existing joins/sorts; the final matcher and server eligibility remain authoritative. Missing/stale metadata uses the existing pipeline. Broad reference sets never truncate results. - Explicit
/api/human-item-summariesand/api/human-item-countsprotocols; legacy item API remains intact. Acceptance detail lives separately from list summaries. Counts traverse all authorized pages and do not return partial totals after errors. - Summary backfill is an explicit, read-only-by-default CLI. Full-document CAS, idempotent reruns, counters only, 64 KiB per-document summary ceiling; no startup/GET writes, index changes or customer-body rewrites. Production apply still needs authorization. Old servers invalidate summaries through source guards; state-only review updates preserve valid summaries without rewriting annotations.
- Statistics reuse the existing shared immutable fact snapshots and add bounded (4 MiB / 128 entries) generation/query-scoped compressed result reuse and concurrent request coalescing. Personal/supplier scope, dates, timezone identity and generation are in the key; decoding/calculation remain bounded. This does NOT implement event-driven source ingestion: existing 120-second projected-source refresh/recovery stays in place. Its 5,000-part/25,000-event cold build measured 240 ms locally; there is no measured reason yet to replace that recovery path.
Evidence so far (not release acceptance)
- Production base
4170c74b, fresh source preserved. Main and unrelated worktrees untouched. - Mongo modern/legacy search parity, payload-only legacy names, tenant poisoning, literal regex characters, category/cursor cases, dry run/retry, old-writer invalidation/rebuild, project rename and state-only review metadata passed locally.
- API summary/full metadata equivalence, shared acceptance revocation and >1,000-row counts passed. Browser search/pagination/retry/unvisited-tab totals passed; quality/acceptance/legacy receipt focus and unsaved-opinion flows passed.
- Dense isolated comparison, identical 81 actors, source payloads, 10-second warmup/60-second measurement: base had 52 errors; candidate 1,765 requests/zero errors. Search p95 7.4–12.4 s -> 257–358 ms; common successful identity sets matched for 1,691 samples, with no differences. Some base actor/endpoint pairs had no successful reference and remain covered by separate equivalence tests, not by that comparison.
- Mongo reply bytes 7.21 GB -> 0.119 GB in that synthetic mix; do not extrapolate it into production savings. Compression settings fixed; no media. CPU/heap readings include the in-process harness, not standalone Pod capacity.
- Statistics isolated 5,000 parts/25,000 events: cold build 240 ms, warm p95 30 ms, 121-request/4-handler cohort burst p95 60 ms, zero history reloads during warm burst.
- First task gate caught legacy no-hit foreign-ID transfer exceeding 64 KiB. Fixed with a tiny legacy-readiness probe; legacy/modern/large-payload Mongo and snapshot regressions passed afterwards. Final task/integration and prolonged repeated performance checks remain pending. Keep this branch local; no production changes, push, PR or deployment authorized for this task.