⚡ 全角色加载性能修复(六轮)+ 审核员领取/专注模式视频/work-buffer 根因修复 #38

Merged
zhanghongming merged 30 commits from codex/slow-page-performance-20261009 into dev 2026-10-12 09:09:42 +08:00
Collaborator

范围(六轮叠加,全部已线上验证)

按 .scratch/slow-page-diagnosis-report.md 实施并逐轮上线验证的加载性能与缺陷修复。

第一轮:全站加载性能(C1–C5/S1–S6)

  • C1/S3 静态资源构建时预压缩(spaHandler 协商直出)+ motion chunk 修正
  • C3/S1 human-items/runs 服务端分页(limit+cursor)、Mongo projection/过滤下推
  • C2/S5 认证 principal TTL 缓存(45s,全失效路径)+ /api/users 分页搜索
  • C4/S2 dashboard 计数聚合下推 + 45s 响应缓存
  • C5 视频预览 BOS 预签名直连(307)+ 后台预转码 + 缓存淘汰 + 免 gzip(回退开关 DROBOTICFLOW_VIDEO_PREVIEW_SYNC=1)
  • S4/S6 POC readiness 点查、human-batches GET 去写、运营页 N+1 扇出消除

第二/三轮(ba5c9909):四角色慢加载

  • 任务池 7 项过滤下推 Mongo + organizationId_createdAt 索引
  • humanItems 改按已过滤任务集反查(organizationId_legacyTaskId 索引)、剔除 annotationDraft 大字段、草稿按需批量读取
  • 管理员快速路径(跳过权限链扇出)、workload-summary 45s 缓存
  • 全接口 >1s 慢请求日志(request_trace kind=slow)

第四轮(efa8f506 + ba1f0ce7):三个用户报障

  • 审核员单任务领取 500 修复:UpsertHumanStageAttempt 改按 (org, item, stage, round) 元组 upsert——review-operation 写入的不同 id 撞 organizationId_itemId_stageId_round_unique 唯一索引(E11000)
  • 专注模式视频占满:填充规则同时匹配 <movi-player>(运行时实际渲染的元素)
  • GET /api/plugins 8 秒迁移扫描:加 legacyMigrationDone 一次性守卫
  • 新增 organizationId_latestAuthorUserId 索引(mine 池 / work-buffer $or 的无索引分支)
  • 任务池服务端分页(limit+cursor)+ 前端 200 条窗口渐进渲染

第五/六轮(a19459d9 + 99900d58):work-buffer 轮询根因

  • 追踪定位:batches_ms 占每 3 秒轮询的 80%(1400–3900ms)——该轮询每次物化 12,009 个批次文档(遗留迁移每任务建一个批次),走 organizationId 单字段索引扫 17,334 行
  • 修复:HasRawHumanAnnotationBatch 投影存在性查询(首条命中即返回)+ 新索引 organizationId_projectId_pluginId
  • 生产实测:2–4s → 200–260ms(batches 段 2000ms → 12ms)

与 dev 的合并(e07f3cac)

dev 的并行化(PR #35 errgroup)与本分支的分页/缓存/聚合计数器同时保留:

  • 路由:保留 /api/dashboard/workforce,production 路由包响应缓存
  • dashboard 加载:errgroup 并行 fetch,runs goroutine 保留游标分页(limit 50 + aggregates)
  • enrichDashboardHumanWork:保留聚合计数器快速路径;无接口的 store 走并行 legacy 回退

验证证据

  • 回归基线:全量 go test ./internal/api/ 失败数改动前后 65=65(上游遗留债,零新增);dashboard/workforce/work-buffer 相关定向测试全绿;go vet + npm run lint:web 干净
  • 真实浏览器复现:修复前领取 POST 500(E11000)→ 修复后 200
  • 50 并发压测(17k 任务/17k 条目合成库,四角色):任务列表 avg 118ms / p100 240ms,0 错误
  • 线上:round4/5/6 已按发布流程部署并 verify(local-data/releases/slow-page-fix-round4/5/6-20261012,含 image-provenance + change-safety 证据);部署后生产日志慢请求数为 0

数据安全

纯读路径优化 + 一处元组 upsert 修复;无 schema 迁移、无客户数据改写;全���索引幂等创建。回退 = 两 Deployment patch 回上一镜像。

## 范围(六轮叠加,全部已线上验证) 按 `.scratch/slow-page-diagnosis-report.md` 实施并逐轮上线验证的加载性能与缺陷修复。 ### 第一轮:全站加载性能(C1–C5/S1–S6) - **C1/S3** 静态资源构建时预压缩(spaHandler 协商直出)+ motion chunk 修正 - **C3/S1** human-items/runs 服务端分页(limit+cursor)、Mongo projection/过滤下推 - **C2/S5** 认证 principal TTL 缓存(45s,全失效路径)+ /api/users 分页搜索 - **C4/S2** dashboard 计数聚合下推 + 45s 响应缓存 - **C5** 视频预览 BOS 预签名直连(307)+ 后台预转码 + 缓存淘汰 + 免 gzip(回退开关 `DROBOTICFLOW_VIDEO_PREVIEW_SYNC=1`) - **S4/S6** POC readiness 点查、human-batches GET 去写、运营页 N+1 扇出消除 ### 第二/三轮(ba5c9909):四角色慢加载 - 任务池 7 项过滤下推 Mongo + `organizationId_createdAt` 索引 - humanItems 改按已过滤任务集反查(`organizationId_legacyTaskId` 索引)、剔除 annotationDraft 大字段、草稿按需批量读取 - 管理员快速路径(跳过权限链扇出)、workload-summary 45s 缓存 - 全接口 >1s 慢请求日志(`request_trace kind=slow`) ### 第四轮(efa8f506 + ba1f0ce7):三个用户报障 - **审核员单任务领取 500 修复**:`UpsertHumanStageAttempt` 改按 (org, item, stage, round) 元组 upsert——review-operation 写入的不同 id 撞 `organizationId_itemId_stageId_round_unique` 唯一索引(E11000) - **专注模式视频占满**:填充规则同时匹配 `<movi-player>`(运行时实际渲染的元素) - **GET /api/plugins 8 秒迁移扫描**:加 `legacyMigrationDone` 一次性守卫 - 新增 `organizationId_latestAuthorUserId` 索引(mine 池 / work-buffer $or 的无索引分支) - 任务池服务端分页(limit+cursor)+ 前端 200 条窗口渐进渲染 ### 第五/六轮(a19459d9 + 99900d58):work-buffer 轮询根因 - 追踪定位:`batches_ms` 占每 3 秒轮询的 80%(1400–3900ms)——该轮询每次物化 **12,009 个批次文档**(遗留迁移每任务建一个批次),走 organizationId 单字段索引扫 17,334 行 - 修复:`HasRawHumanAnnotationBatch` 投影存在性查询(首条命中即返回)+ 新索引 `organizationId_projectId_pluginId` - 生产实测:**2–4s → 200–260ms(batches 段 2000ms → 12ms)** ### 与 dev 的合并(e07f3cac) dev 的并行化(PR #35 errgroup)与本分支的分页/缓存/聚合计数器同时保留: - 路由:保留 `/api/dashboard/workforce`,production 路由包响应缓存 - dashboard 加载:errgroup 并行 fetch,runs goroutine 保留游标分页(limit 50 + aggregates) - `enrichDashboardHumanWork`:保留聚合计数器快速路径;无接口的 store 走并行 legacy 回退 ## 验证证据 - 回归基线:全量 `go test ./internal/api/` 失败数改动前后 65=65(上游遗留债,零新增);dashboard/workforce/work-buffer 相关定向测试全绿;`go vet` + `npm run lint:web` 干净 - 真实浏览器复现:修复前领取 POST 500(E11000)→ 修复后 200 - 50 并发压测(17k 任务/17k 条目合成库,四角色):任务列表 avg 118ms / p100 240ms,0 错误 - 线上:round4/5/6 已按发布流程部署并 verify(`local-data/releases/slow-page-fix-round4/5/6-20261012`,含 image-provenance + change-safety 证据);部署后生产日志**慢请求数为 0** ## 数据安全 纯读路径优化 + 一处元组 upsert 修复;无 schema 迁移、无客户数据改写;全���索引幂等创建。回退 = 两 Deployment patch 回上一镜像。
zhanghongming added 15 commits 2026-10-10 09:34:08 +08:00
Add a PART_INDEX_DELIVERY mode to the package operator (default off): a part
delivers a single part.json carrying both split levels, the task-level slice
boundaries and the action-level segments, plus a classification for every
reviewed interval. Kept intervals land in slices[], discarded intervals in
discardedSlices[] with their discard evidence, which deliberately makes the
discard reason part of the customer deliverable. Media, PTS, IMU, pose and
calibration bytes are no longer published; downstream packaging resolves them
from the pinned materialization revision.

The worker publishes the index through the same frozen-destination path as the
directory form. Extract prepareDeliveryTarget so single-file and directory
publication share one owner marker, overlap guards, delegated-scope check,
conditional create, sha256 verification, idempotent retry and the refusal to
overwrite another delivery. A part index also refuses to adopt a destination
that holds files without an owner marker, so additive migration cannot claim a
legacy directory by accident.

The API and download control plane recognise the new schema, project the split
classification for the browser, and treat a part index as a first-class final
delivery. Cross-language contract tests read the operator's own schema and
field literals so a one-sided rename fails a test.

Verification: 10 operator tests, 8 worker tests and 3 API tests for the new
behaviour pass. Failure sets match the HEAD baseline name-for-name for
internal/api (75), internal/worker (4), tests/backend and the frontend suite;
no regression. The end-to-end local replica stack was not exercised: it needs
image builds and the opt-in Go materialization fixture, so the Go to operator
handoff is covered by the contract tests instead.
Add scripts/part_index_migration.py, which produces the missing part.json for a
part that was delivered in the directory form. It reuses the package operator's
own builder by loading the operator source the way the runner does, so a
migrated index is structurally identical to an online package and
byte-identical to what the same evidence would have produced, apart from a
lineage.migratedFrom marker. A test compares both routes on one evidence set to
keep the two from forking.

The tool never writes object storage: publication has to go through the
API/Worker path so the owner marker, conditional create and no-overwrite
guarantees stay in effect. It refuses to run when the delivery directory lacks a
completion manifest, when a slice has an unknown disposition, when the revision
has no per-slice disposition evidence, and when the part already has a part.json
with different bytes.

Also fix a leak found while building this: the index wrote the operator's
absolute workspace path into calibration[].path. It now records the logical
delivery-relative path, matching the directory form's packages-manifest, and
keeps the local location out of customer-facing output.

Verification: 19 operator and migration tests pass. tests/backend failure set is
identical to the HEAD baseline (25 failed, 829 subtests passed, 516 vs 525
passing), so no regression.
Reading a delivered index through the replica API (object storage disabled,
payload resolved from a workspace file) recognised the document and projected
the full split classification but exposed zero members, because the single
deliverable was only emitted for a durable object ref. The one output of this
delivery mode was therefore not downloadable in that situation. Fall back to the
dataRef payload path when no object ref exists.

Verify against the local replica for the first time: with the API running on the
snapshot Mongo, the members endpoint returns the kept and discarded slices with
their boundaries, action counts and discard reason, and the member content
endpoint returns part.json byte-identical to the delivered file. The delivery
package contents endpoint lists part.json as a final export artifact.

Add a boundary test over real materialization output: the Go acceptance test
builds genuine 60s stereo media (80 frames per slice) and slices it with ffmpeg;
the new test consumes that same revision and asserts the review layer keeps all
seven evidence files per slice while the delivered index contains no media
reference at all. It is opt-in behind FANYU_PART_INDEX_MATERIALIZATION_CONTEXT.

The replica fixture helper lives under the untracked local-data path; it tags and
removes exactly the records it creates and never touches object storage.
Record how to verify the part-index delivery in the local replica: the three
test layers, the real-media boundary check that keeps review media while the
index references none, the reversible replica fixture, and the dry-run
migration command. Also note that the snapshot super admin cannot log in locally
because legacy organization-admin access is off, so the organization admin
account and its phone identifier are the working path.
- C1/S3: build-time precompressed .gz served by spaHandler (writeBundle plugin), motion-vendor chunk rule fix
- C3/S1: /api/runs default cursor pagination (byte-compatible), /api/human-items limit/cursor + Mongo projection + filter pushdown
- C2/S5: in-process principal TTL cache (45s) with full invalidation paths; /api/users limit+q with org isolation
- C4/S2: dashboard counter pushdown ($group/CountDocuments), 45s response/snapshot caches, production stats snapshot cache
- C5: BOS presigned direct-link video preview (307) with background pre-transcode, /tmp cache TTL+LRU, gzip bypass, sync=1 rollback
- S4/S6: POC readiness point queries, human-batches GET no longer writes (60s sweeper), admin ops page N+1 fan-out removed
- Fixes found en route: BSON cursor self-reference recursion, backfill loop cursor advance, supplier cascade session flush
- Frontend: nextCursor passthrough (human-loop/users), acceptance pool no longer truncated at 500

Co-authored-by: AgentTeams slow-page-fix
zhanghongming added 7 commits 2026-10-11 23:13:21 +08:00
- F0: segmented latency logging for annotation tasks list handler
- F1: ListSliceRevisionsByTaskIDs/ListAnnotationRevisionsByTaskIDs replace org-wide revision scans (batched $in, existing indexes)
- F2: TaskListProjection drops 6 heavy payload keys from task-list human item reads (keeps annotationDraft)
- F3: humanEvents (organizationId, actorUserId, eventType) index for reviewer distinct queries
- F4: derive per-project memberships from cached org-wide snapshot (~300 Mongo round trips removed)
- F5: skip per-project members/workflows fan-out on review-only workbench without projectId
- F7: 30s default request timeout with user-facing retry feedback; FormData uploads exempt (timeoutMs: 0)

Co-authored-by: AgentTeams slow-page-fix-round2
E' foundation (per proposal v4, user constraints: no server-side transcode
fallback, no upstream format talks, fast ship):
- video-preview.js: buildBrowserPlayableVideoURL no longer appends
  transcode=browser; legacy param is defensively stripped; bos:// proxy
  URLs are direct too
- LazyVideoPreview.tsx: remove useProxy transcode retry (sync=1 blocking
  stream); direct playback only; error copy points to browser update
- annotation_operations.go: drop unconditional task-detail warmup call
  (<=2 ffmpeg per GET, C-1 complete)
- server.go: mark warmAnnotationVideoPreviews dormant (kept one release
  cycle as rollback hatch)
- video-preview.test.mjs: flip 7 assertions to direct/no-param, add
  bos:// + legacy-strip coverage (14/14 green)

Verified: tsc clean, go build cmd+internal ok, targeted Go tests green
(1 baseline pre-existing failure unchanged via stash A/B), full frontend
suite failures identical to baseline (14), local replica: task GET has
zero warmup/cache/ffmpeg activity, direct artifact content returns 307
presigned redirect, transcode=browser mechanism retained and produces
H.264 960px 12fps cache artifact (rollback path intact).

proposal-final.md v4: new Part E row + §6.6 (E'+D fused decode chain),
recommendation reworked (A backlog, E'+D single-ship, C accelerated,
B dormant pending open question 13), appendices D.8 + full attachment D
implementation & verification record. Not pushed, not deployed.
- vendor public/vendor/movi-player-0.4.1/ (element.slim.js 4.9MB + movi.wasm 5.4MB,
  Apache-2.0, SHA-256 pinned, upgrade = new versioned dir + MOVI_VERSION bump)
- src/lib/movi-player.ts: lazy loader (@vite-ignore dynamic import), runtime
  force flag (movi default / native fleet kill-switch / wasm canary), throttled
  best-effort telemetry reporter
- src/components/MoviVideo.tsx: engine="native wasm" + nohotkeys, HTMLMediaElement
  event bridge, #t= fragment split (startat + wrapper-managed end bound),
  in-place custom-element upgrade preserving node identity, wasmurl before connect
- mount points: LazyVideoPreview (evidence/artifact/object-storage previews),
  MaterializedSliceWorkbench dual-camera, CaptureAnnotationWorkbench main videos
  (moviElementReady gate: transport seek queue needs the element on first render;
  falls back to native E′ path until defined), page-level preload on
  AnnotationOperations
- POST /api/client/movi-stats: session-attributed, field-truncated, store errors
  swallowed (202); Mongo clientMoviStats collection w/ 14d TTL + org/kind indexes
- spikes (CDP, headless Edge 155, production hev1 480p30 sample): key conflicts
  PASS (17 keys all reach container dispatcher, zero playback-state interference,
  nohotkeys effective); frame-seek parity PASS (movi deltas identical to native
  <video>, ±1/30s steps both directions); deep probe: engine=native-wasm hands
  playback to shadow <video> with zero WASM fetches on HW-decode clients
- verification: tsc clean, gofmt/build/vet pass, frontend 644 tests net-zero new
  failures vs E′ baseline (6 new pass), Go API suite net-zero new failures
  (75 pre-existing; the one delta traced to a colleague's uncommitted part-index
  working-tree change), replica E2E: vendor assets served, module registers,
  telemetry lands in Mongo with TTL
- Dockerfile.api web-build never copied public/, so vite publicDir output
  (movi-player 0.4.1 vendor + diagnostics-probe.json) never reached the
  production image; local replica hid this because Vite dev serves the
  source tree directly
- spaHandler: /vendor/ joins the immutable-cache and miss-404 branches
  (version-pinned dirs; a failed module import must 404, not receive
  index.html as a confusing syntax error)
- application/wasm MIME comes from Go's builtin table (verified
  GOROOT/src/mime/type.go; container golang:1.24-bookworm)
- TestSPAHandlerServesVersionedVendorAssets: wasm MIME + immutable header,
  miss 404, probe not cached
- proposal-final.md D.9 records the gap and fix (release prep finding)
zhanghongming added 1 commit 2026-10-11 23:22:48 +08:00
The D-layer commit (54e47906) was staged from the shared working tree and
swept in one unrelated in-flight line: collapsedByDefault={false} on
WorkforceStatisticsPanel — a usage of a prop that exists only in a
colleague's uncommitted panel WIP. The committed history alone therefore
failed tsc (the main tree compiled only because that WIP panel was
present locally). Remove the leaked usage so the release branch's
committed state is self-contained; the colleague's feature line stays
in his uncommitted work exactly where it was.
zhanghongming added 1 commit 2026-10-11 23:35:41 +08:00
zhanghongming added 1 commit 2026-10-12 00:46:38 +08:00
- Task pool: push status/supplier/workflow/run/node/artifact equality filters into ListAnnotationTasksFiltered (new store method, backed by organizationId+createdAt indexes) instead of post-filtering an organization-wide scan
- Human items join: ListHumanItemsByLegacyTaskIDs fetches only the listed tasks' items via the organizationId+legacyTaskId index; TaskListProjection now drops annotationDraft too
- Draft progress: ListHumanItemDraftProgress loads slice/segment drafts per filtered task only (batched $in, new projection reads just legacyTaskId + draft payload); draft overrides revision only when the human item was touched after the latest slice revision
- Manager fast path: admin views skip the request-scoped batch snapshot, membership and supplier-pool reads entirely
- Workload summary: 45s per-user response cache (any /api write invalidates), mirroring the dashboard workforce route
- Tracing: any /api request over 1s now logs method/path/duration so page-level latency regressions surface without per-handler instrumentation; task list log gains drafts_ms segment
- Tests: draft-progress override semantics, manager scan-free path, store filter pushdown; v5 fixture users get SupplierID to match the supplier-unified account model (pre-existing red since c92bca16, 18 stock failures recovered green)

Co-authored-by: AgentTeams slow-page-fix-round3
zhanghongming added 1 commit 2026-10-12 02:31:32 +08:00
- UpsertHumanStageAttempt now upserts on (organization, item, stage, round):
  review operations create attempt rows with operation-scoped ids, so the
  claim path's id-only upsert violated organizationId_itemId_stageId_round_unique
  (E11000 → 500 相同的记录已存在) and reviewers could not claim tasks whose
  round already had an attempt row. Verified against the 2026-08-31 snapshot
  replica with a real browser claim.
- Focus/compact workspace video: the fill rules targeted only <video>, but the
  runtime renders <movi-player> (custom element) whenever browser-transcode
  fallback is not confirmed, leaving the player at clamp-height with dead
  space. Both rules now also match movi-player.
- migrateLegacyAnnotationHumanItems: one-shot guard stamped into the plugin
  record when the sweep completes; GET /api/plugins no longer re-scans
  17k tasks + 17k humanItems every read (8s per call in production).
- New humanItems index organizationId_latestAuthorUserId: the mine-pool /
  work-buffer ActorUserID  previously had an unindexed latestAuthorUserId
  arm, forcing organizationId_legacyTaskId scans of 17k docs (6.2s) on every
  3s poll.
- Task pool progressive rendering (200-card windows with 显示更多): the pool
  state keeps full data, the DOM no longer mounts 17k cards at once.
zhanghongming added 1 commit 2026-10-12 02:44:27 +08:00
- ListAnnotationTasksFiltered (Memory + Mongo) accepts Limit and a descending
  (createdAt, id) cursor; the handler exposes ?limit&cursor and returns
  nextCursor/hasMore metadata only for explicitly limited requests, keeping
  the legacy unbounded body byte-compatible for existing consumers.
- The annotation center fetches the first 200-task window; 显示更多 pulls the
  next page via the opaque cursor and appends, so a 17k-task organization no
  longer serializes the whole pool per page view (17k×~1KB JSON per load).
zhanghongming added 1 commit 2026-10-12 02:54:58 +08:00
The new indexes cut the work-buffer poll from 3-5s to 2-3s, but the remaining
cost is unlocated. This trace records where the time goes per poll so the
next fix targets evidence, not guesses.
zhanghongming added 1 commit 2026-10-12 03:07:26 +08:00
The trace deployed in round5 located the cost: batches_ms carried 80% of
every work-buffer poll (1400-3900ms of 1600-4100ms total). The poller's
enablement check called ListHumanBatches for the project, which — because
the legacy migration created one batch per task — materialized 12,009 full
batch documents on every 3s poll, walking an organizationId-only index over
17,334 docs.

- HasRawHumanAnnotationBatch (Memory + Mongo): projects only the stages
  field and stops at the first three-role raw_human match (105ms on the
  production dataset, was 2000ms+).
- New humanBatches index organizationId_projectId_pluginId backs both the
  enablement query and ListHumanBatches' project-scoped reads.
- store.Store interface gains the method; both store implementations keep
  identical semantics (mid-test profile flips stay visible).
zhanghongming added 1 commit 2026-10-12 08:48:50 +08:00
Resolve server.go and production_stats.go conflicts by combining both
sides' intents:
- routes: keep dev's /api/dashboard/workforce and wrap the production
  dashboard in the response cache
- dashboard loading: dev's parallel errgroup fetches, but the runs
  goroutine keeps the C4 cursor-paginated summary page (limit 50 +
  aggregates) instead of the unbounded list
- enrichDashboardHumanWork: keep the C4 aggregation-counter fast path;
  the legacy fallback (stores without the counting interface) now
  fetches items/packages in parallel like dev
- production_stats.go imports: our superset (time/sort/strings + cache)
zhanghongming changed title from ⚡ 加载性能修复:静态压缩/列表分页/认证缓存/dashboard 瘦身/视频直连 to ⚡ 全角色加载性能修复(六轮)+ 审核员领取/专注模式视频/work-buffer 根因修复 2026-10-12 08:49:40 +08:00
zhanghongming merged commit 4916154490 into dev 2026-10-12 09:09:42 +08:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: longtao.wu/droboticflow#38
No description provided.