From dfb9d7bc655748dfff96c79d04bf14843e86580e Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 9 Nov 2026 21:51:55 -0800 Subject: [PATCH] =?UTF-8?q?=E2=9A=A1=20Batch=20administrator=20pool=20meta?= =?UTF-8?q?data=20joins?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .scratch/query-readmodels-scope-fix/work.md | 14 ++ docs/annotation-admin-task-pool.zh-CN.md | 18 +++ internal/api/admin_annotation_pool.go | 6 +- internal/api/admin_annotation_pool_test.go | 61 +++++++++ internal/api/poll_pool_load_test.go | 22 +++- internal/store/admin_annotation_pool.go | 74 ++++++----- internal/store/admin_annotation_pool_reads.go | 81 ++++++++++++ internal/store/admin_annotation_pool_test.go | 122 +++++++++++++++++- scripts/verification-scopes.json | 24 +++- 9 files changed, 376 insertions(+), 46 deletions(-) create mode 100644 internal/store/admin_annotation_pool_reads.go diff --git a/.scratch/query-readmodels-scope-fix/work.md b/.scratch/query-readmodels-scope-fix/work.md index ded58bce..1968d2f3 100644 --- a/.scratch/query-readmodels-scope-fix/work.md +++ b/.scratch/query-readmodels-scope-fix/work.md @@ -52,3 +52,17 @@ Follow-up base: 7a1231c7 (live API/processor source; freshness checked against o - Final gate verify-1794271033888668000 selected-checks-passed; flight race passed. Six executions have identical harness/candidate patch hashes. Final source freshness includes origin/dev and live commit. - Evidence: final-acceptance-summary.json, mongo8-isolated-formal-comparison.json, six formal reports and resource/execution logs in private diagnostics directory. Docs record failures, corrected protocol, limits, five-index authorization and human-only CAS rollout. - No new push/PR/deployment, production index/backfill or customer writes. Old release remains deployed_pending_acceptance. Local task complete; production enablement separate. + +### Administrator pool timeout follow-up + +Base: ae52e5d4; freshness includes origin/dev and live7a1231c7. Same authorized follow-up worktree; no production mutation/deployment. + +- Production read-only logs: admin-pool500 at8000/8006ms. Separate endpoint missed by earlier81-actor harness (admin only read workforce). +- Red loop: ADMIN_POOL_SCALE=1 TestAdminAnnotationPoolMongoParity,26218 isolated synthetic records; searches6.23–6.51s breach2s. Counts~2.04s. Evidence admin-scale-red.log. +- Batch64 metadata joins for batch/stage validation, search names and source jobs; reuse original source-selection/decision expressions. Match legacy payloads in bounded batches; hydrate only matched page and preserve response shape/order. No new index, cache permission or task writes. +- Batch display priming for admin response. Fix missing nodeId in per-actor counts cache scope. Existing5s expiry and server authorization unchanged. +- Scale green searches0.90–1.02s,counts0.81s; equal reference IDs/counts/paging.192-source-case parity passed, including missing/failed/foreign/superseded proof. +- Gate verify-1794289069683716000 selected-checks-passed, no unmapped/deferred: admin/QC/acceptance frontend+HTTP+Mongo/browser, official Go baseline gate, local81 read mix. +- Post-gate changes only to opt-in harness scheduling for repeated admin measurements; production Go unchanged. First extended81 mix3410 reads,0errors/writes; admin searches0.75–1.56s but all-project available had one3.53s sample. Kept as diagnostic, not stable p95 evidence. Higher-frequency repetitions in progress. + +- Repeated administrator supplement finished:30s warmup+200s steady,1 admin request/second,81 actors;5920 measured reads,0errors/writes. Ten admin scenarios each20samples. All-project availablep95=311.603ms, admin searchp95=475.048–670.738ms; other search maxp95=724.445ms. Same actor/endpoint fingerprints stable. Earlier3.53s sample retained; production cold-load/capacity acceptance pending. Private final-summary.json records limits. No push/PR/deployment or production index/data changes. diff --git a/docs/annotation-admin-task-pool.zh-CN.md b/docs/annotation-admin-task-pool.zh-CN.md index 6cf15396..458213f6 100644 --- a/docs/annotation-admin-task-pool.zh-CN.md +++ b/docs/annotation-admin-task-pool.zh-CN.md @@ -11,3 +11,21 @@ 总数按批次/阶段聚合紧凑计数,再用既有批次权限投影校验;不读取动作正文或草稿来计数。总数使用有身份和范围隔离、容量上限的 5 秒进程缓存,当前进程的 API 写入会使缓存失效;其他副本与后台写入最多延迟到缓存到期。列表不缓存。搜索输入合并 350 毫秒,新搜索与范围切换取消过期请求;旧任务搜索在服务端流式读取投影元数据,查询和范围统计受 8 秒期限限制,不向浏览器传整池,也不把失败当成零条。计数与页面是相邻的只读操作,并非冻结生产时刻的结算快照。 不新增统计集合、数据库字段或索引,不改领取、标注、返工、审核及交付结果。生产环境仍需单独发布与验收。 + +## 2026-11-09 管理员查询超时修复 + +管理员任务池使用独立的 `/admin-pool`,与质检/验收的 `human-item-summaries` 搜索不同。线上只读日志确认前者出现500及8000/8006毫秒超时;截图中的HTTP非安全连接提示是连续工作入口的另一项限制,不能解释该接口的后端超时。 + +旧路径按每条候选关联批次、运行、采集及来源校验任务,搜索旧任务或没有命中时会扫完整个范围;总数读取在页面列表之前执行,失败会阻断首屏。管理员显示名称补充也未调用按页预取。此前81人读取对照中的管理员只请求统计面板,不能作为该独立接口已验证的证据。 + +本次改为每64条窄元数据批量关联,逐条恢复原批次/阶段、来源凭证及范围判定,然后按原创建时间和ID恢复顺序。仅命中的页面按ID读取原列表字段;缺少或过期摘要的历史正文仍按批读取,不丢旧名称。保留搜索字面匹配、全部任务的原环节范围、游标、跨组织隔离及缺失/失败/已替代来源凭证规则。没有新增索引、字段或生产补写要求。 + +管理员显示名称预取改为整页一次;5秒计数缓存保留,补齐节点范围键,避免不同节点复用错误总数。列表仍不缓存,权限检查仍在计数缓存之前执行。其他角色不经过该管理员入口:质检/验收继续使用既有批量搜索路径,标注个人池及来源准入继续使用其有界读取路径。本次回归覆盖这些入口,不能据此宣称所有页面或网络链路都没有性能瓶颈。 + +验证入口:`ADMIN_POOL_SCALE=1 POOL_LOCAL_MONGO_URI=mongodb://127.0.0.1:27018 go test ./internal/store -run '^TestAdminAnnotationPoolMongoParity$' -count=1 -v`。只接受本机27018,创建并清理独立合成库;26,218条记录对照内存参考实现,覆盖历史命中、无命中、特殊字符、自定义Capture、状态和游标。192组来源凭证用原表达式对照批量表达式;管理员HTTP及真实浏览器回归验证权限、范围计数、翻页、取消与失败重试。 + +额外混合读取使用 `TestPollPoolReadMixLocal`,在生产规模合成数据及81人背景下显式设置 `POLL_POOL_ADMIN_TASKS=1`、`POLL_POOL_CLAIM_COUNTS=1`;`POLL_POOL_ADMIN_PERIOD_SECONDS=1` 用于提高单个管理员的重复采样频率,默认仍为12秒。它不改变生产轮询。计入管理员的全部/单项目、可领取、我的任务、状态、命中及无命中。该短时补验排除媒体、写作业、网关和公网传输,不替代生产容量验收。 + +结果:26,218条全量有效候选的历史命中/无命中搜索由约6.2~6.5秒降至0.90~1.02秒,计数由约2.04秒降至0.81秒。首轮81人补验的某次全部项目可领取为3.53秒,保留为异常样本,不用于稳定p95结论。后续提高管理员重复频率:预热30秒、稳态200秒,10种管理员场景各20次;全部项目可领取API p95为312ms,管理员搜索p95为475~671ms,质检/验收搜索最高p95为724ms。5,920次稳态读取零HTTP错误、零业务写入,同账号同接口任务/数量/权限及追溯摘要稳定。该时间不含公网传输,旧响应格式仍保留;生产冷加载及容量仍待上线只读复测。 + +任务门禁 `verify-1794289069683716000` 通过,无未映射或待执行项,含管理员与审查浏览器、Mongo历史兼容、角色权限及项目Go基线门禁。之后仅调整测试脚本的管理员采样周期,API/Store实现未变;调整后的脚本通过上述重复采样。私有证据在工作树 `local-data/diagnostics/acceptance-search/admin-*` 及 `local-data/diagnostics/admin-pool-timeout/final-summary.json`。本轮只读查线上、仅在隔离合成库测试,未推送、部署或修改客户数据。 diff --git a/internal/api/admin_annotation_pool.go b/internal/api/admin_annotation_pool.go index 3961885d..b921a2a3 100644 --- a/internal/api/admin_annotation_pool.go +++ b/internal/api/admin_annotation_pool.go @@ -57,7 +57,7 @@ func (s *server) adminAnnotationCounts(w http.ResponseWriter, r *http.Request, q copyURL := *r.URL countRequest.URL = ©URL values := url.Values{} - for _, key := range []string{"projectId", "workflowId", "runId", "nodeId", "artifactId", "capture"} { + for _, key := range []string{"projectId", "workflowId", "runId", "nodeId", "artifactId", "nodeId", "capture"} { if value := r.URL.Query().Get(key); value != "" { values.Set(key, value) } @@ -110,6 +110,10 @@ func (s *server) handleAdminAnnotationPool(w http.ResponseWriter, r *http.Reques page.NextCursor = encodeHumanItemCursor(&domain.HumanItemCursor{CreatedAt: last.CreatedAt, ID: last.ID}) } resolver := newHumanDisplayResolver(s) + if err := resolver.primeRuns(r.Context(), page.Items); err != nil { + renderError(w, 500, err) + return + } for i := range page.Items { if err := resolver.enrichItem(r.Context(), &page.Items[i]); err != nil { renderError(w, 500, err) diff --git a/internal/api/admin_annotation_pool_test.go b/internal/api/admin_annotation_pool_test.go index a4a5d101..adffa592 100644 --- a/internal/api/admin_annotation_pool_test.go +++ b/internal/api/admin_annotation_pool_test.go @@ -9,6 +9,7 @@ import ( "fmt" "net/http" "net/url" + "strings" "testing" "time" ) @@ -144,3 +145,63 @@ func TestAdminAnnotationPoolFreshPermissionsCacheAndFailures(t *testing.T) { t.Fatal("page fetched after count failure") } } + +func TestAdminAnnotationPoolBatchesPageDisplayReads(t *testing.T) { + f := newFanyuV5FourStageFixture(t, domain.FanyuV5StageSemanticAnnotation, domain.HumanItemReady, "", 0) + admin := domain.UserRecord{ID: "admin-batch-display", OrganizationID: domain.DefaultOrganizationID, Role: domain.UserRoleAdmin, Status: domain.UserStatusActive} + for n := 0; n < 20; n++ { + id := fmt.Sprint(n) + item := f.item + item.ID = "display-item-" + id + item.RunID = "display-run-" + id + item.LegacyTaskID = "" + if err := f.store.UpsertHumanItem(f.ctx, item); err != nil { + t.Fatal(err) + } + if err := f.store.UpsertRun(f.ctx, domain.RunRecord{ID: item.RunID, OrganizationID: item.OrganizationID, TaskID: "capture-" + id}); err != nil { + t.Fatal(err) + } + if err := f.store.UpsertCaptureRecord(f.ctx, domain.CaptureRecord{ID: "capture-" + id, OrganizationID: item.OrganizationID, CaptureID: "directory-" + id}); err != nil { + t.Fatal(err) + } + } + probe := &displayCaptureReadStore{Store: f.store} + f.handler = NewHandler(probe, nil) + r := f.request(http.MethodGet, "/api/plugins/annotation-qa/admin-pool?pool=all&limit=200", admin, nil) + if r.Code != 200 { + t.Fatalf("%d %s", r.Code, r.Body) + } + if probe.batches != 1 || probe.singles != 0 { + t.Fatalf("display fan-out: batch=%d single=%d", probe.batches, probe.singles) + } + var body struct { + Items []domain.HumanItemRecord `json:"items"` + } + if err := json.Unmarshal(r.Body.Bytes(), &body); err != nil { + t.Fatal(err) + } + if len(body.Items) != 21 { + t.Fatalf("lost page entries: %d", len(body.Items)) + } + for _, i := range body.Items { + if strings.HasPrefix(i.ID, "display-item-") && (i.Display == nil || len(i.Display.CaptureDirectories) != 1 || i.Display.CaptureDirectories[0] != "directory-"+strings.TrimPrefix(i.ID, "display-item-")) { + t.Fatalf("display changed: %+v", i) + } + } +} + +func TestAdminAnnotationPoolCountsSeparateNodeScopes(t *testing.T) { + f := newFanyuV5FourStageFixture(t, domain.FanyuV5StageSemanticAnnotation, domain.HumanItemReady, "", 0) + p := &adminPoolProbe{Store: f.store} + f.handler = NewHandler(p, nil) + admin := domain.UserRecord{ID: "admin-node", OrganizationID: domain.DefaultOrganizationID, Role: domain.UserRoleAdmin, Status: domain.UserStatusActive} + for _, node := range []string{"node-a", "node-b", "node-a"} { + r := f.request(http.MethodGet, "/api/plugins/annotation-qa/admin-pool?pool=all&nodeId="+node, admin, nil) + if r.Code != 200 { + t.Fatalf("%d %s", r.Code, r.Body) + } + } + if p.counts != 2 { + t.Fatalf("different node scopes shared counts: %d", p.counts) + } +} diff --git a/internal/api/poll_pool_load_test.go b/internal/api/poll_pool_load_test.go index 1dd99acd..72b5291e 100644 --- a/internal/api/poll_pool_load_test.go +++ b/internal/api/poll_pool_load_test.go @@ -78,6 +78,7 @@ func TestPollPoolReadMixLocal(t *testing.T) { } return n } + adminPeriod := seconds("POLL_POOL_ADMIN_PERIOD_SECONDS", 12) warmup, duration := seconds("POLL_POOL_WARMUP_SECONDS", 5), seconds("POLL_POOL_DURATION_SECONDS", 30) annotators := seconds("POLL_POOL_ANNOTATORS", 60) quality := seconds("POLL_POOL_QUALITY_REVIEWERS", 12) @@ -602,6 +603,21 @@ func TestPollPoolReadMixLocal(t *testing.T) { return result } if a.Role == "admin" { + if os.Getenv("POLL_POOL_ADMIN_TASKS") == "1" { + base := "/api/plugins/annotation-qa/admin-pool?limit=200" + scope + suffix, key := "&pool=available", "available" + switch scenario % 5 { + case 1: + suffix, key = "&pool=all&search=synthetic-available-", "search" + case 2: + suffix, key = "&pool=all&search=__production_shape_no_match__", "search.nohit" + case 3: + suffix, key = "&pool=mine", "mine" + case 4: + suffix, key = "&pool=all&category=pending_slice", "category" + } + return map[string]string{"admin." + scopeName + "." + key: base + suffix} + } return map[string]string{"admin.workforce": "/api/dashboard/workforce?from=2026-10-01T00:00:00Z&to=2026-10-28T00:00:00Z"} } if pageShape { @@ -689,6 +705,7 @@ func TestPollPoolReadMixLocal(t *testing.T) { return result } fixtureSpec := fmt.Sprintf("isolated-preparation-v5|%d|%d|%d|%d|%d|%d|%d|%t|%t|%t|%s|%s", annotators, quality, acceptance, availablePerStage, runPayloadKiB, taskPayloadKiB, authorHistory, productionShape, readModels, claimCounts, authorizationState, os.Getenv("POLL_POOL_SEARCH_INDEXES")) + fixtureSpec += "|admin-tasks=" + os.Getenv("POLL_POOL_ADMIN_TASKS") fixtureDigest := fmt.Sprintf("%x", sha256.Sum256([]byte(fixtureSpec))) if reuseFixture { var manifest struct { @@ -735,7 +752,7 @@ func TestPollPoolReadMixLocal(t *testing.T) { } t.Fatalf("preflight %s status=%d", key, result.Code) } - if (a.Role == "annotation" || a.Role == "admin" || strings.HasSuffix(key, ".available") || strings.HasSuffix(key, ".mine") && !strings.Contains(key, ".acceptance.")) && result.Rows == 0 { + if (a.Role == "annotation" || a.Role == "admin" && os.Getenv("POLL_POOL_ADMIN_TASKS") != "1" || strings.HasSuffix(key, ".available") || strings.HasSuffix(key, ".mine") && !strings.Contains(key, ".acceptance.")) && result.Rows == 0 { t.Fatalf("preflight %s unexpectedly empty for %s", key, a.User.ID) } } @@ -754,6 +771,9 @@ func TestPollPoolReadMixLocal(t *testing.T) { if a.Role == "annotation" { period = 3 * time.Second } + if a.Role == "admin" && os.Getenv("POLL_POOL_ADMIN_TASKS") == "1" { + period = time.Duration(adminPeriod) * time.Second + } ticker := time.NewTicker(period) defer ticker.Stop() cycles := (time.Duration(seconds)*time.Second + period - 1) / period diff --git a/internal/store/admin_annotation_pool.go b/internal/store/admin_annotation_pool.go index f4bfe79a..e36bdaf4 100644 --- a/internal/store/admin_annotation_pool.go +++ b/internal/store/admin_annotation_pool.go @@ -280,34 +280,6 @@ func (s *MemoryStore) ListAdminAnnotationPool(_ context.Context, org string, q A func adminPoolLookup(from, local, as, org string, projection bson.M) bson.D { return bson.D{{Key: "$lookup", Value: bson.M{"from": from, "localField": local, "foreignField": "id", "pipeline": bson.A{bson.M{"$match": bson.M{"organizationId": normalizedOrganizationID(org)}}, bson.M{"$project": projection}}, "as": as}}} } -func adminPoolPagePipeline(org string, q AdminAnnotationPoolQuery) mongo.Pipeline { - f := adminPoolScopeFilter(org, q) - conditions := bson.A{f} - if q.Pool == "available" { - conditions = append(conditions, adminPoolAvailableFilter()) - } else if q.Pool == "mine" { - conditions = append(conditions, adminPoolMineFilter(q.UserID)) - } - if q.Cursor != nil { - conditions = append(conditions, bson.M{"$or": bson.A{bson.M{"createdAt": bson.M{"$lt": q.Cursor.CreatedAt}}, bson.M{"createdAt": q.Cursor.CreatedAt, "id": bson.M{"$lt": q.Cursor.ID}}}}) - } - pipeline := mongo.Pipeline{{{Key: "$match", Value: bson.M{"$and": conditions}}}, {{Key: "$sort", Value: bson.D{{Key: "createdAt", Value: -1}, {Key: "id", Value: -1}}}}, {{Key: "$project", Value: humanItemListProjection(domain.HumanItemQuery{ListProjection: true})}}} - pipeline = append(pipeline, adminPoolLookup("humanBatches", "batchId", "poolBatches", org, bson.M{"_id": 0, "stages.id": 1}), bson.D{{Key: "$unwind", Value: "$poolBatches"}}, bson.D{{Key: "$match", Value: bson.M{"$expr": bson.M{"$in": bson.A{"$currentStageId", bson.M{"$ifNull": bson.A{"$poolBatches.stages.id", bson.A{}}}}}}}}, bson.D{{Key: "$unset", Value: "poolBatches"}}) - if q.Search != "" || q.Capture != "" || q.Category != "" && q.Category != "all" || q.NodeID != "" || q.ArtifactID != "" { - pipeline = append(pipeline, adminPoolLookup("annotationTasks", "legacyTaskId", "poolTasks", org, bson.M{"_id": 0, "id": 1, "status": 1, "nodeId": 1, "artifactId": 1, "rightArtifactId": 1, "sourceVideo": 1})) - } - if q.Search != "" || q.Capture != "" { - pipeline = append(pipeline, adminPoolLookup("runs", "runId", "poolRuns", org, bson.M{"_id": 0, "id": 1, "taskId": 1, "projectName": 1, "captureDirectories": 1, "inputs.ref": 1, "nodeExecutions.artifacts.id": 1, "nodeExecutions.artifacts.name": 1, "nodeExecutions.artifacts.path": 1, "artifacts.id": 1, "artifacts.name": 1, "artifacts.path": 1}), adminPoolLookup("captureRecords", "poolRuns.taskId", "poolCaptures", org, bson.M{"_id": 0, "captureId": 1, "prefix": 1}), adminPoolLookup("projects", "projectId", "poolProjects", org, bson.M{"_id": 0, "name": 1})) - } - pipeline = append(pipeline, annotationSourceAdmissionStatusPipeline()...) - if q.Pool == "available" { - pipeline = append(pipeline, bson.D{{Key: "$match", Value: bson.M{"_sourceReady": true}}}) - } - if q.Category == "source_failed" { - pipeline = append(pipeline, bson.D{{Key: "$match", Value: bson.M{"_sourceAdmissionStatus": "failed"}}}) - } - return append(pipeline, bson.D{{Key: "$unset", Value: "_sourceReady"}}) -} // Stream a narrow metadata cursor; do not materialize the backlog to find an old // search hit. Unfiltered reads stop after pageSize+1. Context bounds rare scans. @@ -319,22 +291,49 @@ func (s *MongoStore) scanAdminAnnotationPool(ctx context.Context, org string, q return err } defer cur.Close(ctx) - for cur.Next(ctx) { - var row adminPoolRow - if err := cur.Decode(&row); err != nil { + for { + rows := []adminPoolRow{} + for len(rows) < 200 && cur.Next(ctx) { + var row adminPoolRow + if err := cur.Decode(&row); err != nil { + return err + } + rows = append(rows, row) + } + if err := cur.Err(); err != nil { return err } - row.SourceValidationStatus = row.SourceAdmissionStatus - if adminPoolRowMatches(row, q) && !visit(row.HumanItemRecord) { + if len(rows) == 0 { break } + if err := s.humanPoolSearchLegacyPayloads(ctx, org, domain.HumanItemQuery{Search: q.Search, Capture: q.Capture}, rows); err != nil { + return err + } + for _, row := range rows { + row.SourceValidationStatus = row.SourceAdmissionStatus + if adminPoolRowMatches(row, q) && !visit(row.HumanItemRecord) { + return nil + } + } } return cur.Err() } func (s *MongoStore) ListAdminAnnotationPool(ctx context.Context, org string, q AdminAnnotationPoolQuery) ([]domain.HumanItemRecord, error) { - out := []domain.HumanItemRecord{} - err := s.scanAdminAnnotationPool(ctx, org, q, func(i domain.HumanItemRecord) bool { out = append(out, i); return len(out) < q.Limit }) - return out, err + ids := []string{} + statuses := map[string]string{} + err := s.scanAdminAnnotationPool(ctx, org, q, func(i domain.HumanItemRecord) bool { + ids = append(ids, i.ID) + statuses[i.ID] = i.SourceValidationStatus + return q.Limit <= 0 || len(ids) < q.Limit + }) + if err != nil { + return nil, err + } + items, err := s.hydrateHumanPoolSearch(ctx, org, domain.HumanItemQuery{ListProjection: true}, ids) + for n := range items { + items[n].SourceValidationStatus = statuses[items[n].ID] + } + return items, err } func (s *MongoStore) CountAdminAnnotationPools(ctx context.Context, org string, q AdminAnnotationPoolQuery) (AdminAnnotationPoolCounts, error) { @@ -399,8 +398,7 @@ func (s *MongoStore) CountAdminAnnotationPools(ctx context.Context, org string, c.Mine += g.Mine } } - eligible := mongo.Pipeline{{{Key: "$match", Value: bson.M{"$and": bson.A{adminPoolScopeFilter(org, q), adminPoolAvailableFilter()}}}}} - eligible = append(eligible, annotationSourceAdmissionPipeline()...) + eligible := adminPoolPagePipeline(org, AdminAnnotationPoolQuery{ProjectID: q.ProjectID, WorkflowID: q.WorkflowID, RunID: q.RunID, Pool: "available"}) eligible = append(eligible, bson.D{{Key: "$group", Value: bson.M{"_id": bson.M{"batch": "$batchId", "stage": "$currentStageId"}, "available": bson.M{"$sum": 1}}}}) ready, err := s.humanItems.Aggregate(ctx, eligible, options.Aggregate().SetAllowDiskUse(false).SetMaxTime(8*time.Second)) if err != nil { diff --git a/internal/store/admin_annotation_pool_reads.go b/internal/store/admin_annotation_pool_reads.go new file mode 100644 index 00000000..1a7872cb --- /dev/null +++ b/internal/store/admin_annotation_pool_reads.go @@ -0,0 +1,81 @@ +package store + +import ( + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/mongo" +) + +// Batch independent id joins before restoring individual rows. The original +// batch/stage and source-admission decisions still run on every candidate. +// Only routing/search metadata participates in blocking sorts, never drafts. +func adminPoolPagePipeline(org string, q AdminAnnotationPoolQuery) mongo.Pipeline { + conditions := bson.A{adminPoolScopeFilter(org, q)} + if q.Pool == "available" { + conditions = append(conditions, adminPoolAvailableFilter()) + } else if q.Pool == "mine" { + conditions = append(conditions, adminPoolMineFilter(q.UserID)) + } + if q.Cursor != nil { + conditions = append(conditions, bson.M{"$or": bson.A{bson.M{"createdAt": bson.M{"$lt": q.Cursor.CreatedAt}}, bson.M{"createdAt": q.Cursor.CreatedAt, "id": bson.M{"$lt": q.Cursor.ID}}}}) + } + projection := annotationSourceAdmissionProjection() + for _, key := range []string{"createdAt", "updatedAt", "poolSearch", "subject.itemKey", "subject.artifactId", "subject.payload.sliceWorkflow.status", "subject.payload.sliceWorkflow.phase", "pendingCombinedSubmissionId", "assignedUserId"} { + projection[key] = 1 + } + projection["_id"] = 0 + p := mongo.Pipeline{{{Key: "$match", Value: bson.M{"$and": conditions}}}, {{Key: "$project", Value: projection}}, + {{Key: "$setWindowFields", Value: bson.M{"sortBy": bson.D{{Key: "createdAt", Value: -1}}, "output": bson.M{"_row": bson.M{"$documentNumber": bson.M{}}}}}}, + {{Key: "$group", Value: bson.M{"_id": bson.M{"$floor": bson.M{"$divide": bson.A{bson.M{"$subtract": bson.A{"$_row", 1}}, 64}}}, "poolItems": bson.M{"$push": "$$ROOT"}}}}} + // Task/source joins intentionally retain foreign scope fields: admission + // rejects mismatching jobs rather than treating them as absent. + lookup := func(from, local, as string, fields bson.M) bson.D { + return bson.D{{Key: "$lookup", Value: bson.M{"from": from, "localField": local, "foreignField": "id", "pipeline": bson.A{bson.M{"$project": fields}}, "as": as}}} + } + p = append(p, adminPoolLookup("humanBatches", "poolItems.batchId", "poolBatches", org, bson.M{"_id": 0, "id": 1, "stages.id": 1}), + lookup("annotationTasks", "poolItems.legacyTaskId", "poolTasks", bson.M{"_id": 0, "id": 1, "organizationId": 1, "status": 1, "nodeId": 1, "artifactId": 1, "rightArtifactId": 1, "sourceVideo": 1, "sourceIsFullVideo": 1, "templateVersion": 1})) + jobFields := bson.M{"_id": 0, "id": 1, "status": 1, "organizationId": 1, "projectId": 1, "runId": 1, "revalidateSource": 1, "sourceFingerprint": 1, "supersededSourceFingerprint": 1} + p = append(p, lookup("annotationIndexJobs", "poolItems.legacyTaskId", "poolTimeline", jobFields), + bson.D{{Key: "$set", Value: bson.M{"_sourceIDs": bson.M{"$map": bson.M{"input": "$poolItems", "as": "item", "in": bson.M{"$concat": bson.A{"source-v1:", "$$item.organizationId", ":", bson.M{"$ifNull": bson.A{"$$item.processingJob.sourceFingerprint", ""}}}}}}}}}, + lookup("annotationIndexJobs", "_sourceIDs", "poolPrepared", jobFields)) + fields := []string{"poolBatches", "poolTasks", "poolTimeline", "poolPrepared"} + names := q.Search != "" || q.Capture != "" + if names { + p = append(p, humanPoolBatchLookup("runs", "poolItems.runId", "poolRuns", org), adminPoolLookup("captureRecords", "poolRuns.taskId", "poolCaptures", org, bson.M{"_id": 0, "id": 1, "captureId": 1, "prefix": 1}), adminPoolLookup("projects", "poolItems.projectId", "poolProjects", org, bson.M{"_id": 0, "id": 1, "name": 1})) + fields = append(fields, "poolRuns", "poolCaptures", "poolProjects") + } + indexes := bson.M{} + for _, field := range fields { + indexes[field] = humanPoolJoinIndex(field) + } + p = append(p, bson.D{{Key: "$set", Value: indexes}}, bson.D{{Key: "$unwind", Value: "$poolItems"}}) + sourceTask := humanPoolItemJoin("poolTasks", "$poolItems.legacyTaskId") + joined := bson.M{"poolBatches": humanPoolItemJoin("poolBatches", "$poolItems.batchId"), "_sourceTask": sourceTask, "_sourceTimeline": humanPoolItemJoin("poolTimeline", "$poolItems.legacyTaskId"), + "_preparedCandidate": bson.M{"$let": bson.M{"vars": bson.M{"sourceID": bson.M{"$concat": bson.A{"source-v1:", "$poolItems.organizationId", ":", bson.M{"$ifNull": bson.A{"$poolItems.processingJob.sourceFingerprint", ""}}}}}, "in": humanPoolItemJoin("poolPrepared", "$$sourceID")}}, + "poolTasks": bson.M{"$filter": bson.M{"input": sourceTask, "as": "task", "cond": bson.M{"$eq": bson.A{"$$task.organizationId", normalizedOrganizationID(org)}}}}} + if names { + joined["poolRuns"] = humanPoolItemJoin("poolRuns", "$poolItems.runId") + joined["poolProjects"] = humanPoolItemJoin("poolProjects", "$poolItems.projectId") + joined["poolCaptures"] = bson.M{"$let": bson.M{"vars": bson.M{"run": humanPoolIndexedRecord("poolRuns", "$poolItems.runId")}, "in": humanPoolItemJoin("poolCaptures", "$$run.taskId")}} + } + p = append(p, bson.D{{Key: "$replaceWith", Value: bson.M{"$mergeObjects": bson.A{"$poolItems", joined}}}}, bson.D{{Key: "$unwind", Value: "$poolBatches"}}, bson.D{{Key: "$match", Value: bson.M{"$expr": bson.M{"$in": bson.A{"$currentStageId", bson.M{"$ifNull": bson.A{"$poolBatches.stages.id", bson.A{}}}}}}}}) + // Reuse the existing source job selection and decision expressions. Replace + // only their lookups with this batch's indexed results. + for _, stage := range annotationSourceAdmissionStatusPipeline() { + if stage[0].Key == "$lookup" { + continue + } + p = append(p, stage) + if set, ok := stage[0].Value.(bson.M); ok { + if _, ok := set["_sourceJobID"]; ok { + p = append(p, bson.D{{Key: "$set", Value: bson.M{"_sourcePrepared": bson.M{"$cond": bson.A{bson.M{"$ne": bson.A{"$_sourceJobID", ""}}, "$_preparedCandidate", bson.A{}}}}}}) + } + } + } + if q.Pool == "available" { + p = append(p, bson.D{{Key: "$match", Value: bson.M{"_sourceReady": true}}}) + } + if q.Category == "source_failed" { + p = append(p, bson.D{{Key: "$match", Value: bson.M{"_sourceAdmissionStatus": "failed"}}}) + } + return append(p, bson.D{{Key: "$unset", Value: bson.A{"_sourceReady", "_preparedCandidate", "poolBatches", "_row"}}}, bson.D{{Key: "$sort", Value: bson.D{{Key: "createdAt", Value: -1}, {Key: "id", Value: -1}}}}) +} diff --git a/internal/store/admin_annotation_pool_test.go b/internal/store/admin_annotation_pool_test.go index 8dfc6617..05f47ff4 100644 --- a/internal/store/admin_annotation_pool_test.go +++ b/internal/store/admin_annotation_pool_test.go @@ -44,7 +44,11 @@ func TestAdminAnnotationPoolMongoParity(t *testing.T) { _ = mem.UpsertHumanBatch(ctx, batch) _ = s.UpsertHumanBatch(ctx, batch) items, tasks, batchDocs := []any{}, []any{}, []any{} - for n := 0; n < 1500; n++ { + count := 1500 + if os.Getenv("ADMIN_POOL_SCALE") == "1" { + count = 26218 + } + for n := 0; n < count; n++ { i := domain.HumanItemRecord{ID: fmt.Sprintf("i-%04d", n), OrganizationID: org, ProjectID: q.ProjectID, PluginID: "annotation-qa", BatchID: batch.ID, CurrentStageID: "annotation_combined", Status: domain.HumanItemReady, LegacyTaskID: fmt.Sprintf("t-%04d", n), CreatedAt: fmt.Sprintf("2026-10-21T%02d:%02d:%02dZ", n/3600, n/60%60, n%60), Subject: domain.HumanSubjectRef{ItemKey: fmt.Sprintf("CAPTURE_%04d_camera_left_part0001.mp4", n), Payload: map[string]any{"sliceWorkflow": map[string]any{"status": "editing", "phase": "slice"}, "annotationWorkspace": map[string]any{"description": strings.Repeat("x", 8192)}}}} b := batch b.ID = fmt.Sprintf("batch-%04d", n) @@ -100,11 +104,12 @@ func TestAdminAnnotationPoolMongoParity(t *testing.T) { if err != nil || counts != want { t.Fatalf("counts=%+v want=%+v err=%v", counts, want, err) } - t.Logf("1500-item counts: %s %+v", time.Since(start), counts) + t.Logf("%d-item counts: %s %+v", count, time.Since(start), counts) for _, scenario := range []struct{ pool, search, category string }{{"available", "", ""}, {"all", "CAPTURE_0003", ""}, {"all", "", "pending_slice"}, {"mine", "", ""}, {"all", "[.*", ""}, {"all", "客户指定Capture", ""}} { q.Pool, q.Search, q.Category = scenario.pool, scenario.search, scenario.category start = time.Now() got, err := s.ListAdminAnnotationPool(ctx, org, q) + elapsed := time.Since(start) expected, _ := mem.ListAdminAnnotationPool(ctx, org, q) if err != nil { t.Fatal(err) @@ -122,7 +127,10 @@ func TestAdminAnnotationPoolMongoParity(t *testing.T) { if !reflect.DeepEqual(ids(got), ids(expected)) { t.Fatalf("%+v Mongo=%v memory=%v", scenario, ids(got), ids(expected)) } - t.Logf("%+v page=%d elapsed=%s", scenario, len(got), time.Since(start)) + t.Logf("%+v page=%d elapsed=%s", scenario, len(got), elapsed) + if os.Getenv("ADMIN_POOL_SCALE") == "1" && elapsed > 2*time.Second { + t.Errorf("administrator pool exceeds 2s at %d rows: %v", count, elapsed) + } if len(got) > 201 { t.Fatal("unbounded page") } @@ -164,3 +172,111 @@ func TestAdminAnnotationPoolMongoParity(t *testing.T) { t.Fatalf("missing batch count=%+v err=%v", again, err) } } + +// Batch source joins must preserve failed, missing, foreign and superseded +// certificates. Compare the existing admission expression on the same records. +func TestAdminAnnotationPoolMongoAdmissionParity(t *testing.T) { + uri := os.Getenv("POOL_LOCAL_MONGO_URI") + if uri == "" { + t.Skip("isolated local Mongo opt-in") + } + u, e := url.Parse(uri) + if e != nil || u.Host != "127.0.0.1:27018" || u.Scheme != "mongodb" { + t.Fatal("local Mongo 27018 only") + } + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) + defer cancel() + client, e := mongo.Connect(ctx, options.Client().ApplyURI(uri)) + if e != nil { + t.Fatal(e) + } + defer client.Disconnect(context.Background()) + db := client.Database(fmt.Sprintf("admin_pool_admission_%d", time.Now().UnixNano())) + defer db.Drop(context.Background()) + s := NewMongoStore(db) + if e := s.EnsureIndexes(ctx); e != nil { + t.Fatal(e) + } + const org = "admin-admission" + batch := domain.HumanBatchRecord{ID: "batch", OrganizationID: org, PluginID: "annotation-qa", Stages: domain.DefaultFanyuV58HumanStages()} + if e := s.UpsertHumanBatch(ctx, batch); e != nil { + t.Fatal(e) + } + for n := 0; n < 192; n++ { + id := fmt.Sprintf("case-%03d", n) + item := domain.HumanItemRecord{ID: id, OrganizationID: org, ProjectID: "project", RunID: "run", PluginID: "annotation-qa", BatchID: batch.ID, CurrentStageID: "annotation_combined", Status: domain.HumanItemReady, LegacyTaskID: id, CreatedAt: id, ProcessingJob: &domain.AnnotationProcessingJob{SourceIndex: n%2 == 0, SourceFingerprint: id}} + item.Subject.ItemKey = id + if n%3 == 0 { + item.Subject.Payload = map[string]any{"materialization": map[string]any{"sourceIndex": true}} + item.ProcessingJob.SourceFingerprint = "" + } + if e := s.UpsertHumanItem(ctx, item); e != nil { + t.Fatal(e) + } + if n%7 != 0 { + task := domain.AnnotationTaskRecord{ID: id, OrganizationID: org, ProjectID: item.ProjectID, RunID: item.RunID, SourceIsFullVideo: n%2 == 0, TemplateVersion: 6, ArtifactID: "video"} + if _, e := s.annotationTasks.InsertOne(ctx, task); e != nil { + t.Fatal(e) + } + } + for k, jobID := range []string{id, "source-v1:" + org + ":" + id} { + if n%5 == k { + continue + } + job := domain.AnnotationIndexJob{ID: jobID, OrganizationID: org, ProjectID: item.ProjectID, RunID: item.RunID, Status: []string{"completed", "failed", "running"}[n%3]} + if n%11 == 0 { + job.OrganizationID = "foreign" + } + if k == 0 && n%4 == 0 { + job.RevalidateSource = true + job.SourceFingerprint = "repaired" + job.SupersededSourceFingerprint = id + } + if _, e := db.Collection("annotationIndexJobs").InsertOne(ctx, job); e != nil { + t.Fatal(e) + } + } + } + legacy := mongo.Pipeline{{{Key: "$match", Value: adminPoolScopeFilter(org, AdminAnnotationPoolQuery{})}}} + legacy = append(legacy, annotationSourceAdmissionStatusPipeline()...) + read := func(p mongo.Pipeline) map[string]string { + t.Helper() + cur, e := s.humanItems.Aggregate(ctx, p) + if e != nil { + t.Fatal(e) + } + defer cur.Close(ctx) + rows, e := decodeMany[adminPoolRow](ctx, cur) + if e != nil { + t.Fatal(e) + } + out := map[string]string{} + for _, r := range rows { + out[r.ID] = r.SourceAdmissionStatus + } + return out + } + expected := read(legacy) + got := read(adminPoolPagePipeline(org, AdminAnnotationPoolQuery{Pool: "all"})) + if len(expected) != 192 || !reflect.DeepEqual(expected, got) { + t.Fatalf("batch admission changed: expected=%v actual=%v", expected, got) + } + available, e := s.ListAdminAnnotationPool(ctx, org, AdminAnnotationPoolQuery{Pool: "available", Limit: 300}) + if e != nil { + t.Fatal(e) + } + ready := 0 + for _, status := range expected { + if status == "" { + ready++ + } + } + if len(available) != ready { + t.Fatalf("available=%d expected=%d", len(available), ready) + } + for _, item := range available { + if expected[item.ID] != "" { + t.Fatalf("unready source exposed: %s", item.ID) + } + } +} diff --git a/scripts/verification-scopes.json b/scripts/verification-scopes.json index ca3d5291..9c42ca16 100644 --- a/scripts/verification-scopes.json +++ b/scripts/verification-scopes.json @@ -246,7 +246,7 @@ "tests/frontend/annotation-task-pools.test.mjs" ], "go": { - "./internal/api": ["^TestHumanPoolSearch", "^TestHumanPoolQCProjectScope"] + "./internal/api": ["^TestHumanPoolSearch", "^TestHumanPoolQCProjectScope", "^TestAdminAnnotationPool"] }, "task": [ { @@ -271,7 +271,7 @@ "test", "./internal/store", "-run", - "^TestHumanPoolSearchMongo", + "^(TestHumanPoolSearchMongo|TestAdminAnnotationPoolMongo)", "-count=1", "-timeout=3m", "-v" @@ -292,7 +292,25 @@ "-timeout=2m", "-v" ], - "env": { "ANNOTATION_BUFFER_MONGO_STORE": "1" }, + "env": { + "ANNOTATION_BUFFER_MONGO_STORE": "1" + }, + "tests": true + }, + { + "argv": [ + "go", + "test", + "./internal/api", + "-run", + "^TestAdminAnnotationPoolBrowserReplica$", + "-count=1", + "-timeout=3m", + "-v" + ], + "env": { + "ADMIN_POOL_BROWSER_TEST": "1" + }, "tests": true } ]