⚡ Batch administrator pool metadata joins

This commit is contained in:
Codex 2026-11-09 21:51:55 -08:00
parent 4562296985
commit dfb9d7bc65
9 changed files with 376 additions and 46 deletions

View File

@ -52,3 +52,17 @@ Follow-up base: 7a1231c7 (live API/processor source; freshness checked against o
- Final gate verify-1794271033888668000 selected-checks-passed; flight race passed. Six executions have identical harness/candidate patch hashes. Final source freshness includes origin/dev and live commit.
- Evidence: final-acceptance-summary.json, mongo8-isolated-formal-comparison.json, six formal reports and resource/execution logs in private diagnostics directory. Docs record failures, corrected protocol, limits, five-index authorization and human-only CAS rollout.
- No new push/PR/deployment, production index/backfill or customer writes. Old release remains deployed_pending_acceptance. Local task complete; production enablement separate.
### Administrator pool timeout follow-up
Base: ae52e5d4; freshness includes origin/dev and live7a1231c7. Same authorized follow-up worktree; no production mutation/deployment.
- Production read-only logs: admin-pool500 at8000/8006ms. Separate endpoint missed by earlier81-actor harness (admin only read workforce).
- Red loop: ADMIN_POOL_SCALE=1 TestAdminAnnotationPoolMongoParity,26218 isolated synthetic records; searches6.23–6.51s breach2s. Counts~2.04s. Evidence admin-scale-red.log.
- Batch64 metadata joins for batch/stage validation, search names and source jobs; reuse original source-selection/decision expressions. Match legacy payloads in bounded batches; hydrate only matched page and preserve response shape/order. No new index, cache permission or task writes.
- Batch display priming for admin response. Fix missing nodeId in per-actor counts cache scope. Existing5s expiry and server authorization unchanged.
- Scale green searches0.90–1.02s,counts0.81s; equal reference IDs/counts/paging.192-source-case parity passed, including missing/failed/foreign/superseded proof.
- Gate verify-1794289069683716000 selected-checks-passed, no unmapped/deferred: admin/QC/acceptance frontend+HTTP+Mongo/browser, official Go baseline gate, local81 read mix.
- Post-gate changes only to opt-in harness scheduling for repeated admin measurements; production Go unchanged. First extended81 mix3410 reads,0errors/writes; admin searches0.75–1.56s but all-project available had one3.53s sample. Kept as diagnostic, not stable p95 evidence. Higher-frequency repetitions in progress.
- Repeated administrator supplement finished:30s warmup+200s steady,1 admin request/second,81 actors;5920 measured reads,0errors/writes. Ten admin scenarios each20samples. All-project availablep95=311.603ms, admin searchp95=475.048–670.738ms; other search maxp95=724.445ms. Same actor/endpoint fingerprints stable. Earlier3.53s sample retained; production cold-load/capacity acceptance pending. Private final-summary.json records limits. No push/PR/deployment or production index/data changes.

View File

@ -11,3 +11,21 @@
总数按批次/阶段聚合紧凑计数,再用既有批次权限投影校验;不读取动作正文或草稿来计数。总数使用有身份和范围隔离、容量上限的 5 秒进程缓存,当前进程的 API 写入会使缓存失效;其他副本与后台写入最多延迟到缓存到期。列表不缓存。搜索输入合并 350 毫秒,新搜索与范围切换取消过期请求;旧任务搜索在服务端流式读取投影元数据,查询和范围统计受 8 秒期限限制,不向浏览器传整池,也不把失败当成零条。计数与页面是相邻的只读操作,并非冻结生产时刻的结算快照。
不新增统计集合、数据库字段或索引,不改领取、标注、返工、审核及交付结果。生产环境仍需单独发布与验收。
## 2026-11-09 管理员查询超时修复
管理员任务池使用独立的 `/admin-pool`,与质检/验收的 `human-item-summaries` 搜索不同。线上只读日志确认前者出现500及8000/8006毫秒超时;截图中的HTTP非安全连接提示是连续工作入口的另一项限制,不能解释该接口的后端超时。
旧路径按每条候选关联批次、运行、采集及来源校验任务,搜索旧任务或没有命中时会扫完整个范围;总数读取在页面列表之前执行,失败会阻断首屏。管理员显示名称补充也未调用按页预取。此前81人读取对照中的管理员只请求统计面板,不能作为该独立接口已验证的证据。
本次改为每64条窄元数据批量关联,逐条恢复原批次/阶段、来源凭证及范围判定,然后按原创建时间和ID恢复顺序。仅命中的页面按ID读取原列表字段;缺少或过期摘要的历史正文仍按批读取,不丢旧名称。保留搜索字面匹配、全部任务的原环节范围、游标、跨组织隔离及缺失/失败/已替代来源凭证规则。没有新增索引、字段或生产补写要求。
管理员显示名称预取改为整页一次;5秒计数缓存保留,补齐节点范围键,避免不同节点复用错误总数。列表仍不缓存,权限检查仍在计数缓存之前执行。其他角色不经过该管理员入口:质检/验收继续使用既有批量搜索路径,标注个人池及来源准入继续使用其有界读取路径。本次回归覆盖这些入口,不能据此宣称所有页面或网络链路都没有性能瓶颈。
验证入口:`ADMIN_POOL_SCALE=1 POOL_LOCAL_MONGO_URI=mongodb://127.0.0.1:27018 go test ./internal/store -run '^TestAdminAnnotationPoolMongoParity$' -count=1 -v`。只接受本机27018,创建并清理独立合成库;26,218条记录对照内存参考实现,覆盖历史命中、无命中、特殊字符、自定义Capture、状态和游标。192组来源凭证用原表达式对照批量表达式;管理员HTTP及真实浏览器回归验证权限、范围计数、翻页、取消与失败重试。
额外混合读取使用 `TestPollPoolReadMixLocal`,在生产规模合成数据及81人背景下显式设置 `POLL_POOL_ADMIN_TASKS=1`、`POLL_POOL_CLAIM_COUNTS=1`;`POLL_POOL_ADMIN_PERIOD_SECONDS=1` 用于提高单个管理员的重复采样频率,默认仍为12秒。它不改变生产轮询。计入管理员的全部/单项目、可领取、我的任务、状态、命中及无命中。该短时补验排除媒体、写作业、网关和公网传输,不替代生产容量验收。
结果:26,218条全量有效候选的历史命中/无命中搜索由约6.2~6.5秒降至0.90~1.02秒,计数由约2.04秒降至0.81秒。首轮81人补验的某次全部项目可领取为3.53秒,保留为异常样本,不用于稳定p95结论。后续提高管理员重复频率:预热30秒、稳态200秒,10种管理员场景各20次;全部项目可领取API p95为312ms,管理员搜索p95为475~671ms,质检/验收搜索最高p95为724ms。5,920次稳态读取零HTTP错误、零业务写入,同账号同接口任务/数量/权限及追溯摘要稳定。该时间不含公网传输,旧响应格式仍保留;生产冷加载及容量仍待上线只读复测。
任务门禁 `verify-1794289069683716000` 通过,无未映射或待执行项,含管理员与审查浏览器、Mongo历史兼容、角色权限及项目Go基线门禁。之后仅调整测试脚本的管理员采样周期,API/Store实现未变;调整后的脚本通过上述重复采样。私有证据在工作树 `local-data/diagnostics/acceptance-search/admin-*` 及 `local-data/diagnostics/admin-pool-timeout/final-summary.json`。本轮只读查线上、仅在隔离合成库测试,未推送、部署或修改客户数据。

View File

@ -57,7 +57,7 @@ func (s *server) adminAnnotationCounts(w http.ResponseWriter, r *http.Request, q
copyURL := *r.URL
countRequest.URL = &copyURL
values := url.Values{}
for _, key := range []string{"projectId", "workflowId", "runId", "nodeId", "artifactId", "capture"} {
for _, key := range []string{"projectId", "workflowId", "runId", "nodeId", "artifactId", "nodeId", "capture"} {
if value := r.URL.Query().Get(key); value != "" {
values.Set(key, value)
}
@ -110,6 +110,10 @@ func (s *server) handleAdminAnnotationPool(w http.ResponseWriter, r *http.Reques
page.NextCursor = encodeHumanItemCursor(&domain.HumanItemCursor{CreatedAt: last.CreatedAt, ID: last.ID})
}
resolver := newHumanDisplayResolver(s)
if err := resolver.primeRuns(r.Context(), page.Items); err != nil {
renderError(w, 500, err)
return
}
for i := range page.Items {
if err := resolver.enrichItem(r.Context(), &page.Items[i]); err != nil {
renderError(w, 500, err)

View File

@ -9,6 +9,7 @@ import (
"fmt"
"net/http"
"net/url"
"strings"
"testing"
"time"
)
@ -144,3 +145,63 @@ func TestAdminAnnotationPoolFreshPermissionsCacheAndFailures(t *testing.T) {
t.Fatal("page fetched after count failure")
}
}
func TestAdminAnnotationPoolBatchesPageDisplayReads(t *testing.T) {
f := newFanyuV5FourStageFixture(t, domain.FanyuV5StageSemanticAnnotation, domain.HumanItemReady, "", 0)
admin := domain.UserRecord{ID: "admin-batch-display", OrganizationID: domain.DefaultOrganizationID, Role: domain.UserRoleAdmin, Status: domain.UserStatusActive}
for n := 0; n < 20; n++ {
id := fmt.Sprint(n)
item := f.item
item.ID = "display-item-" + id
item.RunID = "display-run-" + id
item.LegacyTaskID = ""
if err := f.store.UpsertHumanItem(f.ctx, item); err != nil {
t.Fatal(err)
}
if err := f.store.UpsertRun(f.ctx, domain.RunRecord{ID: item.RunID, OrganizationID: item.OrganizationID, TaskID: "capture-" + id}); err != nil {
t.Fatal(err)
}
if err := f.store.UpsertCaptureRecord(f.ctx, domain.CaptureRecord{ID: "capture-" + id, OrganizationID: item.OrganizationID, CaptureID: "directory-" + id}); err != nil {
t.Fatal(err)
}
}
probe := &displayCaptureReadStore{Store: f.store}
f.handler = NewHandler(probe, nil)
r := f.request(http.MethodGet, "/api/plugins/annotation-qa/admin-pool?pool=all&limit=200", admin, nil)
if r.Code != 200 {
t.Fatalf("%d %s", r.Code, r.Body)
}
if probe.batches != 1 || probe.singles != 0 {
t.Fatalf("display fan-out: batch=%d single=%d", probe.batches, probe.singles)
}
var body struct {
Items []domain.HumanItemRecord `json:"items"`
}
if err := json.Unmarshal(r.Body.Bytes(), &body); err != nil {
t.Fatal(err)
}
if len(body.Items) != 21 {
t.Fatalf("lost page entries: %d", len(body.Items))
}
for _, i := range body.Items {
if strings.HasPrefix(i.ID, "display-item-") && (i.Display == nil || len(i.Display.CaptureDirectories) != 1 || i.Display.CaptureDirectories[0] != "directory-"+strings.TrimPrefix(i.ID, "display-item-")) {
t.Fatalf("display changed: %+v", i)
}
}
}
func TestAdminAnnotationPoolCountsSeparateNodeScopes(t *testing.T) {
f := newFanyuV5FourStageFixture(t, domain.FanyuV5StageSemanticAnnotation, domain.HumanItemReady, "", 0)
p := &adminPoolProbe{Store: f.store}
f.handler = NewHandler(p, nil)
admin := domain.UserRecord{ID: "admin-node", OrganizationID: domain.DefaultOrganizationID, Role: domain.UserRoleAdmin, Status: domain.UserStatusActive}
for _, node := range []string{"node-a", "node-b", "node-a"} {
r := f.request(http.MethodGet, "/api/plugins/annotation-qa/admin-pool?pool=all&nodeId="+node, admin, nil)
if r.Code != 200 {
t.Fatalf("%d %s", r.Code, r.Body)
}
}
if p.counts != 2 {
t.Fatalf("different node scopes shared counts: %d", p.counts)
}
}

View File

@ -78,6 +78,7 @@ func TestPollPoolReadMixLocal(t *testing.T) {
}
return n
}
adminPeriod := seconds("POLL_POOL_ADMIN_PERIOD_SECONDS", 12)
warmup, duration := seconds("POLL_POOL_WARMUP_SECONDS", 5), seconds("POLL_POOL_DURATION_SECONDS", 30)
annotators := seconds("POLL_POOL_ANNOTATORS", 60)
quality := seconds("POLL_POOL_QUALITY_REVIEWERS", 12)
@ -602,6 +603,21 @@ func TestPollPoolReadMixLocal(t *testing.T) {
return result
}
if a.Role == "admin" {
if os.Getenv("POLL_POOL_ADMIN_TASKS") == "1" {
base := "/api/plugins/annotation-qa/admin-pool?limit=200" + scope
suffix, key := "&pool=available", "available"
switch scenario % 5 {
case 1:
suffix, key = "&pool=all&search=synthetic-available-", "search"
case 2:
suffix, key = "&pool=all&search=__production_shape_no_match__", "search.nohit"
case 3:
suffix, key = "&pool=mine", "mine"
case 4:
suffix, key = "&pool=all&category=pending_slice", "category"
}
return map[string]string{"admin." + scopeName + "." + key: base + suffix}
}
return map[string]string{"admin.workforce": "/api/dashboard/workforce?from=2026-10-01T00:00:00Z&to=2026-10-28T00:00:00Z"}
}
if pageShape {
@ -689,6 +705,7 @@ func TestPollPoolReadMixLocal(t *testing.T) {
return result
}
fixtureSpec := fmt.Sprintf("isolated-preparation-v5|%d|%d|%d|%d|%d|%d|%d|%t|%t|%t|%s|%s", annotators, quality, acceptance, availablePerStage, runPayloadKiB, taskPayloadKiB, authorHistory, productionShape, readModels, claimCounts, authorizationState, os.Getenv("POLL_POOL_SEARCH_INDEXES"))
fixtureSpec += "|admin-tasks=" + os.Getenv("POLL_POOL_ADMIN_TASKS")
fixtureDigest := fmt.Sprintf("%x", sha256.Sum256([]byte(fixtureSpec)))
if reuseFixture {
var manifest struct {
@ -735,7 +752,7 @@ func TestPollPoolReadMixLocal(t *testing.T) {
}
t.Fatalf("preflight %s status=%d", key, result.Code)
}
if (a.Role == "annotation" || a.Role == "admin" || strings.HasSuffix(key, ".available") || strings.HasSuffix(key, ".mine") && !strings.Contains(key, ".acceptance.")) && result.Rows == 0 {
if (a.Role == "annotation" || a.Role == "admin" && os.Getenv("POLL_POOL_ADMIN_TASKS") != "1" || strings.HasSuffix(key, ".available") || strings.HasSuffix(key, ".mine") && !strings.Contains(key, ".acceptance.")) && result.Rows == 0 {
t.Fatalf("preflight %s unexpectedly empty for %s", key, a.User.ID)
}
}
@ -754,6 +771,9 @@ func TestPollPoolReadMixLocal(t *testing.T) {
if a.Role == "annotation" {
period = 3 * time.Second
}
if a.Role == "admin" && os.Getenv("POLL_POOL_ADMIN_TASKS") == "1" {
period = time.Duration(adminPeriod) * time.Second
}
ticker := time.NewTicker(period)
defer ticker.Stop()
cycles := (time.Duration(seconds)*time.Second + period - 1) / period

View File

@ -280,34 +280,6 @@ func (s *MemoryStore) ListAdminAnnotationPool(_ context.Context, org string, q A
func adminPoolLookup(from, local, as, org string, projection bson.M) bson.D {
return bson.D{{Key: "$lookup", Value: bson.M{"from": from, "localField": local, "foreignField": "id", "pipeline": bson.A{bson.M{"$match": bson.M{"organizationId": normalizedOrganizationID(org)}}, bson.M{"$project": projection}}, "as": as}}}
}
func adminPoolPagePipeline(org string, q AdminAnnotationPoolQuery) mongo.Pipeline {
f := adminPoolScopeFilter(org, q)
conditions := bson.A{f}
if q.Pool == "available" {
conditions = append(conditions, adminPoolAvailableFilter())
} else if q.Pool == "mine" {
conditions = append(conditions, adminPoolMineFilter(q.UserID))
}
if q.Cursor != nil {
conditions = append(conditions, bson.M{"$or": bson.A{bson.M{"createdAt": bson.M{"$lt": q.Cursor.CreatedAt}}, bson.M{"createdAt": q.Cursor.CreatedAt, "id": bson.M{"$lt": q.Cursor.ID}}}})
}
pipeline := mongo.Pipeline{{{Key: "$match", Value: bson.M{"$and": conditions}}}, {{Key: "$sort", Value: bson.D{{Key: "createdAt", Value: -1}, {Key: "id", Value: -1}}}}, {{Key: "$project", Value: humanItemListProjection(domain.HumanItemQuery{ListProjection: true})}}}
pipeline = append(pipeline, adminPoolLookup("humanBatches", "batchId", "poolBatches", org, bson.M{"_id": 0, "stages.id": 1}), bson.D{{Key: "$unwind", Value: "$poolBatches"}}, bson.D{{Key: "$match", Value: bson.M{"$expr": bson.M{"$in": bson.A{"$currentStageId", bson.M{"$ifNull": bson.A{"$poolBatches.stages.id", bson.A{}}}}}}}}, bson.D{{Key: "$unset", Value: "poolBatches"}})
if q.Search != "" || q.Capture != "" || q.Category != "" && q.Category != "all" || q.NodeID != "" || q.ArtifactID != "" {
pipeline = append(pipeline, adminPoolLookup("annotationTasks", "legacyTaskId", "poolTasks", org, bson.M{"_id": 0, "id": 1, "status": 1, "nodeId": 1, "artifactId": 1, "rightArtifactId": 1, "sourceVideo": 1}))
}
if q.Search != "" || q.Capture != "" {
pipeline = append(pipeline, adminPoolLookup("runs", "runId", "poolRuns", org, bson.M{"_id": 0, "id": 1, "taskId": 1, "projectName": 1, "captureDirectories": 1, "inputs.ref": 1, "nodeExecutions.artifacts.id": 1, "nodeExecutions.artifacts.name": 1, "nodeExecutions.artifacts.path": 1, "artifacts.id": 1, "artifacts.name": 1, "artifacts.path": 1}), adminPoolLookup("captureRecords", "poolRuns.taskId", "poolCaptures", org, bson.M{"_id": 0, "captureId": 1, "prefix": 1}), adminPoolLookup("projects", "projectId", "poolProjects", org, bson.M{"_id": 0, "name": 1}))
}
pipeline = append(pipeline, annotationSourceAdmissionStatusPipeline()...)
if q.Pool == "available" {
pipeline = append(pipeline, bson.D{{Key: "$match", Value: bson.M{"_sourceReady": true}}})
}
if q.Category == "source_failed" {
pipeline = append(pipeline, bson.D{{Key: "$match", Value: bson.M{"_sourceAdmissionStatus": "failed"}}})
}
return append(pipeline, bson.D{{Key: "$unset", Value: "_sourceReady"}})
}
// Stream a narrow metadata cursor; do not materialize the backlog to find an old
// search hit. Unfiltered reads stop after pageSize+1. Context bounds rare scans.
@ -319,22 +291,49 @@ func (s *MongoStore) scanAdminAnnotationPool(ctx context.Context, org string, q
return err
}
defer cur.Close(ctx)
for cur.Next(ctx) {
var row adminPoolRow
if err := cur.Decode(&row); err != nil {
for {
rows := []adminPoolRow{}
for len(rows) < 200 && cur.Next(ctx) {
var row adminPoolRow
if err := cur.Decode(&row); err != nil {
return err
}
rows = append(rows, row)
}
if err := cur.Err(); err != nil {
return err
}
row.SourceValidationStatus = row.SourceAdmissionStatus
if adminPoolRowMatches(row, q) && !visit(row.HumanItemRecord) {
if len(rows) == 0 {
break
}
if err := s.humanPoolSearchLegacyPayloads(ctx, org, domain.HumanItemQuery{Search: q.Search, Capture: q.Capture}, rows); err != nil {
return err
}
for _, row := range rows {
row.SourceValidationStatus = row.SourceAdmissionStatus
if adminPoolRowMatches(row, q) && !visit(row.HumanItemRecord) {
return nil
}
}
}
return cur.Err()
}
func (s *MongoStore) ListAdminAnnotationPool(ctx context.Context, org string, q AdminAnnotationPoolQuery) ([]domain.HumanItemRecord, error) {
out := []domain.HumanItemRecord{}
err := s.scanAdminAnnotationPool(ctx, org, q, func(i domain.HumanItemRecord) bool { out = append(out, i); return len(out) < q.Limit })
return out, err
ids := []string{}
statuses := map[string]string{}
err := s.scanAdminAnnotationPool(ctx, org, q, func(i domain.HumanItemRecord) bool {
ids = append(ids, i.ID)
statuses[i.ID] = i.SourceValidationStatus
return q.Limit <= 0 || len(ids) < q.Limit
})
if err != nil {
return nil, err
}
items, err := s.hydrateHumanPoolSearch(ctx, org, domain.HumanItemQuery{ListProjection: true}, ids)
for n := range items {
items[n].SourceValidationStatus = statuses[items[n].ID]
}
return items, err
}
func (s *MongoStore) CountAdminAnnotationPools(ctx context.Context, org string, q AdminAnnotationPoolQuery) (AdminAnnotationPoolCounts, error) {
@ -399,8 +398,7 @@ func (s *MongoStore) CountAdminAnnotationPools(ctx context.Context, org string,
c.Mine += g.Mine
}
}
eligible := mongo.Pipeline{{{Key: "$match", Value: bson.M{"$and": bson.A{adminPoolScopeFilter(org, q), adminPoolAvailableFilter()}}}}}
eligible = append(eligible, annotationSourceAdmissionPipeline()...)
eligible := adminPoolPagePipeline(org, AdminAnnotationPoolQuery{ProjectID: q.ProjectID, WorkflowID: q.WorkflowID, RunID: q.RunID, Pool: "available"})
eligible = append(eligible, bson.D{{Key: "$group", Value: bson.M{"_id": bson.M{"batch": "$batchId", "stage": "$currentStageId"}, "available": bson.M{"$sum": 1}}}})
ready, err := s.humanItems.Aggregate(ctx, eligible, options.Aggregate().SetAllowDiskUse(false).SetMaxTime(8*time.Second))
if err != nil {

View File

@ -0,0 +1,81 @@
package store
import (
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
)
// Batch independent id joins before restoring individual rows. The original
// batch/stage and source-admission decisions still run on every candidate.
// Only routing/search metadata participates in blocking sorts, never drafts.
func adminPoolPagePipeline(org string, q AdminAnnotationPoolQuery) mongo.Pipeline {
conditions := bson.A{adminPoolScopeFilter(org, q)}
if q.Pool == "available" {
conditions = append(conditions, adminPoolAvailableFilter())
} else if q.Pool == "mine" {
conditions = append(conditions, adminPoolMineFilter(q.UserID))
}
if q.Cursor != nil {
conditions = append(conditions, bson.M{"$or": bson.A{bson.M{"createdAt": bson.M{"$lt": q.Cursor.CreatedAt}}, bson.M{"createdAt": q.Cursor.CreatedAt, "id": bson.M{"$lt": q.Cursor.ID}}}})
}
projection := annotationSourceAdmissionProjection()
for _, key := range []string{"createdAt", "updatedAt", "poolSearch", "subject.itemKey", "subject.artifactId", "subject.payload.sliceWorkflow.status", "subject.payload.sliceWorkflow.phase", "pendingCombinedSubmissionId", "assignedUserId"} {
projection[key] = 1
}
projection["_id"] = 0
p := mongo.Pipeline{{{Key: "$match", Value: bson.M{"$and": conditions}}}, {{Key: "$project", Value: projection}},
{{Key: "$setWindowFields", Value: bson.M{"sortBy": bson.D{{Key: "createdAt", Value: -1}}, "output": bson.M{"_row": bson.M{"$documentNumber": bson.M{}}}}}},
{{Key: "$group", Value: bson.M{"_id": bson.M{"$floor": bson.M{"$divide": bson.A{bson.M{"$subtract": bson.A{"$_row", 1}}, 64}}}, "poolItems": bson.M{"$push": "$$ROOT"}}}}}
// Task/source joins intentionally retain foreign scope fields: admission
// rejects mismatching jobs rather than treating them as absent.
lookup := func(from, local, as string, fields bson.M) bson.D {
return bson.D{{Key: "$lookup", Value: bson.M{"from": from, "localField": local, "foreignField": "id", "pipeline": bson.A{bson.M{"$project": fields}}, "as": as}}}
}
p = append(p, adminPoolLookup("humanBatches", "poolItems.batchId", "poolBatches", org, bson.M{"_id": 0, "id": 1, "stages.id": 1}),
lookup("annotationTasks", "poolItems.legacyTaskId", "poolTasks", bson.M{"_id": 0, "id": 1, "organizationId": 1, "status": 1, "nodeId": 1, "artifactId": 1, "rightArtifactId": 1, "sourceVideo": 1, "sourceIsFullVideo": 1, "templateVersion": 1}))
jobFields := bson.M{"_id": 0, "id": 1, "status": 1, "organizationId": 1, "projectId": 1, "runId": 1, "revalidateSource": 1, "sourceFingerprint": 1, "supersededSourceFingerprint": 1}
p = append(p, lookup("annotationIndexJobs", "poolItems.legacyTaskId", "poolTimeline", jobFields),
bson.D{{Key: "$set", Value: bson.M{"_sourceIDs": bson.M{"$map": bson.M{"input": "$poolItems", "as": "item", "in": bson.M{"$concat": bson.A{"source-v1:", "$$item.organizationId", ":", bson.M{"$ifNull": bson.A{"$$item.processingJob.sourceFingerprint", ""}}}}}}}}},
lookup("annotationIndexJobs", "_sourceIDs", "poolPrepared", jobFields))
fields := []string{"poolBatches", "poolTasks", "poolTimeline", "poolPrepared"}
names := q.Search != "" || q.Capture != ""
if names {
p = append(p, humanPoolBatchLookup("runs", "poolItems.runId", "poolRuns", org), adminPoolLookup("captureRecords", "poolRuns.taskId", "poolCaptures", org, bson.M{"_id": 0, "id": 1, "captureId": 1, "prefix": 1}), adminPoolLookup("projects", "poolItems.projectId", "poolProjects", org, bson.M{"_id": 0, "id": 1, "name": 1}))
fields = append(fields, "poolRuns", "poolCaptures", "poolProjects")
}
indexes := bson.M{}
for _, field := range fields {
indexes[field] = humanPoolJoinIndex(field)
}
p = append(p, bson.D{{Key: "$set", Value: indexes}}, bson.D{{Key: "$unwind", Value: "$poolItems"}})
sourceTask := humanPoolItemJoin("poolTasks", "$poolItems.legacyTaskId")
joined := bson.M{"poolBatches": humanPoolItemJoin("poolBatches", "$poolItems.batchId"), "_sourceTask": sourceTask, "_sourceTimeline": humanPoolItemJoin("poolTimeline", "$poolItems.legacyTaskId"),
"_preparedCandidate": bson.M{"$let": bson.M{"vars": bson.M{"sourceID": bson.M{"$concat": bson.A{"source-v1:", "$poolItems.organizationId", ":", bson.M{"$ifNull": bson.A{"$poolItems.processingJob.sourceFingerprint", ""}}}}}, "in": humanPoolItemJoin("poolPrepared", "$$sourceID")}},
"poolTasks": bson.M{"$filter": bson.M{"input": sourceTask, "as": "task", "cond": bson.M{"$eq": bson.A{"$$task.organizationId", normalizedOrganizationID(org)}}}}}
if names {
joined["poolRuns"] = humanPoolItemJoin("poolRuns", "$poolItems.runId")
joined["poolProjects"] = humanPoolItemJoin("poolProjects", "$poolItems.projectId")
joined["poolCaptures"] = bson.M{"$let": bson.M{"vars": bson.M{"run": humanPoolIndexedRecord("poolRuns", "$poolItems.runId")}, "in": humanPoolItemJoin("poolCaptures", "$$run.taskId")}}
}
p = append(p, bson.D{{Key: "$replaceWith", Value: bson.M{"$mergeObjects": bson.A{"$poolItems", joined}}}}, bson.D{{Key: "$unwind", Value: "$poolBatches"}}, bson.D{{Key: "$match", Value: bson.M{"$expr": bson.M{"$in": bson.A{"$currentStageId", bson.M{"$ifNull": bson.A{"$poolBatches.stages.id", bson.A{}}}}}}}})
// Reuse the existing source job selection and decision expressions. Replace
// only their lookups with this batch's indexed results.
for _, stage := range annotationSourceAdmissionStatusPipeline() {
if stage[0].Key == "$lookup" {
continue
}
p = append(p, stage)
if set, ok := stage[0].Value.(bson.M); ok {
if _, ok := set["_sourceJobID"]; ok {
p = append(p, bson.D{{Key: "$set", Value: bson.M{"_sourcePrepared": bson.M{"$cond": bson.A{bson.M{"$ne": bson.A{"$_sourceJobID", ""}}, "$_preparedCandidate", bson.A{}}}}}})
}
}
}
if q.Pool == "available" {
p = append(p, bson.D{{Key: "$match", Value: bson.M{"_sourceReady": true}}})
}
if q.Category == "source_failed" {
p = append(p, bson.D{{Key: "$match", Value: bson.M{"_sourceAdmissionStatus": "failed"}}})
}
return append(p, bson.D{{Key: "$unset", Value: bson.A{"_sourceReady", "_preparedCandidate", "poolBatches", "_row"}}}, bson.D{{Key: "$sort", Value: bson.D{{Key: "createdAt", Value: -1}, {Key: "id", Value: -1}}}})
}

View File

@ -44,7 +44,11 @@ func TestAdminAnnotationPoolMongoParity(t *testing.T) {
_ = mem.UpsertHumanBatch(ctx, batch)
_ = s.UpsertHumanBatch(ctx, batch)
items, tasks, batchDocs := []any{}, []any{}, []any{}
for n := 0; n < 1500; n++ {
count := 1500
if os.Getenv("ADMIN_POOL_SCALE") == "1" {
count = 26218
}
for n := 0; n < count; n++ {
i := domain.HumanItemRecord{ID: fmt.Sprintf("i-%04d", n), OrganizationID: org, ProjectID: q.ProjectID, PluginID: "annotation-qa", BatchID: batch.ID, CurrentStageID: "annotation_combined", Status: domain.HumanItemReady, LegacyTaskID: fmt.Sprintf("t-%04d", n), CreatedAt: fmt.Sprintf("2026-10-21T%02d:%02d:%02dZ", n/3600, n/60%60, n%60), Subject: domain.HumanSubjectRef{ItemKey: fmt.Sprintf("CAPTURE_%04d_camera_left_part0001.mp4", n), Payload: map[string]any{"sliceWorkflow": map[string]any{"status": "editing", "phase": "slice"}, "annotationWorkspace": map[string]any{"description": strings.Repeat("x", 8192)}}}}
b := batch
b.ID = fmt.Sprintf("batch-%04d", n)
@ -100,11 +104,12 @@ func TestAdminAnnotationPoolMongoParity(t *testing.T) {
if err != nil || counts != want {
t.Fatalf("counts=%+v want=%+v err=%v", counts, want, err)
}
t.Logf("1500-item counts: %s %+v", time.Since(start), counts)
t.Logf("%d-item counts: %s %+v", count, time.Since(start), counts)
for _, scenario := range []struct{ pool, search, category string }{{"available", "", ""}, {"all", "CAPTURE_0003", ""}, {"all", "", "pending_slice"}, {"mine", "", ""}, {"all", "[.*", ""}, {"all", "客户指定Capture", ""}} {
q.Pool, q.Search, q.Category = scenario.pool, scenario.search, scenario.category
start = time.Now()
got, err := s.ListAdminAnnotationPool(ctx, org, q)
elapsed := time.Since(start)
expected, _ := mem.ListAdminAnnotationPool(ctx, org, q)
if err != nil {
t.Fatal(err)
@ -122,7 +127,10 @@ func TestAdminAnnotationPoolMongoParity(t *testing.T) {
if !reflect.DeepEqual(ids(got), ids(expected)) {
t.Fatalf("%+v Mongo=%v memory=%v", scenario, ids(got), ids(expected))
}
t.Logf("%+v page=%d elapsed=%s", scenario, len(got), time.Since(start))
t.Logf("%+v page=%d elapsed=%s", scenario, len(got), elapsed)
if os.Getenv("ADMIN_POOL_SCALE") == "1" && elapsed > 2*time.Second {
t.Errorf("administrator pool exceeds 2s at %d rows: %v", count, elapsed)
}
if len(got) > 201 {
t.Fatal("unbounded page")
}
@ -164,3 +172,111 @@ func TestAdminAnnotationPoolMongoParity(t *testing.T) {
t.Fatalf("missing batch count=%+v err=%v", again, err)
}
}
// Batch source joins must preserve failed, missing, foreign and superseded
// certificates. Compare the existing admission expression on the same records.
func TestAdminAnnotationPoolMongoAdmissionParity(t *testing.T) {
uri := os.Getenv("POOL_LOCAL_MONGO_URI")
if uri == "" {
t.Skip("isolated local Mongo opt-in")
}
u, e := url.Parse(uri)
if e != nil || u.Host != "127.0.0.1:27018" || u.Scheme != "mongodb" {
t.Fatal("local Mongo 27018 only")
}
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
client, e := mongo.Connect(ctx, options.Client().ApplyURI(uri))
if e != nil {
t.Fatal(e)
}
defer client.Disconnect(context.Background())
db := client.Database(fmt.Sprintf("admin_pool_admission_%d", time.Now().UnixNano()))
defer db.Drop(context.Background())
s := NewMongoStore(db)
if e := s.EnsureIndexes(ctx); e != nil {
t.Fatal(e)
}
const org = "admin-admission"
batch := domain.HumanBatchRecord{ID: "batch", OrganizationID: org, PluginID: "annotation-qa", Stages: domain.DefaultFanyuV58HumanStages()}
if e := s.UpsertHumanBatch(ctx, batch); e != nil {
t.Fatal(e)
}
for n := 0; n < 192; n++ {
id := fmt.Sprintf("case-%03d", n)
item := domain.HumanItemRecord{ID: id, OrganizationID: org, ProjectID: "project", RunID: "run", PluginID: "annotation-qa", BatchID: batch.ID, CurrentStageID: "annotation_combined", Status: domain.HumanItemReady, LegacyTaskID: id, CreatedAt: id, ProcessingJob: &domain.AnnotationProcessingJob{SourceIndex: n%2 == 0, SourceFingerprint: id}}
item.Subject.ItemKey = id
if n%3 == 0 {
item.Subject.Payload = map[string]any{"materialization": map[string]any{"sourceIndex": true}}
item.ProcessingJob.SourceFingerprint = ""
}
if e := s.UpsertHumanItem(ctx, item); e != nil {
t.Fatal(e)
}
if n%7 != 0 {
task := domain.AnnotationTaskRecord{ID: id, OrganizationID: org, ProjectID: item.ProjectID, RunID: item.RunID, SourceIsFullVideo: n%2 == 0, TemplateVersion: 6, ArtifactID: "video"}
if _, e := s.annotationTasks.InsertOne(ctx, task); e != nil {
t.Fatal(e)
}
}
for k, jobID := range []string{id, "source-v1:" + org + ":" + id} {
if n%5 == k {
continue
}
job := domain.AnnotationIndexJob{ID: jobID, OrganizationID: org, ProjectID: item.ProjectID, RunID: item.RunID, Status: []string{"completed", "failed", "running"}[n%3]}
if n%11 == 0 {
job.OrganizationID = "foreign"
}
if k == 0 && n%4 == 0 {
job.RevalidateSource = true
job.SourceFingerprint = "repaired"
job.SupersededSourceFingerprint = id
}
if _, e := db.Collection("annotationIndexJobs").InsertOne(ctx, job); e != nil {
t.Fatal(e)
}
}
}
legacy := mongo.Pipeline{{{Key: "$match", Value: adminPoolScopeFilter(org, AdminAnnotationPoolQuery{})}}}
legacy = append(legacy, annotationSourceAdmissionStatusPipeline()...)
read := func(p mongo.Pipeline) map[string]string {
t.Helper()
cur, e := s.humanItems.Aggregate(ctx, p)
if e != nil {
t.Fatal(e)
}
defer cur.Close(ctx)
rows, e := decodeMany[adminPoolRow](ctx, cur)
if e != nil {
t.Fatal(e)
}
out := map[string]string{}
for _, r := range rows {
out[r.ID] = r.SourceAdmissionStatus
}
return out
}
expected := read(legacy)
got := read(adminPoolPagePipeline(org, AdminAnnotationPoolQuery{Pool: "all"}))
if len(expected) != 192 || !reflect.DeepEqual(expected, got) {
t.Fatalf("batch admission changed: expected=%v actual=%v", expected, got)
}
available, e := s.ListAdminAnnotationPool(ctx, org, AdminAnnotationPoolQuery{Pool: "available", Limit: 300})
if e != nil {
t.Fatal(e)
}
ready := 0
for _, status := range expected {
if status == "" {
ready++
}
}
if len(available) != ready {
t.Fatalf("available=%d expected=%d", len(available), ready)
}
for _, item := range available {
if expected[item.ID] != "" {
t.Fatalf("unready source exposed: %s", item.ID)
}
}
}

View File

@ -246,7 +246,7 @@
"tests/frontend/annotation-task-pools.test.mjs"
],
"go": {
"./internal/api": ["^TestHumanPoolSearch", "^TestHumanPoolQCProjectScope"]
"./internal/api": ["^TestHumanPoolSearch", "^TestHumanPoolQCProjectScope", "^TestAdminAnnotationPool"]
},
"task": [
{
@ -271,7 +271,7 @@
"test",
"./internal/store",
"-run",
"^TestHumanPoolSearchMongo",
"^(TestHumanPoolSearchMongo|TestAdminAnnotationPoolMongo)",
"-count=1",
"-timeout=3m",
"-v"
@ -292,7 +292,25 @@
"-timeout=2m",
"-v"
],
"env": { "ANNOTATION_BUFFER_MONGO_STORE": "1" },
"env": {
"ANNOTATION_BUFFER_MONGO_STORE": "1"
},
"tests": true
},
{
"argv": [
"go",
"test",
"./internal/api",
"-run",
"^TestAdminAnnotationPoolBrowserReplica$",
"-count=1",
"-timeout=3m",
"-v"
],
"env": {
"ADMIN_POOL_BROWSER_TEST": "1"
},
"tests": true
}
]