forked from longtao.wu/droboticflow
🐛 Separate administrative pool totals from bounded cursor pages
Administrator counters were derived from the stage-filtered first human-item page, while annotators used an aggregate. Add authorized whole-pool counters with a five-second bounded cache, page human items before joining task summaries, and search older metadata on the server. Cancel stale reads and preserve existing non-admin scope behavior. Cover permissions, mixed stages, real Mongo parity, old-page search, categories, pagination, stale responses and claim invalidation.
This commit is contained in:
parent
867ca69835
commit
2876a14d4a
12
.scratch/admin-pool-counts-20261021/spec.md
Normal file
12
.scratch/admin-pool-counts-20261021/spec.md
Normal file
@ -0,0 +1,12 @@
|
||||
# Correct administrative annotation pool totals and paging
|
||||
Status: resolved
|
||||
|
||||
Base: 867ca698 (contains the prior local editing repairs). Local only.
|
||||
|
||||
The annotation workspace administrator badges must count all authorized human items, independent of the current list page, text search and status category. Project/workflow/run/Capture scope still applies. Lists must page human items, then join tasks by IDs; searching must include older items beyond the former first 500. Never fetch all business payloads into the browser, change customer records, widen annotator visibility or modify claim permissions.
|
||||
|
||||
Use an admin-only read endpoint, compact server-side counts and bounded cursor pages. Keep existing non-admin/review behavior and old APIs for compatibility. Test >500 items, mixed stages, orphan/cross-org batches, permissions, claim changes, old-page search, category filtering, stale page responses and pagination with stable totals. No production deployment in this task.
|
||||
|
||||
## Answer
|
||||
|
||||
Implemented admin-only totals with a 5-second bounded actor/scope cache; list pages follow human item cursors and join only page task IDs. Server-side Capture/status search sees older items; abandoned searches are cancelled and stale responses fenced. Existing role paths and APIs remain. See docs/annotation-admin-task-pool.zh-CN.md and docs/verification/admin-pool-counts-20261021.md.
|
||||
13
docs/annotation-admin-task-pool.zh-CN.md
Normal file
13
docs/annotation-admin-task-pool.zh-CN.md
Normal file
@ -0,0 +1,13 @@
|
||||
# 管理员切分与标注任务池
|
||||
|
||||
2026-10-21 本地修复。适用于平台管理员的“切分与语义标注”队列;标注员随机领取、个人缓冲池及质检/验收原入口保持原合同。
|
||||
|
||||
“可领取/我的任务/全部任务”统计当前权限与项目、工作流、执行、Capture 范围内的完整人工任务池,不再把首屏或已加载列表条数当作总数。搜索词和状态分类只筛选下方列表,不改变这些总数。管理员“全部任务”沿用当前环节范围,包含标注及已送检质检任务,不是全部历史已交付任务。
|
||||
|
||||
新增只读接口 `GET /api/plugins/annotation-qa/admin-pool`,只有组织管理员/平台管理员可用,组织范围由服务端身份决定。有效批次及阶段校验保留;非管理员不能通过传入 pool、用户或组织参数取得公共任务明细。旧接口保留,已有浏览器和在途任务无需迁移。
|
||||
|
||||
列表默认且最多每页 200 条,以人工任务的 `(createdAt,id)` 游标继续,任务摘要只关联这一页的 ID。“显示更多”读取下一页人工任务,而不是继续拉取独立的最新任务窗口。搜索及状态分类在服务端处理,能够找到第 500 条以后的任务;旧 Capture 名称仍通过原有元数据提取规则兼容。切换项目、列表或搜索后,不接受旧请求迟到的内容。
|
||||
|
||||
总数按批次/阶段聚合紧凑计数,再用既有批次权限投影校验;不读取动作正文或草稿来计数。总数使用有身份和范围隔离、容量上限的 5 秒进程缓存,当前进程的 API 写入会使缓存失效;其他副本与后台写入最多延迟到缓存到期。列表不缓存。搜索输入合并 350 毫秒,新搜索与范围切换取消过期请求;旧任务搜索在服务端流式读取投影元数据,查询和范围统计受 8 秒期限限制,不向浏览器传整池,也不把失败当成零条。计数与页面是相邻的只读操作,并非冻结生产时刻的结算快照。
|
||||
|
||||
不新增统计集合、数据库字段或索引,不改领取、标注、返工、审核及交付结果。生产环境仍需单独发布与验收。
|
||||
33
docs/verification/admin-pool-counts-20261021.md
Normal file
33
docs/verification/admin-pool-counts-20261021.md
Normal file
@ -0,0 +1,33 @@
|
||||
# Administrator annotation pool totals — local verification
|
||||
|
||||
Base `867ca698`, branch `codex/rework-boundary-20261021`. Diagnosis captured deployed `fab5be57`: administrator badges were the stage-filtered first 500 human records while annotator blind claim used an aggregate. The original metadata replay produced 495 vs 14902; screenshots were 496 vs 14911 at an earlier time.
|
||||
|
||||
## Regression and implementation
|
||||
|
||||
`TestAdminAnnotationPoolTotalsAndOldPageSearch` first failed with 404 before adding the read endpoint. It now covers 610 added mixed-stage items, stable whole-pool counts across cursors, a search hit older than the previous 500-row window, and denial to annotators. Permission/cache/error tests cover role changes before cache access, normalized count keys independent of page/list filters, bounded page limits, malformed inputs, 5-second expiry and a failed aggregate returning an error rather than false zero.
|
||||
|
||||
The real page/HTTP fixture uses MemoryStore and synthetic records only:
|
||||
|
||||
```
|
||||
ADMIN_POOL_BROWSER_TEST=1 go test ./internal/api -run 'TestAdminAnnotationPool' -count=1 -v
|
||||
```
|
||||
|
||||
Covers 200→400 loaded cards with unchanged 606 available/611 all totals; search for CAPTURE_0001; status category returning zero without changing badges; all-pool review filtering; delayed old search response ignored; real local claim invalidating cached count. No page errors or unexpected browser writes.
|
||||
|
||||
Actual Mongo parity and timings use a new, unique synthetic database on local Mongo 27018, cleaned afterward:
|
||||
|
||||
```
|
||||
POOL_LOCAL_MONGO_URI=mongodb://127.0.0.1:27018 go test ./internal/store -run '^TestAdminAnnotationPoolMongoParity$' -count=1 -v
|
||||
```
|
||||
|
||||
1500 synthetic records in separate batches cover mixed stages/assignees, orphan and cross-organization rows, full old-page searches, literal rather than regex search, cursor de-duplication, persisted workflow categories, legacy Capture names obtained through Capture records, and absence of heavy annotationWorkspace on list responses. Timings are local store measurements, not claims about production latency.
|
||||
|
||||
Legacy API compatibility, shared Capture extraction, existing dashboard-cache defaults, annotator/review editing browser flows, TypeScript, build, production-bundle boot, and full frontend baseline gate are checked separately. No baseline entries are edited. Production data, workflow, indexes, Worker and delivery JSON are not changed in this task; no push, PR or deployment is included.
|
||||
|
||||
## Final evidence
|
||||
|
||||
Store-level measurements with 1500 separate batches: counts 42.8 ms, first page 29.4 ms, old-record search 486.6 ms; these are synthetic local observations only. No new production index or statistical collection is required.
|
||||
|
||||
One combined browser run lost a fixture work session during the draft C/undo/redo scenario; there were no page errors. The same test passed unchanged in an isolated rerun, including the newer-draft recovery path. This is recorded separately rather than erased from the results or accepted by changing test expectations. The administrator browser flow and existing boundary browser passed in that combined run. Logs remain in untracked local test results and `/tmp/admin-pool-all-browser.log` / `/tmp/admin-pool-slice-isolated.log` for this session.
|
||||
|
||||
The final combined administrator + slice-phase browser rerun passed with the newer-draft case, and no runtime errors. Final frontend gate retains only 13 known baseline failures; TypeScript, Go build, web build and production boot pass. Old non-admin project scope/dependency behavior is preserved, so local annotator filters do not trigger the new administrator queries.
|
||||
153
internal/api/admin_annotation_pool.go
Normal file
153
internal/api/admin_annotation_pool.go
Normal file
@ -0,0 +1,153 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/store"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
func (s *server) adminAnnotationPoolQuery(w http.ResponseWriter, r *http.Request) (store.AdminAnnotationPoolQuery, bool) {
|
||||
p, _ := currentPrincipal(r)
|
||||
if !annotationManager(p) {
|
||||
renderJSON(w, 403, map[string]string{"error": "administrator annotation pool required"})
|
||||
return store.AdminAnnotationPoolQuery{}, false
|
||||
}
|
||||
v := r.URL.Query()
|
||||
q := store.AdminAnnotationPoolQuery{ProjectID: v.Get("projectId"), WorkflowID: v.Get("workflowId"), RunID: v.Get("runId"), NodeID: v.Get("nodeId"), ArtifactID: v.Get("artifactId"), Capture: strings.TrimSpace(v.Get("capture")), Search: strings.TrimSpace(v.Get("search")), Category: v.Get("category"), Pool: v.Get("pool"), UserID: p.User.ID}
|
||||
if q.Pool == "" {
|
||||
q.Pool = "available"
|
||||
}
|
||||
if q.Pool != "available" && q.Pool != "mine" && q.Pool != "all" {
|
||||
renderValidationError(w, validationIssue{Field: "pool", Message: "pool must be available, mine, or all"})
|
||||
return q, false
|
||||
}
|
||||
if len(q.Search) > 256 || len(q.Capture) > 1024 {
|
||||
renderValidationError(w, validationIssue{Field: "search", Message: "search is too long"})
|
||||
return q, false
|
||||
}
|
||||
switch q.Category {
|
||||
case "", "all", "available", "pending_slice", "slicing", "pending_annotation", "rework", "pending_quality", "rework_review", "pending_acceptance", "accepted", "submitting", "submission_conflicted":
|
||||
default:
|
||||
renderValidationError(w, validationIssue{Field: "category", Message: "unknown task category"})
|
||||
return q, false
|
||||
}
|
||||
limit, issue := parseDatabaseBrowserPageQuery(v.Get("limit"), "limit", 200, 1, 200)
|
||||
if issue != nil {
|
||||
renderValidationError(w, *issue)
|
||||
return q, false
|
||||
}
|
||||
q.Limit = limit
|
||||
cursor, err := decodeHumanItemCursor(v.Get("cursor"))
|
||||
if err != nil {
|
||||
renderValidationError(w, validationIssue{Field: "cursor", Message: "cursor must be a valid opaque human item cursor"})
|
||||
return q, false
|
||||
}
|
||||
q.Cursor = cursor
|
||||
return q, true
|
||||
}
|
||||
|
||||
// Counts use the existing bounded per-actor cache; list criteria and cursor are
|
||||
// deliberately excluded from its key. Permission is checked before cache hits.
|
||||
func (s *server) adminAnnotationCounts(w http.ResponseWriter, r *http.Request, q store.AdminAnnotationPoolQuery) (*store.AdminAnnotationPoolCounts, bool) {
|
||||
countRequest := r.Clone(r.Context())
|
||||
copyURL := *r.URL
|
||||
countRequest.URL = ©URL
|
||||
values := url.Values{}
|
||||
for _, key := range []string{"projectId", "workflowId", "runId", "nodeId", "artifactId", "capture"} {
|
||||
if value := r.URL.Query().Get(key); value != "" {
|
||||
values.Set(key, value)
|
||||
}
|
||||
}
|
||||
countRequest.URL.RawQuery = values.Encode()
|
||||
key := s.dashboardCacheKey("admin-annotation-pool-counts", countRequest)
|
||||
if entry := s.dashboardResponses.get(key); entry != nil {
|
||||
var counts store.AdminAnnotationPoolCounts
|
||||
if json.Unmarshal(entry.body, &counts) == nil {
|
||||
return &counts, true
|
||||
}
|
||||
}
|
||||
counts, err := s.store.CountAdminAnnotationPools(r.Context(), s.requestOrganizationID(r), q)
|
||||
if err != nil {
|
||||
renderError(w, 500, err)
|
||||
return nil, false
|
||||
}
|
||||
body, err := json.Marshal(counts)
|
||||
if err != nil {
|
||||
renderError(w, 500, err)
|
||||
return nil, false
|
||||
}
|
||||
// Five seconds bounds cross-pod/background changes. Existing API write
|
||||
// invalidation clears these entries immediately on this process.
|
||||
s.dashboardResponses.setTTL(key, 200, "application/json", body, 5*time.Second)
|
||||
return &counts, true
|
||||
}
|
||||
|
||||
func (s *server) handleAdminAnnotationPool(w http.ResponseWriter, r *http.Request) {
|
||||
q, ok := s.adminAnnotationPoolQuery(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
counts, ok := s.adminAnnotationCounts(w, r, q)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
size := q.Limit
|
||||
q.Limit++
|
||||
items, err := s.store.ListAdminAnnotationPool(r.Context(), s.requestOrganizationID(r), q)
|
||||
if err != nil {
|
||||
renderError(w, 500, err)
|
||||
return
|
||||
}
|
||||
page := humanItemListPage{Items: items}
|
||||
if len(items) > size {
|
||||
page.Items = items[:size]
|
||||
last := page.Items[len(page.Items)-1]
|
||||
page.HasMore = true
|
||||
page.NextCursor = encodeHumanItemCursor(&domain.HumanItemCursor{CreatedAt: last.CreatedAt, ID: last.ID})
|
||||
}
|
||||
resolver := newHumanDisplayResolver(s)
|
||||
for i := range page.Items {
|
||||
if err := resolver.enrichItem(r.Context(), &page.Items[i]); err != nil {
|
||||
renderError(w, 500, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
// Join only the page's task IDs, never an independent newest-first window.
|
||||
tasks := []annotationTaskListRecord{}
|
||||
ids := []string{}
|
||||
seen := map[string]bool{}
|
||||
for _, item := range page.Items {
|
||||
if item.LegacyTaskID != "" && !seen[item.LegacyTaskID] {
|
||||
ids = append(ids, item.LegacyTaskID)
|
||||
seen[item.LegacyTaskID] = true
|
||||
}
|
||||
}
|
||||
if len(ids) > 0 {
|
||||
ctx, err := s.withHumanListReads(r.Context(), s.requestOrganizationID(r), q.ProjectID, page.Items)
|
||||
if err != nil {
|
||||
renderError(w, 500, err)
|
||||
return
|
||||
}
|
||||
request := r.Clone(ctx)
|
||||
u := *r.URL
|
||||
request.URL = &u
|
||||
v := url.Values{"ids": {strings.Join(ids, ",")}}
|
||||
if q.ProjectID != "" {
|
||||
v.Set("projectId", q.ProjectID)
|
||||
}
|
||||
request.URL.RawQuery = v.Encode()
|
||||
joined := s.readAnnotationTasksList(w, request)
|
||||
if joined == nil {
|
||||
return
|
||||
}
|
||||
tasks = joined.Items
|
||||
}
|
||||
result := page.response()
|
||||
result["counts"] = counts
|
||||
result["tasks"] = tasks
|
||||
renderJSON(w, 200, result)
|
||||
}
|
||||
70
internal/api/admin_annotation_pool_browser_test.go
Normal file
70
internal/api/admin_annotation_pool_browser_test.go
Normal file
@ -0,0 +1,70 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"droboticflow/internal/domain"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"os/exec"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestAdminAnnotationPoolBrowserReplica(t *testing.T) {
|
||||
if os.Getenv("ADMIN_POOL_BROWSER_TEST") != "1" {
|
||||
t.Skip("local browser opt-in")
|
||||
}
|
||||
f := randomClaimFixture(t, 610)
|
||||
admin := domain.UserRecord{ID: "pool-admin", OrganizationID: domain.DefaultOrganizationID, Role: domain.UserRoleAdmin, Status: domain.UserStatusActive}
|
||||
_ = f.store.UpsertUser(f.ctx, admin)
|
||||
_ = f.store.UpsertOrganizationPlugin(f.ctx, domain.OrganizationPluginRecord{OrganizationID: domain.DefaultOrganizationID, PluginID: "annotation-qa", Enabled: true, Config: defaultAnnotationPluginConfig()})
|
||||
for n := 0; n < 610; n++ {
|
||||
item, _ := f.store.GetHumanItem(f.ctx, fmt.Sprintf("random-item-%d", n))
|
||||
task, _ := f.store.GetAnnotationTask(f.ctx, item.LegacyTaskID)
|
||||
stamp := fmt.Sprintf("2026-10-21T00:%02d:%02dZ", n/60, n%60)
|
||||
item.CreatedAt = stamp
|
||||
task.CreatedAt = stamp
|
||||
item.Subject.ItemKey = fmt.Sprintf("CAPTURE_%04d_camera_left_part0001.mp4", n)
|
||||
task.ArtifactID = item.Subject.ItemKey
|
||||
task.SourceVideo = item.Subject.ItemKey
|
||||
item.Subject.Payload = map[string]any{"sliceWorkflow": map[string]any{"status": "editing", "phase": "slice"}}
|
||||
if n >= 606 {
|
||||
item.CurrentStageID = "quality_review"
|
||||
task.Status = domain.AnnotationTaskStatusPendingQuality
|
||||
}
|
||||
_ = f.store.UpsertHumanItem(f.ctx, *item)
|
||||
_ = f.store.UpsertAnnotationTask(f.ctx, *task)
|
||||
}
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/api/auth/me" {
|
||||
renderJSON(w, 200, map[string]any{"user": sanitizeUser(&admin), "organization": domain.OrganizationRecord{ID: domain.DefaultOrganizationID, Status: domain.OrganizationStatusActive}})
|
||||
return
|
||||
}
|
||||
if r.URL.Path == "/__fixture/claim" {
|
||||
result := f.request(http.MethodPost, "/api/plugins/annotation-qa/random-claims", f.annotator, map[string]any{"projectId": f.task.ProjectID, "requestId": "admin-pool-fixture-claim-01"})
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(result.Code)
|
||||
_, _ = w.Write(result.Body.Bytes())
|
||||
return
|
||||
}
|
||||
if r.Method != "GET" && r.Method != "HEAD" {
|
||||
renderJSON(w, 403, map[string]string{"error": "fixture read-only"})
|
||||
return
|
||||
}
|
||||
r = r.WithContext(context.WithValue(r.Context(), authContextKey{}, &authenticatedPrincipal{User: admin, Organization: domain.OrganizationRecord{ID: domain.DefaultOrganizationID, Status: domain.OrganizationStatusActive}}))
|
||||
f.handler.ServeHTTP(w, r)
|
||||
}))
|
||||
defer server.Close()
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 3*time.Minute)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, "node", "tests/browser/admin-annotation-pool-replica.mjs")
|
||||
cmd.Dir = "../.."
|
||||
cmd.Env = append(os.Environ(), "ADMIN_POOL_API="+server.URL)
|
||||
out, err := cmd.CombinedOutput()
|
||||
t.Log(string(out))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
146
internal/api/admin_annotation_pool_test.go
Normal file
146
internal/api/admin_annotation_pool_test.go
Normal file
@ -0,0 +1,146 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/store"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestAdminAnnotationPoolTotalsAndOldPageSearch(t *testing.T) {
|
||||
f := newFanyuV5FourStageFixture(t, domain.FanyuV5StageSemanticAnnotation, domain.HumanItemReady, "", 0)
|
||||
f.batch.Stages = append(f.batch.Stages, domain.HumanStageDefinition{ID: "quality_review", Kind: domain.HumanStageReview, RequiredRole: "quality_reviewer"})
|
||||
if err := f.store.UpsertHumanBatch(f.ctx, f.batch); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
admin := domain.UserRecord{ID: "pool-admin", OrganizationID: domain.DefaultOrganizationID, Role: domain.UserRoleAdmin, Status: domain.UserStatusActive}
|
||||
for i := 0; i < 610; i++ {
|
||||
item := f.item
|
||||
item.ID = fmt.Sprintf("pool-%04d", i)
|
||||
item.LegacyTaskID = fmt.Sprintf("pool-task-%04d", i)
|
||||
item.CreatedAt = fmt.Sprintf("2026-10-21T00:%02d:%02dZ", i/60, i%60)
|
||||
item.Subject.ItemKey = fmt.Sprintf("CAPTURE_%04d_camera_left_part0001.mp4", i)
|
||||
if i >= 606 {
|
||||
item.CurrentStageID = "quality_review"
|
||||
}
|
||||
if e := f.store.UpsertHumanItem(f.ctx, item); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
}
|
||||
base := "/api/plugins/annotation-qa/admin-pool?projectId=" + f.task.ProjectID
|
||||
read := func(path string) map[string]json.RawMessage {
|
||||
t.Helper()
|
||||
r := f.request(http.MethodGet, path, admin, nil)
|
||||
if r.Code != 200 {
|
||||
t.Fatalf("%d %s", r.Code, r.Body)
|
||||
}
|
||||
var d map[string]json.RawMessage
|
||||
if e := json.Unmarshal(r.Body.Bytes(), &d); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
return d
|
||||
}
|
||||
first := read(base + "&pool=available&limit=100")
|
||||
var counts struct{ Available, All, Mine int }
|
||||
_ = json.Unmarshal(first["counts"], &counts)
|
||||
if counts.Available != 607 || counts.All != 611 || counts.Mine != 0 {
|
||||
t.Fatalf("counts %+v", counts)
|
||||
}
|
||||
var items []domain.HumanItemRecord
|
||||
_ = json.Unmarshal(first["items"], &items)
|
||||
if len(items) != 100 {
|
||||
t.Fatalf("page=%d", len(items))
|
||||
}
|
||||
var cursor string
|
||||
_ = json.Unmarshal(first["nextCursor"], &cursor)
|
||||
if cursor == "" {
|
||||
t.Fatal("missing cursor")
|
||||
}
|
||||
second := read(base + "&pool=available&limit=100&cursor=" + url.QueryEscape(cursor))
|
||||
var nextCounts struct{ Available, All, Mine int }
|
||||
_ = json.Unmarshal(second["counts"], &nextCounts)
|
||||
if counts != nextCounts {
|
||||
t.Fatal("paging changed totals")
|
||||
}
|
||||
old := read(base + "&pool=available&search=CAPTURE_0001")
|
||||
_ = json.Unmarshal(old["items"], &items)
|
||||
if len(items) != 1 || items[0].ID != "pool-0001" {
|
||||
t.Fatalf("old search %+v", items)
|
||||
}
|
||||
_ = json.Unmarshal(old["counts"], &nextCounts)
|
||||
if counts != nextCounts {
|
||||
t.Fatal("search changed badges")
|
||||
}
|
||||
denied := f.request(http.MethodGet, base+"&pool=all", f.annotator, nil)
|
||||
if denied.Code != 403 {
|
||||
t.Fatalf("annotator=%d", denied.Code)
|
||||
}
|
||||
}
|
||||
|
||||
type adminPoolProbe struct {
|
||||
store.Store
|
||||
counts, pages int
|
||||
fail bool
|
||||
}
|
||||
|
||||
func (s *adminPoolProbe) CountAdminAnnotationPools(ctx context.Context, org string, q store.AdminAnnotationPoolQuery) (store.AdminAnnotationPoolCounts, error) {
|
||||
s.counts++
|
||||
if s.fail {
|
||||
return store.AdminAnnotationPoolCounts{}, errors.New("synthetic unavailable")
|
||||
}
|
||||
return s.Store.CountAdminAnnotationPools(ctx, org, q)
|
||||
}
|
||||
func (s *adminPoolProbe) ListAdminAnnotationPool(ctx context.Context, org string, q store.AdminAnnotationPoolQuery) ([]domain.HumanItemRecord, error) {
|
||||
s.pages++
|
||||
if q.Limit > 201 {
|
||||
panic("unbounded administrator page")
|
||||
}
|
||||
return s.Store.ListAdminAnnotationPool(ctx, org, q)
|
||||
}
|
||||
func TestAdminAnnotationPoolFreshPermissionsCacheAndFailures(t *testing.T) {
|
||||
f := newFanyuV5FourStageFixture(t, domain.FanyuV5StageSemanticAnnotation, domain.HumanItemReady, "", 0)
|
||||
probe := &adminPoolProbe{Store: f.store}
|
||||
now := time.Now()
|
||||
srv := &server{store: probe, dashboardResponses: dashboardResponseCache{now: func() time.Time { return now }}}
|
||||
f.handler = http.HandlerFunc(srv.handleAdminAnnotationPool)
|
||||
admin := domain.UserRecord{ID: "admin-cache", OrganizationID: domain.DefaultOrganizationID, Role: domain.UserRoleAdmin, Status: domain.UserStatusActive}
|
||||
base := "/api/plugins/annotation-qa/admin-pool?projectId=" + f.task.ProjectID
|
||||
for _, suffix := range []string{"&pool=available", "&pool=all&search=missing&category=pending_slice", "&pool=mine"} {
|
||||
r := f.request(http.MethodGet, base+suffix, admin, nil)
|
||||
if r.Code != 200 {
|
||||
t.Fatalf("%d %s", r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
if probe.counts != 1 || probe.pages != 3 {
|
||||
t.Fatalf("counts=%d pages=%d", probe.counts, probe.pages)
|
||||
}
|
||||
for _, role := range []domain.UserRole{domain.UserRoleAnnotator, domain.UserRoleQualityReviewer, domain.UserRoleAcceptanceReviewer, domain.UserRoleSupplierManager} {
|
||||
p := admin
|
||||
p.Role = role
|
||||
if r := f.request(http.MethodGet, base, p, nil); r.Code != 403 {
|
||||
t.Fatalf("cached role %s=%d", role, r.Code)
|
||||
}
|
||||
}
|
||||
for _, suffix := range []string{"&cursor=broken", "&limit=bad", "&pool=reviewed", "&category=unknown"} {
|
||||
if r := f.request(http.MethodGet, base+suffix, admin, nil); r.Code != 422 && r.Code != 400 {
|
||||
t.Fatalf("invalid %s=%d", suffix, r.Code)
|
||||
}
|
||||
}
|
||||
// A failed total read must never masquerade as zero, even when the old page
|
||||
// would fit. Cross-process/background changes become visible after 5 seconds.
|
||||
now = now.Add(6 * time.Second)
|
||||
probe.fail = true
|
||||
r := f.request(http.MethodGet, base, admin, nil)
|
||||
if r.Code != 500 {
|
||||
t.Fatalf("failed aggregate=%d %s", r.Code, r.Body)
|
||||
}
|
||||
if probe.pages != 3 {
|
||||
t.Fatal("page fetched after count failure")
|
||||
}
|
||||
}
|
||||
@ -2,7 +2,6 @@ package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
@ -122,27 +121,7 @@ func appendCaptureDirectory(values *[]string, candidate string) {
|
||||
}
|
||||
|
||||
func humanCaptureDirectoriesFromPayload(value any, values *[]string) {
|
||||
switch typed := normalizeStructuredValue(value).(type) {
|
||||
case map[string]any:
|
||||
for key, item := range typed {
|
||||
normalizedKey := strings.ToLower(strings.TrimSpace(key))
|
||||
switch normalizedKey {
|
||||
case "capturedirectory", "captureprefix", "capturepath", "sourcedirectory", "sourceprefix", "recordingdirectory":
|
||||
appendCaptureDirectory(values, domain.CaptureDirectoryFromDirectoryValue(fmt.Sprint(item)))
|
||||
case "sourcevideo", "artifactid", "clipartifactid":
|
||||
appendCaptureDirectory(values, domain.CaptureDirectoryFromArtifactValue(fmt.Sprint(item)))
|
||||
}
|
||||
humanCaptureDirectoriesFromPayload(item, values)
|
||||
}
|
||||
case []any:
|
||||
for _, item := range typed {
|
||||
humanCaptureDirectoriesFromPayload(item, values)
|
||||
}
|
||||
case []map[string]any:
|
||||
for _, item := range typed {
|
||||
humanCaptureDirectoriesFromPayload(item, values)
|
||||
}
|
||||
}
|
||||
domain.AppendCaptureDirectoriesFromPayload(value, values)
|
||||
}
|
||||
|
||||
func uniqueCaptureDirectories(values []string) []string {
|
||||
|
||||
@ -17,6 +17,7 @@ func (server *server) mountAnnotationRoutes(router chi.Router) {
|
||||
router.Get("/api/plugins/annotation-qa/tasks/{id}", server.handleAnnotationTaskGet)
|
||||
router.Get("/api/plugins/annotation-qa/tasks/{id}/frame-timeline", server.handleAnnotationTaskFrameTimelineGet)
|
||||
router.Get("/api/plugins/annotation-qa/sidebar", server.handleAnnotationSidebar)
|
||||
router.Get("/api/plugins/annotation-qa/admin-pool", server.handleAdminAnnotationPool)
|
||||
// The workload summary is a read-only aggregate over 30 days of worklog
|
||||
// events. The shared 45s per-user response cache (invalidated by any
|
||||
// /api write) keeps the annotation center's initial load from re-running
|
||||
|
||||
@ -140,6 +140,9 @@ func (c *dashboardResponseCache) get(key string) *dashboardResponseCacheEntry {
|
||||
}
|
||||
|
||||
func (c *dashboardResponseCache) set(key string, status int, contentType string, body []byte) {
|
||||
c.setTTL(key, status, contentType, body, dashboardResponseCacheTTL)
|
||||
}
|
||||
func (c *dashboardResponseCache) setTTL(key string, status int, contentType string, body []byte, ttl time.Duration) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if c.entries == nil {
|
||||
@ -150,7 +153,7 @@ func (c *dashboardResponseCache) set(key string, status int, contentType string,
|
||||
}
|
||||
stored := make([]byte, len(body))
|
||||
copy(stored, body)
|
||||
c.entries[key] = dashboardResponseCacheEntry{body: stored, contentType: contentType, status: status, expires: c.now().Add(dashboardResponseCacheTTL)}
|
||||
c.entries[key] = dashboardResponseCacheEntry{body: stored, contentType: contentType, status: status, expires: c.now().Add(ttl)}
|
||||
}
|
||||
|
||||
func (c *dashboardResponseCache) invalidateAll() {
|
||||
|
||||
@ -1,8 +1,10 @@
|
||||
package domain
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/url"
|
||||
"path"
|
||||
"reflect"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
@ -143,3 +145,35 @@ func CaptureDirectoriesFromRun(run RunRecord) []string {
|
||||
}
|
||||
return uniqueSortedCaptureDirectories(values)
|
||||
}
|
||||
|
||||
// AppendCaptureDirectoriesFromPayload shares legacy metadata extraction between
|
||||
// task display and administrative search. It accepts string-keyed map aliases
|
||||
// (including BSON maps) and nested slices without depending on a DB driver.
|
||||
func AppendCaptureDirectoriesFromPayload(value any, values *[]string) {
|
||||
if value == nil {
|
||||
return
|
||||
}
|
||||
rv := reflect.ValueOf(value)
|
||||
switch rv.Kind() {
|
||||
case reflect.Map:
|
||||
if rv.Type().Key().Kind() != reflect.String {
|
||||
return
|
||||
}
|
||||
iter := rv.MapRange()
|
||||
for iter.Next() {
|
||||
key := strings.ToLower(strings.TrimSpace(iter.Key().String()))
|
||||
v := iter.Value().Interface()
|
||||
switch key {
|
||||
case "capturedirectory", "captureprefix", "capturepath", "sourcedirectory", "sourceprefix", "recordingdirectory":
|
||||
*values = append(*values, CaptureDirectoryFromDirectoryValue(fmt.Sprint(v)))
|
||||
case "sourcevideo", "artifactid", "clipartifactid":
|
||||
*values = append(*values, CaptureDirectoryFromArtifactValue(fmt.Sprint(v)))
|
||||
}
|
||||
AppendCaptureDirectoriesFromPayload(v, values)
|
||||
}
|
||||
case reflect.Slice, reflect.Array:
|
||||
for n := 0; n < rv.Len(); n++ {
|
||||
AppendCaptureDirectoriesFromPayload(rv.Index(n).Interface(), values)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -43,3 +43,16 @@ func TestCaptureDirectoriesFromRunKeepsPersistedCaptureDirectoryAfterArtifactsAr
|
||||
t.Fatalf("capture directories = %#v", actual)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCapturePayloadDirectoriesAcceptNestedMapAliases(t *testing.T) {
|
||||
type legacyMap map[string]any
|
||||
var values []string
|
||||
AppendCaptureDirectoriesFromPayload(legacyMap{" records ": []legacyMap{{" captureDirectory ": "bos://bucket/path/客户采集-42"}, {"sourceVideo": "CAPTURE_0001_camera_left_part0001.mp4"}}}, &values)
|
||||
found := map[string]bool{}
|
||||
for _, v := range values {
|
||||
found[v] = true
|
||||
}
|
||||
if !found["客户采集-42"] || !found["CAPTURE_0001"] {
|
||||
t.Fatalf("directories=%v", values)
|
||||
}
|
||||
}
|
||||
|
||||
380
internal/store/admin_annotation_pool.go
Normal file
380
internal/store/admin_annotation_pool.go
Normal file
@ -0,0 +1,380 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"droboticflow/internal/domain"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/mongo"
|
||||
"go.mongodb.org/mongo-driver/mongo/options"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Server-authorized administrative read model. Search/category affect only the
|
||||
// page; Capture is a scope and also affects the badge totals. No payload writes.
|
||||
type AdminAnnotationPoolQuery struct {
|
||||
ProjectID, WorkflowID, RunID, NodeID, ArtifactID string
|
||||
UserID, Pool, Search, Capture, Category string
|
||||
Limit int
|
||||
Cursor *domain.HumanItemCursor
|
||||
}
|
||||
type AdminAnnotationPoolCounts struct {
|
||||
Available int `json:"available"`
|
||||
Mine int `json:"mine"`
|
||||
All int `json:"all"`
|
||||
}
|
||||
|
||||
func adminPoolScope(i domain.HumanItemRecord) bool {
|
||||
return i.CurrentStageID == "semantic_annotation" || i.CurrentStageID == "annotation_combined" || i.CurrentStageID == "quality_review" || (i.CurrentStageID == "split_annotation" && i.Status == domain.HumanItemRework)
|
||||
}
|
||||
func adminPoolAvailable(i domain.HumanItemRecord) bool {
|
||||
return i.AssignedUserID == "" && (i.Status == domain.HumanItemReady || i.Status == domain.HumanItemRework || i.Status == domain.HumanItemSubmitted) && (i.CurrentStageID == "semantic_annotation" || i.CurrentStageID == "annotation_combined")
|
||||
}
|
||||
func adminPoolMine(i domain.HumanItemRecord, user string) bool {
|
||||
return i.AssignedUserID == user || (i.LatestAuthorUserID == user && i.CurrentStageID == "quality_review" && (i.Status == domain.HumanItemReady || i.Status == domain.HumanItemSubmitted))
|
||||
}
|
||||
func adminPoolSelected(i domain.HumanItemRecord, q AdminAnnotationPoolQuery) bool {
|
||||
switch q.Pool {
|
||||
case "available":
|
||||
return adminPoolAvailable(i)
|
||||
case "mine":
|
||||
return adminPoolMine(i, q.UserID)
|
||||
}
|
||||
return true
|
||||
}
|
||||
func adminPoolBaseQuery(q AdminAnnotationPoolQuery) domain.HumanItemQuery {
|
||||
return domain.HumanItemQuery{PluginID: "annotation-qa", ProjectID: q.ProjectID, WorkflowID: q.WorkflowID, RunID: q.RunID}
|
||||
}
|
||||
func adminPoolValidBatch(i domain.HumanItemRecord, b domain.HumanBatchRecord, org string) bool {
|
||||
if b.ID == "" || !humanOrganizationMatches(b.OrganizationID, org) {
|
||||
return false
|
||||
}
|
||||
for _, st := range b.Stages {
|
||||
if st.ID == i.CurrentStageID {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
func adminPoolScopeFilter(org string, q AdminAnnotationPoolQuery) bson.M {
|
||||
f := humanItemMongoFilter(org, adminPoolBaseQuery(q))
|
||||
f["batchId"] = bson.M{"$ne": ""}
|
||||
f["$expr"] = bson.M{"$and": bson.A{bson.M{"$eq": bson.A{bson.M{"$type": "$batchId"}, "string"}}, bson.M{"$eq": bson.A{bson.M{"$type": "$currentStageId"}, "string"}}}}
|
||||
f["$or"] = bson.A{bson.M{"currentStageId": bson.M{"$in": bson.A{"semantic_annotation", "annotation_combined", "quality_review"}}}, bson.M{"currentStageId": "split_annotation", "status": domain.HumanItemRework}}
|
||||
return f
|
||||
}
|
||||
func adminPoolAvailableFilter() bson.M {
|
||||
return bson.M{"currentStageId": bson.M{"$in": bson.A{"semantic_annotation", "annotation_combined"}}, "assignedUserId": bson.M{"$in": bson.A{"", nil}}, "status": bson.M{"$in": bson.A{domain.HumanItemReady, domain.HumanItemRework, domain.HumanItemSubmitted}}}
|
||||
}
|
||||
func adminPoolMineFilter(user string) bson.M {
|
||||
return bson.M{"$or": bson.A{bson.M{"assignedUserId": user}, bson.M{"latestAuthorUserId": user, "currentStageId": "quality_review", "status": bson.M{"$in": bson.A{domain.HumanItemReady, domain.HumanItemSubmitted}}}}}
|
||||
}
|
||||
|
||||
// The same business classification as the workspace, using persisted state.
|
||||
func adminPoolCategory(i domain.HumanItemRecord, t domain.AnnotationTaskRecord) string {
|
||||
if i.Status == domain.HumanItemAccepted || t.Status == "accepted" {
|
||||
return "accepted"
|
||||
}
|
||||
if i.PendingCombinedSubmissionID != "" {
|
||||
prefix := "fanyu-v5-combined-" + i.ID + "-v"
|
||||
if strings.HasPrefix(i.PendingCombinedSubmissionID, prefix) {
|
||||
suffix := strings.TrimPrefix(i.PendingCombinedSubmissionID, prefix)
|
||||
n, err := strconv.ParseInt(suffix, 10, 64)
|
||||
if err == nil && strings.IndexFunc(suffix, func(r rune) bool { return r < '0' || r > '9' }) < 0 && n >= 0 && n <= 9007199254740991 && int64(i.Version) > n+1 {
|
||||
return "submission_conflicted"
|
||||
}
|
||||
}
|
||||
return "submitting"
|
||||
}
|
||||
if i.ReviewRound > 0 && (i.CurrentStageID == "quality_review" || i.CurrentStageID == "prepackage_acceptance" || i.CurrentStageID == "acceptance") {
|
||||
return "rework_review"
|
||||
}
|
||||
if t.Status == "pending_quality" || i.CurrentStageID == "quality_review" || i.Status == domain.HumanItemSubmitted {
|
||||
return "pending_quality"
|
||||
}
|
||||
if t.Status == "pending_acceptance" || i.CurrentStageID == "acceptance" || i.CurrentStageID == "prepackage_acceptance" {
|
||||
return "pending_acceptance"
|
||||
}
|
||||
if t.Status == "rework" || i.Status == domain.HumanItemRework || (i.ReviewRound > 0 && i.CurrentStageID == "annotation_combined") {
|
||||
return "rework"
|
||||
}
|
||||
workflow := adminPoolMap(i.Subject.Payload["sliceWorkflow"])
|
||||
if workflow["status"] == "processing" {
|
||||
return "slicing"
|
||||
}
|
||||
if workflow["status"] == "completed" || workflow["phase"] == "annotate" {
|
||||
return "pending_annotation"
|
||||
}
|
||||
if workflow["status"] == "editing" || workflow["phase"] == "slice" || i.AssignedUserID != "" {
|
||||
return "pending_slice"
|
||||
}
|
||||
return "available"
|
||||
}
|
||||
|
||||
type adminPoolRow struct {
|
||||
domain.HumanItemRecord `bson:",inline"`
|
||||
Tasks []domain.AnnotationTaskRecord `bson:"poolTasks"`
|
||||
Runs []domain.RunRecord `bson:"poolRuns"`
|
||||
Captures []domain.CaptureRecord `bson:"poolCaptures"`
|
||||
Projects []domain.ProjectRecord `bson:"poolProjects"`
|
||||
}
|
||||
|
||||
func adminPoolMap(v any) map[string]any {
|
||||
switch x := v.(type) {
|
||||
case map[string]any:
|
||||
return x
|
||||
case bson.M:
|
||||
return map[string]any(x)
|
||||
case bson.D:
|
||||
out := map[string]any{}
|
||||
for _, e := range x {
|
||||
out[e.Key] = e.Value
|
||||
}
|
||||
return out
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func adminPoolRowMatches(row adminPoolRow, q AdminAnnotationPoolQuery) bool {
|
||||
var task domain.AnnotationTaskRecord
|
||||
if len(row.Tasks) > 0 {
|
||||
task = row.Tasks[0]
|
||||
}
|
||||
if q.NodeID != "" && task.NodeID != q.NodeID {
|
||||
return false
|
||||
}
|
||||
if q.ArtifactID != "" && task.ArtifactID != q.ArtifactID && task.RightArtifactID != q.ArtifactID {
|
||||
return false
|
||||
}
|
||||
if q.Category != "" && q.Category != "all" && adminPoolCategory(row.HumanItemRecord, task) != q.Category {
|
||||
return false
|
||||
}
|
||||
if q.Search == "" && q.Capture == "" {
|
||||
return true
|
||||
}
|
||||
values := []string{}
|
||||
if len(row.Captures) > 0 {
|
||||
c := row.Captures[0]
|
||||
if c.CaptureID != "" {
|
||||
values = append(values, c.CaptureID)
|
||||
} else {
|
||||
values = append(values, domain.CaptureDirectoryFromDirectoryValue(c.Prefix))
|
||||
}
|
||||
}
|
||||
if len(values) == 0 {
|
||||
values = append(values, domain.CaptureDirectoryFromArtifactValue(row.Subject.ItemKey))
|
||||
domain.AppendCaptureDirectoriesFromPayload(row.Subject.Payload, &values)
|
||||
}
|
||||
for _, v := range []string{task.ArtifactID, task.SourceVideo} {
|
||||
values = append(values, domain.CaptureDirectoryFromArtifactValue(v))
|
||||
}
|
||||
if len(row.Runs) > 0 {
|
||||
values = append(values, domain.CaptureDirectoriesFromRun(row.Runs[0])...)
|
||||
}
|
||||
if q.Capture != "" {
|
||||
found := false
|
||||
for _, v := range values {
|
||||
if strings.EqualFold(strings.TrimSpace(v), q.Capture) {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return false
|
||||
}
|
||||
}
|
||||
if q.Search != "" {
|
||||
if len(row.Projects) > 0 {
|
||||
values = append(values, row.Projects[0].Name)
|
||||
} else if len(row.Runs) > 0 {
|
||||
values = append(values, row.Runs[0].ProjectName)
|
||||
}
|
||||
if !strings.Contains(strings.ToLower(strings.Join(values, " ")), strings.ToLower(q.Search)) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
func (s *MemoryStore) adminPoolRowsLocked(org string, q AdminAnnotationPoolQuery) []adminPoolRow {
|
||||
rows := []adminPoolRow{}
|
||||
for _, i := range s.humanItems {
|
||||
if !humanOrganizationMatches(i.OrganizationID, org) || !humanItemMatches(i, adminPoolBaseQuery(q)) || !adminPoolScope(i) || !adminPoolValidBatch(i, s.humanBatches[i.BatchID], org) {
|
||||
continue
|
||||
}
|
||||
row := adminPoolRow{HumanItemRecord: humanItemListProjectionClone(i, humanItemListPayloadKeysRemoved)}
|
||||
if t, ok := s.annotationTasks[i.LegacyTaskID]; ok && humanOrganizationMatches(t.OrganizationID, org) {
|
||||
row.Tasks = []domain.AnnotationTaskRecord{t}
|
||||
}
|
||||
if r, ok := s.runs[i.RunID]; ok && humanOrganizationMatches(r.OrganizationID, org) {
|
||||
row.Runs = []domain.RunRecord{r}
|
||||
if c, ok := s.captureRecords[r.TaskID]; ok && humanOrganizationMatches(c.OrganizationID, org) {
|
||||
row.Captures = []domain.CaptureRecord{c}
|
||||
}
|
||||
}
|
||||
if p, ok := s.projects[i.ProjectID]; ok && humanOrganizationMatches(p.OrganizationID, org) {
|
||||
row.Projects = []domain.ProjectRecord{p}
|
||||
}
|
||||
rows = append(rows, row)
|
||||
}
|
||||
return rows
|
||||
}
|
||||
func (s *MemoryStore) CountAdminAnnotationPools(_ context.Context, org string, q AdminAnnotationPoolQuery) (AdminAnnotationPoolCounts, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
c := AdminAnnotationPoolCounts{}
|
||||
q.Search = ""
|
||||
q.Category = ""
|
||||
for _, r := range s.adminPoolRowsLocked(org, q) {
|
||||
if !adminPoolRowMatches(r, q) {
|
||||
continue
|
||||
}
|
||||
c.All++
|
||||
if adminPoolAvailable(r.HumanItemRecord) {
|
||||
c.Available++
|
||||
}
|
||||
if adminPoolMine(r.HumanItemRecord, q.UserID) {
|
||||
c.Mine++
|
||||
}
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
func (s *MemoryStore) ListAdminAnnotationPool(_ context.Context, org string, q AdminAnnotationPoolQuery) ([]domain.HumanItemRecord, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
rows := s.adminPoolRowsLocked(org, q)
|
||||
sort.Slice(rows, func(i, j int) bool {
|
||||
if rows[i].CreatedAt != rows[j].CreatedAt {
|
||||
return rows[i].CreatedAt > rows[j].CreatedAt
|
||||
}
|
||||
return rows[i].ID > rows[j].ID
|
||||
})
|
||||
out := []domain.HumanItemRecord{}
|
||||
for _, r := range rows {
|
||||
if adminPoolSelected(r.HumanItemRecord, q) && humanItemAfterCursor(r.HumanItemRecord, q.Cursor) && adminPoolRowMatches(r, q) {
|
||||
out = append(out, humanItemListProjectionClone(r.HumanItemRecord, humanItemListPayloadKeysRemoved))
|
||||
if len(out) >= q.Limit {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func adminPoolLookup(from, local, as, org string, projection bson.M) bson.D {
|
||||
return bson.D{{Key: "$lookup", Value: bson.M{"from": from, "localField": local, "foreignField": "id", "pipeline": bson.A{bson.M{"$match": bson.M{"organizationId": normalizedOrganizationID(org)}}, bson.M{"$project": projection}}, "as": as}}}
|
||||
}
|
||||
func adminPoolPagePipeline(org string, q AdminAnnotationPoolQuery) mongo.Pipeline {
|
||||
f := adminPoolScopeFilter(org, q)
|
||||
conditions := bson.A{f}
|
||||
if q.Pool == "available" {
|
||||
conditions = append(conditions, adminPoolAvailableFilter())
|
||||
} else if q.Pool == "mine" {
|
||||
conditions = append(conditions, adminPoolMineFilter(q.UserID))
|
||||
}
|
||||
if q.Cursor != nil {
|
||||
conditions = append(conditions, bson.M{"$or": bson.A{bson.M{"createdAt": bson.M{"$lt": q.Cursor.CreatedAt}}, bson.M{"createdAt": q.Cursor.CreatedAt, "id": bson.M{"$lt": q.Cursor.ID}}}})
|
||||
}
|
||||
pipeline := mongo.Pipeline{{{Key: "$match", Value: bson.M{"$and": conditions}}}, {{Key: "$sort", Value: bson.D{{Key: "createdAt", Value: -1}, {Key: "id", Value: -1}}}}, {{Key: "$project", Value: humanItemListProjection(domain.HumanItemQuery{ListProjection: true})}}}
|
||||
pipeline = append(pipeline, adminPoolLookup("humanBatches", "batchId", "poolBatches", org, bson.M{"_id": 0, "stages.id": 1}), bson.D{{Key: "$unwind", Value: "$poolBatches"}}, bson.D{{Key: "$match", Value: bson.M{"$expr": bson.M{"$in": bson.A{"$currentStageId", bson.M{"$ifNull": bson.A{"$poolBatches.stages.id", bson.A{}}}}}}}}, bson.D{{Key: "$unset", Value: "poolBatches"}})
|
||||
if q.Search != "" || q.Capture != "" || q.Category != "" && q.Category != "all" || q.NodeID != "" || q.ArtifactID != "" {
|
||||
pipeline = append(pipeline, adminPoolLookup("annotationTasks", "legacyTaskId", "poolTasks", org, bson.M{"_id": 0, "id": 1, "status": 1, "nodeId": 1, "artifactId": 1, "rightArtifactId": 1, "sourceVideo": 1}))
|
||||
}
|
||||
if q.Search != "" || q.Capture != "" {
|
||||
pipeline = append(pipeline, adminPoolLookup("runs", "runId", "poolRuns", org, bson.M{"_id": 0, "id": 1, "taskId": 1, "projectName": 1, "captureDirectories": 1, "inputs.ref": 1, "nodeExecutions.artifacts.id": 1, "nodeExecutions.artifacts.name": 1, "nodeExecutions.artifacts.path": 1, "artifacts.id": 1, "artifacts.name": 1, "artifacts.path": 1}), adminPoolLookup("captureRecords", "poolRuns.taskId", "poolCaptures", org, bson.M{"_id": 0, "captureId": 1, "prefix": 1}), adminPoolLookup("projects", "projectId", "poolProjects", org, bson.M{"_id": 0, "name": 1}))
|
||||
}
|
||||
return pipeline
|
||||
}
|
||||
|
||||
// Stream a narrow metadata cursor; do not materialize the backlog to find an old
|
||||
// search hit. Unfiltered reads stop after pageSize+1. Context bounds rare scans.
|
||||
func (s *MongoStore) scanAdminAnnotationPool(ctx context.Context, org string, q AdminAnnotationPoolQuery, visit func(domain.HumanItemRecord) bool) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 8*time.Second)
|
||||
defer cancel()
|
||||
cur, err := s.humanItems.Aggregate(ctx, adminPoolPagePipeline(org, q), options.Aggregate().SetAllowDiskUse(false).SetMaxTime(8*time.Second).SetBatchSize(200))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
for cur.Next(ctx) {
|
||||
var row adminPoolRow
|
||||
if err := cur.Decode(&row); err != nil {
|
||||
return err
|
||||
}
|
||||
if adminPoolRowMatches(row, q) && !visit(row.HumanItemRecord) {
|
||||
break
|
||||
}
|
||||
}
|
||||
return cur.Err()
|
||||
}
|
||||
func (s *MongoStore) ListAdminAnnotationPool(ctx context.Context, org string, q AdminAnnotationPoolQuery) ([]domain.HumanItemRecord, error) {
|
||||
out := []domain.HumanItemRecord{}
|
||||
err := s.scanAdminAnnotationPool(ctx, org, q, func(i domain.HumanItemRecord) bool { out = append(out, i); return len(out) < q.Limit })
|
||||
return out, err
|
||||
}
|
||||
|
||||
func (s *MongoStore) CountAdminAnnotationPools(ctx context.Context, org string, q AdminAnnotationPoolQuery) (AdminAnnotationPoolCounts, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 8*time.Second)
|
||||
defer cancel()
|
||||
if q.Capture != "" || q.NodeID != "" || q.ArtifactID != "" {
|
||||
all := q
|
||||
all.Pool = "all"
|
||||
all.Search = ""
|
||||
all.Category = ""
|
||||
all.Cursor = nil
|
||||
all.Limit = int(^uint(0) >> 1)
|
||||
c := AdminAnnotationPoolCounts{}
|
||||
err := s.scanAdminAnnotationPool(ctx, org, all, func(i domain.HumanItemRecord) bool {
|
||||
c.All++
|
||||
if adminPoolAvailable(i) {
|
||||
c.Available++
|
||||
}
|
||||
if adminPoolMine(i, q.UserID) {
|
||||
c.Mine++
|
||||
}
|
||||
return true
|
||||
})
|
||||
return c, err
|
||||
}
|
||||
// Group compact routing counters before decoding anything into the API. Same
|
||||
// projected batch validation used by the existing blind-claim total.
|
||||
available := bson.M{"$and": bson.A{bson.M{"$in": bson.A{"$currentStageId", bson.A{"semantic_annotation", "annotation_combined"}}}, bson.M{"$eq": bson.A{bson.M{"$ifNull": bson.A{"$assignedUserId", ""}}, ""}}, bson.M{"$in": bson.A{"$status", bson.A{"ready", "rework", "submitted"}}}}}
|
||||
mine := bson.M{"$or": bson.A{bson.M{"$eq": bson.A{"$assignedUserId", q.UserID}}, bson.M{"$and": bson.A{bson.M{"$eq": bson.A{"$latestAuthorUserId", q.UserID}}, bson.M{"$eq": bson.A{"$currentStageId", "quality_review"}}, bson.M{"$in": bson.A{"$status", bson.A{"ready", "submitted"}}}}}}}
|
||||
pipeline := mongo.Pipeline{{{Key: "$match", Value: adminPoolScopeFilter(org, q)}}, {{Key: "$group", Value: bson.M{"_id": bson.M{"batch": "$batchId", "stage": "$currentStageId"}, "all": bson.M{"$sum": 1}, "available": bson.M{"$sum": bson.M{"$cond": bson.A{available, 1, 0}}}, "mine": bson.M{"$sum": bson.M{"$cond": bson.A{mine, 1, 0}}}}}}}
|
||||
cur, err := s.humanItems.Aggregate(ctx, pipeline, options.Aggregate().SetAllowDiskUse(false).SetMaxTime(8*time.Second))
|
||||
if err != nil {
|
||||
return AdminAnnotationPoolCounts{}, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
var groups []struct {
|
||||
ID struct{ Batch, Stage string } `bson:"_id"`
|
||||
AdminAnnotationPoolCounts `bson:",inline"`
|
||||
}
|
||||
if err := cur.All(ctx, &groups); err != nil {
|
||||
return AdminAnnotationPoolCounts{}, err
|
||||
}
|
||||
ids := []string{}
|
||||
seen := map[string]bool{}
|
||||
for _, g := range groups {
|
||||
if g.ID.Batch != "" && !seen[g.ID.Batch] {
|
||||
ids = append(ids, g.ID.Batch)
|
||||
seen[g.ID.Batch] = true
|
||||
}
|
||||
}
|
||||
batches, err := s.ListHumanBatchAccessRecords(ctx, org, ids)
|
||||
if err != nil {
|
||||
return AdminAnnotationPoolCounts{}, err
|
||||
}
|
||||
byID := map[string]domain.HumanBatchRecord{}
|
||||
for _, b := range batches {
|
||||
byID[b.ID] = b
|
||||
}
|
||||
c := AdminAnnotationPoolCounts{}
|
||||
for _, g := range groups {
|
||||
if adminPoolValidBatch(domain.HumanItemRecord{CurrentStageID: g.ID.Stage}, byID[g.ID.Batch], org) {
|
||||
c.All += g.All
|
||||
c.Available += g.Available
|
||||
c.Mine += g.Mine
|
||||
}
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
166
internal/store/admin_annotation_pool_test.go
Normal file
166
internal/store/admin_annotation_pool_test.go
Normal file
@ -0,0 +1,166 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"droboticflow/internal/domain"
|
||||
"fmt"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/mongo"
|
||||
"go.mongodb.org/mongo-driver/mongo/options"
|
||||
"net/url"
|
||||
"os"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestAdminAnnotationPoolMongoParity(t *testing.T) {
|
||||
uri := os.Getenv("POOL_LOCAL_MONGO_URI")
|
||||
if uri == "" {
|
||||
t.Skip("isolated local Mongo opt-in")
|
||||
}
|
||||
u, err := url.Parse(uri)
|
||||
if err != nil || u.Scheme != "mongodb" || u.Host != "127.0.0.1:27018" {
|
||||
t.Fatal("local Mongo 27018 only")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
||||
defer cancel()
|
||||
client, err := mongo.Connect(ctx, options.Client().ApplyURI(uri))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer client.Disconnect(context.Background())
|
||||
db := client.Database(fmt.Sprintf("admin_pool_test_%d", time.Now().UnixNano()))
|
||||
defer db.Drop(context.Background())
|
||||
s := NewMongoStore(db)
|
||||
if err := s.EnsureIndexes(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mem := NewMemoryStore()
|
||||
const org = "pool-org"
|
||||
q := AdminAnnotationPoolQuery{ProjectID: "project", UserID: "admin", Pool: "available", Limit: 201}
|
||||
batch := domain.HumanBatchRecord{ID: "batch", OrganizationID: org, ProjectID: q.ProjectID, PluginID: "annotation-qa", Stages: domain.DefaultFanyuV58HumanStages()}
|
||||
_ = mem.UpsertHumanBatch(ctx, batch)
|
||||
_ = s.UpsertHumanBatch(ctx, batch)
|
||||
items, tasks, batchDocs := []any{}, []any{}, []any{}
|
||||
for n := 0; n < 1500; n++ {
|
||||
i := domain.HumanItemRecord{ID: fmt.Sprintf("i-%04d", n), OrganizationID: org, ProjectID: q.ProjectID, PluginID: "annotation-qa", BatchID: batch.ID, CurrentStageID: "annotation_combined", Status: domain.HumanItemReady, LegacyTaskID: fmt.Sprintf("t-%04d", n), CreatedAt: fmt.Sprintf("2026-10-21T%02d:%02d:%02dZ", n/3600, n/60%60, n%60), Subject: domain.HumanSubjectRef{ItemKey: fmt.Sprintf("CAPTURE_%04d_camera_left_part0001.mp4", n), Payload: map[string]any{"sliceWorkflow": map[string]any{"status": "editing", "phase": "slice"}, "annotationWorkspace": map[string]any{"description": strings.Repeat("x", 8192)}}}}
|
||||
b := batch
|
||||
b.ID = fmt.Sprintf("batch-%04d", n)
|
||||
b.RunID = b.ID
|
||||
batchDocs = append(batchDocs, b)
|
||||
_ = mem.UpsertHumanBatch(ctx, b)
|
||||
i.BatchID = b.ID
|
||||
if n%7 == 0 {
|
||||
i.AssignedUserID = "admin"
|
||||
i.Status = domain.HumanItemClaimed
|
||||
}
|
||||
if n%11 == 0 {
|
||||
i.CurrentStageID = "quality_review"
|
||||
}
|
||||
if n == 4 {
|
||||
i.RunID = "custom-run"
|
||||
}
|
||||
if n == 1 {
|
||||
i.BatchID = "missing"
|
||||
}
|
||||
if n == 2 {
|
||||
i.OrganizationID = "other-org"
|
||||
}
|
||||
task := domain.AnnotationTaskRecord{ID: i.LegacyTaskID, OrganizationID: i.OrganizationID, ProjectID: i.ProjectID, Status: domain.AnnotationTaskStatusPendingAnnotation, SourceVideo: i.Subject.ItemKey}
|
||||
items = append(items, i)
|
||||
tasks = append(tasks, task)
|
||||
_ = mem.UpsertHumanItem(ctx, i)
|
||||
_ = mem.UpsertAnnotationTask(ctx, task)
|
||||
}
|
||||
if _, err := s.humanBatches.InsertMany(ctx, batchDocs); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.humanItems.InsertMany(ctx, items); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.annotationTasks.InsertMany(ctx, tasks); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
custom := domain.CaptureRecord{ID: "custom-capture", OrganizationID: org, ProjectID: q.ProjectID, CaptureID: "客户指定Capture"}
|
||||
run := domain.RunRecord{ID: "custom-run", OrganizationID: org, ProjectID: q.ProjectID, TaskID: custom.ID}
|
||||
if e := s.UpsertCaptureRecord(ctx, custom); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
_ = mem.UpsertCaptureRecord(ctx, custom)
|
||||
if e := s.UpsertRun(ctx, run); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
_ = mem.UpsertRun(ctx, run)
|
||||
|
||||
start := time.Now()
|
||||
counts, err := s.CountAdminAnnotationPools(ctx, org, q)
|
||||
want, _ := mem.CountAdminAnnotationPools(ctx, org, q)
|
||||
if err != nil || counts != want {
|
||||
t.Fatalf("counts=%+v want=%+v err=%v", counts, want, err)
|
||||
}
|
||||
t.Logf("1500-item counts: %s %+v", time.Since(start), counts)
|
||||
for _, scenario := range []struct{ pool, search, category string }{{"available", "", ""}, {"all", "CAPTURE_0003", ""}, {"all", "", "pending_slice"}, {"mine", "", ""}, {"all", "[.*", ""}, {"all", "客户指定Capture", ""}} {
|
||||
q.Pool, q.Search, q.Category = scenario.pool, scenario.search, scenario.category
|
||||
start = time.Now()
|
||||
got, err := s.ListAdminAnnotationPool(ctx, org, q)
|
||||
expected, _ := mem.ListAdminAnnotationPool(ctx, org, q)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ids := func(items []domain.HumanItemRecord) []string {
|
||||
out := []string{}
|
||||
for _, i := range items {
|
||||
out = append(out, i.ID)
|
||||
if _, ok := i.Subject.Payload["annotationWorkspace"]; ok {
|
||||
t.Fatal("heavy workspace leaked")
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
if !reflect.DeepEqual(ids(got), ids(expected)) {
|
||||
t.Fatalf("%+v Mongo=%v memory=%v", scenario, ids(got), ids(expected))
|
||||
}
|
||||
t.Logf("%+v page=%d elapsed=%s", scenario, len(got), time.Since(start))
|
||||
if len(got) > 201 {
|
||||
t.Fatal("unbounded page")
|
||||
}
|
||||
if scenario.search == "" && len(got) > 0 {
|
||||
last := got[len(got)-1]
|
||||
q.Cursor = &domain.HumanItemCursor{CreatedAt: last.CreatedAt, ID: last.ID}
|
||||
next, err := s.ListAdminAnnotationPool(ctx, org, q)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, i := range next {
|
||||
for _, old := range got {
|
||||
if i.ID == old.ID {
|
||||
t.Fatal("cursor repeated row")
|
||||
}
|
||||
}
|
||||
}
|
||||
q.Cursor = nil
|
||||
}
|
||||
}
|
||||
scoped := q
|
||||
scoped.Search = ""
|
||||
scoped.Category = ""
|
||||
scoped.Pool = "all"
|
||||
scoped.Capture = "客户指定Capture"
|
||||
gotCounts, e := s.CountAdminAnnotationPools(ctx, org, scoped)
|
||||
wantCounts, _ := mem.CountAdminAnnotationPools(ctx, org, scoped)
|
||||
if e != nil || gotCounts != wantCounts || gotCounts.All != 1 {
|
||||
t.Fatalf("capture counts=%+v want=%+v err=%v", gotCounts, wantCounts, e)
|
||||
}
|
||||
|
||||
// Missing routing IDs must not match an arbitrary batch.
|
||||
_, err = s.humanItems.InsertOne(ctx, bson.M{"id": "bad", "organizationId": org, "projectId": "project", "pluginId": "annotation-qa", "currentStageId": "annotation_combined", "status": "ready", "createdAt": "z"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again, err := s.CountAdminAnnotationPools(ctx, org, q)
|
||||
if err != nil || again != counts {
|
||||
t.Fatalf("missing batch count=%+v err=%v", again, err)
|
||||
}
|
||||
}
|
||||
@ -309,6 +309,8 @@ type Store interface {
|
||||
GetHumanBatch(context.Context, string) (*domain.HumanBatchRecord, error)
|
||||
UpsertHumanBatch(context.Context, domain.HumanBatchRecord) error
|
||||
ListHumanItems(context.Context, string, domain.HumanItemQuery) ([]domain.HumanItemRecord, error)
|
||||
CountAdminAnnotationPools(context.Context, string, AdminAnnotationPoolQuery) (AdminAnnotationPoolCounts, error)
|
||||
ListAdminAnnotationPool(context.Context, string, AdminAnnotationPoolQuery) ([]domain.HumanItemRecord, error)
|
||||
ListAnnotationClaimCandidates(context.Context, string, string) ([]domain.HumanItemRecord, error)
|
||||
CountAnnotationClaimCandidates(context.Context, string, AnnotationClaimCountQuery) (int, error)
|
||||
GetHumanClaimReceipt(context.Context, string) (*HumanClaimReceipt, error)
|
||||
|
||||
@ -246,6 +246,7 @@ type SliceEditQuickConflict = {
|
||||
dragEdit: { before: SliceEditSnapshot; after?: SliceEditSnapshot } | null;
|
||||
};
|
||||
|
||||
const idleAdminPoolFilters: Parameters<typeof annotationOperationsApi.adminPool>[0] = { pool: "available", limit: 200 };
|
||||
const emptyHumanPools: HumanPools = { available: [], mine: [], all: [], reviewed: [] };
|
||||
const projectLoadRecoveryAttempts = 3;
|
||||
const projectLoadRecoveryDelayMs = 2_000;
|
||||
@ -745,6 +746,13 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
const [workbenchDrafts, setWorkbenchDrafts] = useState<AnnotationDraft[]>([]);
|
||||
const workbenchHostRef = useRef<HTMLDivElement>(null);
|
||||
const [taskSearch, setTaskSearch] = useState("");
|
||||
const [adminSearch, setAdminSearch] = useState("");
|
||||
const adminPoolAbortRef = useRef<AbortController | null>(null);
|
||||
useEffect(() => () => adminPoolAbortRef.current?.abort(), []);
|
||||
const [adminPoolTotals, setAdminPoolTotals] = useState<{
|
||||
scope: string;
|
||||
counts: { available: number; mine: number; all: number };
|
||||
} | null>(null);
|
||||
const [annotatorTaskView, setAnnotatorTaskView] = useState<"active" | "submitted">("active");
|
||||
const minePoolCursorRef = useRef("");
|
||||
const [submittedHumanItems, setSubmittedHumanItems] = useState<HumanItem[]>([]);
|
||||
@ -948,6 +956,47 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
// Acceptance pools are loaded together. Switching their local view must not
|
||||
// reload the same project, evidence prerequisites and package history.
|
||||
const projectPoolView = acceptanceWorkspace ? "all" : taskPoolView;
|
||||
const managerAnnotationPool = isManager && workspaceMode === "annotate";
|
||||
useEffect(() => {
|
||||
if (!managerAnnotationPool) {
|
||||
adminPoolAbortRef.current?.abort();
|
||||
return;
|
||||
}
|
||||
const timer = window.setTimeout(() => setAdminSearch(taskSearch.trim()), 350);
|
||||
return () => window.clearTimeout(timer);
|
||||
}, [managerAnnotationPool, taskSearch]);
|
||||
const managerPoolFilters = useMemo(
|
||||
() =>
|
||||
managerAnnotationPool
|
||||
? {
|
||||
projectId: projectId || undefined,
|
||||
workflowId: requestedWorkflowId || undefined,
|
||||
runId: requestedRunId || undefined,
|
||||
nodeId: requestedNodeId || undefined,
|
||||
artifactId: requestedArtifactId || undefined,
|
||||
capture: requestedCaptureDirectory || undefined,
|
||||
pool: (taskPoolView === "mine" || taskPoolView === "all" ? taskPoolView : "available") as
|
||||
| "available"
|
||||
| "mine"
|
||||
| "all",
|
||||
search: adminSearch || undefined,
|
||||
category: taskStatusFilter,
|
||||
limit: 200,
|
||||
}
|
||||
: idleAdminPoolFilters,
|
||||
[
|
||||
managerAnnotationPool,
|
||||
projectId,
|
||||
requestedWorkflowId,
|
||||
requestedRunId,
|
||||
requestedNodeId,
|
||||
requestedArtifactId,
|
||||
requestedCaptureDirectory,
|
||||
taskPoolView,
|
||||
adminSearch,
|
||||
taskStatusFilter,
|
||||
],
|
||||
);
|
||||
const projectDataScope = JSON.stringify([
|
||||
currentUser?.id,
|
||||
role,
|
||||
@ -958,6 +1007,7 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
requestedRunId,
|
||||
requestedWorkflowId,
|
||||
annotatorTaskView,
|
||||
...(managerAnnotationPool ? [managerPoolFilters] : []),
|
||||
]);
|
||||
const reviewActionScopeRef = useRef(projectDataScope);
|
||||
if (reviewActionScopeRef.current !== projectDataScope) skipClaimPoolReload.current = "";
|
||||
@ -1119,6 +1169,13 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
setLoading(foreground);
|
||||
setRefreshingProject(!foreground);
|
||||
setProjectLoadError("");
|
||||
if (managerAnnotationPool) {
|
||||
adminPoolAbortRef.current?.abort();
|
||||
adminPoolAbortRef.current = new AbortController();
|
||||
taskPoolCursorRef.current = "";
|
||||
setHumanPools(emptyHumanPools);
|
||||
setTasks([]);
|
||||
}
|
||||
try {
|
||||
if (blindAnnotator) {
|
||||
// Only load the actor's work, not every available Capture and unrelated
|
||||
@ -1192,29 +1249,47 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
// database never sorts the whole unassigned backlog (17k docs) before
|
||||
// the role filter drops almost every row.
|
||||
const reviewStageIds = humanItemStageIdsForMode(workspaceMode);
|
||||
const poolsRequest = acceptanceWorkspace
|
||||
? humanLoopApi.listAcceptancePools(humanFilters, currentUser?.id || "", isManager)
|
||||
: Promise.all([
|
||||
humanLoopApi.listItems({
|
||||
...humanFilters,
|
||||
pool: "available",
|
||||
...(reviewStageIds ? { stageIds: reviewStageIds } : {}),
|
||||
}),
|
||||
humanLoopApi
|
||||
.listItemsPage({
|
||||
const adminPoolRequest = managerAnnotationPool
|
||||
? annotationOperationsApi
|
||||
.adminPool(managerPoolFilters, { signal: adminPoolAbortRef.current?.signal })
|
||||
.then((page) => {
|
||||
if (requestID === projectLoadRequestRef.current && reviewActionScopeRef.current === projectDataScope) {
|
||||
taskPoolCursorRef.current = page.nextCursor || "";
|
||||
setAdminPoolTotals({ scope: projectDataScope, counts: page.counts });
|
||||
}
|
||||
return page;
|
||||
})
|
||||
: null;
|
||||
const poolsRequest = adminPoolRequest
|
||||
? adminPoolRequest.then((page) => ({
|
||||
available: managerPoolFilters.pool === "available" ? page.items : [],
|
||||
mine: managerPoolFilters.pool === "mine" ? page.items : [],
|
||||
all: managerPoolFilters.pool === "all" ? page.items : [],
|
||||
reviewed: [],
|
||||
}))
|
||||
: acceptanceWorkspace
|
||||
? humanLoopApi.listAcceptancePools(humanFilters, currentUser?.id || "", isManager)
|
||||
: Promise.all([
|
||||
humanLoopApi.listItems({
|
||||
...humanFilters,
|
||||
pool: "mine",
|
||||
...(blindAnnotator ? { annotationView: annotatorTaskView } : {}),
|
||||
pool: "available",
|
||||
...(reviewStageIds ? { stageIds: reviewStageIds } : {}),
|
||||
})
|
||||
.then((page) => {
|
||||
if (requestID === projectLoadRequestRef.current && reviewActionScopeRef.current === projectDataScope)
|
||||
minePoolCursorRef.current = page.nextCursor || "";
|
||||
return page.items;
|
||||
}),
|
||||
isManager ? humanLoopApi.listItems({ ...humanFilters, pool: "all" }) : Promise.resolve([]),
|
||||
reviewOnly ? humanLoopApi.listItems({ ...humanFilters, pool: "reviewed" }) : Promise.resolve([]),
|
||||
]).then(([available, mine, all, reviewed]) => ({ available, mine, all, reviewed }));
|
||||
humanLoopApi
|
||||
.listItemsPage({
|
||||
...humanFilters,
|
||||
pool: "mine",
|
||||
...(blindAnnotator ? { annotationView: annotatorTaskView } : {}),
|
||||
...(reviewStageIds ? { stageIds: reviewStageIds } : {}),
|
||||
})
|
||||
.then((page) => {
|
||||
if (requestID === projectLoadRequestRef.current && reviewActionScopeRef.current === projectDataScope)
|
||||
minePoolCursorRef.current = page.nextCursor || "";
|
||||
return page.items;
|
||||
}),
|
||||
isManager ? humanLoopApi.listItems({ ...humanFilters, pool: "all" }) : Promise.resolve([]),
|
||||
reviewOnly ? humanLoopApi.listItems({ ...humanFilters, pool: "reviewed" }) : Promise.resolve([]),
|
||||
]).then(([available, mine, all, reviewed]) => ({ available, mine, all, reviewed }));
|
||||
// Task acquisition splits by workspace: every non-manager workspace is
|
||||
// pool-driven and joins to the pool page's legacyTaskIds (the pool
|
||||
// decides visibility; a newest-first org task window can miss pool
|
||||
@ -1225,6 +1300,7 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
// role filter empties each fetched window). Only managers keep the
|
||||
// newest-first window with progressive paging.
|
||||
const tasksRequest = (async () => {
|
||||
if (adminPoolRequest) return (await adminPoolRequest).tasks;
|
||||
if (!isManager) {
|
||||
const pools = await poolsRequest;
|
||||
const ids = [
|
||||
@ -1372,6 +1448,8 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
requestedWorkflowId,
|
||||
projectPoolView,
|
||||
annotatorTaskView,
|
||||
managerAnnotationPool,
|
||||
managerPoolFilters,
|
||||
]);
|
||||
|
||||
useEffect(() => {
|
||||
@ -1951,6 +2029,7 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
// List stage — panel search plus status category; scopes only the task list.
|
||||
const matchesPoolListFilters = useCallback(
|
||||
(item: HumanItem) => {
|
||||
if (managerAnnotationPool) return true;
|
||||
const task = tasks.find((t) => t.id === item.legacyTaskId);
|
||||
const directories = captureDirectoriesFromHumanTask(item, task);
|
||||
if (
|
||||
@ -1967,7 +2046,7 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
: "pending_acceptance";
|
||||
return taskStatusFilter === "all" || status === taskStatusFilter;
|
||||
},
|
||||
[tasks, taskSearch, taskStatusFilter],
|
||||
[tasks, taskSearch, taskStatusFilter, managerAnnotationPool],
|
||||
);
|
||||
const matchesPoolFilters = useCallback(
|
||||
(item: HumanItem, pool: TaskPoolView) => matchesPoolScope(item, pool) && matchesPoolListFilters(item),
|
||||
@ -2510,8 +2589,11 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
[combinedEditorPhase, detail?.humanItem, projects, selectedTaskId, visiblePoolItems, visibleTasks, workflows],
|
||||
);
|
||||
const filteredVisibleTasks = useMemo(
|
||||
() => filterAnnotationTasks(visibleTasks, taskContextById, taskSearch, taskStatusFilter),
|
||||
[taskContextById, taskSearch, taskStatusFilter, visibleTasks],
|
||||
() =>
|
||||
managerAnnotationPool
|
||||
? visibleTasks
|
||||
: filterAnnotationTasks(visibleTasks, taskContextById, taskSearch, taskStatusFilter),
|
||||
[taskContextById, taskSearch, taskStatusFilter, visibleTasks, managerAnnotationPool],
|
||||
);
|
||||
// Progressive rendering cap: a 17k-task pool renders fine in the store but
|
||||
// freezes the tab when every card mounts at once. The pool reveals cards in
|
||||
@ -2537,6 +2619,31 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
return;
|
||||
setTaskPoolFetchBusy(true);
|
||||
try {
|
||||
if (managerAnnotationPool) {
|
||||
const scope = projectDataScope;
|
||||
const epoch = projectLoadRequestRef.current;
|
||||
adminPoolAbortRef.current?.abort();
|
||||
const controller = new AbortController();
|
||||
adminPoolAbortRef.current = controller;
|
||||
const page = await annotationOperationsApi.adminPool(
|
||||
{
|
||||
...managerPoolFilters,
|
||||
cursor: taskPoolCursorRef.current,
|
||||
},
|
||||
{ signal: controller.signal },
|
||||
);
|
||||
if (reviewActionScopeRef.current !== scope || projectLoadRequestRef.current !== epoch) return;
|
||||
taskPoolCursorRef.current = page.nextCursor || "";
|
||||
setAdminPoolTotals({ scope, counts: page.counts });
|
||||
const pool = managerPoolFilters.pool;
|
||||
setHumanPools((current) => ({
|
||||
...current,
|
||||
[pool]: [...current[pool], ...page.items.filter((i) => !current[pool].some((old) => old.id === i.id))],
|
||||
}));
|
||||
setTasks((current) => [...current, ...page.tasks.filter((i) => !current.some((old) => old.id === i.id))]);
|
||||
setTaskRenderLimit((current) => current + 200);
|
||||
return;
|
||||
}
|
||||
if (blindAnnotator && taskPoolView === "mine" && minePoolCursorRef.current) {
|
||||
const scope = projectDataScope;
|
||||
const page = await loadAnnotationSidebar(annotationOperationsApi, humanLoopApi, {
|
||||
@ -2595,8 +2702,16 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
const poolCounts = useMemo(
|
||||
// Scope stage only: the search/status panel filters scope the task list
|
||||
// below the tabs, never the pool tab totals (ticket 02).
|
||||
() => poolTabCounts(humanPools, matchesPoolScope),
|
||||
[humanPools, matchesPoolScope],
|
||||
() =>
|
||||
managerAnnotationPool
|
||||
? {
|
||||
...(adminPoolTotals?.scope === projectDataScope
|
||||
? adminPoolTotals.counts
|
||||
: { available: 0, mine: 0, all: 0 }),
|
||||
reviewed: 0,
|
||||
}
|
||||
: poolTabCounts(humanPools, matchesPoolScope),
|
||||
[humanPools, matchesPoolScope, managerAnnotationPool, adminPoolTotals, projectDataScope],
|
||||
);
|
||||
const stageEmptyLabel =
|
||||
workspaceMode === "quality"
|
||||
@ -6688,11 +6803,13 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
className={`rounded-lg border px-3 py-2 text-xs transition-colors ${taskPoolView === value ? "border-blue-500 bg-blue-50 text-blue-600 dark:bg-blue-500/10 dark:text-blue-200 font-medium" : "border-[color:var(--app-border)] text-slate-600 hover:text-slate-900 dark:text-gray-400 dark:hover:text-white"}`}
|
||||
>
|
||||
{label}{" "}
|
||||
{blindAnnotator && value === "available"
|
||||
? claimableCount.projectId === projectId
|
||||
? (claimableCount.count ?? "—")
|
||||
: "—"
|
||||
: count}
|
||||
{managerAnnotationPool && adminPoolTotals?.scope !== projectDataScope
|
||||
? "—"
|
||||
: blindAnnotator && value === "available"
|
||||
? claimableCount.projectId === projectId
|
||||
? (claimableCount.count ?? "—")
|
||||
: "—"
|
||||
: count}
|
||||
{/* Counts only loaded pages; a further page means more exist. */}
|
||||
{blindAnnotator && value === "mine" && minePoolCursorRef.current ? "+" : ""}
|
||||
</button>
|
||||
|
||||
@ -61,6 +61,29 @@ export const annotationOperationsApi = {
|
||||
apiRequest<{ items: HumanItem[]; tasks: AnnotationTaskRecord[]; nextCursor?: string; hasMore?: boolean }>(
|
||||
`/api/plugins/annotation-qa/sidebar${query(filters)}`,
|
||||
),
|
||||
adminPool: (
|
||||
filters: {
|
||||
projectId?: string;
|
||||
workflowId?: string;
|
||||
runId?: string;
|
||||
nodeId?: string;
|
||||
artifactId?: string;
|
||||
capture?: string;
|
||||
search?: string;
|
||||
category?: string;
|
||||
pool: "available" | "mine" | "all";
|
||||
cursor?: string;
|
||||
limit?: number;
|
||||
},
|
||||
options: { signal?: AbortSignal } = {},
|
||||
) =>
|
||||
apiRequest<{
|
||||
items: HumanItem[];
|
||||
tasks: AnnotationTaskRecord[];
|
||||
counts: { available: number; mine: number; all: number };
|
||||
nextCursor?: string;
|
||||
hasMore?: boolean;
|
||||
}>(`/api/plugins/annotation-qa/admin-pool${query(filters)}`, options),
|
||||
claimCount: (projectId: string) =>
|
||||
apiRequest<{ projectId: string; count: number }>(
|
||||
`/api/plugins/annotation-qa/claim-count?projectId=${encodeURIComponent(projectId)}`,
|
||||
|
||||
114
tests/browser/admin-annotation-pool-replica.mjs
Normal file
114
tests/browser/admin-annotation-pool-replica.mjs
Normal file
@ -0,0 +1,114 @@
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { chromium } from "playwright";
|
||||
import { createServer } from "vite";
|
||||
const api = process.env.ADMIN_POOL_API;
|
||||
assert.match(api, /^http:\/\/127\.0\.0\.1:\d+$/);
|
||||
const out = path.resolve("local-data/test-results/admin-annotation-pool");
|
||||
fs.mkdirSync(out, { recursive: true });
|
||||
const report = { pageErrors: [], requests: [], writes: [], cases: [] };
|
||||
const server = await createServer({
|
||||
server: { host: "127.0.0.1", port: 5205, strictPort: true, proxy: { "/api": { target: api, changeOrigin: true } } },
|
||||
});
|
||||
let browser, page;
|
||||
let releaseSlow;
|
||||
let delaySearch = "";
|
||||
try {
|
||||
await server.listen();
|
||||
browser = await chromium.launch({
|
||||
executablePath: process.env.BROWSER_EXECUTABLE || "/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge",
|
||||
headless: true,
|
||||
args: ["--no-proxy-server"],
|
||||
});
|
||||
page = await browser.newPage({ viewport: { width: 1600, height: 1100 } });
|
||||
page.setDefaultTimeout(15000);
|
||||
page.on("pageerror", (e) => report.pageErrors.push(e.message));
|
||||
await page.route("**/*", async (route) => {
|
||||
const r = route.request(),
|
||||
u = new URL(r.url());
|
||||
if (u.hostname !== "127.0.0.1") return route.abort();
|
||||
if (u.pathname.includes("/worker-api/")) return route.fulfill({ json: { status: "ok" } });
|
||||
if (!["GET", "HEAD"].includes(r.method())) report.writes.push(u.pathname);
|
||||
if (u.pathname.includes("/admin-pool")) {
|
||||
report.requests.push(Object.fromEntries(u.searchParams));
|
||||
if (delaySearch && u.searchParams.get("search") === delaySearch) {
|
||||
delaySearch = "";
|
||||
const response = await route.fetch();
|
||||
await new Promise((r) => (releaseSlow = r));
|
||||
return route.fulfill({ response });
|
||||
}
|
||||
}
|
||||
return route.continue();
|
||||
});
|
||||
const base = "http://127.0.0.1:5205/plugins/annotation-qa/workflows/all/annotate?projectId=draft-project";
|
||||
await page.goto(base);
|
||||
const available = page.getByTestId("human-task-pool-available"),
|
||||
all = page.getByTestId("human-task-pool-all"),
|
||||
cards = page.getByTestId("annotation-task-card");
|
||||
await available.getByText("可领取 606", { exact: true }).waitFor();
|
||||
await all.getByText("全部任务 611", { exact: true }).waitFor();
|
||||
assert.equal(await cards.count(), 200);
|
||||
await page.getByTestId("task-pool-show-more").click();
|
||||
await page.waitForTimeout(500);
|
||||
assert.equal(await cards.count(), 400);
|
||||
assert.match(await available.innerText(), /606/);
|
||||
report.cases.push("total independent of human cursor pages");
|
||||
const search = page.getByRole("textbox", { name: "搜索标注任务", exact: true });
|
||||
await search.fill("CAPTURE_0001");
|
||||
await cards.filter({ hasText: "CAPTURE_0001" }).waitFor();
|
||||
assert.equal(await cards.count(), 1);
|
||||
assert.match(await available.innerText(), /606/);
|
||||
report.cases.push("old item search beyond first500");
|
||||
await page.getByRole("combobox", { name: "按任务状态筛选", exact: true }).selectOption("pending_acceptance");
|
||||
await page.waitForTimeout(700);
|
||||
assert.equal(await cards.count(), 0);
|
||||
assert.match(await available.innerText(), /606/);
|
||||
report.cases.push("category does not change total");
|
||||
await page.getByRole("combobox", { name: "按任务状态筛选", exact: true }).selectOption("all");
|
||||
await search.fill("");
|
||||
await available.getByText("可领取 606", { exact: true }).waitFor();
|
||||
await page.waitForTimeout(500);
|
||||
delaySearch = "CAPTURE_0002";
|
||||
await search.fill(delaySearch);
|
||||
for (let n = 0; n < 40 && !releaseSlow; n++) await page.waitForTimeout(50);
|
||||
assert.ok(releaseSlow, "held older search response");
|
||||
await search.fill("CAPTURE_0003");
|
||||
await cards.filter({ hasText: "CAPTURE_0003" }).waitFor();
|
||||
releaseSlow();
|
||||
releaseSlow = null;
|
||||
await page.waitForTimeout(300);
|
||||
assert.equal(await cards.count(), 1);
|
||||
assert.match(await cards.first().innerText(), /CAPTURE_0003/);
|
||||
report.cases.push("stale search response ignored");
|
||||
await search.fill("");
|
||||
await all.click();
|
||||
await page.waitForTimeout(700);
|
||||
await page.getByRole("combobox", { name: "按任务状态筛选", exact: true }).selectOption("pending_quality");
|
||||
for (let n = 0; n < 80 && (await cards.count()) !== 4; n++) await page.waitForTimeout(50);
|
||||
assert.equal(await cards.count(), 4);
|
||||
assert.match(await all.innerText(), /611/);
|
||||
assert.match(await available.innerText(), /606/);
|
||||
report.cases.push("all pool and server status filter include older review-stage rows");
|
||||
const claimed = await fetch(api + "/__fixture/claim", { method: "POST" });
|
||||
assert.equal(claimed.status, 200);
|
||||
await page.reload();
|
||||
await available.getByText("可领取 605", { exact: true }).waitFor();
|
||||
assert.match(await all.innerText(), /611/);
|
||||
report.cases.push("real claim invalidates cached total");
|
||||
assert.deepEqual(report.pageErrors, []);
|
||||
assert.deepEqual(report.writes, []);
|
||||
console.log(JSON.stringify(report));
|
||||
} catch (e) {
|
||||
report.failure = e.message;
|
||||
if (page) {
|
||||
fs.writeFileSync(path.join(out, "failure.txt"), await page.locator("body").innerText());
|
||||
await page.screenshot({ path: path.join(out, "failure.png") });
|
||||
}
|
||||
throw e;
|
||||
} finally {
|
||||
releaseSlow?.();
|
||||
fs.writeFileSync(path.join(out, "report.json"), JSON.stringify(report, null, 2));
|
||||
await browser?.close();
|
||||
await server.close();
|
||||
}
|
||||
Loading…
x
Reference in New Issue
Block a user