⚡ Optimize annotation sidebar and progress reads

This commit is contained in:
d-robotics 2026-10-17 13:11:00 +08:00
parent ba6c46cffa
commit 7b739fd4c6
26 changed files with 1153 additions and 112 deletions

View File

@ -0,0 +1,25 @@
# List read optimization — local implementation
Status: resolved
User authorized implementing the agreed optimization on 2026-10-17. No PR, push, production deployment, bandwidth change, migration, or customer write test is authorized by this task.
## Baseline
Production ba6c46cf: 100-client gzip sidebar mine->task join p95 2925 ms; single-user 205–353 ms. Generic annotator available pool 4.9–6.0 sec; candidate batch distinct examines 22,247 docs, returns 1.24 MB. API CPU not throttled. Runtime steady limits 24 CPU/60 GiB after approved Worker 8Gi correction.
## Delivery
1. Shared typed reads behind legacy list handlers and a read-only actor sidebar interface. Empty page must not query all tasks. Reuse only same-request batch/permission metadata; preserve actor, project, view, pagination, progress, rework and ownership rules. New frontend uses one read; only 404/405 falls back to legacy reads during mixed-version rollout. Later pages use the same interface.
2. Latest progress-only slice/annotation revision queries. Exclude system/revision-zero annotation suggestions before choosing the latest human revision; preserve current draft precedence and progress rules.
3. Remove global candidate-batch distinct from blind pool visibility. Fetch compact exceptional-stage metadata with exact alias/first-duplicate-stage semantics; validate Memory/Mongo equivalence and bounded returned metadata under 25k normal batches. Do not use a client permission cache or discard unusual legacy stages.
4. Persist Worker 8Gi in capacity authority and preserve explicit coordinator port for fresh validators. Update related docs; no runtime resource change.
5. Reproduce red before fixes; run focused regressions, gate commands against registered baseline, local snapshot/browser and bounded synthetic local 100-user read flows. Record limits; local results are not production 100-browser acceptance.
## Proposed performance goals
Single sidebar p95 <=500ms; local 100-client sidebar p95 <=1500ms; no errors. Compare old/new under the same fixture and handler layout. Numeric local thresholds do not guarantee production network/Mongo behavior.
## Implementation evidence
Implementation and verification are recorded in `docs/verification/list-read-optimization-20261017.md`. Local independent-actor HTTP/Mongo acceptance, browser verification, race checks and official baseline gates passed without new failures. This task is delivered as a local commit only; no production operation was performed.

View File

@ -2,7 +2,7 @@
日期:2026-10-17
状态:已实现本地候选;默认关闭,尚未生产启用或完成容量验收。
状态:2026-10-17 已按批准计划在 Fanyu 启用,线上仅完成只读核验,尚未完成生产 100 浏览器容量验收。程序默认开关仍关闭。
## 背景与范围
@ -41,4 +41,4 @@
本地带会话认证的终审浏览器使用合成 HEVC 验证 Movi 挂载、切片源偏移、逐帧、仅右相机与片尾停止。首次切片定位还覆盖了媒体就绪事件丢失:定位队列在就绪轮询中重新派发未完成目标,无需额外用户按键;行为测试覆盖该恢复路径。
首轮保存 p95 约 2 秒,根因是同项目写保护锁串行。2026-10-17 用户随后授权本地实施分区锁及完整回归,后续两轮本地保存 p95 为 389 / 308 ms;实现、两阶段升级要求与限制见 [ADR 0011](0011-project-record-write-stripes.md)。默认仍为旧协议,线上激活需另行授权。
首轮保存 p95 约 2 秒,根因是同项目写保护锁串行。2026-10-17 用户随后授权本地实施分区锁及完整回归,后续两轮本地保存 p95 为 389 / 308 ms;实现、两阶段升级要求与限制见 [ADR 0011](0011-project-record-write-stripes.md)。程序默认仍为旧协议;Fanyu 已按批准的 barrier → striped 顺序激活,其他环境及未来变更仍须另行授权。

View File

@ -2,7 +2,7 @@
日期:2026-10-17
状态:用户已授权本地实施与完整验证;默认关闭,未部署或启用线上。
状态:本地完整验证通过;2026-10-17 已按批准计划在 Fanyu 完成 barrier 兼容升级、写入者核对及 striped 启用。线上仅做只读核验;程序默认仍为 legacy。
## 问题与决定

View File

@ -78,6 +78,17 @@ python3 scripts/generate_operator_docs.py --input-json /path/to/generated-operat
}
```
### Annotation sidebar reads
`GET /api/plugins/annotation-qa/sidebar` is an additive, read-only endpoint for blind annotators. It returns `{ items, tasks, nextCursor?, hasMore? }`: the actor's human-item page plus its authorized annotation task summaries and progress. It never claims work or changes buffer capacity.
- `annotationView=active|submitted` (default `active`), `projectId`, `workflowId`, `runId`, `limit` and opaque `cursor` retain the human-items list contracts. The task join uses only the returned page's legacy task IDs; client-supplied IDs or pool overrides are ignored. An empty page returns empty tasks without an unscoped task query.
- Role, organization, project, supplier and current-stage checks remain authoritative on the server. Access metadata and memberships may be reused only within this request; the next request rechecks current authority. Reviewers continue using their existing endpoints and receive 403 here.
- Progress selects the latest slice revision and latest human annotation revision (`revision > 0`, trimmed author not `system`) per authorized task. Only ranges/statuses are loaded for revision progress. A newer draft retains its existing precedence; history/detail endpoints retain complete records.
- Existing `/api/human-items` and annotation task endpoints remain available. The sidebar client may fall back to them only for 404/405 during a rolling upgrade; 403 and server failures propagate. First load and subsequent pages use the same read interface.
The generic blind available-pool read retains historical stage aliases, first duplicate-stage-ID precedence, and organization-scoped batch lookup even for legacy project mismatches. It queries compact exceptional-stage metadata rather than distinct batch IDs from the unclaimed backlog. This still scans matching organization batch metadata in Mongo and can grow with exceptional custom batches; it is not a constant-cost cache or a permission snapshot across requests.
### Projects
- `GET /api/projects`

View File

@ -75,7 +75,7 @@
},
"limits": {
"cpu": "1",
"memory": "2Gi"
"memory": "8Gi"
},
"env": {
"DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE": "striped",
@ -105,12 +105,12 @@
"enforced": false
},
"constraints": [
"Offline proposal only; no deployment authorization is inferred.",
"Offline rendering only; the baseline was deployed on 2026-10-17, but future plans still require separate deployment authorization.",
"Six CPU cores and remaining memory stay unallocated after the proposed steady services and legacy-export budget.",
"Operator Job Pods have independent resource specifications. The export allowance is a planning reservation, not a namespace quota or implemented global admission limit.",
"Do not enable source-index output until API, revision workers, validators, workflow workers and the package operator all understand the new schemas.",
"Mongo and BOS are external; this budget does not expand their capacity or network bandwidth.",
"The first validation of a source version creates five immutable internal BOS copies, then later revisions write JSON only. No historical customer-data backfill is enabled.",
"Local implementation only: upgrade every Mongo writer to barrier mode and finish all rollouts before enabling striped mode; old gate-only writers are incompatible."
"Upgrade every Mongo writer to barrier mode and finish all rollouts before enabling striped mode; old gate-only writers are incompatible."
]
}

View File

@ -1,6 +1,6 @@
# 源引用切片:100 人容量候选
# 源引用切片:100 人容量基线
本地候选,未部署;100 人本地保存门槛已通过,尚未进行生产 100 浏览器最终容量验收。
2026-10-17 已按批准计划将 ba6c46cf 部署到 Fanyu,并完成线上只读就绪核验。100 人本地保存门槛已通过,尚未进行生产 100 浏览器最终容量验收。Worker 随后按批准修正恢复每副本 8 GiB;本文及 JSON 为修正后的基线,未来变更仍需单独授权。
## 资源与并行
@ -11,12 +11,12 @@
| API | 4 × 2 CPU / 4 GiB | 8 CPU / 16 GiB | 登录、列表、保存、任务状态 |
| 切片修订 | 2 × 2 CPU / 4 GiB | 4 CPU / 8 GiB | 每副本 8 路,共 16 路 JSON 修订 |
| 独立源校验 | 2 × 4 CPU / 8 GiB | 8 CPU / 16 GiB | 每副本 1 路完整校验,共 2 路 |
| Worker | 2 × 1 CPU / 2 GiB | 2 CPU / 4 GiB | 调度、交付发布 |
| Worker | 2 × 1 CPU / 8 GiB | 2 CPU / 16 GiB | 调度、交付发布 |
| Coordinator | 2 × 1 CPU / 2 GiB | 2 CPU / 4 GiB | 同步协调 |
| 常驻合计 | | 24 CPU / 48 GiB | |
| 常驻合计 | | 24 CPU / 60 GiB | |
| 旧媒体导出规划预留 | | 6 CPU / 12 GiB | 独立 Job;尚非强制准入上限 |
余下 6 CPU 与约 55.6 GB 留作余量。各算子 Job 的资源单独定义;该表不是命名空间配额,也不代表已有 GPU/normalize/export Jobs 已受 36 CPU 总限额约束。未改变它们的配置。
余下 6 CPU 与约 42.7 GB 留作余量。各算子 Job 的资源单独定义;该表不是命名空间配额,也不代表已有 GPU/normalize/export Jobs 已受 36 CPU 总限额约束。未改变它们的配置。
提高切片并行只针对完成源校验后的轻量修订。冷源排入独立验证队列,不能把 16 路全部用于解码。更多副本不解除 Mongo 项目写锁,也不增加外部 Mongo、BOS 或客户端网络容量。
@ -40,7 +40,7 @@
| 草稿保存 p95 | 2,024 ms | 389 ms | 308 ms |
| 确认受理 p95 | 2,128 ms | 591 ms | 655 ms |
两轮保存门槛均通过。分区模式默认关闭;删除、撤权、宕机及混用约束见 [ADR 0011](../adr/0011-project-record-write-stripes.md)。第一次仅放开少量集合时出现 91/100 完成的回归,已通过核对普通写入集合修复,不能复用那个中间版本。
两轮保存门槛均通过。程序默认仍为旧协议,Fanyu 已按两阶段门禁启用分区模式;删除、撤权、宕机及混用约束见 [ADR 0011](../adr/0011-project-record-write-stripes.md)。第一次仅放开少量集合时出现 91/100 完成的回归,已通过核对普通写入集合修复,不能复用那个中间版本。
线上仍需测真实 100 浏览器、HEVC 双目解码/跳帧、BOS 带宽、Mongo 延迟及高峰恢复。已审计源的双目约 2.6–4.6 Mbps/人,100 人约 260–460 Mbps 从 BOS 到客户端;不经过 API 不代表无带宽瓶颈。
@ -49,10 +49,14 @@
1. 本地完成兼容性、恢复、交付、权限回归和性能门槛;保留所有未通过项。
2. 构建并证明候选 commit 对应 API/processor、Worker、Coordinator 与独立 package 镜像摘要。全部组件支持新协议、源开关仍关闭;所有 Mongo 写入进程先启用 `barrier` 协议。
已初始化的 Fanyu 库在核对索引兼容后,为全部五类进程显式设置 `DROBOTICFLOW_DATABASE_BOOTSTRAP=verify-existing`。Coordinator 现已支持该模式,仅读取默认组织检查;其默认/`migrate` 行为仍与历史一致,不新增默认组织。容量工具为新 validator 只输出一条 bootstrap 配置,避免重复环境变量。
已初始化的 Fanyu 库在核对索引兼容后,为全部五类进程显式设置 `DROBOTICFLOW_DATABASE_BOOTSTRAP=verify-existing`。Coordinator 现已支持该模式,仅读取默认组织检查;其默认/`migrate` 行为仍与历史一致,不新增默认组织。容量工具为新 validator 只输出一条 bootstrap 配置,避免重复环境变量;同时保留处理器的显式 `DROSYNC_COORDINATOR_PORT` ConfigMap 引用,拒绝缺失、重复或 `tcp://` 服务注入值。首次部署中该引用缺失导致启动失败,已按用户授权补齐,配置模板也已修正。
3. 现有 `scripts/release.py` 仅支持 API/processor 镜像更新,保留原副本和配置。使用其 prepare/check/plan 流程;不得以此工具的旧单处理器策略冒充新资源计划通过。
4. 独立审查 Worker/Coordinator/package 更新、validator 新建和本表资源/config 变更。使用 `scripts/source_index_capacity_plan.py` 从只读 Deployment 快照生成候选补丁;工具只写本地文件,不调用集群。默认生成 `barrier` 阶段,检查镜像、候选 commit、100 人保存门槛;`--lock-stage striped` 还检查全部五个 Deployment 的桥接模式与滚动更新完成状态。另行清点旧 Pods、Jobs、CronJobs、外部脚本等写入者,确认后才可开启分区并行。
5. 确认交付模式:目录保留物理舍弃排除契约;JSON v2 须另行批准下游读取完整源及范围协议,并显式开启新交付开关。未确认前不启用源修订。
6. 核对集群/context/namespace UID、在途任务和回退策略,展示包含源副本存储成本的最终计划;取得紧邻部署的用户授权后执行。生产核验默认只读。
回退遵循 ADR 0010:已有源修订必须保留新版本读者及兼容 package。禁止删除固定源、丢弃在途任务,或直接将所有组件退到旧协议。
## 列表读取优化(本地,待发布)
2026-10-17 增加标注员侧栏合并读取,进度只返回最新有效修订的统计字段,通用盲池不再展开普通批次元数据。CPU/内存基线不增加,不修改并发槽、源修订和交付行为。验收方法与结果见 [列表读取验证](../verification/list-read-optimization-20261017.md)。线上 5 Mbps 入口带宽不在本次变更范围;本地查询提升不能替代公网及媒体并发验收。

View File

@ -0,0 +1,52 @@
# 列表读取优化验证(2026-10-17)
本次仅本地实现和验证;未推送、提交 PR、部署、调整带宽或写入生产客户记录。
## 变更与兼容
- 新增标注员只读侧栏接口,一次 HTTP 返回本人任务页与关联任务进度。首屏和翻页共用;旧接口保留,客户端仅在 404/405 时回退。403/5xx 不回退。
- 批次和成员权限只在一次请求内复用;下一次请求重新读取。空页不查询全量任务,客户端不能用 IDs/pool 参数扩大范围。同任务关联多条旧记录时,以当前关联查询的顺序决定批次;不让本人页的缓存顺序覆盖它。
- 列表进度在 Mongo 选最新修订后,仅返回范围、状态和必要版本字段。人工标注排除 revision 0 和去除空白后的 system 作者,再选最新。较新草稿仍优先;历史、详情接口保留完整内容。数据库仍可能遍历同任务的历史索引,不能宣称每任务只扫描一条记录。复用现有组织/任务/修订索引,无新增迁移。
- 通用盲池移除候选任务的 batchId distinct,仅读取组织内可能属于标注员的非标注阶段元数据,随后按现有领域函数判断。保留历史别名、首个重复阶段 ID、旧批次项目不一致和其他插件规则。正常积压不展开为批次元数据;数据库仍可能扫描组织批次,自定义异常批次很多时仍会增长。
- 容量权威源同步此前已批准的 Worker 每副本 8 GiB,常驻 limit 24 CPU/60 GiB。新校验器保留协调器端口 ConfigMap 引用,拒绝缺失、重复、tcp:// 端口;本次没有改动任何线上资源。
## 回归和门禁
先记录新接口 404、前端仍走旧路径、缺少最新进度接口、容量模板仍为 2 GiB 以及关联记录缓存顺序的失败,再实现修复。
通过的专项覆盖:
- 新旧侧栏结果等价,角色拒绝、空页范围隔离、游标不重复、已送检视图、下一请求撤权生效。
- 前端 15 个行为用例:元数据/计数慢响应不阻塞侧栏、旧响应不能覆盖新账号/视图、404/405 回退、403/500 不回退、空页不扩大查询。
- Memory/Mongo 最新进度等价:历史多版本、其他组织、系统/零版本、作者空白规范化、完整历史仍保留。
- Memory/Mongo 盲池语义等价,全部历史阶段别名 × 配置角色组合。2.5 万普通批次中返回异常批次元数据 0 条,Mongo 命令监控确认 distinct 调用 0 次;本地查询约 44 ms(不含播种)。
- 配置计划 6 项专项测试;完整 release 测试 35 项通过。
- `npm run lint:web`、`npm run format:check`、`npm run build:web`、Go 三组件构建通过。前端正式基线门禁 13 项失败均为已登记旧失败;Go 正式基线门禁 70 项失败均为已登记旧失败,无新增失败。相关 API/Store 并发检测(`go test -race`)通过。
## 本地 HTTP/Mongo 性能
固定本地容器 `127.0.0.1:27018`,每次使用独立临时库,结束后删除该测试库。100 个独立合成账号,每人 8 个任务,每任务 10 个切片/标注历史版本。测试逐项校验归属与进度,200 次并发完整侧栏读取均成功。
旧路径对照通过保留的全历史 Store 方法模拟原进度读取,并发出 mine → tasks 两次 HTTP;新路径使用合并接口和最新进度查询。测试不是旧镜像对新镜像的生产 A/B。
| 热态 p95 | 旧读取路径模拟 | 优化后 |
| --- | --- | --- |
| 单账号(20 次) | 50.5 ms | 35.2 ms |
| 100 账号并发(两轮,共 200 次) | 1,149.5 ms | 632.3 ms |
100 并发 p95 下降约 45%,通过本地 ≤1,500 ms 且零错误门槛。此前两轮为 849.6 → 361.7 ms、979.8 → 526.0 ms;最终代码复测如表,不能把某一轮数字视为固定收益。
复现:
```sh
POOL_LOCAL_MONGO_URI=mongodb://127.0.0.1:27018 go test ./internal/store -run 'TestLatestProgressReads|TestPoolCountPredicates|TestPoolStageRoleCountAndVisibility|TestBlindAnnotationBacklogMetadataBounded' -count=1 -v
FANYU_SIDEBAR_MONGO_LOAD=1 FANYU_SIDEBAR_LOAD_REPORT=/tmp/sidebar-local-load-report.json go test ./internal/api -run '^TestAnnotationSidebar100ActorsMongoHTTP$' -count=1 -v
```
本地浏览器使用独立预览端口 5188/API 3018,连接快照库和既有合成演示账号;确认待处理 4 条、已送检 2 条,切换正常。连续工作开关保持手动,未领取或提交任务。既有预览服务没有停止或替换。
## 验收边界
本地测试排除了公网、BOS、媒体解码、生产 Mongo 距离与真实 100 浏览器。不能用本地 p95 对比此前线上 2.9 秒,或据此宣称生产最终容量验收通过。此前线上 API CPU 未出现限流,5 Mbps 入口仍可能影响冷加载;本次先减少往返和历史传输,不增加 CPU/内存。发布后仍需在批准的只读范围重测真实侧栏和资源曲线,再决定网络或 Mongo 优化。
Mongo 查询设计参考[官方聚合优化文档](https://www.mongodb.com/docs/manual/core/aggregation-pipeline-optimization/),实际收益以本地端到端结果为准。

View File

@ -606,7 +606,30 @@ func annotationTaskProgress(sliceRevision *domain.SliceRevisionRecord, annotatio
return progress
}
type annotationTaskListPage struct {
Items []annotationTaskListRecord
NextCursor string
HasMore bool
}
func annotationTaskPage(records []annotationTaskListRecord, raw []domain.AnnotationTaskRecord, limit int64, ids []string) *annotationTaskListPage {
cursor, more := taskPageCursor(raw, limit, ids)
return &annotationTaskListPage{Items: records, NextCursor: cursor, HasMore: more}
}
func (server *server) handleAnnotationTasksList(writer http.ResponseWriter, request *http.Request) {
page := server.readAnnotationTasksList(writer, request)
if page == nil {
return
}
response := map[string]any{"items": page.Items}
if page.HasMore {
response["nextCursor"] = page.NextCursor
response["hasMore"] = true
}
renderJSON(writer, http.StatusOK, response)
}
func (server *server) readAnnotationTasksList(writer http.ResponseWriter, request *http.Request) *annotationTaskListPage {
started := time.Now()
var listTasksMs, humanItemsMs, listReadsMs, reviewedMs, filterMs, sliceRevisionsMs, annotationRevisionsMs, draftsMs int64
defer func() {
@ -669,7 +692,7 @@ func (server *server) handleAnnotationTasksList(writer http.ResponseWriter, requ
listTasksMs = time.Since(segment).Milliseconds()
if err != nil {
renderError(writer, http.StatusInternalServerError, err)
return
return nil
}
// Human items are joined to the fetched task set via the
// organizationId+legacyTaskId index. The projection drops draft payloads;
@ -683,7 +706,7 @@ func (server *server) handleAnnotationTasksList(writer http.ResponseWriter, requ
humanItemsMs = time.Since(segment).Milliseconds()
if humanItemsErr != nil {
renderError(writer, http.StatusInternalServerError, humanItemsErr)
return
return nil
}
humanItemByTaskID := make(map[string]domain.HumanItemRecord, len(humanItems))
for _, humanItem := range humanItems {
@ -700,7 +723,7 @@ func (server *server) handleAnnotationTasksList(writer http.ResponseWriter, requ
listReadsMs = time.Since(segment).Milliseconds()
if listErr != nil {
renderError(writer, http.StatusInternalServerError, listErr)
return
return nil
}
request = request.WithContext(listCtx)
}
@ -709,7 +732,7 @@ func (server *server) handleAnnotationTasksList(writer http.ResponseWriter, requ
memberships, err = server.annotationMembershipsForUser(request.Context(), server.requestOrganizationID(request), projectID, principal.User.ID)
if err != nil {
renderError(writer, http.StatusInternalServerError, err)
return
return nil
}
}
reviewedIDs := map[string]bool{}
@ -719,7 +742,7 @@ func (server *server) handleAnnotationTasksList(writer http.ResponseWriter, requ
reviewedMs = time.Since(segment).Milliseconds()
if historyErr != nil {
renderError(writer, http.StatusInternalServerError, historyErr)
return
return nil
}
for _, id := range ids {
reviewedIDs[id] = true
@ -740,7 +763,7 @@ func (server *server) handleAnnotationTasksList(writer http.ResponseWriter, requ
item.SupplierPool, item.SupplierPoolClosed, err = server.taskUsesSupplierPool(request.Context(), item)
if err != nil {
renderError(writer, http.StatusInternalServerError, err)
return
return nil
}
if !membershipAllowsTaskRead(memberships, item) {
continue
@ -761,13 +784,7 @@ func (server *server) handleAnnotationTasksList(writer http.ResponseWriter, requ
// caller's own tasks sit on later pages (e.g. an annotator whose
// rework/claimed tasks are older than the newest import wave). Keep
// the paging metadata so limited consumers can continue.
response := map[string]any{"items": []annotationTaskListRecord{}}
if cursor, ok := taskPageCursor(items, limit, requestedIDs); ok {
response["nextCursor"] = cursor
response["hasMore"] = true
}
renderJSON(writer, http.StatusOK, response)
return
return annotationTaskPage([]annotationTaskListRecord{}, items, limit, requestedIDs)
}
// Progress is derived only after all query and authorization filters have
@ -783,14 +800,14 @@ func (server *server) handleAnnotationTasksList(writer http.ResponseWriter, requ
filteredTaskIDs = append(filteredTaskIDs, item.task.ID)
}
segment = time.Now()
sliceRevisions, sliceRevisionsErr := server.store.ListSliceRevisionsByTaskIDs(request.Context(), server.requestOrganizationID(request), filteredTaskIDs)
sliceRevisions, sliceRevisionsErr := server.store.ListLatestSliceProgressByTaskIDs(request.Context(), server.requestOrganizationID(request), filteredTaskIDs)
sliceRevisionsMs = time.Since(segment).Milliseconds()
segment = time.Now()
annotationRevisions, annotationRevisionsErr := server.store.ListAnnotationRevisionsByTaskIDs(request.Context(), server.requestOrganizationID(request), filteredTaskIDs)
annotationRevisions, annotationRevisionsErr := server.store.ListLatestAnnotationProgressByTaskIDs(request.Context(), server.requestOrganizationID(request), filteredTaskIDs)
annotationRevisionsMs = time.Since(segment).Milliseconds()
if sliceRevisionsErr != nil || annotationRevisionsErr != nil {
renderError(writer, http.StatusInternalServerError, fmt.Errorf("load annotation task progress: slices=%v annotations=%v", sliceRevisionsErr, annotationRevisionsErr))
return
return nil
}
latestSliceByTaskID := make(map[string]*domain.SliceRevisionRecord, len(filteredTasks))
for index := range sliceRevisions {
@ -840,7 +857,7 @@ func (server *server) handleAnnotationTasksList(writer http.ResponseWriter, requ
drafts, draftsErr := server.store.ListHumanItemDraftProgress(request.Context(), server.requestOrganizationID(request), draftTaskIDs)
if draftsErr != nil {
renderError(writer, http.StatusInternalServerError, fmt.Errorf("load annotation task drafts: %w", draftsErr))
return
return nil
}
for _, draft := range drafts {
draftProgressByTaskID[draft.LegacyTaskID] = draft
@ -866,14 +883,7 @@ func (server *server) handleAnnotationTasksList(writer http.ResponseWriter, requ
AnnotationEventCount: progress.AnnotationEventCount,
})
}
response := map[string]any{"items": filtered}
// Pagination metadata appears only for explicitly limited requests so the
// legacy unbounded body stays byte-compatible.
if cursor, ok := taskPageCursor(items, limit, requestedIDs); ok {
response["nextCursor"] = cursor
response["hasMore"] = true
}
renderJSON(writer, http.StatusOK, response)
return annotationTaskPage(filtered, items, limit, requestedIDs)
}
// taskPageCursor derives the continuation cursor from the raw store page.

View File

@ -0,0 +1,60 @@
package api
import (
"net/http"
"net/url"
"strings"
)
// handleAnnotationSidebar reads a page of the actor's work and its task
// summaries under one fresh request-scoped authorization snapshot. It never
// claims work, changes capacity, or expands an empty page into an unscoped list.
func (server *server) handleAnnotationSidebar(writer http.ResponseWriter, request *http.Request) {
principal, _ := currentPrincipal(request)
if !randomClaimAnnotator(principal) {
renderJSON(writer, http.StatusForbidden, map[string]string{"error": "annotator sidebar required"})
return
}
source := request.URL.Query()
view := strings.TrimSpace(source.Get("annotationView"))
if view == "" {
view = "active"
}
query := url.Values{"pluginId": {"annotation-qa"}, "pool": {"mine"}, "annotationView": {view}}
for _, key := range []string{"projectId", "workflowId", "runId", "cursor", "limit"} {
if value := source.Get(key); value != "" {
query.Set(key, value)
}
}
poolRequest := request.Clone(request.Context())
poolRequest.URL.RawQuery = query.Encode()
page, poolRequest := server.readHumanItemsList(writer, poolRequest)
if page == nil {
return
}
tasks := []annotationTaskListRecord{}
ids := []string{}
seen := map[string]bool{}
for _, item := range page.Items {
if id := strings.TrimSpace(item.LegacyTaskID); id != "" && !seen[id] {
ids = append(ids, id)
seen[id] = true
}
}
if len(ids) > 0 {
taskQuery := url.Values{"ids": {strings.Join(ids, ",")}}
if project := query.Get("projectId"); project != "" {
taskQuery.Set("projectId", project)
}
taskRequest := poolRequest.Clone(poolRequest.Context())
taskRequest.URL.RawQuery = taskQuery.Encode()
taskPage := server.readAnnotationTasksList(writer, taskRequest)
if taskPage == nil {
return
}
tasks = taskPage.Items
}
response := page.response()
response["tasks"] = tasks
renderJSON(writer, http.StatusOK, response)
}

View File

@ -0,0 +1,248 @@
package api
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"sort"
"strings"
"sync"
"testing"
"time"
"droboticflow/internal/domain"
"droboticflow/internal/store"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
// This wrapper models the previous list-progress read cost. The API's retained
// latest-selection logic makes its result equivalent while the store returns
// full history, as it did before this change.
type sidebarHistoryBaseline struct{ store.Store }
func (s sidebarHistoryBaseline) ListLatestSliceProgressByTaskIDs(ctx context.Context, org string, ids []string) ([]domain.SliceRevisionRecord, error) {
return s.ListSliceRevisionsByTaskIDs(ctx, org, ids)
}
func (s sidebarHistoryBaseline) ListLatestAnnotationProgressByTaskIDs(ctx context.Context, org string, ids []string) ([]domain.AnnotationRevisionRecord, error) {
return s.ListAnnotationRevisionsByTaskIDs(ctx, org, ids)
}
// Fixed loopback Mongo plus a unique disposable database: this opt-in test
// cannot seed or drop any customer database. It exercises real HTTP and Mongo
// with 100 distinct actors, not 100 requests sharing an identity.
func TestAnnotationSidebar100ActorsMongoHTTP(t *testing.T) {
if os.Getenv("FANYU_SIDEBAR_MONGO_LOAD") != "1" {
t.Skip("opt-in local Mongo sidebar acceptance")
}
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
defer cancel()
client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://127.0.0.1:27018/?directConnection=true").SetServerSelectionTimeout(5*time.Second))
if err != nil {
t.Fatal(err)
}
db := client.Database(fmt.Sprintf("sidebar_http_test_%d", time.Now().UnixNano()))
defer func() {
cleanup, stop := context.WithTimeout(context.Background(), 10*time.Second)
defer stop()
if err := db.Drop(cleanup); err != nil {
t.Error(err)
}
client.Disconnect(cleanup)
}()
s := store.NewMongoStore(db)
if err := s.EnsureIndexes(ctx); err != nil {
t.Fatal(err)
}
f := newFanyuV5FourStageFixture(t, domain.FanyuV5StageSemanticAnnotation, domain.HumanItemInProgress, "annotator-v5", 1)
project, _ := f.store.GetProject(ctx, f.task.ProjectID)
supplier, _ := f.store.GetSupplier(ctx, f.annotator.SupplierID)
if err := s.UpsertProject(ctx, *project); err != nil {
t.Fatal(err)
}
if err := s.UpsertSupplier(ctx, *supplier); err != nil {
t.Fatal(err)
}
actors := map[string]domain.UserRecord{}
documents := map[string][]interface{}{}
for a := 0; a < 100; a++ {
actor := f.annotator
actor.ID = fmt.Sprintf("sidebar-actor-%03d", a)
actor.Phone = actor.ID
actors[actor.ID] = actor
documents["users"] = append(documents["users"], actor)
documents["projectMembers"] = append(documents["projectMembers"], domain.ProjectMemberRecord{ID: "member-" + actor.ID, OrganizationID: actor.OrganizationID, ProjectID: f.task.ProjectID, UserID: actor.ID, SupplierID: actor.SupplierID, Roles: domain.AccountProductionRoles(actor)})
for n := 0; n < 8; n++ {
id := fmt.Sprintf("task-%03d-%02d", a, n)
task := f.task
task.ID = id
task.RunID = "run-" + id
task.ArtifactID = "artifact-" + id
task.CreatedAt = fmt.Sprintf("2026-10-17T00:00:%02dZ", n)
task.CurrentRevision = 10
item := f.item
item.ID = "item-" + id
item.LegacyTaskID = id
item.AssignedUserID = actor.ID
item.Subject.ItemKey = id
item.BatchID = "batch-" + id
item.CreatedAt = task.CreatedAt
item.UpdatedAt = "2026-10-17T00:00:00Z"
batch := f.batch
batch.ID = item.BatchID
batch.RunID = task.RunID
item.RunID = task.RunID
documents["annotationTasks"] = append(documents["annotationTasks"], task)
documents["humanItems"] = append(documents["humanItems"], item)
documents["humanBatches"] = append(documents["humanBatches"], batch)
for rev := 1; rev <= 10; rev++ {
slices := []domain.SliceSegment{{Start: 0, End: 5, AnnotationStatus: domain.SliceAnnotationLabeled}, {Start: 5, End: 10, AnnotationStatus: domain.SliceAnnotationEmptyConfirmed}, {Start: 10, End: 15, DeliveryDisposition: "discard"}, {Start: 15, End: 20}}
annotations := []domain.AnnotationSegment{{ID: "event", Start: 0, End: 5, Description: strings.Repeat("history-description-", 100)}}
documents["sliceRevisions"] = append(documents["sliceRevisions"], domain.SliceRevisionRecord{ID: fmt.Sprintf("slice-%s-%d", id, rev), OrganizationID: actor.OrganizationID, TaskID: id, Revision: rev, CreatedAt: "2026-10-17T01:00:00Z", Segments: slices, ChangeReason: strings.Repeat("operation-history-", 100)})
documents["annotationRevisions"] = append(documents["annotationRevisions"], domain.AnnotationRevisionRecord{ID: fmt.Sprintf("annotation-%s-%d", id, rev), OrganizationID: actor.OrganizationID, TaskID: id, Revision: rev, AuthorUserID: actor.ID, Segments: annotations, ObjectHints: []string{strings.Repeat("large hint ", 100)}})
}
}
}
for collection, items := range documents {
if _, err := db.Collection(collection).InsertMany(ctx, items); err != nil {
t.Fatalf("seed %s: %v", collection, err)
}
}
handlers := []http.Handler{NewHandler(sidebarHistoryBaseline{s}, nil), NewHandler(s, nil)}
httpServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
actor, ok := actors[r.Header.Get("X-Synthetic-Actor")]
if !ok {
http.Error(w, "fixture actor required", 403)
return
}
r = r.WithContext(context.WithValue(r.Context(), authContextKey{}, &authenticatedPrincipal{User: actor, Organization: domain.OrganizationRecord{ID: actor.OrganizationID, Status: domain.OrganizationStatusActive}}))
index := 1
if r.Header.Get("X-Baseline") == "1" {
index = 0
}
handlers[index].ServeHTTP(w, r)
}))
defer httpServer.Close()
type page struct {
Items []domain.HumanItemRecord `json:"items"`
Tasks []annotationTaskListRecord `json:"tasks"`
}
call := func(actor, path string, baseline bool, target any) error {
req, e := http.NewRequestWithContext(ctx, http.MethodGet, httpServer.URL+path, nil)
if e != nil {
return e
}
req.Header.Set("X-Synthetic-Actor", actor)
if baseline {
req.Header.Set("X-Baseline", "1")
}
response, e := httpServer.Client().Do(req)
if e != nil {
return e
}
defer response.Body.Close()
if response.StatusCode != 200 {
io.Copy(io.Discard, response.Body)
return fmt.Errorf("HTTP %d", response.StatusCode)
}
return json.NewDecoder(response.Body).Decode(target)
}
journey := func(actor string, baseline bool) error {
var result page
prefix := "?projectId=" + f.task.ProjectID
if !baseline {
if e := call(actor, "/api/plugins/annotation-qa/sidebar"+prefix+"&annotationView=active", false, &result); e != nil {
return e
}
} else {
if e := call(actor, "/api/human-items"+prefix+"&pluginId=annotation-qa&pool=mine&annotationView=active", true, &result); e != nil {
return e
}
ids := []string{}
for _, item := range result.Items {
ids = append(ids, item.LegacyTaskID)
}
var tasks struct {
Items []annotationTaskListRecord `json:"items"`
}
if e := call(actor, "/api/plugins/annotation-qa/tasks"+prefix+"&ids="+strings.Join(ids, ","), true, &tasks); e != nil {
return e
}
result.Tasks = tasks.Items
}
if len(result.Items) != 8 || len(result.Tasks) != 8 {
return fmt.Errorf("actor %s items=%d tasks=%d", actor, len(result.Items), len(result.Tasks))
}
for _, item := range result.Items {
if item.AssignedUserID != actor {
return fmt.Errorf("cross-actor item")
}
}
for _, task := range result.Tasks {
if task.SliceCount != 4 || task.CompletedSliceCount != 3 || task.PendingSliceCount != 1 || task.AnnotationEventCount != 1 {
return fmt.Errorf("progress changed: %+v", task)
}
}
return nil
}
type measure struct {
Count int `json:"count"`
P95 float64 `json:"p95Ms"`
Max float64 `json:"maxMs"`
}
measureRound := func(baseline bool, concurrency int) measure {
values := []float64{}
var mu sync.Mutex
var wg sync.WaitGroup
var failures []string
for n := 0; n < concurrency; n++ {
wg.Add(1)
go func(n int) {
defer wg.Done()
rounds := 2
if concurrency == 1 {
rounds = 20
}
for round := 0; round < rounds; round++ {
started := time.Now()
err := journey(fmt.Sprintf("sidebar-actor-%03d", n), baseline)
elapsed := float64(time.Since(started).Microseconds()) / 1000
mu.Lock()
values = append(values, elapsed)
if err != nil {
failures = append(failures, err.Error())
}
mu.Unlock()
}
}(n)
}
wg.Wait()
if len(failures) > 0 {
t.Fatalf("%d failures: %s", len(failures), failures[0])
}
sort.Float64s(values)
return measure{len(values), values[(len(values)*95-1)/100], values[len(values)-1]}
}
// Warm both paths before measuring; local results exclude WAN/browser/media.
if err := journey("sidebar-actor-000", true); err != nil {
t.Fatal(err)
}
if err := journey("sidebar-actor-000", false); err != nil {
t.Fatal(err)
}
result := map[string]any{"scope": "isolated local Mongo + HTTP; 100 distinct actors; eight tasks and ten revisions each; excludes production WAN/media", "baselineSingle": measureRound(true, 1), "optimizedSingle": measureRound(false, 1), "baseline100": measureRound(true, 100), "optimized100": measureRound(false, 100)}
body, _ := json.MarshalIndent(result, "", " ")
t.Log(string(body))
if output := os.Getenv("FANYU_SIDEBAR_LOAD_REPORT"); output != "" {
if err := os.WriteFile(output, append(body, '\n'), 0600); err != nil {
t.Fatal(err)
}
}
if result["optimizedSingle"].(measure).P95 > 500 || result["optimized100"].(measure).P95 > 1500 {
t.Fatal("local sidebar latency budget exceeded")
}
}

View File

@ -0,0 +1,177 @@
package api
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/url"
"reflect"
"testing"
"droboticflow/internal/domain"
"droboticflow/internal/store"
)
type sidebarReadCounter struct {
store.Store
batchPages, taskPages int
}
func (s *sidebarReadCounter) ListHumanBatchAccessRecords(ctx context.Context, org string, ids []string) ([]domain.HumanBatchRecord, error) {
s.batchPages++
return s.Store.ListHumanBatchAccessRecords(ctx, org, ids)
}
func (s *sidebarReadCounter) ListAnnotationTasksFiltered(ctx context.Context, org string, f store.AnnotationTaskFilter) ([]domain.AnnotationTaskRecord, error) {
s.taskPages++
return s.Store.ListAnnotationTasksFiltered(ctx, org, f)
}
func TestAnnotationSidebarMatchesLegacyReadsAndSharesMetadata(t *testing.T) {
f := newFanyuV5FourStageFixture(t, domain.FanyuV5StageSemanticAnnotation, domain.HumanItemInProgress, "annotator-v5", 1)
pool := f.request(http.MethodGet, "/api/human-items?pluginId=annotation-qa&pool=mine&annotationView=active&projectId="+f.task.ProjectID, f.annotator, nil)
tasks := f.request(http.MethodGet, "/api/plugins/annotation-qa/tasks?projectId="+f.task.ProjectID+"&ids="+f.task.ID, f.annotator, nil)
if pool.Code != 200 || tasks.Code != 200 {
t.Fatalf("legacy reads pool=%s tasks=%s", pool.Body, tasks.Body)
}
var wantPool struct {
Items []domain.HumanItemRecord `json:"items"`
}
var wantTasks struct {
Items []annotationTaskListRecord `json:"items"`
}
_ = json.Unmarshal(pool.Body.Bytes(), &wantPool)
_ = json.Unmarshal(tasks.Body.Bytes(), &wantTasks)
if len(wantPool.Items) != 1 || len(wantTasks.Items) != 1 {
t.Fatal("fixture must contain visible owned work")
}
counter := &sidebarReadCounter{Store: f.store}
f.handler = NewHandler(counter, nil)
response := f.request(http.MethodGet, "/api/plugins/annotation-qa/sidebar?projectId="+f.task.ProjectID+"&annotationView=active", f.annotator, nil)
if response.Code != 200 {
t.Fatalf("status=%d body=%s", response.Code, response.Body)
}
var got struct {
Items []domain.HumanItemRecord `json:"items"`
Tasks []annotationTaskListRecord `json:"tasks"`
}
if err := json.Unmarshal(response.Body.Bytes(), &got); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(got.Items, wantPool.Items) || !reflect.DeepEqual(got.Tasks, wantTasks.Items) {
t.Fatalf("sidebar must preserve legacy read result")
}
if counter.batchPages != 1 || counter.taskPages != 1 {
t.Fatalf("repeated list reads: batches=%d tasks=%d", counter.batchPages, counter.taskPages)
}
if denied := f.request(http.MethodGet, "/api/plugins/annotation-qa/sidebar?projectId="+f.task.ProjectID, f.quality, nil); denied.Code != 403 {
t.Fatalf("reviewer status=%d", denied.Code)
}
}
func TestAnnotationSidebarEmptyPageCannotExpandTaskQuery(t *testing.T) {
f := newFanyuV5FourStageFixture(t, domain.FanyuV5StageSemanticAnnotation, domain.HumanItemInProgress, "annotator-v5", 1)
counter := &sidebarReadCounter{Store: f.store}
f.handler = NewHandler(counter, nil)
response := f.request(http.MethodGet, "/api/plugins/annotation-qa/sidebar?projectId=missing&ids="+url.QueryEscape(f.task.ID), f.annotator, nil)
if response.Code != 200 {
t.Fatalf("status=%d body=%s", response.Code, response.Body)
}
var body struct {
Items []domain.HumanItemRecord `json:"items"`
Tasks []annotationTaskListRecord `json:"tasks"`
}
_ = json.Unmarshal(response.Body.Bytes(), &body)
if len(body.Items) != 0 || len(body.Tasks) != 0 || counter.taskPages != 0 {
t.Fatalf("empty scope queried tasks: %#v calls=%d", body, counter.taskPages)
}
}
func TestAnnotationSidebarPaginationSubmittedAndFreshAuthority(t *testing.T) {
f := newFanyuV5FourStageFixture(t, domain.FanyuV5StageSemanticAnnotation, domain.HumanItemInProgress, "annotator-v5", 1)
for i := 0; i < 3; i++ {
task := f.task
task.ID = fmt.Sprintf("sidebar-task-%d", i)
task.CreatedAt = fmt.Sprintf("2026-10-17T00:00:0%dZ", i)
item := f.item
item.ID = fmt.Sprintf("sidebar-item-%d", i)
item.LegacyTaskID = task.ID
item.CreatedAt = task.CreatedAt
item.Subject.ItemKey = task.ID
if i == 2 {
item.Status = domain.HumanItemSubmitted
item.AssignedUserID = ""
item.LatestAuthorUserID = f.annotator.ID
item.CurrentStageID = domain.FanyuV5StageSemanticReview
}
if err := f.store.UpsertAnnotationTask(f.ctx, task); err != nil {
t.Fatal(err)
}
if err := f.store.UpsertHumanItem(f.ctx, item); err != nil {
t.Fatal(err)
}
}
type response struct {
Items []domain.HumanItemRecord `json:"items"`
Tasks []annotationTaskListRecord `json:"tasks"`
NextCursor string `json:"nextCursor"`
HasMore bool `json:"hasMore"`
}
read := func(view, cursor string) response {
t.Helper()
r := f.request(http.MethodGet, "/api/plugins/annotation-qa/sidebar?projectId="+f.task.ProjectID+"&limit=1&annotationView="+view+"&cursor="+url.QueryEscape(cursor), f.annotator, nil)
if r.Code != 200 {
t.Fatalf("sidebar=%d %s", r.Code, r.Body)
}
var got response
if err := json.Unmarshal(r.Body.Bytes(), &got); err != nil {
t.Fatal(err)
}
return got
}
first := read("active", "")
if len(first.Items) != 1 || len(first.Tasks) != 1 || !first.HasMore {
t.Fatalf("first=%+v", first)
}
next := read("active", first.NextCursor)
if len(next.Items) != 1 || len(next.Tasks) != 1 || first.Items[0].ID == next.Items[0].ID {
t.Fatalf("cursor repeats: %+v", next)
}
submitted := read("submitted", "")
if len(submitted.Items) != 1 || submitted.Items[0].ID != "sidebar-item-2" || len(submitted.Tasks) != 1 {
t.Fatalf("submitted=%+v", submitted)
}
if _, err := f.store.DeleteProjectMember(f.ctx, "project-member-legacy:"+f.task.ProjectID+":"+f.annotator.ID); err != nil {
t.Fatal(err)
}
revoked := read("active", "")
if len(revoked.Items) != 0 || len(revoked.Tasks) != 0 {
t.Fatalf("permission cached across requests: %+v", revoked)
}
}
func TestHumanListReadsReuseKeepsCurrentTaskJoinOrdering(t *testing.T) {
f := newFanyuV5FourStageFixture(t, domain.FanyuV5StageSemanticAnnotation, domain.HumanItemInProgress, "annotator-v5", 1)
server := &server{store: f.store}
old := f.item
newer := old
newer.ID = "newer-linked-item"
newer.BatchID = "newer-batch"
ctx, err := server.withHumanListReads(f.ctx, f.task.OrganizationID, f.task.ProjectID, []domain.HumanItemRecord{old})
if err != nil {
t.Fatal(err)
}
batch := f.batch
batch.ID = newer.BatchID
if err := f.store.UpsertHumanBatch(f.ctx, batch); err != nil {
t.Fatal(err)
}
ctx, err = server.withHumanListReads(ctx, f.task.OrganizationID, f.task.ProjectID, []domain.HumanItemRecord{newer, old})
if err != nil {
t.Fatal(err)
}
item, err := server.humanItemForLegacyTask(ctx, f.task)
if err != nil || item == nil || item.ID != newer.ID {
t.Fatalf("fresh task join ordering lost: item=%+v err=%v", item, err)
}
}

View File

@ -22,17 +22,24 @@ type humanListReads struct {
}
func (server *server) withHumanListReads(ctx context.Context, org, project string, items []domain.HumanItemRecord) (context.Context, error) {
cache := &humanListReads{organization: org, project: project, batches: map[string]*domain.HumanBatchRecord{}, itemsByTask: map[string][]domain.HumanItemRecord{}, memberships: map[humanListMembershipKey]humanListMembershipResult{}}
cache, ok := ctx.Value(humanListReadsKey{}).(*humanListReads)
if !ok || cache.organization != org || cache.project != project {
cache = &humanListReads{organization: org, project: project, batches: map[string]*domain.HumanBatchRecord{}, itemsByTask: map[string][]domain.HumanItemRecord{}, memberships: map[humanListMembershipKey]humanListMembershipResult{}}
}
if len(items) > 0 {
ids := make([]string, 0, len(items))
seen := map[string]bool{}
for _, item := range items {
if !seen[item.BatchID] {
if _, loaded := cache.batches[item.BatchID]; !loaded && !seen[item.BatchID] {
ids = append(ids, item.BatchID)
seen[item.BatchID] = true
}
}
batches, err := server.store.ListHumanBatchAccessRecords(ctx, org, ids)
var batches []domain.HumanBatchRecord
var err error
if len(ids) > 0 {
batches, err = server.store.ListHumanBatchAccessRecords(ctx, org, ids)
}
if err != nil {
return ctx, err
}
@ -40,8 +47,15 @@ func (server *server) withHumanListReads(ctx context.Context, org, project strin
cache.batches[batches[i].ID] = &batches[i]
}
}
// A later task join may include linked records absent from the actor's
// page. Replace each task's group in that read's order; merging the actor
// page first would change which batch governs legacy supplier-pool reads.
groups := map[string][]domain.HumanItemRecord{}
for _, item := range items {
cache.itemsByTask[item.LegacyTaskID] = append(cache.itemsByTask[item.LegacyTaskID], item)
groups[item.LegacyTaskID] = append(groups[item.LegacyTaskID], item)
}
for taskID, group := range groups {
cache.itemsByTask[taskID] = group
}
return context.WithValue(ctx, humanListReadsKey{}, cache), nil
}

View File

@ -276,7 +276,28 @@ func (server *server) handleHumanBatchesList(writer http.ResponseWriter, request
renderJSON(writer, http.StatusOK, map[string]any{"items": items})
}
type humanItemListPage struct {
Items []domain.HumanItemRecord `json:"items"`
NextCursor string `json:"nextCursor,omitempty"`
HasMore bool `json:"hasMore,omitempty"`
}
func (page *humanItemListPage) response() map[string]any {
result := map[string]any{"items": page.Items}
if page.HasMore {
result["nextCursor"] = page.NextCursor
result["hasMore"] = true
}
return result
}
func (server *server) handleHumanItemsList(writer http.ResponseWriter, request *http.Request) {
page, _ := server.readHumanItemsList(writer, request)
if page != nil {
renderJSON(writer, http.StatusOK, page.response())
}
}
func (server *server) readHumanItemsList(writer http.ResponseWriter, request *http.Request) (*humanItemListPage, *http.Request) {
pool := strings.TrimSpace(request.URL.Query().Get("pool"))
principal, _ := currentPrincipal(request)
if pool == "" {
@ -288,26 +309,26 @@ func (server *server) handleHumanItemsList(writer http.ResponseWriter, request *
}
if pool != "available" && pool != "mine" && pool != "all" && pool != "reviewed" {
renderValidationError(writer, validationIssue{Field: "pool", Message: "pool must be available, mine, reviewed, or all"})
return
return nil, request
}
if pool == "all" && !annotationManager(principal) {
renderJSON(writer, http.StatusForbidden, map[string]string{"error": "only administrators may list all human items"})
return
return nil, request
}
query, queryIssue := parseHumanItemQuery(request)
if queryIssue != nil {
renderValidationError(writer, *queryIssue)
return
return nil, request
}
if pool == "reviewed" {
if principal == nil || (principal.User.Role != domain.UserRoleQualityReviewer && principal.User.Role != domain.UserRoleAcceptanceReviewer && !annotationManager(principal)) {
renderJSON(writer, http.StatusForbidden, map[string]string{"error": "review history requires reviewer access"})
return
return nil, request
}
ids, err := server.store.ListReviewedHumanItemIDs(request.Context(), server.requestOrganizationID(request), principal.User.ID)
if err != nil {
renderError(writer, http.StatusInternalServerError, err)
return
return nil, request
}
query.IDs = ids
}
@ -318,11 +339,11 @@ func (server *server) handleHumanItemsList(writer http.ResponseWriter, request *
if view := strings.TrimSpace(request.URL.Query().Get("annotationView")); view != "" {
if pool != "mine" || !randomClaimAnnotator(principal) {
renderJSON(writer, 403, map[string]string{"error": "annotator mine view required"})
return
return nil, request
}
if view != "active" && view != "submitted" {
renderValidationError(writer, validationIssue{Field: "annotationView", Message: "annotationView must be active or submitted"})
return
return nil, request
}
query.AnnotationView = view
}
@ -331,7 +352,7 @@ func (server *server) handleHumanItemsList(writer http.ResponseWriter, request *
parsed, issue := parseDatabaseBrowserPageQuery(rawLimit, "limit", humanItemsDefaultLimit, 1, humanItemsMaxLimit)
if issue != nil {
renderValidationError(writer, *issue)
return
return nil, request
}
limit = parsed
}
@ -361,12 +382,12 @@ func (server *server) handleHumanItemsList(writer http.ResponseWriter, request *
items, err := server.store.ListHumanItems(request.Context(), server.requestOrganizationID(request), query)
if err != nil {
renderError(writer, http.StatusInternalServerError, err)
return
return nil, request
}
listCtx, listErr := server.withHumanListReads(request.Context(), server.requestOrganizationID(request), query.ProjectID, items)
if listErr != nil {
renderError(writer, http.StatusInternalServerError, listErr)
return
return nil, request
}
request = request.WithContext(listCtx)
pageExhausted := len(items) < fetchSize
@ -377,7 +398,7 @@ func (server *server) handleHumanItemsList(writer http.ResponseWriter, request *
batch, err = server.humanBatchForListRead(request.Context(), item.BatchID)
if err != nil {
renderError(writer, http.StatusInternalServerError, err)
return
return nil, request
}
batchByID[item.BatchID] = batch
}
@ -454,7 +475,7 @@ func (server *server) handleHumanItemsList(writer http.ResponseWriter, request *
events, err := server.store.ListLatestReviewEvents(request.Context(), server.requestOrganizationID(request), principal.User.ID, ids)
if err != nil {
renderError(writer, http.StatusInternalServerError, err)
return
return nil, request
}
for index := range filtered {
if event, ok := events[filtered[index].ID]; ok {
@ -476,7 +497,7 @@ func (server *server) handleHumanItemsList(writer http.ResponseWriter, request *
downstream, err := server.store.ListHumanItems(request.Context(), server.requestOrganizationID(request), domain.HumanItemQuery{IDs: acceptanceIDs})
if err != nil {
renderError(writer, http.StatusInternalServerError, err)
return
return nil, request
}
for _, item := range downstream {
accepted[item.ID] = item.Status == domain.HumanItemAccepted
@ -496,22 +517,19 @@ func (server *server) handleHumanItemsList(writer http.ResponseWriter, request *
for index := range filtered {
if err := resolver.enrichItem(request.Context(), &filtered[index]); err != nil {
renderError(writer, http.StatusInternalServerError, err)
return
return nil, request
}
server.anonymizeV58AcceptanceItem(request.Context(), &filtered[index])
if err := server.projectRejectedAcceptanceData(request.Context(), &filtered[index]); err != nil {
renderError(writer, http.StatusInternalServerError, err)
return
return nil, request
}
}
// Keep the legacy {"items"} body byte-compatible when the page is
// exhausted; pagination metadata appears only when a next page exists.
response := map[string]any{"items": filtered}
page := &humanItemListPage{Items: filtered, HasMore: hasMore}
if hasMore {
response["nextCursor"] = encodeHumanItemCursor(nextCursor)
response["hasMore"] = true
page.NextCursor = encodeHumanItemCursor(nextCursor)
}
renderJSON(writer, http.StatusOK, response)
return page, request
}
func (server *server) anonymizeV58AcceptanceItem(ctx context.Context, item *domain.HumanItemRecord) {

View File

@ -1470,6 +1470,7 @@ func NewHandlerWithConfig(dataStore store.Store, aiService ai.Service, appConfig
router.Get("/api/human-batches", instance.handleHumanBatchesList)
router.Get("/api/human-items", instance.handleHumanItemsList)
router.Get("/api/plugins/annotation-qa/claim-projects", instance.handleAnnotationClaimProjects)
router.Get("/api/plugins/annotation-qa/sidebar", instance.handleAnnotationSidebar)
router.Get("/api/plugins/annotation-qa/claim-count", instance.handleAnnotationClaimCount)
router.Get("/api/human-items/{id}", instance.handleHumanItemGet)
router.Get("/api/human-items/{id}/review-session", instance.handleReviewSession)

View File

@ -0,0 +1,120 @@
package store
import (
"context"
"sort"
"strings"
"droboticflow/internal/domain"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
)
// These projections are deliberately not history records: list progress needs
// ranges/statuses only. History consumers keep the full revision APIs.
func compactSliceProgress(r domain.SliceRevisionRecord) domain.SliceRevisionRecord {
result := domain.SliceRevisionRecord{TaskID: r.TaskID, Revision: r.Revision, CreatedAt: r.CreatedAt, Segments: make([]domain.SliceSegment, len(r.Segments))}
for i, s := range r.Segments {
result.Segments[i] = domain.SliceSegment{Start: s.Start, End: s.End, AnnotationStatus: s.AnnotationStatus, DeliveryDisposition: s.DeliveryDisposition}
}
return result
}
func compactAnnotationProgress(r domain.AnnotationRevisionRecord) domain.AnnotationRevisionRecord {
result := domain.AnnotationRevisionRecord{TaskID: r.TaskID, Revision: r.Revision, AuthorUserID: r.AuthorUserID, Segments: make([]domain.AnnotationSegment, len(r.Segments))}
for i, s := range r.Segments {
result.Segments[i] = domain.AnnotationSegment{Start: s.Start, End: s.End}
}
return result
}
func (s *MemoryStore) ListLatestSliceProgressByTaskIDs(_ context.Context, org string, ids []string) ([]domain.SliceRevisionRecord, error) {
allowed := annotationTaskIDSet(ids)
s.mu.Lock()
defer s.mu.Unlock()
latest := map[string]domain.SliceRevisionRecord{}
for _, r := range s.sliceRevisions {
if annotationOrganizationMatches(r.OrganizationID, org) && allowed[r.TaskID] {
if old, ok := latest[r.TaskID]; !ok || r.Revision > old.Revision {
latest[r.TaskID] = r
}
}
}
result := make([]domain.SliceRevisionRecord, 0, len(latest))
for _, r := range latest {
result = append(result, compactSliceProgress(r))
}
sort.Slice(result, func(i, j int) bool { return result[i].TaskID < result[j].TaskID })
return result, nil
}
func (s *MemoryStore) ListLatestAnnotationProgressByTaskIDs(_ context.Context, org string, ids []string) ([]domain.AnnotationRevisionRecord, error) {
allowed := annotationTaskIDSet(ids)
s.mu.Lock()
defer s.mu.Unlock()
latest := map[string]domain.AnnotationRevisionRecord{}
for _, r := range s.annotationRevisions {
if annotationOrganizationMatches(r.OrganizationID, org) && allowed[r.TaskID] && r.Revision > 0 && strings.TrimSpace(r.AuthorUserID) != "system" {
if old, ok := latest[r.TaskID]; !ok || r.Revision > old.Revision {
latest[r.TaskID] = r
}
}
}
result := make([]domain.AnnotationRevisionRecord, 0, len(latest))
for _, r := range latest {
result = append(result, compactAnnotationProgress(r))
}
sort.Slice(result, func(i, j int) bool { return result[i].TaskID < result[j].TaskID })
return result, nil
}
func latestProgressPipeline(org string, ids []string, annotation bool) mongo.Pipeline {
filter := bson.M{"taskId": bson.M{"$in": ids}}
projection := bson.M{"_id": 0, "taskId": 1, "revision": 1, "segments.start": 1, "segments.end": 1}
if annotation {
filter["revision"] = bson.M{"$gt": 0}
// Match Go strings.TrimSpace exactly; Mongo defaults also trim NUL.
filter["$expr"] = bson.M{"$ne": bson.A{bson.M{"$trim": bson.M{"input": bson.M{"$ifNull": bson.A{"$authorUserId", ""}}, "chars": "\t\n\v\f\r \u0085\u00a0\u1680\u2000\u2001\u2002\u2003\u2004\u2005\u2006\u2007\u2008\u2009\u200a\u2028\u2029\u202f\u205f\u3000"}}, "system"}}
projection["authorUserId"] = 1
} else {
projection["createdAt"] = 1
projection["segments.annotationStatus"] = 1
projection["segments.deliveryDisposition"] = 1
}
// Ascending revision order uses the existing org/task/revision index; $last
// selects the latest eligible record. Project before grouping $$ROOT so the
// retained group records never contain operation history or descriptions.
return mongo.Pipeline{
{{Key: "$match", Value: mongoAnnotationFilter(org, filter)}},
{{Key: "$sort", Value: bson.D{{Key: "taskId", Value: 1}, {Key: "revision", Value: 1}}}},
{{Key: "$project", Value: projection}},
{{Key: "$group", Value: bson.M{"_id": "$taskId", "record": bson.M{"$last": "$$ROOT"}}}},
{{Key: "$replaceRoot", Value: bson.M{"newRoot": "$record"}}},
}
}
func listLatestProgress[T any](ctx context.Context, collection *mongo.Collection, org string, ids []string, annotation bool) ([]T, error) {
unique := annotationTaskIDSet(ids)
ids = make([]string, 0, len(unique))
for id := range unique {
ids = append(ids, id)
}
sort.Strings(ids)
items := make([]T, 0, len(ids))
for start := 0; start < len(ids); start += mongoTaskIDBatchSize {
cursor, err := collection.Aggregate(ctx, latestProgressPipeline(org, ids[start:min(start+mongoTaskIDBatchSize, len(ids))], annotation))
if err != nil {
return nil, err
}
var batch []T
err = cursor.All(ctx, &batch)
cursor.Close(ctx)
if err != nil {
return nil, err
}
items = append(items, batch...)
}
return items, nil
}
func (s *MongoStore) ListLatestSliceProgressByTaskIDs(ctx context.Context, org string, ids []string) ([]domain.SliceRevisionRecord, error) {
return listLatestProgress[domain.SliceRevisionRecord](ctx, s.sliceRevisions, org, ids, false)
}
func (s *MongoStore) ListLatestAnnotationProgressByTaskIDs(ctx context.Context, org string, ids []string) ([]domain.AnnotationRevisionRecord, error) {
return listLatestProgress[domain.AnnotationRevisionRecord](ctx, s.annotationRevisions, org, ids, true)
}

View File

@ -0,0 +1,80 @@
package store
import (
"context"
"fmt"
"reflect"
"testing"
"droboticflow/internal/domain"
)
func testLatestProgressReads(t *testing.T, s Store) {
t.Helper()
ctx := context.Background()
org := "progress-org"
for _, task := range []string{"a", "b", "outside"} {
for rev := 0; rev < 20; rev++ {
slice := domain.SliceRevisionRecord{ID: fmt.Sprintf("slice-%s-%d", task, rev), OrganizationID: org, TaskID: task, Revision: rev, CreatedAt: fmt.Sprint(rev), ChangeReason: "private history", Segments: []domain.SliceSegment{{ID: "slice", Start: 0, End: 10, Name: "large name", AnnotationStatus: domain.SliceAnnotationLabeled, DeliveryDisposition: "discard"}}, Operations: []domain.SliceOperation{{}}}
if err := s.UpsertSliceRevision(ctx, slice); err != nil {
t.Fatal(err)
}
annotation := domain.AnnotationRevisionRecord{ID: fmt.Sprintf("annotation-%s-%d", task, rev), OrganizationID: org, TaskID: task, Revision: rev, AuthorUserID: "human", Segments: []domain.AnnotationSegment{{ID: "event", Start: 1, End: 5, Description: "large description"}}, ObjectHints: []string{"large hints"}}
if rev > 17 {
annotation.AuthorUserID = " system "
}
if err := s.UpsertAnnotationRevision(ctx, annotation); err != nil {
t.Fatal(err)
}
}
}
// Organization isolation and human-only selection happen before latest wins.
for _, r := range []domain.AnnotationRevisionRecord{
{ID: "other-org", OrganizationID: "other-org", TaskID: "a", Revision: 99, AuthorUserID: "human"},
{ID: "system-only", OrganizationID: org, TaskID: "system-only", Revision: 5, AuthorUserID: "\u0085system\u0085"},
{ID: "nul-author", OrganizationID: org, TaskID: "nul-author", Revision: 1, AuthorUserID: "\x00system"},
{ID: "suggestion-only", OrganizationID: org, TaskID: "suggestion-only", Revision: 0, AuthorUserID: "human"},
} {
if err := s.UpsertAnnotationRevision(ctx, r); err != nil {
t.Fatal(err)
}
}
ids := []string{"a", "b", "a", "system-only", "suggestion-only", "missing", "nul-author"}
slices, err := s.ListLatestSliceProgressByTaskIDs(ctx, org, ids)
if err != nil || len(slices) != 2 {
t.Fatalf("slices=%v err=%v", slices, err)
}
for _, r := range slices {
if r.Revision != 19 || len(r.Operations) != 0 || r.Segments[0].Name != "" || r.Segments[0].ID != "" || r.Segments[0].End != 10 || r.CreatedAt != "19" || r.Segments[0].DeliveryDisposition != "discard" {
t.Fatalf("invalid slice projection: %+v", r)
}
}
annotations, err := s.ListLatestAnnotationProgressByTaskIDs(ctx, org, ids)
if err != nil || len(annotations) != 3 {
t.Fatalf("annotations=%v err=%v", annotations, err)
}
for _, r := range annotations {
if r.TaskID == "nul-author" {
if r.AuthorUserID != "\x00system" {
t.Fatal("NUL author changed")
}
continue
}
if r.Revision != 17 || r.AuthorUserID != "human" || len(r.ObjectHints) != 0 || !reflect.DeepEqual(r.Segments, []domain.AnnotationSegment{{Start: 1, End: 5}}) {
t.Fatalf("invalid annotation projection: %+v", r)
}
}
empty, err := s.ListLatestSliceProgressByTaskIDs(ctx, org, nil)
if err != nil || len(empty) != 0 {
t.Fatalf("empty query expanded: %v %v", empty, err)
}
all, err := s.ListSliceRevisionsByTaskIDs(ctx, org, []string{"a"})
if err != nil || len(all) != 20 || all[0].Segments[0].Name != "large name" {
t.Fatalf("history changed: %v", err)
}
}
func TestLatestProgressReadsMemory(t *testing.T) { testLatestProgressReads(t, NewMemoryStore()) }
func TestLatestProgressReadsMongo(t *testing.T) {
_, s := poolLocalMongo(t)
testLatestProgressReads(t, s)
}

View File

@ -404,24 +404,34 @@ func blindAnnotationVisible(item domain.HumanItemRecord, batch domain.HumanBatch
return false
}
func (s *MongoStore) blindAnnotationMongoVisibility(ctx context.Context, org string, q domain.HumanItemQuery, baseFilter bson.M) (bson.M, error) {
// Only exceptional non-annotation stages can expose unowned work to a blind
// annotator. Normal annotation/review batches need no metadata expansion. This
// remains organization-scoped (legacy item/batch project mismatches are valid).
// The predicate is deliberately a superset; the domain resolver and first-ID
// rule below remain authoritative, including historical role aliases.
func (s *MongoStore) blindAnnotationExceptionalBatches(ctx context.Context, org string) ([]domain.HumanBatchRecord, error) {
stage := bson.M{"kind": bson.M{"$ne": domain.HumanStageAnnotation}, "$or": bson.A{
bson.M{"id": bson.M{"$in": bson.A{"split_annotation", domain.FanyuV5StageCutAdjustment, "split_quality_review", domain.FanyuV5StageCutReview, domain.FanyuV5StageSemanticAnnotation}}},
bson.M{"id": bson.M{"$nin": bson.A{"quality_review", domain.FanyuV5StageSemanticReview, "acceptance", "prepackage_acceptance"}}, "requiredRole": bson.M{"$nin": bson.A{"quality_reviewer", "acceptance_reviewer"}}},
}}
cursor, err := s.humanBatches.Find(ctx, bson.M{"organizationId": normalizedOrganizationID(org), "stages": bson.M{"$elemMatch": stage}}, options.Find().SetProjection(bson.M{"_id": 0, "id": 1, "stages.id": 1, "stages.kind": 1, "stages.requiredRole": 1}))
if err != nil {
return nil, err
}
defer cursor.Close(ctx)
batches := []domain.HumanBatchRecord{}
if err := cursor.All(ctx, &batches); err != nil {
return nil, err
}
return batches, nil
}
func (s *MongoStore) blindAnnotationMongoVisibility(ctx context.Context, org string, q domain.HumanItemQuery) (bson.M, error) {
own := bson.A{bson.M{"assignedUserId": q.BlindAnnotationActorID}, bson.M{"latestAuthorUserId": q.BlindAnnotationActorID}}
if q.PluginID != "annotation-qa" {
own = append(own, bson.M{"pluginId": bson.M{"$ne": "annotation-qa"}})
}
values, err := s.humanItems.Distinct(ctx, "batchId", baseFilter)
if err != nil {
return nil, err
}
batchIDs := make([]string, 0, len(values))
for _, value := range values {
if id, ok := value.(string); ok {
batchIDs = append(batchIDs, id)
}
}
// Restrict access metadata to matching items. Do not filter batch project:
// legacy cross-project metadata still determines exceptional-stage visibility.
batches, err := s.ListHumanBatchAccessRecords(ctx, org, batchIDs)
batches, err := s.blindAnnotationExceptionalBatches(ctx, org)
if err != nil {
return nil, err
}

View File

@ -748,7 +748,7 @@ func (store *MongoStore) ListHumanItems(ctx context.Context, organizationID stri
}
filter := humanItemMongoFilter(organizationID, query)
if query.BlindAnnotationActorID != "" {
visibility, err := store.blindAnnotationMongoVisibility(ctx, organizationID, query, filter)
visibility, err := store.blindAnnotationMongoVisibility(ctx, organizationID, query)
if err != nil {
return nil, err
}

View File

@ -8,10 +8,12 @@ import (
"os"
"reflect"
"sort"
"sync/atomic"
"testing"
"time"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/event"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
@ -147,7 +149,7 @@ func testPoolCountPredicates(t *testing.T, s Store) {
}
func TestPoolCountPredicatesMemory(t *testing.T) { testPoolCountPredicates(t, NewMemoryStore()) }
func poolLocalMongo(t *testing.T) (context.Context, *MongoStore) {
func poolLocalMongo(t *testing.T, monitors ...*event.CommandMonitor) (context.Context, *MongoStore) {
t.Helper()
uri := os.Getenv("POOL_LOCAL_MONGO_URI")
if uri == "" {
@ -159,7 +161,11 @@ func poolLocalMongo(t *testing.T) (context.Context, *MongoStore) {
}
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
t.Cleanup(cancel)
client, err := mongo.Connect(ctx, options.Client().ApplyURI(uri))
clientOptions := options.Client().ApplyURI(uri)
if len(monitors) > 0 {
clientOptions.SetMonitor(monitors[0])
}
client, err := mongo.Connect(ctx, clientOptions)
if err != nil {
t.Fatal("local Mongo connect failed")
}
@ -208,7 +214,7 @@ func TestPoolPerformanceMongoBacklog(t *testing.T) {
}
indexed := explain(humanItemMongoFilter(batch.OrganizationID, q))
q.BlindAnnotationActorID = "actor"
visibility, err := s.blindAnnotationMongoVisibility(ctx, batch.OrganizationID, q, humanItemMongoFilter(batch.OrganizationID, q))
visibility, err := s.blindAnnotationMongoVisibility(ctx, batch.OrganizationID, q)
if err != nil {
t.Fatal(err)
}
@ -268,3 +274,41 @@ func TestPoolStageRoleCountAndVisibilityMatchesDomainMongo(t *testing.T) {
}
}
}
// A normal unclaimed backlog must not be expanded into batch metadata or a
// distinct batch-ID reply before a private list can return an empty page.
func TestBlindAnnotationBacklogMetadataBounded(t *testing.T) {
var distinctCalls atomic.Int32
monitor := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
if e.CommandName == "distinct" {
distinctCalls.Add(1)
}
}}
ctx, s := poolLocalMongo(t, monitor)
batches := make([]interface{}, 25000)
items := make([]interface{}, len(batches))
for i := range batches {
id := fmt.Sprintf("normal-%05d", i)
batches[i] = domain.HumanBatchRecord{ID: id, OrganizationID: "bounded-org", ProjectID: "p", Stages: domain.DefaultFanyuV58HumanStages()}
items[i] = domain.HumanItemRecord{ID: id, OrganizationID: "bounded-org", ProjectID: "p", BatchID: id, PluginID: "annotation-qa", CurrentStageID: domain.FanyuV58StageAnnotationCombined}
}
if _, err := s.humanBatches.InsertMany(ctx, batches); err != nil {
t.Fatal(err)
}
if _, err := s.humanItems.InsertMany(ctx, items); err != nil {
t.Fatal(err)
}
started := time.Now()
metadata, err := s.blindAnnotationExceptionalBatches(ctx, "bounded-org")
if err != nil || len(metadata) != 0 {
t.Fatalf("normal backlog metadata=%d err=%v", len(metadata), err)
}
result, err := s.ListHumanItems(ctx, "bounded-org", domain.HumanItemQuery{ProjectID: "p", PluginID: "annotation-qa", BlindAnnotationActorID: "actor", Limit: 100, ListProjection: true})
if err != nil || len(result) != 0 {
t.Fatalf("private list=%d err=%v", len(result), err)
}
if got := distinctCalls.Load(); got != 0 {
t.Fatalf("backlog distinct expansion returned: %d commands", got)
}
t.Logf("25k normal batches: exceptional metadata=%d list=%s", len(metadata), time.Since(started))
}

View File

@ -280,6 +280,10 @@ type Store interface {
DeleteAnnotationRevisions(context.Context, string, string) error
ListSliceRevisions(context.Context, string, string) ([]domain.SliceRevisionRecord, error)
ListSliceRevisionsByTaskIDs(context.Context, string, []string) ([]domain.SliceRevisionRecord, error)
// Progress reads return at most one compact revision per requested task.
// Annotation selection excludes system/zero revisions before latest wins.
ListLatestSliceProgressByTaskIDs(context.Context, string, []string) ([]domain.SliceRevisionRecord, error)
ListLatestAnnotationProgressByTaskIDs(context.Context, string, []string) ([]domain.AnnotationRevisionRecord, error)
UpsertSliceRevision(context.Context, domain.SliceRevisionRecord) error
DeleteSliceRevisions(context.Context, string, string) error
PruneAnnotationDataBefore(context.Context, string, string, int) error

View File

@ -79,8 +79,12 @@ def render(snapshot, candidate, profile, package_image=None, load_report=None, l
if "droboticflow-annotation-validator" not in deployments:
pod = processor["spec"]["template"]["spec"]
source = next(c for c in pod["containers"] if c["name"] == "processor")
allowed = {"MONGO_DB", "MONGO_URI", "BOS_ENDPOINT", "WORKSPACE_MOUNT_PATH", "ANNOTATION_WORKSPACE_CONNECTION_ID", "ANNOTATION_WORKSPACE_BUCKET", "ANNOTATION_WORKSPACE_MAPPINGS", "MONGO_CONNECT_RETRIES", "MONGO_CONNECT_RETRY_DELAY_MS"}
allowed = {"DROSYNC_COORDINATOR_PORT", "MONGO_DB", "MONGO_URI", "BOS_ENDPOINT", "WORKSPACE_MOUNT_PATH", "ANNOTATION_WORKSPACE_CONNECTION_ID", "ANNOTATION_WORKSPACE_BUCKET", "ANNOTATION_WORKSPACE_MAPPINGS", "MONGO_CONNECT_RETRIES", "MONGO_CONNECT_RETRY_DELAY_MS"}
env = [copy.deepcopy(e) for e in source.get("env", []) if e["name"] in allowed]
ports = [e for e in env if e["name"] == "DROSYNC_COORDINATOR_PORT"]
if len(ports) != 1 or not (ports[0].get("valueFrom", {}).get("configMapKeyRef")
or re.fullmatch(r"[0-9]+", ports[0].get("value", "")) and 0 < int(ports[0]["value"]) <= 65535):
raise ValueError("Validator requires one explicit integer coordinator port or ConfigMap reference; service-link tcp:// values cannot be parsed")
mongo = next((e for e in env if e["name"] == "MONGO_URI"), {})
if "secretKeyRef" not in mongo.get("valueFrom", {}):
raise ValueError("Mongo must remain a Secret reference; no literal credentials can be rendered")

View File

@ -11,6 +11,22 @@ export async function loadAnnotationTaskJoin(annotation, filters, ids) {
return tasks;
}
// Rolling upgrades may briefly serve an API without this read endpoint. Only
// route absence allows the legacy path; permission and server errors propagate.
export async function loadAnnotationSidebar(annotation, human, filters) {
if (annotation.getSidebar) {
try {
return await annotation.getSidebar(filters);
} catch (error) {
if (error?.status !== 404 && error?.status !== 405) throw error;
}
}
const page = await human.listItemsPage({ pluginId: "annotation-qa", pool: "mine", ...filters });
const ids = page.items.map((item) => item.legacyTaskId).filter(Boolean);
const tasks = await loadAnnotationTaskJoin(annotation, { projectId: filters.projectId }, ids);
return { ...page, tasks };
}
// Keep this orchestration separate from rendering so the slow count and stale
// scope behavior can be exercised with deferred API responses.
export async function loadOwnedAnnotationProject({
@ -36,24 +52,22 @@ export async function loadOwnedAnnotationProject({
// them immediately; project metadata still gates the full result and actions.
let failed = false;
const ownedTasks = Promise.resolve().then(async () => {
const mine = annotationView
? await human
.listItemsPage({ pluginId: "annotation-qa", projectId: projectId || undefined, pool: "mine", annotationView })
.then((page) => {
if (!failed && isCurrent()) onPage?.(page);
return page.items;
})
: await human.listItems({ pluginId: "annotation-qa", projectId: projectId || undefined, pool: "mine" });
const owned = mine.filter((item) => item.legacyTaskId);
const tasks = annotationView
? await loadAnnotationTaskJoin(
annotation,
{ projectId: projectId || undefined },
owned.map((item) => item.legacyTaskId),
)
: (await Promise.all(owned.map((item) => annotation.getTaskSummary(item.legacyTaskId)))).map(
(detail) => detail.task,
);
let mine, tasks;
if (annotationView) {
const page = await loadAnnotationSidebar(annotation, human, {
projectId: projectId || undefined,
annotationView,
});
if (!failed && isCurrent()) onPage?.(page);
mine = page.items;
tasks = page.tasks;
} else {
mine = await human.listItems({ pluginId: "annotation-qa", projectId: projectId || undefined, pool: "mine" });
const owned = mine.filter((item) => item.legacyTaskId);
tasks = (await Promise.all(owned.map((item) => annotation.getTaskSummary(item.legacyTaskId)))).map(
(detail) => detail.task,
);
}
const result = { mine, tasks };
if (!failed && isCurrent()) onTasks?.(result);
return result;

View File

@ -1,5 +1,5 @@
import { AnnotationWorkModeToggle } from "../components/AnnotationWorkModeToggle";
import { loadOwnedAnnotationProject, loadAnnotationTaskJoin } from "../lib/annotation-project-loading.js";
import { loadOwnedAnnotationProject, loadAnnotationSidebar } from "../lib/annotation-project-loading.js";
import { FailedSliceRecoveryNotice } from "../components/FailedSliceRecoveryNotice";
import { initialAnnotationPool } from "../lib/annotation-landing.js";
import { sliceEditDispositionCommands } from "../lib/slice-edit-dispositions.js";
@ -2484,21 +2484,14 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
try {
if (blindAnnotator && taskPoolView === "mine" && minePoolCursorRef.current) {
const scope = projectDataScope;
const page = await humanLoopApi.listItemsPage({
pluginId: "annotation-qa",
const page = await loadAnnotationSidebar(annotationOperationsApi, humanLoopApi, {
projectId: projectId || undefined,
workflowId: requestedWorkflowId || undefined,
runId: requestedRunId || undefined,
pool: "mine",
annotationView: annotatorTaskView,
cursor: minePoolCursorRef.current,
});
const ids = page.items.map((item) => item.legacyTaskId).filter((id): id is string => Boolean(id));
const joinedTasks = await loadAnnotationTaskJoin(
annotationOperationsApi,
{ projectId: projectId || undefined },
ids,
);
const joinedTasks = page.tasks;
if (reviewActionScopeRef.current !== scope) return;
minePoolCursorRef.current = page.nextCursor || "";
if (annotatorTaskView === "submitted")

View File

@ -50,6 +50,17 @@ export const annotationOperationsApi = {
{ method: "PUT" },
{ capacity },
),
getSidebar: (filters: {
projectId?: string;
workflowId?: string;
runId?: string;
annotationView: "active" | "submitted";
cursor?: string;
limit?: number;
}) =>
apiRequest<{ items: HumanItem[]; tasks: AnnotationTaskRecord[]; nextCursor?: string; hasMore?: boolean }>(
`/api/plugins/annotation-qa/sidebar${query(filters)}`,
),
claimCount: (projectId: string) =>
apiRequest<{ projectId: string; count: number }>(
`/api/plugins/annotation-qa/claim-count?projectId=${encodeURIComponent(projectId)}`,

View File

@ -1,6 +1,6 @@
import assert from "node:assert/strict";
import test from "node:test";
import { loadOwnedAnnotationProject } from "../../src/lib/annotation-project-loading.js";
import { loadOwnedAnnotationProject, loadAnnotationSidebar } from "../../src/lib/annotation-project-loading.js";
function deferred() {
let resolve, reject;
@ -197,3 +197,116 @@ test("project failure prevents a later task response from publishing a ready sid
await new Promise((resolve) => setImmediate(resolve));
assert.deepEqual(seen, []);
});
test("owned sidebar uses one read and publishes its cursor before auxiliary metadata", async () => {
const metadata = deferred();
const ready = [];
const pages = [];
const mine = [{ id: "mine", legacyTaskId: "task-a" }];
const tasks = [{ id: "task-a", sliceCount: 2 }];
const options = fixture(deferred()).options;
const loading = loadOwnedAnnotationProject({
...options,
annotationView: "active",
projects: { listMembers: () => metadata.promise, listWorkflows: () => metadata.promise },
human: {
listItemsPage: () => {
throw new Error("legacy pool must not run");
},
},
annotation: {
...options.annotation,
getSidebar: async (filter) => {
assert.deepEqual(filter, { projectId: "project-a", annotationView: "active" });
return { items: mine, tasks, nextCursor: "next", hasMore: true };
},
listTemplates: () => metadata.promise,
listTasksPage: () => {
throw new Error("legacy join must not run");
},
},
onTasks: (value) => ready.push(value),
onPage: (value) => pages.push(value),
});
await new Promise((resolve) => setImmediate(resolve));
try {
assert.deepEqual(ready, [{ mine, tasks }]);
assert.equal(pages[0]?.nextCursor, "next");
} finally {
metadata.resolve([]);
await loading;
}
});
test("sidebar denial never falls back to legacy access paths", async () => {
const options = fixture(deferred()).options;
let legacy = 0;
await assert.rejects(
loadOwnedAnnotationProject({
...options,
annotationView: "active",
human: {
listItemsPage: async () => {
legacy++;
return { items: [] };
},
},
annotation: {
...options.annotation,
getSidebar: async () => {
throw Object.assign(new Error("revoked"), { status: 403 });
},
},
}),
/revoked/,
);
assert.equal(legacy, 0);
});
for (const status of [404, 405, 500]) {
test(`sidebar status ${status} ${status === 500 ? "propagates" : "uses bounded legacy fallback"}`, async () => {
const calls = [];
const loading = loadAnnotationSidebar(
{
getSidebar: async () => {
throw Object.assign(new Error("unavailable"), { status });
},
listTasksPage: async (filter) => {
calls.push(filter.ids);
return { items: [{ id: "t" }] };
},
},
{
listItemsPage: async (filter) => {
calls.push(filter);
return { items: [{ legacyTaskId: "t" }], nextCursor: "cursor-2", hasMore: true };
},
},
{ projectId: "p", annotationView: "submitted", cursor: "cursor-1" },
);
if (status === 500) {
await assert.rejects(loading, /unavailable/);
assert.deepEqual(calls, []);
} else {
const result = await loading;
assert.equal(result.nextCursor, "cursor-2");
assert.deepEqual(result.tasks, [{ id: "t" }]);
assert.equal(calls[0].cursor, "cursor-1");
assert.equal(calls[0].annotationView, "submitted");
assert.deepEqual(calls[1], ["t"]);
}
});
}
test("empty legacy sidebar never issues an unscoped task query", async () => {
const result = await loadAnnotationSidebar(
{
listTasksPage: () => {
throw new Error("unscoped query");
},
},
{ listItemsPage: async () => ({ items: [], hasMore: false }) },
{ annotationView: "active" },
);
assert.deepEqual(result, { items: [], tasks: [], hasMore: false });
});

View File

@ -20,7 +20,8 @@ class SourceCapacityTests(unittest.TestCase):
continue
env = [{"name": "DROBOTICFLOW_BUILD_COMMIT", "value": "c" * 40},
{"name": "MONGO_URI", "valueFrom": {"secretKeyRef": {"name": "mongo-secret", "key": "MONGO_URI"}}},
{"name": "ANNOTATION_PROCESSOR_MODE", "value": "processor"}]
{"name": "ANNOTATION_PROCESSOR_MODE", "value": "processor"},
{"name": "DROSYNC_COORDINATOR_PORT", "valueFrom": {"configMapKeyRef": {"name": "droboticflow-config", "key": "DROSYNC_COORDINATOR_PORT"}}}]
items.append({"kind": "Deployment", "metadata": {"namespace": profile["namespace"], "name": name, "uid": name, "resourceVersion": "1"},
"spec": {"replicas": 1, "template": {"spec": {"containers": [{"name": cfg["container"], "image": "registry.test/api@sha256:" + "a" * 64,
"env": env, "volumeMounts": [{"name": "workspace", "mountPath": "/workspace"}]}],
@ -44,6 +45,33 @@ class SourceCapacityTests(unittest.TestCase):
self.assertEqual(item["patch"][1]["path"], "/metadata/resourceVersion")
self.assertFalse(any(op["path"].endswith("/image") and op["op"] != "test" for op in item["patch"]))
def test_worker_memory_and_validator_port_survive_future_plans(self):
snapshot, profile, evidence = self.fixture()
processor = next(d for d in snapshot["items"] if d["metadata"]["name"].endswith("processor"))
port = {"name": "DROSYNC_COORDINATOR_PORT", "valueFrom": {"configMapKeyRef": {"name": "droboticflow-config", "key": "DROSYNC_COORDINATOR_PORT"}}}
processor["spec"]["template"]["spec"]["containers"][0]["env"][-1] = port
result = capacity.render(snapshot, "c" * 40, profile, "registry.test/package@sha256:" + "b" * 64, evidence)
self.assertEqual(profile["deployments"]["droboticflow-worker"]["limits"]["memory"], "8Gi")
self.assertEqual(result["resourceTotals"]["steadyLimitMemoryBytes"], 60 * 2**30)
self.assertEqual(result["resourceTotals"]["unallocatedMemoryBytesAfterExportAllowance"], 120000000000 - 72 * 2**30)
env = result["createValidator"]["spec"]["template"]["spec"]["containers"][0]["env"]
self.assertEqual([e for e in env if e["name"] == port["name"]], [port])
self.assertEqual(sum(e["name"] == "DROBOTICFLOW_DATABASE_BOOTSTRAP" for e in env), 1)
def test_validator_requires_one_explicit_coordinator_port(self):
for mode in ("missing", "duplicate", "service-link"):
snapshot, profile, evidence = self.fixture()
processor = next(d for d in snapshot["items"] if d["metadata"]["name"].endswith("processor"))
env = processor["spec"]["template"]["spec"]["containers"][0]["env"]
if mode == "missing":
env.pop()
elif mode == "duplicate":
env.append(copy.deepcopy(env[-1]))
else:
env[-1] = {"name": "DROSYNC_COORDINATOR_PORT", "value": "tcp://service:3003"}
with self.subTest(mode=mode), self.assertRaises(ValueError):
capacity.render(snapshot, "c" * 40, profile, load_report=evidence)
def test_missing_images_or_slow_saves_are_blocking_not_success(self):
snapshot, profile, evidence = self.fixture()
evidence["requests"]["draft-save"]["p95Ms"] = 2000