🔀 Merge PR #60: live release line (reviewer dashboard, review video seconds, delivery v4, dashboard stability) into dev

This commit is contained in:
zhanghongming 2026-10-27 22:45:21 +08:00
commit bc17d50508
344 changed files with 27963 additions and 1844 deletions

View File

@ -0,0 +1,12 @@
# Correct administrative annotation pool totals and paging
Status: resolved
Base: 867ca698 (contains the prior local editing repairs). Local only.
The annotation workspace administrator badges must count all authorized human items, independent of the current list page, text search and status category. Project/workflow/run/Capture scope still applies. Lists must page human items, then join tasks by IDs; searching must include older items beyond the former first 500. Never fetch all business payloads into the browser, change customer records, widen annotator visibility or modify claim permissions.
Use an admin-only read endpoint, compact server-side counts and bounded cursor pages. Keep existing non-admin/review behavior and old APIs for compatibility. Test >500 items, mixed stages, orphan/cross-org batches, permissions, claim changes, old-page search, category filtering, stale page responses and pagination with stable totals. No production deployment in this task.
## Answer
Implemented admin-only totals with a 5-second bounded actor/scope cache; list pages follow human item cursors and join only page task IDs. Server-side Capture/status search sees older items; abandoned searches are cancelled and stale responses fenced. Existing role paths and APIs remain. See docs/annotation-admin-task-pool.zh-CN.md and docs/verification/admin-pool-counts-20261021.md.

View File

@ -0,0 +1,8 @@
# Branch rules and status script for parallel agents
Goal: stop branch sprawl: one branch per task from dev, a task card per branch, one command to see every branch's state
Agent: Claude Code (desktop session, frontend-polish thread)
Base: origin/dev @ 1aad4029
Scope: AGENTS.md, docs/agents/branch-workflow.md, scripts/agent_status.py, tests/agents/
Out of scope: moving release.py's ledger to one place; deleting remote branches or worktrees
Risk: tooling and docs only; --prune deletes local branches already contained in origin/dev

View File

@ -0,0 +1,12 @@
# Repair initial slicing and rework feedback
Status: resolved
Base: local rework boundary repair 4d12d8cb, based on deployed fab5be57.
- Initial slice quick conflicts must be visible and confirmable/cancellable by the same keys and buttons as annotation conflicts. Preserve pending, described and invalid records; no customer migration.
- Successful C split, drag and delete must survive save/reload and support same-session undo/redo, respecting the existing one-second acceptance window and stage boundaries.
- Live complete editor drafts update feedback immediately, preserving classification fields. Derived slice status alone must not clear opinions.
- Discarded slices retain evidence and a right-side opinion rail without an action editor. Explicit whole-action invalidation shows its impact.
- Review remains read-only; E remains annotation-only. No production mutation or deployment in this task.
Evidence: [verification](../../docs/verification/cheng-feedback-20261021.md).

View File

@ -1,6 +1,6 @@
# 客户可对账看板
Status: ready-for-agent
Status: resolved
## 已确认
@ -12,10 +12,10 @@ Status: ready-for-agent
- 提交:首次成功送检,按该日期入账;反复保存/返工重提不重复累计。明确重跑或撤销完成证据的规则需沿用正式版本链,不按 revision 大小猜测。
- 质检通过、验收通过:各按有效正式决定实际完成日期入账,重复决定/重试不重复;同一 part 可以在不同日期分别进入各阶段。撤销/打回后的处理保留事件历史及当前状态,明细显示证据,不能直接把当前最新状态当历史结论。
- 返工中:当前状态快照,不是期间返工次数;范围为当前筛选归属、在查询截止时刻仍需返工的 part。日期范围不能把历史已解决返工再次算成当前返工。若要历史截止日快照,必须有可靠状态事件重放;否则仅显示当前时刻并注明。
- 返工中:当前状态快照,不是期间返工次数;范围为当前筛选归属、在查询截止时刻仍需返工的 part。日期范围不能把历史已解决返工再次算成当前返工。若要历史截止日快照,必须有可靠状态事件重放;本版使用当前时刻快照,接口返回 asOf;不添加页面规则说明。
- 时区:显示当前账号配置的统计时区;按本地日界转 UTC 半开区间,查询与导出一致。
- 可结算参考:本期有效验收通过整段时长,不表示已付款或新增结算凭证;本期去重与跨期不重复结算需将来另立实际结算台账,原型不冒充结算系统。
- 归属:使用提交/被审版本冻结的作者、供应商;不能按当前任务分配人或现任角色重写历史。缺证据显示待核对,并显示已核实/待核对覆盖范围。
- 归属:使用提交/被审版本冻结的作者、供应商;不能按当前任务分配人或现任角色重写历史。缺少可靠证据的字段显示“—”;不在产品界面显示“待核对”、覆盖率或调试说明。
## 个人版式
@ -27,10 +27,35 @@ Status: ready-for-agent
## Mongo 补录边界
先读已有证据生成缺口清单。可恢复的字段逐项列来源与冲突,须经用户确认才补录;不能恢复的留空待核对,不为凑齐看板造记录。报表计算、可选内部统计快照与客户交付 JSON 分开。
先读已有证据生成缺口清单。可恢复的字段逐项列来源与冲突,须经用户确认才补录;不能恢复的留空,不为凑齐看板造记录。报表计算、可选内部统计快照与客户交付 JSON 分开。
## 页面与分页
- 数量直接显示数字,不附加“个 part”。缺失值显示“—”,不添加长口径解释、黄色提醒、“待核对”或演示日期。
- 任务、供应商、人员每日表分别分页,默认 5 条,可选 5/10/20/50;展示总条数、页码和上一页/下一页。筛选变更回到第一页,汇总始终覆盖完整筛选结果。
- 导出完整筛选任务明细,不受当前页影响;详情弹层只读,支持关闭和 Esc,返回保留位置。
## 本次交付状态
角色筛选的 role 投影缺陷已修复并完成本地 Mongo/真实 router 回归,待合入和另行发布。本地看板原型仅供版式/交互评审,使用明确标识的合成数据。真实指标接口、完整导出和生产历史覆盖率仍待实现/核实。
2026-10-20 已按用户批准的原型完成正式实现,保存在 `codex/customer-dashboard-20261020` 分支:
原型源码保存在独立 `codex/dashboard-reconciliation-prototype-20261020` 分支,不进入生产代码;入口为 tests/browser/fixtures/dashboard-reconciliation-prototype.html。
- 新增只读 `GET /api/dashboard/annotation-output`,支持阶段归属、日期/项目/供应商/人员筛选、独立分页和完整 CSV 导出。组织管理员看组织内正式纳入的项目,供应商管理员看本供应商,标注员看本人;客户端参数不能扩大服务端权限。
- Mongo 仅投影读取已有事实;统计源使用正式生产范围配置,不添加客户数据、不回填历史、不改变交付 JSON。时长只使用明确的整段视频源时长,不使用保留切片时长或按 15 分钟估算。
- 个人与管理员/供应商管理员接入真实 React 页面;旧的源采集生产进度/运行概览保留在管理员折叠区;质检与验收角色原有工作看板保留。
- 已验证:真实 router 与隔离合成 Mongo 的三角色浏览器流程;权限、返工/重试去重、跨期、冻结归属、缺失时长、207 条分页与完整导出;前端/Go 既有失败基线无新增;生产前端启动与 Go 构建通过。
- 上述初版已按用户授权发布为 `65c19de0`,线上三角色只读读取通过;看板请求实测约 13.7–14.7 秒。生产管理员/供应商管理员专用账号验收和历史完整性覆盖仍未完成,未执行历史补录。
- 该初版随后被同分支 `962ea6da`(共享按需快照)取代;线上当前构建为 `962ea6da`,状态仍为 `deployed_pending_acceptance`,六类业务验收证据为 `unverified`。
## 性能优化:共享按需快照(已发布,业务验收待补)
用户选择方案 2 后,改为所有 API 副本复用按组织/正式项目范围隔离的 part 统计事实,日期、人员和分页请求不再重复读取完整历史。访问触发刷新,默认间隔 120 秒;保留完整可用版本,失败退避,超过 15 分钟不再继续提供陈旧结果。卡片、明细和 CSV 固定同一快照版本,失效后整页恢复;保持确认版式。
每次请求仍重新校验当前身份和正式项目范围。业务记录、历史补录、交付 JSON 不变。该 GET 在缺少快照或刷新时会写入两个内部派生统计集合,不能再描述为服务端零写入;普通数据库浏览/写入入口禁止访问这些集合。原始审核事件保留在业务存储,快照不复制事件列表,报表详情提供阶段时间与归属。
本地验证与资源限制见 [`共享快照验证报告`](../../docs/verification/dashboard-snapshots-20261020/report.md),实现契约见 [`共享按需看板快照`](../dashboard-snapshots-20261020/spec.md)。本次优化已按用户授权于 2026-10-20 22:56 发布为 `962ea6da`;线上发布与有限验收见同一报告「线上发布状态」一节。发布记录为 Git 忽略路径 `local-data/releases/dashboard-snapshots-20261020-962ea6da/`,六类业务验收仍为 `unverified`。
验证记录:[`docs/verification/customer-dashboard-20261020/report.md`](../../docs/verification/customer-dashboard-20261020/report.md)。
## 原型历史
原型源码保存在独立 `codex/dashboard-reconciliation-prototype-20261020` 分支,不进入生产代码;入口为 `tests/browser/fixtures/dashboard-reconciliation-prototype.html`。用户确认版式后,正式页面重新接入真实接口,原型数据没有进入运行服务。

View File

@ -0,0 +1,28 @@
# 共享按需看板快照
Status: resolved
2026-10-20 用户选择方案 2。线上基线三次读取为 13.7–14.7 秒,分页在读历史/重算之后执行。同日 22:56 用户另行授权镜像上传、具体部署及内部派生统计写入,候选 `962ea6da` 已发布为线上构建;6 类业务验收证据仍为 `unverified`,发布记录状态为 `deployed_pending_acceptance`,不宣称完整业务验收通过。
## 契约
- 保留 part、整段时长、正式阶段时间、冻结作者/供应商、跨期累计、返工快照、空值和服务端权限规则。
- 共享已解释的 part 统计事实,按组织/正式项目范围/算法版本区分。所有 API 副本通过 Mongo 复用同一代快照,不按人员、日期或页码重复读历史。
- 请求时校验当前身份与范围;首次缺快照返回构建中,后台生成。有完整快照时立即读取,过期按需触发一个后台构建;无人访问不刷新。刷新默认间隔 120 秒,失败重试退避 120 秒,15 分钟后不把陈旧结果继续当作可用结果。
- 全局一个有效重建租约,续租与发布按 owner/epoch 条件更新。失效工作者不能覆盖新结果。崩溃/重启后可回收过期租约。快照完整生成、校验成功后才原子替换可见指针。
- 独立内部统计集合,不写业务记录、客户描述、媒体或交付 JSON;不把原始审核历史复制到统计快照。保留原有事实读取与纯统计算法作重建来源。
- 固定 16 个存储槽,每槽压缩载荷最多 4 MiB;总压缩载荷最多 64 MiB。每个范围保留当前和前一代,最多 8 个范围;有限复用槽位,不能无限积累历史代次。解压限制 32 MiB、重建源读取限制 64 MiB/250000 文档、同时至多两条来源查询、重建 90 秒超时。字节预算不冒充进程 RSS 硬上限。
- 卡片/明细/导出按 snapshotId 固定同一代;过期或被回收的 token 明确返回冲突,整页切到新版本,不能混用两代或静默重算导出。
- 页面保留确认版式,不加口径长提示;冷启动只显示正常加载状态并有界轮询,取消切换停止轮询。普通刷新不绕过全局限频。
## 验证
真实 router 多 handler/分页共用一次历史重建的回归;Memory/Mongo 两个 Adapter;跨 API 并发、过期刷新、失败保留、租约回收、失效 owner 发布拒绝、槽位/字节上限、固定版本导出、权限变更与范围变化;隔离合成 Mongo 三角色浏览器与有规模的计时/扫描次数;按既有前端/Go 基线门禁核对新增失败。
## 实现结果
已接入真实 API、Mongo/Memory Adapter 和正式看板。统计结果按版本分页、下钻及导出,旧客户端继续获得完整 200 响应;新客户端冷启动采用可取消的构建中轮询。内部集合 `annotationOutputSnapshotState`、`annotationOutputSnapshotSlots` 不通过普通数据库浏览/写入接口开放。
本地 5000 part、25000 条历史事件、4 个 API 实例及 32 次热请求:历史重建 1 次,首次 224 ms,热请求 P50 15 ms / P95 17 ms。该数据来自隔离合成 Mongo,不是生产性能承诺。
验证和兼容边界见 [`report.md`](../../docs/verification/dashboard-snapshots-20261020/report.md),线上发布与有限验收状态见该报告「线上发布状态」一节。本次已发布为 `962ea6da`;新增内部派生集合 `annotationOutputSnapshotState` / `annotationOutputSnapshotSlots` 的写入已列入发布计划并获得用户明确授权。发布记录与原始证据位于工作树内 Git 忽略的 `local-data/releases/dashboard-snapshots-20261020-962ea6da/` 和 `local-data/verification/dashboard-snapshots-release-20261020/`。

View File

@ -0,0 +1,33 @@
# 最终分类交付与内部追溯分离
Status: done
2026-10-21 用户要求将两轮总 JSON 核查的问题一起本地修复,并明确选择独立 `tags` 列表、管理员维护。依据为马迪 10-09 至 10-14 的格式对话与本会话补充。原始核查证据在主仓 `local-data/diagnostics/delivery-json-audit-20261021/`。
## 验收条件
- 字典词条独立 tags,不从 category/aliases 猜补。选择字典后 ID、类型、版本、名称、编码、tags 经服务器校验并冻结,切片/动作修订、物化、审查及最终 JSON 保持一致;字典后改不改写已确认结果。
- 未写动作描述仍保留切片基础分类和全部区间。显式无效与舍弃独立,不从空文本推断;舍弃原因是分类事实,保留。纯留空、混合、全无效及全舍弃均可表达。
- 新分类交付使用显式 v4 协议及默认关闭的 DELIVERY_JSON_V4(要求 PART_INDEX_DELIVERY)。客户主 JSON 保持 capture/part、slices/taskSplit/segments 的约定形态,只含定位、最终分类、字典快照、边界及必要源引用;动作 contentKind 使用马迪约定的 annotated/invalid_marked/gap_placeholder。
- v4 全舍弃 part 仍输出分类 JSON。旧目录/v1/v2/v3 的 no_deliverables 行为、历史文件和在途任务兼容保留,不静默重写旧批次。
- 交付只取最终验收通过的固定物化修订与摘要;返工正式修正须重新通过验收。批注、人员、轮次变化不能改分类,审核者不能通过标注接口修改业务结果。
- 全部已提交追溯留内部 JSON,按 capture/part/结果版本关联:审核人、批注、轮次、退回/复审及正式处理记录。客户主 JSON 不带审核历史、lineage、抽检过程或内部路径引用,客户下载不包含内部追溯文件。
- 不可变批次的选择/身份/SHA/范围/条件发布/重试规则保留;版本必须显式识别,拒绝混搭和不一致证据。历史缺少字典标签不得按最新字典猜补,保留已有名称事实和未知字段边界。
- 舍弃 reasonNote/at 与 note/discardedAt 兼容映射;原始 meta/标定路径基准显式且一致。规范、真实入口样例与校验器枚举一致。
## 验证
定向 Go/Python/前端回归,隔离本机 Mongo 与真实 router 的保存→返工→复审→验收→原生打包→批次及下载,检查最终分类/标签和完整内部追溯;覆盖失败重试、重启/旧版本读取、字典更新、未授权篡改、全舍弃、缺失历史证据。前端与 Go 全量门禁按既有基线对照;真实浏览器检查字典 tags 管理和选择,Web/Go 构建及生产包启动。
不上传镜像、不部署、不写公司数据、不补写历史交付、不 push/PR。本地完成后以新分支提交。
## 本地完成证据(2026-10-21)
- 独立分支 `codex/delivery-classification-20261021`,基于 `bc00467d`。本次只做本地代码、合成数据验证及提交;未 push、提 PR、上传镜像、部署、启用工作流或改写历史交付。
- Go 与前端全量门禁无新增失败,保留原有 70 / 13 个已知失败,基线文件未修改;类型、格式、构建、生成源码一致性及生产页面启动通过。
- 定向 Python 37 项通过(含真实视频物化→v4 的 5 项验证);一个旧版 Go 物化 opt-in 用例未单独启用,新旧真实媒体链路已由独立用例覆盖。
- 本机 27018 随机独立 Mongo 库的三轮退回和 API 重启、原生 v4 的正式接口批次发布/失败恢复/重试/下载、停用与移除词条后的直接及面板撤销重做、v5/v6 标签防伪造均通过。测试库执行后清理。
- 静态复核的 Standards / Spec 两轴发现均已修正并关闭;保留无快照历史的名称事实,不以当前 tags 猜补。未确认临时名称不能绕过最终确认/提交。
- 完整日志、分离的客户/内部样例和浏览器截图保留在本工作树 `local-data/verification/delivery-classification-20261021/`。客户样例子目录 `customer-samples/partNNNN/` 每份只含 `part.json`;`customer-samples/internal/` 不属于客户包。
- 新协议默认关闭,线上尚未验证/启用;后续正式发布需要覆盖 API/processor、Worker、package 镜像及内部 artifact 存储,按现有发布授权流程执行。未把旧镜像 release 摘要伪装成新源码摘要。

View File

@ -0,0 +1,21 @@
Implement approved local v4 contract only. Freeze source ref metadata in revision; require complete customer v4 original source fields. Derive tags from own dictionary or parent frozen dictionary without overwriting selection provenance. Explicit timestamps + local/part seconds, half-open time bases and original PTS resolution. Consumer roundtrip uses only exported JSON + synthetic original source. Legacy formats remain unchanged. Commit local only, no deploy.
Status: resolved
Acceptance:
- Effective tags and action/slice origin are explicit; independent selections and invalid flags remain intact.
- part → slices → segments hierarchy and discard-reason-only semantics remain intact.
- Frozen original BOS references survive metadata-only reuse; no connection credentials or workspace paths enter customer output.
- Action local/part times and hardware PTS boundaries are complete and continuous.
- Missing tags, source, timestamps, source files or conflicting identities reject v4 package/batch publication.
- Independent reader resolves every synthetic original camera frame from JSON alone in physical and source-index modes.
- Immutable retry/recovery, separate private audit, legacy formats and local Mongo publication remain verified.
Compatibility: v4 default stays off; no production changes, retrospective tag guessing, old delivery replacement, downstream recutting or source writes.
Verification:
- 73 relevant Python tests passed, including physical/source-index independent real-media consumption, legacy v1/v2/v3 and 60-second materialization.
- Native Python v4 output passed Go batch/API publication, failure recovery and download tests with an isolated local Mongo 27018 database.
- Go full baseline gate: the same 70 named known failures, no new failures and no baseline edits.
- Operator repository generation check and git diff whitespace check passed.
- Local evidence: `local-data/verification/delivery-consumer-20261021/`; final customer samples under `final-samples/{physical,source-index}/customer/part0001/part.json`.

View File

@ -1,6 +1,6 @@
# 独立内部测试环境落地计划
Status: proposed
Status: superseded — 2026-10-20 ADR-0004 修订替代了专用节点、新云盘、合成素材先行、Ingress 与 5 账号/6 样本等前提;见 [spec.md](spec.md) 当前修订
Date: 2026-09-27
Scope: 本轮仅制定计划,不部署、不申请付费资源、不复制客户样本、不提交 PR。

View File

@ -25,3 +25,7 @@ Blocked by: 无(触发=用户启动执行模式/部署后线上验证)
## 验证命令
- 复现栈/测试通道浏览器手工证据记录到本文件 `## Comments`
## Comments
- 2026-10-20:内部测试环境修订后,第 1 项(无 movi 服务时的浏览器播放)由 [12](12-cluster-http-verifier.md) 的浏览器播放场景覆盖,结果作为 `video-playback` 测试环境部分的证据;本票其余内容不变。

View File

@ -0,0 +1,49 @@
# Render the environment on a pinned node with local storage and a private entry
Type: task
Status: ready-for-agent
Blocked by: none
Spec: Environment. Code: `scripts/internal_test_lane/environment.py`, `tests/internal_test_lane/test_environment.py`.
Today the renderer creates an RWX workspace PVC and an Ingress, forces the Secure cookie and does
not pin nodes. The cluster has only `csi-bos` (no-provisioner) and no IngressClass.
- Replace the RWX PVC and Ingress with static local PVs for Mongo and workspace: `Retain`, RWO,
node affinity on `kubernetes.io/hostname`, explicit claim binding, per-batch names as today.
- Config gains `node` and `hostRoot`. Refuse an empty node, a `hostRoot` outside a dedicated
internal-test prefix, `droboticflow-fanyu`, and any production PV/PVC name.
- Pin API, processor, Mongo and initializer to the node.
- Render a one-time directory-preparation pod (pattern: `k8s/qwen-node-cache.prepare-dir.template.yaml`)
that creates only this batch's directories under `hostRoot`, owned for the Mongo user.
- Entry mode `private-forward`: ClusterIP only, no Ingress or TLS secret, and
`DROBOTICFLOW_SESSION_COOKIE_SECURE=false` only in this mode; other modes keep `true`.
- Processor runs `ANNOTATION_PROCESSOR_ROLE=all`, concurrency 1.
- Refuse any `BOS_*`, `ANNOTATION_WORKSPACE_*` or csi-bos reference.
- `internal-test-quota` keeps its stable name and fits API, processor, Mongo, initializer and the
preparation pod with finite limits.
- Keep default-deny; allow only app pods → Mongo. Whether port-forward needs an extra rule under
Cilium stays an explicit unverified item; never open namespace-external ingress to get it working.
Capacity (PV size and ResourceQuota do not limit what is actually written to a local disk, and
node `.4`'s disk is shared with kubelet and images; filling it evicts pods on that node, which can
include production operator Jobs):
- Config gains a capacity budget: bundle source bytes, an output factor for slice media and results
(default 2.0 × source until a real batch measures it), Mongo, dump, and retention of 3 archived
batches plus the current one. The renderer prints the total.
- The directory-preparation pod reports `df` for the filesystem holding `hostRoot`. The operator
imports that report; the tool refuses to prepare when free space minus the budget would leave less
than 25% of the filesystem free (kubelet hard eviction defaults to 10%). The threshold can be
raised, not set below 15%.
- The same preflight is required before seed, baseline start, upgrade and archive. A crossed
threshold means stop new work (operator scales API and processor to 0) and record it; nothing is
deleted automatically.
- Every pod gets ephemeral-storage requests and limits, so log and container-layer growth evicts the
test pod rather than its neighbours.
- Use a filesystem project quota on `hostRoot` when the node filesystem supports it. Otherwise the
batch record states "disk usage not enforced by the filesystem; bounded by preflight only".
Done when: public-boundary tests cover the rendered shape, the budget arithmetic, the threshold
refusal and each other refusal, existing tests are
updated, and the deployment section of `docs/internal-test-lane.md` matches. No cluster access.

View File

@ -0,0 +1,24 @@
# Derive test runtime config from a production snapshot
Type: task
Status: ready-for-agent
Blocked by: 06
Spec: Environment (runtime configuration).
- Input: a read-only snapshot of the production `droboticflow-api` and
`droboticflow-annotation-processor` Deployments plus the ConfigMaps they reference. Secrets are
never read; only references appear.
- Copy non-secret env from the snapshot, then apply an explicit override allowlist:
- isolation: namespace / `K8S_NAMESPACE`, Mongo URI and database (test Secret),
`WORKSPACE_MOUNT_PATH`, removal of `BOS_ENDPOINT` and every `ANNOTATION_WORKSPACE_*` key,
the private-forward cookie setting;
- capacity: replicas, `ANNOTATION_PROCESSOR_ROLE`, concurrency keys, resources.
- `DROBOTICFLOW_DATABASE_BOOTSTRAP` follows production (currently `verify-existing`).
- A reference to a production Secret or ConfigMap that the test namespace lacks is refused, not dropped.
- Emit `config-diff.json` per batch (key, class, before/after; secret refs shown as references);
`batch.json` binds its SHA256.
- Refuse snapshots from another namespace or with a missing Deployment.
Done when: tests cover the allowlist, refusal of unlisted differences and stable diff output, and
the guide explains the diff and why other differences disqualify a batch as evidence.

View File

@ -0,0 +1,28 @@
# Bundle and seed v2: production projects, two suppliers, five business roles
Type: task
Status: ready-for-agent
Blocked by: none
Spec: Data and accounts.
Today `scripts/internal_test_lane/lifecycle.py` (around lines 155, 201, 476) requires exactly six
samples, three each for BLF-data and SL-data, and `internal/testlane/plan.go` (around lines
133–302) hardcodes six samples, two dictionaries, those two projects, one supplier and the roles
admin / supplier_manager / annotator / quality_reviewer / acceptance_reviewer.
- New schema `droboticflow.internal-test-seed/v2`: projects come from the bundle (name,
workflow-definition snapshot, three dictionaries each); 1–12 samples, at most 3 per project;
kinds `scratch`, `snapshot` at `annotation`, `snapshot` at `quality_review`.
- Suppliers `test-a` and `test-b`, both authorized on every test project; tasks start unassigned
in the shared project pool.
- Accounts: annotator and quality reviewer per supplier, plus a `test-a` acceptance reviewer with
supplier final-acceptance authorization. No platform admin, no supplier manager.
`credentials.py` emits exactly these five.
- The seeded database must start under the production bootstrap mode (`verify-existing`).
- v1 bundles exist only as local synthetic fixtures: refuse v1 with a clear message. Keep every
existing refusal (non-empty target, non-test URI, plan/manifest hash mismatch).
- `tests/internal_test_lane/make_synthetic_bundle.py` produces v2.
Done when: Go and Python tests are updated, a fresh local Mongo seeds from a synthetic v2 bundle,
and the guide documents v2.

View File

@ -0,0 +1,42 @@
# Export samples from a restored backup and download their media
Type: task
Status: ready-for-agent
Blocked by: 08
Spec: Data and accounts, run-sequence steps 1–2.
Restoring the backup (today's `scripts/local/fetch-mongo-snapshot.sh` execs into a production API
pod, stages the archive in chunks under that pod's `/tmp`, then runs `docker rm -f
fanyu-mongo-restore`, recreates 27018 and restores with `--drop`; its default backup directory is
the fixed `fanyu-v58-ca15d60d`):
- Leave that script unchanged for its existing 27018 use; add a separate restore path for the lane.
- List backup directories under `/workspace/system/backups/` and show the chosen one's name, date
and archive size. If the newest backup is older than the selection window, stop: taking a new
production backup is a separate approved operation.
- Read the archive by byte ranges straight from the mounted file, with no staging in the production
pod. If staging cannot be avoided, check the archive size against the pod's free ephemeral space
first and refuse above a threshold. Verify the full SHA256 locally.
- Restore into a new container with an explicit unique name and an explicit unused loopback port.
Refuse when the name exists or the port is in use, and refuse 27017–27020 (existing local
stacks). Never `docker rm -f` an existing container and never `--drop` into one.
Exporting:
- Connect only to an explicit loopback Mongo URI; refuse non-loopback hosts and the existing
local containers on 27017/27018.
- `candidates`: per project active in the backup's recent window (default 14 days, flag), list
tasks by stage with Capture name, part count, required media keys and sizes. No account names
or phone numbers.
- `export`: from chosen task IDs (at most 3 per project, 12 total) write a v2 bundle skeleton:
project config, workflow-definition snapshot, dictionaries, slice/action content for snapshot
kinds, and required media (both cameras, both PTS, IMU) as BOS keys.
- Download only those keys, read-only, with credentials from the untracked local file (same
approach as `scripts/local/download-capture.sh`); record size and SHA256 into bundle assets;
resume safely; never write to BOS.
- Print the total size before downloading; refuse above a cap (default 20 GiB).
Done when: tests use a synthetic Mongo fixture, a mocked BOS client and a mocked `kubectl`/`docker`
boundary that prove the refusals (existing container, busy port, staging over threshold). This ticket does not touch
a real backup or BOS.

View File

@ -0,0 +1,31 @@
# release.py: selectable policy, validator coverage, worker guard
Type: task
Status: ready-for-agent
Blocked by: none
Spec: Promotion. Code: `scripts/release.py` (policy hardcoded at line 19), `docs/release/`.
- `--policy <path>`; without it, behaviour stays on `docs/release/fanyu-policy.json`, so existing
records keep working. Historical records with another policy hash follow the existing read-only rules.
- The Fanyu policy adds `droboticflow-annotation-validator`: same image patch, its own role env
(`validation`) and resources preserved; the cluster check verifies each Deployment's role.
- New `docs/release/internal-test-policy.json`: namespace `droboticflow-internal-test`, local-PV
workspace, no bucket/BOS checks, API + processor (role `all`). Each policy refuses the other
namespace by name and UID.
- Split `annotation-delivery` in the Fanyu policy's `requiredEvidence` into `annotation-results`
and `delivery-packaging`. `delivery-packaging` is satisfied by either:
- a packaging test record from an environment that runs the worker and package operators (the
test lane cannot produce it), or
- an unchanged-delivery-inputs check that `release.py` computes itself: the diff between the
production API build commit and the candidate over a policy-declared path list is empty. The path
list (result-JSON writers, release partition, part-index delivery, package operators) is derived
from the code and reviewed with the policy. A non-empty diff requires the packaging test record.
Records under the old policy hash stay historical; an in-progress record must be prepared again.
- Worker guard: read the production worker's build commit from the read-only snapshot; refuse
the plan if `internal/worker` or `cmd/worker` differ between it and the candidate, pointing to
separate verification.
Done when: `tests/release` (simulated Kubernetes) covers policy selection, three-Deployment patch
and partial failure, the worker guard, cross-policy refusal, and both `delivery-packaging` paths
(empty diff accepted, non-empty diff without a packaging record refused), and `docs/release/README.md` is updated.

View File

@ -0,0 +1,36 @@
# Bind test-lane batch reports to release evidence
Type: task
Status: ready-for-agent
Blocked by: 10, 13
Spec: Promotion.
- `release.py evidence <id> --check <c> --test-lane-batch <batch-id>` reads the local batch
record and requires: state `business-verified`, `candidateImage` equal to the release image
digest, a config diff limited to allowed classes, and unchanged report hashes. It stores the
batch ID, digests and report hash; anything else is refused. Batches in `failed` or
`outcome-unknown` are always refused.
- Start point: for the upgrade-rehearsal part of `change-safety`, the batch's `baselineImage` must
equal the `before` image the release plan recorded for each Deployment it covers. Production may run
different starting digests (today the validator runs `7699d455…` while api and processor run
`1a1ead7a…`); a Deployment whose `before` image differs from `baselineImage` is listed as not
covered by the rehearsal.
- Each check needs named scenarios in the report, all `passed`. A scenario that is missing, failed or
unverified refuses the check.
| Release check | Required scenarios |
| --- | --- |
| `role-login` | five business-role logins; role-boundary refusals (admin logins come from production smoke) |
| `supplier-permissions` | cross-supplier claim refused; other supplier's annotation invisible |
| `review-acceptance` | QC pass; QC return → rework resubmit; final acceptance |
| `slice-recovery` | processor restart with work resumed; in-flight tasks resumed after upgrade |
| `video-playback` | browser playback for both cameras: first frame, playback advancing (test-lane part) |
| `annotation-results` | accepted results contain normal, discard, invalid and blank cases as entered |
| `change-safety` (rehearsal part) | start-point match; in-flight state created on baseline; resumed on candidate |
`delivery-packaging` cannot be satisfied by a batch (see ticket 10).
- Production-side smoke evidence (ticket 14) is still required by `verify`.
Done when: tests cover digest mismatch, start-point mismatch, a partially covered Deployment, a
missing or failed scenario, an unverified or failed batch, a tampered report and a disallowed check.

View File

@ -0,0 +1,29 @@
# HTTP business verifier for cluster batches
Type: task
Status: ready-for-agent
Blocked by: 08
Spec: Batch flow steps 2 and 4. Today `tests/internal_test_lane/verify_local.py` reads fixed
local paths and assumes one supplier.
- Inputs: loopback base URL (port-forward), the private accounts file and the executed plan.
Refuse any non-loopback host.
- Scenarios: five logins and role-boundary refusals; `test-a` claims and `test-b` can neither
claim nor see that annotation; slicing → normal / discard / invalid / blank → submit → QC return
→ rework resubmit → QC pass → final acceptance; snapshot samples; frame range.
- Browser playback with Playwright (already a dependency): for both cameras in normal and focus
layout, the first frame renders (video ready and a non-blank sampled frame) and playback time
advances within ±20% of wall-clock time over 5 s. Record what happens without a movi service; this
covers item 1 of ticket 05 for the test lane. HTTP media responses alone never pass this scenario.
- `--phase in-flight` stops after leaving claimed, slicing-pending and submission-pending tasks;
`--phase after-upgrade` resumes exactly those tasks.
- Processor-restart recovery: the tool prints the restart step for the operator, then waits and
checks the work resumes; it never restarts pods itself.
- Every scenario has a stable ID and a `passed` / `failed` / `unverified` status, matching the table
in ticket 11.
- One final JSON report: batch ID, observed build commit/digest (cross-checked with `batch.json`),
sample keys, task and item IDs, per-check status. No credentials; never overwrites.
Done when: it runs against a fresh local stack with two suppliers, and tests cover report shape
and refusals.

View File

@ -0,0 +1,26 @@
# Batch flow: baseline-to-candidate upgrade rehearsal
Type: task
Status: ready-for-agent
Blocked by: 06, 07, 10, 12
Spec: Batch flow.
- Batch config gains pinned `baselineImage` and `candidateImage`; `batch.json` binds both.
`baselineImage` comes from a read-only snapshot of the production Deployments taken at prepare
time, and the snapshot's per-Deployment images are recorded for the start-point check in ticket 11.
- States: prepared → seeded → baseline-running → in-flight-created → upgraded →
business-verified → archived. Each transition imports operator-provided evidence as today;
the tool never probes the cluster.
- The upgrade runs `release.py` with the internal-test policy, the same code path as production.
- If the candidate needs a migration that production would run as a separate reviewed step, the
rehearsal runs that same procedure in the test namespace; otherwise its change-safety part
cannot be recorded.
- `status` shows which digest ran in each phase.
- Failure path: any phase may move to `failed` or `outcome-unknown`. Either one means stop new work
(operator scales API and processor to 0), archive with that state kept (ticket 15), and never
promote the batch to `business-verified`. The next attempt is a new batch with new identities; a
contaminated batch is not resumed.
Done when: lifecycle tests cover transitions, refusals and both failure states, a local end-to-end run uses two locally
built binaries (production commit → dev-mainline HEAD), and the guide is updated.

View File

@ -0,0 +1,31 @@
# Post-promotion production smoke, no customer-data writes by default
Type: task
Status: ready-for-agent
Blocked by: 12
Spec: Promotion (production smoke).
- Uses the untracked production verification accounts file. Before any write mode, confirm those
accounts belong to the frozen internal organization.
- An HTTP method does not decide whether a call writes. Task detail GET runs interrupted-slice
recovery and can finalize a pending submission (`internal/api/annotation_operations.go`, around
line 999); frame-range GET persists a timeline index on a cache miss
(`internal/api/annotation_timeline_index.go`, around line 102). Keep an endpoint list in the tool,
each entry classified by server-side effect: none / possible write / forbidden.
- Default mode promises no customer-data writes, not zero writes: logins (admin roles when accounts
are supplied), project and task lists, opening an existing internal-organization task, browser
playback (reusing ticket 12's check), frame range. Endpoints marked "possible write" are called
only on internal-organization objects, and the report lists every such call and the object it
touched. Calls outside the list, and any call on a non-internal organization, are refused.
- BOS path: the smoke asserts that frame range loads from the BOS source of an internal-organization
task (this exercises the BOS read and HEAD). Cache invalidation after a source ETag change would
need a production object to change, so it stays unverified.
- Write mode needs `--write --confirm <token>` and explicit internal-organization task IDs:
claim → slicing → submit → QC → acceptance. Refuse customer organization IDs, report how many
internal tasks remain, never create tasks.
- Report JSON bound to the release ID and the observed build commit.
Done when: tests against a mocked HTTP server cover unlisted-endpoint refusal, organization
refusal, the side-effect report (including cold frame-range cache and an interrupted task) and the
confirm token. This ticket does not access production.

View File

@ -0,0 +1,17 @@
# Archive cluster batches to local storage
Type: task
Status: ready-for-agent
Blocked by: 06
Spec: Batch flow step 5.
- Render an archive pod in the test namespace that runs `mongodump` of the batch database into
the batch directory on the node; render the pull commands into
`local-data/internal-test-lane/batches/<batch-id>/` with a SHA256 manifest.
- Dumps count toward the capacity budget in ticket 06, and archiving requires a passing preflight.
- `archive` evidence additionally requires the pulled dump, reports and config diff with hashes.
- Retention: latest 3 archived batches plus the current one. `cleanup-plan` also lists the
node-side directories for a separately approved cluster cleanup; the tool never deletes remotely.
Done when: tests cover manifest verification and retention selection.

View File

@ -128,3 +128,82 @@ Q1–Q3 均已采用推荐项。
- [01 environment](issues/01-environment.md), [02 seed](issues/02-seed.md), [03 lifecycle](issues/03-lifecycle.md) implemented and locally verified.
- [04 integration](issues/04-integration.md): core five-role scratch/snapshot workflows verified through final acceptance; standard delivery-package worker/graph remains uncovered. See [verification and two-axis review](verification.md).
- No customer assets or production runtime changes. Company deployment, real sample copy and production publication remain separately authorized operations.
## 2026-10-19 修订讨论
背景:客户已投产,用户不能再用客户账号测试。希望基于 `codex/dev-mainline-20261019` 在线上单开测试环境,用少量真实客户样本模拟全流程;通过后用脚本把同一版本发布到生产。本节不构成部署、复制或账号创建授权。
### 2026-10-19 Round 1(Q1–Q10 全部采用推荐项)
- Q1:修订 ADR-0004 而非另起方案。三处修订:首次部署即用真实客户样本(不再先走合成素材阶段);全流程覆盖标准交付打包;测试结果作为生产发布验收证据。其余既定决策不变。
- Q2:“推送到生产”定义为**版本晋升**:测试环境运行 dev-mainline 已提交 commit 构建的不可变镜像 digest,通过后经 `scripts/release.py` 将同一 digest 发布到生产;测试数据、账号、配置、存储卷不进生产;测试批次验收报告挂入 release 记录作为写入类证据;每次生产部署仍需当次授权。
- Q3:测试环境运行配置从生产只读导出非机密项,仅覆盖隔离字段(namespace、数据库、存储、入口)与容量字段(并发 1、资源上限);每批记录与生产的配置差异,差异外不一致的项不能作为晋升证据。
- Q4:纳入标准交付打包:测试 namespace 内独立工作流 worker(仅限测试 namespace 创建 Job)、算子镜像、生产打包工作流快照;不启用 `fanyu-high-capacity` 档,打包并发 1,产物只写测试存储、不进客户交付出口。
- Q5:新建测试专用 BOS bucket,用生产只读凭证复制选中 Capture;测试环境的 csi-bos 存储卷与 AK/SK 只对测试 bucket 有权限。不只读挂载生产 bucket,不改用 local PV。
- Q6:测试 namespace 内独立 Mongo 实例(不在生产 Mongo 上另建库);从生产只读导出选中项目配置(项目、工作流定义快照、三类字典)及样本,以新身份生成测试任务副本;不复制客户账号、会话、领取、在途处理和正式审查记录。
- Q7:每个在产项目 3 条(待切片、保留切片/动作快照、质检阶段快照),总数 ≤ 12;精确清单与体积只读盘点后由用户确认。
- Q8:两个测试供应商 `test-a`、`test-b`,各含供应商管理员/标注员/质检员,另加平台管理员和最终验收员,约 8 个账号(替代原 5 账号)。
- Q9:首版仅本机 127.0.0.1 端口转发,供用户本人使用;不建公网入口、不改生产 Ingress/DNS。
- Q10:“多个 pod”指一套测试环境内多个组件,一次一个候选版本;不并行多套环境。
### 集群只读盘点(2026-10-19)
- 三个 Ready 节点:`10.189.16.33`(56 CPU / ~220 GiB / 2 GPU,requests CPU 15%、limits 37%);`10.189.16.4`、`10.189.16.5`(各 8 CPU / ~31 GiB,后者 CPU limits 已超售至 185%)。均无污点。采样时实际 CPU 使用接近 0。
- 存储类仅 `csi-bos`(no-provisioner,静态 PV);无 IngressClass。生产工作区 `droboticflow-workspace-pv` 200Gi 为 csi-bosplugin。
- `droboticflow-internal-test` namespace 于 2026-09-27 已建,仅有 `internal-test-quota`(pods 3、requests 1 CPU/2Gi、limits 3 CPU/10.5Gi、存储 20Gi、PVC 8)、`internal-test-default-deny` 网络策略及 CCE 自动镜像拉取 Secret;无工作负载。配额不足以容纳 worker 与算子 Job。
- 生产 `droboticflow-fanyu` 应用组件:api ×4 与 annotation-processor ×2(同镜像,build commit 630a95c8 = 当前 dev-mainline HEAD,processor role=revision);annotation-validator ×2(另一镜像,963b530f,role=validation,2026-10-17 建);worker ×2(worker 镜像,963b530f);drosync-coordinator ×2;GPU 服务 headpose/wilor 无就绪副本。`release.py` 策略只覆盖 api 与 processor。
- **已有生产内测试组织**:所有生产应用组件的 `ANNOTATION_WORKSPACE_MAPPINGS` 含 `org-internal-0bd5aa2e…`,读取生产 bucket `internal-test/blf-20260927-…/assets`,写 `system/runs/run-internal-*`,交付至 `internal-test/…/deliveries/TEST-internal-BLF__…`。来源为 2026-09-27 release 记录 `test-readonly-fix-20260927`(API-only 配置变更,7 个测试账号覆盖 5 角色)。这是生产库内隔离组织方案,与 ADR-0004 的独立环境方案并存,待决定其去留。
### 2026-10-19 Round 2(Q11–Q15 采用推荐项;Q16 被用户否决)
- Q11:保留既有生产内测试组织 `org-internal-…`,冻结并收窄为**晋升后生产冒烟**:每次晋升后用 1 条样本验证生产自身配置、凭证与存储;不再新增样本或项目。
- Q12:候选版本为一个 dev-mainline commit 构建的镜像;测试环境各组件 1 副本;drosync-coordinator 不在测试环境和候选范围内。(组件范围已被 2026-10-20 范围收窄修订,见下文。)
- Q13:每批先以生产当前 digest 启动并导入样本、制造在途状态,再用**同一晋升脚本**指向测试环境升级到候选版本,演练迁移与在途恢复,产出 change-safety 证据。修订 ADR-0004“一批一版本”为“一批 = 生产版本 → 候选版本”。
- Q14:测试环境固定到 `10.189.16.4`(无生产应用 Pod),不给生产节点加污点、不改生产调度;按需上调测试配额;观察到争用再申请专用节点。
- Q15:测试 Mongo 为测试 namespace 内 Pod,主版本与生产一致(待只读核实),数据放 `.4` 本地盘专用目录的静态卷。
- Q16(否决):用户无法也不愿创建 BOS 子用户或测试 bucket。
### 2026-10-20 用户范围收窄
- 本轮只测试**标注 → 质检 → 最终验收**;Capture 接入及算子处理不在本问题范围。
- 不需要平台管理员的全局管理功能。
- 用户希望直接从 BOS 读取相应文件。代码核实:该链路并非只读——标注处理器在领取后自行生成切片媒体,送检与终审发布结果文件,均写入 `WORKSPACE_MOUNT_PATH` 下;算子 Job 与 worker 确实不参与。依据 [annotation-workspace-mappings.md](../../docs/annotation-workspace-mappings.md):“只读委托仅解决素材与帧范围加载,不支持生成切片产物”。
- 由此 Round 1 Q4(纳入标准交付打包)失效;Q5、Q8、Q12、Q15 中依赖测试 bucket、worker 或平台管理员的部分待 Round 3 重新确认。
- 事实:本地 Capture 样例(单 part、480p)约 235 MB;12 条源素材约数 GB。生产 worker(963b530f)与 api(630a95c8)之间 5 个提交未改 worker 代码。
### 2026-10-20 Round 3(Q16′–Q21 全部采用推荐项)
- Q16′(替代 Q16、Q5):以本机已有 BOS 凭证、按 `scripts/local/download-capture.sh` 方式一次性只读下载选中 Capture,校验 SHA256 后上传到 `.4` 本地盘测试工作区。测试环境为纯本地工作区模式:不挂载 BOS、不持有任何 BOS 凭证,切片与结果写本地盘。bosfs 读取与 BOS ETag 版本检查不在测试环境覆盖范围,由 Q11 晋升后生产冒烟覆盖。不采用只读挂载生产 bucket,不写入生产 bucket 的 `internal-test/` 前缀。
- Q17(撤回 Round 1 Q4):不含 worker、算子 Job 与标准交付打包。测试环境组件为 API ×1、标注处理器 ×1(role `all`,并发 1)、Mongo ×1。终审通过的任务停在待交付。
- Q18(修订 Q12):晋升只更新 API 镜像的三个组件 api、annotation-processor、annotation-validator;worker 保持现版本。候选改动 `internal/worker`、`cmd/worker` 时晋升脚本拒绝并要求另行验证。需扩展 `release.py` 策略纳入 validator。
- Q19(修订 Q8):5 个业务账号——test-a 标注员与质检员、test-b 标注员与质检员、1 名归属 test-a 且开通供应商最终验收授权的最终验收员;无平台管理员与供应商管理员。项目、供应商、成员与任务由 seed 命令建立。管理类登录由生产冒烟只读检查。
- Q20:样本选择、项目配置与字典导出、Mongo 版本确认均基于 `scripts/local/fetch-mongo-snapshot.sh` 拉取的最新备份,在本地独立容器完成,不查询线上库。选样清单交用户确认。
- Q21:测试批次报告记录候选镜像 digest 与配置差异摘要;`release.py evidence` 接收测试报告时校验 digest 一致且批次为 business-verified,否则拒绝。测试环境无法产出的证据(含交付打包)保持 unverified。
### 2026-10-20 Round 4(Q22–Q25 全部采用推荐项)
- Q22:逐步单独授权,前一步通过才询问下一步:① 拉取最新 Mongo 备份到新的本地容器(不动 27018)并选样;② 确认清单后从 BOS 只读下载样本;③ 从 dev-mainline 干净 worktree 构建候选 API 镜像并推送到公司镜像仓库;④ 测试 namespace 写入(配额、`.4` 专用目录、静态卷、Mongo、上传素材、seed、删除初始化 Pod),写前复核集群与 namespace UID;⑤ 以生产当前 digest 启动、制造在途状态、用晋升脚本升级到候选并跑全流程;⑥ 按 `release.py` 流程晋升生产并做生产冒烟。
- Q23:晋升后生产冒烟默认只读(全角色登录、列表、内部组织任务、视频播放与帧范围);仅当候选改动领取、切片、送检或审查写路径时,经当次批准在内部组织剩余任务上做写入冒烟;任务耗尽时报告,不自动补充,不使用客户任务。
- Q24:归档时 `mongodump`、结果报告与配置差异摘要写 `.4` 本地盘并拉回 `local-data/internal-test-lane/batches/<batch-id>/`;保留最近 3 批加当前批;旧批次本地产物预览后确认删除;集群侧清理另列清单与授权。
- Q25:复现特定客户问题时可追加该 Capture 为样本,按 ①→②→④ 流程逐次批准,活动总数 ≤ 12;新身份,不回写来源任务。
- 前沿已清空,待用户确认达成共同理解后修订 ADR-0004 与 spec 并拆分实施票。
### 2026-10-20 共同理解确认
- 用户确认四轮决定达成共同理解。已记录:[ADR-0004 2026-10-20 修订](../../docs/adr/0004-isolated-internal-test-lane.md)、[spec.md 当前修订](spec.md)、实施票 [06](issues/06-renderer-local-node.md)–[15](issues/15-batch-archive.md)(05 为主线既有 movi 冒烟票);[部署计划](deployment-plan.md) 标记为 superseded;术语“内部测试环境”“版本晋升”写入 [CONTEXT.md](../../CONTEXT.md)。
- 依赖:06、08、10、14 无前置;07←06;09←08;12←08;15←06;13←06/07/10/12;11←10/13。既有 05(movi 直连播放冒烟,暂缓)可在本环境部署后执行。
- 未开始实施,未提交;集群、BOS、备份拉取与生产晋升均未执行,按 spec 的 Run sequence 逐步授权。
### 2026-10-20 外部审阅采纳
用户采纳外部审阅的 7 点及补充项,第 6 点选方案 A。均已代码核实并写入票据与 spec:
- 备份恢复:现有 `fetch-mongo-snapshot.sh` 会在生产 API Pod 的 `/tmp` 分块暂存归档,并删除重建 27018 容器、`--drop` 恢复;默认备份目录固定为 `fanyu-v58-ca15d60d`。改为独立容器名与端口、拒绝覆盖、按字节区间直接读取不在生产 Pod 暂存、展示备份日期,过旧则停止([09](issues/09-backup-sample-export.md))。
- 生产冒烟:任务详情 GET 会恢复中断切片并可能完成待定送检,帧范围 GET 缓存缺失时写入索引;“只允许 GET”不等于只读。改为按服务端副作用分类端点,默认承诺不写客户数据,内部组织上的副作用写入列入报告([14](issues/14-production-smoke.md),新增依赖 12)。
- 本地盘容量:预算 + `df` 预检 + 剩余 25% 停止阈值 + 临时存储上限;文件系统配额视节点支持,否则记录为仅预检约束([06](issues/06-renderer-local-node.md)、[15](issues/15-batch-archive.md))。
- 升级起点:`baselineImage` 须等于发布计划记录的各 Deployment 旧镜像;当前 validator(`7699d455…`)与 api/processor(`1a1ead7a…`)起点不同,不一致者列为演练未覆盖([11](issues/11-release-evidence-binding.md)、[13](issues/13-upgrade-rehearsal.md))。
- 检查项→必需场景映射,缺项、失败、未验证一律拒绝([11](issues/11-release-evidence-binding.md)、[12](issues/12-cluster-http-verifier.md))。
- 发布证据:`annotation-delivery` 拆为 `annotation-results` 与 `delivery-packaging`;后者由打包测试记录,或 `release.py` 自行核对策略声明的交付输入路径未改动满足([10](issues/10-release-policy-scope.md))。ADR-0004 修订已补记。
- 播放与 ETag:浏览器(Playwright)首帧与播放推进;ETag 变化后的缓存失效保持未验证;05 号票第 1 项由 12 覆盖。
- 失败路径:`failed` / `outcome-unknown` 停机、保留状态归档、重开新批次([13](issues/13-upgrade-rehearsal.md))。
- 依赖更新:14←12。

View File

@ -1,4 +1,166 @@
# Internal test lane — implementation baseline
# Internal test lane
Status: ready-for-agent
Current revision: 2026-10-20
Code baseline: `codex/dev-mainline-20261019` @ `630a95c8`
Decisions: [map.md](map.md) (2026-10-19/20, Rounds 1–4) and the 2026-10-20 revision of
[ADR-0004](../../docs/adr/0004-isolated-internal-test-lane.md). This revision authorizes local
implementation and local verification of tickets 06–15 only. Pulling a backup, downloading
from BOS, pushing images, writing to the cluster, creating accounts there and promoting to
production are separately approved steps (see Run sequence).
## User-facing outcome
Before promoting a dev-mainline candidate, the developer runs one batch in the isolated test
environment. Up to 12 real customer samples, chosen from the newest production backup and
downloaded once read-only from BOS, are claimed, sliced, annotated, submitted, reviewed and
accepted by five test business accounts in two test suppliers. The batch first runs the
production image, then upgrades to the candidate with the same promotion script used for
production. A business-verified report bound to the candidate digest becomes release evidence;
that digest is then promoted to production's API-image components and checked by a production
smoke that is read-only by default.
## Scope
In: annotation → quality review → final acceptance, including slice generation, source-index
validation, save, submission, rework return, processor restart recovery, supplier claim
exclusivity and invisibility, browser video playback (first frame, playback advancing) and frame
range from the local workspace.
Out, and recorded as unverified wherever release evidence asks for it: workflow worker,
operator Jobs, standard delivery packaging, Capture ingestion, cache invalidation after a BOS
source ETag change, load and capacity, platform-admin and supplier-admin
management.
## Environment
- Namespace `droboticflow-internal-test` (exists; only `internal-test-quota` and
`internal-test-default-deny` today). Every pod is pinned to node `10.189.16.4`; production
nodes get no taints and production scheduling is unchanged.
- Components: API ×1; annotation processor ×1 with role `all`, concurrency 1; Mongo ×1 with
the production major version; an initializer pod during import only; a one-time
directory-preparation pod.
- Storage: static local PVs on a dedicated directory of node `.4` for Mongo and workspace,
`Retain`, node affinity, RWO. No csi-bos volume, BOS endpoint, storage connection,
workspace mapping or BOS credential exists in the namespace.
- Entry: ClusterIP only, reached through `kubectl port-forward` bound to `127.0.0.1`. The
session cookie may drop `Secure` only in this explicit private-forward mode.
- Runtime configuration is derived from a read-only snapshot of the production API and
processor Deployments. Only isolation fields (namespace, database, workspace/storage,
entry/cookie) and capacity fields (replicas, processor role, concurrency, resources) are
overridden; the database bootstrap mode follows production. Each batch records the diff;
any other difference disqualifies the batch as promotion evidence.
- Quota fits API, processor, Mongo and initializer with finite CPU, memory and storage limits.
- Disk: PV size and quota do not bound real writes to the node's local disk, which kubelet and
images share. A budget (samples, slice output, Mongo, dumps, 3 retained batches plus the current
one) is checked against a `df` preflight before seed, start, upgrade and archive; below 25% free
after the budget, work stops. Pods carry ephemeral-storage limits; a filesystem quota is used where
the node supports it, otherwise the batch record says the limit is preflight-only.
## Data and accounts
- Selection and export read a backup restored into a new local container with its own name and
loopback port; the restore refuses an existing container name, a busy port and 27017–27020, and
never drops into an existing database. The archive is read by byte ranges from the backup mount,
without staging inside the production pod. The chosen backup's name and date are shown; if the
newest one is older than the selection window, work stops, because a new production backup is a
separately approved operation. The live database is never queried.
- Per in-production project, up to 3 samples: scratch (pending annotation), annotation
snapshot (existing slices and actions), quality-review snapshot. At most 12 in total. The
user confirms the final list (Capture, project, template kind, size) before download.
- Media is downloaded read-only from production BOS on the developer machine, SHA256-verified
and bundled. Project config, the workflow-definition snapshot and the three dictionaries come
from the backup. Customer accounts, sessions, leases, in-flight work, formal review records
and notification targets are not copied.
- Suppliers `test-a` and `test-b`. Accounts: annotator and quality reviewer for each supplier,
plus one final acceptance reviewer in `test-a` with supplier final-acceptance authorization.
All work identities are new; source IDs are kept only for traceability. The seeded database
must start under the production bootstrap mode.
- Reproduction samples may be added later through the same export → download → import path,
approved each time, keeping the active total at 12 or fewer.
## Batch flow
1. Prepare a batch with `baselineImage` (production's current API digest, from a read-only
Deployment snapshot that also records each Deployment's image) and `candidateImage`.
2. Seed, start on `baselineImage`, and create in-flight state: claimed tasks, slicing pending,
submission pending.
3. Upgrade to `candidateImage` with `release.py` and the internal-test policy.
4. Run the HTTP verifier and the browser playback check on the candidate: five roles, supplier
isolation, normal and rework paths, processor restart recovery. Record `business-verified` with the candidate digest and
the config diff.
5. Archive: `mongodump`, reports and config diff pulled to
`local-data/internal-test-lane/batches/<batch-id>/`. Keep the latest 3 batches plus the
current one; older local outputs are removed only after preview and confirmation. Cluster-side
cleanup is a separate approved list.
Any phase can end in `failed` or `outcome-unknown`: stop new work, archive with that state kept,
never mark the batch `business-verified`, and start over with a new batch.
## Promotion
- `release.py` takes an explicit policy; without one it keeps today's Fanyu behaviour.
- The Fanyu policy covers `droboticflow-api`, `droboticflow-annotation-processor` and
`droboticflow-annotation-validator`. The worker is not touched; promotion is refused when the
candidate differs from the production worker's build commit under `internal/worker` or
`cmd/worker`. Shared-package compatibility with the old worker stays a change-safety review item.
- The Fanyu policy's `annotation-delivery` becomes `annotation-results` (satisfiable by a batch) and
`delivery-packaging` (a packaging test record, or `release.py`'s own check that the policy-declared
delivery-input paths are unchanged between the production API commit and the candidate).
- A test-lane report counts as evidence only if its candidate digest equals the release image
digest, the batch is `business-verified`, and every scenario the requested check needs is present
and `passed` (mapping in ticket 11). For the rehearsal part of `change-safety`, `baselineImage`
must also equal each covered Deployment's `before` image in the release plan; Deployments with a
different starting digest are listed as not covered. Evidence the batch cannot produce stays
unverified.
- After promotion, the production smoke runs in the frozen internal organization and by default
writes no customer data. Some GETs write (task detail recovery, frame-range index on a cache miss),
so endpoints are classified by server-side effect, possible writes are confined to
internal-organization objects and listed in the report. A write smoke runs only when the candidate changes claim, slicing, submission or
review write paths, approved each time, on remaining internal tasks only. Customer tasks are
never used and internal tasks are never replenished automatically.
## Run sequence (each step approved separately)
1. Pull the newest backup (name and date shown) into a new local container, select samples, user
confirms the list.
2. Download the confirmed samples from BOS, read-only.
3. Build the candidate API image from a clean dev-mainline worktree and push it to the company
registry.
4. Test-namespace writes: quota, node directory, PVs/PVCs, Mongo, upload, seed, delete the
initializer. Re-verify the cluster and namespace UID first.
5. Start on the production digest, create in-flight state, upgrade, verify, record evidence.
6. Promote with `release.py` prepare → check → plan → approval → deploy → verify, then run the
production smoke.
## Verification seams
- Public-boundary tests for the renderer, lifecycle CLI, seed planner and `release.py`
(simulated Kubernetes, as today).
- Local end-to-end on a fresh Mongo port and API port (not 27017/27018/3001): a v2 bundle from
synthetic media, two suppliers, five roles, baseline → candidate upgrade, and a verifier report
bound to the digest.
- Cluster behaviour (local PV binding on `.4`, port-forward under Cilium default-deny, quota)
stays unverified until run-sequence step 4 is approved.
## Tickets
[06](issues/06-renderer-local-node.md) ·
[07](issues/07-runtime-config-parity.md) ·
[08](issues/08-bundle-seed-v2.md) ·
[09](issues/09-backup-sample-export.md) ·
[10](issues/10-release-policy-scope.md) ·
[11](issues/11-release-evidence-binding.md) ·
[12](issues/12-cluster-http-verifier.md) ·
[13](issues/13-upgrade-rehearsal.md) ·
[14](issues/14-production-smoke.md) ·
[15](issues/15-batch-archive.md)
Existing [05](issues/05-movi-smoke-and-sl3-evidence.md) (movi direct-playback smoke, deferred by the user)
fits this environment once it is deployed: it has no movi service.
## 2026-09-27 baseline (implemented; superseded where it conflicts)
Status: implemented 2026-09-27 (local synthetic verification); 2026-10-12 drift catch-up re-verified (suites + five-role HTTP flow; movi browser smoke pending), see verification.md new section and final section
Baseline: 17e7de7e (implementation); drift-reviewed 2026-10-12 at 5a956624
@ -9,7 +171,7 @@ The current turn authorizes local implementation and synthetic validation only:
no customer database, BOS assets, production deployment or live credentials are accessed.
Actual cluster provisioning and the six real sample transfers remain separately approved operations.
## User-facing outcome
### User-facing outcome
An internal developer can prepare a small, independently runnable environment
using the same application/business UI, five isolated role accounts and a fixed
@ -17,7 +179,7 @@ six-source sample pack (three BLF-data, three SL-data). They can reproduce a
known state or start from slicing, archive results and explicitly prepare the
next batch/version without putting customer work into a test queue.
## Delivery components
### Delivery components
1. Offline environment renderer and configuration validation. Fixed test namespace,
distinct host, authenticated dedicated Mongo, per-batch database and PVCs,
@ -35,7 +197,7 @@ next batch/version without putting customer work into a test queue.
prepared, seeded and business-verified states; no claim of deployed/accepted
solely because manifests were rendered or records inserted.
## Explicit behavior
### Explicit behavior
- Test supplier `test`; platform admin plus supplier manager, annotator, quality
reviewer and acceptance reviewer. Customer identities are not cloned.
@ -66,7 +228,7 @@ next batch/version without putting customer work into a test queue.
- Same immutable app image is used for eventual promotion, with production runtime
settings retained. Candidate success does not authorize production release.
## Confirmed verification seams
### Confirmed verification seams
The user explicitly accepted these on 2026-09-27:
@ -79,7 +241,7 @@ typecheck/build, then full relevant suites at the end. Review against baseline
17e7de7e along standards and specification axes. Existing full-suite failures
must be reported separately; do not claim a wholly green repository.
## Current-turn acceptance
### Current-turn acceptance
Local commands, manifests, initialization, lifecycle and synthetic workflow evidence
are runnable and documented. No company access is required to demonstrate refusal
@ -87,7 +249,7 @@ checks and business behavior. Real sample inventory, exact company allocation,
external ingress/storage prerequisites, cluster deployment and capacity validation
remain explicitly unverified until separately authorized; do not fabricate them.
## 2026-10-12 漂移复核与追进度工单(用户批准落档)
### 2026-10-12 漂移复核与追进度工单(用户批准落档)
9-27 实现完整存在且未被后续演进破坏:Go 包测试 2026-10-12 重跑全绿(`go test ./internal/testlane/...`),种子命令 `cmd/internal-test-seed` 与 `scripts/internal_test_lane.py` 冒烟正常,五角色账号模型与当前 `internal/api/auth.go` 服务端校验完全一致;实现后零维护提交、零破坏。但距验证基线 `17e7de7e` 约 174 个提交已进入主线,差距如下:

View File

@ -0,0 +1,8 @@
# Add QC and acceptance reviewers to the production output dashboard
Goal: 生产总览 and 供应商产出 show QC and acceptance reviewer output next to annotators (same layout), with per-day efficiency for all three roles, durations in hours (one decimal), and rework re-reviews counted per part each time.
Agent: Claude Code (Opus 5.5), desktop session 2026-10-27
Base: codex/release-review-video-20261027 @ 2840d669 — the build live on Fanyu; origin/dev lacks the reviewer-statistics commits this builds on, so the PR also carries that branch's commits.
Scope: internal/workforcestats (output dataset/report), internal/store/annotation_output.go (projection fields only), internal/api/annotation_output.go (response + pagination), internal/outputsnapshot schema key, src/components/AnnotationOutputDashboard.tsx, src/features/annotation-output, src/services/api/annotation-output.ts, docs/workforce-statistics.zh-CN.md
Out of scope: reviewer self view (reviewers keep WorkforceSummaryPanel), CSV export columns, work-hour (工时) tracking, settlement/ranking formulas
Risk: additive API fields; snapshot schema key bumps to v2 so a fresh generation builds once after deploy; no Mongo writes, migrations or index changes

View File

@ -0,0 +1,13 @@
# Wang Ziyu rework feedback
## Notes
11:41 Enter feedback and 11:42 discard-boundary feedback are tracked separately, with a shared failing save path. A missing Capture title alone does not establish the cause. Repair base is deployed fab5be57; delivery classification v4 work is a separate branch.
## Decisions-so-far
- [Resolved repair](spec.md): route shortcuts through atomic slice application, preserve kept classifications, await confirmation and retain failed choices. Existing archive authority, CAS and undo contracts are reused.
## Fog
Local synthetic verification is not production acceptance. No customer records were changed, and this branch has not been deployed. Historical records that users explicitly invalidated require a separate evidence-based recovery decision; this fix does not infer their intended classifications.

View File

@ -0,0 +1,18 @@
# Repair Wang Ziyu's discard-boundary and Enter feedback
Status: resolved
Base: deployed fab5be57. Local changes only; no deployment, push or production writes.
Use the existing atomic slice application for boundary edits and conflict confirmation, with server-owned archives, version checks, authoritative recovery and one persisted undo step. Keep discarded portions archived and preserve kept-side action text/status/dictionary. Explicit invalidate remains a deliberate whole-action decision.
Enter/button confirmation must await persistence, reject duplicate writes, expose failures beside the confirmation controls, retain choices for retry and prevent cancellation while a result is pending. No outcome is inferred from the Capture title.
Verify through the actual AnnotationOperations page and real HTTP slice API with isolated synthetic data: left/right expansion, pending/described, mouse/keyboard confirmation, fresh read, error/retry, lost response, cancel, undo/redo and first-pass/rework. Keep existing split/merge, permission and autosave contracts.
## Answer
The deployed drag path had bypassed the atomic slice application and validated raw pieces before server archive semantics. Both failures are now handled by the existing authoritative application/reconciliation flow. Discarded portions are archived; kept portions preserve classification. Confirmation is asynchronous with an inline failure and a duplicate/cancel guard.
Regression command: `REWORK_BOUNDARY_BROWSER_TEST=1 go test ./internal/api -run '^TestReworkBoundaryBrowserReplica$' -count=1 -v` (local loopback and isolated MemoryStore; ffmpeg, Node dependencies and a Playwright browser required). Unit regression: `node --test tests/frontend/rework-boundary.test.mjs`.
Details: [map](map.md), [verification](../../docs/verification/rework-boundary-20261021.md).

View File

@ -55,13 +55,17 @@ These principles guide implementation choices. Specific business contracts and t
- Before assuming a remote target, run `git remote -v`.
- After making code or documentation changes for the user, stage only the files changed for that task.
- Commit with an English, emoji-prefixed commit message.
- **Push only when the user explicitly says push.** Default action after commit: stay local.
- When push is approved, push the user's personal feature branch only; never push `dev` or `poc` directly.
- Agents may push their own task branch (`<agent>/<slug>`) without asking. Never push `dev`, `poc` or another agent's branch directly.
- **Creating a pull request requires the user's explicit approval.** When a branch is ready for review, remind the user and wait; after approval, push the final state and create the PR together.
- Do not stage or commit unrelated user changes already present in the working tree.
- One task, one branch: start it from the latest `origin/dev` in its own worktree under `local-data/worktrees/<slug>`, named `<agent>/<slug>`. Never switch branches in the main checkout. Build on another unmerged branch only when the user asks, and record that branch as the card's `Base:`.
- The branch's first commit adds the task card `.scratch/<slug>/work.md`. Follow `docs/agents/branch-workflow.md` for the card, the end-of-task report, and cleanup.
- When a task ends, push the task branch, run `python3 scripts/agent_status.py`, and report this branch's state to the user. If it is ready for review, remind them that a PR is waiting for their approval.
## Release Workflow
- For release preparation, deployment, rollback, or resuming release work, read `docs/release/README.md` first and run `python3 scripts/release.py status <release-id>` when a record exists. Follow the recorded state and evidence; a successful rollout alone is not business acceptance.
- Deploy only commits contained in `origin/dev`. When the user approves deploying an unmerged branch for an urgent fix, name the branch in the approval request and in `--purpose`, and open its PR to `dev` the same day.
- Fanyu image releases use `scripts/release.py`. Configuration changes or migrations outside its supported scope require a separately reviewed plan and the corresponding user authorization.
## Credentials

View File

@ -2,6 +2,14 @@
以下词汇描述人工标注与审核中的业务对象。
**分类标签**:管理员为字典词条明确维护的 tags 集合,用于按已确认类别归组;它不同于词条分类 category、别名 aliases 或动作描述文本。确认后的词条身份、版本和标签共同构成当次分类快照,字典后续更新不改写该快照。
**最终分类结果**:最终验收通过的固定版本中,视频各区间的基础分类、动作细分、显式无效及舍弃判断。未写动作描述不撤销基础分类;审核批注和返工过程不属于分类结果。
**动作有效标签**:动作独立选择描述字典词条时,使用当次选择冻结的标签;未独立选择时,继承所属切片冻结的分类标签。是否填写自由描述不决定继承,无效判断与标签分别保留。
**原始区间定位**:以原始采集及 part 的文件身份和硬件时间戳定位人工确认区间;切片内相对时间与原始 part 相对时间是不同坐标,不能混用。下游按定位和分类处理原始数据,不依赖平台内部审核文件。
## 标注对象
**Capture(源采集)**:一次源采集所关联的双目视频、时间戳与传感器数据,是本流程组织人工处理和审核的对象。
@ -20,6 +28,12 @@ _Avoid_:操作次数(同一份数据可以重复操作,但期间完成份
**返工重提次数**:任务被退回后再次成功送检的次数,与收到正式打回决定的次数分别统计。内部数据版本的增加不代表发生返工。
**审查次数**:质检员或验收员在所选期间作出的正式通过与退回决定,按决定所涉 part 计数;同一 part 返工后再次审查重复计入,其中再次审查另计为复审。终审一次决定涉及多个 part 时按 part 分别计入。它与标注员产出分开,不相加。
_Avoid_:审查份数(同一 part 可以计入多次)
**人天**:一名人员在一个本地自然日至少有一次本角色正式工作即计 1 人天:标注员为成功送检(含返工重提),质检员、验收员为正式审查决定。人均日产出为期间视频小时除以人天,不是实际工时。
_Avoid_:工时、出勤时长
**问题对象数**:每轮正式打回中具有意见的切片、动作及时间点对象数,同一对象的旧版多条意见只计一个对象,原文保留。停用前的整体意见不再计入。它不等于已独立证实的错误数量。
**问题覆盖视频时长**:正式打回意见所定位的视频区间,在每轮同一来源内去除重叠后的覆盖时长,跨轮累计。时间点意见只定位单一时间点,不计入覆盖时长;缺少可靠区间的记录不推定为整段视频,也不等于实际工时或精确缺陷持续时间。
@ -96,6 +110,8 @@ _Avoid_:公共可领取任务池(其中的任务尚未归属当前标注员
## 审查
**审查追溯**:已授权任务的只读事件投影。任务来源与流转记录合并为右栏顶部的一张可折叠卡片,标题行显示当前版本与打回次数,默认展示最近两条流转;普通任务卡可批量附加人员、供应商和退回统计。缺失值用 `-`,已知零次保留 `0`;版本来自修订字段,退回次数来自正式退回事件,不使用轮数或版本相减。历史没有创建起点时,首次提交和总退回次数保留未知。最终验收继续遵守供应商匿名规则。辅助统计失败不阻断任务列表,不回填或改写历史数据。
**审查**:质检审核与最终验收的统称,操作者核对已有结果、记录问题并作出通过或退回决定,不直接修改原标注结果。
**质检审核**:对标注员提交的切片、动作结果及相关证据进行检查的环节。不通过的结果回到标注环节返工。
@ -138,6 +154,8 @@ _Avoid_:删除意见(确认解决不删除任何历史记录)
## 账号与项目授权
**已删除账号身份**:保留原用户 ID、姓名、手机号和历史归属的不可重新启用记录。删除完成后手机号可用于创建新身份,新账号不继承旧账号的任务或历史;仅停用及删除未完成的账号继续占用手机号。
**账号角色**:一个账号只有一个角色。管理员独立;供应商管理员、标注员、质检审核员和最终验收员均归属一个供应商。
**项目成员**:账号参与项目的授权关系,生产职责继承账号角色。一个项目可以同时包含多个供应商,各供应商可以在该项目内处理其归属的任务;不能把项目成员角色当作额外账号权限。
@ -165,6 +183,12 @@ _Avoid_:客户公共池(内部测试不参与客户任务的竞争领取)
**测试任务副本**:以经批准的客户样本为来源、拥有独立身份和处理历史的内部测试任务;副本的修改不回写来源任务。
_Avoid_:供应商任务副本(不代表给每家生产供应商复制客户任务)
**内部测试环境**:与客户生产隔离、一次运行一个候选版本的完整平台,用经批准的少量客户样本模拟客户生产处理流程;其中的账号、任务、审查和交付不计入客户生产、统计或交付出口。
_Avoid_:测试 pod(一套环境包含多个组件)、生产内测试项目(与客户共用业务数据,不构成环境隔离)
**版本晋升**:把已在内部测试环境验收的同一候选版本发布到客户生产;只带走版本本身,不带走测试数据、账号或配置,每次发布仍需单独授权。测试通过不等于获准发布。
_Avoid_:推送 pod、迁移测试环境
**撤销供应商项目授权**:停止供应商领取该项目的新任务,但保留其既有任务的标注、质检与返工处理资格,直至存量完成;不等于停用账号或释放已领取任务。
**生产统计异常**:仅指当前正式处理中的执行故障,包括运行失败、源采集处理失败、切片或送检处理作业失败、交付打包失败。供应商归属不同、部分未领取、归属待核对、审核退回、业务升级处理及正常等待不单独计为异常;这些状态保留在明细,不增加红色异常数量。

View File

@ -0,0 +1,18 @@
# The base must be the reviewed independent package runtime, pinned by digest.
# The runner and its dependencies stay on that base; only package source changes.
ARG PACKAGE_BASE_IMAGE
FROM ${PACKAGE_BASE_IMAGE}
ARG DROBOTICFLOW_BUILD_COMMIT=unknown
ARG DROBOTICFLOW_BUILD_TIME=unknown
ENV DROBOTICFLOW_IMAGE_BUILD_COMMIT=${DROBOTICFLOW_BUILD_COMMIT}
ENV DROBOTICFLOW_IMAGE_BUILD_TIME=${DROBOTICFLOW_BUILD_TIME}
ENV DROBOTICFLOW_OPERATOR_ID=operator-builtin-package
ENV DROBOTICFLOW_OPERATOR_SLUG=package
ENV DROBOTICFLOW_OPERATOR_HANDLER=builtin_result_package
ENV DROBOTICFLOW_OPERATOR_SOURCE=/app/backend/operators/builtin/package/operator.py
LABEL org.opencontainers.image.revision=${DROBOTICFLOW_BUILD_COMMIT}
LABEL org.opencontainers.image.created=${DROBOTICFLOW_BUILD_TIME}
COPY --chown=10001:10001 operator-repos/builtin/package/src/operator.py /app/backend/operators/builtin/package/operator.py
COPY --chown=10001:10001 operator-repos/builtin/package/src/operator.py /app/operator-repos/builtin/package/src/operator.py
USER 10001:10001
ENTRYPOINT ["python", "/app/backend/operator-runner/main.py"]

View File

@ -5,6 +5,7 @@ import hashlib
import bisect
import csv
import json
import math
import pathlib
import re
import shutil
@ -702,9 +703,31 @@ def _part_index_capture_record(summary_payload: dict[str, Any], revision_manifes
return {}
def _part_index_discard_evidence(evidence: dict[str, Any], *, classification_v4: bool = False) -> dict[str, Any]:
# New workflow metadata must not change the bytes of a legacy package.
result = {
"reason": str(evidence.get("reason") or ""),
"reasonLabelId": str(evidence.get("reasonLabelId") or ""),
"reasonName": str(evidence.get("reasonName") or ""),
"note": str(evidence.get("note") or ""),
"discardedByUserId": str(evidence.get("discardedByUserId") or ""),
"discardedAt": str(evidence.get("discardedAt") or ""),
}
if classification_v4:
result.update(
note=str(evidence.get("note") or evidence.get("reasonNote") or ""),
reasonVersion=evidence.get("reasonVersion"),
dictionary=evidence.get("dictionary"),
discardedAt=str(evidence.get("discardedAt") or evidence.get("at") or ""),
)
return result
def _part_index_discarded_slice(
row: dict[str, Any],
source_root: pathlib.Path,
*,
classification_v4: bool = False,
) -> dict[str, Any]:
"""Classify a discarded interval without shipping its media or old text.
@ -748,14 +771,7 @@ def _part_index_discarded_slice(
},
"classification": {
"deliveryDisposition": declared,
"discardEvidence": {
"reason": str(evidence.get("reason") or ""),
"reasonLabelId": str(evidence.get("reasonLabelId") or ""),
"reasonName": str(evidence.get("reasonName") or ""),
"note": str(evidence.get("note") or ""),
"discardedByUserId": str(evidence.get("discardedByUserId") or ""),
"discardedAt": str(evidence.get("discardedAt") or ""),
},
"discardEvidence": _part_index_discard_evidence(evidence, classification_v4=classification_v4),
},
"segments": [],
}
@ -843,13 +859,189 @@ def _description_status(actions: list[dict[str, Any]]) -> str:
return "empty_confirmed"
def _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload):
"""Use the established immutable directory publisher for the two JSON files.
def _v4_tags(selection: Any, kind: str) -> list[str]:
if not isinstance(selection, dict) or selection.get("kind") != kind or not str(selection.get("id") or "").strip() or type(selection.get("version")) is not int or selection["version"] < 1 or not str(selection.get("name") or "").strip():
raise RuntimeError(f"v4 missing frozen {kind} dictionary; confirm historical classification before delivery")
tags = selection.get("tags")
if not isinstance(tags, list) or not tags or any(not isinstance(tag, str) or not tag.strip() or tag != tag.strip() for tag in tags) or len(set(tags)) != len(tags):
raise RuntimeError(f"v4 missing frozen {kind} tags; confirm classification before delivery")
return list(tags)
def _v4_original_sources(manifest, capture_id, part_id, capture_prefix):
source = manifest.get("originalSource")
if not isinstance(source, dict) or source.get("provider") != "BOS" or not str(source.get("bucket") or "").strip():
raise RuntimeError("v4 requires frozen original BOS source selection; workspace paths cannot identify original data")
prefix = str(source.get("prefix") or "")
part_prefix = str(source.get("partPrefix") or "")
if prefix != capture_prefix or part_prefix != prefix + "/" + part_id or source.get("captureUnit") != capture_id + "/" + part_id:
raise RuntimeError("v4 original source identity disagrees with accepted capture/part")
if not prefix or str(pathlib.PurePosixPath(prefix)) != prefix or pathlib.PurePosixPath(prefix).is_absolute() or ".." in pathlib.PurePosixPath(prefix).parts or any(c in prefix for c in ("\\", "\x00")):
raise RuntimeError("unsafe v4 original source prefix")
keys = source.get("objectKeys")
if not isinstance(keys, list) or not keys or any(not isinstance(key, str) for key in keys) or len(set(keys)) != len(keys):
raise RuntimeError("v4 requires an unambiguous original object selection")
files = {}
cap, part = re.escape(capture_id), re.escape(part_id)
patterns = {
"leftVideo": rf"{part}/{cap}_camera_left_{part}(?:_[a-z0-9_]+)?\.(?:mp4|mov|mkv)",
"rightVideo": rf"{part}/{cap}_camera_right_{part}(?:_[a-z0-9_]+)?\.(?:mp4|mov|mkv)",
"leftPts": rf"{part}/{cap}_camera_left_{part}(?:_[a-z0-9_]+)?_pts\.csv",
"rightPts": rf"{part}/{cap}_camera_right_{part}(?:_[a-z0-9_]+)?_pts\.csv",
"imu": rf"{part}/{cap}_imu_{part}\.csv",
"cameraCalibration": rf"{cap}_calibration_camera\.yaml",
"imuCalibration": rf"{cap}_calibration_imu\.yaml",
"metadata": rf"meta_{cap}\.json",
}
for key in keys:
if not key.startswith(prefix + "/") or str(pathlib.PurePosixPath(key)) != key or ".." in pathlib.PurePosixPath(key).parts or any(c in key for c in ("\\", "\x00")):
raise RuntimeError("unsafe v4 original source object")
relative = key[len(prefix) + 1:]
if "/" in relative and not relative.startswith(part_id + "/"):
raise RuntimeError("v4 original selection contains another part")
for role, pattern in patterns.items():
if re.fullmatch(pattern, relative, re.IGNORECASE):
if role in files: raise RuntimeError(f"ambiguous v4 original {role} selection")
files[role] = {"path": relative}
missing = sorted(set(patterns) - files.keys())
if missing: raise RuntimeError("v4 missing original files: " + ", ".join(missing))
return {"provider": "BOS", "bucket": source["bucket"], "prefix": prefix, "partPrefix": part_prefix,
"snapshotFingerprint": str(source.get("snapshotFingerprint") or ""), "files": files}
def _v4_number(value, label):
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value):
raise RuntimeError(f"v4 missing or invalid {label}")
return float(value)
def _v4_timestamp(value, label):
if type(value) is not int or value <= 0: raise RuntimeError(f"v4 missing or invalid {label}")
return value
def _v4_consumer_payload(payload, manifest, source_root):
"""Project immutable accepted facts into a self-contained consumer contract.
No dictionary lookup, source media reads, transcoding or BOS writes here.
Hardware PTS are authoritative; media seconds are explicitly named and never
used to manufacture missing hardware timestamps.
"""
if not isinstance(manifest, dict): raise RuntimeError("v4 requires accepted source manifest")
original = _v4_original_sources(manifest, payload["capture"]["captureId"], payload["partId"], payload["capture"]["prefix"])
payload["originalSource"] = original
payload["consumerContract"] = "droboticflow.classification-consumer/v1"
payload["timeBases"] = {"taskSplitSeconds": "original_left_video", "segmentSeconds": "slice_left_video",
"segmentPartSeconds": "original_left_video", "timestampUs": "original_hardware_pts",
"interval": "half-open", "frameRanges": "zero-based-half-open"}
payload["pathBases"] = {"sources": "original_capture", "sourceMetadata": "original_capture", "calibration": "original_capture"}
# Workspace sources are private transport references, not the original files.
payload["sources"] = {kind: original["files"][kind] for kind in ("leftVideo", "rightVideo", "leftPts", "rightPts", "imu")}
payload["sourceMetadata"]["path"] = original["files"]["metadata"]["path"]
for item in payload["calibration"]:
# Bind by actual calibration kind, not input ordering.
kind = "imuCalibration" if "imu" in item["name"].lower() else "cameraCalibration" if "camera" in item["name"].lower() else ""
if not kind: raise RuntimeError("v4 unknown calibration kind")
item["path"] = original["files"][kind]["path"]
rows = {str(row["sliceId"]): row for row in manifest["items"]}
if len(rows) != len(manifest["items"]): raise RuntimeError("v4 duplicate slice identities")
for item in payload["slices"] + payload["discardedSlices"]:
split = item["taskSplit"]
start, end = (_v4_number(split[k], k) for k in ("startSecond", "endSecond"))
first, last = (_v4_timestamp(split[k], k) for k in ("startTimestampUs", "endTimestampUs"))
if start < 0 or end <= start or last <= first: raise RuntimeError("v4 invalid accepted slice range")
row = rows[item["sliceId"]]
sync = json.loads((source_root / row["directory"] / "synchronized-slice.json").read_text())
seconds, stamps = sync.get("boundarySeconds"), sync.get("boundaryTimestampUs")
# A continuous annotation is not proof of its relation to media seconds.
# Require the accepted revision's mapping for kept and discarded ranges;
# never infer hardware PTS from duration, cadence, or action endpoints.
if not isinstance(seconds, list) or not isinstance(stamps, list) or len(seconds) < 2 or len(seconds) != len(stamps):
raise RuntimeError("v4 missing or invalid hardware boundary mapping")
seconds = [_v4_number(value, "boundary") for value in seconds]
stamps = [_v4_timestamp(value, "boundary timestamp") for value in stamps]
if (abs(seconds[0]) > 1e-7 or abs(seconds[-1] - (end-start)) > 1e-7 or
stamps[0] != first or stamps[-1] != last or
any(a >= b for a, b in zip(seconds, seconds[1:])) or
any(a >= b for a, b in zip(stamps, stamps[1:]))):
raise RuntimeError("v4 invalid hardware boundary mapping")
window = sync.get("window")
if (not isinstance(window, dict) or window.get("semantics") != "half-open" or
_v4_timestamp(window.get("startTimestampUs"), "synchronization window start") != first or
_v4_timestamp(window.get("endTimestampUs"), "synchronization window end") != last):
raise RuntimeError("v4 boundary mapping disagrees with frozen synchronization window")
if item["classification"]["deliveryDisposition"] == "discard":
item["sliceName"] = ""
evidence = item["classification"]["discardEvidence"]
selected = evidence.get("dictionary")
if selected is not None:
if (not isinstance(selected, dict) or selected.get("kind") != "discard_reason" or
not selected.get("id") or not selected.get("name") or
type(selected.get("version")) is not int or selected["version"] < 1):
raise RuntimeError("v4 invalid frozen discard dictionary")
for alias, field in (("reasonLabelId", "id"), ("reasonName", "name"), ("reasonVersion", "version")):
if evidence.get(alias) not in (None, "", 0, selected[field]):
raise RuntimeError("v4 discard reason disagrees with frozen dictionary")
evidence[alias] = selected[field]
continue
parent_tags = _v4_tags(item["classification"].get("dictionary"), "slice")
item["classification"]["tags"] = parent_tags
def timestamp(second):
hits = [stamps[i] for i, value in enumerate(seconds) if abs(value - second) < 1e-7]
if len(hits) != 1: raise RuntimeError("v4 action boundary has no original PTS mapping")
return hits[0]
actions = []
for action in item["segments"]:
a = _v4_number(action.get("clipStartMs"), "action clipStartMs") / 1000
b = _v4_number(action.get("clipEndMs"), "action clipEndMs") / 1000
if a < -1e-7 or b <= a or b > end-start+1e-7: raise RuntimeError("v4 action escapes its parent slice")
ta, tb = (_v4_timestamp(action.get(k), k) for k in ("startTimestampUs", "endTimestampUs"))
if not first <= ta < tb <= last: raise RuntimeError("v4 action timestamps escape its parent slice")
if timestamp(a) != ta or timestamp(b) != tb: raise RuntimeError("v4 action seconds disagree with original PTS")
action.update(startSecond=a, endSecond=b, partStartSecond=start+a, partEndSecond=start+b)
own = action.get("dictionary")
action["tags"] = _v4_tags(own, "action") if own is not None else list(parent_tags)
action["tagSource"] = "action" if own is not None else "slice"
actions.append(action)
actions.sort(key=lambda a: a["startSecond"])
complete, previous = [], 0.0
def gap(a,b):
ta,tb=timestamp(a),timestamp(b)
return {"id":f"{item['sliceId']}:gap:{ta}:{tb}", "description":"", "isInvalid":False,
"contentKind":"gap_placeholder", "startSecond":a,"endSecond":b,
"clipStartMs":a*1000,"clipEndMs":b*1000,"partStartSecond":start+a,"partEndSecond":start+b,
"startTimestampUs":ta,"endTimestampUs":tb,"tags":list(parent_tags),"tagSource":"slice"}
for action in actions:
if action["startSecond"] < previous-1e-7: raise RuntimeError("v4 overlapping actions")
if action["startSecond"] > previous+1e-7: complete.append(gap(previous,action["startSecond"]))
complete.append(action);previous=action["endSecond"]
if previous < end-start-1e-7: complete.append(gap(previous,end-start))
if not complete: raise RuntimeError("v4 kept slice has no classified action coverage")
previous_stamp = first
for action in complete:
if action["startTimestampUs"] != previous_stamp or action["endTimestampUs"] <= previous_stamp:
raise RuntimeError("v4 action PTS coverage is discontinuous")
previous_stamp = action["endTimestampUs"]
if previous_stamp != last: raise RuntimeError("v4 action PTS coverage is incomplete")
item["segments"] = complete
counts = {kind:sum(a["contentKind"]==kind for a in complete) for kind in ("annotated","gap_placeholder","invalid_marked")}
item["classification"]["contentCounts"] = counts
item["classification"]["descriptionStatus"] = _description_status(complete)
if not counts["annotated"] and counts["invalid_marked"] and counts["gap_placeholder"]: item["classification"]["descriptionStatus"]="invalid_and_blank"
item["synchronization"]["rightBoundarySeconds"] = sync.get("rightBoundarySeconds", [])
def _emit_part_index_directory(run_id, node_id, output, node_data, helpers, payload, internal_audit=None, revision_manifest=None, source_root=None):
"""Use the established immutable publisher for versioned customer JSON.
V3 retains its historical sibling review file; v4 stores that audit in a
separate internal artifact before publishing the customer directory.
Legacy single-file publishers deliberately accept only v1/v2. A directory
artifact preserves their ownership/conditional-publication protocol without
teaching an old worker to misinterpret a new single-file schema.
"""
v4 = internal_audit is not None
capture = dict(payload["capture"])
prefix = pathlib.PurePosixPath(str(capture.get("prefix") or ""))
part_id = prefix.name
@ -860,7 +1052,7 @@ def _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload):
if prefix.is_absolute() or ".." in prefix.parts or len(prefix.parts) < 3:
raise RuntimeError("unsafe v3 capture identity prefix")
capture.update(captureId=capture_id, prefix=str(prefix.parent))
payload.update(schemaVersion="droboticflow.delivery.part-index/v3", capture=capture, partId=part_id)
payload.update(schemaVersion="droboticflow.delivery.part-index/v4" if v4 else "droboticflow.delivery.part-index/v3", capture=capture, partId=part_id)
payload["pathBases"] = {"sourceMetadata": "original_capture", "calibration": "original_capture", "sources": "workspace"}
payload["sourceMetadata"] = {"path": f"meta_{capture_id}.json"}
history = {
@ -869,7 +1061,8 @@ def _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload):
"qualityReviews": payload["acceptance"].pop("qualityReviews", []),
}
payload["reviewHistory"] = {"path": "review-history.json", "pathBase": "part_package"}
payload["acceptance"]["finalAcceptance"].pop("confirmedByUserId", None)
if not v4:
payload["acceptance"]["finalAcceptance"].pop("confirmedByUserId", None)
for item in payload["calibration"]:
if pathlib.PurePosixPath(item["name"]).name != item["name"] or "\\" in item["name"]:
raise RuntimeError("unsafe calibration filename")
@ -889,6 +1082,54 @@ def _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload):
classification["contentCounts"] = counts
if not counts["described"] and counts["invalid_marked"] and counts["gap_placeholder"]:
classification["descriptionStatus"] = "invalid_and_blank"
if v4:
if not isinstance(internal_audit.get("trace"), list):
raise RuntimeError("DELIVERY_JSON_V4 requires complete internal trace input")
if _output(node_data, helpers, "deliveryAudit") is None:
raise RuntimeError("DELIVERY_JSON_V4 requires an internal deliveryAudit output")
audit = {
"schemaVersion": "droboticflow.delivery.review-history/v2",
"captureId": capture_id, "partId": part_id, "revision": payload["revision"],
"acceptance": payload.pop("acceptance"), "lineage": payload.pop("lineage"),
"qualityReviews": internal_audit.get("qualityReviews", []),
"trace": internal_audit.get("trace", []),
"acceptedMaterializations": internal_audit.get("acceptedMaterializations", []),
}
audit["acceptance"]["finalAcceptance"] = internal_audit.get("finalAcceptance", {})
payload.pop("reviewHistory", None)
payload["capture"] = {"captureId": capture_id, "prefix": str(prefix.parent)}
for item in payload["slices"] + payload["discardedSlices"]:
classification = item["classification"]
# Blank descriptions retain the independently selected base class.
if classification.get("dictionary") is None:
classification.pop("dictionary", None)
evidence = classification.get("discardEvidence", {})
evidence.pop("discardedAt", None)
if evidence.get("dictionary") is None:
evidence.pop("dictionary", None)
counts = classification.get("contentCounts")
if counts is not None:
counts["annotated"] = counts.pop("described")
for action in item["segments"]:
if action["contentKind"] == "described": action["contentKind"] = "annotated"
for key in ("source", "confidence", "phrases", "labelId"):
action.pop(key, None)
if action.get("dictionary") is None: action.pop("dictionary", None)
audit["materializedSources"] = payload.get("sources", {})
_v4_consumer_payload(payload, revision_manifest, source_root)
body = (json.dumps(payload, ensure_ascii=False, sort_keys=True, indent=2) + "\n").encode("utf8")
audit["partSha256"] = hashlib.sha256(body).hexdigest()
if len(json.dumps(audit, ensure_ascii=False, indent=2).encode("utf8")) > 8 * 1024 * 1024:
raise RuntimeError("internal delivery audit exceeds 8 MiB; refusing incomplete export")
staging = pathlib.Path(tempfile.mkdtemp(prefix="part-index-v4-", dir=output))
(staging / "part.json").write_bytes(body)
# A sibling artifact stays in the internal run, never in the customer tree.
audit_artifact = helpers["write_json_artifact"](run_id, node_id, output, "review-history.json", audit, output=_output(node_data, helpers, "deliveryAudit"))
directory = helpers["build_virtual_artifact"](
run_id, node_id, "delivery.directory.index", str(staging), "file", 0,
data={"schemaVersion": "droboticflow.directory-index/v1", "presentation": "collection", "datasetType": "directory", "dataSource": "FILESYSTEM", "rootPath": str(staging), "sliceCount": len(payload["slices"]), "classificationOnly": True},
output=_output(node_data, helpers, "deliveryDirectory"))
return [directory, audit_artifact]
staging = pathlib.Path(tempfile.mkdtemp(prefix="part-index-v3-", dir=output))
for name, document in (("part.json", payload), ("review-history.json", history)):
(staging / name).write_text(json.dumps(document, ensure_ascii=False, sort_keys=True, indent=2) + "\n", encoding="utf8")
@ -905,6 +1146,8 @@ def _part_index_slice(
source_slice: pathlib.Path,
calibration: list[dict[str, Any]],
pose_policy: dict[str, Any],
*,
classification_v4: bool = False,
) -> dict[str, Any]:
declared = str(row.get("deliveryDisposition") or "")
slice_id = str(row.get("sliceId") or f"slice-{index + 1:06d}")
@ -944,6 +1187,7 @@ def _part_index_slice(
"clipStartMs": _optional_float(action.get("clipStartMs")),
"clipEndMs": _optional_float(action.get("clipEndMs")),
"labelId": str(action.get("descriptionTemplateId") or ""),
**({"dictionary": action.get("dictionary")} if classification_v4 else {}),
"phrases": [str(item) for item in action.get("phrases", [])] if isinstance(action.get("phrases"), list) else [],
"source": str(action.get("source") or ""),
"confidence": _optional_float(action.get("confidence")),
@ -976,6 +1220,7 @@ def _part_index_slice(
"deliveryDisposition": declared,
"taskLabel": str(synchronization.get("taskLabel") or ""),
"taskLabelId": str(synchronization.get("taskLabelId") or ""),
**({"dictionary": synchronization.get("dictionary")} if classification_v4 else {}),
"descriptionStatus": _description_status(actions),
},
"segments": actions,
@ -995,14 +1240,7 @@ def _part_index_slice(
raise RuntimeError(f"discarded slice {slice_id} has no reason evidence")
# Discard classification is exported deliberately: downstream packaging
# must know why an interval is excluded.
entry["classification"]["discardEvidence"] = {
"reason": str(evidence.get("reason") or ""),
"reasonLabelId": str(evidence.get("reasonLabelId") or ""),
"reasonName": str(evidence.get("reasonName") or ""),
"note": str(evidence.get("note") or ""),
"discardedByUserId": str(evidence.get("discardedByUserId") or ""),
"discardedAt": str(evidence.get("discardedAt") or ""),
}
entry["classification"]["discardEvidence"] = _part_index_discard_evidence(evidence, classification_v4=classification_v4)
# Pose frames deliberately stay out of the index: the customer contract for
# this mode is the split information, and embedded per-frame pose would
# make a multi-megabyte part.json out of a kilobyte index. The per-slice
@ -1027,6 +1265,8 @@ def _build_part_index(
capture_record: dict[str, Any],
lineage: dict[str, Any],
discarded_slices: list[dict[str, Any]],
*,
classification_v4: bool = False,
) -> tuple[dict[str, Any], bytes]:
slices: list[dict[str, Any]] = []
for index, row in enumerate(kept_slices):
@ -1038,7 +1278,7 @@ def _build_part_index(
source_slice = source_root / pathlib.Path(*relative.parts)
if not source_slice.is_dir():
raise RuntimeError("accepted slice manifest contains an unavailable slice directory")
slices.append(_part_index_slice(row, index, source_slice, calibration, pose_policy))
slices.append(_part_index_slice(row, index, source_slice, calibration, pose_policy, classification_v4=classification_v4))
quality_reviews = summary_payload.get("qualityReviews") if isinstance(summary_payload.get("qualityReviews"), list) else []
exported_reviews = [
{key: review[key] for key in _QUALITY_REVIEW_EXPORT_FIELDS if key in review}
@ -1061,7 +1301,7 @@ def _build_part_index(
"coverage": coverage,
"slices": slices,
"discardedSlices": [
_part_index_discarded_slice(row, source_root)
_part_index_discarded_slice(row, source_root, classification_v4=classification_v4)
for row in discarded_slices
if isinstance(row, dict)
],
@ -1651,6 +1891,12 @@ def _run_accepted_slice_directory(
raise RuntimeError("acceptedSliceDirectory does not prove gap-free full-video coverage")
summary_artifact = next((item for item in inputs if _port(item) == "acceptanceSummary"), None)
summary_payload = summary_artifact.get("data") if isinstance((summary_artifact or {}).get("data"), dict) else {}
if summary_artifact is not None and not isinstance(summary_payload.get("finalAcceptance"), dict):
summary_path = _materialize_input_file(summary_artifact, output / "_acceptance", helpers)
if summary_path.stat().st_size > 8 * 1024 * 1024:
raise RuntimeError("acceptance summary exceeds 8 MiB")
summary_payload = json.loads(summary_path.read_text(encoding="utf8"))
if not isinstance(summary_payload, dict): raise RuntimeError("invalid acceptance summary reference")
final_acceptance = summary_payload.get("finalAcceptance") if isinstance(summary_payload.get("finalAcceptance"), dict) else {}
capture_record = _part_index_capture_record(summary_payload, revision_manifest)
all_slices = revision_manifest.get("items", [])
@ -1669,7 +1915,11 @@ def _run_accepted_slice_directory(
discarded_slices.append(row)
else:
kept_slices.append(row)
if disposition_contract:
params = helpers["hyperparams"](node_data) if callable(helpers.get("hyperparams")) else {}
classification_v4 = _bool(params, "DELIVERY_JSON_V4", False)
if classification_v4 and not coverage_required:
raise RuntimeError("DELIVERY_JSON_V4 requires verified full-video coverage")
if disposition_contract or classification_v4:
pins = summary_payload.get("acceptedMaterializations", [])
expected_sha = _sha256_file(revision_manifest_path)
if summary_payload.get("decision") != "approved" or final_acceptance.get("confirmed") is not True or not any(
@ -1679,7 +1929,7 @@ def _run_accepted_slice_directory(
raise RuntimeError("slice dispositions are not pinned to the accepted manifest revision")
if int(coverage.get("sliceCount", -1)) != len(all_slices):
raise RuntimeError("review coverage slice count mismatch")
if disposition_contract and not kept_slices:
if disposition_contract and not kept_slices and not _bool(params, "DELIVERY_JSON_V4", False):
if any(p.get("key") == "PROCESSING_PROFILE" and p.get("value") == "raw_human" for p in node_data.get("hyperparams", [])) and (summary_payload.get("processingProfile") != "raw_human" or final_acceptance.get("processingProfile") != "raw_human"):
raise RuntimeError("raw-human package requires matching accepted task profile")
# Internal evidence only: deliberately no deliveryDirectory output and
@ -1697,13 +1947,12 @@ def _run_accepted_slice_directory(
payloads = _payload_tree(_payloads_with_full_pose_manifests(inputs, output, helpers))
headpose = _first_payload(payloads, "droboticflow.headpose/v1")
handpose = _first_payload(payloads, "droboticflow.wilor.hand_pose_result/v1")
params = helpers["hyperparams"](node_data) if callable(helpers.get("hyperparams")) else {}
if _bool(params, "PART_INDEX_DELIVERY", False):
if source_index and not _bool(params, "SOURCE_INDEX_DELIVERY", False):
raise RuntimeError("source-index JSON delivery requires explicit SOURCE_INDEX_DELIVERY approval; use directory delivery to preserve physical discard exclusion")
calibration_refs = _part_index_calibration_refs(inputs)
lineage = {"acceptedSliceDirectoryArtifactId": directory_artifact.get("id"), "humanItemId": revision_manifest.get("humanItemId"), "taskId": revision_manifest.get("taskId"), "sourceRunId": revision_manifest.get("runId")}
payload, body = _build_part_index(revision_manifest, coverage, kept_slices, source_root, calibration_refs, pose_policy, summary_payload, final_acceptance, capture_record, lineage, discarded_slices)
payload, body = _build_part_index(revision_manifest, coverage, kept_slices, source_root, calibration_refs, pose_policy, summary_payload, final_acceptance, capture_record, lineage, discarded_slices, classification_v4=classification_v4)
if source_index:
_source_index_files(source_root, revision_manifest, False)
payload["schemaVersion"] = _SOURCE_PART_INDEX_SCHEMA
@ -1715,8 +1964,10 @@ def _run_accepted_slice_directory(
for item in payload["slices"] + payload["discardedSlices"]:
item["sourceFrames"] = frames[item["sliceId"]]
body = (json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + "\n").encode("utf8")
if _bool(params, "DELIVERY_JSON_V4", False):
return _emit_part_index_directory(run_id, node_id, output, node_data, helpers, payload, summary_payload, revision_manifest, source_root)
if _bool(params, "DELIVERY_JSON_V3", False):
return _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload)
return _emit_part_index_directory(run_id, node_id, output, node_data, helpers, payload)
return _emit_part_index(run_id, node_id, output, node_data, helpers, body)
if source_index:
source_root = _source_index_physical_export(source_root, revision_manifest, kept_slices, output)
@ -1953,6 +2204,8 @@ def _run_multiport(
def run(run_id: str, node_id: str, output_dir: Any, node_data: dict[str, Any], inputs: list[dict[str, Any]], helpers: HelperMap) -> list[dict[str, Any]]:
params = helpers["hyperparams"](node_data) if callable(helpers.get("hyperparams")) else {}
if _bool(params, "DELIVERY_JSON_V4", False) and (not _bool(params, "PART_INDEX_DELIVERY", False) or _bool(params, "DELIVERY_JSON_V3", False)):
raise RuntimeError("DELIVERY_JSON_V4 requires PART_INDEX_DELIVERY and cannot combine with DELIVERY_JSON_V3")
if _bool(params, "DELIVERY_JSON_V3", False) and not _bool(params, "PART_INDEX_DELIVERY", False):
raise RuntimeError("DELIVERY_JSON_V3 requires PART_INDEX_DELIVERY")
# A part-index delivery intentionally has no directory port; it must still

View File

@ -16,8 +16,8 @@
],
"containerEntrypoint": "backend/operator-runner/main.py",
"entrypoint": "operator.py",
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-package:feaaf80-poc-handpose-auto-20260825052001",
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-package@sha256:fa084bbd0e3b6fedff9582d09c965b460f054ad9e0f5edaf9fc4b9aebcc375cc",
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package:delivery-v4-20261022-8f6b531f",
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package@sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d",
"metadata": "operator.json",
"name": "成品打包",
"operatorId": "operator-builtin-package",

View File

@ -0,0 +1,72 @@
// account-phone-index previews (default) or explicitly migrates user phone uniqueness.
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"io"
"os"
"time"
"droboticflow/internal/store"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func main() {
if err := run(os.Args[1:], os.Stdout); err != nil {
fmt.Fprintln(os.Stderr, "phone index maintenance failed:", err)
os.Exit(1)
}
}
func run(args []string, stdout io.Writer) error {
flags := flag.NewFlagSet("account-phone-index", flag.ContinueOnError)
database := flags.String("database", "", "exact database name (required)")
apply := flags.Bool("apply", false, "create live-account unique index, then remove legacy phone_unique")
if err := flags.Parse(args); err != nil {
return err
}
if flags.NArg() != 0 {
return fmt.Errorf("unexpected positional arguments")
}
uri := os.Getenv("MONGO_URI")
if uri == "" || *database == "" {
return fmt.Errorf("MONGO_URI and --database are required")
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
client, err := mongo.Connect(ctx, options.Client().ApplyURI(uri))
if err != nil {
return err
}
defer client.Disconnect(context.Background())
db := client.Database(*database)
col := db.Collection("users")
cur, err := col.Indexes().List(ctx)
if err != nil {
return err
}
var indexes []bson.M
if err = cur.All(ctx, &indexes); err != nil {
return err
}
deleted, err := col.CountDocuments(ctx, bson.M{"deletedAt": bson.M{"$type": "string", "$ne": ""}})
if err != nil {
return err
}
preview := map[string]any{"database": *database, "apply": *apply, "existingIndexes": indexes, "deletedAccountsPreserved": deleted, "createIndex": "phone_live_unique", "dropIndex": "phone_unique", "businessRecordUpdates": 0}
if err = json.NewEncoder(stdout).Encode(preview); err != nil {
return err
}
if !*apply {
return nil
}
if err = store.NewMongoStore(db).MigrateUserPhoneIndex(ctx); err != nil {
return err
}
return json.NewEncoder(stdout).Encode(map[string]any{"completed": true, "database": *database})
}

View File

@ -0,0 +1,76 @@
package main
import (
"bytes"
"context"
"fmt"
"os"
"strings"
"testing"
"time"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func TestPreviewAndApplyPhoneIndexLocalMongo(t *testing.T) {
if os.Getenv("ACCOUNT_PHONE_REUSE_LOCAL_MONGO") != "1" {
t.Skip("isolated local Mongo only")
}
t.Setenv("MONGO_URI", "mongodb://127.0.0.1:27018/?directConnection=true")
ctx, cancel := context.WithTimeout(t.Context(), time.Minute)
defer cancel()
client, err := mongo.Connect(ctx, options.Client().ApplyURI(os.Getenv("MONGO_URI")))
if err != nil {
t.Fatal(err)
}
defer client.Disconnect(context.Background())
db := client.Database(fmt.Sprintf("phone_reuse_cli_test_%d", time.Now().UnixNano()))
defer db.Drop(context.Background())
col := db.Collection("users")
if _, err = col.Indexes().CreateOne(ctx, mongo.IndexModel{Keys: bson.D{{Key: "phone", Value: 1}}, Options: options.Index().SetName("phone_unique").SetUnique(true)}); err != nil {
t.Fatal(err)
}
if _, err = col.InsertOne(ctx, bson.M{"id": "audit", "phone": "13900000001", "deletedAt": "2026-10-21"}); err != nil {
t.Fatal(err)
}
var before []bson.M
cur, err := col.Indexes().List(ctx)
if err != nil {
t.Fatal(err)
}
if err = cur.All(ctx, &before); err != nil {
t.Fatal(err)
}
var out bytes.Buffer
if err = run([]string{"--database", db.Name()}, &out); err != nil {
t.Fatal(err)
}
var after []bson.M
cur, err = col.Indexes().List(ctx)
if err != nil {
t.Fatal(err)
}
if err = cur.All(ctx, &after); err != nil {
t.Fatal(err)
}
if len(before) != len(after) || !strings.Contains(out.String(), `"apply":false`) {
t.Fatal("preview mutated indexes or did not identify read-only mode")
}
if err = run([]string{"--database", db.Name(), "--apply"}, &out); err != nil {
t.Fatal(err)
}
if err = run([]string{"--database", db.Name(), "--apply"}, &out); err != nil {
t.Fatalf("retry: %v", err)
}
if _, err = col.InsertOne(ctx, bson.M{"id": "replacement", "phone": "13900000001"}); err != nil {
t.Fatalf("migration ineffective: %v", err)
}
if count, err := col.CountDocuments(ctx, bson.M{"id": "audit", "phone": "13900000001", "deletedAt": "2026-10-21"}); err != nil || count != 1 {
t.Fatal("historical business record changed")
}
if _, err = col.InsertOne(ctx, bson.M{"id": "duplicate", "phone": "13900000001"}); !mongo.IsDuplicateKeyError(err) {
t.Fatalf("unique protection removed: %v", err)
}
}

View File

@ -0,0 +1,45 @@
# 删除账号后复用手机号
## 业务契约
账号删除完成后,旧用户 ID、姓名、原手机号、删除时间及历史任务归属继续保留。原身份不能重新启用,旧密码和旧会话失效。原手机号可创建一个全新账号,角色及供应商按正常创建权限校验;新账号不继承旧账号的任务、项目成员关系或历史产出。
仅停用账号不释放手机号。删除操作中途失败、尚未写入 `deletedAt` 的账号也继续占用手机号,须按既有删除重试流程处理。未删除账号的手机号在全平台唯一,跨组织也不允许重复。
手机号及旧邮箱登录兼容查询排除已删除身份。`GetUser(id)` 继续读取旧身份供审计;不能把手机号当成历史归属关联键。
## 实现与兼容范围
MemoryStore 与 MongoStore 使用相同的账号删除判断。Mongo 新索引 `phone_live_unique` 为 phone 的部分唯一索引,包含 `deletedAt` 缺失、null 或空字符串的记录;停用但未删除的账号仍在索引中。新索引冲突仍返回原有 phone 校验错误,前端显示“该手机号已被其他账号使用”。
应用启动只创建新索引,不自动移除历史 `phone_unique`。因此老数据库在显式迁移前仍会阻止删除后复用;仅部署代码不足以完成修复。新数据库直接使用新约束。
## 发布与索引迁移顺序
1. 按正式发布流程部署本修复,确认所有账号登录/管理实例均使用新版查询。仍保留旧代码的进程须核实不调用手机号登录/账号管理,且其实际启动模式为 `verify-existing`、不会重建旧索引;不满足时须另行升级该组件。此时旧索引仍保护全量记录。
2. 从既有未跟踪配置取得 `MONGO_URI`。默认只读预览,显式指定数据库:
```sh
go run ./cmd/account-phone-index --database <database>
```
预览包含数据库名、现有索引定义、保留的已删除账号数量及计划操作;不输出手机号、密码或业务正文,不修改索引和业务记录。
3. 将该数据库/索引变更作为独立迁移计划提交授权。取得明确授权后,才执行相同命令并增加 `--apply`。
4. 工具先创建 `phone_live_unique`,成功后才移除旧 `phone_unique`;新索引构建失败就停止,不移除旧约束。整个过程不更新用户、任务或审核记录。中断后可重复执行;任一阶段都有唯一索引保护未删除身份。
5. 线上采用只读验收;创建客户账号或执行其他写入测试须单独获得对应授权。客户之后可通过正常管理界面使用原手机号创建新账号。
原先已经完成删除的账号不需要补写或恢复;新索引自动排除它们。仍有其他未删除账号占用同一手机号时继续拒绝创建,不擅自合并身份。
## 回退边界
索引迁移前,可按原发布流程回退代码,旧索引仍在。迁移后,特别是原手机号已创建新账号时,不能直接恢复旧版手机号查询或旧 `phone_unique`:旧版可能查到已删除身份,旧全量唯一索引也可能因历史同手机号记录而构建失败。
需要回退时保留新版账号查询和新索引,优先前向修复;若必须完整退回,应先冻结账号写入、只读列出冲突并另行审批恢复计划。不得删除旧身份、改写历史手机号、转移历史任务或随意删除新账号来强行恢复旧约束。
## 验证
新增 API 契约覆盖:删除后同手机号创建供应商管理员、新旧用户 ID 分离、新密码登录、旧密码拒绝、停用账号继续占号、历史任务及删除记录不变。
本地真实 Mongo 隔离合成库覆盖:旧索引仍在时的失败、新索引先建后删旧、重复迁移及重新初始化、缺失/null/空删除标记兼容、多次删除后复用、8 路并发仅 1 次创建成功、已删除身份不能复活。迁移工具另验证默认预览不改索引,显式执行不改历史用户记录。
2026-10-21 发布预检:线上 Worker 构建为 `963b530f`,显式使用 `DROBOTICFLOW_DATABASE_BOOTSTRAP=verify-existing`,没有手机号查找/创建/更新账号的生产调用;在保持该模式的条件下,本次可保留 Worker 镜像。之后改变启动模式或引入账号写入前,必须先升级其索引/查询实现。

View File

@ -6,3 +6,11 @@
2026-10-15 旧任务兼容补充:缺少 `sliceWorkflow` 的历史任务,若服务端保存的完整切片分区有可靠生成记录,或非全视频任务有完整标注分区证据,可在当前标注员、当前阶段内撤销/重做。判定只读取服务端记录,不以客户端撤销快照充当证明,也不补写阶段标记。显式但不完整的工作流、生成中/失败处理任务、待完成送检和审核阶段仍禁止恢复;版本、归属、真实帧、覆盖及舍弃审计校验保持生效。
## 2026-10-22:撤销/重做仅在请求进行中互斥
用户确认取消旧交互中的一秒防重复规则:恢复请求完成后不再设置时间冷却,成功后可立即继续撤销/重做,失败(返回失败或抛错)不移动历史栈且可立即重试。撤销后产生新编辑或新草稿仍截断重做分支,新编辑可立即撤销。
防重复由进行中的 pending 锁承担:同一历史实例一次只执行一个恢复回调,期间重复撤销/重做不发起第二次恢复,也不排队。任务或阶段切换清空历史后,旧回包不得推进新历史;旧请求结束前仍保持互斥。恢复回调所属协调器继续负责数据写入及任务身份校验;本修订只取消前端完成后的时间门槛,不改变阶段、持久化确认及页面会话边界。
生成失败任务的例外:仅切片生成失败、无运行中任务且当前草稿证据可靠时,可以在草稿中恢复边界,原生成快照保持冻结;详见 ADR 0007 的 2026-10-22 生成失败编辑修订。上述旧任务规则中对失败状态的一律禁止由此窄范围例外替代。

View File

@ -15,3 +15,17 @@ status: accepted
Q7–Q13 随后明确:必要素材复制到测试存储;5 个独立测试角色账号;固定 6 份样本模板,最多 6 条活动测试任务,手动重建;切片并发 1,资源不足暂停;受控命令管理,按批次切换版本,保留最近 3 批结果并在确认后清理更旧测试产物。
本决策不授权创建账号、复制样本、部署或修改客户数据。具体来源清单、资源额度、账号使用者和执行审批见[需求讨论](../../.scratch/internal-test-lane/map.md),完整方案确认后再推进实施准备。
## 2026-10-20 修订
客户已投产,内部不能再借客户账号验证;候选版本须先在测试环境走完人工标注链路,再把同一镜像晋升到生产。以下决定替代上文及 2026-09-27 部署计划中与之冲突的部分,逐条依据见需求讨论中 2026-10-19/20 四轮记录。
- **范围**:只验证标注 → 质检 → 最终验收。工作流 worker、算子 Job、标准交付打包和 Capture 接入不在测试环境中运行;对应发布证据保持未验证。
- **存储**:不创建 BOS 子用户或测试 bucket。选中样本从生产 BOS 一次性只读下载,导入测试节点本地盘;测试环境是纯本地工作区,不挂载 BOS、不持有任何 BOS 凭证,切片与结果只写本地盘。未采用只读挂载生产 bucket:测试命名空间须持有全桶读写凭证,隔离只剩只读挂载和应用前缀检查。未采用写入生产 bucket 的 `internal-test/` 前缀:测试产物会进入客户生产存储。代价是 BOS 读取与 ETag 版本检查只能在晋升后的生产冒烟中验证。
- **资源**:固定在当前没有生产应用 Pod 的节点 `10.189.16.4`,不新增节点、不改生产调度;Mongo 与工作区用该节点本地盘上的静态卷。生产算子 Job 仍可能调度到该节点,观察到争用再申请专用节点。
- **样本与账号**:从最新生产备份选样,不查询线上库;每个在产项目 3 条、总数不超过 12。两个测试供应商、5 个业务角色账号,不设平台管理员和供应商管理员,用以覆盖多供应商领取与不可见。
- **版本**:每批先以生产当前镜像启动并制造在途状态,再用与生产相同的晋升脚本升级到候选版本。一批对应“生产版本 → 候选版本”,替代上文“按批次一次运行一个版本”。
- **晋升**:只晋升到使用同一镜像的 api、annotation-processor、annotation-validator;worker 不随之更新,候选改动 worker 代码时拒绝晋升。测试报告只有与本次发布镜像 digest 一致时才作为发布证据。原 `annotation-delivery` 证据拆为标注结果与交付打包两项:前者可由测试批次满足,后者需要打包测试,或经核对候选版本未改动交付输入。
- **生产内测试组织**:2026-09-27 起生产库中的内部测试组织冻结,只用于晋升后的生产冒烟,默认只读。
每次执行仍逐步单独授权:拉取备份、下载样本、构建推送镜像、写入测试命名空间、启动与升级演练、晋升生产。

View File

@ -43,3 +43,19 @@ status: accepted
路径基准显式写入:总索引相对本批次项目根,采集清单相对采集交付目录,原始 meta 和标定文件相对原始采集目录;已固定 workspace 来源另标基准。采集清单记录原始 `meta_<captureId>.json` 的 SHA/大小,expected parts 来自原始目录完整分页列举,不用已验收 part 数推定交齐。最终验收结论及最大 acceptedAt 的聚合仅覆盖本批次选择的修订。
总索引生成由管理员显式预览、按预览 snapshotId 条件发布,不随普通 GET/下载自动写对象存储。旧已完成 part 目录不能原地补发 v3,历史迁移仍需另行规划新交付目标。已有调用者和在途任务默认沿用旧协议;回退 API/开关不删除新批次,也不能把新格式任务改成旧格式后重用已完成目录。
## 2026-10-21 修订:分类结果与内部追溯独立
用户明确交付价值为最终人工确认的分类及标签,而非过程记录。新 v4 主 JSON 只包含最终验收通过的分类、区间及必要源定位;审核批注、人员、轮次、返工和抽检过程不进入主 JSON,也不默认进入客户批次。完整正式追溯保存为内部 JSON,并与固定结果修订关联。返工经过重新验收的分类修正应进入新交付版本,单纯修改批注不能改变分类。
独立字典 tags 由管理员维护,与 ID/版本/名称一起在确认时冻结;category 与 aliases 不当作 tags。未写动作描述不等于没有动作,也不撤销切片已有分类。历史没有标签快照时不反查最新字典拼造历史。全舍弃也属于人工分类结果,因此 v4 仍交付该 part 的分类 JSON;这显式替代 v4 路径的旧 no_deliverables 规则,旧媒体目录和 v1–v3 路径继续保持原行为。
采用独立 v4 协议与默认关闭的开关,避免同一 v3 schema 悄悄改变字段、枚举或文件集合。保持已约定的 capture/part、slices/taskSplit/segments 层级及不可变批次身份,动作有描述使用约定的 annotated。保留旧版本读取与发布恢复,不原地修补历史完成目录。单独写内部追溯会增加一份内部产物,但避免客户为读取分类被迫接触人员、批注和平台流程字段。
## 2026-10-21 补充:动作继承及原始数据消费
用户确认切片与动作是父子分类。动作独立选择词条时使用该快照的标签,否则继承父切片冻结标签;不按描述是否为空决定,不伪造动作词条。v4 显式输出有效 tags 和 action/slice 来源,保留独立无效判断;舍弃仅保留原因分类及定位,不带旧动作或审核批注。缺标签不反查最新词条猜补。
下游需要仅凭总 JSON 和原始数据定位、切分、按类归组。新 v4 使用任务原始输入的 BOS 选择冻结八类源文件及 bucket/prefix,主文件不再依赖平台 workspace。动作同时明确切片内秒数、原始 part 秒数与原始硬件时间戳;用原始左右 PTS 各自解析硬件区间,避免用左视频秒数误切另一相机。输入快照指纹不是单个 BOS 对象内容版本的保证。实际切割和分类打包仍由下游执行。
缺少可靠来源、分类标签或时间证据时,选择拒绝新 v4 交付,而非“尽量输出”后让客户承受丢分类或错切的风险。发布同时验证消费契约、来源归属和文件存在。已交付结果不原地改写;历史只按证据补齐并产生新批次。这个要求适用于尚未发布的 v4,不改变 v1–v3;旧本地 v4 候选须重新生成。实现和本地消费验证见 `docs/delivery-classification-v4.zh-CN.md`。

View File

@ -22,6 +22,14 @@
- 复审新增“本轮改动”清单,覆盖全部切片与动作,包括原先没有退回意见的位置。它比较本轮冻结基线与当前实际送检修订,展示净改动及前后内容;不使用客户端操作日志、不把自动保存草稿混入正式结果,不修改冻结基线和正式审查记录。标注员侧显示当前工作区的同一比较结果。
- 缺少旧基线快照或匹配的送检修订时显示“无法确认全部改动”,不以空数组推断全部删除或没有改动。无需数据库迁移;旧结构计数摘要和旧格式退回兼容路径保留。
## 2026-10-21:实时草稿与派生状态比较修正
标注员侧强调层合入区间完整的编辑器草稿,保留分类与左右手等业务字段;未完成终点选择不参与比较,自动保存载荷的字段清洗不用于展示比较。撤回到原文恢复未修改状态。
切片 `annotationStatus` 是派生完成状态,单独变化不代表已按意见修改;旧记录省略用途与显式 keep 等价。名称、分类、边界、用途、舍弃原因及片内动作仍参与比较。灰态继续只表示已修改待复审,不替代正式判定。
选择舍弃切片时仍显示只读舍弃证据与返工意见栏;不提供直接修改归档动作的编辑器。上述均为展示修正,不迁移冻结基线、正式审查记录或客户工作区。
## 背景
B02 上线的返工模式给标注员两个新面板:意见列表上方的"处理反馈"文字编辑器(每条意见写回应,8 秒空闲自动保存)与"上下对比"面板。客户实测后提出三点不满:

View File

@ -16,6 +16,25 @@
- 默认“截断”在边界切开动作,进入舍弃侧的部分归档,保留侧不被整段无效化。“判无效”仍是显式的整段无效判断(两侧几何保留);清单说明影响范围,不将其作为跨保留/舍弃边界的唯一可保存选项。
- 切片重设面板与冲突清单占用右侧编辑区,普通宽屏预留侧栏,窄屏进入正常文档流,避免遮挡视频;替代早期“锚到分界处”的浮卡位置规则。帧号输入及单帧步进不变;Shift+方向键仍为 0.5 秒粗调。
## 2026-10-21:拖拽与冲突清单应用路径补齐
这是 10-19 边界归档语义的入口补齐,不新增分类或舍弃状态。
- 拖拽释放、快捷边界调整、拆分及删除的快捷应用,与 E 面板共用服务端切片原子应用;前端对归档后的活动动作作包含校验,但发送原始分割动作,由服务端维护归档、版本检查与审计。
- 一次拖动及其确认只记录一条持久化历史,撤销基线为拖动前工作区。正常侧保留原描述、状态和分类信息;显式“判无效”的语义不变。
- Enter 和按钮均等待权威保存结果;保存中禁用重复应用、处置选择及取消,错误在清单内显示并保留选择。响应丢失时通过原应用编号回读确认。成功后清理已合入保存的编辑器草稿,避免用旧几何覆盖新工作区。
- 不批量修改已保存、送检或交付数据;现有权限、阶段锁、历史数据恢复接口保持有效。
## 2026-10-21:初始切片快捷冲突兼容
初始切片阶段保存、恢复的工作区可能已有连续待标注动作。切分、拖边和删除遇到这些动作时,与逐片标注阶段一样显示冲突清单,Enter/按钮确认或 Esc 取消;不再因阶段不为 annotate 静默取消。E 重设面板仍限定逐片标注,质检、验收、只读及后台处理阶段不开放编辑。
不按描述为空删除动作,也不批量改写历史草稿。既有自动补齐、空白动作、分类、舍弃证据与版本校验保持原义。成功操作按原子应用记录撤销,失败保留选择及明确错误;原有会话、阶段边界保留;当时保留的一秒防重复规则已由下述 2026-10-22 修订替代。
## 2026-10-22:撤销冷却修复
撤销/重做取消已完成请求后的一秒冷却,仅保留请求进行中的 pending 互斥;成功后可立即按顺序继续,失败不消耗历史且可立即重试。撤销后新编辑截断重做分支并可立即撤销。任务或阶段清空历史后,旧回包不推进新历史。此规则替代旧一秒防重复约定,详见 [ADR 0001](0001-annotation-undo-stage-boundary.md#2026-10-22撤销重做仅在请求进行中互斥)。
## 背景
返修中要改切片结构只能"重新切片":清空全部动作标注、整单退回切片阶段,代价高且不可审计(旧文案"请先返回重新切片"即两把阶段锁的报错)。同时旧 D2 规则在拖动边界穿过动作时自动拆两段标"待核对"——该状态不在服务端白名单(pending/described/invalid)内,不处置则保存被堵死。本决策来自 24 项已拍板裁决(Q1:a–Q8:a、R2-Q1–Q8、R3-1–R3-4)。
@ -43,3 +62,9 @@
- reslice 写端点 409 属行为变更:旧前端调用会得到明确错误与原地编辑指引(同版本部署,无混跑窗口);`sliceResetHistory` 只读残留,无数据迁移或删除。
- 撤销边界不变(ADR 0001):撤销仍限于当前任务当前编辑阶段,生成切片、送检、换任务、刷新截断历史——触发项中原有的"重新切片"随入口删除而消失。
- 边界移动的处置责任前移到拖动现场:任何未处置的越界/贴边动作都会让边界停在原地,标注员不可能"顺手划过"而丢动作;代价是处理卡是强制交互(模态)。
### 2026-10-22:生成失败后的边界编辑
切片生成失败且没有运行中任务时,归属标注员可以继续修改边界、撤销和重做。服务端从归属、版本与时间证据有效的当前草稿校验修改,结果仍保存为草稿;失败生成的确认快照、任务输入和失败状态保持不变。自动保存使用相同校验基线。缺少可靠快照时间的历史记录不自动选择或覆盖。
修改后的草稿与原生成输入不同,普通重试必须拒绝;用户明确重新确认后才用当前草稿启动生成。正在执行的任务、非切片处理失败、非归属用户、待提交状态及过期版本仍不可通过该路径修改。此改动不迁移线上旧快照,也不改变交付时间坐标。

View File

@ -28,3 +28,9 @@
- **折叠显示已解决项**:保留条目、界面折叠。弃用——标注员工作面只关心未完成项,折叠行仍是噪音,J/K 仍要跨过它们;"彻底不显示"是用户明确改判的推荐(R1-Q8)。
- **冻结基线记录上打已解决标记、前端过滤**:要求改写 B02 冻结记录(不可变契约)或让所有消费端各自实现过滤,旧客户端无法过滤,且读写两端要同步部署。弃用——收窄收敛在服务端查询视图一处。
- **复审重写意见表达"仍有问题"**(旧事实行为):每轮产生新条目 ID,条目数与 J/K 跳转随轮次漂移,R2-Q7 明确否决。
## 2026-10-22:复审通过门禁
当前轮退回基线中仍有可见意见未被质检员明确判为已解决时,不允许新的质检通过决定。标注员改动使意见变灰仅表示“已修改待复审”,不能代替质检判定;缺失核对结论按未解决处理。前端在基线加载期间、读取失败或存在未解决意见时禁用通过按钮,正式提交前再次核对;服务端对新请求独立执行同一门禁。旧客户端未传操作标识也不能跳过此检查。
已受理的持久操作沿用受理时的恢复协议;不回写既有审查记录,不自动退回已通过任务,不迁移冻结基线。缺少旧基线快照时,界面保留未修改状态,不以无法比较推定已修复。缺省动作状态按既有内容推导后比较,补齐默认状态本身不算修改。

View File

@ -30,8 +30,18 @@
新的 JSON-only 源索引交付是 **v2 协议**,含固定完整源的逻辑键、哈希、双目帧区间。它只能交给有权读取完整源、承诺按区间解析的下游。完整源含舍弃区间,因此不能把它视为“物理交付视频不含舍弃区间”的等价替代。该变化尚未得到交付协议批准,必须同时显式设置 `PART_INDEX_DELIVERY=true` 和 `SOURCE_INDEX_DELIVERY=true`;仅有旧开关时拒绝新源索引 JSON 交付。目录交付无需放宽旧契约。上线启用前必须确认实际工作流使用哪种交付方式。
## 2026-10-21:MP4 呈现终点量化兼容与旧任务恢复
逐帧时间戳和解码帧数检查保持原要求。总时长检查还需考虑 MP4 编辑列表使用的 movie timescale,而非只看视频 track time base:新的终点容差为原容差与 `min(movie tick + track tick + 浮点余量, 半个帧周期)` 的较大值。无法读取有效 movie clock 时维持原容差,不扩大放行范围。
B06/part0002、B14/part0001、B18/part0003 的本地样本均为连续 30fps,packet 时长全部是 512 track tick;编辑列表导致呈现总时长少 5 个 track tick,约 0.326ms。不得将此误认为丢帧,也不得直接删除时长检查或统一放宽一帧。兼容处理不改源文件、硬件 PTS、统一边界或已有证书,不以忽略编辑列表重新定义播放起点。整帧截短、帧数差异和逐帧断档仍拒绝。
没有 `sliceWorkflow` 的旧任务仍按已有正式生成证据识别。只有草稿而没有有效生成证据的任务不能凭前端推断进入标注;允许操作者显式按服务器保存的完整快照恢复生成,复用持久作业、租约和版本竞争机制,并保留动作、描述及分类。已确认工作区之外还存在较新的本人自动保存草稿时,恢复采用该草稿,与页面恢复内容一致;归属、版本、时间或数据格式无法核对时停止,不能猜选或清空。恢复模式需跨失败重试保留,避免再次退回旧工作区。恢复不是清空后重新切片。
## 成本、兼容性与回退
2026-10-22 补充:普通失败重试不能以旧确认覆盖页面恢复的较新草稿。重试核对确认内容指纹;内容不同需操作者显式按当前保存内容重新确认,保留旧快照及已有动作。后台完成时再次核对,旧队列作业亦适用,不自动回填或重算历史时间。具体契约见 [标注时间完整性](../annotation-time-integrity.zh-CN.md)。
- 固定源每 Run/源版本复制一次,不是跨 Run 去重;首用仍消耗解码 CPU、读取带宽与一份源大小的存储,之后修订为 JSON。
- 固定源不随旧修订目录清理;目前没有自动 GC,以免删除在途审核及已交付索引引用。启用前评估容量,后续需基于引用和保留期设计 GC。
- Worker 识别源索引输入时必须使用 `ANNOTATION_SOURCE_PACKAGE_IMAGE` 指定的不可变 package 镜像摘要;旧 package 镜像不支持该协议,缺配置即启动任务前拒绝。

View File

@ -0,0 +1,59 @@
# Branch workflow for parallel agents
Several agents work on this repository at once. Every branch must answer three questions
without anyone reading its history: what it is for, what it was built on, and whether it
has reached `dev`. Git answers the last two; the task card answers the first.
## Starting a task
```sh
git fetch origin
git worktree add --no-track -b <agent>/<slug> local-data/worktrees/<slug> origin/dev
```
- `<agent>` is the tool's usual prefix (`codex`, `claude`, ...). `<slug>` is a short topic
without a date, and is the same slug as `.scratch/<slug>/`.
- Start from `origin/dev`. To build on an unmerged branch, the user must ask for it. Its
PR then contains that branch's commits too, so say so in the PR description.
- Keep `--no-track`: without it the new branch tracks the remote branch it started from,
and a later `git push` could land there. Push with `git push -u origin <agent>/<slug>`.
## Task card
The first commit on the branch adds `.scratch/<slug>/work.md`. Keep it short. Do not
record status in it: `scripts/agent_status.py` derives status from git.
```md
# <one-line title>
Goal: what the user asked for, in one line
Agent: tool and session that owns the branch
Base: origin/dev @ <short sha> (or the unmerged branch the user chose, and why)
Scope: modules or files this task may change
Out of scope: related work deliberately left out
Risk: API, data, delivery-format or deployment impact; "display only" when none
```
Update `Scope`, `Out of scope` and `Risk` when the plan changes. The card merges into `dev`
with the work and stays as the record of why the change was made.
## Ending a task
1. Run `python3 scripts/agent_status.py`. It fetches `origin` and lists for each branch the
commits ahead of and behind `dev`, unpushed commits, a missing card, stale work, and
the newest deployed release with whether its commit is in `dev`.
2. Push the task branch (no approval needed for your own `<agent>/<slug>` branch) and report
its line to the user. If it is ready for review, remind them; creating the PR needs their
explicit approval, and after it you push the final state and create the PR together.
3. After the PR merges, delete the remote branch in Gitea, remove the worktree, and run
`python3 scripts/agent_status.py --prune`. It deletes only local branches whose
commits are all in `origin/dev` and that no worktree has checked out.
## Release records
`scripts/release.py` writes its ledger under the checkout it runs from, so records are
spread across worktrees. `agent_status.py` reads all of them. Do not delete a worktree that
holds a release record still in `deployed_pending_acceptance` until the record is copied to
the main checkout's `local-data/releases/`.
Tests: `python3 -m unittest discover -s tests/agents -p 'test_*.py'`.

View File

@ -0,0 +1,13 @@
# 管理员切分与标注任务池
2026-10-21 本地修复。适用于平台管理员的“切分与语义标注”队列;标注员随机领取、个人缓冲池及质检/验收原入口保持原合同。
“可领取/我的任务/全部任务”统计当前权限与项目、工作流、执行、Capture 范围内的完整人工任务池,不再把首屏或已加载列表条数当作总数。搜索词和状态分类只筛选下方列表,不改变这些总数。管理员“全部任务”沿用当前环节范围,包含标注及已送检质检任务,不是全部历史已交付任务。
新增只读接口 `GET /api/plugins/annotation-qa/admin-pool`,只有组织管理员/平台管理员可用,组织范围由服务端身份决定。有效批次及阶段校验保留;非管理员不能通过传入 pool、用户或组织参数取得公共任务明细。旧接口保留,已有浏览器和在途任务无需迁移。
列表默认且最多每页 200 条,以人工任务的 `(createdAt,id)` 游标继续,任务摘要只关联这一页的 ID。“显示更多”读取下一页人工任务,而不是继续拉取独立的最新任务窗口。搜索及状态分类在服务端处理,能够找到第 500 条以后的任务;旧 Capture 名称仍通过原有元数据提取规则兼容。切换项目、列表或搜索后,不接受旧请求迟到的内容。
总数按批次/阶段聚合紧凑计数,再用既有批次权限投影校验;不读取动作正文或草稿来计数。总数使用有身份和范围隔离、容量上限的 5 秒进程缓存,当前进程的 API 写入会使缓存失效;其他副本与后台写入最多延迟到缓存到期。列表不缓存。搜索输入合并 350 毫秒,新搜索与范围切换取消过期请求;旧任务搜索在服务端流式读取投影元数据,查询和范围统计受 8 秒期限限制,不向浏览器传整池,也不把失败当成零条。计数与页面是相邻的只读操作,并非冻结生产时刻的结算快照。
不新增统计集合、数据库字段或索引,不改领取、标注、返工、审核及交付结果。生产环境仍需单独发布与验收。

View File

@ -100,6 +100,7 @@ Ctrl/⌘+Z 可连续撤销,Ctrl/⌘+Shift+Z 可重做。最多保留最近 100
- **被边界穿过的动作**(三选条目):`1` 截断(动作在边界处分成两段,各段保留原状态)/`2` 挪片(段迁入对面切片保留原状态,让出的空当自动生成显式无效段)/`3` 判无效(整段截断几何全部转无效)。
- **恰好贴边的有效动作**(两选条目):`1` 跟随伸缩(默认)/`2` 保持原边界。贴边的无效动作不成条目,自动跟随伸缩(零打扰)。
- 清单键位:`↑/↓` 在条目间移动(回绕);`1/2/3` 选当前条目处置,选完自动跳下一条未决;`Enter` 应用(未决条目按默认——三选截断、两选跟随——批量处理);`Esc` 取消整个待应用改动集。不支持部分应用。
- Enter 或按钮确认后显示“正在保存…”,等待完成再继续;保存期间不能重复应用或取消。失败原因显示在清单内,原选择保留,可核对后重试。
- 清单打开期间其它键不响应,其它边界改动也被冻结;一次只决一件事。
- 无论怎么选,时间轴覆盖永不缺口、动作/无效两轨永不重叠;自动生成的无效段与手工无效动作完全同形,可编辑、可撤销。

View File

@ -0,0 +1,36 @@
# 标注确认、时间映射与存量兼容
## 2026-10-22 修复契约
合法的源帧边界不等于用户确认了该边界。页面恢复自动保存草稿,不能使失败重试静默采用另一份已确认工作区。切片、动作、字典版本与标签、舍弃理由必须属于同一次确认。
普通失败重试只接受与原生成确认相同的业务内容。服务端比较规范化的切片、动作和状态,不将保存时间及压缩后的操作历史作为业务差异。有较新的不同草稿时返回 `409 slice_retry_snapshot_conflict`,不入队、不清空或替换两份保存内容。
操作者可显式选择“按当前保存内容重新确认生成”。该请求仅传任务版本和 `reconfirmSaved`,服务端读取已保存草稿,核对操作者、任务版本、保存先后、字典及统一帧边界,建立新的持久作业;客户端提供的重试快照与内容哈希不被信任。它不沿用初次切片确认清空动作的语义。
每次确认记录业务内容指纹。后台生成前及完成发布前分别核对指纹与当前可恢复内容;不一致时生成失败,保留内容及失败作业输入,不能通过删除新草稿发布旧结果。部署前已入队、没有新指纹的作业也执行内容比较。旧租约和任务版本机制继续阻止过期进程写入。
重新确认前,旧工作区和可获得的失败作业快照以内部准备事件保存;准备事件不是已经受理或生成成功的证据。若事件保存失败或超过完整保存上限,请求停止。生成结果及内部事件关联确认指纹;这些过程字段不进入客户总 JSON。
## 时间与来源检查
原时间网格和已保存秒数不在此次修复中重算。硬件 PTS、左右相机各自帧范围及半开区间语义保持不变。新 v4 要求可靠完整的秒数与硬件 PTS 映射;仅有首尾连续性不能证明两种坐标一致,缺证据时拒绝新交付,而不是按固定帧率或时长插值猜补。
接入时的原始 Run 已保存所选对象键及由 `key|ETag|size` 生成的指纹。v4 在打包执行前、写交付目录归属标记前,以及批次预览/发布时重新 HEAD 同一选择并核对该指纹,包括原选择中的可选音频。不能从当前列表、Capture 的新状态或当前文件重新生成“期望值”。读取键有界,不重新下载或哈希整段视频。
ETag/大小指纹用于检测对象身份变化,不等同于逐对象 SHA256 或不可变对象版本;无法证明外部对象从未被覆盖后恢复。保留内部固定源的现有 SHA256 校验。原始选择缺失、范围不一致或身份发生变化时,只拒绝该次 v4 发布,不自动修复历史。
所有 v4 打包节点和源索引输入使用 Worker 已有的 `ANNOTATION_SOURCE_PACKAGE_IMAGE` 不可变镜像配置。物理 v1–v3 输入继续使用原工作流镜像,历史读者与已交付文件保持兼容;此次发布不原地改写工作流定义。
## 旧数据处理
- 编辑中或生成失败的冲突内容:两份保留,操作者核对后显式重新确认;不按保存时间自动择新。
- 已生成、待质检或待验收:被审修订固定。发现确认/生成不一致时走正式返修,不以新草稿替换被审结果。
- 已交付:不覆盖原文件。确认错误后重新验收并建立新不可变批次;证据一致的历史交付无需重生成。
- 缺历史来源或映射:报告缺口,不猜补、插值或把当前证据冒充过去证据。
本次应用升级没有启动历史盘点、回填、时间重算或批量重生成。任何生产修复脚本需要另列具体范围、预览及授权。
## 验证要求
回归覆盖保存版本冲突、显式重新确认、原标签与动作保留、任务归属及版本拒绝、失败/进程重启恢复、旧作业与后来的草稿竞争,以及物理/源索引两种 v4 独立消费路径。构造“秒数连续但 PTS 错误”及“同大小、不同 ETag 的原始文件”,必须在发布前拒绝。

View File

@ -0,0 +1,80 @@
# 标注保存协调与共享切片编辑会话
2026-10-21,继看板展示分层 `ff1cf0be` 后的第二阶段。本轮沿用现有 HTTP、记录结构、权限与工作流;只调整前端职责,并扩充本地测试。公司环境尚未部署本轮代码。
## 保存协调
`src/lib/annotation-persistence.js` 为显式写入集中处理:
- 取得当前任务/账号的独占操作令牌,等待已经在途的自动保存,使用其确认后的最新版本。
- 自动保存结果不明或等待失败时保留当前编辑,释放操作令牌,停止显式写入;编辑资格在等待后再次核对。
- 描述确认、切片用途、结构应用、撤销恢复及原有重新切片兼容入口共用同一读回逻辑。请求失败后只读回服务器,不自动重发写请求。
- 只有任务/账号仍匹配、任务项 ID 相同且版本不倒退时才采纳工作区。格式损坏的响应拒绝,历史 null 列表继续规范化为空列表。
- 更新详情、个人池元数据和保存基线集中在一个位置。`draft` 只更新草稿元数据与保存基线;`confirmed` 更新正式确认基线而保留调用者的编辑展示;`full` 采纳结构和动作。草稿不能冒充正式确认供撤销使用。
原有 `annotation-authoritative-mutation.js` 的令牌与读回算法继续复用;新增写请求前的令牌检查,避免已经失效的操作仍发送请求。各业务操作自己的应用编号/版本/归属判定、历史记录与错误处理留在调用者。
生产 adapter 使用现有 HTTP 客户端;模块测试使用可控的延迟响应。React 页面负责展示和更新自己的状态,服务器仍负责真正的授权与 CAS,前端令牌不是服务器权限机制。
## 共享编辑会话
`src/lib/slice-edit-session.js` 统一 E 面板、拖拽及结构快捷入口的会话快照与异步提交。
- 打开时冻结原始切片、动作、完整描述草稿及任务/账号上下文身份。几何、描述、分类相关字段完整保留,不通过手列字段重建快照。
- 草稿指纹、动作或切片结构发生外部变化时停止应用;拖拽只允许本次手势已知的预览结构不同于基线。相同内容的新数组可通过,不再依赖数组引用判断。
- 未完成动作终点不能被结构应用消费。面板保留原有定位、驻留、Esc 和冲突处置交互。
- 两个入口共用一把在途提交锁;失败保留选择与快照,旧上下文的在途请求不阻塞新任务;迟到响应不能结束另一个新会话。即使切走后回到相同任务/账号,旧上下文身份也无效。
- 正式结构保存成功才消费已合入的编辑草稿,并发送一次统一完成信号。撤销/重做不触发该草稿清理信号,保留恢复快照的机会。
- 独立工作台夹具使用同一实现的本地会话;完整页面由父级提供共享会话。
没有改变冲突分类、无效/舍弃含义、保留侧描述、审核只读限制或交付协议。ADR0001 的任务/阶段撤销范围、ADR0007 的原子归档与单条历史、ADR0010 的旧任务恢复继续有效。原有历史恢复和首次切片重设兼容路径保留。
## 回归中修复的交互问题
合成“描述保存失败→保留文本→鼠标再次确认”实测发现,工作台吸顶错误条遮住确认按钮。已将错误条改为正常文档流,保留 `role=alert` 和错误正文;同一真实鼠标测试已通过,没有用强制点击或快捷键绕过遮挡。
## 验证与发布检查
```sh
npm run test:annotation:persistence
npm run test:browser:annotation-persistence
npm run test:frontend:gate
```
前两项已接入 Fanyu 发布策略。浏览器命令启动真实 Go HTTP、隔离 MemoryStore、合成双目视频/PTS/IMU 与实际页面;没有客户数据库或 BOS 写入。
本轮结果:
- 24 项模块测试通过,覆盖等待版本、锁释放、账号/任务切换、丢响应、未知结果、旧响应拒绝、失败重试及快照校验。
- 18 个完整页面场景通过,覆盖首次标注/返工、左右舍弃边界、Enter/按钮、失败与迟到响应、撤销/重做、E 面板、描述确认和鼠标再次保存。
- 三角色真实 movi 加载失败后的恢复及标注草稿保留回归通过。
- 切片应用、归档、旧任务恢复和审核不可修改标注的定向 Go 回归通过。
- 全量前端基线门禁通过:13 项失败均属已有 14 项基线,没有新增失败,基线未修改。
- TypeScript、格式、生产构建与生产 dist 浏览器启动通过;发布工具 37 项测试通过。
三项旧前端接线测试原先硬匹配函数内部写法,本轮随责任迁移调整为共享会话接线和实际快照行为;未放宽业务断言。完整页面回归继续覆盖这些业务行为。
独立复核发现的旧上下文在途锁阻塞新任务问题已修复,并补两条并发请求的回归;复核后未发现剩余阻断问题。
最终汇总和日志位于忽略目录 `local-data/verification/annotation-architecture-20261021/`。页面细节证据在本工作树忽略目录 `local-data/test-results/rework-boundary/`;错误条遮挡的失败截图、正文和日志单独保留在 `attempts/description-save-overlay/`。它们是合成样本,不含客户正文。
## 范围与后续
这是保存协调和编辑会话的责任收敛,标注页面尚有任务池、审核和页面编排等职责。任务池读取与分页已在[第三阶段](annotation-pool-loading.md)进一步收敛;后续沿现有测试继续处理审核与页面编排。布局 agent 继续只通过展示配置工作,不接入保存与分类判定。独立镜像、JSON v4、数据库迁移及生产发布均不属于本轮实现。
## 2026-10-21 输入取消与强制关闭回归补充
共享会话改造后的稳定性检查中,真实 React 浏览器复现了两个独立问题:
- 数字边界输入按 Esc 时,状态恢复尚未渲染,紧随其后的 blur 仍读取旧输入,将应取消的值提交到面板预览。现在先标记取消本次 blur,再恢复显示;Enter 统一由 blur 提交一次,鼠标和 Tab 也复用这条路径。
- 权限或阶段变化使 E 面板强制关闭时,工作台未通知父页面结束会话,父页面可能持续暂停自动保存。现在强制关闭同步发出结束通知;没有放宽失权后的编辑资格。
新增 `npm run test:browser:slice-input`,在真实 StrictMode 组件中覆盖帧号与秒数输入、Esc、Enter 单次提交、鼠标应用及下一次 Tab 提交;此命令已接入发布本地检查。完整 Go HTTP 标注回归增加取消输入后应用无改动不发送结构写入的场景。现有权限浏览器夹具增加失权关闭及恢复权限后会话状态检查。均为隔离合成样本,不读写客户数据。
兼容范围不变:HTTP、Mongo、工作流、字典、分类与交付格式均未变更,旧任务继续走原路径。父页面的会话标志只是前端保存协调,不替代服务端授权。
上一轮 E 面板应用后偶发未出现冲突清单的超时,尚未获得稳定复现;本轮原有 18 场景及扩充后的 19 场景均通过,但上述两个修复不能据此认定为该超时的根因,仍保留未闭环状态。新增全页用例最初误将 10fps 合成源当成 30fps,已改为读取实际初值;失败证据保留。全量前端首次受限运行的 devboot 挂起后仅终止本次测试子进程,随后在允许本机监听的环境重跑通过,基线未修改。
验证结果:4 个输入交互场景、19 个完整标注场景、权限浏览器回归、类型、格式、构建、生产 dist 启动和 30 个发布工具测试通过;全量前端仍为 13 个已有失败,没有新增,14 项基线未修改。
证据:本工作树忽略目录 `local-data/verification/slice-input-20261021/`,包含修复前失败与修复后结果。本轮未推送或部署。

View File

@ -0,0 +1,50 @@
# 标注任务池读取与分页
2026-10-21,继展示分层与保存协调之后的第三阶段。实现位于同一仓库,前端复用现有 HTTP 客户端;公司环境尚未部署本轮改动。
## 职责
`src/lib/annotation-project-loading.js` 新增 `loadAnnotationPoolPage`,集中选择管理员、标注员侧栏及普通项目/审核工作区的读取协议,返回任务池、关联任务、游标和管理员计数。它不直接更新 React 状态,不接管领取或正式审核写入。
- 管理员标注池仍使用单次 `adminPool` 响应,筛选、总量、任务和游标来自同一服务器查询。
- 标注员首屏仍由 `loadOwnedAnnotationProject` 提前发布可用侧栏,不等待慢计数或辅助元数据;续页复用同一侧栏读取与旧 API 兼容规则。
- 质检、验收及其他非管理员工作区根据服务端可见任务池的任务 ID 关联任务,不用最新导入窗口替代,避免旧返工任务被新导入任务挤掉。
- 验收继续复用 `humanLoopApi.listAcceptancePools`,保留普通验收与包前验收两阶段、完整分页、版本去重及服务端授权。
- 普通管理员项目视图保留任务窗口分页,服务端 `canClaim` 及展示字段原样传递。审核工作区继续下推阶段筛选,已审核历史查询仍保留原口径。
页面保留项目元数据、辅助报表、提交后的临时展示覆盖和错误恢复 UI。首屏只有整组依赖成功且请求序号、页面范围均有效,才接收任务池及游标;元数据失败不会提前推进游标或发布管理员计数。旧闭包入口在改变请求序号、清空列表或中止请求前先检查页面范围。
## 分页并发
首屏刷新会使旧续页失效,清空续页游标并释放旧分页 UI。续页由同步 ref 取得一次请求所有权,避免 React 重新渲染前的连续点击发出重复请求。接收续页时同时核对请求所有权、首屏请求序号和页面范围;切换项目、账号、视图或同范围刷新后,旧结果与旧错误均不能写入当前页面。迟到请求只能释放自己的忙碌状态。
失败保留原列表与原游标,可以重试。去重追加、每次增加 200 条展示上限、服务端总量口径沿用现有行为。
## 关联查询的完整性
原非管理员关联路径只取前 1000 个任务 ID,并拼成一次 GET。本轮改为复用已有 `loadAnnotationTaskJoin`:关联已返回任务池中的全部去重 ID,每个请求最多 50 个 ID,每组最多 4 个并发。空列表不查询无关任务。
每批 HTTP 查询各自排序后,模块按原来的 `createdAt`、`id` 降序恢复整体顺序,避免批次拼接改变任务展示顺序。此处仅补齐当前已返回任务池的关联;没有新增全库扫描,也没有改变普通任务池端点既有的返回窗口或扩大用户可见范围。
## 验证
```sh
npm run test:annotation:pools
npm run test:browser:annotation-pools
npm run test:browser:annotation-persistence
npm run test:frontend:gate
```
前两项接入 Fanyu 本地发布检查。模块测试与页面实际回调测试共 49 项,覆盖三类池、验收双阶段、1221 个关联任务、跨批整体排序、权限错误、旧闭包、元数据失败、同范围刷新、重复点击、旧响应及失败重试。
任务池浏览器测试使用真实页面、Go HTTP 和隔离 MemoryStore 的 611 条合成任务,包含 9 个场景:分页与总量、失败重试、刷新时的迟到续页、重复点击、首批之外搜索、状态筛选、迟到搜索、已审核阶段以及合成领取后的计数更新。页面自身无写请求;唯一领取由测试夹具在隔离数据中执行。不会连接 Mongo 或 BOS。
浏览器与包含 Vite 的前端门禁需串行执行,避免共用工作树依赖缓存相互干扰。保留首次并行尝试的超时和错误断言日志,不把已经完成断言但清理超时的运行算作通过。最终串行结果:49 项定向测试、9 个任务池浏览器场景、18 个标注/返工页面场景通过;TypeScript、格式、生产构建、使用已安装 Edge 的 dist 启动与 37 项发布工具测试通过。全量前端门禁为既有 13 项失败、零新增,14 项基线未修改。两条过时的页面内部查询源码检查随职责迁移改为模块接线/行为验证。独立复核的旧加载闭包和跨批排序问题均已修正并补测,复核无剩余阻断项。
最终结果与日志位于忽略目录 `local-data/verification/pool-architecture-20261021/`。
## 兼容与后续
HTTP、Mongo、审核决定、交付 JSON 和后台任务协议均未修改,不需要数据迁移;回退前端会恢复原来的加载及分页行为。布局 agent 仍只接触展示配置,不接入任务可见性、领取、审核或交付判定。
审核会话、意见草稿和正式决定已经各有现有 hook。正式决定状态机及决定完成后的页面编排已在[第四阶段](review-decision-coordination.md)收敛;意见草稿、会话与连续审核保留各自职责。独立镜像发布、JSON v4 和实际 agent 尚未实现。

View File

@ -0,0 +1,91 @@
# 前端展示配置与业务职责分离
2026-10-21,基于已部署提交 `3221ac65`。用户选择同一仓库分层,先完成可定制布局基础。
## 本轮实现
React 已经通过 `src/services/api` 调用 Go HTTP;当前发布仍将 dist 与 API 放在同一镜像。本轮不搬仓库、不拆生产镜像,首先拆开看板查询流程和展示配置。
- `src/features/annotation-output/useAnnotationOutputData.ts` 集中筛选、查询竞态、统计快照、分页、供应商/人员下钻和导出。通过操作函数暴露业务意图,展示不直接维护快照或请求参数。
- `src/components/AnnotationOutputDashboard.tsx` 组装已有业务内容和交互,按当前账号身份提供内容。默认顺序和统计口径保持一致。
- `src/features/annotation-output/DashboardPresentation.tsx` 仅接受已授权的内容槽,负责顺序、密度、强调色以及预览/取消/保存/恢复默认。它不导入请求客户端、不获取任务数据,不根据配置生成业务按钮。
- `presentation.ts` 是展示文档唯一校验入口。复用已有 Zod,无新增依赖。读取不合法、未知版本或损坏文档时使用默认;保存失败明确报错,不伪称已保存。
- Go 的权限、统计快照和任务规则继续保持权威;业务请求和响应结构未更改。JSON v4 与标注保存/恢复不在本次范围。
这是第一条完整分层路径,不代表八千行标注页已经完成拆分。
## 展示文档 v1
```json
{
"version": 1,
"page": "annotation-output",
"order": ["metrics", "suppliers", "tasks"],
"density": "comfortable",
"accent": "orange"
}
```
三个区块必须各出现一次;个人看板没有供应商内容时不会因排序配置而生成该区块。density 仅 comfortable/compact;accent 仅 orange/blue/teal。固定的查询入口、加载和错误反馈不参与排序。节点使用稳定身份,布局调整不重建查询控制器、不清空分页或筛选;相同源下账号/组织/角色变化重建看板,清除上一身份的内存结果。
首版是此浏览器的个人偏好,按组织 ID、账号 ID、角色组成键隔离。无服务端保存、跨设备同步或组织统一模板;既有深浅主题及标注专注布局设置继续独立工作。配置不存客户数据。可用的本地存储不构成授权边界,所有业务访问仍由服务器检查。
## 后续 agent 如何接入
agent 将来生成上述受限展示文档,走同一校验、预览和显式保存路径;本轮没有创建或接入 agent。拒绝未知字段、JavaScript、HTML、任意 CSS、URL、请求端点、权限字段和统计口径,不用生成代码替换业务 UI。
未来若增加共享模板,应另外定义组织/角色作用域、服务端授权与严格校验、版本竞争、发布和回退;客户端校验不能代替服务器校验。当前只实现实际需要的浏览器存储,不预建插件系统或通用 agent 执行器。
## 后续重构顺序
1. 保留本轮展示 seam,扩展具体页面前先确定该页允许改变的内容与必须保留的业务控件。
2. 深化标注保存/恢复 module:统一 autosave 等待、版本检查、丢响应回读及本地状态采纳。保留草稿/正式工作区、ADR0007 的原子归档和撤销、ADR0010 的旧任务恢复。
3. 在该基础上统一拖拽与 E 面板的编辑会话,保持不同交互方式。
4. 需要独立发布时,再拆 Web 与 Go 构建产物;保留同源入口优先,并测试认证 cookie、CSRF、媒体地址、滚动升级和前后端契约兼容。无需先拆成两个仓库。
## 本地验证
发布门禁已接入 `npm run test:browser:dashboard`:自动启动独立端口的内存 API 与 Vite、运行下列回归、关闭自己启动的进程,证据保留在 `local-data/test-results/`。不依赖既有本地 Mongo。
配置校验测试:`node --test tests/frontend/dashboard-presentation.test.mjs`。
已有真实 HTTP 看板回归扩展了布局预览、取消、保存、刷新恢复、重置、零额外统计请求,以及原有分页、快照一致性、下钻和 CSV 导出。本轮使用内存隔离样本,既没有连接快照 Mongo,也没有连接生产:
```sh
go run ./tests/browser/support/annotation-output-replica -memory -web-port 5201 -api-port 3031
VITE_DEV_API_BASE_URL=http://127.0.0.1:3031 npm run dev:web -- --host 127.0.0.1 --port 5201 --strictPort
BROWSER_EXECUTABLE="/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge" npm run test:browser:annotation-output
```
本地测试账号随机生成于忽略目录 `local-data/verification/customer-dashboard-20261020/accounts.json`,不得提交。未传 `-memory` 时仍保留原有本地 Mongo 测试模式;须先确认该端口实际目标。本轮内存验证不宣称覆盖 Mongo 重启持久化。
相关依据:[React 状态保持与重置](https://react.dev/learn/preserving-and-resetting-state)、[Zod 严格对象校验](https://zod.dev/api)。
## 本轮验证结果(2026-10-21)
- TypeScript、格式、生产构建与生产 dist 浏览器启动通过。
- 展示文档校验 4 组测试通过,包括未知字段、缺失/重复区块、错误版本、账号作用域与存储失败。
- 三角色真实 HTTP + 内存样本共 33 项检查通过;同浏览器账号切换、损坏配置回退、保存失败、窄屏与 Escape 共 4 项补充检查通过,页面无运行时错误。
- 全量前端正式基线门禁通过:13 项历史失败,均在已有 14 项基线中,没有新增失败,未修改基线。首次受限运行在既有 devboot 本机监听测试停滞,中止后允许本机端口重跑通过。
- 发布工具 37 项测试通过;新的自动启动/清理测试命令完整运行通过,已列入发布策略。
- 独立应用代码复核未发现阻断问题。没有执行公司部署、客户任务写入或 PR 提交。
## 第二阶段实施
2026-10-21 已继续完成保存协调与共享切片编辑会话的本地改造,见[实现与兼容性记录](annotation-persistence.md)。第一阶段看板配置不扩展到任务写入。
## 2026-10-21 分层依赖自动检查
保存协调、切片编辑会话、任务池、正式决定和审核草稿的分层已经分别落地,见同目录的 `annotation-persistence.md`、`annotation-pool-loading.md`、`review-decision-coordination.md` 和 `review-draft-persistence.md`。本轮把已拆出的部分职责约束加入 `npm run lint:architecture`,并由 `lint:web` 执行;现有本地提交钩子与发布流程调用 `lint:web`,因此无需另设发布入口。
检查复用项目已安装的 TypeScript AST,不新增依赖:
- 看板展示层只允许现有展示文件与 React、图标、Zod 依赖;业务数据 hook、HTTP 客户端、其他页面、别名导入与其他包不在允许范围内。
- 五个已拆出的协议入口(标注保存、切片会话、审核草稿、正式决定、决定完成协调)递归检查本地 `src/lib` 帮助模块,拒绝 React、业务 HTTP、页面等依赖,避免通过间接帮助模块重新耦合。
- 检查静态 import、export-from、类型导入、动态 import、require 及 module.require;Vite glob 加载因无法逐一静态确认而阻断。协议模块内 JSX 的隐式 React 依赖也阻断。无法静态确定的模块加载、同名不同扩展造成的解析歧义、缺失文件、未解析依赖和语法错误阻断;报告具体文件、行号与依赖链。循环依赖遍历会终止,不因循环漏掉其他分支。
这是当前已分层模块的导入规则,不是整个仓库已完成分层,也不是恶意代码检测或运行时沙箱。直接使用浏览器全局、CSS 内容、回调中夹带业务行为等仍须代码审阅和行为回归;agent 继续只能提交受严格 schema 校验的展示文档。添加新依赖或扩大保护范围时,应根据实际职责调整规则和测试,不加通用豁免。
本轮无应用运行时代码、接口、Mongo、工作流、分类或交付格式变化,不涉及在途任务迁移。10 组回归覆盖正常依赖图以及违规直接/间接导入、动态加载、类型导入、循环、缺失文件、解析失败、扩展名歧义、glob 与隐式 JSX 依赖。独立复核指出的三处漏检已经修正并补用例。三角色看板真实 HTTP 与内存夹具的 33 项检查、4 项布局补充检查通过;全量前端仍为 13 项历史失败,零新增,14 项基线未改。发布工具、类型与格式检查通过。
本轮证据保存于工作树忽略目录 `local-data/verification/architecture-dependencies-20261021/`。未推送、未提交 PR、未部署。

View File

@ -0,0 +1,59 @@
# 正式审核决定与完成编排
2026-10-21,第四阶段。本轮在同一仓库提取前端审核决定状态机和决定完成后的页面编排,沿用现有 Go 回执、会话、草稿、连续审核及旧工作流协议。没有公司部署、客户业务数据写入或数据库迁移。
## 职责
`src/lib/review-decision-controller.js` 负责确认、取消、发送、待确认记录、回执查询和同一操作号重试。生产 adapter 使用现有 `reviewDecisionApi` 和浏览器存储;模块测试使用可控制的延迟和存储失败。`useReviewDecision` 通过 React `useSyncExternalStore` 订阅状态,并提供原有的调用接口。
正式成功仍以服务器 `completed` 回执为准。`prepared`、`applying` 和网络错误均不算完成,也不会自动重发正式决定。`failed_before_effects` 是确认无业务效果的终态,可以解除待确认保护;本轮同时覆盖发送后直接读到此状态的路径。已保存意见继续由原有草稿 hook 保管,不会因正式提交失败而重写。
发送前等待连续审核准备完成,然后取得当前会话头并保存恢复记录;关键恢复记录无法持久化时不发送正式请求。只有显式确认才发正式请求。重复确认、发送中的取消及重复重试不能发起第二个在途决定。已保存操作的重试保留原操作号,不被另一个待核对操作永久阻塞。
每次 controller 激活和页面范围变化都有独立归属身份。离开 A 再回到 A 时,即使存储键相同,旧请求也不能清除新记录、结束新请求的忙碌状态或触发旧恢复回调。生命周期清理只结束本地等待,不删除未知结果的恢复记录。
## 本地恢复记录与兼容
保留原键 `review-decision:<userId>:<projectId>:<stage>` 及其 JSON 格式,同时在同一键前缀下增加 `:operation:<encodedOperationId>` 记录。每条仍是原请求与操作号,不保存会话头或凭证。
新客户端扫描当前账号、项目和阶段的旧记录与操作记录并去重。旧记录可单独恢复;多个待核对操作逐个核对。发现别的未知操作时不接纳新的决定;已存在操作仍可按原操作号恢复或重试。逐操作记录保护两个页面近同时写入时的恢复证据,不依赖跨页面锁。清理只删除自己的操作记录和仍指向自己的旧键,不删除另一操作的记录。
逐操作记录必须写入成功。旧格式兼容记录是尽力写入:它因容量等原因写入失败时,新客户端仍可从已保存的操作记录恢复,不能因此把未发出的操作卡死。服务器已经确认终态后,本地清理失败不推翻终态;当前实例记住已核实终态,避免在本次页面生命周期中重复恢复,重新打开仍可安全查询残留回执。
回退限制:旧前端只识别旧键,不能恢复仅存在于新操作记录中的待确认决定。回退前应先用新版本核对所有待确认决定;不要通过清空浏览器记录处理未知提交。已经正式完成的服务器决定不因前端回退而撤销。部署前仍需按发布流程审查具体候选的兼容性与未完成操作。
## 完成后的编排
`src/lib/review-decision-completion.js` 收敛质检、批次验收和刷新后恢复三条完成路径:
1. 核对当前页面归属,再通知原有连续审核模块完成决定。
2. 等待后再次核对归属;已切换范围或离开后返回的旧操作停止更新页面。
3. 移除已审核条目;只有仍选中该条目时才清空对应输入、关闭任务并显示完成提示。
4. 刷新当前任务池。连续审核已经打开下一条时,保留下一条的选择和输入。
没有 human item 的旧任务审核保留原来的 task-only 分支,不伪造连续审核身份。质检和验收的统计字段、意见规则、通过/打回校验与服务端权限沿用原实现。质检保存状态也使用现有操作序号释放,避免旧决定结束时解除后续质检或验收决定的忙碌状态。
恢复回调中的队列或列表更新失败与正式决定分开处理:正式回执仍完成,不触发重复正式发送;提示核对后续状态,并通过现有失败信号暂停连续审核,避免未处理的 Promise 拒绝。
## 验证
```sh
npm run test:review:decisions
npm run test:browser:review-decisions
npm run test:frontend:gate
```
前两项接入 Fanyu 本地发布检查,现有 Go 定向门禁加入 `TestReviewDecision`。所有浏览器脚本串行运行。
- 43 项定向测试通过:32 项决定状态机、5 项完成编排、6 项页面实际审核回调。包括未知结果、失败终态、存储失败、旧格式、多个待确认操作、兼容键容量不足、终态清理失败、迟到响应、原意见保留及页面归属切换。
- 真实浏览器 hook + Go HTTP + 隔离 MemoryStore 的 4 个场景通过。实际提交一条合成打回决定,丢弃响应后刷新,再执行 A→B→A 和显式核对,最终只有一次正式请求和一条审核记录;React StrictMode 和真实确认对话框无页面错误。
- 现有质检/验收两角色的 8 组 500/409、失败终态、成功回执及未知结果回归通过;2 组连续审核绑定/正式发送竞态与续租回归通过。
- 完整标注/返工页面 18 个场景通过。实际 Go 回执重放、向前恢复、旧工作流和审核不可修改标注的定向回归通过。
- 全量前端门禁为原有 13 项失败、零新增,已有 14 项基线未修改。页面回调测试随归属身份改造更新夹具,并补当前失败可见、旧失败不污染及旧入口不保存草稿的行为断言。
- TypeScript、格式、生产构建、已安装 Edge 下的 dist 页面启动以及 37 项发布工具测试通过。
首次新浏览器夹具遗漏 JSX runtime 的依赖预构建项,已修正夹具并重新通过;失败尝试与最终日志均保存在忽略目录 `local-data/verification/review-architecture-20261021/`。新 HTTP 夹具只允许合成决定和回执核对,完全使用隔离 MemoryStore,不连接 Mongo 或 BOS。
## 范围
这是正式决定和完成路径的分层,尚未拆出整个标注页面。审核草稿的保存协议和输入保留已在[第五阶段](review-draft-persistence.md)进一步整理;审核会话、意见草稿、返工确认和连续审核各自的 hook 继续保留,布局 agent 仍只接触展示配置。独立前后端镜像、实际布局 agent、JSON v4 与生产发布不属于本轮。

View File

@ -0,0 +1,73 @@
# 审核草稿保存协议与编辑保留
2026-10-21,第五阶段。本轮拆出单条审核草稿操作的 HTTP 保存协议,补齐编辑队列与复审核对结论的保存衔接;没有部署公司环境。
## 职责
`src/lib/review-draft-persistence.js` 提供 `createReviewDraftPersistence(...).write(...)`,输入不可变的待确认操作、当前归属检查、会话头读取和已有的落盘证据判定,返回 applied、blocked 或 stale 结果。
它集中处理:未知结果的草稿与回执查询、沿用原操作号/载荷的显式重试、PUT 后读回、版本冲突后的读回、结果版本变化,以及哪些失败足以解除待确认保护。生产使用原有 HTTP wrapper,模块测试使用可控延迟和错误 adapter。模块不直接修改 React 状态,不负责编辑队列或焦点,也不自动重发写请求。
`useReviewDraft` 保留意见目标、文字编辑、旧多条意见整理、删除确认、串行队列、入队载荷、重定基线、播放定位、8 秒空闲保存调度及展示状态。它应用协议返回的结果,继续使用原来的 operationId / resultVersion / version 和服务器 CAS。
## 本轮修复
### 旧请求与输入保留
- 未知结果查询返回 404 后,必须再次检查任务/结果范围、会话代次和挂载状态,才能取会话头并发 PUT。旧查询不能借用新会话发送旧操作。
- 任务范围采用独立身份,离开后返回相同范围不复活旧回调;`saveAll` 和 `prepareSubmit` 每个异步阶段都检查原生命周期。挂载清理覆盖禁用分支,停止旧焦点、位置事件与保存回调。
- 被审结果变化会解除不能再应用的旧 pending,但保留后来输入或排队的文字;只有没有更新的本地文本时才恢复旧快照。
- PUT 明确被拒与 PUT 成功后读回失败分开分类。读回 403、网络失败或无充分证据时不能推断 PUT 没有效果;保留原操作等待核对。
- 新操作在取会话头前就失败时,没有未知写入需要保护,可以解除该 pending 并保留编辑;已有未知操作仍必须读回。
- 每个 flush 仍只释放自己的 busy 标记。代次变化不使旧请求卡死 busy,也不让旧请求释放新的在途保存。
### 复审核对结论
仅修改“已解决/仍有问题”和说明也立即算未保存,参与页面离开检查。`saveAll` / 离开保护中的保存同时处理文字与结论;明确放弃会恢复已保存结论。正式决定前继续要求“仍有问题”填写说明,服务器仍是最终校验者。
结论比较按服务器的 ID/说明去空白、空说明可省略的语义处理;保留输入次序、不合并不同判断、不重写本地正文。避免服务器省略空 note 或裁剪说明空白后,页面永远判为未保存。
刷新读取和同结果的新会话初始化,保留尚未保存的本地结论;没有本地差异时采纳服务器值。即使本地选择改回旧基线,也会比较在途载荷,保留发出请求后的改判,不用“等于旧基线”推断用户没有新操作。读取期间仍允许继续输入当前任务的文字。不同任务或不同被审结果不会复用旧结论。
## 兼容性
没有改变 HTTP、Mongo、服务器操作回执、审核权限、原始标注、冻结返工基线或交付 JSON。没有新增依赖或浏览器存储格式。旧多条意见、历史整体意见与原有人工确认路径保留;ADR0008 的旧问题延续规则和 ADR0009 的停用整体意见继续有效。
后台已经在途的草稿操作沿用原操作号和载荷,先查询再决定是否重试,不把新文字塞入旧操作。无需数据迁移;前端回退会恢复原来的编辑保留和离开检查行为,已经保存的草稿仍按原协议读取。
## 验证
```sh
npm run test:review:drafts
npm run test:browser:review-drafts
npm run test:frontend:gate
```
前两项接入 Fanyu 本地发布检查;服务器定向检查补入草稿及复审核对结论测试。包含 Vite 的测试串行执行。
- 52 项定向测试通过,覆盖单操作保存/读回、未知结果、回执终态、404 后失权、冲突、PUT 与读回错误区分、结论比较、空闲调度及历史兼容接线。
- 质检、验收各 14 个真实 HTTP + React hook 场景通过,共 28 个,全部使用隔离 MemoryStore。覆盖结论离开保护、明确放弃、只保存结论、服务器省略/裁剪说明、旧成功或失败响应后的新输入、刷新期间输入、刷新保留结论、丢响应无重复 PUT、冲突后的明确重试、代次变化及卸载。
- 代次变化在浏览器夹具中模拟客户端收到新会话快照;实际服务器 takeover fencing、进程崩溃恢复、草稿目标/来源隔离及结论冻结由原有定向 Go 回归验证。
- 正式决定的真实回执及两角色失败/连续审查回归通过;完整标注/返工 18 个场景在单独复跑中通过;首次运行在 E 面板第三次场景等待冲突清单时超时,无页面异常,触发条件尚未定位,保留为测试不稳定性待查,不能把这次失败计为通过。
- 全量前端门禁仍为已有 13 项失败、零新增,14 项基线未修改。一条旧测试硬匹配原内联错误前缀的断言已随协议提取改为结果接线,并由真实行为回归补强。
- TypeScript、格式、生产构建、已安装 Edge 的 dist 启动和 37 项发布工具测试通过。
首次验收浏览器夹具误用了质检的切片/来源身份,被现有服务器 400 拒绝;已改用验收聚合来源 ID 和相对时间,再次完整通过。新增代次变更夹具也改为等待真实 GET 完成后再保存,避免把加载未结束时的拒绝保存当作业务失败。失败尝试及最终记录保存在忽略目录 `local-data/verification/review-draft-architecture-20261021/`。夹具只允许当前合成草稿的 PUT,禁止正式决定等其他写入,不连接客户 Mongo、BOS 或公司服务。
## 后续
本轮只提取单操作保存协议,编辑队列与导航仍在 hook 中,没有把整个 hook 搬进新类。后续可按可验证的业务流继续收敛队列与展示职责;布局 agent 继续只使用展示配置,不接入保存或审核判定。独立镜像、实际 agent 与生产发布均不在本轮范围内。
## 2026-10-22 本轮批注集中查看
依据平台调优中王子予 2026-10-21 17:53 的消息与截图,恢复右侧全部批注列表,保留上方当前切片/动作/时间点的编辑卡片。列表直接读取当前任务、当前审查轮次已加载的草稿,不随当前切片筛掉其他批注,也不新增查询、轮询或后台统计。未保存文字继续由原编辑器与未保存提示处理,不把它伪装成已经保存的批注。
列表显示切片、动作、时间点批注的位置与正文,点击复用原意见选择和定位流程;验收使用原有来源到可见切片的映射,避免不同来源使用相同切片 ID 时定位串源。只读时仍可回看,编辑和保存继续受已有会话与服务端校验保护。切换条目时已有未保存文字按原 park/保存协调处理,不新增写入路径;无编辑变化的列表定位不发送保存。
ADR0008 的未解决退回基线和 ADR0009 的隐藏整体意见保持不变。列表不复活历史已解决意见,不改变分类、原标注、审核结论、接口、数据库或交付 JSON,无需迁移。
验证在真实 Go HTTP + 隔离 MemoryStore 中执行,质检与验收共 34 场景通过,包括跨切片/来源同时展示、定位、只读、单条清除后的刷新和其他批注保留,以及原有丢响应、版本冲突、新输入保留与代次切换。测试生成文件曾触发 Vite 热刷新;现将夹具的 `local-data` 排除出监听,失败日志保留,重跑通过。另一次夹具遗漏查看控制器的 focus 实现已补齐,不作为产品缺陷。
当前优化版的 168 项保存/任务池/正式决定/草稿定向测试、19 个完整标注浏览器场景、9 个任务池场景、正式决定及权限回归通过。全量前端仍为 13 项原有失败,基线 14 项未修改。证据保存在本工作树忽略目录 `local-data/verification/wangziyu-review-list-20261021/`。
本轮只在个人分支本地验证,尚未部署。

View File

@ -0,0 +1,68 @@
# 审核追溯、连续领取与源校验等待(2026-10-26)
## 已实现的边界
- 连续领取在存储分页前限定 receipt 协议支持的 `quality_review` / `prepackage_acceptance` 阶段。领取顺序、身份权限、原页面栅栏、待完成领取记录、原子领取与恢复协议不变。旧存储调用不传阶段时保持原语义。
- 追溯接口只读取当前授权任务的审计元数据,排除意见正文、附件和标注 payload;不写客户历史。普通列表显式 `trace=1` 才做批量摘要,最多三次额外查询,与任务条数无关;摘要有两秒预算,失败时仍返回任务列表。专注模式与领取接口不附带列表摘要。
- 每次切换任务取消旧追溯请求;源信息与历史结果按任务及版本匹配,迟到响应不得替换新任务。缺失数据为 `-`,已知零次为 `0`。归档事件若没有创建起点,不推算首次提交或总退回次数。
- 源校验的等待状态从持久化校验队列读取,区分 `waiting_source` 与实际 `validating`。旧前端遇到新增进度值使用原有兜底文本;不改变数据库中 processing job 的状态协议。
- 校验完成记录增加耗时、尝试次数、完成写入是否成功的日志;结束写入限时 15 秒,仍使用原 owner/status 栅栏。未成功写入可由原租约恢复协议接续。
## 只读排查结论与限制
连续领取的最小复现:350 条更早的标注阶段任务排在一条可领取质检任务之前。修复前读取四页批次,修复后只读取一页;原有并发领取、暂停及响应丢失测试均保留。尚未取得客户该次请求的完整 HTTP 耗时分解,不能声称它是所有等待时间的唯一原因,也没有给出线上改善百分比。
切片录屏已匹配到具体任务。持久化记录显示该次请求 14:46:18 开始、15:07:58 完成,等待源校验 25 次,源校验本身尝试一次;没有人工重试或代替用户写入。客户提到的更早 40 分钟等待不在该次请求完整证据中,不能混作同一计时区间。
只读采样观察到多个源任务未领取,随后领取者持续处理下一批任务;两台 validator 各配置一个校验槽。进程正在 `ffprobe -show_frames` 解码,采样没有 CPU 限流。这证实排队容量问题,未发现该任务死锁或无限重试的证据。状态展示与日志改动本身不会提高处理容量。
## 本次解码优化(待发布)
完整逐帧解码从 1 线程调整为 2 线程,保留原整数帧时间戳、帧数、左右目和 IMU 校验以及持久化证书复用。源索引格式、任务状态及租约协议不变。异常输出超过扫描上限时立即终止子进程,避免堵住 stdout 占用校验槽;不积累未使用的 stderr。增加 fetch / media_clock / sensor_index / pin 各阶段耗时日志,日志不含源路径或客户正文。
本机已下载的三段真实 480p 视频,完整解码 6343 / 7957 / 11239 帧,对比全部时间戳输出 SHA256 均一致:
| 并行任务 / 每任务解码线程 | 三段总耗时 | 单进程峰值 RSS |
| --- | --- | --- |
| 1 / 1 | 24.709 秒 | 约 21.3 MB |
| 1 / 2 | 16.798 秒 | 约 22.4 MB |
| 2 / 1 | 17.810 秒 | 约 21.4 MB |
| 2 / 2 | 12.771 秒 | 约 23.1 MB |
这是本机解码阶段测试,不能推算 BOS 下载、固定源复制或线上整体等待时间。本次先保留线上每副本 1 个任务槽、2 副本及原资源限制,仅提高单任务解码并行度。增加任务槽会同时增加下载、临时磁盘与源复制负载;最慢冷源及 scratch 峰值尚未充分测量,因此不随本次启用。
## 后续容量调整候选(未执行)
| 项目 | 当前 | 候选 |
| --- | --- | --- |
| validator 副本数 | 2 | 2 |
| 每副本 ANNOTATION_INDEX_CONCURRENCY | 1 | 2 |
| 总校验槽 | 2 | 4 |
| 每副本 CPU / 内存上限 | 4 CPU / 8 GiB | 不变 |
| 自动历史回填 | 关闭 | 关闭 |
后续候选只调整 validator 的任务并发,保持完整逐帧时钟和 IMU 校验;不跳过校验、不改变 source certificate 格式、不清队列、不批量重试客户任务。既有 source-index-100 配置文件保持原值,避免后续通用部署默默启用候选。
本轮合成测试验证恢复及并发正确性,不能替代真实分辨率、编码、文件体积与 BOS 带宽的冷源容量测试。容量启用前必须取得代表性负载证据(包含最慢源文件、临时磁盘峰值、BOS 读取错误及排队时长),复核 scratch 空间与资源余量。并发增加不保证吞吐线性增长;若资源或错误率恶化,恢复每副本 1 个槽,保留队列和已固定的源证书,由租约恢复接续。
这是独立配置变更方案;上线需按仓库发布规则,在执行前取得该方案的明确批准。该文件不是部署授权。
## 本地验证
- `npm ci --ignore-scripts --offline`、`npx tsc --noEmit`、`npm run build:web`。
- 接口与存储:`go test ./internal/api ./internal/store -run 'Test(ReviewTrace|SourceValidation|ReviewContinuation|SourceIndexJob)' -count=1`。
- 竞态:`go test -race ./internal/api ./internal/store -run 'Test(ReviewTrace|ReviewContinuation|SourceIndex.*(Cold|Recovery|Priority)|SourceValidation|.*Validation.*Wait|HumanTrace)' -count=1`。
- 连续领取重复 50 次;源校验等待退避、1 小时上限、失效 owner 隔离与恢复有既有回归覆盖。
- Mongo:本机 27018 独立合成测试库,验证批量摘要的零值/未知、租户隔离、元数据投影和源任务优先级。首次本地执行因副本容器文件句柄耗尽而失败;已恢复原容器,缩小该只读测试的无关索引初始化后重跑。
- 54 项相关前端状态及加载测试。
- `REVIEW_TRACE_BROWSER_TEST=1 go test ./internal/api -run '^TestReviewTraceBrowserReplica$' -count=1 -v`:实际页面与隔离 MemoryStore/API,质检及最终验收各验证普通/专注模式、位置、折叠、完整记录、Escape 关闭;无页面 JS 错误。该夹具没有真实业务视频,不作为播放质量或线上业务验收证据。
没有执行客户数据写测试、推送或部署。上线后的吞吐改善及长时运行情况仍需按授权范围验收,不能以本地回归替代。
## 发布后只读采样(2026-10-26 17:30,`025896ae` 已上线)
- 切片物化(processor,`ANNOTATION_PROCESSOR_ROLE=revision`,2 副本 × 并发 8,2 CPU / 4 GiB):`materialization_timing` 全部 `queue_ms=0`、`total_ms` 0.2–2.3 秒,空闲 CPU 约 30m。当前不是瓶颈。
- 源校验(validator,2 副本 × `ANNOTATION_INDEX_CONCURRENCY=1`,4 CPU / 8 GiB):上线后 42 分钟两副本无空档连续处理,共 90 个任务(约 2.1 个/分钟);单任务 p50 约 55 秒、p90 约 87–103 秒、最大 110 秒,其中 `media_clock`(逐帧解码)约占 75–80%,`pin` 5–13 秒,`fetch` 1–6 秒。运行中的副本约 1.7 CPU,未触顶 4 CPU 上限。队列饱和,等待时间随同时开新任务的人数线性累积。
- 节点:`10.189.16.33` 可分配 54.6 CPU / 208 GiB,requests 仅 15%;`10.189.16.5`(7.8 CPU)limits 已超分 185%,其中一台 validator 在该节点。
据此,增加资源应投向 validator,而不是 processor 或 API。分阶段扩容方案见 [validator-scale-20261026.md](validator-scale-20261026.md)(未执行,需单独批准)。

View File

@ -3,7 +3,7 @@
"profile": "fanyu-source-index-100-v1",
"namespace": "droboticflow-fanyu",
"budget": {
"cpu": 36,
"cpu": 54,
"memoryBytes": 120000000000
},
"deployments": {
@ -47,20 +47,20 @@
},
"droboticflow-annotation-validator": {
"container": "processor",
"replicas": 2,
"replicas": 4,
"requests": {
"cpu": "2",
"cpu": "6",
"memory": "4Gi"
},
"limits": {
"cpu": "4",
"cpu": "8",
"memory": "8Gi"
},
"env": {
"ANNOTATION_SOURCE_INDEX": "true",
"ANNOTATION_PROCESSOR_ROLE": "validation",
"ANNOTATION_PROCESSOR_CONCURRENCY": "1",
"ANNOTATION_INDEX_CONCURRENCY": "1",
"ANNOTATION_INDEX_CONCURRENCY": "4",
"ANNOTATION_INDEX_BACKFILL": "0",
"DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE": "striped",
"DROBOTICFLOW_DATABASE_BOOTSTRAP": "verify-existing"
@ -106,7 +106,8 @@
},
"constraints": [
"Offline rendering only; the baseline was deployed on 2026-10-17, but future plans still require separate deployment authorization.",
"Six CPU cores and remaining memory stay unallocated after the proposed steady services and legacy-export budget.",
"2026-10-26 user-approved validator scale-up (validator-scale-20261026.md): 4 x 4 validation slots, 8 CPU limit per replica, preferred node 10.189.16.33. Resident limits budget raised to 48 CPU; budget.cpu 54 includes the 6 CPU export reservation.",
"No CPU spare remains after the resident 48 CPU and the 6 CPU legacy-export reservation; memory keeps its remaining headroom.",
"Operator Job Pods have independent resource specifications. The export allowance is a planning reservation, not a namespace quota or implemented global admission limit.",
"Do not enable source-index output until API, revision workers, validators, workflow workers and the package operator all understand the new schemas.",
"Mongo and BOS are external; this budget does not expand their capacity or network bandwidth.",

View File

@ -4,19 +4,19 @@
## 资源与并行
预算按 36 CPU、120,000,000,000 字节计算。配置权威源为 `source-index-100.json`。
预算按 54 CPU(常驻 48 + 导出预留 6,2026-10-26 经用户批准由 36 上调)、120,000,000,000 字节计算。配置权威源为 `source-index-100.json`。
| 组件 | 副本 × 单副本 limit | 总上限 | 用途 |
| --- | --- | --- | --- |
| API | 4 × 2 CPU / 4 GiB | 8 CPU / 16 GiB | 登录、列表、保存、任务状态 |
| 切片修订 | 2 × 2 CPU / 4 GiB | 4 CPU / 8 GiB | 每副本 8 路,共 16 路 JSON 修订 |
| 独立源校验 | 2 × 4 CPU / 8 GiB | 8 CPU / 16 GiB | 每副本 1 路完整校验,共 2 路 |
| 独立源校验 | 4 × 8 CPU / 8 GiB | 32 CPU / 32 GiB | 每副本 4 路完整校验,共 16 路(2026-10-26 扩容,见 validator-scale-20261026.md) |
| Worker | 2 × 1 CPU / 8 GiB | 2 CPU / 16 GiB | 调度、交付发布 |
| Coordinator | 2 × 1 CPU / 2 GiB | 2 CPU / 4 GiB | 同步协调 |
| 常驻合计 | | 24 CPU / 60 GiB | |
| 常驻合计 | | 48 CPU / 76 GiB | |
| 旧媒体导出规划预留 | | 6 CPU / 12 GiB | 独立 Job;尚非强制准入上限 |
余下 6 CPU 与约 42.7 GB 留作余量。各算子 Job 的资源单独定义;该表不是命名空间配额,也不代表已有 GPU/normalize/export Jobs 已受 36 CPU 总限额约束。未改变它们的配置。
CPU 无余量,内存余约 25.5 GB。各算子 Job 的资源单独定义;该表不是命名空间配额,也不代表已有 GPU/normalize/export Jobs 已受 36 CPU 总限额约束。未改变它们的配置。
提高切片并行只针对完成源校验后的轻量修订。冷源排入独立验证队列,不能把 16 路全部用于解码。更多副本不解除 Mongo 项目写锁,也不增加外部 Mongo、BOS 或客户端网络容量。

View File

@ -0,0 +1,99 @@
# 源校验 validator 扩容:100+ 人并发(2026-10-26,已批准并执行)
用户于 2026-10-26 批准本方案及常驻 CPU 预算上调到 48,并已执行,记录见文末。后续再改仍需单独批准。`scripts/release.py` 只覆盖 API / processor 同镜像更新,不检查也不修改 validator;本变更按 `docs/release/README.md` 单独审查,执行前须在紧邻部署时取得用户明确批准。
## 需求实测(只读,Mongo `annotationIndexJobs` / `humanEvents`,UTC 小时)
| 时段(北京时间) | 活跃用户 | 新校验任务 / 小时 | 切片确认 / 小时 |
| --- | ---: | ---: | ---: |
| 10-26 09:00 开工 | 30 | 106 | 115 |
| 10-26 14:00 | 58 | 101 | 110 |
| 10-26 15:00 | 67 | 108 | 116 |
| 10-26 16:00 | 65 | 76 | 148 |
| 10-24 15:00 | 20 | 60 | 63 |
- 校验任务在切片确认时按需创建(每小时新任务数 ≈ 确认数),没有提前预计算。每次确认至少等待一次完整校验(p50 约 55 秒,p90 约 100 秒)。
- 当前容量:2 槽 ≈ 2.1 个/分钟。今天 15:00 的需求约 1.8 个/分钟,利用率约 86%,排队时间迅速拉长。18:54 采样时队列已清空(0 个排队),历史 1901 个完成、2 个失败。
- 人均需求:常态约 1.6 个/人/小时,开工高峰约 3.5 个/人/小时。
## 100+ 人容量估算
| 场景 | 需求 | 所需并发(≈需求×1 分钟) | 16 槽利用率 |
| --- | ---: | ---: | ---: |
| 100 人常态 | 160 个/小时 ≈ 2.7/分钟 | 2.7 | ≈17% |
| 100 人开工高峰 | 350 个/小时 ≈ 5.8/分钟 | 5.8 | ≈36% |
| 150 人开工高峰 | 525 个/小时 ≈ 8.8/分钟 | 8.8 | ≈55% |
按利用率 ≤ 50–60% 留出突发余量,选 **16 槽**:4 副本 × 每副本 4 槽(代码上限每副本 4)。在这个负载下,排队时间应接近 0,用户等待约等于单任务校验时间(约 1 分钟)。扩容不会让单任务变快,消除这 1 分钟需要改代码(见文末)。
## 变更内容(仅 `droboticflow-annotation-validator`)
| 项 | 现在 | 变更后 |
| --- | --- | --- |
| 副本 | 2 | **4** |
| 每副本 `ANNOTATION_INDEX_CONCURRENCY` | 1 | **4** |
| 总校验槽 | 2 | **16** |
| requests CPU / 内存 | 2 / 4Gi | **6** / 4Gi |
| limits CPU / 内存 | 4 / 8Gi | **8** / 8Gi |
| nodeAffinity | 无 | **优先** `10.189.16.33`(weight 100) |
| scratch emptyDir | 20Gi | 不变 |
取值理由:
- **CPU**:每槽 1 路校验 × 2 个 ffprobe 线程,实测约 1.7 CPU/槽,4 槽约 7 CPU。limit 设为 8;request 设为 6,保证调度时预留实际用量。
- **内存不加**:实测 RSS 130–230 MiB/副本,4 槽也不到 1 GiB,8Gi 足够。
- **scratch 不加**:实测 0.4–0.5 GiB/槽,4 槽约 2 GiB,远低于 20Gi。
- **节点**:`10.189.16.33` 可分配 54.6 CPU,当前 requests 8.3 CPU。4 副本都落在该节点时,requests 约 8.3 − 2 + 24 ≈ 30 CPU(55%)。`10.189.16.5` 的 limits 已超分 185%,无法放下 6 CPU request,`10.189.16.4` 最多放一个。用 `preferred` 而不用 `required`,是为了避免单节点故障时 validator 全部无法调度。
- **预算**:`source-index-100.json` 原有 36 CPU 常驻 limits 预算。validator 由 8 增至 32,合计约 48 CPU,超出原预算。集群可分配约 70 CPU,节点 requests 仍有余量,因此需要用户批准把常驻预算上调到 48 CPU,执行后同步更新 JSON 与 md。
- **其他组件不扩**:切片修订 processor 已有 16 槽,单次 0.2–2.3 秒,峰值每小时 148 次确认,远未饱和;API 4×2 CPU 实测每副本 0.1–0.2 CPU。100 人另需的 API / 视频带宽门槛见 `source-index-100.md`,不属于本变更。
补丁(strategic merge,按 `name` 合并 env,不触碰其他字段):
- 变更:`k8s/fanyu-validator-scale.patch.yaml`
- 回退:`k8s/fanyu-validator-scale.rollback.patch.yaml`
已用 `kubectl patch --local` 对线上快照离线渲染:只改变副本、亲和、`ANNOTATION_INDEX_CONCURRENCY` 和 CPU requests/limits;回退后与原 spec 完全一致。快照(uid `38deaddf-6801-4a8d-95c8-6e561f95b66c`,resourceVersion `29531042`,generation 6)及渲染结果在未跟踪的 `local-data/releases/validator-scale-20261026/`。
## 中断与兼容
- RollingUpdate(maxSurge 0 / maxUnavailable 1):逐个替换,滚动期间少一个副本。宜在低峰期执行(18:54 采样时 13 人在线、队列为空)。
- 被终止副本上的在途校验:优雅退出时回到队列,一分钟后重试;进程被杀时,90 秒认领租约过期后由其他副本接续。每个被中断的任务多消耗一次尝试(上限 5 次),不会丢失或重复写入源证书。
- 不改镜像、ConfigMap、Secret、PVC、Service;不清队列、不批量重试、不开历史回填。API / processor / Worker 不动,后续 `release.py` 镜像发布不受影响。
## 执行步骤
前置(只读):context 与 namespace uid(`d6a90ea7-b57d-4c52-b90b-b6c976cba076`)一致;Deployment resourceVersion 仍为 `29531042`,否则重新快照并重新离线渲染;validator Ready 2/2;另存一份备份 YAML。
执行(经批准):
```bash
kubectl -n droboticflow-fanyu patch deploy droboticflow-annotation-validator --type strategic --patch-file k8s/fanyu-validator-scale.patch.yaml
kubectl -n droboticflow-fanyu rollout status deploy/droboticflow-annotation-validator --timeout=15m
kubectl -n droboticflow-fanyu get pods -l app=droboticflow-annotation-validator -o wide
```
部署后核验(只读):
- 4/4 Ready、无重启 / OOMKilled,新 Pod 落在 `10.189.16.33`,日志无启动错误(`annotation index concurrency must be 1..4` 等)。
- 有负载时:`[annotation-validator] job=` 无新增 `status=failed`、无 BOS 读取错误;每副本 CPU < 7.5,内存 < 2Gi,`/tmp` < 8Gi;`source_timing` p50 增幅 ≤ 30%(≤ 72 秒),排队为 0 或接近 0。
- 任一项恶化:执行回退,或在批准后把并发降为每副本 2。
- 业务验收(下一个工作日开工高峰):多人同时确认切片时,“等待源数据校验资源”状态应基本消失,由用户确认。Pod Ready 不等于业务验收通过。
## 回退
```bash
kubectl -n droboticflow-fanyu patch deploy droboticflow-annotation-validator --type strategic --patch-file k8s/fanyu-validator-scale.rollback.patch.yaml
kubectl -n droboticflow-fanyu rollout status deploy/droboticflow-annotation-validator --timeout=15m
```
回退会恢复 2 副本、每副本 1 槽、2/4 CPU、无亲和。源证书与队列保留,在途任务由租约接续。不要整份 `apply` 旧快照(会带上过期的 resourceVersion 和 status)。
## 未验证与风险
- 16 路并发下的 BOS 读取与 pin 拷贝没有实测:单任务 fetch 1–6 秒、pin 5–13 秒,按 16 路估算仍在合理范围,需要用部署后的日志确认。
- `10.189.16.33` 同时运行 GPU 与导出类算子 Job,大批 Job 并发时会争用 CPU。limits 只是上限,不是独占资源。
- 容量估算基于 10-24 与 10-26 两天的数据,人均需求可能随项目视频长度变化。
- **扩容只消除排队,每次确认仍要等约 1 分钟**。要让用户几乎不等,下一步需要改代码:在领取或打开任务时就提前排入校验(标注期间完成),并且左右目并行解码。这项改动需另行评审。
## 执行记录(2026-10-26 19:06 北京时间)
- 前置只读检查:context `2c1284bae51b4bf3a2e623557844c6d6@kubernetes`、namespace uid、Deployment uid 和 resourceVersion `29531042` 都与快照一致;2/2 Ready、无重启;执行前 10 分钟没有校验任务,队列为空。执行前另存备份 `before-20261026T190545.yaml`。
- 执行:补丁附带 `resourceVersion` 前置条件(期间若有他人修改会被拒绝),滚动约 2 分钟完成,generation 6→7,resourceVersion `29573013`。
- 核验:4/4 Ready、0 重启,全部落在 `10.189.16.33`;线上 spec 与离线渲染完全一致;镜像未变(`sha256:b7655059…`);启动日志正常,无 Warning/OOM 事件;API 4/4、processor 2/2、Worker 2/2 未受影响。节点 `10.189.16.33` 的 requests 8.3→30.3 CPU(55%),与估算一致。队列:1901 个完成、2 个历史失败、0 个排队。
- 同步:`source-index-100.json` / `.md` 的 validator 条目与预算(`budget.cpu` 54 = 常驻 48 + 导出预留 6),`tests/release` 39 项通过。
- **尚未验证**:执行时没有负载,16 路并发下的吞吐、单任务耗时、CPU 与 BOS 表现需要在下一个工作日开工高峰(09:00 起)按上文“部署后核验”取证;业务验收由用户确认。

View File

@ -0,0 +1,113 @@
# 看板持续运行与故障恢复
状态:2026-10-26 已按本聊天明确批准的计划部署到 Fanyu,并完成看板只读验收。线上源码提交 `521acce5`,基于原线上 `44e66a68` 与第一轮修复 `8dff3517`。本页补充并更新[初步调查报告](verification/dashboard-stability-20261026/report.md);整个平台的生产写入及故障注入验收仍未完成。
## 本轮落实的保障
统计口径、历史作者/供应商、三种旧 run 关联方式、15 分钟最大陈旧时间和角色权限保持原契约。没有放宽超时/容量上限,没有截断历史后返回“成功”,没有清理原始历史。
- 关联查询 ID 去重并分批(每批最多 256),避免查询表达式随累计任务数无限增长;run 并集按 ID 去重。
- 人员工作事件仅加载解释器使用的三种送检事件;humanEvents 仅加载终审证据。阶段别名 `acceptance`、`prepackage_acceptance`、`final_acceptance` 全部保留。筛选集合由统计模块提供,存储层与解释器共用,避免两份规则漂移。
- 保存、领取等非统计过程历史不再占用返回载荷预算。相关复合索引使数据库可按报告所需键定位,原索引保留以服务其他业务。
- 失败次数、最近失败阶段/类别/时间/耗时保存在现有共享协调记录中;刷新重试不能抹掉失败证据,成功发布才清除活跃失败。旧租约 owner 不能增加新 owner 的失败次数或释放其租约。
- 成功代次附带构建准备耗时(读取、计算与编码,不含发布写入)、压缩/解压体积、part 数和来源 BSON 字节/文档数。序列化仅一次;没有为监控复制一份完整数据集。
- 构建时间或任一载荷达到保护预算的 80% 时生成 `output_snapshot_capacity_warning` 日志;刷新失败继续有阶段化错误日志。失败计数及容量指标也能通过下述只读接口查看。
- 发布策略强制运行真实本地 Mongo 的规模、兼容、权限与恢复门禁。测试跳过、缺失或运行器失败均不能算通过。
## 只读健康检查
`GET /api/dashboard/annotation-output/health` 仅允许组织管理员。使用当前组织与当前正式统计项目范围,不接受查询参数扩权,不返回人员/任务正文、其他组织或项目 ID、原始错误文本、数据库地址、凭据。
这个接口只读取配置及协调元数据;**不会申请租约、触发重建、续租或写入统计集合**。`Cache-Control: no-store`。它说明快照协调状态,不校验压缩 blob 完整性,也不替代一次真实看板读取。
HTTP 200 只说明检查成功,必须解析 `status` 与 `available`:
| status | 含义 | 检查脚本退出码 |
| --- | --- | --- |
| healthy | 代次未过期、无已知失败或容量预警 | 0 |
| idle | 无已知刷新失败,旧代次已过期;原设计无访问不刷新 | 0(available=false,不能当作业务读成功) |
| degraded | 有刷新失败或容量预警,可能仍能服务旧的有效代次 | 1 |
| unavailable | 已知刷新失败,且没有有效代次 | 2 |
| not_built / building | 尚无可验证的已完成代次 | 3,状态未知 |
数据库不可达、401/403、重定向、响应格式错误及未知状态均退出 3,不伪装成健康。脚本只进行一次 GET,不自动登录、轮询或刷新:
```sh
python3 scripts/check_dashboard_health.py \
--url https://<platform-origin> \
--cookie-jar /absolute/path/to/existing-admin-cookies.txt
```
Cookie 文件为既有的 Netscape 格式会话文件,应按凭据文件保护。会话过期必须由既定认证流程处理,检查器不创建账号、扩大权限或自动写登录记录。没有新增未认证的监控入口。
建议将退出码 1/2/3 接到现有监控,保留失败原因与快照年龄;日志系统可匹配 `output_snapshot_build_failed`、`output_snapshot_lease_renew_failed`、`output_snapshot_cleanup_failed`、`output_snapshot_capacity_warning`。本次提供了接口、日志及检查器,**未创建线上监控任务、修改告警配置或向外发送通知**。告警接入需确定现有监控系统和运行身份。
## 存储、索引与兼容
内部快照的 schema 仍为 `annotation-output-facts/v1`,统计 payload 未变。仅向既有协调记录/代次元数据添加可选字段:consecutiveFailures、lastFailure、metrics、activeKey、activeEpoch;无新集合或业务文档迁移。
旧记录缺少指标时显示未知,而非零;旧 failed 标志按“至少一次失败”处理。混合版本运行时,旧写入者可能覆盖新增可选诊断元数据,因此完整故障计数保证以 API 全部升级为前提。activeEpoch 与 epoch 配对,旧写入者沿用过期 activeKey 时不会误报当前刷新者。旧应用可继续读取快照格式,回退不需要回滚客户数据。
候选启动的 EnsureIndexes 会尝试创建以下三个非唯一索引,均需在生产变更计划中列出并单独授权:
| 集合 | 名称 | 键顺序 |
| --- | --- | --- |
| runs | organizationId_taskId | organizationId, taskId |
| humanEvents | organizationId_itemId_stageId | organizationId, itemId, stageId |
| annotationWorklogEvents | organizationId_projectId_eventType | organizationId, projectId, eventType |
不删除/修改现有索引。部署前宜受控预建、检查名称与键一致并观察数据库负载,不能把索引创建当作无影响写入。回退时索引可保留。发布流程仍按 `docs/release/README.md` 执行,新的策略摘要会使旧发布计划失效,必须重新准备。
## 自动验证与证据
```sh
python3 scripts/test_dashboard_stability.py
python3 -m unittest discover -s tests/release -p 'test_*.py'
go build ./...
```
稳定性门禁固定连接 `mongodb://127.0.0.1:27018`,由测试创建/清理独立合成数据库,不能传入生产 URI。运行前应核实端口确实属于本机副本。缺少本地 Mongo、关键测试 skipped、测试名称消失、超时或 race 失败均阻止门禁通过。
已验证:
- 原查询超过关联批次上限的回归先失败后通过;600 与 24000 条/来源均保留所有相关事实、旧关联及组织隔离。
- 200 条非统计过程事件曾使小预算查询失败;筛选后通过,三种终审别名完整保留。
- 共享快照、多实例、重启、遗弃租约、旧 owner、失败退避、损坏代次、私有集合权限,以及新持久化故障状态与旧记录兼容。
- 管理员诊断权限及范围撤销;诊断本身不触发来源加载。路由快照从真实路由生成,仅新增上述管理员 GET。
- 完整门禁在 race 下通过,无 skipped;24,000 条/来源规模读取约 1.82 秒(本机、并发测试、race 开启,不能当作生产容量承诺)。既有真实 HTTP 快照规模场景同时通过。
- 39 项发布工具测试通过;`go build ./...` 通过。
- 独立本地副本真实 HTTP:先登录合成管理员,检查器返回 not_built / 退出 3;读取报表返回 200;随后检查器返回 healthy、available=true / 退出 0。验证后停止副本。
原始本地 HTTP 证据在本分支工作树的 `local-data/verification/dashboard-stability-20261026/`,生成的账号/会话均为本地合成、Git 忽略。未使用生产客户数据写测试。生产复验结果见下节;本地测试不能替代其他未执行的生产业务验收。
## 线上发布与验收(2026-10-26)
已取得本聊天用户对三个索引、具体镜像发布与只读测试的明确批准。发布记录:
`local-data/releases/dashboard-stability-20261026-521acce5/release.json`;计划摘要
`4a8fd68a09c970cfa23eb76ae8bebcf20364ef7016ad23f13e8ef65c67ba6aa7`。
该记录与本节证据位于本隔离工作树,均由 Git 忽略。
- 三个批准的非唯一索引创建成功并独立读回核对,未删除或替换现有索引。
- 处理器 2/2 就绪后才滚动更新 API;API 4/4 就绪,均为提交 `521acce5353250c4285766f229b371b97a7aec46`。
- 已上传、拉取并核实 linux/amd64 镜像及 OCI revision;不可变镜像为
`ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-api@sha256:fad2211cde7774bf53ec6ae31756bc2c8f3b7ea557ecd9d55ebff9a882c7eaf3`。
- 19 项完整发布门禁通过;第一次门禁因缺少 Playwright 默认浏览器未完成,使用已有 Edge 后完整重跑通过,没有忽略失败。
- 原 `503 snapshot_unavailable` 复现请求已变为 200。平台管理员、供应商管理员与标注员返回实际 totals 和明细;初次成功热读约 159–169 ms。质检/验收账号登录及其现有只读仪表盘接口也返回 200,未把这两个角色原本只返回 audienceScope 的接口夸大为产出统计验收。
- 首个新版本快照重建约 69.3 秒,随后自动刷新约 54.1 秒;后续健康检查观测到又一代约 54.5 秒。刷新期间四个 API 均返回同一代完整数据(约 147–210 ms),未返回空统计或 503。
- 四实例构建版本及同代汇总一致。真实分页保持同代与总计;CSV 行数与当次筛选总数一致。生产数据持续变化,不要求跨代总数固定。
- 标注员全部返回明细归属本人,供应商管理员全部明细归属其供应商;试图筛选他人/其他供应商返回零条。非管理员诊断 403,匿名诊断 401。
- 11:46(Asia/Shanghai)最终复核:健康状态 healthy、可用、失败次数 0、无容量预警;来源约 31.9 MB / 127900 文档。Worker、validator、coordinator 及其他非本次目标 Deployment 的 spec 与发布前相同,ConfigMap/PVC/Service 等受保护资源通过工具复核。
本轮没有领取、保存、送检、审核决定或删除客户任务。常规登录记录与内部统计缓存写入在批准范围内。
`role-login` 已记录 passed;完整多供应商业务操作、标注交付、质检/终审写入、生产故障注入及视频播放验收仍如实记录为 unverified,发布状态保留 `deployed_pending_acceptance`。这不否定已完成的看板修复验收,也不代表所有生产业务已全量验收。
原始证据在 `local-data/verification/dashboard-stability-release-20261026/`,主要包括
`online-before.json`、`online-after.json`、`replica-result.json`、`refresh-result.json`、
`read-contract-result.json`、`final-infrastructure.json` 与 `acceptance-evidence.json`。
会话文件和凭据不得提交。
## 扩展边界
本轮让查询表达式和非统计过程历史不会无界拖累统计,并在保护预算前发出可观测信号;它不是无限规模的增量数据仓库。当前仍按需解释相关正式历史,来源 64 MiB / 250000 文档、单代解压 32 MiB、压缩 4 MiB、构建 90 秒等上限保留。
若持续出现 80% 预警,需要在达到硬上限前实施下一阶段的持久化增量事实与对账。不能仅按 createdAt/updatedAt 拉取新记录:迟到事件、已存在 reviewOperation 从 applying 变 completed、返工使旧完成失效、作者/供应商历史归属均可能导致漏算。那一阶段必须具备可靠变更游标、可重放、修正/删除传播、全量重建与跨代切换,并独立验证后迁移。本轮没有把未经这些验证的“增量缓存”投入使用。

View File

@ -0,0 +1,61 @@
# 最终分类总 JSON v4
本地实现对应 [ADR-0005 的 2026-10-21 修订](adr/0005-part-index-delivery-shape.md)。历史 v1/v2/v3 和已交付文件保持原状。开关默认关闭;本文不授权启用工作流或发布。
`DELIVERY_JSON_V4=true` 要求 `PART_INDEX_DELIVERY=true`,与 `DELIVERY_JSON_V3` 互斥;源索引修订另需 `SOURCE_INDEX_DELIVERY=true`。新工作流须有内部 `deliveryAudit` JSON 输出,以及正常的 `deliveryDirectory` 输出。配套 Worker 负责固定审核追溯,大于 inline 限额的验收摘要走现有内部 artifact 存储,打包节点读取固定引用;必须先验证内部存储可用。旧 Worker/旧算子不能启用新协议。
客户格式保持 `capture`、`partId`、`slices[].taskSplit`、`slices[].segments[]` 和 `discardedSlices`。按马迪约定的采集目录/part 组织文件:每个 part 一个总分类 JSON;批次索引 `delivery_index.json` 和采集索引 `annotation_manifest.json` 只负责目录定位、文件校验及汇总,不额外携带审核流水。正式批次目录不可变。
- `sliceName` 是已确认的基础分类名称;`classification.dictionary` 是可获得的字典快照。动作描述为空也保留这个分类,不代表无动作。
- 字典快照固定 `id/kind/version/code/name/tags`。`tags` 是独立分类标签,与 `category`、别名和描述原义不混用。新选择由服务端从项目字典校验,客户不能伪造标签;历史缺失快照不按当前词条补填。
- 每个动作显式导出有效 `tags` 和 `tagSource`。独立选择动作词条用该快照标签,来源为 `action`;没有独立选择则继承本切片冻结标签,来源为 `slice`,不伪造动作 `dictionary`。自由填写文字也沿用这个判定;有文字不代表选择了独立词条,没有文字不代表没有动作。独立词条缺 tags 时拒绝交付,不偷偷退回父标签。
- 动作 `contentKind` 为 `annotated`(有描述)、`invalid_marked`(明确无效)、`gap_placeholder`(保留的留空区间)。`descriptionStatus` 描述文字覆盖情况,不能用于推断整个切片是否有动作。
- `isInvalid` 与标签独立;无效动作可保留继承标签,但仍必须按无效判断排除有效动作处理。历史动作未覆盖的留空范围保留为继承父标签的 `gap_placeholder`,不能把该段视频漏掉。
- `discardedSlices[].classification.discardEvidence` 保留业务舍弃原因、原因词条/版本/标签和补充说明。全舍弃 part 也输出总分类 JSON。审核退回原因不放在这里。
- 舍弃原因有冻结字典时,旧兼容字段 `reasonLabelId/reasonVersion/reasonName` 从同一快照补齐,不导出空名称或版本 0;已有非空值若与快照冲突即拒绝交付。手填原因仍按原有规则保留,不猜造字典选择。
- `originalSource` 冻结 BOS bucket、原采集 prefix、partPrefix、输入快照指纹和所选原始文件路径。来自任务创建时的输入引用,不查询当前目录猜原文件,不输出连接 ID、凭据或签名 URL。八类文件为左右视频、左右 PTS、IMU、相机标定、IMU 标定和采集 meta。`sources`、`sourceMetadata.path` 与标定路径统一相对原采集目录,`pathBases` 显式写为 `original_capture`;原 workspace 路径只留内部追溯。
内部 `droboticflow.delivery.review-history/v2` JSON 按 capture/part/revision 关联,以 `partSha256` 固定原生主文件字节;包含最终验收、抽检、审核人/批注/轮次/确认项、已有正式退回基线、处理记录和阶段事件。该 artifact 留在内部 Run 存储,不进入最终交付列表、不可变客户批次或客户下载。分类选择的服务端快照另外保存在当前任务的恢复记录中,使字典改名、更新或停用后的撤销仍可恢复原版本,客户端不能伪造历史标签。旧记录没有的追溯不制造补写。追溯超出限制时拒绝交付,不静默截断。
最终分类取通过验收的固定物化修订;沿用既有“未抽中”验收策略的任务,内部保留明确豁免,不能伪造人工通过日期。返工正式修改分类后必须重新送检、验收;查看或处理意见本身不修改分类。相同名称而字典版本/标签不同的切片不能自动合并成一个分类。
时间按两级明确表达,全部区间采用左闭右开:
| 字段 | 坐标 |
| --- | --- |
| `taskSplit.startSecond/endSecond` | 原始 part 左相机视频秒数 |
| `segments[].startSecond/endSecond`、`clipStartMs/clipEndMs` | 本切片左相机内秒数/毫秒数 |
| `segments[].partStartSecond/partEndSecond` | 原始 part 左相机视频秒数 |
| `startTimestampUs/endTimestampUs` | 原始硬件时间戳,非墙钟时间、非从秒数估算 |
`timeBases` 描述以上坐标及帧区间基准。下游从原始左右 PTS 分别定位同一硬件时间窗,不能把左相机秒数直接当右相机秒数。原始输入快照指纹标识当时选择,不冒充每个 BOS 对象的版本号或内容 SHA;原始数据应保留,当前发布时仅以 HEAD 校验所引用文件存在,不证明同名文件从未被外部覆盖。
新 v4 要求完整的冻结分类、非空标签、原始来源和可靠时间边界;任一缺失或动作重叠、时间戳不连续即停止交付。批次发布再核对原始 bucket/part 归属、文件存在性和消费契约,不能只凭主文件的 SHA 放行。元数据修订复用原始引用,不追随后来改变的采集记录。未发布的旧本地 v4 样例须重新生成;正式 v1/v2/v3 及其恢复路径保持兼容。历史缺证据应先核实补齐,已交付文件不覆盖,修正交付另建不可变批次。
平台只输出分类索引,不按标签执行 BOS 数据切割、搬运或集中打包。内部审核文件不在客户目录内,也不从主 JSON 引用。
## 2026-10-22 时间与来源门禁补充
新 v4 不再接受缺少秒数↔PTS 映射、仅靠动作首尾连续的旧证据;必须与固定同步窗口一致,不能从帧率或时长猜补。保留原映射与秒数,缺证据的历史结果需另行核对后生成新修订,不能自动迁移。
原始来源检查补充核对 Run 接入时保存的选择指纹:对完全相同的对象键 HEAD 后,按既有 `key|ETag|size` 算法比较,包含当时选择的可选音频。打包前、Worker 发布前和批次预览/发布均校验;不再仅证明对象存在。该指纹不是逐文件 SHA256 或不可变对象版本保证,也不按当前目录重新生成期望指纹。历史 v1–v3 和已交付字节不变,详见 [标注时间完整性](annotation-time-integrity.zh-CN.md)。
本地可用 `python -m tests.backend.export_delivery_v4_fixture <全新输出目录>` 生成明确标记的合成样例。目录中 `partNNNN/part.json` 是客户主文件;`internal/partNNNN/review-history.json` 是分离的内部示例,不属于客户包。不要把整个验证输出目录直接交付给客户。
独立消费验证:设置 `FANYU_DELIVERY_CONSUMER_FIXTURE_DIR` 为全新绝对目录,先运行 Go `TestDeliveryV4ConsumerReplicaFixture`,再以同一环境变量运行 `tests/backend/test_delivery_v4_consumer.py`。合成真实双目视频分别经过物理切片、源索引和元数据复用;独立进程仅读取导出 JSON 与原始文件,核对每帧覆盖、非零 part 起点、标签继承、无效与舍弃。独立读取器 `tests/backend/consume_delivery_v4.py` 只生成处理计划,不执行切割或写源数据。
## 2026-10-22 组合候选与默认关闭阶段
用户确认先与前端分层版本合并验证,`DELIVERY_JSON_V4` 继续保持关闭,启用另行制定计划。新代码的标签编辑、字典快照保存和内部物化元数据并非全部受该打包参数控制;关闭参数只保证不主动选择 v4 交付路径,不代表 API 行为与旧版本逐字节一致。
整采集的多 part 输入可能无法证明单 part 的原始来源。普通物理切片、源索引及元数据复用会省略不可靠的新增 `originalSource` 并保留诊断,不因这项可选证据阻塞旧任务;跨 Run/组织错误与存储失败仍拒绝,已有冻结来源不追随新目录。真正生成或发布 v4 时,缺原始来源、标签或时间证据仍被打包/批次门禁拒绝,不能据此输出不完整 JSON。
回退旧 API 不是无损写入保证:新版本保存分类后,旧版本重建草稿数组可能丢弃字典快照。回退前应暂停相关写操作或准备能保留新字段的兼容回退版本;只保留恢复历史不等于能自动恢复所有后续修改。已交付不可变文件不因此原地迁移或覆盖。
## 2026-10-22 启用决定
用户随后确认启用 v4,范围仅为切换后新启动的工作流;在途任务、它们后续的审核/打包子运行和已交付文件继续原协议。此决定替代上节“继续保持关闭”的准备范围,不代表生产开关已经启用。实际发布仍需完成镜像、兼容性和目标核对,并审查具体部署及工作流切换计划。
用户明确确认当前无 tags 的启用词条以词条名称作为第一个独立 tags 标签。该初始化是一次有界字典变更:保留词条 ID、名称、用途、category、aliases 和既有历史,递增版本并记录新修订;已有非空 tags 不覆盖。历史任务中的冻结快照不因此重写,亦不反查新词条替代历史分类。后续 tags 仍由管理员维护。
新工作流使用新的定义版本及固定打包镜像。旧定义不原地改参数或镜像;切换必须保留已启动采集的原工作流归属,并阻止新监听器重新接入这些目录。新 v4 接入还需完整的原始文件选择,包含采集 meta;旧协议保留原有接入要求。准备与验收步骤见 [启用核对表](release/delivery-v4-activation.md)。

View File

@ -206,3 +206,12 @@ cherry-pick 提交哈希不同简单当成修复丢失,也不能把 patch-id
这是发布前的人工核对步骤,**没有新增团队级发布锁或自动语义兼容检查**。
2026-09-29 核对结果见 [版本一致性盘点](source-consistency-20260929.md)。
## 看板稳定性门禁(2026-10-26)
Fanyu 的 localChecks 新增 `python3 scripts/test_dashboard_stability.py`,固定使用经核实的本地
Mongo 27018 合成隔离库,包含 24000 条/来源规模、历史兼容、故障恢复和权限检查。
关键测试未执行、跳过或失败均阻断,不能以无 Mongo 时的 skip 结果发布。
上线时还需审查三个统计索引的创建授权及可选诊断字段的混合版本行为;验收同时检查真实看板
200 与快照持续更新,不能只看进程健康。接口、检查器和边界见[看板持续运行说明](../dashboard-stability.zh-CN.md)。

View File

@ -0,0 +1,27 @@
# v4 新工作流启用核对表
本页记录 2026-10-22 用户确认的范围,不是已部署证明。实际摘要、目标 UID、镜像和执行状态保存在被 Git 忽略的发布记录中。
## 范围与顺序
1. 在独立工作树验证 API/处理器、Worker、独立 package 镜像,保存源码、基础镜像、产物字节与 digest 的对应证据。API/处理器使用 `scripts/release.py`;Worker、字典和工作流切换另附有界计划。
2. 在 v4 尚未激活时,按发布工具先处理器后 API;随后更新兼容新旧协议的 Worker 镜像与构建标记,确认原来的 2 副本完整就绪且所有工作流仍使用旧协议。再将 Worker 暂停至 0 副本并确认扫描进程退出,完成字典和工作流切换后恢复同一已验证镜像的 2 副本。现有独立算子 Job 不删除,暂停期间调度及状态收敛暂时停顿。不更新 Coordinator、validator、Secret、资源规格或存储配置。
3. 按预览清单为当前无 tags 的启用词条初始化 `tags=[名称]`,递增版本并追加历史。逐条比较原版本、名称、状态、更新时间与标签;发现漂移停止,不覆盖管理员的新修改。此步骤不更新任务、草稿或已冻结字典选择。
4. 对每个目标项目创建隐藏的新工作流版本,仅修改 package 的固定镜像、`PART_INDEX_DELIVERY=true`、`DELIVERY_JSON_V4=true`、`DELIVERY_JSON_V3=false`、`SOURCE_INDEX_DELIVERY=true`、`PIN_OPERATOR_IMAGE=true`,并添加内部 JSON 输出 `deliveryAudit`。保留原节点、边、输入、供应商、接入目录、并发和验收抽样策略;逐项比较允许变更以外的内容。
5. 检查内置 artifact 存储与客户目录隔离、完整的每 part 原始选择及 meta、分类标签、两相机时钟和验收固定修订。全舍弃、无描述继承、无效动作及独立动作词条均须有验证证据。
6. 在停止新接入扫描并确认旧扫描已退出的短窗口内激活新版本。仅未启动采集可以转入新版本;`launching`、有 Run 的采集及其父子运行保留旧归属,新监听器识别并排除这些已接入目录。保留原记录 ID,防止 `(workflowId,prefix)` 冲突;不能删除登记后重扫。
7. 只读检查目标版本、旧任务归属、新旧镜像及实际交付格式。生产写入验收必须另列隔离项目、账号和合成输入范围;单凭 Ready、模拟存储测试或单独算子测试不能标记真实业务验收通过。
## 交付与回退
客户文件只含最终分类、标签、区间和来源。审核人员、批注、返工和阶段事件只存内部 `deliveryAudit`,超限或写失败必须阻断客户发布;不得静默省略。
旧 v1–v3 算子输出须有字节兼容回归,旧工作流继续固定原 package 镜像。不可变交付目录、归属标记和已发布索引不原地覆盖。新批次发布仍按已有预览和确认接口执行。
切换失败或结果未知时停止后续写入,先只读核对实际状态。新版本已经接入的任务不能被重新绑定为旧协议;退回监听器也不能重复接入这些目录。旧 API 会在后续保存时丢弃新字典字段,不能把直接换回旧镜像描述为无损回退。需要保留新字段的兼容回退候选,或单独确认暂停相关写入的恢复方案。回退不撤销已确认标签初始化、不删除内部审核证据或客户新批次。
## 发布门禁
发布必须同时具有具体镜像来源、`release.py` 检查、Worker/工作流/字典计划、上线前漂移复核及用户对展示后计划的部署授权。算子历史 release manifest 的已知漂移不得通过重算旧摘要掩盖;只能为真实构建和发布的新 package 绑定新的来源及摘要,其他环境和算子记录保留。
独立 package 发布后使用 `OPERATOR_RELEASE_ENV=fanyu python3 scripts/generate_operator_repository_manifests.py --slug package --check` 验证这一算子的发布和生成清单;不带 `--slug` 的全量检查行为保持严格。未修改算子的既有漂移另列遗留,不更新其镜像、源码摘要或目录记录来凑齐通过。所选 package 的任何摘要漂移仍阻断本次发布。

View File

@ -16,11 +16,24 @@
"annotation-delivery", "review-acceptance", "slice-recovery", "video-playback"
],
"localChecks": [
["python3", "scripts/test_dashboard_stability.py"],
["python3", "-m", "unittest", "discover", "-s", "tests/release", "-p", "test_*.py"],
["npm", "run", "lint:web"],
["npm", "run", "build:web"],
["npm", "run", "test:production:boot"],
["npm", "run", "test:browser:review-permissions"],
["go", "test", "./internal/api", "-run", "TestDurableAnnotationOutbox|TestSliceRecovery|TestReviewCannotMutateAnnotation|TestRouteSnapshot", "-count=1"]
["npm", "run", "test:browser:dashboard"],
["npm", "run", "test:annotation:persistence"],
["npm", "run", "test:review:decisions"],
["npm", "run", "test:review:drafts"],
["npm", "run", "test:browser:review-drafts"],
["npm", "run", "test:browser:review-decisions"],
["npm", "run", "test:annotation:pools"],
["npm", "run", "test:browser:annotation-pools"],
["npm", "run", "test:browser:slice-input"],
["npm", "run", "test:browser:annotation-persistence"],
["go", "test", "./internal/deliveryindex", "-count=1"],
["go", "test", "./internal/worker", "-run", "TestCaptureCutover|Test.*V4Capture|TestDiscoverCaptureGroupsForWorkflow|TestClassificationAudit|TestInternalAuditFailure", "-count=1"],
["go", "test", "./internal/api", "-run", "TestDurableAnnotationOutbox|TestSliceRecovery|TestReviewCannotMutateAnnotation|TestReviewDecision|Test.*ReviewDraft|TestReviewConfirmations|TestRouteSnapshot|Test.*Dictionary|TestProjectSliceNames|TestLegacyMaterializationWithoutPartScopedOriginalSelection|TestOptionalOriginalSource|TestAnnotationOriginalSource|TestClassificationBatch|TestWorkflowVersionActivation", "-count=1"]
]
}

View File

@ -7,9 +7,9 @@ for this project's selection, so other projects sharing an entry are unaffected.
In the annotator's slice phase, **从切片字典选择** searches active names enabled
for the current project and fills the selected slice's existing `name` field.
The name remains editable. Selection does not seek video, change boundaries or
The name is selected from the project dictionary. Selection does not seek video, change boundaries or
submit work; **确认全部切片** retains the existing save/materialization flow.
No dictionary means manual naming still works. Editing/removing dictionary
An absent or unavailable dictionary blocks a new slice confirmation. Editing/removing dictionary
entries never rewrites already saved slice names or historical delivery data.
## Compatibility and API
@ -26,7 +26,22 @@ entries never rewrites already saved slice names or historical delivery data.
- Project selections store `sliceLabelIds` independently from existing `labelIds`.
Memory/Mongo updates change only the relevant dictionary selection.
- Task/project switches abort stale reads; reopening the picker refreshes names.
Load failures do not disable manual name entry.
Load failures retain saved names and expose retry rather than manual name entry.
## Post-confirmation naming and submission
The annotation view has a visible **选择切片名称 / 修改切片名称** button. It opens
the existing atomic slice-edit panel; choosing a name preserves action content,
boundaries and history. E remains a keyboard entry. Renaming never requires
the destructive initial **重新切片** operation.
Unnamed kept slices may be saved as drafts while editing. Formal submission
requires a nonempty classification name, including after a split/merge was
saved: an unchanged empty workspace name is not a historical-name exemption.
Unchanged nonempty historical names retain their existing compatibility rule;
new/changed names still require the enabled project dictionary. This rule does
not prohibit unnamed or pending actions, which remain distinct from invalid or
discarded data. Existing deliveries and customer records are not rewritten.
Tests cover legacy action compatibility, purpose and project isolation, permissions,
independent updates, removal without history mutation, and the existing name callback.

View File

@ -0,0 +1,33 @@
# Administrator annotation pool totals — local verification
Base `867ca698`, branch `codex/rework-boundary-20261021`. Diagnosis captured deployed `fab5be57`: administrator badges were the stage-filtered first 500 human records while annotator blind claim used an aggregate. The original metadata replay produced 495 vs 14902; screenshots were 496 vs 14911 at an earlier time.
## Regression and implementation
`TestAdminAnnotationPoolTotalsAndOldPageSearch` first failed with 404 before adding the read endpoint. It now covers 610 added mixed-stage items, stable whole-pool counts across cursors, a search hit older than the previous 500-row window, and denial to annotators. Permission/cache/error tests cover role changes before cache access, normalized count keys independent of page/list filters, bounded page limits, malformed inputs, 5-second expiry and a failed aggregate returning an error rather than false zero.
The real page/HTTP fixture uses MemoryStore and synthetic records only:
```
ADMIN_POOL_BROWSER_TEST=1 go test ./internal/api -run 'TestAdminAnnotationPool' -count=1 -v
```
Covers 200→400 loaded cards with unchanged 606 available/611 all totals; search for CAPTURE_0001; status category returning zero without changing badges; all-pool review filtering; delayed old search response ignored; real local claim invalidating cached count. No page errors or unexpected browser writes.
Actual Mongo parity and timings use a new, unique synthetic database on local Mongo 27018, cleaned afterward:
```
POOL_LOCAL_MONGO_URI=mongodb://127.0.0.1:27018 go test ./internal/store -run '^TestAdminAnnotationPoolMongoParity$' -count=1 -v
```
1500 synthetic records in separate batches cover mixed stages/assignees, orphan and cross-organization rows, full old-page searches, literal rather than regex search, cursor de-duplication, persisted workflow categories, legacy Capture names obtained through Capture records, and absence of heavy annotationWorkspace on list responses. Timings are local store measurements, not claims about production latency.
Legacy API compatibility, shared Capture extraction, existing dashboard-cache defaults, annotator/review editing browser flows, TypeScript, build, production-bundle boot, and full frontend baseline gate are checked separately. No baseline entries are edited. Production data, workflow, indexes, Worker and delivery JSON are not changed in this task; no push, PR or deployment is included.
## Final evidence
Store-level measurements with 1500 separate batches: counts 42.8 ms, first page 29.4 ms, old-record search 486.6 ms; these are synthetic local observations only. No new production index or statistical collection is required.
One combined browser run lost a fixture work session during the draft C/undo/redo scenario; there were no page errors. The same test passed unchanged in an isolated rerun, including the newer-draft recovery path. This is recorded separately rather than erased from the results or accepted by changing test expectations. The administrator browser flow and existing boundary browser passed in that combined run. Logs remain in untracked local test results and `/tmp/admin-pool-all-browser.log` / `/tmp/admin-pool-slice-isolated.log` for this session.
The final combined administrator + slice-phase browser rerun passed with the newer-draft case, and no runtime errors. Final frontend gate retains only 13 known baseline failures; TypeScript, Go build, web build and production boot pass. Old non-admin project scope/dependency behavior is preserved, so local annotator filters do not trigger the new administrator queries.

View File

@ -0,0 +1,112 @@
# Annotation generation, naming and media recovery
Scope: October 21 platform-tuning reports B06, B14, B18, B11 and B20. Production
inspection is read-only. Tests use local synthetic media and isolated stores.
## Evidence
- B06/B14/B18 downloaded left videos have 6343/7957/11239 continuous 30fps
frames. Their stream duration is approximately 0.326ms short. All packet
durations are 512 ticks; the difference comes from MP4 edit-list presentation
duration. Unedited duration and decoded cadence agree. This does not establish
the state of every right-camera source in production.
- The user confirmed the 15:26 report is B11/part0001. Its task has no
`sliceWorkflow` or completed materialization evidence, no annotation revision,
and only an old slice revision 0. Its saved workspace contains seven slices
and six actions; its draft contains seven slices and ten actions. Frontend
draft inference can show annotation while the write gate rejects it. Merely
seeing numbered slices does not prove successful generation.
- B20/part0001 recording shows repeated HEAD 405 responses on the authenticated
`subject-content` URL during recovery. Both referenced BOS camera objects
exist; metadata and 1KiB range reads succeed (226335649/217067600 bytes).
This verifies the recovery blocker, without claiming the recording proves
what first interrupted native playback. Its later generation job completed.
- A local HTTP flow reproduced named parent → two unnamed children → saved
workspace → formal submission 201. Unchanged blank names bypassed the old
historical-name exception.
Redacted source metadata and recording are under the main repository's
untracked `local-data/diagnostics/annotation-failures-20261021/`.
## Behavior and compatibility
- Keep strict decoded frame/PTS checks; bound presentation endpoint rounding
by the actual MP4 movie clock and preserve rejection of whole-frame loss.
- Detail GET ships `editorState.phase`, `canConfirmDescriptions` and
`canRecoverGeneration`. The existing description gate remains authoritative.
Status-only polling does not perform the additional legacy evidence lookup.
- Explicit `recoverExisting` on slice confirmation is restricted to legacy
full-source tasks missing generation evidence. The server owns its retained
snapshot; client-supplied replacement snapshots are ignored. If a newer,
owned autosaved draft exists alongside the confirmed workspace, recovery uses
that draft, matching the editor's restored view. Ambiguous ownership, version,
timestamps or malformed content stop recovery without mutation. Queueing,
interruption/reclaim, failure/retry and completion preserve the selected work.
Ordinary failed-generation retries retain their existing confirmation policy.
- A visible post-confirmation name button uses the atomic edit application.
Blank kept names block formal submission; nonempty historical compatibility,
pending actions and discard evidence remain intact.
- Subject and scoped evidence media routes accept HEAD with the same access
checks as GET. The existing storage handler returns metadata without reading
the video body. Hidden cameras remain unloaded.
- The movi wrapper clears its exposed error for each new load instead of
retaining the vendor's previous failure. Transport sets the custom element's
source once, avoiding a second concurrent decoder load. Vendor assets and
media sources are unchanged.
- The recovery mode remains marked in `sliceWorkflow` until completion, so a
failure followed by another retry cannot silently select the older workspace.
API and annotation processing components need compatible versions for this
recovery path. Source-validation processes also need the clock fix before
retrying the affected source checks. Do not roll active recovery jobs back to
an API that lacks these retention rules.
## Verification
- Focused HTTP/API tests pass for server-owned editor readiness, blank-name
submission rejection, name correction, authenticated HEAD and denied access.
- Real MP4 regressions pass for 6343/7957/11239 frames with movie-clock endpoint
rounding. Mismatched frame counts and a whole-frame endpoint truncation remain
rejected. Invalid movie metadata retains strict tolerance.
- `TestLegacyGenerationRecoveryPreservesAnnotationsAcrossRestart` passes:
a newer autosaved draft survives queue interruption/reclaim, a real processor
rejection, retry and successful generation. Duplicate recovery and replacement
snapshot attempts cannot overwrite it. Ambiguous drafts reject without writes.
- The full local page → HTTP handler → isolated store browser flow passes:
the recovery button preserves the newer draft and waits for generation before
entering annotation; post-confirmation naming, first-pass/rework drag edits,
archived discard-side actions, failed saves, lost responses, reload and
persisted undo/redo also pass.
- Real `<movi-player>` software fallback passes injected media failure → reload
for annotator, quality and acceptance views. HEAD succeeds, the hidden camera
stays unloaded, editing position and unsaved description survive, and playback
advances after recovery. Review views do not expose the new naming control.
- The frontend and Go repository gates report no new failures against
`tests/known-failures.json` (13 frontend failures in the 14-case baseline,
70 Go failures in the 70-case baseline). The baseline is unchanged.
- A subsequent Go run also exposed
`TestDispatchTwoWorkersAndRestartDoNotDuplicateClaims`: it intermittently
records six distinct claims where the test expects three. The exact failure
reproduces 15 times in 3000 repetitions on pristine commit `2876a14d`, before
this change. Worker, store and domain files are unchanged. This independent
concurrency issue remains unresolved and is not added to the known-failure
baseline; its failed run is retained alongside the successful gate evidence.
- The pristine full gate also exposes the preceding commit's stale route
snapshot (the admin-pool route was already present). This change regenerates
the snapshot from the actual router, including that route and the new HEAD
routes; it does not change admin-pool access behavior.
- Type checking, formatting, Go build, web build and the production-dist browser
boot check pass. The boot check uses loopback API stubs, with no page errors or
unexpected writes.
Local browser regression commands:
```sh
REWORK_BOUNDARY_BROWSER_TEST=1 go test ./internal/api -run '^TestReworkBoundaryBrowserReplica$' -count=1 -v
REWORK_BOUNDARY_BROWSER_TEST=1 ANNOTATION_FEEDBACK_BROWSER_SCRIPT=tests/browser/annotation-media-recovery-replica.mjs go test ./internal/api -run '^TestReworkBoundaryBrowserReplica$' -count=1 -v
```
The browser runs use local Edge/Chromium and synthetic media; they do not prove
every customer's decoder or network. The original cause of B20's initial native
playback interruption remains unverified. Production B11 generation recovery
has not been executed. No rollout, customer-record repair, index migration or
object-storage write is part of this change.

View File

@ -0,0 +1,19 @@
# Platform feedback repairs — 2026-10-21
Local branch `codex/rework-boundary-20261021`, based on deployed `fab5be57` plus local boundary repair `4d12d8cb`. No push, PR, cluster deployment or customer data writes.
## Cause and compatibility
Initial slicing can recover saved pending actions. The quick edit planner correctly detects action conflicts, but Workbench previously rendered and accepted those conflicts only in annotation phase and silently canceled them in slice phase. Thus attempted edits never entered history. Preserve all existing action records and make the stage contract consistent; keep the server's archive, CAS and restoration paths. No schema, workflow, formal decision or delivery format changes.
Feedback previously omitted complete editor drafts and compared derived slice annotationStatus. It now includes those drafts while preserving display classification/handedness, excludes incomplete end selections, ignores derived slice status and normalizes implicit keep. Discarded slices retain evidence and the opinion rail in the right sidebar. Selecting whole-action invalidation explicitly warns that both sides are affected.
## Regression evidence
- Red before repair: `TestSlicePhaseBrowserReplica` failed `pending actions must show a visible conflict` (0 vs 1); the derived-status unit regression failed uncleared count (1 vs 2).
- `REWORK_BOUNDARY_BROWSER_TEST=1 go test ./internal/api -run '^TestSlicePhaseBrowserReplica$' -count=1 -v`: real AnnotationOperations page, real HTTP handlers, isolated MemoryStore and synthetic H.264 media. Covers empty control, saved pending actions and a newer owner-matched autosave draft over an empty confirmed workspace, C/Enter/save/undo/redo/reload, pending drag/delete undo/redo, actual draft editing and reversal, false-gray prevention, discarded evidence/opinions and sidebar geometry.
- `REWORK_BOUNDARY_BROWSER_TEST=1 go test ./internal/api -run '^TestReworkBoundaryBrowserReplica$' -count=1 -v`: existing discard-boundary, Enter/button, failure/retry, lost-response reconciliation, busy/cancel and persisted undo/redo scenarios remain passing.
- 66 targeted frontend tests and targeted Go slice application, restoration, rework and draft tests pass; TypeScript and web build pass. Production bundle boot has no runtime errors.
- Full frontend gate: 13 existing failures, all in the unchanged known baseline; no new failures. Four source-shape assertions were updated for the new phase contract while retaining review permission assertions. Baseline entries were not edited.
Browser is local Edge/Chromium, not a claim of testing the customer's exact Chrome installation. No production browser/client acceptance or historical result repair is claimed. Old mistaken invalidation results require individual evidence review; they are not guessed or overwritten. Source diagnosis evidence remains under untracked `local-data/diagnostics/chengxiaoming-feedback-20261021/` in the working root.

View File

@ -0,0 +1,45 @@
# 前端分层与交付 v4 组合候选
用户选择:先整合并验证,v4 保持关闭;启用另行确认。新个人分支 `codex/dashboard-delivery-v4-20261022` 从 `2149b862` 合并 v4 分支 `e66c7726`,原两个分支保留。
三处文本冲突已处理:切片字典接口保留 v4 的服务端冻结/历史恢复语义,工作台和标注页保留分层协调与字典选择两侧导入。不能以文本合并成功代替业务验证。
## 已核对的发布范围
2026-10-22 的只读 Deployment 快照显示 Fanyu / droboticflow-fanyu 的 API 与处理器均为 `3221ac65`,Worker 为 `963b530f`;它们均为组合候选的祖先。此事实只说明源码历史没有遗漏,不证明所有语义兼容,也不能代替发布前新快照。
本次只准备候选;没有上传镜像、部署、迁移、启用工作流或改写客户交付。后续若发布默认关闭版本,计划限 API 与处理器,Worker、独立打包算子和工作流继续使用现有版本。v4 需要配套升级 Worker/算子、验证内部审计存储及配置 deliveryAudit 输出,不能只切一个开关。
候选未新增 Store/cmd 迁移或索引修改。内置算子目录新增默认关闭参数和审计输出,目录查询沿现有代码可能更新内置算子定义;这不是现有工作流自动升级,也不能说镜像更新绝不触发数据库元数据写入。发布前仍须审查实际配置、在途任务和数据影响。
## 修复的组合兼容问题
合成多 part 输入引用实测使物理切片、源索引在 v4 关闭时也因新增来源证据检查失败。现将不可信的可选来源证据省略并记录诊断,普通媒体处理继续;跨任务/组织及存储错误仍拒绝。真实 v4 的缺来源、文件归属、时间和标签门禁继续有效。已有冻结来源不被后来目录变化改写。
v4 默认关闭不等于旧版本写入安全:新字段开始保存后,旧 API 的数组重建可能丢失字典快照。后续上线必须明确暂停相关写入或使用兼容回退版本,不能承诺直接回退现网镜像后继续编辑无损。
## 本地证据
- 类型与分层检查、全量前端基线门禁通过;13 项已有失败,14 项基线未修改。
- v4、字典、批次、历史 part-index 的 Go 定向回归通过;新来源兼容回归覆盖物理/源索引两次修订、元数据复用、身份隔离和不伪造来源。
- 新合成真实双目媒体经过 Go 物理切片/源索引,再由独立 Python 消费客户 JSON 与原始文件,确认标签继承、留空、无效、舍弃与硬件时间窗;Python 共 43 通过、1 项旧夹具依赖场景跳过,未将跳过计入通过。
- 组合版完整标注 19 场景、质检/验收草稿 34 场景和审核权限浏览器回归通过。
- 修复前的来源错误、开发态字典夹具依赖优化导致的页面未渲染尝试均保留,不能冒充产品失败或已通过验证。
- 字典标签的生产组件构建及浏览器测试通过;共享保存 24 项、发布工具 37 项、生成源码一致性、Go 编译、前端构建与生产启动通过。
- 完整算子发布 manifest 检查未通过:annotation_gate 的 sourceTreeSha256 漂移在原始 `3221ac65` 工作树逐项复现,期望值/实际值完全相同,属于已有问题;新 v4 package 源码也尚未绑定新算子镜像发布。不得伪造 release.json 摘要使检查通过。它阻断算子升级/v4 启用,不将本轮限定的 API/处理器默认关闭候选冒充为算子可发布版本。
证据存于本工作树忽略目录 `local-data/verification/combined-v4-20261022/`。本记录不是部署批准,也不保证回退旧写入路径无损。
## 后续启用准备
用户随后确认仅新启动的工作流使用 v4,并确认缺 tags 的当前启用词条以词条名称初始化首个标签。新增准备和验收记录位于 `local-data/verification/v4-activation-20261022/`,不覆盖上述默认关闭阶段的证据。
独立 package 镜像已由 `8f6b531f` 实际构建、核对镜像内源码并上传 Fanyu 仓库,摘要为 `sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d`。仅更新 package 的真实发布记录及生成清单,保留 POC 历史镜像记录。独立发布使用 `OPERATOR_RELEASE_ENV=fanyu python3 scripts/generate_operator_repository_manifests.py --slug package --check`,仍强制验证所选算子的实际源码树和镜像摘要;默认不带选择的全量检查继续保留,不能据此声称全库清单通过。未修改、未发布 annotation_gate,其已有清单漂移仍保留为独立遗留项;本次工作流固定新 package digest,不依赖重写全局算子目录来替换旧任务镜像。
切换修复保留已启动采集及父子 Run 的原归属,恢复隐藏版本上未启动登记的迁移,保留原 ID,并在未知启动结果时阻断激活。未知启动结果须先停扫描、核查原归属并单独恢复,不能按记录年龄自动重复创建 Run。新 v4 接入前验证完整八类原始文件,包括 meta;布尔开关与打包器、内部审核保存使用一致口径。详情见启用核对表,实际部署仍须具体计划授权。
## 启用前审核输入回归
正式发布门禁发现保存后立即编辑下一条意见的竞态:模型已切到切片编辑器,但 React 尚未提交新 DOM 时,常驻输入框的旧动作聚焦回调会重新选择动作,将新切片文字提前暂存并打开空动作编辑器。常驻输入框改用控制器的 `editSelected()` 读取当前目标,避免使用渲染时捕获的旧作用域。
真实组件浏览器回归控制聚焦回调在 React 提交前执行,同时验证切片、来源和文字保留、没有额外 PUT。旧实现的质检与验收两项均稳定失败,修复后全套 36 项通过;临时诊断代码已清理。原始发布失败、受控失败和成功证据分别保存在启用证据目录,不以重试到成功替代问题修复。五角色线上登录与自身身份预检已通过;该结果不是候选上线后的业务验收。

View File

@ -0,0 +1,49 @@
# 客户可对账看板:本地实现验证
日期:2026-10-20。实现分支:`codex/customer-dashboard-20261020`,基于 `d6fe5e9be3f9e0002694cffdf776ae7c886235e5`。本报告完成时只做本地修改,不含公司集群发布或客户历史数据补录;此后初版按用户授权发布为 `65c19de0`,并被 `962ea6da` 取代,线上发布状态见 [`共享快照验证报告`](../dashboard-snapshots-20261020/report.md)。
## 最终行为
- 标注员看本人产出与任务;供应商管理员看本供应商;组织管理员看正式生产统计范围内的项目并按供应商、人员下钻。项目、账号现任角色和当前分配变化不重写历史归属。
- 单位为原始 part,整段源视频只计一次。提交取首次有效成功送检日期;质检、最终验收取有效正式决定完成日期;重提、请求重试不重复加产出。完成计划在 applying/未知状态时不计成功。返工按当前状态快照;累计截至所选结束日。
- 时长只取明确的整段源时长;工作日志的 `videoSeconds` 是切片时长,不能用来补数。无法证实的时长为 null,界面和明细显示“—”;含未知时长的汇总也不展示伪装成完整值的部分和。
- 每张表独立分页,默认 5,可选 5/10/20/50;换筛选/页大小回到第一页;汇总与全量 CSV 导出不受当前页影响。
- 保留用户批准版式:不展示“个 part”“待核对”、长规则说明或调试提示。详情只读,支持 Esc 和焦点返回。管理员旧生产进度与运行概览保留在折叠区;审核角色原有看板保留。
## 接口与兼容边界
新增 `GET /api/dashboard/annotation-output`;不改变旧 statistics 接口、Mongo 文档、工作流、任务状态或交付 JSON。
查询参数:`from`/`to` 为包含端点的本地日期,`timeZone` 为 IANA 时区(默认 Asia/Shanghai);服务端转换为 UTC 半开区间,最多 366 个日历日。`projectId`、`supplierId`、`userId` 为筛选。`tasksPage`/`tasksPageSize`、`suppliersPage`/`suppliersPageSize`、`daysPage`/`daysPageSize` 分别分页;每页最大 100,超末页归并到末页。`dailySupplierId`/`dailyUserId` 只收窄每日表。`format=csv` 导出全部筛选明细,超过 100000 行明确拒绝。
组织、本人或供应商边界由服务端主体验证,查询参数不能扩大权限。只读投影不加载草稿、描述正文、媒体或凭证;个人和供应商结果不返回原始审核证据列表。CSV 保留阶段归属、防公式注入并使用 UTF-8 BOM。
## 验证结果
| 范围 | 结果与证据 |
| --- | --- |
| TypeScript、Prettier | `npm run lint:web`、`npm run format:check` 通过。 |
| 定向 Go + 本机 Mongo | `TEST_WORKFORCE_MONGO_URI=mongodb://127.0.0.1:27018 go test ./internal/api ./internal/store ./internal/workforcestats -run TestAnnotationOutput -count=1` 通过。覆盖权限、冻结归属、跨期、重试、返工撤销、未完成计划、多任务合并、来源标识、缺少整段时长、207 条分页及完整导出、时区与夏令时、CSV 公式转义、Memory/Mongo 一致性。 |
| 全量前端基线门禁 | 13 项失败均属于 14 项既有基线;1 项既有失败已不再出现,未擅改基线。 |
| 全量 Go 基线门禁 | 70 项失败均属于 70 项既有基线,无新增失败。最后时长收紧后另跑定向 Go 回归及 API 构建。 |
| 路由快照 | 更新后 369 条,仅新增本接口;处理器权限测试另验未授权角色 403。 |
| 三角色真实浏览器 | 管理员、标注员、供应商管理员,真实 HTTP router + 本机隔离合成 Mongo。验证分页汇总、从第 2 页全量导出、页大小重置、详情与 Esc、供应商/人员/每日下钻、项目切换;无页面异常,无非认证写请求。 |
| 构建与生产入口启动 | `npm run build:web`、Go API 构建通过;生产静态包真实浏览器启动成功,登录表可见,无 React 启动错误。仍有既有大 bundle 提醒。 |
原始日志、浏览器 JSON 报告和截图保存在当前 worktree 的未跟踪目录 `local-data/verification/customer-dashboard-20261020/`。临时测试环境启动与浏览器代理配置故障保留在带 failure 后缀的报告中,最终结果见 `browser-report.json`、`final-targeted-tests.txt`、`production-boot.txt`。
## 本地复现与预览
已存在本机 Mongo `127.0.0.1:27018` 时,在本分支依次启动(各占一个终端):
```sh
go run ./tests/browser/support/annotation-output-replica
VITE_DEV_API_BASE_URL=http://127.0.0.1:3026 DISABLE_HMR=true npx vite --host 127.0.0.1 --port 5196 --strictPort
npm run test:browser:annotation-output
```
入口:`http://127.0.0.1:5196/?view=dashboard`。测试服务仅监听 127.0.0.1:3026,每次创建独立 `annotation_output_browser_*` 合成数据库,账号密码随机生成至本地权限 0600 的 `accounts.json`,不写入仓库。浏览器默认使用本机 Edge;可用 `BROWSER_EXECUTABLE` 指定已安装浏览器。
## 尚未验证的范围
本次没有连接生产环境核实全量历史完整性或真实规模下的查询性能。读取器按组织/正式生产项目投影加载已有事件历史,未增加持久化统计快照或索引迁移。不能关联来源的旧任务暂按独立源任务保留,缺少冻结归属的记录不猜测作者;正式发布前需只读抽样评估这些历史覆盖范围。历史补录必须另列来源与冲突,经用户确认后执行。

View File

@ -0,0 +1,70 @@
# 共享按需看板快照:本地验证
日期:2026-10-20。分支 `codex/customer-dashboard-20261020`,基于已发布的 `65c19de0bf402cd8202714ebbfd8a05551cd5a25`。用户选择方案 2;本报告完成时只做了本地实现,没有上传镜像或写入公司环境。同日 22:56 按用户单独授权的镜像上传、具体部署和内部统计写入完成发布,线上状态见下节。
## 线上发布状态(2026-10-20 22:56)
权威发布记录位于 Git 忽略的本地路径 `local-data/releases/dashboard-snapshots-20261020-962ea6da/release.json`,原始证据在 `local-data/verification/dashboard-snapshots-release-20261020/`。
- 候选与线上构建:`962ea6da3d6932254e1bae293188128db8cbc6ce`;镜像 `ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-api@sha256:5915ff069632516942f80244c03fd9f6d087eee4d60987287def7a5ee21ac113`;计划 `0c2a7c9657fd67b0d4600075b392b2da28b8f9a285514759a9495a37ac398824`;目标 Fanyu / `droboticflow-fanyu`。
- 部署与复核:处理器 2/2、API 4/4 就绪后才通过基础设施复核;Worker、validator、coordinator 及受保护配置、PVC、Service、资源设定与发布前一致。
- 线上看板实测:首次共享快照 18,472 ms(经历 9 次 202 轮询);随后 6 次 92–97 ms;到期刷新触发请求 125 ms 返回旧完整结果,刷新期间查询 87–102 ms。均为不同时点的观察值,不当作容量承诺。
- 业务验收:`role-login`、`supplier-permissions`、`annotation-delivery`、`review-acceptance`、`slice-recovery`、`video-playback` 六项均如实记录为 `unverified`(缺生产管理员/供应商管理员账号,未取得客户任务写测试、生产故障注入和播放验收授权);发布记录状态为 `deployed_pending_acceptance`,不宣称完整业务验收通过。
- 同日早先的三条发布记录(`63560220`、`86a1d7a3`、`65c19de0`)线上镜像已被本版本取代;记录仍保持各自状态,不追认其业务验收。
## 结果
原路由每次加载并解释业务历史,之后才分页;已发布版三次读取实测 14275、14703、13703 ms。本版共享经过解释的 part 统计事实,API 副本、日期、人员和页码复用同一代数据;请求仍按当前身份和正式项目范围过滤。旧缓存因其他 POST 失效时不再连带重建本看板。
后台重建由访问触发,默认间隔 120 秒,无访问不刷新。构建成功并校验后才替换可见指针;失败保留此前完整版本并退避 120 秒。超过 15 分钟不再提供旧结果,返回正常加载/暂不可用状态,不用空统计冒充成功。卡片、三个分页表、人员下钻和全量 CSV 使用同一 snapshotId;被回收、过旧或范围变化的 token 返回 409,页面重置分页后整体加载新版本。
产品页面沿用已确认版式,不新增口径长提示、资源参数或调试信息。初次构建显示正常加载状态,可取消轮询;已部署旧客户端没有请求 shared 模式时,由服务器等待同一共享构建并返回原有 200 结构,避免 202 破坏旧页面。
## 存储与兼容边界
新增内部集合 `annotationOutputSnapshotState` 和 `annotationOutputSnapshotSlots`,分别保存小型全局协调记录及压缩后的不可变统计事实;只依赖现有 Mongo `_id` 索引,不增加启动迁移。普通数据库概览、读取和写入入口拒绝这两个集合,避免全组织快照绕过报表权限。
快照只含 part 标识、项目、明确整段时长、状态、阶段时间及冻结作者/供应商,并含显示名称字典;不复制描述正文、草稿、媒体、凭证或原始事件列表。原始审核和工作日志保留不动。响应字段兼容既有客户端并新增 snapshot 元数据;原始 evidence 列表不复制到缓存报表,详情继续显示阶段时间与归属。保留原有事实读取和纯统计计算作为重建路径。
本接口对业务数据只读,但 GET 可能产生内部派生统计写入。后续公司发布计划必须明确列出这两个集合及其写入范围,不能沿用“零写入验收”的描述。生产统计范围变化使旧 token 失效,角色与供应商变更按每次请求重新授权。
业务文档、任务状态、工作流、交付 JSON 均无需迁移。回退到 65c19de0 不依赖这些快照,但旧版数据库浏览器不认识这两个保留名称;若以后回退,需在回退计划中单独处理遗留内部集合的可见性。本次没有执行任何生产清理。
## 资源和故障边界
| 项目 | 实现限制 |
| --- | --- |
| 历史重建 | 每个数据库一个有效全局租约;owner + epoch 条件更新,60 秒租期、10 秒心跳、90 秒构建超时。 |
| 历史读取 | 最多两条并行来源查询,每条 Mongo 查询 maxTime 15 秒;跨集合累计投影 BSON 最多 64 MiB、250000 文档,超预算终止,不发布部分结果。 |
| 共享存储 | 最多 8 个范围,16 个固定槽;单槽压缩载荷最多 4 MiB,总压缩载荷最多 64 MiB。通常保留当前/前一代;满槽时可提前回收前一代,旧 token 明确失效。 |
| API 本机 | 压缩缓存最多 8 MiB / 16 项;每个 Reader 最多两次完整查询并发,解压载荷最多 32 MiB。 |
| 故障恢复 | 重建不依赖发起请求存活;失败退避,重启后读取共享结果,过期租约可接管。旧 owner 不能发布覆盖新代或释放新 owner 的租约。压缩长度、摘要和解压上限校验失败时不提供部分结果。 |
这些是载荷和并发预算,不是进程 RSS、CPU 或 Mongo 实际扫描文档数的硬配额;业务查询索引和线上数据分布仍影响冷启动。租约使用应用时间,与现有租约模式一致,节点时钟需正常同步;未模拟任意长暂停和跨节点时钟偏差。无后台定时全表扫描,未引入 Redis、新服务或新依赖。
## 验证证据
原始日志与截图位于当前 worktree 未跟踪目录 `local-data/verification/dashboard-snapshots-20261020/`。
| 验证 | 结果 |
| --- | --- |
| 先失败再修复 | `red-repeated-history.txt` 记录两个 handler/分页历史加载 2 次的失败;修复后共享 1 次,非相关 POST 不再触发重建。 |
| Memory / Mongo 与并发 | 真实 Mongo 多 Store/Reader 及重建后新 Reader 共用同一代;四实例并发冷启动仅 1 次来源加载。覆盖失败保留、退避、旧 owner fencing、遗弃租约接管、取消请求/关闭、损坏代次、容量/槽位限制和字节预算。 |
| 统计口径 | 序列化前后与原算法等价,覆盖日期、人员、供应商筛选;既有冻结归属、跨期、返工/重试去重、缺少时长、207 条分页及全量导出回归通过。 |
| 授权与兼容 | 当前角色或项目范围变更及时拒绝;相同数据范围跨身份共享后仍隔离结果;内部集合不能通过通用接口绕过权限。旧客户端等待 200,新客户端冷启动 202;固定版本导出头一致,失效版本 409。 |
| Race 检查 | outputsnapshot/store/workforcestats/api 定向 `go test -race` 通过,包含本机合成 Mongo。 |
| 三角色浏览器 | 管理员、标注员、供应商管理员真实 HTTP + 本机合成 Mongo。通过分页与下钻、固定版本全量 CSV、真实 409 后整页恢复、详情及项目切换;无页面异常、无非认证业务写请求。 |
| 全量门禁 | 前端 13 项失败全部属于 14 项既有基线;Go 70 项失败全部属于 70 项既有基线,无新增失败,未改基线。TypeScript 与格式检查通过。 |
| 构建 | Web 与 Go API 构建通过,生产静态包在真实浏览器显示登录表且无启动异常;保留既有 bundle 大小提醒。 |
故障测试通过受控失败、模拟租约遗弃/时间推进以及重建 Store/Reader 验证恢复,没有宣称做过操作系统强杀、生产 Mongo 故障切换或公司多节点压测。最初浏览器测试的占位行等待/页大小同步问题记录在 `browser-initial-fixture-failures.json`,修正测试同步后通过。
## 本地规模计时
`RUN_OUTPUT_SNAPSHOT_BENCHMARK=1` 的隔离合成 Mongo 测试:5000 个 part、25000 条历史事件、4 个 API 实例、32 次热请求。首次请求 224 ms,热请求 P50 15 ms / P95 17 ms,来源重建 1 次,1 个已写槽,压缩载荷 44634 字节。原始输出 `scale-tests.txt`。
该测试验证共享和负载方向,不能与生产的 13.7–14.7 秒直接计算加速倍数,也不代表生产时延或 RSS 承诺。生产四角色完整验收、冷启动时间与数据库负载仍待具体发布后核验。
## 本地预览
入口 `http://127.0.0.1:5196/?view=dashboard`,Vite 代理到隔离 API `127.0.0.1:3026`;测试仅使用本机 Mongo `27018` 下新建的 `annotation_output_browser_*` 数据库。随机合成账号保存于 `local-data/verification/customer-dashboard-20261020/accounts.json`,不提交账号口令。复现命令沿用 [初版验证记录](../customer-dashboard-20261020/report.md#本地复现与预览),额外设置 `ANNOTATION_OUTPUT_VERIFY_DIR` 可指定本次浏览器证据目录。

View File

@ -0,0 +1,74 @@
# 看板无数据:根因、修复与上线边界
调查日期:2026-10-26(Asia/Shanghai)。线上基线 `44e66a6800b77fe97b3b1db631c732cb2e30c634`;本地隔离分支 `codex/dashboard-stability-20261026`。本报告不是发布验收记录。
后续持续运行加固已实现,并于 2026-10-26 按明确批准计划上线、完成看板只读复验;最新范围、生产结果、三个索引、只读健康接口与强制门禁见[持续运行说明](../../dashboard-stability.zh-CN.md)。下文保留第一轮调查与修复证据。
## 已确认的故障链
1. Fanyu / `droboticflow-fanyu` 的四个 API Pod 正常运行,健康接口 200,观察时约 22 ms;存活探针不能证明看板可用。
2. 使用获准的内部标注测试账号,`GET /api/dashboard/annotation-output?...&snapshotMode=shared` 两次返回 503 / `snapshot_unavailable`(43 ms、62 ms),没有 totals。没有证据表明客户原始数据被删除。
3. 内部快照最后成功时间为 2026-10-26 08:50:53;10:27 和 10:32 观察时均 `failed=true`、租约 owner 已释放。当前代 epoch 564,协调器先后 594 / 595。旧数据超过现有 15 分钟有效期后,服务按契约拒绝提供陈旧统计。
4. 在获准的 API 容器 /tmp 中临时运行同版本只读加载函数(不启动 API、不执行迁移),原查询重现:`runs` 在 15,874 ms 返回 `MaxTimeMSExpired`,整次加载 33,091 ms 失败,关联 humanEvents 查询随之取消。当时累计 23,428,433 BSON bytes / 75,763 documents,未到 64 MiB / 250000 上限。
5. 旧查询把项目、runId 和 capture taskId 三种关联合在一个 `$or` 中;正式范围有 23,264 annotationTasks、24,188 captureRecords、23,376 humanItems,关联数组随存量增长。线上已有索引缺少 organizationId/taskId 前缀;此外独立只读聚合复现中 runs 与 humanEvents 均在 15 秒服务器预算下超时。现场观察到 runs COLLSCAN,但未把其他业务的 currentOp 条目冒充本次查询的 explain。
6. 后台 Reader 丢弃来源读取/发布错误,只保存 failed 标志,导致常规日志无法解释 503,延长诊断时间。
结论:确认的直接根因是 **runs 统计历史查询超过执行时间限制,导致共享快照无法更新,过期后看板不可用**。查询形状和缺失的关联索引是本轮修复目标。其他“所有看板”角色/入口未逐一复现,不能据此宣称每个页面同因。
排除与限制:已保存快照压缩约 1.32 MB、解压约 7.17 MB,明显低于 4/32 MiB 限制;无 API 重启/OOM 证据。最初本地隧道大批传输出现 EOF/客户端超时,已排除出根因证据,最终采用容器内的原查询实测。只读样本同时存在 claim-count 500 和其他慢请求,未将它们归因于本次看板问题。
## 本地修复
- 关联 ID 去重、每批最多 256 个;保留最多两条并行来源读取、每条 15 秒、总构建 90 秒和累计载荷预算。
- runs 三种来源分开查询并按 run ID 合并,项目外旧 runId 引用及 capture taskId 关联均保留,已有项目结果避免再次按 ID 读取。不更改统计口径、权限、历史归属和快照格式。
- 权威索引声明新增 runs 的非唯一 `organizationId_taskId`,键为 `{organizationId:1, taskId:1}`。这是索引结构变更,部署前必须单独审查并授权,不能作为“零线上写入”发布。
- 保存首个来源错误,避免兄弟查询的 context canceled 掩盖根因。预算错误附计数;后台失败日志包含阶段、epoch、耗时及错误,另记录续租/清理失败。不输出统计正文、账号口令、Cookie 或 Mongo URI。
- 15 分钟最大陈旧时间与失败退避保持原契约;没有用过期数据或零值伪装恢复。
MongoDB 官方说明了 [$or 各分支与索引的关系](https://www.mongodb.com/docs/manual/reference/operator/query/or/)。该原则解释修复方向,实际效果仍需上线索引后的生产复核,不能用文档代替测量。
## 验证
测试连接经核实为本机 Colima 的 `fanyu-mongo-restore` / 27018;每次新建独立合成库,结束清理,不使用客户生产库写入。
| 验证 | 结果 |
| --- | --- |
| 原查询回归先红 | 600 条/来源的真实 Mongo loader 测试捕获 `$in` 长度 600、1200 等;修复后通过。 |
| 数据完整性与隔离 | 关联任务、事件、审核、版本全部保留;三条 run 路径并集去重;保留项目外旧引用,拒绝外组织和无关项目数据。 |
| 规模测试 | 24000 条/来源 × 7,额外兼容记录;run 带 4 KiB 非统计负载。读完整事实约 421 ms,合成数据准备约 2.03 s。本机数字不代表生产时延或容量承诺。 |
| 错误诊断先红 | source 失败原本无日志,回归测试失败;新增阶段与原因后通过,并检查租约释放与退避不变。 |
| 并发与持久化 | outputsnapshot 全包 race;store 的 TestAnnotationOutput 系列 race 通过,包含多 Reader/重启、失败保留、lease fencing、固定槽、预算、私有集合隔离。 |
| 接口与统计 | API / workforcestats 的 TestAnnotationOutput / TestOutput 定向回归通过。 |
| 构建 | `go build ./...` 通过。没有前端改动。 |
| 新索引 | 本地空库通过权威 EnsureIndexes 创建并校验声明;尚未创建线上索引。 |
命令:
```sh
TEST_WORKFORCE_MONGO_URI=mongodb://127.0.0.1:27018 go test -race ./internal/store -run '^TestAnnotationOutput' -count=1
go test -race ./internal/outputsnapshot -count=1
RUN_OUTPUT_HISTORY_SCALE=1 TEST_WORKFORCE_MONGO_URI=mongodb://127.0.0.1:27018 go test ./internal/store -run '^TestAnnotationOutputLargeRelatedSetsRemainBoundedAndComplete$' -count=1 -v
TEST_WORKFORCE_MONGO_URI=mongodb://127.0.0.1:27018 go test ./internal/api ./internal/workforcestats -run 'TestAnnotationOutput|TestOutput' -count=1
go build ./...
```
## 发布与恢复计划(未执行)
1. 重新核实线上 API、处理器、Worker 镜像与构建,按 release_source_audit 审查候选是否含当前所有修复;若线上已变化,不能直接用本分支覆盖。
2. 单独批准在 `drobodata-fanyu.runs` 添加上述非唯一索引,不清理/替换现有索引,不改客户文档。先检查同名键是否一致;由受控会话创建一次并核对完成,观察数据库负载。索引构建会读取集合并占用数据库资源,不能把它当成无影响动作。
3. 构建不可变候选镜像,使用 `scripts/release.py` 的检查/计划/证据流程。候选启动会 EnsureIndexes;必须在 change-safety 中明确该行为与预建索引结论。部署需要临近执行时的单独批准。
4. 获准后复验:同一个原始 GET 返回 200 和 totals;快照生成时间推进;跨四副本复用;超过 120 秒刷新仍成功;503 与后台失败日志不再持续。逐角色复核范围/汇总,并抽样对照旧统计计算,写测试另行授权。
5. 回退需新授权;旧镜像可读现有快照格式,新增非唯一索引可保留,不能删除/恢复业务数据。回退应用不能保证消除旧查询的超时,不得把“rollout 成功”记录成看板恢复。
尚未构建/上传发布镜像、创建线上索引或部署;原线上 HTTP 复现仍失败。本地验证不能替代生产验收。
## 长期稳定性:以减少故障驱动改动为目标
本轮先消除已证实的查询退化并补可诊断性,不更换架构。后续按优先级推进:
1. 将真实看板成功率、快照年龄、连续刷新失败和构建耗时接入现有监控,在 15 分钟过期前告警;200 health 不作为业务可用性依据。本轮新增日志,未配置外部告警或定时任务。
2. 将这次有界查询/兼容/恢复测试纳入发布门禁,并明确索引及真实历史规模验收。保留受控发布和回退路径,减少多版本紧急补丁。
3. 当前仍按需全量解释正式范围历史,64 MiB/250000 是保护上限,不是无限扩展能力。增长逼近预算或刷新时间不足时,再设计可重放、可校验的增量统计事实与周期对账;需独立契约及迁移计划,不在本次热修中仓促加入。
4. 单独排查本次观察到的 claim-count 500 和其他业务慢请求;这些未被本次修复覆盖,不能宣称平台所有 bug 已消除。
诊断原始材料在主工作根的 Git 忽略路径 `local-data/diagnostics/dashboard-20261026/`,包含服务器只读聚合、快照状态、索引元数据及诊断源代码;凭据/会话文件不入库。获准临时容器程序已清理。

View File

@ -0,0 +1,39 @@
# 前端分层版本与批注列表验证
2026-10-22。本轮完成王子予在平台调优 2026-10-21 17:53 提出的本轮批注集中查看需求,同时对前几轮架构改造作受控对比。尚未推送或部署。
## 需求验收
[客户消息](https://applink.feishu.cn/client/chat/open?openChatId=oc_ff3c7f66ecfd607ffebd120f0113dbca&position=150):审核员希望在右侧集中看见本轮全部批注,不用逐片查找。现在保留当前目标的编辑卡片,新增带位置、范围、正文的完整列表;点击复用现有定位和编辑流程。验收按来源映射切片,避免跨来源同名 ID 串源;只读可查看,权限不放宽。
复用已经加载的草稿,没有新增请求、轮询、数据库或服务端统计。列表是当前轮次的已保存批注,未保存文字继续由编辑器管理。停用的整体意见及历史已解决意见仍按 ADR0008/0009 隐藏,不修改原标注或交付 JSON。
真实 Go HTTP + MemoryStore 两角色 34 场景通过,覆盖全范围意见、跨切片/来源、只读导航、清除刷新,以及原有草稿丢响应、版本竞争和新输入保留。168 项协议定向测试、19 个完整标注场景、9 个任务池场景、正式决定和权限浏览器回归通过。全量前端有 13 项原有失败,均在原 14 项基线内,未增加或放宽基线。类型、格式、构建和生产 dist 启动通过。
## 性能对比范围
固定已提交版本:改造前 `3221ac65`,架构改造后 `01eed769`。两者均不包含本轮新增列表,以免把功能新增混入重构比较。两个独立源码导出使用相同锁文件及内容完全相同、独立复制的依赖;无共享 node_modules。生产构建、同一浏览器、同一静态服务器策略、相同合成 API、相同视口;两对预热后交替顺序测量 20 对,每次新浏览器上下文。
计时从导航开始到登录密码输入框可见,不等待 networkidle。没有真实账号、客户数据或公司服务;该测试只能反映匿名启动,不代表登录后的看板、切片生成、视频播放、数据库吞吐或多人负载。
| 指标 | 改造前 | 架构改造后 |
| --- | ---: | ---: |
| 启动中位数 | 96.92 ms | 95.95 ms |
| 启动 p90(最近秩) | 123.45 ms | 112.66 ms |
| 登录可见时请求数 | 13 | 13 |
| 登录所需 JS gzip 正文字节 | 128,289 | 128,342 |
| 全部 JS 文件 gzip 合计(每文件 level 9) | 2,102,977 | 2,118,529 |
启动中位数仅约 1ms 差异,配对差异中位数为 −0.79ms,探索性 95% 区间为 −6.77 至 +4.67ms,不能据此宣布提速;p90 为单机单次会话样本,同样不作生产承诺。登录 JS 增 53 字节;全部 JS gzip 合计增 15,552 字节(约 0.74%),包含延迟加载的页面,不等于每次打开下载量。该对比没有测运行时内存、CPU 或后端资源,不能声称这些指标下降。单次构建时间受其他验证任务影响,不作为效率结论。
## 同条件正确性对比
同一生产组件夹具对两个版本执行 5 个相同场景:帧号 Esc 取消、Enter 单次提交、鼠标应用当前输入、秒数 Esc 取消、秒数 Tab 提交。旧版 2 通过、3 失败,优化版 5 全通过。失败对应两个明确缺陷:Esc 的失焦路径误应用取消值;Enter 同时走键盘和失焦导致重复回调。不是“测试更多所以质量更好”,也不能把这些修复单独归因于拆模块。
原先完整页面 E 面板偶发关闭的原因仍未确认;本轮 19 场景通过不代表该历史不稳定项已闭环。
## 证据与结论
证据位于本工作树忽略目录 `local-data/verification/frontend-architecture-comparison-20261021/` 和 `local-data/verification/wangziyu-review-list-20261021/`,保留原始采样、源码与依赖指纹、独立构建、预期/实际状态以及失败尝试。完整批注测试曾受生成文件触发 HMR 干扰,已经排除 local-data 监听并重跑通过;另一次夹具遗漏 focus 回调已补齐,均未作为产品质量差异计入。
当前结论是:已验证的输入正确性改善,登录启动速度基本持平,包体略增;职责分离和依赖门禁有助于控制后续修改范围,但开发效率及业务性能仍须按具体流程继续测量。本轮没有迁移和生产变更。

View File

@ -0,0 +1,42 @@
# 平台调优反馈:排查与首轮修复
日期:2026-10-22。需求来源:客户《数据平台测试》10月21日、22日记录,以及当日群内补充。候选基于 `93925d42`,隔离分支 `codex/platform-feedback-20261022`。本轮仅本地开发、合成数据验证,无推送、PR、部署或客户数据修改。
## 已实现
- 返工意见误变灰:默认动作状态补齐被误判为内容修改;旧基线缺少切片/动作快照被误判为目标已删除。先以失败用例复现,再统一语义状态比较、缺证据保持未修改。已修改仍需质检确认,不能代表已修复。
- 未解决旧意见仍能审核通过:真实 HTTP 用例在修改前返回201且推进任务。现在按钮、实际提交回调、服务端新决定均检查原退回意见;全部明确判为已解决才能通过。基线读取失败不再静默当作没有问题。未核对、明确未解决两种情况都被拦截,检查没有新增正式审查记录或推进任务。
- 提交提示:将“决定已完成”改为“提交完成”;普通及专注布局的连续审查状态采用16px字体,正式提交状态也放大。状态区域用中性色,不以绿色误表示错误或未确认结果。
- 报错可读性:主作业错误区补浅色/深色主题的文字与背景对比,保留具体业务说明及操作指引。
## 仍未闭环
| 项目 | 本轮证据与后续条件 |
| --- | --- |
| 10/21–22切片失败 | 既存本地样本中有源视频时长/帧率校验失败记录;候选已有MP4尾部量化兼容。源时钟定向测试及19个切片/返工作业场景通过。未对线上失败作业自动重试,客户原任务仍需按保存内容显式恢复并复验。 |
| 视频丢失、切换回来播放异常 | 三角色真实播放器加载失败/原URL重载、位置保持和未保存文字保护通过。没有客户发生时的媒体请求及播放状态,不能确认间歇故障根因或宣称根治。 |
| 连续审查等待过长 | 九个服务端接续场景通过;审核回执浏览器恢复通过。尚无客户该次提交到下一任务可用的分段时延证据,没有做猜测性的并发领取或取消权限校验。 |
| 不自动领取下一条 | 客户群已有上线回复;服务端定向测试通过,仍需原业务场景验收。 |
| 全部批注集中展示 | 基线已有实现,本轮未重做;生产交付/客户验收仍需核对。 |
| 突然只读 | 已核对周期权限请求失败会暂停写入的机制;本轮没有改变编辑权保护,客户当次任务已提交,具体原因仍待复现。 |
| 审核人员及流转追溯 | 用户指定低优先级待办,近期不实施;含普通列表、专注页和终审的人员/供应商/时间/轮次及流转记录。 |
| 有效时长、小时显示 | 客户注明非紧急,保留待办。 |
## 验证
- 前端业务定向84项通过:返工语义比较、按钮、实际提交回调、基线读取失败/任务切换、回执生命周期。提示最终调整后相关45项复跑通过。
- Go定向通过:未核对/未解决阻断、全部解决可通过、冻结确认、回执恢复、源时钟与MP4尾部量化、九个连续审查场景。
- `REWORK_BOUNDARY_BROWSER_TEST=1 go test ./internal/api -run '^TestReworkBoundaryBrowserReplica$' -count=1 -v`:真实浏览器与真实Go HTTP/MemoryStore,19个场景通过,包含恢复新草稿、失败重试、响应丢失、保存/刷新、立即撤销重做。
- 同一Go夹具以 `ANNOTATION_FEEDBACK_BROWSER_SCRIPT=tests/browser/annotation-media-recovery-replica.mjs` 执行:标注、质检、终审三角色均通过,页面无异常,不写标注数据。
- `REVIEW_DECISION_BROWSER_TEST=1 go test ./internal/api -run '^TestReviewDecisionControllerBrowserReplica$' -count=1 -v`:通过,正式写入一次,恢复不重复提交。
- 类型、架构门禁、完整格式检查、生产前端构建通过。
- 前端完整门禁仍报两项未登记失败:`review-target-interaction.test.mjs :: source guard: resident editor binds the selected target with a tier label` 与 `scope-cycle-inline-editor.test.mjs :: source guard: resident inline editor promotes into edit state on focus`。未修改的93925d42工作树运行相同门禁得到同名同原因失败,均为旧源码断言仍要求 `selectScope`,实际已使用 `editSelected`。未改业务代码迎合断言,也未更新已知失败清单。
- Go完整门禁另遇 `internal/worker :: TestDispatchTwoWorkersAndRestartDoNotDuplicateClaims`,断言为并发领取多出run-005。在未修改基线对该用例重复20次,同样复现;本轮未修改worker。其他失败属于门禁登记的旧用例。以上全量门禁均不宣称全绿。
当前5173/3001未运行;27018由SSH进程监听,未确认其为本地快照容器,故未连接或写入该端口。浏览器验证使用仅绑定回环地址、只允许夹具操作的隔离HTTP服务;与公司集群/BOS隔离。证据在工作树忽略目录 `local-data/test-results/`,不提交客户消息、账号或源数据。
## 发布前复核补充
双轴复核发现旧通用 `POST /api/human-items/{id}/decisions` 将质检调用归为acceptance,旧请求没有操作标识时会绕过按入口种类判定的门禁。真实HTTP夹具在修改前返回200并推进任务。现改为依据任务当前质检阶段强制检查,并加入 `TestGenericLegacyQualityApprovalRequiresResolvedPriorOpinions`,同时断言阻断后任务状态及版本不变。常规质检门禁及既有回执恢复定向测试均通过。
发布候选依赖已按当前package-lock独立安装,不再共享其他工作树的node_modules。实际dist通过生产启动测试,使用已有Edge浏览器;缺少Playwright默认浏览器的首次启动失败已如实保留,没有当作产品缺陷或跳过门禁。线上只读快照确认API/处理器仍为93925d42,Worker为63512da7;候选包含这三个组件源码祖先,仍需按发布流程完成计划及授权。

View File

@ -0,0 +1,19 @@
# Rework boundary and Enter repair — 2026-10-21
Base: deployed `fab5be57`. Implementation is local, on `codex/rework-boundary-20261021`.
The quick drag path previously ran raw action containment validation and local draft save, bypassing the server archive application. The fix validates the archive preview and sends original action pieces through the existing atomic application. Enter and mouse confirmation await authoritative persistence. A failed save retains choices and shows its error beside the controls; pending saves block duplicate application and cancellation. Persisted undo uses the pre-drag snapshot, and successful quick edits clear frozen drafts.
## Regression evidence
- The new unit regression against pristine `fab5be57` source fails all four left/right × pending/described cases with “舍弃片段不能包含当前有效标注”. Against the fix, all five tests pass, including rejection of an unsupported action status hidden in a discard interval.
- All 26 focused frontend checks pass.
- The real AnnotationOperations page is served by Vite and driven in headless Edge, against actual Go HTTP handlers and an isolated MemoryStore. Synthetic video, hardware timestamps and IMU data exercise frame validation. No customer or snapshot database is used. All 10 browser scenarios pass with zero page errors. Browser assertions cover left/right expansion, pending/described, first-pass/rework, button/Enter, persisted reload, visible failure/retry, delayed-save duplicate and cancel guards, lost-response reconciliation, persisted undo/redo, and Escape cancellation.
- Focused Go archive/application/restoration/legacy/permission contracts pass. No backend business contract or database migration is introduced.
- TypeScript, full frontend baseline gate, frontend/API/worker builds and production-bundle browser boot pass. The frontend gate retains 13 exact failures already registered in `tests/known-failures.json`; no new failure and no baseline change. These are unrelated navigation/project/automation source-shape assertions; the production-shell assertion is now passing.
Run the browser regression with `REWORK_BOUNDARY_BROWSER_TEST=1 go test ./internal/api -run '^TestReworkBoundaryBrowserReplica$' -count=1 -v`. It needs local ffmpeg, npm dependencies and a Playwright browser (Edge by default, or BROWSER_EXECUTABLE). Evidence is written under untracked `local-data/test-results/rework-boundary/`. The pristine unit comparison is in its `baseline/result.log`; the final frontend summary lists exact existing failures and full logs.
## Compatibility and limits
The existing endpoints, domain statuses, permission checks, archive authority, CAS and response-loss recovery remain in use. Other workflows and the separate delivery-format work are not included. This prevents future inadvertent invalidation through the affected shortcut; it does not reconstruct classifications already explicitly changed by a user. Production rollout and customer acceptance remain outstanding.

View File

@ -8,12 +8,26 @@
- 首交任务数以成功送检的 part 任务为单位去重,归入首次提交发生的期间。送检次数包含首次提交和返工重提。未确认、处理中或失败的提交不计入。
- 使用提交记录的业务标记判断首交/返工,不按版本号判断。旧记录优先使用对应标注版本的 `IsRework/CountsAsOutput`,再使用明确的事件类型及正向 `outputDelta`;缺证据单列待核对。
- 视频时长分为首交、返工两项,不再显示成工时。从对应切片版本的区间并集或新提交保存的视频时长读取;没有可靠范围时单列待核对,不拿旧 `durationSeconds` 猜工时或视频时长。
- 审查视频时长与标注员的首交/返工视频时长分开统计,归给作出决定的审核员本人,按决定完成时间入账。每次已完成的正式通过或退回决定都计入一次,复审重复计入;时长取被审那一版切片的区间并集,舍弃切片也计入,双路相机不重复计时。质检决定使用工作记录中的被审版本;终审按每个来源分别求并集后相加,退回与通过都取该来源在终审决定前最近一次质检通过的版本。返工送检后系统会清理被替换的旧切片版本,因此按以下优先级取被审版本的时长:决定提交时冻结的时长(质检记在该次工作记录上,终审按来源逐个记在审核操作上;2026-10-27 起的新决定)→ 该版本现存切片的区间并集 → 该版本送检记录保存的视频时长(2026-09-29 起的送检有此记录,为切片时长之和,同版本切片互不重叠时与并集相等)。三者都没有、来源无法关联或旧交付包验收路径的决定单列待核对,不计为 0;2026-09-09 至 09-29 间被替换的版本切片已被清理且无送检时长,属于无法恢复的历史缺口。调试任务与其他统计一样默认排除。
- 审查产出按审核员作出的正式审查决定次数计:质检与终审的通过、退回都计入,复审重复计入;终审聚合决定按一次计,与来源数量无关。只含调试来源的决定不计入;旧交付包验收路径的决定计入次数,时长待核对。审查产出与标注员产出分开,不相加。
- 审核员(质检、终审)本人看板的“产出”和“视频时长”卡片分别显示审查产出与审查视频时长,代替恒为 0 的标注员口径;团队表格与按天明细新增“审查产出”“审查视频时长”列。
- 成员角色按账号目录解析,团队表格显示成员角色,按角色筛选只保留该角色成员。
- “收到打回”与审核员的“审查操作”分列。质检打回归给当时的标注作者;终审打回归给被审来源版本的质检审核员。无法确认归属的记录单列,不按当前领取人分摊。
- 打回次数按实际完成的正式决定计数;任务数按关联 part 去重。问题对象数、涉及切片数和覆盖视频时长按轮次累计。终审包含多个来源/审核人时,总计决定次数去重,不能简单相加各成员的次数。
- 每次决定内,同一 `scope/source/slice/action` 的旧意见只计一个对象,展开仍完整显示每条原文,不解释句意或自动选最后一条。整体意见没有具体时间范围,不展开成整个视频。
- 片内时间按来源及切片隔离,原视频时间按来源求并集,去除动作与外层切片的覆盖重叠;双路相机不重复计时。不能可靠定位的历史意见明确提示,不伪装成完整的零值。
主表保持精简,默认折叠,展开后按成员展示:成员、供应商、角色、产出、质量通过率、返工次数和视频时长;点击成员再加载按天产出/通过率/返工及分页打回意见。质量通过率只以已通过或退回的决定为分母,返工暂只显示次数,不推算返工率。评分、排名、结算公式仍未启用。
主表保持精简,默认折叠,展开后按成员展示:成员、供应商、角色、产出、质量通过率、返工次数、视频时长、审查产出和审查视频时长;点击成员再加载按天产出/通过率/返工及分页打回意见。质量通过率只以已通过或退回的决定为分母,返工暂只显示次数,不推算返工率。评分、排名、结算公式仍未启用。
## 生产总览与供应商产出中的审核员
`/api/dashboard/annotation-output`(生产总览、供应商产出)在原标注员区块之外,按同样的卡片和表格样式增加质检员、验收员区块;页面布局与原标注员口径不变,时长统一以小时显示并保留一位小数(不足 0.1 小时的非零时长显示“<0.1 小时”)。
- 审查次数按正式通过或退回决定所涉 part 计:质检每次决定计一个 part;终审一次决定涉及多个 part 时按 part 分别计入,同一决定内同一 part 的多个任务只计一次。同一 part 返工后再次审查重复计入,并在“其中复审”中单列(同环节对该 part 已有更早决定即为复审,判断使用全部历史,不受所选期间影响)。草稿保存、未完成或失败的操作、调试任务不计入。
- 审查时长取被审 part 的整段视频时长,与同页标注员各项时长口径一致;它与“人员工作记录统计”面板中按被审切片版本求并集的审查视频时长不是同一口径,两处数值可能不同。任一被审 part 缺少可靠时长时合计显示“—”,不按 0 计。
- 审核员归属其账号所属供应商,不改变被审 part 的生产供应商归属;供应商管理员只看到本供应商审核员的审查。人员筛选同时适用于标注员与审核员。
- 人均日产出/人均日审查=期间视频小时 ÷ 人天。人天为一名人员在一个本地自然日至少有一次本角色正式工作:标注员为成功送检(含返工重提,分子仍为首次提交视频小时),审核员为正式审查决定。它不是实际工时,平台没有记录登录或在线时长。
- 这些事实随共享快照一起计算(每 2 分钟),查询只在内存中过滤与汇总;快照 schema 升至 `annotation-output-facts/v2`,发布后首次访问会重新生成一次快照。新增读取字段仅为审核员身份(`reviewerUserId`、`userId`、`actorUserId`),不新增集合、索引或写入业务数据。
## 接口、权限与兼容
@ -25,7 +39,7 @@
新送检工作记录增加稳定提交 ID、首交/返工标记、视频时长;同一提交的工作记录使用固定 ID 并仅首次写入,重试保留原来的成功时间和归属。读取旧事件时批量关联相关任务的提交历史,同一任务、同一正式版本按最早成功事件去重,避免跨日期重试再次入账。
查询使用区间和组织/供应商/项目过滤、字段投影与批量关联,正文仅在展开明细时加载。未增加线上索引或持久化统计表。
查询使用区间和组织/供应商/项目过滤、字段投影与批量关联,正文仅在展开明细时加载。本人范围只关联本人决定涉及的任务;切片区间只为送检提交、本人质检决定和终审来源读取,不随打回归属关联扩大。未增加线上索引或持久化统计表。
## 数据边界与回退

View File

@ -0,0 +1,118 @@
package api
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"testing"
"time"
"droboticflow/internal/domain"
"droboticflow/internal/store"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func accountPhoneReuseContract(t *testing.T, ds store.Store) {
t.Helper()
accountDeleteContract(t, ds)
users, err := ds.ListUsers(t.Context())
if err != nil {
t.Fatal(err)
}
var old, admin, disabled domain.UserRecord
for _, u := range users {
if u.DeletedAt != "" {
old = u
}
if u.Role == domain.UserRoleAdmin {
admin = u
}
if u.DeletedAt == "" && u.Status == domain.UserStatusDisabled {
disabled = u
}
}
if old.ID == "" || admin.ID == "" || disabled.ID == "" {
t.Fatal("missing contract identities")
}
h := NewHandler(ds, nil)
cookie := loginSession(t, h, admin.Phone, "delete-test-password")
create := func(phone string) *httptest.ResponseRecorder {
body, _ := json.Marshal(map[string]any{"name": "Replacement manager", "phone": phone, "role": domain.UserRoleSupplierManager, "supplierId": old.SupplierID, "password": "replacement-test-password"})
return serveJSONWithCookie(h, http.MethodPost, "/api/users", body, cookie)
}
if rr := create(disabled.Phone); rr.Code != 400 {
t.Fatalf("disabled identity released its phone: %d %s", rr.Code, rr.Body.String())
}
rr := create(old.Phone)
if rr.Code != 201 {
t.Fatalf("deleted phone not reusable: %d %s", rr.Code, rr.Body.String())
}
var replacement domain.UserRecord
if err := json.Unmarshal(rr.Body.Bytes(), &replacement); err != nil {
t.Fatal(err)
}
if replacement.ID == old.ID || replacement.Role != domain.UserRoleSupplierManager {
t.Fatal("old audit identity reused")
}
if rr := create(old.Phone); rr.Code != 400 {
t.Fatalf("duplicate live account admitted: %d", rr.Code)
}
found, err := ds.FindUserByPhone(t.Context(), old.Phone)
if err != nil || found == nil || found.ID != replacement.ID {
t.Fatalf("login lookup selected old audit identity: %+v %v", found, err)
}
newCookie := loginSession(t, h, old.Phone, "replacement-test-password")
rr = serveJSONWithCookie(h, http.MethodGet, "/api/auth/me", nil, newCookie)
if rr.Code != 200 {
t.Fatalf("replacement login failed: %d", rr.Code)
}
body, _ := json.Marshal(map[string]string{"phone": old.Phone, "password": "delete-test-password"})
rr = serveJSONWithCookie(h, http.MethodPost, "/api/auth/login", body, nil)
if rr.Code != 401 {
t.Fatalf("old password admitted: %d", rr.Code)
}
saved, err := ds.GetUser(t.Context(), old.ID)
if err != nil || saved == nil || saved.DeletedAt != old.DeletedAt || saved.Phone != old.Phone || saved.PasswordHash != "" {
t.Fatal("old audit identity changed")
}
historical, err := ds.GetHumanItem(t.Context(), "delete-"+string(domain.HumanItemAccepted))
if err != nil || historical.AssignedUserID != old.ID {
t.Fatal("old history transferred to new account")
}
members, err := ds.ListProjectMembers(t.Context(), old.OrganizationID, "")
if err != nil {
t.Fatal(err)
}
for _, m := range members {
if m.UserID == old.ID {
t.Fatal("deleted membership revived")
}
}
}
func TestDeletedAccountPhoneReuse(t *testing.T) { accountPhoneReuseContract(t, store.NewMemoryStore()) }
func TestDeletedAccountPhoneReuseMongo(t *testing.T) {
if os.Getenv("ACCOUNT_PHONE_REUSE_LOCAL_MONGO") != "1" {
t.Skip("isolated local Mongo only")
}
ctx, cancel := context.WithTimeout(t.Context(), time.Minute)
defer cancel()
client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://127.0.0.1:27018/?directConnection=true"))
if err != nil {
t.Fatal(err)
}
defer client.Disconnect(context.Background())
db := client.Database(fmt.Sprintf("phone_reuse_api_test_%d", time.Now().UnixNano()))
defer db.Drop(context.Background())
ds := store.NewMongoStore(db)
if err = ds.MigrateUserPhoneIndex(ctx); err != nil {
t.Fatal(err)
}
accountPhoneReuseContract(t, ds)
}

View File

@ -0,0 +1,153 @@
package api
import (
"droboticflow/internal/domain"
"droboticflow/internal/store"
"encoding/json"
"net/http"
"net/url"
"strings"
"time"
)
func (s *server) adminAnnotationPoolQuery(w http.ResponseWriter, r *http.Request) (store.AdminAnnotationPoolQuery, bool) {
p, _ := currentPrincipal(r)
if !annotationManager(p) {
renderJSON(w, 403, map[string]string{"error": "administrator annotation pool required"})
return store.AdminAnnotationPoolQuery{}, false
}
v := r.URL.Query()
q := store.AdminAnnotationPoolQuery{ProjectID: v.Get("projectId"), WorkflowID: v.Get("workflowId"), RunID: v.Get("runId"), NodeID: v.Get("nodeId"), ArtifactID: v.Get("artifactId"), Capture: strings.TrimSpace(v.Get("capture")), Search: strings.TrimSpace(v.Get("search")), Category: v.Get("category"), Pool: v.Get("pool"), UserID: p.User.ID}
if q.Pool == "" {
q.Pool = "available"
}
if q.Pool != "available" && q.Pool != "mine" && q.Pool != "all" {
renderValidationError(w, validationIssue{Field: "pool", Message: "pool must be available, mine, or all"})
return q, false
}
if len(q.Search) > 256 || len(q.Capture) > 1024 {
renderValidationError(w, validationIssue{Field: "search", Message: "search is too long"})
return q, false
}
switch q.Category {
case "", "all", "available", "pending_slice", "slicing", "pending_annotation", "rework", "pending_quality", "rework_review", "pending_acceptance", "accepted", "submitting", "submission_conflicted":
default:
renderValidationError(w, validationIssue{Field: "category", Message: "unknown task category"})
return q, false
}
limit, issue := parseDatabaseBrowserPageQuery(v.Get("limit"), "limit", 200, 1, 200)
if issue != nil {
renderValidationError(w, *issue)
return q, false
}
q.Limit = limit
cursor, err := decodeHumanItemCursor(v.Get("cursor"))
if err != nil {
renderValidationError(w, validationIssue{Field: "cursor", Message: "cursor must be a valid opaque human item cursor"})
return q, false
}
q.Cursor = cursor
return q, true
}
// Counts use the existing bounded per-actor cache; list criteria and cursor are
// deliberately excluded from its key. Permission is checked before cache hits.
func (s *server) adminAnnotationCounts(w http.ResponseWriter, r *http.Request, q store.AdminAnnotationPoolQuery) (*store.AdminAnnotationPoolCounts, bool) {
countRequest := r.Clone(r.Context())
copyURL := *r.URL
countRequest.URL = &copyURL
values := url.Values{}
for _, key := range []string{"projectId", "workflowId", "runId", "nodeId", "artifactId", "capture"} {
if value := r.URL.Query().Get(key); value != "" {
values.Set(key, value)
}
}
countRequest.URL.RawQuery = values.Encode()
key := s.dashboardCacheKey("admin-annotation-pool-counts", countRequest)
if entry := s.dashboardResponses.get(key); entry != nil {
var counts store.AdminAnnotationPoolCounts
if json.Unmarshal(entry.body, &counts) == nil {
return &counts, true
}
}
counts, err := s.store.CountAdminAnnotationPools(r.Context(), s.requestOrganizationID(r), q)
if err != nil {
renderError(w, 500, err)
return nil, false
}
body, err := json.Marshal(counts)
if err != nil {
renderError(w, 500, err)
return nil, false
}
// Five seconds bounds cross-pod/background changes. Existing API write
// invalidation clears these entries immediately on this process.
s.dashboardResponses.setTTL(key, 200, "application/json", body, 5*time.Second)
return &counts, true
}
func (s *server) handleAdminAnnotationPool(w http.ResponseWriter, r *http.Request) {
q, ok := s.adminAnnotationPoolQuery(w, r)
if !ok {
return
}
counts, ok := s.adminAnnotationCounts(w, r, q)
if !ok {
return
}
size := q.Limit
q.Limit++
items, err := s.store.ListAdminAnnotationPool(r.Context(), s.requestOrganizationID(r), q)
if err != nil {
renderError(w, 500, err)
return
}
page := humanItemListPage{Items: items}
if len(items) > size {
page.Items = items[:size]
last := page.Items[len(page.Items)-1]
page.HasMore = true
page.NextCursor = encodeHumanItemCursor(&domain.HumanItemCursor{CreatedAt: last.CreatedAt, ID: last.ID})
}
resolver := newHumanDisplayResolver(s)
for i := range page.Items {
if err := resolver.enrichItem(r.Context(), &page.Items[i]); err != nil {
renderError(w, 500, err)
return
}
}
// Join only the page's task IDs, never an independent newest-first window.
tasks := []annotationTaskListRecord{}
ids := []string{}
seen := map[string]bool{}
for _, item := range page.Items {
if item.LegacyTaskID != "" && !seen[item.LegacyTaskID] {
ids = append(ids, item.LegacyTaskID)
seen[item.LegacyTaskID] = true
}
}
if len(ids) > 0 {
ctx, err := s.withHumanListReads(r.Context(), s.requestOrganizationID(r), q.ProjectID, page.Items)
if err != nil {
renderError(w, 500, err)
return
}
request := r.Clone(ctx)
u := *r.URL
request.URL = &u
v := url.Values{"ids": {strings.Join(ids, ",")}}
if q.ProjectID != "" {
v.Set("projectId", q.ProjectID)
}
request.URL.RawQuery = v.Encode()
joined := s.readAnnotationTasksList(w, request)
if joined == nil {
return
}
tasks = joined.Items
}
result := page.response()
result["counts"] = counts
result["tasks"] = tasks
renderJSON(w, 200, result)
}

View File

@ -0,0 +1,70 @@
package api
import (
"context"
"droboticflow/internal/domain"
"fmt"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"testing"
"time"
)
func TestAdminAnnotationPoolBrowserReplica(t *testing.T) {
if os.Getenv("ADMIN_POOL_BROWSER_TEST") != "1" {
t.Skip("local browser opt-in")
}
f := randomClaimFixture(t, 610)
admin := domain.UserRecord{ID: "pool-admin", OrganizationID: domain.DefaultOrganizationID, Role: domain.UserRoleAdmin, Status: domain.UserStatusActive}
_ = f.store.UpsertUser(f.ctx, admin)
_ = f.store.UpsertOrganizationPlugin(f.ctx, domain.OrganizationPluginRecord{OrganizationID: domain.DefaultOrganizationID, PluginID: "annotation-qa", Enabled: true, Config: defaultAnnotationPluginConfig()})
for n := 0; n < 610; n++ {
item, _ := f.store.GetHumanItem(f.ctx, fmt.Sprintf("random-item-%d", n))
task, _ := f.store.GetAnnotationTask(f.ctx, item.LegacyTaskID)
stamp := fmt.Sprintf("2026-10-21T00:%02d:%02dZ", n/60, n%60)
item.CreatedAt = stamp
task.CreatedAt = stamp
item.Subject.ItemKey = fmt.Sprintf("CAPTURE_%04d_camera_left_part0001.mp4", n)
task.ArtifactID = item.Subject.ItemKey
task.SourceVideo = item.Subject.ItemKey
item.Subject.Payload = map[string]any{"sliceWorkflow": map[string]any{"status": "editing", "phase": "slice"}}
if n >= 606 {
item.CurrentStageID = "quality_review"
task.Status = domain.AnnotationTaskStatusPendingQuality
}
_ = f.store.UpsertHumanItem(f.ctx, *item)
_ = f.store.UpsertAnnotationTask(f.ctx, *task)
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/auth/me" {
renderJSON(w, 200, map[string]any{"user": sanitizeUser(&admin), "organization": domain.OrganizationRecord{ID: domain.DefaultOrganizationID, Status: domain.OrganizationStatusActive}})
return
}
if r.URL.Path == "/__fixture/claim" {
result := f.request(http.MethodPost, "/api/plugins/annotation-qa/random-claims", f.annotator, map[string]any{"projectId": f.task.ProjectID, "requestId": "admin-pool-fixture-claim-01"})
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(result.Code)
_, _ = w.Write(result.Body.Bytes())
return
}
if r.Method != "GET" && r.Method != "HEAD" {
renderJSON(w, 403, map[string]string{"error": "fixture read-only"})
return
}
r = r.WithContext(context.WithValue(r.Context(), authContextKey{}, &authenticatedPrincipal{User: admin, Organization: domain.OrganizationRecord{ID: domain.DefaultOrganizationID, Status: domain.OrganizationStatusActive}}))
f.handler.ServeHTTP(w, r)
}))
defer server.Close()
ctx, cancel := context.WithTimeout(t.Context(), 3*time.Minute)
defer cancel()
cmd := exec.CommandContext(ctx, "node", "tests/browser/admin-annotation-pool-replica.mjs")
cmd.Dir = "../.."
cmd.Env = append(os.Environ(), "ADMIN_POOL_API="+server.URL)
out, err := cmd.CombinedOutput()
t.Log(string(out))
if err != nil {
t.Fatal(err)
}
}

View File

@ -0,0 +1,146 @@
package api
import (
"context"
"droboticflow/internal/domain"
"droboticflow/internal/store"
"encoding/json"
"errors"
"fmt"
"net/http"
"net/url"
"testing"
"time"
)
func TestAdminAnnotationPoolTotalsAndOldPageSearch(t *testing.T) {
f := newFanyuV5FourStageFixture(t, domain.FanyuV5StageSemanticAnnotation, domain.HumanItemReady, "", 0)
f.batch.Stages = append(f.batch.Stages, domain.HumanStageDefinition{ID: "quality_review", Kind: domain.HumanStageReview, RequiredRole: "quality_reviewer"})
if err := f.store.UpsertHumanBatch(f.ctx, f.batch); err != nil {
t.Fatal(err)
}
admin := domain.UserRecord{ID: "pool-admin", OrganizationID: domain.DefaultOrganizationID, Role: domain.UserRoleAdmin, Status: domain.UserStatusActive}
for i := 0; i < 610; i++ {
item := f.item
item.ID = fmt.Sprintf("pool-%04d", i)
item.LegacyTaskID = fmt.Sprintf("pool-task-%04d", i)
item.CreatedAt = fmt.Sprintf("2026-10-21T00:%02d:%02dZ", i/60, i%60)
item.Subject.ItemKey = fmt.Sprintf("CAPTURE_%04d_camera_left_part0001.mp4", i)
if i >= 606 {
item.CurrentStageID = "quality_review"
}
if e := f.store.UpsertHumanItem(f.ctx, item); e != nil {
t.Fatal(e)
}
}
base := "/api/plugins/annotation-qa/admin-pool?projectId=" + f.task.ProjectID
read := func(path string) map[string]json.RawMessage {
t.Helper()
r := f.request(http.MethodGet, path, admin, nil)
if r.Code != 200 {
t.Fatalf("%d %s", r.Code, r.Body)
}
var d map[string]json.RawMessage
if e := json.Unmarshal(r.Body.Bytes(), &d); e != nil {
t.Fatal(e)
}
return d
}
first := read(base + "&pool=available&limit=100")
var counts struct{ Available, All, Mine int }
_ = json.Unmarshal(first["counts"], &counts)
if counts.Available != 607 || counts.All != 611 || counts.Mine != 0 {
t.Fatalf("counts %+v", counts)
}
var items []domain.HumanItemRecord
_ = json.Unmarshal(first["items"], &items)
if len(items) != 100 {
t.Fatalf("page=%d", len(items))
}
var cursor string
_ = json.Unmarshal(first["nextCursor"], &cursor)
if cursor == "" {
t.Fatal("missing cursor")
}
second := read(base + "&pool=available&limit=100&cursor=" + url.QueryEscape(cursor))
var nextCounts struct{ Available, All, Mine int }
_ = json.Unmarshal(second["counts"], &nextCounts)
if counts != nextCounts {
t.Fatal("paging changed totals")
}
old := read(base + "&pool=available&search=CAPTURE_0001")
_ = json.Unmarshal(old["items"], &items)
if len(items) != 1 || items[0].ID != "pool-0001" {
t.Fatalf("old search %+v", items)
}
_ = json.Unmarshal(old["counts"], &nextCounts)
if counts != nextCounts {
t.Fatal("search changed badges")
}
denied := f.request(http.MethodGet, base+"&pool=all", f.annotator, nil)
if denied.Code != 403 {
t.Fatalf("annotator=%d", denied.Code)
}
}
type adminPoolProbe struct {
store.Store
counts, pages int
fail bool
}
func (s *adminPoolProbe) CountAdminAnnotationPools(ctx context.Context, org string, q store.AdminAnnotationPoolQuery) (store.AdminAnnotationPoolCounts, error) {
s.counts++
if s.fail {
return store.AdminAnnotationPoolCounts{}, errors.New("synthetic unavailable")
}
return s.Store.CountAdminAnnotationPools(ctx, org, q)
}
func (s *adminPoolProbe) ListAdminAnnotationPool(ctx context.Context, org string, q store.AdminAnnotationPoolQuery) ([]domain.HumanItemRecord, error) {
s.pages++
if q.Limit > 201 {
panic("unbounded administrator page")
}
return s.Store.ListAdminAnnotationPool(ctx, org, q)
}
func TestAdminAnnotationPoolFreshPermissionsCacheAndFailures(t *testing.T) {
f := newFanyuV5FourStageFixture(t, domain.FanyuV5StageSemanticAnnotation, domain.HumanItemReady, "", 0)
probe := &adminPoolProbe{Store: f.store}
now := time.Now()
srv := &server{store: probe, dashboardResponses: dashboardResponseCache{now: func() time.Time { return now }}}
f.handler = http.HandlerFunc(srv.handleAdminAnnotationPool)
admin := domain.UserRecord{ID: "admin-cache", OrganizationID: domain.DefaultOrganizationID, Role: domain.UserRoleAdmin, Status: domain.UserStatusActive}
base := "/api/plugins/annotation-qa/admin-pool?projectId=" + f.task.ProjectID
for _, suffix := range []string{"&pool=available", "&pool=all&search=missing&category=pending_slice", "&pool=mine"} {
r := f.request(http.MethodGet, base+suffix, admin, nil)
if r.Code != 200 {
t.Fatalf("%d %s", r.Code, r.Body)
}
}
if probe.counts != 1 || probe.pages != 3 {
t.Fatalf("counts=%d pages=%d", probe.counts, probe.pages)
}
for _, role := range []domain.UserRole{domain.UserRoleAnnotator, domain.UserRoleQualityReviewer, domain.UserRoleAcceptanceReviewer, domain.UserRoleSupplierManager} {
p := admin
p.Role = role
if r := f.request(http.MethodGet, base, p, nil); r.Code != 403 {
t.Fatalf("cached role %s=%d", role, r.Code)
}
}
for _, suffix := range []string{"&cursor=broken", "&limit=bad", "&pool=reviewed", "&category=unknown"} {
if r := f.request(http.MethodGet, base+suffix, admin, nil); r.Code != 422 && r.Code != 400 {
t.Fatalf("invalid %s=%d", suffix, r.Code)
}
}
// A failed total read must never masquerade as zero, even when the old page
// would fit. Cross-process/background changes become visible after 5 seconds.
now = now.Add(6 * time.Second)
probe.fail = true
r := f.request(http.MethodGet, base, admin, nil)
if r.Code != 500 {
t.Fatalf("failed aggregate=%d %s", r.Code, r.Body)
}
if probe.pages != 3 {
t.Fatal("page fetched after count failure")
}
}

View File

@ -7,6 +7,59 @@ import (
"droboticflow/internal/domain"
)
type annotationEditorState struct {
Phase string `json:"phase"`
CanConfirmDescriptions bool `json:"canConfirmDescriptions"`
CanRecoverGeneration bool `json:"canRecoverGeneration"`
CanReconfirmSavedGeneration bool `json:"canReconfirmSavedGeneration"`
}
// Use the same persisted evidence as the write gate, including legacy tasks
// without sliceWorkflow. Restored draft actions do not prove generation ready.
func (server *server) annotationEditorStateForTask(ctx context.Context, task domain.AnnotationTaskRecord, item *domain.HumanItemRecord) (annotationEditorState, error) {
ready, err := server.legacyAnnotationConfirmationReady(ctx, task, item)
state := annotationEditorState{Phase: "slice", CanConfirmDescriptions: ready}
if err != nil {
return state, err
}
if ready {
state.Phase = "annotate"
return state, nil
}
if item != nil {
state.CanRecoverGeneration = legacyGenerationRecoveryCandidate(task, item)
workflow := mapValue(item.Subject.Payload["sliceWorkflow"])
status := stringValue(workflow["status"])
jobStatus := ""
if item.ProcessingJob != nil {
jobStatus = item.ProcessingJob.Status
}
if status == "failed" || jobStatus == "failed" {
state.CanReconfirmSavedGeneration = failedSliceReconfirmationAvailable(item)
return state, nil
}
if status == "processing" || status == "queued" || status == "running" || status == "materializing" || jobStatus == "queued" || jobStatus == "running" || jobStatus == "processing" || item.PendingCombinedSubmissionID != "" {
state.Phase = "materializing"
}
}
return state, nil
}
func legacyGenerationRecoveryCandidate(task domain.AnnotationTaskRecord, item *domain.HumanItemRecord) bool {
if item == nil || !task.SourceIsFullVideo || item.PendingCombinedSubmissionID != "" || item.Subject.Payload["sliceWorkflow"] != nil {
return false
}
if item.ProcessingJob != nil && item.ProcessingJob.Status != "completed" {
return false
}
for _, workspace := range []map[string]any{annotationWorkspacePayload(item), mapValue(item.Subject.Payload["annotationDraft"])} {
if len(decodeMaterializationValue[domain.SliceSegment](workspace["sliceSegments"])) > 0 && len(decodeMaterializationValue[domain.AnnotationSegment](workspace["segments"])) > 0 {
return true
}
}
return false
}
// legacyAnnotationConfirmationReady recognizes persisted pre-workflow evidence
// for tasks whose old records do not have sliceWorkflow. It never mutates the
// item; the caller remains responsible for checking the submitted version.

View File

@ -0,0 +1,52 @@
package api
import (
"encoding/json"
"testing"
"time"
"droboticflow/internal/domain"
)
func TestAnnotationDetailSharesDescriptionConfirmationReadiness(t *testing.T) {
for _, tc := range []struct {
name, status, phase, expected string
ready bool
}{
{"completed", "completed", "annotate", "annotate", true},
{"failed-with-old-phase", "failed", "annotate", "slice", false},
{"failed-with-draft", "failed", "", "slice", false},
{"inconsistent-completed", "completed", "slice", "slice", false},
{"processing", "processing", "slice", "materializing", false},
{"legacy-confirmed", "", "", "annotate", true},
} {
t.Run(tc.name, func(t *testing.T) {
f := continuousFixture(t)
item := f.item
item.Subject.Payload["annotationWorkspace"] = map[string]any{"sliceSegments": []domain.SliceSegment{{ID: "s", Name: "测试切片", Start: 0, End: 10}}, "segments": []domain.AnnotationSegment{{ID: "a", Start: 0, End: 10, Status: "pending"}}}
if tc.status != "" {
item.Subject.Payload["sliceWorkflow"] = map[string]any{"status": tc.status, "phase": tc.phase, "leaseExpiresAt": time.Now().Add(time.Hour).UTC().Format(time.RFC3339Nano)}
}
if err := f.store.UpsertHumanItem(f.ctx, item); err != nil {
t.Fatal(err)
}
r := f.request("GET", "/api/plugins/annotation-qa/tasks/"+f.task.ID, f.annotator, nil)
if r.Code != 200 {
t.Fatalf("detail: %d %s", r.Code, r.Body.String())
}
var body struct {
EditorState annotationEditorState `json:"editorState"`
}
if err := json.Unmarshal(r.Body.Bytes(), &body); err != nil {
t.Fatal(err)
}
if body.EditorState.Phase != tc.expected || body.EditorState.CanConfirmDescriptions != tc.ready {
t.Fatalf("state=%+v", body.EditorState)
}
ready, err := (&server{store: f.store}).legacyAnnotationConfirmationReady(f.ctx, f.task, &item)
if err != nil || ready != body.EditorState.CanConfirmDescriptions {
t.Fatalf("gate differs: %v %v", ready, err)
}
})
}
}

View File

@ -0,0 +1,55 @@
package api
import (
"encoding/json"
"strings"
"testing"
"droboticflow/internal/domain"
)
func TestNewUnnamedSplitCannotBecomeHistoricalNameAtSubmission(t *testing.T) {
f := continuousFixture(t)
seedCompletedMaterialization(t, f, 1)
item, _ := f.store.GetHumanItem(f.ctx, f.item.ID)
item.Subject.Payload["sliceWorkflow"] = map[string]any{"status": "completed", "phase": "annotate"}
item.Subject.Payload["annotationWorkspace"] = map[string]any{
"sliceSegments": []domain.SliceSegment{{ID: "parent", Name: "测试切片", Start: 0, End: 10}},
"segments": []domain.AnnotationSegment{{ID: "a", Start: 0, End: 10, Status: "pending"}},
}
if err := f.store.UpsertHumanItem(f.ctx, *item); err != nil {
t.Fatal(err)
}
path := "/api/plugins/annotation-qa/tasks/" + f.task.ID
split := []domain.SliceSegment{{ID: "child-a", SourceIDs: []string{"parent"}, Start: 0, End: 5}, {ID: "child-b", SourceIDs: []string{"parent"}, Start: 5, End: 10}}
actions := []domain.AnnotationSegment{{ID: "a1", Start: 0, End: 5, Status: "pending"}, {ID: "a2", Start: 5, End: 10, Status: "pending"}}
r := f.request("POST", path+"/slice-edit-applications", f.annotator, sliceEditApplicationInput{ApplicationID: "unnamed-split", combinedAnnotationDraftInput: combinedAnnotationDraftInput{Version: item.Version, SliceSegments: split, Segments: actions}})
if r.Code != 200 {
t.Fatalf("save split: %d %s", r.Code, r.Body.String())
}
var got struct {
Workspace combinedAnnotationDraftInput `json:"workspace"`
}
if err := json.Unmarshal(r.Body.Bytes(), &got); err != nil {
t.Fatal(err)
}
item, _ = f.store.GetHumanItem(f.ctx, item.ID)
got.Workspace.Version = item.Version
r = f.request("POST", path+"/combined-submissions", f.annotator, got.Workspace)
if r.Code != 400 || !strings.Contains(r.Body.String(), "slice_dictionary_name_required") {
t.Fatalf("unnamed new slices submitted: %d %s", r.Code, r.Body.String())
}
unchanged, _ := f.store.GetHumanItem(f.ctx, item.ID)
if unchanged.Version != item.Version || unchanged.CurrentStageID != item.CurrentStageID {
t.Fatal("rejection changed the task")
}
// Naming the slices must preserve the anonymous pending actions, which are
// valid classification results and must never be silently marked invalid.
for i := range got.Workspace.SliceSegments {
got.Workspace.SliceSegments[i].Name = "测试切片"
}
r = f.request("POST", path+"/combined-submissions", f.annotator, got.Workspace)
if r.Code != 201 {
t.Fatalf("named split: %d %s", r.Code, r.Body.String())
}
}

View File

@ -2,6 +2,7 @@ package api
import (
"fmt"
"reflect"
"sort"
"strings"
@ -19,7 +20,7 @@ func consolidateInvalidActionSlices(input []domain.SliceSegment, actions []domai
eligible := map[string]int{}
for i := 1; i < len(ordered)-1; i++ {
l, c, r := ordered[i-1], ordered[i], ordered[i+1]
if l.IsDiscarded() || c.IsDiscarded() || r.IsDiscarded() || strings.TrimSpace(c.Name) == "" || l.Name != c.Name || r.Name != c.Name || l.End != c.Start || c.End != r.Start {
if l.IsDiscarded() || c.IsDiscarded() || r.IsDiscarded() || strings.TrimSpace(c.Name) == "" || l.Name != c.Name || r.Name != c.Name || !reflect.DeepEqual(l.Dictionary, c.Dictionary) || !reflect.DeepEqual(r.Dictionary, c.Dictionary) || l.End != c.Start || c.End != r.Start {
continue
}
cursor, has, valid := c.Start, false, true

View File

@ -14,6 +14,7 @@ import (
)
type annotationLabelInput struct {
Tags []string `json:"tags"`
RequiresNote *bool `json:"requiresNote"`
Kind string `json:"kind"`
Code string `json:"code"`
@ -46,7 +47,7 @@ func annotationLabelRevision(item domain.AnnotationLabelRecord) domain.Annotatio
return domain.AnnotationLabelRevision{
RequiresNote: item.RequiresNote, Version: item.Version, Code: item.Code, Name: item.Name, Category: item.Category,
Description: item.Description, Example: item.Example, Color: item.Color,
Aliases: append([]string(nil), item.Aliases...), Status: item.Status,
Tags: append([]string(nil), item.Tags...), Aliases: append([]string(nil), item.Aliases...), Status: item.Status,
UpdatedByUserID: item.UpdatedByUserID, UpdatedAt: item.UpdatedAt,
}
}
@ -79,6 +80,19 @@ func normalizeAnnotationLabelInput(input annotationLabelInput) annotationLabelIn
seen[value] = struct{}{}
aliases = append(aliases, value)
}
if input.Tags != nil {
tags := []string{}
seenTags := map[string]bool{}
for _, tag := range input.Tags {
tag = strings.TrimSpace(tag)
if tag != "" && !seenTags[tag] {
tags = append(tags, tag)
seenTags[tag] = true
}
}
input.Tags = tags
}
input.Aliases = aliases
return input
}
@ -97,6 +111,15 @@ func validateAnnotationLabelInput(input annotationLabelInput) []validationIssue
if input.Status != domain.AnnotationLabelActive && input.Status != domain.AnnotationLabelInactive {
issues = append(issues, validationIssue{Field: "status", Message: "标签状态必须是 active 或 inactive"})
}
if len(input.Tags) > 32 {
issues = append(issues, validationIssue{Field: "tags", Message: "每个词条最多 32 个标签"})
}
for _, tag := range input.Tags {
if len([]rune(tag)) > 80 {
issues = append(issues, validationIssue{Field: "tags", Message: "标签最多 80 字"})
break
}
}
return issues
}
@ -165,7 +188,7 @@ func (server *server) handleAnnotationLabelCreate(writer http.ResponseWriter, re
ID: domain.NewID("annotation-label"), OrganizationID: organizationID,
Code: payload.Code, Name: payload.Name, Category: payload.Category,
Description: payload.Description, Example: payload.Example, Color: payload.Color,
Aliases: payload.Aliases, Status: payload.Status, Version: 1,
Tags: payload.Tags, Aliases: payload.Aliases, Status: payload.Status, Version: 1,
CreatedByUserID: actorID, UpdatedByUserID: actorID, CreatedAt: now, UpdatedAt: now,
}
item.History = []domain.AnnotationLabelRevision{annotationLabelRevision(item)}
@ -225,6 +248,9 @@ func (server *server) handleAnnotationLabelPatch(writer http.ResponseWriter, req
item.Code, item.Name, item.Category = payload.Code, payload.Name, payload.Category
item.Description, item.Example, item.Color = payload.Description, payload.Example, payload.Color
item.Aliases, item.Status = payload.Aliases, payload.Status
if payload.Tags != nil {
item.Tags = payload.Tags
}
item.Version++
item.UpdatedByUserID, item.UpdatedAt = principal.User.ID, nowISO()
item.History = append(item.History, annotationLabelRevision(*item))
@ -267,13 +293,16 @@ func (server *server) handleProjectAnnotationLabelsGet(writer http.ResponseWrite
if selection == nil {
selection = &domain.ProjectAnnotationLabelsRecord{ID: "project-annotation-labels-" + project.ID, OrganizationID: project.OrganizationID, ProjectID: project.ID, LabelIDs: []string{}}
}
if projectDictionaryKind(request) != "action" {
{
available, e := server.store.ListAnnotationLabels(request.Context(), project.OrganizationID)
if e != nil {
renderError(writer, 500, e)
return
}
ids := selection.SliceLabelIDs
ids := selection.LabelIDs
if projectDictionaryKind(request) == "slice" {
ids = selection.SliceLabelIDs
}
if projectDictionaryKind(request) == "discard_reason" {
ids = selection.DiscardReasonLabelIDs
}
@ -290,14 +319,20 @@ func (server *server) handleProjectAnnotationLabelsGet(writer http.ResponseWrite
items = append(items, label)
}
}
response := map[string]any{"projectId": project.ID, "labelIds": ids, "items": items}
response := struct {
domain.ProjectAnnotationLabelsRecord
Items []domain.AnnotationLabelRecord `json:"items"`
AllowManual *bool `json:"allowManual,omitempty"`
}{ProjectAnnotationLabelsRecord: *selection, Items: items}
response.LabelIDs = ids
if projectDictionaryKind(request) == "discard_reason" {
response["allowManual"] = selection.OrganizationID == project.OrganizationID && len(ids) == 0
manual := selection.OrganizationID == project.OrganizationID && len(ids) == 0
response.AllowManual = &manual
}
renderJSON(writer, http.StatusOK, response)
return
}
renderJSON(writer, http.StatusOK, selection)
}
func (server *server) handleProjectAnnotationLabelsPut(writer http.ResponseWriter, request *http.Request) {
@ -399,7 +434,7 @@ func (server *server) projectAnnotationLabelDefinitions(ctx context.Context, org
if !exists || item.Status != domain.AnnotationLabelActive || dictionaryKind(item.Kind) != "action" {
continue
}
labels = append(labels, domain.SemanticLabelDefinition{ID: item.Code, Name: item.Name, Description: item.Description, Aliases: item.Aliases, Color: item.Color})
labels = append(labels, domain.SemanticLabelDefinition{Dictionary: item.Selection(), ID: item.Code, Name: item.Name, Description: item.Description, Aliases: item.Aliases, Color: item.Color})
}
return labels, nil
}

View File

@ -72,6 +72,15 @@ func (s *server) tryAnnotationMetadataRevision(ctx context.Context, task domain.
if e != nil {
return nil, nil
}
if manifest["originalSource"] == nil {
if record, err := s.store.GetCaptureRecordByRunID(ctx, task.RunID); err != nil {
return nil, err
} else if original, err := annotationRevisionOriginalSource(run, record); err != nil {
return nil, err
} else if original != nil {
manifest["originalSource"] = original
}
}
entries := listValue(manifest["items"])
if len(entries) != len(slices) {
return nil, nil
@ -134,6 +143,7 @@ func (s *server) tryAnnotationMetadataRevision(ctx context.Context, task domain.
}
entry["sourceSliceId"], entry["sliceName"], entry["deliveryDisposition"] = slice.ID, slice.Name, sliceDisposition(*slice)
synchronized["sliceName"], synchronized["deliveryDisposition"] = slice.Name, sliceDisposition(*slice)
entry["dictionary"], synchronized["dictionary"] = slice.Dictionary, slice.Dictionary
delete(entry, "discardEvidence")
delete(synchronized, "discardEvidence")
if slice.IsDiscarded() {
@ -146,9 +156,9 @@ func (s *server) tryAnnotationMetadataRevision(ctx context.Context, task domain.
if slice.IsDiscarded() || segment.End <= slice.Start || segment.Start >= slice.End {
continue
}
actions = append(actions, map[string]any{"id": segment.ID, "status": effectiveActionStatus(segment), "isInvalid": segment.IsInvalid, "description": segment.Description, "descriptionTemplateId": segment.LabelID, "phrases": segment.Phrases, "source": segment.Source, "confidence": segment.Confidence, "clipStartMs": (segment.Start - slice.Start) * 1000, "clipEndMs": (segment.End - slice.Start) * 1000, "startTimestampUs": segment.StartTimestampUS, "endTimestampUs": segment.EndTimestampUS})
actions = append(actions, map[string]any{"id": segment.ID, "status": effectiveActionStatus(segment), "isInvalid": segment.IsInvalid, "description": segment.Description, "descriptionTemplateId": segment.LabelID, "dictionary": segment.Dictionary, "phrases": segment.Phrases, "source": segment.Source, "confidence": segment.Confidence, "clipStartMs": (segment.Start - slice.Start) * 1000, "clipEndMs": (segment.End - slice.Start) * 1000, "startTimestampUs": segment.StartTimestampUS, "endTimestampUs": segment.EndTimestampUS})
}
annotation := map[string]any{"schemaVersion": "droboticflow.timestamped-description/v3", "boundaryContract": "hardware-frame-v1", "sliceId": entry["sliceId"], "sliceName": slice.Name, "window": map[string]any{"startTimestampUs": slice.StartTimestampUS, "endTimestampUs": slice.EndTimestampUS, "semantics": "half-open"}, "descriptions": actions, "actions": actions}
annotation := map[string]any{"schemaVersion": "droboticflow.timestamped-description/v3", "boundaryContract": "hardware-frame-v1", "sliceId": entry["sliceId"], "sliceName": slice.Name, "dictionary": slice.Dictionary, "window": map[string]any{"startTimestampUs": slice.StartTimestampUS, "endTimestampUs": slice.EndTimestampUS, "semantics": "half-open"}, "descriptions": actions, "actions": actions}
if e = os.MkdirAll(filepath.Join(stage, oldDir), 0755); e != nil {
return nil, e
}

View File

@ -6,6 +6,7 @@ import (
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"log"
"math"
@ -978,7 +979,7 @@ func (server *server) handleAnnotationTaskGet(writer http.ResponseWriter, reques
}
}
if j := linkedHumanItem.ProcessingJob; j != nil {
status["progress"] = map[string]any{"stage": j.Stage, "completedSlices": j.CompletedSlices, "totalSlices": j.TotalSlices, "startedAt": j.CreatedAt, "attempt": j.Attempt}
status["progress"] = map[string]any{"stage": server.annotationProcessingStage(request.Context(), item.OrganizationID, j), "completedSlices": j.CompletedSlices, "totalSlices": j.TotalSlices, "startedAt": j.CreatedAt, "attempt": j.Attempt}
if durableAnnotationActive(linkedHumanItem) {
status["needsRefresh"] = false
}
@ -1090,7 +1091,12 @@ func (server *server) handleAnnotationTaskGet(writer http.ResponseWriter, reques
}
linkedHumanItem.CanClaim = &claimable
}
renderJSON(writer, http.StatusOK, map[string]any{"task": item, "sliceRevisions": sliceRevisions, "revisions": revisions, "reviews": reviews, "humanItem": linkedHumanItem})
editorState, readinessErr := server.annotationEditorStateForTask(request.Context(), *item, linkedHumanItem)
if readinessErr != nil {
renderError(writer, http.StatusInternalServerError, readinessErr)
return
}
renderJSON(writer, http.StatusOK, map[string]any{"task": item, "sliceRevisions": sliceRevisions, "revisions": revisions, "reviews": reviews, "humanItem": linkedHumanItem, "editorState": editorState})
}
func annotationFrameTimelinePayload(camera string, rows []hardwarePTSRow) (map[string]any, error) {
@ -1415,12 +1421,15 @@ type combinedAnnotationDraftInput struct {
}
type sliceConfirmationInput struct {
RetryFailed bool `json:"retryFailed,omitempty"`
RetryWorkspace map[string]any `json:"retryWorkspace,omitempty"`
Version int `json:"version"`
SliceSegments []domain.SliceSegment `json:"sliceSegments"`
SliceOperations []domain.SliceOperation `json:"sliceOperations,omitempty"`
ChangeReason string `json:"changeReason,omitempty"`
ReconfirmSaved bool `json:"reconfirmSaved,omitempty"`
SnapshotContentSHA string `json:"snapshotContentSha256,omitempty"`
RecoverExisting bool `json:"recoverExisting,omitempty"`
RetryFailed bool `json:"retryFailed,omitempty"`
RetryWorkspace map[string]any `json:"retryWorkspace,omitempty"`
Version int `json:"version"`
SliceSegments []domain.SliceSegment `json:"sliceSegments"`
SliceOperations []domain.SliceOperation `json:"sliceOperations,omitempty"`
ChangeReason string `json:"changeReason,omitempty"`
}
type sliceResetInput struct {
@ -1802,20 +1811,25 @@ func (server *server) handleCombinedAnnotationDraftPut(writer http.ResponseWrite
if !decodeJSON(writer, request, &payload) {
return
}
validationItem, baselineErr := sliceEditBaseline(humanItem)
if baselineErr != nil {
renderJSON(writer, http.StatusConflict, map[string]string{"error": baselineErr.Error()})
return
}
issues := validateSliceSegments(payload.SliceSegments, *task)
canonical, dispositionIssues := server.canonicalSliceDispositions(request.Context(), humanItem, payload.SliceSegments, payload.normalizedOperations())
canonical, dispositionIssues := server.canonicalSliceDispositions(request.Context(), validationItem, payload.SliceSegments, payload.normalizedOperations())
if len(dispositionIssues) > 0 {
renderValidationError(writer, dispositionIssues...)
return
}
payload.SliceSegments = canonical
payload.Operations = domain.CheckpointSliceOperations(decodeMaterializationValue[domain.SliceSegment](annotationWorkspacePayload(humanItem)["sliceSegments"]), canonical, payload.normalizedOperations())
payload.Operations = domain.CheckpointSliceOperations(decodeMaterializationValue[domain.SliceSegment](annotationWorkspacePayload(validationItem)["sliceSegments"]), canonical, payload.normalizedOperations())
payload.SliceOperations = nil
if server.continuousActionsForItem(request.Context(), humanItem) {
payload.Segments = fillPendingActionGaps(payload.SliceSegments, payload.Segments)
payload.Statuses = continuousSliceStatuses(payload.SliceSegments, payload.Segments)
issues = append(issues, validateContinuousActions(payload.SliceSegments, payload.Segments, false)...)
issues = append(issues, server.continuousActionFrameIssues(request.Context(), *task, humanItem, payload.Segments)...)
issues = append(issues, server.continuousActionFrameIssues(request.Context(), *task, validationItem, payload.Segments)...)
}
issues = append(issues, validateDraftCombinedAnnotationEvents(payload.Segments, *task)...)
issues = append(issues, validateDraftStatuses(payload.Statuses, payload.SliceSegments)...)
@ -1825,6 +1839,7 @@ func (server *server) handleCombinedAnnotationDraftPut(writer http.ResponseWrite
return
}
template = annotationTemplateWithTaskSnapshot(template, task)
issues = append(issues, server.canonicalActionDictionaries(request.Context(), validationItem, payload.Segments)...)
issues = append(issues, validateDraftFanyuV5CombinedSemanticSegments(payload.Segments, payload.SliceSegments, template)...)
if len(issues) > 0 {
renderValidationError(writer, issues...)
@ -1834,6 +1849,7 @@ func (server *server) handleCombinedAnnotationDraftPut(writer http.ResponseWrite
renderJSON(writer, http.StatusConflict, map[string]string{"error": "Human item version conflict", "code": "version_conflict"})
return
}
copyClassificationRecoveryState(humanItem, validationItem)
probe.validated()
now := nowISO()
draft := map[string]any{
@ -1971,6 +1987,7 @@ func (server *server) handleAnnotationConfirmationCreate(writer http.ResponseWri
return
}
template = annotationTemplateWithTaskSnapshot(template, task)
issues = append(issues, server.canonicalActionDictionaries(request.Context(), item, payload.Segments)...)
issues = append(issues, validateDraftFanyuV5CombinedSemanticSegments(payload.Segments, slices, template)...)
if len(issues) > 0 {
renderValidationError(writer, issues...)
@ -2031,7 +2048,11 @@ func (server *server) finishSliceConfirmation(ctx context.Context, taskID string
Revision: revision, Segments: slices, Operations: payload.SliceOperations, AuthorUserID: actorUserID,
ChangeReason: strings.TrimSpace(payload.ChangeReason), CreatedAt: nowISO(),
}
materialization, processErr := server.materializeCombinedSliceSubmission(ctx, *task, *item, combinedAnnotationSubmissionInput{sliceOnly: true, SliceSegments: slices, Segments: []domain.AnnotationSegment{}}, revision)
processErr := validateSliceCompletionSnapshot(item, payload)
var materialization *humanSliceMaterialization
if processErr == nil {
materialization, processErr = server.materializeCombinedSliceSubmission(ctx, *task, *item, combinedAnnotationSubmissionInput{sliceOnly: true, SliceSegments: slices, Segments: []domain.AnnotationSegment{}}, revision)
}
close(done)
<-heartbeatStopped
if _, durable := ctx.Value(annotationJobLeaseKey{}).(annotationJobLease); durable && (ctx.Err() != nil || server.assertAnnotationLease(ctx) != nil) {
@ -2051,6 +2072,9 @@ func (server *server) finishSliceConfirmation(ctx context.Context, taskID string
if strings.TrimSpace(fmt.Sprint(workflow["requestId"])) != requestID || stringValue(workflow["status"]) != "processing" {
return
}
if processErr == nil {
processErr = validateSliceCompletionSnapshot(current, payload)
}
if processErr == nil {
processErr = server.store.UpsertSliceRevision(ctx, sliceRevision)
}
@ -2062,11 +2086,13 @@ func (server *server) finishSliceConfirmation(ctx context.Context, taskID string
workflow["updatedAt"] = now
} else {
log.Printf("[api] finishSliceConfirmation succeeded for task=%s item=%s requestID=%s revision=%d slices=%d", taskID, itemID, requestID, revision, len(materialization.Slices))
materialization.Payload["confirmationSnapshotSha256"] = payload.SnapshotContentSHA
workflow = map[string]any{
"status": "completed", "phase": "annotate", "requestId": requestID, "revision": revision,
"boundaryContract": "hardware-frame-v1",
"sliceCount": len(materialization.Slices), "materialization": materialization.Payload, "updatedAt": now,
}
workflow["snapshotContentSha256"] = payload.SnapshotContentSHA
if payload.RetryFailed {
workflow["retryOf"] = w["retryOf"]
}
@ -2074,6 +2100,7 @@ func (server *server) finishSliceConfirmation(ctx context.Context, taskID string
"sliceSegments": materialization.Slices, "operations": payload.SliceOperations,
"segments": []domain.AnnotationSegment{}, "statuses": pendingSliceStatuses(materialization.Slices),
"userId": actorUserID, "savedAt": now,
"confirmationSnapshotSha256": payload.SnapshotContentSHA,
}
if payload.RetryFailed {
// Retry only regenerated physical media; retained semantic work and
@ -2134,8 +2161,50 @@ func (server *server) handleSliceConfirmationCreate(writer http.ResponseWriter,
return
}
payload.RetryWorkspace = nil // Private durable snapshot is always server-owned.
payload.SnapshotContentSHA = ""
if payload.Version != item.Version {
renderJSON(writer, http.StatusConflict, map[string]string{"error": "Human item version conflict", "code": "version_conflict"})
return
}
if payload.ReconfirmSaved && (payload.RetryFailed || payload.RecoverExisting) {
renderJSON(writer, http.StatusBadRequest, map[string]string{"error": "请选择一种切片生成操作", "code": "slice_confirmation_mode_invalid"})
return
}
var retained combinedAnnotationDraftInput
if payload.RetryFailed {
if payload.ReconfirmSaved {
workflow := mapValue(item.Subject.Payload["sliceWorkflow"])
job := item.ProcessingJob
if stringValue(workflow["status"]) != "failed" || (job != nil && (job.Status == "queued" || job.Status == "running" || job.Kind != "slice")) {
renderJSON(writer, http.StatusConflict, map[string]string{"error": "只有生成失败且未在后台处理的任务可以重新确认", "code": "slice_retry_unavailable"})
return
}
var recoveryErr error
payload, retained, recoveryErr = retainedSliceRecoveryInput(item, payload.Version)
if recoveryErr != nil {
renderJSON(writer, http.StatusUnprocessableEntity, map[string]string{"error": recoveryErr.Error(), "code": "slice_retry_workspace_invalid"})
return
}
payload.ReconfirmSaved = true
payload.ChangeReason = "按当前已保存内容重新确认切片生成"
} else if payload.RecoverExisting {
ready, err := server.legacyAnnotationConfirmationReady(request.Context(), *task, item)
if err != nil {
renderError(writer, http.StatusInternalServerError, err)
return
}
if ready || !legacyGenerationRecoveryCandidate(*task, item) {
renderJSON(writer, http.StatusConflict, map[string]string{"error": "当前任务不需要恢复历史切片生成,已有内容仍保留", "code": "slice_recovery_unavailable"})
return
}
var recoveryErr error
payload, retained, recoveryErr = retainedSliceRecoveryInput(item, payload.Version)
if recoveryErr != nil {
renderJSON(writer, http.StatusUnprocessableEntity, map[string]string{"error": recoveryErr.Error(), "code": "slice_retry_workspace_invalid"})
return
}
payload.RecoverExisting = true
payload.ChangeReason = "恢复历史切片生成并保留已有标注"
} else if payload.RetryFailed {
workflow := mapValue(item.Subject.Payload["sliceWorkflow"])
job := item.ProcessingJob
if stringValue(workflow["status"]) != "failed" || (job != nil && (job.Status == "queued" || job.Status == "running" || job.Kind != "slice")) {
@ -2143,12 +2212,22 @@ func (server *server) handleSliceConfirmationCreate(writer http.ResponseWriter,
return
}
var retryErr error
recoverExisting := boolValue(workflow["recoverExisting"])
payload, retained, retryErr = retainedSliceRetryInput(item, payload.Version)
payload.RecoverExisting = recoverExisting
if retryErr != nil {
if errors.Is(retryErr, errSliceSnapshotConflict) {
renderJSON(writer, http.StatusConflict, map[string]string{"error": retryErr.Error(), "code": "slice_retry_snapshot_conflict"})
return
}
renderJSON(writer, http.StatusUnprocessableEntity, map[string]string{"error": retryErr.Error(), "code": "slice_retry_workspace_invalid"})
return
}
}
if !payload.RetryFailed && stringValue(mapValue(item.Subject.Payload["sliceWorkflow"])["status"]) == "failed" {
renderJSON(writer, http.StatusConflict, map[string]string{"error": "请按已保存内容重新确认生成,已有标注仍保留", "code": "slice_retry_snapshot_conflict"})
return
}
payload.SliceSegments = descriptionOnlySlices(payload.SliceSegments)
if payload.Version != item.Version {
renderJSON(writer, http.StatusConflict, map[string]string{"error": "Human item version conflict", "code": "version_conflict"})
@ -2218,6 +2297,39 @@ func (server *server) handleSliceConfirmationCreate(writer http.ResponseWriter,
}
probe.validated()
requestID := domain.NewID("slice-materialization")
contentSHA, hashErr := sliceWorkspaceContentSHA(sliceConfirmationWorkspace(payload))
if hashErr != nil {
renderError(writer, http.StatusUnprocessableEntity, hashErr)
return
}
payload.SnapshotContentSHA = contentSHA
if payload.ReconfirmSaved {
prior, archiveErr := compactSliceWorkspaceForArchive(annotationWorkspacePayload(item))
if archiveErr != nil {
renderError(writer, http.StatusUnprocessableEntity, archiveErr)
return
}
archive := map[string]any{"requestId": requestID, "selectedContentSha256": contentSHA, "previousWorkspace": prior}
if job := item.ProcessingJob; job != nil && job.Input != nil {
if raw := mapValue(job.Input["retryWorkspace"]); len(raw) > 0 {
previousInput, err := compactSliceWorkspaceForArchive(raw)
if err != nil {
renderError(writer, http.StatusUnprocessableEntity, err)
return
}
archive["previousFailedSnapshot"] = previousInput
}
}
encoded, err := json.Marshal(archive)
if err != nil || len(encoded) > 8<<20 {
renderJSON(writer, http.StatusRequestEntityTooLarge, map[string]string{"error": "历史快照过大,无法完整保留,未开始重新生成", "code": "slice_snapshot_archive_too_large"})
return
}
if err := server.appendHumanEvent(request.Context(), *item, "slice_reconfirmation_snapshot_prepared", principal.User.ID, archive); err != nil {
renderError(writer, http.StatusInternalServerError, err)
return
}
}
now := nowISO()
if item.Subject.Payload == nil {
item.Subject.Payload = map[string]any{}
@ -2225,13 +2337,17 @@ func (server *server) handleSliceConfirmationCreate(writer http.ResponseWriter,
item.Subject.Payload["sliceWorkflow"] = map[string]any{
"status": "processing", "phase": "materializing", "requestId": requestID, "revision": revision,
"sliceCount": len(payload.SliceSegments), "startedAt": now,
"leaseExpiresAt": time.Now().Add(sliceLeaseDuration).Format(time.RFC3339Nano),
"snapshotContentSha256": contentSHA,
"leaseExpiresAt": time.Now().Add(sliceLeaseDuration).Format(time.RFC3339Nano),
}
if payload.RetryFailed {
// Keep the original snapshot intact while queued/running; all workbench
// mutations remain blocked by the existing processing guard.
workflow := mapValue(item.Subject.Payload["sliceWorkflow"])
workflow["retryOf"] = retryOf
if payload.RecoverExisting {
workflow["recoverExisting"] = true
}
item.Subject.Payload["sliceWorkflow"] = workflow
} else {
delete(item.Subject.Payload, "annotationDraft")
@ -2463,6 +2579,10 @@ func (server *server) handleCombinedAnnotationSubmission(writer http.ResponseWri
return
}
template = annotationTemplateWithTaskSnapshot(template, task)
if issues := server.canonicalActionDictionaries(request.Context(), humanItem, payload.Segments); len(issues) > 0 {
renderValidationError(writer, issues...)
return
}
if issues := validateFanyuV5CombinedSemanticSegmentsWithRequireDesc(payload.Segments, payload.SliceSegments, template, !continuousActions); len(issues) > 0 {
renderValidationError(writer, issues...)
return
@ -3071,13 +3191,18 @@ func (server *server) handleAnnotationRevisionCreate(writer http.ResponseWriter,
renderJSON(writer, http.StatusForbidden, map[string]string{"error": "annotator assignment is required"})
return
}
if _, claimed := server.requireLegacyHumanItemClaim(writer, request, *task, "semantic_annotation"); !claimed {
humanItem, claimed := server.requireLegacyHumanItemClaim(writer, request, *task, "semantic_annotation")
if !claimed {
return
}
var payload annotationRevisionInput
if !decodeJSON(writer, request, &payload) {
return
}
if issues := server.canonicalActionDictionaries(request.Context(), humanItem, payload.Segments); len(issues) > 0 {
renderValidationError(writer, issues...)
return
}
if issues := validateAnnotationSegments(payload.Segments, *task); len(issues) > 0 {
renderValidationError(writer, issues...)
return
@ -3555,7 +3680,7 @@ func (server *server) handleFanyuV5CombinedQualityReview(writer http.ResponseWri
}
_ = server.appendHumanEvent(request.Context(), *item, "combined_quality_"+string(payload.Decision), principal.User.ID, map[string]any{"reviewedRevision": review.Revision, "reasonCodes": review.ReasonCodes, "problemSliceIds": review.ProblemSliceIDs, "comment": review.Comment})
taskMode, isTemp := server.resolveTaskRunMode(request.Context(), task.RunID)
_ = server.store.UpsertAnnotationWorklogEvent(request.Context(), domain.AnnotationWorklogEventRecord{ID: domain.NewID("annotation-worklog"), OrganizationID: task.OrganizationID, ProjectID: task.ProjectID, TaskID: task.ID, UserID: principal.User.ID, SupplierID: task.SupplierID, WorkgroupID: task.WorkgroupID, VendorID: task.VendorID, EventType: "quality_" + string(payload.Decision), Mode: taskMode, Temporary: isTemp, Revision: review.Revision, TemplateVersion: task.TemplateVersion, DurationSeconds: payload.DurationSeconds, OutputDelta: 0, CreatedAt: now})
_ = server.store.UpsertAnnotationWorklogEvent(request.Context(), domain.AnnotationWorklogEventRecord{ID: domain.NewID("annotation-worklog"), OrganizationID: task.OrganizationID, ProjectID: task.ProjectID, TaskID: task.ID, UserID: principal.User.ID, SupplierID: task.SupplierID, WorkgroupID: task.WorkgroupID, VendorID: task.VendorID, EventType: "quality_" + string(payload.Decision), Mode: taskMode, Temporary: isTemp, Revision: review.Revision, TemplateVersion: task.TemplateVersion, DurationSeconds: payload.DurationSeconds, VideoSeconds: server.reviewedRevisionVideoSeconds(request.Context(), task.OrganizationID, task.ID, review.Revision), OutputDelta: 0, CreatedAt: now})
if payload.Decision == domain.AnnotationReviewApproved {
if err := server.flushHumanReleasePartitions(request.Context(), batch, false); err != nil {
renderError(writer, http.StatusInternalServerError, err)

View File

@ -0,0 +1,79 @@
package api
import (
"encoding/json"
"fmt"
"log"
"path"
"sort"
"strings"
"droboticflow/internal/domain"
)
// Freeze only credential-free facts from the run's original input, never the
// current workflow configuration or a later source-directory listing.
func annotationOriginalSource(run *domain.RunRecord, capture *domain.CaptureRecord) (map[string]any, error) {
if run == nil || capture == nil || len(run.Inputs) == 0 {
return nil, nil
}
if capture.RunID != run.ID || capture.OrganizationID != run.OrganizationID {
return nil, fmt.Errorf("original capture scope mismatch")
}
root := capture.SourcePrefix
if root == "" {
root = capture.Prefix
}
var result map[string]any
for _, input := range run.Inputs {
ref := input.Ref
if ref == nil || ref.Provider != domain.StorageProviderBOS || ref.Kind != domain.ArtifactRefKindDirectory || ref.Bucket != capture.Bucket || ref.Prefix != root {
continue
}
if ref.CaptureUnit != "" && ref.CaptureUnit != capture.CaptureID {
continue
}
if len(ref.ObjectKeys) == 0 {
continue
} // Older non-selected directory inputs need explicit recovery.
if root == "" || path.Clean(root) != root || strings.HasPrefix(root, "/") || strings.Contains(root, "\\") {
return nil, fmt.Errorf("unsafe original capture prefix")
}
keys := append([]string(nil), ref.ObjectKeys...)
sort.Strings(keys)
for i, key := range keys {
if !strings.HasPrefix(key, root+"/") || path.Clean(key) != key || strings.Contains(key, "\\") || i > 0 && key == keys[i-1] {
return nil, fmt.Errorf("unsafe original capture object selection")
}
relative := strings.TrimPrefix(key, root+"/")
if strings.Contains(relative, "/") && !strings.HasPrefix(key, capture.Prefix+"/") {
return nil, fmt.Errorf("original capture selection includes another part")
}
}
candidate := map[string]any{"provider": string(ref.Provider), "bucket": ref.Bucket, "prefix": root, "partPrefix": capture.Prefix, "captureUnit": capture.CaptureID, "objectKeys": keys, "snapshotFingerprint": ref.VersionID}
if result != nil {
a, _ := json.Marshal(result)
b, _ := json.Marshal(candidate)
if string(a) != string(b) {
return nil, fmt.Errorf("ambiguous original capture inputs")
}
}
result = candidate
}
return result, nil
}
// Source provenance is optional for legacy materialization, but mandatory at
// v4 export. Never invent a part selection from an ambiguous capture input.
// Tenant/run violations are still fatal; storage failures remain with callers.
func annotationRevisionOriginalSource(run *domain.RunRecord, capture *domain.CaptureRecord) (map[string]any, error) {
source, err := annotationOriginalSource(run, capture)
if err == nil {
return source, nil
}
if run != nil && capture != nil && (capture.RunID != run.ID || capture.OrganizationID != run.OrganizationID) {
return nil, err
}
log.Printf("[api] original_source_evidence_unavailable reason=%s", err)
return nil, nil
}

View File

@ -0,0 +1,238 @@
package api
import (
"encoding/json"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"droboticflow/internal/domain"
)
func consumerSourceRef() *domain.ArtifactReference {
root := "incoming/capture-A"
keys := []string{root + "/meta_capture-A.json", root + "/capture-A_calibration_camera.yaml", root + "/capture-A_calibration_imu.yaml", root + "/part0001/capture-A_imu_part0001.csv"}
for _, side := range []string{"left", "right"} {
keys = append(keys, root+"/part0001/capture-A_camera_"+side+"_part0001.mp4", root+"/part0001/capture-A_camera_"+side+"_part0001_pts.csv")
}
return &domain.ArtifactReference{Provider: domain.StorageProviderBOS, Kind: domain.ArtifactRefKindDirectory, ConnectionID: "PRIVATE_CONNECTION", Bucket: "synthetic-original", Prefix: root, CaptureUnit: "capture-A/part0001", ObjectKeys: keys, VersionID: "frozen-input-version"}
}
func TestAnnotationOriginalSourceUsesFrozenInputSelection(t *testing.T) {
ref := consumerSourceRef()
run := &domain.RunRecord{ID: "run", OrganizationID: "org", Inputs: map[string]domain.ArtifactPayload{"raw": {Ref: ref}}}
capture := &domain.CaptureRecord{RunID: "run", OrganizationID: "org", Bucket: ref.Bucket, Prefix: ref.Prefix + "/part0001", SourcePrefix: ref.Prefix, CaptureID: ref.CaptureUnit, SourceKeys: []string{"later-current-directory-must-not-be-used"}}
source, err := annotationOriginalSource(run, capture)
if err != nil {
t.Fatal(err)
}
if len(source["objectKeys"].([]string)) != 8 {
t.Fatal("frozen source selection lost")
}
if _, ok := source["connectionId"]; ok {
t.Fatal("private connection leaked")
}
ref.ObjectKeys = append(ref.ObjectKeys, ref.Prefix+"/part0002/foreign.mp4")
if _, err = annotationOriginalSource(run, capture); err == nil {
t.Fatal("foreign part accepted")
}
ref.ObjectKeys = ref.ObjectKeys[:len(ref.ObjectKeys)-1]
capture.OrganizationID = "other"
if _, err = annotationOriginalSource(run, capture); err == nil {
t.Fatal("foreign organization accepted")
}
}
// Retain real synthetic source data for the independent Python consumer test.
func TestDeliveryV4ConsumerReplicaFixture(t *testing.T) {
parent := os.Getenv("FANYU_DELIVERY_CONSUMER_FIXTURE_DIR")
if parent == "" {
t.Skip("opt-in retained synthetic fixture")
}
if !filepath.IsAbs(parent) {
t.Fatal("absolute local output required")
}
for _, sourceIndex := range []bool{false, true} {
mode := "physical"
if sourceIndex {
mode = "source-index"
}
t.Run(mode, func(t *testing.T) {
f, temp := annotationProcessorFixture(t)
dir := filepath.Join(parent, mode)
root := filepath.Join(dir, "workspace")
if err := os.MkdirAll(dir, 0700); err != nil {
t.Fatal(err)
}
if err := copyAnnotationTree(f.ctx, temp, root); err != nil {
t.Fatal(err)
}
run, _ := f.store.GetRun(f.ctx, f.task.RunID)
for i := range run.NodeExecutions {
for j := range run.NodeExecutions[i].InputBindings {
for k := range run.NodeExecutions[i].InputBindings[j].Artifacts {
a := &run.NodeExecutions[i].InputBindings[j].Artifacts[k]
a.Path = filepath.Join(root, filepath.Base(a.Path))
}
}
}
ref := consumerSourceRef()
run.Inputs = map[string]domain.ArtifactPayload{"raw": {Ref: ref}}
if err := f.store.UpsertRun(f.ctx, *run); err != nil {
t.Fatal(err)
}
capture := domain.CaptureRecord{ID: "consumer-capture", RunID: run.ID, OrganizationID: run.OrganizationID, Bucket: ref.Bucket, Prefix: ref.Prefix + "/part0001", SourcePrefix: ref.Prefix, CaptureID: ref.CaptureUnit}
if err := f.store.UpsertCaptureRecord(f.ctx, capture); err != nil {
t.Fatal(err)
}
base := &domain.DictionarySelection{ID: "base", Kind: "slice", Version: 2, Code: "operate", Name: "物品操作", Tags: []string{"商品", "手部"}}
reason := &domain.DictionarySelection{ID: "discard", Kind: "discard_reason", Version: 1, Code: "occluded", Name: "遮挡", Tags: []string{"不可用"}}
action := &domain.DictionarySelection{ID: "pick", Kind: "action", Version: 3, Code: "pick", Name: "拿取", Tags: []string{"抓取"}}
input := combinedAnnotationSubmissionInput{SliceSegments: []domain.SliceSegment{{ID: "first", Name: base.Name, Dictionary: base, Start: 0, End: 1}, {ID: "discard", Start: 1, End: 3, DeliveryDisposition: "discard", DiscardReason: "遮挡", DiscardReasonDictionary: reason}, {ID: "last", Name: base.Name, Dictionary: base, Start: 3, End: 4}}, Segments: []domain.AnnotationSegment{{ID: "blank", Start: 0, End: .5}, {ID: "pick", Start: .5, End: 1, Description: "拿起杯子", Dictionary: action}, {ID: "invalid", Start: 3, End: 3.5, IsInvalid: true}, {ID: "free-text", Start: 3.5, End: 4, Description: "人工补充描述"}}}
s := &server{store: f.store, workspaceMountPath: root, annotationSourceIndex: sourceIndex}
result, err := s.materializeCombinedSliceSubmission(f.ctx, f.task, f.item, input, 1)
if err != nil {
t.Fatal(err)
}
if !sourceIndex {
before, err := os.ReadFile(filepath.Join(result.Artifact.Path, "revision-manifest.json"))
if err != nil {
t.Fatal(err)
}
var first map[string]any
if err = json.Unmarshal(before, &first); err != nil {
t.Fatal(err)
}
// Metadata-only reuse must keep the original source, even if discovery changes.
capture.Bucket = "later-bucket"
if err = f.store.UpsertCaptureRecord(f.ctx, capture); err != nil {
t.Fatal(err)
}
run, _ = f.store.GetRun(f.ctx, f.task.RunID)
result, err = s.materializeCombinedSliceSubmission(f.ctx, f.task, f.item, input, 2)
if err != nil {
t.Fatal(err)
}
if result.Payload["metadataOnly"] != true {
t.Fatal("metadata-only recovery not exercised")
}
after, err := os.ReadFile(filepath.Join(result.Artifact.Path, "revision-manifest.json"))
if err != nil {
t.Fatal(err)
}
var second map[string]any
if err = json.Unmarshal(after, &second); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(first["originalSource"], second["originalSource"]) {
t.Fatal("metadata revision followed mutable source")
}
}
original := filepath.Join(dir, "original")
for _, key := range ref.ObjectKeys {
relative := key[len(ref.Prefix)+1:]
dst := filepath.Join(original, filepath.FromSlash(relative))
if err = os.MkdirAll(filepath.Dir(dst), 0700); err != nil {
t.Fatal(err)
}
var body []byte
switch {
case filepath.Ext(dst) == ".yaml":
body = []byte("synthetic: true\n")
case filepath.Ext(dst) == ".json":
body = []byte("{\"synthetic\":true,\"partId\":\"part0001\"}\n")
default:
name := "imu.csv"
for _, side := range []string{"left", "right"} {
if strings.Contains(relative, "camera_"+side) {
name = side + ".mp4"
if strings.HasSuffix(relative, "_pts.csv") {
name = side + "_pts.csv"
}
}
}
body, err = os.ReadFile(filepath.Join(root, name))
if err != nil {
t.Fatal(err)
}
}
if err = os.WriteFile(dst, body, 0600); err != nil {
t.Fatal(err)
}
}
body, _ := json.MarshalIndent(map[string]any{"artifact": result.Artifact, "original": original, "workspace": root}, "", " ")
if err = os.WriteFile(filepath.Join(dir, "context.json"), body, 0600); err != nil {
t.Fatal(err)
}
})
}
}
// Legacy capture inputs can select an entire multi-part directory. They are
// insufficient evidence for v4, but must not block existing annotation work.
func TestLegacyMaterializationWithoutPartScopedOriginalSelection(t *testing.T) {
for _, sourceIndex := range []bool{false, true} {
name := "physical"
if sourceIndex {
name = "source-index"
}
t.Run(name, func(t *testing.T) {
f, root := annotationProcessorFixture(t)
run, err := f.store.GetRun(f.ctx, f.task.RunID)
if err != nil {
t.Fatal(err)
}
ref := consumerSourceRef()
ref.ObjectKeys = append(ref.ObjectKeys, ref.Prefix+"/part0002/other.mp4")
run.Inputs = map[string]domain.ArtifactPayload{"legacy-capture": {Ref: ref}}
if err = f.store.UpsertRun(f.ctx, *run); err != nil {
t.Fatal(err)
}
capture := domain.CaptureRecord{ID: "legacy-capture", RunID: run.ID, OrganizationID: run.OrganizationID, Bucket: ref.Bucket, Prefix: ref.Prefix + "/part0001", SourcePrefix: ref.Prefix, CaptureID: ref.CaptureUnit}
if err = f.store.UpsertCaptureRecord(f.ctx, capture); err != nil {
t.Fatal(err)
}
input := combinedAnnotationSubmissionInput{SliceSegments: []domain.SliceSegment{{ID: "whole", Name: "历史名称", Start: 0, End: 4}}, Segments: []domain.AnnotationSegment{{ID: "old", Start: 0, End: 4, Description: "历史描述"}}}
s := &server{store: f.store, workspaceMountPath: root, annotationSourceIndex: sourceIndex}
for revision := 1; revision <= 2; revision++ {
result, err := s.materializeCombinedSliceSubmission(f.ctx, f.task, f.item, input, revision)
if err != nil {
t.Fatal(err)
}
data, err := os.ReadFile(filepath.Join(result.Artifact.Path, "revision-manifest.json"))
if err != nil {
t.Fatal(err)
}
var manifest map[string]any
if err = json.Unmarshal(data, &manifest); err != nil {
t.Fatal(err)
}
if manifest["originalSource"] != nil {
t.Fatal("unreliable source must not be exported as v4 evidence")
}
if len(listValue(manifest["items"])) != 1 {
t.Fatal("legacy materialization lost its slice")
}
if !sourceIndex && revision == 2 && result.Payload["metadataOnly"] != true {
t.Fatal("metadata-only reuse was not exercised")
}
}
})
}
}
func TestOptionalOriginalSourceStillRejectsIdentityMismatch(t *testing.T) {
ref := consumerSourceRef()
run := &domain.RunRecord{ID: "run", OrganizationID: "org", Inputs: map[string]domain.ArtifactPayload{"raw": {Ref: ref}}}
capture := &domain.CaptureRecord{RunID: "other-run", OrganizationID: "org"}
if _, err := annotationRevisionOriginalSource(run, capture); err == nil {
t.Fatal("run mismatch accepted")
}
capture.RunID = run.ID
capture.OrganizationID = "other-org"
if _, err := annotationRevisionOriginalSource(run, capture); err == nil {
t.Fatal("organization mismatch accepted")
}
}

View File

@ -0,0 +1,319 @@
package api
import (
"context"
"droboticflow/internal/domain"
"droboticflow/internal/outputsnapshot"
"droboticflow/internal/workforcestats"
"encoding/csv"
"errors"
"fmt"
"net/http"
"strconv"
"strings"
"time"
_ "time/tzdata"
"unicode"
)
type outputPage[T any] struct {
Items []T `json:"items"`
Total int `json:"total"`
Page int `json:"page"`
PageSize int `json:"pageSize"`
}
func paginateOutput[T any](items []T, page, size int) outputPage[T] {
pages := (len(items) + size - 1) / size
if pages < 1 {
pages = 1
}
if page > pages {
page = pages
}
start := (page - 1) * size
end := start + size
if end > len(items) {
end = len(items)
}
return outputPage[T]{items[start:end], len(items), page, size}
}
func outputPagination(r *http.Request, key string) (int, int, error) {
parse := func(name string, fallback, max int) (int, error) {
v := r.URL.Query().Get(name)
if v == "" {
return fallback, nil
}
n, e := strconv.Atoi(v)
if e != nil || n < 1 || n > max {
return 0, fmt.Errorf("invalid %s", name)
}
return n, nil
}
page, e := parse(key+"Page", 1, 10000000)
if e != nil {
return 0, 0, e
}
size, e := parse(key+"PageSize", 5, 100)
return page, size, e
}
func outputDateRange(r *http.Request, now time.Time) (time.Time, time.Time, *time.Location, error) {
zone := r.URL.Query().Get("timeZone")
if zone == "" {
zone = "Asia/Shanghai"
}
loc, e := time.LoadLocation(zone)
if e != nil {
return time.Time{}, time.Time{}, nil, fmt.Errorf("invalid time zone")
}
today := now.In(loc)
to := time.Date(today.Year(), today.Month(), today.Day()+1, 0, 0, 0, 0, loc)
from := to.AddDate(0, 0, -30)
if s := r.URL.Query().Get("from"); s != "" {
from, e = time.ParseInLocation("2006-01-02", s, loc)
if e != nil {
return time.Time{}, time.Time{}, nil, fmt.Errorf("invalid start date")
}
}
if s := r.URL.Query().Get("to"); s != "" {
var last time.Time
last, e = time.ParseInLocation("2006-01-02", s, loc)
if e != nil {
return time.Time{}, time.Time{}, nil, fmt.Errorf("invalid end date")
}
to = last.AddDate(0, 0, 1)
}
if !from.Before(to) || to.After(from.AddDate(0, 0, 366)) {
return time.Time{}, time.Time{}, nil, fmt.Errorf("date range must be between 1 and 366 days")
}
return from.UTC(), to.UTC(), loc, nil
}
func (s *server) handleAnnotationOutput(w http.ResponseWriter, r *http.Request) {
p, ok := currentPrincipal(r)
if !ok {
renderJSON(w, 401, map[string]string{"error": "authentication required"})
return
}
admin := isOrganizationAdmin(p)
vendor := p.User.Role == domain.UserRoleSupplierManager
if !admin && !vendor && p.User.Role != domain.UserRoleAnnotator {
renderJSON(w, 403, map[string]string{"error": "annotation output access required"})
return
}
now := time.Now().UTC()
from, to, loc, err := outputDateRange(r, now)
if err != nil {
renderError(w, 400, err)
return
}
tasksPage, tasksSize, err := outputPagination(r, "tasks")
if err != nil {
renderError(w, 400, err)
return
}
suppliersPage, suppliersSize, err := outputPagination(r, "suppliers")
if err != nil {
renderError(w, 400, err)
return
}
daysPage, daysSize, err := outputPagination(r, "days")
if err != nil {
renderError(w, 400, err)
return
}
// Reviewer sections page independently: qualitySuppliers, qualityDays, ...
reviewPages := map[string][2]int{}
for _, role := range []string{"quality", "acceptance"} {
for _, list := range []string{"Suppliers", "Days"} {
page, size, err := outputPagination(r, role+list)
if err != nil {
renderError(w, 400, err)
return
}
reviewPages[role+list] = [2]int{page, size}
}
}
format := r.URL.Query().Get("format")
if format != "" && format != "csv" {
renderError(w, 400, fmt.Errorf("invalid export format"))
return
}
q := workforcestats.OutputQuery{From: from, To: to, Now: now, Location: loc, ProjectID: r.URL.Query().Get("projectId"), SupplierID: r.URL.Query().Get("supplierId"), UserID: r.URL.Query().Get("userId")}
audience := "platform"
if vendor {
audience = "vendor"
q.MatchSupplier = true
q.ScopeSupplierID = p.User.SupplierID
} else if !admin {
audience = "self"
q.MatchUser = true
q.ScopeUserID = p.User.ID
}
ctx, cancel := context.WithTimeout(r.Context(), 25*time.Second)
defer cancel()
org := s.requestOrganizationID(r)
ids, err := s.annotationOutputProjectScope(ctx, org)
if err != nil {
renderError(w, 500, err)
return
}
// A supplier manager without a supplier never gains the organization scope.
if vendor && p.User.SupplierID == "" {
ids = []string{}
}
// Do not narrow the load to a requested person: a review may address sources
// attributed to several authors. The read model applies the principal fence.
s.outputOnce.Do(func() { s.outputReader = outputsnapshot.NewReader(s.store, s.store.LoadAnnotationOutput) })
report, info, err := s.outputReader.Query(ctx, org, ids, r.URL.Query().Get("snapshotId"), q)
// Older deployed clients cannot consume 202. They wait for the same shared
// background generation; they never start a private synchronous rebuild.
for err == nil && report == nil && r.URL.Query().Get("snapshotMode") != "shared" {
timer := time.NewTimer(100 * time.Millisecond)
select {
case <-ctx.Done():
timer.Stop()
err = ctx.Err()
case <-timer.C:
report, info, err = s.outputReader.Query(ctx, org, ids, r.URL.Query().Get("snapshotId"), q)
}
}
if errors.Is(err, outputsnapshot.ErrExpired) {
renderJSON(w, 409, map[string]string{"error": "数据已更新,请刷新后重试", "code": "snapshot_expired"})
return
}
if err != nil {
renderJSON(w, 503, map[string]string{"error": "看板暂时未能加载,请稍后重试", "code": "snapshot_unavailable"})
return
}
w.Header().Set("Cache-Control", "no-store")
if report == nil {
w.Header().Set("Retry-After", "2")
renderJSON(w, 202, map[string]any{"status": "building", "retryAfterMs": 2000})
return
}
if format == "csv" {
if len(report.Tasks) > 100000 {
renderError(w, 400, fmt.Errorf("export exceeds 100000 rows; narrow the filters"))
return
}
w.Header().Set("X-Output-Snapshot", info.ID)
writeOutputCSV(w, report.Tasks, loc)
return
}
dailySupplier, dailyUser := r.URL.Query().Get("dailySupplierId"), r.URL.Query().Get("dailyUserId")
days := filterOutputDays(report.Days, func(day workforcestats.OutputDay) bool {
return (dailySupplier == "" || day.SupplierID == dailySupplier) && (dailyUser == "" || day.UserID == dailyUser)
})
review := func(role string, section workforcestats.OutputReviewReport) map[string]any {
reviewDays := filterOutputDays(section.Days, func(day workforcestats.OutputReviewDay) bool {
return (dailySupplier == "" || day.SupplierID == dailySupplier) && (dailyUser == "" || day.UserID == dailyUser)
})
suppliers, dayPage := reviewPages[role+"Suppliers"], reviewPages[role+"Days"]
return map[string]any{"totals": section.Totals, "supplierOutput": paginateOutput(section.Suppliers, suppliers[0], suppliers[1]), "dailyOutput": paginateOutput(reviewDays, dayPage[0], dayPage[1])}
}
w.Header().Set("Cache-Control", "no-store")
renderJSON(w, 200, map[string]any{"asOf": info.AsOf.Format(time.RFC3339Nano), "snapshot": info, "audienceScope": audience, "from": from.In(loc).Format("2006-01-02"), "to": to.In(loc).AddDate(0, 0, -1).Format("2006-01-02"), "timeZone": loc.String(), "totals": report.Totals, "efficiency": report.Efficiency, "projects": report.Projects, "suppliers": report.SupplierOptions, "users": report.Users, "tasks": paginateOutput(report.Tasks, tasksPage, tasksSize), "supplierOutput": paginateOutput(report.Suppliers, suppliersPage, suppliersSize), "dailyOutput": paginateOutput(days, daysPage, daysSize), "qualityReview": review("quality", report.Quality), "acceptanceReview": review("acceptance", report.Acceptance)})
}
func filterOutputDays[T any](days []T, keep func(T) bool) []T {
out := make([]T, 0, len(days))
for _, day := range days {
if keep(day) {
out = append(out, day)
}
}
return out
}
func outputCSVCell(s string) string {
v := strings.TrimLeftFunc(s, unicode.IsSpace)
if v != "" && strings.ContainsRune("=+-@", rune(v[0])) {
return "'" + s
}
return s
}
func writeOutputCSV(w http.ResponseWriter, tasks []workforcestats.OutputTask, loc *time.Location) {
w.Header().Set("Content-Type", "text/csv; charset=utf-8")
w.Header().Set("Content-Disposition", `attachment; filename="annotation-output.csv"`)
w.Header().Set("Cache-Control", "no-store")
_, _ = w.Write([]byte{0xef, 0xbb, 0xbf})
writer := csv.NewWriter(w)
defer writer.Flush()
_ = writer.Write([]string{"视频编号", "项目", "供应商", "标注员", "整段视频秒数", "首次提交时间", "质检通过时间", "最终验收通过时间", "当前状态", "首次提交标注员", "首次提交供应商", "验收产出标注员", "验收产出供应商"})
date := func(s string) string {
t, e := time.Parse(time.RFC3339Nano, s)
if e != nil {
return ""
}
return t.In(loc).Format("2006-01-02 15:04:05")
}
for _, t := range tasks {
seconds := ""
if t.Seconds != nil {
seconds = strconv.FormatFloat(*t.Seconds, 'f', 3, 64)
}
row := []string{t.VideoID, t.ProjectName, t.SupplierName, t.UserName, seconds, date(t.SubmittedAt), date(t.QualityAt), date(t.AcceptedAt), outputStatusLabel(t.Status), t.Attribution["submitted"].UserName, t.Attribution["submitted"].SupplierName, t.Attribution["acceptance"].UserName, t.Attribution["acceptance"].SupplierName}
for i := range row {
row[i] = outputCSVCell(row[i])
}
if writer.Write(row) != nil {
return
}
}
}
func outputStatusLabel(s string) string {
switch s {
case "accepted":
return "已验收"
case "pending_acceptance":
return "待最终验收"
case "pending_quality":
return "待质检"
case "rework":
return "返工中"
}
return ""
}
// Keep the report and its read-only health inspection on exactly the same
// current organization/project scope; caller-supplied IDs cannot expand it.
func (s *server) annotationOutputProjectScope(ctx context.Context, org string) ([]string, error) {
scope, err := s.store.GetProductionStatsScope(ctx, org)
if err != nil {
return nil, err
}
projects, err := s.store.ListProjectsByOrganization(ctx, org)
if err != nil {
return nil, err
}
admitted := map[string]bool{}
for _, id := range scope.ProjectIDs {
admitted[id] = true
}
ids := []string{}
for _, p := range projects {
if admitted[p.ID] {
ids = append(ids, p.ID)
}
}
return ids, nil
}
func (s *server) handleAnnotationOutputHealth(w http.ResponseWriter, r *http.Request) {
if _, ok := s.productionAdmin(w, r); !ok {
return
}
w.Header().Set("Cache-Control", "no-store")
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
defer cancel()
org := s.requestOrganizationID(r)
ids, err := s.annotationOutputProjectScope(ctx, org)
if err != nil {
renderError(w, 503, err)
return
}
s.outputOnce.Do(func() { s.outputReader = outputsnapshot.NewReader(s.store, s.store.LoadAnnotationOutput) })
health, err := s.outputReader.Inspect(ctx, org, ids)
if err != nil {
renderError(w, 503, err)
return
}
renderJSON(w, 200, health)
}

View File

@ -0,0 +1,120 @@
package api
import (
"context"
"droboticflow/internal/domain"
"droboticflow/internal/productionstats"
"droboticflow/internal/store"
"encoding/json"
"fmt"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"os"
"sort"
"testing"
"time"
)
func TestAnnotationOutputSnapshotScale(t *testing.T) {
if os.Getenv("RUN_OUTPUT_SNAPSHOT_BENCHMARK") != "1" || os.Getenv("TEST_WORKFORCE_MONGO_URI") != "mongodb://127.0.0.1:27018" {
t.Skip("explicit isolated local timing run")
}
ctx, cancel := context.WithTimeout(t.Context(), 60*time.Second)
defer cancel()
c, e := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://127.0.0.1:27018"))
if e != nil {
t.Fatal(e)
}
db := c.Database(fmt.Sprintf("annotation_output_scale_%d", time.Now().UnixNano()))
defer c.Disconnect(context.Background())
defer db.Drop(context.Background())
f := newCombinedDraftFixture(t)
s := store.NewMongoStore(db)
for _, u := range []domain.UserRecord{f.admin, f.annotator} {
if e = s.UpsertUser(ctx, u); e != nil {
t.Fatal(e)
}
}
if e = s.UpsertProject(ctx, domain.ProjectRecord{ID: "scale", OrganizationID: domain.DefaultOrganizationID, Name: "Synthetic scale project"}); e != nil {
t.Fatal(e)
}
if _, e = s.SaveProductionStatsScope(ctx, domain.DefaultOrganizationID, productionstats.Scope{ProjectIDs: []string{"scale"}, Revision: 1}, 0); e != nil {
t.Fatal(e)
}
n := 5000
at := time.Now().Add(-time.Hour).UTC().Format(time.RFC3339Nano)
collections := map[string][]any{"annotationTasks": {}, "annotationWorklogEvents": {}}
for i := 0; i < n; i++ {
id := fmt.Sprintf("scale-%d", i)
collections["annotationTasks"] = append(collections["annotationTasks"], bson.M{"id": id, "organizationId": domain.DefaultOrganizationID, "projectId": "scale", "sourceIsFullVideo": true, "sourceDurationSeconds": 900, "createdAt": at})
for j := 0; j < 5; j++ {
collections["annotationWorklogEvents"] = append(collections["annotationWorklogEvents"], bson.M{"id": fmt.Sprintf("%s-%d", id, j), "organizationId": domain.DefaultOrganizationID, "projectId": "scale", "taskId": id, "userId": f.annotator.ID, "supplierId": f.annotator.SupplierID, "revision": 1, "submissionKind": "first", "eventType": "combined_submitted", "createdAt": at})
}
}
for name, docs := range collections {
for len(docs) > 0 {
size := 1000
if size > len(docs) {
size = len(docs)
}
if _, e = db.Collection(name).InsertMany(ctx, docs[:size]); e != nil {
t.Fatal(e)
}
docs = docs[size:]
}
}
wrapped := &countingOutputStore{Store: s}
f.handler = NewHandler(wrapped, nil)
start := time.Now()
cold := outputReady(t, f, "/api/dashboard/annotation-output", f.admin)
coldMS := time.Since(start).Milliseconds()
var body struct {
Snapshot struct {
ID string `json:"id"`
} `json:"snapshot"`
Tasks struct {
Total int `json:"total"`
} `json:"tasks"`
}
if e = json.Unmarshal(cold.Body.Bytes(), &body); e != nil || body.Tasks.Total != n {
t.Fatalf("scale setup %d %v", body.Tasks.Total, e)
}
samples := []int64{}
// Four independent handlers share only Mongo, including after a restart.
for instance := 0; instance < 4; instance++ {
proxy := &countingOutputStore{Store: store.NewMongoStore(db)}
if instance == 0 {
proxy = wrapped
}
f.handler = NewHandler(proxy, nil)
for page := 1; page <= 8; page++ {
start := time.Now()
r := f.request("GET", fmt.Sprintf("/api/dashboard/annotation-output?snapshotId=%s&tasksPage=%d", body.Snapshot.ID, page), f.admin, nil)
samples = append(samples, time.Since(start).Milliseconds())
if r.Code != 200 {
t.Fatalf("warm %d", r.Code)
}
}
if instance > 0 && proxy.loads.Load() != 0 {
t.Fatal("another API reloaded source history")
}
}
sort.Slice(samples, func(i, j int) bool { return samples[i] < samples[j] })
p95 := samples[(len(samples)*95-1)/100]
if p95 > 1000 {
t.Fatalf("warm report p95 %dms exceeds local target", p95)
}
if wrapped.loads.Load() != 1 {
t.Fatalf("%d raw source reads", wrapped.loads.Load())
}
sizes := db.Collection("annotationOutputSnapshotSlots")
count, _ := sizes.CountDocuments(ctx, bson.M{})
var slot struct {
Data []byte `bson:"data"`
}
_ = sizes.FindOne(ctx, bson.M{}).Decode(&slot)
report := map[string]any{"environment": "local isolated Mongo; synthetic metadata; no production performance claim", "parts": n, "historyEvents": n * 5, "apiInstances": 4, "warmRequests": len(samples), "coldMs": coldMS, "warmP50Ms": samples[len(samples)/2], "warmP95Ms": p95, "sourceBuilds": wrapped.loads.Load(), "storedSlots": count, "compressedBytes": len(slot.Data)}
encoded, _ := json.Marshal(report)
t.Log(string(encoded))
}

View File

@ -0,0 +1,207 @@
package api
import (
"context"
"droboticflow/internal/domain"
"droboticflow/internal/outputsnapshot"
"droboticflow/internal/productionstats"
"droboticflow/internal/store"
"droboticflow/internal/workforcestats"
"encoding/json"
"net/http/httptest"
"sync"
"sync/atomic"
"testing"
"time"
)
type countingOutputStore struct {
store.Store
loads atomic.Int32
gate chan struct{}
}
func (s *countingOutputStore) LoadAnnotationOutput(ctx context.Context, org string, projects []string) (workforcestats.OutputSnapshot, error) {
s.loads.Add(1)
if s.gate != nil {
select {
case <-s.gate:
case <-ctx.Done():
return workforcestats.OutputSnapshot{}, ctx.Err()
}
}
return s.Store.LoadAnnotationOutput(ctx, org, projects)
}
func outputReady(t *testing.T, f combinedDraftFixture, target string, user domain.UserRecord) *httptest.ResponseRecorder {
t.Helper()
deadline := time.Now().Add(3 * time.Second)
for {
r := f.request("GET", target, user, nil)
if r.Code != 202 {
if r.Code != 200 {
t.Fatalf("read %d: %s", r.Code, r.Body.String())
}
return r
}
if time.Now().After(deadline) {
t.Fatal("snapshot never became ready")
}
time.Sleep(time.Millisecond)
}
}
func TestAnnotationOutputSnapshotSharedAcrossHandlersAndPages(t *testing.T) {
f := newCombinedDraftFixture(t)
if _, err := f.store.SaveProductionStatsScope(f.ctx, domain.DefaultOrganizationID, productionstats.Scope{ProjectIDs: []string{f.task.ProjectID}, Revision: 1}, 0); err != nil {
t.Fatal(err)
}
wrapped := &countingOutputStore{Store: f.store}
f.handler = NewHandler(wrapped, nil)
outputReady(t, f, "/api/dashboard/annotation-output?tasksPage=1", f.admin)
// A second API handler must use the already published shared snapshot.
f.handler = NewHandler(wrapped, nil)
outputReady(t, f, "/api/dashboard/annotation-output?tasksPage=2", f.annotator)
f.request("POST", "/api/nonexistent-output-test", f.admin, nil)
outputReady(t, f, "/api/dashboard/annotation-output?tasksPage=3", f.annotator)
if n := wrapped.loads.Load(); n != 1 {
t.Fatalf("two handlers/pages rebuilt history %d times; want one shared build", n)
}
}
func TestAnnotationOutputSnapshotPinAndCurrentPermissions(t *testing.T) {
f := newCombinedDraftFixture(t)
_, e := f.store.SaveProductionStatsScope(f.ctx, domain.DefaultOrganizationID, productionstats.Scope{ProjectIDs: []string{f.task.ProjectID}, Revision: 1}, 0)
if e != nil {
t.Fatal(e)
}
r := outputReady(t, f, "/api/dashboard/annotation-output", f.admin)
var data struct {
Snapshot struct {
ID string `json:"id"`
} `json:"snapshot"`
}
if e = json.Unmarshal(r.Body.Bytes(), &data); e != nil {
t.Fatal(e)
}
pin := data.Snapshot.ID
if pin == "" {
t.Fatal("missing snapshot version")
}
export := f.request("GET", "/api/dashboard/annotation-output?format=csv&snapshotId="+pin, f.admin, nil)
if export.Code != 200 || export.Header().Get("X-Output-Snapshot") != pin {
t.Fatal("CSV did not use selected generation")
}
if r := f.request("GET", "/api/dashboard/annotation-output?snapshotId=expired", f.admin, nil); r.Code != 409 {
t.Fatal("unknown generation silently replaced")
}
changed := f.admin
changed.Role = domain.UserRoleQualityReviewer
if r := f.request("GET", "/api/dashboard/annotation-output?snapshotId="+pin, changed, nil); r.Code != 403 {
t.Fatal("cached generation bypassed current role")
}
if _, e = f.store.SaveProductionStatsScope(f.ctx, domain.DefaultOrganizationID, productionstats.Scope{ProjectIDs: []string{}, Revision: 2}, 1); e != nil {
t.Fatal(e)
}
if r := f.request("GET", "/api/dashboard/annotation-output?snapshotId="+pin, f.admin, nil); r.Code != 409 {
t.Fatal("scope removal kept old cached coverage")
}
}
func TestAnnotationOutputColdSharedResponseAndLegacyCompatibility(t *testing.T) {
f := newCombinedDraftFixture(t)
_, e := f.store.SaveProductionStatsScope(f.ctx, domain.DefaultOrganizationID, productionstats.Scope{ProjectIDs: []string{f.task.ProjectID}, Revision: 1}, 0)
if e != nil {
t.Fatal(e)
}
gate := make(chan struct{})
var release sync.Once
defer release.Do(func() { close(gate) })
wrapped := &countingOutputStore{Store: f.store, gate: gate}
f.handler = NewHandler(wrapped, nil)
started := time.Now()
cold := f.request("GET", "/api/dashboard/annotation-output?snapshotMode=shared", f.admin, nil)
if cold.Code != 202 || time.Since(started) > 250*time.Millisecond {
t.Fatalf("cold request blocked or returned fake numbers: %d", cold.Code)
}
legacy := make(chan *httptest.ResponseRecorder, 1)
go func() { legacy <- f.request("GET", "/api/dashboard/annotation-output", f.admin, nil) }()
select {
case r := <-legacy:
t.Fatalf("legacy client received premature %d", r.Code)
case <-time.After(20 * time.Millisecond):
}
release.Do(func() { close(gate) })
select {
case r := <-legacy:
if r.Code != 200 {
t.Fatalf("legacy client failed %d", r.Code)
}
case <-time.After(2 * time.Second):
t.Fatal("legacy client not released")
}
if wrapped.loads.Load() != 1 {
t.Fatal("legacy and shared requests duplicated source read")
}
}
func TestAnnotationOutputPrivateCollectionsRejectBrowserAndWrites(t *testing.T) {
f := newCombinedDraftFixture(t)
for _, name := range []string{outputsnapshot.StateCollection, outputsnapshot.SlotsCollection} {
for _, operation := range []struct {
method, target string
body any
}{
{"GET", "/api/database/collections/" + name + "/documents?target=platform", nil},
{"POST", "/api/database/collections?target=business", map[string]any{"name": name}},
} {
r := f.request(operation.method, operation.target, f.admin, operation.body)
if r.Code < 400 {
t.Fatalf("internal collection exposed through %s %s: %d", operation.method, operation.target, r.Code)
}
}
}
}
func TestAnnotationOutputHealthIsAdminScopedAndReadOnly(t *testing.T) {
f := newCombinedDraftFixture(t)
_, err := f.store.SaveProductionStatsScope(f.ctx, domain.DefaultOrganizationID, productionstats.Scope{ProjectIDs: []string{f.task.ProjectID}, Revision: 1}, 0)
if err != nil {
t.Fatal(err)
}
wrapped := &countingOutputStore{Store: f.store}
f.handler = NewHandler(wrapped, nil)
url := "/api/dashboard/annotation-output/health"
for _, user := range []domain.UserRecord{f.annotator, f.reviewer} {
if r := f.request("GET", url, user, nil); r.Code != 403 {
t.Fatalf("non-admin read diagnostics: %d", r.Code)
}
}
r := f.request("GET", url, f.admin, nil)
if r.Code != 200 {
t.Fatal(r.Body.String())
}
var h outputsnapshot.Health
if err = json.Unmarshal(r.Body.Bytes(), &h); err != nil {
t.Fatal(err)
}
if h.Status != "not_built" || wrapped.loads.Load() != 0 || r.Header().Get("Cache-Control") != "no-store" {
t.Fatalf("inspection unexpectedly built data: %+v", h)
}
outputReady(t, f, "/api/dashboard/annotation-output", f.admin)
r = f.request("GET", url, f.admin, nil)
if err = json.Unmarshal(r.Body.Bytes(), &h); err != nil {
t.Fatal(err)
}
if h.Status != "healthy" || !h.Available || h.Metrics == nil || wrapped.loads.Load() != 1 {
t.Fatalf("missing shared health %+v", h)
}
_, err = f.store.SaveProductionStatsScope(f.ctx, domain.DefaultOrganizationID, productionstats.Scope{ProjectIDs: []string{}, Revision: 2}, 1)
if err != nil {
t.Fatal(err)
}
r = f.request("GET", url+"?projectId="+f.task.ProjectID, f.admin, nil)
h = outputsnapshot.Health{}
_ = json.Unmarshal(r.Body.Bytes(), &h)
if h.Status != "not_built" || h.Metrics != nil || wrapped.loads.Load() != 1 {
t.Fatal("health exposed excluded scope or rebuilt")
}
}

View File

@ -0,0 +1,113 @@
package api
import (
"droboticflow/internal/domain"
"droboticflow/internal/productionstats"
"droboticflow/internal/workforcestats"
"encoding/csv"
"encoding/json"
"fmt"
"net/http/httptest"
"net/url"
"reflect"
"strings"
"testing"
"time"
)
func TestAnnotationOutputAPIAndCSVScope(t *testing.T) {
f := newCombinedDraftFixture(t, "output-vendor")
check := func(e error) {
t.Helper()
if e != nil {
t.Fatal(e)
}
}
_, e := f.store.SaveProductionStatsScope(f.ctx, domain.DefaultOrganizationID, productionstats.Scope{ProjectIDs: []string{f.task.ProjectID}, Revision: 1}, 0)
check(e)
at := time.Now().Add(-time.Hour).UTC().Format(time.RFC3339Nano)
for i := 0; i < 207; i++ {
id := fmt.Sprintf("output-task-%d", i)
check(f.store.UpsertAnnotationTask(f.ctx, domain.AnnotationTaskRecord{ID: id, ProjectID: f.task.ProjectID, OrganizationID: domain.DefaultOrganizationID, SourceIsFullVideo: true, SourceDurationSeconds: 900, CreatedAt: at}))
check(f.store.UpsertAnnotationRevision(f.ctx, domain.AnnotationRevisionRecord{ID: id + "-rev", OrganizationID: domain.DefaultOrganizationID, TaskID: id, Revision: 1, AuthorUserID: f.annotator.ID, CountsAsOutput: true}))
check(f.store.UpsertAnnotationWorklogEvent(f.ctx, domain.AnnotationWorklogEventRecord{ID: id + "-event", ProjectID: f.task.ProjectID, OrganizationID: domain.DefaultOrganizationID, TaskID: id, Revision: 1, UserID: f.annotator.ID, SupplierID: f.annotator.SupplierID, EventType: "combined_submitted", CreatedAt: at}))
}
for i := 0; i < 2; i++ {
check(f.store.UpsertAnnotationReview(f.ctx, domain.AnnotationReviewRecord{ID: fmt.Sprintf("output-review-%d", i), OrganizationID: domain.DefaultOrganizationID, TaskID: "output-task-0", Revision: 1, Stage: domain.AnnotationReviewStageQuality, Decision: domain.AnnotationReviewRejected, ReviewerUserID: f.reviewer.ID, CreatedAt: at}))
}
before, _ := f.store.GetHumanItem(f.ctx, f.item.ID)
for _, user := range []domain.UserRecord{f.admin, f.annotator} {
r := outputReady(t, f, "/api/dashboard/annotation-output?tasksPage=42&tasksPageSize=5&qualityDaysPageSize=1", user)
if r.Code != 200 {
t.Fatalf("read %d %s", r.Code, r.Body.String())
}
var d struct {
Tasks outputPage[workforcestats.OutputTask] `json:"tasks"`
Totals workforcestats.OutputTotals `json:"totals"`
Quality struct {
Totals workforcestats.OutputReviewMetric `json:"totals"`
Days outputPage[workforcestats.OutputReviewDay] `json:"dailyOutput"`
} `json:"qualityReview"`
}
check(json.Unmarshal(r.Body.Bytes(), &d))
if d.Tasks.Total != 207 || len(d.Tasks.Items) != 2 || d.Totals.Submitted.Count != 207 {
t.Fatalf("pagination changed totals %s", r.Body.String())
}
// Both decisions on the same part count; an annotator never sees reviewer work.
want := 2
if user.ID == f.annotator.ID {
want = 0
}
if d.Quality.Totals.Reviews.Count != want || d.Quality.Totals.Rereviews != want/2 || d.Quality.Days.PageSize != 1 {
t.Fatalf("review output for %s: %s", user.ID, r.Body.String())
}
exported := outputReady(t, f, "/api/dashboard/annotation-output?format=csv&tasksPage=42&tasksPageSize=5", user)
if exported.Code != 200 {
t.Fatalf("export %s", exported.Body.String())
}
rows, e := csv.NewReader(strings.NewReader(strings.TrimPrefix(exported.Body.String(), "\ufeff"))).ReadAll()
check(e)
if len(rows) != 208 {
t.Fatalf("export only current page: %d", len(rows))
}
}
for _, query := range []string{"userId=" + f.reviewer.ID, "supplierId=foreign"} {
r := f.request("GET", "/api/dashboard/annotation-output?"+query, f.annotator, nil)
var body struct {
Tasks outputPage[workforcestats.OutputTask] `json:"tasks"`
}
check(json.Unmarshal(r.Body.Bytes(), &body))
if r.Code != 200 || body.Tasks.Total != 0 {
t.Fatalf("scope enlarged %s", r.Body.String())
}
}
if r := f.request("GET", "/api/dashboard/annotation-output", f.reviewer, nil); r.Code != 403 {
t.Fatal("reviewer gained author report")
}
for _, query := range []string{"tasksPage=-1", "tasksPageSize=1000", "qualityDaysPageSize=1000", "acceptanceSuppliersPage=0", "timeZone=invalid", "from=2026-10-20&to=2026-10-01", "format=unknown"} {
if r := f.request("GET", "/api/dashboard/annotation-output?"+query, f.admin, nil); r.Code != 400 {
t.Fatalf("invalid query accepted %s: %d", query, r.Code)
}
}
after, _ := f.store.GetHumanItem(f.ctx, f.item.ID)
if !reflect.DeepEqual(before, after) {
t.Fatal("report mutated customer task")
}
}
func TestAnnotationOutputDateBoundariesAndCSVCells(t *testing.T) {
for _, tc := range []struct {
zone, from, to string
hours time.Duration
}{{"Asia/Shanghai", "2026-10-01", "2026-10-01", 24}, {"America/New_York", "2026-03-08", "2026-03-08", 23}, {"America/New_York", "2026-11-01", "2026-11-01", 25}} {
req := httptest.NewRequest("GET", "/api/dashboard/annotation-output?"+url.Values{"timeZone": {tc.zone}, "from": {tc.from}, "to": {tc.to}}.Encode(), nil)
from, to, _, e := outputDateRange(req, time.Now())
if e != nil || to.Sub(from) != tc.hours*time.Hour {
t.Fatalf("DST range %+v: %v %v", tc, e, to.Sub(from))
}
}
for _, s := range []string{"=SUM(A1)", " +123", "@unsafe", "\t-formula"} {
if !strings.HasPrefix(outputCSVCell(s), "'") {
t.Fatal("formula passed to spreadsheet")
}
}
}

View File

@ -0,0 +1,27 @@
package api
import (
"context"
"droboticflow/internal/domain"
"droboticflow/internal/store"
)
// A revision waiting for source validation is not itself validating media.
// Read the durable validator lease so all API replicas show the same state.
func (s *server) annotationProcessingStage(ctx context.Context, org string, j *domain.AnnotationProcessingJob) string {
if j.Stage != "validating" || (j.Status != "queued" && j.Status != "running") {
return j.Stage
}
db, ok := s.store.(store.AnnotationProcessingStore)
if !ok || j.SourceFingerprint == "" {
return j.Stage
}
index, err := db.GetAnnotationIndexJob(ctx, store.AnnotationSourceJobID(org, j.SourceFingerprint))
if err != nil {
return j.Stage
}
if index == nil || index.Status == "queued" {
return "waiting_source"
}
return j.Stage
}

View File

@ -0,0 +1,38 @@
package api
import (
"droboticflow/internal/domain"
"droboticflow/internal/store"
"testing"
"time"
)
func TestSourceValidationProgressSeparatesQueueAndWorker(t *testing.T) {
db := store.NewMemoryStore()
s := &server{store: db}
ctx := t.Context()
task := domain.AnnotationTaskRecord{ID: "task", OrganizationID: "org"}
job := &domain.AnnotationProcessingJob{Stage: "validating", Status: "queued", SourceFingerprint: "source"}
if got := s.annotationProcessingStage(ctx, "org", job); got != "waiting_source" {
t.Fatal(got)
}
if err := db.EnqueueAnnotationSourceIndex(ctx, task, "source", false); err != nil {
t.Fatal(err)
}
if got := s.annotationProcessingStage(ctx, "org", job); got != "waiting_source" {
t.Fatal(got)
}
now := time.Now().UTC()
claimed, err := db.ClaimAnnotationIndex(ctx, "worker", now.Format(time.RFC3339Nano), now.Add(time.Minute).Format(time.RFC3339Nano))
if err != nil || claimed == nil {
t.Fatal(claimed, err)
}
if got := s.annotationProcessingStage(ctx, "org", job); got != "validating" {
t.Fatal(got)
}
job.Status = "failed"
job.Stage = "failed"
if got := s.annotationProcessingStage(ctx, "org", job); got != "failed" {
t.Fatal(got)
}
}

View File

@ -172,6 +172,7 @@ func RunAnnotationProcessor(ctx context.Context, db store.Store, cfg config.AppC
if e != nil {
log.Printf("[annotation-processor] index claim: %v", e)
} else if j != nil {
started := time.Now()
indexCtx, cancel := context.WithTimeout(ctx, 15*time.Minute)
stopHeartbeat := keepAnnotationIndexLease(indexCtx, queue, *j, cancel)
taskID := j.TaskID
@ -223,7 +224,10 @@ func RunAnnotationProcessor(ctx context.Context, db store.Store, cfg config.AppC
j.Status = "failed"
}
}
_, _ = queue.FinishAnnotationIndex(context.WithoutCancel(ctx), *j)
finishCtx, finishCancel := context.WithTimeout(context.WithoutCancel(ctx), 15*time.Second)
saved, finishErr := queue.FinishAnnotationIndex(finishCtx, *j)
finishCancel()
log.Printf("[annotation-validator] job=%s status=%s attempt=%d elapsed_ms=%d saved=%t error=%v finish_error=%v", j.ID, j.Status, j.Attempt, time.Since(started).Milliseconds(), saved, readErr, finishErr)
continue
}
select {
@ -309,7 +313,6 @@ func (s *server) executeAnnotationJob(parent context.Context, claimed domain.Hum
}
j.Status = "completed"
j.Stage = "completed"
j.Input = nil
j.CompletedSlices = j.TotalSlices
j.UpdatedAt = nowISO()
if e != nil {
@ -317,6 +320,8 @@ func (s *server) executeAnnotationJob(parent context.Context, claimed domain.Hum
j.Stage = "failed"
j.Error = e.Error()
log.Printf("[annotation-processor] task=%s job=%s failed: %v", j.TaskID, j.ID, e)
} else {
j.Input = nil
}
_, _ = queue.SaveAnnotationProcessing(finishCtx, claimed.ID, j)
if e != nil {

View File

@ -13,12 +13,14 @@ import (
"encoding/json"
"fmt"
"io"
"log"
"math"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"time"
)
const sourceIndexSchema = "droboticflow.source-slice-index/v1"
@ -242,20 +244,24 @@ func validateSourceMediaClock(ctx context.Context, path string, count int, durat
period := 1 / rate
// r_frame_rate is only a candidate cadence: verify every decoded frame
// against it below, rather than trusting the metadata or deriving a cadence
// from rounded total duration. The final frame timestamp and packet duration
// can each round, so allow one time-base tick at the end of the stream.
if math.IsInf(period, 0) || math.Abs(duration-float64(count)*period) > math.Max(0.0001, tick+1e-9) {
// from rounded total duration. MP4 edit lists additionally round the
// presentation endpoint in the movie clock (often 1ms, even for a video
// track at 1/15360). This never relaxes the decoded frame checks below.
if math.IsInf(period, 0) || math.Abs(duration-float64(count)*period) > sourceClockEndTolerance(path, tick, period) {
return fmt.Errorf("source video duration does not match its fixed frame cadence")
}
// Integer timestamps preserve half-tick boundaries; ffprobe's *_time
// decimal output rounds to microseconds and can exceed the tolerance.
cmd := exec.CommandContext(ctx, "ffprobe", "-v", "error", "-threads", "1", "-select_streams", "v:0", "-show_frames", "-show_entries", "frame=best_effort_timestamp", "-of", "csv=p=0", path)
// Bound decoding to two threads within the validator CPU budget.
// Still decode every frame and validate its original presentation timestamp.
cmd := exec.CommandContext(ctx, "ffprobe", "-v", "error", "-threads", "2", "-select_streams", "v:0", "-show_frames", "-show_entries", "frame=best_effort_timestamp", "-of", "csv=p=0", path)
stdout, e := cmd.StdoutPipe()
if e != nil {
return e
}
var stderr bytes.Buffer
cmd.Stderr = &stderr
// Decoder diagnostics are not returned; do not accumulate unbounded stderr
// from malformed media during a long-running validation.
cmd.Stderr = io.Discard
if e = cmd.Start(); e != nil {
return e
}
@ -281,6 +287,9 @@ func validateSourceMediaClock(ctx context.Context, path string, count int, durat
n++
}
readErr := scanner.Err()
if readErr != nil {
_ = cmd.Process.Kill()
}
waitErr := cmd.Wait()
if clockErr != nil {
return clockErr
@ -297,7 +306,18 @@ func validateSourceMediaClock(ctx context.Context, path string, count int, durat
return nil
}
func (s *server) buildAnnotationSourceCertificate(ctx context.Context, t domain.AnnotationTaskRecord, run domain.RunRecord, fingerprint string) (*annotationSourceCertificate, error) {
func (s *server) buildAnnotationSourceCertificate(ctx context.Context, t domain.AnnotationTaskRecord, run domain.RunRecord, fingerprint string) (_ *annotationSourceCertificate, buildErr error) {
started, phaseStarted := time.Now(), time.Now()
phase := "fetch"
durations := map[string]int64{}
nextPhase := func(next string) {
durations[phase] = time.Since(phaseStarted).Milliseconds()
phase, phaseStarted = next, time.Now()
}
defer func() {
durations[phase] = time.Since(phaseStarted).Milliseconds()
log.Printf("[annotation-validator] source_timing task=%s total_ms=%d phase_ms=%v success=%t", t.ID, time.Since(started).Milliseconds(), durations, buildErr == nil)
}()
inputs, e := annotationSourceInputs(t, run)
if e != nil {
return nil, e
@ -328,6 +348,7 @@ func (s *server) buildAnnotationSourceCertificate(ctx context.Context, t domain.
}
paths[kind] = local
}
nextPhase("media_clock")
left, e := readHardwarePTS(paths["leftPts"])
if e != nil {
return nil, e
@ -350,6 +371,7 @@ func (s *server) buildAnnotationSourceCertificate(ctx context.Context, t domain.
if e = validateSourceMediaClock(ctx, paths["rightVideo"], len(right), rd); e != nil {
return nil, e
}
nextPhase("sensor_index")
// The existing builder validates ordered sensor pairs and the exact common
// range. Its compact grid is the same seam used by editor validation.
timeline, e := s.annotationFrameTimelineForTask(ctx, t, "left")
@ -387,6 +409,7 @@ func (s *server) buildAnnotationSourceCertificate(ctx context.Context, t domain.
c.IMURowsBeforeBoundary[position] = count
position++
}
nextPhase("pin")
for _, kind := range []string{"leftVideo", "rightVideo", "leftPts", "rightPts", "imu"} {
original, e := s.safeMaterializationSourcePath(inputs[kind])
if e != nil {

View File

@ -11,6 +11,7 @@ import (
"path/filepath"
"strings"
"testing"
"time"
)
func TestSourceIndexRevisionReusesValidatedSourceWithoutMediaGeneration(t *testing.T) {
@ -287,3 +288,48 @@ func TestSourceClockRejectsDroppedFrameAtNativeTimeBase(t *testing.T) {
t.Fatal("video with frame times 0,0.1,0.3,0.4 admitted as fixed-rate")
}
}
// Frame threading must preserve presentation order even with B-frame reordering.
func TestSourceClockAcceptsBFramesWithParallelDecoder(t *testing.T) {
if _, err := exec.LookPath("ffmpeg"); err != nil {
t.Skip("ffmpeg unavailable")
}
path := filepath.Join(t.TempDir(), "bframes.mp4")
args := []string{"-v", "error", "-y", "-f", "lavfi", "-i", "testsrc2=s=128x96:r=30:d=4", "-c:v", "libx264", "-bf", "3", "-g", "60", path}
if out, err := exec.CommandContext(t.Context(), "ffmpeg", args...).CombinedOutput(); err != nil {
t.Fatalf("encode: %v %s", err, out)
}
duration, err := ffprobeDurationSeconds(t.Context(), path)
if err != nil {
t.Fatal(err)
}
if err := validateSourceMediaClock(t.Context(), path, 120, duration); err != nil {
t.Fatal(err)
}
}
// Oversized decoder output must release its worker slot instead of waiting for
// the job timeout with a blocked stdout writer.
func TestSourceClockOversizedOutputReleasesProcess(t *testing.T) {
dir := t.TempDir()
script := `#!/bin/sh
case "$*" in
*stream=time_base*) printf '%s' '{"streams":[{"time_base":"1/30","r_frame_rate":"30/1"}]}'; exit 0 ;;
esac
while :; do printf '%01000d' 0; done
`
if err := os.WriteFile(filepath.Join(dir, "ffprobe"), []byte(script), 0700); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", dir)
ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
defer cancel()
started := time.Now()
err := validateSourceMediaClock(ctx, filepath.Join(dir, "fake.mp4"), 30, 1)
if err == nil || !strings.Contains(err.Error(), "token too long") {
t.Fatalf("wanted bounded scanner failure, got %v", err)
}
if ctx.Err() != nil || time.Since(started) > 3*time.Second {
t.Fatal("malformed decoder retained worker until timeout")
}
}

View File

@ -0,0 +1,85 @@
package api
import (
"encoding/binary"
"fmt"
"io"
"math"
"os"
)
// MP4 edit-list durations use the movie clock, not the video's track clock.
// Read only atom headers and mvhd: a large mdat is skipped without loading it.
func sourceMovieClockTick(path string) (float64, error) {
f, err := os.Open(path)
if err != nil {
return 0, err
}
defer f.Close()
info, err := f.Stat()
if err != nil {
return 0, err
}
var scan func(int64, int64, bool) (float64, error)
scan = func(start, end int64, inMovie bool) (float64, error) {
for offset, boxes := start, 0; offset+8 <= end && boxes < 4096; boxes++ {
var header [16]byte
if _, err := f.ReadAt(header[:8], offset); err != nil {
return 0, err
}
size, headerSize := uint64(binary.BigEndian.Uint32(header[:4])), int64(8)
if size == 1 {
if _, err := f.ReadAt(header[8:], offset+8); err != nil {
return 0, err
}
size, headerSize = binary.BigEndian.Uint64(header[8:]), 16
} else if size == 0 {
size = uint64(end - offset)
}
if size < uint64(headerSize) || size > uint64(end-offset) {
return 0, fmt.Errorf("invalid MP4 atom size")
}
kind := string(header[4:8])
if kind == "moov" && !inMovie {
return scan(offset+headerSize, offset+int64(size), true)
}
if kind == "mvhd" && inMovie {
var version [1]byte
if _, err := f.ReadAt(version[:], offset+headerSize); err != nil {
return 0, err
}
clockOffset := int64(12)
if version[0] == 1 {
clockOffset = 20
} else if version[0] != 0 {
return 0, fmt.Errorf("unsupported MP4 movie clock")
}
if headerSize+clockOffset+4 > int64(size) {
return 0, io.ErrUnexpectedEOF
}
var clock [4]byte
if _, err := f.ReadAt(clock[:], offset+headerSize+clockOffset); err != nil {
return 0, err
}
timescale := binary.BigEndian.Uint32(clock[:])
if timescale == 0 {
return 0, fmt.Errorf("invalid MP4 movie timescale")
}
return 1 / float64(timescale), nil
}
offset += int64(size)
}
return 0, fmt.Errorf("MP4 movie clock unavailable")
}
return scan(0, info.Size(), false)
}
func sourceClockEndTolerance(path string, trackTick, period float64) float64 {
tolerance := math.Max(0.0001, trackTick+1e-9)
if movieTick, err := sourceMovieClockTick(path); err == nil {
// Both conversions can round. Never turn the container exception into
// permission to lose a whole final frame; decoded cadence stays strict.
tolerance = math.Max(tolerance, math.Min(movieTick+trackTick+1e-9, period/2))
}
return tolerance
}

View File

@ -0,0 +1,78 @@
package api
import (
"bytes"
"encoding/binary"
"fmt"
"math"
"os"
"os/exec"
"path/filepath"
"testing"
)
func TestSourceClockAcceptsEditListTailRoundingWithoutRelaxingFrames(t *testing.T) {
if _, err := exec.LookPath("ffmpeg"); err != nil {
t.Skip("ffmpeg unavailable")
}
for _, count := range []int{6343, 7957, 11239} {
t.Run(fmt.Sprint(count), func(t *testing.T) {
path := filepath.Join(t.TempDir(), "edit-list.mp4")
args := []string{"-v", "error", "-y", "-f", "lavfi", "-i", "color=c=navy:s=32x32:r=30", "-frames:v", fmt.Sprint(count), "-c:v", "libx264", "-preset", "ultrafast", "-bf", "3", "-video_track_timescale", "15360", "-movie_timescale", "1000", "-use_editlist", "1", path}
if out, err := exec.CommandContext(t.Context(), "ffmpeg", args...).CombinedOutput(); err != nil {
t.Fatalf("encode: %v %s", err, out)
}
body, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
at := bytes.Index(body, []byte("elst"))
if at < 4 || at+24 > len(body) || body[at+4] != 0 || binary.BigEndian.Uint32(body[at+8:]) != 1 {
t.Fatal("fixture needs one version-0 edit")
}
writeEndpoint := func(milliseconds uint32) {
binary.BigEndian.PutUint32(body[at+12:], milliseconds)
if err := os.WriteFile(path, body, 0600); err != nil {
t.Fatal(err)
}
}
writeEndpoint(uint32(count * 1000 / 30))
duration, err := ffprobeDurationSeconds(t.Context(), path)
if err != nil {
t.Fatal(err)
}
if delta := float64(count)/30 - duration; delta < .0003 || delta > .0004 {
t.Fatalf("fixture does not reproduce endpoint rounding: %.9f", delta)
}
if err := validateSourceMediaClock(t.Context(), path, count, duration); err != nil {
t.Fatalf("complete CFR source rejected: %v", err)
}
if err := validateSourceMediaClock(t.Context(), path, count-1, duration); err == nil {
t.Fatal("frame/PTS mismatch admitted")
}
writeEndpoint(uint32((count - 1) * 1000 / 30))
duration, err = ffprobeDurationSeconds(t.Context(), path)
if err != nil {
t.Fatal(err)
}
if err := validateSourceMediaClock(t.Context(), path, count, duration); err == nil {
t.Fatal("whole-frame endpoint truncation admitted")
}
})
}
}
func TestSourceMovieClockMalformedInputKeepsStrictTolerance(t *testing.T) {
for _, data := range [][]byte{nil, []byte("not MP4"), {0xff, 0xff, 0xff, 0xff, 'm', 'o', 'o', 'v'}, {0, 0, 0, 1, 'm', 'o', 'o', 'v'}} {
path := filepath.Join(t.TempDir(), "bad.mp4")
if err := os.WriteFile(path, data, 0600); err != nil {
t.Fatal(err)
}
if _, err := sourceMovieClockTick(path); err == nil {
t.Fatal("malformed movie clock accepted")
}
if got := sourceClockEndTolerance(path, 1.0/15360, 1.0/30); math.Abs(got-.0001) > 1e-12 {
t.Fatal(got)
}
}
}

View File

@ -85,10 +85,10 @@ func (s *server) materializeSourceIndexRevision(ctx context.Context, t domain.An
if segment.Start < slice.Start || segment.End > slice.End {
return nil, fmt.Errorf("动作不能跨越切片边界")
}
actions = append(actions, map[string]any{"id": segment.ID, "status": effectiveActionStatus(segment), "isInvalid": segment.IsInvalid, "description": segment.Description, "descriptionTemplateId": segment.LabelID, "phrases": segment.Phrases, "source": segment.Source, "confidence": segment.Confidence, "clipStartMs": (segment.Start - slice.Start) * 1000, "clipEndMs": (segment.End - slice.Start) * 1000, "startTimestampUs": segment.StartTimestampUS, "endTimestampUs": segment.EndTimestampUS})
actions = append(actions, map[string]any{"id": segment.ID, "status": effectiveActionStatus(segment), "isInvalid": segment.IsInvalid, "description": segment.Description, "descriptionTemplateId": segment.LabelID, "dictionary": segment.Dictionary, "phrases": segment.Phrases, "source": segment.Source, "confidence": segment.Confidence, "clipStartMs": (segment.Start - slice.Start) * 1000, "clipEndMs": (segment.End - slice.Start) * 1000, "startTimestampUs": segment.StartTimestampUS, "endTimestampUs": segment.EndTimestampUS})
}
window := map[string]any{"startTimestampUs": slice.StartTimestampUS, "endTimestampUs": slice.EndTimestampUS, "semantics": "half-open"}
annotation := map[string]any{"schemaVersion": "droboticflow.timestamped-description/v3", "boundaryContract": "hardware-frame-v1", "sliceId": id, "sliceName": slice.Name, "window": window, "actions": actions, "descriptions": actions}
annotation := map[string]any{"schemaVersion": "droboticflow.timestamped-description/v3", "boundaryContract": "hardware-frame-v1", "sliceId": id, "sliceName": slice.Name, "dictionary": slice.Dictionary, "window": window, "actions": actions, "descriptions": actions}
seconds, rightSeconds, stamps := []float64{}, []float64{}, []int64{}
for j := a; j <= b; j++ {
seconds = append(seconds, grid.Seconds[j]-slice.Start)
@ -96,8 +96,8 @@ func (s *server) materializeSourceIndexRevision(ctx context.Context, t domain.An
stamps = append(stamps, grid.Timestamps[j])
}
frames := map[string]any{"left": []int{leftStart, leftEnd}, "right": []int{rightStart, rightEnd}}
sync := map[string]any{"schemaVersion": "droboticflow.synchronized-slice/v1", "sourceIndexSchema": sourceIndexSchema, "boundaryContract": "hardware-frame-v1", "collectionId": cert.CollectionID, "sliceIndex": i + 1, "sliceCount": len(slices), "sliceName": slice.Name, "window": window, "boundarySeconds": seconds, "rightBoundarySeconds": rightSeconds, "boundaryTimestampUs": stamps, "sourceFrames": frames, "sourceMediaOffsets": map[string]any{"left": slice.Start, "right": cert.Timeline.RightSeconds[a]}, "deliveryDisposition": sliceDisposition(*slice), "left": map[string]any{"pts": map[string]any{"rowCount": leftEnd - leftStart}}, "right": map[string]any{"pts": map[string]any{"rowCount": rightEnd - rightStart}}, "imu": map[string]any{"sampleCount": cert.IMURowsBeforeBoundary[b] - cert.IMURowsBeforeBoundary[a]}, "commonRange": map[string]any{"startTimestampUs": grid.Timestamps[0], "endTimestampUs": grid.Timestamps[len(grid.Timestamps)-1], "startSecond": grid.Seconds[0], "endSecond": grid.Seconds[len(grid.Seconds)-1]}, "coverage": map[string]any{"mode": "full_common_range", "contiguous": true}}
entry := map[string]any{"sliceId": id, "sourceSliceId": slice.ID, "sliceName": slice.Name, "startSecond": slice.Start, "endSecond": slice.End, "startTimestampUs": slice.StartTimestampUS, "endTimestampUs": slice.EndTimestampUS, "directory": filepath.ToSlash(filepath.Join("slices", id)), "sourceFrames": frames, "deliveryDisposition": sliceDisposition(*slice)}
sync := map[string]any{"schemaVersion": "droboticflow.synchronized-slice/v1", "sourceIndexSchema": sourceIndexSchema, "boundaryContract": "hardware-frame-v1", "collectionId": cert.CollectionID, "sliceIndex": i + 1, "sliceCount": len(slices), "sliceName": slice.Name, "dictionary": slice.Dictionary, "window": window, "boundarySeconds": seconds, "rightBoundarySeconds": rightSeconds, "boundaryTimestampUs": stamps, "sourceFrames": frames, "sourceMediaOffsets": map[string]any{"left": slice.Start, "right": cert.Timeline.RightSeconds[a]}, "deliveryDisposition": sliceDisposition(*slice), "left": map[string]any{"pts": map[string]any{"rowCount": leftEnd - leftStart}}, "right": map[string]any{"pts": map[string]any{"rowCount": rightEnd - rightStart}}, "imu": map[string]any{"sampleCount": cert.IMURowsBeforeBoundary[b] - cert.IMURowsBeforeBoundary[a]}, "commonRange": map[string]any{"startTimestampUs": grid.Timestamps[0], "endTimestampUs": grid.Timestamps[len(grid.Timestamps)-1], "startSecond": grid.Seconds[0], "endSecond": grid.Seconds[len(grid.Seconds)-1]}, "coverage": map[string]any{"mode": "full_common_range", "contiguous": true}}
entry := map[string]any{"sliceId": id, "sourceSliceId": slice.ID, "sliceName": slice.Name, "dictionary": slice.Dictionary, "startSecond": slice.Start, "endSecond": slice.End, "startTimestampUs": slice.StartTimestampUS, "endTimestampUs": slice.EndTimestampUS, "directory": filepath.ToSlash(filepath.Join("slices", id)), "sourceFrames": frames, "deliveryDisposition": sliceDisposition(*slice)}
if slice.IsDiscarded() {
entry["discardEvidence"] = discardReasonEvidence(*slice)
sync["discardEvidence"] = discardReasonEvidence(*slice)
@ -126,12 +126,20 @@ func (s *server) materializeSourceIndexRevision(ctx context.Context, t domain.An
coverage := map[string]any{"mode": "full_common_range", "contiguous": true, "gapMicroseconds": 0, "overlapMicroseconds": 0, "startSecond": grid.Seconds[0], "endSecond": grid.Seconds[previous], "durationSeconds": grid.Seconds[previous] - grid.Seconds[0], "startTimestampUs": grid.Timestamps[0], "endTimestampUs": grid.Timestamps[previous], "ptsToMediaScale": cert.Timeline.Scale, "sliceCount": len(slices), "reconstruction": map[string]any{"leftFrameCount": grid.LeftFrames[previous] - grid.LeftFrames[0], "rightFrameCount": grid.RightFrames[previous] - grid.RightFrames[0], "imuRowCount": cert.IMURowsBeforeBoundary[previous] - cert.IMURowsBeforeBoundary[0]}}
selection := map[string]any{"keptSliceCount": kept, "discardedSliceCount": discarded, "keptDurationSeconds": keepDuration, "discardedDurationSeconds": discardDuration}
capture := map[string]any{}
var originalSource map[string]any
if record, e := s.store.GetCaptureRecordByRunID(ctx, t.RunID); e != nil {
return nil, e
} else if record != nil {
originalSource, e = annotationRevisionOriginalSource(run, record)
if e != nil {
return nil, e
}
capture = map[string]any{"captureRecordId": record.ID, "captureId": record.CaptureID, "prefix": record.Prefix, "fingerprint": record.Fingerprint, "contractRevision": record.ContractRevision}
}
manifest := map[string]any{"schemaVersion": sourceIndexSchema, "revision": revision, "taskId": t.ID, "humanItemId": h.ID, "runId": t.RunID, "collectionId": cert.CollectionID, "capture": capture, "sourceFingerprint": fingerprint, "sources": cert.Sources, "sourceDurations": map[string]any{"left": cert.LeftDuration, "right": cert.RightDuration}, "status": "materialized", "sliceCount": len(items), "coverage": coverage, "deliverySelection": selection, "items": items}
if originalSource != nil {
manifest["originalSource"] = originalSource
}
manifest["sourceFrameCounts"] = map[string]any{"left": len(cert.LeftPTS), "right": len(cert.RightPTS)}
encoded, e := json.Marshal(manifest)
if e != nil || len(encoded)+1 > maxRunArtifactManifestBytes {

View File

@ -7860,6 +7860,12 @@
"type": "boolean",
"value": "false",
"description": "原生 v3:part.json 与独立审核历史,通过目录发布;须同时启用 PART_INDEX_DELIVERY"
},
{
"key": "DELIVERY_JSON_V4",
"type": "boolean",
"value": "false",
"description": "最终分类总 JSON;审核返工追溯单独保存在内部,须启用 PART_INDEX_DELIVERY"
}
],
"id": "operator-builtin-package",
@ -8170,6 +8176,16 @@
"role": "summary",
"transmission": "oss",
"type": "json"
},
{
"id": "deliveryAudit",
"name": "deliveryAudit",
"type": "json",
"container": "single",
"role": "internal-audit",
"payloadMode": "inline",
"transmission": "bus",
"description": "内部审核与返工追溯,不属于客户交付文件"
}
],
"publicationStatus": "built",

View File

@ -10,6 +10,7 @@ import (
"time"
"droboticflow/internal/domain"
"droboticflow/internal/outputsnapshot"
"droboticflow/internal/store"
"github.com/go-chi/chi/v5"
@ -147,6 +148,9 @@ func normalizeDatabaseBrowserTarget(raw string) databaseBrowserTarget {
func validateDatabaseCollectionName(field string, value string) *validationIssue {
name := strings.TrimSpace(value)
if outputsnapshot.IsPrivateCollection(name) {
return &validationIssue{Field: field, Message: "internal statistics collection is not accessible"}
}
label := "collection name"
if field == "newName" {
label = "new collection name"

View File

@ -0,0 +1,125 @@
package api
import (
"encoding/json"
"fmt"
"net/http"
"strings"
"droboticflow/internal/deliveryindex"
"droboticflow/internal/domain"
)
// Read only the audit produced by the same completed, tenant-scoped package.
// Customer JSON deliberately contains no private run pointers.
func (s *server) classificationPackageAudit(r *http.Request, packages []domain.DeliveryPackageRecord, target deliveryDownloadTarget, partBytes []byte) ([]byte, string, error) {
var part struct {
Capture struct {
ID string `json:"captureId"`
} `json:"capture"`
PartID string `json:"partId"`
Revision int `json:"revision"`
}
if json.Unmarshal(partBytes, &part) != nil {
return nil, "", fmt.Errorf("invalid classification part")
}
var matched []byte
sourceRun := ""
for _, pkg := range packages {
run, err := s.store.GetRun(r.Context(), pkg.ChildRunID)
if err != nil {
return nil, "", err
}
if run == nil || run.OrganizationID != s.requestOrganizationID(r) || run.ProjectID != pkg.ProjectID || run.Status != domain.RunStatusCompleted {
continue
}
ownsTarget := false
for _, a := range run.Artifacts {
if !deliveryPackageArtifactAllowed(&pkg, a.ID) || a.RunID != run.ID || a.Payload == nil || a.Payload.Ref == nil {
continue
}
ref := a.Payload.Ref
if ref.ConnectionID == target.connectionID && ref.Bucket == target.bucket && strings.Trim(ref.Prefix, "/") == target.prefix && ref.ManifestKey == target.prefix+"/manifest.json" {
ownsTarget = true
}
}
if !ownsTarget {
continue
}
for _, a := range run.Artifacts {
if a.RunID != run.ID || !domain.IsInternalDeliveryAudit(a) {
continue
}
data, readErr := s.readClassificationAudit(r, run, a)
if readErr != nil {
return nil, "", readErr
}
raw, err := json.Marshal(data)
if err != nil || len(raw) > 8<<20 {
return nil, "", fmt.Errorf("内部追溯不可读或超过 8 MiB")
}
var audit struct {
Schema string `json:"schemaVersion"`
CaptureID string `json:"captureId"`
PartID string `json:"partId"`
Revision int `json:"revision"`
PartSHA string `json:"partSha256"`
Lineage struct {
RunID string `json:"sourceRunId"`
} `json:"lineage"`
}
if json.Unmarshal(raw, &audit) != nil || audit.Schema != "droboticflow.delivery.review-history/v2" || audit.CaptureID != part.Capture.ID || audit.PartID != part.PartID || audit.Revision != part.Revision || audit.PartSHA != deliveryindex.Hash(partBytes) || audit.Lineage.RunID != pkg.ParentRunID {
return nil, "", fmt.Errorf("内部追溯与固定验收结果不一致")
}
if matched != nil {
return nil, "", fmt.Errorf("交付分类对应多个内部追溯")
}
matched, sourceRun = raw, audit.Lineage.RunID
}
}
if matched == nil || sourceRun == "" {
return nil, "", fmt.Errorf("缺少同一正式交付任务的内部验收证据")
}
return matched, sourceRun, nil
}
func (s *server) readClassificationAudit(r *http.Request, run *domain.RunRecord, a domain.ExecutionArtifact) (map[string]any, error) {
if a.Payload != nil && a.Payload.Ref != nil {
ref := a.Payload.Ref
if ref.Kind != domain.ArtifactRefKindObject || ref.SizeBytes <= 0 || ref.SizeBytes > 8<<20 || len(ref.SHA256) != 64 {
return nil, fmt.Errorf("内部追溯引用缺少完整校验")
}
connection, err := s.resolveArtifactStorageConnection(r.Context(), run, a, ref.Bucket)
if err != nil || connection == nil || connection.ID != ref.ConnectionID {
return nil, fmt.Errorf("内部追溯连接不可访问")
}
browser, cleanup, err := storageConnectionBrowserResolver(r.Context(), s.storageConnectionWithResolvedEndpoint(connection))
if err != nil {
return nil, err
}
if cleanup != nil {
defer cleanup()
}
raw, _, err := browser.ReadObjectPreview(r.Context(), ref.Bucket, ref.Key, 8<<20+1)
if err != nil || int64(len(raw)) != ref.SizeBytes || deliveryindex.Hash(raw) != ref.SHA256 {
return nil, fmt.Errorf("内部追溯文件校验失败")
}
var data map[string]any
if json.Unmarshal(raw, &data) != nil {
return nil, fmt.Errorf("内部追溯 JSON 无效")
}
return data, nil
}
if a.Payload != nil && a.Payload.Mode == domain.ArtifactTransferModeInline {
raw, err := json.Marshal(a.Payload.Inline)
if err != nil || len(raw) > 8<<20 {
return nil, fmt.Errorf("内部追溯过大")
}
var data map[string]any
if json.Unmarshal(raw, &data) != nil {
return nil, fmt.Errorf("内部追溯 JSON 无效")
}
return data, nil
}
return artifactStructuredData(&a), nil
}

View File

@ -16,6 +16,7 @@ import (
"droboticflow/internal/deliveryindex"
"droboticflow/internal/domain"
"droboticflow/internal/sourceidentity"
"droboticflow/internal/workspacescope"
)
@ -99,7 +100,7 @@ func (s *server) planIndexBatch(r *http.Request, q indexBatchRequest) (*batchPla
if e != nil {
return nil, e
}
if len(files) != 3 {
if len(files) != 2 && len(files) != 3 {
return nil, fmt.Errorf("原生 v3 part 包必须只含 part.json 与独立审核历史")
}
var partBytes, historyBytes []byte
@ -124,14 +125,24 @@ func (s *server) planIndexBatch(r *http.Request, q indexBatchRequest) (*batchPla
historyBytes = raw
}
}
if len(partBytes) == 0 || len(historyBytes) == 0 {
if len(partBytes) == 0 {
return nil, fmt.Errorf("选择的包尚未由原生 v3 算子交付;旧格式必须另行重交,不会覆盖旧文件")
}
var part struct {
Schema string `json:"schemaVersion"`
Capture struct {
ID string `json:"captureId"`
Prefix string `json:"prefix"`
} `json:"capture"`
OriginalSource struct {
Provider string `json:"provider"`
Bucket string `json:"bucket"`
Prefix string `json:"prefix"`
PartPrefix string `json:"partPrefix"`
Files map[string]struct {
Path string `json:"path"`
} `json:"files"`
} `json:"originalSource"`
PartID string `json:"partId"`
Lineage struct {
RunID string `json:"sourceRunId"`
@ -140,16 +151,57 @@ func (s *server) planIndexBatch(r *http.Request, q indexBatchRequest) (*batchPla
if json.Unmarshal(partBytes, &part) != nil || target.prefix != root+"/"+part.Capture.ID+"/"+part.PartID {
return nil, fmt.Errorf("part 身份与冻结交付目标不一致")
}
switch part.Schema {
case deliveryindex.ClassificationPartSchema:
if len(files) != 2 || len(historyBytes) != 0 {
return nil, fmt.Errorf("分类交付包中包含额外文件")
}
historyBytes, part.Lineage.RunID, e = s.classificationPackageAudit(r, selected, target, partBytes)
if e != nil {
return nil, e
}
case deliveryindex.PartSchema:
if len(files) != 3 || len(historyBytes) == 0 {
return nil, fmt.Errorf("旧 v3 交付缺少审核历史")
}
default:
return nil, fmt.Errorf("不支持的交付格式;历史交付不会被覆盖")
}
record, e := s.store.GetCaptureRecordByRunID(r.Context(), part.Lineage.RunID)
if e != nil || record == nil || record.OrganizationID != project.OrganizationID || record.ProjectID != project.ID || record.Prefix != part.Capture.Prefix+"/"+part.PartID || record.Delivery == nil || record.Delivery.Prefix != target.prefix || record.Delivery.ConnectionID != t.connectionID || record.Delivery.Bucket != t.bucket {
return nil, fmt.Errorf("part 来源与采集记录不一致")
}
if part.Schema == deliveryindex.ClassificationPartSchema && (part.OriginalSource.Provider != "BOS" || part.OriginalSource.Bucket != record.Bucket || part.OriginalSource.Prefix != part.Capture.Prefix || part.OriginalSource.PartPrefix != record.Prefix) {
return nil, fmt.Errorf("分类 JSON 原始存储来源与采集记录不一致,停止发布")
}
sourceRuns = append(sourceRuns, part.Lineage.RunID)
sourceBrowser, closeSource, e := s.deliveryReadBrowser(r.Context(), project.OrganizationID, record.ConnectionID, record.Bucket, part.Capture.Prefix, false)
if e != nil {
return nil, e
}
expected, meta, e := indexBatchSourceEvidence(r.Context(), sourceBrowser, record.Bucket, part.Capture.Prefix, part.Capture.ID)
if e == nil && part.Schema == deliveryindex.ClassificationPartSchema {
originalRun, runErr := s.store.GetRun(r.Context(), part.Lineage.RunID)
if runErr != nil {
e = runErr
} else {
var selection *sourceidentity.Selection
selection, e = sourceidentity.FromRun(originalRun, record)
if e == nil {
paths := map[string]string{}
for role, file := range part.OriginalSource.Files {
paths[role] = file.Path
}
e = selection.MatchFiles(part.OriginalSource.Provider, part.OriginalSource.Bucket, part.OriginalSource.Prefix, part.OriginalSource.PartPrefix, paths)
}
if e == nil {
e = selection.Recheck(r.Context(), sourceBrowser.GetObjectSummary)
}
}
if e != nil {
e = fmt.Errorf("原始数据身份核验失败,停止发布:%w", e)
}
}
if closeSource != nil {
closeSource()
}
@ -425,14 +477,14 @@ func readIndexBatchDownloadManifest(ctx context.Context, b objectStorageBrowser,
} `json:"project"`
Entries []deliveryindex.Entry `json:"entries"`
}
if json.Unmarshal(raw, &index) != nil || index.Schema != 3 || index.Kind != "part-index" || index.ID != id || index.Project.Root != t.prefix || index.Count != len(index.Entries) || index.Count < 1 || index.Count > 400 {
if json.Unmarshal(raw, &index) != nil || (index.Schema != 3 && index.Schema != 4) || index.Kind != "part-index" || index.ID != id || index.Project.Root != t.prefix || index.Count != len(index.Entries) || index.Count < 1 || index.Count > 400 {
return nil, fmt.Errorf("批次总索引身份或成员数量无效")
}
files := []deliveryDownloadFile{}
var size int64
for _, entry := range index.Entries {
sha, shaErr := hex.DecodeString(entry.SHA256)
if !validDeliveryDownloadPath(entry.Path) || entry.Path == "delivery_index.json" || strings.HasPrefix(entry.Path, ".") || entry.Key != t.prefix+"/"+entry.Path || entry.SizeBytes < 0 || entry.SizeBytes > 8<<20 || shaErr != nil || len(sha) != 32 {
if (index.Schema == 4 && !validClassificationBatchPath(entry.Path)) || !validDeliveryDownloadPath(entry.Path) || entry.Path == "delivery_index.json" || strings.HasPrefix(entry.Path, ".") || entry.Key != t.prefix+"/"+entry.Path || entry.SizeBytes < 0 || entry.SizeBytes > 8<<20 || shaErr != nil || len(sha) != 32 {
return nil, fmt.Errorf("批次成员校验信息无效")
}
size += entry.SizeBytes
@ -444,3 +496,15 @@ func readIndexBatchDownloadManifest(ctx context.Context, b objectStorageBrowser,
files = append(files, deliveryDownloadFile{RelativePath: t.relative + "/delivery_index.json", SizeBytes: int64(len(raw)), SHA256: deliveryindex.Hash(raw), key: t.prefix + "/delivery_index.json", bucket: t.bucket, connectionID: t.connectionID})
return files, nil
}
func validClassificationBatchPath(value string) bool {
parts := strings.Split(value, "/")
if len(parts) != 2 {
return false
}
if parts[1] == "annotation_manifest.json" {
return true
}
name := strings.TrimSuffix(parts[1], ".json")
return strings.HasSuffix(parts[1], ".json") && len(name) >= 8 && strings.HasPrefix(name, "part") && strings.Trim(name[4:], "0123456789") == ""
}

View File

@ -15,6 +15,7 @@ import (
"droboticflow/internal/deliveryindex"
"droboticflow/internal/domain"
"droboticflow/internal/sourceidentity"
"droboticflow/internal/store"
)
@ -25,6 +26,14 @@ type indexTestBrowser struct {
ambiguous bool
}
func (b *indexTestBrowser) GetObjectSummary(_ context.Context, bucket, key string) (domain.ObjectStorageObjectSummary, error) {
body, ok := b.objects[key]
if !ok {
return domain.ObjectStorageObjectSummary{}, os.ErrNotExist
}
return domain.ObjectStorageObjectSummary{Bucket: bucket, Key: key, Kind: "file", SizeBytes: int64(len(body)), ETag: deliveryindex.Hash(body)}, nil
}
func (b *indexTestBrowser) PutObjectConditional(_ context.Context, _ string, key string, body []byte, _ string, match, none string, _ map[string]string) (domain.ObjectStorageObjectSummary, error) {
if match != "" || none != "*" {
return domain.ObjectStorageObjectSummary{}, fmt.Errorf("unconditional write")
@ -108,6 +117,10 @@ func seedIndexFixture(t *testing.T, ds store.Store) (*indexTestBrowser, *authent
must(ds.UpsertProject(ctx, domain.ProjectRecord{ID: "project-A", OrganizationID: org, Name: "项目A"}))
must(ds.UpsertStorageConnection(ctx, domain.StorageConnectionRecord{ID: "index-storage", OrganizationID: org, Provider: domain.StorageProviderBOS, Bucket: "test-bucket", Endpoint: "https://bj.bcebos.com"}))
dir := os.Getenv("DELIVERY_V3_FIXTURE_DIR")
v4 := os.Getenv("DELIVERY_CLASSIFICATION_TEST") == "1"
if v4 {
dir = os.Getenv("DELIVERY_V4_FIXTURE_DIR")
}
for _, part := range []string{"part0001", "part0002"} {
prefix := root + "/capture-A/" + part
sourceRun := "source-run-" + part
@ -119,22 +132,80 @@ func seedIndexFixture(t *testing.T, ds store.Store) (*indexTestBrowser, *authent
var e error
body, e = os.ReadFile(filepath.Join(dir, part, "part.json"))
must(e)
history, e = os.ReadFile(filepath.Join(dir, part, "review-history.json"))
historyPath := filepath.Join(dir, part, "review-history.json")
if v4 {
historyPath = filepath.Join(dir, "internal", part, "review-history.json")
}
history, e = os.ReadFile(historyPath)
must(e)
}
if v4 && dir == "" {
var doc map[string]any
must(json.Unmarshal(body, &doc))
acceptance := doc["acceptance"]
delete(doc, "acceptance")
delete(doc, "lineage")
doc["schemaVersion"] = deliveryindex.ClassificationPartSchema
enrichConsumerFixture(doc)
body, _ = json.Marshal(doc)
history, _ = json.Marshal(map[string]any{"schemaVersion": "droboticflow.delivery.review-history/v2", "captureId": "capture-A", "partId": part, "revision": 1, "partSha256": deliveryindex.Hash(body), "acceptance": acceptance, "lineage": map[string]any{"sourceRunId": sourceRun}, "qualityReviews": []any{map[string]any{"comment": "PRIVATE_REWORK"}}, "trace": []any{}})
}
b.objects[prefix+"/.delivery-owner.json"] = []byte(`{"owner":"synthetic-source"}`)
b.objects[prefix+"/part.json"] = body
b.objects[prefix+"/review-history.json"] = history
entries := []deliveryindex.Entry{{Path: "part.json", Key: prefix + "/part.json", SizeBytes: int64(len(body)), SHA256: deliveryindex.Hash(body)}, {Path: "review-history.json", Key: prefix + "/review-history.json", SizeBytes: int64(len(history)), SHA256: deliveryindex.Hash(history)}}
b.objects[prefix+"/manifest.json"], _ = json.Marshal(map[string]any{"schemaVersion": 1, "kind": "directory", "fileCount": 2, "totalBytes": len(body) + len(history), "entries": entries})
if v4 {
delete(b.objects, prefix+"/review-history.json")
entries = entries[:1]
b.objects[prefix+"/manifest.json"], _ = json.Marshal(map[string]any{"schemaVersion": 1, "kind": "directory", "fileCount": 1, "totalBytes": len(body), "entries": entries})
}
b.objects["incoming/capture-A/"+part+"/left.mp4"] = []byte("synthetic")
if v4 {
var doc struct {
Original struct {
Files map[string]struct {
Path string `json:"path"`
} `json:"files"`
} `json:"originalSource"`
}
must(json.Unmarshal(body, &doc))
for _, file := range doc.Original.Files {
b.objects["incoming/capture-A/"+file.Path] = []byte("synthetic original")
}
}
run := domain.RunRecord{ID: runID, OrganizationID: org, ProjectID: "project-A", Status: domain.RunStatusCompleted, Artifacts: []domain.ExecutionArtifact{{ID: artifactID, RunID: runID, Name: "delivery.directory.index", Payload: &domain.ArtifactPayload{Mode: domain.ArtifactTransferModeRef, Ref: &domain.ArtifactReference{Kind: domain.ArtifactRefKindDirectory, Provider: domain.StorageProviderBOS, ConnectionID: "index-storage", Bucket: "test-bucket", Prefix: prefix, ManifestKey: prefix + "/manifest.json"}}}}}
if v4 {
var audit map[string]any
must(json.Unmarshal(history, &audit))
run.Artifacts = append(run.Artifacts, domain.ExecutionArtifact{ID: "audit-" + part, RunID: runID, PortID: "deliveryAudit", Type: domain.PortTypeJSON, Data: audit})
}
run.NodeExecutions = []domain.NodeExecutionRecord{{NodeID: "package", Artifacts: run.Artifacts}}
must(ds.UpsertRun(ctx, run))
must(ds.UpsertDeliveryPackage(ctx, domain.DeliveryPackageRecord{ID: "package-" + part, OrganizationID: org, ProjectID: "project-A", ChildRunID: runID, Status: domain.DeliveryPackageAccepted, Mode: domain.RunModeProduction, FinalExportArtifactIDs: []string{artifactID}}))
must(ds.UpsertDeliveryPackage(ctx, domain.DeliveryPackageRecord{ID: "package-" + part, OrganizationID: org, ProjectID: "project-A", ParentRunID: sourceRun, ChildRunID: runID, Status: domain.DeliveryPackageAccepted, Mode: domain.RunModeProduction, FinalExportArtifactIDs: []string{artifactID}}))
must(ds.UpsertCaptureRecord(ctx, domain.CaptureRecord{ID: "record-" + part, OrganizationID: org, ProjectID: "project-A", RunID: sourceRun, CaptureID: "capture-A/" + part, ConnectionID: "index-storage", Bucket: "test-bucket", Prefix: "incoming/capture-A/" + part, SourcePrefix: "incoming/capture-A", Delivery: &domain.CaptureDeliveryTarget{ConnectionID: "index-storage", Bucket: "test-bucket", Prefix: prefix}}))
}
b.objects["incoming/capture-A/meta_capture-A.json"] = []byte(`{"synthetic":true}`)
if v4 {
for _, part := range []string{"part0001", "part0002"} {
var doc struct {
Original struct {
Files map[string]struct {
Path string `json:"path"`
} `json:"files"`
} `json:"originalSource"`
}
must(json.Unmarshal(b.objects[root+"/capture-A/"+part+"/part.json"], &doc))
keys := []string{}
objects := []sourceidentity.Object{}
for _, file := range doc.Original.Files {
key := "incoming/capture-A/" + file.Path
keys = append(keys, key)
objects = append(objects, sourceidentity.Object{Key: key, ETag: deliveryindex.Hash(b.objects[key]), Size: int64(len(b.objects[key]))})
}
must(ds.UpsertRun(ctx, domain.RunRecord{ID: "source-run-" + part, OrganizationID: org, ProjectID: "project-A", Inputs: map[string]domain.ArtifactPayload{"source": {Ref: &domain.ArtifactReference{Provider: domain.StorageProviderBOS, Kind: domain.ArtifactRefKindDirectory, ConnectionID: "index-storage", Bucket: "test-bucket", Prefix: "incoming/capture-A", CaptureUnit: "capture-A/" + part, ObjectKeys: keys, VersionID: sourceidentity.Fingerprint(objects)}}}}))
}
}
old := storageConnectionBrowserResolver
storageConnectionBrowserResolver = func(_ context.Context, c *domain.StorageConnectionRecord) (objectStorageBrowser, func(), error) {
if c.OrganizationID != org {
@ -207,6 +278,16 @@ func exerciseIndexBatch(t *testing.T, ds store.Store) {
if download.Code != 200 || !strings.Contains(download.Body.String(), "delivery_index.json") {
t.Fatal(download.Code, download.Body.String())
}
if os.Getenv("DELIVERY_CLASSIFICATION_TEST") == "1" {
for _, file := range files {
if strings.Contains(file.RelativePath, "history") || bytes.Contains(b.objects[file.key], []byte("PRIVATE_REWORK")) {
t.Fatal("private audit delivered")
}
}
if strings.Contains(download.Body.String(), "history") {
t.Fatal("download exposes private audit")
}
}
// A later partial snapshot does not alter the old full snapshot.
first := indexCall(h, p, "GET", "/api/delivery-index-batches?projectId=project-A&packageId=package-part0001", nil)
var partial struct {
@ -299,3 +380,113 @@ func TestIndexBatchDelegationKeepsSourceAndDeliveryScopesSeparate(t *testing.T)
t.Fatal("sibling batch prefix escaped delivery grant", w.Code, w.Body.String())
}
}
func TestClassificationBatchHTTPPublicationRecoveryAndDownload(t *testing.T) {
t.Setenv("DELIVERY_CLASSIFICATION_TEST", "1")
exerciseIndexBatch(t, store.NewMemoryStore())
}
func TestClassificationBatchRejectsMissingOrChangedOriginalSource(t *testing.T) {
t.Setenv("DELIVERY_CLASSIFICATION_TEST", "1")
t.Setenv("DELIVERY_V4_FIXTURE_DIR", "")
for _, failure := range []string{"bucket", "missing-file"} {
t.Run(failure, func(t *testing.T) {
ds := store.NewMemoryStore()
b, p, _ := seedIndexFixture(t, ds)
if failure == "bucket" {
record, err := ds.GetCaptureRecordByRunID(t.Context(), "source-run-part0001")
if err != nil {
t.Fatal(err)
}
record.Bucket = "changed-bucket"
if err = ds.UpsertCaptureRecord(t.Context(), *record); err != nil {
t.Fatal(err)
}
} else {
delete(b.objects, "incoming/capture-A/part0001/capture-A_camera_left_part0001.mp4")
}
w := indexCall(NewHandler(ds, nil), p, "GET", "/api/delivery-index-batches?projectId=project-A&packageId=package-part0001", nil)
if w.Code != 409 || len(b.puts) != 0 || !strings.Contains(w.Body.String(), "原始") {
t.Fatalf("incomplete source accepted: %d %s", w.Code, w.Body.String())
}
})
}
}
func enrichConsumerFixture(p map[string]any) {
part := p["partId"].(string)
p["consumerContract"] = "droboticflow.classification-consumer/v1"
p["timeBases"] = map[string]string{"taskSplitSeconds": "original_left_video", "segmentSeconds": "slice_left_video", "segmentPartSeconds": "original_left_video", "timestampUs": "original_hardware_pts", "interval": "half-open", "frameRanges": "zero-based-half-open"}
p["pathBases"] = map[string]string{"sources": "original_capture", "sourceMetadata": "original_capture", "calibration": "original_capture"}
files := map[string]any{}
sources := map[string]any{}
for _, role := range []string{"leftVideo", "rightVideo", "leftPts", "rightPts", "imu", "cameraCalibration", "imuCalibration", "metadata"} {
name := part + "/" + role
switch role {
case "leftVideo":
name = part + "/capture-A_camera_left_" + part + ".mp4"
case "rightVideo":
name = part + "/capture-A_camera_right_" + part + ".mp4"
case "leftPts":
name = part + "/capture-A_camera_left_" + part + "_pts.csv"
case "rightPts":
name = part + "/capture-A_camera_right_" + part + "_pts.csv"
case "imu":
name = part + "/capture-A_imu_" + part + ".csv"
}
if role == "cameraCalibration" || role == "imuCalibration" {
name = "capture-A_calibration_camera.yaml"
if role == "imuCalibration" {
name = "capture-A_calibration_imu.yaml"
}
}
if role == "metadata" {
name = "meta_capture-A.json"
}
files[role] = map[string]any{"path": name}
if role == "leftVideo" || role == "rightVideo" || role == "leftPts" || role == "rightPts" || role == "imu" {
sources[role] = files[role]
}
}
p["sources"] = sources
p["sourceMetadata"] = files["metadata"]
p["originalSource"] = map[string]any{"provider": "BOS", "bucket": "test-bucket", "prefix": "incoming/capture-A", "partPrefix": "incoming/capture-A/" + part, "files": files}
for _, kind := range []string{"slices", "discardedSlices"} {
for _, raw := range p[kind].([]any) {
item := raw.(map[string]any)
split := item["taskSplit"].(map[string]any)
number := func(v any) float64 {
switch n := v.(type) {
case float64:
return n
case int:
return float64(n)
default:
panic("invalid fixture time")
}
}
start, end := number(split["startSecond"]), number(split["endSecond"])
split["startTimestampUs"] = int64(1e6 + start*1e6)
split["endTimestampUs"] = int64(1e6 + end*1e6)
if kind == "discardedSlices" {
continue
}
class := item["classification"].(map[string]any)
class["dictionary"] = map[string]any{"id": "base", "kind": "slice", "version": 1, "name": "操作", "tags": []string{"操作"}}
class["tags"] = []string{"操作"}
actions := item["segments"].([]any)
for i, raw := range actions {
a := raw.(map[string]any)
low, high := float64(i)*(end-start)/float64(len(actions)), float64(i+1)*(end-start)/float64(len(actions))
a["startSecond"] = low
a["endSecond"] = high
a["partStartSecond"] = start + low
a["partEndSecond"] = start + high
a["startTimestampUs"] = int64(1e6 + (start+low)*1e6)
a["endTimestampUs"] = int64(1e6 + (start+high)*1e6)
a["tags"] = []string{"操作"}
a["tagSource"] = "slice"
}
}
}
}

View File

@ -0,0 +1,166 @@
package api
import (
"bytes"
"context"
"encoding/json"
"net/http"
"strings"
"testing"
"droboticflow/internal/deliveryindex"
"droboticflow/internal/domain"
"droboticflow/internal/sourceidentity"
"droboticflow/internal/store"
)
type identityIndexBrowser struct {
*indexTestBrowser
heads int
}
func (b *identityIndexBrowser) GetObjectSummary(ctx context.Context, bucket, key string) (domain.ObjectStorageObjectSummary, error) {
info, err := b.indexTestBrowser.GetObjectSummary(ctx, bucket, key)
if err == nil {
info.ETag = deliveryindex.Hash(b.objects[key])
b.heads++
}
return info, err
}
func seedOriginalIdentityIndex(t *testing.T, ds store.Store) (*identityIndexBrowser, *authenticatedPrincipal) {
t.Helper()
t.Setenv("DELIVERY_CLASSIFICATION_TEST", "1")
t.Setenv("DELIVERY_V4_FIXTURE_DIR", "")
base, p, root := seedIndexFixture(t, ds)
b := &identityIndexBrowser{indexTestBrowser: base}
for _, part := range []string{"part0001", "part0002"} {
prefix := root + "/capture-A/" + part
body := base.objects[prefix+"/part.json"]
replacements := map[string]string{"leftVideo": part + "/capture-A_camera_left_" + part + ".mp4", "rightVideo": part + "/capture-A_camera_right_" + part + ".mp4", "leftPts": part + "/capture-A_camera_left_" + part + "_pts.csv", "rightPts": part + "/capture-A_camera_right_" + part + "_pts.csv", "imu": part + "/capture-A_imu_" + part + ".csv", "cameraCalibration": "capture-A_calibration_camera.yaml", "imuCalibration": "capture-A_calibration_imu.yaml", "metadata": "meta_capture-A.json"}
keys := []string{}
objects := []sourceidentity.Object{}
for role, rel := range replacements {
oldRel := part + "/" + role
if role == "cameraCalibration" || role == "imuCalibration" {
oldRel = role + ".yaml"
}
if role == "metadata" {
oldRel = "meta_capture-A.json"
}
body = bytes.ReplaceAll(body, []byte(`"`+oldRel+`"`), []byte(`"`+rel+`"`))
key := "incoming/capture-A/" + rel
if role != "metadata" {
base.objects[key] = []byte("original synthetic object")
}
keys = append(keys, key)
objects = append(objects, sourceidentity.Object{Key: key, ETag: deliveryindex.Hash(base.objects[key]), Size: int64(len(base.objects[key]))})
}
base.objects[prefix+"/part.json"] = body
entries := []deliveryindex.Entry{{Path: "part.json", Key: prefix + "/part.json", SizeBytes: int64(len(body)), SHA256: deliveryindex.Hash(body)}}
base.objects[prefix+"/manifest.json"], _ = json.Marshal(map[string]any{"schemaVersion": 1, "kind": "directory", "fileCount": 1, "totalBytes": len(body), "entries": entries})
child, err := ds.GetRun(t.Context(), "package-run-"+part)
if err != nil {
t.Fatal(err)
}
for i := range child.Artifacts {
if domain.IsInternalDeliveryAudit(child.Artifacts[i]) {
child.Artifacts[i].Data["partSha256"] = deliveryindex.Hash(body)
}
}
child.NodeExecutions[0].Artifacts = child.Artifacts
if err = ds.UpsertRun(t.Context(), *child); err != nil {
t.Fatal(err)
}
capture, err := ds.GetCaptureRecordByRunID(t.Context(), "source-run-"+part)
if err != nil {
t.Fatal(err)
}
original := domain.RunRecord{ID: capture.RunID, OrganizationID: capture.OrganizationID, ProjectID: capture.ProjectID, WorkflowID: capture.WorkflowID, Inputs: map[string]domain.ArtifactPayload{"source": {Ref: &domain.ArtifactReference{Provider: domain.StorageProviderBOS, Kind: domain.ArtifactRefKindDirectory, ConnectionID: capture.ConnectionID, Bucket: capture.Bucket, Prefix: capture.SourcePrefix, CaptureUnit: capture.CaptureID, ObjectKeys: keys, VersionID: sourceidentity.Fingerprint(objects)}}}}
if err = ds.UpsertRun(t.Context(), original); err != nil {
t.Fatal(err)
}
}
storageConnectionBrowserResolver = func(context.Context, *domain.StorageConnectionRecord) (objectStorageBrowser, func(), error) {
return b, nil, nil
}
return b, p
}
func TestClassificationBatchOriginalIdentityHTTP(t *testing.T) {
for _, scenario := range []string{"correct", "same-size-etag", "changed-size", "missing", "missing-frozen-version", "wrong-tenant", "extra-selection"} {
t.Run(scenario, func(t *testing.T) {
ds := store.NewMemoryStore()
b, p := seedOriginalIdentityIndex(t, ds)
h := NewHandler(ds, nil)
query := "/api/delivery-index-batches?projectId=project-A&packageId=package-part0001"
preview := indexCall(h, p, http.MethodGet, query, nil)
if preview.Code != 200 {
t.Fatalf("valid preview: %d %s", preview.Code, preview.Body.String())
}
var plan struct {
ID string `json:"snapshotId"`
}
json.Unmarshal(preview.Body.Bytes(), &plan)
key := "incoming/capture-A/part0001/capture-A_camera_left_part0001.mp4"
switch scenario {
case "same-size-etag":
b.objects[key] = bytes.Repeat([]byte("x"), len(b.objects[key]))
case "changed-size":
b.objects[key] = append(b.objects[key], 'x')
case "missing":
delete(b.objects, key)
case "missing-frozen-version", "wrong-tenant", "extra-selection":
run, _ := ds.GetRun(t.Context(), "source-run-part0001")
switch scenario {
case "missing-frozen-version":
run.Inputs["source"].Ref.VersionID = ""
case "wrong-tenant":
run.OrganizationID = "other"
case "extra-selection":
run.Inputs["source"].Ref.ObjectKeys = append(run.Inputs["source"].Ref.ObjectKeys, "incoming/capture-A/extra.json")
}
if err := ds.UpsertRun(t.Context(), *run); err != nil {
t.Fatal(err)
}
}
result := indexCall(h, p, http.MethodPost, "/api/delivery-index-batches", indexBatchRequest{ProjectID: "project-A", PackageIDs: []string{"package-part0001"}, SnapshotID: plan.ID})
if scenario == "correct" {
if result.Code != 200 {
t.Fatalf("publish: %d %s", result.Code, result.Body.String())
}
if b.heads < 16 {
t.Fatalf("POST did not recheck: %d", b.heads)
}
} else if result.Code != 409 || len(b.puts) != 0 {
t.Fatalf("changed source published: %d %s writes=%d", result.Code, result.Body.String(), len(b.puts))
}
})
}
}
func TestClassificationBatchRejectsDeliveryFilesOutsideFrozenSelection(t *testing.T) {
ds := store.NewMemoryStore()
b, p := seedOriginalIdentityIndex(t, ds)
// Change the immutable run selection while preserving a valid selection shape.
run, _ := ds.GetRun(t.Context(), "source-run-part0001")
ref := run.Inputs["source"].Ref
for i, key := range ref.ObjectKeys {
if strings.HasSuffix(key, "_left_part0001.mp4") {
ref.ObjectKeys[i] = strings.TrimSuffix(key, ".mp4") + "_alternate.mp4"
b.objects[ref.ObjectKeys[i]] = b.objects[key]
}
}
objects := []sourceidentity.Object{}
for _, key := range ref.ObjectKeys {
objects = append(objects, sourceidentity.Object{Key: key, ETag: deliveryindex.Hash(b.objects[key]), Size: int64(len(b.objects[key]))})
}
ref.VersionID = sourceidentity.Fingerprint(objects)
if err := ds.UpsertRun(t.Context(), *run); err != nil {
t.Fatal(err)
}
result := indexCall(NewHandler(ds, nil), p, http.MethodGet, "/api/delivery-index-batches?projectId=project-A&packageId=package-part0001", nil)
if result.Code != 409 || len(b.puts) > 0 {
t.Fatalf("files outside frozen selection accepted: %d %s", result.Code, result.Body.String())
}
}

Some files were not shown because too many files have changed in this diff Show More