forked from longtao.wu/droboticflow
🔀 Integrate delivery v4 with layered frontend while preserving legacy materialization
This commit is contained in:
commit
3549ad6216
33
.scratch/delivery-classification-20261021/spec.md
Normal file
33
.scratch/delivery-classification-20261021/spec.md
Normal file
@ -0,0 +1,33 @@
|
||||
# 最终分类交付与内部追溯分离
|
||||
|
||||
Status: done
|
||||
|
||||
2026-10-21 用户要求将两轮总 JSON 核查的问题一起本地修复,并明确选择独立 `tags` 列表、管理员维护。依据为马迪 10-09 至 10-14 的格式对话与本会话补充。原始核查证据在主仓 `local-data/diagnostics/delivery-json-audit-20261021/`。
|
||||
|
||||
## 验收条件
|
||||
|
||||
- 字典词条独立 tags,不从 category/aliases 猜补。选择字典后 ID、类型、版本、名称、编码、tags 经服务器校验并冻结,切片/动作修订、物化、审查及最终 JSON 保持一致;字典后改不改写已确认结果。
|
||||
- 未写动作描述仍保留切片基础分类和全部区间。显式无效与舍弃独立,不从空文本推断;舍弃原因是分类事实,保留。纯留空、混合、全无效及全舍弃均可表达。
|
||||
- 新分类交付使用显式 v4 协议及默认关闭的 DELIVERY_JSON_V4(要求 PART_INDEX_DELIVERY)。客户主 JSON 保持 capture/part、slices/taskSplit/segments 的约定形态,只含定位、最终分类、字典快照、边界及必要源引用;动作 contentKind 使用马迪约定的 annotated/invalid_marked/gap_placeholder。
|
||||
- v4 全舍弃 part 仍输出分类 JSON。旧目录/v1/v2/v3 的 no_deliverables 行为、历史文件和在途任务兼容保留,不静默重写旧批次。
|
||||
- 交付只取最终验收通过的固定物化修订与摘要;返工正式修正须重新通过验收。批注、人员、轮次变化不能改分类,审核者不能通过标注接口修改业务结果。
|
||||
- 全部已提交追溯留内部 JSON,按 capture/part/结果版本关联:审核人、批注、轮次、退回/复审及正式处理记录。客户主 JSON 不带审核历史、lineage、抽检过程或内部路径引用,客户下载不包含内部追溯文件。
|
||||
- 不可变批次的选择/身份/SHA/范围/条件发布/重试规则保留;版本必须显式识别,拒绝混搭和不一致证据。历史缺少字典标签不得按最新字典猜补,保留已有名称事实和未知字段边界。
|
||||
- 舍弃 reasonNote/at 与 note/discardedAt 兼容映射;原始 meta/标定路径基准显式且一致。规范、真实入口样例与校验器枚举一致。
|
||||
|
||||
## 验证
|
||||
|
||||
定向 Go/Python/前端回归,隔离本机 Mongo 与真实 router 的保存→返工→复审→验收→原生打包→批次及下载,检查最终分类/标签和完整内部追溯;覆盖失败重试、重启/旧版本读取、字典更新、未授权篡改、全舍弃、缺失历史证据。前端与 Go 全量门禁按既有基线对照;真实浏览器检查字典 tags 管理和选择,Web/Go 构建及生产包启动。
|
||||
|
||||
不上传镜像、不部署、不写公司数据、不补写历史交付、不 push/PR。本地完成后以新分支提交。
|
||||
|
||||
|
||||
## 本地完成证据(2026-10-21)
|
||||
|
||||
- 独立分支 `codex/delivery-classification-20261021`,基于 `bc00467d`。本次只做本地代码、合成数据验证及提交;未 push、提 PR、上传镜像、部署、启用工作流或改写历史交付。
|
||||
- Go 与前端全量门禁无新增失败,保留原有 70 / 13 个已知失败,基线文件未修改;类型、格式、构建、生成源码一致性及生产页面启动通过。
|
||||
- 定向 Python 37 项通过(含真实视频物化→v4 的 5 项验证);一个旧版 Go 物化 opt-in 用例未单独启用,新旧真实媒体链路已由独立用例覆盖。
|
||||
- 本机 27018 随机独立 Mongo 库的三轮退回和 API 重启、原生 v4 的正式接口批次发布/失败恢复/重试/下载、停用与移除词条后的直接及面板撤销重做、v5/v6 标签防伪造均通过。测试库执行后清理。
|
||||
- 静态复核的 Standards / Spec 两轴发现均已修正并关闭;保留无快照历史的名称事实,不以当前 tags 猜补。未确认临时名称不能绕过最终确认/提交。
|
||||
- 完整日志、分离的客户/内部样例和浏览器截图保留在本工作树 `local-data/verification/delivery-classification-20261021/`。客户样例子目录 `customer-samples/partNNNN/` 每份只含 `part.json`;`customer-samples/internal/` 不属于客户包。
|
||||
- 新协议默认关闭,线上尚未验证/启用;后续正式发布需要覆盖 API/processor、Worker、package 镜像及内部 artifact 存储,按现有发布授权流程执行。未把旧镜像 release 摘要伪装成新源码摘要。
|
||||
21
.scratch/delivery-consumer-contract-20261021/spec.md
Normal file
21
.scratch/delivery-consumer-contract-20261021/spec.md
Normal file
@ -0,0 +1,21 @@
|
||||
Implement approved local v4 contract only. Freeze source ref metadata in revision; require complete customer v4 original source fields. Derive tags from own dictionary or parent frozen dictionary without overwriting selection provenance. Explicit timestamps + local/part seconds, half-open time bases and original PTS resolution. Consumer roundtrip uses only exported JSON + synthetic original source. Legacy formats remain unchanged. Commit local only, no deploy.
|
||||
|
||||
Status: resolved
|
||||
|
||||
Acceptance:
|
||||
- Effective tags and action/slice origin are explicit; independent selections and invalid flags remain intact.
|
||||
- part → slices → segments hierarchy and discard-reason-only semantics remain intact.
|
||||
- Frozen original BOS references survive metadata-only reuse; no connection credentials or workspace paths enter customer output.
|
||||
- Action local/part times and hardware PTS boundaries are complete and continuous.
|
||||
- Missing tags, source, timestamps, source files or conflicting identities reject v4 package/batch publication.
|
||||
- Independent reader resolves every synthetic original camera frame from JSON alone in physical and source-index modes.
|
||||
- Immutable retry/recovery, separate private audit, legacy formats and local Mongo publication remain verified.
|
||||
|
||||
Compatibility: v4 default stays off; no production changes, retrospective tag guessing, old delivery replacement, downstream recutting or source writes.
|
||||
|
||||
Verification:
|
||||
- 73 relevant Python tests passed, including physical/source-index independent real-media consumption, legacy v1/v2/v3 and 60-second materialization.
|
||||
- Native Python v4 output passed Go batch/API publication, failure recovery and download tests with an isolated local Mongo 27018 database.
|
||||
- Go full baseline gate: the same 70 named known failures, no new failures and no baseline edits.
|
||||
- Operator repository generation check and git diff whitespace check passed.
|
||||
- Local evidence: `local-data/verification/delivery-consumer-20261021/`; final customer samples under `final-samples/{physical,source-index}/customer/part0001/part.json`.
|
||||
@ -2,6 +2,14 @@
|
||||
|
||||
以下词汇描述人工标注与审核中的业务对象。
|
||||
|
||||
**分类标签**:管理员为字典词条明确维护的 tags 集合,用于按已确认类别归组;它不同于词条分类 category、别名 aliases 或动作描述文本。确认后的词条身份、版本和标签共同构成当次分类快照,字典后续更新不改写该快照。
|
||||
|
||||
**最终分类结果**:最终验收通过的固定版本中,视频各区间的基础分类、动作细分、显式无效及舍弃判断。未写动作描述不撤销基础分类;审核批注和返工过程不属于分类结果。
|
||||
|
||||
**动作有效标签**:动作独立选择描述字典词条时,使用当次选择冻结的标签;未独立选择时,继承所属切片冻结的分类标签。是否填写自由描述不决定继承,无效判断与标签分别保留。
|
||||
|
||||
**原始区间定位**:以原始采集及 part 的文件身份和硬件时间戳定位人工确认区间;切片内相对时间与原始 part 相对时间是不同坐标,不能混用。下游按定位和分类处理原始数据,不依赖平台内部审核文件。
|
||||
|
||||
## 标注对象
|
||||
|
||||
**Capture(源采集)**:一次源采集所关联的双目视频、时间戳与传感器数据,是本流程组织人工处理和审核的对象。
|
||||
|
||||
@ -5,6 +5,7 @@ import hashlib
|
||||
import bisect
|
||||
import csv
|
||||
import json
|
||||
import math
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
@ -752,9 +753,11 @@ def _part_index_discarded_slice(
|
||||
"reason": str(evidence.get("reason") or ""),
|
||||
"reasonLabelId": str(evidence.get("reasonLabelId") or ""),
|
||||
"reasonName": str(evidence.get("reasonName") or ""),
|
||||
"note": str(evidence.get("note") or ""),
|
||||
"note": str(evidence.get("note") or evidence.get("reasonNote") or ""),
|
||||
"reasonVersion": evidence.get("reasonVersion"),
|
||||
"dictionary": evidence.get("dictionary"),
|
||||
"discardedByUserId": str(evidence.get("discardedByUserId") or ""),
|
||||
"discardedAt": str(evidence.get("discardedAt") or ""),
|
||||
"discardedAt": str(evidence.get("discardedAt") or evidence.get("at") or ""),
|
||||
},
|
||||
},
|
||||
"segments": [],
|
||||
@ -843,13 +846,184 @@ def _description_status(actions: list[dict[str, Any]]) -> str:
|
||||
return "empty_confirmed"
|
||||
|
||||
|
||||
def _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload):
|
||||
"""Use the established immutable directory publisher for the two JSON files.
|
||||
def _v4_tags(selection: Any, kind: str) -> list[str]:
|
||||
if not isinstance(selection, dict) or selection.get("kind") != kind or not str(selection.get("id") or "").strip() or type(selection.get("version")) is not int or selection["version"] < 1 or not str(selection.get("name") or "").strip():
|
||||
raise RuntimeError(f"v4 missing frozen {kind} dictionary; confirm historical classification before delivery")
|
||||
tags = selection.get("tags")
|
||||
if not isinstance(tags, list) or not tags or any(not isinstance(tag, str) or not tag.strip() or tag != tag.strip() for tag in tags) or len(set(tags)) != len(tags):
|
||||
raise RuntimeError(f"v4 missing frozen {kind} tags; confirm classification before delivery")
|
||||
return list(tags)
|
||||
|
||||
|
||||
def _v4_original_sources(manifest, capture_id, part_id, capture_prefix):
|
||||
source = manifest.get("originalSource")
|
||||
if not isinstance(source, dict) or source.get("provider") != "BOS" or not str(source.get("bucket") or "").strip():
|
||||
raise RuntimeError("v4 requires frozen original BOS source selection; workspace paths cannot identify original data")
|
||||
prefix = str(source.get("prefix") or "")
|
||||
part_prefix = str(source.get("partPrefix") or "")
|
||||
if prefix != capture_prefix or part_prefix != prefix + "/" + part_id or source.get("captureUnit") != capture_id + "/" + part_id:
|
||||
raise RuntimeError("v4 original source identity disagrees with accepted capture/part")
|
||||
if not prefix or str(pathlib.PurePosixPath(prefix)) != prefix or pathlib.PurePosixPath(prefix).is_absolute() or ".." in pathlib.PurePosixPath(prefix).parts or any(c in prefix for c in ("\\", "\x00")):
|
||||
raise RuntimeError("unsafe v4 original source prefix")
|
||||
keys = source.get("objectKeys")
|
||||
if not isinstance(keys, list) or not keys or any(not isinstance(key, str) for key in keys) or len(set(keys)) != len(keys):
|
||||
raise RuntimeError("v4 requires an unambiguous original object selection")
|
||||
files = {}
|
||||
cap, part = re.escape(capture_id), re.escape(part_id)
|
||||
patterns = {
|
||||
"leftVideo": rf"{part}/{cap}_camera_left_{part}(?:_[a-z0-9_]+)?\.(?:mp4|mov|mkv)",
|
||||
"rightVideo": rf"{part}/{cap}_camera_right_{part}(?:_[a-z0-9_]+)?\.(?:mp4|mov|mkv)",
|
||||
"leftPts": rf"{part}/{cap}_camera_left_{part}(?:_[a-z0-9_]+)?_pts\.csv",
|
||||
"rightPts": rf"{part}/{cap}_camera_right_{part}(?:_[a-z0-9_]+)?_pts\.csv",
|
||||
"imu": rf"{part}/{cap}_imu_{part}\.csv",
|
||||
"cameraCalibration": rf"{cap}_calibration_camera\.yaml",
|
||||
"imuCalibration": rf"{cap}_calibration_imu\.yaml",
|
||||
"metadata": rf"meta_{cap}\.json",
|
||||
}
|
||||
for key in keys:
|
||||
if not key.startswith(prefix + "/") or str(pathlib.PurePosixPath(key)) != key or ".." in pathlib.PurePosixPath(key).parts or any(c in key for c in ("\\", "\x00")):
|
||||
raise RuntimeError("unsafe v4 original source object")
|
||||
relative = key[len(prefix) + 1:]
|
||||
if "/" in relative and not relative.startswith(part_id + "/"):
|
||||
raise RuntimeError("v4 original selection contains another part")
|
||||
for role, pattern in patterns.items():
|
||||
if re.fullmatch(pattern, relative, re.IGNORECASE):
|
||||
if role in files: raise RuntimeError(f"ambiguous v4 original {role} selection")
|
||||
files[role] = {"path": relative}
|
||||
missing = sorted(set(patterns) - files.keys())
|
||||
if missing: raise RuntimeError("v4 missing original files: " + ", ".join(missing))
|
||||
return {"provider": "BOS", "bucket": source["bucket"], "prefix": prefix, "partPrefix": part_prefix,
|
||||
"snapshotFingerprint": str(source.get("snapshotFingerprint") or ""), "files": files}
|
||||
|
||||
|
||||
def _v4_number(value, label):
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value):
|
||||
raise RuntimeError(f"v4 missing or invalid {label}")
|
||||
return float(value)
|
||||
|
||||
|
||||
def _v4_timestamp(value, label):
|
||||
if type(value) is not int or value <= 0: raise RuntimeError(f"v4 missing or invalid {label}")
|
||||
return value
|
||||
|
||||
|
||||
def _v4_consumer_payload(payload, manifest, source_root):
|
||||
"""Project immutable accepted facts into a self-contained consumer contract.
|
||||
|
||||
No dictionary lookup, source media reads, transcoding or BOS writes here.
|
||||
Hardware PTS are authoritative; media seconds are explicitly named and never
|
||||
used to manufacture missing hardware timestamps.
|
||||
"""
|
||||
if not isinstance(manifest, dict): raise RuntimeError("v4 requires accepted source manifest")
|
||||
original = _v4_original_sources(manifest, payload["capture"]["captureId"], payload["partId"], payload["capture"]["prefix"])
|
||||
payload["originalSource"] = original
|
||||
payload["consumerContract"] = "droboticflow.classification-consumer/v1"
|
||||
payload["timeBases"] = {"taskSplitSeconds": "original_left_video", "segmentSeconds": "slice_left_video",
|
||||
"segmentPartSeconds": "original_left_video", "timestampUs": "original_hardware_pts",
|
||||
"interval": "half-open", "frameRanges": "zero-based-half-open"}
|
||||
payload["pathBases"] = {"sources": "original_capture", "sourceMetadata": "original_capture", "calibration": "original_capture"}
|
||||
# Workspace sources are private transport references, not the original files.
|
||||
payload["sources"] = {kind: original["files"][kind] for kind in ("leftVideo", "rightVideo", "leftPts", "rightPts", "imu")}
|
||||
payload["sourceMetadata"]["path"] = original["files"]["metadata"]["path"]
|
||||
for item in payload["calibration"]:
|
||||
# Bind by actual calibration kind, not input ordering.
|
||||
kind = "imuCalibration" if "imu" in item["name"].lower() else "cameraCalibration" if "camera" in item["name"].lower() else ""
|
||||
if not kind: raise RuntimeError("v4 unknown calibration kind")
|
||||
item["path"] = original["files"][kind]["path"]
|
||||
rows = {str(row["sliceId"]): row for row in manifest["items"]}
|
||||
if len(rows) != len(manifest["items"]): raise RuntimeError("v4 duplicate slice identities")
|
||||
for item in payload["slices"] + payload["discardedSlices"]:
|
||||
split = item["taskSplit"]
|
||||
start, end = (_v4_number(split[k], k) for k in ("startSecond", "endSecond"))
|
||||
first, last = (_v4_timestamp(split[k], k) for k in ("startTimestampUs", "endTimestampUs"))
|
||||
if start < 0 or end <= start or last <= first: raise RuntimeError("v4 invalid accepted slice range")
|
||||
if item["classification"]["deliveryDisposition"] == "discard":
|
||||
item["sliceName"] = ""
|
||||
evidence = item["classification"]["discardEvidence"]
|
||||
selected = evidence.get("dictionary")
|
||||
if selected is not None:
|
||||
if (not isinstance(selected, dict) or selected.get("kind") != "discard_reason" or
|
||||
not selected.get("id") or not selected.get("name") or
|
||||
type(selected.get("version")) is not int or selected["version"] < 1):
|
||||
raise RuntimeError("v4 invalid frozen discard dictionary")
|
||||
for alias, field in (("reasonLabelId", "id"), ("reasonName", "name"), ("reasonVersion", "version")):
|
||||
if evidence.get(alias) not in (None, "", 0, selected[field]):
|
||||
raise RuntimeError("v4 discard reason disagrees with frozen dictionary")
|
||||
evidence[alias] = selected[field]
|
||||
continue
|
||||
parent_tags = _v4_tags(item["classification"].get("dictionary"), "slice")
|
||||
item["classification"]["tags"] = parent_tags
|
||||
row = rows[item["sliceId"]]
|
||||
sync = json.loads((source_root / row["directory"] / "synchronized-slice.json").read_text())
|
||||
seconds, stamps = sync.get("boundarySeconds", []), sync.get("boundaryTimestampUs", [])
|
||||
if seconds or stamps:
|
||||
if not isinstance(seconds, list) or not isinstance(stamps, list) or len(seconds) < 2 or len(seconds) != len(stamps):
|
||||
raise RuntimeError("v4 invalid hardware boundary mapping")
|
||||
seconds = [_v4_number(value, "boundary") for value in seconds]
|
||||
stamps = [_v4_timestamp(value, "boundary timestamp") for value in stamps]
|
||||
if (abs(seconds[0]) > 1e-7 or abs(seconds[-1] - (end-start)) > 1e-7 or
|
||||
stamps[0] != first or stamps[-1] != last or
|
||||
any(a >= b for a, b in zip(seconds, seconds[1:])) or
|
||||
any(a >= b for a, b in zip(stamps, stamps[1:]))):
|
||||
raise RuntimeError("v4 invalid hardware boundary mapping")
|
||||
def timestamp(second):
|
||||
if abs(second) < 1e-7: return first
|
||||
if abs(second - (end-start)) < 1e-7: return last
|
||||
hits = [stamps[i] for i, value in enumerate(seconds) if abs(_v4_number(value,"boundary") - second) < 1e-7]
|
||||
if len(hits) != 1: raise RuntimeError("v4 action boundary has no original PTS mapping")
|
||||
return _v4_timestamp(hits[0], "boundary timestamp")
|
||||
actions = []
|
||||
for action in item["segments"]:
|
||||
a = _v4_number(action.get("clipStartMs"), "action clipStartMs") / 1000
|
||||
b = _v4_number(action.get("clipEndMs"), "action clipEndMs") / 1000
|
||||
if a < -1e-7 or b <= a or b > end-start+1e-7: raise RuntimeError("v4 action escapes its parent slice")
|
||||
ta, tb = (_v4_timestamp(action.get(k), k) for k in ("startTimestampUs", "endTimestampUs"))
|
||||
if not first <= ta < tb <= last: raise RuntimeError("v4 action timestamps escape its parent slice")
|
||||
if seconds and (timestamp(a) != ta or timestamp(b) != tb): raise RuntimeError("v4 action seconds disagree with original PTS")
|
||||
action.update(startSecond=a, endSecond=b, partStartSecond=start+a, partEndSecond=start+b)
|
||||
own = action.get("dictionary")
|
||||
action["tags"] = _v4_tags(own, "action") if own is not None else list(parent_tags)
|
||||
action["tagSource"] = "action" if own is not None else "slice"
|
||||
actions.append(action)
|
||||
actions.sort(key=lambda a: a["startSecond"])
|
||||
complete, previous = [], 0.0
|
||||
def gap(a,b):
|
||||
ta,tb=timestamp(a),timestamp(b)
|
||||
return {"id":f"{item['sliceId']}:gap:{ta}:{tb}", "description":"", "isInvalid":False,
|
||||
"contentKind":"gap_placeholder", "startSecond":a,"endSecond":b,
|
||||
"clipStartMs":a*1000,"clipEndMs":b*1000,"partStartSecond":start+a,"partEndSecond":start+b,
|
||||
"startTimestampUs":ta,"endTimestampUs":tb,"tags":list(parent_tags),"tagSource":"slice"}
|
||||
for action in actions:
|
||||
if action["startSecond"] < previous-1e-7: raise RuntimeError("v4 overlapping actions")
|
||||
if action["startSecond"] > previous+1e-7: complete.append(gap(previous,action["startSecond"]))
|
||||
complete.append(action);previous=action["endSecond"]
|
||||
if previous < end-start-1e-7: complete.append(gap(previous,end-start))
|
||||
if not complete: raise RuntimeError("v4 kept slice has no classified action coverage")
|
||||
previous_stamp = first
|
||||
for action in complete:
|
||||
if action["startTimestampUs"] != previous_stamp or action["endTimestampUs"] <= previous_stamp:
|
||||
raise RuntimeError("v4 action PTS coverage is discontinuous")
|
||||
previous_stamp = action["endTimestampUs"]
|
||||
if previous_stamp != last: raise RuntimeError("v4 action PTS coverage is incomplete")
|
||||
item["segments"] = complete
|
||||
counts = {kind:sum(a["contentKind"]==kind for a in complete) for kind in ("annotated","gap_placeholder","invalid_marked")}
|
||||
item["classification"]["contentCounts"] = counts
|
||||
item["classification"]["descriptionStatus"] = _description_status(complete)
|
||||
if not counts["annotated"] and counts["invalid_marked"] and counts["gap_placeholder"]: item["classification"]["descriptionStatus"]="invalid_and_blank"
|
||||
item["synchronization"]["rightBoundarySeconds"] = sync.get("rightBoundarySeconds", [])
|
||||
|
||||
|
||||
def _emit_part_index_directory(run_id, node_id, output, node_data, helpers, payload, internal_audit=None, revision_manifest=None, source_root=None):
|
||||
"""Use the established immutable publisher for versioned customer JSON.
|
||||
|
||||
V3 retains its historical sibling review file; v4 stores that audit in a
|
||||
separate internal artifact before publishing the customer directory.
|
||||
|
||||
Legacy single-file publishers deliberately accept only v1/v2. A directory
|
||||
artifact preserves their ownership/conditional-publication protocol without
|
||||
teaching an old worker to misinterpret a new single-file schema.
|
||||
"""
|
||||
v4 = internal_audit is not None
|
||||
capture = dict(payload["capture"])
|
||||
prefix = pathlib.PurePosixPath(str(capture.get("prefix") or ""))
|
||||
part_id = prefix.name
|
||||
@ -860,7 +1034,7 @@ def _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload):
|
||||
if prefix.is_absolute() or ".." in prefix.parts or len(prefix.parts) < 3:
|
||||
raise RuntimeError("unsafe v3 capture identity prefix")
|
||||
capture.update(captureId=capture_id, prefix=str(prefix.parent))
|
||||
payload.update(schemaVersion="droboticflow.delivery.part-index/v3", capture=capture, partId=part_id)
|
||||
payload.update(schemaVersion="droboticflow.delivery.part-index/v4" if v4 else "droboticflow.delivery.part-index/v3", capture=capture, partId=part_id)
|
||||
payload["pathBases"] = {"sourceMetadata": "original_capture", "calibration": "original_capture", "sources": "workspace"}
|
||||
payload["sourceMetadata"] = {"path": f"meta_{capture_id}.json"}
|
||||
history = {
|
||||
@ -869,7 +1043,8 @@ def _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload):
|
||||
"qualityReviews": payload["acceptance"].pop("qualityReviews", []),
|
||||
}
|
||||
payload["reviewHistory"] = {"path": "review-history.json", "pathBase": "part_package"}
|
||||
payload["acceptance"]["finalAcceptance"].pop("confirmedByUserId", None)
|
||||
if not v4:
|
||||
payload["acceptance"]["finalAcceptance"].pop("confirmedByUserId", None)
|
||||
for item in payload["calibration"]:
|
||||
if pathlib.PurePosixPath(item["name"]).name != item["name"] or "\\" in item["name"]:
|
||||
raise RuntimeError("unsafe calibration filename")
|
||||
@ -889,6 +1064,54 @@ def _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload):
|
||||
classification["contentCounts"] = counts
|
||||
if not counts["described"] and counts["invalid_marked"] and counts["gap_placeholder"]:
|
||||
classification["descriptionStatus"] = "invalid_and_blank"
|
||||
if v4:
|
||||
if not isinstance(internal_audit.get("trace"), list):
|
||||
raise RuntimeError("DELIVERY_JSON_V4 requires complete internal trace input")
|
||||
if _output(node_data, helpers, "deliveryAudit") is None:
|
||||
raise RuntimeError("DELIVERY_JSON_V4 requires an internal deliveryAudit output")
|
||||
audit = {
|
||||
"schemaVersion": "droboticflow.delivery.review-history/v2",
|
||||
"captureId": capture_id, "partId": part_id, "revision": payload["revision"],
|
||||
"acceptance": payload.pop("acceptance"), "lineage": payload.pop("lineage"),
|
||||
"qualityReviews": internal_audit.get("qualityReviews", []),
|
||||
"trace": internal_audit.get("trace", []),
|
||||
"acceptedMaterializations": internal_audit.get("acceptedMaterializations", []),
|
||||
}
|
||||
audit["acceptance"]["finalAcceptance"] = internal_audit.get("finalAcceptance", {})
|
||||
payload.pop("reviewHistory", None)
|
||||
payload["capture"] = {"captureId": capture_id, "prefix": str(prefix.parent)}
|
||||
for item in payload["slices"] + payload["discardedSlices"]:
|
||||
classification = item["classification"]
|
||||
# Blank descriptions retain the independently selected base class.
|
||||
if classification.get("dictionary") is None:
|
||||
classification.pop("dictionary", None)
|
||||
evidence = classification.get("discardEvidence", {})
|
||||
evidence.pop("discardedAt", None)
|
||||
if evidence.get("dictionary") is None:
|
||||
evidence.pop("dictionary", None)
|
||||
counts = classification.get("contentCounts")
|
||||
if counts is not None:
|
||||
counts["annotated"] = counts.pop("described")
|
||||
for action in item["segments"]:
|
||||
if action["contentKind"] == "described": action["contentKind"] = "annotated"
|
||||
for key in ("source", "confidence", "phrases", "labelId"):
|
||||
action.pop(key, None)
|
||||
if action.get("dictionary") is None: action.pop("dictionary", None)
|
||||
audit["materializedSources"] = payload.get("sources", {})
|
||||
_v4_consumer_payload(payload, revision_manifest, source_root)
|
||||
body = (json.dumps(payload, ensure_ascii=False, sort_keys=True, indent=2) + "\n").encode("utf8")
|
||||
audit["partSha256"] = hashlib.sha256(body).hexdigest()
|
||||
if len(json.dumps(audit, ensure_ascii=False, indent=2).encode("utf8")) > 8 * 1024 * 1024:
|
||||
raise RuntimeError("internal delivery audit exceeds 8 MiB; refusing incomplete export")
|
||||
staging = pathlib.Path(tempfile.mkdtemp(prefix="part-index-v4-", dir=output))
|
||||
(staging / "part.json").write_bytes(body)
|
||||
# A sibling artifact stays in the internal run, never in the customer tree.
|
||||
audit_artifact = helpers["write_json_artifact"](run_id, node_id, output, "review-history.json", audit, output=_output(node_data, helpers, "deliveryAudit"))
|
||||
directory = helpers["build_virtual_artifact"](
|
||||
run_id, node_id, "delivery.directory.index", str(staging), "file", 0,
|
||||
data={"schemaVersion": "droboticflow.directory-index/v1", "presentation": "collection", "datasetType": "directory", "dataSource": "FILESYSTEM", "rootPath": str(staging), "sliceCount": len(payload["slices"]), "classificationOnly": True},
|
||||
output=_output(node_data, helpers, "deliveryDirectory"))
|
||||
return [directory, audit_artifact]
|
||||
staging = pathlib.Path(tempfile.mkdtemp(prefix="part-index-v3-", dir=output))
|
||||
for name, document in (("part.json", payload), ("review-history.json", history)):
|
||||
(staging / name).write_text(json.dumps(document, ensure_ascii=False, sort_keys=True, indent=2) + "\n", encoding="utf8")
|
||||
@ -944,6 +1167,7 @@ def _part_index_slice(
|
||||
"clipStartMs": _optional_float(action.get("clipStartMs")),
|
||||
"clipEndMs": _optional_float(action.get("clipEndMs")),
|
||||
"labelId": str(action.get("descriptionTemplateId") or ""),
|
||||
"dictionary": action.get("dictionary"),
|
||||
"phrases": [str(item) for item in action.get("phrases", [])] if isinstance(action.get("phrases"), list) else [],
|
||||
"source": str(action.get("source") or ""),
|
||||
"confidence": _optional_float(action.get("confidence")),
|
||||
@ -976,6 +1200,7 @@ def _part_index_slice(
|
||||
"deliveryDisposition": declared,
|
||||
"taskLabel": str(synchronization.get("taskLabel") or ""),
|
||||
"taskLabelId": str(synchronization.get("taskLabelId") or ""),
|
||||
"dictionary": synchronization.get("dictionary"),
|
||||
"descriptionStatus": _description_status(actions),
|
||||
},
|
||||
"segments": actions,
|
||||
@ -999,9 +1224,11 @@ def _part_index_slice(
|
||||
"reason": str(evidence.get("reason") or ""),
|
||||
"reasonLabelId": str(evidence.get("reasonLabelId") or ""),
|
||||
"reasonName": str(evidence.get("reasonName") or ""),
|
||||
"note": str(evidence.get("note") or ""),
|
||||
"note": str(evidence.get("note") or evidence.get("reasonNote") or ""),
|
||||
"reasonVersion": evidence.get("reasonVersion"),
|
||||
"dictionary": evidence.get("dictionary"),
|
||||
"discardedByUserId": str(evidence.get("discardedByUserId") or ""),
|
||||
"discardedAt": str(evidence.get("discardedAt") or ""),
|
||||
"discardedAt": str(evidence.get("discardedAt") or evidence.get("at") or ""),
|
||||
}
|
||||
# Pose frames deliberately stay out of the index: the customer contract for
|
||||
# this mode is the split information, and embedded per-frame pose would
|
||||
@ -1651,6 +1878,12 @@ def _run_accepted_slice_directory(
|
||||
raise RuntimeError("acceptedSliceDirectory does not prove gap-free full-video coverage")
|
||||
summary_artifact = next((item for item in inputs if _port(item) == "acceptanceSummary"), None)
|
||||
summary_payload = summary_artifact.get("data") if isinstance((summary_artifact or {}).get("data"), dict) else {}
|
||||
if summary_artifact is not None and not isinstance(summary_payload.get("finalAcceptance"), dict):
|
||||
summary_path = _materialize_input_file(summary_artifact, output / "_acceptance", helpers)
|
||||
if summary_path.stat().st_size > 8 * 1024 * 1024:
|
||||
raise RuntimeError("acceptance summary exceeds 8 MiB")
|
||||
summary_payload = json.loads(summary_path.read_text(encoding="utf8"))
|
||||
if not isinstance(summary_payload, dict): raise RuntimeError("invalid acceptance summary reference")
|
||||
final_acceptance = summary_payload.get("finalAcceptance") if isinstance(summary_payload.get("finalAcceptance"), dict) else {}
|
||||
capture_record = _part_index_capture_record(summary_payload, revision_manifest)
|
||||
all_slices = revision_manifest.get("items", [])
|
||||
@ -1669,7 +1902,11 @@ def _run_accepted_slice_directory(
|
||||
discarded_slices.append(row)
|
||||
else:
|
||||
kept_slices.append(row)
|
||||
if disposition_contract:
|
||||
params = helpers["hyperparams"](node_data) if callable(helpers.get("hyperparams")) else {}
|
||||
classification_v4 = _bool(params, "DELIVERY_JSON_V4", False)
|
||||
if classification_v4 and not coverage_required:
|
||||
raise RuntimeError("DELIVERY_JSON_V4 requires verified full-video coverage")
|
||||
if disposition_contract or classification_v4:
|
||||
pins = summary_payload.get("acceptedMaterializations", [])
|
||||
expected_sha = _sha256_file(revision_manifest_path)
|
||||
if summary_payload.get("decision") != "approved" or final_acceptance.get("confirmed") is not True or not any(
|
||||
@ -1679,7 +1916,7 @@ def _run_accepted_slice_directory(
|
||||
raise RuntimeError("slice dispositions are not pinned to the accepted manifest revision")
|
||||
if int(coverage.get("sliceCount", -1)) != len(all_slices):
|
||||
raise RuntimeError("review coverage slice count mismatch")
|
||||
if disposition_contract and not kept_slices:
|
||||
if disposition_contract and not kept_slices and not _bool(params, "DELIVERY_JSON_V4", False):
|
||||
if any(p.get("key") == "PROCESSING_PROFILE" and p.get("value") == "raw_human" for p in node_data.get("hyperparams", [])) and (summary_payload.get("processingProfile") != "raw_human" or final_acceptance.get("processingProfile") != "raw_human"):
|
||||
raise RuntimeError("raw-human package requires matching accepted task profile")
|
||||
# Internal evidence only: deliberately no deliveryDirectory output and
|
||||
@ -1697,7 +1934,6 @@ def _run_accepted_slice_directory(
|
||||
payloads = _payload_tree(_payloads_with_full_pose_manifests(inputs, output, helpers))
|
||||
headpose = _first_payload(payloads, "droboticflow.headpose/v1")
|
||||
handpose = _first_payload(payloads, "droboticflow.wilor.hand_pose_result/v1")
|
||||
params = helpers["hyperparams"](node_data) if callable(helpers.get("hyperparams")) else {}
|
||||
if _bool(params, "PART_INDEX_DELIVERY", False):
|
||||
if source_index and not _bool(params, "SOURCE_INDEX_DELIVERY", False):
|
||||
raise RuntimeError("source-index JSON delivery requires explicit SOURCE_INDEX_DELIVERY approval; use directory delivery to preserve physical discard exclusion")
|
||||
@ -1715,8 +1951,10 @@ def _run_accepted_slice_directory(
|
||||
for item in payload["slices"] + payload["discardedSlices"]:
|
||||
item["sourceFrames"] = frames[item["sliceId"]]
|
||||
body = (json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + "\n").encode("utf8")
|
||||
if _bool(params, "DELIVERY_JSON_V4", False):
|
||||
return _emit_part_index_directory(run_id, node_id, output, node_data, helpers, payload, summary_payload, revision_manifest, source_root)
|
||||
if _bool(params, "DELIVERY_JSON_V3", False):
|
||||
return _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload)
|
||||
return _emit_part_index_directory(run_id, node_id, output, node_data, helpers, payload)
|
||||
return _emit_part_index(run_id, node_id, output, node_data, helpers, body)
|
||||
if source_index:
|
||||
source_root = _source_index_physical_export(source_root, revision_manifest, kept_slices, output)
|
||||
@ -1953,6 +2191,8 @@ def _run_multiport(
|
||||
|
||||
def run(run_id: str, node_id: str, output_dir: Any, node_data: dict[str, Any], inputs: list[dict[str, Any]], helpers: HelperMap) -> list[dict[str, Any]]:
|
||||
params = helpers["hyperparams"](node_data) if callable(helpers.get("hyperparams")) else {}
|
||||
if _bool(params, "DELIVERY_JSON_V4", False) and (not _bool(params, "PART_INDEX_DELIVERY", False) or _bool(params, "DELIVERY_JSON_V3", False)):
|
||||
raise RuntimeError("DELIVERY_JSON_V4 requires PART_INDEX_DELIVERY and cannot combine with DELIVERY_JSON_V3")
|
||||
if _bool(params, "DELIVERY_JSON_V3", False) and not _bool(params, "PART_INDEX_DELIVERY", False):
|
||||
raise RuntimeError("DELIVERY_JSON_V3 requires PART_INDEX_DELIVERY")
|
||||
# A part-index delivery intentionally has no directory port; it must still
|
||||
|
||||
@ -43,3 +43,19 @@ status: accepted
|
||||
路径基准显式写入:总索引相对本批次项目根,采集清单相对采集交付目录,原始 meta 和标定文件相对原始采集目录;已固定 workspace 来源另标基准。采集清单记录原始 `meta_<captureId>.json` 的 SHA/大小,expected parts 来自原始目录完整分页列举,不用已验收 part 数推定交齐。最终验收结论及最大 acceptedAt 的聚合仅覆盖本批次选择的修订。
|
||||
|
||||
总索引生成由管理员显式预览、按预览 snapshotId 条件发布,不随普通 GET/下载自动写对象存储。旧已完成 part 目录不能原地补发 v3,历史迁移仍需另行规划新交付目标。已有调用者和在途任务默认沿用旧协议;回退 API/开关不删除新批次,也不能把新格式任务改成旧格式后重用已完成目录。
|
||||
|
||||
## 2026-10-21 修订:分类结果与内部追溯独立
|
||||
|
||||
用户明确交付价值为最终人工确认的分类及标签,而非过程记录。新 v4 主 JSON 只包含最终验收通过的分类、区间及必要源定位;审核批注、人员、轮次、返工和抽检过程不进入主 JSON,也不默认进入客户批次。完整正式追溯保存为内部 JSON,并与固定结果修订关联。返工经过重新验收的分类修正应进入新交付版本,单纯修改批注不能改变分类。
|
||||
|
||||
独立字典 tags 由管理员维护,与 ID/版本/名称一起在确认时冻结;category 与 aliases 不当作 tags。未写动作描述不等于没有动作,也不撤销切片已有分类。历史没有标签快照时不反查最新字典拼造历史。全舍弃也属于人工分类结果,因此 v4 仍交付该 part 的分类 JSON;这显式替代 v4 路径的旧 no_deliverables 规则,旧媒体目录和 v1–v3 路径继续保持原行为。
|
||||
|
||||
采用独立 v4 协议与默认关闭的开关,避免同一 v3 schema 悄悄改变字段、枚举或文件集合。保持已约定的 capture/part、slices/taskSplit/segments 层级及不可变批次身份,动作有描述使用约定的 annotated。保留旧版本读取与发布恢复,不原地修补历史完成目录。单独写内部追溯会增加一份内部产物,但避免客户为读取分类被迫接触人员、批注和平台流程字段。
|
||||
|
||||
## 2026-10-21 补充:动作继承及原始数据消费
|
||||
|
||||
用户确认切片与动作是父子分类。动作独立选择词条时使用该快照的标签,否则继承父切片冻结标签;不按描述是否为空决定,不伪造动作词条。v4 显式输出有效 tags 和 action/slice 来源,保留独立无效判断;舍弃仅保留原因分类及定位,不带旧动作或审核批注。缺标签不反查最新词条猜补。
|
||||
|
||||
下游需要仅凭总 JSON 和原始数据定位、切分、按类归组。新 v4 使用任务原始输入的 BOS 选择冻结八类源文件及 bucket/prefix,主文件不再依赖平台 workspace。动作同时明确切片内秒数、原始 part 秒数与原始硬件时间戳;用原始左右 PTS 各自解析硬件区间,避免用左视频秒数误切另一相机。输入快照指纹不是单个 BOS 对象内容版本的保证。实际切割和分类打包仍由下游执行。
|
||||
|
||||
缺少可靠来源、分类标签或时间证据时,选择拒绝新 v4 交付,而非“尽量输出”后让客户承受丢分类或错切的风险。发布同时验证消费契约、来源归属和文件存在。已交付结果不原地改写;历史只按证据补齐并产生新批次。这个要求适用于尚未发布的 v4,不改变 v1–v3;旧本地 v4 候选须重新生成。实现和本地消费验证见 `docs/delivery-classification-v4.zh-CN.md`。
|
||||
|
||||
47
docs/delivery-classification-v4.zh-CN.md
Normal file
47
docs/delivery-classification-v4.zh-CN.md
Normal file
@ -0,0 +1,47 @@
|
||||
# 最终分类总 JSON v4
|
||||
|
||||
本地实现对应 [ADR-0005 的 2026-10-21 修订](adr/0005-part-index-delivery-shape.md)。历史 v1/v2/v3 和已交付文件保持原状。开关默认关闭;本文不授权启用工作流或发布。
|
||||
|
||||
`DELIVERY_JSON_V4=true` 要求 `PART_INDEX_DELIVERY=true`,与 `DELIVERY_JSON_V3` 互斥;源索引修订另需 `SOURCE_INDEX_DELIVERY=true`。新工作流须有内部 `deliveryAudit` JSON 输出,以及正常的 `deliveryDirectory` 输出。配套 Worker 负责固定审核追溯,大于 inline 限额的验收摘要走现有内部 artifact 存储,打包节点读取固定引用;必须先验证内部存储可用。旧 Worker/旧算子不能启用新协议。
|
||||
|
||||
客户格式保持 `capture`、`partId`、`slices[].taskSplit`、`slices[].segments[]` 和 `discardedSlices`。按马迪约定的采集目录/part 组织文件:每个 part 一个总分类 JSON;批次索引 `delivery_index.json` 和采集索引 `annotation_manifest.json` 只负责目录定位、文件校验及汇总,不额外携带审核流水。正式批次目录不可变。
|
||||
|
||||
- `sliceName` 是已确认的基础分类名称;`classification.dictionary` 是可获得的字典快照。动作描述为空也保留这个分类,不代表无动作。
|
||||
- 字典快照固定 `id/kind/version/code/name/tags`。`tags` 是独立分类标签,与 `category`、别名和描述原义不混用。新选择由服务端从项目字典校验,客户不能伪造标签;历史缺失快照不按当前词条补填。
|
||||
- 每个动作显式导出有效 `tags` 和 `tagSource`。独立选择动作词条用该快照标签,来源为 `action`;没有独立选择则继承本切片冻结标签,来源为 `slice`,不伪造动作 `dictionary`。自由填写文字也沿用这个判定;有文字不代表选择了独立词条,没有文字不代表没有动作。独立词条缺 tags 时拒绝交付,不偷偷退回父标签。
|
||||
- 动作 `contentKind` 为 `annotated`(有描述)、`invalid_marked`(明确无效)、`gap_placeholder`(保留的留空区间)。`descriptionStatus` 描述文字覆盖情况,不能用于推断整个切片是否有动作。
|
||||
- `isInvalid` 与标签独立;无效动作可保留继承标签,但仍必须按无效判断排除有效动作处理。历史动作未覆盖的留空范围保留为继承父标签的 `gap_placeholder`,不能把该段视频漏掉。
|
||||
- `discardedSlices[].classification.discardEvidence` 保留业务舍弃原因、原因词条/版本/标签和补充说明。全舍弃 part 也输出总分类 JSON。审核退回原因不放在这里。
|
||||
- 舍弃原因有冻结字典时,旧兼容字段 `reasonLabelId/reasonVersion/reasonName` 从同一快照补齐,不导出空名称或版本 0;已有非空值若与快照冲突即拒绝交付。手填原因仍按原有规则保留,不猜造字典选择。
|
||||
- `originalSource` 冻结 BOS bucket、原采集 prefix、partPrefix、输入快照指纹和所选原始文件路径。来自任务创建时的输入引用,不查询当前目录猜原文件,不输出连接 ID、凭据或签名 URL。八类文件为左右视频、左右 PTS、IMU、相机标定、IMU 标定和采集 meta。`sources`、`sourceMetadata.path` 与标定路径统一相对原采集目录,`pathBases` 显式写为 `original_capture`;原 workspace 路径只留内部追溯。
|
||||
|
||||
内部 `droboticflow.delivery.review-history/v2` JSON 按 capture/part/revision 关联,以 `partSha256` 固定原生主文件字节;包含最终验收、抽检、审核人/批注/轮次/确认项、已有正式退回基线、处理记录和阶段事件。该 artifact 留在内部 Run 存储,不进入最终交付列表、不可变客户批次或客户下载。分类选择的服务端快照另外保存在当前任务的恢复记录中,使字典改名、更新或停用后的撤销仍可恢复原版本,客户端不能伪造历史标签。旧记录没有的追溯不制造补写。追溯超出限制时拒绝交付,不静默截断。
|
||||
|
||||
最终分类取通过验收的固定物化修订;沿用既有“未抽中”验收策略的任务,内部保留明确豁免,不能伪造人工通过日期。返工正式修改分类后必须重新送检、验收;查看或处理意见本身不修改分类。相同名称而字典版本/标签不同的切片不能自动合并成一个分类。
|
||||
|
||||
时间按两级明确表达,全部区间采用左闭右开:
|
||||
|
||||
| 字段 | 坐标 |
|
||||
| --- | --- |
|
||||
| `taskSplit.startSecond/endSecond` | 原始 part 左相机视频秒数 |
|
||||
| `segments[].startSecond/endSecond`、`clipStartMs/clipEndMs` | 本切片左相机内秒数/毫秒数 |
|
||||
| `segments[].partStartSecond/partEndSecond` | 原始 part 左相机视频秒数 |
|
||||
| `startTimestampUs/endTimestampUs` | 原始硬件时间戳,非墙钟时间、非从秒数估算 |
|
||||
|
||||
`timeBases` 描述以上坐标及帧区间基准。下游从原始左右 PTS 分别定位同一硬件时间窗,不能把左相机秒数直接当右相机秒数。原始输入快照指纹标识当时选择,不冒充每个 BOS 对象的版本号或内容 SHA;原始数据应保留,当前发布时仅以 HEAD 校验所引用文件存在,不证明同名文件从未被外部覆盖。
|
||||
|
||||
新 v4 要求完整的冻结分类、非空标签、原始来源和可靠时间边界;任一缺失或动作重叠、时间戳不连续即停止交付。批次发布再核对原始 bucket/part 归属、文件存在性和消费契约,不能只凭主文件的 SHA 放行。元数据修订复用原始引用,不追随后来改变的采集记录。未发布的旧本地 v4 样例须重新生成;正式 v1/v2/v3 及其恢复路径保持兼容。历史缺证据应先核实补齐,已交付文件不覆盖,修正交付另建不可变批次。
|
||||
|
||||
平台只输出分类索引,不按标签执行 BOS 数据切割、搬运或集中打包。内部审核文件不在客户目录内,也不从主 JSON 引用。
|
||||
|
||||
本地可用 `python -m tests.backend.export_delivery_v4_fixture <全新输出目录>` 生成明确标记的合成样例。目录中 `partNNNN/part.json` 是客户主文件;`internal/partNNNN/review-history.json` 是分离的内部示例,不属于客户包。不要把整个验证输出目录直接交付给客户。
|
||||
|
||||
独立消费验证:设置 `FANYU_DELIVERY_CONSUMER_FIXTURE_DIR` 为全新绝对目录,先运行 Go `TestDeliveryV4ConsumerReplicaFixture`,再以同一环境变量运行 `tests/backend/test_delivery_v4_consumer.py`。合成真实双目视频分别经过物理切片、源索引和元数据复用;独立进程仅读取导出 JSON 与原始文件,核对每帧覆盖、非零 part 起点、标签继承、无效与舍弃。独立读取器 `tests/backend/consume_delivery_v4.py` 只生成处理计划,不执行切割或写源数据。
|
||||
|
||||
## 2026-10-22 组合候选与默认关闭阶段
|
||||
|
||||
用户确认先与前端分层版本合并验证,`DELIVERY_JSON_V4` 继续保持关闭,启用另行制定计划。新代码的标签编辑、字典快照保存和内部物化元数据并非全部受该打包参数控制;关闭参数只保证不主动选择 v4 交付路径,不代表 API 行为与旧版本逐字节一致。
|
||||
|
||||
整采集的多 part 输入可能无法证明单 part 的原始来源。普通物理切片、源索引及元数据复用会省略不可靠的新增 `originalSource` 并保留诊断,不因这项可选证据阻塞旧任务;跨 Run/组织错误与存储失败仍拒绝,已有冻结来源不追随新目录。真正生成或发布 v4 时,缺原始来源、标签或时间证据仍被打包/批次门禁拒绝,不能据此输出不完整 JSON。
|
||||
|
||||
回退旧 API 不是无损写入保证:新版本保存分类后,旧版本重建草稿数组可能丢弃字典快照。回退前应暂停相关写操作或准备能保留新字段的兼容回退版本;只保留恢复历史不等于能自动恢复所有后续修改。已交付不可变文件不因此原地迁移或覆盖。
|
||||
@ -31,6 +31,7 @@
|
||||
["npm", "run", "test:browser:annotation-pools"],
|
||||
["npm", "run", "test:browser:slice-input"],
|
||||
["npm", "run", "test:browser:annotation-persistence"],
|
||||
["go", "test", "./internal/api", "-run", "TestDurableAnnotationOutbox|TestSliceRecovery|TestReviewCannotMutateAnnotation|TestReviewDecision|Test.*ReviewDraft|TestReviewConfirmations|TestRouteSnapshot", "-count=1"]
|
||||
["go", "test", "./internal/deliveryindex", "-count=1"],
|
||||
["go", "test", "./internal/api", "-run", "TestDurableAnnotationOutbox|TestSliceRecovery|TestReviewCannotMutateAnnotation|TestReviewDecision|Test.*ReviewDraft|TestReviewConfirmations|TestRouteSnapshot|Test.*Dictionary|TestProjectSliceNames|TestLegacyMaterializationWithoutPartScopedOriginalSelection|TestOptionalOriginalSource|TestAnnotationOriginalSource|TestClassificationBatch", "-count=1"]
|
||||
]
|
||||
}
|
||||
|
||||
31
docs/verification/combined-v4-20261022.md
Normal file
31
docs/verification/combined-v4-20261022.md
Normal file
@ -0,0 +1,31 @@
|
||||
# 前端分层与交付 v4 组合候选
|
||||
|
||||
用户选择:先整合并验证,v4 保持关闭;启用另行确认。新个人分支 `codex/dashboard-delivery-v4-20261022` 从 `2149b862` 合并 v4 分支 `e66c7726`,原两个分支保留。
|
||||
|
||||
三处文本冲突已处理:切片字典接口保留 v4 的服务端冻结/历史恢复语义,工作台和标注页保留分层协调与字典选择两侧导入。不能以文本合并成功代替业务验证。
|
||||
|
||||
## 已核对的发布范围
|
||||
|
||||
2026-10-22 的只读 Deployment 快照显示 Fanyu / droboticflow-fanyu 的 API 与处理器均为 `3221ac65`,Worker 为 `963b530f`;它们均为组合候选的祖先。此事实只说明源码历史没有遗漏,不证明所有语义兼容,也不能代替发布前新快照。
|
||||
|
||||
本次只准备候选;没有上传镜像、部署、迁移、启用工作流或改写客户交付。后续若发布默认关闭版本,计划限 API 与处理器,Worker、独立打包算子和工作流继续使用现有版本。v4 需要配套升级 Worker/算子、验证内部审计存储及配置 deliveryAudit 输出,不能只切一个开关。
|
||||
|
||||
候选未新增 Store/cmd 迁移或索引修改。内置算子目录新增默认关闭参数和审计输出,目录查询沿现有代码可能更新内置算子定义;这不是现有工作流自动升级,也不能说镜像更新绝不触发数据库元数据写入。发布前仍须审查实际配置、在途任务和数据影响。
|
||||
|
||||
## 修复的组合兼容问题
|
||||
|
||||
合成多 part 输入引用实测使物理切片、源索引在 v4 关闭时也因新增来源证据检查失败。现将不可信的可选来源证据省略并记录诊断,普通媒体处理继续;跨任务/组织及存储错误仍拒绝。真实 v4 的缺来源、文件归属、时间和标签门禁继续有效。已有冻结来源不被后来目录变化改写。
|
||||
|
||||
v4 默认关闭不等于旧版本写入安全:新字段开始保存后,旧 API 的数组重建可能丢失字典快照。后续上线必须明确暂停相关写入或使用兼容回退版本,不能承诺直接回退现网镜像后继续编辑无损。
|
||||
|
||||
## 本地证据
|
||||
|
||||
- 类型与分层检查、全量前端基线门禁通过;13 项已有失败,14 项基线未修改。
|
||||
- v4、字典、批次、历史 part-index 的 Go 定向回归通过;新来源兼容回归覆盖物理/源索引两次修订、元数据复用、身份隔离和不伪造来源。
|
||||
- 新合成真实双目媒体经过 Go 物理切片/源索引,再由独立 Python 消费客户 JSON 与原始文件,确认标签继承、留空、无效、舍弃与硬件时间窗;Python 共 43 通过、1 项旧夹具依赖场景跳过,未将跳过计入通过。
|
||||
- 组合版完整标注 19 场景、质检/验收草稿 34 场景和审核权限浏览器回归通过。
|
||||
- 修复前的来源错误、开发态字典夹具依赖优化导致的页面未渲染尝试均保留,不能冒充产品失败或已通过验证。
|
||||
- 字典标签的生产组件构建及浏览器测试通过;共享保存 24 项、发布工具 37 项、生成源码一致性、Go 编译、前端构建与生产启动通过。
|
||||
- 完整算子发布 manifest 检查未通过:annotation_gate 的 sourceTreeSha256 漂移在原始 `3221ac65` 工作树逐项复现,期望值/实际值完全相同,属于已有问题;新 v4 package 源码也尚未绑定新算子镜像发布。不得伪造 release.json 摘要使检查通过。它阻断算子升级/v4 启用,不将本轮限定的 API/处理器默认关闭候选冒充为算子可发布版本。
|
||||
|
||||
证据存于本工作树忽略目录 `local-data/verification/combined-v4-20261022/`。本记录不是部署批准,也不保证回退旧写入路径无损。
|
||||
@ -2,6 +2,7 @@ package api
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
@ -19,7 +20,7 @@ func consolidateInvalidActionSlices(input []domain.SliceSegment, actions []domai
|
||||
eligible := map[string]int{}
|
||||
for i := 1; i < len(ordered)-1; i++ {
|
||||
l, c, r := ordered[i-1], ordered[i], ordered[i+1]
|
||||
if l.IsDiscarded() || c.IsDiscarded() || r.IsDiscarded() || strings.TrimSpace(c.Name) == "" || l.Name != c.Name || r.Name != c.Name || l.End != c.Start || c.End != r.Start {
|
||||
if l.IsDiscarded() || c.IsDiscarded() || r.IsDiscarded() || strings.TrimSpace(c.Name) == "" || l.Name != c.Name || r.Name != c.Name || !reflect.DeepEqual(l.Dictionary, c.Dictionary) || !reflect.DeepEqual(r.Dictionary, c.Dictionary) || l.End != c.Start || c.End != r.Start {
|
||||
continue
|
||||
}
|
||||
cursor, has, valid := c.Start, false, true
|
||||
|
||||
@ -14,6 +14,7 @@ import (
|
||||
)
|
||||
|
||||
type annotationLabelInput struct {
|
||||
Tags []string `json:"tags"`
|
||||
RequiresNote *bool `json:"requiresNote"`
|
||||
Kind string `json:"kind"`
|
||||
Code string `json:"code"`
|
||||
@ -46,7 +47,7 @@ func annotationLabelRevision(item domain.AnnotationLabelRecord) domain.Annotatio
|
||||
return domain.AnnotationLabelRevision{
|
||||
RequiresNote: item.RequiresNote, Version: item.Version, Code: item.Code, Name: item.Name, Category: item.Category,
|
||||
Description: item.Description, Example: item.Example, Color: item.Color,
|
||||
Aliases: append([]string(nil), item.Aliases...), Status: item.Status,
|
||||
Tags: append([]string(nil), item.Tags...), Aliases: append([]string(nil), item.Aliases...), Status: item.Status,
|
||||
UpdatedByUserID: item.UpdatedByUserID, UpdatedAt: item.UpdatedAt,
|
||||
}
|
||||
}
|
||||
@ -79,6 +80,19 @@ func normalizeAnnotationLabelInput(input annotationLabelInput) annotationLabelIn
|
||||
seen[value] = struct{}{}
|
||||
aliases = append(aliases, value)
|
||||
}
|
||||
|
||||
if input.Tags != nil {
|
||||
tags := []string{}
|
||||
seenTags := map[string]bool{}
|
||||
for _, tag := range input.Tags {
|
||||
tag = strings.TrimSpace(tag)
|
||||
if tag != "" && !seenTags[tag] {
|
||||
tags = append(tags, tag)
|
||||
seenTags[tag] = true
|
||||
}
|
||||
}
|
||||
input.Tags = tags
|
||||
}
|
||||
input.Aliases = aliases
|
||||
return input
|
||||
}
|
||||
@ -97,6 +111,15 @@ func validateAnnotationLabelInput(input annotationLabelInput) []validationIssue
|
||||
if input.Status != domain.AnnotationLabelActive && input.Status != domain.AnnotationLabelInactive {
|
||||
issues = append(issues, validationIssue{Field: "status", Message: "标签状态必须是 active 或 inactive"})
|
||||
}
|
||||
if len(input.Tags) > 32 {
|
||||
issues = append(issues, validationIssue{Field: "tags", Message: "每个词条最多 32 个标签"})
|
||||
}
|
||||
for _, tag := range input.Tags {
|
||||
if len([]rune(tag)) > 80 {
|
||||
issues = append(issues, validationIssue{Field: "tags", Message: "标签最多 80 字"})
|
||||
break
|
||||
}
|
||||
}
|
||||
return issues
|
||||
}
|
||||
|
||||
@ -165,7 +188,7 @@ func (server *server) handleAnnotationLabelCreate(writer http.ResponseWriter, re
|
||||
ID: domain.NewID("annotation-label"), OrganizationID: organizationID,
|
||||
Code: payload.Code, Name: payload.Name, Category: payload.Category,
|
||||
Description: payload.Description, Example: payload.Example, Color: payload.Color,
|
||||
Aliases: payload.Aliases, Status: payload.Status, Version: 1,
|
||||
Tags: payload.Tags, Aliases: payload.Aliases, Status: payload.Status, Version: 1,
|
||||
CreatedByUserID: actorID, UpdatedByUserID: actorID, CreatedAt: now, UpdatedAt: now,
|
||||
}
|
||||
item.History = []domain.AnnotationLabelRevision{annotationLabelRevision(item)}
|
||||
@ -225,6 +248,9 @@ func (server *server) handleAnnotationLabelPatch(writer http.ResponseWriter, req
|
||||
item.Code, item.Name, item.Category = payload.Code, payload.Name, payload.Category
|
||||
item.Description, item.Example, item.Color = payload.Description, payload.Example, payload.Color
|
||||
item.Aliases, item.Status = payload.Aliases, payload.Status
|
||||
if payload.Tags != nil {
|
||||
item.Tags = payload.Tags
|
||||
}
|
||||
item.Version++
|
||||
item.UpdatedByUserID, item.UpdatedAt = principal.User.ID, nowISO()
|
||||
item.History = append(item.History, annotationLabelRevision(*item))
|
||||
@ -267,13 +293,16 @@ func (server *server) handleProjectAnnotationLabelsGet(writer http.ResponseWrite
|
||||
if selection == nil {
|
||||
selection = &domain.ProjectAnnotationLabelsRecord{ID: "project-annotation-labels-" + project.ID, OrganizationID: project.OrganizationID, ProjectID: project.ID, LabelIDs: []string{}}
|
||||
}
|
||||
if projectDictionaryKind(request) != "action" {
|
||||
{
|
||||
available, e := server.store.ListAnnotationLabels(request.Context(), project.OrganizationID)
|
||||
if e != nil {
|
||||
renderError(writer, 500, e)
|
||||
return
|
||||
}
|
||||
ids := selection.SliceLabelIDs
|
||||
ids := selection.LabelIDs
|
||||
if projectDictionaryKind(request) == "slice" {
|
||||
ids = selection.SliceLabelIDs
|
||||
}
|
||||
if projectDictionaryKind(request) == "discard_reason" {
|
||||
ids = selection.DiscardReasonLabelIDs
|
||||
}
|
||||
@ -290,14 +319,20 @@ func (server *server) handleProjectAnnotationLabelsGet(writer http.ResponseWrite
|
||||
items = append(items, label)
|
||||
}
|
||||
}
|
||||
response := map[string]any{"projectId": project.ID, "labelIds": ids, "items": items}
|
||||
response := struct {
|
||||
domain.ProjectAnnotationLabelsRecord
|
||||
Items []domain.AnnotationLabelRecord `json:"items"`
|
||||
AllowManual *bool `json:"allowManual,omitempty"`
|
||||
}{ProjectAnnotationLabelsRecord: *selection, Items: items}
|
||||
response.LabelIDs = ids
|
||||
if projectDictionaryKind(request) == "discard_reason" {
|
||||
response["allowManual"] = selection.OrganizationID == project.OrganizationID && len(ids) == 0
|
||||
manual := selection.OrganizationID == project.OrganizationID && len(ids) == 0
|
||||
response.AllowManual = &manual
|
||||
}
|
||||
|
||||
renderJSON(writer, http.StatusOK, response)
|
||||
return
|
||||
}
|
||||
renderJSON(writer, http.StatusOK, selection)
|
||||
}
|
||||
|
||||
func (server *server) handleProjectAnnotationLabelsPut(writer http.ResponseWriter, request *http.Request) {
|
||||
@ -399,7 +434,7 @@ func (server *server) projectAnnotationLabelDefinitions(ctx context.Context, org
|
||||
if !exists || item.Status != domain.AnnotationLabelActive || dictionaryKind(item.Kind) != "action" {
|
||||
continue
|
||||
}
|
||||
labels = append(labels, domain.SemanticLabelDefinition{ID: item.Code, Name: item.Name, Description: item.Description, Aliases: item.Aliases, Color: item.Color})
|
||||
labels = append(labels, domain.SemanticLabelDefinition{Dictionary: item.Selection(), ID: item.Code, Name: item.Name, Description: item.Description, Aliases: item.Aliases, Color: item.Color})
|
||||
}
|
||||
return labels, nil
|
||||
}
|
||||
|
||||
@ -72,6 +72,15 @@ func (s *server) tryAnnotationMetadataRevision(ctx context.Context, task domain.
|
||||
if e != nil {
|
||||
return nil, nil
|
||||
}
|
||||
if manifest["originalSource"] == nil {
|
||||
if record, err := s.store.GetCaptureRecordByRunID(ctx, task.RunID); err != nil {
|
||||
return nil, err
|
||||
} else if original, err := annotationRevisionOriginalSource(run, record); err != nil {
|
||||
return nil, err
|
||||
} else if original != nil {
|
||||
manifest["originalSource"] = original
|
||||
}
|
||||
}
|
||||
entries := listValue(manifest["items"])
|
||||
if len(entries) != len(slices) {
|
||||
return nil, nil
|
||||
@ -134,6 +143,7 @@ func (s *server) tryAnnotationMetadataRevision(ctx context.Context, task domain.
|
||||
}
|
||||
entry["sourceSliceId"], entry["sliceName"], entry["deliveryDisposition"] = slice.ID, slice.Name, sliceDisposition(*slice)
|
||||
synchronized["sliceName"], synchronized["deliveryDisposition"] = slice.Name, sliceDisposition(*slice)
|
||||
entry["dictionary"], synchronized["dictionary"] = slice.Dictionary, slice.Dictionary
|
||||
delete(entry, "discardEvidence")
|
||||
delete(synchronized, "discardEvidence")
|
||||
if slice.IsDiscarded() {
|
||||
@ -146,9 +156,9 @@ func (s *server) tryAnnotationMetadataRevision(ctx context.Context, task domain.
|
||||
if slice.IsDiscarded() || segment.End <= slice.Start || segment.Start >= slice.End {
|
||||
continue
|
||||
}
|
||||
actions = append(actions, map[string]any{"id": segment.ID, "status": effectiveActionStatus(segment), "isInvalid": segment.IsInvalid, "description": segment.Description, "descriptionTemplateId": segment.LabelID, "phrases": segment.Phrases, "source": segment.Source, "confidence": segment.Confidence, "clipStartMs": (segment.Start - slice.Start) * 1000, "clipEndMs": (segment.End - slice.Start) * 1000, "startTimestampUs": segment.StartTimestampUS, "endTimestampUs": segment.EndTimestampUS})
|
||||
actions = append(actions, map[string]any{"id": segment.ID, "status": effectiveActionStatus(segment), "isInvalid": segment.IsInvalid, "description": segment.Description, "descriptionTemplateId": segment.LabelID, "dictionary": segment.Dictionary, "phrases": segment.Phrases, "source": segment.Source, "confidence": segment.Confidence, "clipStartMs": (segment.Start - slice.Start) * 1000, "clipEndMs": (segment.End - slice.Start) * 1000, "startTimestampUs": segment.StartTimestampUS, "endTimestampUs": segment.EndTimestampUS})
|
||||
}
|
||||
annotation := map[string]any{"schemaVersion": "droboticflow.timestamped-description/v3", "boundaryContract": "hardware-frame-v1", "sliceId": entry["sliceId"], "sliceName": slice.Name, "window": map[string]any{"startTimestampUs": slice.StartTimestampUS, "endTimestampUs": slice.EndTimestampUS, "semantics": "half-open"}, "descriptions": actions, "actions": actions}
|
||||
annotation := map[string]any{"schemaVersion": "droboticflow.timestamped-description/v3", "boundaryContract": "hardware-frame-v1", "sliceId": entry["sliceId"], "sliceName": slice.Name, "dictionary": slice.Dictionary, "window": map[string]any{"startTimestampUs": slice.StartTimestampUS, "endTimestampUs": slice.EndTimestampUS, "semantics": "half-open"}, "descriptions": actions, "actions": actions}
|
||||
if e = os.MkdirAll(filepath.Join(stage, oldDir), 0755); e != nil {
|
||||
return nil, e
|
||||
}
|
||||
|
||||
@ -1831,6 +1831,7 @@ func (server *server) handleCombinedAnnotationDraftPut(writer http.ResponseWrite
|
||||
return
|
||||
}
|
||||
template = annotationTemplateWithTaskSnapshot(template, task)
|
||||
issues = append(issues, server.canonicalActionDictionaries(request.Context(), humanItem, payload.Segments)...)
|
||||
issues = append(issues, validateDraftFanyuV5CombinedSemanticSegments(payload.Segments, payload.SliceSegments, template)...)
|
||||
if len(issues) > 0 {
|
||||
renderValidationError(writer, issues...)
|
||||
@ -1977,6 +1978,7 @@ func (server *server) handleAnnotationConfirmationCreate(writer http.ResponseWri
|
||||
return
|
||||
}
|
||||
template = annotationTemplateWithTaskSnapshot(template, task)
|
||||
issues = append(issues, server.canonicalActionDictionaries(request.Context(), item, payload.Segments)...)
|
||||
issues = append(issues, validateDraftFanyuV5CombinedSemanticSegments(payload.Segments, slices, template)...)
|
||||
if len(issues) > 0 {
|
||||
renderValidationError(writer, issues...)
|
||||
@ -2496,6 +2498,10 @@ func (server *server) handleCombinedAnnotationSubmission(writer http.ResponseWri
|
||||
return
|
||||
}
|
||||
template = annotationTemplateWithTaskSnapshot(template, task)
|
||||
if issues := server.canonicalActionDictionaries(request.Context(), humanItem, payload.Segments); len(issues) > 0 {
|
||||
renderValidationError(writer, issues...)
|
||||
return
|
||||
}
|
||||
if issues := validateFanyuV5CombinedSemanticSegmentsWithRequireDesc(payload.Segments, payload.SliceSegments, template, !continuousActions); len(issues) > 0 {
|
||||
renderValidationError(writer, issues...)
|
||||
return
|
||||
@ -3104,13 +3110,18 @@ func (server *server) handleAnnotationRevisionCreate(writer http.ResponseWriter,
|
||||
renderJSON(writer, http.StatusForbidden, map[string]string{"error": "annotator assignment is required"})
|
||||
return
|
||||
}
|
||||
if _, claimed := server.requireLegacyHumanItemClaim(writer, request, *task, "semantic_annotation"); !claimed {
|
||||
humanItem, claimed := server.requireLegacyHumanItemClaim(writer, request, *task, "semantic_annotation")
|
||||
if !claimed {
|
||||
return
|
||||
}
|
||||
var payload annotationRevisionInput
|
||||
if !decodeJSON(writer, request, &payload) {
|
||||
return
|
||||
}
|
||||
if issues := server.canonicalActionDictionaries(request.Context(), humanItem, payload.Segments); len(issues) > 0 {
|
||||
renderValidationError(writer, issues...)
|
||||
return
|
||||
}
|
||||
if issues := validateAnnotationSegments(payload.Segments, *task); len(issues) > 0 {
|
||||
renderValidationError(writer, issues...)
|
||||
return
|
||||
|
||||
79
internal/api/annotation_original_source.go
Normal file
79
internal/api/annotation_original_source.go
Normal file
@ -0,0 +1,79 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
)
|
||||
|
||||
// Freeze only credential-free facts from the run's original input, never the
|
||||
// current workflow configuration or a later source-directory listing.
|
||||
func annotationOriginalSource(run *domain.RunRecord, capture *domain.CaptureRecord) (map[string]any, error) {
|
||||
if run == nil || capture == nil || len(run.Inputs) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
if capture.RunID != run.ID || capture.OrganizationID != run.OrganizationID {
|
||||
return nil, fmt.Errorf("original capture scope mismatch")
|
||||
}
|
||||
root := capture.SourcePrefix
|
||||
if root == "" {
|
||||
root = capture.Prefix
|
||||
}
|
||||
var result map[string]any
|
||||
for _, input := range run.Inputs {
|
||||
ref := input.Ref
|
||||
if ref == nil || ref.Provider != domain.StorageProviderBOS || ref.Kind != domain.ArtifactRefKindDirectory || ref.Bucket != capture.Bucket || ref.Prefix != root {
|
||||
continue
|
||||
}
|
||||
if ref.CaptureUnit != "" && ref.CaptureUnit != capture.CaptureID {
|
||||
continue
|
||||
}
|
||||
if len(ref.ObjectKeys) == 0 {
|
||||
continue
|
||||
} // Older non-selected directory inputs need explicit recovery.
|
||||
if root == "" || path.Clean(root) != root || strings.HasPrefix(root, "/") || strings.Contains(root, "\\") {
|
||||
return nil, fmt.Errorf("unsafe original capture prefix")
|
||||
}
|
||||
keys := append([]string(nil), ref.ObjectKeys...)
|
||||
sort.Strings(keys)
|
||||
for i, key := range keys {
|
||||
if !strings.HasPrefix(key, root+"/") || path.Clean(key) != key || strings.Contains(key, "\\") || i > 0 && key == keys[i-1] {
|
||||
return nil, fmt.Errorf("unsafe original capture object selection")
|
||||
}
|
||||
relative := strings.TrimPrefix(key, root+"/")
|
||||
if strings.Contains(relative, "/") && !strings.HasPrefix(key, capture.Prefix+"/") {
|
||||
return nil, fmt.Errorf("original capture selection includes another part")
|
||||
}
|
||||
}
|
||||
candidate := map[string]any{"provider": string(ref.Provider), "bucket": ref.Bucket, "prefix": root, "partPrefix": capture.Prefix, "captureUnit": capture.CaptureID, "objectKeys": keys, "snapshotFingerprint": ref.VersionID}
|
||||
if result != nil {
|
||||
a, _ := json.Marshal(result)
|
||||
b, _ := json.Marshal(candidate)
|
||||
if string(a) != string(b) {
|
||||
return nil, fmt.Errorf("ambiguous original capture inputs")
|
||||
}
|
||||
}
|
||||
result = candidate
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// Source provenance is optional for legacy materialization, but mandatory at
|
||||
// v4 export. Never invent a part selection from an ambiguous capture input.
|
||||
// Tenant/run violations are still fatal; storage failures remain with callers.
|
||||
func annotationRevisionOriginalSource(run *domain.RunRecord, capture *domain.CaptureRecord) (map[string]any, error) {
|
||||
source, err := annotationOriginalSource(run, capture)
|
||||
if err == nil {
|
||||
return source, nil
|
||||
}
|
||||
if run != nil && capture != nil && (capture.RunID != run.ID || capture.OrganizationID != run.OrganizationID) {
|
||||
return nil, err
|
||||
}
|
||||
log.Printf("[api] original_source_evidence_unavailable reason=%s", err)
|
||||
return nil, nil
|
||||
}
|
||||
238
internal/api/annotation_original_source_test.go
Normal file
238
internal/api/annotation_original_source_test.go
Normal file
@ -0,0 +1,238 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
)
|
||||
|
||||
func consumerSourceRef() *domain.ArtifactReference {
|
||||
root := "incoming/capture-A"
|
||||
keys := []string{root + "/meta_capture-A.json", root + "/capture-A_calibration_camera.yaml", root + "/capture-A_calibration_imu.yaml", root + "/part0001/capture-A_imu_part0001.csv"}
|
||||
for _, side := range []string{"left", "right"} {
|
||||
keys = append(keys, root+"/part0001/capture-A_camera_"+side+"_part0001.mp4", root+"/part0001/capture-A_camera_"+side+"_part0001_pts.csv")
|
||||
}
|
||||
return &domain.ArtifactReference{Provider: domain.StorageProviderBOS, Kind: domain.ArtifactRefKindDirectory, ConnectionID: "PRIVATE_CONNECTION", Bucket: "synthetic-original", Prefix: root, CaptureUnit: "capture-A/part0001", ObjectKeys: keys, VersionID: "frozen-input-version"}
|
||||
}
|
||||
|
||||
func TestAnnotationOriginalSourceUsesFrozenInputSelection(t *testing.T) {
|
||||
ref := consumerSourceRef()
|
||||
run := &domain.RunRecord{ID: "run", OrganizationID: "org", Inputs: map[string]domain.ArtifactPayload{"raw": {Ref: ref}}}
|
||||
capture := &domain.CaptureRecord{RunID: "run", OrganizationID: "org", Bucket: ref.Bucket, Prefix: ref.Prefix + "/part0001", SourcePrefix: ref.Prefix, CaptureID: ref.CaptureUnit, SourceKeys: []string{"later-current-directory-must-not-be-used"}}
|
||||
source, err := annotationOriginalSource(run, capture)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(source["objectKeys"].([]string)) != 8 {
|
||||
t.Fatal("frozen source selection lost")
|
||||
}
|
||||
if _, ok := source["connectionId"]; ok {
|
||||
t.Fatal("private connection leaked")
|
||||
}
|
||||
ref.ObjectKeys = append(ref.ObjectKeys, ref.Prefix+"/part0002/foreign.mp4")
|
||||
if _, err = annotationOriginalSource(run, capture); err == nil {
|
||||
t.Fatal("foreign part accepted")
|
||||
}
|
||||
ref.ObjectKeys = ref.ObjectKeys[:len(ref.ObjectKeys)-1]
|
||||
capture.OrganizationID = "other"
|
||||
if _, err = annotationOriginalSource(run, capture); err == nil {
|
||||
t.Fatal("foreign organization accepted")
|
||||
}
|
||||
}
|
||||
|
||||
// Retain real synthetic source data for the independent Python consumer test.
|
||||
func TestDeliveryV4ConsumerReplicaFixture(t *testing.T) {
|
||||
parent := os.Getenv("FANYU_DELIVERY_CONSUMER_FIXTURE_DIR")
|
||||
if parent == "" {
|
||||
t.Skip("opt-in retained synthetic fixture")
|
||||
}
|
||||
if !filepath.IsAbs(parent) {
|
||||
t.Fatal("absolute local output required")
|
||||
}
|
||||
for _, sourceIndex := range []bool{false, true} {
|
||||
mode := "physical"
|
||||
if sourceIndex {
|
||||
mode = "source-index"
|
||||
}
|
||||
t.Run(mode, func(t *testing.T) {
|
||||
f, temp := annotationProcessorFixture(t)
|
||||
dir := filepath.Join(parent, mode)
|
||||
root := filepath.Join(dir, "workspace")
|
||||
if err := os.MkdirAll(dir, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := copyAnnotationTree(f.ctx, temp, root); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
run, _ := f.store.GetRun(f.ctx, f.task.RunID)
|
||||
for i := range run.NodeExecutions {
|
||||
for j := range run.NodeExecutions[i].InputBindings {
|
||||
for k := range run.NodeExecutions[i].InputBindings[j].Artifacts {
|
||||
a := &run.NodeExecutions[i].InputBindings[j].Artifacts[k]
|
||||
a.Path = filepath.Join(root, filepath.Base(a.Path))
|
||||
}
|
||||
}
|
||||
}
|
||||
ref := consumerSourceRef()
|
||||
run.Inputs = map[string]domain.ArtifactPayload{"raw": {Ref: ref}}
|
||||
if err := f.store.UpsertRun(f.ctx, *run); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
capture := domain.CaptureRecord{ID: "consumer-capture", RunID: run.ID, OrganizationID: run.OrganizationID, Bucket: ref.Bucket, Prefix: ref.Prefix + "/part0001", SourcePrefix: ref.Prefix, CaptureID: ref.CaptureUnit}
|
||||
if err := f.store.UpsertCaptureRecord(f.ctx, capture); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
base := &domain.DictionarySelection{ID: "base", Kind: "slice", Version: 2, Code: "operate", Name: "物品操作", Tags: []string{"商品", "手部"}}
|
||||
reason := &domain.DictionarySelection{ID: "discard", Kind: "discard_reason", Version: 1, Code: "occluded", Name: "遮挡", Tags: []string{"不可用"}}
|
||||
action := &domain.DictionarySelection{ID: "pick", Kind: "action", Version: 3, Code: "pick", Name: "拿取", Tags: []string{"抓取"}}
|
||||
input := combinedAnnotationSubmissionInput{SliceSegments: []domain.SliceSegment{{ID: "first", Name: base.Name, Dictionary: base, Start: 0, End: 1}, {ID: "discard", Start: 1, End: 3, DeliveryDisposition: "discard", DiscardReason: "遮挡", DiscardReasonDictionary: reason}, {ID: "last", Name: base.Name, Dictionary: base, Start: 3, End: 4}}, Segments: []domain.AnnotationSegment{{ID: "blank", Start: 0, End: .5}, {ID: "pick", Start: .5, End: 1, Description: "拿起杯子", Dictionary: action}, {ID: "invalid", Start: 3, End: 3.5, IsInvalid: true}, {ID: "free-text", Start: 3.5, End: 4, Description: "人工补充描述"}}}
|
||||
s := &server{store: f.store, workspaceMountPath: root, annotationSourceIndex: sourceIndex}
|
||||
result, err := s.materializeCombinedSliceSubmission(f.ctx, f.task, f.item, input, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !sourceIndex {
|
||||
before, err := os.ReadFile(filepath.Join(result.Artifact.Path, "revision-manifest.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var first map[string]any
|
||||
if err = json.Unmarshal(before, &first); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Metadata-only reuse must keep the original source, even if discovery changes.
|
||||
capture.Bucket = "later-bucket"
|
||||
if err = f.store.UpsertCaptureRecord(f.ctx, capture); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
run, _ = f.store.GetRun(f.ctx, f.task.RunID)
|
||||
result, err = s.materializeCombinedSliceSubmission(f.ctx, f.task, f.item, input, 2)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if result.Payload["metadataOnly"] != true {
|
||||
t.Fatal("metadata-only recovery not exercised")
|
||||
}
|
||||
after, err := os.ReadFile(filepath.Join(result.Artifact.Path, "revision-manifest.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var second map[string]any
|
||||
if err = json.Unmarshal(after, &second); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reflect.DeepEqual(first["originalSource"], second["originalSource"]) {
|
||||
t.Fatal("metadata revision followed mutable source")
|
||||
}
|
||||
}
|
||||
original := filepath.Join(dir, "original")
|
||||
for _, key := range ref.ObjectKeys {
|
||||
relative := key[len(ref.Prefix)+1:]
|
||||
dst := filepath.Join(original, filepath.FromSlash(relative))
|
||||
if err = os.MkdirAll(filepath.Dir(dst), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var body []byte
|
||||
switch {
|
||||
case filepath.Ext(dst) == ".yaml":
|
||||
body = []byte("synthetic: true\n")
|
||||
case filepath.Ext(dst) == ".json":
|
||||
body = []byte("{\"synthetic\":true,\"partId\":\"part0001\"}\n")
|
||||
default:
|
||||
name := "imu.csv"
|
||||
for _, side := range []string{"left", "right"} {
|
||||
if strings.Contains(relative, "camera_"+side) {
|
||||
name = side + ".mp4"
|
||||
if strings.HasSuffix(relative, "_pts.csv") {
|
||||
name = side + "_pts.csv"
|
||||
}
|
||||
}
|
||||
}
|
||||
body, err = os.ReadFile(filepath.Join(root, name))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err = os.WriteFile(dst, body, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
body, _ := json.MarshalIndent(map[string]any{"artifact": result.Artifact, "original": original, "workspace": root}, "", " ")
|
||||
if err = os.WriteFile(filepath.Join(dir, "context.json"), body, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Legacy capture inputs can select an entire multi-part directory. They are
|
||||
// insufficient evidence for v4, but must not block existing annotation work.
|
||||
func TestLegacyMaterializationWithoutPartScopedOriginalSelection(t *testing.T) {
|
||||
for _, sourceIndex := range []bool{false, true} {
|
||||
name := "physical"
|
||||
if sourceIndex {
|
||||
name = "source-index"
|
||||
}
|
||||
t.Run(name, func(t *testing.T) {
|
||||
f, root := annotationProcessorFixture(t)
|
||||
run, err := f.store.GetRun(f.ctx, f.task.RunID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ref := consumerSourceRef()
|
||||
ref.ObjectKeys = append(ref.ObjectKeys, ref.Prefix+"/part0002/other.mp4")
|
||||
run.Inputs = map[string]domain.ArtifactPayload{"legacy-capture": {Ref: ref}}
|
||||
if err = f.store.UpsertRun(f.ctx, *run); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
capture := domain.CaptureRecord{ID: "legacy-capture", RunID: run.ID, OrganizationID: run.OrganizationID, Bucket: ref.Bucket, Prefix: ref.Prefix + "/part0001", SourcePrefix: ref.Prefix, CaptureID: ref.CaptureUnit}
|
||||
if err = f.store.UpsertCaptureRecord(f.ctx, capture); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
input := combinedAnnotationSubmissionInput{SliceSegments: []domain.SliceSegment{{ID: "whole", Name: "历史名称", Start: 0, End: 4}}, Segments: []domain.AnnotationSegment{{ID: "old", Start: 0, End: 4, Description: "历史描述"}}}
|
||||
s := &server{store: f.store, workspaceMountPath: root, annotationSourceIndex: sourceIndex}
|
||||
for revision := 1; revision <= 2; revision++ {
|
||||
result, err := s.materializeCombinedSliceSubmission(f.ctx, f.task, f.item, input, revision)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
data, err := os.ReadFile(filepath.Join(result.Artifact.Path, "revision-manifest.json"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var manifest map[string]any
|
||||
if err = json.Unmarshal(data, &manifest); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if manifest["originalSource"] != nil {
|
||||
t.Fatal("unreliable source must not be exported as v4 evidence")
|
||||
}
|
||||
if len(listValue(manifest["items"])) != 1 {
|
||||
t.Fatal("legacy materialization lost its slice")
|
||||
}
|
||||
if !sourceIndex && revision == 2 && result.Payload["metadataOnly"] != true {
|
||||
t.Fatal("metadata-only reuse was not exercised")
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestOptionalOriginalSourceStillRejectsIdentityMismatch(t *testing.T) {
|
||||
ref := consumerSourceRef()
|
||||
run := &domain.RunRecord{ID: "run", OrganizationID: "org", Inputs: map[string]domain.ArtifactPayload{"raw": {Ref: ref}}}
|
||||
capture := &domain.CaptureRecord{RunID: "other-run", OrganizationID: "org"}
|
||||
if _, err := annotationRevisionOriginalSource(run, capture); err == nil {
|
||||
t.Fatal("run mismatch accepted")
|
||||
}
|
||||
capture.RunID = run.ID
|
||||
capture.OrganizationID = "other-org"
|
||||
if _, err := annotationRevisionOriginalSource(run, capture); err == nil {
|
||||
t.Fatal("organization mismatch accepted")
|
||||
}
|
||||
}
|
||||
@ -85,10 +85,10 @@ func (s *server) materializeSourceIndexRevision(ctx context.Context, t domain.An
|
||||
if segment.Start < slice.Start || segment.End > slice.End {
|
||||
return nil, fmt.Errorf("动作不能跨越切片边界")
|
||||
}
|
||||
actions = append(actions, map[string]any{"id": segment.ID, "status": effectiveActionStatus(segment), "isInvalid": segment.IsInvalid, "description": segment.Description, "descriptionTemplateId": segment.LabelID, "phrases": segment.Phrases, "source": segment.Source, "confidence": segment.Confidence, "clipStartMs": (segment.Start - slice.Start) * 1000, "clipEndMs": (segment.End - slice.Start) * 1000, "startTimestampUs": segment.StartTimestampUS, "endTimestampUs": segment.EndTimestampUS})
|
||||
actions = append(actions, map[string]any{"id": segment.ID, "status": effectiveActionStatus(segment), "isInvalid": segment.IsInvalid, "description": segment.Description, "descriptionTemplateId": segment.LabelID, "dictionary": segment.Dictionary, "phrases": segment.Phrases, "source": segment.Source, "confidence": segment.Confidence, "clipStartMs": (segment.Start - slice.Start) * 1000, "clipEndMs": (segment.End - slice.Start) * 1000, "startTimestampUs": segment.StartTimestampUS, "endTimestampUs": segment.EndTimestampUS})
|
||||
}
|
||||
window := map[string]any{"startTimestampUs": slice.StartTimestampUS, "endTimestampUs": slice.EndTimestampUS, "semantics": "half-open"}
|
||||
annotation := map[string]any{"schemaVersion": "droboticflow.timestamped-description/v3", "boundaryContract": "hardware-frame-v1", "sliceId": id, "sliceName": slice.Name, "window": window, "actions": actions, "descriptions": actions}
|
||||
annotation := map[string]any{"schemaVersion": "droboticflow.timestamped-description/v3", "boundaryContract": "hardware-frame-v1", "sliceId": id, "sliceName": slice.Name, "dictionary": slice.Dictionary, "window": window, "actions": actions, "descriptions": actions}
|
||||
seconds, rightSeconds, stamps := []float64{}, []float64{}, []int64{}
|
||||
for j := a; j <= b; j++ {
|
||||
seconds = append(seconds, grid.Seconds[j]-slice.Start)
|
||||
@ -96,8 +96,8 @@ func (s *server) materializeSourceIndexRevision(ctx context.Context, t domain.An
|
||||
stamps = append(stamps, grid.Timestamps[j])
|
||||
}
|
||||
frames := map[string]any{"left": []int{leftStart, leftEnd}, "right": []int{rightStart, rightEnd}}
|
||||
sync := map[string]any{"schemaVersion": "droboticflow.synchronized-slice/v1", "sourceIndexSchema": sourceIndexSchema, "boundaryContract": "hardware-frame-v1", "collectionId": cert.CollectionID, "sliceIndex": i + 1, "sliceCount": len(slices), "sliceName": slice.Name, "window": window, "boundarySeconds": seconds, "rightBoundarySeconds": rightSeconds, "boundaryTimestampUs": stamps, "sourceFrames": frames, "sourceMediaOffsets": map[string]any{"left": slice.Start, "right": cert.Timeline.RightSeconds[a]}, "deliveryDisposition": sliceDisposition(*slice), "left": map[string]any{"pts": map[string]any{"rowCount": leftEnd - leftStart}}, "right": map[string]any{"pts": map[string]any{"rowCount": rightEnd - rightStart}}, "imu": map[string]any{"sampleCount": cert.IMURowsBeforeBoundary[b] - cert.IMURowsBeforeBoundary[a]}, "commonRange": map[string]any{"startTimestampUs": grid.Timestamps[0], "endTimestampUs": grid.Timestamps[len(grid.Timestamps)-1], "startSecond": grid.Seconds[0], "endSecond": grid.Seconds[len(grid.Seconds)-1]}, "coverage": map[string]any{"mode": "full_common_range", "contiguous": true}}
|
||||
entry := map[string]any{"sliceId": id, "sourceSliceId": slice.ID, "sliceName": slice.Name, "startSecond": slice.Start, "endSecond": slice.End, "startTimestampUs": slice.StartTimestampUS, "endTimestampUs": slice.EndTimestampUS, "directory": filepath.ToSlash(filepath.Join("slices", id)), "sourceFrames": frames, "deliveryDisposition": sliceDisposition(*slice)}
|
||||
sync := map[string]any{"schemaVersion": "droboticflow.synchronized-slice/v1", "sourceIndexSchema": sourceIndexSchema, "boundaryContract": "hardware-frame-v1", "collectionId": cert.CollectionID, "sliceIndex": i + 1, "sliceCount": len(slices), "sliceName": slice.Name, "dictionary": slice.Dictionary, "window": window, "boundarySeconds": seconds, "rightBoundarySeconds": rightSeconds, "boundaryTimestampUs": stamps, "sourceFrames": frames, "sourceMediaOffsets": map[string]any{"left": slice.Start, "right": cert.Timeline.RightSeconds[a]}, "deliveryDisposition": sliceDisposition(*slice), "left": map[string]any{"pts": map[string]any{"rowCount": leftEnd - leftStart}}, "right": map[string]any{"pts": map[string]any{"rowCount": rightEnd - rightStart}}, "imu": map[string]any{"sampleCount": cert.IMURowsBeforeBoundary[b] - cert.IMURowsBeforeBoundary[a]}, "commonRange": map[string]any{"startTimestampUs": grid.Timestamps[0], "endTimestampUs": grid.Timestamps[len(grid.Timestamps)-1], "startSecond": grid.Seconds[0], "endSecond": grid.Seconds[len(grid.Seconds)-1]}, "coverage": map[string]any{"mode": "full_common_range", "contiguous": true}}
|
||||
entry := map[string]any{"sliceId": id, "sourceSliceId": slice.ID, "sliceName": slice.Name, "dictionary": slice.Dictionary, "startSecond": slice.Start, "endSecond": slice.End, "startTimestampUs": slice.StartTimestampUS, "endTimestampUs": slice.EndTimestampUS, "directory": filepath.ToSlash(filepath.Join("slices", id)), "sourceFrames": frames, "deliveryDisposition": sliceDisposition(*slice)}
|
||||
if slice.IsDiscarded() {
|
||||
entry["discardEvidence"] = discardReasonEvidence(*slice)
|
||||
sync["discardEvidence"] = discardReasonEvidence(*slice)
|
||||
@ -126,12 +126,20 @@ func (s *server) materializeSourceIndexRevision(ctx context.Context, t domain.An
|
||||
coverage := map[string]any{"mode": "full_common_range", "contiguous": true, "gapMicroseconds": 0, "overlapMicroseconds": 0, "startSecond": grid.Seconds[0], "endSecond": grid.Seconds[previous], "durationSeconds": grid.Seconds[previous] - grid.Seconds[0], "startTimestampUs": grid.Timestamps[0], "endTimestampUs": grid.Timestamps[previous], "ptsToMediaScale": cert.Timeline.Scale, "sliceCount": len(slices), "reconstruction": map[string]any{"leftFrameCount": grid.LeftFrames[previous] - grid.LeftFrames[0], "rightFrameCount": grid.RightFrames[previous] - grid.RightFrames[0], "imuRowCount": cert.IMURowsBeforeBoundary[previous] - cert.IMURowsBeforeBoundary[0]}}
|
||||
selection := map[string]any{"keptSliceCount": kept, "discardedSliceCount": discarded, "keptDurationSeconds": keepDuration, "discardedDurationSeconds": discardDuration}
|
||||
capture := map[string]any{}
|
||||
var originalSource map[string]any
|
||||
if record, e := s.store.GetCaptureRecordByRunID(ctx, t.RunID); e != nil {
|
||||
return nil, e
|
||||
} else if record != nil {
|
||||
originalSource, e = annotationRevisionOriginalSource(run, record)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
capture = map[string]any{"captureRecordId": record.ID, "captureId": record.CaptureID, "prefix": record.Prefix, "fingerprint": record.Fingerprint, "contractRevision": record.ContractRevision}
|
||||
}
|
||||
manifest := map[string]any{"schemaVersion": sourceIndexSchema, "revision": revision, "taskId": t.ID, "humanItemId": h.ID, "runId": t.RunID, "collectionId": cert.CollectionID, "capture": capture, "sourceFingerprint": fingerprint, "sources": cert.Sources, "sourceDurations": map[string]any{"left": cert.LeftDuration, "right": cert.RightDuration}, "status": "materialized", "sliceCount": len(items), "coverage": coverage, "deliverySelection": selection, "items": items}
|
||||
if originalSource != nil {
|
||||
manifest["originalSource"] = originalSource
|
||||
}
|
||||
manifest["sourceFrameCounts"] = map[string]any{"left": len(cert.LeftPTS), "right": len(cert.RightPTS)}
|
||||
encoded, e := json.Marshal(manifest)
|
||||
if e != nil || len(encoded)+1 > maxRunArtifactManifestBytes {
|
||||
|
||||
@ -7860,6 +7860,12 @@
|
||||
"type": "boolean",
|
||||
"value": "false",
|
||||
"description": "原生 v3:part.json 与独立审核历史,通过目录发布;须同时启用 PART_INDEX_DELIVERY"
|
||||
},
|
||||
{
|
||||
"key": "DELIVERY_JSON_V4",
|
||||
"type": "boolean",
|
||||
"value": "false",
|
||||
"description": "最终分类总 JSON;审核返工追溯单独保存在内部,须启用 PART_INDEX_DELIVERY"
|
||||
}
|
||||
],
|
||||
"id": "operator-builtin-package",
|
||||
@ -8170,6 +8176,16 @@
|
||||
"role": "summary",
|
||||
"transmission": "oss",
|
||||
"type": "json"
|
||||
},
|
||||
{
|
||||
"id": "deliveryAudit",
|
||||
"name": "deliveryAudit",
|
||||
"type": "json",
|
||||
"container": "single",
|
||||
"role": "internal-audit",
|
||||
"payloadMode": "inline",
|
||||
"transmission": "bus",
|
||||
"description": "内部审核与返工追溯,不属于客户交付文件"
|
||||
}
|
||||
],
|
||||
"publicationStatus": "built",
|
||||
|
||||
125
internal/api/delivery_classification_audit.go
Normal file
125
internal/api/delivery_classification_audit.go
Normal file
@ -0,0 +1,125 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"droboticflow/internal/deliveryindex"
|
||||
"droboticflow/internal/domain"
|
||||
)
|
||||
|
||||
// Read only the audit produced by the same completed, tenant-scoped package.
|
||||
// Customer JSON deliberately contains no private run pointers.
|
||||
func (s *server) classificationPackageAudit(r *http.Request, packages []domain.DeliveryPackageRecord, target deliveryDownloadTarget, partBytes []byte) ([]byte, string, error) {
|
||||
var part struct {
|
||||
Capture struct {
|
||||
ID string `json:"captureId"`
|
||||
} `json:"capture"`
|
||||
PartID string `json:"partId"`
|
||||
Revision int `json:"revision"`
|
||||
}
|
||||
if json.Unmarshal(partBytes, &part) != nil {
|
||||
return nil, "", fmt.Errorf("invalid classification part")
|
||||
}
|
||||
var matched []byte
|
||||
sourceRun := ""
|
||||
for _, pkg := range packages {
|
||||
run, err := s.store.GetRun(r.Context(), pkg.ChildRunID)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
if run == nil || run.OrganizationID != s.requestOrganizationID(r) || run.ProjectID != pkg.ProjectID || run.Status != domain.RunStatusCompleted {
|
||||
continue
|
||||
}
|
||||
ownsTarget := false
|
||||
for _, a := range run.Artifacts {
|
||||
if !deliveryPackageArtifactAllowed(&pkg, a.ID) || a.RunID != run.ID || a.Payload == nil || a.Payload.Ref == nil {
|
||||
continue
|
||||
}
|
||||
ref := a.Payload.Ref
|
||||
if ref.ConnectionID == target.connectionID && ref.Bucket == target.bucket && strings.Trim(ref.Prefix, "/") == target.prefix && ref.ManifestKey == target.prefix+"/manifest.json" {
|
||||
ownsTarget = true
|
||||
}
|
||||
}
|
||||
if !ownsTarget {
|
||||
continue
|
||||
}
|
||||
for _, a := range run.Artifacts {
|
||||
if a.RunID != run.ID || !domain.IsInternalDeliveryAudit(a) {
|
||||
continue
|
||||
}
|
||||
data, readErr := s.readClassificationAudit(r, run, a)
|
||||
if readErr != nil {
|
||||
return nil, "", readErr
|
||||
}
|
||||
raw, err := json.Marshal(data)
|
||||
if err != nil || len(raw) > 8<<20 {
|
||||
return nil, "", fmt.Errorf("内部追溯不可读或超过 8 MiB")
|
||||
}
|
||||
var audit struct {
|
||||
Schema string `json:"schemaVersion"`
|
||||
CaptureID string `json:"captureId"`
|
||||
PartID string `json:"partId"`
|
||||
Revision int `json:"revision"`
|
||||
PartSHA string `json:"partSha256"`
|
||||
Lineage struct {
|
||||
RunID string `json:"sourceRunId"`
|
||||
} `json:"lineage"`
|
||||
}
|
||||
if json.Unmarshal(raw, &audit) != nil || audit.Schema != "droboticflow.delivery.review-history/v2" || audit.CaptureID != part.Capture.ID || audit.PartID != part.PartID || audit.Revision != part.Revision || audit.PartSHA != deliveryindex.Hash(partBytes) || audit.Lineage.RunID != pkg.ParentRunID {
|
||||
return nil, "", fmt.Errorf("内部追溯与固定验收结果不一致")
|
||||
}
|
||||
if matched != nil {
|
||||
return nil, "", fmt.Errorf("交付分类对应多个内部追溯")
|
||||
}
|
||||
matched, sourceRun = raw, audit.Lineage.RunID
|
||||
}
|
||||
}
|
||||
if matched == nil || sourceRun == "" {
|
||||
return nil, "", fmt.Errorf("缺少同一正式交付任务的内部验收证据")
|
||||
}
|
||||
return matched, sourceRun, nil
|
||||
}
|
||||
|
||||
func (s *server) readClassificationAudit(r *http.Request, run *domain.RunRecord, a domain.ExecutionArtifact) (map[string]any, error) {
|
||||
if a.Payload != nil && a.Payload.Ref != nil {
|
||||
ref := a.Payload.Ref
|
||||
if ref.Kind != domain.ArtifactRefKindObject || ref.SizeBytes <= 0 || ref.SizeBytes > 8<<20 || len(ref.SHA256) != 64 {
|
||||
return nil, fmt.Errorf("内部追溯引用缺少完整校验")
|
||||
}
|
||||
connection, err := s.resolveArtifactStorageConnection(r.Context(), run, a, ref.Bucket)
|
||||
if err != nil || connection == nil || connection.ID != ref.ConnectionID {
|
||||
return nil, fmt.Errorf("内部追溯连接不可访问")
|
||||
}
|
||||
browser, cleanup, err := storageConnectionBrowserResolver(r.Context(), s.storageConnectionWithResolvedEndpoint(connection))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if cleanup != nil {
|
||||
defer cleanup()
|
||||
}
|
||||
raw, _, err := browser.ReadObjectPreview(r.Context(), ref.Bucket, ref.Key, 8<<20+1)
|
||||
if err != nil || int64(len(raw)) != ref.SizeBytes || deliveryindex.Hash(raw) != ref.SHA256 {
|
||||
return nil, fmt.Errorf("内部追溯文件校验失败")
|
||||
}
|
||||
var data map[string]any
|
||||
if json.Unmarshal(raw, &data) != nil {
|
||||
return nil, fmt.Errorf("内部追溯 JSON 无效")
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
if a.Payload != nil && a.Payload.Mode == domain.ArtifactTransferModeInline {
|
||||
raw, err := json.Marshal(a.Payload.Inline)
|
||||
if err != nil || len(raw) > 8<<20 {
|
||||
return nil, fmt.Errorf("内部追溯过大")
|
||||
}
|
||||
var data map[string]any
|
||||
if json.Unmarshal(raw, &data) != nil {
|
||||
return nil, fmt.Errorf("内部追溯 JSON 无效")
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
return artifactStructuredData(&a), nil
|
||||
}
|
||||
@ -99,7 +99,7 @@ func (s *server) planIndexBatch(r *http.Request, q indexBatchRequest) (*batchPla
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
if len(files) != 3 {
|
||||
if len(files) != 2 && len(files) != 3 {
|
||||
return nil, fmt.Errorf("原生 v3 part 包必须只含 part.json 与独立审核历史")
|
||||
}
|
||||
var partBytes, historyBytes []byte
|
||||
@ -124,14 +124,24 @@ func (s *server) planIndexBatch(r *http.Request, q indexBatchRequest) (*batchPla
|
||||
historyBytes = raw
|
||||
}
|
||||
}
|
||||
if len(partBytes) == 0 || len(historyBytes) == 0 {
|
||||
if len(partBytes) == 0 {
|
||||
return nil, fmt.Errorf("选择的包尚未由原生 v3 算子交付;旧格式必须另行重交,不会覆盖旧文件")
|
||||
}
|
||||
var part struct {
|
||||
Schema string `json:"schemaVersion"`
|
||||
Capture struct {
|
||||
ID string `json:"captureId"`
|
||||
Prefix string `json:"prefix"`
|
||||
} `json:"capture"`
|
||||
OriginalSource struct {
|
||||
Provider string `json:"provider"`
|
||||
Bucket string `json:"bucket"`
|
||||
Prefix string `json:"prefix"`
|
||||
PartPrefix string `json:"partPrefix"`
|
||||
Files map[string]struct {
|
||||
Path string `json:"path"`
|
||||
} `json:"files"`
|
||||
} `json:"originalSource"`
|
||||
PartID string `json:"partId"`
|
||||
Lineage struct {
|
||||
RunID string `json:"sourceRunId"`
|
||||
@ -140,16 +150,53 @@ func (s *server) planIndexBatch(r *http.Request, q indexBatchRequest) (*batchPla
|
||||
if json.Unmarshal(partBytes, &part) != nil || target.prefix != root+"/"+part.Capture.ID+"/"+part.PartID {
|
||||
return nil, fmt.Errorf("part 身份与冻结交付目标不一致")
|
||||
}
|
||||
switch part.Schema {
|
||||
case deliveryindex.ClassificationPartSchema:
|
||||
if len(files) != 2 || len(historyBytes) != 0 {
|
||||
return nil, fmt.Errorf("分类交付包中包含额外文件")
|
||||
}
|
||||
historyBytes, part.Lineage.RunID, e = s.classificationPackageAudit(r, selected, target, partBytes)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
case deliveryindex.PartSchema:
|
||||
if len(files) != 3 || len(historyBytes) == 0 {
|
||||
return nil, fmt.Errorf("旧 v3 交付缺少审核历史")
|
||||
}
|
||||
default:
|
||||
return nil, fmt.Errorf("不支持的交付格式;历史交付不会被覆盖")
|
||||
}
|
||||
record, e := s.store.GetCaptureRecordByRunID(r.Context(), part.Lineage.RunID)
|
||||
if e != nil || record == nil || record.OrganizationID != project.OrganizationID || record.ProjectID != project.ID || record.Prefix != part.Capture.Prefix+"/"+part.PartID || record.Delivery == nil || record.Delivery.Prefix != target.prefix || record.Delivery.ConnectionID != t.connectionID || record.Delivery.Bucket != t.bucket {
|
||||
return nil, fmt.Errorf("part 来源与采集记录不一致")
|
||||
}
|
||||
if part.Schema == deliveryindex.ClassificationPartSchema && (part.OriginalSource.Provider != "BOS" || part.OriginalSource.Bucket != record.Bucket || part.OriginalSource.Prefix != part.Capture.Prefix || part.OriginalSource.PartPrefix != record.Prefix) {
|
||||
return nil, fmt.Errorf("分类 JSON 原始存储来源与采集记录不一致,停止发布")
|
||||
}
|
||||
sourceRuns = append(sourceRuns, part.Lineage.RunID)
|
||||
sourceBrowser, closeSource, e := s.deliveryReadBrowser(r.Context(), project.OrganizationID, record.ConnectionID, record.Bucket, part.Capture.Prefix, false)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
expected, meta, e := indexBatchSourceEvidence(r.Context(), sourceBrowser, record.Bucket, part.Capture.Prefix, part.Capture.ID)
|
||||
if e == nil && part.Schema == deliveryindex.ClassificationPartSchema {
|
||||
// HEAD only: prove the frozen references still resolve without reading
|
||||
// video bodies, recutting media or expanding the original selection.
|
||||
for _, role := range []string{"leftVideo", "rightVideo", "leftPts", "rightPts", "imu", "cameraCalibration", "imuCalibration", "metadata"} {
|
||||
rel := part.OriginalSource.Files[role].Path
|
||||
media := role == "leftVideo" || role == "rightVideo" || role == "leftPts" || role == "rightPts" || role == "imu"
|
||||
if rel == "" || path.Clean(rel) != rel || strings.ContainsAny(rel, "\\\x00") || strings.HasPrefix(rel, "/") || strings.HasPrefix(rel, "../") || rel == ".." || media && !strings.HasPrefix(rel, part.PartID+"/") || !media && path.Base(rel) != rel {
|
||||
e = fmt.Errorf("分类 JSON 原始文件路径无效:%s", role)
|
||||
break
|
||||
}
|
||||
key := part.Capture.Prefix + "/" + rel
|
||||
info, err := sourceBrowser.GetObjectSummary(r.Context(), record.Bucket, key)
|
||||
if err != nil || info.Key != key || info.Kind != "file" || info.SizeBytes <= 0 {
|
||||
e = fmt.Errorf("分类 JSON 原始文件不存在或不可读:%s", role)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if closeSource != nil {
|
||||
closeSource()
|
||||
}
|
||||
@ -425,14 +472,14 @@ func readIndexBatchDownloadManifest(ctx context.Context, b objectStorageBrowser,
|
||||
} `json:"project"`
|
||||
Entries []deliveryindex.Entry `json:"entries"`
|
||||
}
|
||||
if json.Unmarshal(raw, &index) != nil || index.Schema != 3 || index.Kind != "part-index" || index.ID != id || index.Project.Root != t.prefix || index.Count != len(index.Entries) || index.Count < 1 || index.Count > 400 {
|
||||
if json.Unmarshal(raw, &index) != nil || (index.Schema != 3 && index.Schema != 4) || index.Kind != "part-index" || index.ID != id || index.Project.Root != t.prefix || index.Count != len(index.Entries) || index.Count < 1 || index.Count > 400 {
|
||||
return nil, fmt.Errorf("批次总索引身份或成员数量无效")
|
||||
}
|
||||
files := []deliveryDownloadFile{}
|
||||
var size int64
|
||||
for _, entry := range index.Entries {
|
||||
sha, shaErr := hex.DecodeString(entry.SHA256)
|
||||
if !validDeliveryDownloadPath(entry.Path) || entry.Path == "delivery_index.json" || strings.HasPrefix(entry.Path, ".") || entry.Key != t.prefix+"/"+entry.Path || entry.SizeBytes < 0 || entry.SizeBytes > 8<<20 || shaErr != nil || len(sha) != 32 {
|
||||
if (index.Schema == 4 && !validClassificationBatchPath(entry.Path)) || !validDeliveryDownloadPath(entry.Path) || entry.Path == "delivery_index.json" || strings.HasPrefix(entry.Path, ".") || entry.Key != t.prefix+"/"+entry.Path || entry.SizeBytes < 0 || entry.SizeBytes > 8<<20 || shaErr != nil || len(sha) != 32 {
|
||||
return nil, fmt.Errorf("批次成员校验信息无效")
|
||||
}
|
||||
size += entry.SizeBytes
|
||||
@ -444,3 +491,15 @@ func readIndexBatchDownloadManifest(ctx context.Context, b objectStorageBrowser,
|
||||
files = append(files, deliveryDownloadFile{RelativePath: t.relative + "/delivery_index.json", SizeBytes: int64(len(raw)), SHA256: deliveryindex.Hash(raw), key: t.prefix + "/delivery_index.json", bucket: t.bucket, connectionID: t.connectionID})
|
||||
return files, nil
|
||||
}
|
||||
|
||||
func validClassificationBatchPath(value string) bool {
|
||||
parts := strings.Split(value, "/")
|
||||
if len(parts) != 2 {
|
||||
return false
|
||||
}
|
||||
if parts[1] == "annotation_manifest.json" {
|
||||
return true
|
||||
}
|
||||
name := strings.TrimSuffix(parts[1], ".json")
|
||||
return strings.HasSuffix(parts[1], ".json") && len(name) >= 8 && strings.HasPrefix(name, "part") && strings.Trim(name[4:], "0123456789") == ""
|
||||
}
|
||||
|
||||
@ -25,6 +25,14 @@ type indexTestBrowser struct {
|
||||
ambiguous bool
|
||||
}
|
||||
|
||||
func (b *indexTestBrowser) GetObjectSummary(_ context.Context, bucket, key string) (domain.ObjectStorageObjectSummary, error) {
|
||||
body, ok := b.objects[key]
|
||||
if !ok {
|
||||
return domain.ObjectStorageObjectSummary{}, os.ErrNotExist
|
||||
}
|
||||
return domain.ObjectStorageObjectSummary{Bucket: bucket, Key: key, Kind: "file", SizeBytes: int64(len(body))}, nil
|
||||
}
|
||||
|
||||
func (b *indexTestBrowser) PutObjectConditional(_ context.Context, _ string, key string, body []byte, _ string, match, none string, _ map[string]string) (domain.ObjectStorageObjectSummary, error) {
|
||||
if match != "" || none != "*" {
|
||||
return domain.ObjectStorageObjectSummary{}, fmt.Errorf("unconditional write")
|
||||
@ -108,6 +116,10 @@ func seedIndexFixture(t *testing.T, ds store.Store) (*indexTestBrowser, *authent
|
||||
must(ds.UpsertProject(ctx, domain.ProjectRecord{ID: "project-A", OrganizationID: org, Name: "项目A"}))
|
||||
must(ds.UpsertStorageConnection(ctx, domain.StorageConnectionRecord{ID: "index-storage", OrganizationID: org, Provider: domain.StorageProviderBOS, Bucket: "test-bucket", Endpoint: "https://bj.bcebos.com"}))
|
||||
dir := os.Getenv("DELIVERY_V3_FIXTURE_DIR")
|
||||
v4 := os.Getenv("DELIVERY_CLASSIFICATION_TEST") == "1"
|
||||
if v4 {
|
||||
dir = os.Getenv("DELIVERY_V4_FIXTURE_DIR")
|
||||
}
|
||||
for _, part := range []string{"part0001", "part0002"} {
|
||||
prefix := root + "/capture-A/" + part
|
||||
sourceRun := "source-run-" + part
|
||||
@ -119,19 +131,57 @@ func seedIndexFixture(t *testing.T, ds store.Store) (*indexTestBrowser, *authent
|
||||
var e error
|
||||
body, e = os.ReadFile(filepath.Join(dir, part, "part.json"))
|
||||
must(e)
|
||||
history, e = os.ReadFile(filepath.Join(dir, part, "review-history.json"))
|
||||
historyPath := filepath.Join(dir, part, "review-history.json")
|
||||
if v4 {
|
||||
historyPath = filepath.Join(dir, "internal", part, "review-history.json")
|
||||
}
|
||||
history, e = os.ReadFile(historyPath)
|
||||
must(e)
|
||||
}
|
||||
if v4 && dir == "" {
|
||||
var doc map[string]any
|
||||
must(json.Unmarshal(body, &doc))
|
||||
acceptance := doc["acceptance"]
|
||||
delete(doc, "acceptance")
|
||||
delete(doc, "lineage")
|
||||
doc["schemaVersion"] = deliveryindex.ClassificationPartSchema
|
||||
enrichConsumerFixture(doc)
|
||||
body, _ = json.Marshal(doc)
|
||||
history, _ = json.Marshal(map[string]any{"schemaVersion": "droboticflow.delivery.review-history/v2", "captureId": "capture-A", "partId": part, "revision": 1, "partSha256": deliveryindex.Hash(body), "acceptance": acceptance, "lineage": map[string]any{"sourceRunId": sourceRun}, "qualityReviews": []any{map[string]any{"comment": "PRIVATE_REWORK"}}, "trace": []any{}})
|
||||
}
|
||||
b.objects[prefix+"/.delivery-owner.json"] = []byte(`{"owner":"synthetic-source"}`)
|
||||
b.objects[prefix+"/part.json"] = body
|
||||
b.objects[prefix+"/review-history.json"] = history
|
||||
entries := []deliveryindex.Entry{{Path: "part.json", Key: prefix + "/part.json", SizeBytes: int64(len(body)), SHA256: deliveryindex.Hash(body)}, {Path: "review-history.json", Key: prefix + "/review-history.json", SizeBytes: int64(len(history)), SHA256: deliveryindex.Hash(history)}}
|
||||
b.objects[prefix+"/manifest.json"], _ = json.Marshal(map[string]any{"schemaVersion": 1, "kind": "directory", "fileCount": 2, "totalBytes": len(body) + len(history), "entries": entries})
|
||||
if v4 {
|
||||
delete(b.objects, prefix+"/review-history.json")
|
||||
entries = entries[:1]
|
||||
b.objects[prefix+"/manifest.json"], _ = json.Marshal(map[string]any{"schemaVersion": 1, "kind": "directory", "fileCount": 1, "totalBytes": len(body), "entries": entries})
|
||||
}
|
||||
b.objects["incoming/capture-A/"+part+"/left.mp4"] = []byte("synthetic")
|
||||
if v4 {
|
||||
var doc struct {
|
||||
Original struct {
|
||||
Files map[string]struct {
|
||||
Path string `json:"path"`
|
||||
} `json:"files"`
|
||||
} `json:"originalSource"`
|
||||
}
|
||||
must(json.Unmarshal(body, &doc))
|
||||
for _, file := range doc.Original.Files {
|
||||
b.objects["incoming/capture-A/"+file.Path] = []byte("synthetic original")
|
||||
}
|
||||
}
|
||||
run := domain.RunRecord{ID: runID, OrganizationID: org, ProjectID: "project-A", Status: domain.RunStatusCompleted, Artifacts: []domain.ExecutionArtifact{{ID: artifactID, RunID: runID, Name: "delivery.directory.index", Payload: &domain.ArtifactPayload{Mode: domain.ArtifactTransferModeRef, Ref: &domain.ArtifactReference{Kind: domain.ArtifactRefKindDirectory, Provider: domain.StorageProviderBOS, ConnectionID: "index-storage", Bucket: "test-bucket", Prefix: prefix, ManifestKey: prefix + "/manifest.json"}}}}}
|
||||
if v4 {
|
||||
var audit map[string]any
|
||||
must(json.Unmarshal(history, &audit))
|
||||
run.Artifacts = append(run.Artifacts, domain.ExecutionArtifact{ID: "audit-" + part, RunID: runID, PortID: "deliveryAudit", Type: domain.PortTypeJSON, Data: audit})
|
||||
}
|
||||
run.NodeExecutions = []domain.NodeExecutionRecord{{NodeID: "package", Artifacts: run.Artifacts}}
|
||||
must(ds.UpsertRun(ctx, run))
|
||||
must(ds.UpsertDeliveryPackage(ctx, domain.DeliveryPackageRecord{ID: "package-" + part, OrganizationID: org, ProjectID: "project-A", ChildRunID: runID, Status: domain.DeliveryPackageAccepted, Mode: domain.RunModeProduction, FinalExportArtifactIDs: []string{artifactID}}))
|
||||
must(ds.UpsertDeliveryPackage(ctx, domain.DeliveryPackageRecord{ID: "package-" + part, OrganizationID: org, ProjectID: "project-A", ParentRunID: sourceRun, ChildRunID: runID, Status: domain.DeliveryPackageAccepted, Mode: domain.RunModeProduction, FinalExportArtifactIDs: []string{artifactID}}))
|
||||
must(ds.UpsertCaptureRecord(ctx, domain.CaptureRecord{ID: "record-" + part, OrganizationID: org, ProjectID: "project-A", RunID: sourceRun, CaptureID: "capture-A/" + part, ConnectionID: "index-storage", Bucket: "test-bucket", Prefix: "incoming/capture-A/" + part, SourcePrefix: "incoming/capture-A", Delivery: &domain.CaptureDeliveryTarget{ConnectionID: "index-storage", Bucket: "test-bucket", Prefix: prefix}}))
|
||||
}
|
||||
b.objects["incoming/capture-A/meta_capture-A.json"] = []byte(`{"synthetic":true}`)
|
||||
@ -207,6 +257,16 @@ func exerciseIndexBatch(t *testing.T, ds store.Store) {
|
||||
if download.Code != 200 || !strings.Contains(download.Body.String(), "delivery_index.json") {
|
||||
t.Fatal(download.Code, download.Body.String())
|
||||
}
|
||||
if os.Getenv("DELIVERY_CLASSIFICATION_TEST") == "1" {
|
||||
for _, file := range files {
|
||||
if strings.Contains(file.RelativePath, "history") || bytes.Contains(b.objects[file.key], []byte("PRIVATE_REWORK")) {
|
||||
t.Fatal("private audit delivered")
|
||||
}
|
||||
}
|
||||
if strings.Contains(download.Body.String(), "history") {
|
||||
t.Fatal("download exposes private audit")
|
||||
}
|
||||
}
|
||||
// A later partial snapshot does not alter the old full snapshot.
|
||||
first := indexCall(h, p, "GET", "/api/delivery-index-batches?projectId=project-A&packageId=package-part0001", nil)
|
||||
var partial struct {
|
||||
@ -299,3 +359,98 @@ func TestIndexBatchDelegationKeepsSourceAndDeliveryScopesSeparate(t *testing.T)
|
||||
t.Fatal("sibling batch prefix escaped delivery grant", w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassificationBatchHTTPPublicationRecoveryAndDownload(t *testing.T) {
|
||||
t.Setenv("DELIVERY_CLASSIFICATION_TEST", "1")
|
||||
exerciseIndexBatch(t, store.NewMemoryStore())
|
||||
}
|
||||
|
||||
func TestClassificationBatchRejectsMissingOrChangedOriginalSource(t *testing.T) {
|
||||
t.Setenv("DELIVERY_CLASSIFICATION_TEST", "1")
|
||||
t.Setenv("DELIVERY_V4_FIXTURE_DIR", "")
|
||||
for _, failure := range []string{"bucket", "missing-file"} {
|
||||
t.Run(failure, func(t *testing.T) {
|
||||
ds := store.NewMemoryStore()
|
||||
b, p, _ := seedIndexFixture(t, ds)
|
||||
if failure == "bucket" {
|
||||
record, err := ds.GetCaptureRecordByRunID(t.Context(), "source-run-part0001")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
record.Bucket = "changed-bucket"
|
||||
if err = ds.UpsertCaptureRecord(t.Context(), *record); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
} else {
|
||||
delete(b.objects, "incoming/capture-A/part0001/leftVideo")
|
||||
}
|
||||
w := indexCall(NewHandler(ds, nil), p, "GET", "/api/delivery-index-batches?projectId=project-A&packageId=package-part0001", nil)
|
||||
if w.Code != 409 || len(b.puts) != 0 || !strings.Contains(w.Body.String(), "原始") {
|
||||
t.Fatalf("incomplete source accepted: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func enrichConsumerFixture(p map[string]any) {
|
||||
part := p["partId"].(string)
|
||||
p["consumerContract"] = "droboticflow.classification-consumer/v1"
|
||||
p["timeBases"] = map[string]string{"taskSplitSeconds": "original_left_video", "segmentSeconds": "slice_left_video", "segmentPartSeconds": "original_left_video", "timestampUs": "original_hardware_pts", "interval": "half-open", "frameRanges": "zero-based-half-open"}
|
||||
p["pathBases"] = map[string]string{"sources": "original_capture", "sourceMetadata": "original_capture", "calibration": "original_capture"}
|
||||
files := map[string]any{}
|
||||
sources := map[string]any{}
|
||||
for _, role := range []string{"leftVideo", "rightVideo", "leftPts", "rightPts", "imu", "cameraCalibration", "imuCalibration", "metadata"} {
|
||||
name := part + "/" + role
|
||||
if role == "cameraCalibration" || role == "imuCalibration" {
|
||||
name = role + ".yaml"
|
||||
}
|
||||
if role == "metadata" {
|
||||
name = "meta_capture-A.json"
|
||||
}
|
||||
files[role] = map[string]any{"path": name}
|
||||
if role == "leftVideo" || role == "rightVideo" || role == "leftPts" || role == "rightPts" || role == "imu" {
|
||||
sources[role] = files[role]
|
||||
}
|
||||
}
|
||||
p["sources"] = sources
|
||||
p["sourceMetadata"] = files["metadata"]
|
||||
p["originalSource"] = map[string]any{"provider": "BOS", "bucket": "test-bucket", "prefix": "incoming/capture-A", "partPrefix": "incoming/capture-A/" + part, "files": files}
|
||||
for _, kind := range []string{"slices", "discardedSlices"} {
|
||||
for _, raw := range p[kind].([]any) {
|
||||
item := raw.(map[string]any)
|
||||
split := item["taskSplit"].(map[string]any)
|
||||
number := func(v any) float64 {
|
||||
switch n := v.(type) {
|
||||
case float64:
|
||||
return n
|
||||
case int:
|
||||
return float64(n)
|
||||
default:
|
||||
panic("invalid fixture time")
|
||||
}
|
||||
}
|
||||
start, end := number(split["startSecond"]), number(split["endSecond"])
|
||||
split["startTimestampUs"] = int64(1e6 + start*1e6)
|
||||
split["endTimestampUs"] = int64(1e6 + end*1e6)
|
||||
if kind == "discardedSlices" {
|
||||
continue
|
||||
}
|
||||
class := item["classification"].(map[string]any)
|
||||
class["dictionary"] = map[string]any{"id": "base", "kind": "slice", "version": 1, "name": "操作", "tags": []string{"操作"}}
|
||||
class["tags"] = []string{"操作"}
|
||||
actions := item["segments"].([]any)
|
||||
for i, raw := range actions {
|
||||
a := raw.(map[string]any)
|
||||
low, high := float64(i)*(end-start)/float64(len(actions)), float64(i+1)*(end-start)/float64(len(actions))
|
||||
a["startSecond"] = low
|
||||
a["endSecond"] = high
|
||||
a["partStartSecond"] = start + low
|
||||
a["partEndSecond"] = start + high
|
||||
a["startTimestampUs"] = int64(1e6 + (start+low)*1e6)
|
||||
a["endTimestampUs"] = int64(1e6 + (start+high)*1e6)
|
||||
a["tags"] = []string{"操作"}
|
||||
a["tagSource"] = "slice"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -114,7 +114,7 @@ func (server *server) handleDeliveryPackageContents(writer http.ResponseWriter,
|
||||
artifacts := make([]domain.ExecutionArtifact, 0, len(allowed))
|
||||
var manifest map[string]any
|
||||
for _, artifact := range run.Artifacts {
|
||||
if _, exists := allowed[artifact.ID]; !exists || artifact.RunID != run.ID {
|
||||
if _, exists := allowed[artifact.ID]; !exists || artifact.RunID != run.ID || domain.IsInternalDeliveryAudit(artifact) {
|
||||
continue
|
||||
}
|
||||
copy := artifact
|
||||
|
||||
80
internal/api/dictionary_selection_history.go
Normal file
80
internal/api/dictionary_selection_history.go
Normal file
@ -0,0 +1,80 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
)
|
||||
|
||||
const dictionarySelectionHistoryKey = "dictionarySelectionHistory"
|
||||
|
||||
// Only server-resolved selections enter this item-scoped recovery ledger.
|
||||
// It allows undo after a shared word changes or leaves the project, without
|
||||
// treating the client's undo snapshot as trustworthy dictionary metadata.
|
||||
func rememberDictionarySelections(item *domain.HumanItemRecord, selections ...*domain.DictionarySelection) []validationIssue {
|
||||
if item == nil {
|
||||
return nil
|
||||
}
|
||||
history := decodeMaterializationValue[domain.DictionarySelection](item.Subject.Payload[dictionarySelectionHistoryKey])
|
||||
seen := map[string]bool{}
|
||||
key := func(d domain.DictionarySelection) string { return fmt.Sprintf("%s/%s/%d", d.Kind, d.ID, d.Version) }
|
||||
for _, d := range history {
|
||||
seen[key(d)] = true
|
||||
}
|
||||
for _, d := range selections {
|
||||
if d != nil && !seen[key(*d)] {
|
||||
copy := *d
|
||||
copy.Tags = append([]string{}, d.Tags...)
|
||||
history = append(history, copy)
|
||||
seen[key(copy)] = true
|
||||
}
|
||||
}
|
||||
body, err := json.Marshal(history)
|
||||
if err != nil || len(body) > 1<<20 {
|
||||
return []validationIssue{{Field: "dictionary", Message: "分类恢复记录超过限制,请拆分任务后重试"}}
|
||||
}
|
||||
if item.Subject.Payload == nil {
|
||||
item.Subject.Payload = map[string]any{}
|
||||
}
|
||||
if len(history) > 0 {
|
||||
item.Subject.Payload[dictionarySelectionHistoryKey] = history
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func savedDictionarySelection(item *domain.HumanItemRecord, requested *domain.DictionarySelection, kind string) *domain.DictionarySelection {
|
||||
if item == nil || requested == nil {
|
||||
return nil
|
||||
}
|
||||
for _, saved := range decodeMaterializationValue[domain.DictionarySelection](item.Subject.Payload[dictionarySelectionHistoryKey]) {
|
||||
if saved.Kind == kind && saved.ID == requested.ID && saved.Version == requested.Version {
|
||||
copy := saved
|
||||
return ©
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func sliceDictionarySelections(slices []domain.SliceSegment) []*domain.DictionarySelection {
|
||||
selections := []*domain.DictionarySelection{}
|
||||
for _, slice := range slices {
|
||||
selections = append(selections, slice.Dictionary, slice.DiscardReasonDictionary)
|
||||
}
|
||||
return selections
|
||||
}
|
||||
func actionDictionarySelections(actions []domain.AnnotationSegment) []*domain.DictionarySelection {
|
||||
selections := []*domain.DictionarySelection{}
|
||||
for _, action := range actions {
|
||||
selections = append(selections, action.Dictionary)
|
||||
}
|
||||
return selections
|
||||
}
|
||||
|
||||
func copyClassificationRecoveryState(target, source *domain.HumanItemRecord) {
|
||||
for _, key := range []string{dictionarySelectionHistoryKey, "unconfirmedSliceNames"} {
|
||||
if value, ok := source.Subject.Payload[key]; ok {
|
||||
target.Subject.Payload[key] = value
|
||||
} else {
|
||||
delete(target.Subject.Payload, key)
|
||||
}
|
||||
}
|
||||
}
|
||||
106
internal/api/dictionary_selections.go
Normal file
106
internal/api/dictionary_selections.go
Normal file
@ -0,0 +1,106 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
)
|
||||
|
||||
// Always resolve new selections on the server. Existing saved selections are
|
||||
// copied from the workspace/revision, so dictionary maintenance cannot rewrite
|
||||
// a submitted classification and a client cannot forge tags or another tenant.
|
||||
func (s *server) projectDictionarySelections(ctx context.Context, item *domain.HumanItemRecord, kind string) (map[string]domain.AnnotationLabelRecord, error) {
|
||||
selection, err := s.store.GetProjectAnnotationLabels(ctx, item.ProjectID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result := map[string]domain.AnnotationLabelRecord{}
|
||||
if selection == nil {
|
||||
return result, nil
|
||||
}
|
||||
if selection.OrganizationID != item.OrganizationID {
|
||||
return nil, fmt.Errorf("项目字典不属于当前组织")
|
||||
}
|
||||
ids := selection.LabelIDs
|
||||
if kind == "slice" {
|
||||
ids = selection.SliceLabelIDs
|
||||
}
|
||||
for _, id := range ids {
|
||||
label, err := s.store.GetAnnotationLabel(ctx, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if label != nil && label.OrganizationID == item.OrganizationID && dictionaryKind(label.Kind) == kind && label.Status == domain.AnnotationLabelActive {
|
||||
result[id] = *label
|
||||
}
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func sameDictionaryVersion(a, b *domain.DictionarySelection) bool {
|
||||
return a != nil && b != nil && a.ID == b.ID && a.Version == b.Version
|
||||
}
|
||||
|
||||
func (s *server) canonicalActionDictionaries(ctx context.Context, item *domain.HumanItemRecord, segments []domain.AnnotationSegment, mode ...string) []validationIssue {
|
||||
previous := decodeMaterializationValue[domain.AnnotationSegment](annotationWorkspacePayload(item)["segments"])
|
||||
if len(previous) == 0 && item.LegacyTaskID != "" {
|
||||
revisions, err := s.store.ListAnnotationRevisions(ctx, item.OrganizationID, item.LegacyTaskID)
|
||||
if err != nil {
|
||||
return []validationIssue{{Field: "segments", Message: "无法读取已保存的动作分类"}}
|
||||
}
|
||||
for _, rev := range revisions {
|
||||
if rev.Revision == item.CurrentRevision {
|
||||
previous = rev.Segments
|
||||
}
|
||||
}
|
||||
}
|
||||
if issues := rememberDictionarySelections(item, actionDictionarySelections(previous)...); len(issues) > 0 {
|
||||
return issues
|
||||
}
|
||||
old := map[string]domain.AnnotationSegment{}
|
||||
for _, seg := range previous {
|
||||
old[seg.ID] = seg
|
||||
}
|
||||
var allowed map[string]domain.AnnotationLabelRecord
|
||||
for i := range segments {
|
||||
seg := &segments[i]
|
||||
p := old[seg.ID]
|
||||
if seg.ClearDictionary {
|
||||
seg.Dictionary = nil
|
||||
seg.ClearDictionary = false
|
||||
continue
|
||||
}
|
||||
if len(mode) > 0 && mode[0] == "restore" && !seg.IsInvalid && strings.TrimSpace(seg.Description) != "" {
|
||||
if saved := savedDictionarySelection(item, seg.Dictionary, "action"); saved != nil {
|
||||
seg.Dictionary = saved
|
||||
continue
|
||||
}
|
||||
}
|
||||
if seg.IsInvalid || strings.TrimSpace(seg.Description) == "" {
|
||||
seg.Dictionary = nil
|
||||
continue
|
||||
}
|
||||
if p.Dictionary != nil && (sameDictionaryVersion(p.Dictionary, seg.Dictionary) || (seg.Dictionary == nil && seg.Description == p.Description && !p.IsInvalid)) {
|
||||
seg.Dictionary = p.Dictionary
|
||||
continue
|
||||
}
|
||||
if seg.Dictionary == nil {
|
||||
continue
|
||||
} // Legacy/free text is never guessed into a category.
|
||||
if allowed == nil {
|
||||
var err error
|
||||
allowed, err = s.projectDictionarySelections(ctx, item, "action")
|
||||
if err != nil {
|
||||
return []validationIssue{{Field: "segments", Message: "无法读取动作字典,请重试"}}
|
||||
}
|
||||
}
|
||||
label, ok := allowed[seg.Dictionary.ID]
|
||||
if !ok || label.Version != seg.Dictionary.Version {
|
||||
return []validationIssue{{Field: fmt.Sprintf("segments.%d.dictionary", i), Code: "dictionary_selection_stale", Message: "动作词条已变化,请从项目字典重新选择"}}
|
||||
}
|
||||
seg.Dictionary = label.Selection()
|
||||
}
|
||||
return rememberDictionarySelections(item, actionDictionarySelections(segments)...)
|
||||
}
|
||||
411
internal/api/dictionary_selections_test.go
Normal file
411
internal/api/dictionary_selections_test.go
Normal file
@ -0,0 +1,411 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
)
|
||||
|
||||
func TestDictionarySelectionsPersistThroughHTTPAndDictionaryMaintenance(t *testing.T) {
|
||||
f := newCombinedDraftFixture(t)
|
||||
labels := []domain.AnnotationLabelRecord{
|
||||
{ID: "base", OrganizationID: f.item.OrganizationID, Kind: "slice", Code: "base", Name: "物品操作", Tags: []string{"商品"}, Version: 2, Status: domain.AnnotationLabelActive},
|
||||
{ID: "action", OrganizationID: f.item.OrganizationID, Kind: "action", Code: "pick", Name: "拿取", Tags: []string{"抓取"}, Version: 3, Status: domain.AnnotationLabelActive},
|
||||
}
|
||||
for _, label := range labels {
|
||||
if err := f.store.UpsertAnnotationLabel(f.ctx, label); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
selection := domain.ProjectAnnotationLabelsRecord{ProjectID: f.task.ProjectID, OrganizationID: f.item.OrganizationID, LabelIDs: []string{"action"}, SliceLabelIDs: []string{"base"}}
|
||||
if err := f.store.UpsertProjectAnnotationLabels(f.ctx, selection); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.store.UpsertProjectSliceLabels(f.ctx, selection); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
forgedBase := *labels[0].Selection()
|
||||
forgedBase.Tags = []string{"FORGED"}
|
||||
forgedAction := *labels[1].Selection()
|
||||
forgedAction.Tags = []string{"FORGED"}
|
||||
payload := combinedAnnotationDraftInput{Version: f.item.Version, SliceSegments: []domain.SliceSegment{{ID: "s", Name: "物品操作", Start: 0, End: 10, Dictionary: &forgedBase}}, Segments: []domain.AnnotationSegment{{ID: "a", Start: 1, End: 2, Description: "拿起杯子", Dictionary: &forgedAction}}}
|
||||
endpoint := "/api/plugins/annotation-qa/tasks/" + f.task.ID + "/drafts"
|
||||
save := func(actor domain.UserRecord) *domain.HumanItemRecord {
|
||||
t.Helper()
|
||||
r := f.request(http.MethodPut, endpoint, actor, payload)
|
||||
if r.Code != 200 {
|
||||
t.Fatalf("save: %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
item, err := f.store.GetHumanItem(f.ctx, f.item.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return item
|
||||
}
|
||||
saved := save(f.annotator)
|
||||
draft := annotationWorkspacePayload(saved)
|
||||
slices := decodeMaterializationValue[domain.SliceSegment](draft["sliceSegments"])
|
||||
actions := decodeMaterializationValue[domain.AnnotationSegment](draft["segments"])
|
||||
if !reflect.DeepEqual(slices[0].Dictionary, labels[0].Selection()) || !reflect.DeepEqual(actions[0].Dictionary, labels[1].Selection()) {
|
||||
t.Fatal("server trusted forged tags")
|
||||
}
|
||||
// Empty action text never clears the independent base classification.
|
||||
payload.Version = saved.Version
|
||||
payload.SliceSegments = slices
|
||||
payload.Segments = nil
|
||||
saved = save(f.annotator)
|
||||
slices = decodeMaterializationValue[domain.SliceSegment](annotationWorkspacePayload(saved)["sliceSegments"])
|
||||
if slices[0].Dictionary == nil || slices[0].Dictionary.Tags[0] != "商品" {
|
||||
t.Fatal("blank description lost base class")
|
||||
}
|
||||
for i := range labels {
|
||||
labels[i].Version++
|
||||
labels[i].Name += "新版"
|
||||
labels[i].Tags = []string{"新标签"}
|
||||
labels[i].Status = domain.AnnotationLabelInactive
|
||||
f.store.UpsertAnnotationLabel(f.ctx, labels[i])
|
||||
}
|
||||
f.store.UpsertProjectSliceLabels(f.ctx, domain.ProjectAnnotationLabelsRecord{ProjectID: f.task.ProjectID, OrganizationID: f.item.OrganizationID})
|
||||
payload.Version = saved.Version
|
||||
payload.SliceSegments = slices
|
||||
saved = save(f.annotator)
|
||||
// A fresh handler recovers the frozen snapshot, independent of active dictionaries.
|
||||
f.handler = NewHandler(f.store, nil)
|
||||
payload.Version = saved.Version
|
||||
saved = save(f.annotator)
|
||||
slices = decodeMaterializationValue[domain.SliceSegment](annotationWorkspacePayload(saved)["sliceSegments"])
|
||||
if slices[0].Dictionary.Version != 2 || slices[0].Dictionary.Name != "物品操作" {
|
||||
t.Fatal("historical classification rewritten")
|
||||
}
|
||||
payload.Version = saved.Version
|
||||
payload.Segments = []domain.AnnotationSegment{{ID: "new-action", Start: 1, End: 2, Description: "拿取", Dictionary: &forgedAction}}
|
||||
if r := f.request(http.MethodPut, endpoint, f.annotator, payload); r.Code != 400 {
|
||||
t.Fatalf("stale/new action accepted %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
if r := f.request(http.MethodPut, endpoint, f.reviewer, payload); r.Code != 403 {
|
||||
t.Fatal("reviewer can change classification")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDictionaryTagsHaveIndependentRevisionAndAdminAuthorization(t *testing.T) {
|
||||
f := newCombinedDraftFixture(t)
|
||||
body := map[string]any{"kind": "slice", "name": "分拣", "code": "sort", "category": "business-category", "aliases": []string{"别名"}, "tags": []string{" 商品 ", "商品", "分拣"}}
|
||||
r := f.request("POST", "/api/annotation-labels", f.annotator, body)
|
||||
if r.Code != 403 {
|
||||
t.Fatal("annotator maintained dictionary")
|
||||
}
|
||||
r = f.request("POST", "/api/annotation-labels", f.admin, body)
|
||||
if r.Code != 201 {
|
||||
t.Fatalf("%d %s", r.Code, r.Body.String())
|
||||
}
|
||||
var label domain.AnnotationLabelRecord
|
||||
if err := json.Unmarshal(r.Body.Bytes(), &label); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reflect.DeepEqual(label.Tags, []string{"商品", "分拣"}) || label.Category != "business-category" || len(label.Aliases) != 1 {
|
||||
t.Fatal("tags confused with other fields")
|
||||
}
|
||||
body["tags"] = []string{}
|
||||
r = f.request("PATCH", "/api/annotation-labels/"+label.ID, f.admin, body)
|
||||
if r.Code != 200 {
|
||||
t.Fatalf("%d %s", r.Code, r.Body.String())
|
||||
}
|
||||
var updated domain.AnnotationLabelRecord
|
||||
json.Unmarshal(r.Body.Bytes(), &updated)
|
||||
if updated.Version != 2 || len(updated.Tags) != 0 || !reflect.DeepEqual(updated.History[0].Tags, label.Tags) {
|
||||
t.Fatal("tag history lost")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLegacyClassificationDoesNotGuessCurrentTagsAtConfirmation(t *testing.T) {
|
||||
f := newCombinedDraftFixture(t)
|
||||
legacy := domain.SliceSegment{ID: "legacy", Name: "物品操作", Start: 0, End: 10}
|
||||
f.item.Subject.Payload["annotationWorkspace"] = map[string]any{"sliceSegments": []domain.SliceSegment{legacy}}
|
||||
f.store.UpsertHumanItem(f.ctx, f.item)
|
||||
f.store.UpsertAnnotationLabel(f.ctx, domain.AnnotationLabelRecord{ID: "current", OrganizationID: f.item.OrganizationID, Kind: "slice", Code: "base", Name: legacy.Name, Tags: []string{"现在新增的标签"}, Version: 9, Status: domain.AnnotationLabelActive})
|
||||
f.store.UpsertProjectSliceLabels(f.ctx, domain.ProjectAnnotationLabelsRecord{ProjectID: f.task.ProjectID, OrganizationID: f.item.OrganizationID, SliceLabelIDs: []string{"current"}})
|
||||
server := &server{store: f.store}
|
||||
slices := []domain.SliceSegment{legacy}
|
||||
if issues := server.validateProjectSliceNames(f.ctx, &f.item, slices, "confirm"); len(issues) > 0 || slices[0].Dictionary != nil {
|
||||
t.Fatal("legacy tags guessed at confirm")
|
||||
}
|
||||
slices[0].Dictionary = &domain.DictionarySelection{ID: "current", Version: 9}
|
||||
if issues := server.validateProjectSliceNames(f.ctx, &f.item, slices, "confirm"); len(issues) > 0 || slices[0].Dictionary == nil || slices[0].Dictionary.Tags[0] != "现在新增的标签" {
|
||||
t.Fatal("explicit new selection did not freeze current tags")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSliceEditApplicationCanonicalizesAgainstStoredDictionaryState(t *testing.T) {
|
||||
for _, foreign := range []bool{false, true} {
|
||||
t.Run(map[bool]string{false: "forged-tags", true: "foreign-word"}[foreign], func(t *testing.T) {
|
||||
f := continuousFixture(t)
|
||||
selection, _ := f.store.GetProjectAnnotationLabels(f.ctx, f.task.ProjectID)
|
||||
label, _ := f.store.GetAnnotationLabel(f.ctx, selection.SliceLabelIDs[0])
|
||||
label.Tags = []string{"服务端标签"}
|
||||
label.Version = 2
|
||||
f.store.UpsertAnnotationLabel(f.ctx, *label)
|
||||
before := []domain.SliceSegment{{ID: "s", Name: label.Name, Start: 0, End: 10}}
|
||||
item := f.item
|
||||
item.Subject.Payload["sliceWorkflow"] = map[string]any{"status": "completed", "phase": "annotate"}
|
||||
item.Subject.Payload["annotationWorkspace"] = map[string]any{"sliceSegments": before, "segments": []domain.AnnotationSegment{}}
|
||||
f.store.UpsertHumanItem(f.ctx, item)
|
||||
incoming := *label.Selection()
|
||||
incoming.Tags = []string{"FORGED"}
|
||||
if foreign {
|
||||
incoming.ID = "foreign-label"
|
||||
}
|
||||
proposed := append([]domain.SliceSegment(nil), before...)
|
||||
proposed[0].Dictionary = &incoming
|
||||
body := sliceEditApplicationInput{ApplicationID: "classification-apply", combinedAnnotationDraftInput: combinedAnnotationDraftInput{Version: item.Version, SliceSegments: proposed}}
|
||||
r := f.request("POST", "/api/plugins/annotation-qa/tasks/"+f.task.ID+"/slice-edit-applications", f.annotator, body)
|
||||
if foreign {
|
||||
if r.Code != 400 {
|
||||
t.Fatalf("foreign classification accepted %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
return
|
||||
}
|
||||
if r.Code != 200 {
|
||||
t.Fatalf("%d %s", r.Code, r.Body.String())
|
||||
}
|
||||
saved, _ := f.store.GetHumanItem(f.ctx, item.ID)
|
||||
got := decodeMaterializationValue[domain.SliceSegment](annotationWorkspacePayload(saved)["sliceSegments"])
|
||||
if got[0].Dictionary == nil || !reflect.DeepEqual(got[0].Dictionary.Tags, label.Tags) {
|
||||
t.Fatal("client classification snapshot trusted")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestProjectActionDictionaryReturnsVersionedItemsForAnnotator(t *testing.T) {
|
||||
f := newCombinedDraftFixture(t)
|
||||
label := domain.AnnotationLabelRecord{ID: "action-word", OrganizationID: f.item.OrganizationID, Code: "pick", Name: "拿取", Kind: "action", Version: 3, Tags: []string{"抓取"}, Status: domain.AnnotationLabelActive}
|
||||
f.store.UpsertAnnotationLabel(f.ctx, label)
|
||||
f.store.UpsertProjectAnnotationLabels(f.ctx, domain.ProjectAnnotationLabelsRecord{ID: "selection", ProjectID: f.task.ProjectID, OrganizationID: f.item.OrganizationID, LabelIDs: []string{label.ID}})
|
||||
r := f.request("GET", "/api/projects/"+f.task.ProjectID+"/annotation-labels", f.annotator, nil)
|
||||
var response struct {
|
||||
ID string `json:"id"`
|
||||
LabelIDs []string `json:"labelIds"`
|
||||
Items []domain.AnnotationLabelRecord `json:"items"`
|
||||
}
|
||||
if json.Unmarshal(r.Body.Bytes(), &response) != nil || r.Code != 200 || response.ID != "selection" || len(response.LabelIDs) != 1 || len(response.Items) != 1 || response.Items[0].Version != 3 || response.Items[0].Tags[0] != "抓取" {
|
||||
t.Fatalf("%d %s", r.Code, r.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestUndoRestoresOnlyServerRememberedClassificationsAfterDictionaryMaintenance(t *testing.T) {
|
||||
f := continuousFixture(t)
|
||||
srv := &server{store: f.store}
|
||||
base := domain.AnnotationLabelRecord{ID: "base-old", OrganizationID: f.item.OrganizationID, Kind: "slice", Code: "base", Name: "旧基础分类", Version: 2, Tags: []string{"原标签"}, Status: domain.AnnotationLabelActive}
|
||||
action := domain.AnnotationLabelRecord{ID: "action-old", OrganizationID: f.item.OrganizationID, Kind: "action", Code: "pick", Name: "旧动作分类", Version: 3, Tags: []string{"原动作标签"}, Status: domain.AnnotationLabelActive}
|
||||
f.store.UpsertAnnotationLabel(f.ctx, base)
|
||||
f.store.UpsertAnnotationLabel(f.ctx, action)
|
||||
selection := domain.ProjectAnnotationLabelsRecord{ProjectID: f.task.ProjectID, OrganizationID: f.item.OrganizationID, LabelIDs: []string{action.ID}, SliceLabelIDs: []string{base.ID}}
|
||||
f.store.UpsertProjectAnnotationLabels(f.ctx, selection)
|
||||
f.store.UpsertProjectSliceLabels(f.ctx, selection)
|
||||
oldSlices := []domain.SliceSegment{{ID: "s", Name: base.Name, Start: 0, End: 10, Dictionary: base.Selection()}}
|
||||
oldActions := []domain.AnnotationSegment{{ID: "a", Description: "原描述", Start: 0, End: 10, Status: "described", Dictionary: action.Selection()}}
|
||||
item := f.item
|
||||
if issues := srv.validateProjectSliceNames(f.ctx, &item, oldSlices, "draft"); len(issues) > 0 {
|
||||
t.Fatal(issues)
|
||||
}
|
||||
if issues := srv.canonicalActionDictionaries(f.ctx, &item, oldActions); len(issues) > 0 {
|
||||
t.Fatal(issues)
|
||||
}
|
||||
item.Subject.Payload["sliceWorkflow"] = map[string]any{"phase": "annotate", "status": "completed"}
|
||||
// Later editing selected a different classification. The word is subsequently
|
||||
// renamed, retagged, disabled and unlinked by an administrator.
|
||||
item.Subject.Payload["annotationWorkspace"] = map[string]any{"sliceSegments": []domain.SliceSegment{{ID: "s", Name: "另一个分类", Start: 0, End: 10}}, "segments": []domain.AnnotationSegment{{ID: "a", Description: "修改后描述", Start: 0, End: 10, Status: "described"}}}
|
||||
f.store.UpsertHumanItem(f.ctx, item)
|
||||
base.Name = "新版"
|
||||
base.Tags = []string{"新标签"}
|
||||
base.Version++
|
||||
base.Status = domain.AnnotationLabelInactive
|
||||
f.store.UpsertAnnotationLabel(f.ctx, base)
|
||||
action.Version++
|
||||
action.Status = domain.AnnotationLabelInactive
|
||||
f.store.UpsertAnnotationLabel(f.ctx, action)
|
||||
f.store.UpsertProjectSliceLabels(f.ctx, domain.ProjectAnnotationLabelsRecord{ProjectID: f.task.ProjectID, OrganizationID: f.item.OrganizationID})
|
||||
f.store.UpsertProjectAnnotationLabels(f.ctx, domain.ProjectAnnotationLabelsRecord{ProjectID: f.task.ProjectID, OrganizationID: f.item.OrganizationID})
|
||||
// Simulate serialization and restart; do not use process-local snapshot state.
|
||||
f.handler = NewHandler(f.store, nil)
|
||||
oldSlices[0].Dictionary.Tags = []string{"FORGED"}
|
||||
oldActions[0].Dictionary.Tags = []string{"FORGED"}
|
||||
body := combinedAnnotationDraftInput{Version: item.Version, SliceSegments: oldSlices, Segments: oldActions}
|
||||
r := f.request("POST", "/api/plugins/annotation-qa/tasks/"+f.task.ID+"/slice-edit-restorations", f.annotator, body)
|
||||
if r.Code != 200 {
|
||||
t.Fatalf("undo %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
saved, _ := f.store.GetHumanItem(f.ctx, item.ID)
|
||||
slices := decodeMaterializationValue[domain.SliceSegment](annotationWorkspacePayload(saved)["sliceSegments"])
|
||||
actions := decodeMaterializationValue[domain.AnnotationSegment](annotationWorkspacePayload(saved)["segments"])
|
||||
if slices[0].Dictionary.Version != 2 || slices[0].Dictionary.Tags[0] != "原标签" || actions[0].Dictionary.Version != 3 || actions[0].Dictionary.Tags[0] != "原动作标签" {
|
||||
t.Fatal("undo lost or forged frozen classification")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDirectDispositionCannotPersistForeignOrForgedSliceDictionary(t *testing.T) {
|
||||
for _, foreign := range []bool{false, true} {
|
||||
t.Run(map[bool]string{false: "forged-tags", true: "foreign-word"}[foreign], func(t *testing.T) {
|
||||
f := continuousFixture(t)
|
||||
body := dispositionFixturePayload(t, f)
|
||||
selection, _ := f.store.GetProjectAnnotationLabels(f.ctx, f.task.ProjectID)
|
||||
label, _ := f.store.GetAnnotationLabel(f.ctx, selection.SliceLabelIDs[0])
|
||||
label.Tags = []string{"服务端标签"}
|
||||
label.Version = 2
|
||||
f.store.UpsertAnnotationLabel(f.ctx, *label)
|
||||
body.SliceSegments[0].Name = label.Name
|
||||
d := *label.Selection()
|
||||
d.Tags = []string{"FORGED"}
|
||||
if foreign {
|
||||
d.ID = "foreign"
|
||||
}
|
||||
body.SliceSegments[0].Dictionary = &d
|
||||
r := f.request("POST", "/api/plugins/annotation-qa/tasks/"+f.task.ID+"/slice-dispositions", f.annotator, body)
|
||||
if foreign {
|
||||
if r.Code != 400 {
|
||||
t.Fatalf("foreign dictionary saved %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
return
|
||||
}
|
||||
if r.Code != 200 {
|
||||
t.Fatalf("%d %s", r.Code, r.Body.String())
|
||||
}
|
||||
item, _ := f.store.GetHumanItem(f.ctx, f.item.ID)
|
||||
saved := decodeMaterializationValue[domain.SliceSegment](annotationWorkspacePayload(item)["sliceSegments"])
|
||||
if saved[0].Dictionary == nil || !reflect.DeepEqual(saved[0].Dictionary.Tags, label.Tags) {
|
||||
t.Fatal("forged tags persisted through direct disposition")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDispositionUndoRedoPreservesClassificationsAfterDictionaryRemoval(t *testing.T) {
|
||||
for _, application := range []bool{false, true} {
|
||||
t.Run(map[bool]string{false: "direct", true: "panel-application"}[application], func(t *testing.T) {
|
||||
f := continuousFixture(t)
|
||||
body := dispositionFixturePayload(t, f)
|
||||
selection, _ := f.store.GetProjectAnnotationLabels(f.ctx, f.task.ProjectID)
|
||||
base, _ := f.store.GetAnnotationLabel(f.ctx, selection.SliceLabelIDs[0])
|
||||
base.Tags = []string{"基础标签"}
|
||||
base.Version = 2
|
||||
f.store.UpsertAnnotationLabel(f.ctx, *base)
|
||||
reason, _ := f.store.GetAnnotationLabel(f.ctx, body.ReasonLabelID)
|
||||
reason.Tags = []string{"舍弃标签"}
|
||||
f.store.UpsertAnnotationLabel(f.ctx, *reason)
|
||||
for i := range body.SliceSegments {
|
||||
body.SliceSegments[i].Name = base.Name
|
||||
body.SliceSegments[i].Dictionary = base.Selection()
|
||||
}
|
||||
original := append([]domain.SliceSegment(nil), body.SliceSegments...)
|
||||
endpoint := "/api/plugins/annotation-qa/tasks/" + f.task.ID
|
||||
r := f.request("POST", endpoint+"/slice-dispositions", f.annotator, body)
|
||||
if r.Code != 200 {
|
||||
t.Fatalf("discard %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
var first struct {
|
||||
Item domain.HumanItemRecord `json:"item"`
|
||||
Workspace combinedAnnotationDraftInput `json:"workspace"`
|
||||
}
|
||||
json.Unmarshal(r.Body.Bytes(), &first)
|
||||
base.Name = "新名字"
|
||||
base.Version++
|
||||
base.Tags = []string{"新标签"}
|
||||
base.Status = domain.AnnotationLabelInactive
|
||||
f.store.UpsertAnnotationLabel(f.ctx, *base)
|
||||
reason.Version++
|
||||
reason.Status = domain.AnnotationLabelInactive
|
||||
f.store.UpsertAnnotationLabel(f.ctx, *reason)
|
||||
f.store.UpsertProjectSliceLabels(f.ctx, domain.ProjectAnnotationLabelsRecord{ProjectID: f.task.ProjectID, OrganizationID: f.item.OrganizationID})
|
||||
f.store.UpsertProjectDiscardReasonLabels(f.ctx, domain.ProjectAnnotationLabelsRecord{ProjectID: f.task.ProjectID, OrganizationID: f.item.OrganizationID})
|
||||
item, _ := f.store.GetHumanItem(f.ctx, f.item.ID)
|
||||
item.Subject.Payload["sliceWorkflow"] = map[string]any{"phase": "annotate", "status": "completed"}
|
||||
f.store.UpsertHumanItem(f.ctx, *item)
|
||||
f.handler = NewHandler(f.store, nil)
|
||||
if application {
|
||||
input := sliceEditApplicationInput{ApplicationID: "undo-discard", RestoreClassifications: true, combinedAnnotationDraftInput: combinedAnnotationDraftInput{Version: item.Version, SliceSegments: original}, Dispositions: []sliceDispositionInput{{SliceID: body.SliceID, Disposition: "keep"}}}
|
||||
r = f.request("POST", endpoint+"/slice-edit-applications", f.annotator, input)
|
||||
} else {
|
||||
undo := body
|
||||
undo.combinedAnnotationDraftInput = first.Workspace
|
||||
undo.Version = item.Version
|
||||
undo.Disposition = "keep"
|
||||
undo.RestoreClassifications = true
|
||||
undo.RestorePreviousName = true
|
||||
undo.RestoreNameFromVersion = first.Item.Version
|
||||
r = f.request("POST", endpoint+"/slice-dispositions", f.annotator, undo)
|
||||
}
|
||||
if r.Code != 200 {
|
||||
t.Fatalf("undo %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
var restored struct {
|
||||
Item domain.HumanItemRecord `json:"item"`
|
||||
Workspace combinedAnnotationDraftInput `json:"workspace"`
|
||||
}
|
||||
json.Unmarshal(r.Body.Bytes(), &restored)
|
||||
d := restored.Workspace.SliceSegments[1].Dictionary
|
||||
if d == nil || d.Version != 2 || d.Tags[0] != "基础标签" {
|
||||
t.Fatal("undo lost frozen base category")
|
||||
}
|
||||
if application {
|
||||
input := sliceEditApplicationInput{ApplicationID: "redo-discard", RestoreClassifications: true, combinedAnnotationDraftInput: combinedAnnotationDraftInput{Version: restored.Item.Version, SliceSegments: first.Workspace.SliceSegments}, Dispositions: []sliceDispositionInput{{SliceID: body.SliceID, Disposition: "discard", ReasonLabelID: body.ReasonLabelID, ReasonVersion: body.ReasonVersion}}}
|
||||
r = f.request("POST", endpoint+"/slice-edit-applications", f.annotator, input)
|
||||
} else {
|
||||
redo := body
|
||||
redo.combinedAnnotationDraftInput = restored.Workspace
|
||||
redo.Version = restored.Item.Version
|
||||
redo.RestoreClassifications = true
|
||||
r = f.request("POST", endpoint+"/slice-dispositions", f.annotator, redo)
|
||||
}
|
||||
if r.Code != 200 {
|
||||
t.Fatalf("redo %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
var redone struct {
|
||||
Workspace combinedAnnotationDraftInput `json:"workspace"`
|
||||
}
|
||||
json.Unmarshal(r.Body.Bytes(), &redone)
|
||||
d = redone.Workspace.SliceSegments[1].DiscardReasonDictionary
|
||||
if d == nil || d.Version != 1 || d.Tags[0] != "舍弃标签" {
|
||||
t.Fatal("redo lost frozen discard reason")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnconfirmedRestoredNamesCannotBecomeFinalClassification(t *testing.T) {
|
||||
f := newCombinedDraftFixture(t)
|
||||
srv := &server{store: f.store}
|
||||
slices := []domain.SliceSegment{{ID: "s", Name: "未选择字典的临时名称", Start: 0, End: 10}}
|
||||
if issues := srv.validateProjectSliceNames(f.ctx, &f.item, slices, "restore"); len(issues) > 0 {
|
||||
t.Fatal(issues)
|
||||
}
|
||||
f.item.Subject.Payload["annotationWorkspace"] = map[string]any{"sliceSegments": slices}
|
||||
for _, mode := range []string{"confirm", "submit"} {
|
||||
if issues := srv.validateProjectSliceNames(f.ctx, &f.item, slices, mode); len(issues) == 0 {
|
||||
t.Fatalf("unfinished name bypassed %s", mode)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLegacyDispositionCannotSeedDictionaryRecoveryHistory(t *testing.T) {
|
||||
f := newCombinedDraftFixture(t)
|
||||
f.task.TemplateVersion = 5
|
||||
f.store.UpsertAnnotationTask(f.ctx, f.task)
|
||||
body := dispositionFixturePayload(t, f)
|
||||
body.Segments[0].Handedness = "both"
|
||||
body.SliceSegments[0].Name = "原有名称"
|
||||
body.SliceSegments[0].TaskLabelID = "legacy-task-label"
|
||||
body.SliceSegments[0].TaskLabel = "原有任务分类"
|
||||
body.SliceSegments[0].Dictionary = &domain.DictionarySelection{ID: "FORGED", Kind: "action", Version: 9, Name: "伪造分类", Code: "forged", Tags: []string{"FORGED"}}
|
||||
r := f.request("POST", "/api/plugins/annotation-qa/tasks/"+f.task.ID+"/slice-dispositions", f.annotator, body)
|
||||
if r.Code != 200 {
|
||||
t.Fatalf("legacy save %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
saved, _ := f.store.GetHumanItem(f.ctx, f.item.ID)
|
||||
slices := decodeMaterializationValue[domain.SliceSegment](annotationWorkspacePayload(saved)["sliceSegments"])
|
||||
if slices[0].Dictionary != nil || slices[0].Name != "原有名称" || slices[0].TaskLabelID != "legacy-task-label" || savedDictionarySelection(saved, &domain.DictionarySelection{ID: "FORGED", Version: 9}, "action") != nil {
|
||||
t.Fatal("legacy task seeded a dictionary snapshot or lost legacy labels")
|
||||
}
|
||||
}
|
||||
@ -11,6 +11,19 @@ func (s *server) resolveDiscardReason(ctx context.Context, item *domain.HumanIte
|
||||
issue := func(message string) (*domain.AnnotationLabelRecord, []validationIssue) {
|
||||
return nil, []validationIssue{{Field: "reasonLabelId", Code: "discard_reason_dictionary_required", Message: message}}
|
||||
}
|
||||
if body.RestoreClassifications {
|
||||
if prior := historicalDiscardReason(item, body); prior != nil {
|
||||
if prior.DiscardReasonDictionary != nil {
|
||||
d := prior.DiscardReasonDictionary
|
||||
return &domain.AnnotationLabelRecord{ID: d.ID, Kind: d.Kind, Code: d.Code, Name: d.Name, Version: d.Version, Tags: append([]string{}, d.Tags...), OrganizationID: item.OrganizationID}, nil
|
||||
}
|
||||
if prior.DiscardReasonLabelID == "" {
|
||||
return nil, nil
|
||||
}
|
||||
return &domain.AnnotationLabelRecord{ID: prior.DiscardReasonLabelID, Kind: "discard_reason", Name: prior.DiscardReasonName, Version: prior.DiscardReasonVersion, OrganizationID: item.OrganizationID}, nil
|
||||
}
|
||||
return issue("未找到当前任务已保存的舍弃原因,无法恢复该历史选择")
|
||||
}
|
||||
selection, err := s.store.GetProjectAnnotationLabels(ctx, item.ProjectID)
|
||||
if err != nil {
|
||||
return issue("舍弃原因字典加载失败,请重试")
|
||||
@ -73,5 +86,26 @@ func describeSliceNameIssues(issues []validationIssue, slices []domain.SliceSegm
|
||||
}
|
||||
|
||||
func discardReasonEvidence(slice domain.SliceSegment) map[string]any {
|
||||
return map[string]any{"reason": slice.DiscardReason, "reasonLabelId": slice.DiscardReasonLabelID, "reasonVersion": slice.DiscardReasonVersion, "reasonName": slice.DiscardReasonName, "reasonNote": slice.DiscardReasonNote, "at": slice.DiscardedAt, "previousAnnotations": slice.DiscardedAnnotations}
|
||||
return map[string]any{"dictionary": slice.DiscardReasonDictionary, "reason": slice.DiscardReason, "reasonLabelId": slice.DiscardReasonLabelID, "reasonVersion": slice.DiscardReasonVersion, "reasonName": slice.DiscardReasonName, "reasonNote": slice.DiscardReasonNote, "at": slice.DiscardedAt, "previousAnnotations": slice.DiscardedAnnotations}
|
||||
}
|
||||
|
||||
// Restore only a reason/note combination actually recorded for this slice;
|
||||
// administrator changes never let a client invent historical reason metadata.
|
||||
func historicalDiscardReason(item *domain.HumanItemRecord, body sliceDispositionInput) *domain.SliceSegment {
|
||||
history, _ := normalizeStructuredValue(item.Subject.Payload["sliceDispositionHistory"]).([]any)
|
||||
for i := len(history) - 1; i >= 0; i-- {
|
||||
entry := mapValue(history[i])
|
||||
if stringValue(entry["sliceId"]) != body.SliceID {
|
||||
continue
|
||||
}
|
||||
after := decodeMaterializationValue[domain.SliceSegment]([]any{entry["after"]})
|
||||
if len(after) != 1 {
|
||||
continue
|
||||
}
|
||||
saved := after[0]
|
||||
if saved.IsDiscarded() && saved.DiscardReasonLabelID == body.ReasonLabelID && saved.DiscardReasonVersion == body.ReasonVersion && saved.DiscardReasonNote == strings.TrimSpace(body.ReasonNote) && (saved.DiscardReasonLabelID != "" || saved.DiscardReason == strings.TrimSpace(body.Reason)) {
|
||||
return &saved
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@ -30,6 +30,7 @@ func TestDiscardReasonDictionaryValidationAndSnapshots(t *testing.T) {
|
||||
case "other_without_note":
|
||||
label.RequiresNote = true
|
||||
case "success":
|
||||
label.Tags = []string{"不可用"}
|
||||
body.Reason = "forged"
|
||||
body.ReasonNote = "遮挡持续两秒"
|
||||
}
|
||||
@ -54,6 +55,9 @@ func TestDiscardReasonDictionaryValidationAndSnapshots(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
slice := got.Workspace.SliceSegments[1]
|
||||
if slice.DiscardReasonDictionary == nil || len(slice.DiscardReasonDictionary.Tags) != 1 || slice.DiscardReasonDictionary.Tags[0] != "不可用" {
|
||||
t.Fatal("discard tags missing")
|
||||
}
|
||||
if slice.DiscardReasonName != "相机被遮挡" || slice.DiscardReasonVersion != 1 || slice.DiscardReasonLabelID != label.ID || slice.DiscardReasonNote != body.ReasonNote || strings.Contains(slice.DiscardReason, "forged") {
|
||||
t.Fatalf("snapshot %+v", slice)
|
||||
}
|
||||
|
||||
@ -948,9 +948,9 @@ func (server *server) materializeCombinedSliceSubmission(ctx context.Context, ta
|
||||
}
|
||||
relativeStart := segmentStart - slice.Start
|
||||
relativeEnd := segmentEnd - slice.Start
|
||||
actions = append(actions, map[string]any{"id": segment.ID, "status": effectiveActionStatus(segment), "isInvalid": segment.IsInvalid, "description": segment.Description, "descriptionTemplateId": segment.LabelID, "phrases": segment.Phrases, "source": segment.Source, "confidence": segment.Confidence, "clipStartMs": relativeStart * 1000, "clipEndMs": relativeEnd * 1000, "startTimestampUs": mediaSecondToTimestamp(segmentStart), "endTimestampUs": mediaSecondToTimestamp(segmentEnd)})
|
||||
actions = append(actions, map[string]any{"id": segment.ID, "status": effectiveActionStatus(segment), "isInvalid": segment.IsInvalid, "description": segment.Description, "descriptionTemplateId": segment.LabelID, "dictionary": segment.Dictionary, "phrases": segment.Phrases, "source": segment.Source, "confidence": segment.Confidence, "clipStartMs": relativeStart * 1000, "clipEndMs": relativeEnd * 1000, "startTimestampUs": mediaSecondToTimestamp(segmentStart), "endTimestampUs": mediaSecondToTimestamp(segmentEnd)})
|
||||
}
|
||||
annotation := map[string]any{"schemaVersion": "droboticflow.timestamped-description/v3", "boundaryContract": "hardware-frame-v1", "sliceId": sliceID, "sliceName": slice.Name, "window": map[string]any{"startTimestampUs": startUS, "endTimestampUs": endUS, "semantics": "half-open"}, "descriptions": actions, "actions": actions}
|
||||
annotation := map[string]any{"schemaVersion": "droboticflow.timestamped-description/v3", "boundaryContract": "hardware-frame-v1", "sliceId": sliceID, "sliceName": slice.Name, "dictionary": slice.Dictionary, "window": map[string]any{"startTimestampUs": startUS, "endTimestampUs": endUS, "semantics": "half-open"}, "descriptions": actions, "actions": actions}
|
||||
annotationSHA, annotationBytes, err := writeJSONFile(filepath.Join(directory, "annotation.json"), annotation)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@ -968,7 +968,7 @@ func (server *server) materializeCombinedSliceSubmission(ctx context.Context, ta
|
||||
boundaryTimes = append(boundaryTimes, grid.Seconds[k]-slice.Start)
|
||||
rightBoundaryTimes = append(rightBoundaryTimes, float64(grid.RightFrames[k]-rightStart)*rightMediaDuration/float64(len(rightPTS)))
|
||||
}
|
||||
synchronization := map[string]any{"schemaVersion": "droboticflow.synchronized-slice/v1", "collectionId": collectionID, "sliceIndex": index + 1, "sliceCount": len(normalizedSlices), "sliceName": slice.Name, "window": map[string]any{"startTimestampUs": startUS, "endTimestampUs": endUS, "semantics": "half-open"}, "commonRange": map[string]any{"startTimestampUs": commonStart, "endTimestampUs": commonEnd, "startSecond": commonMediaStart, "endSecond": commonMediaEnd}, "coverage": map[string]any{"mode": "full_common_range", "contiguous": true, "previousEndTimestampUs": func() any {
|
||||
synchronization := map[string]any{"schemaVersion": "droboticflow.synchronized-slice/v1", "collectionId": collectionID, "sliceIndex": index + 1, "sliceCount": len(normalizedSlices), "sliceName": slice.Name, "dictionary": slice.Dictionary, "window": map[string]any{"startTimestampUs": startUS, "endTimestampUs": endUS, "semantics": "half-open"}, "commonRange": map[string]any{"startTimestampUs": commonStart, "endTimestampUs": commonEnd, "startSecond": commonMediaStart, "endSecond": commonMediaEnd}, "coverage": map[string]any{"mode": "full_common_range", "contiguous": true, "previousEndTimestampUs": func() any {
|
||||
if index == 0 {
|
||||
return nil
|
||||
}
|
||||
@ -991,7 +991,7 @@ func (server *server) materializeCombinedSliceSubmission(ctx context.Context, ta
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
manifestItems = append(manifestItems, map[string]any{"sliceId": sliceID, "sourceSliceId": slice.ID, "sliceName": slice.Name, "startSecond": slice.Start, "endSecond": slice.End, "startTimestampUs": startUS, "endTimestampUs": endUS, "directory": filepath.ToSlash(filepath.Join("slices", sliceID)), "files": map[string]any{"leftVideo": leftVideo, "rightVideo": rightVideo, "leftPts": leftManifest, "rightPts": rightManifest, "imu": imuManifest, "annotation": map[string]any{"name": "annotation.json", "sha256": annotationSHA, "bytes": annotationBytes}, "synchronization": map[string]any{"name": "synchronized-slice.json", "sha256": syncSHA, "bytes": syncBytes}}})
|
||||
manifestItems = append(manifestItems, map[string]any{"sliceId": sliceID, "sourceSliceId": slice.ID, "sliceName": slice.Name, "dictionary": slice.Dictionary, "startSecond": slice.Start, "endSecond": slice.End, "startTimestampUs": startUS, "endTimestampUs": endUS, "directory": filepath.ToSlash(filepath.Join("slices", sliceID)), "files": map[string]any{"leftVideo": leftVideo, "rightVideo": rightVideo, "leftPts": leftManifest, "rightPts": rightManifest, "imu": imuManifest, "annotation": map[string]any{"name": "annotation.json", "sha256": annotationSHA, "bytes": annotationBytes}, "synchronization": map[string]any{"name": "synchronized-slice.json", "sha256": syncSHA, "bytes": syncBytes}}})
|
||||
manifestItems[len(manifestItems)-1]["deliveryDisposition"] = sliceDisposition(slice)
|
||||
manifestItems[len(manifestItems)-1]["mediaKey"] = mediaKey
|
||||
if slice.IsDiscarded() {
|
||||
@ -1005,15 +1005,23 @@ func (server *server) materializeCombinedSliceSubmission(ctx context.Context, ta
|
||||
}
|
||||
coverage := map[string]any{"mode": "full_common_range", "contiguous": true, "gapMicroseconds": 0, "overlapMicroseconds": 0, "startSecond": commonMediaStart, "endSecond": commonMediaEnd, "durationSeconds": commonMediaEnd - commonMediaStart, "startTimestampUs": commonStart, "endTimestampUs": commonEnd, "ptsToMediaScale": ptsToMediaScale, "sliceCount": len(manifestItems), "reconstruction": map[string]any{"leftFrameCount": deliveredLeftFrames, "rightFrameCount": deliveredRightFrames, "imuRowCount": deliveredIMURows}}
|
||||
captureEvidence := map[string]any{}
|
||||
var originalSource map[string]any
|
||||
if capture, captureErr := server.store.GetCaptureRecordByRunID(ctx, task.RunID); captureErr != nil {
|
||||
return nil, captureErr
|
||||
} else if capture != nil {
|
||||
originalSource, err = annotationRevisionOriginalSource(run, capture)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
captureEvidence = map[string]any{
|
||||
"captureRecordId": capture.ID, "captureId": capture.CaptureID, "prefix": capture.Prefix,
|
||||
"fingerprint": capture.Fingerprint, "contractRevision": capture.ContractRevision,
|
||||
}
|
||||
}
|
||||
revisionManifest := map[string]any{"schemaVersion": "droboticflow.slice-materialization/v2", "revision": revision, "humanItemId": item.ID, "taskId": task.ID, "runId": task.RunID, "collectionId": collectionID, "capture": captureEvidence, "status": "materialized", "sliceCount": len(manifestItems), "coverage": coverage, "items": manifestItems}
|
||||
if originalSource != nil {
|
||||
revisionManifest["originalSource"] = originalSource
|
||||
}
|
||||
deliverySelection := map[string]any{"keptSliceCount": keptSliceCount, "discardedSliceCount": discardedSliceCount, "keptDurationSeconds": keptDuration, "discardedDurationSeconds": discardedDuration}
|
||||
if discardedSliceCount > 0 {
|
||||
revisionManifest["schemaVersion"] = "droboticflow.slice-materialization/v3"
|
||||
|
||||
@ -89,16 +89,25 @@ func TestReworkMongoHTTPRepeatedReturnAcrossRestart(t *testing.T) {
|
||||
batch, e := f.store.GetHumanBatch(ctx, f.item.BatchID)
|
||||
must(e)
|
||||
seed("humanBatches", batch)
|
||||
f.item.Subject.Payload[dictionarySelectionHistoryKey] = []domain.DictionarySelection{{ID: "historical-base", Kind: "slice", Version: 2, Code: "sort", Name: "物品操作", Tags: []string{"商品"}}}
|
||||
seed("humanItems", f.item)
|
||||
seed("annotationTasks", f.task)
|
||||
slices, e := f.store.ListSliceRevisions(ctx, f.item.OrganizationID, f.task.ID)
|
||||
must(e)
|
||||
baseClass := &domain.DictionarySelection{ID: "historical-base", Kind: "slice", Version: 2, Code: "sort", Name: "物品操作", Tags: []string{"商品"}}
|
||||
actionClass := &domain.DictionarySelection{ID: "historical-action", Kind: "action", Version: 3, Code: "pick", Name: "拿取", Tags: []string{"抓取"}}
|
||||
for _, r := range slices {
|
||||
for i := range r.Segments {
|
||||
r.Segments[i].Dictionary = baseClass
|
||||
}
|
||||
seed("sliceRevisions", r)
|
||||
}
|
||||
actions, e := f.store.ListAnnotationRevisions(ctx, f.item.OrganizationID, f.task.ID)
|
||||
must(e)
|
||||
for _, r := range actions {
|
||||
for i := range r.Segments {
|
||||
r.Segments[i].Dictionary = actionClass
|
||||
}
|
||||
seed("annotationRevisions", r)
|
||||
}
|
||||
f.handler = NewHandler(ds, nil)
|
||||
@ -128,6 +137,14 @@ func TestReworkMongoHTTPRepeatedReturnAcrossRestart(t *testing.T) {
|
||||
if w.Code != 200 || len(out.Items) != 1 || out.Items[0].AssignedUserID != f.annotator.ID {
|
||||
t.Fatalf("persisted return missing from author pool: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
if snap := savedDictionarySelection(&out.Items[0], &domain.DictionarySelection{ID: "historical-base", Version: 2}, "slice"); snap == nil || snap.Tags[0] != "商品" {
|
||||
t.Fatal("classification recovery ledger lost across Mongo restart")
|
||||
}
|
||||
baseline, e := ds.GetReworkBaseline(ctx, store.ReworkBaselineID(item.ID, out.Items[0].ReviewRound))
|
||||
must(e)
|
||||
if baseline == nil || len(baseline.Slices) == 0 || baseline.Slices[0].Dictionary == nil || baseline.Slices[0].Dictionary.Version != 2 || baseline.Slices[0].Dictionary.Tags[0] != "商品" {
|
||||
t.Fatal("base classification lost across Mongo review/restart")
|
||||
}
|
||||
t.Logf("%s: round %d persisted and visible after restart", operation, out.Items[0].ReviewRound)
|
||||
}
|
||||
item, e := ds.GetHumanItem(ctx, f.item.ID)
|
||||
|
||||
@ -7,8 +7,8 @@ import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Unnamed drafts are allowed, but new confirmations need project dictionary
|
||||
// names. Unchanged historical names remain valid during annotation/review.
|
||||
// Binding happens before persistence, and never at delivery time. Name-only
|
||||
// historical records remain name-only until a new selection is made.
|
||||
func (s *server) validateProjectSliceNames(ctx context.Context, item *domain.HumanItemRecord, slices []domain.SliceSegment, mode string) []validationIssue {
|
||||
if item == nil {
|
||||
return nil
|
||||
@ -18,61 +18,93 @@ func (s *server) validateProjectSliceNames(ctx context.Context, item *domain.Hum
|
||||
return []validationIssue{{Field: "sliceSegments", Message: "无法读取切片字典规则"}}
|
||||
}
|
||||
if task == nil || task.TemplateVersion < 6 {
|
||||
// Older templates use taskLabelId/taskLabel; they cannot introduce the
|
||||
// independent v6 slice dictionary field, including through undo/discard.
|
||||
for i := range slices {
|
||||
slices[i].Dictionary = nil
|
||||
}
|
||||
return nil
|
||||
}
|
||||
previous := decodeMaterializationValue[domain.SliceSegment](annotationWorkspacePayload(item)["sliceSegments"])
|
||||
if len(previous) == 0 {
|
||||
revisions, e := s.store.ListSliceRevisions(ctx, item.OrganizationID, item.LegacyTaskID)
|
||||
if e != nil {
|
||||
revisions, err := s.store.ListSliceRevisions(ctx, item.OrganizationID, item.LegacyTaskID)
|
||||
if err != nil {
|
||||
return []validationIssue{{Field: "sliceSegments", Message: "无法读取历史切片名称"}}
|
||||
}
|
||||
if len(revisions) > 0 {
|
||||
previous = revisions[len(revisions)-1].Segments
|
||||
for _, rev := range revisions {
|
||||
if rev.Revision == item.CurrentRevision || len(previous) == 0 {
|
||||
previous = rev.Segments
|
||||
}
|
||||
}
|
||||
}
|
||||
old := map[string]string{}
|
||||
if issues := rememberDictionarySelections(item, sliceDictionarySelections(previous)...); len(issues) > 0 {
|
||||
return issues
|
||||
}
|
||||
old := map[string]domain.SliceSegment{}
|
||||
for _, p := range previous {
|
||||
old[p.ID] = p.Name
|
||||
old[p.ID] = p
|
||||
}
|
||||
needs := []int{}
|
||||
for i, c := range slices {
|
||||
if c.IsDiscarded() {
|
||||
continue
|
||||
}
|
||||
if mode != "confirm" && strings.TrimSpace(c.Name) != "" {
|
||||
if name, ok := old[c.ID]; ok && name == c.Name {
|
||||
unconfirmed := mapValue(item.Subject.Payload["unconfirmedSliceNames"])
|
||||
if unconfirmed == nil {
|
||||
unconfirmed = map[string]any{}
|
||||
}
|
||||
var allowed map[string]domain.AnnotationLabelRecord
|
||||
for i := range slices {
|
||||
c := &slices[i]
|
||||
p, exists := old[c.ID]
|
||||
if mode == "restore" && c.Dictionary != nil {
|
||||
if saved := savedDictionarySelection(item, c.Dictionary, "slice"); saved != nil && c.Name == saved.Name {
|
||||
c.Dictionary = saved
|
||||
delete(unconfirmed, c.ID)
|
||||
continue
|
||||
}
|
||||
}
|
||||
if mode == "draft" && strings.TrimSpace(c.Name) == "" {
|
||||
if c.IsDiscarded() {
|
||||
c.Dictionary = nil
|
||||
delete(unconfirmed, c.ID)
|
||||
continue
|
||||
}
|
||||
needs = append(needs, i)
|
||||
}
|
||||
if len(needs) == 0 {
|
||||
return nil
|
||||
}
|
||||
selection, err := s.store.GetProjectAnnotationLabels(ctx, item.ProjectID)
|
||||
if err != nil {
|
||||
return []validationIssue{{Field: "sliceSegments", Message: "无法读取项目切片字典,请重试"}}
|
||||
}
|
||||
allowed := map[string]bool{}
|
||||
if selection != nil {
|
||||
for _, id := range selection.SliceLabelIDs {
|
||||
label, e := s.store.GetAnnotationLabel(ctx, id)
|
||||
if e != nil {
|
||||
unchanged := exists && p.Name == c.Name && (c.Dictionary == nil || sameDictionaryVersion(p.Dictionary, c.Dictionary))
|
||||
finalizing := mode == "confirm" || mode == "submit"
|
||||
pendingName := p.Dictionary == nil && c.Name != "" && stringValue(unconfirmed[c.ID]) == c.Name
|
||||
needsNameCheck := (mode == "confirm" && p.Dictionary == nil) || (mode == "submit" && strings.TrimSpace(c.Name) == "") || (finalizing && pendingName)
|
||||
if unchanged && !needsNameCheck {
|
||||
c.Dictionary = p.Dictionary
|
||||
continue
|
||||
}
|
||||
|
||||
if (mode == "restore" && c.Dictionary == nil) || (mode == "draft" && strings.TrimSpace(c.Name) == "") {
|
||||
c.Dictionary = nil
|
||||
if c.Name != "" {
|
||||
unconfirmed[c.ID] = c.Name
|
||||
}
|
||||
continue
|
||||
}
|
||||
if allowed == nil {
|
||||
allowed, err = s.projectDictionarySelections(ctx, item, "slice")
|
||||
if err != nil {
|
||||
return []validationIssue{{Field: "sliceSegments", Message: "无法读取项目切片字典,请重试"}}
|
||||
}
|
||||
if label != nil && label.OrganizationID == item.OrganizationID && dictionaryKind(label.Kind) == "slice" && label.Status == domain.AnnotationLabelActive {
|
||||
allowed[label.Name] = true
|
||||
}
|
||||
var chosen *domain.DictionarySelection
|
||||
for _, label := range allowed {
|
||||
if label.Name == c.Name && (c.Dictionary == nil || (label.ID == c.Dictionary.ID && label.Version == c.Dictionary.Version)) {
|
||||
chosen = label.Selection()
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
issues := []validationIssue{}
|
||||
for _, i := range needs {
|
||||
if !allowed[slices[i].Name] {
|
||||
issues = append(issues, validationIssue{Field: fmt.Sprintf("sliceSegments.%d.name", i), Code: "slice_dictionary_name_required", Message: "请从本项目切片字典选择名称;不能自由填写或修改。未配置字典请联系管理员"})
|
||||
if chosen == nil {
|
||||
return []validationIssue{{Field: fmt.Sprintf("sliceSegments.%d.name", i), Code: "slice_dictionary_name_required", Message: "请从本项目切片字典选择名称;词条已更新时请重新选择"}}
|
||||
}
|
||||
if !(exists && p.Name == c.Name && p.Dictionary == nil && c.Dictionary == nil && stringValue(unconfirmed[c.ID]) != c.Name) {
|
||||
c.Dictionary = chosen
|
||||
}
|
||||
delete(unconfirmed, c.ID)
|
||||
}
|
||||
return issues
|
||||
if len(unconfirmed) > 0 {
|
||||
item.Subject.Payload["unconfirmedSliceNames"] = unconfirmed
|
||||
} else {
|
||||
delete(item.Subject.Payload, "unconfirmedSliceNames")
|
||||
}
|
||||
return rememberDictionarySelections(item, sliceDictionarySelections(slices)...)
|
||||
}
|
||||
|
||||
@ -142,6 +142,7 @@ func canonicalDispositions(previous, slices []domain.SliceSegment) ([]domain.Sli
|
||||
next[i].DiscardReasonLabelID = p.DiscardReasonLabelID
|
||||
next[i].DiscardReasonVersion = p.DiscardReasonVersion
|
||||
next[i].DiscardReasonName = p.DiscardReasonName
|
||||
next[i].DiscardReasonDictionary = p.DiscardReasonDictionary
|
||||
next[i].DiscardReasonNote = p.DiscardReasonNote
|
||||
next[i].DiscardedByUserID = p.DiscardedByUserID
|
||||
next[i].DiscardedAt = p.DiscardedAt
|
||||
@ -345,6 +346,7 @@ func restoredDiscardAnnotations(slice domain.SliceSegment) []domain.AnnotationSe
|
||||
|
||||
type sliceDispositionInput struct {
|
||||
combinedAnnotationDraftInput
|
||||
RestoreClassifications bool `json:"restoreClassifications,omitempty"`
|
||||
SliceID string `json:"sliceId"`
|
||||
Disposition string `json:"disposition"`
|
||||
Reason string `json:"reason"`
|
||||
@ -357,6 +359,13 @@ type sliceDispositionInput struct {
|
||||
}
|
||||
|
||||
func sliceNameBeforeDiscard(item *domain.HumanItemRecord, sliceID string, version int) (string, bool) {
|
||||
prior := sliceBeforeDiscard(item, sliceID, version)
|
||||
if prior == nil {
|
||||
return "", false
|
||||
}
|
||||
return prior.Name, true
|
||||
}
|
||||
func sliceBeforeDiscard(item *domain.HumanItemRecord, sliceID string, version int) *domain.SliceSegment {
|
||||
history, _ := normalizeStructuredValue(item.Subject.Payload["sliceDispositionHistory"]).([]any)
|
||||
resets, _ := normalizeStructuredValue(item.Subject.Payload["sliceResetHistory"]).([]any)
|
||||
resetVersion := 0
|
||||
@ -375,10 +384,11 @@ func sliceNameBeforeDiscard(item *domain.HumanItemRecord, sliceID string, versio
|
||||
before := decodeMaterializationValue[domain.SliceSegment]([]any{entry["before"]})
|
||||
after := decodeMaterializationValue[domain.SliceSegment]([]any{entry["after"]})
|
||||
if len(before) == 1 && len(after) == 1 && !before[0].IsDiscarded() && after[0].IsDiscarded() {
|
||||
return before[0].Name, true
|
||||
copy := before[0]
|
||||
return ©
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *server) handleSliceDisposition(w http.ResponseWriter, r *http.Request) {
|
||||
@ -468,6 +478,7 @@ func (s *server) prepareSliceDisposition(w http.ResponseWriter, r *http.Request,
|
||||
}
|
||||
|
||||
slices, issues := s.canonicalSliceDispositionState(r.Context(), item, body.normalizedSliceSegments(), body.normalizedOperations())
|
||||
issues = append(issues, s.validateProjectSliceNames(r.Context(), item, slices, "restore")...)
|
||||
if task.SourceIsFullVideo {
|
||||
timeline, e := s.annotationFrameTimelineForTask(r.Context(), *task, "left")
|
||||
if e != nil {
|
||||
@ -480,6 +491,7 @@ func (s *server) prepareSliceDisposition(w http.ResponseWriter, r *http.Request,
|
||||
}
|
||||
issues = append(issues, validateSliceSegments(slices, *task)...)
|
||||
issues = append(issues, validateCompleteSliceCoverage(slices, *task)...)
|
||||
issues = append(issues, s.canonicalActionDictionaries(r.Context(), item, body.Segments, dispositionDictionaryMode(body))...)
|
||||
issues = append(issues, validateDraftCombinedAnnotationEvents(body.Segments, *task)...)
|
||||
idx := -1
|
||||
for i, v := range slices {
|
||||
@ -536,14 +548,20 @@ func (s *server) prepareSliceDisposition(w http.ResponseWriter, r *http.Request,
|
||||
if body.Disposition == "discard" {
|
||||
next.DeliveryDisposition = "discard"
|
||||
next.Name, next.TaskLabelID, next.TaskLabel = "", "", ""
|
||||
next.Dictionary = nil
|
||||
next.DiscardReason = body.Reason
|
||||
next.DiscardReasonLabelID = ""
|
||||
next.DiscardReasonVersion = 0
|
||||
next.DiscardReasonName = ""
|
||||
next.DiscardReasonDictionary = nil
|
||||
if reasonLabel != nil {
|
||||
next.DiscardReasonLabelID = reasonLabel.ID
|
||||
next.DiscardReasonVersion = reasonLabel.Version
|
||||
next.DiscardReasonName = reasonLabel.Name
|
||||
next.DiscardReasonDictionary = reasonLabel.Selection()
|
||||
if body.RestoreClassifications {
|
||||
next.DiscardReasonDictionary = savedDictionarySelection(item, &domain.DictionarySelection{ID: reasonLabel.ID, Version: reasonLabel.Version}, "discard_reason")
|
||||
}
|
||||
}
|
||||
next.DiscardReasonNote = body.ReasonNote
|
||||
next.DiscardedByUserID = actorID
|
||||
@ -622,11 +640,18 @@ func (s *server) prepareSliceDisposition(w http.ResponseWriter, r *http.Request,
|
||||
// Precise undo targets a server-recorded discard, even after later
|
||||
// undo/redo writes. Ordinary restore deliberately remains unnamed.
|
||||
next.Name = restoredName
|
||||
next.Dictionary = nil
|
||||
if body.RestorePreviousName || body.RestoreNameFromVersion != 0 {
|
||||
if prior := sliceBeforeDiscard(item, before.ID, body.RestoreNameFromVersion); prior != nil {
|
||||
next.Dictionary = prior.Dictionary
|
||||
}
|
||||
}
|
||||
}
|
||||
next.DiscardReason = ""
|
||||
next.DiscardReasonLabelID = ""
|
||||
next.DiscardReasonVersion = 0
|
||||
next.DiscardReasonName = ""
|
||||
next.DiscardReasonDictionary = nil
|
||||
next.DiscardReasonNote = ""
|
||||
next.DiscardedByUserID = ""
|
||||
next.DiscardedAt = ""
|
||||
@ -638,13 +663,20 @@ func (s *server) prepareSliceDisposition(w http.ResponseWriter, r *http.Request,
|
||||
next = slices[idx]
|
||||
// Disposition is an incremental edit, not confirmation of the whole slicing
|
||||
// plan. Other slices may still have draft names; confirmation validates them.
|
||||
issues = validateDraftCombinedAnnotationEvents(segments, *task)
|
||||
issues = s.validateProjectSliceNames(r.Context(), item, slices, "restore")
|
||||
issues = append(issues, s.canonicalActionDictionaries(r.Context(), item, segments, "restore")...)
|
||||
issues = append(issues, validateDraftCombinedAnnotationEvents(segments, *task)...)
|
||||
|
||||
if len(issues) > 0 {
|
||||
renderValidationError(w, describeSliceNameIssues(issues, slices)...)
|
||||
return nil, domain.SliceSegment{}, false
|
||||
}
|
||||
statuses := body.normalizedStatuses()
|
||||
statuses[next.ID] = domain.SliceAnnotationPending
|
||||
if issues := rememberDictionarySelections(item, sliceDictionarySelections(slices)...); len(issues) > 0 {
|
||||
renderValidationError(w, issues...)
|
||||
return nil, domain.SliceSegment{}, false
|
||||
}
|
||||
workspace := map[string]any{"sliceSegments": slices, "segments": segments, "statuses": statuses, "operations": body.normalizedOperations(), "userId": actorID, "savedAt": nowISO()}
|
||||
if item.Subject.Payload == nil {
|
||||
item.Subject.Payload = map[string]any{}
|
||||
@ -665,3 +697,10 @@ func (s *server) prepareSliceDisposition(w http.ResponseWriter, r *http.Request,
|
||||
}
|
||||
return workspace, next, true
|
||||
}
|
||||
|
||||
func dispositionDictionaryMode(body sliceDispositionInput) string {
|
||||
if body.RestoreClassifications {
|
||||
return "restore"
|
||||
}
|
||||
return "draft"
|
||||
}
|
||||
|
||||
@ -82,11 +82,14 @@ func TestSliceDiscardMaterialization60Seconds(t *testing.T) {
|
||||
for i := 0; i < 3; i++ {
|
||||
start, end := float64(i*20), float64((i+1)*20)
|
||||
sl := domain.SliceSegment{ID: fmt.Sprintf("segment-%d", i), Name: fmt.Sprintf("片段 %d", i+1), Start: start, End: end}
|
||||
sl.Dictionary = &domain.DictionarySelection{ID: fmt.Sprintf("base-%d", i), Kind: "slice", Version: 2, Code: fmt.Sprintf("class-%d", i), Name: sl.Name, Tags: []string{"基础分类", "商品"}}
|
||||
if mode == "discard" || (mode == "mixed" && i == 1) {
|
||||
sl.DeliveryDisposition = "discard"
|
||||
sl.DiscardReason = "审查测试:画面被遮挡"
|
||||
sl.DiscardReasonLabelID = "reason-test"
|
||||
sl.DiscardReasonVersion = 3
|
||||
sl.Dictionary = nil
|
||||
sl.DiscardReasonDictionary = &domain.DictionarySelection{ID: "reason-test", Kind: "discard_reason", Version: 3, Code: "occluded", Name: "审查测试", Tags: []string{"不可用"}}
|
||||
sl.DiscardReasonName = "审查测试"
|
||||
sl.DiscardReasonNote = "画面被遮挡"
|
||||
sl.DiscardedAt = "2026-09-16T00:00:00Z"
|
||||
@ -94,7 +97,7 @@ func TestSliceDiscardMaterialization60Seconds(t *testing.T) {
|
||||
} else if mode == "partial" && i == 0 {
|
||||
segments = append(segments, domain.AnnotationSegment{ID: "partial", Start: 5, End: 10, Description: "仅描述这五秒", Source: "human"})
|
||||
} else if mode != "blank" && mode != "partial" {
|
||||
segments = append(segments, domain.AnnotationSegment{ID: fmt.Sprintf("action-%d", i), Start: start, End: end, Description: "测试动作", Source: "human"})
|
||||
segments = append(segments, domain.AnnotationSegment{ID: fmt.Sprintf("action-%d", i), Start: start, End: end, Description: "测试动作", Dictionary: &domain.DictionarySelection{ID: "action-test", Kind: "action", Version: 4, Code: "pick", Name: "拿取", Tags: []string{"抓取"}}, Source: "human"})
|
||||
}
|
||||
slices = append(slices, sl)
|
||||
}
|
||||
|
||||
@ -9,8 +9,9 @@ import (
|
||||
|
||||
type sliceEditApplicationInput struct {
|
||||
combinedAnnotationDraftInput
|
||||
ApplicationID string `json:"applicationId"`
|
||||
Dispositions []sliceDispositionInput `json:"dispositions"`
|
||||
RestoreClassifications bool `json:"restoreClassifications,omitempty"`
|
||||
ApplicationID string `json:"applicationId"`
|
||||
Dispositions []sliceDispositionInput `json:"dispositions"`
|
||||
}
|
||||
|
||||
// A panel application is one CAS, including explicit disposition commands and
|
||||
@ -71,6 +72,7 @@ func (s *server) handleSliceEditApplication(w http.ResponseWriter, r *http.Reque
|
||||
}
|
||||
commands := map[string]sliceDispositionInput{}
|
||||
for _, command := range body.Dispositions {
|
||||
command.RestoreClassifications = body.RestoreClassifications
|
||||
if command.SliceID == "" || commands[command.SliceID].SliceID != "" || (command.Disposition != "keep" && command.Disposition != "discard") {
|
||||
renderValidationError(w, validationIssue{Field: "dispositions", Message: "切片用途操作无效或重复"})
|
||||
return
|
||||
@ -81,6 +83,7 @@ func (s *server) handleSliceEditApplication(w http.ResponseWriter, r *http.Reque
|
||||
// Undo can first restore a discarded child before rejoining it. The restore
|
||||
// remains an explicit, audited command in the same atomic application.
|
||||
for _, command := range body.Dispositions {
|
||||
command.RestoreClassifications = body.RestoreClassifications
|
||||
if !ids[command.SliceID] {
|
||||
if command.Disposition != "keep" {
|
||||
renderValidationError(w, validationIssue{Field: "dispositions", Message: "舍弃操作必须保留目标切片"})
|
||||
@ -143,10 +146,15 @@ func (s *server) handleSliceEditApplication(w http.ResponseWriter, r *http.Reque
|
||||
}
|
||||
}
|
||||
baseline[i] = sliceEditPolicyBaseline(candidate, old)
|
||||
if command := commands[candidate.ID]; command.Disposition == "discard" && !old.IsDiscarded() {
|
||||
baseline[i].Name = old.Name
|
||||
baseline[i].Dictionary = old.Dictionary
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
slices, issues := s.canonicalSliceDispositionState(r.Context(), &item, baseline, body.normalizedOperations())
|
||||
issues = append(issues, s.validateProjectSliceNames(r.Context(), stored, slices, applicationDictionaryMode(body))...)
|
||||
issues = append(issues, s.validateSliceEditFrameBounds(r, task, &item, slices, body.Segments)...)
|
||||
issues = append(issues, validateSliceSegments(slices, *task)...)
|
||||
issues = append(issues, validateCompleteSliceCoverage(slices, *task)...)
|
||||
@ -194,10 +202,16 @@ func (s *server) handleSliceEditApplication(w http.ResponseWriter, r *http.Reque
|
||||
segmentsForApplication = retained
|
||||
}
|
||||
}
|
||||
if issues := s.canonicalActionDictionaries(r.Context(), stored, segmentsForApplication, applicationDictionaryMode(body)); len(issues) > 0 {
|
||||
renderValidationError(w, issues...)
|
||||
return
|
||||
}
|
||||
copyClassificationRecoveryState(&item, stored)
|
||||
workspace := map[string]any{"sliceSegments": slices, "segments": segmentsForApplication, "statuses": body.normalizedStatuses(), "operations": body.normalizedOperations(), "userId": principal.User.ID, "savedAt": nowISO()}
|
||||
item.Subject.Payload["annotationWorkspace"] = workspace
|
||||
delete(item.Subject.Payload, "annotationDraft")
|
||||
for _, command := range body.Dispositions {
|
||||
command.RestoreClassifications = body.RestoreClassifications
|
||||
if ids[command.SliceID] {
|
||||
command.combinedAnnotationDraftInput = workspaceSliceEditInput(workspace)
|
||||
var prepared bool
|
||||
@ -239,10 +253,13 @@ func (s *server) handleSliceEditApplication(w http.ResponseWriter, r *http.Reque
|
||||
for _, candidate := range proposed {
|
||||
if candidate.ID == slices[i].ID && !slices[i].IsDiscarded() {
|
||||
slices[i].Name = candidate.Name
|
||||
slices[i].Dictionary = candidate.Dictionary
|
||||
}
|
||||
}
|
||||
}
|
||||
issues = s.validateProjectSliceNames(r.Context(), &item, slices, "draft")
|
||||
issues = s.validateProjectSliceNames(r.Context(), stored, slices, applicationDictionaryMode(body))
|
||||
issues = append(issues, s.canonicalActionDictionaries(r.Context(), stored, segments, applicationDictionaryMode(body))...)
|
||||
copyClassificationRecoveryState(&item, stored)
|
||||
issues = append(issues, s.validateSliceEditFrameBounds(r, task, &item, slices, segments)...)
|
||||
issues = append(issues, validateDraftCombinedAnnotationEvents(segments, *task)...)
|
||||
issues = append(issues, validateDraftFanyuV5CombinedSemanticSegments(segments, slices, annotationTemplateWithTaskSnapshot(template, task))...)
|
||||
@ -265,6 +282,7 @@ func (s *server) handleSliceEditApplication(w http.ResponseWriter, r *http.Reque
|
||||
item.Subject.Payload["annotationWorkspace"] = workspace
|
||||
// Recheck dictionary versions immediately before the single commit.
|
||||
for _, command := range body.Dispositions {
|
||||
command.RestoreClassifications = body.RestoreClassifications
|
||||
if command.Disposition == "discard" {
|
||||
if _, issues := s.resolveDiscardReason(r.Context(), &item, command); len(issues) > 0 {
|
||||
renderValidationError(w, issues...)
|
||||
@ -302,6 +320,7 @@ func sliceEditPolicyBaseline(candidate, old domain.SliceSegment) domain.SliceSeg
|
||||
candidate.DiscardReasonLabelID = old.DiscardReasonLabelID
|
||||
candidate.DiscardReasonVersion = old.DiscardReasonVersion
|
||||
candidate.DiscardReasonName = old.DiscardReasonName
|
||||
candidate.DiscardReasonDictionary = old.DiscardReasonDictionary
|
||||
candidate.DiscardReasonNote = old.DiscardReasonNote
|
||||
candidate.DiscardedAt = old.DiscardedAt
|
||||
candidate.DiscardedByUserID = old.DiscardedByUserID
|
||||
@ -344,3 +363,10 @@ func (s *server) validateSliceEditFrameBounds(r *http.Request, task *domain.Anno
|
||||
}
|
||||
return issues
|
||||
}
|
||||
|
||||
func applicationDictionaryMode(body sliceEditApplicationInput) string {
|
||||
if body.RestoreClassifications {
|
||||
return "restore"
|
||||
}
|
||||
return "draft"
|
||||
}
|
||||
|
||||
@ -122,6 +122,8 @@ func (s *server) handleSliceEditRestoration(w http.ResponseWriter, r *http.Reque
|
||||
issues = append(issues, validateCompleteSliceCoverage(slices, *task)...)
|
||||
issues = append(issues, validateDraftCombinedAnnotationEvents(body.Segments, *task)...)
|
||||
issues = append(issues, validateDraftStatuses(body.normalizedStatuses(), slices)...)
|
||||
issues = append(issues, s.validateProjectSliceNames(r.Context(), item, slices, "restore")...)
|
||||
issues = append(issues, s.canonicalActionDictionaries(r.Context(), item, body.Segments, "restore")...)
|
||||
template, err := s.store.GetAnnotationTemplate(r.Context(), task.TemplateID)
|
||||
if err != nil {
|
||||
renderError(w, 500, err)
|
||||
|
||||
@ -16,6 +16,7 @@ import (
|
||||
)
|
||||
|
||||
const PartSchema = "droboticflow.delivery.part-index/v3"
|
||||
const ClassificationPartSchema = "droboticflow.delivery.part-index/v4"
|
||||
|
||||
var partName = regexp.MustCompile(`^part[0-9]{4,}$`)
|
||||
|
||||
@ -48,7 +49,7 @@ type Summary struct {
|
||||
DiscardedCount int `json:"discardedSliceCount"`
|
||||
KeptDuration float64 `json:"keptDurationSeconds"`
|
||||
DiscardedDuration float64 `json:"discardedDurationSeconds"`
|
||||
Acceptance map[string]string `json:"acceptance"`
|
||||
Acceptance map[string]string `json:"acceptance,omitempty"`
|
||||
}
|
||||
type part struct {
|
||||
Schema string `json:"schemaVersion"`
|
||||
@ -117,6 +118,9 @@ func addSummary(a *Summary, b Summary) {
|
||||
a.DiscardedCount += b.DiscardedCount
|
||||
a.KeptDuration += b.KeptDuration
|
||||
a.DiscardedDuration += b.DiscardedDuration
|
||||
if b.Acceptance == nil {
|
||||
return
|
||||
}
|
||||
if a.Acceptance == nil {
|
||||
a.Acceptance = map[string]string{"decision": "approved"}
|
||||
}
|
||||
@ -128,10 +132,10 @@ func addSummary(a *Summary, b Summary) {
|
||||
}
|
||||
func checkPart(p part) (Summary, error) {
|
||||
s := Summary{PartCount: 1, Acceptance: map[string]string{"decision": "approved", "acceptedAt": p.Acceptance.Final.AcceptedAt}}
|
||||
if p.Schema != PartSchema || !safeName(p.Capture.ID) || !partName.MatchString(p.PartID) || p.Revision < 1 || !p.Acceptance.Final.Confirmed || len(p.Acceptance.Reviews) != 0 {
|
||||
if (p.Schema != PartSchema && p.Schema != ClassificationPartSchema) || !safeName(p.Capture.ID) || !partName.MatchString(p.PartID) || p.Revision < 1 || !p.Acceptance.Final.Confirmed || len(p.Acceptance.Reviews) != 0 {
|
||||
return s, fmt.Errorf("part must pin an accepted native v3 revision without embedded review history")
|
||||
}
|
||||
if _, e := time.Parse(time.RFC3339Nano, p.Acceptance.Final.AcceptedAt); e != nil {
|
||||
if _, e := time.Parse(time.RFC3339Nano, p.Acceptance.Final.AcceptedAt); e != nil && p.Schema != ClassificationPartSchema {
|
||||
return s, fmt.Errorf("missing acceptance timestamp")
|
||||
}
|
||||
if p.SourceMetadata.Path != "meta_"+p.Capture.ID+".json" || path.Base(p.Capture.Prefix) != p.Capture.ID || path.Clean(p.Capture.Prefix) != p.Capture.Prefix || strings.HasPrefix(p.Capture.Prefix, "/") || strings.Contains(p.Capture.Prefix, "../") {
|
||||
@ -168,6 +172,9 @@ func checkPart(p part) (Summary, error) {
|
||||
invalid = true
|
||||
} else if strings.TrimSpace(a.Description) != "" {
|
||||
kind = "described"
|
||||
if p.Schema == ClassificationPartSchema {
|
||||
kind = "annotated"
|
||||
}
|
||||
described = true
|
||||
} else {
|
||||
blank = true
|
||||
@ -191,7 +198,7 @@ func checkPart(p part) (Summary, error) {
|
||||
}
|
||||
}
|
||||
}
|
||||
if s.KeptCount == 0 {
|
||||
if s.KeptCount == 0 && (p.Schema != ClassificationPartSchema || s.DiscardedCount == 0) {
|
||||
return s, fmt.Errorf("no deliverable slices")
|
||||
}
|
||||
if math.IsInf(s.KeptDuration, 0) || math.IsInf(s.DiscardedDuration, 0) {
|
||||
@ -206,6 +213,9 @@ func checkPart(p part) (Summary, error) {
|
||||
return s, fmt.Errorf("slice coverage gap or overlap")
|
||||
}
|
||||
}
|
||||
if p.Schema == ClassificationPartSchema {
|
||||
s.Acceptance = nil
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
@ -223,6 +233,25 @@ func Build(projectID, projectName, projectRoot string, sources []Source) (Snapsh
|
||||
return out, fmt.Errorf("source JSON exceeds 64 MiB")
|
||||
}
|
||||
}
|
||||
schema := ""
|
||||
for _, src := range sources {
|
||||
var p part
|
||||
if err := json.Unmarshal(src.Part, &p); err != nil {
|
||||
return out, err
|
||||
}
|
||||
if schema != "" && schema != p.Schema {
|
||||
return out, fmt.Errorf("cannot mix part schema versions in one batch")
|
||||
}
|
||||
schema = p.Schema
|
||||
if p.Schema == ClassificationPartSchema {
|
||||
if err := validateConsumerPart(src.Part); err != nil {
|
||||
return out, err
|
||||
}
|
||||
if _, err := classificationAudit(src); err != nil {
|
||||
return out, err
|
||||
}
|
||||
}
|
||||
}
|
||||
// Normalize JSON before hashing to ignore transport whitespace/map ordering.
|
||||
for i := range sources {
|
||||
var d, h any
|
||||
@ -235,7 +264,17 @@ func Build(projectID, projectName, projectRoot string, sources []Source) (Snapsh
|
||||
sort.Strings(sources[i].ExpectedParts)
|
||||
}
|
||||
sort.Slice(sources, func(i, j int) bool { return string(sources[i].Part) < string(sources[j].Part) })
|
||||
identity := encode([]any{"droboticflow.delivery.batch/v1", projectID, projectName, projectRoot, sources})
|
||||
identitySources := append([]Source(nil), sources...)
|
||||
if schema == ClassificationPartSchema {
|
||||
for i := range identitySources {
|
||||
identitySources[i].History = nil
|
||||
}
|
||||
}
|
||||
identity := encode([]any{"droboticflow.delivery.batch/v1", projectID, projectName, projectRoot, identitySources})
|
||||
version := 3
|
||||
if schema == ClassificationPartSchema {
|
||||
version = 4
|
||||
}
|
||||
out.ID = Hash(identity)
|
||||
out.Root = projectRoot + "/index-batches/" + out.ID
|
||||
type group struct {
|
||||
@ -259,6 +298,15 @@ func Build(projectID, projectName, projectRoot string, sources []Source) (Snapsh
|
||||
if e := json.Unmarshal(src.Part, &p); e != nil {
|
||||
return out, e
|
||||
}
|
||||
if p.Schema == ClassificationPartSchema {
|
||||
// The original bytes were checked before canonicalization above.
|
||||
var audit classificationHistory
|
||||
if json.Unmarshal(src.History, &audit) != nil {
|
||||
return out, fmt.Errorf("invalid internal audit")
|
||||
}
|
||||
p.Acceptance.Final.Confirmed = audit.Acceptance.Final.Confirmed
|
||||
p.Acceptance.Final.AcceptedAt = audit.Acceptance.Final.AcceptedAt
|
||||
}
|
||||
s, e := checkPart(p)
|
||||
if e != nil {
|
||||
return out, e
|
||||
@ -290,7 +338,7 @@ func Build(projectID, projectName, projectRoot string, sources []Source) (Snapsh
|
||||
Revision int `json:"revision"`
|
||||
Reviews []json.RawMessage `json:"qualityReviews"`
|
||||
}
|
||||
if json.Unmarshal(src.History, &history) != nil || history.Schema != "droboticflow.delivery.review-history/v1" || history.CaptureID != p.Capture.ID || history.PartID != p.PartID || history.Revision != p.Revision || history.Reviews == nil {
|
||||
if p.Schema != ClassificationPartSchema && (json.Unmarshal(src.History, &history) != nil || history.Schema != "droboticflow.delivery.review-history/v1" || history.CaptureID != p.Capture.ID || history.PartID != p.PartID || history.Revision != p.Revision || history.Reviews == nil) {
|
||||
return out, fmt.Errorf("review history revision mismatch")
|
||||
}
|
||||
g := groups[p.Capture.ID]
|
||||
@ -302,10 +350,17 @@ func Build(projectID, projectName, projectRoot string, sources []Source) (Snapsh
|
||||
}
|
||||
var doc map[string]any
|
||||
json.Unmarshal(src.Part, &doc)
|
||||
doc["reviewHistory"] = map[string]string{"path": p.PartID + ".review-history.json", "pathBase": "capture_delivery"}
|
||||
doc["delivery"] = map[string]string{"kind": "part_index", "fileName": p.PartID + ".json", "artifactName": "delivery.part-index"}
|
||||
g.entries = append(g.entries, add(name+".json", encode(doc)), add(name+".review-history.json", src.History))
|
||||
g.parts = append(g.parts, map[string]any{"part": p.PartID, "revision": p.Revision, "path": p.PartID + ".json", "reviewHistoryPath": p.PartID + ".review-history.json"})
|
||||
g.entries = append(g.entries, add(name+".json", encode(doc)))
|
||||
reference := map[string]any{"part": p.PartID, "revision": p.Revision, "path": p.PartID + ".json"}
|
||||
if p.Schema != ClassificationPartSchema {
|
||||
doc["reviewHistory"] = map[string]string{"path": p.PartID + ".review-history.json", "pathBase": "capture_delivery"}
|
||||
totalBytes -= len(out.Files[name+".json"])
|
||||
g.entries[len(g.entries)-1] = add(name+".json", encode(doc))
|
||||
g.entries = append(g.entries, add(name+".review-history.json", src.History))
|
||||
reference["reviewHistoryPath"] = p.PartID + ".review-history.json"
|
||||
}
|
||||
g.parts = append(g.parts, reference)
|
||||
addSummary(&g.summary, s)
|
||||
if math.IsInf(g.summary.KeptDuration, 0) || math.IsInf(g.summary.DiscardedDuration, 0) {
|
||||
return out, fmt.Errorf("capture duration overflow")
|
||||
@ -343,7 +398,7 @@ func Build(projectID, projectName, projectRoot string, sources []Source) (Snapsh
|
||||
size += local[i].SizeBytes
|
||||
}
|
||||
metadata := map[string]any{"path": "meta_" + n + ".json", "pathBase": "original_capture", "sha256": g.metadata.MetadataSHA256, "sizeBytes": g.metadata.MetadataBytes}
|
||||
manifest := map[string]any{"schemaVersion": 3, "kind": "part-index", "capture": g.capture, "pathBase": "capture_delivery", "sourceMetadata": metadata, "summary": g.summary, "parts": g.parts, "entries": local, "fileCount": len(local), "totalBytes": size}
|
||||
manifest := map[string]any{"schemaVersion": version, "kind": "part-index", "capture": g.capture, "pathBase": "capture_delivery", "sourceMetadata": metadata, "summary": g.summary, "parts": g.parts, "entries": local, "fileCount": len(local), "totalBytes": size}
|
||||
allEntries = append(allEntries, g.entries...)
|
||||
allEntries = append(allEntries, add(n+"/annotation_manifest.json", encode(manifest)))
|
||||
captures = append(captures, map[string]any{"captureId": n, "prefix": g.capture["prefix"], "manifestPath": n + "/annotation_manifest.json", "summary": g.summary})
|
||||
@ -360,7 +415,50 @@ func Build(projectID, projectName, projectRoot string, sources []Source) (Snapsh
|
||||
for _, v := range allEntries {
|
||||
size += v.SizeBytes
|
||||
}
|
||||
out.Index = encode(map[string]any{"schemaVersion": 3, "kind": "part-index", "snapshotId": out.ID, "pathBase": "batch_project", "project": map[string]string{"projectId": projectID, "deliveryName": projectName, "rootKey": out.Root}, "summary": total, "captures": captures, "entries": allEntries, "fileCount": len(allEntries), "totalBytes": size})
|
||||
out.Index = encode(map[string]any{"schemaVersion": version, "kind": "part-index", "snapshotId": out.ID, "pathBase": "batch_project", "project": map[string]string{"projectId": projectID, "deliveryName": projectName, "rootKey": out.Root}, "summary": total, "captures": captures, "entries": allEntries, "fileCount": len(allEntries), "totalBytes": size})
|
||||
out.Files["delivery_index.json"] = out.Index
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// classificationHistory is internal acceptance proof, excluded from v4 batch
|
||||
// identity and customer files. The digest binds it to the original native bytes.
|
||||
type classificationHistory struct {
|
||||
Schema string `json:"schemaVersion"`
|
||||
CaptureID string `json:"captureId"`
|
||||
PartID string `json:"partId"`
|
||||
Revision int `json:"revision"`
|
||||
PartSHA string `json:"partSha256"`
|
||||
Acceptance struct {
|
||||
Final struct {
|
||||
Confirmed bool `json:"confirmed"`
|
||||
AcceptedAt string `json:"acceptedAt"`
|
||||
Sampled *bool `json:"sampled"`
|
||||
SamplingDecision string `json:"samplingDecision"`
|
||||
} `json:"finalAcceptance"`
|
||||
} `json:"acceptance"`
|
||||
}
|
||||
|
||||
func classificationAudit(src Source) (classificationHistory, error) {
|
||||
var audit classificationHistory
|
||||
var p part
|
||||
var doc map[string]json.RawMessage
|
||||
if json.Unmarshal(src.Part, &p) != nil || json.Unmarshal(src.Part, &doc) != nil || json.Unmarshal(src.History, &audit) != nil {
|
||||
return audit, fmt.Errorf("invalid classification/audit JSON")
|
||||
}
|
||||
for _, field := range []string{"acceptance", "lineage", "reviewHistory"} {
|
||||
if _, ok := doc[field]; ok {
|
||||
return audit, fmt.Errorf("customer classification contains internal trace")
|
||||
}
|
||||
}
|
||||
if audit.Schema != "droboticflow.delivery.review-history/v2" || audit.CaptureID != p.Capture.ID || audit.PartID != p.PartID || audit.Revision != p.Revision || audit.PartSHA != Hash(src.Part) || !audit.Acceptance.Final.Confirmed {
|
||||
return audit, fmt.Errorf("internal audit does not pin the accepted classification")
|
||||
}
|
||||
if _, err := time.Parse(time.RFC3339Nano, audit.Acceptance.Final.AcceptedAt); err != nil {
|
||||
final := audit.Acceptance.Final
|
||||
exempt := final.AcceptedAt == "" && final.Sampled != nil && !*final.Sampled && final.SamplingDecision == "not_selected"
|
||||
if !exempt {
|
||||
return audit, fmt.Errorf("missing internal acceptance timestamp or explicit sampling exemption")
|
||||
}
|
||||
}
|
||||
return audit, nil
|
||||
}
|
||||
|
||||
176
internal/deliveryindex/classification_test.go
Normal file
176
internal/deliveryindex/classification_test.go
Normal file
@ -0,0 +1,176 @@
|
||||
package deliveryindex
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func classificationFixture(allDiscard bool) Source {
|
||||
src := fixture("part0001", 7)
|
||||
var p map[string]any
|
||||
json.Unmarshal(src.Part, &p)
|
||||
acceptance := p["acceptance"]
|
||||
delete(p, "acceptance")
|
||||
p["schemaVersion"] = ClassificationPartSchema
|
||||
if allDiscard {
|
||||
p["slices"] = []any{}
|
||||
p["discardedSlices"] = []any{map[string]any{"sliceId": "all", "taskSplit": map[string]any{"startSecond": 0, "endSecond": 3, "semantics": "half-open"}, "classification": map[string]any{"deliveryDisposition": "discard", "discardEvidence": map[string]any{"reason": "遮挡"}}, "segments": []any{}}}
|
||||
p["deliveryClassification"] = map[string]any{"keptSliceCount": 0, "discardedSliceCount": 1, "keptDurationSeconds": 0, "discardedDurationSeconds": 3}
|
||||
}
|
||||
enrichConsumerFixture(p)
|
||||
src.Part = encode(p)
|
||||
src.History = encode(map[string]any{"schemaVersion": "droboticflow.delivery.review-history/v2", "captureId": "capture-A", "partId": "part0001", "revision": 7, "partSha256": Hash(src.Part), "acceptance": acceptance, "qualityReviews": []any{map[string]string{"comment": "PRIVATE_REWORK"}}, "trace": []any{}})
|
||||
return src
|
||||
}
|
||||
func TestClassificationBatchNeverIncludesPrivateTrace(t *testing.T) {
|
||||
for _, all := range []bool{false, true} {
|
||||
src := classificationFixture(all)
|
||||
out := build(t, src)
|
||||
if len(out.Files) != 3 {
|
||||
t.Fatalf("customer files %v", out.Files)
|
||||
}
|
||||
for name, body := range out.Files {
|
||||
if strings.Contains(name, "history") || bytes.Contains(body, []byte("PRIVATE_REWORK")) || bytes.Contains(body, []byte(`"acceptance"`)) {
|
||||
t.Fatalf("private trace in %s", name)
|
||||
}
|
||||
}
|
||||
var h map[string]any
|
||||
json.Unmarshal(src.History, &h)
|
||||
h["qualityReviews"] = []any{map[string]string{"comment": "changed"}}
|
||||
src.History = encode(h)
|
||||
again := build(t, src)
|
||||
if out.ID != again.ID || !bytes.Equal(out.Index, again.Index) {
|
||||
t.Fatal("audit-only edit changed customer batch")
|
||||
}
|
||||
src.History = bytes.ReplaceAll(src.History, []byte(Hash(src.Part)), []byte(strings.Repeat("0", 64)))
|
||||
if _, err := Build("project-A", "项目A", "delivery/项目A__project-A", []Source{src}); err == nil {
|
||||
t.Fatal("wrong acceptance digest accepted")
|
||||
}
|
||||
}
|
||||
}
|
||||
func TestClassificationBatchRejectsMixedVersions(t *testing.T) {
|
||||
if _, err := Build("project-A", "项目A", "delivery/项目A__project-A", []Source{classificationFixture(false), fixture("part0002", 1)}); err == nil {
|
||||
t.Fatal("mixed versions accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassificationBatchRejectsIncompleteConsumerContract(t *testing.T) {
|
||||
for _, failure := range []string{"contract", "tags", "tag-source", "part-time", "pts", "original-file", "original-source"} {
|
||||
t.Run(failure, func(t *testing.T) {
|
||||
src := classificationFixture(false)
|
||||
var doc, audit map[string]any
|
||||
json.Unmarshal(src.Part, &doc)
|
||||
slice := doc["slices"].([]any)[0].(map[string]any)
|
||||
action := slice["segments"].([]any)[0].(map[string]any)
|
||||
switch failure {
|
||||
case "contract":
|
||||
delete(doc, "consumerContract")
|
||||
case "tags":
|
||||
delete(action, "tags")
|
||||
case "tag-source":
|
||||
action["tagSource"] = "action"
|
||||
case "part-time":
|
||||
action["partStartSecond"] = -1
|
||||
case "pts":
|
||||
delete(action, "startTimestampUs")
|
||||
case "original-source":
|
||||
delete(doc, "originalSource")
|
||||
case "original-file":
|
||||
delete(doc["originalSource"].(map[string]any)["files"].(map[string]any), "leftPts")
|
||||
}
|
||||
src.Part = encode(doc)
|
||||
json.Unmarshal(src.History, &audit)
|
||||
// Keep acceptance hash valid: rejection must be the consumer guard.
|
||||
audit["partSha256"] = Hash(src.Part)
|
||||
src.History = encode(audit)
|
||||
if _, err := Build("project-A", "项目A", "delivery/项目A__project-A", []Source{src}); err == nil || !strings.Contains(err.Error(), "consumer contract") {
|
||||
t.Fatalf("incomplete %s accepted: %v", failure, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassificationBatchAcceptsExplicitSamplingExemptionWithoutInventingHumanDate(t *testing.T) {
|
||||
src := classificationFixture(false)
|
||||
var audit map[string]any
|
||||
json.Unmarshal(src.History, &audit)
|
||||
final := audit["acceptance"].(map[string]any)["finalAcceptance"].(map[string]any)
|
||||
delete(final, "acceptedAt")
|
||||
final["sampled"] = false
|
||||
final["samplingDecision"] = "not_selected"
|
||||
src.History = encode(audit)
|
||||
out := build(t, src)
|
||||
if bytes.Contains(out.Index, []byte("acceptedAt")) {
|
||||
t.Fatal("invented human acceptance date")
|
||||
}
|
||||
final["samplingDecision"] = "selected"
|
||||
src.History = encode(audit)
|
||||
if _, err := Build("project-A", "项目A", "delivery/项目A__project-A", []Source{src}); err == nil {
|
||||
t.Fatal("unverified human acceptance accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func enrichConsumerFixture(p map[string]any) {
|
||||
part := p["partId"].(string)
|
||||
p["consumerContract"] = "droboticflow.classification-consumer/v1"
|
||||
p["timeBases"] = map[string]string{"taskSplitSeconds": "original_left_video", "segmentSeconds": "slice_left_video", "segmentPartSeconds": "original_left_video", "timestampUs": "original_hardware_pts", "interval": "half-open", "frameRanges": "zero-based-half-open"}
|
||||
p["pathBases"] = map[string]string{"sources": "original_capture", "sourceMetadata": "original_capture", "calibration": "original_capture"}
|
||||
files := map[string]any{}
|
||||
sources := map[string]any{}
|
||||
for _, role := range []string{"leftVideo", "rightVideo", "leftPts", "rightPts", "imu", "cameraCalibration", "imuCalibration", "metadata"} {
|
||||
name := part + "/" + role
|
||||
if role == "cameraCalibration" || role == "imuCalibration" {
|
||||
name = role + ".yaml"
|
||||
}
|
||||
if role == "metadata" {
|
||||
name = "meta_capture-A.json"
|
||||
}
|
||||
files[role] = map[string]any{"path": name}
|
||||
if role == "leftVideo" || role == "rightVideo" || role == "leftPts" || role == "rightPts" || role == "imu" {
|
||||
sources[role] = files[role]
|
||||
}
|
||||
}
|
||||
p["sources"] = sources
|
||||
p["sourceMetadata"] = files["metadata"]
|
||||
p["originalSource"] = map[string]any{"provider": "BOS", "bucket": "test-bucket", "prefix": "incoming/capture-A", "partPrefix": "incoming/capture-A/" + part, "files": files}
|
||||
for _, kind := range []string{"slices", "discardedSlices"} {
|
||||
for _, raw := range p[kind].([]any) {
|
||||
item := raw.(map[string]any)
|
||||
split := item["taskSplit"].(map[string]any)
|
||||
number := func(v any) float64 {
|
||||
switch n := v.(type) {
|
||||
case float64:
|
||||
return n
|
||||
case int:
|
||||
return float64(n)
|
||||
default:
|
||||
panic("invalid fixture time")
|
||||
}
|
||||
}
|
||||
start, end := number(split["startSecond"]), number(split["endSecond"])
|
||||
split["startTimestampUs"] = int64(1e6 + start*1e6)
|
||||
split["endTimestampUs"] = int64(1e6 + end*1e6)
|
||||
if kind == "discardedSlices" {
|
||||
continue
|
||||
}
|
||||
class := item["classification"].(map[string]any)
|
||||
class["dictionary"] = map[string]any{"id": "base", "kind": "slice", "version": 1, "name": "操作", "tags": []string{"操作"}}
|
||||
class["tags"] = []string{"操作"}
|
||||
actions := item["segments"].([]any)
|
||||
for i, raw := range actions {
|
||||
a := raw.(map[string]any)
|
||||
low, high := float64(i)*(end-start)/float64(len(actions)), float64(i+1)*(end-start)/float64(len(actions))
|
||||
a["startSecond"] = low
|
||||
a["endSecond"] = high
|
||||
a["partStartSecond"] = start + low
|
||||
a["partEndSecond"] = start + high
|
||||
a["startTimestampUs"] = int64(1e6 + (start+low)*1e6)
|
||||
a["endTimestampUs"] = int64(1e6 + (start+high)*1e6)
|
||||
a["tags"] = []string{"操作"}
|
||||
a["tagSource"] = "slice"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
148
internal/deliveryindex/consumer.go
Normal file
148
internal/deliveryindex/consumer.go
Normal file
@ -0,0 +1,148 @@
|
||||
package deliveryindex
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"math"
|
||||
"path"
|
||||
"reflect"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// v4 is unreleased: only complete consumer/v1 projections may enter new batches.
|
||||
// Keep this guard at publication as well as in the package producer, so older
|
||||
// local v4 artifacts cannot silently publish incomplete classification JSON.
|
||||
func validateConsumerPart(body []byte) error {
|
||||
type sourceFile struct {
|
||||
Path string `json:"path"`
|
||||
}
|
||||
type selection struct {
|
||||
ID string `json:"id"`
|
||||
Kind string `json:"kind"`
|
||||
Version int `json:"version"`
|
||||
Name string `json:"name"`
|
||||
Tags []string `json:"tags"`
|
||||
}
|
||||
type window struct {
|
||||
Start *float64 `json:"startSecond"`
|
||||
End *float64 `json:"endSecond"`
|
||||
First *int64 `json:"startTimestampUs"`
|
||||
Last *int64 `json:"endTimestampUs"`
|
||||
}
|
||||
type action struct {
|
||||
window
|
||||
PartStart *float64 `json:"partStartSecond"`
|
||||
PartEnd *float64 `json:"partEndSecond"`
|
||||
Tags []string `json:"tags"`
|
||||
TagSource string `json:"tagSource"`
|
||||
Dictionary *selection `json:"dictionary"`
|
||||
}
|
||||
type item struct {
|
||||
Split window `json:"taskSplit"`
|
||||
Class struct {
|
||||
Dictionary *selection `json:"dictionary"`
|
||||
Tags []string `json:"tags"`
|
||||
} `json:"classification"`
|
||||
Actions []action `json:"segments"`
|
||||
}
|
||||
var p struct {
|
||||
Contract string `json:"consumerContract"`
|
||||
Capture struct {
|
||||
Prefix string `json:"prefix"`
|
||||
} `json:"capture"`
|
||||
PartID string `json:"partId"`
|
||||
Original struct {
|
||||
Provider string `json:"provider"`
|
||||
Bucket string `json:"bucket"`
|
||||
Prefix string `json:"prefix"`
|
||||
PartPrefix string `json:"partPrefix"`
|
||||
Files map[string]sourceFile `json:"files"`
|
||||
} `json:"originalSource"`
|
||||
Bases map[string]string `json:"timeBases"`
|
||||
Slices []item `json:"slices"`
|
||||
Discarded []struct {
|
||||
Split window `json:"taskSplit"`
|
||||
} `json:"discardedSlices"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &p); err != nil {
|
||||
return err
|
||||
}
|
||||
fail := func() error { return fmt.Errorf("incomplete v4 classification consumer contract") }
|
||||
var refs struct {
|
||||
Sources map[string]sourceFile `json:"sources"`
|
||||
Metadata sourceFile `json:"sourceMetadata"`
|
||||
Bases map[string]string `json:"pathBases"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &refs); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, role := range []string{"leftVideo", "rightVideo", "leftPts", "rightPts", "imu"} {
|
||||
if refs.Sources[role].Path != p.Original.Files[role].Path || !strings.HasPrefix(refs.Sources[role].Path, p.PartID+"/") {
|
||||
return fail()
|
||||
}
|
||||
}
|
||||
if refs.Metadata.Path != p.Original.Files["metadata"].Path || refs.Bases["sources"] != "original_capture" || refs.Bases["sourceMetadata"] != "original_capture" || refs.Bases["calibration"] != "original_capture" {
|
||||
return fail()
|
||||
}
|
||||
if p.Contract != "droboticflow.classification-consumer/v1" || p.Original.Provider != "BOS" || p.Original.Bucket == "" || p.Original.Prefix == "" || p.Original.Prefix != p.Capture.Prefix || p.Original.PartPrefix != p.Original.Prefix+"/"+p.PartID {
|
||||
return fail()
|
||||
}
|
||||
for key, value := range map[string]string{"taskSplitSeconds": "original_left_video", "segmentSeconds": "slice_left_video", "segmentPartSeconds": "original_left_video", "timestampUs": "original_hardware_pts", "interval": "half-open", "frameRanges": "zero-based-half-open"} {
|
||||
if p.Bases[key] != value {
|
||||
return fail()
|
||||
}
|
||||
}
|
||||
for _, role := range []string{"leftVideo", "rightVideo", "leftPts", "rightPts", "imu", "cameraCalibration", "imuCalibration", "metadata"} {
|
||||
file := p.Original.Files[role].Path
|
||||
if file == "" || path.Clean(file) != file || strings.HasPrefix(file, "/") || strings.ContainsAny(file, "\\\x00") || file == ".." || strings.HasPrefix(file, "../") {
|
||||
return fail()
|
||||
}
|
||||
}
|
||||
validWindow := func(w window) bool {
|
||||
return w.Start != nil && w.End != nil && w.First != nil && w.Last != nil && !math.IsNaN(*w.Start) && !math.IsInf(*w.Start, 0) && !math.IsNaN(*w.End) && !math.IsInf(*w.End, 0) && *w.Start >= 0 && *w.End > *w.Start && *w.First > 0 && *w.Last > *w.First
|
||||
}
|
||||
validTags := func(tags []string) bool {
|
||||
seen := map[string]bool{}
|
||||
if len(tags) == 0 {
|
||||
return false
|
||||
}
|
||||
for _, tag := range tags {
|
||||
if strings.TrimSpace(tag) == "" || tag != strings.TrimSpace(tag) || seen[tag] {
|
||||
return false
|
||||
}
|
||||
seen[tag] = true
|
||||
}
|
||||
return true
|
||||
}
|
||||
for _, slice := range p.Slices {
|
||||
d := slice.Class.Dictionary
|
||||
if !validWindow(slice.Split) || d == nil || strings.TrimSpace(d.ID) == "" || strings.TrimSpace(d.Name) == "" || d.Kind != "slice" || d.Version < 1 || !validTags(d.Tags) || !reflect.DeepEqual(d.Tags, slice.Class.Tags) || len(slice.Actions) == 0 {
|
||||
return fail()
|
||||
}
|
||||
previous := 0.0
|
||||
stamp := *slice.Split.First
|
||||
for _, a := range slice.Actions {
|
||||
if !validWindow(a.window) || a.PartStart == nil || a.PartEnd == nil || !validTags(a.Tags) || math.Abs(*a.Start-previous) > 1e-6 || *a.First != stamp || *a.Last > *slice.Split.Last || math.Abs(*a.PartStart-*slice.Split.Start-*a.Start) > 1e-6 || math.Abs(*a.PartEnd-*slice.Split.Start-*a.End) > 1e-6 {
|
||||
return fail()
|
||||
}
|
||||
if a.Dictionary == nil {
|
||||
if a.TagSource != "slice" || !reflect.DeepEqual(a.Tags, d.Tags) {
|
||||
return fail()
|
||||
}
|
||||
} else if a.TagSource != "action" || a.Dictionary.Kind != "action" || strings.TrimSpace(a.Dictionary.ID) == "" || strings.TrimSpace(a.Dictionary.Name) == "" || a.Dictionary.Version < 1 || !reflect.DeepEqual(a.Tags, a.Dictionary.Tags) {
|
||||
return fail()
|
||||
}
|
||||
previous = *a.End
|
||||
stamp = *a.Last
|
||||
}
|
||||
if math.Abs(previous-(*slice.Split.End-*slice.Split.Start)) > 1e-6 || stamp != *slice.Split.Last {
|
||||
return fail()
|
||||
}
|
||||
}
|
||||
for _, item := range p.Discarded {
|
||||
if !validWindow(item.Split) {
|
||||
return fail()
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@ -29,6 +29,7 @@ type AnnotationLabelRecord struct {
|
||||
Description string `json:"description,omitempty" bson:"description,omitempty"`
|
||||
Example string `json:"example,omitempty" bson:"example,omitempty"`
|
||||
Color string `json:"color,omitempty" bson:"color,omitempty"`
|
||||
Tags []string `json:"tags,omitempty" bson:"tags,omitempty"`
|
||||
Aliases []string `json:"aliases,omitempty" bson:"aliases,omitempty"`
|
||||
Status AnnotationLabelStatus `json:"status" bson:"status"`
|
||||
Version int `json:"version" bson:"version"`
|
||||
@ -48,6 +49,7 @@ type AnnotationLabelRevision struct {
|
||||
Description string `json:"description,omitempty" bson:"description,omitempty"`
|
||||
Example string `json:"example,omitempty" bson:"example,omitempty"`
|
||||
Color string `json:"color,omitempty" bson:"color,omitempty"`
|
||||
Tags []string `json:"tags,omitempty" bson:"tags,omitempty"`
|
||||
Aliases []string `json:"aliases,omitempty" bson:"aliases,omitempty"`
|
||||
Status AnnotationLabelStatus `json:"status" bson:"status"`
|
||||
UpdatedByUserID string `json:"updatedByUserId" bson:"updatedByUserId"`
|
||||
@ -64,3 +66,23 @@ type ProjectAnnotationLabelsRecord struct {
|
||||
UpdatedByUserID string `json:"updatedByUserId" bson:"updatedByUserId"`
|
||||
UpdatedAt string `json:"updatedAt" bson:"updatedAt"`
|
||||
}
|
||||
|
||||
// DictionarySelection is an immutable business classification, never a lookup
|
||||
// against the current dictionary during review or delivery.
|
||||
type DictionarySelection struct {
|
||||
ID string `json:"id" bson:"id"`
|
||||
Kind string `json:"kind" bson:"kind"`
|
||||
Version int `json:"version" bson:"version"`
|
||||
Code string `json:"code" bson:"code"`
|
||||
Name string `json:"name" bson:"name"`
|
||||
Tags []string `json:"tags" bson:"tags"`
|
||||
}
|
||||
|
||||
func (label AnnotationLabelRecord) Selection() *DictionarySelection {
|
||||
kind := label.Kind
|
||||
if kind == "" {
|
||||
kind = "action"
|
||||
}
|
||||
return &DictionarySelection{ID: label.ID, Kind: kind, Version: label.Version,
|
||||
Code: label.Code, Name: label.Name, Tags: append([]string{}, label.Tags...)}
|
||||
}
|
||||
|
||||
@ -66,11 +66,12 @@ type ProjectAnnotationMembershipRecord struct {
|
||||
}
|
||||
|
||||
type SemanticLabelDefinition struct {
|
||||
ID string `json:"id" bson:"id"`
|
||||
Name string `json:"name" bson:"name"`
|
||||
Description string `json:"description,omitempty" bson:"description,omitempty"`
|
||||
Aliases []string `json:"aliases,omitempty" bson:"aliases,omitempty"`
|
||||
Color string `json:"color,omitempty" bson:"color,omitempty"`
|
||||
Dictionary *DictionarySelection `json:"dictionary,omitempty" bson:"dictionary,omitempty"`
|
||||
ID string `json:"id" bson:"id"`
|
||||
Name string `json:"name" bson:"name"`
|
||||
Description string `json:"description,omitempty" bson:"description,omitempty"`
|
||||
Aliases []string `json:"aliases,omitempty" bson:"aliases,omitempty"`
|
||||
Color string `json:"color,omitempty" bson:"color,omitempty"`
|
||||
}
|
||||
|
||||
type AnnotationPrelabelSettings struct {
|
||||
@ -99,20 +100,22 @@ type AnnotationTemplateRecord struct {
|
||||
}
|
||||
|
||||
type AnnotationSegment struct {
|
||||
Status string `json:"status,omitempty" bson:"status,omitempty"`
|
||||
ID string `json:"id" bson:"id"`
|
||||
LabelID string `json:"labelId" bson:"labelId"`
|
||||
Label string `json:"label" bson:"label"`
|
||||
Description string `json:"description,omitempty" bson:"description,omitempty"`
|
||||
Handedness string `json:"handedness,omitempty" bson:"handedness,omitempty"`
|
||||
Start float64 `json:"start" bson:"start"`
|
||||
End float64 `json:"end" bson:"end"`
|
||||
StartTimestampUS int64 `json:"startTimestampUs,omitempty" bson:"startTimestampUs,omitempty"`
|
||||
EndTimestampUS int64 `json:"endTimestampUs,omitempty" bson:"endTimestampUs,omitempty"`
|
||||
Confidence float64 `json:"confidence,omitempty" bson:"confidence,omitempty"`
|
||||
Phrases []string `json:"phrases,omitempty" bson:"phrases,omitempty"`
|
||||
Source string `json:"source,omitempty" bson:"source,omitempty"`
|
||||
IsInvalid bool `json:"isInvalid,omitempty" bson:"isInvalid,omitempty"`
|
||||
ClearDictionary bool `json:"clearDictionary,omitempty" bson:"-"`
|
||||
Dictionary *DictionarySelection `json:"dictionary,omitempty" bson:"dictionary,omitempty"`
|
||||
Status string `json:"status,omitempty" bson:"status,omitempty"`
|
||||
ID string `json:"id" bson:"id"`
|
||||
LabelID string `json:"labelId" bson:"labelId"`
|
||||
Label string `json:"label" bson:"label"`
|
||||
Description string `json:"description,omitempty" bson:"description,omitempty"`
|
||||
Handedness string `json:"handedness,omitempty" bson:"handedness,omitempty"`
|
||||
Start float64 `json:"start" bson:"start"`
|
||||
End float64 `json:"end" bson:"end"`
|
||||
StartTimestampUS int64 `json:"startTimestampUs,omitempty" bson:"startTimestampUs,omitempty"`
|
||||
EndTimestampUS int64 `json:"endTimestampUs,omitempty" bson:"endTimestampUs,omitempty"`
|
||||
Confidence float64 `json:"confidence,omitempty" bson:"confidence,omitempty"`
|
||||
Phrases []string `json:"phrases,omitempty" bson:"phrases,omitempty"`
|
||||
Source string `json:"source,omitempty" bson:"source,omitempty"`
|
||||
IsInvalid bool `json:"isInvalid,omitempty" bson:"isInvalid,omitempty"`
|
||||
}
|
||||
|
||||
type SliceAnnotationStatus string
|
||||
@ -124,10 +127,12 @@ const (
|
||||
)
|
||||
|
||||
type SliceSegment struct {
|
||||
DiscardReasonLabelID string `json:"discardReasonLabelId,omitempty" bson:"discardReasonLabelId,omitempty"`
|
||||
DiscardReasonVersion int `json:"discardReasonVersion,omitempty" bson:"discardReasonVersion,omitempty"`
|
||||
DiscardReasonName string `json:"discardReasonName,omitempty" bson:"discardReasonName,omitempty"`
|
||||
DiscardReasonNote string `json:"discardReasonNote,omitempty" bson:"discardReasonNote,omitempty"`
|
||||
Dictionary *DictionarySelection `json:"dictionary,omitempty" bson:"dictionary,omitempty"`
|
||||
DiscardReasonDictionary *DictionarySelection `json:"discardReasonDictionary,omitempty" bson:"discardReasonDictionary,omitempty"`
|
||||
DiscardReasonLabelID string `json:"discardReasonLabelId,omitempty" bson:"discardReasonLabelId,omitempty"`
|
||||
DiscardReasonVersion int `json:"discardReasonVersion,omitempty" bson:"discardReasonVersion,omitempty"`
|
||||
DiscardReasonName string `json:"discardReasonName,omitempty" bson:"discardReasonName,omitempty"`
|
||||
DiscardReasonNote string `json:"discardReasonNote,omitempty" bson:"discardReasonNote,omitempty"`
|
||||
|
||||
DeliveryDisposition string `json:"deliveryDisposition,omitempty" bson:"deliveryDisposition,omitempty"`
|
||||
DiscardReason string `json:"discardReason,omitempty" bson:"discardReason,omitempty"`
|
||||
|
||||
@ -65,3 +65,9 @@ type DeliveryPackageQuery struct {
|
||||
Status DeliveryPackageStatus
|
||||
Mode RunMode
|
||||
}
|
||||
|
||||
// IsInternalDeliveryAudit also recognizes the schema so historical workflow
|
||||
// output bindings cannot accidentally expose this internal artifact.
|
||||
func IsInternalDeliveryAudit(artifact ExecutionArtifact) bool {
|
||||
return artifact.PortID == "deliveryAudit" || artifact.PortName == "deliveryAudit" || artifact.Data["schemaVersion"] == "droboticflow.delivery.review-history/v2"
|
||||
}
|
||||
|
||||
@ -1292,6 +1292,7 @@ func (runtime *Runtime) releasePartitionArtifacts(
|
||||
}
|
||||
for _, review := range reviews {
|
||||
qualityReviews = append(qualityReviews, map[string]any{
|
||||
"id": review.ID, "reviewRound": review.ReviewRound, "stage": review.Stage, "confirmations": review.Confirmations, "nextRevision": review.NextRevision, "rejectTargets": review.RejectTargets,
|
||||
"taskId": review.TaskID, "revision": review.Revision, "decision": review.Decision,
|
||||
"reasonCodes": review.ReasonCodes, "problemSliceIds": review.ProblemSliceIDs,
|
||||
"comment": review.Comment, "reviewerUserId": review.ReviewerUserID, "createdAt": review.CreatedAt,
|
||||
@ -1308,12 +1309,44 @@ func (runtime *Runtime) releasePartitionArtifacts(
|
||||
}
|
||||
}
|
||||
summary := map[string]any{"schemaVersion": "droboticflow.acceptance-summary/v2", "decision": "approved", "partitionId": partition.ID, "itemCount": len(selected), "capture": captureEvidence, "qualityReviews": qualityReviews, "finalAcceptance": acceptanceEvidence}
|
||||
classificationMode, err := runtime.classificationDeliveryEnabled(ctx, run.WorkflowID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if classificationMode {
|
||||
auditItems := []domain.HumanItemRecord{}
|
||||
for _, itemID := range partition.ItemIDs {
|
||||
auditItems = append(auditItems, selected[itemID])
|
||||
}
|
||||
if acceptanceID, ok := acceptanceEvidence["humanItemId"].(string); ok && acceptanceID != "" {
|
||||
acceptedItem, err := runtime.store.GetHumanItem(ctx, acceptanceID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if acceptedItem == nil {
|
||||
return nil, fmt.Errorf("acceptance audit item missing")
|
||||
}
|
||||
auditItems = append(auditItems, *acceptedItem)
|
||||
}
|
||||
trace, err := runtime.deliveryAuditTrace(ctx, batch.OrganizationID, auditItems)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
summary["trace"] = trace
|
||||
}
|
||||
summary["acceptedMaterializations"] = acceptedMaterializations
|
||||
if batch.ProcessingProfile == domain.ProcessingProfileRawHuman {
|
||||
summary["processingProfile"] = domain.ProcessingProfileRawHuman
|
||||
}
|
||||
encoded, _ := json.Marshal(summary)
|
||||
artifacts = append(artifacts, domain.ExecutionArtifact{ID: fmt.Sprintf("%s-%s-acceptance-summary", run.ID, node.ID), RunID: run.ID, NodeID: node.ID, PortID: summaryPortID, PortName: summaryPortName, Name: "acceptance-summary.json", Type: domain.PortTypeJSON, Size: annotationArtifactSize(int64(len(encoded))), SizeBytes: int64(len(encoded)), Data: summary, Payload: &domain.ArtifactPayload{Mode: domain.ArtifactTransferModeInline, Type: domain.PortTypeJSON, Inline: summary}})
|
||||
summaryArtifact := domain.ExecutionArtifact{ID: fmt.Sprintf("%s-%s-acceptance-summary", run.ID, node.ID), RunID: run.ID, NodeID: node.ID, PortID: summaryPortID, PortName: summaryPortName, Name: "acceptance-summary.json", Type: domain.PortTypeJSON, Size: annotationArtifactSize(int64(len(encoded))), SizeBytes: int64(len(encoded)), Data: summary, Payload: &domain.ArtifactPayload{Mode: domain.ArtifactTransferModeInline, Type: domain.PortTypeJSON, Inline: summary}}
|
||||
if classificationMode {
|
||||
summaryArtifact, err = runtime.persistClassificationSummary(ctx, run, node, partition.ID, summaryArtifact, encoded)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
artifacts = append(artifacts, summaryArtifact)
|
||||
return artifacts, nil
|
||||
}
|
||||
clips := annotationClipItems(execution.InputArtifacts)
|
||||
|
||||
@ -324,7 +324,19 @@ func (runtime *Runtime) normalizeOutputArtifacts(ctx context.Context, workflow *
|
||||
// the exact already-uploaded object refs instead of copying every byte a
|
||||
// second time.
|
||||
sharedDirectoryFileRefs := map[string]domain.ArtifactReference{}
|
||||
order := make([]int, 0, len(artifacts))
|
||||
for index, artifact := range artifacts {
|
||||
if domain.IsInternalDeliveryAudit(artifact) {
|
||||
order = append(order, index)
|
||||
}
|
||||
}
|
||||
for index, artifact := range artifacts {
|
||||
if !domain.IsInternalDeliveryAudit(artifact) {
|
||||
order = append(order, index)
|
||||
}
|
||||
}
|
||||
for _, index := range order {
|
||||
artifact := artifacts[index]
|
||||
result[index] = artifact
|
||||
if artifact.Payload != nil && artifact.Payload.Mode == domain.ArtifactTransferModeRef {
|
||||
if err := domain.ValidateArtifactPayload(*artifact.Payload, artifact.Type); err != nil {
|
||||
|
||||
124
internal/worker/delivery_audit.go
Normal file
124
internal/worker/delivery_audit.go
Normal file
@ -0,0 +1,124 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/store"
|
||||
)
|
||||
|
||||
// Internal evidence is pinned with the accepted package, independently from
|
||||
// customer classification JSON. Never resolve it from a later live workspace.
|
||||
func (runtime *Runtime) deliveryAuditTrace(ctx context.Context, org string, items []domain.HumanItemRecord) ([]map[string]any, error) {
|
||||
trace := []map[string]any{}
|
||||
seen := map[string]bool{}
|
||||
for _, item := range items {
|
||||
if seen[item.ID] {
|
||||
continue
|
||||
}
|
||||
seen[item.ID] = true
|
||||
if item.OrganizationID != org || item.ReviewRound > 10000 {
|
||||
return nil, fmt.Errorf("invalid audit item scope/round")
|
||||
}
|
||||
attempts, err := runtime.store.ListHumanStageAttempts(ctx, org, item.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
events, err := runtime.store.ListHumanEvents(ctx, org, item.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
rejects, err := runtime.store.ListHumanRejects(ctx, org, domain.HumanRejectQuery{ItemID: item.ID})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
baselines := []domain.ReworkBaselineRecord{}
|
||||
handlingRecords := []domain.ReworkHandlingRecord{}
|
||||
for round := 1; round <= item.ReviewRound; round++ {
|
||||
baseline, err := runtime.store.GetReworkBaseline(ctx, store.ReworkBaselineID(item.ID, round))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
handling, err := runtime.store.GetReworkHandling(ctx, store.ReworkHandlingID(item.ID, round))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if handling != nil {
|
||||
handlingRecords = append(handlingRecords, *handling)
|
||||
}
|
||||
if baseline != nil {
|
||||
baselines = append(baselines, *baseline)
|
||||
}
|
||||
}
|
||||
trace = append(trace, map[string]any{"humanItemId": item.ID, "taskId": item.LegacyTaskID, "revision": item.CurrentRevision, "reviewRound": item.ReviewRound, "stageAttempts": attempts, "events": events, "rejects": rejects, "reworkBaselines": baselines, "reworkHandling": handlingRecords})
|
||||
}
|
||||
body, err := json.Marshal(trace)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(body) > 6<<20 {
|
||||
return nil, fmt.Errorf("internal delivery trace exceeds 6 MiB; refusing incomplete export")
|
||||
}
|
||||
return trace, nil
|
||||
}
|
||||
|
||||
func (runtime *Runtime) classificationDeliveryEnabled(ctx context.Context, workflowID string) (bool, error) {
|
||||
workflow, err := runtime.store.GetWorkflow(ctx, workflowID)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if workflow == nil {
|
||||
return false, nil
|
||||
}
|
||||
for _, node := range workflow.Nodes {
|
||||
if node.Data.OperatorID != "operator-builtin-package" {
|
||||
continue
|
||||
}
|
||||
for _, param := range node.Data.Hyperparams {
|
||||
if param.Key == "DELIVERY_JSON_V4" && param.Value == "true" {
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
|
||||
// Reuse normal internal-artifact persistence rather than exceeding the 64 KiB
|
||||
// inline transport limit with a multi-round audit. Customer publication is a
|
||||
// different path and never sees this source artifact.
|
||||
func (runtime *Runtime) persistClassificationSummary(ctx context.Context, run *domain.RunRecord, node domain.WorkflowNode, partitionID string, artifact domain.ExecutionArtifact, body []byte) (domain.ExecutionArtifact, error) {
|
||||
if len(body) > 8<<20 {
|
||||
return artifact, fmt.Errorf("acceptance audit summary exceeds 8 MiB")
|
||||
}
|
||||
if len(body) <= domain.MaxInlinePayloadBytes && !structuredJSONRequiresReference(artifact, artifact.Data) {
|
||||
return artifact, nil
|
||||
}
|
||||
if runtime.workspaceRoot == "" {
|
||||
return artifact, fmt.Errorf("classification audit requires a persistent workspace")
|
||||
}
|
||||
sum := sha256.Sum256(body)
|
||||
directory := filepath.Join(runtime.workspaceRoot, "system", "runs", run.ID, "release-audits", artifactSlug(partitionID), hex.EncodeToString(sum[:]))
|
||||
if err := os.MkdirAll(directory, 0700); err != nil {
|
||||
return artifact, err
|
||||
}
|
||||
artifact.Path = filepath.Join(directory, "acceptance-summary.json")
|
||||
if err := os.WriteFile(artifact.Path, body, 0600); err != nil {
|
||||
return artifact, err
|
||||
}
|
||||
artifact.Payload = nil
|
||||
workflow, err := runtime.store.GetWorkflow(ctx, run.WorkflowID)
|
||||
if err != nil {
|
||||
return artifact, err
|
||||
}
|
||||
result, err := runtime.normalizeOutputArtifacts(ctx, workflow, run, node.ID, []domain.ExecutionArtifact{artifact})
|
||||
if err != nil {
|
||||
return artifact, err
|
||||
}
|
||||
return result[0], nil
|
||||
}
|
||||
98
internal/worker/delivery_audit_test.go
Normal file
98
internal/worker/delivery_audit_test.go
Normal file
@ -0,0 +1,98 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/store"
|
||||
)
|
||||
|
||||
func TestClassificationAuditUsesInternalStorageBeyondInlineLimit(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ds := store.NewMemoryStore()
|
||||
publisher := &memoryArtifactPublisher{}
|
||||
runtime := &Runtime{store: ds, workspaceRoot: t.TempDir(), artifactPublisher: publisher}
|
||||
c := domain.StorageConnectionRecord{ID: "audit-storage", OrganizationID: "audit-org", Provider: domain.StorageProviderBOS, Bucket: "internal-audit", Endpoint: "https://bos.example.test"}
|
||||
if err := ds.UpsertStorageConnection(ctx, c); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
workflow := domain.WorkflowRecord{ID: "wf", OrganizationID: c.OrganizationID, ArtifactStorage: &domain.ArtifactStoragePolicy{ConnectionID: c.ID, Bucket: c.Bucket, PrefixTemplate: "artifacts/{runId}/{nodeId}"}}
|
||||
if err := ds.UpsertWorkflow(ctx, workflow); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
run := &domain.RunRecord{ID: "run-audit", OrganizationID: c.OrganizationID, WorkflowID: workflow.ID}
|
||||
data := map[string]any{"schemaVersion": "droboticflow.acceptance-summary/v2", "trace": []any{map[string]any{"comment": strings.Repeat("详细返工记录", 20000)}}}
|
||||
body, _ := json.Marshal(data)
|
||||
artifact := domain.ExecutionArtifact{ID: "summary", RunID: run.ID, NodeID: "human", Name: "acceptance-summary.json", PortID: "acceptanceSummary", Type: domain.PortTypeJSON, Data: data, SizeBytes: int64(len(body))}
|
||||
saved, err := runtime.persistClassificationSummary(ctx, run, domain.WorkflowNode{ID: "human"}, "partition", artifact, body)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if saved.Payload == nil || saved.Payload.Mode != domain.ArtifactTransferModeRef || saved.Payload.Ref == nil {
|
||||
t.Fatal("oversized trace remained inline")
|
||||
}
|
||||
ref := saved.Payload.Ref
|
||||
if !strings.HasPrefix(ref.Key, "artifacts/run-audit/human/") || strings.Contains(ref.Key, "deliveries/") {
|
||||
t.Fatal("audit in customer location")
|
||||
}
|
||||
raw := publisher.objects[ref.Bucket+"/"+ref.Key]
|
||||
if string(raw) != string(body) || ref.SizeBytes != int64(len(body)) || len(ref.SHA256) != 64 {
|
||||
t.Fatal("internal trace truncated or unpinned")
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassificationAuditTraceRetainsFormalReworkAndScopedEvidence(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ds := store.NewMemoryStore()
|
||||
runtime := &Runtime{store: ds}
|
||||
item := domain.HumanItemRecord{ID: "item", OrganizationID: "audit-org", LegacyTaskID: "task", CurrentRevision: 3, ReviewRound: 2}
|
||||
for round := 1; round <= 2; round++ {
|
||||
if err := ds.UpsertReworkBaseline(ctx, domain.ReworkBaselineRecord{ID: store.ReworkBaselineID(item.ID, round), OrganizationID: item.OrganizationID, ItemID: item.ID, Round: round, Problems: []domain.ReviewProblem{{ID: "problem", Text: "正式退回意见"}}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
ds.AppendHumanEvent(ctx, domain.HumanEventRecord{ID: "event", OrganizationID: item.OrganizationID, ItemID: item.ID, EventType: "review_rejected"})
|
||||
trace, err := runtime.deliveryAuditTrace(ctx, item.OrganizationID, []domain.HumanItemRecord{item})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw, _ := json.Marshal(trace)
|
||||
if !strings.Contains(string(raw), "正式退回意见") || !strings.Contains(string(raw), "review_rejected") {
|
||||
t.Fatal("formal trace lost")
|
||||
}
|
||||
if _, err := runtime.deliveryAuditTrace(ctx, "foreign", []domain.HumanItemRecord{item}); err == nil {
|
||||
t.Fatal("foreign trace accepted")
|
||||
}
|
||||
audit := domain.ExecutionArtifact{Data: compactArtifactData(map[string]any{"schemaVersion": "droboticflow.delivery.review-history/v2"})}
|
||||
if !domain.IsInternalDeliveryAudit(audit) {
|
||||
t.Fatal("compacted audit no longer excluded from final export")
|
||||
}
|
||||
}
|
||||
|
||||
func TestInternalAuditFailurePreventsCustomerDirectoryPublication(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ds := store.NewMemoryStore()
|
||||
p := &deliveryPublisher{}
|
||||
runtime := &Runtime{store: ds, artifactPublisher: p}
|
||||
connection := domain.StorageConnectionRecord{ID: "audit-store", OrganizationID: "org", Provider: domain.StorageProviderBOS, Bucket: "b"}
|
||||
ds.UpsertStorageConnection(ctx, connection)
|
||||
workflow := &domain.WorkflowRecord{ID: "wf", OrganizationID: "org", DefinitionState: domain.WorkflowDefinitionStateReleased, ArtifactStorage: &domain.ArtifactStoragePolicy{ConnectionID: connection.ID, Bucket: connection.Bucket, PrefixTemplate: "artifacts/{runId}/{nodeId}"}}
|
||||
run := &domain.RunRecord{ID: "run-audit-first", OrganizationID: "org", WorkflowID: "wf"}
|
||||
ds.UpsertCaptureRecord(ctx, domain.CaptureRecord{ID: "capture", OrganizationID: "org", RunID: run.ID, Delivery: &domain.CaptureDeliveryTarget{ConnectionID: connection.ID, Bucket: connection.Bucket, Prefix: "deliveries/项目__project/capture/part0001"}})
|
||||
directory := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(directory, "part.json"), []byte(`{"classification":"synthetic"}`), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
artifacts := []domain.ExecutionArtifact{{ID: "customer", RunID: run.ID, Name: "delivery.directory.index", Type: domain.PortTypeFile, Path: directory}, {ID: "audit", RunID: run.ID, PortID: "deliveryAudit", Type: domain.PortTypeJSON, Path: filepath.Join(directory, "missing-internal-audit.json"), Data: map[string]any{"schemaVersion": "droboticflow.delivery.review-history/v2"}}}
|
||||
if _, err := runtime.normalizeOutputArtifacts(ctx, workflow, run, "package", artifacts); err == nil {
|
||||
t.Fatal("missing audit accepted")
|
||||
}
|
||||
if len(p.published) != 0 || len(p.puts) != 0 {
|
||||
t.Fatal("customer directory became visible before audit persisted")
|
||||
}
|
||||
}
|
||||
@ -256,7 +256,7 @@ func (runtime *Runtime) syncDeliveryPackage(ctx context.Context, batch domain.Hu
|
||||
handposeIDs := make([]string, 0)
|
||||
for _, artifact := range child.Artifacts {
|
||||
artifactIDs = append(artifactIDs, artifact.ID)
|
||||
if artifact.RunID == child.ID {
|
||||
if artifact.RunID == child.ID && !domain.IsInternalDeliveryAudit(artifact) {
|
||||
finalArtifactIDs = append(finalArtifactIDs, artifact.ID)
|
||||
}
|
||||
identity := strings.ToLower(artifact.Name + " " + artifact.Path + " " + artifact.NodeID)
|
||||
|
||||
@ -5,6 +5,7 @@ import hashlib
|
||||
import bisect
|
||||
import csv
|
||||
import json
|
||||
import math
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
@ -752,9 +753,11 @@ def _part_index_discarded_slice(
|
||||
"reason": str(evidence.get("reason") or ""),
|
||||
"reasonLabelId": str(evidence.get("reasonLabelId") or ""),
|
||||
"reasonName": str(evidence.get("reasonName") or ""),
|
||||
"note": str(evidence.get("note") or ""),
|
||||
"note": str(evidence.get("note") or evidence.get("reasonNote") or ""),
|
||||
"reasonVersion": evidence.get("reasonVersion"),
|
||||
"dictionary": evidence.get("dictionary"),
|
||||
"discardedByUserId": str(evidence.get("discardedByUserId") or ""),
|
||||
"discardedAt": str(evidence.get("discardedAt") or ""),
|
||||
"discardedAt": str(evidence.get("discardedAt") or evidence.get("at") or ""),
|
||||
},
|
||||
},
|
||||
"segments": [],
|
||||
@ -843,13 +846,184 @@ def _description_status(actions: list[dict[str, Any]]) -> str:
|
||||
return "empty_confirmed"
|
||||
|
||||
|
||||
def _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload):
|
||||
"""Use the established immutable directory publisher for the two JSON files.
|
||||
def _v4_tags(selection: Any, kind: str) -> list[str]:
|
||||
if not isinstance(selection, dict) or selection.get("kind") != kind or not str(selection.get("id") or "").strip() or type(selection.get("version")) is not int or selection["version"] < 1 or not str(selection.get("name") or "").strip():
|
||||
raise RuntimeError(f"v4 missing frozen {kind} dictionary; confirm historical classification before delivery")
|
||||
tags = selection.get("tags")
|
||||
if not isinstance(tags, list) or not tags or any(not isinstance(tag, str) or not tag.strip() or tag != tag.strip() for tag in tags) or len(set(tags)) != len(tags):
|
||||
raise RuntimeError(f"v4 missing frozen {kind} tags; confirm classification before delivery")
|
||||
return list(tags)
|
||||
|
||||
|
||||
def _v4_original_sources(manifest, capture_id, part_id, capture_prefix):
|
||||
source = manifest.get("originalSource")
|
||||
if not isinstance(source, dict) or source.get("provider") != "BOS" or not str(source.get("bucket") or "").strip():
|
||||
raise RuntimeError("v4 requires frozen original BOS source selection; workspace paths cannot identify original data")
|
||||
prefix = str(source.get("prefix") or "")
|
||||
part_prefix = str(source.get("partPrefix") or "")
|
||||
if prefix != capture_prefix or part_prefix != prefix + "/" + part_id or source.get("captureUnit") != capture_id + "/" + part_id:
|
||||
raise RuntimeError("v4 original source identity disagrees with accepted capture/part")
|
||||
if not prefix or str(pathlib.PurePosixPath(prefix)) != prefix or pathlib.PurePosixPath(prefix).is_absolute() or ".." in pathlib.PurePosixPath(prefix).parts or any(c in prefix for c in ("\\", "\x00")):
|
||||
raise RuntimeError("unsafe v4 original source prefix")
|
||||
keys = source.get("objectKeys")
|
||||
if not isinstance(keys, list) or not keys or any(not isinstance(key, str) for key in keys) or len(set(keys)) != len(keys):
|
||||
raise RuntimeError("v4 requires an unambiguous original object selection")
|
||||
files = {}
|
||||
cap, part = re.escape(capture_id), re.escape(part_id)
|
||||
patterns = {
|
||||
"leftVideo": rf"{part}/{cap}_camera_left_{part}(?:_[a-z0-9_]+)?\.(?:mp4|mov|mkv)",
|
||||
"rightVideo": rf"{part}/{cap}_camera_right_{part}(?:_[a-z0-9_]+)?\.(?:mp4|mov|mkv)",
|
||||
"leftPts": rf"{part}/{cap}_camera_left_{part}(?:_[a-z0-9_]+)?_pts\.csv",
|
||||
"rightPts": rf"{part}/{cap}_camera_right_{part}(?:_[a-z0-9_]+)?_pts\.csv",
|
||||
"imu": rf"{part}/{cap}_imu_{part}\.csv",
|
||||
"cameraCalibration": rf"{cap}_calibration_camera\.yaml",
|
||||
"imuCalibration": rf"{cap}_calibration_imu\.yaml",
|
||||
"metadata": rf"meta_{cap}\.json",
|
||||
}
|
||||
for key in keys:
|
||||
if not key.startswith(prefix + "/") or str(pathlib.PurePosixPath(key)) != key or ".." in pathlib.PurePosixPath(key).parts or any(c in key for c in ("\\", "\x00")):
|
||||
raise RuntimeError("unsafe v4 original source object")
|
||||
relative = key[len(prefix) + 1:]
|
||||
if "/" in relative and not relative.startswith(part_id + "/"):
|
||||
raise RuntimeError("v4 original selection contains another part")
|
||||
for role, pattern in patterns.items():
|
||||
if re.fullmatch(pattern, relative, re.IGNORECASE):
|
||||
if role in files: raise RuntimeError(f"ambiguous v4 original {role} selection")
|
||||
files[role] = {"path": relative}
|
||||
missing = sorted(set(patterns) - files.keys())
|
||||
if missing: raise RuntimeError("v4 missing original files: " + ", ".join(missing))
|
||||
return {"provider": "BOS", "bucket": source["bucket"], "prefix": prefix, "partPrefix": part_prefix,
|
||||
"snapshotFingerprint": str(source.get("snapshotFingerprint") or ""), "files": files}
|
||||
|
||||
|
||||
def _v4_number(value, label):
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value):
|
||||
raise RuntimeError(f"v4 missing or invalid {label}")
|
||||
return float(value)
|
||||
|
||||
|
||||
def _v4_timestamp(value, label):
|
||||
if type(value) is not int or value <= 0: raise RuntimeError(f"v4 missing or invalid {label}")
|
||||
return value
|
||||
|
||||
|
||||
def _v4_consumer_payload(payload, manifest, source_root):
|
||||
"""Project immutable accepted facts into a self-contained consumer contract.
|
||||
|
||||
No dictionary lookup, source media reads, transcoding or BOS writes here.
|
||||
Hardware PTS are authoritative; media seconds are explicitly named and never
|
||||
used to manufacture missing hardware timestamps.
|
||||
"""
|
||||
if not isinstance(manifest, dict): raise RuntimeError("v4 requires accepted source manifest")
|
||||
original = _v4_original_sources(manifest, payload["capture"]["captureId"], payload["partId"], payload["capture"]["prefix"])
|
||||
payload["originalSource"] = original
|
||||
payload["consumerContract"] = "droboticflow.classification-consumer/v1"
|
||||
payload["timeBases"] = {"taskSplitSeconds": "original_left_video", "segmentSeconds": "slice_left_video",
|
||||
"segmentPartSeconds": "original_left_video", "timestampUs": "original_hardware_pts",
|
||||
"interval": "half-open", "frameRanges": "zero-based-half-open"}
|
||||
payload["pathBases"] = {"sources": "original_capture", "sourceMetadata": "original_capture", "calibration": "original_capture"}
|
||||
# Workspace sources are private transport references, not the original files.
|
||||
payload["sources"] = {kind: original["files"][kind] for kind in ("leftVideo", "rightVideo", "leftPts", "rightPts", "imu")}
|
||||
payload["sourceMetadata"]["path"] = original["files"]["metadata"]["path"]
|
||||
for item in payload["calibration"]:
|
||||
# Bind by actual calibration kind, not input ordering.
|
||||
kind = "imuCalibration" if "imu" in item["name"].lower() else "cameraCalibration" if "camera" in item["name"].lower() else ""
|
||||
if not kind: raise RuntimeError("v4 unknown calibration kind")
|
||||
item["path"] = original["files"][kind]["path"]
|
||||
rows = {str(row["sliceId"]): row for row in manifest["items"]}
|
||||
if len(rows) != len(manifest["items"]): raise RuntimeError("v4 duplicate slice identities")
|
||||
for item in payload["slices"] + payload["discardedSlices"]:
|
||||
split = item["taskSplit"]
|
||||
start, end = (_v4_number(split[k], k) for k in ("startSecond", "endSecond"))
|
||||
first, last = (_v4_timestamp(split[k], k) for k in ("startTimestampUs", "endTimestampUs"))
|
||||
if start < 0 or end <= start or last <= first: raise RuntimeError("v4 invalid accepted slice range")
|
||||
if item["classification"]["deliveryDisposition"] == "discard":
|
||||
item["sliceName"] = ""
|
||||
evidence = item["classification"]["discardEvidence"]
|
||||
selected = evidence.get("dictionary")
|
||||
if selected is not None:
|
||||
if (not isinstance(selected, dict) or selected.get("kind") != "discard_reason" or
|
||||
not selected.get("id") or not selected.get("name") or
|
||||
type(selected.get("version")) is not int or selected["version"] < 1):
|
||||
raise RuntimeError("v4 invalid frozen discard dictionary")
|
||||
for alias, field in (("reasonLabelId", "id"), ("reasonName", "name"), ("reasonVersion", "version")):
|
||||
if evidence.get(alias) not in (None, "", 0, selected[field]):
|
||||
raise RuntimeError("v4 discard reason disagrees with frozen dictionary")
|
||||
evidence[alias] = selected[field]
|
||||
continue
|
||||
parent_tags = _v4_tags(item["classification"].get("dictionary"), "slice")
|
||||
item["classification"]["tags"] = parent_tags
|
||||
row = rows[item["sliceId"]]
|
||||
sync = json.loads((source_root / row["directory"] / "synchronized-slice.json").read_text())
|
||||
seconds, stamps = sync.get("boundarySeconds", []), sync.get("boundaryTimestampUs", [])
|
||||
if seconds or stamps:
|
||||
if not isinstance(seconds, list) or not isinstance(stamps, list) or len(seconds) < 2 or len(seconds) != len(stamps):
|
||||
raise RuntimeError("v4 invalid hardware boundary mapping")
|
||||
seconds = [_v4_number(value, "boundary") for value in seconds]
|
||||
stamps = [_v4_timestamp(value, "boundary timestamp") for value in stamps]
|
||||
if (abs(seconds[0]) > 1e-7 or abs(seconds[-1] - (end-start)) > 1e-7 or
|
||||
stamps[0] != first or stamps[-1] != last or
|
||||
any(a >= b for a, b in zip(seconds, seconds[1:])) or
|
||||
any(a >= b for a, b in zip(stamps, stamps[1:]))):
|
||||
raise RuntimeError("v4 invalid hardware boundary mapping")
|
||||
def timestamp(second):
|
||||
if abs(second) < 1e-7: return first
|
||||
if abs(second - (end-start)) < 1e-7: return last
|
||||
hits = [stamps[i] for i, value in enumerate(seconds) if abs(_v4_number(value,"boundary") - second) < 1e-7]
|
||||
if len(hits) != 1: raise RuntimeError("v4 action boundary has no original PTS mapping")
|
||||
return _v4_timestamp(hits[0], "boundary timestamp")
|
||||
actions = []
|
||||
for action in item["segments"]:
|
||||
a = _v4_number(action.get("clipStartMs"), "action clipStartMs") / 1000
|
||||
b = _v4_number(action.get("clipEndMs"), "action clipEndMs") / 1000
|
||||
if a < -1e-7 or b <= a or b > end-start+1e-7: raise RuntimeError("v4 action escapes its parent slice")
|
||||
ta, tb = (_v4_timestamp(action.get(k), k) for k in ("startTimestampUs", "endTimestampUs"))
|
||||
if not first <= ta < tb <= last: raise RuntimeError("v4 action timestamps escape its parent slice")
|
||||
if seconds and (timestamp(a) != ta or timestamp(b) != tb): raise RuntimeError("v4 action seconds disagree with original PTS")
|
||||
action.update(startSecond=a, endSecond=b, partStartSecond=start+a, partEndSecond=start+b)
|
||||
own = action.get("dictionary")
|
||||
action["tags"] = _v4_tags(own, "action") if own is not None else list(parent_tags)
|
||||
action["tagSource"] = "action" if own is not None else "slice"
|
||||
actions.append(action)
|
||||
actions.sort(key=lambda a: a["startSecond"])
|
||||
complete, previous = [], 0.0
|
||||
def gap(a,b):
|
||||
ta,tb=timestamp(a),timestamp(b)
|
||||
return {"id":f"{item['sliceId']}:gap:{ta}:{tb}", "description":"", "isInvalid":False,
|
||||
"contentKind":"gap_placeholder", "startSecond":a,"endSecond":b,
|
||||
"clipStartMs":a*1000,"clipEndMs":b*1000,"partStartSecond":start+a,"partEndSecond":start+b,
|
||||
"startTimestampUs":ta,"endTimestampUs":tb,"tags":list(parent_tags),"tagSource":"slice"}
|
||||
for action in actions:
|
||||
if action["startSecond"] < previous-1e-7: raise RuntimeError("v4 overlapping actions")
|
||||
if action["startSecond"] > previous+1e-7: complete.append(gap(previous,action["startSecond"]))
|
||||
complete.append(action);previous=action["endSecond"]
|
||||
if previous < end-start-1e-7: complete.append(gap(previous,end-start))
|
||||
if not complete: raise RuntimeError("v4 kept slice has no classified action coverage")
|
||||
previous_stamp = first
|
||||
for action in complete:
|
||||
if action["startTimestampUs"] != previous_stamp or action["endTimestampUs"] <= previous_stamp:
|
||||
raise RuntimeError("v4 action PTS coverage is discontinuous")
|
||||
previous_stamp = action["endTimestampUs"]
|
||||
if previous_stamp != last: raise RuntimeError("v4 action PTS coverage is incomplete")
|
||||
item["segments"] = complete
|
||||
counts = {kind:sum(a["contentKind"]==kind for a in complete) for kind in ("annotated","gap_placeholder","invalid_marked")}
|
||||
item["classification"]["contentCounts"] = counts
|
||||
item["classification"]["descriptionStatus"] = _description_status(complete)
|
||||
if not counts["annotated"] and counts["invalid_marked"] and counts["gap_placeholder"]: item["classification"]["descriptionStatus"]="invalid_and_blank"
|
||||
item["synchronization"]["rightBoundarySeconds"] = sync.get("rightBoundarySeconds", [])
|
||||
|
||||
|
||||
def _emit_part_index_directory(run_id, node_id, output, node_data, helpers, payload, internal_audit=None, revision_manifest=None, source_root=None):
|
||||
"""Use the established immutable publisher for versioned customer JSON.
|
||||
|
||||
V3 retains its historical sibling review file; v4 stores that audit in a
|
||||
separate internal artifact before publishing the customer directory.
|
||||
|
||||
Legacy single-file publishers deliberately accept only v1/v2. A directory
|
||||
artifact preserves their ownership/conditional-publication protocol without
|
||||
teaching an old worker to misinterpret a new single-file schema.
|
||||
"""
|
||||
v4 = internal_audit is not None
|
||||
capture = dict(payload["capture"])
|
||||
prefix = pathlib.PurePosixPath(str(capture.get("prefix") or ""))
|
||||
part_id = prefix.name
|
||||
@ -860,7 +1034,7 @@ def _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload):
|
||||
if prefix.is_absolute() or ".." in prefix.parts or len(prefix.parts) < 3:
|
||||
raise RuntimeError("unsafe v3 capture identity prefix")
|
||||
capture.update(captureId=capture_id, prefix=str(prefix.parent))
|
||||
payload.update(schemaVersion="droboticflow.delivery.part-index/v3", capture=capture, partId=part_id)
|
||||
payload.update(schemaVersion="droboticflow.delivery.part-index/v4" if v4 else "droboticflow.delivery.part-index/v3", capture=capture, partId=part_id)
|
||||
payload["pathBases"] = {"sourceMetadata": "original_capture", "calibration": "original_capture", "sources": "workspace"}
|
||||
payload["sourceMetadata"] = {"path": f"meta_{capture_id}.json"}
|
||||
history = {
|
||||
@ -869,7 +1043,8 @@ def _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload):
|
||||
"qualityReviews": payload["acceptance"].pop("qualityReviews", []),
|
||||
}
|
||||
payload["reviewHistory"] = {"path": "review-history.json", "pathBase": "part_package"}
|
||||
payload["acceptance"]["finalAcceptance"].pop("confirmedByUserId", None)
|
||||
if not v4:
|
||||
payload["acceptance"]["finalAcceptance"].pop("confirmedByUserId", None)
|
||||
for item in payload["calibration"]:
|
||||
if pathlib.PurePosixPath(item["name"]).name != item["name"] or "\\" in item["name"]:
|
||||
raise RuntimeError("unsafe calibration filename")
|
||||
@ -889,6 +1064,54 @@ def _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload):
|
||||
classification["contentCounts"] = counts
|
||||
if not counts["described"] and counts["invalid_marked"] and counts["gap_placeholder"]:
|
||||
classification["descriptionStatus"] = "invalid_and_blank"
|
||||
if v4:
|
||||
if not isinstance(internal_audit.get("trace"), list):
|
||||
raise RuntimeError("DELIVERY_JSON_V4 requires complete internal trace input")
|
||||
if _output(node_data, helpers, "deliveryAudit") is None:
|
||||
raise RuntimeError("DELIVERY_JSON_V4 requires an internal deliveryAudit output")
|
||||
audit = {
|
||||
"schemaVersion": "droboticflow.delivery.review-history/v2",
|
||||
"captureId": capture_id, "partId": part_id, "revision": payload["revision"],
|
||||
"acceptance": payload.pop("acceptance"), "lineage": payload.pop("lineage"),
|
||||
"qualityReviews": internal_audit.get("qualityReviews", []),
|
||||
"trace": internal_audit.get("trace", []),
|
||||
"acceptedMaterializations": internal_audit.get("acceptedMaterializations", []),
|
||||
}
|
||||
audit["acceptance"]["finalAcceptance"] = internal_audit.get("finalAcceptance", {})
|
||||
payload.pop("reviewHistory", None)
|
||||
payload["capture"] = {"captureId": capture_id, "prefix": str(prefix.parent)}
|
||||
for item in payload["slices"] + payload["discardedSlices"]:
|
||||
classification = item["classification"]
|
||||
# Blank descriptions retain the independently selected base class.
|
||||
if classification.get("dictionary") is None:
|
||||
classification.pop("dictionary", None)
|
||||
evidence = classification.get("discardEvidence", {})
|
||||
evidence.pop("discardedAt", None)
|
||||
if evidence.get("dictionary") is None:
|
||||
evidence.pop("dictionary", None)
|
||||
counts = classification.get("contentCounts")
|
||||
if counts is not None:
|
||||
counts["annotated"] = counts.pop("described")
|
||||
for action in item["segments"]:
|
||||
if action["contentKind"] == "described": action["contentKind"] = "annotated"
|
||||
for key in ("source", "confidence", "phrases", "labelId"):
|
||||
action.pop(key, None)
|
||||
if action.get("dictionary") is None: action.pop("dictionary", None)
|
||||
audit["materializedSources"] = payload.get("sources", {})
|
||||
_v4_consumer_payload(payload, revision_manifest, source_root)
|
||||
body = (json.dumps(payload, ensure_ascii=False, sort_keys=True, indent=2) + "\n").encode("utf8")
|
||||
audit["partSha256"] = hashlib.sha256(body).hexdigest()
|
||||
if len(json.dumps(audit, ensure_ascii=False, indent=2).encode("utf8")) > 8 * 1024 * 1024:
|
||||
raise RuntimeError("internal delivery audit exceeds 8 MiB; refusing incomplete export")
|
||||
staging = pathlib.Path(tempfile.mkdtemp(prefix="part-index-v4-", dir=output))
|
||||
(staging / "part.json").write_bytes(body)
|
||||
# A sibling artifact stays in the internal run, never in the customer tree.
|
||||
audit_artifact = helpers["write_json_artifact"](run_id, node_id, output, "review-history.json", audit, output=_output(node_data, helpers, "deliveryAudit"))
|
||||
directory = helpers["build_virtual_artifact"](
|
||||
run_id, node_id, "delivery.directory.index", str(staging), "file", 0,
|
||||
data={"schemaVersion": "droboticflow.directory-index/v1", "presentation": "collection", "datasetType": "directory", "dataSource": "FILESYSTEM", "rootPath": str(staging), "sliceCount": len(payload["slices"]), "classificationOnly": True},
|
||||
output=_output(node_data, helpers, "deliveryDirectory"))
|
||||
return [directory, audit_artifact]
|
||||
staging = pathlib.Path(tempfile.mkdtemp(prefix="part-index-v3-", dir=output))
|
||||
for name, document in (("part.json", payload), ("review-history.json", history)):
|
||||
(staging / name).write_text(json.dumps(document, ensure_ascii=False, sort_keys=True, indent=2) + "\n", encoding="utf8")
|
||||
@ -944,6 +1167,7 @@ def _part_index_slice(
|
||||
"clipStartMs": _optional_float(action.get("clipStartMs")),
|
||||
"clipEndMs": _optional_float(action.get("clipEndMs")),
|
||||
"labelId": str(action.get("descriptionTemplateId") or ""),
|
||||
"dictionary": action.get("dictionary"),
|
||||
"phrases": [str(item) for item in action.get("phrases", [])] if isinstance(action.get("phrases"), list) else [],
|
||||
"source": str(action.get("source") or ""),
|
||||
"confidence": _optional_float(action.get("confidence")),
|
||||
@ -976,6 +1200,7 @@ def _part_index_slice(
|
||||
"deliveryDisposition": declared,
|
||||
"taskLabel": str(synchronization.get("taskLabel") or ""),
|
||||
"taskLabelId": str(synchronization.get("taskLabelId") or ""),
|
||||
"dictionary": synchronization.get("dictionary"),
|
||||
"descriptionStatus": _description_status(actions),
|
||||
},
|
||||
"segments": actions,
|
||||
@ -999,9 +1224,11 @@ def _part_index_slice(
|
||||
"reason": str(evidence.get("reason") or ""),
|
||||
"reasonLabelId": str(evidence.get("reasonLabelId") or ""),
|
||||
"reasonName": str(evidence.get("reasonName") or ""),
|
||||
"note": str(evidence.get("note") or ""),
|
||||
"note": str(evidence.get("note") or evidence.get("reasonNote") or ""),
|
||||
"reasonVersion": evidence.get("reasonVersion"),
|
||||
"dictionary": evidence.get("dictionary"),
|
||||
"discardedByUserId": str(evidence.get("discardedByUserId") or ""),
|
||||
"discardedAt": str(evidence.get("discardedAt") or ""),
|
||||
"discardedAt": str(evidence.get("discardedAt") or evidence.get("at") or ""),
|
||||
}
|
||||
# Pose frames deliberately stay out of the index: the customer contract for
|
||||
# this mode is the split information, and embedded per-frame pose would
|
||||
@ -1651,6 +1878,12 @@ def _run_accepted_slice_directory(
|
||||
raise RuntimeError("acceptedSliceDirectory does not prove gap-free full-video coverage")
|
||||
summary_artifact = next((item for item in inputs if _port(item) == "acceptanceSummary"), None)
|
||||
summary_payload = summary_artifact.get("data") if isinstance((summary_artifact or {}).get("data"), dict) else {}
|
||||
if summary_artifact is not None and not isinstance(summary_payload.get("finalAcceptance"), dict):
|
||||
summary_path = _materialize_input_file(summary_artifact, output / "_acceptance", helpers)
|
||||
if summary_path.stat().st_size > 8 * 1024 * 1024:
|
||||
raise RuntimeError("acceptance summary exceeds 8 MiB")
|
||||
summary_payload = json.loads(summary_path.read_text(encoding="utf8"))
|
||||
if not isinstance(summary_payload, dict): raise RuntimeError("invalid acceptance summary reference")
|
||||
final_acceptance = summary_payload.get("finalAcceptance") if isinstance(summary_payload.get("finalAcceptance"), dict) else {}
|
||||
capture_record = _part_index_capture_record(summary_payload, revision_manifest)
|
||||
all_slices = revision_manifest.get("items", [])
|
||||
@ -1669,7 +1902,11 @@ def _run_accepted_slice_directory(
|
||||
discarded_slices.append(row)
|
||||
else:
|
||||
kept_slices.append(row)
|
||||
if disposition_contract:
|
||||
params = helpers["hyperparams"](node_data) if callable(helpers.get("hyperparams")) else {}
|
||||
classification_v4 = _bool(params, "DELIVERY_JSON_V4", False)
|
||||
if classification_v4 and not coverage_required:
|
||||
raise RuntimeError("DELIVERY_JSON_V4 requires verified full-video coverage")
|
||||
if disposition_contract or classification_v4:
|
||||
pins = summary_payload.get("acceptedMaterializations", [])
|
||||
expected_sha = _sha256_file(revision_manifest_path)
|
||||
if summary_payload.get("decision") != "approved" or final_acceptance.get("confirmed") is not True or not any(
|
||||
@ -1679,7 +1916,7 @@ def _run_accepted_slice_directory(
|
||||
raise RuntimeError("slice dispositions are not pinned to the accepted manifest revision")
|
||||
if int(coverage.get("sliceCount", -1)) != len(all_slices):
|
||||
raise RuntimeError("review coverage slice count mismatch")
|
||||
if disposition_contract and not kept_slices:
|
||||
if disposition_contract and not kept_slices and not _bool(params, "DELIVERY_JSON_V4", False):
|
||||
if any(p.get("key") == "PROCESSING_PROFILE" and p.get("value") == "raw_human" for p in node_data.get("hyperparams", [])) and (summary_payload.get("processingProfile") != "raw_human" or final_acceptance.get("processingProfile") != "raw_human"):
|
||||
raise RuntimeError("raw-human package requires matching accepted task profile")
|
||||
# Internal evidence only: deliberately no deliveryDirectory output and
|
||||
@ -1697,7 +1934,6 @@ def _run_accepted_slice_directory(
|
||||
payloads = _payload_tree(_payloads_with_full_pose_manifests(inputs, output, helpers))
|
||||
headpose = _first_payload(payloads, "droboticflow.headpose/v1")
|
||||
handpose = _first_payload(payloads, "droboticflow.wilor.hand_pose_result/v1")
|
||||
params = helpers["hyperparams"](node_data) if callable(helpers.get("hyperparams")) else {}
|
||||
if _bool(params, "PART_INDEX_DELIVERY", False):
|
||||
if source_index and not _bool(params, "SOURCE_INDEX_DELIVERY", False):
|
||||
raise RuntimeError("source-index JSON delivery requires explicit SOURCE_INDEX_DELIVERY approval; use directory delivery to preserve physical discard exclusion")
|
||||
@ -1715,8 +1951,10 @@ def _run_accepted_slice_directory(
|
||||
for item in payload["slices"] + payload["discardedSlices"]:
|
||||
item["sourceFrames"] = frames[item["sliceId"]]
|
||||
body = (json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + "\n").encode("utf8")
|
||||
if _bool(params, "DELIVERY_JSON_V4", False):
|
||||
return _emit_part_index_directory(run_id, node_id, output, node_data, helpers, payload, summary_payload, revision_manifest, source_root)
|
||||
if _bool(params, "DELIVERY_JSON_V3", False):
|
||||
return _emit_part_index_v3(run_id, node_id, output, node_data, helpers, payload)
|
||||
return _emit_part_index_directory(run_id, node_id, output, node_data, helpers, payload)
|
||||
return _emit_part_index(run_id, node_id, output, node_data, helpers, body)
|
||||
if source_index:
|
||||
source_root = _source_index_physical_export(source_root, revision_manifest, kept_slices, output)
|
||||
@ -1953,6 +2191,8 @@ def _run_multiport(
|
||||
|
||||
def run(run_id: str, node_id: str, output_dir: Any, node_data: dict[str, Any], inputs: list[dict[str, Any]], helpers: HelperMap) -> list[dict[str, Any]]:
|
||||
params = helpers["hyperparams"](node_data) if callable(helpers.get("hyperparams")) else {}
|
||||
if _bool(params, "DELIVERY_JSON_V4", False) and (not _bool(params, "PART_INDEX_DELIVERY", False) or _bool(params, "DELIVERY_JSON_V3", False)):
|
||||
raise RuntimeError("DELIVERY_JSON_V4 requires PART_INDEX_DELIVERY and cannot combine with DELIVERY_JSON_V3")
|
||||
if _bool(params, "DELIVERY_JSON_V3", False) and not _bool(params, "PART_INDEX_DELIVERY", False):
|
||||
raise RuntimeError("DELIVERY_JSON_V3 requires PART_INDEX_DELIVERY")
|
||||
# A part-index delivery intentionally has no directory port; it must still
|
||||
|
||||
@ -39,6 +39,7 @@ export function AnnotationLabelsPanel({
|
||||
const [error, setError] = useState("");
|
||||
const [editingLabelId, setEditingLabelId] = useState("");
|
||||
const [inputText, setInputText] = useState("");
|
||||
const [tagText, setTagText] = useState("");
|
||||
const [checkedIds, setCheckedIds] = useState<string[]>([]);
|
||||
const [importOpen, setImportOpen] = useState(false);
|
||||
const [importText, setImportText] = useState("");
|
||||
@ -83,6 +84,7 @@ export function AnnotationLabelsPanel({
|
||||
setEnabled([]);
|
||||
setEditingLabelId("");
|
||||
setInputText("");
|
||||
setTagText("");
|
||||
setCheckedIds([]);
|
||||
setImportOpen(false);
|
||||
setImportText("");
|
||||
@ -99,6 +101,7 @@ export function AnnotationLabelsPanel({
|
||||
const resetEditing = () => {
|
||||
setEditingLabelId("");
|
||||
setInputText("");
|
||||
setTagText("");
|
||||
};
|
||||
|
||||
const startEdit = (label: AnnotationLabelRecord) => {
|
||||
@ -106,6 +109,7 @@ export function AnnotationLabelsPanel({
|
||||
setError("");
|
||||
setEditingLabelId(label.id);
|
||||
setInputText(label.name || label.description || "");
|
||||
setTagText((label.tags || []).join(","));
|
||||
};
|
||||
|
||||
const saveLabel = async (e?: React.FormEvent) => {
|
||||
@ -133,7 +137,16 @@ export function AnnotationLabelsPanel({
|
||||
try {
|
||||
const existing = labels.find((item) => item.id === editingLabelId);
|
||||
const code = `desc_${Date.now().toString(36)}_${Math.random().toString(36).slice(2, 6)}`;
|
||||
const tags = [
|
||||
...new Set(
|
||||
tagText
|
||||
.split(/[,,\n]/)
|
||||
.map((tag) => tag.trim())
|
||||
.filter(Boolean),
|
||||
),
|
||||
];
|
||||
const payload = {
|
||||
tags,
|
||||
kind,
|
||||
code,
|
||||
name: text,
|
||||
@ -143,7 +156,7 @@ export function AnnotationLabelsPanel({
|
||||
status: existing?.status || ("active" as const),
|
||||
};
|
||||
|
||||
const saved = await reuseOrCreateDictionaryLabel({
|
||||
let saved = await reuseOrCreateDictionaryLabel({
|
||||
kind,
|
||||
name: text,
|
||||
preferredId: editingLabelId,
|
||||
@ -154,6 +167,19 @@ export function AnnotationLabelsPanel({
|
||||
});
|
||||
if (scope.current !== generation) return;
|
||||
|
||||
// Explicit tag maintenance versions the shared dictionary word; saved
|
||||
// selections retain their original tags.
|
||||
if (
|
||||
(editingLabelId === saved.id || tags.length > 0) &&
|
||||
JSON.stringify(saved.tags || []) !== JSON.stringify(tags)
|
||||
) {
|
||||
saved = await apiRequest<AnnotationLabelRecord>(
|
||||
`/api/annotation-labels/${saved.id}`,
|
||||
{ method: "PATCH" },
|
||||
{ ...saved, tags },
|
||||
);
|
||||
if (scope.current !== generation) return;
|
||||
}
|
||||
// Relink only this project; never rename a term shared by other projects.
|
||||
const selection = await apiRequest<ProjectAnnotationLabelsRecord>(
|
||||
selectionPath,
|
||||
@ -321,6 +347,14 @@ export function AnnotationLabelsPanel({
|
||||
}
|
||||
className="flex-1 min-w-[280px] rounded-lg border border-[color:var(--app-border)] bg-[color:var(--app-panel)] px-4 py-2.5 text-sm text-[color:var(--app-text)] shadow-sm focus:border-blue-500 focus:outline-none"
|
||||
/>
|
||||
<input
|
||||
aria-label="分类标签"
|
||||
placeholder="分类标签,用逗号分隔"
|
||||
value={tagText}
|
||||
onChange={(e) => setTagText(e.target.value)}
|
||||
disabled={loading || saving}
|
||||
className="min-w-48 flex-1 rounded-lg border border-[color:var(--app-border)] bg-[color:var(--app-panel)] px-3 py-2 text-sm"
|
||||
/>
|
||||
<button
|
||||
type="submit"
|
||||
disabled={loading || saving || !inputText.trim()}
|
||||
@ -508,6 +542,14 @@ export function AnnotationLabelsPanel({
|
||||
className="flex-1 text-sm text-slate-800 dark:text-gray-200 cursor-pointer select-none"
|
||||
>
|
||||
{text}
|
||||
{(item.tags || []).map((tag) => (
|
||||
<span
|
||||
key={tag}
|
||||
className="ml-2 rounded bg-blue-50 px-2 py-0.5 text-xs text-blue-700 dark:bg-blue-900/30 dark:text-blue-200"
|
||||
>
|
||||
{tag}
|
||||
</span>
|
||||
))}
|
||||
</span>
|
||||
{canManage && (
|
||||
<div className="flex items-center gap-1 shrink-0">
|
||||
@ -537,9 +579,7 @@ export function AnnotationLabelsPanel({
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
<p className="text-sm text-slate-500">
|
||||
支持导出 CSV、批量导入与勾选批量删除;编辑或删除只影响本项目的可选词条,不影响其他项目及已保存的标注、审核记录。
|
||||
</p>
|
||||
<p className="text-sm text-slate-500">名称增删只影响本项目;标签随字典词条共享,已保存的分类保持原版本。</p>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
|
||||
@ -4,6 +4,8 @@ import {
|
||||
sliceEditWorkspaceChanged,
|
||||
} from "../lib/slice-edit-session.js";
|
||||
import { checkSliceEditResolution } from "../lib/slice-edit-drag-quick-path.js";
|
||||
import type { DictionaryItem } from "../lib/dictionary-selection";
|
||||
import type { DictionarySelection } from "../types";
|
||||
import { formatAnnotationTime } from "../lib/annotation-time.js";
|
||||
import { userErrorMessage } from "../lib/user-error-message.js";
|
||||
import { reviewCommandRequiresWrite, reviewShortcut, reviewShortcutHelp } from "../lib/review-shortcuts.js";
|
||||
@ -156,6 +158,9 @@ type Props = ReviewViewingProps & {
|
||||
slices: SliceSegment[];
|
||||
segments: AnnotationSegment[];
|
||||
labels?: SemanticLabelDefinition[];
|
||||
actionDictionaryLoading?: boolean;
|
||||
actionDictionaryError?: string;
|
||||
onRefreshActionDictionary?: () => void;
|
||||
discardReasons?: AnnotationLabelRecord[];
|
||||
discardReasonsLoading?: boolean;
|
||||
discardReasonsAllowManual?: boolean;
|
||||
@ -163,7 +168,7 @@ type Props = ReviewViewingProps & {
|
||||
operationNotice?: string;
|
||||
discardReasonsError?: string;
|
||||
onRefreshDiscardReasons?: () => void;
|
||||
sliceDictionary?: { id: string; name: string }[];
|
||||
sliceDictionary?: DictionaryItem[];
|
||||
sliceDictionaryLoading?: boolean;
|
||||
sliceDictionaryError?: string;
|
||||
onRefreshSliceDictionary?: () => void;
|
||||
@ -220,7 +225,7 @@ type Props = ReviewViewingProps & {
|
||||
onDraftsChange?: (drafts: AnnotationDraft[]) => void;
|
||||
/** 换片通知(A01 草稿落盘触发):选中切片变化时上报;初始挂载不触发。 */
|
||||
onSelectedSliceChange?: (sliceId: string) => void;
|
||||
onSliceNameChange?: (sliceId: string, name: string) => void;
|
||||
onSliceNameChange?: (sliceId: string, name: string, dictionary?: DictionarySelection) => void;
|
||||
onSliceDispositionChange?: (
|
||||
sliceId: string,
|
||||
disposition: "keep" | "discard",
|
||||
@ -348,6 +353,9 @@ export function CaptureAnnotationWorkbench({
|
||||
slices,
|
||||
segments: providedSegments,
|
||||
labels = [],
|
||||
actionDictionaryLoading = false,
|
||||
actionDictionaryError = "",
|
||||
onRefreshActionDictionary,
|
||||
discardReasons = [],
|
||||
discardReasonsLoading = false,
|
||||
discardReasonsAllowManual = false,
|
||||
@ -2028,8 +2036,27 @@ export function CaptureAnnotationWorkbench({
|
||||
setEditingAnnotation((current) => (current ? { ...current, ...patch } : current));
|
||||
};
|
||||
|
||||
const actionDictionaryStatus = actionDictionaryLoading ? (
|
||||
<p role="status">正在加载动作字典…</p>
|
||||
) : actionDictionaryError ? (
|
||||
<div role="alert">
|
||||
{userErrorMessage(actionDictionaryError)}
|
||||
<button type="button" onClick={onRefreshActionDictionary}>
|
||||
重试加载
|
||||
</button>
|
||||
</div>
|
||||
) : !labels.length ? (
|
||||
<p>暂无字典词条,可直接填写描述。</p>
|
||||
) : undefined;
|
||||
|
||||
const chooseTemplate = (definition: SemanticLabelDefinition) => {
|
||||
updateEditingAnnotation({ labelId: "", label: "", description: definition.description?.trim() || definition.name });
|
||||
updateEditingAnnotation({
|
||||
labelId: "",
|
||||
label: "",
|
||||
dictionary: definition.dictionary,
|
||||
clearDictionary: false,
|
||||
description: definition.description?.trim() || definition.name,
|
||||
});
|
||||
setShowDictionary(false);
|
||||
focusDescriptionEditor();
|
||||
};
|
||||
@ -2255,10 +2282,12 @@ export function CaptureAnnotationWorkbench({
|
||||
if (isInvalid) {
|
||||
candidate.isInvalid = true;
|
||||
candidate.labelId = "";
|
||||
delete candidate.dictionary;
|
||||
} else {
|
||||
delete candidate.isInvalid;
|
||||
if (!description && continuousActions) {
|
||||
candidate.labelId = "";
|
||||
delete candidate.dictionary;
|
||||
delete candidate.phrases;
|
||||
delete candidate.handedness;
|
||||
}
|
||||
@ -5005,7 +5034,7 @@ export function CaptureAnnotationWorkbench({
|
||||
loading={sliceDictionaryLoading}
|
||||
error={userErrorMessage(sliceDictionaryError)}
|
||||
onRefresh={onRefreshSliceDictionary}
|
||||
onSelect={(name) => onSliceNameChange?.(selectedSlice.id, name)}
|
||||
onSelect={(name, dictionary) => onSliceNameChange?.(selectedSlice.id, name, dictionary)}
|
||||
onCloseFocus={focusWorkbench}
|
||||
/>
|
||||
)}
|
||||
@ -5752,7 +5781,7 @@ export function CaptureAnnotationWorkbench({
|
||||
text: item.description || item.name,
|
||||
searchText: `${item.name} ${item.description || ""}`,
|
||||
}))}
|
||||
status={!labels.length ? <p>暂无字典词条,可直接填写描述。</p> : undefined}
|
||||
status={actionDictionaryStatus}
|
||||
onSelect={(id) => {
|
||||
const item = labels.find((item) => item.id === id);
|
||||
if (item) chooseTemplate(item);
|
||||
@ -6029,7 +6058,10 @@ export function CaptureAnnotationWorkbench({
|
||||
disabled={isInvalid || saving}
|
||||
title="从字典选择(F / D,填写文本时不触发)"
|
||||
aria-keyshortcuts="F D"
|
||||
onClick={() => setShowDictionary((value) => !value)}
|
||||
onClick={() => {
|
||||
if (!showDictionary) onRefreshActionDictionary?.();
|
||||
setShowDictionary((value) => !value);
|
||||
}}
|
||||
className="inline-flex items-center gap-1.5 rounded border border-slate-300 px-2 py-1 text-xs text-slate-700 dark:border-slate-600 dark:text-gray-300 disabled:opacity-40 disabled:cursor-not-allowed"
|
||||
>
|
||||
<BookOpen className="h-3.5 w-3.5" />
|
||||
@ -6054,11 +6086,7 @@ export function CaptureAnnotationWorkbench({
|
||||
<p className="px-2 py-1 text-[10px] text-slate-500">
|
||||
↑ / ↓ 选择 · Enter 填入 · Esc 关闭
|
||||
</p>
|
||||
{!labels.length && (
|
||||
<p className="p-2 text-xs text-slate-500">
|
||||
暂无字典词条,可直接填写描述。
|
||||
</p>
|
||||
)}
|
||||
{actionDictionaryStatus}
|
||||
{labels.map((item) => (
|
||||
<button
|
||||
data-dictionary-option
|
||||
@ -6076,6 +6104,24 @@ export function CaptureAnnotationWorkbench({
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{!isInvalid && editingAnnotation.dictionary && (
|
||||
<div className="flex flex-wrap items-center gap-2 text-xs text-slate-600">
|
||||
<span>分类:{editingAnnotation.dictionary.name}</span>
|
||||
{editingAnnotation.dictionary.tags.map((tag) => (
|
||||
<span key={tag} className="rounded bg-blue-50 px-2 py-0.5 text-blue-700">
|
||||
{tag}
|
||||
</span>
|
||||
))}
|
||||
<button
|
||||
type="button"
|
||||
onClick={() =>
|
||||
updateEditingAnnotation({ dictionary: undefined, clearDictionary: true })
|
||||
}
|
||||
>
|
||||
取消分类
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
{isInvalid ? (
|
||||
<div className="flex min-h-24 flex-col items-center justify-center rounded border border-dashed border-amber-300 bg-amber-50/60 p-4 text-center dark:border-amber-800/60 dark:bg-amber-950/30">
|
||||
<div className="flex items-center gap-1.5 text-sm font-semibold text-amber-800 dark:text-amber-300">
|
||||
@ -6091,7 +6137,11 @@ export function CaptureAnnotationWorkbench({
|
||||
ref={descriptionInputRef}
|
||||
aria-label="动作描述"
|
||||
value={editingAnnotation.description || ""}
|
||||
onChange={(event) => updateEditingAnnotation({ description: event.target.value })}
|
||||
onChange={(event) =>
|
||||
updateEditingAnnotation({
|
||||
description: event.target.value,
|
||||
})
|
||||
}
|
||||
placeholder="直接填写详细动作描述..."
|
||||
rows={3}
|
||||
className="min-h-24 w-full resize-y rounded border border-slate-300 p-2.5 text-sm text-slate-800 outline-none focus:border-orange-400 dark:border-[#2a2b2f] dark:bg-[#1a1b1e] dark:text-gray-200"
|
||||
@ -6627,9 +6677,9 @@ export function CaptureAnnotationWorkbench({
|
||||
loading: sliceDictionaryLoading,
|
||||
error: userErrorMessage(sliceDictionaryError),
|
||||
refresh: onRefreshSliceDictionary,
|
||||
onSelect: (name) =>
|
||||
onSelect: (name, dictionary) =>
|
||||
applySliceEditPanelOp(
|
||||
setPendingSliceName(sliceEditPanel.pendingSlices, sliceEditPanel.sliceId, name),
|
||||
setPendingSliceName(sliceEditPanel.pendingSlices, sliceEditPanel.sliceId, name, dictionary),
|
||||
),
|
||||
}}
|
||||
discard={{
|
||||
|
||||
@ -1,3 +1,5 @@
|
||||
import type { DictionaryItem } from "../lib/dictionary-selection";
|
||||
import type { DictionarySelection } from "../types";
|
||||
import { useEffect, useLayoutEffect, useRef, useState } from "react";
|
||||
import type { AnnotationLabelRecord, SliceSegment } from "../types";
|
||||
import { formatAnnotationTime } from "../lib/annotation-time.js";
|
||||
@ -57,11 +59,11 @@ export function SliceEditPanel({
|
||||
onCancel: () => void;
|
||||
error?: string;
|
||||
dictionary: {
|
||||
items: { id: string; name: string }[];
|
||||
items: DictionaryItem[];
|
||||
loading?: boolean;
|
||||
error?: string;
|
||||
refresh?: () => void;
|
||||
onSelect: (name: string) => void;
|
||||
onSelect: (name: string, dictionary?: DictionarySelection) => void;
|
||||
};
|
||||
discard: {
|
||||
draft: SliceEditPanelDiscardDraft | null;
|
||||
|
||||
@ -1,3 +1,5 @@
|
||||
import { dictionarySelection, type DictionaryItem } from "../lib/dictionary-selection";
|
||||
import type { DictionarySelection } from "../types";
|
||||
import { userErrorMessage } from "../lib/user-error-message.js";
|
||||
import { useEffect, useLayoutEffect, useRef, useState } from "react";
|
||||
import { createPortal } from "react-dom";
|
||||
@ -20,13 +22,13 @@ export function SliceNameDictionaryPicker({
|
||||
inline = false,
|
||||
}: {
|
||||
key?: string;
|
||||
items: { id: string; name: string }[];
|
||||
items: DictionaryItem[];
|
||||
value: string;
|
||||
disabled?: boolean;
|
||||
loading?: boolean;
|
||||
error?: string;
|
||||
onRefresh?: () => void;
|
||||
onSelect: (name: string) => void;
|
||||
onSelect: (name: string, dictionary?: DictionarySelection) => void;
|
||||
triggerRef?: RefObject<HTMLButtonElement | null>;
|
||||
onCloseFocus?: () => void;
|
||||
portal?: boolean;
|
||||
@ -128,7 +130,7 @@ export function SliceNameDictionaryPicker({
|
||||
}
|
||||
onSelect={(id) => {
|
||||
const item = items.find((item) => item.id === id);
|
||||
if (item) onSelect(item.name);
|
||||
if (item) onSelect(item.name, dictionarySelection(item));
|
||||
close();
|
||||
}}
|
||||
onClose={close}
|
||||
@ -178,7 +180,7 @@ export function SliceNameDictionaryPicker({
|
||||
type="button"
|
||||
key={item.id}
|
||||
onClick={() => {
|
||||
onSelect(item.name);
|
||||
onSelect(item.name, dictionarySelection(item));
|
||||
close();
|
||||
}}
|
||||
className={`flex w-full items-center justify-between gap-2 rounded px-2 py-2 text-left text-sm hover:bg-orange-50 dark:hover:bg-orange-950/40 ${value === item.name ? "bg-orange-50 text-orange-700 dark:bg-orange-950/40" : ""}`}
|
||||
|
||||
@ -2,7 +2,7 @@ import { useEffect, useState } from "react";
|
||||
import { apiRequest } from "../services/api/client";
|
||||
import type { AnnotationLabelRecord } from "../types";
|
||||
|
||||
export function useProjectSliceDictionary(projectId?: string, taskId?: string) {
|
||||
export function useProjectSliceDictionary(projectId?: string, taskId?: string, kind: "slice" | "action" = "slice") {
|
||||
const [refreshKey, setRefreshKey] = useState(0);
|
||||
const [state, setState] = useState<{
|
||||
owner: string;
|
||||
@ -10,23 +10,26 @@ export function useProjectSliceDictionary(projectId?: string, taskId?: string) {
|
||||
loading: boolean;
|
||||
error: string;
|
||||
}>({ owner: "", items: [], loading: false, error: "" });
|
||||
const owner = `${projectId || ""}:${taskId || ""}`;
|
||||
const owner = `${projectId || ""}:${taskId || ""}:${kind}`;
|
||||
useEffect(() => {
|
||||
const controller = new AbortController();
|
||||
if (!projectId || !taskId) return () => controller.abort();
|
||||
setState({ owner, items: [], loading: true, error: "" });
|
||||
void apiRequest<{ items: AnnotationLabelRecord[] }>(`/api/projects/${projectId}/slice-labels`, {
|
||||
signal: controller.signal,
|
||||
})
|
||||
void apiRequest<{ items: AnnotationLabelRecord[] }>(
|
||||
`/api/projects/${projectId}/${kind === "slice" ? "slice-labels" : "annotation-labels"}`,
|
||||
{
|
||||
signal: controller.signal,
|
||||
},
|
||||
)
|
||||
.then((result) => {
|
||||
if (!controller.signal.aborted) setState({ owner, items: result.items || [], loading: false, error: "" });
|
||||
})
|
||||
.catch(() => {
|
||||
if (!controller.signal.aborted)
|
||||
setState({ owner, items: [], loading: false, error: "切片字典加载失败,仍可手动输入名称。" });
|
||||
setState({ owner, items: [], loading: false, error: "字典加载失败,请刷新重试。" });
|
||||
});
|
||||
return () => controller.abort();
|
||||
}, [projectId, taskId, owner, refreshKey]);
|
||||
}, [projectId, taskId, owner, refreshKey, kind]);
|
||||
return {
|
||||
...(state.owner === owner ? state : { items: [], loading: Boolean(projectId && taskId), error: "" }),
|
||||
refresh: () => setRefreshKey((n) => n + 1),
|
||||
|
||||
@ -27,6 +27,7 @@ export function normalizeAnnotationSegments(segments, clipStart, clipEnd) {
|
||||
return (Array.isArray(segments) ? segments : [])
|
||||
.map((segment, index) => ({
|
||||
id: String(segment?.id || `segment-${index + 1}`),
|
||||
dictionary: segment?.dictionary,
|
||||
labelId: String(segment?.labelId || "").trim(),
|
||||
label: String(segment?.label || "").trim(),
|
||||
description: String(segment?.description || "").trim(),
|
||||
|
||||
15
src/lib/dictionary-selection.ts
Normal file
15
src/lib/dictionary-selection.ts
Normal file
@ -0,0 +1,15 @@
|
||||
import type { AnnotationLabelRecord, DictionarySelection } from "../types";
|
||||
|
||||
export type DictionaryItem = Pick<AnnotationLabelRecord, "id" | "name"> & Partial<AnnotationLabelRecord>;
|
||||
|
||||
export function dictionarySelection(item: DictionaryItem): DictionarySelection | undefined {
|
||||
if (!item.code || !item.version) return undefined;
|
||||
return {
|
||||
id: item.id,
|
||||
kind: item.kind || "action",
|
||||
version: item.version,
|
||||
code: item.code,
|
||||
name: item.name,
|
||||
tags: [...(item.tags || [])],
|
||||
};
|
||||
}
|
||||
@ -1,3 +1,5 @@
|
||||
const dictionaryKey = (value) => value ? JSON.stringify([value.id,value.kind,value.version,value.code,value.name,value.tags || []]) : "";
|
||||
|
||||
// Candidate discovery only. The server revalidates explicit selections and
|
||||
// creates a new revision; viewing or marking an action invalid never merges.
|
||||
export function invalidSliceConsolidationCandidates(slices, actions) {
|
||||
@ -12,6 +14,8 @@ export function invalidSliceConsolidationCandidates(slices, actions) {
|
||||
!current.name?.trim() ||
|
||||
left.name !== current.name ||
|
||||
right.name !== current.name ||
|
||||
dictionaryKey(left.dictionary) !== dictionaryKey(current.dictionary) ||
|
||||
dictionaryKey(right.dictionary) !== dictionaryKey(current.dictionary) ||
|
||||
left.end !== current.start ||
|
||||
current.end !== right.start
|
||||
)
|
||||
|
||||
@ -137,10 +137,10 @@ export function mergePendingSliceBoundary(slices, sliceId, side) {
|
||||
}
|
||||
|
||||
/** ④ 切片字典描述(名称)编辑:只改目标片名称。 */
|
||||
export function setPendingSliceName(slices, sliceId, name) {
|
||||
export function setPendingSliceName(slices, sliceId, name, dictionary = undefined) {
|
||||
const index = indexOfSliceId(slices, sliceId);
|
||||
if (index < 0) return { slices, issue: "未找到目标切片。" };
|
||||
return { slices: slices.map((slice, at) => (at === index ? { ...slice, name } : slice)) };
|
||||
return { slices: slices.map((slice, at) => (at === index ? { ...slice, name, dictionary } : slice)) };
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@ -1,6 +1,8 @@
|
||||
import { createReviewDecisionCompletion } from "../lib/review-decision-completion.js";
|
||||
import { createSliceEditSession, freezeSliceEditSession } from "../lib/slice-edit-session.js";
|
||||
import { createAnnotationPersistence } from "../lib/annotation-persistence.js";
|
||||
import { dictionarySelection } from "../lib/dictionary-selection";
|
||||
import type { DictionarySelection } from "../types";
|
||||
import { AnnotationWorkModeToggle } from "../components/AnnotationWorkModeToggle";
|
||||
import { loadOwnedAnnotationProject, loadAnnotationPoolPage } from "../lib/annotation-project-loading.js";
|
||||
import { FailedSliceRecoveryNotice } from "../components/FailedSliceRecoveryNotice";
|
||||
@ -767,6 +769,7 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
const [detail, setDetailState] = useState<AnnotationTaskDetail | null>(null);
|
||||
const discardReasons = useProjectDiscardReasons(detail?.task.projectId, detail?.task.id);
|
||||
const sliceDictionary = useProjectSliceDictionary(detail?.task.projectId, detail?.task.id);
|
||||
const actionDictionary = useProjectSliceDictionary(detail?.task.projectId, detail?.task.id, "action");
|
||||
const detailRef = useRef<AnnotationTaskDetail | null>(null);
|
||||
const annotationMutationScopeRef = useRef(createAuthoritativeMutationScope());
|
||||
useEffect(() => {
|
||||
@ -1857,7 +1860,17 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
null,
|
||||
[detail?.task.templateId, templates],
|
||||
);
|
||||
const selectedLabels = detail?.task.labelSnapshot || [];
|
||||
const selectedLabels =
|
||||
(detail?.task.templateVersion || 0) >= 6
|
||||
? actionDictionary.items.map((item) => ({
|
||||
id: item.code,
|
||||
name: item.name,
|
||||
description: item.description,
|
||||
aliases: item.aliases,
|
||||
color: item.color,
|
||||
dictionary: dictionarySelection(item),
|
||||
}))
|
||||
: detail?.task.labelSnapshot || [];
|
||||
const modelPrelabels = useMemo(
|
||||
() =>
|
||||
detail?.revisions
|
||||
@ -3750,12 +3763,17 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
setSliceEditQuickConflict(prompt);
|
||||
};
|
||||
|
||||
const updateSliceName = (id: string, name: string) => {
|
||||
const updateSliceName = (id: string, name: string, dictionary?: DictionarySelection) => {
|
||||
if (!annotationAuthoringRef.current) return;
|
||||
if (!exactFrameEditing || saving || combinedSubmissionLocked || annotationMutationScopeRef.current.busy()) return;
|
||||
const slice = sliceSegments.find((item) => item.id === id);
|
||||
if (!slice || slice.deliveryDisposition === "discard" || slice.name === name) return;
|
||||
const next = sliceSegments.map((item) => (item.id === id ? { ...item, name } : item));
|
||||
if (
|
||||
!slice ||
|
||||
slice.deliveryDisposition === "discard" ||
|
||||
(slice.name === name && JSON.stringify(slice.dictionary) === JSON.stringify(dictionary))
|
||||
)
|
||||
return;
|
||||
const next = sliceSegments.map((item) => (item.id === id ? { ...item, name, dictionary } : item));
|
||||
const before = sliceEditSnapshot(id);
|
||||
editHistoryRef.current.record({ before, after: { ...before, slices: next } });
|
||||
setSliceSegments(next);
|
||||
@ -3802,6 +3820,7 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
...(recordHistory && disposition === "keep" ? { resetToPending: true } : {}),
|
||||
restorePreviousName,
|
||||
...(restorePreviousName && restoreNameFromVersion ? { restoreNameFromVersion } : {}),
|
||||
restoreClassifications: !recordHistory,
|
||||
...reason,
|
||||
sliceSegments,
|
||||
segments,
|
||||
@ -3874,6 +3893,7 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
const payload = {
|
||||
version: token.version,
|
||||
applicationId,
|
||||
restoreClassifications: !recordHistory,
|
||||
sliceSegments: nextSlices,
|
||||
segments: nextSegments,
|
||||
operations: restoredOperations,
|
||||
@ -7624,6 +7644,9 @@ export default function AnnotationOperations({ workPage = false }: { workPage?:
|
||||
slices={sliceSegments}
|
||||
segments={segments}
|
||||
labels={selectedLabels}
|
||||
actionDictionaryLoading={actionDictionary.loading}
|
||||
actionDictionaryError={actionDictionary.error}
|
||||
onRefreshActionDictionary={actionDictionary.refresh}
|
||||
sliceDictionary={sliceDictionary.items}
|
||||
sliceDictionaryLoading={sliceDictionary.loading}
|
||||
sliceDictionaryError={userErrorMessage(sliceDictionary.error)}
|
||||
|
||||
16
src/types.ts
16
src/types.ts
@ -1904,7 +1904,17 @@ export interface ObjectStorageUploadSessionSummary {
|
||||
pendingCount: number;
|
||||
}
|
||||
|
||||
export interface DictionarySelection {
|
||||
id: string;
|
||||
kind: "action" | "slice" | "discard_reason";
|
||||
version: number;
|
||||
code: string;
|
||||
name: string;
|
||||
tags: string[];
|
||||
}
|
||||
|
||||
export interface SemanticLabelDefinition {
|
||||
dictionary?: DictionarySelection;
|
||||
id: string;
|
||||
name: string;
|
||||
description?: string;
|
||||
@ -1913,6 +1923,7 @@ export interface SemanticLabelDefinition {
|
||||
}
|
||||
|
||||
export interface AnnotationLabelRecord {
|
||||
tags?: string[];
|
||||
kind?: "action" | "slice" | "discard_reason";
|
||||
requiresNote?: boolean;
|
||||
id: string;
|
||||
@ -1928,6 +1939,7 @@ export interface AnnotationLabelRecord {
|
||||
version: number;
|
||||
updatedAt: string;
|
||||
history?: Array<{
|
||||
tags?: string[];
|
||||
version: number;
|
||||
code: string;
|
||||
name: string;
|
||||
@ -1985,6 +1997,8 @@ export interface AnnotationTemplate {
|
||||
}
|
||||
|
||||
export interface AnnotationSegment {
|
||||
clearDictionary?: boolean;
|
||||
dictionary?: DictionarySelection;
|
||||
status?: "pending" | "described" | "invalid";
|
||||
id: string;
|
||||
labelId: string;
|
||||
@ -2000,6 +2014,8 @@ export interface AnnotationSegment {
|
||||
}
|
||||
|
||||
export interface SliceSegment {
|
||||
dictionary?: DictionarySelection;
|
||||
discardReasonDictionary?: DictionarySelection;
|
||||
deliveryDisposition?: "keep" | "discard";
|
||||
discardReason?: string;
|
||||
discardReasonLabelId?: string;
|
||||
|
||||
68
tests/backend/consume_delivery_v4.py
Normal file
68
tests/backend/consume_delivery_v4.py
Normal file
@ -0,0 +1,68 @@
|
||||
"""Read-only independent fixture consumer: stdlib, JSON and original files only.
|
||||
|
||||
Produces a processing plan; never cuts, copies, uploads or modifies source data.
|
||||
"""
|
||||
import argparse
|
||||
import bisect
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def inspect(part, original):
|
||||
original=original.resolve()
|
||||
def file(relative):
|
||||
p=(original/relative).resolve()
|
||||
if not p.is_relative_to(original) or not p.is_file():raise ValueError("original file unavailable or outside source root")
|
||||
return p
|
||||
assert part["consumerContract"]=="droboticflow.classification-consumer/v1"
|
||||
assert part["timeBases"]["timestampUs"]=="original_hardware_pts"
|
||||
assert part["timeBases"]["interval"]=="half-open"
|
||||
source=part["originalSource"]
|
||||
assert source["provider"]=="BOS" and source["bucket"]
|
||||
assert source["prefix"]==part["capture"]["prefix"]
|
||||
assert source["partPrefix"]==source["prefix"]+"/"+part["partId"]
|
||||
assert part["pathBases"]["sources"]=="original_capture"
|
||||
assert part["sources"]=={k:source["files"][k] for k in ("leftVideo","rightVideo","leftPts","rightPts","imu")}
|
||||
clocks={}
|
||||
for side in ("left","right"):
|
||||
file(part["sources"][side+"Video"]["path"])
|
||||
with file(part["sources"][side+"Pts"]["path"]).open() as f:clocks[side]=[int(row["timestamp_us"]) for row in csv.DictReader(f)]
|
||||
assert clocks[side] and all(a<b for a,b in zip(clocks[side],clocks[side][1:]))
|
||||
for c in part["calibration"]:assert hashlib.sha256(file(c["path"]).read_bytes()).hexdigest()==c["sha256"]
|
||||
file(part["sourceMetadata"]["path"])
|
||||
with file(part["sources"]["imu"]["path"]).open() as f:imu=[int(r["timestamp_us"]) for r in csv.DictReader(f)]
|
||||
covered={side:[] for side in clocks};actions=[];discard=[]
|
||||
def range_for(a,b):return {side:[bisect.bisect_left(pts,a),bisect.bisect_left(pts,b)] for side,pts in clocks.items()}
|
||||
for item in sorted(part["slices"]+part["discardedSlices"],key=lambda s:s["taskSplit"]["startTimestampUs"]):
|
||||
parent=item["taskSplit"];a,b=parent["startTimestampUs"],parent["endTimestampUs"]
|
||||
if item["classification"]["deliveryDisposition"]=="discard":
|
||||
assert item["segments"]==[]
|
||||
reason=item["classification"]["discardEvidence"];assert reason["reason"]
|
||||
frames=range_for(a,b);discard.append({"sliceId":item["sliceId"],"reason":reason,"frames":frames})
|
||||
for side,(start,end) in frames.items():covered[side]+=list(range(start,end))
|
||||
continue
|
||||
previous=a
|
||||
for action in item["segments"]:
|
||||
start,end=action["startTimestampUs"],action["endTimestampUs"]
|
||||
assert start==previous and start<end<=b
|
||||
previous=end
|
||||
assert abs(action["partStartSecond"]-parent["startSecond"]-action["startSecond"])<1e-6
|
||||
assert abs(action["partEndSecond"]-parent["startSecond"]-action["endSecond"])<1e-6
|
||||
dictionary=action.get("dictionary") if action["tagSource"]=="action" else item["classification"]["dictionary"]
|
||||
assert dictionary["tags"]==action["tags"] and action["tags"]
|
||||
frames=range_for(start,end)
|
||||
for side,(first,last) in frames.items():
|
||||
assert last>first
|
||||
covered[side]+=list(range(first,last))
|
||||
actions.append({"slice":item["sliceId"],"tags":action["tags"],"tagSource":action["tagSource"],"dictionaryId":dictionary["id"],"dictionaryVersion":dictionary["version"],"invalid":action["isInvalid"],"range":[action["partStartSecond"],action["partEndSecond"]],"timestampsUs":[start,end],"frames":frames,"imuRows":bisect.bisect_left(imu,end)-bisect.bisect_left(imu,start)})
|
||||
assert previous==b
|
||||
low=part["coverage"]["startTimestampUs"];high=part["coverage"]["endTimestampUs"]
|
||||
for side,pts in clocks.items():assert covered[side]==[i for i,t in enumerate(pts) if low<=t<high]
|
||||
return {"passed":True,"actions":actions,"discarded":discard,"coveredFrames":{side:len(frames) for side,frames in covered.items()},"sourceModified":False}
|
||||
|
||||
|
||||
if __name__=="__main__":
|
||||
parser=argparse.ArgumentParser(description=__doc__);parser.add_argument("part",type=Path);parser.add_argument("original",type=Path)
|
||||
args=parser.parse_args();print(json.dumps(inspect(json.loads(args.part.read_text()),args.original),ensure_ascii=False))
|
||||
39
tests/backend/export_delivery_v4_fixture.py
Normal file
39
tests/backend/export_delivery_v4_fixture.py
Normal file
@ -0,0 +1,39 @@
|
||||
"""Synthetic v4 examples from the actual package entry point, with private audit kept separately."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from backend.operators.builtin.package import operator as package
|
||||
from tests.backend.test_delivery_v4_package import NODE, ACTION, prepare, original_source
|
||||
from tests.backend.test_part_index_package import helpers
|
||||
|
||||
|
||||
def export(destination):
|
||||
destination.mkdir(parents=True, exist_ok=True)
|
||||
for part in ("part0001", "part0002"):
|
||||
work=destination/("work-"+part)
|
||||
work.mkdir()
|
||||
inputs=prepare(work,[{"id":"blank","description":"","clipStartMs":0,"clipEndMs":500,"startTimestampUs":1000000,"endTimestampUs":1500000},{"id":"pick","description":"拿起杯子","dictionary":ACTION,"clipStartMs":500,"clipEndMs":1000,"startTimestampUs":1500000,"endTimestampUs":2000000}])
|
||||
inputs[1]["data"]["capture"].update(captureId="capture-A/"+part,prefix="incoming/capture-A/"+part)
|
||||
manifest_path=Path(inputs[0]["path"])/"revision-manifest.json"
|
||||
m=json.loads(manifest_path.read_text());m["runId"]="source-run-"+part;m["originalSource"]=original_source(part=part)
|
||||
if part=="part0002":
|
||||
m["items"][0]["deliveryDisposition"]="discard"
|
||||
m["items"][0]["discardEvidence"]=m["items"][1]["discardEvidence"]
|
||||
manifest_path.write_text(json.dumps(m))
|
||||
inputs[1]["data"]["acceptedMaterializations"][0]["manifestSha256"]=hashlib.sha256(manifest_path.read_bytes()).hexdigest()
|
||||
artifacts=package.run("package-"+part,"package",work/"out",NODE,inputs,helpers())
|
||||
shutil.copytree(artifacts[0]["path"],destination/part)
|
||||
audit=destination/"internal"/part
|
||||
audit.mkdir(parents=True)
|
||||
shutil.copy2(artifacts[1]["path"],audit/"review-history.json")
|
||||
shutil.rmtree(work)
|
||||
(destination/"meta_capture-A.json").write_text('{"synthetic":true,"parts":["part0001","part0002"]}\n')
|
||||
|
||||
|
||||
if __name__=="__main__":
|
||||
parser=argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("output",type=Path)
|
||||
export(parser.parse_args().output)
|
||||
124
tests/backend/test_delivery_v4_consumer.py
Normal file
124
tests/backend/test_delivery_v4_consumer.py
Normal file
@ -0,0 +1,124 @@
|
||||
"""Consumers must classify and locate every interval without platform workspaces."""
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from backend.operators.builtin.package import operator as package
|
||||
from tests.backend.test_delivery_v4_package import ACTION, BASE, NODE, prepare, read_outputs
|
||||
from tests.backend.test_part_index_package import helpers
|
||||
from tests.backend.test_slice_discard_package import accepted_inputs
|
||||
|
||||
|
||||
def test_tags_inherit_by_dictionary_selection_not_description_text(tmp_path):
|
||||
inputs=prepare(tmp_path,[{"description":""},{"description":"人工补充文字"},{"description":"","dictionary":ACTION},{"description":"无效","isInvalid":True}])
|
||||
doc,_,_=read_outputs(tmp_path,inputs)
|
||||
actions=doc["slices"][0]["segments"]
|
||||
assert [a["tags"] for a in actions]==[BASE["tags"],BASE["tags"],ACTION["tags"],BASE["tags"]]
|
||||
assert [a["tagSource"] for a in actions]==["slice","slice","action","slice"]
|
||||
assert actions[-1]["isInvalid"] is True
|
||||
assert "dictionary" not in actions[0] # Inheritance must not invent an action selection.
|
||||
assert actions[2]["dictionary"]==ACTION
|
||||
assert doc["consumerContract"]=="droboticflow.classification-consumer/v1"
|
||||
assert doc["pathBases"]["sources"]=="original_capture"
|
||||
assert "workspaceKey" not in json.dumps(doc)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("failure,match",[("parent-tags","slice tags"),("action-tags","action tags"),("source","original BOS"),("timestamps","startTimestampUs"),("overlap","overlapping"),("clock","disagree"),("missing-file","missing original files"),("unsafe-key","unsafe"),("boundary","boundary mapping"),("pts-gap","discontinuous"),("discard-dictionary","discard reason disagrees")])
|
||||
def test_v4_refuses_incomplete_consumer_instructions(tmp_path,failure,match):
|
||||
inputs=prepare(tmp_path,[{"description":"one"},{"description":"two","dictionary":dict(ACTION)}])
|
||||
root=Path(inputs[0]["path"])
|
||||
manifest_path=root/"revision-manifest.json"
|
||||
manifest=json.loads(manifest_path.read_text())
|
||||
sync_path=root/"slices/slice-000001/synchronized-slice.json"
|
||||
sync=json.loads(sync_path.read_text())
|
||||
ann_path=root/"slices/slice-000001/annotation.json"
|
||||
ann=json.loads(ann_path.read_text())
|
||||
if failure=="parent-tags":sync["dictionary"]["tags"]=[]
|
||||
elif failure=="action-tags":ann["actions"][1]["dictionary"]["tags"]=[]
|
||||
elif failure=="source":manifest.pop("originalSource")
|
||||
elif failure=="missing-file":manifest["originalSource"]["objectKeys"]=[k for k in manifest["originalSource"]["objectKeys"] if "camera_left" not in k]
|
||||
elif failure=="timestamps":ann["actions"][0].pop("startTimestampUs")
|
||||
elif failure=="overlap":ann["actions"][1].update(clipStartMs=0,startTimestampUs=1000000)
|
||||
elif failure=="clock":ann["actions"][0]["endTimestampUs"]-=1
|
||||
elif failure=="unsafe-key":manifest["originalSource"]["objectKeys"][0]=manifest["originalSource"]["objectKeys"][0].replace("/part0001/", "//part0001/")
|
||||
elif failure=="boundary":sync["boundaryTimestampUs"][-1]-=1
|
||||
elif failure=="discard-dictionary":manifest["items"][1]["discardEvidence"]["reasonVersion"]=999
|
||||
elif failure=="pts-gap":
|
||||
sync.pop("boundarySeconds");sync.pop("boundaryTimestampUs")
|
||||
ann["actions"][1]["startTimestampUs"]+=1
|
||||
sync_path.write_text(json.dumps(sync));ann_path.write_text(json.dumps(ann));manifest_path.write_text(json.dumps(manifest))
|
||||
inputs[1]["data"]["acceptedMaterializations"][0]["manifestSha256"]=hashlib.sha256(manifest_path.read_bytes()).hexdigest()
|
||||
with pytest.raises(RuntimeError,match=match):read_outputs(tmp_path,inputs)
|
||||
|
||||
|
||||
def test_legacy_partial_annotations_keep_inherited_gap_coverage(tmp_path):
|
||||
inputs=prepare(tmp_path,[{"description":"拿起","dictionary":ACTION,"clipStartMs":250,"clipEndMs":750}])
|
||||
doc,_,_=read_outputs(tmp_path,inputs)
|
||||
actions=doc["slices"][0]["segments"]
|
||||
assert [(a["startSecond"],a["endSecond"],a["tagSource"]) for a in actions]==[(0,.25,"slice"),(.25,.75,"action"),(.75,1,"slice")]
|
||||
assert [(a["startTimestampUs"],a["endTimestampUs"]) for a in actions]==[(1000000,1250000),(1250000,1750000),(1750000,2000000)]
|
||||
|
||||
|
||||
def test_discard_aliases_use_frozen_dictionary_without_zero_version(tmp_path):
|
||||
inputs=prepare(tmp_path,[])
|
||||
manifest_path=Path(inputs[0]["path"])/"revision-manifest.json"
|
||||
manifest=json.loads(manifest_path.read_text())
|
||||
reason=manifest["items"][1]["discardEvidence"]
|
||||
reason.update(reasonLabelId="",reasonName="",reasonVersion=0)
|
||||
manifest_path.write_text(json.dumps(manifest))
|
||||
inputs[1]["data"]["acceptedMaterializations"][0]["manifestSha256"]=hashlib.sha256(manifest_path.read_bytes()).hexdigest()
|
||||
doc,_,_=read_outputs(tmp_path,inputs)
|
||||
reason=doc["discardedSlices"][0]["classification"]["discardEvidence"]
|
||||
assert reason["reasonLabelId"]==reason["dictionary"]["id"]
|
||||
assert reason["reasonVersion"]==reason["dictionary"]["version"]==4
|
||||
assert reason["reasonName"]==reason["dictionary"]["name"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode",["physical","source-index"])
|
||||
def test_real_v4_roundtrip_uses_only_customer_json_and_original_data(tmp_path,monkeypatch,mode):
|
||||
root=os.environ.get("FANYU_DELIVERY_CONSUMER_FIXTURE_DIR")
|
||||
if not root:pytest.skip("generate retained local Go source fixture first")
|
||||
context=json.loads((Path(root)/mode/"context.json").read_text())
|
||||
artifact=context["artifact"]
|
||||
inputs=accepted_inputs(artifact)[:2]
|
||||
inputs[1]["data"].update(trace=[],qualityReviews=[])
|
||||
inputs[1]["data"]["finalAcceptance"]["acceptedAt"]="2026-10-21T00:00:00Z"
|
||||
original=Path(context["original"])
|
||||
for kind in ("camera","imu"):
|
||||
p=original/f"capture-A_calibration_{kind}.yaml"
|
||||
inputs.append({"portId":"calibrations","name":p.name,"path":str(p)})
|
||||
node={**NODE,"hyperparams":NODE["hyperparams"]+[{"key":"SOURCE_INDEX_DELIVERY","value":"true"}]}
|
||||
with monkeypatch.context() as guard:
|
||||
def no_subprocess(*args, **kwargs):
|
||||
raise AssertionError("classification export must not transcode or recut source data")
|
||||
original_sha = package._sha256_file
|
||||
def metadata_only(path):
|
||||
assert Path(path).suffix not in {".mp4", ".mov", ".mkv"}, "classification export rehashed video bodies"
|
||||
return original_sha(path)
|
||||
guard.setattr(package.subprocess, "run", no_subprocess)
|
||||
guard.setattr(package.subprocess, "Popen", no_subprocess)
|
||||
guard.setattr(package, "_sha256_file", metadata_only)
|
||||
result=package.run("consumer","package",tmp_path/"out",node,inputs,helpers())
|
||||
doc=json.loads((Path(result[0]["path"])/"part.json").read_text())
|
||||
assert not any(word in json.dumps(doc) for word in ("workspaceKey","system/runs/","PRIVATE_CONNECTION"))
|
||||
# A fresh process can consume copies of only the customer file and original source.
|
||||
consumer=tmp_path/"downstream";shutil.copytree(original,consumer/"original")
|
||||
(consumer/"part.json").write_text(json.dumps(doc))
|
||||
before={p.relative_to(consumer/"original"):hashlib.sha256(p.read_bytes()).hexdigest() for p in (consumer/"original").rglob("*") if p.is_file()}
|
||||
result=subprocess.run([sys.executable,str(Path(__file__).with_name("consume_delivery_v4.py").resolve()),str(consumer/"part.json"),str(consumer/"original")],cwd=consumer,capture_output=True,text=True,check=True)
|
||||
assert before=={p.relative_to(consumer/"original"):hashlib.sha256(p.read_bytes()).hexdigest() for p in (consumer/"original").rglob("*") if p.is_file()}
|
||||
report=json.loads(result.stdout)
|
||||
assert report["passed"] and report["sourceModified"] is False
|
||||
assert report["coveredFrames"]=={"left":32,"right":32}
|
||||
groups=report["actions"]
|
||||
assert [g["tags"] for g in groups]==[["商品","手部"],["抓取"],["商品","手部"],["商品","手部"]]
|
||||
assert [g["invalid"] for g in groups]==[False,False,True,False]
|
||||
assert [g["range"] for g in groups]==[[0,.5],[.5,1],[3,3.5],[3.5,4]]
|
||||
assert doc["slices"][1]["segments"][0]["startSecond"]==0
|
||||
assert doc["slices"][1]["segments"][0]["partStartSecond"]==3
|
||||
120
tests/backend/test_delivery_v4_package.py
Normal file
120
tests/backend/test_delivery_v4_package.py
Normal file
@ -0,0 +1,120 @@
|
||||
"""Final classifications and private review history are different outputs."""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from backend.operators.builtin.package import operator as package
|
||||
from tests.backend.test_delivery_v3_package import inputs_for
|
||||
from tests.backend.test_part_index_package import helpers
|
||||
from tests.backend.test_slice_discard_package import accepted_inputs, fixture
|
||||
|
||||
NODE = {
|
||||
"outputs": [{"id": n, "name": n} for n in ("deliveryDirectory", "deliverySummary", "deliveryAudit")],
|
||||
"hyperparams": [{"key": k, "value": v} for k, v in {
|
||||
"PART_INDEX_DELIVERY": "true", "DELIVERY_JSON_V4": "true", "PROCESSING_PROFILE": "raw_human"
|
||||
}.items()],
|
||||
}
|
||||
BASE = {"id": "slice-word", "kind": "slice", "version": 2, "code": "base", "name": "物品操作", "tags": ["商品", "手部"]}
|
||||
ACTION = {"id": "action-word", "kind": "action", "version": 3, "code": "pick", "name": "拿取", "tags": ["抓取"]}
|
||||
|
||||
|
||||
def original_source(capture="capture-A", part="part0001"):
|
||||
root=f"incoming/{capture}"
|
||||
paths=[f"{part}/{capture}_camera_{side}_{part}{suffix}" for side in ("left","right") for suffix in (".mp4","_pts.csv")]
|
||||
paths += [f"{part}/{capture}_imu_{part}.csv",f"{capture}_calibration_camera.yaml",f"{capture}_calibration_imu.yaml",f"meta_{capture}.json"]
|
||||
return {"provider":"BOS","bucket":"test-bucket","prefix":root,"partPrefix":root+"/"+part,"captureUnit":capture+"/"+part,"objectKeys":[root+"/"+p for p in paths],"snapshotFingerprint":"synthetic-source-version"}
|
||||
|
||||
|
||||
def prepare(tmp_path, actions):
|
||||
inputs = inputs_for(tmp_path, actions)
|
||||
root = Path(inputs[0]["path"])
|
||||
sync = root / "slices/slice-000001/synchronized-slice.json"
|
||||
doc = json.loads(sync.read_text())
|
||||
doc["dictionary"] = BASE
|
||||
n=max(1,len(actions))
|
||||
for i,action in enumerate(actions):
|
||||
action.setdefault("id",f"action-{i}")
|
||||
action.setdefault("clipStartMs",i*1000/n)
|
||||
action.setdefault("clipEndMs",(i+1)*1000/n)
|
||||
action.setdefault("startTimestampUs",1000000+round(action["clipStartMs"]*1000))
|
||||
action.setdefault("endTimestampUs",1000000+round(action["clipEndMs"]*1000))
|
||||
(root/"slices/slice-000001/annotation.json").write_text(json.dumps({"actions":actions}))
|
||||
points=sorted({0.0,1.0,*[a[k]/1000 for a in actions for k in ("clipStartMs","clipEndMs")]})
|
||||
doc.update(boundarySeconds=points,boundaryTimestampUs=[1000000+round(p*1000000) for p in points],rightBoundarySeconds=points)
|
||||
sync.write_text(json.dumps(doc))
|
||||
manifest_path = root / "revision-manifest.json"
|
||||
m = json.loads(manifest_path.read_text())
|
||||
m["runId"] = "source-run"
|
||||
m["originalSource"] = original_source()
|
||||
m["items"][0]["sliceName"] = BASE["name"]
|
||||
reason = m["items"][1]["discardEvidence"]
|
||||
reason.update(reasonLabelId="discard-word", reasonVersion=4, reasonName="遮挡", reasonNote="持续遮挡", at="2026-10-20T00:00:00Z", dictionary={"id":"discard-word","kind":"discard_reason","version":4,"code":"occluded","name":"遮挡","tags":["不可用"]})
|
||||
manifest_path.write_text(json.dumps(m))
|
||||
inputs[1]["data"]["acceptedMaterializations"][0]["manifestSha256"] = hashlib.sha256(manifest_path.read_bytes()).hexdigest()
|
||||
inputs[1]["data"]["qualityReviews"] = [{"reviewRound":0,"decision":"rejected","comment":"PRIVATE_REWORK"},{"reviewRound":1,"decision":"approved","confirmations":[{"opinionId":"old","resolved":True}]}]
|
||||
inputs[1]["data"]["trace"] = [{"humanItemId":"item","reworkBaselines":[{"round":1,"comment":"PRIVATE_BASELINE"}],"reworkHandling":[{"round":1}],"events":[{"type":"rework_submitted"}]}]
|
||||
return inputs
|
||||
|
||||
|
||||
def read_outputs(tmp_path, inputs, out="out"):
|
||||
result = package.run("r", "p", tmp_path / out, NODE, inputs, helpers())
|
||||
assert len(result) == 2
|
||||
directory = Path(result[0]["path"])
|
||||
assert {p.name for p in directory.iterdir()} == {"part.json"}
|
||||
raw = (directory / "part.json").read_bytes()
|
||||
audit = json.loads(Path(result[1]["path"]).read_text())
|
||||
assert result[1]["portId"] == "deliveryAudit"
|
||||
assert Path(result[1]["path"]).parent != directory
|
||||
assert audit["partSha256"] == hashlib.sha256(raw).hexdigest()
|
||||
assert audit["revision"] == json.loads(raw)["revision"]
|
||||
return json.loads(raw), audit, raw
|
||||
|
||||
|
||||
@pytest.mark.parametrize("actions,kinds,status", [
|
||||
([{"description":"", "status":"pending"}], ["gap_placeholder"], "empty_confirmed"),
|
||||
([{"description":"拿起杯子", "dictionary":ACTION},{"description":""},{"description":"无效", "isInvalid":True}], ["annotated","gap_placeholder","invalid_marked"], "described"),
|
||||
([{"description":"无效", "isInvalid":True}], ["invalid_marked"], "invalid_only"),
|
||||
])
|
||||
def test_v4_freezes_classification_without_exporting_review_process(tmp_path, actions, kinds, status):
|
||||
inputs = prepare(tmp_path, actions)
|
||||
doc, audit, raw = read_outputs(tmp_path, inputs)
|
||||
first = doc["slices"][0]
|
||||
assert doc["schemaVersion"] == "droboticflow.delivery.part-index/v4"
|
||||
assert first["classification"]["dictionary"] == BASE
|
||||
assert first["classification"]["descriptionStatus"] == status
|
||||
assert [a["contentKind"] for a in first["segments"]] == kinds
|
||||
if kinds[0] == "annotated": assert first["segments"][0]["dictionary"] == ACTION
|
||||
discard = doc["discardedSlices"][0]["classification"]["discardEvidence"]
|
||||
assert discard["note"] == "持续遮挡" and discard["dictionary"]["tags"] == ["不可用"]
|
||||
assert discard["reasonVersion"] == 4
|
||||
for key in ("acceptance","reviewHistory","lineage"): assert key not in doc
|
||||
for text in (b"PRIVATE_REWORK",b"PRIVATE_BASELINE",b"reviewerUserId",b"sampling",b"discardedAt",b"confirmedByUserId"):
|
||||
assert text not in raw
|
||||
assert audit["qualityReviews"][0]["comment"] == "PRIVATE_REWORK"
|
||||
assert audit["trace"][0]["reworkBaselines"]
|
||||
assert audit["lineage"]["sourceRunId"] == "source-run"
|
||||
assert read_outputs(tmp_path, inputs, "retry")[2] == raw
|
||||
# Intermediate audit-only changes cannot change final business bytes.
|
||||
inputs[1]["data"]["qualityReviews"][0]["comment"] = "OTHER_REVIEW_NOTE"
|
||||
assert read_outputs(tmp_path, inputs, "audit-change")[2] == raw
|
||||
|
||||
|
||||
def test_v4_all_discard_outputs_classification_and_legacy_stays_unchanged(tmp_path):
|
||||
inputs = fixture(tmp_path,["discard","discard"])
|
||||
inputs[1]["data"].update(capture={"captureId":"capture-A/part0001","prefix":"incoming/capture-A/part0001"},trace=[])
|
||||
inputs[1]["data"]["finalAcceptance"]["acceptedAt"] = "2026-10-20T01:00:00Z"
|
||||
root=Path(inputs[0]["path"]);p=root/"revision-manifest.json";m=json.loads(p.read_text());m["originalSource"]=original_source();p.write_text(json.dumps(m));inputs[1]["data"]["acceptedMaterializations"][0]["manifestSha256"]=hashlib.sha256(p.read_bytes()).hexdigest()
|
||||
doc, _, _ = read_outputs(tmp_path, inputs)
|
||||
assert doc["slices"] == [] and len(doc["discardedSlices"]) == 2
|
||||
assert doc["deliveryClassification"]["discardedSliceCount"] == 2
|
||||
legacy={**NODE,"hyperparams":[p for p in NODE["hyperparams"] if p["key"]!="DELIVERY_JSON_V4"]}
|
||||
result=package.run("r","p",tmp_path/"legacy",legacy,inputs,helpers())
|
||||
assert len(result)==1 and result[0]["data"]["outcome"]=="no_deliverables"
|
||||
|
||||
|
||||
def test_v4_rejects_unaccepted_or_wrong_revision(tmp_path):
|
||||
inputs = prepare(tmp_path,[])
|
||||
inputs[1]["data"]["acceptedMaterializations"][0]["revision"] = 999
|
||||
with pytest.raises(RuntimeError,match="not pinned"):
|
||||
read_outputs(tmp_path, inputs)
|
||||
57
tests/backend/test_delivery_v4_real_materialization.py
Normal file
57
tests/backend/test_delivery_v4_real_materialization.py
Normal file
@ -0,0 +1,57 @@
|
||||
"""Consume real Go materialization revisions, including metadata-only reuse."""
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from backend.operators.builtin.package import operator as package
|
||||
from tests.backend.test_part_index_real_materialization import load_context
|
||||
from tests.backend.test_part_index_package import helpers
|
||||
from tests.backend.test_delivery_v4_package import NODE, original_source
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode", ["keep", "mixed", "discard", "blank"])
|
||||
def test_v4_real_materialization_retains_final_dictionary_snapshots(tmp_path, mode):
|
||||
context=load_context()
|
||||
artifact=context[mode]["artifact"]
|
||||
root=tmp_path/"accepted"
|
||||
shutil.copytree(artifact["path"],root)
|
||||
artifact={**artifact,"path":str(root)}
|
||||
mpath=root/"revision-manifest.json"
|
||||
manifest=json.loads(mpath.read_text())
|
||||
# Explicit synthetic original reference for this historical, local-only fixture.
|
||||
manifest["originalSource"]=original_source("real-capture")
|
||||
mpath.write_text(json.dumps(manifest))
|
||||
summary={"decision":"approved","processingProfile":"raw_human","capture":{"captureId":"real-capture/part0001","prefix":"incoming/real-capture/part0001"},"trace":[],"qualityReviews":[],"acceptedMaterializations":[{"artifactId":artifact["id"],"revision":manifest["revision"],"manifestSha256":hashlib.sha256(mpath.read_bytes()).hexdigest()}],"finalAcceptance":{"confirmed":True,"acceptedAt":"2026-10-21T00:00:00Z","processingProfile":"raw_human","confirmedByUserId":"internal-acceptor"}}
|
||||
inputs=[{**artifact,"portId":"acceptedSliceDirectory"},{"portId":"acceptanceSummary","data":summary}]
|
||||
for kind in ("camera","imu"):
|
||||
calibration=tmp_path/f"calibration_{kind}.yaml"
|
||||
calibration.write_text(f"{kind}: synthetic-calibration\n")
|
||||
inputs.append({"portId":"calibrations","name":calibration.name,"path":str(calibration)})
|
||||
# Also exercise a durable referenced acceptance summary (large trace transport).
|
||||
reference=tmp_path/"accepted.json"
|
||||
reference.write_text(json.dumps(summary))
|
||||
inputs[1]={"portId":"acceptanceSummary","path":str(reference),"data":{"schemaVersion":"droboticflow.acceptance-summary/v2"}}
|
||||
artifacts=package.run("real-package","package",tmp_path/"out",NODE,inputs,helpers())
|
||||
customer=Path(artifacts[0]["path"])
|
||||
assert {p.name for p in customer.iterdir()}=={"part.json"}
|
||||
part=json.loads((customer/"part.json").read_text())
|
||||
assert len(part["slices"])+len(part["discardedSlices"])==3
|
||||
for item in part["slices"]:
|
||||
snap=item["classification"]["dictionary"]
|
||||
assert snap["version"]==2 and snap["tags"]==["基础分类","商品"]
|
||||
if mode=="blank": assert item["classification"]["descriptionStatus"]=="empty_confirmed"
|
||||
for action in item["segments"]:
|
||||
if "dictionary" in action:
|
||||
assert action["dictionary"]["version"]==4 and action["dictionary"]["tags"]==["抓取"]
|
||||
assert action["tags"]==["抓取"] and action["tagSource"]=="action"
|
||||
else:
|
||||
assert mode=="blank"
|
||||
assert action["tags"]==snap["tags"] and action["tagSource"]=="slice"
|
||||
for item in part["discardedSlices"]:
|
||||
reason=item["classification"]["discardEvidence"]
|
||||
assert reason["note"]=="画面被遮挡" and reason["dictionary"]["tags"]==["不可用"]
|
||||
audit=json.loads(Path(artifacts[1]["path"]).read_text())
|
||||
assert audit["acceptance"]["finalAcceptance"]["confirmedByUserId"]=="internal-acceptor"
|
||||
assert "internal-acceptor" not in (customer/"part.json").read_text()
|
||||
71
tests/browser/dictionary-classification-tags.mjs
Normal file
71
tests/browser/dictionary-classification-tags.mjs
Normal file
@ -0,0 +1,71 @@
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs/promises";
|
||||
import { chromium } from "playwright";
|
||||
|
||||
const base = process.env.DICTIONARY_TEST_URL || "http://127.0.0.1:5198";
|
||||
const out = process.env.DICTIONARY_TEST_OUTPUT || "local-data/verification/delivery-classification-20261021/browser";
|
||||
await fs.mkdir(out, { recursive: true });
|
||||
const browser = await chromium.launch({
|
||||
executablePath: process.env.BROWSER_EXECUTABLE || "/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge",
|
||||
headless: true,
|
||||
});
|
||||
const page = await browser.newPage({ viewport: { width: 1365, height: 900 } });
|
||||
const errors = [];
|
||||
const failedRequests = [];
|
||||
page.on("requestfailed", (request) => failedRequests.push({ url: request.url(), failure: request.failure() }));
|
||||
page.on("pageerror", (error) => errors.push(error.message));
|
||||
try {
|
||||
page.setDefaultTimeout(60000);
|
||||
|
||||
let label = {
|
||||
id: "base",
|
||||
name: "物品操作",
|
||||
kind: "slice",
|
||||
code: "base",
|
||||
version: 2,
|
||||
tags: ["商品"],
|
||||
status: "active",
|
||||
};
|
||||
let patch;
|
||||
await page.route("**/api/**", async (route) => {
|
||||
const request = route.request(),
|
||||
path = new URL(request.url()).pathname;
|
||||
if (path === "/api/annotation-labels/base" && request.method() === "PATCH") {
|
||||
patch = request.postDataJSON();
|
||||
label = { ...label, ...patch, version: label.version + 1 };
|
||||
return route.fulfill({ json: label });
|
||||
}
|
||||
if (path === "/api/annotation-labels") return route.fulfill({ json: { items: [label] } });
|
||||
if (path === "/api/projects/project/slice-labels")
|
||||
return route.fulfill({ json: { labelIds: [label.id], items: [label] } });
|
||||
return route.fulfill({ status: 404, json: { error: "unexpected route" } });
|
||||
});
|
||||
await page.goto(base + "/tests/browser/fixtures/dictionary-classification-tags.html", { waitUntil: "commit" });
|
||||
await page.getByRole("button", { name: "编辑切片名称", exact: true }).click();
|
||||
await page.getByRole("textbox", { name: "分类标签" }).fill("商品,手部, 商品");
|
||||
await page.getByRole("button", { name: "保存修改", exact: true }).click();
|
||||
await page.getByText("手部", { exact: true }).waitFor();
|
||||
assert.deepEqual(patch.tags, ["商品", "手部"]);
|
||||
assert.equal(label.version, 3);
|
||||
await page.getByRole("button", { name: "从切片字典选择", exact: true }).click();
|
||||
await page.getByRole("button", { name: "物品操作", exact: true }).click();
|
||||
const selected = await page.evaluate(() => window.selected);
|
||||
assert.equal(selected.dictionary.id, "base");
|
||||
assert.equal(selected.dictionary.version, 2); // Picker represents the version actually shown, not the later edit.
|
||||
assert.deepEqual(selected.dictionary.tags, ["商品"]);
|
||||
assert.deepEqual(errors, []);
|
||||
await page.screenshot({ path: `${out}/dictionary-tags.png`, fullPage: true });
|
||||
await fs.writeFile(
|
||||
`${out}/result.json`,
|
||||
JSON.stringify({ passed: true, edited: label, selected, pageErrors: errors }, null, 2),
|
||||
);
|
||||
} catch (error) {
|
||||
await fs.writeFile(
|
||||
`${out}/failure.json`,
|
||||
JSON.stringify({ errors, failedRequests, url: page.url(), html: await page.content() }, null, 2),
|
||||
);
|
||||
await page.screenshot({ path: `${out}/failure.png`, fullPage: true });
|
||||
throw error;
|
||||
} finally {
|
||||
await browser.close();
|
||||
}
|
||||
13
tests/browser/dictionary-classification-vite.config.ts
Normal file
13
tests/browser/dictionary-classification-vite.config.ts
Normal file
@ -0,0 +1,13 @@
|
||||
import { defineConfig } from "vite";
|
||||
import react from "@vitejs/plugin-react";
|
||||
import tailwindcss from "@tailwindcss/vite";
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [react(), tailwindcss()],
|
||||
build: {
|
||||
outDir: "local-data/verification/delivery-classification-20261021/browser-ui-dist",
|
||||
emptyOutDir: true,
|
||||
rollupOptions: { input: "tests/browser/fixtures/dictionary-classification-tags.html" },
|
||||
},
|
||||
preview: { host: "127.0.0.1", port: 5198, strictPort: true },
|
||||
});
|
||||
11
tests/browser/fixtures/dictionary-classification-tags.html
Normal file
11
tests/browser/fixtures/dictionary-classification-tags.html
Normal file
@ -0,0 +1,11 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<title>字典分类验证</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
<script type="module" src="./dictionary-classification-tags.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
34
tests/browser/fixtures/dictionary-classification-tags.tsx
Normal file
34
tests/browser/fixtures/dictionary-classification-tags.tsx
Normal file
@ -0,0 +1,34 @@
|
||||
import React from "react";
|
||||
import { createRoot } from "react-dom/client";
|
||||
import "../../../src/index.css";
|
||||
import { AnnotationLabelsPanel } from "../../../src/components/AnnotationLabelsPanel";
|
||||
import { SliceNameDictionaryPicker } from "../../../src/components/SliceNameDictionaryPicker";
|
||||
import type { AnnotationLabelRecord, DictionarySelection } from "../../../src/types";
|
||||
|
||||
const label: AnnotationLabelRecord = {
|
||||
id: "base",
|
||||
name: "物品操作",
|
||||
kind: "slice",
|
||||
code: "base",
|
||||
version: 2,
|
||||
tags: ["商品"],
|
||||
status: "active",
|
||||
organizationId: "fixture",
|
||||
updatedAt: "2026-10-21T00:00:00Z",
|
||||
};
|
||||
createRoot(document.getElementById("root")!).render(
|
||||
<main className="space-y-8 p-8">
|
||||
<AnnotationLabelsPanel projectId="project" canManage kind="slice" />
|
||||
<SliceNameDictionaryPicker
|
||||
items={[label]}
|
||||
value=""
|
||||
inline
|
||||
onSelect={(name, dictionary) => {
|
||||
(window as Window & { selected?: { name: string; dictionary?: DictionarySelection } }).selected = {
|
||||
name,
|
||||
dictionary,
|
||||
};
|
||||
}}
|
||||
/>
|
||||
</main>,
|
||||
);
|
||||
@ -4,7 +4,10 @@ import fs from "node:fs/promises";
|
||||
const base = process.env.DICTIONARY_TEST_URL || "http://127.0.0.1:5178",
|
||||
out = process.env.DICTIONARY_TEST_OUTPUT || "verification/dictionary-reuse";
|
||||
await fs.mkdir(out, { recursive: true });
|
||||
const browser = await chromium.launch({ channel: "chrome", headless: true });
|
||||
const browser = await chromium.launch({
|
||||
...(process.env.BROWSER_EXECUTABLE ? { executablePath: process.env.BROWSER_EXECUTABLE } : { channel: "chrome" }),
|
||||
headless: true,
|
||||
});
|
||||
try {
|
||||
for (const kind of ["slice", "action", "discard_reason"]) {
|
||||
const page = await browser.newPage({ viewport: { width: 1280, height: 900 } });
|
||||
@ -54,7 +57,7 @@ try {
|
||||
);
|
||||
await page.goto(base + "/__dictionary_reuse");
|
||||
assert.equal(await page.title(), "项目字典复用验证");
|
||||
const input = page.getByRole("textbox"),
|
||||
const input = page.getByRole("textbox").first(),
|
||||
add = page.getByRole("button", {
|
||||
name: kind === "slice" ? "添加名称" : kind === "discard_reason" ? "添加原因" : "添加描述",
|
||||
exact: true,
|
||||
|
||||
@ -38,3 +38,18 @@ test("never propose discarded, different-name, gapped or partially invalid parti
|
||||
test("invalid inside a single business slice stays inside it", () => {
|
||||
assert.deepEqual(candidates([{ id: "a", name: "盘点", start: 0, end: 3 }], actions), []);
|
||||
});
|
||||
|
||||
test("never merge same-name slices with distinct frozen dictionary classifications", () => {
|
||||
const dictionary = { id: "base", kind: "slice", version: 2, code: "sort", name: "盘点", tags: ["商品"] };
|
||||
const classified = slices.map((slice) => ({ ...slice, dictionary }));
|
||||
assert.equal(candidates(classified, actions).length, 1);
|
||||
for (const replacement of [{ ...dictionary, version: 3 }, { ...dictionary, tags: ["其他"] }, undefined]) {
|
||||
assert.deepEqual(
|
||||
candidates(
|
||||
classified.map((slice) => (slice.id === "c" ? { ...slice, dictionary: replacement } : slice)),
|
||||
actions,
|
||||
),
|
||||
[],
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
@ -28,10 +28,12 @@ test("theme utilities follow the application class rather than OS preference", a
|
||||
assert.match(css, /:where\(\.dark, \.dark \*\)/);
|
||||
assert.doesNotMatch(css, /prefers-color-scheme/);
|
||||
});
|
||||
test("project dictionary only shows explicitly selected descriptions and never edits shared labels", () => {
|
||||
test("project dictionary only shows selected descriptions and relinks names while versioning explicit tags", () => {
|
||||
const s = read("components/AnnotationLabelsPanel.tsx");
|
||||
assert.match(s, /enabled\.includes\(item\.id\)/);
|
||||
assert.doesNotMatch(s, /method: "PATCH"/);
|
||||
assert.match(s, /reuseOrCreateDictionaryLabel/);
|
||||
assert.match(s, /editingLabelId === saved.id \|\| tags.length > 0/);
|
||||
assert.match(s, /\{ \.\.\.saved, tags \}/);
|
||||
assert.match(s, /await apiRequest<ProjectAnnotationLabelsRecord>/);
|
||||
assert.match(s, /controller\.abort\(\)/);
|
||||
});
|
||||
|
||||
@ -2,15 +2,15 @@ import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs";
|
||||
const read = (p) => fs.readFileSync(new URL("../../" + p, import.meta.url), "utf8");
|
||||
test("slice dictionary selection only fills the existing name callback", () => {
|
||||
test("slice dictionary selection carries the displayed classification snapshot without changing boundaries", () => {
|
||||
const picker = read("src/components/SliceNameDictionaryPicker.tsx");
|
||||
assert.match(picker, /onSelect\(item.name\)/);
|
||||
assert.match(picker, /onSelect\(item.name, dictionarySelection\(item\)\)/);
|
||||
assert.doesNotMatch(picker, /apiRequest|\.start\s*=|\.end\s*=/);
|
||||
assert.match(picker, /搜索本项目切片字典/);
|
||||
assert.match(picker, /请联系管理员配置后再确认切片/);
|
||||
assert.doesNotMatch(picker, /可直接输入名称/);
|
||||
const workbench = read("src/components/CaptureAnnotationWorkbench.tsx");
|
||||
assert.match(workbench, /onSliceNameChange\?\.\(selectedSlice\.id,\s*name\)/);
|
||||
assert.match(workbench, /onSliceNameChange\?\.\(selectedSlice\.id,\s*name,\s*dictionary\)/);
|
||||
});
|
||||
test("dictionary management is separated and stale task results are hidden", () => {
|
||||
assert.match(read("src/pages/ProjectDetail.tsx"), /动作描述字典/);
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user