forked from longtao.wu/droboticflow
🧾 Migrate delivered directories to a part index additively
Add scripts/part_index_migration.py, which produces the missing part.json for a part that was delivered in the directory form. It reuses the package operator's own builder by loading the operator source the way the runner does, so a migrated index is structurally identical to an online package and byte-identical to what the same evidence would have produced, apart from a lineage.migratedFrom marker. A test compares both routes on one evidence set to keep the two from forking. The tool never writes object storage: publication has to go through the API/Worker path so the owner marker, conditional create and no-overwrite guarantees stay in effect. It refuses to run when the delivery directory lacks a completion manifest, when a slice has an unknown disposition, when the revision has no per-slice disposition evidence, and when the part already has a part.json with different bytes. Also fix a leak found while building this: the index wrote the operator's absolute workspace path into calibration[].path. It now records the logical delivery-relative path, matching the directory form's packages-manifest, and keeps the local location out of customer-facing output. Verification: 19 operator and migration tests pass. tests/backend failure set is identical to the HEAD baseline (25 failed, 829 subtests passed, 516 vs 525 passing), so no regression.
This commit is contained in:
parent
7b5604f9e0
commit
0acbe4dcd2
@ -786,7 +786,10 @@ def _part_index_calibration_refs(inputs: list[dict[str, Any]]) -> list[dict[str,
|
||||
size_bytes = pathlib.Path(path).stat().st_size
|
||||
except OSError:
|
||||
size_bytes = 0
|
||||
refs.append({"name": name, "path": path, "sha256": digest, "bytes": size_bytes})
|
||||
# `path` is a logical delivery-relative location, matching the directory
|
||||
# form's packages-manifest. An absolute workspace path would leak the
|
||||
# worker layout into a customer-facing document.
|
||||
refs.append({"name": name, "path": f"calibration/{name}", "sha256": digest, "bytes": size_bytes})
|
||||
if not refs:
|
||||
raise RuntimeError("part index delivery requires camera/IMU calibration files")
|
||||
return refs
|
||||
@ -933,7 +936,7 @@ def _part_index_slice(
|
||||
"boundaryContract": str(synchronization.get("boundaryContract") or ""),
|
||||
"boundaryTimestampUs": [int(value) for value in synchronization.get("boundaryTimestampUs", [])] if isinstance(synchronization.get("boundaryTimestampUs"), list) else [],
|
||||
"calibration": [
|
||||
{"name": item["name"], "path": item["path"], "sha256": item["sha256"]}
|
||||
{"name": item["name"], "path": item["path"], "sha256": item["sha256"], "bytes": item["bytes"]}
|
||||
for item in calibration
|
||||
],
|
||||
},
|
||||
|
||||
@ -17,6 +17,19 @@ The index carries both split levels in one document: the task-level split (`slic
|
||||
|
||||
`clips/` is absent in this mode, so consumers must branch on the completion manifest `kind` (`part-index` versus `directory`) instead of assuming a directory layout.
|
||||
|
||||
## Migrating an already delivered part
|
||||
|
||||
An already published directory delivery keeps its `clips/`, calibration and evidence files. The migration is additive: it produces the missing `part.json` and publishes it through the same frozen destination. Nothing is deleted and no existing member is rewritten.
|
||||
|
||||
`scripts/part_index_migration.py` produces the candidate index for one part from a local copy of the delivered directory plus the accepted materialization revision. It reuses the package operator's own builder, so a migrated index is byte-identical to what an online part-index package would have produced for the same evidence — apart from a `lineage.migratedFrom` provenance marker.
|
||||
|
||||
Two constraints follow from the shared owner marker:
|
||||
|
||||
- The destination must already carry `.delivery-owner.json`. A directory without an owner marker is rejected rather than adopted, so an arbitrary prefix can never be claimed.
|
||||
- If the part already has a `part.json` with different bytes, the migration refuses instead of overwriting. Resolve the part's delivery state explicitly.
|
||||
|
||||
The tool never writes object storage itself. Publication has to go through the API/Worker path so the ownership marker, conditional create and no-overwrite guarantees stay in effect.
|
||||
|
||||
## Safety and persistence
|
||||
|
||||
- API validates disjoint input/output prefixes (including other workflow listeners), rejects reserved system prefixes and probes write/delete permission before enabling intake. The project directory name is server-owned and reused across subsequent configuration updates.
|
||||
|
||||
@ -786,7 +786,10 @@ def _part_index_calibration_refs(inputs: list[dict[str, Any]]) -> list[dict[str,
|
||||
size_bytes = pathlib.Path(path).stat().st_size
|
||||
except OSError:
|
||||
size_bytes = 0
|
||||
refs.append({"name": name, "path": path, "sha256": digest, "bytes": size_bytes})
|
||||
# `path` is a logical delivery-relative location, matching the directory
|
||||
# form's packages-manifest. An absolute workspace path would leak the
|
||||
# worker layout into a customer-facing document.
|
||||
refs.append({"name": name, "path": f"calibration/{name}", "sha256": digest, "bytes": size_bytes})
|
||||
if not refs:
|
||||
raise RuntimeError("part index delivery requires camera/IMU calibration files")
|
||||
return refs
|
||||
@ -933,7 +936,7 @@ def _part_index_slice(
|
||||
"boundaryContract": str(synchronization.get("boundaryContract") or ""),
|
||||
"boundaryTimestampUs": [int(value) for value in synchronization.get("boundaryTimestampUs", [])] if isinstance(synchronization.get("boundaryTimestampUs"), list) else [],
|
||||
"calibration": [
|
||||
{"name": item["name"], "path": item["path"], "sha256": item["sha256"]}
|
||||
{"name": item["name"], "path": item["path"], "sha256": item["sha256"], "bytes": item["bytes"]}
|
||||
for item in calibration
|
||||
],
|
||||
},
|
||||
|
||||
245
scripts/part_index_migration.py
Normal file
245
scripts/part_index_migration.py
Normal file
@ -0,0 +1,245 @@
|
||||
#!/usr/bin/env python3
|
||||
"""把一个已交付的逐片目录形态交付,增量迁移为 part 级总 JSON 索引。
|
||||
|
||||
只新增 `part.json`,不删除 `clips/`、`calibration/` 或任何既有清单。
|
||||
|
||||
用法:
|
||||
|
||||
python3 scripts/part_index_migration.py --delivery-root /path/to/part0001 \\
|
||||
--revision-root /path/to/human-materializations/<item>/<task>/revision-000001 \\
|
||||
--out /tmp/part-index-dry-run
|
||||
|
||||
设计边界:
|
||||
|
||||
- 本脚本**不写对象存储**。发布必须经过 API/Worker 的 `prepareDeliveryTarget`
|
||||
通道,才能保留 `.delivery-owner.json` 归属校验、条件创建与「绝不覆盖他人交付」
|
||||
语义;直连 BOS 会绕过它们。本脚本只产出候选产物与迁移计划。
|
||||
- 复用打包算子自身的 part-index 构建逻辑(同一份源码),不复制第二套实现,
|
||||
因此迁移产物与在线打包逐字节同构。
|
||||
- 交付目录里的 `annotation.json` 与 `synchronized-slice.json` 是打包时写出的
|
||||
对外版本;索引只从它们读取已对外的事实,舍弃原因取自 `revision-manifest.json`
|
||||
与 `synchronized-slice.json` 的 `discardEvidence`。
|
||||
- 交付目录已有内容不同的 `part.json` 时直接拒绝,不覆盖、不删除任何对象。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
||||
OPERATOR_SOURCE = ROOT / "backend" / "operators" / "builtin" / "package" / "operator.py"
|
||||
PART_INDEX_SCHEMA = "droboticflow.delivery.part-index/v1"
|
||||
PART_INDEX_FILE = "part.json"
|
||||
|
||||
|
||||
def load_operator_module() -> object:
|
||||
"""Load the package operator by path, exactly like the operator runner does.
|
||||
|
||||
Only module-level helpers are used, so no runtime helper map is required.
|
||||
"""
|
||||
spec = importlib.util.spec_from_file_location("droboticflow_part_index_migration", OPERATOR_SOURCE)
|
||||
if spec is None or spec.loader is None:
|
||||
raise RuntimeError(f"cannot load package operator from {OPERATOR_SOURCE}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def sha256_file(path: pathlib.Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(1 << 20), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def load_json(path: pathlib.Path) -> dict:
|
||||
try:
|
||||
decoded = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError) as error:
|
||||
raise SystemExit(f"无法读取 {path}:{error}") from error
|
||||
if not isinstance(decoded, dict):
|
||||
raise SystemExit(f"{path} 不是 JSON 对象")
|
||||
return decoded
|
||||
|
||||
|
||||
def delivery_completion_manifest(delivery_root: pathlib.Path) -> dict:
|
||||
manifest_path = delivery_root / "manifest.json"
|
||||
if not manifest_path.is_file():
|
||||
raise SystemExit(f"{delivery_root} 不是已完成的交付目录:缺少 manifest.json")
|
||||
manifest = load_json(manifest_path)
|
||||
kind = str(manifest.get("kind") or "")
|
||||
if kind != "directory":
|
||||
raise SystemExit(f"该交付的 manifest kind={kind!r},只有 directory 形态需要迁移")
|
||||
return manifest
|
||||
|
||||
|
||||
def delivered_calibrations(delivery_root: pathlib.Path, operator: object) -> list[dict]:
|
||||
"""Reference the calibration bytes that are already delivered.
|
||||
|
||||
Builds the same artifact shape the operator's calibration helper expects,
|
||||
then reads the checksum from the delivered file itself.
|
||||
"""
|
||||
directory = delivery_root / "calibration"
|
||||
if not directory.is_dir():
|
||||
raise SystemExit("交付目录缺少 calibration/,无法生成索引")
|
||||
artifacts = []
|
||||
for path in sorted(directory.iterdir()):
|
||||
if path.is_file():
|
||||
# The helper accepts either a logical or an absolute location; it
|
||||
# records only the logical one in the index.
|
||||
artifacts.append({"name": path.name, "path": str(path), "portId": "calibrations"})
|
||||
if not artifacts:
|
||||
raise SystemExit("交付目录的 calibration/ 为空,无法生成索引")
|
||||
return operator._part_index_calibration_refs(artifacts)
|
||||
|
||||
|
||||
def resolve_pose_policy(acceptance: dict) -> dict:
|
||||
"""Recover the pose decision, or prove it was raw-human with no pose output."""
|
||||
final = acceptance.get("finalAcceptance") if isinstance(acceptance.get("finalAcceptance"), dict) else {}
|
||||
pose = final.get("poseDelivery") if isinstance(final.get("poseDelivery"), dict) else {}
|
||||
if pose.get("headpose") in {"include", "exclude"} and pose.get("handpose") in {"include", "exclude"}:
|
||||
return pose
|
||||
if final.get("processingProfile") == "raw_human":
|
||||
return {"headpose": "not_generated", "handpose": "not_generated"}
|
||||
raise SystemExit("交付证据里没有可用的姿态交付决策,拒绝猜测 HeadPose/HandPose 策略")
|
||||
|
||||
|
||||
def split_slices(revision_manifest: dict) -> tuple[list[dict], list[dict]]:
|
||||
rows = revision_manifest.get("items")
|
||||
if not isinstance(rows, list) or not rows:
|
||||
raise SystemExit("物化修订清单没有切片条目")
|
||||
kept: list[dict] = []
|
||||
discarded: list[dict] = []
|
||||
for row in rows:
|
||||
if not isinstance(row, dict):
|
||||
raise SystemExit("物化修订清单包含无效切片条目")
|
||||
disposition = str(row.get("deliveryDisposition") or "")
|
||||
if disposition == "keep":
|
||||
kept.append(row)
|
||||
elif disposition == "discard":
|
||||
if not str((row.get("discardEvidence") or {}).get("reason") or "").strip():
|
||||
raise SystemExit(f"舍弃切片 {row.get('sliceId')} 没有原因证据")
|
||||
discarded.append(row)
|
||||
else:
|
||||
raise SystemExit(f"切片 {row.get('sliceId')} 的用途未知:{disposition!r}")
|
||||
return kept, discarded
|
||||
|
||||
|
||||
def build_index(delivery_root: pathlib.Path, revision_root: pathlib.Path) -> tuple[dict, bytes]:
|
||||
operator = load_operator_module()
|
||||
delivery_completion_manifest(delivery_root)
|
||||
revision_manifest = load_json(revision_root / "revision-manifest.json")
|
||||
schema = str(revision_manifest.get("schemaVersion") or "")
|
||||
if schema not in {"droboticflow.slice-materialization/v2", "droboticflow.slice-materialization/v3"}:
|
||||
raise SystemExit(f"不支持的物化修订清单:{schema!r}")
|
||||
if schema == "droboticflow.slice-materialization/v2":
|
||||
raise SystemExit("v2 修订清单没有逐片用途证据,无法判定保留与舍弃;请先确认该 part 的交付契约")
|
||||
coverage = revision_manifest.get("coverage") if isinstance(revision_manifest.get("coverage"), dict) else {}
|
||||
kept, discarded = split_slices(revision_manifest)
|
||||
if not kept:
|
||||
raise SystemExit("该 part 没有保留切片;迁移不产生交付物")
|
||||
acceptance = load_json(delivery_root / "acceptance-record.json")
|
||||
capture_manifest = load_json(delivery_root / "capture-manifest.json")
|
||||
delivery_manifest = load_json(delivery_root / "packages-manifest.json")
|
||||
capture_record = delivery_manifest.get("capture") or capture_manifest.get("capture") or {}
|
||||
summary_payload = {
|
||||
"decision": "approved",
|
||||
"processingProfile": (acceptance.get("finalAcceptance") or {}).get("processingProfile"),
|
||||
"qualityReviews": (load_json(delivery_root / "quality-review.json") or {}).get("reviews") or [],
|
||||
"capture": capture_record,
|
||||
}
|
||||
lineage = {
|
||||
"acceptedSliceDirectoryArtifactId": delivery_manifest.get("lineage", {}).get("acceptedSliceDirectoryArtifactId"),
|
||||
"humanItemId": revision_manifest.get("humanItemId"),
|
||||
"taskId": revision_manifest.get("taskId"),
|
||||
"sourceRunId": revision_manifest.get("runId"),
|
||||
"migratedFrom": "droboticflow.delivery.manifest/v6",
|
||||
}
|
||||
return operator._build_part_index(
|
||||
revision_manifest,
|
||||
coverage,
|
||||
kept,
|
||||
revision_root,
|
||||
delivered_calibrations(delivery_root, operator),
|
||||
resolve_pose_policy(acceptance),
|
||||
None,
|
||||
None,
|
||||
summary_payload,
|
||||
acceptance.get("finalAcceptance") or {},
|
||||
capture_record,
|
||||
lineage,
|
||||
discarded,
|
||||
)
|
||||
|
||||
|
||||
def part_name_from_prefix(prefix: str) -> str:
|
||||
"""The frozen delivery prefix already ends with the part directory."""
|
||||
name = prefix.rstrip("/").rsplit("/", 1)[-1]
|
||||
return name or "part0001"
|
||||
|
||||
|
||||
def run_migration(delivery_root: pathlib.Path, revision_root: pathlib.Path, out_dir: pathlib.Path, part_name: str) -> dict:
|
||||
"""Produce the candidate index outside the delivery directory.
|
||||
|
||||
Unchanged-to-delivered content is a no-op; different content is refused so a
|
||||
migration can never silently replace what the customer already received.
|
||||
"""
|
||||
delivery_root = delivery_root.resolve()
|
||||
revision_root = revision_root.resolve()
|
||||
out_dir = out_dir.resolve()
|
||||
if not delivery_root.is_dir():
|
||||
raise SystemExit(f"--delivery-root 不存在:{delivery_root}")
|
||||
if not revision_root.is_dir():
|
||||
raise SystemExit(f"--revision-root 不存在:{revision_root}")
|
||||
if out_dir == delivery_root or delivery_root in out_dir.parents:
|
||||
raise SystemExit("--out 不能位于交付目录内;迁移只新增交付物,不改写本地副本")
|
||||
|
||||
payload, body = build_index(delivery_root, revision_root)
|
||||
existing = delivery_root / PART_INDEX_FILE
|
||||
if existing.is_file():
|
||||
if existing.read_bytes() == body:
|
||||
return {"status": "already-migrated", "path": str(existing), "sha256": sha256_file(existing)}
|
||||
raise SystemExit("交付目录已有内容不同的 part.json,拒绝覆盖;请人工确认该 part 的交付状态")
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
target = out_dir / PART_INDEX_FILE
|
||||
target.write_bytes(body)
|
||||
return {
|
||||
"status": "dry-run",
|
||||
"schemaVersion": PART_INDEX_SCHEMA,
|
||||
"part": part_name,
|
||||
"written": str(target),
|
||||
"sha256": sha256_file(target),
|
||||
"bytes": len(body),
|
||||
"keptSliceCount": len(payload.get("slices") or []),
|
||||
"discardedSliceCount": len(payload.get("discardedSlices") or []),
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="增量迁移已交付的逐片目录为 part 级总 JSON 索引(dry run)。")
|
||||
parser.add_argument("--delivery-root", required=True, help="已交付 part 目录的本地副本")
|
||||
parser.add_argument("--revision-root", required=True, help="对应的物化修订目录(含 revision-manifest.json 与 slices/)")
|
||||
parser.add_argument("--out", required=True, help="写出候选 part.json 的目录(不会写入交付目录)")
|
||||
parser.add_argument("--prefix", help="该 part 的冻结交付前缀,仅用于推断目录名")
|
||||
args = parser.parse_args()
|
||||
|
||||
report = run_migration(
|
||||
pathlib.Path(args.delivery_root),
|
||||
pathlib.Path(args.revision_root),
|
||||
pathlib.Path(args.out),
|
||||
part_name_from_prefix(args.prefix or args.delivery_root),
|
||||
)
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2))
|
||||
if report["status"] == "dry-run":
|
||||
print("未上传:发布必须经过 API/Worker 的交付通道。")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
184
tests/backend/test_part_index_migration.py
Normal file
184
tests/backend/test_part_index_migration.py
Normal file
@ -0,0 +1,184 @@
|
||||
"""The delivered-directory migration must reuse the operator's own builder.
|
||||
|
||||
These tests pin the property that matters for a one-way migration: the index a
|
||||
migration tool produces from an already-delivered directory is byte-identical to
|
||||
the index the package operator would have produced for the same revision. That
|
||||
only holds while both share one implementation, so this test is also the guard
|
||||
against someone forking the builder.
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from backend.operators.builtin.package import operator as package
|
||||
|
||||
from tests.backend.test_part_index_package import PART_NODE, helpers
|
||||
from tests.backend.test_slice_discard_package import fixture
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def migration_module():
|
||||
path = ROOT / "scripts" / "part_index_migration.py"
|
||||
spec = importlib.util.spec_from_file_location("part_index_migration_under_test", path)
|
||||
assert spec is not None and spec.loader is not None
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def delivered_part(tmp_path, dispositions):
|
||||
"""Build a part directory shaped like a published v6 delivery, plus its
|
||||
materialization revision, exactly as the operator and worker leave them."""
|
||||
delivery_root = tmp_path / "part0001"
|
||||
delivery_root.mkdir(parents=True)
|
||||
(delivery_root / "calibration").mkdir()
|
||||
for index, disposition in enumerate(dispositions, start=1):
|
||||
slice_id = f"slice-{index:06d}"
|
||||
folder = delivery_root / "clips" / slice_id
|
||||
folder.mkdir(parents=True)
|
||||
actions = [] if disposition == "discard" else [{"id": f"a{index}", "description": "拿起物品", "isInvalid": False}]
|
||||
(folder / "annotation.json").write_text(json.dumps({"schemaVersion": "droboticflow.timestamped-description/v3", "sliceId": slice_id, "actions": actions, "descriptions": actions}), encoding="utf8")
|
||||
(folder / "synchronized-slice.json").write_text(json.dumps({
|
||||
"schemaVersion": "droboticflow.synchronized-slice/v1",
|
||||
"deliveryDisposition": disposition,
|
||||
"boundaryContract": "hardware-frame-v1",
|
||||
"maxStereoAlignmentErrorUs": 120,
|
||||
"boundaryTimestampUs": [1000000, 2000000],
|
||||
}), encoding="utf8")
|
||||
(folder / "left.mp4").write_bytes(f"{slice_id}-left".encode())
|
||||
(folder / "right.mp4").write_bytes(f"{slice_id}-right".encode())
|
||||
(folder / "left_pts.csv").write_text("frame_index,timestamp_ms,clip_time_ms\n0,1000,0\n", encoding="utf8")
|
||||
(folder / "right_pts.csv").write_text("frame_index,timestamp_ms,clip_time_ms\n0,1000,0\n", encoding="utf8")
|
||||
(folder / "imu.csv").write_text("timestamp_ms,clip_time_ms,x,y,z,type\n1000,0,0,0,0,accel\n", encoding="utf8")
|
||||
|
||||
# Mirrors the runtime: these evidence files are projections of one accepted
|
||||
# summary, so the tool and the online path read the same facts.
|
||||
(delivery_root / "acceptance-record.json").write_text(json.dumps({"schemaVersion": "droboticflow.capture-acceptance/v1", "finalAcceptance": {
|
||||
"confirmed": True, "sampled": True, "samplingDecision": "selected", "processingProfile": "raw_human",
|
||||
"poseDelivery": {"headpose": "exclude", "handpose": "exclude"},
|
||||
}}), encoding="utf8")
|
||||
(delivery_root / "capture-manifest.json").write_text(json.dumps({"schemaVersion": "droboticflow.capture-delivery/v1", "capture": {"captureId": "CAP-1", "prefix": "raw/part0001"}}), encoding="utf8")
|
||||
(delivery_root / "quality-review.json").write_text(json.dumps({"schemaVersion": "droboticflow.capture-quality-review/v1", "reviews": [{"taskId": "task-1", "revision": 1, "decision": "approved", "reviewerUserId": "user-qc", "createdAt": "2026-01-01T00:00:00Z", "comment": "INTERNAL-ONLY"}]}), encoding="utf8")
|
||||
|
||||
# The accepted materialization revision the delivery was built from.
|
||||
inputs = fixture(tmp_path, dispositions)
|
||||
revision_root = Path(inputs[0]["path"])
|
||||
rows = json.loads((revision_root / "revision-manifest.json").read_text(encoding="utf8"))["items"]
|
||||
for row in rows:
|
||||
slice_id = row["sliceId"]
|
||||
source = revision_root / "slices" / slice_id
|
||||
target = delivery_root / "clips" / slice_id
|
||||
# The delivered copy is authoritative for the index; keep them identical
|
||||
# so any drift would show up as a byte difference in the test.
|
||||
for name in ("annotation.json", "synchronized-slice.json"):
|
||||
target.joinpath(name).write_bytes(source.joinpath(name).read_bytes())
|
||||
(delivery_root / "packages-manifest.json").write_text(json.dumps({"schemaVersion": "droboticflow.delivery.manifest/v6", "capture": {"captureId": "CAP-1"}, "lineage": {"acceptedSliceDirectoryArtifactId": "accepted"}}), encoding="utf8")
|
||||
# Calibration is the one input a delivered directory must carry itself, so
|
||||
# copy the very bytes the online package would reference. Both routes then
|
||||
# see identical inputs and any byte difference is a real contract break.
|
||||
calibrations = Path(inputs[0]["path"]).parent / "package-test-calibrations"
|
||||
for source in sorted(calibrations.iterdir()):
|
||||
(delivery_root / "calibration" / source.name).write_bytes(source.read_bytes())
|
||||
refresh_completion_manifest(delivery_root)
|
||||
return delivery_root, revision_root, inputs
|
||||
|
||||
|
||||
def refresh_completion_manifest(delivery_root: Path) -> None:
|
||||
entries = []
|
||||
for path in sorted(delivery_root.rglob("*")):
|
||||
if path.is_file() and path.name != "manifest.json":
|
||||
relative = path.relative_to(delivery_root).as_posix()
|
||||
entries.append({"path": relative, "key": f"prefix/{relative}", "sizeBytes": path.stat().st_size, "sha256": "0" * 64})
|
||||
(delivery_root / "manifest.json").write_text(json.dumps({"schemaVersion": 1, "kind": "directory", "fileCount": len(entries), "totalBytes": sum(e["sizeBytes"] for e in entries), "entries": entries}), encoding="utf8")
|
||||
|
||||
|
||||
def test_migration_output_matches_the_operator_byte_for_byte(tmp_path):
|
||||
delivery_root, revision_root, inputs = delivered_part(tmp_path, ["keep", "discard", "keep"])
|
||||
migrated = migration_module()
|
||||
payload, body = migrated.build_index(delivery_root, revision_root)
|
||||
|
||||
# Compare against an online package built from the same evidence. The single
|
||||
# sanctioned difference is the migration provenance marker.
|
||||
inputs[1]["data"]["finalAcceptance"].update({
|
||||
"samplingDecision": "selected", "processingProfile": "raw_human",
|
||||
"poseDelivery": {"headpose": "exclude", "handpose": "exclude"},
|
||||
})
|
||||
inputs[1]["data"]["capture"] = {"captureId": "CAP-1"}
|
||||
inputs[1]["data"]["qualityReviews"] = [{"taskId": "task-1", "revision": 1, "decision": "approved", "reviewerUserId": "user-qc", "createdAt": "2026-01-01T00:00:00Z", "comment": "INTERNAL-ONLY"}]
|
||||
artifacts = package.run("r", "p", tmp_path / "online", PART_NODE, inputs, helpers())
|
||||
online = json.loads(Path(artifacts[0]["path"]).read_text(encoding="utf8"))
|
||||
comparable = json.loads(json.dumps(payload))
|
||||
assert comparable["lineage"].pop("migratedFrom") == "droboticflow.delivery.manifest/v6"
|
||||
assert online["lineage"].pop("migratedFrom", None) is None
|
||||
assert comparable == online, "migration no longer produces the same structure as an online package"
|
||||
assert body == (json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True) + "\n").encode("utf-8")
|
||||
assert payload["schemaVersion"] == "droboticflow.delivery.part-index/v1"
|
||||
|
||||
|
||||
def test_migrated_index_records_its_provenance(tmp_path):
|
||||
delivery_root, revision_root, _ = delivered_part(tmp_path, ["keep"])
|
||||
payload, _ = migration_module().build_index(delivery_root, revision_root)
|
||||
assert payload["lineage"]["migratedFrom"] == "droboticflow.delivery.manifest/v6"
|
||||
|
||||
|
||||
def test_migration_classifies_both_dispositions_and_keeps_evidence(tmp_path):
|
||||
delivery_root, revision_root, _ = delivered_part(tmp_path, ["keep", "discard"])
|
||||
payload, _ = migration_module().build_index(delivery_root, revision_root)
|
||||
assert [entry["sliceId"] for entry in payload["slices"]] == ["slice-000001"]
|
||||
assert [entry["sliceId"] for entry in payload["discardedSlices"]] == ["slice-000002"]
|
||||
assert payload["discardedSlices"][0]["classification"]["discardEvidence"]["reason"] == "PRIVATE_DISCARD_REASON"
|
||||
assert payload["lineage"]["migratedFrom"] == "droboticflow.delivery.manifest/v6"
|
||||
|
||||
|
||||
def test_migration_refuses_a_part_without_a_completion_manifest(tmp_path):
|
||||
delivery_root, revision_root, _ = delivered_part(tmp_path, ["keep"])
|
||||
(delivery_root / "manifest.json").unlink()
|
||||
with pytest.raises(SystemExit, match="已完成的交付目录"):
|
||||
migration_module().build_index(delivery_root, revision_root)
|
||||
|
||||
|
||||
def test_migration_refuses_an_unknown_disposition(tmp_path):
|
||||
delivery_root, revision_root, _ = delivered_part(tmp_path, ["keep", "discard"])
|
||||
path = revision_root / "revision-manifest.json"
|
||||
value = json.loads(path.read_text(encoding="utf8"))
|
||||
value["items"][1]["deliveryDisposition"] = "guess"
|
||||
path.write_text(json.dumps(value), encoding="utf8")
|
||||
with pytest.raises(SystemExit, match="用途未知"):
|
||||
migration_module().build_index(delivery_root, revision_root)
|
||||
|
||||
|
||||
def test_migration_refuses_to_overwrite_a_different_existing_index(tmp_path):
|
||||
delivery_root, revision_root, _ = delivered_part(tmp_path, ["keep"])
|
||||
(delivery_root / "part.json").write_text('{"different":true}', encoding="utf8")
|
||||
with pytest.raises(SystemExit, match="拒绝覆盖"):
|
||||
migration_module().run_migration(delivery_root, revision_root, tmp_path / "out", "part0001")
|
||||
|
||||
|
||||
def test_migration_is_a_noop_when_the_index_is_already_delivered(tmp_path):
|
||||
delivery_root, revision_root, _ = delivered_part(tmp_path, ["keep"])
|
||||
migrated = migration_module()
|
||||
_, body = migrated.build_index(delivery_root, revision_root)
|
||||
(delivery_root / "part.json").write_bytes(body)
|
||||
report = migrated.run_migration(delivery_root, revision_root, tmp_path / "out", "part0001")
|
||||
assert report["status"] == "already-migrated"
|
||||
assert not (tmp_path / "out" / "part.json").exists()
|
||||
|
||||
|
||||
def test_migration_writes_the_candidate_outside_the_delivery_directory(tmp_path):
|
||||
delivery_root, revision_root, _ = delivered_part(tmp_path, ["keep"])
|
||||
report = migration_module().run_migration(delivery_root, revision_root, tmp_path / "out", "part0001")
|
||||
assert report["status"] == "dry-run"
|
||||
assert Path(report["written"]).is_file()
|
||||
# The delivered directory is untouched: migration only adds the index.
|
||||
assert not list((delivery_root / "clips").glob("**/part.json"))
|
||||
assert not (delivery_root / "part.json").exists()
|
||||
|
||||
|
||||
def test_migration_refuses_to_write_inside_the_delivery_directory(tmp_path):
|
||||
delivery_root, revision_root, _ = delivered_part(tmp_path, ["keep"])
|
||||
with pytest.raises(SystemExit, match="不能位于交付目录内"):
|
||||
migration_module().run_migration(delivery_root, revision_root, delivery_root / "out", "part0001")
|
||||
Loading…
x
Reference in New Issue
Block a user