🛠️ Add resumable V4 previews and record pool query diagnostics

This commit is contained in:
Codex 2026-11-08 09:02:49 -08:00
parent c942a8ced0
commit 6d8b011c67
5 changed files with 456 additions and 0 deletions

View File

@ -0,0 +1,39 @@
# Broad pool search: rejected metadata index experiment
Base: c942a8ce on codex/review-pool-search; live source e4969acc. This is an opt-in follow-up, not a startup migration.
## Evidence
Production read-only phase counts on 2026-11-08 found about 11,900 quality/semantic candidates. The current source-generated wide search still exceeds the 8-second deadline. Root explain examined 11,903 full documents in 1,018 ms in one warm sample; the window/group prefix and foreign joins add more work. These samples are diagnostic, not an end-to-end p95 or a fixed dataset benchmark. Random claim uses a separate path.
Hash batching and removing foreign lookup pipelines did not resolve the timeout and were not adopted. An initial synthetic index with an arbitrary prefix still fetched all documents. Ordering equality scope first is material: organizationId/pluginId/currentStageId, then createdAt/id, followed by projected metadata.
`TestHumanPoolSearchMongoCoveringIndex` uses the actual aggregation with 1,000 synthetic 64 KiB payloads. With the candidate index, its root cursor examines zero documents, and matching results remain identical before creation, after creation, and after dropping the index. There is no hint in the tested aggregation. A separate 10,000-row projection experiment also examined zero documents with the correctly ordered index. It does not establish full-query latency or concurrency capacity.
## Exact proposed production change
- Target: Fanyu cluster, namespace droboticflow-fanyu; Mongo database drobodata-fanyu, collection humanItems.
- Create one non-unique, non-sparse index `pool_search_metadata_v1` using the ordered keys in `local-data/pool-v4-followup-20261108/rejected-index/index-spec.json`.
- No customer record updates, backfill, deletion, existing index removal, Pod/bandwidth changes, or deployment.
- The candidate API code's compact projection is still local; the live legacy projection may not gain the same coverage. After index creation, run the **local candidate's source-generated pipeline read-only** against the same filter/cursor and compare all matching IDs and count. Also check live API latency/errors because planner selection can change.
- Do not automatically add this index in MongoStore startup. Creation requires separate production-change authorization under AGENTS.md.
## Cost, stop and rollback
A 10,000-row synthetic 16 KiB-payload experiment measured about 1.8 MiB for the 16-field version and a 51 ms local build. The final spec adds supplier/author scope fields. These figures do not predict production build time, disk size, replication lag, CPU or write amplification; inspect actual capacity and active builds before creation. All indexed fields are scalar by contract; abort if historical array values make this a multikey/parallel-array index.
Build adds CPU, storage I/O, disk usage and ongoing index maintenance. Monitor the existing service and cancel a build on sustained pressure rather than changing resources simultaneously. If the index worsens plans or gives no useful gain, drop **only this newly created index** with the same authorization. No customer-data rewrite is needed for rollback.
Root coverage alone is not acceptance. If foreign lookups still dominate or total p95 does not meet the agreed threshold, retain pending status and profile the next phase. Do not widen permissions, omit legacy rows, change pagination, or increase the 8-second timeout.
## Executed result and rollback
The user explicitly authorized creation of this index and dropping only this index if ineffective. Preflight: humanItems had 25,802 records, about 250 MB uncompressed data, 62.5 MB storage, 19.3 MB existing indexes; Mongo cache used 3.54 / 7.86 GB. A paged read-only field check examined all 25,802 records and found no arrays in the indexed fields. The original one-shot type aggregate exceeded its 8-second diagnostic bound; the 52 bounded batches completed in 11.3 seconds.
Creation took 4,597 ms; the new index occupied 15,835,136 bytes. The source-generated production root explain really used this index and examined zero documents, but the full **actual Go Store** candidate queries still took 8,001 ms and timed out for broad/all and available scopes. The acceptance example returned one match in 5,915 ms. These are scoped backend reads, not a customer browser/capacity pass.
A direct old/new root projection comparison measured old index: 2,065 ms, 11,903 documents, 11,904 keys, 1 seek; candidate: 3,377 ms, zero documents, 11,904 keys, 9 seeks. Coverage did not translate to reduced overall cost. Hash batching and native equality batching also still timed out. Production activity and warm caches vary, so these do not establish a fixed-dataset percentage regression; they are sufficient to reject retaining an unproven extra index.
The authorized rollback dropped only pool_search_metadata_v1 in 140 ms and verified it absent. Health/readiness returned HTTP 200 in 38/32 ms afterwards. No customer record, old index, resource or application image was changed. The experiment and local synthetic test are retained only under ignored local-data/, not installed by startup or made a permanent CI requirement.
Next work remains: avoid performing full candidate-set sorting/grouping and foreign metadata joins before selective search. Candidate discovery must preserve direct IDs, capture/part combinations, historical payload/input names, categories, ordering and tenant/claim rules. A new implementation needs differential tests and realistic latency evidence before another production change. The broad-pool issue is **open**, not fixed by this experiment.

View File

@ -39,3 +39,13 @@ Scope: Repair worktree hook bootstrap, verification selection/dependencies/resul
Risk: Local/CI tooling only; no business query, delivery contract, customer record, migration, resource or deployment changes in this follow-up. Keep the wide-pool performance limitation above open.
Verification follow-up evidence: local-data/retro-guardrails/. The unchanged browser/product inputs passed all 23 task checks in verify-1793952872699139000/report.json (338.854s, no source drift). Earlier failures and the 600-second budget exhaustion remain recorded. Subsequent Docker COPY ordering and its regression affect only the dependency-install bootstrap: Docker-prepare regression and final quick checks are run separately; application/browser/Mongo inputs are unchanged. No known-failure baseline was expanded. CI wiring is validated locally; the remote CI has not run because this branch remains local.
## 2026-11-08: wide-pool and all-delivery continuation (base c942a8ce)
- Production freshness was confirmed against origin/dev and live e4969acc before investigation. No application deployment or customer-record mutation was performed.
- Full production delivery inventory: 35 records (27 accepted, 8 legacy no_deliverables). Added a local-only resumable V4 preview converter that reuses the native builder, preserves fixed revision checksums, refuses conflicting output, records every blocked item and never claims a complete audit/publication. All-discard conversion requires the fixed child-run confirmation to name the exact revision/artifact/hash. Historical source recovery requires the published Capture snapshot and an exact persisted Run captureUnit/objectKeys selection; it never uses the mutable current Capture association or current dictionaries.
- Converted and independently verified 18 previews: 12 published packages and 6 all-discard records. All 18 pass original left/right PTS coverage and calibration byte checks. The 6 all-discard cases have no historical calibration SHA: their checksums come from current bytes of the frozen BOS object selection, explicitly recorded. Historical object-byte version proof, complete internal audit, video decoding/downstream cuts and formal publication remain unverified. No online package/pointer was replaced.
- 17 records remain blocked: 10 missing frozen slice classification dictionaries; 6 unavailable/incomplete fixed revision JSONs (5 published + 1 all-discard); 1 older all-discard record lacks original-source and dictionary snapshots. Historical original selection was recovered for 9 of the 10 dictionary-blocked packages by cross-checking the published Capture snapshot with the original Run's explicit selection and the existing Go validator; this does not supply missing historical classification tags. A per-record Chinese follow-up list and historical-name confirmation list are included with the local evidence.
- Preview archive: local-data/delivery-v4-all-preview-20261108/V4交付包_18个已校验预览.zip. Full reports/evidence: local-data/pool-v4-followup-20261108/ and local-data/delivery-v4-all-preview-20261108/. Customer ZIP contains only 18 part.json files, README and public validation summary; no video/IMU/audit files.
- Broad pool remains open. The separately authorized metadata-index experiment really achieved zero root document fetches, but actual Go Store broad/all and available queries still timed out at 8,001 ms, acceptance example 5,915 ms. The index was explicitly rolled back and verified absent; health/readiness 200 afterwards. No unproven index or runtime query rewrite was retained. See index-plan.md for exact scope, cost, evidence and rollback.
- Earlier full task evidence (including synthetic covering-index, pool Mongo parity, permission and browser checks): verify-1794155814156557000/report.json passed. The rejected index experiment was then moved to ignored local-data rather than installed/added as a permanent CI requirement. Final converter/compatibility/tooling checks, including exact historical source recovery and all-discard acceptance guards: verify-1794157259610133000/report.json passed (8 converter cases plus native V4/real-materialization/legacy compatibility suites). These are scoped verification, not release or 81-user capacity acceptance.

View File

@ -0,0 +1,201 @@
#!/usr/bin/env python3
"""Resume a local-only V4 conversion of inventoried, frozen delivery revisions.
Input: inventory.json and packages/<id>/{old/,pinned-revision/,pinned-provenance.json}.
This produces customer JSON previews, never a complete audit or publishable run.
No network, current dictionaries, video downloads, or production writes are used.
"""
from __future__ import annotations
import argparse
import copy
import hashlib
import json
from pathlib import Path
import tempfile
from part_index_migration import load_operator_module, resolve_pose_policy, split_slices
def load(path):
return json.loads(path.read_text(encoding="utf8"))
def sha(raw):
return hashlib.sha256(raw).hexdigest()
def convert(package, directory, output):
if package["status"] not in {"accepted", "no_deliverables"}:
raise ValueError("no published accepted manifest; frozen revision recovery required")
revision = directory / "pinned-revision"
manifest_bytes = (revision / "revision-manifest.json").read_bytes()
manifest = json.loads(manifest_bytes)
provenance = load(directory / "pinned-provenance.json")
delivery = load(directory / "old/packages-manifest.json")
lineage = delivery["lineage"]
if (provenance["sha256"] != sha(manifest_bytes)
or provenance["artifactId"] != lineage["acceptedSliceDirectoryArtifactId"]
or provenance["revision"] != manifest["revision"]):
raise ValueError("frozen revision provenance mismatch")
# Verify every input JSON against its accepted manifest before the builder reads it.
before = {}
for row in manifest["items"]:
relative = Path(row["directory"])
if relative.is_absolute() or ".." in relative.parts:
raise ValueError("unsafe accepted slice directory")
verified = set()
files = row.get("files", [])
for entry in files.values() if isinstance(files, dict) else files:
if entry["name"] not in {"annotation.json", "synchronized-slice.json"}:
continue
path = revision / relative / entry["name"]
if not path.resolve().is_relative_to(revision.resolve()):
raise ValueError("accepted file escapes revision directory")
raw = path.read_bytes()
if not entry.get("sha256") or sha(raw) != entry["sha256"]:
raise ValueError("frozen annotation checksum mismatch")
before[path] = sha(raw)
verified.add(entry["name"])
required = {"synchronized-slice.json"}
if row["deliveryDisposition"] == "keep":
required.add("annotation.json")
if not required <= verified:
raise ValueError("missing frozen JSON checksums")
source_basis = "accepted_revision_manifest"
recovery_path = directory / "source-selection-recovery.json"
if not manifest.get("originalSource") and recovery_path.exists():
recovery = load(recovery_path)
raw_input = (directory / "source-selection-input.json").read_bytes()
evidence = json.loads(raw_input)
run, capture = evidence["run"], evidence["capture"]
frozen = delivery["capture"]
refs = [v["ref"] for v in run["inputs"].values()]
source = recovery["originalSource"]
if (recovery.get("basis") != "published_capture_snapshot_and_exact_persisted_run_captureUnit_objectKeys"
or recovery.get("inputSha256") != sha(raw_input)
or recovery.get("manifestSha256") != sha(manifest_bytes)
or run["id"] != lineage["sourceRunId"] or run["id"] != manifest["runId"]
or run["organizationId"] != package["organizationId"]
or capture["runId"] != run["id"] or capture["organizationId"] != run["organizationId"]
or capture["id"] != frozen["captureRecordId"]
or capture["captureId"] != frozen["captureId"] or capture["prefix"] != frozen["prefix"]
or len(refs) != 1):
raise ValueError("historical source recovery provenance mismatch")
ref = refs[0]
if (ref["provider"] != "BOS" or ref["kind"] != "directory"
or ref["captureUnit"] != frozen["captureId"]
or not frozen["prefix"].startswith(ref["prefix"] + "/")
or source["bucket"] != ref["bucket"] or source["prefix"] != ref["prefix"]
or source["partPrefix"] != frozen["prefix"] or source["captureUnit"] != frozen["captureId"]
or source["objectKeys"] != sorted(ref["objectKeys"])
or source.get("snapshotFingerprint", "") != ref.get("versionId", "")):
raise ValueError("historical source selection mismatch")
manifest = copy.deepcopy(manifest)
manifest["originalSource"] = source
source_basis = "published_capture_snapshot_and_exact_persisted_run_captureUnit_objectKeys"
acceptance = load(directory / "old/acceptance-record.json")
kept, discarded = split_slices(manifest)
if package["status"] == "no_deliverables":
proof = load(directory / "empty-acceptance-proof.json")
accepted = proof.get("acceptedMaterialization", {})
if (kept or not discarded or acceptance["finalAcceptance"].get("confirmed") is not True
or proof.get("packageId") != package["id"] or proof.get("confirmed") is not True
or proof.get("summaryArtifactId") not in package.get("artifactIds", [])
or accepted.get("artifactId") != lineage["acceptedSliceDirectoryArtifactId"]
or accepted.get("revision") != manifest["revision"]
or accepted.get("manifestSha256") != sha(manifest_bytes)):
raise ValueError("unconfirmed or mismatched all-discard acceptance evidence")
operator = load_operator_module()
final = acceptance["finalAcceptance"]
# Deliberately incomplete audit input: never export it or claim formal readiness.
preview_context = {"trace": [], "finalAcceptance": final}
bodies = []
with tempfile.TemporaryDirectory(prefix="v4-preview-") as temp:
for attempt in range(2):
staging = Path(temp) / str(attempt)
staging.mkdir()
payload, _ = operator._build_part_index(
manifest, manifest["coverage"], kept, revision,
copy.deepcopy(delivery["calibration"]), resolve_pose_policy(acceptance),
{}, final, copy.deepcopy(delivery["capture"]), lineage, discarded,
classification_v4=True,
)
def internal(*args, **kwargs):
return {"portId": "deliveryAudit", "previewOnly": True}
def virtual(run, node, name, path, kind, size, data=None, output=None):
return {"path": str(path), "portId": output["id"]}
artifacts = operator._emit_part_index_directory(
"local-preview", "package", staging,
{"outputs": [{"id": "deliveryAudit"}, {"id": "deliveryDirectory"}]},
{"write_json_artifact": internal, "build_virtual_artifact": virtual},
payload, preview_context, manifest, revision,
)
folder = Path(artifacts[0]["path"])
if {p.name for p in folder.iterdir()} != {"part.json"}:
raise ValueError("customer directory must contain only part.json")
bodies.append((folder / "part.json").read_bytes())
if bodies[0] != bodies[1]:
raise ValueError("non-deterministic conversion")
for path, digest in before.items():
if sha(path.read_bytes()) != digest:
raise ValueError("source changed during conversion")
if (revision / "revision-manifest.json").read_bytes() != manifest_bytes:
raise ValueError("source manifest changed during conversion")
calibration_proof = directory / "calibration-provenance.json"
calibration_basis = "published_manifest_checksums"
if calibration_proof.exists():
evidence = load(calibration_proof)
if not all(entry.get("historicalStoredSHAAvailable") is True for entry in evidence):
calibration_basis = "original_BOS_selection_read_at_conversion; historical_checksum_unavailable"
document = json.loads(bodies[0])
target = output / package["id"] / "part.json"
target.parent.mkdir(parents=True, exist_ok=True)
if target.exists() and target.read_bytes() != bodies[0]:
raise ValueError("refusing to replace an existing different preview")
target.write_bytes(bodies[0])
return {"packageId": package["id"], "status": "converted-preview", "file": str(target),
"captureId": document["capture"]["captureId"], "partId": document["partId"],
"revision": document["revision"], "sha256": sha(bodies[0]), "bytes": len(bodies[0]),
"keptSlices": len(kept), "discardedSlices": len(discarded),
"manifestSha256": sha(manifest_bytes), "deterministic": True,
"calibrationEvidence": calibration_basis,
"originalSourceEvidence": source_basis,
"formalPublicationReady": False, "completeInternalAuditValidated": False,
"onlinePublished": False}
def run(root, output):
root, output = root.resolve(), output.resolve()
if output == root or root in output.parents:
raise ValueError("output must be outside the evidence directory")
inventory = load(root / "inventory.json")
ids = [p["id"] for p in inventory]
if len(ids) != len(set(ids)):
raise ValueError("duplicate inventory package IDs")
results = []
output.mkdir(parents=True, exist_ok=True)
for package in inventory:
# Do not use an untrusted ID as a filesystem path.
if Path(package["id"]).name != package["id"] or package["id"] in {".", ".."}:
raise ValueError("unsafe package ID")
try:
result = convert(package, root / "packages" / package["id"], output)
except (ValueError, RuntimeError, KeyError, OSError, SystemExit) as error:
result = {"packageId": package["id"], "status": "blocked", "reason": str(error)}
results.append(result)
report = {"total": len(inventory), "processed": len(results),
"converted": sum(r["status"] == "converted-preview" for r in results),
"productionWrites": False, "formalPublicationReady": False, "packages": results}
(output / "conversion-report.json").write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n")
return report
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--evidence", type=Path, required=True)
parser.add_argument("--out", type=Path, required=True)
args = parser.parse_args()
report = run(args.evidence, args.out)
print(json.dumps({k: v for k, v in report.items() if k != "packages"}, ensure_ascii=False))
raise SystemExit(0 if report["converted"] == report["total"] else 2)

View File

@ -1,6 +1,33 @@
{
"version": 1,
"scopes": [
{
"name": "delivery-v4-preview",
"paths": [
"^scripts/delivery_v4_preview\\.py$",
"^tests/backend/test_delivery_v4_preview\\.py$"
],
"frontend": [],
"go": {},
"task": [
{
"argv": [
"python3",
"-m",
"pytest",
"tests/backend/test_delivery_v4_preview.py",
"tests/backend/test_delivery_v4_package.py",
"tests/backend/test_delivery_v4_real_materialization.py",
"tests/backend/test_legacy_package_compatibility.py",
"-q"
],
"tests": true
}
],
"warnings": [
"仅验证本地 V4 转换预览;完整内部审计、真实视频切片与正式发布另行验收。"
]
},
{
"name": "verification-tooling",
"paths": [

View File

@ -0,0 +1,179 @@
"""Batch conversion must preserve revisions, resume safely and report every failure."""
import hashlib
import importlib.util
import json
from pathlib import Path
import pytest
from tests.backend.test_delivery_v4_package import prepare
def converter(monkeypatch):
root = Path(__file__).resolve().parents[2]
monkeypatch.syspath_prepend(str(root / "scripts"))
spec = importlib.util.spec_from_file_location("preview", root / "scripts/delivery_v4_preview.py")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def evidence(tmp_path):
root = tmp_path / "evidence"
directory = root / "packages/package-accepted"
directory.mkdir(parents=True)
inputs = prepare(directory, [{"description": "拿取"}, {"description": "", "isInvalid": True}])
revision = Path(inputs[0]["path"])
revision.rename(directory / "pinned-revision")
revision = directory / "pinned-revision"
manifest = json.loads((revision / "revision-manifest.json").read_text())
for row in manifest["items"]:
row["files"] = [{"name": name, "sha256": hashlib.sha256((revision / row["directory"] / name).read_bytes()).hexdigest()}
for name in ["annotation.json", "synchronized-slice.json"]]
raw = json.dumps(manifest).encode()
(revision / "revision-manifest.json").write_bytes(raw)
(directory / "pinned-provenance.json").write_text(json.dumps({
"sha256": hashlib.sha256(raw).hexdigest(), "revision": manifest["revision"], "artifactId": "frozen-artifact"}))
old = directory / "old"
old.mkdir()
(old / "packages-manifest.json").write_text(json.dumps({
"lineage": {"acceptedSliceDirectoryArtifactId": "frozen-artifact"},
"capture": inputs[1]["data"]["capture"], "calibration": []}))
(old / "acceptance-record.json").write_text(json.dumps({"finalAcceptance": inputs[1]["data"]["finalAcceptance"]}))
inventory = [{"id": "package-accepted", "status": "accepted"}, {"id": "package-empty", "status": "no_deliverables"}]
(root / "inventory.json").write_text(json.dumps(inventory))
return root, directory
def test_batch_resumes_exactly_and_keeps_all_blocked_rows(tmp_path, monkeypatch):
module = converter(monkeypatch)
root, directory = evidence(tmp_path)
output = tmp_path / "output"
before = {p: p.read_bytes() for p in root.rglob("*.json")}
report = module.run(root, output)
assert report["processed"] == report["total"] == 2
assert report["converted"] == 1
assert report["packages"][1]["status"] == "blocked"
assert report["productionWrites"] is report["formalPublicationReady"] is False
customer = output / "package-accepted/part.json"
doc = json.loads(customer.read_text())
assert doc["schemaVersion"].endswith("/v4")
assert not {"lineage", "acceptance", "reviewHistory"} & doc.keys()
assert not list(output.rglob("review-history.json"))
assert "PRIVATE_REWORK" not in customer.read_text()
assert "PRIVATE_BASELINE" not in customer.read_text()
assert module.run(root, output) == report
assert all(p.read_bytes() == body for p, body in before.items())
customer.write_text("other customer bytes")
blocked = module.run(root, output)
assert blocked["converted"] == 0
assert "refusing to replace" in blocked["packages"][0]["reason"]
assert customer.read_text() == "other customer bytes"
@pytest.mark.parametrize("mutation", ["annotation", "provenance", "source", "classification", "checksums"])
def test_corrupt_or_missing_frozen_evidence_blocks_without_guessing(tmp_path, monkeypatch, mutation):
module = converter(monkeypatch)
root, directory = evidence(tmp_path)
revision = directory / "pinned-revision"
path = revision / "revision-manifest.json"
manifest = json.loads(path.read_text())
if mutation == "annotation":
(revision / manifest["items"][0]["directory"] / "annotation.json").write_text("{}")
elif mutation == "provenance":
(directory / "pinned-provenance.json").write_text('{}')
else:
if mutation == "source":
manifest.pop("originalSource")
elif mutation == "classification":
manifest["items"][0].pop("dictionary", None)
sync = revision / manifest["items"][0]["directory"] / "synchronized-slice.json"
value = json.loads(sync.read_text()); value.pop("dictionary", None); sync.write_text(json.dumps(value))
manifest["items"][0]["files"][1]["sha256"] = hashlib.sha256(sync.read_bytes()).hexdigest()
else:
manifest["items"][0].pop("files")
path.write_text(json.dumps(manifest))
provenance = json.loads((directory / "pinned-provenance.json").read_text())
provenance["sha256"] = hashlib.sha256(path.read_bytes()).hexdigest()
(directory / "pinned-provenance.json").write_text(json.dumps(provenance))
report = module.run(root, tmp_path / "output")
assert report["processed"] == 2
assert report["converted"] == 0
assert not list((tmp_path / "output").rglob("part.json"))
def test_all_discard_preview_requires_matching_fixed_acceptance_proof(tmp_path, monkeypatch):
import copy
module = converter(monkeypatch)
root, directory = evidence(tmp_path)
revision = directory / "pinned-revision"
path = revision / "revision-manifest.json"
manifest = json.loads(path.read_text())
row = manifest["items"][0]
row["deliveryDisposition"] = "discard"
row["discardEvidence"] = copy.deepcopy(manifest["items"][1]["discardEvidence"])
sync = revision / row["directory"] / "synchronized-slice.json"
doc = json.loads(sync.read_text())
doc.update(deliveryDisposition="discard", discardEvidence=row["discardEvidence"])
sync.write_text(json.dumps(doc))
row["files"][1]["sha256"] = hashlib.sha256(sync.read_bytes()).hexdigest()
path.write_text(json.dumps(manifest))
digest = hashlib.sha256(path.read_bytes()).hexdigest()
provenance = json.loads((directory / "pinned-provenance.json").read_text())
provenance["sha256"] = digest
(directory / "pinned-provenance.json").write_text(json.dumps(provenance))
package = {"id": "package-accepted", "status": "no_deliverables", "artifactIds": ["frozen-summary"]}
(root / "inventory.json").write_text(json.dumps([package]))
assert module.run(root, tmp_path / "out")["converted"] == 0
proof = {"packageId": package["id"], "confirmed": True, "summaryArtifactId": "frozen-summary",
"acceptedMaterialization": {"artifactId": "frozen-artifact", "revision": manifest["revision"], "manifestSha256": digest}}
(directory / "empty-acceptance-proof.json").write_text(json.dumps(proof))
report = module.run(root, tmp_path / "out")
assert report["converted"] == 1
doc = json.loads(Path(report["packages"][0]["file"]).read_text())
assert doc["slices"] == [] and len(doc["discardedSlices"]) == 2
assert all(not item["segments"] for item in doc["discardedSlices"])
proof["acceptedMaterialization"]["manifestSha256"] = "wrong-revision"
(directory / "empty-acceptance-proof.json").write_text(json.dumps(proof))
assert module.run(root, tmp_path / "other")["converted"] == 0
def test_recovers_source_only_from_matching_published_capture_and_fixed_run(tmp_path, monkeypatch):
module = converter(monkeypatch)
root, directory = evidence(tmp_path)
manifest_path = directory / "pinned-revision/revision-manifest.json"
manifest = json.loads(manifest_path.read_text())
original = manifest.pop("originalSource")
original["objectKeys"] = sorted(original["objectKeys"])
manifest_path.write_text(json.dumps(manifest))
digest = hashlib.sha256(manifest_path.read_bytes()).hexdigest()
provenance = json.loads((directory / "pinned-provenance.json").read_text())
provenance["sha256"] = digest
(directory / "pinned-provenance.json").write_text(json.dumps(provenance))
delivery_path = directory / "old/packages-manifest.json"
delivery = json.loads(delivery_path.read_text())
delivery["lineage"]["sourceRunId"] = manifest["runId"]
delivery_path.write_text(json.dumps(delivery))
package = {"id": "package-accepted", "status": "accepted", "organizationId": "org"}
(root / "inventory.json").write_text(json.dumps([package]))
frozen = delivery["capture"]
source = {"run": {"id": manifest["runId"], "organizationId": "org", "inputs": {"source": {"ref": {
"provider": "BOS", "kind": "directory", "bucket": original["bucket"], "prefix": original["prefix"],
"objectKeys": original["objectKeys"], "captureUnit": original["captureUnit"], "versionId": original["snapshotFingerprint"]}}}},
"capture": {"id": frozen["captureRecordId"], "organizationId": "org", "runId": manifest["runId"],
"captureId": frozen["captureId"], "prefix": frozen["prefix"]}}
recovery = {"basis": "published_capture_snapshot_and_exact_persisted_run_captureUnit_objectKeys",
"manifestSha256": digest, "originalSource": original}
def save():
raw = json.dumps(source).encode()
(directory / "source-selection-input.json").write_bytes(raw)
recovery["inputSha256"] = hashlib.sha256(raw).hexdigest()
(directory / "source-selection-recovery.json").write_text(json.dumps(recovery))
save()
report = module.run(root, tmp_path / "out")
assert report["converted"] == 1
assert report["packages"][0]["originalSourceEvidence"] == recovery["basis"]
assert "originalSource" not in json.loads(manifest_path.read_text())
source["run"]["organizationId"] = "different-tenant"
save()
assert module.run(root, tmp_path / "foreign")["converted"] == 0