🚀 Prepare isolated v4 activation with safe workflow handoff and tag seeding

This commit is contained in:
d-robotics 2026-10-22 00:58:27 +08:00
parent 8f6b531fd3
commit a777c42c4e
20 changed files with 1700 additions and 82 deletions

View File

@ -16,8 +16,8 @@
],
"containerEntrypoint": "backend/operator-runner/main.py",
"entrypoint": "operator.py",
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-package:feaaf80-poc-handpose-auto-20260825052001",
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-package@sha256:fa084bbd0e3b6fedff9582d09c965b460f054ad9e0f5edaf9fc4b9aebcc375cc",
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package:delivery-v4-20261022-8f6b531f",
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package@sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d",
"metadata": "operator.json",
"name": "成品打包",
"operatorId": "operator-builtin-package",

View File

@ -45,3 +45,11 @@
整采集的多 part 输入可能无法证明单 part 的原始来源。普通物理切片、源索引及元数据复用会省略不可靠的新增 `originalSource` 并保留诊断,不因这项可选证据阻塞旧任务;跨 Run/组织错误与存储失败仍拒绝,已有冻结来源不追随新目录。真正生成或发布 v4 时,缺原始来源、标签或时间证据仍被打包/批次门禁拒绝,不能据此输出不完整 JSON。
回退旧 API 不是无损写入保证:新版本保存分类后,旧版本重建草稿数组可能丢弃字典快照。回退前应暂停相关写操作或准备能保留新字段的兼容回退版本;只保留恢复历史不等于能自动恢复所有后续修改。已交付不可变文件不因此原地迁移或覆盖。
## 2026-10-22 启用决定
用户随后确认启用 v4,范围仅为切换后新启动的工作流;在途任务、它们后续的审核/打包子运行和已交付文件继续原协议。此决定替代上节“继续保持关闭”的准备范围,不代表生产开关已经启用。实际发布仍需完成镜像、兼容性和目标核对,并审查具体部署及工作流切换计划。
用户明确确认当前无 tags 的启用词条以词条名称作为第一个独立 tags 标签。该初始化是一次有界字典变更:保留词条 ID、名称、用途、category、aliases 和既有历史,递增版本并记录新修订;已有非空 tags 不覆盖。历史任务中的冻结快照不因此重写,亦不反查新词条替代历史分类。后续 tags 仍由管理员维护。
新工作流使用新的定义版本及固定打包镜像。旧定义不原地改参数或镜像;切换必须保留已启动采集的原工作流归属,并阻止新监听器重新接入这些目录。新 v4 接入还需完整的原始文件选择,包含采集 meta;旧协议保留原有接入要求。准备与验收步骤见 [启用核对表](release/delivery-v4-activation.md)。

View File

@ -0,0 +1,27 @@
# v4 新工作流启用核对表
本页记录 2026-10-22 用户确认的范围,不是已部署证明。实际摘要、目标 UID、镜像和执行状态保存在被 Git 忽略的发布记录中。
## 范围与顺序
1. 在独立工作树验证 API/处理器、Worker、独立 package 镜像,保存源码、基础镜像、产物字节与 digest 的对应证据。API/处理器使用 `scripts/release.py`;Worker、字典和工作流切换另附有界计划。
2. 在 v4 尚未激活时,按发布工具先处理器后 API;随后将 Worker 暂停至 0 副本并确认旧扫描进程退出。现有独立算子 Job 不删除,调度及状态收敛暂时停顿。完成字典和工作流切换后更新兼容新旧协议的 Worker 镜像与构建标记,恢复原来的 2 副本,再开始新版本接入。不更新 Coordinator、validator、Secret、资源规格或存储配置。
3. 按预览清单为当前无 tags 的启用词条初始化 `tags=[名称]`,递增版本并追加历史。逐条比较原版本、名称、状态、更新时间与标签;发现漂移停止,不覆盖管理员的新修改。此步骤不更新任务、草稿或已冻结字典选择。
4. 对每个目标项目创建隐藏的新工作流版本,仅修改 package 的固定镜像、`PART_INDEX_DELIVERY=true`、`DELIVERY_JSON_V4=true`、`DELIVERY_JSON_V3=false`、`SOURCE_INDEX_DELIVERY=true`、`PIN_OPERATOR_IMAGE=true`,并添加内部 JSON 输出 `deliveryAudit`。保留原节点、边、输入、供应商、接入目录、并发和验收抽样策略;逐项比较允许变更以外的内容。
5. 检查内置 artifact 存储与客户目录隔离、完整的每 part 原始选择及 meta、分类标签、两相机时钟和验收固定修订。全舍弃、无描述继承、无效动作及独立动作词条均须有验证证据。
6. 在停止新接入扫描并确认旧扫描已退出的短窗口内激活新版本。仅未启动采集可以转入新版本;`launching`、有 Run 的采集及其父子运行保留旧归属,新监听器识别并排除这些已接入目录。保留原记录 ID,防止 `(workflowId,prefix)` 冲突;不能删除登记后重扫。
7. 只读检查目标版本、旧任务归属、新旧镜像及实际交付格式。生产写入验收必须另列隔离项目、账号和合成输入范围;单凭 Ready、模拟存储测试或单独算子测试不能标记真实业务验收通过。
## 交付与回退
客户文件只含最终分类、标签、区间和来源。审核人员、批注、返工和阶段事件只存内部 `deliveryAudit`,超限或写失败必须阻断客户发布;不得静默省略。
旧 v1–v3 算子输出须有字节兼容回归,旧工作流继续固定原 package 镜像。不可变交付目录、归属标记和已发布索引不原地覆盖。新批次发布仍按已有预览和确认接口执行。
切换失败或结果未知时停止后续写入,先只读核对实际状态。新版本已经接入的任务不能被重新绑定为旧协议;退回监听器也不能重复接入这些目录。旧 API 会在后续保存时丢弃新字典字段,不能把直接换回旧镜像描述为无损回退。需要保留新字段的兼容回退候选,或单独确认暂停相关写入的恢复方案。回退不撤销已确认标签初始化、不删除内部审核证据或客户新批次。
## 发布门禁
发布必须同时具有具体镜像来源、`release.py` 检查、Worker/工作流/字典计划、上线前漂移复核及用户对展示后计划的部署授权。算子历史 release manifest 的已知漂移不得通过重算旧摘要掩盖;只能为真实构建和发布的新 package 绑定新的来源及摘要,其他环境和算子记录保留。
独立 package 发布后使用 `OPERATOR_RELEASE_ENV=fanyu python3 scripts/generate_operator_repository_manifests.py --slug package --check` 验证这一算子的发布和生成清单;不带 `--slug` 的全量检查行为保持严格。未修改算子的既有漂移另列遗留,不更新其镜像、源码摘要或目录记录来凑齐通过。所选 package 的任何摘要漂移仍阻断本次发布。

View File

@ -32,6 +32,7 @@
["npm", "run", "test:browser:slice-input"],
["npm", "run", "test:browser:annotation-persistence"],
["go", "test", "./internal/deliveryindex", "-count=1"],
["go", "test", "./internal/api", "-run", "TestDurableAnnotationOutbox|TestSliceRecovery|TestReviewCannotMutateAnnotation|TestReviewDecision|Test.*ReviewDraft|TestReviewConfirmations|TestRouteSnapshot|Test.*Dictionary|TestProjectSliceNames|TestLegacyMaterializationWithoutPartScopedOriginalSelection|TestOptionalOriginalSource|TestAnnotationOriginalSource|TestClassificationBatch", "-count=1"]
["go", "test", "./internal/worker", "-run", "TestCaptureCutover|Test.*V4Capture|TestDiscoverCaptureGroupsForWorkflow|TestClassificationAudit|TestInternalAuditFailure", "-count=1"],
["go", "test", "./internal/api", "-run", "TestDurableAnnotationOutbox|TestSliceRecovery|TestReviewCannotMutateAnnotation|TestReviewDecision|Test.*ReviewDraft|TestReviewConfirmations|TestRouteSnapshot|Test.*Dictionary|TestProjectSliceNames|TestLegacyMaterializationWithoutPartScopedOriginalSelection|TestOptionalOriginalSource|TestAnnotationOriginalSource|TestClassificationBatch|TestWorkflowVersionActivation", "-count=1"]
]
}

View File

@ -29,3 +29,11 @@ v4 默认关闭不等于旧版本写入安全:新字段开始保存后,旧 A
- 完整算子发布 manifest 检查未通过:annotation_gate 的 sourceTreeSha256 漂移在原始 `3221ac65` 工作树逐项复现,期望值/实际值完全相同,属于已有问题;新 v4 package 源码也尚未绑定新算子镜像发布。不得伪造 release.json 摘要使检查通过。它阻断算子升级/v4 启用,不将本轮限定的 API/处理器默认关闭候选冒充为算子可发布版本。
证据存于本工作树忽略目录 `local-data/verification/combined-v4-20261022/`。本记录不是部署批准,也不保证回退旧写入路径无损。
## 后续启用准备
用户随后确认仅新启动的工作流使用 v4,并确认缺 tags 的当前启用词条以词条名称初始化首个标签。新增准备和验收记录位于 `local-data/verification/v4-activation-20261022/`,不覆盖上述默认关闭阶段的证据。
独立 package 镜像已由 `8f6b531f` 实际构建、核对镜像内源码并上传 Fanyu 仓库,摘要为 `sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d`。仅更新 package 的真实发布记录及生成清单,保留 POC 历史镜像记录。独立发布使用 `OPERATOR_RELEASE_ENV=fanyu python3 scripts/generate_operator_repository_manifests.py --slug package --check`,仍强制验证所选算子的实际源码树和镜像摘要;默认不带选择的全量检查继续保留,不能据此声称全库清单通过。未修改、未发布 annotation_gate,其已有清单漂移仍保留为独立遗留项;本次工作流固定新 package digest,不依赖重写全局算子目录来替换旧任务镜像。
切换修复保留已启动采集及父子 Run 的原归属,恢复隐藏版本上未启动登记的迁移,保留原 ID,并在未知启动结果时阻断激活。未知启动结果须先停扫描、核查原归属并单独恢复,不能按记录年龄自动重复创建 Run。新 v4 接入前验证完整八类原始文件,包括 meta;布尔开关与打包器、内部审核保存使用一致口径。详情见启用核对表,实际部署仍须具体计划授权。

View File

@ -3502,6 +3502,43 @@ func (server *server) handleWorkflowActivateVersion(writer http.ResponseWriter,
renderError(writer, http.StatusInternalServerError, err)
return
}
// Read every exact-scope version, including hidden ancestors left by an
// interrupted activation. A missing Run is an unknown launch outcome, not
// proof that the claim is safe to transfer or repeat. Preflight is read-only.
familyCaptures := make([]domain.CaptureRecord, 0)
for _, version := range versions {
if version.OrganizationID != target.OrganizationID || version.ProjectID != target.ProjectID {
continue
}
records, listErr := server.store.ListCaptureRecords(request.Context(), version.OrganizationID, version.ID, 0)
if listErr != nil {
renderError(writer, http.StatusInternalServerError, listErr)
return
}
for _, record := range records {
if record.OrganizationID != target.OrganizationID || record.ProjectID != target.ProjectID || record.WorkflowID != version.ID {
continue
}
if record.Status == domain.CaptureRecordLaunching && strings.TrimSpace(record.RunID) == "" {
run, lookupErr := server.store.GetRunByWorkflowTaskID(request.Context(), version.ID, record.ID)
if lookupErr != nil {
renderError(writer, http.StatusInternalServerError, lookupErr)
return
}
if run == nil {
renderJSON(writer, http.StatusConflict, map[string]string{
"code": "capture_launch_unresolved",
"error": "Capture launch outcome is unresolved; quiesce Workers and recover the claim under its original workflow before activation",
"captureId": record.ID, "workflowId": version.ID,
})
return
}
// The listener will adopt the observable Run under this original
// owner. Do not rewrite lineage or create another Run here.
}
familyCaptures = append(familyCaptures, record)
}
}
now := nowISO()
updated := make([]domain.WorkflowRecord, 0, len(versions))
rollbackWorkflows := func() {
@ -3509,12 +3546,11 @@ func (server *server) handleWorkflowActivateVersion(writer http.ResponseWriter,
_ = server.store.UpsertWorkflow(request.Context(), updated[index])
}
}
// Close every currently visible sibling before moving the Capture registry.
// This creates a short fail-closed interval in which no listener can scan the
// prefix, instead of exposing the new listener before its idempotency state is
// ready.
// Activation requires a quiesced capture scan window: these writes are not
// atomic with Worker admission. Hide siblings before transferring only their
// never-started captures; started records retain parent/child Run ownership.
for _, item := range versions {
if item.ID == target.ID || item.Hidden {
if item.ID == target.ID || item.Hidden || item.OrganizationID != target.OrganizationID || item.ProjectID != target.ProjectID {
continue
}
previous := item
@ -3537,30 +3573,21 @@ func (server *server) handleWorkflowActivateVersion(writer http.ResponseWriter,
_ = server.store.UpsertCaptureRecord(request.Context(), movedCaptures[index].previous)
}
}
for _, previousWorkflow := range updated {
if previousWorkflow.CaptureAutomation == nil || !previousWorkflow.CaptureAutomation.Enabled {
for _, record := range familyCaptures {
if record.WorkflowID == target.ID || !domain.CaptureAwaitingFirstRun(record) {
continue
}
records, listErr := server.store.ListCaptureRecords(request.Context(), previousWorkflow.OrganizationID, previousWorkflow.ID, 100000)
if listErr != nil {
previous := record
record.WorkflowID = target.ID
record.ProjectID = target.ProjectID
record.UpdatedAt = now
if upsertErr := server.store.UpsertCaptureRecord(request.Context(), record); upsertErr != nil {
rollbackCaptures()
rollbackWorkflows()
renderError(writer, http.StatusInternalServerError, listErr)
renderError(writer, http.StatusInternalServerError, upsertErr)
return
}
for _, record := range records {
previous := record
record.WorkflowID = target.ID
record.ProjectID = target.ProjectID
record.UpdatedAt = now
if upsertErr := server.store.UpsertCaptureRecord(request.Context(), record); upsertErr != nil {
rollbackCaptures()
rollbackWorkflows()
renderError(writer, http.StatusInternalServerError, upsertErr)
return
}
movedCaptures = append(movedCaptures, movedCaptureRecord{previous: previous})
}
movedCaptures = append(movedCaptures, movedCaptureRecord{previous: previous})
}
previousTarget := *target

View File

@ -11538,7 +11538,7 @@ func TestWorkflowVersionActivationHidesSiblingsAndKeepsHistoryReadable(t *testin
versionOne, err := dataStore.CreateWorkflow(ctx, domain.CreateWorkflowInput{
ProjectID: project.ID,
Name: "双目交付-r1",
Type: domain.WorkflowTypeBatch,
Type: domain.WorkflowTypeStreaming,
})
if err != nil {
t.Fatalf("create workflow: %v", err)
@ -11625,12 +11625,12 @@ func TestWorkflowVersionActivationHidesSiblingsAndKeepsHistoryReadable(t *testin
if err != nil || migrated == nil {
t.Fatalf("get migrated Capture registry: %#v err=%v", migrated, err)
}
if migrated.WorkflowID != versionThree.ID || migrated.ID != capture.ID || migrated.RunID != capture.RunID || migrated.Fingerprint != capture.Fingerprint {
t.Fatalf("Capture registry must move without changing identity or lineage: %#v", migrated)
if migrated.WorkflowID != versionOne.ID || migrated.ID != capture.ID || migrated.RunID != capture.RunID || migrated.Fingerprint != capture.Fingerprint {
t.Fatalf("started Capture must retain its original workflow and lineage: %#v", migrated)
}
oldRecords, err := dataStore.ListCaptureRecords(ctx, versionOne.OrganizationID, versionOne.ID, 100)
if err != nil || len(oldRecords) != 0 {
t.Fatalf("old listener must not retain duplicate Capture registry rows: %#v err=%v", oldRecords, err)
if err != nil || len(oldRecords) != 1 {
t.Fatalf("old workflow must retain started Capture ownership: %#v err=%v", oldRecords, err)
}
}

View File

@ -0,0 +1,15 @@
package domain
// CaptureAwaitingFirstRun excludes claims in progress and every historical
// attempt. Only these records may change workflow ownership at activation.
func CaptureAwaitingFirstRun(record CaptureRecord) bool {
if record.RunID != "" {
return false
}
switch record.Status {
case CaptureRecordIncomplete, CaptureRecordStabilizing, CaptureRecordReady:
return true
default:
return false
}
}

View File

@ -371,19 +371,25 @@ func workflowInputDefinition(workflow domain.WorkflowRecord, inputID string) *do
}
func (runtime *Runtime) reconcileCaptureRun(ctx context.Context, record *domain.CaptureRecord) error {
if record == nil || strings.TrimSpace(record.RunID) == "" {
if record == nil || (strings.TrimSpace(record.RunID) == "" && record.Status != domain.CaptureRecordLaunching) {
return nil
}
run, err := runtime.store.GetRunByWorkflowTaskID(ctx, record.WorkflowID, record.ID)
if err != nil {
return err
}
if run == nil {
if run == nil && record.RunID != "" {
run, err = runtime.store.GetRun(ctx, record.RunID)
}
if err != nil || run == nil {
if err != nil {
return err
}
if run == nil {
if record.Status == domain.CaptureRecordLaunching && strings.TrimSpace(record.RunID) == "" {
return fmt.Errorf("capture_launch_unresolved: Capture %s under workflow %s has no observable Run; quiesce Workers and recover this claim under its original workflow before activation; do not reset or repeat admission", record.ID, record.WorkflowID)
}
return nil
}
record.RunID = run.ID
if !captureCancelledByUser(record) {
record.Status = runStatusForCapture(run)
@ -422,7 +428,7 @@ func (runtime *Runtime) launchCaptureRun(ctx context.Context, workflow domain.Wo
return nil
}
}
if record == nil || record.Status != domain.CaptureRecordReady {
if record == nil || record.Status != domain.CaptureRecordReady || record.WorkflowID != workflow.ID || record.OrganizationID != workflow.OrganizationID || record.ProjectID != workflow.ProjectID {
return nil
}
existingRun, err := runtime.store.GetRunByWorkflowTaskID(ctx, workflow.ID, record.ID)
@ -441,16 +447,20 @@ func (runtime *Runtime) launchCaptureRun(ctx context.Context, workflow domain.Wo
if err != nil {
return err
}
if latest == nil || latest.Type != domain.WorkflowTypeStreaming || latest.CaptureAutomation == nil || !latest.CaptureAutomation.Enabled {
if latest == nil || latest.Hidden || latest.DefinitionState != domain.WorkflowDefinitionStateReleased || latest.Type != domain.WorkflowTypeStreaming || latest.CaptureAutomation == nil || !latest.CaptureAutomation.Enabled {
return nil
}
if captureAutomationFingerprint(workflow.ID, normalizedCaptureAutomation(latest.CaptureAutomation)) != captureAutomationFingerprint(workflow.ID, config) {
return nil
}
if err := validateV4CaptureSourceSelection(workflow, record); err != nil {
return err
}
claimed, err := runtime.store.ClaimCaptureRecordForLaunch(ctx, record.ID, record.Fingerprint)
if err != nil || !claimed {
return err
}
record.Status = domain.CaptureRecordLaunching
input := workflowInputDefinition(workflow, config.WorkflowInputID)
if input == nil {
record.Status = domain.CaptureRecordFailed
@ -487,11 +497,13 @@ func (runtime *Runtime) launchCaptureRun(ctx context.Context, workflow domain.Wo
}
run, err := runtime.store.CreateRunWithInputs(ctx, workflow.ID, domain.RunTriggerScheduled, map[string]domain.ArtifactPayload{config.WorkflowInputID: payload}, record.ID)
if err != nil {
record.Status = domain.CaptureRecordReady
record.Error = err.Error()
record.UpdatedAt = nowISO()
_ = runtime.store.UpsertCaptureRecord(ctx, *record)
return err
// CreateRun may have committed before its response failed. Keep the
// persisted launch claim and adopt an observable Run; never reopen
// admission on an unknown result or infer failure from claim age.
if readbackErr := runtime.reconcileCaptureRun(ctx, record); readbackErr != nil {
return fmt.Errorf("create Capture Run failed (%v): %w", err, readbackErr)
}
return nil
}
record.RunID = run.ID
record.Status = domain.CaptureRecordQueued
@ -514,6 +526,55 @@ func (runtime *Runtime) captureAutomationDue(ctx context.Context, workflow domai
return !ok || now.Sub(last) >= time.Duration(config.PollIntervalSeconds)*time.Second, nil
}
// Intake moves between versions, but records with Runs remain with the original
// workflow. Read only this project's exact tenant/family history for deduplication
// and reconcile hidden in-flight owners without migrating their lineage.
func (runtime *Runtime) captureAutomationFamilyRecords(ctx context.Context, workflow domain.WorkflowRecord) ([]domain.CaptureRecord, error) {
domain.NormalizeWorkflowDefinitionMetadata(&workflow)
versions, err := runtime.store.ListWorkflowsByProject(ctx, workflow.ProjectID)
if err != nil {
return nil, err
}
var records []domain.CaptureRecord
for _, version := range versions {
domain.NormalizeWorkflowDefinitionMetadata(&version)
if version.OrganizationID != workflow.OrganizationID || version.ProjectID != workflow.ProjectID || version.DefinitionFamilyID != workflow.DefinitionFamilyID || version.DefinitionState != domain.WorkflowDefinitionStateReleased {
continue
}
items, err := runtime.store.ListCaptureRecords(ctx, version.OrganizationID, version.ID, 0)
if err != nil {
return nil, err
}
for _, item := range items {
if item.OrganizationID != workflow.OrganizationID || item.ProjectID != workflow.ProjectID || item.WorkflowID != version.ID {
continue
}
// Reconcile claims for the current listener as well as hidden owners.
// Missing Runs surface an actionable error without unfenced retries.
if item.Status == domain.CaptureRecordLaunching {
if err := runtime.reconcileCaptureRun(ctx, &item); err != nil {
return nil, err
}
} else if version.ID != workflow.ID {
switch item.Status {
case domain.CaptureRecordQueued, domain.CaptureRecordRunning, domain.CaptureRecordWaiting:
if err := runtime.reconcileCaptureRun(ctx, &item); err != nil {
return nil, err
}
case domain.CaptureRecordRequeueing:
if recoverStaleCaptureRequeue(&item, time.Now().UTC()) {
if err := runtime.store.UpsertCaptureRecord(ctx, item); err != nil {
return nil, err
}
}
}
}
records = append(records, item)
}
}
return records, nil
}
func (runtime *Runtime) processCaptureAutomation(ctx context.Context, workflow domain.WorkflowRecord) error {
if workflow.Type != domain.WorkflowTypeStreaming {
return nil
@ -549,16 +610,35 @@ func (runtime *Runtime) processCaptureAutomation(ctx context.Context, workflow d
if err != nil {
return err
}
groups, err := discoverCaptureGroups(objects, config)
groups, err := discoverCaptureGroupsForWorkflow(objects, config, workflow)
if err != nil {
return err
}
priorRecords, err := runtime.store.ListCaptureRecords(ctx, workflow.OrganizationID, workflow.ID, 0)
// Listing can outlive activation or an intake pause. Do not publish records
// from a captured old listener after the handoff. Deployment still quiesces
// scans because this check and the following store writes are not a transaction.
latest, err := runtime.store.GetWorkflow(ctx, workflow.ID)
if err != nil {
return err
}
if latest == nil || latest.Hidden || latest.DefinitionState != domain.WorkflowDefinitionStateReleased || latest.CaptureAutomation == nil || !latest.CaptureAutomation.Enabled || captureAutomationFingerprint(workflow.ID, normalizedCaptureAutomation(latest.CaptureAutomation)) != fingerprint {
return nil
}
priorRecords, err := runtime.captureAutomationFamilyRecords(ctx, workflow)
if err != nil {
return err
}
byPrefix := map[string]*domain.CaptureRecord{}
admittedPartParents := map[string]bool{}
for _, prior := range priorRecords {
if prior.ConnectionID != config.ConnectionID || prior.Bucket != config.Bucket {
continue
}
if previous := byPrefix[prior.Prefix]; previous != nil && previous.ID != prior.ID {
return fmt.Errorf("capture workflow family has conflicting records for prefix %s", prior.Prefix)
}
copy := prior
byPrefix[prior.Prefix] = &copy
if prior.SourcePrefix != "" && prior.SourcePrefix != prior.Prefix && preserveCaptureParent(&prior) {
admittedPartParents[prior.SourcePrefix] = true
}
@ -570,12 +650,9 @@ func (runtime *Runtime) processCaptureAutomation(ctx context.Context, workflow d
group.Missing = append(group.Missing, "该目录已有 part 接入记录,不能重新作为整目录接入")
}
if group.SourcePrefix != "" && group.SourcePrefix != group.Prefix {
parent, e := runtime.store.GetCaptureRecordByWorkflowPrefix(ctx, workflow.ID, group.SourcePrefix)
if e != nil {
return e
}
parent := byPrefix[group.SourcePrefix]
if preserveCaptureParent(parent) {
if !historicalParents[parent.ID] && parent.RunID != "" {
if !historicalParents[parent.ID] && parent.RunID != "" && parent.WorkflowID == workflow.ID {
if e := runtime.reconcileCaptureRun(ctx, parent); e != nil {
return e
}
@ -584,9 +661,11 @@ func (runtime *Runtime) processCaptureAutomation(ctx context.Context, workflow d
continue
}
}
existing, err := runtime.store.GetCaptureRecordByWorkflowPrefix(ctx, workflow.ID, group.Prefix)
if err != nil {
return err
existing := byPrefix[group.Prefix]
if existing != nil && existing.WorkflowID != workflow.ID {
// Historical admissions (including a launch claim with no Run yet)
// never switch protocols. Unstarted transfers belong to activation.
continue
}
if existing != nil && existing.Status == domain.CaptureRecordDiscarded {
continue
@ -613,6 +692,9 @@ func (runtime *Runtime) processCaptureAutomation(ctx context.Context, workflow d
}
continue
}
if existing != nil && !domain.CaptureAwaitingFirstRun(*existing) {
continue
}
missing := append([]string(nil), group.Missing...)
nowText := nowISO()
record := domain.CaptureRecord{
@ -625,6 +707,7 @@ func (runtime *Runtime) processCaptureAutomation(ctx context.Context, workflow d
ContractRevision: domain.FanyuV58CaptureContractRevision,
}
if existing != nil {
record.ID = existing.ID
record.Delivery = existing.Delivery
record.DiscoveredAt = existing.DiscoveredAt
record.Actions = append([]domain.CaptureActionRecord(nil), existing.Actions...)

View File

@ -0,0 +1,538 @@
package worker
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
"droboticflow/internal/api"
"droboticflow/internal/config"
"droboticflow/internal/domain"
"droboticflow/internal/store"
)
type captureCutoverFixture struct {
t *testing.T
ctx context.Context
db *store.MemoryStore
runtime *Runtime
old, next domain.WorkflowRecord
objects []captureObject
beforeList func()
}
func newCaptureCutoverFixture(t *testing.T, names ...string) *captureCutoverFixture {
t.Helper()
f := &captureCutoverFixture{t: t, ctx: context.Background(), db: store.NewMemoryStore()}
for _, name := range names {
for _, object := range append(calibrationObjects(), partObjects("part0001")...) {
object.Key = strings.ReplaceAll(object.Key, layoutCapture, name)
f.objects = append(f.objects, object)
}
f.objects = append(f.objects, captureObject{Key: "incoming/" + name + "/meta_" + name + ".json", Size: 10, ETag: "metadata"})
}
bos := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if f.beforeList != nil {
f.beforeList()
}
contents := []map[string]any{}
for _, object := range f.objects {
contents = append(contents, map[string]any{"key": object.Key, "eTag": object.ETag, "size": object.Size})
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"contents": contents, "isTruncated": false})
}))
t.Cleanup(bos.Close)
project, err := f.db.CreateProject(f.ctx, domain.CreateProjectInput{OrganizationID: domain.DefaultOrganizationID, Name: "cutover"})
if err != nil {
t.Fatal(err)
}
if err := f.db.UpsertStorageConnection(f.ctx, domain.StorageConnectionRecord{ID: "cutover-storage", OrganizationID: project.OrganizationID, Provider: domain.StorageProviderBOS, Bucket: "synthetic", Endpoint: bos.URL, AccessKeyID: "synthetic", SecretAccessKey: "synthetic"}); err != nil {
t.Fatal(err)
}
config := &domain.CaptureAutomationConfig{Enabled: true, ConnectionID: "cutover-storage", Bucket: "synthetic", Prefix: "incoming", WorkflowInputID: "sourceDirectory", DeliveryPrefix: "deliveries", DeliveryProjectDirectory: "cutover__project", StabilityWindowSeconds: 1}
f.old = domain.WorkflowRecord{ID: "cutover-old", OrganizationID: project.OrganizationID, ProjectID: project.ID, Type: domain.WorkflowTypeStreaming, ExecutionScope: domain.WorkflowExecutionScopeProduction, DefinitionState: domain.WorkflowDefinitionStateReleased, DefinitionFamilyID: "shared-template-family", DefinitionVersion: 1, CaptureAutomation: config, WorkflowInputs: []domain.WorkflowInputDefinition{{ID: "sourceDirectory", Name: "sourceDirectory", Type: domain.PortTypeFile, Container: domain.PortContainerDirectory, PayloadMode: domain.ArtifactTransferModeRef, Required: true}}, Nodes: []domain.WorkflowNode{{ID: "package", Data: domain.WorkflowNodeData{OperatorID: "operator-builtin-package", Image: "registry.example/package:legacy", OperatorImageID: "registry.example/package@sha256:" + strings.Repeat("a", 64), Hyperparams: []domain.HyperParam{{Key: "DELIVERY_JSON_V4", Value: "false"}, {Key: "PIN_OPERATOR_IMAGE", Value: "true"}}}}}}
f.next = f.old
f.next.ID, f.next.Hidden, f.next.DefinitionVersion, f.next.SupersedesWorkflowID = "cutover-new", true, 2, f.old.ID
f.next.Nodes = []domain.WorkflowNode{{ID: "package", Data: domain.WorkflowNodeData{OperatorID: "operator-builtin-package", Image: "registry.example/package:v4", OperatorImageID: "registry.example/package@sha256:" + strings.Repeat("b", 64), Hyperparams: []domain.HyperParam{{Key: "DELIVERY_JSON_V4", Value: "true"}, {Key: "PIN_OPERATOR_IMAGE", Value: "true"}}}}}
for _, workflow := range []domain.WorkflowRecord{f.old, f.next} {
if err := f.db.UpsertWorkflow(f.ctx, workflow); err != nil {
t.Fatal(err)
}
}
if err := f.db.UpsertOperator(f.ctx, domain.OperatorRecord{ID: "operator-builtin-package", Type: "builtin", Image: "registry.example/package:catalog", ImageID: "registry.example/package@sha256:" + strings.Repeat("c", 64)}); err != nil {
t.Fatal(err)
}
f.runtime = &Runtime{store: f.db}
return f
}
func (f *captureCutoverFixture) seed(name string, status domain.CaptureRecordStatus, runID string) domain.CaptureRecord {
f.t.Helper()
groups, err := discoverCaptureGroups(f.objects, normalizedCaptureAutomation(f.old.CaptureAutomation))
if err != nil {
f.t.Fatal(err)
}
var selected captureGroup
for _, group := range groups {
if group.CaptureID == name+"/part0001" {
selected = group
}
}
if selected.Prefix == "" || len(selected.Missing) != 0 {
f.t.Fatalf("invalid cutover source fixture: %+v", selected)
}
c := domain.CaptureRecord{ID: captureRecordID(f.old.ID, selected.Prefix), OrganizationID: f.old.OrganizationID, ProjectID: f.old.ProjectID, WorkflowID: f.old.ID, ConnectionID: "cutover-storage", Bucket: "synthetic", CaptureID: selected.CaptureID, Prefix: selected.Prefix, SourcePrefix: selected.SourcePrefix, SourceKeys: selected.SourceKeys, Fingerprint: selected.Fingerprint, Status: status, RunID: runID, StabilityChecks: 1, StabilityObservedAt: time.Now().Add(-time.Minute).UTC().Format(time.RFC3339Nano), DiscoveredAt: "2026-01-01T00:00:00Z", UpdatedAt: "2026-01-01T00:00:00Z", Delivery: &domain.CaptureDeliveryTarget{ConnectionID: "cutover-storage", Bucket: "synthetic", Prefix: "deliveries/cutover__project/" + selected.CaptureID}}
if err := f.db.UpsertCaptureRecord(f.ctx, c); err != nil {
f.t.Fatal(err)
}
saved, err := f.db.GetCaptureRecord(f.ctx, c.ID)
if err != nil {
f.t.Fatal(err)
}
return *saved
}
func (f *captureCutoverFixture) activate() {
f.t.Helper()
w := httptest.NewRecorder()
api.NewHandlerWithConfig(f.db, nil, config.AppConfig{}).ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/api/workflows/"+f.next.ID+"/activate", nil))
if w.Code != http.StatusOK {
f.t.Fatalf("activation: %d %s", w.Code, w.Body.String())
}
}
func (f *captureCutoverFixture) scan() {
f.t.Helper()
if err := f.db.DeleteSourceStatesByWorkflow(f.ctx, f.next.ID); err != nil {
f.t.Fatal(err)
}
w, err := f.db.GetWorkflow(f.ctx, f.next.ID)
if err != nil {
f.t.Fatal(err)
}
if err := f.runtime.processCaptureAutomation(f.ctx, *w); err != nil {
f.t.Fatal(err)
}
}
func TestCaptureCutoverPreservesStartedOwnershipAndParentChildDelivery(t *testing.T) {
f := newCaptureCutoverFixture(t, "active", "done", "launching", "fresh")
active := f.seed("active", domain.CaptureRecordWaiting, "old-parent")
done := f.seed("done", domain.CaptureRecordCompleted, "delivered-parent")
launching := f.seed("launching", domain.CaptureRecordLaunching, "old-claimed")
fresh := f.seed("fresh", domain.CaptureRecordStabilizing, "")
parent := domain.RunRecord{ID: active.RunID, OrganizationID: active.OrganizationID, ProjectID: active.ProjectID, WorkflowID: f.old.ID, TaskID: active.ID, Status: domain.RunStatusWaitingForHuman, CreatedAt: "2026-01-01T00:00:00Z"}
child := parent
child.ID, child.ParentRunID, child.Trigger, child.Status, child.CreatedAt = "old-child", parent.ID, domain.RunTriggerHumanRelease, domain.RunStatusRunning, "2026-01-01T00:00:01Z"
for _, run := range []domain.RunRecord{parent, child, {ID: launching.RunID, OrganizationID: launching.OrganizationID, ProjectID: launching.ProjectID, WorkflowID: f.old.ID, TaskID: launching.ID, Status: domain.RunStatusQueued}} {
if err := f.db.UpsertRun(f.ctx, run); err != nil {
t.Fatal(err)
}
}
f.activate()
for _, before := range []domain.CaptureRecord{active, done, launching} {
after, _ := f.db.GetCaptureRecord(f.ctx, before.ID)
if after == nil || after.WorkflowID != f.old.ID || after.RunID != before.RunID {
t.Fatalf("started Capture moved: before=%+v after=%+v", before, after)
}
}
lookup, err := f.db.GetCaptureRecordByRunID(f.ctx, child.ID)
if err != nil || lookup == nil || lookup.ID != active.ID {
t.Fatalf("old child lost Capture: %+v %v", lookup, err)
}
publisher := &deliveryPublisher{}
f.runtime.artifactPublisher = publisher
root := t.TempDir()
if err := os.WriteFile(filepath.Join(root, "legacy.json"), []byte("legacy bytes"), 0600); err != nil {
t.Fatal(err)
}
if _, err := f.runtime.publishCaptureDelivery(f.ctx, &child, "package", domain.ExecutionArtifact{Name: "delivery.directory.index", Type: domain.PortTypeFile, Path: root}); err != nil {
t.Fatalf("old child cannot publish: %v", err)
}
child.Status = domain.RunStatusCompleted
if err := f.db.UpsertRun(f.ctx, child); err != nil {
t.Fatal(err)
}
f.scan()
oldAfter, _ := f.db.GetCaptureRecord(f.ctx, active.ID)
if oldAfter.WorkflowID != f.old.ID || oldAfter.RunID != child.ID || oldAfter.Status != domain.CaptureRecordCompleted {
t.Fatalf("hidden lineage did not reconcile: %+v", oldAfter)
}
if lookup, err = f.db.GetCaptureRecordByRunID(f.ctx, parent.ID); err != nil || lookup == nil || lookup.ID != active.ID {
t.Fatalf("old parent lost child-owned Capture: %+v %v", lookup, err)
}
f.scan()
newCapture, _ := f.db.GetCaptureRecord(f.ctx, fresh.ID)
if newCapture == nil || newCapture.WorkflowID != f.next.ID || newCapture.RunID == "" {
t.Fatalf("unstarted Capture not admitted: %+v", newCapture)
}
newRun, _ := f.db.GetRun(f.ctx, newCapture.RunID)
if newRun.WorkflowID != f.next.ID || newRun.TaskID != fresh.ID {
t.Fatalf("wrong new Run binding: %+v", newRun)
}
oldMode, err := f.runtime.classificationDeliveryEnabled(f.ctx, child.WorkflowID)
if err != nil || oldMode {
t.Fatalf("old child selected v4: %v %v", oldMode, err)
}
newMode, err := f.runtime.classificationDeliveryEnabled(f.ctx, newRun.WorkflowID)
if err != nil || !newMode {
t.Fatalf("new parent did not select v4: %v %v", newMode, err)
}
for _, run := range []domain.RunRecord{parent, child, *newRun} {
workflow, err := f.db.GetWorkflow(f.ctx, run.WorkflowID)
if err != nil {
t.Fatal(err)
}
resolved, err := f.runtime.resolveBuiltinNodeRuntime(f.ctx, workflow.Nodes[0])
want := f.old.Nodes[0]
if run.WorkflowID == f.next.ID {
want = f.next.Nodes[0]
}
if err != nil || resolved.Data.Image != want.Data.Image || resolved.Data.OperatorImageID != want.Data.OperatorImageID {
t.Fatalf("Run %s lost pinned original runtime: got=%+v want=%+v err=%v", run.ID, resolved.Data, want.Data, err)
}
}
completed, _ := f.db.GetCaptureRecord(f.ctx, done.ID)
if !reflect.DeepEqual(*completed, done) {
t.Fatalf("delivered Capture changed: before=%+v after=%+v", done, completed)
}
all, _ := f.db.ListCaptureRecords(f.ctx, f.old.OrganizationID, "", 0)
if len(all) != 4 {
t.Fatalf("Capture re-ingestion or ID collision: %+v", all)
}
}
func TestCaptureCutoverMigratedUnstartedKeepsIdentity(t *testing.T) {
f := newCaptureCutoverFixture(t, "fresh")
before := f.seed("fresh", domain.CaptureRecordStabilizing, "")
f.activate()
f.scan()
f.scan()
after, _ := f.db.GetCaptureRecord(f.ctx, before.ID)
all, _ := f.db.ListCaptureRecords(f.ctx, f.next.OrganizationID, f.next.ID, 0)
if after == nil || after.RunID == "" || len(all) != 1 || all[0].ID != before.ID {
t.Fatalf("migrated identity lost/duplicated: %+v %+v", after, all)
}
}
func TestCaptureCutoverDoesNotDeduplicateOtherScopes(t *testing.T) {
for _, scope := range []string{"project", "organization", "family"} {
t.Run(scope, func(t *testing.T) {
f := newCaptureCutoverFixture(t, "fresh")
foreign := f.old
foreign.ID = "foreign-workflow"
switch scope {
case "project":
foreign.ProjectID = "other-project"
case "organization":
foreign.OrganizationID = "other-org"
case "family":
foreign.DefinitionFamilyID = "other-family"
}
if err := f.db.UpsertWorkflow(f.ctx, foreign); err != nil {
t.Fatal(err)
}
original := f.seed("fresh", domain.CaptureRecordStabilizing, "")
other := original
other.ID, other.WorkflowID, other.OrganizationID, other.ProjectID, other.RunID, other.Status = "foreign-capture", foreign.ID, foreign.OrganizationID, foreign.ProjectID, "foreign-run", domain.CaptureRecordCompleted
if err := f.db.UpsertCaptureRecord(f.ctx, other); err != nil {
t.Fatal(err)
}
f.activate()
f.scan()
got, _ := f.db.GetCaptureRecord(f.ctx, original.ID)
untouched, _ := f.db.GetCaptureRecord(f.ctx, other.ID)
if got.RunID == "" || got.WorkflowID != f.next.ID || untouched.WorkflowID != foreign.ID || untouched.RunID != other.RunID {
t.Fatalf("scope crossed: own=%+v foreign=%+v", got, untouched)
}
})
}
}
func TestCaptureCutoverStopsScanCapturedBeforeActivation(t *testing.T) {
f := newCaptureCutoverFixture(t, "fresh")
before := f.seed("fresh", domain.CaptureRecordStabilizing, "")
f.beforeList = f.activate
if err := f.runtime.processCaptureAutomation(f.ctx, f.old); err != nil {
t.Fatal(err)
}
after, _ := f.db.GetCaptureRecord(f.ctx, before.ID)
if after.WorkflowID != f.next.ID || after.RunID != "" {
t.Fatalf("stale old scan wrote/admitted after activation: %+v", after)
}
}
func TestCaptureCutoverReadsAllHiddenAncestorsAndReconcilesMissingSources(t *testing.T) {
f := newCaptureCutoverFixture(t, "ancient", "missing", "claim")
ancestor := f.old
ancestor.ID, ancestor.Hidden = "hidden-ancestor", true
f.old.SupersedesWorkflowID, f.old.DefinitionVersion = ancestor.ID, 2
f.next.DefinitionVersion = 3
for _, workflow := range []domain.WorkflowRecord{ancestor, f.old, f.next} {
if err := f.db.UpsertWorkflow(f.ctx, workflow); err != nil {
t.Fatal(err)
}
}
ancient := f.seed("ancient", domain.CaptureRecordCompleted, "delivered-ancestor")
missing := f.seed("missing", domain.CaptureRecordWaiting, "ancestor-run")
claim := f.seed("claim", domain.CaptureRecordLaunching, "")
for _, record := range []*domain.CaptureRecord{&ancient, &missing, &claim} {
record.WorkflowID = ancestor.ID
if err := f.db.UpsertCaptureRecord(f.ctx, *record); err != nil {
t.Fatal(err)
}
}
for _, run := range []domain.RunRecord{
{ID: missing.RunID, OrganizationID: ancestor.OrganizationID, ProjectID: ancestor.ProjectID, WorkflowID: ancestor.ID, TaskID: missing.ID, Status: domain.RunStatusCompleted},
// The Worker died after CreateRun but before recording its ID on Capture.
{ID: "claimed-old-run", OrganizationID: ancestor.OrganizationID, ProjectID: ancestor.ProjectID, WorkflowID: ancestor.ID, TaskID: claim.ID, Status: domain.RunStatusQueued},
} {
if err := f.db.UpsertRun(f.ctx, run); err != nil {
t.Fatal(err)
}
}
filtered := []captureObject{}
for _, object := range f.objects {
if !strings.Contains(object.Key, "/missing/") {
filtered = append(filtered, object)
}
}
f.objects = filtered
f.activate()
f.scan()
f.scan()
completed, _ := f.db.GetCaptureRecord(f.ctx, ancient.ID)
if !reflect.DeepEqual(*completed, ancient) {
t.Fatalf("historical delivery changed: %+v", completed)
}
reconciled, _ := f.db.GetCaptureRecord(f.ctx, missing.ID)
if reconciled.WorkflowID != ancestor.ID || reconciled.Status != domain.CaptureRecordCompleted {
t.Fatalf("hidden ancestor without source was not reconciled: %+v", reconciled)
}
recovered, _ := f.db.GetCaptureRecord(f.ctx, claim.ID)
if recovered.WorkflowID != ancestor.ID || recovered.RunID != "claimed-old-run" || recovered.Status != domain.CaptureRecordQueued {
t.Fatalf("interrupted old launch lost its owner: %+v", recovered)
}
all, _ := f.db.ListCaptureRecords(f.ctx, f.old.OrganizationID, "", 0)
if len(all) != 3 {
t.Fatalf("hidden ancestor re-ingested: %+v", all)
}
}
func TestCaptureCutoverTransfersOnlyNeverStartedStates(t *testing.T) {
states := []domain.CaptureRecordStatus{domain.CaptureRecordIncomplete, domain.CaptureRecordStabilizing, domain.CaptureRecordReady, domain.CaptureRecordLaunching, domain.CaptureRecordRequeueing, domain.CaptureRecordQueued, domain.CaptureRecordRunning, domain.CaptureRecordWaiting, domain.CaptureRecordCompleted, domain.CaptureRecordFailed, domain.CaptureRecordCancelled, domain.CaptureRecordDiscarded}
names := []string{}
for _, state := range states {
names = append(names, string(state))
}
f := newCaptureCutoverFixture(t, names...)
before := []domain.CaptureRecord{}
for _, state := range states {
runID := "old-" + string(state)
if state == domain.CaptureRecordIncomplete || state == domain.CaptureRecordStabilizing || state == domain.CaptureRecordReady || state == domain.CaptureRecordRequeueing || state == domain.CaptureRecordDiscarded {
runID = ""
}
before = append(before, f.seed(string(state), state, runID))
}
f.activate()
for _, previous := range before {
after, _ := f.db.GetCaptureRecord(f.ctx, previous.ID)
move := previous.Status == domain.CaptureRecordIncomplete || previous.Status == domain.CaptureRecordStabilizing || previous.Status == domain.CaptureRecordReady
if move {
if after.WorkflowID != f.next.ID || after.RunID != "" || after.ID != previous.ID {
t.Fatalf("unstarted transfer: %+v", after)
}
} else if !reflect.DeepEqual(*after, previous) {
t.Fatalf("historical/claimed record changed: before=%+v after=%+v", previous, after)
}
}
}
// Store faults exercise partial activation and unknown Run creation outcomes.
type captureCutoverFaultStore struct {
store.Store
upsertWorkflow func(context.Context, domain.WorkflowRecord) error
getRun func(context.Context, string, string) (*domain.RunRecord, error)
createRun func(context.Context, string, domain.RunTrigger, map[string]domain.ArtifactPayload, ...string) (*domain.RunRecord, error)
}
func (s *captureCutoverFaultStore) UpsertWorkflow(ctx context.Context, w domain.WorkflowRecord) error {
if s.upsertWorkflow != nil {
return s.upsertWorkflow(ctx, w)
}
return s.Store.UpsertWorkflow(ctx, w)
}
func (s *captureCutoverFaultStore) GetRunByWorkflowTaskID(ctx context.Context, workflowID, taskID string) (*domain.RunRecord, error) {
if s.getRun != nil {
return s.getRun(ctx, workflowID, taskID)
}
return s.Store.GetRunByWorkflowTaskID(ctx, workflowID, taskID)
}
func (s *captureCutoverFaultStore) CreateRunWithInputs(ctx context.Context, workflowID string, trigger domain.RunTrigger, inputs map[string]domain.ArtifactPayload, taskIDs ...string) (*domain.RunRecord, error) {
if s.createRun != nil {
return s.createRun(ctx, workflowID, trigger, inputs, taskIDs...)
}
return s.Store.CreateRunWithInputs(ctx, workflowID, trigger, inputs, taskIDs...)
}
func (f *captureCutoverFixture) activateResponse() *httptest.ResponseRecorder {
w := httptest.NewRecorder()
api.NewHandlerWithConfig(f.runtime.store, nil, config.AppConfig{}).ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/api/workflows/"+f.next.ID+"/activate", nil))
return w
}
func TestCaptureCutoverRetryTransfersHiddenUnstartedAndRollsBackOnlyOwnWrites(t *testing.T) {
f := newCaptureCutoverFixture(t, "stranded", "already-moved")
stranded := f.seed("stranded", domain.CaptureRecordReady, "")
already := f.seed("already-moved", domain.CaptureRecordReady, "")
already.WorkflowID = f.next.ID
if err := f.db.UpsertCaptureRecord(f.ctx, already); err != nil {
t.Fatal(err)
}
// Persisted state after an earlier process died after hiding the old owner.
f.old.Hidden = true
if err := f.db.UpsertWorkflow(f.ctx, f.old); err != nil {
t.Fatal(err)
}
fault := &captureCutoverFaultStore{Store: f.db}
fault.upsertWorkflow = func(ctx context.Context, w domain.WorkflowRecord) error {
if w.ID == f.next.ID && !w.Hidden {
return fmt.Errorf("injected activation write failure")
}
return f.db.UpsertWorkflow(ctx, w)
}
f.runtime.store = fault
if got := f.activateResponse(); got.Code != http.StatusInternalServerError {
t.Fatalf("expected injected failure: %d %s", got.Code, got.Body.String())
}
old, _ := f.db.GetWorkflow(f.ctx, f.old.ID)
got, _ := f.db.GetCaptureRecord(f.ctx, stranded.ID)
untouched, _ := f.db.GetCaptureRecord(f.ctx, already.ID)
if !old.Hidden || !reflect.DeepEqual(*got, stranded) || !reflect.DeepEqual(*untouched, already) {
t.Fatalf("rollback changed prior request state: old=%+v stranded=%+v moved=%+v", old, got, untouched)
}
f.runtime.store = f.db
f.activate()
got, _ = f.db.GetCaptureRecord(f.ctx, stranded.ID)
if got.WorkflowID != f.next.ID || got.ID != stranded.ID {
t.Fatalf("hidden unstarted capture stranded: %+v", got)
}
f.scan()
got, _ = f.db.GetCaptureRecord(f.ctx, stranded.ID)
if got.RunID == "" {
t.Fatalf("transferred hidden capture not admitted: %+v", got)
}
}
func TestCaptureCutoverUnresolvedClaimBlocksBeforeAnyWrites(t *testing.T) {
for _, hidden := range []bool{false, true} {
for _, lookupFails := range []bool{false, true} {
t.Run(fmt.Sprintf("hidden=%v/lookupFails=%v", hidden, lookupFails), func(t *testing.T) {
f := newCaptureCutoverFixture(t, "claim", "fresh")
claim := f.seed("claim", domain.CaptureRecordLaunching, "")
fresh := f.seed("fresh", domain.CaptureRecordReady, "")
f.old.Hidden = hidden
if err := f.db.UpsertWorkflow(f.ctx, f.old); err != nil {
t.Fatal(err)
}
if lookupFails {
f.runtime.store = &captureCutoverFaultStore{Store: f.db, getRun: func(context.Context, string, string) (*domain.RunRecord, error) {
return nil, fmt.Errorf("injected Run lookup failure")
}}
}
got := f.activateResponse()
want := http.StatusConflict
if lookupFails {
want = http.StatusInternalServerError
}
if got.Code != want || (!lookupFails && !strings.Contains(got.Body.String(), "capture_launch_unresolved")) {
t.Fatalf("unsafe activation: %d %s", got.Code, got.Body.String())
}
for _, before := range []domain.WorkflowRecord{f.old, f.next} {
after, _ := f.db.GetWorkflow(f.ctx, before.ID)
if !reflect.DeepEqual(*after, before) {
t.Fatalf("preflight wrote workflow: %+v", after)
}
}
for _, before := range []domain.CaptureRecord{claim, fresh} {
after, _ := f.db.GetCaptureRecord(f.ctx, before.ID)
if !reflect.DeepEqual(*after, before) {
t.Fatalf("preflight wrote Capture: %+v", after)
}
}
})
}
}
}
func TestCaptureCutoverScannerSurfacesUnresolvedClaimWithoutRewriting(t *testing.T) {
for _, hidden := range []bool{false, true} {
t.Run(fmt.Sprintf("hidden=%v", hidden), func(t *testing.T) {
f := newCaptureCutoverFixture(t, "claim")
claim := f.seed("claim", domain.CaptureRecordLaunching, "")
owner := f.old
if hidden {
f.old.Hidden, f.next.Hidden = true, false
for _, w := range []domain.WorkflowRecord{f.old, f.next} {
if err := f.db.UpsertWorkflow(f.ctx, w); err != nil {
t.Fatal(err)
}
}
owner = f.next
}
err := f.runtime.processCaptureAutomation(f.ctx, owner)
if err == nil || !strings.Contains(err.Error(), "capture_launch_unresolved") || !strings.Contains(err.Error(), claim.ID) || !strings.Contains(err.Error(), f.old.ID) {
t.Fatalf("unresolved claim silently skipped: %v", err)
}
after, _ := f.db.GetCaptureRecord(f.ctx, claim.ID)
runs, _ := f.db.ListRuns(f.ctx, "")
if !reflect.DeepEqual(*after, claim) || len(runs) != 0 {
t.Fatalf("unfenced recovery: Capture=%+v runs=%+v", after, runs)
}
})
}
}
func TestCaptureCutoverUnknownCreateOutcomeNeverReopensAdmission(t *testing.T) {
for _, persisted := range []bool{false, true} {
t.Run(fmt.Sprintf("persisted=%v", persisted), func(t *testing.T) {
f := newCaptureCutoverFixture(t, "claim")
claim := f.seed("claim", domain.CaptureRecordReady, "")
calls := 0
f.runtime.store = &captureCutoverFaultStore{Store: f.db, createRun: func(ctx context.Context, id string, trigger domain.RunTrigger, inputs map[string]domain.ArtifactPayload, taskIDs ...string) (*domain.RunRecord, error) {
calls++
if persisted {
if _, err := f.db.CreateRunWithInputs(ctx, id, trigger, inputs, taskIDs...); err != nil {
t.Fatal(err)
}
}
return nil, fmt.Errorf("injected lost CreateRun response")
}}
err := f.runtime.launchCaptureRun(f.ctx, f.old, normalizedCaptureAutomation(f.old.CaptureAutomation), &claim)
if !persisted && (err == nil || !strings.Contains(err.Error(), "capture_launch_unresolved")) {
t.Fatalf("missing actionable diagnostic: %v", err)
}
after, _ := f.db.GetCaptureRecord(f.ctx, claim.ID)
if persisted {
if after.RunID == "" {
t.Fatalf("lost response not reconciled: %+v (%v)", after, err)
}
} else if after.Status != domain.CaptureRecordLaunching || after.RunID != "" {
t.Fatalf("unknown outcome reopened admission: %+v", after)
}
_ = f.runtime.processCaptureAutomation(f.ctx, f.old)
if calls != 1 {
t.Fatalf("duplicated unknown CreateRun: %d", calls)
}
})
}
}

View File

@ -1,6 +1,7 @@
package worker
import (
"path"
"regexp"
"sort"
"strings"
@ -186,3 +187,27 @@ func discoverCaptureGroups(objects []captureObject, config domain.CaptureAutomat
}
return result, nil
}
// Keep legacy discovery unchanged. Only workflows explicitly requesting v4
// need all eight original-source roles before a part is eligible for launch.
func discoverCaptureGroupsForWorkflow(objects []captureObject, config domain.CaptureAutomationConfig, workflow domain.WorkflowRecord) ([]captureGroup, error) {
groups, err := discoverCaptureGroups(objects, config)
if err != nil || !workflowUsesV4CaptureDelivery(workflow) {
return groups, err
}
for i := range groups {
group := &groups[i]
record := domain.CaptureRecord{Bucket: config.Bucket, CaptureID: group.CaptureID, Prefix: group.Prefix,
SourcePrefix: group.SourcePrefix, SourceKeys: group.SourceKeys, Fingerprint: group.Fingerprint}
if err := validateV4CaptureSourceSelection(workflow, &record); err != nil {
group.Missing = append(group.Missing, err.Error())
}
metadataKey := group.SourcePrefix + "/meta_" + path.Base(group.SourcePrefix) + ".json"
for _, object := range group.Objects {
if strings.EqualFold(object.Key, metadataKey) && object.Size <= 0 {
group.Missing = append(group.Missing, "v4 原始源文件为空 "+object.Key)
}
}
}
return groups, nil
}

View File

@ -0,0 +1,89 @@
package worker
import (
"fmt"
"path"
"regexp"
"strings"
"droboticflow/internal/domain"
)
func workflowUsesV4CaptureDelivery(workflow domain.WorkflowRecord) bool {
for _, node := range workflow.Nodes {
if node.Data.OperatorID != "operator-builtin-package" {
continue
}
// Match the package operator's boolean parser so alternate true values
// cannot select v4 while silently bypassing the source preflight.
switch strings.ToLower(strings.TrimSpace(nodeHyperparamsMap(node)["DELIVERY_JSON_V4"])) {
case "1", "true", "yes", "on":
return true
}
}
return false
}
// validateV4CaptureSourceSelection checks the exact frozen selection, without
// listing storage or reconstructing a part from a whole-capture input. Call it
// before claiming a new launch; already-created runs keep their own inputs.
func validateV4CaptureSourceSelection(workflow domain.WorkflowRecord, record *domain.CaptureRecord) error {
if !workflowUsesV4CaptureDelivery(workflow) {
return nil
}
if record == nil {
return fmt.Errorf("v4 original source requires a capture record")
}
root, prefix := record.SourcePrefix, record.Prefix
if root == "" || path.Clean(root) != root || strings.HasPrefix(root, "/") || strings.ContainsAny(root, "\\\x00") || root == ".." || strings.HasPrefix(root, "../") {
return fmt.Errorf("v4 original source requires a safe capture root")
}
part := path.Base(prefix)
capture := path.Base(root)
if !regexp.MustCompile(`^part[0-9]{4,}$`).MatchString(part) || prefix != root+"/"+part || record.CaptureID != capture+"/"+part {
return fmt.Errorf("v4 original source requires one explicit capture/part selection")
}
if strings.TrimSpace(record.Bucket) == "" || strings.TrimSpace(record.Fingerprint) == "" {
return fmt.Errorf("v4 original source requires a bucket and snapshot fingerprint")
}
capPattern, partPattern := regexp.QuoteMeta(capture), regexp.QuoteMeta(part)
// These paths match the eight original-source roles consumed by v4. In
// particular, metadata is rooted at the capture, never at the part directory.
requirements := []captureRequirement{
{"左相机视频", partPattern + `/` + capPattern + `_camera_left_` + partPattern + `(?:_[a-z0-9_]+)?\.(?:mp4|mov|mkv)`},
{"右相机视频", partPattern + `/` + capPattern + `_camera_right_` + partPattern + `(?:_[a-z0-9_]+)?\.(?:mp4|mov|mkv)`},
{"左相机 PTS", partPattern + `/` + capPattern + `_camera_left_` + partPattern + `(?:_[a-z0-9_]+)?_pts\.csv`},
{"右相机 PTS", partPattern + `/` + capPattern + `_camera_right_` + partPattern + `(?:_[a-z0-9_]+)?_pts\.csv`},
{"IMU", partPattern + `/` + capPattern + `_imu_` + partPattern + `\.csv`},
{"相机标定", capPattern + `_calibration_camera\.yaml`},
{"IMU 标定", capPattern + `_calibration_imu\.yaml`},
{"meta_" + capture + ".json", `meta_` + capPattern + `\.json`},
}
counts := make([]int, len(requirements))
patterns := make([]*regexp.Regexp, len(requirements))
for i, requirement := range requirements {
patterns[i] = regexp.MustCompile(`(?i)^` + requirement.Pattern + `$`)
}
seen := map[string]bool{}
for _, key := range record.SourceKeys {
if !strings.HasPrefix(key, root+"/") || path.Clean(key) != key || strings.ContainsAny(key, "\\\x00") || seen[key] {
return fmt.Errorf("v4 original source contains an unsafe or duplicate object selection")
}
seen[key] = true
relative := strings.TrimPrefix(key, root+"/")
if strings.Contains(relative, "/") && !strings.HasPrefix(key, prefix+"/") {
return fmt.Errorf("v4 original source selection includes another part")
}
for i, pattern := range patterns {
if pattern.MatchString(relative) {
counts[i]++
}
}
}
for i, count := range counts {
if count != 1 {
return fmt.Errorf("v4 original source requires exactly one %s (found %d)", requirements[i].Name, count)
}
}
return nil
}

View File

@ -0,0 +1,242 @@
package worker
import (
"context"
"reflect"
"strings"
"testing"
"droboticflow/internal/domain"
"droboticflow/internal/store"
)
func captureV4Workflow(enabled bool) domain.WorkflowRecord {
value := "false"
if enabled {
value = "true"
}
return domain.WorkflowRecord{Nodes: []domain.WorkflowNode{{ID: "package", Data: domain.WorkflowNodeData{
OperatorID: "operator-builtin-package", Hyperparams: []domain.HyperParam{{Key: "DELIVERY_JSON_V4", Value: value}},
}}}}
}
func captureV4Objects() []captureObject {
objects := append(calibrationObjects(), partObjects("part0001")...)
return append(objects, captureObject{Key: "incoming/" + layoutCapture + "/meta_" + layoutCapture + ".json", Size: 20, ETag: "metadata-version"})
}
func TestDiscoverCaptureGroupsForWorkflowV4RequiresCaptureMetadata(t *testing.T) {
config := normalizedCaptureAutomation(&domain.CaptureAutomationConfig{Prefix: "incoming", Bucket: "synthetic-bucket"})
objects := append(calibrationObjects(), partObjects("part0001")...)
for _, enabled := range []bool{false, true} {
groups, err := discoverCaptureGroupsForWorkflow(objects, config, captureV4Workflow(enabled))
if err != nil || len(groups) != 1 {
t.Fatalf("groups: %+v, %v", groups, err)
}
if enabled {
if !strings.Contains(strings.Join(groups[0].Missing, " "), "meta_"+layoutCapture+".json") {
t.Fatal("v4 admitted a seven-file part", groups[0])
}
} else if len(groups[0].Missing) != 0 || len(groups[0].SourceKeys) != 7 {
t.Fatal("legacy intake must keep metadata optional", groups[0])
}
}
}
func TestV4CaptureMetadataMustBeNonemptyAndAtItsOwnCaptureRoot(t *testing.T) {
config := normalizedCaptureAutomation(&domain.CaptureAutomationConfig{Prefix: "incoming", Bucket: "synthetic-bucket"})
for _, location := range []string{"missing", "other-capture", "inside-part", "empty", "complete"} {
t.Run(location, func(t *testing.T) {
objects := captureV4Objects()
meta := &objects[len(objects)-1]
switch location {
case "missing":
objects = objects[:len(objects)-1]
case "other-capture":
meta.Key = "incoming/other/meta_" + layoutCapture + ".json"
case "inside-part":
meta.Key = "incoming/" + layoutCapture + "/part0001/meta_" + layoutCapture + ".json"
case "empty":
meta.Size = 0
}
groups, err := discoverCaptureGroupsForWorkflow(objects, config, captureV4Workflow(true))
if err != nil {
t.Fatal(err)
}
for _, group := range groups {
if group.CaptureID != layoutCapture+"/part0001" {
continue
}
if (len(group.Missing) == 0) != (location == "complete") {
t.Fatalf("%s: %+v", location, group)
}
return
}
t.Fatal("target part not discovered")
})
}
}
func TestV4CapturePreflightRejectsIncompleteOrUnprovableSelections(t *testing.T) {
config := normalizedCaptureAutomation(&domain.CaptureAutomationConfig{Prefix: "incoming", Bucket: "synthetic-bucket"})
groups, err := discoverCaptureGroups(captureV4Objects(), config)
if err != nil || len(groups) != 1 || len(groups[0].Missing) != 0 {
t.Fatalf("complete fixture: %+v, %v", groups, err)
}
group := groups[0]
base := domain.CaptureRecord{Bucket: config.Bucket, CaptureID: group.CaptureID, Prefix: group.Prefix,
SourcePrefix: group.SourcePrefix, SourceKeys: group.SourceKeys, Fingerprint: group.Fingerprint}
for i, key := range base.SourceKeys {
t.Run("missing/"+key, func(t *testing.T) {
record := base
record.SourceKeys = append(append([]string{}, base.SourceKeys[:i]...), base.SourceKeys[i+1:]...)
if err := validateV4CaptureSourceSelection(captureV4Workflow(true), &record); err == nil {
t.Fatal("incomplete selection admitted", key)
}
if err := validateV4CaptureSourceSelection(captureV4Workflow(false), &record); err != nil {
t.Fatal("legacy preflight changed", err)
}
})
}
for _, kind := range []string{"whole-capture", "cross-part", "duplicate-key", "duplicate-role", "wrong-metadata", "unsafe-key", "missing-fingerprint"} {
t.Run(kind, func(t *testing.T) {
record := base
record.SourceKeys = append([]string{}, base.SourceKeys...)
switch kind {
case "whole-capture":
record.Prefix, record.CaptureID = record.SourcePrefix, layoutCapture
case "cross-part":
record.SourceKeys = append(record.SourceKeys, record.SourcePrefix+"/part0002/foreign.mp4")
case "duplicate-key":
record.SourceKeys = append(record.SourceKeys, record.SourceKeys[0])
case "duplicate-role":
record.SourceKeys = append(record.SourceKeys, record.Prefix+"/"+layoutCapture+"_camera_left_part0001.mp4")
case "wrong-metadata":
for i, key := range record.SourceKeys {
if strings.HasSuffix(key, ".json") {
record.SourceKeys[i] = record.SourcePrefix + "/meta_another_capture.json"
}
}
case "unsafe-key":
record.SourceKeys[0] = record.SourcePrefix + "/../foreign.yaml"
case "missing-fingerprint":
record.Fingerprint = ""
}
if err := validateV4CaptureSourceSelection(captureV4Workflow(true), &record); err == nil {
t.Fatal("unprovable v4 selection admitted", kind)
}
if err := validateV4CaptureSourceSelection(captureV4Workflow(false), &record); err != nil {
t.Fatal("legacy preflight changed", err)
}
})
}
}
func TestV4CaptureSourceAndAuditMatchPackageBooleanValues(t *testing.T) {
ctx := context.Background()
ds := store.NewMemoryStore()
runtime := &Runtime{store: ds}
for _, tc := range []struct {
value string
enabled bool
}{
{"true", true}, {"TRUE", true}, {"1", true}, {"yes", true}, {" on ", true},
{"false", false}, {"FALSE", false}, {"0", false}, {"no", false}, {"off", false}, {"", false},
} {
workflow := captureV4Workflow(true)
workflow.ID = "source-audit-alias"
workflow.Nodes[0].Data.Hyperparams[0].Value = tc.value
if err := ds.UpsertWorkflow(ctx, workflow); err != nil {
t.Fatal(err)
}
if err := validateV4CaptureSourceSelection(workflow, &domain.CaptureRecord{}); (err != nil) != tc.enabled {
t.Fatalf("source preflight for package boolean %q: %v", tc.value, err)
}
if audit, err := runtime.classificationDeliveryEnabled(ctx, workflow.ID); err != nil || audit != tc.enabled {
t.Fatalf("audit for package boolean %q: enabled=%v, err=%v", tc.value, audit, err)
}
}
workflow := captureV4Workflow(true)
workflow.ID = "non-package-alias"
workflow.Nodes[0].Data.OperatorID = "unrelated-operator"
if err := ds.UpsertWorkflow(ctx, workflow); err != nil {
t.Fatal(err)
}
if err := validateV4CaptureSourceSelection(workflow, &domain.CaptureRecord{}); err != nil {
t.Fatal("non-package parameter enabled the v4 gate", err)
}
if audit, err := runtime.classificationDeliveryEnabled(ctx, workflow.ID); err != nil || audit {
t.Fatal("non-package parameter enabled v4 audit", audit, err)
}
if audit, err := runtime.classificationDeliveryEnabled(ctx, "missing-workflow"); err != nil || audit {
t.Fatal("missing workflow enabled v4 audit", audit, err)
}
}
func TestV4CaptureLaunchFreezesCompletePartSourceSelection(t *testing.T) {
ctx := context.Background()
ds := store.NewMemoryStore()
workflow, err := ds.CreateWorkflow(ctx, domain.CreateWorkflowInput{
OrganizationID: domain.DefaultOrganizationID, Name: "new v4 capture workflow", Type: domain.WorkflowTypeStreaming,
WorkflowInputs: []domain.WorkflowInputDefinition{{ID: "sourceDirectory", Type: domain.PortTypeFile, Container: domain.PortContainerDirectory, PayloadMode: domain.ArtifactTransferModeRef, Required: true}},
})
if err != nil {
t.Fatal(err)
}
config := normalizedCaptureAutomation(&domain.CaptureAutomationConfig{
Enabled: true, ConnectionID: "synthetic-source", Bucket: "synthetic-bucket", Prefix: "incoming",
DeliveryPrefix: "deliveries", DeliveryProjectDirectory: "Synthetic__project",
})
workflow.Nodes = captureV4Workflow(true).Nodes
workflow.DefinitionState = domain.WorkflowDefinitionStateReleased
workflow.CaptureAutomation = &config
if err = ds.UpsertWorkflow(ctx, *workflow); err != nil {
t.Fatal(err)
}
objects := append(captureV4Objects(), partObjects("part0002")...)
groups, err := discoverCaptureGroupsForWorkflow(objects, config, *workflow)
if err != nil || len(groups) != 2 {
t.Fatalf("groups: %+v, %v", groups, err)
}
runtime := &Runtime{store: ds}
for _, group := range groups {
if len(group.Missing) != 0 {
t.Fatal("complete source rejected", group)
}
record := domain.CaptureRecord{
ID: captureRecordID(workflow.ID, group.Prefix), OrganizationID: workflow.OrganizationID, WorkflowID: workflow.ID,
ConnectionID: config.ConnectionID, Bucket: config.Bucket, CaptureID: group.CaptureID, Prefix: group.Prefix,
SourcePrefix: group.SourcePrefix, SourceKeys: group.SourceKeys, Fingerprint: group.Fingerprint, Status: domain.CaptureRecordReady,
}
if err = validateV4CaptureSourceSelection(*workflow, &record); err != nil {
t.Fatal(err)
}
if err = ds.UpsertCaptureRecord(ctx, record); err != nil {
t.Fatal(err)
}
if err = runtime.launchCaptureRun(ctx, *workflow, config, &record); err != nil {
t.Fatal(err)
}
run, err := ds.GetRun(ctx, record.RunID)
if err != nil || run == nil {
t.Fatalf("run missing: %v", err)
}
ref := run.Inputs["sourceDirectory"].Ref
if ref == nil || ref.Prefix != "incoming/"+layoutCapture || ref.CaptureUnit != group.CaptureID || ref.VersionID != group.Fingerprint || len(ref.ObjectKeys) != 8 || !reflect.DeepEqual(ref.ObjectKeys, group.SourceKeys) {
t.Fatalf("source proof did not survive run creation: %+v", ref)
}
if err = domain.ValidateArtifactPayload(run.Inputs["sourceDirectory"], domain.PortTypeFile); err != nil {
t.Fatal("v5.8 input contract rejected selected part", err)
}
// Later discovery cannot rewrite the input used by annotationOriginalSource.
before := append([]string(nil), ref.ObjectKeys...)
record.SourceKeys[0] = "incoming/unrelated/file"
if err = ds.UpsertCaptureRecord(ctx, record); err != nil {
t.Fatal(err)
}
run, err = ds.GetRun(ctx, record.RunID)
if err != nil || !reflect.DeepEqual(run.Inputs["sourceDirectory"].Ref.ObjectKeys, before) {
t.Fatal("frozen run selection followed later discovery", err)
}
}
}

View File

@ -76,17 +76,7 @@ func (runtime *Runtime) classificationDeliveryEnabled(ctx context.Context, workf
if workflow == nil {
return false, nil
}
for _, node := range workflow.Nodes {
if node.Data.OperatorID != "operator-builtin-package" {
continue
}
for _, param := range node.Data.Hyperparams {
if param.Key == "DELIVERY_JSON_V4" && param.Value == "true" {
return true, nil
}
}
}
return false, nil
return workflowUsesV4CaptureDelivery(*workflow), nil
}
// Reuse normal internal-artifact persistence rather than exceeding the 64 KiB

View File

@ -1,14 +1,14 @@
{
"acceptance": {},
"builtAt": "2026-08-31T19:36:30Z",
"digest": "sha256:87965e34785dcc8027934a4291a4d35de0918cf68b681a5ed5cc184e2a4bddf2",
"builtAt": "2026-10-21T16:50:46Z",
"digest": "sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d",
"environmentReleases": {
"fanyu": {
"digest": "sha256:87965e34785dcc8027934a4291a4d35de0918cf68b681a5ed5cc184e2a4bddf2",
"image": "ccr-29eug8s3-vpc.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package:b085e371-fanyu-20260901-capture-v5",
"imageId": "ccr-29eug8s3-vpc.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package@sha256:87965e34785dcc8027934a4291a4d35de0918cf68b681a5ed5cc184e2a4bddf2",
"registry": "ccr-29eug8s3-vpc.cnc.bj.baidubce.com",
"tag": "b085e371-fanyu-20260901-capture-v5"
"digest": "sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d",
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package:delivery-v4-20261022-8f6b531f",
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package@sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d",
"registry": "ccr-29eug8s3-pub.cnc.bj.baidubce.com",
"tag": "delivery-v4-20261022-8f6b531f"
},
"poc": {
"digest": "sha256:fa084bbd0e3b6fedff9582d09c965b460f054ad9e0f5edaf9fc4b9aebcc375cc",
@ -18,14 +18,14 @@
"tag": "feaaf80-poc-handpose-auto-20260825052001"
}
},
"image": "ccr-29eug8s3-vpc.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package:b085e371-fanyu-20260901-capture-v5",
"imageId": "ccr-29eug8s3-vpc.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package@sha256:87965e34785dcc8027934a4291a4d35de0918cf68b681a5ed5cc184e2a4bddf2",
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package:delivery-v4-20261022-8f6b531f",
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package@sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d",
"kanikoJob": "droboticflow-kaniko-op-package-b085e371-fanyu-20260901-capture",
"operatorId": "operator-builtin-package",
"registry": "ccr-29eug8s3-vpc.cnc.bj.baidubce.com",
"registry": "ccr-29eug8s3-pub.cnc.bj.baidubce.com",
"schemaVersion": "droboticflow.operator.release/v1",
"slug": "package",
"sourceCommit": "b085e371b230b304fb0446040dee98793dc2e545",
"sourceTreeSha256": "8cb9d11686fa3c44c33117340ee6b115b6880a0e73e1fc0807d9b997a117fe3d",
"tag": "b085e371-fanyu-20260901-capture-v5"
"sourceCommit": "8f6b531fd322f04d0ecf02d74f77348169f773e9",
"sourceTreeSha256": "1db72650763169528362fa2e66d14689d76b2141de1c8aba0f4721559d3882cc",
"tag": "delivery-v4-20261022-8f6b531f"
}

View File

@ -19,8 +19,8 @@
],
"containerEntrypoint": "backend/operator-runner/main.py",
"entrypoint": "src/operator.py",
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-package:feaaf80-poc-handpose-auto-20260825052001",
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-package@sha256:fa084bbd0e3b6fedff9582d09c965b460f054ad9e0f5edaf9fc4b9aebcc375cc",
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package:delivery-v4-20261022-8f6b531f",
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package@sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d",
"metadata": "operator.json",
"name": "成品打包",
"operatorId": "operator-builtin-package",
@ -83,7 +83,7 @@
"runtimeSource": "operator-repos/builtin/package/src/operator.py",
"schemaVersion": "droboticflow.operator.repository/v1",
"sourceRoot": "operator-repos/builtin/package",
"sourceTreeSha256": "9c91471144466566083b516e71ac10ae948c1d1490d59708ff9b2068daed759a",
"sourceTreeSha256": "1db72650763169528362fa2e66d14689d76b2141de1c8aba0f4721559d3882cc",
"strictPocAcceptanceExternals": [],
"type": "builtin"
}

View File

@ -198,13 +198,19 @@ def legacy_manifest(
return manifest
def expected_manifests() -> dict[pathlib.Path, dict[str, Any]]:
def expected_manifests(slugs: set[str] | None = None) -> dict[pathlib.Path, dict[str, Any]]:
catalog_items = {
item["id"]: item for item in load_json(CATALOG_PATH) if item.get("type") == "builtin"
}
manifests: dict[pathlib.Path, dict[str, Any]] = {}
split_operator_ids: set[str] = set()
for metadata_path in sorted(OPERATOR_REPO_ROOT.glob("*/operator.json")):
metadata_paths = sorted(OPERATOR_REPO_ROOT.glob("*/operator.json"))
if slugs is not None:
available = {path.parent.name for path in metadata_paths} - UNMIGRATED_HISTORICAL_REPOSITORIES
if not slugs or slugs - available:
raise RuntimeError("unknown or empty operator repository selection")
metadata_paths = [path for path in metadata_paths if path.parent.name in slugs]
for metadata_path in metadata_paths:
if metadata_path.parent.name in UNMIGRATED_HISTORICAL_REPOSITORIES:
continue
metadata = load_json(metadata_path)
@ -286,7 +292,7 @@ def expected_manifests() -> dict[pathlib.Path, dict[str, Any]]:
image_id=image_id,
dependencies=dependencies,
)
missing_catalog = sorted(set(catalog_items) - split_operator_ids)
missing_catalog = sorted(set(catalog_items) - split_operator_ids) if slugs is None else []
if missing_catalog:
raise RuntimeError(f"catalog operators missing repository manifests: {', '.join(missing_catalog)}")
return manifests
@ -319,9 +325,10 @@ def main() -> int:
mode = parser.add_mutually_exclusive_group()
mode.add_argument("--write", action="store_true", help="write repository.json files")
mode.add_argument("--check", action="store_true", help="check repository.json files without writing")
parser.add_argument("--slug", action="append", help="publish/check only these independent repositories; default checks every operator")
args = parser.parse_args()
manifests = expected_manifests()
manifests = expected_manifests(set(args.slug) if args.slug is not None else None)
if args.check:
return check_manifests(manifests)
write_manifests(manifests)

View File

@ -0,0 +1,243 @@
#!/usr/bin/env python3
"""Offline name-as-first-tag plan; Mongo apply requires its exact confirmed hash.
Plan: --snapshot FILE --database DB --organization ORG --updated-by USER
[--at ISO_TIMESTAMP] [--output PLAN.json]
Apply: --plan PLAN.json --apply --confirm-plan SHA256
(URI only from DICTIONARY_TAGS_MONGO_URI).
Without --apply no database connection is made. Only annotationLabels changes;
saved classifications and existing nonempty tags are never rewritten. History
is copied/appended inside Mongo, never fetched by this script.
"""
from __future__ import annotations
import argparse
import datetime as dt
import hashlib
import hmac
import json
import os
from pathlib import Path
FIELDS = ("id", "organizationId", "name", "kind", "tags", "version", "status", "updatedAt")
PROJECTION = {**dict.fromkeys(FIELDS, 1), "_id": 0}
COLLECTION = "annotationLabels"
SCHEMA = "droboticflow.dictionary-tags-seed/v1"
ENTRY_FIELDS = {"id", "name", "version", "expectedHash"}
PLAN_FIELDS = {"schemaVersion", "database", "organizationId", "collection", "updatedAt", "updatedByUserId", "entries", "planHash"}
def digest(value):
return hashlib.sha256(json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False).encode()).hexdigest()
def projected(row):
if not isinstance(row, dict) or set(row) - set(FIELDS):
raise ValueError("snapshot/read contains fields outside the metadata whitelist")
return {key: row[key] for key in FIELDS if key in row}
def text(value, field):
if not isinstance(value, str) or not value.strip():
raise ValueError(f"{field} must be a nonempty string")
return value
def timestamp(value):
text(value, "updatedAt")
parsed = dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
if parsed.tzinfo is None:
raise ValueError("updatedAt must include a timezone")
return value
def build_plan(snapshot, *, database, organization, updated_by, at):
rows = snapshot.get("active-dictionary-tag-readiness") if isinstance(snapshot, dict) else snapshot
if not isinstance(rows, list):
raise ValueError("snapshot must contain a list of projected dictionary records")
entries, seen = [], set()
for raw in rows:
row = projected(raw)
if row.get("organizationId") != organization:
raise ValueError("snapshot organization differs from fixed scope")
label_id = text(row.get("id"), "id")
if label_id in seen:
raise ValueError("duplicate dictionary id")
seen.add(label_id)
tags = row.get("tags")
if tags is not None and (not isinstance(tags, list) or any(not isinstance(tag, str) for tag in tags)):
raise ValueError("tags must be an array of strings or absent/null")
if row.get("status") != "active" or tags:
continue
name = text(row.get("name"), "name")
version = row.get("version")
if type(version) is not int or version < 1:
raise ValueError("active dictionary requires a positive integer version")
text(row.get("updatedAt"), "existing updatedAt")
entries.append({"id": label_id, "name": name, "version": version, "expectedHash": digest(row)})
plan = {
"schemaVersion": SCHEMA, "database": text(database, "database"),
"organizationId": text(organization, "organization"), "collection": COLLECTION,
"updatedAt": timestamp(at), "updatedByUserId": text(updated_by, "updated-by"),
"entries": sorted(entries, key=lambda entry: entry["id"]),
}
plan["planHash"] = digest(plan)
validate_plan(plan)
return plan
def validate_plan(plan, confirmation=None):
if not isinstance(plan, dict) or set(plan) != PLAN_FIELDS or plan.get("schemaVersion") != SCHEMA or plan.get("collection") != COLLECTION:
raise ValueError("invalid plan shape or collection")
expected = digest({key: value for key, value in plan.items() if key != "planHash"})
if plan.get("planHash") != expected or (confirmation is not None and not hmac.compare_digest(confirmation, expected)):
raise ValueError("plan hash/confirmation mismatch")
for field in ("database", "organizationId", "updatedByUserId"):
text(plan[field], field)
timestamp(plan["updatedAt"])
if not isinstance(plan["entries"], list):
raise ValueError("entries must be an array")
ids = set()
for entry in plan["entries"]:
if not isinstance(entry, dict) or set(entry) != ENTRY_FIELDS:
raise ValueError("invalid entry shape")
for field in ("id", "name"):
text(entry[field], field)
if entry["id"] in ids or type(entry["version"]) is not int or entry["version"] < 1:
raise ValueError("duplicate id or invalid version")
ids.add(entry["id"])
value = entry["expectedHash"]
if not isinstance(value, str) or len(value) != 64 or any(c not in "0123456789abcdef" for c in value):
raise ValueError("invalid expected metadata hash")
def revision_expression():
# These expressions execute on the server, not in a fetched document.
defaults = {
"requiresNote": False, "version": 1, "code": "", "name": "", "category": "",
"description": "", "example": "", "color": "", "tags": [], "aliases": [],
"status": "", "updatedByUserId": "", "updatedAt": "",
}
return {key: {"$ifNull": ["$" + key, default]} for key, default in defaults.items()}
def update_pipeline(plan, entry):
literal = lambda value: {"$literal": value}
version = {"$add": ["$version", 1]}
old = revision_expression()
changes = {"tags": literal([entry["name"]]), "version": version,
"updatedAt": literal(plan["updatedAt"]), "updatedByUserId": literal(plan["updatedByUserId"])}
new = {**old, **changes}
history = {"$ifNull": ["$history", []]}
return [{"$set": {
**changes,
"history": {"$concatArrays": [
{"$cond": [{"$eq": [{"$size": history}, 0]}, [old], history]}, [new],
]},
}}]
def cas_filter(row):
# Preserve missing-vs-null and compare arrays exactly through $expr.
return {
"id": row["id"], "organizationId": row["organizationId"],
"$and": [{field: {"$exists": field in row}} for field in FIELDS],
"$expr": {"$and": [{"$eq": ["$" + field, {"$literal": value}]} for field, value in row.items()]},
}
def apply_plan(database, plan, confirmation):
validate_plan(plan, confirmation)
if database.name != plan["database"]:
raise ValueError("database differs from fixed plan scope")
collection = database[COLLECTION]
entries = plan["entries"]
if not entries:
return {"updated": 0, "alreadyApplied": 0}
rows = {}
for raw in collection.find({"organizationId": plan["organizationId"], "id": {"$in": [entry["id"] for entry in entries]}}, PROJECTION):
row = projected(raw)
if row.get("id") in rows:
raise ValueError("duplicate current dictionary id")
rows[row.get("id")] = row
pending, already = [], 0
# Preflight every entry before the first write. CAS also checks the interval
# between this read and each atomic update; partial runs can be retried.
for entry in entries:
row = rows.get(entry["id"])
if row and all(row.get(key) == value for key, value in {
"id": entry["id"], "organizationId": plan["organizationId"], "name": entry["name"],
"status": "active", "tags": [entry["name"]], "version": entry["version"] + 1,
"updatedAt": plan["updatedAt"],
}.items()):
already += 1
continue
if row is None or digest(row) != entry["expectedHash"]:
raise ValueError(f"metadata drift or missing dictionary: {entry['id']}")
if row.get("status") != "active" or row.get("tags") or row.get("name") != entry["name"] or row.get("version") != entry["version"] or row.get("organizationId") != plan["organizationId"]:
raise ValueError("entry is not eligible for name-as-first-tag initialization")
pending.append((entry, row))
updated = 0
for entry, row in pending:
result = collection.update_one(cas_filter(row), update_pipeline(plan, entry), upsert=False)
if result.matched_count != 1 or result.modified_count != 1:
raise ValueError(f"concurrent dictionary change; stop and replan: {entry['id']}")
updated += 1
return {"updated": updated, "alreadyApplied": already}
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
source = parser.add_mutually_exclusive_group(required=True)
source.add_argument("--snapshot", type=Path)
source.add_argument("--plan", type=Path)
for option in ("database", "organization", "updated-by", "at"):
parser.add_argument("--" + option)
parser.add_argument("--output", type=Path)
parser.add_argument("--apply", action="store_true")
parser.add_argument("--confirm-plan")
args = parser.parse_args(argv)
if args.apply and (not args.plan or not args.confirm_plan or args.output):
parser.error("apply requires --plan and --confirm-plan; --output is plan-only")
if args.confirm_plan and not args.apply:
parser.error("--confirm-plan requires --apply")
if args.snapshot:
if not all((args.database, args.organization, args.updated_by)):
parser.error("snapshot planning requires --database, --organization and --updated-by")
plan = build_plan(json.loads(args.snapshot.read_text()), database=args.database, organization=args.organization,
updated_by=args.updated_by, at=args.at or dt.datetime.now(dt.timezone.utc).isoformat().replace("+00:00", "Z"))
else:
if any((args.database, args.organization, args.updated_by, args.at)):
parser.error("saved plan scope and timestamps cannot be overridden")
plan = json.loads(args.plan.read_text())
validate_plan(plan)
if not args.apply:
rendered = json.dumps(plan, ensure_ascii=False, sort_keys=True, indent=2) + "\n"
if args.output:
args.output.write_text(rendered)
else:
print(rendered, end="")
return 0
validate_plan(plan, args.confirm_plan)
uri = os.environ.get("DICTIONARY_TAGS_MONGO_URI")
if not uri:
parser.error("DICTIONARY_TAGS_MONGO_URI must be set for apply")
from pymongo import MongoClient
from pymongo.errors import PyMongoError
try:
with MongoClient(uri, serverSelectionTimeoutMS=10000) as client:
result = apply_plan(client[plan["database"]], plan, args.confirm_plan)
except PyMongoError:
raise ValueError("Mongo operation failed; inspect state with the same narrow projection before retrying") from None
print(json.dumps({"planHash": plan["planHash"], **result}, sort_keys=True))
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except ValueError as error:
raise SystemExit(str(error)) from None

View File

@ -0,0 +1,40 @@
"""A scoped publication validates its own release; the full check stays strict."""
import pytest
from scripts import generate_operator_repository_manifests as manifests
def test_package_publication_checks_its_actual_release_without_loading_other_releases(monkeypatch):
load = manifests.load_release
calls = []
def selected(slug):
calls.append(slug)
if slug != "package":
raise RuntimeError("unrelated release must still fail the full check")
return load(slug)
monkeypatch.setattr(manifests, "load_release", selected)
generated = manifests.expected_manifests({"package"})
assert calls == ["package"]
assert len(generated) == 2
assert all(path.parent.name == "package" for path in generated)
assert all(value["operatorId"] == "operator-builtin-package" for value in generated.values())
with pytest.raises(RuntimeError, match="unrelated release"):
manifests.expected_manifests()
@pytest.mark.parametrize("selection", [set(), {"not-an-operator"}, {"../package"}, {"temporal_causal_eval"}])
def test_unknown_or_empty_selection_cannot_silently_pass(selection):
with pytest.raises(RuntimeError, match="selection"):
manifests.expected_manifests(selection)
def test_selected_release_errors_are_not_ignored(monkeypatch):
def invalid(_slug):
raise RuntimeError("selected digest drift")
monkeypatch.setattr(manifests, "load_release", invalid)
with pytest.raises(RuntimeError, match="selected digest drift"):
manifests.expected_manifests({"package"})

View File

@ -0,0 +1,275 @@
"""Offline plan/CAS checks; opt-in integration uses a new synthetic database."""
import copy
import json
import os
from types import SimpleNamespace
from urllib.parse import urlsplit
import uuid
import pytest
from scripts import seed_dictionary_tags as seed
AT = "2026-10-22T08:00:00Z"
MISSING = object()
def label(label_id="one", **changes):
return {
"id": label_id, "organizationId": "synthetic-org", "kind": "action", "name": "拿取",
"version": 1, "status": "active", "updatedAt": "2026-10-21T08:00:00Z",
"code": "pick", "description": "synthetic description", "example": "synthetic example",
"aliases": ["synthetic alias"], "category": "category", "requiresNote": True,
"color": "blue", "updatedByUserId": "old-user", **changes,
}
def metadata(doc):
return {field: copy.deepcopy(doc[field]) for field in seed.FIELDS if field in doc}
def plan_for(docs, database="synthetic-dictionary-tags"):
return seed.build_plan({"active-dictionary-tag-readiness": [metadata(doc) for doc in docs]},
database=database, organization="synthetic-org", updated_by="seed-test", at=AT)
def expression(value, doc):
"""Small generic Mongo expression evaluator for the offline fake server."""
if isinstance(value, str) and value.startswith("$"):
return doc.get(value[1:], MISSING)
if isinstance(value, list):
return [expression(item, doc) for item in value]
if not isinstance(value, dict):
return value
if "$literal" in value:
return copy.deepcopy(value["$literal"])
if "$ifNull" in value:
first, fallback = value["$ifNull"]
result = expression(first, doc)
return expression(fallback, doc) if result is MISSING or result is None else result
if "$cond" in value:
condition, yes, no = value["$cond"]
return expression(yes if expression(condition, doc) else no, doc)
if "$eq" in value:
a, b = expression(value["$eq"], doc)
return a == b
if "$and" in value:
return all(expression(item, doc) for item in value["$and"])
if "$size" in value:
return len(expression(value["$size"], doc))
if "$add" in value:
return sum(expression(value["$add"], doc))
if "$concatArrays" in value:
return sum(expression(value["$concatArrays"], doc), [])
return {key: expression(item, doc) for key, item in value.items()}
class FakeCollection:
def __init__(self, docs):
self.docs = copy.deepcopy(docs)
self.writes = []
self.reads = []
self.before_update = None
def find(self, query, projection):
assert projection == seed.PROJECTION # No full-document/history read.
self.reads.append(copy.deepcopy(projection))
return [metadata(doc) for doc in self.docs if doc["organizationId"] == query["organizationId"] and doc["id"] in query["id"]["$in"]]
def update_one(self, query, pipeline, *, upsert):
assert upsert is False
if self.before_update:
self.before_update(self)
self.writes.append((query, pipeline))
for doc in self.docs:
if doc["id"] != query["id"] or doc["organizationId"] != query["organizationId"]:
continue
if not all((field in doc) == options["$exists"] for clause in query["$and"] for field, options in clause.items()):
continue
if not expression(query["$expr"], doc):
continue
assert len(pipeline) == 1 and set(pipeline[0]) == {"$set"}
changes = expression(pipeline[0]["$set"], doc)
assert set(changes) == {"tags", "version", "updatedAt", "updatedByUserId", "history"}
doc.update(changes)
return SimpleNamespace(matched_count=1, modified_count=1)
return SimpleNamespace(matched_count=0, modified_count=0)
class FakeDatabase:
def __init__(self, docs, name="synthetic-dictionary-tags"):
self.name = name
self.collection = FakeCollection(docs)
def __getitem__(self, name):
assert name == "annotationLabels"
return self.collection
def test_offline_plan_is_fixed_minimal_and_skips_existing_tags_and_inactive():
docs = [label(), label("tagged", tags=["existing"]), label("inactive", status="inactive")]
plan = plan_for(docs)
assert plan == plan_for(docs)
assert len(plan["entries"]) == 1
assert set(plan["entries"][0]) == seed.ENTRY_FIELDS
assert "description" not in json.dumps(plan)
with pytest.raises(ValueError, match="whitelist"):
seed.build_plan([docs[0]], database="x", organization="synthetic-org", updated_by="user", at=AT)
with pytest.raises(ValueError, match="organization"):
plan_for([label(organizationId="other")])
@pytest.mark.parametrize("history", [MISSING, None, []])
def test_empty_history_gets_old_and_new_revisions_without_fetching_fields(history):
doc = label(name="$literal-name")
if history is not MISSING:
doc["history"] = history
database = FakeDatabase([doc])
plan = plan_for([doc])
assert seed.apply_plan(database, plan, plan["planHash"]) == {"updated": 1, "alreadyApplied": 0}
result = database.collection.docs[0]
assert result["tags"] == ["$literal-name"] and result["version"] == 2
assert [rev["version"] for rev in result["history"]] == [1, 2]
assert result["history"][0]["tags"] == []
for field in ("description", "example", "aliases", "category", "code", "requiresNote", "color", "name"):
assert result[field] == doc[field]
assert result["history"][0][field] == result["history"][1][field] == doc[field]
assert result["history"][0]["updatedByUserId"] == "old-user"
assert result["history"][1]["updatedByUserId"] == "seed-test"
assert result["history"][1]["updatedAt"] == AT
assert seed.apply_plan(database, plan, plan["planHash"]) == {"updated": 0, "alreadyApplied": 1}
assert len(database.collection.writes) == 1
def test_existing_history_is_preserved_without_truncation():
history = [{"version": index, "description": f"history-{index}"} for index in range(1, 105)]
doc = label(version=104, history=history, tags=[])
database = FakeDatabase([doc])
plan = plan_for([doc])
seed.apply_plan(database, plan, plan["planHash"])
result = database.collection.docs[0]
assert result["history"][:-1] == history
assert result["history"][-1]["version"] == 105
assert result["history"][-1]["tags"] == [doc["name"]]
@pytest.mark.parametrize("field,value", [
("id", "changed"), ("organizationId", "other"), ("name", "changed"), ("kind", "slice"),
("tags", ["other"]), ("version", 2), ("status", "inactive"), ("updatedAt", AT),
])
def test_any_projected_drift_aborts_before_writes(field, value):
docs = [label(), label("two")]
plan = plan_for(docs)
database = FakeDatabase(docs)
database.collection.docs[1][field] = value
with pytest.raises(ValueError, match="drift|missing"):
seed.apply_plan(database, plan, plan["planHash"])
assert database.collection.writes == []
assert "tags" not in database.collection.docs[0]
@pytest.mark.parametrize("change", [{"version": 2}, {"updatedAt": AT}, {"tags": None}, {"kind": "slice"}])
def test_atomic_cas_rejects_change_after_narrow_preflight(change):
doc = label()
database = FakeDatabase([doc])
database.collection.before_update = lambda collection: collection.docs[0].update(change)
plan = plan_for([doc])
with pytest.raises(ValueError, match="concurrent"):
seed.apply_plan(database, plan, plan["planHash"])
assert "history" not in database.collection.docs[0]
assert database.collection.docs[0].get("tags") is None
def test_pipeline_uses_current_server_fields_even_without_versioned_edit():
doc = label()
database = FakeDatabase([doc])
database.collection.before_update = lambda collection: collection.docs[0].update(description="new synthetic server text")
plan = plan_for([doc])
seed.apply_plan(database, plan, plan["planHash"])
assert database.collection.docs[0]["history"][-1]["description"] == "new synthetic server text"
def test_confirmation_and_scope_fail_before_database_reads():
database = FakeDatabase([label()])
plan = plan_for([label()])
with pytest.raises(ValueError, match="hash"):
seed.apply_plan(database, plan, "wrong")
tampered = {**plan, "updatedByUserId": "other"}
with pytest.raises(ValueError, match="hash"):
seed.apply_plan(database, tampered, plan["planHash"])
database.name = "wrong-database"
with pytest.raises(ValueError, match="database"):
seed.apply_plan(database, plan, plan["planHash"])
assert database.collection.reads == [] and database.collection.writes == []
def test_cli_default_is_offline_and_saved_plan_scope_cannot_be_overridden(tmp_path, monkeypatch):
path = tmp_path / "snapshot.json"
path.write_text(json.dumps({"active-dictionary-tag-readiness": [metadata(label())]}))
output = tmp_path / "plan.json"
monkeypatch.delenv("DICTIONARY_TAGS_MONGO_URI", raising=False)
assert seed.main(["--snapshot", str(path), "--database", "synthetic-dictionary-tags", "--organization", "synthetic-org",
"--updated-by", "seed-test", "--at", AT, "--output", str(output)]) == 0
assert json.loads(output.read_text()) == plan_for([label()])
assert seed.main(["--plan", str(output)]) == 0
with pytest.raises(SystemExit):
seed.main(["--plan", str(output), "--database", "other"])
with pytest.raises(SystemExit):
seed.main(["--plan", str(output), "--apply"])
@pytest.mark.skipif(not os.environ.get("DICTIONARY_TAGS_TEST_MONGO_URI"), reason="requires isolated synthetic Mongo on 127.0.0.1:27029")
def test_real_mongo_pipeline_preserves_history_and_uses_only_metadata_projection():
from pymongo import MongoClient, monitoring
uri = os.environ["DICTIONARY_TAGS_TEST_MONGO_URI"]
parsed = urlsplit(uri)
assert parsed.scheme == "mongodb" and parsed.hostname == "127.0.0.1" and parsed.port == 27029
assert not parsed.username and not parsed.password and parsed.path in ("", "/")
class Reads(monitoring.CommandListener):
def __init__(self):
self.projections = []
def started(self, event):
if event.command_name == "find":
self.projections.append(dict(event.command.get("projection", {})))
def succeeded(self, event):
pass
def failed(self, event):
pass
listener = Reads()
client = MongoClient(uri, serverSelectionTimeoutMS=5000, event_listeners=[listener])
name = "dictionary_tags_seed_test_" + uuid.uuid4().hex
database = client[name]
old_history = [{"version": 1, "description": "original historical description", "tags": []}]
docs = [label("history", history=old_history), label("empty", history=[], name="$server-field"), label("missing"), label("null", history=None)]
try:
database.annotationLabels.insert_many(copy.deepcopy(docs))
plan = plan_for(docs, database=name)
assert seed.apply_plan(database, plan, plan["planHash"]) == {"updated": 4, "alreadyApplied": 0}
assert listener.projections == [seed.PROJECTION]
# Full reads here are exclusively test assertions on our synthetic DB.
results = {doc["id"]: doc for doc in database.annotationLabels.find({})}
assert results["history"]["history"][:-1] == old_history
for original in docs:
result = results[original["id"]]
assert result["version"] == 2 and result["tags"] == [original["name"]]
assert result["history"][-1]["version"] == 2
assert result["history"][-1]["tags"] == [original["name"]]
for field in ("description", "example", "aliases", "code", "category", "requiresNote", "color"):
assert result[field] == result["history"][-1][field] == original[field]
if original["id"] != "history":
assert len(result["history"]) == 2
assert result["history"][0]["version"] == 1 and result["history"][0]["tags"] == []
listener.projections.clear()
assert seed.apply_plan(database, plan, plan["planHash"]) == {"updated": 0, "alreadyApplied": 4}
assert listener.projections == [seed.PROJECTION]
finally:
client.drop_database(name)
client.close()