🚀 Prepare isolated v4 activation with safe workflow handoff and tag seeding
This commit is contained in:
parent
8f6b531fd3
commit
a777c42c4e
@ -16,8 +16,8 @@
|
||||
],
|
||||
"containerEntrypoint": "backend/operator-runner/main.py",
|
||||
"entrypoint": "operator.py",
|
||||
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-package:feaaf80-poc-handpose-auto-20260825052001",
|
||||
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-package@sha256:fa084bbd0e3b6fedff9582d09c965b460f054ad9e0f5edaf9fc4b9aebcc375cc",
|
||||
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package:delivery-v4-20261022-8f6b531f",
|
||||
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package@sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d",
|
||||
"metadata": "operator.json",
|
||||
"name": "成品打包",
|
||||
"operatorId": "operator-builtin-package",
|
||||
|
||||
@ -45,3 +45,11 @@
|
||||
整采集的多 part 输入可能无法证明单 part 的原始来源。普通物理切片、源索引及元数据复用会省略不可靠的新增 `originalSource` 并保留诊断,不因这项可选证据阻塞旧任务;跨 Run/组织错误与存储失败仍拒绝,已有冻结来源不追随新目录。真正生成或发布 v4 时,缺原始来源、标签或时间证据仍被打包/批次门禁拒绝,不能据此输出不完整 JSON。
|
||||
|
||||
回退旧 API 不是无损写入保证:新版本保存分类后,旧版本重建草稿数组可能丢弃字典快照。回退前应暂停相关写操作或准备能保留新字段的兼容回退版本;只保留恢复历史不等于能自动恢复所有后续修改。已交付不可变文件不因此原地迁移或覆盖。
|
||||
|
||||
## 2026-10-22 启用决定
|
||||
|
||||
用户随后确认启用 v4,范围仅为切换后新启动的工作流;在途任务、它们后续的审核/打包子运行和已交付文件继续原协议。此决定替代上节“继续保持关闭”的准备范围,不代表生产开关已经启用。实际发布仍需完成镜像、兼容性和目标核对,并审查具体部署及工作流切换计划。
|
||||
|
||||
用户明确确认当前无 tags 的启用词条以词条名称作为第一个独立 tags 标签。该初始化是一次有界字典变更:保留词条 ID、名称、用途、category、aliases 和既有历史,递增版本并记录新修订;已有非空 tags 不覆盖。历史任务中的冻结快照不因此重写,亦不反查新词条替代历史分类。后续 tags 仍由管理员维护。
|
||||
|
||||
新工作流使用新的定义版本及固定打包镜像。旧定义不原地改参数或镜像;切换必须保留已启动采集的原工作流归属,并阻止新监听器重新接入这些目录。新 v4 接入还需完整的原始文件选择,包含采集 meta;旧协议保留原有接入要求。准备与验收步骤见 [启用核对表](release/delivery-v4-activation.md)。
|
||||
|
||||
27
docs/release/delivery-v4-activation.md
Normal file
27
docs/release/delivery-v4-activation.md
Normal file
@ -0,0 +1,27 @@
|
||||
# v4 新工作流启用核对表
|
||||
|
||||
本页记录 2026-10-22 用户确认的范围,不是已部署证明。实际摘要、目标 UID、镜像和执行状态保存在被 Git 忽略的发布记录中。
|
||||
|
||||
## 范围与顺序
|
||||
|
||||
1. 在独立工作树验证 API/处理器、Worker、独立 package 镜像,保存源码、基础镜像、产物字节与 digest 的对应证据。API/处理器使用 `scripts/release.py`;Worker、字典和工作流切换另附有界计划。
|
||||
2. 在 v4 尚未激活时,按发布工具先处理器后 API;随后将 Worker 暂停至 0 副本并确认旧扫描进程退出。现有独立算子 Job 不删除,调度及状态收敛暂时停顿。完成字典和工作流切换后更新兼容新旧协议的 Worker 镜像与构建标记,恢复原来的 2 副本,再开始新版本接入。不更新 Coordinator、validator、Secret、资源规格或存储配置。
|
||||
3. 按预览清单为当前无 tags 的启用词条初始化 `tags=[名称]`,递增版本并追加历史。逐条比较原版本、名称、状态、更新时间与标签;发现漂移停止,不覆盖管理员的新修改。此步骤不更新任务、草稿或已冻结字典选择。
|
||||
4. 对每个目标项目创建隐藏的新工作流版本,仅修改 package 的固定镜像、`PART_INDEX_DELIVERY=true`、`DELIVERY_JSON_V4=true`、`DELIVERY_JSON_V3=false`、`SOURCE_INDEX_DELIVERY=true`、`PIN_OPERATOR_IMAGE=true`,并添加内部 JSON 输出 `deliveryAudit`。保留原节点、边、输入、供应商、接入目录、并发和验收抽样策略;逐项比较允许变更以外的内容。
|
||||
5. 检查内置 artifact 存储与客户目录隔离、完整的每 part 原始选择及 meta、分类标签、两相机时钟和验收固定修订。全舍弃、无描述继承、无效动作及独立动作词条均须有验证证据。
|
||||
6. 在停止新接入扫描并确认旧扫描已退出的短窗口内激活新版本。仅未启动采集可以转入新版本;`launching`、有 Run 的采集及其父子运行保留旧归属,新监听器识别并排除这些已接入目录。保留原记录 ID,防止 `(workflowId,prefix)` 冲突;不能删除登记后重扫。
|
||||
7. 只读检查目标版本、旧任务归属、新旧镜像及实际交付格式。生产写入验收必须另列隔离项目、账号和合成输入范围;单凭 Ready、模拟存储测试或单独算子测试不能标记真实业务验收通过。
|
||||
|
||||
## 交付与回退
|
||||
|
||||
客户文件只含最终分类、标签、区间和来源。审核人员、批注、返工和阶段事件只存内部 `deliveryAudit`,超限或写失败必须阻断客户发布;不得静默省略。
|
||||
|
||||
旧 v1–v3 算子输出须有字节兼容回归,旧工作流继续固定原 package 镜像。不可变交付目录、归属标记和已发布索引不原地覆盖。新批次发布仍按已有预览和确认接口执行。
|
||||
|
||||
切换失败或结果未知时停止后续写入,先只读核对实际状态。新版本已经接入的任务不能被重新绑定为旧协议;退回监听器也不能重复接入这些目录。旧 API 会在后续保存时丢弃新字典字段,不能把直接换回旧镜像描述为无损回退。需要保留新字段的兼容回退候选,或单独确认暂停相关写入的恢复方案。回退不撤销已确认标签初始化、不删除内部审核证据或客户新批次。
|
||||
|
||||
## 发布门禁
|
||||
|
||||
发布必须同时具有具体镜像来源、`release.py` 检查、Worker/工作流/字典计划、上线前漂移复核及用户对展示后计划的部署授权。算子历史 release manifest 的已知漂移不得通过重算旧摘要掩盖;只能为真实构建和发布的新 package 绑定新的来源及摘要,其他环境和算子记录保留。
|
||||
|
||||
独立 package 发布后使用 `OPERATOR_RELEASE_ENV=fanyu python3 scripts/generate_operator_repository_manifests.py --slug package --check` 验证这一算子的发布和生成清单;不带 `--slug` 的全量检查行为保持严格。未修改算子的既有漂移另列遗留,不更新其镜像、源码摘要或目录记录来凑齐通过。所选 package 的任何摘要漂移仍阻断本次发布。
|
||||
@ -32,6 +32,7 @@
|
||||
["npm", "run", "test:browser:slice-input"],
|
||||
["npm", "run", "test:browser:annotation-persistence"],
|
||||
["go", "test", "./internal/deliveryindex", "-count=1"],
|
||||
["go", "test", "./internal/api", "-run", "TestDurableAnnotationOutbox|TestSliceRecovery|TestReviewCannotMutateAnnotation|TestReviewDecision|Test.*ReviewDraft|TestReviewConfirmations|TestRouteSnapshot|Test.*Dictionary|TestProjectSliceNames|TestLegacyMaterializationWithoutPartScopedOriginalSelection|TestOptionalOriginalSource|TestAnnotationOriginalSource|TestClassificationBatch", "-count=1"]
|
||||
["go", "test", "./internal/worker", "-run", "TestCaptureCutover|Test.*V4Capture|TestDiscoverCaptureGroupsForWorkflow|TestClassificationAudit|TestInternalAuditFailure", "-count=1"],
|
||||
["go", "test", "./internal/api", "-run", "TestDurableAnnotationOutbox|TestSliceRecovery|TestReviewCannotMutateAnnotation|TestReviewDecision|Test.*ReviewDraft|TestReviewConfirmations|TestRouteSnapshot|Test.*Dictionary|TestProjectSliceNames|TestLegacyMaterializationWithoutPartScopedOriginalSelection|TestOptionalOriginalSource|TestAnnotationOriginalSource|TestClassificationBatch|TestWorkflowVersionActivation", "-count=1"]
|
||||
]
|
||||
}
|
||||
|
||||
@ -29,3 +29,11 @@ v4 默认关闭不等于旧版本写入安全:新字段开始保存后,旧 A
|
||||
- 完整算子发布 manifest 检查未通过:annotation_gate 的 sourceTreeSha256 漂移在原始 `3221ac65` 工作树逐项复现,期望值/实际值完全相同,属于已有问题;新 v4 package 源码也尚未绑定新算子镜像发布。不得伪造 release.json 摘要使检查通过。它阻断算子升级/v4 启用,不将本轮限定的 API/处理器默认关闭候选冒充为算子可发布版本。
|
||||
|
||||
证据存于本工作树忽略目录 `local-data/verification/combined-v4-20261022/`。本记录不是部署批准,也不保证回退旧写入路径无损。
|
||||
|
||||
## 后续启用准备
|
||||
|
||||
用户随后确认仅新启动的工作流使用 v4,并确认缺 tags 的当前启用词条以词条名称初始化首个标签。新增准备和验收记录位于 `local-data/verification/v4-activation-20261022/`,不覆盖上述默认关闭阶段的证据。
|
||||
|
||||
独立 package 镜像已由 `8f6b531f` 实际构建、核对镜像内源码并上传 Fanyu 仓库,摘要为 `sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d`。仅更新 package 的真实发布记录及生成清单,保留 POC 历史镜像记录。独立发布使用 `OPERATOR_RELEASE_ENV=fanyu python3 scripts/generate_operator_repository_manifests.py --slug package --check`,仍强制验证所选算子的实际源码树和镜像摘要;默认不带选择的全量检查继续保留,不能据此声称全库清单通过。未修改、未发布 annotation_gate,其已有清单漂移仍保留为独立遗留项;本次工作流固定新 package digest,不依赖重写全局算子目录来替换旧任务镜像。
|
||||
|
||||
切换修复保留已启动采集及父子 Run 的原归属,恢复隐藏版本上未启动登记的迁移,保留原 ID,并在未知启动结果时阻断激活。未知启动结果须先停扫描、核查原归属并单独恢复,不能按记录年龄自动重复创建 Run。新 v4 接入前验证完整八类原始文件,包括 meta;布尔开关与打包器、内部审核保存使用一致口径。详情见启用核对表,实际部署仍须具体计划授权。
|
||||
|
||||
@ -3502,6 +3502,43 @@ func (server *server) handleWorkflowActivateVersion(writer http.ResponseWriter,
|
||||
renderError(writer, http.StatusInternalServerError, err)
|
||||
return
|
||||
}
|
||||
// Read every exact-scope version, including hidden ancestors left by an
|
||||
// interrupted activation. A missing Run is an unknown launch outcome, not
|
||||
// proof that the claim is safe to transfer or repeat. Preflight is read-only.
|
||||
familyCaptures := make([]domain.CaptureRecord, 0)
|
||||
for _, version := range versions {
|
||||
if version.OrganizationID != target.OrganizationID || version.ProjectID != target.ProjectID {
|
||||
continue
|
||||
}
|
||||
records, listErr := server.store.ListCaptureRecords(request.Context(), version.OrganizationID, version.ID, 0)
|
||||
if listErr != nil {
|
||||
renderError(writer, http.StatusInternalServerError, listErr)
|
||||
return
|
||||
}
|
||||
for _, record := range records {
|
||||
if record.OrganizationID != target.OrganizationID || record.ProjectID != target.ProjectID || record.WorkflowID != version.ID {
|
||||
continue
|
||||
}
|
||||
if record.Status == domain.CaptureRecordLaunching && strings.TrimSpace(record.RunID) == "" {
|
||||
run, lookupErr := server.store.GetRunByWorkflowTaskID(request.Context(), version.ID, record.ID)
|
||||
if lookupErr != nil {
|
||||
renderError(writer, http.StatusInternalServerError, lookupErr)
|
||||
return
|
||||
}
|
||||
if run == nil {
|
||||
renderJSON(writer, http.StatusConflict, map[string]string{
|
||||
"code": "capture_launch_unresolved",
|
||||
"error": "Capture launch outcome is unresolved; quiesce Workers and recover the claim under its original workflow before activation",
|
||||
"captureId": record.ID, "workflowId": version.ID,
|
||||
})
|
||||
return
|
||||
}
|
||||
// The listener will adopt the observable Run under this original
|
||||
// owner. Do not rewrite lineage or create another Run here.
|
||||
}
|
||||
familyCaptures = append(familyCaptures, record)
|
||||
}
|
||||
}
|
||||
now := nowISO()
|
||||
updated := make([]domain.WorkflowRecord, 0, len(versions))
|
||||
rollbackWorkflows := func() {
|
||||
@ -3509,12 +3546,11 @@ func (server *server) handleWorkflowActivateVersion(writer http.ResponseWriter,
|
||||
_ = server.store.UpsertWorkflow(request.Context(), updated[index])
|
||||
}
|
||||
}
|
||||
// Close every currently visible sibling before moving the Capture registry.
|
||||
// This creates a short fail-closed interval in which no listener can scan the
|
||||
// prefix, instead of exposing the new listener before its idempotency state is
|
||||
// ready.
|
||||
// Activation requires a quiesced capture scan window: these writes are not
|
||||
// atomic with Worker admission. Hide siblings before transferring only their
|
||||
// never-started captures; started records retain parent/child Run ownership.
|
||||
for _, item := range versions {
|
||||
if item.ID == target.ID || item.Hidden {
|
||||
if item.ID == target.ID || item.Hidden || item.OrganizationID != target.OrganizationID || item.ProjectID != target.ProjectID {
|
||||
continue
|
||||
}
|
||||
previous := item
|
||||
@ -3537,30 +3573,21 @@ func (server *server) handleWorkflowActivateVersion(writer http.ResponseWriter,
|
||||
_ = server.store.UpsertCaptureRecord(request.Context(), movedCaptures[index].previous)
|
||||
}
|
||||
}
|
||||
for _, previousWorkflow := range updated {
|
||||
if previousWorkflow.CaptureAutomation == nil || !previousWorkflow.CaptureAutomation.Enabled {
|
||||
for _, record := range familyCaptures {
|
||||
if record.WorkflowID == target.ID || !domain.CaptureAwaitingFirstRun(record) {
|
||||
continue
|
||||
}
|
||||
records, listErr := server.store.ListCaptureRecords(request.Context(), previousWorkflow.OrganizationID, previousWorkflow.ID, 100000)
|
||||
if listErr != nil {
|
||||
previous := record
|
||||
record.WorkflowID = target.ID
|
||||
record.ProjectID = target.ProjectID
|
||||
record.UpdatedAt = now
|
||||
if upsertErr := server.store.UpsertCaptureRecord(request.Context(), record); upsertErr != nil {
|
||||
rollbackCaptures()
|
||||
rollbackWorkflows()
|
||||
renderError(writer, http.StatusInternalServerError, listErr)
|
||||
renderError(writer, http.StatusInternalServerError, upsertErr)
|
||||
return
|
||||
}
|
||||
for _, record := range records {
|
||||
previous := record
|
||||
record.WorkflowID = target.ID
|
||||
record.ProjectID = target.ProjectID
|
||||
record.UpdatedAt = now
|
||||
if upsertErr := server.store.UpsertCaptureRecord(request.Context(), record); upsertErr != nil {
|
||||
rollbackCaptures()
|
||||
rollbackWorkflows()
|
||||
renderError(writer, http.StatusInternalServerError, upsertErr)
|
||||
return
|
||||
}
|
||||
movedCaptures = append(movedCaptures, movedCaptureRecord{previous: previous})
|
||||
}
|
||||
movedCaptures = append(movedCaptures, movedCaptureRecord{previous: previous})
|
||||
}
|
||||
|
||||
previousTarget := *target
|
||||
|
||||
@ -11538,7 +11538,7 @@ func TestWorkflowVersionActivationHidesSiblingsAndKeepsHistoryReadable(t *testin
|
||||
versionOne, err := dataStore.CreateWorkflow(ctx, domain.CreateWorkflowInput{
|
||||
ProjectID: project.ID,
|
||||
Name: "双目交付-r1",
|
||||
Type: domain.WorkflowTypeBatch,
|
||||
Type: domain.WorkflowTypeStreaming,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("create workflow: %v", err)
|
||||
@ -11625,12 +11625,12 @@ func TestWorkflowVersionActivationHidesSiblingsAndKeepsHistoryReadable(t *testin
|
||||
if err != nil || migrated == nil {
|
||||
t.Fatalf("get migrated Capture registry: %#v err=%v", migrated, err)
|
||||
}
|
||||
if migrated.WorkflowID != versionThree.ID || migrated.ID != capture.ID || migrated.RunID != capture.RunID || migrated.Fingerprint != capture.Fingerprint {
|
||||
t.Fatalf("Capture registry must move without changing identity or lineage: %#v", migrated)
|
||||
if migrated.WorkflowID != versionOne.ID || migrated.ID != capture.ID || migrated.RunID != capture.RunID || migrated.Fingerprint != capture.Fingerprint {
|
||||
t.Fatalf("started Capture must retain its original workflow and lineage: %#v", migrated)
|
||||
}
|
||||
oldRecords, err := dataStore.ListCaptureRecords(ctx, versionOne.OrganizationID, versionOne.ID, 100)
|
||||
if err != nil || len(oldRecords) != 0 {
|
||||
t.Fatalf("old listener must not retain duplicate Capture registry rows: %#v err=%v", oldRecords, err)
|
||||
if err != nil || len(oldRecords) != 1 {
|
||||
t.Fatalf("old workflow must retain started Capture ownership: %#v err=%v", oldRecords, err)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
15
internal/domain/capture_activation.go
Normal file
15
internal/domain/capture_activation.go
Normal file
@ -0,0 +1,15 @@
|
||||
package domain
|
||||
|
||||
// CaptureAwaitingFirstRun excludes claims in progress and every historical
|
||||
// attempt. Only these records may change workflow ownership at activation.
|
||||
func CaptureAwaitingFirstRun(record CaptureRecord) bool {
|
||||
if record.RunID != "" {
|
||||
return false
|
||||
}
|
||||
switch record.Status {
|
||||
case CaptureRecordIncomplete, CaptureRecordStabilizing, CaptureRecordReady:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
@ -371,19 +371,25 @@ func workflowInputDefinition(workflow domain.WorkflowRecord, inputID string) *do
|
||||
}
|
||||
|
||||
func (runtime *Runtime) reconcileCaptureRun(ctx context.Context, record *domain.CaptureRecord) error {
|
||||
if record == nil || strings.TrimSpace(record.RunID) == "" {
|
||||
if record == nil || (strings.TrimSpace(record.RunID) == "" && record.Status != domain.CaptureRecordLaunching) {
|
||||
return nil
|
||||
}
|
||||
run, err := runtime.store.GetRunByWorkflowTaskID(ctx, record.WorkflowID, record.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if run == nil {
|
||||
if run == nil && record.RunID != "" {
|
||||
run, err = runtime.store.GetRun(ctx, record.RunID)
|
||||
}
|
||||
if err != nil || run == nil {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if run == nil {
|
||||
if record.Status == domain.CaptureRecordLaunching && strings.TrimSpace(record.RunID) == "" {
|
||||
return fmt.Errorf("capture_launch_unresolved: Capture %s under workflow %s has no observable Run; quiesce Workers and recover this claim under its original workflow before activation; do not reset or repeat admission", record.ID, record.WorkflowID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
record.RunID = run.ID
|
||||
if !captureCancelledByUser(record) {
|
||||
record.Status = runStatusForCapture(run)
|
||||
@ -422,7 +428,7 @@ func (runtime *Runtime) launchCaptureRun(ctx context.Context, workflow domain.Wo
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if record == nil || record.Status != domain.CaptureRecordReady {
|
||||
if record == nil || record.Status != domain.CaptureRecordReady || record.WorkflowID != workflow.ID || record.OrganizationID != workflow.OrganizationID || record.ProjectID != workflow.ProjectID {
|
||||
return nil
|
||||
}
|
||||
existingRun, err := runtime.store.GetRunByWorkflowTaskID(ctx, workflow.ID, record.ID)
|
||||
@ -441,16 +447,20 @@ func (runtime *Runtime) launchCaptureRun(ctx context.Context, workflow domain.Wo
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if latest == nil || latest.Type != domain.WorkflowTypeStreaming || latest.CaptureAutomation == nil || !latest.CaptureAutomation.Enabled {
|
||||
if latest == nil || latest.Hidden || latest.DefinitionState != domain.WorkflowDefinitionStateReleased || latest.Type != domain.WorkflowTypeStreaming || latest.CaptureAutomation == nil || !latest.CaptureAutomation.Enabled {
|
||||
return nil
|
||||
}
|
||||
if captureAutomationFingerprint(workflow.ID, normalizedCaptureAutomation(latest.CaptureAutomation)) != captureAutomationFingerprint(workflow.ID, config) {
|
||||
return nil
|
||||
}
|
||||
if err := validateV4CaptureSourceSelection(workflow, record); err != nil {
|
||||
return err
|
||||
}
|
||||
claimed, err := runtime.store.ClaimCaptureRecordForLaunch(ctx, record.ID, record.Fingerprint)
|
||||
if err != nil || !claimed {
|
||||
return err
|
||||
}
|
||||
record.Status = domain.CaptureRecordLaunching
|
||||
input := workflowInputDefinition(workflow, config.WorkflowInputID)
|
||||
if input == nil {
|
||||
record.Status = domain.CaptureRecordFailed
|
||||
@ -487,11 +497,13 @@ func (runtime *Runtime) launchCaptureRun(ctx context.Context, workflow domain.Wo
|
||||
}
|
||||
run, err := runtime.store.CreateRunWithInputs(ctx, workflow.ID, domain.RunTriggerScheduled, map[string]domain.ArtifactPayload{config.WorkflowInputID: payload}, record.ID)
|
||||
if err != nil {
|
||||
record.Status = domain.CaptureRecordReady
|
||||
record.Error = err.Error()
|
||||
record.UpdatedAt = nowISO()
|
||||
_ = runtime.store.UpsertCaptureRecord(ctx, *record)
|
||||
return err
|
||||
// CreateRun may have committed before its response failed. Keep the
|
||||
// persisted launch claim and adopt an observable Run; never reopen
|
||||
// admission on an unknown result or infer failure from claim age.
|
||||
if readbackErr := runtime.reconcileCaptureRun(ctx, record); readbackErr != nil {
|
||||
return fmt.Errorf("create Capture Run failed (%v): %w", err, readbackErr)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
record.RunID = run.ID
|
||||
record.Status = domain.CaptureRecordQueued
|
||||
@ -514,6 +526,55 @@ func (runtime *Runtime) captureAutomationDue(ctx context.Context, workflow domai
|
||||
return !ok || now.Sub(last) >= time.Duration(config.PollIntervalSeconds)*time.Second, nil
|
||||
}
|
||||
|
||||
// Intake moves between versions, but records with Runs remain with the original
|
||||
// workflow. Read only this project's exact tenant/family history for deduplication
|
||||
// and reconcile hidden in-flight owners without migrating their lineage.
|
||||
func (runtime *Runtime) captureAutomationFamilyRecords(ctx context.Context, workflow domain.WorkflowRecord) ([]domain.CaptureRecord, error) {
|
||||
domain.NormalizeWorkflowDefinitionMetadata(&workflow)
|
||||
versions, err := runtime.store.ListWorkflowsByProject(ctx, workflow.ProjectID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var records []domain.CaptureRecord
|
||||
for _, version := range versions {
|
||||
domain.NormalizeWorkflowDefinitionMetadata(&version)
|
||||
if version.OrganizationID != workflow.OrganizationID || version.ProjectID != workflow.ProjectID || version.DefinitionFamilyID != workflow.DefinitionFamilyID || version.DefinitionState != domain.WorkflowDefinitionStateReleased {
|
||||
continue
|
||||
}
|
||||
items, err := runtime.store.ListCaptureRecords(ctx, version.OrganizationID, version.ID, 0)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, item := range items {
|
||||
if item.OrganizationID != workflow.OrganizationID || item.ProjectID != workflow.ProjectID || item.WorkflowID != version.ID {
|
||||
continue
|
||||
}
|
||||
// Reconcile claims for the current listener as well as hidden owners.
|
||||
// Missing Runs surface an actionable error without unfenced retries.
|
||||
if item.Status == domain.CaptureRecordLaunching {
|
||||
if err := runtime.reconcileCaptureRun(ctx, &item); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
} else if version.ID != workflow.ID {
|
||||
switch item.Status {
|
||||
case domain.CaptureRecordQueued, domain.CaptureRecordRunning, domain.CaptureRecordWaiting:
|
||||
if err := runtime.reconcileCaptureRun(ctx, &item); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
case domain.CaptureRecordRequeueing:
|
||||
if recoverStaleCaptureRequeue(&item, time.Now().UTC()) {
|
||||
if err := runtime.store.UpsertCaptureRecord(ctx, item); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
records = append(records, item)
|
||||
}
|
||||
}
|
||||
return records, nil
|
||||
}
|
||||
|
||||
func (runtime *Runtime) processCaptureAutomation(ctx context.Context, workflow domain.WorkflowRecord) error {
|
||||
if workflow.Type != domain.WorkflowTypeStreaming {
|
||||
return nil
|
||||
@ -549,16 +610,35 @@ func (runtime *Runtime) processCaptureAutomation(ctx context.Context, workflow d
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
groups, err := discoverCaptureGroups(objects, config)
|
||||
groups, err := discoverCaptureGroupsForWorkflow(objects, config, workflow)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
priorRecords, err := runtime.store.ListCaptureRecords(ctx, workflow.OrganizationID, workflow.ID, 0)
|
||||
// Listing can outlive activation or an intake pause. Do not publish records
|
||||
// from a captured old listener after the handoff. Deployment still quiesces
|
||||
// scans because this check and the following store writes are not a transaction.
|
||||
latest, err := runtime.store.GetWorkflow(ctx, workflow.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if latest == nil || latest.Hidden || latest.DefinitionState != domain.WorkflowDefinitionStateReleased || latest.CaptureAutomation == nil || !latest.CaptureAutomation.Enabled || captureAutomationFingerprint(workflow.ID, normalizedCaptureAutomation(latest.CaptureAutomation)) != fingerprint {
|
||||
return nil
|
||||
}
|
||||
priorRecords, err := runtime.captureAutomationFamilyRecords(ctx, workflow)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
byPrefix := map[string]*domain.CaptureRecord{}
|
||||
admittedPartParents := map[string]bool{}
|
||||
for _, prior := range priorRecords {
|
||||
if prior.ConnectionID != config.ConnectionID || prior.Bucket != config.Bucket {
|
||||
continue
|
||||
}
|
||||
if previous := byPrefix[prior.Prefix]; previous != nil && previous.ID != prior.ID {
|
||||
return fmt.Errorf("capture workflow family has conflicting records for prefix %s", prior.Prefix)
|
||||
}
|
||||
copy := prior
|
||||
byPrefix[prior.Prefix] = ©
|
||||
if prior.SourcePrefix != "" && prior.SourcePrefix != prior.Prefix && preserveCaptureParent(&prior) {
|
||||
admittedPartParents[prior.SourcePrefix] = true
|
||||
}
|
||||
@ -570,12 +650,9 @@ func (runtime *Runtime) processCaptureAutomation(ctx context.Context, workflow d
|
||||
group.Missing = append(group.Missing, "该目录已有 part 接入记录,不能重新作为整目录接入")
|
||||
}
|
||||
if group.SourcePrefix != "" && group.SourcePrefix != group.Prefix {
|
||||
parent, e := runtime.store.GetCaptureRecordByWorkflowPrefix(ctx, workflow.ID, group.SourcePrefix)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
parent := byPrefix[group.SourcePrefix]
|
||||
if preserveCaptureParent(parent) {
|
||||
if !historicalParents[parent.ID] && parent.RunID != "" {
|
||||
if !historicalParents[parent.ID] && parent.RunID != "" && parent.WorkflowID == workflow.ID {
|
||||
if e := runtime.reconcileCaptureRun(ctx, parent); e != nil {
|
||||
return e
|
||||
}
|
||||
@ -584,9 +661,11 @@ func (runtime *Runtime) processCaptureAutomation(ctx context.Context, workflow d
|
||||
continue
|
||||
}
|
||||
}
|
||||
existing, err := runtime.store.GetCaptureRecordByWorkflowPrefix(ctx, workflow.ID, group.Prefix)
|
||||
if err != nil {
|
||||
return err
|
||||
existing := byPrefix[group.Prefix]
|
||||
if existing != nil && existing.WorkflowID != workflow.ID {
|
||||
// Historical admissions (including a launch claim with no Run yet)
|
||||
// never switch protocols. Unstarted transfers belong to activation.
|
||||
continue
|
||||
}
|
||||
if existing != nil && existing.Status == domain.CaptureRecordDiscarded {
|
||||
continue
|
||||
@ -613,6 +692,9 @@ func (runtime *Runtime) processCaptureAutomation(ctx context.Context, workflow d
|
||||
}
|
||||
continue
|
||||
}
|
||||
if existing != nil && !domain.CaptureAwaitingFirstRun(*existing) {
|
||||
continue
|
||||
}
|
||||
missing := append([]string(nil), group.Missing...)
|
||||
nowText := nowISO()
|
||||
record := domain.CaptureRecord{
|
||||
@ -625,6 +707,7 @@ func (runtime *Runtime) processCaptureAutomation(ctx context.Context, workflow d
|
||||
ContractRevision: domain.FanyuV58CaptureContractRevision,
|
||||
}
|
||||
if existing != nil {
|
||||
record.ID = existing.ID
|
||||
record.Delivery = existing.Delivery
|
||||
record.DiscoveredAt = existing.DiscoveredAt
|
||||
record.Actions = append([]domain.CaptureActionRecord(nil), existing.Actions...)
|
||||
|
||||
538
internal/worker/capture_cutover_test.go
Normal file
538
internal/worker/capture_cutover_test.go
Normal file
@ -0,0 +1,538 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"droboticflow/internal/api"
|
||||
"droboticflow/internal/config"
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/store"
|
||||
)
|
||||
|
||||
type captureCutoverFixture struct {
|
||||
t *testing.T
|
||||
ctx context.Context
|
||||
db *store.MemoryStore
|
||||
runtime *Runtime
|
||||
old, next domain.WorkflowRecord
|
||||
objects []captureObject
|
||||
beforeList func()
|
||||
}
|
||||
|
||||
func newCaptureCutoverFixture(t *testing.T, names ...string) *captureCutoverFixture {
|
||||
t.Helper()
|
||||
f := &captureCutoverFixture{t: t, ctx: context.Background(), db: store.NewMemoryStore()}
|
||||
for _, name := range names {
|
||||
for _, object := range append(calibrationObjects(), partObjects("part0001")...) {
|
||||
object.Key = strings.ReplaceAll(object.Key, layoutCapture, name)
|
||||
f.objects = append(f.objects, object)
|
||||
}
|
||||
f.objects = append(f.objects, captureObject{Key: "incoming/" + name + "/meta_" + name + ".json", Size: 10, ETag: "metadata"})
|
||||
}
|
||||
bos := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if f.beforeList != nil {
|
||||
f.beforeList()
|
||||
}
|
||||
contents := []map[string]any{}
|
||||
for _, object := range f.objects {
|
||||
contents = append(contents, map[string]any{"key": object.Key, "eTag": object.ETag, "size": object.Size})
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"contents": contents, "isTruncated": false})
|
||||
}))
|
||||
t.Cleanup(bos.Close)
|
||||
project, err := f.db.CreateProject(f.ctx, domain.CreateProjectInput{OrganizationID: domain.DefaultOrganizationID, Name: "cutover"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.db.UpsertStorageConnection(f.ctx, domain.StorageConnectionRecord{ID: "cutover-storage", OrganizationID: project.OrganizationID, Provider: domain.StorageProviderBOS, Bucket: "synthetic", Endpoint: bos.URL, AccessKeyID: "synthetic", SecretAccessKey: "synthetic"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
config := &domain.CaptureAutomationConfig{Enabled: true, ConnectionID: "cutover-storage", Bucket: "synthetic", Prefix: "incoming", WorkflowInputID: "sourceDirectory", DeliveryPrefix: "deliveries", DeliveryProjectDirectory: "cutover__project", StabilityWindowSeconds: 1}
|
||||
f.old = domain.WorkflowRecord{ID: "cutover-old", OrganizationID: project.OrganizationID, ProjectID: project.ID, Type: domain.WorkflowTypeStreaming, ExecutionScope: domain.WorkflowExecutionScopeProduction, DefinitionState: domain.WorkflowDefinitionStateReleased, DefinitionFamilyID: "shared-template-family", DefinitionVersion: 1, CaptureAutomation: config, WorkflowInputs: []domain.WorkflowInputDefinition{{ID: "sourceDirectory", Name: "sourceDirectory", Type: domain.PortTypeFile, Container: domain.PortContainerDirectory, PayloadMode: domain.ArtifactTransferModeRef, Required: true}}, Nodes: []domain.WorkflowNode{{ID: "package", Data: domain.WorkflowNodeData{OperatorID: "operator-builtin-package", Image: "registry.example/package:legacy", OperatorImageID: "registry.example/package@sha256:" + strings.Repeat("a", 64), Hyperparams: []domain.HyperParam{{Key: "DELIVERY_JSON_V4", Value: "false"}, {Key: "PIN_OPERATOR_IMAGE", Value: "true"}}}}}}
|
||||
f.next = f.old
|
||||
f.next.ID, f.next.Hidden, f.next.DefinitionVersion, f.next.SupersedesWorkflowID = "cutover-new", true, 2, f.old.ID
|
||||
f.next.Nodes = []domain.WorkflowNode{{ID: "package", Data: domain.WorkflowNodeData{OperatorID: "operator-builtin-package", Image: "registry.example/package:v4", OperatorImageID: "registry.example/package@sha256:" + strings.Repeat("b", 64), Hyperparams: []domain.HyperParam{{Key: "DELIVERY_JSON_V4", Value: "true"}, {Key: "PIN_OPERATOR_IMAGE", Value: "true"}}}}}
|
||||
for _, workflow := range []domain.WorkflowRecord{f.old, f.next} {
|
||||
if err := f.db.UpsertWorkflow(f.ctx, workflow); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := f.db.UpsertOperator(f.ctx, domain.OperatorRecord{ID: "operator-builtin-package", Type: "builtin", Image: "registry.example/package:catalog", ImageID: "registry.example/package@sha256:" + strings.Repeat("c", 64)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.runtime = &Runtime{store: f.db}
|
||||
return f
|
||||
}
|
||||
|
||||
func (f *captureCutoverFixture) seed(name string, status domain.CaptureRecordStatus, runID string) domain.CaptureRecord {
|
||||
f.t.Helper()
|
||||
groups, err := discoverCaptureGroups(f.objects, normalizedCaptureAutomation(f.old.CaptureAutomation))
|
||||
if err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
var selected captureGroup
|
||||
for _, group := range groups {
|
||||
if group.CaptureID == name+"/part0001" {
|
||||
selected = group
|
||||
}
|
||||
}
|
||||
if selected.Prefix == "" || len(selected.Missing) != 0 {
|
||||
f.t.Fatalf("invalid cutover source fixture: %+v", selected)
|
||||
}
|
||||
c := domain.CaptureRecord{ID: captureRecordID(f.old.ID, selected.Prefix), OrganizationID: f.old.OrganizationID, ProjectID: f.old.ProjectID, WorkflowID: f.old.ID, ConnectionID: "cutover-storage", Bucket: "synthetic", CaptureID: selected.CaptureID, Prefix: selected.Prefix, SourcePrefix: selected.SourcePrefix, SourceKeys: selected.SourceKeys, Fingerprint: selected.Fingerprint, Status: status, RunID: runID, StabilityChecks: 1, StabilityObservedAt: time.Now().Add(-time.Minute).UTC().Format(time.RFC3339Nano), DiscoveredAt: "2026-01-01T00:00:00Z", UpdatedAt: "2026-01-01T00:00:00Z", Delivery: &domain.CaptureDeliveryTarget{ConnectionID: "cutover-storage", Bucket: "synthetic", Prefix: "deliveries/cutover__project/" + selected.CaptureID}}
|
||||
if err := f.db.UpsertCaptureRecord(f.ctx, c); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
saved, err := f.db.GetCaptureRecord(f.ctx, c.ID)
|
||||
if err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
return *saved
|
||||
}
|
||||
|
||||
func (f *captureCutoverFixture) activate() {
|
||||
f.t.Helper()
|
||||
w := httptest.NewRecorder()
|
||||
api.NewHandlerWithConfig(f.db, nil, config.AppConfig{}).ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/api/workflows/"+f.next.ID+"/activate", nil))
|
||||
if w.Code != http.StatusOK {
|
||||
f.t.Fatalf("activation: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func (f *captureCutoverFixture) scan() {
|
||||
f.t.Helper()
|
||||
if err := f.db.DeleteSourceStatesByWorkflow(f.ctx, f.next.ID); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
w, err := f.db.GetWorkflow(f.ctx, f.next.ID)
|
||||
if err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
if err := f.runtime.processCaptureAutomation(f.ctx, *w); err != nil {
|
||||
f.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCaptureCutoverPreservesStartedOwnershipAndParentChildDelivery(t *testing.T) {
|
||||
f := newCaptureCutoverFixture(t, "active", "done", "launching", "fresh")
|
||||
active := f.seed("active", domain.CaptureRecordWaiting, "old-parent")
|
||||
done := f.seed("done", domain.CaptureRecordCompleted, "delivered-parent")
|
||||
launching := f.seed("launching", domain.CaptureRecordLaunching, "old-claimed")
|
||||
fresh := f.seed("fresh", domain.CaptureRecordStabilizing, "")
|
||||
parent := domain.RunRecord{ID: active.RunID, OrganizationID: active.OrganizationID, ProjectID: active.ProjectID, WorkflowID: f.old.ID, TaskID: active.ID, Status: domain.RunStatusWaitingForHuman, CreatedAt: "2026-01-01T00:00:00Z"}
|
||||
child := parent
|
||||
child.ID, child.ParentRunID, child.Trigger, child.Status, child.CreatedAt = "old-child", parent.ID, domain.RunTriggerHumanRelease, domain.RunStatusRunning, "2026-01-01T00:00:01Z"
|
||||
for _, run := range []domain.RunRecord{parent, child, {ID: launching.RunID, OrganizationID: launching.OrganizationID, ProjectID: launching.ProjectID, WorkflowID: f.old.ID, TaskID: launching.ID, Status: domain.RunStatusQueued}} {
|
||||
if err := f.db.UpsertRun(f.ctx, run); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
f.activate()
|
||||
for _, before := range []domain.CaptureRecord{active, done, launching} {
|
||||
after, _ := f.db.GetCaptureRecord(f.ctx, before.ID)
|
||||
if after == nil || after.WorkflowID != f.old.ID || after.RunID != before.RunID {
|
||||
t.Fatalf("started Capture moved: before=%+v after=%+v", before, after)
|
||||
}
|
||||
}
|
||||
lookup, err := f.db.GetCaptureRecordByRunID(f.ctx, child.ID)
|
||||
if err != nil || lookup == nil || lookup.ID != active.ID {
|
||||
t.Fatalf("old child lost Capture: %+v %v", lookup, err)
|
||||
}
|
||||
publisher := &deliveryPublisher{}
|
||||
f.runtime.artifactPublisher = publisher
|
||||
root := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(root, "legacy.json"), []byte("legacy bytes"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.runtime.publishCaptureDelivery(f.ctx, &child, "package", domain.ExecutionArtifact{Name: "delivery.directory.index", Type: domain.PortTypeFile, Path: root}); err != nil {
|
||||
t.Fatalf("old child cannot publish: %v", err)
|
||||
}
|
||||
child.Status = domain.RunStatusCompleted
|
||||
if err := f.db.UpsertRun(f.ctx, child); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.scan()
|
||||
oldAfter, _ := f.db.GetCaptureRecord(f.ctx, active.ID)
|
||||
if oldAfter.WorkflowID != f.old.ID || oldAfter.RunID != child.ID || oldAfter.Status != domain.CaptureRecordCompleted {
|
||||
t.Fatalf("hidden lineage did not reconcile: %+v", oldAfter)
|
||||
}
|
||||
if lookup, err = f.db.GetCaptureRecordByRunID(f.ctx, parent.ID); err != nil || lookup == nil || lookup.ID != active.ID {
|
||||
t.Fatalf("old parent lost child-owned Capture: %+v %v", lookup, err)
|
||||
}
|
||||
f.scan()
|
||||
newCapture, _ := f.db.GetCaptureRecord(f.ctx, fresh.ID)
|
||||
if newCapture == nil || newCapture.WorkflowID != f.next.ID || newCapture.RunID == "" {
|
||||
t.Fatalf("unstarted Capture not admitted: %+v", newCapture)
|
||||
}
|
||||
newRun, _ := f.db.GetRun(f.ctx, newCapture.RunID)
|
||||
if newRun.WorkflowID != f.next.ID || newRun.TaskID != fresh.ID {
|
||||
t.Fatalf("wrong new Run binding: %+v", newRun)
|
||||
}
|
||||
oldMode, err := f.runtime.classificationDeliveryEnabled(f.ctx, child.WorkflowID)
|
||||
if err != nil || oldMode {
|
||||
t.Fatalf("old child selected v4: %v %v", oldMode, err)
|
||||
}
|
||||
newMode, err := f.runtime.classificationDeliveryEnabled(f.ctx, newRun.WorkflowID)
|
||||
if err != nil || !newMode {
|
||||
t.Fatalf("new parent did not select v4: %v %v", newMode, err)
|
||||
}
|
||||
for _, run := range []domain.RunRecord{parent, child, *newRun} {
|
||||
workflow, err := f.db.GetWorkflow(f.ctx, run.WorkflowID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resolved, err := f.runtime.resolveBuiltinNodeRuntime(f.ctx, workflow.Nodes[0])
|
||||
want := f.old.Nodes[0]
|
||||
if run.WorkflowID == f.next.ID {
|
||||
want = f.next.Nodes[0]
|
||||
}
|
||||
if err != nil || resolved.Data.Image != want.Data.Image || resolved.Data.OperatorImageID != want.Data.OperatorImageID {
|
||||
t.Fatalf("Run %s lost pinned original runtime: got=%+v want=%+v err=%v", run.ID, resolved.Data, want.Data, err)
|
||||
}
|
||||
}
|
||||
completed, _ := f.db.GetCaptureRecord(f.ctx, done.ID)
|
||||
if !reflect.DeepEqual(*completed, done) {
|
||||
t.Fatalf("delivered Capture changed: before=%+v after=%+v", done, completed)
|
||||
}
|
||||
all, _ := f.db.ListCaptureRecords(f.ctx, f.old.OrganizationID, "", 0)
|
||||
if len(all) != 4 {
|
||||
t.Fatalf("Capture re-ingestion or ID collision: %+v", all)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCaptureCutoverMigratedUnstartedKeepsIdentity(t *testing.T) {
|
||||
f := newCaptureCutoverFixture(t, "fresh")
|
||||
before := f.seed("fresh", domain.CaptureRecordStabilizing, "")
|
||||
f.activate()
|
||||
f.scan()
|
||||
f.scan()
|
||||
after, _ := f.db.GetCaptureRecord(f.ctx, before.ID)
|
||||
all, _ := f.db.ListCaptureRecords(f.ctx, f.next.OrganizationID, f.next.ID, 0)
|
||||
if after == nil || after.RunID == "" || len(all) != 1 || all[0].ID != before.ID {
|
||||
t.Fatalf("migrated identity lost/duplicated: %+v %+v", after, all)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCaptureCutoverDoesNotDeduplicateOtherScopes(t *testing.T) {
|
||||
for _, scope := range []string{"project", "organization", "family"} {
|
||||
t.Run(scope, func(t *testing.T) {
|
||||
f := newCaptureCutoverFixture(t, "fresh")
|
||||
foreign := f.old
|
||||
foreign.ID = "foreign-workflow"
|
||||
switch scope {
|
||||
case "project":
|
||||
foreign.ProjectID = "other-project"
|
||||
case "organization":
|
||||
foreign.OrganizationID = "other-org"
|
||||
case "family":
|
||||
foreign.DefinitionFamilyID = "other-family"
|
||||
}
|
||||
if err := f.db.UpsertWorkflow(f.ctx, foreign); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
original := f.seed("fresh", domain.CaptureRecordStabilizing, "")
|
||||
other := original
|
||||
other.ID, other.WorkflowID, other.OrganizationID, other.ProjectID, other.RunID, other.Status = "foreign-capture", foreign.ID, foreign.OrganizationID, foreign.ProjectID, "foreign-run", domain.CaptureRecordCompleted
|
||||
if err := f.db.UpsertCaptureRecord(f.ctx, other); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.activate()
|
||||
f.scan()
|
||||
got, _ := f.db.GetCaptureRecord(f.ctx, original.ID)
|
||||
untouched, _ := f.db.GetCaptureRecord(f.ctx, other.ID)
|
||||
if got.RunID == "" || got.WorkflowID != f.next.ID || untouched.WorkflowID != foreign.ID || untouched.RunID != other.RunID {
|
||||
t.Fatalf("scope crossed: own=%+v foreign=%+v", got, untouched)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCaptureCutoverStopsScanCapturedBeforeActivation(t *testing.T) {
|
||||
f := newCaptureCutoverFixture(t, "fresh")
|
||||
before := f.seed("fresh", domain.CaptureRecordStabilizing, "")
|
||||
f.beforeList = f.activate
|
||||
if err := f.runtime.processCaptureAutomation(f.ctx, f.old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
after, _ := f.db.GetCaptureRecord(f.ctx, before.ID)
|
||||
if after.WorkflowID != f.next.ID || after.RunID != "" {
|
||||
t.Fatalf("stale old scan wrote/admitted after activation: %+v", after)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCaptureCutoverReadsAllHiddenAncestorsAndReconcilesMissingSources(t *testing.T) {
|
||||
f := newCaptureCutoverFixture(t, "ancient", "missing", "claim")
|
||||
ancestor := f.old
|
||||
ancestor.ID, ancestor.Hidden = "hidden-ancestor", true
|
||||
f.old.SupersedesWorkflowID, f.old.DefinitionVersion = ancestor.ID, 2
|
||||
f.next.DefinitionVersion = 3
|
||||
for _, workflow := range []domain.WorkflowRecord{ancestor, f.old, f.next} {
|
||||
if err := f.db.UpsertWorkflow(f.ctx, workflow); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
ancient := f.seed("ancient", domain.CaptureRecordCompleted, "delivered-ancestor")
|
||||
missing := f.seed("missing", domain.CaptureRecordWaiting, "ancestor-run")
|
||||
claim := f.seed("claim", domain.CaptureRecordLaunching, "")
|
||||
for _, record := range []*domain.CaptureRecord{&ancient, &missing, &claim} {
|
||||
record.WorkflowID = ancestor.ID
|
||||
if err := f.db.UpsertCaptureRecord(f.ctx, *record); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for _, run := range []domain.RunRecord{
|
||||
{ID: missing.RunID, OrganizationID: ancestor.OrganizationID, ProjectID: ancestor.ProjectID, WorkflowID: ancestor.ID, TaskID: missing.ID, Status: domain.RunStatusCompleted},
|
||||
// The Worker died after CreateRun but before recording its ID on Capture.
|
||||
{ID: "claimed-old-run", OrganizationID: ancestor.OrganizationID, ProjectID: ancestor.ProjectID, WorkflowID: ancestor.ID, TaskID: claim.ID, Status: domain.RunStatusQueued},
|
||||
} {
|
||||
if err := f.db.UpsertRun(f.ctx, run); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
filtered := []captureObject{}
|
||||
for _, object := range f.objects {
|
||||
if !strings.Contains(object.Key, "/missing/") {
|
||||
filtered = append(filtered, object)
|
||||
}
|
||||
}
|
||||
f.objects = filtered
|
||||
f.activate()
|
||||
f.scan()
|
||||
f.scan()
|
||||
completed, _ := f.db.GetCaptureRecord(f.ctx, ancient.ID)
|
||||
if !reflect.DeepEqual(*completed, ancient) {
|
||||
t.Fatalf("historical delivery changed: %+v", completed)
|
||||
}
|
||||
reconciled, _ := f.db.GetCaptureRecord(f.ctx, missing.ID)
|
||||
if reconciled.WorkflowID != ancestor.ID || reconciled.Status != domain.CaptureRecordCompleted {
|
||||
t.Fatalf("hidden ancestor without source was not reconciled: %+v", reconciled)
|
||||
}
|
||||
recovered, _ := f.db.GetCaptureRecord(f.ctx, claim.ID)
|
||||
if recovered.WorkflowID != ancestor.ID || recovered.RunID != "claimed-old-run" || recovered.Status != domain.CaptureRecordQueued {
|
||||
t.Fatalf("interrupted old launch lost its owner: %+v", recovered)
|
||||
}
|
||||
all, _ := f.db.ListCaptureRecords(f.ctx, f.old.OrganizationID, "", 0)
|
||||
if len(all) != 3 {
|
||||
t.Fatalf("hidden ancestor re-ingested: %+v", all)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCaptureCutoverTransfersOnlyNeverStartedStates(t *testing.T) {
|
||||
states := []domain.CaptureRecordStatus{domain.CaptureRecordIncomplete, domain.CaptureRecordStabilizing, domain.CaptureRecordReady, domain.CaptureRecordLaunching, domain.CaptureRecordRequeueing, domain.CaptureRecordQueued, domain.CaptureRecordRunning, domain.CaptureRecordWaiting, domain.CaptureRecordCompleted, domain.CaptureRecordFailed, domain.CaptureRecordCancelled, domain.CaptureRecordDiscarded}
|
||||
names := []string{}
|
||||
for _, state := range states {
|
||||
names = append(names, string(state))
|
||||
}
|
||||
f := newCaptureCutoverFixture(t, names...)
|
||||
before := []domain.CaptureRecord{}
|
||||
for _, state := range states {
|
||||
runID := "old-" + string(state)
|
||||
if state == domain.CaptureRecordIncomplete || state == domain.CaptureRecordStabilizing || state == domain.CaptureRecordReady || state == domain.CaptureRecordRequeueing || state == domain.CaptureRecordDiscarded {
|
||||
runID = ""
|
||||
}
|
||||
before = append(before, f.seed(string(state), state, runID))
|
||||
}
|
||||
f.activate()
|
||||
for _, previous := range before {
|
||||
after, _ := f.db.GetCaptureRecord(f.ctx, previous.ID)
|
||||
move := previous.Status == domain.CaptureRecordIncomplete || previous.Status == domain.CaptureRecordStabilizing || previous.Status == domain.CaptureRecordReady
|
||||
if move {
|
||||
if after.WorkflowID != f.next.ID || after.RunID != "" || after.ID != previous.ID {
|
||||
t.Fatalf("unstarted transfer: %+v", after)
|
||||
}
|
||||
} else if !reflect.DeepEqual(*after, previous) {
|
||||
t.Fatalf("historical/claimed record changed: before=%+v after=%+v", previous, after)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Store faults exercise partial activation and unknown Run creation outcomes.
|
||||
type captureCutoverFaultStore struct {
|
||||
store.Store
|
||||
upsertWorkflow func(context.Context, domain.WorkflowRecord) error
|
||||
getRun func(context.Context, string, string) (*domain.RunRecord, error)
|
||||
createRun func(context.Context, string, domain.RunTrigger, map[string]domain.ArtifactPayload, ...string) (*domain.RunRecord, error)
|
||||
}
|
||||
|
||||
func (s *captureCutoverFaultStore) UpsertWorkflow(ctx context.Context, w domain.WorkflowRecord) error {
|
||||
if s.upsertWorkflow != nil {
|
||||
return s.upsertWorkflow(ctx, w)
|
||||
}
|
||||
return s.Store.UpsertWorkflow(ctx, w)
|
||||
}
|
||||
func (s *captureCutoverFaultStore) GetRunByWorkflowTaskID(ctx context.Context, workflowID, taskID string) (*domain.RunRecord, error) {
|
||||
if s.getRun != nil {
|
||||
return s.getRun(ctx, workflowID, taskID)
|
||||
}
|
||||
return s.Store.GetRunByWorkflowTaskID(ctx, workflowID, taskID)
|
||||
}
|
||||
func (s *captureCutoverFaultStore) CreateRunWithInputs(ctx context.Context, workflowID string, trigger domain.RunTrigger, inputs map[string]domain.ArtifactPayload, taskIDs ...string) (*domain.RunRecord, error) {
|
||||
if s.createRun != nil {
|
||||
return s.createRun(ctx, workflowID, trigger, inputs, taskIDs...)
|
||||
}
|
||||
return s.Store.CreateRunWithInputs(ctx, workflowID, trigger, inputs, taskIDs...)
|
||||
}
|
||||
|
||||
func (f *captureCutoverFixture) activateResponse() *httptest.ResponseRecorder {
|
||||
w := httptest.NewRecorder()
|
||||
api.NewHandlerWithConfig(f.runtime.store, nil, config.AppConfig{}).ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/api/workflows/"+f.next.ID+"/activate", nil))
|
||||
return w
|
||||
}
|
||||
|
||||
func TestCaptureCutoverRetryTransfersHiddenUnstartedAndRollsBackOnlyOwnWrites(t *testing.T) {
|
||||
f := newCaptureCutoverFixture(t, "stranded", "already-moved")
|
||||
stranded := f.seed("stranded", domain.CaptureRecordReady, "")
|
||||
already := f.seed("already-moved", domain.CaptureRecordReady, "")
|
||||
already.WorkflowID = f.next.ID
|
||||
if err := f.db.UpsertCaptureRecord(f.ctx, already); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Persisted state after an earlier process died after hiding the old owner.
|
||||
f.old.Hidden = true
|
||||
if err := f.db.UpsertWorkflow(f.ctx, f.old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fault := &captureCutoverFaultStore{Store: f.db}
|
||||
fault.upsertWorkflow = func(ctx context.Context, w domain.WorkflowRecord) error {
|
||||
if w.ID == f.next.ID && !w.Hidden {
|
||||
return fmt.Errorf("injected activation write failure")
|
||||
}
|
||||
return f.db.UpsertWorkflow(ctx, w)
|
||||
}
|
||||
f.runtime.store = fault
|
||||
if got := f.activateResponse(); got.Code != http.StatusInternalServerError {
|
||||
t.Fatalf("expected injected failure: %d %s", got.Code, got.Body.String())
|
||||
}
|
||||
old, _ := f.db.GetWorkflow(f.ctx, f.old.ID)
|
||||
got, _ := f.db.GetCaptureRecord(f.ctx, stranded.ID)
|
||||
untouched, _ := f.db.GetCaptureRecord(f.ctx, already.ID)
|
||||
if !old.Hidden || !reflect.DeepEqual(*got, stranded) || !reflect.DeepEqual(*untouched, already) {
|
||||
t.Fatalf("rollback changed prior request state: old=%+v stranded=%+v moved=%+v", old, got, untouched)
|
||||
}
|
||||
f.runtime.store = f.db
|
||||
f.activate()
|
||||
got, _ = f.db.GetCaptureRecord(f.ctx, stranded.ID)
|
||||
if got.WorkflowID != f.next.ID || got.ID != stranded.ID {
|
||||
t.Fatalf("hidden unstarted capture stranded: %+v", got)
|
||||
}
|
||||
f.scan()
|
||||
got, _ = f.db.GetCaptureRecord(f.ctx, stranded.ID)
|
||||
if got.RunID == "" {
|
||||
t.Fatalf("transferred hidden capture not admitted: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCaptureCutoverUnresolvedClaimBlocksBeforeAnyWrites(t *testing.T) {
|
||||
for _, hidden := range []bool{false, true} {
|
||||
for _, lookupFails := range []bool{false, true} {
|
||||
t.Run(fmt.Sprintf("hidden=%v/lookupFails=%v", hidden, lookupFails), func(t *testing.T) {
|
||||
f := newCaptureCutoverFixture(t, "claim", "fresh")
|
||||
claim := f.seed("claim", domain.CaptureRecordLaunching, "")
|
||||
fresh := f.seed("fresh", domain.CaptureRecordReady, "")
|
||||
f.old.Hidden = hidden
|
||||
if err := f.db.UpsertWorkflow(f.ctx, f.old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if lookupFails {
|
||||
f.runtime.store = &captureCutoverFaultStore{Store: f.db, getRun: func(context.Context, string, string) (*domain.RunRecord, error) {
|
||||
return nil, fmt.Errorf("injected Run lookup failure")
|
||||
}}
|
||||
}
|
||||
got := f.activateResponse()
|
||||
want := http.StatusConflict
|
||||
if lookupFails {
|
||||
want = http.StatusInternalServerError
|
||||
}
|
||||
if got.Code != want || (!lookupFails && !strings.Contains(got.Body.String(), "capture_launch_unresolved")) {
|
||||
t.Fatalf("unsafe activation: %d %s", got.Code, got.Body.String())
|
||||
}
|
||||
for _, before := range []domain.WorkflowRecord{f.old, f.next} {
|
||||
after, _ := f.db.GetWorkflow(f.ctx, before.ID)
|
||||
if !reflect.DeepEqual(*after, before) {
|
||||
t.Fatalf("preflight wrote workflow: %+v", after)
|
||||
}
|
||||
}
|
||||
for _, before := range []domain.CaptureRecord{claim, fresh} {
|
||||
after, _ := f.db.GetCaptureRecord(f.ctx, before.ID)
|
||||
if !reflect.DeepEqual(*after, before) {
|
||||
t.Fatalf("preflight wrote Capture: %+v", after)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCaptureCutoverScannerSurfacesUnresolvedClaimWithoutRewriting(t *testing.T) {
|
||||
for _, hidden := range []bool{false, true} {
|
||||
t.Run(fmt.Sprintf("hidden=%v", hidden), func(t *testing.T) {
|
||||
f := newCaptureCutoverFixture(t, "claim")
|
||||
claim := f.seed("claim", domain.CaptureRecordLaunching, "")
|
||||
owner := f.old
|
||||
if hidden {
|
||||
f.old.Hidden, f.next.Hidden = true, false
|
||||
for _, w := range []domain.WorkflowRecord{f.old, f.next} {
|
||||
if err := f.db.UpsertWorkflow(f.ctx, w); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
owner = f.next
|
||||
}
|
||||
err := f.runtime.processCaptureAutomation(f.ctx, owner)
|
||||
if err == nil || !strings.Contains(err.Error(), "capture_launch_unresolved") || !strings.Contains(err.Error(), claim.ID) || !strings.Contains(err.Error(), f.old.ID) {
|
||||
t.Fatalf("unresolved claim silently skipped: %v", err)
|
||||
}
|
||||
after, _ := f.db.GetCaptureRecord(f.ctx, claim.ID)
|
||||
runs, _ := f.db.ListRuns(f.ctx, "")
|
||||
if !reflect.DeepEqual(*after, claim) || len(runs) != 0 {
|
||||
t.Fatalf("unfenced recovery: Capture=%+v runs=%+v", after, runs)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCaptureCutoverUnknownCreateOutcomeNeverReopensAdmission(t *testing.T) {
|
||||
for _, persisted := range []bool{false, true} {
|
||||
t.Run(fmt.Sprintf("persisted=%v", persisted), func(t *testing.T) {
|
||||
f := newCaptureCutoverFixture(t, "claim")
|
||||
claim := f.seed("claim", domain.CaptureRecordReady, "")
|
||||
calls := 0
|
||||
f.runtime.store = &captureCutoverFaultStore{Store: f.db, createRun: func(ctx context.Context, id string, trigger domain.RunTrigger, inputs map[string]domain.ArtifactPayload, taskIDs ...string) (*domain.RunRecord, error) {
|
||||
calls++
|
||||
if persisted {
|
||||
if _, err := f.db.CreateRunWithInputs(ctx, id, trigger, inputs, taskIDs...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return nil, fmt.Errorf("injected lost CreateRun response")
|
||||
}}
|
||||
err := f.runtime.launchCaptureRun(f.ctx, f.old, normalizedCaptureAutomation(f.old.CaptureAutomation), &claim)
|
||||
if !persisted && (err == nil || !strings.Contains(err.Error(), "capture_launch_unresolved")) {
|
||||
t.Fatalf("missing actionable diagnostic: %v", err)
|
||||
}
|
||||
after, _ := f.db.GetCaptureRecord(f.ctx, claim.ID)
|
||||
if persisted {
|
||||
if after.RunID == "" {
|
||||
t.Fatalf("lost response not reconciled: %+v (%v)", after, err)
|
||||
}
|
||||
} else if after.Status != domain.CaptureRecordLaunching || after.RunID != "" {
|
||||
t.Fatalf("unknown outcome reopened admission: %+v", after)
|
||||
}
|
||||
_ = f.runtime.processCaptureAutomation(f.ctx, f.old)
|
||||
if calls != 1 {
|
||||
t.Fatalf("duplicated unknown CreateRun: %d", calls)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@ -1,6 +1,7 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"path"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
@ -186,3 +187,27 @@ func discoverCaptureGroups(objects []captureObject, config domain.CaptureAutomat
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// Keep legacy discovery unchanged. Only workflows explicitly requesting v4
|
||||
// need all eight original-source roles before a part is eligible for launch.
|
||||
func discoverCaptureGroupsForWorkflow(objects []captureObject, config domain.CaptureAutomationConfig, workflow domain.WorkflowRecord) ([]captureGroup, error) {
|
||||
groups, err := discoverCaptureGroups(objects, config)
|
||||
if err != nil || !workflowUsesV4CaptureDelivery(workflow) {
|
||||
return groups, err
|
||||
}
|
||||
for i := range groups {
|
||||
group := &groups[i]
|
||||
record := domain.CaptureRecord{Bucket: config.Bucket, CaptureID: group.CaptureID, Prefix: group.Prefix,
|
||||
SourcePrefix: group.SourcePrefix, SourceKeys: group.SourceKeys, Fingerprint: group.Fingerprint}
|
||||
if err := validateV4CaptureSourceSelection(workflow, &record); err != nil {
|
||||
group.Missing = append(group.Missing, err.Error())
|
||||
}
|
||||
metadataKey := group.SourcePrefix + "/meta_" + path.Base(group.SourcePrefix) + ".json"
|
||||
for _, object := range group.Objects {
|
||||
if strings.EqualFold(object.Key, metadataKey) && object.Size <= 0 {
|
||||
group.Missing = append(group.Missing, "v4 原始源文件为空 "+object.Key)
|
||||
}
|
||||
}
|
||||
}
|
||||
return groups, nil
|
||||
}
|
||||
|
||||
89
internal/worker/capture_v4_source.go
Normal file
89
internal/worker/capture_v4_source.go
Normal file
@ -0,0 +1,89 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
)
|
||||
|
||||
func workflowUsesV4CaptureDelivery(workflow domain.WorkflowRecord) bool {
|
||||
for _, node := range workflow.Nodes {
|
||||
if node.Data.OperatorID != "operator-builtin-package" {
|
||||
continue
|
||||
}
|
||||
// Match the package operator's boolean parser so alternate true values
|
||||
// cannot select v4 while silently bypassing the source preflight.
|
||||
switch strings.ToLower(strings.TrimSpace(nodeHyperparamsMap(node)["DELIVERY_JSON_V4"])) {
|
||||
case "1", "true", "yes", "on":
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// validateV4CaptureSourceSelection checks the exact frozen selection, without
|
||||
// listing storage or reconstructing a part from a whole-capture input. Call it
|
||||
// before claiming a new launch; already-created runs keep their own inputs.
|
||||
func validateV4CaptureSourceSelection(workflow domain.WorkflowRecord, record *domain.CaptureRecord) error {
|
||||
if !workflowUsesV4CaptureDelivery(workflow) {
|
||||
return nil
|
||||
}
|
||||
if record == nil {
|
||||
return fmt.Errorf("v4 original source requires a capture record")
|
||||
}
|
||||
root, prefix := record.SourcePrefix, record.Prefix
|
||||
if root == "" || path.Clean(root) != root || strings.HasPrefix(root, "/") || strings.ContainsAny(root, "\\\x00") || root == ".." || strings.HasPrefix(root, "../") {
|
||||
return fmt.Errorf("v4 original source requires a safe capture root")
|
||||
}
|
||||
part := path.Base(prefix)
|
||||
capture := path.Base(root)
|
||||
if !regexp.MustCompile(`^part[0-9]{4,}$`).MatchString(part) || prefix != root+"/"+part || record.CaptureID != capture+"/"+part {
|
||||
return fmt.Errorf("v4 original source requires one explicit capture/part selection")
|
||||
}
|
||||
if strings.TrimSpace(record.Bucket) == "" || strings.TrimSpace(record.Fingerprint) == "" {
|
||||
return fmt.Errorf("v4 original source requires a bucket and snapshot fingerprint")
|
||||
}
|
||||
capPattern, partPattern := regexp.QuoteMeta(capture), regexp.QuoteMeta(part)
|
||||
// These paths match the eight original-source roles consumed by v4. In
|
||||
// particular, metadata is rooted at the capture, never at the part directory.
|
||||
requirements := []captureRequirement{
|
||||
{"左相机视频", partPattern + `/` + capPattern + `_camera_left_` + partPattern + `(?:_[a-z0-9_]+)?\.(?:mp4|mov|mkv)`},
|
||||
{"右相机视频", partPattern + `/` + capPattern + `_camera_right_` + partPattern + `(?:_[a-z0-9_]+)?\.(?:mp4|mov|mkv)`},
|
||||
{"左相机 PTS", partPattern + `/` + capPattern + `_camera_left_` + partPattern + `(?:_[a-z0-9_]+)?_pts\.csv`},
|
||||
{"右相机 PTS", partPattern + `/` + capPattern + `_camera_right_` + partPattern + `(?:_[a-z0-9_]+)?_pts\.csv`},
|
||||
{"IMU", partPattern + `/` + capPattern + `_imu_` + partPattern + `\.csv`},
|
||||
{"相机标定", capPattern + `_calibration_camera\.yaml`},
|
||||
{"IMU 标定", capPattern + `_calibration_imu\.yaml`},
|
||||
{"meta_" + capture + ".json", `meta_` + capPattern + `\.json`},
|
||||
}
|
||||
counts := make([]int, len(requirements))
|
||||
patterns := make([]*regexp.Regexp, len(requirements))
|
||||
for i, requirement := range requirements {
|
||||
patterns[i] = regexp.MustCompile(`(?i)^` + requirement.Pattern + `$`)
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, key := range record.SourceKeys {
|
||||
if !strings.HasPrefix(key, root+"/") || path.Clean(key) != key || strings.ContainsAny(key, "\\\x00") || seen[key] {
|
||||
return fmt.Errorf("v4 original source contains an unsafe or duplicate object selection")
|
||||
}
|
||||
seen[key] = true
|
||||
relative := strings.TrimPrefix(key, root+"/")
|
||||
if strings.Contains(relative, "/") && !strings.HasPrefix(key, prefix+"/") {
|
||||
return fmt.Errorf("v4 original source selection includes another part")
|
||||
}
|
||||
for i, pattern := range patterns {
|
||||
if pattern.MatchString(relative) {
|
||||
counts[i]++
|
||||
}
|
||||
}
|
||||
}
|
||||
for i, count := range counts {
|
||||
if count != 1 {
|
||||
return fmt.Errorf("v4 original source requires exactly one %s (found %d)", requirements[i].Name, count)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
242
internal/worker/capture_v4_source_test.go
Normal file
242
internal/worker/capture_v4_source_test.go
Normal file
@ -0,0 +1,242 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/store"
|
||||
)
|
||||
|
||||
func captureV4Workflow(enabled bool) domain.WorkflowRecord {
|
||||
value := "false"
|
||||
if enabled {
|
||||
value = "true"
|
||||
}
|
||||
return domain.WorkflowRecord{Nodes: []domain.WorkflowNode{{ID: "package", Data: domain.WorkflowNodeData{
|
||||
OperatorID: "operator-builtin-package", Hyperparams: []domain.HyperParam{{Key: "DELIVERY_JSON_V4", Value: value}},
|
||||
}}}}
|
||||
}
|
||||
|
||||
func captureV4Objects() []captureObject {
|
||||
objects := append(calibrationObjects(), partObjects("part0001")...)
|
||||
return append(objects, captureObject{Key: "incoming/" + layoutCapture + "/meta_" + layoutCapture + ".json", Size: 20, ETag: "metadata-version"})
|
||||
}
|
||||
|
||||
func TestDiscoverCaptureGroupsForWorkflowV4RequiresCaptureMetadata(t *testing.T) {
|
||||
config := normalizedCaptureAutomation(&domain.CaptureAutomationConfig{Prefix: "incoming", Bucket: "synthetic-bucket"})
|
||||
objects := append(calibrationObjects(), partObjects("part0001")...)
|
||||
for _, enabled := range []bool{false, true} {
|
||||
groups, err := discoverCaptureGroupsForWorkflow(objects, config, captureV4Workflow(enabled))
|
||||
if err != nil || len(groups) != 1 {
|
||||
t.Fatalf("groups: %+v, %v", groups, err)
|
||||
}
|
||||
if enabled {
|
||||
if !strings.Contains(strings.Join(groups[0].Missing, " "), "meta_"+layoutCapture+".json") {
|
||||
t.Fatal("v4 admitted a seven-file part", groups[0])
|
||||
}
|
||||
} else if len(groups[0].Missing) != 0 || len(groups[0].SourceKeys) != 7 {
|
||||
t.Fatal("legacy intake must keep metadata optional", groups[0])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestV4CaptureMetadataMustBeNonemptyAndAtItsOwnCaptureRoot(t *testing.T) {
|
||||
config := normalizedCaptureAutomation(&domain.CaptureAutomationConfig{Prefix: "incoming", Bucket: "synthetic-bucket"})
|
||||
for _, location := range []string{"missing", "other-capture", "inside-part", "empty", "complete"} {
|
||||
t.Run(location, func(t *testing.T) {
|
||||
objects := captureV4Objects()
|
||||
meta := &objects[len(objects)-1]
|
||||
switch location {
|
||||
case "missing":
|
||||
objects = objects[:len(objects)-1]
|
||||
case "other-capture":
|
||||
meta.Key = "incoming/other/meta_" + layoutCapture + ".json"
|
||||
case "inside-part":
|
||||
meta.Key = "incoming/" + layoutCapture + "/part0001/meta_" + layoutCapture + ".json"
|
||||
case "empty":
|
||||
meta.Size = 0
|
||||
}
|
||||
groups, err := discoverCaptureGroupsForWorkflow(objects, config, captureV4Workflow(true))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, group := range groups {
|
||||
if group.CaptureID != layoutCapture+"/part0001" {
|
||||
continue
|
||||
}
|
||||
if (len(group.Missing) == 0) != (location == "complete") {
|
||||
t.Fatalf("%s: %+v", location, group)
|
||||
}
|
||||
return
|
||||
}
|
||||
t.Fatal("target part not discovered")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestV4CapturePreflightRejectsIncompleteOrUnprovableSelections(t *testing.T) {
|
||||
config := normalizedCaptureAutomation(&domain.CaptureAutomationConfig{Prefix: "incoming", Bucket: "synthetic-bucket"})
|
||||
groups, err := discoverCaptureGroups(captureV4Objects(), config)
|
||||
if err != nil || len(groups) != 1 || len(groups[0].Missing) != 0 {
|
||||
t.Fatalf("complete fixture: %+v, %v", groups, err)
|
||||
}
|
||||
group := groups[0]
|
||||
base := domain.CaptureRecord{Bucket: config.Bucket, CaptureID: group.CaptureID, Prefix: group.Prefix,
|
||||
SourcePrefix: group.SourcePrefix, SourceKeys: group.SourceKeys, Fingerprint: group.Fingerprint}
|
||||
for i, key := range base.SourceKeys {
|
||||
t.Run("missing/"+key, func(t *testing.T) {
|
||||
record := base
|
||||
record.SourceKeys = append(append([]string{}, base.SourceKeys[:i]...), base.SourceKeys[i+1:]...)
|
||||
if err := validateV4CaptureSourceSelection(captureV4Workflow(true), &record); err == nil {
|
||||
t.Fatal("incomplete selection admitted", key)
|
||||
}
|
||||
if err := validateV4CaptureSourceSelection(captureV4Workflow(false), &record); err != nil {
|
||||
t.Fatal("legacy preflight changed", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
for _, kind := range []string{"whole-capture", "cross-part", "duplicate-key", "duplicate-role", "wrong-metadata", "unsafe-key", "missing-fingerprint"} {
|
||||
t.Run(kind, func(t *testing.T) {
|
||||
record := base
|
||||
record.SourceKeys = append([]string{}, base.SourceKeys...)
|
||||
switch kind {
|
||||
case "whole-capture":
|
||||
record.Prefix, record.CaptureID = record.SourcePrefix, layoutCapture
|
||||
case "cross-part":
|
||||
record.SourceKeys = append(record.SourceKeys, record.SourcePrefix+"/part0002/foreign.mp4")
|
||||
case "duplicate-key":
|
||||
record.SourceKeys = append(record.SourceKeys, record.SourceKeys[0])
|
||||
case "duplicate-role":
|
||||
record.SourceKeys = append(record.SourceKeys, record.Prefix+"/"+layoutCapture+"_camera_left_part0001.mp4")
|
||||
case "wrong-metadata":
|
||||
for i, key := range record.SourceKeys {
|
||||
if strings.HasSuffix(key, ".json") {
|
||||
record.SourceKeys[i] = record.SourcePrefix + "/meta_another_capture.json"
|
||||
}
|
||||
}
|
||||
case "unsafe-key":
|
||||
record.SourceKeys[0] = record.SourcePrefix + "/../foreign.yaml"
|
||||
case "missing-fingerprint":
|
||||
record.Fingerprint = ""
|
||||
}
|
||||
if err := validateV4CaptureSourceSelection(captureV4Workflow(true), &record); err == nil {
|
||||
t.Fatal("unprovable v4 selection admitted", kind)
|
||||
}
|
||||
if err := validateV4CaptureSourceSelection(captureV4Workflow(false), &record); err != nil {
|
||||
t.Fatal("legacy preflight changed", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestV4CaptureSourceAndAuditMatchPackageBooleanValues(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ds := store.NewMemoryStore()
|
||||
runtime := &Runtime{store: ds}
|
||||
for _, tc := range []struct {
|
||||
value string
|
||||
enabled bool
|
||||
}{
|
||||
{"true", true}, {"TRUE", true}, {"1", true}, {"yes", true}, {" on ", true},
|
||||
{"false", false}, {"FALSE", false}, {"0", false}, {"no", false}, {"off", false}, {"", false},
|
||||
} {
|
||||
workflow := captureV4Workflow(true)
|
||||
workflow.ID = "source-audit-alias"
|
||||
workflow.Nodes[0].Data.Hyperparams[0].Value = tc.value
|
||||
if err := ds.UpsertWorkflow(ctx, workflow); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := validateV4CaptureSourceSelection(workflow, &domain.CaptureRecord{}); (err != nil) != tc.enabled {
|
||||
t.Fatalf("source preflight for package boolean %q: %v", tc.value, err)
|
||||
}
|
||||
if audit, err := runtime.classificationDeliveryEnabled(ctx, workflow.ID); err != nil || audit != tc.enabled {
|
||||
t.Fatalf("audit for package boolean %q: enabled=%v, err=%v", tc.value, audit, err)
|
||||
}
|
||||
}
|
||||
workflow := captureV4Workflow(true)
|
||||
workflow.ID = "non-package-alias"
|
||||
workflow.Nodes[0].Data.OperatorID = "unrelated-operator"
|
||||
if err := ds.UpsertWorkflow(ctx, workflow); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := validateV4CaptureSourceSelection(workflow, &domain.CaptureRecord{}); err != nil {
|
||||
t.Fatal("non-package parameter enabled the v4 gate", err)
|
||||
}
|
||||
if audit, err := runtime.classificationDeliveryEnabled(ctx, workflow.ID); err != nil || audit {
|
||||
t.Fatal("non-package parameter enabled v4 audit", audit, err)
|
||||
}
|
||||
if audit, err := runtime.classificationDeliveryEnabled(ctx, "missing-workflow"); err != nil || audit {
|
||||
t.Fatal("missing workflow enabled v4 audit", audit, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestV4CaptureLaunchFreezesCompletePartSourceSelection(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ds := store.NewMemoryStore()
|
||||
workflow, err := ds.CreateWorkflow(ctx, domain.CreateWorkflowInput{
|
||||
OrganizationID: domain.DefaultOrganizationID, Name: "new v4 capture workflow", Type: domain.WorkflowTypeStreaming,
|
||||
WorkflowInputs: []domain.WorkflowInputDefinition{{ID: "sourceDirectory", Type: domain.PortTypeFile, Container: domain.PortContainerDirectory, PayloadMode: domain.ArtifactTransferModeRef, Required: true}},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
config := normalizedCaptureAutomation(&domain.CaptureAutomationConfig{
|
||||
Enabled: true, ConnectionID: "synthetic-source", Bucket: "synthetic-bucket", Prefix: "incoming",
|
||||
DeliveryPrefix: "deliveries", DeliveryProjectDirectory: "Synthetic__project",
|
||||
})
|
||||
workflow.Nodes = captureV4Workflow(true).Nodes
|
||||
workflow.DefinitionState = domain.WorkflowDefinitionStateReleased
|
||||
workflow.CaptureAutomation = &config
|
||||
if err = ds.UpsertWorkflow(ctx, *workflow); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
objects := append(captureV4Objects(), partObjects("part0002")...)
|
||||
groups, err := discoverCaptureGroupsForWorkflow(objects, config, *workflow)
|
||||
if err != nil || len(groups) != 2 {
|
||||
t.Fatalf("groups: %+v, %v", groups, err)
|
||||
}
|
||||
runtime := &Runtime{store: ds}
|
||||
for _, group := range groups {
|
||||
if len(group.Missing) != 0 {
|
||||
t.Fatal("complete source rejected", group)
|
||||
}
|
||||
record := domain.CaptureRecord{
|
||||
ID: captureRecordID(workflow.ID, group.Prefix), OrganizationID: workflow.OrganizationID, WorkflowID: workflow.ID,
|
||||
ConnectionID: config.ConnectionID, Bucket: config.Bucket, CaptureID: group.CaptureID, Prefix: group.Prefix,
|
||||
SourcePrefix: group.SourcePrefix, SourceKeys: group.SourceKeys, Fingerprint: group.Fingerprint, Status: domain.CaptureRecordReady,
|
||||
}
|
||||
if err = validateV4CaptureSourceSelection(*workflow, &record); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = ds.UpsertCaptureRecord(ctx, record); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = runtime.launchCaptureRun(ctx, *workflow, config, &record); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
run, err := ds.GetRun(ctx, record.RunID)
|
||||
if err != nil || run == nil {
|
||||
t.Fatalf("run missing: %v", err)
|
||||
}
|
||||
ref := run.Inputs["sourceDirectory"].Ref
|
||||
if ref == nil || ref.Prefix != "incoming/"+layoutCapture || ref.CaptureUnit != group.CaptureID || ref.VersionID != group.Fingerprint || len(ref.ObjectKeys) != 8 || !reflect.DeepEqual(ref.ObjectKeys, group.SourceKeys) {
|
||||
t.Fatalf("source proof did not survive run creation: %+v", ref)
|
||||
}
|
||||
if err = domain.ValidateArtifactPayload(run.Inputs["sourceDirectory"], domain.PortTypeFile); err != nil {
|
||||
t.Fatal("v5.8 input contract rejected selected part", err)
|
||||
}
|
||||
// Later discovery cannot rewrite the input used by annotationOriginalSource.
|
||||
before := append([]string(nil), ref.ObjectKeys...)
|
||||
record.SourceKeys[0] = "incoming/unrelated/file"
|
||||
if err = ds.UpsertCaptureRecord(ctx, record); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
run, err = ds.GetRun(ctx, record.RunID)
|
||||
if err != nil || !reflect.DeepEqual(run.Inputs["sourceDirectory"].Ref.ObjectKeys, before) {
|
||||
t.Fatal("frozen run selection followed later discovery", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -76,17 +76,7 @@ func (runtime *Runtime) classificationDeliveryEnabled(ctx context.Context, workf
|
||||
if workflow == nil {
|
||||
return false, nil
|
||||
}
|
||||
for _, node := range workflow.Nodes {
|
||||
if node.Data.OperatorID != "operator-builtin-package" {
|
||||
continue
|
||||
}
|
||||
for _, param := range node.Data.Hyperparams {
|
||||
if param.Key == "DELIVERY_JSON_V4" && param.Value == "true" {
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
return false, nil
|
||||
return workflowUsesV4CaptureDelivery(*workflow), nil
|
||||
}
|
||||
|
||||
// Reuse normal internal-artifact persistence rather than exceeding the 64 KiB
|
||||
|
||||
@ -1,14 +1,14 @@
|
||||
{
|
||||
"acceptance": {},
|
||||
"builtAt": "2026-08-31T19:36:30Z",
|
||||
"digest": "sha256:87965e34785dcc8027934a4291a4d35de0918cf68b681a5ed5cc184e2a4bddf2",
|
||||
"builtAt": "2026-10-21T16:50:46Z",
|
||||
"digest": "sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d",
|
||||
"environmentReleases": {
|
||||
"fanyu": {
|
||||
"digest": "sha256:87965e34785dcc8027934a4291a4d35de0918cf68b681a5ed5cc184e2a4bddf2",
|
||||
"image": "ccr-29eug8s3-vpc.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package:b085e371-fanyu-20260901-capture-v5",
|
||||
"imageId": "ccr-29eug8s3-vpc.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package@sha256:87965e34785dcc8027934a4291a4d35de0918cf68b681a5ed5cc184e2a4bddf2",
|
||||
"registry": "ccr-29eug8s3-vpc.cnc.bj.baidubce.com",
|
||||
"tag": "b085e371-fanyu-20260901-capture-v5"
|
||||
"digest": "sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d",
|
||||
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package:delivery-v4-20261022-8f6b531f",
|
||||
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package@sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d",
|
||||
"registry": "ccr-29eug8s3-pub.cnc.bj.baidubce.com",
|
||||
"tag": "delivery-v4-20261022-8f6b531f"
|
||||
},
|
||||
"poc": {
|
||||
"digest": "sha256:fa084bbd0e3b6fedff9582d09c965b460f054ad9e0f5edaf9fc4b9aebcc375cc",
|
||||
@ -18,14 +18,14 @@
|
||||
"tag": "feaaf80-poc-handpose-auto-20260825052001"
|
||||
}
|
||||
},
|
||||
"image": "ccr-29eug8s3-vpc.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package:b085e371-fanyu-20260901-capture-v5",
|
||||
"imageId": "ccr-29eug8s3-vpc.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package@sha256:87965e34785dcc8027934a4291a4d35de0918cf68b681a5ed5cc184e2a4bddf2",
|
||||
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package:delivery-v4-20261022-8f6b531f",
|
||||
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package@sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d",
|
||||
"kanikoJob": "droboticflow-kaniko-op-package-b085e371-fanyu-20260901-capture",
|
||||
"operatorId": "operator-builtin-package",
|
||||
"registry": "ccr-29eug8s3-vpc.cnc.bj.baidubce.com",
|
||||
"registry": "ccr-29eug8s3-pub.cnc.bj.baidubce.com",
|
||||
"schemaVersion": "droboticflow.operator.release/v1",
|
||||
"slug": "package",
|
||||
"sourceCommit": "b085e371b230b304fb0446040dee98793dc2e545",
|
||||
"sourceTreeSha256": "8cb9d11686fa3c44c33117340ee6b115b6880a0e73e1fc0807d9b997a117fe3d",
|
||||
"tag": "b085e371-fanyu-20260901-capture-v5"
|
||||
"sourceCommit": "8f6b531fd322f04d0ecf02d74f77348169f773e9",
|
||||
"sourceTreeSha256": "1db72650763169528362fa2e66d14689d76b2141de1c8aba0f4721559d3882cc",
|
||||
"tag": "delivery-v4-20261022-8f6b531f"
|
||||
}
|
||||
|
||||
@ -19,8 +19,8 @@
|
||||
],
|
||||
"containerEntrypoint": "backend/operator-runner/main.py",
|
||||
"entrypoint": "src/operator.py",
|
||||
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-package:feaaf80-poc-handpose-auto-20260825052001",
|
||||
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-operator-package@sha256:fa084bbd0e3b6fedff9582d09c965b460f054ad9e0f5edaf9fc4b9aebcc375cc",
|
||||
"image": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package:delivery-v4-20261022-8f6b531f",
|
||||
"imageId": "ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/droboticflow-operator-package@sha256:dc22ee3b77f00a8e27aa4d4cf11b1b225e8b8d476b8dd3429d43c4d73396ca1d",
|
||||
"metadata": "operator.json",
|
||||
"name": "成品打包",
|
||||
"operatorId": "operator-builtin-package",
|
||||
@ -83,7 +83,7 @@
|
||||
"runtimeSource": "operator-repos/builtin/package/src/operator.py",
|
||||
"schemaVersion": "droboticflow.operator.repository/v1",
|
||||
"sourceRoot": "operator-repos/builtin/package",
|
||||
"sourceTreeSha256": "9c91471144466566083b516e71ac10ae948c1d1490d59708ff9b2068daed759a",
|
||||
"sourceTreeSha256": "1db72650763169528362fa2e66d14689d76b2141de1c8aba0f4721559d3882cc",
|
||||
"strictPocAcceptanceExternals": [],
|
||||
"type": "builtin"
|
||||
}
|
||||
|
||||
@ -198,13 +198,19 @@ def legacy_manifest(
|
||||
return manifest
|
||||
|
||||
|
||||
def expected_manifests() -> dict[pathlib.Path, dict[str, Any]]:
|
||||
def expected_manifests(slugs: set[str] | None = None) -> dict[pathlib.Path, dict[str, Any]]:
|
||||
catalog_items = {
|
||||
item["id"]: item for item in load_json(CATALOG_PATH) if item.get("type") == "builtin"
|
||||
}
|
||||
manifests: dict[pathlib.Path, dict[str, Any]] = {}
|
||||
split_operator_ids: set[str] = set()
|
||||
for metadata_path in sorted(OPERATOR_REPO_ROOT.glob("*/operator.json")):
|
||||
metadata_paths = sorted(OPERATOR_REPO_ROOT.glob("*/operator.json"))
|
||||
if slugs is not None:
|
||||
available = {path.parent.name for path in metadata_paths} - UNMIGRATED_HISTORICAL_REPOSITORIES
|
||||
if not slugs or slugs - available:
|
||||
raise RuntimeError("unknown or empty operator repository selection")
|
||||
metadata_paths = [path for path in metadata_paths if path.parent.name in slugs]
|
||||
for metadata_path in metadata_paths:
|
||||
if metadata_path.parent.name in UNMIGRATED_HISTORICAL_REPOSITORIES:
|
||||
continue
|
||||
metadata = load_json(metadata_path)
|
||||
@ -286,7 +292,7 @@ def expected_manifests() -> dict[pathlib.Path, dict[str, Any]]:
|
||||
image_id=image_id,
|
||||
dependencies=dependencies,
|
||||
)
|
||||
missing_catalog = sorted(set(catalog_items) - split_operator_ids)
|
||||
missing_catalog = sorted(set(catalog_items) - split_operator_ids) if slugs is None else []
|
||||
if missing_catalog:
|
||||
raise RuntimeError(f"catalog operators missing repository manifests: {', '.join(missing_catalog)}")
|
||||
return manifests
|
||||
@ -319,9 +325,10 @@ def main() -> int:
|
||||
mode = parser.add_mutually_exclusive_group()
|
||||
mode.add_argument("--write", action="store_true", help="write repository.json files")
|
||||
mode.add_argument("--check", action="store_true", help="check repository.json files without writing")
|
||||
parser.add_argument("--slug", action="append", help="publish/check only these independent repositories; default checks every operator")
|
||||
args = parser.parse_args()
|
||||
|
||||
manifests = expected_manifests()
|
||||
manifests = expected_manifests(set(args.slug) if args.slug is not None else None)
|
||||
if args.check:
|
||||
return check_manifests(manifests)
|
||||
write_manifests(manifests)
|
||||
|
||||
243
scripts/seed_dictionary_tags.py
Normal file
243
scripts/seed_dictionary_tags.py
Normal file
@ -0,0 +1,243 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Offline name-as-first-tag plan; Mongo apply requires its exact confirmed hash.
|
||||
|
||||
Plan: --snapshot FILE --database DB --organization ORG --updated-by USER
|
||||
[--at ISO_TIMESTAMP] [--output PLAN.json]
|
||||
Apply: --plan PLAN.json --apply --confirm-plan SHA256
|
||||
(URI only from DICTIONARY_TAGS_MONGO_URI).
|
||||
|
||||
Without --apply no database connection is made. Only annotationLabels changes;
|
||||
saved classifications and existing nonempty tags are never rewritten. History
|
||||
is copied/appended inside Mongo, never fetched by this script.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
FIELDS = ("id", "organizationId", "name", "kind", "tags", "version", "status", "updatedAt")
|
||||
PROJECTION = {**dict.fromkeys(FIELDS, 1), "_id": 0}
|
||||
COLLECTION = "annotationLabels"
|
||||
SCHEMA = "droboticflow.dictionary-tags-seed/v1"
|
||||
ENTRY_FIELDS = {"id", "name", "version", "expectedHash"}
|
||||
PLAN_FIELDS = {"schemaVersion", "database", "organizationId", "collection", "updatedAt", "updatedByUserId", "entries", "planHash"}
|
||||
|
||||
|
||||
def digest(value):
|
||||
return hashlib.sha256(json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False).encode()).hexdigest()
|
||||
|
||||
|
||||
def projected(row):
|
||||
if not isinstance(row, dict) or set(row) - set(FIELDS):
|
||||
raise ValueError("snapshot/read contains fields outside the metadata whitelist")
|
||||
return {key: row[key] for key in FIELDS if key in row}
|
||||
|
||||
|
||||
def text(value, field):
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise ValueError(f"{field} must be a nonempty string")
|
||||
return value
|
||||
|
||||
|
||||
def timestamp(value):
|
||||
text(value, "updatedAt")
|
||||
parsed = dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
if parsed.tzinfo is None:
|
||||
raise ValueError("updatedAt must include a timezone")
|
||||
return value
|
||||
|
||||
|
||||
def build_plan(snapshot, *, database, organization, updated_by, at):
|
||||
rows = snapshot.get("active-dictionary-tag-readiness") if isinstance(snapshot, dict) else snapshot
|
||||
if not isinstance(rows, list):
|
||||
raise ValueError("snapshot must contain a list of projected dictionary records")
|
||||
entries, seen = [], set()
|
||||
for raw in rows:
|
||||
row = projected(raw)
|
||||
if row.get("organizationId") != organization:
|
||||
raise ValueError("snapshot organization differs from fixed scope")
|
||||
label_id = text(row.get("id"), "id")
|
||||
if label_id in seen:
|
||||
raise ValueError("duplicate dictionary id")
|
||||
seen.add(label_id)
|
||||
tags = row.get("tags")
|
||||
if tags is not None and (not isinstance(tags, list) or any(not isinstance(tag, str) for tag in tags)):
|
||||
raise ValueError("tags must be an array of strings or absent/null")
|
||||
if row.get("status") != "active" or tags:
|
||||
continue
|
||||
name = text(row.get("name"), "name")
|
||||
version = row.get("version")
|
||||
if type(version) is not int or version < 1:
|
||||
raise ValueError("active dictionary requires a positive integer version")
|
||||
text(row.get("updatedAt"), "existing updatedAt")
|
||||
entries.append({"id": label_id, "name": name, "version": version, "expectedHash": digest(row)})
|
||||
plan = {
|
||||
"schemaVersion": SCHEMA, "database": text(database, "database"),
|
||||
"organizationId": text(organization, "organization"), "collection": COLLECTION,
|
||||
"updatedAt": timestamp(at), "updatedByUserId": text(updated_by, "updated-by"),
|
||||
"entries": sorted(entries, key=lambda entry: entry["id"]),
|
||||
}
|
||||
plan["planHash"] = digest(plan)
|
||||
validate_plan(plan)
|
||||
return plan
|
||||
|
||||
|
||||
def validate_plan(plan, confirmation=None):
|
||||
if not isinstance(plan, dict) or set(plan) != PLAN_FIELDS or plan.get("schemaVersion") != SCHEMA or plan.get("collection") != COLLECTION:
|
||||
raise ValueError("invalid plan shape or collection")
|
||||
expected = digest({key: value for key, value in plan.items() if key != "planHash"})
|
||||
if plan.get("planHash") != expected or (confirmation is not None and not hmac.compare_digest(confirmation, expected)):
|
||||
raise ValueError("plan hash/confirmation mismatch")
|
||||
for field in ("database", "organizationId", "updatedByUserId"):
|
||||
text(plan[field], field)
|
||||
timestamp(plan["updatedAt"])
|
||||
if not isinstance(plan["entries"], list):
|
||||
raise ValueError("entries must be an array")
|
||||
ids = set()
|
||||
for entry in plan["entries"]:
|
||||
if not isinstance(entry, dict) or set(entry) != ENTRY_FIELDS:
|
||||
raise ValueError("invalid entry shape")
|
||||
for field in ("id", "name"):
|
||||
text(entry[field], field)
|
||||
if entry["id"] in ids or type(entry["version"]) is not int or entry["version"] < 1:
|
||||
raise ValueError("duplicate id or invalid version")
|
||||
ids.add(entry["id"])
|
||||
value = entry["expectedHash"]
|
||||
if not isinstance(value, str) or len(value) != 64 or any(c not in "0123456789abcdef" for c in value):
|
||||
raise ValueError("invalid expected metadata hash")
|
||||
|
||||
|
||||
def revision_expression():
|
||||
# These expressions execute on the server, not in a fetched document.
|
||||
defaults = {
|
||||
"requiresNote": False, "version": 1, "code": "", "name": "", "category": "",
|
||||
"description": "", "example": "", "color": "", "tags": [], "aliases": [],
|
||||
"status": "", "updatedByUserId": "", "updatedAt": "",
|
||||
}
|
||||
return {key: {"$ifNull": ["$" + key, default]} for key, default in defaults.items()}
|
||||
|
||||
|
||||
def update_pipeline(plan, entry):
|
||||
literal = lambda value: {"$literal": value}
|
||||
version = {"$add": ["$version", 1]}
|
||||
old = revision_expression()
|
||||
changes = {"tags": literal([entry["name"]]), "version": version,
|
||||
"updatedAt": literal(plan["updatedAt"]), "updatedByUserId": literal(plan["updatedByUserId"])}
|
||||
new = {**old, **changes}
|
||||
history = {"$ifNull": ["$history", []]}
|
||||
return [{"$set": {
|
||||
**changes,
|
||||
"history": {"$concatArrays": [
|
||||
{"$cond": [{"$eq": [{"$size": history}, 0]}, [old], history]}, [new],
|
||||
]},
|
||||
}}]
|
||||
|
||||
|
||||
def cas_filter(row):
|
||||
# Preserve missing-vs-null and compare arrays exactly through $expr.
|
||||
return {
|
||||
"id": row["id"], "organizationId": row["organizationId"],
|
||||
"$and": [{field: {"$exists": field in row}} for field in FIELDS],
|
||||
"$expr": {"$and": [{"$eq": ["$" + field, {"$literal": value}]} for field, value in row.items()]},
|
||||
}
|
||||
|
||||
|
||||
def apply_plan(database, plan, confirmation):
|
||||
validate_plan(plan, confirmation)
|
||||
if database.name != plan["database"]:
|
||||
raise ValueError("database differs from fixed plan scope")
|
||||
collection = database[COLLECTION]
|
||||
entries = plan["entries"]
|
||||
if not entries:
|
||||
return {"updated": 0, "alreadyApplied": 0}
|
||||
rows = {}
|
||||
for raw in collection.find({"organizationId": plan["organizationId"], "id": {"$in": [entry["id"] for entry in entries]}}, PROJECTION):
|
||||
row = projected(raw)
|
||||
if row.get("id") in rows:
|
||||
raise ValueError("duplicate current dictionary id")
|
||||
rows[row.get("id")] = row
|
||||
pending, already = [], 0
|
||||
# Preflight every entry before the first write. CAS also checks the interval
|
||||
# between this read and each atomic update; partial runs can be retried.
|
||||
for entry in entries:
|
||||
row = rows.get(entry["id"])
|
||||
if row and all(row.get(key) == value for key, value in {
|
||||
"id": entry["id"], "organizationId": plan["organizationId"], "name": entry["name"],
|
||||
"status": "active", "tags": [entry["name"]], "version": entry["version"] + 1,
|
||||
"updatedAt": plan["updatedAt"],
|
||||
}.items()):
|
||||
already += 1
|
||||
continue
|
||||
if row is None or digest(row) != entry["expectedHash"]:
|
||||
raise ValueError(f"metadata drift or missing dictionary: {entry['id']}")
|
||||
if row.get("status") != "active" or row.get("tags") or row.get("name") != entry["name"] or row.get("version") != entry["version"] or row.get("organizationId") != plan["organizationId"]:
|
||||
raise ValueError("entry is not eligible for name-as-first-tag initialization")
|
||||
pending.append((entry, row))
|
||||
updated = 0
|
||||
for entry, row in pending:
|
||||
result = collection.update_one(cas_filter(row), update_pipeline(plan, entry), upsert=False)
|
||||
if result.matched_count != 1 or result.modified_count != 1:
|
||||
raise ValueError(f"concurrent dictionary change; stop and replan: {entry['id']}")
|
||||
updated += 1
|
||||
return {"updated": updated, "alreadyApplied": already}
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
source = parser.add_mutually_exclusive_group(required=True)
|
||||
source.add_argument("--snapshot", type=Path)
|
||||
source.add_argument("--plan", type=Path)
|
||||
for option in ("database", "organization", "updated-by", "at"):
|
||||
parser.add_argument("--" + option)
|
||||
parser.add_argument("--output", type=Path)
|
||||
parser.add_argument("--apply", action="store_true")
|
||||
parser.add_argument("--confirm-plan")
|
||||
args = parser.parse_args(argv)
|
||||
if args.apply and (not args.plan or not args.confirm_plan or args.output):
|
||||
parser.error("apply requires --plan and --confirm-plan; --output is plan-only")
|
||||
if args.confirm_plan and not args.apply:
|
||||
parser.error("--confirm-plan requires --apply")
|
||||
if args.snapshot:
|
||||
if not all((args.database, args.organization, args.updated_by)):
|
||||
parser.error("snapshot planning requires --database, --organization and --updated-by")
|
||||
plan = build_plan(json.loads(args.snapshot.read_text()), database=args.database, organization=args.organization,
|
||||
updated_by=args.updated_by, at=args.at or dt.datetime.now(dt.timezone.utc).isoformat().replace("+00:00", "Z"))
|
||||
else:
|
||||
if any((args.database, args.organization, args.updated_by, args.at)):
|
||||
parser.error("saved plan scope and timestamps cannot be overridden")
|
||||
plan = json.loads(args.plan.read_text())
|
||||
validate_plan(plan)
|
||||
if not args.apply:
|
||||
rendered = json.dumps(plan, ensure_ascii=False, sort_keys=True, indent=2) + "\n"
|
||||
if args.output:
|
||||
args.output.write_text(rendered)
|
||||
else:
|
||||
print(rendered, end="")
|
||||
return 0
|
||||
validate_plan(plan, args.confirm_plan)
|
||||
uri = os.environ.get("DICTIONARY_TAGS_MONGO_URI")
|
||||
if not uri:
|
||||
parser.error("DICTIONARY_TAGS_MONGO_URI must be set for apply")
|
||||
from pymongo import MongoClient
|
||||
from pymongo.errors import PyMongoError
|
||||
try:
|
||||
with MongoClient(uri, serverSelectionTimeoutMS=10000) as client:
|
||||
result = apply_plan(client[plan["database"]], plan, args.confirm_plan)
|
||||
except PyMongoError:
|
||||
raise ValueError("Mongo operation failed; inspect state with the same narrow projection before retrying") from None
|
||||
print(json.dumps({"planHash": plan["planHash"], **result}, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except ValueError as error:
|
||||
raise SystemExit(str(error)) from None
|
||||
40
tests/backend/test_operator_manifest_selection.py
Normal file
40
tests/backend/test_operator_manifest_selection.py
Normal file
@ -0,0 +1,40 @@
|
||||
"""A scoped publication validates its own release; the full check stays strict."""
|
||||
|
||||
import pytest
|
||||
|
||||
from scripts import generate_operator_repository_manifests as manifests
|
||||
|
||||
|
||||
def test_package_publication_checks_its_actual_release_without_loading_other_releases(monkeypatch):
|
||||
load = manifests.load_release
|
||||
calls = []
|
||||
|
||||
def selected(slug):
|
||||
calls.append(slug)
|
||||
if slug != "package":
|
||||
raise RuntimeError("unrelated release must still fail the full check")
|
||||
return load(slug)
|
||||
|
||||
monkeypatch.setattr(manifests, "load_release", selected)
|
||||
generated = manifests.expected_manifests({"package"})
|
||||
assert calls == ["package"]
|
||||
assert len(generated) == 2
|
||||
assert all(path.parent.name == "package" for path in generated)
|
||||
assert all(value["operatorId"] == "operator-builtin-package" for value in generated.values())
|
||||
with pytest.raises(RuntimeError, match="unrelated release"):
|
||||
manifests.expected_manifests()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("selection", [set(), {"not-an-operator"}, {"../package"}, {"temporal_causal_eval"}])
|
||||
def test_unknown_or_empty_selection_cannot_silently_pass(selection):
|
||||
with pytest.raises(RuntimeError, match="selection"):
|
||||
manifests.expected_manifests(selection)
|
||||
|
||||
|
||||
def test_selected_release_errors_are_not_ignored(monkeypatch):
|
||||
def invalid(_slug):
|
||||
raise RuntimeError("selected digest drift")
|
||||
|
||||
monkeypatch.setattr(manifests, "load_release", invalid)
|
||||
with pytest.raises(RuntimeError, match="selected digest drift"):
|
||||
manifests.expected_manifests({"package"})
|
||||
275
tests/backend/test_seed_dictionary_tags.py
Normal file
275
tests/backend/test_seed_dictionary_tags.py
Normal file
@ -0,0 +1,275 @@
|
||||
"""Offline plan/CAS checks; opt-in integration uses a new synthetic database."""
|
||||
|
||||
import copy
|
||||
import json
|
||||
import os
|
||||
from types import SimpleNamespace
|
||||
from urllib.parse import urlsplit
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
|
||||
from scripts import seed_dictionary_tags as seed
|
||||
|
||||
|
||||
AT = "2026-10-22T08:00:00Z"
|
||||
MISSING = object()
|
||||
|
||||
|
||||
def label(label_id="one", **changes):
|
||||
return {
|
||||
"id": label_id, "organizationId": "synthetic-org", "kind": "action", "name": "拿取",
|
||||
"version": 1, "status": "active", "updatedAt": "2026-10-21T08:00:00Z",
|
||||
"code": "pick", "description": "synthetic description", "example": "synthetic example",
|
||||
"aliases": ["synthetic alias"], "category": "category", "requiresNote": True,
|
||||
"color": "blue", "updatedByUserId": "old-user", **changes,
|
||||
}
|
||||
|
||||
|
||||
def metadata(doc):
|
||||
return {field: copy.deepcopy(doc[field]) for field in seed.FIELDS if field in doc}
|
||||
|
||||
|
||||
def plan_for(docs, database="synthetic-dictionary-tags"):
|
||||
return seed.build_plan({"active-dictionary-tag-readiness": [metadata(doc) for doc in docs]},
|
||||
database=database, organization="synthetic-org", updated_by="seed-test", at=AT)
|
||||
|
||||
|
||||
def expression(value, doc):
|
||||
"""Small generic Mongo expression evaluator for the offline fake server."""
|
||||
if isinstance(value, str) and value.startswith("$"):
|
||||
return doc.get(value[1:], MISSING)
|
||||
if isinstance(value, list):
|
||||
return [expression(item, doc) for item in value]
|
||||
if not isinstance(value, dict):
|
||||
return value
|
||||
if "$literal" in value:
|
||||
return copy.deepcopy(value["$literal"])
|
||||
if "$ifNull" in value:
|
||||
first, fallback = value["$ifNull"]
|
||||
result = expression(first, doc)
|
||||
return expression(fallback, doc) if result is MISSING or result is None else result
|
||||
if "$cond" in value:
|
||||
condition, yes, no = value["$cond"]
|
||||
return expression(yes if expression(condition, doc) else no, doc)
|
||||
if "$eq" in value:
|
||||
a, b = expression(value["$eq"], doc)
|
||||
return a == b
|
||||
if "$and" in value:
|
||||
return all(expression(item, doc) for item in value["$and"])
|
||||
if "$size" in value:
|
||||
return len(expression(value["$size"], doc))
|
||||
if "$add" in value:
|
||||
return sum(expression(value["$add"], doc))
|
||||
if "$concatArrays" in value:
|
||||
return sum(expression(value["$concatArrays"], doc), [])
|
||||
return {key: expression(item, doc) for key, item in value.items()}
|
||||
|
||||
|
||||
class FakeCollection:
|
||||
def __init__(self, docs):
|
||||
self.docs = copy.deepcopy(docs)
|
||||
self.writes = []
|
||||
self.reads = []
|
||||
self.before_update = None
|
||||
|
||||
def find(self, query, projection):
|
||||
assert projection == seed.PROJECTION # No full-document/history read.
|
||||
self.reads.append(copy.deepcopy(projection))
|
||||
return [metadata(doc) for doc in self.docs if doc["organizationId"] == query["organizationId"] and doc["id"] in query["id"]["$in"]]
|
||||
|
||||
def update_one(self, query, pipeline, *, upsert):
|
||||
assert upsert is False
|
||||
if self.before_update:
|
||||
self.before_update(self)
|
||||
self.writes.append((query, pipeline))
|
||||
for doc in self.docs:
|
||||
if doc["id"] != query["id"] or doc["organizationId"] != query["organizationId"]:
|
||||
continue
|
||||
if not all((field in doc) == options["$exists"] for clause in query["$and"] for field, options in clause.items()):
|
||||
continue
|
||||
if not expression(query["$expr"], doc):
|
||||
continue
|
||||
assert len(pipeline) == 1 and set(pipeline[0]) == {"$set"}
|
||||
changes = expression(pipeline[0]["$set"], doc)
|
||||
assert set(changes) == {"tags", "version", "updatedAt", "updatedByUserId", "history"}
|
||||
doc.update(changes)
|
||||
return SimpleNamespace(matched_count=1, modified_count=1)
|
||||
return SimpleNamespace(matched_count=0, modified_count=0)
|
||||
|
||||
|
||||
class FakeDatabase:
|
||||
def __init__(self, docs, name="synthetic-dictionary-tags"):
|
||||
self.name = name
|
||||
self.collection = FakeCollection(docs)
|
||||
|
||||
def __getitem__(self, name):
|
||||
assert name == "annotationLabels"
|
||||
return self.collection
|
||||
|
||||
|
||||
def test_offline_plan_is_fixed_minimal_and_skips_existing_tags_and_inactive():
|
||||
docs = [label(), label("tagged", tags=["existing"]), label("inactive", status="inactive")]
|
||||
plan = plan_for(docs)
|
||||
assert plan == plan_for(docs)
|
||||
assert len(plan["entries"]) == 1
|
||||
assert set(plan["entries"][0]) == seed.ENTRY_FIELDS
|
||||
assert "description" not in json.dumps(plan)
|
||||
with pytest.raises(ValueError, match="whitelist"):
|
||||
seed.build_plan([docs[0]], database="x", organization="synthetic-org", updated_by="user", at=AT)
|
||||
with pytest.raises(ValueError, match="organization"):
|
||||
plan_for([label(organizationId="other")])
|
||||
|
||||
|
||||
@pytest.mark.parametrize("history", [MISSING, None, []])
|
||||
def test_empty_history_gets_old_and_new_revisions_without_fetching_fields(history):
|
||||
doc = label(name="$literal-name")
|
||||
if history is not MISSING:
|
||||
doc["history"] = history
|
||||
database = FakeDatabase([doc])
|
||||
plan = plan_for([doc])
|
||||
assert seed.apply_plan(database, plan, plan["planHash"]) == {"updated": 1, "alreadyApplied": 0}
|
||||
result = database.collection.docs[0]
|
||||
assert result["tags"] == ["$literal-name"] and result["version"] == 2
|
||||
assert [rev["version"] for rev in result["history"]] == [1, 2]
|
||||
assert result["history"][0]["tags"] == []
|
||||
for field in ("description", "example", "aliases", "category", "code", "requiresNote", "color", "name"):
|
||||
assert result[field] == doc[field]
|
||||
assert result["history"][0][field] == result["history"][1][field] == doc[field]
|
||||
assert result["history"][0]["updatedByUserId"] == "old-user"
|
||||
assert result["history"][1]["updatedByUserId"] == "seed-test"
|
||||
assert result["history"][1]["updatedAt"] == AT
|
||||
assert seed.apply_plan(database, plan, plan["planHash"]) == {"updated": 0, "alreadyApplied": 1}
|
||||
assert len(database.collection.writes) == 1
|
||||
|
||||
|
||||
def test_existing_history_is_preserved_without_truncation():
|
||||
history = [{"version": index, "description": f"history-{index}"} for index in range(1, 105)]
|
||||
doc = label(version=104, history=history, tags=[])
|
||||
database = FakeDatabase([doc])
|
||||
plan = plan_for([doc])
|
||||
seed.apply_plan(database, plan, plan["planHash"])
|
||||
result = database.collection.docs[0]
|
||||
assert result["history"][:-1] == history
|
||||
assert result["history"][-1]["version"] == 105
|
||||
assert result["history"][-1]["tags"] == [doc["name"]]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("field,value", [
|
||||
("id", "changed"), ("organizationId", "other"), ("name", "changed"), ("kind", "slice"),
|
||||
("tags", ["other"]), ("version", 2), ("status", "inactive"), ("updatedAt", AT),
|
||||
])
|
||||
def test_any_projected_drift_aborts_before_writes(field, value):
|
||||
docs = [label(), label("two")]
|
||||
plan = plan_for(docs)
|
||||
database = FakeDatabase(docs)
|
||||
database.collection.docs[1][field] = value
|
||||
with pytest.raises(ValueError, match="drift|missing"):
|
||||
seed.apply_plan(database, plan, plan["planHash"])
|
||||
assert database.collection.writes == []
|
||||
assert "tags" not in database.collection.docs[0]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("change", [{"version": 2}, {"updatedAt": AT}, {"tags": None}, {"kind": "slice"}])
|
||||
def test_atomic_cas_rejects_change_after_narrow_preflight(change):
|
||||
doc = label()
|
||||
database = FakeDatabase([doc])
|
||||
database.collection.before_update = lambda collection: collection.docs[0].update(change)
|
||||
plan = plan_for([doc])
|
||||
with pytest.raises(ValueError, match="concurrent"):
|
||||
seed.apply_plan(database, plan, plan["planHash"])
|
||||
assert "history" not in database.collection.docs[0]
|
||||
assert database.collection.docs[0].get("tags") is None
|
||||
|
||||
|
||||
def test_pipeline_uses_current_server_fields_even_without_versioned_edit():
|
||||
doc = label()
|
||||
database = FakeDatabase([doc])
|
||||
database.collection.before_update = lambda collection: collection.docs[0].update(description="new synthetic server text")
|
||||
plan = plan_for([doc])
|
||||
seed.apply_plan(database, plan, plan["planHash"])
|
||||
assert database.collection.docs[0]["history"][-1]["description"] == "new synthetic server text"
|
||||
|
||||
|
||||
def test_confirmation_and_scope_fail_before_database_reads():
|
||||
database = FakeDatabase([label()])
|
||||
plan = plan_for([label()])
|
||||
with pytest.raises(ValueError, match="hash"):
|
||||
seed.apply_plan(database, plan, "wrong")
|
||||
tampered = {**plan, "updatedByUserId": "other"}
|
||||
with pytest.raises(ValueError, match="hash"):
|
||||
seed.apply_plan(database, tampered, plan["planHash"])
|
||||
database.name = "wrong-database"
|
||||
with pytest.raises(ValueError, match="database"):
|
||||
seed.apply_plan(database, plan, plan["planHash"])
|
||||
assert database.collection.reads == [] and database.collection.writes == []
|
||||
|
||||
|
||||
def test_cli_default_is_offline_and_saved_plan_scope_cannot_be_overridden(tmp_path, monkeypatch):
|
||||
path = tmp_path / "snapshot.json"
|
||||
path.write_text(json.dumps({"active-dictionary-tag-readiness": [metadata(label())]}))
|
||||
output = tmp_path / "plan.json"
|
||||
monkeypatch.delenv("DICTIONARY_TAGS_MONGO_URI", raising=False)
|
||||
assert seed.main(["--snapshot", str(path), "--database", "synthetic-dictionary-tags", "--organization", "synthetic-org",
|
||||
"--updated-by", "seed-test", "--at", AT, "--output", str(output)]) == 0
|
||||
assert json.loads(output.read_text()) == plan_for([label()])
|
||||
assert seed.main(["--plan", str(output)]) == 0
|
||||
with pytest.raises(SystemExit):
|
||||
seed.main(["--plan", str(output), "--database", "other"])
|
||||
with pytest.raises(SystemExit):
|
||||
seed.main(["--plan", str(output), "--apply"])
|
||||
|
||||
|
||||
@pytest.mark.skipif(not os.environ.get("DICTIONARY_TAGS_TEST_MONGO_URI"), reason="requires isolated synthetic Mongo on 127.0.0.1:27029")
|
||||
def test_real_mongo_pipeline_preserves_history_and_uses_only_metadata_projection():
|
||||
from pymongo import MongoClient, monitoring
|
||||
|
||||
uri = os.environ["DICTIONARY_TAGS_TEST_MONGO_URI"]
|
||||
parsed = urlsplit(uri)
|
||||
assert parsed.scheme == "mongodb" and parsed.hostname == "127.0.0.1" and parsed.port == 27029
|
||||
assert not parsed.username and not parsed.password and parsed.path in ("", "/")
|
||||
|
||||
class Reads(monitoring.CommandListener):
|
||||
def __init__(self):
|
||||
self.projections = []
|
||||
|
||||
def started(self, event):
|
||||
if event.command_name == "find":
|
||||
self.projections.append(dict(event.command.get("projection", {})))
|
||||
|
||||
def succeeded(self, event):
|
||||
pass
|
||||
|
||||
def failed(self, event):
|
||||
pass
|
||||
|
||||
listener = Reads()
|
||||
client = MongoClient(uri, serverSelectionTimeoutMS=5000, event_listeners=[listener])
|
||||
name = "dictionary_tags_seed_test_" + uuid.uuid4().hex
|
||||
database = client[name]
|
||||
old_history = [{"version": 1, "description": "original historical description", "tags": []}]
|
||||
docs = [label("history", history=old_history), label("empty", history=[], name="$server-field"), label("missing"), label("null", history=None)]
|
||||
try:
|
||||
database.annotationLabels.insert_many(copy.deepcopy(docs))
|
||||
plan = plan_for(docs, database=name)
|
||||
assert seed.apply_plan(database, plan, plan["planHash"]) == {"updated": 4, "alreadyApplied": 0}
|
||||
assert listener.projections == [seed.PROJECTION]
|
||||
# Full reads here are exclusively test assertions on our synthetic DB.
|
||||
results = {doc["id"]: doc for doc in database.annotationLabels.find({})}
|
||||
assert results["history"]["history"][:-1] == old_history
|
||||
for original in docs:
|
||||
result = results[original["id"]]
|
||||
assert result["version"] == 2 and result["tags"] == [original["name"]]
|
||||
assert result["history"][-1]["version"] == 2
|
||||
assert result["history"][-1]["tags"] == [original["name"]]
|
||||
for field in ("description", "example", "aliases", "code", "category", "requiresNote", "color"):
|
||||
assert result[field] == result["history"][-1][field] == original[field]
|
||||
if original["id"] != "history":
|
||||
assert len(result["history"]) == 2
|
||||
assert result["history"][0]["version"] == 1 and result["history"][0]["tags"] == []
|
||||
listener.projections.clear()
|
||||
assert seed.apply_plan(database, plan, plan["planHash"]) == {"updated": 0, "alreadyApplied": 4}
|
||||
assert listener.projections == [seed.PROJECTION]
|
||||
finally:
|
||||
client.drop_database(name)
|
||||
client.close()
|
||||
Loading…
x
Reference in New Issue
Block a user