⚡ Parallelize project saves while fencing scope mutations

Same-project saves queued behind one project lock. Add default-off record stripes while deletion, authorization and claim operations retain exclusive protection across all stripes. Require a complete barrier rollout before striped activation. Verify local Mongo deletion, revocation, crash recovery and two 100-user HTTP runs with save p95 below 400ms.
This commit is contained in:
d-robotics 2026-10-17 01:32:16 +08:00
parent 04e872ac85
commit 94a840c2ad
14 changed files with 905 additions and 164 deletions

View File

@ -12,7 +12,13 @@ Production and BOS verification remain read-only. No customer writes, deployment
- Source-backed revisions, immutable per-run source pins, independent durable validation queue, frozen acceptance playback, compatible directory export and separately gated JSON v2 export are implemented behind a default-off source flag. See ADR 0010.
- Candidate resource profile: 4 API replicas, 2 x 8 revision slots, 2 x 1 validation slots; steady limits 24 CPU / 48 GiB. Offline plan renderer guards image/config identity and remains blocked on missing candidate images and capacity evidence.
- Local 100-user HTTP/Mongo test completed all 100 tasks with 15-minute index/sensor cardinality; draft-save p95 approximately 2 seconds fails the 750ms acceptance threshold. Existing project deletion/authorization mutex is unchanged. Isolated striped-lock prototype reproduces unsafe mixed-version behavior; production implementation awaits explicit authorization after automatic review rejection.
- Initial local 100-user HTTP/Mongo test completed all tasks with 15-minute index/sensor cardinality, but draft-save p95 approximately 2 seconds failed the 750ms threshold. The user then explicitly authorized local project-lock implementation and full verification. Two fixed-version runs now complete 100/100 with draft-save p95 389/308ms and confirmation p95 591/655ms. ADR 0011 documents the implemented default-off protocol, bridge rollout and limits; the isolated prototype is replaced by shipping-path Mongo regression tests.
- Go targeted/race tests passed. Full Go baseline gate reports 70 existing failures, no new test names; frontend baseline gate reports 13 existing failures, no new test names. Type checking and builds passed. Python source-index/legacy package tests: 26 passed, 1 skipped; release tools: 32 passed.
- Authenticated local browser replica uses synthetic HEVC, an isolated acceptance reviewer and in-memory records. Found and fixed missed initial media-readiness seek; a behavior regression uses the actual transport hook with controllable readiness.
- Production, BOS and resource state unchanged; final release/100-browser acceptance is outstanding. No push or PR submitted.
## Project lock verification
The legacy implementation failed the deterministic different-record concurrency test. The replacement passes guarded-write overlap, same record/hash collision exclusion, cross-project isolation, exclusive nested writes, version conflicts, default legacy compatibility, bridge/new protocol interoperability, supplier cascade preservation and stale-write rejection, changed deletion-scope refusal, authorization drain inventory and post-revocation claim refusal, cancelled partial acquisition, and real subprocess crash recovery with stale-owner release fencing. The original account mutation lease implementation is reused. Go race detection passes.
Activation remains local-only: all Mongo writers must run barrier-capable binaries before any striped writer starts. The offline planner defaults to barrier and blocks striped activation for missing, stale, or incomplete Deployment rollouts. Pods/Jobs/CronJobs/external writers still require explicit inventory review before deployment approval.

View File

@ -39,4 +39,4 @@
本地带会话认证的终审浏览器使用合成 HEVC 验证 Movi 挂载、切片源偏移、逐帧、仅右相机与片尾停止。首次切片定位还覆盖了媒体就绪事件丢失:定位队列在就绪轮询中重新派发未完成目标,无需额外用户按键;行为测试覆盖该恢复路径。
保存 p95 约 2 秒,未达到本次本地 750ms 门槛。根因是同项目写保护锁串行;该保护覆盖删除及权限变化,当前未改动。分区锁仅有隔离 `_test.go` 原型,混用旧删除进程和新分区写入已复现竞态。其正式实现与上线激活须另行授权、验证。
首轮保存 p95 约 2 秒,根因是同项目写保护锁串行。2026-10-17 用户随后授权本地实施分区锁及完整回归,后续两轮本地保存 p95 为 389 / 308 ms;实现、两阶段升级要求与限制见 [ADR 0011](0011-project-record-write-stripes.md)。默认仍为旧协议,线上激活需另行授权。

View File

@ -0,0 +1,45 @@
# 0011:项目普通写入分区,范围变更独占
日期:2026-10-17
状态:用户已授权本地实施与完整验证;默认关闭,未部署或启用线上。
## 问题与决定
100 个已领取任务的合成用户同时保存,同一项目的全部记录共用 `supplier-project-write:<project>` Mongo 租约锁,草稿保存 p95 为 2,024 ms。锁同时保护供应商删除范围和授权撤销清单,不能直接去掉。
普通记录写入按 `(collection, id)` 的稳定 FNV-1a 散列,映射到每项目固定 32 个 Mongo 租约锁。相同记录、散列碰撞仍串行;不同项目的锁独立。允许集合仅为 `humanItems`、`annotationTasks`、`annotationIndexJobs`、`annotationTimelineIndexes`、`runs`、`sliceRevisions`、`humanEvents`。集合之外的写入以及显式范围操作继续独占。
删除冻结、删除执行、供应商授权、领取和批次转换都保留旧项目门锁,并拿齐全部 32 个分区锁后才运行。多个项目按项目 ID 排序,每项目按分区 0–31 获取;独占上下文中的嵌套写入复用已有保护,仍执行原有供应商删除检查。禁止从单分区升级独占或嵌套取得其他分区,避免锁顺序倒置。
复用已有账户变更租约的续期、超时和按 owner 释放机制,不引入新集合或迁移。项目锁文档数量至多新增每项目 32 个,按记录数不增长。独占操作要额外取得和释放 32 个锁,有数据库往返成本,因此普通事件和切片历史不能误走独占。
## 两阶段兼容协议
`DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE` 有三个值,空值默认 `legacy`,其他拼写阻止应用启动。
| 模式 | 普通允许集合写入 | 独占操作 |
| --- | --- | --- |
| `legacy` | 旧项目门锁 | 旧项目门锁 |
| `barrier` | 旧项目门锁 + 对应分区锁 | 旧项目门锁 + 全部分区锁 |
| `striped` | 对应分区锁 | 旧项目门锁 + 全部分区锁 |
1. 先将全部 Mongo 写入者升级到支持本协议的候选镜像,以 `barrier` 运行。该阶段仍能与旧门锁进程共存。
2. 核对全部 Deployment 的候选镜像、构建标记、滚动更新完成状态;另行审查 Pods、Jobs、CronJobs、外部脚本和其他直接 Mongo 写入者。没有旧门锁进程后才可启用 `striped`。`barrier` 和 `striped` 能混用。
3. **旧进程/`legacy` 与 `striped` 不兼容**:旧独占操作不拿分区锁,会与新普通写入并行。测试有意复现此危险,不能把“默认关闭”误当成任何混用都安全。
4. 回退先让所有 `striped` 写入者停止或切回 `barrier`,确认旧执行者及租约已经排空,再考虑 `legacy`。不能在仍有分区写入时直接滚回旧二进制。源引用修订另受 ADR 0010 的读写兼容约束。
离线容量工具默认生成 `barrier` 阶段计划;`--lock-stage striped` 检查 API、处理器、校验器、Worker、Coordinator 的模式、候选构建标记和完整滚动更新。Deployment 状态不等于完整进程清单,工具不会替代人工写入者清点,也不执行部署。
## 本地验证与限制
- 真实 Mongo 回归先在旧代码复现同项目不同记录无法并行,再验证分区写入可并行。
- 同记录/散列碰撞串行、跨项目隔离、旧/桥接版本兼容、版本冲突仅一方成功、嵌套独占复用、禁止锁升级、取消的部分独占不执行。
- 完整供应商级联删除契约通过:保留另一供应商及共享数据、拒绝旧后台写入复活;并发新记录使删除范围摘要失效,必须重取确认范围。
- 撤权等待已开始的写入完成后冻结 drain 清单,已开始任务纳入、未开始任务排除,撤权后新领取拒绝。
- 实际测试子进程拿齐全部锁后被终止,租约过期可回收,旧 owner 的释放无法匹配新 owner;并发竞态检测通过。
- 两轮 100 用户 HTTP/Mongo 回归均 100/100 完成。保存 p95 为 389 / 308 ms,确认受理 p95 为 591 / 655 ms。先前最小允许集合遗漏事件/Run/切片历史,曾造成 91/100 完成,扩大到上述经核对的单记录集合后修复;失败记录保留在本地证据中。
以上是单机本地 4 个 API handler、2×8 修订槽,15 分钟索引量级、预校验合成源、已领取任务的突发负载。没有覆盖真实 100 浏览器 HEVC 解码、生产网络/BOS、100 人同时新领取、完整送检/终审峰值或外部 Mongo 扩容。因此通过本地保存门槛不等于生产 100 人最终验收。
租约失效的安全边界沿用已有实现,不宣称能防止任意长进程暂停、网络分区或失效进程恢复后继续写的所有情形。未修改现有租约有效期、等待超时和数据库写入语义;这类故障需继续按现有恢复流程核实。

View File

@ -2,22 +2,110 @@
"schemaVersion": 1,
"profile": "fanyu-source-index-100-v1",
"namespace": "droboticflow-fanyu",
"budget": {"cpu": 36, "memoryBytes": 120000000000},
"deployments": {
"droboticflow-api": {"container": "api", "replicas": 4, "requests": {"cpu": "1", "memory": "2Gi"}, "limits": {"cpu": "2", "memory": "4Gi"}, "env": {"ANNOTATION_SOURCE_INDEX": "true"}},
"droboticflow-annotation-processor": {"container": "processor", "replicas": 2, "requests": {"cpu": "1", "memory": "2Gi"}, "limits": {"cpu": "2", "memory": "4Gi"}, "env": {"ANNOTATION_SOURCE_INDEX": "true", "ANNOTATION_PROCESSOR_ROLE": "revision", "ANNOTATION_PROCESSOR_CONCURRENCY": "8", "ANNOTATION_INDEX_CONCURRENCY": "1", "ANNOTATION_INDEX_BACKFILL": "0"}},
"droboticflow-annotation-validator": {"container": "processor", "replicas": 2, "requests": {"cpu": "2", "memory": "4Gi"}, "limits": {"cpu": "4", "memory": "8Gi"}, "env": {"ANNOTATION_SOURCE_INDEX": "true", "ANNOTATION_PROCESSOR_ROLE": "validation", "ANNOTATION_PROCESSOR_CONCURRENCY": "1", "ANNOTATION_INDEX_CONCURRENCY": "1", "ANNOTATION_INDEX_BACKFILL": "0"}},
"droboticflow-worker": {"container": "worker", "replicas": 2, "requests": {"cpu": "250m", "memory": "512Mi"}, "limits": {"cpu": "1", "memory": "2Gi"}, "env": {}},
"drosync-coordinator": {"container": "coordinator", "replicas": 2, "requests": {"cpu": "200m", "memory": "256Mi"}, "limits": {"cpu": "1", "memory": "2Gi"}, "env": {}}
"budget": {
"cpu": 36,
"memoryBytes": 120000000000
},
"deployments": {
"droboticflow-api": {
"container": "api",
"replicas": 4,
"requests": {
"cpu": "1",
"memory": "2Gi"
},
"limits": {
"cpu": "2",
"memory": "4Gi"
},
"env": {
"ANNOTATION_SOURCE_INDEX": "true",
"DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE": "striped"
}
},
"droboticflow-annotation-processor": {
"container": "processor",
"replicas": 2,
"requests": {
"cpu": "1",
"memory": "2Gi"
},
"limits": {
"cpu": "2",
"memory": "4Gi"
},
"env": {
"ANNOTATION_SOURCE_INDEX": "true",
"ANNOTATION_PROCESSOR_ROLE": "revision",
"ANNOTATION_PROCESSOR_CONCURRENCY": "8",
"ANNOTATION_INDEX_CONCURRENCY": "1",
"ANNOTATION_INDEX_BACKFILL": "0",
"DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE": "striped"
}
},
"droboticflow-annotation-validator": {
"container": "processor",
"replicas": 2,
"requests": {
"cpu": "2",
"memory": "4Gi"
},
"limits": {
"cpu": "4",
"memory": "8Gi"
},
"env": {
"ANNOTATION_SOURCE_INDEX": "true",
"ANNOTATION_PROCESSOR_ROLE": "validation",
"ANNOTATION_PROCESSOR_CONCURRENCY": "1",
"ANNOTATION_INDEX_CONCURRENCY": "1",
"ANNOTATION_INDEX_BACKFILL": "0",
"DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE": "striped"
}
},
"droboticflow-worker": {
"container": "worker",
"replicas": 2,
"requests": {
"cpu": "250m",
"memory": "512Mi"
},
"limits": {
"cpu": "1",
"memory": "2Gi"
},
"env": {
"DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE": "striped"
}
},
"drosync-coordinator": {
"container": "coordinator",
"replicas": 2,
"requests": {
"cpu": "200m",
"memory": "256Mi"
},
"limits": {
"cpu": "1",
"memory": "2Gi"
},
"env": {
"DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE": "striped"
}
}
},
"legacyExportBudget": {
"cpu": 6,
"memoryBytes": 12884901888,
"enforced": false
},
"legacyExportBudget": {"cpu": 6, "memoryBytes": 12884901888, "enforced": false},
"constraints": [
"Offline proposal only; no deployment authorization is inferred.",
"Six CPU cores and remaining memory stay unallocated after the proposed steady services and legacy-export budget.",
"Operator Job Pods have independent resource specifications. The export allowance is a planning reservation, not a namespace quota or implemented global admission limit.",
"Do not enable source-index output until API, revision workers, validators, workflow workers and the package operator all understand the new schemas.",
"The project write-lock mechanism remains unchanged; 100-user save latency acceptance is separate.",
"Mongo and BOS are external; this budget does not expand their capacity or network bandwidth.",
"The first validation of a source version creates five immutable internal BOS copies, then later revisions write JSON only. No historical customer-data backfill is enabled."
"The first validation of a source version creates five immutable internal BOS copies, then later revisions write JSON only. No historical customer-data backfill is enabled.",
"Local implementation only: upgrade every Mongo writer to barrier mode and finish all rollouts before enabling striped mode; old gate-only writers are incompatible."
]
}

View File

@ -1,6 +1,6 @@
# 源引用切片:100 人容量候选
本地候选,未部署;尚未达到“100 人操作不卡”的验收条件。
本地候选,未部署;100 人本地保存门槛已通过,尚未进行生产 100 浏览器最终容量验收。
## 资源与并行
@ -32,16 +32,24 @@
| 切片确认受理 | 2,128 ms |
| 异步处理状态查询 | 52 ms |
完成时间 p95 约 5.2 秒,包含每 2 秒轮询的观测延迟。保存门槛为 p95 ≤ 750 ms,目前红灯,不能因全部成功而宣布容量通过。同项目保存共用项目写锁,是主要排队来源;锁优化的隔离原型不进入生产程序。
旧锁完成时间 p95 约 5.2 秒,包含每 2 秒轮询的观测延迟。保存门槛为 p95 ≤ 750 ms;首轮因为同项目写锁串行而未通过。用户随后授权本地实施分区写入,结果如下:
| 指标 | 旧项目锁 | 分区第 1 轮 | 分区第 2 轮 |
| --- | --- | --- | --- |
| 成功完成 | 100/100 | 100/100 | 100/100 |
| 草稿保存 p95 | 2,024 ms | 389 ms | 308 ms |
| 确认受理 p95 | 2,128 ms | 591 ms | 655 ms |
两轮保存门槛均通过。分区模式默认关闭;删除、撤权、宕机及混用约束见 [ADR 0011](../adr/0011-project-record-write-stripes.md)。第一次仅放开少量集合时出现 91/100 完成的回归,已通过核对普通写入集合修复,不能复用那个中间版本。
线上仍需测真实 100 浏览器、HEVC 双目解码/跳帧、BOS 带宽、Mongo 延迟及高峰恢复。已审计源的双目约 2.6–4.6 Mbps/人,100 人约 260–460 Mbps 从 BOS 到客户端;不经过 API 不代表无带宽瓶颈。
## 变更与上线顺序
1. 本地完成兼容性、恢复、交付、权限回归和性能门槛;保留所有未通过项。
2. 构建并证明候选 commit 对应 API/processor、Worker 与独立 package 镜像摘要。全部组件支持新协议、源开关仍关闭。
2. 构建并证明候选 commit 对应 API/processor、Worker、Coordinator 与独立 package 镜像摘要。全部组件支持新协议、源开关仍关闭;所有 Mongo 写入进程先启用 `barrier` 协议。
3. 现有 `scripts/release.py` 仅支持 API/processor 镜像更新,保留原副本和配置。使用其 prepare/check/plan 流程;不得以此工具的旧单处理器策略冒充新资源计划通过。
4. 独立审查 Worker/package 更新、validator 新建和本表资源/config 变更。使用 `scripts/source_index_capacity_plan.py` 从只读 Deployment 快照生成候选补丁;工具只写本地文件,不调用集群。其检查镜像、候选 commit、100 人保存门槛,缺一则标记 blocked。
4. 独立审查 Worker/Coordinator/package 更新、validator 新建和本表资源/config 变更。使用 `scripts/source_index_capacity_plan.py` 从只读 Deployment 快照生成候选补丁;工具只写本地文件,不调用集群。默认生成 `barrier` 阶段,检查镜像、候选 commit、100 人保存门槛;`--lock-stage striped` 还检查全部五个 Deployment 的桥接模式与滚动更新完成状态。另行清点旧 Pods、Jobs、CronJobs、外部脚本等写入者,确认后才可开启分区并行。
5. 确认交付模式:目录保留物理舍弃排除契约;JSON v2 须另行批准下游读取完整源及范围协议,并显式开启新交付开关。未确认前不启用源修订。
6. 核对集群/context/namespace UID、在途任务和回退策略,展示包含源副本存储成本的最终计划;取得紧邻部署的用户授权后执行。生产核验默认只读。

View File

@ -264,6 +264,7 @@ func TestSourceIndex100ActiveOperatorsMongoHTTP(t *testing.T) {
summary[kind] = map[string]any{"requests": len(values), "p50Ms": percentile(values, .5), "p95Ms": percentile(values, .95), "maxMs": percentile(values, 1)}
}
report := map[string]any{"activeOperators": 100, "apiHandlerInstances": 4, "revisionReplicas": 2, "revisionConcurrencyPerReplica": 8, "sourceSecondsPerTask": 900, "sourceFramesPerCamera": 27000, "sourceIMURows": 1800000, "completed": len(completions), "wallSeconds": time.Since(began).Seconds(), "completionP95Ms": percentile(completions, .95), "requests": summary, "scope": "isolated local Mongo and real HTTP, 100 distinct run/source certificate identities with production-scale prevalidated synthetic indexes; excludes login, production BOS/network and 100 browser decoders"}
report["projectWriteLockMode"], _ = config.ProjectWriteLockMode()
b, _ := json.MarshalIndent(report, "", " ")
t.Log(string(b))
if out := os.Getenv("FANYU_SOURCE_INDEX_LOAD_REPORT"); out != "" {

View File

@ -57,6 +57,9 @@ type AppConfig struct {
}
func LoadValidated() (AppConfig, error) {
if _, err := ProjectWriteLockMode(); err != nil {
return AppConfig{}, err
}
workingDirectory, _ := os.Getwd()
port, err := parseIntEnv("PORT", 3001, 1, 65535)
if err != nil {
@ -189,6 +192,16 @@ func LoadValidated() (AppConfig, error) {
}, nil
}
// Protocol changes require every Mongo writer to be upgraded to barrier before
// any process enables striped. This validates spelling, not fleet readiness.
func ProjectWriteLockMode() (string, error) {
mode := getenvTrimmed("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "legacy")
if mode != "legacy" && mode != "barrier" && mode != "striped" {
return "", fmt.Errorf("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE must be legacy, barrier or striped")
}
return mode, nil
}
func firstNonEmptyTrimmed(values ...string) string {
for _, value := range values {
if trimmed := strings.TrimSpace(value); trimmed != "" {

View File

@ -6,6 +6,7 @@ import (
)
var configEnvKeys = []string{
"DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE",
"PORT",
"WORKER_PORT",
"MONGO_URI",
@ -50,6 +51,27 @@ var configEnvKeys = []string{
"DROBOTICFLOW_IMAGE",
}
func TestProjectWriteLockModeValidation(t *testing.T) {
for _, value := range []string{"", "legacy", "barrier", "striped"} {
t.Run("valid-"+value, func(t *testing.T) {
clearConfigEnv(t)
setRequiredConfigEnv(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", value)
if _, e := LoadValidated(); e != nil {
t.Fatal(e)
}
})
}
t.Run("typo-blocks-startup", func(t *testing.T) {
clearConfigEnv(t)
setRequiredConfigEnv(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "stripe")
if _, e := LoadValidated(); e == nil {
t.Fatal("invalid write lock protocol accepted")
}
})
}
func clearConfigEnv(t *testing.T) {
t.Helper()
for _, key := range configEnvKeys {

View File

@ -1,119 +0,0 @@
package store
// ISOLATED DESIGN EXPERIMENT ONLY. These functions are in a _test.go file and
// are never compiled into API/Worker binaries. No production lock is modified.
import (
"context"
"fmt"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"os"
"testing"
"time"
)
func prototypeProjectStripe(s *MongoStore, ctx context.Context, stripe int, fn func(context.Context) (any, error)) (any, error) {
return s.withAccountMutationLock(ctx, fmt.Sprintf("prototype-project-record:%02d", stripe), fn)
}
func prototypeProjectExclusive(s *MongoStore, ctx context.Context, fn func(context.Context) (any, error)) (any, error) {
var acquire func(context.Context, int) (any, error)
acquire = func(ctx context.Context, stripe int) (any, error) {
if stripe == 32 {
return fn(ctx)
}
return prototypeProjectStripe(s, ctx, stripe, func(ctx context.Context) (any, error) { return acquire(ctx, stripe+1) })
}
return s.withAccountMutationLock(ctx, "prototype-project-gate", func(ctx context.Context) (any, error) { return acquire(ctx, 0) })
}
func TestProjectWriteStripesPrototypeLocalMongo(t *testing.T) {
if os.Getenv("FANYU_PROJECT_LOCK_PROTOTYPE") != "1" {
t.Skip("opt-in isolated synchronization prototype; no production behavior change")
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
client, e := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://127.0.0.1:27018/?directConnection=true").SetServerSelectionTimeout(3*time.Second))
if e != nil {
t.Fatal(e)
}
defer client.Disconnect(context.Background())
db := client.Database(fmt.Sprintf("project_lock_prototype_%d", time.Now().UnixNano()))
defer db.Drop(context.Background())
s := NewMongoStore(db)
aEntered, bEntered, release := make(chan struct{}), make(chan struct{}), make(chan struct{})
done := make(chan error, 2)
go func() {
_, e := prototypeProjectStripe(s, ctx, 2, func(context.Context) (any, error) { close(aEntered); <-release; return nil, nil })
done <- e
}()
select {
case <-aEntered:
case <-ctx.Done():
t.Fatal("first writer did not enter")
}
go func() {
_, e := prototypeProjectStripe(s, ctx, 3, func(context.Context) (any, error) { close(bEntered); <-release; return nil, nil })
done <- e
}()
select {
case <-bEntered:
case <-time.After(time.Second):
t.Fatal("independent ordinary writers serialized")
}
exclusiveEntered, exclusiveRelease, exclusiveDone := make(chan struct{}), make(chan struct{}), make(chan error, 1)
go func() {
_, e := prototypeProjectExclusive(s, ctx, func(context.Context) (any, error) { close(exclusiveEntered); <-exclusiveRelease; return nil, nil })
exclusiveDone <- e
}()
select {
case <-exclusiveEntered:
t.Fatal("exclusive operation entered while ordinary writes were active")
case <-time.After(100 * time.Millisecond):
}
close(release)
for n := 0; n < 2; n++ {
if e := <-done; e != nil {
t.Fatal(e)
}
}
select {
case <-exclusiveEntered:
case <-ctx.Done():
t.Fatal("exclusive operation never entered")
}
lateEntered, lateDone := make(chan struct{}), make(chan error, 1)
go func() {
_, e := prototypeProjectStripe(s, ctx, 3, func(context.Context) (any, error) { close(lateEntered); return nil, nil })
lateDone <- e
}()
select {
case <-lateEntered:
t.Fatal("ordinary write entered during exclusive operation")
case <-time.After(100 * time.Millisecond):
}
close(exclusiveRelease)
if e := <-exclusiveDone; e != nil {
t.Fatal(e)
}
if e := <-lateDone; e != nil {
t.Fatal(e)
}
// Deliberately demonstrate the compatibility hazard: an old gate-only
// exclusive implementation does not block stripe writers. Rollout therefore
// requires every exclusive writer to be upgraded before enabling the path.
legacyHeld, legacyRelease, legacyDone := make(chan struct{}), make(chan struct{}), make(chan error, 1)
go func() {
_, e := s.withAccountMutationLock(ctx, "prototype-project-gate", func(context.Context) (any, error) { close(legacyHeld); <-legacyRelease; return nil, nil })
legacyDone <- e
}()
<-legacyHeld
_, e = prototypeProjectStripe(s, ctx, 3, func(context.Context) (any, error) { return "old-gate-does-not-fence-stripes", nil })
if e != nil {
t.Fatal(e)
}
close(legacyRelease)
if e = <-legacyDone; e != nil {
t.Fatal(e)
}
t.Log("parallel stripe operations and new exclusive fencing passed; old-exclusive/new-stripe overlap reproduced, so mixed-version enablement is unsafe")
}

View File

@ -3,7 +3,6 @@ package store
import (
"context"
"encoding/json"
"fmt"
"strings"
"droboticflow/internal/domain"
@ -13,27 +12,6 @@ import (
type supplierProjectLockKey struct{}
func (s *MongoStore) withSupplierProjectLocks(ctx context.Context, ids []string, fn func(context.Context) (any, error)) (any, error) {
if len(ids) == 0 {
return fn(ctx)
}
id := ids[0]
held, _ := ctx.Value(supplierProjectLockKey{}).(map[string]bool)
if id == "" || held[id] {
return s.withSupplierProjectLocks(ctx, ids[1:], fn)
}
if mongo.SessionFromContext(ctx) != nil {
return nil, fmt.Errorf("supplier write lock must precede transaction")
}
return s.withAccountMutationLock(ctx, "supplier-project-write:"+id, func(ctx context.Context) (any, error) {
next := map[string]bool{}
for k, v := range held {
next[k] = v
}
next[id] = true
return s.withSupplierProjectLocks(context.WithValue(ctx, supplierProjectLockKey{}, next), ids[1:], fn)
})
}
func supplierRecordRelated(p domain.SupplierDeletionPlan, col, id string, d bson.M) bool {
if planHas(p, col, id) || docSupplier(d) == p.SupplierID {
return true
@ -209,7 +187,7 @@ func supplierGuardedMongoWrite(ctx context.Context, col *mongo.Collection, id st
}
}
}
_, e := s.withSupplierProjectLocks(ctx, []string{pid}, func(ctx context.Context) (any, error) {
_, e := s.withSupplierRecordWriteLock(ctx, pid, col.Name(), id, func(ctx context.Context) (any, error) {
jobs, e := s.supplierRecordJobs(ctx, col.Name(), id, d)
if e != nil {
return nil, e

View File

@ -0,0 +1,142 @@
package store
import (
"context"
"droboticflow/internal/config"
"fmt"
"hash/fnv"
"sort"
"go.mongodb.org/mongo-driver/mongo"
)
const supplierProjectStripeCount = 32
type supplierRecordStripeKey struct{}
// legacy preserves the deployed protocol. barrier is the bridge release:
// ordinary writes retain the legacy gate and also acquire their record stripe;
// exclusive operations acquire the gate and all stripes. Only after EVERY
// process that writes these records is at least barrier-capable may striped
// ordinary writers omit the gate. Never mix striped with an old/legacy process.
func supplierProjectLockMode() (string, error) {
return config.ProjectWriteLockMode()
}
func supplierRecordStripe(collection, id string) int {
h := fnv.New32a()
_, _ = h.Write([]byte(collection + "\x00" + id))
return int(h.Sum32() % supplierProjectStripeCount)
}
func supplierRecordStripeLockID(project string, stripe int) string {
return fmt.Sprintf("supplier-project-record:%s:%02d", project, stripe)
}
func stripedSupplierCollection(collection string) bool {
switch collection {
case "humanItems", "annotationTasks", "annotationIndexJobs", "annotationTimelineIndexes",
"runs", "sliceRevisions", "humanEvents":
return true
}
return false
}
func (s *MongoStore) withSupplierRecordWriteLock(ctx context.Context, project, collection, id string, fn func(context.Context) (any, error)) (any, error) {
exclusive, _ := ctx.Value(supplierProjectLockKey{}).(map[string]bool)
if project == "" || exclusive[project] {
return fn(ctx)
}
mode, e := supplierProjectLockMode()
if e != nil {
return nil, e
}
if mode == "legacy" || !stripedSupplierCollection(collection) {
return s.withSupplierProjectLocks(ctx, []string{project}, fn)
}
stripe := supplierRecordStripe(collection, id)
held, _ := ctx.Value(supplierRecordStripeKey{}).(map[string]int)
if previous, ok := held[project]; ok {
if previous != stripe {
return nil, fmt.Errorf("nested supplier writes cannot acquire another stripe")
}
return fn(ctx)
}
if len(held) != 0 {
return nil, fmt.Errorf("nested supplier writes cannot acquire another project stripe")
}
if mongo.SessionFromContext(ctx) != nil {
return nil, fmt.Errorf("supplier write lock must precede transaction")
}
acquire := func(ctx context.Context) (any, error) {
return s.withAccountMutationLock(ctx, supplierRecordStripeLockID(project, stripe), func(ctx context.Context) (any, error) {
next := make(map[string]int, len(held)+1)
for k, v := range held {
next[k] = v
}
next[project] = stripe
return fn(context.WithValue(ctx, supplierRecordStripeKey{}, next))
})
}
if mode == "barrier" {
return s.withAccountMutationLock(ctx, "supplier-project-write:"+project, acquire)
}
return acquire(ctx)
}
// All broad mutations retain the old gate and, in the new protocol, hold all
// stripes before examining/changing scope. Multi-project locks use global
// project order, then ascending stripe order. Context reentry is exclusive-only.
func (s *MongoStore) withSupplierProjectLocks(ctx context.Context, ids []string, fn func(context.Context) (any, error)) (any, error) {
mode, e := supplierProjectLockMode()
if e != nil {
return nil, e
}
ordered := append([]string(nil), ids...)
sort.Strings(ordered)
var acquire func(context.Context, int) (any, error)
acquire = func(ctx context.Context, index int) (any, error) {
if index == len(ordered) {
return fn(ctx)
}
id := ordered[index]
held, _ := ctx.Value(supplierProjectLockKey{}).(map[string]bool)
if id == "" || held[id] {
return acquire(ctx, index+1)
}
stripes, _ := ctx.Value(supplierRecordStripeKey{}).(map[string]int)
if len(stripes) != 0 {
return nil, fmt.Errorf("cannot upgrade a supplier record stripe to an exclusive lock")
}
for previous := range held {
if previous > id {
return nil, fmt.Errorf("supplier project locks must follow global order")
}
}
if mongo.SessionFromContext(ctx) != nil {
return nil, fmt.Errorf("supplier write lock must precede transaction")
}
return s.withAccountMutationLock(ctx, "supplier-project-write:"+id, func(ctx context.Context) (any, error) {
enter := func(ctx context.Context) (any, error) {
next := make(map[string]bool, len(held)+1)
for k, v := range held {
next[k] = v
}
next[id] = true
return acquire(context.WithValue(ctx, supplierProjectLockKey{}, next), index+1)
}
if mode == "legacy" {
return enter(ctx)
}
var lockStripe func(context.Context, int) (any, error)
lockStripe = func(ctx context.Context, stripe int) (any, error) {
if stripe == supplierProjectStripeCount {
return enter(ctx)
}
return s.withAccountMutationLock(ctx, supplierRecordStripeLockID(id, stripe), func(ctx context.Context) (any, error) { return lockStripe(ctx, stripe+1) })
}
return lockStripe(ctx, 0)
})
}
return acquire(ctx, 0)
}

View File

@ -0,0 +1,514 @@
package store
import (
"bufio"
"context"
"errors"
"fmt"
"os"
"os/exec"
"strings"
"sync"
"testing"
"time"
"droboticflow/internal/domain"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func projectStripeTestStore(t *testing.T) (*MongoStore, context.Context) {
t.Helper()
if os.Getenv("FANYU_PROJECT_WRITE_LOCK_TEST") != "1" {
t.Skip("opt-in isolated local Mongo project lock contract")
}
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
t.Cleanup(cancel)
client, e := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://127.0.0.1:27018/?directConnection=true").SetServerSelectionTimeout(3*time.Second))
if e != nil {
t.Fatal(e)
}
db := client.Database(fmt.Sprintf("project_stripes_test_%d", time.Now().UnixNano()))
t.Cleanup(func() {
cleanup, stop := context.WithTimeout(context.Background(), 10*time.Second)
defer stop()
if e := db.Drop(cleanup); e != nil {
t.Error(e)
}
client.Disconnect(cleanup)
})
return NewMongoStore(db), ctx
}
func waitProjectTest(t *testing.T, ctx context.Context, ch <-chan struct{}) {
t.Helper()
select {
case <-ch:
case <-ctx.Done():
t.Fatal(ctx.Err())
}
}
func waitProjectGate(t *testing.T, ctx context.Context, s *MongoStore, key string) {
t.Helper()
for ctx.Err() == nil {
var row bson.M
if e := s.db.Collection("accountMutationLocks").FindOne(ctx, bson.M{"_id": key, "locked": true}).Decode(&row); e == nil {
return
}
time.Sleep(time.Millisecond)
}
t.Fatal(ctx.Err())
}
func TestProjectWriteStripesExclusiveWaitsAndBlocksLateWriters(t *testing.T) {
s, ctx := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "striped")
id := ""
for i := 0; ; i++ {
id = fmt.Sprint(i)
if supplierRecordStripe("humanItems", id) == 31 {
break
}
}
writer, release, exclusive, releaseExclusive, late := make(chan struct{}), make(chan struct{}), make(chan struct{}), make(chan struct{}), make(chan struct{})
var writerOnce, exclusiveOnce sync.Once
releaseWriter := func() { writerOnce.Do(func() { close(release) }) }
releaseBarrier := func() { exclusiveOnce.Do(func() { close(releaseExclusive) }) }
defer releaseWriter()
defer releaseBarrier()
done := make(chan error, 3)
go func() {
_, e := s.withSupplierRecordWriteLock(ctx, "p", "humanItems", id, func(context.Context) (any, error) { close(writer); <-release; return nil, nil })
done <- e
}()
waitProjectTest(t, ctx, writer)
go func() {
_, e := s.withSupplierProjectLocks(ctx, []string{"p"}, func(ctx context.Context) (any, error) {
close(exclusive)
// Guarded nested writes reuse the exclusive context, including a
// different collection, without a deadlock or skipped deletion checks.
e := supplierGuardedMongoWrite(ctx, s.annotationTasks, "nested", bson.M{"id": "nested", "projectId": "p"}, func(context.Context, bson.M) error { return nil })
<-releaseExclusive
return nil, e
})
done <- e
}()
waitProjectGate(t, ctx, s, supplierRecordStripeLockID("p", 0))
select {
case <-exclusive:
t.Fatal("exclusive operation passed active writer")
default:
}
releaseWriter()
waitProjectTest(t, ctx, exclusive)
go func() {
_, e := s.withSupplierRecordWriteLock(ctx, "p", "humanItems", "late", func(context.Context) (any, error) { close(late); return nil, nil })
done <- e
}()
select {
case <-late:
t.Fatal("late writer passed exclusive operation")
case <-time.After(50 * time.Millisecond):
}
releaseBarrier()
waitProjectTest(t, ctx, late)
for i := 0; i < 3; i++ {
if e := <-done; e != nil {
t.Fatal(e)
}
}
}
func TestProjectWriteStripesLegacyAndBarrierCompatibility(t *testing.T) {
for _, mode := range []string{"legacy", "barrier"} {
t.Run(mode, func(t *testing.T) {
s, ctx := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", mode)
entered, release := make(chan struct{}), make(chan struct{})
done := make(chan error, 1)
go func() {
_, e := s.withSupplierProjectLocks(ctx, []string{"p"}, func(context.Context) (any, error) { close(entered); <-release; return nil, nil })
done <- e
}()
waitProjectTest(t, ctx, entered)
// Changing the test process mode models a second binary's independent
// config. The first operation already captured its protocol above.
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "striped")
probe, stop := context.WithTimeout(ctx, 100*time.Millisecond)
_, e := s.withSupplierRecordWriteLock(probe, "p", "humanItems", "record", func(context.Context) (any, error) { return "entered", nil })
stop()
close(release)
if err := <-done; err != nil {
t.Fatal(err)
}
if mode == "legacy" && e != nil {
t.Fatal("legacy/new overlap hazard was not reproduced", e)
}
if mode == "barrier" && e == nil {
t.Fatal("bridge binary failed to exclude a striped writer")
}
})
}
}
func TestProjectWriteStripesVersionConflictAndScopeIsolation(t *testing.T) {
s, ctx := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "striped")
item := domain.HumanItemRecord{ID: "record", OrganizationID: "org", ProjectID: "p", Version: 3, Status: domain.HumanItemInProgress}
if e := s.UpsertHumanItem(ctx, item); e != nil {
t.Fatal(e)
}
results := make(chan bool, 2)
errs := make(chan error, 2)
for i := 0; i < 2; i++ {
go func() {
next := item
next.Version = 4
ok, e := s.UpdateHumanItemVersioned(ctx, next, 3)
results <- ok
errs <- e
}()
}
count := 0
for i := 0; i < 2; i++ {
if <-results {
count++
}
if e := <-errs; e != nil {
t.Fatal(e)
}
}
if count != 1 {
t.Fatal("concurrent stale versions both wrote or both failed", count)
}
if e := s.db.Collection("accountMutationLocks").FindOne(ctx, bson.M{"_id": "supplier-project-write:p", "locked": true}).Err(); e != mongo.ErrNoDocuments {
t.Fatal("ordinary writer unexpectedly holds project gate", e)
}
_, e := s.withSupplierProjectLocks(ctx, []string{"other", "p", "other"}, func(ctx context.Context) (any, error) {
_, e := s.withSupplierProjectLocks(ctx, []string{"p"}, func(context.Context) (any, error) { return nil, nil })
return nil, e
})
if e != nil {
t.Fatal("sorted/deduplicated multi-project reentry", e)
}
_, e = s.withSupplierRecordWriteLock(ctx, "p", "humanItems", "record", func(ctx context.Context) (any, error) {
return s.withSupplierProjectLocks(ctx, []string{"p"}, func(context.Context) (any, error) { t.Error("unsafe lock upgrade entered"); return nil, nil })
})
if e == nil {
t.Fatal("stripe-to-exclusive upgrade should fail before deadlocking")
}
}
func TestProjectWriteStripesCollisionAndBridgeWritersSerialize(t *testing.T) {
for _, mode := range []string{"striped", "barrier"} {
t.Run(mode, func(t *testing.T) {
s, ctx := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", mode)
collision := ""
for n := 0; ; n++ {
collision = fmt.Sprintf("collision-%d", n)
if supplierRecordStripe("humanItems", collision) == supplierRecordStripe("humanItems", "record") {
break
}
}
entered, release := make(chan struct{}), make(chan struct{})
var once sync.Once
releaseWriter := func() { once.Do(func() { close(release) }) }
defer releaseWriter()
done := make(chan error, 1)
go func() {
_, e := s.withSupplierRecordWriteLock(ctx, "p", "humanItems", "record", func(context.Context) (any, error) { close(entered); <-release; return nil, nil })
done <- e
}()
waitProjectTest(t, ctx, entered)
// Model a newer process while the holder remains in bridge mode.
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "striped")
for _, id := range []string{"record", collision} {
probe, stop := context.WithTimeout(ctx, 60*time.Millisecond)
_, e := s.withSupplierRecordWriteLock(probe, "p", "humanItems", id, func(context.Context) (any, error) { t.Error("same record/stripe overlapped holder"); return nil, nil })
stop()
if e == nil {
t.Fatal("conflicting stripe not excluded")
}
}
// Project identity is part of the durable key, so an unrelated project
// with the same record/hash does not wait for this project's holder.
if _, e := s.withSupplierRecordWriteLock(ctx, "other", "humanItems", "record", func(context.Context) (any, error) { return nil, nil }); e != nil {
t.Fatal(e)
}
releaseWriter()
if e := <-done; e != nil {
t.Fatal(e)
}
})
}
}
func TestProjectWriteStripesDefaultKeepsLegacyProtocol(t *testing.T) {
s, ctx := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "")
if e := supplierGuardedMongoWrite(ctx, s.humanItems, "record", bson.M{"id": "record", "projectId": "p"}, func(context.Context, bson.M) error { return nil }); e != nil {
t.Fatal(e)
}
n, e := s.db.Collection("accountMutationLocks").CountDocuments(ctx, bson.M{"_id": bson.M{"$regex": "^supplier-project-record:"}})
if e != nil || n != 0 {
t.Fatal("default path changed protocol", n, e)
}
if e := s.db.Collection("accountMutationLocks").FindOne(ctx, bson.M{"_id": "supplier-project-write:p", "locked": false}).Err(); e != nil {
t.Fatal("legacy gate missing", e)
}
}
func TestProjectWriteStripesSupplierCascadeContract(t *testing.T) {
s, _ := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "striped")
supplierCascadeContract(t, s)
}
func TestProjectWriteStripesDeletionRechecksScopeAfterConcurrentWrite(t *testing.T) {
s, ctx := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "striped")
org, project := seedSupplierCascade(t, s)
plan, e := s.PreviewSupplierDeletion(ctx, org, "A")
if e != nil {
t.Fatal(e)
}
plan.ScopeDigest = plan.Digest
entered, release := make(chan struct{}), make(chan struct{})
writer := make(chan error, 1)
go func() {
writer <- supplierGuardedMongoWrite(ctx, s.humanItems, "late-new", bson.M{"id": "late-new", "organizationId": org, "projectId": project, "supplierId": "A", "runId": "run-A", "batchId": "batch-A"}, func(ctx context.Context, d bson.M) error {
close(entered)
<-release
_, e := s.humanItems.InsertOne(ctx, d)
return e
})
}()
waitProjectTest(t, ctx, entered)
deletion := make(chan error, 1)
go func() {
_, e := s.BeginSupplierDeletion(ctx, domain.SupplierDeletionTask{ID: supplierTaskID("A"), OrganizationID: org, Plan: plan, Status: "prepared", Phase: "freezing"})
deletion <- e
}()
waitProjectGate(t, ctx, s, "supplier-project-write:"+project)
close(release)
if e := <-writer; e != nil {
t.Fatal(e)
}
if e := <-deletion; !errors.Is(e, ErrSupplierDeletionScopeChanged) {
t.Fatal("stale deletion inventory was accepted", e)
}
supplier, e := s.GetSupplier(ctx, "A")
if e != nil || supplier.Status != "active" {
t.Fatal("failed freeze partially deleted supplier", supplier, e)
}
}
func TestProjectWriteStripesRevocationFreezesAfterActiveWrite(t *testing.T) {
s, ctx := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "striped")
org, project, supplier := "org", "project", "supplier"
for _, e := range []error{
s.UpsertProject(ctx, domain.ProjectRecord{ID: project, OrganizationID: org}),
s.UpsertSupplier(ctx, domain.SupplierRecord{ID: supplier, OrganizationID: org, Status: "active"}),
s.UpsertUser(ctx, domain.UserRecord{ID: "user", Phone: "13900009231", OrganizationID: org, SupplierID: supplier, Role: domain.UserRoleAnnotator, Status: domain.UserStatusActive}),
s.UpsertHumanBatch(ctx, domain.HumanBatchRecord{ID: "batch", OrganizationID: org, ProjectID: project, PluginID: "annotation-qa", ProcessingProfile: domain.ProcessingProfileRawHuman, Stages: domain.DefaultFanyuV58HumanStages()}),
} {
if e != nil {
t.Fatal(e)
}
}
if _, e := s.SetSupplierProjectAuthorization(ctx, org, project, supplier, true, "admin"); e != nil {
t.Fatal(e)
}
for _, id := range []string{"started", "fresh"} {
if e := s.UpsertHumanItem(ctx, domain.HumanItemRecord{ID: id, OrganizationID: org, ProjectID: project, SupplierID: supplier, PluginID: "annotation-qa", LegacyTaskID: id + "-task", BatchID: "batch", Status: domain.HumanItemReady, Version: 1}); e != nil {
t.Fatal(e)
}
}
entered, release := make(chan struct{}), make(chan struct{})
writeDone := make(chan error, 1)
go func() {
writeDone <- supplierGuardedMongoWrite(ctx, s.humanItems, "started", bson.M{"id": "started", "organizationId": org, "projectId": project, "supplierId": supplier}, func(ctx context.Context, _ bson.M) error {
close(entered)
<-release
_, e := s.humanItems.UpdateOne(ctx, bson.M{"id": "started"}, bson.M{"$set": bson.M{"status": domain.HumanItemClaimed, "assignedUserId": "user"}})
return e
})
}()
waitProjectTest(t, ctx, entered)
grantDone := make(chan *domain.SupplierProjectAuthorization, 1)
grantErr := make(chan error, 1)
go func() {
grant, e := s.SetSupplierProjectAuthorization(ctx, org, project, supplier, false, "admin")
grantDone <- grant
grantErr <- e
}()
waitProjectGate(t, ctx, s, "supplier-project-write:"+project)
close(release)
if e := <-writeDone; e != nil {
t.Fatal(e)
}
grant := <-grantDone
if e := <-grantErr; e != nil {
t.Fatal(e)
}
if len(grant.DrainTaskIDs) != 1 || grant.DrainTaskIDs[0] != "started-task" {
t.Fatal("revocation missed active work or admitted unstarted work", grant)
}
fresh, e := s.GetHumanItem(ctx, "fresh")
if e != nil {
t.Fatal(e)
}
fresh.Status = domain.HumanItemClaimed
fresh.AssignedUserID = "user"
fresh.Version++
if ok, e := s.ClaimHumanItemVersioned(ctx, *fresh, 1, domain.UserRoleAnnotator); ok || !errors.Is(e, ErrHumanClaimConflict) {
t.Fatal("fresh claim passed revoked authorization", ok, e)
}
}
func TestProjectWriteStripesCrashChild(t *testing.T) {
dbName := os.Getenv("FANYU_STRIPE_CRASH_CHILD_DB")
if dbName == "" {
t.Skip("subprocess helper")
}
if !strings.HasPrefix(dbName, "project_stripes_test_") {
t.Fatal("isolated DB required")
}
ctx := context.Background()
client, e := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://127.0.0.1:27018/?directConnection=true"))
if e != nil {
t.Fatal(e)
}
s := NewMongoStore(client.Database(dbName))
_, e = s.withSupplierProjectLocks(ctx, []string{"crash-project"}, func(context.Context) (any, error) { fmt.Println("STRIPE_CRASH_LOCKS_HELD"); select {} })
t.Fatal("crash helper unexpectedly returned", e)
}
func TestProjectWriteStripesCrashRecoveryAndStaleRelease(t *testing.T) {
s, ctx := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "striped")
t.Setenv("ACCOUNT_MUTATION_LOCK_LEASE_SECONDS", "1")
exe, e := os.Executable()
if e != nil {
t.Fatal(e)
}
child := exec.CommandContext(ctx, exe, "-test.run=^TestProjectWriteStripesCrashChild$", "-test.v")
child.Env = append(os.Environ(), "FANYU_STRIPE_CRASH_CHILD_DB="+s.db.Name())
pipe, e := child.StdoutPipe()
if e != nil {
t.Fatal(e)
}
if e = child.Start(); e != nil {
t.Fatal(e)
}
t.Cleanup(func() {
if child.ProcessState == nil {
child.Process.Kill()
child.Wait()
}
})
ready := make(chan struct{})
go func() {
scanner := bufio.NewScanner(pipe)
for scanner.Scan() {
if scanner.Text() == "STRIPE_CRASH_LOCKS_HELD" {
close(ready)
return
}
}
}()
waitProjectTest(t, ctx, ready)
key := supplierRecordStripeLockID("crash-project", 0)
var old bson.M
if e = s.db.Collection("accountMutationLocks").FindOne(ctx, bson.M{"_id": key}).Decode(&old); e != nil {
t.Fatal(e)
}
if e = child.Process.Kill(); e != nil {
t.Fatal(e)
}
_ = child.Wait()
// No callback or renewal remains alive; expiration is a real process crash.
time.Sleep(1100 * time.Millisecond)
_, e = s.withSupplierProjectLocks(ctx, []string{"crash-project"}, func(ctx context.Context) (any, error) {
r, e := s.db.Collection("accountMutationLocks").UpdateOne(ctx, bson.M{"_id": key, "owner": old["owner"], "locked": true}, bson.M{"$set": bson.M{"locked": false}})
if e != nil {
return nil, e
}
if r.MatchedCount != 0 {
return nil, fmt.Errorf("dead owner's release matched replacement lock")
}
return nil, nil
})
if e != nil {
t.Fatal(e)
}
}
func TestProjectWriteStripesCancelledExclusiveNeverRuns(t *testing.T) {
s, ctx := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "striped")
entered, release := make(chan struct{}), make(chan struct{})
done := make(chan error, 1)
go func() {
_, e := s.withAccountMutationLock(ctx, supplierRecordStripeLockID("p", 10), func(context.Context) (any, error) { close(entered); <-release; return nil, nil })
done <- e
}()
waitProjectTest(t, ctx, entered)
short, stop := context.WithTimeout(ctx, 150*time.Millisecond)
_, e := s.withSupplierProjectLocks(short, []string{"p"}, func(context.Context) (any, error) { t.Error("cancelled partial exclusive entered"); return nil, nil })
stop()
close(release)
if err := <-done; err != nil {
t.Fatal(err)
}
if e == nil {
t.Fatal("cancelled exclusive should fail closed")
}
}
func TestProjectWriteStripesIndependentRecordsOverlap(t *testing.T) {
s, ctx := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "striped")
first, second, release := make(chan struct{}), make(chan struct{}), make(chan struct{})
done := make(chan error, 2)
start := func(id string, entered chan struct{}) {
go func() {
e := supplierGuardedMongoWrite(ctx, s.humanItems, id, bson.M{"id": id, "projectId": "project"}, func(ctx context.Context, d bson.M) error {
close(entered)
select {
case <-release:
case <-ctx.Done():
return ctx.Err()
}
return nil
})
done <- e
}()
}
start("first", first)
select {
case <-first:
case <-ctx.Done():
t.Fatal(ctx.Err())
}
start("second", second)
parallel := false
select {
case <-second:
parallel = true
case <-time.After(300 * time.Millisecond):
}
close(release)
for i := 0; i < 2; i++ {
if e := <-done; e != nil {
t.Fatal(e)
}
}
if !parallel {
t.Fatal("independent same-project saves serialized behind the global project lock")
}
}

View File

@ -11,14 +11,17 @@ ROOT = Path(__file__).resolve().parents[1]
PROFILE = ROOT / "docs/capacity/source-index-100.json"
def render(snapshot, candidate, profile, package_image=None, load_report=None):
def render(snapshot, candidate, profile, package_image=None, load_report=None, lock_stage="barrier"):
if not re.fullmatch(r"[a-f0-9]{40}", candidate):
raise ValueError("A full candidate commit is required")
if lock_stage not in {"barrier", "striped"}:
raise ValueError("Lock stage must be barrier or striped")
namespace = profile["namespace"]
deployments = {d["metadata"]["name"]: d for d in snapshot["items"] if d.get("kind", "Deployment") == "Deployment"}
blockers, patches = [], []
if (not load_report or load_report.get("activeOperators") != 100 or load_report.get("completed") != 100
or load_report.get("sourceFramesPerCamera") != 27000
or load_report.get("projectWriteLockMode") != "striped"
or load_report.get("requests", {}).get("draft-save", {}).get("p95Ms", float("inf")) > 750):
blockers.append("100-operator production-scale local HTTP acceptance with draft-save p95 <=750ms is required")
if package_image is None:
@ -27,6 +30,8 @@ def render(snapshot, candidate, profile, package_image=None, load_report=None):
raise ValueError("Package operator image must be pinned by digest")
for name, desired in profile["deployments"].items():
if name == "droboticflow-annotation-validator" and name not in deployments:
if lock_stage == "striped":
blockers.append(name + ": bridge validator must already exist before enabling striped writes")
continue
d = deployments.get(name)
if d is None or d["metadata"].get("namespace") != namespace:
@ -38,8 +43,18 @@ def render(snapshot, candidate, profile, package_image=None, load_report=None):
index, container = matches[0]
env = container.get("env", [])
values = {e["name"]: e.get("value") for e in env}
if name != "drosync-coordinator" and values.get("DROBOTICFLOW_BUILD_COMMIT") != candidate:
if values.get("DROBOTICFLOW_BUILD_COMMIT") != candidate:
blockers.append(name + ": candidate image/build stamp must be deployed before enabling this profile")
if lock_stage == "striped":
status = d.get("status", {})
desired_replicas = d["spec"]["replicas"]
generation = d["metadata"].get("generation")
complete = (desired_replicas > 0 and generation is not None
and status.get("observedGeneration", 0) >= generation
and all(status.get(k, 0) == desired_replicas for k in ("replicas", "updatedReplicas", "readyReplicas", "availableReplicas"))
and status.get("unavailableReplicas", 0) == 0 and status.get("terminatingReplicas", 0) == 0)
if values.get("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE") not in {"barrier", "striped"} or not complete:
blockers.append(name + ": completed barrier-capable rollout is required before striped activation")
base = f"/spec/template/spec/containers/{index}"
patch = [
{"op": "test", "path": "/metadata/uid", "value": d["metadata"]["uid"]},
@ -49,6 +64,7 @@ def render(snapshot, candidate, profile, package_image=None, load_report=None):
{"op": "add", "path": base + "/resources", "value": {k: desired[k] for k in ("requests", "limits")}},
]
desired_env = dict(desired["env"])
desired_env["DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE"] = lock_stage
if name == "droboticflow-worker" and package_image:
desired_env["ANNOTATION_SOURCE_PACKAGE_IMAGE"] = package_image
for key, value in desired_env.items():
@ -69,6 +85,9 @@ def render(snapshot, candidate, profile, package_image=None, load_report=None):
if "secretKeyRef" not in mongo.get("valueFrom", {}):
raise ValueError("Mongo must remain a Secret reference; no literal credentials can be rendered")
env += [{"name": k, "value": v} for k, v in desired["env"].items()]
for variable in env:
if variable["name"] == "DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE":
variable["value"] = lock_stage
env += [{"name": "ANNOTATION_PROCESSOR_MODE", "value": "processor"},
{"name": "DROBOTICFLOW_DATABASE_BOOTSTRAP", "value": "verify-existing"}]
env.append({"name": "DROBOTICFLOW_BUILD_COMMIT", "value": candidate})
@ -82,10 +101,11 @@ def render(snapshot, candidate, profile, package_image=None, load_report=None):
"resources": {k: desired[k] for k in ("requests", "limits")}, "volumeMounts": source["volumeMounts"], "ports": source.get("ports", []), "readinessProbe": source.get("readinessProbe", {})}],
"volumes": pod["volumes"],
}}}}
result = {"schemaVersion": 1, "profile": profile["profile"], "candidate": candidate, "namespace": namespace,
result = {"schemaVersion": 1, "profile": profile["profile"], "candidate": candidate, "namespace": namespace, "lockStage": lock_stage,
"state": "blocked_until_candidate_and_review" if blockers else "review_only_not_authorized",
"blockers": blockers, "patches": patches, "createValidator": validator, "budget": profile["budget"], "constraints": profile["constraints"],
"approvalRequired": "Separate explicit approval for final images, resource/config changes, new validators, source-copy writes, startup indexes and read/write verification scope"}
"approvalRequired": "Separate explicit approval for final images, resource/config changes, new validators, source-copy writes, startup indexes and read/write verification scope",
"writerInventoryReviewRequired": "Deployment status is not a full writer inventory: review Pods, Jobs, CronJobs, maintenance scripts and external Mongo writers; no old gate-only writer may remain before striped activation"}
cpu = sum(float(d["limits"]["cpu"]) * d["replicas"] for d in profile["deployments"].values())
memory = sum(int(d["limits"]["memory"].removesuffix("Gi")) * 2**30 * d["replicas"] for d in profile["deployments"].values())
allowance = profile["legacyExportBudget"]
@ -106,9 +126,10 @@ def main():
parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--package-image")
parser.add_argument("--load-report", type=Path)
parser.add_argument("--lock-stage", choices=("barrier", "striped"), default="barrier")
args = parser.parse_args()
result = render(json.loads(args.snapshot.read_text()), args.candidate, json.loads(PROFILE.read_text()), args.package_image,
json.loads(args.load_report.read_text()) if args.load_report else None)
json.loads(args.load_report.read_text()) if args.load_report else None, args.lock_stage)
args.output.write_text(json.dumps(result, ensure_ascii=False, indent=2) + "\n")
print(json.dumps({"state": result["state"], "planHash": result["planHash"], "blockers": result["blockers"]}))

View File

@ -25,7 +25,7 @@ class SourceCapacityTests(unittest.TestCase):
"spec": {"replicas": 1, "template": {"spec": {"containers": [{"name": cfg["container"], "image": "registry.test/api@sha256:" + "a" * 64,
"env": env, "volumeMounts": [{"name": "workspace", "mountPath": "/workspace"}]}],
"volumes": [{"name": "workspace", "persistentVolumeClaim": {"claimName": "workspace"}}]}}}})
evidence = {"activeOperators": 100, "completed": 100, "sourceFramesPerCamera": 27000, "requests": {"draft-save": {"p95Ms": 400}}}
evidence = {"activeOperators": 100, "completed": 100, "sourceFramesPerCamera": 27000, "projectWriteLockMode": "striped", "requests": {"draft-save": {"p95Ms": 400}}}
return {"items": items}, profile, evidence
def test_bounded_patches_and_separate_validator_do_not_include_secrets(self):
@ -38,6 +38,7 @@ class SourceCapacityTests(unittest.TestCase):
validator_env = {e["name"]: e.get("value") for e in result["createValidator"]["spec"]["template"]["spec"]["containers"][0]["env"]}
self.assertEqual(validator_env["ANNOTATION_PROCESSOR_MODE"], "processor")
self.assertEqual(validator_env["DROBOTICFLOW_DATABASE_BOOTSTRAP"], "verify-existing")
self.assertEqual(validator_env["DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE"], "barrier")
for item in result["patches"]:
self.assertEqual(item["patch"][0]["path"], "/metadata/uid")
self.assertEqual(item["patch"][1]["path"], "/metadata/resourceVersion")
@ -61,6 +62,27 @@ class SourceCapacityTests(unittest.TestCase):
with self.assertRaises(ValueError):
capacity.render(snapshot, "c" * 40, profile)
def test_striped_activation_requires_all_five_complete_bridge_rollouts(self):
snapshot, profile, evidence = self.fixture()
image = "registry.test/package@sha256:" + "b" * 64
result = capacity.render(snapshot, "c" * 40, profile, image, evidence, "striped")
self.assertEqual(result["state"], "blocked_until_candidate_and_review")
self.assertTrue(any("bridge validator" in b for b in result["blockers"]))
bridge = capacity.render(snapshot, "c" * 40, profile, image, evidence)
snapshot["items"].append(bridge["createValidator"])
for d in snapshot["items"]:
d["metadata"].update({"generation": 2, "uid": d["metadata"]["name"], "resourceVersion": "2"})
replicas = d["spec"]["replicas"]
d["status"] = {"observedGeneration": 2, **{k: replicas for k in ("replicas", "updatedReplicas", "readyReplicas", "availableReplicas")}}
env = d["spec"]["template"]["spec"]["containers"][0]["env"]
if not any(e["name"] == "DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE" for e in env):
env.append({"name": "DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "value": "barrier"})
result = capacity.render(snapshot, "c" * 40, profile, image, evidence, "striped")
self.assertEqual(result["state"], "review_only_not_authorized")
snapshot["items"][0]["status"]["updatedReplicas"] = 0
result = capacity.render(snapshot, "c" * 40, profile, image, evidence, "striped")
self.assertEqual(result["state"], "blocked_until_candidate_and_review")
if __name__ == "__main__":
unittest.main()