⚡ Pace query cold builds and recover confirmed leases

This commit is contained in:
d-robotics 2026-11-13 07:13:01 -08:00
parent c20dbacab6
commit c83f332ba9
23 changed files with 1052 additions and 82 deletions

View File

@ -190,3 +190,51 @@ IO保护新增严格的FUSE归因分支,并有先失败后通过的回归:
下一步先处理独立查询库冷构建的写入节奏、共享控制写和失败恢复:记录每类后台写入/确认等待、检查点及限速窗口,减少重复写和突发积压;保持多数提交和版本隔离,不靠延长HTTP超时、降低写持久性或扩大IO限额掩盖问题。完成后再运行同规模两轮;任何更长时间或更大资源对照需另行界定。之前扩大20轮被自动审批以共享节点持续负载风险拒绝,未执行。
本轮自有合成源Pod及两条策略已按UID清理,桥接按PID/可执行文件身份停止,配额CAS恢复1Pod/1CPU/1Gi。25Gi卷及持久标记保留,查询Pod仍按3600秒deadline退出,未延长;清理证据 `io-rca/fixture-6/cleanup.json`。本批三项功能实现与本地回归完成,完整规模性能验收仍不通过,不准备生产发布。
## 冷构建确认等待:根因与写入入口补齐(2026-11-13)
以 `c20dbaca` 为任务起点继续本地实施,不部署、不切生产路由、不改客户数据。联网新鲜度确认工作树包含 `origin/dev` 和线上提交。
证据将本次等待定位到独立查询库受限IO中的 journal/检查点竞争,叠加未受控的数据写入和失败处理放大,并非原权威Mongo的4核CPU本身造成这些确认等待。`scale-81-9-cold-recovery/slow-metadata.json` 中搜索读表一次200行插入耗时31,844ms,其中31,843ms是写确认等待,CPU约1.4ms;该轮统计事实尚未开始写入。搜索写入绕过了最初的事实/BLOB节奏控制,生产读表还存在2MiB单块突发。因此仅处理统计事实或单纯延迟提交,不能算解决了所有入口。
8MiB随机合成诊断保留 majority+j、三秒心跳和原资源/IO保护,不是HTTP容量证据:
| 诊断 | 总时间 | 心跳最长 | 心跳超时 | 期间检查点累计时间增量 |
| --- | ---: | ---: | ---: | ---: |
| 直接提交32批 | 21.83秒 | 1,938ms | 0 | 7.55秒 |
| 同二进制、128KiB/s提交 | 64.01秒 | 3,002ms | 7 | 29.98秒 |
限速诊断跨过一次检查点,超时与检查点执行、journal同步计数停滞重合;这是否定“固定字节限速单独足够”的证据,不将不同检查点相位解释为严格稳态吞吐收益。宿主盘忙碌很低,而查询cgroup写入接近本次硬限额;`scale-81-9`采样峰值约30写IOPS、约1MiB/s。采样窗口允许短暂突发,不是毫秒级严格速率保证。Mongo多数提交默认等待磁盘journal的语义保持不变([官方写关注说明](https://www.mongodb.com/docs/manual/reference/write-concern/)、[journal说明](https://www.mongodb.com/docs/manual/core/journaling/))。
修正范围:
- 仅显式验证的独立查询库构建器安装共享写入预算,默认512KiB/s逻辑载荷、64KiB预算粒度,覆盖搜索行、生产读表块、统计事实和报表分片;搜索/事实按约64KiB封批,单条记录不截断。生产块为128KiB,报表分片保持原上限,因此预算不保证每次物理写入都小于64KiB。检查点执行时后台数据写入等待,控制续租不进入该等待队列;不在Mongo写事务中等待。监控读取有250ms预算、约1秒复用;不支持/无监控权限时保留字节预算,不增加客户作业准入证书。逻辑节奏不能代替内核硬限额或生产容量验收。
- 搜索批量按字节拆分且完整发布;新生产读表块使用128KiB。清单新增可选 `chunkBytes`,缺失时按原2MiB读取,旧数据无需改写。旧API不能读取某些新多块清单,实际发布须先升级读取方、后启新构建器;回退先关闭新读取,不能将旧API直接指向新清单。
- 统计事实省去零值计数,Mongo聚合仍按零处理;不改统计归属、日期、去重或正文。已有同哈希事实不为压缩而强制重写。
- 每个报表分片不再额外写一次续租,仅读取相同归属/代次/到期条件,独立心跳和最终发布仍多数提交。暂时性续租超时只在最后一次已确认的原租约有效期内有限重试;明确失权或原租约过期立即取消。单次三秒、原60秒租约、90秒报表构建及四分钟就绪等待均未延长。等待后重新计算租约校验时间,防止用等待前时间继续写入。
- 增加阶段耗时及搜索构建行数元数据,不输出业务正文或凭证。诊断报告采用压缩传输;限速小探针第一次原始报告下载失败,原 `exit.json`保留,通过只读gzip恢复的报告及补充验证另存,不改写为HTTP通过。
先失败后通过的回归:一次续租超时错误取消可恢复构建、每片重复续租写入;另验证失权立即停止、原租约到期、检查点等待取消/恢复及期间控制写可用、等待后过期禁止写入、搜索批次和结果完整性、旧大块与新小块往返等价。真实本机Mongo配合race通过。完整固定源码任务检查 `verify-1794578222366219000/report.json` 所选检查通过,包括看板、Go已知失败基线、查询服务、81人本机读取冒烟和查询副本 `--race --cache`。此前测试标签误匹配造成的跳过,以及一次原搜索MaxTimeMS失败均保留;后者原阈值下定点连续3次及完整专项重跑通过,未通过放宽超时修正。
`scale-81-9-cold-recovery` 保留为失败:回填318.133秒、一次租约恢复,四分钟物化就绪超时,未进入HTTP阶段,没有p95或源CPU节省结论。其源Pod/策略/桥接已按身份清理并恢复配额。随后在相同25Gi卷、镜像、1CPU/1Gi和原IO预算下再次验证冷Pod重建,持久标记保留;证据 `query25-cold-writers-resume/`。新一轮固定候选 `scale-81-10-all-writers` 仍为26,218任务、81会话、两轮;以最终独立记录判断,不将本节实现和本地通过当成集群验收。
### 固定预算下的后续验证与重复读取修正
`scale-81-10-all-writers` 使用256KiB/s逻辑预算:搜索26,218行完成于140.694秒,随后统计阶段97.976秒后被原四分钟就绪预算取消;回填328.573秒,未进入HTTP。`scale-81-11-balanced-writers` 将逻辑预算调到512KiB/s,内核硬IO上限、资源、持久性和超时不变:搜索92.939秒,统计阶段145.833秒后仍未就绪;回填393.224秒。两轮均保留失败,不能据此报告查询达标或CPU节省。它们表明搜索写突发得到约束后,还需分离统计加载、计算与分批确认等待。
进一步将统计构建阶段日志拆为历史加载/物化完成及已确认写入行数、字节数。旧历史加载为每200个任务生成一个复杂OR查询;后台构建现在仅合并**完全相同的历史版本集合**,每组最多2,000个任务ID并限制查询字节预算;零散版本仍保留原200条OR分批,避免每任务一次查询。权威请求路径继续原读取方式,不扩大读取的任务/版本组合。
真实隔离Mongo回归先在原实现复现6次读取,再验证新实现2次,1,200条目标切片及历史统计结果等价;6,000条共同版本集纯计划由30批降为3批,601条零散版本保留4批。首次定点绿色运行遇到原MaxTimeMS失败,原记录保留,未放宽阈值;完整相关race回归及完整查询副本runner随后通过。最新快速检查 `verify-1794581789355119000/report.json` 所选检查通过,无跳过;格式检查通过。完整任务检查中未受改动影响的浏览器/基线证据沿用前述固定记录,最新历史读取、写入和恢复相关用例重新运行。
最终同规模记录为 `scale-81-12-grouped-slices`:仍为26,218任务、81会话、两轮,冷构建保持原四分钟就绪、三秒单次续期及多数提交/journal要求;结果在下节补充。单次短测不替代长稳态或生产容量验收。
### 最新同规模对照:环境变更保护停止,未形成性能结论
`scale-81-12-grouped-slices` 在原资源/IO预算下启动,历史回填到80批约145.2秒,发生一次正常租约恢复,尚未进入后台读表冷构建和HTTP对照。15:06:48,`droboticflow-fanyu` 新增 `wishforge-build-api-e9089081` 构建Pod(调度在 `.5` 节点),全量Pod集合保护检测变化并自动停止。只读核对显示当前API、处理器、校验器和Worker重启数均为0,未见本轮期间重新创建;不能把新增构建Pod解释为本测试导致业务故障,也不能据此放松保护阈值。该轮没有可用p95、结果等价或源CPU下降比例,性能验收仍待完成。
证据:`scale-81-12-grouped-slices/stopped.json`、`production-state-readonly.json`、`manifest.json`和原始 `test.log`。测试源Pod由保护器按UID删除;随后桥接/两条策略按身份清理,配额CAS恢复原1Pod/1CPU/1Gi。仅删除本轮精确随机合成查询数据库,保留25Gi文件系统、持久标记与查询Pod原3600秒deadline,见 `fixture-9/cleanup.json` 与 `synthetic-db-cleanup.json`。没有修改生产数据、线上读路由、资源、超时或持久性。
下一步在无并发构建/发布的稳定窗口重跑原规模两轮,先确认冷构建在原期限内完成,再判断HTTP正确性、p95与源MongoCPU/扫描下降。未完成这一证据前不准备上线;也不通过扩大IO、延长超时、关闭journal或将未确认写入当成功来取得通过。

View File

@ -0,0 +1,134 @@
package api
import (
"context"
"crypto/rand"
"encoding/json"
"os"
"sync"
"testing"
"time"
"droboticflow/internal/domain"
"droboticflow/internal/testsupport"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/mongo/writeconcern"
)
// A bounded diagnostic, not HTTP capacity evidence. The outer runner retains
// the existing node/production health guards. Never accepts a production URI.
func TestQueryReplicaColdWriteProbeLocal(t *testing.T) {
if os.Getenv("QUERY_REPLICA_COLD_WRITE_PROBE") != "1" {
t.Skip("explicit isolated cold-write diagnostic only")
}
if os.Getenv("QUERY_REPLICA_CLUSTER_TEST") != "1" {
t.Fatal("owned synthetic cluster required")
}
pace := isolatedComparisonInt(t, "QUERY_COLD_PACE_KIB", 0, 0, 128)
if pace != 0 && pace != 128 {
t.Fatal("only unpaced / 128KiB per second diagnostics")
}
ctx, cancel := context.WithTimeout(t.Context(), 3*time.Minute)
defer cancel()
c, err := mongo.Connect(ctx, options.Client().ApplyURI(testsupport.QueryReplicaTargetURI(t)).SetMaxPoolSize(6).SetWriteConcern(writeconcern.New(writeconcern.WMajority(), writeconcern.J(true))))
if err != nil {
t.Fatal(err)
}
defer c.Disconnect(context.Background())
var marker bson.M
if err = c.Database("query25_synthetic").Collection("marker").FindOne(ctx, bson.M{"_id": "query25-confirmed"}).Decode(&marker); err != nil || marker["value"] != "preserve-query25-v1" {
t.Fatal("owned target marker absent", err)
}
db := c.Database(domain.NewID("cold_write_synthetic"))
defer func() {
cleanup, stop := context.WithTimeout(context.Background(), 30*time.Second)
defer stop()
if e := db.Drop(cleanup); e != nil {
t.Error("synthetic cleanup", e)
}
}()
if _, err = db.Collection("control").InsertOne(ctx, bson.M{"_id": "heartbeat", "at": time.Now()}); err != nil {
t.Fatal(err)
}
var mu sync.Mutex
samples := []bson.M{}
record := func(v bson.M) { mu.Lock(); samples = append(samples, v); mu.Unlock() }
status := func() {
short, stop := context.WithTimeout(ctx, 3*time.Second)
defer stop()
var s bson.M
if e := c.Database("admin").RunCommand(short, bson.M{"serverStatus": 1}).Decode(&s); e == nil {
wt, _ := s["wiredTiger"].(bson.M)
record(bson.M{"kind": "storage", "at": time.Now().UTC(), "log": wt["log"], "checkpoint": wt["checkpoint"], "cache": wt["cache"]})
}
}
status()
stopHeartbeat := make(chan struct{})
done := make(chan struct{})
go func() {
defer close(done)
tick := time.NewTicker(time.Second)
defer tick.Stop()
for {
select {
case <-stopHeartbeat:
return
case <-ctx.Done():
return
case <-tick.C:
short, stop := context.WithTimeout(ctx, 3*time.Second)
start := time.Now()
_, e := db.Collection("control").UpdateOne(short, bson.M{"_id": "heartbeat"}, bson.M{"$set": bson.M{"at": time.Now()}})
stop()
record(bson.M{"kind": "heartbeat", "at": time.Now().UTC(), "ms": float64(time.Since(start).Microseconds()) / 1000, "failed": e != nil})
status()
}
}
}()
var stopOnce sync.Once
stop := func() { stopOnce.Do(func() { close(stopHeartbeat); <-done }) }
defer stop()
started := time.Now()
completed := 0
for batch := 0; batch < 32 && ctx.Err() == nil; batch++ {
rows := []any{}
for i := 0; i < 8; i++ {
payload := make([]byte, 32768)
if _, e := rand.Read(payload); e != nil {
t.Fatal(e)
}
rows = append(rows, bson.M{"_id": batch*8 + i, "payload": payload})
}
start := time.Now()
_, e := db.Collection("facts").InsertMany(ctx, rows)
record(bson.M{"kind": "batch", "at": time.Now().UTC(), "ms": float64(time.Since(start).Microseconds()) / 1000, "failed": e != nil})
if e != nil {
break
}
completed++
if pace > 0 {
due := started.Add(time.Duration(completed) * 2 * time.Second)
if wait := time.Until(due); wait > 0 {
select {
case <-ctx.Done():
case <-time.After(wait):
}
}
}
}
stop()
status()
report := bson.M{"kind": "cold-write-probe", "paceKiB": pace, "payloadBytes": 8 << 20, "completedBatches": completed, "majority": true, "journal": true, "heartbeatTimeoutMs": 3000, "seconds": time.Since(started).Seconds(), "samples": samples, "capacityAccepted": false}
raw, err := json.MarshalIndent(report, "", " ")
if err != nil {
t.Fatal(err)
}
if err = os.WriteFile(os.Getenv("QUERY_COMPARE_REPORT"), raw, 0600); err != nil {
t.Fatal(err)
}
if completed != 32 {
t.Fatal("bounded diagnostic incomplete", completed, ctx.Err())
}
}

View File

@ -0,0 +1,64 @@
package outputsnapshot
import (
"context"
"errors"
"log"
"time"
)
func transientLeaseError(err error) bool {
var timeout interface{ Timeout() bool }
return errors.Is(err, context.DeadlineExceeded) || (errors.As(err, &timeout) && timeout.Timeout())
}
// An acknowledgement timeout is not evidence that ownership was lost. Retry
// only while the LAST CONFIRMED original lease remains valid. All backend
// stages/publications still independently fence owner, epoch and expiry.
func (r *Reader) keepBuildLease(ctx context.Context, l Lease, acquiredAt time.Time, cancel context.CancelFunc, stop <-chan struct{}) {
confirmedUntil := acquiredAt.Add(LeaseDuration)
delay, retry := 10*time.Second, time.Second
for {
remaining := confirmedUntil.Sub(r.now())
if remaining <= 0 {
log.Printf("output_snapshot_lease_renew_failed epoch=%d result=confirmed_lease_expired", l.Epoch)
cancel()
return
}
timer := time.NewTimer(min(delay, remaining))
select {
case <-stop:
timer.Stop()
return
case <-ctx.Done():
timer.Stop()
return
case <-timer.C:
}
at := r.now()
remaining = confirmedUntil.Sub(at)
if remaining <= 0 {
cancel()
return
}
renewal, finish := context.WithTimeout(ctx, min(3*time.Second, remaining))
ok, err := r.backend.RenewOutputSnapshot(renewal, l, at)
finish()
if err == nil && ok {
confirmedUntil = at.Add(LeaseDuration)
delay, retry = 10*time.Second, time.Second
continue
}
if ctx.Err() != nil {
return
}
if err != nil && transientLeaseError(err) && r.now().Before(confirmedUntil) {
log.Printf("output_snapshot_lease_renew_retry epoch=%d remaining_ms=%d", l.Epoch, confirmedUntil.Sub(r.now()).Milliseconds())
delay, retry = retry, min(retry*2, 10*time.Second)
continue
}
log.Printf("output_snapshot_lease_renew_failed epoch=%d renewed=%t error=%v", l.Epoch, ok, err)
cancel()
return
}
}

View File

@ -0,0 +1,89 @@
package outputsnapshot
import (
"context"
"errors"
"sync/atomic"
"testing"
"time"
"droboticflow/internal/workforcestats"
)
type interruptedRenewal struct {
MemoryBackend
calls atomic.Int32
recovered chan struct{}
lost bool
}
func TestSnapshotRenewalRetryNeverExtendsUnconfirmedLease(t *testing.T) {
base := time.Now()
b := &MemoryBackend{}
r := NewReader(b, nil)
defer r.Close()
r.now = func() time.Time { return base.Add(LeaseDuration) }
ctx, cancel := context.WithCancel(t.Context())
r.keepBuildLease(ctx, Lease{Owner: "old", Epoch: 1}, base, cancel, make(chan struct{}))
if !errors.Is(ctx.Err(), context.Canceled) {
t.Fatal("expired unconfirmed ownership remained active")
}
}
func (b *interruptedRenewal) RenewOutputSnapshot(ctx context.Context, l Lease, now time.Time) (bool, error) {
if b.calls.Add(1) == 1 {
if b.lost {
return false, nil
}
return false, context.DeadlineExceeded
}
ok, err := b.MemoryBackend.RenewOutputSnapshot(ctx, l, now)
if ok && err == nil {
select {
case <-b.recovered:
default:
close(b.recovered)
}
}
return ok, err
}
func TestSnapshotBuildRecoversTransientRenewalWithinOriginalLease(t *testing.T) {
b := &interruptedRenewal{recovered: make(chan struct{})}
r := NewReader(b, func(ctx context.Context, _ string, _ []string) (workforcestats.OutputSnapshot, error) {
select {
case <-b.recovered:
return workforcestats.OutputSnapshot{}, nil
case <-ctx.Done():
return workforcestats.OutputSnapshot{}, ctx.Err()
}
})
defer r.Close()
ctx, stop := context.WithTimeout(r.ctx, 20*time.Second)
defer stop()
r.ctx = ctx
key := scopeKey("org", []string{"p"})
r.build(key, "org", []string{"p"})
s, _ := b.OutputSnapshotState(t.Context())
if b.calls.Load() < 2 || s.Heads[key].Current == nil || s.Heads[key].Failed {
t.Fatalf("a single uncertain renewal discarded a recoverable build: calls=%d published=%t failed=%t", b.calls.Load(), s.Heads[key].Current != nil, s.Heads[key].Failed)
}
}
func TestSnapshotBuildStopsImmediatelyAfterDefinitiveLeaseLoss(t *testing.T) {
b := &interruptedRenewal{recovered: make(chan struct{}), lost: true}
r := NewReader(b, func(ctx context.Context, _ string, _ []string) (workforcestats.OutputSnapshot, error) {
<-ctx.Done()
return workforcestats.OutputSnapshot{}, ctx.Err()
})
defer r.Close()
ctx, stop := context.WithTimeout(r.ctx, 15*time.Second)
defer stop()
r.ctx = ctx
key := scopeKey("org", []string{"p"})
r.build(key, "org", []string{"p"})
s, _ := b.OutputSnapshotState(t.Context())
if b.calls.Load() != 1 || s.Heads[key].Current != nil || !s.Heads[key].Failed {
t.Fatalf("lost owner retried or published: calls=%d state=%+v", b.calls.Load(), s.Heads[key])
}
}

View File

@ -186,7 +186,8 @@ func (r *Reader) request(key, org string, projects []string, head Head, now time
func (r *Reader) build(key, org string, projects []string) {
ctx, cancel := context.WithTimeout(r.ctx, BuildTimeout)
defer cancel()
lease, ok, err := r.backend.AcquireOutputSnapshot(ctx, key, r.now())
acquiredAt := r.now()
lease, ok, err := r.backend.AcquireOutputSnapshot(ctx, key, acquiredAt)
if err != nil || !ok {
return
}
@ -214,25 +215,7 @@ func (r *Reader) build(key, org string, projects []string) {
done := make(chan struct{})
go func() {
defer close(done)
ticker := time.NewTicker(10 * time.Second)
defer ticker.Stop()
for {
select {
case <-stop:
return
case <-ctx.Done():
return
case <-ticker.C:
renewal, finish := context.WithTimeout(ctx, 3*time.Second)
ok, err := r.backend.RenewOutputSnapshot(renewal, lease, r.now())
finish()
if err != nil || !ok {
log.Printf("output_snapshot_lease_renew_failed epoch=%d renewed=%t error=%v", lease.Epoch, ok, err)
cancel()
return
}
}
}
r.keepBuildLease(ctx, lease, acquiredAt, cancel, stop)
}()
defer func() { close(stop); <-done }()
asOf := r.now().UTC().Truncate(time.Millisecond)

View File

@ -96,6 +96,8 @@ func (s *MongoStore) RenewOutputSnapshot(ctx context.Context, l outputsnapshot.L
return err == nil && res.MatchedCount == 1, err
}
func (s *MongoStore) PublishOutputSnapshot(ctx context.Context, l outputsnapshot.Lease, g outputsnapshot.Generation, data []byte, now time.Time) (bool, error) {
started := time.Now()
current := func() time.Time { return now.Add(time.Since(started)) }
if replica, err := s.reportReplica(ctx); err != nil {
return false, err
} else if replica != nil {
@ -104,7 +106,10 @@ func (s *MongoStore) PublishOutputSnapshot(ctx context.Context, l outputsnapshot
if !outputsnapshot.ValidPublication(l, g, data) {
return false, outputsnapshot.ErrCapacity
}
ok, err := s.RenewOutputSnapshot(ctx, l, now)
if err := s.paceQueryBuild(ctx, len(data)); err != nil {
return false, err
}
ok, err := s.RenewOutputSnapshot(ctx, l, current())
if err != nil || !ok {
return false, err
}
@ -120,7 +125,7 @@ func (s *MongoStore) PublishOutputSnapshot(ctx context.Context, l outputsnapshot
return false, err
}
head := outputsnapshot.Head{Current: &g, Previous: l.Previous, NextAttempt: g.AsOf.Add(outputsnapshot.RefreshInterval)}
res, err := s.outputCoordinator().UpdateOne(write, outputLeaseFilter(l, now), bson.M{"$set": bson.M{"heads." + l.Key: head, "owner": "", "until": now}})
res, err := s.outputCoordinator().UpdateOne(write, outputLeaseFilter(l, current()), bson.M{"$set": bson.M{"heads." + l.Key: head, "owner": "", "until": current()}})
if err == nil && res.MatchedCount == 1 {
// Same slot only: current and previous generations occupy different slots.
// Epoch-qualified immutable keys protect against late expired writers.
@ -169,6 +174,7 @@ func outputShardID(slot int, epoch int64, index int) string {
return fmt.Sprintf("of-shard:%02d:%020d:%09d", slot, epoch, index)
}
func (s *MongoStore) StageOutputSnapshotShard(ctx context.Context, l outputsnapshot.Lease, index int, data []byte, now time.Time) (bool, error) {
started := time.Now()
if replica, err := s.reportReplica(ctx); err != nil {
return false, err
} else if replica != nil {
@ -177,8 +183,22 @@ func (s *MongoStore) StageOutputSnapshotShard(ctx context.Context, l outputsnaps
if l.Slot < 0 || l.Slot >= outputsnapshot.SlotCount || index < 0 || index >= outputsnapshot.MaxShards || len(data) > outputsnapshot.MaxEncodedBytes {
return false, outputsnapshot.ErrCapacity
}
ok, err := s.RenewOutputSnapshot(ctx, l, now)
if err != nil || !ok {
if err := s.paceQueryBuild(ctx, len(data)); err != nil {
return false, err
}
// Admission may wait for a checkpoint: never test the lease against the
// timestamp from BEFORE that wait. Preserve explicitly injected test clocks.
now = now.Add(time.Since(started))
// The reader's heartbeat renews the lease. Per-shard renewal adds a
// majority journal flush for every chunk without strengthening publication:
// immutable epoch IDs cannot overwrite a successor and publish is fenced.
// Keep the same ownership/expiry check, as in MemoryBackend, without writing.
var owner bson.M
err := s.outputCoordinator().FindOne(ctx, outputLeaseFilter(l, now), options.FindOne().SetProjection(bson.M{"_id": 1})).Decode(&owner)
if err == mongo.ErrNoDocuments {
return false, nil
}
if err != nil {
return false, err
}
if index == 0 {

View File

@ -80,7 +80,7 @@ func TestAnnotationOutputMongoSharedAcrossInstancesAndRestart(t *testing.T) {
t.Fatalf("cross instance history rebuilt %d times", n)
}
}
func TestAnnotationOutputLeaseFencingAndFixedSlots(t *testing.T) {
func TestAnnotationOutputMongoLeaseFencingAndFixedSlots(t *testing.T) {
db := localOutputMongo(t)
for name, backend := range map[string]outputsnapshot.Backend{"memory": NewMemoryStore(), "mongo": NewMongoStore(db)} {
t.Run(name, func(t *testing.T) {
@ -121,7 +121,7 @@ func TestAnnotationOutputLeaseFencingAndFixedSlots(t *testing.T) {
}
}
func TestAnnotationOutputSourceBudgetAndPayloadProjection(t *testing.T) {
func TestAnnotationOutputMongoSourceBudgetAndPayloadProjection(t *testing.T) {
db := localOutputMongo(t)
s := NewMongoStore(db)
seedAnnotationOutput(t, s)
@ -142,7 +142,7 @@ func TestAnnotationOutputSourceBudgetAndPayloadProjection(t *testing.T) {
}
}
func TestAnnotationOutputPrivateCollectionsCannotBypassReportScopes(t *testing.T) {
func TestAnnotationOutputMongoPrivateCollectionsCannotBypassReportScopes(t *testing.T) {
db := localOutputMongo(t)
s := NewMongoStore(db)
for _, name := range []string{outputsnapshot.StateCollection, outputsnapshot.SlotsCollection} {

View File

@ -24,6 +24,7 @@ type MongoStore struct {
querySyncEnabled atomic.Bool
queryReplicaReads *queryReplicaReads
queryReplicaFactStore bool
queryBuildWrites *queryBuildWriteGate
queryReplicaCounts queryCountCache
queryReplicaBatchFacts queryBatchFacts
queryServing *queryServingOptions

View File

@ -0,0 +1,85 @@
package store
import (
"context"
"sync"
"time"
"go.mongodb.org/mongo-driver/bson"
"golang.org/x/time/rate"
)
const queryBuildBatchBytes = 64 << 10
// This admission budget belongs ONLY to a verified independent read-store
// builder. It never gates authority commands or customer requests. It leaves
// IO headroom for journal/control writes; the kernel hard limits still apply.
type queryBuildWriteGate struct {
limiter *rate.Limiter
mu sync.Mutex
next time.Time
checkpoint bool
inspect func(context.Context) (bool, error)
}
func (s *MongoStore) newQueryBuildWriteGate() *queryBuildWriteGate {
return &queryBuildWriteGate{limiter: rate.NewLimiter(512<<10, queryBuildBatchBytes), inspect: func(ctx context.Context) (bool, error) {
var state struct {
WT struct {
Checkpoint struct {
Progress int `bson:"progress state"`
} `bson:"checkpoint"`
Transaction struct {
Running int `bson:"transaction checkpoint currently running"`
} `bson:"transaction"`
} `bson:"wiredTiger"`
}
err := s.db.Client().Database("admin").RunCommand(ctx, bson.M{"serverStatus": 1}).Decode(&state)
return state.WT.Checkpoint.Progress != 0 || state.WT.Transaction.Running != 0, err
}}
}
func (g *queryBuildWriteGate) wait(ctx context.Context, bytes int) error {
if g == nil {
return ctx.Err()
}
for bytes > 0 {
n := min(bytes, queryBuildBatchBytes)
if err := g.limiter.WaitN(ctx, n); err != nil {
return err
}
bytes -= n
}
for {
if err := ctx.Err(); err != nil {
return err
}
g.mu.Lock()
if !time.Now().Before(g.next) {
probe, stop := context.WithTimeout(ctx, 250*time.Millisecond)
busy, err := g.inspect(probe)
stop()
// Unsupported metrics/monitor privileges do not become another
// admission certificate. Keep byte pacing, never block core work.
g.checkpoint = err == nil && busy
g.next = time.Now().Add(time.Second)
}
busy, next := g.checkpoint, g.next
g.mu.Unlock()
if !busy {
break
}
timer := time.NewTimer(max(time.Millisecond, time.Until(next)))
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
}
return ctx.Err()
}
func (s *MongoStore) paceQueryBuild(ctx context.Context, bytes int) error {
return s.queryBuildWrites.wait(ctx, bytes)
}

View File

@ -0,0 +1,44 @@
package store
import (
"context"
"errors"
"testing"
"time"
"golang.org/x/time/rate"
)
func TestQueryBuildWriteWaitCancelsDuringCheckpoint(t *testing.T) {
g := &queryBuildWriteGate{limiter: rate.NewLimiter(rate.Inf, queryBuildBatchBytes), inspect: func(context.Context) (bool, error) { return true, nil }}
ctx, stop := context.WithTimeout(t.Context(), 20*time.Millisecond)
defer stop()
if err := g.wait(ctx, 1); !errors.Is(err, context.DeadlineExceeded) {
t.Fatal("checkpoint waiting ignored cancellation", err)
}
}
func TestQueryBuildWriteMissingMetricsKeepsByteBudget(t *testing.T) {
calls := 0
g := &queryBuildWriteGate{limiter: rate.NewLimiter(1024, queryBuildBatchBytes), inspect: func(context.Context) (bool, error) {
calls++
return true, errors.New("monitor permission unavailable")
}}
if err := g.wait(t.Context(), queryBuildBatchBytes); err != nil || calls != 1 {
t.Fatal("optional metrics became an admission certificate", err, calls)
}
ctx, stop := context.WithTimeout(t.Context(), 20*time.Millisecond)
defer stop()
if err := g.wait(ctx, queryBuildBatchBytes); err == nil {
t.Fatal("byte budget disappeared with monitor privilege")
}
}
func TestQueryBuildWriteBudgetIsNotInstalledOnAuthority(t *testing.T) {
s := &MongoStore{}
b := NewQueryReadBuilder(s)
defer b.Close()
if s.queryBuildWrites != nil {
t.Fatal("authority store acquired background write admission")
}
}

View File

@ -0,0 +1,291 @@
package store
import (
"bytes"
"compress/gzip"
"context"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"os"
"reflect"
"sort"
"strings"
"sync/atomic"
"testing"
"time"
"droboticflow/internal/domain"
"droboticflow/internal/outputsnapshot"
"droboticflow/internal/workforcestats"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/event"
"golang.org/x/time/rate"
)
func TestQueryBuildSliceReadsGroupEqualVersionSetsLocal(t *testing.T) {
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
t.Skip("owned synthetic replica required")
}
var reads atomic.Int32
m := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
if e.CommandName == "find" {
if n, _ := e.Command.Lookup("find").StringValueOK(); n == "sliceRevisions" {
reads.Add(1)
}
}
}}
s, _, _ := newLocalReplica(t, m, nil)
tasks, events, slices := []any{}, []any{}, []any{}
for i := 0; i < 1200; i++ {
id := fmt.Sprintf("version-task-%04d", i)
v := i%2 + 1
tasks = append(tasks, bson.M{"id": id, "organizationId": "o", "projectId": "p"})
events = append(events, bson.M{"id": "event-" + id, "organizationId": "o", "projectId": "p", "taskId": id, "eventType": "combined_submitted", "revision": v, "createdAt": "2026-11-01T10:00:00Z"})
for r := 1; r <= 3; r++ {
slices = append(slices, bson.M{"id": fmt.Sprintf("%s/%d", id, r), "organizationId": "o", "taskId": id, "revision": r, "segments": bson.A{bson.M{"start": 0, "end": r}}})
}
}
for _, x := range []struct {
c mongoCollection
rows []any
}{{s.annotationTasks, tasks}, {s.annotationWorklogEvents, events}, {s.sliceRevisions, slices}} {
if _, err := x.c.InsertMany(t.Context(), x.rows); err != nil {
t.Fatal(err)
}
}
q := workforcestats.Query{Mode: domain.RunModeAll}
reads.Store(0)
old, err := s.LoadWorkforceStats(t.Context(), "o", q)
if err != nil {
t.Fatal(err)
}
oldReads := reads.Load()
reads.Store(0)
ctx := context.WithValue(t.Context(), workforceBuildBudgetKey{}, &workforceBuildBudget{})
next, err := s.LoadWorkforceStats(ctx, "o", q)
if err != nil {
t.Fatal(err)
}
if reads.Load() != 2 || oldReads != 6 {
t.Fatalf("background slice queries not merged: old=%d new=%d", oldReads, reads.Load())
}
less := func(rows []domain.SliceRevisionRecord) {
sort.Slice(rows, func(i, j int) bool {
if rows[i].TaskID == rows[j].TaskID {
return rows[i].Revision < rows[j].Revision
}
return rows[i].TaskID < rows[j].TaskID
})
}
less(old.Slices)
less(next.Slices)
if len(next.Slices) != 1200 || !reflect.DeepEqual(old.Slices, next.Slices) || !reflect.DeepEqual(workforcestats.Build(old, q), workforcestats.Build(next, q)) {
t.Fatal("grouping changed exact versions or historical statistics")
}
}
func TestQueryBuildSmallAndLegacyBlobChunksLocal(t *testing.T) {
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
t.Skip("owned synthetic replica required")
}
_, target, _ := newLocalReplica(t, nil, nil)
data := make([]byte, 300<<10)
if _, err := rand.Read(data); err != nil {
t.Fatal(err)
}
value := map[string]string{"payload": hex.EncodeToString(data)}
h, err := target.stageQueryModel(t.Context(), "small", value, time.Now())
if err != nil {
t.Fatal(err)
}
if h.ChunkBytes != queryModelBuildChunkBytes || h.Chunks < 2 {
t.Fatal("new construction still emits large bursts", h)
}
var got map[string]string
if err = target.loadQueryModel(t.Context(), "small", h, &got); err != nil || !reflect.DeepEqual(got, value) {
t.Fatal("small chunk round trip", err)
}
var raw bytes.Buffer
zw := gzip.NewWriter(&raw)
if err = json.NewEncoder(zw).Encode(value); err != nil {
t.Fatal(err)
}
if err = zw.Close(); err != nil {
t.Fatal(err)
}
sum := sha256.Sum256(raw.Bytes())
legacy := QueryModelHead{Generation: "legacy", Bytes: raw.Len(), Chunks: (raw.Len() + queryModelChunkBytes - 1) / queryModelChunkBytes, Digest: hex.EncodeToString(sum[:])}
for i := 0; i < legacy.Chunks; i++ {
if _, err = target.queryData().InsertOne(t.Context(), bson.M{"_id": queryModelBlobKey("old", legacy.Generation, i), "data": raw.Bytes()[i*queryModelChunkBytes : min((i+1)*queryModelChunkBytes, raw.Len())]}); err != nil {
t.Fatal(err)
}
}
got = nil
if err = target.loadQueryModel(t.Context(), "old", legacy, &got); err != nil || !reflect.DeepEqual(got, value) {
t.Fatal("old manifest without chunkBytes lost compatibility", err)
}
}
func TestQueryBuildSearchUsesBoundedPacedBatchesLocal(t *testing.T) {
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
t.Skip("owned synthetic replica required")
}
var maxBytes atomic.Int32
monitor := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
if e.CommandName != "insert" {
return
}
name, _ := e.Command.Lookup("insert").StringValueOK()
if name != "queryReadModelData" {
return
}
values, _ := e.Command.Lookup("documents").Array().Values()
size := 0
for _, v := range values {
size += len(v.Document())
}
for before := maxBytes.Load(); int32(size) > before; before = maxBytes.Load() {
if maxBytes.CompareAndSwap(before, int32(size)) {
break
}
}
}}
_, target, _ := newLocalReplica(t, nil, monitor)
for i := 0; i < 300; i++ {
if err := target.UpsertHumanItem(t.Context(), domain.HumanItemRecord{ID: fmt.Sprint(i), OrganizationID: "o", PluginID: "annotation-qa", ProjectID: "p", Status: domain.HumanItemReady, Subject: domain.HumanSubjectRef{ItemKey: strings.Repeat("synthetic-name-", 100) + fmt.Sprint(i)}}); err != nil {
t.Fatal(err)
}
}
l, ok, err := target.acquireQueryModelLease(t.Context())
if err != nil || !ok {
t.Fatal(ok, err)
}
var inspected atomic.Int32
target.queryBuildWrites = &queryBuildWriteGate{limiter: rate.NewLimiter(rate.Inf, queryBuildBatchBytes), inspect: func(context.Context) (bool, error) { inspected.Add(1); return false, nil }}
if err = target.buildQuerySearchModel(t.Context(), l, "o"); err != nil {
t.Fatal(err)
}
st, err := target.querySearchState(t.Context())
if err != nil {
t.Fatal(err)
}
if st.Heads[queryModelKey("search", "o", nil)].Rows != 300 || inspected.Load() == 0 || maxBytes.Load() > queryBuildBatchBytes {
t.Fatalf("search writer bypassed budget or lost rows: rows=%d inspect=%d maxBytes=%d", st.Heads[queryModelKey("search", "o", nil)].Rows, inspected.Load(), maxBytes.Load())
}
}
func TestQueryBuildShardStagingChecksLeaseWithoutHeartbeatWritesLocal(t *testing.T) {
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
t.Skip("owned synthetic replica required")
}
var renews atomic.Int32
monitor := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
if e.CommandName == "update" {
if v, ok := e.Command.Lookup("update").StringValueOK(); ok && v == outputsnapshot.StateCollection {
renews.Add(1)
}
}
}}
_, target, _ := newLocalReplica(t, nil, monitor)
now := time.Now().UTC().Truncate(time.Millisecond)
l, ok, err := target.AcquireOutputSnapshot(t.Context(), "cold-stage", now)
if err != nil || !ok {
t.Fatal(ok, err)
}
before := renews.Load()
for i := 0; i < 3; i++ {
ok, err := target.StageOutputSnapshotShard(t.Context(), l, i, []byte{byte(i)}, now.Add(time.Second))
if err != nil || !ok {
t.Fatal(ok, err)
}
}
if got := renews.Load() - before; got != 0 {
t.Fatalf("staging redundantly persisted %d heartbeats", got)
}
st, err := target.OutputSnapshotState(t.Context())
if err != nil || !st.Until.Equal(now.Add(outputsnapshot.LeaseDuration)) {
t.Fatalf("staging changed the existing lease: %+v %v", st, err)
}
if ok, err := target.StageOutputSnapshotShard(t.Context(), l, 3, []byte("expired"), now.Add(outputsnapshot.LeaseDuration+time.Second)); err != nil || ok {
t.Fatal("expired writer staged", ok, err)
}
}
func TestQueryBuildShardWaitsForCheckpointWithoutHoldingWriteTransactionLocal(t *testing.T) {
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
t.Skip("owned synthetic replica required")
}
var inserts atomic.Int32
monitor := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
if e.CommandName == "insert" {
if v, ok := e.Command.Lookup("insert").StringValueOK(); ok && v == outputsnapshot.SlotsCollection {
inserts.Add(1)
}
}
}}
_, target, _ := newLocalReplica(t, nil, monitor)
l, ok, err := target.AcquireOutputSnapshot(t.Context(), "checkpoint", time.Now())
if err != nil || !ok {
t.Fatal(ok, err)
}
var busy atomic.Bool
busy.Store(true)
entered := make(chan struct{}, 1)
target.queryBuildWrites = &queryBuildWriteGate{limiter: rate.NewLimiter(rate.Inf, queryBuildBatchBytes), inspect: func(context.Context) (bool, error) {
select {
case entered <- struct{}{}:
default:
}
return busy.Load(), nil
}}
done := make(chan error, 1)
go func() {
ok, e := target.StageOutputSnapshotShard(t.Context(), l, 0, []byte("complete"), time.Now())
if e == nil && !ok {
e = context.Canceled
}
done <- e
}()
<-entered
if inserts.Load() != 0 {
t.Fatal("data write entered checkpoint IO queue")
}
// The independent heartbeat remains available while the data producer waits.
if ok, e := target.RenewOutputSnapshot(t.Context(), l, time.Now()); e != nil || !ok {
t.Fatal("checkpoint pause blocked control writes", ok, e)
}
busy.Store(false)
select {
case e := <-done:
if e != nil {
t.Fatal(e)
}
case <-time.After(3 * time.Second):
t.Fatal("producer did not resume")
}
if inserts.Load() != 1 {
t.Fatal("resumed staging was duplicated", inserts.Load())
}
}
func TestQueryBuildShardRechecksExpiryAfterCheckpointWaitLocal(t *testing.T) {
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
t.Skip("owned synthetic replica required")
}
_, target, _ := newLocalReplica(t, nil, nil)
// The lease is valid at admission but expires while waiting. A timestamp
// captured before that wait must not authorize the subsequent staging.
now := time.Now()
l, ok, err := target.AcquireOutputSnapshot(t.Context(), "expiring", now.Add(-outputsnapshot.LeaseDuration+500*time.Millisecond))
if err != nil || !ok {
t.Fatal(ok, err)
}
calls := 0
target.queryBuildWrites = &queryBuildWriteGate{limiter: rate.NewLimiter(rate.Inf, queryBuildBatchBytes), inspect: func(context.Context) (bool, error) { calls++; return calls == 1, nil }}
if ok, err = target.StageOutputSnapshotShard(t.Context(), l, 0, []byte("late"), now); err != nil || ok {
t.Fatal("pre-wait time authorized an expired owner", ok, err)
}
}

View File

@ -33,6 +33,9 @@ type QueryReadBuilder struct {
}
func NewQueryReadBuilder(s *MongoStore) *QueryReadBuilder {
if s.queryReplicaFactStore {
s.queryBuildWrites = s.newQueryBuildWriteGate()
}
r := outputsnapshot.NewStreamReader(s, s.StreamAnnotationOutput)
return &QueryReadBuilder{store: s, dirty: map[string]bool{}, all: true, reader: r, searchDirty: map[string]bool{}, allSearch: true, sourceDigests: map[string]querySourceDigest{}}
}
@ -107,8 +110,14 @@ func (b *QueryReadBuilder) scopes(ctx context.Context) (map[string][]string, err
}
return out, nil
}
func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org string, projects []string) error {
func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org string, projects []string) (err error) {
started := time.Now()
stage := "source"
defer func() {
if err != nil {
log.Printf("query_model_build result=failed stage=%s elapsed_ms=%d", stage, time.Since(started).Milliseconds())
}
}()
var source struct {
Revision int64 `bson:"productionRevision"`
Global int64 `bson:"searchGlobalRevision"`
@ -127,21 +136,25 @@ func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org s
return err
}
key := queryModelKey("production", org, projects)
stage = "production_blob"
head, err := b.store.stageQueryModel(ctx, key, productionstats.Build(snapshot), asOf)
if err != nil {
return err
}
if b.store.queryReplicaFactStore {
stage = "workforce_facts"
if err = b.store.rebuildWorkforceFacts(buildCtx, l, org, source.Revision); err != nil {
return err
}
}
head.SourceRevision = source.Revision
stage = "publish"
head.GlobalRevision = source.Global
if err = b.store.publishQueryModelHead(ctx, l, key, head); err != nil {
return err
}
if len(projects) > 0 {
stage = "output_prewarm"
if _, err = b.reader.Prewarm(ctx, org, projects); err != nil {
return err
}

View File

@ -24,6 +24,7 @@ import (
const queryModelSchema = "query-model/v3"
const queryModelChunkBytes = 2 << 20
const queryModelBuildChunkBytes = 128 << 10
const queryModelMaxBytes = 64 << 20
type queryServingOptions struct {
@ -71,6 +72,7 @@ func (s *MongoStore) SharedQueryCache() queryserving.Cache {
}
type QueryModelHead struct {
ChunkBytes int `bson:"chunkBytes,omitempty" json:"chunkBytes,omitempty"`
FactEpoch int64 `bson:"factEpoch,omitempty" json:"factEpoch,omitempty"`
SourceRevision int64 `bson:"sourceRevision" json:"sourceRevision"`
Revision int64 `bson:"revision" json:"revision"`
@ -168,11 +170,14 @@ func (s *MongoStore) stageQueryModel(ctx context.Context, key string, value any,
if data.Len() > queryModelMaxBytes {
return QueryModelHead{}, errors.New("query materialization exceeds budget")
}
h := QueryModelHead{Generation: domain.NewID("query-generation"), AsOf: asOf, Bytes: data.Len(), Chunks: (data.Len() + queryModelChunkBytes - 1) / queryModelChunkBytes}
h := QueryModelHead{Generation: domain.NewID("query-generation"), AsOf: asOf, Bytes: data.Len(), ChunkBytes: queryModelBuildChunkBytes, Chunks: (data.Len() + queryModelBuildChunkBytes - 1) / queryModelBuildChunkBytes}
sum := sha256.Sum256(data.Bytes())
h.Digest = hex.EncodeToString(sum[:])
for i := 0; i < h.Chunks; i++ {
b := data.Bytes()[i*queryModelChunkBytes : min((i+1)*queryModelChunkBytes, data.Len())]
b := data.Bytes()[i*h.ChunkBytes : min((i+1)*h.ChunkBytes, data.Len())]
if err := s.paceQueryBuild(ctx, len(b)); err != nil {
return QueryModelHead{}, err
}
_, err := s.queryData().InsertOne(ctx, bson.M{"_id": queryModelBlobKey(key, h.Generation, i), "generation": h.Generation, "data": b, "createdAt": time.Now().UTC()})
if err != nil {
return QueryModelHead{}, err
@ -184,7 +189,11 @@ func (s *MongoStore) stageQueryModel(ctx context.Context, key string, value any,
return h, nil
}
func (s *MongoStore) loadQueryModel(ctx context.Context, key string, h QueryModelHead, out any) error {
if h.Chunks < 1 || h.Bytes < 1 || h.Bytes > queryModelMaxBytes || h.Chunks != (h.Bytes+queryModelChunkBytes-1)/queryModelChunkBytes {
chunkBytes := h.ChunkBytes
if chunkBytes == 0 {
chunkBytes = queryModelChunkBytes
} // Existing 2MiB manifests remain readable.
if chunkBytes < 64<<10 || chunkBytes > queryModelChunkBytes || h.Chunks < 1 || h.Bytes < 1 || h.Bytes > queryModelMaxBytes || h.Chunks != (h.Bytes+chunkBytes-1)/chunkBytes {
return errors.New("invalid query manifest")
}
var data bytes.Buffer
@ -207,7 +216,7 @@ func (s *MongoStore) loadQueryModel(ctx context.Context, key string, h QueryMode
cache.Put(ctx, blobKey, b, 15*time.Minute)
}
}
if len(b) > queryModelChunkBytes {
if len(b) > chunkBytes {
return errors.New("invalid query chunk")
}
data.Write(b)

View File

@ -5,6 +5,7 @@ import (
"crypto/sha256"
"encoding/hex"
"errors"
"log"
"net/url"
"regexp"
"strings"
@ -54,6 +55,10 @@ func (s *MongoStore) buildQuerySearchModel(ctx context.Context, l queryModelLeas
return err
}
h := QueryModelHead{Generation: domain.NewID("query-search"), AsOf: time.Now().UTC(), Revision: state.Revisions[key], GlobalRevision: state.Global}
started := time.Now()
defer func() {
log.Printf("query_search_build rows=%d elapsed_ms=%d", h.Rows, time.Since(started).Milliseconds())
}()
if s.queryServing != nil && s.queryServing.search != nil {
sum := sha256.Sum256([]byte(s.db.Name() + ":" + key + ":" + h.Generation))
h.Collection = "dfq_" + hex.EncodeToString(sum[:])
@ -103,6 +108,26 @@ func (s *MongoStore) buildQuerySearchModel(ctx context.Context, l queryModelLeas
}
docs := make([]any, 0, len(rows))
searchDocs := make([]queryserving.SearchDocument, 0, len(rows))
batchBytes := 0
flush := func() error {
if len(docs) == 0 {
return nil
}
if err := s.paceQueryBuild(ctx, batchBytes); err != nil {
return err
}
if _, err := s.queryData().InsertMany(ctx, docs); err != nil {
return err
}
if h.Collection != "" {
if err := s.queryServing.search.Import(ctx, h.Collection, searchDocs); err != nil {
return err
}
}
h.Rows += len(docs)
docs, searchDocs, batchBytes = nil, nil, 0
return nil
}
for _, row := range rows {
for _, task := range row.Tasks {
if querySafePath(task.SourceVideo) != task.SourceVideo {
@ -121,18 +146,23 @@ func (s *MongoStore) buildQuerySearchModel(ctx context.Context, l queryModelLeas
// Store only the conservative searchable text and identity. No
// task body, draft, review opinion, eligibility or credential.
id := queryserving.Key("row", h.Generation, row.ID)
docs = append(docs, bson.M{"_id": id, "generation": h.Generation, "organizationId": normalizedOrganizationID(org), "itemId": row.ID, "text": text, "createdAt": time.Now().UTC()})
doc := bson.M{"_id": id, "generation": h.Generation, "organizationId": normalizedOrganizationID(org), "itemId": row.ID, "text": text, "createdAt": time.Now().UTC()}
raw, e := bson.Marshal(doc)
if e != nil {
return e
}
if batchBytes+len(raw) > queryBuildBatchBytes {
if err = flush(); err != nil {
return err
}
}
docs = append(docs, doc)
batchBytes += len(raw)
searchDocs = append(searchDocs, queryserving.SearchDocument{ID: id, ItemID: row.ID, Grams: queryserving.IndexGrams(text)})
}
if _, err = s.queryData().InsertMany(ctx, docs); err != nil {
if err = flush(); err != nil {
return err
}
if h.Collection != "" {
if err = s.queryServing.search.Import(ctx, h.Collection, searchDocs); err != nil {
return err
}
}
h.Rows += len(rows)
}
filter := queryLeaseFilter(l)
filter["searchGlobalRevision"] = state.Global

View File

@ -140,6 +140,7 @@ func (s *MongoStore) restoreWorkforceStage(ctx context.Context, l queryModelLeas
return nil
}
func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLease, org string, revision int64) error {
started := time.Now()
c := s.db.Collection(workforceFactCollection)
key := workforceFactKey(org)
var control struct {
@ -169,6 +170,7 @@ func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLeas
return err
}
facts := workforcestats.Materialize(d)
log.Printf("workforce_fact_phase stage=loaded elapsed_ms=%d events=%d slices=%d logical_rows=%d", time.Since(started).Milliseconds(), len(d.Events), len(d.Slices), len(facts))
type previous struct {
ID string `bson:"_id"`
Key string `bson:"key"`
@ -194,17 +196,33 @@ func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLeas
return err
}
models := []mongo.WriteModel{}
batchBytes := 0
batchRows, confirmedRows, confirmedBytes := 0, 0, 0
defer func() {
log.Printf("workforce_fact_phase stage=write confirmed_rows=%d confirmed_bytes=%d elapsed_ms=%d", confirmedRows, confirmedBytes, time.Since(started).Milliseconds())
}()
changed := false
flush := func() error {
if len(models) == 0 {
return nil
}
batch := models
rows, bytes := batchRows, batchBytes
if err := s.paceQueryBuild(ctx, batchBytes); err != nil {
return err
}
models = nil
return s.workforceFactTransaction(ctx, l, func(sc mongo.SessionContext) error {
batchBytes = 0
batchRows = 0
err := s.workforceFactTransaction(ctx, l, func(sc mongo.SessionContext) error {
_, err := c.BulkWrite(sc, batch, options.BulkWrite().SetOrdered(true))
return err
})
if err == nil {
confirmedRows += rows
confirmedBytes += bytes
}
return err
}
for _, fact := range facts {
// Keep comments/text in authority. Source hashes and exact problem identities
@ -222,6 +240,7 @@ func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLeas
changed = true
if found {
models = append(models, mongo.NewUpdateOneModel().SetFilter(bson.M{"_id": prev.ID}).SetUpdate(bson.M{"$set": bson.M{"toEpoch": epoch, "retiredAt": time.Now().UTC()}}))
batchBytes += 256
}
encoded, err := bson.Marshal(fact)
if err != nil {
@ -255,7 +274,13 @@ func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLeas
doc["order"] = fact.Detail.ID + fact.Detail.UserID + fact.Detail.SupplierID
}
models = append(models, mongo.NewInsertOneModel().SetDocument(doc))
if len(models) >= 200 {
batchRows++
stored, err := bson.Marshal(doc)
if err != nil {
return err
}
batchBytes += len(stored)
if len(models) >= 200 || batchBytes >= queryBuildBatchBytes {
if err = flush(); err != nil {
return err
}
@ -264,7 +289,8 @@ func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLeas
for _, prev := range old {
changed = true
models = append(models, mongo.NewUpdateOneModel().SetFilter(bson.M{"_id": prev.ID}).SetUpdate(bson.M{"$set": bson.M{"toEpoch": epoch, "retiredAt": time.Now().UTC()}}))
if len(models) >= 200 {
batchBytes += 256
if len(models) >= 200 || batchBytes >= queryBuildBatchBytes {
if err = flush(); err != nil {
return err
}

View File

@ -0,0 +1,94 @@
package store
import (
"fmt"
"sort"
"go.mongodb.org/mongo-driver/bson"
)
// Group only IDENTICAL version sets. Flattening all revisions into one global
// $in would read unrelated historical versions. Rare sets retain the old OR
// batches, so fragmented legacy histories do not turn into one query per task.
func workforceSliceReadFilters(versions map[string]map[int]bool, grouped bool) []bson.M {
type group struct {
ids []string
revisions []int
}
groups := map[string]*group{}
ids := make([]string, 0, len(versions))
for id := range versions {
ids = append(ids, id)
}
sort.Strings(ids)
if !grouped {
return workforceLegacySliceFilters(nil, ids, versions)
}
for _, id := range ids {
revs := []int{}
for v := range versions[id] {
revs = append(revs, v)
}
sort.Ints(revs)
key := fmt.Sprint(revs)
g := groups[key]
if g == nil {
g = &group{revisions: revs}
groups[key] = g
}
g.ids = append(g.ids, id)
}
filters := []bson.M{}
keys := []string{}
for k := range groups {
keys = append(keys, k)
}
sort.Strings(keys)
left := []string{}
for _, key := range keys {
g := groups[key]
if len(g.ids) < 200 {
left = append(left, g.ids...)
continue
}
batch := []string{}
bytes := 128 + len(g.revisions)*16
flush := func() {
if len(batch) > 0 {
filters = append(filters, bson.M{"taskId": bson.M{"$in": batch}, "revision": bson.M{"$in": g.revisions}})
}
batch = nil
bytes = 128 + len(g.revisions)*16
}
for _, id := range g.ids {
if len(batch) >= 2000 || (len(batch) > 0 && bytes+len(id)+16 > 256<<10) {
flush()
}
batch = append(batch, id)
bytes += len(id) + 16
}
flush()
}
sort.Strings(left)
return workforceLegacySliceFilters(filters, left, versions)
}
func workforceLegacySliceFilters(filters []bson.M, ids []string, versions map[string]map[int]bool) []bson.M {
clauses := bson.A{}
for _, id := range ids {
revs := []int{}
for v := range versions[id] {
revs = append(revs, v)
}
sort.Ints(revs)
clauses = append(clauses, bson.M{"taskId": id, "revision": bson.M{"$in": revs}})
if len(clauses) == 200 {
filters = append(filters, bson.M{"$or": clauses})
clauses = nil
}
}
if len(clauses) > 0 {
filters = append(filters, bson.M{"$or": clauses})
}
return filters
}

View File

@ -0,0 +1,26 @@
package store
import (
"fmt"
"testing"
)
func TestQueryBuildSlicePlansAvoidPerTaskFanout(t *testing.T) {
fragmented := map[string]map[int]bool{}
for i := 0; i < 601; i++ {
fragmented[fmt.Sprint(i)] = map[int]bool{i: true}
}
if got := len(workforceSliceReadFilters(fragmented, true)); got != 4 {
t.Fatal("rare histories became one query per task", got)
}
dense := map[string]map[int]bool{}
for i := 0; i < 6000; i++ {
dense[fmt.Sprint(i)] = map[int]bool{1: true, 2: true}
}
if got := len(workforceSliceReadFilters(dense, true)); got != 3 {
t.Fatal("common version sets remain fragmented", got)
}
if got := len(workforceSliceReadFilters(dense, false)); got != 30 {
t.Fatal("legacy API query strategy changed", got)
}
}

View File

@ -574,39 +574,13 @@ func (s *MongoStore) LoadWorkforceStats(ctx context.Context, org string, q workf
// Reading every historical slice version of these tasks grows with
// autosaves rather than the report's output and can dominate the load.
versions := workforcestats.SliceVersions(d)
clauses := bson.A{}
taskIDs := make([]string, 0, len(versions))
for id := range versions {
taskIDs = append(taskIDs, id)
}
sort.Strings(taskIDs)
flush := func() error {
if len(clauses) == 0 {
return nil
}
_, grouped := ctx.Value(workforceBuildBudgetKey{}).(*workforceBuildBudget)
for _, filter := range workforceSliceReadFilters(versions, grouped) {
var rows []domain.SliceRevisionRecord
err := read(s.sliceRevisions, organizationScopedFilter(org, bson.M{"$or": clauses}), fields("taskId", "revision", "segments.start", "segments.end"), &rows)
if err == nil {
d.Slices = append(d.Slices, rows...)
if err := read(s.sliceRevisions, organizationScopedFilter(org, filter), fields("taskId", "revision", "segments.start", "segments.end"), &rows); err != nil {
return d, err
}
clauses = nil
return err
}
for _, id := range taskIDs {
revisions := []int{}
for version := range versions[id] {
revisions = append(revisions, version)
}
sort.Ints(revisions)
clauses = append(clauses, bson.M{"taskId": id, "revision": bson.M{"$in": revisions}})
if len(clauses) == 200 {
if err := flush(); err != nil {
return d, err
}
}
}
if err := flush(); err != nil {
return d, err
d.Slices = append(d.Slices, rows...)
}
if err := read(s.annotationTasks, organizationScopedFilter(org, bson.M{"id": bson.M{"$in": ids}}), fields("id", "projectId", "runId", "supplierId", "templateVersion"), &d.Tasks); err != nil {

View File

@ -104,7 +104,12 @@ func MetricCounterValues(m domain.DashboardWorkforceMetric) map[string]any {
out := map[string]any{}
v := reflect.ValueOf(m)
for _, f := range workforceCounterFields {
out[f.name] = v.FieldByIndex(f.index).Interface()
counter := v.FieldByIndex(f.index)
// Mongo $sum treats an absent counter as zero. Most return facts
// have no event counters: don't journal dozens of redundant zeros.
if !counter.IsZero() {
out[f.name] = counter.Interface()
}
}
return out
}

View File

@ -7,6 +7,16 @@ import (
"testing"
)
func TestFactStorageOmitsOnlyZeroCounters(t *testing.T) {
if got := MetricCounterValues(domain.DashboardWorkforceMetric{}); len(got) != 0 {
t.Fatal("empty return contribution journals redundant counters", got)
}
m := domain.DashboardWorkforceMetric{SubmittedCount: 3, DurationSeconds: 12}
if got := MetricCounterValues(m); !reflect.DeepEqual(got, map[string]any{"submittedCount": int64(3), "durationSeconds": int64(12)}) {
t.Fatal("sparse counters changed nonzero values", got)
}
}
// Run the existing business fixtures against both paths. Source loaders have
// always narrowed event/operation scope BEFORE calling Build; retain that seam.
func scopeFactComparison(d Snapshot, q Query) Snapshot {

View File

@ -3,6 +3,7 @@
import argparse
import hashlib
import gzip
import json
import os
from pathlib import Path
@ -128,7 +129,12 @@ def main():
ap.add_argument("--waves", type=int, default=2, choices=range(1, 101))
ap.add_argument("--max-actors", type=int, default=81, choices=(81, 200, 500))
ap.add_argument("--fixture-plan", help="owned.json from the bounded in-cluster fixture")
ap.add_argument("--cold-write-probe", action="store_true", help="bounded 8MiB synthetic diagnostic; never HTTP capacity evidence")
ap.add_argument("--cold-pace-kib", type=int, choices=(0, 128), default=0)
args = ap.parse_args()
assert args.cold_write_probe or args.cold_pace_kib == 0
assert not args.cold_write_probe or args.fixture_plan, "cold diagnostic requires the owned in-cluster fixture"
test_name = "TestQueryReplicaColdWriteProbeLocal" if args.cold_write_probe else TEST_NAME
out = Path(args.output).resolve()
out.mkdir(parents=True, exist_ok=False)
k = ["kubectl", "--kubeconfig", str(Path(args.kubeconfig).resolve())]
@ -158,8 +164,8 @@ def main():
assert api_binary.startswith("/synthetic/query-http") and api_binary.endswith(".test") and "/../" not in api_binary
actual = run(k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", "sha256sum", api_binary]).split()[0]
assert actual == owned["binarySHA256"]["api"]
listed = run(k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", api_binary, "-test.list=^" + TEST_NAME + "$"])
assert listed.splitlines() == [TEST_NAME], "stale binary or missing comparison test; no load started"
listed = run(k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", api_binary, "-test.list=^" + test_name + "$"])
assert listed.splitlines() == [test_name], "stale binary or missing comparison test; no load started"
source = {"Id": fixture["metadata"]["uid"], "Image": fixture["status"]["containerStatuses"][0]["imageID"]}
else:
source = json.loads(run(["docker", "inspect", CONTAINER]))[0]
@ -239,6 +245,7 @@ def main():
manifest = {"sourceCommit": run(["git", "-C", str(ROOT), "rev-parse", "HEAD"]), "sourceFingerprint": source_fingerprint(), "sourceContainer": source["Id"], "sourceImage": source["Image"], "sourceCPU": 4, "sourceMemory": 4_294_967_296, "queryPodUID": query["metadata"]["uid"], "queryImage": query["spec"]["containers"][0]["image"], "queryCPU": 1, "queryMemory": "1Gi", "tasks": args.tasks, "waves": args.waves, "maxActors": args.max_actors, "wireCompressionBothVariants": os.getenv("QUERY_COMPARE_WIRE_COMPRESSION", ""), "minimumPoolBothVariants": os.getenv("QUERY_COMPARE_MIN_POOL", "0"), "limits": limits, "limitations": ["source Docker VM shares CPU with existing local containers", "target has 2MiB read / 1MiB write and 64/32 IOPS hard limits", "short comparison; not 30-minute three-round production capacity acceptance", "health probes do not establish customer workflow latency"]}
manifest["baselineSamples"] = len(baseline)
manifest["baselineSeconds"] = baseline[-1]["time"] - baseline[0]["time"]
manifest["diagnostic"] = {"kind": "cold-write-probe", "payloadBytes": 8 << 20, "paceKiB": args.cold_pace_kib} if args.cold_write_probe else None
if fixture:
manifest["limitations"][0] = "source Mongo and native HTTP/projector/builder/client share one fixed 4CPU/4Gi Pod; source storage is bounded tmpfs"
manifest["sourceCPUCounter"] = "mongod process utime+stime; whole fixture Pod cgroup sampled separately"
@ -249,7 +256,9 @@ def main():
remote_report = "/synthetic/query-comparison-" + str(int(time.time())) + ".json"
if fixture:
settings = ["TMPDIR=/synthetic/tmp", "QUERY_REPLICA_ISOLATED_E2E=1", "QUERY_REPLICA_CLUSTER_TEST=1", "QUERY_REPLICA_TEST_TARGET_URI=mongodb://127.0.0.1:27229/?directConnection=true&replicaSet=query-lab-rs", "QUERY_COMPARE_IN_CLUSTER=1", "QUERY_COMPARE_TASKS=" + str(args.tasks), "QUERY_COMPARE_WAVES=" + str(args.waves), "QUERY_COMPARE_MAX_ACTORS=" + str(args.max_actors), "QUERY_COMPARE_REPORT=" + remote_report, "QUERY_COMPARE_WIRE_COMPRESSION=" + os.getenv("QUERY_COMPARE_WIRE_COMPRESSION", ""), "QUERY_COMPARE_MIN_POOL=" + os.getenv("QUERY_COMPARE_MIN_POOL", "0")]
argv = k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", "env"] + settings + [api_binary, "-test.run=^TestQueryReplicaIsolatedHTTPComparisonLocal$", "-test.count=1", "-test.timeout=45m", "-test.v"]
if args.cold_write_probe:
settings += ["QUERY_REPLICA_COLD_WRITE_PROBE=1", "QUERY_COLD_PACE_KIB=" + str(args.cold_pace_kib)]
argv = k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", "env"] + settings + [api_binary, "-test.run=^" + test_name + "$", "-test.count=1", "-test.timeout=45m", "-test.v"]
with (out / "test.log").open("w") as log:
child = subprocess.Popen(argv, cwd=ROOT, env=env, stdout=log, stderr=subprocess.STDOUT, start_new_session=True)
bad = 0
@ -313,13 +322,21 @@ def main():
except Exception as error:
final_health_error = str(error)
if fixture:
result = subprocess.run(k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", "cat", remote_report], capture_output=True, text=True, timeout=30)
result = subprocess.run(k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", "gzip", "-c", remote_report], capture_output=True, timeout=30)
if result.returncode == 0:
(out / "comparison.json").write_text(result.stdout)
(out / "comparison.json").write_bytes(gzip.decompress(result.stdout))
changed = source_fingerprint() != manifest["sourceFingerprint"]
exit_record = {"returncode": child.returncode, "sourceChanged": changed, "coverageComplete": False, "finalHealthError": final_health_error}
try:
validate_comparison(out / "comparison.json", args.tasks, args.waves, args.max_actors)
if args.cold_write_probe:
probe = json.loads((out / "comparison.json").read_text())
assert probe["kind"] == "cold-write-probe" and probe["capacityAccepted"] is False
assert probe["payloadBytes"] == 8 << 20 and probe["completedBatches"] == 32
assert probe["majority"] is True and probe["journal"] is True and probe["heartbeatTimeoutMs"] == 3000
assert probe["paceKiB"] == args.cold_pace_kib
assert len(probe["samples"]) >= 34
else:
validate_comparison(out / "comparison.json", args.tasks, args.waves, args.max_actors)
exit_record["coverageComplete"] = True
except (OSError, ValueError, KeyError, AssertionError) as error:
exit_record["coverageError"] = str(error)

View File

@ -23,6 +23,7 @@ CASES = ('internal/store/query_replica_local_test.go',
'internal/store/query_replica_valkey_local_test.go',
'internal/store/workforce_report_cache_local_test.go',
'internal/store/workforce_facts_local_test.go',
'internal/store/query_cold_write_local_test.go',
'internal/store/query_pool_buckets_local_test.go')

View File

@ -608,9 +608,14 @@
"frontend": [],
"go": {
"./internal/api": ["^TestHumanPoolSummary", "^TestHumanPoolCounts", "^TestHumanDisplay", "^TestRouteSnapshot$"],
"./internal/store": ["^TestPoolSearchMetadata"]
"./internal/store": ["^TestPoolSearchMetadataOwnsOnlyNames$"]
},
"task": [
{
"argv": ["go", "test", "./internal/store", "-run", "^TestPoolSearchMetadataWriteCost$", "-count=1", "-v"],
"env": { "SUBMISSION_LOCK_COST_TEST": "1" },
"tests": true
},
{
"argv": ["npm", "run", "test:go:gate"],
"tests": true
@ -1015,6 +1020,7 @@
"^internal/api/query_replica.*\\.go$",
"^internal/store/workforce_(report_cache|heap_diagnostic|facts|return_page).*\\.go$",
"^internal/store/query_pool_buckets.*\\.go$",
"^internal/store/query_cold_write.*\\.go$",
"^internal/api/query_fact_routes_test\\.go$",
"^internal/api/workforce_compact_report_test\\.go$",
"^internal/testsupport/query_replica_lab.*\\.go$",