forked from longtao.wu/droboticflow
⚡ Pace query cold builds and recover confirmed leases
This commit is contained in:
parent
c20dbacab6
commit
c83f332ba9
@ -190,3 +190,51 @@ IO保护新增严格的FUSE归因分支,并有先失败后通过的回归:
|
||||
下一步先处理独立查询库冷构建的写入节奏、共享控制写和失败恢复:记录每类后台写入/确认等待、检查点及限速窗口,减少重复写和突发积压;保持多数提交和版本隔离,不靠延长HTTP超时、降低写持久性或扩大IO限额掩盖问题。完成后再运行同规模两轮;任何更长时间或更大资源对照需另行界定。之前扩大20轮被自动审批以共享节点持续负载风险拒绝,未执行。
|
||||
|
||||
本轮自有合成源Pod及两条策略已按UID清理,桥接按PID/可执行文件身份停止,配额CAS恢复1Pod/1CPU/1Gi。25Gi卷及持久标记保留,查询Pod仍按3600秒deadline退出,未延长;清理证据 `io-rca/fixture-6/cleanup.json`。本批三项功能实现与本地回归完成,完整规模性能验收仍不通过,不准备生产发布。
|
||||
|
||||
|
||||
## 冷构建确认等待:根因与写入入口补齐(2026-11-13)
|
||||
|
||||
以 `c20dbaca` 为任务起点继续本地实施,不部署、不切生产路由、不改客户数据。联网新鲜度确认工作树包含 `origin/dev` 和线上提交。
|
||||
|
||||
证据将本次等待定位到独立查询库受限IO中的 journal/检查点竞争,叠加未受控的数据写入和失败处理放大,并非原权威Mongo的4核CPU本身造成这些确认等待。`scale-81-9-cold-recovery/slow-metadata.json` 中搜索读表一次200行插入耗时31,844ms,其中31,843ms是写确认等待,CPU约1.4ms;该轮统计事实尚未开始写入。搜索写入绕过了最初的事实/BLOB节奏控制,生产读表还存在2MiB单块突发。因此仅处理统计事实或单纯延迟提交,不能算解决了所有入口。
|
||||
|
||||
8MiB随机合成诊断保留 majority+j、三秒心跳和原资源/IO保护,不是HTTP容量证据:
|
||||
|
||||
| 诊断 | 总时间 | 心跳最长 | 心跳超时 | 期间检查点累计时间增量 |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| 直接提交32批 | 21.83秒 | 1,938ms | 0 | 7.55秒 |
|
||||
| 同二进制、128KiB/s提交 | 64.01秒 | 3,002ms | 7 | 29.98秒 |
|
||||
|
||||
限速诊断跨过一次检查点,超时与检查点执行、journal同步计数停滞重合;这是否定“固定字节限速单独足够”的证据,不将不同检查点相位解释为严格稳态吞吐收益。宿主盘忙碌很低,而查询cgroup写入接近本次硬限额;`scale-81-9`采样峰值约30写IOPS、约1MiB/s。采样窗口允许短暂突发,不是毫秒级严格速率保证。Mongo多数提交默认等待磁盘journal的语义保持不变([官方写关注说明](https://www.mongodb.com/docs/manual/reference/write-concern/)、[journal说明](https://www.mongodb.com/docs/manual/core/journaling/))。
|
||||
|
||||
修正范围:
|
||||
|
||||
- 仅显式验证的独立查询库构建器安装共享写入预算,默认512KiB/s逻辑载荷、64KiB预算粒度,覆盖搜索行、生产读表块、统计事实和报表分片;搜索/事实按约64KiB封批,单条记录不截断。生产块为128KiB,报表分片保持原上限,因此预算不保证每次物理写入都小于64KiB。检查点执行时后台数据写入等待,控制续租不进入该等待队列;不在Mongo写事务中等待。监控读取有250ms预算、约1秒复用;不支持/无监控权限时保留字节预算,不增加客户作业准入证书。逻辑节奏不能代替内核硬限额或生产容量验收。
|
||||
- 搜索批量按字节拆分且完整发布;新生产读表块使用128KiB。清单新增可选 `chunkBytes`,缺失时按原2MiB读取,旧数据无需改写。旧API不能读取某些新多块清单,实际发布须先升级读取方、后启新构建器;回退先关闭新读取,不能将旧API直接指向新清单。
|
||||
- 统计事实省去零值计数,Mongo聚合仍按零处理;不改统计归属、日期、去重或正文。已有同哈希事实不为压缩而强制重写。
|
||||
- 每个报表分片不再额外写一次续租,仅读取相同归属/代次/到期条件,独立心跳和最终发布仍多数提交。暂时性续租超时只在最后一次已确认的原租约有效期内有限重试;明确失权或原租约过期立即取消。单次三秒、原60秒租约、90秒报表构建及四分钟就绪等待均未延长。等待后重新计算租约校验时间,防止用等待前时间继续写入。
|
||||
- 增加阶段耗时及搜索构建行数元数据,不输出业务正文或凭证。诊断报告采用压缩传输;限速小探针第一次原始报告下载失败,原 `exit.json`保留,通过只读gzip恢复的报告及补充验证另存,不改写为HTTP通过。
|
||||
|
||||
先失败后通过的回归:一次续租超时错误取消可恢复构建、每片重复续租写入;另验证失权立即停止、原租约到期、检查点等待取消/恢复及期间控制写可用、等待后过期禁止写入、搜索批次和结果完整性、旧大块与新小块往返等价。真实本机Mongo配合race通过。完整固定源码任务检查 `verify-1794578222366219000/report.json` 所选检查通过,包括看板、Go已知失败基线、查询服务、81人本机读取冒烟和查询副本 `--race --cache`。此前测试标签误匹配造成的跳过,以及一次原搜索MaxTimeMS失败均保留;后者原阈值下定点连续3次及完整专项重跑通过,未通过放宽超时修正。
|
||||
|
||||
`scale-81-9-cold-recovery` 保留为失败:回填318.133秒、一次租约恢复,四分钟物化就绪超时,未进入HTTP阶段,没有p95或源CPU节省结论。其源Pod/策略/桥接已按身份清理并恢复配额。随后在相同25Gi卷、镜像、1CPU/1Gi和原IO预算下再次验证冷Pod重建,持久标记保留;证据 `query25-cold-writers-resume/`。新一轮固定候选 `scale-81-10-all-writers` 仍为26,218任务、81会话、两轮;以最终独立记录判断,不将本节实现和本地通过当成集群验收。
|
||||
|
||||
|
||||
### 固定预算下的后续验证与重复读取修正
|
||||
|
||||
`scale-81-10-all-writers` 使用256KiB/s逻辑预算:搜索26,218行完成于140.694秒,随后统计阶段97.976秒后被原四分钟就绪预算取消;回填328.573秒,未进入HTTP。`scale-81-11-balanced-writers` 将逻辑预算调到512KiB/s,内核硬IO上限、资源、持久性和超时不变:搜索92.939秒,统计阶段145.833秒后仍未就绪;回填393.224秒。两轮均保留失败,不能据此报告查询达标或CPU节省。它们表明搜索写突发得到约束后,还需分离统计加载、计算与分批确认等待。
|
||||
|
||||
进一步将统计构建阶段日志拆为历史加载/物化完成及已确认写入行数、字节数。旧历史加载为每200个任务生成一个复杂OR查询;后台构建现在仅合并**完全相同的历史版本集合**,每组最多2,000个任务ID并限制查询字节预算;零散版本仍保留原200条OR分批,避免每任务一次查询。权威请求路径继续原读取方式,不扩大读取的任务/版本组合。
|
||||
|
||||
真实隔离Mongo回归先在原实现复现6次读取,再验证新实现2次,1,200条目标切片及历史统计结果等价;6,000条共同版本集纯计划由30批降为3批,601条零散版本保留4批。首次定点绿色运行遇到原MaxTimeMS失败,原记录保留,未放宽阈值;完整相关race回归及完整查询副本runner随后通过。最新快速检查 `verify-1794581789355119000/report.json` 所选检查通过,无跳过;格式检查通过。完整任务检查中未受改动影响的浏览器/基线证据沿用前述固定记录,最新历史读取、写入和恢复相关用例重新运行。
|
||||
|
||||
最终同规模记录为 `scale-81-12-grouped-slices`:仍为26,218任务、81会话、两轮,冷构建保持原四分钟就绪、三秒单次续期及多数提交/journal要求;结果在下节补充。单次短测不替代长稳态或生产容量验收。
|
||||
|
||||
|
||||
### 最新同规模对照:环境变更保护停止,未形成性能结论
|
||||
|
||||
`scale-81-12-grouped-slices` 在原资源/IO预算下启动,历史回填到80批约145.2秒,发生一次正常租约恢复,尚未进入后台读表冷构建和HTTP对照。15:06:48,`droboticflow-fanyu` 新增 `wishforge-build-api-e9089081` 构建Pod(调度在 `.5` 节点),全量Pod集合保护检测变化并自动停止。只读核对显示当前API、处理器、校验器和Worker重启数均为0,未见本轮期间重新创建;不能把新增构建Pod解释为本测试导致业务故障,也不能据此放松保护阈值。该轮没有可用p95、结果等价或源CPU下降比例,性能验收仍待完成。
|
||||
|
||||
证据:`scale-81-12-grouped-slices/stopped.json`、`production-state-readonly.json`、`manifest.json`和原始 `test.log`。测试源Pod由保护器按UID删除;随后桥接/两条策略按身份清理,配额CAS恢复原1Pod/1CPU/1Gi。仅删除本轮精确随机合成查询数据库,保留25Gi文件系统、持久标记与查询Pod原3600秒deadline,见 `fixture-9/cleanup.json` 与 `synthetic-db-cleanup.json`。没有修改生产数据、线上读路由、资源、超时或持久性。
|
||||
|
||||
下一步在无并发构建/发布的稳定窗口重跑原规模两轮,先确认冷构建在原期限内完成,再判断HTTP正确性、p95与源MongoCPU/扫描下降。未完成这一证据前不准备上线;也不通过扩大IO、延长超时、关闭journal或将未确认写入当成功来取得通过。
|
||||
|
||||
134
internal/api/query_replica_cold_write_local_test.go
Normal file
134
internal/api/query_replica_cold_write_local_test.go
Normal file
@ -0,0 +1,134 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/testsupport"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/mongo"
|
||||
"go.mongodb.org/mongo-driver/mongo/options"
|
||||
"go.mongodb.org/mongo-driver/mongo/writeconcern"
|
||||
)
|
||||
|
||||
// A bounded diagnostic, not HTTP capacity evidence. The outer runner retains
|
||||
// the existing node/production health guards. Never accepts a production URI.
|
||||
func TestQueryReplicaColdWriteProbeLocal(t *testing.T) {
|
||||
if os.Getenv("QUERY_REPLICA_COLD_WRITE_PROBE") != "1" {
|
||||
t.Skip("explicit isolated cold-write diagnostic only")
|
||||
}
|
||||
if os.Getenv("QUERY_REPLICA_CLUSTER_TEST") != "1" {
|
||||
t.Fatal("owned synthetic cluster required")
|
||||
}
|
||||
pace := isolatedComparisonInt(t, "QUERY_COLD_PACE_KIB", 0, 0, 128)
|
||||
if pace != 0 && pace != 128 {
|
||||
t.Fatal("only unpaced / 128KiB per second diagnostics")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 3*time.Minute)
|
||||
defer cancel()
|
||||
c, err := mongo.Connect(ctx, options.Client().ApplyURI(testsupport.QueryReplicaTargetURI(t)).SetMaxPoolSize(6).SetWriteConcern(writeconcern.New(writeconcern.WMajority(), writeconcern.J(true))))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer c.Disconnect(context.Background())
|
||||
var marker bson.M
|
||||
if err = c.Database("query25_synthetic").Collection("marker").FindOne(ctx, bson.M{"_id": "query25-confirmed"}).Decode(&marker); err != nil || marker["value"] != "preserve-query25-v1" {
|
||||
t.Fatal("owned target marker absent", err)
|
||||
}
|
||||
db := c.Database(domain.NewID("cold_write_synthetic"))
|
||||
defer func() {
|
||||
cleanup, stop := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer stop()
|
||||
if e := db.Drop(cleanup); e != nil {
|
||||
t.Error("synthetic cleanup", e)
|
||||
}
|
||||
}()
|
||||
if _, err = db.Collection("control").InsertOne(ctx, bson.M{"_id": "heartbeat", "at": time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var mu sync.Mutex
|
||||
samples := []bson.M{}
|
||||
record := func(v bson.M) { mu.Lock(); samples = append(samples, v); mu.Unlock() }
|
||||
status := func() {
|
||||
short, stop := context.WithTimeout(ctx, 3*time.Second)
|
||||
defer stop()
|
||||
var s bson.M
|
||||
if e := c.Database("admin").RunCommand(short, bson.M{"serverStatus": 1}).Decode(&s); e == nil {
|
||||
wt, _ := s["wiredTiger"].(bson.M)
|
||||
record(bson.M{"kind": "storage", "at": time.Now().UTC(), "log": wt["log"], "checkpoint": wt["checkpoint"], "cache": wt["cache"]})
|
||||
}
|
||||
}
|
||||
status()
|
||||
stopHeartbeat := make(chan struct{})
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
tick := time.NewTicker(time.Second)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-stopHeartbeat:
|
||||
return
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-tick.C:
|
||||
short, stop := context.WithTimeout(ctx, 3*time.Second)
|
||||
start := time.Now()
|
||||
_, e := db.Collection("control").UpdateOne(short, bson.M{"_id": "heartbeat"}, bson.M{"$set": bson.M{"at": time.Now()}})
|
||||
stop()
|
||||
record(bson.M{"kind": "heartbeat", "at": time.Now().UTC(), "ms": float64(time.Since(start).Microseconds()) / 1000, "failed": e != nil})
|
||||
status()
|
||||
}
|
||||
}
|
||||
}()
|
||||
var stopOnce sync.Once
|
||||
stop := func() { stopOnce.Do(func() { close(stopHeartbeat); <-done }) }
|
||||
defer stop()
|
||||
started := time.Now()
|
||||
completed := 0
|
||||
for batch := 0; batch < 32 && ctx.Err() == nil; batch++ {
|
||||
rows := []any{}
|
||||
for i := 0; i < 8; i++ {
|
||||
payload := make([]byte, 32768)
|
||||
if _, e := rand.Read(payload); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
rows = append(rows, bson.M{"_id": batch*8 + i, "payload": payload})
|
||||
}
|
||||
start := time.Now()
|
||||
_, e := db.Collection("facts").InsertMany(ctx, rows)
|
||||
record(bson.M{"kind": "batch", "at": time.Now().UTC(), "ms": float64(time.Since(start).Microseconds()) / 1000, "failed": e != nil})
|
||||
if e != nil {
|
||||
break
|
||||
}
|
||||
completed++
|
||||
if pace > 0 {
|
||||
due := started.Add(time.Duration(completed) * 2 * time.Second)
|
||||
if wait := time.Until(due); wait > 0 {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
case <-time.After(wait):
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
stop()
|
||||
status()
|
||||
report := bson.M{"kind": "cold-write-probe", "paceKiB": pace, "payloadBytes": 8 << 20, "completedBatches": completed, "majority": true, "journal": true, "heartbeatTimeoutMs": 3000, "seconds": time.Since(started).Seconds(), "samples": samples, "capacityAccepted": false}
|
||||
raw, err := json.MarshalIndent(report, "", " ")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = os.WriteFile(os.Getenv("QUERY_COMPARE_REPORT"), raw, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if completed != 32 {
|
||||
t.Fatal("bounded diagnostic incomplete", completed, ctx.Err())
|
||||
}
|
||||
}
|
||||
64
internal/outputsnapshot/lease_recovery.go
Normal file
64
internal/outputsnapshot/lease_recovery.go
Normal file
@ -0,0 +1,64 @@
|
||||
package outputsnapshot
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log"
|
||||
"time"
|
||||
)
|
||||
|
||||
func transientLeaseError(err error) bool {
|
||||
var timeout interface{ Timeout() bool }
|
||||
return errors.Is(err, context.DeadlineExceeded) || (errors.As(err, &timeout) && timeout.Timeout())
|
||||
}
|
||||
|
||||
// An acknowledgement timeout is not evidence that ownership was lost. Retry
|
||||
// only while the LAST CONFIRMED original lease remains valid. All backend
|
||||
// stages/publications still independently fence owner, epoch and expiry.
|
||||
func (r *Reader) keepBuildLease(ctx context.Context, l Lease, acquiredAt time.Time, cancel context.CancelFunc, stop <-chan struct{}) {
|
||||
confirmedUntil := acquiredAt.Add(LeaseDuration)
|
||||
delay, retry := 10*time.Second, time.Second
|
||||
for {
|
||||
remaining := confirmedUntil.Sub(r.now())
|
||||
if remaining <= 0 {
|
||||
log.Printf("output_snapshot_lease_renew_failed epoch=%d result=confirmed_lease_expired", l.Epoch)
|
||||
cancel()
|
||||
return
|
||||
}
|
||||
timer := time.NewTimer(min(delay, remaining))
|
||||
select {
|
||||
case <-stop:
|
||||
timer.Stop()
|
||||
return
|
||||
case <-ctx.Done():
|
||||
timer.Stop()
|
||||
return
|
||||
case <-timer.C:
|
||||
}
|
||||
at := r.now()
|
||||
remaining = confirmedUntil.Sub(at)
|
||||
if remaining <= 0 {
|
||||
cancel()
|
||||
return
|
||||
}
|
||||
renewal, finish := context.WithTimeout(ctx, min(3*time.Second, remaining))
|
||||
ok, err := r.backend.RenewOutputSnapshot(renewal, l, at)
|
||||
finish()
|
||||
if err == nil && ok {
|
||||
confirmedUntil = at.Add(LeaseDuration)
|
||||
delay, retry = 10*time.Second, time.Second
|
||||
continue
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
if err != nil && transientLeaseError(err) && r.now().Before(confirmedUntil) {
|
||||
log.Printf("output_snapshot_lease_renew_retry epoch=%d remaining_ms=%d", l.Epoch, confirmedUntil.Sub(r.now()).Milliseconds())
|
||||
delay, retry = retry, min(retry*2, 10*time.Second)
|
||||
continue
|
||||
}
|
||||
log.Printf("output_snapshot_lease_renew_failed epoch=%d renewed=%t error=%v", l.Epoch, ok, err)
|
||||
cancel()
|
||||
return
|
||||
}
|
||||
}
|
||||
89
internal/outputsnapshot/lease_recovery_test.go
Normal file
89
internal/outputsnapshot/lease_recovery_test.go
Normal file
@ -0,0 +1,89 @@
|
||||
package outputsnapshot
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"droboticflow/internal/workforcestats"
|
||||
)
|
||||
|
||||
type interruptedRenewal struct {
|
||||
MemoryBackend
|
||||
calls atomic.Int32
|
||||
recovered chan struct{}
|
||||
lost bool
|
||||
}
|
||||
|
||||
func TestSnapshotRenewalRetryNeverExtendsUnconfirmedLease(t *testing.T) {
|
||||
base := time.Now()
|
||||
b := &MemoryBackend{}
|
||||
r := NewReader(b, nil)
|
||||
defer r.Close()
|
||||
r.now = func() time.Time { return base.Add(LeaseDuration) }
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
r.keepBuildLease(ctx, Lease{Owner: "old", Epoch: 1}, base, cancel, make(chan struct{}))
|
||||
if !errors.Is(ctx.Err(), context.Canceled) {
|
||||
t.Fatal("expired unconfirmed ownership remained active")
|
||||
}
|
||||
}
|
||||
|
||||
func (b *interruptedRenewal) RenewOutputSnapshot(ctx context.Context, l Lease, now time.Time) (bool, error) {
|
||||
if b.calls.Add(1) == 1 {
|
||||
if b.lost {
|
||||
return false, nil
|
||||
}
|
||||
return false, context.DeadlineExceeded
|
||||
}
|
||||
ok, err := b.MemoryBackend.RenewOutputSnapshot(ctx, l, now)
|
||||
if ok && err == nil {
|
||||
select {
|
||||
case <-b.recovered:
|
||||
default:
|
||||
close(b.recovered)
|
||||
}
|
||||
}
|
||||
return ok, err
|
||||
}
|
||||
|
||||
func TestSnapshotBuildRecoversTransientRenewalWithinOriginalLease(t *testing.T) {
|
||||
b := &interruptedRenewal{recovered: make(chan struct{})}
|
||||
r := NewReader(b, func(ctx context.Context, _ string, _ []string) (workforcestats.OutputSnapshot, error) {
|
||||
select {
|
||||
case <-b.recovered:
|
||||
return workforcestats.OutputSnapshot{}, nil
|
||||
case <-ctx.Done():
|
||||
return workforcestats.OutputSnapshot{}, ctx.Err()
|
||||
}
|
||||
})
|
||||
defer r.Close()
|
||||
ctx, stop := context.WithTimeout(r.ctx, 20*time.Second)
|
||||
defer stop()
|
||||
r.ctx = ctx
|
||||
key := scopeKey("org", []string{"p"})
|
||||
r.build(key, "org", []string{"p"})
|
||||
s, _ := b.OutputSnapshotState(t.Context())
|
||||
if b.calls.Load() < 2 || s.Heads[key].Current == nil || s.Heads[key].Failed {
|
||||
t.Fatalf("a single uncertain renewal discarded a recoverable build: calls=%d published=%t failed=%t", b.calls.Load(), s.Heads[key].Current != nil, s.Heads[key].Failed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSnapshotBuildStopsImmediatelyAfterDefinitiveLeaseLoss(t *testing.T) {
|
||||
b := &interruptedRenewal{recovered: make(chan struct{}), lost: true}
|
||||
r := NewReader(b, func(ctx context.Context, _ string, _ []string) (workforcestats.OutputSnapshot, error) {
|
||||
<-ctx.Done()
|
||||
return workforcestats.OutputSnapshot{}, ctx.Err()
|
||||
})
|
||||
defer r.Close()
|
||||
ctx, stop := context.WithTimeout(r.ctx, 15*time.Second)
|
||||
defer stop()
|
||||
r.ctx = ctx
|
||||
key := scopeKey("org", []string{"p"})
|
||||
r.build(key, "org", []string{"p"})
|
||||
s, _ := b.OutputSnapshotState(t.Context())
|
||||
if b.calls.Load() != 1 || s.Heads[key].Current != nil || !s.Heads[key].Failed {
|
||||
t.Fatalf("lost owner retried or published: calls=%d state=%+v", b.calls.Load(), s.Heads[key])
|
||||
}
|
||||
}
|
||||
@ -186,7 +186,8 @@ func (r *Reader) request(key, org string, projects []string, head Head, now time
|
||||
func (r *Reader) build(key, org string, projects []string) {
|
||||
ctx, cancel := context.WithTimeout(r.ctx, BuildTimeout)
|
||||
defer cancel()
|
||||
lease, ok, err := r.backend.AcquireOutputSnapshot(ctx, key, r.now())
|
||||
acquiredAt := r.now()
|
||||
lease, ok, err := r.backend.AcquireOutputSnapshot(ctx, key, acquiredAt)
|
||||
if err != nil || !ok {
|
||||
return
|
||||
}
|
||||
@ -214,25 +215,7 @@ func (r *Reader) build(key, org string, projects []string) {
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
ticker := time.NewTicker(10 * time.Second)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-stop:
|
||||
return
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
renewal, finish := context.WithTimeout(ctx, 3*time.Second)
|
||||
ok, err := r.backend.RenewOutputSnapshot(renewal, lease, r.now())
|
||||
finish()
|
||||
if err != nil || !ok {
|
||||
log.Printf("output_snapshot_lease_renew_failed epoch=%d renewed=%t error=%v", lease.Epoch, ok, err)
|
||||
cancel()
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
r.keepBuildLease(ctx, lease, acquiredAt, cancel, stop)
|
||||
}()
|
||||
defer func() { close(stop); <-done }()
|
||||
asOf := r.now().UTC().Truncate(time.Millisecond)
|
||||
|
||||
@ -96,6 +96,8 @@ func (s *MongoStore) RenewOutputSnapshot(ctx context.Context, l outputsnapshot.L
|
||||
return err == nil && res.MatchedCount == 1, err
|
||||
}
|
||||
func (s *MongoStore) PublishOutputSnapshot(ctx context.Context, l outputsnapshot.Lease, g outputsnapshot.Generation, data []byte, now time.Time) (bool, error) {
|
||||
started := time.Now()
|
||||
current := func() time.Time { return now.Add(time.Since(started)) }
|
||||
if replica, err := s.reportReplica(ctx); err != nil {
|
||||
return false, err
|
||||
} else if replica != nil {
|
||||
@ -104,7 +106,10 @@ func (s *MongoStore) PublishOutputSnapshot(ctx context.Context, l outputsnapshot
|
||||
if !outputsnapshot.ValidPublication(l, g, data) {
|
||||
return false, outputsnapshot.ErrCapacity
|
||||
}
|
||||
ok, err := s.RenewOutputSnapshot(ctx, l, now)
|
||||
if err := s.paceQueryBuild(ctx, len(data)); err != nil {
|
||||
return false, err
|
||||
}
|
||||
ok, err := s.RenewOutputSnapshot(ctx, l, current())
|
||||
if err != nil || !ok {
|
||||
return false, err
|
||||
}
|
||||
@ -120,7 +125,7 @@ func (s *MongoStore) PublishOutputSnapshot(ctx context.Context, l outputsnapshot
|
||||
return false, err
|
||||
}
|
||||
head := outputsnapshot.Head{Current: &g, Previous: l.Previous, NextAttempt: g.AsOf.Add(outputsnapshot.RefreshInterval)}
|
||||
res, err := s.outputCoordinator().UpdateOne(write, outputLeaseFilter(l, now), bson.M{"$set": bson.M{"heads." + l.Key: head, "owner": "", "until": now}})
|
||||
res, err := s.outputCoordinator().UpdateOne(write, outputLeaseFilter(l, current()), bson.M{"$set": bson.M{"heads." + l.Key: head, "owner": "", "until": current()}})
|
||||
if err == nil && res.MatchedCount == 1 {
|
||||
// Same slot only: current and previous generations occupy different slots.
|
||||
// Epoch-qualified immutable keys protect against late expired writers.
|
||||
@ -169,6 +174,7 @@ func outputShardID(slot int, epoch int64, index int) string {
|
||||
return fmt.Sprintf("of-shard:%02d:%020d:%09d", slot, epoch, index)
|
||||
}
|
||||
func (s *MongoStore) StageOutputSnapshotShard(ctx context.Context, l outputsnapshot.Lease, index int, data []byte, now time.Time) (bool, error) {
|
||||
started := time.Now()
|
||||
if replica, err := s.reportReplica(ctx); err != nil {
|
||||
return false, err
|
||||
} else if replica != nil {
|
||||
@ -177,8 +183,22 @@ func (s *MongoStore) StageOutputSnapshotShard(ctx context.Context, l outputsnaps
|
||||
if l.Slot < 0 || l.Slot >= outputsnapshot.SlotCount || index < 0 || index >= outputsnapshot.MaxShards || len(data) > outputsnapshot.MaxEncodedBytes {
|
||||
return false, outputsnapshot.ErrCapacity
|
||||
}
|
||||
ok, err := s.RenewOutputSnapshot(ctx, l, now)
|
||||
if err != nil || !ok {
|
||||
if err := s.paceQueryBuild(ctx, len(data)); err != nil {
|
||||
return false, err
|
||||
}
|
||||
// Admission may wait for a checkpoint: never test the lease against the
|
||||
// timestamp from BEFORE that wait. Preserve explicitly injected test clocks.
|
||||
now = now.Add(time.Since(started))
|
||||
// The reader's heartbeat renews the lease. Per-shard renewal adds a
|
||||
// majority journal flush for every chunk without strengthening publication:
|
||||
// immutable epoch IDs cannot overwrite a successor and publish is fenced.
|
||||
// Keep the same ownership/expiry check, as in MemoryBackend, without writing.
|
||||
var owner bson.M
|
||||
err := s.outputCoordinator().FindOne(ctx, outputLeaseFilter(l, now), options.FindOne().SetProjection(bson.M{"_id": 1})).Decode(&owner)
|
||||
if err == mongo.ErrNoDocuments {
|
||||
return false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if index == 0 {
|
||||
|
||||
@ -80,7 +80,7 @@ func TestAnnotationOutputMongoSharedAcrossInstancesAndRestart(t *testing.T) {
|
||||
t.Fatalf("cross instance history rebuilt %d times", n)
|
||||
}
|
||||
}
|
||||
func TestAnnotationOutputLeaseFencingAndFixedSlots(t *testing.T) {
|
||||
func TestAnnotationOutputMongoLeaseFencingAndFixedSlots(t *testing.T) {
|
||||
db := localOutputMongo(t)
|
||||
for name, backend := range map[string]outputsnapshot.Backend{"memory": NewMemoryStore(), "mongo": NewMongoStore(db)} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
@ -121,7 +121,7 @@ func TestAnnotationOutputLeaseFencingAndFixedSlots(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnnotationOutputSourceBudgetAndPayloadProjection(t *testing.T) {
|
||||
func TestAnnotationOutputMongoSourceBudgetAndPayloadProjection(t *testing.T) {
|
||||
db := localOutputMongo(t)
|
||||
s := NewMongoStore(db)
|
||||
seedAnnotationOutput(t, s)
|
||||
@ -142,7 +142,7 @@ func TestAnnotationOutputSourceBudgetAndPayloadProjection(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnnotationOutputPrivateCollectionsCannotBypassReportScopes(t *testing.T) {
|
||||
func TestAnnotationOutputMongoPrivateCollectionsCannotBypassReportScopes(t *testing.T) {
|
||||
db := localOutputMongo(t)
|
||||
s := NewMongoStore(db)
|
||||
for _, name := range []string{outputsnapshot.StateCollection, outputsnapshot.SlotsCollection} {
|
||||
|
||||
@ -24,6 +24,7 @@ type MongoStore struct {
|
||||
querySyncEnabled atomic.Bool
|
||||
queryReplicaReads *queryReplicaReads
|
||||
queryReplicaFactStore bool
|
||||
queryBuildWrites *queryBuildWriteGate
|
||||
queryReplicaCounts queryCountCache
|
||||
queryReplicaBatchFacts queryBatchFacts
|
||||
queryServing *queryServingOptions
|
||||
|
||||
85
internal/store/query_build_writes.go
Normal file
85
internal/store/query_build_writes.go
Normal file
@ -0,0 +1,85 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"golang.org/x/time/rate"
|
||||
)
|
||||
|
||||
const queryBuildBatchBytes = 64 << 10
|
||||
|
||||
// This admission budget belongs ONLY to a verified independent read-store
|
||||
// builder. It never gates authority commands or customer requests. It leaves
|
||||
// IO headroom for journal/control writes; the kernel hard limits still apply.
|
||||
type queryBuildWriteGate struct {
|
||||
limiter *rate.Limiter
|
||||
mu sync.Mutex
|
||||
next time.Time
|
||||
checkpoint bool
|
||||
inspect func(context.Context) (bool, error)
|
||||
}
|
||||
|
||||
func (s *MongoStore) newQueryBuildWriteGate() *queryBuildWriteGate {
|
||||
return &queryBuildWriteGate{limiter: rate.NewLimiter(512<<10, queryBuildBatchBytes), inspect: func(ctx context.Context) (bool, error) {
|
||||
var state struct {
|
||||
WT struct {
|
||||
Checkpoint struct {
|
||||
Progress int `bson:"progress state"`
|
||||
} `bson:"checkpoint"`
|
||||
Transaction struct {
|
||||
Running int `bson:"transaction checkpoint currently running"`
|
||||
} `bson:"transaction"`
|
||||
} `bson:"wiredTiger"`
|
||||
}
|
||||
err := s.db.Client().Database("admin").RunCommand(ctx, bson.M{"serverStatus": 1}).Decode(&state)
|
||||
return state.WT.Checkpoint.Progress != 0 || state.WT.Transaction.Running != 0, err
|
||||
}}
|
||||
}
|
||||
|
||||
func (g *queryBuildWriteGate) wait(ctx context.Context, bytes int) error {
|
||||
if g == nil {
|
||||
return ctx.Err()
|
||||
}
|
||||
for bytes > 0 {
|
||||
n := min(bytes, queryBuildBatchBytes)
|
||||
if err := g.limiter.WaitN(ctx, n); err != nil {
|
||||
return err
|
||||
}
|
||||
bytes -= n
|
||||
}
|
||||
for {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
g.mu.Lock()
|
||||
if !time.Now().Before(g.next) {
|
||||
probe, stop := context.WithTimeout(ctx, 250*time.Millisecond)
|
||||
busy, err := g.inspect(probe)
|
||||
stop()
|
||||
// Unsupported metrics/monitor privileges do not become another
|
||||
// admission certificate. Keep byte pacing, never block core work.
|
||||
g.checkpoint = err == nil && busy
|
||||
g.next = time.Now().Add(time.Second)
|
||||
}
|
||||
busy, next := g.checkpoint, g.next
|
||||
g.mu.Unlock()
|
||||
if !busy {
|
||||
break
|
||||
}
|
||||
timer := time.NewTimer(max(time.Millisecond, time.Until(next)))
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
timer.Stop()
|
||||
return ctx.Err()
|
||||
case <-timer.C:
|
||||
}
|
||||
}
|
||||
return ctx.Err()
|
||||
}
|
||||
|
||||
func (s *MongoStore) paceQueryBuild(ctx context.Context, bytes int) error {
|
||||
return s.queryBuildWrites.wait(ctx, bytes)
|
||||
}
|
||||
44
internal/store/query_build_writes_test.go
Normal file
44
internal/store/query_build_writes_test.go
Normal file
@ -0,0 +1,44 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"golang.org/x/time/rate"
|
||||
)
|
||||
|
||||
func TestQueryBuildWriteWaitCancelsDuringCheckpoint(t *testing.T) {
|
||||
g := &queryBuildWriteGate{limiter: rate.NewLimiter(rate.Inf, queryBuildBatchBytes), inspect: func(context.Context) (bool, error) { return true, nil }}
|
||||
ctx, stop := context.WithTimeout(t.Context(), 20*time.Millisecond)
|
||||
defer stop()
|
||||
if err := g.wait(ctx, 1); !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatal("checkpoint waiting ignored cancellation", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryBuildWriteMissingMetricsKeepsByteBudget(t *testing.T) {
|
||||
calls := 0
|
||||
g := &queryBuildWriteGate{limiter: rate.NewLimiter(1024, queryBuildBatchBytes), inspect: func(context.Context) (bool, error) {
|
||||
calls++
|
||||
return true, errors.New("monitor permission unavailable")
|
||||
}}
|
||||
if err := g.wait(t.Context(), queryBuildBatchBytes); err != nil || calls != 1 {
|
||||
t.Fatal("optional metrics became an admission certificate", err, calls)
|
||||
}
|
||||
ctx, stop := context.WithTimeout(t.Context(), 20*time.Millisecond)
|
||||
defer stop()
|
||||
if err := g.wait(ctx, queryBuildBatchBytes); err == nil {
|
||||
t.Fatal("byte budget disappeared with monitor privilege")
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryBuildWriteBudgetIsNotInstalledOnAuthority(t *testing.T) {
|
||||
s := &MongoStore{}
|
||||
b := NewQueryReadBuilder(s)
|
||||
defer b.Close()
|
||||
if s.queryBuildWrites != nil {
|
||||
t.Fatal("authority store acquired background write admission")
|
||||
}
|
||||
}
|
||||
291
internal/store/query_cold_write_local_test.go
Normal file
291
internal/store/query_cold_write_local_test.go
Normal file
@ -0,0 +1,291 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/outputsnapshot"
|
||||
"droboticflow/internal/workforcestats"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/event"
|
||||
"golang.org/x/time/rate"
|
||||
)
|
||||
|
||||
func TestQueryBuildSliceReadsGroupEqualVersionSetsLocal(t *testing.T) {
|
||||
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
|
||||
t.Skip("owned synthetic replica required")
|
||||
}
|
||||
var reads atomic.Int32
|
||||
m := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
|
||||
if e.CommandName == "find" {
|
||||
if n, _ := e.Command.Lookup("find").StringValueOK(); n == "sliceRevisions" {
|
||||
reads.Add(1)
|
||||
}
|
||||
}
|
||||
}}
|
||||
s, _, _ := newLocalReplica(t, m, nil)
|
||||
tasks, events, slices := []any{}, []any{}, []any{}
|
||||
for i := 0; i < 1200; i++ {
|
||||
id := fmt.Sprintf("version-task-%04d", i)
|
||||
v := i%2 + 1
|
||||
tasks = append(tasks, bson.M{"id": id, "organizationId": "o", "projectId": "p"})
|
||||
events = append(events, bson.M{"id": "event-" + id, "organizationId": "o", "projectId": "p", "taskId": id, "eventType": "combined_submitted", "revision": v, "createdAt": "2026-11-01T10:00:00Z"})
|
||||
for r := 1; r <= 3; r++ {
|
||||
slices = append(slices, bson.M{"id": fmt.Sprintf("%s/%d", id, r), "organizationId": "o", "taskId": id, "revision": r, "segments": bson.A{bson.M{"start": 0, "end": r}}})
|
||||
}
|
||||
}
|
||||
for _, x := range []struct {
|
||||
c mongoCollection
|
||||
rows []any
|
||||
}{{s.annotationTasks, tasks}, {s.annotationWorklogEvents, events}, {s.sliceRevisions, slices}} {
|
||||
if _, err := x.c.InsertMany(t.Context(), x.rows); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
q := workforcestats.Query{Mode: domain.RunModeAll}
|
||||
reads.Store(0)
|
||||
old, err := s.LoadWorkforceStats(t.Context(), "o", q)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
oldReads := reads.Load()
|
||||
reads.Store(0)
|
||||
ctx := context.WithValue(t.Context(), workforceBuildBudgetKey{}, &workforceBuildBudget{})
|
||||
next, err := s.LoadWorkforceStats(ctx, "o", q)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if reads.Load() != 2 || oldReads != 6 {
|
||||
t.Fatalf("background slice queries not merged: old=%d new=%d", oldReads, reads.Load())
|
||||
}
|
||||
less := func(rows []domain.SliceRevisionRecord) {
|
||||
sort.Slice(rows, func(i, j int) bool {
|
||||
if rows[i].TaskID == rows[j].TaskID {
|
||||
return rows[i].Revision < rows[j].Revision
|
||||
}
|
||||
return rows[i].TaskID < rows[j].TaskID
|
||||
})
|
||||
}
|
||||
less(old.Slices)
|
||||
less(next.Slices)
|
||||
if len(next.Slices) != 1200 || !reflect.DeepEqual(old.Slices, next.Slices) || !reflect.DeepEqual(workforcestats.Build(old, q), workforcestats.Build(next, q)) {
|
||||
t.Fatal("grouping changed exact versions or historical statistics")
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryBuildSmallAndLegacyBlobChunksLocal(t *testing.T) {
|
||||
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
|
||||
t.Skip("owned synthetic replica required")
|
||||
}
|
||||
_, target, _ := newLocalReplica(t, nil, nil)
|
||||
data := make([]byte, 300<<10)
|
||||
if _, err := rand.Read(data); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
value := map[string]string{"payload": hex.EncodeToString(data)}
|
||||
h, err := target.stageQueryModel(t.Context(), "small", value, time.Now())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if h.ChunkBytes != queryModelBuildChunkBytes || h.Chunks < 2 {
|
||||
t.Fatal("new construction still emits large bursts", h)
|
||||
}
|
||||
var got map[string]string
|
||||
if err = target.loadQueryModel(t.Context(), "small", h, &got); err != nil || !reflect.DeepEqual(got, value) {
|
||||
t.Fatal("small chunk round trip", err)
|
||||
}
|
||||
var raw bytes.Buffer
|
||||
zw := gzip.NewWriter(&raw)
|
||||
if err = json.NewEncoder(zw).Encode(value); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = zw.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sum := sha256.Sum256(raw.Bytes())
|
||||
legacy := QueryModelHead{Generation: "legacy", Bytes: raw.Len(), Chunks: (raw.Len() + queryModelChunkBytes - 1) / queryModelChunkBytes, Digest: hex.EncodeToString(sum[:])}
|
||||
for i := 0; i < legacy.Chunks; i++ {
|
||||
if _, err = target.queryData().InsertOne(t.Context(), bson.M{"_id": queryModelBlobKey("old", legacy.Generation, i), "data": raw.Bytes()[i*queryModelChunkBytes : min((i+1)*queryModelChunkBytes, raw.Len())]}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
got = nil
|
||||
if err = target.loadQueryModel(t.Context(), "old", legacy, &got); err != nil || !reflect.DeepEqual(got, value) {
|
||||
t.Fatal("old manifest without chunkBytes lost compatibility", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryBuildSearchUsesBoundedPacedBatchesLocal(t *testing.T) {
|
||||
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
|
||||
t.Skip("owned synthetic replica required")
|
||||
}
|
||||
var maxBytes atomic.Int32
|
||||
monitor := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
|
||||
if e.CommandName != "insert" {
|
||||
return
|
||||
}
|
||||
name, _ := e.Command.Lookup("insert").StringValueOK()
|
||||
if name != "queryReadModelData" {
|
||||
return
|
||||
}
|
||||
values, _ := e.Command.Lookup("documents").Array().Values()
|
||||
size := 0
|
||||
for _, v := range values {
|
||||
size += len(v.Document())
|
||||
}
|
||||
for before := maxBytes.Load(); int32(size) > before; before = maxBytes.Load() {
|
||||
if maxBytes.CompareAndSwap(before, int32(size)) {
|
||||
break
|
||||
}
|
||||
}
|
||||
}}
|
||||
_, target, _ := newLocalReplica(t, nil, monitor)
|
||||
for i := 0; i < 300; i++ {
|
||||
if err := target.UpsertHumanItem(t.Context(), domain.HumanItemRecord{ID: fmt.Sprint(i), OrganizationID: "o", PluginID: "annotation-qa", ProjectID: "p", Status: domain.HumanItemReady, Subject: domain.HumanSubjectRef{ItemKey: strings.Repeat("synthetic-name-", 100) + fmt.Sprint(i)}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
l, ok, err := target.acquireQueryModelLease(t.Context())
|
||||
if err != nil || !ok {
|
||||
t.Fatal(ok, err)
|
||||
}
|
||||
var inspected atomic.Int32
|
||||
target.queryBuildWrites = &queryBuildWriteGate{limiter: rate.NewLimiter(rate.Inf, queryBuildBatchBytes), inspect: func(context.Context) (bool, error) { inspected.Add(1); return false, nil }}
|
||||
if err = target.buildQuerySearchModel(t.Context(), l, "o"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st, err := target.querySearchState(t.Context())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if st.Heads[queryModelKey("search", "o", nil)].Rows != 300 || inspected.Load() == 0 || maxBytes.Load() > queryBuildBatchBytes {
|
||||
t.Fatalf("search writer bypassed budget or lost rows: rows=%d inspect=%d maxBytes=%d", st.Heads[queryModelKey("search", "o", nil)].Rows, inspected.Load(), maxBytes.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryBuildShardStagingChecksLeaseWithoutHeartbeatWritesLocal(t *testing.T) {
|
||||
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
|
||||
t.Skip("owned synthetic replica required")
|
||||
}
|
||||
var renews atomic.Int32
|
||||
monitor := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
|
||||
if e.CommandName == "update" {
|
||||
if v, ok := e.Command.Lookup("update").StringValueOK(); ok && v == outputsnapshot.StateCollection {
|
||||
renews.Add(1)
|
||||
}
|
||||
}
|
||||
}}
|
||||
_, target, _ := newLocalReplica(t, nil, monitor)
|
||||
now := time.Now().UTC().Truncate(time.Millisecond)
|
||||
l, ok, err := target.AcquireOutputSnapshot(t.Context(), "cold-stage", now)
|
||||
if err != nil || !ok {
|
||||
t.Fatal(ok, err)
|
||||
}
|
||||
before := renews.Load()
|
||||
for i := 0; i < 3; i++ {
|
||||
ok, err := target.StageOutputSnapshotShard(t.Context(), l, i, []byte{byte(i)}, now.Add(time.Second))
|
||||
if err != nil || !ok {
|
||||
t.Fatal(ok, err)
|
||||
}
|
||||
}
|
||||
if got := renews.Load() - before; got != 0 {
|
||||
t.Fatalf("staging redundantly persisted %d heartbeats", got)
|
||||
}
|
||||
st, err := target.OutputSnapshotState(t.Context())
|
||||
if err != nil || !st.Until.Equal(now.Add(outputsnapshot.LeaseDuration)) {
|
||||
t.Fatalf("staging changed the existing lease: %+v %v", st, err)
|
||||
}
|
||||
if ok, err := target.StageOutputSnapshotShard(t.Context(), l, 3, []byte("expired"), now.Add(outputsnapshot.LeaseDuration+time.Second)); err != nil || ok {
|
||||
t.Fatal("expired writer staged", ok, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryBuildShardWaitsForCheckpointWithoutHoldingWriteTransactionLocal(t *testing.T) {
|
||||
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
|
||||
t.Skip("owned synthetic replica required")
|
||||
}
|
||||
var inserts atomic.Int32
|
||||
monitor := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
|
||||
if e.CommandName == "insert" {
|
||||
if v, ok := e.Command.Lookup("insert").StringValueOK(); ok && v == outputsnapshot.SlotsCollection {
|
||||
inserts.Add(1)
|
||||
}
|
||||
}
|
||||
}}
|
||||
_, target, _ := newLocalReplica(t, nil, monitor)
|
||||
l, ok, err := target.AcquireOutputSnapshot(t.Context(), "checkpoint", time.Now())
|
||||
if err != nil || !ok {
|
||||
t.Fatal(ok, err)
|
||||
}
|
||||
var busy atomic.Bool
|
||||
busy.Store(true)
|
||||
entered := make(chan struct{}, 1)
|
||||
target.queryBuildWrites = &queryBuildWriteGate{limiter: rate.NewLimiter(rate.Inf, queryBuildBatchBytes), inspect: func(context.Context) (bool, error) {
|
||||
select {
|
||||
case entered <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
return busy.Load(), nil
|
||||
}}
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
ok, e := target.StageOutputSnapshotShard(t.Context(), l, 0, []byte("complete"), time.Now())
|
||||
if e == nil && !ok {
|
||||
e = context.Canceled
|
||||
}
|
||||
done <- e
|
||||
}()
|
||||
<-entered
|
||||
if inserts.Load() != 0 {
|
||||
t.Fatal("data write entered checkpoint IO queue")
|
||||
}
|
||||
// The independent heartbeat remains available while the data producer waits.
|
||||
if ok, e := target.RenewOutputSnapshot(t.Context(), l, time.Now()); e != nil || !ok {
|
||||
t.Fatal("checkpoint pause blocked control writes", ok, e)
|
||||
}
|
||||
busy.Store(false)
|
||||
select {
|
||||
case e := <-done:
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("producer did not resume")
|
||||
}
|
||||
if inserts.Load() != 1 {
|
||||
t.Fatal("resumed staging was duplicated", inserts.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryBuildShardRechecksExpiryAfterCheckpointWaitLocal(t *testing.T) {
|
||||
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
|
||||
t.Skip("owned synthetic replica required")
|
||||
}
|
||||
_, target, _ := newLocalReplica(t, nil, nil)
|
||||
// The lease is valid at admission but expires while waiting. A timestamp
|
||||
// captured before that wait must not authorize the subsequent staging.
|
||||
now := time.Now()
|
||||
l, ok, err := target.AcquireOutputSnapshot(t.Context(), "expiring", now.Add(-outputsnapshot.LeaseDuration+500*time.Millisecond))
|
||||
if err != nil || !ok {
|
||||
t.Fatal(ok, err)
|
||||
}
|
||||
calls := 0
|
||||
target.queryBuildWrites = &queryBuildWriteGate{limiter: rate.NewLimiter(rate.Inf, queryBuildBatchBytes), inspect: func(context.Context) (bool, error) { calls++; return calls == 1, nil }}
|
||||
if ok, err = target.StageOutputSnapshotShard(t.Context(), l, 0, []byte("late"), now); err != nil || ok {
|
||||
t.Fatal("pre-wait time authorized an expired owner", ok, err)
|
||||
}
|
||||
}
|
||||
@ -33,6 +33,9 @@ type QueryReadBuilder struct {
|
||||
}
|
||||
|
||||
func NewQueryReadBuilder(s *MongoStore) *QueryReadBuilder {
|
||||
if s.queryReplicaFactStore {
|
||||
s.queryBuildWrites = s.newQueryBuildWriteGate()
|
||||
}
|
||||
r := outputsnapshot.NewStreamReader(s, s.StreamAnnotationOutput)
|
||||
return &QueryReadBuilder{store: s, dirty: map[string]bool{}, all: true, reader: r, searchDirty: map[string]bool{}, allSearch: true, sourceDigests: map[string]querySourceDigest{}}
|
||||
}
|
||||
@ -107,8 +110,14 @@ func (b *QueryReadBuilder) scopes(ctx context.Context) (map[string][]string, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org string, projects []string) error {
|
||||
func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org string, projects []string) (err error) {
|
||||
started := time.Now()
|
||||
stage := "source"
|
||||
defer func() {
|
||||
if err != nil {
|
||||
log.Printf("query_model_build result=failed stage=%s elapsed_ms=%d", stage, time.Since(started).Milliseconds())
|
||||
}
|
||||
}()
|
||||
var source struct {
|
||||
Revision int64 `bson:"productionRevision"`
|
||||
Global int64 `bson:"searchGlobalRevision"`
|
||||
@ -127,21 +136,25 @@ func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org s
|
||||
return err
|
||||
}
|
||||
key := queryModelKey("production", org, projects)
|
||||
stage = "production_blob"
|
||||
head, err := b.store.stageQueryModel(ctx, key, productionstats.Build(snapshot), asOf)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if b.store.queryReplicaFactStore {
|
||||
stage = "workforce_facts"
|
||||
if err = b.store.rebuildWorkforceFacts(buildCtx, l, org, source.Revision); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
head.SourceRevision = source.Revision
|
||||
stage = "publish"
|
||||
head.GlobalRevision = source.Global
|
||||
if err = b.store.publishQueryModelHead(ctx, l, key, head); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(projects) > 0 {
|
||||
stage = "output_prewarm"
|
||||
if _, err = b.reader.Prewarm(ctx, org, projects); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@ -24,6 +24,7 @@ import (
|
||||
|
||||
const queryModelSchema = "query-model/v3"
|
||||
const queryModelChunkBytes = 2 << 20
|
||||
const queryModelBuildChunkBytes = 128 << 10
|
||||
const queryModelMaxBytes = 64 << 20
|
||||
|
||||
type queryServingOptions struct {
|
||||
@ -71,6 +72,7 @@ func (s *MongoStore) SharedQueryCache() queryserving.Cache {
|
||||
}
|
||||
|
||||
type QueryModelHead struct {
|
||||
ChunkBytes int `bson:"chunkBytes,omitempty" json:"chunkBytes,omitempty"`
|
||||
FactEpoch int64 `bson:"factEpoch,omitempty" json:"factEpoch,omitempty"`
|
||||
SourceRevision int64 `bson:"sourceRevision" json:"sourceRevision"`
|
||||
Revision int64 `bson:"revision" json:"revision"`
|
||||
@ -168,11 +170,14 @@ func (s *MongoStore) stageQueryModel(ctx context.Context, key string, value any,
|
||||
if data.Len() > queryModelMaxBytes {
|
||||
return QueryModelHead{}, errors.New("query materialization exceeds budget")
|
||||
}
|
||||
h := QueryModelHead{Generation: domain.NewID("query-generation"), AsOf: asOf, Bytes: data.Len(), Chunks: (data.Len() + queryModelChunkBytes - 1) / queryModelChunkBytes}
|
||||
h := QueryModelHead{Generation: domain.NewID("query-generation"), AsOf: asOf, Bytes: data.Len(), ChunkBytes: queryModelBuildChunkBytes, Chunks: (data.Len() + queryModelBuildChunkBytes - 1) / queryModelBuildChunkBytes}
|
||||
sum := sha256.Sum256(data.Bytes())
|
||||
h.Digest = hex.EncodeToString(sum[:])
|
||||
for i := 0; i < h.Chunks; i++ {
|
||||
b := data.Bytes()[i*queryModelChunkBytes : min((i+1)*queryModelChunkBytes, data.Len())]
|
||||
b := data.Bytes()[i*h.ChunkBytes : min((i+1)*h.ChunkBytes, data.Len())]
|
||||
if err := s.paceQueryBuild(ctx, len(b)); err != nil {
|
||||
return QueryModelHead{}, err
|
||||
}
|
||||
_, err := s.queryData().InsertOne(ctx, bson.M{"_id": queryModelBlobKey(key, h.Generation, i), "generation": h.Generation, "data": b, "createdAt": time.Now().UTC()})
|
||||
if err != nil {
|
||||
return QueryModelHead{}, err
|
||||
@ -184,7 +189,11 @@ func (s *MongoStore) stageQueryModel(ctx context.Context, key string, value any,
|
||||
return h, nil
|
||||
}
|
||||
func (s *MongoStore) loadQueryModel(ctx context.Context, key string, h QueryModelHead, out any) error {
|
||||
if h.Chunks < 1 || h.Bytes < 1 || h.Bytes > queryModelMaxBytes || h.Chunks != (h.Bytes+queryModelChunkBytes-1)/queryModelChunkBytes {
|
||||
chunkBytes := h.ChunkBytes
|
||||
if chunkBytes == 0 {
|
||||
chunkBytes = queryModelChunkBytes
|
||||
} // Existing 2MiB manifests remain readable.
|
||||
if chunkBytes < 64<<10 || chunkBytes > queryModelChunkBytes || h.Chunks < 1 || h.Bytes < 1 || h.Bytes > queryModelMaxBytes || h.Chunks != (h.Bytes+chunkBytes-1)/chunkBytes {
|
||||
return errors.New("invalid query manifest")
|
||||
}
|
||||
var data bytes.Buffer
|
||||
@ -207,7 +216,7 @@ func (s *MongoStore) loadQueryModel(ctx context.Context, key string, h QueryMode
|
||||
cache.Put(ctx, blobKey, b, 15*time.Minute)
|
||||
}
|
||||
}
|
||||
if len(b) > queryModelChunkBytes {
|
||||
if len(b) > chunkBytes {
|
||||
return errors.New("invalid query chunk")
|
||||
}
|
||||
data.Write(b)
|
||||
|
||||
@ -5,6 +5,7 @@ import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"log"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strings"
|
||||
@ -54,6 +55,10 @@ func (s *MongoStore) buildQuerySearchModel(ctx context.Context, l queryModelLeas
|
||||
return err
|
||||
}
|
||||
h := QueryModelHead{Generation: domain.NewID("query-search"), AsOf: time.Now().UTC(), Revision: state.Revisions[key], GlobalRevision: state.Global}
|
||||
started := time.Now()
|
||||
defer func() {
|
||||
log.Printf("query_search_build rows=%d elapsed_ms=%d", h.Rows, time.Since(started).Milliseconds())
|
||||
}()
|
||||
if s.queryServing != nil && s.queryServing.search != nil {
|
||||
sum := sha256.Sum256([]byte(s.db.Name() + ":" + key + ":" + h.Generation))
|
||||
h.Collection = "dfq_" + hex.EncodeToString(sum[:])
|
||||
@ -103,6 +108,26 @@ func (s *MongoStore) buildQuerySearchModel(ctx context.Context, l queryModelLeas
|
||||
}
|
||||
docs := make([]any, 0, len(rows))
|
||||
searchDocs := make([]queryserving.SearchDocument, 0, len(rows))
|
||||
batchBytes := 0
|
||||
flush := func() error {
|
||||
if len(docs) == 0 {
|
||||
return nil
|
||||
}
|
||||
if err := s.paceQueryBuild(ctx, batchBytes); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := s.queryData().InsertMany(ctx, docs); err != nil {
|
||||
return err
|
||||
}
|
||||
if h.Collection != "" {
|
||||
if err := s.queryServing.search.Import(ctx, h.Collection, searchDocs); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
h.Rows += len(docs)
|
||||
docs, searchDocs, batchBytes = nil, nil, 0
|
||||
return nil
|
||||
}
|
||||
for _, row := range rows {
|
||||
for _, task := range row.Tasks {
|
||||
if querySafePath(task.SourceVideo) != task.SourceVideo {
|
||||
@ -121,18 +146,23 @@ func (s *MongoStore) buildQuerySearchModel(ctx context.Context, l queryModelLeas
|
||||
// Store only the conservative searchable text and identity. No
|
||||
// task body, draft, review opinion, eligibility or credential.
|
||||
id := queryserving.Key("row", h.Generation, row.ID)
|
||||
docs = append(docs, bson.M{"_id": id, "generation": h.Generation, "organizationId": normalizedOrganizationID(org), "itemId": row.ID, "text": text, "createdAt": time.Now().UTC()})
|
||||
doc := bson.M{"_id": id, "generation": h.Generation, "organizationId": normalizedOrganizationID(org), "itemId": row.ID, "text": text, "createdAt": time.Now().UTC()}
|
||||
raw, e := bson.Marshal(doc)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if batchBytes+len(raw) > queryBuildBatchBytes {
|
||||
if err = flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
docs = append(docs, doc)
|
||||
batchBytes += len(raw)
|
||||
searchDocs = append(searchDocs, queryserving.SearchDocument{ID: id, ItemID: row.ID, Grams: queryserving.IndexGrams(text)})
|
||||
}
|
||||
if _, err = s.queryData().InsertMany(ctx, docs); err != nil {
|
||||
if err = flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
if h.Collection != "" {
|
||||
if err = s.queryServing.search.Import(ctx, h.Collection, searchDocs); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
h.Rows += len(rows)
|
||||
}
|
||||
filter := queryLeaseFilter(l)
|
||||
filter["searchGlobalRevision"] = state.Global
|
||||
|
||||
@ -140,6 +140,7 @@ func (s *MongoStore) restoreWorkforceStage(ctx context.Context, l queryModelLeas
|
||||
return nil
|
||||
}
|
||||
func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLease, org string, revision int64) error {
|
||||
started := time.Now()
|
||||
c := s.db.Collection(workforceFactCollection)
|
||||
key := workforceFactKey(org)
|
||||
var control struct {
|
||||
@ -169,6 +170,7 @@ func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLeas
|
||||
return err
|
||||
}
|
||||
facts := workforcestats.Materialize(d)
|
||||
log.Printf("workforce_fact_phase stage=loaded elapsed_ms=%d events=%d slices=%d logical_rows=%d", time.Since(started).Milliseconds(), len(d.Events), len(d.Slices), len(facts))
|
||||
type previous struct {
|
||||
ID string `bson:"_id"`
|
||||
Key string `bson:"key"`
|
||||
@ -194,17 +196,33 @@ func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLeas
|
||||
return err
|
||||
}
|
||||
models := []mongo.WriteModel{}
|
||||
batchBytes := 0
|
||||
batchRows, confirmedRows, confirmedBytes := 0, 0, 0
|
||||
defer func() {
|
||||
log.Printf("workforce_fact_phase stage=write confirmed_rows=%d confirmed_bytes=%d elapsed_ms=%d", confirmedRows, confirmedBytes, time.Since(started).Milliseconds())
|
||||
}()
|
||||
changed := false
|
||||
flush := func() error {
|
||||
if len(models) == 0 {
|
||||
return nil
|
||||
}
|
||||
batch := models
|
||||
rows, bytes := batchRows, batchBytes
|
||||
if err := s.paceQueryBuild(ctx, batchBytes); err != nil {
|
||||
return err
|
||||
}
|
||||
models = nil
|
||||
return s.workforceFactTransaction(ctx, l, func(sc mongo.SessionContext) error {
|
||||
batchBytes = 0
|
||||
batchRows = 0
|
||||
err := s.workforceFactTransaction(ctx, l, func(sc mongo.SessionContext) error {
|
||||
_, err := c.BulkWrite(sc, batch, options.BulkWrite().SetOrdered(true))
|
||||
return err
|
||||
})
|
||||
if err == nil {
|
||||
confirmedRows += rows
|
||||
confirmedBytes += bytes
|
||||
}
|
||||
return err
|
||||
}
|
||||
for _, fact := range facts {
|
||||
// Keep comments/text in authority. Source hashes and exact problem identities
|
||||
@ -222,6 +240,7 @@ func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLeas
|
||||
changed = true
|
||||
if found {
|
||||
models = append(models, mongo.NewUpdateOneModel().SetFilter(bson.M{"_id": prev.ID}).SetUpdate(bson.M{"$set": bson.M{"toEpoch": epoch, "retiredAt": time.Now().UTC()}}))
|
||||
batchBytes += 256
|
||||
}
|
||||
encoded, err := bson.Marshal(fact)
|
||||
if err != nil {
|
||||
@ -255,7 +274,13 @@ func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLeas
|
||||
doc["order"] = fact.Detail.ID + fact.Detail.UserID + fact.Detail.SupplierID
|
||||
}
|
||||
models = append(models, mongo.NewInsertOneModel().SetDocument(doc))
|
||||
if len(models) >= 200 {
|
||||
batchRows++
|
||||
stored, err := bson.Marshal(doc)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
batchBytes += len(stored)
|
||||
if len(models) >= 200 || batchBytes >= queryBuildBatchBytes {
|
||||
if err = flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
@ -264,7 +289,8 @@ func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLeas
|
||||
for _, prev := range old {
|
||||
changed = true
|
||||
models = append(models, mongo.NewUpdateOneModel().SetFilter(bson.M{"_id": prev.ID}).SetUpdate(bson.M{"$set": bson.M{"toEpoch": epoch, "retiredAt": time.Now().UTC()}}))
|
||||
if len(models) >= 200 {
|
||||
batchBytes += 256
|
||||
if len(models) >= 200 || batchBytes >= queryBuildBatchBytes {
|
||||
if err = flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
94
internal/store/workforce_slice_reads.go
Normal file
94
internal/store/workforce_slice_reads.go
Normal file
@ -0,0 +1,94 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
)
|
||||
|
||||
// Group only IDENTICAL version sets. Flattening all revisions into one global
|
||||
// $in would read unrelated historical versions. Rare sets retain the old OR
|
||||
// batches, so fragmented legacy histories do not turn into one query per task.
|
||||
func workforceSliceReadFilters(versions map[string]map[int]bool, grouped bool) []bson.M {
|
||||
type group struct {
|
||||
ids []string
|
||||
revisions []int
|
||||
}
|
||||
groups := map[string]*group{}
|
||||
ids := make([]string, 0, len(versions))
|
||||
for id := range versions {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
sort.Strings(ids)
|
||||
if !grouped {
|
||||
return workforceLegacySliceFilters(nil, ids, versions)
|
||||
}
|
||||
for _, id := range ids {
|
||||
revs := []int{}
|
||||
for v := range versions[id] {
|
||||
revs = append(revs, v)
|
||||
}
|
||||
sort.Ints(revs)
|
||||
key := fmt.Sprint(revs)
|
||||
g := groups[key]
|
||||
if g == nil {
|
||||
g = &group{revisions: revs}
|
||||
groups[key] = g
|
||||
}
|
||||
g.ids = append(g.ids, id)
|
||||
}
|
||||
filters := []bson.M{}
|
||||
keys := []string{}
|
||||
for k := range groups {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
left := []string{}
|
||||
for _, key := range keys {
|
||||
g := groups[key]
|
||||
if len(g.ids) < 200 {
|
||||
left = append(left, g.ids...)
|
||||
continue
|
||||
}
|
||||
batch := []string{}
|
||||
bytes := 128 + len(g.revisions)*16
|
||||
flush := func() {
|
||||
if len(batch) > 0 {
|
||||
filters = append(filters, bson.M{"taskId": bson.M{"$in": batch}, "revision": bson.M{"$in": g.revisions}})
|
||||
}
|
||||
batch = nil
|
||||
bytes = 128 + len(g.revisions)*16
|
||||
}
|
||||
for _, id := range g.ids {
|
||||
if len(batch) >= 2000 || (len(batch) > 0 && bytes+len(id)+16 > 256<<10) {
|
||||
flush()
|
||||
}
|
||||
batch = append(batch, id)
|
||||
bytes += len(id) + 16
|
||||
}
|
||||
flush()
|
||||
}
|
||||
sort.Strings(left)
|
||||
return workforceLegacySliceFilters(filters, left, versions)
|
||||
}
|
||||
|
||||
func workforceLegacySliceFilters(filters []bson.M, ids []string, versions map[string]map[int]bool) []bson.M {
|
||||
clauses := bson.A{}
|
||||
for _, id := range ids {
|
||||
revs := []int{}
|
||||
for v := range versions[id] {
|
||||
revs = append(revs, v)
|
||||
}
|
||||
sort.Ints(revs)
|
||||
clauses = append(clauses, bson.M{"taskId": id, "revision": bson.M{"$in": revs}})
|
||||
if len(clauses) == 200 {
|
||||
filters = append(filters, bson.M{"$or": clauses})
|
||||
clauses = nil
|
||||
}
|
||||
}
|
||||
if len(clauses) > 0 {
|
||||
filters = append(filters, bson.M{"$or": clauses})
|
||||
}
|
||||
return filters
|
||||
}
|
||||
26
internal/store/workforce_slice_reads_test.go
Normal file
26
internal/store/workforce_slice_reads_test.go
Normal file
@ -0,0 +1,26 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestQueryBuildSlicePlansAvoidPerTaskFanout(t *testing.T) {
|
||||
fragmented := map[string]map[int]bool{}
|
||||
for i := 0; i < 601; i++ {
|
||||
fragmented[fmt.Sprint(i)] = map[int]bool{i: true}
|
||||
}
|
||||
if got := len(workforceSliceReadFilters(fragmented, true)); got != 4 {
|
||||
t.Fatal("rare histories became one query per task", got)
|
||||
}
|
||||
dense := map[string]map[int]bool{}
|
||||
for i := 0; i < 6000; i++ {
|
||||
dense[fmt.Sprint(i)] = map[int]bool{1: true, 2: true}
|
||||
}
|
||||
if got := len(workforceSliceReadFilters(dense, true)); got != 3 {
|
||||
t.Fatal("common version sets remain fragmented", got)
|
||||
}
|
||||
if got := len(workforceSliceReadFilters(dense, false)); got != 30 {
|
||||
t.Fatal("legacy API query strategy changed", got)
|
||||
}
|
||||
}
|
||||
@ -574,39 +574,13 @@ func (s *MongoStore) LoadWorkforceStats(ctx context.Context, org string, q workf
|
||||
// Reading every historical slice version of these tasks grows with
|
||||
// autosaves rather than the report's output and can dominate the load.
|
||||
versions := workforcestats.SliceVersions(d)
|
||||
clauses := bson.A{}
|
||||
taskIDs := make([]string, 0, len(versions))
|
||||
for id := range versions {
|
||||
taskIDs = append(taskIDs, id)
|
||||
}
|
||||
sort.Strings(taskIDs)
|
||||
flush := func() error {
|
||||
if len(clauses) == 0 {
|
||||
return nil
|
||||
}
|
||||
_, grouped := ctx.Value(workforceBuildBudgetKey{}).(*workforceBuildBudget)
|
||||
for _, filter := range workforceSliceReadFilters(versions, grouped) {
|
||||
var rows []domain.SliceRevisionRecord
|
||||
err := read(s.sliceRevisions, organizationScopedFilter(org, bson.M{"$or": clauses}), fields("taskId", "revision", "segments.start", "segments.end"), &rows)
|
||||
if err == nil {
|
||||
d.Slices = append(d.Slices, rows...)
|
||||
if err := read(s.sliceRevisions, organizationScopedFilter(org, filter), fields("taskId", "revision", "segments.start", "segments.end"), &rows); err != nil {
|
||||
return d, err
|
||||
}
|
||||
clauses = nil
|
||||
return err
|
||||
}
|
||||
for _, id := range taskIDs {
|
||||
revisions := []int{}
|
||||
for version := range versions[id] {
|
||||
revisions = append(revisions, version)
|
||||
}
|
||||
sort.Ints(revisions)
|
||||
clauses = append(clauses, bson.M{"taskId": id, "revision": bson.M{"$in": revisions}})
|
||||
if len(clauses) == 200 {
|
||||
if err := flush(); err != nil {
|
||||
return d, err
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := flush(); err != nil {
|
||||
return d, err
|
||||
d.Slices = append(d.Slices, rows...)
|
||||
}
|
||||
|
||||
if err := read(s.annotationTasks, organizationScopedFilter(org, bson.M{"id": bson.M{"$in": ids}}), fields("id", "projectId", "runId", "supplierId", "templateVersion"), &d.Tasks); err != nil {
|
||||
|
||||
@ -104,7 +104,12 @@ func MetricCounterValues(m domain.DashboardWorkforceMetric) map[string]any {
|
||||
out := map[string]any{}
|
||||
v := reflect.ValueOf(m)
|
||||
for _, f := range workforceCounterFields {
|
||||
out[f.name] = v.FieldByIndex(f.index).Interface()
|
||||
counter := v.FieldByIndex(f.index)
|
||||
// Mongo $sum treats an absent counter as zero. Most return facts
|
||||
// have no event counters: don't journal dozens of redundant zeros.
|
||||
if !counter.IsZero() {
|
||||
out[f.name] = counter.Interface()
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
@ -7,6 +7,16 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestFactStorageOmitsOnlyZeroCounters(t *testing.T) {
|
||||
if got := MetricCounterValues(domain.DashboardWorkforceMetric{}); len(got) != 0 {
|
||||
t.Fatal("empty return contribution journals redundant counters", got)
|
||||
}
|
||||
m := domain.DashboardWorkforceMetric{SubmittedCount: 3, DurationSeconds: 12}
|
||||
if got := MetricCounterValues(m); !reflect.DeepEqual(got, map[string]any{"submittedCount": int64(3), "durationSeconds": int64(12)}) {
|
||||
t.Fatal("sparse counters changed nonzero values", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Run the existing business fixtures against both paths. Source loaders have
|
||||
// always narrowed event/operation scope BEFORE calling Build; retain that seam.
|
||||
func scopeFactComparison(d Snapshot, q Query) Snapshot {
|
||||
|
||||
@ -3,6 +3,7 @@
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import gzip
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
@ -128,7 +129,12 @@ def main():
|
||||
ap.add_argument("--waves", type=int, default=2, choices=range(1, 101))
|
||||
ap.add_argument("--max-actors", type=int, default=81, choices=(81, 200, 500))
|
||||
ap.add_argument("--fixture-plan", help="owned.json from the bounded in-cluster fixture")
|
||||
ap.add_argument("--cold-write-probe", action="store_true", help="bounded 8MiB synthetic diagnostic; never HTTP capacity evidence")
|
||||
ap.add_argument("--cold-pace-kib", type=int, choices=(0, 128), default=0)
|
||||
args = ap.parse_args()
|
||||
assert args.cold_write_probe or args.cold_pace_kib == 0
|
||||
assert not args.cold_write_probe or args.fixture_plan, "cold diagnostic requires the owned in-cluster fixture"
|
||||
test_name = "TestQueryReplicaColdWriteProbeLocal" if args.cold_write_probe else TEST_NAME
|
||||
out = Path(args.output).resolve()
|
||||
out.mkdir(parents=True, exist_ok=False)
|
||||
k = ["kubectl", "--kubeconfig", str(Path(args.kubeconfig).resolve())]
|
||||
@ -158,8 +164,8 @@ def main():
|
||||
assert api_binary.startswith("/synthetic/query-http") and api_binary.endswith(".test") and "/../" not in api_binary
|
||||
actual = run(k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", "sha256sum", api_binary]).split()[0]
|
||||
assert actual == owned["binarySHA256"]["api"]
|
||||
listed = run(k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", api_binary, "-test.list=^" + TEST_NAME + "$"])
|
||||
assert listed.splitlines() == [TEST_NAME], "stale binary or missing comparison test; no load started"
|
||||
listed = run(k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", api_binary, "-test.list=^" + test_name + "$"])
|
||||
assert listed.splitlines() == [test_name], "stale binary or missing comparison test; no load started"
|
||||
source = {"Id": fixture["metadata"]["uid"], "Image": fixture["status"]["containerStatuses"][0]["imageID"]}
|
||||
else:
|
||||
source = json.loads(run(["docker", "inspect", CONTAINER]))[0]
|
||||
@ -239,6 +245,7 @@ def main():
|
||||
manifest = {"sourceCommit": run(["git", "-C", str(ROOT), "rev-parse", "HEAD"]), "sourceFingerprint": source_fingerprint(), "sourceContainer": source["Id"], "sourceImage": source["Image"], "sourceCPU": 4, "sourceMemory": 4_294_967_296, "queryPodUID": query["metadata"]["uid"], "queryImage": query["spec"]["containers"][0]["image"], "queryCPU": 1, "queryMemory": "1Gi", "tasks": args.tasks, "waves": args.waves, "maxActors": args.max_actors, "wireCompressionBothVariants": os.getenv("QUERY_COMPARE_WIRE_COMPRESSION", ""), "minimumPoolBothVariants": os.getenv("QUERY_COMPARE_MIN_POOL", "0"), "limits": limits, "limitations": ["source Docker VM shares CPU with existing local containers", "target has 2MiB read / 1MiB write and 64/32 IOPS hard limits", "short comparison; not 30-minute three-round production capacity acceptance", "health probes do not establish customer workflow latency"]}
|
||||
manifest["baselineSamples"] = len(baseline)
|
||||
manifest["baselineSeconds"] = baseline[-1]["time"] - baseline[0]["time"]
|
||||
manifest["diagnostic"] = {"kind": "cold-write-probe", "payloadBytes": 8 << 20, "paceKiB": args.cold_pace_kib} if args.cold_write_probe else None
|
||||
if fixture:
|
||||
manifest["limitations"][0] = "source Mongo and native HTTP/projector/builder/client share one fixed 4CPU/4Gi Pod; source storage is bounded tmpfs"
|
||||
manifest["sourceCPUCounter"] = "mongod process utime+stime; whole fixture Pod cgroup sampled separately"
|
||||
@ -249,7 +256,9 @@ def main():
|
||||
remote_report = "/synthetic/query-comparison-" + str(int(time.time())) + ".json"
|
||||
if fixture:
|
||||
settings = ["TMPDIR=/synthetic/tmp", "QUERY_REPLICA_ISOLATED_E2E=1", "QUERY_REPLICA_CLUSTER_TEST=1", "QUERY_REPLICA_TEST_TARGET_URI=mongodb://127.0.0.1:27229/?directConnection=true&replicaSet=query-lab-rs", "QUERY_COMPARE_IN_CLUSTER=1", "QUERY_COMPARE_TASKS=" + str(args.tasks), "QUERY_COMPARE_WAVES=" + str(args.waves), "QUERY_COMPARE_MAX_ACTORS=" + str(args.max_actors), "QUERY_COMPARE_REPORT=" + remote_report, "QUERY_COMPARE_WIRE_COMPRESSION=" + os.getenv("QUERY_COMPARE_WIRE_COMPRESSION", ""), "QUERY_COMPARE_MIN_POOL=" + os.getenv("QUERY_COMPARE_MIN_POOL", "0")]
|
||||
argv = k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", "env"] + settings + [api_binary, "-test.run=^TestQueryReplicaIsolatedHTTPComparisonLocal$", "-test.count=1", "-test.timeout=45m", "-test.v"]
|
||||
if args.cold_write_probe:
|
||||
settings += ["QUERY_REPLICA_COLD_WRITE_PROBE=1", "QUERY_COLD_PACE_KIB=" + str(args.cold_pace_kib)]
|
||||
argv = k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", "env"] + settings + [api_binary, "-test.run=^" + test_name + "$", "-test.count=1", "-test.timeout=45m", "-test.v"]
|
||||
with (out / "test.log").open("w") as log:
|
||||
child = subprocess.Popen(argv, cwd=ROOT, env=env, stdout=log, stderr=subprocess.STDOUT, start_new_session=True)
|
||||
bad = 0
|
||||
@ -313,13 +322,21 @@ def main():
|
||||
except Exception as error:
|
||||
final_health_error = str(error)
|
||||
if fixture:
|
||||
result = subprocess.run(k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", "cat", remote_report], capture_output=True, text=True, timeout=30)
|
||||
result = subprocess.run(k + ["-n", "droboticflow-query-lab", "exec", "node4-query-fixture", "-c", "lab", "--", "gzip", "-c", remote_report], capture_output=True, timeout=30)
|
||||
if result.returncode == 0:
|
||||
(out / "comparison.json").write_text(result.stdout)
|
||||
(out / "comparison.json").write_bytes(gzip.decompress(result.stdout))
|
||||
changed = source_fingerprint() != manifest["sourceFingerprint"]
|
||||
exit_record = {"returncode": child.returncode, "sourceChanged": changed, "coverageComplete": False, "finalHealthError": final_health_error}
|
||||
try:
|
||||
validate_comparison(out / "comparison.json", args.tasks, args.waves, args.max_actors)
|
||||
if args.cold_write_probe:
|
||||
probe = json.loads((out / "comparison.json").read_text())
|
||||
assert probe["kind"] == "cold-write-probe" and probe["capacityAccepted"] is False
|
||||
assert probe["payloadBytes"] == 8 << 20 and probe["completedBatches"] == 32
|
||||
assert probe["majority"] is True and probe["journal"] is True and probe["heartbeatTimeoutMs"] == 3000
|
||||
assert probe["paceKiB"] == args.cold_pace_kib
|
||||
assert len(probe["samples"]) >= 34
|
||||
else:
|
||||
validate_comparison(out / "comparison.json", args.tasks, args.waves, args.max_actors)
|
||||
exit_record["coverageComplete"] = True
|
||||
except (OSError, ValueError, KeyError, AssertionError) as error:
|
||||
exit_record["coverageError"] = str(error)
|
||||
|
||||
@ -23,6 +23,7 @@ CASES = ('internal/store/query_replica_local_test.go',
|
||||
'internal/store/query_replica_valkey_local_test.go',
|
||||
'internal/store/workforce_report_cache_local_test.go',
|
||||
'internal/store/workforce_facts_local_test.go',
|
||||
'internal/store/query_cold_write_local_test.go',
|
||||
'internal/store/query_pool_buckets_local_test.go')
|
||||
|
||||
|
||||
|
||||
@ -608,9 +608,14 @@
|
||||
"frontend": [],
|
||||
"go": {
|
||||
"./internal/api": ["^TestHumanPoolSummary", "^TestHumanPoolCounts", "^TestHumanDisplay", "^TestRouteSnapshot$"],
|
||||
"./internal/store": ["^TestPoolSearchMetadata"]
|
||||
"./internal/store": ["^TestPoolSearchMetadataOwnsOnlyNames$"]
|
||||
},
|
||||
"task": [
|
||||
{
|
||||
"argv": ["go", "test", "./internal/store", "-run", "^TestPoolSearchMetadataWriteCost$", "-count=1", "-v"],
|
||||
"env": { "SUBMISSION_LOCK_COST_TEST": "1" },
|
||||
"tests": true
|
||||
},
|
||||
{
|
||||
"argv": ["npm", "run", "test:go:gate"],
|
||||
"tests": true
|
||||
@ -1015,6 +1020,7 @@
|
||||
"^internal/api/query_replica.*\\.go$",
|
||||
"^internal/store/workforce_(report_cache|heap_diagnostic|facts|return_page).*\\.go$",
|
||||
"^internal/store/query_pool_buckets.*\\.go$",
|
||||
"^internal/store/query_cold_write.*\\.go$",
|
||||
"^internal/api/query_fact_routes_test\\.go$",
|
||||
"^internal/api/workforce_compact_report_test\\.go$",
|
||||
"^internal/testsupport/query_replica_lab.*\\.go$",
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user