🛡️ Harden query recovery, conservative search and shared prewarming

This commit is contained in:
Codex 2026-11-12 03:40:36 -08:00
parent bd15b19cf1
commit 352c405b11
23 changed files with 1193 additions and 126 deletions

View File

@ -15,8 +15,18 @@ Base: 9232bace
## 本地进展
后台读表、Valkey、Typesense 已实现,默认关闭。真实三组件12项(含竞态、范围、失联、重启、旧数据和缓存损坏)通过;既有五阶段作业/恢复、搜索与统计浏览器回归通过。官方 Go 基线为68项已有失败、无新增;前端基线与构建/生产启动检查通过。范围总报告保留了原生产诊断跳过、原测试债务和最初三组件瞬时回退断言失败,不将其写成全绿;修正的专项已重跑通过。
后台读表、Valkey、Typesense 已实现,默认关闭。真实三组件22项(含竞态、范围、失联、重启、旧数据和缓存损坏)通过;既有五阶段作业/恢复、搜索与统计浏览器回归通过。官方 Go 基线为68项已有失败、无新增;前端基线与构建/生产启动检查通过。范围总报告保留了原生产诊断跳过、原测试债务和最初三组件瞬时回退断言失败,不将其写成全绿;修正的专项已重跑通过。
81人、约2.6万任务/34万事件的首轮短测:候选零错误,347组账号/接口业务结果一致;管理员全部项目搜索p95 2592→799ms、生产总览3516→162ms;Mongo内部返回字节199215338→152507258,Go CPU +3.3%。这是短测,不是容量/生产验收。1GiB Mongo 对照OOM已作废,两版统一固定4GiB重新测得上述结果。
独立后台镜像构建和启动帮助检查已通过。待完成:冻结源码后的三轮持续对照和资源审查。未达标不准备发布;生产拓扑、新组件及派生索引/构建范围仍需具体计划与单独授权。现有两个受保护文件的哈希保持不变。
复查新增两项保护:Capture 候选按原 EqualFold 的 Unicode 简单折叠规范化,保留自由搜索原 ToLower/Contains 语义(含终形 sigma、长 s 和 Kelvin 符号回归);新候选与原历史条件组合超出 BSON 预算时退回旧路径,不截断。派生 schema 已升至v2,避免读旧候选。首次长测因发现此边界主动中止,不能作验收证据;22项真实三组件回归已通过,长测须重新冻结运行。
恢复边界补充:后台启动先递增观察代次使旧追平证明失效,只有变更流能推进 observedAt;单次预热复用完整观察路径,不能在重启或迟滞时重新信任旧索引。构建失败有界退避,搜索失联时普通孤儿载荷仍回收,远端意图恢复后清理。新增未重建失败窗口、发布不伪造追平、退避及断连清理回归。进程最大RSS(含预热)纳入规模报告。
新增代次发布前活跃查询预热,避免120秒代次切换造成80人同时解码;另补长词筛选预算、完整BSON预算、凭证字段回退和时钟健康验收。长词超过Typesense默认运算限制的根因已修复;测试夹具补齐既有计数覆盖索引,两版相同。最新Docker短测仍失败,日志证实本机时钟异常,不能作为稳定性通过;已采用独立原生Mongo重建夹具继续验证,环境资源差异必须单列。
原生Mongo 8.0.17(官方SHA256已核对)、同一约2.6万任务/34万事件、81账号的180秒短测:552组结果一致、零错误、零业务写入,全部搜索p95<2秒,生产总览1763→101ms,Go CPU约-11.9%、保留堆约-52.7%。但审核/验收/管理员面板各有一次约0.94秒尖峰,10秒预热短测不通过500ms门槛,不能作为最终验收。两版最大时钟变化62/28ms,旧Docker时钟失败单列保留。接下来必须按完整5分钟预热/30分钟稳态、三轮对照验证,不能丢弃尖峰或放宽门槛。
最新官方Go门禁仍为68项已知失败、0新增;22项真实三组件竞态/恢复回归、6项证据判定测试、格式和独立后台镜像构建/启动检查通过。尚未推送、部署、补写生产或变更生产索引。

View File

@ -6,11 +6,11 @@
Mongo 仍是任务、权限、归属、版本与正式审查记录的权威存储。独立 `query-read-builder` 只写 `queryReadModelControl`、`queryReadModelData` 及既有 `annotationOutputSnapshot*` 派生集合,不领取、退回、重置、补验或改写客户任务。读取新集合不触发构建。
后台使用租约代次隔离写者,先完整构建再发布清单。单个失败不会发布半份结果。进程重启先打开变更流,再核对源身份和重建;不把遗留索引视为最新。副本集/分片集群变更流是持续搜索加速的前提,普通独立 Mongo 不支持时后台暂停,API 继续旧路径。
后台使用租约代次隔离写者,先完整构建再发布清单。单个失败不会发布半份结果。进程重启先使旧追平证明失效,再打开变更流、核对源身份和重建;发布代次不能推进观察器追平时间。`--once` 也执行这条路径并有5分钟上限,退出后搜索新鲜度随心跳过期,不把遗留索引视为最新。副本集/分片集群变更流是持续搜索加速的前提,普通独立 Mongo 不支持时后台暂停,API 继续旧路径。
- 生产总览复用原事实读取与 `productionstats.Build`,保留原日期、供应商、人员和首次送检批次口径。当前配置项目集合生成一份报告;其他未预建项目组合走原路径。来源无变化且观察器新鲜时复用同一代次;有变化按原事实缓存时限停止使用旧结果。
- 人员与供应商看板复用既有分片快照和聚合器,后台无人访问也预热。Valkey 共享不可变分片及按完整范围、用户、供应商、日期、时区、代次隔离的结果;本进程已算出的结果优先,避免每次重复访问缓存服务器。
- Typesense 只存任务 ID 与名称搜索窗口,返回保守候选 ID。用 Unicode 窗口交集保持任意子串、Capture/part 和标点语义;最终匹配、排序、分页、实时可领取及权限仍走原服务端规则。候选过大、超时、索引过期或不完整时退回原查询,不能截断候选凑性能指标。
- 人员与供应商看板复用既有分片快照和聚合器,后台无人访问也预热。Valkey 共享不可变分片及按完整范围、用户、供应商、日期、时区、代次隔离的结果;本进程已算出的结果优先,避免每次重复访问缓存服务器。已访问的个人/供应商查询登记保留4分钟,每范围最多256条,单个身份最多预热2条;新代次发布前复用原聚合器、每分片只解码一次,最多预热128条/5秒。预热不完整不缓存半份结果,失败也不阻断完整快照发布。时区规则必须与当前服务端标准规则一致,自定义时区保留原计算。
- Typesense 只存任务 ID 与名称搜索窗口,返回保守候选 ID。长词最多选取40个必要窗口约束,避免默认100次筛选运算限制;完整搜索词仍由原匹配器复核,候选不截断。用 Unicode 窗口交集保持任意子串、Capture/part 和标点语义;最终匹配、排序、分页、实时可领取及权限仍走原服务端规则。候选过大、超时、索引过期或不完整时退回原查询,不能截断候选凑性能指标。
- 派生集合对通用数据库浏览入口隐藏,防止绕过组织和个人权限。Valkey、Typesense 只允许服务端私网访问,不开放浏览器直连。
## 配置与本地验证
@ -21,10 +21,16 @@ API、后台分别显式设置 `QUERY_SERVING_READS=1` 与唯一 `QUERY_SERVING_
功能与故障回归使用真实本机三组件,`--race` 检查并发。规模对照入口为 `python3 scripts/test_query_serving_performance.py --production-shape`,使用同一合成数据、索引、资源及固定压缩设置;可通过 `--database` 复用脚本生成的专用数据库。完整稳态验收另设 `--rounds 3 --warmup 300 --duration 1800`;短测通过不代表线上容量验收。
本机规模夹具统一使用 Mongo 2 CPU/4GiB(WiredTiger 0.5GiB)、Valkey 1 CPU/256MiB、Typesense 2 CPU/1GiB;这是隔离对照的固定配置,不是生产扩容建议。1GiB Mongo 的首轮对照因旧路径 OOM 作废,不能计为优化收益。
Docker 本机规模夹具统一使用 Mongo 2 CPU/4GiB(WiredTiger 0.5GiB)、Valkey 1 CPU/256MiB、Typesense 2 CPU/1GiB;这是隔离对照的固定配置,不是生产扩容建议。1GiB Mongo 的首轮对照因旧路径 OOM 作废,不能计为优化收益。
本机 Docker 已发现服务端时钟前跳、数百毫秒即触发8秒 MaxTimeMS 的既有环境问题。性能入口持续采样 Mongo 时钟;两版任一存在超过500ms跳变或缺失采样,整轮不能通过。时钟异常记录不得重写为成功。可用官方原生 Mongo 的独立 loopback 27019/query-rs、新数据目录重测;`--native-mongo` 仅接受本任务 `local-data/query-serving-local/native/mongod.pid` 对应进程,不自动替换服务。此模式采集原生进程RSS/CPU,固定0.5GiB WiredTiger,但没有容器CPU/RSS上限,必须单列环境,不能与Docker对照拼接或作生产容量承诺。原二进制来源和SHA256保存在本机忽略目录,不安装系统服务。
两版夹具都显式安装仓库已有 `PoolSearchReadIndexes` 与 `PlatformReadIndexes`,报告记录身份。准备阶段合成写入使用w=1,正式计时重新连接默认写关注,包括后台派生发布写入。生产预检须只读核对这些既有索引;缺失时另列成本与授权方案,不能因本地具备而假定现网具备。该优化不在API启动创建业务索引。
## 运行与清理
名称构建失败按5~60秒退避,统计失败按30~300秒退避;组件失联时先清理普通不可达载荷,保留远端集合清理意图,恢复后再完成删除,避免垃圾载荷因搜索不可用而一直滞留。
后台单进程生成、跨副本租约;Valkey 不持久化,清空后仅损失速度。Typesense 索引可重建。清理只删除当前数据库登记、已不可达且超过保留期的派生代次,当前清单和其他部署不在清理范围。固定构建预算和载荷预算超限时保留旧路径,不改任务准入。
显式 `query-read-builder --ensure-indexes` 只为派生行创建 `query_model_rows_v1`;API 启动不建索引。生产新增组件、该索引及派生数据构建会产生数据库读取和索引维护开销,需另列具体部署、资源和授权计划。不上线时不向公司集群安装或写入这些数据。

View File

@ -58,6 +58,7 @@ type pollMongoObservation struct {
// Rusage covers this test process (handler plus harness compression/sampling),
// not Mongo, another process, or Kubernetes Pod throttling/RSS.
func pollLoadProcessCPU() (float64, error) {
var usage syscall.Rusage
if err := syscall.Getrusage(syscall.RUSAGE_SELF, &usage); err != nil {
return 0, err
@ -198,6 +199,50 @@ func TestPollPoolReadMixLocal(t *testing.T) {
t.Fatal(err)
}
defer client.Disconnect(context.Background())
clockCtx, stopClock := context.WithCancel(ctx)
clockDone := make(chan struct{})
var clockJumpMS atomic.Int64
var clockSamples atomic.Int64
go func() {
defer close(clockDone)
var previousServer, previousLocal time.Time
ticker := time.NewTicker(100 * time.Millisecond)
defer ticker.Stop()
for {
select {
case <-clockCtx.Done():
return
case <-ticker.C:
started := time.Now()
var reply struct {
LocalTime time.Time `bson:"localTime"`
}
probe, cancelProbe := context.WithTimeout(clockCtx, time.Second)
err := client.Database("admin").RunCommand(probe, map[string]any{"hello": 1}).Decode(&reply)
cancelProbe()
finished := time.Now()
if err != nil || finished.Sub(started) > 200*time.Millisecond {
continue
}
local := started.Add(finished.Sub(started) / 2)
if !previousServer.IsZero() {
drift := reply.LocalTime.Sub(previousServer) - local.Sub(previousLocal)
if drift < 0 {
drift = -drift
}
for old := clockJumpMS.Load(); drift.Milliseconds() > old; old = clockJumpMS.Load() {
if clockJumpMS.CompareAndSwap(old, drift.Milliseconds()) {
break
}
}
}
previousServer, previousLocal = reply.LocalTime, local
clockSamples.Add(1)
}
}
}()
defer func() { stopClock(); <-clockDone }()
reuseFixture := os.Getenv("POLL_POOL_REUSE_FIXTURE") == "1"
fixtureName := os.Getenv("POLL_POOL_FIXTURE_DATABASE")
if fixtureName == "" {
@ -244,6 +289,14 @@ func TestPollPoolReadMixLocal(t *testing.T) {
if !reuseFixture {
must(ds.EnsureIndexes(ctx))
}
if os.Getenv("POLL_POOL_QUERY_SERVING_PREPARE") == "1" {
// Both variants use the already-documented platform read indexes as
// well as the pool-search indexes. These are isolated fixtures only.
for collection, models := range store.PlatformReadIndexes() {
_, e := db.Collection(collection).Indexes().CreateMany(ctx, models)
must(e)
}
}
const org = "load-org"
projects := []string{"load-project-a", "load-project-b"}
if os.Getenv("POLL_POOL_QUERY_SERVING_PREPARE") == "1" && !reuseFixture {
@ -489,10 +542,14 @@ func TestPollPoolReadMixLocal(t *testing.T) {
var mu sync.Mutex
samples := []pollLoadSample{}
coldSamples := []pollLoadSample{}
lastPaths := map[string]string{}
recordingCold := true
failureSamples := []map[string]any{}
var completedRequests, completedErrors atomic.Int64
call := func(a pollLoadActor, key, path string, record bool) pollLoadSample {
mu.Lock()
lastPaths[a.User.ID+"\x00"+key] = path
mu.Unlock()
req := httptest.NewRequest(http.MethodGet, path, nil)
requestCtx, done := context.WithTimeout(ctx, 15*time.Second)
defer done()
@ -993,7 +1050,48 @@ func TestPollPoolReadMixLocal(t *testing.T) {
}
}
}
report := map[string]any{"query_serving": queryServing, "query_preparation_cpu_seconds": preparationCPU, "query_serving_statistics": ds.QueryServingStatistics(), "query_panels": queryPanels, "derived_write_commands": derivedWrites.Load(), "reused_prepared_fixture": reuseFixture, "fixture_digest": fixtureDigest, "cold_samples": coldSamples, "slow_search_endpoints": slowSearch, "baseline_failure_collection": allowBaselineFailure, "production_shape": productionShape, "page_shape": pageShape, "fixture_counts": fixtureCounts, "search_indexes": os.Getenv("POLL_POOL_SEARCH_INDEXES") == "1", "read_models": readModels, "search_queries": os.Getenv("POLL_POOL_SEARCH") == "1", "schema": 2, "resource_samples": resourceRows, "process_cpu_seconds": cpuAfter - cpuBefore, "authorization_state": authorizationState, "fixture": "read-mix-v5-isolated-preparation", "available_per_stage": availablePerStage, "task_payload_kib": taskPayloadKiB, "run_payload_kib": runPayloadKiB, "author_history_per_reviewer": authorHistory, "mode": mode, "revision": os.Getenv("POLL_POOL_REVISION"), "actors": totalActors, "actor_counts": map[string]int{"annotation": annotators, "quality": quality, "acceptance": acceptance, "admin": 1}, "claim_counts": claimCounts, "warmup_seconds": warmup, "scheduled_seconds": duration, "actual_seconds": actual.Seconds(), "errors": failures, "failure_samples": failureSamples, "metrics": metrics, "mongo_breakdown": observations, "mongo_commands": commands.Load(), "mongo_reply_bytes": mongoBytes.Load(), "mongo_command_ms": float64(mongoNS.Load()) / 1e6, "mongo_write_commands": writes.Load(), "go_heap_before": memBefore.HeapAlloc, "go_heap_after": memAfter.HeapAlloc, "go_heap_after_gc": memAfterGC.HeapAlloc, "go_gc_cycles": memAfter.NumGC - memBefore.NumGC, "go_allocated_bytes": memAfter.TotalAlloc - memBefore.TotalAlloc, "limitations": []string{"handler timing excludes ingress/network/authentication", "gzip computed separately with fixed Go default compression", "no continuation renew, media or workflow writes", "synthetic fixture; not capacity acceptance", "CPU covers handler and in-process load harness, not Mongo/Pod throttling", "resource heap includes retained benchmark request samples; not standalone service RSS"}, "samples": samples}
outputState, stateErr := ds.OutputSnapshotState(ctx)
must(stateErr)
preparedQueries := 0
for _, head := range outputState.Heads {
if head.Current != nil && head.Current.Metrics != nil {
preparedQueries += head.Current.Metrics.PreparedQueries
}
}
var usage syscall.Rusage
must(syscall.Getrusage(syscall.RUSAGE_SELF, &usage))
maxRSS := usage.Maxrss
if runtime.GOOS != "darwin" {
maxRSS *= 1024
}
// Outside all measured latency, CPU, heap and Mongo counters: a serial
// retry supplies correctness evidence for baseline endpoints that never
// completed under load. Its latency is NOT mixed into benchmark metrics.
references := []pollLoadSample{}
if queryPanels {
succeeded := map[string]bool{}
tried := map[string]bool{}
all := append(append([]pollLoadSample{}, coldSamples...), samples...)
for _, sample := range all {
if sample.Code == 200 {
succeeded[sample.Actor+"\x00"+sample.Key] = true
}
}
for _, sample := range all {
identity := sample.Actor + "\x00" + sample.Key
if succeeded[identity] || tried[identity] || len(references) >= 32 {
continue
}
tried[identity] = true
for _, actor := range actors {
if actor.User.ID == sample.Actor {
references = append(references, call(actor, sample.Key, lastPaths[identity], false))
break
}
}
}
}
report := map[string]any{"mongo_clock_samples": clockSamples.Load(), "mongo_clock_max_jump_ms": clockJumpMS.Load(), "reference_samples": references, "platform_read_indexes": os.Getenv("POLL_POOL_QUERY_SERVING_PREPARE") == "1", "output_prepared_queries": preparedQueries, "process_max_rss_bytes": maxRSS, "query_serving": queryServing, "query_preparation_cpu_seconds": preparationCPU, "query_serving_statistics": ds.QueryServingStatistics(), "query_panels": queryPanels, "derived_write_commands": derivedWrites.Load(), "reused_prepared_fixture": reuseFixture, "fixture_digest": fixtureDigest, "cold_samples": coldSamples, "slow_search_endpoints": slowSearch, "baseline_failure_collection": allowBaselineFailure, "production_shape": productionShape, "page_shape": pageShape, "fixture_counts": fixtureCounts, "search_indexes": os.Getenv("POLL_POOL_SEARCH_INDEXES") == "1", "read_models": readModels, "search_queries": os.Getenv("POLL_POOL_SEARCH") == "1", "schema": 2, "resource_samples": resourceRows, "process_cpu_seconds": cpuAfter - cpuBefore, "authorization_state": authorizationState, "fixture": "read-mix-v5-isolated-preparation", "available_per_stage": availablePerStage, "task_payload_kib": taskPayloadKiB, "run_payload_kib": runPayloadKiB, "author_history_per_reviewer": authorHistory, "mode": mode, "revision": os.Getenv("POLL_POOL_REVISION"), "actors": totalActors, "actor_counts": map[string]int{"annotation": annotators, "quality": quality, "acceptance": acceptance, "admin": 1}, "claim_counts": claimCounts, "warmup_seconds": warmup, "scheduled_seconds": duration, "actual_seconds": actual.Seconds(), "errors": failures, "failure_samples": failureSamples, "metrics": metrics, "mongo_breakdown": observations, "mongo_commands": commands.Load(), "mongo_reply_bytes": mongoBytes.Load(), "mongo_command_ms": float64(mongoNS.Load()) / 1e6, "mongo_write_commands": writes.Load(), "go_heap_before": memBefore.HeapAlloc, "go_heap_after": memAfter.HeapAlloc, "go_heap_after_gc": memAfterGC.HeapAlloc, "go_gc_cycles": memAfter.NumGC - memBefore.NumGC, "go_allocated_bytes": memAfter.TotalAlloc - memBefore.TotalAlloc, "limitations": []string{"handler timing excludes ingress/network/authentication", "gzip computed separately with fixed Go default compression", "no continuation renew, media or workflow writes", "synthetic fixture; not capacity acceptance", "CPU covers handler and in-process load harness, not Mongo/Pod throttling", "resource heap includes retained benchmark request samples; not standalone service RSS"}, "samples": samples}
encoded, e := json.MarshalIndent(report, "", " ")
must(e)
must(os.WriteFile(out, encoded, 0600))

View File

@ -0,0 +1,204 @@
package outputsnapshot
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"strings"
"sync"
"time"
_ "time/tzdata"
"droboticflow/internal/queryserving"
"droboticflow/internal/workforcestats"
)
type activeReportQuery struct {
Version int
Scope string
Query workforcestats.OutputQuery
Location, Rules string
RulesUntil int
}
var reportZoneRules = struct {
sync.Mutex
values map[string]string
}{values: map[string]string{}}
// A canonical reporting location can be reconstructed in another process only
// when its offset rules match exactly, including historical/DST transitions.
// Custom locations or non-progressing standard-library zone bounds keep the
// existing cold calculation path. Historical rules start at year one; the end
// covers the requested dates and current-generation time, with a full-year pad.
func reportLocationRules(location *time.Location, untilYear int) string {
if location == nil || untilYear < 2 || untilYear > 10001 {
return ""
}
canonical, err := workforcestats.ReportingLocation(location.String())
if err != nil || canonical != location {
return ""
}
reportZoneRules.Lock()
defer reportZoneRules.Unlock()
cacheKey := fmt.Sprintf("%p:%d", location, untilYear)
if value := reportZoneRules.values[cacheKey]; value != "" {
return value
}
hash := sha256.New()
cursor := time.Date(1, 1, 1, 0, 0, 0, 0, time.UTC)
end := time.Date(untilYear, 1, 1, 0, 0, 0, 0, time.UTC)
complete := false
for i := 0; i < 30000; i++ {
_, offset := cursor.In(location).Zone()
fmt.Fprintf(hash, "%d:%d;", cursor.Unix(), offset)
_, next := cursor.In(location).ZoneBounds()
if next.IsZero() || !next.Before(end) {
complete = true
break
}
if !next.After(cursor) {
return ""
}
cursor = next
}
if !complete {
return ""
}
value := hex.EncodeToString(hash.Sum(nil))
if len(reportZoneRules.values) < 64 {
reportZoneRules.values[cacheKey] = value
}
return value
}
func safeActiveQuery(q workforcestats.OutputQuery) bool {
if !(q.MatchUser && q.ScopeUserID != "" || q.MatchSupplier && q.ScopeSupplierID != "") {
return false
}
for _, id := range []string{q.ScopeUserID, q.ScopeSupplierID, q.ProjectID, q.SupplierID, q.UserID} {
if len(id) > 256 || strings.Contains(id, "://") || strings.ContainsAny(id, "\x00\r\n?#=") {
return false
}
}
return !q.From.IsZero() && !q.To.IsZero() && q.From.Before(q.To)
}
func (r *Reader) sharedCache() queryserving.Cache {
if provider, ok := r.backend.(interface{ SharedQueryCache() queryserving.Cache }); ok {
return provider.SharedQueryCache()
}
return nil
}
func outputSharedQueryKey(scope, generation string, q workforcestats.OutputQuery) string {
location := ""
if q.Location != nil {
_, from := q.From.In(q.Location).Zone()
_, to := q.To.In(q.Location).Zone()
location = fmt.Sprintf("%s:%d:%d", q.Location.String(), from, to)
}
return queryserving.Key("output-query/v1", scope, generation, q, location)
}
func (r *Reader) rememberActiveQuery(ctx context.Context, scope string, q workforcestats.OutputQuery) {
registry, ok := r.sharedCache().(queryserving.QueryRegistry)
if !ok || !safeActiveQuery(q) {
return
}
if q.Location == nil {
q.Location = time.UTC
}
until := max(q.From.Year(), q.To.Year(), q.Now.Year()) + 2
rules := reportLocationRules(q.Location, until)
if rules == "" {
return
}
entry := activeReportQuery{Version: 1, Scope: scope, Query: q, Location: q.Location.String(), Rules: rules, RulesUntil: until}
entry.Query.Now = time.Time{}
entry.Query.Location = nil
raw, err := json.Marshal(entry)
if err != nil {
return
}
id := queryserving.Key("report-query/v1", entry)
now := time.Now()
r.mu.Lock()
if r.closed || now.Sub(r.registeredQueries[id]) < time.Minute {
r.mu.Unlock()
return
}
if len(r.registeredQueries) >= 512 {
r.registeredQueries = map[string]time.Time{}
}
r.registeredQueries[id] = now
r.mu.Unlock()
registry.RememberQuery(ctx, scope, id, raw)
}
// Warm current registered queries against a complete staged generation before
// its head is published. Decode each shard once and reuse the original reducers;
// no stale-generation result or alternate business calculation is introduced.
func (r *Reader) prewarmActiveQueries(ctx context.Context, scope string, g Generation, payload []byte) int {
registry, ok := r.sharedCache().(queryserving.QueryRegistry)
if !ok {
return 0
}
budget := 5 * time.Second
if deadline, ok := ctx.Deadline(); ok {
budget = min(budget, time.Until(deadline)/2)
}
if budget < 100*time.Millisecond {
return 0
}
warmCtx, stop := context.WithTimeout(ctx, budget)
defer stop()
queries := []workforcestats.OutputQuery{}
reducers := []*workforcestats.OutputReducer{}
perPrincipal := map[string]int{}
for _, raw := range registry.RecentQueries(warmCtx, scope) {
var entry activeReportQuery
if len(raw) > 16<<10 || json.Unmarshal(raw, &entry) != nil || entry.Version != 1 || entry.Scope != scope || !safeActiveQuery(entry.Query) {
continue
}
location, err := workforcestats.ReportingLocation(entry.Location)
if err != nil || g.AsOf.Year() >= entry.RulesUntil || reportLocationRules(location, entry.RulesUntil) != entry.Rules {
continue
}
q := entry.Query
q.Location = location
q.Now = g.AsOf
principal := fmt.Sprintf("%t:%s:%s", q.MatchUser, q.ScopeUserID, q.ScopeSupplierID)
if perPrincipal[principal] >= 2 {
continue
}
perPrincipal[principal]++
queries = append(queries, q)
reducers = append(reducers, workforcestats.NewOutputReducer(q))
if len(queries) >= 128 {
break
}
}
if len(queries) == 0 {
return 0
}
if err := r.visitGenerationDatasets(warmCtx, g, payload, func(data workforcestats.OutputDataset) error {
for _, reducer := range reducers {
if err := warmCtx.Err(); err != nil {
return err
}
reducer.Add(data)
}
return nil
}); err != nil {
return 0
}
prepared := 0
for i, reducer := range reducers {
if warmCtx.Err() != nil {
break
}
report := reducer.Report()
queryserving.PutJSON(warmCtx, r.sharedCache(), outputSharedQueryKey(scope, g.ID, queries[i]), report, MaxAge)
prepared++
}
return prepared
}

View File

@ -4,6 +4,8 @@ import (
"context"
"droboticflow/internal/queryserving"
"droboticflow/internal/workforcestats"
"encoding/json"
"reflect"
"sync"
"sync/atomic"
"testing"
@ -11,8 +13,9 @@ import (
)
type queryTestCache struct {
mu sync.Mutex
data map[string][]byte
mu sync.Mutex
data map[string][]byte
registry map[string]map[string][]byte
}
func (c *queryTestCache) Get(_ context.Context, k string) ([]byte, bool) {
@ -30,6 +33,27 @@ func (c *queryTestCache) Put(_ context.Context, k string, v []byte, _ time.Durat
c.data[k] = append([]byte(nil), v...)
}
func (c *queryTestCache) RememberQuery(_ context.Context, scope, id string, value []byte) {
c.mu.Lock()
defer c.mu.Unlock()
if c.registry == nil {
c.registry = map[string]map[string][]byte{}
}
if c.registry[scope] == nil {
c.registry[scope] = map[string][]byte{}
}
c.registry[scope][id] = append([]byte(nil), value...)
}
func (c *queryTestCache) RecentQueries(_ context.Context, scope string) [][]byte {
c.mu.Lock()
defer c.mu.Unlock()
out := [][]byte{}
for _, v := range c.registry[scope] {
out = append(out, append([]byte(nil), v...))
}
return out
}
type queryCachedBackend struct {
MemoryBackend
cache queryTestCache
@ -84,3 +108,104 @@ func TestQueryServingSharedOutputCacheKeepsScopeLocationAndGeneration(t *testing
t.Fatal("changed project scope borrowed generation", err)
}
}
func TestQueryServingActiveReportsWarmBeforeGenerationPublication(t *testing.T) {
now := time.Now().UTC().Truncate(time.Millisecond)
raw := testData(now)
backend := &queryCachedBackend{}
source := func(ctx context.Context, _ string, _ []string, yield func(workforcestats.OutputSnapshot) error) (*workforcestats.OutputSourceStats, error) {
return nil, yield(raw)
}
reader := NewStreamReader(backend, source)
defer reader.Close()
var clock atomic.Int64
clock.Store(now.UnixNano())
reader.now = func() time.Time { return time.Unix(0, clock.Load()).UTC() }
if _, e := reader.Prewarm(t.Context(), "org", []string{"p"}); e != nil {
t.Fatal(e)
}
var first Info
firstDeadline := time.Now().Add(5 * time.Second)
for {
_, info, e := reader.resolve(t.Context(), "org", []string{"p"}, "")
if e != nil {
t.Fatal(e)
}
if info.ID != "" {
first = info
break
}
if time.Now().After(firstDeadline) {
t.Fatal("first generation not ready")
}
time.Sleep(10 * time.Millisecond)
}
queries := []workforcestats.OutputQuery{
{From: now.Add(-24 * time.Hour), To: now.Add(time.Hour), Location: time.UTC, MatchUser: true, ScopeUserID: "u"},
{From: now.Add(-24 * time.Hour), To: now.Add(time.Hour), Location: time.UTC, MatchUser: true, ScopeUserID: "u", ProductionQuality: true, ReportRole: "annotation"},
{From: now.Add(-24 * time.Hour), To: now.Add(time.Hour), Location: time.UTC, MatchSupplier: true, ScopeSupplierID: "v"},
}
for _, q := range queries {
if _, _, e := reader.Query(t.Context(), "org", []string{"p"}, first.ID, q); e != nil {
t.Fatal(e)
}
}
clock.Store(now.Add(RefreshInterval + time.Second).UnixNano())
if _, e := reader.Prewarm(t.Context(), "org", []string{"p"}); e != nil {
t.Fatal(e)
}
var next Generation
deadline := time.Now().Add(5 * time.Second)
for {
state, e := backend.OutputSnapshotState(t.Context())
if e != nil {
t.Fatal(e)
}
current := state.Heads[scopeKey("org", []string{"p"})].Current
if current != nil && current.ID != first.ID {
next = *current
break
}
if time.Now().After(deadline) {
t.Fatal("next generation not published")
}
time.Sleep(10 * time.Millisecond)
}
if next.Metrics.PreparedQueries != len(queries) {
t.Fatal("published without active report preparation", next.Metrics)
}
fresh := NewStreamReader(backend, source)
defer fresh.Close()
fresh.now = reader.now
reads := backend.blobs.Load()
for _, q := range queries {
got, info, e := fresh.Query(t.Context(), "org", []string{"p"}, next.ID, q)
if e != nil || info.ID != next.ID {
t.Fatal("prepared generation", e, info)
}
dataset := workforcestats.BuildOutputDataset(raw, next.AsOf)
before, _ := json.Marshal(dataset)
want := workforcestats.QueryOutputDataset(dataset, q)
after, _ := json.Marshal(dataset)
if !reflect.DeepEqual(before, after) || !reflect.DeepEqual(got, &want) {
t.Fatal("prepared report changed calculation or source data")
}
}
if backend.blobs.Load() != reads {
t.Fatal("first query decoded source after prepared publication")
}
}
func TestQueryServingActiveQueryDoesNotSubstituteCustomTimeZones(t *testing.T) {
custom := time.FixedZone("UTC", 3600)
if reportLocationRules(custom, 2028) != "" {
t.Fatal("custom zone treated as canonical UTC")
}
location, err := workforcestats.ReportingLocation("America/New_York")
if err != nil || reportLocationRules(location, 2028) == "" {
t.Fatal("canonical DST rules not reusable", err)
}
if reportLocationRules(time.UTC, 2028) == "" {
t.Fatal("canonical UTC not reusable")
}
}

View File

@ -20,28 +20,29 @@ import (
)
type Reader struct {
queryMemo map[string][]byte
queryMemoBytes int
queryFlights singleflight.Group
querySlots chan struct{}
backend Backend
source Source
stream StreamSource
now func() time.Time
ctx context.Context
cancel context.CancelFunc
wg sync.WaitGroup
mu sync.Mutex
closed bool
attempts map[string]time.Time
cache map[string][]byte // Compressed bytes only; no unbounded decoded object cache.
cacheBytes int
reads chan struct{}
queryMemo map[string][]byte
registeredQueries map[string]time.Time
queryMemoBytes int
queryFlights singleflight.Group
querySlots chan struct{}
backend Backend
source Source
stream StreamSource
now func() time.Time
ctx context.Context
cancel context.CancelFunc
wg sync.WaitGroup
mu sync.Mutex
closed bool
attempts map[string]time.Time
cache map[string][]byte // Compressed bytes only; no unbounded decoded object cache.
cacheBytes int
reads chan struct{}
}
func NewReader(b Backend, source Source) *Reader {
ctx, cancel := context.WithCancel(context.Background())
return &Reader{queryMemo: map[string][]byte{}, querySlots: make(chan struct{}, 2), backend: b, source: source, now: time.Now, ctx: ctx, cancel: cancel, attempts: map[string]time.Time{}, cache: map[string][]byte{}, reads: make(chan struct{}, 2)}
return &Reader{registeredQueries: map[string]time.Time{}, queryMemo: map[string][]byte{}, querySlots: make(chan struct{}, 2), backend: b, source: source, now: time.Now, ctx: ctx, cancel: cancel, attempts: map[string]time.Time{}, cache: map[string][]byte{}, reads: make(chan struct{}, 2)}
}
func (r *Reader) Close() { r.mu.Lock(); r.closed = true; r.cancel(); r.mu.Unlock(); r.wg.Wait() }
@ -242,6 +243,11 @@ func (r *Reader) build(key, org string, projects []string) {
buildErr = err
return
}
stage = "query_prewarm"
warmStart := time.Now()
gen.Metrics.PreparedQueries = r.prewarmActiveQueries(ctx, key, gen, payload)
gen.Metrics.PrewarmMS = time.Since(warmStart).Milliseconds()
gen.Metrics.DurationMS = time.Since(started).Milliseconds()
stage = "publish"
published, buildErr = r.backend.PublishOutputSnapshot(ctx, lease, gen, payload, r.now())
return
@ -265,6 +271,11 @@ func (r *Reader) build(key, org string, projects []string) {
}
hash := sha256.Sum256(payload)
gen := Generation{Metrics: &BuildMetrics{DurationMS: time.Since(started).Milliseconds(), DecodedBytes: decodedBytes, Parts: len(data.Parts), Source: raw.SourceStats}, ID: GenerationID(lease), Slot: lease.Slot, Epoch: lease.Epoch, AsOf: asOf, SHA256: hex.EncodeToString(hash[:]), Bytes: len(payload)}
stage = "query_prewarm"
warmStart := time.Now()
gen.Metrics.PreparedQueries = r.prewarmActiveQueries(ctx, key, gen, payload)
gen.Metrics.PrewarmMS = time.Since(warmStart).Milliseconds()
gen.Metrics.DurationMS = time.Since(started).Milliseconds()
stage = "publish"
published, buildErr = r.backend.PublishOutputSnapshot(ctx, lease, gen, payload, r.now())
if published {
@ -357,17 +368,10 @@ func (r *Reader) Query(ctx context.Context, org string, projects []string, pin s
// generation and location identity participate in the key, including personal
// and supplier isolation. No permissions or mutable datasets are cached.
q.Now = gen.AsOf
var shared queryserving.Cache
if provider, ok := r.backend.(interface{ SharedQueryCache() queryserving.Cache }); ok {
shared = provider.SharedQueryCache()
}
location := ""
if q.Location != nil {
_, fromOffset := q.From.In(q.Location).Zone()
_, toOffset := q.To.In(q.Location).Zone()
location = fmt.Sprintf("%s:%d:%d", q.Location.String(), fromOffset, toOffset)
}
sharedKey := queryserving.Key("output-query/v1", scopeKey(org, projects), gen.ID, q, location)
shared := r.sharedCache()
scope := scopeKey(org, projects)
r.rememberActiveQuery(ctx, scope, q)
sharedKey := outputSharedQueryKey(scope, gen.ID, q)
keyData, _ := json.Marshal([]any{gen.ID, q, fmt.Sprintf("%p", q.Location)})
key := string(keyData)
select {

View File

@ -119,64 +119,84 @@ func decodeShard(payload []byte, g Generation) (*workforcestats.OutputDataset, e
func (r *Reader) queryGeneration(ctx context.Context, g Generation, q workforcestats.OutputQuery) (*workforcestats.OutputReport, error) {
q.Now = g.AsOf
if g.Shards == 0 {
d, err := r.load(ctx, g)
data, err := r.load(ctx, g)
if err != nil {
return nil, err
}
report := workforcestats.QueryOutputDataset(*d, q)
report := workforcestats.QueryOutputDataset(*data, q)
return &report, ctx.Err()
}
backend, ok := r.backend.(ShardedBackend)
if !ok {
return nil, ErrUnavailable
}
if g.Shards < 1 || g.Shards > MaxShards || g.Bytes > MaxGenerationBytes {
return nil, ErrCapacity
}
payload, err := r.backend.OutputSnapshotBlob(ctx, g)
if err != nil {
return nil, err
}
reducer := workforcestats.NewOutputReducer(q)
err = r.visitGenerationDatasets(ctx, g, payload, func(data workforcestats.OutputDataset) error { reducer.Add(data); return nil })
if err != nil {
return nil, err
}
report := reducer.Report()
return &report, ctx.Err()
}
func (r *Reader) visitGenerationDatasets(ctx context.Context, g Generation, payload []byte, visit func(workforcestats.OutputDataset) error) error {
if g.Shards == 0 {
if len(payload) != g.Bytes || len(payload) > MaxEncodedBytes || digest(payload) != g.SHA256 {
return ErrUnavailable
}
data, err := decodeShard(payload, g)
if err != nil {
return err
}
return visit(*data)
}
backend, ok := r.backend.(ShardedBackend)
if !ok {
return ErrUnavailable
}
if g.Shards < 1 || g.Shards > MaxShards || g.Bytes > MaxGenerationBytes {
return ErrCapacity
}
var err error
if len(payload) != g.ManifestBytes || len(payload) > MaxEncodedBytes || digest(payload) != g.SHA256 {
return nil, ErrUnavailable
return ErrUnavailable
}
var manifest snapshotManifest
if err = json.Unmarshal(payload, &manifest); err != nil {
return nil, err
return err
}
if len(manifest.Shards) != g.Shards {
return nil, ErrUnavailable
return ErrUnavailable
}
total := 0
for _, d := range manifest.Shards {
if d.Bytes < 1 || d.Bytes > MaxEncodedBytes {
return nil, ErrCapacity
return ErrCapacity
}
total += d.Bytes
}
if total != g.Bytes {
return nil, ErrUnavailable
return ErrUnavailable
}
reducer := workforcestats.NewOutputReducer(q)
for i, descriptor := range manifest.Shards {
if err = ctx.Err(); err != nil {
return nil, err
return err
}
encoded, err := backend.OutputSnapshotShard(ctx, g, i)
if err != nil {
return nil, err
return err
}
if len(encoded) != descriptor.Bytes || digest(encoded) != descriptor.SHA256 {
return nil, ErrUnavailable
return ErrUnavailable
}
d, err := decodeShard(encoded, g)
if err != nil {
return nil, err
return err
}
if err = visit(*d); err != nil {
return err
}
reducer.Add(*d)
}
report := reducer.Report()
return &report, ctx.Err()
return ctx.Err()
}
func (r *Reader) queryResolved(ctx context.Context, g *Generation, info Info, pin string, q workforcestats.OutputQuery) (*workforcestats.OutputReport, Info, error) {
report, err := r.queryGeneration(ctx, *g, q)

View File

@ -31,8 +31,11 @@ var (
ErrCapacity = errors.New("snapshot capacity exceeded")
)
// DurationMS covers source reads, calculation and encoding, before publication.
// DurationMS covers source reads, calculation, encoding and optional active
// query preparation, before publication.
type BuildMetrics struct {
PreparedQueries int `bson:"preparedQueries,omitempty" json:"preparedQueries,omitempty"`
PrewarmMS int64 `bson:"prewarmMs,omitempty" json:"prewarmMs,omitempty"`
PeakEncodedBytes int `bson:"peakEncodedBytes,omitempty" json:"peakEncodedBytes,omitempty"`
PeakDecodedBytes int `bson:"peakDecodedBytes,omitempty" json:"peakDecodedBytes,omitempty"`
DurationMS int64 `bson:"durationMs" json:"durationMs"`

View File

@ -140,17 +140,13 @@ func (v *Valkey) Get(ctx context.Context, key string) ([]byte, bool) {
v.Misses.Add(1)
return nil, false
}
if len(b) < 36 || len(b) > MaxValueBytes+36 || string(b[:4]) != "DFQ1" {
v.Misses.Add(1)
return nil, false
}
sum := sha256.Sum256(append([]byte(key+"\x00"), b[36:]...))
if !bytes.Equal(sum[:], b[4:36]) {
decoded, valid := decodeCacheValue(key, b)
if !valid {
v.Misses.Add(1)
return nil, false
}
v.Hits.Add(1)
return append([]byte(nil), b[36:]...), true
return decoded, true
}
func (v *Valkey) Put(ctx context.Context, key string, b []byte, ttl time.Duration) {
parent := ctx
@ -162,8 +158,7 @@ func (v *Valkey) Put(ctx context.Context, key string, b []byte, ttl time.Duratio
}
ctx, cancel := context.WithTimeout(ctx, 40*time.Millisecond)
defer cancel()
sum := sha256.Sum256(append([]byte(key+"\x00"), b...))
encoded := append(append([]byte("DFQ1"), sum[:]...), b...)
encoded := encodeCacheValue(key, b)
if err := v.client.Do(ctx, v.client.B().Set().Key(v.prefix+":"+key).Value(string(encoded)).Px(ttl).Build()).Error(); err != nil {
if parent.Err() == nil {
v.failed()
@ -197,3 +192,18 @@ func (m *ManagedValkey) Statistics() map[string]uint64 {
}
return map[string]uint64{"hits": v.Hits.Load(), "misses": v.Misses.Load(), "errors": v.Errors.Load()}
}
func encodeCacheValue(key string, b []byte) []byte {
sum := sha256.Sum256(append([]byte(key+"\x00"), b...))
return append(append([]byte("DFQ1"), sum[:]...), b...)
}
func decodeCacheValue(key string, b []byte) ([]byte, bool) {
if len(b) < 36 || len(b) > MaxValueBytes+36 || string(b[:4]) != "DFQ1" {
return nil, false
}
sum := sha256.Sum256(append([]byte(key+"\x00"), b[36:]...))
if !bytes.Equal(sum[:], b[4:36]) {
return nil, false
}
return append([]byte(nil), b[36:]...), true
}

View File

@ -0,0 +1,96 @@
package queryserving
import (
"context"
"strconv"
"time"
)
// QueryRegistry keeps only bounded, short-lived, server-derived report queries.
// It contains no permission decisions and never controls task mutations.
type QueryRegistry interface {
RememberQuery(context.Context, string, string, []byte)
RecentQueries(context.Context, string) [][]byte
}
const rememberQueryScript = `
redis.call('ZADD',KEYS[1],ARGV[1],ARGV[2])
redis.call('HSET',KEYS[2],ARGV[2],ARGV[3])
local old=redis.call('ZRANGEBYSCORE',KEYS[1],'-inf',ARGV[4])
for _,id in ipairs(old) do redis.call('ZREM',KEYS[1],id);redis.call('HDEL',KEYS[2],id) end
local excess=redis.call('ZCARD',KEYS[1])-256
if excess>0 then
local ids=redis.call('ZRANGE',KEYS[1],0,excess-1)
for _,id in ipairs(ids) do redis.call('ZREM',KEYS[1],id);redis.call('HDEL',KEYS[2],id) end
end
redis.call('EXPIRE',KEYS[1],480);redis.call('EXPIRE',KEYS[2],480)
return 1`
const recentQueriesScript = `
local ids=redis.call('ZREVRANGEBYSCORE',KEYS[1],'+inf',ARGV[1],'LIMIT',0,256)
local out={}
for _,id in ipairs(ids) do
local value=redis.call('HGET',KEYS[2],id)
if value then table.insert(out,id);table.insert(out,value) end
end
return out`
func (v *Valkey) registryKeys(scope string) (string, string) {
root := v.prefix + ":{" + Key("active-query/v1", scope) + "}"
return root + ":order", root + ":data"
}
func (v *Valkey) RememberQuery(parent context.Context, scope, id string, b []byte) {
if parent.Err() != nil || len(b) > 16<<10 || time.Now().UnixNano() < v.until.Load() {
return
}
ctx, cancel := context.WithTimeout(parent, 40*time.Millisecond)
defer cancel()
order, data := v.registryKeys(scope)
now := time.Now().Unix()
encoded := encodeCacheValue(scope+":"+id, b)
err := v.client.Do(ctx, v.client.B().Eval().Script(rememberQueryScript).Numkeys(2).Key(order, data).Arg(strconv.FormatInt(now, 10), id, string(encoded), strconv.FormatInt(now-240, 10)).Build()).Error()
if err != nil && parent.Err() == nil {
v.failed()
}
}
func (v *Valkey) RecentQueries(parent context.Context, scope string) [][]byte {
if parent.Err() != nil || time.Now().UnixNano() < v.until.Load() {
return nil
}
ctx, cancel := context.WithTimeout(parent, 100*time.Millisecond)
defer cancel()
order, data := v.registryKeys(scope)
values, err := v.client.Do(ctx, v.client.B().Eval().Script(recentQueriesScript).Numkeys(2).Key(order, data).Arg(strconv.FormatInt(time.Now().Unix()-240, 10)).Build()).AsStrSlice()
if err != nil {
if parent.Err() == nil {
v.failed()
}
return nil
}
if len(values)%2 != 0 || len(values) > 512 {
return nil
}
out := [][]byte{}
for i := 0; i < len(values); i += 2 {
if b, ok := decodeCacheValue(scope+":"+values[i], []byte(values[i+1])); ok && len(b) <= 16<<10 {
out = append(out, b)
}
}
return out
}
func (m *ManagedValkey) RememberQuery(ctx context.Context, scope, id string, b []byte) {
m.mu.RLock()
v := m.value
m.mu.RUnlock()
if v != nil {
v.RememberQuery(ctx, scope, id, b)
}
}
func (m *ManagedValkey) RecentQueries(ctx context.Context, scope string) [][]byte {
m.mu.RLock()
v := m.value
m.mu.RUnlock()
if v != nil {
return v.RecentQueries(ctx, scope)
}
return nil
}

View File

@ -3,6 +3,7 @@ package queryserving
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
@ -136,3 +137,94 @@ func TestQueryServingValkeyRealIsolationCorruptionAndColdRecoveryLocal(t *testin
t.Fatal("cache payload changed")
}
}
func TestQueryServingValkeyActiveQueryRegistryLocal(t *testing.T) {
if os.Getenv("QUERY_SERVING_LOCAL_TEST") != "1" {
t.Skip("isolated Valkey only")
}
namespace := fmt.Sprintf("registry-%d", time.Now().UnixNano())
one, e := NewValkey("valkey://127.0.0.1:16379", namespace)
if e != nil {
t.Fatal(e)
}
defer one.Close()
two, e := NewValkey("valkey://127.0.0.1:16379", namespace)
if e != nil {
t.Fatal(e)
}
defer two.Close()
foreign, e := NewValkey("valkey://127.0.0.1:16379", namespace+"-foreign")
if e != nil {
t.Fatal(e)
}
defer foreign.Close()
ctx := t.Context()
scope := "scope"
order, data := one.registryKeys(scope)
defer one.client.Do(context.Background(), one.client.B().Del().Key(order, data).Build())
one.RememberQuery(ctx, scope, "one", []byte("query-one"))
if got := two.RecentQueries(ctx, scope); len(got) != 1 || string(got[0]) != "query-one" {
t.Fatal("registry not shared", got)
}
if len(foreign.RecentQueries(ctx, scope)) != 0 || len(two.RecentQueries(ctx, "other-scope")) != 0 {
t.Fatal("registry crossed namespace/scope")
}
for i := 0; i < 270; i++ {
one.RememberQuery(ctx, scope, fmt.Sprintf("q-%03d", i), []byte("query"))
}
if got := two.RecentQueries(ctx, scope); len(got) != 256 {
t.Fatal("registry retention not bounded", len(got), one.Errors.Load())
}
// A syntactically valid JSON value without its key-bound integrity frame is
// not allowed to become a registered precomputation.
if e = one.client.Do(ctx, one.client.B().Hset().Key(data).FieldValue().FieldValue("bad", "{}").Build()).Error(); e != nil {
t.Fatal(e)
}
if e = one.client.Do(ctx, one.client.B().Zadd().Key(order).ScoreMember().ScoreMember(float64(time.Now().Unix()), "bad").Build()).Error(); e != nil {
t.Fatal(e)
}
for _, value := range two.RecentQueries(ctx, scope) {
if string(value) == "{}" {
t.Fatal("corrupt registry value accepted")
}
}
cancelled, stop := context.WithCancel(ctx)
stop()
before := one.Errors.Load()
one.RememberQuery(cancelled, scope, "cancelled", []byte("x"))
_ = one.RecentQueries(cancelled, scope)
if one.Errors.Load() != before {
t.Fatal("caller cancellation tripped registry breaker")
}
}
func TestQueryServingLongSearchKeepsConservativeOperatorBudget(t *testing.T) {
text := "synthetic-available-load-project-a-load-supplier-a-prepackage_acceptance-0"
grams := candidateGrams(text, "another-long-capture/part0001")
if len(grams) > 40 || len(grams) == 0 {
t.Fatal("filter operation budget", len(grams))
}
full := map[string]bool{}
for _, g := range IndexGrams(text + " another-long-capture/part0001") {
full[g] = true
}
for _, g := range grams {
if !full[g] {
t.Fatal("selected constraint can lose a true result")
}
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if count := strings.Count(r.URL.Query().Get("filter_by"), "grams:="); count > 40 {
t.Error("long query rejected by server budget", count)
w.WriteHeader(400)
return
}
_, _ = w.Write([]byte("{\"item_id\":\"expected\"}\n"))
}))
defer server.Close()
client, _ := NewTypesense(server.URL, "synthetic")
ids, e := client.Candidates(t.Context(), "dfq_budget", text, "another-long-capture/part0001")
if e != nil || !reflect.DeepEqual(ids, []string{"expected"}) {
t.Fatal("long search response", ids, e)
}
}

View File

@ -149,10 +149,8 @@ func (t *Typesense) Candidates(ctx context.Context, c string, terms ...string) (
return nil, errors.New("invalid search collection")
}
filters := []string{}
for _, term := range terms {
for _, g := range Grams(term) {
filters = append(filters, "grams:=["+g+"]")
}
for _, g := range candidateGrams(terms...) {
filters = append(filters, "grams:=["+g+"]")
}
params := url.Values{"include_fields": {"item_id"}}
if len(filters) > 0 {
@ -188,3 +186,30 @@ func (t *Typesense) Delete(ctx context.Context, c string) error {
}
return err
}
// Typesense's default parser counts both clauses and boolean operators. Keep
// well below its 100-operation limit without changing that server setting.
// Dropping prefilter constraints only adds candidates; the original complete
// query is still matched by Go, and no task/result inventory is truncated.
func candidateGrams(terms ...string) []string {
set := map[string]bool{}
for _, term := range terms {
for _, gram := range Grams(term) {
set[gram] = true
}
}
all := make([]string, 0, len(set))
for gram := range set {
all = append(all, gram)
}
sort.Strings(all)
const limit = 40
if len(all) <= limit {
return all
}
selected := make([]string, limit)
for i := range selected {
selected[i] = all[i*(len(all)-1)/(limit-1)]
}
return selected
}

View File

@ -305,7 +305,9 @@ func (s *MongoStore) scanAdminAnnotationPool(ctx context.Context, org string, q
if len(ids) == 0 {
return nil
}
pipeline[0] = bson.D{{Key: "$match", Value: bson.M{"$and": bson.A{pipeline[0][0].Value, bson.M{"id": bson.M{"$in": ids}}}}}}
if scoped, withinBudget := queryCandidateFilter(pipeline[0][0].Value, ids); withinBudget {
pipeline[0] = bson.D{{Key: "$match", Value: scoped}}
}
}
cur, err := s.humanItems.Aggregate(ctx, pipeline, options.Aggregate().SetAllowDiskUse(false).SetMaxTime(8*time.Second).SetBatchSize(200))
if err != nil {

View File

@ -128,7 +128,9 @@ func (s *MongoStore) prepareHumanPoolSearchFilter(ctx context.Context, org strin
if len(ids) == 0 {
return nil, true, nil
}
return bson.M{"$and": bson.A{filter, bson.M{"id": bson.M{"$in": ids}}}}, false, nil
if scoped, withinBudget := queryCandidateFilter(filter, ids); withinBudget {
return scoped, false, nil
}
}
candidates, err := s.humanPoolSearchCandidates(ctx, org, q, filter)
if err != nil {

View File

@ -26,6 +26,11 @@ func (s *MongoStore) cleanQueryModelOrphans(ctx context.Context) error {
keep = append(keep, h.Generation)
}
filter := bson.M{"generation": bson.M{"$nin": keep}, "createdAt": bson.M{"$lt": time.Now().UTC().Add(-15 * time.Minute)}}
// Ordinary orphan chunks/rows must still be reclaimed when Typesense is
// offline. Retain only its collection intents until remote deletion succeeds.
if _, err := s.queryData().DeleteMany(ctx, bson.M{"$and": bson.A{filter, bson.M{"kind": bson.M{"$ne": "search_collection"}}}}); err != nil {
return err
}
// Only this database's registered, unreachable search generations are
// eligible; other deployments and current generations remain untouched.
cur, err := s.queryData().Find(ctx, bson.M{"$and": bson.A{filter, bson.M{"kind": "search_collection"}}}, options.Find().SetProjection(bson.M{"_id": 1, "collection": 1}))
@ -45,6 +50,9 @@ func (s *MongoStore) cleanQueryModelOrphans(ctx context.Context) error {
if err = s.queryServing.search.Delete(ctx, row.Collection); err != nil {
return err
}
if _, err = s.queryData().DeleteOne(ctx, bson.M{"_id": row.ID}); err != nil {
return err
}
} else {
return nil
}
@ -52,8 +60,7 @@ func (s *MongoStore) cleanQueryModelOrphans(ctx context.Context) error {
if err = cur.Err(); err != nil {
return err
}
_, err = s.queryData().DeleteMany(ctx, filter)
return err
return nil
}
// Obsolete configured scopes cannot pin an unbounded history of generations.

View File

@ -44,8 +44,9 @@ func (b *QueryReadBuilder) WaitReady(ctx context.Context, org string, projects [
tick := time.NewTicker(100 * time.Millisecond)
defer tick.Stop()
for {
_, production := b.store.readQueryModelHead(ctx, queryModelKey("production", org, projects), 5*time.Minute)
productionHead, production := b.store.readQueryModelHead(ctx, queryModelKey("production", org, projects), 5*time.Minute)
st, err := b.store.querySearchState(ctx)
production = production && productionHead.GlobalRevision == st.Global
h, exists := st.Heads[queryModelKey("search", org, nil)]
search := err == nil && exists && h.Generation != "" && h.Revision == st.Revisions[queryModelKey("search", org, nil)] && h.GlobalRevision == st.Global && time.Since(st.ObservedAt) >= 0 && time.Since(st.ObservedAt) < 5*time.Second
output := len(projects) == 0
@ -106,8 +107,9 @@ func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org s
started := time.Now()
var source struct {
Revision int64 `bson:"productionRevision"`
Global int64 `bson:"searchGlobalRevision"`
}
if err := b.store.queryControl().FindOne(ctx, bson.M{"_id": "global"}, options.FindOne().SetProjection(bson.M{"productionRevision": 1})).Decode(&source); err != nil {
if err := b.store.queryControl().FindOne(ctx, bson.M{"_id": "global"}, options.FindOne().SetProjection(bson.M{"productionRevision": 1, "searchGlobalRevision": 1})).Decode(&source); err != nil {
return err
}
buildCtx, finish := context.WithTimeout(ctx, 5*time.Minute)
@ -126,6 +128,7 @@ func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org s
return err
}
head.SourceRevision = source.Revision
head.GlobalRevision = source.Global
if err = b.store.publishQueryModelHead(ctx, l, key, head); err != nil {
return err
}
@ -141,30 +144,40 @@ func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org s
// RunOnce is for bounded initial prewarming and isolated tests, not a freshness
// substitute for Run. After the heartbeat expires search falls back to old reads.
func (b *QueryReadBuilder) RunOnce(ctx context.Context) error {
l, ok, err := b.store.acquireQueryModelLease(ctx)
func (b *QueryReadBuilder) RunOnce(parent context.Context) error {
ctx, cancel := context.WithTimeout(parent, 5*time.Minute)
defer cancel()
started := make(chan struct{})
done := make(chan error, 1)
go func() { done <- b.run(ctx, started); cancel() }()
select {
case <-started:
case err := <-done:
return err
case <-ctx.Done():
cancel()
<-done
return ctx.Err()
}
scopes, err := b.scopes(ctx)
if err == nil {
for org, ids := range scopes {
if err = b.WaitReady(ctx, org, ids); err != nil {
break
}
}
}
cancel()
runErr := <-done
if err != nil {
return err
}
if !ok {
return errors.New("query builder already active")
if runErr != nil && !errors.Is(runErr, context.Canceled) {
return runErr
}
return b.withLease(ctx, l, func(ctx context.Context) error {
scopes, err := b.scopes(ctx)
if err != nil {
return err
}
for org, ids := range scopes {
if err = b.rebuild(ctx, l, org, ids); err != nil {
return err
}
if err = b.store.buildQuerySearchModel(ctx, l, org); err != nil {
return err
}
}
return nil
})
return nil
}
func (b *QueryReadBuilder) withLease(ctx context.Context, l queryModelLease, run func(context.Context) error) error {
ctx, cancel := context.WithCancel(ctx)
defer cancel()
@ -194,7 +207,9 @@ func (b *QueryReadBuilder) withLease(ctx context.Context, l queryModelLease, run
return err
}
func (b *QueryReadBuilder) Run(ctx context.Context) error {
func (b *QueryReadBuilder) Run(ctx context.Context) error { return b.run(ctx, nil) }
func (b *QueryReadBuilder) run(ctx context.Context, started chan<- struct{}) error {
b.mu.Lock()
b.all, b.allSearch = true, true
b.sourceDigests = map[string]querySourceDigest{}
@ -207,6 +222,17 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
return errors.New("query builder already active")
}
return b.withLease(ctx, l, func(ctx context.Context) error {
ctx, cancelWork := context.WithCancel(ctx)
defer cancelWork()
// A new observer has not covered changes during downtime. Old search
// generations must remain unavailable until a complete reconciliation.
result, e := b.store.queryControl().UpdateOne(ctx, queryLeaseFilter(l), bson.M{"$inc": bson.M{"searchGlobalRevision": int64(1), "productionRevision": int64(1)}, "$set": bson.M{"observedAt": time.Time{}}})
if e != nil {
return e
}
if result.MatchedCount != 1 {
return errors.New("query restart fenced")
}
// No fullDocument lookup: projecting only event identity avoids copying
// annotation bodies, credentials or full video payloads on each save.
pipeline := mongo.Pipeline{{{Key: "$match", Value: bson.M{"ns.coll": bson.M{"$in": querySourceCollections}}}}, {{Key: "$project", Value: bson.M{"_id": 1, "ns": 1, "documentKey": 1, "operationType": 1, "clusterTime": 1}}}}
@ -214,7 +240,11 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
if err != nil {
return fmt.Errorf("query builder needs available Mongo change streams: %w", err)
}
defer stream.Close(context.Background())
defer func() {
closeCtx, closeCancel := context.WithTimeout(context.Background(), time.Second)
defer closeCancel()
_ = stream.Close(closeCtx)
}()
// Seed before consuming: events already captured by the open stream
// close the initial snapshot/change gap without retaining customer data.
seedCtx, seedCancel := context.WithTimeout(ctx, 2*time.Minute)
@ -227,6 +257,14 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
defer stop()
watchDone := make(chan error, 1)
watchExited := make(chan struct{})
var watchFailure error
failWatch := func(e error) {
if watchCtx.Err() == nil {
watchFailure = e
}
watchDone <- e
cancelWork()
}
go func() {
defer close(watchExited)
lastHeartbeat := time.Time{}
@ -243,7 +281,7 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
ClusterTime primitive.Timestamp `bson:"clusterTime"`
}
if e := stream.Decode(&event); e != nil {
watchDone <- e
failWatch(e)
return
}
pendingChange = true
@ -251,18 +289,18 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
observed = time.Unix(int64(event.ClusterTime.T), 0).UTC()
}
if e := b.observeQuerySourceEvent(watchCtx, l, event.NS.Collection, event.Operation, event.Key); e != nil {
watchDone <- e
failWatch(e)
return
}
} else if e := stream.Err(); e != nil {
watchDone <- e
failWatch(e)
return
}
if time.Since(lastHeartbeat) < time.Second {
continue
}
if e := b.recordObserved(watchCtx, l, observed, pendingChange); e != nil {
watchDone <- e
failWatch(e)
return
}
pendingChange = false
@ -270,11 +308,24 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
}
watchDone <- watchCtx.Err()
}()
defer func() { stop(); <-watchExited }()
defer func() {
stop()
<-watchExited
if watchFailure != nil {
cleanup, finish := context.WithTimeout(context.Background(), time.Second)
defer finish()
_, _ = b.store.queryControl().UpdateOne(cleanup, queryLeaseFilter(l), bson.M{"$set": bson.M{"observedAt": time.Time{}}})
}
}()
if started != nil {
close(started)
}
tick := time.NewTicker(time.Second)
defer tick.Stop()
pendingStats, pendingSearch := map[string]bool{}, map[string]bool{}
lastStats := map[string]time.Time{}
searchRetries := map[string]queryBuildRetry{}
statsRetries := map[string]queryBuildRetry{}
var scopes map[string][]string
refreshScopes := true
lastScopeRead := time.Time{}
@ -316,9 +367,16 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
delete(pendingSearch, org)
continue
}
if time.Now().Before(searchRetries[org].Next) {
continue
}
if e := b.store.buildQuerySearchModel(ctx, l, org); e == nil {
delete(searchRetries, org)
delete(pendingSearch, org)
} else {
retry := searchRetries[org]
retry.fail(5*time.Second, time.Minute)
searchRetries[org] = retry
log.Print("query_search_build result=retry category=source_or_publication")
}
}
@ -331,10 +389,17 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
if time.Since(lastStats[org]) < 30*time.Second {
continue
}
if time.Now().Before(statsRetries[org].Next) {
continue
}
if e := b.rebuild(ctx, l, org, ids); e == nil {
delete(statsRetries, org)
delete(pendingStats, org)
lastStats[org] = time.Now()
} else {
retry := statsRetries[org]
retry.fail(30*time.Second, 5*time.Minute)
statsRetries[org] = retry
log.Print("query_model_build result=retry category=source_or_publication")
}
}
@ -352,8 +417,6 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
}
select {
case <-ctx.Done():
stop()
<-watchDone
return ctx.Err()
case e := <-watchDone:
return e
@ -382,3 +445,17 @@ func querySourceCollection(name string) bool {
}
return false
}
type queryBuildRetry struct {
Next time.Time
Delay time.Duration
}
func (r *queryBuildRetry) fail(initial, maximum time.Duration) {
if r.Delay == 0 {
r.Delay = initial
} else {
r.Delay = min(r.Delay*2, maximum)
}
r.Next = time.Now().Add(r.Delay)
}

View File

@ -22,7 +22,7 @@ import (
"go.mongodb.org/mongo-driver/mongo/options"
)
const queryModelSchema = "query-model/v1"
const queryModelSchema = "query-model/v2"
const queryModelChunkBytes = 2 << 20
const queryModelMaxBytes = 64 << 20

View File

@ -44,15 +44,17 @@ func newQueryModelFixture(t *testing.T, n int) queryModelFixture {
t.Cleanup(func() {
cleanup, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var control struct {
Heads map[string]QueryModelHead `bson:"heads"`
}
if s.queryControl().FindOne(cleanup, bson.M{"_id": "global"}).Decode(&control) == nil {
for _, h := range control.Heads {
if h.Collection != "" {
_ = search.Delete(cleanup, h.Collection)
cur, e := s.queryData().Find(cleanup, bson.M{"kind": "search_collection"}, options.Find().SetProjection(bson.M{"collection": 1}))
if e == nil {
for cur.Next(cleanup) {
var row struct {
Collection string `bson:"collection"`
}
if cur.Decode(&row) == nil {
_ = search.Delete(cleanup, row.Collection)
}
}
_ = cur.Close(cleanup)
}
_ = db.Drop(cleanup)
_ = client.Disconnect(cleanup)

View File

@ -4,8 +4,13 @@ import (
"context"
"droboticflow/internal/domain"
"droboticflow/internal/queryserving"
"fmt"
"go.mongodb.org/mongo-driver/bson"
"net/http"
"net/http/httptest"
"reflect"
"strings"
"sync/atomic"
"testing"
"time"
)
@ -229,3 +234,212 @@ func TestQueryReadModelsCachedResponseScopeAndStaleFallbackLocal(t *testing.T) {
t.Fatal("stale model counted as fresh")
}
}
func TestQuerySearchUnicodeCaptureFoldCompatibilityLocal(t *testing.T) {
f := newQueryModelFixture(t, 3)
s, ctx := f.s, t.Context()
names := []string{"Σ", "S", "K"}
variants := []string{"ς", "ſ", "K"}
for i, name := range names {
if e := s.UpsertCaptureRecord(ctx, domain.CaptureRecord{ID: fmt.Sprintf("capture-item-%06d", i), OrganizationID: f.org, ProjectID: "p", CaptureID: name, Prefix: name}); e != nil {
t.Fatal(e)
}
if !strings.EqualFold(name, variants[i]) || queryCaptureFold(name) != queryCaptureFold(variants[i]) {
t.Fatal("capture folding is incompatible")
}
}
b := NewQueryReadBuilder(s)
defer b.Close()
if e := b.RunOnce(ctx); e != nil {
t.Fatal(e)
}
for i, q := range variants {
ids, ready := s.querySearchCandidates(ctx, f.org, "", q)
want := fmt.Sprintf("item-%06d", i)
found := false
for _, id := range ids {
found = found || id == want
}
if !ready || !found {
t.Fatal("Unicode capture lost from candidates", q, ids, ready)
}
items, e := s.ListHumanItems(ctx, f.org, domain.HumanItemQuery{PluginID: "annotation-qa", Capture: q, Limit: 10})
if e != nil || !reflect.DeepEqual(queryIDs(items), []string{want}) {
t.Fatal("exact capture changed", q, queryIDs(items), e)
}
}
}
func TestQuerySearchCombinedFilterBudgetRetainsLegacyPath(t *testing.T) {
small, ok := queryCandidateFilter(bson.M{"organizationId": "org"}, []string{"a"})
if !ok || len(small) == 0 {
t.Fatal("ordinary filter rejected")
}
// Previously valid large history filters must not exceed BSON's command
// budget only because an optional candidate index was added.
original := bson.M{"history": strings.Repeat("x", 12<<20)}
if _, ok = queryCandidateFilter(original, []string{"a"}); ok {
t.Fatal("combined filter budget ignored")
}
if len(original) != 1 {
t.Fatal("fallback changed original scope")
}
}
func TestQueryReadModelsRestartCannotTrustOldCatalogDuringFailedRebuildLocal(t *testing.T) {
f := newQueryModelFixture(t, 3)
s, ctx := f.s, t.Context()
b := NewQueryReadBuilder(s)
defer b.Close()
if e := b.RunOnce(ctx); e != nil {
t.Fatal(e)
}
original, _ := s.querySearchState(ctx)
if e := s.UpsertProject(ctx, domain.ProjectRecord{ID: "p", OrganizationID: f.org, Name: "downtime-project-name"}); e != nil {
t.Fatal(e)
}
var calls atomic.Int64
offline := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { calls.Add(1); w.WriteHeader(503) }))
defer offline.Close()
backend, e := queryserving.NewTypesense(offline.URL, "synthetic")
if e != nil {
t.Fatal(e)
}
s.queryServing.search = backend
runCtx, stop := context.WithCancel(ctx)
done := make(chan error, 1)
go func() { done <- b.Run(runCtx) }()
defer func() { stop(); <-done; s.queryServing.search = f.search }()
deadline := time.Now().Add(10 * time.Second)
for {
st, e := s.querySearchState(ctx)
if e == nil && st.Global > original.Global && time.Since(st.ObservedAt) < 5*time.Second {
break
}
if time.Now().After(deadline) {
t.Fatal("observer did not restart")
}
time.Sleep(50 * time.Millisecond)
}
if _, ready := s.querySearchCandidates(ctx, f.org, "downtime-project-name", ""); ready {
t.Fatal("old catalog trusted before reconciliation")
}
items, e := s.ListHumanItems(ctx, f.org, domain.HumanItemQuery{Search: "downtime-project-name", Limit: 10})
if e != nil || len(items) != 3 {
t.Fatal("failed rebuild blocked legacy search", e, len(items))
}
time.Sleep(1200 * time.Millisecond)
if calls.Load() > 2 {
t.Fatal("backend outage caused a tight rebuild loop", calls.Load())
}
}
func TestQueryReadModelsPublicationCannotAdvanceObserverProofLocal(t *testing.T) {
f := newQueryModelFixture(t, 3)
s, ctx := f.s, t.Context()
b := NewQueryReadBuilder(s)
defer b.Close()
if e := b.RunOnce(ctx); e != nil {
t.Fatal(e)
}
l, ok, e := s.acquireQueryModelLease(ctx)
if e != nil || !ok {
t.Fatal("lease", e)
}
_, e = s.queryControl().UpdateOne(ctx, queryLeaseFilter(l), bson.M{"$set": bson.M{"observedAt": time.Now().Add(-time.Minute)}})
if e != nil {
t.Fatal(e)
}
if e = s.buildQuerySearchModel(ctx, l, f.org); e != nil {
t.Fatal(e)
}
if _, ready := s.querySearchCandidates(ctx, f.org, "part0002", ""); ready {
t.Fatal("publication forged fresh observer proof")
}
// Backend outage retains remote cleanup intent, without retaining all orphan
// report/search payloads. Restoring the backend completes cleanup.
_, e = s.queryData().InsertOne(ctx, bson.M{"_id": "orphan-payload", "generation": "orphan-generation", "createdAt": time.Now().Add(-time.Hour), "data": []byte("old")})
if e != nil {
t.Fatal(e)
}
_, e = s.queryData().InsertOne(ctx, bson.M{"_id": "orphan-intent", "generation": "orphan-generation", "kind": "search_collection", "collection": fmt.Sprintf("dfq_gc_%d", time.Now().UnixNano()), "createdAt": time.Now().Add(-time.Hour)})
if e != nil {
t.Fatal(e)
}
s.queryServing.search = nil
if e = s.cleanQueryModelOrphans(ctx); e != nil {
t.Fatal(e)
}
n, e := s.queryData().CountDocuments(ctx, bson.M{"_id": "orphan-payload"})
if e != nil || n != 0 {
t.Fatal("unavailable search backend retained payload garbage", e)
}
n, e = s.queryData().CountDocuments(ctx, bson.M{"_id": "orphan-intent"})
if e != nil || n != 1 {
t.Fatal("remote cleanup intent lost", e)
}
s.queryServing.search = f.search
if e = s.cleanQueryModelOrphans(ctx); e != nil {
t.Fatal(e)
}
n, e = s.queryData().CountDocuments(ctx, bson.M{"_id": "orphan-intent"})
if e != nil || n != 0 {
t.Fatal("recovered backend did not finish cleanup", e)
}
}
func TestQueryReadModelsRetryBackoffIsBounded(t *testing.T) {
var retry queryBuildRetry
for i := 0; i < 20; i++ {
retry.fail(5*time.Second, time.Minute)
if retry.Delay < 5*time.Second || retry.Delay > time.Minute || time.Until(retry.Next) > time.Minute {
t.Fatal("unbounded retry", retry)
}
}
if retry.Delay != time.Minute {
t.Fatal("backoff did not reach limit")
}
}
func TestQuerySearchCredentialsInLegacyNamesKeepOriginalPathLocal(t *testing.T) {
f := newQueryModelFixture(t, 1)
s, ctx := f.s, t.Context()
item, e := s.GetHumanItem(ctx, "item-000000")
if e != nil {
t.Fatal(e)
}
item.Subject.ArtifactID = "https://synthetic.invalid/video.mp4?signature=synthetic-private"
if e = s.UpsertHumanItem(ctx, *item); e != nil {
t.Fatal(e)
}
before, e := s.GetHumanItem(ctx, item.ID)
if e != nil {
t.Fatal(e)
}
l, ok, e := s.acquireQueryModelLease(ctx)
if e != nil || !ok {
t.Fatal("lease", e)
}
if e = s.buildQuerySearchModel(ctx, l, f.org); e == nil {
t.Fatal("credential-bearing legacy name materialized")
}
st, e := s.querySearchState(ctx)
if e != nil {
t.Fatal(e)
}
if st.Heads[queryModelKey("search", f.org, nil)].Generation != "" {
t.Fatal("partial catalog published")
}
n, e := s.queryData().CountDocuments(ctx, bson.M{"text": bson.M{"$regex": "synthetic-private"}})
if e != nil || n != 0 {
t.Fatal("credential in derived rows", n, e)
}
after, e := s.GetHumanItem(ctx, item.ID)
if e != nil || !reflect.DeepEqual(before, after) {
t.Fatal("fallback changed annotation", e)
}
rows, e := s.ListHumanItems(ctx, f.org, domain.HumanItemQuery{Search: "synthetic-private", Limit: 10})
if e != nil || len(rows) != 1 {
t.Fatal("legacy search lost", len(rows), e)
}
}

View File

@ -9,6 +9,7 @@ import (
"regexp"
"strings"
"time"
"unicode"
"droboticflow/internal/domain"
"droboticflow/internal/queryserving"
@ -41,7 +42,7 @@ func querySafePath(value string) string {
func searchRowText(row adminPoolRow) string {
// Both exact historical search corpora remain contiguous; adding fields in
// between could otherwise lose cross-field substring matches.
return adminPoolSearchText(row) + "\n" + humanPoolFallbackSearchText(row)
return adminPoolSearchText(row) + "\n" + humanPoolFallbackSearchText(row) + "\n" + queryCaptureFold(strings.Join(adminPoolCaptureValues(row), "\n"))
}
func (s *MongoStore) buildQuerySearchModel(ctx context.Context, l queryModelLease, org string) error {
@ -109,6 +110,11 @@ func (s *MongoStore) buildQuerySearchModel(ctx context.Context, l queryModelLeas
}
}
text := searchRowText(row)
for _, value := range strings.Fields(text) {
if querySafePath(value) != value {
return errors.New("credential-bearing search field requires legacy search")
}
}
if len(text) > 64<<10 {
return errors.New("search names exceed materialization budget")
}
@ -133,7 +139,7 @@ func (s *MongoStore) buildQuerySearchModel(ctx context.Context, l queryModelLeas
// Missing revision zero is equivalent to zero, without writing business data.
filter["$expr"] = bson.M{"$and": bson.A{bson.M{"$eq": bson.A{bson.M{"$ifNull": bson.A{"$searchRevisions." + key, int64(0)}}, h.Revision}}, bson.M{"$eq": bson.A{bson.M{"$ifNull": bson.A{"$searchGlobalRevision", int64(0)}}, h.GlobalRevision}}}}
delete(filter, "searchGlobalRevision")
res, err := s.queryControl().UpdateOne(ctx, filter, bson.M{"$set": bson.M{"heads." + key: h, "observedAt": time.Now().UTC()}})
res, err := s.queryControl().UpdateOne(ctx, filter, bson.M{"$set": bson.M{"heads." + key: h}})
if err != nil {
return err
}
@ -168,7 +174,7 @@ func (s *MongoStore) querySearchCandidates(ctx context.Context, org, search, cap
return ids, true
}
if h.Collection != "" && s.queryServing.search != nil {
if out, err := s.queryServing.search.Candidates(short, h.Collection, search, capture); err == nil && queryCandidateBudget(out) {
if out, err := s.queryServing.search.Candidates(short, h.Collection, search, queryCaptureFold(capture)); err == nil && queryCandidateBudget(out) {
s.queryServing.stats.searchHits.Add(1)
s.queryServing.stats.typesenseReads.Add(1)
queryserving.PutJSON(short, s.SharedQueryCache(), cacheKey, out, 5*time.Minute)
@ -176,7 +182,7 @@ func (s *MongoStore) querySearchCandidates(ctx context.Context, org, search, cap
}
}
clauses := bson.A{bson.M{"generation": h.Generation, "organizationId": normalizedOrganizationID(org)}}
for _, term := range []string{search, capture} {
for _, term := range []string{search, queryCaptureFold(capture)} {
if term != "" {
clauses = append(clauses, bson.M{"text": bson.M{"$regex": regexp.QuoteMeta(strings.ToLower(term))}})
}
@ -187,6 +193,7 @@ func (s *MongoStore) querySearchCandidates(ctx context.Context, org, search, cap
}
defer cur.Close(short)
ids = []string{}
candidateBytes := 0
for cur.Next(short) {
var row struct {
ID string `bson:"itemId"`
@ -195,7 +202,8 @@ func (s *MongoStore) querySearchCandidates(ctx context.Context, org, search, cap
return nil, false
}
ids = append(ids, row.ID)
if len(ids) > 100000 {
candidateBytes += len(row.ID) + 16
if len(ids) > 100000 || candidateBytes > 4<<20 {
return nil, false
}
}
@ -229,3 +237,29 @@ func queryCandidateBudget(ids []string) bool {
}
return true
}
// Capture is exact EqualFold, whereas free search uses ToLower+Contains.
// Canonicalize each Unicode simple-fold cycle so sigma/final sigma and long-s
// cannot disappear from the conservative prefilter. Extra hits are still
// rejected by the original exact matcher.
func queryCaptureFold(value string) string {
var b strings.Builder
for _, r := range value {
smallest := r
for next := unicode.SimpleFold(r); next != r; next = unicode.SimpleFold(next) {
if next < smallest {
smallest = next
}
}
b.WriteRune(smallest)
}
return strings.ToLower(b.String())
}
// Leave room for joins/projections under Mongo's command size limit. Oversized
// optional acceleration falls through to the original query without truncation.
func queryCandidateFilter(filter any, ids []string) (bson.M, bool) {
joined := bson.M{"$and": bson.A{filter, bson.M{"id": bson.M{"$in": ids}}}}
encoded, err := bson.Marshal(joined)
return joined, err == nil && len(encoded) <= 12<<20
}

View File

@ -4,6 +4,8 @@ def compare(base, candidate):
problems=[]
if base.get('fixture_digest')!=candidate.get('fixture_digest') or not base.get('fixture_digest'):
problems.append('fixture identity mismatch')
if any(r.get('mongo_clock_samples',0)<2 or r.get('mongo_clock_max_jump_ms',float('inf'))>500 for r in (base,candidate)):
problems.append('Mongo clock evidence missing or unstable')
if not base.get('metrics') or not candidate.get('metrics'):
problems.append('missing request measurements')
if candidate.get('actors')!=81 or candidate.get('errors')!=0 or candidate.get('mongo_write_commands')!=0:
@ -11,6 +13,8 @@ def compare(base, candidate):
counters=candidate.get('query_serving_statistics',{})
if not all(counters.get(k,0)>0 for k in ('report_hits','search_hits','typesense_reads','cache_hits')):
problems.append('accelerated paths not proven active')
if candidate.get('scheduled_seconds',0)>=180 and candidate.get('output_prepared_queries',0)<=0:
problems.append('generation-change prewarming not proven active')
for key,m in candidate.get('metrics',{}).items():
limit=2000 if '.search' in key else 500 if key.endswith('.panel') or key=='admin.production' else None
if limit and m.get('p95_ms',float('inf'))>limit:problems.append('latency exceeded: '+key)
@ -19,7 +23,7 @@ def compare(base, candidate):
problems.append('prewarmed first visit exceeded: '+row['Key'])
def hashes(report):
out={}
for r in report.get('cold_samples',[])+report.get('samples',[]):
for r in report.get('cold_samples',[])+report.get('samples',[])+report.get('reference_samples',[]):
if r.get('Code')==200:out.setdefault((r['Actor'],r['Key']),set()).add(r['Hash'])
return out
old,new=hashes(base),hashes(candidate)

View File

@ -21,7 +21,14 @@ def main():
parser.add_argument('--warmup',type=int,default=5)
parser.add_argument('--duration',type=int,default=30)
parser.add_argument('--database',help='reuse this script\'s existing synthetic fixture')
parser.add_argument('--native-mongo',action='store_true',help='use task-owned native Mongo on loopback 27019; never starts/stops another service')
args=parser.parse_args()
native_pid=None
if args.native_mongo:
if args.start:parser.error('start the native replica set and optional Docker components separately')
native_pid=int((PRIVATE/'native/mongod.pid').read_text().strip())
command=subprocess.run(['ps','-p',str(native_pid),'-o','command='],capture_output=True,text=True,check=True).stdout
if 'local-data/query-serving-local/native/' not in command or '--port 27019' not in command:parser.error('task-owned native Mongo process required')
if not 1<=args.rounds<=3 or not 1<=args.duration<=1800 or not 1<=args.warmup<=300:parser.error('bounded rounds/times required')
if args.database and (not args.database.startswith('poll_pool_read_mix_query_') or not args.database.replace('_','').isalnum()):parser.error('synthetic database only')
env={k:v for k,v in environment().items() if not k.startswith('POLL_POOL_')}
@ -41,7 +48,8 @@ def main():
POLL_POOL_TASK_PAYLOAD_KIB='16', POLL_POOL_AVAILABLE_PER_STAGE='215',
POLL_POOL_AUTHOR_HISTORY='100', GOMAXPROCS='2')
before=source_fingerprint(ROOT)
report={'database':database,'inputDigest':before,'shape':'production' if args.production_shape else 'small','rounds':[],'resourceSamples':[],'passed':False,'limits':['synthetic loopback API harness, no media or production writes','process resource sampling includes builder and handler; not Kubernetes Pod limits']}
report={'database':database,'inputDigest':before,'mongoEnvironment':'native-uncapped' if native_pid else 'docker-2cpu-4g','shape':'production' if args.production_shape else 'small','rounds':[],'resourceSamples':[],'passed':False,'limits':['synthetic loopback API harness, no media or production writes','process resource sampling includes builder and handler; not Kubernetes Pod limits']}
if native_pid:report['limits'].append('native Mongo has fixed 0.5GiB WiredTiger cache but no container CPU/RSS cap; not comparable with earlier Docker timings')
argv=['go','test','./internal/api','-run','^TestPollPoolReadMixLocal$','-count=1','-timeout=45m','-v']
def run(name,updates):
settings={**env,**updates,'POLL_POOL_OUTPUT':str(folder/(name+'.json'))}
@ -56,13 +64,24 @@ def main():
return result.returncode==0
active={'phase':'prepare'}
if not args.database:
if not run('prepare',{'POLL_POOL_FIXTURE_ONLY':'1'}):return 1
# Synthetic setup is not measured. Avoid a separate majority journal
# barrier for every seed upsert; measured phases reconnect with the
# normal URI/write concern, including background publication writes.
setup={'POLL_POOL_FIXTURE_ONLY':'1','POLL_POOL_LOCAL_MONGO_URI':env['POLL_POOL_LOCAL_MONGO_URI']+'&w=1'}
if not run('prepare',setup):return 1
stopped=threading.Event()
def resources():
while not stopped.is_set():
try:
p=subprocess.run(['docker','stats','--no-stream','--format','{{json .}}','fanyu-query-local-mongo-1','fanyu-query-local-valkey-1','fanyu-query-local-typesense-1'],capture_output=True,text=True,timeout=8)
if p.returncode==0:report['resourceSamples'].append({'phase':active['phase'],'at':dt.datetime.now(dt.timezone.utc).isoformat(),'containers':[json.loads(line) for line in p.stdout.splitlines()]})
containers=['fanyu-query-local-valkey-1','fanyu-query-local-typesense-1']
if not native_pid:containers.insert(0,'fanyu-query-local-mongo-1')
p=subprocess.run(['docker','stats','--no-stream','--format','{{json .}}',*containers],capture_output=True,text=True,timeout=8)
if p.returncode==0:
row={'phase':active['phase'],'at':dt.datetime.now(dt.timezone.utc).isoformat(),'containers':[json.loads(line) for line in p.stdout.splitlines()]}
if native_pid:
process=subprocess.run(['ps','-p',str(native_pid),'-o','%cpu=,rss=,vsz=,time='],capture_output=True,text=True,timeout=2,check=True)
row['nativeMongo']=dict(zip(('cpuPercentLifetime','rssKiB','virtualKiB','cpuTime'),process.stdout.split()))
report['resourceSamples'].append(row)
except (OSError,ValueError,subprocess.TimeoutExpired):pass
(folder/'progress.json').write_text(json.dumps({'phase':active['phase'],'at':dt.datetime.now(dt.timezone.utc).isoformat(),'samples':len(report['resourceSamples'])})+'\n')
stopped.wait(2)

View File

@ -7,7 +7,7 @@ from lib.query_serving_evidence import compare
class QueryEvidenceTests(unittest.TestCase):
def fixture(self):
sample={'Actor':'reviewer','Key':'acceptance.all.search','Code':200,'MS':20,'Hash':'business-id-claim-counts'}
return {'fixture_digest':'same','actors':81,'errors':0,'mongo_write_commands':0,'metrics':{'acceptance.all.search':{'p95_ms':20}},'samples':[sample], 'cold_samples':[], 'query_serving_statistics':dict(report_hits=1,search_hits=1,typesense_reads=1,cache_hits=1)}
return {'mongo_clock_samples':100, 'mongo_clock_max_jump_ms':1, 'fixture_digest':'same','actors':81,'errors':0,'mongo_write_commands':0,'metrics':{'acceptance.all.search':{'p95_ms':20}},'samples':[sample], 'cold_samples':[], 'query_serving_statistics':dict(report_hits=1,search_hits=1,typesense_reads=1,cache_hits=1)}
def test_requires_real_measurements_and_active_components(self):
a=self.fixture();b=self.fixture();self.assertTrue(compare(a,b)['passed'])
b['query_serving_statistics']={};self.assertFalse(compare(a,b)['passed'])
@ -26,4 +26,17 @@ class QueryEvidenceTests(unittest.TestCase):
a=self.fixture();b=self.fixture();a['samples'][0]['Code']=500
r=compare(a,b);self.assertFalse(r['passed']);self.assertTrue(r['baselineUnavailableComparisons'])
def test_unstable_clock_cannot_be_accepted(self):
a=self.fixture();b=self.fixture();a['mongo_clock_max_jump_ms']=18000
self.assertFalse(compare(a,b)['passed'])
a=self.fixture();del a['mongo_clock_samples']
self.assertFalse(compare(a,b)['passed'])
def test_serial_reference_closes_gap_without_changing_latency(self):
a=self.fixture();b=self.fixture();sample=dict(a['samples'][0]);a['samples'][0]['Code']=500
a['reference_samples']=[sample]
self.assertTrue(compare(a,b)['passed'])
b['metrics']['acceptance.all.search']['p95_ms']=2001
self.assertFalse(compare(a,b)['passed'])
if __name__=='__main__':unittest.main()