🛡️ Harden query recovery, conservative search and shared prewarming
This commit is contained in:
parent
bd15b19cf1
commit
352c405b11
@ -15,8 +15,18 @@ Base: 9232bace
|
||||
|
||||
## 本地进展
|
||||
|
||||
后台读表、Valkey、Typesense 已实现,默认关闭。真实三组件12项(含竞态、范围、失联、重启、旧数据和缓存损坏)通过;既有五阶段作业/恢复、搜索与统计浏览器回归通过。官方 Go 基线为68项已有失败、无新增;前端基线与构建/生产启动检查通过。范围总报告保留了原生产诊断跳过、原测试债务和最初三组件瞬时回退断言失败,不将其写成全绿;修正的专项已重跑通过。
|
||||
后台读表、Valkey、Typesense 已实现,默认关闭。真实三组件22项(含竞态、范围、失联、重启、旧数据和缓存损坏)通过;既有五阶段作业/恢复、搜索与统计浏览器回归通过。官方 Go 基线为68项已有失败、无新增;前端基线与构建/生产启动检查通过。范围总报告保留了原生产诊断跳过、原测试债务和最初三组件瞬时回退断言失败,不将其写成全绿;修正的专项已重跑通过。
|
||||
|
||||
81人、约2.6万任务/34万事件的首轮短测:候选零错误,347组账号/接口业务结果一致;管理员全部项目搜索p95 2592→799ms、生产总览3516→162ms;Mongo内部返回字节199215338→152507258,Go CPU +3.3%。这是短测,不是容量/生产验收。1GiB Mongo 对照OOM已作废,两版统一固定4GiB重新测得上述结果。
|
||||
|
||||
独立后台镜像构建和启动帮助检查已通过。待完成:冻结源码后的三轮持续对照和资源审查。未达标不准备发布;生产拓扑、新组件及派生索引/构建范围仍需具体计划与单独授权。现有两个受保护文件的哈希保持不变。
|
||||
|
||||
复查新增两项保护:Capture 候选按原 EqualFold 的 Unicode 简单折叠规范化,保留自由搜索原 ToLower/Contains 语义(含终形 sigma、长 s 和 Kelvin 符号回归);新候选与原历史条件组合超出 BSON 预算时退回旧路径,不截断。派生 schema 已升至v2,避免读旧候选。首次长测因发现此边界主动中止,不能作验收证据;22项真实三组件回归已通过,长测须重新冻结运行。
|
||||
|
||||
恢复边界补充:后台启动先递增观察代次使旧追平证明失效,只有变更流能推进 observedAt;单次预热复用完整观察路径,不能在重启或迟滞时重新信任旧索引。构建失败有界退避,搜索失联时普通孤儿载荷仍回收,远端意图恢复后清理。新增未重建失败窗口、发布不伪造追平、退避及断连清理回归。进程最大RSS(含预热)纳入规模报告。
|
||||
|
||||
新增代次发布前活跃查询预热,避免120秒代次切换造成80人同时解码;另补长词筛选预算、完整BSON预算、凭证字段回退和时钟健康验收。长词超过Typesense默认运算限制的根因已修复;测试夹具补齐既有计数覆盖索引,两版相同。最新Docker短测仍失败,日志证实本机时钟异常,不能作为稳定性通过;已采用独立原生Mongo重建夹具继续验证,环境资源差异必须单列。
|
||||
|
||||
原生Mongo 8.0.17(官方SHA256已核对)、同一约2.6万任务/34万事件、81账号的180秒短测:552组结果一致、零错误、零业务写入,全部搜索p95<2秒,生产总览1763→101ms,Go CPU约-11.9%、保留堆约-52.7%。但审核/验收/管理员面板各有一次约0.94秒尖峰,10秒预热短测不通过500ms门槛,不能作为最终验收。两版最大时钟变化62/28ms,旧Docker时钟失败单列保留。接下来必须按完整5分钟预热/30分钟稳态、三轮对照验证,不能丢弃尖峰或放宽门槛。
|
||||
|
||||
最新官方Go门禁仍为68项已知失败、0新增;22项真实三组件竞态/恢复回归、6项证据判定测试、格式和独立后台镜像构建/启动检查通过。尚未推送、部署、补写生产或变更生产索引。
|
||||
|
||||
@ -6,11 +6,11 @@
|
||||
|
||||
Mongo 仍是任务、权限、归属、版本与正式审查记录的权威存储。独立 `query-read-builder` 只写 `queryReadModelControl`、`queryReadModelData` 及既有 `annotationOutputSnapshot*` 派生集合,不领取、退回、重置、补验或改写客户任务。读取新集合不触发构建。
|
||||
|
||||
后台使用租约代次隔离写者,先完整构建再发布清单。单个失败不会发布半份结果。进程重启先打开变更流,再核对源身份和重建;不把遗留索引视为最新。副本集/分片集群变更流是持续搜索加速的前提,普通独立 Mongo 不支持时后台暂停,API 继续旧路径。
|
||||
后台使用租约代次隔离写者,先完整构建再发布清单。单个失败不会发布半份结果。进程重启先使旧追平证明失效,再打开变更流、核对源身份和重建;发布代次不能推进观察器追平时间。`--once` 也执行这条路径并有5分钟上限,退出后搜索新鲜度随心跳过期,不把遗留索引视为最新。副本集/分片集群变更流是持续搜索加速的前提,普通独立 Mongo 不支持时后台暂停,API 继续旧路径。
|
||||
|
||||
- 生产总览复用原事实读取与 `productionstats.Build`,保留原日期、供应商、人员和首次送检批次口径。当前配置项目集合生成一份报告;其他未预建项目组合走原路径。来源无变化且观察器新鲜时复用同一代次;有变化按原事实缓存时限停止使用旧结果。
|
||||
- 人员与供应商看板复用既有分片快照和聚合器,后台无人访问也预热。Valkey 共享不可变分片及按完整范围、用户、供应商、日期、时区、代次隔离的结果;本进程已算出的结果优先,避免每次重复访问缓存服务器。
|
||||
- Typesense 只存任务 ID 与名称搜索窗口,返回保守候选 ID。用 Unicode 窗口交集保持任意子串、Capture/part 和标点语义;最终匹配、排序、分页、实时可领取及权限仍走原服务端规则。候选过大、超时、索引过期或不完整时退回原查询,不能截断候选凑性能指标。
|
||||
- 人员与供应商看板复用既有分片快照和聚合器,后台无人访问也预热。Valkey 共享不可变分片及按完整范围、用户、供应商、日期、时区、代次隔离的结果;本进程已算出的结果优先,避免每次重复访问缓存服务器。已访问的个人/供应商查询登记保留4分钟,每范围最多256条,单个身份最多预热2条;新代次发布前复用原聚合器、每分片只解码一次,最多预热128条/5秒。预热不完整不缓存半份结果,失败也不阻断完整快照发布。时区规则必须与当前服务端标准规则一致,自定义时区保留原计算。
|
||||
- Typesense 只存任务 ID 与名称搜索窗口,返回保守候选 ID。长词最多选取40个必要窗口约束,避免默认100次筛选运算限制;完整搜索词仍由原匹配器复核,候选不截断。用 Unicode 窗口交集保持任意子串、Capture/part 和标点语义;最终匹配、排序、分页、实时可领取及权限仍走原服务端规则。候选过大、超时、索引过期或不完整时退回原查询,不能截断候选凑性能指标。
|
||||
- 派生集合对通用数据库浏览入口隐藏,防止绕过组织和个人权限。Valkey、Typesense 只允许服务端私网访问,不开放浏览器直连。
|
||||
|
||||
## 配置与本地验证
|
||||
@ -21,10 +21,16 @@ API、后台分别显式设置 `QUERY_SERVING_READS=1` 与唯一 `QUERY_SERVING_
|
||||
|
||||
功能与故障回归使用真实本机三组件,`--race` 检查并发。规模对照入口为 `python3 scripts/test_query_serving_performance.py --production-shape`,使用同一合成数据、索引、资源及固定压缩设置;可通过 `--database` 复用脚本生成的专用数据库。完整稳态验收另设 `--rounds 3 --warmup 300 --duration 1800`;短测通过不代表线上容量验收。
|
||||
|
||||
本机规模夹具统一使用 Mongo 2 CPU/4GiB(WiredTiger 0.5GiB)、Valkey 1 CPU/256MiB、Typesense 2 CPU/1GiB;这是隔离对照的固定配置,不是生产扩容建议。1GiB Mongo 的首轮对照因旧路径 OOM 作废,不能计为优化收益。
|
||||
Docker 本机规模夹具统一使用 Mongo 2 CPU/4GiB(WiredTiger 0.5GiB)、Valkey 1 CPU/256MiB、Typesense 2 CPU/1GiB;这是隔离对照的固定配置,不是生产扩容建议。1GiB Mongo 的首轮对照因旧路径 OOM 作废,不能计为优化收益。
|
||||
|
||||
本机 Docker 已发现服务端时钟前跳、数百毫秒即触发8秒 MaxTimeMS 的既有环境问题。性能入口持续采样 Mongo 时钟;两版任一存在超过500ms跳变或缺失采样,整轮不能通过。时钟异常记录不得重写为成功。可用官方原生 Mongo 的独立 loopback 27019/query-rs、新数据目录重测;`--native-mongo` 仅接受本任务 `local-data/query-serving-local/native/mongod.pid` 对应进程,不自动替换服务。此模式采集原生进程RSS/CPU,固定0.5GiB WiredTiger,但没有容器CPU/RSS上限,必须单列环境,不能与Docker对照拼接或作生产容量承诺。原二进制来源和SHA256保存在本机忽略目录,不安装系统服务。
|
||||
|
||||
两版夹具都显式安装仓库已有 `PoolSearchReadIndexes` 与 `PlatformReadIndexes`,报告记录身份。准备阶段合成写入使用w=1,正式计时重新连接默认写关注,包括后台派生发布写入。生产预检须只读核对这些既有索引;缺失时另列成本与授权方案,不能因本地具备而假定现网具备。该优化不在API启动创建业务索引。
|
||||
|
||||
## 运行与清理
|
||||
|
||||
名称构建失败按5~60秒退避,统计失败按30~300秒退避;组件失联时先清理普通不可达载荷,保留远端集合清理意图,恢复后再完成删除,避免垃圾载荷因搜索不可用而一直滞留。
|
||||
|
||||
后台单进程生成、跨副本租约;Valkey 不持久化,清空后仅损失速度。Typesense 索引可重建。清理只删除当前数据库登记、已不可达且超过保留期的派生代次,当前清单和其他部署不在清理范围。固定构建预算和载荷预算超限时保留旧路径,不改任务准入。
|
||||
|
||||
显式 `query-read-builder --ensure-indexes` 只为派生行创建 `query_model_rows_v1`;API 启动不建索引。生产新增组件、该索引及派生数据构建会产生数据库读取和索引维护开销,需另列具体部署、资源和授权计划。不上线时不向公司集群安装或写入这些数据。
|
||||
|
||||
@ -58,6 +58,7 @@ type pollMongoObservation struct {
|
||||
// Rusage covers this test process (handler plus harness compression/sampling),
|
||||
// not Mongo, another process, or Kubernetes Pod throttling/RSS.
|
||||
func pollLoadProcessCPU() (float64, error) {
|
||||
|
||||
var usage syscall.Rusage
|
||||
if err := syscall.Getrusage(syscall.RUSAGE_SELF, &usage); err != nil {
|
||||
return 0, err
|
||||
@ -198,6 +199,50 @@ func TestPollPoolReadMixLocal(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer client.Disconnect(context.Background())
|
||||
clockCtx, stopClock := context.WithCancel(ctx)
|
||||
clockDone := make(chan struct{})
|
||||
var clockJumpMS atomic.Int64
|
||||
var clockSamples atomic.Int64
|
||||
go func() {
|
||||
defer close(clockDone)
|
||||
var previousServer, previousLocal time.Time
|
||||
ticker := time.NewTicker(100 * time.Millisecond)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-clockCtx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
started := time.Now()
|
||||
var reply struct {
|
||||
LocalTime time.Time `bson:"localTime"`
|
||||
}
|
||||
probe, cancelProbe := context.WithTimeout(clockCtx, time.Second)
|
||||
err := client.Database("admin").RunCommand(probe, map[string]any{"hello": 1}).Decode(&reply)
|
||||
cancelProbe()
|
||||
finished := time.Now()
|
||||
if err != nil || finished.Sub(started) > 200*time.Millisecond {
|
||||
continue
|
||||
}
|
||||
local := started.Add(finished.Sub(started) / 2)
|
||||
if !previousServer.IsZero() {
|
||||
drift := reply.LocalTime.Sub(previousServer) - local.Sub(previousLocal)
|
||||
if drift < 0 {
|
||||
drift = -drift
|
||||
}
|
||||
for old := clockJumpMS.Load(); drift.Milliseconds() > old; old = clockJumpMS.Load() {
|
||||
if clockJumpMS.CompareAndSwap(old, drift.Milliseconds()) {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
previousServer, previousLocal = reply.LocalTime, local
|
||||
clockSamples.Add(1)
|
||||
}
|
||||
}
|
||||
}()
|
||||
defer func() { stopClock(); <-clockDone }()
|
||||
|
||||
reuseFixture := os.Getenv("POLL_POOL_REUSE_FIXTURE") == "1"
|
||||
fixtureName := os.Getenv("POLL_POOL_FIXTURE_DATABASE")
|
||||
if fixtureName == "" {
|
||||
@ -244,6 +289,14 @@ func TestPollPoolReadMixLocal(t *testing.T) {
|
||||
if !reuseFixture {
|
||||
must(ds.EnsureIndexes(ctx))
|
||||
}
|
||||
if os.Getenv("POLL_POOL_QUERY_SERVING_PREPARE") == "1" {
|
||||
// Both variants use the already-documented platform read indexes as
|
||||
// well as the pool-search indexes. These are isolated fixtures only.
|
||||
for collection, models := range store.PlatformReadIndexes() {
|
||||
_, e := db.Collection(collection).Indexes().CreateMany(ctx, models)
|
||||
must(e)
|
||||
}
|
||||
}
|
||||
const org = "load-org"
|
||||
projects := []string{"load-project-a", "load-project-b"}
|
||||
if os.Getenv("POLL_POOL_QUERY_SERVING_PREPARE") == "1" && !reuseFixture {
|
||||
@ -489,10 +542,14 @@ func TestPollPoolReadMixLocal(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
samples := []pollLoadSample{}
|
||||
coldSamples := []pollLoadSample{}
|
||||
lastPaths := map[string]string{}
|
||||
recordingCold := true
|
||||
failureSamples := []map[string]any{}
|
||||
var completedRequests, completedErrors atomic.Int64
|
||||
call := func(a pollLoadActor, key, path string, record bool) pollLoadSample {
|
||||
mu.Lock()
|
||||
lastPaths[a.User.ID+"\x00"+key] = path
|
||||
mu.Unlock()
|
||||
req := httptest.NewRequest(http.MethodGet, path, nil)
|
||||
requestCtx, done := context.WithTimeout(ctx, 15*time.Second)
|
||||
defer done()
|
||||
@ -993,7 +1050,48 @@ func TestPollPoolReadMixLocal(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
report := map[string]any{"query_serving": queryServing, "query_preparation_cpu_seconds": preparationCPU, "query_serving_statistics": ds.QueryServingStatistics(), "query_panels": queryPanels, "derived_write_commands": derivedWrites.Load(), "reused_prepared_fixture": reuseFixture, "fixture_digest": fixtureDigest, "cold_samples": coldSamples, "slow_search_endpoints": slowSearch, "baseline_failure_collection": allowBaselineFailure, "production_shape": productionShape, "page_shape": pageShape, "fixture_counts": fixtureCounts, "search_indexes": os.Getenv("POLL_POOL_SEARCH_INDEXES") == "1", "read_models": readModels, "search_queries": os.Getenv("POLL_POOL_SEARCH") == "1", "schema": 2, "resource_samples": resourceRows, "process_cpu_seconds": cpuAfter - cpuBefore, "authorization_state": authorizationState, "fixture": "read-mix-v5-isolated-preparation", "available_per_stage": availablePerStage, "task_payload_kib": taskPayloadKiB, "run_payload_kib": runPayloadKiB, "author_history_per_reviewer": authorHistory, "mode": mode, "revision": os.Getenv("POLL_POOL_REVISION"), "actors": totalActors, "actor_counts": map[string]int{"annotation": annotators, "quality": quality, "acceptance": acceptance, "admin": 1}, "claim_counts": claimCounts, "warmup_seconds": warmup, "scheduled_seconds": duration, "actual_seconds": actual.Seconds(), "errors": failures, "failure_samples": failureSamples, "metrics": metrics, "mongo_breakdown": observations, "mongo_commands": commands.Load(), "mongo_reply_bytes": mongoBytes.Load(), "mongo_command_ms": float64(mongoNS.Load()) / 1e6, "mongo_write_commands": writes.Load(), "go_heap_before": memBefore.HeapAlloc, "go_heap_after": memAfter.HeapAlloc, "go_heap_after_gc": memAfterGC.HeapAlloc, "go_gc_cycles": memAfter.NumGC - memBefore.NumGC, "go_allocated_bytes": memAfter.TotalAlloc - memBefore.TotalAlloc, "limitations": []string{"handler timing excludes ingress/network/authentication", "gzip computed separately with fixed Go default compression", "no continuation renew, media or workflow writes", "synthetic fixture; not capacity acceptance", "CPU covers handler and in-process load harness, not Mongo/Pod throttling", "resource heap includes retained benchmark request samples; not standalone service RSS"}, "samples": samples}
|
||||
outputState, stateErr := ds.OutputSnapshotState(ctx)
|
||||
must(stateErr)
|
||||
preparedQueries := 0
|
||||
for _, head := range outputState.Heads {
|
||||
if head.Current != nil && head.Current.Metrics != nil {
|
||||
preparedQueries += head.Current.Metrics.PreparedQueries
|
||||
}
|
||||
}
|
||||
var usage syscall.Rusage
|
||||
must(syscall.Getrusage(syscall.RUSAGE_SELF, &usage))
|
||||
maxRSS := usage.Maxrss
|
||||
if runtime.GOOS != "darwin" {
|
||||
maxRSS *= 1024
|
||||
}
|
||||
// Outside all measured latency, CPU, heap and Mongo counters: a serial
|
||||
// retry supplies correctness evidence for baseline endpoints that never
|
||||
// completed under load. Its latency is NOT mixed into benchmark metrics.
|
||||
references := []pollLoadSample{}
|
||||
if queryPanels {
|
||||
succeeded := map[string]bool{}
|
||||
tried := map[string]bool{}
|
||||
all := append(append([]pollLoadSample{}, coldSamples...), samples...)
|
||||
for _, sample := range all {
|
||||
if sample.Code == 200 {
|
||||
succeeded[sample.Actor+"\x00"+sample.Key] = true
|
||||
}
|
||||
}
|
||||
for _, sample := range all {
|
||||
identity := sample.Actor + "\x00" + sample.Key
|
||||
if succeeded[identity] || tried[identity] || len(references) >= 32 {
|
||||
continue
|
||||
}
|
||||
tried[identity] = true
|
||||
for _, actor := range actors {
|
||||
if actor.User.ID == sample.Actor {
|
||||
references = append(references, call(actor, sample.Key, lastPaths[identity], false))
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
report := map[string]any{"mongo_clock_samples": clockSamples.Load(), "mongo_clock_max_jump_ms": clockJumpMS.Load(), "reference_samples": references, "platform_read_indexes": os.Getenv("POLL_POOL_QUERY_SERVING_PREPARE") == "1", "output_prepared_queries": preparedQueries, "process_max_rss_bytes": maxRSS, "query_serving": queryServing, "query_preparation_cpu_seconds": preparationCPU, "query_serving_statistics": ds.QueryServingStatistics(), "query_panels": queryPanels, "derived_write_commands": derivedWrites.Load(), "reused_prepared_fixture": reuseFixture, "fixture_digest": fixtureDigest, "cold_samples": coldSamples, "slow_search_endpoints": slowSearch, "baseline_failure_collection": allowBaselineFailure, "production_shape": productionShape, "page_shape": pageShape, "fixture_counts": fixtureCounts, "search_indexes": os.Getenv("POLL_POOL_SEARCH_INDEXES") == "1", "read_models": readModels, "search_queries": os.Getenv("POLL_POOL_SEARCH") == "1", "schema": 2, "resource_samples": resourceRows, "process_cpu_seconds": cpuAfter - cpuBefore, "authorization_state": authorizationState, "fixture": "read-mix-v5-isolated-preparation", "available_per_stage": availablePerStage, "task_payload_kib": taskPayloadKiB, "run_payload_kib": runPayloadKiB, "author_history_per_reviewer": authorHistory, "mode": mode, "revision": os.Getenv("POLL_POOL_REVISION"), "actors": totalActors, "actor_counts": map[string]int{"annotation": annotators, "quality": quality, "acceptance": acceptance, "admin": 1}, "claim_counts": claimCounts, "warmup_seconds": warmup, "scheduled_seconds": duration, "actual_seconds": actual.Seconds(), "errors": failures, "failure_samples": failureSamples, "metrics": metrics, "mongo_breakdown": observations, "mongo_commands": commands.Load(), "mongo_reply_bytes": mongoBytes.Load(), "mongo_command_ms": float64(mongoNS.Load()) / 1e6, "mongo_write_commands": writes.Load(), "go_heap_before": memBefore.HeapAlloc, "go_heap_after": memAfter.HeapAlloc, "go_heap_after_gc": memAfterGC.HeapAlloc, "go_gc_cycles": memAfter.NumGC - memBefore.NumGC, "go_allocated_bytes": memAfter.TotalAlloc - memBefore.TotalAlloc, "limitations": []string{"handler timing excludes ingress/network/authentication", "gzip computed separately with fixed Go default compression", "no continuation renew, media or workflow writes", "synthetic fixture; not capacity acceptance", "CPU covers handler and in-process load harness, not Mongo/Pod throttling", "resource heap includes retained benchmark request samples; not standalone service RSS"}, "samples": samples}
|
||||
encoded, e := json.MarshalIndent(report, "", " ")
|
||||
must(e)
|
||||
must(os.WriteFile(out, encoded, 0600))
|
||||
|
||||
204
internal/outputsnapshot/active_queries.go
Normal file
204
internal/outputsnapshot/active_queries.go
Normal file
@ -0,0 +1,204 @@
|
||||
package outputsnapshot
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
_ "time/tzdata"
|
||||
|
||||
"droboticflow/internal/queryserving"
|
||||
"droboticflow/internal/workforcestats"
|
||||
)
|
||||
|
||||
type activeReportQuery struct {
|
||||
Version int
|
||||
Scope string
|
||||
Query workforcestats.OutputQuery
|
||||
Location, Rules string
|
||||
RulesUntil int
|
||||
}
|
||||
|
||||
var reportZoneRules = struct {
|
||||
sync.Mutex
|
||||
values map[string]string
|
||||
}{values: map[string]string{}}
|
||||
|
||||
// A canonical reporting location can be reconstructed in another process only
|
||||
// when its offset rules match exactly, including historical/DST transitions.
|
||||
// Custom locations or non-progressing standard-library zone bounds keep the
|
||||
// existing cold calculation path. Historical rules start at year one; the end
|
||||
// covers the requested dates and current-generation time, with a full-year pad.
|
||||
func reportLocationRules(location *time.Location, untilYear int) string {
|
||||
if location == nil || untilYear < 2 || untilYear > 10001 {
|
||||
return ""
|
||||
}
|
||||
canonical, err := workforcestats.ReportingLocation(location.String())
|
||||
if err != nil || canonical != location {
|
||||
return ""
|
||||
}
|
||||
reportZoneRules.Lock()
|
||||
defer reportZoneRules.Unlock()
|
||||
cacheKey := fmt.Sprintf("%p:%d", location, untilYear)
|
||||
if value := reportZoneRules.values[cacheKey]; value != "" {
|
||||
return value
|
||||
}
|
||||
hash := sha256.New()
|
||||
cursor := time.Date(1, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
end := time.Date(untilYear, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
complete := false
|
||||
for i := 0; i < 30000; i++ {
|
||||
_, offset := cursor.In(location).Zone()
|
||||
fmt.Fprintf(hash, "%d:%d;", cursor.Unix(), offset)
|
||||
_, next := cursor.In(location).ZoneBounds()
|
||||
if next.IsZero() || !next.Before(end) {
|
||||
complete = true
|
||||
break
|
||||
}
|
||||
if !next.After(cursor) {
|
||||
return ""
|
||||
}
|
||||
cursor = next
|
||||
}
|
||||
if !complete {
|
||||
return ""
|
||||
}
|
||||
value := hex.EncodeToString(hash.Sum(nil))
|
||||
if len(reportZoneRules.values) < 64 {
|
||||
reportZoneRules.values[cacheKey] = value
|
||||
}
|
||||
return value
|
||||
}
|
||||
func safeActiveQuery(q workforcestats.OutputQuery) bool {
|
||||
if !(q.MatchUser && q.ScopeUserID != "" || q.MatchSupplier && q.ScopeSupplierID != "") {
|
||||
return false
|
||||
}
|
||||
for _, id := range []string{q.ScopeUserID, q.ScopeSupplierID, q.ProjectID, q.SupplierID, q.UserID} {
|
||||
if len(id) > 256 || strings.Contains(id, "://") || strings.ContainsAny(id, "\x00\r\n?#=") {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return !q.From.IsZero() && !q.To.IsZero() && q.From.Before(q.To)
|
||||
}
|
||||
func (r *Reader) sharedCache() queryserving.Cache {
|
||||
if provider, ok := r.backend.(interface{ SharedQueryCache() queryserving.Cache }); ok {
|
||||
return provider.SharedQueryCache()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func outputSharedQueryKey(scope, generation string, q workforcestats.OutputQuery) string {
|
||||
location := ""
|
||||
if q.Location != nil {
|
||||
_, from := q.From.In(q.Location).Zone()
|
||||
_, to := q.To.In(q.Location).Zone()
|
||||
location = fmt.Sprintf("%s:%d:%d", q.Location.String(), from, to)
|
||||
}
|
||||
return queryserving.Key("output-query/v1", scope, generation, q, location)
|
||||
}
|
||||
func (r *Reader) rememberActiveQuery(ctx context.Context, scope string, q workforcestats.OutputQuery) {
|
||||
registry, ok := r.sharedCache().(queryserving.QueryRegistry)
|
||||
if !ok || !safeActiveQuery(q) {
|
||||
return
|
||||
}
|
||||
if q.Location == nil {
|
||||
q.Location = time.UTC
|
||||
}
|
||||
until := max(q.From.Year(), q.To.Year(), q.Now.Year()) + 2
|
||||
rules := reportLocationRules(q.Location, until)
|
||||
if rules == "" {
|
||||
return
|
||||
}
|
||||
entry := activeReportQuery{Version: 1, Scope: scope, Query: q, Location: q.Location.String(), Rules: rules, RulesUntil: until}
|
||||
entry.Query.Now = time.Time{}
|
||||
entry.Query.Location = nil
|
||||
raw, err := json.Marshal(entry)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
id := queryserving.Key("report-query/v1", entry)
|
||||
now := time.Now()
|
||||
r.mu.Lock()
|
||||
if r.closed || now.Sub(r.registeredQueries[id]) < time.Minute {
|
||||
r.mu.Unlock()
|
||||
return
|
||||
}
|
||||
if len(r.registeredQueries) >= 512 {
|
||||
r.registeredQueries = map[string]time.Time{}
|
||||
}
|
||||
r.registeredQueries[id] = now
|
||||
r.mu.Unlock()
|
||||
registry.RememberQuery(ctx, scope, id, raw)
|
||||
}
|
||||
|
||||
// Warm current registered queries against a complete staged generation before
|
||||
// its head is published. Decode each shard once and reuse the original reducers;
|
||||
// no stale-generation result or alternate business calculation is introduced.
|
||||
func (r *Reader) prewarmActiveQueries(ctx context.Context, scope string, g Generation, payload []byte) int {
|
||||
registry, ok := r.sharedCache().(queryserving.QueryRegistry)
|
||||
if !ok {
|
||||
return 0
|
||||
}
|
||||
budget := 5 * time.Second
|
||||
if deadline, ok := ctx.Deadline(); ok {
|
||||
budget = min(budget, time.Until(deadline)/2)
|
||||
}
|
||||
if budget < 100*time.Millisecond {
|
||||
return 0
|
||||
}
|
||||
warmCtx, stop := context.WithTimeout(ctx, budget)
|
||||
defer stop()
|
||||
queries := []workforcestats.OutputQuery{}
|
||||
reducers := []*workforcestats.OutputReducer{}
|
||||
perPrincipal := map[string]int{}
|
||||
for _, raw := range registry.RecentQueries(warmCtx, scope) {
|
||||
var entry activeReportQuery
|
||||
if len(raw) > 16<<10 || json.Unmarshal(raw, &entry) != nil || entry.Version != 1 || entry.Scope != scope || !safeActiveQuery(entry.Query) {
|
||||
continue
|
||||
}
|
||||
location, err := workforcestats.ReportingLocation(entry.Location)
|
||||
if err != nil || g.AsOf.Year() >= entry.RulesUntil || reportLocationRules(location, entry.RulesUntil) != entry.Rules {
|
||||
continue
|
||||
}
|
||||
q := entry.Query
|
||||
q.Location = location
|
||||
q.Now = g.AsOf
|
||||
principal := fmt.Sprintf("%t:%s:%s", q.MatchUser, q.ScopeUserID, q.ScopeSupplierID)
|
||||
if perPrincipal[principal] >= 2 {
|
||||
continue
|
||||
}
|
||||
perPrincipal[principal]++
|
||||
queries = append(queries, q)
|
||||
reducers = append(reducers, workforcestats.NewOutputReducer(q))
|
||||
if len(queries) >= 128 {
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(queries) == 0 {
|
||||
return 0
|
||||
}
|
||||
if err := r.visitGenerationDatasets(warmCtx, g, payload, func(data workforcestats.OutputDataset) error {
|
||||
for _, reducer := range reducers {
|
||||
if err := warmCtx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
reducer.Add(data)
|
||||
}
|
||||
return nil
|
||||
}); err != nil {
|
||||
return 0
|
||||
}
|
||||
prepared := 0
|
||||
for i, reducer := range reducers {
|
||||
if warmCtx.Err() != nil {
|
||||
break
|
||||
}
|
||||
report := reducer.Report()
|
||||
queryserving.PutJSON(warmCtx, r.sharedCache(), outputSharedQueryKey(scope, g.ID, queries[i]), report, MaxAge)
|
||||
prepared++
|
||||
}
|
||||
return prepared
|
||||
}
|
||||
@ -4,6 +4,8 @@ import (
|
||||
"context"
|
||||
"droboticflow/internal/queryserving"
|
||||
"droboticflow/internal/workforcestats"
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
@ -11,8 +13,9 @@ import (
|
||||
)
|
||||
|
||||
type queryTestCache struct {
|
||||
mu sync.Mutex
|
||||
data map[string][]byte
|
||||
mu sync.Mutex
|
||||
data map[string][]byte
|
||||
registry map[string]map[string][]byte
|
||||
}
|
||||
|
||||
func (c *queryTestCache) Get(_ context.Context, k string) ([]byte, bool) {
|
||||
@ -30,6 +33,27 @@ func (c *queryTestCache) Put(_ context.Context, k string, v []byte, _ time.Durat
|
||||
c.data[k] = append([]byte(nil), v...)
|
||||
}
|
||||
|
||||
func (c *queryTestCache) RememberQuery(_ context.Context, scope, id string, value []byte) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if c.registry == nil {
|
||||
c.registry = map[string]map[string][]byte{}
|
||||
}
|
||||
if c.registry[scope] == nil {
|
||||
c.registry[scope] = map[string][]byte{}
|
||||
}
|
||||
c.registry[scope][id] = append([]byte(nil), value...)
|
||||
}
|
||||
func (c *queryTestCache) RecentQueries(_ context.Context, scope string) [][]byte {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
out := [][]byte{}
|
||||
for _, v := range c.registry[scope] {
|
||||
out = append(out, append([]byte(nil), v...))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
type queryCachedBackend struct {
|
||||
MemoryBackend
|
||||
cache queryTestCache
|
||||
@ -84,3 +108,104 @@ func TestQueryServingSharedOutputCacheKeepsScopeLocationAndGeneration(t *testing
|
||||
t.Fatal("changed project scope borrowed generation", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryServingActiveReportsWarmBeforeGenerationPublication(t *testing.T) {
|
||||
now := time.Now().UTC().Truncate(time.Millisecond)
|
||||
raw := testData(now)
|
||||
backend := &queryCachedBackend{}
|
||||
source := func(ctx context.Context, _ string, _ []string, yield func(workforcestats.OutputSnapshot) error) (*workforcestats.OutputSourceStats, error) {
|
||||
return nil, yield(raw)
|
||||
}
|
||||
reader := NewStreamReader(backend, source)
|
||||
defer reader.Close()
|
||||
var clock atomic.Int64
|
||||
clock.Store(now.UnixNano())
|
||||
reader.now = func() time.Time { return time.Unix(0, clock.Load()).UTC() }
|
||||
if _, e := reader.Prewarm(t.Context(), "org", []string{"p"}); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
var first Info
|
||||
firstDeadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
_, info, e := reader.resolve(t.Context(), "org", []string{"p"}, "")
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
if info.ID != "" {
|
||||
first = info
|
||||
break
|
||||
}
|
||||
if time.Now().After(firstDeadline) {
|
||||
t.Fatal("first generation not ready")
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
queries := []workforcestats.OutputQuery{
|
||||
{From: now.Add(-24 * time.Hour), To: now.Add(time.Hour), Location: time.UTC, MatchUser: true, ScopeUserID: "u"},
|
||||
{From: now.Add(-24 * time.Hour), To: now.Add(time.Hour), Location: time.UTC, MatchUser: true, ScopeUserID: "u", ProductionQuality: true, ReportRole: "annotation"},
|
||||
{From: now.Add(-24 * time.Hour), To: now.Add(time.Hour), Location: time.UTC, MatchSupplier: true, ScopeSupplierID: "v"},
|
||||
}
|
||||
for _, q := range queries {
|
||||
if _, _, e := reader.Query(t.Context(), "org", []string{"p"}, first.ID, q); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
}
|
||||
clock.Store(now.Add(RefreshInterval + time.Second).UnixNano())
|
||||
if _, e := reader.Prewarm(t.Context(), "org", []string{"p"}); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
var next Generation
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
state, e := backend.OutputSnapshotState(t.Context())
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
current := state.Heads[scopeKey("org", []string{"p"})].Current
|
||||
if current != nil && current.ID != first.ID {
|
||||
next = *current
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("next generation not published")
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
if next.Metrics.PreparedQueries != len(queries) {
|
||||
t.Fatal("published without active report preparation", next.Metrics)
|
||||
}
|
||||
fresh := NewStreamReader(backend, source)
|
||||
defer fresh.Close()
|
||||
fresh.now = reader.now
|
||||
reads := backend.blobs.Load()
|
||||
for _, q := range queries {
|
||||
got, info, e := fresh.Query(t.Context(), "org", []string{"p"}, next.ID, q)
|
||||
if e != nil || info.ID != next.ID {
|
||||
t.Fatal("prepared generation", e, info)
|
||||
}
|
||||
dataset := workforcestats.BuildOutputDataset(raw, next.AsOf)
|
||||
before, _ := json.Marshal(dataset)
|
||||
want := workforcestats.QueryOutputDataset(dataset, q)
|
||||
after, _ := json.Marshal(dataset)
|
||||
if !reflect.DeepEqual(before, after) || !reflect.DeepEqual(got, &want) {
|
||||
t.Fatal("prepared report changed calculation or source data")
|
||||
}
|
||||
}
|
||||
if backend.blobs.Load() != reads {
|
||||
t.Fatal("first query decoded source after prepared publication")
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryServingActiveQueryDoesNotSubstituteCustomTimeZones(t *testing.T) {
|
||||
custom := time.FixedZone("UTC", 3600)
|
||||
if reportLocationRules(custom, 2028) != "" {
|
||||
t.Fatal("custom zone treated as canonical UTC")
|
||||
}
|
||||
location, err := workforcestats.ReportingLocation("America/New_York")
|
||||
if err != nil || reportLocationRules(location, 2028) == "" {
|
||||
t.Fatal("canonical DST rules not reusable", err)
|
||||
}
|
||||
if reportLocationRules(time.UTC, 2028) == "" {
|
||||
t.Fatal("canonical UTC not reusable")
|
||||
}
|
||||
}
|
||||
|
||||
@ -20,28 +20,29 @@ import (
|
||||
)
|
||||
|
||||
type Reader struct {
|
||||
queryMemo map[string][]byte
|
||||
queryMemoBytes int
|
||||
queryFlights singleflight.Group
|
||||
querySlots chan struct{}
|
||||
backend Backend
|
||||
source Source
|
||||
stream StreamSource
|
||||
now func() time.Time
|
||||
ctx context.Context
|
||||
cancel context.CancelFunc
|
||||
wg sync.WaitGroup
|
||||
mu sync.Mutex
|
||||
closed bool
|
||||
attempts map[string]time.Time
|
||||
cache map[string][]byte // Compressed bytes only; no unbounded decoded object cache.
|
||||
cacheBytes int
|
||||
reads chan struct{}
|
||||
queryMemo map[string][]byte
|
||||
registeredQueries map[string]time.Time
|
||||
queryMemoBytes int
|
||||
queryFlights singleflight.Group
|
||||
querySlots chan struct{}
|
||||
backend Backend
|
||||
source Source
|
||||
stream StreamSource
|
||||
now func() time.Time
|
||||
ctx context.Context
|
||||
cancel context.CancelFunc
|
||||
wg sync.WaitGroup
|
||||
mu sync.Mutex
|
||||
closed bool
|
||||
attempts map[string]time.Time
|
||||
cache map[string][]byte // Compressed bytes only; no unbounded decoded object cache.
|
||||
cacheBytes int
|
||||
reads chan struct{}
|
||||
}
|
||||
|
||||
func NewReader(b Backend, source Source) *Reader {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
return &Reader{queryMemo: map[string][]byte{}, querySlots: make(chan struct{}, 2), backend: b, source: source, now: time.Now, ctx: ctx, cancel: cancel, attempts: map[string]time.Time{}, cache: map[string][]byte{}, reads: make(chan struct{}, 2)}
|
||||
return &Reader{registeredQueries: map[string]time.Time{}, queryMemo: map[string][]byte{}, querySlots: make(chan struct{}, 2), backend: b, source: source, now: time.Now, ctx: ctx, cancel: cancel, attempts: map[string]time.Time{}, cache: map[string][]byte{}, reads: make(chan struct{}, 2)}
|
||||
}
|
||||
func (r *Reader) Close() { r.mu.Lock(); r.closed = true; r.cancel(); r.mu.Unlock(); r.wg.Wait() }
|
||||
|
||||
@ -242,6 +243,11 @@ func (r *Reader) build(key, org string, projects []string) {
|
||||
buildErr = err
|
||||
return
|
||||
}
|
||||
stage = "query_prewarm"
|
||||
warmStart := time.Now()
|
||||
gen.Metrics.PreparedQueries = r.prewarmActiveQueries(ctx, key, gen, payload)
|
||||
gen.Metrics.PrewarmMS = time.Since(warmStart).Milliseconds()
|
||||
gen.Metrics.DurationMS = time.Since(started).Milliseconds()
|
||||
stage = "publish"
|
||||
published, buildErr = r.backend.PublishOutputSnapshot(ctx, lease, gen, payload, r.now())
|
||||
return
|
||||
@ -265,6 +271,11 @@ func (r *Reader) build(key, org string, projects []string) {
|
||||
}
|
||||
hash := sha256.Sum256(payload)
|
||||
gen := Generation{Metrics: &BuildMetrics{DurationMS: time.Since(started).Milliseconds(), DecodedBytes: decodedBytes, Parts: len(data.Parts), Source: raw.SourceStats}, ID: GenerationID(lease), Slot: lease.Slot, Epoch: lease.Epoch, AsOf: asOf, SHA256: hex.EncodeToString(hash[:]), Bytes: len(payload)}
|
||||
stage = "query_prewarm"
|
||||
warmStart := time.Now()
|
||||
gen.Metrics.PreparedQueries = r.prewarmActiveQueries(ctx, key, gen, payload)
|
||||
gen.Metrics.PrewarmMS = time.Since(warmStart).Milliseconds()
|
||||
gen.Metrics.DurationMS = time.Since(started).Milliseconds()
|
||||
stage = "publish"
|
||||
published, buildErr = r.backend.PublishOutputSnapshot(ctx, lease, gen, payload, r.now())
|
||||
if published {
|
||||
@ -357,17 +368,10 @@ func (r *Reader) Query(ctx context.Context, org string, projects []string, pin s
|
||||
// generation and location identity participate in the key, including personal
|
||||
// and supplier isolation. No permissions or mutable datasets are cached.
|
||||
q.Now = gen.AsOf
|
||||
var shared queryserving.Cache
|
||||
if provider, ok := r.backend.(interface{ SharedQueryCache() queryserving.Cache }); ok {
|
||||
shared = provider.SharedQueryCache()
|
||||
}
|
||||
location := ""
|
||||
if q.Location != nil {
|
||||
_, fromOffset := q.From.In(q.Location).Zone()
|
||||
_, toOffset := q.To.In(q.Location).Zone()
|
||||
location = fmt.Sprintf("%s:%d:%d", q.Location.String(), fromOffset, toOffset)
|
||||
}
|
||||
sharedKey := queryserving.Key("output-query/v1", scopeKey(org, projects), gen.ID, q, location)
|
||||
shared := r.sharedCache()
|
||||
scope := scopeKey(org, projects)
|
||||
r.rememberActiveQuery(ctx, scope, q)
|
||||
sharedKey := outputSharedQueryKey(scope, gen.ID, q)
|
||||
keyData, _ := json.Marshal([]any{gen.ID, q, fmt.Sprintf("%p", q.Location)})
|
||||
key := string(keyData)
|
||||
select {
|
||||
|
||||
@ -119,64 +119,84 @@ func decodeShard(payload []byte, g Generation) (*workforcestats.OutputDataset, e
|
||||
func (r *Reader) queryGeneration(ctx context.Context, g Generation, q workforcestats.OutputQuery) (*workforcestats.OutputReport, error) {
|
||||
q.Now = g.AsOf
|
||||
if g.Shards == 0 {
|
||||
d, err := r.load(ctx, g)
|
||||
data, err := r.load(ctx, g)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
report := workforcestats.QueryOutputDataset(*d, q)
|
||||
report := workforcestats.QueryOutputDataset(*data, q)
|
||||
return &report, ctx.Err()
|
||||
}
|
||||
backend, ok := r.backend.(ShardedBackend)
|
||||
if !ok {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
if g.Shards < 1 || g.Shards > MaxShards || g.Bytes > MaxGenerationBytes {
|
||||
return nil, ErrCapacity
|
||||
}
|
||||
payload, err := r.backend.OutputSnapshotBlob(ctx, g)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
reducer := workforcestats.NewOutputReducer(q)
|
||||
err = r.visitGenerationDatasets(ctx, g, payload, func(data workforcestats.OutputDataset) error { reducer.Add(data); return nil })
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
report := reducer.Report()
|
||||
return &report, ctx.Err()
|
||||
}
|
||||
func (r *Reader) visitGenerationDatasets(ctx context.Context, g Generation, payload []byte, visit func(workforcestats.OutputDataset) error) error {
|
||||
if g.Shards == 0 {
|
||||
if len(payload) != g.Bytes || len(payload) > MaxEncodedBytes || digest(payload) != g.SHA256 {
|
||||
return ErrUnavailable
|
||||
}
|
||||
data, err := decodeShard(payload, g)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return visit(*data)
|
||||
}
|
||||
backend, ok := r.backend.(ShardedBackend)
|
||||
if !ok {
|
||||
return ErrUnavailable
|
||||
}
|
||||
if g.Shards < 1 || g.Shards > MaxShards || g.Bytes > MaxGenerationBytes {
|
||||
return ErrCapacity
|
||||
}
|
||||
var err error
|
||||
if len(payload) != g.ManifestBytes || len(payload) > MaxEncodedBytes || digest(payload) != g.SHA256 {
|
||||
return nil, ErrUnavailable
|
||||
return ErrUnavailable
|
||||
}
|
||||
var manifest snapshotManifest
|
||||
if err = json.Unmarshal(payload, &manifest); err != nil {
|
||||
return nil, err
|
||||
return err
|
||||
}
|
||||
if len(manifest.Shards) != g.Shards {
|
||||
return nil, ErrUnavailable
|
||||
return ErrUnavailable
|
||||
}
|
||||
total := 0
|
||||
for _, d := range manifest.Shards {
|
||||
if d.Bytes < 1 || d.Bytes > MaxEncodedBytes {
|
||||
return nil, ErrCapacity
|
||||
return ErrCapacity
|
||||
}
|
||||
total += d.Bytes
|
||||
}
|
||||
if total != g.Bytes {
|
||||
return nil, ErrUnavailable
|
||||
return ErrUnavailable
|
||||
}
|
||||
reducer := workforcestats.NewOutputReducer(q)
|
||||
for i, descriptor := range manifest.Shards {
|
||||
if err = ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
return err
|
||||
}
|
||||
encoded, err := backend.OutputSnapshotShard(ctx, g, i)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return err
|
||||
}
|
||||
if len(encoded) != descriptor.Bytes || digest(encoded) != descriptor.SHA256 {
|
||||
return nil, ErrUnavailable
|
||||
return ErrUnavailable
|
||||
}
|
||||
d, err := decodeShard(encoded, g)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return err
|
||||
}
|
||||
if err = visit(*d); err != nil {
|
||||
return err
|
||||
}
|
||||
reducer.Add(*d)
|
||||
}
|
||||
report := reducer.Report()
|
||||
return &report, ctx.Err()
|
||||
return ctx.Err()
|
||||
}
|
||||
func (r *Reader) queryResolved(ctx context.Context, g *Generation, info Info, pin string, q workforcestats.OutputQuery) (*workforcestats.OutputReport, Info, error) {
|
||||
report, err := r.queryGeneration(ctx, *g, q)
|
||||
|
||||
@ -31,8 +31,11 @@ var (
|
||||
ErrCapacity = errors.New("snapshot capacity exceeded")
|
||||
)
|
||||
|
||||
// DurationMS covers source reads, calculation and encoding, before publication.
|
||||
// DurationMS covers source reads, calculation, encoding and optional active
|
||||
// query preparation, before publication.
|
||||
type BuildMetrics struct {
|
||||
PreparedQueries int `bson:"preparedQueries,omitempty" json:"preparedQueries,omitempty"`
|
||||
PrewarmMS int64 `bson:"prewarmMs,omitempty" json:"prewarmMs,omitempty"`
|
||||
PeakEncodedBytes int `bson:"peakEncodedBytes,omitempty" json:"peakEncodedBytes,omitempty"`
|
||||
PeakDecodedBytes int `bson:"peakDecodedBytes,omitempty" json:"peakDecodedBytes,omitempty"`
|
||||
DurationMS int64 `bson:"durationMs" json:"durationMs"`
|
||||
|
||||
@ -140,17 +140,13 @@ func (v *Valkey) Get(ctx context.Context, key string) ([]byte, bool) {
|
||||
v.Misses.Add(1)
|
||||
return nil, false
|
||||
}
|
||||
if len(b) < 36 || len(b) > MaxValueBytes+36 || string(b[:4]) != "DFQ1" {
|
||||
v.Misses.Add(1)
|
||||
return nil, false
|
||||
}
|
||||
sum := sha256.Sum256(append([]byte(key+"\x00"), b[36:]...))
|
||||
if !bytes.Equal(sum[:], b[4:36]) {
|
||||
decoded, valid := decodeCacheValue(key, b)
|
||||
if !valid {
|
||||
v.Misses.Add(1)
|
||||
return nil, false
|
||||
}
|
||||
v.Hits.Add(1)
|
||||
return append([]byte(nil), b[36:]...), true
|
||||
return decoded, true
|
||||
}
|
||||
func (v *Valkey) Put(ctx context.Context, key string, b []byte, ttl time.Duration) {
|
||||
parent := ctx
|
||||
@ -162,8 +158,7 @@ func (v *Valkey) Put(ctx context.Context, key string, b []byte, ttl time.Duratio
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 40*time.Millisecond)
|
||||
defer cancel()
|
||||
sum := sha256.Sum256(append([]byte(key+"\x00"), b...))
|
||||
encoded := append(append([]byte("DFQ1"), sum[:]...), b...)
|
||||
encoded := encodeCacheValue(key, b)
|
||||
if err := v.client.Do(ctx, v.client.B().Set().Key(v.prefix+":"+key).Value(string(encoded)).Px(ttl).Build()).Error(); err != nil {
|
||||
if parent.Err() == nil {
|
||||
v.failed()
|
||||
@ -197,3 +192,18 @@ func (m *ManagedValkey) Statistics() map[string]uint64 {
|
||||
}
|
||||
return map[string]uint64{"hits": v.Hits.Load(), "misses": v.Misses.Load(), "errors": v.Errors.Load()}
|
||||
}
|
||||
|
||||
func encodeCacheValue(key string, b []byte) []byte {
|
||||
sum := sha256.Sum256(append([]byte(key+"\x00"), b...))
|
||||
return append(append([]byte("DFQ1"), sum[:]...), b...)
|
||||
}
|
||||
func decodeCacheValue(key string, b []byte) ([]byte, bool) {
|
||||
if len(b) < 36 || len(b) > MaxValueBytes+36 || string(b[:4]) != "DFQ1" {
|
||||
return nil, false
|
||||
}
|
||||
sum := sha256.Sum256(append([]byte(key+"\x00"), b[36:]...))
|
||||
if !bytes.Equal(sum[:], b[4:36]) {
|
||||
return nil, false
|
||||
}
|
||||
return append([]byte(nil), b[36:]...), true
|
||||
}
|
||||
|
||||
96
internal/queryserving/registry.go
Normal file
96
internal/queryserving/registry.go
Normal file
@ -0,0 +1,96 @@
|
||||
package queryserving
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
// QueryRegistry keeps only bounded, short-lived, server-derived report queries.
|
||||
// It contains no permission decisions and never controls task mutations.
|
||||
type QueryRegistry interface {
|
||||
RememberQuery(context.Context, string, string, []byte)
|
||||
RecentQueries(context.Context, string) [][]byte
|
||||
}
|
||||
|
||||
const rememberQueryScript = `
|
||||
redis.call('ZADD',KEYS[1],ARGV[1],ARGV[2])
|
||||
redis.call('HSET',KEYS[2],ARGV[2],ARGV[3])
|
||||
local old=redis.call('ZRANGEBYSCORE',KEYS[1],'-inf',ARGV[4])
|
||||
for _,id in ipairs(old) do redis.call('ZREM',KEYS[1],id);redis.call('HDEL',KEYS[2],id) end
|
||||
local excess=redis.call('ZCARD',KEYS[1])-256
|
||||
if excess>0 then
|
||||
local ids=redis.call('ZRANGE',KEYS[1],0,excess-1)
|
||||
for _,id in ipairs(ids) do redis.call('ZREM',KEYS[1],id);redis.call('HDEL',KEYS[2],id) end
|
||||
end
|
||||
redis.call('EXPIRE',KEYS[1],480);redis.call('EXPIRE',KEYS[2],480)
|
||||
return 1`
|
||||
const recentQueriesScript = `
|
||||
local ids=redis.call('ZREVRANGEBYSCORE',KEYS[1],'+inf',ARGV[1],'LIMIT',0,256)
|
||||
local out={}
|
||||
for _,id in ipairs(ids) do
|
||||
local value=redis.call('HGET',KEYS[2],id)
|
||||
if value then table.insert(out,id);table.insert(out,value) end
|
||||
end
|
||||
return out`
|
||||
|
||||
func (v *Valkey) registryKeys(scope string) (string, string) {
|
||||
root := v.prefix + ":{" + Key("active-query/v1", scope) + "}"
|
||||
return root + ":order", root + ":data"
|
||||
}
|
||||
func (v *Valkey) RememberQuery(parent context.Context, scope, id string, b []byte) {
|
||||
if parent.Err() != nil || len(b) > 16<<10 || time.Now().UnixNano() < v.until.Load() {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(parent, 40*time.Millisecond)
|
||||
defer cancel()
|
||||
order, data := v.registryKeys(scope)
|
||||
now := time.Now().Unix()
|
||||
encoded := encodeCacheValue(scope+":"+id, b)
|
||||
err := v.client.Do(ctx, v.client.B().Eval().Script(rememberQueryScript).Numkeys(2).Key(order, data).Arg(strconv.FormatInt(now, 10), id, string(encoded), strconv.FormatInt(now-240, 10)).Build()).Error()
|
||||
if err != nil && parent.Err() == nil {
|
||||
v.failed()
|
||||
}
|
||||
}
|
||||
func (v *Valkey) RecentQueries(parent context.Context, scope string) [][]byte {
|
||||
if parent.Err() != nil || time.Now().UnixNano() < v.until.Load() {
|
||||
return nil
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(parent, 100*time.Millisecond)
|
||||
defer cancel()
|
||||
order, data := v.registryKeys(scope)
|
||||
values, err := v.client.Do(ctx, v.client.B().Eval().Script(recentQueriesScript).Numkeys(2).Key(order, data).Arg(strconv.FormatInt(time.Now().Unix()-240, 10)).Build()).AsStrSlice()
|
||||
if err != nil {
|
||||
if parent.Err() == nil {
|
||||
v.failed()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if len(values)%2 != 0 || len(values) > 512 {
|
||||
return nil
|
||||
}
|
||||
out := [][]byte{}
|
||||
for i := 0; i < len(values); i += 2 {
|
||||
if b, ok := decodeCacheValue(scope+":"+values[i], []byte(values[i+1])); ok && len(b) <= 16<<10 {
|
||||
out = append(out, b)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
func (m *ManagedValkey) RememberQuery(ctx context.Context, scope, id string, b []byte) {
|
||||
m.mu.RLock()
|
||||
v := m.value
|
||||
m.mu.RUnlock()
|
||||
if v != nil {
|
||||
v.RememberQuery(ctx, scope, id, b)
|
||||
}
|
||||
}
|
||||
func (m *ManagedValkey) RecentQueries(ctx context.Context, scope string) [][]byte {
|
||||
m.mu.RLock()
|
||||
v := m.value
|
||||
m.mu.RUnlock()
|
||||
if v != nil {
|
||||
return v.RecentQueries(ctx, scope)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@ -3,6 +3,7 @@ package queryserving
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
@ -136,3 +137,94 @@ func TestQueryServingValkeyRealIsolationCorruptionAndColdRecoveryLocal(t *testin
|
||||
t.Fatal("cache payload changed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryServingValkeyActiveQueryRegistryLocal(t *testing.T) {
|
||||
if os.Getenv("QUERY_SERVING_LOCAL_TEST") != "1" {
|
||||
t.Skip("isolated Valkey only")
|
||||
}
|
||||
namespace := fmt.Sprintf("registry-%d", time.Now().UnixNano())
|
||||
one, e := NewValkey("valkey://127.0.0.1:16379", namespace)
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
defer one.Close()
|
||||
two, e := NewValkey("valkey://127.0.0.1:16379", namespace)
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
defer two.Close()
|
||||
foreign, e := NewValkey("valkey://127.0.0.1:16379", namespace+"-foreign")
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
defer foreign.Close()
|
||||
ctx := t.Context()
|
||||
scope := "scope"
|
||||
order, data := one.registryKeys(scope)
|
||||
defer one.client.Do(context.Background(), one.client.B().Del().Key(order, data).Build())
|
||||
one.RememberQuery(ctx, scope, "one", []byte("query-one"))
|
||||
if got := two.RecentQueries(ctx, scope); len(got) != 1 || string(got[0]) != "query-one" {
|
||||
t.Fatal("registry not shared", got)
|
||||
}
|
||||
if len(foreign.RecentQueries(ctx, scope)) != 0 || len(two.RecentQueries(ctx, "other-scope")) != 0 {
|
||||
t.Fatal("registry crossed namespace/scope")
|
||||
}
|
||||
for i := 0; i < 270; i++ {
|
||||
one.RememberQuery(ctx, scope, fmt.Sprintf("q-%03d", i), []byte("query"))
|
||||
}
|
||||
if got := two.RecentQueries(ctx, scope); len(got) != 256 {
|
||||
t.Fatal("registry retention not bounded", len(got), one.Errors.Load())
|
||||
}
|
||||
// A syntactically valid JSON value without its key-bound integrity frame is
|
||||
// not allowed to become a registered precomputation.
|
||||
if e = one.client.Do(ctx, one.client.B().Hset().Key(data).FieldValue().FieldValue("bad", "{}").Build()).Error(); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
if e = one.client.Do(ctx, one.client.B().Zadd().Key(order).ScoreMember().ScoreMember(float64(time.Now().Unix()), "bad").Build()).Error(); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
for _, value := range two.RecentQueries(ctx, scope) {
|
||||
if string(value) == "{}" {
|
||||
t.Fatal("corrupt registry value accepted")
|
||||
}
|
||||
}
|
||||
cancelled, stop := context.WithCancel(ctx)
|
||||
stop()
|
||||
before := one.Errors.Load()
|
||||
one.RememberQuery(cancelled, scope, "cancelled", []byte("x"))
|
||||
_ = one.RecentQueries(cancelled, scope)
|
||||
if one.Errors.Load() != before {
|
||||
t.Fatal("caller cancellation tripped registry breaker")
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryServingLongSearchKeepsConservativeOperatorBudget(t *testing.T) {
|
||||
text := "synthetic-available-load-project-a-load-supplier-a-prepackage_acceptance-0"
|
||||
grams := candidateGrams(text, "another-long-capture/part0001")
|
||||
if len(grams) > 40 || len(grams) == 0 {
|
||||
t.Fatal("filter operation budget", len(grams))
|
||||
}
|
||||
full := map[string]bool{}
|
||||
for _, g := range IndexGrams(text + " another-long-capture/part0001") {
|
||||
full[g] = true
|
||||
}
|
||||
for _, g := range grams {
|
||||
if !full[g] {
|
||||
t.Fatal("selected constraint can lose a true result")
|
||||
}
|
||||
}
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if count := strings.Count(r.URL.Query().Get("filter_by"), "grams:="); count > 40 {
|
||||
t.Error("long query rejected by server budget", count)
|
||||
w.WriteHeader(400)
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte("{\"item_id\":\"expected\"}\n"))
|
||||
}))
|
||||
defer server.Close()
|
||||
client, _ := NewTypesense(server.URL, "synthetic")
|
||||
ids, e := client.Candidates(t.Context(), "dfq_budget", text, "another-long-capture/part0001")
|
||||
if e != nil || !reflect.DeepEqual(ids, []string{"expected"}) {
|
||||
t.Fatal("long search response", ids, e)
|
||||
}
|
||||
}
|
||||
|
||||
@ -149,10 +149,8 @@ func (t *Typesense) Candidates(ctx context.Context, c string, terms ...string) (
|
||||
return nil, errors.New("invalid search collection")
|
||||
}
|
||||
filters := []string{}
|
||||
for _, term := range terms {
|
||||
for _, g := range Grams(term) {
|
||||
filters = append(filters, "grams:=["+g+"]")
|
||||
}
|
||||
for _, g := range candidateGrams(terms...) {
|
||||
filters = append(filters, "grams:=["+g+"]")
|
||||
}
|
||||
params := url.Values{"include_fields": {"item_id"}}
|
||||
if len(filters) > 0 {
|
||||
@ -188,3 +186,30 @@ func (t *Typesense) Delete(ctx context.Context, c string) error {
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// Typesense's default parser counts both clauses and boolean operators. Keep
|
||||
// well below its 100-operation limit without changing that server setting.
|
||||
// Dropping prefilter constraints only adds candidates; the original complete
|
||||
// query is still matched by Go, and no task/result inventory is truncated.
|
||||
func candidateGrams(terms ...string) []string {
|
||||
set := map[string]bool{}
|
||||
for _, term := range terms {
|
||||
for _, gram := range Grams(term) {
|
||||
set[gram] = true
|
||||
}
|
||||
}
|
||||
all := make([]string, 0, len(set))
|
||||
for gram := range set {
|
||||
all = append(all, gram)
|
||||
}
|
||||
sort.Strings(all)
|
||||
const limit = 40
|
||||
if len(all) <= limit {
|
||||
return all
|
||||
}
|
||||
selected := make([]string, limit)
|
||||
for i := range selected {
|
||||
selected[i] = all[i*(len(all)-1)/(limit-1)]
|
||||
}
|
||||
return selected
|
||||
}
|
||||
|
||||
@ -305,7 +305,9 @@ func (s *MongoStore) scanAdminAnnotationPool(ctx context.Context, org string, q
|
||||
if len(ids) == 0 {
|
||||
return nil
|
||||
}
|
||||
pipeline[0] = bson.D{{Key: "$match", Value: bson.M{"$and": bson.A{pipeline[0][0].Value, bson.M{"id": bson.M{"$in": ids}}}}}}
|
||||
if scoped, withinBudget := queryCandidateFilter(pipeline[0][0].Value, ids); withinBudget {
|
||||
pipeline[0] = bson.D{{Key: "$match", Value: scoped}}
|
||||
}
|
||||
}
|
||||
cur, err := s.humanItems.Aggregate(ctx, pipeline, options.Aggregate().SetAllowDiskUse(false).SetMaxTime(8*time.Second).SetBatchSize(200))
|
||||
if err != nil {
|
||||
|
||||
@ -128,7 +128,9 @@ func (s *MongoStore) prepareHumanPoolSearchFilter(ctx context.Context, org strin
|
||||
if len(ids) == 0 {
|
||||
return nil, true, nil
|
||||
}
|
||||
return bson.M{"$and": bson.A{filter, bson.M{"id": bson.M{"$in": ids}}}}, false, nil
|
||||
if scoped, withinBudget := queryCandidateFilter(filter, ids); withinBudget {
|
||||
return scoped, false, nil
|
||||
}
|
||||
}
|
||||
candidates, err := s.humanPoolSearchCandidates(ctx, org, q, filter)
|
||||
if err != nil {
|
||||
|
||||
@ -26,6 +26,11 @@ func (s *MongoStore) cleanQueryModelOrphans(ctx context.Context) error {
|
||||
keep = append(keep, h.Generation)
|
||||
}
|
||||
filter := bson.M{"generation": bson.M{"$nin": keep}, "createdAt": bson.M{"$lt": time.Now().UTC().Add(-15 * time.Minute)}}
|
||||
// Ordinary orphan chunks/rows must still be reclaimed when Typesense is
|
||||
// offline. Retain only its collection intents until remote deletion succeeds.
|
||||
if _, err := s.queryData().DeleteMany(ctx, bson.M{"$and": bson.A{filter, bson.M{"kind": bson.M{"$ne": "search_collection"}}}}); err != nil {
|
||||
return err
|
||||
}
|
||||
// Only this database's registered, unreachable search generations are
|
||||
// eligible; other deployments and current generations remain untouched.
|
||||
cur, err := s.queryData().Find(ctx, bson.M{"$and": bson.A{filter, bson.M{"kind": "search_collection"}}}, options.Find().SetProjection(bson.M{"_id": 1, "collection": 1}))
|
||||
@ -45,6 +50,9 @@ func (s *MongoStore) cleanQueryModelOrphans(ctx context.Context) error {
|
||||
if err = s.queryServing.search.Delete(ctx, row.Collection); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err = s.queryData().DeleteOne(ctx, bson.M{"_id": row.ID}); err != nil {
|
||||
return err
|
||||
}
|
||||
} else {
|
||||
return nil
|
||||
}
|
||||
@ -52,8 +60,7 @@ func (s *MongoStore) cleanQueryModelOrphans(ctx context.Context) error {
|
||||
if err = cur.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = s.queryData().DeleteMany(ctx, filter)
|
||||
return err
|
||||
return nil
|
||||
}
|
||||
|
||||
// Obsolete configured scopes cannot pin an unbounded history of generations.
|
||||
|
||||
@ -44,8 +44,9 @@ func (b *QueryReadBuilder) WaitReady(ctx context.Context, org string, projects [
|
||||
tick := time.NewTicker(100 * time.Millisecond)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
_, production := b.store.readQueryModelHead(ctx, queryModelKey("production", org, projects), 5*time.Minute)
|
||||
productionHead, production := b.store.readQueryModelHead(ctx, queryModelKey("production", org, projects), 5*time.Minute)
|
||||
st, err := b.store.querySearchState(ctx)
|
||||
production = production && productionHead.GlobalRevision == st.Global
|
||||
h, exists := st.Heads[queryModelKey("search", org, nil)]
|
||||
search := err == nil && exists && h.Generation != "" && h.Revision == st.Revisions[queryModelKey("search", org, nil)] && h.GlobalRevision == st.Global && time.Since(st.ObservedAt) >= 0 && time.Since(st.ObservedAt) < 5*time.Second
|
||||
output := len(projects) == 0
|
||||
@ -106,8 +107,9 @@ func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org s
|
||||
started := time.Now()
|
||||
var source struct {
|
||||
Revision int64 `bson:"productionRevision"`
|
||||
Global int64 `bson:"searchGlobalRevision"`
|
||||
}
|
||||
if err := b.store.queryControl().FindOne(ctx, bson.M{"_id": "global"}, options.FindOne().SetProjection(bson.M{"productionRevision": 1})).Decode(&source); err != nil {
|
||||
if err := b.store.queryControl().FindOne(ctx, bson.M{"_id": "global"}, options.FindOne().SetProjection(bson.M{"productionRevision": 1, "searchGlobalRevision": 1})).Decode(&source); err != nil {
|
||||
return err
|
||||
}
|
||||
buildCtx, finish := context.WithTimeout(ctx, 5*time.Minute)
|
||||
@ -126,6 +128,7 @@ func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org s
|
||||
return err
|
||||
}
|
||||
head.SourceRevision = source.Revision
|
||||
head.GlobalRevision = source.Global
|
||||
if err = b.store.publishQueryModelHead(ctx, l, key, head); err != nil {
|
||||
return err
|
||||
}
|
||||
@ -141,30 +144,40 @@ func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org s
|
||||
|
||||
// RunOnce is for bounded initial prewarming and isolated tests, not a freshness
|
||||
// substitute for Run. After the heartbeat expires search falls back to old reads.
|
||||
func (b *QueryReadBuilder) RunOnce(ctx context.Context) error {
|
||||
l, ok, err := b.store.acquireQueryModelLease(ctx)
|
||||
func (b *QueryReadBuilder) RunOnce(parent context.Context) error {
|
||||
ctx, cancel := context.WithTimeout(parent, 5*time.Minute)
|
||||
defer cancel()
|
||||
started := make(chan struct{})
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- b.run(ctx, started); cancel() }()
|
||||
select {
|
||||
case <-started:
|
||||
case err := <-done:
|
||||
return err
|
||||
case <-ctx.Done():
|
||||
cancel()
|
||||
<-done
|
||||
return ctx.Err()
|
||||
}
|
||||
scopes, err := b.scopes(ctx)
|
||||
if err == nil {
|
||||
for org, ids := range scopes {
|
||||
if err = b.WaitReady(ctx, org, ids); err != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
cancel()
|
||||
runErr := <-done
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !ok {
|
||||
return errors.New("query builder already active")
|
||||
if runErr != nil && !errors.Is(runErr, context.Canceled) {
|
||||
return runErr
|
||||
}
|
||||
return b.withLease(ctx, l, func(ctx context.Context) error {
|
||||
scopes, err := b.scopes(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for org, ids := range scopes {
|
||||
if err = b.rebuild(ctx, l, org, ids); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = b.store.buildQuerySearchModel(ctx, l, org); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *QueryReadBuilder) withLease(ctx context.Context, l queryModelLease, run func(context.Context) error) error {
|
||||
ctx, cancel := context.WithCancel(ctx)
|
||||
defer cancel()
|
||||
@ -194,7 +207,9 @@ func (b *QueryReadBuilder) withLease(ctx context.Context, l queryModelLease, run
|
||||
return err
|
||||
}
|
||||
|
||||
func (b *QueryReadBuilder) Run(ctx context.Context) error {
|
||||
func (b *QueryReadBuilder) Run(ctx context.Context) error { return b.run(ctx, nil) }
|
||||
|
||||
func (b *QueryReadBuilder) run(ctx context.Context, started chan<- struct{}) error {
|
||||
b.mu.Lock()
|
||||
b.all, b.allSearch = true, true
|
||||
b.sourceDigests = map[string]querySourceDigest{}
|
||||
@ -207,6 +222,17 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
|
||||
return errors.New("query builder already active")
|
||||
}
|
||||
return b.withLease(ctx, l, func(ctx context.Context) error {
|
||||
ctx, cancelWork := context.WithCancel(ctx)
|
||||
defer cancelWork()
|
||||
// A new observer has not covered changes during downtime. Old search
|
||||
// generations must remain unavailable until a complete reconciliation.
|
||||
result, e := b.store.queryControl().UpdateOne(ctx, queryLeaseFilter(l), bson.M{"$inc": bson.M{"searchGlobalRevision": int64(1), "productionRevision": int64(1)}, "$set": bson.M{"observedAt": time.Time{}}})
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if result.MatchedCount != 1 {
|
||||
return errors.New("query restart fenced")
|
||||
}
|
||||
// No fullDocument lookup: projecting only event identity avoids copying
|
||||
// annotation bodies, credentials or full video payloads on each save.
|
||||
pipeline := mongo.Pipeline{{{Key: "$match", Value: bson.M{"ns.coll": bson.M{"$in": querySourceCollections}}}}, {{Key: "$project", Value: bson.M{"_id": 1, "ns": 1, "documentKey": 1, "operationType": 1, "clusterTime": 1}}}}
|
||||
@ -214,7 +240,11 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
|
||||
if err != nil {
|
||||
return fmt.Errorf("query builder needs available Mongo change streams: %w", err)
|
||||
}
|
||||
defer stream.Close(context.Background())
|
||||
defer func() {
|
||||
closeCtx, closeCancel := context.WithTimeout(context.Background(), time.Second)
|
||||
defer closeCancel()
|
||||
_ = stream.Close(closeCtx)
|
||||
}()
|
||||
// Seed before consuming: events already captured by the open stream
|
||||
// close the initial snapshot/change gap without retaining customer data.
|
||||
seedCtx, seedCancel := context.WithTimeout(ctx, 2*time.Minute)
|
||||
@ -227,6 +257,14 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
|
||||
defer stop()
|
||||
watchDone := make(chan error, 1)
|
||||
watchExited := make(chan struct{})
|
||||
var watchFailure error
|
||||
failWatch := func(e error) {
|
||||
if watchCtx.Err() == nil {
|
||||
watchFailure = e
|
||||
}
|
||||
watchDone <- e
|
||||
cancelWork()
|
||||
}
|
||||
go func() {
|
||||
defer close(watchExited)
|
||||
lastHeartbeat := time.Time{}
|
||||
@ -243,7 +281,7 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
|
||||
ClusterTime primitive.Timestamp `bson:"clusterTime"`
|
||||
}
|
||||
if e := stream.Decode(&event); e != nil {
|
||||
watchDone <- e
|
||||
failWatch(e)
|
||||
return
|
||||
}
|
||||
pendingChange = true
|
||||
@ -251,18 +289,18 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
|
||||
observed = time.Unix(int64(event.ClusterTime.T), 0).UTC()
|
||||
}
|
||||
if e := b.observeQuerySourceEvent(watchCtx, l, event.NS.Collection, event.Operation, event.Key); e != nil {
|
||||
watchDone <- e
|
||||
failWatch(e)
|
||||
return
|
||||
}
|
||||
} else if e := stream.Err(); e != nil {
|
||||
watchDone <- e
|
||||
failWatch(e)
|
||||
return
|
||||
}
|
||||
if time.Since(lastHeartbeat) < time.Second {
|
||||
continue
|
||||
}
|
||||
if e := b.recordObserved(watchCtx, l, observed, pendingChange); e != nil {
|
||||
watchDone <- e
|
||||
failWatch(e)
|
||||
return
|
||||
}
|
||||
pendingChange = false
|
||||
@ -270,11 +308,24 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
|
||||
}
|
||||
watchDone <- watchCtx.Err()
|
||||
}()
|
||||
defer func() { stop(); <-watchExited }()
|
||||
defer func() {
|
||||
stop()
|
||||
<-watchExited
|
||||
if watchFailure != nil {
|
||||
cleanup, finish := context.WithTimeout(context.Background(), time.Second)
|
||||
defer finish()
|
||||
_, _ = b.store.queryControl().UpdateOne(cleanup, queryLeaseFilter(l), bson.M{"$set": bson.M{"observedAt": time.Time{}}})
|
||||
}
|
||||
}()
|
||||
if started != nil {
|
||||
close(started)
|
||||
}
|
||||
tick := time.NewTicker(time.Second)
|
||||
defer tick.Stop()
|
||||
pendingStats, pendingSearch := map[string]bool{}, map[string]bool{}
|
||||
lastStats := map[string]time.Time{}
|
||||
searchRetries := map[string]queryBuildRetry{}
|
||||
statsRetries := map[string]queryBuildRetry{}
|
||||
var scopes map[string][]string
|
||||
refreshScopes := true
|
||||
lastScopeRead := time.Time{}
|
||||
@ -316,9 +367,16 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
|
||||
delete(pendingSearch, org)
|
||||
continue
|
||||
}
|
||||
if time.Now().Before(searchRetries[org].Next) {
|
||||
continue
|
||||
}
|
||||
if e := b.store.buildQuerySearchModel(ctx, l, org); e == nil {
|
||||
delete(searchRetries, org)
|
||||
delete(pendingSearch, org)
|
||||
} else {
|
||||
retry := searchRetries[org]
|
||||
retry.fail(5*time.Second, time.Minute)
|
||||
searchRetries[org] = retry
|
||||
log.Print("query_search_build result=retry category=source_or_publication")
|
||||
}
|
||||
}
|
||||
@ -331,10 +389,17 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
|
||||
if time.Since(lastStats[org]) < 30*time.Second {
|
||||
continue
|
||||
}
|
||||
if time.Now().Before(statsRetries[org].Next) {
|
||||
continue
|
||||
}
|
||||
if e := b.rebuild(ctx, l, org, ids); e == nil {
|
||||
delete(statsRetries, org)
|
||||
delete(pendingStats, org)
|
||||
lastStats[org] = time.Now()
|
||||
} else {
|
||||
retry := statsRetries[org]
|
||||
retry.fail(30*time.Second, 5*time.Minute)
|
||||
statsRetries[org] = retry
|
||||
log.Print("query_model_build result=retry category=source_or_publication")
|
||||
}
|
||||
}
|
||||
@ -352,8 +417,6 @@ func (b *QueryReadBuilder) Run(ctx context.Context) error {
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
stop()
|
||||
<-watchDone
|
||||
return ctx.Err()
|
||||
case e := <-watchDone:
|
||||
return e
|
||||
@ -382,3 +445,17 @@ func querySourceCollection(name string) bool {
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
type queryBuildRetry struct {
|
||||
Next time.Time
|
||||
Delay time.Duration
|
||||
}
|
||||
|
||||
func (r *queryBuildRetry) fail(initial, maximum time.Duration) {
|
||||
if r.Delay == 0 {
|
||||
r.Delay = initial
|
||||
} else {
|
||||
r.Delay = min(r.Delay*2, maximum)
|
||||
}
|
||||
r.Next = time.Now().Add(r.Delay)
|
||||
}
|
||||
|
||||
@ -22,7 +22,7 @@ import (
|
||||
"go.mongodb.org/mongo-driver/mongo/options"
|
||||
)
|
||||
|
||||
const queryModelSchema = "query-model/v1"
|
||||
const queryModelSchema = "query-model/v2"
|
||||
const queryModelChunkBytes = 2 << 20
|
||||
const queryModelMaxBytes = 64 << 20
|
||||
|
||||
|
||||
@ -44,15 +44,17 @@ func newQueryModelFixture(t *testing.T, n int) queryModelFixture {
|
||||
t.Cleanup(func() {
|
||||
cleanup, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
var control struct {
|
||||
Heads map[string]QueryModelHead `bson:"heads"`
|
||||
}
|
||||
if s.queryControl().FindOne(cleanup, bson.M{"_id": "global"}).Decode(&control) == nil {
|
||||
for _, h := range control.Heads {
|
||||
if h.Collection != "" {
|
||||
_ = search.Delete(cleanup, h.Collection)
|
||||
cur, e := s.queryData().Find(cleanup, bson.M{"kind": "search_collection"}, options.Find().SetProjection(bson.M{"collection": 1}))
|
||||
if e == nil {
|
||||
for cur.Next(cleanup) {
|
||||
var row struct {
|
||||
Collection string `bson:"collection"`
|
||||
}
|
||||
if cur.Decode(&row) == nil {
|
||||
_ = search.Delete(cleanup, row.Collection)
|
||||
}
|
||||
}
|
||||
_ = cur.Close(cleanup)
|
||||
}
|
||||
_ = db.Drop(cleanup)
|
||||
_ = client.Disconnect(cleanup)
|
||||
|
||||
@ -4,8 +4,13 @@ import (
|
||||
"context"
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/queryserving"
|
||||
"fmt"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
@ -229,3 +234,212 @@ func TestQueryReadModelsCachedResponseScopeAndStaleFallbackLocal(t *testing.T) {
|
||||
t.Fatal("stale model counted as fresh")
|
||||
}
|
||||
}
|
||||
|
||||
func TestQuerySearchUnicodeCaptureFoldCompatibilityLocal(t *testing.T) {
|
||||
f := newQueryModelFixture(t, 3)
|
||||
s, ctx := f.s, t.Context()
|
||||
names := []string{"Σ", "S", "K"}
|
||||
variants := []string{"ς", "ſ", "K"}
|
||||
for i, name := range names {
|
||||
if e := s.UpsertCaptureRecord(ctx, domain.CaptureRecord{ID: fmt.Sprintf("capture-item-%06d", i), OrganizationID: f.org, ProjectID: "p", CaptureID: name, Prefix: name}); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
if !strings.EqualFold(name, variants[i]) || queryCaptureFold(name) != queryCaptureFold(variants[i]) {
|
||||
t.Fatal("capture folding is incompatible")
|
||||
}
|
||||
}
|
||||
b := NewQueryReadBuilder(s)
|
||||
defer b.Close()
|
||||
if e := b.RunOnce(ctx); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
for i, q := range variants {
|
||||
ids, ready := s.querySearchCandidates(ctx, f.org, "", q)
|
||||
want := fmt.Sprintf("item-%06d", i)
|
||||
found := false
|
||||
for _, id := range ids {
|
||||
found = found || id == want
|
||||
}
|
||||
if !ready || !found {
|
||||
t.Fatal("Unicode capture lost from candidates", q, ids, ready)
|
||||
}
|
||||
items, e := s.ListHumanItems(ctx, f.org, domain.HumanItemQuery{PluginID: "annotation-qa", Capture: q, Limit: 10})
|
||||
if e != nil || !reflect.DeepEqual(queryIDs(items), []string{want}) {
|
||||
t.Fatal("exact capture changed", q, queryIDs(items), e)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestQuerySearchCombinedFilterBudgetRetainsLegacyPath(t *testing.T) {
|
||||
small, ok := queryCandidateFilter(bson.M{"organizationId": "org"}, []string{"a"})
|
||||
if !ok || len(small) == 0 {
|
||||
t.Fatal("ordinary filter rejected")
|
||||
}
|
||||
// Previously valid large history filters must not exceed BSON's command
|
||||
// budget only because an optional candidate index was added.
|
||||
original := bson.M{"history": strings.Repeat("x", 12<<20)}
|
||||
if _, ok = queryCandidateFilter(original, []string{"a"}); ok {
|
||||
t.Fatal("combined filter budget ignored")
|
||||
}
|
||||
if len(original) != 1 {
|
||||
t.Fatal("fallback changed original scope")
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryReadModelsRestartCannotTrustOldCatalogDuringFailedRebuildLocal(t *testing.T) {
|
||||
f := newQueryModelFixture(t, 3)
|
||||
s, ctx := f.s, t.Context()
|
||||
b := NewQueryReadBuilder(s)
|
||||
defer b.Close()
|
||||
if e := b.RunOnce(ctx); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
original, _ := s.querySearchState(ctx)
|
||||
if e := s.UpsertProject(ctx, domain.ProjectRecord{ID: "p", OrganizationID: f.org, Name: "downtime-project-name"}); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
var calls atomic.Int64
|
||||
offline := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { calls.Add(1); w.WriteHeader(503) }))
|
||||
defer offline.Close()
|
||||
backend, e := queryserving.NewTypesense(offline.URL, "synthetic")
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
s.queryServing.search = backend
|
||||
runCtx, stop := context.WithCancel(ctx)
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- b.Run(runCtx) }()
|
||||
defer func() { stop(); <-done; s.queryServing.search = f.search }()
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for {
|
||||
st, e := s.querySearchState(ctx)
|
||||
if e == nil && st.Global > original.Global && time.Since(st.ObservedAt) < 5*time.Second {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("observer did not restart")
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
if _, ready := s.querySearchCandidates(ctx, f.org, "downtime-project-name", ""); ready {
|
||||
t.Fatal("old catalog trusted before reconciliation")
|
||||
}
|
||||
items, e := s.ListHumanItems(ctx, f.org, domain.HumanItemQuery{Search: "downtime-project-name", Limit: 10})
|
||||
if e != nil || len(items) != 3 {
|
||||
t.Fatal("failed rebuild blocked legacy search", e, len(items))
|
||||
}
|
||||
time.Sleep(1200 * time.Millisecond)
|
||||
if calls.Load() > 2 {
|
||||
t.Fatal("backend outage caused a tight rebuild loop", calls.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryReadModelsPublicationCannotAdvanceObserverProofLocal(t *testing.T) {
|
||||
f := newQueryModelFixture(t, 3)
|
||||
s, ctx := f.s, t.Context()
|
||||
b := NewQueryReadBuilder(s)
|
||||
defer b.Close()
|
||||
if e := b.RunOnce(ctx); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
l, ok, e := s.acquireQueryModelLease(ctx)
|
||||
if e != nil || !ok {
|
||||
t.Fatal("lease", e)
|
||||
}
|
||||
_, e = s.queryControl().UpdateOne(ctx, queryLeaseFilter(l), bson.M{"$set": bson.M{"observedAt": time.Now().Add(-time.Minute)}})
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
if e = s.buildQuerySearchModel(ctx, l, f.org); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
if _, ready := s.querySearchCandidates(ctx, f.org, "part0002", ""); ready {
|
||||
t.Fatal("publication forged fresh observer proof")
|
||||
}
|
||||
// Backend outage retains remote cleanup intent, without retaining all orphan
|
||||
// report/search payloads. Restoring the backend completes cleanup.
|
||||
_, e = s.queryData().InsertOne(ctx, bson.M{"_id": "orphan-payload", "generation": "orphan-generation", "createdAt": time.Now().Add(-time.Hour), "data": []byte("old")})
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
_, e = s.queryData().InsertOne(ctx, bson.M{"_id": "orphan-intent", "generation": "orphan-generation", "kind": "search_collection", "collection": fmt.Sprintf("dfq_gc_%d", time.Now().UnixNano()), "createdAt": time.Now().Add(-time.Hour)})
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
s.queryServing.search = nil
|
||||
if e = s.cleanQueryModelOrphans(ctx); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
n, e := s.queryData().CountDocuments(ctx, bson.M{"_id": "orphan-payload"})
|
||||
if e != nil || n != 0 {
|
||||
t.Fatal("unavailable search backend retained payload garbage", e)
|
||||
}
|
||||
n, e = s.queryData().CountDocuments(ctx, bson.M{"_id": "orphan-intent"})
|
||||
if e != nil || n != 1 {
|
||||
t.Fatal("remote cleanup intent lost", e)
|
||||
}
|
||||
s.queryServing.search = f.search
|
||||
if e = s.cleanQueryModelOrphans(ctx); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
n, e = s.queryData().CountDocuments(ctx, bson.M{"_id": "orphan-intent"})
|
||||
if e != nil || n != 0 {
|
||||
t.Fatal("recovered backend did not finish cleanup", e)
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryReadModelsRetryBackoffIsBounded(t *testing.T) {
|
||||
var retry queryBuildRetry
|
||||
for i := 0; i < 20; i++ {
|
||||
retry.fail(5*time.Second, time.Minute)
|
||||
if retry.Delay < 5*time.Second || retry.Delay > time.Minute || time.Until(retry.Next) > time.Minute {
|
||||
t.Fatal("unbounded retry", retry)
|
||||
}
|
||||
}
|
||||
if retry.Delay != time.Minute {
|
||||
t.Fatal("backoff did not reach limit")
|
||||
}
|
||||
}
|
||||
|
||||
func TestQuerySearchCredentialsInLegacyNamesKeepOriginalPathLocal(t *testing.T) {
|
||||
f := newQueryModelFixture(t, 1)
|
||||
s, ctx := f.s, t.Context()
|
||||
item, e := s.GetHumanItem(ctx, "item-000000")
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
item.Subject.ArtifactID = "https://synthetic.invalid/video.mp4?signature=synthetic-private"
|
||||
if e = s.UpsertHumanItem(ctx, *item); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
before, e := s.GetHumanItem(ctx, item.ID)
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
l, ok, e := s.acquireQueryModelLease(ctx)
|
||||
if e != nil || !ok {
|
||||
t.Fatal("lease", e)
|
||||
}
|
||||
if e = s.buildQuerySearchModel(ctx, l, f.org); e == nil {
|
||||
t.Fatal("credential-bearing legacy name materialized")
|
||||
}
|
||||
st, e := s.querySearchState(ctx)
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
if st.Heads[queryModelKey("search", f.org, nil)].Generation != "" {
|
||||
t.Fatal("partial catalog published")
|
||||
}
|
||||
n, e := s.queryData().CountDocuments(ctx, bson.M{"text": bson.M{"$regex": "synthetic-private"}})
|
||||
if e != nil || n != 0 {
|
||||
t.Fatal("credential in derived rows", n, e)
|
||||
}
|
||||
after, e := s.GetHumanItem(ctx, item.ID)
|
||||
if e != nil || !reflect.DeepEqual(before, after) {
|
||||
t.Fatal("fallback changed annotation", e)
|
||||
}
|
||||
rows, e := s.ListHumanItems(ctx, f.org, domain.HumanItemQuery{Search: "synthetic-private", Limit: 10})
|
||||
if e != nil || len(rows) != 1 {
|
||||
t.Fatal("legacy search lost", len(rows), e)
|
||||
}
|
||||
}
|
||||
|
||||
@ -9,6 +9,7 @@ import (
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/queryserving"
|
||||
@ -41,7 +42,7 @@ func querySafePath(value string) string {
|
||||
func searchRowText(row adminPoolRow) string {
|
||||
// Both exact historical search corpora remain contiguous; adding fields in
|
||||
// between could otherwise lose cross-field substring matches.
|
||||
return adminPoolSearchText(row) + "\n" + humanPoolFallbackSearchText(row)
|
||||
return adminPoolSearchText(row) + "\n" + humanPoolFallbackSearchText(row) + "\n" + queryCaptureFold(strings.Join(adminPoolCaptureValues(row), "\n"))
|
||||
}
|
||||
|
||||
func (s *MongoStore) buildQuerySearchModel(ctx context.Context, l queryModelLease, org string) error {
|
||||
@ -109,6 +110,11 @@ func (s *MongoStore) buildQuerySearchModel(ctx context.Context, l queryModelLeas
|
||||
}
|
||||
}
|
||||
text := searchRowText(row)
|
||||
for _, value := range strings.Fields(text) {
|
||||
if querySafePath(value) != value {
|
||||
return errors.New("credential-bearing search field requires legacy search")
|
||||
}
|
||||
}
|
||||
if len(text) > 64<<10 {
|
||||
return errors.New("search names exceed materialization budget")
|
||||
}
|
||||
@ -133,7 +139,7 @@ func (s *MongoStore) buildQuerySearchModel(ctx context.Context, l queryModelLeas
|
||||
// Missing revision zero is equivalent to zero, without writing business data.
|
||||
filter["$expr"] = bson.M{"$and": bson.A{bson.M{"$eq": bson.A{bson.M{"$ifNull": bson.A{"$searchRevisions." + key, int64(0)}}, h.Revision}}, bson.M{"$eq": bson.A{bson.M{"$ifNull": bson.A{"$searchGlobalRevision", int64(0)}}, h.GlobalRevision}}}}
|
||||
delete(filter, "searchGlobalRevision")
|
||||
res, err := s.queryControl().UpdateOne(ctx, filter, bson.M{"$set": bson.M{"heads." + key: h, "observedAt": time.Now().UTC()}})
|
||||
res, err := s.queryControl().UpdateOne(ctx, filter, bson.M{"$set": bson.M{"heads." + key: h}})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@ -168,7 +174,7 @@ func (s *MongoStore) querySearchCandidates(ctx context.Context, org, search, cap
|
||||
return ids, true
|
||||
}
|
||||
if h.Collection != "" && s.queryServing.search != nil {
|
||||
if out, err := s.queryServing.search.Candidates(short, h.Collection, search, capture); err == nil && queryCandidateBudget(out) {
|
||||
if out, err := s.queryServing.search.Candidates(short, h.Collection, search, queryCaptureFold(capture)); err == nil && queryCandidateBudget(out) {
|
||||
s.queryServing.stats.searchHits.Add(1)
|
||||
s.queryServing.stats.typesenseReads.Add(1)
|
||||
queryserving.PutJSON(short, s.SharedQueryCache(), cacheKey, out, 5*time.Minute)
|
||||
@ -176,7 +182,7 @@ func (s *MongoStore) querySearchCandidates(ctx context.Context, org, search, cap
|
||||
}
|
||||
}
|
||||
clauses := bson.A{bson.M{"generation": h.Generation, "organizationId": normalizedOrganizationID(org)}}
|
||||
for _, term := range []string{search, capture} {
|
||||
for _, term := range []string{search, queryCaptureFold(capture)} {
|
||||
if term != "" {
|
||||
clauses = append(clauses, bson.M{"text": bson.M{"$regex": regexp.QuoteMeta(strings.ToLower(term))}})
|
||||
}
|
||||
@ -187,6 +193,7 @@ func (s *MongoStore) querySearchCandidates(ctx context.Context, org, search, cap
|
||||
}
|
||||
defer cur.Close(short)
|
||||
ids = []string{}
|
||||
candidateBytes := 0
|
||||
for cur.Next(short) {
|
||||
var row struct {
|
||||
ID string `bson:"itemId"`
|
||||
@ -195,7 +202,8 @@ func (s *MongoStore) querySearchCandidates(ctx context.Context, org, search, cap
|
||||
return nil, false
|
||||
}
|
||||
ids = append(ids, row.ID)
|
||||
if len(ids) > 100000 {
|
||||
candidateBytes += len(row.ID) + 16
|
||||
if len(ids) > 100000 || candidateBytes > 4<<20 {
|
||||
return nil, false
|
||||
}
|
||||
}
|
||||
@ -229,3 +237,29 @@ func queryCandidateBudget(ids []string) bool {
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// Capture is exact EqualFold, whereas free search uses ToLower+Contains.
|
||||
// Canonicalize each Unicode simple-fold cycle so sigma/final sigma and long-s
|
||||
// cannot disappear from the conservative prefilter. Extra hits are still
|
||||
// rejected by the original exact matcher.
|
||||
func queryCaptureFold(value string) string {
|
||||
var b strings.Builder
|
||||
for _, r := range value {
|
||||
smallest := r
|
||||
for next := unicode.SimpleFold(r); next != r; next = unicode.SimpleFold(next) {
|
||||
if next < smallest {
|
||||
smallest = next
|
||||
}
|
||||
}
|
||||
b.WriteRune(smallest)
|
||||
}
|
||||
return strings.ToLower(b.String())
|
||||
}
|
||||
|
||||
// Leave room for joins/projections under Mongo's command size limit. Oversized
|
||||
// optional acceleration falls through to the original query without truncation.
|
||||
func queryCandidateFilter(filter any, ids []string) (bson.M, bool) {
|
||||
joined := bson.M{"$and": bson.A{filter, bson.M{"id": bson.M{"$in": ids}}}}
|
||||
encoded, err := bson.Marshal(joined)
|
||||
return joined, err == nil && len(encoded) <= 12<<20
|
||||
}
|
||||
|
||||
@ -4,6 +4,8 @@ def compare(base, candidate):
|
||||
problems=[]
|
||||
if base.get('fixture_digest')!=candidate.get('fixture_digest') or not base.get('fixture_digest'):
|
||||
problems.append('fixture identity mismatch')
|
||||
if any(r.get('mongo_clock_samples',0)<2 or r.get('mongo_clock_max_jump_ms',float('inf'))>500 for r in (base,candidate)):
|
||||
problems.append('Mongo clock evidence missing or unstable')
|
||||
if not base.get('metrics') or not candidate.get('metrics'):
|
||||
problems.append('missing request measurements')
|
||||
if candidate.get('actors')!=81 or candidate.get('errors')!=0 or candidate.get('mongo_write_commands')!=0:
|
||||
@ -11,6 +13,8 @@ def compare(base, candidate):
|
||||
counters=candidate.get('query_serving_statistics',{})
|
||||
if not all(counters.get(k,0)>0 for k in ('report_hits','search_hits','typesense_reads','cache_hits')):
|
||||
problems.append('accelerated paths not proven active')
|
||||
if candidate.get('scheduled_seconds',0)>=180 and candidate.get('output_prepared_queries',0)<=0:
|
||||
problems.append('generation-change prewarming not proven active')
|
||||
for key,m in candidate.get('metrics',{}).items():
|
||||
limit=2000 if '.search' in key else 500 if key.endswith('.panel') or key=='admin.production' else None
|
||||
if limit and m.get('p95_ms',float('inf'))>limit:problems.append('latency exceeded: '+key)
|
||||
@ -19,7 +23,7 @@ def compare(base, candidate):
|
||||
problems.append('prewarmed first visit exceeded: '+row['Key'])
|
||||
def hashes(report):
|
||||
out={}
|
||||
for r in report.get('cold_samples',[])+report.get('samples',[]):
|
||||
for r in report.get('cold_samples',[])+report.get('samples',[])+report.get('reference_samples',[]):
|
||||
if r.get('Code')==200:out.setdefault((r['Actor'],r['Key']),set()).add(r['Hash'])
|
||||
return out
|
||||
old,new=hashes(base),hashes(candidate)
|
||||
|
||||
@ -21,7 +21,14 @@ def main():
|
||||
parser.add_argument('--warmup',type=int,default=5)
|
||||
parser.add_argument('--duration',type=int,default=30)
|
||||
parser.add_argument('--database',help='reuse this script\'s existing synthetic fixture')
|
||||
parser.add_argument('--native-mongo',action='store_true',help='use task-owned native Mongo on loopback 27019; never starts/stops another service')
|
||||
args=parser.parse_args()
|
||||
native_pid=None
|
||||
if args.native_mongo:
|
||||
if args.start:parser.error('start the native replica set and optional Docker components separately')
|
||||
native_pid=int((PRIVATE/'native/mongod.pid').read_text().strip())
|
||||
command=subprocess.run(['ps','-p',str(native_pid),'-o','command='],capture_output=True,text=True,check=True).stdout
|
||||
if 'local-data/query-serving-local/native/' not in command or '--port 27019' not in command:parser.error('task-owned native Mongo process required')
|
||||
if not 1<=args.rounds<=3 or not 1<=args.duration<=1800 or not 1<=args.warmup<=300:parser.error('bounded rounds/times required')
|
||||
if args.database and (not args.database.startswith('poll_pool_read_mix_query_') or not args.database.replace('_','').isalnum()):parser.error('synthetic database only')
|
||||
env={k:v for k,v in environment().items() if not k.startswith('POLL_POOL_')}
|
||||
@ -41,7 +48,8 @@ def main():
|
||||
POLL_POOL_TASK_PAYLOAD_KIB='16', POLL_POOL_AVAILABLE_PER_STAGE='215',
|
||||
POLL_POOL_AUTHOR_HISTORY='100', GOMAXPROCS='2')
|
||||
before=source_fingerprint(ROOT)
|
||||
report={'database':database,'inputDigest':before,'shape':'production' if args.production_shape else 'small','rounds':[],'resourceSamples':[],'passed':False,'limits':['synthetic loopback API harness, no media or production writes','process resource sampling includes builder and handler; not Kubernetes Pod limits']}
|
||||
report={'database':database,'inputDigest':before,'mongoEnvironment':'native-uncapped' if native_pid else 'docker-2cpu-4g','shape':'production' if args.production_shape else 'small','rounds':[],'resourceSamples':[],'passed':False,'limits':['synthetic loopback API harness, no media or production writes','process resource sampling includes builder and handler; not Kubernetes Pod limits']}
|
||||
if native_pid:report['limits'].append('native Mongo has fixed 0.5GiB WiredTiger cache but no container CPU/RSS cap; not comparable with earlier Docker timings')
|
||||
argv=['go','test','./internal/api','-run','^TestPollPoolReadMixLocal$','-count=1','-timeout=45m','-v']
|
||||
def run(name,updates):
|
||||
settings={**env,**updates,'POLL_POOL_OUTPUT':str(folder/(name+'.json'))}
|
||||
@ -56,13 +64,24 @@ def main():
|
||||
return result.returncode==0
|
||||
active={'phase':'prepare'}
|
||||
if not args.database:
|
||||
if not run('prepare',{'POLL_POOL_FIXTURE_ONLY':'1'}):return 1
|
||||
# Synthetic setup is not measured. Avoid a separate majority journal
|
||||
# barrier for every seed upsert; measured phases reconnect with the
|
||||
# normal URI/write concern, including background publication writes.
|
||||
setup={'POLL_POOL_FIXTURE_ONLY':'1','POLL_POOL_LOCAL_MONGO_URI':env['POLL_POOL_LOCAL_MONGO_URI']+'&w=1'}
|
||||
if not run('prepare',setup):return 1
|
||||
stopped=threading.Event()
|
||||
def resources():
|
||||
while not stopped.is_set():
|
||||
try:
|
||||
p=subprocess.run(['docker','stats','--no-stream','--format','{{json .}}','fanyu-query-local-mongo-1','fanyu-query-local-valkey-1','fanyu-query-local-typesense-1'],capture_output=True,text=True,timeout=8)
|
||||
if p.returncode==0:report['resourceSamples'].append({'phase':active['phase'],'at':dt.datetime.now(dt.timezone.utc).isoformat(),'containers':[json.loads(line) for line in p.stdout.splitlines()]})
|
||||
containers=['fanyu-query-local-valkey-1','fanyu-query-local-typesense-1']
|
||||
if not native_pid:containers.insert(0,'fanyu-query-local-mongo-1')
|
||||
p=subprocess.run(['docker','stats','--no-stream','--format','{{json .}}',*containers],capture_output=True,text=True,timeout=8)
|
||||
if p.returncode==0:
|
||||
row={'phase':active['phase'],'at':dt.datetime.now(dt.timezone.utc).isoformat(),'containers':[json.loads(line) for line in p.stdout.splitlines()]}
|
||||
if native_pid:
|
||||
process=subprocess.run(['ps','-p',str(native_pid),'-o','%cpu=,rss=,vsz=,time='],capture_output=True,text=True,timeout=2,check=True)
|
||||
row['nativeMongo']=dict(zip(('cpuPercentLifetime','rssKiB','virtualKiB','cpuTime'),process.stdout.split()))
|
||||
report['resourceSamples'].append(row)
|
||||
except (OSError,ValueError,subprocess.TimeoutExpired):pass
|
||||
(folder/'progress.json').write_text(json.dumps({'phase':active['phase'],'at':dt.datetime.now(dt.timezone.utc).isoformat(),'samples':len(report['resourceSamples'])})+'\n')
|
||||
stopped.wait(2)
|
||||
|
||||
@ -7,7 +7,7 @@ from lib.query_serving_evidence import compare
|
||||
class QueryEvidenceTests(unittest.TestCase):
|
||||
def fixture(self):
|
||||
sample={'Actor':'reviewer','Key':'acceptance.all.search','Code':200,'MS':20,'Hash':'business-id-claim-counts'}
|
||||
return {'fixture_digest':'same','actors':81,'errors':0,'mongo_write_commands':0,'metrics':{'acceptance.all.search':{'p95_ms':20}},'samples':[sample], 'cold_samples':[], 'query_serving_statistics':dict(report_hits=1,search_hits=1,typesense_reads=1,cache_hits=1)}
|
||||
return {'mongo_clock_samples':100, 'mongo_clock_max_jump_ms':1, 'fixture_digest':'same','actors':81,'errors':0,'mongo_write_commands':0,'metrics':{'acceptance.all.search':{'p95_ms':20}},'samples':[sample], 'cold_samples':[], 'query_serving_statistics':dict(report_hits=1,search_hits=1,typesense_reads=1,cache_hits=1)}
|
||||
def test_requires_real_measurements_and_active_components(self):
|
||||
a=self.fixture();b=self.fixture();self.assertTrue(compare(a,b)['passed'])
|
||||
b['query_serving_statistics']={};self.assertFalse(compare(a,b)['passed'])
|
||||
@ -26,4 +26,17 @@ class QueryEvidenceTests(unittest.TestCase):
|
||||
a=self.fixture();b=self.fixture();a['samples'][0]['Code']=500
|
||||
r=compare(a,b);self.assertFalse(r['passed']);self.assertTrue(r['baselineUnavailableComparisons'])
|
||||
|
||||
def test_unstable_clock_cannot_be_accepted(self):
|
||||
a=self.fixture();b=self.fixture();a['mongo_clock_max_jump_ms']=18000
|
||||
self.assertFalse(compare(a,b)['passed'])
|
||||
a=self.fixture();del a['mongo_clock_samples']
|
||||
self.assertFalse(compare(a,b)['passed'])
|
||||
|
||||
def test_serial_reference_closes_gap_without_changing_latency(self):
|
||||
a=self.fixture();b=self.fixture();sample=dict(a['samples'][0]);a['samples'][0]['Code']=500
|
||||
a['reference_samples']=[sample]
|
||||
self.assertTrue(compare(a,b)['passed'])
|
||||
b['metrics']['acceptance.all.search']['p95_ms']=2001
|
||||
self.assertFalse(compare(a,b)['passed'])
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user