forked from longtao.wu/droboticflow
⚡ Bound production overview queries and shard statistics history
This commit is contained in:
parent
e99febdac1
commit
6bf0b7662d
@ -31,3 +31,9 @@
|
||||
## 兼容与回退
|
||||
|
||||
新增字段均可选。共享事实 schema v3 防止旧事实缺失归属字段。回退应用可重建对应旧 schema 快照,不要求重写业务数据。原任务流转、正式决定、标注草稿和交付内容不变;仅新增验收意见误报处理记录,旧应用忽略这些记录。
|
||||
|
||||
## 2026-11-10:增长与读取实现
|
||||
|
||||
事实 schema v5 按 canonical part 分片构建和持久化,完整清单原子发布,查询流式聚合;单个 part 的全部尝试、历史关联和未完成操作屏蔽留在同一分片。比率合并分子/分母,人员天数及跨片成员去重,当前权限逐请求执行。旧 v4 应用按自身 schema 重建,业务数据不迁移。空客户端操作 ID 不再屏蔽无关历史验收。
|
||||
|
||||
容量与生产总览索引的预算、验证证据、兼容和待授权步骤见 [运行说明](../operations/production-overview-statistics-growth.md)。生产仍待端到端验收,不以本地规模测试替代线上容量结论。
|
||||
|
||||
24
docs/operations/production-overview-indexes.json
Normal file
24
docs/operations/production-overview-indexes.json
Normal file
@ -0,0 +1,24 @@
|
||||
{
|
||||
"database": "drobodata-fanyu",
|
||||
"minimumMongoVersion": "6.0",
|
||||
"indexes": [
|
||||
{
|
||||
"collection": "runs",
|
||||
"name": "production_runs_cover_v1",
|
||||
"unique": false,
|
||||
"keys": [["organizationId",1],["projectId",1],["id",1],["workflowId",1],["taskId",1],["trigger",1],["mode",1],["parentRunId",1],["createdAt",1],["endedAt",1],["status",1],["deliveryOutcome",1],["failure",1],["error",1]]
|
||||
},
|
||||
{
|
||||
"collection": "humanEvents",
|
||||
"name": "production_event_facts_v1",
|
||||
"unique": false,
|
||||
"keys": [["organizationId",1],["itemId",1]],
|
||||
"partialFilterExpression": {
|
||||
"$or": [
|
||||
{"eventType":{"$in":["approved","rejected"]}},
|
||||
{"payload.deliveryPackageId":{"$exists":true}}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
59
docs/operations/production-overview-statistics-growth.md
Normal file
59
docs/operations/production-overview-statistics-growth.md
Normal file
@ -0,0 +1,59 @@
|
||||
# 生产总览超时与统计快照容量,2026-11-10
|
||||
|
||||
任务基线与当时线上源码为 `e99febda`。本批完成本地修改、隔离数据验证和生产只读诊断;随后取得明确授权,仅新增下述两个生产索引。未部署候选应用、未补写业务数据。
|
||||
|
||||
## 根因与处理
|
||||
|
||||
生产总览有三处成本叠加。约 29,762 个 Capture ID 被装进一个 runs 的 OR 查询,查询规划约 14.7 秒,explain 结果还超过 BSON 限制;项目 runs 仍会 Fetch 含日志和节点正文的大文档,5 秒诊断预算内只返回 18,791 条;humanEvents 按 256 个 ID 拆分后,每批仍 Fetch 约 3,500 条编辑事件,约 1–1.6 秒。分批本身不能证明生产超时已解决。
|
||||
|
||||
代码改为互不重复的项目 runs、历史 Capture 关联、缺失 run ID 三段查询,关联 ID 每批至多 256;保留全量结果、组织隔离、所有尝试与旧事件交付关联。两个可选索引定义在 `production-overview-indexes.json`,不在启动迁移中创建:
|
||||
|
||||
- runs 覆盖生产统计字段,项目查询免读大文档;历史 taskId/id 查找继续使用原有索引,不能误用项目领先索引。
|
||||
- humanEvents 仅索引正式决定或含旧交付关联的事件,跳过编辑正文。读取拆成两个互斥分支:正式决定,以及非决定的旧交付关联。直接把 OR 原样交给 partial index 会扫描整个统计子集,局部回归已覆盖该陷阱。
|
||||
|
||||
索引缺失时保留原路径和超时保护,不返回部分统计、伪造零值或延长 20 秒请求预算。索引缺失时仍可能慢,不能宣称生产已修复。
|
||||
|
||||
共享统计快照原来在每次刷新中将全组织历史读入一个 64 MiB/25 万条源预算,再编码成一个 4 MiB 压缩/32 MiB 解压对象。上次精简后源体积仍为 56.8 MiB,已接近上限;继续增大固定上限不能解决增长问题。
|
||||
|
||||
新路径将投影后的原始元数据流式写入私有临时目录,以临时 bbolt 索引连接历史关联,按 canonical part 分区解释。一个 part 的多次处理不能分开统计。跨 part 验收与未完成操作的状态屏蔽保留。派生事实单片仍执行原 4/32 MiB 限制;只在所有片完成后原子发布清单。查询逐片聚合计数、时长、比率的分子分母及人员天数,不常驻全部历史,也不平均百分比。权限仍逐请求判定。
|
||||
|
||||
临时索引选择及事务约束依据 [bbolt 官方说明](https://github.com/etcd-io/bbolt):可重建数据关闭 fsync,事务分批提交,回调前结束读事务,返回数据复制后使用。文件 0600、目录 0700;正常/取消/异常均清理。崩溃遗留超过 10 分钟且没有活动锁的目录在下一次构建清理,不删除活动构建。
|
||||
|
||||
## 边界、兼容与恢复
|
||||
|
||||
事实 schema 改为 v5,旧 v4 代次不被新程序误读;回退旧应用后按旧 schema 重建,不回写业务数据。单对象旧 Reader 路径仍兼容,生产 Mongo 使用流式 Reader。跨副本租约、epoch fencing、完整代次 pin 和前一完整代次回退保留;任一分片失败不发布部分结果。
|
||||
|
||||
本批同时修复旧数据漏洞:空的客户端 operationId 不能覆盖不相关的、缺少 operationId 的历史验收事件。原先漏算的历史验收可能恢复计数,这是口径纠错,不是分片丢失等价性。
|
||||
|
||||
增长仍有明确预算,不承诺无限容量:源按约 8 MiB 目标分区,单 part 硬限仍为 64 MiB/25 万条;临时目录每次构建 2 GiB、目录/关联事务内存计量 256 MiB、压缩事实总量 256 MiB、至多 4096 片、构建 90 秒、单次查询 10 秒。容量预警分别看批峰值、单片峰值、清单、总存储、临时磁盘和构建耗时的 80%,保留全量读字节/条数供观察;失败次数和最后错误继续暴露。达到预算应排查增长来源、归档/分层或资源,不自动截断历史。
|
||||
|
||||
生产 API 当前 request 1 CPU/2 GiB、limit 2 CPU/4 GiB。本地峰值不等于生产容量验收;上线前需确认 Pod 临时磁盘可用且有至少 2 GiB 构建预算。未调整资源、副本或带宽。
|
||||
|
||||
## 证据与未完成项
|
||||
|
||||
本地隔离库每次使用唯一名称并自动删除。三轮 72,000 part、504,000 条投影记录:源 90,049,350 字节,136 片,批峰值 5,062,804 字节;独立三轮构建 6.36–6.74 秒,构建加首次查询 6.77–7.18 秒。解压事实总量约 38 MiB,已超过旧 32 MiB 单对象限制;单片解压约 2.37 MiB。所有 part 均计入。构建后 GC 堆约 1.0–1.1 MiB、临时文件 0;构建堆峰值约 63–73 MiB,进程最高 RSS 230 MiB,临时磁盘峰值约 370 MiB。独立三轮 CPU 为 8.54/8.98/8.75 秒,无持续增长;早一组与其他门禁并发的样本单独保留,不用于容量承诺。
|
||||
|
||||
600 片段带编辑正文的索引回归:runs 扫描文档 0;两任务事件查询仅扫描 3 条,保留非决定旧交付关联;有无索引完整结果等价。24,000 片段、约 50 万事件的大载荷对照保留全部 24,001 条有效事件,五次完整读取为 364–530 毫秒,首次原路径 1.066 秒、索引路径 530 毫秒。该读耗时不包含真实浏览器与生产网络。
|
||||
|
||||
诊断和原始验证日志位于忽略目录 `local-data/diagnostics/statistics-growth/`。生产只读查询证明根因和现有 Mongo 8.0 版本兼容,并验证下述索引扫描收益。候选应用的生产端到端耗时、并发、CPU/内存及长期索引维护开销仍待发布后的真实对照;不以本地快、Pod 就绪或压缩体积小作为线上验收。
|
||||
|
||||
## 单独授权的生产索引计划
|
||||
|
||||
1. 核对泛宇 namespace、数据库与 Mongo 版本;盘点两集合记录数、当前负载和可用空间。逐项比对名称、字段顺序及 partial 条件;同名不同定义即停止,不替换已有索引。
|
||||
2. 按清单一次只创建一个非唯一索引,先 runs,再 humanEvents。构建扫描全集合并消耗 IO/CPU/磁盘;runs 后续状态/错误更新增加索引维护,humanEvents 编辑事件被 partial 排除。实际大小和构建时间尚未在线量化,观察负载,不同时扩容或运行压测。超时先查实际完成状态,不能盲目重试。
|
||||
3. 只读执行完整项目查询、256-ID 两分支查询及 explain,检查 runs documentsExamined=0、事件扫描只覆盖所需记录,核验完整数量与旧路径等价。API 发布需另按 release.py 审批,实际页面完整请求必须在 20 秒内且按性能目标验收,冷/热均测。
|
||||
4. 无收益或写延迟恶化时,在授权范围内仅删除本次两个索引回退;先回退代码也安全,缺索引走兼容路径。不得删除原索引、修改客户记录或回滚客户状态。
|
||||
|
||||
## 已授权执行与当前线上状态
|
||||
|
||||
用户授权引用 `call_ac06f16e8fe8413ba93f38d7dd2e401a`,仅允许新增清单中的两个索引及只读对照。已核对 namespace `droboticflow-fanyu`,UID `d6a90ea7-b57d-4c52-b90b-b6c976cba076`,现用 API 摘要 `51395a98…`/源码 `e99febda`。
|
||||
|
||||
2026-11-11 UTC 依次创建两索引(本地日期 11 月 10 日)。runs 创建成功后的核对脚本复用了上一索引的属性,首次核对误报;随即停止后续创建、只读核实实际完整定义,再修正脚本。未重复创建 runs。humanEvents 创建用时 17.979 秒,精确定义核对完成。日志保留首次错误与后续核对,不把错误记录擦除。
|
||||
|
||||
新索引实际占用:runs 28,639,232 字节、humanEvents 20,480 字节。只读 explain:runs 项目完整覆盖查询返回 26,489 行,文档读取 0,数据库执行约 2–3 秒;代表性事件分支每批扫描 0–3 条,数据库执行 7–13 毫秒,原路径约 1–1.6 秒。操作后单点采样数据库读/写等待队列均为 0,不代表长期维护成本已验收。其他原索引及业务记录未改动。
|
||||
|
||||
现用旧应用复测仍为两次 HTTP 500,约 20.04/20.03 秒;尚未部署本批分批查询代码,不能宣称整页恢复。其余 humanItems/annotationTasks/worklogs 仍有秒级全量 Fetch 成本,后续发布后必须继续分段观察;本批不扩建未授权索引。一次通过 kubectl 控制链路运行完整候选 loader 的尝试受到传输与诊断连接 deadline 实现限制,未完成读取,不能用其约 49 秒结果判断生产候选 API 耗时;原始失败日志与限制说明保留。
|
||||
|
||||
本地 API/CSV 权限、浏览器看板、快照恢复、race、真实 Mongo、增长和官方前后端基线门禁已执行。官方 Go 门禁为 68 个已有基线失败、无新增失败。`verify:task` 的 26 项中 25 项通过;自动 store 组因未开启的可选测试标为 incomplete,所需本地统计专项随后在显式隔离环境执行通过。受保护的另一任务生产探针与生产端到端/持续容量仍未验收;不把总报告改写为全部通过。新增门禁及索引清单已补充验证映射,清单与本地实际索引字段/partial 条件由测试比对。
|
||||
|
||||
上述索引已完成,推送及应用部署仍为独立授权。索引语义参考 [MongoDB 官方 partial index 说明](https://www.mongodb.com/docs/v8.0/core/index-partial/)。
|
||||
1
go.mod
1
go.mod
@ -6,6 +6,7 @@ require (
|
||||
github.com/aliyun/aliyun-oss-go-sdk v3.0.2+incompatible
|
||||
github.com/baidubce/bce-sdk-go v0.9.264
|
||||
github.com/go-chi/chi/v5 v5.2.3
|
||||
go.etcd.io/bbolt v1.4.3
|
||||
go.mongodb.org/mongo-driver v1.17.6
|
||||
golang.org/x/crypto v0.36.0
|
||||
golang.org/x/sync v0.12.0
|
||||
|
||||
2
go.sum
2
go.sum
@ -108,6 +108,8 @@ github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78/go.mod h1:aL8wCCfTfS
|
||||
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
|
||||
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
|
||||
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
|
||||
go.etcd.io/bbolt v1.4.3 h1:dEadXpI6G79deX5prL3QRNP6JB8UxVkqo4UPnHaNXJo=
|
||||
go.etcd.io/bbolt v1.4.3/go.mod h1:tKQlpPaYCVFctUIgFKFnAlvbmB3tpy1vkTnDWohtc0E=
|
||||
go.mongodb.org/mongo-driver v1.17.6 h1:87JUG1wZfWsr6rIz3ZmpH90rL5tea7O3IHuSwHUpsss=
|
||||
go.mongodb.org/mongo-driver v1.17.6/go.mod h1:Hy04i7O2kC4RS06ZrhPRqj/u4DTYkFDAAccj+rVKqgQ=
|
||||
go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
|
||||
|
||||
@ -168,7 +168,7 @@ func (s *server) handleAnnotationOutput(w http.ResponseWriter, r *http.Request)
|
||||
}
|
||||
// Do not narrow the load to a requested person: a review may address sources
|
||||
// attributed to several authors. The read model applies the principal fence.
|
||||
s.outputOnce.Do(func() { s.outputReader = outputsnapshot.NewReader(s.store, s.store.LoadAnnotationOutput) })
|
||||
s.outputOnce.Do(func() { s.outputReader = s.newOutputSnapshotReader() })
|
||||
report, info, err := s.outputReader.Query(ctx, org, ids, r.URL.Query().Get("snapshotId"), q)
|
||||
// Older deployed clients cannot consume 202. They wait for the same shared
|
||||
// background generation; they never start a private synchronous rebuild.
|
||||
@ -314,7 +314,7 @@ func (s *server) handleAnnotationOutputHealth(w http.ResponseWriter, r *http.Req
|
||||
renderError(w, 503, err)
|
||||
return
|
||||
}
|
||||
s.outputOnce.Do(func() { s.outputReader = outputsnapshot.NewReader(s.store, s.store.LoadAnnotationOutput) })
|
||||
s.outputOnce.Do(func() { s.outputReader = s.newOutputSnapshotReader() })
|
||||
health, err := s.outputReader.Inspect(ctx, org, ids)
|
||||
if err != nil {
|
||||
renderError(w, 503, err)
|
||||
@ -322,3 +322,14 @@ func (s *server) handleAnnotationOutputHealth(w http.ResponseWriter, r *http.Req
|
||||
}
|
||||
renderJSON(w, 200, health)
|
||||
}
|
||||
|
||||
func (s *server) newOutputSnapshotReader() *outputsnapshot.Reader {
|
||||
if source, ok := any(s.store).(interface {
|
||||
StreamAnnotationOutput(context.Context, string, []string, func(workforcestats.OutputSnapshot) error) (*workforcestats.OutputSourceStats, error)
|
||||
}); ok {
|
||||
if _, ok := any(s.store).(outputsnapshot.ShardedBackend); ok {
|
||||
return outputsnapshot.NewStreamReader(s.store, source.StreamAnnotationOutput)
|
||||
}
|
||||
}
|
||||
return outputsnapshot.NewReader(s.store, s.store.LoadAnnotationOutput)
|
||||
}
|
||||
|
||||
@ -105,7 +105,7 @@ func (s *server) handleProductionQuality(w http.ResponseWriter, r *http.Request)
|
||||
if vendor && p.User.SupplierID == "" {
|
||||
ids = []string{}
|
||||
}
|
||||
s.outputOnce.Do(func() { s.outputReader = outputsnapshot.NewReader(s.store, s.store.LoadAnnotationOutput) })
|
||||
s.outputOnce.Do(func() { s.outputReader = s.newOutputSnapshotReader() })
|
||||
report, info, err := s.outputReader.Query(ctx, org, ids, r.URL.Query().Get("snapshotId"), q)
|
||||
if errors.Is(err, outputsnapshot.ErrExpired) {
|
||||
renderJSON(w, 409, map[string]string{"code": "snapshot_expired", "error": "数据已更新,请重试"})
|
||||
|
||||
@ -60,13 +60,30 @@ func snapshotHealth(state State, key string, now time.Time) Health {
|
||||
out.Warnings = append(out.Warnings, code)
|
||||
}
|
||||
}
|
||||
warn(g.Bytes >= MaxEncodedBytes*4/5, "encoded_capacity")
|
||||
if g.Shards == 0 {
|
||||
warn(g.Bytes >= MaxEncodedBytes*4/5, "encoded_capacity")
|
||||
} else {
|
||||
warn(g.Bytes >= MaxGenerationBytes*4/5, "generation_capacity")
|
||||
warn(g.Shards >= MaxShards*4/5, "shard_count_capacity")
|
||||
warn(g.ManifestBytes >= MaxEncodedBytes*4/5, "manifest_capacity")
|
||||
}
|
||||
if m := g.Metrics; m != nil {
|
||||
warn(m.DurationMS >= BuildTimeout.Milliseconds()*4/5, "build_time_budget")
|
||||
warn(m.DecodedBytes >= MaxDecodedBytes*4/5, "decoded_capacity")
|
||||
decoded := m.DecodedBytes
|
||||
if g.Shards > 0 {
|
||||
decoded = m.PeakDecodedBytes
|
||||
warn(m.PeakEncodedBytes >= MaxEncodedBytes*4/5, "encoded_capacity")
|
||||
}
|
||||
warn(decoded >= MaxDecodedBytes*4/5, "decoded_capacity")
|
||||
if m.Source != nil {
|
||||
warn(m.Source.Bytes >= workforcestats.OutputSourceMaxBytes*4/5, "source_bytes_capacity")
|
||||
warn(m.Source.Documents >= workforcestats.OutputSourceMaxDocuments*4/5, "source_documents_capacity")
|
||||
bytes, documents := m.Source.Bytes, m.Source.Documents
|
||||
if g.Shards > 0 {
|
||||
bytes, documents = m.Source.PeakBatchBytes, m.Source.PeakBatchDocuments
|
||||
warn(m.Source.DirectoryBytes >= workforcestats.OutputDirectoryMaxBytes*4/5, "directory_capacity")
|
||||
warn(m.Source.PeakSpoolBytes >= workforcestats.OutputSpoolMaxBytes*4/5, "temporary_disk_capacity")
|
||||
}
|
||||
warn(bytes >= workforcestats.OutputSourceMaxBytes*4/5, "source_bytes_capacity")
|
||||
warn(documents >= workforcestats.OutputSourceMaxDocuments*4/5, "source_documents_capacity")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -10,9 +10,10 @@ import (
|
||||
// MemoryBackend is the in-memory Adapter used by API fixtures. Its coordinator
|
||||
// and bounded slots have the same ownership/publication rules as Mongo.
|
||||
type MemoryBackend struct {
|
||||
mu sync.Mutex
|
||||
state State
|
||||
slots [SlotCount]memoryBlob
|
||||
mu sync.Mutex
|
||||
state State
|
||||
slots [SlotCount]memoryBlob
|
||||
shards map[string][]byte
|
||||
}
|
||||
type memoryBlob struct {
|
||||
epoch int64
|
||||
@ -67,12 +68,20 @@ func (b *MemoryBackend) PublishOutputSnapshot(_ context.Context, l Lease, g Gene
|
||||
if !b.owns(l, now) {
|
||||
return false, nil
|
||||
}
|
||||
if l.Slot < 0 || l.Slot >= SlotCount || g.Slot != l.Slot || g.Epoch != l.Epoch || g.Bytes != len(data) || len(data) > MaxEncodedBytes {
|
||||
if !ValidPublication(l, g, data) {
|
||||
return false, ErrCapacity
|
||||
}
|
||||
b.slots[l.Slot] = memoryBlob{l.Epoch, g.ID, append([]byte(nil), data...)}
|
||||
b.state.Heads[l.Key] = Head{Current: &g, Previous: l.Previous, NextAttempt: g.AsOf.Add(RefreshInterval)}
|
||||
b.state.Owner = ""
|
||||
for key := range b.shards {
|
||||
var slot int
|
||||
var epoch int64
|
||||
var index int
|
||||
if _, err := fmt.Sscanf(key, "%d/%d/%d", &slot, &epoch, &index); err == nil && slot == l.Slot && epoch < l.Epoch {
|
||||
delete(b.shards, key)
|
||||
}
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
func (b *MemoryBackend) FailOutputSnapshot(_ context.Context, l Lease, now time.Time, failure Failure) error {
|
||||
@ -98,3 +107,45 @@ func (b *MemoryBackend) OutputSnapshotBlob(_ context.Context, g Generation) ([]b
|
||||
}
|
||||
return append([]byte(nil), v.data...), nil
|
||||
}
|
||||
|
||||
func shardKey(slot int, epoch int64, index int) string {
|
||||
return fmt.Sprintf("%d/%d/%d", slot, epoch, index)
|
||||
}
|
||||
func (b *MemoryBackend) StageOutputSnapshotShard(_ context.Context, l Lease, index int, data []byte, now time.Time) (bool, error) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
if !b.owns(l, now) {
|
||||
return false, nil
|
||||
}
|
||||
if l.Slot < 0 || l.Slot >= SlotCount || index < 0 || index >= MaxShards || len(data) > MaxEncodedBytes {
|
||||
return false, ErrCapacity
|
||||
}
|
||||
if b.shards == nil {
|
||||
b.shards = map[string][]byte{}
|
||||
}
|
||||
key := shardKey(l.Slot, l.Epoch, index)
|
||||
if _, exists := b.shards[key]; exists {
|
||||
return false, ErrUnavailable
|
||||
}
|
||||
if index == 0 {
|
||||
for existing := range b.shards {
|
||||
var slot int
|
||||
var epoch int64
|
||||
var index int
|
||||
if _, err := fmt.Sscanf(existing, "%d/%d/%d", &slot, &epoch, &index); err == nil && slot == l.Slot && epoch < l.Epoch {
|
||||
delete(b.shards, existing)
|
||||
}
|
||||
}
|
||||
}
|
||||
b.shards[key] = append([]byte(nil), data...)
|
||||
return true, nil
|
||||
}
|
||||
func (b *MemoryBackend) OutputSnapshotShard(_ context.Context, g Generation, index int) ([]byte, error) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
data, ok := b.shards[shardKey(g.Slot, g.Epoch, index)]
|
||||
if !ok {
|
||||
return nil, ErrExpired
|
||||
}
|
||||
return append([]byte(nil), data...), nil
|
||||
}
|
||||
|
||||
@ -25,6 +25,7 @@ type Reader struct {
|
||||
querySlots chan struct{}
|
||||
backend Backend
|
||||
source Source
|
||||
stream StreamSource
|
||||
now func() time.Time
|
||||
ctx context.Context
|
||||
cancel context.CancelFunc
|
||||
@ -117,6 +118,9 @@ func (r *Reader) Read(ctx context.Context, org string, projects []string, pin st
|
||||
}
|
||||
func (r *Reader) readResolved(ctx context.Context, gen *Generation, info Info, pin string) (*workforcestats.OutputDataset, Info, error) {
|
||||
now := r.now()
|
||||
if gen.Shards > 0 {
|
||||
return nil, info, ErrCapacity
|
||||
} // Query streams shards; Read retains its bounded single-dataset contract.
|
||||
|
||||
select {
|
||||
case r.reads <- struct{}{}:
|
||||
@ -223,6 +227,18 @@ func (r *Reader) build(key, org string, projects []string) {
|
||||
}()
|
||||
defer func() { close(stop); <-done }()
|
||||
asOf := r.now().UTC().Truncate(time.Millisecond)
|
||||
if r.stream != nil {
|
||||
stage = "partitioned_build"
|
||||
gen, payload, err := r.buildShards(ctx, lease, org, projects, asOf, started)
|
||||
if err != nil {
|
||||
buildErr = err
|
||||
return
|
||||
}
|
||||
stage = "publish"
|
||||
published, buildErr = r.backend.PublishOutputSnapshot(ctx, lease, gen, payload, r.now())
|
||||
return
|
||||
}
|
||||
|
||||
raw, err := r.source(ctx, org, projects)
|
||||
if err != nil || ctx.Err() != nil {
|
||||
buildErr = err
|
||||
@ -369,16 +385,15 @@ func (r *Reader) Query(ctx context.Context, org string, projects []string, pin s
|
||||
if report := r.memoReport(key); report != nil {
|
||||
return answer{report, info}, nil
|
||||
}
|
||||
data, actual, err := r.readResolved(buildCtx, gen, info, pin)
|
||||
if err != nil || data == nil {
|
||||
report, actual, err := r.queryResolved(buildCtx, gen, info, pin, q)
|
||||
if err != nil || report == nil {
|
||||
return nil, err
|
||||
}
|
||||
report := workforcestats.QueryOutputDataset(*data, q)
|
||||
// Corrupt-current fallback must not poison the current generation's key.
|
||||
if actual.ID == gen.ID {
|
||||
r.rememberReport(key, &report)
|
||||
r.rememberReport(key, report)
|
||||
}
|
||||
return answer{&report, actual}, nil
|
||||
return answer{report, actual}, nil
|
||||
})
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
|
||||
205
internal/outputsnapshot/shards.go
Normal file
205
internal/outputsnapshot/shards.go
Normal file
@ -0,0 +1,205 @@
|
||||
package outputsnapshot
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"time"
|
||||
|
||||
"droboticflow/internal/workforcestats"
|
||||
)
|
||||
|
||||
type shardDescriptor struct {
|
||||
Bytes int `json:"b"`
|
||||
SHA256 string `json:"h"`
|
||||
}
|
||||
type snapshotManifest struct {
|
||||
Shards []shardDescriptor `json:"shards"`
|
||||
}
|
||||
|
||||
const MaxGenerationBytes = 256 << 20
|
||||
|
||||
func NewStreamReader(b Backend, source StreamSource) *Reader {
|
||||
r := NewReader(b, nil)
|
||||
r.stream = source
|
||||
return r
|
||||
}
|
||||
func digest(data []byte) string { h := sha256.Sum256(data); return hex.EncodeToString(h[:]) }
|
||||
|
||||
func (r *Reader) buildShards(ctx context.Context, lease Lease, org string, projects []string, asOf time.Time, started time.Time) (Generation, []byte, error) {
|
||||
backend, ok := r.backend.(ShardedBackend)
|
||||
if !ok {
|
||||
return Generation{}, nil, ErrUnavailable
|
||||
}
|
||||
manifest := snapshotManifest{Shards: []shardDescriptor{}}
|
||||
metrics := &BuildMetrics{}
|
||||
total := 0
|
||||
yield := func(raw workforcestats.OutputSnapshot) error {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
data := workforcestats.BuildOutputDataset(raw, asOf)
|
||||
encoded, decoded, err := encode(data)
|
||||
if errors.Is(err, ErrCapacity) {
|
||||
return workforcestats.ErrOutputPartitionSplit
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(manifest.Shards) >= MaxShards || total+len(encoded) > MaxGenerationBytes {
|
||||
return ErrCapacity
|
||||
}
|
||||
if err = ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
ok, err := backend.StageOutputSnapshotShard(ctx, lease, len(manifest.Shards), encoded, r.now())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !ok {
|
||||
return fmt.Errorf("snapshot shard lease lost")
|
||||
}
|
||||
manifest.Shards = append(manifest.Shards, shardDescriptor{len(encoded), digest(encoded)})
|
||||
total += len(encoded)
|
||||
metrics.DecodedBytes += decoded
|
||||
metrics.Parts += len(data.Parts)
|
||||
metrics.PeakEncodedBytes = max(metrics.PeakEncodedBytes, len(encoded))
|
||||
metrics.PeakDecodedBytes = max(metrics.PeakDecodedBytes, decoded)
|
||||
return nil
|
||||
}
|
||||
source, err := r.stream(ctx, org, projects, yield)
|
||||
if err != nil {
|
||||
return Generation{}, nil, err
|
||||
}
|
||||
if len(manifest.Shards) == 0 {
|
||||
if err = yield(workforcestats.OutputSnapshot{}); err != nil {
|
||||
return Generation{}, nil, err
|
||||
}
|
||||
}
|
||||
payload, err := json.Marshal(manifest)
|
||||
if err != nil {
|
||||
return Generation{}, nil, err
|
||||
}
|
||||
if len(payload) > MaxEncodedBytes {
|
||||
return Generation{}, nil, ErrCapacity
|
||||
}
|
||||
metrics.Source = source
|
||||
metrics.DurationMS = time.Since(started).Milliseconds()
|
||||
g := Generation{ID: fmt.Sprintf("of1-%s-%d", lease.Owner, lease.Epoch), Slot: lease.Slot, Epoch: lease.Epoch, AsOf: asOf, SHA256: digest(payload), Bytes: total, Shards: len(manifest.Shards), ManifestBytes: len(payload), Metrics: metrics}
|
||||
return g, payload, nil
|
||||
}
|
||||
func decodeShard(payload []byte, g Generation) (*workforcestats.OutputDataset, error) {
|
||||
z, err := gzip.NewReader(bytes.NewReader(payload))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer z.Close()
|
||||
raw, err := io.ReadAll(io.LimitReader(z, MaxDecodedBytes+1))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(raw) > MaxDecodedBytes {
|
||||
return nil, ErrCapacity
|
||||
}
|
||||
var d workforcestats.OutputDataset
|
||||
if err = json.Unmarshal(raw, &d); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !d.AsOf.Equal(g.AsOf) {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
return &d, nil
|
||||
}
|
||||
func (r *Reader) queryGeneration(ctx context.Context, g Generation, q workforcestats.OutputQuery) (*workforcestats.OutputReport, error) {
|
||||
q.Now = g.AsOf
|
||||
if g.Shards == 0 {
|
||||
d, err := r.load(ctx, g)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
report := workforcestats.QueryOutputDataset(*d, q)
|
||||
return &report, ctx.Err()
|
||||
}
|
||||
backend, ok := r.backend.(ShardedBackend)
|
||||
if !ok {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
if g.Shards < 1 || g.Shards > MaxShards || g.Bytes > MaxGenerationBytes {
|
||||
return nil, ErrCapacity
|
||||
}
|
||||
payload, err := r.backend.OutputSnapshotBlob(ctx, g)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(payload) != g.ManifestBytes || len(payload) > MaxEncodedBytes || digest(payload) != g.SHA256 {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
var manifest snapshotManifest
|
||||
if err = json.Unmarshal(payload, &manifest); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(manifest.Shards) != g.Shards {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
total := 0
|
||||
for _, d := range manifest.Shards {
|
||||
if d.Bytes < 1 || d.Bytes > MaxEncodedBytes {
|
||||
return nil, ErrCapacity
|
||||
}
|
||||
total += d.Bytes
|
||||
}
|
||||
if total != g.Bytes {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
reducer := workforcestats.NewOutputReducer(q)
|
||||
for i, descriptor := range manifest.Shards {
|
||||
if err = ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
encoded, err := backend.OutputSnapshotShard(ctx, g, i)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(encoded) != descriptor.Bytes || digest(encoded) != descriptor.SHA256 {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
d, err := decodeShard(encoded, g)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
reducer.Add(*d)
|
||||
}
|
||||
report := reducer.Report()
|
||||
return &report, ctx.Err()
|
||||
}
|
||||
func (r *Reader) queryResolved(ctx context.Context, g *Generation, info Info, pin string, q workforcestats.OutputQuery) (*workforcestats.OutputReport, Info, error) {
|
||||
report, err := r.queryGeneration(ctx, *g, q)
|
||||
if err != nil && ctx.Err() == nil && pin == "" {
|
||||
state, stateErr := r.backend.OutputSnapshotState(ctx)
|
||||
if stateErr == nil {
|
||||
for _, head := range state.Heads {
|
||||
if head.Current != nil && head.Current.ID == g.ID && head.Previous != nil && r.now().Sub(head.Previous.AsOf) <= MaxAge {
|
||||
g = head.Previous
|
||||
report, err = r.queryGeneration(ctx, *g, q)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil, info, ctx.Err()
|
||||
}
|
||||
return nil, info, ErrUnavailable
|
||||
}
|
||||
info.ID = g.ID
|
||||
info.AsOf = g.AsOf
|
||||
info.Stale = r.now().Sub(g.AsOf) >= RefreshInterval
|
||||
return report, info, nil
|
||||
}
|
||||
114
internal/outputsnapshot/shards_test.go
Normal file
114
internal/outputsnapshot/shards_test.go
Normal file
@ -0,0 +1,114 @@
|
||||
package outputsnapshot
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"droboticflow/internal/workforcestats"
|
||||
)
|
||||
|
||||
func TestShardedGenerationMatchesLegacyAndSurvivesCorruptionAndRestart(t *testing.T) {
|
||||
now := time.Now().UTC().Truncate(time.Millisecond)
|
||||
var clock atomic.Int64
|
||||
clock.Store(now.UnixMilli())
|
||||
nowFn := func() time.Time { return time.UnixMilli(clock.Load()).UTC() }
|
||||
b := &MemoryBackend{}
|
||||
raw := testData(now)
|
||||
source := func(ctx context.Context, _ string, _ []string, yield func(workforcestats.OutputSnapshot) error) (*workforcestats.OutputSourceStats, error) {
|
||||
for i := range raw.Tasks {
|
||||
d := raw
|
||||
d.Tasks = raw.Tasks[i : i+1]
|
||||
d.Events = raw.Events[i : i+1]
|
||||
if err := yield(d); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
return &workforcestats.OutputSourceStats{Bytes: 90 << 20, Documents: 300000, Partitions: 12, PeakBatchBytes: 8 << 20, PeakBatchDocuments: 25000}, nil
|
||||
}
|
||||
r := NewStreamReader(b, source)
|
||||
defer r.Close()
|
||||
r.now = nowFn
|
||||
q := workforcestats.OutputQuery{From: now.Add(-24 * time.Hour), To: now.Add(time.Hour)}
|
||||
var first Info
|
||||
eventually(t, func() bool {
|
||||
report, info, err := r.Query(t.Context(), "org", []string{"p"}, "", q)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if report == nil {
|
||||
return false
|
||||
}
|
||||
want := workforcestats.QueryOutputDataset(workforcestats.BuildOutputDataset(raw, now), q)
|
||||
if !reflect.DeepEqual(*report, want) {
|
||||
t.Fatal("report differs")
|
||||
}
|
||||
first = info
|
||||
return true
|
||||
})
|
||||
h, err := r.Inspect(t.Context(), "org", []string{"p"})
|
||||
if err != nil || len(h.Warnings) != 0 {
|
||||
t.Fatalf("partition capacity warns on total history: %+v %v", h, err)
|
||||
}
|
||||
now = now.Add(RefreshInterval + time.Second)
|
||||
clock.Store(now.UnixMilli())
|
||||
eventually(t, func() bool {
|
||||
_, info, err := r.Query(t.Context(), "org", []string{"p"}, "", q)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return info.ID != "" && info.ID != first.ID
|
||||
})
|
||||
b.mu.Lock()
|
||||
head := b.state.Heads[scopeKey("org", []string{"p"})]
|
||||
b.shards[shardKey(head.Current.Slot, head.Current.Epoch, 3)] = []byte("broken")
|
||||
b.mu.Unlock()
|
||||
// New process has no memoized report. A corrupt shard must discard the entire
|
||||
// partial reduction and use the previous complete generation.
|
||||
restarted := NewStreamReader(b, source)
|
||||
defer restarted.Close()
|
||||
restarted.now = nowFn
|
||||
report, info, err := restarted.Query(t.Context(), "org", []string{"p"}, "", q)
|
||||
if err != nil || report == nil || info.ID != first.ID {
|
||||
t.Fatalf("previous fallback: %v %+v", err, info)
|
||||
}
|
||||
if report.Totals.Submitted.Count != 12 {
|
||||
t.Fatal("partial generation contaminated fallback")
|
||||
}
|
||||
if _, _, err = restarted.Query(t.Context(), "org", []string{"p"}, head.Current.ID, q); !errors.Is(err, ErrUnavailable) {
|
||||
t.Fatal("corrupt pin silently changed", err)
|
||||
}
|
||||
}
|
||||
func TestShardedBuildFailureNeverPublishesPartialManifest(t *testing.T) {
|
||||
now := time.Now().UTC().Truncate(time.Millisecond)
|
||||
b := &MemoryBackend{}
|
||||
r := NewStreamReader(b, func(ctx context.Context, _ string, _ []string, yield func(workforcestats.OutputSnapshot) error) (*workforcestats.OutputSourceStats, error) {
|
||||
if err := yield(testData(now)); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return nil, errors.New("source stopped")
|
||||
})
|
||||
defer r.Close()
|
||||
r.now = func() time.Time { return now }
|
||||
r.build(scopeKey("org", []string{"p"}), "org", []string{"p"})
|
||||
state, _ := b.OutputSnapshotState(t.Context())
|
||||
h := state.Heads[scopeKey("org", []string{"p"})]
|
||||
if h.Current != nil || h.ConsecutiveFailures != 1 {
|
||||
t.Fatalf("partial generation visible: %+v", h)
|
||||
}
|
||||
lease, ok, err := b.AcquireOutputSnapshot(t.Context(), scopeKey("org", []string{"p"}), now.Add(RetryInterval+time.Second))
|
||||
if err != nil || !ok {
|
||||
t.Fatal("retry unavailable")
|
||||
}
|
||||
if ok, err = b.StageOutputSnapshotShard(t.Context(), lease, 0, []byte("next"), now.Add(RetryInterval+time.Second)); err != nil || !ok {
|
||||
t.Fatal("retry stage failed")
|
||||
}
|
||||
stale := lease
|
||||
stale.Epoch--
|
||||
if ok, err = b.StageOutputSnapshotShard(t.Context(), stale, 1, []byte("stale"), now.Add(RetryInterval+time.Second)); err != nil || ok {
|
||||
t.Fatal("expired writer staged")
|
||||
}
|
||||
}
|
||||
@ -10,7 +10,7 @@ import (
|
||||
)
|
||||
|
||||
const (
|
||||
Schema = "annotation-output-facts/v4"
|
||||
Schema = "annotation-output-facts/v5"
|
||||
RefreshInterval = 120 * time.Second
|
||||
RetryInterval = 120 * time.Second
|
||||
MaxAge = 15 * time.Minute
|
||||
@ -30,10 +30,12 @@ var (
|
||||
|
||||
// DurationMS covers source reads, calculation and encoding, before publication.
|
||||
type BuildMetrics struct {
|
||||
DurationMS int64 `bson:"durationMs" json:"durationMs"`
|
||||
DecodedBytes int `bson:"decodedBytes" json:"decodedBytes"`
|
||||
Parts int `bson:"parts" json:"parts"`
|
||||
Source *workforcestats.OutputSourceStats `bson:"source,omitempty" json:"source,omitempty"`
|
||||
PeakEncodedBytes int `bson:"peakEncodedBytes,omitempty" json:"peakEncodedBytes,omitempty"`
|
||||
PeakDecodedBytes int `bson:"peakDecodedBytes,omitempty" json:"peakDecodedBytes,omitempty"`
|
||||
DurationMS int64 `bson:"durationMs" json:"durationMs"`
|
||||
DecodedBytes int `bson:"decodedBytes" json:"decodedBytes"`
|
||||
Parts int `bson:"parts" json:"parts"`
|
||||
Source *workforcestats.OutputSourceStats `bson:"source,omitempty" json:"source,omitempty"`
|
||||
}
|
||||
|
||||
// Failure stores bounded machine-readable categories; raw errors stay in logs.
|
||||
@ -44,13 +46,15 @@ type Failure struct {
|
||||
DurationMS int64 `bson:"durationMs" json:"durationMs"`
|
||||
}
|
||||
type Generation struct {
|
||||
Metrics *BuildMetrics `bson:"metrics,omitempty" json:"-"`
|
||||
ID string `bson:"id" json:"id"`
|
||||
Slot int `bson:"slot" json:"-"`
|
||||
Epoch int64 `bson:"epoch" json:"-"`
|
||||
AsOf time.Time `bson:"asOf" json:"asOf"`
|
||||
SHA256 string `bson:"sha256" json:"-"`
|
||||
Bytes int `bson:"bytes" json:"-"`
|
||||
Shards int `bson:"shards,omitempty" json:"-"`
|
||||
ManifestBytes int `bson:"manifestBytes,omitempty" json:"-"`
|
||||
Metrics *BuildMetrics `bson:"metrics,omitempty" json:"-"`
|
||||
ID string `bson:"id" json:"id"`
|
||||
Slot int `bson:"slot" json:"-"`
|
||||
Epoch int64 `bson:"epoch" json:"-"`
|
||||
AsOf time.Time `bson:"asOf" json:"asOf"`
|
||||
SHA256 string `bson:"sha256" json:"-"`
|
||||
Bytes int `bson:"bytes" json:"-"`
|
||||
}
|
||||
type Head struct {
|
||||
ConsecutiveFailures int64 `bson:"consecutiveFailures,omitempty"`
|
||||
@ -143,3 +147,25 @@ const StateCollection = "annotationOutputSnapshotState"
|
||||
const SlotsCollection = "annotationOutputSnapshotSlots"
|
||||
|
||||
func IsPrivateCollection(name string) bool { return name == StateCollection || name == SlotsCollection }
|
||||
|
||||
// ShardedBackend stages immutable, epoch-addressed blobs. Only publication of
|
||||
// the manifest makes a complete generation visible to readers.
|
||||
type ShardedBackend interface {
|
||||
StageOutputSnapshotShard(context.Context, Lease, int, []byte, time.Time) (bool, error)
|
||||
OutputSnapshotShard(context.Context, Generation, int) ([]byte, error)
|
||||
}
|
||||
type StreamSource func(context.Context, string, []string, func(workforcestats.OutputSnapshot) error) (*workforcestats.OutputSourceStats, error)
|
||||
|
||||
func ValidPublication(l Lease, g Generation, data []byte) bool {
|
||||
if l.Slot < 0 || l.Slot >= SlotCount || g.Slot != l.Slot || g.Epoch != l.Epoch || len(data) > MaxEncodedBytes {
|
||||
return false
|
||||
}
|
||||
if g.Shards > 0 {
|
||||
return g.ManifestBytes == len(data) && g.Shards <= MaxShards && g.Bytes >= 0
|
||||
}
|
||||
return g.Shards == 0 && g.Bytes == len(data)
|
||||
}
|
||||
|
||||
// Bound manifest size and total per-generation derived storage independently of
|
||||
// historical source size. Oversized single parts remain explicit failures.
|
||||
const MaxShards = 4096
|
||||
|
||||
@ -132,6 +132,9 @@ func (s *MongoStore) LoadAnnotationOutput(ctx context.Context, org string, proje
|
||||
return s.loadAnnotationOutput(ctx, org, projects, outputSourceBudget{workforcestats.OutputSourceMaxBytes, workforcestats.OutputSourceMaxDocuments})
|
||||
}
|
||||
func (s *MongoStore) loadAnnotationOutput(ctx context.Context, org string, projects []string, budget outputSourceBudget) (workforcestats.OutputSnapshot, error) {
|
||||
return s.loadAnnotationOutputTo(ctx, org, projects, budget, nil)
|
||||
}
|
||||
func (s *MongoStore) loadAnnotationOutputTo(ctx context.Context, org string, projects []string, budget outputSourceBudget, spool *outputSourceSpool) (workforcestats.OutputSnapshot, error) {
|
||||
d := workforcestats.OutputSnapshot{}
|
||||
if len(projects) == 0 {
|
||||
return d, nil
|
||||
@ -181,11 +184,17 @@ func (s *MongoStore) loadAnnotationOutput(ctx context.Context, org string, proje
|
||||
totalBytes += len(cursor.Current)
|
||||
totalDocs++
|
||||
bytes, documents := totalBytes, totalDocs
|
||||
exceeded := bytes > budget.bytes || documents > budget.documents
|
||||
exceeded := spool == nil && (bytes > budget.bytes || documents > budget.documents)
|
||||
budgetMu.Unlock()
|
||||
if exceeded {
|
||||
return fmt.Errorf("%w: bytes=%d/%d documents=%d/%d", workforcestats.ErrOutputSourceBudget, bytes, budget.bytes, documents, budget.documents)
|
||||
}
|
||||
if spool != nil {
|
||||
if err = spool.add(c.Name(), cursor.Current); err != nil {
|
||||
return err
|
||||
}
|
||||
continue
|
||||
}
|
||||
v := reflect.New(target.Type().Elem())
|
||||
if err = cursor.Decode(v.Interface()); err != nil {
|
||||
return err
|
||||
@ -276,6 +285,13 @@ func (s *MongoStore) loadAnnotationOutput(ctx context.Context, org string, proje
|
||||
runIDs = append(runIDs, c.RunID)
|
||||
captureIDs = append(captureIDs, c.ID)
|
||||
}
|
||||
readDirectory := func(table string, c *mongo.Collection, key string, keys []string, projection bson.M, out any, constraints ...bson.M) error {
|
||||
if spool == nil {
|
||||
return readRelated(c, key, keys, projection, out, constraints...)
|
||||
}
|
||||
return spool.index.eachKeys(readCtx, table, func(batch []string) error { return readRelated(c, key, batch, projection, out, constraints...) })
|
||||
}
|
||||
|
||||
g.Go(func() error {
|
||||
projection := fields("id", "taskId", "mode", "createdAt")
|
||||
if err := readRelated(s.runs, "projectId", projects, projection, &d.Runs); err != nil {
|
||||
@ -285,7 +301,7 @@ func (s *MongoStore) loadAnnotationOutput(ctx context.Context, org string, proje
|
||||
// links are disjoint. Deduplicating after fetching still spends the
|
||||
// global byte/document budget on every repeated row.
|
||||
var relatedRuns []domain.RunRecord
|
||||
if err := readRelated(s.runs, "taskId", captureIDs, projection, &relatedRuns, bson.M{"projectId": bson.M{"$nin": projects}}); err != nil {
|
||||
if err := readDirectory("captures", s.runs, "taskId", captureIDs, projection, &relatedRuns, bson.M{"projectId": bson.M{"$nin": projects}}); err != nil {
|
||||
return err
|
||||
}
|
||||
d.Runs = append(d.Runs, relatedRuns...)
|
||||
@ -293,6 +309,22 @@ func (s *MongoStore) loadAnnotationOutput(ctx context.Context, org string, proje
|
||||
for _, run := range d.Runs {
|
||||
seen[run.ID] = true
|
||||
}
|
||||
if spool != nil {
|
||||
return spool.index.eachKeys(readCtx, "runRefs", func(refs []string) error {
|
||||
missing := make([]string, 0, len(refs))
|
||||
for _, id := range refs {
|
||||
seen, err := spool.index.contains("runs", id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !seen {
|
||||
missing = append(missing, id)
|
||||
}
|
||||
}
|
||||
return readRelated(s.runs, "id", missing, projection, &relatedRuns)
|
||||
})
|
||||
}
|
||||
|
||||
missing := make([]string, 0, len(runIDs))
|
||||
for _, id := range runIDs {
|
||||
if !seen[id] {
|
||||
@ -310,24 +342,26 @@ func (s *MongoStore) loadAnnotationOutput(ctx context.Context, org string, proje
|
||||
for _, t := range d.Tasks {
|
||||
ids = append(ids, t.ID)
|
||||
}
|
||||
|
||||
g.Go(func() error {
|
||||
return readRelated(s.annotationReviews, "taskId", ids, fields("id", "taskId", "revision", "stage", "decision", "reviewerUserId", "createdAt"), &d.Reviews)
|
||||
return readDirectory("tasks", s.annotationReviews, "taskId", ids, fields("id", "taskId", "revision", "stage", "decision", "reviewerUserId", "createdAt"), &d.Reviews)
|
||||
})
|
||||
g.Go(func() error {
|
||||
return readRelated(s.annotationRevisions, "taskId", ids, fields("taskId", "revision", "authorUserId", "isRework", "countsAsOutput"), &d.Revisions)
|
||||
return readDirectory("tasks", s.annotationRevisions, "taskId", ids, fields("taskId", "revision", "authorUserId", "isRework", "countsAsOutput"), &d.Revisions)
|
||||
})
|
||||
itemIDs := []string{}
|
||||
for _, i := range d.Items {
|
||||
itemIDs = append(itemIDs, i.ID)
|
||||
}
|
||||
|
||||
g.Go(func() error {
|
||||
return readRelated(s.humanEvents, "itemId", itemIDs, fields("id", "itemId", "stageId", "eventType", "actorUserId", "createdAt", "payload.operationId", "payload.sourceItemIds", "payload.deliveryPackageId", "payload.reviewedRevision"), &d.HumanEvents, bson.M{"stageId": bson.M{"$in": workforcestats.OutputAcceptanceStageIDs()}})
|
||||
return readDirectory("items", s.humanEvents, "itemId", itemIDs, fields("id", "itemId", "stageId", "eventType", "actorUserId", "createdAt", "payload.operationId", "payload.sourceItemIds", "payload.deliveryPackageId", "payload.reviewedRevision"), &d.HumanEvents, bson.M{"stageId": bson.M{"$in": workforcestats.OutputAcceptanceStageIDs()}})
|
||||
})
|
||||
g.Go(func() error {
|
||||
return read(s.db.Collection("reworkBaselines"), f, fields("_id", "itemId", "round", "stageId", "reviewOperationId", "problems.id", "problems.scope", "problems.sourceItemId", "problems.origin.stageId", "createdAt"), &d.Baselines)
|
||||
})
|
||||
g.Go(func() error {
|
||||
return readRelated(s.db.Collection("reworkHandlings"), "itemId", itemIDs, fields("_id", "itemId", "round", "operationId", "skippedOpinionIds", "updatedAt"), &d.Handlings)
|
||||
return readDirectory("items", s.db.Collection("reworkHandlings"), "itemId", itemIDs, fields("_id", "itemId", "round", "operationId", "skippedOpinionIds", "updatedAt"), &d.Handlings)
|
||||
})
|
||||
err := g.Wait()
|
||||
d.SourceStats = &workforcestats.OutputSourceStats{Bytes: totalBytes, Documents: totalDocs}
|
||||
|
||||
163
internal/store/annotation_output_growth_test.go
Normal file
163
internal/store/annotation_output_growth_test.go
Normal file
@ -0,0 +1,163 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"droboticflow/internal/outputsnapshot"
|
||||
"droboticflow/internal/workforcestats"
|
||||
"fmt"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/mongo"
|
||||
"os"
|
||||
"runtime"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// localOutputMongo creates annotation_output_snapshots_test_<UnixNano> and
|
||||
// registers db.Drop in t.Cleanup. No shared snapshot database is mutated.
|
||||
func TestStatisticsPartitionGrowthBeyondGlobalBudget(t *testing.T) {
|
||||
if os.Getenv("RUN_OUTPUT_HISTORY_SCALE") != "1" {
|
||||
t.Skip("explicit isolated growth benchmark")
|
||||
}
|
||||
tempRoot := t.TempDir()
|
||||
t.Setenv("TMPDIR", tempRoot)
|
||||
db := localOutputMongo(t)
|
||||
s := NewMongoStore(db)
|
||||
ctx := t.Context()
|
||||
now := time.Now().UTC().Truncate(time.Millisecond)
|
||||
const n = 72000
|
||||
for _, name := range []string{"annotationTasks", "humanItems", "annotationRevisions", "annotationWorklogEvents", "annotationReviews", "captureRecords", "runs"} {
|
||||
for _, keys := range []bson.D{{{Key: "organizationId", Value: 1}, {Key: "projectId", Value: 1}, {Key: "taskId", Value: 1}}, {{Key: "organizationId", Value: 1}, {Key: "taskId", Value: 1}}, {{Key: "organizationId", Value: 1}, {Key: "id", Value: 1}}} {
|
||||
if _, err := db.Collection(name).Indexes().CreateOne(ctx, mongo.IndexModel{Keys: keys}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for start := 0; start < n; start += 1000 {
|
||||
rows := []any{}
|
||||
for i := start; i < min(start+1000, n); i++ {
|
||||
id := fmt.Sprint("t", i)
|
||||
row := bson.M{"id": id, "organizationId": "out", "projectId": "p"}
|
||||
switch name {
|
||||
case "annotationTasks":
|
||||
row["runId"] = id
|
||||
row["sourceIsFullVideo"] = true
|
||||
row["sourceDurationSeconds"] = 60
|
||||
case "humanItems":
|
||||
row["legacyTaskId"] = id
|
||||
case "annotationRevisions":
|
||||
row["taskId"] = id
|
||||
row["revision"] = 1
|
||||
row["countsAsOutput"] = true
|
||||
row["authorUserId"] = "u"
|
||||
case "annotationWorklogEvents":
|
||||
row["id"] = id + strings.Repeat("x", 512)
|
||||
row["taskId"] = id
|
||||
row["userId"] = "u"
|
||||
row["supplierId"] = "v"
|
||||
row["eventType"] = "combined_submitted"
|
||||
row["revision"] = 1
|
||||
row["createdAt"] = now.Add(-time.Hour).Format(time.RFC3339Nano)
|
||||
case "annotationReviews":
|
||||
row["taskId"] = id
|
||||
row["stage"] = "quality"
|
||||
row["decision"] = "approved"
|
||||
row["reviewerUserId"] = "qc"
|
||||
row["revision"] = 1
|
||||
row["createdAt"] = now.Add(-time.Minute).Format(time.RFC3339Nano)
|
||||
case "captureRecords":
|
||||
row["runId"] = id
|
||||
row["prefix"] = "capture/" + id + "/part0001"
|
||||
row["captureId"] = id + "/part0001"
|
||||
case "runs":
|
||||
row["taskId"] = id
|
||||
}
|
||||
rows = append(rows, row)
|
||||
}
|
||||
if _, err := db.Collection(name).InsertMany(ctx, rows); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
runtime.GC()
|
||||
var initial runtime.MemStats
|
||||
runtime.ReadMemStats(&initial)
|
||||
var peak atomic.Uint64
|
||||
peak.Store(initial.HeapAlloc)
|
||||
stop := make(chan struct{})
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
ticker := time.NewTicker(20 * time.Millisecond)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-stop:
|
||||
return
|
||||
case <-ticker.C:
|
||||
var m runtime.MemStats
|
||||
runtime.ReadMemStats(&m)
|
||||
for old := peak.Load(); m.HeapAlloc > old; old = peak.Load() {
|
||||
if peak.CompareAndSwap(old, m.HeapAlloc) {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
defer func() { close(stop); <-done }()
|
||||
var beforeCPU syscall.Rusage
|
||||
_ = syscall.Getrusage(syscall.RUSAGE_SELF, &beforeCPU)
|
||||
reader := outputsnapshot.NewStreamReader(s, s.StreamAnnotationOutput)
|
||||
defer reader.Close()
|
||||
q := workforcestats.OutputQuery{From: now.Add(-24 * time.Hour), To: now.Add(time.Hour), ProductionQuality: true}
|
||||
started := time.Now()
|
||||
deadline := started.Add(100 * time.Second)
|
||||
for {
|
||||
report, info, err := reader.Query(ctx, "out", []string{"p"}, "", q)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if report != nil {
|
||||
if report.Production.Totals.Submitted.Count != n {
|
||||
t.Fatalf("lost parts: %+v", report.Production.Totals.Submitted)
|
||||
}
|
||||
health, err := reader.Inspect(ctx, "out", []string{"p"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
source := health.Metrics.Source
|
||||
if source.Bytes <= workforcestats.OutputSourceMaxBytes || source.Documents <= workforcestats.OutputSourceMaxDocuments || source.PeakBatchBytes > workforcestats.OutputSourceMaxBytes {
|
||||
t.Fatalf("fixture did not cross old limits or violated batch bound: %+v", source)
|
||||
}
|
||||
if len(health.Warnings) > 0 {
|
||||
t.Fatalf("growth capacity warning %+v", health)
|
||||
}
|
||||
entries, err := os.ReadDir(tempRoot)
|
||||
if err != nil || len(entries) != 0 {
|
||||
t.Fatalf("temporary build files leaked: %d %v", len(entries), err)
|
||||
}
|
||||
reader.Close()
|
||||
runtime.GC()
|
||||
var retained runtime.MemStats
|
||||
runtime.ReadMemStats(&retained)
|
||||
t.Logf("post_gc_heap=%d temporary_files=%d", retained.HeapAlloc, len(entries))
|
||||
if retained.HeapAlloc > initial.HeapAlloc+16<<20 {
|
||||
t.Fatal("retained build heap grew beyond 16 MiB allowance")
|
||||
}
|
||||
var afterCPU syscall.Rusage
|
||||
_ = syscall.Getrusage(syscall.RUSAGE_SELF, &afterCPU)
|
||||
cpu := func(r syscall.Rusage) float64 {
|
||||
return float64(r.Utime.Sec+r.Stime.Sec) + float64(r.Utime.Usec+r.Stime.Usec)/1e6
|
||||
}
|
||||
t.Logf("heap_baseline=%d heap_peak=%d cpu_seconds=%.3f process_maxrss=%d", initial.HeapAlloc, peak.Load(), cpu(afterCPU)-cpu(beforeCPU), afterCPU.Maxrss)
|
||||
t.Logf("parts=%d source=%+v metrics=%+v encoded=%d build_and_cold_query=%s generation=%s", n, source, health.Metrics, health.EncodedBytes, time.Since(started), info.ID)
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("growth build did not publish")
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
92
internal/store/annotation_output_shards_test.go
Normal file
92
internal/store/annotation_output_shards_test.go
Normal file
@ -0,0 +1,92 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"droboticflow/internal/outputsnapshot"
|
||||
"droboticflow/internal/workforcestats"
|
||||
"errors"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestAnnotationOutputMongoShardedRestartAndFailedBuild(t *testing.T) {
|
||||
db := localOutputMongo(t)
|
||||
s := NewMongoStore(db)
|
||||
seedAnnotationOutput(t, s)
|
||||
r := outputsnapshot.NewStreamReader(s, s.StreamAnnotationOutput)
|
||||
defer r.Close()
|
||||
q := workforcestats.OutputQuery{From: time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC), To: time.Date(2026, 10, 21, 0, 0, 0, 0, time.UTC)}
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
var info outputsnapshot.Info
|
||||
for {
|
||||
report, current, err := r.Query(t.Context(), "out", []string{"p"}, "", q)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if report != nil {
|
||||
if report.Totals.Acceptance.Count != 1 {
|
||||
t.Fatal("wrong persisted report")
|
||||
}
|
||||
info = current
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("not published")
|
||||
}
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
restarted := outputsnapshot.NewStreamReader(NewMongoStore(db), s.StreamAnnotationOutput)
|
||||
defer restarted.Close()
|
||||
report, got, err := restarted.Query(t.Context(), "out", []string{"p"}, info.ID, q)
|
||||
if err != nil || report == nil || got.ID != info.ID || report.Totals.Acceptance.Count != 1 {
|
||||
t.Fatalf("restart mismatch %v %+v", err, got)
|
||||
}
|
||||
state, err := s.OutputSnapshotState(t.Context())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, head := range state.Heads {
|
||||
if head.Current == nil || head.Current.Shards == 0 {
|
||||
t.Fatal("legacy single blob used")
|
||||
}
|
||||
}
|
||||
now := time.Now()
|
||||
old, ok, err := s.AcquireOutputSnapshot(t.Context(), "isolated", now)
|
||||
if err != nil || !ok {
|
||||
t.Fatal("lease unavailable", err)
|
||||
}
|
||||
next, ok, err := s.AcquireOutputSnapshot(t.Context(), "isolated", now.Add(outputsnapshot.LeaseDuration+time.Second))
|
||||
if err != nil || !ok {
|
||||
t.Fatal("lease recovery", err)
|
||||
}
|
||||
if ok, err = s.StageOutputSnapshotShard(t.Context(), old, 0, []byte("stale"), now.Add(outputsnapshot.LeaseDuration+time.Second)); err != nil || ok {
|
||||
t.Fatal("stale writer staged", err)
|
||||
}
|
||||
if ok, err = s.StageOutputSnapshotShard(t.Context(), next, 0, []byte("new"), now.Add(outputsnapshot.LeaseDuration+time.Second)); err != nil || !ok {
|
||||
t.Fatal("new writer unable to stage", err)
|
||||
}
|
||||
}
|
||||
func TestAnnotationOutputStreamFailureCleansPrivateFiles(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("TMPDIR", root)
|
||||
s := NewMongoStore(localOutputMongo(t))
|
||||
seedAnnotationOutput(t, s)
|
||||
sentinel := errors.New("consumer failed")
|
||||
_, err := s.StreamAnnotationOutput(context.Background(), "out", []string{"p"}, func(workforcestats.OutputSnapshot) error { return sentinel })
|
||||
if !errors.Is(err, sentinel) {
|
||||
t.Fatal("consumer failure lost", err)
|
||||
}
|
||||
entries, err := os.ReadDir(root)
|
||||
if err != nil || len(entries) != 0 {
|
||||
t.Fatalf("temporary files leaked: %d %v", len(entries), err)
|
||||
}
|
||||
_, err = s.StreamAnnotationOutput(context.Background(), "out", []string{"p"}, func(workforcestats.OutputSnapshot) error { return workforcestats.ErrOutputPartitionSplit })
|
||||
if !errors.Is(err, workforcestats.ErrOutputSourceBudget) {
|
||||
t.Fatal("indivisible part did not fail explicitly", err)
|
||||
}
|
||||
entries, err = os.ReadDir(root)
|
||||
if err != nil || len(entries) != 0 {
|
||||
t.Fatal("overflow files leaked")
|
||||
}
|
||||
}
|
||||
@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/outputsnapshot"
|
||||
"fmt"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/mongo"
|
||||
"go.mongodb.org/mongo-driver/mongo/options"
|
||||
@ -79,7 +80,7 @@ func (s *MongoStore) RenewOutputSnapshot(ctx context.Context, l outputsnapshot.L
|
||||
return err == nil && res.MatchedCount == 1, err
|
||||
}
|
||||
func (s *MongoStore) PublishOutputSnapshot(ctx context.Context, l outputsnapshot.Lease, g outputsnapshot.Generation, data []byte, now time.Time) (bool, error) {
|
||||
if g.Slot != l.Slot || g.Epoch != l.Epoch || len(data) > outputsnapshot.MaxEncodedBytes || g.Bytes != len(data) || l.Slot < 0 || l.Slot >= outputsnapshot.SlotCount {
|
||||
if !outputsnapshot.ValidPublication(l, g, data) {
|
||||
return false, outputsnapshot.ErrCapacity
|
||||
}
|
||||
ok, err := s.RenewOutputSnapshot(ctx, l, now)
|
||||
@ -99,6 +100,12 @@ func (s *MongoStore) PublishOutputSnapshot(ctx context.Context, l outputsnapshot
|
||||
}
|
||||
head := outputsnapshot.Head{Current: &g, Previous: l.Previous, NextAttempt: g.AsOf.Add(outputsnapshot.RefreshInterval)}
|
||||
res, err := s.outputCoordinator().UpdateOne(write, outputLeaseFilter(l, now), bson.M{"$set": bson.M{"heads." + l.Key: head, "owner": "", "until": now}})
|
||||
if err == nil && res.MatchedCount == 1 {
|
||||
// Same slot only: current and previous generations occupy different slots.
|
||||
// Epoch-qualified immutable keys protect against late expired writers.
|
||||
prefix := fmt.Sprintf("of-shard:%02d:", l.Slot)
|
||||
_, _ = s.outputBlobs().DeleteMany(write, bson.M{"_id": bson.M{"$gte": prefix, "$lt": fmt.Sprintf("%s%020d:", prefix, l.Epoch)}})
|
||||
}
|
||||
return err == nil && res.MatchedCount == 1, err
|
||||
}
|
||||
func (s *MongoStore) FailOutputSnapshot(ctx context.Context, l outputsnapshot.Lease, now time.Time, failure outputsnapshot.Failure) error {
|
||||
@ -115,3 +122,42 @@ func (s *MongoStore) OutputSnapshotBlob(ctx context.Context, g outputsnapshot.Ge
|
||||
}
|
||||
return doc.Data, err
|
||||
}
|
||||
|
||||
func outputShardID(slot int, epoch int64, index int) string {
|
||||
return fmt.Sprintf("of-shard:%02d:%020d:%09d", slot, epoch, index)
|
||||
}
|
||||
func (s *MongoStore) StageOutputSnapshotShard(ctx context.Context, l outputsnapshot.Lease, index int, data []byte, now time.Time) (bool, error) {
|
||||
if l.Slot < 0 || l.Slot >= outputsnapshot.SlotCount || index < 0 || index >= outputsnapshot.MaxShards || len(data) > outputsnapshot.MaxEncodedBytes {
|
||||
return false, outputsnapshot.ErrCapacity
|
||||
}
|
||||
ok, err := s.RenewOutputSnapshot(ctx, l, now)
|
||||
if err != nil || !ok {
|
||||
return false, err
|
||||
}
|
||||
if index == 0 {
|
||||
prefix := fmt.Sprintf("of-shard:%02d:", l.Slot)
|
||||
if _, err = s.outputBlobs().DeleteMany(ctx, bson.M{"_id": bson.M{"$gte": prefix, "$lt": fmt.Sprintf("%s%020d:", prefix, l.Epoch)}}); err != nil {
|
||||
return false, err
|
||||
}
|
||||
}
|
||||
// Immutable: never overwrite a staged shard.
|
||||
id := outputShardID(l.Slot, l.Epoch, index)
|
||||
_, err = s.outputBlobs().InsertOne(ctx, bson.M{"_id": id, "data": data})
|
||||
return err == nil, err
|
||||
}
|
||||
func (s *MongoStore) OutputSnapshotShard(ctx context.Context, g outputsnapshot.Generation, index int) ([]byte, error) {
|
||||
var doc struct {
|
||||
Data []byte `bson:"data"`
|
||||
}
|
||||
err := s.outputBlobs().FindOne(ctx, bson.M{"_id": outputShardID(g.Slot, g.Epoch, index)}, options.FindOne().SetProjection(bson.M{"data": 1})).Decode(&doc)
|
||||
if err == mongo.ErrNoDocuments {
|
||||
err = outputsnapshot.ErrExpired
|
||||
}
|
||||
return doc.Data, err
|
||||
}
|
||||
func (s *MemoryStore) StageOutputSnapshotShard(ctx context.Context, l outputsnapshot.Lease, i int, b []byte, t time.Time) (bool, error) {
|
||||
return s.outputSnapshots.StageOutputSnapshotShard(ctx, l, i, b, t)
|
||||
}
|
||||
func (s *MemoryStore) OutputSnapshotShard(ctx context.Context, g outputsnapshot.Generation, i int) ([]byte, error) {
|
||||
return s.outputSnapshots.OutputSnapshotShard(ctx, g, i)
|
||||
}
|
||||
|
||||
45
internal/store/annotation_output_spool_cleanup.go
Normal file
45
internal/store/annotation_output_spool_cleanup.go
Normal file
@ -0,0 +1,45 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Private build files have no recovery value. A live advisory lock protects
|
||||
// slow/canceling builds; after a process crash the OS releases it. Restrict
|
||||
// cleanup to old, real directories with our marker, never follow symlinks.
|
||||
func cleanupOutputSpools(root string, now time.Time) {
|
||||
entries, err := os.ReadDir(root)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
cleaned := 0
|
||||
for _, entry := range entries {
|
||||
if cleaned >= 64 {
|
||||
return
|
||||
}
|
||||
if !entry.IsDir() || !strings.HasPrefix(entry.Name(), "droboticflow-output-") {
|
||||
continue
|
||||
}
|
||||
path := filepath.Join(root, entry.Name())
|
||||
info, err := os.Lstat(path)
|
||||
if err != nil || !info.IsDir() || now.Sub(info.ModTime()) < 10*time.Minute {
|
||||
continue
|
||||
}
|
||||
guard, err := os.OpenFile(filepath.Join(path, ".build-lock"), os.O_RDWR, 0)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if syscall.Flock(int(guard.Fd()), syscall.LOCK_EX|syscall.LOCK_NB) != nil {
|
||||
guard.Close()
|
||||
continue
|
||||
}
|
||||
_ = os.RemoveAll(path)
|
||||
_ = syscall.Flock(int(guard.Fd()), syscall.LOCK_UN)
|
||||
guard.Close()
|
||||
cleaned++
|
||||
}
|
||||
}
|
||||
226
internal/store/annotation_output_spool_index.go
Normal file
226
internal/store/annotation_output_spool_index.go
Normal file
@ -0,0 +1,226 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"droboticflow/internal/workforcestats"
|
||||
bolt "go.etcd.io/bbolt"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
)
|
||||
|
||||
// Rebuildable, per-build temporary associations, never customer persistence.
|
||||
// Transactions are bounded; no read transaction survives a callback that may
|
||||
// write/grow the mmap. All returned bytes are owned copies.
|
||||
type outputSpoolIndex struct {
|
||||
mu sync.Mutex
|
||||
db *bolt.DB
|
||||
tx *bolt.Tx
|
||||
pending, operations, peak int
|
||||
disk int64
|
||||
charge func(int64) error
|
||||
}
|
||||
|
||||
func newOutputSpoolIndex(path string, charge func(int64) error) (*outputSpoolIndex, error) {
|
||||
db, e := bolt.Open(path, 0600, &bolt.Options{Timeout: time.Second, NoSync: true, NoFreelistSync: true})
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
return &outputSpoolIndex{db: db, charge: charge}, nil
|
||||
}
|
||||
func outputIndexKey(key string) []byte { return append([]byte{1}, []byte(key)...) }
|
||||
func (k *outputSpoolIndex) commitLocked() error {
|
||||
if k.tx != nil {
|
||||
e := k.tx.Commit()
|
||||
k.tx = nil
|
||||
k.pending = 0
|
||||
k.operations = 0
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
var e error
|
||||
var size int64
|
||||
// DataSize is the mapped address-space size, not a retained Go object graph.
|
||||
// Charge the actual backing file rather than speculative mmap growth.
|
||||
if stat, se := k.dbFileSize(); se == nil {
|
||||
size = stat
|
||||
} else {
|
||||
e = se
|
||||
}
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if size > k.disk {
|
||||
if e = k.charge(size - k.disk); e != nil {
|
||||
return e
|
||||
}
|
||||
k.disk = size
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (k *outputSpoolIndex) putLocked(table, key string, value []byte) error {
|
||||
if len(value) > 16<<20 {
|
||||
return fmt.Errorf("%w: one reporting association exceeds BSON size", workforcestats.ErrOutputSourceBudget)
|
||||
}
|
||||
if k.tx != nil && (k.pending+len(value) > 2<<20 || k.operations >= 512) {
|
||||
if e := k.commitLocked(); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
if k.tx == nil {
|
||||
tx, e := k.db.Begin(true)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
k.tx = tx
|
||||
}
|
||||
bucket, e := k.tx.CreateBucketIfNotExists([]byte(table))
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
keyBytes := outputIndexKey(key)
|
||||
if e = bucket.Put(keyBytes, append([]byte(nil), value...)); e != nil {
|
||||
return e
|
||||
}
|
||||
k.pending += len(value) + len(keyBytes) + 64
|
||||
k.operations++
|
||||
k.peak = max(k.peak, k.pending)
|
||||
return nil
|
||||
}
|
||||
func (k *outputSpoolIndex) put(table, key string, value []byte) error {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
return k.putLocked(table, key, value)
|
||||
}
|
||||
func (k *outputSpoolIndex) getLocked(table, key string) ([]byte, error) {
|
||||
var out []byte
|
||||
read := func(tx *bolt.Tx) error {
|
||||
if b := tx.Bucket([]byte(table)); b != nil {
|
||||
out = append(out, b.Get(outputIndexKey(key))...)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if k.tx != nil {
|
||||
e := read(k.tx)
|
||||
return out, e
|
||||
}
|
||||
e := k.db.View(read)
|
||||
return out, e
|
||||
}
|
||||
func (k *outputSpoolIndex) get(table, key string) ([]byte, error) {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
return k.getLocked(table, key)
|
||||
}
|
||||
func (k *outputSpoolIndex) contains(table, key string) (bool, error) {
|
||||
v, e := k.get(table, key)
|
||||
return len(v) > 0, e
|
||||
}
|
||||
func (k *outputSpoolIndex) addParts(table, key string, parts []string) error {
|
||||
if key == "" {
|
||||
return nil
|
||||
}
|
||||
parts = uniqueOutputParts(parts)
|
||||
if len(parts) == 0 {
|
||||
return nil
|
||||
}
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
raw, e := k.getLocked(table, key)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
var old struct {
|
||||
Parts []string `bson:"p"`
|
||||
}
|
||||
if len(raw) > 0 {
|
||||
if e = bson.Unmarshal(raw, &old); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
value, e := bson.Marshal(bson.M{"p": uniqueOutputParts(append(old.Parts, parts...))})
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
return k.putLocked(table, key, value)
|
||||
}
|
||||
func (k *outputSpoolIndex) parts(table, key string) ([]string, error) {
|
||||
raw, e := k.get(table, key)
|
||||
if e != nil || len(raw) == 0 {
|
||||
return nil, e
|
||||
}
|
||||
var v struct {
|
||||
Parts []string `bson:"p"`
|
||||
}
|
||||
e = bson.Unmarshal(raw, &v)
|
||||
return v.Parts, e
|
||||
}
|
||||
func (k *outputSpoolIndex) eachKeys(ctx context.Context, table string, visit func([]string) error) error {
|
||||
var last []byte
|
||||
for {
|
||||
if e := ctx.Err(); e != nil {
|
||||
return e
|
||||
}
|
||||
keys := []string{}
|
||||
k.mu.Lock()
|
||||
read := func(tx *bolt.Tx) error {
|
||||
b := tx.Bucket([]byte(table))
|
||||
if b == nil {
|
||||
return nil
|
||||
}
|
||||
c := b.Cursor()
|
||||
key, _ := c.First()
|
||||
if last != nil {
|
||||
key, _ = c.Seek(last)
|
||||
if bytes.Equal(key, last) {
|
||||
key, _ = c.Next()
|
||||
}
|
||||
}
|
||||
for key != nil && len(keys) < 256 {
|
||||
keys = append(keys, string(key[1:]))
|
||||
last = append(last[:0], key...)
|
||||
key, _ = c.Next()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
var e error
|
||||
if k.tx != nil {
|
||||
e = read(k.tx)
|
||||
} else {
|
||||
e = k.db.View(read)
|
||||
}
|
||||
k.mu.Unlock()
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if len(keys) == 0 {
|
||||
return nil
|
||||
}
|
||||
if e = visit(keys); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
}
|
||||
func (k *outputSpoolIndex) flush() error { k.mu.Lock(); defer k.mu.Unlock(); return k.commitLocked() }
|
||||
func (k *outputSpoolIndex) close() error {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
if k.tx != nil {
|
||||
_ = k.tx.Rollback()
|
||||
k.tx = nil
|
||||
}
|
||||
return k.db.Close()
|
||||
}
|
||||
|
||||
func (k *outputSpoolIndex) dbFileSize() (int64, error) {
|
||||
v, e := os.Stat(k.db.Path())
|
||||
if e != nil {
|
||||
return 0, e
|
||||
}
|
||||
return v.Size(), nil
|
||||
}
|
||||
94
internal/store/annotation_output_spool_index_test.go
Normal file
94
internal/store/annotation_output_spool_index_test.go
Normal file
@ -0,0 +1,94 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestOutputSpoolIndexCopiesPagesAndAllowsWritesDuringIteration(t *testing.T) {
|
||||
var disk int64
|
||||
index, err := newOutputSpoolIndex(filepath.Join(t.TempDir(), "index.db"), func(n int64) error { disk += n; return nil })
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer index.close()
|
||||
value := []byte("before")
|
||||
for i := 0; i < 1600; i++ {
|
||||
if err = index.put("a", fmt.Sprintf("%05d", i), value); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
value[0] = 'X'
|
||||
got, err := index.get("a", "00000")
|
||||
if err != nil || string(got) != "before" {
|
||||
t.Fatal("borrowed input", string(got), err)
|
||||
}
|
||||
got[0] = 'Z'
|
||||
visited := 0
|
||||
err = index.eachKeys(context.Background(), "a", func(keys []string) error {
|
||||
if len(keys) > 256 {
|
||||
t.Fatal("unbounded page")
|
||||
}
|
||||
for _, key := range keys {
|
||||
visited++
|
||||
if err := index.put("b", key, make([]byte, 2048)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil || visited != 1600 {
|
||||
t.Fatalf("page loss %d %v", visited, err)
|
||||
}
|
||||
got, err = index.get("a", "00000")
|
||||
if err != nil || string(got) != "before" {
|
||||
t.Fatal("borrowed result", string(got), err)
|
||||
}
|
||||
if err = index.flush(); err != nil || disk == 0 {
|
||||
t.Fatalf("disk accounting %d %v", disk, err)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if err = index.eachKeys(ctx, "a", func([]string) error { t.Fatal("cancel ignored"); return nil }); err == nil {
|
||||
t.Fatal("cancel ignored")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOutputSpoolCleanupRecoversCrashedBuildButKeepsActive(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
t.Setenv("TMPDIR", root)
|
||||
active, err := newOutputSourceSpool(t.Context())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer active.close()
|
||||
abandoned, err := newOutputSourceSpool(t.Context())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = abandoned.index.close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
abandoned.index = nil
|
||||
if err = abandoned.guard.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
abandoned.guard = nil
|
||||
old := time.Now().Add(-time.Hour)
|
||||
for _, path := range []string{active.root, abandoned.root} {
|
||||
if err = os.Chtimes(path, old, old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
cleanupOutputSpools(root, time.Now())
|
||||
if _, err = os.Stat(active.root); err != nil {
|
||||
t.Fatal("deleted active build", err)
|
||||
}
|
||||
if _, err = os.Stat(abandoned.root); !os.IsNotExist(err) {
|
||||
t.Fatal("abandoned build not cleaned", err)
|
||||
}
|
||||
}
|
||||
847
internal/store/annotation_output_stream.go
Normal file
847
internal/store/annotation_output_stream.go
Normal file
@ -0,0 +1,847 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/binary"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/workforcestats"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
)
|
||||
|
||||
const outputBatchTarget = 8 << 20
|
||||
|
||||
// A build spills projected metadata to private temporary files. The directory
|
||||
// retains identities, not revision/decision history. Only one bounded canonical
|
||||
// part partition is interpreted at a time; a part never spans partitions.
|
||||
type outputSourceSpool struct {
|
||||
ctx context.Context
|
||||
root string
|
||||
guard *os.File
|
||||
mu sync.Mutex
|
||||
files map[string]*os.File
|
||||
disk, peakDisk int64
|
||||
directoryBytes int
|
||||
index *outputSpoolIndex
|
||||
directories workforcestats.OutputSnapshot
|
||||
}
|
||||
type outputTaskRef struct {
|
||||
ID string `bson:"id"`
|
||||
Project string `bson:"projectId"`
|
||||
Run string `bson:"runId"`
|
||||
Data string `bson:"dataTaskId"`
|
||||
}
|
||||
type outputItemRef struct {
|
||||
ID string `bson:"id"`
|
||||
Task string `bson:"legacyTaskId"`
|
||||
Release string `bson:"releasePartitionId"`
|
||||
}
|
||||
type outputRunRef struct {
|
||||
ID string `bson:"id"`
|
||||
Task string `bson:"taskId"`
|
||||
}
|
||||
type outputEnvelope struct {
|
||||
Kind string `bson:"k"`
|
||||
Parts []string `bson:"p"`
|
||||
Video string `bson:"v,omitempty"`
|
||||
Data []byte `bson:"d"`
|
||||
}
|
||||
|
||||
func newOutputSourceSpool(ctx context.Context) (*outputSourceSpool, error) {
|
||||
cleanupOutputSpools(os.TempDir(), time.Now())
|
||||
root, err := os.MkdirTemp("", "droboticflow-output-")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
p := &outputSourceSpool{ctx: ctx, root: root, files: map[string]*os.File{}}
|
||||
p.guard, err = os.OpenFile(filepath.Join(root, ".build-lock"), os.O_CREATE|os.O_RDWR, 0600)
|
||||
if err == nil {
|
||||
err = syscall.Flock(int(p.guard.Fd()), syscall.LOCK_EX|syscall.LOCK_NB)
|
||||
}
|
||||
if err != nil {
|
||||
if p.guard != nil {
|
||||
_ = p.guard.Close()
|
||||
}
|
||||
_ = os.RemoveAll(root)
|
||||
return nil, err
|
||||
}
|
||||
p.index, err = newOutputSpoolIndex(filepath.Join(root, "associations.db"), p.reserve)
|
||||
if err != nil {
|
||||
_ = p.guard.Close()
|
||||
_ = os.RemoveAll(root)
|
||||
return nil, err
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
func (p *outputSourceSpool) close() {
|
||||
for _, f := range p.files {
|
||||
_ = f.Close()
|
||||
}
|
||||
if p.index != nil {
|
||||
_ = p.index.close()
|
||||
}
|
||||
_ = os.RemoveAll(p.root)
|
||||
if p.guard != nil {
|
||||
_ = p.guard.Close()
|
||||
}
|
||||
}
|
||||
func (p *outputSourceSpool) reserve(n int64) error {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
p.disk += n
|
||||
if p.disk > p.peakDisk {
|
||||
p.peakDisk = p.disk
|
||||
}
|
||||
if p.disk > workforcestats.OutputSpoolMaxBytes {
|
||||
return fmt.Errorf("%w: temporary spool budget", workforcestats.ErrOutputSourceBudget)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func writeOutputRecord(w io.Writer, raw []byte) error {
|
||||
var size [4]byte
|
||||
binary.LittleEndian.PutUint32(size[:], uint32(len(raw)))
|
||||
if _, e := w.Write(size[:]); e != nil {
|
||||
return e
|
||||
}
|
||||
_, e := w.Write(raw)
|
||||
return e
|
||||
}
|
||||
func readOutputRecords(ctx context.Context, path string, visit func([]byte) error) error {
|
||||
f, e := os.Open(path)
|
||||
if errors.Is(e, os.ErrNotExist) {
|
||||
return nil
|
||||
}
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
defer f.Close()
|
||||
r := bufio.NewReaderSize(f, 64<<10)
|
||||
for {
|
||||
if e := ctx.Err(); e != nil {
|
||||
return e
|
||||
}
|
||||
var size [4]byte
|
||||
_, e := io.ReadFull(r, size[:])
|
||||
if e == io.EOF {
|
||||
return nil
|
||||
}
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
n := int(binary.LittleEndian.Uint32(size[:]))
|
||||
if n < 5 || n > 32<<20 {
|
||||
return fmt.Errorf("invalid reporting spool record size")
|
||||
}
|
||||
raw := make([]byte, n)
|
||||
if _, e = io.ReadFull(r, raw); e != nil {
|
||||
return e
|
||||
}
|
||||
if e = visit(raw); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
}
|
||||
func (p *outputSourceSpool) add(kind string, raw []byte) error {
|
||||
if e := p.ctx.Err(); e != nil {
|
||||
return e
|
||||
}
|
||||
directory := false
|
||||
switch kind {
|
||||
case "annotationTasks":
|
||||
var v outputTaskRef
|
||||
if e := bson.Unmarshal(raw, &v); e != nil {
|
||||
return e
|
||||
}
|
||||
if e := p.index.put("tasks", v.ID, raw); e != nil {
|
||||
return e
|
||||
}
|
||||
if v.Run != "" {
|
||||
if e := p.index.put("runRefs", v.Run, []byte{1}); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
case "humanItems":
|
||||
var v outputItemRef
|
||||
if e := bson.Unmarshal(raw, &v); e != nil {
|
||||
return e
|
||||
}
|
||||
if e := p.index.put("items", v.ID, raw); e != nil {
|
||||
return e
|
||||
}
|
||||
case "captureRecords":
|
||||
var v struct {
|
||||
ID string `bson:"id"`
|
||||
Run string `bson:"runId"`
|
||||
}
|
||||
if e := bson.Unmarshal(raw, &v); e != nil {
|
||||
return e
|
||||
}
|
||||
if e := p.index.put("captures", v.ID, raw); e != nil {
|
||||
return e
|
||||
}
|
||||
if v.Run != "" {
|
||||
if e := p.index.put("captureByRun", v.Run, []byte(v.ID)); e != nil {
|
||||
return e
|
||||
}
|
||||
if e := p.index.put("runRefs", v.Run, []byte{1}); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
case "runs":
|
||||
var v outputRunRef
|
||||
if e := bson.Unmarshal(raw, &v); e != nil {
|
||||
return e
|
||||
}
|
||||
if e := p.index.put("runs", v.ID, raw); e != nil {
|
||||
return e
|
||||
}
|
||||
has, e := p.index.contains("captures", v.Task)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if has {
|
||||
if e = p.index.put("captureByRun", v.ID, []byte(v.Task)); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
case "users":
|
||||
var v domain.UserRecord
|
||||
if e := bson.Unmarshal(raw, &v); e != nil {
|
||||
return e
|
||||
}
|
||||
p.directories.Users = append(p.directories.Users, v)
|
||||
directory = true
|
||||
case "suppliers":
|
||||
var v domain.SupplierRecord
|
||||
if e := bson.Unmarshal(raw, &v); e != nil {
|
||||
return e
|
||||
}
|
||||
p.directories.Suppliers = append(p.directories.Suppliers, v)
|
||||
directory = true
|
||||
case "projects":
|
||||
var v domain.ProjectRecord
|
||||
if e := bson.Unmarshal(raw, &v); e != nil {
|
||||
return e
|
||||
}
|
||||
p.directories.Projects = append(p.directories.Projects, v)
|
||||
directory = true
|
||||
}
|
||||
if directory {
|
||||
p.mu.Lock()
|
||||
p.directoryBytes += len(raw) + 512
|
||||
over := p.directoryBytes > workforcestats.OutputDirectoryMaxBytes
|
||||
p.mu.Unlock()
|
||||
if over {
|
||||
return fmt.Errorf("%w: display directory budget", workforcestats.ErrOutputSourceBudget)
|
||||
}
|
||||
}
|
||||
if kind == "captureRecords" || kind == "users" || kind == "suppliers" || kind == "projects" {
|
||||
return nil
|
||||
}
|
||||
if e := p.reserve(int64(len(raw) + 4)); e != nil {
|
||||
return e
|
||||
}
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
f := p.files[kind]
|
||||
if f == nil {
|
||||
var e error
|
||||
f, e = os.OpenFile(filepath.Join(p.root, kind+".raw"), os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0600)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
p.files[kind] = f
|
||||
}
|
||||
return writeOutputRecord(f, raw)
|
||||
}
|
||||
func (p *outputSourceSpool) identities() error {
|
||||
if e := p.index.eachKeys(p.ctx, "tasks", func(ids []string) error {
|
||||
for _, id := range ids {
|
||||
raw, e := p.index.get("tasks", id)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
var t outputTaskRef
|
||||
if e = bson.Unmarshal(raw, &t); e != nil {
|
||||
return e
|
||||
}
|
||||
capID, e := p.index.get("captureByRun", t.Run)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
cRaw, e := p.index.get("captures", string(capID))
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if len(cRaw) == 0 {
|
||||
cRaw, e = p.index.get("captures", t.Data)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
var capture *domain.CaptureRecord
|
||||
if len(cRaw) > 0 {
|
||||
var c domain.CaptureRecord
|
||||
if e = bson.Unmarshal(cRaw, &c); e != nil {
|
||||
return e
|
||||
}
|
||||
capture = &c
|
||||
}
|
||||
identity := workforcestats.OutputPartIdentityForCapture(t.Project, t.ID, capture)
|
||||
value, e := bson.Marshal(identity)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if e = p.index.put("taskParts", id, value); e != nil {
|
||||
return e
|
||||
}
|
||||
if e = p.index.addParts("runParts", t.Run, []string{identity.ID}); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}); e != nil {
|
||||
return e
|
||||
}
|
||||
return p.index.eachKeys(p.ctx, "items", func(ids []string) error {
|
||||
for _, id := range ids {
|
||||
raw, e := p.index.get("items", id)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
var i outputItemRef
|
||||
if e = bson.Unmarshal(raw, &i); e != nil {
|
||||
return e
|
||||
}
|
||||
identity, e := p.taskIdentity(i.Task)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if identity.ID != "" {
|
||||
if e = p.index.put("itemParts", id, []byte(identity.ID)); e != nil {
|
||||
return e
|
||||
}
|
||||
if e = p.index.addParts("releaseParts", i.Release, []string{identity.ID}); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
func (p *outputSourceSpool) taskIdentity(id string) (workforcestats.OutputPartIdentity, error) {
|
||||
var v workforcestats.OutputPartIdentity
|
||||
raw, e := p.index.get("taskParts", id)
|
||||
if e == nil && len(raw) > 0 {
|
||||
e = bson.Unmarshal(raw, &v)
|
||||
}
|
||||
return v, e
|
||||
}
|
||||
func outputPartHash(part string) string {
|
||||
h := sha256.Sum256([]byte(part))
|
||||
return hex.EncodeToString(h[:])
|
||||
}
|
||||
func uniqueOutputParts(parts []string) []string {
|
||||
out := make([]string, 0, len(parts))
|
||||
seen := map[string]bool{}
|
||||
for _, part := range parts {
|
||||
if part != "" && !seen[part] {
|
||||
seen[part] = true
|
||||
out = append(out, part)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
func (p *outputSourceSpool) route(kind string, raw []byte) (outputEnvelope, error) {
|
||||
e := outputEnvelope{Kind: kind, Data: raw}
|
||||
var routeErr error
|
||||
task := func(id string) {
|
||||
v, err := p.taskIdentity(id)
|
||||
if err != nil {
|
||||
routeErr = err
|
||||
}
|
||||
e.Parts = append(e.Parts, v.ID)
|
||||
}
|
||||
item := func(id string) {
|
||||
v, err := p.index.get("itemParts", id)
|
||||
if err != nil {
|
||||
routeErr = err
|
||||
}
|
||||
e.Parts = append(e.Parts, string(v))
|
||||
}
|
||||
refs := func(table, id string) {
|
||||
v, err := p.index.parts(table, id)
|
||||
if err != nil {
|
||||
routeErr = err
|
||||
}
|
||||
e.Parts = append(e.Parts, v...)
|
||||
}
|
||||
switch kind {
|
||||
case "annotationTasks":
|
||||
var v outputTaskRef
|
||||
if err := bson.Unmarshal(raw, &v); err != nil {
|
||||
return e, err
|
||||
}
|
||||
identity, err := p.taskIdentity(v.ID)
|
||||
if err != nil {
|
||||
return e, err
|
||||
}
|
||||
e.Parts = append(e.Parts, identity.ID)
|
||||
e.Video = identity.Video
|
||||
case "humanItems":
|
||||
var v outputItemRef
|
||||
if err := bson.Unmarshal(raw, &v); err != nil {
|
||||
return e, err
|
||||
}
|
||||
item(v.ID)
|
||||
case "runs":
|
||||
var v outputRunRef
|
||||
if err := bson.Unmarshal(raw, &v); err != nil {
|
||||
return e, err
|
||||
}
|
||||
refs("runParts", v.ID)
|
||||
case "annotationWorklogEvents":
|
||||
var v domain.AnnotationWorklogEventRecord
|
||||
if err := bson.Unmarshal(raw, &v); err != nil {
|
||||
return e, err
|
||||
}
|
||||
task(v.TaskID)
|
||||
if err := p.index.addParts("worklogParts", v.ID, e.Parts); err != nil {
|
||||
return e, err
|
||||
}
|
||||
case "annotationReviews":
|
||||
var v domain.AnnotationReviewRecord
|
||||
if err := bson.Unmarshal(raw, &v); err != nil {
|
||||
return e, err
|
||||
}
|
||||
task(v.TaskID)
|
||||
if err := p.index.addParts("reviewParts", v.ID, e.Parts); err != nil {
|
||||
return e, err
|
||||
}
|
||||
case "annotationRevisions":
|
||||
var v domain.AnnotationRevisionRecord
|
||||
if err := bson.Unmarshal(raw, &v); err != nil {
|
||||
return e, err
|
||||
}
|
||||
task(v.TaskID)
|
||||
case "humanEvents":
|
||||
var v domain.HumanEventRecord
|
||||
if err := bson.Unmarshal(raw, &v); err != nil {
|
||||
return e, err
|
||||
}
|
||||
for _, id := range outputStringIDs(v.Payload["sourceItemIds"]) {
|
||||
item(id)
|
||||
}
|
||||
if op, ok := v.Payload["operationId"].(string); ok {
|
||||
if err := p.index.addParts("operationParts", op, e.Parts); err != nil {
|
||||
return e, err
|
||||
}
|
||||
}
|
||||
case "reworkBaselines":
|
||||
var v domain.ReworkBaselineRecord
|
||||
if err := bson.Unmarshal(raw, &v); err != nil {
|
||||
return e, err
|
||||
}
|
||||
item(v.ItemID)
|
||||
if err := p.index.addParts("operationParts", v.ReviewOperationID, e.Parts); err != nil {
|
||||
return e, err
|
||||
}
|
||||
case "reworkHandlings":
|
||||
var v domain.ReworkHandlingRecord
|
||||
if err := bson.Unmarshal(raw, &v); err != nil {
|
||||
return e, err
|
||||
}
|
||||
item(v.ItemID)
|
||||
case "reviewOperations":
|
||||
var v domain.ReviewOperation
|
||||
if err := bson.Unmarshal(raw, &v); err != nil {
|
||||
return e, err
|
||||
}
|
||||
task(v.TaskID)
|
||||
item(v.ItemID)
|
||||
for _, id := range v.SourceItemIDs {
|
||||
item(id)
|
||||
}
|
||||
for _, effect := range v.Effects {
|
||||
if effect.Review != nil {
|
||||
task(effect.Review.TaskID)
|
||||
refs("reviewParts", effect.Review.ID)
|
||||
}
|
||||
if effect.Worklog != nil {
|
||||
refs("worklogParts", effect.Worklog.ID)
|
||||
}
|
||||
}
|
||||
refs("operationParts", v.ID)
|
||||
refs("operationParts", v.OperationID)
|
||||
case "deliveryPackages":
|
||||
var v domain.DeliveryPackageRecord
|
||||
if err := bson.Unmarshal(raw, &v); err != nil {
|
||||
return e, err
|
||||
}
|
||||
for _, id := range v.ItemIDs {
|
||||
item(id)
|
||||
}
|
||||
refs("releaseParts", v.ReleasePartitionID)
|
||||
default:
|
||||
return e, fmt.Errorf("unknown reporting spool collection %s", kind)
|
||||
}
|
||||
e.Parts = uniqueOutputParts(e.Parts)
|
||||
return e, routeErr
|
||||
}
|
||||
func outputStringIDs(v any) []string {
|
||||
switch v := v.(type) {
|
||||
case bson.A:
|
||||
out := []string{}
|
||||
for _, x := range v {
|
||||
if id, ok := x.(string); ok {
|
||||
out = append(out, id)
|
||||
}
|
||||
}
|
||||
return out
|
||||
case []string:
|
||||
return v
|
||||
case []any:
|
||||
out := []string{}
|
||||
for _, x := range v {
|
||||
if id, ok := x.(string); ok {
|
||||
out = append(out, id)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (p *outputSourceSpool) remove(path string) {
|
||||
if info, e := os.Stat(path); e == nil {
|
||||
_ = os.Remove(path)
|
||||
p.mu.Lock()
|
||||
p.disk -= info.Size()
|
||||
p.mu.Unlock()
|
||||
}
|
||||
}
|
||||
func (p *outputSourceSpool) partition(writers map[string]*bufio.Writer, files map[string]*os.File, prefix string, e outputEnvelope) error {
|
||||
groups := map[string][]string{}
|
||||
for _, part := range e.Parts {
|
||||
h := outputPartHash(part)
|
||||
if strings.HasPrefix(h, prefix) {
|
||||
key := h[:len(prefix)+1]
|
||||
groups[key] = append(groups[key], part)
|
||||
}
|
||||
}
|
||||
for key, parts := range groups {
|
||||
copy := e
|
||||
copy.Parts = parts
|
||||
raw, err := bson.Marshal(copy)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = p.reserve(int64(len(raw) + 4)); err != nil {
|
||||
return err
|
||||
}
|
||||
w := writers[key]
|
||||
if w == nil {
|
||||
f, err := os.OpenFile(filepath.Join(p.root, "part-"+key), os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
files[key] = f
|
||||
w = bufio.NewWriterSize(f, 64<<10)
|
||||
writers[key] = w
|
||||
}
|
||||
if err = writeOutputRecord(w, raw); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func closeOutputWriters(writers map[string]*bufio.Writer, files map[string]*os.File) error {
|
||||
var first error
|
||||
for key, w := range writers {
|
||||
if e := w.Flush(); first == nil {
|
||||
first = e
|
||||
}
|
||||
if e := files[key].Close(); first == nil {
|
||||
first = e
|
||||
}
|
||||
}
|
||||
return first
|
||||
}
|
||||
|
||||
// StreamAnnotationOutput preserves the old source interpreter and API contracts,
|
||||
// but replaces its global history-size cliff with per-part bounded partitions.
|
||||
func (s *MongoStore) StreamAnnotationOutput(ctx context.Context, org string, projects []string, yield func(workforcestats.OutputSnapshot) error) (*workforcestats.OutputSourceStats, error) {
|
||||
if len(projects) == 0 {
|
||||
return &workforcestats.OutputSourceStats{}, nil
|
||||
}
|
||||
p, err := newOutputSourceSpool(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer p.close()
|
||||
raw, err := s.loadAnnotationOutputTo(ctx, org, projects, outputSourceBudget{}, p)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, f := range p.files {
|
||||
if err = f.Close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
p.files = map[string]*os.File{}
|
||||
if err = p.identities(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
writers, files := map[string]*bufio.Writer{}, map[string]*os.File{}
|
||||
// References must be resolved before operations are routed; acceptance can
|
||||
// cover parts in other projects/buckets and pending plans still fence events.
|
||||
order := []string{"annotationTasks", "humanItems", "runs", "annotationWorklogEvents", "annotationReviews", "annotationRevisions", "humanEvents", "reworkBaselines", "reworkHandlings", "deliveryPackages", "reviewOperations"}
|
||||
for _, kind := range order {
|
||||
path := filepath.Join(p.root, kind+".raw")
|
||||
err = readOutputRecords(ctx, path, func(b []byte) error {
|
||||
e, err := p.route(kind, b)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return p.partition(writers, files, "", e)
|
||||
})
|
||||
if err != nil {
|
||||
_ = closeOutputWriters(writers, files)
|
||||
return nil, err
|
||||
}
|
||||
p.remove(path)
|
||||
}
|
||||
if err = closeOutputWriters(writers, files); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Association history stays on disk; release its mmap/transactions before
|
||||
// interpreting batches. Each envelope now owns its canonical routing.
|
||||
if err = p.index.flush(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
stats := raw.SourceStats
|
||||
if stats == nil {
|
||||
stats = &workforcestats.OutputSourceStats{}
|
||||
}
|
||||
stats.DirectoryBytes = p.directoryBytes + p.index.peak
|
||||
if err = p.index.close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
p.index = nil
|
||||
p.remove(filepath.Join(p.root, "associations.db"))
|
||||
var emit func(string) error
|
||||
emit = func(prefix string) error {
|
||||
path := filepath.Join(p.root, "part-"+prefix)
|
||||
info, e := os.Stat(path)
|
||||
if errors.Is(e, os.ErrNotExist) {
|
||||
return nil
|
||||
}
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
split := func() error {
|
||||
// An indivisible part must fail explicitly, not spill 64 identical copies.
|
||||
first := ""
|
||||
multiple := errors.New("multiple canonical parts")
|
||||
check := readOutputRecords(ctx, path, func(b []byte) error {
|
||||
var env outputEnvelope
|
||||
if err := bson.Unmarshal(b, &env); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, part := range env.Parts {
|
||||
if first == "" {
|
||||
first = part
|
||||
} else if part != first {
|
||||
return multiple
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if check != nil && !errors.Is(check, multiple) {
|
||||
return check
|
||||
}
|
||||
if check == nil {
|
||||
return fmt.Errorf("%w: one canonical part exceeds partition budget", workforcestats.ErrOutputSourceBudget)
|
||||
}
|
||||
if len(prefix) >= 64 {
|
||||
return fmt.Errorf("%w: one canonical part exceeds partition budget", workforcestats.ErrOutputSourceBudget)
|
||||
}
|
||||
ws, fs := map[string]*bufio.Writer{}, map[string]*os.File{}
|
||||
e := readOutputRecords(ctx, path, func(b []byte) error {
|
||||
var env outputEnvelope
|
||||
if e := bson.Unmarshal(b, &env); e != nil {
|
||||
return e
|
||||
}
|
||||
return p.partition(ws, fs, prefix, env)
|
||||
})
|
||||
ce := closeOutputWriters(ws, fs)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if ce != nil {
|
||||
return ce
|
||||
}
|
||||
p.remove(path)
|
||||
for _, c := range "0123456789abcdef" {
|
||||
if e := emit(prefix + string(c)); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if info.Size() > outputBatchTarget {
|
||||
distinct := map[string]bool{}
|
||||
rawBytes := 0
|
||||
stop := errors.New("multiple parts")
|
||||
check := readOutputRecords(ctx, path, func(b []byte) error {
|
||||
var env outputEnvelope
|
||||
if e := bson.Unmarshal(b, &env); e != nil {
|
||||
return e
|
||||
}
|
||||
rawBytes += len(env.Data)
|
||||
for _, part := range env.Parts {
|
||||
distinct[part] = true
|
||||
if len(distinct) > 1 {
|
||||
return stop
|
||||
}
|
||||
}
|
||||
if rawBytes > workforcestats.OutputSourceMaxBytes {
|
||||
return fmt.Errorf("%w: one canonical part exceeds source budget", workforcestats.ErrOutputSourceBudget)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if errors.Is(check, stop) {
|
||||
return split()
|
||||
}
|
||||
if check != nil {
|
||||
return check
|
||||
}
|
||||
}
|
||||
d := p.directories
|
||||
d.Identities = map[string]workforcestats.OutputPartIdentity{}
|
||||
docs, bytes := 0, 0
|
||||
e = readOutputRecords(ctx, path, func(b []byte) error {
|
||||
var env outputEnvelope
|
||||
if e := bson.Unmarshal(b, &env); e != nil {
|
||||
return e
|
||||
}
|
||||
docs++
|
||||
bytes += len(env.Data)
|
||||
if bytes > workforcestats.OutputSourceMaxBytes || docs > workforcestats.OutputSourceMaxDocuments {
|
||||
return workforcestats.ErrOutputPartitionSplit
|
||||
}
|
||||
return appendOutputEnvelope(&d, env)
|
||||
})
|
||||
if errors.Is(e, workforcestats.ErrOutputPartitionSplit) {
|
||||
return split()
|
||||
}
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if bytes > workforcestats.OutputSourceMaxBytes || docs > workforcestats.OutputSourceMaxDocuments {
|
||||
return split()
|
||||
}
|
||||
if e = yield(d); errors.Is(e, workforcestats.ErrOutputPartitionSplit) {
|
||||
return split()
|
||||
} else if e != nil {
|
||||
return e
|
||||
}
|
||||
stats.Partitions++
|
||||
stats.PeakBatchBytes = max(stats.PeakBatchBytes, bytes)
|
||||
stats.PeakBatchDocuments = max(stats.PeakBatchDocuments, docs)
|
||||
p.remove(path)
|
||||
return nil
|
||||
}
|
||||
for _, c := range "0123456789abcdef" {
|
||||
if err = emit(string(c)); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
stats.PeakSpoolBytes = p.peakDisk
|
||||
return stats, nil
|
||||
}
|
||||
func appendOutputEnvelope(d *workforcestats.OutputSnapshot, e outputEnvelope) error {
|
||||
switch e.Kind {
|
||||
case "annotationTasks":
|
||||
var v domain.AnnotationTaskRecord
|
||||
if err := bson.Unmarshal(e.Data, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
d.Tasks = append(d.Tasks, v)
|
||||
d.Identities[v.ID] = workforcestats.OutputPartIdentity{ID: e.Parts[0], Video: e.Video}
|
||||
case "humanItems":
|
||||
var v domain.HumanItemRecord
|
||||
if err := bson.Unmarshal(e.Data, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
d.Items = append(d.Items, v)
|
||||
case "runs":
|
||||
var v domain.RunRecord
|
||||
if err := bson.Unmarshal(e.Data, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
d.Runs = append(d.Runs, v)
|
||||
case "annotationWorklogEvents":
|
||||
var v domain.AnnotationWorklogEventRecord
|
||||
if err := bson.Unmarshal(e.Data, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
d.Events = append(d.Events, v)
|
||||
case "annotationReviews":
|
||||
var v domain.AnnotationReviewRecord
|
||||
if err := bson.Unmarshal(e.Data, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
d.Reviews = append(d.Reviews, v)
|
||||
case "annotationRevisions":
|
||||
var v domain.AnnotationRevisionRecord
|
||||
if err := bson.Unmarshal(e.Data, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
d.Revisions = append(d.Revisions, v)
|
||||
case "humanEvents":
|
||||
var v domain.HumanEventRecord
|
||||
if err := bson.Unmarshal(e.Data, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
d.HumanEvents = append(d.HumanEvents, v)
|
||||
case "reworkBaselines":
|
||||
var v domain.ReworkBaselineRecord
|
||||
if err := bson.Unmarshal(e.Data, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
d.Baselines = append(d.Baselines, v)
|
||||
case "reworkHandlings":
|
||||
var v domain.ReworkHandlingRecord
|
||||
if err := bson.Unmarshal(e.Data, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
d.Handlings = append(d.Handlings, v)
|
||||
case "reviewOperations":
|
||||
var v domain.ReviewOperation
|
||||
if err := bson.Unmarshal(e.Data, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
d.Operations = append(d.Operations, v)
|
||||
case "deliveryPackages":
|
||||
var v domain.DeliveryPackageRecord
|
||||
if err := bson.Unmarshal(e.Data, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
d.Packages = append(d.Packages, v)
|
||||
default:
|
||||
return fmt.Errorf("unknown reporting batch %s", e.Kind)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
133
internal/store/annotation_output_stream_test.go
Normal file
133
internal/store/annotation_output_stream_test.go
Normal file
@ -0,0 +1,133 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/workforcestats"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
)
|
||||
|
||||
func collectStreamedOutput(t *testing.T, s *MongoStore, asOf time.Time) (workforcestats.OutputDataset, *workforcestats.OutputSourceStats) {
|
||||
t.Helper()
|
||||
d := workforcestats.OutputDataset{AsOf: asOf, Projects: map[string]string{}, Users: map[string]string{}, Suppliers: map[string]string{}}
|
||||
seen := map[string]bool{}
|
||||
stats, e := s.StreamAnnotationOutput(t.Context(), "out", []string{"p"}, func(raw workforcestats.OutputSnapshot) error {
|
||||
batch := workforcestats.BuildOutputDataset(raw, asOf)
|
||||
offset := len(d.Parts)
|
||||
for _, part := range batch.Parts {
|
||||
if seen[part.ID] {
|
||||
return fmt.Errorf("canonical part split across batches")
|
||||
}
|
||||
seen[part.ID] = true
|
||||
}
|
||||
d.Parts = append(d.Parts, batch.Parts...)
|
||||
for _, w := range batch.Work {
|
||||
w.Part += offset
|
||||
d.Work = append(d.Work, w)
|
||||
}
|
||||
for _, p := range batch.Problems {
|
||||
p.Part += offset
|
||||
d.Problems = append(d.Problems, p)
|
||||
}
|
||||
for id, name := range batch.Projects {
|
||||
d.Projects[id] = name
|
||||
}
|
||||
for id, name := range batch.Users {
|
||||
d.Users[id] = name
|
||||
}
|
||||
for id, name := range batch.Suppliers {
|
||||
d.Suppliers[id] = name
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
return d, stats
|
||||
}
|
||||
func TestAnnotationOutputStreamMatchesWholeHistoryAcrossLegacyReferences(t *testing.T) {
|
||||
db := localOutputMongo(t)
|
||||
s := NewMongoStore(db)
|
||||
seedAnnotationOutput(t, s)
|
||||
ctx := t.Context()
|
||||
check := func(e error) {
|
||||
t.Helper()
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
}
|
||||
prefix := "source/c2/part0001"
|
||||
first := workforcestats.OutputPartIdentityForCapture("p", "t", &domain.CaptureRecord{ProjectID: "p", Prefix: "source/c/part0001", CaptureID: "c/part0001"})
|
||||
for n := 0; ; n++ {
|
||||
c := domain.CaptureRecord{ProjectID: "p", Prefix: prefix}
|
||||
part := workforcestats.OutputPartIdentityForCapture("p", "t2", &c)
|
||||
if outputPartHash(part.ID)[:1] != outputPartHash(first.ID)[:1] {
|
||||
break
|
||||
}
|
||||
prefix = fmt.Sprintf("source/c2-%d/part0001", n)
|
||||
}
|
||||
check(s.UpsertRun(ctx, domain.RunRecord{ID: "r2", OrganizationID: "out", ProjectID: "p", TaskID: "c2", CreatedAt: "2026-10-01T00:00:00Z"}))
|
||||
check(s.UpsertCaptureRecord(ctx, domain.CaptureRecord{ID: "c2", OrganizationID: "out", ProjectID: "p", RunID: "r2", Prefix: prefix, CaptureID: "c2/part0001"}))
|
||||
check(s.UpsertAnnotationTask(ctx, domain.AnnotationTaskRecord{ID: "t2", OrganizationID: "out", ProjectID: "p", RunID: "r2", SourceIsFullVideo: true, SourceDurationSeconds: 600, CurrentRevision: 1}))
|
||||
check(s.UpsertHumanItem(ctx, domain.HumanItemRecord{ID: "i2", OrganizationID: "out", ProjectID: "p", LegacyTaskID: "t2"}))
|
||||
check(s.UpsertAnnotationRevision(ctx, domain.AnnotationRevisionRecord{ID: "rev2", OrganizationID: "out", TaskID: "t2", Revision: 1, AuthorUserID: "u", CountsAsOutput: true}))
|
||||
check(s.UpsertAnnotationWorklogEvent(ctx, domain.AnnotationWorklogEventRecord{ID: "s2", OrganizationID: "out", ProjectID: "p", TaskID: "t2", UserID: "u", SupplierID: "v", EventType: "combined_submitted", Revision: 1, CreatedAt: "2026-10-02T00:00:00Z"}))
|
||||
check(s.UpsertAnnotationReview(ctx, domain.AnnotationReviewRecord{ID: "q2", OrganizationID: "out", TaskID: "t2", Revision: 1, Stage: domain.AnnotationReviewStageQuality, Decision: domain.AnnotationReviewApproved, CreatedAt: "2026-10-03T00:00:00Z"}))
|
||||
// A second processing attempt for the same capture must stay in the same
|
||||
// canonical partition; it cannot be independently counted as another part.
|
||||
check(s.UpsertAnnotationTask(ctx, domain.AnnotationTaskRecord{ID: "retry", OrganizationID: "out", ProjectID: "p", RunID: "r", SourceIsFullVideo: true, SourceDurationSeconds: 900, CurrentRevision: 2}))
|
||||
check(s.UpsertHumanItem(ctx, domain.HumanItemRecord{ID: "retry-item", OrganizationID: "out", ProjectID: "p", LegacyTaskID: "retry"}))
|
||||
check(s.UpsertAnnotationRevision(ctx, domain.AnnotationRevisionRecord{ID: "retry-rev", OrganizationID: "out", TaskID: "retry", Revision: 2, AuthorUserID: "u", CountsAsOutput: true}))
|
||||
check(s.UpsertAnnotationWorklogEvent(ctx, domain.AnnotationWorklogEventRecord{ID: "retry-submit", OrganizationID: "out", ProjectID: "p", TaskID: "retry", UserID: "u", SupplierID: "v", Revision: 2, EventType: "combined_submitted", CreatedAt: "2026-10-04T01:00:00Z"}))
|
||||
// Missing direct task/source references are legitimate legacy plans: their
|
||||
// effect IDs and linked human events still have to route the status fence.
|
||||
op := domain.ReviewOperation{ID: "pending-accept", OperationID: "client-accept", OrganizationID: "out", ProjectID: "p", StageID: "prepackage_acceptance", Status: "applying", Decision: "approved"}
|
||||
if ok, e := s.CompareAndSwapReviewOperation(ctx, op, 0); e != nil || !ok {
|
||||
t.Fatalf("seed operation %t %v", ok, e)
|
||||
}
|
||||
check(s.AppendHumanEvent(ctx, domain.HumanEventRecord{ID: "linked-both", OrganizationID: "out", ItemID: "i", StageID: "prepackage_acceptance", EventType: "approved", CreatedAt: "2026-10-05T00:00:00Z", Payload: map[string]any{"operationId": "client-accept", "sourceItemIds": []string{"i", "i2"}}}))
|
||||
// A legacy event without a coordinator ID is unrelated to the first part's
|
||||
// modern acceptance operation, whose optional client operation ID is empty.
|
||||
check(s.AppendHumanEvent(ctx, domain.HumanEventRecord{ID: "legacy-second", OrganizationID: "out", ItemID: "i2", StageID: "final_acceptance", EventType: "approved", CreatedAt: "2026-10-04T00:00:00Z", Payload: map[string]any{"sourceItemIds": []string{"i2"}}}))
|
||||
at := time.Date(2026, 10, 20, 0, 0, 0, 0, time.UTC)
|
||||
for _, state := range []string{"applying", "completed"} {
|
||||
t.Run(state, func(t *testing.T) {
|
||||
_, e := db.Collection("reviewOperations").UpdateOne(ctx, bson.M{"_id": "pending-accept"}, bson.M{"$set": bson.M{"status": state, "sourceItemIds": []string{"i", "i2"}, "completedAt": "2026-10-05T00:00:00Z"}})
|
||||
check(e)
|
||||
raw, e := s.LoadAnnotationOutput(ctx, "out", []string{"p"})
|
||||
check(e)
|
||||
want := workforcestats.BuildOutputDataset(raw, at)
|
||||
got, stats := collectStreamedOutput(t, s, at)
|
||||
if stats.Partitions < 2 {
|
||||
t.Fatal("fixture did not span partitions")
|
||||
}
|
||||
for _, scope := range []struct{ user, vendor string }{{}, {user: "u"}, {vendor: "v"}, {user: "foreign"}, {vendor: "foreign"}} {
|
||||
for _, production := range []bool{false, true} {
|
||||
for _, cohort := range []bool{false, true} {
|
||||
q := workforcestats.OutputQuery{From: time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC), To: at, Now: at, Location: time.UTC, ProductionQuality: production, FirstSubmissionCohort: cohort, MatchUser: scope.user != "", ScopeUserID: scope.user, MatchSupplier: scope.vendor != "", ScopeSupplierID: scope.vendor}
|
||||
a, b := workforcestats.QueryOutputDataset(got, q), workforcestats.QueryOutputDataset(want, q)
|
||||
if !reflect.DeepEqual(a, b) {
|
||||
t.Fatalf("stream changed report production=%t cohort=%t scope=%+v\ngot=%+v\nwant=%+v", production, cohort, scope, a, b)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
func TestAnnotationOutputStreamCancellationAndEmptyScope(t *testing.T) {
|
||||
s := NewMongoStore(localOutputMongo(t))
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
cancel()
|
||||
if _, e := s.StreamAnnotationOutput(ctx, "out", []string{"p"}, func(workforcestats.OutputSnapshot) error { return nil }); e == nil {
|
||||
t.Fatal("cancelled build succeeded")
|
||||
}
|
||||
if stats, e := s.StreamAnnotationOutput(t.Context(), "out", nil, func(workforcestats.OutputSnapshot) error { t.Fatal("empty scope emitted data"); return nil }); e != nil || stats.Documents != 0 {
|
||||
t.Fatalf("empty scope %v %v", stats, e)
|
||||
}
|
||||
}
|
||||
@ -5,6 +5,7 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
@ -163,42 +164,56 @@ func (s *MongoStore) loadProductionStats(ctx context.Context, org string, projec
|
||||
return d, nil
|
||||
}
|
||||
filter := organizationScopedFilter(org, bson.M{"projectId": bson.M{"$in": projects}})
|
||||
read := func(c *mongo.Collection, f bson.M, projection bson.M, target any) error {
|
||||
opts := options.Find().SetProjection(projection).SetMaxTime(15 * time.Second)
|
||||
if c.Name() == "captureRecords" {
|
||||
indexes, e := c.Indexes().List(ctx)
|
||||
if e == nil {
|
||||
defer indexes.Close(ctx)
|
||||
for indexes.Next(ctx) {
|
||||
var index struct {
|
||||
Name string `bson:"name"`
|
||||
}
|
||||
if e = indexes.Decode(&index); e != nil {
|
||||
return e
|
||||
}
|
||||
if index.Name == "production_capture_cover_v1" {
|
||||
opts.SetHint(index.Name)
|
||||
break
|
||||
}
|
||||
}
|
||||
if e = indexes.Err(); e != nil {
|
||||
return e
|
||||
}
|
||||
} else {
|
||||
var command mongo.CommandError
|
||||
if !errors.As(e, &command) || command.Code != 26 {
|
||||
return e
|
||||
}
|
||||
|
||||
hints := map[string]string{}
|
||||
for collection, wanted := range map[*mongo.Collection]string{s.captureRecords: "production_capture_cover_v1", s.runs: productionRunsCoverIndex, s.humanEvents: productionEventFactsIndex} {
|
||||
indexes, e := collection.Indexes().List(ctx)
|
||||
if e != nil {
|
||||
var command mongo.CommandError
|
||||
if errors.As(e, &command) && command.Code == 26 {
|
||||
continue
|
||||
}
|
||||
return d, e
|
||||
}
|
||||
for indexes.Next(ctx) {
|
||||
var index struct {
|
||||
Name string `bson:"name"`
|
||||
}
|
||||
if e = indexes.Decode(&index); e != nil {
|
||||
indexes.Close(ctx)
|
||||
return d, e
|
||||
}
|
||||
if index.Name == wanted {
|
||||
hints[collection.Name()] = wanted
|
||||
}
|
||||
}
|
||||
e = indexes.Err()
|
||||
indexes.Close(ctx)
|
||||
if e != nil {
|
||||
return d, e
|
||||
}
|
||||
}
|
||||
read := func(c *mongo.Collection, f bson.M, projection bson.M, target any, preferred ...string) error {
|
||||
opts := options.Find().SetProjection(projection).SetMaxTime(15 * time.Second)
|
||||
hint := hints[c.Name()]
|
||||
// A project-leading covering index is unsuitable for legacy task-ID lookups.
|
||||
if c.Name() == "runs" && len(preferred) == 0 {
|
||||
hint = ""
|
||||
}
|
||||
if hint != "" {
|
||||
opts.SetHint(hint)
|
||||
}
|
||||
cursor, err := c.Find(ctx, f, opts)
|
||||
if err != nil {
|
||||
return fmt.Errorf("production stats %s: %w", c.Name(), err)
|
||||
}
|
||||
defer cursor.Close(ctx)
|
||||
if err := cursor.All(ctx, target); err != nil {
|
||||
value := reflect.ValueOf(target).Elem()
|
||||
batch := reflect.New(value.Type())
|
||||
if err := cursor.All(ctx, batch.Interface()); err != nil {
|
||||
return fmt.Errorf("production stats %s: %w", c.Name(), err)
|
||||
}
|
||||
value.Set(reflect.AppendSlice(value, batch.Elem()))
|
||||
return nil
|
||||
}
|
||||
fields := func(names ...string) bson.M {
|
||||
@ -219,10 +234,51 @@ func (s *MongoStore) loadProductionStats(ctx context.Context, org string, projec
|
||||
runIDs = append(runIDs, c.RunID)
|
||||
}
|
||||
}
|
||||
runFilter := organizationScopedFilter(org, bson.M{"$or": bson.A{bson.M{"projectId": bson.M{"$in": projects}}, bson.M{"taskId": bson.M{"$in": captureIDs}}, bson.M{"id": bson.M{"$in": runIDs}}}})
|
||||
// Bound each related-key interval set before Mongo query planning. A single
|
||||
// OR containing every capture/run ID can take the entire request budget.
|
||||
readRelated := func(c *mongo.Collection, key string, ids []string, projection bson.M, out any, extra bson.M) error {
|
||||
unique := make([]string, 0, len(ids))
|
||||
seen := map[string]bool{}
|
||||
for _, id := range ids {
|
||||
if !seen[id] {
|
||||
seen[id] = true
|
||||
unique = append(unique, id)
|
||||
}
|
||||
}
|
||||
for start := 0; start < len(unique); start += 256 {
|
||||
end := min(start+256, len(unique))
|
||||
f := organizationScopedFilter(org, bson.M{key: bson.M{"$in": unique[start:end]}})
|
||||
if len(extra) > 0 {
|
||||
f = bson.M{"$and": bson.A{f, extra}}
|
||||
}
|
||||
if err := read(c, f, projection, out); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
var g errgroup.Group
|
||||
g.Go(func() error {
|
||||
return read(s.runs, runFilter, fields("id", "projectId", "workflowId", "taskId", "trigger", "mode", "parentRunId", "createdAt", "endedAt", "status", "deliveryOutcome", "failure", "error"), &d.Runs)
|
||||
projection := fields("id", "projectId", "workflowId", "taskId", "trigger", "mode", "parentRunId", "createdAt", "endedAt", "status", "deliveryOutcome", "failure", "error")
|
||||
if err := read(s.runs, filter, projection, &d.Runs, productionRunsCoverIndex); err != nil {
|
||||
return err
|
||||
}
|
||||
// Disjoint union: normal project runs, legacy capture links outside
|
||||
// those projects, then explicitly referenced runs still missing.
|
||||
if err := readRelated(s.runs, "taskId", captureIDs, projection, &d.Runs, bson.M{"projectId": bson.M{"$nin": projects}}); err != nil {
|
||||
return err
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, run := range d.Runs {
|
||||
seen[run.ID] = true
|
||||
}
|
||||
missing := make([]string, 0, len(runIDs))
|
||||
for _, id := range runIDs {
|
||||
if !seen[id] {
|
||||
missing = append(missing, id)
|
||||
}
|
||||
}
|
||||
return readRelated(s.runs, "id", missing, projection, &d.Runs, nil)
|
||||
})
|
||||
g.Go(func() error {
|
||||
return read(s.humanItems, filter, fields("id", "projectId", "supplierId", "runId", "batchId", "legacyTaskId", "currentStageId", "status", "pendingCombinedSubmissionId", "releasePartitionId", "processingJob.status"), &d.Items)
|
||||
@ -244,15 +300,20 @@ func (s *MongoStore) loadProductionStats(ctx context.Context, org string, projec
|
||||
for _, item := range d.Items {
|
||||
ids = append(ids, item.ID)
|
||||
}
|
||||
if len(ids) > 0 {
|
||||
if err := read(s.humanEvents, organizationScopedFilter(org, bson.M{
|
||||
"itemId": bson.M{"$in": ids},
|
||||
// Decisions drive milestones. Historical package linkage may also
|
||||
// be recorded on created/reopened events, so keep those links.
|
||||
"$or": bson.A{bson.M{"eventType": bson.M{"$in": bson.A{"approved", "rejected"}}}, bson.M{"payload.deliveryPackageId": bson.M{"$exists": true}}},
|
||||
}), fields("id", "itemId", "stageId", "eventType", "createdAt", "payload.sourceItemIds", "payload.deliveryPackageId"), &d.Events); err != nil {
|
||||
|
||||
projection := fields("id", "itemId", "stageId", "eventType", "createdAt", "payload.sourceItemIds", "payload.deliveryPackageId")
|
||||
if hints["humanEvents"] != "" {
|
||||
// Keep both branches disjoint. An OR equal to a partial-index predicate can
|
||||
// make Mongo scan the whole partial index rather than bound each item range.
|
||||
if err := readRelated(s.humanEvents, "itemId", ids, projection, &d.Events, bson.M{"eventType": bson.M{"$in": bson.A{"approved", "rejected"}}}); err != nil {
|
||||
return d, err
|
||||
}
|
||||
if err := readRelated(s.humanEvents, "itemId", ids, projection, &d.Events, bson.M{"eventType": bson.M{"$nin": bson.A{"approved", "rejected"}}, "payload.deliveryPackageId": bson.M{"$exists": true}}); err != nil {
|
||||
return d, err
|
||||
}
|
||||
} else if err := readRelated(s.humanEvents, "itemId", ids, projection, &d.Events, productionEventFactsFilter()); err != nil {
|
||||
return d, err
|
||||
}
|
||||
|
||||
return d, nil
|
||||
}
|
||||
|
||||
131
internal/store/production_stats_bounded_read_test.go
Normal file
131
internal/store/production_stats_bounded_read_test.go
Normal file
@ -0,0 +1,131 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"reflect"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/event"
|
||||
"go.mongodb.org/mongo-driver/mongo"
|
||||
"go.mongodb.org/mongo-driver/mongo/options"
|
||||
)
|
||||
|
||||
func TestProductionOverviewBoundsRelatedQueriesAndKeepsLegacyUnion(t *testing.T) {
|
||||
if os.Getenv("TEST_WORKFORCE_MONGO_URI") != "mongodb://127.0.0.1:27018" {
|
||||
t.Skip("isolated loopback Mongo required")
|
||||
}
|
||||
var mu sync.Mutex
|
||||
largest := 0
|
||||
var visit func(any)
|
||||
visit = func(v any) {
|
||||
switch x := v.(type) {
|
||||
case bson.M:
|
||||
for k, v := range x {
|
||||
if k == "$in" {
|
||||
if a, ok := v.(bson.A); ok && len(a) > largest {
|
||||
largest = len(a)
|
||||
}
|
||||
}
|
||||
visit(v)
|
||||
}
|
||||
case bson.A:
|
||||
for _, v := range x {
|
||||
visit(v)
|
||||
}
|
||||
}
|
||||
}
|
||||
client, err := mongo.Connect(t.Context(), options.Client().ApplyURI("mongodb://127.0.0.1:27018").SetMonitor(&event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
|
||||
if e.CommandName != "find" {
|
||||
return
|
||||
}
|
||||
var m bson.M
|
||||
if bson.Unmarshal(e.Command, &m) == nil {
|
||||
mu.Lock()
|
||||
visit(m)
|
||||
mu.Unlock()
|
||||
}
|
||||
}}))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db := client.Database(fmt.Sprintf("production_bounded_%d", time.Now().UnixNano()))
|
||||
t.Cleanup(func() { _ = db.Drop(context.Background()); _ = client.Disconnect(context.Background()) })
|
||||
n := 600
|
||||
if os.Getenv("RUN_OUTPUT_HISTORY_SCALE") == "1" {
|
||||
n = 24000
|
||||
}
|
||||
for name, keys := range map[string]bson.D{"captureRecords": {{Key: "organizationId", Value: 1}, {Key: "projectId", Value: 1}}, "humanItems": {{Key: "organizationId", Value: 1}, {Key: "projectId", Value: 1}}, "humanEvents": {{Key: "organizationId", Value: 1}, {Key: "itemId", Value: 1}}} {
|
||||
if _, err = db.Collection(name).Indexes().CreateOne(t.Context(), mongo.IndexModel{Keys: keys}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for _, keys := range []bson.D{{{Key: "organizationId", Value: 1}, {Key: "projectId", Value: 1}}, {{Key: "organizationId", Value: 1}, {Key: "taskId", Value: 1}}, {{Key: "id", Value: 1}}} {
|
||||
if _, err = db.Collection("runs").Indexes().CreateOne(t.Context(), mongo.IndexModel{Keys: keys}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for _, col := range []string{"captureRecords", "runs", "humanItems", "humanEvents"} {
|
||||
docs := make([]any, 0, n)
|
||||
for i := 0; i < n; i++ {
|
||||
d := bson.M{"id": fmt.Sprintf("%s-%d", col, i), "organizationId": "o", "projectId": "p"}
|
||||
switch col {
|
||||
case "captureRecords":
|
||||
d["id"] = fmt.Sprint("c", i)
|
||||
d["runId"] = fmt.Sprint("r", i)
|
||||
case "runs":
|
||||
d["id"] = fmt.Sprint("r", i)
|
||||
d["taskId"] = fmt.Sprint("c", i)
|
||||
case "humanItems":
|
||||
d["id"] = fmt.Sprint("i", i)
|
||||
case "humanEvents":
|
||||
d["itemId"] = fmt.Sprint("i", i)
|
||||
d["stageId"] = "acceptance"
|
||||
d["eventType"] = "approved"
|
||||
delete(d, "projectId")
|
||||
}
|
||||
docs = append(docs, d)
|
||||
}
|
||||
if _, err = db.Collection(col).InsertMany(t.Context(), docs); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
_, err = db.Collection("runs").UpdateOne(t.Context(), bson.M{"id": "r0"}, bson.M{"$set": bson.M{"projectId": "legacy", "taskId": "not-the-capture"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Collection("runs").InsertMany(t.Context(), []any{bson.M{"id": "legacy-capture", "organizationId": "o", "taskId": "c0"}, bson.M{"id": "foreign-run", "organizationId": "other", "taskId": "c0"}, bson.M{"id": "unrelated", "organizationId": "o", "projectId": "other", "taskId": "elsewhere"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = db.Collection("humanEvents").InsertMany(t.Context(), []any{bson.M{"id": "legacy-link", "organizationId": "o", "itemId": "i0", "eventType": "created", "payload": bson.M{"deliveryPackageId": "pkg"}}, bson.M{"id": "foreign-event", "organizationId": "other", "itemId": "i0", "eventType": "approved"}, bson.M{"id": "edit-noise", "organizationId": "o", "itemId": "i0", "eventType": "draft_saved"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
started := time.Now()
|
||||
data, err := NewMongoStore(db).LoadProductionStats(t.Context(), "o", []string{"p"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(data.Runs) != n+1 || len(data.Events) != n+1 {
|
||||
t.Fatalf("legacy union/scope changed: runs=%d events=%d", len(data.Runs), len(data.Events))
|
||||
}
|
||||
ids := map[string]bool{}
|
||||
for _, r := range data.Runs {
|
||||
if ids[r.ID] {
|
||||
t.Fatal("duplicate run", r.ID)
|
||||
}
|
||||
ids[r.ID] = true
|
||||
}
|
||||
if !reflect.DeepEqual([]bool{ids["r0"], ids["legacy-capture"], ids["foreign-run"], ids["unrelated"]}, []bool{true, true, false, false}) {
|
||||
t.Fatal("run linkage/tenant boundary changed")
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if largest > 256 {
|
||||
t.Fatalf("unbounded overview related query: largest $in=%d, maximum=256", largest)
|
||||
}
|
||||
t.Logf("records_per_source=%d runs=%d events=%d max_in=%d read=%s", n, len(data.Runs), len(data.Events), largest, time.Since(started))
|
||||
}
|
||||
22
internal/store/production_stats_indexes.go
Normal file
22
internal/store/production_stats_indexes.go
Normal file
@ -0,0 +1,22 @@
|
||||
package store
|
||||
|
||||
import "go.mongodb.org/mongo-driver/bson"
|
||||
|
||||
// Opt-in production indexes. No startup creation or migration: rollout requires
|
||||
// separately approved index construction and a measured read/write comparison.
|
||||
const productionEventFactsIndex = "production_event_facts_v1"
|
||||
const productionRunsCoverIndex = "production_runs_cover_v1"
|
||||
|
||||
func productionEventFactsFilter() bson.M {
|
||||
return bson.M{"$or": bson.A{bson.M{"eventType": bson.M{"$in": bson.A{"approved", "rejected"}}}, bson.M{"payload.deliveryPackageId": bson.M{"$exists": true}}}}
|
||||
}
|
||||
func productionRunsCoverKeys() bson.D {
|
||||
keys := bson.D{}
|
||||
for _, name := range []string{"organizationId", "projectId", "id", "workflowId", "taskId", "trigger", "mode", "parentRunId", "createdAt", "endedAt", "status", "deliveryOutcome", "failure", "error"} {
|
||||
keys = append(keys, bson.E{Key: name, Value: 1})
|
||||
}
|
||||
return keys
|
||||
}
|
||||
func productionEventFactsKeys() bson.D {
|
||||
return bson.D{{Key: "organizationId", Value: 1}, {Key: "itemId", Value: 1}}
|
||||
}
|
||||
198
internal/store/production_stats_indexes_test.go
Normal file
198
internal/store/production_stats_indexes_test.go
Normal file
@ -0,0 +1,198 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/mongo"
|
||||
"go.mongodb.org/mongo-driver/mongo/options"
|
||||
"os"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestProductionFactsIndexesAvoidLargeDocumentsAndKeepLegacyLinks(t *testing.T) {
|
||||
db := localOutputMongo(t)
|
||||
ctx := t.Context()
|
||||
s := NewMongoStore(db)
|
||||
check := func(err error) {
|
||||
t.Helper()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
_, err := db.Collection("runs").Indexes().CreateOne(ctx, mongo.IndexModel{Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "projectId", Value: 1}}})
|
||||
check(err)
|
||||
_, err = db.Collection("humanEvents").Indexes().CreateOne(ctx, mongo.IndexModel{Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "itemId", Value: 1}, {Key: "stageId", Value: 1}}})
|
||||
check(err)
|
||||
n := 600
|
||||
if os.Getenv("RUN_PRODUCTION_STATS_INDEX_SCALE") == "1" {
|
||||
n = 24000
|
||||
}
|
||||
for _, name := range []string{"captureRecords", "runs", "humanItems", "humanEvents"} {
|
||||
rows := []any{}
|
||||
for i := 0; i < n; i++ {
|
||||
id := fmt.Sprint("i", i)
|
||||
r := bson.M{"id": id, "organizationId": "o", "projectId": "p"}
|
||||
switch name {
|
||||
case "captureRecords":
|
||||
r["runId"] = id
|
||||
r["prefix"] = "source/" + id + "/part0001"
|
||||
case "runs":
|
||||
r["taskId"] = id
|
||||
r["logs"] = strings.Repeat("large unneeded run log", 1000)
|
||||
r["failure"] = bson.M{"message": "invalid PTS", "code": "source"}
|
||||
case "humanEvents":
|
||||
r["itemId"] = id
|
||||
r["stageId"] = "acceptance"
|
||||
r["eventType"] = "approved"
|
||||
delete(r, "projectId")
|
||||
}
|
||||
rows = append(rows, r)
|
||||
if name == "humanEvents" {
|
||||
for j := 0; j < 20; j++ {
|
||||
rows = append(rows, bson.M{"id": fmt.Sprintf("edit-%d-%d", i, j), "organizationId": "o", "itemId": id, "stageId": "annotation", "eventType": "description_confirmed", "payload": bson.M{"draft": strings.Repeat("x", 1024)}})
|
||||
}
|
||||
}
|
||||
if len(rows) >= 1000 {
|
||||
_, err = db.Collection(name).InsertMany(ctx, rows)
|
||||
check(err)
|
||||
rows = nil
|
||||
}
|
||||
}
|
||||
if len(rows) > 0 {
|
||||
_, err = db.Collection(name).InsertMany(ctx, rows)
|
||||
check(err)
|
||||
}
|
||||
}
|
||||
_, err = db.Collection("humanEvents").InsertOne(ctx, bson.M{"id": "legacy", "organizationId": "o", "itemId": "i0", "eventType": "created", "payload": bson.M{"deliveryPackageId": "old-package"}})
|
||||
check(err)
|
||||
started := time.Now()
|
||||
before, err := s.LoadProductionStats(ctx, "o", []string{"p"})
|
||||
check(err)
|
||||
beforeTime := time.Since(started)
|
||||
_, err = db.Collection("runs").Indexes().CreateOne(ctx, mongo.IndexModel{Keys: productionRunsCoverKeys(), Options: options.Index().SetName(productionRunsCoverIndex)})
|
||||
check(err)
|
||||
_, err = db.Collection("humanEvents").Indexes().CreateOne(ctx, mongo.IndexModel{Keys: productionEventFactsKeys(), Options: options.Index().SetName(productionEventFactsIndex).SetPartialFilterExpression(productionEventFactsFilter())})
|
||||
check(err)
|
||||
started = time.Now()
|
||||
after, err := s.LoadProductionStats(ctx, "o", []string{"p"})
|
||||
check(err)
|
||||
afterTime := time.Since(started)
|
||||
before.AsOf, after.AsOf = "", ""
|
||||
sort.Slice(before.Runs, func(i, j int) bool { return before.Runs[i].ID < before.Runs[j].ID })
|
||||
sort.Slice(after.Runs, func(i, j int) bool { return after.Runs[i].ID < after.Runs[j].ID })
|
||||
sort.Slice(before.Events, func(i, j int) bool { return before.Events[i].ID < before.Events[j].ID })
|
||||
sort.Slice(after.Events, func(i, j int) bool { return after.Events[i].ID < after.Events[j].ID })
|
||||
if !reflect.DeepEqual(before, after) || len(after.Events) != n+1 {
|
||||
t.Fatal("index changed complete history/legacy links")
|
||||
}
|
||||
for _, name := range []string{"runs", "humanEvents"} {
|
||||
filter := bson.M{"organizationId": "o", "projectId": "p"}
|
||||
hint := productionRunsCoverIndex
|
||||
projection := bson.M{"_id": 0}
|
||||
for _, key := range productionRunsCoverKeys() {
|
||||
if key.Key != "organizationId" {
|
||||
projection[key.Key] = 1
|
||||
}
|
||||
}
|
||||
if name == "humanEvents" {
|
||||
hint = productionEventFactsIndex
|
||||
filter = bson.M{"$and": bson.A{bson.M{"organizationId": "o", "itemId": bson.M{"$in": bson.A{"i0", "i1"}}}, bson.M{"eventType": bson.M{"$in": bson.A{"approved", "rejected"}}}}}
|
||||
projection = bson.M{"_id": 0, "id": 1, "payload.deliveryPackageId": 1}
|
||||
}
|
||||
var result struct {
|
||||
Stats struct {
|
||||
Docs int `bson:"totalDocsExamined"`
|
||||
Keys int `bson:"totalKeysExamined"`
|
||||
Returned int `bson:"nReturned"`
|
||||
} `bson:"executionStats"`
|
||||
}
|
||||
check(db.RunCommand(ctx, bson.D{{Key: "explain", Value: bson.D{{Key: "find", Value: name}, {Key: "filter", Value: filter}, {Key: "projection", Value: projection}, {Key: "hint", Value: hint}}}, {Key: "verbosity", Value: "executionStats"}}).Decode(&result))
|
||||
if name == "runs" && result.Stats.Docs != 0 {
|
||||
t.Fatalf("covering run query fetched large documents: %+v", result.Stats)
|
||||
}
|
||||
if name == "humanEvents" && result.Stats.Docs > 3 {
|
||||
t.Fatalf("editing history still scanned: %+v", result.Stats)
|
||||
}
|
||||
t.Logf("collection=%s docs=%d keys=%d returned=%d", name, result.Stats.Docs, result.Stats.Keys, result.Stats.Returned)
|
||||
}
|
||||
t.Logf("same fixture read fallback=%s indexed=%s; records preserved=%d", beforeTime, afterTime, len(after.Events))
|
||||
if n > 600 {
|
||||
samples := []time.Duration{afterTime}
|
||||
for range 4 {
|
||||
started = time.Now()
|
||||
v, e := s.LoadProductionStats(ctx, "o", []string{"p"})
|
||||
check(e)
|
||||
if len(v.Runs) != n || len(v.Events) != n+1 {
|
||||
t.Fatal("incomplete benchmark read")
|
||||
}
|
||||
samples = append(samples, time.Since(started))
|
||||
}
|
||||
sort.Slice(samples, func(i, j int) bool { return samples[i] < samples[j] })
|
||||
t.Logf("indexed complete reads=%v max=%s", samples, samples[len(samples)-1])
|
||||
if samples[len(samples)-1] > 2*time.Second {
|
||||
t.Fatal("indexed representative read exceeds 2 seconds")
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
func TestProductionIndexManifestMatchesVerifiedDefinitions(t *testing.T) {
|
||||
raw, err := os.ReadFile("../../docs/operations/production-overview-indexes.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var manifest struct {
|
||||
Database string `json:"database"`
|
||||
Indexes []struct {
|
||||
Collection, Name string
|
||||
Keys [][]any
|
||||
Partial json.RawMessage `json:"partialFilterExpression"`
|
||||
}
|
||||
}
|
||||
if err = json.Unmarshal(raw, &manifest); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if manifest.Database != "drobodata-fanyu" || len(manifest.Indexes) != 2 {
|
||||
t.Fatal("unexpected index scope")
|
||||
}
|
||||
for _, index := range manifest.Indexes {
|
||||
var keys bson.D
|
||||
switch index.Collection {
|
||||
case "runs":
|
||||
if index.Name != productionRunsCoverIndex {
|
||||
t.Fatal("run name mismatch")
|
||||
}
|
||||
keys = productionRunsCoverKeys()
|
||||
case "humanEvents":
|
||||
if index.Name != productionEventFactsIndex {
|
||||
t.Fatal("event name mismatch")
|
||||
}
|
||||
keys = productionEventFactsKeys()
|
||||
var actual map[string]any
|
||||
if err = json.Unmarshal(index.Partial, &actual); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a, _ := json.Marshal(actual)
|
||||
b, _ := json.Marshal(productionEventFactsFilter())
|
||||
if string(a) != string(b) {
|
||||
t.Fatal("partial predicate drift")
|
||||
}
|
||||
default:
|
||||
t.Fatal("unexpected collection")
|
||||
}
|
||||
expected := [][]any{}
|
||||
for _, key := range keys {
|
||||
expected = append(expected, []any{key.Key, key.Value})
|
||||
}
|
||||
a, _ := json.Marshal(expected)
|
||||
b, _ := json.Marshal(index.Keys)
|
||||
if string(a) != string(b) {
|
||||
t.Fatal("ordered index fields drift")
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -144,7 +144,9 @@ func buildOutputParts(d OutputSnapshot, q OutputQuery) ([]outputPart, []OutputWo
|
||||
worklogOps := map[string]domain.ReviewOperation{}
|
||||
for _, o := range d.Operations {
|
||||
operations[o.ID] = o
|
||||
operations[o.OperationID] = o
|
||||
if o.OperationID != "" {
|
||||
operations[o.OperationID] = o
|
||||
}
|
||||
for _, e := range o.Effects {
|
||||
if e.Review != nil {
|
||||
reviewOps[e.Review.ID] = o
|
||||
@ -171,6 +173,9 @@ func buildOutputParts(d OutputSnapshot, q OutputQuery) ([]outputPart, []OutputWo
|
||||
video = path.Base(path.Dir(strings.TrimRight(c.Prefix, "/"))) + "/" + path.Base(strings.TrimRight(c.Prefix, "/"))
|
||||
}
|
||||
}
|
||||
if identity, ok := d.Identities[t.ID]; ok {
|
||||
key, video = identity.ID, identity.Video
|
||||
}
|
||||
attempt := t.RunID
|
||||
if attempt == "" {
|
||||
attempt = t.ID
|
||||
@ -293,7 +298,9 @@ func buildOutputParts(d OutputSnapshot, q OutputQuery) ([]outputPart, []OutputWo
|
||||
continue
|
||||
}
|
||||
covered[o.ID] = true
|
||||
covered[o.OperationID] = true
|
||||
if o.OperationID != "" {
|
||||
covered[o.OperationID] = true
|
||||
}
|
||||
if o.Status == "completed" {
|
||||
addAcceptance(o.ID, o.CompletedAt, o.Decision, o.UserID, o.ReviewerSupplierID, o.SourceItemIDs)
|
||||
}
|
||||
@ -303,10 +310,10 @@ func buildOutputParts(d OutputSnapshot, q OutputQuery) ([]outputPart, []OutputWo
|
||||
continue
|
||||
}
|
||||
opID, _ := e.Payload["operationId"].(string)
|
||||
if covered[opID] {
|
||||
if opID != "" && covered[opID] {
|
||||
continue
|
||||
}
|
||||
if o, ok := operations[opID]; ok && o.Status != "completed" {
|
||||
if o, ok := operations[opID]; opID != "" && ok && o.Status != "completed" {
|
||||
continue
|
||||
}
|
||||
addAcceptance(e.ID, e.CreatedAt, e.EventType, e.ActorUserID, "", outputIDs(e.Payload["sourceItemIds"]))
|
||||
@ -755,6 +762,10 @@ func (a *outputReviewAccumulator) report(suppliers map[string]string) OutputRevi
|
||||
}
|
||||
|
||||
func outputSummarize(parts []outputPart, work []OutputWorkFact, q OutputQuery, projects, users, suppliers map[string]string) OutputReport {
|
||||
out, _ := outputSummarizeWithMeta(parts, work, q, projects, users, suppliers)
|
||||
return out
|
||||
}
|
||||
func outputSummarizeWithMeta(parts []outputPart, work []OutputWorkFact, q OutputQuery, projects, users, suppliers map[string]string) (OutputReport, *outputMergeMeta) {
|
||||
out := OutputReport{Tasks: []OutputTask{}, Suppliers: []OutputSupplier{}, Days: []OutputDay{}, Projects: []OutputOption{}, SupplierOptions: []OutputOption{}, Users: []OutputOption{}}
|
||||
allowed := func(o outputOwner) bool {
|
||||
return (!q.MatchUser || o.user == q.ScopeUserID) && (!q.MatchSupplier || o.supplier == q.ScopeSupplierID)
|
||||
@ -952,5 +963,9 @@ func outputSummarize(parts []outputPart, work []OutputWorkFact, q OutputQuery, p
|
||||
}
|
||||
return a.SupplierID+a.UserID < b.SupplierID+b.UserID
|
||||
})
|
||||
return out
|
||||
meta := &outputMergeMeta{days: annotationDays, members: members, cumulative: map[string][2]OutputMetric{}}
|
||||
for key, c := range cumulative {
|
||||
meta.cumulative[key] = [2]OutputMetric{c[0].metric(), c[1].metric()}
|
||||
}
|
||||
return out, meta
|
||||
}
|
||||
|
||||
@ -69,10 +69,14 @@ func BuildOutputDataset(d OutputSnapshot, asOf time.Time) OutputDataset {
|
||||
return out
|
||||
}
|
||||
func QueryOutputDataset(d OutputDataset, q OutputQuery) OutputReport {
|
||||
out, _ := queryOutputDatasetWithMeta(d, q)
|
||||
return out
|
||||
}
|
||||
func queryOutputDatasetWithMeta(d OutputDataset, q OutputQuery) (OutputReport, *outputMergeMeta) {
|
||||
if q.ProductionQuality {
|
||||
q.ProductionQuality = false
|
||||
result := BuildProductionQuality(d, q)
|
||||
return OutputReport{Production: &result}
|
||||
return OutputReport{Production: &result}, nil
|
||||
}
|
||||
if q.FirstSubmissionCohort {
|
||||
d = SelectFirstSubmissionCohort(d, q.From, q.To)
|
||||
@ -103,9 +107,9 @@ func QueryOutputDataset(d OutputDataset, q OutputQuery) OutputReport {
|
||||
}
|
||||
parts = append(parts, outputPart{row, fs[0], fs[1], fs[2], fs[3]})
|
||||
}
|
||||
report := outputSummarize(parts, d.Work, q, d.Projects, d.Users, d.Suppliers)
|
||||
report, meta := outputSummarizeWithMeta(parts, d.Work, q, d.Projects, d.Users, d.Suppliers)
|
||||
if q.FirstSubmissionCohort {
|
||||
report.QualityPeople = CohortQuality(d, q)
|
||||
}
|
||||
return report
|
||||
return report, meta
|
||||
}
|
||||
|
||||
19
internal/workforcestats/output_identity.go
Normal file
19
internal/workforcestats/output_identity.go
Normal file
@ -0,0 +1,19 @@
|
||||
package workforcestats
|
||||
|
||||
import (
|
||||
"droboticflow/internal/domain"
|
||||
"path"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func OutputPartIdentityForCapture(project, task string, capture *domain.CaptureRecord) OutputPartIdentity {
|
||||
identity := OutputPartIdentity{ID: "task:" + project + ":" + task, Video: task}
|
||||
if capture != nil && capture.Prefix != "" {
|
||||
identity.ID = outputPartKey(*capture)
|
||||
identity.Video = capture.CaptureID
|
||||
if identity.Video == "" {
|
||||
identity.Video = path.Base(path.Dir(strings.TrimRight(capture.Prefix, "/"))) + "/" + path.Base(strings.TrimRight(capture.Prefix, "/"))
|
||||
}
|
||||
}
|
||||
return identity
|
||||
}
|
||||
@ -0,0 +1,41 @@
|
||||
package workforcestats
|
||||
|
||||
import (
|
||||
"droboticflow/internal/domain"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestLegacyAcceptanceWithoutOperationIDIsNotCoveredByUnrelatedPlan(t *testing.T) {
|
||||
d := outputFixture()
|
||||
c := d.Captures[0]
|
||||
c.ID = "c2"
|
||||
c.RunID = "r2"
|
||||
c.Prefix = "source/other/part0001"
|
||||
d.Captures = append(d.Captures, c)
|
||||
r := d.Runs[0]
|
||||
r.ID = "r2"
|
||||
r.TaskID = "c2"
|
||||
d.Runs = append(d.Runs, r)
|
||||
task := d.Tasks[0]
|
||||
task.ID = "t2"
|
||||
task.RunID = "r2"
|
||||
d.Tasks = append(d.Tasks, task)
|
||||
item := d.Items[0]
|
||||
item.ID = "i2"
|
||||
item.LegacyTaskID = "t2"
|
||||
d.Items = append(d.Items, item)
|
||||
event := d.Events[0]
|
||||
event.ID = "s2"
|
||||
event.TaskID = "t2"
|
||||
d.Events = append(d.Events, event)
|
||||
revision := d.Revisions[0]
|
||||
revision.TaskID = "t2"
|
||||
d.Revisions = append(d.Revisions, revision)
|
||||
d.HumanEvents = append(d.HumanEvents, domain.HumanEventRecord{ID: "legacy-acceptance", StageID: "final_acceptance", EventType: "approved", CreatedAt: "2026-10-04T00:00:00Z", Payload: map[string]any{"sourceItemIds": []string{"i2"}}})
|
||||
at := time.Date(2026, 10, 20, 0, 0, 0, 0, time.UTC)
|
||||
got := BuildOutput(d, OutputQuery{From: time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC), To: at, Now: at})
|
||||
if got.Totals.Acceptance.Count != 2 {
|
||||
t.Fatalf("unrelated empty client operation ID suppressed legacy acceptance: got=%d want=2", got.Totals.Acceptance.Count)
|
||||
}
|
||||
}
|
||||
404
internal/workforcestats/output_reduce.go
Normal file
404
internal/workforcestats/output_reduce.go
Normal file
@ -0,0 +1,404 @@
|
||||
package workforcestats
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
type outputMergeMeta struct {
|
||||
days outputDays
|
||||
members map[string]map[string]bool
|
||||
cumulative map[string][2]OutputMetric
|
||||
}
|
||||
|
||||
// OutputReducer merges disjoint canonical-part shards. It retains result rows
|
||||
// and small person/day sets, never all raw decisions or decoded datasets.
|
||||
// Ratios are merged through their evidence, never by averaging percentages.
|
||||
type OutputReducer struct {
|
||||
q OutputQuery
|
||||
out OutputReport
|
||||
days outputDays
|
||||
members map[string]map[string]bool
|
||||
cumulative map[string][2]OutputMetric
|
||||
suppliers map[string]*OutputSupplier
|
||||
daily map[string]*OutputDay
|
||||
qualityPeople map[string]*QualityPerson
|
||||
productionPeople map[string]*ProductionPerson
|
||||
productionSuppliers map[string]*ProductionSupplier
|
||||
productionQuality map[string]*QualityPerson
|
||||
comparisons [6]map[string]*QualityPerson
|
||||
options [3]map[string]string
|
||||
}
|
||||
|
||||
func NewOutputReducer(q OutputQuery) *OutputReducer {
|
||||
if q.Location == nil {
|
||||
q.Location = time.UTC
|
||||
}
|
||||
r := &OutputReducer{q: q, days: outputDays{}, members: map[string]map[string]bool{}, cumulative: map[string][2]OutputMetric{}, suppliers: map[string]*OutputSupplier{}, daily: map[string]*OutputDay{}, qualityPeople: map[string]*QualityPerson{}, productionPeople: map[string]*ProductionPerson{}, productionSuppliers: map[string]*ProductionSupplier{}, productionQuality: map[string]*QualityPerson{}}
|
||||
r.out = OutputReport{Tasks: []OutputTask{}, Suppliers: []OutputSupplier{}, Days: []OutputDay{}, Projects: []OutputOption{}, SupplierOptions: []OutputOption{}, Users: []OutputOption{}}
|
||||
for i := range r.options {
|
||||
r.options[i] = map[string]string{}
|
||||
}
|
||||
for i := range r.comparisons {
|
||||
r.comparisons[i] = map[string]*QualityPerson{}
|
||||
}
|
||||
return r
|
||||
}
|
||||
func mergeOutputMetric(a, b OutputMetric) OutputMetric {
|
||||
out := OutputMetric{Count: a.Count + b.Count}
|
||||
if (a.Count > 0 && a.Seconds == nil) || (b.Count > 0 && b.Seconds == nil) {
|
||||
return out
|
||||
}
|
||||
if a.Seconds != nil || b.Seconds != nil {
|
||||
value := float64(0)
|
||||
if a.Seconds != nil {
|
||||
value += *a.Seconds
|
||||
}
|
||||
if b.Seconds != nil {
|
||||
value += *b.Seconds
|
||||
}
|
||||
out.Seconds = &value
|
||||
}
|
||||
return out
|
||||
}
|
||||
func mergeOutputTotals(a, b OutputTotals) OutputTotals {
|
||||
return OutputTotals{mergeOutputMetric(a.Submitted, b.Submitted), mergeOutputMetric(a.Quality, b.Quality), mergeOutputMetric(a.Acceptance, b.Acceptance), mergeOutputMetric(a.Rework, b.Rework)}
|
||||
}
|
||||
func mergeQualityRatio(a *QualityRatio, b QualityRatio) {
|
||||
a.Numerator += b.Numerator
|
||||
a.Denominator += b.Denominator
|
||||
a.Unknown += b.Unknown
|
||||
a.Value = nil
|
||||
}
|
||||
func qualityIdentity(p QualityPerson) string {
|
||||
return p.Role + "\x00" + p.SupplierID + "\x00" + p.UserID
|
||||
}
|
||||
func mergeQualityRow(m map[string]*QualityPerson, key string, v QualityPerson) {
|
||||
p := m[key]
|
||||
if p == nil {
|
||||
p = &QualityPerson{UserID: v.UserID, SupplierID: v.SupplierID, Role: v.Role}
|
||||
m[key] = p
|
||||
}
|
||||
mergeQualityRatio(&p.FirstPass, v.FirstPass)
|
||||
mergeQualityRatio(&p.FirstReworkPass, v.FirstReworkPass)
|
||||
mergeQualityRatio(&p.OpenRework, v.OpenRework)
|
||||
mergeQualityRatio(&p.MeanReworks, v.MeanReworks)
|
||||
mergeQualityRatio(&p.Misreport, v.Misreport)
|
||||
}
|
||||
func finishQuality(p *QualityPerson) {
|
||||
p.FirstPass.finish(100)
|
||||
p.FirstReworkPass.finish(100)
|
||||
p.OpenRework.finish(100)
|
||||
p.MeanReworks.finish(1)
|
||||
p.Misreport.finish(100)
|
||||
}
|
||||
func mergeStringSets(dst map[string]map[string]bool, src map[string]map[string]bool) {
|
||||
for key, values := range src {
|
||||
if dst[key] == nil {
|
||||
dst[key] = map[string]bool{}
|
||||
}
|
||||
for value := range values {
|
||||
dst[key][value] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
func (r *OutputReducer) addOptions(projects, suppliers, users []OutputOption) {
|
||||
for i, values := range [][]OutputOption{projects, suppliers, users} {
|
||||
for _, v := range values {
|
||||
r.options[i][v.ID] = v.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
func outputOptions(m map[string]string) []OutputOption {
|
||||
out := []OutputOption{}
|
||||
for id, name := range m {
|
||||
out = append(out, OutputOption{ID: id, Name: name})
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].Name != out[j].Name {
|
||||
return out[i].Name < out[j].Name
|
||||
}
|
||||
return out[i].ID < out[j].ID
|
||||
})
|
||||
return out
|
||||
}
|
||||
func (r *OutputReducer) Add(d OutputDataset) {
|
||||
if r.q.ProductionQuality {
|
||||
r.addProduction(d)
|
||||
return
|
||||
}
|
||||
report, meta := queryOutputDatasetWithMeta(d, r.q)
|
||||
r.addOptions(report.Projects, report.SupplierOptions, report.Users)
|
||||
r.out.Totals = mergeOutputTotals(r.out.Totals, report.Totals)
|
||||
r.out.Tasks = append(r.out.Tasks, report.Tasks...)
|
||||
r.out.Quality = mergeOutputReview(r.out.Quality, report.Quality)
|
||||
r.out.Acceptance = mergeOutputReview(r.out.Acceptance, report.Acceptance)
|
||||
if meta != nil {
|
||||
mergeStringSets(r.days, meta.days)
|
||||
mergeStringSets(r.members, meta.members)
|
||||
for key, v := range meta.cumulative {
|
||||
old := r.cumulative[key]
|
||||
r.cumulative[key] = [2]OutputMetric{mergeOutputMetric(old[0], v[0]), mergeOutputMetric(old[1], v[1])}
|
||||
}
|
||||
}
|
||||
for _, v := range report.Suppliers {
|
||||
p := r.suppliers[v.ID]
|
||||
if p == nil {
|
||||
p = &OutputSupplier{ID: v.ID, Name: v.Name}
|
||||
r.suppliers[v.ID] = p
|
||||
}
|
||||
p.OutputTotals = mergeOutputTotals(p.OutputTotals, v.OutputTotals)
|
||||
}
|
||||
for _, v := range report.Days {
|
||||
key := v.Date + "\x00" + v.SupplierID + "\x00" + v.UserID
|
||||
p := r.daily[key]
|
||||
if p == nil {
|
||||
copy := v
|
||||
copy.Submitted = OutputMetric{}
|
||||
copy.Acceptance = OutputMetric{}
|
||||
p = ©
|
||||
r.daily[key] = p
|
||||
}
|
||||
p.Submitted = mergeOutputMetric(p.Submitted, v.Submitted)
|
||||
p.Acceptance = mergeOutputMetric(p.Acceptance, v.Acceptance)
|
||||
}
|
||||
for _, v := range report.QualityPeople {
|
||||
mergeQualityRow(r.qualityPeople, qualityIdentity(v), v)
|
||||
}
|
||||
}
|
||||
func mergeOutputReview(a, b OutputReviewReport) OutputReviewReport {
|
||||
out := OutputReviewReport{Suppliers: []OutputReviewSupplier{}, Days: []OutputReviewDay{}}
|
||||
merge := func(a, b OutputReviewMetric) OutputReviewMetric {
|
||||
return OutputReviewMetric{Reviews: mergeOutputMetric(a.Reviews, b.Reviews), Approved: a.Approved + b.Approved, Rejected: a.Rejected + b.Rejected, Rereviews: a.Rereviews + b.Rereviews}
|
||||
}
|
||||
out.Totals = merge(a.Totals, b.Totals)
|
||||
suppliers := map[string]*OutputReviewSupplier{}
|
||||
days := map[string]*OutputReviewDay{}
|
||||
for _, report := range []OutputReviewReport{a, b} {
|
||||
for _, v := range report.Suppliers {
|
||||
p := suppliers[v.ID]
|
||||
if p == nil {
|
||||
p = &OutputReviewSupplier{ID: v.ID, Name: v.Name}
|
||||
suppliers[v.ID] = p
|
||||
}
|
||||
p.OutputReviewMetric = merge(p.OutputReviewMetric, v.OutputReviewMetric)
|
||||
}
|
||||
for _, v := range report.Days {
|
||||
key := v.Date + "\x00" + v.SupplierID + "\x00" + v.UserID
|
||||
p := days[key]
|
||||
if p == nil {
|
||||
copy := v
|
||||
copy.Reviews = OutputMetric{}
|
||||
copy.Approved = 0
|
||||
copy.Rejected = 0
|
||||
copy.Rereviews = 0
|
||||
p = ©
|
||||
days[key] = p
|
||||
}
|
||||
p.Reviews = mergeOutputMetric(p.Reviews, v.Reviews)
|
||||
p.Approved += v.Approved
|
||||
p.Rejected += v.Rejected
|
||||
p.Rereviews += v.Rereviews
|
||||
}
|
||||
}
|
||||
members := map[string]map[string]bool{}
|
||||
workDays := outputDays{}
|
||||
for _, v := range days {
|
||||
out.Days = append(out.Days, *v)
|
||||
workDays.add(v.SupplierID, v.UserID, v.Date)
|
||||
if members[v.SupplierID] == nil {
|
||||
members[v.SupplierID] = map[string]bool{}
|
||||
}
|
||||
if v.UserID != "" {
|
||||
members[v.SupplierID][v.UserID] = true
|
||||
}
|
||||
}
|
||||
all := map[string]bool{}
|
||||
for id, p := range suppliers {
|
||||
p.MemberCount = len(members[id])
|
||||
for user := range members[id] {
|
||||
all[user] = true
|
||||
}
|
||||
p.Efficiency = outputEfficiency(p.Reviews, workDays.count(id))
|
||||
out.Suppliers = append(out.Suppliers, *p)
|
||||
}
|
||||
out.Totals.MemberCount = len(all)
|
||||
out.Totals.Efficiency = outputEfficiency(out.Totals.Reviews, workDays.total())
|
||||
sort.Slice(out.Suppliers, func(i, j int) bool { return out.Suppliers[i].ID < out.Suppliers[j].ID })
|
||||
sort.Slice(out.Days, func(i, j int) bool {
|
||||
a, b := out.Days[i], out.Days[j]
|
||||
if a.Date != b.Date {
|
||||
return a.Date > b.Date
|
||||
}
|
||||
return a.SupplierID+"\x00"+a.UserID < b.SupplierID+"\x00"+b.UserID
|
||||
})
|
||||
return out
|
||||
}
|
||||
func (r *OutputReducer) addProduction(d OutputDataset) {
|
||||
report := buildProductionQualityCore(d, r.q)
|
||||
r.addOptions(report.Projects, report.SupplierOptions, report.Users)
|
||||
if r.out.Production == nil {
|
||||
r.out.Production = &ProductionQualityReport{Suppliers: []ProductionSupplier{}, People: []ProductionPerson{}, Quality: []QualityPerson{}, Projects: []OutputOption{}, SupplierOptions: []OutputOption{}, Users: []OutputOption{}}
|
||||
}
|
||||
r.out.Production.Totals = mergeOutputTotals(r.out.Production.Totals, report.Totals)
|
||||
r.out.Production.Deliverable = mergeOutputMetric(r.out.Production.Deliverable, report.Deliverable)
|
||||
for _, v := range report.People {
|
||||
key := v.Role + "\x00" + v.SupplierID + "\x00" + v.ID
|
||||
p := r.productionPeople[key]
|
||||
if p == nil {
|
||||
p = &ProductionPerson{ID: v.ID, Name: v.Name, SupplierID: v.SupplierID, SupplierName: v.SupplierName, Role: v.Role}
|
||||
r.productionPeople[key] = p
|
||||
}
|
||||
p.Totals = mergeOutputTotals(p.Totals, v.Totals)
|
||||
p.Deliverable = mergeOutputMetric(p.Deliverable, v.Deliverable)
|
||||
if v.Quality != nil {
|
||||
mergeQualityRow(r.qualityPeople, key, *v.Quality)
|
||||
}
|
||||
}
|
||||
for _, v := range report.Suppliers {
|
||||
p := r.productionSuppliers[v.ID]
|
||||
if p == nil {
|
||||
p = &ProductionSupplier{ID: v.ID, Name: v.Name, Members: map[string]int{}}
|
||||
r.productionSuppliers[v.ID] = p
|
||||
}
|
||||
p.Totals = mergeOutputTotals(p.Totals, v.Totals)
|
||||
p.Deliverable = mergeOutputMetric(p.Deliverable, v.Deliverable)
|
||||
for _, q := range v.Quality {
|
||||
mergeQualityRow(r.productionQuality, "supplier\x00"+qualityIdentity(q), q)
|
||||
}
|
||||
}
|
||||
for _, q := range report.Quality {
|
||||
mergeQualityRow(r.productionQuality, "total\x00"+qualityIdentity(q), q)
|
||||
}
|
||||
now := d.AsOf.In(r.q.Location)
|
||||
today := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, r.q.Location)
|
||||
for day := 0; day < 6; day++ {
|
||||
start := today.AddDate(0, 0, day-5)
|
||||
cohort := SelectFirstSubmissionCohort(d, start, start.AddDate(0, 0, 1))
|
||||
parts := qualityWorkByPart(cohort)
|
||||
for _, group := range []string{"total", "supplier", "person"} {
|
||||
for _, q := range cohortQualityFromParts(cohort, r.q, group, parts) {
|
||||
mergeQualityRow(r.comparisons[day], group+"\x00"+qualityIdentity(q), q)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
func (r *OutputReducer) attachComparisons(p *QualityPerson, group string) {
|
||||
p.Trend = make([]QualityValues, 6)
|
||||
for day := 0; day < 6; day++ {
|
||||
if metric := r.comparisons[day][group+"\x00"+qualityIdentity(*p)]; metric != nil {
|
||||
finishQuality(metric)
|
||||
values := qualityValues(*metric)
|
||||
p.Trend[day] = values
|
||||
if day == 4 {
|
||||
copy := values
|
||||
p.Yesterday = ©
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
func (r *OutputReducer) Report() OutputReport {
|
||||
if r.q.ProductionQuality {
|
||||
return r.productionReport()
|
||||
}
|
||||
r.out.Suppliers = []OutputSupplier{}
|
||||
r.out.Days = []OutputDay{}
|
||||
r.out.Projects = outputOptions(r.options[0])
|
||||
r.out.SupplierOptions = outputOptions(r.options[1])
|
||||
r.out.Users = outputOptions(r.options[2])
|
||||
r.out.Efficiency = outputEfficiency(r.out.Totals.Submitted, r.days.total())
|
||||
for _, p := range r.suppliers {
|
||||
p.MemberCount = len(r.members[p.ID])
|
||||
p.Efficiency = outputEfficiency(p.Submitted, r.days.count(p.ID))
|
||||
r.out.Suppliers = append(r.out.Suppliers, *p)
|
||||
}
|
||||
for _, p := range r.daily {
|
||||
total := r.cumulative[p.SupplierID+"\x00"+p.UserID]
|
||||
p.CumulativeSubmitted, p.CumulativeAcceptance = total[0], total[1]
|
||||
r.out.Days = append(r.out.Days, *p)
|
||||
}
|
||||
if r.q.FirstSubmissionCohort {
|
||||
r.out.QualityPeople = []QualityPerson{}
|
||||
for _, p := range r.qualityPeople {
|
||||
finishQuality(p)
|
||||
r.out.QualityPeople = append(r.out.QualityPeople, *p)
|
||||
}
|
||||
sort.Slice(r.out.QualityPeople, func(i, j int) bool {
|
||||
return qualityIdentity(r.out.QualityPeople[i]) < qualityIdentity(r.out.QualityPeople[j])
|
||||
})
|
||||
}
|
||||
sort.Slice(r.out.Tasks, func(i, j int) bool {
|
||||
a, b := r.out.Tasks[i], r.out.Tasks[j]
|
||||
if a.SubmittedAt != b.SubmittedAt {
|
||||
return outputEarlier(b.SubmittedAt, a.SubmittedAt)
|
||||
}
|
||||
return a.ID < b.ID
|
||||
})
|
||||
sort.Slice(r.out.Suppliers, func(i, j int) bool { return r.out.Suppliers[i].ID < r.out.Suppliers[j].ID })
|
||||
sort.Slice(r.out.Days, func(i, j int) bool {
|
||||
a, b := r.out.Days[i], r.out.Days[j]
|
||||
if a.Date != b.Date {
|
||||
return a.Date > b.Date
|
||||
}
|
||||
return a.SupplierID+a.UserID < b.SupplierID+b.UserID
|
||||
})
|
||||
return r.out
|
||||
}
|
||||
func (r *OutputReducer) productionReport() OutputReport {
|
||||
if r.out.Production == nil {
|
||||
empty := OutputDataset{AsOf: r.q.Now, Projects: map[string]string{}, Users: map[string]string{}, Suppliers: map[string]string{}}
|
||||
r.addProduction(empty)
|
||||
}
|
||||
out := r.out.Production
|
||||
out.People = []ProductionPerson{}
|
||||
out.Suppliers = []ProductionSupplier{}
|
||||
out.Quality = []QualityPerson{}
|
||||
out.Projects = outputOptions(r.options[0])
|
||||
out.SupplierOptions = outputOptions(r.options[1])
|
||||
out.Users = outputOptions(r.options[2])
|
||||
members := map[string]map[string]map[string]bool{}
|
||||
for key, p := range r.productionPeople {
|
||||
if q := r.qualityPeople[key]; q != nil {
|
||||
finishQuality(q)
|
||||
r.attachComparisons(q, "person")
|
||||
p.Quality = q
|
||||
}
|
||||
out.People = append(out.People, *p)
|
||||
if members[p.SupplierID] == nil {
|
||||
members[p.SupplierID] = map[string]map[string]bool{}
|
||||
}
|
||||
if members[p.SupplierID][p.Role] == nil {
|
||||
members[p.SupplierID][p.Role] = map[string]bool{}
|
||||
}
|
||||
members[p.SupplierID][p.Role][p.ID] = true
|
||||
}
|
||||
for id, p := range r.productionSuppliers {
|
||||
p.Quality = nil
|
||||
for role, users := range members[id] {
|
||||
p.Members[role] = len(users)
|
||||
}
|
||||
for key, q := range r.productionQuality {
|
||||
if len(key) >= 9 && key[:9] == "supplier\x00" && q.SupplierID == id {
|
||||
finishQuality(q)
|
||||
r.attachComparisons(q, "supplier")
|
||||
p.Quality = append(p.Quality, *q)
|
||||
}
|
||||
}
|
||||
sort.Slice(p.Quality, func(i, j int) bool { return qualityIdentity(p.Quality[i]) < qualityIdentity(p.Quality[j]) })
|
||||
out.Suppliers = append(out.Suppliers, *p)
|
||||
}
|
||||
for key, q := range r.productionQuality {
|
||||
if len(key) >= 6 && key[:6] == "total\x00" {
|
||||
finishQuality(q)
|
||||
r.attachComparisons(q, "total")
|
||||
out.Quality = append(out.Quality, *q)
|
||||
}
|
||||
}
|
||||
sort.Slice(out.People, func(i, j int) bool {
|
||||
a, b := out.People[i], out.People[j]
|
||||
return a.Role+"\x00"+a.SupplierID+"\x00"+a.ID < b.Role+"\x00"+b.SupplierID+"\x00"+b.ID
|
||||
})
|
||||
sort.Slice(out.Suppliers, func(i, j int) bool { return out.Suppliers[i].ID < out.Suppliers[j].ID })
|
||||
sort.Slice(out.Quality, func(i, j int) bool { return qualityIdentity(out.Quality[i]) < qualityIdentity(out.Quality[j]) })
|
||||
return OutputReport{Production: out}
|
||||
}
|
||||
128
internal/workforcestats/output_reduce_test.go
Normal file
128
internal/workforcestats/output_reduce_test.go
Normal file
@ -0,0 +1,128 @@
|
||||
package workforcestats
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"math"
|
||||
"reflect"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestOutputReducerMatchesWholeDataset(t *testing.T) {
|
||||
d := BuildOutputDataset(outputReviewFixture(), outputQuery().Now)
|
||||
d.AsOf = d.AsOf.AddDate(0, 0, 3)
|
||||
// Exercise distinct parts sharing people/days, unknown durations and rework.
|
||||
originalParts, originalWork := append([]OutputPartFact(nil), d.Parts...), append([]OutputWorkFact(nil), d.Work...)
|
||||
for n := 1; n < 7; n++ {
|
||||
for _, p := range originalParts {
|
||||
p.ID += string(rune('a' + n))
|
||||
for j, stage := range p.Stages {
|
||||
if stage != nil {
|
||||
copy := *stage
|
||||
at, err := time.Parse(time.RFC3339Nano, copy.At)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
copy.At = at.AddDate(0, 0, n+12).Format(time.RFC3339Nano)
|
||||
p.Stages[j] = ©
|
||||
}
|
||||
}
|
||||
if n == 3 {
|
||||
p.Seconds = nil
|
||||
} else if p.Seconds != nil {
|
||||
seconds := *p.Seconds + float64(n)/30
|
||||
p.Seconds = &seconds
|
||||
}
|
||||
d.Parts = append(d.Parts, p)
|
||||
}
|
||||
for _, w := range originalWork {
|
||||
w.Part += n * len(originalParts)
|
||||
at, err := time.Parse(time.RFC3339Nano, w.At)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
w.At = at.AddDate(0, 0, n+12).Format(time.RFC3339Nano)
|
||||
d.Work = append(d.Work, w)
|
||||
}
|
||||
}
|
||||
for _, production := range []bool{false, true} {
|
||||
for _, cohort := range []bool{false, true} {
|
||||
for _, supplier := range []string{"", "v", "qv", "missing"} {
|
||||
for _, user := range []string{"", "a", "qc"} {
|
||||
q := outputQuery()
|
||||
q.ProductionQuality = production
|
||||
q.FirstSubmissionCohort = cohort
|
||||
q.SupplierID = supplier
|
||||
q.UserID = user
|
||||
r := NewOutputReducer(q)
|
||||
for i, p := range d.Parts {
|
||||
shard := OutputDataset{AsOf: d.AsOf, Parts: []OutputPartFact{p}, Projects: d.Projects, Users: d.Users, Suppliers: d.Suppliers}
|
||||
for _, w := range d.Work {
|
||||
if w.Part == i {
|
||||
w.Part = 0
|
||||
shard.Work = append(shard.Work, w)
|
||||
}
|
||||
}
|
||||
for _, problem := range d.Problems {
|
||||
if problem.Part == i {
|
||||
problem.Part = 0
|
||||
shard.Problems = append(shard.Problems, problem)
|
||||
}
|
||||
}
|
||||
r.Add(shard)
|
||||
}
|
||||
want := QueryOutputDataset(d, q)
|
||||
got := r.Report()
|
||||
a, _ := json.Marshal(want)
|
||||
b, _ := json.Marshal(got)
|
||||
again, _ := json.Marshal(r.Report())
|
||||
if !reflect.DeepEqual(b, again) {
|
||||
t.Fatal("Report is not idempotent")
|
||||
}
|
||||
var expectedValue, actualValue any
|
||||
_ = json.Unmarshal(a, &expectedValue)
|
||||
_ = json.Unmarshal(b, &actualValue)
|
||||
if !equalOutputReportValue(expectedValue, actualValue) {
|
||||
t.Fatalf("production=%v cohort=%v supplier=%s user=%s\nwant %s\ngot %s", production, cohort, supplier, user, a, b)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Disjoint-part merge order may differ in the final floating-point bit. The
|
||||
// tolerance is far below one frame; structure, unknowns and integer counts stay
|
||||
// exact. Percentages are still calculated from merged numerators/denominators.
|
||||
func equalOutputReportValue(a, b any) bool {
|
||||
switch x := a.(type) {
|
||||
case float64:
|
||||
y, ok := b.(float64)
|
||||
return ok && math.Abs(x-y) < 1e-7
|
||||
case map[string]any:
|
||||
y, ok := b.(map[string]any)
|
||||
if !ok || len(x) != len(y) {
|
||||
return false
|
||||
}
|
||||
for k, v := range x {
|
||||
value, ok := y[k]
|
||||
if !ok || !equalOutputReportValue(v, value) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
case []any:
|
||||
y, ok := b.([]any)
|
||||
if !ok || len(x) != len(y) {
|
||||
return false
|
||||
}
|
||||
for i, v := range x {
|
||||
if !equalOutputReportValue(v, y[i]) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
default:
|
||||
return reflect.DeepEqual(a, b)
|
||||
}
|
||||
}
|
||||
@ -8,7 +8,14 @@ import (
|
||||
|
||||
// OutputSnapshot contains reporting facts only. Loading it must never recover,
|
||||
// claim or modify a task. Mutable drafts and media payloads are not needed.
|
||||
type OutputPartIdentity struct {
|
||||
ID string `bson:"id"`
|
||||
Video string `bson:"video"`
|
||||
}
|
||||
|
||||
type OutputSnapshot struct {
|
||||
Identities map[string]OutputPartIdentity // Internal pre-resolved canonical identity for streamed batches.
|
||||
|
||||
Baselines []domain.ReworkBaselineRecord
|
||||
Handlings []domain.ReworkHandlingRecord
|
||||
SourceStats *OutputSourceStats
|
||||
@ -181,6 +188,16 @@ const OutputSourceMaxDocuments = 250000
|
||||
var ErrOutputSourceBudget = errors.New("annotation output source budget exceeded")
|
||||
|
||||
type OutputSourceStats struct {
|
||||
Bytes int `json:"bytes" bson:"bytes"`
|
||||
Documents int `json:"documents" bson:"documents"`
|
||||
Bytes int `json:"bytes" bson:"bytes"`
|
||||
Documents int `json:"documents" bson:"documents"`
|
||||
Partitions int `json:"partitions,omitempty" bson:"partitions,omitempty"`
|
||||
PeakBatchBytes int `json:"peakBatchBytes,omitempty" bson:"peakBatchBytes,omitempty"`
|
||||
PeakBatchDocuments int `json:"peakBatchDocuments,omitempty" bson:"peakBatchDocuments,omitempty"`
|
||||
DirectoryBytes int `json:"directoryBytes,omitempty" bson:"directoryBytes,omitempty"`
|
||||
PeakSpoolBytes int64 `json:"peakSpoolBytes,omitempty" bson:"peakSpoolBytes,omitempty"`
|
||||
}
|
||||
|
||||
const OutputDirectoryMaxBytes = 256 << 20
|
||||
const OutputSpoolMaxBytes = 2 << 30
|
||||
|
||||
var ErrOutputPartitionSplit = errors.New("reporting partition needs subdivision")
|
||||
|
||||
@ -35,7 +35,11 @@ type ProductionPerson struct {
|
||||
}
|
||||
|
||||
func BuildProductionQuality(d OutputDataset, q OutputQuery) ProductionQualityReport {
|
||||
original := d
|
||||
result := buildProductionQualityCore(d, q)
|
||||
attachQualityComparisons(&result, d, q)
|
||||
return result
|
||||
}
|
||||
func buildProductionQualityCore(d OutputDataset, q OutputQuery) ProductionQualityReport {
|
||||
d = SelectFirstSubmissionCohort(d, q.From, q.To)
|
||||
q.FirstSubmissionCohort = false
|
||||
q.From = time.Time{}
|
||||
@ -184,7 +188,6 @@ func BuildProductionQuality(d OutputDataset, q OutputQuery) ProductionQualityRep
|
||||
if q.MatchUser || q.UserID != "" {
|
||||
result.Totals = OutputTotals{selectedSums[0].metric(), selectedSums[1].metric(), selectedSums[2].metric(), selectedSums[3].metric()}
|
||||
}
|
||||
attachQualityComparisons(&result, original, q)
|
||||
return result
|
||||
}
|
||||
func attachQualityComparisons(result *ProductionQualityReport, d OutputDataset, q OutputQuery) {
|
||||
|
||||
@ -20,6 +20,16 @@ REQUIRED = {
|
||||
"TestHealthWarnsBeforeExpiryAndDoesNotTriggerBuild",
|
||||
"TestAnnotationOutputDatasetPreservesRulesAcrossFilters",
|
||||
"TestRouteSnapshot",
|
||||
"TestAnnotationOutputMongoShardedRestartAndFailedBuild",
|
||||
"TestAnnotationOutputStreamFailureCleansPrivateFiles",
|
||||
"TestAnnotationOutputStreamMatchesWholeHistoryAcrossLegacyReferences",
|
||||
"TestShardedGenerationMatchesLegacyAndSurvivesCorruptionAndRestart",
|
||||
"TestShardedBuildFailureNeverPublishesPartialManifest",
|
||||
"TestOutputReducerMatchesWholeDataset",
|
||||
"TestOutputSpoolCleanupRecoversCrashedBuildButKeepsActive",
|
||||
"TestProductionFactsIndexesAvoidLargeDocumentsAndKeepLegacyLinks",
|
||||
"TestProductionOverviewBoundsRelatedQueriesAndKeepsLegacyUnion",
|
||||
"TestLegacyAcceptanceWithoutOperationIDIsNotCoveredByUnrelatedPlan",
|
||||
}
|
||||
|
||||
|
||||
@ -37,11 +47,12 @@ def main():
|
||||
"./internal/workforcestats", "./internal/api", "-run",
|
||||
"TestAnnotationOutput|TestOutput|TestSharedReaders|TestRefreshFailure|TestLeaseRecovery|"
|
||||
"TestCorruptCurrent|TestBuildCancellation|TestSnapshotScope|TestFailedBuild|"
|
||||
"TestFailureDiagnostics|TestHealth|TestRouteSnapshot", "-count=1", "-timeout=120s"]
|
||||
"TestFailureDiagnostics|TestHealth|TestRouteSnapshot|TestSharded|TestProductionFactsIndexes|TestProductionOverviewBounds|TestLegacyAcceptanceWithout", "-count=1", "-timeout=120s"]
|
||||
timing = ["go", "test", "-race=false", "-json", "./internal/api", "-run", "^TestAnnotationOutputSnapshotScale$", "-count=1", "-timeout=120s"]
|
||||
scale = {"TestAnnotationOutputSnapshotScale", "TestAnnotationOutputSnapshotScale/production-quality-concurrent-warm-reads"}
|
||||
growth = ["go", "test", "-json", "./internal/store", "-run", "^TestStatisticsPartitionGrowthBeyondGlobalBudget$", "-count=1", "-timeout=120s"]
|
||||
verdicts = []
|
||||
for name, args, required in [("race", command, REQUIRED | scale), ("timing", timing, scale)]:
|
||||
for name, args, required in [("race", command, REQUIRED | scale), ("timing", timing, scale), ("partition-growth", growth, {"TestStatisticsPartitionGrowthBeyondGlobalBudget"})]:
|
||||
try:
|
||||
result = subprocess.run(args, cwd=ROOT, env=env, capture_output=True, text=True, timeout=240)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
|
||||
@ -410,6 +410,8 @@
|
||||
{
|
||||
"name": "dashboards",
|
||||
"paths": [
|
||||
"^docs/operations/production-overview-(indexes\\.json|statistics-growth\\.md)$",
|
||||
"^(scripts/test_dashboard_stability|tests/release/test_dashboard_stability)\\.py$",
|
||||
"^src/.*([Dd]ashboard|WorkforceStatistics|annotation-output)",
|
||||
"^internal/(api|store)/(production_(dashboard|stats)|annotation_output|workforce)",
|
||||
"^internal/(outputsnapshot|workforcestats)/",
|
||||
@ -435,6 +437,10 @@
|
||||
{
|
||||
"argv": ["python3", "scripts/test_dashboard_stability.py"]
|
||||
},
|
||||
{
|
||||
"argv": ["python3", "-m", "unittest", "discover", "-s", "tests/release", "-p", "test_dashboard_stability.py"],
|
||||
"tests": true
|
||||
},
|
||||
{
|
||||
"argv": [
|
||||
"go",
|
||||
|
||||
@ -44,12 +44,14 @@ class DashboardStabilityTests(unittest.TestCase):
|
||||
calls = []
|
||||
def run(command, **kwargs):
|
||||
calls.append(command)
|
||||
names = gate.REQUIRED | scale if "-race" in command else scale
|
||||
names = gate.REQUIRED | scale if "-race" in command else ({"TestStatisticsPartitionGrowthBeyondGlobalBudget"} if "^TestStatisticsPartitionGrowthBeyondGlobalBudget$" in command else scale)
|
||||
return SimpleNamespace(returncode=int(fail_timing and "-race" not in command),
|
||||
stdout="\n".join(json.dumps({"Action": "pass", "Test": name}) for name in names), stderr="")
|
||||
with patch.object(gate.subprocess, "run", side_effect=run), patch("builtins.print"):
|
||||
self.assertEqual(gate.main(), int(fail_timing))
|
||||
self.assertEqual(len(calls), 2)
|
||||
self.assertEqual(len(calls), 3)
|
||||
self.assertIn("-race", calls[0])
|
||||
self.assertNotIn("-race", calls[1])
|
||||
self.assertIn("-race=false", calls[1])
|
||||
self.assertNotIn("-race", calls[2])
|
||||
self.assertIn("^TestStatisticsPartitionGrowthBeyondGlobalBudget$", calls[2])
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user