⚡ Bound production overview queries and shard statistics history

This commit is contained in:
Codex 2026-11-10 21:39:47 -08:00
parent e99febdac1
commit 6bf0b7662d
37 changed files with 3360 additions and 87 deletions

View File

@ -31,3 +31,9 @@
## 兼容与回退
新增字段均可选。共享事实 schema v3 防止旧事实缺失归属字段。回退应用可重建对应旧 schema 快照,不要求重写业务数据。原任务流转、正式决定、标注草稿和交付内容不变;仅新增验收意见误报处理记录,旧应用忽略这些记录。
## 2026-11-10:增长与读取实现
事实 schema v5 按 canonical part 分片构建和持久化,完整清单原子发布,查询流式聚合;单个 part 的全部尝试、历史关联和未完成操作屏蔽留在同一分片。比率合并分子/分母,人员天数及跨片成员去重,当前权限逐请求执行。旧 v4 应用按自身 schema 重建,业务数据不迁移。空客户端操作 ID 不再屏蔽无关历史验收。
容量与生产总览索引的预算、验证证据、兼容和待授权步骤见 [运行说明](../operations/production-overview-statistics-growth.md)。生产仍待端到端验收,不以本地规模测试替代线上容量结论。

View File

@ -0,0 +1,24 @@
{
"database": "drobodata-fanyu",
"minimumMongoVersion": "6.0",
"indexes": [
{
"collection": "runs",
"name": "production_runs_cover_v1",
"unique": false,
"keys": [["organizationId",1],["projectId",1],["id",1],["workflowId",1],["taskId",1],["trigger",1],["mode",1],["parentRunId",1],["createdAt",1],["endedAt",1],["status",1],["deliveryOutcome",1],["failure",1],["error",1]]
},
{
"collection": "humanEvents",
"name": "production_event_facts_v1",
"unique": false,
"keys": [["organizationId",1],["itemId",1]],
"partialFilterExpression": {
"$or": [
{"eventType":{"$in":["approved","rejected"]}},
{"payload.deliveryPackageId":{"$exists":true}}
]
}
}
]
}

View File

@ -0,0 +1,59 @@
# 生产总览超时与统计快照容量,2026-11-10
任务基线与当时线上源码为 `e99febda`。本批完成本地修改、隔离数据验证和生产只读诊断;随后取得明确授权,仅新增下述两个生产索引。未部署候选应用、未补写业务数据。
## 根因与处理
生产总览有三处成本叠加。约 29,762 个 Capture ID 被装进一个 runs 的 OR 查询,查询规划约 14.7 秒,explain 结果还超过 BSON 限制;项目 runs 仍会 Fetch 含日志和节点正文的大文档,5 秒诊断预算内只返回 18,791 条;humanEvents 按 256 个 ID 拆分后,每批仍 Fetch 约 3,500 条编辑事件,约 1–1.6 秒。分批本身不能证明生产超时已解决。
代码改为互不重复的项目 runs、历史 Capture 关联、缺失 run ID 三段查询,关联 ID 每批至多 256;保留全量结果、组织隔离、所有尝试与旧事件交付关联。两个可选索引定义在 `production-overview-indexes.json`,不在启动迁移中创建:
- runs 覆盖生产统计字段,项目查询免读大文档;历史 taskId/id 查找继续使用原有索引,不能误用项目领先索引。
- humanEvents 仅索引正式决定或含旧交付关联的事件,跳过编辑正文。读取拆成两个互斥分支:正式决定,以及非决定的旧交付关联。直接把 OR 原样交给 partial index 会扫描整个统计子集,局部回归已覆盖该陷阱。
索引缺失时保留原路径和超时保护,不返回部分统计、伪造零值或延长 20 秒请求预算。索引缺失时仍可能慢,不能宣称生产已修复。
共享统计快照原来在每次刷新中将全组织历史读入一个 64 MiB/25 万条源预算,再编码成一个 4 MiB 压缩/32 MiB 解压对象。上次精简后源体积仍为 56.8 MiB,已接近上限;继续增大固定上限不能解决增长问题。
新路径将投影后的原始元数据流式写入私有临时目录,以临时 bbolt 索引连接历史关联,按 canonical part 分区解释。一个 part 的多次处理不能分开统计。跨 part 验收与未完成操作的状态屏蔽保留。派生事实单片仍执行原 4/32 MiB 限制;只在所有片完成后原子发布清单。查询逐片聚合计数、时长、比率的分子分母及人员天数,不常驻全部历史,也不平均百分比。权限仍逐请求判定。
临时索引选择及事务约束依据 [bbolt 官方说明](https://github.com/etcd-io/bbolt):可重建数据关闭 fsync,事务分批提交,回调前结束读事务,返回数据复制后使用。文件 0600、目录 0700;正常/取消/异常均清理。崩溃遗留超过 10 分钟且没有活动锁的目录在下一次构建清理,不删除活动构建。
## 边界、兼容与恢复
事实 schema 改为 v5,旧 v4 代次不被新程序误读;回退旧应用后按旧 schema 重建,不回写业务数据。单对象旧 Reader 路径仍兼容,生产 Mongo 使用流式 Reader。跨副本租约、epoch fencing、完整代次 pin 和前一完整代次回退保留;任一分片失败不发布部分结果。
本批同时修复旧数据漏洞:空的客户端 operationId 不能覆盖不相关的、缺少 operationId 的历史验收事件。原先漏算的历史验收可能恢复计数,这是口径纠错,不是分片丢失等价性。
增长仍有明确预算,不承诺无限容量:源按约 8 MiB 目标分区,单 part 硬限仍为 64 MiB/25 万条;临时目录每次构建 2 GiB、目录/关联事务内存计量 256 MiB、压缩事实总量 256 MiB、至多 4096 片、构建 90 秒、单次查询 10 秒。容量预警分别看批峰值、单片峰值、清单、总存储、临时磁盘和构建耗时的 80%,保留全量读字节/条数供观察;失败次数和最后错误继续暴露。达到预算应排查增长来源、归档/分层或资源,不自动截断历史。
生产 API 当前 request 1 CPU/2 GiB、limit 2 CPU/4 GiB。本地峰值不等于生产容量验收;上线前需确认 Pod 临时磁盘可用且有至少 2 GiB 构建预算。未调整资源、副本或带宽。
## 证据与未完成项
本地隔离库每次使用唯一名称并自动删除。三轮 72,000 part、504,000 条投影记录:源 90,049,350 字节,136 片,批峰值 5,062,804 字节;独立三轮构建 6.36–6.74 秒,构建加首次查询 6.77–7.18 秒。解压事实总量约 38 MiB,已超过旧 32 MiB 单对象限制;单片解压约 2.37 MiB。所有 part 均计入。构建后 GC 堆约 1.0–1.1 MiB、临时文件 0;构建堆峰值约 63–73 MiB,进程最高 RSS 230 MiB,临时磁盘峰值约 370 MiB。独立三轮 CPU 为 8.54/8.98/8.75 秒,无持续增长;早一组与其他门禁并发的样本单独保留,不用于容量承诺。
600 片段带编辑正文的索引回归:runs 扫描文档 0;两任务事件查询仅扫描 3 条,保留非决定旧交付关联;有无索引完整结果等价。24,000 片段、约 50 万事件的大载荷对照保留全部 24,001 条有效事件,五次完整读取为 364–530 毫秒,首次原路径 1.066 秒、索引路径 530 毫秒。该读耗时不包含真实浏览器与生产网络。
诊断和原始验证日志位于忽略目录 `local-data/diagnostics/statistics-growth/`。生产只读查询证明根因和现有 Mongo 8.0 版本兼容,并验证下述索引扫描收益。候选应用的生产端到端耗时、并发、CPU/内存及长期索引维护开销仍待发布后的真实对照;不以本地快、Pod 就绪或压缩体积小作为线上验收。
## 单独授权的生产索引计划
1. 核对泛宇 namespace、数据库与 Mongo 版本;盘点两集合记录数、当前负载和可用空间。逐项比对名称、字段顺序及 partial 条件;同名不同定义即停止,不替换已有索引。
2. 按清单一次只创建一个非唯一索引,先 runs,再 humanEvents。构建扫描全集合并消耗 IO/CPU/磁盘;runs 后续状态/错误更新增加索引维护,humanEvents 编辑事件被 partial 排除。实际大小和构建时间尚未在线量化,观察负载,不同时扩容或运行压测。超时先查实际完成状态,不能盲目重试。
3. 只读执行完整项目查询、256-ID 两分支查询及 explain,检查 runs documentsExamined=0、事件扫描只覆盖所需记录,核验完整数量与旧路径等价。API 发布需另按 release.py 审批,实际页面完整请求必须在 20 秒内且按性能目标验收,冷/热均测。
4. 无收益或写延迟恶化时,在授权范围内仅删除本次两个索引回退;先回退代码也安全,缺索引走兼容路径。不得删除原索引、修改客户记录或回滚客户状态。
## 已授权执行与当前线上状态
用户授权引用 `call_ac06f16e8fe8413ba93f38d7dd2e401a`,仅允许新增清单中的两个索引及只读对照。已核对 namespace `droboticflow-fanyu`,UID `d6a90ea7-b57d-4c52-b90b-b6c976cba076`,现用 API 摘要 `51395a98…`/源码 `e99febda`。
2026-11-11 UTC 依次创建两索引(本地日期 11 月 10 日)。runs 创建成功后的核对脚本复用了上一索引的属性,首次核对误报;随即停止后续创建、只读核实实际完整定义,再修正脚本。未重复创建 runs。humanEvents 创建用时 17.979 秒,精确定义核对完成。日志保留首次错误与后续核对,不把错误记录擦除。
新索引实际占用:runs 28,639,232 字节、humanEvents 20,480 字节。只读 explain:runs 项目完整覆盖查询返回 26,489 行,文档读取 0,数据库执行约 2–3 秒;代表性事件分支每批扫描 0–3 条,数据库执行 7–13 毫秒,原路径约 1–1.6 秒。操作后单点采样数据库读/写等待队列均为 0,不代表长期维护成本已验收。其他原索引及业务记录未改动。
现用旧应用复测仍为两次 HTTP 500,约 20.04/20.03 秒;尚未部署本批分批查询代码,不能宣称整页恢复。其余 humanItems/annotationTasks/worklogs 仍有秒级全量 Fetch 成本,后续发布后必须继续分段观察;本批不扩建未授权索引。一次通过 kubectl 控制链路运行完整候选 loader 的尝试受到传输与诊断连接 deadline 实现限制,未完成读取,不能用其约 49 秒结果判断生产候选 API 耗时;原始失败日志与限制说明保留。
本地 API/CSV 权限、浏览器看板、快照恢复、race、真实 Mongo、增长和官方前后端基线门禁已执行。官方 Go 门禁为 68 个已有基线失败、无新增失败。`verify:task` 的 26 项中 25 项通过;自动 store 组因未开启的可选测试标为 incomplete,所需本地统计专项随后在显式隔离环境执行通过。受保护的另一任务生产探针与生产端到端/持续容量仍未验收;不把总报告改写为全部通过。新增门禁及索引清单已补充验证映射,清单与本地实际索引字段/partial 条件由测试比对。
上述索引已完成,推送及应用部署仍为独立授权。索引语义参考 [MongoDB 官方 partial index 说明](https://www.mongodb.com/docs/v8.0/core/index-partial/)。

1
go.mod
View File

@ -6,6 +6,7 @@ require (
github.com/aliyun/aliyun-oss-go-sdk v3.0.2+incompatible
github.com/baidubce/bce-sdk-go v0.9.264
github.com/go-chi/chi/v5 v5.2.3
go.etcd.io/bbolt v1.4.3
go.mongodb.org/mongo-driver v1.17.6
golang.org/x/crypto v0.36.0
golang.org/x/sync v0.12.0

2
go.sum
View File

@ -108,6 +108,8 @@ github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78/go.mod h1:aL8wCCfTfS
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.etcd.io/bbolt v1.4.3 h1:dEadXpI6G79deX5prL3QRNP6JB8UxVkqo4UPnHaNXJo=
go.etcd.io/bbolt v1.4.3/go.mod h1:tKQlpPaYCVFctUIgFKFnAlvbmB3tpy1vkTnDWohtc0E=
go.mongodb.org/mongo-driver v1.17.6 h1:87JUG1wZfWsr6rIz3ZmpH90rL5tea7O3IHuSwHUpsss=
go.mongodb.org/mongo-driver v1.17.6/go.mod h1:Hy04i7O2kC4RS06ZrhPRqj/u4DTYkFDAAccj+rVKqgQ=
go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=

View File

@ -168,7 +168,7 @@ func (s *server) handleAnnotationOutput(w http.ResponseWriter, r *http.Request)
}
// Do not narrow the load to a requested person: a review may address sources
// attributed to several authors. The read model applies the principal fence.
s.outputOnce.Do(func() { s.outputReader = outputsnapshot.NewReader(s.store, s.store.LoadAnnotationOutput) })
s.outputOnce.Do(func() { s.outputReader = s.newOutputSnapshotReader() })
report, info, err := s.outputReader.Query(ctx, org, ids, r.URL.Query().Get("snapshotId"), q)
// Older deployed clients cannot consume 202. They wait for the same shared
// background generation; they never start a private synchronous rebuild.
@ -314,7 +314,7 @@ func (s *server) handleAnnotationOutputHealth(w http.ResponseWriter, r *http.Req
renderError(w, 503, err)
return
}
s.outputOnce.Do(func() { s.outputReader = outputsnapshot.NewReader(s.store, s.store.LoadAnnotationOutput) })
s.outputOnce.Do(func() { s.outputReader = s.newOutputSnapshotReader() })
health, err := s.outputReader.Inspect(ctx, org, ids)
if err != nil {
renderError(w, 503, err)
@ -322,3 +322,14 @@ func (s *server) handleAnnotationOutputHealth(w http.ResponseWriter, r *http.Req
}
renderJSON(w, 200, health)
}
func (s *server) newOutputSnapshotReader() *outputsnapshot.Reader {
if source, ok := any(s.store).(interface {
StreamAnnotationOutput(context.Context, string, []string, func(workforcestats.OutputSnapshot) error) (*workforcestats.OutputSourceStats, error)
}); ok {
if _, ok := any(s.store).(outputsnapshot.ShardedBackend); ok {
return outputsnapshot.NewStreamReader(s.store, source.StreamAnnotationOutput)
}
}
return outputsnapshot.NewReader(s.store, s.store.LoadAnnotationOutput)
}

View File

@ -105,7 +105,7 @@ func (s *server) handleProductionQuality(w http.ResponseWriter, r *http.Request)
if vendor && p.User.SupplierID == "" {
ids = []string{}
}
s.outputOnce.Do(func() { s.outputReader = outputsnapshot.NewReader(s.store, s.store.LoadAnnotationOutput) })
s.outputOnce.Do(func() { s.outputReader = s.newOutputSnapshotReader() })
report, info, err := s.outputReader.Query(ctx, org, ids, r.URL.Query().Get("snapshotId"), q)
if errors.Is(err, outputsnapshot.ErrExpired) {
renderJSON(w, 409, map[string]string{"code": "snapshot_expired", "error": "数据已更新,请重试"})

View File

@ -60,13 +60,30 @@ func snapshotHealth(state State, key string, now time.Time) Health {
out.Warnings = append(out.Warnings, code)
}
}
warn(g.Bytes >= MaxEncodedBytes*4/5, "encoded_capacity")
if g.Shards == 0 {
warn(g.Bytes >= MaxEncodedBytes*4/5, "encoded_capacity")
} else {
warn(g.Bytes >= MaxGenerationBytes*4/5, "generation_capacity")
warn(g.Shards >= MaxShards*4/5, "shard_count_capacity")
warn(g.ManifestBytes >= MaxEncodedBytes*4/5, "manifest_capacity")
}
if m := g.Metrics; m != nil {
warn(m.DurationMS >= BuildTimeout.Milliseconds()*4/5, "build_time_budget")
warn(m.DecodedBytes >= MaxDecodedBytes*4/5, "decoded_capacity")
decoded := m.DecodedBytes
if g.Shards > 0 {
decoded = m.PeakDecodedBytes
warn(m.PeakEncodedBytes >= MaxEncodedBytes*4/5, "encoded_capacity")
}
warn(decoded >= MaxDecodedBytes*4/5, "decoded_capacity")
if m.Source != nil {
warn(m.Source.Bytes >= workforcestats.OutputSourceMaxBytes*4/5, "source_bytes_capacity")
warn(m.Source.Documents >= workforcestats.OutputSourceMaxDocuments*4/5, "source_documents_capacity")
bytes, documents := m.Source.Bytes, m.Source.Documents
if g.Shards > 0 {
bytes, documents = m.Source.PeakBatchBytes, m.Source.PeakBatchDocuments
warn(m.Source.DirectoryBytes >= workforcestats.OutputDirectoryMaxBytes*4/5, "directory_capacity")
warn(m.Source.PeakSpoolBytes >= workforcestats.OutputSpoolMaxBytes*4/5, "temporary_disk_capacity")
}
warn(bytes >= workforcestats.OutputSourceMaxBytes*4/5, "source_bytes_capacity")
warn(documents >= workforcestats.OutputSourceMaxDocuments*4/5, "source_documents_capacity")
}
}
}

View File

@ -10,9 +10,10 @@ import (
// MemoryBackend is the in-memory Adapter used by API fixtures. Its coordinator
// and bounded slots have the same ownership/publication rules as Mongo.
type MemoryBackend struct {
mu sync.Mutex
state State
slots [SlotCount]memoryBlob
mu sync.Mutex
state State
slots [SlotCount]memoryBlob
shards map[string][]byte
}
type memoryBlob struct {
epoch int64
@ -67,12 +68,20 @@ func (b *MemoryBackend) PublishOutputSnapshot(_ context.Context, l Lease, g Gene
if !b.owns(l, now) {
return false, nil
}
if l.Slot < 0 || l.Slot >= SlotCount || g.Slot != l.Slot || g.Epoch != l.Epoch || g.Bytes != len(data) || len(data) > MaxEncodedBytes {
if !ValidPublication(l, g, data) {
return false, ErrCapacity
}
b.slots[l.Slot] = memoryBlob{l.Epoch, g.ID, append([]byte(nil), data...)}
b.state.Heads[l.Key] = Head{Current: &g, Previous: l.Previous, NextAttempt: g.AsOf.Add(RefreshInterval)}
b.state.Owner = ""
for key := range b.shards {
var slot int
var epoch int64
var index int
if _, err := fmt.Sscanf(key, "%d/%d/%d", &slot, &epoch, &index); err == nil && slot == l.Slot && epoch < l.Epoch {
delete(b.shards, key)
}
}
return true, nil
}
func (b *MemoryBackend) FailOutputSnapshot(_ context.Context, l Lease, now time.Time, failure Failure) error {
@ -98,3 +107,45 @@ func (b *MemoryBackend) OutputSnapshotBlob(_ context.Context, g Generation) ([]b
}
return append([]byte(nil), v.data...), nil
}
func shardKey(slot int, epoch int64, index int) string {
return fmt.Sprintf("%d/%d/%d", slot, epoch, index)
}
func (b *MemoryBackend) StageOutputSnapshotShard(_ context.Context, l Lease, index int, data []byte, now time.Time) (bool, error) {
b.mu.Lock()
defer b.mu.Unlock()
if !b.owns(l, now) {
return false, nil
}
if l.Slot < 0 || l.Slot >= SlotCount || index < 0 || index >= MaxShards || len(data) > MaxEncodedBytes {
return false, ErrCapacity
}
if b.shards == nil {
b.shards = map[string][]byte{}
}
key := shardKey(l.Slot, l.Epoch, index)
if _, exists := b.shards[key]; exists {
return false, ErrUnavailable
}
if index == 0 {
for existing := range b.shards {
var slot int
var epoch int64
var index int
if _, err := fmt.Sscanf(existing, "%d/%d/%d", &slot, &epoch, &index); err == nil && slot == l.Slot && epoch < l.Epoch {
delete(b.shards, existing)
}
}
}
b.shards[key] = append([]byte(nil), data...)
return true, nil
}
func (b *MemoryBackend) OutputSnapshotShard(_ context.Context, g Generation, index int) ([]byte, error) {
b.mu.Lock()
defer b.mu.Unlock()
data, ok := b.shards[shardKey(g.Slot, g.Epoch, index)]
if !ok {
return nil, ErrExpired
}
return append([]byte(nil), data...), nil
}

View File

@ -25,6 +25,7 @@ type Reader struct {
querySlots chan struct{}
backend Backend
source Source
stream StreamSource
now func() time.Time
ctx context.Context
cancel context.CancelFunc
@ -117,6 +118,9 @@ func (r *Reader) Read(ctx context.Context, org string, projects []string, pin st
}
func (r *Reader) readResolved(ctx context.Context, gen *Generation, info Info, pin string) (*workforcestats.OutputDataset, Info, error) {
now := r.now()
if gen.Shards > 0 {
return nil, info, ErrCapacity
} // Query streams shards; Read retains its bounded single-dataset contract.
select {
case r.reads <- struct{}{}:
@ -223,6 +227,18 @@ func (r *Reader) build(key, org string, projects []string) {
}()
defer func() { close(stop); <-done }()
asOf := r.now().UTC().Truncate(time.Millisecond)
if r.stream != nil {
stage = "partitioned_build"
gen, payload, err := r.buildShards(ctx, lease, org, projects, asOf, started)
if err != nil {
buildErr = err
return
}
stage = "publish"
published, buildErr = r.backend.PublishOutputSnapshot(ctx, lease, gen, payload, r.now())
return
}
raw, err := r.source(ctx, org, projects)
if err != nil || ctx.Err() != nil {
buildErr = err
@ -369,16 +385,15 @@ func (r *Reader) Query(ctx context.Context, org string, projects []string, pin s
if report := r.memoReport(key); report != nil {
return answer{report, info}, nil
}
data, actual, err := r.readResolved(buildCtx, gen, info, pin)
if err != nil || data == nil {
report, actual, err := r.queryResolved(buildCtx, gen, info, pin, q)
if err != nil || report == nil {
return nil, err
}
report := workforcestats.QueryOutputDataset(*data, q)
// Corrupt-current fallback must not poison the current generation's key.
if actual.ID == gen.ID {
r.rememberReport(key, &report)
r.rememberReport(key, report)
}
return answer{&report, actual}, nil
return answer{report, actual}, nil
})
select {
case <-ctx.Done():

View File

@ -0,0 +1,205 @@
package outputsnapshot
import (
"bytes"
"compress/gzip"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"time"
"droboticflow/internal/workforcestats"
)
type shardDescriptor struct {
Bytes int `json:"b"`
SHA256 string `json:"h"`
}
type snapshotManifest struct {
Shards []shardDescriptor `json:"shards"`
}
const MaxGenerationBytes = 256 << 20
func NewStreamReader(b Backend, source StreamSource) *Reader {
r := NewReader(b, nil)
r.stream = source
return r
}
func digest(data []byte) string { h := sha256.Sum256(data); return hex.EncodeToString(h[:]) }
func (r *Reader) buildShards(ctx context.Context, lease Lease, org string, projects []string, asOf time.Time, started time.Time) (Generation, []byte, error) {
backend, ok := r.backend.(ShardedBackend)
if !ok {
return Generation{}, nil, ErrUnavailable
}
manifest := snapshotManifest{Shards: []shardDescriptor{}}
metrics := &BuildMetrics{}
total := 0
yield := func(raw workforcestats.OutputSnapshot) error {
if err := ctx.Err(); err != nil {
return err
}
data := workforcestats.BuildOutputDataset(raw, asOf)
encoded, decoded, err := encode(data)
if errors.Is(err, ErrCapacity) {
return workforcestats.ErrOutputPartitionSplit
}
if err != nil {
return err
}
if len(manifest.Shards) >= MaxShards || total+len(encoded) > MaxGenerationBytes {
return ErrCapacity
}
if err = ctx.Err(); err != nil {
return err
}
ok, err := backend.StageOutputSnapshotShard(ctx, lease, len(manifest.Shards), encoded, r.now())
if err != nil {
return err
}
if !ok {
return fmt.Errorf("snapshot shard lease lost")
}
manifest.Shards = append(manifest.Shards, shardDescriptor{len(encoded), digest(encoded)})
total += len(encoded)
metrics.DecodedBytes += decoded
metrics.Parts += len(data.Parts)
metrics.PeakEncodedBytes = max(metrics.PeakEncodedBytes, len(encoded))
metrics.PeakDecodedBytes = max(metrics.PeakDecodedBytes, decoded)
return nil
}
source, err := r.stream(ctx, org, projects, yield)
if err != nil {
return Generation{}, nil, err
}
if len(manifest.Shards) == 0 {
if err = yield(workforcestats.OutputSnapshot{}); err != nil {
return Generation{}, nil, err
}
}
payload, err := json.Marshal(manifest)
if err != nil {
return Generation{}, nil, err
}
if len(payload) > MaxEncodedBytes {
return Generation{}, nil, ErrCapacity
}
metrics.Source = source
metrics.DurationMS = time.Since(started).Milliseconds()
g := Generation{ID: fmt.Sprintf("of1-%s-%d", lease.Owner, lease.Epoch), Slot: lease.Slot, Epoch: lease.Epoch, AsOf: asOf, SHA256: digest(payload), Bytes: total, Shards: len(manifest.Shards), ManifestBytes: len(payload), Metrics: metrics}
return g, payload, nil
}
func decodeShard(payload []byte, g Generation) (*workforcestats.OutputDataset, error) {
z, err := gzip.NewReader(bytes.NewReader(payload))
if err != nil {
return nil, err
}
defer z.Close()
raw, err := io.ReadAll(io.LimitReader(z, MaxDecodedBytes+1))
if err != nil {
return nil, err
}
if len(raw) > MaxDecodedBytes {
return nil, ErrCapacity
}
var d workforcestats.OutputDataset
if err = json.Unmarshal(raw, &d); err != nil {
return nil, err
}
if !d.AsOf.Equal(g.AsOf) {
return nil, ErrUnavailable
}
return &d, nil
}
func (r *Reader) queryGeneration(ctx context.Context, g Generation, q workforcestats.OutputQuery) (*workforcestats.OutputReport, error) {
q.Now = g.AsOf
if g.Shards == 0 {
d, err := r.load(ctx, g)
if err != nil {
return nil, err
}
report := workforcestats.QueryOutputDataset(*d, q)
return &report, ctx.Err()
}
backend, ok := r.backend.(ShardedBackend)
if !ok {
return nil, ErrUnavailable
}
if g.Shards < 1 || g.Shards > MaxShards || g.Bytes > MaxGenerationBytes {
return nil, ErrCapacity
}
payload, err := r.backend.OutputSnapshotBlob(ctx, g)
if err != nil {
return nil, err
}
if len(payload) != g.ManifestBytes || len(payload) > MaxEncodedBytes || digest(payload) != g.SHA256 {
return nil, ErrUnavailable
}
var manifest snapshotManifest
if err = json.Unmarshal(payload, &manifest); err != nil {
return nil, err
}
if len(manifest.Shards) != g.Shards {
return nil, ErrUnavailable
}
total := 0
for _, d := range manifest.Shards {
if d.Bytes < 1 || d.Bytes > MaxEncodedBytes {
return nil, ErrCapacity
}
total += d.Bytes
}
if total != g.Bytes {
return nil, ErrUnavailable
}
reducer := workforcestats.NewOutputReducer(q)
for i, descriptor := range manifest.Shards {
if err = ctx.Err(); err != nil {
return nil, err
}
encoded, err := backend.OutputSnapshotShard(ctx, g, i)
if err != nil {
return nil, err
}
if len(encoded) != descriptor.Bytes || digest(encoded) != descriptor.SHA256 {
return nil, ErrUnavailable
}
d, err := decodeShard(encoded, g)
if err != nil {
return nil, err
}
reducer.Add(*d)
}
report := reducer.Report()
return &report, ctx.Err()
}
func (r *Reader) queryResolved(ctx context.Context, g *Generation, info Info, pin string, q workforcestats.OutputQuery) (*workforcestats.OutputReport, Info, error) {
report, err := r.queryGeneration(ctx, *g, q)
if err != nil && ctx.Err() == nil && pin == "" {
state, stateErr := r.backend.OutputSnapshotState(ctx)
if stateErr == nil {
for _, head := range state.Heads {
if head.Current != nil && head.Current.ID == g.ID && head.Previous != nil && r.now().Sub(head.Previous.AsOf) <= MaxAge {
g = head.Previous
report, err = r.queryGeneration(ctx, *g, q)
break
}
}
}
}
if err != nil {
if ctx.Err() != nil {
return nil, info, ctx.Err()
}
return nil, info, ErrUnavailable
}
info.ID = g.ID
info.AsOf = g.AsOf
info.Stale = r.now().Sub(g.AsOf) >= RefreshInterval
return report, info, nil
}

View File

@ -0,0 +1,114 @@
package outputsnapshot
import (
"context"
"errors"
"reflect"
"sync/atomic"
"testing"
"time"
"droboticflow/internal/workforcestats"
)
func TestShardedGenerationMatchesLegacyAndSurvivesCorruptionAndRestart(t *testing.T) {
now := time.Now().UTC().Truncate(time.Millisecond)
var clock atomic.Int64
clock.Store(now.UnixMilli())
nowFn := func() time.Time { return time.UnixMilli(clock.Load()).UTC() }
b := &MemoryBackend{}
raw := testData(now)
source := func(ctx context.Context, _ string, _ []string, yield func(workforcestats.OutputSnapshot) error) (*workforcestats.OutputSourceStats, error) {
for i := range raw.Tasks {
d := raw
d.Tasks = raw.Tasks[i : i+1]
d.Events = raw.Events[i : i+1]
if err := yield(d); err != nil {
return nil, err
}
}
return &workforcestats.OutputSourceStats{Bytes: 90 << 20, Documents: 300000, Partitions: 12, PeakBatchBytes: 8 << 20, PeakBatchDocuments: 25000}, nil
}
r := NewStreamReader(b, source)
defer r.Close()
r.now = nowFn
q := workforcestats.OutputQuery{From: now.Add(-24 * time.Hour), To: now.Add(time.Hour)}
var first Info
eventually(t, func() bool {
report, info, err := r.Query(t.Context(), "org", []string{"p"}, "", q)
if err != nil {
t.Fatal(err)
}
if report == nil {
return false
}
want := workforcestats.QueryOutputDataset(workforcestats.BuildOutputDataset(raw, now), q)
if !reflect.DeepEqual(*report, want) {
t.Fatal("report differs")
}
first = info
return true
})
h, err := r.Inspect(t.Context(), "org", []string{"p"})
if err != nil || len(h.Warnings) != 0 {
t.Fatalf("partition capacity warns on total history: %+v %v", h, err)
}
now = now.Add(RefreshInterval + time.Second)
clock.Store(now.UnixMilli())
eventually(t, func() bool {
_, info, err := r.Query(t.Context(), "org", []string{"p"}, "", q)
if err != nil {
t.Fatal(err)
}
return info.ID != "" && info.ID != first.ID
})
b.mu.Lock()
head := b.state.Heads[scopeKey("org", []string{"p"})]
b.shards[shardKey(head.Current.Slot, head.Current.Epoch, 3)] = []byte("broken")
b.mu.Unlock()
// New process has no memoized report. A corrupt shard must discard the entire
// partial reduction and use the previous complete generation.
restarted := NewStreamReader(b, source)
defer restarted.Close()
restarted.now = nowFn
report, info, err := restarted.Query(t.Context(), "org", []string{"p"}, "", q)
if err != nil || report == nil || info.ID != first.ID {
t.Fatalf("previous fallback: %v %+v", err, info)
}
if report.Totals.Submitted.Count != 12 {
t.Fatal("partial generation contaminated fallback")
}
if _, _, err = restarted.Query(t.Context(), "org", []string{"p"}, head.Current.ID, q); !errors.Is(err, ErrUnavailable) {
t.Fatal("corrupt pin silently changed", err)
}
}
func TestShardedBuildFailureNeverPublishesPartialManifest(t *testing.T) {
now := time.Now().UTC().Truncate(time.Millisecond)
b := &MemoryBackend{}
r := NewStreamReader(b, func(ctx context.Context, _ string, _ []string, yield func(workforcestats.OutputSnapshot) error) (*workforcestats.OutputSourceStats, error) {
if err := yield(testData(now)); err != nil {
return nil, err
}
return nil, errors.New("source stopped")
})
defer r.Close()
r.now = func() time.Time { return now }
r.build(scopeKey("org", []string{"p"}), "org", []string{"p"})
state, _ := b.OutputSnapshotState(t.Context())
h := state.Heads[scopeKey("org", []string{"p"})]
if h.Current != nil || h.ConsecutiveFailures != 1 {
t.Fatalf("partial generation visible: %+v", h)
}
lease, ok, err := b.AcquireOutputSnapshot(t.Context(), scopeKey("org", []string{"p"}), now.Add(RetryInterval+time.Second))
if err != nil || !ok {
t.Fatal("retry unavailable")
}
if ok, err = b.StageOutputSnapshotShard(t.Context(), lease, 0, []byte("next"), now.Add(RetryInterval+time.Second)); err != nil || !ok {
t.Fatal("retry stage failed")
}
stale := lease
stale.Epoch--
if ok, err = b.StageOutputSnapshotShard(t.Context(), stale, 1, []byte("stale"), now.Add(RetryInterval+time.Second)); err != nil || ok {
t.Fatal("expired writer staged")
}
}

View File

@ -10,7 +10,7 @@ import (
)
const (
Schema = "annotation-output-facts/v4"
Schema = "annotation-output-facts/v5"
RefreshInterval = 120 * time.Second
RetryInterval = 120 * time.Second
MaxAge = 15 * time.Minute
@ -30,10 +30,12 @@ var (
// DurationMS covers source reads, calculation and encoding, before publication.
type BuildMetrics struct {
DurationMS int64 `bson:"durationMs" json:"durationMs"`
DecodedBytes int `bson:"decodedBytes" json:"decodedBytes"`
Parts int `bson:"parts" json:"parts"`
Source *workforcestats.OutputSourceStats `bson:"source,omitempty" json:"source,omitempty"`
PeakEncodedBytes int `bson:"peakEncodedBytes,omitempty" json:"peakEncodedBytes,omitempty"`
PeakDecodedBytes int `bson:"peakDecodedBytes,omitempty" json:"peakDecodedBytes,omitempty"`
DurationMS int64 `bson:"durationMs" json:"durationMs"`
DecodedBytes int `bson:"decodedBytes" json:"decodedBytes"`
Parts int `bson:"parts" json:"parts"`
Source *workforcestats.OutputSourceStats `bson:"source,omitempty" json:"source,omitempty"`
}
// Failure stores bounded machine-readable categories; raw errors stay in logs.
@ -44,13 +46,15 @@ type Failure struct {
DurationMS int64 `bson:"durationMs" json:"durationMs"`
}
type Generation struct {
Metrics *BuildMetrics `bson:"metrics,omitempty" json:"-"`
ID string `bson:"id" json:"id"`
Slot int `bson:"slot" json:"-"`
Epoch int64 `bson:"epoch" json:"-"`
AsOf time.Time `bson:"asOf" json:"asOf"`
SHA256 string `bson:"sha256" json:"-"`
Bytes int `bson:"bytes" json:"-"`
Shards int `bson:"shards,omitempty" json:"-"`
ManifestBytes int `bson:"manifestBytes,omitempty" json:"-"`
Metrics *BuildMetrics `bson:"metrics,omitempty" json:"-"`
ID string `bson:"id" json:"id"`
Slot int `bson:"slot" json:"-"`
Epoch int64 `bson:"epoch" json:"-"`
AsOf time.Time `bson:"asOf" json:"asOf"`
SHA256 string `bson:"sha256" json:"-"`
Bytes int `bson:"bytes" json:"-"`
}
type Head struct {
ConsecutiveFailures int64 `bson:"consecutiveFailures,omitempty"`
@ -143,3 +147,25 @@ const StateCollection = "annotationOutputSnapshotState"
const SlotsCollection = "annotationOutputSnapshotSlots"
func IsPrivateCollection(name string) bool { return name == StateCollection || name == SlotsCollection }
// ShardedBackend stages immutable, epoch-addressed blobs. Only publication of
// the manifest makes a complete generation visible to readers.
type ShardedBackend interface {
StageOutputSnapshotShard(context.Context, Lease, int, []byte, time.Time) (bool, error)
OutputSnapshotShard(context.Context, Generation, int) ([]byte, error)
}
type StreamSource func(context.Context, string, []string, func(workforcestats.OutputSnapshot) error) (*workforcestats.OutputSourceStats, error)
func ValidPublication(l Lease, g Generation, data []byte) bool {
if l.Slot < 0 || l.Slot >= SlotCount || g.Slot != l.Slot || g.Epoch != l.Epoch || len(data) > MaxEncodedBytes {
return false
}
if g.Shards > 0 {
return g.ManifestBytes == len(data) && g.Shards <= MaxShards && g.Bytes >= 0
}
return g.Shards == 0 && g.Bytes == len(data)
}
// Bound manifest size and total per-generation derived storage independently of
// historical source size. Oversized single parts remain explicit failures.
const MaxShards = 4096

View File

@ -132,6 +132,9 @@ func (s *MongoStore) LoadAnnotationOutput(ctx context.Context, org string, proje
return s.loadAnnotationOutput(ctx, org, projects, outputSourceBudget{workforcestats.OutputSourceMaxBytes, workforcestats.OutputSourceMaxDocuments})
}
func (s *MongoStore) loadAnnotationOutput(ctx context.Context, org string, projects []string, budget outputSourceBudget) (workforcestats.OutputSnapshot, error) {
return s.loadAnnotationOutputTo(ctx, org, projects, budget, nil)
}
func (s *MongoStore) loadAnnotationOutputTo(ctx context.Context, org string, projects []string, budget outputSourceBudget, spool *outputSourceSpool) (workforcestats.OutputSnapshot, error) {
d := workforcestats.OutputSnapshot{}
if len(projects) == 0 {
return d, nil
@ -181,11 +184,17 @@ func (s *MongoStore) loadAnnotationOutput(ctx context.Context, org string, proje
totalBytes += len(cursor.Current)
totalDocs++
bytes, documents := totalBytes, totalDocs
exceeded := bytes > budget.bytes || documents > budget.documents
exceeded := spool == nil && (bytes > budget.bytes || documents > budget.documents)
budgetMu.Unlock()
if exceeded {
return fmt.Errorf("%w: bytes=%d/%d documents=%d/%d", workforcestats.ErrOutputSourceBudget, bytes, budget.bytes, documents, budget.documents)
}
if spool != nil {
if err = spool.add(c.Name(), cursor.Current); err != nil {
return err
}
continue
}
v := reflect.New(target.Type().Elem())
if err = cursor.Decode(v.Interface()); err != nil {
return err
@ -276,6 +285,13 @@ func (s *MongoStore) loadAnnotationOutput(ctx context.Context, org string, proje
runIDs = append(runIDs, c.RunID)
captureIDs = append(captureIDs, c.ID)
}
readDirectory := func(table string, c *mongo.Collection, key string, keys []string, projection bson.M, out any, constraints ...bson.M) error {
if spool == nil {
return readRelated(c, key, keys, projection, out, constraints...)
}
return spool.index.eachKeys(readCtx, table, func(batch []string) error { return readRelated(c, key, batch, projection, out, constraints...) })
}
g.Go(func() error {
projection := fields("id", "taskId", "mode", "createdAt")
if err := readRelated(s.runs, "projectId", projects, projection, &d.Runs); err != nil {
@ -285,7 +301,7 @@ func (s *MongoStore) loadAnnotationOutput(ctx context.Context, org string, proje
// links are disjoint. Deduplicating after fetching still spends the
// global byte/document budget on every repeated row.
var relatedRuns []domain.RunRecord
if err := readRelated(s.runs, "taskId", captureIDs, projection, &relatedRuns, bson.M{"projectId": bson.M{"$nin": projects}}); err != nil {
if err := readDirectory("captures", s.runs, "taskId", captureIDs, projection, &relatedRuns, bson.M{"projectId": bson.M{"$nin": projects}}); err != nil {
return err
}
d.Runs = append(d.Runs, relatedRuns...)
@ -293,6 +309,22 @@ func (s *MongoStore) loadAnnotationOutput(ctx context.Context, org string, proje
for _, run := range d.Runs {
seen[run.ID] = true
}
if spool != nil {
return spool.index.eachKeys(readCtx, "runRefs", func(refs []string) error {
missing := make([]string, 0, len(refs))
for _, id := range refs {
seen, err := spool.index.contains("runs", id)
if err != nil {
return err
}
if !seen {
missing = append(missing, id)
}
}
return readRelated(s.runs, "id", missing, projection, &relatedRuns)
})
}
missing := make([]string, 0, len(runIDs))
for _, id := range runIDs {
if !seen[id] {
@ -310,24 +342,26 @@ func (s *MongoStore) loadAnnotationOutput(ctx context.Context, org string, proje
for _, t := range d.Tasks {
ids = append(ids, t.ID)
}
g.Go(func() error {
return readRelated(s.annotationReviews, "taskId", ids, fields("id", "taskId", "revision", "stage", "decision", "reviewerUserId", "createdAt"), &d.Reviews)
return readDirectory("tasks", s.annotationReviews, "taskId", ids, fields("id", "taskId", "revision", "stage", "decision", "reviewerUserId", "createdAt"), &d.Reviews)
})
g.Go(func() error {
return readRelated(s.annotationRevisions, "taskId", ids, fields("taskId", "revision", "authorUserId", "isRework", "countsAsOutput"), &d.Revisions)
return readDirectory("tasks", s.annotationRevisions, "taskId", ids, fields("taskId", "revision", "authorUserId", "isRework", "countsAsOutput"), &d.Revisions)
})
itemIDs := []string{}
for _, i := range d.Items {
itemIDs = append(itemIDs, i.ID)
}
g.Go(func() error {
return readRelated(s.humanEvents, "itemId", itemIDs, fields("id", "itemId", "stageId", "eventType", "actorUserId", "createdAt", "payload.operationId", "payload.sourceItemIds", "payload.deliveryPackageId", "payload.reviewedRevision"), &d.HumanEvents, bson.M{"stageId": bson.M{"$in": workforcestats.OutputAcceptanceStageIDs()}})
return readDirectory("items", s.humanEvents, "itemId", itemIDs, fields("id", "itemId", "stageId", "eventType", "actorUserId", "createdAt", "payload.operationId", "payload.sourceItemIds", "payload.deliveryPackageId", "payload.reviewedRevision"), &d.HumanEvents, bson.M{"stageId": bson.M{"$in": workforcestats.OutputAcceptanceStageIDs()}})
})
g.Go(func() error {
return read(s.db.Collection("reworkBaselines"), f, fields("_id", "itemId", "round", "stageId", "reviewOperationId", "problems.id", "problems.scope", "problems.sourceItemId", "problems.origin.stageId", "createdAt"), &d.Baselines)
})
g.Go(func() error {
return readRelated(s.db.Collection("reworkHandlings"), "itemId", itemIDs, fields("_id", "itemId", "round", "operationId", "skippedOpinionIds", "updatedAt"), &d.Handlings)
return readDirectory("items", s.db.Collection("reworkHandlings"), "itemId", itemIDs, fields("_id", "itemId", "round", "operationId", "skippedOpinionIds", "updatedAt"), &d.Handlings)
})
err := g.Wait()
d.SourceStats = &workforcestats.OutputSourceStats{Bytes: totalBytes, Documents: totalDocs}

View File

@ -0,0 +1,163 @@
package store
import (
"droboticflow/internal/outputsnapshot"
"droboticflow/internal/workforcestats"
"fmt"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"os"
"runtime"
"strings"
"sync/atomic"
"syscall"
"testing"
"time"
)
// localOutputMongo creates annotation_output_snapshots_test_<UnixNano> and
// registers db.Drop in t.Cleanup. No shared snapshot database is mutated.
func TestStatisticsPartitionGrowthBeyondGlobalBudget(t *testing.T) {
if os.Getenv("RUN_OUTPUT_HISTORY_SCALE") != "1" {
t.Skip("explicit isolated growth benchmark")
}
tempRoot := t.TempDir()
t.Setenv("TMPDIR", tempRoot)
db := localOutputMongo(t)
s := NewMongoStore(db)
ctx := t.Context()
now := time.Now().UTC().Truncate(time.Millisecond)
const n = 72000
for _, name := range []string{"annotationTasks", "humanItems", "annotationRevisions", "annotationWorklogEvents", "annotationReviews", "captureRecords", "runs"} {
for _, keys := range []bson.D{{{Key: "organizationId", Value: 1}, {Key: "projectId", Value: 1}, {Key: "taskId", Value: 1}}, {{Key: "organizationId", Value: 1}, {Key: "taskId", Value: 1}}, {{Key: "organizationId", Value: 1}, {Key: "id", Value: 1}}} {
if _, err := db.Collection(name).Indexes().CreateOne(ctx, mongo.IndexModel{Keys: keys}); err != nil {
t.Fatal(err)
}
}
for start := 0; start < n; start += 1000 {
rows := []any{}
for i := start; i < min(start+1000, n); i++ {
id := fmt.Sprint("t", i)
row := bson.M{"id": id, "organizationId": "out", "projectId": "p"}
switch name {
case "annotationTasks":
row["runId"] = id
row["sourceIsFullVideo"] = true
row["sourceDurationSeconds"] = 60
case "humanItems":
row["legacyTaskId"] = id
case "annotationRevisions":
row["taskId"] = id
row["revision"] = 1
row["countsAsOutput"] = true
row["authorUserId"] = "u"
case "annotationWorklogEvents":
row["id"] = id + strings.Repeat("x", 512)
row["taskId"] = id
row["userId"] = "u"
row["supplierId"] = "v"
row["eventType"] = "combined_submitted"
row["revision"] = 1
row["createdAt"] = now.Add(-time.Hour).Format(time.RFC3339Nano)
case "annotationReviews":
row["taskId"] = id
row["stage"] = "quality"
row["decision"] = "approved"
row["reviewerUserId"] = "qc"
row["revision"] = 1
row["createdAt"] = now.Add(-time.Minute).Format(time.RFC3339Nano)
case "captureRecords":
row["runId"] = id
row["prefix"] = "capture/" + id + "/part0001"
row["captureId"] = id + "/part0001"
case "runs":
row["taskId"] = id
}
rows = append(rows, row)
}
if _, err := db.Collection(name).InsertMany(ctx, rows); err != nil {
t.Fatal(err)
}
}
}
runtime.GC()
var initial runtime.MemStats
runtime.ReadMemStats(&initial)
var peak atomic.Uint64
peak.Store(initial.HeapAlloc)
stop := make(chan struct{})
done := make(chan struct{})
go func() {
defer close(done)
ticker := time.NewTicker(20 * time.Millisecond)
defer ticker.Stop()
for {
select {
case <-stop:
return
case <-ticker.C:
var m runtime.MemStats
runtime.ReadMemStats(&m)
for old := peak.Load(); m.HeapAlloc > old; old = peak.Load() {
if peak.CompareAndSwap(old, m.HeapAlloc) {
break
}
}
}
}
}()
defer func() { close(stop); <-done }()
var beforeCPU syscall.Rusage
_ = syscall.Getrusage(syscall.RUSAGE_SELF, &beforeCPU)
reader := outputsnapshot.NewStreamReader(s, s.StreamAnnotationOutput)
defer reader.Close()
q := workforcestats.OutputQuery{From: now.Add(-24 * time.Hour), To: now.Add(time.Hour), ProductionQuality: true}
started := time.Now()
deadline := started.Add(100 * time.Second)
for {
report, info, err := reader.Query(ctx, "out", []string{"p"}, "", q)
if err != nil {
t.Fatal(err)
}
if report != nil {
if report.Production.Totals.Submitted.Count != n {
t.Fatalf("lost parts: %+v", report.Production.Totals.Submitted)
}
health, err := reader.Inspect(ctx, "out", []string{"p"})
if err != nil {
t.Fatal(err)
}
source := health.Metrics.Source
if source.Bytes <= workforcestats.OutputSourceMaxBytes || source.Documents <= workforcestats.OutputSourceMaxDocuments || source.PeakBatchBytes > workforcestats.OutputSourceMaxBytes {
t.Fatalf("fixture did not cross old limits or violated batch bound: %+v", source)
}
if len(health.Warnings) > 0 {
t.Fatalf("growth capacity warning %+v", health)
}
entries, err := os.ReadDir(tempRoot)
if err != nil || len(entries) != 0 {
t.Fatalf("temporary build files leaked: %d %v", len(entries), err)
}
reader.Close()
runtime.GC()
var retained runtime.MemStats
runtime.ReadMemStats(&retained)
t.Logf("post_gc_heap=%d temporary_files=%d", retained.HeapAlloc, len(entries))
if retained.HeapAlloc > initial.HeapAlloc+16<<20 {
t.Fatal("retained build heap grew beyond 16 MiB allowance")
}
var afterCPU syscall.Rusage
_ = syscall.Getrusage(syscall.RUSAGE_SELF, &afterCPU)
cpu := func(r syscall.Rusage) float64 {
return float64(r.Utime.Sec+r.Stime.Sec) + float64(r.Utime.Usec+r.Stime.Usec)/1e6
}
t.Logf("heap_baseline=%d heap_peak=%d cpu_seconds=%.3f process_maxrss=%d", initial.HeapAlloc, peak.Load(), cpu(afterCPU)-cpu(beforeCPU), afterCPU.Maxrss)
t.Logf("parts=%d source=%+v metrics=%+v encoded=%d build_and_cold_query=%s generation=%s", n, source, health.Metrics, health.EncodedBytes, time.Since(started), info.ID)
break
}
if time.Now().After(deadline) {
t.Fatal("growth build did not publish")
}
time.Sleep(50 * time.Millisecond)
}
}

View File

@ -0,0 +1,92 @@
package store
import (
"context"
"droboticflow/internal/outputsnapshot"
"droboticflow/internal/workforcestats"
"errors"
"os"
"testing"
"time"
)
func TestAnnotationOutputMongoShardedRestartAndFailedBuild(t *testing.T) {
db := localOutputMongo(t)
s := NewMongoStore(db)
seedAnnotationOutput(t, s)
r := outputsnapshot.NewStreamReader(s, s.StreamAnnotationOutput)
defer r.Close()
q := workforcestats.OutputQuery{From: time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC), To: time.Date(2026, 10, 21, 0, 0, 0, 0, time.UTC)}
deadline := time.Now().Add(5 * time.Second)
var info outputsnapshot.Info
for {
report, current, err := r.Query(t.Context(), "out", []string{"p"}, "", q)
if err != nil {
t.Fatal(err)
}
if report != nil {
if report.Totals.Acceptance.Count != 1 {
t.Fatal("wrong persisted report")
}
info = current
break
}
if time.Now().After(deadline) {
t.Fatal("not published")
}
time.Sleep(time.Millisecond)
}
restarted := outputsnapshot.NewStreamReader(NewMongoStore(db), s.StreamAnnotationOutput)
defer restarted.Close()
report, got, err := restarted.Query(t.Context(), "out", []string{"p"}, info.ID, q)
if err != nil || report == nil || got.ID != info.ID || report.Totals.Acceptance.Count != 1 {
t.Fatalf("restart mismatch %v %+v", err, got)
}
state, err := s.OutputSnapshotState(t.Context())
if err != nil {
t.Fatal(err)
}
for _, head := range state.Heads {
if head.Current == nil || head.Current.Shards == 0 {
t.Fatal("legacy single blob used")
}
}
now := time.Now()
old, ok, err := s.AcquireOutputSnapshot(t.Context(), "isolated", now)
if err != nil || !ok {
t.Fatal("lease unavailable", err)
}
next, ok, err := s.AcquireOutputSnapshot(t.Context(), "isolated", now.Add(outputsnapshot.LeaseDuration+time.Second))
if err != nil || !ok {
t.Fatal("lease recovery", err)
}
if ok, err = s.StageOutputSnapshotShard(t.Context(), old, 0, []byte("stale"), now.Add(outputsnapshot.LeaseDuration+time.Second)); err != nil || ok {
t.Fatal("stale writer staged", err)
}
if ok, err = s.StageOutputSnapshotShard(t.Context(), next, 0, []byte("new"), now.Add(outputsnapshot.LeaseDuration+time.Second)); err != nil || !ok {
t.Fatal("new writer unable to stage", err)
}
}
func TestAnnotationOutputStreamFailureCleansPrivateFiles(t *testing.T) {
root := t.TempDir()
t.Setenv("TMPDIR", root)
s := NewMongoStore(localOutputMongo(t))
seedAnnotationOutput(t, s)
sentinel := errors.New("consumer failed")
_, err := s.StreamAnnotationOutput(context.Background(), "out", []string{"p"}, func(workforcestats.OutputSnapshot) error { return sentinel })
if !errors.Is(err, sentinel) {
t.Fatal("consumer failure lost", err)
}
entries, err := os.ReadDir(root)
if err != nil || len(entries) != 0 {
t.Fatalf("temporary files leaked: %d %v", len(entries), err)
}
_, err = s.StreamAnnotationOutput(context.Background(), "out", []string{"p"}, func(workforcestats.OutputSnapshot) error { return workforcestats.ErrOutputPartitionSplit })
if !errors.Is(err, workforcestats.ErrOutputSourceBudget) {
t.Fatal("indivisible part did not fail explicitly", err)
}
entries, err = os.ReadDir(root)
if err != nil || len(entries) != 0 {
t.Fatal("overflow files leaked")
}
}

View File

@ -4,6 +4,7 @@ import (
"context"
"droboticflow/internal/domain"
"droboticflow/internal/outputsnapshot"
"fmt"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
@ -79,7 +80,7 @@ func (s *MongoStore) RenewOutputSnapshot(ctx context.Context, l outputsnapshot.L
return err == nil && res.MatchedCount == 1, err
}
func (s *MongoStore) PublishOutputSnapshot(ctx context.Context, l outputsnapshot.Lease, g outputsnapshot.Generation, data []byte, now time.Time) (bool, error) {
if g.Slot != l.Slot || g.Epoch != l.Epoch || len(data) > outputsnapshot.MaxEncodedBytes || g.Bytes != len(data) || l.Slot < 0 || l.Slot >= outputsnapshot.SlotCount {
if !outputsnapshot.ValidPublication(l, g, data) {
return false, outputsnapshot.ErrCapacity
}
ok, err := s.RenewOutputSnapshot(ctx, l, now)
@ -99,6 +100,12 @@ func (s *MongoStore) PublishOutputSnapshot(ctx context.Context, l outputsnapshot
}
head := outputsnapshot.Head{Current: &g, Previous: l.Previous, NextAttempt: g.AsOf.Add(outputsnapshot.RefreshInterval)}
res, err := s.outputCoordinator().UpdateOne(write, outputLeaseFilter(l, now), bson.M{"$set": bson.M{"heads." + l.Key: head, "owner": "", "until": now}})
if err == nil && res.MatchedCount == 1 {
// Same slot only: current and previous generations occupy different slots.
// Epoch-qualified immutable keys protect against late expired writers.
prefix := fmt.Sprintf("of-shard:%02d:", l.Slot)
_, _ = s.outputBlobs().DeleteMany(write, bson.M{"_id": bson.M{"$gte": prefix, "$lt": fmt.Sprintf("%s%020d:", prefix, l.Epoch)}})
}
return err == nil && res.MatchedCount == 1, err
}
func (s *MongoStore) FailOutputSnapshot(ctx context.Context, l outputsnapshot.Lease, now time.Time, failure outputsnapshot.Failure) error {
@ -115,3 +122,42 @@ func (s *MongoStore) OutputSnapshotBlob(ctx context.Context, g outputsnapshot.Ge
}
return doc.Data, err
}
func outputShardID(slot int, epoch int64, index int) string {
return fmt.Sprintf("of-shard:%02d:%020d:%09d", slot, epoch, index)
}
func (s *MongoStore) StageOutputSnapshotShard(ctx context.Context, l outputsnapshot.Lease, index int, data []byte, now time.Time) (bool, error) {
if l.Slot < 0 || l.Slot >= outputsnapshot.SlotCount || index < 0 || index >= outputsnapshot.MaxShards || len(data) > outputsnapshot.MaxEncodedBytes {
return false, outputsnapshot.ErrCapacity
}
ok, err := s.RenewOutputSnapshot(ctx, l, now)
if err != nil || !ok {
return false, err
}
if index == 0 {
prefix := fmt.Sprintf("of-shard:%02d:", l.Slot)
if _, err = s.outputBlobs().DeleteMany(ctx, bson.M{"_id": bson.M{"$gte": prefix, "$lt": fmt.Sprintf("%s%020d:", prefix, l.Epoch)}}); err != nil {
return false, err
}
}
// Immutable: never overwrite a staged shard.
id := outputShardID(l.Slot, l.Epoch, index)
_, err = s.outputBlobs().InsertOne(ctx, bson.M{"_id": id, "data": data})
return err == nil, err
}
func (s *MongoStore) OutputSnapshotShard(ctx context.Context, g outputsnapshot.Generation, index int) ([]byte, error) {
var doc struct {
Data []byte `bson:"data"`
}
err := s.outputBlobs().FindOne(ctx, bson.M{"_id": outputShardID(g.Slot, g.Epoch, index)}, options.FindOne().SetProjection(bson.M{"data": 1})).Decode(&doc)
if err == mongo.ErrNoDocuments {
err = outputsnapshot.ErrExpired
}
return doc.Data, err
}
func (s *MemoryStore) StageOutputSnapshotShard(ctx context.Context, l outputsnapshot.Lease, i int, b []byte, t time.Time) (bool, error) {
return s.outputSnapshots.StageOutputSnapshotShard(ctx, l, i, b, t)
}
func (s *MemoryStore) OutputSnapshotShard(ctx context.Context, g outputsnapshot.Generation, i int) ([]byte, error) {
return s.outputSnapshots.OutputSnapshotShard(ctx, g, i)
}

View File

@ -0,0 +1,45 @@
package store
import (
"os"
"path/filepath"
"strings"
"syscall"
"time"
)
// Private build files have no recovery value. A live advisory lock protects
// slow/canceling builds; after a process crash the OS releases it. Restrict
// cleanup to old, real directories with our marker, never follow symlinks.
func cleanupOutputSpools(root string, now time.Time) {
entries, err := os.ReadDir(root)
if err != nil {
return
}
cleaned := 0
for _, entry := range entries {
if cleaned >= 64 {
return
}
if !entry.IsDir() || !strings.HasPrefix(entry.Name(), "droboticflow-output-") {
continue
}
path := filepath.Join(root, entry.Name())
info, err := os.Lstat(path)
if err != nil || !info.IsDir() || now.Sub(info.ModTime()) < 10*time.Minute {
continue
}
guard, err := os.OpenFile(filepath.Join(path, ".build-lock"), os.O_RDWR, 0)
if err != nil {
continue
}
if syscall.Flock(int(guard.Fd()), syscall.LOCK_EX|syscall.LOCK_NB) != nil {
guard.Close()
continue
}
_ = os.RemoveAll(path)
_ = syscall.Flock(int(guard.Fd()), syscall.LOCK_UN)
guard.Close()
cleaned++
}
}

View File

@ -0,0 +1,226 @@
package store
import (
"bytes"
"context"
"fmt"
"os"
"sync"
"time"
"droboticflow/internal/workforcestats"
bolt "go.etcd.io/bbolt"
"go.mongodb.org/mongo-driver/bson"
)
// Rebuildable, per-build temporary associations, never customer persistence.
// Transactions are bounded; no read transaction survives a callback that may
// write/grow the mmap. All returned bytes are owned copies.
type outputSpoolIndex struct {
mu sync.Mutex
db *bolt.DB
tx *bolt.Tx
pending, operations, peak int
disk int64
charge func(int64) error
}
func newOutputSpoolIndex(path string, charge func(int64) error) (*outputSpoolIndex, error) {
db, e := bolt.Open(path, 0600, &bolt.Options{Timeout: time.Second, NoSync: true, NoFreelistSync: true})
if e != nil {
return nil, e
}
return &outputSpoolIndex{db: db, charge: charge}, nil
}
func outputIndexKey(key string) []byte { return append([]byte{1}, []byte(key)...) }
func (k *outputSpoolIndex) commitLocked() error {
if k.tx != nil {
e := k.tx.Commit()
k.tx = nil
k.pending = 0
k.operations = 0
if e != nil {
return e
}
}
var e error
var size int64
// DataSize is the mapped address-space size, not a retained Go object graph.
// Charge the actual backing file rather than speculative mmap growth.
if stat, se := k.dbFileSize(); se == nil {
size = stat
} else {
e = se
}
if e != nil {
return e
}
if size > k.disk {
if e = k.charge(size - k.disk); e != nil {
return e
}
k.disk = size
}
return nil
}
func (k *outputSpoolIndex) putLocked(table, key string, value []byte) error {
if len(value) > 16<<20 {
return fmt.Errorf("%w: one reporting association exceeds BSON size", workforcestats.ErrOutputSourceBudget)
}
if k.tx != nil && (k.pending+len(value) > 2<<20 || k.operations >= 512) {
if e := k.commitLocked(); e != nil {
return e
}
}
if k.tx == nil {
tx, e := k.db.Begin(true)
if e != nil {
return e
}
k.tx = tx
}
bucket, e := k.tx.CreateBucketIfNotExists([]byte(table))
if e != nil {
return e
}
keyBytes := outputIndexKey(key)
if e = bucket.Put(keyBytes, append([]byte(nil), value...)); e != nil {
return e
}
k.pending += len(value) + len(keyBytes) + 64
k.operations++
k.peak = max(k.peak, k.pending)
return nil
}
func (k *outputSpoolIndex) put(table, key string, value []byte) error {
k.mu.Lock()
defer k.mu.Unlock()
return k.putLocked(table, key, value)
}
func (k *outputSpoolIndex) getLocked(table, key string) ([]byte, error) {
var out []byte
read := func(tx *bolt.Tx) error {
if b := tx.Bucket([]byte(table)); b != nil {
out = append(out, b.Get(outputIndexKey(key))...)
}
return nil
}
if k.tx != nil {
e := read(k.tx)
return out, e
}
e := k.db.View(read)
return out, e
}
func (k *outputSpoolIndex) get(table, key string) ([]byte, error) {
k.mu.Lock()
defer k.mu.Unlock()
return k.getLocked(table, key)
}
func (k *outputSpoolIndex) contains(table, key string) (bool, error) {
v, e := k.get(table, key)
return len(v) > 0, e
}
func (k *outputSpoolIndex) addParts(table, key string, parts []string) error {
if key == "" {
return nil
}
parts = uniqueOutputParts(parts)
if len(parts) == 0 {
return nil
}
k.mu.Lock()
defer k.mu.Unlock()
raw, e := k.getLocked(table, key)
if e != nil {
return e
}
var old struct {
Parts []string `bson:"p"`
}
if len(raw) > 0 {
if e = bson.Unmarshal(raw, &old); e != nil {
return e
}
}
value, e := bson.Marshal(bson.M{"p": uniqueOutputParts(append(old.Parts, parts...))})
if e != nil {
return e
}
return k.putLocked(table, key, value)
}
func (k *outputSpoolIndex) parts(table, key string) ([]string, error) {
raw, e := k.get(table, key)
if e != nil || len(raw) == 0 {
return nil, e
}
var v struct {
Parts []string `bson:"p"`
}
e = bson.Unmarshal(raw, &v)
return v.Parts, e
}
func (k *outputSpoolIndex) eachKeys(ctx context.Context, table string, visit func([]string) error) error {
var last []byte
for {
if e := ctx.Err(); e != nil {
return e
}
keys := []string{}
k.mu.Lock()
read := func(tx *bolt.Tx) error {
b := tx.Bucket([]byte(table))
if b == nil {
return nil
}
c := b.Cursor()
key, _ := c.First()
if last != nil {
key, _ = c.Seek(last)
if bytes.Equal(key, last) {
key, _ = c.Next()
}
}
for key != nil && len(keys) < 256 {
keys = append(keys, string(key[1:]))
last = append(last[:0], key...)
key, _ = c.Next()
}
return nil
}
var e error
if k.tx != nil {
e = read(k.tx)
} else {
e = k.db.View(read)
}
k.mu.Unlock()
if e != nil {
return e
}
if len(keys) == 0 {
return nil
}
if e = visit(keys); e != nil {
return e
}
}
}
func (k *outputSpoolIndex) flush() error { k.mu.Lock(); defer k.mu.Unlock(); return k.commitLocked() }
func (k *outputSpoolIndex) close() error {
k.mu.Lock()
defer k.mu.Unlock()
if k.tx != nil {
_ = k.tx.Rollback()
k.tx = nil
}
return k.db.Close()
}
func (k *outputSpoolIndex) dbFileSize() (int64, error) {
v, e := os.Stat(k.db.Path())
if e != nil {
return 0, e
}
return v.Size(), nil
}

View File

@ -0,0 +1,94 @@
package store
import (
"context"
"fmt"
"os"
"path/filepath"
"testing"
"time"
)
func TestOutputSpoolIndexCopiesPagesAndAllowsWritesDuringIteration(t *testing.T) {
var disk int64
index, err := newOutputSpoolIndex(filepath.Join(t.TempDir(), "index.db"), func(n int64) error { disk += n; return nil })
if err != nil {
t.Fatal(err)
}
defer index.close()
value := []byte("before")
for i := 0; i < 1600; i++ {
if err = index.put("a", fmt.Sprintf("%05d", i), value); err != nil {
t.Fatal(err)
}
}
value[0] = 'X'
got, err := index.get("a", "00000")
if err != nil || string(got) != "before" {
t.Fatal("borrowed input", string(got), err)
}
got[0] = 'Z'
visited := 0
err = index.eachKeys(context.Background(), "a", func(keys []string) error {
if len(keys) > 256 {
t.Fatal("unbounded page")
}
for _, key := range keys {
visited++
if err := index.put("b", key, make([]byte, 2048)); err != nil {
return err
}
}
return nil
})
if err != nil || visited != 1600 {
t.Fatalf("page loss %d %v", visited, err)
}
got, err = index.get("a", "00000")
if err != nil || string(got) != "before" {
t.Fatal("borrowed result", string(got), err)
}
if err = index.flush(); err != nil || disk == 0 {
t.Fatalf("disk accounting %d %v", disk, err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if err = index.eachKeys(ctx, "a", func([]string) error { t.Fatal("cancel ignored"); return nil }); err == nil {
t.Fatal("cancel ignored")
}
}
func TestOutputSpoolCleanupRecoversCrashedBuildButKeepsActive(t *testing.T) {
root := t.TempDir()
t.Setenv("TMPDIR", root)
active, err := newOutputSourceSpool(t.Context())
if err != nil {
t.Fatal(err)
}
defer active.close()
abandoned, err := newOutputSourceSpool(t.Context())
if err != nil {
t.Fatal(err)
}
if err = abandoned.index.close(); err != nil {
t.Fatal(err)
}
abandoned.index = nil
if err = abandoned.guard.Close(); err != nil {
t.Fatal(err)
}
abandoned.guard = nil
old := time.Now().Add(-time.Hour)
for _, path := range []string{active.root, abandoned.root} {
if err = os.Chtimes(path, old, old); err != nil {
t.Fatal(err)
}
}
cleanupOutputSpools(root, time.Now())
if _, err = os.Stat(active.root); err != nil {
t.Fatal("deleted active build", err)
}
if _, err = os.Stat(abandoned.root); !os.IsNotExist(err) {
t.Fatal("abandoned build not cleaned", err)
}
}

View File

@ -0,0 +1,847 @@
package store
import (
"bufio"
"context"
"crypto/sha256"
"encoding/binary"
"encoding/hex"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"sync"
"syscall"
"time"
"droboticflow/internal/domain"
"droboticflow/internal/workforcestats"
"go.mongodb.org/mongo-driver/bson"
)
const outputBatchTarget = 8 << 20
// A build spills projected metadata to private temporary files. The directory
// retains identities, not revision/decision history. Only one bounded canonical
// part partition is interpreted at a time; a part never spans partitions.
type outputSourceSpool struct {
ctx context.Context
root string
guard *os.File
mu sync.Mutex
files map[string]*os.File
disk, peakDisk int64
directoryBytes int
index *outputSpoolIndex
directories workforcestats.OutputSnapshot
}
type outputTaskRef struct {
ID string `bson:"id"`
Project string `bson:"projectId"`
Run string `bson:"runId"`
Data string `bson:"dataTaskId"`
}
type outputItemRef struct {
ID string `bson:"id"`
Task string `bson:"legacyTaskId"`
Release string `bson:"releasePartitionId"`
}
type outputRunRef struct {
ID string `bson:"id"`
Task string `bson:"taskId"`
}
type outputEnvelope struct {
Kind string `bson:"k"`
Parts []string `bson:"p"`
Video string `bson:"v,omitempty"`
Data []byte `bson:"d"`
}
func newOutputSourceSpool(ctx context.Context) (*outputSourceSpool, error) {
cleanupOutputSpools(os.TempDir(), time.Now())
root, err := os.MkdirTemp("", "droboticflow-output-")
if err != nil {
return nil, err
}
p := &outputSourceSpool{ctx: ctx, root: root, files: map[string]*os.File{}}
p.guard, err = os.OpenFile(filepath.Join(root, ".build-lock"), os.O_CREATE|os.O_RDWR, 0600)
if err == nil {
err = syscall.Flock(int(p.guard.Fd()), syscall.LOCK_EX|syscall.LOCK_NB)
}
if err != nil {
if p.guard != nil {
_ = p.guard.Close()
}
_ = os.RemoveAll(root)
return nil, err
}
p.index, err = newOutputSpoolIndex(filepath.Join(root, "associations.db"), p.reserve)
if err != nil {
_ = p.guard.Close()
_ = os.RemoveAll(root)
return nil, err
}
return p, nil
}
func (p *outputSourceSpool) close() {
for _, f := range p.files {
_ = f.Close()
}
if p.index != nil {
_ = p.index.close()
}
_ = os.RemoveAll(p.root)
if p.guard != nil {
_ = p.guard.Close()
}
}
func (p *outputSourceSpool) reserve(n int64) error {
p.mu.Lock()
defer p.mu.Unlock()
p.disk += n
if p.disk > p.peakDisk {
p.peakDisk = p.disk
}
if p.disk > workforcestats.OutputSpoolMaxBytes {
return fmt.Errorf("%w: temporary spool budget", workforcestats.ErrOutputSourceBudget)
}
return nil
}
func writeOutputRecord(w io.Writer, raw []byte) error {
var size [4]byte
binary.LittleEndian.PutUint32(size[:], uint32(len(raw)))
if _, e := w.Write(size[:]); e != nil {
return e
}
_, e := w.Write(raw)
return e
}
func readOutputRecords(ctx context.Context, path string, visit func([]byte) error) error {
f, e := os.Open(path)
if errors.Is(e, os.ErrNotExist) {
return nil
}
if e != nil {
return e
}
defer f.Close()
r := bufio.NewReaderSize(f, 64<<10)
for {
if e := ctx.Err(); e != nil {
return e
}
var size [4]byte
_, e := io.ReadFull(r, size[:])
if e == io.EOF {
return nil
}
if e != nil {
return e
}
n := int(binary.LittleEndian.Uint32(size[:]))
if n < 5 || n > 32<<20 {
return fmt.Errorf("invalid reporting spool record size")
}
raw := make([]byte, n)
if _, e = io.ReadFull(r, raw); e != nil {
return e
}
if e = visit(raw); e != nil {
return e
}
}
}
func (p *outputSourceSpool) add(kind string, raw []byte) error {
if e := p.ctx.Err(); e != nil {
return e
}
directory := false
switch kind {
case "annotationTasks":
var v outputTaskRef
if e := bson.Unmarshal(raw, &v); e != nil {
return e
}
if e := p.index.put("tasks", v.ID, raw); e != nil {
return e
}
if v.Run != "" {
if e := p.index.put("runRefs", v.Run, []byte{1}); e != nil {
return e
}
}
case "humanItems":
var v outputItemRef
if e := bson.Unmarshal(raw, &v); e != nil {
return e
}
if e := p.index.put("items", v.ID, raw); e != nil {
return e
}
case "captureRecords":
var v struct {
ID string `bson:"id"`
Run string `bson:"runId"`
}
if e := bson.Unmarshal(raw, &v); e != nil {
return e
}
if e := p.index.put("captures", v.ID, raw); e != nil {
return e
}
if v.Run != "" {
if e := p.index.put("captureByRun", v.Run, []byte(v.ID)); e != nil {
return e
}
if e := p.index.put("runRefs", v.Run, []byte{1}); e != nil {
return e
}
}
case "runs":
var v outputRunRef
if e := bson.Unmarshal(raw, &v); e != nil {
return e
}
if e := p.index.put("runs", v.ID, raw); e != nil {
return e
}
has, e := p.index.contains("captures", v.Task)
if e != nil {
return e
}
if has {
if e = p.index.put("captureByRun", v.ID, []byte(v.Task)); e != nil {
return e
}
}
case "users":
var v domain.UserRecord
if e := bson.Unmarshal(raw, &v); e != nil {
return e
}
p.directories.Users = append(p.directories.Users, v)
directory = true
case "suppliers":
var v domain.SupplierRecord
if e := bson.Unmarshal(raw, &v); e != nil {
return e
}
p.directories.Suppliers = append(p.directories.Suppliers, v)
directory = true
case "projects":
var v domain.ProjectRecord
if e := bson.Unmarshal(raw, &v); e != nil {
return e
}
p.directories.Projects = append(p.directories.Projects, v)
directory = true
}
if directory {
p.mu.Lock()
p.directoryBytes += len(raw) + 512
over := p.directoryBytes > workforcestats.OutputDirectoryMaxBytes
p.mu.Unlock()
if over {
return fmt.Errorf("%w: display directory budget", workforcestats.ErrOutputSourceBudget)
}
}
if kind == "captureRecords" || kind == "users" || kind == "suppliers" || kind == "projects" {
return nil
}
if e := p.reserve(int64(len(raw) + 4)); e != nil {
return e
}
p.mu.Lock()
defer p.mu.Unlock()
f := p.files[kind]
if f == nil {
var e error
f, e = os.OpenFile(filepath.Join(p.root, kind+".raw"), os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0600)
if e != nil {
return e
}
p.files[kind] = f
}
return writeOutputRecord(f, raw)
}
func (p *outputSourceSpool) identities() error {
if e := p.index.eachKeys(p.ctx, "tasks", func(ids []string) error {
for _, id := range ids {
raw, e := p.index.get("tasks", id)
if e != nil {
return e
}
var t outputTaskRef
if e = bson.Unmarshal(raw, &t); e != nil {
return e
}
capID, e := p.index.get("captureByRun", t.Run)
if e != nil {
return e
}
cRaw, e := p.index.get("captures", string(capID))
if e != nil {
return e
}
if len(cRaw) == 0 {
cRaw, e = p.index.get("captures", t.Data)
if e != nil {
return e
}
}
var capture *domain.CaptureRecord
if len(cRaw) > 0 {
var c domain.CaptureRecord
if e = bson.Unmarshal(cRaw, &c); e != nil {
return e
}
capture = &c
}
identity := workforcestats.OutputPartIdentityForCapture(t.Project, t.ID, capture)
value, e := bson.Marshal(identity)
if e != nil {
return e
}
if e = p.index.put("taskParts", id, value); e != nil {
return e
}
if e = p.index.addParts("runParts", t.Run, []string{identity.ID}); e != nil {
return e
}
}
return nil
}); e != nil {
return e
}
return p.index.eachKeys(p.ctx, "items", func(ids []string) error {
for _, id := range ids {
raw, e := p.index.get("items", id)
if e != nil {
return e
}
var i outputItemRef
if e = bson.Unmarshal(raw, &i); e != nil {
return e
}
identity, e := p.taskIdentity(i.Task)
if e != nil {
return e
}
if identity.ID != "" {
if e = p.index.put("itemParts", id, []byte(identity.ID)); e != nil {
return e
}
if e = p.index.addParts("releaseParts", i.Release, []string{identity.ID}); e != nil {
return e
}
}
}
return nil
})
}
func (p *outputSourceSpool) taskIdentity(id string) (workforcestats.OutputPartIdentity, error) {
var v workforcestats.OutputPartIdentity
raw, e := p.index.get("taskParts", id)
if e == nil && len(raw) > 0 {
e = bson.Unmarshal(raw, &v)
}
return v, e
}
func outputPartHash(part string) string {
h := sha256.Sum256([]byte(part))
return hex.EncodeToString(h[:])
}
func uniqueOutputParts(parts []string) []string {
out := make([]string, 0, len(parts))
seen := map[string]bool{}
for _, part := range parts {
if part != "" && !seen[part] {
seen[part] = true
out = append(out, part)
}
}
return out
}
func (p *outputSourceSpool) route(kind string, raw []byte) (outputEnvelope, error) {
e := outputEnvelope{Kind: kind, Data: raw}
var routeErr error
task := func(id string) {
v, err := p.taskIdentity(id)
if err != nil {
routeErr = err
}
e.Parts = append(e.Parts, v.ID)
}
item := func(id string) {
v, err := p.index.get("itemParts", id)
if err != nil {
routeErr = err
}
e.Parts = append(e.Parts, string(v))
}
refs := func(table, id string) {
v, err := p.index.parts(table, id)
if err != nil {
routeErr = err
}
e.Parts = append(e.Parts, v...)
}
switch kind {
case "annotationTasks":
var v outputTaskRef
if err := bson.Unmarshal(raw, &v); err != nil {
return e, err
}
identity, err := p.taskIdentity(v.ID)
if err != nil {
return e, err
}
e.Parts = append(e.Parts, identity.ID)
e.Video = identity.Video
case "humanItems":
var v outputItemRef
if err := bson.Unmarshal(raw, &v); err != nil {
return e, err
}
item(v.ID)
case "runs":
var v outputRunRef
if err := bson.Unmarshal(raw, &v); err != nil {
return e, err
}
refs("runParts", v.ID)
case "annotationWorklogEvents":
var v domain.AnnotationWorklogEventRecord
if err := bson.Unmarshal(raw, &v); err != nil {
return e, err
}
task(v.TaskID)
if err := p.index.addParts("worklogParts", v.ID, e.Parts); err != nil {
return e, err
}
case "annotationReviews":
var v domain.AnnotationReviewRecord
if err := bson.Unmarshal(raw, &v); err != nil {
return e, err
}
task(v.TaskID)
if err := p.index.addParts("reviewParts", v.ID, e.Parts); err != nil {
return e, err
}
case "annotationRevisions":
var v domain.AnnotationRevisionRecord
if err := bson.Unmarshal(raw, &v); err != nil {
return e, err
}
task(v.TaskID)
case "humanEvents":
var v domain.HumanEventRecord
if err := bson.Unmarshal(raw, &v); err != nil {
return e, err
}
for _, id := range outputStringIDs(v.Payload["sourceItemIds"]) {
item(id)
}
if op, ok := v.Payload["operationId"].(string); ok {
if err := p.index.addParts("operationParts", op, e.Parts); err != nil {
return e, err
}
}
case "reworkBaselines":
var v domain.ReworkBaselineRecord
if err := bson.Unmarshal(raw, &v); err != nil {
return e, err
}
item(v.ItemID)
if err := p.index.addParts("operationParts", v.ReviewOperationID, e.Parts); err != nil {
return e, err
}
case "reworkHandlings":
var v domain.ReworkHandlingRecord
if err := bson.Unmarshal(raw, &v); err != nil {
return e, err
}
item(v.ItemID)
case "reviewOperations":
var v domain.ReviewOperation
if err := bson.Unmarshal(raw, &v); err != nil {
return e, err
}
task(v.TaskID)
item(v.ItemID)
for _, id := range v.SourceItemIDs {
item(id)
}
for _, effect := range v.Effects {
if effect.Review != nil {
task(effect.Review.TaskID)
refs("reviewParts", effect.Review.ID)
}
if effect.Worklog != nil {
refs("worklogParts", effect.Worklog.ID)
}
}
refs("operationParts", v.ID)
refs("operationParts", v.OperationID)
case "deliveryPackages":
var v domain.DeliveryPackageRecord
if err := bson.Unmarshal(raw, &v); err != nil {
return e, err
}
for _, id := range v.ItemIDs {
item(id)
}
refs("releaseParts", v.ReleasePartitionID)
default:
return e, fmt.Errorf("unknown reporting spool collection %s", kind)
}
e.Parts = uniqueOutputParts(e.Parts)
return e, routeErr
}
func outputStringIDs(v any) []string {
switch v := v.(type) {
case bson.A:
out := []string{}
for _, x := range v {
if id, ok := x.(string); ok {
out = append(out, id)
}
}
return out
case []string:
return v
case []any:
out := []string{}
for _, x := range v {
if id, ok := x.(string); ok {
out = append(out, id)
}
}
return out
}
return nil
}
func (p *outputSourceSpool) remove(path string) {
if info, e := os.Stat(path); e == nil {
_ = os.Remove(path)
p.mu.Lock()
p.disk -= info.Size()
p.mu.Unlock()
}
}
func (p *outputSourceSpool) partition(writers map[string]*bufio.Writer, files map[string]*os.File, prefix string, e outputEnvelope) error {
groups := map[string][]string{}
for _, part := range e.Parts {
h := outputPartHash(part)
if strings.HasPrefix(h, prefix) {
key := h[:len(prefix)+1]
groups[key] = append(groups[key], part)
}
}
for key, parts := range groups {
copy := e
copy.Parts = parts
raw, err := bson.Marshal(copy)
if err != nil {
return err
}
if err = p.reserve(int64(len(raw) + 4)); err != nil {
return err
}
w := writers[key]
if w == nil {
f, err := os.OpenFile(filepath.Join(p.root, "part-"+key), os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0600)
if err != nil {
return err
}
files[key] = f
w = bufio.NewWriterSize(f, 64<<10)
writers[key] = w
}
if err = writeOutputRecord(w, raw); err != nil {
return err
}
}
return nil
}
func closeOutputWriters(writers map[string]*bufio.Writer, files map[string]*os.File) error {
var first error
for key, w := range writers {
if e := w.Flush(); first == nil {
first = e
}
if e := files[key].Close(); first == nil {
first = e
}
}
return first
}
// StreamAnnotationOutput preserves the old source interpreter and API contracts,
// but replaces its global history-size cliff with per-part bounded partitions.
func (s *MongoStore) StreamAnnotationOutput(ctx context.Context, org string, projects []string, yield func(workforcestats.OutputSnapshot) error) (*workforcestats.OutputSourceStats, error) {
if len(projects) == 0 {
return &workforcestats.OutputSourceStats{}, nil
}
p, err := newOutputSourceSpool(ctx)
if err != nil {
return nil, err
}
defer p.close()
raw, err := s.loadAnnotationOutputTo(ctx, org, projects, outputSourceBudget{}, p)
if err != nil {
return nil, err
}
for _, f := range p.files {
if err = f.Close(); err != nil {
return nil, err
}
}
p.files = map[string]*os.File{}
if err = p.identities(); err != nil {
return nil, err
}
writers, files := map[string]*bufio.Writer{}, map[string]*os.File{}
// References must be resolved before operations are routed; acceptance can
// cover parts in other projects/buckets and pending plans still fence events.
order := []string{"annotationTasks", "humanItems", "runs", "annotationWorklogEvents", "annotationReviews", "annotationRevisions", "humanEvents", "reworkBaselines", "reworkHandlings", "deliveryPackages", "reviewOperations"}
for _, kind := range order {
path := filepath.Join(p.root, kind+".raw")
err = readOutputRecords(ctx, path, func(b []byte) error {
e, err := p.route(kind, b)
if err != nil {
return err
}
return p.partition(writers, files, "", e)
})
if err != nil {
_ = closeOutputWriters(writers, files)
return nil, err
}
p.remove(path)
}
if err = closeOutputWriters(writers, files); err != nil {
return nil, err
}
// Association history stays on disk; release its mmap/transactions before
// interpreting batches. Each envelope now owns its canonical routing.
if err = p.index.flush(); err != nil {
return nil, err
}
stats := raw.SourceStats
if stats == nil {
stats = &workforcestats.OutputSourceStats{}
}
stats.DirectoryBytes = p.directoryBytes + p.index.peak
if err = p.index.close(); err != nil {
return nil, err
}
p.index = nil
p.remove(filepath.Join(p.root, "associations.db"))
var emit func(string) error
emit = func(prefix string) error {
path := filepath.Join(p.root, "part-"+prefix)
info, e := os.Stat(path)
if errors.Is(e, os.ErrNotExist) {
return nil
}
if e != nil {
return e
}
split := func() error {
// An indivisible part must fail explicitly, not spill 64 identical copies.
first := ""
multiple := errors.New("multiple canonical parts")
check := readOutputRecords(ctx, path, func(b []byte) error {
var env outputEnvelope
if err := bson.Unmarshal(b, &env); err != nil {
return err
}
for _, part := range env.Parts {
if first == "" {
first = part
} else if part != first {
return multiple
}
}
return nil
})
if check != nil && !errors.Is(check, multiple) {
return check
}
if check == nil {
return fmt.Errorf("%w: one canonical part exceeds partition budget", workforcestats.ErrOutputSourceBudget)
}
if len(prefix) >= 64 {
return fmt.Errorf("%w: one canonical part exceeds partition budget", workforcestats.ErrOutputSourceBudget)
}
ws, fs := map[string]*bufio.Writer{}, map[string]*os.File{}
e := readOutputRecords(ctx, path, func(b []byte) error {
var env outputEnvelope
if e := bson.Unmarshal(b, &env); e != nil {
return e
}
return p.partition(ws, fs, prefix, env)
})
ce := closeOutputWriters(ws, fs)
if e != nil {
return e
}
if ce != nil {
return ce
}
p.remove(path)
for _, c := range "0123456789abcdef" {
if e := emit(prefix + string(c)); e != nil {
return e
}
}
return nil
}
if info.Size() > outputBatchTarget {
distinct := map[string]bool{}
rawBytes := 0
stop := errors.New("multiple parts")
check := readOutputRecords(ctx, path, func(b []byte) error {
var env outputEnvelope
if e := bson.Unmarshal(b, &env); e != nil {
return e
}
rawBytes += len(env.Data)
for _, part := range env.Parts {
distinct[part] = true
if len(distinct) > 1 {
return stop
}
}
if rawBytes > workforcestats.OutputSourceMaxBytes {
return fmt.Errorf("%w: one canonical part exceeds source budget", workforcestats.ErrOutputSourceBudget)
}
return nil
})
if errors.Is(check, stop) {
return split()
}
if check != nil {
return check
}
}
d := p.directories
d.Identities = map[string]workforcestats.OutputPartIdentity{}
docs, bytes := 0, 0
e = readOutputRecords(ctx, path, func(b []byte) error {
var env outputEnvelope
if e := bson.Unmarshal(b, &env); e != nil {
return e
}
docs++
bytes += len(env.Data)
if bytes > workforcestats.OutputSourceMaxBytes || docs > workforcestats.OutputSourceMaxDocuments {
return workforcestats.ErrOutputPartitionSplit
}
return appendOutputEnvelope(&d, env)
})
if errors.Is(e, workforcestats.ErrOutputPartitionSplit) {
return split()
}
if e != nil {
return e
}
if bytes > workforcestats.OutputSourceMaxBytes || docs > workforcestats.OutputSourceMaxDocuments {
return split()
}
if e = yield(d); errors.Is(e, workforcestats.ErrOutputPartitionSplit) {
return split()
} else if e != nil {
return e
}
stats.Partitions++
stats.PeakBatchBytes = max(stats.PeakBatchBytes, bytes)
stats.PeakBatchDocuments = max(stats.PeakBatchDocuments, docs)
p.remove(path)
return nil
}
for _, c := range "0123456789abcdef" {
if err = emit(string(c)); err != nil {
return nil, err
}
}
stats.PeakSpoolBytes = p.peakDisk
return stats, nil
}
func appendOutputEnvelope(d *workforcestats.OutputSnapshot, e outputEnvelope) error {
switch e.Kind {
case "annotationTasks":
var v domain.AnnotationTaskRecord
if err := bson.Unmarshal(e.Data, &v); err != nil {
return err
}
d.Tasks = append(d.Tasks, v)
d.Identities[v.ID] = workforcestats.OutputPartIdentity{ID: e.Parts[0], Video: e.Video}
case "humanItems":
var v domain.HumanItemRecord
if err := bson.Unmarshal(e.Data, &v); err != nil {
return err
}
d.Items = append(d.Items, v)
case "runs":
var v domain.RunRecord
if err := bson.Unmarshal(e.Data, &v); err != nil {
return err
}
d.Runs = append(d.Runs, v)
case "annotationWorklogEvents":
var v domain.AnnotationWorklogEventRecord
if err := bson.Unmarshal(e.Data, &v); err != nil {
return err
}
d.Events = append(d.Events, v)
case "annotationReviews":
var v domain.AnnotationReviewRecord
if err := bson.Unmarshal(e.Data, &v); err != nil {
return err
}
d.Reviews = append(d.Reviews, v)
case "annotationRevisions":
var v domain.AnnotationRevisionRecord
if err := bson.Unmarshal(e.Data, &v); err != nil {
return err
}
d.Revisions = append(d.Revisions, v)
case "humanEvents":
var v domain.HumanEventRecord
if err := bson.Unmarshal(e.Data, &v); err != nil {
return err
}
d.HumanEvents = append(d.HumanEvents, v)
case "reworkBaselines":
var v domain.ReworkBaselineRecord
if err := bson.Unmarshal(e.Data, &v); err != nil {
return err
}
d.Baselines = append(d.Baselines, v)
case "reworkHandlings":
var v domain.ReworkHandlingRecord
if err := bson.Unmarshal(e.Data, &v); err != nil {
return err
}
d.Handlings = append(d.Handlings, v)
case "reviewOperations":
var v domain.ReviewOperation
if err := bson.Unmarshal(e.Data, &v); err != nil {
return err
}
d.Operations = append(d.Operations, v)
case "deliveryPackages":
var v domain.DeliveryPackageRecord
if err := bson.Unmarshal(e.Data, &v); err != nil {
return err
}
d.Packages = append(d.Packages, v)
default:
return fmt.Errorf("unknown reporting batch %s", e.Kind)
}
return nil
}

View File

@ -0,0 +1,133 @@
package store
import (
"context"
"fmt"
"reflect"
"testing"
"time"
"droboticflow/internal/domain"
"droboticflow/internal/workforcestats"
"go.mongodb.org/mongo-driver/bson"
)
func collectStreamedOutput(t *testing.T, s *MongoStore, asOf time.Time) (workforcestats.OutputDataset, *workforcestats.OutputSourceStats) {
t.Helper()
d := workforcestats.OutputDataset{AsOf: asOf, Projects: map[string]string{}, Users: map[string]string{}, Suppliers: map[string]string{}}
seen := map[string]bool{}
stats, e := s.StreamAnnotationOutput(t.Context(), "out", []string{"p"}, func(raw workforcestats.OutputSnapshot) error {
batch := workforcestats.BuildOutputDataset(raw, asOf)
offset := len(d.Parts)
for _, part := range batch.Parts {
if seen[part.ID] {
return fmt.Errorf("canonical part split across batches")
}
seen[part.ID] = true
}
d.Parts = append(d.Parts, batch.Parts...)
for _, w := range batch.Work {
w.Part += offset
d.Work = append(d.Work, w)
}
for _, p := range batch.Problems {
p.Part += offset
d.Problems = append(d.Problems, p)
}
for id, name := range batch.Projects {
d.Projects[id] = name
}
for id, name := range batch.Users {
d.Users[id] = name
}
for id, name := range batch.Suppliers {
d.Suppliers[id] = name
}
return nil
})
if e != nil {
t.Fatal(e)
}
return d, stats
}
func TestAnnotationOutputStreamMatchesWholeHistoryAcrossLegacyReferences(t *testing.T) {
db := localOutputMongo(t)
s := NewMongoStore(db)
seedAnnotationOutput(t, s)
ctx := t.Context()
check := func(e error) {
t.Helper()
if e != nil {
t.Fatal(e)
}
}
prefix := "source/c2/part0001"
first := workforcestats.OutputPartIdentityForCapture("p", "t", &domain.CaptureRecord{ProjectID: "p", Prefix: "source/c/part0001", CaptureID: "c/part0001"})
for n := 0; ; n++ {
c := domain.CaptureRecord{ProjectID: "p", Prefix: prefix}
part := workforcestats.OutputPartIdentityForCapture("p", "t2", &c)
if outputPartHash(part.ID)[:1] != outputPartHash(first.ID)[:1] {
break
}
prefix = fmt.Sprintf("source/c2-%d/part0001", n)
}
check(s.UpsertRun(ctx, domain.RunRecord{ID: "r2", OrganizationID: "out", ProjectID: "p", TaskID: "c2", CreatedAt: "2026-10-01T00:00:00Z"}))
check(s.UpsertCaptureRecord(ctx, domain.CaptureRecord{ID: "c2", OrganizationID: "out", ProjectID: "p", RunID: "r2", Prefix: prefix, CaptureID: "c2/part0001"}))
check(s.UpsertAnnotationTask(ctx, domain.AnnotationTaskRecord{ID: "t2", OrganizationID: "out", ProjectID: "p", RunID: "r2", SourceIsFullVideo: true, SourceDurationSeconds: 600, CurrentRevision: 1}))
check(s.UpsertHumanItem(ctx, domain.HumanItemRecord{ID: "i2", OrganizationID: "out", ProjectID: "p", LegacyTaskID: "t2"}))
check(s.UpsertAnnotationRevision(ctx, domain.AnnotationRevisionRecord{ID: "rev2", OrganizationID: "out", TaskID: "t2", Revision: 1, AuthorUserID: "u", CountsAsOutput: true}))
check(s.UpsertAnnotationWorklogEvent(ctx, domain.AnnotationWorklogEventRecord{ID: "s2", OrganizationID: "out", ProjectID: "p", TaskID: "t2", UserID: "u", SupplierID: "v", EventType: "combined_submitted", Revision: 1, CreatedAt: "2026-10-02T00:00:00Z"}))
check(s.UpsertAnnotationReview(ctx, domain.AnnotationReviewRecord{ID: "q2", OrganizationID: "out", TaskID: "t2", Revision: 1, Stage: domain.AnnotationReviewStageQuality, Decision: domain.AnnotationReviewApproved, CreatedAt: "2026-10-03T00:00:00Z"}))
// A second processing attempt for the same capture must stay in the same
// canonical partition; it cannot be independently counted as another part.
check(s.UpsertAnnotationTask(ctx, domain.AnnotationTaskRecord{ID: "retry", OrganizationID: "out", ProjectID: "p", RunID: "r", SourceIsFullVideo: true, SourceDurationSeconds: 900, CurrentRevision: 2}))
check(s.UpsertHumanItem(ctx, domain.HumanItemRecord{ID: "retry-item", OrganizationID: "out", ProjectID: "p", LegacyTaskID: "retry"}))
check(s.UpsertAnnotationRevision(ctx, domain.AnnotationRevisionRecord{ID: "retry-rev", OrganizationID: "out", TaskID: "retry", Revision: 2, AuthorUserID: "u", CountsAsOutput: true}))
check(s.UpsertAnnotationWorklogEvent(ctx, domain.AnnotationWorklogEventRecord{ID: "retry-submit", OrganizationID: "out", ProjectID: "p", TaskID: "retry", UserID: "u", SupplierID: "v", Revision: 2, EventType: "combined_submitted", CreatedAt: "2026-10-04T01:00:00Z"}))
// Missing direct task/source references are legitimate legacy plans: their
// effect IDs and linked human events still have to route the status fence.
op := domain.ReviewOperation{ID: "pending-accept", OperationID: "client-accept", OrganizationID: "out", ProjectID: "p", StageID: "prepackage_acceptance", Status: "applying", Decision: "approved"}
if ok, e := s.CompareAndSwapReviewOperation(ctx, op, 0); e != nil || !ok {
t.Fatalf("seed operation %t %v", ok, e)
}
check(s.AppendHumanEvent(ctx, domain.HumanEventRecord{ID: "linked-both", OrganizationID: "out", ItemID: "i", StageID: "prepackage_acceptance", EventType: "approved", CreatedAt: "2026-10-05T00:00:00Z", Payload: map[string]any{"operationId": "client-accept", "sourceItemIds": []string{"i", "i2"}}}))
// A legacy event without a coordinator ID is unrelated to the first part's
// modern acceptance operation, whose optional client operation ID is empty.
check(s.AppendHumanEvent(ctx, domain.HumanEventRecord{ID: "legacy-second", OrganizationID: "out", ItemID: "i2", StageID: "final_acceptance", EventType: "approved", CreatedAt: "2026-10-04T00:00:00Z", Payload: map[string]any{"sourceItemIds": []string{"i2"}}}))
at := time.Date(2026, 10, 20, 0, 0, 0, 0, time.UTC)
for _, state := range []string{"applying", "completed"} {
t.Run(state, func(t *testing.T) {
_, e := db.Collection("reviewOperations").UpdateOne(ctx, bson.M{"_id": "pending-accept"}, bson.M{"$set": bson.M{"status": state, "sourceItemIds": []string{"i", "i2"}, "completedAt": "2026-10-05T00:00:00Z"}})
check(e)
raw, e := s.LoadAnnotationOutput(ctx, "out", []string{"p"})
check(e)
want := workforcestats.BuildOutputDataset(raw, at)
got, stats := collectStreamedOutput(t, s, at)
if stats.Partitions < 2 {
t.Fatal("fixture did not span partitions")
}
for _, scope := range []struct{ user, vendor string }{{}, {user: "u"}, {vendor: "v"}, {user: "foreign"}, {vendor: "foreign"}} {
for _, production := range []bool{false, true} {
for _, cohort := range []bool{false, true} {
q := workforcestats.OutputQuery{From: time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC), To: at, Now: at, Location: time.UTC, ProductionQuality: production, FirstSubmissionCohort: cohort, MatchUser: scope.user != "", ScopeUserID: scope.user, MatchSupplier: scope.vendor != "", ScopeSupplierID: scope.vendor}
a, b := workforcestats.QueryOutputDataset(got, q), workforcestats.QueryOutputDataset(want, q)
if !reflect.DeepEqual(a, b) {
t.Fatalf("stream changed report production=%t cohort=%t scope=%+v\ngot=%+v\nwant=%+v", production, cohort, scope, a, b)
}
}
}
}
})
}
}
func TestAnnotationOutputStreamCancellationAndEmptyScope(t *testing.T) {
s := NewMongoStore(localOutputMongo(t))
ctx, cancel := context.WithCancel(t.Context())
cancel()
if _, e := s.StreamAnnotationOutput(ctx, "out", []string{"p"}, func(workforcestats.OutputSnapshot) error { return nil }); e == nil {
t.Fatal("cancelled build succeeded")
}
if stats, e := s.StreamAnnotationOutput(t.Context(), "out", nil, func(workforcestats.OutputSnapshot) error { t.Fatal("empty scope emitted data"); return nil }); e != nil || stats.Documents != 0 {
t.Fatalf("empty scope %v %v", stats, e)
}
}

View File

@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"fmt"
"reflect"
"sort"
"strings"
"time"
@ -163,42 +164,56 @@ func (s *MongoStore) loadProductionStats(ctx context.Context, org string, projec
return d, nil
}
filter := organizationScopedFilter(org, bson.M{"projectId": bson.M{"$in": projects}})
read := func(c *mongo.Collection, f bson.M, projection bson.M, target any) error {
opts := options.Find().SetProjection(projection).SetMaxTime(15 * time.Second)
if c.Name() == "captureRecords" {
indexes, e := c.Indexes().List(ctx)
if e == nil {
defer indexes.Close(ctx)
for indexes.Next(ctx) {
var index struct {
Name string `bson:"name"`
}
if e = indexes.Decode(&index); e != nil {
return e
}
if index.Name == "production_capture_cover_v1" {
opts.SetHint(index.Name)
break
}
}
if e = indexes.Err(); e != nil {
return e
}
} else {
var command mongo.CommandError
if !errors.As(e, &command) || command.Code != 26 {
return e
}
hints := map[string]string{}
for collection, wanted := range map[*mongo.Collection]string{s.captureRecords: "production_capture_cover_v1", s.runs: productionRunsCoverIndex, s.humanEvents: productionEventFactsIndex} {
indexes, e := collection.Indexes().List(ctx)
if e != nil {
var command mongo.CommandError
if errors.As(e, &command) && command.Code == 26 {
continue
}
return d, e
}
for indexes.Next(ctx) {
var index struct {
Name string `bson:"name"`
}
if e = indexes.Decode(&index); e != nil {
indexes.Close(ctx)
return d, e
}
if index.Name == wanted {
hints[collection.Name()] = wanted
}
}
e = indexes.Err()
indexes.Close(ctx)
if e != nil {
return d, e
}
}
read := func(c *mongo.Collection, f bson.M, projection bson.M, target any, preferred ...string) error {
opts := options.Find().SetProjection(projection).SetMaxTime(15 * time.Second)
hint := hints[c.Name()]
// A project-leading covering index is unsuitable for legacy task-ID lookups.
if c.Name() == "runs" && len(preferred) == 0 {
hint = ""
}
if hint != "" {
opts.SetHint(hint)
}
cursor, err := c.Find(ctx, f, opts)
if err != nil {
return fmt.Errorf("production stats %s: %w", c.Name(), err)
}
defer cursor.Close(ctx)
if err := cursor.All(ctx, target); err != nil {
value := reflect.ValueOf(target).Elem()
batch := reflect.New(value.Type())
if err := cursor.All(ctx, batch.Interface()); err != nil {
return fmt.Errorf("production stats %s: %w", c.Name(), err)
}
value.Set(reflect.AppendSlice(value, batch.Elem()))
return nil
}
fields := func(names ...string) bson.M {
@ -219,10 +234,51 @@ func (s *MongoStore) loadProductionStats(ctx context.Context, org string, projec
runIDs = append(runIDs, c.RunID)
}
}
runFilter := organizationScopedFilter(org, bson.M{"$or": bson.A{bson.M{"projectId": bson.M{"$in": projects}}, bson.M{"taskId": bson.M{"$in": captureIDs}}, bson.M{"id": bson.M{"$in": runIDs}}}})
// Bound each related-key interval set before Mongo query planning. A single
// OR containing every capture/run ID can take the entire request budget.
readRelated := func(c *mongo.Collection, key string, ids []string, projection bson.M, out any, extra bson.M) error {
unique := make([]string, 0, len(ids))
seen := map[string]bool{}
for _, id := range ids {
if !seen[id] {
seen[id] = true
unique = append(unique, id)
}
}
for start := 0; start < len(unique); start += 256 {
end := min(start+256, len(unique))
f := organizationScopedFilter(org, bson.M{key: bson.M{"$in": unique[start:end]}})
if len(extra) > 0 {
f = bson.M{"$and": bson.A{f, extra}}
}
if err := read(c, f, projection, out); err != nil {
return err
}
}
return nil
}
var g errgroup.Group
g.Go(func() error {
return read(s.runs, runFilter, fields("id", "projectId", "workflowId", "taskId", "trigger", "mode", "parentRunId", "createdAt", "endedAt", "status", "deliveryOutcome", "failure", "error"), &d.Runs)
projection := fields("id", "projectId", "workflowId", "taskId", "trigger", "mode", "parentRunId", "createdAt", "endedAt", "status", "deliveryOutcome", "failure", "error")
if err := read(s.runs, filter, projection, &d.Runs, productionRunsCoverIndex); err != nil {
return err
}
// Disjoint union: normal project runs, legacy capture links outside
// those projects, then explicitly referenced runs still missing.
if err := readRelated(s.runs, "taskId", captureIDs, projection, &d.Runs, bson.M{"projectId": bson.M{"$nin": projects}}); err != nil {
return err
}
seen := map[string]bool{}
for _, run := range d.Runs {
seen[run.ID] = true
}
missing := make([]string, 0, len(runIDs))
for _, id := range runIDs {
if !seen[id] {
missing = append(missing, id)
}
}
return readRelated(s.runs, "id", missing, projection, &d.Runs, nil)
})
g.Go(func() error {
return read(s.humanItems, filter, fields("id", "projectId", "supplierId", "runId", "batchId", "legacyTaskId", "currentStageId", "status", "pendingCombinedSubmissionId", "releasePartitionId", "processingJob.status"), &d.Items)
@ -244,15 +300,20 @@ func (s *MongoStore) loadProductionStats(ctx context.Context, org string, projec
for _, item := range d.Items {
ids = append(ids, item.ID)
}
if len(ids) > 0 {
if err := read(s.humanEvents, organizationScopedFilter(org, bson.M{
"itemId": bson.M{"$in": ids},
// Decisions drive milestones. Historical package linkage may also
// be recorded on created/reopened events, so keep those links.
"$or": bson.A{bson.M{"eventType": bson.M{"$in": bson.A{"approved", "rejected"}}}, bson.M{"payload.deliveryPackageId": bson.M{"$exists": true}}},
}), fields("id", "itemId", "stageId", "eventType", "createdAt", "payload.sourceItemIds", "payload.deliveryPackageId"), &d.Events); err != nil {
projection := fields("id", "itemId", "stageId", "eventType", "createdAt", "payload.sourceItemIds", "payload.deliveryPackageId")
if hints["humanEvents"] != "" {
// Keep both branches disjoint. An OR equal to a partial-index predicate can
// make Mongo scan the whole partial index rather than bound each item range.
if err := readRelated(s.humanEvents, "itemId", ids, projection, &d.Events, bson.M{"eventType": bson.M{"$in": bson.A{"approved", "rejected"}}}); err != nil {
return d, err
}
if err := readRelated(s.humanEvents, "itemId", ids, projection, &d.Events, bson.M{"eventType": bson.M{"$nin": bson.A{"approved", "rejected"}}, "payload.deliveryPackageId": bson.M{"$exists": true}}); err != nil {
return d, err
}
} else if err := readRelated(s.humanEvents, "itemId", ids, projection, &d.Events, productionEventFactsFilter()); err != nil {
return d, err
}
return d, nil
}

View File

@ -0,0 +1,131 @@
package store
import (
"context"
"fmt"
"os"
"reflect"
"sync"
"testing"
"time"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/event"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func TestProductionOverviewBoundsRelatedQueriesAndKeepsLegacyUnion(t *testing.T) {
if os.Getenv("TEST_WORKFORCE_MONGO_URI") != "mongodb://127.0.0.1:27018" {
t.Skip("isolated loopback Mongo required")
}
var mu sync.Mutex
largest := 0
var visit func(any)
visit = func(v any) {
switch x := v.(type) {
case bson.M:
for k, v := range x {
if k == "$in" {
if a, ok := v.(bson.A); ok && len(a) > largest {
largest = len(a)
}
}
visit(v)
}
case bson.A:
for _, v := range x {
visit(v)
}
}
}
client, err := mongo.Connect(t.Context(), options.Client().ApplyURI("mongodb://127.0.0.1:27018").SetMonitor(&event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
if e.CommandName != "find" {
return
}
var m bson.M
if bson.Unmarshal(e.Command, &m) == nil {
mu.Lock()
visit(m)
mu.Unlock()
}
}}))
if err != nil {
t.Fatal(err)
}
db := client.Database(fmt.Sprintf("production_bounded_%d", time.Now().UnixNano()))
t.Cleanup(func() { _ = db.Drop(context.Background()); _ = client.Disconnect(context.Background()) })
n := 600
if os.Getenv("RUN_OUTPUT_HISTORY_SCALE") == "1" {
n = 24000
}
for name, keys := range map[string]bson.D{"captureRecords": {{Key: "organizationId", Value: 1}, {Key: "projectId", Value: 1}}, "humanItems": {{Key: "organizationId", Value: 1}, {Key: "projectId", Value: 1}}, "humanEvents": {{Key: "organizationId", Value: 1}, {Key: "itemId", Value: 1}}} {
if _, err = db.Collection(name).Indexes().CreateOne(t.Context(), mongo.IndexModel{Keys: keys}); err != nil {
t.Fatal(err)
}
}
for _, keys := range []bson.D{{{Key: "organizationId", Value: 1}, {Key: "projectId", Value: 1}}, {{Key: "organizationId", Value: 1}, {Key: "taskId", Value: 1}}, {{Key: "id", Value: 1}}} {
if _, err = db.Collection("runs").Indexes().CreateOne(t.Context(), mongo.IndexModel{Keys: keys}); err != nil {
t.Fatal(err)
}
}
for _, col := range []string{"captureRecords", "runs", "humanItems", "humanEvents"} {
docs := make([]any, 0, n)
for i := 0; i < n; i++ {
d := bson.M{"id": fmt.Sprintf("%s-%d", col, i), "organizationId": "o", "projectId": "p"}
switch col {
case "captureRecords":
d["id"] = fmt.Sprint("c", i)
d["runId"] = fmt.Sprint("r", i)
case "runs":
d["id"] = fmt.Sprint("r", i)
d["taskId"] = fmt.Sprint("c", i)
case "humanItems":
d["id"] = fmt.Sprint("i", i)
case "humanEvents":
d["itemId"] = fmt.Sprint("i", i)
d["stageId"] = "acceptance"
d["eventType"] = "approved"
delete(d, "projectId")
}
docs = append(docs, d)
}
if _, err = db.Collection(col).InsertMany(t.Context(), docs); err != nil {
t.Fatal(err)
}
}
_, err = db.Collection("runs").UpdateOne(t.Context(), bson.M{"id": "r0"}, bson.M{"$set": bson.M{"projectId": "legacy", "taskId": "not-the-capture"}})
if err != nil {
t.Fatal(err)
}
if _, err = db.Collection("runs").InsertMany(t.Context(), []any{bson.M{"id": "legacy-capture", "organizationId": "o", "taskId": "c0"}, bson.M{"id": "foreign-run", "organizationId": "other", "taskId": "c0"}, bson.M{"id": "unrelated", "organizationId": "o", "projectId": "other", "taskId": "elsewhere"}}); err != nil {
t.Fatal(err)
}
if _, err = db.Collection("humanEvents").InsertMany(t.Context(), []any{bson.M{"id": "legacy-link", "organizationId": "o", "itemId": "i0", "eventType": "created", "payload": bson.M{"deliveryPackageId": "pkg"}}, bson.M{"id": "foreign-event", "organizationId": "other", "itemId": "i0", "eventType": "approved"}, bson.M{"id": "edit-noise", "organizationId": "o", "itemId": "i0", "eventType": "draft_saved"}}); err != nil {
t.Fatal(err)
}
started := time.Now()
data, err := NewMongoStore(db).LoadProductionStats(t.Context(), "o", []string{"p"})
if err != nil {
t.Fatal(err)
}
if len(data.Runs) != n+1 || len(data.Events) != n+1 {
t.Fatalf("legacy union/scope changed: runs=%d events=%d", len(data.Runs), len(data.Events))
}
ids := map[string]bool{}
for _, r := range data.Runs {
if ids[r.ID] {
t.Fatal("duplicate run", r.ID)
}
ids[r.ID] = true
}
if !reflect.DeepEqual([]bool{ids["r0"], ids["legacy-capture"], ids["foreign-run"], ids["unrelated"]}, []bool{true, true, false, false}) {
t.Fatal("run linkage/tenant boundary changed")
}
mu.Lock()
defer mu.Unlock()
if largest > 256 {
t.Fatalf("unbounded overview related query: largest $in=%d, maximum=256", largest)
}
t.Logf("records_per_source=%d runs=%d events=%d max_in=%d read=%s", n, len(data.Runs), len(data.Events), largest, time.Since(started))
}

View File

@ -0,0 +1,22 @@
package store
import "go.mongodb.org/mongo-driver/bson"
// Opt-in production indexes. No startup creation or migration: rollout requires
// separately approved index construction and a measured read/write comparison.
const productionEventFactsIndex = "production_event_facts_v1"
const productionRunsCoverIndex = "production_runs_cover_v1"
func productionEventFactsFilter() bson.M {
return bson.M{"$or": bson.A{bson.M{"eventType": bson.M{"$in": bson.A{"approved", "rejected"}}}, bson.M{"payload.deliveryPackageId": bson.M{"$exists": true}}}}
}
func productionRunsCoverKeys() bson.D {
keys := bson.D{}
for _, name := range []string{"organizationId", "projectId", "id", "workflowId", "taskId", "trigger", "mode", "parentRunId", "createdAt", "endedAt", "status", "deliveryOutcome", "failure", "error"} {
keys = append(keys, bson.E{Key: name, Value: 1})
}
return keys
}
func productionEventFactsKeys() bson.D {
return bson.D{{Key: "organizationId", Value: 1}, {Key: "itemId", Value: 1}}
}

View File

@ -0,0 +1,198 @@
package store
import (
"encoding/json"
"fmt"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"os"
"reflect"
"sort"
"strings"
"testing"
"time"
)
func TestProductionFactsIndexesAvoidLargeDocumentsAndKeepLegacyLinks(t *testing.T) {
db := localOutputMongo(t)
ctx := t.Context()
s := NewMongoStore(db)
check := func(err error) {
t.Helper()
if err != nil {
t.Fatal(err)
}
}
_, err := db.Collection("runs").Indexes().CreateOne(ctx, mongo.IndexModel{Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "projectId", Value: 1}}})
check(err)
_, err = db.Collection("humanEvents").Indexes().CreateOne(ctx, mongo.IndexModel{Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "itemId", Value: 1}, {Key: "stageId", Value: 1}}})
check(err)
n := 600
if os.Getenv("RUN_PRODUCTION_STATS_INDEX_SCALE") == "1" {
n = 24000
}
for _, name := range []string{"captureRecords", "runs", "humanItems", "humanEvents"} {
rows := []any{}
for i := 0; i < n; i++ {
id := fmt.Sprint("i", i)
r := bson.M{"id": id, "organizationId": "o", "projectId": "p"}
switch name {
case "captureRecords":
r["runId"] = id
r["prefix"] = "source/" + id + "/part0001"
case "runs":
r["taskId"] = id
r["logs"] = strings.Repeat("large unneeded run log", 1000)
r["failure"] = bson.M{"message": "invalid PTS", "code": "source"}
case "humanEvents":
r["itemId"] = id
r["stageId"] = "acceptance"
r["eventType"] = "approved"
delete(r, "projectId")
}
rows = append(rows, r)
if name == "humanEvents" {
for j := 0; j < 20; j++ {
rows = append(rows, bson.M{"id": fmt.Sprintf("edit-%d-%d", i, j), "organizationId": "o", "itemId": id, "stageId": "annotation", "eventType": "description_confirmed", "payload": bson.M{"draft": strings.Repeat("x", 1024)}})
}
}
if len(rows) >= 1000 {
_, err = db.Collection(name).InsertMany(ctx, rows)
check(err)
rows = nil
}
}
if len(rows) > 0 {
_, err = db.Collection(name).InsertMany(ctx, rows)
check(err)
}
}
_, err = db.Collection("humanEvents").InsertOne(ctx, bson.M{"id": "legacy", "organizationId": "o", "itemId": "i0", "eventType": "created", "payload": bson.M{"deliveryPackageId": "old-package"}})
check(err)
started := time.Now()
before, err := s.LoadProductionStats(ctx, "o", []string{"p"})
check(err)
beforeTime := time.Since(started)
_, err = db.Collection("runs").Indexes().CreateOne(ctx, mongo.IndexModel{Keys: productionRunsCoverKeys(), Options: options.Index().SetName(productionRunsCoverIndex)})
check(err)
_, err = db.Collection("humanEvents").Indexes().CreateOne(ctx, mongo.IndexModel{Keys: productionEventFactsKeys(), Options: options.Index().SetName(productionEventFactsIndex).SetPartialFilterExpression(productionEventFactsFilter())})
check(err)
started = time.Now()
after, err := s.LoadProductionStats(ctx, "o", []string{"p"})
check(err)
afterTime := time.Since(started)
before.AsOf, after.AsOf = "", ""
sort.Slice(before.Runs, func(i, j int) bool { return before.Runs[i].ID < before.Runs[j].ID })
sort.Slice(after.Runs, func(i, j int) bool { return after.Runs[i].ID < after.Runs[j].ID })
sort.Slice(before.Events, func(i, j int) bool { return before.Events[i].ID < before.Events[j].ID })
sort.Slice(after.Events, func(i, j int) bool { return after.Events[i].ID < after.Events[j].ID })
if !reflect.DeepEqual(before, after) || len(after.Events) != n+1 {
t.Fatal("index changed complete history/legacy links")
}
for _, name := range []string{"runs", "humanEvents"} {
filter := bson.M{"organizationId": "o", "projectId": "p"}
hint := productionRunsCoverIndex
projection := bson.M{"_id": 0}
for _, key := range productionRunsCoverKeys() {
if key.Key != "organizationId" {
projection[key.Key] = 1
}
}
if name == "humanEvents" {
hint = productionEventFactsIndex
filter = bson.M{"$and": bson.A{bson.M{"organizationId": "o", "itemId": bson.M{"$in": bson.A{"i0", "i1"}}}, bson.M{"eventType": bson.M{"$in": bson.A{"approved", "rejected"}}}}}
projection = bson.M{"_id": 0, "id": 1, "payload.deliveryPackageId": 1}
}
var result struct {
Stats struct {
Docs int `bson:"totalDocsExamined"`
Keys int `bson:"totalKeysExamined"`
Returned int `bson:"nReturned"`
} `bson:"executionStats"`
}
check(db.RunCommand(ctx, bson.D{{Key: "explain", Value: bson.D{{Key: "find", Value: name}, {Key: "filter", Value: filter}, {Key: "projection", Value: projection}, {Key: "hint", Value: hint}}}, {Key: "verbosity", Value: "executionStats"}}).Decode(&result))
if name == "runs" && result.Stats.Docs != 0 {
t.Fatalf("covering run query fetched large documents: %+v", result.Stats)
}
if name == "humanEvents" && result.Stats.Docs > 3 {
t.Fatalf("editing history still scanned: %+v", result.Stats)
}
t.Logf("collection=%s docs=%d keys=%d returned=%d", name, result.Stats.Docs, result.Stats.Keys, result.Stats.Returned)
}
t.Logf("same fixture read fallback=%s indexed=%s; records preserved=%d", beforeTime, afterTime, len(after.Events))
if n > 600 {
samples := []time.Duration{afterTime}
for range 4 {
started = time.Now()
v, e := s.LoadProductionStats(ctx, "o", []string{"p"})
check(e)
if len(v.Runs) != n || len(v.Events) != n+1 {
t.Fatal("incomplete benchmark read")
}
samples = append(samples, time.Since(started))
}
sort.Slice(samples, func(i, j int) bool { return samples[i] < samples[j] })
t.Logf("indexed complete reads=%v max=%s", samples, samples[len(samples)-1])
if samples[len(samples)-1] > 2*time.Second {
t.Fatal("indexed representative read exceeds 2 seconds")
}
}
}
func TestProductionIndexManifestMatchesVerifiedDefinitions(t *testing.T) {
raw, err := os.ReadFile("../../docs/operations/production-overview-indexes.json")
if err != nil {
t.Fatal(err)
}
var manifest struct {
Database string `json:"database"`
Indexes []struct {
Collection, Name string
Keys [][]any
Partial json.RawMessage `json:"partialFilterExpression"`
}
}
if err = json.Unmarshal(raw, &manifest); err != nil {
t.Fatal(err)
}
if manifest.Database != "drobodata-fanyu" || len(manifest.Indexes) != 2 {
t.Fatal("unexpected index scope")
}
for _, index := range manifest.Indexes {
var keys bson.D
switch index.Collection {
case "runs":
if index.Name != productionRunsCoverIndex {
t.Fatal("run name mismatch")
}
keys = productionRunsCoverKeys()
case "humanEvents":
if index.Name != productionEventFactsIndex {
t.Fatal("event name mismatch")
}
keys = productionEventFactsKeys()
var actual map[string]any
if err = json.Unmarshal(index.Partial, &actual); err != nil {
t.Fatal(err)
}
a, _ := json.Marshal(actual)
b, _ := json.Marshal(productionEventFactsFilter())
if string(a) != string(b) {
t.Fatal("partial predicate drift")
}
default:
t.Fatal("unexpected collection")
}
expected := [][]any{}
for _, key := range keys {
expected = append(expected, []any{key.Key, key.Value})
}
a, _ := json.Marshal(expected)
b, _ := json.Marshal(index.Keys)
if string(a) != string(b) {
t.Fatal("ordered index fields drift")
}
}
}

View File

@ -144,7 +144,9 @@ func buildOutputParts(d OutputSnapshot, q OutputQuery) ([]outputPart, []OutputWo
worklogOps := map[string]domain.ReviewOperation{}
for _, o := range d.Operations {
operations[o.ID] = o
operations[o.OperationID] = o
if o.OperationID != "" {
operations[o.OperationID] = o
}
for _, e := range o.Effects {
if e.Review != nil {
reviewOps[e.Review.ID] = o
@ -171,6 +173,9 @@ func buildOutputParts(d OutputSnapshot, q OutputQuery) ([]outputPart, []OutputWo
video = path.Base(path.Dir(strings.TrimRight(c.Prefix, "/"))) + "/" + path.Base(strings.TrimRight(c.Prefix, "/"))
}
}
if identity, ok := d.Identities[t.ID]; ok {
key, video = identity.ID, identity.Video
}
attempt := t.RunID
if attempt == "" {
attempt = t.ID
@ -293,7 +298,9 @@ func buildOutputParts(d OutputSnapshot, q OutputQuery) ([]outputPart, []OutputWo
continue
}
covered[o.ID] = true
covered[o.OperationID] = true
if o.OperationID != "" {
covered[o.OperationID] = true
}
if o.Status == "completed" {
addAcceptance(o.ID, o.CompletedAt, o.Decision, o.UserID, o.ReviewerSupplierID, o.SourceItemIDs)
}
@ -303,10 +310,10 @@ func buildOutputParts(d OutputSnapshot, q OutputQuery) ([]outputPart, []OutputWo
continue
}
opID, _ := e.Payload["operationId"].(string)
if covered[opID] {
if opID != "" && covered[opID] {
continue
}
if o, ok := operations[opID]; ok && o.Status != "completed" {
if o, ok := operations[opID]; opID != "" && ok && o.Status != "completed" {
continue
}
addAcceptance(e.ID, e.CreatedAt, e.EventType, e.ActorUserID, "", outputIDs(e.Payload["sourceItemIds"]))
@ -755,6 +762,10 @@ func (a *outputReviewAccumulator) report(suppliers map[string]string) OutputRevi
}
func outputSummarize(parts []outputPart, work []OutputWorkFact, q OutputQuery, projects, users, suppliers map[string]string) OutputReport {
out, _ := outputSummarizeWithMeta(parts, work, q, projects, users, suppliers)
return out
}
func outputSummarizeWithMeta(parts []outputPart, work []OutputWorkFact, q OutputQuery, projects, users, suppliers map[string]string) (OutputReport, *outputMergeMeta) {
out := OutputReport{Tasks: []OutputTask{}, Suppliers: []OutputSupplier{}, Days: []OutputDay{}, Projects: []OutputOption{}, SupplierOptions: []OutputOption{}, Users: []OutputOption{}}
allowed := func(o outputOwner) bool {
return (!q.MatchUser || o.user == q.ScopeUserID) && (!q.MatchSupplier || o.supplier == q.ScopeSupplierID)
@ -952,5 +963,9 @@ func outputSummarize(parts []outputPart, work []OutputWorkFact, q OutputQuery, p
}
return a.SupplierID+a.UserID < b.SupplierID+b.UserID
})
return out
meta := &outputMergeMeta{days: annotationDays, members: members, cumulative: map[string][2]OutputMetric{}}
for key, c := range cumulative {
meta.cumulative[key] = [2]OutputMetric{c[0].metric(), c[1].metric()}
}
return out, meta
}

View File

@ -69,10 +69,14 @@ func BuildOutputDataset(d OutputSnapshot, asOf time.Time) OutputDataset {
return out
}
func QueryOutputDataset(d OutputDataset, q OutputQuery) OutputReport {
out, _ := queryOutputDatasetWithMeta(d, q)
return out
}
func queryOutputDatasetWithMeta(d OutputDataset, q OutputQuery) (OutputReport, *outputMergeMeta) {
if q.ProductionQuality {
q.ProductionQuality = false
result := BuildProductionQuality(d, q)
return OutputReport{Production: &result}
return OutputReport{Production: &result}, nil
}
if q.FirstSubmissionCohort {
d = SelectFirstSubmissionCohort(d, q.From, q.To)
@ -103,9 +107,9 @@ func QueryOutputDataset(d OutputDataset, q OutputQuery) OutputReport {
}
parts = append(parts, outputPart{row, fs[0], fs[1], fs[2], fs[3]})
}
report := outputSummarize(parts, d.Work, q, d.Projects, d.Users, d.Suppliers)
report, meta := outputSummarizeWithMeta(parts, d.Work, q, d.Projects, d.Users, d.Suppliers)
if q.FirstSubmissionCohort {
report.QualityPeople = CohortQuality(d, q)
}
return report
return report, meta
}

View File

@ -0,0 +1,19 @@
package workforcestats
import (
"droboticflow/internal/domain"
"path"
"strings"
)
func OutputPartIdentityForCapture(project, task string, capture *domain.CaptureRecord) OutputPartIdentity {
identity := OutputPartIdentity{ID: "task:" + project + ":" + task, Video: task}
if capture != nil && capture.Prefix != "" {
identity.ID = outputPartKey(*capture)
identity.Video = capture.CaptureID
if identity.Video == "" {
identity.Video = path.Base(path.Dir(strings.TrimRight(capture.Prefix, "/"))) + "/" + path.Base(strings.TrimRight(capture.Prefix, "/"))
}
}
return identity
}

View File

@ -0,0 +1,41 @@
package workforcestats
import (
"droboticflow/internal/domain"
"testing"
"time"
)
func TestLegacyAcceptanceWithoutOperationIDIsNotCoveredByUnrelatedPlan(t *testing.T) {
d := outputFixture()
c := d.Captures[0]
c.ID = "c2"
c.RunID = "r2"
c.Prefix = "source/other/part0001"
d.Captures = append(d.Captures, c)
r := d.Runs[0]
r.ID = "r2"
r.TaskID = "c2"
d.Runs = append(d.Runs, r)
task := d.Tasks[0]
task.ID = "t2"
task.RunID = "r2"
d.Tasks = append(d.Tasks, task)
item := d.Items[0]
item.ID = "i2"
item.LegacyTaskID = "t2"
d.Items = append(d.Items, item)
event := d.Events[0]
event.ID = "s2"
event.TaskID = "t2"
d.Events = append(d.Events, event)
revision := d.Revisions[0]
revision.TaskID = "t2"
d.Revisions = append(d.Revisions, revision)
d.HumanEvents = append(d.HumanEvents, domain.HumanEventRecord{ID: "legacy-acceptance", StageID: "final_acceptance", EventType: "approved", CreatedAt: "2026-10-04T00:00:00Z", Payload: map[string]any{"sourceItemIds": []string{"i2"}}})
at := time.Date(2026, 10, 20, 0, 0, 0, 0, time.UTC)
got := BuildOutput(d, OutputQuery{From: time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC), To: at, Now: at})
if got.Totals.Acceptance.Count != 2 {
t.Fatalf("unrelated empty client operation ID suppressed legacy acceptance: got=%d want=2", got.Totals.Acceptance.Count)
}
}

View File

@ -0,0 +1,404 @@
package workforcestats
import (
"sort"
"time"
)
type outputMergeMeta struct {
days outputDays
members map[string]map[string]bool
cumulative map[string][2]OutputMetric
}
// OutputReducer merges disjoint canonical-part shards. It retains result rows
// and small person/day sets, never all raw decisions or decoded datasets.
// Ratios are merged through their evidence, never by averaging percentages.
type OutputReducer struct {
q OutputQuery
out OutputReport
days outputDays
members map[string]map[string]bool
cumulative map[string][2]OutputMetric
suppliers map[string]*OutputSupplier
daily map[string]*OutputDay
qualityPeople map[string]*QualityPerson
productionPeople map[string]*ProductionPerson
productionSuppliers map[string]*ProductionSupplier
productionQuality map[string]*QualityPerson
comparisons [6]map[string]*QualityPerson
options [3]map[string]string
}
func NewOutputReducer(q OutputQuery) *OutputReducer {
if q.Location == nil {
q.Location = time.UTC
}
r := &OutputReducer{q: q, days: outputDays{}, members: map[string]map[string]bool{}, cumulative: map[string][2]OutputMetric{}, suppliers: map[string]*OutputSupplier{}, daily: map[string]*OutputDay{}, qualityPeople: map[string]*QualityPerson{}, productionPeople: map[string]*ProductionPerson{}, productionSuppliers: map[string]*ProductionSupplier{}, productionQuality: map[string]*QualityPerson{}}
r.out = OutputReport{Tasks: []OutputTask{}, Suppliers: []OutputSupplier{}, Days: []OutputDay{}, Projects: []OutputOption{}, SupplierOptions: []OutputOption{}, Users: []OutputOption{}}
for i := range r.options {
r.options[i] = map[string]string{}
}
for i := range r.comparisons {
r.comparisons[i] = map[string]*QualityPerson{}
}
return r
}
func mergeOutputMetric(a, b OutputMetric) OutputMetric {
out := OutputMetric{Count: a.Count + b.Count}
if (a.Count > 0 && a.Seconds == nil) || (b.Count > 0 && b.Seconds == nil) {
return out
}
if a.Seconds != nil || b.Seconds != nil {
value := float64(0)
if a.Seconds != nil {
value += *a.Seconds
}
if b.Seconds != nil {
value += *b.Seconds
}
out.Seconds = &value
}
return out
}
func mergeOutputTotals(a, b OutputTotals) OutputTotals {
return OutputTotals{mergeOutputMetric(a.Submitted, b.Submitted), mergeOutputMetric(a.Quality, b.Quality), mergeOutputMetric(a.Acceptance, b.Acceptance), mergeOutputMetric(a.Rework, b.Rework)}
}
func mergeQualityRatio(a *QualityRatio, b QualityRatio) {
a.Numerator += b.Numerator
a.Denominator += b.Denominator
a.Unknown += b.Unknown
a.Value = nil
}
func qualityIdentity(p QualityPerson) string {
return p.Role + "\x00" + p.SupplierID + "\x00" + p.UserID
}
func mergeQualityRow(m map[string]*QualityPerson, key string, v QualityPerson) {
p := m[key]
if p == nil {
p = &QualityPerson{UserID: v.UserID, SupplierID: v.SupplierID, Role: v.Role}
m[key] = p
}
mergeQualityRatio(&p.FirstPass, v.FirstPass)
mergeQualityRatio(&p.FirstReworkPass, v.FirstReworkPass)
mergeQualityRatio(&p.OpenRework, v.OpenRework)
mergeQualityRatio(&p.MeanReworks, v.MeanReworks)
mergeQualityRatio(&p.Misreport, v.Misreport)
}
func finishQuality(p *QualityPerson) {
p.FirstPass.finish(100)
p.FirstReworkPass.finish(100)
p.OpenRework.finish(100)
p.MeanReworks.finish(1)
p.Misreport.finish(100)
}
func mergeStringSets(dst map[string]map[string]bool, src map[string]map[string]bool) {
for key, values := range src {
if dst[key] == nil {
dst[key] = map[string]bool{}
}
for value := range values {
dst[key][value] = true
}
}
}
func (r *OutputReducer) addOptions(projects, suppliers, users []OutputOption) {
for i, values := range [][]OutputOption{projects, suppliers, users} {
for _, v := range values {
r.options[i][v.ID] = v.Name
}
}
}
func outputOptions(m map[string]string) []OutputOption {
out := []OutputOption{}
for id, name := range m {
out = append(out, OutputOption{ID: id, Name: name})
}
sort.Slice(out, func(i, j int) bool {
if out[i].Name != out[j].Name {
return out[i].Name < out[j].Name
}
return out[i].ID < out[j].ID
})
return out
}
func (r *OutputReducer) Add(d OutputDataset) {
if r.q.ProductionQuality {
r.addProduction(d)
return
}
report, meta := queryOutputDatasetWithMeta(d, r.q)
r.addOptions(report.Projects, report.SupplierOptions, report.Users)
r.out.Totals = mergeOutputTotals(r.out.Totals, report.Totals)
r.out.Tasks = append(r.out.Tasks, report.Tasks...)
r.out.Quality = mergeOutputReview(r.out.Quality, report.Quality)
r.out.Acceptance = mergeOutputReview(r.out.Acceptance, report.Acceptance)
if meta != nil {
mergeStringSets(r.days, meta.days)
mergeStringSets(r.members, meta.members)
for key, v := range meta.cumulative {
old := r.cumulative[key]
r.cumulative[key] = [2]OutputMetric{mergeOutputMetric(old[0], v[0]), mergeOutputMetric(old[1], v[1])}
}
}
for _, v := range report.Suppliers {
p := r.suppliers[v.ID]
if p == nil {
p = &OutputSupplier{ID: v.ID, Name: v.Name}
r.suppliers[v.ID] = p
}
p.OutputTotals = mergeOutputTotals(p.OutputTotals, v.OutputTotals)
}
for _, v := range report.Days {
key := v.Date + "\x00" + v.SupplierID + "\x00" + v.UserID
p := r.daily[key]
if p == nil {
copy := v
copy.Submitted = OutputMetric{}
copy.Acceptance = OutputMetric{}
p = &copy
r.daily[key] = p
}
p.Submitted = mergeOutputMetric(p.Submitted, v.Submitted)
p.Acceptance = mergeOutputMetric(p.Acceptance, v.Acceptance)
}
for _, v := range report.QualityPeople {
mergeQualityRow(r.qualityPeople, qualityIdentity(v), v)
}
}
func mergeOutputReview(a, b OutputReviewReport) OutputReviewReport {
out := OutputReviewReport{Suppliers: []OutputReviewSupplier{}, Days: []OutputReviewDay{}}
merge := func(a, b OutputReviewMetric) OutputReviewMetric {
return OutputReviewMetric{Reviews: mergeOutputMetric(a.Reviews, b.Reviews), Approved: a.Approved + b.Approved, Rejected: a.Rejected + b.Rejected, Rereviews: a.Rereviews + b.Rereviews}
}
out.Totals = merge(a.Totals, b.Totals)
suppliers := map[string]*OutputReviewSupplier{}
days := map[string]*OutputReviewDay{}
for _, report := range []OutputReviewReport{a, b} {
for _, v := range report.Suppliers {
p := suppliers[v.ID]
if p == nil {
p = &OutputReviewSupplier{ID: v.ID, Name: v.Name}
suppliers[v.ID] = p
}
p.OutputReviewMetric = merge(p.OutputReviewMetric, v.OutputReviewMetric)
}
for _, v := range report.Days {
key := v.Date + "\x00" + v.SupplierID + "\x00" + v.UserID
p := days[key]
if p == nil {
copy := v
copy.Reviews = OutputMetric{}
copy.Approved = 0
copy.Rejected = 0
copy.Rereviews = 0
p = &copy
days[key] = p
}
p.Reviews = mergeOutputMetric(p.Reviews, v.Reviews)
p.Approved += v.Approved
p.Rejected += v.Rejected
p.Rereviews += v.Rereviews
}
}
members := map[string]map[string]bool{}
workDays := outputDays{}
for _, v := range days {
out.Days = append(out.Days, *v)
workDays.add(v.SupplierID, v.UserID, v.Date)
if members[v.SupplierID] == nil {
members[v.SupplierID] = map[string]bool{}
}
if v.UserID != "" {
members[v.SupplierID][v.UserID] = true
}
}
all := map[string]bool{}
for id, p := range suppliers {
p.MemberCount = len(members[id])
for user := range members[id] {
all[user] = true
}
p.Efficiency = outputEfficiency(p.Reviews, workDays.count(id))
out.Suppliers = append(out.Suppliers, *p)
}
out.Totals.MemberCount = len(all)
out.Totals.Efficiency = outputEfficiency(out.Totals.Reviews, workDays.total())
sort.Slice(out.Suppliers, func(i, j int) bool { return out.Suppliers[i].ID < out.Suppliers[j].ID })
sort.Slice(out.Days, func(i, j int) bool {
a, b := out.Days[i], out.Days[j]
if a.Date != b.Date {
return a.Date > b.Date
}
return a.SupplierID+"\x00"+a.UserID < b.SupplierID+"\x00"+b.UserID
})
return out
}
func (r *OutputReducer) addProduction(d OutputDataset) {
report := buildProductionQualityCore(d, r.q)
r.addOptions(report.Projects, report.SupplierOptions, report.Users)
if r.out.Production == nil {
r.out.Production = &ProductionQualityReport{Suppliers: []ProductionSupplier{}, People: []ProductionPerson{}, Quality: []QualityPerson{}, Projects: []OutputOption{}, SupplierOptions: []OutputOption{}, Users: []OutputOption{}}
}
r.out.Production.Totals = mergeOutputTotals(r.out.Production.Totals, report.Totals)
r.out.Production.Deliverable = mergeOutputMetric(r.out.Production.Deliverable, report.Deliverable)
for _, v := range report.People {
key := v.Role + "\x00" + v.SupplierID + "\x00" + v.ID
p := r.productionPeople[key]
if p == nil {
p = &ProductionPerson{ID: v.ID, Name: v.Name, SupplierID: v.SupplierID, SupplierName: v.SupplierName, Role: v.Role}
r.productionPeople[key] = p
}
p.Totals = mergeOutputTotals(p.Totals, v.Totals)
p.Deliverable = mergeOutputMetric(p.Deliverable, v.Deliverable)
if v.Quality != nil {
mergeQualityRow(r.qualityPeople, key, *v.Quality)
}
}
for _, v := range report.Suppliers {
p := r.productionSuppliers[v.ID]
if p == nil {
p = &ProductionSupplier{ID: v.ID, Name: v.Name, Members: map[string]int{}}
r.productionSuppliers[v.ID] = p
}
p.Totals = mergeOutputTotals(p.Totals, v.Totals)
p.Deliverable = mergeOutputMetric(p.Deliverable, v.Deliverable)
for _, q := range v.Quality {
mergeQualityRow(r.productionQuality, "supplier\x00"+qualityIdentity(q), q)
}
}
for _, q := range report.Quality {
mergeQualityRow(r.productionQuality, "total\x00"+qualityIdentity(q), q)
}
now := d.AsOf.In(r.q.Location)
today := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, r.q.Location)
for day := 0; day < 6; day++ {
start := today.AddDate(0, 0, day-5)
cohort := SelectFirstSubmissionCohort(d, start, start.AddDate(0, 0, 1))
parts := qualityWorkByPart(cohort)
for _, group := range []string{"total", "supplier", "person"} {
for _, q := range cohortQualityFromParts(cohort, r.q, group, parts) {
mergeQualityRow(r.comparisons[day], group+"\x00"+qualityIdentity(q), q)
}
}
}
}
func (r *OutputReducer) attachComparisons(p *QualityPerson, group string) {
p.Trend = make([]QualityValues, 6)
for day := 0; day < 6; day++ {
if metric := r.comparisons[day][group+"\x00"+qualityIdentity(*p)]; metric != nil {
finishQuality(metric)
values := qualityValues(*metric)
p.Trend[day] = values
if day == 4 {
copy := values
p.Yesterday = &copy
}
}
}
}
func (r *OutputReducer) Report() OutputReport {
if r.q.ProductionQuality {
return r.productionReport()
}
r.out.Suppliers = []OutputSupplier{}
r.out.Days = []OutputDay{}
r.out.Projects = outputOptions(r.options[0])
r.out.SupplierOptions = outputOptions(r.options[1])
r.out.Users = outputOptions(r.options[2])
r.out.Efficiency = outputEfficiency(r.out.Totals.Submitted, r.days.total())
for _, p := range r.suppliers {
p.MemberCount = len(r.members[p.ID])
p.Efficiency = outputEfficiency(p.Submitted, r.days.count(p.ID))
r.out.Suppliers = append(r.out.Suppliers, *p)
}
for _, p := range r.daily {
total := r.cumulative[p.SupplierID+"\x00"+p.UserID]
p.CumulativeSubmitted, p.CumulativeAcceptance = total[0], total[1]
r.out.Days = append(r.out.Days, *p)
}
if r.q.FirstSubmissionCohort {
r.out.QualityPeople = []QualityPerson{}
for _, p := range r.qualityPeople {
finishQuality(p)
r.out.QualityPeople = append(r.out.QualityPeople, *p)
}
sort.Slice(r.out.QualityPeople, func(i, j int) bool {
return qualityIdentity(r.out.QualityPeople[i]) < qualityIdentity(r.out.QualityPeople[j])
})
}
sort.Slice(r.out.Tasks, func(i, j int) bool {
a, b := r.out.Tasks[i], r.out.Tasks[j]
if a.SubmittedAt != b.SubmittedAt {
return outputEarlier(b.SubmittedAt, a.SubmittedAt)
}
return a.ID < b.ID
})
sort.Slice(r.out.Suppliers, func(i, j int) bool { return r.out.Suppliers[i].ID < r.out.Suppliers[j].ID })
sort.Slice(r.out.Days, func(i, j int) bool {
a, b := r.out.Days[i], r.out.Days[j]
if a.Date != b.Date {
return a.Date > b.Date
}
return a.SupplierID+a.UserID < b.SupplierID+b.UserID
})
return r.out
}
func (r *OutputReducer) productionReport() OutputReport {
if r.out.Production == nil {
empty := OutputDataset{AsOf: r.q.Now, Projects: map[string]string{}, Users: map[string]string{}, Suppliers: map[string]string{}}
r.addProduction(empty)
}
out := r.out.Production
out.People = []ProductionPerson{}
out.Suppliers = []ProductionSupplier{}
out.Quality = []QualityPerson{}
out.Projects = outputOptions(r.options[0])
out.SupplierOptions = outputOptions(r.options[1])
out.Users = outputOptions(r.options[2])
members := map[string]map[string]map[string]bool{}
for key, p := range r.productionPeople {
if q := r.qualityPeople[key]; q != nil {
finishQuality(q)
r.attachComparisons(q, "person")
p.Quality = q
}
out.People = append(out.People, *p)
if members[p.SupplierID] == nil {
members[p.SupplierID] = map[string]map[string]bool{}
}
if members[p.SupplierID][p.Role] == nil {
members[p.SupplierID][p.Role] = map[string]bool{}
}
members[p.SupplierID][p.Role][p.ID] = true
}
for id, p := range r.productionSuppliers {
p.Quality = nil
for role, users := range members[id] {
p.Members[role] = len(users)
}
for key, q := range r.productionQuality {
if len(key) >= 9 && key[:9] == "supplier\x00" && q.SupplierID == id {
finishQuality(q)
r.attachComparisons(q, "supplier")
p.Quality = append(p.Quality, *q)
}
}
sort.Slice(p.Quality, func(i, j int) bool { return qualityIdentity(p.Quality[i]) < qualityIdentity(p.Quality[j]) })
out.Suppliers = append(out.Suppliers, *p)
}
for key, q := range r.productionQuality {
if len(key) >= 6 && key[:6] == "total\x00" {
finishQuality(q)
r.attachComparisons(q, "total")
out.Quality = append(out.Quality, *q)
}
}
sort.Slice(out.People, func(i, j int) bool {
a, b := out.People[i], out.People[j]
return a.Role+"\x00"+a.SupplierID+"\x00"+a.ID < b.Role+"\x00"+b.SupplierID+"\x00"+b.ID
})
sort.Slice(out.Suppliers, func(i, j int) bool { return out.Suppliers[i].ID < out.Suppliers[j].ID })
sort.Slice(out.Quality, func(i, j int) bool { return qualityIdentity(out.Quality[i]) < qualityIdentity(out.Quality[j]) })
return OutputReport{Production: out}
}

View File

@ -0,0 +1,128 @@
package workforcestats
import (
"encoding/json"
"math"
"reflect"
"testing"
"time"
)
func TestOutputReducerMatchesWholeDataset(t *testing.T) {
d := BuildOutputDataset(outputReviewFixture(), outputQuery().Now)
d.AsOf = d.AsOf.AddDate(0, 0, 3)
// Exercise distinct parts sharing people/days, unknown durations and rework.
originalParts, originalWork := append([]OutputPartFact(nil), d.Parts...), append([]OutputWorkFact(nil), d.Work...)
for n := 1; n < 7; n++ {
for _, p := range originalParts {
p.ID += string(rune('a' + n))
for j, stage := range p.Stages {
if stage != nil {
copy := *stage
at, err := time.Parse(time.RFC3339Nano, copy.At)
if err != nil {
t.Fatal(err)
}
copy.At = at.AddDate(0, 0, n+12).Format(time.RFC3339Nano)
p.Stages[j] = &copy
}
}
if n == 3 {
p.Seconds = nil
} else if p.Seconds != nil {
seconds := *p.Seconds + float64(n)/30
p.Seconds = &seconds
}
d.Parts = append(d.Parts, p)
}
for _, w := range originalWork {
w.Part += n * len(originalParts)
at, err := time.Parse(time.RFC3339Nano, w.At)
if err != nil {
t.Fatal(err)
}
w.At = at.AddDate(0, 0, n+12).Format(time.RFC3339Nano)
d.Work = append(d.Work, w)
}
}
for _, production := range []bool{false, true} {
for _, cohort := range []bool{false, true} {
for _, supplier := range []string{"", "v", "qv", "missing"} {
for _, user := range []string{"", "a", "qc"} {
q := outputQuery()
q.ProductionQuality = production
q.FirstSubmissionCohort = cohort
q.SupplierID = supplier
q.UserID = user
r := NewOutputReducer(q)
for i, p := range d.Parts {
shard := OutputDataset{AsOf: d.AsOf, Parts: []OutputPartFact{p}, Projects: d.Projects, Users: d.Users, Suppliers: d.Suppliers}
for _, w := range d.Work {
if w.Part == i {
w.Part = 0
shard.Work = append(shard.Work, w)
}
}
for _, problem := range d.Problems {
if problem.Part == i {
problem.Part = 0
shard.Problems = append(shard.Problems, problem)
}
}
r.Add(shard)
}
want := QueryOutputDataset(d, q)
got := r.Report()
a, _ := json.Marshal(want)
b, _ := json.Marshal(got)
again, _ := json.Marshal(r.Report())
if !reflect.DeepEqual(b, again) {
t.Fatal("Report is not idempotent")
}
var expectedValue, actualValue any
_ = json.Unmarshal(a, &expectedValue)
_ = json.Unmarshal(b, &actualValue)
if !equalOutputReportValue(expectedValue, actualValue) {
t.Fatalf("production=%v cohort=%v supplier=%s user=%s\nwant %s\ngot %s", production, cohort, supplier, user, a, b)
}
}
}
}
}
}
// Disjoint-part merge order may differ in the final floating-point bit. The
// tolerance is far below one frame; structure, unknowns and integer counts stay
// exact. Percentages are still calculated from merged numerators/denominators.
func equalOutputReportValue(a, b any) bool {
switch x := a.(type) {
case float64:
y, ok := b.(float64)
return ok && math.Abs(x-y) < 1e-7
case map[string]any:
y, ok := b.(map[string]any)
if !ok || len(x) != len(y) {
return false
}
for k, v := range x {
value, ok := y[k]
if !ok || !equalOutputReportValue(v, value) {
return false
}
}
return true
case []any:
y, ok := b.([]any)
if !ok || len(x) != len(y) {
return false
}
for i, v := range x {
if !equalOutputReportValue(v, y[i]) {
return false
}
}
return true
default:
return reflect.DeepEqual(a, b)
}
}

View File

@ -8,7 +8,14 @@ import (
// OutputSnapshot contains reporting facts only. Loading it must never recover,
// claim or modify a task. Mutable drafts and media payloads are not needed.
type OutputPartIdentity struct {
ID string `bson:"id"`
Video string `bson:"video"`
}
type OutputSnapshot struct {
Identities map[string]OutputPartIdentity // Internal pre-resolved canonical identity for streamed batches.
Baselines []domain.ReworkBaselineRecord
Handlings []domain.ReworkHandlingRecord
SourceStats *OutputSourceStats
@ -181,6 +188,16 @@ const OutputSourceMaxDocuments = 250000
var ErrOutputSourceBudget = errors.New("annotation output source budget exceeded")
type OutputSourceStats struct {
Bytes int `json:"bytes" bson:"bytes"`
Documents int `json:"documents" bson:"documents"`
Bytes int `json:"bytes" bson:"bytes"`
Documents int `json:"documents" bson:"documents"`
Partitions int `json:"partitions,omitempty" bson:"partitions,omitempty"`
PeakBatchBytes int `json:"peakBatchBytes,omitempty" bson:"peakBatchBytes,omitempty"`
PeakBatchDocuments int `json:"peakBatchDocuments,omitempty" bson:"peakBatchDocuments,omitempty"`
DirectoryBytes int `json:"directoryBytes,omitempty" bson:"directoryBytes,omitempty"`
PeakSpoolBytes int64 `json:"peakSpoolBytes,omitempty" bson:"peakSpoolBytes,omitempty"`
}
const OutputDirectoryMaxBytes = 256 << 20
const OutputSpoolMaxBytes = 2 << 30
var ErrOutputPartitionSplit = errors.New("reporting partition needs subdivision")

View File

@ -35,7 +35,11 @@ type ProductionPerson struct {
}
func BuildProductionQuality(d OutputDataset, q OutputQuery) ProductionQualityReport {
original := d
result := buildProductionQualityCore(d, q)
attachQualityComparisons(&result, d, q)
return result
}
func buildProductionQualityCore(d OutputDataset, q OutputQuery) ProductionQualityReport {
d = SelectFirstSubmissionCohort(d, q.From, q.To)
q.FirstSubmissionCohort = false
q.From = time.Time{}
@ -184,7 +188,6 @@ func BuildProductionQuality(d OutputDataset, q OutputQuery) ProductionQualityRep
if q.MatchUser || q.UserID != "" {
result.Totals = OutputTotals{selectedSums[0].metric(), selectedSums[1].metric(), selectedSums[2].metric(), selectedSums[3].metric()}
}
attachQualityComparisons(&result, original, q)
return result
}
func attachQualityComparisons(result *ProductionQualityReport, d OutputDataset, q OutputQuery) {

View File

@ -20,6 +20,16 @@ REQUIRED = {
"TestHealthWarnsBeforeExpiryAndDoesNotTriggerBuild",
"TestAnnotationOutputDatasetPreservesRulesAcrossFilters",
"TestRouteSnapshot",
"TestAnnotationOutputMongoShardedRestartAndFailedBuild",
"TestAnnotationOutputStreamFailureCleansPrivateFiles",
"TestAnnotationOutputStreamMatchesWholeHistoryAcrossLegacyReferences",
"TestShardedGenerationMatchesLegacyAndSurvivesCorruptionAndRestart",
"TestShardedBuildFailureNeverPublishesPartialManifest",
"TestOutputReducerMatchesWholeDataset",
"TestOutputSpoolCleanupRecoversCrashedBuildButKeepsActive",
"TestProductionFactsIndexesAvoidLargeDocumentsAndKeepLegacyLinks",
"TestProductionOverviewBoundsRelatedQueriesAndKeepsLegacyUnion",
"TestLegacyAcceptanceWithoutOperationIDIsNotCoveredByUnrelatedPlan",
}
@ -37,11 +47,12 @@ def main():
"./internal/workforcestats", "./internal/api", "-run",
"TestAnnotationOutput|TestOutput|TestSharedReaders|TestRefreshFailure|TestLeaseRecovery|"
"TestCorruptCurrent|TestBuildCancellation|TestSnapshotScope|TestFailedBuild|"
"TestFailureDiagnostics|TestHealth|TestRouteSnapshot", "-count=1", "-timeout=120s"]
"TestFailureDiagnostics|TestHealth|TestRouteSnapshot|TestSharded|TestProductionFactsIndexes|TestProductionOverviewBounds|TestLegacyAcceptanceWithout", "-count=1", "-timeout=120s"]
timing = ["go", "test", "-race=false", "-json", "./internal/api", "-run", "^TestAnnotationOutputSnapshotScale$", "-count=1", "-timeout=120s"]
scale = {"TestAnnotationOutputSnapshotScale", "TestAnnotationOutputSnapshotScale/production-quality-concurrent-warm-reads"}
growth = ["go", "test", "-json", "./internal/store", "-run", "^TestStatisticsPartitionGrowthBeyondGlobalBudget$", "-count=1", "-timeout=120s"]
verdicts = []
for name, args, required in [("race", command, REQUIRED | scale), ("timing", timing, scale)]:
for name, args, required in [("race", command, REQUIRED | scale), ("timing", timing, scale), ("partition-growth", growth, {"TestStatisticsPartitionGrowthBeyondGlobalBudget"})]:
try:
result = subprocess.run(args, cwd=ROOT, env=env, capture_output=True, text=True, timeout=240)
except (OSError, subprocess.TimeoutExpired):

View File

@ -410,6 +410,8 @@
{
"name": "dashboards",
"paths": [
"^docs/operations/production-overview-(indexes\\.json|statistics-growth\\.md)$",
"^(scripts/test_dashboard_stability|tests/release/test_dashboard_stability)\\.py$",
"^src/.*([Dd]ashboard|WorkforceStatistics|annotation-output)",
"^internal/(api|store)/(production_(dashboard|stats)|annotation_output|workforce)",
"^internal/(outputsnapshot|workforcestats)/",
@ -435,6 +437,10 @@
{
"argv": ["python3", "scripts/test_dashboard_stability.py"]
},
{
"argv": ["python3", "-m", "unittest", "discover", "-s", "tests/release", "-p", "test_dashboard_stability.py"],
"tests": true
},
{
"argv": [
"go",

View File

@ -44,12 +44,14 @@ class DashboardStabilityTests(unittest.TestCase):
calls = []
def run(command, **kwargs):
calls.append(command)
names = gate.REQUIRED | scale if "-race" in command else scale
names = gate.REQUIRED | scale if "-race" in command else ({"TestStatisticsPartitionGrowthBeyondGlobalBudget"} if "^TestStatisticsPartitionGrowthBeyondGlobalBudget$" in command else scale)
return SimpleNamespace(returncode=int(fail_timing and "-race" not in command),
stdout="\n".join(json.dumps({"Action": "pass", "Test": name}) for name in names), stderr="")
with patch.object(gate.subprocess, "run", side_effect=run), patch("builtins.print"):
self.assertEqual(gate.main(), int(fail_timing))
self.assertEqual(len(calls), 2)
self.assertEqual(len(calls), 3)
self.assertIn("-race", calls[0])
self.assertNotIn("-race", calls[1])
self.assertIn("-race=false", calls[1])
self.assertNotIn("-race", calls[2])
self.assertIn("^TestStatisticsPartitionGrowthBeyondGlobalBudget$", calls[2])