diff --git a/docs/adr/0014-independent-query-database.md b/docs/adr/0014-independent-query-database.md index 458f8319..04c9ec2e 100644 --- a/docs/adr/0014-independent-query-database.md +++ b/docs/adr/0014-independent-query-database.md @@ -30,3 +30,12 @@ ## 验收 详见 [操作与验收说明](../operations/query-replica.md)。必须比较原库总 CPU、扫描和读取字节,包含新索引、标记、确认和对账成本。短时本机测试和无 CPU 配额的 native 测试不构成生产容量验收。 + + +## 查询库内事实聚合(2026-11-13,本地实施) + +查询库事务同时维护按服务端可见性字段分组的计数桶与代次证书。读取桶、证书及待同步旧记录使用同一查询库快照;服务端实时运行原权限判定。旧工作者导致证书代次落后、身份敏感授权及复杂筛选继续原候选路径,不使用近似计数。 + +人员统计复用原历史计算器生成无正文贡献,按不可变版本发布。完整新版本提交前始终保留上一版本;重启清理未提交暂存,已提交历史按有界宽限清理。历史供应商归属和正式操作抑制旧审查仍依查询范围解析,日期边界保留纳秒。后台当前使用有界组织快照计算与差量写入,尚非逐任务分区增量计算;超预算保留旧版本,不截断数据、不增加核心作业准入条件。 + +作业记录在查询库筛选计数及分页后,仅补选中页详情与权威文字,来源指纹不一致时拒绝拼接。旧任务/未首次物化的库保留读库历史兼容路径。极少数等价时间字符串的排序统一为绝对时间和稳定身份,避免偏移页漂移。以上实现仍需完整规模性能及持久化确认压力验收,不能将本地功能测试当作上线承诺。 diff --git a/docs/operations/query-replica-query25-comparison.md b/docs/operations/query-replica-query25-comparison.md index 00fc83ff..b6d962c1 100644 --- a/docs/operations/query-replica-query25-comparison.md +++ b/docs/operations/query-replica-query25-comparison.md @@ -164,3 +164,29 @@ IO保护新增严格的FUSE归因分支,并有先失败后通过的回归: 下一步应先减少读库端每请求的全范围工作:任务池计数建立按权限范围可组合的轻量事实/索引聚合;人员统计按冻结作者和阶段事实增量物化;作业记录先在事实表筛选分页,再补当前页文本;控制后台构建并发和发布节奏,保留上一完整代次。权限撤销与领取/提交仍实时核验,不能用过期事实取代权威判断。共享Valkey的跨进程正确性已在本机验证,本轮集群夹具未接Valkey,不能声称已验证完整共享缓存组合。单纯提高API核数、延长HTTP超时、跳过冷查询或放宽本地盘保护都不算修复。若后续评估更大的查询库内存/IO预算,须单列资源方案与对照,不能混入本轮结果。 本轮临时源/驱动Pod与两条策略已按UID清理,桥接按PID/可执行文件身份停止,配额CAS恢复1Pod/1CPU/1Gi;25Gi库和持久标记保留,查询Pod仍按2小时deadline退出。未切换任何生产读取或修改客户数据。清理证据在`io-rca/fixture-5/cleanup.json`。 + +## 查询库事实物化与分页(2026-11-13) + +本批以 `5e59ae50` 为起点,仅在 `codex/mongo-cpu` 和合成隔离环境实现,未切换生产路由,未修改客户数据。 + +- 计数:独立查询库事务内维护 `queryPoolCountBuckets`;权限维度分组只减少候选传输,不缓存授权结果。服务端仍按当前成员、批次、供应商及验收权限判定。脏权威记录在同一请求扣旧加新;桶、代次证书及旧行在同一查询库快照读取。身份敏感授权、复杂搜索和旧桶未建好的库继续原逐条路径。旧投影工作者导致桶代次不一致时安全回退。 +- 统计:复用原历史计算器输出无正文的事件、冻结作者、返工和验收贡献,在 `queryWorkforceFacts` 发布不可变代次。筛选和聚合在查询库执行,最终摘要进入有界进程/Valkey缓存。返工任务去重不按日去重数量相加;历史供应商归属及正式操作抑制旧记录仍按原查询范围解析。时间边界保留纳秒精度。显式空用户兼容查询继续原读库路径。 +- 作业记录:先筛选、排序、分页及计数,再读取本页事实详情;只按本页操作/旧审查 ID 从权威库补文字。来源元数据指纹不符时返回未就绪,避免新旧归属或批注拼接。相同绝对时间但不同时区/小数写法的旧记录改用稳定身份顺序,避免分页漂移。 +- 恢复:新代次未完整写完不切换 head;重启只清理未来暂存行/退役标记。无变化不重复写事实。已提交旧代次至少保留15分钟,后台每次最多200行清理,不使用可能误删当前版本的TTL。定期范围清理保留新增事实 head。 + +兼容回归使用真实隔离 Mongo,覆盖旧记录、权限范围、脏记录覆盖、旧工作者、删除、指纹变化、重启恢复、无变化零写入和分页正文恢复。纯贡献浮点时长允许因加法顺序产生 `1e-8` 内误差,整数/身份严格一致;HTTP合成对照仍要求业务JSON精确等价。完整任务检查 `verify-1794573005303254000/report.json` 全部所选检查通过,含浏览器、Mongo `--race --cache` 及查询服务故障恢复。此前运行因执行中源码变化而标为未完成,原记录保留。 + +限制:后台当前是有界组织快照计算+差量发布,并非已完成逐任务分区计算;单次构建限制64MiB原始BSON/250,000行,超限保留上一完整版本并报错,不截断数据,也不阻断标注作业。规模超过此预算仍需分区计算验证。事实尚未首次发布时兼容读取独立查询库原历史,不回扫生产权威历史。集群本轮没有Valkey,不能当作完整共享缓存容量证据。短两轮不等于30分钟三轮或200/500人验收。 + + +### 本轮同规模结果:冷构建仍未通过,尚无HTTP容量结论 + +`io-rca/scale-81-8-facts` 按原26,218任务、81会话、两轮及固定资源/IO预算实际执行。源码未变化;回填294.551秒,2次租约沿检查点恢复。统计事实及生产读表构建88.372秒后成功;随后产出报表快照续租在3秒预算内未完成,分片构建取消,失败状态清理也超时。4分钟就绪等待失败,测试总计565.97秒,尚未进入管理员HTTP等价和81人请求阶段。`coverageComplete=false`,没有 `comparison.json`;因此不能报告本轮p95或源库节省比例,也不能说新路径已达到2秒。 + +`slow-metadata-complete.json` 仅保留隔离库命令名称、扫描/CPU/存储与确认等待,不含正文或过滤条件。多条小控制写CPU约0.3~0.8ms,总等待14~21秒,`waitForWriteConcernDurationMillis`几乎覆盖总耗时(最高20,857ms)。这说明本轮明确卡在受限IO下的持久化确认,不能继续把所有超时都归为大范围历史读取或CPU不足;是否由检查点、journal或不同后台写入共同触发仍需单独分段验证,不直接认定某一种机制。 + +122次保护采样覆盖约689.5秒:节点CPU最高6%,物理盘平均等待最高约1.37ms、忙碌最高约7.74%;保护器未停测,最终健康采样无错误。这只能说明本轮未触发既有业务健康/IO保护,不代表已证明对所有客户作业零影响。原业务、客户记录、资源与读路由均未改动。 + +下一步先处理独立查询库冷构建的写入节奏、共享控制写和失败恢复:记录每类后台写入/确认等待、检查点及限速窗口,减少重复写和突发积压;保持多数提交和版本隔离,不靠延长HTTP超时、降低写持久性或扩大IO限额掩盖问题。完成后再运行同规模两轮;任何更长时间或更大资源对照需另行界定。之前扩大20轮被自动审批以共享节点持续负载风险拒绝,未执行。 + +本轮自有合成源Pod及两条策略已按UID清理,桥接按PID/可执行文件身份停止,配额CAS恢复1Pod/1CPU/1Gi。25Gi卷及持久标记保留,查询Pod仍按3600秒deadline退出,未延长;清理证据 `io-rca/fixture-6/cleanup.json`。本批三项功能实现与本地回归完成,完整规模性能验收仍不通过,不准备生产发布。 diff --git a/internal/api/human_pool_reads.go b/internal/api/human_pool_reads.go index d14408c9..693cf3ae 100644 --- a/internal/api/human_pool_reads.go +++ b/internal/api/human_pool_reads.go @@ -205,6 +205,46 @@ func (s *server) countHumanPoolCandidates(w http.ResponseWriter, r *http.Request } count := 0 batches := map[string]*domain.HumanBatchRecord{} + tryGroups := true + if mode, ok := s.store.(interface{ UsesQueryReplicaCountRead(context.Context) bool }); ok { + tryGroups = mode.UsesQueryReplicaCountRead(r.Context()) + } + if grouped, yes := s.store.(interface { + VisitHumanPoolCountGroups(context.Context, string, domain.HumanItemQuery, func(domain.HumanItemRecord, int) error) (bool, error) + }); yes && tryGroups { + principal, _ := currentPrincipal(r) + if principal != nil && !annotationManager(principal) { + roles, err := s.annotationMembershipsForUser(r.Context(), s.requestOrganizationID(r), query.ProjectID, principal.User.ID) + if err != nil { + renderError(w, 500, err) + return 0, r, false + } + for _, role := range roles { + if len(role.DrainTaskIDs) > 0 { + query.CountIdentitySensitive = true + } + } + } + handled, err := grouped.VisitHumanPoolCountGroups(r.Context(), s.requestOrganizationID(r), query, func(item domain.HumanItemRecord, weight int) error { + ctx, err := s.withHumanListReads(r.Context(), s.requestOrganizationID(r), query.ProjectID, []domain.HumanItemRecord{item}) + if err != nil { + return err + } + r = r.WithContext(ctx) + _, visible, err := s.filterHumanPoolItem(r, item, pool, query, batches) + if visible { + count += weight + } + return err + }) + if err != nil { + renderError(w, 500, err) + return 0, r, false + } + if handled { + return count, r, true + } + } err := s.store.VisitHumanPoolCountCandidates(r.Context(), s.requestOrganizationID(r), query, func(items []domain.HumanItemRecord) error { ctx, err := s.withHumanListReads(r.Context(), s.requestOrganizationID(r), query.ProjectID, items) if err != nil { diff --git a/internal/api/query_fact_routes_test.go b/internal/api/query_fact_routes_test.go new file mode 100644 index 00000000..9f3bfcf5 --- /dev/null +++ b/internal/api/query_fact_routes_test.go @@ -0,0 +1,54 @@ +package api + +import ( + "context" + "droboticflow/internal/domain" + "droboticflow/internal/store" + "droboticflow/internal/workforcestats" + "encoding/json" + "testing" +) + +type queryFactRoutesProbe struct { + store.Store + pageCalls, groupCalls, rawCalls int + item domain.HumanItemRecord +} + +func (p *queryFactRoutesProbe) LoadWorkforceReturnPage(ctx context.Context, org string, q workforcestats.Query, limit, offset int) ([]domain.WorkforceReturnDetail, int, error) { + p.pageCalls++ + return []domain.WorkforceReturnDetail{{ID: "selected-page", Problems: []domain.ReviewProblem{}}}, 37, nil +} +func (p *queryFactRoutesProbe) LoadWorkforceStats(ctx context.Context, org string, q workforcestats.Query) (workforcestats.Snapshot, error) { + p.rawCalls++ + return p.Store.LoadWorkforceStats(ctx, org, q) +} +func (p *queryFactRoutesProbe) VisitHumanPoolCountGroups(ctx context.Context, org string, q domain.HumanItemQuery, visit func(domain.HumanItemRecord, int) error) (bool, error) { + p.groupCalls++ + return true, visit(p.item, 7) +} +func TestWorkforceReturnsUsesAuthorizedPagedReader(t *testing.T) { + f := sharedAcceptanceFixture(t) + p := &queryFactRoutesProbe{Store: f.store, item: f.item} + f.handler = NewHandler(p, nil) + r := f.request("GET", "/api/workforce-returns?from=2026-11-01T00:00:00Z&to=2026-11-13T00:00:00Z&limit=20&offset=20", f.reviewer, nil) + var body struct { + Items []domain.WorkforceReturnDetail + Total int + } + _ = json.Unmarshal(r.Body.Bytes(), &body) + if r.Code != 200 || p.pageCalls != 1 || p.rawCalls != 0 || body.Total != 37 || len(body.Items) != 1 { + t.Fatalf("page reader not used: status=%d page=%d raw=%d body=%s", r.Code, p.pageCalls, p.rawCalls, r.Body.String()) + } +} +func TestHumanPoolCountsUsesWeightedFactsWithCurrentAuthorization(t *testing.T) { + f := sharedAcceptanceFixture(t) + p := &queryFactRoutesProbe{Store: f.store, item: f.item} + f.handler = NewHandler(p, nil) + r := f.request("GET", "/api/human-item-counts?pluginId=annotation-qa", f.reviewer, nil) + var body struct{ Counts map[string]int } + _ = json.Unmarshal(r.Body.Bytes(), &body) + if r.Code != 200 || p.groupCalls == 0 || body.Counts["available"] != 7 { + t.Fatalf("groups not used: %d calls=%d %s", r.Code, p.groupCalls, r.Body.String()) + } +} diff --git a/internal/api/workforce_statistics.go b/internal/api/workforce_statistics.go index a01c284f..671d14e7 100644 --- a/internal/api/workforce_statistics.go +++ b/internal/api/workforce_statistics.go @@ -1,6 +1,7 @@ package api import ( + "context" "fmt" "net/http" "strconv" @@ -88,12 +89,22 @@ func (s *server) handleAnnotationWorkloadSummary(w http.ResponseWriter, r *http. return } q.GroupTemplate = true - d, err := s.store.LoadWorkforceStats(r.Context(), s.requestOrganizationID(r), q) + var report workforcestats.Report + if prepared, ok := s.store.(interface { + LoadWorkforceReport(context.Context, string, workforcestats.Query) (workforcestats.Report, error) + }); ok { + report, err = prepared.LoadWorkforceReport(r.Context(), s.requestOrganizationID(r), q) + } else { + var d workforcestats.Snapshot + d, err = s.store.LoadWorkforceStats(r.Context(), s.requestOrganizationID(r), q) + if err == nil { + report = workforcestats.Build(d, q) + } + } if err != nil { renderError(w, 500, err) return } - report := workforcestats.Build(d, q) items := make([]domain.AnnotationWorkloadAggregate, 0, len(report.Items)) for _, m := range report.Items { items = append(items, domain.AnnotationWorkloadAggregate{ @@ -155,6 +166,17 @@ func (s *server) handleWorkforceReturns(w http.ResponseWriter, r *http.Request) } } q.Details = true + if paged, ok := s.store.(interface { + LoadWorkforceReturnPage(context.Context, string, workforcestats.Query, int, int) ([]domain.WorkforceReturnDetail, int, error) + }); ok { + items, total, e := paged.LoadWorkforceReturnPage(r.Context(), s.requestOrganizationID(r), q, limit, offset) + if e != nil { + renderError(w, 500, e) + return + } + renderJSON(w, 200, map[string]any{"items": items, "total": total, "from": q.From.Format(time.RFC3339Nano), "to": q.To.Format(time.RFC3339Nano)}) + return + } d, err := s.store.LoadWorkforceStats(r.Context(), s.requestOrganizationID(r), q) if err != nil { renderError(w, 500, err) diff --git a/internal/domain/human_loop.go b/internal/domain/human_loop.go index 5d3b2141..1ec4e61b 100644 --- a/internal/domain/human_loop.go +++ b/internal/domain/human_loop.go @@ -359,6 +359,9 @@ type HumanItemCursor struct { } type HumanItemQuery struct { + // Server-only: draining supplier grants require per-task identity checks. + CountIdentitySensitive bool + InheritedUnassigned bool NeedsReworkRouting bool diff --git a/internal/store/query_model_maintenance.go b/internal/store/query_model_maintenance.go index 70dc39e3..fcf2e7b9 100644 --- a/internal/store/query_model_maintenance.go +++ b/internal/store/query_model_maintenance.go @@ -11,6 +11,11 @@ import ( // This is an explicit derived-data index operation, never API startup behavior. // Production execution requires its own reviewed migration/index authorization. func (s *MongoStore) EnsureQueryReadModelIndexes(ctx context.Context) error { + if s.queryReplicaFactStore { + if err := s.ensureWorkforceFactIndexes(ctx); err != nil { + return err + } + } _, err := s.queryData().Indexes().CreateOne(ctx, mongo.IndexModel{Keys: bson.D{{Key: "generation", Value: 1}, {Key: "organizationId", Value: 1}, {Key: "itemId", Value: 1}}, Options: options.Index().SetName("query_model_rows_v1")}) return err } @@ -76,6 +81,7 @@ func (s *MongoStore) pruneQueryModelScopes(ctx context.Context, l queryModelLeas for org, projects := range scopes { keep[queryModelKey("production", org, projects)] = true keep[queryModelKey("search", org, nil)] = true + keep[workforceFactKey(org)] = true } unset := bson.M{} for key := range st.Heads { diff --git a/internal/store/query_pool_buckets.go b/internal/store/query_pool_buckets.go new file mode 100644 index 00000000..ad3b1f81 --- /dev/null +++ b/internal/store/query_pool_buckets.go @@ -0,0 +1,219 @@ +package store + +import ( + "context" + "droboticflow/internal/domain" + "droboticflow/internal/queryserving" + "errors" + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/mongo" + "go.mongodb.org/mongo-driver/mongo/options" + "go.mongodb.org/mongo-driver/mongo/readconcern" + "time" +) + +const queryPoolBuckets = "queryPoolCountBuckets" +const queryPoolBucketSchema = "pool-count/v1" + +// These are the inputs of the API's visibility predicate, not an authorization +// result. Identity-sensitive grant draining uses the legacy per-record reader. +var poolBucketFields = []string{"organizationId", "pluginId", "projectId", "batchId", "supplierId", "processingScope", "processingProfile", "workgroupId", "vendorId", "currentStageId", "status", "assignedUserId", "latestAuthorUserId", "acceptanceClaim", "reviewRound", "claimedAt", "inheritedRework"} + +func poolBucketProjection() bson.M { + p := bson.M{"_id": 1} + for _, field := range poolBucketFields { + p[field] = 1 + } + return p +} +func poolBucket(doc bson.M) (string, bson.M, error) { + projected := bson.M{} + for _, field := range poolBucketFields { + if value, ok := doc[field]; ok { + projected[field] = value + } + } + raw, err := bson.Marshal(projected) + if err != nil { + return "", nil, err + } + var typed domain.HumanItemRecord + if err = bson.Unmarshal(raw, &typed); err != nil { + return "", nil, err + } + out := bson.M{} + for _, field := range poolBucketFields { + value := doc[field] + switch field { + case "reviewRound": + value = int(supplierNumber(value)) + case "inheritedRework": + value = value == true + case "acceptanceClaim": + if value == nil { + value = nil + } + default: + text, _ := value.(string) + value = text + } + out[field] = value + } + return queryserving.Key(queryPoolBucketSchema, out), out, nil +} + +type poolBucketDelta struct { + doc bson.M + count int64 +} + +func addPoolBucketDelta(deltas map[string]*poolBucketDelta, doc bson.M, count int64) error { + if len(doc) == 0 { + return nil + } + key, row, err := poolBucket(doc) + if err != nil { + return err + } + v := deltas[key] + if v == nil { + v = &poolBucketDelta{doc: row} + deltas[key] = v + } + v.count += count + return nil +} +func (q *QueryReplica) writePoolBucketDeltas(ctx mongo.SessionContext, deltas map[string]*poolBucketDelta) error { + models := []mongo.WriteModel{} + for key, delta := range deltas { + if delta.count != 0 { + models = append(models, mongo.NewUpdateOneModel().SetFilter(bson.M{"_id": key}).SetUpdate(bson.M{"$setOnInsert": delta.doc, "$inc": bson.M{"count": delta.count}}).SetUpsert(true)) + } + } + if len(models) == 0 { + return nil + } + _, err := q.Target.db.Collection(queryPoolBuckets).BulkWrite(ctx, models) + return err +} +func (q *QueryReplica) ensurePoolBucketIndexes(ctx context.Context) error { + _, err := q.Target.db.Collection(queryPoolBuckets).Indexes().CreateOne(ctx, mongo.IndexModel{Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "pluginId", Value: 1}, {Key: "currentStageId", Value: 1}, {Key: "projectId", Value: 1}}, Options: options.Index().SetName("pool_bucket_scope_v1")}) + if err != nil { + return err + } + // Existing read stores without buckets retain the original reader until an + // explicit rebuild. A newly empty replica starts counting in its transactions. + n, err := q.Target.humanItems.CountDocuments(ctx, bson.M{}, options.Count().SetLimit(1)) + if err != nil || n != 0 { + return err + } + _, err = q.control().UpdateOne(ctx, bson.M{"_id": "state"}, bson.M{"$setOnInsert": bson.M{"poolBucketSchema": queryPoolBucketSchema, "poolBucketEpoch": int64(0)}}, options.Update().SetUpsert(true)) + return err +} + +// VisitHumanPoolCountGroups never supplies an authorized total. Every group is +// checked using current server memberships/batches. Dirty authoritative rows +// subtract their old group and add their current row in this same request. +func (s *MongoStore) VisitHumanPoolCountGroups(ctx context.Context, org string, q domain.HumanItemQuery, visit func(domain.HumanItemRecord, int) error) (bool, error) { + target, err := s.poolReplica(ctx) + if err != nil { + return true, err + } + if target == nil { + return false, nil + } + if humanPoolFiltered(q) || q.BlindAnnotationActorID != "" || q.NeedsReworkRouting || q.InheritedUnassigned || q.WorkflowID != "" || q.RunID != "" || q.LegacyTaskID != "" || q.AnnotationView != "" || q.IDs != nil || q.CountIdentitySensitive { + return false, nil + } + b, err := s.countReplicaBarrier(ctx, org) + if err != nil { + return true, err + } + type weightedRow struct { + domain.HumanItemRecord `bson:",inline"` + Count int `bson:"count"` + } + rows := []weightedRow{} + session, e := target.db.Client().StartSession() + if e != nil { + return true, e + } + defer session.EndSession(ctx) + capacity := errors.New("count groups exceed request budget") + incompatible := errors.New("count bucket writer compatibility changed") + _, err = session.WithTransaction(ctx, func(sc mongo.SessionContext) (any, error) { + rows = nil + var state struct { + Schema string `bson:"poolBucketSchema"` + FactEpoch int64 `bson:"factEpoch"` + BucketEpoch int64 `bson:"poolBucketEpoch"` + } + if e := target.db.Collection(queryReplicaControl).FindOne(sc, bson.M{"_id": "state"}).Decode(&state); e != nil { + return nil, e + } + if state.Schema != queryPoolBucketSchema || state.FactEpoch != state.BucketEpoch { + return nil, incompatible + } + filter := humanItemMongoFilter(org, q) + filter["count"] = bson.M{"$gt": 0} + cur, e := target.db.Collection(queryPoolBuckets).Find(sc, filter, options.Find().SetMaxTime(8*time.Second).SetLimit(10001)) + if e != nil { + return nil, e + } + e = cur.All(sc, &rows) + cur.Close(sc) + if e != nil { + return nil, e + } + if len(rows) > 10000 { + return nil, capacity + } + if b == nil || len(b.ids) == 0 { + return nil, nil + } + ids := make([]string, 0, len(b.ids)) + for id := range b.ids { + ids = append(ids, id) + } + oldFilter := bson.M{"$and": bson.A{humanItemMongoFilter(org, q), bson.M{"id": bson.M{"$in": ids}}}} + old, e := target.humanItems.Find(sc, oldFilter, options.Find().SetProjection(poolBucketProjection())) + if e != nil { + return nil, e + } + defer old.Close(sc) + for old.Next(sc) { + var row domain.HumanItemRecord + if e = old.Decode(&row); e != nil { + return nil, e + } + rows = append(rows, weightedRow{HumanItemRecord: row, Count: -1}) + } + return nil, old.Err() + }, options.Transaction().SetReadConcern(readconcern.Snapshot())) + if errors.Is(err, capacity) || errors.Is(err, incompatible) { + return false, nil + } + if err != nil { + return true, err + } + for _, row := range rows { + if e := ctx.Err(); e != nil { + return true, e + } + if e := visit(row.HumanItemRecord, row.Count); e != nil { + return true, e + } + } + if b == nil { + return true, ctx.Err() + } + err = s.visitReplicaPendingCounts(ctx, org, q, b, func(rows []domain.HumanItemRecord) error { + for _, row := range rows { + if e := visit(row, 1); e != nil { + return e + } + } + return nil + }) + return true, err +} diff --git a/internal/store/query_pool_buckets_local_test.go b/internal/store/query_pool_buckets_local_test.go new file mode 100644 index 00000000..e5edbb08 --- /dev/null +++ b/internal/store/query_pool_buckets_local_test.go @@ -0,0 +1,95 @@ +package store + +import ( + "context" + "droboticflow/internal/domain" + "fmt" + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/event" + "os" + "sync/atomic" + "testing" + "time" +) + +func TestQueryReplicaPoolBucketsIncrementalAndPendingLocal(t *testing.T) { + if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" { + t.Skip("owned synthetic replica required") + } + var rowReads atomic.Int64 + monitor := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) { + if e.CommandName == "find" { + if v, ok := e.Command.Lookup("find").StringValueOK(); ok && v == "humanItems" { + rowReads.Add(1) + } + } + }} + source, _, replica := newLocalReplica(t, nil, monitor) + ctx := t.Context() + for i := 0; i < 120; i++ { + if err := source.UpsertHumanItem(ctx, domain.HumanItemRecord{ID: fmt.Sprint("i", i), OrganizationID: "o", ProjectID: "p", BatchID: "b", PluginID: "annotation-qa", CurrentStageID: "quality_review", Status: domain.HumanItemReady, LatestAuthorUserID: fmt.Sprint("u", i%3)}); err != nil { + t.Fatal(err) + } + } + finishLocalReplica(t, replica) + if err := source.ConfigureQueryReplicaReports(replica.Target, time.Minute); err != nil { + t.Fatal(err) + } + source.EnableQueryReplicaCountReads(true) + count := func(q domain.HumanItemQuery) (int, int) { + t.Helper() + n, groups := 0, 0 + handled, err := source.VisitHumanPoolCountGroups(WithQueryReplicaCountRead(ctx), "o", q, func(row domain.HumanItemRecord, weight int) error { n += weight; groups++; return nil }) + if err != nil || !handled { + t.Fatalf("group reader unavailable: %v/%v", handled, err) + } + return n, groups + } + q := domain.HumanItemQuery{PluginID: "annotation-qa"} + rowReads.Store(0) + if n, groups := count(q); n != 120 || groups != 3 { + t.Fatalf("count=%d groups=%d", n, groups) + } + if rowReads.Load() != 0 { + t.Fatal("clean count loaded individual query items") + } + if _, err := source.humanItems.UpdateOne(ctx, bson.M{"id": "i0"}, bson.M{"$set": bson.M{"assignedUserId": "claimant", "status": "claimed"}}); err != nil { + t.Fatal(err) + } + q.UnassignedOnly = true + q.Statuses = []domain.HumanItemStatus{domain.HumanItemReady} + if n, _ := count(q); n != 119 { + t.Fatalf("pending overlay=%d", n) + } + if _, err := replica.RunOnce(ctx, false); err != nil { + t.Fatal(err) + } + if n, _ := count(q); n != 119 { + t.Fatalf("committed overlay=%d", n) + } + if _, err := source.humanItems.DeleteOne(ctx, bson.M{"id": "i1"}); err != nil { + t.Fatal(err) + } + for i := 0; i < 10; i++ { + if _, err := replica.RunOnce(ctx, false); err != nil { + t.Fatal(err) + } + } + if n, _ := count(q); n != 118 { + t.Fatalf("deletion left a counted bucket: %d", n) + } + // An old projector can still update the ordinary replica after rollback. + // Its missing bucket epoch invalidates the fast path, not permissions. + if _, err := replica.control().UpdateOne(ctx, bson.M{"_id": "state"}, bson.M{"$inc": bson.M{"factEpoch": int64(1)}}); err != nil { + t.Fatal(err) + } + used, e := source.VisitHumanPoolCountGroups(WithQueryReplicaCountRead(ctx), "o", q, func(domain.HumanItemRecord, int) error { t.Fatal("mixed old projector used stale buckets"); return nil }) + if e != nil || used { + t.Fatal(used, e) + } + q.CountIdentitySensitive = true + handled, err := source.VisitHumanPoolCountGroups(WithQueryReplicaCountRead(ctx), "o", q, func(domain.HumanItemRecord, int) error { t.Fatal("identity grant was grouped"); return nil }) + if handled || err != nil { + t.Fatal("identity fallback changed", handled, err) + } +} diff --git a/internal/store/query_read_builder.go b/internal/store/query_read_builder.go index 3e32edbe..452cea3e 100644 --- a/internal/store/query_read_builder.go +++ b/internal/store/query_read_builder.go @@ -40,7 +40,7 @@ func (b *QueryReadBuilder) Close() { b.reader.Close() } // WaitReady is for explicit prewarming and isolated acceptance, not HTTP reads. func (b *QueryReadBuilder) WaitReady(ctx context.Context, org string, projects []string) error { - tick := time.NewTicker(100 * time.Millisecond) + tick := time.NewTicker(500 * time.Millisecond) defer tick.Stop() for { productionHead, production := b.store.readQueryModelHead(ctx, queryModelKey("production", org, projects), 5*time.Minute) @@ -53,7 +53,12 @@ func (b *QueryReadBuilder) WaitReady(ctx context.Context, org string, projects [ health, e := b.reader.Inspect(ctx, org, projects) output = e == nil && health.Available } - if production && search && output { + workforce := true + if b.store.queryReplicaFactStore { + _, workforce, err = b.store.workforceHead(ctx, org) + workforce = workforce && err == nil + } + if production && search && output && workforce { return nil } select { @@ -126,6 +131,11 @@ func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org s if err != nil { return err } + if b.store.queryReplicaFactStore { + if err = b.store.rebuildWorkforceFacts(buildCtx, l, org, source.Revision); err != nil { + return err + } + } head.SourceRevision = source.Revision head.GlobalRevision = source.Global if err = b.store.publishQueryModelHead(ctx, l, key, head); err != nil { @@ -346,6 +356,7 @@ func (b *QueryReadBuilder) run(ctx context.Context, started chan<- struct{}) err refreshScopes := true lastScopeRead := time.Time{} lastCleanup := time.Now() + lastWorkforceCleanup := time.Now() for { b.mu.Lock() all, allSearch := b.all, b.allSearch @@ -396,7 +407,16 @@ func (b *QueryReadBuilder) run(ctx context.Context, started chan<- struct{}) err log.Print("query_search_build result=retry category=source_or_publication") } } + outputBusy := false + if state, e := b.store.OutputSnapshotState(ctx); e != nil { + outputBusy = true + } else { + outputBusy = state.Owner != "" && time.Now().Before(state.Until) && state.ActiveEpoch == state.Epoch + } for org := range pendingStats { + if outputBusy { + continue + } ids, exists := scopes[org] if !exists { delete(pendingStats, org) @@ -422,9 +442,16 @@ func (b *QueryReadBuilder) run(ctx context.Context, started chan<- struct{}) err if time.Since(lastCleanup) > time.Minute { cleanup, finish := context.WithTimeout(ctx, 5*time.Second) _ = b.store.cleanQueryModelOrphans(cleanup) + finish() lastCleanup = time.Now() } + if b.store.queryReplicaFactStore && time.Since(lastWorkforceCleanup) > 5*time.Second { + cleanup, finish := context.WithTimeout(ctx, 2*time.Second) + _ = b.store.maintainWorkforceVersions(cleanup, l) + finish() + lastWorkforceCleanup = time.Now() + } // Prewarm due generations even when no browser is connected. for org, ids := range scopes { if len(ids) > 0 { diff --git a/internal/store/query_read_models.go b/internal/store/query_read_models.go index 110501ac..fa328a8c 100644 --- a/internal/store/query_read_models.go +++ b/internal/store/query_read_models.go @@ -71,6 +71,7 @@ func (s *MongoStore) SharedQueryCache() queryserving.Cache { } type QueryModelHead struct { + FactEpoch int64 `bson:"factEpoch,omitempty" json:"factEpoch,omitempty"` SourceRevision int64 `bson:"sourceRevision" json:"sourceRevision"` Revision int64 `bson:"revision" json:"revision"` GlobalRevision int64 `bson:"globalRevision" json:"globalRevision"` diff --git a/internal/store/query_replica.go b/internal/store/query_replica.go index 4c801230..cdb0e768 100644 --- a/internal/store/query_replica.go +++ b/internal/store/query_replica.go @@ -169,11 +169,17 @@ func (q *QueryReplica) apply(ctx mongo.SessionContext, n string, records []query seen[key] = true ids = append(ids, r.Document["_id"]) } - cur, e := c.Find(ctx, bson.M{"_id": bson.M{"$in": ids}}, options.Find().SetProjection(bson.M{"_queryReplica": 1})) + projection := bson.M{"_queryReplica": 1} + if n == "humanItems" { + projection = poolBucketProjection() + projection["_queryReplica"] = 1 + } + cur, e := c.Find(ctx, bson.M{"_id": bson.M{"$in": ids}}, options.Find().SetProjection(projection)) if e != nil { return e } current := map[string]bson.M{} + oldPoolRows := map[string]bson.M{} for cur.Next(ctx) { var d bson.M if e = cur.Decode(&d); e != nil { @@ -181,6 +187,9 @@ func (q *QueryReplica) apply(ctx mongo.SessionContext, n string, records []query return e } current[querySourceIdentity(n, d["_id"])] = deletionDoc(d["_queryReplica"]) + if n == "humanItems" { + oldPoolRows[querySourceIdentity(n, d["_id"])] = d + } } e = cur.Err() cur.Close(ctx) @@ -188,6 +197,7 @@ func (q *QueryReplica) apply(ctx mongo.SessionContext, n string, records []query return e } models := make([]mongo.WriteModel, 0, len(records)) + deltas := map[string]*poolBucketDelta{} for i, r := range records { m := current[querySourceIdentity(n, r.Document["_id"])] records[i].TargetSourceID = m["sourceId"] @@ -197,6 +207,14 @@ func (q *QueryReplica) apply(ctx mongo.SessionContext, n string, records []query } records[i].TargetSourceID = r.SourceID records[i].TargetRevision = r.Revision + if n == "humanItems" { + if err := addPoolBucketDelta(deltas, oldPoolRows[querySourceIdentity(n, r.Document["_id"])], -1); err != nil { + return err + } + if err := addPoolBucketDelta(deltas, r.Document, 1); err != nil { + return err + } + } models = append(models, mongo.NewReplaceOneModel().SetFilter(bson.M{"_id": r.Document["_id"]}).SetReplacement(r.Document).SetUpsert(true)) } if len(models) == 0 { @@ -206,7 +224,10 @@ func (q *QueryReplica) apply(ctx mongo.SessionContext, n string, records []query if e != nil { return e } - _, e = q.control().UpdateOne(ctx, bson.M{"_id": "state"}, bson.M{"$inc": bson.M{"factEpoch": int64(1), "collectionEpoch." + n: int64(1)}}, options.Update().SetUpsert(true)) + if e = q.writePoolBucketDeltas(ctx, deltas); e != nil { + return e + } + _, e = q.control().UpdateOne(ctx, bson.M{"_id": "state"}, bson.M{"$inc": bson.M{"factEpoch": int64(1), "poolBucketEpoch": int64(1), "collectionEpoch." + n: int64(1)}}, options.Update().SetUpsert(true)) return e } diff --git a/internal/store/query_replica_delete.go b/internal/store/query_replica_delete.go index ff6a7519..c06b8641 100644 --- a/internal/store/query_replica_delete.go +++ b/internal/store/query_replica_delete.go @@ -96,13 +96,31 @@ func (q *QueryReplica) ReconcileDeleteStep(ctx context.Context) (bool, error) { } } e = q.transaction(ctx, func(sc mongo.SessionContext) error { + deltas := map[string]*poolBucketDelta{} for _, r := range absent { - if _, e := q.Target.db.Collection(i.Collection).DeleteOne(sc, bson.M{"_id": r["_id"], "_queryReplica.sourceId": deletionDoc(r["_queryReplica"])["sourceId"]}); e != nil { + filter := bson.M{"_id": r["_id"], "_queryReplica.sourceId": deletionDoc(r["_queryReplica"])["sourceId"]} + var old bson.M + if i.Collection == "humanItems" { + e := q.Target.humanItems.FindOne(sc, filter, options.FindOne().SetProjection(poolBucketProjection())).Decode(&old) + if e != nil && e != mongo.ErrNoDocuments { + return e + } + } + result, e := q.Target.db.Collection(i.Collection).DeleteOne(sc, filter) + if e != nil { return e } + if result.DeletedCount == 1 && i.Collection == "humanItems" { + if err := addPoolBucketDelta(deltas, old, -1); err != nil { + return err + } + } + } + if e := q.writePoolBucketDeltas(sc, deltas); e != nil { + return e } if len(absent) > 0 { - if _, e := q.control().UpdateOne(sc, bson.M{"_id": "state"}, bson.M{"$inc": bson.M{"factEpoch": int64(1), "collectionEpoch." + i.Collection: int64(1)}}, options.Update().SetUpsert(true)); e != nil { + if _, e := q.control().UpdateOne(sc, bson.M{"_id": "state"}, bson.M{"$inc": bson.M{"factEpoch": int64(1), "poolBucketEpoch": int64(1), "collectionEpoch." + i.Collection: int64(1)}}, options.Update().SetUpsert(true)); e != nil { return e } } diff --git a/internal/store/query_replica_indexes.go b/internal/store/query_replica_indexes.go index 2d20979d..72234ae0 100644 --- a/internal/store/query_replica_indexes.go +++ b/internal/store/query_replica_indexes.go @@ -10,6 +10,12 @@ import ( // EnsureTargetIndexes operates only on the explicitly supplied independent // read database. No business migrations/bootstrapping run on this database. func (q *QueryReplica) EnsureTargetIndexes(ctx context.Context) error { + if err := q.Target.ensureWorkforceFactIndexes(ctx); err != nil { + return err + } + if err := q.ensurePoolBucketIndexes(ctx); err != nil { + return err + } for _, name := range replicaCollections() { models := []mongo.IndexModel{{Keys: bson.D{{Key: "id", Value: 1}}, Options: options.Index().SetName("query_id_v1")}, {Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "projectId", Value: 1}, {Key: "createdAt", Value: -1}, {Key: "id", Value: -1}}, Options: options.Index().SetName("query_scope_v1")}} if name == "annotationReviews" || name == "annotationRevisions" || name == "sliceRevisions" || name == "annotationWorklogEvents" { diff --git a/internal/store/query_replica_reads.go b/internal/store/query_replica_reads.go index 959d4c0c..35001a0d 100644 --- a/internal/store/query_replica_reads.go +++ b/internal/store/query_replica_reads.go @@ -284,5 +284,6 @@ func (s *MongoStore) VerifyQueryReplicaTarget(ctx context.Context) error { if state.Schema != queryReplicaSchema || state.Binding.SourceDatabase == "" || state.Binding.Namespace != os.Getenv("QUERY_SERVING_NAMESPACE") || state.Binding.Namespace == "" { return ErrQueryReplicaNotReady } + s.queryReplicaFactStore = true return nil } diff --git a/internal/store/workforce_facts.go b/internal/store/workforce_facts.go new file mode 100644 index 00000000..060efa1f --- /dev/null +++ b/internal/store/workforce_facts.go @@ -0,0 +1,658 @@ +package store + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "log" + "reflect" + "sort" + "time" + + "droboticflow/internal/domain" + "droboticflow/internal/queryserving" + "droboticflow/internal/workforcestats" + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/mongo" + "go.mongodb.org/mongo-driver/mongo/options" + "go.mongodb.org/mongo-driver/mongo/readconcern" + "go.mongodb.org/mongo-driver/mongo/writeconcern" +) + +const workforceFactCollection = "queryWorkforceFacts" +const workforceFactSchema = "workforce-facts/v1" + +type workforceBuildBudgetKey struct{} +type workforceBuildBudget struct{ bytes, rows int } + +var errWorkforceBuildBudget = errors.New("workforce fact source exceeds isolated build budget") + +func workforceBoundedRead(ctx context.Context, cur *mongo.Cursor, target any, budget *workforceBuildBudget) error { + slice := reflect.ValueOf(target).Elem() + for cur.Next(ctx) { + budget.bytes += len(cur.Current) + budget.rows++ + if budget.bytes > 64<<20 || budget.rows > 250000 { + return errWorkforceBuildBudget + } + v := reflect.New(slice.Type().Elem()) + if err := cur.Decode(v.Interface()); err != nil { + return err + } + slice.Set(reflect.Append(slice, v.Elem())) + } + return cur.Err() +} +func workforceFactKey(org string) string { return queryModelKey("workforce-facts", org, nil) } +func (s *MongoStore) ensureWorkforceFactIndexes(ctx context.Context) error { + indexes := []mongo.IndexModel{ + {Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "retiredAt", Value: 1}, {Key: "toEpoch", Value: 1}}, Options: options.Index().SetName("workforce_fact_retirement_v1")}, + {Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "mode", Value: 1}, {Key: "groupTemplate", Value: 1}, {Key: "userId", Value: 1}, {Key: "at", Value: -1}, {Key: "order", Value: 1}}, Options: options.Index().SetName("workforce_fact_user_v1")}, + {Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "mode", Value: 1}, {Key: "groupTemplate", Value: 1}, {Key: "supplierId", Value: 1}, {Key: "projectId", Value: 1}, {Key: "at", Value: -1}}, Options: options.Index().SetName("workforce_fact_scope_v1")}, + {Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "fromEpoch", Value: 1}, {Key: "toEpoch", Value: 1}}, Options: options.Index().SetName("workforce_fact_versions_v1")}, + } + _, err := s.db.Collection(workforceFactCollection).Indexes().CreateMany(ctx, indexes) + return err +} +func (s *MongoStore) workforceHead(ctx context.Context, org string) (QueryModelHead, bool, error) { + var state struct { + Heads map[string]QueryModelHead `bson:"heads"` + Revision int64 `bson:"productionRevision"` + Observed time.Time `bson:"observedAt"` + } + key := workforceFactKey(org) + err := s.queryControl().FindOne(ctx, bson.M{"_id": "global"}, options.FindOne().SetProjection(bson.M{"heads." + key: 1, "productionRevision": 1, "observedAt": 1})).Decode(&state) + if errors.Is(err, mongo.ErrNoDocuments) { + return QueryModelHead{}, false, nil + } + if err != nil { + return QueryModelHead{}, false, err + } + h, exists := state.Heads[key] + if !exists { + return h, false, nil + } + unchanged := h.SourceRevision == state.Revision && !state.Observed.IsZero() && time.Since(state.Observed) >= 0 && time.Since(state.Observed) <= 5*time.Second + if h.Collection != workforceFactCollection || h.FactEpoch < 1 || h.Generation != workforceFactSchema || time.Since(h.AsOf) < 0 || (!unchanged && time.Since(h.AsOf) > dashboardStatsCacheTTL) { + return h, true, ErrQueryReplicaNotReady + } + return h, true, nil +} +func workforceVersionFilter(org string, epoch int64) bson.M { + return bson.M{"organizationId": org, "fromEpoch": bson.M{"$lte": epoch}, "$or": bson.A{bson.M{"toEpoch": bson.M{"$exists": false}}, bson.M{"toEpoch": bson.M{"$gt": epoch}}}} +} +func (s *MongoStore) workforceFactTransaction(ctx context.Context, l queryModelLease, fn func(mongo.SessionContext) error) error { + session, err := s.db.Client().StartSession() + if err != nil { + return err + } + defer session.EndSession(ctx) + _, err = session.WithTransaction(ctx, func(sc mongo.SessionContext) (any, error) { + if err := s.renewQueryModelLease(sc, l); err != nil { + return nil, err + } + return nil, fn(sc) + }, options.Transaction().SetReadConcern(readconcern.Snapshot()).SetWriteConcern(writeconcern.Majority())) + return err +} + +// A failed stage never changes the committed view. On restart remove only +// future versions and undo future retirements, fenced by the builder lease. +func (s *MongoStore) restoreWorkforceStage(ctx context.Context, l queryModelLease, org string, head int64) error { + c := s.db.Collection(workforceFactCollection) + for _, inserted := range []bool{true, false} { + field := "toEpoch" + if inserted { + field = "fromEpoch" + } + for { + cur, err := c.Find(ctx, bson.M{"organizationId": org, field: bson.M{"$gt": head}}, options.Find().SetProjection(bson.M{"_id": 1}).SetLimit(200)) + if err != nil { + return err + } + var ids []bson.M + err = cur.All(ctx, &ids) + cur.Close(ctx) + if err != nil { + return err + } + if len(ids) == 0 { + break + } + values := bson.A{} + for _, d := range ids { + values = append(values, d["_id"]) + } + err = s.workforceFactTransaction(ctx, l, func(sc mongo.SessionContext) error { + if inserted { + _, err := c.DeleteMany(sc, bson.M{"_id": bson.M{"$in": values}, "fromEpoch": bson.M{"$gt": head}}) + return err + } + _, err := c.UpdateMany(sc, bson.M{"_id": bson.M{"$in": values}, "toEpoch": bson.M{"$gt": head}}, bson.M{"$unset": bson.M{"toEpoch": "", "retiredAt": "", "expireAt": ""}}) + return err + }) + if err != nil { + return err + } + } + } + return nil +} +func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLease, org string, revision int64) error { + c := s.db.Collection(workforceFactCollection) + key := workforceFactKey(org) + var control struct { + Heads map[string]QueryModelHead `bson:"heads"` + } + err := s.queryControl().FindOne(ctx, bson.M{"_id": "global"}, options.FindOne().SetProjection(bson.M{"heads." + key: 1})).Decode(&control) + if err != nil { + return err + } + oldHead := control.Heads[key] + epoch := oldHead.FactEpoch + 1 + if err = s.restoreWorkforceStage(ctx, l, org, oldHead.FactEpoch); err != nil { + return err + } + var d workforcestats.Snapshot + session, err := s.db.Client().StartSession() + if err != nil { + return err + } + defer session.EndSession(ctx) + _, err = session.WithTransaction(ctx, func(sc mongo.SessionContext) (any, error) { + var e error + d, e = s.LoadWorkforceStats(context.WithValue(sc, workforceBuildBudgetKey{}, &workforceBuildBudget{}), org, workforcestats.Query{Mode: domain.RunModeAll}) + return nil, e + }, options.Transaction().SetReadConcern(readconcern.Snapshot())) + if err != nil { + return err + } + facts := workforcestats.Materialize(d) + type previous struct { + ID string `bson:"_id"` + Key string `bson:"key"` + Hash string `bson:"hash"` + } + cur, err := c.Find(ctx, workforceVersionFilter(org, oldHead.FactEpoch), options.Find().SetProjection(bson.M{"_id": 1, "key": 1, "hash": 1})) + if err != nil { + return err + } + old := map[string]previous{} + for cur.Next(ctx) { + var row previous + if err = cur.Decode(&row); err != nil { + break + } + old[row.Key] = row + } + if err == nil { + err = cur.Err() + } + cur.Close(ctx) + if err != nil { + return err + } + models := []mongo.WriteModel{} + changed := false + flush := func() error { + if len(models) == 0 { + return nil + } + batch := models + models = nil + return s.workforceFactTransaction(ctx, l, func(sc mongo.SessionContext) error { + _, err := c.BulkWrite(sc, batch, options.BulkWrite().SetOrdered(true)) + return err + }) + } + for _, fact := range facts { + // Keep comments/text in authority. Source hashes and exact problem identities + // fence page hydration; no client draft or media is stored in derived facts. + raw, err := json.Marshal(fact) + if err != nil { + return err + } + hash := queryserving.Key(workforceFactSchema, string(raw)) + prev, found := old[fact.Key] + delete(old, fact.Key) + if found && prev.Hash == hash { + continue + } + changed = true + if found { + models = append(models, mongo.NewUpdateOneModel().SetFilter(bson.M{"_id": prev.ID}).SetUpdate(bson.M{"$set": bson.M{"toEpoch": epoch, "retiredAt": time.Now().UTC()}})) + } + encoded, err := bson.Marshal(fact) + if err != nil { + return err + } + var doc bson.M + if err = bson.Unmarshal(encoded, &doc); err != nil { + return err + } + at, err := time.Parse(time.RFC3339Nano, fact.CreatedAt) + if err != nil { + return err + } + doc["_id"], doc["organizationId"], doc["fromEpoch"], doc["hash"], doc["at"] = queryserving.Key(workforceFactSchema, org, fact.Key, epoch), org, epoch, hash, at + doc["atSeconds"], doc["atNanos"] = at.Unix(), at.Nanosecond() + doc["metric"] = workforcestats.MetricCounterValues(fact.Metric) + if fact.Detail != nil { + raw, e := json.Marshal(fact.Detail) + if e != nil { + return e + } + var detail bson.M + if e = bson.UnmarshalExtJSON(raw, false, &detail); e != nil { + return e + } + doc["detail"] = detail + doc["return"] = bson.M{"id": fact.Detail.ID, "stage": fact.Detail.Stage, "taskIds": fact.Detail.TaskIDs, "metrics": detail["metrics"]} + } + doc["order"] = "" + if fact.Detail != nil { + doc["order"] = fact.Detail.ID + fact.Detail.UserID + fact.Detail.SupplierID + } + models = append(models, mongo.NewInsertOneModel().SetDocument(doc)) + if len(models) >= 200 { + if err = flush(); err != nil { + return err + } + } + } + for _, prev := range old { + changed = true + models = append(models, mongo.NewUpdateOneModel().SetFilter(bson.M{"_id": prev.ID}).SetUpdate(bson.M{"$set": bson.M{"toEpoch": epoch, "retiredAt": time.Now().UTC()}})) + if len(models) >= 200 { + if err = flush(); err != nil { + return err + } + } + } + if err = flush(); err != nil { + return err + } + if !changed && oldHead.FactEpoch > 0 { + epoch = oldHead.FactEpoch + } + head := QueryModelHead{Generation: workforceFactSchema, Collection: workforceFactCollection, FactEpoch: epoch, Rows: len(facts), SourceRevision: revision, AsOf: time.Now().UTC()} + if err = s.publishQueryModelHead(ctx, l, key, head); err != nil { + return err + } + // Garbage collection never expires a partially staged retirement. It + // deletes only committed versions after a long in-flight reader grace. + cleanup, stop := context.WithTimeout(ctx, 2*time.Second) + defer stop() + if e := s.cleanWorkforceVersions(cleanup, l, org, epoch); e != nil { + log.Print("workforce_fact_cleanup result=retry") + } + return nil +} +func (s *MongoStore) cleanWorkforceVersions(ctx context.Context, l queryModelLease, org string, epoch int64) error { + c := s.db.Collection(workforceFactCollection) + cur, err := c.Find(ctx, bson.M{"organizationId": org, "toEpoch": bson.M{"$lte": epoch}, "retiredAt": bson.M{"$lt": time.Now().UTC().Add(-15 * time.Minute)}}, options.Find().SetProjection(bson.M{"_id": 1}).SetLimit(200)) + if err != nil { + return err + } + var docs []bson.M + err = cur.All(ctx, &docs) + cur.Close(ctx) + if err != nil || len(docs) == 0 { + return err + } + ids := bson.A{} + for _, d := range docs { + ids = append(ids, d["_id"]) + } + return s.workforceFactTransaction(ctx, l, func(sc mongo.SessionContext) error { + _, err := c.DeleteMany(sc, bson.M{"_id": bson.M{"$in": ids}, "toEpoch": bson.M{"$lte": epoch}}) + return err + }) +} + +func (s *MongoStore) maintainWorkforceVersions(ctx context.Context, l queryModelLease) error { + var control struct { + Heads map[string]QueryModelHead `bson:"heads"` + } + if err := s.queryControl().FindOne(ctx, bson.M{"_id": "global"}).Decode(&control); err != nil { + return err + } + cur, err := s.organizations.Find(ctx, bson.M{}, options.Find().SetProjection(bson.M{"id": 1})) + if err != nil { + return err + } + defer cur.Close(ctx) + for cur.Next(ctx) { + var row struct { + ID string `bson:"id"` + } + if err = cur.Decode(&row); err != nil { + return err + } + if h, ok := control.Heads[workforceFactKey(row.ID)]; ok && h.FactEpoch > 0 { + if err = s.cleanWorkforceVersions(ctx, l, row.ID, h.FactEpoch); err != nil { + return err + } + } + } + return cur.Err() +} + +// BSON dates are millisecond precision. The indexed date predicate is broad; +// these two integer fields retain exact historical RFC3339Nano boundaries. +func workforceTimeBound(seconds, nanos any, bound time.Time, lower bool) bson.M { + major, minor := "$lt", "$lt" + if lower { + major, minor = "$gt", "$gte" + } + return bson.M{"$or": bson.A{bson.M{major: bson.A{seconds, bound.Unix()}}, bson.M{"$and": bson.A{bson.M{"$eq": bson.A{seconds, bound.Unix()}}, bson.M{minor: bson.A{nanos, bound.Nanosecond()}}}}}} +} +func workforceFactFilter(org string, h QueryModelHead, q workforcestats.Query) bson.M { + f := workforceVersionFilter(org, h.FactEpoch) + f["mode"] = workforcestats.ModeVariant(q.Mode) + f["groupTemplate"] = q.GroupTemplate + if !q.From.IsZero() || !q.To.IsZero() { + at := bson.M{} + if !q.From.IsZero() { + at["$gte"] = q.From.Truncate(time.Millisecond) + } + if !q.To.IsZero() { + at["$lt"] = q.To.Add(time.Millisecond).Truncate(time.Millisecond) + } + f["at"] = at + exact := bson.A{} + if !q.From.IsZero() { + exact = append(exact, workforceTimeBound("$atSeconds", "$atNanos", q.From, true)) + } + if !q.To.IsZero() { + exact = append(exact, workforceTimeBound("$atSeconds", "$atNanos", q.To, false)) + } + f["$expr"] = bson.M{"$and": exact} + } + if q.ProjectID != "" { + f["projectId"] = q.ProjectID + } + if q.SupplierID != "" { + f["supplierId"] = q.SupplierID + } + if q.MatchUser || q.UserID != "" { + f["userId"] = q.UserID + } + if q.Role != "" { + f["role"] = q.Role + } + if q.TemplateVersion != nil { + f["templateVersion"] = *q.TemplateVersion + } + clauses := bson.A{f} + for field, allowed := range map[string]map[string]bool{"projectId": q.AllowedProjects, "supplierId": q.AllowedSuppliers} { + if allowed != nil { + ids := []string{} + for id, on := range allowed { + if on { + ids = append(ids, id) + } + } + clauses = append(clauses, bson.M{field: bson.M{"$in": ids}}) + } + } + return bson.M{"$and": clauses} +} + +// Legacy formal reviews used scope-filtered event/suppressor context in the +// original loader. Resolve those small descriptors BEFORE supplier visibility. +func workforceFactReadStages(org string, h QueryModelHead, q workforcestats.Query) mongo.Pipeline { + base := q + base.SupplierID = "" + base.AllowedSuppliers = nil + conditions := func(variable string, actor bool) bson.A { + out := bson.A{bson.M{"$literal": true}} + if q.ProjectID != "" { + out = append(out, bson.M{"$eq": bson.A{variable + ".projectId", q.ProjectID}}) + } + if q.AllowedProjects != nil { + ids := []string{} + for id, on := range q.AllowedProjects { + if on { + ids = append(ids, id) + } + } + out = append(out, bson.M{"$in": bson.A{variable + ".projectId", ids}}) + } + if actor { + if q.MatchUser || q.UserID != "" { + out = append(out, bson.M{"$eq": bson.A{variable + ".userId", q.UserID}}) + } + if q.SupplierID != "" { + out = append(out, bson.M{"$eq": bson.A{variable + ".supplierId", q.SupplierID}}) + } + } else { + if q.SupplierID != "" { + out = append(out, bson.M{"$in": bson.A{q.SupplierID, variable + ".suppliers"}}) + } + if !q.From.IsZero() { + out = append(out, workforceTimeBound(variable+".seconds", variable+".nanos", q.From, true)) + } + if !q.To.IsZero() { + out = append(out, workforceTimeBound(variable+".seconds", variable+".nanos", q.To, false)) + } + } + return out + } + events := bson.M{"$filter": bson.M{"input": bson.M{"$ifNull": bson.A{"$legacy.events", bson.A{}}}, "as": "e", "cond": bson.M{"$and": conditions("$$e", true)}}} + suppressed := bson.M{"$filter": bson.M{"input": bson.M{"$ifNull": bson.A{"$legacy.suppressors", bson.A{}}}, "as": "op", "cond": bson.M{"$and": conditions("$$op", false)}}} + legacy := bson.M{"$eq": bson.A{"$sourceKind", "review"}} + chosen := func(field string) any { + return bson.M{"$cond": bson.A{legacy, bson.M{"$ifNull": bson.A{"$legacyEvent." + field, "$legacy." + field}}, "$" + field}} + } + first := workforceFactFilter(org, h, base) + if q.Details { + first = bson.M{"$and": bson.A{first, bson.M{"kind": "return"}}} + } + stages := mongo.Pipeline{{{Key: "$match", Value: first}}, {{Key: "$set", Value: bson.M{"legacyEvent": bson.M{"$first": events}, "legacyHidden": bson.M{"$gt": bson.A{bson.M{"$size": suppressed}, 0}}}}}, {{Key: "$set", Value: bson.M{"supplierId": chosen("supplierId"), "supplierName": chosen("supplierName")}}}, {{Key: "$set", Value: bson.M{"effectiveUnknown": bson.M{"$cond": bson.A{legacy, bson.M{"$or": bson.A{bson.M{"$eq": bson.A{"$userId", ""}}, bson.M{"$eq": bson.A{"$supplierId", ""}}}}, false}}, "order": bson.M{"$concat": bson.A{"$return.id", "$userId", "$supplierId"}}}}}} + + match := bson.M{"legacyHidden": false} + parts := bson.A{match} + if q.SupplierID != "" { + parts = append(parts, bson.M{"supplierId": q.SupplierID}) + } + if q.AllowedSuppliers != nil { + ids := []string{} + for id, on := range q.AllowedSuppliers { + if on { + ids = append(ids, id) + } + } + parts = append(parts, bson.M{"supplierId": bson.M{"$in": ids}}) + } + return append(stages, bson.D{{Key: "$match", Value: bson.M{"$and": parts}}}) +} + +func workforceMetricPipeline(q workforcestats.Query, totals bool) mongo.Pipeline { + id := any(nil) + if !totals { + template := any(bson.M{"$literal": 0}) + if q.GroupTemplate { + template = "$templateVersion" + } + day := any(bson.M{"$literal": ""}) + if q.GroupBy == "day" { + day = "$day" + } + id = bson.M{"userId": "$userId", "supplierId": "$supplierId", "templateVersion": template, "date": day} + } + group := bson.M{"_id": id, "eventCount": bson.M{"$sum": "$eventCount"}} + project := bson.M{"_id": 0, "eventCount": 1} + for _, name := range workforcestats.MetricCounterNames() { + group[name] = bson.M{"$sum": "$metric." + name} + project[name] = 1 + } + if !totals { + for _, name := range []string{"userId", "supplierId", "templateVersion", "date"} { + project[name] = "$_id." + name + } + for _, name := range []string{"userName", "supplierName", "role"} { + group[name] = bson.M{"$first": "$" + name} + project[name] = 1 + } + } + for _, stage := range []string{"quality", "acceptance"} { + when := bson.M{"$and": bson.A{bson.M{"$eq": bson.A{"$kind", "return"}}, bson.M{"$eq": bson.A{"$return.stage", stage}}}} + choose := func(yes, no any) any { return bson.M{"$cond": bson.A{when, yes, no}} } + group[stage+"Decisions"] = bson.M{"$addToSet": choose("$return.id", nil)} + group[stage+"Tasks"] = bson.M{"$addToSet": choose("$return.taskIds", bson.A{})} + group[stage+"Unknown"] = bson.M{"$addToSet": choose(bson.M{"$cond": bson.A{bson.M{"$gt": bson.A{"$return.metrics.unknownDetailCount", 0}}, "$return.id", nil}}, nil)} + out := bson.M{"decisionCount": bson.M{"$size": bson.M{"$setDifference": bson.A{"$" + stage + "Decisions", bson.A{nil}}}}, "unknownDetailCount": bson.M{"$size": bson.M{"$setDifference": bson.A{"$" + stage + "Unknown", bson.A{nil}}}}, "taskCount": bson.M{"$size": bson.M{"$reduce": bson.M{"input": "$" + stage + "Tasks", "initialValue": bson.A{}, "in": bson.M{"$setUnion": bson.A{"$$value", "$$this"}}}}}} + for _, name := range []string{"problemCount", "sliceCount", "videoSeconds", "unlocatedCount"} { + sumKey := stage + "_" + name + group[sumKey] = bson.M{"$sum": choose("$return.metrics."+name, 0)} + out[name] = "$" + sumKey + } + project[stage+"Returns"] = out + } + return mongo.Pipeline{{{Key: "$group", Value: group}}, {{Key: "$project", Value: project}}} +} +func (s *MongoStore) loadWorkforceFactReport(ctx context.Context, org string, q workforcestats.Query) (workforcestats.Report, bool, error) { + // Explicit empty-user queries distinguish Mongo missing/null/empty values. + // Keep their exact legacy loader until provenance is separately indexed. + if q.MatchUser && q.UserID == "" { + return workforcestats.Report{}, false, nil + } + h, exists, err := s.workforceHead(ctx, org) + if err != nil || !exists { + return workforcestats.Report{}, exists, err + } + key := queryserving.Key(workforceFactSchema, s.db.Name(), org, h.FactEpoch, q) + if raw, ok := workforceReportCache.get(key, dashboardStatsCacheTTL); ok { + var report workforcestats.Report + if json.Unmarshal(raw, &report) == nil { + return report, true, nil + } + } + if cache := s.SharedQueryCache(); cache != nil { + if raw, ok := cache.Get(ctx, key); ok && len(raw) <= workforceReportMaxBytes { + var report workforcestats.Report + if json.Unmarshal(raw, &report) == nil { + workforceReportCache.set(key, raw, dashboardStatsCacheTTL) + return report, true, nil + } + } + } + pipeline := append(workforceFactReadStages(org, h, q), bson.D{{Key: "$project", Value: bson.M{"kind": 1, "metric": 1, "return": 1, "eventCount": 1, "userId": 1, "userName": 1, "supplierId": 1, "supplierName": 1, "role": 1, "templateVersion": 1, "day": 1}}}, bson.D{{Key: "$facet", Value: bson.M{"items": workforceMetricPipeline(q, false), "totals": workforceMetricPipeline(q, true)}}}) + cur, err := s.db.Collection(workforceFactCollection).Aggregate(ctx, pipeline, options.Aggregate().SetMaxTime(8*time.Second)) + if err != nil { + return workforcestats.Report{}, true, err + } + defer cur.Close(ctx) + var rows []struct { + Items []bson.M `bson:"items"` + Totals []bson.M `bson:"totals"` + } + if err = cur.All(ctx, &rows); err != nil { + return workforcestats.Report{}, true, err + } + report := workforcestats.Report{Items: []domain.DashboardWorkforceMetric{}} + if len(rows) > 0 { + for _, m := range rows[0].Items { + var v domain.DashboardWorkforceMetric + raw, _ := json.Marshal(m) + if err = json.Unmarshal(raw, &v); err != nil { + return report, true, err + } + report.Items = append(report.Items, v) + } + if len(rows[0].Totals) > 0 { + raw, _ := json.Marshal(rows[0].Totals[0]) + if err = json.Unmarshal(raw, &report.Totals); err != nil { + return report, true, err + } + report.EventCount = int(supplierNumber(rows[0].Totals[0]["eventCount"])) + } + } + sort.Slice(report.Items, func(i, j int) bool { + a, b := report.Items[i], report.Items[j] + return fmt.Sprintf("%s/%s/%s/%s/%d", a.Date, a.SupplierName, a.UserName, a.UserID, a.TemplateVersion) < fmt.Sprintf("%s/%s/%s/%s/%d", b.Date, b.SupplierName, b.UserName, b.UserID, b.TemplateVersion) + }) + if raw, err := json.Marshal(report); err == nil && len(raw) <= workforceReportMaxBytes { + workforceReportCache.set(key, raw, dashboardStatsCacheTTL) + if cache := s.SharedQueryCache(); cache != nil { + cache.Put(ctx, key, raw, dashboardStatsCacheTTL) + } + } + return report, true, nil +} +func (s *MongoStore) loadWorkforceFactPage(ctx context.Context, org string, q workforcestats.Query, limit, offset int) ([]workforcePageFact, int, bool, error) { + if q.MatchUser && q.UserID == "" { + return nil, 0, false, nil + } + h, exists, err := s.workforceHead(ctx, org) + if err != nil || !exists { + return nil, 0, exists, err + } + selected := bson.M{"_id": 1, "atSeconds": 1, "atNanos": 1, "order": 1, "userId": 1, "supplierId": 1, "sourceKind": 1, "effectiveUnknown": 1, "returnId": "$return.id"} + pipeline := append(workforceFactReadStages(org, h, q), bson.D{{Key: "$match", Value: bson.M{"kind": "return"}}}, bson.D{{Key: "$project", Value: selected}}, bson.D{{Key: "$facet", Value: bson.M{"total": bson.A{bson.M{"$count": "count"}}, "page": bson.A{bson.M{"$sort": bson.D{{Key: "atSeconds", Value: -1}, {Key: "atNanos", Value: -1}, {Key: "order", Value: 1}, {Key: "returnId", Value: 1}, {Key: "userId", Value: 1}, {Key: "supplierId", Value: 1}, {Key: "sourceKind", Value: 1}}}, bson.M{"$skip": offset}, bson.M{"$limit": limit}}}}}) + cur, err := s.db.Collection(workforceFactCollection).Aggregate(ctx, pipeline, options.Aggregate().SetMaxTime(8*time.Second)) + if err != nil { + return nil, 0, true, err + } + defer cur.Close(ctx) + type selection struct { + ID string `bson:"_id"` + Supplier string `bson:"supplierId"` + Unknown bool `bson:"effectiveUnknown"` + Kind string `bson:"sourceKind"` + } + var rows []struct { + Page []selection `bson:"page"` + Total []struct { + Count int `bson:"count"` + } `bson:"total"` + } + if err = cur.All(ctx, &rows); err != nil { + return nil, 0, true, err + } + out := []workforcePageFact{} + count := 0 + if len(rows) == 0 { + return out, count, true, nil + } + if len(rows[0].Total) > 0 { + count = rows[0].Total[0].Count + } + if len(rows[0].Page) == 0 { + return out, count, true, nil + } + ids := []string{} + for _, row := range rows[0].Page { + ids = append(ids, row.ID) + } + detailCursor, err := s.db.Collection(workforceFactCollection).Find(ctx, bson.M{"_id": bson.M{"$in": ids}}, options.Find().SetProjection(bson.M{"_id": 1, "detail": 1, "sourceKind": 1, "sourceHash": 1})) + if err != nil { + return nil, 0, true, err + } + defer detailCursor.Close(ctx) + var details []struct { + ID string `bson:"_id"` + Data workforcePageFact `bson:",inline"` + } + if err = detailCursor.All(ctx, &details); err != nil { + return nil, 0, true, err + } + byID := map[string]workforcePageFact{} + for _, row := range details { + byID[row.ID] = row.Data + } + for _, row := range rows[0].Page { + detail, ok := byID[row.ID] + if !ok { + return nil, 0, true, ErrQueryReplicaNotReady + } + if row.Kind == "review" { + detail.Detail.SupplierID = row.Supplier + detail.Detail.AttributionUnknown = row.Unknown + } + out = append(out, detail) + } + return out, count, true, nil +} diff --git a/internal/store/workforce_facts_local_test.go b/internal/store/workforce_facts_local_test.go new file mode 100644 index 00000000..3081766f --- /dev/null +++ b/internal/store/workforce_facts_local_test.go @@ -0,0 +1,310 @@ +package store + +import ( + "context" + "droboticflow/internal/domain" + "droboticflow/internal/workforcestats" + "encoding/json" + "fmt" + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/event" + "os" + "reflect" + "strings" + "sync" + "testing" + "time" +) + +func TestWorkforceMaterializedFactsAndBoundedDetailsLocal(t *testing.T) { + if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" { + t.Skip("owned synthetic replica required") + } + var mu sync.Mutex + historyReads := 0 + factWrites := 0 + var textIDs []string + sourceMonitor := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) { + if e.CommandName != "find" { + return + } + name, ok := e.Command.Lookup("find").StringValueOK() + if !ok || name != "reviewOperations" { + return + } + var filter bson.M + _ = bson.Unmarshal(e.Command.Lookup("filter").Document(), &filter) + var walk func(bson.M) + walk = func(doc bson.M) { + if in := deletionDoc(doc["_id"])["$in"]; in != nil { + mu.Lock() + for _, id := range in.(bson.A) { + if text, ok := id.(string); ok { + textIDs = append(textIDs, text) + } + } + mu.Unlock() + } + if parts, ok := doc["$and"].(bson.A); ok { + for _, part := range parts { + walk(deletionDoc(part)) + } + } + } + walk(filter) + }} + targetMonitor := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) { + if e.CommandName == "insert" || e.CommandName == "update" || e.CommandName == "delete" { + if v, ok := e.Command.Lookup(e.CommandName).StringValueOK(); ok && v == workforceFactCollection { + mu.Lock() + factWrites++ + mu.Unlock() + } + } + if e.CommandName != "find" { + return + } + name, ok := e.Command.Lookup("find").StringValueOK() + if ok && (name == "reviewOperations" || name == "annotationWorklogEvents" || name == "annotationTasks" || name == "annotationReviews" || name == "sliceRevisions" || name == "annotationRevisions") { + mu.Lock() + historyReads++ + mu.Unlock() + } + }} + source, target, replica := newLocalReplica(t, sourceMonitor, targetMonitor) + ctx := t.Context() + seedWorkforceStats(t, source) + original, err := source.GetReviewOperation(ctx, "review-operation-test") + if err != nil { + t.Fatal(err) + } + for i := 0; i < 52; i++ { + op := *original + op.ID = fmt.Sprintf("review-%03d", i) + op.ReviewRound = i + 2 + op.CreatedAt = time.Date(2026, 9, 28, 13, 0, i, 0, time.UTC).Format(time.RFC3339) + op.CompletedAt = op.CreatedAt + ok, err := source.CompareAndSwapReviewOperation(ctx, op, 0) + if err != nil || !ok { + t.Fatal(ok, err) + } + } + if err = source.UpsertAnnotationReview(ctx, domain.AnnotationReviewRecord{ID: "legacy-review", OrganizationID: "workforce-test", TaskID: "task", Revision: 4, ReviewRound: 1, Stage: domain.AnnotationReviewStageQuality, Decision: domain.AnnotationReviewRejected, CreatedAt: "2026-09-28T14:00:00Z", Comment: "legacy text survives", ProblemSliceIDs: []string{"old-slice"}}); err != nil { + t.Fatal(err) + } + all, err := source.LoadWorkforceStats(ctx, "workforce-test", workforcestats.Query{Details: true, Mode: domain.RunModeAll}) + if err != nil { + t.Fatal(err) + } + finishLocalReplica(t, replica) + lease, ok, err := target.acquireQueryModelLease(ctx) + if err != nil || !ok { + t.Fatal(ok, err) + } + if err = target.rebuildWorkforceFacts(ctx, lease, "workforce-test", 0); err != nil { + t.Fatal(err) + } + if err = source.ConfigureQueryReplicaReports(target, time.Minute); err != nil { + t.Fatal(err) + } + base := workforcestats.Query{UserID: "u", MatchUser: true, Details: true, From: time.Date(2026, 9, 28, 0, 0, 0, 0, time.UTC), To: time.Date(2026, 9, 29, 0, 0, 0, 0, time.UTC)} + queries := []workforcestats.Query{base, base, base, base, base, base, base, base} + queries[1].GroupBy = "day" + queries[2].GroupTemplate = true + queries[3].AllowedProjects = map[string]bool{"no": true} + queries[4].AllowedSuppliers = map[string]bool{"no": true} + queries[5].Role = string(domain.UserRoleQualityReviewer) + queries[6].From = base.From.Add(12*time.Hour + time.Nanosecond) + queries[7].To = base.From.Add(12*time.Hour + time.Nanosecond) + mu.Lock() + historyReads = 0 + mu.Unlock() + for _, q := range queries { + want := workforcestats.Build(all, q) + q.Details = false + got, err := source.LoadWorkforceReportCached(ctx, "workforce-test", q) + want.Details = nil + if err != nil || !reflect.DeepEqual(got, want) { + t.Fatalf("summary mismatch q=%+v: %v\ngot=%+v\nwant=%+v", q, err, got, want) + } + } + mu.Lock() + reads := historyReads + mu.Unlock() + if reads != 0 { + t.Fatalf("summary reloaded %d raw histories", reads) + } + for _, offset := range []int{0, 20, 40, 100} { + q := base + want := workforcestats.Build(all, q).Details + start, end := workforcePageBounds(len(want), 20, offset) + mu.Lock() + textIDs = nil + mu.Unlock() + got, total, err := source.LoadWorkforceReturnPage(ctx, "workforce-test", q, 20, offset) + if err != nil || total != len(want) || !reflect.DeepEqual(got, want[start:end]) { + t.Fatalf("page offset=%d total=%d: %v\ngot=%+v\nwant=%+v", offset, total, err, got, want[start:end]) + } + mu.Lock() + n := len(textIDs) + mu.Unlock() + expectedIDs := 0 + for _, v := range want[start:end] { + if v.ID != "legacy-review" { + expectedIDs++ + } + } + if n != expectedIDs || n > 20 { + t.Fatalf("hydrated %d operation texts for 20 rows", n) + } + } + mu.Lock() + factWrites = 0 + mu.Unlock() + if err = target.rebuildWorkforceFacts(ctx, lease, "workforce-test", 0); err != nil { + t.Fatal(err) + } + mu.Lock() + writes := factWrites + mu.Unlock() + if writes != 0 { + t.Fatalf("unchanged facts caused %d data writes", writes) + } + // Completed records changing after publication may not lend new attribution + // or text to an old fact. The authority annotation itself is never rewritten. + if _, err = source.db.Collection("reviewOperations").UpdateOne(ctx, bson.M{"_id": "review-051"}, bson.M{"$set": bson.M{"effects.1.beforeTask.supplierId": "changed"}}); err != nil { + t.Fatal(err) + } + if _, _, err = source.LoadWorkforceReturnPage(ctx, "workforce-test", base, 20, 0); err == nil { + t.Fatal("changed formal identity was hydrated into old facts") + } + // Simulate a crashed partial stage: old head stays readable and restart + // removes only future versions, then republishes equal current facts. + var head struct { + Heads map[string]QueryModelHead `bson:"heads"` + } + if err = target.queryControl().FindOne(ctx, bson.M{"_id": "global"}).Decode(&head); err != nil { + t.Fatal(err) + } + h := head.Heads[workforceFactKey("workforce-test")] + c := target.db.Collection(workforceFactCollection) + var active bson.M + if err = c.FindOne(ctx, workforceVersionFilter("workforce-test", h.FactEpoch)).Decode(&active); err != nil { + t.Fatal(err) + } + future := bson.M{"_id": "crashed-future", "organizationId": "workforce-test", "fromEpoch": h.FactEpoch + 1, "key": "never published"} + if _, err = c.InsertOne(ctx, future); err != nil { + t.Fatal(err) + } + if _, err = c.UpdateOne(ctx, bson.M{"_id": active["_id"]}, bson.M{"$set": bson.M{"toEpoch": h.FactEpoch + 1}}); err != nil { + t.Fatal(err) + } + if err = target.rebuildWorkforceFacts(ctx, lease, "workforce-test", 0); err != nil { + t.Fatal(err) + } + if err = c.FindOne(ctx, bson.M{"_id": "crashed-future"}).Err(); err == nil { + t.Fatal("future stage survived recovery") + } + // Derived rows never contain customer comment text. + cur, err := c.Find(ctx, bson.M{}) + if err != nil { + t.Fatal(err) + } + defer cur.Close(ctx) + for cur.Next(ctx) { + var doc bson.M + if err = cur.Decode(&doc); err != nil { + t.Fatal(err) + } + raw, _ := json.Marshal(doc) + if strings.Contains(string(raw), "保留意见") || strings.Contains(string(raw), "legacy text survives") { + t.Fatal("customer text copied into facts") + } + } +} + +func TestWorkforceLegacyScopeVariantsStayEquivalentLocal(t *testing.T) { + if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" { + t.Skip("owned synthetic replica required") + } + source, target, replica := newLocalReplica(t) + ctx := t.Context() + seedWorkforceStats(t, source) + when := "2026-09-28T14:00:00Z" + if err := source.UpsertAnnotationReview(ctx, domain.AnnotationReviewRecord{ID: "legacy-dependent", OrganizationID: "workforce-test", TaskID: "task", Revision: 4, Stage: domain.AnnotationReviewStageQuality, Decision: domain.AnnotationReviewRejected, CreatedAt: when, Comment: "old opinion"}); err != nil { + t.Fatal(err) + } + if err := source.UpsertAnnotationWorklogEvent(ctx, domain.AnnotationWorklogEventRecord{ID: "legacy-supplier-event", OrganizationID: "workforce-test", TaskID: "task", ProjectID: "p", UserID: "reviewer", SupplierID: "reviewer-v", EventType: "quality_rejected", Revision: 4, CreatedAt: when}); err != nil { + t.Fatal(err) + } + base, err := source.GetReviewOperation(ctx, "review-operation-test") + if err != nil { + t.Fatal(err) + } + op := *base + op.ID = "formal-scope-suppressor" + op.CreatedAt, op.CompletedAt = "2026-09-28T13:00:00Z", "2026-09-28T13:00:00Z" + op.Effects = []domain.ReviewDecisionEffect{{Review: &domain.AnnotationReviewRecord{ID: "legacy-dependent", TaskID: "task", Revision: 4}}, {BeforeTask: &domain.AnnotationTaskRecord{ID: "task", SupplierID: "foreign-v"}}, {BeforeItem: &domain.HumanItemRecord{ID: "item", LegacyTaskID: "task", SupplierID: "foreign-v", LatestAuthorUserID: "u"}}} + if ok, err := source.CompareAndSwapReviewOperation(ctx, op, 0); err != nil || !ok { + t.Fatal(ok, err) + } + queries := []workforcestats.Query{{Details: true}, {Details: true, MatchUser: true, UserID: "u"}, {Details: true, SupplierID: "v"}, {Details: true, SupplierID: "foreign-v"}, {Details: true, SupplierID: "reviewer-v"}, {Details: true, AllowedProjects: map[string]bool{"other": true}}, {Details: true, GroupTemplate: true, MatchUser: true, UserID: "u"}} + queries = append(queries, workforcestats.Query{Details: true, From: time.Date(2026, 9, 28, 13, 0, 0, 1, time.UTC)}) + wants := []workforcestats.Report{} + for _, q := range queries { + d, e := source.LoadWorkforceStats(ctx, "workforce-test", q) + if e != nil { + t.Fatal(e) + } + wants = append(wants, workforcestats.Build(d, q)) + } + finishLocalReplica(t, replica) + l, ok, err := target.acquireQueryModelLease(ctx) + if err != nil || !ok { + t.Fatal(ok, err) + } + if err = target.rebuildWorkforceFacts(ctx, l, "workforce-test", 0); err != nil { + t.Fatal(err) + } + if err = source.ConfigureQueryReplicaReports(target, time.Minute); err != nil { + t.Fatal(err) + } + for i, q := range queries { + want := wants[i] + details := want.Details + want.Details = nil + q.Details = false + got, e := source.LoadWorkforceReportCached(ctx, "workforce-test", q) + if e != nil || !reflect.DeepEqual(want, got) { + t.Fatalf("legacy scope %d: %v\ngot=%+v\nwant=%+v", i, e, got, want) + } + q.Details = true + page, total, e := source.LoadWorkforceReturnPage(ctx, "workforce-test", q, 50, 0) + if e != nil || total != len(details) || !reflect.DeepEqual(page, details) { + t.Fatalf("legacy page scope %d total=%d: %v\ngot=%+v\nwant=%+v", i, total, e, page, details) + } + } +} + +func TestWorkforceFactsSurviveConfiguredScopePruningLocal(t *testing.T) { + if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" { + t.Skip("owned synthetic replica required") + } + _, target, _ := newLocalReplica(t) + ctx := t.Context() + l, ok, err := target.acquireQueryModelLease(ctx) + if err != nil || !ok { + t.Fatal(ok, err) + } + head := QueryModelHead{Generation: workforceFactSchema, Collection: workforceFactCollection, FactEpoch: 1, AsOf: time.Now().UTC()} + if err = target.publishQueryModelHead(ctx, l, workforceFactKey("o"), head); err != nil { + t.Fatal(err) + } + if err = target.pruneQueryModelScopes(ctx, l, map[string][]string{"o": {"p"}}); err != nil { + t.Fatal(err) + } + _, exists, err := target.workforceHead(ctx, "o") + if err != nil || !exists { + t.Fatalf("scope maintenance deleted published workforce facts: %v/%v", exists, err) + } +} diff --git a/internal/store/workforce_report_cache.go b/internal/store/workforce_report_cache.go index e16be821..6af654b1 100644 --- a/internal/store/workforce_report_cache.go +++ b/internal/store/workforce_report_cache.go @@ -34,6 +34,17 @@ func (s *MongoStore) LoadWorkforceReportCached(ctx context.Context, org string, if err != nil { return workforcestats.Report{}, err } + if target != nil { + report, used, e := target.loadWorkforceFactReport(ctx, org, q) + if e != nil || used { + return report, e + } + } else if s.queryReplicaFactStore { + report, used, e := s.loadWorkforceFactReport(ctx, org, q) + if e != nil || used { + return report, e + } + } epoch := int64(0) identity := fmt.Sprintf("%p:%s", s.db.Client(), s.db.Name()) if target != nil { @@ -91,3 +102,23 @@ func (s *MongoStore) LoadWorkforceReportCached(ctx context.Context, org string, } return report, nil } + +// LoadWorkforceReport preserves uncached legacy summary behavior when read +// replicas are disabled; the enabled replacement reads only published facts. +func (s *MongoStore) LoadWorkforceReport(ctx context.Context, org string, q workforcestats.Query) (workforcestats.Report, error) { + target, err := s.reportReplica(ctx) + if err != nil { + return workforcestats.Report{}, err + } + if target != nil { + report, used, e := target.loadWorkforceFactReport(ctx, org, q) + if e != nil || used { + return report, e + } + } + d, err := s.LoadWorkforceStats(ctx, org, q) + if err != nil { + return workforcestats.Report{}, err + } + return workforcestats.Build(d, q), nil +} diff --git a/internal/store/workforce_return_page.go b/internal/store/workforce_return_page.go new file mode 100644 index 00000000..1a969bbf --- /dev/null +++ b/internal/store/workforce_return_page.go @@ -0,0 +1,161 @@ +package store + +import ( + "context" + "droboticflow/internal/domain" + "droboticflow/internal/workforcestats" + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/mongo/options" + "reflect" +) + +type workforcePageFact struct { + Detail domain.WorkforceReturnDetail `bson:"detail"` + SourceKind string `bson:"sourceKind"` + SourceHash string `bson:"sourceHash"` +} + +func workforcePageBounds(count, limit, offset int) (int, int) { + if offset > count { + offset = count + } + end := offset + limit + if end < offset || end > count { + end = count + } + return offset, end +} +func (s *MongoStore) LoadWorkforceReturnPage(ctx context.Context, org string, q workforcestats.Query, limit, offset int) ([]domain.WorkforceReturnDetail, int, error) { + q.Details = true + target, err := s.reportReplica(ctx) + if err != nil { + return nil, 0, err + } + if target == nil { + d, err := s.LoadWorkforceStats(ctx, org, q) + if err != nil { + return nil, 0, err + } + q.Details = true + all := workforcestats.Build(d, q).Details + start, end := workforcePageBounds(len(all), limit, offset) + return all[start:end], len(all), nil + } + page, total, used, e := target.loadWorkforceFactPage(ctx, org, q, limit, offset) + if e != nil { + return nil, 0, e + } + if used { + items, e := s.hydrateWorkforcePage(ctx, org, page) + return items, total, e + } + // Compatibility before facts are first published: calculate only metadata on + // the independent replica; hydrate authority text AFTER selecting this page. + metadata := q + metadata.Details = false + d, err := target.LoadWorkforceStats(ctx, org, metadata) + if err != nil { + return nil, 0, err + } + q.Details = true + all := workforcestats.ReturnFacts(d, q) + start, end := workforcePageBounds(len(all), limit, offset) + page = make([]workforcePageFact, 0, end-start) + for _, fact := range all[start:end] { + page = append(page, workforcePageFact{Detail: *fact.Detail, SourceKind: fact.SourceKind, SourceHash: fact.SourceHash}) + } + + items, err := s.hydrateWorkforcePage(ctx, org, page) + return items, len(all), err +} +func (s *MongoStore) hydrateWorkforcePage(ctx context.Context, org string, page []workforcePageFact) ([]domain.WorkforceReturnDetail, error) { + out := make([]domain.WorkforceReturnDetail, 0, len(page)) + opIDs, reviewIDs := []string{}, []string{} + for _, v := range page { + if v.SourceKind == "operation" { + opIDs = append(opIDs, v.Detail.ID) + } else { + reviewIDs = append(reviewIDs, v.Detail.ID) + } + } + ops := map[string]domain.ReviewOperation{} + reviews := map[string]domain.AnnotationReviewRecord{} + if len(opIDs) > 0 { + cur, err := s.db.Collection("reviewOperations").Find(ctx, organizationScopedFilter(org, bson.M{"_id": bson.M{"$in": opIDs}, "status": "completed", "decision": "rejected"}), options.Find().SetProjection(workforceOperationProjection(true))) + if err != nil { + return nil, err + } + var rows []domain.ReviewOperation + err = cur.All(ctx, &rows) + cur.Close(ctx) + if err != nil { + return nil, err + } + for _, v := range rows { + ops[v.ID] = v + } + } + if len(reviewIDs) > 0 { + cur, err := s.annotationReviews.Find(ctx, organizationScopedFilter(org, bson.M{"id": bson.M{"$in": reviewIDs}, "decision": "rejected"}), options.Find().SetProjection(workforceFields("id", "taskId", "revision", "reviewRound", "stage", "decision", "reviewerUserId", "createdAt", "problemSliceIds", "comment"))) + if err != nil { + return nil, err + } + var rows []domain.AnnotationReviewRecord + err = cur.All(ctx, &rows) + cur.Close(ctx) + if err != nil { + return nil, err + } + for _, v := range rows { + reviews[v.ID] = v + } + } + for _, v := range page { + detail := v.Detail + if v.SourceKind == "operation" { + op, ok := ops[detail.ID] + if !ok || workforcestats.OperationFingerprint(op) != v.SourceHash { + return nil, ErrQueryReplicaNotReady + } + next := 0 + for i, p := range detail.Problems { + found := false + for next < len(op.Draft.Problems) { + candidate := op.Draft.Problems[next] + next++ + p.Text = "" + text := candidate.Text + candidate.Text = "" + if reflect.DeepEqual(p, candidate) { + detail.Problems[i].Text = text + found = true + break + } + } + if !found { + return nil, ErrQueryReplicaNotReady + } + } + comment := "" + for _, effect := range op.Effects { + if effect.Review != nil { + comment = effect.Review.Comment + } + if effect.Reject != nil && comment == "" { + comment = effect.Reject.Comment + } + } + if detail.Stage == "quality" || len(op.Draft.Problems) == 0 { + detail.Comment = comment + } + } else { + r, ok := reviews[detail.ID] + if !ok || workforcestats.ReviewFingerprint(r) != v.SourceHash { + return nil, ErrQueryReplicaNotReady + } + detail.Comment = r.Comment + } + out = append(out, detail) + } + return out, ctx.Err() +} diff --git a/internal/store/workforce_stats.go b/internal/store/workforce_stats.go index 9d6213a4..36dc237c 100644 --- a/internal/store/workforce_stats.go +++ b/internal/store/workforce_stats.go @@ -460,6 +460,9 @@ func (s *MongoStore) LoadWorkforceStats(ctx context.Context, org string, q workf return err } defer cur.Close(ctx) + if budget, ok := ctx.Value(workforceBuildBudgetKey{}).(*workforceBuildBudget); ok { + return workforceBoundedRead(ctx, cur, target, budget) + } return cur.All(ctx, target) } window := func(field string) bson.M { @@ -512,12 +515,7 @@ func (s *MongoStore) LoadWorkforceStats(ctx context.Context, org string, q workf if q.SupplierID != "" { of["$or"] = bson.A{bson.M{"effects.beforeTask.supplierId": q.SupplierID}, bson.M{"effects.beforeItem.supplierId": q.SupplierID}, bson.M{"effects.worklog.supplierId": q.SupplierID}, bson.M{"effects.item.supplierId": q.SupplierID}} } - projection := fields("_id", "operationId", "organizationId", "projectId", "itemId", "taskId", "stageId", "reviewRound", "decision", "status", "createdAt", "completedAt", "draft.modelVersion", "draft.problems.id", "draft.problems.scope", "draft.problems.sourceItemId", "draft.problems.sliceId", "draft.problems.actionId", "draft.problems.start", "draft.problems.end", "draft.problems.timeBasis", "effects.review.id", "effects.review.taskId", "effects.review.revision", "effects.beforeTask.id", "effects.beforeTask.supplierId", "effects.beforeTask.projectId", "effects.beforeTask.templateVersion", "effects.beforeItem.id", "effects.beforeItem.legacyTaskId", "effects.beforeItem.latestAuthorUserId", "effects.beforeItem.acceptedByUserId", "effects.beforeItem.supplierId", "effects.beforeItem.projectId", "effects.beforeItem.currentRevision", "effects.beforeItem.subject.start", "effects.beforeItem.subject.end", "effects.item.id", "effects.item.legacyTaskId", "effects.item.latestAuthorUserId", "effects.item.supplierId", "effects.item.currentRevision", "effects.item.acceptedByUserId") - if q.Details { - projection["draft.problems.text"] = 1 - projection["effects.review.comment"] = 1 - projection["effects.reject.comment"] = 1 - } + projection := workforceOperationProjection(q.Details) if err := read(s.queryCollection("reviewOperations"), organizationScopedFilter(org, of), projection, &d.Operations); err != nil { return d, err } @@ -634,3 +632,21 @@ func (s *MongoStore) LoadWorkforceStats(ctx context.Context, org string, q workf } return d, nil } + +func workforceFields(names ...string) bson.M { + m := bson.M{} + for _, name := range names { + m[name] = 1 + } + return m +} + +func workforceOperationProjection(details bool) bson.M { + projection := workforceFields("_id", "operationId", "organizationId", "projectId", "itemId", "taskId", "stageId", "reviewRound", "decision", "status", "createdAt", "completedAt", "draft.modelVersion", "draft.problems.id", "draft.problems.scope", "draft.problems.sourceItemId", "draft.problems.sliceId", "draft.problems.actionId", "draft.problems.start", "draft.problems.end", "draft.problems.timeBasis", "effects.review.id", "effects.review.taskId", "effects.review.revision", "effects.worklog.supplierId", "effects.beforeTask.id", "effects.beforeTask.supplierId", "effects.beforeTask.projectId", "effects.beforeTask.templateVersion", "effects.beforeItem.id", "effects.beforeItem.legacyTaskId", "effects.beforeItem.latestAuthorUserId", "effects.beforeItem.acceptedByUserId", "effects.beforeItem.supplierId", "effects.beforeItem.projectId", "effects.beforeItem.currentRevision", "effects.beforeItem.subject.start", "effects.beforeItem.subject.end", "effects.item.id", "effects.item.legacyTaskId", "effects.item.latestAuthorUserId", "effects.item.supplierId", "effects.item.currentRevision", "effects.item.acceptedByUserId") + if details { + projection["draft.problems.text"] = 1 + projection["effects.review.comment"] = 1 + projection["effects.reject.comment"] = 1 + } + return projection +} diff --git a/internal/workforcestats/calculate.go b/internal/workforcestats/calculate.go index dd69f741..f9fdd269 100644 --- a/internal/workforcestats/calculate.go +++ b/internal/workforcestats/calculate.go @@ -181,7 +181,9 @@ type row struct { quality, acceptance returnAccumulator } -func Build(d Snapshot, q Query) Report { +func Build(d Snapshot, q Query) Report { return build(d, q, nil) } + +func build(d Snapshot, q Query, emit func(Fact)) Report { result := Report{Items: []domain.DashboardWorkforceMetric{}, Details: []domain.WorkforceReturnDetail{}} tasks := map[string]domain.AnnotationTaskRecord{} for _, t := range d.Tasks { @@ -335,7 +337,13 @@ func Build(d Snapshot, q Query) Report { day = localDay(e.CreatedAt) } r := get(e.UserID, e.SupplierID, e.TemplateVersion, day) - for _, dest := range []*row{r, &total} { + var contribution *row + dests := []*row{r, &total} + if emit != nil { + contribution = &row{first: map[string]bool{}} + dests = append(dests, contribution) + } + for _, dest := range dests { AddEvent(&dest.metric, e, revision) if seconds, known, review := reviewSeconds(e); review { dest.metric.ReviewDecisionCount++ @@ -370,6 +378,12 @@ func Build(d Snapshot, q Query) Report { dest.metric.ReworkVideoSeconds += seconds } } + if emit != nil { + f := factForRow(r.metric, e.ProjectID, e.UserID, e.SupplierID, e.CreatedAt, e.TemplateVersion) + f.Kind, f.Key, f.EventCount = "event", key, 1 + f.Metric = contribution.metric + emit(f) + } } // Attribution uses the frozen reviewed version, never the current task owner. @@ -399,7 +413,41 @@ func Build(d Snapshot, q Query) Report { } return user } - addReturn := func(detail domain.WorkforceReturnDetail, template int) { + operationSuppressors := map[string][]LegacyOperationScope{} + if emit != nil { + for _, op := range d.Operations { + if op.Status != "completed" || op.Decision != "rejected" { + continue + } + ids := []string{} + instant, e := time.Parse(time.RFC3339Nano, op.CompletedAt) + if e != nil { + continue + } + scope := LegacyOperationScope{ProjectID: op.ProjectID, At: op.CompletedAt, Seconds: instant.Unix(), Nanos: instant.Nanosecond(), Suppliers: []string{}} + for _, effect := range op.Effects { + if effect.Review != nil { + ids = append(ids, effect.Review.ID) + } + if effect.BeforeTask != nil { + scope.Suppliers = append(scope.Suppliers, effect.BeforeTask.SupplierID) + } + if effect.BeforeItem != nil { + scope.Suppliers = append(scope.Suppliers, effect.BeforeItem.SupplierID) + } + if effect.Worklog != nil { + scope.Suppliers = append(scope.Suppliers, effect.Worklog.SupplierID) + } + if effect.Item != nil { + scope.Suppliers = append(scope.Suppliers, effect.Item.SupplierID) + } + } + for _, id := range ids { + operationSuppressors[id] = append(operationSuppressors[id], scope) + } + } + } + addReturn := func(detail domain.WorkforceReturnDetail, template int, source any) { if q.TemplateVersion != nil && template != *q.TemplateVersion { return } @@ -429,6 +477,45 @@ func Build(d Snapshot, q Query) Report { } sort.Strings(detail.TaskIDs) detail.Metrics.TaskCount = int64(len(detail.TaskIDs)) + if emit != nil { + f := factForRow(r.metric, detail.ProjectID, detail.UserID, detail.SupplierID, detail.CreatedAt, template) + stored := detail + stored.Comment = "" + if detail.Problems != nil { + stored.Problems = append([]domain.ReviewProblem{}, detail.Problems...) + for i := range stored.Problems { + stored.Problems[i].Text = "" + } + } + f.Kind, f.Key, f.Detail = "return", detail.ID, &stored + switch v := source.(type) { + case domain.ReviewOperation: + f.SourceKind = "operation" + f.SourceHash = OperationFingerprint(v) + case domain.AnnotationReviewRecord: + f.SourceKind = "review" + f.SourceHash = ReviewFingerprint(v) + supplierName := func(id string) string { + if name := suppliers[id]; name != "" { + return name + } + if id == "" { + return "未分配供应商" + } + return "历史供应商(名称待核对)" + } + baseSupplier := canonical["submit:"+revisionKey(v.TaskID, v.Revision)].SupplierID + legacy := &LegacyReturnScope{SupplierID: baseSupplier, SupplierName: supplierName(baseSupplier), Events: []LegacyEventScope{}, Suppressors: []LegacyOperationScope{}} + for _, event := range eventsByTask[v.TaskID] { + if event.Revision == v.Revision && event.CreatedAt == v.CreatedAt && event.EventType == detail.Stage+"_rejected" { + legacy.Events = append(legacy.Events, LegacyEventScope{UserID: event.UserID, SupplierID: event.SupplierID, SupplierName: supplierName(event.SupplierID), ProjectID: event.ProjectID}) + } + } + legacy.Suppressors = append(legacy.Suppressors, operationSuppressors[v.ID]...) + f.Legacy = legacy + } + emit(f) + } if q.Details { result.Details = append(result.Details, detail) } @@ -491,7 +578,7 @@ func Build(d Snapshot, q Query) Report { if len(detail.Problems) == 0 { detail.Metrics.UnknownDetailCount = 1 } - addReturn(detail, t.TemplateVersion) + addReturn(detail, t.TemplateVersion, op) } else if op.StageID == "acceptance" || op.StageID == "prepackage_acceptance" { // Acceptance can contain multiple sources. Keep overall opinions unassigned // when several reviewers/suppliers are involved rather than duplicating blame. @@ -564,14 +651,14 @@ func Build(d Snapshot, q Query) Report { detail.Metrics.UnknownDetailCount = 1 detail.Comment = comment } - addReturn(detail, o.template) + addReturn(detail, o.template, op) } } } // Old formal records without a frozen structured draft still contribute a // rejection. Their problem count/time is explicitly unknown, never guessed. for _, r := range d.Reviews { - if r.Decision != domain.AnnotationReviewRejected || !InRange(r.CreatedAt, q) || handledReviews[r.ID] || strings.HasPrefix(r.ID, "review-operation-") { + if r.Decision != domain.AnnotationReviewRejected || !InRange(r.CreatedAt, q) || (handledReviews[r.ID] && emit == nil) || strings.HasPrefix(r.ID, "review-operation-") { continue } t := tasks[r.TaskID] @@ -603,7 +690,7 @@ func Build(d Snapshot, q Query) Report { } } detail.Metrics.SliceCount = int64(len(unique)) - addReturn(detail, t.TemplateVersion) + addReturn(detail, t.TemplateVersion, r) } // Final acceptance decides aggregated sources. Approvals freeze no source // version, so each source is priced at its latest quality-approved version @@ -672,11 +759,23 @@ func Build(d Snapshot, q Query) Report { day = localDay(op.CompletedAt) } r := get(op.UserID, supplier, template, day) - for _, dest := range []*row{r, &total} { + var contribution *row + dests := []*row{r, &total} + if emit != nil { + contribution = &row{first: map[string]bool{}} + dests = append(dests, contribution) + } + for _, dest := range dests { dest.metric.ReviewDecisionCount++ dest.metric.ReviewVideoSeconds += seconds dest.metric.UnknownReviewVideoCount += unknown } + if emit != nil { + f := factForRow(r.metric, op.ProjectID, op.UserID, supplier, op.CompletedAt, template) + f.Kind, f.Key = "acceptance", op.ID + f.Metric = contribution.metric + emit(f) + } } for _, r := range rows { r.metric.QualityReturns = r.quality.metrics @@ -690,13 +789,7 @@ func Build(d Snapshot, q Query) Report { total.metric.QualityReturns = total.quality.metrics total.metric.AcceptanceReturns = total.acceptance.metrics result.Totals = total.metric - sort.Slice(result.Details, func(i, j int) bool { - a, b := result.Details[i], result.Details[j] - if a.CreatedAt != b.CreatedAt { - return parsedAfter(a.CreatedAt, b.CreatedAt) - } - return a.ID+a.UserID+a.SupplierID < b.ID+b.UserID+b.SupplierID - }) + sort.SliceStable(result.Details, func(i, j int) bool { return ReturnDetailLess(result.Details[i], result.Details[j]) }) return result } diff --git a/internal/workforcestats/calculate_test.go b/internal/workforcestats/calculate_test.go index ac1f2909..e4d0bcb3 100644 --- a/internal/workforcestats/calculate_test.go +++ b/internal/workforcestats/calculate_test.go @@ -24,7 +24,7 @@ func TestSubmissionsUseBusinessEvidenceAndStableIdentity(t *testing.T) { Slices: []domain.SliceRevisionRecord{{TaskID: "one", Revision: 4, Segments: []domain.SliceSegment{{Start: 0, End: 79}}}, {TaskID: "one", Revision: 7, Segments: []domain.SliceSegment{{Start: 0, End: 79}}}}, Suppliers: []domain.SupplierRecord{{ID: "vendor", Name: "倍赛"}}, } - r := Build(d, Query{}) + r := buildWithFactParity(t, d, Query{}) m := r.Totals if m.FirstSubmittedTasks != 1 || m.SubmissionCount != 3 || m.ReworkSubmissionCount != 1 || m.UnknownSubmissionCount != 1 || m.FirstVideoSeconds != 79 || m.ReworkVideoSeconds != 79 || m.UnknownVideoCount != 1 { t.Fatalf("incorrect counters: %+v", m) @@ -38,7 +38,7 @@ func TestSubmissionsUseBusinessEvidenceAndStableIdentity(t *testing.T) { d.Events[2].VideoSeconds = &seconds d.Revisions = nil d.Slices = nil - r = Build(d, Query{}) + r = buildWithFactParity(t, d, Query{}) if r.Totals.ReworkSubmissionCount != 1 || r.Totals.ReworkVideoSeconds != 79 || r.Totals.FirstSubmittedTasks != 1 { t.Fatalf("durable evidence lost: %+v", r.Totals) } @@ -53,12 +53,12 @@ func TestSubmissionPeriodAndScope(t *testing.T) { {ID: "other", TaskID: "two", Revision: 1, EventType: "combined_submitted", OutputDelta: 1, CreatedAt: at, UserID: "foreign", SupplierID: "foreign", ProjectID: "other"}, }} q := Query{From: from, To: to, UserID: "u", SupplierID: "v", AllowedProjects: map[string]bool{"p": true}} - m := Build(d, q).Totals + m := buildWithFactParity(t, d, q).Totals if m.FirstSubmittedTasks != 0 || m.ReworkSubmissionCount != 1 || m.SubmissionCount != 1 { t.Fatalf("cross-period/scope: %+v", m) } q.AllowedProjects = map[string]bool{} - if len(Build(d, q).Items) != 0 { + if len(buildWithFactParity(t, d, q).Items) != 0 { t.Fatal("denied project leaked") } } @@ -74,7 +74,7 @@ func TestWorkforceRoleAndDayGroupingWithRates(t *testing.T) { Users: []domain.UserRecord{{ID: "ann", Name: "标注员", Role: domain.UserRoleAnnotator}, {ID: "qc", Name: "质检", Role: domain.UserRoleQualityReviewer}}, } q := Query{Role: string(domain.UserRoleQualityReviewer), GroupBy: "day"} - r := Build(d, q) + r := buildWithFactParity(t, d, q) if len(r.Items) != 2 || r.Items[0].Role != string(domain.UserRoleQualityReviewer) || r.Items[0].Date == "" { t.Fatalf("role/day grouping: %+v", r.Items) } @@ -96,7 +96,7 @@ func TestCrossDayRetryKeepsOriginalSubmissionDate(t *testing.T) { retry.CreatedAt = at from, _ := time.Parse(time.RFC3339, "2026-09-28T00:00:00Z") d := Snapshot{Events: []domain.AnnotationWorklogEventRecord{retry}, SubmissionHistory: []domain.AnnotationWorklogEventRecord{original, retry}} - if m := Build(d, Query{From: from, To: from.Add(24 * time.Hour)}).Totals; m.SubmissionCount != 0 || m.FirstSubmittedTasks != 0 { + if m := buildWithFactParity(t, d, Query{From: from, To: from.Add(24 * time.Hour)}).Totals; m.SubmissionCount != 0 || m.FirstSubmittedTasks != 0 { t.Fatalf("retry moved first submission into another day: %+v", m) } // A later valid first-kind record for an already submitted task can count as @@ -104,7 +104,7 @@ func TestCrossDayRetryKeepsOriginalSubmissionDate(t *testing.T) { retry.Revision = 5 d.Events[0] = retry d.SubmissionHistory[1] = retry - if m := Build(d, Query{From: from, To: from.Add(24 * time.Hour)}).Totals; m.SubmissionCount != 1 || m.FirstSubmittedTasks != 0 { + if m := buildWithFactParity(t, d, Query{From: from, To: from.Add(24 * time.Hour)}).Totals; m.SubmissionCount != 1 || m.FirstSubmittedTasks != 0 { t.Fatalf("task counted twice as first: %+v", m) } } @@ -166,7 +166,7 @@ func TestCompletedReturnsRoundTotalsFrozenOwnershipAndLegacy(t *testing.T) { {ID: one.ID + ":review", TaskID: "task", Revision: 4, Decision: domain.AnnotationReviewRejected, Stage: domain.AnnotationReviewStageQuality, CreatedAt: at}, {ID: failed.ID + ":review", TaskID: "task", Revision: 4, Decision: domain.AnnotationReviewRejected, Stage: domain.AnnotationReviewStageQuality, CreatedAt: at}, }} - r := Build(d, Query{Details: true}) + r := buildWithFactParity(t, d, Query{Details: true}) if len(r.Items) != 1 || r.Items[0].UserID != "original-author" || r.Items[0].SupplierID != "original-vendor" { t.Fatalf("not frozen ownership: %+v", r.Items) } @@ -177,7 +177,7 @@ func TestCompletedReturnsRoundTotalsFrozenOwnershipAndLegacy(t *testing.T) { if len(r.Details) != 2 || r.Details[0].Problems[0].Text != "保留原文" { t.Fatal("lost detail") } - if len(Build(d, Query{UserID: "new-author"}).Items) != 0 { + if len(buildWithFactParity(t, d, Query{UserID: "new-author"}).Items) != 0 { t.Fatal("assigned current user historical blame") } if !reflect.DeepEqual(one.Draft, d.Operations[0].Draft) { @@ -199,12 +199,12 @@ func TestAcceptanceTargetsQualityReviewerAndSkipsTaskScope(t *testing.T) { {TaskID: "one", Revision: 4, Stage: domain.AnnotationReviewStageQuality, Decision: domain.AnnotationReviewApproved, ReviewerUserID: "qc1", CreatedAt: "2026-09-27T00:00:00Z"}, {TaskID: "two", Revision: 2, Stage: domain.AnnotationReviewStageQuality, Decision: domain.AnnotationReviewApproved, ReviewerUserID: "qc2", CreatedAt: "2026-09-27T00:00:00Z"}, }} - r := Build(d, Query{Details: true}) + r := buildWithFactParity(t, d, Query{Details: true}) m := r.Totals.AcceptanceReturns if m.DecisionCount != 1 || m.TaskCount != 2 || m.ProblemCount != 1 || m.VideoSeconds != 10 || len(r.Items) != 2 { t.Fatalf("acceptance counts %+v rows %+v", m, r.Items) } - r = Build(d, Query{SupplierID: "v1", Details: true}) + r = buildWithFactParity(t, d, Query{SupplierID: "v1", Details: true}) if len(r.Items) != 1 || r.Items[0].UserID != "qc1" || r.Totals.AcceptanceReturns.ProblemCount != 1 || len(r.Details[0].Problems) != 1 { t.Fatalf("foreign overall/source leaked: %+v", r) } @@ -247,7 +247,7 @@ func TestReviewVideoSecondsPerDecision(t *testing.T) { SourceItems: []domain.HumanItemRecord{{ID: "src-a", LegacyTaskID: "a"}, {ID: "src-b", LegacyTaskID: "b"}}, Users: []domain.UserRecord{{ID: "acc", SupplierID: "v"}, {ID: "qc", SupplierID: "v"}, {ID: "author", SupplierID: "v"}}, } - r := Build(d, Query{}) + r := buildWithFactParity(t, d, Query{}) byUser := map[string]domain.DashboardWorkforceMetric{} for _, item := range r.Items { byUser[item.UserID] = item @@ -282,12 +282,12 @@ func TestReviewVideoSecondsPerDecision(t *testing.T) { t.Fatalf("review output totals: %+v", r.Totals) } // Scope: a reviewer's own report only carries their own decisions. - self := Build(d, Query{UserID: "acc", MatchUser: true}).Totals + self := buildWithFactParity(t, d, Query{UserID: "acc", MatchUser: true}).Totals if self.ReviewVideoSeconds != 72 || self.FirstVideoSeconds != 0 { t.Fatalf("self scope: %+v", self) } // Day grouping attributes acceptance by its completion date. - days := Build(d, Query{UserID: "acc", MatchUser: true, GroupBy: "day"}).Items + days := buildWithFactParity(t, d, Query{UserID: "acc", MatchUser: true, GroupBy: "day"}).Items if len(days) != 1 || days[0].Date != "2026-09-28" || days[0].ReviewVideoSeconds != 72 { t.Fatalf("day grouping: %+v", days) } @@ -338,7 +338,7 @@ func TestReviewVideoSecondsFrozenThenSubmissionFallback(t *testing.T) { Users: []domain.UserRecord{{ID: "acc", SupplierID: "v"}, {ID: "qc", SupplierID: "v"}}, } byUser := map[string]domain.DashboardWorkforceMetric{} - for _, item := range Build(d, Query{}).Items { + for _, item := range buildWithFactParity(t, d, Query{}).Items { byUser[item.UserID] = item } if got := byUser["qc"]; got.ReviewVideoSeconds != 88 || got.UnknownReviewVideoCount != 1 || got.ReviewDecisionCount != 3 { diff --git a/internal/workforcestats/facts.go b/internal/workforcestats/facts.go new file mode 100644 index 00000000..f3e35493 --- /dev/null +++ b/internal/workforcestats/facts.go @@ -0,0 +1,289 @@ +package workforcestats + +import ( + "droboticflow/internal/domain" + "fmt" + "reflect" + "sort" + "strings" +) + +// Fact is a text-free contribution produced by the SAME historical calculator. +// Separate modes/template grouping preserve mixed acceptance attribution rather +// than attempting to add incompatible daily totals or guess frozen authors. +type LegacyEventScope struct { + UserID string `bson:"userId" json:"userId"` + SupplierID string `bson:"supplierId" json:"supplierId"` + SupplierName string `bson:"supplierName" json:"supplierName"` + ProjectID string `bson:"projectId" json:"projectId"` +} +type LegacyOperationScope struct { + Seconds int64 `bson:"seconds" json:"seconds"` + Nanos int `bson:"nanos" json:"nanos"` + ProjectID string `bson:"projectId" json:"projectId"` + At string `bson:"at" json:"at"` + Suppliers []string `bson:"suppliers" json:"suppliers"` +} +type LegacyReturnScope struct { + SupplierID string `bson:"supplierId" json:"supplierId"` + SupplierName string `bson:"supplierName" json:"supplierName"` + Events []LegacyEventScope `bson:"events" json:"events"` + Suppressors []LegacyOperationScope `bson:"suppressors" json:"suppressors"` +} +type Fact struct { + Legacy *LegacyReturnScope `bson:"legacy,omitempty" json:"legacy,omitempty"` + Key string `bson:"key" json:"key"` + Kind string `bson:"kind" json:"kind"` + Mode domain.RunMode `bson:"mode" json:"mode"` + GroupTemplate bool `bson:"groupTemplate" json:"groupTemplate"` + ProjectID string `bson:"projectId" json:"projectId"` + UserID string `bson:"userId" json:"userId"` + SupplierID string `bson:"supplierId" json:"supplierId"` + UserName string `bson:"userName" json:"userName"` + SupplierName string `bson:"supplierName" json:"supplierName"` + Role string `bson:"role" json:"role"` + TemplateVersion int `bson:"templateVersion" json:"templateVersion"` + CreatedAt string `bson:"createdAt" json:"createdAt"` + Day string `bson:"day" json:"day"` + Metric domain.DashboardWorkforceMetric `bson:"-" json:"metric"` + EventCount int `bson:"eventCount" json:"eventCount"` + Detail *domain.WorkforceReturnDetail `bson:"detail,omitempty" json:"detail,omitempty"` + SourceKind string `bson:"sourceKind,omitempty" json:"sourceKind,omitempty"` + SourceHash string `bson:"sourceHash,omitempty" json:"sourceHash,omitempty"` +} + +func factForRow(m domain.DashboardWorkforceMetric, project, user, supplier, at string, template int) Fact { + return Fact{ProjectID: project, UserID: user, SupplierID: supplier, UserName: m.UserName, SupplierName: m.SupplierName, Role: m.Role, TemplateVersion: template, CreatedAt: at, Day: localDay(at)} +} +func ModeVariant(mode domain.RunMode) domain.RunMode { + if mode == domain.RunModeDebug || mode == domain.RunModeAll { + return mode + } + return domain.RunModeProduction +} + +// Numeric counters are shared by in-memory reduction and Mongo aggregation. +// Identity fields and undefined rate pointers never participate in addition. +type counterField struct { + name string + index []int + kind reflect.Kind +} + +var workforceCounterFields = func() []counterField { + out := []counterField{} + var walk func(reflect.Type, []int) + walk = func(t reflect.Type, parent []int) { + for i := 0; i < t.NumField(); i++ { + f := t.Field(i) + idx := append(append([]int(nil), parent...), i) + if f.Anonymous && f.Type.Kind() == reflect.Struct { + walk(f.Type, idx) + continue + } + if f.Name == "TemplateVersion" { + continue + } + if f.Type.Kind() == reflect.Int || f.Type.Kind() == reflect.Int64 || f.Type.Kind() == reflect.Float64 { + out = append(out, counterField{strings.Split(f.Tag.Get("json"), ",")[0], idx, f.Type.Kind()}) + } + } + } + walk(reflect.TypeOf(domain.DashboardWorkforceMetric{}), nil) + return out +}() + +func MetricCounterNames() []string { + out := []string{} + for _, f := range workforceCounterFields { + out = append(out, f.name) + } + return out +} +func MetricCounterValues(m domain.DashboardWorkforceMetric) map[string]any { + out := map[string]any{} + v := reflect.ValueOf(m) + for _, f := range workforceCounterFields { + out[f.name] = v.FieldByIndex(f.index).Interface() + } + return out +} +func addMetricCounters(dst *domain.DashboardWorkforceMetric, src domain.DashboardWorkforceMetric) { + a, b := reflect.ValueOf(dst).Elem(), reflect.ValueOf(src) + for _, f := range workforceCounterFields { + x, y := a.FieldByIndex(f.index), b.FieldByIndex(f.index) + if f.kind == reflect.Float64 { + x.SetFloat(x.Float() + y.Float()) + } else { + x.SetInt(x.Int() + y.Int()) + } + } +} + +// Materialize emits canonical contributions once in the background. Equal +// event timestamps/dimensions combine without rounding the requested boundaries. +func Materialize(d Snapshot) []Fact { + byKey := map[string]Fact{} + for _, mode := range []domain.RunMode{domain.RunModeProduction, domain.RunModeDebug, domain.RunModeAll} { + for _, templates := range []bool{false, true} { + q := Query{Mode: mode, GroupTemplate: templates} + build(d, q, func(f Fact) { + f.Mode, f.GroupTemplate = mode, templates + identity := f.Key + if f.Kind == "event" { + identity = "" + } + stage := "" + if f.Detail != nil { + stage = f.Detail.Stage + } + f.Key = workforceFingerprint([]any{mode, templates, f.Kind, f.SourceKind, stage, identity, f.ProjectID, f.UserID, f.SupplierID, f.TemplateVersion, f.CreatedAt}) + if old, ok := byKey[f.Key]; ok && f.Kind == "event" { + addMetricCounters(&old.Metric, f.Metric) + old.EventCount += f.EventCount + byKey[f.Key] = old + } else { + byKey[f.Key] = f + } + }) + } + } + keys := make([]string, 0, len(byKey)) + for key := range byKey { + keys = append(keys, key) + } + sort.Strings(keys) + out := make([]Fact, 0, len(keys)) + for _, key := range keys { + out = append(out, byKey[key]) + } + sort.SliceStable(out, func(i, j int) bool { + if out[i].SourceKind != out[j].SourceKind { + return out[i].SourceKind < out[j].SourceKind + } + return out[i].Key < out[j].Key + }) + return out +} +func EffectiveFact(f Fact, q Query) (Fact, bool) { + if f.Legacy == nil { + return f, true + } + l := f.Legacy + for _, op := range l.Suppressors { + if !InRange(op.At, q) || (q.ProjectID != "" && q.ProjectID != op.ProjectID) || (q.AllowedProjects != nil && !q.AllowedProjects[op.ProjectID]) { + continue + } + if q.SupplierID != "" { + found := false + for _, supplier := range op.Suppliers { + if supplier == q.SupplierID { + found = true + } + } + if !found { + continue + } + } + return f, false + } + f.SupplierID, f.SupplierName = l.SupplierID, l.SupplierName + for _, e := range l.Events { + if (q.ProjectID != "" && q.ProjectID != e.ProjectID) || (q.AllowedProjects != nil && !q.AllowedProjects[e.ProjectID]) || ((q.MatchUser || q.UserID != "") && q.UserID != e.UserID) || (q.SupplierID != "" && q.SupplierID != e.SupplierID) { + continue + } + f.SupplierID, f.SupplierName = e.SupplierID, e.SupplierName + break + } + if f.Detail != nil { + v := *f.Detail + v.SupplierID = f.SupplierID + v.AttributionUnknown = v.UserID == "" || v.SupplierID == "" + f.Detail = &v + } + return f, true +} +func factVisible(f Fact, q Query) bool { + return f.Mode == ModeVariant(q.Mode) && f.GroupTemplate == q.GroupTemplate && InRange(f.CreatedAt, q) && (q.ProjectID == "" || q.ProjectID == f.ProjectID) && (q.SupplierID == "" || q.SupplierID == f.SupplierID) && (q.Role == "" || q.Role == f.Role) && ((!q.MatchUser && q.UserID == "") || q.UserID == f.UserID) && (q.TemplateVersion == nil || *q.TemplateVersion == f.TemplateVersion) && (q.AllowedProjects == nil || q.AllowedProjects[f.ProjectID]) && (q.AllowedSuppliers == nil || q.AllowedSuppliers[f.SupplierID]) +} +func BuildFacts(facts []Fact, q Query) Report { + result := Report{Items: []domain.DashboardWorkforceMetric{}, Details: []domain.WorkforceReturnDetail{}} + rows := map[string]*row{} + total := row{} + for _, f := range facts { + var visible bool + f, visible = EffectiveFact(f, q) + if !visible { + continue + } + if !factVisible(f, q) { + continue + } + day := "" + if q.GroupBy == "day" { + day = f.Day + } + template := 0 + if q.GroupTemplate { + template = f.TemplateVersion + } + key := fmt.Sprintf("%s\x00%s\x00%d\x00%s", f.UserID, f.SupplierID, template, day) + r := rows[key] + if r == nil { + r = &row{metric: domain.DashboardWorkforceMetric{UserID: f.UserID, UserName: f.UserName, Role: f.Role, SupplierID: f.SupplierID, SupplierName: f.SupplierName, TemplateVersion: template, Date: day}} + rows[key] = r + } + for _, dst := range []*row{r, &total} { + addMetricCounters(&dst.metric, f.Metric) + if f.Detail != nil { + if f.Detail.Stage == "quality" { + dst.quality.add(*f.Detail) + } else { + dst.acceptance.add(*f.Detail) + } + } + } + result.EventCount += f.EventCount + if q.Details && f.Detail != nil { + result.Details = append(result.Details, *f.Detail) + } + } + for _, r := range rows { + r.metric.QualityReturns = r.quality.metrics + r.metric.AcceptanceReturns = r.acceptance.metrics + result.Items = append(result.Items, r.metric) + } + sort.Slice(result.Items, func(i, j int) bool { + a, b := result.Items[i], result.Items[j] + return fmt.Sprintf("%s/%s/%s/%s/%d", a.Date, a.SupplierName, a.UserName, a.UserID, a.TemplateVersion) < fmt.Sprintf("%s/%s/%s/%s/%d", b.Date, b.SupplierName, b.UserName, b.UserID, b.TemplateVersion) + }) + total.metric.QualityReturns, total.metric.AcceptanceReturns = total.quality.metrics, total.acceptance.metrics + result.Totals = total.metric + sort.SliceStable(result.Details, func(i, j int) bool { return ReturnDetailLess(result.Details[i], result.Details[j]) }) + return result +} + +// ReturnFacts preserves the source kind even if old review/operation collections +// contain the same ID. It is used only for pre-materialization compatibility. +func ReturnFacts(d Snapshot, q Query) []Fact { + out := []Fact{} + build(d, q, func(f Fact) { + if f.Kind != "return" { + return + } + if effective, ok := EffectiveFact(f, q); ok { + out = append(out, effective) + } + }) + sort.SliceStable(out, func(i, j int) bool { + a, b := *out[i].Detail, *out[j].Detail + if ReturnDetailLess(a, b) { + return true + } + if ReturnDetailLess(b, a) { + return false + } + return out[i].SourceKind < out[j].SourceKind + }) + return out +} diff --git a/internal/workforcestats/facts_test.go b/internal/workforcestats/facts_test.go new file mode 100644 index 00000000..6c972e97 --- /dev/null +++ b/internal/workforcestats/facts_test.go @@ -0,0 +1,110 @@ +package workforcestats + +import ( + "droboticflow/internal/domain" + "math" + "reflect" + "testing" +) + +// Run the existing business fixtures against both paths. Source loaders have +// always narrowed event/operation scope BEFORE calling Build; retain that seam. +func scopeFactComparison(d Snapshot, q Query) Snapshot { + out := d + out.SubmissionHistory = append(append([]domain.AnnotationWorklogEventRecord{}, d.SubmissionHistory...), d.Events...) + out.Events = nil + for _, e := range d.Events { + if !InRange(e.CreatedAt, q) || (q.ProjectID != "" && q.ProjectID != e.ProjectID) || (q.AllowedProjects != nil && !q.AllowedProjects[e.ProjectID]) || (q.SupplierID != "" && q.SupplierID != e.SupplierID) || ((q.MatchUser || q.UserID != "") && q.UserID != e.UserID) { + continue + } + out.Events = append(out.Events, e) + } + out.Operations = nil + for _, op := range d.Operations { + if !InRange(op.CompletedAt, q) || (q.ProjectID != "" && q.ProjectID != op.ProjectID) || (q.AllowedProjects != nil && !q.AllowedProjects[op.ProjectID]) { + continue + } + if q.SupplierID != "" { + found := false + for _, e := range op.Effects { + found = found || (e.BeforeTask != nil && e.BeforeTask.SupplierID == q.SupplierID) || (e.BeforeItem != nil && e.BeforeItem.SupplierID == q.SupplierID) || (e.Item != nil && e.Item.SupplierID == q.SupplierID) || (e.Worklog != nil && e.Worklog.SupplierID == q.SupplierID) + } + if !found { + continue + } + } + out.Operations = append(out.Operations, op) + } + out.AcceptanceDecisions = nil + for _, op := range d.AcceptanceDecisions { + if !InRange(op.CompletedAt, q) || (q.ProjectID != "" && q.ProjectID != op.ProjectID) || (q.AllowedProjects != nil && !q.AllowedProjects[op.ProjectID]) || ((q.MatchUser || q.UserID != "") && q.UserID != op.UserID) { + continue + } + out.AcceptanceDecisions = append(out.AcceptanceDecisions, op) + } + return out +} +func textFreeFactReport(r Report) Report { + for i := range r.Details { + r.Details[i].Comment = "" + if r.Details[i].Problems != nil { + r.Details[i].Problems = append([]domain.ReviewProblem{}, r.Details[i].Problems...) + for j := range r.Details[i].Problems { + r.Details[i].Problems[j].Text = "" + } + } + } + return r +} +func factReportsEqual(a, b reflect.Value) bool { + if a.Type() != b.Type() { + return false + } + switch a.Kind() { + case reflect.Float64: + return math.Abs(a.Float()-b.Float()) <= 1e-8 + case reflect.Struct: + for i := 0; i < a.NumField(); i++ { + if !factReportsEqual(a.Field(i), b.Field(i)) { + return false + } + } + return true + case reflect.Slice: + if a.IsNil() != b.IsNil() || a.Len() != b.Len() { + return false + } + for i := 0; i < a.Len(); i++ { + if !factReportsEqual(a.Index(i), b.Index(i)) { + return false + } + } + return true + case reflect.Pointer: + if a.IsNil() || b.IsNil() { + return a.IsNil() == b.IsNil() + } + return factReportsEqual(a.Elem(), b.Elem()) + default: + return reflect.DeepEqual(a.Interface(), b.Interface()) + } +} +func buildWithFactParity(t *testing.T, d Snapshot, q Query) Report { + t.Helper() + original := Build(d, q) + facts := Materialize(d) + want := textFreeFactReport(Build(scopeFactComparison(d, q), q)) + got := BuildFacts(facts, q) + if !factReportsEqual(reflect.ValueOf(got), reflect.ValueOf(want)) { + t.Fatalf("materialized facts differ from scoped historical calculator q=%+v\ngot=%+v\nwant=%+v", q, got, want) + } + return original +} + +func TestReturnPaginationEquivalentTimestampsHaveStableOrder(t *testing.T) { + a := domain.WorkforceReturnDetail{ID: "a", CreatedAt: "2026-11-01T00:00:00.000Z"} + b := domain.WorkforceReturnDetail{ID: "b", CreatedAt: "2026-11-01T08:00:00+08:00"} + if !ReturnDetailLess(a, b) || ReturnDetailLess(b, a) { + t.Fatal("equivalent timestamps lost stable identity order") + } +} diff --git a/internal/workforcestats/fingerprint.go b/internal/workforcestats/fingerprint.go new file mode 100644 index 00000000..a8a1c5f7 --- /dev/null +++ b/internal/workforcestats/fingerprint.go @@ -0,0 +1,37 @@ +package workforcestats + +import ( + "crypto/sha256" + "droboticflow/internal/domain" + "encoding/hex" + "encoding/json" +) + +// Fingerprints compare the exact projected formal record, excluding only text +// fetched later. Different revisions/attribution cannot lend text to old facts. +func OperationFingerprint(op domain.ReviewOperation) string { + raw, _ := json.Marshal(op) + var copy domain.ReviewOperation + _ = json.Unmarshal(raw, ©) + for i := range copy.Draft.Problems { + copy.Draft.Problems[i].Text = "" + } + for _, v := range copy.Effects { + if v.Review != nil { + v.Review.Comment = "" + } + if v.Reject != nil { + v.Reject.Comment = "" + } + } + return workforceFingerprint(copy) +} +func ReviewFingerprint(r domain.AnnotationReviewRecord) string { + r.Comment = "" + return workforceFingerprint(r) +} +func workforceFingerprint(v any) string { + raw, _ := json.Marshal(v) + sum := sha256.Sum256(raw) + return hex.EncodeToString(sum[:]) +} diff --git a/internal/workforcestats/slice_versions_test.go b/internal/workforcestats/slice_versions_test.go index eea3868f..508f55e8 100644 --- a/internal/workforcestats/slice_versions_test.go +++ b/internal/workforcestats/slice_versions_test.go @@ -30,7 +30,7 @@ func TestSliceVersionsPreservesReportAndExcludesAutosaves(t *testing.T) { if len(needed) != 2 || !needed["task"][2] || !needed["task"][3] || !needed["legacy"][7] { t.Fatalf("incorrect needed versions: %v", needed) } - before := Build(d, Query{}) + before := buildWithFactParity(t, d, Query{}) pruned := d pruned.Slices = nil for _, s := range d.Slices { @@ -41,7 +41,7 @@ func TestSliceVersionsPreservesReportAndExcludesAutosaves(t *testing.T) { if len(pruned.Slices) != 3 { t.Fatalf("read %d versions instead of 3", len(pruned.Slices)) } - if after := Build(pruned, Query{}); !reflect.DeepEqual(before, after) { + if after := buildWithFactParity(t, pruned, Query{}); !reflect.DeepEqual(before, after) { t.Fatal("report changed after excluding unused historical slice versions") } d.AcceptanceDecisions[0].ReviewVideoSources = []domain.ReviewVideoSource{{SourceItemID: "source", TaskID: "legacy", VideoSeconds: &frozen}} diff --git a/internal/workforcestats/types.go b/internal/workforcestats/types.go index 307b68ee..0f77a51d 100644 --- a/internal/workforcestats/types.go +++ b/internal/workforcestats/types.go @@ -60,3 +60,25 @@ func ModeMatches(mode domain.RunMode, q Query) bool { } return mode != domain.RunModeDebug } + +// ReturnDetailLess orders equivalent RFC3339 spellings by stable identity too, +// so offset pages cannot reshuffle records representing the same instant. +func ReturnDetailLess(a, b domain.WorkforceReturnDetail) bool { + if parsedAfter(a.CreatedAt, b.CreatedAt) { + return true + } + if parsedAfter(b.CreatedAt, a.CreatedAt) { + return false + } + x, y := a.ID+a.UserID+a.SupplierID, b.ID+b.UserID+b.SupplierID + if x != y { + return x < y + } + if a.ID != b.ID { + return a.ID < b.ID + } + if a.UserID != b.UserID { + return a.UserID < b.UserID + } + return a.SupplierID < b.SupplierID +} diff --git a/scripts/test_query_replica.py b/scripts/test_query_replica.py index 9b2a01c1..00c5726f 100644 --- a/scripts/test_query_replica.py +++ b/scripts/test_query_replica.py @@ -21,7 +21,9 @@ CASES = ('internal/store/query_replica_local_test.go', 'internal/api/query_replica_local_test.go', 'cmd/query-cdc-probe/main_test.go', 'internal/store/query_replica_valkey_local_test.go', - 'internal/store/workforce_report_cache_local_test.go') + 'internal/store/workforce_report_cache_local_test.go', + 'internal/store/workforce_facts_local_test.go', + 'internal/store/query_pool_buckets_local_test.go') def owned_native(): diff --git a/scripts/verification-scopes.json b/scripts/verification-scopes.json index 76b8a30e..4aacf09b 100644 --- a/scripts/verification-scopes.json +++ b/scripts/verification-scopes.json @@ -1013,7 +1013,9 @@ "paths": [ "^internal/store/query_(replica|sync|tracked_collection|collection_interface).*\\.go$", "^internal/api/query_replica.*\\.go$", - "^internal/store/workforce_(report_cache|heap_diagnostic).*\\.go$", + "^internal/store/workforce_(report_cache|heap_diagnostic|facts|return_page).*\\.go$", + "^internal/store/query_pool_buckets.*\\.go$", + "^internal/api/query_fact_routes_test\\.go$", "^internal/api/workforce_compact_report_test\\.go$", "^internal/testsupport/query_replica_lab.*\\.go$", "^cmd/(query-replica|query-cdc-probe)/",