⚡ Materialize query facts and paginate return details
This commit is contained in:
parent
5e59ae50eb
commit
c20dbacab6
@ -30,3 +30,12 @@
|
||||
## 验收
|
||||
|
||||
详见 [操作与验收说明](../operations/query-replica.md)。必须比较原库总 CPU、扫描和读取字节,包含新索引、标记、确认和对账成本。短时本机测试和无 CPU 配额的 native 测试不构成生产容量验收。
|
||||
|
||||
|
||||
## 查询库内事实聚合(2026-11-13,本地实施)
|
||||
|
||||
查询库事务同时维护按服务端可见性字段分组的计数桶与代次证书。读取桶、证书及待同步旧记录使用同一查询库快照;服务端实时运行原权限判定。旧工作者导致证书代次落后、身份敏感授权及复杂筛选继续原候选路径,不使用近似计数。
|
||||
|
||||
人员统计复用原历史计算器生成无正文贡献,按不可变版本发布。完整新版本提交前始终保留上一版本;重启清理未提交暂存,已提交历史按有界宽限清理。历史供应商归属和正式操作抑制旧审查仍依查询范围解析,日期边界保留纳秒。后台当前使用有界组织快照计算与差量写入,尚非逐任务分区增量计算;超预算保留旧版本,不截断数据、不增加核心作业准入条件。
|
||||
|
||||
作业记录在查询库筛选计数及分页后,仅补选中页详情与权威文字,来源指纹不一致时拒绝拼接。旧任务/未首次物化的库保留读库历史兼容路径。极少数等价时间字符串的排序统一为绝对时间和稳定身份,避免偏移页漂移。以上实现仍需完整规模性能及持久化确认压力验收,不能将本地功能测试当作上线承诺。
|
||||
|
||||
@ -164,3 +164,29 @@ IO保护新增严格的FUSE归因分支,并有先失败后通过的回归:
|
||||
下一步应先减少读库端每请求的全范围工作:任务池计数建立按权限范围可组合的轻量事实/索引聚合;人员统计按冻结作者和阶段事实增量物化;作业记录先在事实表筛选分页,再补当前页文本;控制后台构建并发和发布节奏,保留上一完整代次。权限撤销与领取/提交仍实时核验,不能用过期事实取代权威判断。共享Valkey的跨进程正确性已在本机验证,本轮集群夹具未接Valkey,不能声称已验证完整共享缓存组合。单纯提高API核数、延长HTTP超时、跳过冷查询或放宽本地盘保护都不算修复。若后续评估更大的查询库内存/IO预算,须单列资源方案与对照,不能混入本轮结果。
|
||||
|
||||
本轮临时源/驱动Pod与两条策略已按UID清理,桥接按PID/可执行文件身份停止,配额CAS恢复1Pod/1CPU/1Gi;25Gi库和持久标记保留,查询Pod仍按2小时deadline退出。未切换任何生产读取或修改客户数据。清理证据在`io-rca/fixture-5/cleanup.json`。
|
||||
|
||||
## 查询库事实物化与分页(2026-11-13)
|
||||
|
||||
本批以 `5e59ae50` 为起点,仅在 `codex/mongo-cpu` 和合成隔离环境实现,未切换生产路由,未修改客户数据。
|
||||
|
||||
- 计数:独立查询库事务内维护 `queryPoolCountBuckets`;权限维度分组只减少候选传输,不缓存授权结果。服务端仍按当前成员、批次、供应商及验收权限判定。脏权威记录在同一请求扣旧加新;桶、代次证书及旧行在同一查询库快照读取。身份敏感授权、复杂搜索和旧桶未建好的库继续原逐条路径。旧投影工作者导致桶代次不一致时安全回退。
|
||||
- 统计:复用原历史计算器输出无正文的事件、冻结作者、返工和验收贡献,在 `queryWorkforceFacts` 发布不可变代次。筛选和聚合在查询库执行,最终摘要进入有界进程/Valkey缓存。返工任务去重不按日去重数量相加;历史供应商归属及正式操作抑制旧记录仍按原查询范围解析。时间边界保留纳秒精度。显式空用户兼容查询继续原读库路径。
|
||||
- 作业记录:先筛选、排序、分页及计数,再读取本页事实详情;只按本页操作/旧审查 ID 从权威库补文字。来源元数据指纹不符时返回未就绪,避免新旧归属或批注拼接。相同绝对时间但不同时区/小数写法的旧记录改用稳定身份顺序,避免分页漂移。
|
||||
- 恢复:新代次未完整写完不切换 head;重启只清理未来暂存行/退役标记。无变化不重复写事实。已提交旧代次至少保留15分钟,后台每次最多200行清理,不使用可能误删当前版本的TTL。定期范围清理保留新增事实 head。
|
||||
|
||||
兼容回归使用真实隔离 Mongo,覆盖旧记录、权限范围、脏记录覆盖、旧工作者、删除、指纹变化、重启恢复、无变化零写入和分页正文恢复。纯贡献浮点时长允许因加法顺序产生 `1e-8` 内误差,整数/身份严格一致;HTTP合成对照仍要求业务JSON精确等价。完整任务检查 `verify-1794573005303254000/report.json` 全部所选检查通过,含浏览器、Mongo `--race --cache` 及查询服务故障恢复。此前运行因执行中源码变化而标为未完成,原记录保留。
|
||||
|
||||
限制:后台当前是有界组织快照计算+差量发布,并非已完成逐任务分区计算;单次构建限制64MiB原始BSON/250,000行,超限保留上一完整版本并报错,不截断数据,也不阻断标注作业。规模超过此预算仍需分区计算验证。事实尚未首次发布时兼容读取独立查询库原历史,不回扫生产权威历史。集群本轮没有Valkey,不能当作完整共享缓存容量证据。短两轮不等于30分钟三轮或200/500人验收。
|
||||
|
||||
|
||||
### 本轮同规模结果:冷构建仍未通过,尚无HTTP容量结论
|
||||
|
||||
`io-rca/scale-81-8-facts` 按原26,218任务、81会话、两轮及固定资源/IO预算实际执行。源码未变化;回填294.551秒,2次租约沿检查点恢复。统计事实及生产读表构建88.372秒后成功;随后产出报表快照续租在3秒预算内未完成,分片构建取消,失败状态清理也超时。4分钟就绪等待失败,测试总计565.97秒,尚未进入管理员HTTP等价和81人请求阶段。`coverageComplete=false`,没有 `comparison.json`;因此不能报告本轮p95或源库节省比例,也不能说新路径已达到2秒。
|
||||
|
||||
`slow-metadata-complete.json` 仅保留隔离库命令名称、扫描/CPU/存储与确认等待,不含正文或过滤条件。多条小控制写CPU约0.3~0.8ms,总等待14~21秒,`waitForWriteConcernDurationMillis`几乎覆盖总耗时(最高20,857ms)。这说明本轮明确卡在受限IO下的持久化确认,不能继续把所有超时都归为大范围历史读取或CPU不足;是否由检查点、journal或不同后台写入共同触发仍需单独分段验证,不直接认定某一种机制。
|
||||
|
||||
122次保护采样覆盖约689.5秒:节点CPU最高6%,物理盘平均等待最高约1.37ms、忙碌最高约7.74%;保护器未停测,最终健康采样无错误。这只能说明本轮未触发既有业务健康/IO保护,不代表已证明对所有客户作业零影响。原业务、客户记录、资源与读路由均未改动。
|
||||
|
||||
下一步先处理独立查询库冷构建的写入节奏、共享控制写和失败恢复:记录每类后台写入/确认等待、检查点及限速窗口,减少重复写和突发积压;保持多数提交和版本隔离,不靠延长HTTP超时、降低写持久性或扩大IO限额掩盖问题。完成后再运行同规模两轮;任何更长时间或更大资源对照需另行界定。之前扩大20轮被自动审批以共享节点持续负载风险拒绝,未执行。
|
||||
|
||||
本轮自有合成源Pod及两条策略已按UID清理,桥接按PID/可执行文件身份停止,配额CAS恢复1Pod/1CPU/1Gi。25Gi卷及持久标记保留,查询Pod仍按3600秒deadline退出,未延长;清理证据 `io-rca/fixture-6/cleanup.json`。本批三项功能实现与本地回归完成,完整规模性能验收仍不通过,不准备生产发布。
|
||||
|
||||
@ -205,6 +205,46 @@ func (s *server) countHumanPoolCandidates(w http.ResponseWriter, r *http.Request
|
||||
}
|
||||
count := 0
|
||||
batches := map[string]*domain.HumanBatchRecord{}
|
||||
tryGroups := true
|
||||
if mode, ok := s.store.(interface{ UsesQueryReplicaCountRead(context.Context) bool }); ok {
|
||||
tryGroups = mode.UsesQueryReplicaCountRead(r.Context())
|
||||
}
|
||||
if grouped, yes := s.store.(interface {
|
||||
VisitHumanPoolCountGroups(context.Context, string, domain.HumanItemQuery, func(domain.HumanItemRecord, int) error) (bool, error)
|
||||
}); yes && tryGroups {
|
||||
principal, _ := currentPrincipal(r)
|
||||
if principal != nil && !annotationManager(principal) {
|
||||
roles, err := s.annotationMembershipsForUser(r.Context(), s.requestOrganizationID(r), query.ProjectID, principal.User.ID)
|
||||
if err != nil {
|
||||
renderError(w, 500, err)
|
||||
return 0, r, false
|
||||
}
|
||||
for _, role := range roles {
|
||||
if len(role.DrainTaskIDs) > 0 {
|
||||
query.CountIdentitySensitive = true
|
||||
}
|
||||
}
|
||||
}
|
||||
handled, err := grouped.VisitHumanPoolCountGroups(r.Context(), s.requestOrganizationID(r), query, func(item domain.HumanItemRecord, weight int) error {
|
||||
ctx, err := s.withHumanListReads(r.Context(), s.requestOrganizationID(r), query.ProjectID, []domain.HumanItemRecord{item})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
r = r.WithContext(ctx)
|
||||
_, visible, err := s.filterHumanPoolItem(r, item, pool, query, batches)
|
||||
if visible {
|
||||
count += weight
|
||||
}
|
||||
return err
|
||||
})
|
||||
if err != nil {
|
||||
renderError(w, 500, err)
|
||||
return 0, r, false
|
||||
}
|
||||
if handled {
|
||||
return count, r, true
|
||||
}
|
||||
}
|
||||
err := s.store.VisitHumanPoolCountCandidates(r.Context(), s.requestOrganizationID(r), query, func(items []domain.HumanItemRecord) error {
|
||||
ctx, err := s.withHumanListReads(r.Context(), s.requestOrganizationID(r), query.ProjectID, items)
|
||||
if err != nil {
|
||||
|
||||
54
internal/api/query_fact_routes_test.go
Normal file
54
internal/api/query_fact_routes_test.go
Normal file
@ -0,0 +1,54 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/store"
|
||||
"droboticflow/internal/workforcestats"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type queryFactRoutesProbe struct {
|
||||
store.Store
|
||||
pageCalls, groupCalls, rawCalls int
|
||||
item domain.HumanItemRecord
|
||||
}
|
||||
|
||||
func (p *queryFactRoutesProbe) LoadWorkforceReturnPage(ctx context.Context, org string, q workforcestats.Query, limit, offset int) ([]domain.WorkforceReturnDetail, int, error) {
|
||||
p.pageCalls++
|
||||
return []domain.WorkforceReturnDetail{{ID: "selected-page", Problems: []domain.ReviewProblem{}}}, 37, nil
|
||||
}
|
||||
func (p *queryFactRoutesProbe) LoadWorkforceStats(ctx context.Context, org string, q workforcestats.Query) (workforcestats.Snapshot, error) {
|
||||
p.rawCalls++
|
||||
return p.Store.LoadWorkforceStats(ctx, org, q)
|
||||
}
|
||||
func (p *queryFactRoutesProbe) VisitHumanPoolCountGroups(ctx context.Context, org string, q domain.HumanItemQuery, visit func(domain.HumanItemRecord, int) error) (bool, error) {
|
||||
p.groupCalls++
|
||||
return true, visit(p.item, 7)
|
||||
}
|
||||
func TestWorkforceReturnsUsesAuthorizedPagedReader(t *testing.T) {
|
||||
f := sharedAcceptanceFixture(t)
|
||||
p := &queryFactRoutesProbe{Store: f.store, item: f.item}
|
||||
f.handler = NewHandler(p, nil)
|
||||
r := f.request("GET", "/api/workforce-returns?from=2026-11-01T00:00:00Z&to=2026-11-13T00:00:00Z&limit=20&offset=20", f.reviewer, nil)
|
||||
var body struct {
|
||||
Items []domain.WorkforceReturnDetail
|
||||
Total int
|
||||
}
|
||||
_ = json.Unmarshal(r.Body.Bytes(), &body)
|
||||
if r.Code != 200 || p.pageCalls != 1 || p.rawCalls != 0 || body.Total != 37 || len(body.Items) != 1 {
|
||||
t.Fatalf("page reader not used: status=%d page=%d raw=%d body=%s", r.Code, p.pageCalls, p.rawCalls, r.Body.String())
|
||||
}
|
||||
}
|
||||
func TestHumanPoolCountsUsesWeightedFactsWithCurrentAuthorization(t *testing.T) {
|
||||
f := sharedAcceptanceFixture(t)
|
||||
p := &queryFactRoutesProbe{Store: f.store, item: f.item}
|
||||
f.handler = NewHandler(p, nil)
|
||||
r := f.request("GET", "/api/human-item-counts?pluginId=annotation-qa", f.reviewer, nil)
|
||||
var body struct{ Counts map[string]int }
|
||||
_ = json.Unmarshal(r.Body.Bytes(), &body)
|
||||
if r.Code != 200 || p.groupCalls == 0 || body.Counts["available"] != 7 {
|
||||
t.Fatalf("groups not used: %d calls=%d %s", r.Code, p.groupCalls, r.Body.String())
|
||||
}
|
||||
}
|
||||
@ -1,6 +1,7 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strconv"
|
||||
@ -88,12 +89,22 @@ func (s *server) handleAnnotationWorkloadSummary(w http.ResponseWriter, r *http.
|
||||
return
|
||||
}
|
||||
q.GroupTemplate = true
|
||||
d, err := s.store.LoadWorkforceStats(r.Context(), s.requestOrganizationID(r), q)
|
||||
var report workforcestats.Report
|
||||
if prepared, ok := s.store.(interface {
|
||||
LoadWorkforceReport(context.Context, string, workforcestats.Query) (workforcestats.Report, error)
|
||||
}); ok {
|
||||
report, err = prepared.LoadWorkforceReport(r.Context(), s.requestOrganizationID(r), q)
|
||||
} else {
|
||||
var d workforcestats.Snapshot
|
||||
d, err = s.store.LoadWorkforceStats(r.Context(), s.requestOrganizationID(r), q)
|
||||
if err == nil {
|
||||
report = workforcestats.Build(d, q)
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
renderError(w, 500, err)
|
||||
return
|
||||
}
|
||||
report := workforcestats.Build(d, q)
|
||||
items := make([]domain.AnnotationWorkloadAggregate, 0, len(report.Items))
|
||||
for _, m := range report.Items {
|
||||
items = append(items, domain.AnnotationWorkloadAggregate{
|
||||
@ -155,6 +166,17 @@ func (s *server) handleWorkforceReturns(w http.ResponseWriter, r *http.Request)
|
||||
}
|
||||
}
|
||||
q.Details = true
|
||||
if paged, ok := s.store.(interface {
|
||||
LoadWorkforceReturnPage(context.Context, string, workforcestats.Query, int, int) ([]domain.WorkforceReturnDetail, int, error)
|
||||
}); ok {
|
||||
items, total, e := paged.LoadWorkforceReturnPage(r.Context(), s.requestOrganizationID(r), q, limit, offset)
|
||||
if e != nil {
|
||||
renderError(w, 500, e)
|
||||
return
|
||||
}
|
||||
renderJSON(w, 200, map[string]any{"items": items, "total": total, "from": q.From.Format(time.RFC3339Nano), "to": q.To.Format(time.RFC3339Nano)})
|
||||
return
|
||||
}
|
||||
d, err := s.store.LoadWorkforceStats(r.Context(), s.requestOrganizationID(r), q)
|
||||
if err != nil {
|
||||
renderError(w, 500, err)
|
||||
|
||||
@ -359,6 +359,9 @@ type HumanItemCursor struct {
|
||||
}
|
||||
|
||||
type HumanItemQuery struct {
|
||||
// Server-only: draining supplier grants require per-task identity checks.
|
||||
CountIdentitySensitive bool
|
||||
|
||||
InheritedUnassigned bool
|
||||
NeedsReworkRouting bool
|
||||
|
||||
|
||||
@ -11,6 +11,11 @@ import (
|
||||
// This is an explicit derived-data index operation, never API startup behavior.
|
||||
// Production execution requires its own reviewed migration/index authorization.
|
||||
func (s *MongoStore) EnsureQueryReadModelIndexes(ctx context.Context) error {
|
||||
if s.queryReplicaFactStore {
|
||||
if err := s.ensureWorkforceFactIndexes(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
_, err := s.queryData().Indexes().CreateOne(ctx, mongo.IndexModel{Keys: bson.D{{Key: "generation", Value: 1}, {Key: "organizationId", Value: 1}, {Key: "itemId", Value: 1}}, Options: options.Index().SetName("query_model_rows_v1")})
|
||||
return err
|
||||
}
|
||||
@ -76,6 +81,7 @@ func (s *MongoStore) pruneQueryModelScopes(ctx context.Context, l queryModelLeas
|
||||
for org, projects := range scopes {
|
||||
keep[queryModelKey("production", org, projects)] = true
|
||||
keep[queryModelKey("search", org, nil)] = true
|
||||
keep[workforceFactKey(org)] = true
|
||||
}
|
||||
unset := bson.M{}
|
||||
for key := range st.Heads {
|
||||
|
||||
219
internal/store/query_pool_buckets.go
Normal file
219
internal/store/query_pool_buckets.go
Normal file
@ -0,0 +1,219 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/queryserving"
|
||||
"errors"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/mongo"
|
||||
"go.mongodb.org/mongo-driver/mongo/options"
|
||||
"go.mongodb.org/mongo-driver/mongo/readconcern"
|
||||
"time"
|
||||
)
|
||||
|
||||
const queryPoolBuckets = "queryPoolCountBuckets"
|
||||
const queryPoolBucketSchema = "pool-count/v1"
|
||||
|
||||
// These are the inputs of the API's visibility predicate, not an authorization
|
||||
// result. Identity-sensitive grant draining uses the legacy per-record reader.
|
||||
var poolBucketFields = []string{"organizationId", "pluginId", "projectId", "batchId", "supplierId", "processingScope", "processingProfile", "workgroupId", "vendorId", "currentStageId", "status", "assignedUserId", "latestAuthorUserId", "acceptanceClaim", "reviewRound", "claimedAt", "inheritedRework"}
|
||||
|
||||
func poolBucketProjection() bson.M {
|
||||
p := bson.M{"_id": 1}
|
||||
for _, field := range poolBucketFields {
|
||||
p[field] = 1
|
||||
}
|
||||
return p
|
||||
}
|
||||
func poolBucket(doc bson.M) (string, bson.M, error) {
|
||||
projected := bson.M{}
|
||||
for _, field := range poolBucketFields {
|
||||
if value, ok := doc[field]; ok {
|
||||
projected[field] = value
|
||||
}
|
||||
}
|
||||
raw, err := bson.Marshal(projected)
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
var typed domain.HumanItemRecord
|
||||
if err = bson.Unmarshal(raw, &typed); err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
out := bson.M{}
|
||||
for _, field := range poolBucketFields {
|
||||
value := doc[field]
|
||||
switch field {
|
||||
case "reviewRound":
|
||||
value = int(supplierNumber(value))
|
||||
case "inheritedRework":
|
||||
value = value == true
|
||||
case "acceptanceClaim":
|
||||
if value == nil {
|
||||
value = nil
|
||||
}
|
||||
default:
|
||||
text, _ := value.(string)
|
||||
value = text
|
||||
}
|
||||
out[field] = value
|
||||
}
|
||||
return queryserving.Key(queryPoolBucketSchema, out), out, nil
|
||||
}
|
||||
|
||||
type poolBucketDelta struct {
|
||||
doc bson.M
|
||||
count int64
|
||||
}
|
||||
|
||||
func addPoolBucketDelta(deltas map[string]*poolBucketDelta, doc bson.M, count int64) error {
|
||||
if len(doc) == 0 {
|
||||
return nil
|
||||
}
|
||||
key, row, err := poolBucket(doc)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
v := deltas[key]
|
||||
if v == nil {
|
||||
v = &poolBucketDelta{doc: row}
|
||||
deltas[key] = v
|
||||
}
|
||||
v.count += count
|
||||
return nil
|
||||
}
|
||||
func (q *QueryReplica) writePoolBucketDeltas(ctx mongo.SessionContext, deltas map[string]*poolBucketDelta) error {
|
||||
models := []mongo.WriteModel{}
|
||||
for key, delta := range deltas {
|
||||
if delta.count != 0 {
|
||||
models = append(models, mongo.NewUpdateOneModel().SetFilter(bson.M{"_id": key}).SetUpdate(bson.M{"$setOnInsert": delta.doc, "$inc": bson.M{"count": delta.count}}).SetUpsert(true))
|
||||
}
|
||||
}
|
||||
if len(models) == 0 {
|
||||
return nil
|
||||
}
|
||||
_, err := q.Target.db.Collection(queryPoolBuckets).BulkWrite(ctx, models)
|
||||
return err
|
||||
}
|
||||
func (q *QueryReplica) ensurePoolBucketIndexes(ctx context.Context) error {
|
||||
_, err := q.Target.db.Collection(queryPoolBuckets).Indexes().CreateOne(ctx, mongo.IndexModel{Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "pluginId", Value: 1}, {Key: "currentStageId", Value: 1}, {Key: "projectId", Value: 1}}, Options: options.Index().SetName("pool_bucket_scope_v1")})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Existing read stores without buckets retain the original reader until an
|
||||
// explicit rebuild. A newly empty replica starts counting in its transactions.
|
||||
n, err := q.Target.humanItems.CountDocuments(ctx, bson.M{}, options.Count().SetLimit(1))
|
||||
if err != nil || n != 0 {
|
||||
return err
|
||||
}
|
||||
_, err = q.control().UpdateOne(ctx, bson.M{"_id": "state"}, bson.M{"$setOnInsert": bson.M{"poolBucketSchema": queryPoolBucketSchema, "poolBucketEpoch": int64(0)}}, options.Update().SetUpsert(true))
|
||||
return err
|
||||
}
|
||||
|
||||
// VisitHumanPoolCountGroups never supplies an authorized total. Every group is
|
||||
// checked using current server memberships/batches. Dirty authoritative rows
|
||||
// subtract their old group and add their current row in this same request.
|
||||
func (s *MongoStore) VisitHumanPoolCountGroups(ctx context.Context, org string, q domain.HumanItemQuery, visit func(domain.HumanItemRecord, int) error) (bool, error) {
|
||||
target, err := s.poolReplica(ctx)
|
||||
if err != nil {
|
||||
return true, err
|
||||
}
|
||||
if target == nil {
|
||||
return false, nil
|
||||
}
|
||||
if humanPoolFiltered(q) || q.BlindAnnotationActorID != "" || q.NeedsReworkRouting || q.InheritedUnassigned || q.WorkflowID != "" || q.RunID != "" || q.LegacyTaskID != "" || q.AnnotationView != "" || q.IDs != nil || q.CountIdentitySensitive {
|
||||
return false, nil
|
||||
}
|
||||
b, err := s.countReplicaBarrier(ctx, org)
|
||||
if err != nil {
|
||||
return true, err
|
||||
}
|
||||
type weightedRow struct {
|
||||
domain.HumanItemRecord `bson:",inline"`
|
||||
Count int `bson:"count"`
|
||||
}
|
||||
rows := []weightedRow{}
|
||||
session, e := target.db.Client().StartSession()
|
||||
if e != nil {
|
||||
return true, e
|
||||
}
|
||||
defer session.EndSession(ctx)
|
||||
capacity := errors.New("count groups exceed request budget")
|
||||
incompatible := errors.New("count bucket writer compatibility changed")
|
||||
_, err = session.WithTransaction(ctx, func(sc mongo.SessionContext) (any, error) {
|
||||
rows = nil
|
||||
var state struct {
|
||||
Schema string `bson:"poolBucketSchema"`
|
||||
FactEpoch int64 `bson:"factEpoch"`
|
||||
BucketEpoch int64 `bson:"poolBucketEpoch"`
|
||||
}
|
||||
if e := target.db.Collection(queryReplicaControl).FindOne(sc, bson.M{"_id": "state"}).Decode(&state); e != nil {
|
||||
return nil, e
|
||||
}
|
||||
if state.Schema != queryPoolBucketSchema || state.FactEpoch != state.BucketEpoch {
|
||||
return nil, incompatible
|
||||
}
|
||||
filter := humanItemMongoFilter(org, q)
|
||||
filter["count"] = bson.M{"$gt": 0}
|
||||
cur, e := target.db.Collection(queryPoolBuckets).Find(sc, filter, options.Find().SetMaxTime(8*time.Second).SetLimit(10001))
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
e = cur.All(sc, &rows)
|
||||
cur.Close(sc)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
if len(rows) > 10000 {
|
||||
return nil, capacity
|
||||
}
|
||||
if b == nil || len(b.ids) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
ids := make([]string, 0, len(b.ids))
|
||||
for id := range b.ids {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
oldFilter := bson.M{"$and": bson.A{humanItemMongoFilter(org, q), bson.M{"id": bson.M{"$in": ids}}}}
|
||||
old, e := target.humanItems.Find(sc, oldFilter, options.Find().SetProjection(poolBucketProjection()))
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
defer old.Close(sc)
|
||||
for old.Next(sc) {
|
||||
var row domain.HumanItemRecord
|
||||
if e = old.Decode(&row); e != nil {
|
||||
return nil, e
|
||||
}
|
||||
rows = append(rows, weightedRow{HumanItemRecord: row, Count: -1})
|
||||
}
|
||||
return nil, old.Err()
|
||||
}, options.Transaction().SetReadConcern(readconcern.Snapshot()))
|
||||
if errors.Is(err, capacity) || errors.Is(err, incompatible) {
|
||||
return false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return true, err
|
||||
}
|
||||
for _, row := range rows {
|
||||
if e := ctx.Err(); e != nil {
|
||||
return true, e
|
||||
}
|
||||
if e := visit(row.HumanItemRecord, row.Count); e != nil {
|
||||
return true, e
|
||||
}
|
||||
}
|
||||
if b == nil {
|
||||
return true, ctx.Err()
|
||||
}
|
||||
err = s.visitReplicaPendingCounts(ctx, org, q, b, func(rows []domain.HumanItemRecord) error {
|
||||
for _, row := range rows {
|
||||
if e := visit(row, 1); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return true, err
|
||||
}
|
||||
95
internal/store/query_pool_buckets_local_test.go
Normal file
95
internal/store/query_pool_buckets_local_test.go
Normal file
@ -0,0 +1,95 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"droboticflow/internal/domain"
|
||||
"fmt"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/event"
|
||||
"os"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestQueryReplicaPoolBucketsIncrementalAndPendingLocal(t *testing.T) {
|
||||
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
|
||||
t.Skip("owned synthetic replica required")
|
||||
}
|
||||
var rowReads atomic.Int64
|
||||
monitor := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
|
||||
if e.CommandName == "find" {
|
||||
if v, ok := e.Command.Lookup("find").StringValueOK(); ok && v == "humanItems" {
|
||||
rowReads.Add(1)
|
||||
}
|
||||
}
|
||||
}}
|
||||
source, _, replica := newLocalReplica(t, nil, monitor)
|
||||
ctx := t.Context()
|
||||
for i := 0; i < 120; i++ {
|
||||
if err := source.UpsertHumanItem(ctx, domain.HumanItemRecord{ID: fmt.Sprint("i", i), OrganizationID: "o", ProjectID: "p", BatchID: "b", PluginID: "annotation-qa", CurrentStageID: "quality_review", Status: domain.HumanItemReady, LatestAuthorUserID: fmt.Sprint("u", i%3)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
finishLocalReplica(t, replica)
|
||||
if err := source.ConfigureQueryReplicaReports(replica.Target, time.Minute); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
source.EnableQueryReplicaCountReads(true)
|
||||
count := func(q domain.HumanItemQuery) (int, int) {
|
||||
t.Helper()
|
||||
n, groups := 0, 0
|
||||
handled, err := source.VisitHumanPoolCountGroups(WithQueryReplicaCountRead(ctx), "o", q, func(row domain.HumanItemRecord, weight int) error { n += weight; groups++; return nil })
|
||||
if err != nil || !handled {
|
||||
t.Fatalf("group reader unavailable: %v/%v", handled, err)
|
||||
}
|
||||
return n, groups
|
||||
}
|
||||
q := domain.HumanItemQuery{PluginID: "annotation-qa"}
|
||||
rowReads.Store(0)
|
||||
if n, groups := count(q); n != 120 || groups != 3 {
|
||||
t.Fatalf("count=%d groups=%d", n, groups)
|
||||
}
|
||||
if rowReads.Load() != 0 {
|
||||
t.Fatal("clean count loaded individual query items")
|
||||
}
|
||||
if _, err := source.humanItems.UpdateOne(ctx, bson.M{"id": "i0"}, bson.M{"$set": bson.M{"assignedUserId": "claimant", "status": "claimed"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
q.UnassignedOnly = true
|
||||
q.Statuses = []domain.HumanItemStatus{domain.HumanItemReady}
|
||||
if n, _ := count(q); n != 119 {
|
||||
t.Fatalf("pending overlay=%d", n)
|
||||
}
|
||||
if _, err := replica.RunOnce(ctx, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n, _ := count(q); n != 119 {
|
||||
t.Fatalf("committed overlay=%d", n)
|
||||
}
|
||||
if _, err := source.humanItems.DeleteOne(ctx, bson.M{"id": "i1"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := 0; i < 10; i++ {
|
||||
if _, err := replica.RunOnce(ctx, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if n, _ := count(q); n != 118 {
|
||||
t.Fatalf("deletion left a counted bucket: %d", n)
|
||||
}
|
||||
// An old projector can still update the ordinary replica after rollback.
|
||||
// Its missing bucket epoch invalidates the fast path, not permissions.
|
||||
if _, err := replica.control().UpdateOne(ctx, bson.M{"_id": "state"}, bson.M{"$inc": bson.M{"factEpoch": int64(1)}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
used, e := source.VisitHumanPoolCountGroups(WithQueryReplicaCountRead(ctx), "o", q, func(domain.HumanItemRecord, int) error { t.Fatal("mixed old projector used stale buckets"); return nil })
|
||||
if e != nil || used {
|
||||
t.Fatal(used, e)
|
||||
}
|
||||
q.CountIdentitySensitive = true
|
||||
handled, err := source.VisitHumanPoolCountGroups(WithQueryReplicaCountRead(ctx), "o", q, func(domain.HumanItemRecord, int) error { t.Fatal("identity grant was grouped"); return nil })
|
||||
if handled || err != nil {
|
||||
t.Fatal("identity fallback changed", handled, err)
|
||||
}
|
||||
}
|
||||
@ -40,7 +40,7 @@ func (b *QueryReadBuilder) Close() { b.reader.Close() }
|
||||
|
||||
// WaitReady is for explicit prewarming and isolated acceptance, not HTTP reads.
|
||||
func (b *QueryReadBuilder) WaitReady(ctx context.Context, org string, projects []string) error {
|
||||
tick := time.NewTicker(100 * time.Millisecond)
|
||||
tick := time.NewTicker(500 * time.Millisecond)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
productionHead, production := b.store.readQueryModelHead(ctx, queryModelKey("production", org, projects), 5*time.Minute)
|
||||
@ -53,7 +53,12 @@ func (b *QueryReadBuilder) WaitReady(ctx context.Context, org string, projects [
|
||||
health, e := b.reader.Inspect(ctx, org, projects)
|
||||
output = e == nil && health.Available
|
||||
}
|
||||
if production && search && output {
|
||||
workforce := true
|
||||
if b.store.queryReplicaFactStore {
|
||||
_, workforce, err = b.store.workforceHead(ctx, org)
|
||||
workforce = workforce && err == nil
|
||||
}
|
||||
if production && search && output && workforce {
|
||||
return nil
|
||||
}
|
||||
select {
|
||||
@ -126,6 +131,11 @@ func (b *QueryReadBuilder) rebuild(ctx context.Context, l queryModelLease, org s
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if b.store.queryReplicaFactStore {
|
||||
if err = b.store.rebuildWorkforceFacts(buildCtx, l, org, source.Revision); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
head.SourceRevision = source.Revision
|
||||
head.GlobalRevision = source.Global
|
||||
if err = b.store.publishQueryModelHead(ctx, l, key, head); err != nil {
|
||||
@ -346,6 +356,7 @@ func (b *QueryReadBuilder) run(ctx context.Context, started chan<- struct{}) err
|
||||
refreshScopes := true
|
||||
lastScopeRead := time.Time{}
|
||||
lastCleanup := time.Now()
|
||||
lastWorkforceCleanup := time.Now()
|
||||
for {
|
||||
b.mu.Lock()
|
||||
all, allSearch := b.all, b.allSearch
|
||||
@ -396,7 +407,16 @@ func (b *QueryReadBuilder) run(ctx context.Context, started chan<- struct{}) err
|
||||
log.Print("query_search_build result=retry category=source_or_publication")
|
||||
}
|
||||
}
|
||||
outputBusy := false
|
||||
if state, e := b.store.OutputSnapshotState(ctx); e != nil {
|
||||
outputBusy = true
|
||||
} else {
|
||||
outputBusy = state.Owner != "" && time.Now().Before(state.Until) && state.ActiveEpoch == state.Epoch
|
||||
}
|
||||
for org := range pendingStats {
|
||||
if outputBusy {
|
||||
continue
|
||||
}
|
||||
ids, exists := scopes[org]
|
||||
if !exists {
|
||||
delete(pendingStats, org)
|
||||
@ -422,9 +442,16 @@ func (b *QueryReadBuilder) run(ctx context.Context, started chan<- struct{}) err
|
||||
if time.Since(lastCleanup) > time.Minute {
|
||||
cleanup, finish := context.WithTimeout(ctx, 5*time.Second)
|
||||
_ = b.store.cleanQueryModelOrphans(cleanup)
|
||||
|
||||
finish()
|
||||
lastCleanup = time.Now()
|
||||
}
|
||||
if b.store.queryReplicaFactStore && time.Since(lastWorkforceCleanup) > 5*time.Second {
|
||||
cleanup, finish := context.WithTimeout(ctx, 2*time.Second)
|
||||
_ = b.store.maintainWorkforceVersions(cleanup, l)
|
||||
finish()
|
||||
lastWorkforceCleanup = time.Now()
|
||||
}
|
||||
// Prewarm due generations even when no browser is connected.
|
||||
for org, ids := range scopes {
|
||||
if len(ids) > 0 {
|
||||
|
||||
@ -71,6 +71,7 @@ func (s *MongoStore) SharedQueryCache() queryserving.Cache {
|
||||
}
|
||||
|
||||
type QueryModelHead struct {
|
||||
FactEpoch int64 `bson:"factEpoch,omitempty" json:"factEpoch,omitempty"`
|
||||
SourceRevision int64 `bson:"sourceRevision" json:"sourceRevision"`
|
||||
Revision int64 `bson:"revision" json:"revision"`
|
||||
GlobalRevision int64 `bson:"globalRevision" json:"globalRevision"`
|
||||
|
||||
@ -169,11 +169,17 @@ func (q *QueryReplica) apply(ctx mongo.SessionContext, n string, records []query
|
||||
seen[key] = true
|
||||
ids = append(ids, r.Document["_id"])
|
||||
}
|
||||
cur, e := c.Find(ctx, bson.M{"_id": bson.M{"$in": ids}}, options.Find().SetProjection(bson.M{"_queryReplica": 1}))
|
||||
projection := bson.M{"_queryReplica": 1}
|
||||
if n == "humanItems" {
|
||||
projection = poolBucketProjection()
|
||||
projection["_queryReplica"] = 1
|
||||
}
|
||||
cur, e := c.Find(ctx, bson.M{"_id": bson.M{"$in": ids}}, options.Find().SetProjection(projection))
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
current := map[string]bson.M{}
|
||||
oldPoolRows := map[string]bson.M{}
|
||||
for cur.Next(ctx) {
|
||||
var d bson.M
|
||||
if e = cur.Decode(&d); e != nil {
|
||||
@ -181,6 +187,9 @@ func (q *QueryReplica) apply(ctx mongo.SessionContext, n string, records []query
|
||||
return e
|
||||
}
|
||||
current[querySourceIdentity(n, d["_id"])] = deletionDoc(d["_queryReplica"])
|
||||
if n == "humanItems" {
|
||||
oldPoolRows[querySourceIdentity(n, d["_id"])] = d
|
||||
}
|
||||
}
|
||||
e = cur.Err()
|
||||
cur.Close(ctx)
|
||||
@ -188,6 +197,7 @@ func (q *QueryReplica) apply(ctx mongo.SessionContext, n string, records []query
|
||||
return e
|
||||
}
|
||||
models := make([]mongo.WriteModel, 0, len(records))
|
||||
deltas := map[string]*poolBucketDelta{}
|
||||
for i, r := range records {
|
||||
m := current[querySourceIdentity(n, r.Document["_id"])]
|
||||
records[i].TargetSourceID = m["sourceId"]
|
||||
@ -197,6 +207,14 @@ func (q *QueryReplica) apply(ctx mongo.SessionContext, n string, records []query
|
||||
}
|
||||
records[i].TargetSourceID = r.SourceID
|
||||
records[i].TargetRevision = r.Revision
|
||||
if n == "humanItems" {
|
||||
if err := addPoolBucketDelta(deltas, oldPoolRows[querySourceIdentity(n, r.Document["_id"])], -1); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := addPoolBucketDelta(deltas, r.Document, 1); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
models = append(models, mongo.NewReplaceOneModel().SetFilter(bson.M{"_id": r.Document["_id"]}).SetReplacement(r.Document).SetUpsert(true))
|
||||
}
|
||||
if len(models) == 0 {
|
||||
@ -206,7 +224,10 @@ func (q *QueryReplica) apply(ctx mongo.SessionContext, n string, records []query
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
_, e = q.control().UpdateOne(ctx, bson.M{"_id": "state"}, bson.M{"$inc": bson.M{"factEpoch": int64(1), "collectionEpoch." + n: int64(1)}}, options.Update().SetUpsert(true))
|
||||
if e = q.writePoolBucketDeltas(ctx, deltas); e != nil {
|
||||
return e
|
||||
}
|
||||
_, e = q.control().UpdateOne(ctx, bson.M{"_id": "state"}, bson.M{"$inc": bson.M{"factEpoch": int64(1), "poolBucketEpoch": int64(1), "collectionEpoch." + n: int64(1)}}, options.Update().SetUpsert(true))
|
||||
return e
|
||||
}
|
||||
|
||||
|
||||
@ -96,13 +96,31 @@ func (q *QueryReplica) ReconcileDeleteStep(ctx context.Context) (bool, error) {
|
||||
}
|
||||
}
|
||||
e = q.transaction(ctx, func(sc mongo.SessionContext) error {
|
||||
deltas := map[string]*poolBucketDelta{}
|
||||
for _, r := range absent {
|
||||
if _, e := q.Target.db.Collection(i.Collection).DeleteOne(sc, bson.M{"_id": r["_id"], "_queryReplica.sourceId": deletionDoc(r["_queryReplica"])["sourceId"]}); e != nil {
|
||||
filter := bson.M{"_id": r["_id"], "_queryReplica.sourceId": deletionDoc(r["_queryReplica"])["sourceId"]}
|
||||
var old bson.M
|
||||
if i.Collection == "humanItems" {
|
||||
e := q.Target.humanItems.FindOne(sc, filter, options.FindOne().SetProjection(poolBucketProjection())).Decode(&old)
|
||||
if e != nil && e != mongo.ErrNoDocuments {
|
||||
return e
|
||||
}
|
||||
}
|
||||
result, e := q.Target.db.Collection(i.Collection).DeleteOne(sc, filter)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if result.DeletedCount == 1 && i.Collection == "humanItems" {
|
||||
if err := addPoolBucketDelta(deltas, old, -1); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if e := q.writePoolBucketDeltas(sc, deltas); e != nil {
|
||||
return e
|
||||
}
|
||||
if len(absent) > 0 {
|
||||
if _, e := q.control().UpdateOne(sc, bson.M{"_id": "state"}, bson.M{"$inc": bson.M{"factEpoch": int64(1), "collectionEpoch." + i.Collection: int64(1)}}, options.Update().SetUpsert(true)); e != nil {
|
||||
if _, e := q.control().UpdateOne(sc, bson.M{"_id": "state"}, bson.M{"$inc": bson.M{"factEpoch": int64(1), "poolBucketEpoch": int64(1), "collectionEpoch." + i.Collection: int64(1)}}, options.Update().SetUpsert(true)); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
|
||||
@ -10,6 +10,12 @@ import (
|
||||
// EnsureTargetIndexes operates only on the explicitly supplied independent
|
||||
// read database. No business migrations/bootstrapping run on this database.
|
||||
func (q *QueryReplica) EnsureTargetIndexes(ctx context.Context) error {
|
||||
if err := q.Target.ensureWorkforceFactIndexes(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := q.ensurePoolBucketIndexes(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, name := range replicaCollections() {
|
||||
models := []mongo.IndexModel{{Keys: bson.D{{Key: "id", Value: 1}}, Options: options.Index().SetName("query_id_v1")}, {Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "projectId", Value: 1}, {Key: "createdAt", Value: -1}, {Key: "id", Value: -1}}, Options: options.Index().SetName("query_scope_v1")}}
|
||||
if name == "annotationReviews" || name == "annotationRevisions" || name == "sliceRevisions" || name == "annotationWorklogEvents" {
|
||||
|
||||
@ -284,5 +284,6 @@ func (s *MongoStore) VerifyQueryReplicaTarget(ctx context.Context) error {
|
||||
if state.Schema != queryReplicaSchema || state.Binding.SourceDatabase == "" || state.Binding.Namespace != os.Getenv("QUERY_SERVING_NAMESPACE") || state.Binding.Namespace == "" {
|
||||
return ErrQueryReplicaNotReady
|
||||
}
|
||||
s.queryReplicaFactStore = true
|
||||
return nil
|
||||
}
|
||||
|
||||
658
internal/store/workforce_facts.go
Normal file
658
internal/store/workforce_facts.go
Normal file
@ -0,0 +1,658 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"reflect"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/queryserving"
|
||||
"droboticflow/internal/workforcestats"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/mongo"
|
||||
"go.mongodb.org/mongo-driver/mongo/options"
|
||||
"go.mongodb.org/mongo-driver/mongo/readconcern"
|
||||
"go.mongodb.org/mongo-driver/mongo/writeconcern"
|
||||
)
|
||||
|
||||
const workforceFactCollection = "queryWorkforceFacts"
|
||||
const workforceFactSchema = "workforce-facts/v1"
|
||||
|
||||
type workforceBuildBudgetKey struct{}
|
||||
type workforceBuildBudget struct{ bytes, rows int }
|
||||
|
||||
var errWorkforceBuildBudget = errors.New("workforce fact source exceeds isolated build budget")
|
||||
|
||||
func workforceBoundedRead(ctx context.Context, cur *mongo.Cursor, target any, budget *workforceBuildBudget) error {
|
||||
slice := reflect.ValueOf(target).Elem()
|
||||
for cur.Next(ctx) {
|
||||
budget.bytes += len(cur.Current)
|
||||
budget.rows++
|
||||
if budget.bytes > 64<<20 || budget.rows > 250000 {
|
||||
return errWorkforceBuildBudget
|
||||
}
|
||||
v := reflect.New(slice.Type().Elem())
|
||||
if err := cur.Decode(v.Interface()); err != nil {
|
||||
return err
|
||||
}
|
||||
slice.Set(reflect.Append(slice, v.Elem()))
|
||||
}
|
||||
return cur.Err()
|
||||
}
|
||||
func workforceFactKey(org string) string { return queryModelKey("workforce-facts", org, nil) }
|
||||
func (s *MongoStore) ensureWorkforceFactIndexes(ctx context.Context) error {
|
||||
indexes := []mongo.IndexModel{
|
||||
{Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "retiredAt", Value: 1}, {Key: "toEpoch", Value: 1}}, Options: options.Index().SetName("workforce_fact_retirement_v1")},
|
||||
{Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "mode", Value: 1}, {Key: "groupTemplate", Value: 1}, {Key: "userId", Value: 1}, {Key: "at", Value: -1}, {Key: "order", Value: 1}}, Options: options.Index().SetName("workforce_fact_user_v1")},
|
||||
{Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "mode", Value: 1}, {Key: "groupTemplate", Value: 1}, {Key: "supplierId", Value: 1}, {Key: "projectId", Value: 1}, {Key: "at", Value: -1}}, Options: options.Index().SetName("workforce_fact_scope_v1")},
|
||||
{Keys: bson.D{{Key: "organizationId", Value: 1}, {Key: "fromEpoch", Value: 1}, {Key: "toEpoch", Value: 1}}, Options: options.Index().SetName("workforce_fact_versions_v1")},
|
||||
}
|
||||
_, err := s.db.Collection(workforceFactCollection).Indexes().CreateMany(ctx, indexes)
|
||||
return err
|
||||
}
|
||||
func (s *MongoStore) workforceHead(ctx context.Context, org string) (QueryModelHead, bool, error) {
|
||||
var state struct {
|
||||
Heads map[string]QueryModelHead `bson:"heads"`
|
||||
Revision int64 `bson:"productionRevision"`
|
||||
Observed time.Time `bson:"observedAt"`
|
||||
}
|
||||
key := workforceFactKey(org)
|
||||
err := s.queryControl().FindOne(ctx, bson.M{"_id": "global"}, options.FindOne().SetProjection(bson.M{"heads." + key: 1, "productionRevision": 1, "observedAt": 1})).Decode(&state)
|
||||
if errors.Is(err, mongo.ErrNoDocuments) {
|
||||
return QueryModelHead{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return QueryModelHead{}, false, err
|
||||
}
|
||||
h, exists := state.Heads[key]
|
||||
if !exists {
|
||||
return h, false, nil
|
||||
}
|
||||
unchanged := h.SourceRevision == state.Revision && !state.Observed.IsZero() && time.Since(state.Observed) >= 0 && time.Since(state.Observed) <= 5*time.Second
|
||||
if h.Collection != workforceFactCollection || h.FactEpoch < 1 || h.Generation != workforceFactSchema || time.Since(h.AsOf) < 0 || (!unchanged && time.Since(h.AsOf) > dashboardStatsCacheTTL) {
|
||||
return h, true, ErrQueryReplicaNotReady
|
||||
}
|
||||
return h, true, nil
|
||||
}
|
||||
func workforceVersionFilter(org string, epoch int64) bson.M {
|
||||
return bson.M{"organizationId": org, "fromEpoch": bson.M{"$lte": epoch}, "$or": bson.A{bson.M{"toEpoch": bson.M{"$exists": false}}, bson.M{"toEpoch": bson.M{"$gt": epoch}}}}
|
||||
}
|
||||
func (s *MongoStore) workforceFactTransaction(ctx context.Context, l queryModelLease, fn func(mongo.SessionContext) error) error {
|
||||
session, err := s.db.Client().StartSession()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer session.EndSession(ctx)
|
||||
_, err = session.WithTransaction(ctx, func(sc mongo.SessionContext) (any, error) {
|
||||
if err := s.renewQueryModelLease(sc, l); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return nil, fn(sc)
|
||||
}, options.Transaction().SetReadConcern(readconcern.Snapshot()).SetWriteConcern(writeconcern.Majority()))
|
||||
return err
|
||||
}
|
||||
|
||||
// A failed stage never changes the committed view. On restart remove only
|
||||
// future versions and undo future retirements, fenced by the builder lease.
|
||||
func (s *MongoStore) restoreWorkforceStage(ctx context.Context, l queryModelLease, org string, head int64) error {
|
||||
c := s.db.Collection(workforceFactCollection)
|
||||
for _, inserted := range []bool{true, false} {
|
||||
field := "toEpoch"
|
||||
if inserted {
|
||||
field = "fromEpoch"
|
||||
}
|
||||
for {
|
||||
cur, err := c.Find(ctx, bson.M{"organizationId": org, field: bson.M{"$gt": head}}, options.Find().SetProjection(bson.M{"_id": 1}).SetLimit(200))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var ids []bson.M
|
||||
err = cur.All(ctx, &ids)
|
||||
cur.Close(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
break
|
||||
}
|
||||
values := bson.A{}
|
||||
for _, d := range ids {
|
||||
values = append(values, d["_id"])
|
||||
}
|
||||
err = s.workforceFactTransaction(ctx, l, func(sc mongo.SessionContext) error {
|
||||
if inserted {
|
||||
_, err := c.DeleteMany(sc, bson.M{"_id": bson.M{"$in": values}, "fromEpoch": bson.M{"$gt": head}})
|
||||
return err
|
||||
}
|
||||
_, err := c.UpdateMany(sc, bson.M{"_id": bson.M{"$in": values}, "toEpoch": bson.M{"$gt": head}}, bson.M{"$unset": bson.M{"toEpoch": "", "retiredAt": "", "expireAt": ""}})
|
||||
return err
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (s *MongoStore) rebuildWorkforceFacts(ctx context.Context, l queryModelLease, org string, revision int64) error {
|
||||
c := s.db.Collection(workforceFactCollection)
|
||||
key := workforceFactKey(org)
|
||||
var control struct {
|
||||
Heads map[string]QueryModelHead `bson:"heads"`
|
||||
}
|
||||
err := s.queryControl().FindOne(ctx, bson.M{"_id": "global"}, options.FindOne().SetProjection(bson.M{"heads." + key: 1})).Decode(&control)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
oldHead := control.Heads[key]
|
||||
epoch := oldHead.FactEpoch + 1
|
||||
if err = s.restoreWorkforceStage(ctx, l, org, oldHead.FactEpoch); err != nil {
|
||||
return err
|
||||
}
|
||||
var d workforcestats.Snapshot
|
||||
session, err := s.db.Client().StartSession()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer session.EndSession(ctx)
|
||||
_, err = session.WithTransaction(ctx, func(sc mongo.SessionContext) (any, error) {
|
||||
var e error
|
||||
d, e = s.LoadWorkforceStats(context.WithValue(sc, workforceBuildBudgetKey{}, &workforceBuildBudget{}), org, workforcestats.Query{Mode: domain.RunModeAll})
|
||||
return nil, e
|
||||
}, options.Transaction().SetReadConcern(readconcern.Snapshot()))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
facts := workforcestats.Materialize(d)
|
||||
type previous struct {
|
||||
ID string `bson:"_id"`
|
||||
Key string `bson:"key"`
|
||||
Hash string `bson:"hash"`
|
||||
}
|
||||
cur, err := c.Find(ctx, workforceVersionFilter(org, oldHead.FactEpoch), options.Find().SetProjection(bson.M{"_id": 1, "key": 1, "hash": 1}))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
old := map[string]previous{}
|
||||
for cur.Next(ctx) {
|
||||
var row previous
|
||||
if err = cur.Decode(&row); err != nil {
|
||||
break
|
||||
}
|
||||
old[row.Key] = row
|
||||
}
|
||||
if err == nil {
|
||||
err = cur.Err()
|
||||
}
|
||||
cur.Close(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
models := []mongo.WriteModel{}
|
||||
changed := false
|
||||
flush := func() error {
|
||||
if len(models) == 0 {
|
||||
return nil
|
||||
}
|
||||
batch := models
|
||||
models = nil
|
||||
return s.workforceFactTransaction(ctx, l, func(sc mongo.SessionContext) error {
|
||||
_, err := c.BulkWrite(sc, batch, options.BulkWrite().SetOrdered(true))
|
||||
return err
|
||||
})
|
||||
}
|
||||
for _, fact := range facts {
|
||||
// Keep comments/text in authority. Source hashes and exact problem identities
|
||||
// fence page hydration; no client draft or media is stored in derived facts.
|
||||
raw, err := json.Marshal(fact)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
hash := queryserving.Key(workforceFactSchema, string(raw))
|
||||
prev, found := old[fact.Key]
|
||||
delete(old, fact.Key)
|
||||
if found && prev.Hash == hash {
|
||||
continue
|
||||
}
|
||||
changed = true
|
||||
if found {
|
||||
models = append(models, mongo.NewUpdateOneModel().SetFilter(bson.M{"_id": prev.ID}).SetUpdate(bson.M{"$set": bson.M{"toEpoch": epoch, "retiredAt": time.Now().UTC()}}))
|
||||
}
|
||||
encoded, err := bson.Marshal(fact)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var doc bson.M
|
||||
if err = bson.Unmarshal(encoded, &doc); err != nil {
|
||||
return err
|
||||
}
|
||||
at, err := time.Parse(time.RFC3339Nano, fact.CreatedAt)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
doc["_id"], doc["organizationId"], doc["fromEpoch"], doc["hash"], doc["at"] = queryserving.Key(workforceFactSchema, org, fact.Key, epoch), org, epoch, hash, at
|
||||
doc["atSeconds"], doc["atNanos"] = at.Unix(), at.Nanosecond()
|
||||
doc["metric"] = workforcestats.MetricCounterValues(fact.Metric)
|
||||
if fact.Detail != nil {
|
||||
raw, e := json.Marshal(fact.Detail)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
var detail bson.M
|
||||
if e = bson.UnmarshalExtJSON(raw, false, &detail); e != nil {
|
||||
return e
|
||||
}
|
||||
doc["detail"] = detail
|
||||
doc["return"] = bson.M{"id": fact.Detail.ID, "stage": fact.Detail.Stage, "taskIds": fact.Detail.TaskIDs, "metrics": detail["metrics"]}
|
||||
}
|
||||
doc["order"] = ""
|
||||
if fact.Detail != nil {
|
||||
doc["order"] = fact.Detail.ID + fact.Detail.UserID + fact.Detail.SupplierID
|
||||
}
|
||||
models = append(models, mongo.NewInsertOneModel().SetDocument(doc))
|
||||
if len(models) >= 200 {
|
||||
if err = flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, prev := range old {
|
||||
changed = true
|
||||
models = append(models, mongo.NewUpdateOneModel().SetFilter(bson.M{"_id": prev.ID}).SetUpdate(bson.M{"$set": bson.M{"toEpoch": epoch, "retiredAt": time.Now().UTC()}}))
|
||||
if len(models) >= 200 {
|
||||
if err = flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if err = flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
if !changed && oldHead.FactEpoch > 0 {
|
||||
epoch = oldHead.FactEpoch
|
||||
}
|
||||
head := QueryModelHead{Generation: workforceFactSchema, Collection: workforceFactCollection, FactEpoch: epoch, Rows: len(facts), SourceRevision: revision, AsOf: time.Now().UTC()}
|
||||
if err = s.publishQueryModelHead(ctx, l, key, head); err != nil {
|
||||
return err
|
||||
}
|
||||
// Garbage collection never expires a partially staged retirement. It
|
||||
// deletes only committed versions after a long in-flight reader grace.
|
||||
cleanup, stop := context.WithTimeout(ctx, 2*time.Second)
|
||||
defer stop()
|
||||
if e := s.cleanWorkforceVersions(cleanup, l, org, epoch); e != nil {
|
||||
log.Print("workforce_fact_cleanup result=retry")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (s *MongoStore) cleanWorkforceVersions(ctx context.Context, l queryModelLease, org string, epoch int64) error {
|
||||
c := s.db.Collection(workforceFactCollection)
|
||||
cur, err := c.Find(ctx, bson.M{"organizationId": org, "toEpoch": bson.M{"$lte": epoch}, "retiredAt": bson.M{"$lt": time.Now().UTC().Add(-15 * time.Minute)}}, options.Find().SetProjection(bson.M{"_id": 1}).SetLimit(200))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var docs []bson.M
|
||||
err = cur.All(ctx, &docs)
|
||||
cur.Close(ctx)
|
||||
if err != nil || len(docs) == 0 {
|
||||
return err
|
||||
}
|
||||
ids := bson.A{}
|
||||
for _, d := range docs {
|
||||
ids = append(ids, d["_id"])
|
||||
}
|
||||
return s.workforceFactTransaction(ctx, l, func(sc mongo.SessionContext) error {
|
||||
_, err := c.DeleteMany(sc, bson.M{"_id": bson.M{"$in": ids}, "toEpoch": bson.M{"$lte": epoch}})
|
||||
return err
|
||||
})
|
||||
}
|
||||
|
||||
func (s *MongoStore) maintainWorkforceVersions(ctx context.Context, l queryModelLease) error {
|
||||
var control struct {
|
||||
Heads map[string]QueryModelHead `bson:"heads"`
|
||||
}
|
||||
if err := s.queryControl().FindOne(ctx, bson.M{"_id": "global"}).Decode(&control); err != nil {
|
||||
return err
|
||||
}
|
||||
cur, err := s.organizations.Find(ctx, bson.M{}, options.Find().SetProjection(bson.M{"id": 1}))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
for cur.Next(ctx) {
|
||||
var row struct {
|
||||
ID string `bson:"id"`
|
||||
}
|
||||
if err = cur.Decode(&row); err != nil {
|
||||
return err
|
||||
}
|
||||
if h, ok := control.Heads[workforceFactKey(row.ID)]; ok && h.FactEpoch > 0 {
|
||||
if err = s.cleanWorkforceVersions(ctx, l, row.ID, h.FactEpoch); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return cur.Err()
|
||||
}
|
||||
|
||||
// BSON dates are millisecond precision. The indexed date predicate is broad;
|
||||
// these two integer fields retain exact historical RFC3339Nano boundaries.
|
||||
func workforceTimeBound(seconds, nanos any, bound time.Time, lower bool) bson.M {
|
||||
major, minor := "$lt", "$lt"
|
||||
if lower {
|
||||
major, minor = "$gt", "$gte"
|
||||
}
|
||||
return bson.M{"$or": bson.A{bson.M{major: bson.A{seconds, bound.Unix()}}, bson.M{"$and": bson.A{bson.M{"$eq": bson.A{seconds, bound.Unix()}}, bson.M{minor: bson.A{nanos, bound.Nanosecond()}}}}}}
|
||||
}
|
||||
func workforceFactFilter(org string, h QueryModelHead, q workforcestats.Query) bson.M {
|
||||
f := workforceVersionFilter(org, h.FactEpoch)
|
||||
f["mode"] = workforcestats.ModeVariant(q.Mode)
|
||||
f["groupTemplate"] = q.GroupTemplate
|
||||
if !q.From.IsZero() || !q.To.IsZero() {
|
||||
at := bson.M{}
|
||||
if !q.From.IsZero() {
|
||||
at["$gte"] = q.From.Truncate(time.Millisecond)
|
||||
}
|
||||
if !q.To.IsZero() {
|
||||
at["$lt"] = q.To.Add(time.Millisecond).Truncate(time.Millisecond)
|
||||
}
|
||||
f["at"] = at
|
||||
exact := bson.A{}
|
||||
if !q.From.IsZero() {
|
||||
exact = append(exact, workforceTimeBound("$atSeconds", "$atNanos", q.From, true))
|
||||
}
|
||||
if !q.To.IsZero() {
|
||||
exact = append(exact, workforceTimeBound("$atSeconds", "$atNanos", q.To, false))
|
||||
}
|
||||
f["$expr"] = bson.M{"$and": exact}
|
||||
}
|
||||
if q.ProjectID != "" {
|
||||
f["projectId"] = q.ProjectID
|
||||
}
|
||||
if q.SupplierID != "" {
|
||||
f["supplierId"] = q.SupplierID
|
||||
}
|
||||
if q.MatchUser || q.UserID != "" {
|
||||
f["userId"] = q.UserID
|
||||
}
|
||||
if q.Role != "" {
|
||||
f["role"] = q.Role
|
||||
}
|
||||
if q.TemplateVersion != nil {
|
||||
f["templateVersion"] = *q.TemplateVersion
|
||||
}
|
||||
clauses := bson.A{f}
|
||||
for field, allowed := range map[string]map[string]bool{"projectId": q.AllowedProjects, "supplierId": q.AllowedSuppliers} {
|
||||
if allowed != nil {
|
||||
ids := []string{}
|
||||
for id, on := range allowed {
|
||||
if on {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
}
|
||||
clauses = append(clauses, bson.M{field: bson.M{"$in": ids}})
|
||||
}
|
||||
}
|
||||
return bson.M{"$and": clauses}
|
||||
}
|
||||
|
||||
// Legacy formal reviews used scope-filtered event/suppressor context in the
|
||||
// original loader. Resolve those small descriptors BEFORE supplier visibility.
|
||||
func workforceFactReadStages(org string, h QueryModelHead, q workforcestats.Query) mongo.Pipeline {
|
||||
base := q
|
||||
base.SupplierID = ""
|
||||
base.AllowedSuppliers = nil
|
||||
conditions := func(variable string, actor bool) bson.A {
|
||||
out := bson.A{bson.M{"$literal": true}}
|
||||
if q.ProjectID != "" {
|
||||
out = append(out, bson.M{"$eq": bson.A{variable + ".projectId", q.ProjectID}})
|
||||
}
|
||||
if q.AllowedProjects != nil {
|
||||
ids := []string{}
|
||||
for id, on := range q.AllowedProjects {
|
||||
if on {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
}
|
||||
out = append(out, bson.M{"$in": bson.A{variable + ".projectId", ids}})
|
||||
}
|
||||
if actor {
|
||||
if q.MatchUser || q.UserID != "" {
|
||||
out = append(out, bson.M{"$eq": bson.A{variable + ".userId", q.UserID}})
|
||||
}
|
||||
if q.SupplierID != "" {
|
||||
out = append(out, bson.M{"$eq": bson.A{variable + ".supplierId", q.SupplierID}})
|
||||
}
|
||||
} else {
|
||||
if q.SupplierID != "" {
|
||||
out = append(out, bson.M{"$in": bson.A{q.SupplierID, variable + ".suppliers"}})
|
||||
}
|
||||
if !q.From.IsZero() {
|
||||
out = append(out, workforceTimeBound(variable+".seconds", variable+".nanos", q.From, true))
|
||||
}
|
||||
if !q.To.IsZero() {
|
||||
out = append(out, workforceTimeBound(variable+".seconds", variable+".nanos", q.To, false))
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
events := bson.M{"$filter": bson.M{"input": bson.M{"$ifNull": bson.A{"$legacy.events", bson.A{}}}, "as": "e", "cond": bson.M{"$and": conditions("$$e", true)}}}
|
||||
suppressed := bson.M{"$filter": bson.M{"input": bson.M{"$ifNull": bson.A{"$legacy.suppressors", bson.A{}}}, "as": "op", "cond": bson.M{"$and": conditions("$$op", false)}}}
|
||||
legacy := bson.M{"$eq": bson.A{"$sourceKind", "review"}}
|
||||
chosen := func(field string) any {
|
||||
return bson.M{"$cond": bson.A{legacy, bson.M{"$ifNull": bson.A{"$legacyEvent." + field, "$legacy." + field}}, "$" + field}}
|
||||
}
|
||||
first := workforceFactFilter(org, h, base)
|
||||
if q.Details {
|
||||
first = bson.M{"$and": bson.A{first, bson.M{"kind": "return"}}}
|
||||
}
|
||||
stages := mongo.Pipeline{{{Key: "$match", Value: first}}, {{Key: "$set", Value: bson.M{"legacyEvent": bson.M{"$first": events}, "legacyHidden": bson.M{"$gt": bson.A{bson.M{"$size": suppressed}, 0}}}}}, {{Key: "$set", Value: bson.M{"supplierId": chosen("supplierId"), "supplierName": chosen("supplierName")}}}, {{Key: "$set", Value: bson.M{"effectiveUnknown": bson.M{"$cond": bson.A{legacy, bson.M{"$or": bson.A{bson.M{"$eq": bson.A{"$userId", ""}}, bson.M{"$eq": bson.A{"$supplierId", ""}}}}, false}}, "order": bson.M{"$concat": bson.A{"$return.id", "$userId", "$supplierId"}}}}}}
|
||||
|
||||
match := bson.M{"legacyHidden": false}
|
||||
parts := bson.A{match}
|
||||
if q.SupplierID != "" {
|
||||
parts = append(parts, bson.M{"supplierId": q.SupplierID})
|
||||
}
|
||||
if q.AllowedSuppliers != nil {
|
||||
ids := []string{}
|
||||
for id, on := range q.AllowedSuppliers {
|
||||
if on {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
}
|
||||
parts = append(parts, bson.M{"supplierId": bson.M{"$in": ids}})
|
||||
}
|
||||
return append(stages, bson.D{{Key: "$match", Value: bson.M{"$and": parts}}})
|
||||
}
|
||||
|
||||
func workforceMetricPipeline(q workforcestats.Query, totals bool) mongo.Pipeline {
|
||||
id := any(nil)
|
||||
if !totals {
|
||||
template := any(bson.M{"$literal": 0})
|
||||
if q.GroupTemplate {
|
||||
template = "$templateVersion"
|
||||
}
|
||||
day := any(bson.M{"$literal": ""})
|
||||
if q.GroupBy == "day" {
|
||||
day = "$day"
|
||||
}
|
||||
id = bson.M{"userId": "$userId", "supplierId": "$supplierId", "templateVersion": template, "date": day}
|
||||
}
|
||||
group := bson.M{"_id": id, "eventCount": bson.M{"$sum": "$eventCount"}}
|
||||
project := bson.M{"_id": 0, "eventCount": 1}
|
||||
for _, name := range workforcestats.MetricCounterNames() {
|
||||
group[name] = bson.M{"$sum": "$metric." + name}
|
||||
project[name] = 1
|
||||
}
|
||||
if !totals {
|
||||
for _, name := range []string{"userId", "supplierId", "templateVersion", "date"} {
|
||||
project[name] = "$_id." + name
|
||||
}
|
||||
for _, name := range []string{"userName", "supplierName", "role"} {
|
||||
group[name] = bson.M{"$first": "$" + name}
|
||||
project[name] = 1
|
||||
}
|
||||
}
|
||||
for _, stage := range []string{"quality", "acceptance"} {
|
||||
when := bson.M{"$and": bson.A{bson.M{"$eq": bson.A{"$kind", "return"}}, bson.M{"$eq": bson.A{"$return.stage", stage}}}}
|
||||
choose := func(yes, no any) any { return bson.M{"$cond": bson.A{when, yes, no}} }
|
||||
group[stage+"Decisions"] = bson.M{"$addToSet": choose("$return.id", nil)}
|
||||
group[stage+"Tasks"] = bson.M{"$addToSet": choose("$return.taskIds", bson.A{})}
|
||||
group[stage+"Unknown"] = bson.M{"$addToSet": choose(bson.M{"$cond": bson.A{bson.M{"$gt": bson.A{"$return.metrics.unknownDetailCount", 0}}, "$return.id", nil}}, nil)}
|
||||
out := bson.M{"decisionCount": bson.M{"$size": bson.M{"$setDifference": bson.A{"$" + stage + "Decisions", bson.A{nil}}}}, "unknownDetailCount": bson.M{"$size": bson.M{"$setDifference": bson.A{"$" + stage + "Unknown", bson.A{nil}}}}, "taskCount": bson.M{"$size": bson.M{"$reduce": bson.M{"input": "$" + stage + "Tasks", "initialValue": bson.A{}, "in": bson.M{"$setUnion": bson.A{"$$value", "$$this"}}}}}}
|
||||
for _, name := range []string{"problemCount", "sliceCount", "videoSeconds", "unlocatedCount"} {
|
||||
sumKey := stage + "_" + name
|
||||
group[sumKey] = bson.M{"$sum": choose("$return.metrics."+name, 0)}
|
||||
out[name] = "$" + sumKey
|
||||
}
|
||||
project[stage+"Returns"] = out
|
||||
}
|
||||
return mongo.Pipeline{{{Key: "$group", Value: group}}, {{Key: "$project", Value: project}}}
|
||||
}
|
||||
func (s *MongoStore) loadWorkforceFactReport(ctx context.Context, org string, q workforcestats.Query) (workforcestats.Report, bool, error) {
|
||||
// Explicit empty-user queries distinguish Mongo missing/null/empty values.
|
||||
// Keep their exact legacy loader until provenance is separately indexed.
|
||||
if q.MatchUser && q.UserID == "" {
|
||||
return workforcestats.Report{}, false, nil
|
||||
}
|
||||
h, exists, err := s.workforceHead(ctx, org)
|
||||
if err != nil || !exists {
|
||||
return workforcestats.Report{}, exists, err
|
||||
}
|
||||
key := queryserving.Key(workforceFactSchema, s.db.Name(), org, h.FactEpoch, q)
|
||||
if raw, ok := workforceReportCache.get(key, dashboardStatsCacheTTL); ok {
|
||||
var report workforcestats.Report
|
||||
if json.Unmarshal(raw, &report) == nil {
|
||||
return report, true, nil
|
||||
}
|
||||
}
|
||||
if cache := s.SharedQueryCache(); cache != nil {
|
||||
if raw, ok := cache.Get(ctx, key); ok && len(raw) <= workforceReportMaxBytes {
|
||||
var report workforcestats.Report
|
||||
if json.Unmarshal(raw, &report) == nil {
|
||||
workforceReportCache.set(key, raw, dashboardStatsCacheTTL)
|
||||
return report, true, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
pipeline := append(workforceFactReadStages(org, h, q), bson.D{{Key: "$project", Value: bson.M{"kind": 1, "metric": 1, "return": 1, "eventCount": 1, "userId": 1, "userName": 1, "supplierId": 1, "supplierName": 1, "role": 1, "templateVersion": 1, "day": 1}}}, bson.D{{Key: "$facet", Value: bson.M{"items": workforceMetricPipeline(q, false), "totals": workforceMetricPipeline(q, true)}}})
|
||||
cur, err := s.db.Collection(workforceFactCollection).Aggregate(ctx, pipeline, options.Aggregate().SetMaxTime(8*time.Second))
|
||||
if err != nil {
|
||||
return workforcestats.Report{}, true, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
var rows []struct {
|
||||
Items []bson.M `bson:"items"`
|
||||
Totals []bson.M `bson:"totals"`
|
||||
}
|
||||
if err = cur.All(ctx, &rows); err != nil {
|
||||
return workforcestats.Report{}, true, err
|
||||
}
|
||||
report := workforcestats.Report{Items: []domain.DashboardWorkforceMetric{}}
|
||||
if len(rows) > 0 {
|
||||
for _, m := range rows[0].Items {
|
||||
var v domain.DashboardWorkforceMetric
|
||||
raw, _ := json.Marshal(m)
|
||||
if err = json.Unmarshal(raw, &v); err != nil {
|
||||
return report, true, err
|
||||
}
|
||||
report.Items = append(report.Items, v)
|
||||
}
|
||||
if len(rows[0].Totals) > 0 {
|
||||
raw, _ := json.Marshal(rows[0].Totals[0])
|
||||
if err = json.Unmarshal(raw, &report.Totals); err != nil {
|
||||
return report, true, err
|
||||
}
|
||||
report.EventCount = int(supplierNumber(rows[0].Totals[0]["eventCount"]))
|
||||
}
|
||||
}
|
||||
sort.Slice(report.Items, func(i, j int) bool {
|
||||
a, b := report.Items[i], report.Items[j]
|
||||
return fmt.Sprintf("%s/%s/%s/%s/%d", a.Date, a.SupplierName, a.UserName, a.UserID, a.TemplateVersion) < fmt.Sprintf("%s/%s/%s/%s/%d", b.Date, b.SupplierName, b.UserName, b.UserID, b.TemplateVersion)
|
||||
})
|
||||
if raw, err := json.Marshal(report); err == nil && len(raw) <= workforceReportMaxBytes {
|
||||
workforceReportCache.set(key, raw, dashboardStatsCacheTTL)
|
||||
if cache := s.SharedQueryCache(); cache != nil {
|
||||
cache.Put(ctx, key, raw, dashboardStatsCacheTTL)
|
||||
}
|
||||
}
|
||||
return report, true, nil
|
||||
}
|
||||
func (s *MongoStore) loadWorkforceFactPage(ctx context.Context, org string, q workforcestats.Query, limit, offset int) ([]workforcePageFact, int, bool, error) {
|
||||
if q.MatchUser && q.UserID == "" {
|
||||
return nil, 0, false, nil
|
||||
}
|
||||
h, exists, err := s.workforceHead(ctx, org)
|
||||
if err != nil || !exists {
|
||||
return nil, 0, exists, err
|
||||
}
|
||||
selected := bson.M{"_id": 1, "atSeconds": 1, "atNanos": 1, "order": 1, "userId": 1, "supplierId": 1, "sourceKind": 1, "effectiveUnknown": 1, "returnId": "$return.id"}
|
||||
pipeline := append(workforceFactReadStages(org, h, q), bson.D{{Key: "$match", Value: bson.M{"kind": "return"}}}, bson.D{{Key: "$project", Value: selected}}, bson.D{{Key: "$facet", Value: bson.M{"total": bson.A{bson.M{"$count": "count"}}, "page": bson.A{bson.M{"$sort": bson.D{{Key: "atSeconds", Value: -1}, {Key: "atNanos", Value: -1}, {Key: "order", Value: 1}, {Key: "returnId", Value: 1}, {Key: "userId", Value: 1}, {Key: "supplierId", Value: 1}, {Key: "sourceKind", Value: 1}}}, bson.M{"$skip": offset}, bson.M{"$limit": limit}}}}})
|
||||
cur, err := s.db.Collection(workforceFactCollection).Aggregate(ctx, pipeline, options.Aggregate().SetMaxTime(8*time.Second))
|
||||
if err != nil {
|
||||
return nil, 0, true, err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
type selection struct {
|
||||
ID string `bson:"_id"`
|
||||
Supplier string `bson:"supplierId"`
|
||||
Unknown bool `bson:"effectiveUnknown"`
|
||||
Kind string `bson:"sourceKind"`
|
||||
}
|
||||
var rows []struct {
|
||||
Page []selection `bson:"page"`
|
||||
Total []struct {
|
||||
Count int `bson:"count"`
|
||||
} `bson:"total"`
|
||||
}
|
||||
if err = cur.All(ctx, &rows); err != nil {
|
||||
return nil, 0, true, err
|
||||
}
|
||||
out := []workforcePageFact{}
|
||||
count := 0
|
||||
if len(rows) == 0 {
|
||||
return out, count, true, nil
|
||||
}
|
||||
if len(rows[0].Total) > 0 {
|
||||
count = rows[0].Total[0].Count
|
||||
}
|
||||
if len(rows[0].Page) == 0 {
|
||||
return out, count, true, nil
|
||||
}
|
||||
ids := []string{}
|
||||
for _, row := range rows[0].Page {
|
||||
ids = append(ids, row.ID)
|
||||
}
|
||||
detailCursor, err := s.db.Collection(workforceFactCollection).Find(ctx, bson.M{"_id": bson.M{"$in": ids}}, options.Find().SetProjection(bson.M{"_id": 1, "detail": 1, "sourceKind": 1, "sourceHash": 1}))
|
||||
if err != nil {
|
||||
return nil, 0, true, err
|
||||
}
|
||||
defer detailCursor.Close(ctx)
|
||||
var details []struct {
|
||||
ID string `bson:"_id"`
|
||||
Data workforcePageFact `bson:",inline"`
|
||||
}
|
||||
if err = detailCursor.All(ctx, &details); err != nil {
|
||||
return nil, 0, true, err
|
||||
}
|
||||
byID := map[string]workforcePageFact{}
|
||||
for _, row := range details {
|
||||
byID[row.ID] = row.Data
|
||||
}
|
||||
for _, row := range rows[0].Page {
|
||||
detail, ok := byID[row.ID]
|
||||
if !ok {
|
||||
return nil, 0, true, ErrQueryReplicaNotReady
|
||||
}
|
||||
if row.Kind == "review" {
|
||||
detail.Detail.SupplierID = row.Supplier
|
||||
detail.Detail.AttributionUnknown = row.Unknown
|
||||
}
|
||||
out = append(out, detail)
|
||||
}
|
||||
return out, count, true, nil
|
||||
}
|
||||
310
internal/store/workforce_facts_local_test.go
Normal file
310
internal/store/workforce_facts_local_test.go
Normal file
@ -0,0 +1,310 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/workforcestats"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/event"
|
||||
"os"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestWorkforceMaterializedFactsAndBoundedDetailsLocal(t *testing.T) {
|
||||
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
|
||||
t.Skip("owned synthetic replica required")
|
||||
}
|
||||
var mu sync.Mutex
|
||||
historyReads := 0
|
||||
factWrites := 0
|
||||
var textIDs []string
|
||||
sourceMonitor := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
|
||||
if e.CommandName != "find" {
|
||||
return
|
||||
}
|
||||
name, ok := e.Command.Lookup("find").StringValueOK()
|
||||
if !ok || name != "reviewOperations" {
|
||||
return
|
||||
}
|
||||
var filter bson.M
|
||||
_ = bson.Unmarshal(e.Command.Lookup("filter").Document(), &filter)
|
||||
var walk func(bson.M)
|
||||
walk = func(doc bson.M) {
|
||||
if in := deletionDoc(doc["_id"])["$in"]; in != nil {
|
||||
mu.Lock()
|
||||
for _, id := range in.(bson.A) {
|
||||
if text, ok := id.(string); ok {
|
||||
textIDs = append(textIDs, text)
|
||||
}
|
||||
}
|
||||
mu.Unlock()
|
||||
}
|
||||
if parts, ok := doc["$and"].(bson.A); ok {
|
||||
for _, part := range parts {
|
||||
walk(deletionDoc(part))
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(filter)
|
||||
}}
|
||||
targetMonitor := &event.CommandMonitor{Started: func(_ context.Context, e *event.CommandStartedEvent) {
|
||||
if e.CommandName == "insert" || e.CommandName == "update" || e.CommandName == "delete" {
|
||||
if v, ok := e.Command.Lookup(e.CommandName).StringValueOK(); ok && v == workforceFactCollection {
|
||||
mu.Lock()
|
||||
factWrites++
|
||||
mu.Unlock()
|
||||
}
|
||||
}
|
||||
if e.CommandName != "find" {
|
||||
return
|
||||
}
|
||||
name, ok := e.Command.Lookup("find").StringValueOK()
|
||||
if ok && (name == "reviewOperations" || name == "annotationWorklogEvents" || name == "annotationTasks" || name == "annotationReviews" || name == "sliceRevisions" || name == "annotationRevisions") {
|
||||
mu.Lock()
|
||||
historyReads++
|
||||
mu.Unlock()
|
||||
}
|
||||
}}
|
||||
source, target, replica := newLocalReplica(t, sourceMonitor, targetMonitor)
|
||||
ctx := t.Context()
|
||||
seedWorkforceStats(t, source)
|
||||
original, err := source.GetReviewOperation(ctx, "review-operation-test")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := 0; i < 52; i++ {
|
||||
op := *original
|
||||
op.ID = fmt.Sprintf("review-%03d", i)
|
||||
op.ReviewRound = i + 2
|
||||
op.CreatedAt = time.Date(2026, 9, 28, 13, 0, i, 0, time.UTC).Format(time.RFC3339)
|
||||
op.CompletedAt = op.CreatedAt
|
||||
ok, err := source.CompareAndSwapReviewOperation(ctx, op, 0)
|
||||
if err != nil || !ok {
|
||||
t.Fatal(ok, err)
|
||||
}
|
||||
}
|
||||
if err = source.UpsertAnnotationReview(ctx, domain.AnnotationReviewRecord{ID: "legacy-review", OrganizationID: "workforce-test", TaskID: "task", Revision: 4, ReviewRound: 1, Stage: domain.AnnotationReviewStageQuality, Decision: domain.AnnotationReviewRejected, CreatedAt: "2026-09-28T14:00:00Z", Comment: "legacy text survives", ProblemSliceIDs: []string{"old-slice"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
all, err := source.LoadWorkforceStats(ctx, "workforce-test", workforcestats.Query{Details: true, Mode: domain.RunModeAll})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
finishLocalReplica(t, replica)
|
||||
lease, ok, err := target.acquireQueryModelLease(ctx)
|
||||
if err != nil || !ok {
|
||||
t.Fatal(ok, err)
|
||||
}
|
||||
if err = target.rebuildWorkforceFacts(ctx, lease, "workforce-test", 0); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = source.ConfigureQueryReplicaReports(target, time.Minute); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
base := workforcestats.Query{UserID: "u", MatchUser: true, Details: true, From: time.Date(2026, 9, 28, 0, 0, 0, 0, time.UTC), To: time.Date(2026, 9, 29, 0, 0, 0, 0, time.UTC)}
|
||||
queries := []workforcestats.Query{base, base, base, base, base, base, base, base}
|
||||
queries[1].GroupBy = "day"
|
||||
queries[2].GroupTemplate = true
|
||||
queries[3].AllowedProjects = map[string]bool{"no": true}
|
||||
queries[4].AllowedSuppliers = map[string]bool{"no": true}
|
||||
queries[5].Role = string(domain.UserRoleQualityReviewer)
|
||||
queries[6].From = base.From.Add(12*time.Hour + time.Nanosecond)
|
||||
queries[7].To = base.From.Add(12*time.Hour + time.Nanosecond)
|
||||
mu.Lock()
|
||||
historyReads = 0
|
||||
mu.Unlock()
|
||||
for _, q := range queries {
|
||||
want := workforcestats.Build(all, q)
|
||||
q.Details = false
|
||||
got, err := source.LoadWorkforceReportCached(ctx, "workforce-test", q)
|
||||
want.Details = nil
|
||||
if err != nil || !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("summary mismatch q=%+v: %v\ngot=%+v\nwant=%+v", q, err, got, want)
|
||||
}
|
||||
}
|
||||
mu.Lock()
|
||||
reads := historyReads
|
||||
mu.Unlock()
|
||||
if reads != 0 {
|
||||
t.Fatalf("summary reloaded %d raw histories", reads)
|
||||
}
|
||||
for _, offset := range []int{0, 20, 40, 100} {
|
||||
q := base
|
||||
want := workforcestats.Build(all, q).Details
|
||||
start, end := workforcePageBounds(len(want), 20, offset)
|
||||
mu.Lock()
|
||||
textIDs = nil
|
||||
mu.Unlock()
|
||||
got, total, err := source.LoadWorkforceReturnPage(ctx, "workforce-test", q, 20, offset)
|
||||
if err != nil || total != len(want) || !reflect.DeepEqual(got, want[start:end]) {
|
||||
t.Fatalf("page offset=%d total=%d: %v\ngot=%+v\nwant=%+v", offset, total, err, got, want[start:end])
|
||||
}
|
||||
mu.Lock()
|
||||
n := len(textIDs)
|
||||
mu.Unlock()
|
||||
expectedIDs := 0
|
||||
for _, v := range want[start:end] {
|
||||
if v.ID != "legacy-review" {
|
||||
expectedIDs++
|
||||
}
|
||||
}
|
||||
if n != expectedIDs || n > 20 {
|
||||
t.Fatalf("hydrated %d operation texts for 20 rows", n)
|
||||
}
|
||||
}
|
||||
mu.Lock()
|
||||
factWrites = 0
|
||||
mu.Unlock()
|
||||
if err = target.rebuildWorkforceFacts(ctx, lease, "workforce-test", 0); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mu.Lock()
|
||||
writes := factWrites
|
||||
mu.Unlock()
|
||||
if writes != 0 {
|
||||
t.Fatalf("unchanged facts caused %d data writes", writes)
|
||||
}
|
||||
// Completed records changing after publication may not lend new attribution
|
||||
// or text to an old fact. The authority annotation itself is never rewritten.
|
||||
if _, err = source.db.Collection("reviewOperations").UpdateOne(ctx, bson.M{"_id": "review-051"}, bson.M{"$set": bson.M{"effects.1.beforeTask.supplierId": "changed"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err = source.LoadWorkforceReturnPage(ctx, "workforce-test", base, 20, 0); err == nil {
|
||||
t.Fatal("changed formal identity was hydrated into old facts")
|
||||
}
|
||||
// Simulate a crashed partial stage: old head stays readable and restart
|
||||
// removes only future versions, then republishes equal current facts.
|
||||
var head struct {
|
||||
Heads map[string]QueryModelHead `bson:"heads"`
|
||||
}
|
||||
if err = target.queryControl().FindOne(ctx, bson.M{"_id": "global"}).Decode(&head); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h := head.Heads[workforceFactKey("workforce-test")]
|
||||
c := target.db.Collection(workforceFactCollection)
|
||||
var active bson.M
|
||||
if err = c.FindOne(ctx, workforceVersionFilter("workforce-test", h.FactEpoch)).Decode(&active); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
future := bson.M{"_id": "crashed-future", "organizationId": "workforce-test", "fromEpoch": h.FactEpoch + 1, "key": "never published"}
|
||||
if _, err = c.InsertOne(ctx, future); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = c.UpdateOne(ctx, bson.M{"_id": active["_id"]}, bson.M{"$set": bson.M{"toEpoch": h.FactEpoch + 1}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = target.rebuildWorkforceFacts(ctx, lease, "workforce-test", 0); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = c.FindOne(ctx, bson.M{"_id": "crashed-future"}).Err(); err == nil {
|
||||
t.Fatal("future stage survived recovery")
|
||||
}
|
||||
// Derived rows never contain customer comment text.
|
||||
cur, err := c.Find(ctx, bson.M{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
for cur.Next(ctx) {
|
||||
var doc bson.M
|
||||
if err = cur.Decode(&doc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw, _ := json.Marshal(doc)
|
||||
if strings.Contains(string(raw), "保留意见") || strings.Contains(string(raw), "legacy text survives") {
|
||||
t.Fatal("customer text copied into facts")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkforceLegacyScopeVariantsStayEquivalentLocal(t *testing.T) {
|
||||
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
|
||||
t.Skip("owned synthetic replica required")
|
||||
}
|
||||
source, target, replica := newLocalReplica(t)
|
||||
ctx := t.Context()
|
||||
seedWorkforceStats(t, source)
|
||||
when := "2026-09-28T14:00:00Z"
|
||||
if err := source.UpsertAnnotationReview(ctx, domain.AnnotationReviewRecord{ID: "legacy-dependent", OrganizationID: "workforce-test", TaskID: "task", Revision: 4, Stage: domain.AnnotationReviewStageQuality, Decision: domain.AnnotationReviewRejected, CreatedAt: when, Comment: "old opinion"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := source.UpsertAnnotationWorklogEvent(ctx, domain.AnnotationWorklogEventRecord{ID: "legacy-supplier-event", OrganizationID: "workforce-test", TaskID: "task", ProjectID: "p", UserID: "reviewer", SupplierID: "reviewer-v", EventType: "quality_rejected", Revision: 4, CreatedAt: when}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
base, err := source.GetReviewOperation(ctx, "review-operation-test")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
op := *base
|
||||
op.ID = "formal-scope-suppressor"
|
||||
op.CreatedAt, op.CompletedAt = "2026-09-28T13:00:00Z", "2026-09-28T13:00:00Z"
|
||||
op.Effects = []domain.ReviewDecisionEffect{{Review: &domain.AnnotationReviewRecord{ID: "legacy-dependent", TaskID: "task", Revision: 4}}, {BeforeTask: &domain.AnnotationTaskRecord{ID: "task", SupplierID: "foreign-v"}}, {BeforeItem: &domain.HumanItemRecord{ID: "item", LegacyTaskID: "task", SupplierID: "foreign-v", LatestAuthorUserID: "u"}}}
|
||||
if ok, err := source.CompareAndSwapReviewOperation(ctx, op, 0); err != nil || !ok {
|
||||
t.Fatal(ok, err)
|
||||
}
|
||||
queries := []workforcestats.Query{{Details: true}, {Details: true, MatchUser: true, UserID: "u"}, {Details: true, SupplierID: "v"}, {Details: true, SupplierID: "foreign-v"}, {Details: true, SupplierID: "reviewer-v"}, {Details: true, AllowedProjects: map[string]bool{"other": true}}, {Details: true, GroupTemplate: true, MatchUser: true, UserID: "u"}}
|
||||
queries = append(queries, workforcestats.Query{Details: true, From: time.Date(2026, 9, 28, 13, 0, 0, 1, time.UTC)})
|
||||
wants := []workforcestats.Report{}
|
||||
for _, q := range queries {
|
||||
d, e := source.LoadWorkforceStats(ctx, "workforce-test", q)
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
wants = append(wants, workforcestats.Build(d, q))
|
||||
}
|
||||
finishLocalReplica(t, replica)
|
||||
l, ok, err := target.acquireQueryModelLease(ctx)
|
||||
if err != nil || !ok {
|
||||
t.Fatal(ok, err)
|
||||
}
|
||||
if err = target.rebuildWorkforceFacts(ctx, l, "workforce-test", 0); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = source.ConfigureQueryReplicaReports(target, time.Minute); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i, q := range queries {
|
||||
want := wants[i]
|
||||
details := want.Details
|
||||
want.Details = nil
|
||||
q.Details = false
|
||||
got, e := source.LoadWorkforceReportCached(ctx, "workforce-test", q)
|
||||
if e != nil || !reflect.DeepEqual(want, got) {
|
||||
t.Fatalf("legacy scope %d: %v\ngot=%+v\nwant=%+v", i, e, got, want)
|
||||
}
|
||||
q.Details = true
|
||||
page, total, e := source.LoadWorkforceReturnPage(ctx, "workforce-test", q, 50, 0)
|
||||
if e != nil || total != len(details) || !reflect.DeepEqual(page, details) {
|
||||
t.Fatalf("legacy page scope %d total=%d: %v\ngot=%+v\nwant=%+v", i, total, e, page, details)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkforceFactsSurviveConfiguredScopePruningLocal(t *testing.T) {
|
||||
if os.Getenv("QUERY_REPLICA_LOCAL_TEST") != "1" {
|
||||
t.Skip("owned synthetic replica required")
|
||||
}
|
||||
_, target, _ := newLocalReplica(t)
|
||||
ctx := t.Context()
|
||||
l, ok, err := target.acquireQueryModelLease(ctx)
|
||||
if err != nil || !ok {
|
||||
t.Fatal(ok, err)
|
||||
}
|
||||
head := QueryModelHead{Generation: workforceFactSchema, Collection: workforceFactCollection, FactEpoch: 1, AsOf: time.Now().UTC()}
|
||||
if err = target.publishQueryModelHead(ctx, l, workforceFactKey("o"), head); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = target.pruneQueryModelScopes(ctx, l, map[string][]string{"o": {"p"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, exists, err := target.workforceHead(ctx, "o")
|
||||
if err != nil || !exists {
|
||||
t.Fatalf("scope maintenance deleted published workforce facts: %v/%v", exists, err)
|
||||
}
|
||||
}
|
||||
@ -34,6 +34,17 @@ func (s *MongoStore) LoadWorkforceReportCached(ctx context.Context, org string,
|
||||
if err != nil {
|
||||
return workforcestats.Report{}, err
|
||||
}
|
||||
if target != nil {
|
||||
report, used, e := target.loadWorkforceFactReport(ctx, org, q)
|
||||
if e != nil || used {
|
||||
return report, e
|
||||
}
|
||||
} else if s.queryReplicaFactStore {
|
||||
report, used, e := s.loadWorkforceFactReport(ctx, org, q)
|
||||
if e != nil || used {
|
||||
return report, e
|
||||
}
|
||||
}
|
||||
epoch := int64(0)
|
||||
identity := fmt.Sprintf("%p:%s", s.db.Client(), s.db.Name())
|
||||
if target != nil {
|
||||
@ -91,3 +102,23 @@ func (s *MongoStore) LoadWorkforceReportCached(ctx context.Context, org string,
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
|
||||
// LoadWorkforceReport preserves uncached legacy summary behavior when read
|
||||
// replicas are disabled; the enabled replacement reads only published facts.
|
||||
func (s *MongoStore) LoadWorkforceReport(ctx context.Context, org string, q workforcestats.Query) (workforcestats.Report, error) {
|
||||
target, err := s.reportReplica(ctx)
|
||||
if err != nil {
|
||||
return workforcestats.Report{}, err
|
||||
}
|
||||
if target != nil {
|
||||
report, used, e := target.loadWorkforceFactReport(ctx, org, q)
|
||||
if e != nil || used {
|
||||
return report, e
|
||||
}
|
||||
}
|
||||
d, err := s.LoadWorkforceStats(ctx, org, q)
|
||||
if err != nil {
|
||||
return workforcestats.Report{}, err
|
||||
}
|
||||
return workforcestats.Build(d, q), nil
|
||||
}
|
||||
|
||||
161
internal/store/workforce_return_page.go
Normal file
161
internal/store/workforce_return_page.go
Normal file
@ -0,0 +1,161 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/workforcestats"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/mongo/options"
|
||||
"reflect"
|
||||
)
|
||||
|
||||
type workforcePageFact struct {
|
||||
Detail domain.WorkforceReturnDetail `bson:"detail"`
|
||||
SourceKind string `bson:"sourceKind"`
|
||||
SourceHash string `bson:"sourceHash"`
|
||||
}
|
||||
|
||||
func workforcePageBounds(count, limit, offset int) (int, int) {
|
||||
if offset > count {
|
||||
offset = count
|
||||
}
|
||||
end := offset + limit
|
||||
if end < offset || end > count {
|
||||
end = count
|
||||
}
|
||||
return offset, end
|
||||
}
|
||||
func (s *MongoStore) LoadWorkforceReturnPage(ctx context.Context, org string, q workforcestats.Query, limit, offset int) ([]domain.WorkforceReturnDetail, int, error) {
|
||||
q.Details = true
|
||||
target, err := s.reportReplica(ctx)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
if target == nil {
|
||||
d, err := s.LoadWorkforceStats(ctx, org, q)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
q.Details = true
|
||||
all := workforcestats.Build(d, q).Details
|
||||
start, end := workforcePageBounds(len(all), limit, offset)
|
||||
return all[start:end], len(all), nil
|
||||
}
|
||||
page, total, used, e := target.loadWorkforceFactPage(ctx, org, q, limit, offset)
|
||||
if e != nil {
|
||||
return nil, 0, e
|
||||
}
|
||||
if used {
|
||||
items, e := s.hydrateWorkforcePage(ctx, org, page)
|
||||
return items, total, e
|
||||
}
|
||||
// Compatibility before facts are first published: calculate only metadata on
|
||||
// the independent replica; hydrate authority text AFTER selecting this page.
|
||||
metadata := q
|
||||
metadata.Details = false
|
||||
d, err := target.LoadWorkforceStats(ctx, org, metadata)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
q.Details = true
|
||||
all := workforcestats.ReturnFacts(d, q)
|
||||
start, end := workforcePageBounds(len(all), limit, offset)
|
||||
page = make([]workforcePageFact, 0, end-start)
|
||||
for _, fact := range all[start:end] {
|
||||
page = append(page, workforcePageFact{Detail: *fact.Detail, SourceKind: fact.SourceKind, SourceHash: fact.SourceHash})
|
||||
}
|
||||
|
||||
items, err := s.hydrateWorkforcePage(ctx, org, page)
|
||||
return items, len(all), err
|
||||
}
|
||||
func (s *MongoStore) hydrateWorkforcePage(ctx context.Context, org string, page []workforcePageFact) ([]domain.WorkforceReturnDetail, error) {
|
||||
out := make([]domain.WorkforceReturnDetail, 0, len(page))
|
||||
opIDs, reviewIDs := []string{}, []string{}
|
||||
for _, v := range page {
|
||||
if v.SourceKind == "operation" {
|
||||
opIDs = append(opIDs, v.Detail.ID)
|
||||
} else {
|
||||
reviewIDs = append(reviewIDs, v.Detail.ID)
|
||||
}
|
||||
}
|
||||
ops := map[string]domain.ReviewOperation{}
|
||||
reviews := map[string]domain.AnnotationReviewRecord{}
|
||||
if len(opIDs) > 0 {
|
||||
cur, err := s.db.Collection("reviewOperations").Find(ctx, organizationScopedFilter(org, bson.M{"_id": bson.M{"$in": opIDs}, "status": "completed", "decision": "rejected"}), options.Find().SetProjection(workforceOperationProjection(true)))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var rows []domain.ReviewOperation
|
||||
err = cur.All(ctx, &rows)
|
||||
cur.Close(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, v := range rows {
|
||||
ops[v.ID] = v
|
||||
}
|
||||
}
|
||||
if len(reviewIDs) > 0 {
|
||||
cur, err := s.annotationReviews.Find(ctx, organizationScopedFilter(org, bson.M{"id": bson.M{"$in": reviewIDs}, "decision": "rejected"}), options.Find().SetProjection(workforceFields("id", "taskId", "revision", "reviewRound", "stage", "decision", "reviewerUserId", "createdAt", "problemSliceIds", "comment")))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var rows []domain.AnnotationReviewRecord
|
||||
err = cur.All(ctx, &rows)
|
||||
cur.Close(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, v := range rows {
|
||||
reviews[v.ID] = v
|
||||
}
|
||||
}
|
||||
for _, v := range page {
|
||||
detail := v.Detail
|
||||
if v.SourceKind == "operation" {
|
||||
op, ok := ops[detail.ID]
|
||||
if !ok || workforcestats.OperationFingerprint(op) != v.SourceHash {
|
||||
return nil, ErrQueryReplicaNotReady
|
||||
}
|
||||
next := 0
|
||||
for i, p := range detail.Problems {
|
||||
found := false
|
||||
for next < len(op.Draft.Problems) {
|
||||
candidate := op.Draft.Problems[next]
|
||||
next++
|
||||
p.Text = ""
|
||||
text := candidate.Text
|
||||
candidate.Text = ""
|
||||
if reflect.DeepEqual(p, candidate) {
|
||||
detail.Problems[i].Text = text
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return nil, ErrQueryReplicaNotReady
|
||||
}
|
||||
}
|
||||
comment := ""
|
||||
for _, effect := range op.Effects {
|
||||
if effect.Review != nil {
|
||||
comment = effect.Review.Comment
|
||||
}
|
||||
if effect.Reject != nil && comment == "" {
|
||||
comment = effect.Reject.Comment
|
||||
}
|
||||
}
|
||||
if detail.Stage == "quality" || len(op.Draft.Problems) == 0 {
|
||||
detail.Comment = comment
|
||||
}
|
||||
} else {
|
||||
r, ok := reviews[detail.ID]
|
||||
if !ok || workforcestats.ReviewFingerprint(r) != v.SourceHash {
|
||||
return nil, ErrQueryReplicaNotReady
|
||||
}
|
||||
detail.Comment = r.Comment
|
||||
}
|
||||
out = append(out, detail)
|
||||
}
|
||||
return out, ctx.Err()
|
||||
}
|
||||
@ -460,6 +460,9 @@ func (s *MongoStore) LoadWorkforceStats(ctx context.Context, org string, q workf
|
||||
return err
|
||||
}
|
||||
defer cur.Close(ctx)
|
||||
if budget, ok := ctx.Value(workforceBuildBudgetKey{}).(*workforceBuildBudget); ok {
|
||||
return workforceBoundedRead(ctx, cur, target, budget)
|
||||
}
|
||||
return cur.All(ctx, target)
|
||||
}
|
||||
window := func(field string) bson.M {
|
||||
@ -512,12 +515,7 @@ func (s *MongoStore) LoadWorkforceStats(ctx context.Context, org string, q workf
|
||||
if q.SupplierID != "" {
|
||||
of["$or"] = bson.A{bson.M{"effects.beforeTask.supplierId": q.SupplierID}, bson.M{"effects.beforeItem.supplierId": q.SupplierID}, bson.M{"effects.worklog.supplierId": q.SupplierID}, bson.M{"effects.item.supplierId": q.SupplierID}}
|
||||
}
|
||||
projection := fields("_id", "operationId", "organizationId", "projectId", "itemId", "taskId", "stageId", "reviewRound", "decision", "status", "createdAt", "completedAt", "draft.modelVersion", "draft.problems.id", "draft.problems.scope", "draft.problems.sourceItemId", "draft.problems.sliceId", "draft.problems.actionId", "draft.problems.start", "draft.problems.end", "draft.problems.timeBasis", "effects.review.id", "effects.review.taskId", "effects.review.revision", "effects.beforeTask.id", "effects.beforeTask.supplierId", "effects.beforeTask.projectId", "effects.beforeTask.templateVersion", "effects.beforeItem.id", "effects.beforeItem.legacyTaskId", "effects.beforeItem.latestAuthorUserId", "effects.beforeItem.acceptedByUserId", "effects.beforeItem.supplierId", "effects.beforeItem.projectId", "effects.beforeItem.currentRevision", "effects.beforeItem.subject.start", "effects.beforeItem.subject.end", "effects.item.id", "effects.item.legacyTaskId", "effects.item.latestAuthorUserId", "effects.item.supplierId", "effects.item.currentRevision", "effects.item.acceptedByUserId")
|
||||
if q.Details {
|
||||
projection["draft.problems.text"] = 1
|
||||
projection["effects.review.comment"] = 1
|
||||
projection["effects.reject.comment"] = 1
|
||||
}
|
||||
projection := workforceOperationProjection(q.Details)
|
||||
if err := read(s.queryCollection("reviewOperations"), organizationScopedFilter(org, of), projection, &d.Operations); err != nil {
|
||||
return d, err
|
||||
}
|
||||
@ -634,3 +632,21 @@ func (s *MongoStore) LoadWorkforceStats(ctx context.Context, org string, q workf
|
||||
}
|
||||
return d, nil
|
||||
}
|
||||
|
||||
func workforceFields(names ...string) bson.M {
|
||||
m := bson.M{}
|
||||
for _, name := range names {
|
||||
m[name] = 1
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func workforceOperationProjection(details bool) bson.M {
|
||||
projection := workforceFields("_id", "operationId", "organizationId", "projectId", "itemId", "taskId", "stageId", "reviewRound", "decision", "status", "createdAt", "completedAt", "draft.modelVersion", "draft.problems.id", "draft.problems.scope", "draft.problems.sourceItemId", "draft.problems.sliceId", "draft.problems.actionId", "draft.problems.start", "draft.problems.end", "draft.problems.timeBasis", "effects.review.id", "effects.review.taskId", "effects.review.revision", "effects.worklog.supplierId", "effects.beforeTask.id", "effects.beforeTask.supplierId", "effects.beforeTask.projectId", "effects.beforeTask.templateVersion", "effects.beforeItem.id", "effects.beforeItem.legacyTaskId", "effects.beforeItem.latestAuthorUserId", "effects.beforeItem.acceptedByUserId", "effects.beforeItem.supplierId", "effects.beforeItem.projectId", "effects.beforeItem.currentRevision", "effects.beforeItem.subject.start", "effects.beforeItem.subject.end", "effects.item.id", "effects.item.legacyTaskId", "effects.item.latestAuthorUserId", "effects.item.supplierId", "effects.item.currentRevision", "effects.item.acceptedByUserId")
|
||||
if details {
|
||||
projection["draft.problems.text"] = 1
|
||||
projection["effects.review.comment"] = 1
|
||||
projection["effects.reject.comment"] = 1
|
||||
}
|
||||
return projection
|
||||
}
|
||||
|
||||
@ -181,7 +181,9 @@ type row struct {
|
||||
quality, acceptance returnAccumulator
|
||||
}
|
||||
|
||||
func Build(d Snapshot, q Query) Report {
|
||||
func Build(d Snapshot, q Query) Report { return build(d, q, nil) }
|
||||
|
||||
func build(d Snapshot, q Query, emit func(Fact)) Report {
|
||||
result := Report{Items: []domain.DashboardWorkforceMetric{}, Details: []domain.WorkforceReturnDetail{}}
|
||||
tasks := map[string]domain.AnnotationTaskRecord{}
|
||||
for _, t := range d.Tasks {
|
||||
@ -335,7 +337,13 @@ func Build(d Snapshot, q Query) Report {
|
||||
day = localDay(e.CreatedAt)
|
||||
}
|
||||
r := get(e.UserID, e.SupplierID, e.TemplateVersion, day)
|
||||
for _, dest := range []*row{r, &total} {
|
||||
var contribution *row
|
||||
dests := []*row{r, &total}
|
||||
if emit != nil {
|
||||
contribution = &row{first: map[string]bool{}}
|
||||
dests = append(dests, contribution)
|
||||
}
|
||||
for _, dest := range dests {
|
||||
AddEvent(&dest.metric, e, revision)
|
||||
if seconds, known, review := reviewSeconds(e); review {
|
||||
dest.metric.ReviewDecisionCount++
|
||||
@ -370,6 +378,12 @@ func Build(d Snapshot, q Query) Report {
|
||||
dest.metric.ReworkVideoSeconds += seconds
|
||||
}
|
||||
}
|
||||
if emit != nil {
|
||||
f := factForRow(r.metric, e.ProjectID, e.UserID, e.SupplierID, e.CreatedAt, e.TemplateVersion)
|
||||
f.Kind, f.Key, f.EventCount = "event", key, 1
|
||||
f.Metric = contribution.metric
|
||||
emit(f)
|
||||
}
|
||||
}
|
||||
|
||||
// Attribution uses the frozen reviewed version, never the current task owner.
|
||||
@ -399,7 +413,41 @@ func Build(d Snapshot, q Query) Report {
|
||||
}
|
||||
return user
|
||||
}
|
||||
addReturn := func(detail domain.WorkforceReturnDetail, template int) {
|
||||
operationSuppressors := map[string][]LegacyOperationScope{}
|
||||
if emit != nil {
|
||||
for _, op := range d.Operations {
|
||||
if op.Status != "completed" || op.Decision != "rejected" {
|
||||
continue
|
||||
}
|
||||
ids := []string{}
|
||||
instant, e := time.Parse(time.RFC3339Nano, op.CompletedAt)
|
||||
if e != nil {
|
||||
continue
|
||||
}
|
||||
scope := LegacyOperationScope{ProjectID: op.ProjectID, At: op.CompletedAt, Seconds: instant.Unix(), Nanos: instant.Nanosecond(), Suppliers: []string{}}
|
||||
for _, effect := range op.Effects {
|
||||
if effect.Review != nil {
|
||||
ids = append(ids, effect.Review.ID)
|
||||
}
|
||||
if effect.BeforeTask != nil {
|
||||
scope.Suppliers = append(scope.Suppliers, effect.BeforeTask.SupplierID)
|
||||
}
|
||||
if effect.BeforeItem != nil {
|
||||
scope.Suppliers = append(scope.Suppliers, effect.BeforeItem.SupplierID)
|
||||
}
|
||||
if effect.Worklog != nil {
|
||||
scope.Suppliers = append(scope.Suppliers, effect.Worklog.SupplierID)
|
||||
}
|
||||
if effect.Item != nil {
|
||||
scope.Suppliers = append(scope.Suppliers, effect.Item.SupplierID)
|
||||
}
|
||||
}
|
||||
for _, id := range ids {
|
||||
operationSuppressors[id] = append(operationSuppressors[id], scope)
|
||||
}
|
||||
}
|
||||
}
|
||||
addReturn := func(detail domain.WorkforceReturnDetail, template int, source any) {
|
||||
if q.TemplateVersion != nil && template != *q.TemplateVersion {
|
||||
return
|
||||
}
|
||||
@ -429,6 +477,45 @@ func Build(d Snapshot, q Query) Report {
|
||||
}
|
||||
sort.Strings(detail.TaskIDs)
|
||||
detail.Metrics.TaskCount = int64(len(detail.TaskIDs))
|
||||
if emit != nil {
|
||||
f := factForRow(r.metric, detail.ProjectID, detail.UserID, detail.SupplierID, detail.CreatedAt, template)
|
||||
stored := detail
|
||||
stored.Comment = ""
|
||||
if detail.Problems != nil {
|
||||
stored.Problems = append([]domain.ReviewProblem{}, detail.Problems...)
|
||||
for i := range stored.Problems {
|
||||
stored.Problems[i].Text = ""
|
||||
}
|
||||
}
|
||||
f.Kind, f.Key, f.Detail = "return", detail.ID, &stored
|
||||
switch v := source.(type) {
|
||||
case domain.ReviewOperation:
|
||||
f.SourceKind = "operation"
|
||||
f.SourceHash = OperationFingerprint(v)
|
||||
case domain.AnnotationReviewRecord:
|
||||
f.SourceKind = "review"
|
||||
f.SourceHash = ReviewFingerprint(v)
|
||||
supplierName := func(id string) string {
|
||||
if name := suppliers[id]; name != "" {
|
||||
return name
|
||||
}
|
||||
if id == "" {
|
||||
return "未分配供应商"
|
||||
}
|
||||
return "历史供应商(名称待核对)"
|
||||
}
|
||||
baseSupplier := canonical["submit:"+revisionKey(v.TaskID, v.Revision)].SupplierID
|
||||
legacy := &LegacyReturnScope{SupplierID: baseSupplier, SupplierName: supplierName(baseSupplier), Events: []LegacyEventScope{}, Suppressors: []LegacyOperationScope{}}
|
||||
for _, event := range eventsByTask[v.TaskID] {
|
||||
if event.Revision == v.Revision && event.CreatedAt == v.CreatedAt && event.EventType == detail.Stage+"_rejected" {
|
||||
legacy.Events = append(legacy.Events, LegacyEventScope{UserID: event.UserID, SupplierID: event.SupplierID, SupplierName: supplierName(event.SupplierID), ProjectID: event.ProjectID})
|
||||
}
|
||||
}
|
||||
legacy.Suppressors = append(legacy.Suppressors, operationSuppressors[v.ID]...)
|
||||
f.Legacy = legacy
|
||||
}
|
||||
emit(f)
|
||||
}
|
||||
if q.Details {
|
||||
result.Details = append(result.Details, detail)
|
||||
}
|
||||
@ -491,7 +578,7 @@ func Build(d Snapshot, q Query) Report {
|
||||
if len(detail.Problems) == 0 {
|
||||
detail.Metrics.UnknownDetailCount = 1
|
||||
}
|
||||
addReturn(detail, t.TemplateVersion)
|
||||
addReturn(detail, t.TemplateVersion, op)
|
||||
} else if op.StageID == "acceptance" || op.StageID == "prepackage_acceptance" {
|
||||
// Acceptance can contain multiple sources. Keep overall opinions unassigned
|
||||
// when several reviewers/suppliers are involved rather than duplicating blame.
|
||||
@ -564,14 +651,14 @@ func Build(d Snapshot, q Query) Report {
|
||||
detail.Metrics.UnknownDetailCount = 1
|
||||
detail.Comment = comment
|
||||
}
|
||||
addReturn(detail, o.template)
|
||||
addReturn(detail, o.template, op)
|
||||
}
|
||||
}
|
||||
}
|
||||
// Old formal records without a frozen structured draft still contribute a
|
||||
// rejection. Their problem count/time is explicitly unknown, never guessed.
|
||||
for _, r := range d.Reviews {
|
||||
if r.Decision != domain.AnnotationReviewRejected || !InRange(r.CreatedAt, q) || handledReviews[r.ID] || strings.HasPrefix(r.ID, "review-operation-") {
|
||||
if r.Decision != domain.AnnotationReviewRejected || !InRange(r.CreatedAt, q) || (handledReviews[r.ID] && emit == nil) || strings.HasPrefix(r.ID, "review-operation-") {
|
||||
continue
|
||||
}
|
||||
t := tasks[r.TaskID]
|
||||
@ -603,7 +690,7 @@ func Build(d Snapshot, q Query) Report {
|
||||
}
|
||||
}
|
||||
detail.Metrics.SliceCount = int64(len(unique))
|
||||
addReturn(detail, t.TemplateVersion)
|
||||
addReturn(detail, t.TemplateVersion, r)
|
||||
}
|
||||
// Final acceptance decides aggregated sources. Approvals freeze no source
|
||||
// version, so each source is priced at its latest quality-approved version
|
||||
@ -672,11 +759,23 @@ func Build(d Snapshot, q Query) Report {
|
||||
day = localDay(op.CompletedAt)
|
||||
}
|
||||
r := get(op.UserID, supplier, template, day)
|
||||
for _, dest := range []*row{r, &total} {
|
||||
var contribution *row
|
||||
dests := []*row{r, &total}
|
||||
if emit != nil {
|
||||
contribution = &row{first: map[string]bool{}}
|
||||
dests = append(dests, contribution)
|
||||
}
|
||||
for _, dest := range dests {
|
||||
dest.metric.ReviewDecisionCount++
|
||||
dest.metric.ReviewVideoSeconds += seconds
|
||||
dest.metric.UnknownReviewVideoCount += unknown
|
||||
}
|
||||
if emit != nil {
|
||||
f := factForRow(r.metric, op.ProjectID, op.UserID, supplier, op.CompletedAt, template)
|
||||
f.Kind, f.Key = "acceptance", op.ID
|
||||
f.Metric = contribution.metric
|
||||
emit(f)
|
||||
}
|
||||
}
|
||||
for _, r := range rows {
|
||||
r.metric.QualityReturns = r.quality.metrics
|
||||
@ -690,13 +789,7 @@ func Build(d Snapshot, q Query) Report {
|
||||
total.metric.QualityReturns = total.quality.metrics
|
||||
total.metric.AcceptanceReturns = total.acceptance.metrics
|
||||
result.Totals = total.metric
|
||||
sort.Slice(result.Details, func(i, j int) bool {
|
||||
a, b := result.Details[i], result.Details[j]
|
||||
if a.CreatedAt != b.CreatedAt {
|
||||
return parsedAfter(a.CreatedAt, b.CreatedAt)
|
||||
}
|
||||
return a.ID+a.UserID+a.SupplierID < b.ID+b.UserID+b.SupplierID
|
||||
})
|
||||
sort.SliceStable(result.Details, func(i, j int) bool { return ReturnDetailLess(result.Details[i], result.Details[j]) })
|
||||
return result
|
||||
}
|
||||
|
||||
|
||||
@ -24,7 +24,7 @@ func TestSubmissionsUseBusinessEvidenceAndStableIdentity(t *testing.T) {
|
||||
Slices: []domain.SliceRevisionRecord{{TaskID: "one", Revision: 4, Segments: []domain.SliceSegment{{Start: 0, End: 79}}}, {TaskID: "one", Revision: 7, Segments: []domain.SliceSegment{{Start: 0, End: 79}}}},
|
||||
Suppliers: []domain.SupplierRecord{{ID: "vendor", Name: "倍赛"}},
|
||||
}
|
||||
r := Build(d, Query{})
|
||||
r := buildWithFactParity(t, d, Query{})
|
||||
m := r.Totals
|
||||
if m.FirstSubmittedTasks != 1 || m.SubmissionCount != 3 || m.ReworkSubmissionCount != 1 || m.UnknownSubmissionCount != 1 || m.FirstVideoSeconds != 79 || m.ReworkVideoSeconds != 79 || m.UnknownVideoCount != 1 {
|
||||
t.Fatalf("incorrect counters: %+v", m)
|
||||
@ -38,7 +38,7 @@ func TestSubmissionsUseBusinessEvidenceAndStableIdentity(t *testing.T) {
|
||||
d.Events[2].VideoSeconds = &seconds
|
||||
d.Revisions = nil
|
||||
d.Slices = nil
|
||||
r = Build(d, Query{})
|
||||
r = buildWithFactParity(t, d, Query{})
|
||||
if r.Totals.ReworkSubmissionCount != 1 || r.Totals.ReworkVideoSeconds != 79 || r.Totals.FirstSubmittedTasks != 1 {
|
||||
t.Fatalf("durable evidence lost: %+v", r.Totals)
|
||||
}
|
||||
@ -53,12 +53,12 @@ func TestSubmissionPeriodAndScope(t *testing.T) {
|
||||
{ID: "other", TaskID: "two", Revision: 1, EventType: "combined_submitted", OutputDelta: 1, CreatedAt: at, UserID: "foreign", SupplierID: "foreign", ProjectID: "other"},
|
||||
}}
|
||||
q := Query{From: from, To: to, UserID: "u", SupplierID: "v", AllowedProjects: map[string]bool{"p": true}}
|
||||
m := Build(d, q).Totals
|
||||
m := buildWithFactParity(t, d, q).Totals
|
||||
if m.FirstSubmittedTasks != 0 || m.ReworkSubmissionCount != 1 || m.SubmissionCount != 1 {
|
||||
t.Fatalf("cross-period/scope: %+v", m)
|
||||
}
|
||||
q.AllowedProjects = map[string]bool{}
|
||||
if len(Build(d, q).Items) != 0 {
|
||||
if len(buildWithFactParity(t, d, q).Items) != 0 {
|
||||
t.Fatal("denied project leaked")
|
||||
}
|
||||
}
|
||||
@ -74,7 +74,7 @@ func TestWorkforceRoleAndDayGroupingWithRates(t *testing.T) {
|
||||
Users: []domain.UserRecord{{ID: "ann", Name: "标注员", Role: domain.UserRoleAnnotator}, {ID: "qc", Name: "质检", Role: domain.UserRoleQualityReviewer}},
|
||||
}
|
||||
q := Query{Role: string(domain.UserRoleQualityReviewer), GroupBy: "day"}
|
||||
r := Build(d, q)
|
||||
r := buildWithFactParity(t, d, q)
|
||||
if len(r.Items) != 2 || r.Items[0].Role != string(domain.UserRoleQualityReviewer) || r.Items[0].Date == "" {
|
||||
t.Fatalf("role/day grouping: %+v", r.Items)
|
||||
}
|
||||
@ -96,7 +96,7 @@ func TestCrossDayRetryKeepsOriginalSubmissionDate(t *testing.T) {
|
||||
retry.CreatedAt = at
|
||||
from, _ := time.Parse(time.RFC3339, "2026-09-28T00:00:00Z")
|
||||
d := Snapshot{Events: []domain.AnnotationWorklogEventRecord{retry}, SubmissionHistory: []domain.AnnotationWorklogEventRecord{original, retry}}
|
||||
if m := Build(d, Query{From: from, To: from.Add(24 * time.Hour)}).Totals; m.SubmissionCount != 0 || m.FirstSubmittedTasks != 0 {
|
||||
if m := buildWithFactParity(t, d, Query{From: from, To: from.Add(24 * time.Hour)}).Totals; m.SubmissionCount != 0 || m.FirstSubmittedTasks != 0 {
|
||||
t.Fatalf("retry moved first submission into another day: %+v", m)
|
||||
}
|
||||
// A later valid first-kind record for an already submitted task can count as
|
||||
@ -104,7 +104,7 @@ func TestCrossDayRetryKeepsOriginalSubmissionDate(t *testing.T) {
|
||||
retry.Revision = 5
|
||||
d.Events[0] = retry
|
||||
d.SubmissionHistory[1] = retry
|
||||
if m := Build(d, Query{From: from, To: from.Add(24 * time.Hour)}).Totals; m.SubmissionCount != 1 || m.FirstSubmittedTasks != 0 {
|
||||
if m := buildWithFactParity(t, d, Query{From: from, To: from.Add(24 * time.Hour)}).Totals; m.SubmissionCount != 1 || m.FirstSubmittedTasks != 0 {
|
||||
t.Fatalf("task counted twice as first: %+v", m)
|
||||
}
|
||||
}
|
||||
@ -166,7 +166,7 @@ func TestCompletedReturnsRoundTotalsFrozenOwnershipAndLegacy(t *testing.T) {
|
||||
{ID: one.ID + ":review", TaskID: "task", Revision: 4, Decision: domain.AnnotationReviewRejected, Stage: domain.AnnotationReviewStageQuality, CreatedAt: at},
|
||||
{ID: failed.ID + ":review", TaskID: "task", Revision: 4, Decision: domain.AnnotationReviewRejected, Stage: domain.AnnotationReviewStageQuality, CreatedAt: at},
|
||||
}}
|
||||
r := Build(d, Query{Details: true})
|
||||
r := buildWithFactParity(t, d, Query{Details: true})
|
||||
if len(r.Items) != 1 || r.Items[0].UserID != "original-author" || r.Items[0].SupplierID != "original-vendor" {
|
||||
t.Fatalf("not frozen ownership: %+v", r.Items)
|
||||
}
|
||||
@ -177,7 +177,7 @@ func TestCompletedReturnsRoundTotalsFrozenOwnershipAndLegacy(t *testing.T) {
|
||||
if len(r.Details) != 2 || r.Details[0].Problems[0].Text != "保留原文" {
|
||||
t.Fatal("lost detail")
|
||||
}
|
||||
if len(Build(d, Query{UserID: "new-author"}).Items) != 0 {
|
||||
if len(buildWithFactParity(t, d, Query{UserID: "new-author"}).Items) != 0 {
|
||||
t.Fatal("assigned current user historical blame")
|
||||
}
|
||||
if !reflect.DeepEqual(one.Draft, d.Operations[0].Draft) {
|
||||
@ -199,12 +199,12 @@ func TestAcceptanceTargetsQualityReviewerAndSkipsTaskScope(t *testing.T) {
|
||||
{TaskID: "one", Revision: 4, Stage: domain.AnnotationReviewStageQuality, Decision: domain.AnnotationReviewApproved, ReviewerUserID: "qc1", CreatedAt: "2026-09-27T00:00:00Z"},
|
||||
{TaskID: "two", Revision: 2, Stage: domain.AnnotationReviewStageQuality, Decision: domain.AnnotationReviewApproved, ReviewerUserID: "qc2", CreatedAt: "2026-09-27T00:00:00Z"},
|
||||
}}
|
||||
r := Build(d, Query{Details: true})
|
||||
r := buildWithFactParity(t, d, Query{Details: true})
|
||||
m := r.Totals.AcceptanceReturns
|
||||
if m.DecisionCount != 1 || m.TaskCount != 2 || m.ProblemCount != 1 || m.VideoSeconds != 10 || len(r.Items) != 2 {
|
||||
t.Fatalf("acceptance counts %+v rows %+v", m, r.Items)
|
||||
}
|
||||
r = Build(d, Query{SupplierID: "v1", Details: true})
|
||||
r = buildWithFactParity(t, d, Query{SupplierID: "v1", Details: true})
|
||||
if len(r.Items) != 1 || r.Items[0].UserID != "qc1" || r.Totals.AcceptanceReturns.ProblemCount != 1 || len(r.Details[0].Problems) != 1 {
|
||||
t.Fatalf("foreign overall/source leaked: %+v", r)
|
||||
}
|
||||
@ -247,7 +247,7 @@ func TestReviewVideoSecondsPerDecision(t *testing.T) {
|
||||
SourceItems: []domain.HumanItemRecord{{ID: "src-a", LegacyTaskID: "a"}, {ID: "src-b", LegacyTaskID: "b"}},
|
||||
Users: []domain.UserRecord{{ID: "acc", SupplierID: "v"}, {ID: "qc", SupplierID: "v"}, {ID: "author", SupplierID: "v"}},
|
||||
}
|
||||
r := Build(d, Query{})
|
||||
r := buildWithFactParity(t, d, Query{})
|
||||
byUser := map[string]domain.DashboardWorkforceMetric{}
|
||||
for _, item := range r.Items {
|
||||
byUser[item.UserID] = item
|
||||
@ -282,12 +282,12 @@ func TestReviewVideoSecondsPerDecision(t *testing.T) {
|
||||
t.Fatalf("review output totals: %+v", r.Totals)
|
||||
}
|
||||
// Scope: a reviewer's own report only carries their own decisions.
|
||||
self := Build(d, Query{UserID: "acc", MatchUser: true}).Totals
|
||||
self := buildWithFactParity(t, d, Query{UserID: "acc", MatchUser: true}).Totals
|
||||
if self.ReviewVideoSeconds != 72 || self.FirstVideoSeconds != 0 {
|
||||
t.Fatalf("self scope: %+v", self)
|
||||
}
|
||||
// Day grouping attributes acceptance by its completion date.
|
||||
days := Build(d, Query{UserID: "acc", MatchUser: true, GroupBy: "day"}).Items
|
||||
days := buildWithFactParity(t, d, Query{UserID: "acc", MatchUser: true, GroupBy: "day"}).Items
|
||||
if len(days) != 1 || days[0].Date != "2026-09-28" || days[0].ReviewVideoSeconds != 72 {
|
||||
t.Fatalf("day grouping: %+v", days)
|
||||
}
|
||||
@ -338,7 +338,7 @@ func TestReviewVideoSecondsFrozenThenSubmissionFallback(t *testing.T) {
|
||||
Users: []domain.UserRecord{{ID: "acc", SupplierID: "v"}, {ID: "qc", SupplierID: "v"}},
|
||||
}
|
||||
byUser := map[string]domain.DashboardWorkforceMetric{}
|
||||
for _, item := range Build(d, Query{}).Items {
|
||||
for _, item := range buildWithFactParity(t, d, Query{}).Items {
|
||||
byUser[item.UserID] = item
|
||||
}
|
||||
if got := byUser["qc"]; got.ReviewVideoSeconds != 88 || got.UnknownReviewVideoCount != 1 || got.ReviewDecisionCount != 3 {
|
||||
|
||||
289
internal/workforcestats/facts.go
Normal file
289
internal/workforcestats/facts.go
Normal file
@ -0,0 +1,289 @@
|
||||
package workforcestats
|
||||
|
||||
import (
|
||||
"droboticflow/internal/domain"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Fact is a text-free contribution produced by the SAME historical calculator.
|
||||
// Separate modes/template grouping preserve mixed acceptance attribution rather
|
||||
// than attempting to add incompatible daily totals or guess frozen authors.
|
||||
type LegacyEventScope struct {
|
||||
UserID string `bson:"userId" json:"userId"`
|
||||
SupplierID string `bson:"supplierId" json:"supplierId"`
|
||||
SupplierName string `bson:"supplierName" json:"supplierName"`
|
||||
ProjectID string `bson:"projectId" json:"projectId"`
|
||||
}
|
||||
type LegacyOperationScope struct {
|
||||
Seconds int64 `bson:"seconds" json:"seconds"`
|
||||
Nanos int `bson:"nanos" json:"nanos"`
|
||||
ProjectID string `bson:"projectId" json:"projectId"`
|
||||
At string `bson:"at" json:"at"`
|
||||
Suppliers []string `bson:"suppliers" json:"suppliers"`
|
||||
}
|
||||
type LegacyReturnScope struct {
|
||||
SupplierID string `bson:"supplierId" json:"supplierId"`
|
||||
SupplierName string `bson:"supplierName" json:"supplierName"`
|
||||
Events []LegacyEventScope `bson:"events" json:"events"`
|
||||
Suppressors []LegacyOperationScope `bson:"suppressors" json:"suppressors"`
|
||||
}
|
||||
type Fact struct {
|
||||
Legacy *LegacyReturnScope `bson:"legacy,omitempty" json:"legacy,omitempty"`
|
||||
Key string `bson:"key" json:"key"`
|
||||
Kind string `bson:"kind" json:"kind"`
|
||||
Mode domain.RunMode `bson:"mode" json:"mode"`
|
||||
GroupTemplate bool `bson:"groupTemplate" json:"groupTemplate"`
|
||||
ProjectID string `bson:"projectId" json:"projectId"`
|
||||
UserID string `bson:"userId" json:"userId"`
|
||||
SupplierID string `bson:"supplierId" json:"supplierId"`
|
||||
UserName string `bson:"userName" json:"userName"`
|
||||
SupplierName string `bson:"supplierName" json:"supplierName"`
|
||||
Role string `bson:"role" json:"role"`
|
||||
TemplateVersion int `bson:"templateVersion" json:"templateVersion"`
|
||||
CreatedAt string `bson:"createdAt" json:"createdAt"`
|
||||
Day string `bson:"day" json:"day"`
|
||||
Metric domain.DashboardWorkforceMetric `bson:"-" json:"metric"`
|
||||
EventCount int `bson:"eventCount" json:"eventCount"`
|
||||
Detail *domain.WorkforceReturnDetail `bson:"detail,omitempty" json:"detail,omitempty"`
|
||||
SourceKind string `bson:"sourceKind,omitempty" json:"sourceKind,omitempty"`
|
||||
SourceHash string `bson:"sourceHash,omitempty" json:"sourceHash,omitempty"`
|
||||
}
|
||||
|
||||
func factForRow(m domain.DashboardWorkforceMetric, project, user, supplier, at string, template int) Fact {
|
||||
return Fact{ProjectID: project, UserID: user, SupplierID: supplier, UserName: m.UserName, SupplierName: m.SupplierName, Role: m.Role, TemplateVersion: template, CreatedAt: at, Day: localDay(at)}
|
||||
}
|
||||
func ModeVariant(mode domain.RunMode) domain.RunMode {
|
||||
if mode == domain.RunModeDebug || mode == domain.RunModeAll {
|
||||
return mode
|
||||
}
|
||||
return domain.RunModeProduction
|
||||
}
|
||||
|
||||
// Numeric counters are shared by in-memory reduction and Mongo aggregation.
|
||||
// Identity fields and undefined rate pointers never participate in addition.
|
||||
type counterField struct {
|
||||
name string
|
||||
index []int
|
||||
kind reflect.Kind
|
||||
}
|
||||
|
||||
var workforceCounterFields = func() []counterField {
|
||||
out := []counterField{}
|
||||
var walk func(reflect.Type, []int)
|
||||
walk = func(t reflect.Type, parent []int) {
|
||||
for i := 0; i < t.NumField(); i++ {
|
||||
f := t.Field(i)
|
||||
idx := append(append([]int(nil), parent...), i)
|
||||
if f.Anonymous && f.Type.Kind() == reflect.Struct {
|
||||
walk(f.Type, idx)
|
||||
continue
|
||||
}
|
||||
if f.Name == "TemplateVersion" {
|
||||
continue
|
||||
}
|
||||
if f.Type.Kind() == reflect.Int || f.Type.Kind() == reflect.Int64 || f.Type.Kind() == reflect.Float64 {
|
||||
out = append(out, counterField{strings.Split(f.Tag.Get("json"), ",")[0], idx, f.Type.Kind()})
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(reflect.TypeOf(domain.DashboardWorkforceMetric{}), nil)
|
||||
return out
|
||||
}()
|
||||
|
||||
func MetricCounterNames() []string {
|
||||
out := []string{}
|
||||
for _, f := range workforceCounterFields {
|
||||
out = append(out, f.name)
|
||||
}
|
||||
return out
|
||||
}
|
||||
func MetricCounterValues(m domain.DashboardWorkforceMetric) map[string]any {
|
||||
out := map[string]any{}
|
||||
v := reflect.ValueOf(m)
|
||||
for _, f := range workforceCounterFields {
|
||||
out[f.name] = v.FieldByIndex(f.index).Interface()
|
||||
}
|
||||
return out
|
||||
}
|
||||
func addMetricCounters(dst *domain.DashboardWorkforceMetric, src domain.DashboardWorkforceMetric) {
|
||||
a, b := reflect.ValueOf(dst).Elem(), reflect.ValueOf(src)
|
||||
for _, f := range workforceCounterFields {
|
||||
x, y := a.FieldByIndex(f.index), b.FieldByIndex(f.index)
|
||||
if f.kind == reflect.Float64 {
|
||||
x.SetFloat(x.Float() + y.Float())
|
||||
} else {
|
||||
x.SetInt(x.Int() + y.Int())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Materialize emits canonical contributions once in the background. Equal
|
||||
// event timestamps/dimensions combine without rounding the requested boundaries.
|
||||
func Materialize(d Snapshot) []Fact {
|
||||
byKey := map[string]Fact{}
|
||||
for _, mode := range []domain.RunMode{domain.RunModeProduction, domain.RunModeDebug, domain.RunModeAll} {
|
||||
for _, templates := range []bool{false, true} {
|
||||
q := Query{Mode: mode, GroupTemplate: templates}
|
||||
build(d, q, func(f Fact) {
|
||||
f.Mode, f.GroupTemplate = mode, templates
|
||||
identity := f.Key
|
||||
if f.Kind == "event" {
|
||||
identity = ""
|
||||
}
|
||||
stage := ""
|
||||
if f.Detail != nil {
|
||||
stage = f.Detail.Stage
|
||||
}
|
||||
f.Key = workforceFingerprint([]any{mode, templates, f.Kind, f.SourceKind, stage, identity, f.ProjectID, f.UserID, f.SupplierID, f.TemplateVersion, f.CreatedAt})
|
||||
if old, ok := byKey[f.Key]; ok && f.Kind == "event" {
|
||||
addMetricCounters(&old.Metric, f.Metric)
|
||||
old.EventCount += f.EventCount
|
||||
byKey[f.Key] = old
|
||||
} else {
|
||||
byKey[f.Key] = f
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
keys := make([]string, 0, len(byKey))
|
||||
for key := range byKey {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
out := make([]Fact, 0, len(keys))
|
||||
for _, key := range keys {
|
||||
out = append(out, byKey[key])
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
if out[i].SourceKind != out[j].SourceKind {
|
||||
return out[i].SourceKind < out[j].SourceKind
|
||||
}
|
||||
return out[i].Key < out[j].Key
|
||||
})
|
||||
return out
|
||||
}
|
||||
func EffectiveFact(f Fact, q Query) (Fact, bool) {
|
||||
if f.Legacy == nil {
|
||||
return f, true
|
||||
}
|
||||
l := f.Legacy
|
||||
for _, op := range l.Suppressors {
|
||||
if !InRange(op.At, q) || (q.ProjectID != "" && q.ProjectID != op.ProjectID) || (q.AllowedProjects != nil && !q.AllowedProjects[op.ProjectID]) {
|
||||
continue
|
||||
}
|
||||
if q.SupplierID != "" {
|
||||
found := false
|
||||
for _, supplier := range op.Suppliers {
|
||||
if supplier == q.SupplierID {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
continue
|
||||
}
|
||||
}
|
||||
return f, false
|
||||
}
|
||||
f.SupplierID, f.SupplierName = l.SupplierID, l.SupplierName
|
||||
for _, e := range l.Events {
|
||||
if (q.ProjectID != "" && q.ProjectID != e.ProjectID) || (q.AllowedProjects != nil && !q.AllowedProjects[e.ProjectID]) || ((q.MatchUser || q.UserID != "") && q.UserID != e.UserID) || (q.SupplierID != "" && q.SupplierID != e.SupplierID) {
|
||||
continue
|
||||
}
|
||||
f.SupplierID, f.SupplierName = e.SupplierID, e.SupplierName
|
||||
break
|
||||
}
|
||||
if f.Detail != nil {
|
||||
v := *f.Detail
|
||||
v.SupplierID = f.SupplierID
|
||||
v.AttributionUnknown = v.UserID == "" || v.SupplierID == ""
|
||||
f.Detail = &v
|
||||
}
|
||||
return f, true
|
||||
}
|
||||
func factVisible(f Fact, q Query) bool {
|
||||
return f.Mode == ModeVariant(q.Mode) && f.GroupTemplate == q.GroupTemplate && InRange(f.CreatedAt, q) && (q.ProjectID == "" || q.ProjectID == f.ProjectID) && (q.SupplierID == "" || q.SupplierID == f.SupplierID) && (q.Role == "" || q.Role == f.Role) && ((!q.MatchUser && q.UserID == "") || q.UserID == f.UserID) && (q.TemplateVersion == nil || *q.TemplateVersion == f.TemplateVersion) && (q.AllowedProjects == nil || q.AllowedProjects[f.ProjectID]) && (q.AllowedSuppliers == nil || q.AllowedSuppliers[f.SupplierID])
|
||||
}
|
||||
func BuildFacts(facts []Fact, q Query) Report {
|
||||
result := Report{Items: []domain.DashboardWorkforceMetric{}, Details: []domain.WorkforceReturnDetail{}}
|
||||
rows := map[string]*row{}
|
||||
total := row{}
|
||||
for _, f := range facts {
|
||||
var visible bool
|
||||
f, visible = EffectiveFact(f, q)
|
||||
if !visible {
|
||||
continue
|
||||
}
|
||||
if !factVisible(f, q) {
|
||||
continue
|
||||
}
|
||||
day := ""
|
||||
if q.GroupBy == "day" {
|
||||
day = f.Day
|
||||
}
|
||||
template := 0
|
||||
if q.GroupTemplate {
|
||||
template = f.TemplateVersion
|
||||
}
|
||||
key := fmt.Sprintf("%s\x00%s\x00%d\x00%s", f.UserID, f.SupplierID, template, day)
|
||||
r := rows[key]
|
||||
if r == nil {
|
||||
r = &row{metric: domain.DashboardWorkforceMetric{UserID: f.UserID, UserName: f.UserName, Role: f.Role, SupplierID: f.SupplierID, SupplierName: f.SupplierName, TemplateVersion: template, Date: day}}
|
||||
rows[key] = r
|
||||
}
|
||||
for _, dst := range []*row{r, &total} {
|
||||
addMetricCounters(&dst.metric, f.Metric)
|
||||
if f.Detail != nil {
|
||||
if f.Detail.Stage == "quality" {
|
||||
dst.quality.add(*f.Detail)
|
||||
} else {
|
||||
dst.acceptance.add(*f.Detail)
|
||||
}
|
||||
}
|
||||
}
|
||||
result.EventCount += f.EventCount
|
||||
if q.Details && f.Detail != nil {
|
||||
result.Details = append(result.Details, *f.Detail)
|
||||
}
|
||||
}
|
||||
for _, r := range rows {
|
||||
r.metric.QualityReturns = r.quality.metrics
|
||||
r.metric.AcceptanceReturns = r.acceptance.metrics
|
||||
result.Items = append(result.Items, r.metric)
|
||||
}
|
||||
sort.Slice(result.Items, func(i, j int) bool {
|
||||
a, b := result.Items[i], result.Items[j]
|
||||
return fmt.Sprintf("%s/%s/%s/%s/%d", a.Date, a.SupplierName, a.UserName, a.UserID, a.TemplateVersion) < fmt.Sprintf("%s/%s/%s/%s/%d", b.Date, b.SupplierName, b.UserName, b.UserID, b.TemplateVersion)
|
||||
})
|
||||
total.metric.QualityReturns, total.metric.AcceptanceReturns = total.quality.metrics, total.acceptance.metrics
|
||||
result.Totals = total.metric
|
||||
sort.SliceStable(result.Details, func(i, j int) bool { return ReturnDetailLess(result.Details[i], result.Details[j]) })
|
||||
return result
|
||||
}
|
||||
|
||||
// ReturnFacts preserves the source kind even if old review/operation collections
|
||||
// contain the same ID. It is used only for pre-materialization compatibility.
|
||||
func ReturnFacts(d Snapshot, q Query) []Fact {
|
||||
out := []Fact{}
|
||||
build(d, q, func(f Fact) {
|
||||
if f.Kind != "return" {
|
||||
return
|
||||
}
|
||||
if effective, ok := EffectiveFact(f, q); ok {
|
||||
out = append(out, effective)
|
||||
}
|
||||
})
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
a, b := *out[i].Detail, *out[j].Detail
|
||||
if ReturnDetailLess(a, b) {
|
||||
return true
|
||||
}
|
||||
if ReturnDetailLess(b, a) {
|
||||
return false
|
||||
}
|
||||
return out[i].SourceKind < out[j].SourceKind
|
||||
})
|
||||
return out
|
||||
}
|
||||
110
internal/workforcestats/facts_test.go
Normal file
110
internal/workforcestats/facts_test.go
Normal file
@ -0,0 +1,110 @@
|
||||
package workforcestats
|
||||
|
||||
import (
|
||||
"droboticflow/internal/domain"
|
||||
"math"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Run the existing business fixtures against both paths. Source loaders have
|
||||
// always narrowed event/operation scope BEFORE calling Build; retain that seam.
|
||||
func scopeFactComparison(d Snapshot, q Query) Snapshot {
|
||||
out := d
|
||||
out.SubmissionHistory = append(append([]domain.AnnotationWorklogEventRecord{}, d.SubmissionHistory...), d.Events...)
|
||||
out.Events = nil
|
||||
for _, e := range d.Events {
|
||||
if !InRange(e.CreatedAt, q) || (q.ProjectID != "" && q.ProjectID != e.ProjectID) || (q.AllowedProjects != nil && !q.AllowedProjects[e.ProjectID]) || (q.SupplierID != "" && q.SupplierID != e.SupplierID) || ((q.MatchUser || q.UserID != "") && q.UserID != e.UserID) {
|
||||
continue
|
||||
}
|
||||
out.Events = append(out.Events, e)
|
||||
}
|
||||
out.Operations = nil
|
||||
for _, op := range d.Operations {
|
||||
if !InRange(op.CompletedAt, q) || (q.ProjectID != "" && q.ProjectID != op.ProjectID) || (q.AllowedProjects != nil && !q.AllowedProjects[op.ProjectID]) {
|
||||
continue
|
||||
}
|
||||
if q.SupplierID != "" {
|
||||
found := false
|
||||
for _, e := range op.Effects {
|
||||
found = found || (e.BeforeTask != nil && e.BeforeTask.SupplierID == q.SupplierID) || (e.BeforeItem != nil && e.BeforeItem.SupplierID == q.SupplierID) || (e.Item != nil && e.Item.SupplierID == q.SupplierID) || (e.Worklog != nil && e.Worklog.SupplierID == q.SupplierID)
|
||||
}
|
||||
if !found {
|
||||
continue
|
||||
}
|
||||
}
|
||||
out.Operations = append(out.Operations, op)
|
||||
}
|
||||
out.AcceptanceDecisions = nil
|
||||
for _, op := range d.AcceptanceDecisions {
|
||||
if !InRange(op.CompletedAt, q) || (q.ProjectID != "" && q.ProjectID != op.ProjectID) || (q.AllowedProjects != nil && !q.AllowedProjects[op.ProjectID]) || ((q.MatchUser || q.UserID != "") && q.UserID != op.UserID) {
|
||||
continue
|
||||
}
|
||||
out.AcceptanceDecisions = append(out.AcceptanceDecisions, op)
|
||||
}
|
||||
return out
|
||||
}
|
||||
func textFreeFactReport(r Report) Report {
|
||||
for i := range r.Details {
|
||||
r.Details[i].Comment = ""
|
||||
if r.Details[i].Problems != nil {
|
||||
r.Details[i].Problems = append([]domain.ReviewProblem{}, r.Details[i].Problems...)
|
||||
for j := range r.Details[i].Problems {
|
||||
r.Details[i].Problems[j].Text = ""
|
||||
}
|
||||
}
|
||||
}
|
||||
return r
|
||||
}
|
||||
func factReportsEqual(a, b reflect.Value) bool {
|
||||
if a.Type() != b.Type() {
|
||||
return false
|
||||
}
|
||||
switch a.Kind() {
|
||||
case reflect.Float64:
|
||||
return math.Abs(a.Float()-b.Float()) <= 1e-8
|
||||
case reflect.Struct:
|
||||
for i := 0; i < a.NumField(); i++ {
|
||||
if !factReportsEqual(a.Field(i), b.Field(i)) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
case reflect.Slice:
|
||||
if a.IsNil() != b.IsNil() || a.Len() != b.Len() {
|
||||
return false
|
||||
}
|
||||
for i := 0; i < a.Len(); i++ {
|
||||
if !factReportsEqual(a.Index(i), b.Index(i)) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
case reflect.Pointer:
|
||||
if a.IsNil() || b.IsNil() {
|
||||
return a.IsNil() == b.IsNil()
|
||||
}
|
||||
return factReportsEqual(a.Elem(), b.Elem())
|
||||
default:
|
||||
return reflect.DeepEqual(a.Interface(), b.Interface())
|
||||
}
|
||||
}
|
||||
func buildWithFactParity(t *testing.T, d Snapshot, q Query) Report {
|
||||
t.Helper()
|
||||
original := Build(d, q)
|
||||
facts := Materialize(d)
|
||||
want := textFreeFactReport(Build(scopeFactComparison(d, q), q))
|
||||
got := BuildFacts(facts, q)
|
||||
if !factReportsEqual(reflect.ValueOf(got), reflect.ValueOf(want)) {
|
||||
t.Fatalf("materialized facts differ from scoped historical calculator q=%+v\ngot=%+v\nwant=%+v", q, got, want)
|
||||
}
|
||||
return original
|
||||
}
|
||||
|
||||
func TestReturnPaginationEquivalentTimestampsHaveStableOrder(t *testing.T) {
|
||||
a := domain.WorkforceReturnDetail{ID: "a", CreatedAt: "2026-11-01T00:00:00.000Z"}
|
||||
b := domain.WorkforceReturnDetail{ID: "b", CreatedAt: "2026-11-01T08:00:00+08:00"}
|
||||
if !ReturnDetailLess(a, b) || ReturnDetailLess(b, a) {
|
||||
t.Fatal("equivalent timestamps lost stable identity order")
|
||||
}
|
||||
}
|
||||
37
internal/workforcestats/fingerprint.go
Normal file
37
internal/workforcestats/fingerprint.go
Normal file
@ -0,0 +1,37 @@
|
||||
package workforcestats
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"droboticflow/internal/domain"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
)
|
||||
|
||||
// Fingerprints compare the exact projected formal record, excluding only text
|
||||
// fetched later. Different revisions/attribution cannot lend text to old facts.
|
||||
func OperationFingerprint(op domain.ReviewOperation) string {
|
||||
raw, _ := json.Marshal(op)
|
||||
var copy domain.ReviewOperation
|
||||
_ = json.Unmarshal(raw, ©)
|
||||
for i := range copy.Draft.Problems {
|
||||
copy.Draft.Problems[i].Text = ""
|
||||
}
|
||||
for _, v := range copy.Effects {
|
||||
if v.Review != nil {
|
||||
v.Review.Comment = ""
|
||||
}
|
||||
if v.Reject != nil {
|
||||
v.Reject.Comment = ""
|
||||
}
|
||||
}
|
||||
return workforceFingerprint(copy)
|
||||
}
|
||||
func ReviewFingerprint(r domain.AnnotationReviewRecord) string {
|
||||
r.Comment = ""
|
||||
return workforceFingerprint(r)
|
||||
}
|
||||
func workforceFingerprint(v any) string {
|
||||
raw, _ := json.Marshal(v)
|
||||
sum := sha256.Sum256(raw)
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
@ -30,7 +30,7 @@ func TestSliceVersionsPreservesReportAndExcludesAutosaves(t *testing.T) {
|
||||
if len(needed) != 2 || !needed["task"][2] || !needed["task"][3] || !needed["legacy"][7] {
|
||||
t.Fatalf("incorrect needed versions: %v", needed)
|
||||
}
|
||||
before := Build(d, Query{})
|
||||
before := buildWithFactParity(t, d, Query{})
|
||||
pruned := d
|
||||
pruned.Slices = nil
|
||||
for _, s := range d.Slices {
|
||||
@ -41,7 +41,7 @@ func TestSliceVersionsPreservesReportAndExcludesAutosaves(t *testing.T) {
|
||||
if len(pruned.Slices) != 3 {
|
||||
t.Fatalf("read %d versions instead of 3", len(pruned.Slices))
|
||||
}
|
||||
if after := Build(pruned, Query{}); !reflect.DeepEqual(before, after) {
|
||||
if after := buildWithFactParity(t, pruned, Query{}); !reflect.DeepEqual(before, after) {
|
||||
t.Fatal("report changed after excluding unused historical slice versions")
|
||||
}
|
||||
d.AcceptanceDecisions[0].ReviewVideoSources = []domain.ReviewVideoSource{{SourceItemID: "source", TaskID: "legacy", VideoSeconds: &frozen}}
|
||||
|
||||
@ -60,3 +60,25 @@ func ModeMatches(mode domain.RunMode, q Query) bool {
|
||||
}
|
||||
return mode != domain.RunModeDebug
|
||||
}
|
||||
|
||||
// ReturnDetailLess orders equivalent RFC3339 spellings by stable identity too,
|
||||
// so offset pages cannot reshuffle records representing the same instant.
|
||||
func ReturnDetailLess(a, b domain.WorkforceReturnDetail) bool {
|
||||
if parsedAfter(a.CreatedAt, b.CreatedAt) {
|
||||
return true
|
||||
}
|
||||
if parsedAfter(b.CreatedAt, a.CreatedAt) {
|
||||
return false
|
||||
}
|
||||
x, y := a.ID+a.UserID+a.SupplierID, b.ID+b.UserID+b.SupplierID
|
||||
if x != y {
|
||||
return x < y
|
||||
}
|
||||
if a.ID != b.ID {
|
||||
return a.ID < b.ID
|
||||
}
|
||||
if a.UserID != b.UserID {
|
||||
return a.UserID < b.UserID
|
||||
}
|
||||
return a.SupplierID < b.SupplierID
|
||||
}
|
||||
|
||||
@ -21,7 +21,9 @@ CASES = ('internal/store/query_replica_local_test.go',
|
||||
'internal/api/query_replica_local_test.go',
|
||||
'cmd/query-cdc-probe/main_test.go',
|
||||
'internal/store/query_replica_valkey_local_test.go',
|
||||
'internal/store/workforce_report_cache_local_test.go')
|
||||
'internal/store/workforce_report_cache_local_test.go',
|
||||
'internal/store/workforce_facts_local_test.go',
|
||||
'internal/store/query_pool_buckets_local_test.go')
|
||||
|
||||
|
||||
def owned_native():
|
||||
|
||||
@ -1013,7 +1013,9 @@
|
||||
"paths": [
|
||||
"^internal/store/query_(replica|sync|tracked_collection|collection_interface).*\\.go$",
|
||||
"^internal/api/query_replica.*\\.go$",
|
||||
"^internal/store/workforce_(report_cache|heap_diagnostic).*\\.go$",
|
||||
"^internal/store/workforce_(report_cache|heap_diagnostic|facts|return_page).*\\.go$",
|
||||
"^internal/store/query_pool_buckets.*\\.go$",
|
||||
"^internal/api/query_fact_routes_test\\.go$",
|
||||
"^internal/api/workforce_compact_report_test\\.go$",
|
||||
"^internal/testsupport/query_replica_lab.*\\.go$",
|
||||
"^cmd/(query-replica|query-cdc-probe)/",
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user