🐛 Release unused project locks and resume transient submissions

This commit is contained in:
Codex 2026-11-10 00:17:00 -08:00
parent 85ce47e4d5
commit b2ac2cb3e4
16 changed files with 568 additions and 28 deletions

View File

@ -3,6 +3,6 @@
Goal: Prevent failed submissions or slice-only regeneration from erasing saved annotations; investigate the reported B11 rework without overwriting production data.
Agent: Codex
Base: origin/dev fast-forwarded to deployed 7a1231c7, required by the live-commit freshness rule; independent of the unpushed pool-search optimizations.
Scope: Submission error classification, failed-input recovery evidence, blank regeneration guards, revision cleanup publication gates and failure/restart/legacy regressions.
Out of scope: Push, PR, deployment, production restores/mutations, project-lock redesign and unrelated pool performance.
Scope: Submission error classification, failed-input recovery evidence, blank regeneration guards, revision cleanup publication gates, cancellation-safe project lock acquisition, durable bounded submission retry, related search-only metadata projection and failure/restart/legacy regressions.
Out of scope: Push, PR, deployment, production restores/mutations, lock protocol/lease duration changes and unrelated pool performance.
Risk: Preserve old-client recovery, in-flight job fencing, legitimate explicit reset and successful submission; never replace current production work with an old return baseline.

View File

@ -0,0 +1,39 @@
# Project lock contention and durable submission recovery
Base: `d1dc4a38` on verified live API/processor source `7a1231c7`. This follows the annotation preservation guard; unrelated unpushed pool optimizations remain excluded.
## Confirmed and inferred causes
- Actual processor logs contain seven failed jobs with eight busy-error lines in the UTC 06:38–06:40 incident window. API traces immediately before/during this window contain 30-second continuation failures, query timeouts and many roughly five-second write failures. Logs are under ignored `local-data/diagnostics/write-lock/`.
- Isolated Mongo reproduced a concrete amplification defect: cancellation before the final project stripe executes no protected body but leaves 32 acquired prerequisite locks held. The old cleanup treats cancellation during lock acquisition like an uncertain business write; default leases last two minutes. After the patch, unused locks are released by owner. Actual started writes returning an unknown result retain all 33 protection locks.
- The project/record/account lock code is unchanged between query optimization parent `8c2ae878` and live `7a1231c7`. Therefore recent query code did not introduce this lock-retention mechanism. There is no historical lock-owner trace to prove which exact request first triggered the incident; database read pressure remains a possible trigger, not a proven sole cause.
- Actual async submission with one injected `ErrAccountMutationBusy` permanently failed and cleared its reservation before the patch. It now retains the same job/input/reservation, defers with bounded backoff, revalidates permissions/source, and finishes idempotently. Business/ownership errors are not retried. Post-handoff projection recovery does not duplicate submission revisions.
- A synthetic 2.7 MB / 10,000-action payload added about 2.7 ms of metadata computation in the local sample, not seconds. This single-machine measurement cannot exclude contention under production CPU/I/O pressure.
## Recent query overhead and correction
Same Mongo, dependencies and 420-record large-payload fixture: optimization-parent source versus live-compatible migrated metadata, then the trimmed candidate. Result identities and permissions/legacy tests agree; no record count reduction or permission cache was introduced.
| Query | Pre-optimization Mongo reply bytes | Live-compatible bytes | Trimmed bytes |
| ------------------------- | ---------------------------------: | --------------------: | ------------: |
| Broad name, 200 rows | 131542 | 268737 | 234137 |
| Pending quality, 200 rows | 153342 | 309142 | 239942 |
Search-only `poolSearch.text/source` had been carried through search/list hydration. Removing these duplicate fields reduces these representative reads by 12.9% / 22.4% against live-compatible code. Display, duration, freshness and legacy fallback evidence remain. Other summary overhead remains; this is not proof that all database pressure or all-project production latency is resolved. These are Mongo reply bytes, not an EIP/video bandwidth measurement. Timing samples are small and not a capacity claim.
Comparison logs: `pre-query-large-payload.log`, `query-and-safety.log`, `query-trimmed.log`, `query-comparison.json` under the ignored write-lock diagnostics directory.
## Verification
- Red reproduction: `submission-busy-red.log`, `lock-contention-red.log`, `metadata-list-red.log`.
- Task report: `local-data/test-results/verify-1794297878860663000/report.json`, frozen inputs. API recovery/permission/CAS checks, explicit Mongo lock and private snapshot checks, migrated and legacy query parity, 81-reader smoke and official Go baseline passed. The quick store group records opt-in skips; the relevant cases were explicitly enabled in task/integration runs, not counted as quick passes.
- Integration report: `local-data/test-results/verify-1794298148552521000/report.json`, frozen inputs: 19 commands passed, one browser command failed. Type/architecture, formatting, frontend and Go baseline gates, web build and anonymous production-bundle startup passed.
- The failed browser command contains passing `TestReworkBoundaryBrowserReplica` and failing `TestSlicePhaseBrowserReplica` (Delete slice button timeout). The same latter failure was already reproduced on untouched live source and recorded in the prior validation document. No known-failure baseline was changed; full integration is not green.
- Additional local Mongo tests cover live lease renewal, expired lease recovery, cancellation immediately after acquisition, preserved protection after actual writes, restart/backoff persistence, revoked users and old-worker fencing. A race run is recorded in `final-lock-race.log`.
- Local shared-helper/diagnostic changes require no migration, index or lock-mode change. No production records, resources or configuration were changed. No push, PR or deployment performed.
## Release and rollback bounds
All lock users must receive the fix for full coverage: API, annotation processor, validator and Worker. Updating only API leaves older lock-acquisition behavior in other writers. The mixed-version lock protocol remains compatible, but old processes can still retain unused locks; existing leases are not cleared by this patch.
Retry bookkeeping is optional BSON data and private JSON state. Rollback readers can consume queued jobs but lose the new transient recovery behavior. Preserve the prior annotation-loss guard, and retain current drafts/recovery evidence. The five-minute store-wait budget does not promise successful submission during a prolonged database outage. Production verification and release approval remain pending.

View File

@ -45,3 +45,11 @@
以上是单机本地 4 个 API handler、2×8 修订槽,15 分钟索引量级、预校验合成源、已领取任务的突发负载。没有覆盖真实 100 浏览器 HEVC 解码、生产网络/BOS、100 人同时新领取、完整送检/终审峰值或外部 Mongo 扩容。因此通过本地保存门槛不等于生产 100 人最终验收。
租约失效的安全边界沿用已有实现,不宣称能防止任意长进程暂停、网络分区或失效进程恢复后继续写的所有情形。未修改现有租约有效期、等待超时和数据库写入语义;这类故障需继续按现有恢复流程核实。
## 2026-11-10:取消获取锁时的释放边界
项目独占操作只有取得门锁和全部分区锁后才执行受保护业务。若在此之前被取消或超时,业务写入尚未开始,已取得的前置锁按原 owner 立即释放;不能把单纯锁获取失败当作未知业务写入,继续保留整项目租约。barrier 模式等待分区时同样处理。获取本身结果未知时,只尝试释放本次新 owner 的锁,不触碰其他 owner。
受保护业务一旦开始,仍保留原有未知写入保护:上下文取消、网络错误或数据库超时不提前放锁,按既有租约恢复。不更改锁模式、分区数、锁顺序、等待上限、租约时长、删除和撤权边界。
隔离 Mongo 回归覆盖取消在最后一个分区前、barrier 门锁释放、实际写入后结果未知仍保留 33 把锁,以及原有分区/删除/撤权并发契约。送检的有限重试与事故调查边界见 [送检恢复保护](../operations/submission-recovery-guards.md)。

View File

@ -23,3 +23,15 @@
调查时修订 2、3 的 BOS 清单返回 404,桶未开启版本控制;Mongo 未提供操作日志,也没有现成备份/PITR 入口。原浏览器已刷新或关闭。仍存的返工基线是退回时内容,不是刚完成的返工结果;当前访问范围内未找到可证实的完整最新快照。没有覆盖、回滚或恢复任何生产客户记录。
恢复仍需云数据库备份或其他独立副本证据;取得候选后应先离线比对身份、轮次、版本和时间,再单独批准写入,禁止直接以旧退回基线覆盖现有内容。
## 写锁集中繁忙的后续修复
隔离 Mongo 已复现:项目操作在取得最后一个分区锁前超时,业务尚未执行,但旧实现会遗留已取得的门锁及分区锁,等默认两分钟租约过期。一次短暂拥堵因此可能扩大为同项目其他任务连续等待五秒后报繁忙。修复仅释放确认尚未开始业务的前置锁;实际写入后的未知结果仍按原规则保锁。
异步送检将数据库繁忙、网络/数据库超时的原始错误类型保留到处理器,不再从响应文本猜测原因。同一送检作业保留输入、编号、预约和来源指纹,以 2/4/8/16 秒、之后最多 30 秒间隔重试,五分钟等待预算;等待不消耗崩溃恢复次数。等待信息以可选 BSON 字段持久化且不进入公开 JSON。每次重试重新检查归属、账号、权限和源版本;正式送检已生效时只补齐已有的任务投影,不再生成第二次送检。业务冲突、撤权及输入错误不自动重试。五分钟内仍未恢复则保留输入并按原失败恢复入口处理,不承诺数据库持续不可用时仍能送检成功。
查询优化前后的 Git 对照证明锁获取/保留及处理器永久失败逻辑未在近期查询优化中新增。事故窗口同时存在查询及连接超时,随后出现约两分钟的集中繁忙;缺少当时的锁持有者记录,不能把已复现机制等同于已证明具体请求的完整因果链,也不能排除新增读取压力是触发因素。
同一批 420 条大载荷合成记录对照中,近期版本将搜索专用 `poolSearch.text/source` 连同列表重复读回。此次仅精简 Mongo 列表与候选投影中的这两项;仍保留显示名称、版本/更新时间有效性证据和旧记录回退,服务端过滤、权限及可领取判断不变。该对照不代表生产容量或混合写入性能验收。
完整覆盖需同时升级使用共享锁的 API、修订处理器、校验器及 Worker;仅升级 API 不会改变其他旧写入者的锁获取行为。此次不自动清理已有生产锁,也不更改锁协议或租约时长。

View File

@ -27,6 +27,7 @@ import (
)
type asynchronousAnnotationResponseWriter struct {
cause error
header http.Header
status int
body strings.Builder
@ -2723,14 +2724,14 @@ func (server *server) handleCombinedAnnotationSubmission(writer http.ResponseWri
func (server *server) completeCombinedAnnotationSubmission(writer http.ResponseWriter, request *http.Request, task *domain.AnnotationTaskRecord, humanItem *domain.HumanItemRecord, payload combinedAnnotationSubmissionInput, principal *authenticatedPrincipal, submissionRef string, resumed bool) {
if err := server.assertAnnotationLease(request.Context()); err != nil {
renderError(writer, http.StatusConflict, err)
renderAnnotationSubmissionError(writer, http.StatusConflict, err)
return
}
sliceRevisions, sliceErr := server.store.ListSliceRevisions(request.Context(), task.OrganizationID, task.ID)
annotationRevisions, annotationErr := server.store.ListAnnotationRevisions(request.Context(), task.OrganizationID, task.ID)
reviews, reviewErr := server.store.ListAnnotationReviews(request.Context(), task.OrganizationID, task.ID)
if sliceErr != nil || annotationErr != nil || reviewErr != nil {
renderError(writer, http.StatusInternalServerError, fmt.Errorf("load combined annotation revisions: slices=%v annotations=%v reviews=%v", sliceErr, annotationErr, reviewErr))
renderAnnotationSubmissionError(writer, http.StatusInternalServerError, fmt.Errorf("load combined annotation revisions: slices=%v annotations=%v reviews=%v", sliceErr, annotationErr, reviewErr))
return
}
sliceID := "slice-" + submissionRef
@ -2767,10 +2768,14 @@ func (server *server) completeCombinedAnnotationSubmission(writer http.ResponseW
annotationRevision := domain.AnnotationRevisionRecord{ID: annotationID, OrganizationID: task.OrganizationID, TaskID: task.ID, Revision: revisionNumber, TemplateVersion: task.TemplateVersion, Segments: payload.Segments, ObjectHints: payload.ObjectHints, AuthorUserID: principal.User.ID, ChangeReason: submissionRef + ":" + strings.TrimSpace(payload.ChangeReason), IsRework: isRework, CountsAsOutput: !isRework, DurationSeconds: payload.DurationSeconds, CreatedAt: now}
materialization, materializationErr := server.materializeCombinedSliceSubmission(request.Context(), *task, *humanItem, payload, revisionNumber)
if request.Context().Err() != nil {
renderError(writer, http.StatusConflict, request.Context().Err())
renderAnnotationSubmissionError(writer, http.StatusConflict, request.Context().Err())
return
}
if materializationErr != nil {
if transientAnnotationStoreError(materializationErr) {
renderAnnotationSubmissionError(writer, http.StatusServiceUnavailable, materializationErr)
return
}
previousVersion := humanItem.Version
humanItem.PendingCombinedSubmissionID = ""
humanItem.PendingCombinedPayloadSHA = ""
@ -2797,7 +2802,7 @@ func (server *server) completeCombinedAnnotationSubmission(writer http.ResponseW
if humanItem.ReviewRound > 0 {
baseline, baselineErr := server.store.GetReworkBaseline(request.Context(), store.ReworkBaselineID(humanItem.ID, humanItem.ReviewRound))
if baselineErr != nil {
renderError(writer, http.StatusInternalServerError, baselineErr)
renderAnnotationSubmissionError(writer, http.StatusInternalServerError, baselineErr)
return
}
if baseline != nil {
@ -2806,16 +2811,16 @@ func (server *server) completeCombinedAnnotationSubmission(writer http.ResponseW
}
}
if err := server.store.UpsertSliceRevision(request.Context(), sliceRevision); err != nil {
renderError(writer, http.StatusInternalServerError, err)
renderAnnotationSubmissionError(writer, http.StatusInternalServerError, err)
return
}
if err := server.store.UpsertAnnotationRevision(request.Context(), annotationRevision); err != nil {
renderError(writer, http.StatusInternalServerError, err)
renderAnnotationSubmissionError(writer, http.StatusInternalServerError, err)
return
}
previousVersion := humanItem.Version
if err := server.assertAnnotationLease(request.Context()); err != nil {
renderError(writer, http.StatusConflict, err)
renderAnnotationSubmissionError(writer, http.StatusConflict, err)
return
}
humanItem.CurrentRevision = annotationRevision.Revision
@ -2828,7 +2833,7 @@ func (server *server) completeCombinedAnnotationSubmission(writer http.ResponseW
humanItem.AcceptedByUserID = ""
humanItem.RejectedAt = ""
if err := server.freezeSubmissionAcceptancePolicy(request.Context(), humanItem); err != nil {
renderError(writer, http.StatusInternalServerError, err)
renderAnnotationSubmissionError(writer, http.StatusInternalServerError, err)
return
}
humanItem.LatestAuthorUserID = principal.User.ID
@ -2851,7 +2856,7 @@ func (server *server) completeCombinedAnnotationSubmission(writer http.ResponseW
updated, updateErr := server.store.UpdateHumanItemVersioned(request.Context(), *humanItem, previousVersion)
if updateErr != nil || !updated {
if updateErr != nil {
renderError(writer, http.StatusInternalServerError, updateErr)
renderAnnotationSubmissionError(writer, http.StatusInternalServerError, updateErr)
} else {
renderJSON(writer, http.StatusConflict, map[string]string{"error": "Human item version conflict", "code": "version_conflict"})
}
@ -2864,7 +2869,7 @@ func (server *server) completeCombinedAnnotationSubmission(writer http.ResponseW
task.AssignedUserID = ""
task.UpdatedAt = now
if err := server.store.UpsertAnnotationTask(request.Context(), *task); err != nil {
renderError(writer, http.StatusInternalServerError, err)
renderAnnotationSubmissionError(writer, http.StatusInternalServerError, err)
return
}
server.cleanupReplacedAnnotationData(request.Context(), *task, humanItem.ID, annotationRevision.Revision)

View File

@ -314,6 +314,14 @@ func (s *server) executeAnnotationJob(parent context.Context, claimed domain.Hum
}
e = fmt.Errorf("源文件预校验等待超过 %s,编辑内容已保留;请确认校验进程在运行后重试", sourceValidationWaitLimit)
}
if j.Kind == "submit" && transientAnnotationStoreError(e) {
if deferForAnnotationStore(&j, time.Now()) {
if saved, saveErr := queue.SaveAnnotationProcessing(finishCtx, claimed.ID, j); saveErr != nil || !saved {
log.Printf("[annotation-processor] submission retry persistence unavailable task=%s", j.TaskID)
}
return
}
}
j.Status = "completed"
j.Stage = "completed"
j.CompletedSlices = j.TotalSlices
@ -454,6 +462,9 @@ func (s *server) processAnnotationJob(ctx context.Context, h domain.HumanItemRec
}
s.completeCombinedAnnotationSubmission(w, r, t, &h, p, principal, j.ID, j.Attempt > 1)
if w.status >= 400 {
if w.cause != nil {
return fmt.Errorf("annotation submission failed: %w", w.cause)
}
return fmt.Errorf("annotation submission failed: %s", w.body.String())
}
default:

View File

@ -0,0 +1,129 @@
package api
import (
"context"
"encoding/json"
"fmt"
"net/http"
"strings"
"sync/atomic"
"testing"
"time"
"droboticflow/internal/config"
"droboticflow/internal/domain"
"droboticflow/internal/store"
)
type transientSubmissionBusyStore struct {
*store.MemoryStore
sliceWrites atomic.Int32
failAt string
}
func (s *transientSubmissionBusyStore) UpsertSliceRevision(ctx context.Context, r domain.SliceRevisionRecord) error {
if (s.failAt == "" || s.failAt == "revision") && s.sliceWrites.Add(1) == 1 {
return store.ErrAccountMutationBusy
}
return s.MemoryStore.UpsertSliceRevision(ctx, r)
}
func (s *transientSubmissionBusyStore) UpdateHumanItemVersioned(ctx context.Context, h domain.HumanItemRecord, v int) (bool, error) {
if s.failAt == "handoff" && h.CurrentStageID == domain.FanyuV58StageQualityReview && s.sliceWrites.Add(1) == 1 {
return false, store.ErrAccountMutationBusy
}
return s.MemoryStore.UpdateHumanItemVersioned(ctx, h, v)
}
func (s *transientSubmissionBusyStore) UpsertAnnotationTask(ctx context.Context, r domain.AnnotationTaskRecord) error {
if s.failAt == "published" && r.Status == domain.AnnotationTaskStatusPendingQuality && s.sliceWrites.Add(1) == 1 {
return store.ErrAccountMutationBusy
}
return s.MemoryStore.UpsertAnnotationTask(ctx, r)
}
func TestSubmissionBusyRetainsReservationAndResumes(t *testing.T) {
for _, point := range []string{"revision", "handoff", "published", "revoked"} {
t.Run(point, func(t *testing.T) {
t.Setenv("ANNOTATION_PROCESSOR_MODE", "external")
f, root := annotationProcessorFixture(t)
faulty := &transientSubmissionBusyStore{MemoryStore: f.store, failAt: point}
if point == "revoked" {
faulty.failAt = "revision"
}
handler := NewHandlerWithConfig(faulty, nil, config.AppConfig{WorkspaceMountPath: root})
f.handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
r.Header.Set("Prefer", "respond-async")
handler.ServeHTTP(w, r)
})
r := f.request(http.MethodPost, "/api/plugins/annotation-qa/tasks/"+f.task.ID+"/combined-submissions", f.annotator, map[string]any{"version": f.item.Version, "sliceSegments": []domain.SliceSegment{{ID: "s", Name: "测试切片", Start: 0, End: 4}}, "segments": []domain.AnnotationSegment{{ID: "a", Start: 0, End: 4, Description: "返工完成的内容"}}})
if r.Code != 202 {
t.Fatal(r.Code, r.Body.String())
}
now := time.Now().UTC()
claimed, e := f.store.ClaimAnnotationProcessing(f.ctx, "worker-1", now.Format(time.RFC3339Nano), now.Add(time.Minute).Format(time.RFC3339Nano))
if e != nil || claimed == nil {
t.Fatal("not queued", e)
}
processor := &server{store: faulty, workspaceMountPath: root, externalAnnotationProcessor: true}
processor.executeAnnotationJob(f.ctx, *claimed)
after, _ := f.store.GetHumanItem(f.ctx, f.item.ID)
if after.ProcessingJob.Status != "queued" || (point != "published" && after.PendingCombinedSubmissionID != claimed.ProcessingJob.ID) || len(after.ProcessingJob.Input) == 0 {
t.Fatalf("temporary lock contention permanently failed submission: status=%s reservation=%q inputFields=%d", after.ProcessingJob.Status, after.PendingCombinedSubmissionID, len(after.ProcessingJob.Input))
}
if point == "revoked" {
user := f.annotator
user.Status = "disabled"
if e := f.store.UpsertUser(f.ctx, user); e != nil {
t.Fatal(e)
}
}
// A fresh processor instance must resume the same durable input, not create a
// second submission or require the annotator to reconstruct their work.
later := now.Add(5 * time.Minute)
retry, e := f.store.ClaimAnnotationProcessing(f.ctx, "worker-2", later.Format(time.RFC3339Nano), later.Add(time.Minute).Format(time.RFC3339Nano))
if e != nil || retry == nil {
t.Fatal("retry not claimable", e)
}
(&server{store: faulty, workspaceMountPath: root, externalAnnotationProcessor: true}).executeAnnotationJob(f.ctx, *retry)
after, _ = f.store.GetHumanItem(f.ctx, f.item.ID)
if point == "revoked" {
if after.ProcessingJob.Status != "failed" || after.CurrentStageID == domain.FanyuV58StageQualityReview || len(after.ProcessingJob.Input) == 0 {
t.Fatal("retry bypassed revoked authorization or lost input")
}
return
}
if after.ProcessingJob.Status != "completed" || after.CurrentStageID != domain.FanyuV58StageQualityReview {
t.Fatalf("retry failed: %+v", after.ProcessingJob)
}
revisions, _ := f.store.ListAnnotationRevisions(f.ctx, f.task.OrganizationID, f.task.ID)
if len(revisions) != 1 || len(revisions[0].Segments) != 1 || revisions[0].Segments[0].Description != "返工完成的内容" {
t.Fatal("retry lost or duplicated submitted work")
}
})
}
}
func TestSubmissionBusyBackoffIsBoundedAndPrivate(t *testing.T) {
now := time.Now().UTC()
j := domain.AnnotationProcessingJob{Attempt: 1, Kind: "submit"}
for n := 0; n < 8; n++ {
j.Attempt++
if !deferForAnnotationStore(&j, now.Add(time.Duration(n)*time.Second)) {
t.Fatal("retry ended too early")
}
next, e := time.Parse(time.RFC3339Nano, j.NextAttemptAt)
if e != nil || next.Sub(now.Add(time.Duration(n)*time.Second)) > 30*time.Second || j.Status != "queued" {
t.Fatal("unbounded delay")
}
}
if deferForAnnotationStore(&j, now.Add(5*time.Minute)) {
t.Fatal("retry budget never expires")
}
if transientAnnotationStoreError(fmt.Errorf("authorization revoked")) || transientAnnotationStoreError(errCombinedSubmissionConflict) {
t.Fatal("business failures retried")
}
b, _ := json.Marshal(j)
if strings.Contains(string(b), "storeBusy") {
t.Fatal("private retry state leaked")
}
}

View File

@ -0,0 +1,51 @@
package api
import (
"context"
"droboticflow/internal/domain"
"droboticflow/internal/store"
"errors"
"go.mongodb.org/mongo-driver/mongo"
"net/http"
"time"
)
func transientAnnotationStoreError(err error) bool {
return errors.Is(err, store.ErrAccountMutationBusy) || errors.Is(err, context.DeadlineExceeded) || mongo.IsTimeout(err) || mongo.IsNetworkError(err)
}
// Preserve error identity across the internal HTTP-shaped submission handler.
// Text matching would confuse validation/ownership errors with safe retry work.
func renderAnnotationSubmissionError(w http.ResponseWriter, status int, err error) {
if response, ok := w.(*asynchronousAnnotationResponseWriter); ok {
response.cause = err
}
if transientAnnotationStoreError(err) {
renderCombinedSubmissionReservationError(w, err)
return
}
renderError(w, status, err)
}
// Durable, bounded retry: keep the same submission ID, input and reservation.
// Every new attempt revalidates ownership, permissions, source and publication.
func deferForAnnotationStore(j *domain.AnnotationProcessingJob, now time.Time) bool {
since, err := time.Parse(time.RFC3339Nano, j.StoreBusySince)
if err != nil {
since = now
j.StoreBusySince = now.UTC().Format(time.RFC3339Nano)
}
if now.Sub(since) >= 5*time.Minute {
return false
}
j.StoreBusyWaits++
delay := 30 * time.Second
if j.StoreBusyWaits <= 4 {
delay = 2 * time.Second << (j.StoreBusyWaits - 1)
}
j.Status, j.Stage, j.Error = "queued", "queued", ""
j.UpdatedAt = now.UTC().Format(time.RFC3339Nano)
j.NextAttemptAt = now.Add(delay).UTC().Format(time.RFC3339Nano)
j.Attempt = max(0, j.Attempt-1)
return true
}

View File

@ -21,6 +21,8 @@ type AnnotationProcessingJob struct {
LeaseUntil string `bson:"leaseUntil,omitempty" json:"-"`
NextAttemptAt string `bson:"nextAttemptAt,omitempty" json:"-"`
// Set while a revision waits for source validation; absent on other jobs.
StoreBusySince string `bson:"storeBusySince,omitempty" json:"-"`
StoreBusyWaits int `bson:"storeBusyWaits,omitempty" json:"-"`
ValidationSince string `bson:"validationSince,omitempty" json:"-"`
ValidationWaits int `bson:"validationWaits,omitempty" json:"-"`
Attempt int `bson:"attempt" json:"attempt"`

View File

@ -2,8 +2,10 @@ package store
import (
"context"
"crypto/sha256"
"errors"
"fmt"
"log"
"os"
"strconv"
"strings"
@ -50,6 +52,20 @@ func accountMutationLockLeaseDuration() time.Duration {
// verify, exactly as before; only the "must wait for an operator to notice
// and clear it by hand" half of the old design is gone.
func (s *MongoStore) withAccountMutationLock(ctx context.Context, key string, operation func(context.Context) (any, error)) (result any, err error) {
return s.withAccountMutationLockBeforeOperation(ctx, key, operation, nil)
}
// operationStarted is shared only by the synchronous lock-acquisition chain.
// Until its protected body starts, cancellation cannot hide a business write:
// acquired prerequisite locks may be released immediately. Once started (or
// for callers without this proof), preserve uncertain-write lease semantics.
func (s *MongoStore) withAccountMutationLockBeforeOperation(ctx context.Context, key string, operation func(context.Context) (any, error), operationStarted *bool) (result any, err error) {
startedAt := time.Now()
diagnose := func(event string) {
digest := sha256.Sum256([]byte(key))
log.Printf("[account-mutation-lock] event=%s resource=%s key_hash=%x elapsed_ms=%d", event, strings.SplitN(key, ":", 2)[0], digest[:6], time.Since(startedAt).Milliseconds())
}
col := s.db.Collection("accountMutationLocks")
_, e := col.InsertOne(ctx, bson.M{"_id": key, "locked": false})
if e != nil && !mongo.IsDuplicateKeyError(e) {
@ -57,6 +73,12 @@ func (s *MongoStore) withAccountMutationLock(ctx context.Context, key string, op
}
leaseDuration := accountMutationLockLeaseDuration()
owner := domain.NewID("mutation")
releaseOwned := func() (bool, error) {
releaseCtx, releaseCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer releaseCancel()
r, e := col.UpdateOne(releaseCtx, bson.M{"_id": key, "owner": owner, "locked": true}, bson.M{"$set": bson.M{"locked": false}, "$unset": bson.M{"owner": "", "process": "", "startedAt": "", "leaseExpiresAt": ""}})
return e == nil && r.MatchedCount == 1, e
}
waitCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
for {
@ -71,6 +93,12 @@ func (s *MongoStore) withAccountMutationLock(ctx context.Context, key string, op
}}
r, e := col.UpdateOne(waitCtx, filter, update)
if e != nil {
// No protected operation has run. Even an ambiguous acquisition
// may safely release only this fresh owner; never another owner.
if ctx.Err() != nil || mongo.IsTimeout(e) || mongo.IsNetworkError(e) {
_, _ = releaseOwned()
diagnose("acquisition_result_unknown")
}
return nil, e
}
if r.MatchedCount == 1 {
@ -78,6 +106,7 @@ func (s *MongoStore) withAccountMutationLock(ctx context.Context, key string, op
}
select {
case <-waitCtx.Done():
diagnose("acquisition_wait_expired")
return nil, ErrAccountMutationBusy
case <-time.After(25 * time.Millisecond):
}
@ -118,20 +147,25 @@ func (s *MongoStore) withAccountMutationLock(ctx context.Context, key string, op
}
}
}()
bodyStarted := false
defer func() {
cancelRenew()
<-renewalDone
// Do not unlock an uncertain write and admit a conflicting operation.
if ctx.Err() != nil || mongo.IsNetworkError(err) || mongo.IsTimeout(err) {
if bodyStarted && (operationStarted == nil || *operationStarted) && (ctx.Err() != nil || mongo.IsNetworkError(err) || mongo.IsTimeout(err)) {
diagnose("started_operation_lease_preserved")
return
}
releaseCtx, releaseCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer releaseCancel()
r, e := col.UpdateOne(releaseCtx, bson.M{"_id": key, "owner": owner, "locked": true}, bson.M{"$set": bson.M{"locked": false}, "$unset": bson.M{"owner": "", "process": "", "startedAt": "", "leaseExpiresAt": ""}})
if e != nil || r.MatchedCount != 1 {
released, e := releaseOwned()
if e != nil || !released {
diagnose("release_needs_verification")
err = fmt.Errorf("%w: operation finished but lock release needs verification", ErrAccountMutationBusy)
}
}()
if ctx.Err() != nil {
return nil, ctx.Err()
}
bodyStarted = true
return operation(ctx)
}

View File

@ -739,7 +739,7 @@ func humanItemMongoFilter(organizationID string, query domain.HumanItemQuery) bs
// video/artifact locators, duration — stays in the document.
func humanItemListProjection(query domain.HumanItemQuery) bson.M {
if query.PoolCandidateProjection {
return bson.M{"subject.payload": 0, "processingJob.input": 0}
return bson.M{"subject.payload": 0, "processingJob.input": 0, "poolSearch.text": 0, "poolSearch.source": 0}
}
if query.IdentityProjection {
return bson.M{"_id": 0, "id": 1}
@ -747,7 +747,7 @@ func humanItemListProjection(query domain.HumanItemQuery) bson.M {
if query.BufferSummaryProjection {
return bson.M{"_id": 0, "id": 1, "organizationId": 1, "projectId": 1, "workflowId": 1, "runId": 1, "batchId": 1, "pluginId": 1, "legacyTaskId": 1, "assignedUserId": 1, "currentStageId": 1, "status": 1, "version": 1, "reviewRound": 1, "claimedAt": 1, "pendingCombinedSubmissionId": 1, "processingJob.sourceFingerprint": 1, "processingJob.sourceIndex": 1, "subject.payload.materialization.sourceIndex": 1, "processingJob.status": 1, "processingJob.kind": 1, "processingJob.error": 1, "subject.payload.sliceWorkflow.status": 1, "subject.payload.sliceWorkflow.phase": 1}
}
projection := bson.M{"processingJob.input": 0}
projection := bson.M{"processingJob.input": 0, "poolSearch.text": 0, "poolSearch.source": 0}
for _, key := range humanItemPayloadKeysRemoved(query) {
projection["subject.payload."+key] = 0
}

View File

@ -13,7 +13,12 @@ import (
func humanPoolSearchPipeline(org string, q domain.HumanItemQuery, filter bson.M) mongo.Pipeline {
// Only search/category metadata participates in global sorts and joins.
// Hydrate matched IDs afterwards; the public record shape stays unchanged.
projection := bson.M{"version": 1, "updatedAt": 1, "poolSearch": 1, "_id": 0, "id": 1, "organizationId": 1, "pluginId": 1, "projectId": 1, "runId": 1, "legacyTaskId": 1, "createdAt": 1, "currentStageId": 1, "status": 1, "reviewRound": 1, "pendingCombinedSubmissionId": 1, "assignedUserId": 1, "subject.itemKey": 1, "subject.artifactId": 1, "subject.payload.sliceWorkflow.status": 1, "subject.payload.sliceWorkflow.phase": 1}
projection := bson.M{"version": 1, "updatedAt": 1, "_id": 0, "id": 1, "organizationId": 1, "pluginId": 1, "projectId": 1, "runId": 1, "legacyTaskId": 1, "createdAt": 1, "currentStageId": 1, "status": 1, "reviewRound": 1, "pendingCombinedSubmissionId": 1, "assignedUserId": 1, "subject.itemKey": 1, "subject.artifactId": 1, "subject.payload.sliceWorkflow.status": 1, "subject.payload.sliceWorkflow.phase": 1}
// Reverse lookup validity was checked before projection. The Go matcher
// needs display/legacy evidence, not duplicated regex text/source snapshots.
for _, field := range []string{"schema", "sourceVersion", "sourceUpdatedAt", "sourceItemKey", "sourceArtifactId", "payloadDirectories", "durationSeconds"} {
projection["poolSearch."+field] = 1
}
pipeline := mongo.Pipeline{{{Key: "$match", Value: filter}}, {{Key: "$project", Value: projection}}}
pipeline = append(pipeline,
// Number candidate batches; restore the stable createdAt/id order below.

View File

@ -41,3 +41,7 @@ func TestPoolSearchMetadataOwnsOnlyNames(t *testing.T) {
t.Fatal(err)
}
}
func TestHumanPoolSearchMongoMigratedLargePayload(t *testing.T) {
testHumanPoolSearchMongo(t, 420, true, true, true)
}

View File

@ -0,0 +1,195 @@
package store
import (
"context"
"droboticflow/internal/domain"
"fmt"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/event"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"os"
"strings"
"sync/atomic"
"testing"
"time"
)
func TestProjectLockCanceledAcquisitionReleasesUnusedLocks(t *testing.T) {
s, ctx := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "striped")
held, release := make(chan struct{}), make(chan struct{})
done := make(chan error, 1)
go func() {
_, e := s.withAccountMutationLock(ctx, supplierRecordStripeLockID("p", 31), func(context.Context) (any, error) { close(held); <-release; return nil, nil })
done <- e
}()
waitProjectTest(t, ctx, held)
deadline, stop := context.WithTimeout(ctx, 400*time.Millisecond)
invoked := false
_, e := s.withSupplierProjectLocks(deadline, []string{"p"}, func(context.Context) (any, error) { invoked = true; return nil, nil })
stop()
close(release)
if holderErr := <-done; holderErr != nil {
t.Fatal(holderErr)
}
if e == nil || invoked {
t.Fatal("incomplete acquisition entered operation", e)
}
n, err := s.db.Collection("accountMutationLocks").CountDocuments(ctx, bson.M{"locked": true})
if err != nil {
t.Fatal(err)
}
if n != 0 {
t.Fatalf("canceled acquisition left %d unused project locks blocking submissions until lease expiry", n)
}
}
func TestPoolSearchMetadataWriteCost(t *testing.T) {
if os.Getenv("SUBMISSION_LOCK_COST_TEST") != "1" {
t.Skip("synthetic metadata timing opt-in")
}
for _, count := range []int{100, 1000, 10000} {
actions := bson.A{}
for n := 0; n < count; n++ {
actions = append(actions, bson.M{"id": fmt.Sprint(n), "start": float64(n), "end": float64(n + 1), "description": strings.Repeat("synthetic ", 20)})
}
d := bson.M{"id": "synthetic-item", "version": 8, "updatedAt": "2026-11-09T00:00:00Z", "subject": bson.M{"itemKey": "synthetic/part0001", "payload": bson.M{"annotationWorkspace": bson.M{"segments": actions}}}}
raw, _ := bson.Marshal(d)
start := time.Now()
for n := 0; n < 10; n++ {
stampPoolSearchMetadata("humanItems", d)
}
t.Logf("actions=%d payload_bytes=%d added_metadata_mean_us=%d", count, len(raw), time.Since(start).Microseconds()/10)
}
}
func TestProjectLockStartedUncertainWriteKeepsProtection(t *testing.T) {
s, ctx := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "striped")
_, err := s.withSupplierProjectLocks(ctx, []string{"p"}, func(ctx context.Context) (any, error) {
if _, e := s.db.Collection("syntheticWrites").InsertOne(ctx, bson.M{"id": "committed-but-response-unknown"}); e != nil {
return nil, e
}
return nil, context.DeadlineExceeded
})
if err == nil {
t.Fatal("uncertain write reported success")
}
n, e := s.db.Collection("accountMutationLocks").CountDocuments(ctx, bson.M{"locked": true})
if e != nil || n != 33 {
t.Fatalf("started uncertain write lost its protection: locks=%d err=%v", n, e)
}
}
func TestProjectBarrierCanceledAcquisitionReleasesGate(t *testing.T) {
s, ctx := projectStripeTestStore(t)
t.Setenv("DROBOTICFLOW_PROJECT_WRITE_LOCK_MODE", "barrier")
id := ""
for n := 0; ; n++ {
id = fmt.Sprint(n)
if supplierRecordStripe("humanItems", id) == 31 {
break
}
}
held, release := make(chan struct{}), make(chan struct{})
done := make(chan error, 1)
go func() {
_, e := s.withAccountMutationLock(ctx, supplierRecordStripeLockID("p", 31), func(context.Context) (any, error) { close(held); <-release; return nil, nil })
done <- e
}()
waitProjectTest(t, ctx, held)
deadline, stop := context.WithTimeout(ctx, 100*time.Millisecond)
invoked := false
_, e := s.withSupplierRecordWriteLock(deadline, "p", "humanItems", id, func(context.Context) (any, error) { invoked = true; return nil, nil })
stop()
close(release)
if holderErr := <-done; holderErr != nil {
t.Fatal(holderErr)
}
if e == nil || invoked {
t.Fatal("incomplete barrier acquisition entered operation", e)
}
n, err := s.db.Collection("accountMutationLocks").CountDocuments(ctx, bson.M{"locked": true})
if err != nil || n != 0 {
t.Fatalf("canceled barrier left locks=%d err=%v", n, err)
}
}
func TestSubmissionRetryMongoStateSurvivesRestart(t *testing.T) {
s, ctx := projectStripeTestStore(t)
now := time.Now().UTC()
j := domain.AnnotationProcessingJob{ID: "submission", Kind: "submit", Status: "running", Owner: "old-worker", Input: map[string]any{"segments": []any{map[string]any{"description": "synthetic saved work"}}}, Attempt: 1}
_, e := s.humanItems.InsertOne(ctx, domain.HumanItemRecord{ID: "item", ProcessingJob: &j})
if e != nil {
t.Fatal(e)
}
j.Status = "queued"
j.StoreBusySince = now.Format(time.RFC3339Nano)
j.StoreBusyWaits = 2
j.NextAttemptAt = now.Add(4 * time.Second).Format(time.RFC3339Nano)
if ok, e := s.SaveAnnotationProcessing(ctx, "item", j); e != nil || !ok {
t.Fatal(ok, e)
}
restarted := NewMongoStore(s.db)
h, e := restarted.GetHumanItem(ctx, "item")
if e != nil || h == nil || h.ProcessingJob.StoreBusySince != j.StoreBusySince || h.ProcessingJob.StoreBusyWaits != 2 || len(h.ProcessingJob.Input) == 0 {
t.Fatal("retry state lost on restart", e)
}
if h, e := restarted.ClaimAnnotationProcessing(ctx, "new-worker", now.Format(time.RFC3339Nano), now.Add(time.Minute).Format(time.RFC3339Nano)); e != nil || h != nil {
t.Fatal("backoff ignored", e)
}
later := now.Add(5 * time.Second)
h, e = restarted.ClaimAnnotationProcessing(ctx, "new-worker", later.Format(time.RFC3339Nano), later.Add(time.Minute).Format(time.RFC3339Nano))
if e != nil || h == nil || h.ProcessingJob.ID != j.ID || h.ProcessingJob.StoreBusyWaits != 2 || len(h.ProcessingJob.Input) == 0 {
t.Fatal("same input did not resume", e)
}
}
func TestPoolListDoesNotReadSearchOnlyMetadata(t *testing.T) {
s, ctx := projectStripeTestStore(t)
m := domain.HumanPoolSearchMetadata{Schema: 1, SourceVersion: 3, SourceUpdatedAt: "now", SourceItemKey: "cap/part0001", Text: []string{"cap", "part0001"}, Source: map[string]any{"0": 3}, PayloadDirectories: []string{"cap"}}
_, e := s.humanItems.InsertOne(ctx, domain.HumanItemRecord{ID: "i", OrganizationID: "org", Version: 3, UpdatedAt: "now", Subject: domain.HumanSubjectRef{ItemKey: "cap/part0001"}, PoolSearch: &m})
if e != nil {
t.Fatal(e)
}
for _, q := range []domain.HumanItemQuery{{ListProjection: true}, {PoolCandidateProjection: true}} {
rows, e := s.ListHumanItems(ctx, "org", q)
if e != nil || len(rows) != 1 {
t.Fatal("list failed", e)
}
got := rows[0]
if !got.PoolSearchCurrent() || len(got.PoolSearch.PayloadDirectories) != 1 {
t.Fatal("display/legacy validity evidence removed")
}
if len(got.PoolSearch.Text) != 0 || len(got.PoolSearch.Source) != 0 {
t.Fatal("list redundantly reads search-only metadata")
}
}
}
func TestProjectLockCanceledAfterAcquisitionDoesNotEnterBody(t *testing.T) {
s, ctx := projectStripeTestStore(t)
acquire, cancel := context.WithCancel(ctx)
defer cancel()
var once atomic.Bool
client, e := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://127.0.0.1:27018/?directConnection=true").SetMonitor(&event.CommandMonitor{Succeeded: func(_ context.Context, e *event.CommandSucceededEvent) {
if e.CommandName == "update" && once.CompareAndSwap(false, true) {
cancel()
}
}}))
if e != nil {
t.Fatal(e)
}
defer client.Disconnect(context.Background())
other := NewMongoStore(client.Database(s.db.Name()))
invoked := false
_, e = other.withAccountMutationLock(acquire, "cancel-after-acquire", func(context.Context) (any, error) { invoked = true; return nil, nil })
if e == nil || invoked {
t.Fatal("canceled operation entered protected body", e)
}
n, err := s.db.Collection("accountMutationLocks").CountDocuments(ctx, bson.M{"locked": true})
if err != nil || n != 0 {
t.Fatalf("canceled unstarted acquisition retained lock=%d err=%v", n, err)
}
}

View File

@ -76,18 +76,23 @@ func (s *MongoStore) withSupplierRecordWriteLock(ctx context.Context, project, c
if mongo.SessionFromContext(ctx) != nil {
return nil, fmt.Errorf("supplier write lock must precede transaction")
}
operationStarted := false
acquire := func(ctx context.Context) (any, error) {
return s.withAccountMutationLock(ctx, supplierRecordStripeLockID(project, stripe), func(ctx context.Context) (any, error) {
return s.withAccountMutationLockBeforeOperation(ctx, supplierRecordStripeLockID(project, stripe), func(ctx context.Context) (any, error) {
next := make(map[string]int, len(held)+1)
for k, v := range held {
next[k] = v
}
next[project] = stripe
if ctx.Err() != nil {
return nil, ctx.Err()
}
operationStarted = true
return fn(context.WithValue(ctx, supplierRecordStripeKey{}, next))
})
}, &operationStarted)
}
if gates, _ := ctx.Value(supplierProjectGateKey{}).(map[string]bool); mode == "barrier" && !gates[project] {
return s.withAccountMutationLock(ctx, "supplier-project-write:"+project, acquire)
return s.withAccountMutationLockBeforeOperation(ctx, "supplier-project-write:"+project, acquire, &operationStarted)
}
return acquire(ctx)
}
@ -127,11 +132,16 @@ func (s *MongoStore) withSupplierProjectLocks(ctx context.Context, ids []string,
if e != nil {
return nil, e
}
operationStarted := false
ordered := append([]string(nil), ids...)
sort.Strings(ordered)
var acquire func(context.Context, int) (any, error)
acquire = func(ctx context.Context, index int) (any, error) {
if index == len(ordered) {
if ctx.Err() != nil {
return nil, ctx.Err()
}
operationStarted = true
return fn(ctx)
}
id := ordered[index]
@ -152,7 +162,7 @@ func (s *MongoStore) withSupplierProjectLocks(ctx context.Context, ids []string,
if mongo.SessionFromContext(ctx) != nil {
return nil, fmt.Errorf("supplier write lock must precede transaction")
}
return s.withAccountMutationLock(ctx, "supplier-project-write:"+id, func(ctx context.Context) (any, error) {
return s.withAccountMutationLockBeforeOperation(ctx, "supplier-project-write:"+id, func(ctx context.Context) (any, error) {
enter := func(ctx context.Context) (any, error) {
next := make(map[string]bool, len(held)+1)
for k, v := range held {
@ -169,10 +179,10 @@ func (s *MongoStore) withSupplierProjectLocks(ctx context.Context, ids []string,
if stripe == supplierProjectStripeCount {
return enter(ctx)
}
return s.withAccountMutationLock(ctx, supplierRecordStripeLockID(id, stripe), func(ctx context.Context) (any, error) { return lockStripe(ctx, stripe+1) })
return s.withAccountMutationLockBeforeOperation(ctx, supplierRecordStripeLockID(id, stripe), func(ctx context.Context) (any, error) { return lockStripe(ctx, stripe+1) }, &operationStarted)
}
return lockStripe(ctx, 0)
})
}, &operationStarted)
}
return acquire(ctx, 0)
}

View File

@ -487,7 +487,9 @@
"paths": [
"^internal/api/annotation_(submission_recovery|current|operations|processor)(.*)\\.go$",
"^internal/store/human_event_recovery.*\\.go$",
"^docs/operations/submission-recovery.*\\.md$"
"^docs/operations/submission-recovery.*\\.md$",
"^internal/api/annotation_submission_(busy_test|retry)\\.go$",
"^internal/store/(account_mutation_lock|supplier_project_stripes|submission_lock_contention).*\\.go$"
],
"frontend": [
"tests/frontend/annotation-submission-state.test.mjs",
@ -503,7 +505,8 @@
"^TestFailedSliceRetry",
"^TestReconfirmSaved",
"^TestInvalidatedSubmission",
"^TestCombinedSubmissionFinalCAS"
"^TestCombinedSubmissionFinalCAS",
"^TestSubmissionBusy"
],
"./internal/store": ["^TestSubmissionRecoveryEventMemory"]
},
@ -524,6 +527,38 @@
},
"tests": true
},
{
"argv": [
"go",
"test",
"./internal/store",
"-run",
"^TestProjectLock|^TestProjectBarrier|^TestProjectWriteStripes(Cancelled|Collision|CrashRecovery|Default|Deletion|Exclusive|Independent|Legacy|Nested|NonStriped|Revocation|Supplier|Version)|^TestSubmissionRetryMongoState|^TestPoolListDoesNotReadSearchOnlyMetadata",
"-count=1",
"-timeout=3m",
"-v"
],
"env": {
"FANYU_PROJECT_WRITE_LOCK_TEST": "1"
},
"tests": true
},
{
"argv": [
"go",
"test",
"./internal/store",
"-run",
"^TestHumanPoolSearchMongoMigratedLargePayload$",
"-count=1",
"-timeout=2m",
"-v"
],
"env": {
"POOL_LOCAL_MONGO_URI": "mongodb://127.0.0.1:27018"
},
"tests": true
},
{
"argv": [
"go",