⚡ Add an expiring, renewed lease to the account mutation lock #18

Merged
longtao.wu merged 1 commits from fix/account-mutation-lock-lease into dev 2026-09-23 14:38:19 +08:00
Owner

Hit three times live on Fanyu in one session: the account-mutation lock (guards account/project-member/supplier writes) had no expiring lease by design, so a worker pod replacement mid-hold (a routine rolling deploy) left the lock permanently stuck, requiring a manual mongosh intervention each time.

Adds a renewed lease (ACCOUNT_MUTATION_LOCK_LEASE_SECONDS, default 2m), mirroring the project-deletion task lease's own renewal ticker already in this codebase. A live owner renews every leaseDuration/4 so its lease never lapses; only a lock whose owner stopped renewing (crashed/replaced) becomes reclaimable, once lapsed. Reclaiming assigns a fresh owner id, which already acts as a fencing token — the original holder's release (matched on owner) can never succeed afterward or touch the new holder's lock. Transient renewal errors don't give up early; only a confirmed loss does.

Tests: go build/vet clean, go test ./internal/api/... ./internal/store/... ./internal/worker/... — identical failure set to unmodified dev (verified via stash diff, baseline is 76, unrelated to this change), race detector clean on internal/store. Added TestAccountMutationLockLeaseReclaimAndRenewal (live-Mongo gated) covering normal round-trip, lapsed-lease reclaim, and renewed-lease blocking a concurrent reclaim.

🤖 Generated with Claude Code

Hit three times live on Fanyu in one session: the account-mutation lock (guards account/project-member/supplier writes) had no expiring lease by design, so a worker pod replacement mid-hold (a routine rolling deploy) left the lock permanently stuck, requiring a manual mongosh intervention each time. Adds a renewed lease (ACCOUNT_MUTATION_LOCK_LEASE_SECONDS, default 2m), mirroring the project-deletion task lease's own renewal ticker already in this codebase. A live owner renews every leaseDuration/4 so its lease never lapses; only a lock whose owner stopped renewing (crashed/replaced) becomes reclaimable, once lapsed. Reclaiming assigns a fresh owner id, which already acts as a fencing token — the original holder's release (matched on owner) can never succeed afterward or touch the new holder's lock. Transient renewal errors don't give up early; only a confirmed loss does. Tests: go build/vet clean, go test ./internal/api/... ./internal/store/... ./internal/worker/... — identical failure set to unmodified dev (verified via stash diff, baseline is 76, unrelated to this change), race detector clean on internal/store. Added TestAccountMutationLockLeaseReclaimAndRenewal (live-Mongo gated) covering normal round-trip, lapsed-lease reclaim, and renewed-lease blocking a concurrent reclaim. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
longtao.wu added 1 commit 2026-09-23 14:38:14 +08:00
The per-resource mutex behind account/project-member/supplier writes had
no expiring lease by design: a paused process must never resume after
another writer has stolen its lock, so recovery required an operator to
verify the owner had actually stopped and clear the lock by hand. In
practice, on Fanyu this meant every worker pod replacement (a routine
rolling deploy) mid-hold left the lock permanently stuck — hit three
times in one session, each requiring a manual mongosh intervention.

Adds a lease (ACCOUNT_MUTATION_LOCK_LEASE_SECONDS, default 2 minutes)
renewed by a background ticker for as long as the guarded operation
keeps running — mirroring the project-deletion task lease's own renewal
pattern already in this codebase. A live owner's lease is renewed well
before it can lapse (every leaseDuration/4), so it's never reclaimed out
from under it; only a lock whose owner stopped renewing entirely —
because it crashed or was replaced — becomes reclaimable, once its lease
lapses. Reclaiming always assigns a fresh owner id, so the original
holder's own release (matched on "owner": owner) can never succeed
afterward and can never touch the new holder's lock — the fresh id
already acts as the fencing token a naive TTL-lock would be missing. A
transient renewal error (network blip) doesn't give up the lease
early — only a confirmed "someone else now owns this" does. Genuine
crashes or ambiguous network results are still surfaced to the caller to
verify, exactly as before; only the "must wait for an operator to
notice and clear it by hand" half of the old design is gone.

Added TestAccountMutationLockLeaseReclaimAndRenewal (live-Mongo gated,
same convention as the existing index-migration/duplicate-key tests):
normal acquire/release still round-trips, a lapsed lease from a
simulated dead owner is reclaimable, and a live owner's renewed lease
blocks a concurrent reclaim attempt.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
longtao.wu merged commit f8941d59b4 into dev 2026-09-23 14:38:19 +08:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: longtao.wu/droboticflow#18
No description provided.