⚡ Process deletion targets concurrently within each phase #14

Merged
longtao.wu merged 1 commits from feat/project-deletion-parallel-target-execution into dev 2026-09-23 04:30:33 +08:00
Owner

Follow-up to #11/#12/#13, requested live while BLF's deletion was running on Fanyu (real runtime phase taking tens of minutes to stop 4,641 runs one at a time). Replaces the per-target sequential loop in executeProjectDeletion with a bounded worker pool (PROJECT_DELETION_TARGET_CONCURRENCY, default 12) applied within each phase; phases (runtime -> workspace -> bos -> records) still run strictly in order, preserving the plan's intended cleanup ordering. Shared task state (target Status/Error, persist()) is mutex-guarded; the actual K8s/BOS/Mongo I/O runs unlocked so workers genuinely overlap.

Tests: go build ./..., go test ./internal/api/... ./internal/store/... -race (all pass, zero race warnings, 75 pre-existing unrelated failures unchanged). Added TestProjectDeletionExecutesTargetsConcurrentlyAndCompletes (30 concurrent runs) to lock in that concurrent execution reaches the same end state as the old sequential loop.

🤖 Generated with Claude Code

Follow-up to #11/#12/#13, requested live while BLF's deletion was running on Fanyu (real runtime phase taking tens of minutes to stop 4,641 runs one at a time). Replaces the per-target sequential loop in executeProjectDeletion with a bounded worker pool (PROJECT_DELETION_TARGET_CONCURRENCY, default 12) applied within each phase; phases (runtime -> workspace -> bos -> records) still run strictly in order, preserving the plan's intended cleanup ordering. Shared task state (target Status/Error, persist()) is mutex-guarded; the actual K8s/BOS/Mongo I/O runs unlocked so workers genuinely overlap. Tests: go build ./..., go test ./internal/api/... ./internal/store/... -race (all pass, zero race warnings, 75 pre-existing unrelated failures unchanged). Added TestProjectDeletionExecutesTargetsConcurrentlyAndCompletes (30 concurrent runs) to lock in that concurrent execution reaches the same end state as the old sequential loop. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
longtao.wu added 1 commit 2026-09-23 04:30:27 +08:00
executeProjectDeletion processed every target (runtime, then workspace,
then bos, then records) strictly one at a time — for a project with
thousands of real runs, that meant thousands of sequential K8s/BOS/Mongo
round trips. Observed live on Fanyu: BLF-data's runtime phase (stopping
4,641 runs) took tens of minutes at that pace.

Replaced the per-target sequential loop with a bounded worker pool
(PROJECT_DELETION_TARGET_CONCURRENCY, default 12) applied within each
phase — targets of the same kind now run concurrently, but phases still
run strictly in order (runtime, then workspace, then bos, then records),
preserving the "stop/verify runtime, then remove files, then DB"
ordering the plan is built with. The one-time phase-transition safety
checks (wait for in-flight human work, re-verify runtime absence,
revalidate references) still run exactly once, at the same point,
between the runtime and workspace phases.

Every read-modify-write of shared task state (a target's Status/Error,
and every persist() call) is guarded by a mutex; the actual I/O
(deleteProjectTarget) runs unlocked so workers genuinely overlap. Added
TestProjectDeletionExecutesTargetsConcurrentlyAndCompletes (30 runs) to
lock in that concurrent execution reaches the same end state — every
target done, every workspace directory removed, project record gone —
as the old sequential loop. Full suite passes under -race with no
warnings; 75 pre-existing unrelated failures unchanged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
longtao.wu merged commit 50683fc89d into dev 2026-09-23 04:30:33 +08:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: longtao.wu/droboticflow#14
No description provided.