⚡ Fix O(targets×runs) nested substring scan in deletion-reference validation #12

Merged
longtao.wu merged 1 commits from fix/project-deletion-reference-check-complexity into dev 2026-09-22 21:39:53 +08:00
Owner

Follow-up to #11. The org-scoping fix got the queries in validateDeletionReferences down to one organization, but SL-data/BLF-data deletion previews still hung for minutes. Live debug instrumentation on Fanyu found the real cost: ~2,600 BOS targets × ~8,000 other runs (~30KB marshaled JSON each) = ~21M strings.Contains pair-wise scans over ~240MB of JSON, confirmed pegging the api pod at 1 full core for minutes.

Replaced the O(targets×runs) nested loop with a single combined regexp (RE2, linear in text length regardless of alternative count) scanned once per other-run, collecting referenced URIs into a set. Same exact-substring semantics as before — verified with a new test (TestProjectDeletionBlocksTargetReferencedByOtherRun) plus a local benchmark at SL-data's real scale: 241ms, down from a multi-minute hang.

Tests: go build ./..., go vet ./..., go test ./internal/api/... ./internal/store/... (all deletion tests pass, 75 pre-existing unrelated failures unchanged).

🤖 Generated with Claude Code

Follow-up to #11. The org-scoping fix got the queries in validateDeletionReferences down to one organization, but SL-data/BLF-data deletion previews still hung for minutes. Live debug instrumentation on Fanyu found the real cost: ~2,600 BOS targets × ~8,000 other runs (~30KB marshaled JSON each) = ~21M strings.Contains pair-wise scans over ~240MB of JSON, confirmed pegging the api pod at 1 full core for minutes. Replaced the O(targets×runs) nested loop with a single combined regexp (RE2, linear in text length regardless of alternative count) scanned once per other-run, collecting referenced URIs into a set. Same exact-substring semantics as before — verified with a new test (TestProjectDeletionBlocksTargetReferencedByOtherRun) plus a local benchmark at SL-data's real scale: 241ms, down from a multi-minute hang. Tests: go build ./..., go vet ./..., go test ./internal/api/... ./internal/store/... (all deletion tests pass, 75 pre-existing unrelated failures unchanged). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
longtao.wu added 1 commit 2026-09-22 21:39:49 +08:00
The scoping fix in #11 got validateDeletionReferences's queries down to a
single organization, but SL-data/BLF-data deletion previews still hung
for minutes. Live debug instrumentation on Fanyu found the real cost:
for a project like SL-data, buildProjectDeletionPlan produces ~2,600 BOS
deletion targets, and the org has ~8,000 other runs averaging ~30KB of
marshaled JSON each. The old code did a strings.Contains scan of every
other run's JSON for every target — ~21M pair-wise substring searches
over ~240MB of JSON, which is what actually burned the CPU (confirmed
via `kubectl top`: pegged at 1 full core for minutes).

Replaced the O(targets×runs) nested loop with a single combined regexp
(Go's regexp is RE2-based, so an alternation of N literal patterns still
matches in time linear in the text length) scanned once per other-run,
collecting which target URIs are actually referenced. The boolean result
for every (target, run) pair is unchanged — same exact-substring
semantics as the original strings.Contains check — just computed once
per distinct URI/run instead of once per pair. Added
TestProjectDeletionBlocksTargetReferencedByOtherRun to lock in that the
safety check still blocks a target another run's artifact references and
still allows one nothing references.

Local benchmark at SL-data's real scale (8,000 other runs, 2,600 BOS
targets): 241ms, down from a multi-minute hang.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
longtao.wu merged commit 74125862f6 into dev 2026-09-22 21:39:53 +08:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: longtao.wu/droboticflow#12
No description provided.