✨ Archive old terminal runs to bound org-wide scan cost #17

Merged
longtao.wu merged 1 commits from feat/archive-terminal-runs into dev 2026-09-23 14:02:54 +08:00
Owner

Found live on Fanyu: an org run count that only ever grows (nothing ever deletes runs except a full project deletion) was making the org-wide safety scan in validateDeletionReferences get slower every day, independent of what was actually being deleted — driven by an unrelated capture-processing backlog continuously adding runs.

Adds RunRecord.ArchivedAt + a periodic worker sweep (archiveTerminalRuns, same throttle pattern as the existing orphan-runtime reconciliation) marking completed/failed/cancelled runs older than WORKER_RUN_ARCHIVE_MIN_AGE_SECONDS (default 30d) as archived, checked at most every WORKER_RUN_ARCHIVE_CHECK_INTERVAL_SECONDS (default 1h). Archiving never deletes data: ListRunArtifactsByOrganization (the hot-path org-wide scan) now excludes archived runs, but ListRunArtifactsForProject (used by an actual project deletion) still includes them — a project deleting itself must still clean up every run it ever had. Data is only actually removed when its project is deleted, same as today.

Tests: go build/vet clean, go test ./internal/api/... ./internal/store/... ./internal/worker/... — new tests pass, no new failures vs the pre-existing baselines (confirmed the one worker-package failure reproduces identically on plain dev without this change).

🤖 Generated with Claude Code

Found live on Fanyu: an org run count that only ever grows (nothing ever deletes runs except a full project deletion) was making the org-wide safety scan in validateDeletionReferences get slower every day, independent of what was actually being deleted — driven by an unrelated capture-processing backlog continuously adding runs. Adds RunRecord.ArchivedAt + a periodic worker sweep (archiveTerminalRuns, same throttle pattern as the existing orphan-runtime reconciliation) marking completed/failed/cancelled runs older than WORKER_RUN_ARCHIVE_MIN_AGE_SECONDS (default 30d) as archived, checked at most every WORKER_RUN_ARCHIVE_CHECK_INTERVAL_SECONDS (default 1h). Archiving never deletes data: ListRunArtifactsByOrganization (the hot-path org-wide scan) now excludes archived runs, but ListRunArtifactsForProject (used by an actual project deletion) still includes them — a project deleting itself must still clean up every run it ever had. Data is only actually removed when its project is deleted, same as today. Tests: go build/vet clean, go test ./internal/api/... ./internal/store/... ./internal/worker/... — new tests pass, no new failures vs the pre-existing baselines (confirmed the one worker-package failure reproduces identically on plain dev without this change). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
longtao.wu added 1 commit 2026-09-23 14:02:49 +08:00
An organization's run history grows unbounded and nothing ever shrinks
it — completed/failed/cancelled runs stay in the runs collection
forever, only removed if their whole project is later deleted. That
made every org-wide scan (most notably validateDeletionReferences'
deletion-safety check, and any other query through
ListRunArtifactsByOrganization) get proportionally slower over time as
an org's run count kept climbing, independent of the size of whatever
was actually being operated on. Live on Fanyu this showed up as project
deletion getting slower every day, driven entirely by an unrelated
capture-processing backlog continuously adding new runs to the same
organization.

Adds RunRecord.ArchivedAt and a periodic worker sweep
(archiveTerminalRuns, mirroring the existing orphan-runtime
reconciliation's throttle pattern) that marks completed/failed/cancelled
runs whose EndedAt is older than WORKER_RUN_ARCHIVE_MIN_AGE_SECONDS
(default 30 days) as archived, checking at most every
WORKER_RUN_ARCHIVE_CHECK_INTERVAL_SECONDS (default 1 hour). Archiving
never deletes anything — ListRunArtifactsByOrganization now excludes
archived runs so the hot-path org-wide scans stay bounded, but
ListRunArtifactsForProject (used when a project is actually being
deleted) still includes them, since a project's own deletion must still
clean up every run it ever had regardless of archive state. A run's
data is only actually removed when its project is deleted, unchanged
from today.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
longtao.wu merged commit 9ad4bb9438 into dev 2026-09-23 14:02:54 +08:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: longtao.wu/droboticflow#17
No description provided.