An organization's run history grows unbounded and nothing ever shrinks
it — completed/failed/cancelled runs stay in the runs collection
forever, only removed if their whole project is later deleted. That
made every org-wide scan (most notably validateDeletionReferences'
deletion-safety check, and any other query through
ListRunArtifactsByOrganization) get proportionally slower over time as
an org's run count kept climbing, independent of the size of whatever
was actually being operated on. Live on Fanyu this showed up as project
deletion getting slower every day, driven entirely by an unrelated
capture-processing backlog continuously adding new runs to the same
organization.
Adds RunRecord.ArchivedAt and a periodic worker sweep
(archiveTerminalRuns, mirroring the existing orphan-runtime
reconciliation's throttle pattern) that marks completed/failed/cancelled
runs whose EndedAt is older than WORKER_RUN_ARCHIVE_MIN_AGE_SECONDS
(default 30 days) as archived, checking at most every
WORKER_RUN_ARCHIVE_CHECK_INTERVAL_SECONDS (default 1 hour). Archiving
never deletes anything — ListRunArtifactsByOrganization now excludes
archived runs so the hot-path org-wide scans stay bounded, but
ListRunArtifactsForProject (used when a project is actually being
deleted) still includes them, since a project's own deletion must still
clean up every run it ever had regardless of archive state. A run's
data is only actually removed when its project is deleted, unchanged
from today.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two platform bugs surfaced by the SAM3 video-operator run:
P6 — workflow PUT reverted mid-run (lost update). The worker held a workflow
snapshot for the whole run duration and bumped run state via UpsertWorkflow
(full-document $set), clobbering concurrent PUT edits to Nodes — e.g. a node's
containerCommand reverted to its pre-run value while a long GPU run was in
flight. Add store.UpdateWorkflowRunState, a field-level $set of only
status/lastRunId/updatedAt, and switch the 5 worker run-state writes to it.
PUT edits to Nodes/Edges now survive a long-running workflow.
P7 — run.status "queued" while the pod was already running (reconcile race).
RunStatusRunning + CurrentNodeID are persisted BEFORE the k8s job is created
(async dispatch goroutine). reconcileActiveRuns had a freshness guard only on
the Queued branch; a freshly-dispatched Running run was Inspect→JobStateMissing
→ re-queued + its nodeExecution dropped, while the pod kept running (orphan,
GPU still 44-91%). Add the inverse freshness guard for the Running branch
(skip Inspect only while the active nodeExecution is fresh) and clean up
orphan pods via TerminateRun before re-queueing in the Missing branch.
- store.go/mongo.go/memory.go: UpdateWorkflowRunState field-level update
- runtime.go: Running-branch freshness guard; TerminateRun on JobStateMissing;
5 run-state writes use UpdateWorkflowRunState; jobLauncher + testLauncher
gain TerminateRun
Co-Authored-By: Claude <noreply@anthropic.com>