bench dcf976dc95 recovery: hand a stopped collector's downtime back to its rollouts
A rollout whose collector died (handover, crash) is paused on the next start
(CONTROLLER_RESTARTED) with paused_at taken from execution.json's updated_at.
But reconcile_run first settles interrupted operations through the stage
Worker, which rewrites execution.json, so updated_at was already "now": no
downtime was handed back and any stop longer than the remaining budget timed
the rollout out the moment it resumed. Those timeouts look like real 24 h runs
to the sibling rule, so their requeued attempts were never claimed either
(11-06..08: a 33 h stop, 10 of 12 Qwen rollouts lost).

reconcile() now takes owner_lost_at (the dead owner's last write, read before
the caller's own saves) and adds the whole downtime to the deadline, uncapped:
the paused-credit cap is for pauses that may never resume, while a restarted
collector resumes at once. downtime_s records it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:33:08 +08:00
..