bench
dcf976dc95
recovery: hand a stopped collector's downtime back to its rollouts
...
A rollout whose collector died (handover, crash) is paused on the next start
(CONTROLLER_RESTARTED) with paused_at taken from execution.json's updated_at.
But reconcile_run first settles interrupted operations through the stage
Worker, which rewrites execution.json, so updated_at was already "now": no
downtime was handed back and any stop longer than the remaining budget timed
the rollout out the moment it resumed. Those timeouts look like real 24 h runs
to the sibling rule, so their requeued attempts were never claimed either
(11-06..08: a 33 h stop, 10 of 12 Qwen rollouts lost).
reconcile() now takes owner_lost_at (the dead owner's last write, read before
the caller's own saves) and adds the whole downtime to the deadline, uncapped:
the paused-credit cap is for pauses that may never resume, while a restarted
collector resumes at once. downtime_s records it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:33:08 +08:00
..
2026-09-22 15:35:28 +08:00
2026-09-19 17:52:03 +08:00
2026-10-12 17:05:59 +08:00
2026-09-15 08:10:26 +08:00
2026-10-24 00:55:46 +08:00
2026-10-12 16:50:51 +08:00
2026-09-15 08:10:26 +08:00
2026-09-15 08:10:26 +08:00
2026-09-22 14:56:41 +08:00
2026-09-15 12:35:36 +08:00
2026-11-07 22:56:56 +08:00
2026-09-15 08:10:26 +08:00
2026-09-15 12:46:30 +08:00
2026-10-22 21:51:17 +08:00
2026-09-22 15:35:28 +08:00
2026-09-19 17:52:03 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-22 14:56:41 +08:00
2026-09-22 14:56:41 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-22 14:56:41 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-22 14:56:41 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-15 12:42:44 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-19 16:52:42 +08:00
2026-09-22 15:35:28 +08:00
2026-10-29 11:00:43 +08:00
2026-10-12 01:51:49 +08:00
2026-09-15 12:21:38 +08:00
2026-10-22 10:54:18 +08:00
2026-10-24 12:34:49 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-09-19 17:52:03 +08:00
2026-09-22 15:35:28 +08:00
2026-10-16 01:18:03 +08:00
2026-09-22 20:28:42 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-22 21:51:17 +08:00
2026-10-20 03:35:59 +08:00
2026-10-04 09:18:48 +08:00
2026-11-07 22:55:19 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-11-07 22:55:19 +08:00
2026-10-24 00:41:02 +08:00
2026-10-15 15:10:04 +08:00
2026-09-22 15:35:28 +08:00
2026-09-19 17:52:03 +08:00
2026-10-18 10:33:37 +08:00
2026-09-23 01:28:21 +08:00
2026-09-29 16:09:01 +08:00
2026-09-22 15:35:28 +08:00
2026-10-08 09:30:27 +08:00
2026-10-12 21:44:24 +08:00
2026-09-22 15:35:28 +08:00
2026-10-15 15:28:35 +08:00
2026-09-29 16:09:01 +08:00
2026-09-29 16:09:01 +08:00
2026-10-19 11:02:59 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-10-15 14:16:48 +08:00
2026-09-19 17:52:03 +08:00
2026-10-04 09:50:23 +08:00
2026-11-08 08:33:08 +08:00
2026-09-22 15:35:28 +08:00
2026-10-22 11:24:49 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-11-07 22:55:19 +08:00
2026-10-19 09:29:26 +08:00
2026-09-22 15:35:28 +08:00
2026-10-19 16:52:42 +08:00
2026-09-19 17:52:03 +08:00
2026-09-29 16:09:01 +08:00
2026-09-16 11:04:30 +08:00
2026-10-20 01:55:47 +08:00
2026-09-22 15:35:28 +08:00
2026-09-29 18:29:59 +08:00
2026-11-04 15:23:41 +08:00
2026-10-15 20:46:51 +08:00
2026-09-15 08:10:26 +08:00
2026-10-19 16:56:46 +08:00
2026-10-19 16:52:42 +08:00
2026-09-22 15:35:28 +08:00
2026-10-24 00:15:37 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-19 14:03:04 +08:00
2026-11-04 15:23:42 +08:00
2026-09-22 15:35:28 +08:00
2026-09-29 12:55:35 +08:00
2026-10-14 12:48:17 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-03 21:17:40 +08:00
2026-10-15 14:16:48 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-09 14:19:48 +08:00
2026-11-04 15:23:42 +08:00
2026-10-24 12:34:49 +08:00
2026-10-12 16:38:07 +08:00
2026-10-19 09:29:26 +08:00
2026-10-19 09:29:26 +08:00
2026-10-21 07:38:06 +08:00
2026-10-16 15:24:22 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-19 23:07:10 +08:00
2026-10-19 09:29:26 +08:00
2026-09-15 09:42:56 +08:00
2026-09-15 12:27:43 +08:00