bench
44f3142cfa
Reboot recovery: a reboot-killed scope proves a failed step; speed_ref sets aside an interrupted run
...
The 10-21 03:16 Mac mini reboot left 7 GLM rollouts (05: densenet201,
efficientnet.b2, rtmo.s; 06: rtmpose.x, rtmo.m, hypersim.base, yolo11n.seg)
quarantined OPERATION_STATE_UNKNOWN while still counted RUNNING: each had a
05-compile in flight, its reaper died with the VM before writing a receipt, and
_validate_recovered_receipt demanded one ('scope cleanup unconfirmed') although
scope_cleaned already accepts an earlier boot. Without a receipt an earlier-boot
scope now proves cleanup for a failed result and for the inline 04 whose meta is
hashed; any other success still needs its receipt.
speed_ref: stages/ref of an interrupted run kept bundle.tar, which salty_dev
creates exclusively, so the 3 in-flight references failed with FileExistsError
every 2 minutes and took all 3 workers. The leftover dir is renamed aside.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 07:38:06 +08:00
..
2026-09-22 15:35:28 +08:00
2026-09-19 17:52:03 +08:00
2026-10-12 17:05:59 +08:00
2026-09-15 08:10:26 +08:00
2026-10-12 16:50:51 +08:00
2026-09-15 08:10:26 +08:00
2026-09-15 08:10:26 +08:00
2026-09-22 14:56:41 +08:00
2026-09-15 12:35:36 +08:00
2026-10-12 16:38:07 +08:00
2026-09-15 08:10:26 +08:00
2026-09-15 12:46:30 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-09-19 17:52:03 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-22 14:56:41 +08:00
2026-09-22 14:56:41 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-22 14:56:41 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-22 14:56:41 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-09-15 12:42:44 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-19 16:52:42 +08:00
2026-09-22 15:35:28 +08:00
2026-10-20 14:02:07 +08:00
2026-10-12 01:51:49 +08:00
2026-09-15 12:21:38 +08:00
2026-09-22 15:35:28 +08:00
2026-10-21 07:38:06 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-09-19 17:52:03 +08:00
2026-09-22 15:35:28 +08:00
2026-10-16 01:18:03 +08:00
2026-09-22 20:28:42 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-20 03:35:59 +08:00
2026-10-04 09:18:48 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-15 13:43:16 +08:00
2026-10-15 15:10:04 +08:00
2026-09-22 15:35:28 +08:00
2026-09-19 17:52:03 +08:00
2026-10-18 10:33:37 +08:00
2026-09-23 01:28:21 +08:00
2026-09-29 16:09:01 +08:00
2026-09-22 15:35:28 +08:00
2026-10-08 09:30:27 +08:00
2026-10-12 21:44:24 +08:00
2026-09-22 15:35:28 +08:00
2026-10-15 15:28:35 +08:00
2026-09-29 16:09:01 +08:00
2026-09-29 16:09:01 +08:00
2026-10-19 11:02:59 +08:00
2026-09-19 17:52:03 +08:00
2026-09-19 17:52:03 +08:00
2026-10-15 14:16:48 +08:00
2026-09-19 17:52:03 +08:00
2026-10-04 09:50:23 +08:00
2026-10-15 13:43:16 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-20 15:21:32 +08:00
2026-10-19 09:29:26 +08:00
2026-09-22 15:35:28 +08:00
2026-10-19 16:52:42 +08:00
2026-09-19 17:52:03 +08:00
2026-09-29 16:09:01 +08:00
2026-09-16 11:04:30 +08:00
2026-10-20 01:55:47 +08:00
2026-09-22 15:35:28 +08:00
2026-09-29 18:29:59 +08:00
2026-10-15 11:43:53 +08:00
2026-10-15 20:46:51 +08:00
2026-09-15 08:10:26 +08:00
2026-10-19 16:56:46 +08:00
2026-10-19 16:52:42 +08:00
2026-09-22 15:35:28 +08:00
2026-09-23 21:57:25 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-19 14:03:04 +08:00
2026-10-19 16:52:42 +08:00
2026-09-22 15:35:28 +08:00
2026-09-29 12:55:35 +08:00
2026-10-14 12:48:17 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-03 21:17:40 +08:00
2026-10-15 14:16:48 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-09 14:19:48 +08:00
2026-10-20 19:47:14 +08:00
2026-10-21 07:25:54 +08:00
2026-10-12 16:38:07 +08:00
2026-10-19 09:29:26 +08:00
2026-10-19 09:29:26 +08:00
2026-10-21 07:38:06 +08:00
2026-10-16 15:24:22 +08:00
2026-09-22 15:35:28 +08:00
2026-09-22 15:35:28 +08:00
2026-10-19 23:07:10 +08:00
2026-10-19 09:29:26 +08:00
2026-09-15 09:42:56 +08:00
2026-09-15 12:27:43 +08:00