10 Commits

Author SHA1 Message Date
bench
dcf976dc95 recovery: hand a stopped collector's downtime back to its rollouts
A rollout whose collector died (handover, crash) is paused on the next start
(CONTROLLER_RESTARTED) with paused_at taken from execution.json's updated_at.
But reconcile_run first settles interrupted operations through the stage
Worker, which rewrites execution.json, so updated_at was already "now": no
downtime was handed back and any stop longer than the remaining budget timed
the rollout out the moment it resumed. Those timeouts look like real 24 h runs
to the sibling rule, so their requeued attempts were never claimed either
(11-06..08: a 33 h stop, 10 of 12 Qwen rollouts lost).

reconcile() now takes owner_lost_at (the dead owner's last write, read before
the caller's own saves) and adds the whole downtime to the deadline, uncapped:
the paused-credit cap is for pauses that may never resume, while a restarted
collector resumes at once. downtime_s records it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:33:08 +08:00
bench
4709c7e44c boards: borrow from the RDK Sandbox platform per use, give back after
The platform now hands out boards per request and resets a board on release
(11-07 measured: free again after 42 s, new password, new host key, our key
gone). Boards are no longer ours between uses, so:

- harness/board_lease.py: request by board id (looked up by SSH port), install
  our public key with that lease's password (SSH_ASKPASS, never on a command
  line), release. A board-<slot>.rdk.json marker records each borrow, so a
  crashed holder's board is found and released; boards a person holds on the
  same account are never touched. Token in .env.local (RDK_SANDBOX_TOKEN) stays
  in controller processes; without it boards are used as statically configured.
- scheduler: borrow after taking the slot lock, give back right after use; an
  unavailable board rests 30 s per waiter. The reset replaces the board
  quarantine: receipts/idle-probe release and the orphan-marker loop in
  reconcile_leases become one sweep that gives back a dead holder's board.
- Host keys are no longer pinned (every borrow changes them): board_perf,
  the idle probe, Salty containers stop copying known_hosts.
- health_check: platform status replaces TCP and pinned-key ssh probes; warns
  when a march has no borrowable board or a borrow is older than 2 hours.

The agent's board access is unchanged (bench dev --board, stage run
board/eval); it never sees the token, passwords or which board it gets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 22:55:19 +08:00
bench
67303f7a06 execution: a paused rollout past its deadline after a reboot ends as DEADLINE_EXCEEDED
After a host reboot _reason() returned BOOT_CHANGED_RECONCILE_REQUIRED for every paused
execution before looking at the deadline, so one already past 24 h was offered for resume,
refused by resume() without saving why, and offered again. 10-25 after the Mac mini hang
seven rollouts looped ~43k times and held 7 lanes until they were cancelled by hand.

When the boot changed, judge a paused execution's deadline by wall time (the old boot's
monotonic clock means nothing). _resume_snapshot already settles EXECUTION_DEADLINE_EXCEEDED.

Checked on staging (r20261026-st): rollout made to look "previous boot, started 49 h ago,
paused 1 h"; the collector settled it as DEADLINE_EXCEEDED within a minute of start, no loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 10d5546f884053c6d1bf062e140183ff172f70dc)
2026-11-04 15:23:41 +08:00
bench
bf88f78f45 暂停等基础设施的时间不算 agent 的 24 小时:续跑时补回,暂停期间判超时也扣除,累计最多补一个预算
10-14 三条轨迹被 GLM 断线停了 7–16 小时,续上时只剩不到 1 小时,编译或评测做到一半被截断
(06 fcn.resnet101、depth.anything.v1.large、dpt.hybrid)。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:43:16 +08:00
bench
f83fa3948c 网络小故障不再拖垮整条轨迹:原地重发、没发出去的暂停不扣续跑次数、结果不明的请求可以续跑
GLM 请求失败率约 0.3%,但一次失败就让整条轨迹暂停;3 次续跑次数被网络故障用完后,
轨迹只能挂到 24 小时上限(10-14 有 4 条);回复中途断开的请求过了等待期会被自动放弃,
同时整条执行被取消(10-13/14 共 16 条)。

A. 连接失败、域名解析失败(RETRY_UNSENT,确定什么都没发出去)在 broker 里等一会儿原地重发。
   每次重试把等待时间加上一次连接时限计入排队预算 wait_spent,保持
   客户端超时 = 排队预算 + 次数×单次时限 的上界;预算用完才照旧暂停。
B. 续跑次数只在最近一次请求可能已经送达时才扣(看账本最近一次 attempt 是否 sent)。
   连接/解析失败、因为别的请求结果不明而在准入时被挡下的暂停,续跑不扣次数。
   执行记录和账本两处计数同步,调度程序不再按原始次数预先过滤。
C. reconcile 自动放弃时(截止时间再过一个 token 窗口)不再取消执行,而是把这一轮
   标成 REQUEST_OUTCOME_UNKNOWN、可以续跑:重发一轮只花一次请求的 token,并计入续跑次数。
   人工放弃(operator)仍然永久隔离。续跑资格检查不再被已放弃的 attempt 挡住。
都在调度程序进程里运行,下次重启生效。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 15:41:49 +08:00
bench
ec1a58edf8 续跑不再重新核对已结算的步骤;领题后不再重复查额度
续跑时 stageworker 会重新核对旧步骤回执。同一轨迹每次编译写同一个 .hbm,
旧回执核对失败后被当成状态不明,整条轨迹隔离;10-12/13 共 19 条编译过多次的轨迹,
续跑几秒内就判 INFRA_ERROR。现在已结算的操作直接沿用旧结果,判断条件与
reconcile_completed 相同。
collector 领题前已经查过额度,run_rollout 领到后又查一次(只等 0.5 秒);
机器忙时读超时,题被判 UNVERIFIED,一步都没跑(59 条)。领到后不再重复查。
这两处代码都在调度程序进程里运行,重启后才生效。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 11:15:05 +08:00
e1e73816af fix: recover stage registration and reclaim completed local leases 2026-09-29 12:55:35 +08:00
Codex
b81bc80875 fix: reuse managed waits and drain accepted stage work 2026-09-23 21:57:25 +08:00
Codex
1c21193221 fix: parse proc bytes and honor quota holds before claim 2026-09-23 01:28:21 +08:00
Codex
068c56043f Merge managed natural-agent runtime into retained benchmark mainline 2026-09-22 15:35:28 +08:00