Mac mini 主控只跑 agent、调度、账本、配额和锁;agent 的每次 bench stage run
(01–08)都在 Salty 起一个容器跑完整的 Stage.run(含查缓存),容器用软链接
复刻老编译机路径,冻结的任务路径和缓存原样可用,跑完只取回阶段目录(≤32MB 的
文件)和板卡回执。配置键 execution.salty_remote,未开启时行为不变。
已验证:本地往返测试;Mac mini → Salty 容器 → 结果取回的真实链路(06-hbm 空跑)。
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A queued rollout was never claimed once any sibling ended DEADLINE_EXCEEDED,
CANCELLED, INFRA_ERROR, LOST or BUDGET_EXHAUSTED. Almost all of those were
system stops (tool cap, context rejected by the local quota, board cleanup
or request outcome unknown, pauses that expired, operator restarts), so
1107 queued rollouts across r20260929-04/05/06 sat behind them.
Siblings still run one at a time, and two stops still hold the rest: an
exhausted turn/token budget, and a DEADLINE_EXCEEDED whose verdict says
the trajectory really worked until the 24 h execution deadline. Agent-
finished FAIL/UNVERIFIED keep not holding siblings, as before.
Tests that pinned the old hold now assert the repeat is claimable; the
lease-recovery waves no longer depend on sibling holds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bash has no default timeout, so any command running past the 900 s system
cap ended the whole trajectory (81 of 590 finished rollouts). Give bash a
native timeout one minute before the cap; only that command fails and the
agent continues. The persisted tool call keeps the model's own arguments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The probe after an unknown request was only allowed at the moment the
unknown attempt was resolved, and only if no other attempt was open. On
2026-10-07 08:26 the Qwen group's unknown attempt was auto-abandoned while
an innocent request was still streaming (ended 08:27), probe_allowed was
false, and nothing re-checked: the group stayed PAUSED for 15 hours.
abandon_unknown, confirm_end and reconcile() now share _probe_if_settled():
an unknown-outcome pause turns into one COOLING probe as soon as no attempt
in the group is open, whichever of them sees that first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covers the branch itself, not only the counter: at cap-1 the compile runs;
at the cap stagerun publishes COMPILE_BUDGET_EXHAUSTED without starting a
stage row or calling the stage. Verified to fail when the check is removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compile attempts are now the primary per-trajectory budget: they map to
the real compute cost, unlike turns or tokens, and make the stop reason
obvious. stagerun refuses a 4th 05-compile with COMPILE_BUDGET_EXHAUSTED
before inline calibration or slot acquisition.
- Counted from the stages ledger (05-compile rows that got a slot), so
resume/recovery cannot reset it; a compile that timed out in the queue
is free; recompiles after an accuracy miss count.
- Board/eval/review of the last artifact still run after the cap.
- Turn/token/wall budgets, per-phase deadlines, cleanup and API quota
protection are unchanged as backstops.
- e2e prompt drops "no retry limit"; SYSTEM.md names compiles as the
primary budget; agent/VERSION 12 -> 13.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three r20260929-06 rollouts sat in PAUSED_INFRA although their stage had
finished (status DONE/FAIL, no process left): the completion replay in
stageworker._validate_recovered_receipt raised ValueError, so the operation
stayed DISPATCHED and the execution was quarantined (one-way).
- Nested scopes: a deadline kill of the outer stage leaves the inner reaper
with owner_lost=True although both levels reaped via waitpid(ECHILD).
owned_processes() already accepted that when the enclosing verified receipt
names the exact supervisor; receipt validation demanded VERIFIED. Both now
share execution.scope_cleaned().
- Board stages killed by QUEUE_DEADLINE_EXCEEDED never held a board slot
(phase_running fires only under the board lock) and so have no board
receipt; that missing receipt no longer counts as unconfirmed cleanup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Board ssh ports are public and constantly hold 15-35 unauthenticated
scanner connections; whenever that exceeds sshd MaxStartups, our handshake
is closed with kex_exchange_identification. Only upload steps retried, so
a single drop failed the bench call itself (exit 255) or lost the cleanup
receipt fetch, which then quarantined the board.
run() and guarded_run() now replay a call up to twice when ssh reports
kex_exchange_identification: the connection closed before auth, so the
board never saw the command and replay cannot duplicate work. Any other
failure is returned unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
board_guard writes .board-inference.quarantine.json before spawning the
inference child. If the guard is killed (ssh drop, SIGKILL) the file stays
and every later 07-board/08-eval on that board exits 78 until a human
deletes it; rdk-sandbox-40009 was blocked this way for 1.5 days.
The guard already holds the board lock every admitted child inherits. It
now also checks /proc for any process carrying the old operation token;
with none left, the record is stale: it is removed and logged as
stale_quarantine_cleared, then the normal occupancy check still applies.
Survivors keep the board quarantined. Unverified-cleanup receipts now
store the token so the same rule covers them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When board ssh drops during cleanup, no receipt is written and the board
lease stayed quarantined forever; all 8 boards ended up locked and every
rollout timed out in the board queue (QUEUE_DEADLINE_EXCEEDED).
reconcile_leases now probes the board when the holder is dead, the
execution is terminal and no local process remains. If the board runs
nothing but sshd/ps the slot is released and board-release-<slot>.json
records the evidence; busy or unreachable boards stay quarantined.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- quota: once an UNKNOWN attempt is a full token window past its local
deadline, abandon it via the operator path (evidence kept, ended_at NULL,
generation fenced); the group then gets one probe.
- quota: generations paused at admission by that group pause may resume
when none of their own attempts is unended.
- pipeline: isolate an identifiable unreconciled rollout and keep
collecting; unidentifiable reports still stop. Drive broker reconcile
every loop so a fully paused group can recover.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reuse execution, stage and review evidence to recover database registration and
SFT publication without replaying completed work. Keep finished futures during
SQLite contention, and hand durable results to startup recovery when draining.
Release proven local leases independently of unresolved board isolation.
Keep UNKNOWN review retries explicit and archived, publish completed receipts
after transient export failures, and prevent a malformed sample from aborting
the entire dataset. Scope routine reconciliation to newly finished attempts.
Add failure-injection coverage and the production acceptance plan.