A rollout whose collector died (handover, crash) is paused on the next start
(CONTROLLER_RESTARTED) with paused_at taken from execution.json's updated_at.
But reconcile_run first settles interrupted operations through the stage
Worker, which rewrites execution.json, so updated_at was already "now": no
downtime was handed back and any stop longer than the remaining budget timed
the rollout out the moment it resumed. Those timeouts look like real 24 h runs
to the sibling rule, so their requeued attempts were never claimed either
(11-06..08: a 33 h stop, 10 of 12 Qwen rollouts lost).
reconcile() now takes owner_lost_at (the dead owner's last write, read before
the caller's own saves) and adds the whole downtime to the deadline, uncapped:
the paused-credit cap is for pauses that may never resume, while a restarted
collector resumes at once. downtime_s records it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The platform now hands out boards per request and resets a board on release
(11-07 measured: free again after 42 s, new password, new host key, our key
gone). Boards are no longer ours between uses, so:
- harness/board_lease.py: request by board id (looked up by SSH port), install
our public key with that lease's password (SSH_ASKPASS, never on a command
line), release. A board-<slot>.rdk.json marker records each borrow, so a
crashed holder's board is found and released; boards a person holds on the
same account are never touched. Token in .env.local (RDK_SANDBOX_TOKEN) stays
in controller processes; without it boards are used as statically configured.
- scheduler: borrow after taking the slot lock, give back right after use; an
unavailable board rests 30 s per waiter. The reset replaces the board
quarantine: receipts/idle-probe release and the orphan-marker loop in
reconcile_leases become one sweep that gives back a dead holder's board.
- Host keys are no longer pinned (every borrow changes them): board_perf,
the idle probe, Salty containers stop copying known_hosts.
- health_check: platform status replaces TCP and pinned-key ssh probes; warns
when a march has no borrowable board or a borrow is older than 2 hours.
The agent's board access is unchanged (bench dev --board, stage run
board/eval); it never sees the token, passwords or which board it gets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a host reboot _reason() returned BOOT_CHANGED_RECONCILE_REQUIRED for every paused
execution before looking at the deadline, so one already past 24 h was offered for resume,
refused by resume() without saving why, and offered again. 10-25 after the Mac mini hang
seven rollouts looped ~43k times and held 7 lanes until they were cancelled by hand.
When the boot changed, judge a paused execution's deadline by wall time (the old boot's
monotonic clock means nothing). _resume_snapshot already settles EXECUTION_DEADLINE_EXCEEDED.
Checked on staging (r20261026-st): rollout made to look "previous boot, started 49 h ago,
paused 1 h"; the collector settled it as DEADLINE_EXCEEDED within a minute of start, no loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>