Board ssh ports are public and constantly hold 15-35 unauthenticated
scanner connections; whenever that exceeds sshd MaxStartups, our handshake
is closed with kex_exchange_identification. Only upload steps retried, so
a single drop failed the bench call itself (exit 255) or lost the cleanup
receipt fetch, which then quarantined the board.
run() and guarded_run() now replay a call up to twice when ssh reports
kex_exchange_identification: the connection closed before auth, so the
board never saw the command and replay cannot duplicate work. Any other
failure is returned unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
board_guard writes .board-inference.quarantine.json before spawning the
inference child. If the guard is killed (ssh drop, SIGKILL) the file stays
and every later 07-board/08-eval on that board exits 78 until a human
deletes it; rdk-sandbox-40009 was blocked this way for 1.5 days.
The guard already holds the board lock every admitted child inherits. It
now also checks /proc for any process carrying the old operation token;
with none left, the record is stale: it is removed and logged as
stale_quarantine_cleared, then the normal occupancy check still applies.
Survivors keep the board quarantined. Unverified-cleanup receipts now
store the token so the same rule covers them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When board ssh drops during cleanup, no receipt is written and the board
lease stayed quarantined forever; all 8 boards ended up locked and every
rollout timed out in the board queue (QUEUE_DEADLINE_EXCEEDED).
reconcile_leases now probes the board when the holder is dead, the
execution is terminal and no local process remains. If the board runs
nothing but sshd/ps the slot is released and board-release-<slot>.json
records the evidence; busy or unreachable boards stay quarantined.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- quota: once an UNKNOWN attempt is a full token window past its local
deadline, abandon it via the operator path (evidence kept, ended_at NULL,
generation fenced); the group then gets one probe.
- quota: generations paused at admission by that group pause may resume
when none of their own attempts is unended.
- pipeline: isolate an identifiable unreconciled rollout and keep
collecting; unidentifiable reports still stop. Drive broker reconcile
every loop so a fully paused group can recover.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reuse execution, stage and review evidence to recover database registration and
SFT publication without replaying completed work. Keep finished futures during
SQLite contention, and hand durable results to startup recovery when draining.
Release proven local leases independently of unresolved board isolation.
Keep UNKNOWN review retries explicit and archived, publish completed receipts
after transient export failures, and prevent a malformed sample from aborting
the entire dataset. Scope routine reconciliation to newly finished attempts.
Add failure-injection coverage and the production acceptance plan.
Record the deployed continuation, process reaping, recovery, and cleanup fixes with the current board registry. Add focused regression coverage and document the shared production branch and existing startup configuration.
Cancellation can end a compiler parent before its nested reaper. The nested
receipt then reports owner loss even when both reapers reached waitpid(ECHILD).
Recognize the verified enclosing receipt only when execution, epoch, immediate
parent, and full supervisor birth identity match and the nested children were
fully reaped. Preserve the original receipts and fail closed without that proof.
Add eight regression tests, including identity mismatch, incomplete cleanup,
missing parent proof, and live-process preservation. The positive regression
fails on the previous code and passes on the patch; recorded incident receipts
also distinguish the two implementations without changing the cancelled outcome.
Pi requests could hit Node fetch's independent five-minute header deadline,
and queued requests could outlive their local client or terminal rollout.
Set header/body deadlines only for the loopback broker origin, cancel scopes
when clients disconnect or executions terminate, and retain uncertain upstream
reservations instead of retrying after cancellation. Resumable pauses remain open.
Validation: real Pi SDK accepted a response delayed 310 seconds (314.139 s test);
quota, Pi session, acceptance, cancellation, and lifecycle regression coverage.