A batch paused at the free-space floor recorded its in-flight attempts as disk_limit,
which paid their JOBS lines, so rerunning the same command would have skipped them.
Host disk stops are now their own exclusion, disk_low, and count like interrupted
attempts; attempt_size stays disk_limit and still pays its line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rollout.salty_run keeps the gateway, grading and evidence on 36 and moves only the
agent step into a Salty job. The gateway listens on 127.0.0.1 with a per-attempt
bearer token; the job reaches it over port 22 with a restricted one-time key that is
removed after each attempt. rollout.batch gains --backend salty and repeatable
--credential-file, and writes pass/fail SFT exports through rollout.sft at the end.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
36's /tmp has 28 GiB free, below the 30 GiB floor, so the runner correctly
paused before launching and four tests failed there. Pin the floor to 0 in
those tests and cover the low-disk pause on its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 6 lost 26 of 103 attempts to Qwen requests that sent their preamble and
then stalled; Pi's own Node fetch gave up after 300 s and reported "terminated".
- model_proxy: hold the SSE preamble until the first generation event, retry a
stall (180 s) within the existing retry budget, and send SSE keepalive
comments so Pi never times out first; record first_event_seconds.
- report: name one exclusion reason per incomplete attempt and compute the
JOBS lines still owed, with a per-line resample budget.
- batch: run a JOBS file to completion, resume from receipts, resample
infrastructure failures, pause after three environment failures in a row,
forward SIGTERM to running controllers, and refuse a second runner.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>