2 Commits

Author SHA1 Message Date
amos
a3365b5583 Keep batch tests independent of the host's free disk
36's /tmp has 28 GiB free, below the 30 GiB floor, so the runner correctly
paused before launching and four tests failed there. Pin the floor to 0 in
those tests and cover the low-disk pause on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 08:31:29 +08:00
amos
3adab31da7 Self-heal rollouts: retry stalled model streams and run batches from receipts
Batch 6 lost 26 of 103 attempts to Qwen requests that sent their preamble and
then stalled; Pi's own Node fetch gave up after 300 s and reported "terminated".

- model_proxy: hold the SSE preamble until the first generation event, retry a
  stall (180 s) within the existing retry budget, and send SSE keepalive
  comments so Pi never times out first; record first_event_seconds.
- report: name one exclusion reason per incomplete attempt and compute the
  JOBS lines still owed, with a per-line resample budget.
- batch: run a JOBS file to completion, resume from receipts, resample
  infrastructure failures, pause after three environment failures in a row,
  forward SIGTERM to running controllers, and refuse a second runner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 08:28:53 +08:00