terminal_exec's body moves to agent/extensions/terminal-core.js, shared by
Pi's terminal.ts (unchanged schema/behaviour) and OpenCode's bench.ts.
OpenCode rollouts now get a managed Execution and a broker-scoped token
(the upstream key no longer enters its environment); terminal requires both.
No checkpoint/resume for OpenCode yet: the resume scan skips it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 10-21 03:16 Mac mini reboot left 7 GLM rollouts (05: densenet201,
efficientnet.b2, rtmo.s; 06: rtmpose.x, rtmo.m, hypersim.base, yolo11n.seg)
quarantined OPERATION_STATE_UNKNOWN while still counted RUNNING: each had a
05-compile in flight, its reaper died with the VM before writing a receipt, and
_validate_recovered_receipt demanded one ('scope cleanup unconfirmed') although
scope_cleaned already accepts an earlier boot. Without a receipt an earlier-boot
scope now proves cleanup for a failed result and for the inline 04 whose meta is
hashed; any other success still needs its receipt.
speed_ref: stages/ref of an interrupted run kept bundle.tar, which salty_dev
creates exclusively, so the 3 in-flight references failed with FileExistsError
every 2 minutes and took all 3 workers. The leftover dir is renamed aside.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 10-21 03:16 Mac mini reboot left zero-byte files that had been installed
(tmp write + rename, no fsync) just before: 04-calibrate/a00/meta.json of 05
cls--swin.b k2, 07-board meta/perf of 06 pose--rtmo.l k2, salty/result.json of
several dev commands. _memory json-parsed every attempt meta of the model, so
every later 04/05 of swin.b failed in 0.03 s as REMOTE_UNAVAILABLE.
_install/_fetch/dev install now fsync the data before the rename; the dev sync
manifest uses execution.atomic_json; _memory skips an unreadable meta.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dev container OOM (no reason given, the agent cannot raise the limit) was retried
at the same size and failed again every time (10-20: efficientnet.b6 3x at 16Gi,
unidepth.v2.vit.l 2x at 32Gi). remote.json now records memory and job_state; a failed
dev job of the rollout doubles the next one's memory (cap 128Gi) and stderr says so.
.bench-session.json.tmp (pi_session atomic write) was listed then renamed away before
tar read it: FileNotFoundError, dev command died as STAGE_EXECUTOR_EXIT (3x since 10-16).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker returned only artifacts that changed during the stage. 10-20
hrnet.w48 / ocrnet.hrnet.w18: the first 06-hbm container copied the HBM
into the (persistent) remote artifacts, then hit its memory limit; the
retry found it unchanged and never sent it, so the controller verified
'current HBM missing' and both ended UNVERIFIED after a good board eval.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jobs were kept until their dir delete succeeded; every storage delete
has failed server side since 10-19, so ~8000 finished jobs piled up and
from 10-20 17:00 /jobs grew past the CLI's response limit: salty ps and
describe by name failed and every Salty step ended REMOTE_UNAVAILABLE.
The job record is only an index (logs/describe are in stderr.log), so it
now goes first; the dir delete keeps its own retry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 07-board step killed at its phase deadline never reached the lease release,
and a board lease was only freed once the whole rollout ended, so two S600
boards stayed locked for 11 hours (10-20). A board slot is now reused when its
holder is dead, the stage is DONE with cleanup VERIFIED, and every board-guard
receipt says VERIFIED - the same proof the stage worker already requires.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Retries add a task record each (storage tasks lists them all). Once a delete
succeeded its record goes via storage rm-task (data untouched; the ID stays in
the receipt). Failed records stay for the Salty side's server investigation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With Salty 0.2.4 a delete is a background task; since 10-19 every one fails
server side, and a failure was recorded once and never retried, so garbage
would pile up for good. Now a submission the server refused (an HTTP error
such as 405) or a task that failed/was interrupted is submitted again by the
reaper after 30 min, doubling to 6 h; a task failing with not_found means an
earlier try removed it. A submission with no reply stays unknown and is never
resent blindly. The result path still makes one try. The reaper also covers
ended jobs ps no longer lists, logs only removals, and salty_offload gains
--reap-only so staging gets a background cleaner without archive/export.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On board kernels without /proc/<pid>/task/*/children, children() scans /proc and
reads each stat; a process that exits between the listing and the read answers
ESRCH (ProcessLookupError), which was not caught. 10-20 staging: the board guard
crashed in its own cleanup after a 20-minute self-test, wrote no result, and the
platform recorded BOARD_CLEANUP_UNCONFIRMED, quarantining a 5-hour rollout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stage worker runs one ticket at a time, but terminal_exec wait_handle gave up
after 1 s if the ticket had no operation yet ("Unknown protected stage handle").
10-20 staging: `bench stage run baseline` during a 40-minute bench dev compile,
and the agent followed the printed "# Pi wait" hint straight into that error.
In the last 3 days 85% of these errors (production 05/06 and staging) were real,
submitted tickets. A submitted ticket now waits under the execution's own stop
and deadline checks; a ticket the worker rejected before admission says why;
a handle that was never submitted is still unknown.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
10-19 staging: a 25-minute board self-test spent all of it queued for an S600
board shared with production, and the child's TERMINAL_DEADLINE_EXCEEDED was
overwritten by a bare STAGE_EXECUTOR_EXIT because it exited nonzero. The queue
is now covered by the request deadline (as for bench dev) and the command's
time starts at acquisition; a DONE status written by the child stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From 16:50 on 10-19 Salty's storage rm answered 405 for every object; the
cleanup retried it for 30 minutes before each stage or bench dev result went
back. Garbage removal now tries once (dir first, so a failure leaves the job
listed) and salty_offload.reap_orphans tries again on its next pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pi's bash tool (shellPath), bench dev and the board terminal run commands with
pipe-note.sh: the exit code stays the last step's (no pipefail, so | head is
still 0), and a failed earlier step of the final pipeline gets one stderr line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recorded FAILED CONTEXT_TOO_LONG like a provider overflow and answered with
the overflow body, so Pi compacts and continues instead of ending the rollout
on an unrecognised 413. Content review keeps its REQUEST_EXCEEDS_INTERNAL_TPM.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A checkout whose state is elsewhere (BENCH_STATE_DIR, staging) was read as
<checkout>/state, so content review and export found no database and called
a PASS rollout unfinished. One helper, paths.state_dir, used by every lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex review of the staging collector (10-19): a model built in bench dev was missing for 04-calibrate,
because only bench dev requests were rewritten to the container's bench/runs (reverts 56230c0). The
container now links the controller checkout's own runs/ to the same personal storage, so every
absolute path resolves alike; a no-op for production's bench/. And a paused rollout that waited ~2 h
left no trace of why: the collector logs each change of its wait reason.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Staging (10-19 14:06): a rollout whose lifetime wait total had passed wait_budget_s paused the first
time it was not admitted, resumed uncounted, and paused again every minute, starving the other paused
rollouts. 7 executions in production's s600 group are already over the total. The per-call bound and
the HTTP attempt budget are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Staging handover (10-19 11:40): the 2-minute hold on the dead collector's requests expired into an
unknown-outcome group pause that bounced every rollout already resumed. Its process exit closed the
connections, so the reply was cut off, not in doubt; and a platform restart no longer spends the
rollout's resume budget.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Staging kill -9 with an immediate restart (10-19 11:17): a turn waiting behind the group's concurrency
read as not resumable and lost its slot, and a calibrate step whose executor was still exiting got
quarantined as SIDE_EFFECT_UNKNOWN.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found on the staging collector (10-19): after kill -9 two rollouts whose reply was cut off were read as
not resumable for request_total_s while new rollouts took their slots, and a rollout handed over with a
bash command running was quarantined because the next collector looked before the reaper finished.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Processes are disposable; rollout state is resumable. When the collector dies,
the next one reconciles each running rollout from durable state instead of
quarantining or losing it:
- execution: a process of an earlier boot on this host is DEAD; a top-level
reaper whose owner (the collector) died still proves its subtree empty
(scopes record the owner that started them); reconcile() pauses a rollout
with a proven-clean crash checkpoint as CONTROLLER_RESTARTED and marks its
running tool calls INTERRUPTED; resume() across a reboot rebases monotonic
clocks by wall time and credits the downtime as a pause
- stageworker.settle_interrupted: an admitted step whose executor is gone ends
as STAGE_EXECUTOR_EXIT (free compile) and its handle is told directly; a
board step without a clean board receipt still quarantines; nothing replays
- collection_runtime.crash_checkpoint: capture the real system prompt, freeze
the session at the last saved message (mid-work phases only); configure_pi
skips a generation ID the ledger already completed
- pi_session.js: checkpoint after every saved message (and compaction); on a
recovered resume drop an unfinished trailing reply and answer open tool
calls with an honest "interrupted by a platform restart" result
- scheduler: a paused rollout's local slot held by a dead holder is released
once every process is proven gone
- pipeline: SIGUSR1 = planned handover (stop claiming, let board steps
finish, shorten in-flight request holds, exit); SIGTERM still drains
- export: bench_recovery closes the cut segment; the dropped unfinished reply
is excluded (UNFINISHED_BEFORE_RESTART)
- tests: test_controller_restart (real Pi on Linux: kill -9 mid-command,
kill -9 mid-reply, SIGUSR1 handover, power loss + reboot), test_settle_
interrupted; three tests of the old quarantine-everything rule now assert
the new outcome while keeping their guarantees (no replay, unrelated PID
untouched, board lease kept)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hb_compile's working_dir lives on the VM's own disk (work_root) and only
05-compile/06-hbm read it. Nothing removed it: on 10-18 742 finished rollouts
held 36 GB there, the Mac mini disk (which backs the VM disk) neared full,
and the VM's disk went read-only at 18:46.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
freeze_checkpoint read the session through config_bytes, which caps files at
1 MiB for tickets/configs. Long rollouts pass that, so finalization failed:
10-15..10-17, 17 rollouts, 4 recorded INFRA_ERROR instead of their real
outcome. The session read keeps the symlink/regular-file checks, no cap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- broker: a context-window 400 fails its generation (fences nothing) and goes
back to Pi as context_length_exceeded, so Pi compacts and resends
- runner: new generation ID after compaction, wait for Pi's resumed run,
record kept_messages; drop agent/context-mask.js and context_mask_chars
- models.json: contextWindow large so Pi compacts only on a real overflow,
never on an estimate right before a pause (resume would not match)
- export: compactions[] gives what the model actually saw after each one
- prompt: keep facts and acceptance, drop how-to rules and coaching
(SYSTEM.md 89->64 lines, e2e.md 27->20); stage status == stage wait
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>