- 06-hbm records the convolutions/matmuls hb_compile left on the CPU (its node table's ON column); G1 fails on any
when the batch's gates say require_bpu_compute (new in gates.yaml; frozen batches without it are not judged)
- batch creation freezes agent/speed_ref/<job>.json into G4: latency_budget_ms = 1.5 x board_latency_ms (median
board latency of plain int8 builds) and cpu_fp32_latency_ms; G4 also fails when the model is not faster than
the original FP32 model on the board CPU, and records the speedup
- scripts/board_refs.py computes both; the CPU number is onnxruntime on the board itself under the board lease
- e2e prompt states the three rules; agent/VERSION 25
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 2 px limit was a hand-set number from the first baseline (9-15): mean distance of all 17 joints in raw image
pixels, invisible joints included, not scale-normalized. On 10-22 SimpleBaseline r101 failed it with OKS-AP 68.34
vs FP32 68.22. OKS-AP already scores the task against ground truth with person-scale normalization, and FP32/quant
agreement is G2's job. New runs drop it from G3; the shift is still measured and recorded. Pose contracts no
longer state it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RTMPose evidence never carried official_model, so the top-down shift check reported every rtmpose run
UNVERIFIED (06 x2, pilot x2). run_quant already rejects a baseline of another model, so the key added nothing.
The contracts now say G3 also bounds the mean keypoint shift (2 px), which pilot agents could not know.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
139 adapter models, 16 contract files. Each output is defined as an official module's own output (verifiable
against the official source and the bench stage run export signature), plus what the platform decodes on the host.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The agent could read and copy harness/adapters (model loading, preprocessing, export, decoding), and every
bench dev container carried the whole harness, registry, scripts and export trees, evaluator and gates
included. That did the model-specific half of the deployment for it.
- guard and private-home jail no longer expose harness/adapters; bench dev packs only container_site
- the evaluator still decodes outputs with the platform's own code, so agent/io-contracts states what the
graph's inputs and outputs mean (9 families, 71 models so far); the prompt appends the task's contract
- SYSTEM.md: says where official sources, weights and environments are; the compile.yaml template is
replaced by the facts that differ from the docs
- e2e prompt: whole-class int16/float fallback and >1.5x board latency do not pass
- agent/VERSION 24
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A hand-written package.json without its lock made OpenCode reify again over the
symlink. With its package.json + package-lock.json and node_modules linked to
the pinned copy, startup to init is 0.1 s and the home 672 KB (VM probe).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bench plugin applies guard-core like Pi's private-home path-guard: file
tools against the allowlist, bash only for root scans and direct shared-
resource commands; external_directory is left to that guard (job.json was
unreadable). Each rollout's config dir now links the pinned
agent/opencode/node_modules and models.json is not fetched: startup no longer
installs ~160 MB over the network.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A staging rollout (mobilenet.v2) hit 2M tokens at turn 63: 1.95M were cache reads
of the same context. Pi on the same task counted 120k for 96 turns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
terminal_exec's body moves to agent/extensions/terminal-core.js, shared by
Pi's terminal.ts (unchanged schema/behaviour) and OpenCode's bench.ts.
OpenCode rollouts now get a managed Execution and a broker-scoped token
(the upstream key no longer enters its environment); terminal requires both.
No checkpoint/resume for OpenCode yet: the resume scan skips it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stage worker runs one ticket at a time, but terminal_exec wait_handle gave up
after 1 s if the ticket had no operation yet ("Unknown protected stage handle").
10-20 staging: `bench stage run baseline` during a 40-minute bench dev compile,
and the agent followed the printed "# Pi wait" hint straight into that error.
In the last 3 days 85% of these errors (production 05/06 and staging) were real,
submitted tickets. A submitted ticket now waits under the execution's own stop
and deadline checks; a ticket the worker rejected before admission says why;
a handle that was never submitted is still unknown.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pi's bash tool (shellPath), bench dev and the board terminal run commands with
pipe-note.sh: the exit code stays the last step's (no pipefail, so | head is
still 0), and a failed earlier step of the final pipeline gets one stderr line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Processes are disposable; rollout state is resumable. When the collector dies,
the next one reconciles each running rollout from durable state instead of
quarantining or losing it:
- execution: a process of an earlier boot on this host is DEAD; a top-level
reaper whose owner (the collector) died still proves its subtree empty
(scopes record the owner that started them); reconcile() pauses a rollout
with a proven-clean crash checkpoint as CONTROLLER_RESTARTED and marks its
running tool calls INTERRUPTED; resume() across a reboot rebases monotonic
clocks by wall time and credits the downtime as a pause
- stageworker.settle_interrupted: an admitted step whose executor is gone ends
as STAGE_EXECUTOR_EXIT (free compile) and its handle is told directly; a
board step without a clean board receipt still quarantines; nothing replays
- collection_runtime.crash_checkpoint: capture the real system prompt, freeze
the session at the last saved message (mid-work phases only); configure_pi
skips a generation ID the ledger already completed
- pi_session.js: checkpoint after every saved message (and compaction); on a
recovered resume drop an unfinished trailing reply and answer open tool
calls with an honest "interrupted by a platform restart" result
- scheduler: a paused rollout's local slot held by a dead holder is released
once every process is proven gone
- pipeline: SIGUSR1 = planned handover (stop claiming, let board steps
finish, shorten in-flight request holds, exit); SIGTERM still drains
- export: bench_recovery closes the cut segment; the dropped unfinished reply
is excluded (UNFINISHED_BEFORE_RESTART)
- tests: test_controller_restart (real Pi on Linux: kill -9 mid-command,
kill -9 mid-reply, SIGUSR1 handover, power loss + reboot), test_settle_
interrupted; three tests of the old quarantine-everything rule now assert
the new outcome while keeping their guarantees (no replay, unrelated PID
untouched, board lease kept)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- broker: a context-window 400 fails its generation (fences nothing) and goes
back to Pi as context_length_exceeded, so Pi compacts and resends
- runner: new generation ID after compaction, wait for Pi's resumed run,
record kept_messages; drop agent/context-mask.js and context_mask_chars
- models.json: contextWindow large so Pi compacts only on a real overflow,
never on an estimate right before a pause (resume would not match)
- export: compactions[] gives what the model actually saw after each one
- prompt: keep facts and acceptance, drop how-to rules and coaching
(SYSTEM.md 89->64 lines, e2e.md 27->20); stage status == stage wait
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bash has no default timeout, so any command running past the 900 s system
cap ended the whole trajectory (81 of 590 finished rollouts). Give bash a
native timeout one minute before the cap; only that command fails and the
agent continues. The persisted tool call keeps the model's own arguments.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Historical r20260929-04/05/06 data: a cap of 3 would have lost 19 of 142
PASS trajectories (13%); a cap of 5 loses 2 (1.4%). Prompt states the same
number; agent/VERSION 13 -> 14.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compile attempts are now the primary per-trajectory budget: they map to
the real compute cost, unlike turns or tokens, and make the stop reason
obvious. stagerun refuses a 4th 05-compile with COMPILE_BUDGET_EXHAUSTED
before inline calibration or slot acquisition.
- Counted from the stages ledger (05-compile rows that got a slot), so
resume/recovery cannot reset it; a compile that timed out in the queue
is free; recompiles after an accuracy miss count.
- Board/eval/review of the last artifact still run after the cap.
- Turn/token/wall budgets, per-phase deadlines, cleanup and API quota
protection are unchanged as backstops.
- e2e prompt drops "no retry limit"; SYSTEM.md names compiles as the
primary budget; agent/VERSION 12 -> 13.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pi requests could hit Node fetch's independent five-minute header deadline,
and queued requests could outlive their local client or terminal rollout.
Set header/body deadlines only for the loopback broker origin, cancel scopes
when clients disconnect or executions terminate, and retain uncertain upstream
reservations instead of retrying after cancellation. Resumable pauses remain open.
Validation: real Pi SDK accepted a response delayed 310 seconds (314.139 s test);
quota, Pi session, acceptance, cancellation, and lifecycle regression coverage.