285 Commits

Author SHA1 Message Date
bench
1dcf331c90 Gates: compute must run on the BPU; board latency budget and the original model on the board CPU as speed floors
- 06-hbm records the convolutions/matmuls hb_compile left on the CPU (its node table's ON column); G1 fails on any
  when the batch's gates say require_bpu_compute (new in gates.yaml; frozen batches without it are not judged)
- batch creation freezes agent/speed_ref/<job>.json into G4: latency_budget_ms = 1.5 x board_latency_ms (median
  board latency of plain int8 builds) and cpu_fp32_latency_ms; G4 also fails when the model is not faster than
  the original FP32 model on the board CPU, and records the speedup
- scripts/board_refs.py computes both; the CPU number is onnxruntime on the board itself under the board lease
- e2e prompt states the three rules; agent/VERSION 25

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 00:41:02 +08:00
bench
f6e7dfe56a docs(memory): pilot contract checksum incident and 10-24 tally
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 00:16:26 +08:00
bench
ff3c8926df Revert the pose evaluator edit from 8a1ee24: running batches froze its checksum
harness/eval/coco_extra.py is frozen per batch by task_contract (only listed equivalent rewrites may replace it).
The 8a1ee24 edit made every pose stage of the pilot batches fail the contract check (Qwen rtmpose.s never got a
formal stage). The keypoint shift gate it fed was removed in e0c224f, so the shift reaching 'unverified' no longer
decides anything; back to the approved version 505d37.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 00:15:37 +08:00
bench
a0e86e71ea health_check: alert on board_offline stage failures and on unreachable board ports
10-23 08:00-11:00 all 8 boards behind rdk-sandbox refused connections; board/eval steps failed as
board_offline for 3 hours and nothing in the check covered it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 11:19:15 +08:00
bench
23b55cb86e docs(memory): production pose shift rule removed; rejudge limits with offloaded HBM
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 08:39:53 +08:00
bench
f13c41da82 docs(memory): pose shift gate removed, pilot tally 10-23 08:00
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 07:56:43 +08:00
bench
e0c224f881 Pose: keypoint shift is recorded, no longer a pass/fail gate
The 2 px limit was a hand-set number from the first baseline (9-15): mean distance of all 17 joints in raw image
pixels, invisible joints included, not scale-normalized. On 10-22 SimpleBaseline r101 failed it with OKS-AP 68.34
vs FP32 68.22. OKS-AP already scores the task against ground truth with person-scale normalization, and FP32/quant
agreement is G2's job. New runs drop it from G3; the shift is still measured and recorded. Pose contracts no
longer state it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 07:55:51 +08:00
bench
8a1ee24338 Pose shift check no longer requires official_model; pose contracts state the 2 px keypoint shift limit
RTMPose evidence never carried official_model, so the top-down shift check reported every rtmpose run
UNVERIFIED (06 x2, pilot x2). run_quant already rejects a baseline of another model, so the key added nothing.
The contracts now say G3 also bounds the mean keypoint shift (2 px), which pilot agents could not know.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 01:39:15 +08:00
bench
bdfd600a54 docs(memory): opencode path guard, shared deps, 3/3 PASS rerun
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 22:19:15 +08:00
bench
89c06d4cba docs(memory): GLM vs Qwen pilot with platform code hidden
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 22:03:20 +08:00
bench
a8fdb288ec io-contracts for every adapter family: pose (HRNet/SimCC/MSPN/RTMO), depth, flow, features, DETR/Mask2Former/SOLO, EfficientViT-seg, SSD300
139 adapter models, 16 contract files. Each output is defined as an official module's own output (verifiable
against the official source and the bench stage run export signature), plus what the platform decodes on the host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 22:02:02 +08:00
bench
8fb9349907 Agent no longer gets the platform's model code; each task states its I/O contract
The agent could read and copy harness/adapters (model loading, preprocessing, export, decoding), and every
bench dev container carried the whole harness, registry, scripts and export trees, evaluator and gates
included. That did the model-specific half of the deployment for it.

- guard and private-home jail no longer expose harness/adapters; bench dev packs only container_site
- the evaluator still decodes outputs with the platform's own code, so agent/io-contracts states what the
  graph's inputs and outputs mean (9 families, 71 models so far); the prompt appends the task's contract
- SYSTEM.md: says where official sources, weights and environments are; the compile.yaml template is
  replaced by the facts that differ from the docs
- e2e prompt: whole-class int16/float fallback and >1.5x board latency do not pass
- agent/VERSION 24

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 21:51:17 +08:00
bench
061f3ed168 opencode: hard-link the pinned node_modules into the config dir
A symlinked node_modules made OpenCode hang inside the jail (read-only
target). A hard-linked copy is an ordinary installed tree for its reify and
takes no extra disk; agent/ and runs/ share one filesystem on the VM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 11:24:49 +08:00
bench
abc351cdc5 opencode: seed the config dir with OpenCode's own package files
A hand-written package.json without its lock made OpenCode reify again over the
symlink. With its package.json + package-lock.json and node_modules linked to
the pinned copy, startup to init is 0.1 s and the home 672 KB (VM probe).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 11:08:17 +08:00
bench
d166593ec6 opencode: Pi's path guard and the pinned plugin deps instead of a per-rollout install
The bench plugin applies guard-core like Pi's private-home path-guard: file
tools against the allowlist, bash only for root scans and direct shared-
resource commands; external_directory is left to that guard (job.json was
unreadable). Each rollout's config dir now links the pinned
agent/opencode/node_modules and models.json is not fetched: startup no longer
installs ~160 MB over the network.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 10:54:18 +08:00
bench
91ed6add7c docs(memory): opencode rerun results on a separate staging run
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 08:33:48 +08:00
bench
161d0a27e0 docs(memory): outcome of the 7 resumed 04 rollouts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 02:27:56 +08:00
bench
69954d7c0d docs(memory): breakdown of export rollouts missing a verdict
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 21:52:09 +08:00
bench
b98113c8af docs(memory): Salty CLI 0.2.6 fixes job list parse failure
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 20:19:04 +08:00
bench
2d164cd2a4 docs(memory): opencode on staging, fixes and pitfalls
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 19:41:24 +08:00
bench
f61a083d5a opencode: export the session to a file; through a pipe it is cut at 64 KiB
Staging mobilenet.v2: 380 KB export came back as 65536 or 131072 bytes and
the rollout had no trajectory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 19:40:51 +08:00
bench
a595d49c05 opencode: budget counts fresh input and output, not cached context, like Pi
A staging rollout (mobilenet.v2) hit 2M tokens at turn 63: 1.95M were cache reads
of the same context. Pi on the same task counted 120k for 96 turns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 19:40:51 +08:00
bench
064e1a025f docs(memory): 04 paused rollouts resumed or cancelled after Qwen redeploy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 14:26:18 +08:00
bench
216d5d0a05 docs(memory): reboot fixes shipped and 7 quarantined rollouts cancelled
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 14:13:58 +08:00
bench
5fb89d66b1 opencode: terminal_exec and the quota broker, like Pi
terminal_exec's body moves to agent/extensions/terminal-core.js, shared by
Pi's terminal.ts (unchanged schema/behaviour) and OpenCode's bench.ts.
OpenCode rollouts now get a managed Execution and a broker-scoped token
(the upstream key no longer enters its environment); terminal requires both.
No checkpoint/resume for OpenCode yet: the resume scan skips it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 12:46:53 +08:00
bench
44f3142cfa Reboot recovery: a reboot-killed scope proves a failed step; speed_ref sets aside an interrupted run
The 10-21 03:16 Mac mini reboot left 7 GLM rollouts (05: densenet201,
efficientnet.b2, rtmo.s; 06: rtmpose.x, rtmo.m, hypersim.base, yolo11n.seg)
quarantined OPERATION_STATE_UNKNOWN while still counted RUNNING: each had a
05-compile in flight, its reaper died with the VM before writing a receipt, and
_validate_recovered_receipt demanded one ('scope cleanup unconfirmed') although
scope_cleaned already accepts an earlier boot. Without a receipt an earlier-boot
scope now proves cleanup for a failed result and for the inline 04 whose meta is
hashed; any other success still needs its receipt.

speed_ref: stages/ref of an interrupted run kept bundle.tar, which salty_dev
creates exclusively, so the 3 in-flight references failed with FileExistsError
every 2 minutes and took all 3 workers. The leftover dir is renamed aside.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 07:38:06 +08:00
bench
e1753ad70d Salty results: fsync before rename; an empty attempt meta no longer breaks _memory
The 10-21 03:16 Mac mini reboot left zero-byte files that had been installed
(tmp write + rename, no fsync) just before: 04-calibrate/a00/meta.json of 05
cls--swin.b k2, 07-board meta/perf of 06 pose--rtmo.l k2, salty/result.json of
several dev commands. _memory json-parsed every attempt meta of the model, so
every later 04/05 of swin.b failed in 0.03 s as REMOTE_UNAVAILABLE.

_install/_fetch/dev install now fsync the data before the rename; the dev sync
manifest uses execution.atomic_json; _memory skips an unreadable meta.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 07:25:54 +08:00
bench
e3d14ad820 docs(memory): 10-21 Mac mini reboot and recovery
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 07:22:20 +08:00
bench
60c85d87e5 docs(memory): Qwen service context length, concurrency and stall pattern
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 01:50:31 +08:00
bench
93a09d2a83 docs(memory): dev OOM memory doubling shipped to production
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 21:43:10 +08:00
bench
50f163e993 bench dev: double memory after a failed dev job; never upload the pi checkpoint temp file
A dev container OOM (no reason given, the agent cannot raise the limit) was retried
at the same size and failed again every time (10-20: efficientnet.b6 3x at 16Gi,
unidepth.v2.vit.l 2x at 32Gi). remote.json now records memory and job_state; a failed
dev job of the rollout doubles the next one's memory (cap 128Gi) and stderr says so.

.bench-session.json.tmp (pi_session atomic write) was listed then renamed away before
tar read it: FileNotFoundError, dev command died as STAGE_EXECUTOR_EXIT (3x since 10-16).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 21:02:23 +08:00
bench
16b99ee1ed docs(memory): 06-hbm artifact return fix and two rollouts re-verified
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:47:19 +08:00
bench
19814889aa Salty worker: 06-hbm always returns its HBM/BC
The worker returned only artifacts that changed during the stage. 10-20
hrnet.w48 / ocrnet.hrnet.w18: the first 06-hbm container copied the HBM
into the (persistent) remote artifacts, then hit its memory limit; the
retry found it unchanged and never sent it, so the controller verified
'current HBM missing' and both ended UNVERIFIED after a good board eval.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:46:05 +08:00
bench
e50bf15278 docs(memory): offload and staging reaper restarted on job-first cleanup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:26:59 +08:00
bench
8064feb2fc docs(memory): Salty job-list outage and v23 SFT chain review
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:52:10 +08:00
bench
89c4801f53 Salty cleanup: remove the job before its storage dir
Jobs were kept until their dir delete succeeded; every storage delete
has failed server side since 10-19, so ~8000 finished jobs piled up and
from 10-20 17:00 /jobs grew past the CLI's response limit: salty ps and
describe by name failed and every Salty step ended REMOTE_UNAVAILABLE.
The job record is only an index (logs/describe are in stderr.log), so it
now goes first; the dir delete keeps its own retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:47:14 +08:00
bench
eab2dbcc37 docs(memory): board lease fix hot-swapped to production
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 16:28:08 +08:00
bench
aad8d5b597 docs(memory): production switched to the new code on 10-20
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 15:23:13 +08:00
bench
f75930e0f5 Scheduler: free a board whose stage died after its cleanup was proven
A 07-board step killed at its phase deadline never reached the lease release,
and a board lease was only freed once the whole rollout ended, so two S600
boards stayed locked for 11 hours (10-20). A board slot is now reused when its
holder is dead, the stage is DONE with cleanup VERIFIED, and every board-guard
receipt says VERIFIED - the same proof the stage worker already requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 15:21:32 +08:00
bench
3fa2e236f6 docs(memory): branches renamed to salty-prod / salty-test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 15:14:04 +08:00
bench
a6e2ce0ccb docs(memory): production hotfix of the one-shot Salty cleanup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 14:47:30 +08:00
bench
6badb5d44a docs(memory): Salty 0.2.4 on staging, background delete retries
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 14:30:32 +08:00
bench
36432d1c4e Salty cleanup: drop a succeeded delete task's record
Retries add a task record each (storage tasks lists them all). Once a delete
succeeded its record goes via storage rm-task (data untouched; the ID stays in
the receipt). Failed records stay for the Salty side's server investigation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 14:08:53 +08:00
bench
77031cd05b Salty cleanup: refused or failed deletes are retried from the background
With Salty 0.2.4 a delete is a background task; since 10-19 every one fails
server side, and a failure was recorded once and never retried, so garbage
would pile up for good. Now a submission the server refused (an HTTP error
such as 405) or a task that failed/was interrupted is submitted again by the
reaper after 30 min, doubling to 6 h; a task failing with not_found means an
earlier try removed it. A submission with no reply stays unknown and is never
resent blindly. The result path still makes one try. The reaper also covers
ended jobs ps no longer lists, logs only removals, and salty_offload gains
--reap-only so staging gets a background cleaner without archive/export.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 14:02:07 +08:00
bench
5998ab4c81 fix(staging): support Salty CLI 0.2.4 lifecycle 2026-10-20 13:52:49 +08:00
bench
2c5a08015f docs(memory): overnight staging supervision results
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 05:25:04 +08:00
bench
b4c4b83a88 Process reaper: a process exiting mid-scan is not a cleanup failure
On board kernels without /proc/<pid>/task/*/children, children() scans /proc and
reads each stat; a process that exits between the listing and the read answers
ESRCH (ProcessLookupError), which was not caught. 10-20 staging: the board guard
crashed in its own cleanup after a 20-minute self-test, wrote no result, and the
platform recorded BOARD_CLEANUP_UNCONFIRMED, quarantining a 5-hour rollout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 03:35:59 +08:00
bench
4b5b118909 terminal_exec: wait for a ticket queued behind the rollout's own worker
The stage worker runs one ticket at a time, but terminal_exec wait_handle gave up
after 1 s if the ticket had no operation yet ("Unknown protected stage handle").
10-20 staging: `bench stage run baseline` during a 40-minute bench dev compile,
and the agent followed the printed "# Pi wait" hint straight into that error.
In the last 3 days 85% of these errors (production 05/06 and staging) were real,
submitted tickets. A submitted ticket now waits under the execution's own stop
and deadline checks; a ticket the worker rejected before admission says why;
a handle that was never submitted is still unknown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 01:55:47 +08:00
bench
37df6933d7 Board commands: time starts with the board; a stage's own verdict is kept
10-19 staging: a 25-minute board self-test spent all of it queued for an S600
board shared with production, and the child's TERMINAL_DEADLINE_EXCEEDED was
overwritten by a bare STAGE_EXECUTOR_EXIT because it exited nonzero. The queue
is now covered by the request deadline (as for bench dev) and the command's
time starts at acquisition; a DONE status written by the child stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 23:07:10 +08:00
bench
98d85800ea docs(memory): Salty storage rm 405 incident and the one-shot cleanup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 18:01:35 +08:00