144 Commits

Author SHA1 Message Date
bench
f61a083d5a opencode: export the session to a file; through a pipe it is cut at 64 KiB
Staging mobilenet.v2: 380 KB export came back as 65536 or 131072 bytes and
the rollout had no trajectory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 19:40:51 +08:00
bench
a595d49c05 opencode: budget counts fresh input and output, not cached context, like Pi
A staging rollout (mobilenet.v2) hit 2M tokens at turn 63: 1.95M were cache reads
of the same context. Pi on the same task counted 120k for 96 turns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 19:40:51 +08:00
bench
5fb89d66b1 opencode: terminal_exec and the quota broker, like Pi
terminal_exec's body moves to agent/extensions/terminal-core.js, shared by
Pi's terminal.ts (unchanged schema/behaviour) and OpenCode's bench.ts.
OpenCode rollouts now get a managed Execution and a broker-scoped token
(the upstream key no longer enters its environment); terminal requires both.
No checkpoint/resume for OpenCode yet: the resume scan skips it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 12:46:53 +08:00
bench
44f3142cfa Reboot recovery: a reboot-killed scope proves a failed step; speed_ref sets aside an interrupted run
The 10-21 03:16 Mac mini reboot left 7 GLM rollouts (05: densenet201,
efficientnet.b2, rtmo.s; 06: rtmpose.x, rtmo.m, hypersim.base, yolo11n.seg)
quarantined OPERATION_STATE_UNKNOWN while still counted RUNNING: each had a
05-compile in flight, its reaper died with the VM before writing a receipt, and
_validate_recovered_receipt demanded one ('scope cleanup unconfirmed') although
scope_cleaned already accepts an earlier boot. Without a receipt an earlier-boot
scope now proves cleanup for a failed result and for the inline 04 whose meta is
hashed; any other success still needs its receipt.

speed_ref: stages/ref of an interrupted run kept bundle.tar, which salty_dev
creates exclusively, so the 3 in-flight references failed with FileExistsError
every 2 minutes and took all 3 workers. The leftover dir is renamed aside.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 07:38:06 +08:00
bench
e1753ad70d Salty results: fsync before rename; an empty attempt meta no longer breaks _memory
The 10-21 03:16 Mac mini reboot left zero-byte files that had been installed
(tmp write + rename, no fsync) just before: 04-calibrate/a00/meta.json of 05
cls--swin.b k2, 07-board meta/perf of 06 pose--rtmo.l k2, salty/result.json of
several dev commands. _memory json-parsed every attempt meta of the model, so
every later 04/05 of swin.b failed in 0.03 s as REMOTE_UNAVAILABLE.

_install/_fetch/dev install now fsync the data before the rename; the dev sync
manifest uses execution.atomic_json; _memory skips an unreadable meta.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 07:25:54 +08:00
bench
50f163e993 bench dev: double memory after a failed dev job; never upload the pi checkpoint temp file
A dev container OOM (no reason given, the agent cannot raise the limit) was retried
at the same size and failed again every time (10-20: efficientnet.b6 3x at 16Gi,
unidepth.v2.vit.l 2x at 32Gi). remote.json now records memory and job_state; a failed
dev job of the rollout doubles the next one's memory (cap 128Gi) and stderr says so.

.bench-session.json.tmp (pi_session atomic write) was listed then renamed away before
tar read it: FileNotFoundError, dev command died as STAGE_EXECUTOR_EXIT (3x since 10-16).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 21:02:23 +08:00
bench
19814889aa Salty worker: 06-hbm always returns its HBM/BC
The worker returned only artifacts that changed during the stage. 10-20
hrnet.w48 / ocrnet.hrnet.w18: the first 06-hbm container copied the HBM
into the (persistent) remote artifacts, then hit its memory limit; the
retry found it unchanged and never sent it, so the controller verified
'current HBM missing' and both ended UNVERIFIED after a good board eval.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:46:05 +08:00
bench
89c4801f53 Salty cleanup: remove the job before its storage dir
Jobs were kept until their dir delete succeeded; every storage delete
has failed server side since 10-19, so ~8000 finished jobs piled up and
from 10-20 17:00 /jobs grew past the CLI's response limit: salty ps and
describe by name failed and every Salty step ended REMOTE_UNAVAILABLE.
The job record is only an index (logs/describe are in stderr.log), so it
now goes first; the dir delete keeps its own retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:47:14 +08:00
bench
f75930e0f5 Scheduler: free a board whose stage died after its cleanup was proven
A 07-board step killed at its phase deadline never reached the lease release,
and a board lease was only freed once the whole rollout ended, so two S600
boards stayed locked for 11 hours (10-20). A board slot is now reused when its
holder is dead, the stage is DONE with cleanup VERIFIED, and every board-guard
receipt says VERIFIED - the same proof the stage worker already requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 15:21:32 +08:00
bench
36432d1c4e Salty cleanup: drop a succeeded delete task's record
Retries add a task record each (storage tasks lists them all). Once a delete
succeeded its record goes via storage rm-task (data untouched; the ID stays in
the receipt). Failed records stay for the Salty side's server investigation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 14:08:53 +08:00
bench
77031cd05b Salty cleanup: refused or failed deletes are retried from the background
With Salty 0.2.4 a delete is a background task; since 10-19 every one fails
server side, and a failure was recorded once and never retried, so garbage
would pile up for good. Now a submission the server refused (an HTTP error
such as 405) or a task that failed/was interrupted is submitted again by the
reaper after 30 min, doubling to 6 h; a task failing with not_found means an
earlier try removed it. A submission with no reply stays unknown and is never
resent blindly. The result path still makes one try. The reaper also covers
ended jobs ps no longer lists, logs only removals, and salty_offload gains
--reap-only so staging gets a background cleaner without archive/export.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 14:02:07 +08:00
bench
5998ab4c81 fix(staging): support Salty CLI 0.2.4 lifecycle 2026-10-20 13:52:49 +08:00
bench
b4c4b83a88 Process reaper: a process exiting mid-scan is not a cleanup failure
On board kernels without /proc/<pid>/task/*/children, children() scans /proc and
reads each stat; a process that exits between the listing and the read answers
ESRCH (ProcessLookupError), which was not caught. 10-20 staging: the board guard
crashed in its own cleanup after a 20-minute self-test, wrote no result, and the
platform recorded BOARD_CLEANUP_UNCONFIRMED, quarantining a 5-hour rollout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 03:35:59 +08:00
bench
4b5b118909 terminal_exec: wait for a ticket queued behind the rollout's own worker
The stage worker runs one ticket at a time, but terminal_exec wait_handle gave up
after 1 s if the ticket had no operation yet ("Unknown protected stage handle").
10-20 staging: `bench stage run baseline` during a 40-minute bench dev compile,
and the agent followed the printed "# Pi wait" hint straight into that error.
In the last 3 days 85% of these errors (production 05/06 and staging) were real,
submitted tickets. A submitted ticket now waits under the execution's own stop
and deadline checks; a ticket the worker rejected before admission says why;
a handle that was never submitted is still unknown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 01:55:47 +08:00
bench
37df6933d7 Board commands: time starts with the board; a stage's own verdict is kept
10-19 staging: a 25-minute board self-test spent all of it queued for an S600
board shared with production, and the child's TERMINAL_DEADLINE_EXCEEDED was
overwritten by a bare STAGE_EXECUTOR_EXIT because it exited nonzero. The queue
is now covered by the request deadline (as for bench dev) and the command's
time starts at acquisition; a DONE status written by the child stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 23:07:10 +08:00
bench
45dcce1e7d Salty cleanup: one try, never hold a result back
From 16:50 on 10-19 Salty's storage rm answered 405 for every object; the
cleanup retried it for 30 minutes before each stage or bench dev result went
back. Garbage removal now tries once (dir first, so a failure leaves the job
listed) and salty_offload.reap_orphans tries again on its next pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 17:29:53 +08:00
bench
f6a8318891 Tests: real Pi compacts a request the account refuses, and runs bash through the agent shell
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 16:56:46 +08:00
bench
ed962b11cc Agent commands: say when an earlier pipeline step failed
Pi's bash tool (shellPath), bench dev and the board terminal run commands with
pipe-note.sh: the exit code stays the last step's (no pipefail, so | head is
still 0), and a failed earlier step of the final pipeline gets one stderr line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 16:52:42 +08:00
bench
6e1668c9a8 Quota: a request over the account's limit is a full context for Pi
Recorded FAILED CONTEXT_TOO_LONG like a provider overflow and answered with
the overflow body, so Pi compacts and continues instead of ending the rollout
on an unrecognised 413. Content review keeps its REQUEST_EXCEEDS_INTERNAL_TPM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 16:52:42 +08:00
bench
db5a7a839d Export: read the running checkout's own state wherever it lives
A checkout whose state is elsewhere (BENCH_STATE_DIR, staging) was read as
<checkout>/state, so content review and export found no database and called
a PASS rollout unfinished. One helper, paths.state_dir, used by every lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 16:52:42 +08:00
bench
20c876fd5c Staging: one rollout storage for bench dev and pipeline stages; log why paused rollouts wait
Codex review of the staging collector (10-19): a model built in bench dev was missing for 04-calibrate,
because only bench dev requests were rewritten to the container's bench/runs (reverts 56230c0). The
container now links the controller checkout's own runs/ to the same personal storage, so every
absolute path resolves alike; a no-op for production's bench/. And a paused rollout that waited ~2 h
left no trace of why: the collector logs each change of its wait reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:21:38 +08:00
bench
752943baaf Quota: each resume or restart starts a fresh wait allowance
Staging (10-19 14:06): a rollout whose lifetime wait total had passed wait_budget_s paused the first
time it was not admitted, resumed uncounted, and paused again every minute, starving the other paused
rollouts. 7 executions in production's s600 group are already over the total. The per-call bound and
the HTTP attempt budget are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:15:34 +08:00
bench
787f61fa80 Restart recovery: a dead collector's cut-off request replays at once, without pausing the shared group
Staging handover (10-19 11:40): the 2-minute hold on the dead collector's requests expired into an
unknown-outcome group pause that bounced every rollout already resumed. Its process exit closed the
connections, so the reply was cut off, not in doubt; and a platform restart no longer spends the
rollout's resume budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:03:04 +08:00
bench
c575ea0baf Restart recovery: resume turns queued for admission, wait for exiting stage reapers
Staging kill -9 with an immediate restart (10-19 11:17): a turn waiting behind the group's concurrency
read as not resumable and lost its slot, and a calibrate step whose executor was still exiting got
quarantined as SIDE_EFFECT_UNKNOWN.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 11:25:40 +08:00
bench
72912a6b17 Restart recovery: keep slots for cut-off replies, wait for exiting reapers
Found on the staging collector (10-19): after kill -9 two rollouts whose reply was cut off were read as
not resumable for request_total_s while new rollouts took their slots, and a rollout handed over with a
bash command running was quarantined because the next collector looked before the reaper finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 11:02:59 +08:00
bench
0da66edde0 Resume rollouts after the collector dies (crash, kill, reboot, planned handover)
Processes are disposable; rollout state is resumable. When the collector dies,
the next one reconciles each running rollout from durable state instead of
quarantining or losing it:

- execution: a process of an earlier boot on this host is DEAD; a top-level
  reaper whose owner (the collector) died still proves its subtree empty
  (scopes record the owner that started them); reconcile() pauses a rollout
  with a proven-clean crash checkpoint as CONTROLLER_RESTARTED and marks its
  running tool calls INTERRUPTED; resume() across a reboot rebases monotonic
  clocks by wall time and credits the downtime as a pause
- stageworker.settle_interrupted: an admitted step whose executor is gone ends
  as STAGE_EXECUTOR_EXIT (free compile) and its handle is told directly; a
  board step without a clean board receipt still quarantines; nothing replays
- collection_runtime.crash_checkpoint: capture the real system prompt, freeze
  the session at the last saved message (mid-work phases only); configure_pi
  skips a generation ID the ledger already completed
- pi_session.js: checkpoint after every saved message (and compaction); on a
  recovered resume drop an unfinished trailing reply and answer open tool
  calls with an honest "interrupted by a platform restart" result
- scheduler: a paused rollout's local slot held by a dead holder is released
  once every process is proven gone
- pipeline: SIGUSR1 = planned handover (stop claiming, let board steps
  finish, shorten in-flight request holds, exit); SIGTERM still drains
- export: bench_recovery closes the cut segment; the dropped unfinished reply
  is excluded (UNFINISHED_BEFORE_RESTART)
- tests: test_controller_restart (real Pi on Linux: kill -9 mid-command,
  kill -9 mid-reply, SIGUSR1 handover, power loss + reboot), test_settle_
  interrupted; three tests of the old quarantine-everything rule now assert
  the new outcome while keeping their guarantees (no replay, unrelated PID
  untouched, board lease kept)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 09:29:26 +08:00
bench
6c7b19f616 offload: delete a rollout's compile scratch once it is archived
hb_compile's working_dir lives on the VM's own disk (work_root) and only
05-compile/06-hbm read it. Nothing removed it: on 10-18 742 finished rollouts
held 36 GB there, the Mac mini disk (which backs the VM disk) neared full,
and the VM's disk went read-only at 18:46.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 23:55:11 +08:00
bench
6437b572a2 fix: freeze long Pi sessions at rollout end (no 1 MiB cap)
freeze_checkpoint read the session through config_bytes, which caps files at
1 MiB for tickets/configs. Long rollouts pass that, so finalization failed:
10-15..10-17, 17 rollouts, 4 recorded INFRA_ERROR instead of their real
outcome. The session read keeps the symlink/regular-file checks, no cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 10:33:37 +08:00
bench
18379401e4 v23: hand context compaction back to Pi, cut process rules from the prompt
- broker: a context-window 400 fails its generation (fences nothing) and goes
  back to Pi as context_length_exceeded, so Pi compacts and resends
- runner: new generation ID after compaction, wait for Pi's resumed run,
  record kept_messages; drop agent/context-mask.js and context_mask_chars
- models.json: contextWindow large so Pi compacts only on a real overflow,
  never on an estimate right before a pause (resume would not match)
- export: compactions[] gives what the model actually saw after each one
- prompt: keep facts and acceptance, drop how-to rules and coaching
  (SYSTEM.md 89->64 lines, e2e.md 27->20); stage status == stage wait

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 17:04:58 +08:00
bench
f2322ac926 test_pi_session:真 Pi 0.73.1 下 context_mask_chars 让请求里出现占位、最近输出完整、会话文件不变
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:49:58 +08:00
bench
b248a28fd7 上下文满记为预算用完;broker 内部错误可续跑并记日志;断线只占账号 2 分钟;04 Qwen 只发出时省去最早的工具输出
- quota:400 且正文说上下文超长(vLLM maximum context length、GLM exceeds max length)记 CONTEXT_TOO_LONG,不续跑;
  rollout 结束记 CONTEXT_BUDGET_EXHAUSTED(结局 BUDGET_EXHAUSTED,rollout.json TIMEOUT),不再是 INFRA。10-16 当天 7 条 Qwen。
- quota:do_POST 吞掉的异常打出类型、消息、位置(不含请求);一次都没发出去的 generation 记 BROKER_ERROR 暂停,
  能像连不上一样续跑。10-16 05 hrnet.w32 白跑 2.9h。
- quota:连接被断(REQUEST_CONNECTION_INTERRUPTED)的请求只占账号 UNKNOWN_HOLD_S=120 秒,不再等到请求自己的期限
  (20–27 分钟、整个账号的轨迹全停)。
- agent/context-mask.js:配置 rollouts.context_mask_chars 时(只给 04 Qwen),发出的上下文超过这个字数就把最早的工具输出
  换成占位,降到 60% 以下;会话文件保留原文;切点只在再次超限时移动。Pi 自带压缩只在两次 prompt 之间触发,用不上。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:37:55 +08:00
bench
21fce5e2e0 v22:bench stage wait 只看一眼不轮询;板上自测先查大小、拒绝有原因、失败也删副本;速度参考编不出来写 unavailable
- bench stage wait(stageclient 与 stagerun 两个入口)不再睡,没结束就返回 2 并指向 terminal_exec wait_handle。
  10-16 efficientvit.b0 400 轮里 307 轮是轮询。
- bench dev --board:先用 salty storage ls 量开发环境里的目录(>512 MiB 或 >40 个子目录直接拒绝),
  取回后已到时限也明确拒绝;拒绝原因作为命令结果回给 agent;snapshot/dev-files/tar 在任何路径都删。
  10-16 四次自测三次从 1.2 GB 的目录提交,取回 40 分钟、超时、exit 2 无原因、VM 留下 2.5 GB。
- speed_ref:默认配置编不出来时写 latency_ms_est=null、status=unavailable,并补齐以前的 FAILED。
- SYSTEM.md 对应三处;agent/VERSION 22。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:24:22 +08:00
bench
e543d7bd5f bench stage wait 一次最多等 10 分钟(原 60 秒),结束就返回
10-16:轮询的 agent 每分钟烧一轮,efficientvit.b0(v18)400 轮里 302 轮是 bench stage wait,没编译就用完预算;
vkitti.base 73 轮、zoedepth.nk 55 轮。bash 一条命令上限约 14 分钟,10 分钟在内。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 06:39:50 +08:00
bench
b3293851b9 bench dev:排队(dev 名额、Salty 容器)不算命令自己的时间
prepare 给走 Salty 的 dev 请求多 3 小时排队余量(仍不超过整条轨迹的时限),命令自己的 timeout 在容器启动后才开始算
(worker 用 spec timeout)。10-15:64Gi 的 dev 容器在 Salty 排队超过 agent 给的 300 秒,连 ls 都超时。
prepare 在 collector 里,下次重启后生效。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 04:59:46 +08:00
bench
22bf19d04e v20:板上自测直接在板卡容器里跑,S600 也开放;跑完删板上目录
用户定:板卡本身是测试环境、容器就是隔离,不再套命名空间(S600 的容器 seccomp 本来就禁)。命令在会话目录里跑,
$BENCH_OUT 拉回,会话目录照旧由 RemoteSession.close 删掉。S100P/S600 实测 0.75/0.97 ms,板上目录已删。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 04:52:13 +08:00
bench
16a9c50c8b v19:bench dev --board 板上自测(S100P),隔离环境在真板上验证过
- terminal_jail:系统目录改递归绑定(S100P 沙箱容器在 /usr 下有只读的主机挂载,用户命名空间里非递归绑定被拒)、
  加 /host_lib 和只读 /sys(BPU 运行库读核心类型)、透传板上 LD_LIBRARY_PATH(libdnn/libhbucp 在 /usr/hobot/lib)
- collector 的请求检查只认 development 目标且不能热换,所以 --board 在命令前加标记,由新进程里的 terminal.run 转成板上请求;
  只对 v18+、nash-m 开放;上传当前目录(本地文件 + 只在开发环境里的大文件,如 hbm),一次最多 30 分钟、4 GiB;跑完删副本
- S600 板卡容器的 seccomp 禁止所有命名空间,暂不开放
- 实测:rdk-sandbox 上隔离环境里 /etc/shadow、/userdata、/root 不可见、无网络,mobilenet.v2 0.79 ms,回执确认清理

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 01:18:03 +08:00
bench
80aa571a0c v18 开发环境给出本题校准图片目录 $BENCH_CALIB_IMAGES;speed_ref 加 --follow 滚动补算
10-15 夜:提示词只说把 /data/calib/ 换成 /data/calib-raw/,agent 去 /data/calib-raw 找,每条花 5-8 次 dev 命令。
speed_ref 按轮跑时一轮要等最慢的那道编完,新题排不上。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 00:57:42 +08:00
bench
2ed2f5fec0 speed_ref:只给有 v18 轨迹的题按需编(--active,默认 3 路),失败的记 FAILED 不重试
Salty 和在跑的轨迹共用:一次加 8 路时 agent 自己的 dev 作业开始排队。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 21:42:43 +08:00
bench
1f8a7e20d1 v18:图和校准由 agent 自己做(平台不兜底),速度参考由平台预算
- 04 对 v18 起的轨迹(rollout.json 版本 >= 18,stagerun 放进 ctx.extra)要求 compile.yaml 写自己的
  onnx_model 和 cal_data_dir,缺了编译前拒绝、不占编译次数;更早的轨迹规则不变
- bench dev 对 v18 轨迹不挂平台 ONNX 和校准 npy,改挂 data/calib-raw(平台 128 张训练图,
  反归一化成模型输入尺寸的 8 位图,归一化由 agent 自己做)
- scripts/speed_ref.py:每题用默认配置在 dev 容器里编一次,hb_model_info 估计延迟写到
  agent/speed_ref/<job_id>.json(jail 里只读可见、实时更新),path-guard 放行这个目录
- 提示词:删掉自己编默认配置做参考、平台图和校准集位置;VERSION 18

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 20:46:51 +08:00
bench
5bd0f1a40c Salty 失败作业不再留着:阶段失败拷完日志就删作业和目录,孤儿清理也收已结束的失败作业
失败作业的 logs、describe(OOM/驱逐原因)和内存记录早已写进本地 stderr.log,远端留着没人看,
10-15 攒了 327 个失败、95 个成功未删的作业和 485 个 personal 目录(用户同意清理,不是 SFT 数据)。
主控端被打断的那次尝试,其作业跑完后由 salty_offload.reap_orphans 删除,不再标 ponytail 遗留。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:32:57 +08:00
bench
85ae1750f1 Salty 容器里 onnxruntime 线程数限到分给容器的核数(阶段和 bench dev 都生效)
容器看得到宿主 244 个核,配额只有 4–6 个;onnxruntime 默认按核数开 125 个线程。
10-15 容器实测 resnet50(4 核):默认 320–500 ms/次,限到 4 线程 22 ms;hmct 校准写死的 10 线程约 35–48 ms。
做法:worker 把 harness/container_site 放到 PYTHONPATH 最前,其中 sitecustomize 在 onnxruntime 第一次 import 时
给 InferenceSession 加上限(OMP_NUM_THREADS),不用 onnxruntime 的进程什么都不做。它盖掉的镜像自带
sitecustomize 只装 Ubuntu 的 apport 崩溃钩子。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:28:35 +08:00
bench
b7577de0e7 板上加载模型卡住 10 分钟就按可重试的卡住结束,不再等满约 49 分钟
10-15 S600 40015、40017 两次在 load_start 之后再无进展,一直等到 guard 的期限。正常加载几秒(S600 约 4 秒)。
只限加载阶段:慢模型推理本身可能很久。主机关掉连接后,板上 guard 按断连回收并写清理回执。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:12:04 +08:00
bench
6ab088d7ee 评测分批不建新批次也能上:冻结规则增加逐对批准的等价改写,任务文件换成分批版本
- harness/eval/__init__.py:EQUIVALENT_SOURCES 列出精确的「冻结指纹 → 现指纹」对,contract_holds 判断现文件
  仍实现冻结合同(完全相同,或同一模块的已批准改写)。阶段检查、基线检查、expand 比对三处都走它。
- 批准的只有 f56ac2f 的 5 个任务文件:上板由逐张改为按实测字节分批,板上仍逐个样本推理
  (board_bench.infer_inputs),每张的预处理、后处理和指标一字未改。
- 批次的冻结记录不动,基线缓存按冻结指纹寻址,照常命中,不用重算。
- 用户 10-15:不建新批次。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:10:04 +08:00
bench
96d72b2f8c 判分版本固定处跟到 v0.8;导出清单说明改读 RECIPE_VERSION
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 14:16:48 +08:00
bench
41eb1d2193 agent v17:提示词删掉「评测数据只读可见」;判分 v0.8:板上磁盘满和板子连不上一样免罚
- dev 环境 10-14 起只挂校准集,VM 上 bench/data 为空,agent 已看不到评测数据,提示词与实际不符。
- 用户 10-15 定:板上磁盘满(disk_full)与 board_offline 同等处理,不扣 G2/G3、不扣超时分。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:53:34 +08:00
bench
b3be5cb102 板上每次调用按实测字节分批、G2 余弦按组算:大输出模型不再一次 12 GB(只取 f56ac2f 中不在冻结指纹里的部分)
task_contract 只哈希 harness/eval/<任务>.py;board.py、common.py、run.py 不在指纹里,在跑的 run 可以上。
各任务文件里的分组调用(分割/深度/光流/特征一次多张)仍等新建 run。
10-15 05 solo.r50 的 G2 一批 32 张 12 GB,容器内存撑爆,回执带不回来,整条轨迹作废。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:43:16 +08:00
bench
bf88f78f45 暂停等基础设施的时间不算 agent 的 24 小时:续跑时补回,暂停期间判超时也扣除,累计最多补一个预算
10-14 三条轨迹被 GLM 断线停了 7–16 小时,续上时只剩不到 1 小时,编译或评测做到一半被截断
(06 fcn.resnet101、depth.anything.v1.large、dpt.hybrid)。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 13:43:16 +08:00
bench
4c98f9fd4e 板上清理回执读取:超时 15→60 秒并重试 3 次,慢握手不再把已清理的轨迹判成清理未确认
10-12 至 10-15 共 9 次(8 次 TimeoutExpired、1 次 ssh 退出),每次整条轨迹作废;
最近一次 05 fcn.resnet50 r02:板上已跑完并 waitpid 确认回收,回执读取超时。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:48:46 +08:00
bench
383710c917 板卡阶段在派发前结束时补一张 not_started 回执,不再被当成清理未确认
板卡会话在第一条板上命令之前就写 UNKNOWN 回执(board_perf.bench),阶段结束时进程也都收完了,
所以本地执行完、目录里还没有回执,就证明板子没被碰过。原来这种情况(比如阶段开头的 harness 检查失败)
会被 stagerun/stageworker 判 BOARD_CLEANUP_UNCONFIRMED,执行被隔离、整条轨迹作废
(10-15 04 solov2.r50、05 deeplabv3plus.resnet50)。
只在真正执行阶段的一侧补(Salty 容器内或本地),控制端包装 Salty 的那一层拿不到这个证明,不补。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:20:15 +08:00
bench
2b024178a9 回退 f56ac2f(板上评测分批):任务评测文件被 task_contract 冻结,在跑的 run 不能热改
harness/eval/<任务>.py 的 sha256 在建 run 时写进 job.task_contract,每个阶段开头都核对。
10-15 11:08 热部署改了 seg/mde/flow/feat/coco_extra.py,之后这些任务的阶段全报
frozen task implementation changed:板卡阶段因此被判清理未确认,整条轨迹作废
(04 solov2.r50、05 deeplabv3plus.resnet50;06 yolov8m.seg 各阶段全失败后判 FAIL)。
11:45:38 已在主控上恢复旧文件。分批改动等建新 run 时再 cherry-pick f56ac2f。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:47:23 +08:00