70 Commits

Author SHA1 Message Date
bench
c8a93c9605 feat: salty_remote 按题记内存需求:同一 job 任何轨迹里该阶段被 OOM 杀过,就从当时内存的两倍起步
内存需要是模型的属性;之前按单条轨迹计数,补号重跑的轨迹又从 16Gi 起,重复被杀。
meta 里记 salty_memory。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:36:40 +08:00
bench
45cff52749 fix: salty_remote 按需加内存的上限从 64Gi 提到 128Gi
r2d2 编译在 32Gi 下内存仍按每 15 秒约 1.3G 线性增长后被杀;旧编译机内存大,
上限放到 128Gi 让重模型的环境接近原来。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:13:42 +08:00
bench
f1f44c9d1e fix: 调度程序因读整个 events.jsonl 被 OOM 杀;message_update 不再逐 token 复制整条消息
Pi 每个 token 发一次 message_update,里面 message 和 assistantMessageEvent.partial
各带一份到目前为止的整条消息,events.jsonl 每条轨迹 300–800MB(99.9% 是它)。
budget_used 每次 read_text() 整个文件,调度程序峰值 5GB,10-11 07:08 在 10GB 的
Mac mini 虚拟机里被内核 OOM 杀掉。下游只读 message_update 的 type/delta,
完整文本在 message_end。budget_used 改为逐行读。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 09:47:41 +08:00
bench
df64aa85b2 feat: salty_remote 按需加内存:同一阶段容器失败(非停止)过几次,就把申请翻几倍(上限 64Gi)
统一给编译 32Gi 时共享队列只放得下一个编译作业;改为默认值不变,
只有真的被 OOM 杀过(memory.oom.group=1 无声死亡)的阶段重试时才多要内存。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 21:50:49 +08:00
bench
dd20d199d7 feat: salty_remote 容器每 15 秒记录内存用量(usage.log)
Salty 容器开着 memory.oom.group=1:任一子进程超内存,整个容器被杀,
原因留空、日志为空,看起来和磁盘驱逐一样。记录内存才能分清。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 19:34:42 +08:00
bench
8a613c63f3 feat: salty_remote 容器每分钟把 /tmp 剩余空间和最大文件记到个人空间,失败时取回
大模型编译的容器会因节点磁盘压力被驱逐(实测本地写到约 39G 时被杀,日志和原因全丢),
这份记录不随容器丢失,用来看清是哪些文件撑爆了本地盘。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 19:27:40 +08:00
bench
6a013d6b00 fix: salty_remote 容器异常退出时补写的 meta 带上操作身份
否则 stageworker 校验回执时报 completed attempt metadata mismatch,
轨迹被暂停(PAUSED_INFRA)而不是让 agent 看到一次普通的阶段失败。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 16:03:49 +08:00
bench
356fdaa0f8 fix: salty_remote 改用可重试的传输,网络抖动不再废掉轨迹
Mac mini 虚拟机里约 17% 的请求会失败,原传输(salty_client)任何一步失败都判
"副作用不明"并隔离整条轨迹:上传、下载、清理里的 describe 都中过招。改为每步可重试:
作业名每次唯一、重提前先按名字查(回复丢失也不会重复建作业),上传覆盖自己的目录,
查询/下载只读,删除遇 404 即成功;连续失败 30 分钟才判这一步失败(REMOTE_UNAVAILABLE,
普通失败、agent 可重试,板卡阶段仍由回执检查隔离板卡)。清理失败只留垃圾,不影响结果。
超时只发 SIGTERM,不强杀。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 12:51:43 +08:00
bench
0cf3c2005f fix: salty_remote 查询作业状态失败时重试,长时间断网才判失败
Mac mini 网络抖动时 salty describe 报"网络错误",原来一次失败就把阶段判成
SIDE_EFFECT_UNKNOWN、整条轨迹被隔离(10-10 06 批次 11 条、04 批次 1 条)。describe
是只读的,现在连续失败 30 分钟内持续重试;轮询间隔 15s→30s,减轻 Salty 压力。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 11:06:49 +08:00
bench
236766c19b fix: salty_remote 按快照取回新产物,不看修改时间
06 用 copy2 把 HBM 放进 artifacts/,保留了编译时的 mtime,按"本阶段开始后修改"筛会
漏掉它,主控判定时找不到 HBM(G1/G3 unverified)。改为阶段前后比较文件大小和 mtime。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 10:44:34 +08:00
bench
f6aec1f419 fix: salty_remote 取回 HBM,并去掉 32MB 上限
主控在本地重算编译回执里 HBM 的哈希(stagerun._completion_status、stageworker 复核),
最终判定也要读 rollout 的 artifacts/*.hbm,所以 05 工作目录里的 HBM、本阶段新写的
artifacts 必须回到主控同一路径;评测预测文件也要核哈希,不能截断。落盘只允许写进
本题目录和编译工作目录。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 06:32:06 +08:00
bench
e376b593fe feat: 每个阶段整步在 Salty 容器执行(salty_remote,默认关闭)
Mac mini 主控只跑 agent、调度、账本、配额和锁;agent 的每次 bench stage run
(01–08)都在 Salty 起一个容器跑完整的 Stage.run(含查缓存),容器用软链接
复刻老编译机路径,冻结的任务路径和缓存原样可用,跑完只取回阶段目录(≤32MB 的
文件)和板卡回执。配置键 execution.salty_remote,未开启时行为不变。

已验证:本地往返测试;Mac mini → Salty 容器 → 结果取回的真实链路(06-hbm 空跑)。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:59:57 +08:00
bench
45b1262f01 feat: Salty 远程编译(05-compile PTQ),默认关闭
把本地工作区里编译上云的改动单独拿出来,放到当前生产代码(c2c52ed)上:
新增 salty_client / salty_stage / salty_stage_worker 及两组测试,并在
stagerun、pipeline、stages/base、s06_hbm 里接线。只有配置里写了
execution.salty.enabled: true 且 stages 含 05-compile 时才生效,
现有配置不受影响。

合到生产代码上时补的衔接:
- 编译次数上限:已记录过提交的 Salty 编译只做找回、绝不重提,不受上限
  阻拦;只准备好、从未提交的照常拒绝(新增 salty_stage.dispatch_recorded)。
- selected() 与 cleanup_state 读取在上下文缺字段时按"未启用 / VERIFIED"处理。

未包含:进程优雅退出(owned_process / namespace_owner 等)和其余板卡侧改动,
仍留在本地工作区。

测试(Linux, python3.10):test_salty_client、test_salty_stage 全过;
stagerun/pipeline/阶段相关的 13 个已有测试与 c2c52ed 结果一致。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:19:48 +08:00
10b070e05d fix: stop holding a task's queued rollouts after a system stop
A queued rollout was never claimed once any sibling ended DEADLINE_EXCEEDED,
CANCELLED, INFRA_ERROR, LOST or BUDGET_EXHAUSTED. Almost all of those were
system stops (tool cap, context rejected by the local quota, board cleanup
or request outcome unknown, pauses that expired, operator restarts), so
1107 queued rollouts across r20260929-04/05/06 sat behind them.

Siblings still run one at a time, and two stops still hold the rest: an
exhausted turn/token budget, and a DEADLINE_EXCEEDED whose verdict says
the trajectory really worked until the 24 h execution deadline. Agent-
finished FAIL/UNVERIFIED keep not holding siblings, as before.

Tests that pinned the old hold now assert the repeat is claimable; the
lease-recovery waves no longer depend on sibling holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:30:27 +08:00
5617f57f08 fix: let a long bash command time out alone instead of ending the attempt
Bash has no default timeout, so any command running past the 900 s system
cap ended the whole trajectory (81 of 590 finished rollouts). Give bash a
native timeout one minute before the cap; only that command fails and the
agent continues. The persisted tool call keeps the model's own arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 07:34:41 +08:00
ea0335bc24 fix: release an unknown-outcome pause once the group has nothing open
The probe after an unknown request was only allowed at the moment the
unknown attempt was resolved, and only if no other attempt was open. On
2026-10-07 08:26 the Qwen group's unknown attempt was auto-abandoned while
an innocent request was still streaming (ended 08:27), probe_allowed was
false, and nothing re-checked: the group stayed PAUSED for 15 hours.

abandon_unknown, confirm_end and reconcile() now share _probe_if_settled():
an unknown-outcome pause turns into one COOLING probe as soon as no attempt
in the group is open, whichever of them sees that first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 23:47:04 +08:00
f9b3fa6871 test: drive stagerun through the compile-cap refusal path
Covers the branch itself, not only the counter: at cap-1 the compile runs;
at the cap stagerun publishes COMPILE_BUDGET_EXHAUSTED without starting a
stage row or calling the stage. Verified to fail when the check is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 00:19:16 +08:00
4332f43dab feat: cap each trajectory at three started compiles
Compile attempts are now the primary per-trajectory budget: they map to
the real compute cost, unlike turns or tokens, and make the stop reason
obvious. stagerun refuses a 4th 05-compile with COMPILE_BUDGET_EXHAUSTED
before inline calibration or slot acquisition.

- Counted from the stages ledger (05-compile rows that got a slot), so
  resume/recovery cannot reset it; a compile that timed out in the queue
  is free; recompiles after an accuracy miss count.
- Board/eval/review of the last artifact still run after the cap.
- Turn/token/wall budgets, per-phase deadlines, cleanup and API quota
  protection are unchanged as backstops.
- e2e prompt drops "no retry limit"; SYSTEM.md names compiles as the
  primary budget; agent/VERSION 12 -> 13.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:37:15 +08:00
44f643d290 fix: stop false OPERATION_STATE_UNKNOWN on finished stages
Three r20260929-06 rollouts sat in PAUSED_INFRA although their stage had
finished (status DONE/FAIL, no process left): the completion replay in
stageworker._validate_recovered_receipt raised ValueError, so the operation
stayed DISPATCHED and the execution was quarantined (one-way).

- Nested scopes: a deadline kill of the outer stage leaves the inner reaper
  with owner_lost=True although both levels reaped via waitpid(ECHILD).
  owned_processes() already accepted that when the enclosing verified receipt
  names the exact supervisor; receipt validation demanded VERIFIED. Both now
  share execution.scope_cleaned().
- Board stages killed by QUEUE_DEADLINE_EXCEEDED never held a board slot
  (phase_running fires only under the board lock) and so have no board
  receipt; that missing receipt no longer counts as unconfirmed cleanup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 09:50:23 +08:00
f61d2efebb fix: replay board ssh calls dropped before authentication
Board ssh ports are public and constantly hold 15-35 unauthenticated
scanner connections; whenever that exceeds sshd MaxStartups, our handshake
is closed with kex_exchange_identification. Only upload steps retried, so
a single drop failed the bench call itself (exit 255) or lost the cleanup
receipt fetch, which then quarantined the board.

run() and guarded_run() now replay a call up to twice when ssh reports
kex_exchange_identification: the connection closed before auth, so the
board never saw the command and replay cannot duplicate work. Any other
failure is returned unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 09:18:48 +08:00
f315496d18 fix: clear a board-side quarantine once its operation has no survivors
board_guard writes .board-inference.quarantine.json before spawning the
inference child. If the guard is killed (ssh drop, SIGKILL) the file stays
and every later 07-board/08-eval on that board exits 78 until a human
deletes it; rdk-sandbox-40009 was blocked this way for 1.5 days.

The guard already holds the board lock every admitted child inherits. It
now also checks /proc for any process carrying the old operation token;
with none left, the record is stale: it is removed and logged as
stale_quarantine_cleared, then the normal occupancy check still applies.
Survivors keep the board quarantined. Unverified-cleanup receipts now
store the token so the same rule covers them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:44:20 +08:00
80a8e246c0 fix: release board slots whose holder died without a cleanup receipt
When board ssh drops during cleanup, no receipt is written and the board
lease stayed quarantined forever; all 8 boards ended up locked and every
rollout timed out in the board queue (QUEUE_DEADLINE_EXCEEDED).

reconcile_leases now probes the board when the holder is dead, the
execution is terminal and no local process remains. If the board runs
nothing but sshd/ps the slot is released and board-release-<slot>.json
records the evidence; busy or unreachable boards stay quarantined.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:07:22 +08:00
2f54d407cb fix: recover from unknown request outcomes without stopping the collector
- quota: once an UNKNOWN attempt is a full token window past its local
  deadline, abandon it via the operator path (evidence kept, ended_at NULL,
  generation fenced); the group then gets one probe.
- quota: generations paused at admission by that group pause may resume
  when none of their own attempts is unended.
- pipeline: isolate an identifiable unreconciled rollout and keep
  collecting; unidentifiable reports still stop. Drive broker reconcile
  every loop so a fully paused group can recover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:18:11 +08:00
2eb29caa83 fix: let dead, quarantined failed executions stop blocking unrelated work
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:17:40 +08:00
10663a800f fix: preserve completed calibration when later compilation is cancelled 2026-09-29 19:04:50 +08:00
13f1e99aee fix: distinguish empty documentation searches from tool failures 2026-09-29 18:29:59 +08:00
3a30aac297 fix: persist board dispatch evidence before remote execution 2026-09-29 16:33:16 +08:00
498187569c fix: close rollout finalization and review publication gaps
Reuse execution, stage and review evidence to recover database registration and
SFT publication without replaying completed work. Keep finished futures during
SQLite contention, and hand durable results to startup recovery when draining.
Release proven local leases independently of unresolved board isolation.

Keep UNKNOWN review retries explicit and archived, publish completed receipts
after transient export failures, and prevent a malformed sample from aborting
the entire dataset. Scope routine reconciliation to newly finished attempts.
Add failure-injection coverage and the production acceptance plan.
2026-09-29 16:09:01 +08:00
e1e73816af fix: recover stage registration and reclaim completed local leases 2026-09-29 12:55:35 +08:00
2da29f8050 fix: preserve deployed recovery and board cleanup behavior
Record the deployed continuation, process reaping, recovery, and cleanup fixes with the current board registry. Add focused regression coverage and document the shared production branch and existing startup configuration.
2026-09-29 01:27:19 +08:00
8bb1694499 fix: retry transient tool event reads once 2026-09-24 02:00:12 +08:00
b398495d54 fix: allow 32768 tokens for content review 2026-09-23 23:29:04 +08:00
6b62baf1fd fix: scope execution readiness to the requested run 2026-09-23 22:34:41 +08:00
Codex
9af7952ab5 fix: consume tool boundaries under lock and diagnose transport errors 2026-09-23 22:00:17 +08:00
Codex
e1d5a96252 fix: match fixed top-down poses by source identity 2026-09-23 21:57:25 +08:00
Codex
b81bc80875 fix: reuse managed waits and drain accepted stage work 2026-09-23 21:57:25 +08:00
Codex
d35c82a8bf fix: integrate managed content review into dataset refresh 2026-09-23 21:30:40 +08:00
Codex
1c21193221 fix: parse proc bytes and honor quota holds before claim 2026-09-23 01:28:21 +08:00
7440ba8cb8 fix: verify nested reaper cleanup through enclosing receipts
Cancellation can end a compiler parent before its nested reaper. The nested
receipt then reports owner loss even when both reapers reached waitpid(ECHILD).
Recognize the verified enclosing receipt only when execution, epoch, immediate
parent, and full supervisor birth identity match and the nested children were
fully reaped. Preserve the original receipts and fail closed without that proof.

Add eight regression tests, including identity mismatch, incomplete cleanup,
missing parent proof, and live-process preservation. The positive regression
fails on the previous code and passes on the patch; recorded incident receipts
also distinguish the two implementations without changing the cancelled outcome.
2026-09-22 20:29:45 +08:00
f611ef29ab fix: keep managed API requests within the broker lifecycle
Pi requests could hit Node fetch's independent five-minute header deadline,
and queued requests could outlive their local client or terminal rollout.
Set header/body deadlines only for the loopback broker origin, cancel scopes
when clients disconnect or executions terminate, and retain uncertain upstream
reservations instead of retrying after cancellation. Resumable pauses remain open.

Validation: real Pi SDK accepted a response delayed 310 seconds (314.139 s test);
quota, Pi session, acceptance, cancellation, and lifecycle regression coverage.
2026-09-22 20:28:42 +08:00
Codex
068c56043f Merge managed natural-agent runtime into retained benchmark mainline 2026-09-22 15:35:28 +08:00
Codex
5884a6d7c6 Snapshot existing adapters and environment before extension merge 2026-09-22 14:56:41 +08:00
bench
d309109f2b pre-grind snapshot 2026-09-19: batches 1-7 adapters/probes/recipes + eval additive changes 2026-09-19 17:52:03 +08:00
bench
ee4c7fce1c reward v0.4: board_offline 不加分只免罚;board-backfill 支持 --rollout 用现成 HBM 补测
- export/reward.py: G3 unverified 且最后一次 07/08 为 board_offline → G2/G3 权重 0、G1 保持 1.0、免 TIMEOUT 罚分,
  返回 unverifiable=board_offline;RECIPE v0.4。实测 r20260916-01 regnet.y.800mf −0.97 → +0.03
- export/to_agentrl.py: 记录 reward_unverifiable
- harness/pipeline.py: board_backfill_rollout:用 rollouts/kk/artifacts 的 HBM 补跑 07/08(lane=backfill)后 verify_rollout
- harness/cli.py: bench board-backfill --rollout JOB_ID[:KK]
- 单测 scripts/test_reward_board_offline.py、scripts/test_rollout_backfill.py;CONTRACT §9 / README 同步
2026-09-16 15:58:24 +08:00
bench
668a7b1d16 Schedule board stages across independent S100P devices 2026-09-16 11:04:30 +08:00
bench
3e722f3e94 Collect trajectories across models before repeated sampling 2026-09-16 08:23:00 +08:00
bench
cd220cc81e fix: preserve deployment outcomes and export completed run data 2026-09-16 07:55:44 +08:00
bench
864c20a087 fix: regenerate configured repairs and use glm-5.3 for collection 2026-09-16 02:32:06 +08:00
bench
4a0e7d521f fix: run S100P board accuracy through the existing pipeline 2026-09-16 02:17:36 +08:00
bench
c717d01d97 sync 2026-09-15_12:46:30 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:46:30 +08:00