56 Commits

Author SHA1 Message Date
Codex
12ef41c42e chore: preserve existing local target and collection plan changes 2026-09-23 21:31:43 +08:00
bench
5d46a47296 Own board inference lifetime across SSH disconnects and reject contention 2026-09-19 12:44:07 +08:00
bench
c44d3f3f5e 清理:归档三份历史文档,删掉零引用死代码,修同步链路的三个洞
归档到 docs/archive/(全部零代码引用):
- PLAN.md —— §7 前三条运维教训已被 CONTRACT §12 覆盖,只把缺失的"换代必须 drain"
  补进 §12;后两条(events.jsonl 在爆、heartbeat 健康时零输出)是未修的 bug 不是
  规矩,迁进 CONTINUE.md 当待办。
- SPEC-two-phase-compile.md —— 归档说明写明是"未实现,暂不做"。之前记成"被 long
  车道替代"是错的:long 只是 STALLED 后把看门狗阈值抬到 3600s 的重试车道,与两阶段
  编译无关;本文的动机测量(量化 107s / BPU 编译 4680s)至今仍成立。
- REPORT-codex.md + evidence-codex.md —— 同批改 HANDBOOK §8 的链接和 REPORT 内部
  指向 evidence 的相对链接(现在同目录)。

删除(全库零引用,已逐个 grep 确认):
- harness/oracle.py(329 字节,无 importer)
- scripts/gate_summary.py
- scripts/_probe.py —— 删而不归档:它自述"一次性诊断",且做的正是被禁止的
  runs/ 全量 glob 遍历,归档只会让下一个人捡起来再跑一次。

同步链路三个洞:
- docs 不在 rsync 源列表里 —— 不补的话归档是静默空操作:docs/archive 永远推不到
  机器上,而 --delete 也管不到不在源列表里的远端残留。
- REPORT-codex.md 从源列表移除(它已归档进 docs/)。
- 两处 __pycache__ 清理的 find 都漏了 bin,导致 bin/__pycache__ 删了会再生。

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 21:54:03 +08:00
bench
cea5cb5415 watchdog: 按线程而非线程组长判定 D 状态,修复链接期误杀
_proc_tree_stat 只读 /proc/<pid>/stat,拿到的是线程组长的状态。实测 ld.lld 组长是
S(futex_wait),但 17 个线程里有 6~16 个 D 在 FUSE 回写(wait_on_page_bit_common /
balance_dirty_pages / fuse_wait_on_page_writeback)。1001 次观测中 59.7% 属于
"组长口径 0、线程口径 >0",于是正在往 JuiceFS 写盘的链接期被判成挂死:日志不涨 +
CPU 空闲 + d_count=0 三条同时成立,long 车道跑满 3600s 被 SIGTERM。

活体证据:efficientnet.v2.l 日志 511 字节连续 11 次轮询不变、CPU 0.0~0.3%,
旧口径 d_count 恒 0(正在累加 stalled_for),新口径 10~11。对照组 raft.large 在
103% CPU 下新口径恒 0,正常编译不受影响;非链接期 hb_compile 897 次采样 0 次命中,
豁免不会泛化。近期 36 次 05-compile 尝试中 24 次 STALLED、约 49.7 机时。

改动:
- 新增 _d_threads(),遍历 /proc/<pid>/task/*/stat 数 D 线程。task/ 含组长自身,
  是原进程级判定的严格超集。只数 D,不把 futex 等待算豁免——每个 hb_compile 常驻
  40+ 个 futex_wait 线程,算进来会让 stall 判定永久失效。
- 接上一直没接线的 io_stall_s(config 里补键并由 base.py 传入)。它此前是死代码,
  因为 d_count 恒 0;这次保持 21600 不下调,避免把误杀换个理由重演。
- 新增 no_progress_ceiling_s(默认 None,仅 base.py 显式开启):stalled_for 与
  io_blocked_for 会互相清零,d_count 抖动时两个阈值都到不了,仿真显示会塌到 24h
  hard_ceiling。它只看"距上次日志增长或 CPU 活动",免疫 D 抖动。默认关闭是因为
  run() 另有 rollout.py / s06_hbm.py 四个调用点,rollout.py 刻意把 stall_s 设成
  hard_ceiling_s(不 stall-kill agent),改默认值会静默腰斩它。
- _proc_tree_stat 与 _d_threads 的 stat 解析一并移入 try:进程/线程随时消失会给出
  空串或截断串,异常冲出轮询循环会跳过 _kill_tree,把子进程留成孤儿。

heartbeat.py:89/93 复用 _proc_tree_stat 的 d_count,其假 STALLED 告警一并修复。
性能:整树多读约 2.2ms/次,24 路并发折合 0.18% 单核。

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 19:51:59 +08:00
bench
96bef15dac sync 2026-09-17_12:09:56 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 12:09:56 +08:00
bench
5bf70ad83a sync 2026-09-17_11:58:14 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 11:58:14 +08:00
bench
716e269f73 sync 2026-09-17_09:38:54 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 09:38:54 +08:00
bench
555b48ee11 sync 2026-09-17_09:15:58 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 09:15:58 +08:00
bench
321372373f sync 2026-09-17_08:47:20 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 08:47:20 +08:00
bench
cd7ed0aad5 sync 2026-09-17_08:30:13 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 08:30:13 +08:00
bench
add895fa52 sync 2026-09-17_08:17:15 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 08:17:15 +08:00
bench
03bd5cdea2 sync 2026-09-17_07:56:11 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 07:56:11 +08:00
bench
2afbe41670 sync 2026-09-17_07:40:01 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 07:40:01 +08:00
bench
185c9ba67e sync 2026-09-17_07:26:26 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 07:26:27 +08:00
bench
f3be247163 sync 2026-09-17_07:17:18 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 07:17:18 +08:00
bench
1bafa14a80 sync 2026-09-17_07:11:45 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 07:11:45 +08:00
bench
ca45825e53 sync 2026-09-17_07:05:37 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 07:05:37 +08:00
bench
1cdd8ffb63 sync 2026-09-17_06:56:29 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 06:56:29 +08:00
bench
3ed3c670ad sync 2026-09-17_06:50:41 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 06:50:41 +08:00
bench
76cd25322e sync 2026-09-17_06:36:34 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 06:36:34 +08:00
bench
d6959325d3 sync 2026-09-17_01:38:21 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 01:38:21 +08:00
bench
9d058deb66 sync 2026-09-17_00:17:32 d-robotics@d-roboticsdeMacBook-Pro 2026-09-17 00:17:32 +08:00
bench
67264799e8 sync 2026-09-16_23:33:45 d-robotics@d-roboticsdeMacBook-Pro 2026-09-16 23:33:45 +08:00
bench
b0fc22c65c sync 2026-09-16_23:32:00 d-robotics@d-roboticsdeMacBook-Pro 2026-09-16 23:32:00 +08:00
bench
bc00f70c39 sync 2026-09-16_23:26:25 d-robotics@d-roboticsdeMacBook-Pro 2026-09-16 23:26:25 +08:00
bench
e1c8b06562 sync 2026-09-16_23:00:48 d-robotics@d-roboticsdeMacBook-Pro 2026-09-16 23:00:49 +08:00
bench
5a5c5b4fd3 sync 2026-09-16_22:18:53 d-robotics@d-roboticsdeMacBook-Pro 2026-09-16 22:18:53 +08:00
bench
3df455d19d sync 2026-09-16_22:01:31 d-robotics@d-roboticsdeMacBook-Pro 2026-09-16 22:01:31 +08:00
bench
fc7eb3b2ab sync 2026-09-16_20:46:00 d-robotics@d-roboticsdeMacBook-Pro 2026-09-16 20:46:00 +08:00
bench
ee4c7fce1c reward v0.4: board_offline 不加分只免罚;board-backfill 支持 --rollout 用现成 HBM 补测
- export/reward.py: G3 unverified 且最后一次 07/08 为 board_offline → G2/G3 权重 0、G1 保持 1.0、免 TIMEOUT 罚分,
  返回 unverifiable=board_offline;RECIPE v0.4。实测 r20260916-01 regnet.y.800mf −0.97 → +0.03
- export/to_agentrl.py: 记录 reward_unverifiable
- harness/pipeline.py: board_backfill_rollout:用 rollouts/kk/artifacts 的 HBM 补跑 07/08(lane=backfill)后 verify_rollout
- harness/cli.py: bench board-backfill --rollout JOB_ID[:KK]
- 单测 scripts/test_reward_board_offline.py、scripts/test_rollout_backfill.py;CONTRACT §9 / README 同步
2026-09-16 15:58:24 +08:00
bench
668a7b1d16 Schedule board stages across independent S100P devices 2026-09-16 11:04:30 +08:00
bench
3e722f3e94 Collect trajectories across models before repeated sampling 2026-09-16 08:23:00 +08:00
bench
cd220cc81e fix: preserve deployment outcomes and export completed run data 2026-09-16 07:55:44 +08:00
bench
864c20a087 fix: regenerate configured repairs and use glm-5.3 for collection 2026-09-16 02:32:06 +08:00
bench
4a0e7d521f fix: run S100P board accuracy through the existing pipeline 2026-09-16 02:17:36 +08:00
bench
c717d01d97 sync 2026-09-15_12:46:30 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:46:30 +08:00
bench
cb68619267 sync 2026-09-15_12:42:44 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:42:44 +08:00
bench
ed9c7381c0 sync 2026-09-15_12:39:05 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:39:05 +08:00
bench
3dce78e49d sync 2026-09-15_12:38:10 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:38:10 +08:00
bench
5ad7d4b05c sync 2026-09-15_12:36:44 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:36:44 +08:00
bench
546ec40019 sync 2026-09-15_12:35:36 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:35:36 +08:00
bench
1fd42dff20 sync 2026-09-15_12:32:50 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:32:50 +08:00
bench
b55efc94ce sync 2026-09-15_12:30:11 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:30:12 +08:00
bench
04190b7b4c sync 2026-09-15_12:27:43 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:27:43 +08:00
bench
5bafe1d7f4 sync 2026-09-15_12:26:22 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:26:22 +08:00
bench
1838ceb36d sync 2026-09-15_12:24:58 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:24:58 +08:00
bench
0e61344b34 sync 2026-09-15_12:23:21 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:23:21 +08:00
bench
50edd8c2a1 sync 2026-09-15_12:21:38 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 12:21:38 +08:00
bench
ba1e1a7bd1 sync 2026-09-15_11:34:17 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 11:34:17 +08:00
bench
d191b68b88 sync 2026-09-15_11:26:25 d-robotics@d-roboticsdeMacBook-Pro 2026-09-15 11:26:25 +08:00