87 Commits

Author SHA1 Message Date
bench
f83fa3948c 网络小故障不再拖垮整条轨迹:原地重发、没发出去的暂停不扣续跑次数、结果不明的请求可以续跑
GLM 请求失败率约 0.3%,但一次失败就让整条轨迹暂停;3 次续跑次数被网络故障用完后,
轨迹只能挂到 24 小时上限(10-14 有 4 条);回复中途断开的请求过了等待期会被自动放弃,
同时整条执行被取消(10-13/14 共 16 条)。

A. 连接失败、域名解析失败(RETRY_UNSENT,确定什么都没发出去)在 broker 里等一会儿原地重发。
   每次重试把等待时间加上一次连接时限计入排队预算 wait_spent,保持
   客户端超时 = 排队预算 + 次数×单次时限 的上界;预算用完才照旧暂停。
B. 续跑次数只在最近一次请求可能已经送达时才扣(看账本最近一次 attempt 是否 sent)。
   连接/解析失败、因为别的请求结果不明而在准入时被挡下的暂停,续跑不扣次数。
   执行记录和账本两处计数同步,调度程序不再按原始次数预先过滤。
C. reconcile 自动放弃时(截止时间再过一个 token 窗口)不再取消执行,而是把这一轮
   标成 REQUEST_OUTCOME_UNKNOWN、可以续跑:重发一轮只花一次请求的 token,并计入续跑次数。
   人工放弃(operator)仍然永久隔离。续跑资格检查不再被已放弃的 attempt 挡住。
都在调度程序进程里运行,下次重启生效。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 15:41:49 +08:00
bench
f74fe00aaf hbm/bc 模拟器对比改为 agent 按需开启,只作诊断
阶段配置新增 verification_parameters.hbm_bc_samples(0–16,默认 0)。agent 用
bench stage run hbm --config <yaml> 指定 N 张校准样本,用 x86 模拟器对比 HBM 与 BC,
结果写在 G1.hbm_bc_cos_min,不改 verifier_ok:开了不会被扣分,真正把关的仍是 08-eval 板端实测。
编译配置只取三段,所以同一个 yaml 拿去编译也不受影响。适用任何有 .bc 的后端;跑失败记 None。
s06_hbm 随阶段打包,部署后马上生效(默认不跑);配置校验在 stageworker、说明在 SYSTEM.md,
都要下次重启时一起上线,否则按说明提交会被旧校验拒绝。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:48:17 +08:00
bench
ba143faaae 06-hbm 去掉 hbm/bc 模拟器抽查
x86 模拟器跑 2 张图占这一步 95% 的时间(本地中位 7 分钟,Salty 上 1–2 小时);
10-12 17:16 起 Salty 上撞 2 小时上限 103 次、约 206 小时。历史 388 次抽查报出 13 次
不一致,hb_verifier 都已先判掉,从没单独发现过问题;08-eval 在板上跑真 HBM。
与 salty_stage.py 一样记 hbm_bc_cos_min=None 并注明。阶段代码随每次提交打包,
部署后新起的步骤马上生效。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:24:02 +08:00
bench
ad51e86f5f 导出进程每轮停掉本地已结束的 Salty 作业,停好后清理
阶段时限只管超时;轨迹被取消、进程崩溃时,远端容器照样跑完,没人取结果。
10-13 晚有 12 个这样的作业占着约 64 核,同时我们自己的作业在排队。
现在每轮先查 salty ps:本地这一步已经写了 meta.json、远端还在 running/pending 的,
就停掉;已经 stopped 的,按 salty_remote 处理自己停掉的作业那样删除作业和临时目录。
failed 的照旧保留,供排查。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 22:24:47 +08:00
bench
ec1a58edf8 续跑不再重新核对已结算的步骤;领题后不再重复查额度
续跑时 stageworker 会重新核对旧步骤回执。同一轨迹每次编译写同一个 .hbm,
旧回执核对失败后被当成状态不明,整条轨迹隔离;10-12/13 共 19 条编译过多次的轨迹,
续跑几秒内就判 INFRA_ERROR。现在已结算的操作直接沿用旧结果,判断条件与
reconcile_completed 相同。
collector 领题前已经查过额度,run_rollout 领到后又查一次(只等 0.5 秒);
机器忙时读超时,题被判 UNVERIFIED,一步都没跑(59 条)。领到后不再重复查。
这两处代码都在调度程序进程里运行,重启后才生效。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 11:15:05 +08:00
bench
7093c321f0 超时被杀的步骤归还名额、远端容器随阶段时限停止
步骤到了阶段时限被看门狗结束时来不及删 cpu 名额的占用记录,_try_slots 见记录就跳过,
只有调度程序重启才清;10-12 晚到 10-13 上午累计锁死 26/32 个名额。现在占用者已死、
且该步骤已记 DONE、进程清理已确认的名额可以直接接手(板卡名额不变)。
同时远端容器不随本地步骤停,7 个容器空跑最久 7.5 小时;提交时用 timeout 把容器
运行时长限制在该阶段的执行时限内(镜像自带 /usr/bin/timeout,已实测)。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 11:15:05 +08:00
bench
a7c454c0ae salty_remote: 容器里线程数按分到的核数设
容器看到整机 244 核,实际只给 4–6 核;torch/numpy 按 244 开线程,大部分时间在被限流。
10-12 实测(4 核容器):torch 默认 122 线程 2.6 次/秒,4 线程 42 次/秒;numpy 31 → 146–158 次/秒。
提交时设 OMP/MKL/OPENBLAS_NUM_THREADS=分到的核数。onnxruntime 不认这些变量,另行处理。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 10:43:02 +08:00
bench
375ce4d1f5 stagerun: 编译超时、看门狗判卡死也不扣编译次数
和容器被杀、执行器退出一样算系统的错,每条轨迹合计最多免 3 次。
10-12 17:16 后 DETR 两次编译一小时没日志没 CPU 被看门狗杀掉、
segformer.b5 编译逼近 4 小时上限,原来都会白扣模型一次机会。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 21:44:24 +08:00
bench
51d6cda544 alert: 启用小金飞书私聊报警,发送失败记日志
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:05:59 +08:00
bench
da7b77b62f alert: 可选用小金(飞书群答疑助手)私聊报警;记忆补充排空与 06 断流
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:03:38 +08:00
bench
8d39d2b69b fix(drain_restart): 修空数组在 set -u 下报错;加 NO_TERM 防重复发信号;删掉排空期间手动续跑(旧进程在世时按设计不可接手)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 16:50:51 +08:00
bench
12787a289d fix: 系统失败不占编译次数、连不上可自动续跑、暂停轨迹不被新题抢名额;加健康检查与排空重启脚本
- stagerun: 容器被杀/执行器退出不计入 5 次编译(每条最多免 3 次)
- quota: CONNECT_FAILED / REQUEST_DEADLINE_BEFORE_SEND 证明未发送,允许自动续跑
- pipeline: 有暂停轨迹只差额度时,空出的名额留给它
- scripts/health_check.py + bench_health_alert.sh: 每 10 分钟检查,异常弹本机通知
- scripts/drain_restart.sh: 排空旧调度程序后重启,排空期间用新代码续跑卡住的轨迹

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 16:38:07 +08:00
bench
df51a7c13f fix: Salty 导出只用存档里的文件,不混进容器替已放弃尝试写的东西
xfeat(05,10-11)的 06-hbm a00 主控已判超时放弃,容器却继续跑完,把 stdout.log 等写进了
个人空间的 runs。导出照搬个人空间的阶段目录,门证据多出 06-hbm/a00/stdout.log,
审核绑定和主控算的对不上,通过了审核仍显示"未审"。
现在解包时记下每个包的成员,本地目录只按成员清单搭(阶段文件和小文件拷贝、大文件链接),
存档没带的 HBM 从个人空间链接;旧的空标记会触发重新解包补清单。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 02:04:11 +08:00
bench
b4e4f6ee09 feat: scripts/salty_review_again.py 补审已搬走的 PASS 轨迹
最终数据集里"没审过/审核未知"、文件已被 salty_offload 删掉的轨迹,逐条从个人空间存档取回、
用配置的审核模型审、回执作补丁包传回、再删本地;一次只留一条(旧格式对话最大 650MB,
调度程序每次本地导出都会重读留在本地的文件)。已有 COMPLETE UNKNOWN 就走重试。

--release-started:崩溃留下的 STARTED 回执按设计不授权新请求。06 shufflenet.v2.x1.5 那条
其实请求已完成(账本 generation COMPLETE),按原执行号重审会被当作已完成而不发。放行时
原件进 history/,回执记成 COMPLETE UNKNOWN(OPERATOR_RELEASED_STARTED_RECEIPT),
由正常重试换新执行号发一次新请求。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 01:51:49 +08:00
bench
6735b703a7 feat: 轨迹搬到 Salty 个人空间,数据集在 Salty 上导出(scripts/salty_offload.py)
Mac mini 只留小文件和最终数据集。常驻进程每 10 分钟:
- 结束满 30 分钟(UNVERIFIED 24 小时)、执行 TERMINAL、PASS 已审完的轨迹打包到
  personal:/bench-root/archive/<run>/<job>/<kk>.tgz,个人空间已有同大小副本的产物不进包;
  之后的改动(如审核回执)另传 <kk>.u<时间>.tgz,旧包从不覆盖;传完核对大小再删本地 >1MB 文件。
- 有新包时最多每小时起一个 Salty 作业(salty_remote_worker.export):把包解到 personal 的 runs
  (不解 workspace),用账本快照跑 export/to_agentrl.py,数据集留在 personal:/bench-root/datasets/,
  拷回 bench/datasets-final/。
调度程序每条结束后的本地导出保留:它在文件还在本地时发现并审核新 PASS,搬运等它审完。

导出作业里 runs 不能是指向个人空间的软链接:verify.py 会 resolve 轨迹目录、拒绝软链接的预测文件,
路径对不上时 G3 变未验证、审核绑定也变(实测 SFT 0 条)。所以容器里重建真实的轨迹和阶段目录,
agent/、artifacts/ 等大目录用软链接。实测和 Mac mini 本地导出逐条一致(3/3/2)。
半截上传的包解包失败就跳过,下次再解。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:10:30 +08:00
bench
0855486585 test: usage_log 检查只跑一拍,不在临时目录删掉后报错
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 11:22:50 +08:00
bench
aa3fabf8e9 feat: salty_remote 内存需求按模型记,跨批次/板型共享
同一模型在 04/05/06 里只是 march 不同,之前每个批次都要先被 OOM 杀一次。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 11:22:33 +08:00
bench
c8a93c9605 feat: salty_remote 按题记内存需求:同一 job 任何轨迹里该阶段被 OOM 杀过,就从当时内存的两倍起步
内存需要是模型的属性;之前按单条轨迹计数,补号重跑的轨迹又从 16Gi 起,重复被杀。
meta 里记 salty_memory。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:36:40 +08:00
bench
45cff52749 fix: salty_remote 按需加内存的上限从 64Gi 提到 128Gi
r2d2 编译在 32Gi 下内存仍按每 15 秒约 1.3G 线性增长后被杀;旧编译机内存大,
上限放到 128Gi 让重模型的环境接近原来。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:13:42 +08:00
bench
f1f44c9d1e fix: 调度程序因读整个 events.jsonl 被 OOM 杀;message_update 不再逐 token 复制整条消息
Pi 每个 token 发一次 message_update,里面 message 和 assistantMessageEvent.partial
各带一份到目前为止的整条消息,events.jsonl 每条轨迹 300–800MB(99.9% 是它)。
budget_used 每次 read_text() 整个文件,调度程序峰值 5GB,10-11 07:08 在 10GB 的
Mac mini 虚拟机里被内核 OOM 杀掉。下游只读 message_update 的 type/delta,
完整文本在 message_end。budget_used 改为逐行读。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 09:47:41 +08:00
bench
df64aa85b2 feat: salty_remote 按需加内存:同一阶段容器失败(非停止)过几次,就把申请翻几倍(上限 64Gi)
统一给编译 32Gi 时共享队列只放得下一个编译作业;改为默认值不变,
只有真的被 OOM 杀过(memory.oom.group=1 无声死亡)的阶段重试时才多要内存。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 21:50:49 +08:00
bench
dd20d199d7 feat: salty_remote 容器每 15 秒记录内存用量(usage.log)
Salty 容器开着 memory.oom.group=1:任一子进程超内存,整个容器被杀,
原因留空、日志为空,看起来和磁盘驱逐一样。记录内存才能分清。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 19:34:42 +08:00
bench
8a613c63f3 feat: salty_remote 容器每分钟把 /tmp 剩余空间和最大文件记到个人空间,失败时取回
大模型编译的容器会因节点磁盘压力被驱逐(实测本地写到约 39G 时被杀,日志和原因全丢),
这份记录不随容器丢失,用来看清是哪些文件撑爆了本地盘。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 19:27:40 +08:00
bench
6a013d6b00 fix: salty_remote 容器异常退出时补写的 meta 带上操作身份
否则 stageworker 校验回执时报 completed attempt metadata mismatch,
轨迹被暂停(PAUSED_INFRA)而不是让 agent 看到一次普通的阶段失败。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 16:03:49 +08:00
bench
356fdaa0f8 fix: salty_remote 改用可重试的传输,网络抖动不再废掉轨迹
Mac mini 虚拟机里约 17% 的请求会失败,原传输(salty_client)任何一步失败都判
"副作用不明"并隔离整条轨迹:上传、下载、清理里的 describe 都中过招。改为每步可重试:
作业名每次唯一、重提前先按名字查(回复丢失也不会重复建作业),上传覆盖自己的目录,
查询/下载只读,删除遇 404 即成功;连续失败 30 分钟才判这一步失败(REMOTE_UNAVAILABLE,
普通失败、agent 可重试,板卡阶段仍由回执检查隔离板卡)。清理失败只留垃圾,不影响结果。
超时只发 SIGTERM,不强杀。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 12:51:43 +08:00
bench
0cf3c2005f fix: salty_remote 查询作业状态失败时重试,长时间断网才判失败
Mac mini 网络抖动时 salty describe 报"网络错误",原来一次失败就把阶段判成
SIDE_EFFECT_UNKNOWN、整条轨迹被隔离(10-10 06 批次 11 条、04 批次 1 条)。describe
是只读的,现在连续失败 30 分钟内持续重试;轮询间隔 15s→30s,减轻 Salty 压力。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 11:06:49 +08:00
bench
236766c19b fix: salty_remote 按快照取回新产物,不看修改时间
06 用 copy2 把 HBM 放进 artifacts/,保留了编译时的 mtime,按"本阶段开始后修改"筛会
漏掉它,主控判定时找不到 HBM(G1/G3 unverified)。改为阶段前后比较文件大小和 mtime。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 10:44:34 +08:00
bench
f6aec1f419 fix: salty_remote 取回 HBM,并去掉 32MB 上限
主控在本地重算编译回执里 HBM 的哈希(stagerun._completion_status、stageworker 复核),
最终判定也要读 rollout 的 artifacts/*.hbm,所以 05 工作目录里的 HBM、本阶段新写的
artifacts 必须回到主控同一路径;评测预测文件也要核哈希,不能截断。落盘只允许写进
本题目录和编译工作目录。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 06:32:06 +08:00
bench
e376b593fe feat: 每个阶段整步在 Salty 容器执行(salty_remote,默认关闭)
Mac mini 主控只跑 agent、调度、账本、配额和锁;agent 的每次 bench stage run
(01–08)都在 Salty 起一个容器跑完整的 Stage.run(含查缓存),容器用软链接
复刻老编译机路径,冻结的任务路径和缓存原样可用,跑完只取回阶段目录(≤32MB 的
文件)和板卡回执。配置键 execution.salty_remote,未开启时行为不变。

已验证:本地往返测试;Mac mini → Salty 容器 → 结果取回的真实链路(06-hbm 空跑)。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:59:57 +08:00
bench
45b1262f01 feat: Salty 远程编译(05-compile PTQ),默认关闭
把本地工作区里编译上云的改动单独拿出来,放到当前生产代码(c2c52ed)上:
新增 salty_client / salty_stage / salty_stage_worker 及两组测试,并在
stagerun、pipeline、stages/base、s06_hbm 里接线。只有配置里写了
execution.salty.enabled: true 且 stages 含 05-compile 时才生效,
现有配置不受影响。

合到生产代码上时补的衔接:
- 编译次数上限:已记录过提交的 Salty 编译只做找回、绝不重提,不受上限
  阻拦;只准备好、从未提交的照常拒绝(新增 salty_stage.dispatch_recorded)。
- selected() 与 cleanup_state 读取在上下文缺字段时按"未启用 / VERIFIED"处理。

未包含:进程优雅退出(owned_process / namespace_owner 等)和其余板卡侧改动,
仍留在本地工作区。

测试(Linux, python3.10):test_salty_client、test_salty_stage 全过;
stagerun/pipeline/阶段相关的 13 个已有测试与 c2c52ed 结果一致。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:19:48 +08:00
10b070e05d fix: stop holding a task's queued rollouts after a system stop
A queued rollout was never claimed once any sibling ended DEADLINE_EXCEEDED,
CANCELLED, INFRA_ERROR, LOST or BUDGET_EXHAUSTED. Almost all of those were
system stops (tool cap, context rejected by the local quota, board cleanup
or request outcome unknown, pauses that expired, operator restarts), so
1107 queued rollouts across r20260929-04/05/06 sat behind them.

Siblings still run one at a time, and two stops still hold the rest: an
exhausted turn/token budget, and a DEADLINE_EXCEEDED whose verdict says
the trajectory really worked until the 24 h execution deadline. Agent-
finished FAIL/UNVERIFIED keep not holding siblings, as before.

Tests that pinned the old hold now assert the repeat is claimable; the
lease-recovery waves no longer depend on sibling holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:30:27 +08:00
5617f57f08 fix: let a long bash command time out alone instead of ending the attempt
Bash has no default timeout, so any command running past the 900 s system
cap ended the whole trajectory (81 of 590 finished rollouts). Give bash a
native timeout one minute before the cap; only that command fails and the
agent continues. The persisted tool call keeps the model's own arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 07:34:41 +08:00
ea0335bc24 fix: release an unknown-outcome pause once the group has nothing open
The probe after an unknown request was only allowed at the moment the
unknown attempt was resolved, and only if no other attempt was open. On
2026-10-07 08:26 the Qwen group's unknown attempt was auto-abandoned while
an innocent request was still streaming (ended 08:27), probe_allowed was
false, and nothing re-checked: the group stayed PAUSED for 15 hours.

abandon_unknown, confirm_end and reconcile() now share _probe_if_settled():
an unknown-outcome pause turns into one COOLING probe as soon as no attempt
in the group is open, whichever of them sees that first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 23:47:04 +08:00
f9b3fa6871 test: drive stagerun through the compile-cap refusal path
Covers the branch itself, not only the counter: at cap-1 the compile runs;
at the cap stagerun publishes COMPILE_BUDGET_EXHAUSTED without starting a
stage row or calling the stage. Verified to fail when the check is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 00:19:16 +08:00
4332f43dab feat: cap each trajectory at three started compiles
Compile attempts are now the primary per-trajectory budget: they map to
the real compute cost, unlike turns or tokens, and make the stop reason
obvious. stagerun refuses a 4th 05-compile with COMPILE_BUDGET_EXHAUSTED
before inline calibration or slot acquisition.

- Counted from the stages ledger (05-compile rows that got a slot), so
  resume/recovery cannot reset it; a compile that timed out in the queue
  is free; recompiles after an accuracy miss count.
- Board/eval/review of the last artifact still run after the cap.
- Turn/token/wall budgets, per-phase deadlines, cleanup and API quota
  protection are unchanged as backstops.
- e2e prompt drops "no retry limit"; SYSTEM.md names compiles as the
  primary budget; agent/VERSION 12 -> 13.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:37:15 +08:00
44f643d290 fix: stop false OPERATION_STATE_UNKNOWN on finished stages
Three r20260929-06 rollouts sat in PAUSED_INFRA although their stage had
finished (status DONE/FAIL, no process left): the completion replay in
stageworker._validate_recovered_receipt raised ValueError, so the operation
stayed DISPATCHED and the execution was quarantined (one-way).

- Nested scopes: a deadline kill of the outer stage leaves the inner reaper
  with owner_lost=True although both levels reaped via waitpid(ECHILD).
  owned_processes() already accepted that when the enclosing verified receipt
  names the exact supervisor; receipt validation demanded VERIFIED. Both now
  share execution.scope_cleaned().
- Board stages killed by QUEUE_DEADLINE_EXCEEDED never held a board slot
  (phase_running fires only under the board lock) and so have no board
  receipt; that missing receipt no longer counts as unconfirmed cleanup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 09:50:23 +08:00
f61d2efebb fix: replay board ssh calls dropped before authentication
Board ssh ports are public and constantly hold 15-35 unauthenticated
scanner connections; whenever that exceeds sshd MaxStartups, our handshake
is closed with kex_exchange_identification. Only upload steps retried, so
a single drop failed the bench call itself (exit 255) or lost the cleanup
receipt fetch, which then quarantined the board.

run() and guarded_run() now replay a call up to twice when ssh reports
kex_exchange_identification: the connection closed before auth, so the
board never saw the command and replay cannot duplicate work. Any other
failure is returned unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 09:18:48 +08:00
f315496d18 fix: clear a board-side quarantine once its operation has no survivors
board_guard writes .board-inference.quarantine.json before spawning the
inference child. If the guard is killed (ssh drop, SIGKILL) the file stays
and every later 07-board/08-eval on that board exits 78 until a human
deletes it; rdk-sandbox-40009 was blocked this way for 1.5 days.

The guard already holds the board lock every admitted child inherits. It
now also checks /proc for any process carrying the old operation token;
with none left, the record is stale: it is removed and logged as
stale_quarantine_cleared, then the normal occupancy check still applies.
Survivors keep the board quarantined. Unverified-cleanup receipts now
store the token so the same rule covers them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:44:20 +08:00
80a8e246c0 fix: release board slots whose holder died without a cleanup receipt
When board ssh drops during cleanup, no receipt is written and the board
lease stayed quarantined forever; all 8 boards ended up locked and every
rollout timed out in the board queue (QUEUE_DEADLINE_EXCEEDED).

reconcile_leases now probes the board when the holder is dead, the
execution is terminal and no local process remains. If the board runs
nothing but sshd/ps the slot is released and board-release-<slot>.json
records the evidence; busy or unreachable boards stay quarantined.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 08:07:22 +08:00
2f54d407cb fix: recover from unknown request outcomes without stopping the collector
- quota: once an UNKNOWN attempt is a full token window past its local
  deadline, abandon it via the operator path (evidence kept, ended_at NULL,
  generation fenced); the group then gets one probe.
- quota: generations paused at admission by that group pause may resume
  when none of their own attempts is unended.
- pipeline: isolate an identifiable unreconciled rollout and keep
  collecting; unidentifiable reports still stop. Drive broker reconcile
  every loop so a fully paused group can recover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:18:11 +08:00
2eb29caa83 fix: let dead, quarantined failed executions stop blocking unrelated work
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 21:17:40 +08:00
10663a800f fix: preserve completed calibration when later compilation is cancelled 2026-09-29 19:04:50 +08:00
13f1e99aee fix: distinguish empty documentation searches from tool failures 2026-09-29 18:29:59 +08:00
3a30aac297 fix: persist board dispatch evidence before remote execution 2026-09-29 16:33:16 +08:00
498187569c fix: close rollout finalization and review publication gaps
Reuse execution, stage and review evidence to recover database registration and
SFT publication without replaying completed work. Keep finished futures during
SQLite contention, and hand durable results to startup recovery when draining.
Release proven local leases independently of unresolved board isolation.

Keep UNKNOWN review retries explicit and archived, publish completed receipts
after transient export failures, and prevent a malformed sample from aborting
the entire dataset. Scope routine reconciliation to newly finished attempts.
Add failure-injection coverage and the production acceptance plan.
2026-09-29 16:09:01 +08:00
e1e73816af fix: recover stage registration and reclaim completed local leases 2026-09-29 12:55:35 +08:00
2da29f8050 fix: preserve deployed recovery and board cleanup behavior
Record the deployed continuation, process reaping, recovery, and cleanup fixes with the current board registry. Add focused regression coverage and document the shared production branch and existing startup configuration.
2026-09-29 01:27:19 +08:00
8bb1694499 fix: retry transient tool event reads once 2026-09-24 02:00:12 +08:00
b398495d54 fix: allow 32768 tokens for content review 2026-09-23 23:29:04 +08:00
6b62baf1fd fix: scope execution readiness to the requested run 2026-09-23 22:34:41 +08:00