fix: let a long bash command time out alone instead of ending the attempt

Bash has no default timeout, so any command running past the 900 s system
cap ended the whole trajectory (81 of 590 finished rollouts). Give bash a
native timeout one minute before the cap; only that command fails and the
agent continues. The persisted tool call keeps the model's own arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
ming01.wang 2026-10-08 07:34:41 +08:00
parent ea0335bc24
commit 5617f57f08
4 changed files with 21 additions and 1 deletions

View File

@ -7,6 +7,7 @@
- 本 rollout 的日志、阶段产物、job.json 与任务数据可读;正式提交和阶段证据只读。其他 rollout、状态库、凭据与 oracle 不属于本任务。
- 公共可复用工具位于 `$BENCH_ROOT/harness/adapters/`;`BENCH_ROOT` 已由运行环境提供,可读、可复制修改自己的副本,不是新增模型的必经入口。评判代码仍不作为 Agent 的实现依赖。
- 查找文件先用 `rg --files`,搜索内容用 `rg`;限定在 workspace、任务返回的日志目录或上述公共 adapters 目录。不要执行 `find /` 或遍历整台服务器来寻找代码;路径不可访问时依据工具报错处理。
- 单条 bash 命令最长约 14 分钟,到时只结束这条命令(返回 `Command timed out`),轨迹继续。更久的工作放后台运行(`nohup … > log 2>&1 &`),再分次查看日志;不要用 `sleep` 一次等待超过 10 分钟。
- 现有 Pi legacy 沙箱仍保留 shell 路径限制;工具报限制时按其说明使用明确路径。不要为私有环境安装修改共享工具链。
- 目标板卡、数据、指标与阈值以 job.json 为准。编译失败、图变化、精度下降都如实保留;只有平台的实际测量能作为验收证据。
- 阶段在后台运行;提交返回 handle 后调用 `terminal_exec({"wait_handle": "<handle>"})`,一次等待原阶段完成或原期限终止,不消耗模型轮数反复轮询。不要为等待重提交阶段;诊断时只查看该 handle 返回的日志与对应阶段,不遍历所有历史产物。旧 `bench stage wait <handle> --poll 60` 仍可用于一次短查询,返回 2 仅表示仍在排队或运行。

View File

@ -1 +1 @@
14
15

View File

@ -170,6 +170,14 @@ async function main(configFile) {
if (toolStop || signal?.aborted) return {block: true, reason: toolStop?.reason || 'ATTEMPT_ABORTED'};
const active = activeTools.get(context.toolCall.id);
if (!active) throw new Error('TOOL_START_NOT_PERSISTED');
// Bash has no default timeout, so a long command would reach the system cap
// below and end the whole attempt. Give it a native timeout a minute earlier
// so only that command fails. args is Pi's private copy; the persisted call
// keeps the model's own arguments.
if (context.toolCall.name === 'bash') {
const room = Math.floor(active.call.deadline_monotonic - monotonic()) - 60;
if (room > 0 && !(context.args.timeout > 0 && context.args.timeout <= room)) context.args.timeout = room;
}
// A shorter native bash timeout remains a normal, recoverable tool error.
// This independent system cap ends the attempt and cannot be extended by args.
active.timer = setTimeout(() => expireTool(active.call),

View File

@ -206,6 +206,17 @@ class PiSDKTests(unittest.TestCase):
self.assertTrue(checkpoint['closure'])
self.assertEqual((self.root / 'side-effect.txt').read_text(), 'once')
def test_bash_without_timeout_fails_alone_before_system_cap(self):
self.tool_name, self.tool_args = 'bash', {'command': 'sleep 5; echo late'}
self.config.update(tools=['bash'], ordinary_tool_deadline_s=62)
result = self.run_sdk()
self.assertEqual(result.returncode, 0, result.stderr.decode())
self.assertEqual(json.loads((self.root / '.bench-session.json').read_bytes())['phase'], 'COMPLETED')
messages = self.bodies[1]['messages']
self.assertEqual(messages[-1]['content'], 'Command timed out after 1 seconds')
call = next(m for m in messages if m.get('tool_calls'))['tool_calls'][0]
self.assertEqual(json.loads(call['function']['arguments']), self.tool_args)
def _events(self):
return [json.loads(line) for line in (self.root / 'events.jsonl').read_text().splitlines()]