fix: let a long bash command time out alone instead of ending the attempt
Bash has no default timeout, so any command running past the 900 s system cap ended the whole trajectory (81 of 590 finished rollouts). Give bash a native timeout one minute before the cap; only that command fails and the agent continues. The persisted tool call keeps the model's own arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
ea0335bc24
commit
5617f57f08
@ -7,6 +7,7 @@
|
||||
- 本 rollout 的日志、阶段产物、job.json 与任务数据可读;正式提交和阶段证据只读。其他 rollout、状态库、凭据与 oracle 不属于本任务。
|
||||
- 公共可复用工具位于 `$BENCH_ROOT/harness/adapters/`;`BENCH_ROOT` 已由运行环境提供,可读、可复制修改自己的副本,不是新增模型的必经入口。评判代码仍不作为 Agent 的实现依赖。
|
||||
- 查找文件先用 `rg --files`,搜索内容用 `rg`;限定在 workspace、任务返回的日志目录或上述公共 adapters 目录。不要执行 `find /` 或遍历整台服务器来寻找代码;路径不可访问时依据工具报错处理。
|
||||
- 单条 bash 命令最长约 14 分钟,到时只结束这条命令(返回 `Command timed out`),轨迹继续。更久的工作放后台运行(`nohup … > log 2>&1 &`),再分次查看日志;不要用 `sleep` 一次等待超过 10 分钟。
|
||||
- 现有 Pi legacy 沙箱仍保留 shell 路径限制;工具报限制时按其说明使用明确路径。不要为私有环境安装修改共享工具链。
|
||||
- 目标板卡、数据、指标与阈值以 job.json 为准。编译失败、图变化、精度下降都如实保留;只有平台的实际测量能作为验收证据。
|
||||
- 阶段在后台运行;提交返回 handle 后调用 `terminal_exec({"wait_handle": "<handle>"})`,一次等待原阶段完成或原期限终止,不消耗模型轮数反复轮询。不要为等待重提交阶段;诊断时只查看该 handle 返回的日志与对应阶段,不遍历所有历史产物。旧 `bench stage wait <handle> --poll 60` 仍可用于一次短查询,返回 2 仅表示仍在排队或运行。
|
||||
|
||||
@ -1 +1 @@
|
||||
14
|
||||
15
|
||||
|
||||
@ -170,6 +170,14 @@ async function main(configFile) {
|
||||
if (toolStop || signal?.aborted) return {block: true, reason: toolStop?.reason || 'ATTEMPT_ABORTED'};
|
||||
const active = activeTools.get(context.toolCall.id);
|
||||
if (!active) throw new Error('TOOL_START_NOT_PERSISTED');
|
||||
// Bash has no default timeout, so a long command would reach the system cap
|
||||
// below and end the whole attempt. Give it a native timeout a minute earlier
|
||||
// so only that command fails. args is Pi's private copy; the persisted call
|
||||
// keeps the model's own arguments.
|
||||
if (context.toolCall.name === 'bash') {
|
||||
const room = Math.floor(active.call.deadline_monotonic - monotonic()) - 60;
|
||||
if (room > 0 && !(context.args.timeout > 0 && context.args.timeout <= room)) context.args.timeout = room;
|
||||
}
|
||||
// A shorter native bash timeout remains a normal, recoverable tool error.
|
||||
// This independent system cap ends the attempt and cannot be extended by args.
|
||||
active.timer = setTimeout(() => expireTool(active.call),
|
||||
|
||||
@ -206,6 +206,17 @@ class PiSDKTests(unittest.TestCase):
|
||||
self.assertTrue(checkpoint['closure'])
|
||||
self.assertEqual((self.root / 'side-effect.txt').read_text(), 'once')
|
||||
|
||||
def test_bash_without_timeout_fails_alone_before_system_cap(self):
|
||||
self.tool_name, self.tool_args = 'bash', {'command': 'sleep 5; echo late'}
|
||||
self.config.update(tools=['bash'], ordinary_tool_deadline_s=62)
|
||||
result = self.run_sdk()
|
||||
self.assertEqual(result.returncode, 0, result.stderr.decode())
|
||||
self.assertEqual(json.loads((self.root / '.bench-session.json').read_bytes())['phase'], 'COMPLETED')
|
||||
messages = self.bodies[1]['messages']
|
||||
self.assertEqual(messages[-1]['content'], 'Command timed out after 1 seconds')
|
||||
call = next(m for m in messages if m.get('tool_calls'))['tool_calls'][0]
|
||||
self.assertEqual(json.loads(call['function']['arguments']), self.tool_args)
|
||||
|
||||
def _events(self):
|
||||
return [json.loads(line) for line in (self.root / 'events.jsonl').read_text().splitlines()]
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user