data-gen-bpu/harness/rollout.py
bench f83fa3948c 网络小故障不再拖垮整条轨迹:原地重发、没发出去的暂停不扣续跑次数、结果不明的请求可以续跑
GLM 请求失败率约 0.3%,但一次失败就让整条轨迹暂停;3 次续跑次数被网络故障用完后,
轨迹只能挂到 24 小时上限(10-14 有 4 条);回复中途断开的请求过了等待期会被自动放弃,
同时整条执行被取消(10-13/14 共 16 条)。

A. 连接失败、域名解析失败(RETRY_UNSENT,确定什么都没发出去)在 broker 里等一会儿原地重发。
   每次重试把等待时间加上一次连接时限计入排队预算 wait_spent,保持
   客户端超时 = 排队预算 + 次数×单次时限 的上界;预算用完才照旧暂停。
B. 续跑次数只在最近一次请求可能已经送达时才扣(看账本最近一次 attempt 是否 sent)。
   连接/解析失败、因为别的请求结果不明而在准入时被挡下的暂停,续跑不扣次数。
   执行记录和账本两处计数同步,调度程序不再按原始次数预先过滤。
C. reconcile 自动放弃时(截止时间再过一个 token 窗口)不再取消执行,而是把这一轮
   标成 REQUEST_OUTCOME_UNKNOWN、可以续跑:重发一轮只花一次请求的 token,并计入续跑次数。
   人工放弃(operator)仍然永久隔离。续跑资格检查不再被已放弃的 attempt 挡住。
都在调度程序进程里运行,下次重启生效。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 15:41:49 +08:00

605 lines
36 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

"""rollout 驱动器:起一个 pi 进程对一个 job 作答一次,然后独立核验。
bench rollout <job_id> --kk 0 --mode e2e --policy teacher
bench agent setup # 从 .env.local 渲染 ~/.pi/agent/models.json
每次 rollout 一个目录、全新上下文、看不到其他 rollout。Agent 没有板卡或平台凭据;
新配方任务只传当前 LLM endpoint 必需的 key,旧 Pi 任务保持原配置读取方式。
"""
from __future__ import annotations
import json
import os
import shutil
import time
from pathlib import Path
from . import paths, db, watchdog
from .registry import gates as gates_cfg
from .stages.s02_baseline import baseline_dir
from .stages.s03_export import onnx_path
from .verify import verify_rollout
def rollout_tools(cfg=None):
cfg = paths.config() if cfg is None else cfg
names = [name for name in cfg['rollouts']['tools'] if name != 'terminal_exec']
names.append('terminal_exec') # Stage-only tasks use its protected wait_handle branch.
return names
def rollout_dir(run_id: str, job_id: str, kk: int) -> Path:
return paths.RUNS / run_id / job_id / "rollouts" / f"{kk:02d}"
def _render(tmpl: str, **kw) -> str:
out = tmpl
for k, v in kw.items():
out = out.replace("{" + k + "}", "" if v is None else str(v))
return out
def _recipe_prompt(tmpl: str, job: dict) -> str:
"""Describe tool inputs, not an exploration order, for a task-local submission."""
before, marker, remainder = tmpl.partition("## 输入和可复用缓存\n")
_, budget, after = remainder.partition("\n预算:")
if not marker or not budget:
raise ValueError("recipe prompt requires the standard input/workflow and budget sections")
policy = json.dumps(job["model_recipe"]["reference"], indent=2, ensure_ascii=False)
body = """## 工作区与提交
你可以复用只读的 `harness/adapters/`,复制到 workspace 修改,也可以自行实现模型接入、修改导出代码或直接修改 ONNX。由你决定探索顺序和重试方法;配方只用于提交这一版实际需要的文件,不是开始工作的前置条件。
任务、数据、验收标准以 job.json 为准。以下官方参考由平台固定,执行参考基线时核对源码和权重摘要,不能用候选代码代替:
```json
""" + policy + """
```
提交清单 `recipe.json` 的 `schema` 为 `bench.recipe/1`,`files` 列出需要冻结的 workspace 相对路径。`--recipe` 和清单内全部路径都以 workspace 为起点,不是相对清单所在子目录。
| 操作 | 实际需要的材料 |
| --- | --- |
| `export` 接收已有图 | `onnx` 指向图,图及全部 external_data 在 files 中;张量路径相对图所在目录。无需先跑基线。 |
| `export` 从代码导出 | `entrypoint` 和该导出使用的文件;使用 build 时再提供实际 `weights`。 |
| `baseline` | 平台固定参考文件和匹配的权重,用独立参考生成 T1 基线。 |
| `compile` / `calibrate` | 已提交的 ONNX、实际可用校准输入,以及 workspace 中的编译配置。 |
| `hbm` / `board` | 对应编译产物;共享工具链、板卡仍由原阶段分配。 |
| `eval` | 对应 HBM、独立参考基线,以及产生完整任务预测所需的前处理/解码。由 08 和 verifier 固定评分。 |
代码可沿用 `build(weights, input_spec)`、`preprocess(image, input_spec)`、可选 `decode(outputs, input_spec)`,也可提供 `export(weights, input_spec, out)`。`input_spec` / `preprocess` 记录候选实际输入约定。当前正式评分支持已有语义分割协议;图去头后可由 decode 恢复完整任务预测。编译成功、图不等价和精度下降都按真实结果记录,不预先当成完整任务通过。
## 按需调用原阶段
每次 `--recipe` 冻结这一版并返回 handle;随后用该 handle 的 `--submission` 继续使用同一版。例如已有图可以先试编译:
```
bench stage run export --recipe recipe.json
# 等待 export:调用 terminal_exec({"wait_handle": "SUBMISSION_HANDLE"})
bench stage run compile --submission SUBMISSION_HANDLE --config compile.yaml
```
这里 SUBMISSION_HANDLE 替换为 export 实际返回的 handle。其他可按需调用的工具:
```
bench stage run fetch --recipe recipe.json
bench stage run baseline --submission SUBMISSION_HANDLE
bench stage run hbm --submission SUBMISSION_HANDLE
bench stage run board --submission SUBMISSION_HANDLE
bench stage run eval --submission SUBMISSION_HANDLE
```
`fetch` 可以核对权重,不是所有提交必经步骤。compile 会准备 04 配置;需要单独检查时用 `bench stage run calibrate --submission SUBMISSION_HANDLE --config compile.yaml`。使用某步结果前等待它完成,读返回的 stdout/stderr/meta;`recipe_outputs` 给出实际产物路径与摘要,不猜“最新文件”。
提交后仍可修改 workspace;再次带 `--recipe recipe.json` 得到新的提交 handle。正式产物只读,需要改图先复制到 workspace。新图不继承旧 HBM 或旧成绩;失败日志仍保留,可以继续试验。最后用自然语言如实说明完成内容、测量范围和未验证事项;任务要求板端精度而缺少对应证据时,不能声称完整通过。
"""
return before + body + "\n预算:" + after
def _recipe_system_prompt(system: str) -> str:
"""Keep the shared boundaries and replace only legacy stage invocation syntax."""
before, marker, remainder = system.partition("## 共享资源与正式验收\n")
_, end, after = remainder.partition("## 结束说明\n")
if not marker or not end:
raise ValueError("recipe system prompt is missing the existing stage section")
section = """## 共享资源与正式验收
按任务提示为当前操作提交必要文件:`bench stage run <stage> --recipe recipe.json` 冻结新版本,`--submission SUBMISSION_HANDLE` 复用那次提交。探索次序由你决定;已有 ONNX 可先 export,再试 compile。编译使用 `--config compile.yaml`,文件位于 workspace。
共享工具链、板卡和正式评分通过原 stage 工具,使用结果前调用 `terminal_exec({"wait_handle": "<handle>"})`,一次等待原阶段完成。日志和 meta 保存真实执行与失败;按返回的产物路径使用结果。固定任务目标、评测数据与评分阈值不可修改,claim 不能代替独立验收。
"""
result = before + section + end + after
return result.replace("现有 Pi legacy 沙箱仍保留 shell 路径限制;工具报限制时按其说明使用明确路径。不要为私有环境安装修改共享工具链。",
"本任务使用私有 HOME;workspace 内可 cd、使用变量并安装私有依赖。共享工具链保持只读。")
def setup_models_json(force: bool = False) -> Path:
"""渲染 ~/.pi/agent/models.json。key 只从 .env.local 来。"""
dst = Path(os.path.expanduser("~/.pi/agent/models.json"))
if dst.exists() and not force:
return dst
cfg = paths.config()["endpoints"]
tmpl = (paths.AGENT / "models.json.tmpl").read_text()
text = _render(tmpl,
TEACHER_BASE_URL=cfg["teacher"]["base_url"],
TEACHER_MODEL=cfg["teacher"]["model"],
TEACHER_API_KEY=paths.secret(cfg["teacher"]["api_key_env"]) or "",
STUDENT_BASE_URL=cfg["student"].get("base_url") or "http://127.0.0.1:8000/v1",
STUDENT_API_KEY=paths.secret(cfg["student"]["api_key_env"]) or "EMPTY",
STUDENT_MODEL=cfg["student"].get("model") or "student")
dst.parent.mkdir(parents=True, exist_ok=True)
dst.write_text(text)
os.chmod(dst, 0o600)
return dst
def prepare(run_id: str, job_id: str, kk: int, mode: str, policy: str, dry_run: bool = False, runner: str = "pi") -> tuple[Path, dict, str]:
"""dry_run=True 只在内存里渲染,不建目录、不写 rollout.json、不动 DB(踩过:dry-run 把 3 个 kk 占成了空壳 RUNNING)。"""
if mode == "repair":
raise ValueError("Oracle repair collection is retired; existing rows and artifacts remain historical")
from .runners import get as get_runner
get_runner(runner).check(policy, execute=False)
jd = paths.RUNS / run_id / job_id
job = json.loads((jd / "job.json").read_text())
with (db.readonly() if dry_run else db.connect()) as con:
db.assert_run_open(con, run_id)
rd = rollout_dir(run_id, job_id, kk)
if job.get("model_recipe") and not paths.config()["rollouts"].get("jail", True):
raise ValueError("task recipes require the outer worker and mount jail")
if (rd / "rollout.json").exists():
raise SystemExit(f"rollout 已存在(append-only):{rd}。想再答一次就用下一个 kk。")
if not dry_run:
for d in ("workspace", "agent", "stages", "handles", "workspace/.pi"):
(rd / d).mkdir(parents=True, exist_ok=True)
# 输入缓存可复用;缺少导出或基线时由 Agent 在自己的轨迹中提交。
onnx = None if job.get("model_recipe") else onnx_path(job)
bdir = None if job.get("model_recipe") else baseline_dir(job)
cfg = paths.config()
ep = cfg["endpoints"][policy]
if runner == "pi":
settings = _render((paths.AGENT / "settings.json.tmpl").read_text(),
PROJECT_ROOT=str(paths.PROJECT_ROOT), BENCH_ROOT=str(paths.BENCH_ROOT), MODEL=ep["model"])
sj = json.loads(settings)
sj.pop("_comment", None)
sj["defaultProvider"] = "dr-gateway" if policy == "teacher" else "student-vllm"
sj["tools"] = rollout_tools(cfg)
if not dry_run:
(rd / "workspace" / ".pi" / "settings.json").write_text(json.dumps(sj, indent=2, ensure_ascii=False))
if job.get("model_recipe") or cfg["rollouts"].get("private_home", False):
# Only this task's model configuration; never copy the host HOME.
model_config = json.loads(_render((paths.AGENT / "models.json.tmpl").read_text(),
TEACHER_BASE_URL=ep["base_url"], TEACHER_MODEL=ep["model"],
TEACHER_API_KEY="BENCH_RUNNER_API_KEY", STUDENT_BASE_URL=ep["base_url"],
STUDENT_MODEL=ep["model"], STUDENT_API_KEY="BENCH_RUNNER_API_KEY"))
provider = sj["defaultProvider"]
model_dir = rd / "workspace/.agent-home/.pi/agent"
model_dir.mkdir(parents=True)
(model_dir / "models.json").write_text(json.dumps({
"providers": {provider: model_config["providers"][provider]}}))
# prompt
tname = job.get("prompt") or (f"{mode}.md")
tmpl = (paths.AGENT / "prompts" / tname).read_text()
g = job["gates"]
prompt = _render(
tmpl, job_id=job_id, run_id=run_id, model=job["model"], task=job["task"], march=job["march"],
backend=job["backend"], qcfg=job["qcfg"], job_json=str(jd / "job.json"), data_dir=str(jd / "data"),
input_spec=job["model_spec"]["input_spec"], onnx=str(onnx) if onnx else "尚未生成;由 fetch/export 阶段返回路径", baseline_dir=str(bdir),
calib_dir=job["data"]["calib"]["path"], calib_n=job["data"]["calib"]["n"],
t1_tar=job["data"]["t1"]["path"], t1_n=job["data"]["t1"]["n"],
g2_threshold=g["G2"].get("threshold"), g3_metric=g["G3"].get("metric"), g3_tol=g["G3"].get("tol"),
g3_kind=g["G3"].get("kind"), max_turns=cfg["rollouts"]["max_turns"], max_tokens=cfg["rollouts"]["max_tokens"],
)
if job.get("model_recipe"):
# Insert the fixed policy after placeholder expansion so literal braces
# inside its file names/metadata are never interpreted as template keys.
prompt = _recipe_prompt(prompt, job)
if dry_run:
return rd, job, prompt
(rd / "agent" / "prompt.md").write_text(prompt)
# 先保存追加提示词;SDK 启动后另存实际完整 system/tools 上下文。
# legacy 只有追加快照,严格导出会保留但拒绝其 SFT 准入。
if job.get("model_recipe"):
system = _recipe_system_prompt((paths.AGENT / "SYSTEM.md").read_text())
(rd / "agent" / "system.md").write_text(system)
else:
shutil.copy2(paths.AGENT / "SYSTEM.md", rd / "agent" / "system.md")
shutil.copy2(rd / "agent/system.md", rd / "agent/append-system.md")
(rd / "rollout.json").write_text(json.dumps({"schema": "bench.rollout/1", "run_id": run_id, "job_id": job_id,
"kk": kk, "mode": mode, "policy": policy, "runner": runner, "runner_version": get_runner(runner).VERSION, "model": ep["model"],
"agent_config_version": agent_config_version() if runner == "pi" else "1",
"max_turns": cfg["rollouts"]["max_turns"], "max_tokens": cfg["rollouts"]["max_tokens"],
"state": "RUNNING", "started_at": time.time(),
**({"natural_agent_version": "natural-agent-v1"} if cfg.get("terminal", {}).get("enabled") else {}),
**({"jail_policy": "private-home"} if job.get("model_recipe") or cfg["rollouts"].get("private_home", False) else {})}, indent=2))
# 工具定义快照:pi 不把工具 schema 写进 events.jsonl,导出 SFT 样本的 tools 字段要靠它。
# 从 pi 包的 create*Tool 工厂现取(权威),失败不致命——导出时还能按 cmd.json 的白名单补跑一次。
try:
if runner != "pi":
raise ValueError("runner owns its tool definitions")
import subprocess as _sp
node = paths.expand(cfg["paths"].get("node_bin") or "node")
td = _sp.run([str(node), str(paths.AGENT / "dump_tools.js"), ",".join(rollout_tools(cfg))],
capture_output=True, text=True, timeout=60, env={**os.environ, "BENCH_ROLLOUT_DIR": str(rd)})
if td.returncode == 0 and td.stdout.strip():
(rd / "agent" / "tools.json").write_text(td.stdout)
except Exception:
pass
with db.connect() as con:
db.rollout_upsert(con, run_id, job_id, kk, mode, policy, runner=runner)
db.rollout_set(con, run_id, job_id, kk, state="RUNNING", started_at=time.time())
if runner != "pi":
get_runner(runner).prepare(rd, policy)
return rd, job, prompt
def agent_config_version() -> str:
"""agent/VERSION:提示词 / 扩展 / 预算规则的口径版本。改了 SYSTEM.md、prompts、extensions 就 +1;导出可按版本筛。"""
p = paths.AGENT / "VERSION"
return p.read_text().strip() if p.exists() else "1"
def agent_env(rd: Path, mode: str = "e2e") -> dict:
"""路径与预算;新配方 Pi 仅额外传当前模型调用必需的 key。"""
cfg = paths.config()
path_parts = [str(paths.BENCH_ROOT / "bin")]
if cfg["paths"].get("node_bin"):
path_parts.append(str(Path(paths.expand(cfg["paths"]["node_bin"])).parent))
path_parts.append(os.environ.get("PATH", "/usr/bin:/bin"))
# path-guard 的白名单按"项目树内路径"判;state/ 是指向 /home 的符号链接、work_root 是本地盘,
# 用真实路径就绕开了 → 把它们的 realpath 作为额外拒绝项传给 guard;本 rollout 自己的编译工作目录放行。
wr = paths.expand(cfg["paths"]["work_root"])
rel = rd.relative_to(paths.RUNS) # <run>/<job>/rollouts/<kk>
deny = [str(Path(paths.STATE).resolve()), str(wr)]
allow = [str(wr / rel.parts[0] / rel.parts[1] / f"r{int(rel.parts[3]):02d}")]
env = {
"BENCH_GUARD_DENY": ":".join(deny),
"BENCH_GUARD_ALLOW": ":".join(allow),
"PATH": ":".join(path_parts),
"HOME": os.environ.get("HOME", "/tmp"),
"LANG": "C.UTF-8",
"TERM": "dumb",
"BENCH_ROOT": str(paths.BENCH_ROOT),
"BENCH_CONFIG": str(paths.CONFIG_PATH),
"BENCH_STATE_DIR": str(paths.STATE),
"BENCH_HOST_HOME": os.environ.get("HOME", "/tmp"),
"BENCH_OUTER_WORKER": "1",
"BENCH_ROLLOUT_DIR": str(rd),
"BENCH_MODE": mode, # path-guard:e2e 看不到 oracle/,repair 可以
"BENCH_MAX_TURNS": str(cfg["rollouts"]["max_turns"]),
"BENCH_MAX_TOKENS": str(cfg["rollouts"]["max_tokens"]),
"PYTHONPATH": str(paths.BENCH_ROOT),
}
snapshot = rd / "rollout.json"
recorded = json.loads(snapshot.read_text()) if snapshot.exists() else {}
if recorded.get("runner", "pi") == "pi" and recorded.get("jail_policy") == "private-home":
home = rd / "workspace/.agent-home"
ep = cfg["endpoints"][recorded["policy"]]
env.update(HOME=str(home), PI_CODING_AGENT_DIR=str(home / ".pi/agent"),
XDG_CONFIG_HOME=str(home / "config"), XDG_CACHE_HOME=str(home / "cache"),
XDG_DATA_HOME=str(home / "data"), XDG_STATE_HOME=str(home / "state"),
BENCH_RUNNER_API_KEY=paths.secret(ep["api_key_env"]) or "EMPTY")
return env
def jail_prefix(rd: Path, mode: str) -> list[str]:
"""把 pi 关进 mount namespace:oracle/(e2e)、兄弟 rollout、~/.pi、.env.local 直接从文件系统里摘掉。
path-guard 是解析 shell 字符串的白名单,v1→v3 每一轮都被新写法穿透(换行分隔的 cd、$IFS、
$(…)、$VAR/ 拼接、workspace 里造软链)。这一层不解析任何东西,绕法再多也看不见那些路径。
unshare 不可用时拒绝启动,不降低隔离要求。
"""
if not paths.config()["rollouts"].get("jail", True):
return []
js = paths.BENCH_ROOT / "scripts" / "agent_jail.sh"
if not js.exists() or not shutil.which("unshare"):
raise RuntimeError("agent_jail 不可用:拒绝降级隔离")
return [str(js), str(rd), mode, "--"]
def pi_command(rd: Path, prompt: str, mode: str = "e2e") -> list[str]:
cfg = paths.config()
if cfg.get("collection_v1"):
node = str(paths.expand(cfg["paths"].get("node_bin") or "node"))
return jail_prefix(rd, mode) + [node, str(paths.AGENT / "pi_session.js"), str(rd / "agent/pi-session.json")]
pi = str(paths.expand(cfg["paths"].get("pi_bin") or "pi")) if "/" in (cfg["paths"].get("pi_bin") or "") else (cfg["paths"].get("pi_bin") or "pi")
# 实测 pi-coding-agent 0.73.1(pi --help):--mode json 单独就是非交互 JSON 事件流;没有 -a 这个参数;
# --append-system-prompt 接受文本或文件路径;--tools 是逗号分隔白名单;extensions/skills 由 workspace/.pi/settings.json 发现。
return jail_prefix(rd, mode) + [
pi, "--mode", "json", "--session-dir", str(rd / "workspace" / ".pi-sessions") if paths.config()["rollouts"].get("jail", True) else str(rd / "agent"),
"--append-system-prompt", str(rd / "agent" / "system.md"), # 用 rollout 内的快照,与导出补回的 system 消息逐字一致
"--tools", ",".join(rollout_tools(cfg)),
prompt]
def run_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str, dry_run: bool = False,
*, _claimed: bool = False, runner: str = "pi") -> dict:
from .runners import get as get_runner
# The collector checked quota right before claiming; a second 0.5 s ledger read here turned busy
# moments into UNVERIFIED rollouts that never ran (10-13: 59).
get_runner(runner).check(policy, execute=not dry_run and not _claimed)
if not dry_run and runner == "pi":
from .collection_runtime import config
value, _ = config()
if policy not in value['credential_groups']:
raise ValueError('QUOTA_CONFIG_INCOMPLETE: ' + policy)
if not _claimed:
from .recovery import reconcile_run
if not reconcile_run(run_id)['ready']:
raise ValueError('EXECUTION_RECONCILIATION_REQUIRED')
if mode == "repair":
raise ValueError("Oracle repair collection is retired; existing rows and artifacts remain historical")
with (db.readonly() if dry_run else db.connect()) as con:
db.assert_run_open(con, run_id)
run = db.get_run(con, run_id)
row = con.execute("SELECT * FROM rollouts WHERE run_id=? AND job_id=? AND kk=?", (run_id, job_id, kk)).fetchone()
if run and json.loads(run["manifest"]).get("continuation"):
if not row or (row["mode"], row["policy"]) != (mode, policy):
raise ValueError("rollout is not in the continuation queue")
if row and db.rollout_runner(con, run_id, job_id, kk) != runner:
raise ValueError("rollout runner differs from the queued task")
if row and (row["mode"], row["policy"]) != (mode, policy):
raise ValueError("rollout mode/policy differs from the queued task")
if not dry_run:
if _claimed:
if not row or row["state"] != "RUNNING":
raise ValueError("collector has not claimed this rollout")
else:
if not row:
db.rollout_upsert(con, run_id, job_id, kk, mode, policy, runner=runner)
con.commit()
if not db.claim_rollout(con, run_id, job_id, kk):
raise ValueError("rollout already claimed or finished")
try:
return _execute_rollout(run_id, job_id, kk, mode, policy, dry_run, runner=runner)
except (SystemExit, Exception):
if not dry_run:
from .execution import Execution, owned_processes
rd = rollout_dir(run_id, job_id, kk)
if (rd / 'execution.json').exists():
execution = Execution(rd).attach()
snapshot = execution.snapshot()
if runner == 'pi':
from .collection_runtime import broker
broker().cancel(snapshot['execution_id'])
if snapshot['execution_state'] not in ('TERMINAL', 'PAUSED_INFRA'):
clean = owned_processes(snapshot) == []
execution.finish(None, protocol_complete=False, stop_reason='INFRA_EXECUTION_EXCEPTION',
cleanup_state='VERIFIED' if clean else 'QUARANTINED')
if not dry_run and not _claimed:
with db.connect() as con:
db.rollout_set(con, run_id, job_id, kk, state="UNVERIFIED", finished_at=time.time())
raise
def settle_execution(run_id, job_id, kk, execution, reason, *, cancel=False, timeout_s=None):
"""Close control-plane stops and retry their registration without rerunning work."""
from .execution import atomic_json
from .collection_runtime import broker
snapshot = execution.snapshot()
if snapshot['execution_state'] != 'TERMINAL':
if cancel:
snapshot = execution.cancel(reason)
if snapshot['execution_state'] != 'TERMINAL':
try:
broker().cancel(execution.snapshot()['execution_id'])
except (OSError, ValueError):
# Durable cancellation still fences new work; never refund an unknown request.
pass
execution.cleanup()
state = execution.snapshot()
snapshot = execution.finish(None, protocol_complete=False, stop_reason=reason,
cleanup_state=state['cleanup_state'])
if snapshot['execution_outcome'] == 'COMPLETED':
return {k: v for k, v in snapshot.items() if k != 'token'}
rd = rollout_dir(run_id, job_id, kk)
metadata = json.loads((rd / 'rollout.json').read_text())
if rd.resolve() != execution.rd.resolve():
raise ValueError('rollout settlement identity mismatch')
if metadata.get('state') in ('RUNNING', 'VERIFYING', 'PAUSED_INFRA'):
metadata.update(state=snapshot['execution_outcome'], execution_outcome=snapshot['execution_outcome'],
stop_reason=snapshot['stop_reason'], cleanup_state=snapshot['cleanup_state'],
finished_at=snapshot['finished_at'], collection_hold=True)
atomic_json(rd / 'rollout.json', metadata)
with db.connect(**({'timeout_s': timeout_s} if timeout_s is not None else {})) as con:
# A verified task verdict is not replaced by a later control-plane retry.
con.execute("UPDATE rollouts SET state=?,finished_at=? WHERE run_id=? AND job_id=? AND kk=? "
"AND state IN ('RUNNING','VERIFYING','PAUSED_INFRA')",
(snapshot['execution_outcome'], snapshot['finished_at'], run_id, job_id, kk))
return {k: v for k, v in snapshot.items() if k != 'token'}
def resume_rollout(run_id: str, job_id: str, kk: int) -> dict:
"""Resume one protected checkpoint, with the original attempt and budgets."""
from .execution import Execution
from . import collection_runtime as runtime
rd = rollout_dir(run_id, job_id, kk)
metadata = json.loads((rd / 'rollout.json').read_text())
if metadata.get('runner', 'pi') != 'pi' or metadata.get('state') != 'PAUSED_INFRA':
raise ValueError('RESUME_NOT_ELIGIBLE')
execution = Execution(rd).attach()
state = execution.snapshot()
runtime.resume_policy(state)
checkpoint = state.get('checkpoint')
if not checkpoint:
raise ValueError('RESUME_CHECKPOINT_MISSING')
readiness = runtime.broker().resume_readiness(state['execution_id'])
if not readiness['eligible']:
raise ValueError('RESUME_NOT_READY: ' + str(readiness.get('reason')))
execution.resume(counted=readiness.get('counted', True))
try:
runtime.broker().resume(state['execution_id'], checkpoint['generation_id'],
checkpoint_complete=checkpoint.get('resumable', False), side_effects_known=True)
except Exception:
execution.pause(state.get('stop_reason') or 'QUOTA_WAIT_BUDGET_EXHAUSTED', checkpoint)
raise
with db.connect() as con:
changed = con.execute("UPDATE rollouts SET state='RUNNING' WHERE run_id=? AND job_id=? AND kk=? AND state='PAUSED_INFRA'",
(run_id, job_id, kk)).rowcount
if changed != 1:
execution.pause('RESUME_DB_CONFLICT', checkpoint)
raise ValueError('RESUME_DB_CONFLICT')
metadata.update(state='RUNNING')
(rd / 'rollout.json').write_text(json.dumps(metadata, indent=2))
try:
return _execute_rollout(run_id, job_id, kk, metadata['mode'], metadata['policy'], False, 'pi', resume=True)
except Exception:
# No new attempt and no implicit replay after an uncertain resumed run.
runtime.broker().cancel(state['execution_id'])
execution.quarantine('RESUME_EXECUTION_UNCONFIRMED')
with db.connect() as con:
db.rollout_set(con, run_id, job_id, kk, state='PAUSED_INFRA')
raise
def _execute_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str, dry_run: bool, runner: str = "pi", *, resume=False) -> dict:
from .runners import get as get_runner
adapter = get_runner(runner)
if resume:
rd = rollout_dir(run_id, job_id, kk)
job = json.loads((rd.parent.parent / 'job.json').read_text())
prompt = (rd / 'agent/prompt.md').read_text()
else:
rd, job, prompt = prepare(run_id, job_id, kk, mode, policy, dry_run=dry_run, runner=runner)
if not dry_run and runner == "pi" and not job.get("model_recipe") and not paths.config()["rollouts"].get("private_home", False):
setup_models_json()
cmd = adapter.command(rd, prompt, mode)
if not dry_run:
(rd / "agent" / "cmd.json").write_text(json.dumps({"cmd": cmd, "env_keys": sorted(adapter.environment(rd, mode))}, indent=2, ensure_ascii=False))
if dry_run:
print(f"[dry-run] rollout 已准备:{rd}\n cwd={rd / 'workspace'}\n cmd={' '.join(cmd[:8])} …")
return {"rollout_dir": str(rd), "dry_run": True}
cfg = paths.config()["rollouts"]
from contextlib import nullcontext
from .stageworker import Worker, config_bytes
(paths.expand(paths.config()['paths']['work_root']) / run_id / job_id / f'r{kk:02d}').mkdir(parents=True, exist_ok=True)
jailed = bool(jail_prefix(rd, mode))
execution, scope = None, None
env = adapter.environment(rd, mode)
if runner == 'pi':
from .execution import Execution
from . import collection_runtime as runtime
runtime_config, _ = runtime.config()
execution = Execution(rd)
if resume:
execution.attach()
else:
execution.start(runtime_config['lifecycle'], activity='QUEUED')
scope = runtime.configure_pi(rd, execution, policy, resume=resume)
tool_deadlines = runtime.ToolDeadlines(rd, execution)
env['BENCH_RUNNER_API_KEY'] = scope.api_key
used = runtime.budget_used(rd)
env.update(BENCH_INITIAL_TURNS=str(used['turns']), BENCH_INITIAL_TOKENS=str(used['tokens']))
def stop_for_budget():
if runner == "pi":
tool_reason = tool_deadlines.poll()
if tool_reason:
return tool_reason
state = execution.snapshot()
if state.get('cancel_requested'):
runtime.broker().cancel(state['execution_id'])
elif state['execution_state'] == 'ACTIVE':
quota_state = runtime.broker().execution_status(state['execution_id']) or {}
generations = quota_state.get('generations', [])
if generations:
latest = generations[-1]
activity = 'WAITING_QUOTA' if latest['state'] in ('QUEUED', 'RESUMABLE') else 'WAITING_LLM'
if (state['activity'] != 'RUNNING_TOOL' and latest['state'] not in ('COMPLETE', 'PAUSED', 'CANCELLED')
and state['activity'] != activity):
execution.activity(activity, generation_id=latest['generation_id'], reason=latest.get('reason'))
return None
cost = adapter.cost(rd)
return "runner budget exhausted" if (cost["turns"] >= cfg["max_turns"] or cost["tokens"] >= cfg["max_tokens"]) else None
from . import scheduler
finalization_error = None
checkpoint = {}
with execution.scope() if execution else nullcontext():
with scheduler.acquire('agent'):
if execution:
execution.activity('WAITING_LLM')
with Worker(rd, execution=execution) if jailed else nullcontext():
r = watchdog.run(cmd, cwd=rd / "workspace", stdout=rd / "agent" / "events.jsonl", stderr=rd / "agent" / f"{runner}.stderr.log",
env=env, stall_s=cfg["hard_ceiling_s"], poll_s=5 if runner == "pi" else 1,
hard_ceiling_s=cfg["hard_ceiling_s"], stop_check=stop_for_budget,
execution=execution, role='agent',
on_finish=(lambda _: tool_deadlines.poll()) if execution else None)
# Freeze the Agent's first exit before draining its accepted stages.
try:
adapter.finish(rd)
if execution:
runtime.capture_context(rd)
checkpoint = runtime.freeze_checkpoint(rd, execution)
except Exception as exc:
finalization_error = str(exc)
(rd / "agent/session-error.json").write_text(json.dumps({"error": type(exc).__name__, "detail": str(exc)}))
if execution:
quota = runtime.broker().execution_status(execution.snapshot()['execution_id']) or {}
generations = quota.get('generations', [])
used = runtime.budget_used(rd)
limits = json.loads((rd / 'rollout.json').read_text())
first_reason = ('SESSION_FINALIZATION_ERROR' if finalization_error else
((generations[-1].get('reason') if generations else None) or 'INCOMPLETE_GENERATION') if r.exit_code == 76 else
'BUDGET_EXHAUSTED' if r.exit_code == 75 or used['turns'] >= limits['max_turns'] or used['tokens'] >= limits['max_tokens'] else
r.stalled_reason)
with execution.mutation() as state:
execution._save(state, 'AGENT_EXITED', exit_code=r.exit_code,
checkpoint_phase=checkpoint.get('phase'), reason=first_reason,
turns=used['turns'], tokens=used['tokens'])
if execution and not finalization_error and r.exit_code == 76 and not execution.snapshot().get('cancel_requested'):
quota_state = runtime.broker().execution_status(execution.snapshot()['execution_id'])
pending = (quota_state or {}).get('generations', [])
reason = pending[-1].get('reason') if pending else 'INCOMPLETE_GENERATION'
if checkpoint.get('resumable') and checkpoint.get('phase') == 'PAUSED_INFRA' and checkpoint.get('closure') and pending and pending[-1]['state'] == 'PAUSED':
execution.pause(reason, checkpoint)
rj = json.loads((rd / 'rollout.json').read_text())
rj.update(state='PAUSED_INFRA', stop_reason=reason, exit_code=r.exit_code)
(rd / 'rollout.json').write_text(json.dumps(rj, indent=2))
with db.connect() as con:
db.rollout_set(con, run_id, job_id, kk, state='PAUSED_INFRA')
return {'rollout_dir': str(rd), 'terminal': 'PAUSED_INFRA', 'reason': reason}
if jailed:
# These remain untrusted claims/logs, never verdict or authoritative budgets.
for src, dst in [('claim.json', 'claim.json'), ('guard.log', 'agent/guard.log')]:
try:
(rd / dst).write_bytes(config_bytes(rd / 'workspace', str(rd / 'workspace' / src)))
except (OSError, ValueError):
pass
rj = json.loads((rd / "rollout.json").read_text())
from .verify import _cost
cost = runtime.budget_used(rd) if execution else _cost(rd)
budget_exhausted = cost["turns"] >= rj["max_turns"] or cost["tokens"] >= rj["max_tokens"]
if budget_exhausted:
rj["state"] = "TIMEOUT"
rj["timeout_reason"] = "budget (events verified)"
if r.state == "STALLED" or r.exit_code == 75:
rj["state"] = "TIMEOUT"; rj.setdefault("timeout_reason", r.stalled_reason or "budget")
elif rj.get("state") != "TIMEOUT":
rj["state"] = "ERROR" if finalization_error or r.exit_code != 0 else "DONE"
if finalization_error:
rj["session_error"] = finalization_error
rj.update({"exit_code": r.exit_code, "wall_s": round(r.duration_s, 1), "finished_at": time.time()})
if execution:
protocol_complete = not finalization_error and checkpoint.get('phase') == 'COMPLETED'
stop_reason = execution.snapshot().get('stop_reason') or (
'BUDGET_EXHAUSTED' if budget_exhausted or r.exit_code == 75 else r.stalled_reason)
if r.exit_code == 76 and not stop_reason:
quota_state = runtime.broker().execution_status(execution.snapshot()['execution_id']) or {}
pending = quota_state.get('generations', [])
stop_reason = 'INFRA_' + ((pending[-1].get('reason') if pending else None) or 'INCOMPLETE_GENERATION')
# A resumable quota pause returned above. Every terminal path must fence
# the broker BEFORE publishing a finished rollout, including SDK errors.
runtime.broker().cancel(execution.snapshot()['execution_id'])
finished = execution.finish(r.exit_code, protocol_complete=protocol_complete, stop_reason=stop_reason,
cleanup_state=r.cleanup_state)
rj['execution_outcome'] = finished['execution_outcome']
rj['cleanup_state'] = finished['cleanup_state']
if finished['stop_reason'] == 'TOOL_DEADLINE_EXCEEDED':
rj.update(state='DEADLINE_EXCEEDED', stop_reason=finished['stop_reason'], collection_hold=True)
if finished['execution_outcome'] != 'COMPLETED' and rj['state'] == 'DONE':
rj['state'] = 'ERROR'
(rd / "rollout.json").write_text(json.dumps(rj, indent=2))
with db.connect() as con:
db.rollout_set(con, run_id, job_id, kk, state="VERIFYING")
v = verify_rollout(rd)
terminal = (v.get('execution_control') or {}).get('db_state', v['terminal'])
return {"rollout_dir": str(rd), "terminal": terminal, "claim_consistent": v["claim_consistent"], "review": v["review"]}