The agent could read and copy harness/adapters (model loading, preprocessing, export, decoding), and every bench dev container carried the whole harness, registry, scripts and export trees, evaluator and gates included. That did the model-specific half of the deployment for it. - guard and private-home jail no longer expose harness/adapters; bench dev packs only container_site - the evaluator still decodes outputs with the platform's own code, so agent/io-contracts states what the graph's inputs and outputs mean (9 families, 71 models so far); the prompt appends the task's contract - SYSTEM.md: says where official sources, weights and environments are; the compile.yaml template is replaced by the facts that differ from the docs - e2e prompt: whole-class int16/float fallback and >1.5x board latency do not pass - agent/VERSION 24 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
635 lines
38 KiB
Python
635 lines
38 KiB
Python
"""rollout 驱动器:起一个 pi 进程对一个 job 作答一次,然后独立核验。
|
||
|
||
bench rollout <job_id> --kk 0 --mode e2e --policy teacher
|
||
bench agent setup # 从 .env.local 渲染 ~/.pi/agent/models.json
|
||
|
||
每次 rollout 一个目录、全新上下文、看不到其他 rollout。Agent 没有板卡或平台凭据;
|
||
新配方任务只传当前 LLM endpoint 必需的 key,旧 Pi 任务保持原配置读取方式。
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import json
|
||
import os
|
||
import shutil
|
||
import time
|
||
from pathlib import Path
|
||
|
||
from . import paths, db, watchdog
|
||
from .registry import gates as gates_cfg
|
||
from .stages.s02_baseline import baseline_dir
|
||
from .stages.s03_export import onnx_path
|
||
from .verify import verify_rollout
|
||
|
||
|
||
def rollout_tools(cfg=None):
|
||
cfg = paths.config() if cfg is None else cfg
|
||
names = [name for name in cfg['rollouts']['tools'] if name != 'terminal_exec']
|
||
names.append('terminal_exec') # Stage-only tasks use its protected wait_handle branch.
|
||
return names
|
||
|
||
|
||
def rollout_dir(run_id: str, job_id: str, kk: int) -> Path:
|
||
return paths.RUNS / run_id / job_id / "rollouts" / f"{kk:02d}"
|
||
|
||
|
||
def _render(tmpl: str, **kw) -> str:
|
||
out = tmpl
|
||
for k, v in kw.items():
|
||
out = out.replace("{" + k + "}", "" if v is None else str(v))
|
||
return out
|
||
|
||
|
||
def _io_contract(model: str) -> str | None:
|
||
"""The evaluator feeds the graph and decodes its outputs with the platform's own code (harness/adapters), which
|
||
the agent no longer sees; agent/io-contracts states what those inputs and outputs mean, per model family."""
|
||
index = json.loads((paths.AGENT / "io-contracts" / "index.json").read_text())
|
||
name = index.get(model.replace(".", "_").replace("-", "_"))
|
||
return (paths.AGENT / "io-contracts" / name).read_text().strip() if name else None
|
||
|
||
|
||
def _recipe_prompt(tmpl: str, job: dict) -> str:
|
||
"""Describe tool inputs, not an exploration order, for a task-local submission."""
|
||
before, marker, remainder = tmpl.partition("## 输入和可复用缓存\n")
|
||
_, budget, after = remainder.partition("\n预算:")
|
||
if not marker or not budget:
|
||
raise ValueError("recipe prompt requires the standard input/workflow and budget sections")
|
||
policy = json.dumps(job["model_recipe"]["reference"], indent=2, ensure_ascii=False)
|
||
body = """## 工作区与提交
|
||
你可以自行实现模型接入、修改导出代码或直接修改 ONNX。由你决定探索顺序和重试方法;配方只用于提交这一版实际需要的文件,不是开始工作的前置条件。
|
||
|
||
任务、数据、验收标准以 job.json 为准。以下官方参考由平台固定,执行参考基线时核对源码和权重摘要,不能用候选代码代替:
|
||
```json
|
||
""" + policy + """
|
||
```
|
||
|
||
提交清单 `recipe.json` 的 `schema` 为 `bench.recipe/1`,`files` 列出需要冻结的 workspace 相对路径。`--recipe` 和清单内全部路径都以 workspace 为起点,不是相对清单所在子目录。
|
||
|
||
| 操作 | 实际需要的材料 |
|
||
| --- | --- |
|
||
| `export` 接收已有图 | `onnx` 指向图,图及全部 external_data 在 files 中;张量路径相对图所在目录。无需先跑基线。 |
|
||
| `export` 从代码导出 | `entrypoint` 和该导出使用的文件;使用 build 时再提供实际 `weights`。 |
|
||
| `baseline` | 平台固定参考文件和匹配的权重,用独立参考生成 T1 基线。 |
|
||
| `compile` / `calibrate` | 已提交的 ONNX、实际可用校准输入,以及 workspace 中的编译配置。 |
|
||
| `hbm` / `board` | 对应编译产物;共享工具链、板卡仍由原阶段分配。 |
|
||
| `eval` | 对应 HBM、独立参考基线,以及产生完整任务预测所需的前处理/解码。由 08 和 verifier 固定评分。 |
|
||
|
||
代码可沿用 `build(weights, input_spec)`、`preprocess(image, input_spec)`、可选 `decode(outputs, input_spec)`,也可提供 `export(weights, input_spec, out)`。`input_spec` / `preprocess` 记录候选实际输入约定。当前正式评分支持已有语义分割协议;图去头后可由 decode 恢复完整任务预测。编译成功、图不等价和精度下降都按真实结果记录,不预先当成完整任务通过。
|
||
|
||
## 按需调用原阶段
|
||
每次 `--recipe` 冻结这一版并返回 handle;随后用该 handle 的 `--submission` 继续使用同一版。例如已有图可以先试编译:
|
||
|
||
```
|
||
bench stage run export --recipe recipe.json
|
||
# 等待 export:调用 terminal_exec({"wait_handle": "SUBMISSION_HANDLE"})
|
||
bench stage run compile --submission SUBMISSION_HANDLE --config compile.yaml
|
||
```
|
||
|
||
这里 SUBMISSION_HANDLE 替换为 export 实际返回的 handle。其他可按需调用的工具:
|
||
|
||
```
|
||
bench stage run fetch --recipe recipe.json
|
||
bench stage run baseline --submission SUBMISSION_HANDLE
|
||
bench stage run hbm --submission SUBMISSION_HANDLE
|
||
bench stage run board --submission SUBMISSION_HANDLE
|
||
bench stage run eval --submission SUBMISSION_HANDLE
|
||
```
|
||
|
||
`fetch` 可以核对权重,不是所有提交必经步骤。compile 会准备 04 配置;需要单独检查时用 `bench stage run calibrate --submission SUBMISSION_HANDLE --config compile.yaml`。使用某步结果前等待它完成,读返回的 stdout/stderr/meta;`recipe_outputs` 给出实际产物路径与摘要,不猜“最新文件”。
|
||
|
||
提交后仍可修改 workspace;再次带 `--recipe recipe.json` 得到新的提交 handle。正式产物只读,需要改图先复制到 workspace。新图不继承旧 HBM 或旧成绩;失败日志仍保留,可以继续试验。最后用自然语言如实说明完成内容、测量范围和未验证事项;任务要求板端精度而缺少对应证据时,不能声称完整通过。
|
||
"""
|
||
return before + body + "\n预算:" + after
|
||
|
||
|
||
def _recipe_system_prompt(system: str) -> str:
|
||
"""Keep the shared boundaries and replace only legacy stage invocation syntax."""
|
||
before, marker, remainder = system.partition("## 共享资源与正式验收\n")
|
||
_, end, after = remainder.partition("## 结束说明\n")
|
||
if not marker or not end:
|
||
raise ValueError("recipe system prompt is missing the existing stage section")
|
||
section = """## 共享资源与正式验收
|
||
按任务提示为当前操作提交必要文件:`bench stage run <stage> --recipe recipe.json` 冻结新版本,`--submission SUBMISSION_HANDLE` 复用那次提交。探索次序由你决定;已有 ONNX 可先 export,再试 compile。编译使用 `--config compile.yaml`,文件位于 workspace。
|
||
共享工具链、板卡和正式评分通过原 stage 工具,使用结果前调用 `terminal_exec({"wait_handle": "<handle>"})`,一次等待原阶段完成。日志和 meta 保存真实执行与失败;按返回的产物路径使用结果。固定任务目标、评测数据与评分阈值不可修改,claim 不能代替独立验收。
|
||
本任务使用私有 HOME;workspace 内可 cd、使用变量并安装私有依赖。共享工具链保持只读。
|
||
|
||
"""
|
||
return before + section + end + after
|
||
|
||
|
||
def setup_models_json(force: bool = False) -> Path:
|
||
"""渲染 ~/.pi/agent/models.json。key 只从 .env.local 来。"""
|
||
dst = Path(os.path.expanduser("~/.pi/agent/models.json"))
|
||
if dst.exists() and not force:
|
||
return dst
|
||
cfg = paths.config()["endpoints"]
|
||
tmpl = (paths.AGENT / "models.json.tmpl").read_text()
|
||
text = _render(tmpl,
|
||
TEACHER_BASE_URL=cfg["teacher"]["base_url"],
|
||
TEACHER_MODEL=cfg["teacher"]["model"],
|
||
TEACHER_API_KEY=paths.secret(cfg["teacher"]["api_key_env"]) or "",
|
||
STUDENT_BASE_URL=cfg["student"].get("base_url") or "http://127.0.0.1:8000/v1",
|
||
STUDENT_API_KEY=paths.secret(cfg["student"]["api_key_env"]) or "EMPTY",
|
||
STUDENT_MODEL=cfg["student"].get("model") or "student")
|
||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||
dst.write_text(text)
|
||
os.chmod(dst, 0o600)
|
||
return dst
|
||
|
||
|
||
def prepare(run_id: str, job_id: str, kk: int, mode: str, policy: str, dry_run: bool = False, runner: str = "pi") -> tuple[Path, dict, str]:
|
||
"""dry_run=True 只在内存里渲染,不建目录、不写 rollout.json、不动 DB(踩过:dry-run 把 3 个 kk 占成了空壳 RUNNING)。"""
|
||
if mode == "repair":
|
||
raise ValueError("Oracle repair collection is retired; existing rows and artifacts remain historical")
|
||
from .runners import get as get_runner
|
||
get_runner(runner).check(policy, execute=False)
|
||
jd = paths.RUNS / run_id / job_id
|
||
job = json.loads((jd / "job.json").read_text())
|
||
with (db.readonly() if dry_run else db.connect()) as con:
|
||
db.assert_run_open(con, run_id)
|
||
rd = rollout_dir(run_id, job_id, kk)
|
||
if job.get("model_recipe") and not paths.config()["rollouts"].get("jail", True):
|
||
raise ValueError("task recipes require the outer worker and mount jail")
|
||
if (rd / "rollout.json").exists():
|
||
raise SystemExit(f"rollout 已存在(append-only):{rd}。想再答一次就用下一个 kk。")
|
||
if not dry_run:
|
||
for d in ("workspace", "agent", "stages", "handles", "workspace/.pi"):
|
||
(rd / d).mkdir(parents=True, exist_ok=True)
|
||
|
||
# 输入缓存可复用;缺少导出或基线时由 Agent 在自己的轨迹中提交。
|
||
onnx = None if job.get("model_recipe") else onnx_path(job)
|
||
bdir = None if job.get("model_recipe") else baseline_dir(job)
|
||
|
||
cfg = paths.config()
|
||
ep = cfg["endpoints"][policy]
|
||
if runner == "pi":
|
||
settings = _render((paths.AGENT / "settings.json.tmpl").read_text(),
|
||
PROJECT_ROOT=str(paths.PROJECT_ROOT), BENCH_ROOT=str(paths.BENCH_ROOT), MODEL=ep["model"])
|
||
sj = json.loads(settings)
|
||
sj.pop("_comment", None)
|
||
sj["defaultProvider"] = "dr-gateway" if policy == "teacher" else "student-vllm"
|
||
sj["tools"] = rollout_tools(cfg)
|
||
if not dry_run:
|
||
(rd / "workspace" / ".pi" / "settings.json").write_text(json.dumps(sj, indent=2, ensure_ascii=False))
|
||
if job.get("model_recipe") or cfg["rollouts"].get("private_home", False):
|
||
# Only this task's model configuration; never copy the host HOME.
|
||
model_config = json.loads(_render((paths.AGENT / "models.json.tmpl").read_text(),
|
||
TEACHER_BASE_URL=ep["base_url"], TEACHER_MODEL=ep["model"],
|
||
TEACHER_API_KEY="BENCH_RUNNER_API_KEY", STUDENT_BASE_URL=ep["base_url"],
|
||
STUDENT_MODEL=ep["model"], STUDENT_API_KEY="BENCH_RUNNER_API_KEY"))
|
||
provider = sj["defaultProvider"]
|
||
model_dir = rd / "workspace/.agent-home/.pi/agent"
|
||
model_dir.mkdir(parents=True)
|
||
(model_dir / "models.json").write_text(json.dumps({
|
||
"providers": {provider: model_config["providers"][provider]}}))
|
||
|
||
# prompt
|
||
tname = job.get("prompt") or (f"{mode}.md")
|
||
tmpl = (paths.AGENT / "prompts" / tname).read_text()
|
||
g = job["gates"]
|
||
prompt = _render(
|
||
tmpl, job_id=job_id, run_id=run_id, model=job["model"], task=job["task"], march=job["march"],
|
||
backend=job["backend"], qcfg=job["qcfg"], job_json=str(jd / "job.json"), data_dir=str(jd / "data"),
|
||
input_spec=job["model_spec"]["input_spec"], onnx=str(onnx) if onnx else "尚未生成;由 fetch/export 阶段返回路径", baseline_dir=str(bdir),
|
||
calib_dir=job["data"]["calib"]["path"], calib_n=job["data"]["calib"]["n"],
|
||
t1_tar=job["data"]["t1"]["path"], t1_n=job["data"]["t1"]["n"],
|
||
g2_threshold=g["G2"].get("threshold"), g3_metric=g["G3"].get("metric"), g3_tol=g["G3"].get("tol"),
|
||
g3_kind=g["G3"].get("kind"), max_turns=cfg["rollouts"]["max_turns"], max_tokens=cfg["rollouts"]["max_tokens"],
|
||
)
|
||
if job.get("model_recipe"):
|
||
# Insert the fixed policy after placeholder expansion so literal braces
|
||
# inside its file names/metadata are never interpreted as template keys.
|
||
prompt = _recipe_prompt(prompt, job)
|
||
if contract := _io_contract(job["model"]):
|
||
prompt += "\n\n" + contract + "\n"
|
||
if dry_run:
|
||
return rd, job, prompt
|
||
(rd / "agent" / "prompt.md").write_text(prompt)
|
||
# 先保存追加提示词;SDK 启动后另存实际完整 system/tools 上下文。
|
||
# legacy 只有追加快照,严格导出会保留但拒绝其 SFT 准入。
|
||
if job.get("model_recipe"):
|
||
system = _recipe_system_prompt((paths.AGENT / "SYSTEM.md").read_text())
|
||
(rd / "agent" / "system.md").write_text(system)
|
||
else:
|
||
shutil.copy2(paths.AGENT / "SYSTEM.md", rd / "agent" / "system.md")
|
||
shutil.copy2(rd / "agent/system.md", rd / "agent/append-system.md")
|
||
(rd / "rollout.json").write_text(json.dumps({"schema": "bench.rollout/1", "run_id": run_id, "job_id": job_id,
|
||
"kk": kk, "mode": mode, "policy": policy, "runner": runner, "runner_version": get_runner(runner).VERSION, "model": ep["model"],
|
||
"agent_config_version": agent_config_version() if runner == "pi" else "1",
|
||
"max_turns": cfg["rollouts"]["max_turns"], "max_tokens": cfg["rollouts"]["max_tokens"],
|
||
"state": "RUNNING", "started_at": time.time(),
|
||
**({"natural_agent_version": "natural-agent-v1"} if cfg.get("terminal", {}).get("enabled") else {}),
|
||
**({"jail_policy": "private-home"} if job.get("model_recipe") or cfg["rollouts"].get("private_home", False) else {})}, indent=2))
|
||
# 工具定义快照:pi 不把工具 schema 写进 events.jsonl,导出 SFT 样本的 tools 字段要靠它。
|
||
# 从 pi 包的 create*Tool 工厂现取(权威),失败不致命——导出时还能按 cmd.json 的白名单补跑一次。
|
||
try:
|
||
if runner != "pi":
|
||
raise ValueError("runner owns its tool definitions")
|
||
import subprocess as _sp
|
||
node = paths.expand(cfg["paths"].get("node_bin") or "node")
|
||
td = _sp.run([str(node), str(paths.AGENT / "dump_tools.js"), ",".join(rollout_tools(cfg))],
|
||
capture_output=True, text=True, timeout=60, env={**os.environ, "BENCH_ROLLOUT_DIR": str(rd)})
|
||
if td.returncode == 0 and td.stdout.strip():
|
||
(rd / "agent" / "tools.json").write_text(td.stdout)
|
||
except Exception:
|
||
pass
|
||
with db.connect() as con:
|
||
db.rollout_upsert(con, run_id, job_id, kk, mode, policy, runner=runner)
|
||
db.rollout_set(con, run_id, job_id, kk, state="RUNNING", started_at=time.time())
|
||
if runner != "pi":
|
||
get_runner(runner).prepare(rd, policy)
|
||
return rd, job, prompt
|
||
|
||
|
||
def agent_config_version() -> str:
|
||
"""agent/VERSION:提示词 / 扩展 / 预算规则的口径版本。改了 SYSTEM.md、prompts、extensions 就 +1;导出可按版本筛。"""
|
||
p = paths.AGENT / "VERSION"
|
||
return p.read_text().strip() if p.exists() else "1"
|
||
|
||
|
||
def agent_env(rd: Path, mode: str = "e2e") -> dict:
|
||
"""路径与预算;新配方 Pi 仅额外传当前模型调用必需的 key。"""
|
||
cfg = paths.config()
|
||
path_parts = [str(paths.BENCH_ROOT / "bin")]
|
||
if cfg["paths"].get("node_bin"):
|
||
path_parts.append(str(Path(paths.expand(cfg["paths"]["node_bin"])).parent))
|
||
path_parts.append(os.environ.get("PATH", "/usr/bin:/bin"))
|
||
# path-guard 的白名单按"项目树内路径"判;state/ 是指向 /home 的符号链接、work_root 是本地盘,
|
||
# 用真实路径就绕开了 → 把它们的 realpath 作为额外拒绝项传给 guard;本 rollout 自己的编译工作目录放行。
|
||
wr = paths.expand(cfg["paths"]["work_root"])
|
||
rel = rd.relative_to(paths.RUNS) # <run>/<job>/rollouts/<kk>
|
||
deny = [str(Path(paths.STATE).resolve()), str(wr)]
|
||
allow = [str(wr / rel.parts[0] / rel.parts[1] / f"r{int(rel.parts[3]):02d}")]
|
||
env = {
|
||
"BENCH_GUARD_DENY": ":".join(deny),
|
||
"BENCH_GUARD_ALLOW": ":".join(allow),
|
||
"PATH": ":".join(path_parts),
|
||
"HOME": os.environ.get("HOME", "/tmp"),
|
||
"LANG": "C.UTF-8",
|
||
"TERM": "dumb",
|
||
"BENCH_ROOT": str(paths.BENCH_ROOT),
|
||
"BENCH_CONFIG": str(paths.CONFIG_PATH),
|
||
"BENCH_STATE_DIR": str(paths.STATE),
|
||
"BENCH_HOST_HOME": os.environ.get("HOME", "/tmp"),
|
||
"BENCH_OUTER_WORKER": "1",
|
||
"BENCH_ROLLOUT_DIR": str(rd),
|
||
"BENCH_MODE": mode, # path-guard:e2e 看不到 oracle/,repair 可以
|
||
"BENCH_MAX_TURNS": str(cfg["rollouts"]["max_turns"]),
|
||
"BENCH_MAX_TOKENS": str(cfg["rollouts"]["max_tokens"]),
|
||
"PYTHONPATH": str(paths.BENCH_ROOT),
|
||
}
|
||
|
||
snapshot = rd / "rollout.json"
|
||
recorded = json.loads(snapshot.read_text()) if snapshot.exists() else {}
|
||
if recorded.get("runner", "pi") == "pi" and recorded.get("jail_policy") == "private-home":
|
||
home = rd / "workspace/.agent-home"
|
||
ep = cfg["endpoints"][recorded["policy"]]
|
||
env.update(HOME=str(home), PI_CODING_AGENT_DIR=str(home / ".pi/agent"),
|
||
XDG_CONFIG_HOME=str(home / "config"), XDG_CACHE_HOME=str(home / "cache"),
|
||
XDG_DATA_HOME=str(home / "data"), XDG_STATE_HOME=str(home / "state"),
|
||
BENCH_RUNNER_API_KEY=paths.secret(ep["api_key_env"]) or "EMPTY")
|
||
return env
|
||
|
||
|
||
def jail_prefix(rd: Path, mode: str) -> list[str]:
|
||
"""把 pi 关进 mount namespace:oracle/(e2e)、兄弟 rollout、~/.pi、.env.local 直接从文件系统里摘掉。
|
||
|
||
path-guard 是解析 shell 字符串的白名单,v1→v3 每一轮都被新写法穿透(换行分隔的 cd、$IFS、
|
||
$(…)、$VAR/ 拼接、workspace 里造软链)。这一层不解析任何东西,绕法再多也看不见那些路径。
|
||
unshare 不可用时拒绝启动,不降低隔离要求。
|
||
"""
|
||
if not paths.config()["rollouts"].get("jail", True):
|
||
return []
|
||
js = paths.BENCH_ROOT / "scripts" / "agent_jail.sh"
|
||
if not js.exists() or not shutil.which("unshare"):
|
||
raise RuntimeError("agent_jail 不可用:拒绝降级隔离")
|
||
return [str(js), str(rd), mode, "--"]
|
||
|
||
|
||
def pi_command(rd: Path, prompt: str, mode: str = "e2e") -> list[str]:
|
||
cfg = paths.config()
|
||
if cfg.get("collection_v1"):
|
||
node = str(paths.expand(cfg["paths"].get("node_bin") or "node"))
|
||
return jail_prefix(rd, mode) + [node, str(paths.AGENT / "pi_session.js"), str(rd / "agent/pi-session.json")]
|
||
pi = str(paths.expand(cfg["paths"].get("pi_bin") or "pi")) if "/" in (cfg["paths"].get("pi_bin") or "") else (cfg["paths"].get("pi_bin") or "pi")
|
||
# 实测 pi-coding-agent 0.73.1(pi --help):--mode json 单独就是非交互 JSON 事件流;没有 -a 这个参数;
|
||
# --append-system-prompt 接受文本或文件路径;--tools 是逗号分隔白名单;extensions/skills 由 workspace/.pi/settings.json 发现。
|
||
return jail_prefix(rd, mode) + [
|
||
pi, "--mode", "json", "--session-dir", str(rd / "workspace" / ".pi-sessions") if paths.config()["rollouts"].get("jail", True) else str(rd / "agent"),
|
||
"--append-system-prompt", str(rd / "agent" / "system.md"), # 用 rollout 内的快照,与导出补回的 system 消息逐字一致
|
||
"--tools", ",".join(rollout_tools(cfg)),
|
||
prompt]
|
||
|
||
|
||
def run_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str, dry_run: bool = False,
|
||
*, _claimed: bool = False, runner: str = "pi") -> dict:
|
||
from .runners import get as get_runner
|
||
# The collector checked quota right before claiming; a second 0.5 s ledger read here turned busy
|
||
# moments into UNVERIFIED rollouts that never ran (10-13: 59).
|
||
get_runner(runner).check(policy, execute=not dry_run and not _claimed)
|
||
if not dry_run and runner == "pi":
|
||
from .collection_runtime import config
|
||
value, _ = config()
|
||
if policy not in value['credential_groups']:
|
||
raise ValueError('QUOTA_CONFIG_INCOMPLETE: ' + policy)
|
||
if not _claimed:
|
||
from .recovery import reconcile_run
|
||
if not reconcile_run(run_id)['ready']:
|
||
raise ValueError('EXECUTION_RECONCILIATION_REQUIRED')
|
||
if mode == "repair":
|
||
raise ValueError("Oracle repair collection is retired; existing rows and artifacts remain historical")
|
||
with (db.readonly() if dry_run else db.connect()) as con:
|
||
db.assert_run_open(con, run_id)
|
||
run = db.get_run(con, run_id)
|
||
row = con.execute("SELECT * FROM rollouts WHERE run_id=? AND job_id=? AND kk=?", (run_id, job_id, kk)).fetchone()
|
||
if run and json.loads(run["manifest"]).get("continuation"):
|
||
if not row or (row["mode"], row["policy"]) != (mode, policy):
|
||
raise ValueError("rollout is not in the continuation queue")
|
||
if row and db.rollout_runner(con, run_id, job_id, kk) != runner:
|
||
raise ValueError("rollout runner differs from the queued task")
|
||
if row and (row["mode"], row["policy"]) != (mode, policy):
|
||
raise ValueError("rollout mode/policy differs from the queued task")
|
||
if not dry_run:
|
||
if _claimed:
|
||
if not row or row["state"] != "RUNNING":
|
||
raise ValueError("collector has not claimed this rollout")
|
||
else:
|
||
if not row:
|
||
db.rollout_upsert(con, run_id, job_id, kk, mode, policy, runner=runner)
|
||
con.commit()
|
||
if not db.claim_rollout(con, run_id, job_id, kk):
|
||
raise ValueError("rollout already claimed or finished")
|
||
try:
|
||
return _execute_rollout(run_id, job_id, kk, mode, policy, dry_run, runner=runner)
|
||
except (SystemExit, Exception):
|
||
if not dry_run:
|
||
from .execution import Execution, owned_processes
|
||
rd = rollout_dir(run_id, job_id, kk)
|
||
if (rd / 'execution.json').exists():
|
||
execution = Execution(rd).attach()
|
||
snapshot = execution.snapshot()
|
||
if runner == 'pi':
|
||
from .collection_runtime import broker
|
||
broker().cancel(snapshot['execution_id'])
|
||
if snapshot['execution_state'] not in ('TERMINAL', 'PAUSED_INFRA'):
|
||
clean = owned_processes(snapshot) == []
|
||
execution.finish(None, protocol_complete=False, stop_reason='INFRA_EXECUTION_EXCEPTION',
|
||
cleanup_state='VERIFIED' if clean else 'QUARANTINED')
|
||
if not dry_run and not _claimed:
|
||
with db.connect() as con:
|
||
db.rollout_set(con, run_id, job_id, kk, state="UNVERIFIED", finished_at=time.time())
|
||
raise
|
||
|
||
|
||
def settle_execution(run_id, job_id, kk, execution, reason, *, cancel=False, timeout_s=None):
|
||
"""Close control-plane stops and retry their registration without rerunning work."""
|
||
from .execution import atomic_json
|
||
from .collection_runtime import broker
|
||
snapshot = execution.snapshot()
|
||
if snapshot['execution_state'] != 'TERMINAL':
|
||
if cancel:
|
||
snapshot = execution.cancel(reason)
|
||
if snapshot['execution_state'] != 'TERMINAL':
|
||
try:
|
||
broker().cancel(execution.snapshot()['execution_id'])
|
||
except (OSError, ValueError):
|
||
# Durable cancellation still fences new work; never refund an unknown request.
|
||
pass
|
||
execution.cleanup()
|
||
state = execution.snapshot()
|
||
snapshot = execution.finish(None, protocol_complete=False, stop_reason=reason,
|
||
cleanup_state=state['cleanup_state'])
|
||
if snapshot['execution_outcome'] == 'COMPLETED':
|
||
return {k: v for k, v in snapshot.items() if k != 'token'}
|
||
rd = rollout_dir(run_id, job_id, kk)
|
||
metadata = json.loads((rd / 'rollout.json').read_text())
|
||
if rd.resolve() != execution.rd.resolve():
|
||
raise ValueError('rollout settlement identity mismatch')
|
||
if metadata.get('state') in ('RUNNING', 'VERIFYING', 'PAUSED_INFRA'):
|
||
metadata.update(state=snapshot['execution_outcome'], execution_outcome=snapshot['execution_outcome'],
|
||
stop_reason=snapshot['stop_reason'], cleanup_state=snapshot['cleanup_state'],
|
||
finished_at=snapshot['finished_at'], collection_hold=True)
|
||
atomic_json(rd / 'rollout.json', metadata)
|
||
with db.connect(**({'timeout_s': timeout_s} if timeout_s is not None else {})) as con:
|
||
# A verified task verdict is not replaced by a later control-plane retry.
|
||
con.execute("UPDATE rollouts SET state=?,finished_at=? WHERE run_id=? AND job_id=? AND kk=? "
|
||
"AND state IN ('RUNNING','VERIFYING','PAUSED_INFRA')",
|
||
(snapshot['execution_outcome'], snapshot['finished_at'], run_id, job_id, kk))
|
||
return {k: v for k, v in snapshot.items() if k != 'token'}
|
||
|
||
|
||
def resume_rollout(run_id: str, job_id: str, kk: int) -> dict:
|
||
"""Resume one protected checkpoint, with the original attempt and budgets."""
|
||
from .execution import Execution
|
||
from . import collection_runtime as runtime
|
||
rd = rollout_dir(run_id, job_id, kk)
|
||
metadata = json.loads((rd / 'rollout.json').read_text())
|
||
if metadata.get('runner', 'pi') != 'pi' or metadata.get('state') != 'PAUSED_INFRA':
|
||
raise ValueError('RESUME_NOT_ELIGIBLE')
|
||
execution = Execution(rd).attach()
|
||
state = execution.snapshot()
|
||
runtime.resume_policy(state)
|
||
checkpoint = state.get('checkpoint')
|
||
if not checkpoint:
|
||
raise ValueError('RESUME_CHECKPOINT_MISSING')
|
||
readiness = runtime.broker().resume_readiness(state['execution_id'])
|
||
# A collector restart with no request in flight has nothing for the ledger to arm; the next request is new.
|
||
restarted = state.get('stop_reason') == 'CONTROLLER_RESTARTED'
|
||
in_flight = readiness.get('reason') != 'NO_PAUSED_GENERATION'
|
||
if not readiness['eligible'] and not (restarted and not in_flight):
|
||
raise ValueError('RESUME_NOT_READY: ' + str(readiness.get('reason')))
|
||
execution.resume(counted=readiness.get('counted', True) and not restarted)
|
||
if in_flight:
|
||
try:
|
||
runtime.broker().resume(state['execution_id'], checkpoint['generation_id'],
|
||
checkpoint_complete=checkpoint.get('resumable', False), side_effects_known=True)
|
||
except Exception:
|
||
execution.pause(state.get('stop_reason') or 'QUOTA_WAIT_BUDGET_EXHAUSTED', checkpoint)
|
||
raise
|
||
with db.connect() as con:
|
||
changed = con.execute("UPDATE rollouts SET state='RUNNING' WHERE run_id=? AND job_id=? AND kk=? AND state='PAUSED_INFRA'",
|
||
(run_id, job_id, kk)).rowcount
|
||
if changed != 1:
|
||
execution.pause('RESUME_DB_CONFLICT', checkpoint)
|
||
raise ValueError('RESUME_DB_CONFLICT')
|
||
metadata.update(state='RUNNING')
|
||
(rd / 'rollout.json').write_text(json.dumps(metadata, indent=2))
|
||
try:
|
||
return _execute_rollout(run_id, job_id, kk, metadata['mode'], metadata['policy'], False, 'pi', resume=True)
|
||
except Exception:
|
||
# No new attempt and no implicit replay after an uncertain resumed run.
|
||
runtime.broker().cancel(state['execution_id'])
|
||
execution.quarantine('RESUME_EXECUTION_UNCONFIRMED')
|
||
with db.connect() as con:
|
||
db.rollout_set(con, run_id, job_id, kk, state='PAUSED_INFRA')
|
||
raise
|
||
|
||
|
||
def _last_generation_reason(execution):
|
||
from . import collection_runtime as runtime
|
||
pending = (runtime.broker().execution_status(execution.snapshot()['execution_id']) or {}).get('generations', [])
|
||
return pending[-1].get('reason') if pending else None
|
||
|
||
|
||
def _execute_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str, dry_run: bool, runner: str = "pi", *, resume=False) -> dict:
|
||
from .runners import get as get_runner
|
||
adapter = get_runner(runner)
|
||
if resume:
|
||
rd = rollout_dir(run_id, job_id, kk)
|
||
job = json.loads((rd.parent.parent / 'job.json').read_text())
|
||
prompt = (rd / 'agent/prompt.md').read_text()
|
||
else:
|
||
rd, job, prompt = prepare(run_id, job_id, kk, mode, policy, dry_run=dry_run, runner=runner)
|
||
if not dry_run and runner == "pi" and not job.get("model_recipe") and not paths.config()["rollouts"].get("private_home", False):
|
||
setup_models_json()
|
||
cmd = adapter.command(rd, prompt, mode)
|
||
if not dry_run:
|
||
(rd / "agent" / "cmd.json").write_text(json.dumps({"cmd": cmd, "env_keys": sorted(adapter.environment(rd, mode))}, indent=2, ensure_ascii=False))
|
||
if dry_run:
|
||
print(f"[dry-run] rollout 已准备:{rd}\n cwd={rd / 'workspace'}\n cmd={' '.join(cmd[:8])} …")
|
||
return {"rollout_dir": str(rd), "dry_run": True}
|
||
cfg = paths.config()["rollouts"]
|
||
from contextlib import nullcontext
|
||
from .stageworker import Worker, config_bytes
|
||
(paths.expand(paths.config()['paths']['work_root']) / run_id / job_id / f'r{kk:02d}').mkdir(parents=True, exist_ok=True)
|
||
jailed = bool(jail_prefix(rd, mode))
|
||
execution, scope = None, None
|
||
env = adapter.environment(rd, mode)
|
||
tool_deadlines = None
|
||
if runner in ('pi', 'opencode'):
|
||
from .execution import Execution
|
||
from . import collection_runtime as runtime
|
||
runtime_config, _ = runtime.config()
|
||
execution = Execution(rd)
|
||
if resume:
|
||
execution.attach()
|
||
else:
|
||
execution.start(runtime_config['lifecycle'], activity='QUEUED')
|
||
if runner == 'pi':
|
||
scope = runtime.configure_pi(rd, execution, policy, resume=resume)
|
||
tool_deadlines = runtime.ToolDeadlines(rd, execution)
|
||
used = runtime.budget_used(rd)
|
||
env.update(BENCH_INITIAL_TURNS=str(used['turns']), BENCH_INITIAL_TOKENS=str(used['tokens']))
|
||
else:
|
||
# ponytail: no checkpoint/resume or per-tool deadlines for OpenCode yet; a pause or restart ends it.
|
||
scope = adapter.configure(rd, execution, policy)
|
||
env['BENCH_RUNNER_API_KEY'] = scope.api_key
|
||
def stop_for_budget():
|
||
if runner == "pi":
|
||
tool_reason = tool_deadlines.poll()
|
||
if tool_reason:
|
||
return tool_reason
|
||
state = execution.snapshot()
|
||
if state.get('cancel_requested'):
|
||
runtime.broker().cancel(state['execution_id'])
|
||
elif state['execution_state'] == 'ACTIVE':
|
||
quota_state = runtime.broker().execution_status(state['execution_id']) or {}
|
||
generations = quota_state.get('generations', [])
|
||
if generations:
|
||
latest = generations[-1]
|
||
activity = 'WAITING_QUOTA' if latest['state'] in ('QUEUED', 'RESUMABLE') else 'WAITING_LLM'
|
||
if (state['activity'] != 'RUNNING_TOOL' and latest['state'] not in ('COMPLETE', 'PAUSED', 'CANCELLED')
|
||
and state['activity'] != activity):
|
||
execution.activity(activity, generation_id=latest['generation_id'], reason=latest.get('reason'))
|
||
return None
|
||
if execution and execution.snapshot().get('cancel_requested'):
|
||
runtime.broker().cancel(execution.snapshot()['execution_id'])
|
||
cost = adapter.cost(rd)
|
||
return "runner budget exhausted" if (cost["turns"] >= cfg["max_turns"] or cost["tokens"] >= cfg["max_tokens"]) else None
|
||
from . import scheduler
|
||
finalization_error = None
|
||
checkpoint = {}
|
||
with execution.scope() if execution else nullcontext():
|
||
with scheduler.acquire('agent'):
|
||
if execution:
|
||
execution.activity('WAITING_LLM')
|
||
with Worker(rd, execution=execution) if jailed else nullcontext():
|
||
r = watchdog.run(cmd, cwd=rd / "workspace", stdout=rd / "agent" / "events.jsonl", stderr=rd / "agent" / f"{runner}.stderr.log",
|
||
env=env, stall_s=cfg["hard_ceiling_s"], poll_s=5 if runner == "pi" else 1,
|
||
hard_ceiling_s=cfg["hard_ceiling_s"], stop_check=stop_for_budget,
|
||
execution=execution, role='agent',
|
||
on_finish=(lambda _: tool_deadlines.poll()) if tool_deadlines else None)
|
||
# Freeze the Agent's first exit before draining its accepted stages.
|
||
try:
|
||
adapter.finish(rd)
|
||
if runner == 'pi':
|
||
runtime.capture_context(rd)
|
||
checkpoint = runtime.freeze_checkpoint(rd, execution)
|
||
except Exception as exc:
|
||
finalization_error = str(exc)
|
||
(rd / "agent/session-error.json").write_text(json.dumps({"error": type(exc).__name__, "detail": str(exc)}))
|
||
if execution:
|
||
quota = runtime.broker().execution_status(execution.snapshot()['execution_id']) or {}
|
||
generations = quota.get('generations', [])
|
||
used = runtime.budget_used(rd) if runner == 'pi' else adapter.cost(rd)
|
||
limits = json.loads((rd / 'rollout.json').read_text())
|
||
first_reason = ('SESSION_FINALIZATION_ERROR' if finalization_error else
|
||
((generations[-1].get('reason') if generations else None) or 'INCOMPLETE_GENERATION') if r.exit_code == 76 else
|
||
'BUDGET_EXHAUSTED' if r.exit_code == 75 or used['turns'] >= limits['max_turns'] or used['tokens'] >= limits['max_tokens'] else
|
||
r.stalled_reason)
|
||
with execution.mutation() as state:
|
||
execution._save(state, 'AGENT_EXITED', exit_code=r.exit_code,
|
||
checkpoint_phase=checkpoint.get('phase'), reason=first_reason,
|
||
turns=used['turns'], tokens=used['tokens'])
|
||
if execution and not finalization_error and r.exit_code == 76 and not execution.snapshot().get('cancel_requested'):
|
||
quota_state = runtime.broker().execution_status(execution.snapshot()['execution_id'])
|
||
pending = (quota_state or {}).get('generations', [])
|
||
reason = pending[-1].get('reason') if pending else 'INCOMPLETE_GENERATION'
|
||
# A 400 is the provider refusing this exact request (e.g. the context outgrew the model): no resume can accept
|
||
# it, so a pause only held the rollout until its deadline (10-15 04 retinanet.r101: 760k-token request).
|
||
if (reason not in ('UPSTREAM_REJECTED', 'CONTEXT_TOO_LONG') and checkpoint.get('resumable') and checkpoint.get('phase') == 'PAUSED_INFRA'
|
||
and checkpoint.get('closure') and pending and pending[-1]['state'] == 'PAUSED'):
|
||
execution.pause(reason, checkpoint)
|
||
rj = json.loads((rd / 'rollout.json').read_text())
|
||
rj.update(state='PAUSED_INFRA', stop_reason=reason, exit_code=r.exit_code)
|
||
(rd / 'rollout.json').write_text(json.dumps(rj, indent=2))
|
||
with db.connect() as con:
|
||
db.rollout_set(con, run_id, job_id, kk, state='PAUSED_INFRA')
|
||
return {'rollout_dir': str(rd), 'terminal': 'PAUSED_INFRA', 'reason': reason}
|
||
if jailed:
|
||
# These remain untrusted claims/logs, never verdict or authoritative budgets.
|
||
for src, dst in [('claim.json', 'claim.json'), ('guard.log', 'agent/guard.log')]:
|
||
try:
|
||
(rd / dst).write_bytes(config_bytes(rd / 'workspace', str(rd / 'workspace' / src)))
|
||
except (OSError, ValueError):
|
||
pass
|
||
rj = json.loads((rd / "rollout.json").read_text())
|
||
from .verify import _cost
|
||
cost = runtime.budget_used(rd) if runner == 'pi' else _cost(rd)
|
||
budget_exhausted = cost["turns"] >= rj["max_turns"] or cost["tokens"] >= rj["max_tokens"]
|
||
context_full = bool(execution) and r.exit_code == 76 and _last_generation_reason(execution) == 'CONTEXT_TOO_LONG'
|
||
if budget_exhausted or context_full:
|
||
rj["state"] = "TIMEOUT"
|
||
rj["timeout_reason"] = "context window full" if context_full else "budget (events verified)"
|
||
if r.state == "STALLED" or r.exit_code == 75:
|
||
rj["state"] = "TIMEOUT"; rj.setdefault("timeout_reason", r.stalled_reason or "budget")
|
||
elif rj.get("state") != "TIMEOUT":
|
||
rj["state"] = "ERROR" if finalization_error or r.exit_code != 0 else "DONE"
|
||
if finalization_error:
|
||
rj["session_error"] = finalization_error
|
||
rj.update({"exit_code": r.exit_code, "wall_s": round(r.duration_s, 1), "finished_at": time.time()})
|
||
if execution:
|
||
protocol_complete = not finalization_error and (checkpoint.get('phase') == 'COMPLETED' if runner == 'pi' else r.exit_code == 0)
|
||
stop_reason = execution.snapshot().get('stop_reason') or (
|
||
'BUDGET_EXHAUSTED' if budget_exhausted or r.exit_code == 75 else r.stalled_reason)
|
||
if r.exit_code == 76 and not stop_reason:
|
||
# The agent filled the model's window: its own budget ran out, like turns or tokens (outcome BUDGET_EXHAUSTED).
|
||
last = _last_generation_reason(execution)
|
||
stop_reason = 'CONTEXT_BUDGET_EXHAUSTED' if last == 'CONTEXT_TOO_LONG' else 'INFRA_' + (last or 'INCOMPLETE_GENERATION')
|
||
# A resumable quota pause returned above. Every terminal path must fence
|
||
# the broker BEFORE publishing a finished rollout, including SDK errors.
|
||
runtime.broker().cancel(execution.snapshot()['execution_id'])
|
||
finished = execution.finish(r.exit_code, protocol_complete=protocol_complete, stop_reason=stop_reason,
|
||
cleanup_state=r.cleanup_state)
|
||
rj['execution_outcome'] = finished['execution_outcome']
|
||
rj['cleanup_state'] = finished['cleanup_state']
|
||
if finished['stop_reason'] == 'TOOL_DEADLINE_EXCEEDED':
|
||
rj.update(state='DEADLINE_EXCEEDED', stop_reason=finished['stop_reason'], collection_hold=True)
|
||
if finished['execution_outcome'] != 'COMPLETED' and rj['state'] == 'DONE':
|
||
rj['state'] = 'ERROR'
|
||
(rd / "rollout.json").write_text(json.dumps(rj, indent=2))
|
||
with db.connect() as con:
|
||
db.rollout_set(con, run_id, job_id, kk, state="VERIFYING")
|
||
v = verify_rollout(rd)
|
||
terminal = (v.get('execution_control') or {}).get('db_state', v['terminal'])
|
||
return {"rollout_dir": str(rd), "terminal": terminal, "claim_consistent": v["claim_consistent"], "review": v["review"]}
|