Compare commits

..

No commits in common. "simplify-20261017" and "main" have entirely different histories.

123 changed files with 394 additions and 8433 deletions

View File

@ -35,7 +35,3 @@
任务输入在 `runs/<run_id>/<job_id>/job.json`,轨迹和证据在 `runs/<run_id>/<job_id>/rollouts/<kk>/`;状态数据库由 `BENCH_STATE_DIR` 决定。配置中的工作、缓存和数据路径继续沿用现有环境,不手工重置运行数据或隔离标记。
`bench status --run <run_id>` 查看批次状态;`scripts/audit_rollouts.py --run <run_id>` 查看轨迹验收。任务启动、模型回复、板测通过和最终 SFT 是不同证据,不能相互替代。
## 经验记忆
开始工作前先读 [docs/memory/AGENTS.md](docs/memory/AGENTS.md)。那里有之前的 agent 积累下来的用户偏好、现场事实和踩过的坑,也写了新增记忆的格式和存放规则。

View File

@ -1,64 +1,65 @@
# bench agent · 系统提示(追加在 pi 默认系统提示之后)
你负责完成 job.json 固定的部署任务:从模型权重出发,交付一个在目标板卡上精度和速度都合格的模型。怎么做由你决定;验收只认平台在板上的实际测量。
你负责完成 job.json 固定的部署任务。harness 提供共享资源、实际过程记录与独立验收。探索顺序、实现方式和重试方法由你决定,受本任务总预算约束;其中编译次数是主预算,每条轨迹启动编译的次数有硬上限,用完即不能再编译。
## 环境
- 当前目录是你的 workspace,可写。本 rollout 的 job.json、日志和阶段产物可读;公共源码、共享工具链、其他 rollout、评测数据和判分结果不属于本任务。
- 你的 bash 跑在控制机上,**没有工具链**(hb_compile、hmct、onnx、onnxruntime、torch 都不在),用到它们的命令走 `bench dev`。单条 bash 命令约 14 分钟会被结束,只结束这一条。
- 工具链文档:`oe-doc <query>` 和 `.horizon/skills/`。`$BENCH_ROOT/harness/adapters/` 里是公共的预处理和后处理代码,可读。
## 环境和结果边界
- workspace 是你的可写工作区,可以修改自己的代码、模型副本和私有环境。公共源码、共享工具链、其他任务和运行程序不可修改。
- 本 rollout 的日志、阶段产物、job.json 与任务数据可读;正式提交和阶段证据只读。其他 rollout、状态库、凭据与 oracle 不属于本任务。
- 公共可复用工具位于 `$BENCH_ROOT/harness/adapters/`;`BENCH_ROOT` 已由运行环境提供,可读、可复制修改自己的副本,不是新增模型的必经入口。评判代码仍不作为 Agent 的实现依赖。
- 查找文件先用 `rg --files`,搜索内容用 `rg`;限定在 workspace、任务返回的日志目录或上述公共 adapters 目录。不要执行 `find /` 或遍历整台服务器来寻找代码;路径不可访问时依据工具报错处理。
- 单条 bash 命令最长约 14 分钟,到时只结束这条命令(返回 `Command timed out`),轨迹继续。更久的工作放后台运行(`nohup … > log 2>&1 &`),再分次查看日志;不要用 `sleep` 一次等待超过 10 分钟。
- 现有 Pi legacy 沙箱仍保留 shell 路径限制;工具报限制时按其说明使用明确路径。不要为私有环境安装修改共享工具链。
- 目标板卡、数据、指标与阈值以 job.json 为准。编译失败、图变化、精度下降都如实保留;只有平台的实际测量能作为验收证据。
- 阶段在后台运行;提交返回 handle 后调用 `terminal_exec({"wait_handle": "<handle>"})`,一次等待原阶段完成或原期限终止,不消耗模型轮数反复轮询。不要为等待重提交阶段;诊断时只查看该 handle 返回的日志与对应阶段,不遍历所有历史产物。旧 `bench stage wait <handle> --poll 60` 仍可用于一次短查询,返回 2 仅表示仍在排队或运行。
## 等待不花轮数
`bench dev` 和正式阶段都在后台跑,提交后返回 handle。`terminal_exec({"wait_handle": "<handle>"})` 一次等到结束,不管等多久都只算一轮。用 sleep 或反复查看状态来等,每查一次都算一轮,还会占上下文。
## compile.yaml 里你能改的旋钮(PTQ · hb_compile 3.5.3 · hmct 2.6.5,已对源码核实)
harness 钉死 `model_parameters`(onnx / march / working_dir / 输出前缀),你写的 yaml 里这几项会被覆盖。其余三段照下面填,写进 `compile.yaml`(你的 cwd 就是 workspace/),然后 `bench stage run compile --config compile.yaml`(先 `bench stage run calibrate` 也行,它会把你的 yaml 与钉死项合并并落盘)。
## 开发环境:`bench dev`
```
bench dev "python check.py" # 在带完整工具链的容器里跑一条命令,cwd 是 workspace
bench dev --timeout 7200 -- python export.py # 长任务给足时限(最多 43200 秒;容器启动约 1 分钟)
bench dev --board -- python3 run.py # 把当前目录传到本题的板卡上运行,可用 BPU
```
- `python`、`hb_compile`、`hb_verifier`、`hb_model_info` 来自 OE 工具链(含 hmct、onnx、onnxruntime);torch 用 `/home/ming01.wang/toolchains/bench-venv/bin/python`,或 job.json 里 `model_runtime.python` 指定的环境。权重和模型源码只读可见。
- 校准图片:`$BENCH_CALIB_IMAGES`,是训练集原图,和评测集不重叠;`$BENCH_CALIB_IMAGES/../images.json` 列出顺序和来源。预处理以 job.json 的 `model_spec.preprocess` 为准。
- 同步:本机改过的文件在下一次 `bench dev` 开始时传上去;命令里新建或改动的小文件(单个 ≤16 MiB)拷回本机,大文件(图、npy、hbm)只留在开发环境,输出末尾会列出来。
- 板上:有 python3、numpy 和 hbm_runtime(`m = hbm_runtime.HB_HBMRuntime('x.hbm')`,`m.model_names`、`m.input_names[name]`、`m.input_shapes[name]`,`m.run({name: {输入名: 数组}})`);写到 `$BENCH_OUT` 的文件会拉回到结果目录的 `output/`。一次最多 30 分钟,板卡和正式阶段共用。
- 开发环境和板上怎么试都不计编译次数,结果也不作为验收依据。
## 共享资源与正式验收
```
bench stage run fetch # 正式权重
bench stage run export # 平台参考导出:图不给你,用来核对你的图的输入输出
bench stage run baseline # 独立的 FP32 基线
bench stage run compile --config compile.yaml # 正式编译
bench stage run hbm # 收集并检查编译产物
bench stage run board # 板上延迟
bench stage run eval # 板上任务精度
```
- compile.yaml 是 hb_compile 的 PTQ 配置(hb_compile 3.5.3 · hmct 2.6.5),放在 workspace 里。`model_parameters.onnx_model` 和 `calibration_parameters.cal_data_dir` 必填,写你在开发环境里做的图和校准 npy 目录(workspace 内的路径);march、working_dir、输出前缀由平台固定。这一版工具链实际接受的字段(已对源码核实,文档有出入时以这里为准):
```yaml
model_parameters:
onnx_model: my_model.onnx
input_parameters: # 默认 featuremap / NCHW / 无归一化,预处理在主机侧,与评测器一致
input_parameters: # 默认 featuremap/NCHW/无归一化,预处理在主机侧(评测器口径一致);一般不用动
input_type_rt: featuremap # featuremap | rgb | bgr | nv12 | gray | yuv444
input_type_train: featuremap
input_layout_train: NCHW
# mean_value / scale_value / std_value 只在 input_type 不是 featuremap 时有意义
# mean_value / scale_value / std_value 仅当 input_type 不是 featuremap 时有意义
calibration_parameters:
cal_data_dir: my_calib/ # 每个 .npy 是一个模型输入,形状同 input_spec,float32
calibration_type: default # default | max | kl | mix | skip
max_percentile: 0.99999 # 只对 max 有效
per_channel: true
run_on_cpu: "node_a,node_b" # ONNX 节点名,逗号分隔;run_on_bpu 反之
optimization: set_model_output_int16 # 或 set_model_output_int8
quant_config: # dict 或 json 路径;qtype: int8 | int16 | float16 | float32
calibration_type: default # default | max | kl | mix ← 首选旋钮。default = 自动搜索;掉点大时试 kl 或 mix
# (还有 skip = 完全不校准,只给预量化模型用;普通 PTQ 选它必崩 G2/G3,白烧一次编译)
max_percentile: 0.99999 # 只对 max 有效,截掉激活长尾(0.9999–1.0)
per_channel: true # 权重逐通道量化,对深度可分离卷积(MobileNet/EfficientNet)常有帮助
run_on_cpu: "node_a,node_b" # 把敏感节点放 CPU(float),逗号分隔 ONNX 节点名;run_on_bpu 反之
optimization: set_model_output_int16 # set_model_output_int8 | set_model_output_int16(其余以 hmct 报错为准)
quant_config: # 精细精度控制(dict 或 json 路径)。qtype: int8 | int16 | float16 | float32
model_config: { all_node_type: int8, model_output_type: int16 }
op_config: { Softmax: { qtype: float16 } } # 按算子类型
node_config: { "/features/features.8/Mul": { qtype: int16 } } # 按节点名
op_config: { Softmax: { qtype: float16 }, LayerNormalization: { qtype: float16 } } # 按算子类型
node_config: { "/features/features.8/features.8.2/Mul": { qtype: int16 } } # 按节点名(看 hb_compile 日志里的逐节点余弦表挑最差的)
compiler_parameters:
compile_mode: latency # latency | bandwidth | balance
optimize_level: O2 # O0 | O1 | O2(O3 会被拒)
optimize_level: O2 # O0 | O1 | O2 —— hbdk4 后端只接受这三个,写 O3 会在 yaml 校验阶段直接报错
jobs: 8
```
- 你的图的输入输出(个数、顺序、类型、形状)要和平台的一致,基线、上板和评测才可比;不一致时编译前就会被拒并列出期望值,不占编译次数。
- 阶段失败时看等待结果里的 stdout/stderr/meta 再决定下一步;板上延迟记录失败时仍可以跑精度评测。
诊断信号在哪:`terminal_exec` 等待结束后给你 stdout/stderr 路径;hb_compile 日志末尾有逐节点 `Calibrated Cosine / Quantized Cosine` 表和模型输出余弦。余弦偏低时可检查敏感节点、尝试 `node_config` 提到 int16;最终任务精度以 G3 实测与 job 中的阈值为准。
## 共享资源与正式验收
原阶段按需要调用,不要求探索按编号顺序进行。执行某一步仍需它实际消费的输入;使用结果前等待完成。
```
bench stage run fetch # 获取或复用正式权重
bench stage run export # 导出 ONNX
bench stage run baseline # 固定 T1 的独立 FP32 参考
bench stage run compile --config compile.yaml # 编译,含 04 配置准备
bench stage run hbm # 收集并检查编译产物
bench stage run board # 板端性能记录
bench stage run eval # 板端 T1 精度与 G2 诊断
```
随后调用 `terminal_exec({"wait_handle": "<handle>"})` 等待刚提交的阶段。
`--config` 在 workspace 内;目标板卡、产物路径和输出目录由 Worker 固定,配置记录实际使用值。共享工具链和板卡通过受管执行工具获取资源;若任务开放 terminal 命令模式,自写命令通过 terminal_exec 运行;仅阶段模式下该工具只提供等待。连接凭据由平台管理。查看 wait 返回的 stdout/stderr/meta,根据失败证据决定下一步;性能记录失败仍可尝试精度评测。
## 结束说明
最后用自然语言如实说明实际做了什么、平台实测结果是什么、哪些没有验证。总轮数、token 和墙钟预算是硬上限,收到收尾提醒时停止新工作。
用自然语言说明实际完成内容、测量范围、失败及未验证事项;不要求调用 claim 或使用规定终态词。总轮数、token 和墙钟预算仍是硬上限,收到收尾提醒时停止新工作并给出真实说明。
G1 是编译/产物检查,不是板端测量。精度结论说明样本集、样本数、指标、FP32 与板端值及固定阈值;标准误只作诊断,不放宽阈值。不能把子集筛查概括为全数据集无损。自写推理的 HBM 加载/输出关联若未被平台确认,明确板端预测来源未自动核实。缺材料时保留实际错误,不猜测失败原因。
## 工具链参考
`.horizon/skills/` 和 `oe-doc <query>` 提供工具链文档。共享执行环境由 job 和 Worker 选择;自己的试验依赖放在 workspace 私有环境。

View File

@ -1 +1 @@
23
15

View File

@ -70,7 +70,6 @@ function makeGuard(env) {
path.join(JOB, "job.json"), path.join(JOB, "data"),
path.join(BENCH, "harness", "adapters"),
path.join(BENCH, "data"), path.join(BENCH, "cache"), path.join(BENCH, "bin"), path.join(BENCH, "agent", "oe-doc.sh"),
path.join(BENCH, "agent", "speed_ref"), // default-config latency per job (scripts/speed_ref.py)
path.join(PROJ, ".horizon"), path.join(PROJ, "AGENTS.md"),
];
if (MODE === "repair") allowRead.push(path.join(JOB, "oracle"));

View File

@ -88,8 +88,6 @@ expect("url", e2e.judgeCommand("echo https://x.y/z"), "allow");
expect("read own rollout.json", e2e.judgeCommand("cat ../rollout.json"), "allow");
expect("ls ..(own rollout)", e2e.judgeCommand("ls .."), "allow");
expect("ls ../.. (rollouts dir)", e2e.judgeCommand("ls ../.."), "block");
expect("read speed reference", e2e.judgeCommand("cat " + B + "/agent/speed_ref/cls--m--nash-e--ptq--int8.json"), "allow");
expect("other agent files stay closed", e2e.judgeCommand("cat " + B + "/agent/SYSTEM.md"), "block");
expect("repair reads oracle yaml", repair.judgeCommand("cp ../../../oracle/stages/04-calibrate/compile.yaml ./compile.yaml"), "allow");
async function extensionCases() {

View File

@ -10,7 +10,7 @@
* 禁止直接调 hb_compile 等重命令(走 bench stage run);`sleep 100+`、`git push`。
*
* 白名单(e2e):workspace/(可写)· 本 rollout 根(agent/ stages/ handles/ rollout.json)· <job>/job.json · <job>/data
* · bench/data · bench/cache · bench/bin · bench/agent/oe-doc.sh · bench/agent/speed_ref · <proj>/.horizon · <proj>/AGENTS.md ;repair 额外 <job>/oracle。
* · bench/data · bench/cache · bench/bin · bench/agent/oe-doc.sh · <proj>/.horizon · <proj>/AGENTS.md ;repair 额外 <job>/oracle。
* 项目树内其余一律拒(别的 rollout、harness/、state/、export/、别的 job);项目树外只拒 ~/.pi、~/toolchains、bench/.env.local。
*
* 每次拒绝写一行到 <rollout>/agent/guard.log(L4 导出统计"越界尝试次数")。第二道保险是文件系统权限。
@ -83,7 +83,7 @@ log("init", JSON.stringify({ mode: MODE, deny_extra: DENY_EXTRA, allow_extra: AL
const FORBIDDEN_CMDS = ["hb_compile", "hb_mapper", "hb_verifier", "hrt_model_exec", "hb_perf",
"ssh", "scp", ...(PRIVATE_JAIL ? [] : ["rsync", "wget", "curl", "pip", "pip3", "conda", "npm"])];
const WRITE_PATTERNS: RegExp[] = [
/\bsleep\s+[0-9]{3,}/, // 用 terminal_exec wait_handle,不要 sleep 等编译
/\bsleep\s+[0-9]{3,}/, // 用 bench stage wait,不要 sleep 等编译
/\bgit\s+push\b/,
];

View File

@ -1,5 +1,5 @@
{
"_comment": "渲染到 ~/.pi/agent/models.json(chmod 600)。{TEACHER_API_KEY}/{STUDENT_*} 由 `bench agent setup` 从 bench/.env.local 填。字段名对照 pi 版本文档(P3 核对:providers.<name>.{baseUrl, api, apiKey, models[]})。contextWindow 故意写大:Pi 只在服务端报上下文超长时才压缩并续跑,不会在暂停退出前按估算先压缩(那样续跑的请求就和原来对不上)。",
"_comment": "渲染到 ~/.pi/agent/models.json(chmod 600)。{TEACHER_API_KEY}/{STUDENT_*} 由 `bench agent setup` 从 bench/.env.local 填。字段名对照 pi 版本文档(P3 核对:providers.<name>.{baseUrl, api, apiKey, models[]})。",
"providers": {
"dr-gateway": {
"baseUrl": "{TEACHER_BASE_URL}",
@ -11,7 +11,7 @@
"name": "{TEACHER_MODEL} (teacher)",
"reasoning": false,
"input": ["text"],
"contextWindow": 10000000,
"contextWindow": 128000,
"maxTokens": 16384,
"cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }
}
@ -27,7 +27,7 @@
"name": "student (vLLM)",
"reasoning": false,
"input": ["text"],
"contextWindow": 10000000,
"contextWindow": 65536,
"maxTokens": 8192,
"cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }
}

View File

@ -234,17 +234,8 @@ async function main(configFile) {
},
};
};
let compactedRetry = false;
session.subscribe(event => {
lastEvent = event.type;
let compaction;
if (event.type === 'compaction_end' && event.willRetry) {
// Pi compacted after a context overflow and resends: the failed generation's ID is spent. kept_messages is
// how many earlier messages the model still sees after the summary (export/trajectory.py builds that view).
generation += 1; compactedRetry = true;
compaction = {kept_messages: session.agent.state.messages
.filter(m => m.role !== 'compactionSummary' && !emptyProviderError(m)).length};
}
if (event.type === 'message_update') {
const delta = event.assistantMessageEvent;
if (delta?.type?.endsWith('_delta') && delta.delta) nonemptyDelta = true;
@ -269,14 +260,9 @@ async function main(configFile) {
activeTools.delete(event.toolCallId);
}
}
// message_update repeats the whole message twice (message, assistantMessageEvent.partial) on every
// token, ~800MB of events.jsonl per rollout; readers use only its type and delta, message_end has the text.
const written = event.type === 'message_update'
? {...event, message: undefined, assistantMessageEvent: {...event.assistantMessageEvent, partial: undefined}}
: event;
// Metadata decorates real native boundaries; it never adds conversation text.
output(boundary ? {...written, bench_tool: {...boundary, execution_id: config.execution_id,
epoch: config.epoch ?? 1, observed_monotonic: monotonic()}} : compaction ? {...written, bench_compaction: compaction} : written);
output(boundary ? {...event, bench_tool: {...boundary, execution_id: config.execution_id,
epoch: config.epoch ?? 1, observed_monotonic: monotonic()}} : event);
});
const stop = () => session.agent.abort();
process.on('SIGTERM', stop); process.on('SIGINT', stop);
@ -295,13 +281,6 @@ async function main(configFile) {
await session.prompt(fs.readFileSync(config.prompt_path, 'utf8'));
}
await session._agentEventQueue;
// Pi resumes 100 ms after an overflow compaction (agent-session.js _runAutoCompaction): wait for that run too.
while (compactedRetry) {
compactedRetry = false;
await new Promise(resolve => setTimeout(resolve, 300));
await session.agent.waitForIdle();
await session._agentEventQueue;
}
const last = session.state.messages.at(-1);
const complete = last?.role === 'assistant' && last.stopReason === 'stop' && closed(session.state.messages);
const safeResume = !nonemptyDelta && emptyProviderError(last) && closed(session.state.messages) &&

View File

@ -3,18 +3,26 @@
job:`{job_id}` run:`{run_id}`
job.json:`{job_json}`(只读) 数据:`{data_dir}`(只读) 你的可写区:当前目录
## 验收(以 job.json 的 `gates.required` 和阈值为准)
1. G1 编译:产出 `.hbm`,march = {march},输入 {input_spec},`hb_verifier` 通过
2. G2 数值:校准外样本的逐输出余弦,参考阈值 {g2_threshold},只作诊断
3. G3 任务精度:板上 T1 子集({t1_n} 张)的 {g3_metric} 与同一子集的 FP32 基线比较,容差 {g3_tol}({g3_kind})
4. G4 速度:不比平台默认量化配置(compile.yaml 只写两项必填)编出的模型慢 1.5 倍以上。参考在 `$BENCH_ROOT/agent/speed_ref/{job_id}.json`:`latency_ms_est` 是默认配置的编译器静态估计,不含落到 CPU 的算子;文件还没有表示还在算,值为 null 表示默认配置没编出来、本题没有参考。
## 目标
1. G1 compile:产出 `.hbm`,march = {march},输入 {input_spec},`hb_verifier` 通过
2. G2 numeric:记录校准外样本逐输出余弦,参考阈值 {g2_threshold},用于诊断
3. G3 task:真实板端 T1 的 {g3_metric} 与同一子集 FP32 基线比较;容差 {g3_tol}({g3_kind}),以 job.json 的判定规则为准
4. G4 perf:只记录,不判
缺少板上精度结果不能报完整 PASS;缺权重、工具或材料时如实保留实际错误。
必选项以 job.json 的 `gates.required` 为准。
## 输入和可复用缓存
- 权重、模型身份和输入约定见 job.json 的 `model_spec`;`bench stage run fetch` 获取正式权重。
- FP32 基线:`bench stage run baseline` 生成,缓存在 `{baseline_dir}/metrics.json`(可能还没有,或属于旧子集)。
- `bench stage run export` 是平台参考导出:图不给你,只用来核对你的图的输入输出。
- 校准图片在开发环境的 `$BENCH_CALIB_IMAGES`({calib_n} 张);T1 评测子集 {t1_n} 张,不对你开放。
- 权重来源、模型身份和输入约定:见 job.json 的 model_spec
- ONNX 缓存位置:`{onnx}`(可能尚未生成)
- FP32 基线缓存位置:`{baseline_dir}/metrics.json`(可能尚未生成或属于旧子集)
- 校准集位置:`{calib_dir}`(目标 {calib_n} 个 npy,与 val 零交集)
- T1 评测子集:`{t1_tar}`({t1_n} 张)
预算:本轨迹最多启动 5 次编译(排队超时没开始的不算,续跑不清零),用完后再提交编译会返回 `COMPILE_BUDGET_EXHAUSTED`,此时用最后一次产物跑完 board/eval。{max_turns} 轮、{max_tokens} tokens;等待阶段完成不计轮数,每次模型生成计一轮。
## 流程
阅读 job.json 后按需要调用工具:`bench stage run fetch` 获取权重,`bench stage run export` 导出图,`bench stage run baseline` 生成参考基线。已有有效缓存可直接复用;这些操作没有强制的探索顺序。
图与校准输入可用时,用 `bench stage run compile --config <你的>.yaml` 尝试编译。各阶段返回 handle,使用其结果前先调用 `terminal_exec({"wait_handle": "<handle>"})` 等待完成,再读取实际日志。`hbm` 收集编译结果;`board` 记录性能;`eval` 需要 HBM 和独立基线,执行固定板端精度评测。失败后可继续改图、改配置或更换方法,但本轨迹最多启动 5 次编译(首次 1 次 + 重试 4 次);精度不达标后的重编译也计入,排队超时未开始的不计,续跑不清零。用完后再提交编译会直接返回 `COMPILE_BUDGET_EXHAUSTED`,此时用最后一次产物完成 board/eval 并如实报告。
最后自然语言如实报告。缺少真实板端精度不能报完整 PASS;缺权重、工具或材料时保留实际错误,不能把材料缺口说成量化失败。
预算:{max_turns} 轮、{max_tokens} tokens。后台编译的等待时间不计模型轮数,每次模型生成仍计一轮。预算收尾时停止新工作并给出实际状态。

View File

@ -1,7 +1,8 @@
{
"_comment": "渲染到 <rollout>/workspace/.pi/settings.json。{PROJECT_ROOT}/{BENCH_ROOT}/{MODEL} 由 harness/rollout.py 替换。上下文超长时 Pi 自己压缩后接着跑(默认开);导出按 compactions 还原模型实际看到的内容(export/trajectory.py)。",
"_comment": "渲染到 <rollout>/workspace/.pi/settings.json。{PROJECT_ROOT}/{BENCH_ROOT}/{MODEL} 由 harness/rollout.py 替换。compaction 关:训练数据里出现 CompactionSummaryMessage 是脏的,上下文靠 max_turns 控制。",
"defaultProvider": "dr-gateway",
"defaultModel": "{MODEL}",
"compaction": { "enabled": false },
"retry": { "enabled": false, "provider": { "maxRetries": 0 } },
"skills": [
"{PROJECT_ROOT}/.horizon/skills/*/*",

View File

@ -6,8 +6,7 @@
"job_suffix": "--nash-m--ptq--int8",
"protocol": {
"rollouts_per_job": 2,
"count_rollouts_started_after": "2026-09-29T00:00:00+08:00",
"count_rule_note": "2026-10-12 user decision: base-Qwen PASS/FAIL from all of r20260929-04 counts, incl. results before 2026-10-08 07:30.",
"count_rollouts_started_after": "2026-10-08T07:30:00+08:00",
"harness_commit": "10b070e",
"agent_version": 15,
"system_interrupted": "not a model failure; excluded and rerun",

View File

@ -4,7 +4,6 @@ import argparse
import json
import os
import re
import shlex
import sys
import time
import uuid
@ -52,38 +51,33 @@ def submit(stage, config, poll_hint=60, *, recipe=None, submission=None):
tmp.rename(hd / 'request.json')
print(hid)
print('# Pi wait: terminal_exec(' + json.dumps({'wait_handle': hid}) + ')')
print(f'# wait: bench stage wait {hid} --poll {poll_hint}')
return hid
def dev(command, timeout=3600, cwd=None):
"""bench dev: run one command in the development environment (Salty container with the toolchain)."""
rd = rollout_dir()
ws = (rd / 'workspace').resolve()
here = Path.cwd().resolve()
cwd = cwd or (str(here.relative_to(ws)) if here.is_relative_to(ws) else '.')
hid = 'terminal-' + uuid.uuid4().hex
hd = rd / 'handles' / hid
hd.mkdir()
(hd / 'status.json').write_text(json.dumps({'state': 'QUEUED', 'queued_at': time.time()}))
tmp = hd / 'request.tmp'
tmp.write_text(json.dumps({'stage': 'terminal', 'target': 'development', 'command': command,
'timeout': timeout, 'cwd': cwd}))
tmp.rename(hd / 'request.json')
print(hid)
print('# Pi wait: terminal_exec(' + json.dumps({'wait_handle': hid}) + ')')
return hid
def wait(handle: str, poll: int = 60) -> int:
"""不再轮询,看一眼就返回:结束了打印结果;没结束返回 2,并指向 terminal_exec wait_handle(一次等到阶段结束、不占轮数)。
10-16:efficientvit.b0 400 轮里 307 轮是 bench stage wait,每次还往上下文里加一段输出。"""
def wait(handle: str, poll: int = 60, quiet: bool = False, max_wait_s: float = 60) -> int:
"""单次最多等 60 秒;返回 2 表示仍在排队/运行,阶段继续执行。"""
if not 0 < max_wait_s <= 60:
raise ValueError("stage wait budget must be in (0, 60] seconds")
hd = rollout_dir() / "handles" / handle
if not hd.exists():
raise SystemExit(f"handle 不存在:{handle}")
st = json.loads((hd / "status.json").read_text())
if st.get("state") != "DONE":
print(f"{handle}: {st.get('state')}"
f"{'(阶段计算已结束,等待数据库登记;无需重跑)' if st.get('registration') == 'PENDING' else ''}"
f"。不要反复查:调用 terminal_exec({json.dumps({'wait_handle': handle})}),它一次等到这个阶段结束,不占轮数。")
return 2
t0 = time.time()
deadline = time.monotonic() + max_wait_s
while True:
st = json.loads((hd / "status.json").read_text())
if st.get("state") == "DONE":
break
if not quiet:
since = st.get("started_at") or st.get("queued_at") or t0
print(f"[{time.strftime('%H:%M:%S')}] {handle}: {st.get('state')}"
f"{'(等 ' + st.get('pool', '?') + ' 池槽位)' if st.get('state') == 'QUEUED' else ''}"
f"{'(阶段计算已结束,等待数据库登记;无需重跑)' if st.get('registration') == 'PENDING' else ''}"
f" 已 {int(time.time() - since)}s", flush=True)
remaining = deadline - time.monotonic()
if remaining <= 0:
print(f"(已等 {int(max_wait_s - remaining)}s 仍未结束;这不是失败,再调一次 bench stage wait {handle} 继续等)")
return 2
time.sleep(min(max(1, poll), remaining))
_print_status(st)
return 0 if st["result"] in ("OK", "SKIPPED") else 1
@ -100,9 +94,6 @@ def _print_status(st: dict) -> None:
print(json.dumps(st)); return
print(f"result: {st['result']} exit_code: {st['exit_code']} error_class: {st['error_class']} {st['duration_s']}s")
print(f"stdout: {st['stdout']}\nstderr: {st['stderr']}\nmeta: {st['meta']}")
for name in ('stdout_tail', 'stderr_tail'):
if st.get(name):
print(f"--- {name} ---\n{st[name]}")
if st.get("measurements"):
print("measurements: " + json.dumps(st["measurements"], ensure_ascii=False))
if st["result"] != "OK":
@ -114,20 +105,6 @@ def _print_status(st: dict) -> None:
def main():
if sys.argv[1:2] == ['dev']:
p = argparse.ArgumentParser(prog='bench dev', description='Run a command in the development environment')
p.add_argument('--timeout', type=int, default=3600, help='seconds, at most 43200')
p.add_argument('--cwd', help='workspace-relative directory; default: the current one')
p.add_argument('--board', action='store_true', help='run on the task board instead (S100P tasks): this directory, '
'with files that only exist in the development workspace such as a compiled HBM, at most 30 min')
p.add_argument('command', nargs=argparse.REMAINDER)
a = p.parse_args(sys.argv[2:])
words = a.command[1:] if a.command[:1] == ['--'] else a.command
if not words:
p.error('command required')
command = words[0] if len(words) == 1 else shlex.join(words)
dev(('#bench:board\n' if a.board else '') + command, a.timeout, a.cwd) # harness/terminal.py BOARD_MARK
return 0
p = argparse.ArgumentParser()
p.add_argument('command', choices=['stage'])
p.add_argument('op', choices=['run', 'wait', 'status'])

View File

@ -1,93 +0,0 @@
# 经验记忆
这里是在本项目上工作的 agent(Claude、Codex 等)积累下来的经验:用户的偏好和纠正、项目现场的事实、踩过的坑。代码和 git 历史里能直接查到的东西不放在这里。
## 怎么用
- 开始工作前先扫一遍下面的索引,按每行的提示挑相关的文件读。
- 记忆只代表写下那一刻的情况。里面提到的路径、进程、配置、机器,用之前先到现场核实;和现场不符时以现场为准,并更新或删掉那条记忆。
## 怎么写
1. 一个文件只记一件事。文件名用英文小写加短横线,例如 `bench-restart-hazards.md`。
2. 文件开头写 frontmatter:
```markdown
---
name: <和文件名相同,不带 .md>
description: <一句话,别人靠它判断要不要读这个文件>
metadata:
type: user | feedback | project | reference
---
```
| type | 记什么 |
| --- | --- |
| user | 用户是谁、懂什么、偏好什么 |
| feedback | 用户的要求和纠正;做对了被肯定的做法也记 |
| project | 进行中的工作、现场事实、约束,代码里看不出来的 |
| reference | 外部资源在哪:地址、看板、工单 |
3. 正文先写事实。feedback 和 project 类再加两行:`**Why:**` 写原因,`**How to apply:**` 写以后具体怎么做。
4. 时间一律写绝对日期(2026-10-11),不写"昨天""下周"。
5. 相关的记忆用 `[[name]]` 互相链接。目标文件还没写也可以先留着链接。
6. 新写之前先查有没有已经覆盖这件事的文件。有就改那个文件,不要另起一份。发现某条记错了,就地改正并注明"更正(日期)",或者整个删掉。
7. 写完在下面的索引里加一行:`- [标题](文件名.md) — 一句话提示`。索引只放链接和提示,不放内容。
## 不能写的
- 密码、API key、token、私钥,连前几位、后几位都不写。只写变量名或存放位置。
- 代码结构、git 历史,以及 README、CONTRACT、HANDBOOK 里已经写了的东西。
- 只对当前这一次对话有用的临时状态。
## 存放和同步
- 本目录 `docs/memory/` 是共享的记忆库,跟着 `feature/salty-20261009` 分支提交和推送。不要推到 `main` 或生产分支。
- Claude 在本机另有一份工作副本(`~/.claude/projects/<项目>/memory/`,索引文件叫 `MEMORY.md`)。两边内容保持一致:改了一边,同步到另一边再提交。
- 推送前 grep 一遍新增内容,确认没有密钥或密钥片段。
## 索引
- [优先做减法而非加法](prefer-subtraction-over-addition.md) — 修 bug 先想能删什么,别动辄加钩子和配置项
- [汇报说人话](plain-language-no-jargon.md) — 不要用金丝雀这类行话,先给结论和数字
- [bench 调度程序重启的坑](bench-restart-hazards.md) — 改文件不算换代、kill -9 条件、drain 单向、密钥在旧进程环境、启动要 10–20 分钟、导出会卡、全局并发上限要三个一起换、新账号用新组名
- [第一版交付包](delivery-package-v1.md) — 位置、冻结点 62 条、出第二版要避开的坑
- [采集稳定性补丁](quota-unknown-autorecover-fix.md) — 2f54d40→ea0335b:请求不明、板卡锁死、板端隔离、ssh 断连、回执误判、编译上限 5 次、请求结束后补放行试探、bash 超时可恢复、GLM tpm 50万→200万、连坐只挡预算用完和真跑满24h
- [成功率低的真实原因](success-rate-breakdown-20261008.md) — 10-08:真做不到的只有 2 条;GLM/Qwen 同题打平是考卷区分不出(简单题满分、难题被砍或判分卡住)
- [轨迹时间花在哪](rollout-time-breakdown-20261008.md) — 10-08:编译占 33%、编译机 CPU 只用 31%、排队中位 30 秒;04 时长是重复计算;编译上云暂无收益
- [编译机要被回收](compile-server-reclaim.md) — 10-08 晚已回收、换到 36.144.20.182;账本只有 9-28 备份;10-11 起 36 能读写网盘上的旧轨迹;120 已关机清理
- [第二个 GLM 账号与提并发](glm-second-account-20261008.md) — 10-08:05/06 原来同一账号;06 换新账号 B,05/06 各 16 路,04 不变;接续脚本位置
- [Qwen SFT benchmark](qwen-sft-benchmark.md) — 10-08 建 qwen-sft-v1:37 题;04 的 Qwen 轨迹只做训练前基准、不进 SFT;10-12 起 04 的旧结果也算,训练前过 20 败 3、14 题没结果
- [仓库主线整理](git-mainline-20261009.md) — 10-09:main=生产分支 c2c52ed,旧 main 打标签;Salty 在 feature/salty-20261009;本地还剩优雅退出/板卡改动
- [Salty 个人空间迁移](salty-personal-migration-20261009.md) — 10-10 完成:120 的 bench-cache/data 全在 personal:/bench/,逐文件核对通过;Salty 单文件上限 7.3–12.7G、拒软链接、du 会夸大
- [别原地改运行中的脚本](never-edit-running-script.md) — bash 边读边执行;要改就写新文件或新文件+mv 替换
- [Mac mini 主控 + Salty 全阶段执行](mini-salty-controller-20261009.md) — 10-09 至 10-12:salty_remote 设计与踩坑(工作目录、HBM 取回、网络抖动、容器内存超限、重启恢复)、轨迹搬到个人空间、导出挪到 Salty、强停写坏执行文件的处理、各处位置
- [记忆同步到仓库](memory-synced-to-repo.md) — 改完记忆同步到 docs/memory/ 并推 salty 分支;入口 docs/memory/AGENTS.md
- [04 的 Qwen 服务速度](qwen-nvfp4-service-speed-20261012.md) — 10-12:单请求中位 6 秒、一次只处理 1 个请求;04 开 5 路只忙 15%,不是瓶颈
- [迁到 Salty 后通过数暴跌](salty-throughput-drop-20261012.md) — 10-12 清查:主因是剩下全是重题(同模型在 Salty 不更慢);系统问题:整批重启、容器被杀占编译次数、连不上 GLM 永不自动恢复、metric3d、Wi-Fi
- [自愈与报警](selfheal-and-alert-20261012.md) — 10-12:系统失败(被杀、执行器退出、超时、卡死)不占编译次数;连不上自动续,但暂停名额仍会被抢;depthpro/metric3d 撤出;健康检查弹通知
- [容器线程被限流](salty-container-threads-20261013.md) — 10-13:容器看到 244 核只给 4–6 核,torch 慢 16 倍、numpy 5 倍;已加线程上限,onnxruntime 未处理
- [验证器核查](verifier-audit-20261012.md) — 10-12:Qwen 20 个通过是真的;但验证器不查是否真 int8(全 int16 也算过)、只抽 100 张;打平是因为系统包办了大半
- [10-13 四个基建漏洞](infra-leaks-20261013.md) — 超时步骤锁死 CPU 名额(32 剩 6)、超时后 Salty 容器不停、续跑必死、领题后重复查额度把题作废;10-13 20:41 重启后全部修复上线,22:26 导出进程加自动清理孤儿容器
- [训练前 Qwen 能力失败题](qwen-ability-failures-20261013.md) — 10-14:34/37 有结果、过 28、能力失败 6 道;短板:配置写错、不先本地诊断、归咎平台
- [Mac mini 流量绕美国](mac-mini-clash-global-proxy-20261013.md) — 10-13:Clash 全局+TUN,Qwen/Salty/GLM 全走洛杉矶 JMS 节点,是断线和 Salty 网络错误的共同根因;要用户改 Clash
- [HBM 步骤成新瓶颈](hbm-step-bottleneck-20261014.md) — 10-14:HBM≡BC 模拟器抽查在 Salty 上 1–2 小时、一夜撞时限 45 次,从没独立判出过问题;DNS_FAILED 暂停不会自动续
- [06-hbm 抽查改为按需](hbm-spotcheck-optional-20261014.md) — 10-14:模拟器抽查不再必跑(已部署);agent 可用配置开启、只作诊断;配置校验/说明与 DNS 续跑等下次重启
- [网络容错与板卡时限](net-resilience-board-deadline-20261014.md) — 10-14:原地重发/免费续跑/结果不明可续跑;板卡时限对齐;19:00 连同开发环境/自带图/速度要求一起重启上线(-v16 配置、脚本在 restart-20261014)
- [智能体自主性核查](agent-autonomy-audit-20261014.md) — 10-14:不让改图、兜底配置多、VM 无工具链;已实现 bench dev(Salty 沙箱)+ 自带图/校准(a92510d,等重启);速度 1.5 倍门槛离线重判 06 翻 37/178;上线 2 小时内还没人自带图,dev 曾能看到评测集(已修)
- [10-15 早上复查](overnight-check-20261015.md) — dev 槽 8→37(配置有缓存,只对新命令生效);S600 板盘满因失败会话目录不删(a263374 已修);暂停名额被新题抢;全 fp16 落 CPU 仍判过;4 条自带图/校准;评测文件被 task_contract 冻结,在跑的 run 不能热改(11:08 改坏三条)
- [v16 轨迹质量对比](v16-quality-20261015.md) — 10-15:dev 52/53 在用、兜底和慢速通过变少但样本少;240 轮上限截断最自主的轨迹;Qwen 3过3败是不再用全 int16 兜底,基准要在 v16 下重测
- [10-15 下午重启](restart-20261015.md) — 重启前清查:回执 15 秒超时、暂停吃掉 24 小时、400 永久暂停、账本忙让出名额、评测一次 12 GB;v17(400 轮)、判分 v0.8;之后热部署:评测分批靠批准的等价改写(不建新 run)、加载卡住检测、容器 onnxruntime 限线程、Salty 失败作业自动清
- [最新三条 SFT 细读](sft-sample-review-20261015.md) — 10-15:真通过但决策含量低;两条是 Qwen 自己的不该进训练;v16 仅 8/315、含评测题、上下文过长
- [v17 自主性实测](v17-autonomy-check-20261015.md) — 10-15 19:40:32 条 GLM 跑 5 小时 0 条结束;自己导图 0、自备校准 0、图手术 1;72% 时间在开发环境,9 条各花 4 小时编默认配置
- [v18–v22 整条链路给 agent](v18-own-inputs-20261015.md) — 自己导图/校准(v21 原图)、速度参考(编不出写 unavailable)、板上自测(≤512 MiB);10-16 16:21 重启:上下文满记预算用完、断线只停 2 分钟、Qwen 省去旧工具输出、GLM token 2M、stage wait 不轮询
- [10-16 17:00 复查](check-20261016-1700.md) — v18–21 自己导图 41/46、真改图约 3 条、0 条通过;05 的 GLM S600 账号 16:00 起全 429,17:22 换 key 和新组名(s600-r05-v22c)恢复
- [PASS/SFT 自主性统计](sft-autonomy-tally-20261016.md) — 10-16:GLM 310 条 PASS 里 71% 只写配置,成品真用自己改的图只有 4 条(3 条进 SFT);v18+ 还没通过;文本搜 onnx_model 会误报
- [第二版 0 通过的原因](v18-zero-pass-why-20261016.md) — 10-16:重启取消 42/90、82/90 是从没过的难题、到评测的都差精度;10 条想找评测集做本地验证被挡、28 条照搬平台导出脚本
- [稳定性体检](stability-sweep-20261016.md) — 10-16:最大问题是生产批次当试验场(3 天 8 版、重启即取消);19:00 复核更正:本机有健康检查但漏报限流、4000 小时实为约 1660、盘不是急事、Qwen 进 SFT/版本混杂不用改代码
- [10-16 断电停机与重启](poweroff-20261016.md) — 停/起/清理脚本在 configs/poweroff-20261016;开机 VM 自起;取消会留 dev lease 占槽;speed_ref 残留要删
- [回复用中文](reply-in-chinese.md) — 汇报、提问一律中文;命令和路径保持原样
- [10-17 02:15 底层复查](check-20261017-0215.md) — v18+ 134 条:真改图约 10、板上自测 6、通过 3(1 条板上 493 ms 没被速度挡住);平台复测不判速度;重启后很稳,只剩 Qwen 续跑必死(context-mask 状态在内存)
- [能否当强化学习环境](rl-readiness-20261017.md) — 10-17:能用,瓶颈是慢不是稳;收尾 5 件后冻结,清单外默认不再改
- [少管过程的删减清单](simplify-agent-rules-20261017.md) — 10-17:交还 Pi 压缩要改 5 处(不是纯减法,导出也要认压缩);提示词逐条分留事实/删规矩/交给系统;并入冻结版本
- [10-17 停 06](stop-06-20261017.md) — 账号 B 没额度,11:30 只 SIGTERM 不取消;原命令/环境在 configs/stop-06-20261017;换 key 要新组名并取消暂停的

View File

@ -1,42 +0,0 @@
- [优先做减法而非加法](prefer-subtraction-over-addition.md) — 修 bug 先想能删什么,别动辄加钩子和配置项
- [汇报说人话](plain-language-no-jargon.md) — 不要用金丝雀这类行话,先给结论和数字
- [bench 调度程序重启的坑](bench-restart-hazards.md) — 改文件不算换代、kill -9 条件、drain 单向、密钥在旧进程环境、启动要 10–20 分钟、导出会卡、全局并发上限要三个一起换、新账号用新组名
- [第一版交付包](delivery-package-v1.md) — 位置、冻结点 62 条、出第二版要避开的坑
- [采集稳定性补丁](quota-unknown-autorecover-fix.md) — 2f54d40→ea0335b:请求不明、板卡锁死、板端隔离、ssh 断连、回执误判、编译上限 5 次、请求结束后补放行试探、bash 超时可恢复、GLM tpm 50万→200万、连坐只挡预算用完和真跑满24h
- [成功率低的真实原因](success-rate-breakdown-20261008.md) — 10-08:真做不到的只有 2 条;GLM/Qwen 同题打平是考卷区分不出(简单题满分、难题被砍或判分卡住)
- [轨迹时间花在哪](rollout-time-breakdown-20261008.md) — 10-08:编译占 33%、编译机 CPU 只用 31%、排队中位 30 秒;04 时长是重复计算;编译上云暂无收益
- [编译机要被回收](compile-server-reclaim.md) — 10-08 晚已回收、换到 36.144.20.182;账本只有 9-28 备份;10-11 起 36 能读写网盘上的旧轨迹;120 已关机清理
- [第二个 GLM 账号与提并发](glm-second-account-20261008.md) — 10-08:05/06 原来同一账号;06 换新账号 B,05/06 各 16 路,04 不变;接续脚本位置
- [Qwen SFT benchmark](qwen-sft-benchmark.md) — 10-08 建 qwen-sft-v1:37 题;04 的 Qwen 轨迹只做训练前基准、不进 SFT;10-12 起 04 的旧结果也算,训练前过 20 败 3、14 题没结果
- [仓库主线整理](git-mainline-20261009.md) — 10-09:main=生产分支 c2c52ed,旧 main 打标签;Salty 在 feature/salty-20261009;本地还剩优雅退出/板卡改动
- [Salty 个人空间迁移](salty-personal-migration-20261009.md) — 10-10 完成:120 的 bench-cache/data 全在 personal:/bench/,逐文件核对通过;Salty 单文件上限 7.3–12.7G、拒软链接、du 会夸大
- [别原地改运行中的脚本](never-edit-running-script.md) — bash 边读边执行;要改就写新文件或新文件+mv 替换
- [Mac mini 主控 + Salty 全阶段执行](mini-salty-controller-20261009.md) — 10-09 至 10-12:salty_remote 设计与踩坑(工作目录、HBM 取回、网络抖动、容器内存超限、重启恢复)、轨迹搬到个人空间、导出挪到 Salty、强停写坏执行文件的处理、各处位置
- [记忆同步到仓库](memory-synced-to-repo.md) — 改完记忆同步到 docs/memory/ 并推 salty 分支;入口 docs/memory/AGENTS.md
- [04 的 Qwen 服务速度](qwen-nvfp4-service-speed-20261012.md) — 10-12:单请求中位 6 秒、一次只处理 1 个请求;04 开 5 路只忙 15%,不是瓶颈
- [迁到 Salty 后通过数暴跌](salty-throughput-drop-20261012.md) — 10-12 清查:主因是剩下全是重题(同模型在 Salty 不更慢);系统问题:整批重启、容器被杀占编译次数、连不上 GLM 永不自动恢复、metric3d、Wi-Fi
- [自愈与报警](selfheal-and-alert-20261012.md) — 10-12:系统失败(被杀、执行器退出、超时、卡死)不占编译次数;连不上自动续,但暂停名额仍会被抢;depthpro/metric3d 撤出;健康检查弹通知
- [容器线程被限流](salty-container-threads-20261013.md) — 10-13:容器看到 244 核只给 4–6 核,torch 慢 16 倍、numpy 5 倍;已加线程上限,onnxruntime 未处理
- [验证器核查](verifier-audit-20261012.md) — 10-12:Qwen 20 个通过是真的;但验证器不查是否真 int8(全 int16 也算过)、只抽 100 张;打平是因为系统包办了大半
- [10-13 四个基建漏洞](infra-leaks-20261013.md) — 超时步骤锁死 CPU 名额(32 剩 6)、超时后 Salty 容器不停、续跑必死、领题后重复查额度把题作废;10-13 20:41 重启后全部修复上线,22:26 导出进程加自动清理孤儿容器
- [训练前 Qwen 能力失败题](qwen-ability-failures-20261013.md) — 10-14:34/37 有结果、过 28、能力失败 6 道;短板:配置写错、不先本地诊断、归咎平台
- [Mac mini 流量绕美国](mac-mini-clash-global-proxy-20261013.md) — 10-13:Clash 全局+TUN,Qwen/Salty/GLM 全走洛杉矶 JMS 节点,是断线和 Salty 网络错误的共同根因;要用户改 Clash
- [HBM 步骤成新瓶颈](hbm-step-bottleneck-20261014.md) — 10-14:HBM≡BC 模拟器抽查在 Salty 上 1–2 小时、一夜撞时限 45 次,从没独立判出过问题;DNS_FAILED 暂停不会自动续
- [06-hbm 抽查改为按需](hbm-spotcheck-optional-20261014.md) — 10-14:模拟器抽查不再必跑(已部署);agent 可用配置开启、只作诊断;配置校验/说明与 DNS 续跑等下次重启
- [网络容错与板卡时限](net-resilience-board-deadline-20261014.md) — 10-14:原地重发/免费续跑/结果不明可续跑;板卡时限对齐;19:00 连同开发环境/自带图/速度要求一起重启上线(-v16 配置、脚本在 restart-20261014)
- [智能体自主性核查](agent-autonomy-audit-20261014.md) — 10-14:不让改图、兜底配置多、VM 无工具链;已实现 bench dev(Salty 沙箱)+ 自带图/校准(a92510d,等重启);速度 1.5 倍门槛离线重判 06 翻 37/178;上线 2 小时内还没人自带图,dev 曾能看到评测集(已修)
- [10-15 早上复查](overnight-check-20261015.md) — dev 槽 8→37(配置有缓存,只对新命令生效);S600 板盘满因失败会话目录不删(a263374 已修);暂停名额被新题抢;全 fp16 落 CPU 仍判过;4 条自带图/校准;评测文件被 task_contract 冻结,在跑的 run 不能热改(11:08 改坏三条)
- [v16 轨迹质量对比](v16-quality-20261015.md) — 10-15:dev 52/53 在用、兜底和慢速通过变少但样本少;240 轮上限截断最自主的轨迹;Qwen 3过3败是不再用全 int16 兜底,基准要在 v16 下重测
- [10-15 下午重启](restart-20261015.md) — 重启前清查:回执 15 秒超时、暂停吃掉 24 小时、400 永久暂停、账本忙让出名额、评测一次 12 GB;v17(400 轮)、判分 v0.8;之后热部署:评测分批靠批准的等价改写(不建新 run)、加载卡住检测、容器 onnxruntime 限线程、Salty 失败作业自动清
- [最新三条 SFT 细读](sft-sample-review-20261015.md) — 10-15:真通过但决策含量低;两条是 Qwen 自己的不该进训练;v16 仅 8/315、含评测题、上下文过长
- [v17 自主性实测](v17-autonomy-check-20261015.md) — 10-15 19:40:32 条 GLM 跑 5 小时 0 条结束;自己导图 0、自备校准 0、图手术 1;72% 时间在开发环境,9 条各花 4 小时编默认配置
- [v18–v22 整条链路给 agent](v18-own-inputs-20261015.md) — 自己导图/校准(v21 原图)、速度参考(编不出写 unavailable)、板上自测(≤512 MiB);10-16 16:21 重启:上下文满记预算用完、断线只停 2 分钟、Qwen 省去旧工具输出、GLM token 2M、stage wait 不轮询
- [10-16 17:00 复查](check-20261016-1700.md) — v18–21 自己导图 41/46、真改图约 3 条、0 条通过;05 的 GLM S600 账号 16:00 起全 429,17:22 换 key 和新组名(s600-r05-v22c)恢复
- [PASS/SFT 自主性统计](sft-autonomy-tally-20261016.md) — 10-16:GLM 310 条 PASS 里 71% 只写配置,成品真用自己改的图只有 4 条(3 条进 SFT);v18+ 还没通过;文本搜 onnx_model 会误报
- [第二版 0 通过的原因](v18-zero-pass-why-20261016.md) — 10-16:重启取消 42/90、82/90 是从没过的难题、到评测的都差精度;10 条想找评测集做本地验证被挡、28 条照搬平台导出脚本
- [稳定性体检](stability-sweep-20261016.md) — 10-16:最大问题是生产批次当试验场(3 天 8 版、重启即取消);19:00 复核更正:本机有健康检查但漏报限流、4000 小时实为约 1660、盘不是急事、Qwen 进 SFT/版本混杂不用改代码
- [10-16 断电停机与重启](poweroff-20261016.md) — 停/起/清理脚本在 configs/poweroff-20261016;开机 VM 自起;取消会留 dev lease 占槽;speed_ref 残留要删
- [回复用中文](reply-in-chinese.md) — 汇报、提问一律中文;命令和路径保持原样
- [10-17 02:15 底层复查](check-20261017-0215.md) — v18+ 134 条:真改图约 10、板上自测 6、通过 3(1 条板上 493 ms 没被速度挡住);平台复测不判速度;重启后很稳,只剩 Qwen 续跑必死(context-mask 状态在内存)
- [能否当强化学习环境](rl-readiness-20261017.md) — 10-17:能用,瓶颈是慢不是稳;收尾 5 件后冻结,清单外默认不再改
- [10-17 停 06](stop-06-20261017.md) — 账号 B 没额度,11:30 只 SIGTERM 不取消;原命令/环境在 configs/stop-06-20261017;换 key 要新组名并取消暂停的

View File

@ -1,129 +0,0 @@
---
name: agent-autonomy-audit-20261014
description: "10-14 核查智能体自主性:04/05/06 共 627 题都不让改图(ONNX 钉死),难题被堵死;通过的配置多是\"全换 int16 / 全换 fp16\"的兜底,按模型诊断的只有约一成;照抄模板不是主因"
metadata:
node_type: memory
type: project
originSessionId: 2b85e929-b893-42c3-a7a0-3a72bda14705
modified: 2026-10-14T08:58:17.164Z
---
用户问:系统包办 + 模板照抄,是不是让智能体体现不出能力(实际工作要自己导出 ONNX、拆算子)。2026-10-14 查代码和会话的结论:
**1. 真正的硬限制是不能改图**
- compile 用的 ONNX 由平台钉死:ptq.compile_config 覆盖 model_parameters.onnx_model。agent 只能改 compile.yaml 的三段(stageworker.snapshot_config 白名单),custom_op 之类会被拒。
- 代码里其实有"提交自己的图/导出代码"的入口:`bench stage run export --recipe`,见 docs/extensions.md、harness/recipes.py。但它要求 job 里有 model_recipe.reference,而且只支持 PTQ 语义分割。04/05/06 的 627 道题一道都没开。
- 盘上会话里有 11 条失败轨迹试过 `--recipe`,都被拒("job must freeze a model_recipe reference policy")。
- GLM 在 05 有 24 题、06 有 26 题从没通过,约一半连一次编译都没成功,正好是要改图的题:
- mask2former:Einsum 算子不支持;
- flownet2:resample 输出的形状推不出来;
- unidepth:解码器是 float64;
- depthpro:导出失败;
- 还有 aliked、keynet、detr。
- GLM 的报告诊断得很准,最后都写"没有任何改图入口"。flownet2 那条怀疑是 harness 自带的 vendor/flownet2_cpu_ops.py 里的 resample 有问题,**还没核实**。
**2. 照抄模板不是主因,主因是兜底配置**
- 374 次写配置,没有一次照抄模板里的示例节点名。
- 通过轨迹最终用的配置(还在盘上的样本):
- 06(S100P int8):43 条里 24 条全模型 int16,都是先 int8 失败、平均第 3 次编译才换的;只有 4 条按节点调。
- 05(S600 fp16int8):44 条里约 28 条是"除卷积外全 fp16",有的连卷积也 int16、完全没有 int8;按节点调的约 6 条。
- 04(Qwen):32 条里 14 条全 int16。
- 验证器不查 int8 比例(见 [[verifier-audit-20261012]]),所以这些都判通过。SFT 会学到"失败就全换高精度"。
- 一次编译就通过的比例:GLM-06 111/186,GLM-05 76/109,Qwen-04 31/66。通过的题里分类题最多(06 有 94/186)。
**3. 最像智能体的行为都在失败轨迹里**:本地复现、克隆官方源码比对、定位算子。这些轨迹被平台堵死后判 FAIL,进不了训练数据。
**4. 给用户的改法(10-14,等拍板)**
开放改图,走窄路,不推广现有的配方框架:
- 配方框架要每题登记参考源码哈希,只支持分割题,评测还要跑 agent 写的前后处理,太重。
- 窄路是 compile.yaml 允许写 `model_parameters.onnx_model: <workspace 内 .onnx>`,要求输入输出的个数、顺序、形状与平台 ONNX 一致。这样基线、上板、评测都不用改。已核实:quant 评测不读平台 ONNX,基线来自 torch 参考。
- 要改的地方:
- stageworker.snapshot_config:允许这个字段,用 recipes._stream 复制进受保护的队列目录;
- ptq.compile_config:用这份图;
- salty_remote._bundle:把图带进容器;
- s04:在容器里比对输入输出,不一致就直接失败,不占编译次数,同时记录图的来源和 sha;
- SYSTEM.md。
- 前两处和 SYSTEM.md 要重启才生效,s04 部署即生效。
- 开放后能做:本地导出、改写算子、交预量化模型(calibration_type: skip)、等价改权重。
- 风险:测试集 agent 可读,改过的图可能把 100 张的答案写死。做 SFT 先接受,记录图的 sha 供抽查;转 RL 前要加一份 agent 看不到的评测子集。
堵兜底:
- 编译日志节点表里的输出类型不能用。卷积是 int8,但输出 fp16 时也显示 f16。
- 可靠的度量是编译器估计的延迟,和同一题默认 int8 编译比。默认编译每题跑一次,缓存后共享。
- 先离线重判给用户看会翻多少条,阈值由用户定。
开发环境(10-14 下午复查,**排第一**):
- 迁到 Mac mini 后,agent 的 bash 跑在 VM 里,但 VM 里没有工具链:
- 没有 /home/ming01.wang/toolchains;
- 系统 python 没有 hmct、onnxruntime、torch;
- 连提示词让用的 rg 都没装。
- 10-13 以来 05/06 的 171 条会话里,只有 5 条跑成过一次本地试验。GLM 在难题上靠的"先本地诊断再编译",现在做不了。
- 所以问题不是算力不够,是工具根本不在。
- 改法:把受管终端(terminal.py 的 development 目标,现在关着,而且只在本机跑)改成走 Salty,复用 salty_remote 的打包和取回。workspace 带进去,跑完带回来。
- 开发环境里可以随便跑 hb_compile、hmct、torch.onnx.export;正式编译仍走 stage、计次数。
顺带发现:e2e-occ 提示词说可以指向自备校准 npy,但白名单不收 cal_data_dir。(已修,见下)
**5. 已实现(10-14 a92510d,等重启上线)**
- 一、`bench dev`:
- agent 用 `bench dev "<命令>"` 提交,经 stageclient 写成 terminal 票(target development)。
- terminal.run 发现 salty_remote.stages 里有 terminal,就走 harness/salty_dev.py;worker 用 dev 模式跑。
- 沙箱:setpriv 降到 uid nobody,再 `unshare -Urm`;tmpfs 盖住 /data/personal,只把工具链、outputs、cache、data(靠属主只读)和本轨迹 workspace 绑回来,最后去掉全部能力再执行命令。
- Salty 实测:
- 板卡密钥、/root/.ssh、其他轨迹都看不到;
- 不去能力时 umount 能看到整个个人空间,去掉后挡住了;
- 能 import hmct、onnxruntime、torch,小卷积网络的 hb_compile 能出 hbm(玩具全连接模型在沙箱外一样崩,是 hbdk 自己的问题)。
- 每条命令开销约 75–80 秒。
- 工作区同步:本地改动按 stages/_dev-sync.json 上传;单个 ≤32MiB 的文件拷回本地;更大的只留在 personal:/bench-root/runs/.../workspace。
- 局限:容器里起的后台进程随命令结束;开发环境的大文件不会自动清理。
- 二、compile.yaml 可以写 `model_parameters.onnx_model`、`calibration_parameters.cal_data_dir`(workspace 内的相对路径):
- stageworker 先归一化;s04 在容器里比对输入输出,不一致就以 own_input_invalid 失败,不占编译次数;
- 记录图的 sha256 和 calib_source=workspace;hb_verifier 仍用平台的校准样本。
- 历史上 1377 份配置里只有 4 份写了 onnx_model,全是 agent 在试自带的图。
- SYSTEM.md 加了开发环境一节,rg 改成 grep;agent/VERSION 15→16(新旧轨迹按版本区分)。
- 三、速度门槛离线重判:
- 方法:最终模型的编译器估计延迟 ÷ 同题平台默认配置编出的延迟,在 Salty 上读 personal:/bench-root/runs 全量。
- 默认配置通过的比值 0.91–1.12,所以噪声约 ±10%。全 int16 是 1.5–6.2 倍。
- 阈值 1.5 倍时会翻掉的通过轨迹:
- 06:37/178(36 条全 int16,1 条 yolo11n);
- 04:24/64(18 条全 int16,6 条按节点调的,板上慢 7–500 倍,多是落 CPU);
- 05:只有 41/109 有参考(S600 上 agent 很少编平台默认配置),翻 10 条。
- 板上延迟比估计更能抓到落 CPU,正式门槛要用板上测的默认配置做参考。05 要每题补一次默认编译。
- 结果文件在 scratchpad 的 speed-out.json,脚本是 speed.py。
- 四、上线后 1 小时 50 分钟核查(10-14 20:50,v16 共 34 条轨迹,还没有一条跑完):
- 自带图提交 0 条、自带校准提交 0 条、改图命令 0 条。29/34 用了 bench dev,共 140 条命令。
- 命令主要是:看平台 ONNX 结构、读 harness 源码(编译和评测代码)、按速度要求在 dev 里编一次默认配置(约 20 次)。
- 首次提交的配置仍是默认加 per_channel 加 set_model_output_int16,跟以前一样。
- 结论:工具到位了,行为还没变。改图只会在配置调不过时才出现,要等这批难题跑完再看。
- 发现并修复(f16faee,20:47 热部署,不用重启):
- bench dev 能看到评测集 t1(带标签)和 pool,是 a92510d 引入的。efficientvit.b3 读过 t1。现在只挂 bench/data/calib。
- salty_dev 重复上传一次 bundle,已去掉。
- SYSTEM.md 第 57 行还写着「评测数据只读可见」,下次换版本时删掉。
- 五、23:50 复查:
- f16faee 两处都生效:dev 里读 t1/pool 报不存在;成功命令中位 71 秒(之前 78)。
- 开始真改图了:maxvit.t 把 Einsum 改写成 MatMul+Transpose(跑通),reppoints 物化 Reshape 目标(被排队饿死)。另有 4 条在逐节点诊断。还没有提交自带图。
- 新瓶颈:dev 池只有 8 槽(三个 run 共用锁),被 agent 自设 3–12 小时超时的长编译占住(最长 272 分钟)。短命令排到自己超时就报 STAGE_EXECUTOR_EXIT,19:56 起共 47 条,echo 也排了 37 分钟。Salty 当时不排队,瓶颈在我们自己的槽数。
- pools.dev.workers 每次现读配置文件,三个 -v16 配置一起改就能生效、不用重启;改不改等用户拍板。
- 六、10-15 10:43 自带图结果:
- 真改图的 4 条,都还没有最终结论:
- deeplabv3plus.r101(05):拆卷积加自做校准,编译、HBM、上板都过了,评测在跑。这题以前 12 小时没过。板上 506 ms,静态估计 38 ms,BPU 利用率 3.5%,大半跑在 CPU。agent 看到 job.json 里 G4 不是必选,就不管速度要求了。
- maxvit.t(06):Einsum→MatMul,编译和 HBM 过了,08:13 起卡在一条 dev 长编译上。
- deeplabv3plus.r50、fcn.r101(05):同样是拆大卷积(2560→512、2304→256),10:00 起在正式编译。fcn 还在一个校准样本上验证了浮点等价。
- 04 superpoint 是「自带图自带校准」的 PASS,但两样都是 cp 的平台文件,不算。以后统计自带图要比哈希,排除这种复制。
- vit.h.14(06)自做校准编过两次,240 轮用完没上板(BUDGET_EXHAUSTED)。
**Why:** 用户要把这套当训练数据来源和以后 RSI 的基础设施。考卷决定了能学到什么。
**How to apply:**
- 改进顺序(10-14 给用户的方案):
1. 开发环境上 Salty;
2. 窄路开放改图,加 cal_data_dir 自备校准;
3. 验收加速度要求;
4. 删掉提示词里喂饭的话;
5. 以后再做:nv12 输入、板上前后处理、板上终端、隐藏的验收子集、QAT。
- 改判分标准、改题目都要用户拍板。
- 样本只覆盖还在 VM 盘上的会话(04 有 38 条、05 和 06 各 60 条,偏向较新、较难的题);更早的已归档到个人空间。
- 统计脚本在 scratchpad 的 sess.py、cfg.py、cfg2.py。
相关:[[verifier-audit-20261012]] [[qwen-ability-failures-20261013]] [[success-rate-breakdown-20261008]]

View File

@ -1,57 +0,0 @@
---
name: bench-restart-hazards
description: model-compile-hbdk 的 bench 调度程序重启时三个没写进文档的陷阱
metadata:
node_type: memory
type: project
originSessionId: 2636af57-8cef-4006-92c2-7d78ac925e56
modified: 2026-10-03T16:33:23.678Z
---
2026-09-18 实测确认,`bench collect` 换代时有三个坑,CONTRACT.md 只写了其中第一个:
1. **改文件不算换代**:stage 在长驻 `bench collect` 进程内执行(`hb_compile` 的父进程就是它),
必须重启进程才生效。CONTRACT.md:184 有记录。
2. **绝不能 kill -9 调度程序**:`pools.agent.workers=12`,而 `pipeline.py:509` / `db.py:238` 的 active
计数是 `SELECT COUNT(*) FROM rollouts WHERE state IN ('RUNNING','VERIFYING')`,**跨 run 无过滤**。
全仓没有 rollouts RUNNING→QUEUED 的路径(`reset-stale` 只动 jobs 和 stages),硬杀会让那些行
永久占死全局槽位,无 CLI 可救。只能 SIGTERM。
3. **`bench drain` 是单向闸门**:`collect_drain_file` 只有 `cli.py:182` 的 `.touch()`,全仓无 unlink,
也没有 undrain 子命令。用了 drain 就必须在重启前手工 `rm state/drain-collect-<run>`,否则新进程启动即自停。
优先用 SIGTERM,它不产生哨兵文件。
4. **密钥可能只在旧进程的环境变量里**(2026-10-04 踩坑):`paths.secret()` 先读环境变量,再读 `bench/.env.local`。
04 批次(Qwen)用的 `QWEN_EVAL_API_KEY` 不在 `.env.local` 里,只存在于旧进程的环境中;我结束旧进程后,
只照搬 `BENCH_*` 重启,结果报 `CREDENTIAL_UNAVAILABLE: teacher`。
重启前先把配置里所有 `api_key_env` 和 `.env.local` 对一遍,只比对变量名;缺的先让用户补进 `.env.local`,再动旧进程。
另:`--first-pass` 的过滤器按 **job 聚合**(`pipeline.py:503`)且会并入源 run 历史,对 continuation run
(如 r20260916-01-n100)会把 206 条 QUEUED 全部挡掉、可放行 0 条——这正是它空转 14 小时的原因。
换代时务必自己算一遍可放行数,不要相信转述。
5. **重启要等很久,而且旧进程会卡在最后的导出上**(2026-10-04 实测):
- 新采集器在网盘上启动时要逐条核对 836 条任务,要过 10–20 分钟才会认领第一条,期间进程处于 D 状态、CPU 很低,这是正常的,不要误判成卡死。
- 旧采集器排空后会一直卡在最后的数据导出(`datasets/<run>-training/.refresh-*`)。确认这个批次的运行中、校验中、清理中都是 0 条之后,再等 5 分钟仍未退出,可以 kill -9;之后删掉留下的 `.refresh-*` 目录。
- 正在跑的任务没有"暂停后接着跑"的办法:配额管理器在采集器进程里面,进程一停,正在跑的任务就断了。想马上重启只能先用 `bench execution cancel` 取消正在跑的任务;不想丢任务就只能等它们跑完。
- 重启时照搬旧进程的命令行、工作目录、环境变量(尤其是 `BENCH_CONFIG`),日志用 `>>` 追加到原来的日志文件里,就不会产生新的批次或文件。
- 看日志时不要用时间字符串大小比较来过滤(日志会跨天),按每次启动时打印的 "collect <run>: parallel" 那一行的行号来截取。
- `bench execution cancel` 偶尔会卡在网盘读写上(进程等待 FUSE,wchan=request_wait_answer),重启脚本就停在 `wait` 不往下走(2026-10-08 实测卡了 20 分钟)。先查这条轨迹在库里是否已经是终态;是的话直接 kill 这个取消命令,脚本会继续往下走。
6. **想提并发,三个 collector 得一起换代**(2026-10-08 实测,代码读过):
- `pools.agent.workers` 是**所有批次加总**的上限(`pipeline.py:595`),而且只在 collector 启动时读一次。只给一个批次调大,别的批次进程里还是旧上限,总数一超过旧上限它们就补不上位置,会被挤掉。
- 同一批次只能有一个 collector(`locks/collect-<run>.lock` 排它锁),不能靠"同批次再开一个"加并发。
- broker 启动时就把 key 读进内存,之后改 `endpoints` 不会换 key;改了 `collection_v1` 段,在跑的进程会报 `RUNTIME_CONFIG_CHANGED`。所以**永远另写一份新配置,不改正在用的那份**。
- 账本(`~/.local/state/bench-shared/quota.sqlite`)存着每个账号组的规则,配置里同名组的规则只要差一个字段,启动就报 `QUOTA_POLICY_CONFLICT`。新账号要用**新组名**;老组规则不要动。
- **`paths.config()` 带 `@lru_cache(maxsize=1)`(paths.py:55),每个进程只读一次配置**;scheduler.py:156 注释说"每轮重读"已过时。改 `pools.cpu.workers` 只对改之后新起的 stage 进程生效(每次编译/HBM 是新进程,所以很快普遍生效),已经在排队的仍按旧槽位数等。10-08 我起初漏看这个装饰器,误判过一次。
7. **不等排空、直接重启的流程**(2026-10-08 跑通,21 条全部成功):先 SIGTERM 旧 collector,再对每条在跑轨迹执行
`bench execution cancel --run R --job J --kk K --reason ...`(环境照抄旧进程,`BENCH_CONFIG` 用旧配置)。每条约 30 秒;
取消后库里会先短暂显示 `VERIFYING`,片刻后变为 `CANCELLED`,这是旧进程在收尾,不是失败。旧进程随后照例卡在导出,按第 5 条处理。
脚本在远端 `/home/ming01.wang/raise-concurrency-20261008/cancel-inflight.sh`、`relaunch.sh`。
- 没有"暂停后换进程续跑":`execution resume` 只接受系统因额度等待主动暂停(`PAUSED_INFRA` 且有断点)的轨迹;进程被杀后 reconcile 会把在跑的记成 `LOST`。
- 取舍依据:按已跑时长算条件成功率,跑过 5 小时的只有 10–15% 能 PASS;10-08 那次 21 条预计只值约 4 条 PASS。
- 本机 shell 是 zsh,`set -- $var` 不拆词;本机写轮询判断时用 `bash -c` 或把判断放到远端 bash 里。
8. **取消在跑轨迹会让账号组停约 21 分钟**(2026-10-12 实测):`bench execution cancel` 打断发到一半的模型请求,这些请求记成"结果不明",整个账号组进入 RECONCILING,新调度程序起来后也领不了题,要等 request_total_s(1200)+window(60) 后自动放弃才恢复。10-12 17:17 取消 46 条后,05 到 17:39 才开始领题(06 刚好没有发到一半的请求,立刻领满)。立即重启时预留这段时间,别误判成卡死。
9. **重启虚拟机(断电)时,暂停的轨迹也续不了**(2026-10-16 实测):process_status 对旧开机的进程返回 UNKNOWN,开机后 reconcile 把 PAUSED_INFRA 轨迹标成 OWNERSHIP_UNKNOWN 隔离,resume 也拒绝开机变过的。停 VM 前要把可续跑的暂停一起取消补号;事后取消要等 30 分钟清理时限。详见 [[poweroff-20261016]]。

View File

@ -1,44 +0,0 @@
---
name: check-20261016-1700
description: 10-16 17:00 复查:v18–v21 共 46 条,自己导图 41、自做校准 43、真改图约 3 条、板上自测 2 条,但 0 条通过;05 的 GLM S600 账号 16:00 起全是 429;Qwen 重启后等了 30 分钟
metadata:
node_type: memory
type: project
originSessionId: 8477d54c-f423-4a01-ab7c-492675d618dd
modified: 2026-10-16T08:55:20.221Z
---
10-16 16:50–17:00 复查(16:21 刚用 v22 重启)。
**v18–v21 的 46 条轨迹(10-15 21:43 到 10-16 16:20)**
- 行为:
- 43 条用了 dev;41 条在 dev 里自己导图(v17 是 3/40);43 条自做校准。
- 真改图约 3 条:
- 06 efficientvit.b1:在线性注意力里缩放 relu(q)/relu(k);
- 04 yolov8n.seg(Qwen):把 proto 的 SiLU 换成 tanh 形式;
- 06 depth.anything:自己造小模型探测 int8/int16 MatMul 的延迟。
- 板上自测:2 条用过,8 次里成功 2 次。
- 时间:dev 202 小时,正式阶段 24 小时。
- 结果:**0 条通过**。
- 25 条被 16:20 的重启取消。
- 8 条 INFRA_ERROR:Qwen 上下文超 256k 有 5 条,其余是 broker 503 等。
- 3 条预算用完,都是 GLM 的 80 万 token。
- 还有 10 条在跑。
- v17(40 条):7 条通过(Qwen 4、GLM 3);15 条预算用完,12 条超 24 小时。
**卡住的情况**
- 05 用的 GLM 账号 glm-coding-s600-account:
- 16:00:59 起每次请求都返回 HTTP 429 RATE_LIMIT。账本每 15 分钟试一次,05 的 16 条全部 WAITING_QUOTA。
- 前 3 天一次 429 都没有。15 点那一小时请求 429 次、1860 万 token,约为平常的 2–4 倍。
- 429 的正文没有落盘,看不出是哪种限额、什么时候恢复。
- 06 的账号 B 正常;16 点那一小时 749 次请求、3040 万 token,也明显偏高。
- 04 Qwen:重启时取消了在途请求,结果不明,组进入 RECONCILING,等满 30 分钟的请求期限(16:59 自动作废)才恢复,16:55 已经有 5 条在跑。UNKNOWN_HOLD_S=120 没有覆盖到"取消造成的结果不明"。
- 速度参考只算完 7 道,失败 4 道。
- 主机时钟:账本时间比 VM 的 date 快约 6 分钟,看时间对不上时注意。
**Why:** 用户问"会不会自己改图、进度、有没有卡死、数据是不是更像人用 agent"。
**How to apply:**
- 看 05 停摆先查 s600 账号的 attempts.http_status。
- 想看 429 正文,broker 要先把它记下来;不要拿密钥自己去打接口。
相关:[[v18-own-inputs-20261015]] [[v17-autonomy-check-20261015]]

View File

@ -1,30 +0,0 @@
---
name: check-20261017-0215
description: 10-17 02:15 底层复查:v18+ 134 条的流程漏斗(导图 105、真改图约 10、板上自测 6、正式评测 20、通过 3);平台独立复测怎么做、漏了速度;重启后 5 小时很稳,只剩 Qwen 续跑必死(context-mask 有内存状态,续跑时请求对不上)
metadata:
node_type: memory
type: project
originSessionId: e00ddcbc-d4da-4f81-a8f7-1fd7eeb8e8df
modified: 2026-10-16T18:24:07.553Z
---
10-16 19:20 用户为断电取消 36 条(POWEROFF_20261016),Mac mini 20:26 重启,20:57 三个 collector 用 v22 配置起来(05 是 s600-r05-v22c)。仍走 Wi‑Fi,网关变成 192.168.1.1。
**v18+ 134 条漏斗(02:15,按 terminal_result.json 命令 + workspace 脚本 + stages 表)**
- 用开发环境 114;导图 105(72 条用了平台的 harness.tools.export_onnx,92 条有 torch.onnx.export,有重叠);原图做校准 105;开发环境本地编译 98;本地精度诊断 101。
- 真改交付图约 10 条:maxvit.t×2 改写 einsum、depth.anything.v2.large、efficientvit.b1、mask2former 改 einsum、05 fcn.resnet50 和 deeplabv3plus.resnet50 拆卷积、04 yolov8n.seg 改 proto、efficientnet.b0 拆 swish、retinanet.r50 拆 FPN(失败)。swin.b、两条 depth 的 make_node 是探针或库文件,不算。
- 板上自测只有 6 条(重启后 25 次命令、19 次成功,用的人在变多)。
- 正式编译提交 49、成功 32、HBM 30、上板 22、评测跑完 20、PASS 3。状态:取消 78、在跑 37、系统出错 12、预算用完 4、PASS 3。
- 3 条 PASS 都在 05(S600):efficientnet.b7 = 平台导出脚本 + 默认配置 + per_channel,一次编过;fcn.resnet50 = 自写导出、PyTorch 层拆卷积、开发环境小样本试编;deeplabv3plus.resnet50 = ONNX 拆卷积 + 本地编译诊断 + 2 次正式编译,但板上 493 ms、BPU 2%,自己的静态估计 26 ms,平台参考(默认配置静态估计)38.8 ms。三条都没用板上自测。
- 训练数据:05 SFT 101(v16–17 只有 5)、06 164(v16–17 只有 2),v18+ 0 条。
**平台独立复测**:verify.verify_rollout 只读平台阶段(06-hbm/07-board/08-eval)的测量,HBM sha256 在 G1/G3/G4 绑定;agent 的开发环境、板上自测结果标 UNVERIFIED、不计。agent 不调用正式阶段就是 UNVERIFIED,平台不自己补跑。缺口:G4 速度只记录(verdict na);速度参考是 hb_model_info 静态估计,不含落 CPU 的算子,所以抓不到 deeplab 这种情况。要在板上测默认配置的延迟才行。
**重启后 5.3 小时**:健康检查一直 ok;GLM 请求 4072 次,只有 1 次中断(已续上);阶段 98 次成功、3 次失败(conditional.detr 编译挂死、pspnet.r101 容器被杀、1 次 own_input_invalid 不计次数);开发命令 949 条,系统原因的失败基本消失(执行器退出 0 次,超时 12 次)。
**唯一新的系统 bug**:01:24 04 两条 Qwen(efficientnet.b0 在拆 swish、retinanet.r50 在拆 FPN,各跑了约 4.5 小时)续跑 4 秒就以 INFRA_ERROR 结束。链条:00:53 一个 Qwen 请求流式空闲超时 → 整个 Qwen 组暂停 → 01:24 自动放弃那次请求并续跑 → 续跑重发的请求被 broker 拒,返回 409 EXECUTION_HAS_UNRESOLVED_GENERATION(quota._begin 要求重发内容和原来逐字节相同)→ 结束。根因是 agent/context-mask.js 的 `cut` 存在进程内存里(只在超限时移动),续跑是新进程,cut 从 0 重算,省略的位置不同,请求内容就变了。只影响开了省略的 04;10-13 以来本地还能查到的续跑:GLM 9 次全成功,Qwen 5 成 3 败。
- 最简修法:让 context-mask 只由消息列表决定(按每个 assistant 消息前的前缀把原来的逐步省略重放一遍,得到同一个 cut),约 10 行;agent 进程启动和续跑时加载,热部署即可,不用重启 collector。还没改,等用户定。
**Why:** 用户重启后问:数据是否走完整条链路、平台是否自己复测、系统稳没稳、底层错误和最简修法。
**How to apply:** 复查用同一口径(terminal_result.json 的 target=board 是板上自测;workspace 小文件在归档后仍在本地,events.jsonl 归档后会被删)。DB 里有 417 条 9 月旧批次的 PAUSED_INFRA,不计入并发,可以忽略。
相关:[[stability-sweep-20261016]] [[v18-own-inputs-20261015]] [[verifier-audit-20261012]] [[check-20261016-1700]]

View File

@ -1,40 +0,0 @@
---
name: compile-server-reclaim
description: model-compile 编译机要被回收(10-08 告知);主控实际占用、必须跟着搬的东西、Mac 不能当主控的原因
metadata:
node_type: memory
type: project
originSessionId: 1b3cd916-3faa-4ef7-be14-c3fb07244493
modified: 2026-10-08T08:25:37.022Z
---
2026-10-08 用户告知:model-compile(120.48.151.164,244 核 / 1.4 TB,x86)要被回收。这才是"编译上 Salty、主控放 Mac mini"方案的真实动机,提速不是目的。
10-08 实测:
- 主控很轻:3 个采集器约 1 核、7.8 GB;21 个 agent(42 个进程)约 2.9 GB、几乎不占 CPU。
- 吃算力的是编译:hb_compile 138 个进程,其中 77 个属于 bench 轨迹,其余是同机器上别人的实验(tune1.py、local_val.py;web_review.py 一个进程就占 124 GB 内存)。
- 本机盘 /home 上 bench 必须搬走的东西:`/home/ming01.wang/bench-state`(sqlite 账本和锁,131 MB)、`/home/ming01.wang/toolchains`(32 GB x86 环境)、生产配置 `/home/ming01.wang/s100p-glm-8way-20260928-2124/`。代码、轨迹、缓存都在网盘 /mnt/data(JuiceFS robotwin-archive)上。
- 只能在 Linux 上跑的部分:agent 沙箱是 `unshare -Urmpf --mount-proc`(scripts/jail_setup.py),进程管理读 /proc、boot_id、pidfd。06-hbm 用 hb_verifier/hb_model_info(地平线 x86 工具链),占轨迹总时长 16%。
- 不止 05 要 x86:02 基线、03 导出、04 校准数据、08 评测主机侧都用 `paths.model_python(job)`(网盘 cache/venvs 里的 x86 模型环境),05/06 用地平线工具链。只有 07(板端测速只是 ssh)和 agent/采集器不依赖 x86。
- 模型接口:GLM 走公网 open.bigmodel.cn,Qwen 走 ai-api.d-robotics.cc。
- 用户 10-08 的想法:Mac mini 只做任务下发,集群只提供执行环境。我的建议:能要到小云主机(8 核 32 GB)就用它当主控,因为它和网盘、集群在同一个机房,省掉数据搬运。
- Mac mini:`ssh amos@192.168.66.251`(10-08 已装本机 id_ed25519 公钥,不要用密码),工作目录 `/Users/amos/work`(10-08 为空)。
M6 12 核、**16 GB 内存**、剩余 121 GB 磁盘,macOS 27.0(arm64),没装虚拟机工具和 node。
电源设置:sleep=1、autorestart=0(断电后不会自动开机)。改电源设置属于系统设置,要用户自己执行。
10-08 实测能连通:GLM、Qwen、板卡 40009 端口、编译机 22 端口。
- 我当时操作用的 Mac 是 Mac16,1(MacBook Pro M4,16 GB),不是那台 Mac mini。
**Why:** 换机器时最容易漏的是本机盘上的账本和工具链,以及"只把 05 搬走"不够:06 也离不开 x86 工具链。
**How to apply:** 讨论新主控放哪时,先问能否拿到同机房的 Linux 机器;搬之前按 [[bench-restart-hazards]] 正常排空(只用 SIGTERM)。相关:[[rollout-time-breakdown-20261008]]
**2026-10-08 实际回收**(19:13–21:02 之间,ssh 主机密钥三种全换):没跑 `migration-meta-20260928/stop-time-refresh.sh`,所以本机盘上的东西只有 9-28 22:17 的 tar 备份(`/mnt/data/ming01.wang/migration-meta-20260928/home-tars/`:bench-state.tar、toolchains.tar 33G、s100p-glm-8way-*.tar、.local.tar 含额度账本)。9-28 的 bench-state 比 r20260929-0x 批次还早,**这几个批次的账本没有备份**;今天写的 raise-concurrency-20261008 配置也没备份。
新机器 `ming01.wang@36.144.20.182`:112 核、944 GB、Ubuntu 22.04,JuiceFS 已挂 `/mnt/data`,`/home` 盘只剩 23 GB(放不下 32 GB 工具链),unshare 沙箱可用、python3.10 在、没有 node。**新机器上 ming01.wang 是 uid 1006,网盘文件属 uid 1004(新机器上叫 xiaoyan.zhao)**:bench 目录、批次目录全部不可写,`.env.local` 和每条 rollout.json 都读不了。网盘上的记录文件数(04=119、05=214、06=319)与回收前一致。
**更正(2026-10-11 实测)**:网盘 bench/runs 下的文件现在属 uid 1006(36 上的 ming01.wang)、组 1004,36 上能读能写,rollout.json、agent/events.jsonl、HBM 都读得到。120 已于 10-11 关机清理,旧轨迹以后从 36 读。
**2026-10-09 捞回**:老机器重装后仍是 `root@120.48.151.164`(我用本机 `~/.ssh/id_ed25519_codex_180_76_192_183` 登录,用户自己 ssh-copy-id 装的),原 home 整盘在 `/home/cce/ming01.wang`(属 uid 1004)。账本 jobs.sqlite / quota.sqlite 记到 10-08 20:47,完整性 OK。已拷到新机器暂存区 `~/restore-20261009`(sha256 一致):bench-state、.local/state/bench-shared、三个配置目录、resume-glm-20261002-104649 日志、.ssh、toolchains/node。
- toolchains 只有 node 在 9-28 后变过(多了 mcp-remote),其余 bench-venv 7.5G / oe-3.7.0 13G / oellm2 11G 可从网盘 `migration-meta-20260928/home-tars/toolchains.tar` 解;四块都在配置里,沙箱会全挂载,缺一不可。venv 的 base 都是 /usr/bin python3.10。
- `paths.work_root=/home/ming01.wang/bench-work` 是每条轨迹的编译临时目录,老机器上 444G,不用搬(续跑不依赖),但新机器需要大盘装它(重模型一条 10–17G)。
- 停机时在跑的 37 条(04=5、05=16、06=16)账本里仍是 RUNNING,启动前要先处理。
**更正(10-09)**:网盘上 `bench/cache`、`bench/data`、`bench/state` 都是指向 `/home/ming01.wang/bench-{cache,data,state}` 的软链接,**缓存和数据其实在本机盘**,不在网盘。老机器上:bench-cache 415G(baseline 160、onnx 88、weights 81、adapter-probes 80、venvs 1G),9-28 备份(374G,19:43 打包)之后新增 397 个文件 71G(baseline 49G + onnx 22G,可由 02/03 阶段自动重算,约 12.3+1 小时串行累计);bench-data 46G,备份后零改动,网盘备份即最新。LLM 网关 `.local/state/litellm-formal-gateway` 备份后改动 231G(不属 bench,用户定)。
`cache_contract` 锁的是模型 Python 路径和导出器版本,不是缓存内容;路径用 abspath 比对、不解软链接,所以大目录可放网盘、原路径放软链接。开 Salty 后只有 05(PTQ)上云、06 读 Salty 回执;01/02/03/04/08 仍在主控跑,需要 bench-venv + 缓存 + 数据(约 380G+)。

View File

@ -1,39 +0,0 @@
---
name: delivery-package-v1
description: 给内部评测同事的第一版交付包:位置、冻结点、口径、已知缺口
metadata:
node_type: memory
type: project
originSessionId: cf15f913-cbc4-4a5a-9046-1031104e17bb
modified: 2026-09-19T07:48:01.108Z
---
第一版评测交付包在服务器 `/mnt/data/ming01.wang/val-model-compile`,13 GB / 3932 个文件。
本地构建副本在 `/home/ming01.wang/delivery-20260918`(未删,待用户决定)。
**冻结点:2026-09-18 17:18:27,62 条 `rollouts.verdict_terminal='PASS'`,40 个模型。**
冻结当天池子还在跑,次日已涨到 87 条——出第二版时这个数还会变,必须重新冻结快照。
关键口径(出第二版仍适用):
- 唯一正确的"完成"口径是 `rollouts.verdict_terminal='PASS'`。
`state='TRANSFERRED'` 那 597 条**一次都没跑过**,是任务槽位转交的墓碑标记,
按它打包会打出几百个空目录。
- `completion.workflow_completed=true` 比 PASS 多出来的那些是"跑通但掉点超容差"。
- `jobs.state='PASS'` 是 oracle 标准答案流水线,不是 agent 轨迹。
- `jobs` 表按基准批次记账,n100 派生批次没有独立行——62 条里 26 条查不到,
查询一律走 `rollouts` 表。
打包时踩过、第二版要直接避开的坑:
- `bin/bench`(711) 和 `harness/cli.py`(600) 权限不对外,拷完必须 `chmod a+r`,否则同事用不了。
- `stages/*/meta.json` 必须带(62 条共 443 个,543 KB)。有它同事没板卡也能离线重算出
PASS;没有它 62 条全变 UNVERIFIED。
- 必须按工作区拷,不能 `git archive`——大量适配器没提交。
- 缓存目录要按需挑:baseline 全量 59 GB(大头是 G2 参考张量,G2 不参与判定,不带)、
onnx 27 GB、calib 16 GB,实际只用到其中 12.4 GB。
- 排除 `.env.local`、`.sync/`、`.git/`、`__pycache__`、`docs/archive/`。
`.env.local.example` 和 `.gitignore` 可以带(模板是空的)。
已知缺口(已写进包内说明):分割/姿态/光流/深度/特征/占据/双目 7 类零样本;
实例分割仅 1 条且其轨迹因版本过滤未导出;4 条合格轨迹不在数据集里(轨迹只覆盖 58/62)。
相关:[[prefer-subtraction-over-addition]] [[bench-restart-hazards]]

View File

@ -1,16 +0,0 @@
---
name: git-mainline-20261009
description: 10-09 仓库主线整理:main 改为生产分支、旧 main 打标签、Salty 单独分支;本地工作区还剩未提交的优雅退出/板卡改动
metadata:
type: project
---
仓库 `bench/`(gitea `ming01.wang/data-gen-bpu`)。2026-10-09 用户决定:生产分支 `production/s100p-infra-fix-20260928` 就是完整主线,旧 `main`(9-16 起分叉、基本没用)不要了。
- 网盘上在跑的代码 = `c2c52ed`(gitea 生产分支也已是它)。旧 main 留标签 `archive/main-20260917`(12ef41c);`main` 改指 `c2c52ed`(10-09 已用 `--force-with-lease=main:c44d3f3` 推到 gitea,标签和 feature 分支也已推)。
- 旧 main 的 8 个真改动里,watchdog 按线程判 D 状态、06-hbm 回收中间 ONNX 已核实在 c2c52ed 里。
- Salty 远程编译单独成分支 `feature/salty-20261009`(45b1262,基于 c2c52ed),默认关闭(需 `execution.salty.enabled: true` 且 stages 含 05-compile)。合到生产代码时补了:编译上限只放行"已记录过 submit"的 Salty 找回(`salty_stage.dispatch_recorded`)。
- 本地工作区仍在旧的 10663a8 上,留着未提交的"进程优雅退出(owned_process/namespace_owner)+ 板卡侧"改动;其中 `board_guard.py` 与生产分支 f315496 的隔离释放策略相反,合之前要用户定。
- 在新机器上跑单测:`PYTHONPATH=. timeout 900 python3.10 scripts/test_x.py`;`test_agent_first`、`test_board_pipeline` 在干净的 c2c52ed 上也失败(环境未装好),不是回归。
**Why:** 以后问"main 怎么变了""Salty 在哪个分支""本地那堆改动是什么"时直接查这里。
**How to apply:** 新工作从 `main` 开分支;动本地工作区那批非 Salty 改动前先问用户。相关:[[compile-server-reclaim]]

View File

@ -1,19 +0,0 @@
---
name: glm-second-account-20261008
description: 10-08 加第二个 GLM 账号、三个批次换代提并发的现场:配置位置、接续脚本、目标路数、账号对应关系
metadata:
type: project
---
2026-10-08 用户要求提采集并发。事实:
- 05(S600,`S600_API_KEY`)和 06(S100P,`TEACHER_API_KEY`)在 GLM 那边是**同一个账号 A**;`S100P_API_KEY`是**新账号 B**,之前从没用过,只能走 coding 套餐接口。
- 换代前 24 小时两个 GLM 组共约 5470 次请求、零次 429;8 路时在途请求 76% 时间为 0、峰值 8。账号额度不是瓶颈,瓶颈是每个 collector 只开 8 路、全局上限 24。
- 做法(用户选的方案 B):06 改用账号 B(新账号组 `glm-coding-account-b`,规则照抄老组),并发 8→16;05 留在账号 A,8→16;04(Qwen)配置不变、仍 5 路,只为抬全局上限重启。三份配置的全局上限都改成 40。
- 新配置和接续脚本在远端 `/home/ming01.wang/raise-concurrency-20261008/`;`relaunch-<run>.log` 记录每个批次何时排空、何时起新进程。日志仍追加到 `/home/ming01.wang/resume-glm-20261002-104649/restored-r20260929-0X-fix-20261003.log`。
- 16:26 SIGTERM;30 分钟内 05/06 一条没跑完,按用户要求不等了:16:58 起取消全部 21 条在跑轨迹(预计只损失约 4 条 PASS),17:14–17:17 三个新进程起来(04=5 路、05=16 路、06=16 路,06 走 `glm-coding-account-b`)。
- 18:07 核对:05、06 各 16 路满载,06 只走 `glm-coding-account-b`,零 429。04 在 17:44–18:00 被另一个会话("qwen-3.8-27B评估")接管:取消了新 04 的 5 条,重启为只跑 `benchmarks/qwen-sft-v1.json` 的 37 个任务并改了 Qwen 超时和账本规则。动 04 前先确认那个会话是否还在用它。
- 19:04 核对:换代 1.8 小时 05/06 一条没结束。原因是 32 条同时开工、正好排到最重的一批模型(vit.h.14、dab.detr 等编译历史中位 1.5–4 小时);编译本身没变慢(换代后完成的编译中位 31 分 vs 之前 28 分)。编译槽位 24/24 满,7 条在排队、最长等 38 分钟,负载 130/244;用户同意后 19:08 把 05/06 配置的 `pools.cpu.workers` 改为 32(原子替换,collection_v1 不变);只对之后新起的编译/HBM 进程生效,之前已排队的 3 个仍按 24 等。查编译耗时:`stages` 表里成功的状态是 `OK` 不是 `DONE`。
**Why:** 下次再问"并发怎么变了""换回去"时,要知道动了哪些、老配置原样留在原处。
**How to apply:** 回退就用原目录 `/home/ming01.wang/s100p-glm-8way-20260928-2124/` 和 `/home/ming01.wang/qwen-autofill-20260928/` 的老配置照同样流程换代。相关:[[bench-restart-hazards]] [[compile-server-reclaim]]

View File

@ -1,40 +0,0 @@
---
name: hbm-spotcheck-optional-20261014
description: 10-14 06-hbm 的 hbm/bc 模拟器抽查改成 agent 按需开启、只作诊断;默认不跑已部署,配置校验与说明等下次重启;还有 DNS 续跑也等重启
metadata:
node_type: memory
type: project
originSessionId: 2b85e929-b893-42c3-a7a0-3a72bda14705
modified: 2026-10-14T04:48:46.602Z
---
**问题**:06-hbm 以前每次必跑 hbm/bc 模拟器抽查,用 x86 模拟器跑 2 张校准图。
- 本地跑时,它占这一步 95% 的时间(中位 7 分钟)。
- Salty 上要 1–2 小时;hb_verifier 只要约 40 秒。
- 10-12 17:16 到 10-14,Salty 上撞 2 小时上限 103 次,约 206 小时。
它从没单独发现过问题:历史 388 次抽查报出 13 次不一致,hb_verifier 都已先判掉;08-eval 还会在板上跑真 HBM。
**用户的决定**:不加大时限,也不按参数量设时限。把抽查改成 agent 自主判断能不能用,代码要通用。
- 不按参数量设:模拟器耗时主要看计算量和输入分辨率,不看参数量。
- 不加大时限:时间照样花掉、占着 Salty 核,模型每重编一次就要再生成一次 HBM,会吃掉 24 小时总时限。
**做法**(f74fe00,叠在 ba143fa 上):
- 阶段配置新增 `verification_parameters.hbm_bc_samples`,取值 0–16,默认 0。
- agent 用 `bench stage run hbm --config x.yaml` 开启。
- 结果写在 G1.hbm_bc_cos_min,**不改 verifier_ok**。否则开了只会被扣分,agent 就永远不会开。
- 适用于任何有 .bc 的后端。
- 编译配置只取三段(input、calibration、compiler 参数),所以同一个 yaml 拿去编译不受影响。
**上线情况**:
- 已部署:s06_hbm.py(10-14 12:48),默认不跑抽查。这个文件随阶段打包进容器,部署即生效。
- **等下次重启**:
- stageworker.py 的配置校验和 agent/SYSTEM.md 的说明(f74fe00)。说明不能先上线,否则 agent 按说明提交会被旧校验拒绝(invalid_request)。
- quota.py 的 DNS_FAILED / DNS_TIMEOUT 可以续跑(e4e1513)。06 的 fcn.resnet101、depth.anything.v1.large 从 10-14 凌晨起一直停着,等重启后续上。
**Why:** 用户要"合理方案",不是简单加大时限;而且要通用。
**How to apply:**
- 下次重启要部署 stageworker.py、quota.py、agent/SYSTEM.md。部署前先和分支比对 md5。
- 看 06 用时要分 Salty 和本地。Salty 上的文件时间是解包时刻,不能拿来分段,要看日志里的时间戳。
相关:[[infra-leaks-20261013]] [[salty-container-threads-20261013]] [[bench-restart-hazards]]

View File

@ -1,31 +0,0 @@
---
name: hbm-step-bottleneck-20261014
description: 10-14 发现 06-hbm 成了新瓶颈:HBM≡BC 模拟器抽查在 Salty 上 1–2 小时,一夜 45 次撞 2 小时上限;DNS_FAILED 暂停不在自动续跑名单里
metadata:
node_type: memory
type: project
originSessionId: 24818155-52a9-4a37-a413-b2ac9f163981
modified: 2026-10-14T04:13:47.996Z
---
2026-10-14 12:00 复查 10-13 20:41 重启后 15 小时:
- **06-hbm 撞时限 45 次,浪费约 89 小时**,占所有超时的大头。05(S600 nash-p)成功时的中位耗时也要 48 分钟,06 是 16 分钟。
- 根因:s06_hbm._hbm_bc_spotcheck 用 x86 模拟器跑 HBM 推 2 张图。旧编译机上 swin.s 这一步就占 36/37 分钟;Salty 容器只有 4 核,要 1–2 小时以上。
- 这个抽查没有实际作用:历史 401 次里报出 13 次不一致,这 13 次 hb_verifier 的余弦都已经低于 0.97,早被它判掉了。07/08 也会在板上跑真 HBM。
- 模型撞时限后会反复重试同一步。比如 05 efficientvit.b2 kk2 连撞 6 次,烧掉 12 小时。
- 超时的作业日志拿不回来,只能从 Salty 远端目录里的 usage.log 看。
- **DNS_FAILED / DNS_TIMEOUT 暂停不会自动续跑**:
- quota._resume_readiness 的可续跑原因名单里没有这两项,但 DNS 失败发生在发送之前,重放是安全的。
- 06 有 fcn.resnet101 kk1 从 03:01 起、depth.anything.v1.large kk2 从 06:01 起一直停着。
- 根因还是 Clash 的 DNS,见 [[mac-mini-clash-global-proxy-20261013]]。
- 已确认正常的部分:
- CONNECT_FAILED、REQUEST_OUTCOME_UNKNOWN 暂停后 1–30 分钟内自动续上(共 9 次)。
- 孤儿容器清理在工作。
- CPU 名额没有泄漏。
**Why:** 修完 [[infra-leaks-20261013]] 后吞吐回升,06-hbm 接着成了下一个卡点。
**How to apply:**
- 提速先砍 HBM 抽查,或把它降到 1 张,不要先加时限。
- 改 s06_hbm 是阶段代码,新起的步骤马上生效。
- 改 quota 可续跑名单要等重启调度程序才生效。

View File

@ -1,60 +0,0 @@
---
name: infra-leaks-20261013
description: 10-13 查出四个基建问题:CPU 名额被超时步骤锁死(32 剩 6)、超时后 Salty 容器不停、续跑必死、领题后重复查额度把题作废
metadata:
node_type: memory
type: project
originSessionId: 2b85e929-b893-42c3-a7a0-3a72bda14705
modified: 2026-10-13T14:25:52.406Z
---
2026-10-13 11:00 核查 Codex 报的"新增 51 UNVERIFIED、21 INFRA_ERROR、25 条在等阶段队列",几乎全是系统问题:
1. **CPU 名额泄漏**:state/locks/cpu-N.lease.json 由步骤进程退出时删(scheduler._release)。步骤超时(PHASE_EXECUTION_DEADLINE_EXCEEDED)被看门狗终止时来不及删,_try_slots 见到凭证就跳过这个名额。scheduler.reconcile_leases 只在 recovery.reconcile_run 里调(调度程序启动时),而且只放已结束(TERMINAL)的轨迹。10-12 18:07 起累计 26/32 个被锁死,只剩 6 个,排队步骤最长等 3 小时。pool_usage() 只数 flock,看不出这个问题,要看 lease 文件的持有进程是否还活着。
2. **超时后远端容器不停**:本地步骤被杀,Salty 容器继续跑。10-13 11:00 有 7 个,最久跑了 7.5 小时,约 34 核。查法:salty/remote.json 的 job_id 在 salty ps 里是 running,但同目录 meta.json 已经写了结果。
3. **暂停后续跑必死**:续跑时 stageworker.scan 重新核对每个旧步骤的回执。同一条轨迹每次编译都写同一个文件(05-compile-work/<job>.hbm、stages/04-calibrate/compile.yaml),旧回执核对时 ValueError("completed artifact changed"),接着 OPERATION_STATE_UNKNOWN 隔离整条轨迹。只要编译过两次以上,续跑后几秒就判 INFRA_ERROR(10-12 晚到 10-13 共 19 条)。reconcile_completed 里已经有"已结算就跳过"的判断,scan/_execute 没有。
4. **领题后重复查额度**:pipeline 领题前调用 runner.check(execute=True),查不到额度会把题留在队列;rollout.run_rollout:317 领到之后又查一次,quota_snapshot 只等 0.5 秒。00:25–00:51 机器忙时读超时,QuotaError: QUOTA_SOURCE_UNAVAILABLE,题直接记成 UNVERIFIED,根本没开始跑(05 有 43 条,06 有 7 条)。
**已做(10-13 11:00–11:30,用户同意)**:
- 停掉 7 个无人等结果的 Salty 容器(只停,没删作业和个人空间目录)。
- 回收 26 个名额:把 lease 文件挪到 configs/config-backup-20261013/leases/。20 秒内 32 个名额又全部用上。
- 59 条从没跑过的 UNVERIFIED(05 有 45 条,06 有 14 条)改回 QUEUED,并记了注释 REQUEUED_20261013。在跑的调度程序把它们记在已领过的名单里,下次重启后才会重新领。
- 7093c32(11:11 部署,新起的步骤进程马上生效):
- scheduler._lease_settled:占用者已死、该步骤已 DONE、清理已确认,就接手这个名额。板卡名额不在此列。
- salty_remote 提交容器时加 `timeout <阶段执行时限>`。镜像里有 /usr/bin/timeout,实测到时限退出码 124。
- ec1a58e(**未部署,等这批跑完、重启时一起上**):
- stageworker._execute:已结算的操作直接沿用旧结果,不再重新核对。
- rollout.run_rollout:领到题后不再查额度。
**10-13 20:28–20:41 重启(用户选"今晚取消在跑的直接重启")**:
- 脚本和清单在 configs/restart-20261013/:
- stop.sh:存旧环境,SIGTERM,并行取消。
- start.sh:部署,补号,用 withenv.sh 照抄旧环境重启。
- 取消了 43 条:37 条在跑,加上 6 条不可能续跑的暂停轨迹(账本里已放弃,或续跑次数用完)。06 的 5 条可续跑的暂停轨迹保留。
- 05 yolov8s kk2 的取消命令卡在账本收尾 13 分钟。它的执行记录已经结束,所以用 SIGTERM 结束了这个命令。
- 部署 ec1a58e(D、E)和 8a1ce8e:
- 额度账本读取时限 0.5 秒改为 5 秒,领题前和续跑前两处检查都改了。下午续跑失败的真因其实是这个,不是"名额被抢"。
- INCOMPLETE_GENERATION 可以续跑。
- 补号:42 条取消的各补一个新 kk;202 条一步没跑就被作废的 UNVERIFIED 放回 QUEUED(有些在上午放回后又被作废了一次)。
- 没重启 salty_offload:它当时正在跑一个导出作业,等它空闲时再重启,这样导出作业也能带上线程上限。
- 测试注意:VM 里有代理环境变量,跑 scripts/test_quota 要 `env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY`,否则本机测试服务器全回 502。test_collection 的 test_real_database_lock_* 本来就时好时坏。
**10-13 21:53 核对,前三项都已修好**:
- 11:11 后又有 36 次步骤超时,CPU 名额没有一个被死进程锁住;等名额的中位时间从 105–127 分钟降到 0。
- 重启后 5 条续跑全部存活。
- 读额度账本超时 0 次。
- 漏洞 2 的修复只管超时,不管取消:20:28 重启取消的轨迹留下 11 个容器白跑,另有 1 个是 11:11 修复前提交、没有时限的 gmflow。
- 已停掉 9 个,另外 3 个已经自己结束。
- ad51e86:salty_offload 每轮开头调用 reap_orphans。本地这一步已有 meta.json、远端还在 running/pending 的就停;已 stopped 的用 sr._cleanup 删除作业和临时目录;failed 的保留。
- 部署后由 configs/restart-20261013/restart-offload.sh 等旧进程空闲再重启它。
- 踩坑:
- VM 里 Python 的 time.sleep 显示的 wchan 是 do_select,不是 nanosleep。
- pkill -f 会匹配到命令行里含同名字符串的 ssh/bash 自身,把自己杀掉。
**Why:** 用户以为是模型能力问题,实际上吞吐被系统卡住,数据被系统作废。
**How to apply:**
- 看吞吐先查 lease 文件,看持有进程是否还活着,不要只看 pool_usage。
- 第 3、4 条的代码跑在调度程序进程里,要重启才生效。
- 删 lease 文件和停 Salty 作业要先问用户。
相关:[[salty-container-threads-20261013]] [[selfheal-and-alert-20261012]] [[bench-restart-hazards]]

View File

@ -1,19 +0,0 @@
---
name: mac-mini-clash-global-proxy-20261013
description: 10-13 查明:Mac mini 上 Clash Verge 全局模式+TUN,采集的全部流量(Qwen 自建服务、Salty、GLM)都绕美国洛杉矶 JMS 节点;断线、Salty 网络错误、请求慢的共同根因
metadata:
node_type: memory
type: project
originSessionId: 6c848f33-e8d9-4981-ade2-dc77987005d7
modified: 2026-10-13T08:13:56.815Z
---
2026-10-13 16:30 查 Qwen 卡住的原因时发现:
- VM 里采集进程环境 http(s)_proxy=192.168.5.2:7897(Mac mini 上的 Clash Verge / mihomo),没有 no_proxy。
- Clash 是 **mode: global + TUN 开启**(verge.yaml enable_tun_mode: true,enable_system_proxy: true),GLOBAL 选的是 JMS 节点(日志 10-09 21:39 起就是),规则末尾 MATCH,JMS,没有 GEOIP,CN 直连规则。
- 实测:VM 走代理、VM "直连"、Mac mini 本机直连,出口 IP 全是 **美国加州洛杉矶 IT7 网络**。对国内 Qwen 服务器一个小请求要 1.5–2.7 秒。
- Clash 每 8 分钟更新一次订阅(profiles.yaml update_interval: 8),每次都 unset/set system DNS,10-13 一天 121 次;12:13 有一次"直接更新失败"。
- 10-13 Salty API 网络错误集中在 12:10–12:50(245 次)和 14:00–14:10(132 次),与 Qwen 请求断在半路(12:20、12:53、13:54、14:25)、GLM CONNECT_FAILED 同时发生。
**Why:** 这是断线、Salty 网络错误、请求慢的共同根因(之前我以为是 Wi-Fi,不对)。**更正(10-14)**:10-13 Qwen 停摆的主因还有另一会话查出的系统漏洞——CPU 名额被超时步骤锁死(我当时看到的"等 Salty 容器几十分钟"其实是等本地名额)、续跑必死、读额度账本 0.5 秒超时,见 [[infra-leaks-20261013]];10-13 20:41 已修。代理问题到 10-14 中午仍未改。
**How to apply:** 改 Clash 属于用户本机网络设置,我不能动,让用户在 Clash Verge 里改:要么 GLOBAL 选 DIRECT,要么切规则模式并给 114.111.18.115、*.d-robotics.cc、open.bigmodel.cn、板卡网段加 DIRECT 规则;订阅自动更新间隔调大。Mac mini 上还跑着 ChatGPT/Codex,可能需要境外代理,所以推荐规则模式而不是关掉。只设 no_proxy 不够:TUN 仍会接管。改完用 VM 里 `curl --noproxy '*' http://cip.cc` 确认出口在国内。相关:[[selfheal-and-alert-20261012]] [[mini-salty-controller-20261009]] [[qwen-ability-failures-20261013]]

View File

@ -1,14 +0,0 @@
---
name: memory-synced-to-repo
description: 记忆要同步到仓库 docs/memory/(feature/salty-20261009 分支);入口 docs/memory/AGENTS.md 写了创建和存放规则
metadata:
node_type: memory
type: feedback
originSessionId: 0f0f2556-2514-40e5-8ef7-f2c4e67c5701
modified: 2026-10-11T05:22:28.020Z
---
2026-10-11 用户要求把记忆推到远端仓库:放在 docs 目录下,用 AGENTS.md 引导,并写清楚怎么创建、怎么存放。现在仓库里的 `docs/memory/` 就是共享记忆库(Codex 等别的 agent 也读),入口是 `docs/memory/AGENTS.md`(规则和索引),根目录 AGENTS.md 里有指向它的说明。
**Why:** 用户同时在用 Codex 等别的 agent(10-11 起监控交给了 Codex),经验不能只留在我本机。
**How to apply:** 每次新增或修改记忆后,把文件同步到工作树的 docs/memory/,更新 AGENTS.md 的索引,提交到 feature/salty-20261009 并推送。推之前先 grep 一遍,确认没有密钥或密钥片段。相关:[[git-mainline-20261009]]

View File

@ -1,75 +0,0 @@
---
name: mini-salty-controller-20261009
description: 10-09 起 Mac mini 主控 + 每个阶段都在 Salty 容器里跑(salty_remote);各文件位置、为什么这样设计、还缺什么
metadata:
node_type: memory
type: project
originSessionId: a9e0f820-63b5-4eba-bcf3-24ad6f20af42
modified: 2026-10-12T03:03:09.463Z
---
10-09 用户要求把 04/05/06 三个批次搬到"Mac mini 主控 + Salty 执行 + 板卡验证",并让我直接打通。
**设计**:Mac mini 只跑 agent、调度、账本、配额、锁;agent 每次 `bench stage run`(01–08 全部)都在 Salty 起一个容器跑完整的 Stage.run(含查缓存),再把阶段目录(≤32MB 的文件)和板卡回执取回。代码:feature/salty-20261009 上的 harness/salty_remote.py + salty_remote_worker.py,钩子在 stages/base.py Stage.run 开头;配置键 execution.salty_remote。旧的 05-only salty_stage 保留但不启用。
**Why:** 缓存/数据只在 Salty 个人空间,Mac mini 看不到;只远程跑命令不行,查缓存也必须在容器里。Salty 不能在新命名空间挂 /proc(09-30 实测),所以 agent 沙箱只能在 Mac mini;剩余题都不带 recipe,不需要容器里的沙箱。
**How to apply:** 容器里用软链接复刻老编译机路径:/home/ming01.wang/{toolchains,outputs}→/data/personal/bench-runtime/home/…,{bench-cache,bench-data}→/data/personal/bench/{cache,data},/mnt/data/ming01.wang/model-compile-hbdk/bench 是真目录(BENCH_ROOT 会 resolve,不能是软链),runs→/data/personal/bench-root/runs。Mac mini 虚拟机里同一路径也是真目录。
位置(Mac mini 10-14 20:2x 起从外面连用 `ssh -p 443 amos@180.76.192.183`(公网转发,Mac mini 内网仍是 192.168.66.251,本机不在同一网段时内网地址连不上),报警脚本写成 ssh://amos@180.76.192.183:443;10-12 起内网是 amos@192.168.66.251;10-10 22:30 至 10-12 是 192.168.1.3;地址随重启/换网会变,连不上先问用户;仍走 Wi‑Fi,网线口 en0 空着):
- Mac mini 虚拟机:bench 根 /mnt/data/ming01.wang/model-compile-hbdk/bench(state/ 是 10-08 20:47 账本,sha 2d583fb0…),配额 ~/.local/state/bench-shared/,配置 /mnt/data/ming01.wang/configs/mini/*.yaml,板卡私钥/指纹/端口 ~/.bench-board/,key 录入脚本 /mnt/data/ming01.wang/configs/set-keys.sh(用户自己跑)
- 个人空间:personal:/bench-runtime/(三套环境 tar、board.env、ssh/、controller/ 账本备份);personal:/bench/ 归"salty迁移"会话,它负责解压 _tars
- 120 临时目录 /root/bench-runtime-20261009/(迁完要删);36 临时 ~/runs-pack-20261009.*、~/runs-essentials-*.tar.gz
- 本地 worktree:scratchpad/bench-salty(feature/salty-20261009),用完 `git worktree remove`
板卡:8 块都用 rdk_sandbox 私钥直连,指纹已固定;没用用户给的密码。rdk-sandbox(slot0)=40010。
网络:36→本机 Mac 只有约 60KB/s;Mac mini 从个人空间下载约 1.3MB/s,大件走个人空间中转。
10-10 开跑后的教训(代码都在 feature/salty-20261009,0cf3c20/949c5da 等):
- 编译工作目录必须在每个容器都是本地 /tmp/bench-work(base.work_dir 会 resolve 软链,04 写进 compile.yaml 的路径要和 05 一致);05/06 之间经 personal:/bench-root/work 交接。
- 主控要在本地复核 HBM(编译回执哈希、最终判定读 rollouts/NN/artifacts/*.hbm),所以 05 的 HBM 和新写的 artifacts 必须取回;06 用 copy2 保留旧 mtime,要按快照比对取回。
- Mac mini 走 Wi‑Fi 会网络抖动:salty describe 失败要重试(只读),大模型请求会变 PAUSED_INFRA(采集 --watch 自动恢复)。
- Salty 没有个人额度上限(10-11 复现:我们同时占 42 核/128G 没排队;平台方也确认总额度不缺、无"每人 3 个")。之前说的"约 25 核上限""两小时自动失败"都是我猜错的,已收回。`salty ps` 的状态会滞后(作业日志已开跑,ps 还显示 pending),短暂 pending 可能只是显示慢。编译用 6 核(运行中改 cpu_by_stage 只影响之后新起的 stagerun)。
- 作废名额的补法:旧行保留,db.rollout_upsert 新 kk + annotate 写原因;残留远程作业用 salty stop/rm/storage rm -r 清。
- Mac mini 关机/换网会打断在跑轨迹(重启后 PAUSED_INFRA 且隔离,resume 不可用,只能取消+补回)。
10-10 下午的教训(6a013d6):
- 356fdaa 起 Salty 传输层可重试(连续失败 30 分钟才算失败),上线后 20 次网络错误全被吸收;之前的 INFRA_ERROR 全是旧传输层的 OPERATION_STATE_UNKNOWN。
- 容器异常退出时主控补写的 meta.json 必须带 ctx.extra["operation_identity"],否则 stageworker 报 completed attempt metadata mismatch → 轨迹 PAUSED_INFRA 且续跑也过不去,只能作废补号(6a013d6 已修)。
- `bench execution cancel` 要等满轨迹自己的 cleanup_deadline_s(执行级 1800 秒),有操作停在 DISPATCHED 时就会等满 30 分钟,然后标"清理未确认"——不是卡死。
- 大模型请求断网(CONNECT_FAILED/REQUEST_OUTCOME_UNKNOWN)是设计上的"先暂停、不重试",整组账号一起暂停,靠 collect --watch 自动续(每条最多 3 次)。根治靠网络。
**更正(2026-10-12):只有 REQUEST_OUTCOME_UNKNOWN 和额度等待超时会自动续;CONNECT_FAILED 永远不会自动恢复(quota.py _resume_readiness),见 [[salty-throughput-drop-20261012]]。**
- Mini 整机会掉线约 1 分钟(10-10 16:00 实测 ping 不通);Wi‑Fi 是 5GHz 160MHz(信道 40),可能撞雷达避让。
- 04 里 iseg yolo11s.seg/yolo11x.seg/yolov8m.seg 的 kk0 PAUSED 是 10-08 旧行、不在 37 题清单里,不用管。
- 容器"无声死亡"(REMOTE_WORKER_FAILED,Salty 原因为空、日志为空):大模型编译(unimatch.flow ×4、mask2former.swin.b ×2、sea.raft、ocrnet)和 depth.anything 的 06-hbm(8G,2 分钟内死)。Salty 容器是 cgroup v2 且 memory.oom.group=1:任一子进程超内存就整个容器被杀、原因留空;worker 本身不往 stdout 打字,所以日志为空分不出死法。往 /tmp 写满约 39G 也会被驱逐(节点盘 99G 共用),但 06-hbm 死时 /tmp 还剩 54G,主因是内存。10-10 19:40 起 05-compile 32Gi、06-hbm 16Gi;dd20d19 起容器每 15 秒把内存/磁盘/最大文件记到 personal:/bench-stage-*/usage.log,失败时附进 stderr.log。
- 10-10 19:45 Mac mini 整机意外停机(无关机记录、无 panic),20:32 才重新开机;虚拟机不会自己起来,且因 ha.pid/vz.pid 的进程号被别的系统进程复用而显示 Broken。恢复:确认没有 limactl/VZ 进程后,把 ~/.lima/bench/{ha.pid,vz.pid,ha.sock,default_*.sock} 挪到 ~/lima-stale-20261010/,再 `limactl start bench`(别用 stop --force,会按旧进程号发信号)。
- 重启后上次开机的所有执行都会被隔离(OWNERSHIP_UNKNOWN / SIDE_EFFECT_UNKNOWN),这是有测试守着的设计,不改。处理:Salty 上孤儿作业全部 stop/rm、目录 rm -r;`bench execution reconcile` 列出非 TERMINAL 的;并行 `execution cancel`(有阶段在跑的几秒结束,只是暂停的要等满 30 分钟清理期限);configs/requeue-after-crash-20261010.py 补新 kk;旧开机的 lease 归档到 state/locks-stale-*;再起采集。reconcile 的 ready=False 来自历史上 TERMINAL+QUARANTINED 的旧执行,不挡采集启动。
- 监控脚本要报"连不上",否则掉线一小时都是静默(scratchpad/watch-prod.sh 已改)。
- 重启恢复已做成脚本:configs/recover-after-reboot.sh "<原因>"(先手动 salty stop/rm 孤儿作业和 bench-stage 目录,再后台跑它;它会 reconcile→并行 cancel→requeue-cancelled.py 补 kk→归档旧开机 lease)。10-10 22:34 第三次重启用它恢复,8 条作废补回。
- 按需加内存(df64aa8):默认编译 16Gi、06-hbm 8Gi,同一阶段容器 failed 过几次就翻几倍(上限 64Gi);统一给 32Gi 时共享队列只放得下 1 个编译。10-10 晚上队列很挤,6 个 16Gi 编译全排队,采集降到 05/06 各 2、04 一路。
- 容器无声死亡已坐实是内存超限:10-11 01:04 UTC retinanet.r101 编译(salty-f4b63fd8…,6核16G)死前 15 秒内存冲到 15.5G;复现作业 bench-repro-oom-20261011(salty-de281d92…)同样 failed、原因 "-"。平台侧要做的是 describe 原因里带 OOMKilled/Evicted/退出码。665bbef 起失败作业不再 rm,留给平台取证(要手动清)。
- 开不到 16/16/5 的真原因在 Mac mini(10GB 虚拟机),不在 Salty:① 调度程序内存涨到 5GB,10-11 07:08 05 的调度程序被内核 OOM 杀——Pi 每个 token 写一条 message_update,带两份整条消息(message、assistantMessageEvent.partial),events.jsonl 每条轨迹 300–800MB,budget_used 每次 read_text 整个文件。f1f44c9 修:Pi 不再写这两份快照(下游只用 type/delta),budget_used 逐行读。② 同一原因磁盘一天涨 16G(盘 78G)。③ 修完后每条轨迹在主控约 210MB(Pi 约 150MB),37 路约 8.7GB,太紧;10-11 起 05/06/04 = 12/12/4,要上 16/16/5 需把虚拟机内存调到 12GB(要用户同意,且需重启虚拟机)。
- 调度程序的 Python 代码只在启动时加载:改 harness/*.py 后,正在跑的调度程序仍是旧代码,必须排空重启(configs/restart-after-drain-20261011.sh <旧pid> <配置> <run> <并发> [job清单])。
- 10-11 用户要求"尽量用别处资源,Mac mini 只留最终数据集":configs/archive-finished.py 常驻(日志 logs/archive-finished.log),轨迹结束满 30 分钟后打包到 personal:/bench-root/archive/<run>/<job>/<kk>.tgz(容器产物在 personal:/bench-root/runs 已有同大小副本的不进包),核对后只删本地 .bc/spot npz/onnx 等。**坑:调度程序每结束一条轨迹就跑导出器(refresh_run_dataset),它重读每条轨迹的 agent/(SFT 原文)并按 sha256 重验 artifacts/<job>.hbm;删了它们导出会把 PASS 变 UNVERIFIED、SFT 变 0(10-11 12:00 踩过,已用 restore-agent.py / restore-hbm.py 从个人空间取回)。这两样必须留在本地,除非导出改到 Salty 上跑。**
- 10-11 10:22 虚拟机内存 10→12GiB(备份 ~/lima-bench-lima.yaml.before-12g-20261011;宿主机 16G、已用 3.4G 交换,别再加);之后 06/04/05 = 16/5/16。正常 limactl stop 时调度程序把在跑轨迹记成 INFRA_ERROR(不是隔离),recover 脚本不会补它们,要按 finished_at 那一分钟单独补号。
- c8a93c9 内存按题记:同一 job 任一轨迹该阶段 failed 过,就从当时内存(meta.salty_memory)的两倍起,上限 128Gi。
- 10-11 用户定:①导出挪到 Salty;②hb_verifier 没测就崩溃算"未验证"(2018b62,只对之后的轨迹生效,yolo11n.seg 那条历史 FAIL 不改);③监控交给 Codex,我不再挂监控;④120 已关机清理。
- 导出挪到 Salty(scripts/salty_offload.py,常驻在 Mac mini 上,取代 configs/archive-finished.py):每 10 分钟把结束满 30 分钟(UNVERIFIED 24 小时)、执行 TERMINAL、PASS 已审完的轨迹打包到 personal:/bench-root/archive/<run>/<job>/<kk>.tgz(之后的改动另传 <kk>.u<时间>.tgz,不覆盖旧包),然后删本地所有 >1MB 文件;有新包时最多每小时起一个 Salty 作业(salty_remote_worker.export)把包解到 personal:/bench-root/runs、跑 export/to_agentrl.py,数据集留在 personal:/bench-root/datasets/,再拷回 Mac mini 的 bench/datasets-final/(这是最终数据集)。调度程序自己每条结束后的本地导出(bench/datasets/)保留:它负责在文件还在本地时发现并审核新的 PASS,搬运要等它审完。
- Salty 上导出的坑:容器里 runs 不能是指向个人空间的软链接。verify.py 会 resolve 轨迹目录,并拒绝走软链接的预测文件,路径一对不上,G3 就变"未验证"、审核回执的绑定也跟着变(第一次实测 SFT 0 条)。现在 export() 在容器本地建真实的轨迹和阶段目录,agent/、artifacts/ 用软链接;解包不解 workspace(它是大头,导出不读)。修完后和 Mac mini 本地导出逐条一致(3/3/2)。最终数据集在 bench/datasets-final/,Salty 上一份在 personal:/bench-root/datasets/。
- 数据集版本门槛:旧网盘上 04/05/06 数据集的 min_agent_version 是 12;Mac mini 上的数据集是新建的,默认用了当前的 15,迁移前的 PASS 有 260 条(版本 12/14)会被直接丢掉。10-11 已用 `to_agentrl.py --min-agent-version 12` 改回 12(本地三份 filter_history 有记录;Salty 上的 manifest 删了,按本地的重新播种)。新建数据集时先查原批次的门槛。
- 10-11 迁移前的 291 条 PASS(04/05/06 早于 10-09 开始的)已从 36 经 Mac mini 中转到 personal:/bench-root/archive/(13:28–16:46,零失败;不带 workspace、.bc/.onnx、06-hbm 抽检 npz;中转脚本已删)。
- 10-11 补审:Salty 导出列出的"没审过/审核未知"44 条,用 configs/review-oldera-20261011.py 逐条取回 → 审 → 回执作为补丁包传回 → 立刻删本地(旧格式对话最大 650MB,读一条要 1.3G 内存,一次全取回会让三个调度程序的本地导出一起重读、撑爆内存)。结果 33 过、7 不过、4 未知:3 条是请求体 68–77 万字节超过审核账号 tpm 70 万的本地估算(字节当 token 算,改 collection_v1 要排空重启,等用户定),1 条是 10-08 旧服务器审到一半留下的 STARTED 回执(按设计不自动重发)。审核模型偶尔回格式不对(INVALID_REVIEW_SCHEMA 等),重试一次就过。
- 10-11 19:32 最终数据集 SFT 04/05/06 = 30/76/136(共 242;旧网盘 10-08 是 177)。迁移前 291 条 PASS 全部有判定:进 SFT 234、审核不过 46、未知 3、轨迹不完整 8。一次性脚本(中转、补审、restore-*)都已删。
- 10-12 用户确认审核模型 deepseek v4 flash 有 1M 上下文:三份配置新增额度组 deepseek-review-account-1m(规则照抄、tpm 70万→210万,覆盖 content_review 200 万字节的请求上限),reviewer 改走新组(原配置备份 configs/config-backup-20261012/)。额度账本按组名锁规则(同名改字段报 QUOTA_POLICY_CONFLICT),所以只能新开组。在跑的调度程序配置是启动时读的(lru_cache),下次重启才走新组;在那之前它们审长轨迹仍可能得 UNKNOWN(REQUEST_EXCEEDS_INTERNAL_TPM),用 scripts/salty_review_again.py 补(先停 salty_offload.py)。
- 补审工具 scripts/salty_review_again.py RUN CONFIG [--release-started] [--only ...]:逐条从存档取回 → 审(有 COMPLETE UNKNOWN 就按重试走)→ 回执作补丁包传回 → 删本地。STARTED 回执的坑:06 shufflenet.v2.x1.5 的 10-08 请求其实完成了(账本 generation COMPLETE、HTTP 200),是写回执时崩了;按原执行号重审会被账本当作"已完成"直接不发。正确放行是把 STARTED 记成 COMPLETE UNKNOWN(OPERATOR_RELEASED_STARTED_RECEIPT,原件进 history/),走重试换新执行号(工具 --release-started 就这么做,用户 10-12 批准)。
- 10-12 补审结果:04 yolov8m、mspn.r50,05 simplebaseline.r101、xfeat,06 shufflenet.v2.x1.5 通过;04 mobilenet.v3.large 不通过。迁移前 291 条 PASS 已无"未知"。
- 36 上 ~/restore-20261009 只留 .ssh(120 的两把私钥)、qwen-autofill/s100p-glm-8way 旧配置、resume-glm 日志(别处没有);账本/额度/raise-concurrency/node 已核对个人空间 controller-state.tar.gz 和 Mac mini 有完整副本后删除(备份只缺空锁文件)。~/restore-logs 是别的会话恢复家目录的核对记录,不是我的,别删。
- 导出的第二个坑(df51a7c):个人空间 runs 里还有容器替主控已放弃的尝试写的文件(xfeat 06-hbm a00 主控判超时,容器继续跑完写了 stdout.log),照搬会让门证据多出文件、审核绑定对不上。现在解包时把成员清单写进 .unpacked-* 标记,导出目录只按清单搭;HBM 从个人空间链接。
- 10-12 03:40 最终 SFT 32/79/139(共 250);迁移前 291 条:SFT 237、审核不过 46、轨迹不完整 8;迁移后新跑 PASS 17 条:SFT 13、审核不过 4。没有"未审/未知"。
- 每次 Salty 导出 45–75 分钟(搭目录约 30 分钟、个人空间读取时快时慢,05/06 各十几分钟),常驻程序等导出时不搬运;导出只在有新存档时发生,最多每小时一次。想更快就把 salty_offload.py 的导出间隔调大或减少阶段文件拷贝,目前没必要。
- 10-12 10:06 用户直接重启 Mac mini,没先停虚拟机,虚拟机被强停(last -x 里上次开机没有关机记录)。在跑的 37 条轨迹的 execution.json 恰好在那一秒(10:06:37)写入,结果变成全零或半截,尽管 atomic_json 有 fsync。reconcile、recover-rollout 诊断读到它们都会抛 JSONDecodeError,恢复脚本因此第一步就失败。采集程序本地的 06 数据集文件(datasets/*-training/*.jsonl)也被清零,但它每次刷新都整份重写,会自己恢复;datasets-final 没受影响。sqlite(jobs、quota)quick_check 正常。
处理办法:
1. 先清 Salty 孤儿作业:在跑的 stop+rm,已成功没取回的 rm,再 storage rm -r 对应的 bench-stage 目录;失败的作业留着。
2. 把坏的 execution.json 改名为 execution.json.corrupt-<日期> 留档。recover-rollout 只接受没有 execution.json 的轨迹。
3. 执行 `bench recover-rollout --run R --job J --kk K --apply --reason ... --confirm-owner-dead --confirm-work-stopped`,旧号会变成 ABANDONED,同时补一个 QUEUED 新号。
4. 再跑 configs/recover-after-reboot.sh,处理状态完好但没跑完的 3 条和旧开机留下的锁。
清单和日志:logs/corrupt-execution-1012.txt、logs/recover-rollout-1012.log、logs/orphans-1012.log。
**以后重启 Mac mini 之前先 `limactl stop bench`**,让虚拟机正常关机:在跑的轨迹会记成 INFRA_ERROR、文件不会写坏,但要按 finished_at 那一分钟单独补号。

View File

@ -1,96 +0,0 @@
---
name: net-resilience-board-deadline-20261014
description: 10-14 网络小故障拖垮轨迹的三处修复(原地重发/免费续跑/结果不明可续跑,等重启)、板卡清理没确认的根因与修复(已部署)、下次重启要上的完整清单
metadata:
node_type: memory
type: project
originSessionId: 2b85e929-b893-42c3-a7a0-3a72bda14705
modified: 2026-10-14T07:55:38.311Z
---
**网络(f83fa39,等重启)**
GLM 请求失败率约 0.3%,常成堆出现(10-14 12 点那一小时有 14 次连不上)。以前一次失败就让整条轨迹暂停,续跑 3 次的额度被网络故障用完后就卡死;回复中途断开的请求还会被自动放弃,同时取消整条执行(36 小时里 16 次)。
- A:broker 遇到确定一个字节都没发出去的失败(RETRY_UNSENT:CONNECT_FAILED / DNS_*),在原地等 cooldown_initial_s 后重发。每次把"等待 + 一次连接时限"计入 wait_spent,保证客户端超时的上界不变。
- B:只有最近一次请求可能已送达(最后一个 attempt 的 sent ≠ 0),续跑才扣次数。账本的 readiness 返回 counted;Execution.resume(counted=) 只在 counted 时检查上限、增加计数;pipeline 不再按原始次数预先过滤。
- C:reconcile 自动放弃时(resumable=True),把这一轮改成 PAUSED / REQUEST_OUTCOME_UNKNOWN,可以续跑并计次数,不再取消执行。人工放弃仍然永久隔离。续跑资格检查忽略已放弃的 attempt。
- 测试:test_quota 新增 2 个;test_quota_auto_abandon 改了("自己那一轮永不重放"的旧规定被有意推翻);test_execution 增加不扣次数的续跑。
**板卡清理没确认(8a14b41,10-14 15:50 已部署)**
10-12 到 10-14 共 18 条:
- 12 条是 07-board 卡满 1 小时、一张回执都没有。同一模型平时 0.2–2.6 分钟就完成,所以是真卡死。
- 3 条是重启时主动取消。
- 其余是回执 UNKNOWN。
板上证据(rdk-sandbox-40007,10-14 14:35 那次):
- 模型 2 分钟传完,测量程序 14:37:40 启动,BPU 加载后内核日志再无动静;
- guard-result 写的是 host_disconnected(容器被时限结束时)。
- 板子负载约 6,可能有别人在用。卡住的具体原因:测量进度日志随容器一起丢了,查不到。
系统问题:board_perf.bench 的时限,本地执行时会收进阶段截止时间,Salty 容器里没有执行状态,固定 3600 秒,比 07-board 的 1 小时阶段时限还长。结果容器先被结束,回执带不回来,只能判 BOARD_CLEANUP_UNCONFIRMED,整条轨迹作废。
修法:
- salty_remote 打包时把阶段截止时刻(墙钟)放进 BENCH_PHASE_DEADLINE;
- board_perf 预留 600 秒交回结果,到时作为普通 FAIL 结束,agent 可以重试。
- VM 和 Salty 时钟偏差约 1 秒。已核对部署后的打包里带着截止时刻。
还没做:板上进度停滞检测(例如 10 分钟没有 board_progress 就提前结束)。08-eval 每个样本批次可能好几分钟,窗口不好定,先不做。
**下次重启要部署的完整清单**(部署前和分支比对 md5):
- harness/stageworker.py(f74fe00,hbm 抽查的配置校验)
- agent/SYSTEM.md(f74fe00,hbm 抽查说明)
- harness/quota.py(e4e1513 DNS 可续跑 + f83fa39 A/B/C)
- harness/execution.py、harness/rollout.py、harness/pipeline.py(f83fa39)
- 智能体自主性(a92510d,见 [[agent-autonomy-audit-20261014]]):
- harness/stageworker.py、terminal.py、salty_dev.py(新文件)、salty_remote_worker.py、stages/s04_calibrate.py;
- bin/stageclient.py、agent/SYSTEM.md、agent/VERSION。
- **s04 和 worker 不能提前部署**:它们随阶段打包、部署就生效,而 s04 依赖新的 stageworker。
- 三份新配置都要加:
- salty_remote.stages 加 terminal;
- cpu_by_stage.terminal: 8,memory_by_stage.terminal: 32Gi;
- terminal.enabled: true;
- pools.dev.workers: 16(三份要一致)。
- SYSTEM.md 三个批次共用。04 不开的话,Qwen 会读到 bench dev 却用不了,所以要用户定 04 是否一起开。
**10-14 18:27–19:00 已重启上线(用户定:速度门槛 1.5 倍、04 一起开、马上重启)**
- 脚本和清单在 configs/restart-20261014/:
- stop.sh:SIGTERM 后最多等 30 分钟,让已在上板、评测的轨迹跑完;随后按新续跑规则用 targets.py 选出要取消的,并行取消;
- start.sh:先备份,再部署 26 个文件,补号,用 -v16 配置启动;
- restart-offload.sh:等 offload 空闲后用新配置重启它。
- 结果:
- 18:58 取消 35 条,其中 34 条在跑、1 条暂停但账本已作废;保留 7 条可续跑的暂停轨迹,重启后全部续上(仍是 v15)。等待那 30 分钟里有 3 条自然结束(1 PASS)。
- 19:00 三个 collector 起来,1 分钟内开始领题。
- 部署后和分支 9b7544a 逐文件比对 md5 一致。只有 scripts/bench_health_alert.sh 有意没部署:它是别的会话的飞书报警脚本。
- 新配置:configs/mini/*-v16.yaml,在原配置基础上:
- salty_remote.stages 加 terminal(4 核、16Gi);
- terminal.enabled: true;
- pools.dev.workers: 8。
- 踩坑:
- stop.sh 第一版用 grep 匹配 /proc/PID/cmdline,参数之间是 \0,匹配不上,安全中止,什么都没做;要先 tr 成空格。
- Mac 上没有 timeout 命令。
- g.sh 起 setsid 后台进程时 ssh 不会返回,要另开一条命令看日志。
- Qwen 组因取消时有一条请求结果不明,进入 RECONCILING,04 要等约 31 分钟(request_total_s 1800 + 60)才领题;GLM 两个组 RECOVERING。
**重启后头 40 分钟(19:00–19:40)**
- 恢复情况:
- 05/06 各 16 条、04 有 5 条在跑;保留的暂停轨迹全部续上。新轨迹都是 v16(提示词里有 bench dev 和速度要求)。
- 04 等 Qwen 组对账到约 19:31 才领题。
- 19:29 S600 组有一条 GLM 回复中途断开(REQUEST_CONNECTION_INTERRUPTED)。整组 RECONCILING 约 20 分钟,05 有 9 条暂停;19:40 按 C 方案自动作废那一条请求,暂停的全部续上。断线的根因仍是 Clash 走洛杉矶。
- bench dev 使用:
- 40 分钟 91 次:74 次成功,14 次是 agent 自己的命令出错,3 次容器 OOM(vit.h.14、deeplabv3plus 在 16Gi 里跑 hb_compile)。中位 78 秒。
- agent 拿它查平台 ONNX 和缓存:VM 上的 bench/cache 几乎是空的,缓存只在个人空间。
- Salty 上 bench dev 同时只有 3–8 个,没有排队。
- 当场修掉的两处(都不用重启):
- 67d306d:offload 的孤儿清理和作业自己的清理撞车,stop 返回 404 后会重试 30 分钟,这期间归档和导出都停着。改为跳过已清理的作业,404 当作已停。offload 已在 19:17 重启。
- d565796:bench dev 的内存取本模型正式编译需要量的下限(OOM 过就翻倍),失败时把容器的内存记录附进 stderr。terminal 每次都是新进程,部署后马上生效。
- 还没做:
- 速度门槛的强制判定:打算放到导出时按参考延迟过滤,不用重启。
- 补参考延迟:每题用默认配置编译并上板测一次,要占 Salty 和板卡时间,等用户同意。
- 开发环境大文件的清理。
**Why:** 用户要把这套系统当以后 RSI 的基础设施。原则:能在底层重试的不让上层知道;系统故障不扣模型的次数;结果不明就核对后续跑,不整条作废。
**How to apply:** 看到新的"暂停没恢复"报警,先用 resume_readiness 看 counted 和 reason,再判断是不是这几类。
相关:[[infra-leaks-20261013]] [[hbm-spotcheck-optional-20261014]] [[mac-mini-clash-global-proxy-20261013]] [[bench-restart-hazards]]

View File

@ -1,11 +0,0 @@
---
name: never-edit-running-script
description: 不要原地改正在运行的 shell 脚本;要改就写新文件或"新文件+改名替换"
metadata:
type: feedback
---
不要原地修改正在执行的 bash 脚本(open(p,"w") / sed -i 都会改同一个 inode)。bash 边读边执行,运行中的进程会在下一段按旧字节偏移去读已变化的内容,行为不可预测。
**Why:** 2026-10-09 我在 120 上原地改了正在跑的 `/root/salty-migrate-20261009/migrate.sh` 的 up(),"salty迁移"会话指出后只能结束主 bash,改用新脚本 phase2.sh / verify.sh 接手,我的改动也就作废了。
**How to apply:** 脚本已在运行时,改动写进新文件(或写好后 `mv` 替换,旧进程继续持有旧 inode),并明确下一轮用哪个文件。相关:[[salty-personal-migration-20261009]]

View File

@ -1,84 +0,0 @@
---
name: overnight-check-20261015
description: 10-15 早上复查:dev 槽 8→37 已改;S600 板盘满(失败会话目录留板不删);暂停名额被新题抢;全 fp16 落 CPU 仍判过;4 条自带图/校准提交
metadata:
node_type: memory
type: project
originSessionId: cadbef09-8744-472b-81b1-a1ff37b0a735
modified: 2026-10-15T01:41:51.665Z
---
10-15 07:50–08:10 复查(v16 上线约 13 小时,50 条 v16 轨迹)。
- dev 槽 8→37:三份 -v16 配置一起改,07:50 生效,备份在 config-backup-20261014/before-restart-20261014/*.before-dev37。
- paths.config() 有 lru_cache(09-22 合并带进来的),每个进程只读一次配置。改 workers 只对新起的进程生效;已在排队的还按旧槽数等。scheduler.acquire 里「每轮重读 config」的注释已过时。
- 之前 8 槽被 agent 自设 3–12 小时的长编译占住,一夜 199 条 dev 命令排到超时(STAGE_EXECUTOR_EXIT),04–05 点比成功的还多。
- 底层问题:
- S600 板 /userdata 两次写满(01:31–03:25、06:50–06:56),05 的 07-board/08-eval 报「No space left on device」却记成 unknown,像 agent 的错。都自己恢复了。
- 根因:board_perf.py 的 close() 在失败时把板上会话目录「留作诊断」,从不删。日志里 10-13 起留了 9 个(S600 上 3 个:dedode.v2、gmflow、yolo11m.pose)。
- 再加上 solo.r50 一次评测输出 12 GB。
- 板卡密钥只在 Salty 容器里,主控上看不了板子剩余空间。
- 已修(a263374,10-15 09:41 热部署,阶段容器每次打包现读,不用重启):
- 清理已确认的失败会话照样删;
- 清理没确认的只留空目录和回执;
- 每个新会话先删 6 小时没人动的 eval-* 并在阶段 stderr 记「removed stale board sessions」;
- No space left on device → disk_full(瞬态)。
- 不能直接删「活」目录:board_guard 的 finally 先往会话目录写 guard-result.json,再更新 /userdata/bench/.board-inference.quarantine.json。目录没了就抛异常,标记卡在 PENDING,这块板以后的推理都会被拦。test_failed_guard_retains_evidence_without_erasing_live_directory 守的就是这个。
- 上线后核实(09:41–10:40):板上测试和评测全部 OK,没有磁盘满。3 块板的清扫共删了 31 个旧目录(S100P rdk-sandbox 18 个,两块 S600 各 10 和 3 个),比日志里看到的 9 个多,多出来的应该是容器被杀留下的。
- reward.py 只对 board_offline 免罚,disk_full 还没加;这是判分改动,跟导出那批一起等用户。
- 板上评测分批(f56ac2f)11:08 热部署后出事,11:45:38 回退(分支上 2b02417 撤销):
- harness/eval/<任务>.py 的 sha256 在建 run 时冻进 job.task_contract,每个阶段开头都核对(stages/base.py:261)。
- 改了 seg/mde/flow/feat/coco_extra.py 之后,这些任务的所有阶段都报 frozen task implementation changed。
- 板卡阶段在派发前失败、没有回执,会被判 BOARD_CLEANUP_UNCONFIRMED,整条轨迹作废。
- 损失三条:04 solov2.r50、05 deeplabv3plus.resnet50(自带图那条)、06 yolov8m.seg(阶段全失败后判 FAIL)。另有 3 条在跑的白白失败了几次阶段。没有编译失败,不占编译次数。
- 规则:任务评测文件不能直接热改;要么建新 run,要么把精确的「旧指纹 → 新指纹」写进 harness/eval/__init__.py 的 EQUIVALENT_SOURCES(只限结果完全等价的改写)。common.py、board.py、run.py、board_perf.py、errors.py、pipeline.py、adapters/ 不在指纹里。
- 另一个坑:板卡阶段派发前的任何 harness 异常都会被当成清理未确认,害死整条轨迹,应该写 not_started 回执。还没修。
- 断线暂停的位子(10-15 11:32 用户同意换回):
- 取消了 11:21 新开的 4 条。deeplabv3plus.resnet101、fcn.resnet50 11:33 续上,deeplabv3plus.resnet50 11:37 续上,efficientvit.seg.b1 还在等。
- 11:33 又有 2 个位子被新题抢了。根因是 pipeline.py:813:同一轮里有一条就绪,新题就会跟着进来。
- f191baf(_claimable:有暂停在等时这一轮只续跑;额度组 PAUSED/RECONCILING 一律算在等)11:44 已部署,collector 重启后生效。
- collector 重启会杀掉它管的在跑轨迹(配额管理在进程里),不能随手重启。
- deeplabv3plus.resnet101(05)11:34 PASS:第一条真自带图的通过。
- G3 mIoU 32.53→31.80,降 2.24%,门槛 5%;G2 0.986 没过但不是必选;板上 506 ms。
- 「暂停名额被抢」又发生:00:56–01:11 GLM 断线,05 停了 12 条。01:17 一起续跑,6 条因为额度还没恢复(RESUME_NOT_READY: WAITING_QUOTA)又被暂停,空出的位子给了 6 道新题。两条 efficientvit.seg.b1/b3 停了 7 小时还在等。修要改 collector、要重启。
- 离线归档:06 zoedepth.n kk2 的包 23:00–00:12 一直传不上(网络 body error),跳过了。
- Salty 上留着 327 个失败、94 个成功的 stage 作业没清(317 个在重启前)。不占 CPU,可能占个人空间。
- collector 无 traceback;VM 负载 15、内存剩 4G、盘 60%。
- 自主性:
- 4 条编译用了自己的东西,都过了 I/O 检查:
- maxvit.t 把 Einsum 改成 MatMul 的图;
- deeplabv3plus.r101 把 2560→512 的 1×1 卷积拆成 5 个绕编译器 bug;
- vit.h.14 自做校准集(两次)。
- 还没有一条走到结论。
- 12/50 写了产图脚本,9 条跑通。
- 速度门槛只在提示里:05 efficientvit.seg.b0 最终全 float16,板上 4.1 秒一帧、BPU 利用率 0.003%,照样 PASS。导出时按速度过滤还没做。
- 板卡派发前失败误判(383710c,10-15 12:20 热部署):
- base.Stage.run 本地执行完、board 池、非 SKIPPED、回执目录为空时,补一张 not_started(VERIFIED)回执。
- 依据:board_perf.bench 在第一条板上命令前就写 UNKNOWN 回执。控制端包装 Salty 的那层拿不到这个证明,不补。
- 待办总表(10-15 中午,按生效方式分):
- 热部署已完成:板上会话目录清扫、disk_full、dev 不可见评测集、dev 去重复上传、dev 37 槽、派发前误判。
- 要重启 collector:暂停名额修复 f191baf(已部署文件)。重启会断在跑轨迹,等计划窗口。
- ~~要新建 run:评测分批 f56ac2f~~ 15:55 已不建新 run 上线(6ab088d 逐对批准的等价改写,见 [[restart-20261015]])。
- 要新 agent 版本:SYSTEM.md 删「评测数据只读可见」;速度门槛(用户暂缓)。
- 外部:Clash 走洛杉矶导致断线(用户侧)。
- 遗留债:
- salty_remote.py:294 失败作业和目录永不清(327 个);
- s04 自带图大于 2 GB 的外部权重不哈希;
- paths.config 有 lru_cache,scheduler 注释过时;
- zoedepth.n 归档包上传失败;
- agent 在长 dev 编译上同步干等几小时;
- reward 里 disk_full 没免罚。
- 热部署前必查:
- 改的文件在不在 task_contract 指纹里(harness/eval/<任务>.py);
- 在不在常驻进程里(collector 不重载);
- VM 和 Salty 容器各跑一遍相关测试,base 和 new 对比。
**Why:** 用户要这套当 SFT 数据和以后 RSI 的底座:底层错误不能算到 agent 头上,速度兜底不能当通过。
**How to apply:**
- 下次重启要带上:
- 续跑优先于新认领;
- 额度没好时别把位子让出去。
- 清 Salty 旧作业要先问用户。板上残留目录由会话进入时的 6 小时清扫处理,用户 10-15 已同意。
相关:[[agent-autonomy-audit-20261014]] [[net-resilience-board-deadline-20261014]] [[selfheal-and-alert-20261012]]

View File

@ -1,15 +0,0 @@
---
name: plain-language-no-jargon
description: 汇报要用大白话,不要用英文行话或比喻性术语(金丝雀、canary 等)
metadata:
type: feedback
---
用户原话:「你要让我看的话,那你就说点人话,好不好?全是一些黑话,什么金丝雀,什么鬼?你说的那些我完全听不懂。」
**Why:** 用户是项目负责人,关心的是「能不能开始收集数据了」这类结果,不是实现术语。
行话让他无法判断进展,等于汇报无效。同时他对长篇分析而不出结果非常不满。
**How to apply:** 用「先拿一个试,确认好用了再动其他的」代替「金丝雀」;用「本机硬盘 / 网盘」代替
「local disk / JuiceFS」。先给结论和数字,再给理由。能用一张小表说清就别写段落。
承认没做好的地方要直接说,不要辩解。参见 [[prefer-subtraction-over-addition]]。

View File

@ -1,32 +0,0 @@
---
name: poweroff-20261016
description: 10-16 19:29 Mac mini 断电前停机(取消 36 条、停 Salty 35 个、limactl stop)→ 20:57 开机后补号、原命令/环境拉起 5 个进程、垃圾已清;暂停轨迹跨开机续不了要一起取消;现成脚本和坑
metadata:
node_type: memory
type: project
originSessionId: c8e679af-e7a3-4129-a0e4-e1821fa37ce1
modified: 2026-10-16T13:01:01.011Z
---
10-16 19:29 用户要给 Mac mini 断电;20:50 开机后让我把 VM、进程、任务按原进度拉起并清垃圾。
**停(19:29–19:31)**:VM 里 /mnt/data/ming01.wang/configs/poweroff-20261016/stop.sh——存 3 个 collector + offload + speed_ref 的 environ/cmdline/日志/cwd → SIGTERM → 用 restart-20261016/targets.py 选出 36 条立即取消(不等上板/评测)→ 34 CANCELLED、2 INFRA_ERROR;04 的 5 条可续跑暂停保留。collector 自己退出、没卡导出。Salty 上 bench-* running/pending 35 个 `salty stop`。`limactl stop bench` 正常关机。
**起(20:57)**:同目录 start.sh——挪走旧开机的 lease → requeue.py 补号 36 → withenv 按存档原命令拉起 05/06/04/offload/speed_ref(env 自带 BENCH_CONFIG:04 qwen-s100p-r04-v22、05 s600-r05-v22c、06 s100p-r06-v22)。几秒内就开始领题,21:00 是 04 5/5(+5 暂停)、05 16/16、06 16/16。
**新踩的坑(下次停/起照做)**
- Mac mini 开机时 launchd cc.d-robotics.bench-controller.vm(~/work/controller/start-existing-bench-vm.py,只 RunAtLoad)会自动把 VM 拉起,但不起任何进程。
- 取消正在跑 dev 命令的轨迹会留下 state/locks/dev-*.lease.json(这次 24 个)。_lease_settled 只认 DONE+VERIFIED,所以不会自动释放、会一直占 dev 槽。重启后要把 holder.boot_id 不是本次开机的 lease 挪走(start.sh 里有)。
- speed_ref 被中断的题会留下 runs/zz-speedref/<job>/rollouts/00/stages/ref/salty/bundle.tar,再跑时 FileExistsError;异常路径不写 FAILED,所以删掉 stages/ref 和 _dev-sync.* 就能重算。
- offload 一启动就会把 04/05/06 停掉的 Salty 作业连目录一起回收(35 个里回收了 31 个);speed_ref 和导出的作业不归它管,要手工 `salty rm` + `salty storage rm -r personal:/<名字>`(cleanup.sh)。
**更正(10-16 22:45 体检发现):可续跑的暂停轨迹跨开机续不了**
- 断电时保留的 04 两条暂停(dpt.hybrid #5、simplebaseline.r50 #5)开机后被标成 OWNERSHIP_UNKNOWN 隔离,健康检查每 10 分钟报"暂停 N 分钟没恢复"。
- 原因:execution.process_status 对不是本次开机的进程一律返回 UNKNOWN(同一主机名也一样),reconcile 就隔离;Execution.resume 遇到开机变了直接拒(BOOT_CHANGED_RECONCILE_REQUIRED),deadline_monotonic 是按开机计时的,跨开机也没意义。
- 22:48 用 fix-stuck-04.sh 取消并补号;取消要等清理旧开机的子进程到 cleanup_deadline_s(1800 秒)才返回。
- 以后断电/重启 VM:可续跑的暂停也要一起取消(targets.py 里 KEEP 的那些),否则开机后会卡死。代码上可以让"同主机、不同开机"的进程算已死,但要重启 collector 才生效,等下次统一上线。
**清理(用户要求)**:Salty 剩下 4 个作业和目录、两道 speed_ref 的残留、挪走的 24 个 lease、含密钥的 env-* 存档、jobs.sqlite 备份和临时文件都删了;目录里只留脚本、cmd/log/cwd、名单。
**Why:** 10-12 用户直接重启没停 VM,37 条 execution.json 写坏;这次先取消再关 VM,文件和两个 sqlite 都完好(quick_check ok)。
**How to apply:** 再断电就复制这个目录的 stop.sh/start.sh/cleanup.sh 改 PID 和名单用;见 [[bench-restart-hazards]] [[mini-salty-controller-20261009]]。

View File

@ -1,17 +0,0 @@
---
name: prefer-subtraction-over-addition
description: 修复问题时优先删代码/消除不一致,而不是新增扩展点、配置项或抽象层
metadata:
type: feedback
---
用户明确要求:「不一定只关注于做加法,而且要做减法。其实更主要的是要做减法,我不喜欢就是你来不来就做加法。」
同时强调鲁棒性、优雅性、简洁性。
**Why:** 给一个本不该存在的默认值打补丁(例如新增只有一个子类使用的钩子),是在掩盖问题而不是消除它;
代码库会因此越积越重。2026-09-18 修 JuiceFS 编译卡死时,第一版方案是给 `Stage` 加 `run_cwd()` 钩子,
被用户否掉;改成删掉 `cwd=stage_dir` 这个从没人复核过的默认值,净 -10 行、0 新增抽象,反而更对。
**How to apply:** 提方案前先问「能不能靠删掉某个东西来修」。诚实报告增删行数与新增抽象数——用户会核对。
把顺带发现的死代码、永真守卫、过时配置一并清掉,但不要让范围膨胀到另一个 bug 上。
参见 [[plain-language-no-jargon]]。

View File

@ -1,74 +0,0 @@
---
name: quota-unknown-autorecover-fix
description: 请求不明自动恢复(2f54d40)+板卡死锁自动释放(80a8e24);04/05/06 在跑;板卡核对/释放记录位置
metadata:
node_type: memory
type: project
originSessionId: 64b486b6-4970-404f-9b9a-46e9ca4808d3
modified: 2026-10-03T14:30:37.255Z
---
2026-10-03 已推送到 gitea `production/s100p-infra-fix-20260928`:`2eb29ca`(现场原有的 recovery 隔离修改)、
`2f54d40`(请求结果不明时自动恢复 + 单条任务只隔离、采集器不退出)。
服务器上没有 git 身份和推送凭证:提交时用 `-c user.name=ming01.wang -c user.email=ming01.wang@d-robotics.cc`,
推送时在服务器上打 bundle,传回 Mac 再推(Mac 有 gitea 凭证)。
- 06:21:06 用新代码重启,运行正常。
- 04/05:21:5x 发了 SIGTERM 排空,由 `/home/ming01.wang/bench-relaunch-20261003/relaunch.sh` 等旧进程退出后在原批次上重启
(新日志 `resume-glm-20261002-104649/restored-<run>-fix-20261003.log`)。
之后 `bench-relaunch-cleanup.sh` 删除重启脚本目录,结果写入 `/home/ming01.wang/bench-relaunch-20261003.result`。
**下次先看这个 .result 文件,确认 04/05 已经重启。**
- 04/05 当时实际写日志的位置是 `/home/ming01.wang/sampling-recovery-20261003-1125/{qwen,s600}.log`,
而不是 resume-glm 目录里的旧日志。
- UniMatch r20260929-06#00:经人工核对(583 个进程全部已死,板端评估回执 VERIFIED),数据库结束为 INFRA_ERROR,
释放了 agent-7、cpu-4、cpu-6 三个槽位;核对记录在 rollout 目录的 `operator-release.json`。
坑:经 ssh 单引号传 heredoc 脚本时,脚本里的引号和中文会出问题,写成文件 scp 过去再执行;
`pkill -f`、`pgrep -f` 会匹配到自己所在的 `bash -c`,等进程退出时按 PID 用 `kill -0`。
网盘上 `git status --ignored` 慢到不可用。重启要点见 [[bench-restart-hazards]]。
2026-10-04 00:2x–00:48 进展:按用户要求取消了 04/05 剩余任务。取消后旧采集器卡在退出前的数据导出上,
确认"运行中"记录为 0 条后用 kill -9 结束。05 于 00:21 重启。04 缺 `QWEN_EVAL_API_KEY`,用户提供后已写入
`.env.local`,00:39 重启。Qwen 账户自动解除暂停,试探请求成功,进入恢复中。
并发:06、05 各 8 路(GLM-5.3),04 为 5 路(Qwen3.8-27B),全局 Agent 槽位 24。
04 的 RetinaNet-R50 是自己的请求结果不明(INCOMPLETE_GENERATION),按设计永远停在暂停状态。
2026-10-04 08:00 板卡锁死事故:8 块可用板卡全部被已结束任务占着(占用进程已死、清理回执缺失),
从 10-03 23:19 起没有任务能上板,排队满 4 小时报 QUEUE_DEADLINE_EXCEEDED。逐块核对(进程死、任务已结束、板上只有 sshd)后释放,
核对记录在各 rollout 目录 `operator-release-board-<slot>.json`。修复 `80a8e24` 已推送:`scheduler.reconcile_leases` 遇到这种情况时
会用 ssh 检查板卡,确认空闲后自动释放,并写入 `board-release-<slot>.json`。
旧采集器没有重启;临时用 `~/board-reconcile-bridge.sh` 每 10 分钟跑一次,三个旧 PID 都退出后它自行退出并删除自身,日志写到 `~/bench-board-reconcile-bridge.log`。
板卡 ssh 时断时连的原因:板端 sshd 达到 MaxStartups 上限("3 of 3-10 startups")。另外 `~/.ssh/known_hosts` 里 S600 的主机指纹已过期,
连板卡要用密钥目录下的 known_hosts。
2026-10-04 09:00 板卡链路三处根因修复(均已推送,上板步骤每次都是新进程,无需重启采集器即生效):
- `f315496` 板端 `/userdata/bench/.board-inference.quarantine.json`:guard 被杀后这个标记会永久留在板上,之后每次评估都返回 78
(40009 因此被锁 1.5 天)。现在若板上已没有任何进程带着该次评估的标识,guard 会自动清掉它。
- `f61d2ef` ssh 在登录前就被断开(kex_exchange_identification)时,`board_perf.run`/`guarded_run` 最多重放两次。
- 板端 sshd 的 MaxStartups 改为 32:30:100(09-24 改过 40009/40007,本次补齐 40011/40015/40016 和 BOARD_HOST)。
原因是公网端口常年挂着 15–35 个扫描器的未登录连接。每块板的备份和记录在 `/root/bench-ssh-recovery-20261004/`。
- 不需要 S600 密码;用 harness 的密钥,加上密钥目录下的 known_hosts 即可。`~/.ssh/known_hosts` 里的旧指纹 harness 不会读取,不要拿它判断板卡状态。
- 06 那 3 条暂停(yolov8l.seg、两个 DA-v2-large):阶段其实已结束,是回执校验误判。修复 `44f643d` 已推送:
嵌套进程组校验和 owned_processes 统一走 `execution.scope_cleaned`;排队超时、从未拿到板卡的上板阶段不再要求板卡回执。
这 3 条用新代码补登记阶段结果后已结算为 CANCELLED(清理确认),释放 agent-12/18/20。
注意:隔离状态是单向的,处于"清理未确认"的暂停任务只能结算,不能恢复。
2026-10-04 10:03 用户要求立即重启让修复全部生效:取消了正在跑的 21 条(结案为 CANCELLED,原因 USER_APPROVED_IMMEDIATE_RESTART_20261004);三个采集器在原批次上重启,06 于 10:12、04 于 10:17、05 于 10:23 重启(运行的是 44f643d),10:38 恢复满并发 5+8+8。过渡用的 bridge 脚本已停止并删除。
2026-10-04 `4332f43`:编译次数成为每条轨迹的主预算,最多 3 次;`106a96a` 应用户要求改为 5 次(VERSION 14)(`stagerun.MAX_COMPILE_ATTEMPTS`,改它要同步改 e2e.md)。
计数来源是 stages 表里 05-compile 中 lane!='queued' 的行,续跑不会清零。第 4 次提交编译直接返回 COMPILE_BUDGET_EXHAUSTED。stagerun 每个阶段都是新进程,所以不用重启就已生效;提示词只对新开始的轨迹生效。
历史数据:142 条 PASS 中有 19 条(13%)用了 4 次以上编译;如果上限设成 4,只损失 4 条(约 3%)。
提示词模板、SYSTEM.md、VERSION 每次准备新任务时都从磁盘读取,stagerun 每个阶段都是新进程,所以改这几处不用重启采集器;只有改采集器进程里的 Python 代码才需要重启。
2026-10-08 `ea0335b`:04 的 Qwen 账户从 10-07 08:26 起停在 PAUSED 达 15 小时。原因是自动作废那个结果不明的请求时,账户里另有一个正常请求还在进行,于是 probe_allowed=False,之后再也没人检查。
现在作废、确认结束、reconcile 三处共用 `_probe_if_settled`:只要账户里没有未结束的请求,就放行一次试探。
00:00 在原批次上重启(取消了 16 条正在跑的任务,04 暂停的 5 条从检查点续跑)。00:15 又出现一次结果不明,账户按新逻辑自动恢复。
新采集器启动耗时变长:05 约 25 分钟,06 约 38 分钟。
2026-10-08 两处系统终止修复(已推送):
- `5617f57`:bash 没写超时或超时超过上限时,pi_session.js 在执行前自动设成"系统上限减 60 秒"(约 14 分钟),超时只让这一条命令失败,轨迹继续;落盘的模型调用参数不变。SYSTEM.md 加了说明(VERSION 15)。每条轨迹都会新起 pi 进程,所以不用重启就生效,04 也已生效。
- `e2bdd24`:之前说的"GLM 回复不完整"68 条是误判。其中 57 条是本地配额代理返回 413 REQUEST_EXCEEDS_INTERNAL_TPM:代理按字节数估算 token(约是真实值的 3.7 倍),单次请求超过 tpm 就直接拒绝,所以上下文一过约 13.5 万 token 轨迹就死;另外 11 条是重启期间代理不可用。05/06 两个 GLM 组的 tpm 从 50 万调到 200 万(和 Qwen 组一样)。生产配置(/home/ming01.wang/s100p-glm-8way-20260928-2124/s100p.yaml、s600.yaml)、账本 quota_groups.policy 和仓库 config.yaml 默认值都已同步修改。
注意:采集器运行期间改配置文件会触发 RUNTIME_CONFIG_CHANGED;账本里存的策略和配置不一致会触发 QUOTA_POLICY_CONFLICT。所以必须先停采集器,再同时改配置和账本。
- 07:38 按老规矩只重启了 05/06:取消 16 条(原因 USER_APPROVED_RESTART_20261008_GLM_TPM,其中 3 条已在校验,按实际结果结案)。06 于 07:46、05 于 07:51 在原批次上重启;04 没有动。
2026-10-08 `10b070e` 改了"连坐"规则(pipeline._collection_snapshot 的 SQL):同一任务里有一条被系统原因终止(命令超时、基础设施错误、取消、丢失、暂停后过期),不再挡住其余排队的轨迹。仍然会挡的只有两种:预算用完(BUDGET_EXHAUSTED),以及 verdict.timeout_reason=EXECUTION_DEADLINE_EXCEEDED(真正干满 24 小时)。正在跑或暂停中的同伴仍然一条一条排。用户确认的分类:系统原因 379 条放开,7 条继续挡。"导出必然失败"这类 agent 自己判 FAIL 的,本来就不挡,用户决定先不加规则。改之前被挡 1107 条(04 267、05 409、06 431),改后放出 1086 条。只重启了 06(09:58,取消 8 条);04/05 等下次重启时生效。重启时有一个取消命令卡在网盘 IO 上 20 分钟,轨迹其实早已结束,kill 掉之后脚本继续。

View File

@ -1,29 +0,0 @@
---
name: qwen-ability-failures-20261013
description: 训练前 Qwen 真正因能力做不出的题清单与原因(对照 GLM 怎么做成的);训练前后对比应盯这组题,而不是已通过的题
metadata:
node_type: memory
type: project
originSessionId: 6c848f33-e8d9-4981-ade2-dc77987005d7
modified: 2026-10-13T03:09:29.425Z
---
2026-10-13 用户明确:对比训练前后,要看的是**训练前 Qwen 因模型能力没通过的题**,已通过的题看不出提升。
**10-14 12:30 更新:能力失败共 6 道**(34/37 有结果、过 28;系统失败不计,depth.anything.v1.small 的 FAIL 是 06-hbm 超时,已剔除)。新增三道:
- retinanet.r50.fpn(k4,3 次编译):板上 mAP 达标(+0.3%),G1 hb_verifier 余弦 0.66<0.99(P6/P7 头)。想调校准时把 per_channel/node_config/calibration_type 写进 compiler_parameters、自造 calib/quantization 段,stageworker.snapshot_config 白名单拒收(报错只给段名),Qwen 误判"平台不让改校准"。GLM-06 过 1。
- simplebaseline.r50(k4,5 次编译):逐步把骨干改 int16,oks_ap 只掉 0.7%,但附加约束 keypoint_mean_shift_px_max=2 实测 2.33。GLM-06 过 1 败 1。
- efficientvit.seg.b1(k3,5 次编译):int8 mIoU 5.1,全 int16 24.7(要 ≥29.1);Qwen 归因板端运行时、未验证。GLM-06 过 2,各 2 次编译。
共同短板第三条:把自己的问题归到平台上。
**明确能力失败(verdict FAIL、系统正常)**:
- mobilenet.v3.small(04 k1,5 次编译用完,板端 top1 2%):默认 int8 → verifier 0.983;之后尾部 int16 更差、尾部 run_on_cpu 无效且板端输出坏掉;第 4、5 次把 `activation.asymmetric` 等写成 hmct 不认的格式直接崩溃,白烧两次。全程 0 次本地试验。GLM-06 三条全过:k0 本地做了 28 次 hmct/ORT 试验找敏感节点,9 个节点 int16 后过(4 次编译);k1 本地试出全 int16 后 2 次过(全 int16 是验证器漏洞,见 [[verifier-audit-20261012]])。
- solov2.r50(04 k0,10-07,1 次编译、mask_ap 掉 7.4% 就交失败;会话没迁过来,看不到为何停)。GLM-06 k0:首编掉 10.1%,本地 9 次试验定位 head/neck 里 GroupNorm 分解链,neck+heads int16 后掉 2.0% 过(2 次编译)。
- yolov8n.seg(04 k1,5 次编译,mask_ap 掉 21.6%):Qwen 正确定位到 proto/mask 分支,但 int16 混合更差、run_on_cpu 被 HzSwish 转换挡住。GLM 也从没过过——是老师也做不到的上限题。
**很可能是能力问题(10-08 前 24 小时到期时精度还差很远,04 旧轨迹)**:YOLO 检测一族 det--yolov8n(掉 42%)、yolov8s(39%)、yolov5s(40%)、yolo11x(38%)、yolo11n(6 次编译仍掉 5.6%);flow--sea.raft 崩坏(GLM 也全败)。convnext.base 旧轨迹掉 9.6% 但后来过了,不算。其余 04 的 DEADLINE_EXCEEDED 大多 0–3 次编译、没有精度证据,多半是当时系统慢。
**能力差距的具体表现**:难题上 GLM 先在自己 workspace 里用 hmct API / onnxruntime 本地试验选参数,再花正式编译;Qwen 直接拿正式编译次数去猜,还会写错配置格式。全体会话里"做过本地试验"的比例两边差不多(Qwen 14/38、GLM 32/97),差别只出现在难题上。
**Why:** SFT 用 GLM 轨迹,能学到的正是"先本地诊断再编译、写对配置"这类做法;训练后 Qwen 在这组题上通过、或编译次数减少,才说明训练有效。
**How to apply:** 训练前后对比报告以这组题为主;老师也做不到的(yolov8n.seg、sea.raft)单列为上限题。用户要扩题组时,把"很可能"那几道 YOLO 检测题给训练前 Qwen 各补跑 2 条确认。相关:[[qwen-sft-benchmark]] [[verifier-audit-20261012]]

View File

@ -1,19 +0,0 @@
---
name: qwen-nvfp4-service-speed-20261012
description: 10-12 实测 04 用的 Qwen 服务(114.111.18.115:18080 NVFP4):单请求快、但一次只处理 1 个请求;04 开 5 路时服务只忙 15%,不是瓶颈
metadata:
node_type: memory
type: project
originSessionId: cbd0a330-0959-4662-b4f7-9df1eaf61d31
modified: 2026-10-12T07:42:08.439Z
---
2026-10-12 15:40 实测 04 的 Qwen 服务(`http://114.111.18.115:18080/v1`,模型 qwen3.8-27b-nvfp4,配置 /mnt/data/ming01.wang/configs/mini/qwen-s100p-r04.yaml,额度组 qwen38-nvfp4-selfhost-11411118115-18080,max_concurrency 5):
- 单个请求:输出约 96 token/秒;1.5 万 token 的全新输入首字 3.2 秒。前缀缓存命中 96%,04 实际请求平均输入 6.4 万 token、输出 583 token,首字平均 1.9 秒。
- 账本(attempts 表)里 04 实际请求:中位 5.6–6 秒、p90 17–22 秒(旧原版服务中位 39 秒,GLM 3.7 秒)。
- **服务一次只处理 1 个请求**:同时发 4/8/16 个,总速度都停在约 65 token/秒,首字等待线性涨到 15/28/48 秒(最慢 88 秒);/metrics 里 running 始终 ≤1,排队原因是 capacity。显存够同时跑约 10 个满长度请求,所以多半是启动参数 `--max-num-seqs 1`。我没有这台机器的登录权限,要改得找服务负责人。
- 但现在它不是瓶颈:04 开 5 路,最近 6 小时 383 个请求,服务只忙 15%;每条轨迹花在等模型上的时间约 3–5%,其余是编译/HBM/上板等工具。
- 10-08 18:00 到 10-12 15:35,37 题基准共开 44 条,有效结果只有 12 条(10 PASS、2 FAIL),27 条被取消/作废/系统错误(主要是 Mac mini 重启和换代),5 条在跑。
**Why:** 用户以为 04 "一题一题做"、服务太慢。实际是 5 路并行,慢在单条轨迹要几小时、加上大量轨迹被系统打断。
**How to apply:** 04 想提速,先在全局上限 40 内把 04 提到 8 路(16+16+8,不用动 05/06);超过约 15 路、或服务忙碌超过一半时,才需要让服务负责人调大并发。用户可以自己挑题,重启 04 时换 `--job` 清单即可(configs/restart-after-drain-20261011.sh)。相关:[[qwen-sft-benchmark]] [[mini-salty-controller-20261009]] [[bench-restart-hazards]]

View File

@ -1,39 +0,0 @@
---
name: qwen-sft-benchmark
description: 用来比较原版 Qwen3.8-27B 和 GLM-5.3 SFT 版的 benchmark qwen-sft-v1:37 道题、规则、04 的跑法、训练数据污染情况
metadata:
node_type: memory
type: project
originSessionId: 61038635-ef70-484b-8f19-363102d54e44
modified: 2026-10-12T12:07:05.163Z
---
2026-10-08 建立 benchmark qwen-sft-v1,题单冻结在 `bench/benchmarks/qwen-sft-v1.json`(提交 c2c52ed)。
- 共 37 道 S100P 题:
- 对照 4 道:GLM 一次编译就过。
- 中等 26 道:GLM 要编译 2–9 次才通过。
- 难题探针 7 道:姿态 mspn.r50;深度 depth.anything.v1.small、v2.small、dpt.hybrid;实例分割 yolov8n.seg、yolo11m.seg;语义分割 fast.scnn。
- 规则:每道题 2 条;只计 2026-10-08 07:30 之后开始的轨迹;被系统打断的不算模型失败,要补跑。指标分档报:通过率、编译次数、轮次、工具报错率、扣掉等模型回复后的用时。
- 原版 Qwen 的跑法:04 现在用的配置是 `/home/ming01.wang/raise-concurrency-20261008/qwen-s100p-r04.yaml`(全局上限 40)。10-08 18:0x 用它重启,加 37 个 `--job` 只跑这些题。Qwen 配额组的三个超时放宽为 first_byte 600、stream_idle 300、request_total 1800,新旧两份配置和账本策略三处都已改成一样。17:50 那次我误用了旧配置路径重启,18:0x 已纠正,期间没丢任务。
- SFT 版:同一套系统、同样的题,只换模型地址,单独开一个批次。Qwen 服务现在部署的是原版(`/data/qwen38-27b/models/Qwen3.8-27B`,max_model_len 262144)。
- 训练数据污染:`datasets/*/sft.jsonl` 里,37 道题大多已有 GLM 轨迹。9-29 之前的批次就包含 solov2、yolo11l.seg、fsaf、yolo11n、yolox.l、mobilenet.v3.large、regnet.y.400mf/1.6gf、vit.b.32;retinanet.r50.fpn 哪份都没有。导出文件里还有 42 条 Qwen 自己的轨迹也被标成了可训练。
- 跟用户约定:旧的 Qwen 结果不删,只是不计入;"Qwen 一遍就能过"的文档已写成 Claude Doc。
**Why:** 用户用 GLM-5.3 的轨迹给 Qwen 做 SFT。在说明书写得很细的简单题上,两者同题打平,必须专门挑题才能看出微调前后的差别。
**How to apply:**
- 等用户告诉我训练实际用了哪份数据,再把题分成"训练里见过"和"没见过"两部分。
- 原版和 SFT 版必须用同一个说明书版本(VERSION 15)在相近时间跑;系统再改动就要重测原版。
- 10-08 17:00 有另一方用 "raise-concurrency-20261008" 把三个采集器都重启过,05/06 并发改成了 16。重启前先确认没有别人同时在操作。
相关:[[success-rate-breakdown-20261008]] [[quota-unknown-autorecover-fix]] [[bench-restart-hazards]]
**更正(2026-10-12):04 的 Qwen 轨迹只做训练前基准,不进 SFT。** 用户明确说过:Qwen 产生的数据不是用来做 SFT 的。衡量 Qwen 只看最终判定(PASS / FAIL),不要拿内容审核打回率或"进 SFT 的条数"当 Qwen 的指标。datasets-final/r20260929-04-training 里那 32 条 Qwen 轨迹不要混进训练数据。训练后的 Qwen 要用同样的部署方式(10-10 起 04 连的是 NVFP4 服务,配置备份 qwen-s100p-r04.before-nvfp4.yaml)、同样的 37 道题来跑,前后才可比。用户不想细分 Qwen 原版和 NVFP4,汇报时合在一起算。
**10-13 10:50 进度**:昨晚新过 4 题(depth.anything.v2.small 难题、yolo11l.seg、simplebaseline.r152、pidnet.s,各编译 1–2 次),无新失败;合计 27 题有结果、过 24 败 3,10 题没结果。系统问题(不算 Qwen):① 06-hbm 阶段时限 2 小时(config lifecycle.phases.06-hbm.execution_deadline_s=7200),depth.anything.v1.small 的 06-hbm 每次超 2 小时被杀,同一轨迹连杀 6 次;dpt.hybrid 超时后板端也超时→INFRA_ERROR;v2.small 用了 83 分钟刚好过。时限存在每条轨迹的 execution.json 里,改配置只影响新轨迹,config 每次现读、不用重启。② 10-13 10:14 网络断一下,efficientvit.seg.b1 停在 REQUEST_CONNECTION_INTERRUPTED:请求已发出、结果不明的不自动重发(quota.py 设计),会停到 24 小时过期。
**规则更正(10-12 20:10 用户决定):04 在 10-08 07:30 之前的 PASS/FAIL 也算。** 题单文件 count_rollouts_started_after 已改成 2026-09-29(提交 1e0b849,VM 上已同步,没有代码读它)。按新口径训练前 Qwen:23 题有结果、过 20 败 3(对照 3/3、中等 14/16、难题 3/4);败的三道都是真失败:mobilenet.v3.small、yolov8n.seg、solov2(mask_ap 掉 7.4%,只编译 1 次就收手)。两边都过的 20 题,编译平均 2.2 vs GLM-06 3.0,轮次中位 58 vs 61.5。这样调度"有旧结果的排后面"就和规则一致了,只剩 14 题没结果。下面 20:00 那段是旧口径。
**10-12 20:00 进度(训练后模型还没接,只有训练前基准)**:按规则(10-08 07:30 后)13 题有结果,过 11 败 2(对照 2/2、中等 7/8、难题 2/3);同题 11 道对比 GLM-06,轮次中位 56 vs 57、编译平均 2.4 vs 3.0,基本打平。10-08 前 04 还有 10 题的旧结果(过 9:densenet121、两道 regnet、fsaf、rtmdet、三道 yolox、yolo11m.seg;败 1:solov2),不计入但可参考,合起来 23 题过 20。从没出过结果的 14 题,5 条在跑(三道深度估计、yolo11l.seg、simplebaseline.r152,17:16/18:25 起)。**调度按"已完成次数"排队且算上 10-08 前的旧结果**(pipeline.py `_todo` 排序 `completed>0` 优先级最低),所以有旧结果的 9 题会排到最后才重跑。Qwen 的 tokens 计数口径和 GLM 不同(大一两个数量级),别拿来比。
- 能拉开训练前后差距的只剩约 8 题:mobilenet.v3.small、yolov8n.seg、solov2、三道深度估计、两道 simplebaseline;GLM 自己也没过过 depth.anything.v1.small、dpt.hybrid、yolov8n.seg。其余题训练前已接近满分,只能比编译次数和轮次。
**10-12 15:40 进度**:37 题里只有 12 题有有效结果(对照 1/4、中等 8/26、难题 3/7),Qwen 过 10、败 2;25 题一条有效结果都没有(被重启/取消/作废打断),5 条在跑(都是中等题)。有结果、GLM 也做过的 11 题:Qwen 过 10,GLM 全过;同题编译次数两边差不多(2–5 次)。Qwen 的 2 个失败都是真失败,5 次编译用完仍没达标:mobilenet.v3.small(输出相似度 0.983,门槛 0.99;GLM 06 三条全过,各编译 2–4 次)、yolov8n.seg(mask_ap 掉 21.6%,容差 5%;GLM 从没跑完过)。最可能拉开差距的 3 道深度估计和 2 道 simplebaseline 姿态一条都没跑完,GLM 历史上在深度估计 3 题上也只过过 1 条。编译上限在 harness/stagerun.py 的 MAX_COMPILE_ATTEMPTS=5。

View File

@ -1,14 +0,0 @@
---
name: reply-in-chinese
description: "回复用户一律用中文(10-16 用户看到英文回复后要求\"转为中文\")"
metadata:
node_type: memory
type: feedback
originSessionId: c8e679af-e7a3-4129-a0e4-e1821fa37ce1
modified: 2026-10-16T14:33:35.172Z
---
2026-10-16 我用英文汇报断电停机/重启结果,用户回"转为中文"。
**Why:** 用户一直用中文沟通,英文汇报读着费劲。
**How to apply:** 所有回复(包括汇报、提问、表格)都用中文;命令、路径、代码保持原样。配合 [[plain-language-no-jargon]]。

View File

@ -1,77 +0,0 @@
---
name: restart-20261015
description: 10-15 下午重启前清查出的底层 bug(回执超时、暂停吃掉 24 小时、400 永久暂停、账本忙让出名额、评测一次 12 GB)、修法、v17/判分 v0.8,以及重启做法
metadata:
node_type: memory
type: project
originSessionId: cadbef09-8744-472b-81b1-a1ff37b0a735
modified: 2026-10-15T06:39:25.114Z
---
10-15 13:00–14:00 清查 10-14 19:00 重启以来所有结束的轨迹、阶段、dev 命令、collector/offload 日志、额度事件,用户要求"重启前查清所有已暴露的 bug"。
**新发现并修复(分支 feature/salty-20261009)**
- 4c98f9f 板上清理回执读取:原来 15 秒超时、不重试,比 ssh 自己的 20 秒连接超时还短;板上已跑完并回收也会判清理未确认、整条作废。历史 9 次(8 次 TimeoutExpired),10-15 12:44 05 fcn.resnet50 r02。改为 60 秒、最多 3 次。
- bf88f78 暂停期间 24 小时照算:10-14 三条 v15 轨迹(06 fcn.resnet101、depth.anything.v1.large、dpt.hybrid)被断线停了 7–16 小时,续上只剩不到 1 小时,编译/评测做到一半被截断。改为 execution.paused_credit:暂停时判超时扣除暂停时长,续跑时补回,累计最多补一个预算(防止永远续不上的暂停轨迹永不结束)。
- c2ceff5:
- 模型服务返回 400(Qwen 上下文超出,76 万 token)也被当成可续跑暂停,但续跑资格永远不收,挂到 24 小时(04 retinanet.r101)。历史 8 次全是 Qwen。现在 400 直接结束(INFRA_UPSTREAM_REJECTED)。
- resume_readiness 在额度库忙时返回 QUOTA_SOURCE_UNAVAILABLE,f191baf 不算"在等",名额会被新题拿走;现在算在等。
- b3be5cb 只取 f56ac2f 里 board.py/common.py/run.py(不在 task_contract 指纹里):每次上板按实测字节 ≤1 GiB 分批、G2 余弦按组算。05 solo.r50 G2 一批 32 张 12 GB 撑爆 8Gi 容器、回执带不回来、整条作废。v17 配置另把 08-eval 内存 8Gi→16Gi(coco_extra 一次 16 张仍在冻结文件里)。
- 41eb1d2(用户定):agent v17,提示词删「评测数据只读可见」(VM 上 bench/data 为空,dev 只挂校准集);判分 v0.8,板上 disk_full 与 board_offline 一样免罚(export/reward.py,导出时生效)。
- v17 配置(用户定):max_turns 240→400,token 上限不变;08-eval 16Gi。configs/mini/*-v17.yaml。
**查过、不是 bug**
- offload 大包上传失败(convnext.large、zoedepth.n 等)后来都重试成功,ARCHIVED.json 在。
- 阶段 OOM 后内存自动翻倍(salty_remote._memory,上限 128Gi)在起作用;depth.anything.v1.large 的 06-hbm 到 53 GB 还不够,是模型太大。
- dev 的 STAGE_EXECUTOR_EXIT 07:50 改 37 槽后归零;之后的 TERMINAL_DEADLINE_EXCEEDED 是 agent 自设超时。
- 07-board stalled 两次:S600 40015/40017 加载模型卡住,按瞬态重试;conditional.detr.r50 是 hbdk 编译挂死/崩溃,模型问题。
**已知限制,以后做**
- Salty 容器里 onnxruntime 线程没限(只限了 OMP/MKL/OPENBLAS),大模型编译常超 4 小时阶段时限(vit.h.14、dab.detr、dpt.hybrid、conditional.detr、solo.r50)。
- 板上进度停滞检测(加载卡住要等满 ~49 分钟)。
- 分割/深度/光流/特征一次多张上板:各任务文件在指纹里,等新建 run 再上 f56ac2f 其余部分。
- personal 空间有 425 个 bench-stage-*、28 个 bench-dev-* 目录没清,删要用户同意。
**重启做法**:configs/restart-20261015/(stop.sh、targets.py、requeue.py、start.sh、mkconfigs.py、deploy/ 18 个文件,md5 与分支 96d72b2 一致)。
- 测试:VM 全量 base/new 对比(72 个文件)+ Salty 容器全量对比(VM 没有 numpy)。只有固定 v0.7 的两处测试不同,已跟到 v0.8。
- VM 上跑测试要 `env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY`,否则 test_quota 的本地假上游走代理报 502(36 个假失败)。
- test_collection、test_explicit_batch 在 base 上也时过时不过,是老的不稳定测试。
- 过程:14:18 SIGTERM;等 3 条上板/评测的跑完(14:35);取消 35 条(34 在跑 + retinanet 的 400 暂停),补号 35;旧 collector 14:37 自己退出(没卡在导出)。14:37:18 用 -v17 起来,几秒内就开始领题(以前要 10–20 分钟)。
- 06 的 GLM 组因取消打断一条在途请求进入 RECONCILING,约 21 分钟后自动恢复。
- 启动时打印的「已隔离」清单(05 24 条、06 28 条、04 9 条)与上次启动完全相同,都是 10-10/10-12 的旧账。
- 14:39 核实:05 16 条、04 5 条新轨迹全是 v17、max_turns 400;部署文件 md5 与分支一致;collector 环境里是 -v17 配置。
**15:00–16:10 追加(用户:不要新建批次;没进 SFT 的可以清;继续改进),全部热部署、没重启 collector**
- 评测分批不建新批次(6ab088d,15:55 上线):
- harness/eval/__init__.py 加 EQUIVALENT_SOURCES(精确的「冻结指纹 → 现指纹」对)和 contract_holds;阶段检查(stages/base.py)、基线检查(s02_baseline.validate_baseline)、expand 比对(pipeline._check_frozen_job)都走它。
- 只批准 f56ac2f 的 5 个任务文件(seg/mde/flow/feat/coco_extra)。依据:上板逐个样本推理(board_bench.infer_inputs),每张的前后处理和指标不变。
- 批次的冻结记录不动;基线缓存按冻结指纹寻址(baseline_identity 用 job 里的合同),照常命中。
- 冻结检查在每个阶段新起的 stagerun 进程和容器里跑,不在 collector 里,所以能热部署:先上规则、5 秒后再上任务文件。
- 上线后:iseg/pose 阶段都放行,没有 frozen 报错;iseg yolov8n.seg 评测 9 次上板 7.8 分钟。
- 漏洞:harness/adapters/*.py 不在指纹里。顶层姿态和 mmdet 实例分割的适配器(topdown_pose_common、topdown_pose_batch8、rtmpose_common、mspn_common、rtmo_common、mmdet_iseg_common、mmdet_solo_common)仍然一张图一次上板:simplebaseline.r152 用了 176 次、37 分钟。要改 7 个适配器的循环,并在容器里做新旧对比,下一步再做。
- 板上加载卡住 10 分钟就按 stalled 结束(b7577de,board_perf.LOAD_STALL_S)。
- Salty 容器 onnxruntime 限线程(85ae175):
- 实测 4 核容器 resnet50:默认 125 线程 320–500 ms,限 4 线程 22 ms;hmct 校准写死 10 线程,约 35–48 ms。
- harness/container_site/sitecustomize.py 在 onnxruntime 第一次 import 时把 InferenceSession 限到 OMP_NUM_THREADS。
- worker 把这个目录放进阶段和 dev 的 PYTHONPATH。盖掉的镜像 sitecustomize 只装 apport 钩子。
- Salty 失败作业不再留(5bd0f1a):阶段失败拷完日志就删;offload.reap_orphans 也删已结束的失败作业。offload 用 configs/ops-20261015/restart-offload3.sh 等空闲时重启(原命令、原 -v16 配置,导出口径不变)。
- 一次性清理(用户同意):删 442 个已结束的 bench-stage/bench-dev 作业和目录,外加 7 个空挂目录。只删主控端那次尝试已结束的;名字不是 bench-stage-/bench-dev- 的不动(bench-repro-pending、untar 探针、失败的 export)。脚本 /tmp/saltyclean-amos.py,日志 logs/ops-20261015/saltyclean.log。
- 测试:VM 全量(去掉代理变量)和 Salty 全量,最终版 5bd0f1a 对比 a526af7,只多了新测试,其余一致。
- 自己踩的坑:两组 Salty 测试共用 /tmp/saltytest-<label> 目录,一组结束时删掉了另一组的,要用不同 label。
- 19:18 核实(热部署后 3 小时 23 分):
- 66 次阶段尝试全部 OK/SKIPPED,冻结检查误拦 0 次(iseg 24、pose 15)。
- 一次性清理删了 410 个作业、411 个目录、3 个空挂目录;32 个删除失败,多半是被同时清理的进程先删了。19:18 时 Salty 上只剩 1 个失败的阶段作业、11 个阶段目录,没有再攒。
- offload 16:06 已按原命令重启。
- 姿态 simplebaseline.r152 三次评测仍是 176 次上板、35–37 分钟(适配器逐张路径)。
- 19:24 堵 dev 泄露(172a174,热部署):
- [[sft-sample-review-20261015]] 发现 agent 能经 bench-cache/adapter-probes/*/private-source 读到 harness/registry 私有副本和参考配置。
- dev 原来挂整个 personal:/bench/cache,还能看到 dataset-downloads(带标注原始数据集)和 baseline。
- 现在只挂 onnx/weights/sources/venvs;v16/v17 的 dev 命令只用过 sources(7) 和 weights(1)。容器内实测三处都不可见,onnx/hmct 正常。
- VM 侧 bench/cache 只有 weights,没有这些副本。
**Why:** 用户原则:底层错误不能算到 agent 头上,系统故障不扣模型的时间和次数。
**How to apply:** 看到 DEADLINE_EXCEEDED / BOARD_CLEANUP_UNCONFIRMED / 长期 PAUSED_INFRA,先查是不是这几类;task_contract 只哈希 harness/eval/<任务>.py。
相关:[[overnight-check-20261015]] [[bench-restart-hazards]] [[net-resilience-board-deadline-20261014]]

View File

@ -1,25 +0,0 @@
---
name: rl-readiness-20261017
description: 10-17 03:30 评估能否当强化学习环境:能用,但瓶颈是慢不是稳;收尾 5 件(速度进奖励、Qwen 省略改成确定性、冻结版本、挑有好有坏的题、Qwen 推理服务扩容并记模型版本),之后不再无边界优化
metadata:
node_type: memory
type: project
originSessionId: e00ddcbc-d4da-4f81-a8f7-1fd7eeb8e8df
modified: 2026-10-16T19:20:54.154Z
---
用户问:想把它做成 RL rollout 环境,是不是还不稳定、要不要放弃优化。我的判断:能用,不建议放弃,也不建议继续无边界优化;做完收尾清单就冻结。
**依据(10-17 03:30 数据)**
- 稳定:20:57 重启后 6 小时健康检查全 ok,正式步骤 101 次失败 3 次,大模型请求 4000+ 次断 1 次;不算取消,系统出错占已结束轨迹 10-13 约 37%,10-14、10-15 约 13–16%,10-16 约 27%(含 Qwen 上下文超长和断电)。系统原因结束的轨迹都单独标记;导出时只有 PASS/FAIL 进 reward_learning(to_agentrl.py:823),所以不稳只伤吞吐、不污染奖励。
- 慢:10-10 以来 agent 自己结束的轨迹,通过的中位 3.0h(04)/4.7h(05)/6.4h(06),失败 6–11h,预算用完 13–15h,超时 24h;每天 agent 自己结束的 5–73 条(多数 30–50)。Qwen(要训的模型)只开 5 路,服务一次只处理 1 个请求。
- 学习信号:有 ≥2 个结果的题里,结果有好有坏的共 74 道(04 9、05 14、06 32,按 job 合并后 74),中位轨迹 3.8h;其中编译中位 ≤30 分钟的 31 道。全过或全不过的题对组内比较没有信号。注意多是 GLM 和旧规则下的历史。
- 奖励 v0.8(export/reward.py):G1 1 分 + G2 1 分 + G3 2 分 + 成本最多扣 1 分 + 超时扣 1 分;速度 0 分;只编译成功就拿 1 分。
- context-mask.js 的省略只发生在发送时,会话文件存原文:Qwen 训练数据和模型实际看到的不一致;和续跑必死是同一个根因(见 [[check-20261017-0215]])。
**收尾清单(都没动,等用户定)**:①速度进判定和奖励(板上实测默认配置作参考)并统一奖励口径;②context-mask 改成只由消息决定;③冻结一个训练专用版本;④从 74 道题起步挑题;⑤Qwen 推理服务支持并发、每条记模型版本。然后小规模试跑 30 题×4–8 条,测一步多久、多少题有信号,再决定在线 RL 还是批量采样+筛选训练。
**Why:** 用户在决定还要不要继续投入。
**How to apply:** 用户再提"再优化一下"时,先对照这份清单,清单外的改动默认不做;冻结后只修会把系统原因算错的 bug。
相关:[[check-20261017-0215]] [[stability-sweep-20261016]] [[verifier-audit-20261012]]

View File

@ -1,20 +0,0 @@
---
name: rollout-time-breakdown-20261008
description: "10-08 实测:轨迹时间花在哪、编译机 CPU 余量、04 时长重复计算——判断\"编译上云\"值不值的依据"
metadata:
node_type: memory
type: project
originSessionId: 1b3cd916-3faa-4ef7-be14-c3fb07244493
modified: 2026-10-08T05:21:23.910Z
---
2026-10-08 13:20 在 model-compile(120.48.151.164,244 核 / 1.4 TB)上实测,04/05/06 三个批次,608 条已结束轨迹:
- 轨迹总时长按阶段:05-compile 33%,06-hbm 16%,08-eval 7%,07-board 1%,其余约 43% 是 agent 思考和杂项。
- **stages 表里 04-calibrate 的时长是重复计算**:受保护模式下 04 的结束时间跟着 05 一起登记,1270 对里有 1085 对两者结束时间相差不到 2 分钟。看板上 04 占 34% 是假象。
- 05 拿到 cpu 槽位后的编译时长:中位 29 分钟,p90 102 分钟(OMP_NUM_THREADS=8)。
- 从提交到真正开始编译(04 合并配置 + 排队):中位 0.5 分钟,p90 18 分钟。05 从没出现过排队超时。
- 当时机器 CPU 占用约 31%(空闲 67%,负载 82–89),cpu 槽位 13/24,板卡占用 0。
**Why:** 用户 10-08 提出把 05-compile 搬到 Salty 集群(他人写的方案,要求 Salty 作业生命周期、冻结输入、epoch 对账等)。数据显示目前编译机既不缺 CPU 也不排队,搬上云不会提高吞吐,单次编译反而多了传输和拉镜像的时间。
**How to apply:** 再讨论扩并发或编译上云时,先重新采这几项数;只有目标并发大约超过当前的两倍、编译机 CPU 接近打满时,上云才有收益。相关:[[success-rate-breakdown-20261008]] [[prefer-subtraction-over-addition]]

View File

@ -1,36 +0,0 @@
---
name: salty-container-threads-20261013
description: Salty 容器看到整机 244 核、只给 4–6 核,torch/numpy/onnxruntime 按 244 开线程被限流,评测慢 25 倍;10-13 已加线程上限,onnxruntime 还没处理
metadata:
node_type: memory
type: project
originSessionId: 977b3506-bd83-4ae1-9791-f48a3caf1e3d
modified: 2026-10-13T02:44:48.465Z
---
2026-10-12 晚发现,同一个模型在 Salty 容器里有几步突然变得很慢。
- 评测:rtmdet 6 分钟变成 160–173 分钟。rtmdet.l 那次,上板推理前三批几分钟就跑完,之后容器在本地计算那一步停了 107 分钟。
- 生成 HBM:yolo11s 5 分钟变成 36 分钟,efficientvit.seg 15 分钟变成 80 分钟。
- 编译:多数慢 1–1.5 倍,个别慢 2.5 倍。
stage 的 meta.duration_s 是容器自己计的时,不含排队,所以慢在容器里面。
**原因(测试容器实测,4 核、8G、cpu-shared 队列)**:
- 容器里 os.cpu_count() 和亲和性都是 244(整机),cgroup cpu.max 是 400000/100000,也就是只给 4 核。整机负载约 73,机器本身不忙。
- torch 默认开 122 线程,几乎每个 100ms 周期都被限流。1536×1536 矩阵乘每秒 2.6 次,set_num_threads(4) 后每秒 42 次,快 16 倍。
- numpy:默认每秒 31 次,设 OMP/MKL/OPENBLAS_NUM_THREADS=4 后每秒 146–158 次,快 5 倍。
- onnxruntime 1.19:默认开约 185 个线程。它不认 OMP_NUM_THREADS 这些环境变量,也不认进程亲和性。只有在 SessionOptions 里写 intra_op_num_threads=4 才快 2.2 倍(每秒约 300 次变成约 680 次)。
**已做(a7c454c,10-13 10:43 部署)**:salty_remote._submit 提交容器时加 `-e OMP_NUM_THREADS/MKL_NUM_THREADS/OPENBLAS_NUM_THREADS=分到的核数`。每个阶段都是新起的进程,部署后马上生效,不用重启调度程序。已核对:10:43 以后提交的新容器 describe 里带着这三个变量。salty_offload.py 常驻进程还是旧代码,它起的导出作业要等它重启才带上。
**没做**:onnxruntime 的线程数。我们自己的代码里只有 harness/tools/recipe_model.py 一处用它。主要用户是厂商工具链:oe-3.7.0 里 hb_compile 校准时用原始模型推理。工具链只在 personal:/bench-runtime 里,虚拟机里看不到源码。要处理,得在容器启动时加一个自动加载的脚本(sitecustomize),把 InferenceSession 默认的 intra_op_num_threads 设成核数,或者找厂商的参数。
**Why:** 用户发现迁到 Salty 后出结果极慢,怀疑系统有问题。这是一个跟题难不难无关、实打实的系统问题。
**How to apply:**
- 比较部署前后各阶段用时,要按同一个模型配对比。
- 以后任何在容器里跑的程序,线程数都按 cgroup 的 cpu.max 定,别按 cpu_count。
- 测试容器的写法:`salty run <镜像> --cpu 4 --memory 8Gi -- /bin/sh -c "ln -sfn /data/personal/bench-runtime/home/{toolchains,outputs} ... && exec <venv>/bin/python <脚本>"`,脚本先 storage cp 到个人空间。用完 salty rm 并 storage rm -r。
另:10-12 晚 Salty 共享队列一度排队 16 个作业,最久 45 分钟,我们最多同时跑约 22 个作业、116 核;10-13 上午不再排队。注意 usage.log 里的时间是北京时间,salty describe 的创建时间是 UTC。
相关:[[salty-throughput-drop-20261012]] [[selfheal-and-alert-20261012]] [[mini-salty-controller-20261009]]

View File

@ -1,38 +0,0 @@
---
name: salty-personal-migration-20261009
description: 10-09 起把老机器 bench-cache/bench-data 传进 Salty 个人空间 personal:/bench/;脚本位置、速度、打包规则、Salty 存储的限制
metadata:
type: project
---
用户 10-09 定:主控用 Mac mini(Lima `bench` arm64 VM,另一会话在搭,见 outputs/mac-mini-controller-20261008),编译用 Salty,板子验证;任务数据放 **Salty 个人空间**。
- 目标:`personal:/bench/cache`、`personal:/bench/data`(容器内 `/data/personal/bench/...`);带软链接或小文件极多的 `pool`、`venvs`、`sources`、`adapter-probes` 打成 8G 切片的 tar 放 `personal:/bench/_tars/`(附 .sha256),以后要在 Salty 作业里解开。`bench-data/bench-data` 是自指软链接,未传。
- 老机器 120 上:salty 0.2.1 装在 `/root/.local/bin`,root 已登录(用户网页批准;迁移完应让用户在网页撤销)。脚本 `/root/salty-migrate-20261009/migrate.sh`(可重跑,done.txt 记完成单元),日志 migrate.log,文件清单 manifest.tsv(100532 条,用于核对)。10-09 16:56 启动,646 个目录单元 + tar 切片。
- 实测:单路 12.4 MB/s,6 路合计约 20 MB/s,全量约 460G 需 6–7 小时;单文件最大 12.7G,已设 `--timeout 14400`。
- Salty 存储限制:`salty storage cp` 源只能是本地路径或 personal:/shared:/;服务端后台复制只在 Salty 两空间之间,**不能直接从 JuiceFS 网盘拉**;拒绝传软链接;远端目标结尾带 `/` 报非法;`salty run` 无挂卷选项,容器只挂 /data/personal 与 /data/shared;文档没写个人空间配额。
- 跑任务的真正卡点不在上传:Salty 05 编译自带打包输入;01–04/08 需 x86 模型环境,arm64 Mac mini 跑不了,要改成在 Salty 上执行(未开发)。
- **坑**:`salty storage cp 目录 远端` 在远端已存在同名目录时会把源**整个套进去一层**(远端/名字/名字/…)。失败重传前必须先 `salty storage rm -r` 自己的半截目录(migrate.sh 的 up() 已加)。服务端复制(空间之间)实测 1.89G 不到 3 秒,约 700MB/s,只是源必须已在 Salty。
- 10-09 17:43 weights/501f… 遇网络错误失败,17:45 有**另一个会话**用 ssh root 登 120 手工补传,造成嵌套;我 17:54 删除重传纠正。动迁移前先确认没别的会话在同时操作。
- **10-09 18:0x 起迁移由"salty迁移"会话统一负责**(对方要求,我已同意并停掉自己的后台检查):本会话不要再补传、删除、移动 personal:/bench/ 下文件,也不改 120 上的迁移目录;发现问题发消息给"salty迁移"。
**Why:** 下次接着核对上传、解 tar、或问"数据在哪/还差多少"时直接用。
**How to apply:** 看进度:ssh 120 读 migrate.log 和 done.txt;失败就重跑 migrate.sh。传完用 manifest.tsv 核对后才让用户交回 120。相关:[[compile-server-reclaim]] [[git-mainline-20261009]]
**10-09 17:43 踩坑**:一个权重单元上传 493MB 文件时网络错误;我整目录补传时远端目录已存在,结果多套一层(`<N>/<N>/`),而且 CLI 报"已上传 2 个文件"但大文件在哪层都列不出来。改为单文件写明目标路径补传后正常。
- 补传目录单元:远端目录已存在时只拷内容(源写成 `目录/.`,先 `mkdir -p` 目标),别重跑整个 migrate.sh 原样补;要另写补传脚本(不要原地改正在跑的脚本,bash 边跑边读文件)。
- **CLI 报成功不等于在**:传完必须按 manifest.tsv 逐文件核对路径和大小,通过后才让用户交回 120。
**10-09 18:1x 接管与改道**:当时有两个会话同时在处理这次迁移("数据采集进度"和"salty迁移"),已约定由"salty迁移"会话统一负责。对方 17:55 **原地改了正在运行的 migrate.sh**(同一个 inode),主 bash 跑完第一轮后会按旧偏移读改过的文件,行为不可预测;我只结束了主 bash(pid 948591),第一轮 xargs(949300)继续传。后续由新文件接手:`phase2.sh`(等 949300 退出 → 失败单元逐文件补齐、只传不删 → 上传 tar 分片和 .sha256 → 调 verify.sh),`verify.sh`(逐层 `salty storage ls`,按 manifest.tsv 比对路径和大小,结果写 verify-report.txt,日志里出现"核对结束")。migrate.sh 不再运行。
- 以后:正在运行的 bash 脚本绝不原地改;要改就另写新文件。
**解压待办(10-09 约定)**:"数据采集进度"会话要把 04/05/06 改成 Mac mini 主控 + 每阶段都在 Salty 容器里跑,容器里把 /home/ming01.wang/bench-cache→/data/personal/bench/cache、bench-data→/data/personal/bench/data 软链。它请我在 **verify 通过后** 用 Salty 作业解压 _tars,并在完成后发消息告诉它(它在那之前不让作业读写 personal:/bench/)。
- 已备好 120 上的 `/root/salty-migrate-20261009/extract.sh`(未上传、未提交):用法 `extract.sh 包名 目标父目录 顶层名 应有文件数 应有链接数`;先 sha256sum -c,目标已存在就停。镜像用 `registry.d-robotics.cc/algo/bench-runtime@sha256:c530f4ff…`。
- 应有数:data-pool 62913 文件/0 链接 → /data/personal/bench/data/pool;cache-venvs 11959/5、cache-sources 7132/1 → /data/personal/bench/cache/;cache-adapter-probes 6916/16 最后单独解。_tars 先不删,等用户定。
- 对方另在 personal:/bench-runtime/ 上传三套环境 tar(约 22G,用同一个 120 登录),与我无关。
**结果(10-09 23:48 核对 / 10-10 解压)**:直接上传 11589 个文件,缺 4(2 个 .gitkeep 空占位、solo.r50--coco2017 的 2 个 g2_fp32.npz,后者由 02-baseline 用到时重算),大小不符 0、多出 0;18 个 tar 分片和校验文件全部一致。解压脚本在 `personal:/bench/_tools/extract.sh`;10-10 从本机提交两个 Salty 作业:`salty-835ccd4a-…`(venvs、sources、pool)和 `salty-fe327bed-…`(adapter-probes),4 CPU / 8Gi,镜像 bench-runtime@c530f4ff…。解完要发消息告诉"数据采集进度"会话。
- **Salty 单文件上传上限**:在 7.3GiB(成功)到 12.7GiB(约 30 秒必失败,报 request or response body error)之间,确切值未知;大文件切到 8GiB 以内(8GiB 分片实测都成功)。solo.r50 两个 g2_fp32.npz 内容相同(sha256 ce06383e…,12727502206 字节,我已在 120 独立核实),由"数据采集进度"会话切 4G 分片传到 personal:/bench-runtime/solo-g2/ 后起作业拼回原位;约定只新建这两个路径、目标存在就停、拼完核对 sha256 和字节数后告诉我。
- 10-10 04:4x:venvs/sources/adapter-probes 解完并逐文件核对通过(作业 salty-cb102738…,26029 项全对,合计 91564045377 字节),已通知"数据采集进度";solo.r50 两文件由对方补齐、我核过大小。pool 仍在解(作业一 salty-835ccd4a…,6.3 万小文件很慢),解完用 `personal:/bench/_tools/compare.sh data/pool` 核对后再通知对方。
- **/data/personal 上别用 du 判断大小**:目录 st_size 是下层合计(类 CephFS),du -sb 会层层重复计算(venvs 显示 7.7G 实为 0.98G);按 find -printf %s 文件大小相加。工具 compare.sh + manifest-packed.tsv 在 personal:/bench/_tools/。
- **10-10 05:1x 迁移完成**:pool 解完(salty-835ccd4a… succeeded)并逐文件核对通过(salty-c8d9e4f9…:62913 项全对,10877004191 字节)。personal:/bench/ 全部就绪:直接上传 11589 个文件只缺 2 个 .gitkeep;打包部分 88942 项全对。已通知"数据采集进度",我不再写 personal:/bench/。待用户:交回 120、网页撤销 120 的 salty 授权、决定 _tars(约 97G)删不删。
- 小坑:salty run 回显"作业已创建:salty-…"是中文冒号无空格,脚本里取 ID 要用 `grep -o 'salty-[0-9a-f-]*'`,别按空格切。

View File

@ -1,23 +0,0 @@
---
name: salty-throughput-drop-20261012
description: 10-12 清查:迁到 Salty 后通过数掉到 1/20,主因是剩下的全是重题(同模型在 Salty 不更慢);系统问题是整批重启、容器被杀占编译次数、连不上 GLM 永不自动恢复、metric3d 从没跑通、Mac mini Wi-Fi
metadata:
type: project
---
2026-10-12 18:00 查 jobs.sqlite(04/05/06):
- 每天 PASS:10-06 约 37、10-08 68(05 27 + 06 38 + 04 3);迁移后 10-10 4、10-11 13、10-12 3。端到端能跑通(迁移后共 20 条 PASS),但效率约为原来的十分之一。
- 整批丢轨迹 114 条:10-09 迁移作废 37,10-10 重启取消 40,10-12 10:06 Mac mini 强制重启作废 37。
- 阶段成功率(迁移前 10-05..08 → 迁移后 10-10..12):05-compile 84%→44%(容器被杀 REMOTE_WORKER_FAILED 93、执行器退出 15、超 4 小时 28、排队超时 10);06-hbm 80%→53%;07/08 没变差。
- **更正(2026-10-12 16:15):同一个模型在 Salty 上并不更慢**。同题配对:编译时长比值中位 1.04,06-hbm 0.54、08-eval 0.50;容器排队+取回每步约 1.5 分钟。"中位 31→52 分钟"是题变重了。迁移前 06 跑的 104 题里 62 题早已 PASS(多是分类补采样,编译中位 25 分钟);迁移后 98 题 0 题 PASS 过(深度估计/光流/实例分割,编译中位 56 分钟)。05 同理(32/100 → 2/96,34→58 分钟)。每 100 路·小时 PASS:06 17.9→0.8,05 6.7→0.5,主因是题,不是 Salty。
- **连不上 GLM 永不自动恢复**:harness/quota.py `_resume_readiness` 只放行 QUOTA_WAIT_BUDGET_EXHAUSTED / REQUEST_OUTCOME_UNKNOWN,CONNECT_FAILED(请求根本没发出去)被判 RESUME_NOT_ELIGIBLE。10-12 05 有 11 条因此停住(11:47–12:41 连续连不上 + 16:03 断网),auto_resume_count 0,不修会在 24h 期限判超时。broker 在调度程序进程里,改了要重启。
- 06 的 glm-coding-account-b 只要有一个请求结果不明就整组 RECONCILING,16 路全停,直到本地时限过了自动放弃(10-12 有 5 次;16:03 断网后又停)。
- Mac mini 10-12 15:50–16:05 断网 15 分钟(没重启);虚拟机时钟会漂 2–3 分钟。导出正常:datasets-final SFT 32/79/139 14:4x 刷新过,offload 日志只有断网时的网络错误。
- 编译次数上限检查在 stagerun 里,每次 `bench stage run` 都是新进程,改 compile_attempts_used 不用重启就生效(之前说要重启是错的)。
- **5 次编译上限把系统失败也算进去**:harness/stagerun.py compile_attempts_used 只排除 lane='queued'。10-10 以来 175 条有编译的轨迹里 73 条至少被系统吃掉 1 次机会,30 条 ≥2 次;用满 5 次的 19 条里 13 条含系统失败,被记成模型 FAIL(depth.anything.v2 metric 系列、flowformer、unimatch 等),数据被污染。
- metric3d / metric3dv2(05)在任何机器上都没过过 02/03:适配器 harness/adapters/metric3d_common.py 重复导入 torchvision,报 "already a kernel registered ... roi_align",被包成 "Failed to find function"。10-10 以来 02/03 的 59 次失败里 47 次是它,agent 一条轨迹重试导出到 a14。
- Salty 作业列表 123 failed / 5 succeeded 是假象:665bbef 起失败作业留着给平台取证,成功的取回后就删了。
- Qwen 的 mobilenet.v3.small 失败核实为模型自己的配置错误(node_config 里写了不支持的 activation),不是系统问题。
**Why:** 用户怀疑系统有问题,确实有。
**How to apply:** 修复优先级:①编译上限只数真正跑完的编译(排除 REMOTE_WORKER_FAILED、STAGE_EXECUTOR_EXIT),并把被污染的 FAIL 补号重跑(不用重启);②CONNECT_FAILED 加进可自动恢复原因(要重启);③metric3d 先从 05 清单拿掉;④Mac mini 插网线;⑤重启 Mac mini 前先 `limactl stop bench`。比较迁移前后吞吐时必须按题配对或按题难度分层。相关:[[mini-salty-controller-20261009]] [[bench-restart-hazards]] [[qwen-sft-benchmark]]

View File

@ -1,39 +0,0 @@
---
name: selfheal-and-alert-20261012
description: 10-12 修复与自愈:系统失败不占编译次数、连不上可自动续、暂停轨迹不被抢名额(12787a2/8d39d2b);本机每 10 分钟健康检查弹通知;排空重启脚本;metric3d 撤出队列
metadata:
type: project
---
2026-10-12 用户要求"都修好,出问题能自己修或者提醒我"。改动在 feature/salty-20261009(12787a2、8d39d2b),已部署到 Mac mini:
- harness/stagerun.py:容器被杀(REMOTE_WORKER_FAILED)、执行器退出(STAGE_EXECUTOR_EXIT)不计入 5 次编译,每条轨迹最多免 3 次(FREE_INFRA_COMPILES)。stagerun 每次是新进程,部署即生效。提示词 e2e.md 没改(保持 VERSION 15,Qwen 前后可比)。
- harness/quota.py `_resume_readiness`:CONNECT_FAILED、REQUEST_DEADLINE_BEFORE_SEND(_not_sent 证明没发出去)允许自动续。
- harness/pipeline.py:有暂停轨迹只差额度(resume_pending)时,空出的名额不给新轨迹。之前新轨迹总抢在它就绪前,06 segformer.b5 因此饿了 4 小时。
- 后两条在调度程序进程里,要重启才生效。**10-12 17:16 已按用户要求立即重启(不再排空)**:取消在跑和暂停的 46 条(42 CANCELLED、4 INFRA_ERROR,原因 USER_RESTART_20261012,清单 logs/cancel-1012-restart.txt),各补一个新 kk;Salty 上 34 个残留 bench-stage 作业 stop/rm/storage rm(logs/orphans-1012-restart.log,bench-export 没动);新调度程序 05=16、06=16、04=5(pid 95662/95665/95675,日志 logs/collect-*-1012-171614.log)。用户要"在原来的基础上跑",没改题单优先级、没降到 14 路。之前的排空方案(作废):16:43 起 scripts/drain_restart.sh(pid 90713,日志 /mnt/data/ming01.wang/logs/drain-restart-1012.log)在排空 04/05/06:旧进程跑完后按新代码重启,05/06 改 14 路(给训练后 Qwen 批次留 5 路),04 仍 5 路 + qwen-sft-v1-jobs.txt。旧进程无子进程且无校验中轨迹满 10 分钟就 kill -9(卡在导出)。
- **坑**:`bench execution resume` 在原主人进程还活着时报 OWNER_NOT_RELEASABLE(execution.py resume),排空期间不能手动接手暂停轨迹,只能等旧进程退出由新进程续。drain_restart.sh 第一版因 `set -u` 下空关联数组报错直接退出,已修;重跑要加 NO_TERM=1,别重复发 SIGTERM。
- metric3d / metric3dv2:05/06 排队的 19 条改为 CANCELLED(annotation METRIC3D_ADAPTER_BROKEN_20261012),在跑/暂停的 4 条已 execution cancel。04 里的 16 条 QUEUED 不在 37 题清单、不会被领,没动。修好适配器后按原 job 补新 kk。
- 健康检查:scripts/health_check.py(虚拟机里只读,有问题每条一行);本机 ~/.bench-health/bench_health_alert.sh 由 launchd `cc.d-robotics.bench-health`(~/Library/LaunchAgents/)每 10 分钟跑,弹 macOS 通知(同样问题 2 小时提醒一次,恢复也通知),记录 ~/.bench-health/{log,alerts.log}。查:Mac mini 连不上(连续 2 次)、虚拟机没运行、调度程序没在跑、暂停超 1 小时、3 小时没新步骤、1 小时内容器失败 ≥8 或轨迹被中断 ≥5、账号 PAUSED/RECONCILING 超 45 分钟、磁盘 <10G、内存 <1G。脚本顶部 MINI(Mac mini 地址)和 RUNS(批次)要随现场改。
- 没做自动修的:Mac mini 重启/断网/调度程序崩溃,只报警(恢复要清 Salty 孤儿作业,不适合无人值守)。飞书机器人(本机 cc-connect-feishu 等)没用来报警:会以用户身份发消息,要用户点头。
**Why:** 用户不想再出现"暂停没人管、系统失败算模型失败、掉线没人知道"。
**How to apply:** 加训练后 Qwen 批次时:新开 run、跑 37 题、5 路,并把 run id 加进 ~/.bench-health/bench_health_alert.sh 的 RUNS(仓库 scripts/ 同步)。相关:[[salty-throughput-drop-20261012]] [[bench-restart-hazards]] [[mini-salty-controller-20261009]]
10-12 17:00 补充:
- 用户说的"小金"=本机 lark-cli 的机器人(飞书应用 cli_aa870f4a38f91bd3,后台名"飞书群答疑助手"),5-27 Codex 会话里用户截图的"小金那条消息"就是它发的。已知可用的发法:`lark-cli im +messages-send --as bot --user-id <open_id>`;用户本人 汪明 open_id ou_cc299d6a3684a6c45d56d91f7127b185。报警脚本 FEISHU_TO 10-12 17:1x 经用户同意已填上并发过确认消息(发送失败会记进 alerts.log)。launchd 环境里 PATH 由脚本自己 export,lark-cli 能用;测试时别用 `source <(...)`(macOS bash 3.2 不支持)。
- 排空期间旧调度程序不领新题、也不续跑任何暂停的轨迹(resume 在 pipeline 的 not stopped() 里),暂停轨迹会越积越多——这是用户 17:00 看到 11 项报警的原因。
- 排空不会卡死在"核对中":每个等额度的请求都会先调 broker.reconcile()(quota.py 请求循环),不靠调度程序主循环。
- 06 的 glm-coding-account-b 10-12 16:03、16:24、16:46 连续三次请求结果不明(INCOMPLETE_GENERATION / REQUEST_CONNECTION_INTERRUPTED,流中途断),每次整组停约 21 分钟(request_total_s 1200 + window 60)。05 同一 GLM 地址表现为 CONNECT_FAILED。根子是 Mac mini Wi-Fi(17:00 又掉线约 1 分钟)。
- 用户要让 Codex 值守,交接提示词给的是:读 docs/memory/AGENTS.md 四篇 → 每 30 分钟跑 health_check.py、看 drain-restart-1012.log → 只读排查和补起没起来的调度程序可自己做,取消/杀进程/重启/改配置/删文件/停 Salty 作业先问。
10-12 22:00 补充(375ce4d,已部署,stagerun 每次新进程,马上生效):
- 编译超过阶段时限(PHASE_EXECUTION_DEADLINE_EXCEEDED)、被看门狗判卡死(stalled,一小时没日志没 CPU)也不扣编译次数,和容器被杀、执行器退出合计每条最多免 3 次(stagerun.INFRA_COMPILE_ERRORS)。
- depthpro:05 的 kk1–3、06 的 kk3 共 4 条 QUEUED 改成 CANCELLED(annotation DEPTHPRO_EXPORT_BROKEN_20261012)。三个批次里导出原始模型 8 次全失败、从没通过过。04 的 3 条不在 37 题清单里,不会被领到,没动。
- 还没修:暂停轨迹的名额只在它"只差额度"时才保留。05 upernet.swin.b 18:01 因 CONNECT_FAILED 暂停时还不能续,名额马上被新轨迹(convnext.large)占走;之后能续了却没空位,干等 3 个多小时。修法:pipeline._resume_snapshot 对还有续跑次数、没被隔离的暂停轨迹一律算 pending。要重启才生效,等下次排空重启。
- 姿态题 2 像素规则、校验工具比对不一致(上板精度合格)照旧判 FAIL,等用户定。
- 8 块板子 10-12 22:00 逐块直连正常(4–8 秒);17:16 后上板 11 次、评测 12 次全部成功。
**10-13 补充**:自动续跑每条最多 3 次(lifecycle.max_auto_resumes=3),用完就停到 24 小时过期;05 夜里 02–04 点 GLM S600 账号多次"结果不明",yolov8x.seg/ocrnet/pspnet 三条用完 3 次,报警文字却写"暂停 N 分钟没恢复",有误导。upernet.swin.b 只用了 2 次也没续上,原因未查清。"请求断在半路"(REQUEST_CONNECTION_INTERRUPTED)按设计不自动续。
**10-13 14:50 Qwen 批次停摆的原因**:10-13 08:19 之后 04 没有新结果,13:42 后连新阶段都没有。链条:① Mac mini 仍走 Wi-Fi(en1,网线口 en0 inactive),12:20–14:25 断了约 5 次,GLM 同时刻也有 CONNECT_FAILED,Qwen 服务端 abort/error/preemption 都是 0 → 是网络不是服务端。② GLM 多数断在发出前(CONNECT_FAILED 能自动续);Qwen 服务一次只处理 1 个请求、每个请求流式输出好几分钟,断在半路(REQUEST_CONNECTION_INTERRUPTED / STREAM_IDLE_TIMEOUT)→ broker 把该 execution 记 cancelled=1、generation ABANDONED_UNKNOWN,永不重发,这条轨迹卡死到 24h。③ 每次断都让 Qwen 组 RECONCILING 约 21 分钟,其他在跑的 Qwen 轨迹跟着暂停。④ pipeline.py 我加的"有待续轨迹时空位不领新题"让剩下 3 个空位也闲着。⑤ 在跑的 2 条:depth v1 卡在 06-hbm 2 小时超时循环,convnext.large 在 06-hbm。
**更正(10-14)**:上面 10-13 14:50 那段把"步骤等容器"当成 Salty 排队、把续跑失败归到我的空位改动,都不准;真因见 [[infra-leaks-20261013]](名额泄漏、续跑核对旧回执、账本读取 0.5 秒超时),10-13 20:41 已修。活跃配置是 configs/mini/*.yaml(BENCH_CONFIG),不是 bench/config.yaml;06-hbm 时限按执行记录里的 config 看(仍是 7200)。

View File

@ -1,58 +0,0 @@
---
name: sft-autonomy-tally-20261016
description: "10-16 17:40 按 collector 统计 PASS/SFT/通过题数,以及\"不只写配置\"的条数:GLM 310 条 PASS 里 71% 只写配置,真正用自己改的图编出成品的只有 4 条(3 条进 SFT);v18+ 还没有通过"
metadata:
node_type: memory
type: project
originSessionId: 94b62e94-8a51-48f6-a116-dccb712c9118
modified: 2026-10-16T09:45:04.546Z
---
10-16 17:40 统计(数据来自 DB 和 datasets-final 的 sft/teacher/retained.jsonl 消息记录)。
| | PASS 条数 | 通过的题 | SFT |
|---|---|---|---|
| 04 Qwen | 77 | 59 | 54(按约定不进训练) |
| 05 GLM S600 | 117 | 92 | 101 |
| 06 GLM S100P | 193 | 112 | 164 |
**按做法分(PASS / SFT)**
- 只写配置:
- 04:53 / 36
- 05:78 / 65
- 06:142 / 122
- GLM 合计约 71%。
- 做过本地试验(输入仍用平台的):
- 04:22 / 17
- 05:36 / 33
- 06:47 / 39
- 大多是 v12/v14(旧编译机上有工具);v15 迁到 VM 后几乎没有;v16 起靠 dev 恢复。
- 写过并运行改图或导图代码:
- 04:2 / 1
- 05:3 / 3
- 06:4 / 3
- **正式编译的成品真用了自己的图或校准(逐条核对编译配置):4 条,3 条进 SFT**:
- 05 deeplabv3plus.resnet101 k2:自己改的图 + 自做的 32 张校准;
- 05 fcn.resnet101 k2:拆卷积的图;
- 06 deeplabv3plus.resnet50 k3:拆卷积的图;
- 06 fcn.resnet50 k3:拆卷积的图,没进 SFT。
- 05 yolo11x(v12)虽然写了改图代码,但有绕守卫、在预算外本地编译的问题,见 [[sft-sample-review-20261015]]。
**v18+(平台不给图、必须自己导)**:还没有一条 PASS。数据集里有 4 条 Qwen FAIL、1 条 UNVERIFIED,37 条在跑或有判定。
**v16–v17(10-14 19:00 至 10-15 21:43:有 dev 和速度要求,可自带图,平台仍给图)**
- GLM 共 11 条 PASS(05 有 7 条,06 有 4 条),7 条进 SFT。其中 3 条是成品用自己改的图编的。
- Qwen 共 8 条 PASS,5 条进 SFT。
- 没进 SFT 的通过轨迹,原因都是内容审查 fail 或 unknown,也就是报告和实测结果对不上,或者还没审。
- 用户说"新规则"时要先问清是 v16–17 还是 v18+。
**口径坑**:
- 按消息文本搜 yaml 里的 `onnx_model:`、`cal_data_dir` 会误报,模板文字里就有。要以 stages/_queue/05-compile-*/config.yaml 为准。
- 归档到个人空间的轨迹在 VM 上没有目录,只能看消息。
**Why:** 用户要知道"不再只是写配置"的轨迹有多少、有几条通过并进了 SFT。
**How to apply:**
- 下次复查用同一口径比较,看 v18+ 通过后"成品用自己的图"这一档能不能涨起来。
- 脚本思路:消息里的工具调用做特征,再用磁盘上的编译配置核对。
相关:[[check-20261016-1700]] [[sft-sample-review-20261015]] [[v18-own-inputs-20261015]]

View File

@ -1,39 +0,0 @@
---
name: sft-sample-review-20261015
description: 10-15 抽最新三条 SFT 细读 + 全量 315 条统计:真通过但决策含量低;两条是 Qwen 自己的(不该进训练);版本混杂、过长、含评测题
metadata:
node_type: memory
type: project
originSessionId: 8ceff15d-eec5-43fd-994d-f4fd6b293526
modified: 2026-10-15T07:06:27.533Z
---
用户问"最新三条 SFT 数据真的有效吗"。最终数据集在 Mac mini 虚拟机 `bench/datasets-final/r20260929-0{4,5,6}-training/sft.jsonl`(本机没有;`ssh -p 443 amos@180.76.192.183` 后 `limactl shell bench`)。10-15 15:00 共 315 条(04=52 全是 Qwen,05=100、06=163 全是 GLM),193 个不同的题,每题最多 5 条。
**最新三条(按结束时间)**:05 deeplabv3plus.resnet101(GLM,126 轮,编译 3 次,16.6 小时)、04 superpoint(Qwen,59 轮,1 次编译)、04 foveabox.r50.fpn(Qwen,56 轮,1 次编译)。
- 都是真通过、没作弊,板端评测来自平台;最后报告基本如实。
- foveabox 的 compile.yaml 就是系统提示里的模板原样;superpoint 用平台默认配置。两条都是"一次过",答案在提示词里,决策含量近零。
- 只有 deeplabv3plus 有真诊断:首次编译量化崩(余弦 0.16),逐节点定位到 ASPP 后 3×3 conv(2560→512),CPU/int16 对照实验,图手术拆成 5 个 conv,编译 #2 容器内存超限被杀,改 kl+32 张校准后 #3 过。速度结论只用了静态估计(38.1ms vs 50.7ms),板上实测 506ms 没和默认模型对比。
- 三条 G2(数值余弦)都没过:0.9894 / 0.958 / 0.9856(只记录、不挡通过)。
**全量统计**:
- 版本混杂:v12=110、v14=102、v15=95、v16 只有 8;v16 才有 bench dev 和速度要求,10-15 14:37 又换 v17。最新三条在 8 条里,不能代表整体。
- 上下文长:中位 5.1 万 token,39% 超 6.4 万,8% 超 12.8 万,最长 22.7 万;GLM 轨迹的 usage 把缓存命中单列(input+cacheRead 才是真上下文)。thinking 占大头(GLM deeplab 13.8 万字符)。
- 编译次数(粗数):GLM 263 条里 62% 一次、25% ≥3 次;真有迭代的主要在这 1/4。
- 全部 G3 都是 pass-screen(100 张),G2 失败 136/315(43%);沿用 [[verifier-audit-20261012]] 的结论。
- 工具报错 3/60、7/66、10/125:平台坑(terminal_exec "Unknown protected stage handle"、onnx_model/cal_data_dir 必须在 workspace 内、dev 里 mkdir 不同步)会被学成绕路。
**训练集问题**:
1. 04 的 52 条 Qwen 自己的轨迹被导出器标成可训练,但用户定过 Qwen 数据不进 SFT([[qwen-sft-benchmark]])。训练前要剔掉。
2. 37 道基准题里 29 道在 06 里有 GLM 轨迹(43 条),污染早已知,仍存在;训练实际用哪份数据还没告诉我。
**Why:** 用户想知道这批数据能不能指望它提升 Qwen;Qwen 短板是配置写错、不先本地诊断、归咎平台([[qwen-ability-failures-20261013]]),对应的示范只在少数有迭代的 GLM 轨迹里。
**How to apply:** 回答"数据有没有用"时按"有没有真决策点"分,不按条数;建议按(GLM、编译≥2 次、配置偏离模板/默认、有诊断动作)筛子集,统一提示版本,先量上下文长度再定训练窗口;真正验证只能靠 37 题前后对照。未同步到仓库 docs/memory(本机 bench 检出在别的分支且有未提交改动)。
## 第二轮(10-15 晚):GLM 全量按"整条链路"筛
用户要的好数据 = 像人把活交给 Agent、Agent 自己扛整条链路(不是只写 compile.yaml)。263 条 GLM 先用特征打分,读 21 条候选 + 3 条低分对照(6 个子 agent 分读,我核实了私有源码读取、yolo11x 钻空子、solov2/yolox.x 的关键点)。清单在 `outputs/glm-sft-review-20261015/selection.json`。
- **没有一条符合最理想的样子**:263 条里 0 条自己导图(导出一直是平台阶段),自带图/校准的也极少;只有 v16 的 5 条带 bench dev。我的"自带图/校准"自动标签有误(rtmdet.l、upernet 其实没有),以子 agent 读原文为准。
- 可用(评分≥4)约 10 条,约占 4%:yolox.x、solov2、efficientvit.seg.b1(kk0、kk1)、fcn.resnet101、deeplabv3plus.resnet101、rtmdet.l、efficientnet.b1、foveabox(nash-m kk0/kk1)。其中只有 yolox.x、fcn、deeplab 是"精准修复且速度代价小",其余多靠全 int16 / fp16 落 CPU 兜底(板上 98ms 对静态 9ms、685ms 对 6.5ms),v16 的 1.5 倍速度门槛下会不合格。
- **要剔除**:GLM 8 条(v12×7、v15×1)读过 `cache/adapter-probes/*/private-source`(评测器/阶段源码/参考配置),其中 regnet.y.800mf 直接抄了参考配置;平台漏洞:agent 能经缓存目录读到评判代码。另 yolo11x 用 `python -m horizon_tc_ui.hb_compile` 绕守卫并在预算外本地编译约 14 次;mobilenet.v3.small 用评测集 100 张选配置。
- 最自主的 v16 轨迹没进 SFT:teacher.jsonl 里 GLM v16 共 27 条,用开发环境最多的 rtmdet.s(66 次)、rtmdet.tiny(59)、efficientvit.seg.b1 nash-p(43)等都 TIMEOUT/FAIL,只有 5 条 PASS 进 SFT。v17(400 轮)下午 14:37 起在跑,好数据更可能出在那里。
- 这批 sft.jsonl 是活的:行号会随新增而变,引用请用 id。

View File

@ -1,28 +0,0 @@
---
name: simplify-agent-rules-20261017
description: "10-17 核对:把压缩交还 Pi 不是纯减法(台账改写报错并锁住执行、入口不等 Pi 续跑、导出不认压缩、总开关关着);提示词 SYSTEM.md 89 行 + e2e.md 27 行逐条分\"留事实/删规矩/交给系统\""
metadata:
node_type: memory
type: project
originSessionId: e52f9d49-06fe-4649-af69-fee51d593c88
modified: 2026-10-17T08:03:54.758Z
---
用户方向(10-17):严格管结果、不管过程;少给 agent 定规矩,让模型自己想办法。只列清单、没动系统;计划并入冻结训练版本一次上线。
**Pi 压缩核实(Mac mini 上 Pi 0.73.1 源码 agent-session.js)**
- Pi 在一个长任务里只在“上下文超长报错”时压缩(agent_end 后 `_checkCompaction` 的 overflow 分支),压完用 `setTimeout(continue)` 自动续跑;每次成功回复后重置,可多次压缩。按阈值主动压缩只在整个 prompt 结束后才有,长任务里等于没有。
- 要改的 5 处:①settings.json.tmpl `compaction.enabled:false`;②quota.py do_POST 把上游 400 换成 503 `CONTEXT_TOO_LONG`,Pi 的 isContextOverflow 不认(要原样回 400 + 上游原文,vLLM "maximum context length is N tokens" 能认);③同一处 `_pause` 把这次请求挂成暂停,锁住整个执行,Pi 的压缩请求和续跑都会 409,要改成直接记 FAILED;④pi_session.js 等完事件队列就存档退出,Pi 排的续跑来不及跑,要等它跑完;⑤export/trajectory.py 只从 message_end 重建消息、不认 compaction 事件,不改的话训练数据还是原文、和模型看到的不一致。删掉 context-mask.js 和 04 配置的 context_mask_chars: 450000。
- 风险:Qwen 256k 下压缩请求本身接近窗口,上线前要实测一次;GLM 上下文满以后也会改成压缩(现在算预算用完),所以算新版本。
- 结论:不做 10 行临时修复(04 Qwen 只做训练前基准),直接在冻结版本里换。
**提示词核对(运行中的 agent/VERSION 22)**
- 留(事实/验收):任务和必选门、编译 5 次上限、job.json 为准、只认平台测量、控制机没工具链要用 bench dev、单条命令约 14 分钟、大文件只留在开发环境、图的输入输出要和平台一致、阶段命令清单、等待用 wait_handle、校准图片位置、板上 hbm_runtime 用法、如实报告。
- 删(规矩/教学/系统已执行):不要 sleep>10 分钟、不要 find /、没有 rg、legacy 沙箱说明、stage wait 别轮询(直接删 stage wait 命令更省)、compile.yaml 旋钮的教学注释(kl/mix 首选、per_channel、挑最差节点提 int16)、速度一段的 speed_ref 机制和“不要自己编默认配置”、harness.tools.export_onnx 的提及(72 条照抄)、hbm_bc_samples 选项、512 MiB/40 子目录长说明(系统拒绝时报原因)、“terminal 命令模式”那段、结束说明里的 G1/标准误细则、e2e.md 和 SYSTEM.md 的重复段。
- 交给系统:export 阶段“第一次编译前要跑过”改为编译时自动补;速度进判分后平台自己在板上测默认配置,提示词只留 1.5 倍一句。
- 旋钮教学删掉可能让通过率降,要拿新版本数据对比。repair.md 现在批次没用到,不动。
**10-17 已实现(v23,未提交、未上线)**:工作树 scratchpad/wt(分支 simplify-20261017,基于 origin/feature/salty-20261009 e3697f0)。台账超长记 FAILED 并回 400 context_length_exceeded;入口压缩后编号 +1、等 Pi 续跑完、compaction_end 带 bench_compaction.kept_messages;导出输出 compactions(at/kept/summary_message);删 context-mask.js 和配置透传;models.json contextWindow 改 1000 万(否则暂停退出前 Pi 按 128k 估算先压缩,续跑请求对不上);stage status 与 wait 合一;SYSTEM.md 89→64 行、e2e.md 27→20 行,规矩全删(旋钮字段表保留为事实,文档里 node_config 写法和实际不同)、e2e.md 保留“## 输入和可复用缓存”和“预算:”两个配方改写要用的标记。真实 Pi 测试在 Mac mini /tmp/simplify-test 通过(超长→摘要→新编号重发→完成→导出校验通过)。已知:Pi 拆分轮次时两个摘要请求并发,第二个会被台账 409(只在预算提醒消息落在切点前时发生),失败就按上下文满结束。上线要重启三个采集程序,04 配置里的 context_mask_chars 届时删掉。
**Why:** 用户认为规则太多限制模型能力,且这几天的 bug 多出在“替 agent 安排流程”的代码上。
**How to apply:** 做冻结版本时按这份清单删;以后不再往提示词加规矩,只修把系统故障算到 agent 头上的错误。相关:[[rl-readiness-20261017]] [[check-20261017-0215]] [[v17-autonomy-check-20261015]]

View File

@ -1,65 +0,0 @@
---
name: stability-sweep-20261016
description: "10-16 18:00 全面体检(为\"建成稳定系统\"):最大问题是生产批次同时当试验场(3 天 8 个提示版本、每次改动要重启、重启就取消在跑的),近 7 天 40–93% 的轨迹因系统原因结束、约 4000 轨迹小时白费;其余缺口按层列表"
metadata:
node_type: memory
type: project
originSessionId: 94b62e94-8a51-48f6-a116-dccb712c9118
modified: 2026-10-16T10:55:46.194Z
---
10-16 18:00 体检时的数字和事实:
- **系统原因结束的比例**(CANCELLED / INFRA / UNVERIFIED / LOST / ABANDONED),10-09 到 10-16 每天分别是 100%、93%、52%、89%、58%、40%、54%、65%。合计约 3960 轨迹小时,其中取消占大头。
- **版本和重启太频繁**:
- 10-14 到 10-16 三天里起了 v15 到 v22 共 8 个提示版本;
- 10-12 以来,三个 collector 的日志里分别有 5–6 次启动记录;
- configs 下有 7 个 restart/ops 目录、14 份 mini 配置、5 个备份目录。
- **部署一致**:VM 上 401 个代码文件与分支 HEAD 一致,只有 bench_health_alert.sh 有意没部署。
- **没有自动化保障**:
- 没有 CI,77 个测试文件靠手跑;
- VM 和主机上都没有 cron/timer,监控靠 Codex 会话和人工复查。
- **主机**:
- 默认路由走 Wi‑Fi(en1),网线口空着;
- Clash Verge 在跑,collector 环境里仍有代理变量;
- launchd 里 cc.d-robotics.bench-controller.vm 的上次退出码是 75;
- 断电后能不能自动拉起 collector 没有验证过;
- 密钥以权限 400 的 env 文件存在多个 restart 目录里,共 12 份以上。
- **VM 资源**:
- 内存 11G、负载 30、盘 81%(剩 16G);
- 1.5 小时涨了 4G,其中 3G 是 offload 在 /tmp 打的归档包,上传失败在重试;
- /tmp 里还留着 10-13 的 salty-archive 临时目录;
- 有一个 1.3G 的 dev snapshot 残留;
- quota.sqlite 已到 144MB,还在长。
- **48 小时外部依赖**:
- GLM 失败率约 0.3%(429 一共 10 次,DNS 失败 15 次);
- Qwen 被拒 400 共 10 次;
- 阶段失败里大量记成 unknown:07-board 16 次,08-eval 8 次;
- dev 命令 3568 次,其中执行器退出 206 次、超时 157 次;
- Salty 当时 9 个作业排队。
- **数据口径**:
- 导出时没有速度过滤;
- 验证器不看精度构成;
- 04 的 Qwen 被标成 teacher,所以 54 条进了 SFT;
- SFT 里混着 v12–v17;
- dev 里能读到 harness 源码和评测路径,有 10 条去找过评测集;
- 37 道基准题里有 29 道 GLM 训练数据里也有。
**10-16 19:00 逐条复核,更正上面几处(用户要求解释每条、判断做法是否合理)**
- "没有独立监控"是错的:本机 MacBook 的 launchd `cc.d-robotics.bench-health` 每 10 分钟跑 ~/.bench-health/bench_health_alert.sh,发飞书。盲区:
- health_check.py 只查账号 PAUSED/RECONCILING;05 被 429 时账号状态是 COOLING,16:00–17:22 没报。
- 15:33–16:14 误报"虚拟机没在运行":ssh 的 "remote port forwarding failed for listen port 7897" 警告混进输出第一行被当成状态(虚拟机 10-12 10:17 起没重启过),这 50 分钟其他检查全被挡住。
- "约 4000 轨迹小时"是建号到结束的钟点数(含排队、暂停、74 条 ABANDONED 的 1720 小时)。agent 实际干活(wall_s):取消 922 + 系统出错 637 + 未验证 99 ≈ 1660 小时,占 10-09 以来 agent 总时长约 42%。按 10-14 以来同时长轨迹的通过率估,取消的 245 条约值 65 条通过(偏高,被取消的多是难题);同期实际通过 104 条。245 条取消里 149 条编译过至少一次。
- 盘 19:00 是 68%、剩 25G(当时 81% 是归档包上传重试的临时高点);/tmp 有 737 个 bench-jail-* 残留目录约 1G。
- Clash 已从全局改成规则模式,但只有 bigmodel.cn、36.144.20.182:22、向日葵直连,其余 MATCH,JMS;VM `curl --noproxy '*' cip.cc` 出口仍是洛杉矶;订阅仍 8 分钟更新;en0 仍 inactive。
- 07-board 的 unknown 48 小时 11 次全在 10-15 01–12 点(板子盘满,a263374 已修),10-16 0 次;但各阶段合计 unknown 约 47 次(01-fetch 10、05 6、06 6、08 5…)。
- 429 只记了 RATE_LIMIT,response_headers 为空,正文没存;账本里 429 共 18 次。
- datasets-final 按批次分目录,sft.jsonl 每条带 model 和 agent_config_version:Qwen 进 SFT、版本混杂都不用改代码,训练时不用 04 的文件、按字段筛即可。
- 复核后建议去掉:配置热加载、托管 CI、开机自动拉起采集、扩 VM 内存、量化比例门槛(速度过滤已覆盖全 int16)、按版本分数据集、账本清理、账号用量预警、板卡定时巡检、日报;无损重启和加核先观察/先测。
**Why:** 用户要把它建成稳定系统,问最大问题和还差什么。
**How to apply:**
- 先上低风险的告警和清理,再上发布纪律(冻结版本、小批量试跑、无损重启),最后加固判分。
- 改判分、改提示词、重启、删东西都要用户拍板。
相关:[[v18-zero-pass-why-20261016]] [[sft-autonomy-tally-20261016]] [[bench-restart-hazards]] [[mac-mini-clash-global-proxy-20261013]] [[verifier-audit-20261012]]

View File

@ -1,20 +0,0 @@
---
name: stop-06-20261017
description: 10-17 11:30 06 的 GLM 账号 B 从 06:29 起一直 429,用户说 key 没额度了、让停 06;只 SIGTERM 不取消,原命令/环境存在 configs/stop-06-20261017
metadata:
node_type: memory
type: project
originSessionId: 912a804a-505b-4f45-ac78-d189a02a99ab
modified: 2026-10-17T03:30:49.519Z
---
- 10-17 06:28:57 起 glm-coding-account-b 每次请求都 429(RATE_LIMIT,正文没存),到 11:26 仍是;05 的 glm-coding-s600-account-c 一直正常。用户确认"那个 key 没有额度了"。
- 11:30:04 给 06 collector(pid 1671)发 SIGTERM,没有取消任何轨迹。当时 06:14 条 PAUSED_INFRA(等额度)、2 条 RUNNING,进程等这 2 条结束或暂停后才会退出。
- 拉起用的东西在 VM 的 /mnt/data/ming01.wang/configs/stop-06-20261017/(700 目录,文件 600):cmdline.txt、cwd.txt、logs.txt(日志追加到 logs/collect-s100p-r06-1012-171614.log)、environ.bin(含密钥,用完要删)。BENCH_CONFIG=configs/mini/s100p-r06-v22.yaml。
- 之后两种走法:
- 账号 B 额度恢复:同一次 VM 开机内按存档原命令/环境重启 06,暂停的会自动续。
- 换新 key:要用新组名写新配置(老组规则不能改,见 [[bench-restart-hazards]]),暂停的 14 条续不了,要取消补号。
- 健康检查的 RUNS 里还有 06,collector 退出后会报"调度程序没在跑"、暂停超 1 小时也会报。
**Why:** 用户不想再等这个账号。
**How to apply:** 恢复 06 前先问用户用哪个 key;换 key 时取消暂停的再补号。相关 [[check-20261016-1700]] [[poweroff-20261016]]

View File

@ -1,28 +0,0 @@
---
name: success-rate-breakdown-20261008
description: 04/05/06 成功率低的原因、GLM 和 Qwen 为什么看不出差距(同题对比打平、考卷区分不出)、各类失败的条数
metadata:
node_type: memory
type: project
originSessionId: 27581662-7413-4d12-9a88-979ce883a5f0
modified: 2026-10-07T22:10:15.554Z
---
2026-10-08 分析(04/05/06,排除取消,共 590 条,PASS 249 条,占 42%):
- agent 自己走完、但精度真的超出容差的,只有 2 条。其余失败的构成:
- 被系统中途终止约 200 条:单条命令超 900s 81 条;"模型回复不完整"68 条(实为本地 413 上下文过长 57 条 + 代理不可用 11 条,见 [[quota-unknown-autorecover-fix]]);板卡清理/请求状态不明 53 条。
- "暂停后过期" 70 条:请求状态不明导致暂停,一直停到 24h 被判超时,并没有真正在跑。之前说的"满 24 小时 72 条"其实就是这 70 条,真正跑满 24 小时的只有 3 条。
- 判分和评测问题约 31 条:
- pose 要求关键点平均偏移不超过 2px(registry/gates.yaml:59),算全部 17 个关节、不做置信度过滤;有 7 条 OKS 精度在容差内,只因偏移 2.4–8.4px 判失败。PASS 的那几条偏移在 1.0–1.3px。
- rtmpose 的 11 条全部 UNVERIFIED,原因是 "fixed crop official_model mismatch",评测器验证不了。
- 阶段超时 13 条,导出失败 3 条。
- GLM 和 Qwen 的对比:
- 同任务、同轨迹序号、两边都真正跑了的 56 对:Qwen 30 条 PASS,GLM 33 条。两边都自己走完的 28 对:26 比 27,基本打平。
- 重叠的题几乎全是 cls/det,默认流程一次编译就能过,属于天花板效应。
- Qwen 还没轮到 pose/mde/seg。早先"Qwen 自己走完的成功率 92%"是幸存者偏差,不能拿来和 GLM 比。
- Qwen 自建服务没有前缀缓存,单次请求中位数 39s,GLM 是 3.7s。Qwen 用时长、请求状态不明多,主要是服务的原因,不是模型本身。
**Why:** 用户怀疑 GLM-5.3 应该明显强于 Qwen-3.8-27B。结论是现在这张考卷区分不出两者:简单题两边都满分,难题两边都被系统砍掉或被判分规则卡住。
**How to apply:**
- 比较模型时只用同题配对、并且两边都真正跑过的轨迹;先把"暂停后过期"和"系统终止"剔掉。
- pose 的 2px 规则和 rtmpose 评测器属于判分标准的改动,要用户拍板。
相关:[[quota-unknown-autorecover-fix]]

View File

@ -1,49 +0,0 @@
---
name: v16-quality-20261015
description: 10-15 09:30 对比 v16 与 v15 轨迹:dev 用得多、兜底配置少、慢 1.5 倍的通过少,但样本只有 8 条通过;240 轮上限开始卡最自主的轨迹;吞吐暂时降到约 1/4
metadata:
node_type: memory
type: project
originSessionId: 24818155-52a9-4a37-a413-b2ac9f163981
modified: 2026-10-15T01:43:46.343Z
---
10-15 09:30 复查。v16 上线 14.5 小时,共 53 条轨迹,有判定的 12 条(8 过)。
**自主性(v16 vs v15)**
- 用了 bench dev 的有 52/53 条,共 993 条命令,成功 614 条。v15 迁到 VM 后,171 条里只有 5 条做过本地试验。
- 17 条在 dev 里改图或导出;48 条做过编译或推理试验。
- 3 条提交了自带的图或校准:06 maxvit.t(Einsum 改写)、05 deeplabv3plus.r101(拆 1×1 卷积)、06 vit.h.14(自做校准)。
- 通过轨迹的最终配置:
- v16(8 条):5 条是默认配置一次过,2 条按节点调,1 条全 fp16 兜底(05 efficientvit.seg.b0,板上慢 110 倍仍 PASS)。
- v15:06 有 25/46 全 int16;05 有 31/46 全 fp16;按节点调的只有 4 条和 1 条。
- 比平台默认慢 1.5 倍以上的通过:v16 1/8;v15 的 06 是 17/39,04 是 11/25。
- 说明:先跑完的都是简单题,v16 样本太小,下不了结论。
**代价**
- **240 轮上限开始卡人**:06 已结束的 v16 轨迹里,有 3 条 BUDGET_EXHAUSTED,都是跑满 240 轮。当时 token 只用了 24–51 万,上限 80 万。
- 每条 bench dev 命令算一轮。vit.h.14 自做校准后就被截掉了。
- v15 的预算用完都是 token 到 87 万左右(中位 108 轮)。
- 当时在跑的 v16 轨迹,工具调用已有 150–207 次的不少。
- max_turns 在 configs/mini/*-v16.yaml 的 rollouts 段。collector 里有配置缓存,改了大概率要等重启。
- **吞吐**:v16 上线后 14.5 小时只有 8 条 PASS。前一段 22 小时(10-13 20:41 到 10-14 19:00)有 44 条。原因有三:
- 重启后所有轨迹从头跑;
- 夜里 dev 只有 8 个槽,198 条命令排到执行器超时。07:50 改到 37 个后,只剩 2 条;
- 每条 dev 命令要 75 秒左右。
**Qwen(04)在 v16 下:3 过 3 败,以前是 41 过 8 败**
- 失败的 3 道(mspn.r50、simcc.resnet50、yolo11m.seg),以前都是靠"全模型 int16"通过的,板上比默认配置慢 2.8 倍、1.5 倍、3.4 倍。
- v16 提示里有速度要求,Qwen 不再用这种兜底,改成按节点调,结果没调过:
- 两道 pose 题差在"关键点平均偏移 ≤2 像素"这条附加约束;
- yolo11m.seg 的 mask_ap 从 46 掉到 16。
- mspn 的 25 条 dev 命令里,16 条死在夜里 dev 排队(07:50 已修)。
- 结论:这是考卷变难了,不是 Qwen 变差。训练前基准大多在 v15 下测的,**训练后要在 v16 下比,基准也得在 v16 下重测**。04 按 5 路并行,37 道全测一遍约 4 天。
**Why:** 用户要的是能体现自主性的 SFT 数据;兜底配置和速度兜底不该算好轨迹。
**How to apply:**
- 下次重启考虑把 max_turns 提到约 400,token 上限不动。要用户拍板。
- 导出时按速度过滤(不用重启)还没做。
- 等难题跑完再重新对比 v16 和 v15。
- 访问路径:本机不在 192.168.66.x 网段时,用 `ssh -p 443 amos@180.76.192.183`。
相关:[[agent-autonomy-audit-20261014]] [[overnight-check-20261015]]

View File

@ -1,18 +0,0 @@
---
name: v17-autonomy-check-20261015
description: 10-15 19:40 查 v17 是否真做到"整条链路 agent 自己扛":32 条 GLM 跑 5 小时 0 条结束;全用开发环境,但从零自己导图 0、自备校准 0、只有 1 条图手术;72% 时间在开发环境,9 条各花约 4 小时编默认配置做速度自查
metadata:
type: project
---
用户问:拿到权重/代码/工具链/验证集/板卡后,agent 是否已能自己导图、准备校准、本地诊断、编译、上板、验证并长时间跑完。10-15 19:40 统计 v17(14:37 起)的 32 条 GLM(05 16 条、06 16 条)。
- 32 条全部还在跑,0 条结束;轮数用了 30–134/400。
- 都用了 bench dev(每条 3–40 次);但 32 条都先跑平台导出(03-export),从零自己导图 0 条;正式配置里自带 cal_data_dir 0 条。
- 唯一用自己图的:06 deeplabv3.resnet50 r03,在平台图上把 ASPP 后 2560→512 卷积拆开(fixed_split.onnx,浮点 cos=1.0),正式编译后在上板。
- 正式编译过 11/32;评测 6 次(yolov8n、yolox.x、yolo 系列 seg)量化精度都远低于浮点(如 yolov8n 26.3 对 40.4 mAP),还在迭代。
- 时间:共 154 小时,开发环境 111 小时(72%,其中开发环境里编译 61 小时),正式阶段 15 小时(10%)。9 条(gma、dpt.hybrid、dpt.large、unidepth.convnext.l、flowformer++、raft.large、depth.anything.v1.large/v2.base、segformer.b2)各花约 4 小时在开发环境用默认配置编一次,是提示词里 1.5 倍速度自查要求的;大模型在 4–6 核开发容器里要 4 小时以上。
- 设计上的限制:看不到板卡(只能走正式阶段)、看不到评测集(只能 08-eval)、平台导出永远兜底("不写就用平台导出的图")。
**Why:** 用户要的是"整条链路由 agent 自己扛"的轨迹;当前只是"允许",没有"做到"。
**How to apply:** 提议过(未定):平台按题预算默认配置的延迟写进题目信息、删掉"自己编一次默认";是否去掉平台导出/校准兜底由用户定。v17 第一批结果要几小时后才有。相关 [[agent-autonomy-audit-20261014]] [[sft-sample-review-20261015]] [[restart-20261015]]

View File

@ -1,68 +0,0 @@
---
name: v18-own-inputs-20261015
description: v18–v21(10-15/16):agent 自己导图、做校准(v21 起给训练集原图)、速度参考平台算、bench dev --board 板上自测(板卡容器里直接跑,S100P/S600 都开);dev 排队不算命令时间(待重启);stage wait 10 分钟
metadata:
node_type: memory
type: project
originSessionId: dfb570b8-7fd2-43b1-9acf-f30e5447c88b
modified: 2026-10-16T08:00:47.044Z
---
用户定(10-15 晚):A 平台预算默认配置延迟、删"自己编一次默认";B 去掉平台导图和校准兜底;热部署。
**怎么做的(分支 1f8a7e2、2ed2f5f;21:43 上线,部署脚本 configs/ops-20261015/v18/deploy.sh,备份 config-backup-20261015/before-v18-2143)**
- 只对新轨迹生效:rollout.json 的 agent_config_version >= 18(stagerun.OWN_INPUTS_FROM)。stagerun._ctx_from_rollout 把 own_inputs_required 放进 ctx.extra,04 和 bench dev 都从这里读;v17 在跑的轨迹规则不变。
- 04:v18 的 compile.yaml 必须写 model_parameters.onnx_model 和 calibration_parameters.cal_data_dir,缺了 own_input_invalid(编译前拒、不占次数)。没 --config 也会拒。
- bench dev(worker spec own_inputs):不挂 cache/onnx,data/calib 换成 data/calib-raw。home/outputs 里只有 openmmlab 环境和 pose 人框,没有平台图。
- calib-raw:原始训练图大多没迁过来(个人空间只有 ADE20K、HPatches 的 zip),所以是把平台 128 张 npy 反归一化成模型输入尺寸的 8 位 PNG(personal:/bench/data/calib-raw/<同 calib 路径>/png/,多帧是 _0/_1)。62 套全转好,抽看颜色正常。反归一化按候选(声明的 recipe、imagenet、0-1、0-255、[-1,1])挑落在 0–255 且像照片的那个。缺点:缩放/裁剪已经做了,agent 只需要做归一化、通道、布局。
- 速度参考:scripts/speed_ref.py 在 dev 容器里用默认配置编一次,hb_model_info 估计延迟写 bench/agent/speed_ref/<job_id>.json(jail 只绑 job.json 单个文件,看不到 job 目录;agent/ 目录是绑定的、实时可见),path-guard 放行这个目录。
- 一次开 8 路时 Salty 满了,agent 自己的 dev 作业开始排队,已停掉删掉。改成 configs/ops-20261015/speedref-loop.sh:每 10 分钟只给有 v18 轨迹的题编,3 路、6 核、32Gi;失败的在 runs/zz-speedref/<job>/FAILED 记一笔不重试。
- 已有:mobilenet.v2 0.182 ms、yolov8n(S600) 0.8214 ms;每题 18–40 分钟。
- 提示词 v18:SYSTEM.md / prompts/e2e.md 删平台图和校准位置、删"自己编默认配置";04 Qwen 也共用这套提示词,新 Qwen 轨迹同样是 v18。
- 测试:Salty 全量 base/new 一致(73 个文件),VM 上在沙箱里失败的那些也逐个对比一致;guard-core.test.js 加了两条。
**10-16 夜间(v18 跑了 3 小时、11 条)**
- agent 真的在自己导图、做校准(04 yolo11l.seg 自己的图+校准编过、上板、评测 39.0/48.1)。
- 问题:提示词说"把 /data/calib/ 换成 /data/calib-raw/",agent 去 /data/calib-raw 找,每条花 5–8 次 dev 命令。00:55 改成 dev 环境变量 $BENCH_CALIB_IMAGES(salty_dev.calib_images),实测指向 128 张。
- speed_ref 按轮跑时一轮等最慢的那道,改 --follow 滚动 3 路(configs/ops-20261016/speedref-follow.sh)。
- dev 命令 TERMINAL_DEADLINE_EXCEEDED:有的轨迹 dev 容器按编译历史要 64Gi,Salty 排队超过 agent 给的超时,连 ls 都超时;没修(本地时限在 collector 里定,热改不了)。
**板上自测 v19(10-16 01:18 上线,16a9c50)**
- 用户同意开。代码原有(terminal target=board),但 collector 里的请求检查只认 development 且不能热换,所以 `bench dev --board` 在命令前加标记 `#bench:board\n`,由新进程里的 terminal.run(board_request)转成板上请求。只对 v18+、nash-m 开放;上传当前目录(本地 + 只在开发环境里的大文件,salty_dev.download_dir 从个人空间拉),一次最多 30 分钟、4 GiB,跑完删 VM 上的副本(VM 盘只剩 25G)。
- 板上隔离实测:S100P(rdk-sandbox,容器)要三处改 terminal_jail:系统目录递归绑定(/usr 下有只读主机挂载,用户命名空间里非递归绑定 EINVAL)、加 /host_lib 和只读 /sys(BPU 运行库读核心类型)、透传 LD_LIBRARY_PATH(/usr/hobot/lib)。之后隔离环境里 shadow、/userdata、/root 不可见、无网络,mobilenet.v2 在 BPU 上 0.79 ms,回执确认清理;端到端(hbm 只在开发环境、脚本在本地)也通过。
- S600(s600-40015/40017)是 Docker 默认 seccomp,unshare 任何命名空间都 EPERM,要板卡主机改容器启动参数才行,等用户/板卡负责人定。
- 探测脚本在会话 scratchpad(boardprobe.py、e2eboard.py),用 configs/restart-20261015/withenv.sh + env-06/05 拿板卡密钥,不打印。
**10-16 早上(用户定:板卡容器本身就是隔离、跑完删目录;原图去 36 找;排队不算命令时间)**
- v20(04:52,22bf19d):板上自测去掉命名空间隔离,命令在板卡容器的会话目录里直接跑(board_guard 管进程和板卡锁),$BENCH_OUT 拉回,会话目录照旧删;S100P 0.75 ms、S600 0.97 ms,板上目录确认已删。所有 v18+ 题都能用。
- v21(06:36,dbb717f):calib-raw/<set>/images/ 是训练集原图(原尺寸)+ images.json(顺序、来源、姿态题人框);v18–v20 的轨迹仍用 png/。来源:36 的 JuiceFS 很慢(小文件约 70 秒、并行也只有 1.6 MB/s),所以 COCO 走 images.cocodataset.org(字节一致)、ImageNet 把 4 个 parquet 分片并行分块拉到本地再取同一批行、KITTI/Cityscapes 并行读(zip 按偏移读+CRC)、NYU 从官网下 .mat(sha 与平台一致,索引从 1 开始)、ADE20K/HPatches 从 pool。62 套全过,NYU 每张 rgb 哈希与平台一致。临时文件已清。
- b329385:prepare 给走 Salty 的 dev 请求多 3 小时排队余量,命令 timeout 从容器启动算;prepare 在 collector 里,**下次重启才生效**。
- e543d7b:bench stage wait 一次最多等 10 分钟(原 60 秒)。v18 的 efficientvit.b0 400 轮里 302 轮是轮询,没编译就 BUDGET_EXHAUSTED。
- 截至 06:40:v18 11 条、v19 6 条、v20 2 条;板上自测还没人用;速度参考只算完 3 道(S600 大深度模型一道约 3.5 小时)。
**10-16 下午修复(用户定:A 批热上、B 批写完直接重启、GLM token 2M、可删残留)**
- A 批 v22(21fce5e,15:36 热部署,脚本 configs/ops-20261016/v22/deploy.sh,备份 config-backup-20261016/before-v22-1536):
bench stage wait 两个入口都只看一眼不睡、指向 terminal_exec wait_handle;板上自测先 salty storage ls 量大小(>512 MiB/40 个子目录拒绝)、
取回后到时限也拒绝,拒绝原因作为命令结果回给 agent,snapshot/dev-files 任何路径都删(删了 2.5 GB 残留);
speed_ref 编不出来写 latency_ms_est=null + status unavailable,已给 4 道补上;SYSTEM.md 三处;VERSION 22。
在跑的 speed_ref --follow 进程还是旧代码(新失败只写 FAILED),等它手上 3 道编完再重启它。
- B 批(b248a28、f2322ac,重启上线,脚本 configs/restart-20261016/,配置 configs/mini/*-v22.yaml):
quota 400 且正文说上下文超长 → CONTEXT_TOO_LONG(不续)→ rollout CONTEXT_BUDGET_EXHAUSTED(结局 BUDGET_EXHAUSTED);
broker 内部异常打日志(类型/消息/位置),没发出的 generation 记 BROKER_ERROR 可续;连接被断的请求只占账号 120 秒(UNKNOWN_HOLD_S);
agent/context-mask.js + rollouts.context_mask_chars=450000(只 04 Qwen):发出时省去最早的工具输出、降到 60%,会话文件不动——
Pi 自带 compaction 只在两次 prompt 之间触发(agent_end),单个长任务里永远不触发,所以没用它。GLM max_tokens 2M、不压缩。
- 测试:VM 全量 base/new 对比只有已知不稳定的(explicit_batch、collection、watchdog_stop、execution 的 0.1 秒计时);
test_pi_session 要设 BENCH_TEST_NODE=/home/amos.guest/.local/lib/salty-controller/runtime/bin/node 才不跳过,真 SDK 下掩码测试通过。
- 经 Mac mini 的 ssh 长命令约 5 分钟会被断,远端轮询每次 ≤4 分钟。
**10-16 15:00 体检**
- 板上自测 4 次败 3:agent 在整个工作区(1.2 GB)跑 --board,从个人空间拉回要 40 分钟,超过命令 25 分钟时限,报 exit 2 无原因,VM 留下 2.5 GB snapshot(失败路径不删)。成功那次是 29 MB 小目录、2 分钟。
- broker 内部异常(503 BROKER_UNAVAILABLE,异常被吞不记日志)在 rollout.py 判成 INFRA_INCOMPLETE_GENERATION 直接结束(05 hrnet.w32 v21 白跑 2.9h),应可续。
- 断网一次(14:34–14:46)→ 05 GLM 组和 Qwen 组各卡 RECONCILING 到未知请求的期限(约 20–27 分钟),组里所有轨迹 PAUSED(自动恢复)。
- GLM 真正卡住的是 max_tokens 800k(未命中缓存的输入+输出),约 90–150 轮就用完,400 轮上限摸不到;在跑的有 3 条到 88–96%。
- 速度参考 18 小时只算完 3 道、败 4 道(S600 大深度模型默认配置 6 核 10 小时编不完);32 道 v18+ 题 29 道没有参考,agent 又开始自己编默认配置(maxvit.t)。
- 14 小时 750 条 dev 命令里 42 条排队没开跑就超时(b329385 等重启);agent 常在 dev 里 4 核跑 4–10 小时的 hb_compile。
- Qwen 当天 7 条因上下文超 256k 结束(v18+ 4、v17 3)。
**Why:** 用户要"整条链路 agent 自己扛"的轨迹;见 [[v17-autonomy-check-20261015]]。
**How to apply:** 看 v18 效果时比 v17:自带图/校准是否真自己做(不是 cp)、开发环境里编默认配置的时间是否消失、编译被 own_input_invalid 拒的次数;calib-raw 的局限要告诉用户。相关 [[restart-20261015]] [[agent-autonomy-audit-20261014]]

View File

@ -1,58 +0,0 @@
---
name: v18-zero-pass-why-20261016
description: 10-16 18:00 查 v18+(自己导图)90 条 0 通过的原因:42 条被今天两次重启取消、82/90 是从没通过过的难题、走到评测的都差精度、预算/上下文/4 小时编译上限;值得注意的行为:10 条想找评测集做本地验证(被沙箱挡住),28 条照搬平台导出脚本
metadata:
node_type: memory
type: project
originSessionId: 94b62e94-8a51-48f6-a116-dccb712c9118
modified: 2026-10-16T09:55:21.565Z
---
10-16 18:00 统计 v18+ 共 90 条轨迹(GLM 75,Qwen 15)。
**0 通过的原因(按影响大小)**
1. **重启取消 42 条**:
- 16:20 全量重启,取消 26 条;
- 17:21 05 换 GLM key 再重启,取消 16 条。
- 这 42 条里,34 条还没走到正式编译。
2. **题太难**:82/90 是任何版本都没通过过的题。按类型:深度 26、检测 16、大分类 15、姿态 11、分割 10、光流 7、实例分割 5。
3. **走到评测的 9 条都差精度**:
- Qwen:yolo11l.seg、yolov8n.seg、efficientvit.seg.b1、depth.anything.v2.small;
- GLM:vkitti.small、convnext.small 和 efficientnet.b0 的 verifier 不过;
- vkitti.base 板上评测卡住。
4. **预算和时限**:
- GLM 80 万 token 用完(efficientnet.v2.l、convnext.small);
- efficientvit.b0 有 302 轮在轮询 stage wait,已修;
- Qwen 上下文超 256k 有 5 条;
- S600 大深度模型(zoedepth.n/nk、hypersim.large)正式编译超 4 小时阶段时限,4 次。
5. **开发环境里耗时长**:不少轨迹先在 dev 里 4 核跑 2–6 小时的 hb_compile,dev 累计 5–14 小时还没正式编译。agent 没有办法终止自己卡住的 dev 命令。
6. 05 的 S600 账号 16:00–17:22 一直返回 429,整批停摆。
**值得注意的行为**
- **想找评测集做本地验证:10 条**(GLM 6、Qwen 4)。
- 它们照着 harness 源码里的路径找 /home/ming01.wang/bench-data/t1/<数据集>/n100-s20260914、bench/data/pool、cache/baseline。
- 想做的事:先在本地算精度,再花正式次数。比如 hypersim.small 自己写了 proxy_acc 代理精度评估。
- 沙箱全部挡住:dev 里看不到这些路径。
- 背后的真实需求:缺一份 agent 能用、又不是考题的验证集。
- **"自己导图"多半照搬平台导出脚本**:
- 28 条读了或复制了 harness/tools/export_onnx.py,有的写着"platform-identical recipe";
- 自己写 torch.onnx.export 的有 27 条,两者有重叠。
- harness 源码在 dev 里可见($BENCH_ROOT/harness),评测器的预处理也能读到(eval/common.py),有的就照它来做校准。
- **像工程师的做法**:
- 逐层对比精度:GLM 10 条、Qwen 8 条,例如 swin.b 用 HB_ONNXRuntime 沿网络深度追踪余弦;
- 微基准:造小探针模型测 int8/int16 的开销,4 条,vkitti.base 还试了板上实测;
- 读工具链手册(.horizon/skills):37 条;
- 等价改写:efficientvit.b1 在线性注意力里缩放 relu(q)/relu(k),yolov8n.seg 把 SiLU 换成 tanh 形式。
- **越界尝试**:06 convnext.small 试过一次 `sudo ls` 看缓存权重。
**Why:** 用户问第二版为什么一条都没过,以及改完以后有没有让人惊讶的操作。
**How to apply:**
- 想让 v18+ 出通过,可以考虑:
- 少重启;
- 给 agent 一份不是考题的验证子集;
- dev 命令要能取消;
- 大模型 dev 容器给更多核;
- 重排队列,别只剩难题。
- 这些都要用户拍板。
相关:[[sft-autonomy-tally-20261016]] [[check-20261016-1700]] [[v18-own-inputs-20261015]]

View File

@ -1,28 +0,0 @@
---
name: verifier-audit-20261012
description: 10-12 核查 Qwen 训练前 20 个通过:成绩是真的、没抄没改;但验证器不查是否真 int8(全 int16 也算过)、只抽 100 张图;Qwen 与 GLM 打平是因为系统包办了大半
metadata:
node_type: memory
type: project
originSessionId: 6c848f33-e8d9-4981-ade2-dc77987005d7
modified: 2026-10-12T12:29:53.734Z
---
2026-10-12 用户不信 Qwen 训练前过 20/23、平均编译 2.2 次,要求查验证器。结论:
**成绩是真的**(04 的 20 个通过 + 3 个失败逐条查):
- 全部有板端实测(source=board, n=100),自己按 fp32/quant 重算掉点与 verdict 一致;FP32 基线与 GLM-06 完全相同。
- HBM sha256 全库无重复,没抄别人的模型文件。
- 会话还在盘上的 13 条(10-08 后的;10-08 前 9 条会话已归档没翻):没读别的 run/rollout、没碰 oracle、没写 stages/verdict/claim、没绕开 `bench stage run compile` 直接调 hb_compile;Agent 发起编译次数 = stages 表计数。
- 验证器会判失败:全库 G3 fail 88 次;Qwen 三个失败都是真的(mobilenet.v3.small top1 67→2、yolov8n.seg 掉 21.6%、solov2 掉 7.4%)。
**为什么像"很厉害"**:导出、FP32 基线、上板、评测全是 harness 做的,Agent 主要写 compile.yaml,系统提示(agent/prompts 里 append-system)直接给了一份模板。mspn、simcc、midas、yolox.l 四题 Qwen 与 GLM 板端成绩小数点后 16 位相同——配置几乎一样、编译器结果确定。盘上 10 份 Qwen 通过配置里 6 份就是模板默认值。
**验证器两处偏松(对 GLM、Qwen 一视同仁)**:
1. 不检查是否真 int8:job.json gates 只要 G1(有 hbm、march、输入格式、hb_verifier 余弦≥0.99、日志无 ERROR)+ G3(板端 100 张、相对掉点≤5%);G4 速度只记录。`quant_config.model_config.all_node_type: int16` 照样通过。Qwen efficientnet.b1、mobilenet.v3.large(1.85ms vs GLM 1.31ms)是全 int16 过的;GLM-06 yolo11m.seg 也是。盘上能查到的通过配置:Qwen 2/10、GLM 2/6 全 int16。SFT 用的 GLM 轨迹里也有这招。
2. 只抽 100 张(verdict 'pass-screen'):3–4 张差异在噪声内,如 regnet.y.400mf 78→75 算过,efficientnet.b1 量化后反比 FP32 高 3 张。
**逐图核对(同日 20:30,页面 https://claude.ai/artifact/T4CcvQHeBFFZFrhzqU32A8)**:5 道分类题从板端逐图预测重数,答对张数与报告完全一致;8 位模型与原始模型同答案 95–100/100,GLM 与 Qwen 之间 96–100/100;ConvNeXt-Base 有 14 张三者错得一样,量化只改了 3 张。Qwen 坏掉的 mobilenet.v3.small 只对 2 张、答案集中在几个类。取数方法:分类的 `08-eval/a*/predictions.jsonl`(sample_id 末尾是真实类别号,取 eval.json 的 quant 与 verdict 一致的那个 attempt);原始逐图预测在 FP32 基线缓存 metrics.json 的 `_pred`(顺序同 predictions.jsonl),基线目录从 08-eval/meta.json 的 `--baseline` 参数取,在 personal:/bench/cache/ 下;测试图 personal:/bench/data/t1/<数据集>/n100-s20260914/20260914.tar;类名在 personal:/bench-runtime/home/toolchains/bench-venv/lib/python3.10/site-packages/torchvision/models/_meta.py。检测/分割/姿态/深度只存总分、不存逐图结果。`salty storage cp` 本地目标要写绝对路径,相对路径会静默失败。
**Why:** 用户要靠训练前后对比给训练方向反馈;验证器偏松 + 系统包办会让两边都接近满分,看不出差别。
**How to apply:** 汇报通过率时说明这两处偏松。若用户决定收紧(禁全 int16/float、或加速度门槛),旧结果可按 compile.yaml + 已记录延迟重判,不必重跑;归档的配置在 personal:/bench-root/archive/<run>/<job>/<kk>.tgz。相关:[[qwen-sft-benchmark]] [[success-rate-breakdown-20261008]]

View File

@ -1,7 +1,7 @@
"""reward v0.8:只按本次真实门证据、成本、内容审核和超时计算。
"""reward v0.7:只按本次真实门证据、成本、内容审核和超时计算。
不读取 Oracle 对照或 BLOCKED 正确性标签。BLOCKED 的已测阶段仍保留分数,
但导出器不将未证实的 BLOCKED 纳入奖励训练。板卡不可用(连不上或板上磁盘满)仍按既有规则免超时罚分。
但导出器不将未证实的 BLOCKED 纳入奖励训练。板卡不可用仍按既有规则免超时罚分。
"""
from __future__ import annotations
@ -9,22 +9,21 @@ import json
import sys
from pathlib import Path
RECIPE_VERSION = "v0.8"
RECIPE_VERSION = "v0.7"
BOARD_STAGES = ("07-board", "08-eval")
BOARD_UNAVAILABLE = ("board_offline", "disk_full") # v0.8: 10-15 S600 /userdata filled by kept sessions
PASSING = {"pass", "pass-screen", "pass-weak"}
W = {"G1": 1.0, "G2": 1.0, "G3": 2.0, "cost_turns": 0.5, "cost_tokens": 0.5,
"timeout": 1.0}
def board_unavailable(verdict: dict) -> bool:
"""板卡不可用导致 G2/G3 没法测:G3 unverified,且 07-board 或 08-eval 的最后一次尝试以 board_offline / disk_full 失败。
"""板卡不可用导致 G2/G3 没法测:G3 unverified,且 07-board 或 08-eval 的最后一次尝试以 board_offline 失败。
verdict.stages 只记最后一次 attempt(verify._latest_meta),所以"最后一次"就是这条轨迹结束时板子的状态。"""
g3 = (verdict.get("gates") or {}).get("G3") or {}
if g3.get("verdict") != "unverified":
return False
stages = verdict.get("stages") or {}
return any((stages.get(s) or {}).get("error_class") in BOARD_UNAVAILABLE for s in BOARD_STAGES)
return any((stages.get(s) or {}).get("error_class") == "board_offline" for s in BOARD_STAGES)
def compute(verdict: dict, max_turns: int = 40, max_tokens: int = 200000) -> dict:

View File

@ -202,7 +202,7 @@ class MaterializationTests(unittest.TestCase):
def test_timeout_keeps_verified_stage_rewards(self):
rd = self.add(execution="TIMEOUT")
reward = compute(json.loads((rd / "verdict.json").read_text()), 60, 100)
self.assertEqual(reward["recipe"], "v0.8")
self.assertEqual(reward["recipe"], "v0.7")
self.assertEqual(reward["terms"]["G3"], 2.0)
self.assertEqual(reward["terms"]["timeout"], -1.0)
manifest = ex.materialize(["r"], "data")

View File

@ -661,7 +661,7 @@ def _write_dataset(a, work: Path, ds_id: str) -> dict:
"include_timeout": True, "purpose": a.purpose, "min_reward": a.min_reward},
"stats": stats, "files": file_info,
"notes": ["pipeline.jsonl contains historical deterministic evidence only for purpose=all; SFT/RL export reads Agent evidence only.",
f"Verdict v4 and reward {RECIPE_VERSION} use only this rollout evidence; Oracle references are history only.",
"Verdict v4 and reward v0.7 use only this rollout evidence; Oracle references are history only.",
"SFT/reward candidates are labels; no train split, framework conversion, or online RL readiness is implied.",
"Writers hold .export.lock; manifest is published last. Readers can lock or verify file hashes for a coherent refresh."]}
(work / "manifest.json").write_text(json.dumps(manifest, indent=2, ensure_ascii=False))

View File

@ -25,10 +25,7 @@ TERMINAL_STATES = frozenset({"DONE", "FAILED", "FAIL", "TIMEOUT", "CANCELLED", "
"BUDGET_EXHAUSTED", "INTERRUPTED", "COMPLETED", "ERROR", "DEADLINE_EXCEEDED",
"INFRA_ERROR", "AGENT_ERROR", "LOST"})
OPERATIONS = frozenset({"session", "agent_start", "turn_start", "auto_retry_start", "auto_retry_end",
"tool_execution_start", "tool_execution_update", "tool_execution_end", "compaction_start"})
# Pi 0.73.1 core/messages.js: how a compaction summary reaches the model.
COMPACTION_PREFIX = "The conversation history before this point was compacted into the following summary:\n\n<summary>\n"
COMPACTION_SUFFIX = "\n</summary>"
"tool_execution_start", "tool_execution_update", "tool_execution_end"})
SUPPORTED_UPDATES = frozenset(f"{kind}_{suffix}" for kind in ("text", "thinking", "toolcall")
for suffix in ("start", "delta", "end"))
@ -144,7 +141,7 @@ def inventory(source: FrozenSource) -> dict:
errors.append("UNSUPPORTED_SOURCE")
if metadata.get("state") not in TERMINAL_STATES:
errors.append("SOURCE_NOT_TERMINAL")
messages, operations, segment, turn, compactions = [], [], [], [], []
messages, operations, segment, turn = [], [], [], []
pending = None
ended = False
sdk_headers = []
@ -272,17 +269,6 @@ def inventory(source: FrozenSource) -> dict:
add(payload, observation, event_type="unresolved_mirror")
if len(payloads) != len(candidates):
errors.append("OBSERVATION_UNACCOUNTED")
elif kind == "compaction_end":
# Only a compaction Pi resumed after changed what the model saw; failed ones end the rollout.
operations.append({"type": kind, "observation_id": location, "payload_sha256": payload_sha(event)})
if event.get("willRetry"):
kept = (event.get("bench_compaction") or {}).get("kept_messages")
summary = (event.get("result") or {}).get("summary")
if not isinstance(kept, int) or kept < 0 or not isinstance(summary, str):
errors.append("UNSUPPORTED_SOURCE")
else:
compactions.append({"before": {item["message_id"] for item in messages}, "kept": kept,
"summary": summary, "observation_id": location})
elif kind == "agent_end":
payloads = event.get("messages")
if not segment and isinstance(payloads, list):
@ -360,17 +346,7 @@ def inventory(source: FrozenSource) -> dict:
retained.append(item)
if msg.get("role") == "assistant" and msg.get("stopReason") in ("error", "aborted"):
errors.append("ERROR_CONTENT_RETAINED")
# messages[at:] came after the compaction; the model then saw messages[0] (system), the summary message and
# messages[at - kept:at]. The full history stays in messages.
views = []
for record in compactions:
at = sum(item["message_id"] in record["before"] or item["event_type"] == "system_snapshot" for item in retained)
if record["kept"] > at - 1:
errors.append("COMPACTION_VIEW_UNKNOWN")
views.append({"at": at, "kept": record["kept"], "observation_id": record["observation_id"],
"summary_message": {"role": "user", "content": [
{"type": "text", "text": COMPACTION_PREFIX + record["summary"] + COMPACTION_SUFFIX}]}})
return {"messages": retained, "excluded": excluded, "operations": operations, "compactions": views,
return {"messages": retained, "excluded": excluded, "operations": operations,
"errors": sorted(set(errors)), "tools": tools, "agent_end": ended, "context": context}
@ -398,14 +374,14 @@ def closure(messages):
return {"pass": not any(metrics.values()), **metrics}
def _view(items, tools, compactions):
def _view(items, tools):
# Pi-native blocks preserve thinking, tool args/results and unknown fields.
return redact_preserving({"messages": [item["payload"] for item in items], "tools": tools, "compactions": compactions})
return redact_preserving({"messages": [item["payload"] for item in items], "tools": tools})
def reconstruct(source: FrozenSource) -> dict:
scanned = inventory(source)
view, redaction = _view(scanned["messages"], scanned["tools"], scanned["compactions"])
view, redaction = _view(scanned["messages"], scanned["tools"])
lineage = [{k: item[k] for k in ("message_id", "observations", "event_type")} for item in scanned["messages"]]
excluded = [{k: item[k] for k in ("message_id", "observations", "event_type", "reason")} for item in scanned["excluded"]]
result = {**view, "format": "pi-native-messages/1", "rules_version": RULES_VERSION,
@ -653,7 +629,7 @@ def validate(source: FrozenSource, output: dict) -> dict:
wanted_observations.append(f"{source.manifest['agent/system.md']['sha256']}:0:/system")
supplied = output.get("lineage", []) + output.get("excluded", [])
observed = [obs["observation_id"] for item in supplied for obs in item.get("observations", [])]
expected_view, redaction = _view(scanned["messages"], scanned["tools"], scanned["compactions"])
expected_view, redaction = _view(scanned["messages"], scanned["tools"])
expected_lineage = [{k: item[k] for k in ("message_id", "observations", "event_type")} for item in scanned["messages"]]
expected_excluded = [{k: item[k] for k in ("message_id", "observations", "event_type", "reason")} for item in scanned["excluded"]]
counts = Counter(actual)
@ -668,8 +644,7 @@ def validate(source: FrozenSource, output: dict) -> dict:
"expected": len(wanted_observations), "actual": len(observed)},
"VALID_MESSAGE_COVERAGE": {"pass": covered == len(expected) > 0 and not (set(actual) - set(expected))
and len(actual) == len(expected), "numerator": covered, "denominator": len(expected)},
"PAYLOAD_COVERAGE": {"pass": output.get("messages") == expected_view["messages"] and output.get("tools") == expected_view["tools"]
and output.get("compactions") == expected_view["compactions"]},
"PAYLOAD_COVERAGE": {"pass": output.get("messages") == expected_view["messages"] and output.get("tools") == expected_view["tools"]},
"IDENTITY_DEDUP": {"pass": output.get("lineage") == expected_lineage and output.get("excluded") == expected_excluded
and all(ident in native_ids for ident in direct_order)
and not ({"IDENTITY_AMBIGUOUS", "IDENTITY_PAYLOAD_CONFLICT"} & set(scanned["errors"]))},

View File

@ -10,7 +10,8 @@
bench report [--run r…]
bench verify <rollout_dir>
bench stage run <stage> [--config path] (rollout 内,agent 用)
bench stage status <handle> (只看一眼;等待用 terminal_exec wait_handle)
bench stage wait <handle> [--poll 60]
bench stage status <handle>
"""
from __future__ import annotations
@ -401,8 +402,10 @@ def cmd_stage(a) -> int:
if a.op == "run":
stagerun.submit(a.stage, a.config, recipe=a.recipe, submission=a.submission)
return 0
if a.op in ("wait", "status"): # wait: old name, kept so older prompts' habits still get the hint
if a.op == "wait":
return stagerun.wait(a.stage, a.poll)
if a.op == "status":
return stagerun.status(a.stage)
raise SystemExit("stage op: run | wait | status")

View File

@ -51,8 +51,7 @@ def preflight(policy):
if not paths.secret(endpoint['api_key_env']):
raise ValueError('CREDENTIAL_UNAVAILABLE: ' + alias)
broker() # Validate the persistent shared policy before claim; never send a request.
# 0.5 s (the default) timed out whenever the controller was busy (10-13: ~17 rollouts/hour lost).
quota = quota_snapshot(Path(value['quota_ledger']), timeout_s=5)
quota = quota_snapshot(Path(value['quota_ledger']))
group = next((row for row in quota.get('groups', [])
if row['group_id'] == value['credential_groups'][policy]), None)
if quota['status'] != 'OBSERVED' or group is None:
@ -207,8 +206,7 @@ def freeze_checkpoint(rd, execution):
path = Path(checkpoint['session_path'])
if not path.resolve().is_relative_to((rd / 'workspace/.pi-sessions').resolve()):
raise ValueError('CHECKPOINT_PATH_INVALID')
# A long rollout's session passes 1 MiB (10-15..10-17: 17 rollouts, 4 ended as INFRA_ERROR); no size cap here.
data = config_bytes(rd / 'workspace', str(path), limit=None)
data = config_bytes(rd / 'workspace', str(path))
if hashlib.sha256(data).hexdigest() != checkpoint.get('sha256'):
raise ValueError('CHECKPOINT_CHANGED')
# Binary session snapshots require fsync, like JSON lifecycle records.
@ -253,9 +251,7 @@ def budget_used(rd):
from export.trajectory import _empty_content
path = Path(rd) / 'agent/events.jsonl'
if path.exists():
with path.open(errors='replace') as stream: # events.jsonl reaches hundreds of MB; never read it whole
lines = [line for line in stream if '"turn_end"' in line]
for line in lines:
for line in path.read_text().splitlines():
try:
event = json.loads(line)
except ValueError:

View File

@ -1,44 +0,0 @@
"""Imported by every Python started in a Salty container (the worker puts this directory on PYTHONPATH).
The container sees every host CPU (244) but gets a quota of OMP_NUM_THREADS. onnxruntime sizes its default pool from
the CPU count (125 threads) and ran ~15x slower: 10-15 probe, resnet50 on 4 CPUs, 320-500 ms per run by default vs
22 ms at 4 threads (hmct's fixed 10 threads: ~35 ms). Cap every session at the quota, only once onnxruntime is imported.
"""
import os
import sys
def _cap(module, limit):
original = module.InferenceSession.__init__
def __init__(self, path_or_bytes, sess_options=None, *args, **kwargs):
sess_options = sess_options or module.SessionOptions()
if not 0 < sess_options.intra_op_num_threads <= limit:
sess_options.intra_op_num_threads = limit
original(self, path_or_bytes, sess_options, *args, **kwargs)
module.InferenceSession.__init__ = __init__
class _OnImport:
def __init__(self, limit):
self.limit = limit
def find_spec(self, name, path=None, target=None):
if name != "onnxruntime":
return None
sys.meta_path.remove(self)
import importlib.util
spec = importlib.util.find_spec(name)
if spec is not None and spec.loader is not None:
run = spec.loader.exec_module
def exec_module(module):
run(module)
_cap(module, self.limit)
spec.loader.exec_module = exec_module
return spec
if os.environ.get("OMP_NUM_THREADS", "").isdigit() and int(os.environ["OMP_NUM_THREADS"]) > 0:
sys.meta_path.insert(0, _OnImport(int(os.environ["OMP_NUM_THREADS"])))

View File

@ -9,13 +9,12 @@ import re
DETERMINISTIC = ["unsupported_op", "shape_infer", "export_fail", "calib_nan", "verifier_mismatch", "config_error",
# 门失败(gates.classify_gate_failure)
"accuracy_collapse", "accuracy_drop", "numeric_mismatch", "march_mismatch", "perf_over_budget"]
TRANSIENT = ["oom", "board_offline", "board_busy", "disk_full", "stalled"]
TRANSIENT = ["oom", "board_offline", "board_busy", "stalled"]
ALL = DETERMINISTIC + TRANSIENT + ["unknown"]
# (error_class, compiled regex)。写规则时对着真实 stderr 写,别猜。
RULES: list[tuple[str, re.Pattern]] = [
("board_busy", re.compile(r'"event":\s*"board_busy"|board busy after .*refusing contended', re.I)),
("disk_full", re.compile(r"No space left on device")), # 10-15 S600 /userdata: was "unknown", read as the agent's fault
("config_error", re.compile(r"Key '[^']+' error:|Wrong key '|Missing key|yaml.*(invalid|error)|schema.*error", re.I)), # hb_compile yaml 校验不过:harness/agent 配置写错
("oom", re.compile(r"CUDA out of memory|OutOfMemoryError|std::bad_alloc|Cannot allocate memory|MemoryError|Killed process .* out of memory", re.I)),
("board_offline", re.compile(r"Connection (refused|timed out|reset by peer)|No route to host|ssh: connect to host|board .*offline|hrt_.*: failed to connect", re.I)),

View File

@ -20,25 +20,6 @@ TASK_MODULES = {
}
# A frozen task module may be replaced inside a running run only by a rewrite approved here, pair by pair, as giving
# identical per-sample outputs and metrics; anything else still needs a new run. 10-15 (f56ac2f): board calls batched
# by measured bytes instead of one image per call; the board still infers sample by sample (board_bench.infer_inputs).
EQUIVALENT_SOURCES = {
("95f5b4a5682feaf08920c7beb3a0455931f7d7cab270f6ba2d298a3989025c94", "f348981b3be85712b7c3874036e1ac5b1a8f4ab625f500467f7ea62586653969"), # seg
("3feef7cd074a9e09a0ef98c68bd8d5db09c44b0fb3f0588cfd9185c84237788a", "dca658bfc25d906d5a155d51efdfe9300d9ff7fd1430c715574d64bd61177dd3"), # mde
("5bdd6fd229cd3cf4accb0190e74920d71fc4a5d6a0d0aced5872ca84f7983bb3", "ca570f710ecd4428d02b6b81765de51a4d3b692754c6630d1462c5bc5d1073cd"), # flow
("210f0ea503018b17ed4cb98fe79f82a0a761a80c6308821c020ca7fdc5aab32e", "9a321c31f29a6fd73d4072ac281fd6a5d00dbf5f88f0fc15bc96c6fe092fde32"), # feat
("599cf728e77f3d6ffc8e678457a5b703d4e0de615ead12498275f37142db24d7", "505d37ffc245dc095e7d0d4ec215ed28e4f0105d542e660c5e3ff9ba32bdd9da"), # coco_extra
}
def contract_holds(frozen: dict, current: dict) -> bool:
"""The current module still implements the frozen contract: identical, or an approved rewrite of the same module."""
return current == frozen or (
{**current, "source_sha256": None} == {**frozen, "source_sha256": None}
and (frozen.get("source_sha256"), current.get("source_sha256")) in EQUIVALENT_SOURCES)
def get(task: str):
"""One dispatch point. A task module owns inputs and both metric paths."""
import importlib

View File

@ -15,7 +15,6 @@ class BoardRunner:
"""每次评测传一次 HBM,以小块 NPZ 传输入/输出,退出时清理临时文件。"""
CHUNK = 32
MAX_BYTES = 1 << 30 # inputs + outputs per board call (10-15: one 32-sample solo.r50 call was 12 GB, filled an S600)
def __init__(self, hbm: str, runtime_dir: str = "/userdata/bench"):
self.hbm = str(hbm)
@ -32,12 +31,6 @@ class BoardRunner:
self._session: RemoteSession | None = None
self._closed = False
self._samples = 0
self._sample_bytes = 0
def group_size(self) -> int:
"""Samples per board call: one until a sample's inputs + outputs are measured, then as many as fit
MAX_BYTES. Evaluators group by it too, so one group's outputs also fit in memory."""
return max(1, min(self.CHUNK, self.MAX_BYTES // self._sample_bytes)) if self._sample_bytes else 1
def __enter__(self):
if self._closed:
@ -59,10 +52,8 @@ class BoardRunner:
results: list[list[np.ndarray]] = []
fin, fout = self.tmp / "inputs.npz", self.tmp / "outputs.npz"
try:
start = 0
while start < len(xs):
chunk = xs[start:start + self.group_size()]
start += len(chunk)
for start in range(0, len(xs), self.CHUNK):
chunk = xs[start:start + self.CHUNK]
for x in chunk:
if x.dtype != np.float32:
raise ValueError(f"board evaluation requires float32 inputs, found {x.dtype}")
@ -74,7 +65,6 @@ class BoardRunner:
if info.get("samples") != len(chunk):
raise RuntimeError("board returned an unexpected sample count")
session.download("outputs.npz", fout)
self._sample_bytes = max(self._sample_bytes, (fin.stat().st_size + fout.stat().st_size) // len(chunk))
output_count = len(info.get("outputs", []))
if output_count == 0:
raise RuntimeError("board returned no model outputs")

View File

@ -8,7 +8,7 @@ from __future__ import annotations
import json
import numpy as np
from .common import groups, letterbox, run_batched
from .common import letterbox, run_batched
from .det import CONF, IOU, MAX_DET, _nms
METRICS = {"iseg": "mask_ap", "pose": "oks_ap"}
@ -66,7 +66,8 @@ def predict(runner, t1, input_spec, task, pose_evidence=None, model=""):
from pycocotools import mask as mask_utils
size = int(input_spec.split("x")[-1])
predictions = []
for items in groups(runner, t1.items, 16):
for offset in range(0, len(t1.items), 16):
items = t1.items[offset:offset + 16]
images = [t1.image(it) for it in items]
pre = [letterbox(im, size) for im in images]
outputs = run_batched(runner, [v[0] for v in pre])

View File

@ -163,16 +163,6 @@ def run_batched(runner, xs: list[np.ndarray]) -> list[list[np.ndarray]]:
return [runner(x) for x in xs]
def groups(runner, items: list, size: int = 1):
"""评测器每次交给 run_batched 的样本组:板卡 runner 按实测字节定组大小(一组输出也放得进内存),
其他 runner 用 size。10-15 前分割/深度/光流/特征逐张上板,每张约 10 秒连接开销。"""
start = 0
while start < len(items):
n = runner.group_size() if hasattr(runner, "group_size") else size
yield items[start:start + n]
start += n
class RecipeRunner:
"""A data-only proxy: model callbacks never enter the scoring process."""

View File

@ -97,15 +97,13 @@ def auc(errors, thresholds=(1, 3, 5)):
def evaluate(runner, t1, input_spec, model):
if t1.manifest.get('dataset') != 'hpatches': raise ValueError('HPatches T1 required')
from .common import groups, run_batched
cache = {}; sizes = {}; samples = []
for names in groups(runner, list(dict.fromkeys(item[f] for item in t1.items for f in ('file', 'file2')))):
images = [t1.image({'file': name}) for name in names]
outs = run_batched(runner, [common.preprocess(image, input_spec, model) for image in images])
for name, image, out in zip(names, images, outs, strict=True):
sizes[name] = image.size
cache[name] = common.decode(model, out)
for item in t1.items:
for field in ('file', 'file2'):
name = item[field]
if name not in cache:
image = t1.image({'file': name}); sizes[name] = image.size
cache[name] = common.decode(model, runner(common.preprocess(image, input_spec, model)))
samples.append({'id': item['id'], **pair_metrics(cache[item['file']], cache[item['file2']],
item['H_0to1'], sizes[item['file']], sizes[item['file2']])})
values = auc([s['corner_error_px'] for s in samples])

View File

@ -101,13 +101,12 @@ def aggregate(samples: list[dict]) -> dict:
def evaluate(runner, t1, input_spec: str, model: str) -> tuple[dict, list[dict]]:
from .common import groups, run_batched
samples = []
for items in groups(runner, t1.items):
for item, out in zip(items, run_batched(runner, [inputs(t1, item, input_spec, model) for item in items]), strict=True):
truth, valid = ground_truth(t1, item)
restored = restore_flow(out, input_spec, truth.shape[-2:])
samples.append({"id": item["id"], **sample_metrics(restored, truth, valid)})
for item in t1.items:
truth, valid = ground_truth(t1, item)
out = runner(inputs(t1, item, input_spec, model))
restored = restore_flow(out, input_spec, truth.shape[-2:])
samples.append({"id": item["id"], **sample_metrics(restored, truth, valid)})
return aggregate(samples), samples

View File

@ -152,13 +152,11 @@ def evaluate(runner, t1, input_spec: str, model: str) -> tuple[dict, list[dict]]
raise ValueError("MDE evaluation input recipe mismatch")
if t1.manifest.get("evaluation_protocol") != PROTOCOL:
raise ValueError("T1 must freeze the explicit relative-depth evaluation protocol")
from .common import groups, run_batched
samples = []
for items in groups(runner, t1.items):
for item, out in zip(items, run_batched(runner, [inputs(t1, item, input_spec, model) for item in items]), strict=True):
depth, mask = ground_truth(t1, item)
prediction = restore_prediction(out, depth.shape, model)
samples.append({"id": item["id"], **aligned_metrics(prediction, depth, mask)})
for item in t1.items:
depth, mask = ground_truth(t1, item)
prediction = restore_prediction(runner(inputs(t1, item, input_spec, model)), depth.shape, model)
samples.append({"id": item["id"], **aligned_metrics(prediction, depth, mask)})
if not samples:
raise ValueError("empty depth evaluation")
metrics = {key: float(np.mean([item[key] for item in samples])) for key in ("absrel", "rmse", "delta1")}

View File

@ -177,17 +177,15 @@ def numeric_diagnostic(runner, bdir: Path, calib_source: str) -> dict:
prepared = runner.prepare([images[key] for key in keys])
else:
prepared = [xs[k] for k in keys]
from .common import groups
for group in groups(runner, list(range(len(keys))), len(keys)):
results = run_batched(runner, [prepared[j] for j in group])
if len(results) != len(group):
raise ValueError("G2 runner changed sample count")
for j, outs in zip(group, results):
for i, o in enumerate(outs):
ref_key = f"{keys[j]}/out{i}"
if ref_key not in fp or fp[ref_key].shape != o.shape:
return {"cos_min": None, "reason": "G2 output shapes differ"}
per_out.setdefault(f"out{i}", []).append(cosine(fp[ref_key], o))
results = run_batched(runner, prepared)
if len(results) != len(keys):
raise ValueError("G2 runner changed sample count")
for k, outs in zip(keys, results):
for i, o in enumerate(outs):
ref_key = f"{k}/out{i}"
if ref_key not in fp or fp[ref_key].shape != o.shape:
return {"cos_min": None, "reason": "G2 output shapes differ"}
per_out.setdefault(f"out{i}", []).append(cosine(fp[ref_key], o))
cos_per_output = {k: float(min(v)) for k, v in per_out.items()}
return {"cos_min": min(cos_per_output.values()) if cos_per_output else None,
"cos_per_output": cos_per_output, "n": len(keys), "source": calib_source}

View File

@ -126,9 +126,10 @@ def aggregate(samples: list[dict]) -> dict:
def run_fp32(runner, t1, input_spec: str, out: dict, resize: int = 0) -> None:
adapter, dataset, classes = _recipe(t1, out["model"], runner)
samples = []
from .common import groups, run_batched
from .common import run_batched
size = getattr(runner, "CHUNK", 16) if hasattr(runner, "recipe_adapter") else 1
for items in groups(runner, t1.items, size):
for start in range(0, len(t1.items), size):
items = t1.items[start:start + size]
xs = (runner.prepare([t1.image(item) for item in items]) if hasattr(runner, "recipe_adapter")
else [inputs(t1, item, input_spec, out["model"]) for item in items])
outputs = run_batched(runner, xs)

View File

@ -218,16 +218,6 @@ def resume_policy(state):
raise ExecutionStopped('RESUME_TOOL_POLICY_MISSING')
def paused_credit(state):
"""Time paused for infrastructure (quota, network) is not the agent's: it is handed back on resume, at most one
execution budget in total so a paused rollout that can never resume still ends. 10-14: three rollouts paused
7–16 h by GLM outages resumed with < 1 h left and were cut off mid-compile or mid-eval."""
paused = state.get('paused_at') if state['execution_state'] == 'PAUSED_INFRA' else None
if not paused:
return 0.0
return max(0.0, min(time.time() - paused, state['config']['execution_deadline_s'] - state.get('paused_s', 0.0)))
def current():
return getattr(_local, 'execution', None)
@ -309,7 +299,7 @@ class Execution:
return state.get('stop_reason') or state['execution_state']
if state.get('boot_id') != boot_id():
return 'BOOT_CHANGED_RECONCILE_REQUIRED'
if time.monotonic() >= state['deadline_monotonic'] + paused_credit(state):
if time.monotonic() >= state['deadline_monotonic']:
return 'EXECUTION_DEADLINE_EXCEEDED'
owner_status = process_status(state['owner'])
if owner_status == 'DEAD':
@ -568,7 +558,7 @@ class Execution:
checkpoint=checkpoint, paused_at=time.time())
return self._save(state, 'EXECUTION_PAUSED', reason=reason)
def resume(self, *, automatic=True, counted=True):
def resume(self, *, automatic=True):
with self.locked():
state = self.snapshot()
if self.epoch is not None and self.epoch != state['epoch']:
@ -580,10 +570,6 @@ class Execution:
raise ExecutionStopped('SIDE_EFFECT_UNKNOWN')
if state.get('boot_id') != boot_id():
raise ExecutionStopped('BOOT_CHANGED_RECONCILE_REQUIRED')
credit = paused_credit(state)
state['deadline_monotonic'] += credit
state['paused_s'] = state.get('paused_s', 0.0) + credit
state.pop('paused_at', None)
if time.monotonic() >= state['deadline_monotonic']:
raise ExecutionStopped('EXECUTION_DEADLINE_EXCEEDED')
self._checkpoint(state.get('checkpoint'))
@ -596,12 +582,12 @@ class Execution:
if members != [] or pending:
raise ExecutionStopped('SIDE_EFFECT_UNKNOWN' if members is None else 'WORK_STILL_ACTIVE')
count = state.get('auto_resume_count', 0)
if automatic and counted and count >= state['config'].get('max_auto_resumes', 3):
if automatic and count >= state['config'].get('max_auto_resumes', 3):
raise ExecutionStopped('AUTO_RESUME_BUDGET_EXHAUSTED')
state['epoch'] += 1
self.epoch = state['epoch']
state.update(owner=process_identity(os.getpid()), execution_state='ACTIVE', stop_reason=None,
auto_resume_count=count + int(automatic and counted), resumed_at=time.time(), activity='WAITING_QUOTA')
auto_resume_count=count + int(automatic), resumed_at=time.time(), activity='WAITING_QUOTA')
return self._save(state, 'EXECUTION_RESUMED', automatic=automatic)
def cleanup(self):

View File

@ -364,9 +364,6 @@ def _check_frozen_job(path: Path, requested: dict) -> None:
if "toolchain" not in old and tc.get("profile") is None and tc.get("version") == "legacy-config":
new.pop("toolchain", None)
contract = new.get("task_contract") or {}
from .eval import contract_holds
if old.get("task_contract") and contract_holds(old["task_contract"], contract):
new["task_contract"] = contract = old["task_contract"] # an approved rewrite keeps the frozen contract
legacy_modules = {task: f"harness.eval.{task}" for task in ("cls", "det", "flow", "seg", "mde", "feat")}
legacy_modules.update(pose="harness.eval.coco_extra", iseg="harness.eval.coco_extra", stereo=None, occ=None)
if "task_contract" not in old and new.get("task") in legacy_modules and contract.get("module") == legacy_modules[new["task"]]:
@ -563,7 +560,7 @@ def _resume_snapshot(run_id, job_ids=None, modes=None):
cancel=bool(state.get('cancel_requested')), timeout_s=0)
continue
if (state['execution_state'] != 'PAUSED_INFRA' or state['cleanup_state'] == 'QUARANTINED'
or state.get('cancel_requested')):
or state.get('cancel_requested') or state.get('auto_resume_count', 0) >= state['config'].get('max_auto_resumes', 3)):
continue
try:
resume_policy(state)
@ -572,19 +569,11 @@ def _resume_snapshot(run_id, job_ids=None, modes=None):
readiness = broker().resume_readiness(state['execution_id'])
if readiness['eligible']:
ready.append({**row, '_resume': True, 'runner': 'pi', 'job_state': 'RESUME'})
elif (readiness.get('reason') in ('REQUEST_OUTCOME_UNKNOWN', 'QUOTA_SOURCE_UNAVAILABLE') # ledger busy: unknown, keep the slot
or (readiness.get('quota_group') or {}).get('state') in ('PAUSED', 'RECONCILING') # any cause, e.g. 10-15 00:56 connection cut
or readiness.get('not_before') is not None and readiness.get('reason') in ('COOLING', 'QUOTA_UNAVAILABLE', 'WAITING_QUOTA')):
elif readiness.get('reason') == 'REQUEST_OUTCOME_UNKNOWN' or readiness.get('not_before') is not None and readiness.get('reason') in ('COOLING', 'QUOTA_UNAVAILABLE', 'WAITING_QUOTA'):
pending = True
return ready, pending
def _claimable(resumes, resume_pending, queued):
"""有暂停的轨迹在等(已就绪或只差额度)时,这一轮只续跑、不领新题,空出的名额留给它们;否则它们会一直饿到 24h 过期。
10-15 11:21、11:33:一条续上的同一轮里新题跟着进来,把还在等额度的几条的名额占了。"""
return resumes if (resumes or resume_pending) else queued
def collect_rollouts(run_id: str, job_ids: list[str] | None = None, modes: list[str] | None = None,
parallel: int | None = None, dry_run: bool = False, limit: int | None = None,
log=_stamped, coverage: str = "matrix", watch: bool = False,
@ -820,7 +809,7 @@ def collect_rollouts(run_id: str, job_ids: list[str] | None = None, modes: list[
# A fully paused group has no request left to drive its own recovery.
collection_runtime._broker.reconcile()
resumes, resume_pending = _resume_snapshot(run_id, job_ids, modes)
ready = _claimable(resumes, resume_pending, ready)
ready = resumes + ready
pending = pending or resume_pending
active_jobs = {r["job_id"] for r in futures.values()}
free = n - len(futures)
@ -1122,11 +1111,6 @@ def run_oracle_job(run_id: str, job_id: str, dry_run: bool = False, log=_stamped
for S in (ORDER[:3] if prepare_only else ORDER):
stage = S()
attempt, lane = 0, "normal"
from . import salty_stage
if salty_stage.selected(ctx, stage.name):
pending = salty_stage.pending_attempt(ctx.stage_root)
if pending is not None:
attempt = pending
while True:
ctx.attempt, ctx.lane = attempt, lane
with db.connect() as con:
@ -1140,8 +1124,6 @@ def run_oracle_job(run_id: str, job_id: str, dry_run: bool = False, log=_stamped
with scheduler.acquire_stage(ctx, stage.pool) as slot:
ctx.extra["slot"] = slot
out = stage.run(ctx)
if out.error_class == "SIDE_EFFECT_UNKNOWN" or out.meta.get("cleanup_state") in {"QUARANTINED", "PENDING", "UNKNOWN"}:
out.state, out.error_class = "FAIL", "SIDE_EFFECT_UNKNOWN"
with db.connect() as con:
db.stage_finish(con, run_id, job_id, stage.name, attempt, out.state, out.exit_code, out.error_class, -1)
log(f"[{job_id}] {stage.name} a{attempt:02d} {lane}: {out.state} ec={out.error_class} {out.duration_s:.1f}s")
@ -1156,15 +1138,6 @@ def run_oracle_job(run_id: str, job_id: str, dry_run: bool = False, log=_stamped
if calib_meta:
ctx.extra["calib_source"] = calib_meta.get("calib_source")
break
# Unknown side effects cannot be converted into a fresh attempt,
# even when a lower layer returned the legacy STALLED state.
if out.error_class == "SIDE_EFFECT_UNKNOWN" or out.meta.get("cleanup_state") in {"QUARANTINED", "PENDING", "UNKNOWN"}:
out.state, out.error_class = "FAIL", "SIDE_EFFECT_UNKNOWN"
rec.update(state=out.state, error_class=out.error_class)
from . import execution
if execution.current():
execution.current().quarantine("SIDE_EFFECT_UNKNOWN")
break
# 失败:决定重试
if out.state == "STALLED" and attempt < cfg["retry"]["stalled"]:
attempt, lane = attempt + 1, "long"; continue
@ -1173,12 +1146,6 @@ def run_oracle_job(run_id: str, job_id: str, dry_run: bool = False, log=_stamped
break
if out.state not in ("OK", "SKIPPED"):
ec = out.error_class or ("stalled" if out.state == "STALLED" else "unknown")
if ec == "SIDE_EFFECT_UNKNOWN" or out.meta.get("cleanup_state") in {"QUARANTINED", "PENDING", "UNKNOWN"}:
# A board's ordinary failed measurement may be nonblocking.
# Unknown cleanup/remote effects must stop the entire pipeline.
terminal, error_class, stage_failed = "UNVERIFIED", "SIDE_EFFECT_UNKNOWN", stage.name
result["cleanup_state"] = "QUARANTINED"
break
if stage.name == "07-board":
result["measurements"]["G4"] = None # unverified,不阻塞
result["stages"][stage.name]["note"] = "board failed after retries → G4 unverified"

View File

@ -20,31 +20,15 @@ import json
import math
import os
from pathlib import Path
import re
import secrets
import socket
import sqlite3
import sys
import threading
import time
import traceback
from urllib.parse import urlsplit, quote
import uuid
# Failures proved to have sent nothing: wait and resend inside the broker instead of pausing the rollout.
RETRY_UNSENT = ('CONNECT_FAILED', 'DNS_FAILED', 'DNS_TIMEOUT')
# A 400 saying the request outgrew the model's window (vLLM "maximum context length", GLM "exceeds max length"):
# the agent used up its context, not an infrastructure fault. The generation fails (it fences nothing) and Pi gets a
# 400 its isContextOverflow() recognizes, so Pi compacts and continues; same request again would fail, never resumed.
CONTEXT_OVERFLOW_BODY = b'{"error":{"code":"context_length_exceeded","message":"context_length_exceeded"}}'
CONTEXT_FULL = re.compile(rb'maximum context|context length|context window|exceeds? (the )?max(imum)? length|prompt is too long', re.I)
# How long a request whose connection dropped holds its account (it used to hold until the request's own
# deadline, ~20-27 min: 10-16 14:35 one drop paused every 05 GLM and 04 Qwen rollout). Most servers stop
# generating when the client goes away. ponytail: fixed for every group, a policy field when one needs longer.
UNKNOWN_HOLD_S = 120
class QuotaError(Exception):
def __init__(self, reason: str, status: int = 503):
self.reason, self.status = reason, status
@ -504,13 +488,7 @@ class Broker:
'message':exc.reason}}).encode()
except (ValueError, socket.timeout):
status, content_type, payload = 400, 'application/json', b'{"error":{"code":"INVALID_LOCAL_REQUEST"}}'
except Exception as exc:
# 10-16 05 hrnet.w32: swallowed with no trace and the rollout ended. Type, message and place only
# (never the request); a generation that sent nothing resumes like any other unsent one.
where = traceback.extract_tb(exc.__traceback__)[-1:]
print(f'BROKER_UNAVAILABLE {type(exc).__name__}: {str(exc)[:200]} at '
f'{[(f.filename.rsplit("/", 1)[-1], f.lineno) for f in where]}', file=sys.stderr, flush=True)
broker.pause_unsent(self.headers.get('X-Bench-Generation-Id'))
except Exception:
status, content_type, payload = 503, 'application/json', b'{"error":{"code":"BROKER_UNAVAILABLE"}}'
try:
self.send_response(status)
@ -582,27 +560,6 @@ class Broker:
if owner == execution_id:
self._interrupt(conn)
def pause_unsent(self, generation):
"""After an internal broker error: a generation none of whose attempts may have reached the provider is
paused as BROKER_ERROR, which resumes like a connect failure. Anything possibly sent is left as it is."""
if not generation:
return
try:
with self._db() as con:
row = con.execute('SELECT e.group_id FROM generations g JOIN executions e ON e.execution_id=g.execution_id '
'WHERE g.generation_id=?', (generation,)).fetchone()
if not row:
return
with self._group_lock(row['group_id']), self._db() as con:
con.execute('BEGIN IMMEDIATE')
state = con.execute('SELECT state FROM generations WHERE generation_id=?', (generation,)).fetchone()['state']
maybe_sent = con.execute('SELECT 1 FROM attempts WHERE generation_id=? AND (sent IS NOT 0 OR ended_at IS NULL)',
(generation,)).fetchone()
if state in ('QUEUED', 'RESUMABLE') and not maybe_sent:
self._pause(con, generation, 'BROKER_ERROR')
except sqlite3.Error:
pass # the rollout then ends as before
def reconcile(self):
"""Unknown transport end is a quarantine, not a lease-based refund.
@ -627,7 +584,7 @@ class Broker:
self._event(con, group, 'UNKNOWN_SETTLED_PROBE_ALLOWED')
for attempt_id in stale: # outside the group flock: abandon_unknown takes it itself
try:
self.abandon_unknown(attempt_id, operator_reason='AUTO: local deadline + token window elapsed', resumable=True)
self.abandon_unknown(attempt_id, operator_reason='AUTO: local deadline + token window elapsed')
except QuotaError:
pass # another broker settled it first
@ -643,14 +600,11 @@ class Broker:
con.execute("UPDATE quota_groups SET state='COOLING',epoch=epoch+1,target=1,reason=NULL,cooldown_until=MAX(cooldown_until,?),updated_at=? WHERE group_id=?", (now, now, group))
return True
def abandon_unknown(self, attempt_id: str, *, operator_reason: str, resumable: bool = False):
def abandon_unknown(self, attempt_id: str, *, operator_reason: str):
"""Explicit operator release of local capacity, never provider-end proof.
The attempt stays unknown for audit (usage and ended_at untouched). An operator
release fences the generation forever. reconcile() calls this with resumable=True
once a token window has passed: the provider can no longer charge that window, so
the turn may be replayed (one request's tokens, counted against the resume budget)
instead of losing the rollout (10-13/14: 16 rollouts lost this way).
The old generation is fenced forever. Original usage and ended_at stay
untouched. reconcile() also calls this once a token window has passed.
"""
if not isinstance(operator_reason, str) or not operator_reason.strip():
raise ValueError('explicit operator reason required')
@ -671,19 +625,15 @@ class Broker:
execution_id = con.execute('SELECT execution_id FROM generations WHERE generation_id=?',
(generation,)).fetchone()[0]
con.execute("UPDATE attempts SET state='ABANDONED_UNKNOWN' WHERE attempt_id=?", (attempt_id,))
if resumable:
con.execute("UPDATE generations SET state='PAUSED',reason='REQUEST_OUTCOME_UNKNOWN',updated_at=? WHERE generation_id=?", (now, generation))
else:
con.execute("UPDATE generations SET state='ABANDONED_UNKNOWN',reason='OPERATOR_ABANDONED_UNKNOWN',updated_at=? WHERE generation_id=?", (now, generation))
con.execute('UPDATE executions SET cancelled=1 WHERE execution_id=?', (execution_id,))
con.execute("UPDATE generations SET state='ABANDONED_UNKNOWN',reason='OPERATOR_ABANDONED_UNKNOWN',updated_at=? WHERE generation_id=?", (now, generation))
con.execute('UPDATE executions SET cancelled=1 WHERE execution_id=?', (execution_id,))
probe_allowed = self._probe_if_settled(con, group)
self._event(con, group, 'OPERATOR_ABANDONED_UNKNOWN', generation,
attempt_id=attempt_id, operator_reason=operator_reason,
released_at=now, provider_end_confirmed=False,
original_reserved_tokens=row['reserved_tokens'],
original_charged_tokens=row['charged_tokens'], probe_allowed=bool(probe_allowed), resumable=resumable)
if not resumable:
self._cancel_requested.setdefault(execution_id, threading.Event()).set()
original_charged_tokens=row['charged_tokens'], probe_allowed=bool(probe_allowed))
self._cancel_requested.setdefault(execution_id, threading.Event()).set()
return True
def confirm_end(self, attempt_id: str, receipt_id: str):
@ -718,19 +668,13 @@ class Broker:
if not gen:
return result | {'reason':'NO_PAUSED_GENERATION'}
result['generation_id']=gen['generation_id']
# Only a resume after a possibly delivered request spends the budget; connect/DNS failures and
# waiting behind another request's unknown end sent nothing (10-14: 4 rollouts stuck at 3/3 on those).
last=con.execute('SELECT sent FROM attempts WHERE generation_id=? ORDER BY started_at DESC LIMIT 1',(gen['generation_id'],)).fetchone()
result['counted']=last is not None and last['sent'] != 0
policy=self.groups[ex['group_id']]
if (ex['cancelled'] or gen['state']!='PAUSED'
# CONNECT_FAILED / REQUEST_DEADLINE_BEFORE_SEND / DNS_*: nothing was sent, replay is safe.
# INCOMPLETE_GENERATION: a cut-off reply whose end the ledger recorded (the open-attempt check below still applies).
or gen['reason'] not in ('QUOTA_WAIT_BUDGET_EXHAUSTED','REQUEST_OUTCOME_UNKNOWN','CONNECT_FAILED','REQUEST_DEADLINE_BEFORE_SEND','INCOMPLETE_GENERATION','DNS_FAILED','DNS_TIMEOUT','BROKER_ERROR')
or gen['reason'] not in ('QUOTA_WAIT_BUDGET_EXHAUSTED','REQUEST_OUTCOME_UNKNOWN')
# A generation whose own request end is unknown never replays.
or con.execute("SELECT 1 FROM attempts WHERE generation_id=? AND ended_at IS NULL AND state <> 'ABANDONED_UNKNOWN'",(gen['generation_id'],)).fetchone()):
or con.execute('SELECT 1 FROM attempts WHERE generation_id=? AND ended_at IS NULL',(gen['generation_id'],)).fetchone()):
return result | {'reason':'RESUME_NOT_ELIGIBLE'}
if (result['counted'] and ex['resumes']>=policy.max_auto_resumes) or gen['attempts']>=policy.max_attempts:
if ex['resumes']>=policy.max_auto_resumes or gen['attempts']>=policy.max_attempts:
return result | {'reason':'RESUME_BUDGET_EXHAUSTED'}
if gen['reserved_tokens']<=0:
return result | {'reason':'RESERVATION_METADATA_MISSING'}
@ -756,7 +700,7 @@ class Broker:
def resume_readiness(self, execution_id: str):
"""Bounded read-only hint, never a promise of future HTTP admission."""
with self._db() as con:
deadline=time.monotonic()+5 # .5 s failed under load; paused rollouts bounced back for hours
deadline=time.monotonic()+.5
con.set_progress_handler(lambda:int(time.monotonic()>=deadline),1000)
try:
return self._resume_readiness(con,execution_id,self._now())
@ -776,7 +720,7 @@ class Broker:
readiness=self._resume_readiness(con,execution_id,self._now())
if not readiness['eligible'] or readiness['generation_id']!=generation_id:
raise QuotaError(readiness['reason'] if not readiness['eligible'] else 'GENERATION_MISMATCH',409)
con.execute('UPDATE executions SET resumes=resumes+? WHERE execution_id=?',(int(readiness['counted']),execution_id))
con.execute('UPDATE executions SET resumes=resumes+1 WHERE execution_id=?',(execution_id,))
con.execute("UPDATE generations SET state='RESUMABLE',wait_since=? WHERE generation_id=?",(self._now(),generation_id))
def _begin(self, token, body, requested_id):
@ -969,8 +913,6 @@ class Broker:
('FINISHED' if known_end else 'UNKNOWN',now if known_end else None,status,reason,json.dumps(_allowed_headers(headers)),usage if usage is not None else row['reserved_tokens'],'REPORTED' if usage is not None else 'ESTIMATED',attempt))
con.execute('UPDATE attempts SET transport_reason=? WHERE attempt_id=?',(transport_reason,attempt))
if not known_end:
if transport_reason=='REQUEST_CONNECTION_INTERRUPTED':
con.execute('UPDATE attempts SET deadline=MIN(deadline,?) WHERE attempt_id=?',(now+UNKNOWN_HOLD_S,attempt))
self._pause(con,generation,'CANCELLED' if ex['cancelled'] else reason)
con.execute("UPDATE quota_groups SET state='RECONCILING',reason='REQUEST_OUTCOME_UNKNOWN',updated_at=? WHERE group_id=? AND NOT (state='PAUSED' AND reason IS NOT NULL AND reason!='REQUEST_OUTCOME_UNKNOWN')",(now,group))
elif usage is not None and usage>row['reserved_tokens']:
@ -992,8 +934,6 @@ class Broker:
con.execute('UPDATE quota_groups SET state=?,target=?,recovery_at=?,successes=0,failures=0,updated_at=? WHERE group_id=?',('OPEN' if target==policy.max_concurrency else 'RECOVERING',target,now,now,group))
else:
con.execute('UPDATE quota_groups SET successes=? WHERE group_id=?',(successes,group))
elif reason=='CONTEXT_TOO_LONG':
con.execute("UPDATE generations SET state='FAILED',reason=?,updated_at=? WHERE generation_id=?",(reason,now,generation))
elif reason in ('RATE_LIMIT','UPSTREAM_TRANSIENT'):
if reason!='RATE_LIMIT' or row['http_status']!=429:
self._cool(con,group,policy,headers,row['epoch'])
@ -1102,37 +1042,12 @@ class Broker:
watcher = threading.Thread(target=watch_client, daemon=True)
watcher.start()
try:
while True:
try:
return self._generate(scope, upstream, policy, generation, data, body, reserve, client_connected)
except QuotaError as exc:
if exc.reason not in RETRY_UNSENT or not self._wait_unsent(upstream.quota_group, generation, policy):
raise
return self._generate(scope, upstream, policy, generation, data, body, reserve, client_connected)
finally:
stopped.set()
if watcher is not None:
watcher.join()
def _wait_unsent(self, group, generation, policy):
"""Nothing reached the provider: wait here and resend rather than pause the whole rollout.
Each retry is charged to the execution's queue budget (the wait plus one
connect/DNS deadline), so the client timeout bound holds; once the budget
is spent the generation stays paused as before.
"""
if self._closing.wait(policy.cooldown_initial_s):
return False
charge = policy.cooldown_initial_s + policy.connect_timeout_s
with self._group_lock(group), self._db() as con:
con.execute('BEGIN IMMEDIATE')
row = con.execute('SELECT g.state,g.execution_id,e.wait_spent,e.cancelled FROM generations g JOIN executions e ON e.execution_id=g.execution_id WHERE g.generation_id=?', (generation,)).fetchone()
if row['state'] != 'PAUSED' or row['cancelled'] or row['wait_spent'] + charge > policy.wait_budget_s:
return False
con.execute('UPDATE executions SET wait_spent=wait_spent+? WHERE execution_id=?', (charge, row['execution_id']))
con.execute("UPDATE generations SET state='QUEUED',updated_at=? WHERE generation_id=?", (self._now(), generation))
self._event(con, group, 'UNSENT_RETRY', generation)
return True
def _generate(self, scope, upstream, policy, generation, data, body, reserve, client_connected):
url=urlsplit(upstream.base_url)
started=time.monotonic()
@ -1167,15 +1082,13 @@ class Broker:
elif status is not None and status>=500:
reason='UPSTREAM_TRANSIENT'
elif status is None or not 200<=status<300:
reason='CONTEXT_TOO_LONG' if status==400 and CONTEXT_FULL.search(payload or b'') else 'UPSTREAM_REJECTED'
reason='UPSTREAM_REJECTED'
else:
reason=validate_completion(payload,bool(data.get('stream')))
usage=_usage(payload,bool(data.get('stream'))) if reason=='COMPLETE' else None
self._finish(upstream.quota_group,generation,attempt,policy,reason,known_end=known_end,status=status,headers=headers,usage=usage,transport_reason=error)
if reason in ('RATE_LIMIT','UPSTREAM_TRANSIENT') and known_end:
continue
if reason=='CONTEXT_TOO_LONG':
return 400,'application/json',CONTEXT_OVERFLOW_BODY
if reason!='COMPLETE':
raise QuotaError(reason)
with self._db() as con:

View File

@ -102,10 +102,11 @@ def _recipe_system_prompt(system: str) -> str:
section = """## 共享资源与正式验收
按任务提示为当前操作提交必要文件:`bench stage run <stage> --recipe recipe.json` 冻结新版本,`--submission SUBMISSION_HANDLE` 复用那次提交。探索次序由你决定;已有 ONNX 可先 export,再试 compile。编译使用 `--config compile.yaml`,文件位于 workspace。
共享工具链、板卡和正式评分通过原 stage 工具,使用结果前调用 `terminal_exec({"wait_handle": "<handle>"})`,一次等待原阶段完成。日志和 meta 保存真实执行与失败;按返回的产物路径使用结果。固定任务目标、评测数据与评分阈值不可修改,claim 不能代替独立验收。
本任务使用私有 HOME;workspace 内可 cd、使用变量并安装私有依赖。共享工具链保持只读。
"""
return before + section + end + after
result = before + section + end + after
return result.replace("现有 Pi legacy 沙箱仍保留 shell 路径限制;工具报限制时按其说明使用明确路径。不要为私有环境安装修改共享工具链。",
"本任务使用私有 HOME;workspace 内可 cd、使用变量并安装私有依赖。共享工具链保持只读。")
def setup_models_json(force: bool = False) -> Path:
@ -313,9 +314,7 @@ def pi_command(rd: Path, prompt: str, mode: str = "e2e") -> list[str]:
def run_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str, dry_run: bool = False,
*, _claimed: bool = False, runner: str = "pi") -> dict:
from .runners import get as get_runner
# The collector checked quota right before claiming; a second 0.5 s ledger read here turned busy
# moments into UNVERIFIED rollouts that never ran (10-13: 59).
get_runner(runner).check(policy, execute=not dry_run and not _claimed)
get_runner(runner).check(policy, execute=not dry_run)
if not dry_run and runner == "pi":
from .collection_runtime import config
value, _ = config()
@ -424,7 +423,7 @@ def resume_rollout(run_id: str, job_id: str, kk: int) -> dict:
readiness = runtime.broker().resume_readiness(state['execution_id'])
if not readiness['eligible']:
raise ValueError('RESUME_NOT_READY: ' + str(readiness.get('reason')))
execution.resume(counted=readiness.get('counted', True))
execution.resume()
try:
runtime.broker().resume(state['execution_id'], checkpoint['generation_id'],
checkpoint_complete=checkpoint.get('resumable', False), side_effects_known=True)
@ -450,12 +449,6 @@ def resume_rollout(run_id: str, job_id: str, kk: int) -> dict:
raise
def _last_generation_reason(execution):
from . import collection_runtime as runtime
pending = (runtime.broker().execution_status(execution.snapshot()['execution_id']) or {}).get('generations', [])
return pending[-1].get('reason') if pending else None
def _execute_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str, dry_run: bool, runner: str = "pi", *, resume=False) -> dict:
from .runners import get as get_runner
adapter = get_runner(runner)
@ -553,10 +546,7 @@ def _execute_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str,
quota_state = runtime.broker().execution_status(execution.snapshot()['execution_id'])
pending = (quota_state or {}).get('generations', [])
reason = pending[-1].get('reason') if pending else 'INCOMPLETE_GENERATION'
# A 400 is the provider refusing this exact request (e.g. the context outgrew the model): no resume can accept
# it, so a pause only held the rollout until its deadline (10-15 04 retinanet.r101: 760k-token request).
if (reason not in ('UPSTREAM_REJECTED', 'CONTEXT_TOO_LONG') and checkpoint.get('resumable') and checkpoint.get('phase') == 'PAUSED_INFRA'
and checkpoint.get('closure') and pending and pending[-1]['state'] == 'PAUSED'):
if checkpoint.get('resumable') and checkpoint.get('phase') == 'PAUSED_INFRA' and checkpoint.get('closure') and pending and pending[-1]['state'] == 'PAUSED':
execution.pause(reason, checkpoint)
rj = json.loads((rd / 'rollout.json').read_text())
rj.update(state='PAUSED_INFRA', stop_reason=reason, exit_code=r.exit_code)
@ -575,10 +565,9 @@ def _execute_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str,
from .verify import _cost
cost = runtime.budget_used(rd) if execution else _cost(rd)
budget_exhausted = cost["turns"] >= rj["max_turns"] or cost["tokens"] >= rj["max_tokens"]
context_full = bool(execution) and r.exit_code == 76 and _last_generation_reason(execution) == 'CONTEXT_TOO_LONG'
if budget_exhausted or context_full:
if budget_exhausted:
rj["state"] = "TIMEOUT"
rj["timeout_reason"] = "context window full" if context_full else "budget (events verified)"
rj["timeout_reason"] = "budget (events verified)"
if r.state == "STALLED" or r.exit_code == 75:
rj["state"] = "TIMEOUT"; rj.setdefault("timeout_reason", r.stalled_reason or "budget")
elif rj.get("state") != "TIMEOUT":
@ -591,9 +580,9 @@ def _execute_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str,
stop_reason = execution.snapshot().get('stop_reason') or (
'BUDGET_EXHAUSTED' if budget_exhausted or r.exit_code == 75 else r.stalled_reason)
if r.exit_code == 76 and not stop_reason:
# The agent filled the model's window: its own budget ran out, like turns or tokens (outcome BUDGET_EXHAUSTED).
last = _last_generation_reason(execution)
stop_reason = 'CONTEXT_BUDGET_EXHAUSTED' if last == 'CONTEXT_TOO_LONG' else 'INFRA_' + (last or 'INCOMPLETE_GENERATION')
quota_state = runtime.broker().execution_status(execution.snapshot()['execution_id']) or {}
pending = quota_state.get('generations', [])
stop_reason = 'INFRA_' + ((pending[-1].get('reason') if pending else None) or 'INCOMPLETE_GENERATION')
# A resumable quota pause returned above. Every terminal path must fence
# the broker BEFORE publishing a finished rollout, including SDK errors.
runtime.broker().cancel(execution.snapshot()['execution_id'])

View File

@ -1,776 +0,0 @@
"""Durable, single-submission boundary for an already authorized Salty 0.2 CLI.
This module never installs/logs in, builds an image, stops jobs, reads credentials
or publishes to shared storage. A caller supplies a credential-free bundle and
worker plus an immutable, explicit image. Local timeouts leave the CLI process
running and quarantine the operation. Receipt recovery never submits again.
prepare() creates an exclusive receipt directory and snapshots its two inputs.
submit(), poll(), fetch() and cleanup() serialize through the receipt lock. The
remote directory is derived solely from the receipt UUID, never caller text.
Job completion only describes transport state; the caller must verify returned
worker results before treating a stage as successful.
"""
from __future__ import annotations
from contextlib import contextmanager
from dataclasses import dataclass
import errno
import fcntl
import hashlib
import json
import math
import os
from pathlib import Path, PurePosixPath
import re
import shutil
import subprocess
import time
from typing import Callable, Mapping, Sequence
import uuid
SCHEMA = "bench-salty-transport.v1"
JOB_PATTERN = r"salty-[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"
TERMINAL = {"succeeded", "failed"}
SAFE_ENV = {
"OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS",
"NUMEXPR_NUM_THREADS", "PYTHONDONTWRITEBYTECODE", "PYTHONUNBUFFERED",
}
PLAN_FIELDS = ("attempt_uuid", "job_name", "remote_directory", "container_root", "image", "salty",
"cpu", "memory", "command", "env", "payloads")
class TransportError(RuntimeError):
"""No automatic retry or destructive action was performed."""
class AmbiguousOperation(TransportError):
"""The exact operation may have taken effect; retain and reconcile it."""
class FormatError(TransportError):
"""The audited CLI format or exact identity did not match."""
@dataclass(frozen=True)
class CommandResult:
exit_code: int | None
pid: int | None = None
timed_out: bool = False
process_left_running: bool = False
birth_identity: dict | None = None
Runner = Callable[[Sequence[str], Path, Path, float], CommandResult]
def _cli_birth_identity(pid: int) -> dict:
"""Read the owned Linux child's birth metadata; never inspect argv/env."""
identity = {"pid": pid, "status": "UNVERIFIED"}
try:
raw = Path("/proc/%d/stat" % pid).read_text()
fields = raw[raw.rfind(")") + 2:].split()
if int(fields[1]) != os.getpid():
return identity
identity.update(status="KNOWN", start_ticks=int(fields[19]),
boot_id=Path("/proc/sys/kernel/random/boot_id").read_text().strip(),
ppid=int(fields[1]))
except (OSError, ValueError, IndexError):
pass
return identity
class _CliProcessUnknown(RuntimeError):
def __init__(self, result: CommandResult):
self.process_result = result
super().__init__("owned CLI process outcome unknown")
class CliRunner:
"""File-backed subprocess output; a deadline never terminates a process."""
def __init__(self) -> None:
self._unresolved: dict[int, subprocess.Popen] = {}
def __call__(self, argv: Sequence[str], stdout: Path, stderr: Path,
timeout: float, *, on_started=None) -> CommandResult:
# Do not inspect or serialize the inherited environment. Salty manages
# its own existing login. No env-file or credential options are used.
with stdout.open("xb") as out, stderr.open("xb") as err:
os.chmod(stdout, 0o600)
os.chmod(stderr, 0o600)
process = subprocess.Popen(list(argv), stdin=subprocess.DEVNULL,
stdout=out, stderr=err)
# Retain the handle before any callback or interrupt can fail.
self._unresolved[process.pid] = process
identity = _cli_birth_identity(process.pid)
try:
if on_started is not None:
on_started(process.pid, identity)
started = time.monotonic()
while process.poll() is None:
if time.monotonic() - started >= timeout:
return CommandResult(None, process.pid, True, True, identity)
time.sleep(min(0.1, timeout))
self._unresolved.pop(process.pid, None)
return CommandResult(process.returncode, process.pid, birth_identity=identity)
except BaseException as error:
raise _CliProcessUnknown(CommandResult(None, process.pid, False, True, identity)) from error
def _sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def _atomic_json(path: Path, value: object) -> None:
temporary = path.with_name(path.name + "." + uuid.uuid4().hex + ".tmp")
try:
with temporary.open("x", encoding="utf-8") as stream:
os.chmod(temporary, 0o600)
json.dump(value, stream, indent=2, sort_keys=True, ensure_ascii=False)
stream.write("\n")
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary, path)
descriptor = os.open(path.parent, os.O_RDONLY)
try:
os.fsync(descriptor)
finally:
os.close(descriptor)
finally:
if temporary.exists():
temporary.unlink()
def validate_image(image: str) -> str:
"""Require an explicit registry/repository and a SHA256 digest, no tag."""
if not isinstance(image, str) or "@sha256:" not in image:
raise ValueError("Salty image requires registry/repository@sha256:<64 lowercase hex>")
repository, digest = image.rsplit("@sha256:", 1)
if not re.fullmatch(r"[0-9a-f]{64}", digest) or "/" not in repository:
raise ValueError("invalid immutable image reference")
registry, name = repository.split("/", 1)
if not re.fullmatch(r"[a-z0-9][a-z0-9.-]*(?::[0-9]{1,5})?", registry):
raise ValueError("invalid explicit image registry")
if "." not in registry and ":" not in registry and registry != "localhost":
raise ValueError("image registry must be explicit")
if not re.fullmatch(r"[a-z0-9]+(?:[._-][a-z0-9]+)*(?:/[a-z0-9]+(?:[._-][a-z0-9]+)*)*", name):
raise ValueError("image repository must not contain tags, credentials or whitespace")
return image
def _safe_env(env: Mapping[str, str] | None) -> dict[str, str]:
result = {}
for key, value in (env or {}).items():
if key not in SAFE_ENV or not isinstance(value, str):
raise ValueError("only explicit non-sensitive numeric worker environment settings are allowed")
if not re.fullmatch(r"[0-9]{1,3}", value):
raise ValueError("worker environment values must be literal numbers")
if key in {"PYTHONDONTWRITEBYTECODE", "PYTHONUNBUFFERED"}:
valid = value in {"0", "1"}
else:
valid = 1 <= int(value) <= 256
if not valid:
raise ValueError("worker environment value outside allowed range")
result[key] = value
return result
def _plan_hash(receipt: dict) -> str:
raw = json.dumps({key: receipt[key] for key in PLAN_FIELDS}, sort_keys=True,
separators=(",", ":"), ensure_ascii=False).encode("utf-8")
return hashlib.sha256(raw).hexdigest()
def parse_describe(output: str, job_id: str) -> str:
"""Salty 0.2's evidenced Chinese text contract; drift fails closed."""
identities = re.findall(r"^ID:\s*(" + JOB_PATTERN + r")\s*$", output, re.MULTILINE)
states = re.findall(r"^状态:\s*([a-z]+)\s*$", output, re.MULTILINE)
if identities != [job_id] or len(states) != 1:
raise FormatError("describe format or precise job identity mismatch")
if states[0] not in {"pending", "running", *TERMINAL}:
raise FormatError("unknown Salty job state")
return states[0]
def _parse_submission(output: str) -> str:
identities = set(re.findall(r"(?<![\w-])" + JOB_PATTERN + r"(?![\w-])", output))
if len(identities) != 1:
raise FormatError("submission did not return one unique full Salty job ID")
return identities.pop()
class SaltyClient:
def __init__(self, receipt_dir: Path | str, *, runner: Runner | None = None,
timeout_seconds: float = 180) -> None:
if not math.isfinite(timeout_seconds) or timeout_seconds <= 0:
raise ValueError("CLI deadline must be positive")
raw = Path(receipt_dir).absolute()
if raw.is_symlink() or not raw.is_dir():
raise ValueError("receipt directory must be an existing regular directory")
self.directory = raw.resolve()
self.receipt_path = self.directory / "receipt.json"
self.runner = runner or CliRunner()
self.timeout_seconds = timeout_seconds
self._read()
@classmethod
def prepare(cls, receipt_dir: Path | str, *, image: str, bundle: Path | str,
worker: Path | str, command: Sequence[str],
env: Mapping[str, str] | None = None, cpu: int | None = None,
memory: str | None = None, salty: str = "salty",
runner: Runner | None = None, timeout_seconds: float = 180) -> "SaltyClient":
image = validate_image(image)
safe_env = _safe_env(env)
if cpu is not None and (isinstance(cpu, bool) or not isinstance(cpu, int) or not 1 <= cpu <= 64):
raise ValueError("CPU must be an explicit integer from 1 to 64")
if memory is not None and (not isinstance(memory, str) or not re.fullmatch(r"[1-9][0-9]{0,5}(?:Mi|Gi)", memory)):
raise ValueError("memory must be an explicit Mi/Gi quantity")
if not isinstance(salty, str) or not salty or "\x00" in salty:
raise ValueError("Salty executable must be a literal path/name")
if isinstance(command, (str, bytes)) or not command:
raise ValueError("worker command must be a nonempty argv list")
if any(not isinstance(token, str) or not token or "\x00" in token for token in command):
raise ValueError("worker command argv contains an invalid literal")
sources = {"bundle.tar": Path(bundle).absolute(), "worker.py": Path(worker).absolute()}
for source in sources.values():
if source.is_symlink() or not source.is_file() or source.name.startswith(".env"):
raise ValueError("payload sources must be caller-provided regular, non-credential files")
if not math.isfinite(timeout_seconds) or timeout_seconds <= 0:
raise ValueError("CLI deadline must be positive")
directory = Path(receipt_dir).absolute()
# Caller creates the parent. Never reuse an existing attempt directory.
directory.mkdir(mode=0o700, parents=False, exist_ok=False)
directory = directory.resolve()
identifier = str(uuid.uuid4())
name = "bench-stage-" + identifier
container_root = "/data/personal/" + name
payload = directory / "payload"
payload.mkdir(mode=0o700)
snapshots = {}
try:
for filename, source in sources.items():
before = _sha256(source)
destination = payload / filename
with source.open("rb") as stream, destination.open("xb") as target:
shutil.copyfileobj(stream, target)
target.flush()
os.fsync(target.fileno())
os.chmod(destination, 0o600)
after = _sha256(destination)
if before != after or _sha256(source) != before:
raise ValueError("payload changed during immutable snapshot creation")
snapshots[filename] = {"sha256": after, "bytes": destination.stat().st_size}
receipt = {
"schema": SCHEMA, "attempt_uuid": identifier, "job_name": name,
"remote_directory": "personal:/" + name, "container_root": container_root,
"image": image, "salty": salty, "cpu": cpu, "memory": memory,
"command": [token.replace("{container_root}", container_root) for token in command],
"env": safe_env, "payloads": snapshots, "phase": "PREPARED",
"submission_state": "NOT_SENT", "job_id": None, "job_state": None,
"commands": [], "downloads": [], "cleanup": {}, "created_at": time.time(),
}
receipt["plan_sha256"] = _plan_hash(receipt)
_atomic_json(directory / "receipt.json", receipt)
except BaseException:
# No remote action exists yet. Remove only this exclusive directory.
shutil.rmtree(directory)
raise
return cls(directory, runner=runner, timeout_seconds=timeout_seconds)
def _read(self) -> dict:
if self.receipt_path.is_symlink() or not self.receipt_path.is_file():
raise ValueError("transport receipt must be a regular file")
receipt = json.loads(self.receipt_path.read_text(encoding="utf-8"))
if receipt.get("schema") != SCHEMA:
raise ValueError("unknown Salty transport receipt schema")
identifier = receipt.get("attempt_uuid")
try:
valid_uuid = isinstance(identifier, str) and str(uuid.UUID(identifier)) == identifier
except ValueError:
valid_uuid = False
if not valid_uuid:
raise ValueError("invalid transport UUID")
name = "bench-stage-" + identifier
if (receipt.get("job_name") != name or receipt.get("remote_directory") != "personal:/" + name
or receipt.get("container_root") != "/data/personal/" + name):
raise ValueError("receipt remote path/ownership guard mismatch")
job_id = receipt.get("job_id")
if job_id is not None and (not isinstance(job_id, str) or not re.fullmatch(JOB_PATTERN, job_id)):
raise ValueError("receipt requires a complete precise Salty job ID")
validate_image(receipt["image"])
_safe_env(receipt.get("env"))
if _plan_hash(receipt) != receipt.get("plan_sha256"):
raise ValueError("immutable transport plan changed on receipt recovery")
payloads = receipt.get("payloads")
if not isinstance(payloads, dict) or set(payloads) != {"bundle.tar", "worker.py"}:
raise ValueError("receipt payload identity guard mismatch")
for filename, metadata in payloads.items():
if (not isinstance(metadata, dict) or not re.fullmatch(r"[0-9a-f]{64}", metadata.get("sha256", ""))
or not isinstance(metadata.get("bytes"), int) or metadata["bytes"] < 0):
raise ValueError("invalid immutable payload receipt")
# After authorized local discard (including an unreconciled mkdir),
# no future job/upload can use these snapshots. A completed remote
# cleanup may also have partly removed them after a crash.
if not (receipt.get("cleanup", {}).get("storage_removed") or
receipt.get("cleanup", {}).get("local_payload_removal_started")):
payload_root = self.directory / "payload"
path = payload_root / filename
if (payload_root.is_symlink() or path.is_symlink() or not path.is_file()
or path.stat().st_size != metadata["bytes"] or _sha256(path) != metadata["sha256"]):
raise ValueError("immutable payload/source hash differs on receipt recovery")
if job_id is not None and not receipt.get("cleanup", {}).get("job_removed"):
submissions = [entry for entry in receipt["commands"] if entry["label"] == "submit"]
if len(submissions) != 1:
raise ValueError("known job must retain its unique successful submission receipt")
entry = submissions[0]
filename = entry.get("stdout")
if not isinstance(filename, str) or Path(filename).name != filename:
raise ValueError("submission recovery output path guard failed")
output = self.directory / filename
if (entry.get("state") != "COMPLETE" or entry.get("exit_code") != 0
or output.is_symlink() or not output.is_file()
or _sha256(output) != entry.get("stdout_sha256")
or _parse_submission(output.read_text(encoding="utf-8")) != job_id):
raise ValueError("known job ID differs from completed exact submission evidence")
return receipt
@contextmanager
def _lock(self):
lock_path = self.directory / "receipt.lock"
if lock_path.is_symlink():
raise ValueError("receipt lock cannot be a symlink")
descriptor = os.open(lock_path, os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600)
try:
try:
fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError as exc:
raise TransportError("another caller owns this transport receipt") from exc
yield self._read()
finally:
fcntl.flock(descriptor, fcntl.LOCK_UN)
os.close(descriptor)
def _save(self, receipt: dict) -> None:
receipt["updated_at"] = time.time()
_atomic_json(self.receipt_path, receipt)
@property
def receipt(self) -> dict:
with self._lock() as receipt:
return receipt
@property
def container_root(self) -> str:
return self.receipt["container_root"]
def _execute(self, receipt: dict, arguments: Sequence[str], label: str) -> str:
index = len(receipt["commands"])
stdout = self.directory / ("%03d-%s.stdout.log" % (index, label))
stderr = self.directory / ("%03d-%s.stderr.log" % (index, label))
entry = {"label": label, "argv": [receipt["salty"], *arguments],
"stdout": stdout.name, "stderr": stderr.name,
"state": "STARTED", "started_at": time.time()}
receipt["commands"].append(entry)
# Persist before starting every CLI operation, especially run/rm/cp.
self._save(receipt)
try:
if isinstance(self.runner, CliRunner):
def started(pid, identity):
entry.update(local_pid=pid, local_birth_identity=identity,
process_left_running=True)
self._save(receipt)
result = self.runner(entry["argv"], stdout, stderr, self.timeout_seconds,
on_started=started)
else:
result = self.runner(entry["argv"], stdout, stderr, self.timeout_seconds)
except BaseException as exc:
entry["state"] = "UNKNOWN"
entry["exception_type"] = type(exc).__name__
process = getattr(exc, "process_result", None)
if isinstance(process, CommandResult):
entry.update(local_pid=process.pid, local_birth_identity=process.birth_identity,
process_left_running=True)
self._save(receipt)
raise AmbiguousOperation("CLI operation outcome unknown; no automatic retry: " + label) from exc
entry.update(exit_code=result.exit_code, local_pid=result.pid, timed_out=result.timed_out,
process_left_running=result.process_left_running, ended_at=time.time())
if result.birth_identity is not None:
entry["local_birth_identity"] = result.birth_identity
for name, path in (("stdout", stdout), ("stderr", stderr)):
if path.is_file() and not path.is_symlink() and not result.process_left_running:
entry[name + "_sha256"] = _sha256(path)
if result.timed_out or result.process_left_running or result.exit_code is None:
entry["state"] = "UNKNOWN"
self._save(receipt)
raise AmbiguousOperation("CLI deadline/unknown outcome; process left untouched: " + label)
entry["state"] = "COMPLETE" if result.exit_code == 0 else "FAILED"
self._save(receipt)
if result.exit_code != 0:
raise TransportError("CLI %s failed (%s); retained receipt, no retry" % (label, result.exit_code))
if stdout.is_symlink() or not stdout.is_file():
raise FormatError("CLI runner did not produce regular output")
return stdout.read_text(encoding="utf-8", errors="replace")
def submit(self) -> dict:
with self._lock() as receipt:
if receipt["job_id"] is not None:
return receipt # Recovery uses its precise existing identity.
if receipt["cleanup"]:
raise TransportError("cleanup already started; payload disposal cannot be followed by submission")
if receipt["phase"] != "PREPARED" or receipt["submission_state"] != "NOT_SENT":
raise AmbiguousOperation("attempt already started; never resubmit an unknown or partial attempt")
receipt["phase"] = "PREPARING"
self._save(receipt)
try:
version = self._execute(receipt, ["version"], "version").strip()
if version != "salty 0.2.0":
raise FormatError("expected audited Salty 0.2.0 CLI")
remote = receipt["remote_directory"]
self._execute(receipt, ["storage", "mkdir", remote], "mkdir-exclusive")
for filename, metadata in receipt["payloads"].items():
path = self.directory / "payload" / filename
if path.is_symlink() or not path.is_file() or _sha256(path) != metadata["sha256"]:
raise ValueError("immutable local payload mismatch")
self._execute(receipt, ["storage", "cp", str(path), remote + "/" + filename],
"upload-" + filename.replace(".", "-"))
receipt["phase"] = "SUBMITTING"
receipt["submission_state"] = "UNKNOWN"
self._save(receipt)
arguments = ["run", receipt["image"], "--name", receipt["job_name"]]
if receipt["cpu"] is not None:
arguments.extend(["--cpu", str(receipt["cpu"])])
if receipt["memory"] is not None:
arguments.extend(["--memory", receipt["memory"]])
for key, value in sorted(receipt["env"].items()):
arguments.extend(["-e", key + "=" + value])
submitted = self._execute(receipt, [*arguments, "--", *receipt["command"]], "submit")
receipt["job_id"] = _parse_submission(submitted)
receipt["submission_state"] = "KNOWN"
receipt["phase"] = "SUBMITTED"
self._save(receipt)
return receipt
except BaseException:
receipt["phase"] = ("SUBMISSION_UNKNOWN" if receipt["submission_state"] == "UNKNOWN"
else "PREPARATION_FAILED")
self._save(receipt)
raise
def recover_recorded_submission(self) -> dict:
"""Recover a crash after recorded successful run, never discover by name.
Only the persisted completed command's exact output may supply the ID.
An unresolved process, nonzero exit, missing/changed output or format
drift needs operator reconciliation; no job lookup or submit follows.
"""
with self._lock() as receipt:
if receipt["job_id"] is not None:
return receipt
submissions = [entry for entry in receipt["commands"] if entry["label"] == "submit"]
if len(submissions) != 1:
raise AmbiguousOperation("no unique recorded submission to recover")
entry = submissions[0]
if entry.get("state") != "COMPLETE" or entry.get("exit_code") != 0:
raise AmbiguousOperation("submission exit remains unknown or failed")
output = self.directory / entry["stdout"]
if (output.is_symlink() or not output.is_file()
or _sha256(output) != entry.get("stdout_sha256")):
raise FormatError("recorded submission output is missing or changed")
receipt["job_id"] = _parse_submission(output.read_text(encoding="utf-8"))
receipt["submission_state"] = "KNOWN"
receipt["phase"] = "SUBMITTED"
self._save(receipt)
return receipt
def _describe(self, receipt: dict) -> str:
job_id = receipt["job_id"]
if job_id is None:
raise AmbiguousOperation("no known precise job ID; cannot poll or clean up")
output = self._execute(receipt, ["describe", job_id], "describe")
state = parse_describe(output, job_id)
previous = receipt.get("job_state")
if previous in TERMINAL and state != previous:
raise FormatError("terminal job state changed unexpectedly")
receipt["job_state"] = state
receipt["phase"] = "TERMINAL" if state in TERMINAL else "SUBMITTED"
self._save(receipt)
return state
def poll(self) -> dict:
with self._lock() as receipt:
if receipt["phase"] == "CLEANED":
return receipt
if receipt["cleanup"].get("job_removed"):
raise TransportError("job already deleted during partial cleanup; resume cleanup explicitly")
self._describe(receipt)
return receipt
def fetch(self, remote_relative: str, local_destination: Path | str) -> dict:
"""Download one new file from this attempt after exact terminal status.
The CLI writes only a unique file inside this receipt. A regular file
and its hash are recorded before an atomic, no-replace hard link
publishes it. Cross-filesystem destinations are explicitly rejected;
there is no copy/replace fallback. Caller verifies the returned bytes.
Any partial/unknown transfer remains quarantined for reconciliation.
"""
if not isinstance(remote_relative, str) or not re.fullmatch(r"[A-Za-z0-9_./-]+", remote_relative):
raise ValueError("download requires a canonical result file inside this attempt")
relative = PurePosixPath(remote_relative)
if (not remote_relative or relative.is_absolute()
or ".." in relative.parts or relative.as_posix() != remote_relative
or remote_relative in {".", "bundle.tar", "worker.py"}):
raise ValueError("download requires a canonical result file inside this attempt")
destination = Path(local_destination).absolute()
if destination.exists() or destination.is_symlink() or not destination.parent.is_dir():
raise ValueError("download destination must be a new file under an existing directory")
destination = destination.parent.resolve() / destination.name
with self._lock() as receipt:
if receipt["cleanup"] or receipt["phase"] == "CLEANED":
raise TransportError("cannot fetch after cleanup has started")
if any(item["remote_relative"] == remote_relative or item["destination"] == str(destination)
for item in receipt["downloads"]):
raise AmbiguousOperation("download already attempted; inspect exact destination, never overwrite")
state = self._describe(receipt)
if state not in TERMINAL:
raise TransportError("job is active; cannot fetch final results")
temporary = self.directory / ("download-%03d-%s.part" %
(len(receipt["downloads"]), uuid.uuid4().hex))
download = {"remote_relative": remote_relative, "destination": str(destination),
"temporary": temporary.name, "state": "STARTED"}
receipt["downloads"].append(download)
self._save(receipt)
try:
self._execute(receipt, ["storage", "cp", receipt["remote_directory"] + "/" + remote_relative,
str(temporary)], "download")
if temporary.is_symlink() or not temporary.is_file():
raise FormatError("download did not produce a regular result file")
download.update(state="DOWNLOADED", sha256=_sha256(temporary),
bytes=temporary.stat().st_size)
self._save(receipt)
try:
# The kernel arbitrates different receipts publishing the
# same name. EEXIST must never fall back to an overwrite.
os.link(temporary, destination, follow_symlinks=False)
except OSError as exc:
download.update(state="PUBLISH_FAILED", publication_errno=exc.errno)
self._save(receipt)
temporary.unlink()
download["temporary_removed"] = True
self._save(receipt)
reason = ("destination already exists; preserved its bytes" if exc.errno == errno.EEXIST else
"destination crosses filesystems; no overwrite/copy fallback" if exc.errno == errno.EXDEV else
"atomic no-replace publication failed")
raise TransportError(reason) from exc
download["state"] = "COMPLETE"
self._save(receipt)
temporary.unlink()
download["temporary_removed"] = True
self._save(receipt)
return dict(download)
except BaseException:
if download["state"] not in {"COMPLETE", "PUBLISH_FAILED"}:
download["state"] = "UNKNOWN"
self._save(receipt)
raise
def _remove_local_payload(self, receipt: dict) -> None:
cleanup = receipt["cleanup"]
if cleanup.get("local_payload_removed"):
return
# A crash partway through unlinking must still allow receipt recovery.
cleanup["local_payload_removal_started"] = True
self._save(receipt)
payload = self.directory / "payload"
if payload.is_symlink():
raise ValueError("payload cleanup refuses symlink")
if payload.exists():
for filename in ("bundle.tar", "worker.py"):
path = payload / filename
if path.exists() or path.is_symlink():
if path.is_symlink() or not path.is_file():
raise ValueError("payload cleanup requires regular owned snapshots")
path.unlink()
payload.rmdir()
cleanup["local_payload_removed"] = True
self._save(receipt)
def cleanup(self, *, authorized: bool = False) -> dict:
"""Remove only this known terminal job and UUID directory, once authorized.
Never-submitted preparation can also be discarded: no submit record,
no unresolved CLI process, and remote removal only after a verified
successful exclusive mkdir. A failed/unknown mkdir never grants
ownership or proves absence: retain an unreconciled diagnostic receipt.
The caller preserves verified stage outputs before requesting cleanup.
A partially completed/unknown destructive CLI operation is never blindly
repeated; completed recorded steps can safely continue after restart.
Receipt plus hashes remain; payload snapshots and finished CLI logs go.
"""
if authorized is not True:
raise TransportError("cleanup requires explicit caller authorization for this attempt")
with self._lock() as receipt:
if receipt["phase"] == "CLEANED":
return receipt
if any(item.get("state") not in {"COMPLETE", "PUBLISH_FAILED"} for item in receipt["downloads"]):
raise AmbiguousOperation("unresolved download must be reconciled before cleanup")
cleanup = receipt["cleanup"]
if receipt["job_id"] is None:
if receipt["submission_state"] != "NOT_SENT" or any(
entry.get("label") == "submit" or entry.get("argv", [None, None])[1:2] == ["run"]
for entry in receipt["commands"]):
raise AmbiguousOperation("submission may exist; no cleanup without its precise known job")
mkdir_entries = [entry for entry in receipt["commands"] if entry.get("label") == "mkdir-exclusive"]
if len(mkdir_entries) > 1:
raise ValueError("preparation has multiple exclusive-directory operations")
if mkdir_entries and mkdir_entries[0].get("state") != "COMPLETE":
entry = mkdir_entries[0]
expected = [receipt["salty"], "storage", "mkdir", receipt["remote_directory"]]
if entry.get("argv") != expected:
raise ValueError("exclusive directory ownership receipt mismatch")
# Even a nonzero exit may follow successful server creation.
# Never delete a directory whose ownership is unconfirmed.
cleanup.update(authorized=True, job_absent=True, job_removed=False,
never_submitted=True, remote_owned=False,
storage_removed=False, remote_cleanup_unknown=True)
cleanup.pop("storage_not_created_by_attempt", None)
receipt["phase"] = "UNRECONCILED"
self._save(receipt)
# mkdir does not read payloads. Retain its logs (including
# a still-running writer), while discarding unused snapshots.
if not any(other is not entry and (other.get("state") in {"STARTED", "UNKNOWN"}
or other.get("process_left_running"))
for other in receipt["commands"]):
self._remove_local_payload(receipt)
for other in receipt["commands"]:
if other is entry:
continue
for key in ("stdout", "stderr"):
filename = other[key]
if Path(filename).name != filename:
raise ValueError("operation log cleanup path guard failed")
path = self.directory / filename
if path.is_symlink():
raise ValueError("log cleanup refuses symlink")
if path.is_file():
path.unlink()
other["logs_removed"] = True
self._save(receipt)
raise AmbiguousOperation("remote mkdir outcome/ownership is unreconciled; diagnostic receipt retained")
# Includes version: an unresolved local process can still write
# its log. A timed-out/unknown mutation can continue after us.
if any(entry.get("state") in {"STARTED", "UNKNOWN"} or entry.get("process_left_running")
for entry in receipt["commands"]):
raise AmbiguousOperation("unresolved preparation CLI process; retain exact owned files")
remote_owned = bool(cleanup.get("remote_owned"))
already_removed = cleanup.get("never_submitted") and cleanup.get("storage_removed")
if not already_removed and mkdir_entries and mkdir_entries[0].get("state") == "COMPLETE":
entry = mkdir_entries[0]
expected = [receipt["salty"], "storage", "mkdir", receipt["remote_directory"]]
if entry.get("argv") != expected or entry.get("exit_code") != 0:
raise ValueError("exclusive directory ownership receipt mismatch")
for key in ("stdout", "stderr"):
filename = entry.get(key)
if not isinstance(filename, str) or Path(filename).name != filename:
raise ValueError("exclusive-directory output path guard failed")
output = self.directory / filename
if (output.is_symlink() or not output.is_file()
or _sha256(output) != entry.get(key + "_sha256")):
raise FormatError("exclusive directory ownership evidence missing or changed")
remote_owned = True
cleanup.update(authorized=True, job_absent=True, job_removed=False,
never_submitted=True, remote_owned=remote_owned)
if not remote_owned:
cleanup["storage_removed"] = True
cleanup["storage_not_created_by_attempt"] = True
cleanup["remote_cleanup_unknown"] = False
self._save(receipt)
if cleanup.get("operation_state") == "UNKNOWN":
# A crash after _execute persisted rc=0 but before the step flag
# need not repeat rm. Verify the completed precise command/logs
# and finish only its local receipt bookkeeping.
label = cleanup.get("operation")
expected = ([receipt["salty"], "rm", receipt["job_id"]]
if label == "remove-terminal-job" else
[receipt["salty"], "storage", "rm", "-r", receipt["remote_directory"]]
if label == "remove-owned-directory" else None)
entry = receipt["commands"][-1] if receipt["commands"] else {}
if (expected is None or entry.get("label") != label or entry.get("argv") != expected
or entry.get("state") != "COMPLETE" or entry.get("exit_code") != 0):
raise AmbiguousOperation("cleanup operation outcome unknown; never repeat blindly")
for key in ("stdout", "stderr"):
filename = entry.get(key)
if not isinstance(filename, str) or Path(filename).name != filename:
raise ValueError("cleanup recovery log path guard failed")
path = self.directory / filename
if (path.is_symlink() or not path.is_file()
or _sha256(path) != entry.get(key + "_sha256")):
raise FormatError("completed cleanup command evidence missing or changed")
cleanup["job_removed" if label == "remove-terminal-job" else "storage_removed"] = True
cleanup["operation_state"] = "COMPLETE"
self._save(receipt)
# A prior STARTED operation can still own the directory or write a
# log even after a new describe confirms job completion. Refuse all
# remote deletion before touching those materials, not afterward.
if receipt["job_id"] is not None and any(
entry.get("state") in {"STARTED", "UNKNOWN"}
or entry.get("process_left_running") for entry in receipt["commands"]):
raise AmbiguousOperation("unresolved CLI operation; retain job, directory and logs")
if not (cleanup.get("job_removed") or cleanup.get("job_absent")):
state = self._describe(receipt)
if state not in TERMINAL:
raise TransportError("pending/running job cannot be stopped or deleted")
# Persist unknown before rm. A crash cannot replay deletion.
cleanup.update(operation_state="UNKNOWN", operation="remove-terminal-job",
authorized=True, terminal_state=state)
self._save(receipt)
self._execute(receipt, ["rm", receipt["job_id"]], "remove-terminal-job")
cleanup.update(operation_state="COMPLETE", job_removed=True)
self._save(receipt)
if not cleanup.get("storage_removed"):
cleanup.update(operation_state="UNKNOWN", operation="remove-owned-directory")
self._save(receipt)
self._execute(receipt, ["storage", "rm", "-r", receipt["remote_directory"]],
"remove-owned-directory")
cleanup.update(operation_state="COMPLETE", storage_removed=True)
self._save(receipt)
unresolved = [entry for entry in receipt["commands"]
if entry.get("process_left_running") or entry.get("state") in {"STARTED", "UNKNOWN"}]
if unresolved:
raise AmbiguousOperation("unresolved local CLI process/logs retained; cleanup not complete")
# These paths are derived internally. No caller-owned result file is
# removed, and never recurse through symlinks in the receipt tree.
self._remove_local_payload(receipt)
for download in receipt["downloads"]:
if download.get("temporary_removed") or "temporary" not in download:
continue
filename = download["temporary"]
if not isinstance(filename, str) or not re.fullmatch(r"download-[0-9]{3,}-[0-9a-f]{32}\.part", filename):
raise ValueError("download temporary cleanup path guard failed")
path = self.directory / filename
if path.is_symlink() or (path.exists() and not path.is_file()):
raise ValueError("download temporary cleanup requires regular owned file")
if path.exists():
path.unlink()
download["temporary_removed"] = True
self._save(receipt)
for entry in receipt["commands"]:
for key in ("stdout", "stderr"):
filename = entry[key]
if Path(filename).name != filename:
raise ValueError("operation log cleanup path guard failed")
path = self.directory / filename
if path.is_symlink():
raise ValueError("log cleanup refuses symlink")
if path.is_file():
path.unlink()
entry["logs_removed"] = True
receipt["phase"] = "CLEANED"
cleanup.update(local_temporary_removed=True, completed_at=time.time())
self._save(receipt)
return receipt

View File

@ -1,254 +0,0 @@
"""`bench dev`: one agent command in a Salty container that has the toolchain (hb_compile, hmct,
onnxruntime, torch). The agent's bash runs on the controller VM, which has none of it.
The workspace lives twice: the controller copy the agent edits, and the development copy on personal
storage (the same absolute path inside containers, so 04/05 read the graph and calibration made here).
Before a command, files the agent changed locally since the last sync go up; afterwards, files the
command changed come back if small, and big ones (graphs, calibration sets) stay remote and are listed.
The worker sandboxes the command (salty_remote_worker.dev): only toolchains, caches and data read-only
plus this workspace are visible, no capabilities, no board credentials.
"""
from __future__ import annotations
import fcntl
import hashlib
import io
import json
import os
import re
import shutil
import tarfile
import time
import uuid
from contextlib import contextmanager
from pathlib import Path, PurePosixPath
from types import SimpleNamespace
from . import paths, salty_remote
from .salty_client import TransportError
from .watchdog import RunResult
EXCLUDED = {'.pi', '.pi-sessions', '.agent-home', '.bench-session.json', '.env', '.env.local', 'guard.log'}
UPLOAD_LIMIT = 256 << 20 # the Mac mini uplink is ~1 MB/s; big inputs are made inside bench dev
GRACE_S = 300
FETCH_MARGIN_S = 120 # the command ends this long before the request deadline so its results still come back
def selected(cfg=None) -> bool:
policy = ((cfg or paths.config()).get('execution') or {}).get('salty_remote') or {}
return policy.get('enabled') is True and 'terminal' in policy.get('stages', [])
def local_files(ws: Path) -> dict:
files = {}
for root, dirs, names in os.walk(ws):
top = Path(root) == ws
dirs[:] = [d for d in dirs if not (top and d in EXCLUDED) and not os.path.islink(os.path.join(root, d))]
for name in names:
p = Path(root) / name
if (top and name in EXCLUDED) or p.is_symlink() or not p.is_file():
continue
st = p.stat()
files[p.relative_to(ws).as_posix()] = [st.st_size, st.st_mtime_ns]
return files
@contextmanager
def synced(rd: Path):
"""{relative path: [size, mtime_ns]} of the controller copy as last synced; changes since are uploads.
Held only around reads/updates, so dev commands of one rollout still run in parallel."""
path = rd / 'stages' / '_dev-sync.json'
path.parent.mkdir(parents=True, exist_ok=True)
with (rd / 'stages' / '_dev-sync.lock').open('a') as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
state = json.loads(path.read_text()) if path.exists() else {}
yield state
tmp = path.with_suffix('.tmp')
tmp.write_text(json.dumps(state))
os.replace(tmp, path)
def bundle(path: Path, ws: Path, uploads: list, request: dict) -> None:
with tarfile.open(path, 'x') as tar:
data = json.dumps(request).encode()
info = tarfile.TarInfo('request.json'); info.size = len(data)
tar.addfile(info, io.BytesIO(data))
for rel in uploads:
tar.add(ws / rel, 'ws/' + rel, recursive=False)
for name in salty_remote.CODE_DIRS: # harness tools (export_onnx, adapters) for the agent to reuse
if (paths.BENCH_ROOT / name).is_dir():
tar.add(paths.BENCH_ROOT / name, f'code/{name}',
filter=lambda m: None if '__pycache__' in m.name or m.name.endswith('.pyc') else m)
def install(ws: Path, archive: Path, result: dict) -> list:
"""Changed small files back into the controller copy; only regular files under the workspace."""
data = archive.read_bytes()
if hashlib.sha256(data).hexdigest() != result['archive']['sha256']:
raise ValueError('dev output archive bytes differ from its receipt')
installed = []
with tarfile.open(fileobj=io.BytesIO(data)) as tar:
for member in tar.getmembers():
rel = PurePosixPath(member.name)
if not member.isfile() or rel.is_absolute() or '..' in rel.parts or rel.parts[0] in EXCLUDED:
raise ValueError('unsafe dev output member: ' + member.name)
body = tar.extractfile(member).read()
if hashlib.sha256(body).hexdigest() != result['outputs'][member.name]['sha256']:
raise ValueError('dev output bytes fail verification: ' + member.name)
target = ws / rel
if any(p.is_symlink() for p in [target, *target.parents] if p.is_relative_to(ws) and p != ws):
raise ValueError('dev output would follow a workspace symlink: ' + member.name)
target.parent.mkdir(parents=True, exist_ok=True)
tmp = target.with_name(target.name + '.dev-tmp')
tmp.write_bytes(body)
os.replace(tmp, target)
installed.append(member.name)
return installed
def _storage(rd: Path, rel: str):
policy = salty_remote._policy(SimpleNamespace(cfg=paths.config()))
cli = salty_remote._Cli(policy.get('salty', 'salty'), policy.get('outage_seconds', 1800), time.sleep, time.time, lambda m: None)
remote = 'personal:/bench-root/runs/' + (rd / 'workspace').relative_to(paths.BENCH_ROOT / 'runs').as_posix()
if rel not in ('', '.'):
remote += '/' + PurePosixPath(rel).as_posix()
return cli, remote
def remote_size(rd: Path, rel: str, cap: int, max_dirs: int = 40) -> int | None:
"""Bytes under the development workspace's <rel>, or None once past cap bytes or max_dirs directories
(one `salty storage ls` per directory, ~3 s each)."""
cli, root = _storage(rd, rel)
todo, total = [root], 0
while todo:
if max_dirs <= 0:
return None
max_dirs -= 1
here = todo.pop()
_, out, _ = cli.call('storage', 'ls', here, ok_404=True)
for line in (out or '').splitlines()[1:]: # header: 名称 类型 大小 修改时间
name, kind, size = (line.split('\t') + ['', '', ''])[:3]
if kind == '目录':
todo.append(here + '/' + name)
elif size.isdigit():
total += int(size)
if total > cap:
return None
return total
def download_dir(rd: Path, rel: str, destination: Path) -> None:
"""The development workspace's copy of <rel> (files over the copy-back limit only exist there); a directory
never synced up gives an empty destination."""
cli, remote = _storage(rd, rel)
part = destination.with_name(destination.name + '.part')
def fresh():
shutil.rmtree(part, ignore_errors=True); part.mkdir(parents=True)
rc, _, err = cli.call('storage', 'cp', remote, str(part), timeout=1800, ok_404=True, before=fresh)
got = part / PurePosixPath(remote).name # `salty storage cp <dir> <existing dir>` lands in <existing dir>/<name>
if rc == 0 and got.is_dir():
os.replace(got, destination)
else:
destination.mkdir(parents=True)
shutil.rmtree(part)
def calib_images(rd: Path, job: dict) -> str:
"""The job's calibration images for agent v18+ (calib-raw mirrors calib; 10-15 night: told only to
"replace /data/calib/", agents searched /data/calib-raw and spent 5-8 commands finding it). From v21 the
original training images (images/, from 36's JuiceFS); v18-v20 rollouts keep the resized png/ they started with."""
version = int(json.loads((rd / 'rollout.json').read_text()).get('agent_config_version', 0))
return job['data']['calib']['path'].replace('/data/calib/', '/data/calib-raw/', 1) + ('/images' if version >= 21 else '/png')
def execute(ctx, hd: Path, req: dict, deadline: float, stdout: Path, stderr: Path) -> RunResult:
"""Run req['command'] remotely; returns a watchdog-shaped result for terminal.run."""
started = time.time()
rd = ctx.stage_root.parent
ws = rd / 'workspace'
adapter = hd / 'salty'
adapter.mkdir(exist_ok=True)
policy = salty_remote._policy(SimpleNamespace(cfg=paths.config()))
def log(message):
with (adapter / 'transport.log').open('a') as f:
f.write(f"{time.strftime('%F %T')} {message}\n")
cli = salty_remote._Cli(policy.get('salty', 'salty'), policy.get('outage_seconds', 1800), time.sleep, time.time, log)
state = {'name': 'bench-dev-' + str(uuid.uuid4()), 'job_id': None}
state['remote'] = 'personal:/' + state['name']
remaining = lambda: deadline - time.monotonic()
def done(st, code, reason=None, cleanup='VERIFIED'):
stdout.touch(); stderr.touch()
return RunResult(st, code, time.time() - started, stdout, stderr, stalled_reason=reason, cleanup_state=cleanup)
with synced(rd) as manifest:
local = local_files(ws)
uploads = sorted(rel for rel, st in local.items() if manifest.get(rel) != st)
total = sum(local[rel][0] for rel in uploads)
if total > UPLOAD_LIMIT:
stderr.write_text(f'bench dev: {total >> 20} MiB of local changes to upload (limit {UPLOAD_LIMIT >> 20} MiB); '
'make big files inside bench dev instead\n')
return done('FAIL', 2)
request = {'identity': {'stage': 'terminal', 'operation_id': hd.name},
'dev': {'workspace': str(ws), 'command': req['command'], 'cwd': req.get('cwd', '.'),
'deadline_wall': time.time() + remaining() - FETCH_MARGIN_S,
'timeout': req.get('timeout'), # the command's own time, counted from container start
'own_inputs': bool(ctx.extra.get('own_inputs_required')),
'calib_images': calib_images(rd, ctx.job) if ctx.extra.get('own_inputs_required') else None,
'cpu': str((policy.get('cpu_by_stage') or {}).get('terminal', policy.get('cpu', 4)))}}
bundle(adapter / 'bundle.tar', ws, uploads, request)
shutil.copyfile(salty_remote.WORKER, adapter / 'worker.py')
try:
# The heaviest dev use is compiling this model: give it at least what its formal compiles needed
# (salty_remote._memory doubles after an OOM; 10-14: vit.h.14 died at 16Gi in bench dev).
memory = f"{max(int(str(m).removesuffix('Gi')) for m in ((policy.get('memory_by_stage') or {}).get('terminal', policy.get('memory', '16Gi')), salty_remote._memory(policy, SimpleNamespace(name='05-compile'), ctx)))}Gi"
state['job_id'] = salty_remote._submit(cli, state, policy, SimpleNamespace(name='terminal'), adapter, memory)
(adapter / 'remote.json').write_text(json.dumps(state))
with synced(rd) as manifest: # the remote copy has them now, whatever the command does
manifest.update({rel: local[rel] for rel in uploads})
job_state = None
while job_state not in ('succeeded', 'failed', 'stopped'):
if remaining() < -GRACE_S: # the worker's own timeout should have ended it already
cli.call('stop', state['job_id'], ok_404=True)
time.sleep(5 if time.time() - started < 120 else 30) # short commands answer fast; long ones poll gently
out = cli.call('describe', state['job_id'])[1]
job_state = (re.search(r'^状态:\s*(\w+)', out, re.MULTILINE) or [None, None])[1]
if job_state != 'succeeded':
salty_remote._remote_logs(policy, state['job_id'], stderr)
# A container OOM gives no reason; its memory log shows it (the agent cannot raise the limit itself).
if cli.once(['storage', 'cp', state['remote'] + '/usage.log', str(adapter / 'usage.log')])[0] == 0:
with stderr.open('a') as f:
f.write(f'\n[bench dev container: {memory} memory; usage every 15s — a peak near the limit means it ran out of memory]\n'
+ (adapter / 'usage.log').read_text()[-4000:])
return done('FAIL', None, 'TERMINAL_DEADLINE_EXCEEDED' if remaining() < 0 else 'REMOTE_WORKER_FAILED')
for f in ('result.json', 'outputs.tar', 'stdout.log', 'stderr.log'):
salty_remote._fetch(cli, f"{state['remote']}/result/{f}", adapter / f)
result = json.loads((adapter / 'result.json').read_text())
if result.get('identity') != request['identity']:
raise ValueError('remote result belongs to another dev command')
installed = install(ws, adapter / 'outputs.tar', result)
with synced(rd) as manifest:
for rel in installed:
st = (ws / rel).stat()
manifest[rel] = [st.st_size, st.st_mtime_ns]
shutil.copyfile(adapter / 'stdout.log', stdout)
shutil.copyfile(adapter / 'stderr.log', stderr)
with stdout.open('a') as f:
if installed:
f.write(f"\n[bench dev] copied back to the workspace: {', '.join(installed[:50])}\n")
if result.get('remote_only'):
f.write('[bench dev] kept only in the development workspace (too big to copy back; usable from bench dev '
'and compile.yaml): ' + ', '.join(f'{r} ({s >> 20} MiB)' for r, s in result['remote_only'][:50]) + '\n')
code = result.get('exit_code')
return done('OK' if code == 0 else 'FAIL', code, 'TERMINAL_DEADLINE_EXCEEDED' if code == 124 else None)
except (TransportError, ValueError, OSError) as exc:
with stderr.open('a') as f:
f.write(f'\n[bench dev] {type(exc).__name__}: {exc}\n')
# The container ends by its own timeout and writes only to this workspace.
return done('FAIL', None, 'REMOTE_UNAVAILABLE', 'REMOTE_BOUNDED')
finally:
if state['job_id']:
salty_remote._cleanup(cli, state)
for f in ('bundle.tar', 'outputs.tar', 'worker.py'):
(adapter / f).unlink(missing_ok=True)

View File

@ -1,326 +0,0 @@
"""Run one whole stage attempt in a Salty container (opt-in).
Configuration: execution.salty_remote = {enabled: true, stages: [...], image: REGISTRY/...@sha256:...,
salty: /path/to/salty, cpu: 2, cpu_by_stage: {05-compile: 8}, memory: 8Gi, memory_by_stage: {05-compile: 16Gi},
wait_seconds: 21600, poll_seconds: 30, outage_seconds: 1800}.
The controller keeps the agent, ledger, quota and slot/board locks. The container
(salty_remote_worker.py) recreates the old compile-server paths over personal
storage, so frozen job paths and caches resolve unchanged. Only the stage
directory, board receipts, new artifacts and 05's HBM come back to the controller.
"""
from __future__ import annotations
import hashlib
import io
import json
import os
import uuid
import shutil
import re
from pathlib import Path, PurePosixPath
import subprocess
import tarfile
import time
import yaml
from . import execution, paths
from .salty_client import TransportError, JOB_PATTERN
WORKER = Path(__file__).with_name("salty_remote_worker.py")
CONTAINER_PYTHON = "/home/ming01.wang/toolchains/bench-venv/bin/python"
BOOT = ("mkdir -p /home/ming01.wang && ln -sfn /data/personal/bench-runtime/home/toolchains /home/ming01.wang/toolchains"
" && exec " + CONTAINER_PYTHON + " -B {container_root}/worker.py {container_root}")
CODE_DIRS = ("harness", "registry", "scripts", "bin", "export")
def selected(ctx, name: str) -> bool:
if os.environ.get("BENCH_SALTY_WORKER"):
return False
policy = ((ctx.cfg.get("execution") or {}).get("salty_remote")) or {}
return policy.get("enabled") is True and name in policy.get("stages", [])
def _policy(ctx) -> dict:
policy = dict(ctx.cfg["execution"]["salty_remote"])
if "@sha256:" not in policy.get("image", ""):
raise ValueError("salty_remote.image must be pinned by digest")
return policy
def _work_dir(stage, ctx) -> Path | None:
"""05 leaves its HBM in the compile work dir (the completion receipt hashes it there); 06 reads that dir."""
from .stages.base import work_dir
return work_dir(ctx) if stage.name in ("05-compile", "06-hbm") else None
def _bundle(path: Path, stage, ctx) -> None:
config = yaml.safe_load(paths.CONFIG_PATH.read_text())
config["paths"]["harness_python"] = CONTAINER_PYTHON
job_json = ctx.job_dir / "job.json"
files = {str(job_json): job_json}
if ctx.agent_config:
files[str(ctx.agent_config)] = ctx.agent_config
links = {}
if (ctx.job_dir / "data").is_symlink():
links[str(ctx.job_dir / "data")] = os.readlink(ctx.job_dir / "data")
env = {key: os.environ[key] for key in ("BENCH_BOARD_RECEIPT_DIR", "BENCH_OPERATION_ID") if key in os.environ}
current = execution.current()
phase = current.snapshot().get("phases", {}).get(execution.current_operation()) if current else None
if phase and phase.get("execution_deadline_monotonic"):
# The container has no execution state; board_perf caps its board call with this wall-clock end.
env["BENCH_PHASE_DEADLINE"] = str(time.time() + phase["execution_deadline_monotonic"] - time.monotonic())
request = {"identity": {"run_id": ctx.run_id, "job_id": ctx.job["job_id"], "rollout": ctx.rollout,
"stage": stage.name, "attempt": ctx.attempt},
"run_id": ctx.run_id, "stage": stage.name, "attempt": ctx.attempt, "lane": ctx.lane,
"rollout": ctx.rollout, "job_json": str(job_json), "job_dir": str(ctx.job_dir),
"stage_root": str(ctx.stage_root), "artifacts": str(ctx.artifacts),
"agent_config": str(ctx.agent_config) if ctx.agent_config else None,
"extra": json.loads(json.dumps(ctx.extra, default=str)), "env": env, "links": links,
"work_dir": str(_work_dir(stage, ctx)) if _work_dir(stage, ctx) else None,
"files": {destination: f"files/{i}" for i, destination in enumerate(files)}}
root = paths.BENCH_ROOT
with tarfile.open(path, "x") as tar:
for name, data in (("request.json", json.dumps(request).encode()),
("run-config.yaml", yaml.safe_dump(config, allow_unicode=True).encode())):
info = tarfile.TarInfo(name)
info.size = len(data)
tar.addfile(info, io.BytesIO(data))
for i, source in enumerate(files.values()):
tar.add(source, f"files/{i}")
for name in CODE_DIRS:
if (root / name).is_dir():
tar.add(root / name, f"code/{name}",
filter=lambda m: None if "__pycache__" in m.name or m.name.endswith(".pyc") else m)
def _install(result: dict, archive: Path, roots: list[Path]) -> None:
"""Members are absolute paths without the leading slash; only the job dir and the compile work dir are writable."""
data = archive.read_bytes()
if hashlib.sha256(data).hexdigest() != result["archive"]["sha256"]:
raise ValueError("stage output archive bytes differ from its receipt")
with tarfile.open(fileobj=io.BytesIO(data)) as tar:
members = tar.getmembers()
if {m.name for m in members} != set(result["outputs"]):
raise ValueError("stage output archive members differ from receipt")
for member in members:
rel = PurePosixPath(member.name)
target = Path("/", rel)
if (not member.isfile() or rel.is_absolute() or ".." in rel.parts
or not any(target.is_relative_to(root) for root in roots)):
raise ValueError("unsafe stage output member: " + member.name)
body = tar.extractfile(member).read()
if hashlib.sha256(body).hexdigest() != result["outputs"][member.name]["sha256"]:
raise ValueError("stage output bytes fail verification: " + member.name)
target.parent.mkdir(parents=True, exist_ok=True)
tmp = target.with_name(target.name + ".salty-tmp")
tmp.write_bytes(body)
os.replace(tmp, target)
def _not_found(err: str) -> bool:
return "not_found" in err or "404" in err
class _Cli:
"""Salty CLI with retries. Every step used here is safe to repeat: the job name is unique per
attempt (submit looks it up by name first), uploads overwrite this attempt's own directory,
describe/download only read, and rm of a missing object is success. A step fails only after
`outage` seconds of consecutive errors (the Mac mini's link drops requests in bursts)."""
def __init__(self, salty, outage, sleep, clock, log):
self.salty, self.outage, self.sleep, self.clock, self.log = salty, outage, sleep, clock, log
def once(self, args, timeout=120):
proc = subprocess.Popen([self.salty, *args], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
try:
out, err = proc.communicate(timeout=timeout)
return proc.returncode, out, err
except subprocess.TimeoutExpired:
proc.terminate() # never SIGKILL; a CLI ignoring SIGTERM is left alone and the step retried
try:
proc.communicate(timeout=10)
except subprocess.TimeoutExpired:
pass
return None, "", "timeout"
def call(self, *args, timeout=120, ok_404=False, before=None):
failing_since = None
while True:
if before:
before()
rc, out, err = self.once(args, timeout)
if rc == 0 or (ok_404 and rc is not None and _not_found(err)):
return rc, out, err
failing_since = failing_since or self.clock()
self.log(f"salty {' '.join(args[:2])} failed: {err.strip()[-200:]}")
if self.clock() - failing_since >= self.outage:
raise TransportError(f"salty {' '.join(args[:2])} kept failing for {self.outage}s: {err.strip()[-200:]}")
self.sleep(min(60, 5 + (self.clock() - failing_since) / 4))
def _memory(policy, stage, ctx) -> str:
"""Containers run with memory.oom.group=1: an OOM kills the whole job with no reason given. The need is the
model's, so a failed (not stopped) job of this stage in any rollout of any job of this model (other runs and
marches too) sets the floor: twice the memory it had, cap 128Gi (near the old compile server). Only the
models that need it hold more of the queue."""
base = int((policy.get("memory_by_stage") or {}).get(stage.name, policy.get("memory", "16Gi")).removesuffix("Gi"))
gi = base
model = "--".join(ctx.job_dir.name.split("--")[:2]) # <task>--<model>--<march>--...
for meta in ctx.job_dir.parent.parent.glob(f"*/{model}--*/rollouts/*/stages/{stage.name}/a*/meta.json"):
m = json.loads(meta.read_text())
if m.get("error_class") == "REMOTE_WORKER_FAILED" and m.get("salty_job_state") == "failed":
gi = max(gi, 2 * int(str(m.get("salty_memory") or f"{base}Gi").removesuffix("Gi")))
return f"{min(gi, 128)}Gi"
def _submit(cli, state, policy, stage, adapter, memory) -> str:
remote, name = state["remote"], state["name"]
cli.call("storage", "mkdir", "-p", remote)
for f in ("bundle.tar", "worker.py"):
cli.call("storage", "cp", str(adapter / f), f"{remote}/{f}", timeout=1800)
cpu = str((policy.get("cpu_by_stage") or {}).get(stage.name, policy.get("cpu", 4)))
# The container sees every host CPU (244) but is throttled to `cpu`; torch/numpy size their thread
# pools from the CPU count and then spend most of the time throttled (10-12 probe: 16x slower).
caps = [arg for var in ("OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS")
for arg in ("-e", f"{var}={max(1, int(float(cpu)))}")]
boot = BOOT.replace("{container_root}", "/data/personal/" + name)
current = execution.current()
if current:
# The phase deadline kills only the local side; without this the container ran on for
# hours with nobody waiting for it (10-13: 7 such jobs, ~34 cores).
boot = boot.replace("&& exec ", f"&& exec timeout {int(current.policy(stage.name)['execution_deadline_s'])} ", 1)
run = ["run", policy["image"], "--name", name, "-e", "PYTHONUNBUFFERED=1", *caps,
"--cpu", cpu,
"--memory", memory,
"--", "/bin/sh", "-c", boot]
failing_since = None
while True: # a lost reply must not create a second job: look the unique name up before every (re)submit
rc, out, err = cli.call("describe", name, ok_404=True)
if rc == 0:
return re.search(JOB_PATTERN, out).group(0)
rc, out, err = cli.once(run)
if rc == 0 and re.search(JOB_PATTERN, out):
return re.search(JOB_PATTERN, out).group(0)
failing_since = failing_since or cli.clock()
cli.log(f"salty run failed: {err.strip()[-200:]}")
if cli.clock() - failing_since >= cli.outage:
raise TransportError("salty run kept failing: " + err.strip()[-200:])
cli.sleep(10)
def _wait(cli, job_id, policy, started, clock, sleep) -> str:
while True:
out = cli.call("describe", job_id)[1]
state = (re.search(r"^状态:\s*(\w+)", out, re.MULTILINE) or [None, None])[1]
if state in ("succeeded", "failed", "stopped"):
return state
if clock() - started >= policy.get("wait_seconds", 21600):
raise TransportError("stage wait budget reached; Salty job retained")
sleep(policy.get("poll_seconds", 30))
def _fetch(cli, remote_file: str, destination: Path) -> None:
part = destination.with_name(destination.name + ".part")
cli.call("storage", "cp", remote_file, str(part), timeout=1800, before=lambda: part.unlink(missing_ok=True))
os.replace(part, destination)
def _cleanup(cli, state) -> None:
"""Garbage only: the stage result is already installed, so a lasting outage just leaves the job/dir behind."""
try:
cli.call("rm", state["job_id"], ok_404=True)
cli.call("storage", "rm", "-r", state["remote"], ok_404=True)
state["cleaned"] = True
except TransportError as exc:
cli.log(f"cleanup deferred: {exc}")
def run(stage, ctx, *, sleep=time.sleep, clock=time.time):
"""An attempt interrupted on the controller is not resumed (stagerun opens a new attempt); its Salty job
finishes alone and salty_offload.reap_orphans removes it and personal:/bench-stage-<uuid> once it ends."""
out = _run(stage, ctx, sleep=sleep, clock=clock)
sd = stage.stage_dir(ctx)
if not (sd / "meta.json").exists(): # failures before the container wrote its own meta
from .execution import atomic_json
# Same identity fields as Stage.run, or the stage worker rejects the receipt (metadata mismatch).
meta = {"stage": stage.name, "attempt": ctx.attempt, "rollout": ctx.rollout,
**ctx.extra.get("operation_identity", {}),
"state": out.state, "exit_code": out.exit_code, "error_class": out.error_class,
"duration_s": round(out.duration_s, 3), "finished_at": time.time(),
"measurements": out.measurements, **out.meta}
atomic_json(sd / "meta.json", meta)
atomic_json(ctx.stage_root / stage.name / "meta.json", meta)
return out
def _run(stage, ctx, *, sleep, clock):
from .stages.base import StageOutcome
from .execution import atomic_json
started = clock()
sd = stage.stage_dir(ctx)
adapter = sd / "salty"
adapter.mkdir(exist_ok=True)
policy = _policy(ctx)
def log(message):
with (adapter / "transport.log").open("a") as f:
f.write(f"{time.strftime('%F %T')} {message}\n")
cli = _Cli(policy.get("salty", "salty"), policy.get("outage_seconds", 1800), sleep, clock, log)
state_file = adapter / "remote.json"
state = json.loads(state_file.read_text()) if state_file.exists() else None
try:
if state is None:
_bundle(adapter / "bundle.tar", stage, ctx)
shutil.copyfile(WORKER, adapter / "worker.py")
name = "bench-stage-" + str(uuid.uuid4())
state = {"name": name, "remote": "personal:/" + name, "job_id": None}
atomic_json(state_file, state)
if not state["job_id"]:
state["memory"] = _memory(policy, stage, ctx)
state["job_id"] = _submit(cli, state, policy, stage, adapter, state["memory"])
atomic_json(state_file, state)
job_state = _wait(cli, state["job_id"], policy, started, clock, sleep)
meta = {"executor": "salty", "salty_job_id": state["job_id"], "salty_memory": state.get("memory")}
if job_state != "succeeded":
# The worker never reached a result: no stage outcome exists, board receipts decide quarantine.
_remote_logs(policy, state["job_id"], sd / "stderr.log")
if cli.once(["storage", "cp", state["remote"] + "/usage.log", str(adapter / "usage.log")])[0] == 0:
with (sd / "stderr.log").open("a") as f:
f.write("\n[container memory and disk, every 15s]\n" + (adapter / "usage.log").read_text()[-20000:])
# Logs, describe (OOM/eviction reason) and the memory log are in stderr.log now; keeping the job and its
# personal:/bench-stage-* dir only piled up (10-15: 327 failed jobs, nobody looked at them).
_cleanup(cli, state)
atomic_json(state_file, state)
return StageOutcome("FAIL", 1, "REMOTE_WORKER_FAILED", clock() - started, dict(meta, salty_job_state=job_state))
_fetch(cli, state["remote"] + "/result/result.json", adapter / "result.json")
result = json.loads((adapter / "result.json").read_text())
if result.get("identity", {}).get("attempt") != ctx.attempt or result["identity"]["stage"] != stage.name:
raise ValueError("remote result belongs to another stage attempt")
_fetch(cli, state["remote"] + "/result/outputs.tar", adapter / "outputs.tar")
_install(result, adapter / "outputs.tar", [p for p in (ctx.job_dir, _work_dir(stage, ctx)) if p])
_cleanup(cli, state)
atomic_json(state_file, state)
for name in ("bundle.tar", "outputs.tar", "worker.py"):
(adapter / name).unlink(missing_ok=True)
ctx.extra.update({k: v for k, v in result["extra"].items() if k not in ctx.extra})
o = result["outcome"]
return StageOutcome(o["state"], o["exit_code"], o["error_class"], o["duration_s"], dict(o["meta"], **meta),
o["measurements"] or {})
except (TransportError, ValueError, OSError) as exc:
# An ordinary stage failure the agent can retry; board stages still quarantine on missing receipts.
with (sd / "stderr.log").open("a") as f:
f.write(f"\n[salty remote] {type(exc).__name__}: {exc}\n")
return StageOutcome("FAIL", None, "REMOTE_UNAVAILABLE", clock() - started,
{"executor": "salty", "salty_job_id": (state or {}).get("job_id"),
"salty_remote": (state or {}).get("remote")})
def _remote_logs(policy: dict, job_id: str | None, destination: Path) -> None:
if not job_id:
return
cli = _Cli(policy.get("salty", "salty"), 0, None, None, None)
rc, out, err = cli.once(["logs", job_id])
rc, desc, derr = cli.once(["describe", job_id]) # the reason (OOM, eviction) lives here; the job is removed next
destination.write_text(out[-200000:] + err[-20000:] + "\n[salty describe]\n" + desc + derr)

View File

@ -1,383 +0,0 @@
"""Container side of salty_remote: rebuild the old compile-server paths, run one stage, pack its directory.
Usage (inside the Salty job, under the bench-venv interpreter): worker.py <container_root>
<container_root>/bundle.tar holds request.json, run-config.yaml, code/ and files/.
Writes <container_root>/result/{result.json,outputs.tar}.
"""
from __future__ import annotations
import hashlib
import io
import json
import os
from pathlib import Path
import shutil
import subprocess
import sys
import tarfile
import threading
import time
HOME = Path("/home/ming01.wang")
BENCH = Path("/mnt/data/ming01.wang/model-compile-hbdk/bench")
# harness/container_site/sitecustomize.py caps onnxruntime at the container's CPU quota (it shadows the image's own
# sitecustomize, which only installs Ubuntu's apport crash hook).
CONTAINER_PYTHONPATH = f"{BENCH / 'harness/container_site'}{os.pathsep}{BENCH}"
PERSONAL = Path("/data/personal")
RUNTIME = PERSONAL / "bench-runtime"
LOCAL_WORK = Path("/tmp/bench-work")
PERSISTENT_WORK = PERSONAL / "bench-root/work"
UNUSED_ONNX = ("*_original_float_model.onnx", "*_optimized_float_model.onnx", "*_calibrated_model.onnx") # no reader (s06_hbm)
def _link(path: Path, target: Path) -> None:
if path.is_symlink() or path.exists():
if path.is_symlink() and os.readlink(path) == str(target):
return
raise RuntimeError(f"layout path already exists: {path}")
path.parent.mkdir(parents=True, exist_ok=True)
path.symlink_to(target)
def layout(stage: str) -> None:
"""Frozen job paths point at /home/ming01.wang/... and the old bench root; keep them byte-identical."""
# Always local: hb_compile/ld.lld crawl on JuiceFS, and base.work_dir() resolves this link, so every
# container must agree on /tmp/bench-work (04 writes it into compile.yaml, 05 compiles there).
LOCAL_WORK.mkdir(parents=True, exist_ok=True)
for name, target in (("toolchains", RUNTIME / "home/toolchains"), ("outputs", RUNTIME / "home/outputs"),
("bench-cache", PERSONAL / "bench/cache"), ("bench-data", PERSONAL / "bench/data"),
("bench-work", LOCAL_WORK)):
_link(HOME / name, target)
(PERSONAL / "bench-root/runs").mkdir(parents=True, exist_ok=True)
links = [("cache", PERSONAL / "bench/cache"), ("data", PERSONAL / "bench/data"), (".env.local", RUNTIME / "board.env")]
if stage != "export": # export() builds real rollout dirs instead
links.append(("runs", PERSONAL / "bench-root/runs"))
for name, target in links:
_link(BENCH / name, target)
(BENCH / "state").mkdir(exist_ok=True)
cuda_links()
ssh = Path("/root/.ssh")
ssh.mkdir(mode=0o700, exist_ok=True)
for name in ("rdk_sandbox", "known_hosts"):
if not (ssh / name).exists():
shutil.copyfile(RUNTIME / "ssh" / name, ssh / name)
os.chmod(ssh / name, 0o600)
def cuda_links() -> None:
# PTQ's libhbdnn.so loads CUDA 12 / cuDNN 9 from here (installed on the old server); bench-venv's torch wheels carry them.
cuda = Path("/usr/local/cuda/lib64")
cuda.mkdir(parents=True, exist_ok=True)
for lib in (HOME / "toolchains/bench-venv/lib/python3.10/site-packages/nvidia").glob("*/lib/*.so*"):
if not (cuda / lib.name).exists():
(cuda / lib.name).symlink_to(lib)
def unpack(bundle: Path) -> dict:
with tarfile.open(bundle) as tar:
request = json.load(tar.extractfile("request.json"))
for member in tar.getmembers():
if not member.isfile() or ".." in Path(member.name).parts or member.name.startswith("/"):
continue
if member.name.startswith("code/"):
target = BENCH / member.name[len("code/"):]
elif member.name == "run-config.yaml":
target = BENCH / "run-config.yaml"
else:
continue
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(tar.extractfile(member).read())
for destination, member in request["files"].items():
data = tar.extractfile(member).read()
path = Path(destination)
if path.exists() and path.read_bytes() == data:
continue
if path.name == "job.json" and path.exists():
raise RuntimeError("job.json differs from the controller copy")
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(data)
for path, target in request.get("links", {}).items():
_link(Path(path), Path(target))
return request
def run_stage(request: dict) -> dict:
os.environ.update(request["env"], BENCH_ROOT=str(BENCH), BENCH_CONFIG=str(BENCH / "run-config.yaml"),
BENCH_STATE_DIR=str(BENCH / "state"), HOME=str(HOME), BENCH_SALTY_WORKER="1",
PYTHONPATH=CONTAINER_PYTHONPATH)
sys.path.insert(0, str(BENCH))
from harness.stages import BY_NAME, StageContext
ctx = StageContext(run_id=request["run_id"], job=json.loads(Path(request["job_json"]).read_text()),
job_dir=Path(request["job_dir"]), stage_root=Path(request["stage_root"]),
artifacts=Path(request["artifacts"]), rollout=request["rollout"], attempt=request["attempt"],
lane=request["lane"], agent_config=Path(request["agent_config"]) if request["agent_config"] else None,
extra=request["extra"])
out = BY_NAME[request["stage"]]().run(ctx)
return {"outcome": {"state": out.state, "exit_code": out.exit_code, "error_class": out.error_class,
"duration_s": out.duration_s, "meta": out.meta, "measurements": out.measurements},
"extra": ctx.extra}
def snapshot(directory: Path) -> dict:
"""Artifacts are copied with copy2 (source mtime kept), so new/changed files are found by comparing stats."""
return {p: (p.stat().st_size, p.stat().st_mtime_ns) for p in directory.rglob("*")
if p.is_file() and not p.is_symlink()} if directory.is_dir() else {}
def collect(request: dict, result_dir: Path, before: dict) -> dict:
"""Everything the controller re-reads: stage-level files, this attempt's directory, board receipts,
artifacts written during this stage (HBM/BC) and, for 05, the HBM in its work dir (receipts hash it).
Members are absolute paths without the leading slash; the controller only installs under allowed roots."""
stage_dir = Path(request["stage_root"]) / request["stage"]
files = [p for p in stage_dir.iterdir() if p.is_file()] if stage_dir.is_dir() else []
for directory in (stage_dir / f"a{request['attempt']:02d}", Path(request["env"].get("BENCH_BOARD_RECEIPT_DIR", "/nonexistent"))):
if directory.is_dir():
files += [p for p in directory.rglob("*") if p.is_file() and not p.is_symlink()]
files += [p for p, stat in snapshot(Path(request["artifacts"])).items() if before.get(p) != stat]
if request["stage"] == "05-compile" and request.get("work_dir") and Path(request["work_dir"]).is_dir():
files += list(Path(request["work_dir"]).rglob("*.hbm"))
outputs = {}
result_dir.mkdir(parents=True, exist_ok=True)
with tarfile.open(result_dir / "outputs.tar", "w") as tar:
for path in sorted(set(files)):
name = str(path).lstrip("/")
data = path.read_bytes()
info = tarfile.TarInfo(name)
info.size, info.mode = len(data), 0o644
tar.addfile(info, io.BytesIO(data))
outputs[name] = {"sha256": hashlib.sha256(data).hexdigest(), "bytes": len(data)}
archive = (result_dir / "outputs.tar").read_bytes()
return {"outputs": outputs, "archive": {"sha256": hashlib.sha256(archive).hexdigest(), "bytes": len(archive)}}
def _memory() -> str:
cg = Path("/sys/fs/cgroup")
for current, peak in (("memory.current", "memory.peak"), ("memory/memory.usage_in_bytes", "memory/memory.max_usage_in_bytes")):
if (cg / current).exists():
read = lambda n: int((cg / n).read_text()) >> 20 if (cg / n).exists() else -1
return f"mem={read(current)}M peak={read(peak)}M"
return "mem=?"
def usage_log(path: Path, every: float = 15) -> None:
"""Memory, free /tmp space and the largest local files, kept on personal storage: a container
killed for OOM or evicted for node disk pressure leaves no reason and no logs, this file survives."""
while True:
files = []
for p in Path("/tmp").rglob("*"):
try:
if p.is_file() and not p.is_symlink():
files.append((p.stat().st_size, p))
except OSError:
pass
top = ", ".join(f"{p}={size >> 20}M" for size, p in sorted(files, reverse=True)[:5])
with path.open("a") as f:
f.write(f"{time.strftime('%T')} {_memory()} tmp_free={shutil.disk_usage('/tmp').free >> 30}G "
f"tmp_files={sum(s for s, _ in files) >> 30}G top: {top}\n")
time.sleep(every)
def export(request: dict, root: Path) -> None:
"""scripts/salty_offload.py's job: unpack each archived rollout (the controller's files) into personal runs,
then run the controller's exporter over a local tree of exactly those files.
An archive is <kk>.tgz plus <kk>.u<time>.tgz overlays of later changes, unpacked in name order; each one's
marker lists the members it unpacked."""
runs = PERSONAL / "bench-root/runs"
for tgz in sorted((PERSONAL / "bench-root/archive").glob("*/*/*.tgz")):
rel = (tgz.parent.parent.name, tgz.parent.name, "rollouts", tgz.name.split(".")[0])
st = tgz.stat()
marker = runs.joinpath(*rel) / f".unpacked-{tgz.name}-{st.st_size}-{st.st_mtime_ns}"
if marker.exists() and marker.stat().st_size: # an empty marker predates member lists: unpack again
continue
try:
with tarfile.open(tgz) as tar:
# workspace/ stays in the archive: the exporter never reads it (verify.py) and it is the bulk.
members = [m for m in tar.getmembers() if m.isfile() and Path(m.name).parts[:4] == rel
and ".." not in Path(m.name).parts and Path(m.name).parts[4:5] != ("workspace",)]
tar.extractall(runs, members=members)
except (tarfile.TarError, EOFError, OSError) as exc: # still uploading: the next export takes it
print(f"skip {tgz}: {type(exc).__name__}: {exc}", flush=True)
continue
marker.parent.mkdir(parents=True, exist_ok=True)
marker.write_text("\n".join(m.name for m in members) + "\n")
# Only the archived files: personal runs also holds what containers wrote for attempts the controller had
# already abandoned (xfeat 06-hbm a00 ran on past its deadline), and those would change the gate evidence.
# verify.py resolves the rollout dir and refuses symlinked prediction sources and terminal_result.json, so
# dirs, stage files and the rollout's own files are real copies; the rest is linked (no JuiceFS read).
for run in request["export"]:
for rd in sorted((runs / run).glob("*/rollouts/*")):
names = {n for m in rd.glob(".unpacked-*") for n in m.read_text().splitlines() if n}
for name in names:
source, target, parts = runs / name, BENCH / "runs" / name, Path(name).parts
target.parent.mkdir(parents=True, exist_ok=True)
if parts[4] == "stages" or len(parts) == 5:
shutil.copyfile(source, target)
elif not target.is_symlink():
target.symlink_to(source)
for hbm in (rd / "artifacts").glob("*.hbm") if names else (): # left out of the archive as mirrored
target = BENCH / "runs" / hbm.relative_to(runs)
if not target.exists() and not target.is_symlink():
target.parent.mkdir(parents=True, exist_ok=True)
target.symlink_to(hbm)
(PERSONAL / "bench-root/datasets").mkdir(exist_ok=True)
_link(BENCH / "datasets", PERSONAL / "bench-root/datasets") # kept between runs: filters and history carry over
env = dict(os.environ, BENCH_ROOT=str(BENCH), BENCH_STATE_DIR=str(BENCH / "state"), HOME=str(HOME),
PYTHONPATH=str(BENCH))
done = {}
for run, config in request["export"].items():
dataset = BENCH / "datasets" / f"{run}-training"
seed = BENCH / "export-seed" / f"{run}.json"
if not (dataset / "manifest.json").exists() and seed.exists(): # first run: the controller's filters
dataset.mkdir(exist_ok=True)
shutil.copyfile(seed, dataset / "manifest.json")
done[run] = subprocess.run([sys.executable, "-B", "export/to_agentrl.py", "--runs", run, "--dataset-id",
dataset.name, "--refresh"], cwd=BENCH, env=dict(env, BENCH_CONFIG=config),
stdout=subprocess.DEVNULL).returncode
(root / "result").mkdir(exist_ok=True)
with tarfile.open(root / "result/datasets.tar.gz", "w:gz") as tar:
for run, rc in done.items():
if rc == 0:
for f in sorted((BENCH / "datasets" / f"{run}-training").glob("*.json*")):
tar.add(f, f"{run}-training/{f.name}")
(root / "result/result.json").write_text(json.dumps({"identity": request["identity"], "exit_codes": done}))
NOBODY = 65534
DEV_COPY_BACK = 16 << 20 # per file; bigger outputs (graphs, calibration sets) stay on personal storage
DEV_COPY_TOTAL = 64 << 20 # the controller downloads at ~1 MB/s
DEV_LOG_TAIL = 4 << 20
# Inside a user+mount namespace as uid nobody: cover personal storage, re-mount only what the agent may see,
# then drop every capability so the cover cannot be unmounted (10-14 probe: without the drop, umount
# revealed all of personal storage). /proc/<pid>/root of outside processes is not reachable from there.
SANDBOX = r"""set -e
i=0
for src in $DEV_MOUNTS; do i=$((i+1)); mkdir -p /tmp/devsbx/$i; mount -n --rbind "$src" /tmp/devsbx/$i; done
mount -n -t tmpfs none /data/personal
i=0
for src in $DEV_MOUNTS; do i=$((i+1)); mkdir -p "$src"; mount -n --move /tmp/devsbx/$i "$src"; done
cd "$DEV_WORKSPACE/$DEV_CWD"
exec setpriv --bounding-set=-all --inh-caps=-all --no-new-privs timeout -k 30 "$DEV_TIMEOUT" bash -c "$DEV_COMMAND"
"""
def dev(root: Path) -> None:
"""bench dev (harness/salty_dev.py): the agent's command with the toolchain, sandboxed; workspace changes
are reported back. Read-only by ownership: everything visible but the workspace belongs to root."""
with tarfile.open(root / "bundle.tar") as tar:
request = json.load(tar.extractfile("request.json"))
spec = request["dev"]
for name, target in (("toolchains", RUNTIME / "home/toolchains"), ("outputs", RUNTIME / "home/outputs"),
("bench-cache", PERSONAL / "bench/cache"), ("bench-data", PERSONAL / "bench/data")):
_link(HOME / name, target)
for name, target in (("cache", PERSONAL / "bench/cache"), ("data", PERSONAL / "bench/data"),
("runs", PERSONAL / "bench-root/runs")):
_link(BENCH / name, target)
cuda_links() # hb_compile / hmct need them; board keys and .env.local are deliberately not laid out
ws = Path(spec["workspace"]).resolve() # BENCH/runs/... -> this rollout's personal-storage copy
if not ws.is_relative_to(PERSONAL / "bench-root/runs") or ws.name != "workspace":
raise RuntimeError("dev workspace must be a rollout workspace on personal storage")
ws.mkdir(parents=True, exist_ok=True)
os.chown(ws, NOBODY, NOBODY)
for member in tar.getmembers():
rel = Path(member.name)
if not member.isfile() or ".." in rel.parts or rel.is_absolute():
continue
if rel.parts[0] == "code":
target = BENCH / Path(*rel.parts[1:])
elif rel.parts[0] == "ws":
target = ws / Path(*rel.parts[1:])
for parent in reversed([p for p in target.parents if p.is_relative_to(ws) and p != ws]):
if not parent.exists():
parent.mkdir(); os.chown(parent, NOBODY, NOBODY)
else:
continue
target.parent.mkdir(parents=True, exist_ok=True)
if target.is_symlink():
target.unlink()
target.write_bytes(tar.extractfile(member).read())
if rel.parts[0] == "ws":
os.chown(target, NOBODY, NOBODY)
home = Path("/tmp/devhome"); home.mkdir(exist_ok=True); os.chown(home, NOBODY, NOBODY)
# Only calibration data: bench/data/t1 (graded evaluation set, with labels) and pool must stay unseen,
# or an agent can pick configs or calibrate on the answers (10-14: efficientvit.b3 read t1 in bench dev).
# Likewise only the cache a model is built from: adapter-probes holds private harness/registry copies with
# reference configs (8 GLM rollouts read them, regnet.y.800mf copied one), dataset-downloads the labelled
# datasets T1 is drawn from, baseline the evaluation images' FP32 outputs.
# From agent v18 (spec["own_inputs"]) the graph and calibration set are the agent's: no platform ONNX, and
# the calibration training images unpreprocessed (calib-raw/<dataset>) instead of the platform's npy.
own = spec.get("own_inputs")
cache = [PERSONAL / "bench/cache" / name for name in ("weights", "sources", "venvs", *(() if own else ("onnx",)))]
calib = PERSONAL / "bench/data" / ("calib-raw" if own else "calib")
mounts = [RUNTIME / "home/toolchains", RUNTIME / "home/outputs", *cache, calib, ws]
tc, cpu = HOME / "toolchains", spec["cpu"]
env = {"PATH": f"{tc}/oe-3.7.0/venv/bin:{tc}/bench-venv/bin:/usr/local/bin:/usr/bin:/bin", "HOME": str(home),
"LANG": "C.UTF-8", "PYTHONPATH": CONTAINER_PYTHONPATH, "BENCH_ROOT": str(BENCH), "PYTHONUNBUFFERED": "1",
"OMP_NUM_THREADS": cpu, "MKL_NUM_THREADS": cpu, "OPENBLAS_NUM_THREADS": cpu,
"DEV_MOUNTS": " ".join(str(p) for p in mounts if p.is_dir()), "DEV_WORKSPACE": str(ws),
"DEV_CWD": spec.get("cwd", "."), "DEV_COMMAND": spec["command"],
**({"BENCH_CALIB_IMAGES": spec["calib_images"]} if spec.get("calib_images") else {}),
# Wall clock: container start-up must not eat into the time the results need to come back.
"DEV_TIMEOUT": str(max(1, int(min(spec["deadline_wall"] - time.time(), spec.get("timeout") or float("inf")))))}
before = snapshot(ws)
out_dir = root / "result"
out_dir.mkdir(exist_ok=True)
with (out_dir / "stdout.log").open("wb") as out, (out_dir / "stderr.log").open("wb") as err:
code = subprocess.run(["setpriv", f"--reuid={NOBODY}", f"--regid={NOBODY}", "--clear-groups",
"unshare", "-Urm", "sh", "-c", SANDBOX], stdout=out, stderr=err, env=env).returncode
for name in ("stdout.log", "stderr.log"):
path = out_dir / name
if path.stat().st_size > DEV_LOG_TAIL:
with path.open("rb") as f:
f.seek(-DEV_LOG_TAIL, os.SEEK_END)
tail = f.read()
path.write_bytes(b"[earlier output truncated]\n" + tail)
changed = sorted((p, st[0]) for p, st in snapshot(ws).items() if before.get(p) != st)
outputs, remote_only, total = {}, [], 0
with tarfile.open(out_dir / "outputs.tar", "w") as tar:
for path, size in changed:
rel = path.relative_to(ws).as_posix()
if size > DEV_COPY_BACK or total + size > DEV_COPY_TOTAL:
remote_only.append((rel, size)); continue
data = path.read_bytes()
info = tarfile.TarInfo(rel)
info.size, info.mode = len(data), 0o644
tar.addfile(info, io.BytesIO(data))
outputs[rel] = {"sha256": hashlib.sha256(data).hexdigest(), "bytes": len(data)}
total += size
archive = (out_dir / "outputs.tar").read_bytes()
(out_dir / "result.json").write_text(json.dumps({
"identity": request["identity"], "exit_code": code, "outputs": outputs, "remote_only": remote_only,
"archive": {"sha256": hashlib.sha256(archive).hexdigest(), "bytes": len(archive)}}))
def main() -> int:
root = Path(sys.argv[1])
threading.Thread(target=usage_log, args=(root / "usage.log",), daemon=True).start()
request = json.load(tarfile.open(root / "bundle.tar").extractfile("request.json"))
if "dev" in request:
dev(root)
return 0
layout(request["stage"])
request = unpack(root / "bundle.tar")
if "export" in request:
export(request, root)
return 0
# 06 reads 05's work dir in a later container: persist it after 05/06, restore it before 06.
rel = Path(request["work_dir"]).relative_to(HOME / "bench-work") if request.get("work_dir") else None
if rel and request["stage"] == "06-hbm" and (PERSISTENT_WORK / rel).is_dir():
shutil.copytree(PERSISTENT_WORK / rel, LOCAL_WORK / rel, symlinks=True, dirs_exist_ok=True)
before = snapshot(Path(request["artifacts"]))
result = run_stage(request)
if rel and request["stage"] in ("05-compile", "06-hbm"):
shutil.copytree(LOCAL_WORK / rel, PERSISTENT_WORK / rel, symlinks=True, dirs_exist_ok=True,
ignore=shutil.ignore_patterns(*UNUSED_ONNX))
if request["stage"] == "06-hbm": # s06 drops *_model.onnx after its verifier; mirror that
for f in (PERSISTENT_WORK / rel).rglob("*_model.onnx"):
f.unlink()
result.update(collect(request, root / "result", before), identity=request["identity"])
(root / "result/result.json").write_text(json.dumps(result, default=str))
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@ -1,509 +0,0 @@
"""Opt-in Stage adapter for one immutable PTQ compilation attempt on Salty.
Configuration: execution.salty = {enabled: true, stages: [05-compile],
image: REGISTRY/OWNER/IMAGE@sha256:..., image_source_sha256: ...,
image_runtime_contract_sha256: ..., verifier_sha256: ...,
salty: /absolute/path/to/salty, wait_seconds: 3600, poll_seconds: 10,
cleanup_terminal: true}. CPU/memory remain Salty defaults unless specified.
Only a normalized compiler configuration, one ONNX and calibration .npy bytes
are uploaded. Job configuration, credentials, source tree, state and board locks
never enter the bundle. Receipt existence always wins over submission intent.
"""
from __future__ import annotations
from contextlib import contextmanager
import fcntl
import hashlib
import io
import json
import math
import os
from pathlib import Path
import re
import shutil
import tarfile
import tempfile
import time
import yaml
from . import salty_stage_worker as protocol
from .salty_client import SaltyClient, AmbiguousOperation, TransportError, validate_image
def selected(ctx, stage):
policy = ((getattr(ctx, "cfg", None) or {}).get("execution") or {}).get("salty") or {}
if policy.get("enabled") is not True or stage not in policy.get("stages", []):
return False
if stage != "05-compile" or ctx.job.get("backend") != "ptq":
raise ValueError("Salty stage routing currently supports 05-compile PTQ only")
return True
def pending_attempt(stage_root):
"""Return the original unresolved compile attempt for trusted driver recovery.
A caller must resume this attempt, not assign len(previous_attempts). A
missing/partial transport is unresolved too; it is never a new submission.
"""
pending = []
for directory in sorted((Path(stage_root) / "05-compile").glob("a*")):
if not re.fullmatch(r"a[0-9]{2,}", directory.name):
continue
plan = directory / "salty/plan.json"
if plan.exists():
protocol.regular(plan)
value = json.loads(plan.read_text())
# Byte verification does not certify job/storage deletion. Older
# plans have no cleanup policy field, so require reconciliation.
unresolved = (value.get("state") != "VERIFIED" or
(value.get("cleanup_required", True) and value.get("cleanup_complete") is not True))
transport = directory / "salty/transport/receipt.json"
if not unresolved:
protocol.regular(transport)
receipt = json.loads(transport.read_text())
unresolved = (any(command.get("state") in {"STARTED", "UNKNOWN"} or command.get("process_left_running")
for command in receipt.get("commands", [])) or
any(download.get("state") not in {"COMPLETE", "PUBLISH_FAILED"}
for download in receipt.get("downloads", [])) or
(value.get("cleanup_required", True) and receipt.get("phase") != "CLEANED"))
if unresolved:
pending.append(int(directory.name[1:]))
elif (directory / "salty/transport").exists() or (directory / "salty/inputs.tar").exists():
pending.append(int(directory.name[1:]))
if len(pending) > 1:
raise AmbiguousOperation("multiple unresolved Salty attempts; operator reconciliation required")
return pending[0] if pending else None
def dispatch_recorded(stage_root, attempt):
"""True once this attempt's submit CLI was recorded (or a job ID is known).
Only that is dispatch evidence: a prepared bundle or a bare phase field is
not, so a spent compile budget still refuses it. An unreadable receipt is
never treated as undispatched.
"""
for directory in (Path(stage_root) / "05-compile").glob("a*"):
if re.fullmatch(r"a[0-9]{2,}", directory.name) and int(directory.name[1:]) == attempt:
transport = directory / "salty/transport/receipt.json"
if not transport.exists():
return False
try:
protocol.regular(transport)
receipt = json.loads(transport.read_text())
except (OSError, ValueError):
return True
return receipt.get("job_id") is not None or any(
command.get("label") == "submit" for command in receipt.get("commands", []))
return False
def _policy(ctx):
value = dict(ctx.cfg["execution"]["salty"])
allowed = {"enabled", "stages", "image", "image_source_sha256", "image_runtime_contract_sha256",
"verifier_sha256", "salty", "cpu", "memory", "wait_seconds", "poll_seconds", "cleanup_terminal"}
if set(value) - allowed or value.get("stages") != ["05-compile"]:
raise ValueError("unsupported Salty stage configuration")
value["image"] = validate_image(value["image"])
for key in ("image_source_sha256", "image_runtime_contract_sha256", "verifier_sha256"):
if not isinstance(value.get(key), str) or not protocol.HEX.fullmatch(value[key]):
raise ValueError("Salty route requires frozen " + key)
for key, default in (("wait_seconds", 3600), ("poll_seconds", 10)):
number = value.setdefault(key, default)
if isinstance(number, bool) or not isinstance(number, (float, int)) or not math.isfinite(number) or number <= 0:
raise ValueError("invalid Salty wait/poll budget")
if value["poll_seconds"] > 60:
raise ValueError("Salty polls must be at most 60 seconds")
if type(value.setdefault("cleanup_terminal", True)) is not bool:
raise ValueError("cleanup_terminal must be explicit boolean")
return value
def _atomic(path, value):
from .execution import atomic_json
atomic_json(path, value)
@contextmanager
def _lock(directory):
for part in (directory, *directory.parents):
if part.is_symlink():
raise ValueError("stage receipt path contains symlink")
directory.mkdir(parents=True, exist_ok=True)
path = directory / "stage.lock"
if path.is_symlink():
raise ValueError("stage lock symlink")
fd = os.open(path, os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600)
try:
try:
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError as exc:
raise TransportError("another caller owns this Salty stage attempt") from exc
yield
finally:
fcntl.flock(fd, fcntl.LOCK_UN)
os.close(fd)
def _identity(ctx):
return {"run_id": ctx.run_id, "job_id": ctx.job["job_id"], "rollout": ctx.rollout, "attempt": ctx.attempt,
"task": ctx.job["task"], "backend": ctx.job["backend"], "march": ctx.job["march"],
"qcfg": ctx.job["qcfg"], "recipe_submission": (ctx.job.get("recipe") or {}).get("submission_sha256")}
def _request(ctx, policy):
from .stages.s04_calibrate import compile_yaml_path
file = compile_yaml_path(ctx)
if protocol.regular(file).st_size > 1 << 20:
raise ValueError("oversized compile configuration")
config = yaml.safe_load(file.read_text())
if not isinstance(config, dict) or set(config) != set(protocol.CONFIG_FIELDS):
raise ValueError("unsupported compile configuration")
config = json.loads(json.dumps(config))
model = config["model_parameters"]
onnx = Path(model.pop("onnx_model"))
cwd = model.pop("working_dir")
from .stages.base import work_dir
if cwd != str(work_dir(ctx)):
raise ValueError("compile configuration belongs to another work directory")
calibration = Path(config["calibration_parameters"].pop("cal_data_dir"))
if not onnx.is_absolute() or not calibration.is_absolute() or onnx.suffix != ".onnx":
raise ValueError("local compile inputs require explicit absolute paths")
protocol.regular(onnx)
for part in (calibration, *calibration.parents):
if part.is_symlink():
raise ValueError("calibration path contains symlink")
if not calibration.is_dir():
raise ValueError("calibration directory missing")
entries = sorted(calibration.iterdir())
# The generated calibration directory is .npy-only; a legacy manifest is
# deliberately ignored, not packed. Any other non-.npy object is rejected.
sources = {"input/model.onnx": onnx}
npys = []
for path in entries:
if path.name == "manifest.json" and not path.is_symlink() and path.is_file():
continue
if path.suffix != ".npy":
raise ValueError("unexpected object in calibration input directory")
protocol.regular(path)
npys.append(path)
if not 1 <= len(npys) <= protocol.MAX_FILES:
raise ValueError("calibration sample count outside bounds")
sources.update({"input/calibration/%06d.npy" % i: file for i, file in enumerate(npys)})
inputs, total = {}, 0
for name, file in sources.items():
if any(part.startswith(".env") or part in {".ssh", "state"} for part in file.parts):
raise ValueError("credential/state path cannot be a compile input")
size = protocol.regular(file).st_size
total += size
if size <= 0 or total > protocol.MAX_TOTAL:
raise ValueError("empty/oversized compile input")
inputs[name] = {"sha256": protocol.digest(file), "bytes": size}
worker = Path(protocol.__file__).absolute()
root = Path(__file__).parent
controller_files = [root / name for name in ("salty_stage.py", "salty_client.py", "stageworker.py", "stagerun.py",
"pipeline.py", "stages/base.py", "stages/s05_compile.py", "stages/s06_hbm.py")]
code = {file.name: protocol.digest(file) for file in controller_files}
request = {"schema": protocol.SCHEMA, "identity": _identity(ctx), "image": policy["image"],
"image_source_sha256": policy["image_source_sha256"],
"image_runtime_contract_sha256": policy["image_runtime_contract_sha256"],
"verifier_sha256": policy["verifier_sha256"], "worker_sha256": protocol.digest(worker),
"controller_code_sha256": hashlib.sha256(protocol.canonical(code)).hexdigest(),
"config": protocol.config_parameters(config, ctx.job["march"], ctx.job["job_id"]), "inputs": inputs}
return request, sources, worker
def _bundle(path, request, sources):
with tarfile.open(path, "x:") as packed:
data = protocol.canonical(request)
info = tarfile.TarInfo("request.json")
info.size, info.mode, info.mtime = len(data), 0o600, 0
packed.addfile(info, io.BytesIO(data))
for name, source in sources.items():
info = tarfile.TarInfo(name)
info.size, info.mode, info.mtime = request["inputs"][name]["bytes"], 0o600, 0
with protocol.open_regular(source) as stream:
packed.addfile(info, stream)
if protocol.digest(source) != request["inputs"][name]["sha256"]:
raise ValueError("compile input changed during freeze")
protocol.read_manifest(path)
def _fetch(client, remote, destination):
receipt = client.receipt
prior = [row for row in receipt["downloads"] if row["remote_relative"] == remote]
if prior:
if (len(prior) != 1 or prior[0]["state"] != "COMPLETE" or prior[0]["destination"] != str(destination)
or not destination.is_file() or destination.is_symlink()
or destination.stat().st_size != prior[0]["bytes"] or protocol.digest(destination) != prior[0]["sha256"]):
raise AmbiguousOperation("previous stage download is incomplete or changed; no repeat")
return
client.fetch(remote, destination)
def _verified_result(path, request, bundle_sha):
if protocol.regular(path).st_size > 2 << 20:
raise ValueError("oversized stage result receipt")
result = json.loads(path.read_text())
for key in ("schema", "identity", "image", "image_source_sha256", "image_runtime_contract_sha256",
"worker_sha256", "controller_code_sha256", "verifier_sha256"):
if result.get(key) != request[key]:
raise ValueError("remote result identity differs: " + key)
if (result.get("request_sha256") != hashlib.sha256(protocol.canonical(request)).hexdigest()
or result.get("bundle_sha256") != bundle_sha or result.get("state") not in {"OK", "FAIL"}):
raise ValueError("remote result input/state identity mismatch")
if type(result.get("exit_code")) is not int or result["exit_code"] < 0:
raise ValueError("invalid remote compile exit status")
for key in ("model_info_exit_code", "verifier_exit_code"):
code = result.get(key)
if code is not None and (type(code) is not int or code < 0):
raise ValueError("invalid remote verification exit status")
if result["state"] == "OK" and (result["exit_code"] != 0 or result.get("model_info_exit_code") != 0
or result.get("march_read") != request["identity"]["march"]):
raise ValueError("claimed successful compile lacks target/tool evidence")
files = result.get("outputs")
if not isinstance(files, dict) or len(files) > protocol.MAX_FILES:
raise ValueError("invalid output manifest")
total = 0
for name, row in files.items():
protocol.relative(name)
if (Path(name).suffix not in {".hbm", ".bc", ".onnx", ".log"} or not isinstance(row, dict)
or set(row) != {"sha256", "bytes"} or not protocol.HEX.fullmatch(row.get("sha256", ""))
or type(row.get("bytes")) is not int or row["bytes"] < 0):
raise ValueError("invalid stage output descriptor")
total += row["bytes"]
if total > protocol.MAX_TOTAL:
raise ValueError("oversized compile results")
archive = result.get("archive") or {}
if (set(archive) != {"sha256", "bytes"} or not protocol.HEX.fullmatch(archive.get("sha256", ""))
or type(archive.get("bytes")) is not int or not 0 < archive["bytes"] <= protocol.MAX_TOTAL + (16 << 20)):
raise ValueError("invalid result archive identity")
if result["state"] == "OK" and (len([name for name in files if name.endswith(".hbm")]) != 1
or not any(name.endswith("quantized_model.bc") for name in files)
or not any(name.endswith("_ptq_model.onnx") for name in files)):
raise ValueError("successful PTQ result requires HBM, BC and quantized ONNX")
return result
def _install(archive, result, destination):
protocol.regular(archive)
if archive.stat().st_size != result["archive"]["bytes"] or protocol.digest(archive) != result["archive"]["sha256"]:
raise ValueError("downloaded result archive bytes differ")
destination.mkdir(parents=True, exist_ok=True)
for part in (destination, *destination.parents):
if part.is_symlink():
raise ValueError("result destination contains symlink")
with tempfile.TemporaryDirectory(prefix=".salty-output-", dir=destination) as temporary:
staging = Path(temporary)
with tarfile.open(archive, "r:") as packed:
members = []
for member in packed:
members.append(member)
if len(members) > protocol.MAX_FILES:
raise ValueError("too many output members")
if len(members) != len(result["outputs"]) or {m.name for m in members} != set(result["outputs"]):
raise ValueError("output archive members differ from receipt")
for member in members:
protocol.relative(member.name)
row = result["outputs"][member.name]
if not member.isfile() or member.size != row["bytes"]:
raise ValueError("special/changed stage output member")
path = staging / member.name
path.parent.mkdir(parents=True, exist_ok=True)
with packed.extractfile(member) as source, path.open("xb") as out:
shutil.copyfileobj(source, out)
if protocol.digest(path) != row["sha256"]:
raise ValueError("stage output bytes fail verification")
# Validate every output before publishing any byte. Existing identical
# bytes are crash-recovery only; an existing different file is preserved.
for name, row in result["outputs"].items():
path = destination / name
if path.exists() or path.is_symlink():
if protocol.regular(path).st_size != row["bytes"] or protocol.digest(path) != row["sha256"]:
raise ValueError("existing compile output differs; no overwrite")
for name in result["outputs"]:
source, path = staging / name, destination / name
if not path.exists():
path.parent.mkdir(parents=True, exist_ok=True)
for part in (path.parent, *path.parent.parents):
if part.is_symlink():
raise ValueError("result publication parent symlink")
os.link(source, path, follow_symlinks=False)
source.unlink()
def _check_installed(result, destination):
for name, row in result["outputs"].items():
file = destination / str(protocol.relative(name))
if protocol.regular(file).st_size != row["bytes"] or protocol.digest(file) != row["sha256"]:
raise ValueError("installed compile result changed")
def run(stage, ctx, stage_dir, *, runner=None, sleep=time.sleep, clock=time.monotonic):
from .stages.base import StageOutcome, work_dir
started = clock()
adapter = stage_dir / "salty"
with _lock(adapter):
try:
policy = _policy(ctx)
previous = pending_attempt(ctx.stage_root)
if previous is not None and previous != ctx.attempt:
raise AmbiguousOperation("prior Salty attempt unresolved; resume attempt %d, never submit anew" % previous)
ctx.extra.pop("salty_work_dir", None)
request, sources, worker = _request(ctx, policy)
output_directory = stage_dir / "salty" / "work"
plan_file = adapter / "plan.json"
bundle = adapter / "inputs.tar"
if plan_file.exists():
protocol.regular(plan_file)
plan = json.loads(plan_file.read_text())
if plan.get("request") != request or plan.get("work_dir") != str(output_directory):
raise ValueError("current task/input/code identity differs from saved stage attempt")
if plan.get("cleanup_required", True) != policy["cleanup_terminal"]:
raise ValueError("saved Salty attempt cleanup policy differs; reconcile original policy")
else:
if (adapter / "transport").exists() or bundle.exists():
raise AmbiguousOperation("partial stage preparation; preserve exact receipt and reconcile")
_bundle(bundle, request, sources)
plan = {"schema": protocol.SCHEMA, "request": request, "bundle_sha256": protocol.digest(bundle),
"work_dir": str(output_directory), "state": "PREPARED",
"cleanup_required": policy["cleanup_terminal"]}
_atomic(plan_file, plan)
transport = adapter / "transport"
if transport.exists():
client = SaltyClient(transport, runner=runner)
if client.receipt["image"] != request["image"]:
raise ValueError("stage/transport image identity differs")
else:
if plan["state"] != "PREPARED" or protocol.digest(bundle) != plan["bundle_sha256"]:
raise AmbiguousOperation("stage transport missing after dispatch; never resubmit")
client = SaltyClient.prepare(transport, image=request["image"], bundle=bundle, worker=worker,
command=["/opt/bench/ptq/bin/python", "-B", "{container_root}/worker.py",
"--bundle", "{container_root}/bundle.tar", "--bundle-sha256", plan["bundle_sha256"],
"--result-dir", "{container_root}/result"],
env={"PYTHONDONTWRITEBYTECODE": "1", "PYTHONUNBUFFERED": "1",
"OMP_NUM_THREADS": "1", "MKL_NUM_THREADS": "1", "OPENBLAS_NUM_THREADS": "1"},
salty=policy.get("salty", "salty"), cpu=policy.get("cpu"), memory=policy.get("memory"), runner=runner)
receipt = client.receipt
if receipt["phase"] == "PREPARED":
client.submit()
elif receipt["job_id"] is None:
client.recover_recorded_submission()
plan.update(state="RESULT_VERIFIED" if "verified_result" in plan else "DISPATCHED", job_id=client.receipt["job_id"])
_atomic(plan_file, plan)
if "verified_result" not in plan:
while client.poll()["job_state"] not in {"succeeded", "failed"}:
if clock() - started >= policy["wait_seconds"]:
raise AmbiguousOperation("stage wait budget reached; existing Salty job retained for continuation")
sleep(min(policy["poll_seconds"], max(0.001, policy["wait_seconds"] - (clock() - started))))
result_file = adapter / "result.json"
_fetch(client, "result/result.json", result_file)
result = _verified_result(result_file, request, plan["bundle_sha256"])
archive = adapter / "outputs.tar"
_fetch(client, "result/outputs.tar", archive)
_install(archive, result, output_directory)
plan.update(state="RESULT_VERIFIED", verified_result=result, result_sha256=protocol.digest(result_file))
_atomic(plan_file, plan)
else:
result = plan["verified_result"]
verified = _verified_result(adapter / "result.json", request, plan["bundle_sha256"])
if verified != result or protocol.digest(adapter / "result.json") != plan["result_sha256"]:
raise ValueError("persisted verified result changed")
_check_installed(result, output_directory)
if policy["cleanup_terminal"]:
client.cleanup(authorized=True)
for file in (bundle, adapter / "outputs.tar"):
if file.exists():
protocol.regular(file)
file.unlink()
plan["cleanup_complete"] = True
plan["state"] = "VERIFIED"
_atomic(plan_file, plan)
for kind in ("stdout", "stderr"):
source = output_directory / ("compile." + kind + ".log")
if source.exists():
shutil.copyfile(source, stage_dir / (kind + ".log"))
meta = {"executor": "salty", "salty_job_id": plan["job_id"], "salty_receipt": str(adapter / "plan.json"),
"image": request["image"], "input_identity_sha256": hashlib.sha256(protocol.canonical(request)).hexdigest(),
"work_dir": str(output_directory), "cleanup_complete": plan.get("cleanup_complete", False)}
ctx.extra["salty_work_dir"] = str(output_directory)
from .errors import classify
error_class = None
if result["state"] != "OK":
error_class = classify((stage_dir / "stderr.log").read_text(errors="replace"),
(stage_dir / "stdout.log").read_text(errors="replace"), result["exit_code"])
return StageOutcome(result["state"], result["exit_code"], error_class,
clock() - started, meta)
except (AmbiguousOperation, TransportError) as exc:
(stage_dir / "stderr.log").write_text(str(exc) + "\n")
# Legacy pipeline retries STALLED with a new attempt. FAIL plus
# SIDE_EFFECT_UNKNOWN preserves the lifecycle quarantine boundary.
return StageOutcome("FAIL", None, "SIDE_EFFECT_UNKNOWN", clock() - started,
{"executor": "salty", "salty_receipt": str(adapter / "plan.json"),
"cleanup_state": "QUARANTINED", "reconciliation_required": True})
def hbm(ctx, stage_dir):
"""06-hbm consumes verified 05 receipts and original parsers, never x86 tools."""
from .stages.base import StageOutcome, work_dir, recipe_receipt, recipe_output
from .stages import s06_hbm as local
attempts = sorted((ctx.stage_root / "05-compile").glob("a*"))
if not attempts:
return None
file = attempts[-1] / "salty/plan.json"
if not file.exists():
return None
if ctx.job.get("recipe"):
parent = recipe_receipt(ctx, "05-compile")
file = ctx.stage_root / "05-compile" / ("a%02d" % parent["attempt"]) / "salty/plan.json"
protocol.regular(file)
plan = json.loads(file.read_text())
if plan.get("state") != "VERIFIED" or not plan.get("verified_result"):
raise ValueError("latest remote compilation is unverified; 06-hbm cannot use an older attempt")
request, result = plan["request"], plan["verified_result"]
identity = _identity(ctx)
identity["attempt"] = request["identity"]["attempt"]
if request["identity"] != identity or result["state"] != "OK":
raise ValueError("06-hbm requires successful 05 receipt for this exact task")
verified = _verified_result(file.parent / "result.json", request, plan["bundle_sha256"])
if verified != result or protocol.digest(file.parent / "result.json") != plan["result_sha256"]:
raise ValueError("compile verification receipt changed")
wd = Path(plan["work_dir"])
_check_installed(result, wd)
hbms = [wd / name for name in result["outputs"] if name.endswith(".hbm")]
bcs = [wd / name for name in result["outputs"] if name.endswith("quantized_model.bc")]
if len(hbms) != 1 or len(bcs) != 1:
raise ValueError("06-hbm requires one verified HBM and BC")
ctx.artifacts.mkdir(parents=True, exist_ok=True)
dst, bc = local.hbm_path(ctx), local.bc_path(ctx)
for source, target in ((hbms[0], dst), (bcs[0], bc)):
if target.exists():
if protocol.digest(target) != protocol.digest(source):
raise ValueError("artifact exists with different bytes; no overwrite")
else:
with source.open("rb") as stream, target.open("xb") as out:
shutil.copyfileobj(stream, out)
info = "\n".join((wd / ("model_info." + kind + ".log")).read_text(errors="replace") for kind in ("stdout", "stderr"))
io_fields = local._parse_io(info)
verifier = "\n".join((wd / ("verifier." + kind + ".log")).read_text(errors="replace") for kind in ("stdout", "stderr")) if result.get("verifier_exit_code") is not None else ""
cosine = local._verifier_output_cosine(verifier)
compile_text = "\n".join((wd / ("compile." + kind + ".log")).read_text(errors="replace") for kind in ("stdout", "stderr"))
measurements = {"hbm_exists": True, "hbm_sha256": protocol.digest(dst), "model_info_ok": result["model_info_exit_code"] == 0,
"march_read": local._find_march(info), "march_ok": local._find_march(info) == ctx.job["march"],
"io": io_fields, "io_ok": local._check_io(io_fields, ctx.job["model_spec"]["input_spec"]),
"static_perf": local._parse_static_perf(info), "verifier_rc": result.get("verifier_exit_code"),
"verifier_cosine": cosine, "verifier_ok": result.get("verifier_exit_code") == 0 and (cosine is None or cosine >= ctx.job["gates"]["G1"].get("verifier_cosine_threshold", 0.99)),
"hbm_bc_cos_min": None, "hbm_bc_spotcheck": "not run; board HBM verification remains required",
"log_has_error": bool(re.search(r"\bERROR\b", compile_text)), "hmct_output_cosine": local._hmct_output_cosine(compile_text)}
for name in ("model_info.stdout.log", "model_info.stderr.log", "verifier.stdout.log", "verifier.stderr.log"):
if (wd / name).exists():
shutil.copyfile(wd / name, stage_dir / name)
(stage_dir / "stdout.log").write_text(json.dumps(measurements, indent=2) + "\n")
(stage_dir / "stderr.log").touch()
meta = {"hbm": str(dst), "hbm_sha256": protocol.digest(dst), "source_hbm": str(hbms[0]),
"size": dst.stat().st_size, "work_dir": str(wd), "executor": "salty-receipt",
"salty_job_id": plan["job_id"], "compile_receipt_sha256": protocol.digest(file)}
if ctx.job.get("recipe"):
meta["recipe_outputs"] = {"hbm": recipe_output(dst), "bc": recipe_output(bc)}
return StageOutcome("OK", 0, None, 0, meta, {"G1": measurements})

View File

@ -1,327 +0,0 @@
"""Standalone, credential-free PTQ worker uploaded with one frozen input bundle.
Only its CLI runs a compiler. Importing this module performs no runtime work.
No shell, timeout, process signalling, board access, or production state exists here.
"""
from __future__ import annotations
import argparse
from contextlib import contextmanager
import hashlib
import importlib.metadata
import json
import math
import os
from pathlib import Path, PurePosixPath
import re
import stat
import subprocess
import sys
import tarfile
import tempfile
import time
SCHEMA = "bench-salty-ptq-stage.v1"
HEX = re.compile(r"[0-9a-f]{64}")
MAX_FILES = 4096
MAX_TOTAL = 8 * 1024 ** 3
CONFIG_FIELDS = {
"model_parameters": {"march", "output_model_file_prefix", "layer_out_dump"},
"input_parameters": {"input_name", "input_shape", "input_type_rt", "input_type_train",
"input_layout_rt", "input_layout_train", "mean_value", "scale_value", "input_batch"},
"calibration_parameters": {"calibration_type", "num_bin", "max_percentile", "per_channel", "quant_config"},
"compiler_parameters": {"compile_mode", "optimize_level", "debug", "jobs"},
}
def canonical(value):
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode()
def digest(path):
h = hashlib.sha256()
with open_regular(path) as stream:
for block in iter(lambda: stream.read(1 << 20), b""):
h.update(block)
return h.hexdigest()
@contextmanager
def open_regular(path):
"""Walk with openat/O_NOFOLLOW so a racing leaf/parent cannot redirect IO."""
path = Path(os.path.abspath(path))
descriptor = os.open("/", os.O_RDONLY | os.O_DIRECTORY)
try:
for part in path.parts[1:-1]:
next_fd = os.open(part, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, dir_fd=descriptor)
os.close(descriptor)
descriptor = next_fd
fd = os.open(path.name, os.O_RDONLY | os.O_NOFOLLOW, dir_fd=descriptor)
try:
info = os.fstat(fd)
if not stat.S_ISREG(info.st_mode) or info.st_nlink != 1:
raise ValueError("input/output must be an independent regular file")
with os.fdopen(fd, "rb", closefd=False) as stream:
yield stream
finally:
os.close(fd)
finally:
os.close(descriptor)
def relative(name):
if not isinstance(name, str) or not re.fullmatch(r"[A-Za-z0-9_./-]+", name):
raise ValueError("noncanonical bundle/output name")
p = PurePosixPath(name)
if p.is_absolute() or any(part.startswith(".") for part in p.parts) or p.as_posix() != name or not p.parts:
raise ValueError("bundle/output path traversal")
return p
def regular(path):
path = Path(os.path.abspath(path))
for part in (path, *path.parents):
if part.is_symlink():
raise ValueError("symlink in input/output path")
info = path.stat()
if not stat.S_ISREG(info.st_mode) or info.st_nlink != 1:
raise ValueError("input/output must be an independent regular file")
return info
def config_parameters(config, march, prefix):
if not isinstance(config, dict) or set(config) != set(CONFIG_FIELDS):
raise ValueError("unsupported PTQ configuration sections")
clean = {}
for section, fields in CONFIG_FIELDS.items():
values = config[section]
if not isinstance(values, dict) or set(values) - fields:
raise ValueError("unsupported PTQ parameters: " + section)
clean[section] = values
def check(value, depth=0):
if depth > 12:
raise ValueError("PTQ parameter nesting too deep")
if isinstance(value, dict):
for key, val in value.items():
if (not isinstance(key, str) or not re.fullmatch(r"[A-Za-z0-9_.:/-]{1,200}", key)
or any(word in key.lower() for word in ("credential", "secret", "password", "token", "api_key"))):
raise ValueError("unsafe PTQ parameter key")
check(val, depth + 1)
elif isinstance(value, list):
if len(value) > 1024:
raise ValueError("PTQ parameter list too long")
for val in value:
check(val, depth + 1)
elif isinstance(value, str):
if (len(value) > 4096 or "\x00" in value or "://" in value or ".." in value
or value.startswith(("/", "~")) or "PRIVATE KEY" in value):
raise ValueError("external path/credential-like PTQ parameter")
elif value is not None and type(value) not in (bool, int, float):
raise ValueError("unsupported PTQ parameter value")
elif isinstance(value, float) and not math.isfinite(value):
raise ValueError("nonfinite PTQ parameter")
check(clean)
if march not in {"nash-e", "nash-m", "nash-p"} or not re.fullmatch(r"[A-Za-z0-9_-]{1,128}", prefix):
raise ValueError("invalid march/output prefix")
if (clean["model_parameters"].get("march") != march
or clean["model_parameters"].get("output_model_file_prefix") != prefix):
raise ValueError("PTQ configuration target/task mismatch")
if clean["model_parameters"].get("layer_out_dump", False) is not False:
raise ValueError("remote stage does not return layer dumps")
# Resource use is controlled by the outer job, never agent-supplied compiler concurrency.
clean = json.loads(json.dumps(clean))
clean["compiler_parameters"]["jobs"] = 1
return clean
def read_manifest(bundle):
with tarfile.open(bundle, "r:") as packed:
members = []
for member in packed:
members.append(member)
if len(members) > MAX_FILES + 2:
raise ValueError("too many input members")
names = [member.name for member in members]
if len(set(names)) != len(names) or "request.json" not in names:
raise ValueError("duplicate or missing input manifest")
for member in members:
relative(member.name)
if not member.isfile() or not 0 <= member.size <= MAX_TOTAL:
raise ValueError("input archive contains special/oversized members")
member = packed.getmember("request.json")
if member.size > 2 << 20:
raise ValueError("oversized input manifest")
request = json.loads(packed.extractfile(member).read())
if request.get("schema") != SCHEMA or set(request) != {
"schema", "identity", "image", "image_source_sha256", "image_runtime_contract_sha256",
"worker_sha256", "controller_code_sha256", "verifier_sha256", "config", "inputs"}:
raise ValueError("invalid input manifest schema/fields")
for key in ("image_source_sha256", "image_runtime_contract_sha256", "worker_sha256", "controller_code_sha256", "verifier_sha256"):
if not isinstance(request[key], str) or not HEX.fullmatch(request[key]):
raise ValueError("invalid code/runtime identity")
if not isinstance(request["image"], str) or not re.fullmatch(r"[A-Za-z0-9._:/-]+@sha256:[0-9a-f]{64}", request["image"]):
raise ValueError("image digest required")
identity = request["identity"]
if not isinstance(identity, dict) or set(identity) != {
"run_id", "job_id", "rollout", "attempt", "task", "backend", "march", "qcfg", "recipe_submission"}:
raise ValueError("invalid task identity")
for key in ("run_id", "job_id", "task", "qcfg"):
if not isinstance(identity[key], str) or not re.fullmatch(r"[A-Za-z0-9_-]{1,128}", identity[key]):
raise ValueError("invalid task identity field")
if (identity["backend"] != "ptq" or type(identity["attempt"]) is not int or identity["attempt"] < 0
or type(identity["rollout"]) is not int or identity["rollout"] < -1
or (identity["recipe_submission"] is not None and not HEX.fullmatch(identity["recipe_submission"]))):
raise ValueError("unsupported stage identity")
request["config"] = config_parameters(request["config"], identity["march"], identity["job_id"])
inputs = request["inputs"]
if not isinstance(inputs, dict) or "input/model.onnx" not in inputs:
raise ValueError("missing ONNX input")
if not 2 <= len(inputs) <= MAX_FILES + 1 or set(names) != {"request.json", *inputs}:
raise ValueError("input archive/manifest members differ")
if sum(packed.getmember(name).size for name in inputs) > MAX_TOTAL:
raise ValueError("oversized input bundle")
for name, row in inputs.items():
if name != "input/model.onnx" and not re.fullmatch(r"input/calibration/[0-9]{6}\.npy", name):
raise ValueError("unapproved model/calibration input name")
if (not isinstance(row, dict) or set(row) != {"sha256", "bytes"}
or not isinstance(row["sha256"], str) or not HEX.fullmatch(row["sha256"])
or type(row["bytes"]) is not int or row["bytes"] <= 0
or packed.getmember(name).size != row["bytes"]):
raise ValueError("input size/hash descriptor mismatch")
return request
def unpack_inputs(bundle, request, destination):
with tarfile.open(bundle, "r:") as packed:
for name, row in request["inputs"].items():
target = destination / str(relative(name))
target.parent.mkdir(parents=True, exist_ok=True)
h, count = hashlib.sha256(), 0
with packed.extractfile(name) as source, target.open("xb") as out:
for chunk in iter(lambda: source.read(1 << 20), b""):
count += len(chunk)
h.update(chunk)
out.write(chunk)
if count != row["bytes"] or h.hexdigest() != row["sha256"]:
raise ValueError("input bytes differ from frozen manifest")
def runtime_identity(request):
root = Path("/opt/bench")
release = json.loads((root / "RELEASE_MANIFEST.json").read_text())
caps = json.loads((root / "COMPILE_CAPABILITIES.json").read_text())
contract = caps["runtime_contract"]
if (release["source_files_sha256"] != request["image_source_sha256"]
or hashlib.sha256(canonical(contract)).hexdigest() != request["image_runtime_contract_sha256"]
or release["runtime_contract_sha256"] != request["image_runtime_contract_sha256"]
or caps.get("toolchain_prefix") != "/opt/bench/ptq"):
raise ValueError("image source/toolchain identity changed")
if ".".join(map(str, sys.version_info[:3])) != contract["python_version"]:
raise ValueError("runtime Python version changed")
for package, version in contract["packages"].items():
if importlib.metadata.version(package) != version:
raise ValueError("runtime package changed: " + package)
for name, row in contract["commands"].items():
if digest(row["path"]) != row["sha256"]:
raise ValueError("runtime compiler command changed: " + name)
if digest("/usr/local/bin/hb_verifier") != request["verifier_sha256"]:
raise ValueError("runtime verifier command changed")
return contract
def execute(bundle, result_dir, expected_bundle_sha256):
regular(bundle)
if digest(bundle) != expected_bundle_sha256:
raise ValueError("uploaded bundle identity changed")
request = read_manifest(bundle)
if digest(__file__) != request["worker_sha256"]:
raise ValueError("uploaded worker code changed")
contract = runtime_identity(request)
result_dir.mkdir(parents=False, exist_ok=False)
started = time.time()
# Temporary compiler intermediates are isolated from the personal mount.
with tempfile.TemporaryDirectory(prefix="bench-salty-ptq-") as name:
temp = Path(name)
unpack_inputs(bundle, request, temp)
import onnx
import yaml
model = onnx.load(str(temp / "input/model.onnx"), load_external_data=False)
if any(tensor.external_data or tensor.data_location == onnx.TensorProto.EXTERNAL
for tensor in onnx.external_data_helper._get_all_tensors(model)):
raise ValueError("external ONNX tensor references are unsupported")
work = temp / "work"
work.mkdir()
config = request["config"]
config["model_parameters"].update(onnx_model=str(temp / "input/model.onnx"), working_dir=str(work))
config["calibration_parameters"]["cal_data_dir"] = str(temp / "input/calibration")
config_file = temp / "compile.yaml"
config_file.write_text(yaml.safe_dump(config, sort_keys=False))
env = {"PATH": "/usr/local/bin:/usr/bin:/bin", "HOME": str(temp), "LANG": "C.UTF-8",
"PYTHONDONTWRITEBYTECODE": "1", "PYTHONUNBUFFERED": "1",
**{key: "1" for key in ("OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS", "NUMEXPR_NUM_THREADS")}}
def run(argv, label):
with (work / (label + ".stdout.log")).open("xb") as stdout, (work / (label + ".stderr.log")).open("xb") as stderr:
return subprocess.Popen(argv, cwd=work, env=env, stdout=stdout, stderr=stderr).wait()
rc = run([contract["commands"]["hb_compile"]["path"], "-c", str(config_file)], "compile")
hbms = sorted(work.rglob("*.hbm"))
info_rc, march_read = None, None
if rc == 0 and len(hbms) == 1 and regular(hbms[0]).st_size > 0:
info_rc = run([contract["commands"]["hb_model_info"]["path"], str(hbms[0])], "model_info")
text = "\n".join((work / ("model_info." + kind + ".log")).read_text(errors="replace") for kind in ("stdout", "stderr"))
targets = set(re.findall(r"\bnash-[emp]\b", text))
if len(targets) == 1:
march_read = targets.pop()
bcs = sorted(work.rglob("*quantized_model.bc"))
onnx_outputs = sorted(work.rglob("*_ptq_model.onnx"))
verifier_rc = None
if rc == 0 and len(bcs) == 1 and len(onnx_outputs) == 1:
sample = sorted((temp / "input/calibration").glob("*.npy"))[0]
verifier_rc = run(["/usr/local/bin/hb_verifier", "-m", str(onnx_outputs[0]) + "," + str(bcs[0]),
"-i", str(sample)], "verifier")
selected = set(hbms)
selected.update(work.rglob("*quantized_model.bc"))
selected.update(work.rglob("*_ptq_model.onnx"))
selected.update(work.glob("*.log"))
outputs, total = {}, 0
for path in sorted(selected):
row = {"sha256": digest(path), "bytes": regular(path).st_size}
total += row["bytes"]
if total > MAX_TOTAL:
raise ValueError("oversized compile results")
outputs[path.relative_to(work).as_posix()] = row
archive = result_dir / "outputs.tar"
with tarfile.open(archive, "x:") as packed:
for path in sorted(selected):
relative(path.relative_to(work).as_posix())
packed.add(path, arcname=path.relative_to(work).as_posix(), recursive=False)
ok = rc == 0 and info_rc == 0 and march_read == request["identity"]["march"] and len(hbms) == 1
result = {"schema": SCHEMA, "request_sha256": hashlib.sha256(canonical(read_manifest(bundle))).hexdigest(),
"bundle_sha256": expected_bundle_sha256, "identity": request["identity"], "image": request["image"],
"worker_sha256": request["worker_sha256"], "controller_code_sha256": request["controller_code_sha256"],
"image_source_sha256": request["image_source_sha256"],
"image_runtime_contract_sha256": request["image_runtime_contract_sha256"],
"verifier_sha256": request["verifier_sha256"], "verifier_exit_code": verifier_rc,
"state": "OK" if ok else "FAIL", "exit_code": rc, "model_info_exit_code": info_rc,
"march_read": march_read, "duration_s": time.time() - started, "outputs": outputs,
"archive": {"sha256": digest(archive), "bytes": archive.stat().st_size}}
(result_dir / "result.json").write_text(json.dumps(result, indent=2) + "\n")
return 0 if ok else 1
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--bundle", type=Path, required=True)
parser.add_argument("--bundle-sha256", required=True)
parser.add_argument("--result-dir", type=Path, required=True)
args = parser.parse_args()
if not HEX.fullmatch(args.bundle_sha256):
raise ValueError("invalid bundle SHA")
parent = args.result_dir.parent
if (not re.fullmatch(r"/data/personal/bench-stage-[0-9a-f-]{36}", str(parent))
or args.result_dir.name != "result" or args.bundle != parent / "bundle.tar"):
raise ValueError("remote attempt/result ownership guard failed")
return execute(args.bundle, args.result_dir, args.bundle_sha256)
if __name__ == "__main__":
raise SystemExit(main())

View File

@ -52,7 +52,7 @@ def _try_slots(pool: str, order: list[int]):
try:
fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
lease = d / f"{pool}-{i}.lease.json"
if lease.exists() and not (pool != "board" and _lease_settled(lease)):
if lease.exists():
# A flock disappearing after a crash is not proof of remote/local cleanup.
fcntl.flock(f, fcntl.LOCK_UN)
f.close()
@ -76,19 +76,6 @@ def _try_slots(pool: str, order: list[int]):
return None, None
def _lease_settled(lease: Path) -> bool:
"""A stage killed at its phase deadline dies without releasing its slot; the stage worker has
since recorded its process cleanup as verified. Before 10-13 these leaked until a collector
restart (26 of 32 cpu slots in one night)."""
try:
x = json.loads(lease.read_text())
status = json.loads((Path(x["execution_dir"]) / "stages" / "_queue" / x["operation_id"] / "status.json").read_text())
except (OSError, ValueError, KeyError, TypeError):
return False
return (execution.process_status(x["holder"]) == "DEAD" and status.get("state") == "DONE"
and status.get("cleanup_state") == "VERIFIED")
def _queue_budget(timeout_s):
current = execution.current()
if timeout_s is None:
@ -166,7 +153,7 @@ def acquire(pool: str, poll_s: float = 2.0, lane: str = "normal", march: str | N
started, timeout_s = _queue_budget(timeout_s)
while fh is None:
_check_wait(started, timeout_s, stop_check)
# paths.config() 每个进程只读一次(lru_cache):改 workers / reserve_for_agent 只对之后新起的进程生效
# 每轮重读 config:排队中的进程也能吃到 workers / reserve_for_agent 的改动
# (踩过:cpu 8→10 之后,已在排队的 agent stagerun 还盯着旧的 8 个槽,白等)
n, reserve = pool_size(pool), pool_reserve(pool)
order = (list(range(reserve)) + list(range(reserve, n))) if lane == "agent" else list(range(reserve, n))

View File

@ -2,7 +2,7 @@
agent 不在 bash 里直接跑 hb_compile。它提交:
bench stage run compile --config workspace/compile.yaml → 打印 handle
terminal_exec({"wait_handle": <handle>}) → 一次等到结束(bench stage wait 只看一眼)
bench stage wait <handle> --poll 60 → 最多等待60秒;完成打印结果,未完成返回2并提示继续等待
harness 在 agent 进程外执行阶段:按 backend 选 venv、拿资源池、看门狗、日志落 rollout/stages/。
四个好处:编译时长与 pi 的 bash 超时脱钩;venv 不可能选错;预算与看门狗一视同仁;日志位置统一。
@ -21,7 +21,6 @@ import uuid
from pathlib import Path
from . import paths, db, scheduler, execution as lifecycle
from .salty_client import AmbiguousOperation
from .stages import BY_NAME, AGENT_RUNNABLE, StageContext, short
@ -32,25 +31,12 @@ def rollout_dir() -> Path:
return Path(d).resolve()
OWN_INPUTS_FROM = 18 # agent/VERSION from which the graph and the calibration set must be the agent's own
def own_inputs_required(rd: Path) -> bool:
"""Rollouts started under agent v18+: no platform graph or calibration fallback (04 rejects, bench dev hides them).
Earlier rollouts keep the rules they started with."""
try:
return int(json.loads((rd / "rollout.json").read_text()).get("agent_config_version", 0)) >= OWN_INPUTS_FROM
except (OSError, ValueError):
return False
def _ctx_from_rollout(rd: Path, agent_config: Path | None) -> StageContext:
jd = rd.parent.parent
job = json.loads((jd / "job.json").read_text())
kk = int(rd.name)
return StageContext(run_id=job["run_id"], job=job, job_dir=jd, stage_root=rd / "stages",
artifacts=rd / "workspace" / "artifacts", rollout=kk, agent_config=agent_config,
extra={"own_inputs_required": own_inputs_required(rd)})
artifacts=rd / "workspace" / "artifacts", rollout=kk, agent_config=agent_config)
def submit(stage: str, config: str | None, poll_hint: int = 60, *, recipe=None, submission=None) -> str:
@ -101,7 +87,7 @@ def submit(stage: str, config: str | None, poll_hint: int = 60, *, recipe=None,
if execution:
execution.register_locked(state, proc, "stage", scope_token)
print(hid)
print('# wait: terminal_exec(' + json.dumps({'wait_handle': hid}) + ')')
print(f"# wait: bench stage wait {hid} --poll {poll_hint}")
return hid
@ -121,7 +107,6 @@ def _completion_status(ctx, name, out):
outputs = {'compile_yaml': recipe_output(ctx.stage_root / name / 'compile.yaml')}
return dict(state='DONE', stage=name, result=out.state, exit_code=out.exit_code,
error_class=out.error_class, duration_s=round(out.duration_s, 1), attempt=ctx.attempt,
cleanup_state=(getattr(out, 'meta', None) or {}).get('cleanup_state', 'VERIFIED'),
stdout=str(sd / 'stdout.log'), stderr=str(sd / 'stderr.log'), meta=str(sd / 'meta.json'),
measurements=out.measurements, finished_at=meta['finished_at'],
meta_sha256=hashlib.sha256(raw).hexdigest(), outputs=outputs, db_registered=False,
@ -129,20 +114,13 @@ def _completion_status(ctx, name, out):
MAX_COMPILE_ATTEMPTS = 5 # 每条轨迹的主预算;改这里要同步改 agent/prompts/e2e.md 里的次数
# 容器被杀(多是内存超限,下次自动加倍)、执行器退出、超过阶段时限、看门狗判卡死(一小时没日志没 CPU):
# 系统的错,不扣模型的次数;每条轨迹最多免这么多次,防止一个怎么都编不过的模型无限重试。
FREE_INFRA_COMPILES = 3
INFRA_COMPILE_ERRORS = ('REMOTE_WORKER_FAILED', 'STAGE_EXECUTOR_EXIT', 'PHASE_EXECUTION_DEADLINE_EXCEEDED', 'stalled')
def compile_attempts_used(con, run_id, job_id, rollout) -> int:
"""Compiles that got a slot; a compile that timed out in the queue never ran and is free.
Read from the stage ledger, so resume/recovery cannot reset it."""
total, infra = con.execute(
f"SELECT COUNT(*), COALESCE(SUM(error_class IN ({','.join('?' * len(INFRA_COMPILE_ERRORS))})),0) FROM stages "
"WHERE run_id=? AND job_id=? AND rollout=? AND stage='05-compile' AND lane!='queued'",
(*INFRA_COMPILE_ERRORS, run_id, job_id, rollout)).fetchone()
return total - min(infra, FREE_INFRA_COMPILES)
return con.execute("SELECT COUNT(*) FROM stages WHERE run_id=? AND job_id=? AND rollout=? "
"AND stage='05-compile' AND lane!='queued'", (run_id, job_id, rollout)).fetchone()[0]
def _exec_impl(hd: Path, *, rd=None, publish_status=None, protected=False) -> int:
@ -197,16 +175,6 @@ def _exec_impl(hd: Path, *, rd=None, publish_status=None, protected=False) -> in
# attempt = 该 rollout 内此阶段已提交次数(agent 自己的重试是同一条轨迹)
prev = sorted((rd / "stages" / stage.name).glob("a*")) if (rd / "stages" / stage.name).exists() else []
ctx.attempt = len(prev)
pending_salty = None
from . import salty_stage
if salty_stage.selected(ctx, stage.name):
pending_salty = salty_stage.pending_attempt(ctx.stage_root)
if pending_salty is not None:
ctx.attempt = pending_salty # Continue the exact durable job ID, never a new attempt.
if ctx.agent_config:
configs = sorted((ctx.stage_root / '04-calibrate').glob('a*/agent_config.yaml'))
if not configs or hashlib.sha256(configs[-1].read_bytes()).hexdigest() != hashlib.sha256(ctx.agent_config.read_bytes()).hexdigest():
raise AmbiguousOperation('unresolved Salty attempt cannot accept new compiler overrides')
# 05-compile 一律走 long 车道(stall_s 3600)。hbdk4 的 BPU 编译阶段会长时间不打日志:
# 实测 convnext.tiny 编译 3342s 才成功,而 900s 的 stall 判定把同一个 job 的另外两条 rollout
# 在正常编译中途 SIGTERM 掉了(记成 error_class=stalled,白烧一条轨迹)。
@ -217,10 +185,7 @@ def _exec_impl(hd: Path, *, rd=None, publish_status=None, protected=False) -> in
limit = MAX_COMPILE_ATTEMPTS
with db.connect() as con:
used = compile_attempts_used(con, ctx.run_id, ctx.job["job_id"], ctx.rollout)
# 已记录过提交的 Salty 编译只做找回、绝不重提,它早已计入 used,不受上限阻拦;
# 只准备好、从未提交的仍按上限拒绝。
recovering = pending_salty is not None and salty_stage.dispatch_recorded(ctx.stage_root, pending_salty)
if used >= limit and not recovering:
if used >= limit:
put_status({"state": "DONE", "result": "FAIL", "exit_code": 2, "error_class": "COMPILE_BUDGET_EXHAUSTED",
"duration_s": 0, "stdout": "", "stderr": "", "meta": "", "finished_at": time.time(),
"note": f"本轨迹已启动 {used}/{limit} 次编译,不再编译;用已有产物完成 board/eval 并如实报告"})
@ -231,7 +196,7 @@ def _exec_impl(hd: Path, *, rd=None, publish_status=None, protected=False) -> in
# (v4/v5 两条 canary 都踩了)。04 对 PTQ 只是"合并 yaml + 落盘"(2 s),这里内联补跑:
# 带了 --config 就按新配置重新合并;没带且 04 没跑过也补一次默认合并。
yp = rd / "stages" / "04-calibrate" / "compile.yaml"
if pending_salty is None and ctx.job.get("backend") == "ptq" and (ctx.job.get("recipe") or ctx.agent_config or not yp.exists()):
if ctx.job.get("backend") == "ptq" and (ctx.job.get("recipe") or ctx.agent_config or not yp.exists()):
cal = BY_NAME["04-calibrate"]()
cprev = sorted((rd / "stages" / cal.name).glob("a*")) if (rd / "stages" / cal.name).exists() else []
cctx = _ctx_from_rollout(rd, ctx.agent_config)
@ -301,7 +266,6 @@ def _exec_impl(hd: Path, *, rd=None, publish_status=None, protected=False) -> in
lifecycle.current().quarantine("BOARD_CLEANUP_UNCONFIRMED")
sd = rd / "stages" / stage.name / f"a{ctx.attempt:02d}"
status = {"state": "DONE", "result": out.state, "exit_code": out.exit_code, "error_class": out.error_class,
"cleanup_state": (getattr(out, 'meta', None) or {}).get('cleanup_state', 'VERIFIED'),
"duration_s": round(out.duration_s, 1), "attempt": ctx.attempt,
"board_name": (ctx.extra.get("board") or {}).get("name"),
"stdout": str(sd / "stdout.log"), "stderr": str(sd / "stderr.log"), "meta": str(sd / "meta.json"),
@ -323,38 +287,51 @@ def _exec(hd: Path, *, rd=None, publish_status=None, protected=False) -> int:
try:
with execution.scope(req["stage"], hd.name) if execution else nullcontext():
return _exec_impl(hd, rd=rd, publish_status=publish_status, protected=protected)
except (lifecycle.ExecutionStopped, AmbiguousOperation) as exc:
unknown = isinstance(exc, AmbiguousOperation)
status = {"state": "DONE", "result": "FAIL", "exit_code": 2,
"error_class": "SIDE_EFFECT_UNKNOWN" if unknown else str(exc),
except lifecycle.ExecutionStopped as exc:
status = {"state": "DONE", "result": "FAIL", "exit_code": 2, "error_class": str(exc),
"duration_s": 0, "stdout": "", "stderr": "", "meta": "", "finished_at": time.time()}
if unknown:
status.update(cleanup_state='QUARANTINED', reconciliation_required=True)
with execution.mutation() if execution else nullcontext():
lifecycle.atomic_json(hd / "status.json", status)
if publish_status:
publish_status(status)
if unknown and execution:
execution.quarantine('SIDE_EFFECT_UNKNOWN')
return 2
def wait(handle: str, poll: int = 60) -> int:
"""不再轮询,看一眼就返回:结束了打印结果;没结束返回 2,并指向 terminal_exec wait_handle(一次等到阶段结束、不占轮数)。
10-16:efficientvit.b0 400 轮里 307 轮是 bench stage wait,每次还往上下文里加一段输出。"""
def wait(handle: str, poll: int = 60, quiet: bool = False, max_wait_s: float = 60) -> int:
"""单次最多等 60 秒;返回 2 表示仍在排队/运行,阶段继续执行。"""
if not 0 < max_wait_s <= 60:
raise ValueError("stage wait budget must be in (0, 60] seconds")
hd = rollout_dir() / "handles" / handle
if not hd.exists():
raise SystemExit(f"handle 不存在:{handle}")
st = json.loads((hd / "status.json").read_text())
if st.get("state") != "DONE":
print(f"{handle}: {st.get('state')}"
f"{'(阶段计算已结束,等待数据库登记;无需重跑)' if st.get('registration') == 'PENDING' else ''}"
f"。不要反复查:调用 terminal_exec({json.dumps({'wait_handle': handle})}),它一次等到这个阶段结束,不占轮数。")
return 2
t0 = time.time()
deadline = time.monotonic() + max_wait_s
while True:
st = json.loads((hd / "status.json").read_text())
if st.get("state") == "DONE":
break
if not quiet:
since = st.get("started_at") or st.get("queued_at") or t0
print(f"[{time.strftime('%H:%M:%S')}] {handle}: {st.get('state')}"
f"{'(等 ' + st.get('pool', '?') + ' 池槽位)' if st.get('state') == 'QUEUED' else ''}"
f"{'(阶段计算已结束,等待数据库登记;无需重跑)' if st.get('registration') == 'PENDING' else ''}"
f" 已 {int(time.time() - since)}s", flush=True)
remaining = deadline - time.monotonic()
if remaining <= 0:
print(f"(已等 {int(max_wait_s - remaining)}s 仍未结束;这不是失败,再调一次 bench stage wait {handle} 继续等)")
return 2
time.sleep(min(max(1, poll), remaining))
_print_status(st)
return 0 if st["result"] in ("OK", "SKIPPED") else 1
def status(handle: str) -> int:
hd = rollout_dir() / "handles" / handle
st = json.loads((hd / "status.json").read_text())
_print_status(st)
return 0
def _print_status(st: dict) -> None:
if st.get("state") != "DONE":
print(json.dumps(st)); return

View File

@ -54,8 +54,6 @@ def work_dir(ctx: StageContext) -> Path:
相等跳过自拷日志(horizon_tc_ui/hb_compile.py:50),不等会在编译成功后删掉日志再抛错。
叶子名暂留 05-compile-work:在飞 job 已按此路径写完,上移一层会让 s06_hbm 的 hb_compile.log 落空。
"""
if ctx.extra.get("salty_work_dir"):
return Path(ctx.extra["salty_work_dir"])
rel = f"{ctx.run_id}/{ctx.job['job_id']}" + (f"/r{ctx.rollout:02d}" if ctx.rollout >= 0 else "/oracle")
d = paths.expand(paths.config()["paths"]["work_root"]) / rel / "05-compile-work"
if ctx.job.get("recipe"):
@ -233,13 +231,8 @@ class Stage:
return d
def run(self, ctx: StageContext) -> StageOutcome:
from .. import salty_remote
if salty_remote.selected(ctx, self.name):
return salty_remote.run(self, ctx)
sd = self.stage_dir(ctx)
from .. import salty_stage
remote = salty_stage.selected(ctx, self.name)
if ctx.job.get("recipe") and (sd / "meta.json").exists() and not (remote and (sd / "salty/plan.json").exists()):
if ctx.job.get("recipe") and (sd / "meta.json").exists():
raise ValueError("recipe stage attempt already exists; submit a new attempt")
t0 = time.time()
meta = {"stage": self.name, "level": self.level, "pool": self.pool, "attempt": ctx.attempt,
@ -256,8 +249,8 @@ class Stage:
meta["board"] = {k: ctx.extra["board"].get(k) for k in ("name", "slot", "march", "board_type")}
try:
if ctx.job.get("task_contract"):
from ..eval import contract_holds, task_contract
if not contract_holds(ctx.job["task_contract"], task_contract(ctx.job["task"])):
from ..eval import task_contract
if task_contract(ctx.job["task"]) != ctx.job["task_contract"]:
raise ValueError("frozen task implementation changed; create a new run")
if ctx.job.get("toolchain"):
from ..toolchains import validate
@ -276,12 +269,8 @@ class Stage:
would = f"<command unavailable in dry-run: {e}>"
out = StageOutcome("OK", 0, None, 0.0, {"dry_run": True, "would_run": would})
else:
if remote:
ctx.extra.pop("salty_work_dir", None)
cmd = self.command(ctx)
if remote:
out = salty_stage.run(self, ctx, sd)
elif cmd is None:
if cmd is None:
out = self.native(ctx, sd)
else:
wd = ctx.watchdog_params()
@ -306,23 +295,7 @@ class Stage:
except Exception as e: # harness 自身的 bug 也要落盘,不能吞
with (sd / "stderr.log").open("a") as log:
log.write(f"\n[harness exception] {type(e).__name__}: {e}\n")
uncertain_error = getattr(e, "error_class", None) == "SIDE_EFFECT_UNKNOWN" or getattr(e, "cleanup_state", None) == "QUARANTINED"
unresolved = uncertain_error or remote and (sd / "salty/transport").exists()
out = StageOutcome("FAIL", -1, "SIDE_EFFECT_UNKNOWN" if unresolved else "unknown", time.time() - t0,
{"harness_exception": repr(e), **({"cleanup_state": "QUARANTINED",
**({"salty_receipt": str(sd / "salty/plan.json")} if remote else {}),
**({"process_receipt": str(e.receipt_path)} if hasattr(e, "receipt_path") else {})} if unresolved else {})})
if out.error_class == "SIDE_EFFECT_UNKNOWN" or out.meta.get("cleanup_state") == "QUARANTINED":
from .. import execution
if execution.current():
execution.current().quarantine("SIDE_EFFECT_UNKNOWN")
elif (self.pool == "board" and not remote and out.state != "SKIPPED" and os.environ.get("BENCH_BOARD_RECEIPT_DIR")
and not any(Path(os.environ["BENCH_BOARD_RECEIPT_DIR"]).glob("*.json"))):
# A board session writes an UNKNOWN receipt before its first board command (board_perf.bench) and this
# stage's processes are gone now: no receipt proves the board was never touched. Without one, a failure
# before dispatch read as BOARD_CLEANUP_UNCONFIRMED and ended the rollout (10-15: two lost that way).
from ..tools.board_perf import not_started
not_started("STAGE_ENDED_BEFORE_BOARD_DISPATCH")
out = StageOutcome("FAIL", -1, "unknown", time.time() - t0, {"harness_exception": repr(e)})
out.duration_s = out.duration_s or (time.time() - t0)
meta.update({"state": out.state, "exit_code": out.exit_code, "error_class": out.error_class,
"duration_s": round(out.duration_s, 3), "finished_at": time.time(),

View File

@ -29,8 +29,8 @@ def baseline_identity(job: dict) -> dict | None:
def validate_baseline(job: dict) -> None:
if paths.cache_contract(job, validate=True) is not None:
from ..eval import contract_holds, task_contract
if not contract_holds(job.get("task_contract") or {}, task_contract(job["task"])):
from ..eval import task_contract
if task_contract(job["task"]) != job.get("task_contract"):
raise ValueError("frozen task implementation changed; create a new run")

View File

@ -5,7 +5,6 @@ rollout 里 agent 提交的 --config 在这里与 harness 钉死项合并。
"""
from __future__ import annotations
import hashlib
import json
import shlex
from pathlib import Path
@ -108,22 +107,10 @@ class Calibrate(Stage):
meta = {"calib_dir": str(cd), "calib_npy_dir": str(calib_npy_dir(cd)), "calib_source": source,
"calib_n": manifest.get("count"), "calib_manifest_sha": manifest.get("sha256")}
own = {}
if (agent_overrides or ctx.extra.get("own_inputs_required")) and not job.get("recipe") and job["backend"] == "ptq":
try:
own = own_inputs(ctx, agent_overrides or {}, onnx)
except (OSError, ValueError) as exc:
(stage_dir / "stderr.log").write_text(f"{exc}\n")
(stage_dir / "stdout.log").touch()
return StageOutcome("FAIL", 2, "own_input_invalid", 0.0, {"reason": str(exc)[:500]})
onnx = own.get("onnx", onnx)
meta.update(own.get("meta", {}))
if job["backend"] == "ptq":
cfg = ptq.compile_config(job, onnx, calib_npy_dir(cd), work_dir(ctx), agent_overrides)
if job.get("recipe") or not own.get("calib"):
if job.get("recipe"):
cfg["calibration_parameters"]["cal_data_dir"] = str(calib_npy_dir(cd))
else:
cfg["calibration_parameters"]["cal_data_dir"] = str(own["calib"])
yp = stage_dir / "compile.yaml" if job.get("recipe") else compile_yaml_path(ctx)
yp.parent.mkdir(parents=True, exist_ok=True)
ptq.write_compile_config(cfg, yp)
@ -138,70 +125,6 @@ class Calibrate(Stage):
return StageOutcome("OK", 0, None, 0.0, meta, {"calib": meta})
def io_signature(path: Path) -> list:
"""Graph inputs (initializers excluded) and outputs as (element type, shape): what the platform's
baseline, board runner and evaluator rely on. Names are free."""
import onnx
graph = onnx.load(str(path), load_external_data=False).graph
weights = {t.name for t in graph.initializer}
def sig(values):
return [(v.type.tensor_type.elem_type,
[d.dim_value if d.HasField("dim_value") else d.dim_param for d in v.type.tensor_type.shape.dim])
for v in values]
return [sig(v for v in graph.input if v.name not in weights), sig(graph.output)]
def own_inputs(ctx: StageContext, overrides: dict, platform_onnx: Path | None) -> dict:
"""The agent's own graph / calibration set, named in compile.yaml as workspace paths and made in
`bench dev` (the development environment shares this workspace). Same I/O as the platform graph keeps
baseline, board and evaluation unchanged."""
ws = (ctx.stage_root.parent / "workspace").resolve()
out, meta = {}, {}
if ctx.extra.get("own_inputs_required"):
need = [f"{section}.{key}" for section, key in (("model_parameters", "onnx_model"), ("calibration_parameters", "cal_data_dir"))
if not (overrides.get(section) or {}).get(key)]
if need:
raise ValueError("no platform graph or calibration set to fall back on: compile.yaml must name your own "
+ " and ".join(need) + " (workspace paths, made in bench dev)")
def resolve(rel: str) -> Path:
p = (ws / rel).resolve()
if not p.is_relative_to(ws):
raise ValueError(f"{rel} is outside the workspace")
return p
rel = (overrides.get("model_parameters") or {}).get("onnx_model")
if rel:
path = resolve(rel)
if not path.is_file():
raise FileNotFoundError(f"model_parameters.onnx_model: {rel} not found in the development workspace "
"(files made in bench dev are there; local edits sync on the next bench dev)")
if not (platform_onnx and platform_onnx.exists()):
raise ValueError("run `bench stage run export` first: your graph is checked against the platform graph's inputs/outputs")
mine, ref = io_signature(path), io_signature(platform_onnx)
if mine != ref:
raise ValueError(f"model_parameters.onnx_model inputs/outputs {mine} differ from the platform graph {ref}; "
"keep their count, order, element type and shape so evaluation stays comparable")
digest = hashlib.sha256()
with path.open("rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
digest.update(chunk)
# ponytail: external-data tensors (graphs > 2 GB) are not hashed; the main file pins the structure.
out["onnx"] = path
meta["graph"] = {"source": "workspace", "path": rel, "sha256": digest.hexdigest()}
rel = (overrides.get("calibration_parameters") or {}).get("cal_data_dir")
if rel:
path = resolve(rel)
n = sum(1 for _ in path.glob("*.npy")) if path.is_dir() else 0
if not n:
raise ValueError(f"calibration_parameters.cal_data_dir: {rel} has no .npy files in the development workspace")
out["calib"] = path
meta.update(calib_source="workspace", calib_npy_dir=str(path), calib_n=n,
calibration={"source": "workspace", "path": rel, "count": n})
out["meta"] = meta
return out
def _make_synthetic(out: Path, input_spec: str, n: int) -> None:
"""占据类第三级回退:输入范围内的随机张量。G2 结果只作参考(pass-weak)。"""
import hashlib

View File

@ -12,8 +12,6 @@ import re
import shutil
from pathlib import Path
import yaml
from .. import watchdog
from ..backends import get as backend
from ..registry import boards
@ -49,10 +47,6 @@ class Hbm(Stage):
pool = "cpu"
def native(self, ctx: StageContext, stage_dir: Path) -> StageOutcome:
from .. import salty_stage
remote = salty_stage.hbm(ctx, stage_dir)
if remote is not None:
return remote
job = ctx.job
wd = work_dir(ctx)
cands = ([recipe_artifact(ctx, "05-compile", "hbm")] if job.get("recipe") else
@ -107,22 +101,18 @@ class Hbm(Stage):
m["verifier_cosine"] = cos
m["verifier_ok"] = (rv.state == "OK") and (cos is None or cos >= thr)
m["verifier_rc"] = rv.exit_code
if rv.state != "OK" and cos is None:
# hb_verifier died before printing its table ("launch function crashed", rc 255): nothing was
# compared, so consistency is unknown (G1 unverified), not a mismatch.
m["verifier_ok"] = None
m["verifier_note"] = f"hb_verifier {rv.state} rc={rv.exit_code} before measuring"
else:
m["verifier_ok"] = None
m["verifier_note"] = "ptq_model.onnx or calib sample missing"
else:
m["verifier_ok"] = True # QAT 的一致性在 08-eval 的 G2 里判
# hbm ≡ bc 模拟器对比:agent 在配置里要了才跑,只报数不判分(10-14 起)。x86 模拟器每张图在 Salty 上要几十分钟,
# 以前每次必跑 2 张,撞 2 小时上限 103 次;388 次报出的 13 次不一致 hb_verifier 都已先判掉;08-eval 在板上跑真 HBM。
n = ((yaml.safe_load(ctx.agent_config.read_text()) or {}) if ctx.agent_config else {}).get("verification_parameters", {}).get("hbm_bc_samples", 0)
m["hbm_bc_cos_min"] = _hbm_bc_spotcheck(ctx, stage_dir, dst, bc_path(ctx), be, wdp, n) if n and bc else None
m["hbm_bc_spotcheck"] = f"{n} samples, diagnostic only" if n and bc else "not requested; board HBM verification remains required"
# ---- hbm ≡ bc 抽查(编译一致性诊断;08-eval 另跑板端 HBM)
if bc and job["backend"] == "ptq":
m["hbm_bc_cos_min"] = _hbm_bc_spotcheck(ctx, stage_dir, dst, bc_path(ctx), be, wdp)
if m["hbm_bc_cos_min"] is not None and m["hbm_bc_cos_min"] < 0.999:
m["verifier_ok"] = False
m["verifier_note"] = f"hbm vs bc cosine {m['hbm_bc_cos_min']:.6f} < 0.999"
# ---- 编译日志扫 ERROR(只看 hb_compile 自己的日志级别,不看 INFO 里的字样)
clog = ""
@ -153,7 +143,7 @@ class Hbm(Stage):
"toolchain": job.get("toolchain"), "size": dst.stat().st_size, "work_dir": str(wd), **output_meta}, {"G1": m})
def _hbm_bc_spotcheck(ctx: StageContext, stage_dir: Path, hbm: Path, bc: Path, be, wdp: dict, n: int) -> float | None:
def _hbm_bc_spotcheck(ctx: StageContext, stage_dir: Path, hbm: Path, bc: Path, be, wdp: dict, n: int = 2) -> float | None:
"""同 n 张校准样本分别过 hbm(run_sim) 与 bc(HBIR run),返回逐输出余弦的最小值。任何一步失败返回 None(记录,不判)。"""
try:
import numpy as np
@ -169,7 +159,7 @@ def _hbm_bc_spotcheck(ctx: StageContext, stage_dir: Path, hbm: Path, bc: Path, b
fo = stage_dir / f"spot_{tag}.npz"
r = watchdog.run([str(be.python(ctx.job)), "-m", "harness.tools.infer_hbm", "--model", str(model), "--inputs", str(fin), "--out", str(fo)],
cwd=work_dir(ctx), stdout=stage_dir / f"spot_{tag}.log", stderr=stage_dir / f"spot_{tag}.err.log",
env=Hbm().env(ctx), stall_s=wdp["stall_s"], poll_s=5)
env=self_env(ctx), stall_s=wdp["stall_s"], poll_s=5)
if r.state != "OK":
return None
outs[tag] = np.load(fo)
@ -184,6 +174,10 @@ def _hbm_bc_spotcheck(ctx: StageContext, stage_dir: Path, hbm: Path, bc: Path, b
return None
def self_env(ctx: StageContext) -> dict:
return Hbm().env(ctx)
def _find_march(info: str) -> str | None:
low = info.lower()
strings = boards().get("march_strings", {})

View File

@ -19,20 +19,17 @@ from . import stagerun, execution as lifecycle, watchdog
from .stages import AGENT_RUNNABLE
def read_regular(parent_fd, name, limit=1024 * 1024):
def read_regular(parent_fd, name):
fd = os.open(name, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=parent_fd)
try:
if not stat.S_ISREG(os.fstat(fd).st_mode) or (limit is not None and os.fstat(fd).st_size > limit):
if not stat.S_ISREG(os.fstat(fd).st_mode) or os.fstat(fd).st_size > 1024 * 1024:
raise ValueError('ticket/config must be a regular file <= 1 MiB')
if limit is not None:
return os.read(fd, limit + 1)
with os.fdopen(os.dup(fd), 'rb') as stream:
return stream.read()
return os.read(fd, 1024 * 1024 + 1)
finally:
os.close(fd)
def config_bytes(ws, value, limit=1024 * 1024):
def config_bytes(ws, value):
p = Path(value)
rel = p.relative_to(ws)
if not rel.parts or '..' in rel.parts:
@ -42,54 +39,29 @@ def config_bytes(ws, value, limit=1024 * 1024):
for part in rel.parts[:-1]:
nxt = os.open(part, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, dir_fd=fd)
os.close(fd); fd = nxt
return read_regular(fd, rel.name, limit)
return read_regular(fd, rel.name)
finally:
os.close(fd)
def workspace_relative(ws, value, field):
"""A path the agent names inside its workspace, kept relative: 04 resolves it in the development
environment, where files made by `bench dev` live (they may not exist on the controller)."""
p = Path(value) if isinstance(value, str) and value else None
if p is not None and p.is_absolute():
try:
p = p.relative_to(ws)
except ValueError:
p = None
if p is None or not p.parts or '..' in p.parts:
raise ValueError(f'{field} must be a path inside the workspace')
return p.as_posix()
def snapshot_config(ws, value):
"""Keep path-bearing options from smuggling outer-worker file access."""
cfg = yaml.safe_load(config_bytes(ws, value)) or {}
allowed = {
'input_parameters': {'input_type_rt', 'input_type_train', 'input_layout_train', 'input_layout_rt',
'input_name', 'input_shape', 'mean_value', 'scale_value', 'std_value'},
'calibration_parameters': {'calibration_type', 'max_percentile', 'per_channel', 'cal_data_dir',
'calibration_parameters': {'calibration_type', 'max_percentile', 'per_channel',
'run_on_cpu', 'run_on_bpu', 'optimization', 'quant_config'},
'compiler_parameters': {'compile_mode', 'optimize_level', 'jobs', 'debug'},
'verification_parameters': {'hbm_bc_samples'},
}
if not isinstance(cfg, dict):
raise ValueError('config must be a mapping')
for section, values in cfg.items():
if section == 'model_parameters':
# The graph is the agent's (04 checks its inputs/outputs against the platform graph);
# march, working_dir and output names stay pinned, so other keys are ignored as before.
if isinstance(values, dict) and 'onnx_model' in values:
values['onnx_model'] = workspace_relative(ws, values['onnx_model'], 'model_parameters.onnx_model')
continue
continue # Existing backend ignores/pins this section.
if section not in allowed or not isinstance(values, dict) or set(values)-allowed[section]:
extra = sorted(set(values) - allowed[section]) if section in allowed and isinstance(values, dict) else []
raise ValueError('unsupported config keys: ' + str(section) + (f' {extra}' if extra else ''))
n = (cfg.get('verification_parameters') or {}).get('hbm_bc_samples', 0)
if type(n) is not int or not 0 <= n <= 16:
raise ValueError('verification_parameters.hbm_bc_samples must be an integer in [0, 16]')
raise ValueError('unsupported config keys: ' + str(section))
cal = cfg.get('calibration_parameters', {})
if 'cal_data_dir' in cal:
cal['cal_data_dir'] = workspace_relative(ws, cal['cal_data_dir'], 'calibration_parameters.cal_data_dir')
quant = cal.get('quant_config')
if isinstance(quant, str):
p = Path(quant)
@ -251,14 +223,6 @@ class Worker:
status = queue / 'status.json'
if status.exists() and json.loads(status.read_text()).get('state') == 'DONE':
result = json.loads(status.read_text())
ledger = self.execution.snapshot().get('operations', {}).get(hid) if self.execution else None
if (result.get('error_class') != 'SIDE_EFFECT_UNKNOWN' and result.get('db_registered') is not False
and previous.get('state') == 'DONE' and ledger == previous):
# Settled before a pause (same skip as reconcile_completed). A later attempt of the
# rollout may overwrite its outputs (one .hbm path per rollout), so re-proving them
# on resume quarantined every multi-compile rollout (10-12/13: 19 rollouts).
self._publish(fd, result)
return
if result.get('error_class') == 'SIDE_EFFECT_UNKNOWN':
if self.execution:
self.execution.quarantine('SIDE_EFFECT_UNKNOWN')

View File

@ -19,39 +19,6 @@ from . import execution as lifecycle, paths, scheduler, watchdog
from .recipe_runtime import regular_bytes
VERSION = 'bench.terminal/1'
# `bench dev --board` (bin/stageclient.py) marks the command: the collectors' request check (prepare) predates
# board commands and only admits the development target, while run() executes in a fresh process.
BOARD_MARK = '#bench:board\n'
BOARD_MAX_S = 1800 # the board is shared with formal board/eval stages
BOARD_MAX_BYTES = 512 << 20 # dev-only files come back from personal storage at ~0.5 MB/s (10-16: 1.2 GB took 40 min)
DEV_QUEUE_S = 3 * 3600 # bench dev: waiting for a dev slot / a Salty container is not the command's own time
class BoardRefused(ValueError):
"""A board command that cannot run; the message goes back to the agent as the command's result."""
def board_request(ctx, rd, hd, req):
"""A development request marked for the board: the requesting directory (local view plus the files
that exist only in the development workspace, e.g. a compiled HBM) is what goes up."""
if not ctx.extra.get('own_inputs_required'):
raise BoardRefused('BOARD_TERMINAL_UNAVAILABLE: board commands are open to agent v18+ tasks')
from . import salty_dev
small = '新建一个只放 .hbm、脚本和几个输入的小目录,cd 进去再运行 bench dev --board'
# Size first: 10-16 three of four self-tests ran from the whole workspace (1.2 GB), spent 40 min fetching
# it, passed the deadline and failed with no reason given.
if salty_dev.remote_size(rd, req['cwd'], BOARD_MAX_BYTES) is None:
raise BoardRefused(f'BOARD_INPUT_TOO_LARGE: {req["cwd"]} 在开发环境里超过 {BOARD_MAX_BYTES >> 20} MiB(或子目录超过 40 个),'
f'从开发环境取回约 0.5 MB/s,时限内传不完;{small}')
started = time.monotonic()
remote = hd / 'dev-files'
salty_dev.download_dir(rd, req['cwd'], remote)
if time.monotonic() >= req['deadline_monotonic'] - 60:
total = sum(f.stat().st_size for f in remote.rglob('*') if f.is_file())
raise BoardRefused(f'BOARD_TRANSFER_TOO_SLOW: 从开发环境取回 {req["cwd"]}({total >> 20} MiB)用了 '
f'{(time.monotonic() - started) / 60:.0f} 分钟,这条命令的时限已到;{small}')
return dict(req, target='board', command=req['command'][len(BOARD_MARK):], source=req['cwd'], cwd='.',
deadline_monotonic=min(req['deadline_monotonic'], time.monotonic() + BOARD_MAX_S)), remote
def relative(value, field):
@ -87,21 +54,12 @@ def prepare(rd, queue, request):
if not (rd / 'execution.json').is_file():
raise ValueError('terminal requires a managed execution')
metadata = json.loads((rd / 'rollout.json').read_text())
from . import salty_dev
if req['target'] == 'development' and salty_dev.selected():
# bench dev: the Salty container's sandbox is the isolation (salty_remote_worker.dev); commands only,
# formal results still come from the stages.
if req['evaluate'] or req['source'] != '.':
raise ValueError('bench dev runs commands only; evaluation and HBM results come from the stages')
elif metadata.get('natural_agent_version') != 'natural-agent-v1' or metadata.get('jail_policy') != 'private-home':
if metadata.get('natural_agent_version') != 'natural-agent-v1' or metadata.get('jail_policy') != 'private-home':
raise ValueError('terminal requires a new task with natural-agent-v1 private-home isolation')
if paths.config().get('terminal', {}).get('enabled') is not True:
raise ValueError('managed terminal disabled until isolation acceptance')
state = lifecycle.Execution(rd).snapshot()
# The command's timeout starts when its container does (salty_remote_worker.dev); the request also covers the
# queue (10-15: 64Gi dev containers waited in Salty past a 300 s timeout, so even `ls` failed).
waiting = DEV_QUEUE_S if req['target'] == 'development' and salty_dev.selected() else 0
req['deadline_monotonic'] = min(time.monotonic() + timeout + waiting, state['deadline_monotonic'])
req['deadline_monotonic'] = min(time.monotonic() + timeout, state['deadline_monotonic'])
if req['evaluate']:
# Accepting a new formal candidate invalidates earlier stage success even
# if preparation or the executor crashes before producing any prediction.
@ -214,34 +172,12 @@ def score_predictions(raw, labels):
def run(hd, rd, publish):
req = json.loads((hd / 'request.json').read_text())
board = req['target'] == 'board' or req['command'].startswith(BOARD_MARK)
try:
return _run(hd, rd, publish, req)
except BoardRefused as exc:
(hd / 'command.stderr.log').write_text(str(exc) + '\n')
publish({'state': 'DONE', 'result': 'FAIL', 'exit_code': 2, 'error_class': str(exc).split(':', 1)[0],
'duration_s': 0, 'stdout': '', 'stderr': str(hd / 'command.stderr.log'), 'meta': '',
'stderr_tail': str(exc), 'cleanup_state': 'VERIFIED', 'measurements': {},
'model_execution_source': 'UNVERIFIED', 'finished_at': time.time()})
return 0
finally:
if board: # copies of HBMs and inputs would pile up on the controller (10-16: 2.5 GB after two failures)
for name in ('snapshot', 'dev-files', 'dev-files.part'):
shutil.rmtree(hd / name, ignore_errors=True)
for name in ('board-input.tar', 'board-output.tar'): # output/ keeps the extracted files
(hd / name).unlink(missing_ok=True)
def _run(hd, rd, publish, req):
from .stagerun import _ctx_from_rollout
req = json.loads((hd / 'request.json').read_text())
execution = lifecycle.current()
if execution is None:
raise RuntimeError('terminal requires execution ownership')
ctx = _ctx_from_rollout(rd, None)
dev_files = None
if req['target'] == 'development' and req['command'].startswith(BOARD_MARK):
req, dev_files = board_request(ctx, rd, hd, req)
output, inputs = hd / 'output', hd / 'input'
output.mkdir(); inputs.mkdir()
formal = req['evaluate'] or req['target'] == 'board'
@ -249,20 +185,6 @@ def _run(hd, rd, publish, req):
manifest = None
if formal:
manifest = freeze(workspace / req['source'], hd / 'snapshot')
if dev_files is not None:
# Local edits win; big outputs that stayed in the development workspace fill the gaps.
for f in sorted(dev_files.rglob('*')):
target = hd / 'snapshot' / f.relative_to(dev_files)
if f.is_file() and not f.is_symlink() and not target.exists():
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(f, target)
digest = hashlib.sha256()
with target.open('rb') as stream:
for chunk in iter(lambda: stream.read(1 << 20), b''):
digest.update(chunk)
manifest[str(f.relative_to(dev_files))] = {'sha256': digest.hexdigest(), 'bytes': target.stat().st_size,
'from': 'development'}
shutil.rmtree(dev_files)
workspace = hd / 'snapshot'
lifecycle.atomic_json(hd / 'submission.json', manifest)
elif req['source'] != '.':
@ -281,8 +203,7 @@ def _run(hd, rd, publish, req):
(hd / 'jail').mkdir()
lifecycle.atomic_json(hd / 'jail.json', config)
stdout, stderr = hd / 'command.stdout.log', hd / 'command.stderr.log'
# A separate dev pool, when configured, keeps long agent experiments from holding compile slots.
pool = 'board' if req['target'] == 'board' else 'dev' if 'dev' in paths.config().get('pools', {}) else 'cpu'
pool = 'board' if req['target'] == 'board' else 'cpu'
publish({'state': 'QUEUED', 'pool': pool, 'queued_at': time.time()})
started = time.time()
try:
@ -290,12 +211,9 @@ def _run(hd, rd, publish, req):
if time.monotonic() >= deadline:
raise lifecycle.ExecutionStopped('TERMINAL_DEADLINE_EXCEEDED')
publish({'state': 'RUNNING', 'pool': pool, 'started_at': time.time()})
from . import salty_dev
if req['target'] == 'board':
from .terminal_board import execute
result = execute(ctx, hd, config, deadline) # run() drops the copies; submission.json keeps the hashes
elif salty_dev.selected():
result = salty_dev.execute(ctx, hd, req, deadline, stdout, stderr)
result = execute(ctx, hd, config, deadline)
else:
command = ['unshare', '-Urmpf' + ('n' if formal else ''), '--mount-proc', '--kill-child=KILL',
paths.config()['paths'].get('harness_python', sys.executable),
@ -304,7 +222,7 @@ def _run(hd, rd, publish, req):
env={'PATH': '/usr/bin:/bin', 'LANG': 'C.UTF-8'}, execution=execution,
role='terminal-command', hard_ceiling_s=max(.01, deadline - time.monotonic()), poll_s=.1,
stop_check=lambda: 'TERMINAL_DEADLINE_EXCEEDED' if time.monotonic() >= deadline else None)
if result.cleanup_state not in ('VERIFIED', 'REMOTE_BOUNDED'):
if result.cleanup_state != 'VERIFIED':
execution.quarantine('TERMINAL_CLEANUP_UNCONFIRMED')
except BaseException:
# Board helper quarantines inside the lease if a remote receipt is unknown.

View File

@ -1,8 +1,7 @@
"""Finite board transport for candidate commands, never substitutes board_bench.
The existing board guard owns remote descendants and the physical device lock. The command runs directly in
the board's container (the boards are test devices; the S600 containers forbid namespaces anyway, 10-16) in
this session's directory, which is removed afterwards; $BENCH_OUT is the directory brought back.
The existing board guard owns remote descendants and the physical device lock.
Arbitrary commands are admitted only after a namespace capability probe succeeds.
"""
import json
import os
@ -13,7 +12,7 @@ import time
import uuid
from contextlib import contextmanager
from . import execution as lifecycle, watchdog
from . import execution as lifecycle, paths, watchdog
from .stages.s07_board import board_env
from .tools.board_perf import RemoteSession, HERE, guarded_run, run as transport
@ -56,7 +55,8 @@ class TerminalSession(RemoteSession):
def __enter__(self):
self.active = True
self.setup(self.execute, ['mkdir', '-p', '--', self.directory])
for source in (HERE / 'board_guard.py', HERE / 'board_probe.py', HERE.parent / 'process_reaper.py'):
for source in (HERE / 'board_guard.py', HERE / 'board_probe.py',
HERE.parent / 'process_reaper.py', paths.BENCH_ROOT / 'scripts/terminal_jail.py'):
self.setup(self.upload, source, source.name)
return self
@ -79,6 +79,12 @@ def execute(ctx, hd, config, deadline):
session = TerminalSession(ctx.extra['board'].get('runtime_dir', '/userdata/bench'), deadline)
except (ValueError, SystemExit):
return not_started('BOARD_CONNECTION_CONFIGURATION_MISSING')
# Read-only capability probe; no candidate process or remote job directory.
try:
session.execute(['unshare', '-Urmpfn', '--mount-proc', '--kill-child=KILL',
'sh', '-c', 'command -v setpriv >/dev/null'], timeout=max(.01, min(20, deadline - time.monotonic())))
except (Exception, SystemExit):
return not_started('BOARD_ISOLATION_UNAVAILABLE')
op = uuid.uuid4().hex
dispatched = False
receipt = {'operation_id': op, 'guard_version': 4, 'cleanup_state': 'UNKNOWN'}
@ -92,11 +98,18 @@ def execute(ctx, hd, config, deadline):
session.execute(['python3', '-c',
'import tarfile,sys,pathlib; p=pathlib.Path(sys.argv[1]); '
'tarfile.open(p/"input.tar").extractall(p, filter="data"); '
'(p/"output").mkdir()', session.directory])
'(p/"output").mkdir(); (p/"jail").mkdir()', session.directory])
board_config = dict(config, jailroot=session.path('jail'), workspace=session.path('workspace'),
output=session.path('output'), input=session.path('input'),
runtimes=[], path='/usr/local/bin:/usr/bin:/bin',
devices=['/dev/bpu', '/dev/bpu_core0', '/dev/bpu_core1', '/dev/bpu_core2', '/dev/bpu_core3', '/dev/ion'])
board_config['timeout'] = max(1, int(session.remaining()))
lifecycle.atomic_json(hd / 'board-jail.json', board_config)
session.setup(session.upload, hd / 'board-jail.json', 'jail.json')
command = ['python3', '-u', session.path('board_guard.py'), '--directory', session.directory,
'--operation-id', op, '--timeout', str(session.remaining()), '--',
'env', 'BENCH_OUT=' + session.path('output'), 'bash', '-c',
f"cd {shlex.quote(session.path('workspace'))}/{shlex.quote(config['cwd'])} && {config['command']}"]
'unshare', '-Urmpfn', '--mount-proc', '--kill-child=KILL', 'python3',
session.path('terminal_jail.py'), session.path('jail.json')]
timeout = session.remaining()
dispatched = True
try:
@ -118,7 +131,7 @@ def execute(ctx, hd, config, deadline):
receipt['error'] = type(exc).__name__
lifecycle.atomic_json(hd / 'board-receipts' / (op + '.json'), receipt)
if receipt['cleanup_state'] != 'VERIFIED':
session.cleanup_unconfirmed = True
session.failed_bench = True
current.quarantine('BOARD_CLEANUP_UNCONFIRMED')
raise RuntimeError('BOARD_CLEANUP_UNCONFIRMED')
# Download only candidate-generated regular files after all children exit.
@ -145,7 +158,7 @@ def execute(ctx, hd, config, deadline):
proof='not_started', reason='BOARD_SETUP_FAILED')
lifecycle.atomic_json(hd / 'board-receipts' / (op + '.json'), receipt)
elif receipt['cleanup_state'] != 'VERIFIED':
session.cleanup_unconfirmed = True
session.failed_bench = True
current.quarantine('BOARD_CLEANUP_UNCONFIRMED')
raise
finally:

View File

@ -85,11 +85,6 @@ class _PreAuthDrop(Exception):
pass
# Loading a model takes seconds (S600 ~4 s). 10-15: two S600 loads hung until the guard's ~49 min deadline; end such a
# call early as a retryable stall. Only the load phase is bounded: inference on slow models can legitimately take long.
LOAD_STALL_S = 600
def guarded_run(cmd: list[str], timeout: float) -> str:
for attempt in range(3):
try:
@ -110,10 +105,9 @@ def _guarded_once(cmd: list[str], timeout: float) -> str:
execution = from_environment()
with execution.admission() if execution else nullcontext():
proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, errors, partial = bytearray(), bytearray(), b''
output, errors = bytearray(), bytearray()
deadline = time.monotonic() + timeout + 15 # board deadline includes its cleanup acknowledgement
next_lease = 0.0
loading = None
try:
with selectors.DefaultSelector() as selector:
selector.register(proc.stdout, selectors.EVENT_READ, 'stdout')
@ -122,10 +116,6 @@ def _guarded_once(cmd: list[str], timeout: float) -> str:
now = time.monotonic()
if now > deadline:
raise subprocess.TimeoutExpired(cmd, timeout)
if loading is not None and now - loading > LOAD_STALL_S:
print(f"__BENCH_WATCHDOG_STALLED__ board model load: no load_done after {LOAD_STALL_S} s",
file=sys.stderr, flush=True)
raise SystemExit(124)
if now >= next_lease and proc.poll() is None:
try:
proc.stdin.write(b'lease\n'); proc.stdin.flush()
@ -139,11 +129,6 @@ def _guarded_once(cmd: list[str], timeout: float) -> str:
elif key.data == 'stderr':
sys.stderr.write(data.decode(errors='replace')); sys.stderr.flush()
errors.extend(data); del errors[:-4096]
*lines, partial = (partial + data).split(b'\n')
partial = partial[-4096:]
for line in lines:
if line.startswith(b'{"board_progress": "load_'):
loading = time.monotonic() if b'"load_start"' in line else None
else:
output.extend(data)
# Only the final JSON and a bounded diagnostic tail are needed.
@ -181,7 +166,7 @@ def not_started(reason, directory=None):
class RemoteSession:
"""一次评测拥有一个临时目录。HBM/脚本只传一次,退出时清理自己的目录;进入时顺手删 6 小时没人动过的旧会话目录。"""
"""一次评测拥有一个临时目录。HBM/脚本只传一次,退出时只清理自己的文件。"""
def __init__(self, hbm: str | Path, runtime_dir: str = "/userdata/bench"):
self.hbm = Path(hbm)
@ -189,21 +174,12 @@ class RemoteSession:
self.scp_opts = ["-P" if o == "-p" else o for o in self.opts]
self.directory = runtime_dir.rstrip("/") + "/eval-" + uuid.uuid4().hex
self.active = False
self.cleanup_unconfirmed = False
self.failed_bench = False
def __enter__(self):
if self.active:
return self
self.active = True
try:
# Sessions whose container was killed never reach close(). A live session touches its directory
# around every board call (each ≤ 1 h), so 6 h untouched means nobody owns it.
swept = self.execute(["find", self.directory.rsplit("/", 1)[0], "-maxdepth", "1", "-name", "eval-*", "-mmin", "+360",
"-print", "-exec", "rm", "-rf", "--", "{}", "+"], timeout=120, cleanup=True)
if swept.strip():
print(f"removed stale board sessions: {' '.join(swept.split())}", file=sys.stderr, flush=True)
except (Exception, SystemExit) as e:
print(f"WARN stale board session sweep failed: {e}", file=sys.stderr)
try:
self.setup(self.execute, ["mkdir", "-p", "--", self.directory])
for source, name in ((self.hbm, "model.hbm"), (HERE / "board_bench.py", "board_bench.py"),
@ -256,19 +232,12 @@ class RemoteSession:
# Recompute after uploads; the managed phase may already be partly spent.
from harness.execution import from_environment, current_operation
execution = from_environment()
deadlines = []
if execution:
state = execution.snapshot()
phase = state.get("phases", {}).get(current_operation(), {})
deadlines = [state["deadline_monotonic"]]
if phase.get("execution_deadline_monotonic") is not None:
deadlines.append(phase["execution_deadline_monotonic"])
if os.environ.get("BENCH_PHASE_DEADLINE"):
# Salty containers carry no execution state: end before the phase does, leaving time to hand the
# result and cleanup receipt back. 10-12/14: 12 board hangs outlived the phase, no receipt came
# back and each rollout was lost as BOARD_CLEANUP_UNCONFIRMED instead of a retryable FAIL.
deadlines.append(time.monotonic() + float(os.environ["BENCH_PHASE_DEADLINE"]) - time.time() - 540)
if deadlines:
timeout = min(timeout, min(deadlines) - time.monotonic() - 60)
if timeout <= 0:
self.not_started("BOARD_CLEANUP_BUDGET_INSUFFICIENT")
@ -292,29 +261,26 @@ class RemoteSession:
info = result_json(guarded_run(["ssh", *self.opts, self.target, shlex.join(command)], timeout))
from harness.board_identity import validate
validate(info, os.environ.get("BENCH_EXPECT_BOARD_TYPE", ""), os.environ.get("BENCH_EXPECT_UCP", ""))
self.failed_bench = False
return info
except BaseException:
self.failed_bench = True
raise
finally:
# 15 s with no retry was below ssh's own 20 s ConnectTimeout: one slow handshake lost a rollout whose
# board run had finished and been reaped (10-12..10-15: 9 times, 8 TimeoutExpired, 1 ssh exit).
for attempt in range(3):
try:
actual = json.loads(self.execute(["cat", "--", self.path("guard-result.json")], timeout=60, cleanup=True))
if actual.get("operation_id") != operation_id or actual.get("guard_version") != 4:
raise ValueError("board cleanup receipt identity mismatch")
if (actual.get("all_children_reaped") is not True or
actual.get("proof") not in ("waitpid:ECHILD", "not_started")):
raise ValueError("board cleanup receipt has no kernel proof")
receipt.update(actual)
receipt.pop("receipt_error", None)
break
except (Exception, SystemExit) as exc:
receipt["receipt_error"] = type(exc).__name__
if attempt < 2:
time.sleep(10)
try:
actual = json.loads(self.execute(["cat", "--", self.path("guard-result.json")], timeout=15, cleanup=True))
if actual.get("operation_id") != operation_id or actual.get("guard_version") != 4:
raise ValueError("board cleanup receipt identity mismatch")
if (actual.get("all_children_reaped") is not True or
actual.get("proof") not in ("waitpid:ECHILD", "not_started")):
raise ValueError("board cleanup receipt has no kernel proof")
receipt.update(actual)
except (Exception, SystemExit) as exc:
receipt["receipt_error"] = type(exc).__name__
if destination:
atomic_json(Path(destination) / (operation_id + ".json"), receipt)
if receipt["cleanup_state"] != "VERIFIED":
self.cleanup_unconfirmed = True
self.failed_bench = True
if sys.exc_info()[0] is None:
raise RuntimeError("BOARD_CLEANUP_UNCONFIRMED")
@ -324,15 +290,11 @@ class RemoteSession:
def close(self):
if self.active:
self.active = False
command = ["rm", "-rf", "--", self.directory]
if self.cleanup_unconfirmed:
# Its guard may still run and writes its receipt here, then the board quarantine marker: keep the
# directory and the receipt, drop the rest (10-15: kept sessions filled the S600s' /userdata).
print(f"board session cleanup unconfirmed, directory kept: {self.directory}", file=sys.stderr, flush=True)
command = ["find", self.directory, "-mindepth", "1", "-maxdepth", "1", "!", "-name", "*guard-result.json",
"-exec", "rm", "-rf", "--", "{}", "+"]
if self.failed_bench:
print(f"board failed session retained for diagnosis: {self.directory}", file=sys.stderr, flush=True)
return
try:
self.execute(command, timeout=60, cleanup=True)
self.execute(["rm", "-rf", "--", self.directory], timeout=60, cleanup=True)
except (Exception, SystemExit) as e:
print(f"WARN board temporary directory cleanup failed: {self.directory}: {e}", file=sys.stderr)

View File

@ -1,39 +0,0 @@
#!/bin/bash
# 本机每 10 分钟(launchd)查一次 Mac mini 上的采集;有问题弹系统通知,同样的问题每 2 小时再提醒一次,恢复了也通知。
# 安装:cp 到 ~/.bench-health/,plist 放 ~/Library/LaunchAgents/cc.d-robotics.bench-health.plist。记录在 ~/.bench-health/log。
MINI=amos@192.168.66.251 # Mac mini 换了地址改这里
RUNS="r20260929-04 r20260929-05 r20260929-06" # 加新批次时加到这里
FEISHU_TO="ou_cc299d6a3684a6c45d56d91f7127b185" # 填飞书 open_id 就同时用小金(飞书群答疑助手)私聊提醒
export PATH=$HOME/.npm-global/bin:/usr/local/bin:/opt/homebrew/bin:/usr/bin:/bin
D=~/.bench-health; mkdir -p "$D"
notify() {
osascript -e 'on run argv' -e 'display notification (item 1 of argv) with title (item 2 of argv) sound name "Glass"' -e 'end run' "$1" "$2"
if [ -n "$FEISHU_TO" ] && ! lark-cli im +messages-send --as bot --user-id "$FEISHU_TO" --text "【$2】"$'\n'"$1" > /dev/null 2>&1; then
echo "$(date '+%m-%d %H:%M') 飞书发送失败" >> "$D/alerts.log"
fi
printf '%s %s\n%s\n' "$(date '+%m-%d %H:%M')" "$2" "$1" >> "$D/alerts.log"
}
if ! ssh -o ConnectTimeout=15 -o BatchMode=yes "$MINI" true 2>/dev/null; then
fails=$(( $(cat "$D/ssh_fails" 2>/dev/null || echo 0) + 1 )); echo $fails > "$D/ssh_fails"
[ $fails -ge 2 ] && problems="Mac mini($MINI)连不上,已 $((fails * 10)) 分钟。Wi-Fi 掉了、换了地址或关机了" # Wi-Fi 抖一下不报
else
rm -f "$D/ssh_fails"
out=$(ssh -o ConnectTimeout=15 -o BatchMode=yes "$MINI" \
"export PATH=/opt/homebrew/bin:\$PATH; limactl list --format '{{.Status}}' bench && limactl shell bench -- python3 /mnt/data/ming01.wang/model-compile-hbdk/bench/scripts/health_check.py $RUNS" 2>&1)
rc=$?
status=$(head -1 <<<"$out")
if [ "$status" != Running ]; then problems="bench 虚拟机没在运行($status),Mac mini 可能重启过,需要恢复"
elif [ $rc != 0 ]; then problems="健康检查本身出错:$(tail -1 <<<"$out")"
else problems=$(tail -n +2 <<<"$out"); fi
fi
key=$(tr -d '0-9' <<<"$problems") # 去掉数字,分钟数变了不算新问题
now=$(date +%s); last=$(cat "$D/last_t" 2>/dev/null || echo 0)
if [ -n "$problems" ]; then
if [ "$key" != "$(cat "$D/last_key" 2>/dev/null)" ] || [ $((now - last)) -ge 7200 ]; then
notify "$problems" "采集报警($(wc -l <<<"$problems" | tr -d ' ') 项)"
echo "$key" > "$D/last_key"; echo $now > "$D/last_t"
fi
elif [ -s "$D/last_key" ]; then
notify "之前报的问题都已消失" "采集恢复正常"; rm -f "$D/last_key"
fi
echo "$(date '+%m-%d %H:%M') ${problems:-ok}" | head -1 >> "$D/log"

View File

@ -1,34 +0,0 @@
#!/bin/bash
# 排空旧调度程序后换新代码重启,不丢在跑的轨迹。在 Mac mini 的 bench 虚拟机里后台跑:
# setsid nohup scripts/drain_restart.sh "<run> <旧pid> <配置名> <新并发> [题单文件]" ... >> logs/drain-restart.log 2>&1 &
# 暂停的轨迹归旧进程所有(OWNER_NOT_RELEASABLE),旧进程退出后由新进程自动续跑,不用手动 resume。
# 旧进程没有子进程、也没有校验中的轨迹满 10 分钟,说明卡在最后的导出上,按 bench-restart-hazards 第 5 条 kill -9。
set -u
B=/mnt/data/ming01.wang/model-compile-hbdk/bench; C=/mnt/data/ming01.wang/configs
export BENCH_ROOT=$B BENCH_STATE_DIR=$B/state
export PATH=/home/amos.guest/.local/lib/salty-controller/python/bin:/home/amos.guest/.local/lib/salty-controller/runtime/bin:/home/amos.guest/.local/bin:/usr/local/bin:/usr/bin:/bin
log() { echo "$(date '+%m-%d %H:%M:%S') $*"; }
SPECS=("$@"); declare -A idle=() done_=()
# 已经发过 SIGTERM 时用 NO_TERM=1 重跑本脚本:第二次信号可能让调度程序直接退出、丢掉在跑的轨迹。
[ -z "${NO_TERM:-}" ] && for s in "${SPECS[@]}"; do set -- $s; kill -TERM "$2" && log "SIGTERM $1 pid $2,开始排空"; done
while [ ${#done_[@]} -lt ${#SPECS[@]} ]; do
for s in "${SPECS[@]}"; do
set -- $s; run=$1 pid=$2 cfg=$3 n=$4 jobs=${5:-}
[ -n "${done_[$run]:-}" ] && continue
if kill -0 "$pid" 2>/dev/null; then
busy=$(python3 -c "import sqlite3,sys; print(sqlite3.connect('file:$B/state/jobs.sqlite?mode=ro',uri=True).execute(\"SELECT COUNT(*) FROM rollouts WHERE run_id=? AND state IN ('VERIFYING','CLEANING')\",(sys.argv[1],)).fetchone()[0])" "$run")
if [ -z "$(pgrep -P "$pid")" ] && [ "$busy" = 0 ]; then idle[$run]=$(( ${idle[$run]:-0} + 1 )); else idle[$run]=0; fi
if [ "${idle[$run]}" -ge 10 ]; then
log "$run 旧进程 $pid 已无在跑轨迹 10 分钟仍未退出(卡在导出),kill -9"
kill -9 "$pid"; sleep 5; rm -rf "$B/datasets/$run-training"/.refresh-*
fi
continue
fi
args="--run $run --modes e2e --parallel $n --watch"
[ -n "$jobs" ] && args="$args $(sed 's/^/--job /' "$jobs" | tr '\n' ' ')"
log "$run 旧进程已退出,起新进程:$($C/run-collect.sh "$cfg" $args)"
done_[$run]=1
done
sleep 60
done
log "全部重启完成"

View File

@ -1,61 +0,0 @@
"""Mac mini 主控的健康检查:有问题每条打印一行,健康时什么都不打印。只读。
用法(bench 虚拟机里):python3 scripts/health_check.py RUN [RUN ...]
由本机 scripts/bench_health_alert.sh 每 10 分钟经 ssh 调用,有输出就弹通知。"""
import json
import os
import shutil
import sqlite3
import subprocess
import sys
import time
B = os.environ.get("BENCH_ROOT", "/mnt/data/ming01.wang/model-compile-hbdk/bench")
QUOTA = os.path.expanduser("~/.local/state/bench-shared/quota.sqlite")
INFRA = ("REMOTE_WORKER_FAILED", "STAGE_EXECUTOR_EXIT", "QUEUE_DEADLINE_EXCEEDED")
def problems(runs, now):
ps = subprocess.run(["ps", "-eo", "args"], capture_output=True, text=True).stdout
for run in runs:
if f"collect --run {run} " not in ps:
yield f"{run} 的调度程序没在跑"
c = sqlite3.connect(f"file:{B}/state/jobs.sqlite?mode=ro", uri=True, timeout=30)
q = ",".join("?" * len(runs))
for run, job, kk in c.execute(f"SELECT run_id, job_id, kk FROM rollouts WHERE state='PAUSED_INFRA' AND run_id IN ({q})", runs):
try:
e = json.load(open(f"{B}/runs/{run}/{job}/rollouts/{kk:02d}/execution.json"))
except (OSError, ValueError):
continue
paused = now - (e.get("paused_at") or 0)
if 3600 < paused < 86400: # 正常的暂停几分钟内就会自动续;超过一天的是老问题,不反复报
yield f"{run[-2:]} {job} #{kk} 暂停 {paused // 60:.0f} 分钟没恢复({e.get('stop_reason')})"
for run in runs:
alive = c.execute("SELECT COUNT(*) FROM rollouts WHERE run_id=? AND state='RUNNING'", (run,)).fetchone()[0]
last = c.execute("SELECT MAX(started_at) FROM stages WHERE run_id=?", (run,)).fetchone()[0] or 0
if alive and now - last > 3 * 3600:
yield f"{run} 有 {alive} 条在跑,但 {(now - last) / 3600:.0f} 小时没有开始任何新步骤,调度程序可能卡住了"
n = c.execute(f"SELECT COUNT(*) FROM stages WHERE run_id IN ({q}) AND state='FAIL' AND started_at>?"
f" AND error_class IN ({','.join('?' * len(INFRA))})", (*runs, now - 3600, *INFRA)).fetchone()[0]
if n >= 8:
yield f"最近 1 小时 Salty 容器失败 {n} 次"
n = c.execute(f"SELECT COUNT(*) FROM rollouts WHERE run_id IN ({q}) AND finished_at>?"
" AND state IN ('INFRA_ERROR','ABANDONED','LOST','CANCELLED')", (*runs, now - 3600)).fetchone()[0]
if n >= 5:
yield f"最近 1 小时有 {n} 条轨迹被系统中断"
qc = sqlite3.connect(f"file:{QUOTA}?mode=ro", uri=True, timeout=30)
for group, state, updated in qc.execute(
"SELECT group_id, state, updated_at FROM quota_groups WHERE state IN ('PAUSED','RECONCILING')"
" AND group_id IN (SELECT group_id FROM attempts WHERE started_at>?)", (now - 86400,)):
if now - updated > 2700: # 请求结果不明时本来就会整组核对约半小时,超过 45 分钟才算卡住
yield f"模型账号 {group} 处于 {state} 已 {(now - updated) / 60:.0f} 分钟,用这个账号的轨迹都发不出请求"
free = shutil.disk_usage(B).free / 2**30
if free < 10:
yield f"虚拟机磁盘只剩 {free:.0f} GB"
mem = int(next(l.split()[1] for l in open("/proc/meminfo") if l.startswith("MemAvailable"))) // 1024
if mem < 1024:
yield f"虚拟机可用内存只剩 {mem} MB,调度程序可能被系统杀掉"
if __name__ == "__main__":
for line in problems(sys.argv[1:], time.time()):
print(line)

View File

@ -1,272 +0,0 @@
#!/usr/bin/env python3
"""Move finished rollouts to Salty personal space and build the datasets there; the controller keeps
only small files and the final datasets.
Every 10 minutes:
1. archive: a rollout that ended over 30 min ago (UNVERIFIED: 24 h, it can still be re-verified), whose
execution is TERMINAL and, if PASS, whose content review is settled, goes to
personal:/bench-root/archive/<run>/<job>/<kk>.tgz. Stage products already in personal:/bench-root/runs
with the same name and size are left out. Later changes go to <kk>.u<time>.tgz overlays; an archive is
never rewritten. Then every local file over 1 MB is deleted.
2. export: at most hourly and only after new archives, one Salty job unpacks the archives into personal
runs and runs export/to_agentrl.py there (salty_remote_worker.export); the datasets stay in
personal:/bench-root/datasets/ and are copied to bench/datasets-final/.
The collectors keep their own per-rollout export into bench/datasets/: it finds and reviews new PASS
rollouts while their files are still here, which is what step 1 waits for.
Usage: salty_offload.py RUN=CONFIG [RUN=CONFIG ...] [--once] [--export-now]
"""
from __future__ import annotations
import argparse
import contextlib
import hashlib
import io
import json
import os
import shutil
import sqlite3
import sys
import tarfile
import tempfile
import time
import uuid
from pathlib import Path
from types import SimpleNamespace
PRODUCTS = (".hbm", ".bc", ".npz", ".onnx")
BIG = 1 << 20
def log(message):
print(f"{time.strftime('%F %T')} {message}", flush=True)
def remote_sizes(cli, remote_dir):
rc, out, _ = cli.once(["storage", "ls", remote_dir])
rows = (line.split("\t") for line in out.splitlines()[1:]) if rc == 0 else ()
return {f[0]: int(f[2]) for f in rows if len(f) > 2 and f[1] == "文件"}
def settled(rd, reasons):
"""PASS leaves once the collector's export has judged it: a finished review, or excluded for another reason."""
for receipt in (rd / "content-reviews").glob("*.json"):
with contextlib.suppress(ValueError, OSError):
if json.loads(receipt.read_text()).get("state") == "COMPLETE":
return True
return reasons is not None and not any(r.endswith("content-review-unreviewed") for r in reasons)
def archive(cli, paths, rd):
"""Upload what changed since the last archive of this rollout, then free its local bulk."""
marker = rd / "ARCHIVED.json"
info = json.loads(marker.read_text()) if marker.exists() else {}
started = time.time()
files = [p for p in rd.rglob("*") if p.is_file() and not p.is_symlink() and p != marker]
changed = [p for p in files if p.stat().st_mtime > info.get("at", 0)]
if changed:
mirrored, listings = set(), {}
for p in changed:
if p.suffix in PRODUCTS and p.stat().st_size > BIG:
d = str(p.parent.relative_to(paths.RUNS))
listings.setdefault(d, remote_sizes(cli, f"personal:/bench-root/runs/{d}"))
if listings[d].get(p.name) == p.stat().st_size:
mirrored.add(p)
run, job, kk = rd.parts[-4], rd.parts[-3], rd.name
name = f"{kk}.tgz" if not info else f"{kk}.u{int(started)}.tgz"
remote = f"personal:/bench-root/archive/{run}/{job}/{name}"
with tempfile.TemporaryDirectory(prefix="salty-archive-") as tmp:
tgz = Path(tmp) / name
with tarfile.open(tgz, "w:gz") as tar:
for p in changed:
if p not in mirrored:
tar.add(p, str(p.relative_to(paths.RUNS)))
cli.call("storage", "mkdir", "-p", remote.rsplit("/", 1)[0])
cli.call("storage", "cp", str(tgz), remote, timeout=3600)
size = tgz.stat().st_size
if remote_sizes(cli, remote.rsplit("/", 1)[0]).get(name) != size:
raise RuntimeError(f"uploaded {name} size differs")
entry = {"archive": remote, "bytes": size, "sha256": hashlib.sha256(tgz.read_bytes()).hexdigest(),
"mirrored": sorted(str(p.relative_to(paths.RUNS)) for p in mirrored)}
info = {**entry, **info, "at": started}
if name != f"{kk}.tgz":
info["overlays"] = [*info.get("overlays", []), entry]
marker.write_text(json.dumps(info, ensure_ascii=False, indent=1))
freed = 0
for p in files:
if p.exists() and p.stat().st_size > BIG:
freed += p.stat().st_size
p.unlink()
return len(changed), freed
def archive_all(cli, paths, runs, terminal):
con = sqlite3.connect(f"file:{paths.DB_PATH}?mode=ro", uri=True)
rows = con.execute(f"SELECT run_id, job_id, kk, state, finished_at FROM rollouts WHERE run_id IN "
f"({','.join('?' * len(runs))})", tuple(runs)).fetchall()
con.close()
reasons = {}
for run in runs:
with contextlib.suppress(OSError):
for line in (paths.DATASETS / f"{run}-training/disposition.jsonl").read_text().splitlines():
item = json.loads(line)
reasons[(run, item["job_id"], item["rollout"])] = item["reasons"]
uploaded = 0
for run, job, kk, state, finished in rows:
rd = paths.RUNS / run / job / "rollouts" / f"{kk:02d}"
delay = 86400 if state == "UNVERIFIED" else 1800
if state not in terminal or not finished or finished > time.time() - delay or not rd.is_dir():
continue
marker = rd / "ARCHIVED.json"
at = json.loads(marker.read_text())["at"] if marker.exists() else None
if at is not None and all(p.stat().st_size <= BIG and p.stat().st_mtime <= at for p in rd.rglob("*")
if p.is_file() and not p.is_symlink() and p != marker):
continue # archived, freed and unchanged
execution = rd / "execution.json"
if execution.exists() and json.loads(execution.read_text()).get("execution_state") != "TERMINAL":
continue
if state == "PASS" and not settled(rd, reasons.get((run, job, kk))):
continue
try:
changed, freed = archive(cli, paths, rd)
uploaded += bool(changed)
log(f"{run} {job} {kk}: archived {changed} files, freed {freed >> 20}MB")
except Exception as exc: # next pass retries; nothing local was deleted before the upload was checked
log(f"{run} {job} {kk}: SKIP {type(exc).__name__}: {exc}")
return uploaded
def reap_orphans(cli, paths, sr, runs):
"""A Salty job whose controller attempt already ended (phase deadline, cancel, crash) has no reader:
stop it, and once it has ended remove it and its dir like salty_remote removes its own jobs (their logs and
describe are already in the attempt's stderr.log). 10-13: 12 such jobs held ~64 cores while ours queued."""
rc, out, _ = cli.once(["ps"])
if rc:
return
status = {f[0]: f[2] for f in (line.split("\t") for line in out.splitlines()[1:]) if len(f) > 2}
for run in runs:
attempts = (paths.RUNS / run).glob("*/rollouts/*/stages/*/a*/salty/remote.json")
# bench dev commands (harness/salty_dev.py): ended when their ticket is DONE; nothing to keep for forensics.
dev = (paths.RUNS / run).glob("*/rollouts/*/stages/_queue/terminal-*/salty/remote.json")
for p in [*attempts, *dev]:
state = json.loads(p.read_text())
if state.get("cleaned"): # its own run already removed it; ps can still list it for a moment
continue
s = status.get(state.get("job_id"))
is_dev = p.parent.parent.name.startswith("terminal-")
if is_dev:
try:
ended = json.loads((p.parent.parent / "status.json").read_text()).get("state") == "DONE"
except (OSError, ValueError):
ended = False
else:
ended = (p.parent.parent / "meta.json").exists()
if s not in ("running", "pending", "stopped", "succeeded", "failed") or not ended:
continue
if s in ("stopped", "succeeded", "failed"):
sr._cleanup(cli, state)
else:
cli.call("stop", state["job_id"], ok_404=True) # 404: gone meanwhile, nothing to stop (10-14: 30 min of retries)
log(f"{state['job_id']} {s}, its attempt {p.parent.parent.relative_to(paths.RUNS)} already ended: "
+ ("removed" if s in ("stopped", "succeeded", "failed") else "stop requested"))
def export(cli, paths, sr, policy, runs):
name = "bench-export-" + str(uuid.uuid4())
bench = paths.BENCH_ROOT
with tempfile.TemporaryDirectory(prefix="salty-export-") as tmp:
work = Path(tmp)
with contextlib.closing(sqlite3.connect(paths.DB_PATH)) as src, \
contextlib.closing(sqlite3.connect(work / "jobs.sqlite")) as dst:
src.backup(dst)
files, configs = {str(bench / "state/jobs.sqlite"): work / "jobs.sqlite"}, {}
with contextlib.closing(sqlite3.connect(work / "jobs.sqlite")) as con:
for run, config in runs.items():
configs[run] = str(bench / "export-configs" / f"{run}.yaml")
files[configs[run]] = Path(config)
seed = paths.DATASETS / f"{run}-training/manifest.json"
if seed.exists():
files[str(bench / "export-seed" / f"{run}.json")] = seed
for (job,) in con.execute("SELECT job_id FROM jobs WHERE run_id=?", (run,)):
if (paths.RUNS / run / job / "job.json").exists():
files[str(paths.RUNS / run / job / "job.json")] = paths.RUNS / run / job / "job.json"
request = {"identity": {"export": name}, "stage": "export", "export": configs, "env": {}, "links": {},
"files": {destination: f"files/{i}" for i, destination in enumerate(files)}}
data = json.dumps(request).encode()
with tarfile.open(work / "bundle.tar", "w:gz") as tar: # the worker opens it with tarfile's autodetect
info = tarfile.TarInfo("request.json")
info.size = len(data)
tar.addfile(info, io.BytesIO(data))
for i, source in enumerate(files.values()):
tar.add(source, f"files/{i}")
for d in (*sr.CODE_DIRS, "agent"):
tar.add(bench / d, f"code/{d}", filter=lambda m: None if "__pycache__" in m.name else m)
shutil.copyfile(sr.WORKER, work / "worker.py")
state = {"name": name, "remote": "personal:/" + name}
state["job_id"] = sr._submit(cli, state, policy, SimpleNamespace(name="export"), work,
policy.get("export_memory", "32Gi"))
log(f"export job {state['job_id']} submitted")
job_state = sr._wait(cli, state["job_id"], policy, time.time(), time.time, time.sleep)
if job_state != "succeeded":
sr._remote_logs(policy, state["job_id"], work / "remote.log")
log(f"export job {job_state}, kept for forensics:\n{(work / 'remote.log').read_text()[-4000:]}")
return False
sr._fetch(cli, state["remote"] + "/result/result.json", work / "result.json")
sr._fetch(cli, state["remote"] + "/result/datasets.tar.gz", work / "datasets.tar.gz")
codes = json.loads((work / "result.json").read_text())["exit_codes"]
with tarfile.open(work / "datasets.tar.gz") as tar:
tar.extractall(work / "out", members=[m for m in tar.getmembers() if m.isfile() and ".." not in m.name
and not m.name.startswith("/")])
final = bench / "datasets-final"
final.mkdir(exist_ok=True)
for run, rc in codes.items():
if rc != 0:
log(f"export of {run} failed (exit {rc}); its last good dataset stays")
continue
target, old = final / f"{run}-training", final / f".{run}-training.old"
shutil.rmtree(old, ignore_errors=True)
if target.exists():
target.rename(old)
(work / "out" / f"{run}-training").rename(target)
shutil.rmtree(old, ignore_errors=True)
manifest = json.loads((target / "manifest.json").read_text())
log(f"{run}: sft {manifest['stats']['sft_eligible']} of {manifest['stats']['scanned']} scanned")
sr._cleanup(cli, state)
return all(rc == 0 for rc in codes.values())
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("runs", nargs="+", metavar="RUN=CONFIG")
ap.add_argument("--once", action="store_true", help="one pass, then exit")
ap.add_argument("--export-now", action="store_true", help="export on the first pass even if nothing new was archived")
a = ap.parse_args()
runs = dict(item.split("=", 1) for item in a.runs)
os.environ["BENCH_CONFIG"] = next(iter(runs.values())) # salty CLI and image come from the first config
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from harness import paths, salty_remote as sr
from export.to_agentrl import DB_TERMINAL_STATES
policy = dict(paths.config()["execution"]["salty_remote"])
if "@sha256:" not in policy.get("image", ""):
raise SystemExit("execution.salty_remote.image must be pinned by digest")
cli = sr._Cli(policy.get("salty", "salty"), policy.get("outage_seconds", 1800), time.sleep, time.time, log)
pending, last_export = a.export_now, 0.0
while True:
try:
reap_orphans(cli, paths, sr, runs)
except Exception as exc: # retried next pass
log(f"reap failed: {type(exc).__name__}: {exc}")
pending = archive_all(cli, paths, runs, DB_TERMINAL_STATES) > 0 or pending
if pending and time.time() - last_export >= 3600:
last_export = time.time()
try:
pending = not export(cli, paths, sr, policy, runs)
except Exception as exc: # retried next hour; the last good datasets stay in place
log(f"export failed: {type(exc).__name__}: {exc}")
if a.once:
return 0
time.sleep(600)
if __name__ == "__main__":
sys.exit(main())

View File

@ -1,101 +0,0 @@
#!/usr/bin/env python3
"""Review again the PASS rollouts the last Salty export (bench/datasets-final) left unreviewed or UNKNOWN
whose files are no longer on the controller (salty_offload.py freed them).
One at a time: fetch <kk>.tgz from personal:/bench-root/archive (and the HBM from personal runs if the
archive left it out), review with the configured reviewer, upload what the review wrote as an overlay and
free the files again. Old-format events reach 650 MB, and every collector's export would re-read whatever
sits here, so nothing stays longer than its own review. Stop salty_offload.py first: it would free a
rollout in the middle of its review.
--release-started: a receipt left STARTED by a crash never authorizes a new request (the old one may have
been sent and answered); with this flag the operator accepts a second request: the STARTED bytes go to
content-reviews/history/ and the receipt becomes COMPLETE UNKNOWN (OPERATOR_RELEASED_STARTED_RECEIPT), which
the normal UNKNOWN retry replaces under a new execution id.
Usage: salty_review_again.py RUN CONFIG [--release-started] [--only RUN/JOB/rollouts/KK,...] [--dry-run]
"""
from __future__ import annotations
import hashlib
import json
import os
import subprocess
import sys
import tarfile
import time
from pathlib import Path
WANT = {"content-review-unreviewed", "sft:content-review-unreviewed", "content-review-unknown", "sft:content-review-unknown"}
def main() -> int:
run, config, flags = sys.argv[1], sys.argv[2], sys.argv[3:]
if subprocess.run(["pgrep", "-f", r"^python[0-9.]* .*salty_offload\.py"], capture_output=True).returncode == 0:
raise SystemExit("stop salty_offload.py first")
os.environ["BENCH_CONFIG"] = config
sys.path[:0] = [str(Path(__file__).resolve().parent.parent), str(Path(__file__).resolve().parent)]
from harness import paths, salty_remote as sr
from export.content_review import review_once
import salty_offload
only = set(flags[flags.index("--only") + 1].split(",")) if "--only" in flags else None
policy = paths.config()["execution"]["salty_remote"]
cli = sr._Cli(policy.get("salty", "salty"), 1800, time.sleep, time.time, salty_offload.log)
todo = [(d["source"], d["reasons"][0]) for d in map(json.loads, open(paths.BENCH_ROOT / f"datasets-final/{run}-training/disposition.jsonl"))
if d["reasons"] and set(d["reasons"]) <= WANT and (only is None or d["source"].removeprefix("runs/") in only)]
salty_offload.log(f"{run}: {len(todo)} to review again")
for source, reason in todo:
rd = paths.BENCH_ROOT / source
job, kk = rd.parent.parent.name, rd.name
if "--dry-run" in flags or (rd / "agent/events.jsonl").exists():
salty_offload.log(f"{'would review' if '--dry-run' in flags else 'still local, skip'} {source} {reason}")
continue
remote = f"personal:/bench-root/archive/{run}/{job}/{kk}.tgz"
tgz = Path(f"/tmp/review-again-{job}-{kk}.tgz")
try:
cli.call("storage", "cp", remote, str(tgz), timeout=3600)
with tarfile.open(tgz) as tar:
tar.extractall(paths.RUNS, members=[m for m in tar.getmembers() if m.isfile()
and Path(m.name).parts[:4] == rd.relative_to(paths.RUNS).parts])
hbm = rd / "artifacts" / f"{job}.hbm"
if not hbm.exists():
cli.call("storage", "cp", f"personal:/bench-root/runs/{rd.relative_to(paths.RUNS)}/artifacts/{hbm.name}",
str(hbm), timeout=3600)
os.utime(hbm, (0, 0))
# Restored files keep old mtimes: only what the review writes after `at` goes into the overlay.
marker = rd / "ARCHIVED.json"
info = json.loads(marker.read_text()) if marker.exists() else {"archive": remote}
marker.write_text(json.dumps({**info, "at": time.time()}, ensure_ascii=False, indent=1))
if "--release-started" in flags:
for receipt in (rd / "content-reviews").glob("*.json"):
data = receipt.read_bytes()
record = json.loads(data)
if record.get("state") == "STARTED":
# Close it as a COMPLETE UNKNOWN the operator released: the retry below then gets a new
# execution id. Reusing the old one replays the ledger's finished generation, no new request.
history = receipt.parent / "history" / (hashlib.sha256(data).hexdigest() + ".json")
history.parent.mkdir(mode=0o700, exist_ok=True)
history.write_bytes(data)
receipt.write_text(json.dumps(dict(record, state="COMPLETE", decision="UNKNOWN", finished_at=time.time(),
reason="OPERATOR_RELEASED_STARTED_RECEIPT", issues=[]), ensure_ascii=False))
salty_offload.log(f"{source}: released STARTED receipt {receipt.name}")
try:
result = review_once(rd, managed=True, retry_unknown=True) # also an UNKNOWN written after the export
except ValueError as exc:
if "REVIEW_RETRY_REQUIRES_COMPLETE_UNKNOWN" not in str(exc):
raise
result = review_once(rd, managed=True) # no COMPLETE UNKNOWN receipt: a first review
salty_offload.log(f"{source}: {result.get('state')} {result.get('decision')} {(result.get('reason') or '')[:120]}")
except Exception as exc:
salty_offload.log(f"{source}: FAILED {type(exc).__name__}: {str(exc)[:300]}")
finally:
tgz.unlink(missing_ok=True)
if (rd / "ARCHIVED.json").exists():
changed, freed = salty_offload.archive(cli, paths, rd)
salty_offload.log(f"{source}: overlay {changed} files, freed {freed >> 20}MB")
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@ -1,156 +0,0 @@
"""Speed reference per job: compile the platform's default configuration once (in a bench dev container,
outside any rollout) and write the compiler's latency estimate to agent/speed_ref/<job_id>.json: the agent's jail shows bench/agent
read-only and live (not the job directory), so a reference computed after a rollout started still reaches it.
The agent compares its own model against it (SYSTEM.md 速度要求) instead of compiling the default itself
(10-15: 9 of 32 v17 rollouts spent ~4 h each doing that). Jobs that already have the file are skipped.
BENCH_CONFIG=<a run config> python scripts/speed_ref.py r20260929-04 r20260929-05 r20260929-06 [--active --follow] [--workers 3]
--active: only jobs a v18+ rollout is on. Salty is shared with the live rollouts (10-15 20:40: their own dev
jobs queued once 8 of these were added), so references follow demand instead of compiling all 418 up front.
"""
from __future__ import annotations
import argparse
import json
import shlex
import sys
import time
import traceback
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from types import SimpleNamespace
import yaml
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from harness import paths, salty_dev # noqa: E402
from harness.backends import ptq # noqa: E402
from harness.stages.s01_fetch import weights_dir # noqa: E402
from harness.stages.s03_export import onnx_path # noqa: E402
from harness.stages.s06_hbm import _parse_static_perf # noqa: E402
from harness.stagerun import own_inputs_required # noqa: E402
DEADLINE_S = 10 * 3600
def command(job: dict, ws: Path) -> str:
onnx = onnx_path(job)
weights = weights_dir(job["model"]) # its files are on personal storage, listed in the container
if onnx is None or weights is None:
raise ValueError("weights not fetched yet")
mine = ws / "model.onnx"
calib = Path(job["data"]["calib"]["path"]) / "npy_f32"
cfg = ptq.compile_config(job, Path("model.onnx"), calib, Path("ref"), None)
(ws / "ref.yaml").write_text(yaml.safe_dump(cfg, sort_keys=False))
export = shlex.join(ptq.export_cmd(job, mine, Path("WEIGHTS"))).replace("WEIGHTS", '"$W"')
export = f"W=$(ls -d {shlex.quote(str(weights))}/* | grep -v /meta.json | head -1); {export}"
return (f"set -e; if [ -f {shlex.quote(str(onnx))} ]; then ln -sfn {shlex.quote(str(onnx))} model.onnx; "
f"else {export} > export.log 2>&1; fi; "
"hb_compile -c ref.yaml > ref_compile.log 2>&1 || { tail -40 ref_compile.log; exit 1; }; "
"hb_model_info ref/*.hbm > ref_info.log 2>&1; " # the dev PATH's OE toolchain, as the agent uses it
"sha256sum ref/*.hbm > ref_hbm.sha256; tail -20 ref_info.log; rm -rf ref model.onnx") # keep personal storage small
OUT = paths.AGENT / "speed_ref"
def one(job_dir: Path) -> str:
job = json.loads((job_dir / "job.json").read_text())
jid = job["job_id"]
rd = paths.BENCH_ROOT / "runs" / "zz-speedref" / jid / "rollouts" / "00"
ws = rd / "workspace"
ws.mkdir(parents=True, exist_ok=True)
(rd / "stages").mkdir(exist_ok=True)
(rd.parent.parent / "job.json").write_text(json.dumps(job))
hd = rd / "stages" / "ref"
hd.mkdir(exist_ok=True)
ctx = SimpleNamespace(stage_root=rd / "stages", job=job, job_dir=rd.parent.parent, extra={})
result = salty_dev.execute(ctx, hd, {"command": command(job, ws), "cwd": "."},
time.monotonic() + DEADLINE_S, hd / "out.log", hd / "err.log")
info = (ws / "ref_info.log").read_text(errors="ignore") if (ws / "ref_info.log").exists() else ""
perf = _parse_static_perf(info)
if result.state != "OK" or "latency_ms_est" not in perf:
(rd.parent.parent / "FAILED").write_text((hd / "out.log").read_text(errors="ignore")[-4000:]) # not retried
unavailable(job, f"{DEADLINE_S // 3600} 小时内没编完" if result.exit_code == 124 else f"编译失败,退出码 {result.exit_code}")
return f"{jid}\tFAIL {result.state} {result.exit_code}\t{(hd / 'out.log').read_text(errors='ignore')[-300:]!r}"
sha = (ws / "ref_hbm.sha256").read_text().split()[0] if (ws / "ref_hbm.sha256").exists() else None
ref = {"schema": "bench.speed_ref/1", "job_id": jid, "march": job["march"], "qcfg": job["qcfg"],
"config": "platform default (compile.yaml with no overrides)", **perf, "hbm_sha256": sha,
"measured": "hb_model_info static estimate, not a board run", "computed_at": time.time()}
publish(jid, ref)
return f"{jid}\tOK\t{perf['latency_ms_est']} ms"
def publish(jid: str, ref: dict) -> None:
OUT.mkdir(exist_ok=True)
tmp = OUT / f".{jid}.json.tmp"
tmp.write_text(json.dumps(ref, indent=2, ensure_ascii=False))
tmp.replace(OUT / f"{jid}.json")
def unavailable(job: dict, why: str) -> None:
"""The default configuration did not compile here (10-16: big S600 depth models, >10 h on 6 cores): say so,
so agents stop waiting for the file or compiling the default themselves."""
publish(job["job_id"], {"schema": "bench.speed_ref/1", "job_id": job["job_id"], "march": job["march"], "qcfg": job["qcfg"],
"latency_ms_est": None, "status": "unavailable",
"note": "平台默认配置在这道题上没编出来(" + why + "),本题没有速度参考值:不用再等,也不要自己编默认配置;"
"只要别整模型提到 int16/fp16、别让大段算子落到 CPU,并用 board 阶段实测确认。",
"computed_at": time.time()})
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("runs", nargs="+")
ap.add_argument("--workers", type=int, default=3)
ap.add_argument("--cpu", default="6")
ap.add_argument("--only", action="append", help="job ids")
ap.add_argument("--active", action="store_true", help="only jobs with a rollout under agent v18+")
ap.add_argument("--skip", action="append", default=[], help="job ids (e.g. one another run is doing)")
ap.add_argument("--follow", action="store_true", help="keep scanning every 2 min; at most --workers at a time")
a = ap.parse_args()
cfg = paths.config()
cfg["execution"]["salty_remote"].setdefault("cpu_by_stage", {})["terminal"] = int(a.cpu)
cfg["execution"]["salty_remote"].setdefault("memory_by_stage", {})["terminal"] = "32Gi"
failed = paths.BENCH_ROOT / "runs" / "zz-speedref"
def pending() -> tuple[list[str], dict[str, Path]]:
jobs: dict[str, Path] = {}
started: dict[str, int] = {}
for run in a.runs:
for jd in sorted((paths.BENCH_ROOT / "runs" / run).iterdir()):
if (jd / "job.json").exists() and (not a.only or jd.name in a.only) and (
not a.active or any(own_inputs_required(r) for r in (jd / "rollouts").glob("*"))):
jobs.setdefault(jd.name, jd) # same job id = same model, march and qcfg in every run
started[jd.name] = started.get(jd.name, 0) + len(list((jd / "rollouts").glob("*")))
for j in jobs: # failures recorded before the unavailable file existed (or by an older process)
if (failed / j / "FAILED").exists() and not (OUT / f"{j}.json").exists():
unavailable(json.loads((jobs[j] / "job.json").read_text()), "编译失败或超时")
todo = sorted((j for j in jobs if not (OUT / f"{j}.json").exists() and j not in a.skip),
key=lambda j: started.get(j, 0)) # untouched jobs first: new rollouts go there
return todo, jobs
def safe(jid, jd):
try:
line = one(jd)
except Exception as exc:
line = f"{jid}\tERROR {type(exc).__name__}: {exc}\t{traceback.format_exc(limit=2)[-300:]!r}"
print(time.strftime("%F %T"), line, flush=True)
running: dict[str, object] = {}
with ThreadPoolExecutor(a.workers) as pool:
while True:
running = {j: f for j, f in running.items() if not f.done()}
todo, jobs = pending()
for jid in [j for j in todo if j not in running][:a.workers - len(running)]:
print(time.strftime("%F %T"), "start", jid, flush=True)
running[jid] = pool.submit(safe, jid, jobs[jid])
if not a.follow and not running:
break
time.sleep(120 if a.follow else 5) # --follow: v18 rollouts keep starting on new jobs
return 0
if __name__ == "__main__":
sys.exit(main())

Some files were not shown because too many files have changed in this diff Show More