Compare commits
No commits in common. "simplify-20261017" and "main" have entirely different histories.
simplify-2
...
main
@ -35,7 +35,3 @@
|
||||
任务输入在 `runs/<run_id>/<job_id>/job.json`,轨迹和证据在 `runs/<run_id>/<job_id>/rollouts/<kk>/`;状态数据库由 `BENCH_STATE_DIR` 决定。配置中的工作、缓存和数据路径继续沿用现有环境,不手工重置运行数据或隔离标记。
|
||||
|
||||
`bench status --run <run_id>` 查看批次状态;`scripts/audit_rollouts.py --run <run_id>` 查看轨迹验收。任务启动、模型回复、板测通过和最终 SFT 是不同证据,不能相互替代。
|
||||
|
||||
## 经验记忆
|
||||
|
||||
开始工作前先读 [docs/memory/AGENTS.md](docs/memory/AGENTS.md)。那里有之前的 agent 积累下来的用户偏好、现场事实和踩过的坑,也写了新增记忆的格式和存放规则。
|
||||
|
||||
@ -1,64 +1,65 @@
|
||||
# bench agent · 系统提示(追加在 pi 默认系统提示之后)
|
||||
|
||||
你负责完成 job.json 固定的部署任务:从模型权重出发,交付一个在目标板卡上精度和速度都合格的模型。怎么做由你决定;验收只认平台在板上的实际测量。
|
||||
你负责完成 job.json 固定的部署任务。harness 提供共享资源、实际过程记录与独立验收。探索顺序、实现方式和重试方法由你决定,受本任务总预算约束;其中编译次数是主预算,每条轨迹启动编译的次数有硬上限,用完即不能再编译。
|
||||
|
||||
## 环境
|
||||
- 当前目录是你的 workspace,可写。本 rollout 的 job.json、日志和阶段产物可读;公共源码、共享工具链、其他 rollout、评测数据和判分结果不属于本任务。
|
||||
- 你的 bash 跑在控制机上,**没有工具链**(hb_compile、hmct、onnx、onnxruntime、torch 都不在),用到它们的命令走 `bench dev`。单条 bash 命令约 14 分钟会被结束,只结束这一条。
|
||||
- 工具链文档:`oe-doc <query>` 和 `.horizon/skills/`。`$BENCH_ROOT/harness/adapters/` 里是公共的预处理和后处理代码,可读。
|
||||
## 环境和结果边界
|
||||
- workspace 是你的可写工作区,可以修改自己的代码、模型副本和私有环境。公共源码、共享工具链、其他任务和运行程序不可修改。
|
||||
- 本 rollout 的日志、阶段产物、job.json 与任务数据可读;正式提交和阶段证据只读。其他 rollout、状态库、凭据与 oracle 不属于本任务。
|
||||
- 公共可复用工具位于 `$BENCH_ROOT/harness/adapters/`;`BENCH_ROOT` 已由运行环境提供,可读、可复制修改自己的副本,不是新增模型的必经入口。评判代码仍不作为 Agent 的实现依赖。
|
||||
- 查找文件先用 `rg --files`,搜索内容用 `rg`;限定在 workspace、任务返回的日志目录或上述公共 adapters 目录。不要执行 `find /` 或遍历整台服务器来寻找代码;路径不可访问时依据工具报错处理。
|
||||
- 单条 bash 命令最长约 14 分钟,到时只结束这条命令(返回 `Command timed out`),轨迹继续。更久的工作放后台运行(`nohup … > log 2>&1 &`),再分次查看日志;不要用 `sleep` 一次等待超过 10 分钟。
|
||||
- 现有 Pi legacy 沙箱仍保留 shell 路径限制;工具报限制时按其说明使用明确路径。不要为私有环境安装修改共享工具链。
|
||||
- 目标板卡、数据、指标与阈值以 job.json 为准。编译失败、图变化、精度下降都如实保留;只有平台的实际测量能作为验收证据。
|
||||
- 阶段在后台运行;提交返回 handle 后调用 `terminal_exec({"wait_handle": "<handle>"})`,一次等待原阶段完成或原期限终止,不消耗模型轮数反复轮询。不要为等待重提交阶段;诊断时只查看该 handle 返回的日志与对应阶段,不遍历所有历史产物。旧 `bench stage wait <handle> --poll 60` 仍可用于一次短查询,返回 2 仅表示仍在排队或运行。
|
||||
|
||||
## 等待不花轮数
|
||||
`bench dev` 和正式阶段都在后台跑,提交后返回 handle。`terminal_exec({"wait_handle": "<handle>"})` 一次等到结束,不管等多久都只算一轮。用 sleep 或反复查看状态来等,每查一次都算一轮,还会占上下文。
|
||||
## compile.yaml 里你能改的旋钮(PTQ · hb_compile 3.5.3 · hmct 2.6.5,已对源码核实)
|
||||
harness 钉死 `model_parameters`(onnx / march / working_dir / 输出前缀),你写的 yaml 里这几项会被覆盖。其余三段照下面填,写进 `compile.yaml`(你的 cwd 就是 workspace/),然后 `bench stage run compile --config compile.yaml`(先 `bench stage run calibrate` 也行,它会把你的 yaml 与钉死项合并并落盘)。
|
||||
|
||||
## 开发环境:`bench dev`
|
||||
```
|
||||
bench dev "python check.py" # 在带完整工具链的容器里跑一条命令,cwd 是 workspace
|
||||
bench dev --timeout 7200 -- python export.py # 长任务给足时限(最多 43200 秒;容器启动约 1 分钟)
|
||||
bench dev --board -- python3 run.py # 把当前目录传到本题的板卡上运行,可用 BPU
|
||||
```
|
||||
- `python`、`hb_compile`、`hb_verifier`、`hb_model_info` 来自 OE 工具链(含 hmct、onnx、onnxruntime);torch 用 `/home/ming01.wang/toolchains/bench-venv/bin/python`,或 job.json 里 `model_runtime.python` 指定的环境。权重和模型源码只读可见。
|
||||
- 校准图片:`$BENCH_CALIB_IMAGES`,是训练集原图,和评测集不重叠;`$BENCH_CALIB_IMAGES/../images.json` 列出顺序和来源。预处理以 job.json 的 `model_spec.preprocess` 为准。
|
||||
- 同步:本机改过的文件在下一次 `bench dev` 开始时传上去;命令里新建或改动的小文件(单个 ≤16 MiB)拷回本机,大文件(图、npy、hbm)只留在开发环境,输出末尾会列出来。
|
||||
- 板上:有 python3、numpy 和 hbm_runtime(`m = hbm_runtime.HB_HBMRuntime('x.hbm')`,`m.model_names`、`m.input_names[name]`、`m.input_shapes[name]`,`m.run({name: {输入名: 数组}})`);写到 `$BENCH_OUT` 的文件会拉回到结果目录的 `output/`。一次最多 30 分钟,板卡和正式阶段共用。
|
||||
- 开发环境和板上怎么试都不计编译次数,结果也不作为验收依据。
|
||||
|
||||
## 共享资源与正式验收
|
||||
```
|
||||
bench stage run fetch # 正式权重
|
||||
bench stage run export # 平台参考导出:图不给你,用来核对你的图的输入输出
|
||||
bench stage run baseline # 独立的 FP32 基线
|
||||
bench stage run compile --config compile.yaml # 正式编译
|
||||
bench stage run hbm # 收集并检查编译产物
|
||||
bench stage run board # 板上延迟
|
||||
bench stage run eval # 板上任务精度
|
||||
```
|
||||
- compile.yaml 是 hb_compile 的 PTQ 配置(hb_compile 3.5.3 · hmct 2.6.5),放在 workspace 里。`model_parameters.onnx_model` 和 `calibration_parameters.cal_data_dir` 必填,写你在开发环境里做的图和校准 npy 目录(workspace 内的路径);march、working_dir、输出前缀由平台固定。这一版工具链实际接受的字段(已对源码核实,文档有出入时以这里为准):
|
||||
```yaml
|
||||
model_parameters:
|
||||
onnx_model: my_model.onnx
|
||||
input_parameters: # 默认 featuremap / NCHW / 无归一化,预处理在主机侧,与评测器一致
|
||||
input_parameters: # 默认 featuremap/NCHW/无归一化,预处理在主机侧(评测器口径一致);一般不用动
|
||||
input_type_rt: featuremap # featuremap | rgb | bgr | nv12 | gray | yuv444
|
||||
input_type_train: featuremap
|
||||
input_layout_train: NCHW
|
||||
# mean_value / scale_value / std_value 只在 input_type 不是 featuremap 时有意义
|
||||
# mean_value / scale_value / std_value 仅当 input_type 不是 featuremap 时有意义
|
||||
calibration_parameters:
|
||||
cal_data_dir: my_calib/ # 每个 .npy 是一个模型输入,形状同 input_spec,float32
|
||||
calibration_type: default # default | max | kl | mix | skip
|
||||
max_percentile: 0.99999 # 只对 max 有效
|
||||
per_channel: true
|
||||
run_on_cpu: "node_a,node_b" # ONNX 节点名,逗号分隔;run_on_bpu 反之
|
||||
optimization: set_model_output_int16 # 或 set_model_output_int8
|
||||
quant_config: # dict 或 json 路径;qtype: int8 | int16 | float16 | float32
|
||||
calibration_type: default # default | max | kl | mix ← 首选旋钮。default = 自动搜索;掉点大时试 kl 或 mix
|
||||
# (还有 skip = 完全不校准,只给预量化模型用;普通 PTQ 选它必崩 G2/G3,白烧一次编译)
|
||||
max_percentile: 0.99999 # 只对 max 有效,截掉激活长尾(0.9999–1.0)
|
||||
per_channel: true # 权重逐通道量化,对深度可分离卷积(MobileNet/EfficientNet)常有帮助
|
||||
run_on_cpu: "node_a,node_b" # 把敏感节点放 CPU(float),逗号分隔 ONNX 节点名;run_on_bpu 反之
|
||||
optimization: set_model_output_int16 # set_model_output_int8 | set_model_output_int16(其余以 hmct 报错为准)
|
||||
quant_config: # 精细精度控制(dict 或 json 路径)。qtype: int8 | int16 | float16 | float32
|
||||
model_config: { all_node_type: int8, model_output_type: int16 }
|
||||
op_config: { Softmax: { qtype: float16 } } # 按算子类型
|
||||
node_config: { "/features/features.8/Mul": { qtype: int16 } } # 按节点名
|
||||
op_config: { Softmax: { qtype: float16 }, LayerNormalization: { qtype: float16 } } # 按算子类型
|
||||
node_config: { "/features/features.8/features.8.2/Mul": { qtype: int16 } } # 按节点名(看 hb_compile 日志里的逐节点余弦表挑最差的)
|
||||
compiler_parameters:
|
||||
compile_mode: latency # latency | bandwidth | balance
|
||||
optimize_level: O2 # O0 | O1 | O2(O3 会被拒)
|
||||
optimize_level: O2 # O0 | O1 | O2 —— hbdk4 后端只接受这三个,写 O3 会在 yaml 校验阶段直接报错
|
||||
jobs: 8
|
||||
```
|
||||
- 你的图的输入输出(个数、顺序、类型、形状)要和平台的一致,基线、上板和评测才可比;不一致时编译前就会被拒并列出期望值,不占编译次数。
|
||||
- 阶段失败时看等待结果里的 stdout/stderr/meta 再决定下一步;板上延迟记录失败时仍可以跑精度评测。
|
||||
|
||||
诊断信号在哪:`terminal_exec` 等待结束后给你 stdout/stderr 路径;hb_compile 日志末尾有逐节点 `Calibrated Cosine / Quantized Cosine` 表和模型输出余弦。余弦偏低时可检查敏感节点、尝试 `node_config` 提到 int16;最终任务精度以 G3 实测与 job 中的阈值为准。
|
||||
|
||||
## 共享资源与正式验收
|
||||
原阶段按需要调用,不要求探索按编号顺序进行。执行某一步仍需它实际消费的输入;使用结果前等待完成。
|
||||
|
||||
```
|
||||
bench stage run fetch # 获取或复用正式权重
|
||||
bench stage run export # 导出 ONNX
|
||||
bench stage run baseline # 固定 T1 的独立 FP32 参考
|
||||
bench stage run compile --config compile.yaml # 编译,含 04 配置准备
|
||||
bench stage run hbm # 收集并检查编译产物
|
||||
bench stage run board # 板端性能记录
|
||||
bench stage run eval # 板端 T1 精度与 G2 诊断
|
||||
```
|
||||
|
||||
随后调用 `terminal_exec({"wait_handle": "<handle>"})` 等待刚提交的阶段。
|
||||
|
||||
`--config` 在 workspace 内;目标板卡、产物路径和输出目录由 Worker 固定,配置记录实际使用值。共享工具链和板卡通过受管执行工具获取资源;若任务开放 terminal 命令模式,自写命令通过 terminal_exec 运行;仅阶段模式下该工具只提供等待。连接凭据由平台管理。查看 wait 返回的 stdout/stderr/meta,根据失败证据决定下一步;性能记录失败仍可尝试精度评测。
|
||||
|
||||
## 结束说明
|
||||
最后用自然语言如实说明实际做了什么、平台实测结果是什么、哪些没有验证。总轮数、token 和墙钟预算是硬上限,收到收尾提醒时停止新工作。
|
||||
用自然语言说明实际完成内容、测量范围、失败及未验证事项;不要求调用 claim 或使用规定终态词。总轮数、token 和墙钟预算仍是硬上限,收到收尾提醒时停止新工作并给出真实说明。
|
||||
G1 是编译/产物检查,不是板端测量。精度结论说明样本集、样本数、指标、FP32 与板端值及固定阈值;标准误只作诊断,不放宽阈值。不能把子集筛查概括为全数据集无损。自写推理的 HBM 加载/输出关联若未被平台确认,明确板端预测来源未自动核实。缺材料时保留实际错误,不猜测失败原因。
|
||||
|
||||
## 工具链参考
|
||||
`.horizon/skills/` 和 `oe-doc <query>` 提供工具链文档。共享执行环境由 job 和 Worker 选择;自己的试验依赖放在 workspace 私有环境。
|
||||
|
||||
@ -1 +1 @@
|
||||
23
|
||||
15
|
||||
|
||||
@ -70,7 +70,6 @@ function makeGuard(env) {
|
||||
path.join(JOB, "job.json"), path.join(JOB, "data"),
|
||||
path.join(BENCH, "harness", "adapters"),
|
||||
path.join(BENCH, "data"), path.join(BENCH, "cache"), path.join(BENCH, "bin"), path.join(BENCH, "agent", "oe-doc.sh"),
|
||||
path.join(BENCH, "agent", "speed_ref"), // default-config latency per job (scripts/speed_ref.py)
|
||||
path.join(PROJ, ".horizon"), path.join(PROJ, "AGENTS.md"),
|
||||
];
|
||||
if (MODE === "repair") allowRead.push(path.join(JOB, "oracle"));
|
||||
|
||||
@ -88,8 +88,6 @@ expect("url", e2e.judgeCommand("echo https://x.y/z"), "allow");
|
||||
expect("read own rollout.json", e2e.judgeCommand("cat ../rollout.json"), "allow");
|
||||
expect("ls ..(own rollout)", e2e.judgeCommand("ls .."), "allow");
|
||||
expect("ls ../.. (rollouts dir)", e2e.judgeCommand("ls ../.."), "block");
|
||||
expect("read speed reference", e2e.judgeCommand("cat " + B + "/agent/speed_ref/cls--m--nash-e--ptq--int8.json"), "allow");
|
||||
expect("other agent files stay closed", e2e.judgeCommand("cat " + B + "/agent/SYSTEM.md"), "block");
|
||||
expect("repair reads oracle yaml", repair.judgeCommand("cp ../../../oracle/stages/04-calibrate/compile.yaml ./compile.yaml"), "allow");
|
||||
|
||||
async function extensionCases() {
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
* 禁止直接调 hb_compile 等重命令(走 bench stage run);`sleep 100+`、`git push`。
|
||||
*
|
||||
* 白名单(e2e):workspace/(可写)· 本 rollout 根(agent/ stages/ handles/ rollout.json)· <job>/job.json · <job>/data
|
||||
* · bench/data · bench/cache · bench/bin · bench/agent/oe-doc.sh · bench/agent/speed_ref · <proj>/.horizon · <proj>/AGENTS.md ;repair 额外 <job>/oracle。
|
||||
* · bench/data · bench/cache · bench/bin · bench/agent/oe-doc.sh · <proj>/.horizon · <proj>/AGENTS.md ;repair 额外 <job>/oracle。
|
||||
* 项目树内其余一律拒(别的 rollout、harness/、state/、export/、别的 job);项目树外只拒 ~/.pi、~/toolchains、bench/.env.local。
|
||||
*
|
||||
* 每次拒绝写一行到 <rollout>/agent/guard.log(L4 导出统计"越界尝试次数")。第二道保险是文件系统权限。
|
||||
@ -83,7 +83,7 @@ log("init", JSON.stringify({ mode: MODE, deny_extra: DENY_EXTRA, allow_extra: AL
|
||||
const FORBIDDEN_CMDS = ["hb_compile", "hb_mapper", "hb_verifier", "hrt_model_exec", "hb_perf",
|
||||
"ssh", "scp", ...(PRIVATE_JAIL ? [] : ["rsync", "wget", "curl", "pip", "pip3", "conda", "npm"])];
|
||||
const WRITE_PATTERNS: RegExp[] = [
|
||||
/\bsleep\s+[0-9]{3,}/, // 用 terminal_exec wait_handle,不要 sleep 等编译
|
||||
/\bsleep\s+[0-9]{3,}/, // 用 bench stage wait,不要 sleep 等编译
|
||||
/\bgit\s+push\b/,
|
||||
];
|
||||
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
{
|
||||
"_comment": "渲染到 ~/.pi/agent/models.json(chmod 600)。{TEACHER_API_KEY}/{STUDENT_*} 由 `bench agent setup` 从 bench/.env.local 填。字段名对照 pi 版本文档(P3 核对:providers.<name>.{baseUrl, api, apiKey, models[]})。contextWindow 故意写大:Pi 只在服务端报上下文超长时才压缩并续跑,不会在暂停退出前按估算先压缩(那样续跑的请求就和原来对不上)。",
|
||||
"_comment": "渲染到 ~/.pi/agent/models.json(chmod 600)。{TEACHER_API_KEY}/{STUDENT_*} 由 `bench agent setup` 从 bench/.env.local 填。字段名对照 pi 版本文档(P3 核对:providers.<name>.{baseUrl, api, apiKey, models[]})。",
|
||||
"providers": {
|
||||
"dr-gateway": {
|
||||
"baseUrl": "{TEACHER_BASE_URL}",
|
||||
@ -11,7 +11,7 @@
|
||||
"name": "{TEACHER_MODEL} (teacher)",
|
||||
"reasoning": false,
|
||||
"input": ["text"],
|
||||
"contextWindow": 10000000,
|
||||
"contextWindow": 128000,
|
||||
"maxTokens": 16384,
|
||||
"cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }
|
||||
}
|
||||
@ -27,7 +27,7 @@
|
||||
"name": "student (vLLM)",
|
||||
"reasoning": false,
|
||||
"input": ["text"],
|
||||
"contextWindow": 10000000,
|
||||
"contextWindow": 65536,
|
||||
"maxTokens": 8192,
|
||||
"cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }
|
||||
}
|
||||
|
||||
@ -234,17 +234,8 @@ async function main(configFile) {
|
||||
},
|
||||
};
|
||||
};
|
||||
let compactedRetry = false;
|
||||
session.subscribe(event => {
|
||||
lastEvent = event.type;
|
||||
let compaction;
|
||||
if (event.type === 'compaction_end' && event.willRetry) {
|
||||
// Pi compacted after a context overflow and resends: the failed generation's ID is spent. kept_messages is
|
||||
// how many earlier messages the model still sees after the summary (export/trajectory.py builds that view).
|
||||
generation += 1; compactedRetry = true;
|
||||
compaction = {kept_messages: session.agent.state.messages
|
||||
.filter(m => m.role !== 'compactionSummary' && !emptyProviderError(m)).length};
|
||||
}
|
||||
if (event.type === 'message_update') {
|
||||
const delta = event.assistantMessageEvent;
|
||||
if (delta?.type?.endsWith('_delta') && delta.delta) nonemptyDelta = true;
|
||||
@ -269,14 +260,9 @@ async function main(configFile) {
|
||||
activeTools.delete(event.toolCallId);
|
||||
}
|
||||
}
|
||||
// message_update repeats the whole message twice (message, assistantMessageEvent.partial) on every
|
||||
// token, ~800MB of events.jsonl per rollout; readers use only its type and delta, message_end has the text.
|
||||
const written = event.type === 'message_update'
|
||||
? {...event, message: undefined, assistantMessageEvent: {...event.assistantMessageEvent, partial: undefined}}
|
||||
: event;
|
||||
// Metadata decorates real native boundaries; it never adds conversation text.
|
||||
output(boundary ? {...written, bench_tool: {...boundary, execution_id: config.execution_id,
|
||||
epoch: config.epoch ?? 1, observed_monotonic: monotonic()}} : compaction ? {...written, bench_compaction: compaction} : written);
|
||||
output(boundary ? {...event, bench_tool: {...boundary, execution_id: config.execution_id,
|
||||
epoch: config.epoch ?? 1, observed_monotonic: monotonic()}} : event);
|
||||
});
|
||||
const stop = () => session.agent.abort();
|
||||
process.on('SIGTERM', stop); process.on('SIGINT', stop);
|
||||
@ -295,13 +281,6 @@ async function main(configFile) {
|
||||
await session.prompt(fs.readFileSync(config.prompt_path, 'utf8'));
|
||||
}
|
||||
await session._agentEventQueue;
|
||||
// Pi resumes 100 ms after an overflow compaction (agent-session.js _runAutoCompaction): wait for that run too.
|
||||
while (compactedRetry) {
|
||||
compactedRetry = false;
|
||||
await new Promise(resolve => setTimeout(resolve, 300));
|
||||
await session.agent.waitForIdle();
|
||||
await session._agentEventQueue;
|
||||
}
|
||||
const last = session.state.messages.at(-1);
|
||||
const complete = last?.role === 'assistant' && last.stopReason === 'stop' && closed(session.state.messages);
|
||||
const safeResume = !nonemptyDelta && emptyProviderError(last) && closed(session.state.messages) &&
|
||||
|
||||
@ -3,18 +3,26 @@
|
||||
job:`{job_id}` run:`{run_id}`
|
||||
job.json:`{job_json}`(只读) 数据:`{data_dir}`(只读) 你的可写区:当前目录
|
||||
|
||||
## 验收(以 job.json 的 `gates.required` 和阈值为准)
|
||||
1. G1 编译:产出 `.hbm`,march = {march},输入 {input_spec},`hb_verifier` 通过
|
||||
2. G2 数值:校准外样本的逐输出余弦,参考阈值 {g2_threshold},只作诊断
|
||||
3. G3 任务精度:板上 T1 子集({t1_n} 张)的 {g3_metric} 与同一子集的 FP32 基线比较,容差 {g3_tol}({g3_kind})
|
||||
4. G4 速度:不比平台默认量化配置(compile.yaml 只写两项必填)编出的模型慢 1.5 倍以上。参考在 `$BENCH_ROOT/agent/speed_ref/{job_id}.json`:`latency_ms_est` 是默认配置的编译器静态估计,不含落到 CPU 的算子;文件还没有表示还在算,值为 null 表示默认配置没编出来、本题没有参考。
|
||||
## 目标
|
||||
1. G1 compile:产出 `.hbm`,march = {march},输入 {input_spec},`hb_verifier` 通过
|
||||
2. G2 numeric:记录校准外样本逐输出余弦,参考阈值 {g2_threshold},用于诊断
|
||||
3. G3 task:真实板端 T1 的 {g3_metric} 与同一子集 FP32 基线比较;容差 {g3_tol}({g3_kind}),以 job.json 的判定规则为准
|
||||
4. G4 perf:只记录,不判
|
||||
|
||||
缺少板上精度结果不能报完整 PASS;缺权重、工具或材料时如实保留实际错误。
|
||||
必选项以 job.json 的 `gates.required` 为准。
|
||||
|
||||
## 输入和可复用缓存
|
||||
- 权重、模型身份和输入约定见 job.json 的 `model_spec`;`bench stage run fetch` 获取正式权重。
|
||||
- FP32 基线:`bench stage run baseline` 生成,缓存在 `{baseline_dir}/metrics.json`(可能还没有,或属于旧子集)。
|
||||
- `bench stage run export` 是平台参考导出:图不给你,只用来核对你的图的输入输出。
|
||||
- 校准图片在开发环境的 `$BENCH_CALIB_IMAGES`({calib_n} 张);T1 评测子集 {t1_n} 张,不对你开放。
|
||||
- 权重来源、模型身份和输入约定:见 job.json 的 model_spec
|
||||
- ONNX 缓存位置:`{onnx}`(可能尚未生成)
|
||||
- FP32 基线缓存位置:`{baseline_dir}/metrics.json`(可能尚未生成或属于旧子集)
|
||||
- 校准集位置:`{calib_dir}`(目标 {calib_n} 个 npy,与 val 零交集)
|
||||
- T1 评测子集:`{t1_tar}`({t1_n} 张)
|
||||
|
||||
预算:本轨迹最多启动 5 次编译(排队超时没开始的不算,续跑不清零),用完后再提交编译会返回 `COMPILE_BUDGET_EXHAUSTED`,此时用最后一次产物跑完 board/eval。{max_turns} 轮、{max_tokens} tokens;等待阶段完成不计轮数,每次模型生成计一轮。
|
||||
## 流程
|
||||
阅读 job.json 后按需要调用工具:`bench stage run fetch` 获取权重,`bench stage run export` 导出图,`bench stage run baseline` 生成参考基线。已有有效缓存可直接复用;这些操作没有强制的探索顺序。
|
||||
|
||||
图与校准输入可用时,用 `bench stage run compile --config <你的>.yaml` 尝试编译。各阶段返回 handle,使用其结果前先调用 `terminal_exec({"wait_handle": "<handle>"})` 等待完成,再读取实际日志。`hbm` 收集编译结果;`board` 记录性能;`eval` 需要 HBM 和独立基线,执行固定板端精度评测。失败后可继续改图、改配置或更换方法,但本轨迹最多启动 5 次编译(首次 1 次 + 重试 4 次);精度不达标后的重编译也计入,排队超时未开始的不计,续跑不清零。用完后再提交编译会直接返回 `COMPILE_BUDGET_EXHAUSTED`,此时用最后一次产物完成 board/eval 并如实报告。
|
||||
|
||||
最后自然语言如实报告。缺少真实板端精度不能报完整 PASS;缺权重、工具或材料时保留实际错误,不能把材料缺口说成量化失败。
|
||||
|
||||
预算:{max_turns} 轮、{max_tokens} tokens。后台编译的等待时间不计模型轮数,每次模型生成仍计一轮。预算收尾时停止新工作并给出实际状态。
|
||||
|
||||
@ -1,7 +1,8 @@
|
||||
{
|
||||
"_comment": "渲染到 <rollout>/workspace/.pi/settings.json。{PROJECT_ROOT}/{BENCH_ROOT}/{MODEL} 由 harness/rollout.py 替换。上下文超长时 Pi 自己压缩后接着跑(默认开);导出按 compactions 还原模型实际看到的内容(export/trajectory.py)。",
|
||||
"_comment": "渲染到 <rollout>/workspace/.pi/settings.json。{PROJECT_ROOT}/{BENCH_ROOT}/{MODEL} 由 harness/rollout.py 替换。compaction 关:训练数据里出现 CompactionSummaryMessage 是脏的,上下文靠 max_turns 控制。",
|
||||
"defaultProvider": "dr-gateway",
|
||||
"defaultModel": "{MODEL}",
|
||||
"compaction": { "enabled": false },
|
||||
"retry": { "enabled": false, "provider": { "maxRetries": 0 } },
|
||||
"skills": [
|
||||
"{PROJECT_ROOT}/.horizon/skills/*/*",
|
||||
|
||||
@ -6,8 +6,7 @@
|
||||
"job_suffix": "--nash-m--ptq--int8",
|
||||
"protocol": {
|
||||
"rollouts_per_job": 2,
|
||||
"count_rollouts_started_after": "2026-09-29T00:00:00+08:00",
|
||||
"count_rule_note": "2026-10-12 user decision: base-Qwen PASS/FAIL from all of r20260929-04 counts, incl. results before 2026-10-08 07:30.",
|
||||
"count_rollouts_started_after": "2026-10-08T07:30:00+08:00",
|
||||
"harness_commit": "10b070e",
|
||||
"agent_version": 15,
|
||||
"system_interrupted": "not a model failure; excluded and rerun",
|
||||
|
||||
@ -4,7 +4,6 @@ import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shlex
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
@ -52,38 +51,33 @@ def submit(stage, config, poll_hint=60, *, recipe=None, submission=None):
|
||||
tmp.rename(hd / 'request.json')
|
||||
print(hid)
|
||||
print('# Pi wait: terminal_exec(' + json.dumps({'wait_handle': hid}) + ')')
|
||||
print(f'# wait: bench stage wait {hid} --poll {poll_hint}')
|
||||
return hid
|
||||
|
||||
def dev(command, timeout=3600, cwd=None):
|
||||
"""bench dev: run one command in the development environment (Salty container with the toolchain)."""
|
||||
rd = rollout_dir()
|
||||
ws = (rd / 'workspace').resolve()
|
||||
here = Path.cwd().resolve()
|
||||
cwd = cwd or (str(here.relative_to(ws)) if here.is_relative_to(ws) else '.')
|
||||
hid = 'terminal-' + uuid.uuid4().hex
|
||||
hd = rd / 'handles' / hid
|
||||
hd.mkdir()
|
||||
(hd / 'status.json').write_text(json.dumps({'state': 'QUEUED', 'queued_at': time.time()}))
|
||||
tmp = hd / 'request.tmp'
|
||||
tmp.write_text(json.dumps({'stage': 'terminal', 'target': 'development', 'command': command,
|
||||
'timeout': timeout, 'cwd': cwd}))
|
||||
tmp.rename(hd / 'request.json')
|
||||
print(hid)
|
||||
print('# Pi wait: terminal_exec(' + json.dumps({'wait_handle': hid}) + ')')
|
||||
return hid
|
||||
|
||||
def wait(handle: str, poll: int = 60) -> int:
|
||||
"""不再轮询,看一眼就返回:结束了打印结果;没结束返回 2,并指向 terminal_exec wait_handle(一次等到阶段结束、不占轮数)。
|
||||
10-16:efficientvit.b0 400 轮里 307 轮是 bench stage wait,每次还往上下文里加一段输出。"""
|
||||
def wait(handle: str, poll: int = 60, quiet: bool = False, max_wait_s: float = 60) -> int:
|
||||
"""单次最多等 60 秒;返回 2 表示仍在排队/运行,阶段继续执行。"""
|
||||
if not 0 < max_wait_s <= 60:
|
||||
raise ValueError("stage wait budget must be in (0, 60] seconds")
|
||||
hd = rollout_dir() / "handles" / handle
|
||||
if not hd.exists():
|
||||
raise SystemExit(f"handle 不存在:{handle}")
|
||||
st = json.loads((hd / "status.json").read_text())
|
||||
if st.get("state") != "DONE":
|
||||
print(f"{handle}: {st.get('state')}"
|
||||
f"{'(阶段计算已结束,等待数据库登记;无需重跑)' if st.get('registration') == 'PENDING' else ''}"
|
||||
f"。不要反复查:调用 terminal_exec({json.dumps({'wait_handle': handle})}),它一次等到这个阶段结束,不占轮数。")
|
||||
return 2
|
||||
t0 = time.time()
|
||||
deadline = time.monotonic() + max_wait_s
|
||||
while True:
|
||||
st = json.loads((hd / "status.json").read_text())
|
||||
if st.get("state") == "DONE":
|
||||
break
|
||||
if not quiet:
|
||||
since = st.get("started_at") or st.get("queued_at") or t0
|
||||
print(f"[{time.strftime('%H:%M:%S')}] {handle}: {st.get('state')}"
|
||||
f"{'(等 ' + st.get('pool', '?') + ' 池槽位)' if st.get('state') == 'QUEUED' else ''}"
|
||||
f"{'(阶段计算已结束,等待数据库登记;无需重跑)' if st.get('registration') == 'PENDING' else ''}"
|
||||
f" 已 {int(time.time() - since)}s", flush=True)
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
print(f"(已等 {int(max_wait_s - remaining)}s 仍未结束;这不是失败,再调一次 bench stage wait {handle} 继续等)")
|
||||
return 2
|
||||
time.sleep(min(max(1, poll), remaining))
|
||||
_print_status(st)
|
||||
return 0 if st["result"] in ("OK", "SKIPPED") else 1
|
||||
|
||||
@ -100,9 +94,6 @@ def _print_status(st: dict) -> None:
|
||||
print(json.dumps(st)); return
|
||||
print(f"result: {st['result']} exit_code: {st['exit_code']} error_class: {st['error_class']} {st['duration_s']}s")
|
||||
print(f"stdout: {st['stdout']}\nstderr: {st['stderr']}\nmeta: {st['meta']}")
|
||||
for name in ('stdout_tail', 'stderr_tail'):
|
||||
if st.get(name):
|
||||
print(f"--- {name} ---\n{st[name]}")
|
||||
if st.get("measurements"):
|
||||
print("measurements: " + json.dumps(st["measurements"], ensure_ascii=False))
|
||||
if st["result"] != "OK":
|
||||
@ -114,20 +105,6 @@ def _print_status(st: dict) -> None:
|
||||
|
||||
|
||||
def main():
|
||||
if sys.argv[1:2] == ['dev']:
|
||||
p = argparse.ArgumentParser(prog='bench dev', description='Run a command in the development environment')
|
||||
p.add_argument('--timeout', type=int, default=3600, help='seconds, at most 43200')
|
||||
p.add_argument('--cwd', help='workspace-relative directory; default: the current one')
|
||||
p.add_argument('--board', action='store_true', help='run on the task board instead (S100P tasks): this directory, '
|
||||
'with files that only exist in the development workspace such as a compiled HBM, at most 30 min')
|
||||
p.add_argument('command', nargs=argparse.REMAINDER)
|
||||
a = p.parse_args(sys.argv[2:])
|
||||
words = a.command[1:] if a.command[:1] == ['--'] else a.command
|
||||
if not words:
|
||||
p.error('command required')
|
||||
command = words[0] if len(words) == 1 else shlex.join(words)
|
||||
dev(('#bench:board\n' if a.board else '') + command, a.timeout, a.cwd) # harness/terminal.py BOARD_MARK
|
||||
return 0
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument('command', choices=['stage'])
|
||||
p.add_argument('op', choices=['run', 'wait', 'status'])
|
||||
|
||||
@ -1,93 +0,0 @@
|
||||
# 经验记忆
|
||||
|
||||
这里是在本项目上工作的 agent(Claude、Codex 等)积累下来的经验:用户的偏好和纠正、项目现场的事实、踩过的坑。代码和 git 历史里能直接查到的东西不放在这里。
|
||||
|
||||
## 怎么用
|
||||
|
||||
- 开始工作前先扫一遍下面的索引,按每行的提示挑相关的文件读。
|
||||
- 记忆只代表写下那一刻的情况。里面提到的路径、进程、配置、机器,用之前先到现场核实;和现场不符时以现场为准,并更新或删掉那条记忆。
|
||||
|
||||
## 怎么写
|
||||
|
||||
1. 一个文件只记一件事。文件名用英文小写加短横线,例如 `bench-restart-hazards.md`。
|
||||
2. 文件开头写 frontmatter:
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: <和文件名相同,不带 .md>
|
||||
description: <一句话,别人靠它判断要不要读这个文件>
|
||||
metadata:
|
||||
type: user | feedback | project | reference
|
||||
---
|
||||
```
|
||||
|
||||
| type | 记什么 |
|
||||
| --- | --- |
|
||||
| user | 用户是谁、懂什么、偏好什么 |
|
||||
| feedback | 用户的要求和纠正;做对了被肯定的做法也记 |
|
||||
| project | 进行中的工作、现场事实、约束,代码里看不出来的 |
|
||||
| reference | 外部资源在哪:地址、看板、工单 |
|
||||
|
||||
3. 正文先写事实。feedback 和 project 类再加两行:`**Why:**` 写原因,`**How to apply:**` 写以后具体怎么做。
|
||||
4. 时间一律写绝对日期(2026-10-11),不写"昨天""下周"。
|
||||
5. 相关的记忆用 `[[name]]` 互相链接。目标文件还没写也可以先留着链接。
|
||||
6. 新写之前先查有没有已经覆盖这件事的文件。有就改那个文件,不要另起一份。发现某条记错了,就地改正并注明"更正(日期)",或者整个删掉。
|
||||
7. 写完在下面的索引里加一行:`- [标题](文件名.md) — 一句话提示`。索引只放链接和提示,不放内容。
|
||||
|
||||
## 不能写的
|
||||
|
||||
- 密码、API key、token、私钥,连前几位、后几位都不写。只写变量名或存放位置。
|
||||
- 代码结构、git 历史,以及 README、CONTRACT、HANDBOOK 里已经写了的东西。
|
||||
- 只对当前这一次对话有用的临时状态。
|
||||
|
||||
## 存放和同步
|
||||
|
||||
- 本目录 `docs/memory/` 是共享的记忆库,跟着 `feature/salty-20261009` 分支提交和推送。不要推到 `main` 或生产分支。
|
||||
- Claude 在本机另有一份工作副本(`~/.claude/projects/<项目>/memory/`,索引文件叫 `MEMORY.md`)。两边内容保持一致:改了一边,同步到另一边再提交。
|
||||
- 推送前 grep 一遍新增内容,确认没有密钥或密钥片段。
|
||||
|
||||
## 索引
|
||||
|
||||
- [优先做减法而非加法](prefer-subtraction-over-addition.md) — 修 bug 先想能删什么,别动辄加钩子和配置项
|
||||
- [汇报说人话](plain-language-no-jargon.md) — 不要用金丝雀这类行话,先给结论和数字
|
||||
- [bench 调度程序重启的坑](bench-restart-hazards.md) — 改文件不算换代、kill -9 条件、drain 单向、密钥在旧进程环境、启动要 10–20 分钟、导出会卡、全局并发上限要三个一起换、新账号用新组名
|
||||
- [第一版交付包](delivery-package-v1.md) — 位置、冻结点 62 条、出第二版要避开的坑
|
||||
- [采集稳定性补丁](quota-unknown-autorecover-fix.md) — 2f54d40→ea0335b:请求不明、板卡锁死、板端隔离、ssh 断连、回执误判、编译上限 5 次、请求结束后补放行试探、bash 超时可恢复、GLM tpm 50万→200万、连坐只挡预算用完和真跑满24h
|
||||
- [成功率低的真实原因](success-rate-breakdown-20261008.md) — 10-08:真做不到的只有 2 条;GLM/Qwen 同题打平是考卷区分不出(简单题满分、难题被砍或判分卡住)
|
||||
- [轨迹时间花在哪](rollout-time-breakdown-20261008.md) — 10-08:编译占 33%、编译机 CPU 只用 31%、排队中位 30 秒;04 时长是重复计算;编译上云暂无收益
|
||||
- [编译机要被回收](compile-server-reclaim.md) — 10-08 晚已回收、换到 36.144.20.182;账本只有 9-28 备份;10-11 起 36 能读写网盘上的旧轨迹;120 已关机清理
|
||||
- [第二个 GLM 账号与提并发](glm-second-account-20261008.md) — 10-08:05/06 原来同一账号;06 换新账号 B,05/06 各 16 路,04 不变;接续脚本位置
|
||||
- [Qwen SFT benchmark](qwen-sft-benchmark.md) — 10-08 建 qwen-sft-v1:37 题;04 的 Qwen 轨迹只做训练前基准、不进 SFT;10-12 起 04 的旧结果也算,训练前过 20 败 3、14 题没结果
|
||||
- [仓库主线整理](git-mainline-20261009.md) — 10-09:main=生产分支 c2c52ed,旧 main 打标签;Salty 在 feature/salty-20261009;本地还剩优雅退出/板卡改动
|
||||
- [Salty 个人空间迁移](salty-personal-migration-20261009.md) — 10-10 完成:120 的 bench-cache/data 全在 personal:/bench/,逐文件核对通过;Salty 单文件上限 7.3–12.7G、拒软链接、du 会夸大
|
||||
- [别原地改运行中的脚本](never-edit-running-script.md) — bash 边读边执行;要改就写新文件或新文件+mv 替换
|
||||
- [Mac mini 主控 + Salty 全阶段执行](mini-salty-controller-20261009.md) — 10-09 至 10-12:salty_remote 设计与踩坑(工作目录、HBM 取回、网络抖动、容器内存超限、重启恢复)、轨迹搬到个人空间、导出挪到 Salty、强停写坏执行文件的处理、各处位置
|
||||
- [记忆同步到仓库](memory-synced-to-repo.md) — 改完记忆同步到 docs/memory/ 并推 salty 分支;入口 docs/memory/AGENTS.md
|
||||
- [04 的 Qwen 服务速度](qwen-nvfp4-service-speed-20261012.md) — 10-12:单请求中位 6 秒、一次只处理 1 个请求;04 开 5 路只忙 15%,不是瓶颈
|
||||
- [迁到 Salty 后通过数暴跌](salty-throughput-drop-20261012.md) — 10-12 清查:主因是剩下全是重题(同模型在 Salty 不更慢);系统问题:整批重启、容器被杀占编译次数、连不上 GLM 永不自动恢复、metric3d、Wi-Fi
|
||||
- [自愈与报警](selfheal-and-alert-20261012.md) — 10-12:系统失败(被杀、执行器退出、超时、卡死)不占编译次数;连不上自动续,但暂停名额仍会被抢;depthpro/metric3d 撤出;健康检查弹通知
|
||||
- [容器线程被限流](salty-container-threads-20261013.md) — 10-13:容器看到 244 核只给 4–6 核,torch 慢 16 倍、numpy 5 倍;已加线程上限,onnxruntime 未处理
|
||||
- [验证器核查](verifier-audit-20261012.md) — 10-12:Qwen 20 个通过是真的;但验证器不查是否真 int8(全 int16 也算过)、只抽 100 张;打平是因为系统包办了大半
|
||||
- [10-13 四个基建漏洞](infra-leaks-20261013.md) — 超时步骤锁死 CPU 名额(32 剩 6)、超时后 Salty 容器不停、续跑必死、领题后重复查额度把题作废;10-13 20:41 重启后全部修复上线,22:26 导出进程加自动清理孤儿容器
|
||||
- [训练前 Qwen 能力失败题](qwen-ability-failures-20261013.md) — 10-14:34/37 有结果、过 28、能力失败 6 道;短板:配置写错、不先本地诊断、归咎平台
|
||||
- [Mac mini 流量绕美国](mac-mini-clash-global-proxy-20261013.md) — 10-13:Clash 全局+TUN,Qwen/Salty/GLM 全走洛杉矶 JMS 节点,是断线和 Salty 网络错误的共同根因;要用户改 Clash
|
||||
- [HBM 步骤成新瓶颈](hbm-step-bottleneck-20261014.md) — 10-14:HBM≡BC 模拟器抽查在 Salty 上 1–2 小时、一夜撞时限 45 次,从没独立判出过问题;DNS_FAILED 暂停不会自动续
|
||||
- [06-hbm 抽查改为按需](hbm-spotcheck-optional-20261014.md) — 10-14:模拟器抽查不再必跑(已部署);agent 可用配置开启、只作诊断;配置校验/说明与 DNS 续跑等下次重启
|
||||
- [网络容错与板卡时限](net-resilience-board-deadline-20261014.md) — 10-14:原地重发/免费续跑/结果不明可续跑;板卡时限对齐;19:00 连同开发环境/自带图/速度要求一起重启上线(-v16 配置、脚本在 restart-20261014)
|
||||
- [智能体自主性核查](agent-autonomy-audit-20261014.md) — 10-14:不让改图、兜底配置多、VM 无工具链;已实现 bench dev(Salty 沙箱)+ 自带图/校准(a92510d,等重启);速度 1.5 倍门槛离线重判 06 翻 37/178;上线 2 小时内还没人自带图,dev 曾能看到评测集(已修)
|
||||
- [10-15 早上复查](overnight-check-20261015.md) — dev 槽 8→37(配置有缓存,只对新命令生效);S600 板盘满因失败会话目录不删(a263374 已修);暂停名额被新题抢;全 fp16 落 CPU 仍判过;4 条自带图/校准;评测文件被 task_contract 冻结,在跑的 run 不能热改(11:08 改坏三条)
|
||||
- [v16 轨迹质量对比](v16-quality-20261015.md) — 10-15:dev 52/53 在用、兜底和慢速通过变少但样本少;240 轮上限截断最自主的轨迹;Qwen 3过3败是不再用全 int16 兜底,基准要在 v16 下重测
|
||||
- [10-15 下午重启](restart-20261015.md) — 重启前清查:回执 15 秒超时、暂停吃掉 24 小时、400 永久暂停、账本忙让出名额、评测一次 12 GB;v17(400 轮)、判分 v0.8;之后热部署:评测分批靠批准的等价改写(不建新 run)、加载卡住检测、容器 onnxruntime 限线程、Salty 失败作业自动清
|
||||
- [最新三条 SFT 细读](sft-sample-review-20261015.md) — 10-15:真通过但决策含量低;两条是 Qwen 自己的不该进训练;v16 仅 8/315、含评测题、上下文过长
|
||||
- [v17 自主性实测](v17-autonomy-check-20261015.md) — 10-15 19:40:32 条 GLM 跑 5 小时 0 条结束;自己导图 0、自备校准 0、图手术 1;72% 时间在开发环境,9 条各花 4 小时编默认配置
|
||||
- [v18–v22 整条链路给 agent](v18-own-inputs-20261015.md) — 自己导图/校准(v21 原图)、速度参考(编不出写 unavailable)、板上自测(≤512 MiB);10-16 16:21 重启:上下文满记预算用完、断线只停 2 分钟、Qwen 省去旧工具输出、GLM token 2M、stage wait 不轮询
|
||||
- [10-16 17:00 复查](check-20261016-1700.md) — v18–21 自己导图 41/46、真改图约 3 条、0 条通过;05 的 GLM S600 账号 16:00 起全 429,17:22 换 key 和新组名(s600-r05-v22c)恢复
|
||||
- [PASS/SFT 自主性统计](sft-autonomy-tally-20261016.md) — 10-16:GLM 310 条 PASS 里 71% 只写配置,成品真用自己改的图只有 4 条(3 条进 SFT);v18+ 还没通过;文本搜 onnx_model 会误报
|
||||
- [第二版 0 通过的原因](v18-zero-pass-why-20261016.md) — 10-16:重启取消 42/90、82/90 是从没过的难题、到评测的都差精度;10 条想找评测集做本地验证被挡、28 条照搬平台导出脚本
|
||||
- [稳定性体检](stability-sweep-20261016.md) — 10-16:最大问题是生产批次当试验场(3 天 8 版、重启即取消);19:00 复核更正:本机有健康检查但漏报限流、4000 小时实为约 1660、盘不是急事、Qwen 进 SFT/版本混杂不用改代码
|
||||
- [10-16 断电停机与重启](poweroff-20261016.md) — 停/起/清理脚本在 configs/poweroff-20261016;开机 VM 自起;取消会留 dev lease 占槽;speed_ref 残留要删
|
||||
- [回复用中文](reply-in-chinese.md) — 汇报、提问一律中文;命令和路径保持原样
|
||||
- [10-17 02:15 底层复查](check-20261017-0215.md) — v18+ 134 条:真改图约 10、板上自测 6、通过 3(1 条板上 493 ms 没被速度挡住);平台复测不判速度;重启后很稳,只剩 Qwen 续跑必死(context-mask 状态在内存)
|
||||
- [能否当强化学习环境](rl-readiness-20261017.md) — 10-17:能用,瓶颈是慢不是稳;收尾 5 件后冻结,清单外默认不再改
|
||||
- [少管过程的删减清单](simplify-agent-rules-20261017.md) — 10-17:交还 Pi 压缩要改 5 处(不是纯减法,导出也要认压缩);提示词逐条分留事实/删规矩/交给系统;并入冻结版本
|
||||
- [10-17 停 06](stop-06-20261017.md) — 账号 B 没额度,11:30 只 SIGTERM 不取消;原命令/环境在 configs/stop-06-20261017;换 key 要新组名并取消暂停的
|
||||
@ -1,42 +0,0 @@
|
||||
- [优先做减法而非加法](prefer-subtraction-over-addition.md) — 修 bug 先想能删什么,别动辄加钩子和配置项
|
||||
- [汇报说人话](plain-language-no-jargon.md) — 不要用金丝雀这类行话,先给结论和数字
|
||||
- [bench 调度程序重启的坑](bench-restart-hazards.md) — 改文件不算换代、kill -9 条件、drain 单向、密钥在旧进程环境、启动要 10–20 分钟、导出会卡、全局并发上限要三个一起换、新账号用新组名
|
||||
- [第一版交付包](delivery-package-v1.md) — 位置、冻结点 62 条、出第二版要避开的坑
|
||||
- [采集稳定性补丁](quota-unknown-autorecover-fix.md) — 2f54d40→ea0335b:请求不明、板卡锁死、板端隔离、ssh 断连、回执误判、编译上限 5 次、请求结束后补放行试探、bash 超时可恢复、GLM tpm 50万→200万、连坐只挡预算用完和真跑满24h
|
||||
- [成功率低的真实原因](success-rate-breakdown-20261008.md) — 10-08:真做不到的只有 2 条;GLM/Qwen 同题打平是考卷区分不出(简单题满分、难题被砍或判分卡住)
|
||||
- [轨迹时间花在哪](rollout-time-breakdown-20261008.md) — 10-08:编译占 33%、编译机 CPU 只用 31%、排队中位 30 秒;04 时长是重复计算;编译上云暂无收益
|
||||
- [编译机要被回收](compile-server-reclaim.md) — 10-08 晚已回收、换到 36.144.20.182;账本只有 9-28 备份;10-11 起 36 能读写网盘上的旧轨迹;120 已关机清理
|
||||
- [第二个 GLM 账号与提并发](glm-second-account-20261008.md) — 10-08:05/06 原来同一账号;06 换新账号 B,05/06 各 16 路,04 不变;接续脚本位置
|
||||
- [Qwen SFT benchmark](qwen-sft-benchmark.md) — 10-08 建 qwen-sft-v1:37 题;04 的 Qwen 轨迹只做训练前基准、不进 SFT;10-12 起 04 的旧结果也算,训练前过 20 败 3、14 题没结果
|
||||
- [仓库主线整理](git-mainline-20261009.md) — 10-09:main=生产分支 c2c52ed,旧 main 打标签;Salty 在 feature/salty-20261009;本地还剩优雅退出/板卡改动
|
||||
- [Salty 个人空间迁移](salty-personal-migration-20261009.md) — 10-10 完成:120 的 bench-cache/data 全在 personal:/bench/,逐文件核对通过;Salty 单文件上限 7.3–12.7G、拒软链接、du 会夸大
|
||||
- [别原地改运行中的脚本](never-edit-running-script.md) — bash 边读边执行;要改就写新文件或新文件+mv 替换
|
||||
- [Mac mini 主控 + Salty 全阶段执行](mini-salty-controller-20261009.md) — 10-09 至 10-12:salty_remote 设计与踩坑(工作目录、HBM 取回、网络抖动、容器内存超限、重启恢复)、轨迹搬到个人空间、导出挪到 Salty、强停写坏执行文件的处理、各处位置
|
||||
- [记忆同步到仓库](memory-synced-to-repo.md) — 改完记忆同步到 docs/memory/ 并推 salty 分支;入口 docs/memory/AGENTS.md
|
||||
- [04 的 Qwen 服务速度](qwen-nvfp4-service-speed-20261012.md) — 10-12:单请求中位 6 秒、一次只处理 1 个请求;04 开 5 路只忙 15%,不是瓶颈
|
||||
- [迁到 Salty 后通过数暴跌](salty-throughput-drop-20261012.md) — 10-12 清查:主因是剩下全是重题(同模型在 Salty 不更慢);系统问题:整批重启、容器被杀占编译次数、连不上 GLM 永不自动恢复、metric3d、Wi-Fi
|
||||
- [自愈与报警](selfheal-and-alert-20261012.md) — 10-12:系统失败(被杀、执行器退出、超时、卡死)不占编译次数;连不上自动续,但暂停名额仍会被抢;depthpro/metric3d 撤出;健康检查弹通知
|
||||
- [容器线程被限流](salty-container-threads-20261013.md) — 10-13:容器看到 244 核只给 4–6 核,torch 慢 16 倍、numpy 5 倍;已加线程上限,onnxruntime 未处理
|
||||
- [验证器核查](verifier-audit-20261012.md) — 10-12:Qwen 20 个通过是真的;但验证器不查是否真 int8(全 int16 也算过)、只抽 100 张;打平是因为系统包办了大半
|
||||
- [10-13 四个基建漏洞](infra-leaks-20261013.md) — 超时步骤锁死 CPU 名额(32 剩 6)、超时后 Salty 容器不停、续跑必死、领题后重复查额度把题作废;10-13 20:41 重启后全部修复上线,22:26 导出进程加自动清理孤儿容器
|
||||
- [训练前 Qwen 能力失败题](qwen-ability-failures-20261013.md) — 10-14:34/37 有结果、过 28、能力失败 6 道;短板:配置写错、不先本地诊断、归咎平台
|
||||
- [Mac mini 流量绕美国](mac-mini-clash-global-proxy-20261013.md) — 10-13:Clash 全局+TUN,Qwen/Salty/GLM 全走洛杉矶 JMS 节点,是断线和 Salty 网络错误的共同根因;要用户改 Clash
|
||||
- [HBM 步骤成新瓶颈](hbm-step-bottleneck-20261014.md) — 10-14:HBM≡BC 模拟器抽查在 Salty 上 1–2 小时、一夜撞时限 45 次,从没独立判出过问题;DNS_FAILED 暂停不会自动续
|
||||
- [06-hbm 抽查改为按需](hbm-spotcheck-optional-20261014.md) — 10-14:模拟器抽查不再必跑(已部署);agent 可用配置开启、只作诊断;配置校验/说明与 DNS 续跑等下次重启
|
||||
- [网络容错与板卡时限](net-resilience-board-deadline-20261014.md) — 10-14:原地重发/免费续跑/结果不明可续跑;板卡时限对齐;19:00 连同开发环境/自带图/速度要求一起重启上线(-v16 配置、脚本在 restart-20261014)
|
||||
- [智能体自主性核查](agent-autonomy-audit-20261014.md) — 10-14:不让改图、兜底配置多、VM 无工具链;已实现 bench dev(Salty 沙箱)+ 自带图/校准(a92510d,等重启);速度 1.5 倍门槛离线重判 06 翻 37/178;上线 2 小时内还没人自带图,dev 曾能看到评测集(已修)
|
||||
- [10-15 早上复查](overnight-check-20261015.md) — dev 槽 8→37(配置有缓存,只对新命令生效);S600 板盘满因失败会话目录不删(a263374 已修);暂停名额被新题抢;全 fp16 落 CPU 仍判过;4 条自带图/校准;评测文件被 task_contract 冻结,在跑的 run 不能热改(11:08 改坏三条)
|
||||
- [v16 轨迹质量对比](v16-quality-20261015.md) — 10-15:dev 52/53 在用、兜底和慢速通过变少但样本少;240 轮上限截断最自主的轨迹;Qwen 3过3败是不再用全 int16 兜底,基准要在 v16 下重测
|
||||
- [10-15 下午重启](restart-20261015.md) — 重启前清查:回执 15 秒超时、暂停吃掉 24 小时、400 永久暂停、账本忙让出名额、评测一次 12 GB;v17(400 轮)、判分 v0.8;之后热部署:评测分批靠批准的等价改写(不建新 run)、加载卡住检测、容器 onnxruntime 限线程、Salty 失败作业自动清
|
||||
- [最新三条 SFT 细读](sft-sample-review-20261015.md) — 10-15:真通过但决策含量低;两条是 Qwen 自己的不该进训练;v16 仅 8/315、含评测题、上下文过长
|
||||
- [v17 自主性实测](v17-autonomy-check-20261015.md) — 10-15 19:40:32 条 GLM 跑 5 小时 0 条结束;自己导图 0、自备校准 0、图手术 1;72% 时间在开发环境,9 条各花 4 小时编默认配置
|
||||
- [v18–v22 整条链路给 agent](v18-own-inputs-20261015.md) — 自己导图/校准(v21 原图)、速度参考(编不出写 unavailable)、板上自测(≤512 MiB);10-16 16:21 重启:上下文满记预算用完、断线只停 2 分钟、Qwen 省去旧工具输出、GLM token 2M、stage wait 不轮询
|
||||
- [10-16 17:00 复查](check-20261016-1700.md) — v18–21 自己导图 41/46、真改图约 3 条、0 条通过;05 的 GLM S600 账号 16:00 起全 429,17:22 换 key 和新组名(s600-r05-v22c)恢复
|
||||
- [PASS/SFT 自主性统计](sft-autonomy-tally-20261016.md) — 10-16:GLM 310 条 PASS 里 71% 只写配置,成品真用自己改的图只有 4 条(3 条进 SFT);v18+ 还没通过;文本搜 onnx_model 会误报
|
||||
- [第二版 0 通过的原因](v18-zero-pass-why-20261016.md) — 10-16:重启取消 42/90、82/90 是从没过的难题、到评测的都差精度;10 条想找评测集做本地验证被挡、28 条照搬平台导出脚本
|
||||
- [稳定性体检](stability-sweep-20261016.md) — 10-16:最大问题是生产批次当试验场(3 天 8 版、重启即取消);19:00 复核更正:本机有健康检查但漏报限流、4000 小时实为约 1660、盘不是急事、Qwen 进 SFT/版本混杂不用改代码
|
||||
- [10-16 断电停机与重启](poweroff-20261016.md) — 停/起/清理脚本在 configs/poweroff-20261016;开机 VM 自起;取消会留 dev lease 占槽;speed_ref 残留要删
|
||||
- [回复用中文](reply-in-chinese.md) — 汇报、提问一律中文;命令和路径保持原样
|
||||
- [10-17 02:15 底层复查](check-20261017-0215.md) — v18+ 134 条:真改图约 10、板上自测 6、通过 3(1 条板上 493 ms 没被速度挡住);平台复测不判速度;重启后很稳,只剩 Qwen 续跑必死(context-mask 状态在内存)
|
||||
- [能否当强化学习环境](rl-readiness-20261017.md) — 10-17:能用,瓶颈是慢不是稳;收尾 5 件后冻结,清单外默认不再改
|
||||
- [10-17 停 06](stop-06-20261017.md) — 账号 B 没额度,11:30 只 SIGTERM 不取消;原命令/环境在 configs/stop-06-20261017;换 key 要新组名并取消暂停的
|
||||
@ -1,129 +0,0 @@
|
||||
---
|
||||
name: agent-autonomy-audit-20261014
|
||||
description: "10-14 核查智能体自主性:04/05/06 共 627 题都不让改图(ONNX 钉死),难题被堵死;通过的配置多是\"全换 int16 / 全换 fp16\"的兜底,按模型诊断的只有约一成;照抄模板不是主因"
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 2b85e929-b893-42c3-a7a0-3a72bda14705
|
||||
modified: 2026-10-14T08:58:17.164Z
|
||||
---
|
||||
|
||||
用户问:系统包办 + 模板照抄,是不是让智能体体现不出能力(实际工作要自己导出 ONNX、拆算子)。2026-10-14 查代码和会话的结论:
|
||||
|
||||
**1. 真正的硬限制是不能改图**
|
||||
- compile 用的 ONNX 由平台钉死:ptq.compile_config 覆盖 model_parameters.onnx_model。agent 只能改 compile.yaml 的三段(stageworker.snapshot_config 白名单),custom_op 之类会被拒。
|
||||
- 代码里其实有"提交自己的图/导出代码"的入口:`bench stage run export --recipe`,见 docs/extensions.md、harness/recipes.py。但它要求 job 里有 model_recipe.reference,而且只支持 PTQ 语义分割。04/05/06 的 627 道题一道都没开。
|
||||
- 盘上会话里有 11 条失败轨迹试过 `--recipe`,都被拒("job must freeze a model_recipe reference policy")。
|
||||
- GLM 在 05 有 24 题、06 有 26 题从没通过,约一半连一次编译都没成功,正好是要改图的题:
|
||||
- mask2former:Einsum 算子不支持;
|
||||
- flownet2:resample 输出的形状推不出来;
|
||||
- unidepth:解码器是 float64;
|
||||
- depthpro:导出失败;
|
||||
- 还有 aliked、keynet、detr。
|
||||
- GLM 的报告诊断得很准,最后都写"没有任何改图入口"。flownet2 那条怀疑是 harness 自带的 vendor/flownet2_cpu_ops.py 里的 resample 有问题,**还没核实**。
|
||||
|
||||
**2. 照抄模板不是主因,主因是兜底配置**
|
||||
- 374 次写配置,没有一次照抄模板里的示例节点名。
|
||||
- 通过轨迹最终用的配置(还在盘上的样本):
|
||||
- 06(S100P int8):43 条里 24 条全模型 int16,都是先 int8 失败、平均第 3 次编译才换的;只有 4 条按节点调。
|
||||
- 05(S600 fp16int8):44 条里约 28 条是"除卷积外全 fp16",有的连卷积也 int16、完全没有 int8;按节点调的约 6 条。
|
||||
- 04(Qwen):32 条里 14 条全 int16。
|
||||
- 验证器不查 int8 比例(见 [[verifier-audit-20261012]]),所以这些都判通过。SFT 会学到"失败就全换高精度"。
|
||||
- 一次编译就通过的比例:GLM-06 111/186,GLM-05 76/109,Qwen-04 31/66。通过的题里分类题最多(06 有 94/186)。
|
||||
|
||||
**3. 最像智能体的行为都在失败轨迹里**:本地复现、克隆官方源码比对、定位算子。这些轨迹被平台堵死后判 FAIL,进不了训练数据。
|
||||
|
||||
**4. 给用户的改法(10-14,等拍板)**
|
||||
|
||||
开放改图,走窄路,不推广现有的配方框架:
|
||||
- 配方框架要每题登记参考源码哈希,只支持分割题,评测还要跑 agent 写的前后处理,太重。
|
||||
- 窄路是 compile.yaml 允许写 `model_parameters.onnx_model: <workspace 内 .onnx>`,要求输入输出的个数、顺序、形状与平台 ONNX 一致。这样基线、上板、评测都不用改。已核实:quant 评测不读平台 ONNX,基线来自 torch 参考。
|
||||
- 要改的地方:
|
||||
- stageworker.snapshot_config:允许这个字段,用 recipes._stream 复制进受保护的队列目录;
|
||||
- ptq.compile_config:用这份图;
|
||||
- salty_remote._bundle:把图带进容器;
|
||||
- s04:在容器里比对输入输出,不一致就直接失败,不占编译次数,同时记录图的来源和 sha;
|
||||
- SYSTEM.md。
|
||||
- 前两处和 SYSTEM.md 要重启才生效,s04 部署即生效。
|
||||
- 开放后能做:本地导出、改写算子、交预量化模型(calibration_type: skip)、等价改权重。
|
||||
- 风险:测试集 agent 可读,改过的图可能把 100 张的答案写死。做 SFT 先接受,记录图的 sha 供抽查;转 RL 前要加一份 agent 看不到的评测子集。
|
||||
|
||||
堵兜底:
|
||||
- 编译日志节点表里的输出类型不能用。卷积是 int8,但输出 fp16 时也显示 f16。
|
||||
- 可靠的度量是编译器估计的延迟,和同一题默认 int8 编译比。默认编译每题跑一次,缓存后共享。
|
||||
- 先离线重判给用户看会翻多少条,阈值由用户定。
|
||||
|
||||
开发环境(10-14 下午复查,**排第一**):
|
||||
- 迁到 Mac mini 后,agent 的 bash 跑在 VM 里,但 VM 里没有工具链:
|
||||
- 没有 /home/ming01.wang/toolchains;
|
||||
- 系统 python 没有 hmct、onnxruntime、torch;
|
||||
- 连提示词让用的 rg 都没装。
|
||||
- 10-13 以来 05/06 的 171 条会话里,只有 5 条跑成过一次本地试验。GLM 在难题上靠的"先本地诊断再编译",现在做不了。
|
||||
- 所以问题不是算力不够,是工具根本不在。
|
||||
- 改法:把受管终端(terminal.py 的 development 目标,现在关着,而且只在本机跑)改成走 Salty,复用 salty_remote 的打包和取回。workspace 带进去,跑完带回来。
|
||||
- 开发环境里可以随便跑 hb_compile、hmct、torch.onnx.export;正式编译仍走 stage、计次数。
|
||||
|
||||
顺带发现:e2e-occ 提示词说可以指向自备校准 npy,但白名单不收 cal_data_dir。(已修,见下)
|
||||
|
||||
**5. 已实现(10-14 a92510d,等重启上线)**
|
||||
- 一、`bench dev`:
|
||||
- agent 用 `bench dev "<命令>"` 提交,经 stageclient 写成 terminal 票(target development)。
|
||||
- terminal.run 发现 salty_remote.stages 里有 terminal,就走 harness/salty_dev.py;worker 用 dev 模式跑。
|
||||
- 沙箱:setpriv 降到 uid nobody,再 `unshare -Urm`;tmpfs 盖住 /data/personal,只把工具链、outputs、cache、data(靠属主只读)和本轨迹 workspace 绑回来,最后去掉全部能力再执行命令。
|
||||
- Salty 实测:
|
||||
- 板卡密钥、/root/.ssh、其他轨迹都看不到;
|
||||
- 不去能力时 umount 能看到整个个人空间,去掉后挡住了;
|
||||
- 能 import hmct、onnxruntime、torch,小卷积网络的 hb_compile 能出 hbm(玩具全连接模型在沙箱外一样崩,是 hbdk 自己的问题)。
|
||||
- 每条命令开销约 75–80 秒。
|
||||
- 工作区同步:本地改动按 stages/_dev-sync.json 上传;单个 ≤32MiB 的文件拷回本地;更大的只留在 personal:/bench-root/runs/.../workspace。
|
||||
- 局限:容器里起的后台进程随命令结束;开发环境的大文件不会自动清理。
|
||||
- 二、compile.yaml 可以写 `model_parameters.onnx_model`、`calibration_parameters.cal_data_dir`(workspace 内的相对路径):
|
||||
- stageworker 先归一化;s04 在容器里比对输入输出,不一致就以 own_input_invalid 失败,不占编译次数;
|
||||
- 记录图的 sha256 和 calib_source=workspace;hb_verifier 仍用平台的校准样本。
|
||||
- 历史上 1377 份配置里只有 4 份写了 onnx_model,全是 agent 在试自带的图。
|
||||
- SYSTEM.md 加了开发环境一节,rg 改成 grep;agent/VERSION 15→16(新旧轨迹按版本区分)。
|
||||
- 三、速度门槛离线重判:
|
||||
- 方法:最终模型的编译器估计延迟 ÷ 同题平台默认配置编出的延迟,在 Salty 上读 personal:/bench-root/runs 全量。
|
||||
- 默认配置通过的比值 0.91–1.12,所以噪声约 ±10%。全 int16 是 1.5–6.2 倍。
|
||||
- 阈值 1.5 倍时会翻掉的通过轨迹:
|
||||
- 06:37/178(36 条全 int16,1 条 yolo11n);
|
||||
- 04:24/64(18 条全 int16,6 条按节点调的,板上慢 7–500 倍,多是落 CPU);
|
||||
- 05:只有 41/109 有参考(S600 上 agent 很少编平台默认配置),翻 10 条。
|
||||
- 板上延迟比估计更能抓到落 CPU,正式门槛要用板上测的默认配置做参考。05 要每题补一次默认编译。
|
||||
- 结果文件在 scratchpad 的 speed-out.json,脚本是 speed.py。
|
||||
|
||||
- 四、上线后 1 小时 50 分钟核查(10-14 20:50,v16 共 34 条轨迹,还没有一条跑完):
|
||||
- 自带图提交 0 条、自带校准提交 0 条、改图命令 0 条。29/34 用了 bench dev,共 140 条命令。
|
||||
- 命令主要是:看平台 ONNX 结构、读 harness 源码(编译和评测代码)、按速度要求在 dev 里编一次默认配置(约 20 次)。
|
||||
- 首次提交的配置仍是默认加 per_channel 加 set_model_output_int16,跟以前一样。
|
||||
- 结论:工具到位了,行为还没变。改图只会在配置调不过时才出现,要等这批难题跑完再看。
|
||||
- 发现并修复(f16faee,20:47 热部署,不用重启):
|
||||
- bench dev 能看到评测集 t1(带标签)和 pool,是 a92510d 引入的。efficientvit.b3 读过 t1。现在只挂 bench/data/calib。
|
||||
- salty_dev 重复上传一次 bundle,已去掉。
|
||||
- SYSTEM.md 第 57 行还写着「评测数据只读可见」,下次换版本时删掉。
|
||||
- 五、23:50 复查:
|
||||
- f16faee 两处都生效:dev 里读 t1/pool 报不存在;成功命令中位 71 秒(之前 78)。
|
||||
- 开始真改图了:maxvit.t 把 Einsum 改写成 MatMul+Transpose(跑通),reppoints 物化 Reshape 目标(被排队饿死)。另有 4 条在逐节点诊断。还没有提交自带图。
|
||||
- 新瓶颈:dev 池只有 8 槽(三个 run 共用锁),被 agent 自设 3–12 小时超时的长编译占住(最长 272 分钟)。短命令排到自己超时就报 STAGE_EXECUTOR_EXIT,19:56 起共 47 条,echo 也排了 37 分钟。Salty 当时不排队,瓶颈在我们自己的槽数。
|
||||
- pools.dev.workers 每次现读配置文件,三个 -v16 配置一起改就能生效、不用重启;改不改等用户拍板。
|
||||
- 六、10-15 10:43 自带图结果:
|
||||
- 真改图的 4 条,都还没有最终结论:
|
||||
- deeplabv3plus.r101(05):拆卷积加自做校准,编译、HBM、上板都过了,评测在跑。这题以前 12 小时没过。板上 506 ms,静态估计 38 ms,BPU 利用率 3.5%,大半跑在 CPU。agent 看到 job.json 里 G4 不是必选,就不管速度要求了。
|
||||
- maxvit.t(06):Einsum→MatMul,编译和 HBM 过了,08:13 起卡在一条 dev 长编译上。
|
||||
- deeplabv3plus.r50、fcn.r101(05):同样是拆大卷积(2560→512、2304→256),10:00 起在正式编译。fcn 还在一个校准样本上验证了浮点等价。
|
||||
- 04 superpoint 是「自带图自带校准」的 PASS,但两样都是 cp 的平台文件,不算。以后统计自带图要比哈希,排除这种复制。
|
||||
- vit.h.14(06)自做校准编过两次,240 轮用完没上板(BUDGET_EXHAUSTED)。
|
||||
|
||||
**Why:** 用户要把这套当训练数据来源和以后 RSI 的基础设施。考卷决定了能学到什么。
|
||||
**How to apply:**
|
||||
- 改进顺序(10-14 给用户的方案):
|
||||
1. 开发环境上 Salty;
|
||||
2. 窄路开放改图,加 cal_data_dir 自备校准;
|
||||
3. 验收加速度要求;
|
||||
4. 删掉提示词里喂饭的话;
|
||||
5. 以后再做:nv12 输入、板上前后处理、板上终端、隐藏的验收子集、QAT。
|
||||
- 改判分标准、改题目都要用户拍板。
|
||||
- 样本只覆盖还在 VM 盘上的会话(04 有 38 条、05 和 06 各 60 条,偏向较新、较难的题);更早的已归档到个人空间。
|
||||
- 统计脚本在 scratchpad 的 sess.py、cfg.py、cfg2.py。
|
||||
|
||||
相关:[[verifier-audit-20261012]] [[qwen-ability-failures-20261013]] [[success-rate-breakdown-20261008]]
|
||||
@ -1,57 +0,0 @@
|
||||
---
|
||||
name: bench-restart-hazards
|
||||
description: model-compile-hbdk 的 bench 调度程序重启时三个没写进文档的陷阱
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 2636af57-8cef-4006-92c2-7d78ac925e56
|
||||
modified: 2026-10-03T16:33:23.678Z
|
||||
---
|
||||
|
||||
2026-09-18 实测确认,`bench collect` 换代时有三个坑,CONTRACT.md 只写了其中第一个:
|
||||
|
||||
1. **改文件不算换代**:stage 在长驻 `bench collect` 进程内执行(`hb_compile` 的父进程就是它),
|
||||
必须重启进程才生效。CONTRACT.md:184 有记录。
|
||||
2. **绝不能 kill -9 调度程序**:`pools.agent.workers=12`,而 `pipeline.py:509` / `db.py:238` 的 active
|
||||
计数是 `SELECT COUNT(*) FROM rollouts WHERE state IN ('RUNNING','VERIFYING')`,**跨 run 无过滤**。
|
||||
全仓没有 rollouts RUNNING→QUEUED 的路径(`reset-stale` 只动 jobs 和 stages),硬杀会让那些行
|
||||
永久占死全局槽位,无 CLI 可救。只能 SIGTERM。
|
||||
3. **`bench drain` 是单向闸门**:`collect_drain_file` 只有 `cli.py:182` 的 `.touch()`,全仓无 unlink,
|
||||
也没有 undrain 子命令。用了 drain 就必须在重启前手工 `rm state/drain-collect-<run>`,否则新进程启动即自停。
|
||||
优先用 SIGTERM,它不产生哨兵文件。
|
||||
|
||||
4. **密钥可能只在旧进程的环境变量里**(2026-10-04 踩坑):`paths.secret()` 先读环境变量,再读 `bench/.env.local`。
|
||||
04 批次(Qwen)用的 `QWEN_EVAL_API_KEY` 不在 `.env.local` 里,只存在于旧进程的环境中;我结束旧进程后,
|
||||
只照搬 `BENCH_*` 重启,结果报 `CREDENTIAL_UNAVAILABLE: teacher`。
|
||||
重启前先把配置里所有 `api_key_env` 和 `.env.local` 对一遍,只比对变量名;缺的先让用户补进 `.env.local`,再动旧进程。
|
||||
|
||||
另:`--first-pass` 的过滤器按 **job 聚合**(`pipeline.py:503`)且会并入源 run 历史,对 continuation run
|
||||
(如 r20260916-01-n100)会把 206 条 QUEUED 全部挡掉、可放行 0 条——这正是它空转 14 小时的原因。
|
||||
换代时务必自己算一遍可放行数,不要相信转述。
|
||||
|
||||
5. **重启要等很久,而且旧进程会卡在最后的导出上**(2026-10-04 实测):
|
||||
- 新采集器在网盘上启动时要逐条核对 836 条任务,要过 10–20 分钟才会认领第一条,期间进程处于 D 状态、CPU 很低,这是正常的,不要误判成卡死。
|
||||
- 旧采集器排空后会一直卡在最后的数据导出(`datasets/<run>-training/.refresh-*`)。确认这个批次的运行中、校验中、清理中都是 0 条之后,再等 5 分钟仍未退出,可以 kill -9;之后删掉留下的 `.refresh-*` 目录。
|
||||
- 正在跑的任务没有"暂停后接着跑"的办法:配额管理器在采集器进程里面,进程一停,正在跑的任务就断了。想马上重启只能先用 `bench execution cancel` 取消正在跑的任务;不想丢任务就只能等它们跑完。
|
||||
- 重启时照搬旧进程的命令行、工作目录、环境变量(尤其是 `BENCH_CONFIG`),日志用 `>>` 追加到原来的日志文件里,就不会产生新的批次或文件。
|
||||
- 看日志时不要用时间字符串大小比较来过滤(日志会跨天),按每次启动时打印的 "collect <run>: parallel" 那一行的行号来截取。
|
||||
- `bench execution cancel` 偶尔会卡在网盘读写上(进程等待 FUSE,wchan=request_wait_answer),重启脚本就停在 `wait` 不往下走(2026-10-08 实测卡了 20 分钟)。先查这条轨迹在库里是否已经是终态;是的话直接 kill 这个取消命令,脚本会继续往下走。
|
||||
|
||||
6. **想提并发,三个 collector 得一起换代**(2026-10-08 实测,代码读过):
|
||||
- `pools.agent.workers` 是**所有批次加总**的上限(`pipeline.py:595`),而且只在 collector 启动时读一次。只给一个批次调大,别的批次进程里还是旧上限,总数一超过旧上限它们就补不上位置,会被挤掉。
|
||||
- 同一批次只能有一个 collector(`locks/collect-<run>.lock` 排它锁),不能靠"同批次再开一个"加并发。
|
||||
- broker 启动时就把 key 读进内存,之后改 `endpoints` 不会换 key;改了 `collection_v1` 段,在跑的进程会报 `RUNTIME_CONFIG_CHANGED`。所以**永远另写一份新配置,不改正在用的那份**。
|
||||
- 账本(`~/.local/state/bench-shared/quota.sqlite`)存着每个账号组的规则,配置里同名组的规则只要差一个字段,启动就报 `QUOTA_POLICY_CONFLICT`。新账号要用**新组名**;老组规则不要动。
|
||||
- **`paths.config()` 带 `@lru_cache(maxsize=1)`(paths.py:55),每个进程只读一次配置**;scheduler.py:156 注释说"每轮重读"已过时。改 `pools.cpu.workers` 只对改之后新起的 stage 进程生效(每次编译/HBM 是新进程,所以很快普遍生效),已经在排队的仍按旧槽位数等。10-08 我起初漏看这个装饰器,误判过一次。
|
||||
|
||||
7. **不等排空、直接重启的流程**(2026-10-08 跑通,21 条全部成功):先 SIGTERM 旧 collector,再对每条在跑轨迹执行
|
||||
`bench execution cancel --run R --job J --kk K --reason ...`(环境照抄旧进程,`BENCH_CONFIG` 用旧配置)。每条约 30 秒;
|
||||
取消后库里会先短暂显示 `VERIFYING`,片刻后变为 `CANCELLED`,这是旧进程在收尾,不是失败。旧进程随后照例卡在导出,按第 5 条处理。
|
||||
脚本在远端 `/home/ming01.wang/raise-concurrency-20261008/cancel-inflight.sh`、`relaunch.sh`。
|
||||
- 没有"暂停后换进程续跑":`execution resume` 只接受系统因额度等待主动暂停(`PAUSED_INFRA` 且有断点)的轨迹;进程被杀后 reconcile 会把在跑的记成 `LOST`。
|
||||
- 取舍依据:按已跑时长算条件成功率,跑过 5 小时的只有 10–15% 能 PASS;10-08 那次 21 条预计只值约 4 条 PASS。
|
||||
- 本机 shell 是 zsh,`set -- $var` 不拆词;本机写轮询判断时用 `bash -c` 或把判断放到远端 bash 里。
|
||||
|
||||
8. **取消在跑轨迹会让账号组停约 21 分钟**(2026-10-12 实测):`bench execution cancel` 打断发到一半的模型请求,这些请求记成"结果不明",整个账号组进入 RECONCILING,新调度程序起来后也领不了题,要等 request_total_s(1200)+window(60) 后自动放弃才恢复。10-12 17:17 取消 46 条后,05 到 17:39 才开始领题(06 刚好没有发到一半的请求,立刻领满)。立即重启时预留这段时间,别误判成卡死。
|
||||
|
||||
9. **重启虚拟机(断电)时,暂停的轨迹也续不了**(2026-10-16 实测):process_status 对旧开机的进程返回 UNKNOWN,开机后 reconcile 把 PAUSED_INFRA 轨迹标成 OWNERSHIP_UNKNOWN 隔离,resume 也拒绝开机变过的。停 VM 前要把可续跑的暂停一起取消补号;事后取消要等 30 分钟清理时限。详见 [[poweroff-20261016]]。
|
||||
@ -1,44 +0,0 @@
|
||||
---
|
||||
name: check-20261016-1700
|
||||
description: 10-16 17:00 复查:v18–v21 共 46 条,自己导图 41、自做校准 43、真改图约 3 条、板上自测 2 条,但 0 条通过;05 的 GLM S600 账号 16:00 起全是 429;Qwen 重启后等了 30 分钟
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 8477d54c-f423-4a01-ab7c-492675d618dd
|
||||
modified: 2026-10-16T08:55:20.221Z
|
||||
---
|
||||
|
||||
10-16 16:50–17:00 复查(16:21 刚用 v22 重启)。
|
||||
|
||||
**v18–v21 的 46 条轨迹(10-15 21:43 到 10-16 16:20)**
|
||||
- 行为:
|
||||
- 43 条用了 dev;41 条在 dev 里自己导图(v17 是 3/40);43 条自做校准。
|
||||
- 真改图约 3 条:
|
||||
- 06 efficientvit.b1:在线性注意力里缩放 relu(q)/relu(k);
|
||||
- 04 yolov8n.seg(Qwen):把 proto 的 SiLU 换成 tanh 形式;
|
||||
- 06 depth.anything:自己造小模型探测 int8/int16 MatMul 的延迟。
|
||||
- 板上自测:2 条用过,8 次里成功 2 次。
|
||||
- 时间:dev 202 小时,正式阶段 24 小时。
|
||||
- 结果:**0 条通过**。
|
||||
- 25 条被 16:20 的重启取消。
|
||||
- 8 条 INFRA_ERROR:Qwen 上下文超 256k 有 5 条,其余是 broker 503 等。
|
||||
- 3 条预算用完,都是 GLM 的 80 万 token。
|
||||
- 还有 10 条在跑。
|
||||
- v17(40 条):7 条通过(Qwen 4、GLM 3);15 条预算用完,12 条超 24 小时。
|
||||
|
||||
**卡住的情况**
|
||||
- 05 用的 GLM 账号 glm-coding-s600-account:
|
||||
- 16:00:59 起每次请求都返回 HTTP 429 RATE_LIMIT。账本每 15 分钟试一次,05 的 16 条全部 WAITING_QUOTA。
|
||||
- 前 3 天一次 429 都没有。15 点那一小时请求 429 次、1860 万 token,约为平常的 2–4 倍。
|
||||
- 429 的正文没有落盘,看不出是哪种限额、什么时候恢复。
|
||||
- 06 的账号 B 正常;16 点那一小时 749 次请求、3040 万 token,也明显偏高。
|
||||
- 04 Qwen:重启时取消了在途请求,结果不明,组进入 RECONCILING,等满 30 分钟的请求期限(16:59 自动作废)才恢复,16:55 已经有 5 条在跑。UNKNOWN_HOLD_S=120 没有覆盖到"取消造成的结果不明"。
|
||||
- 速度参考只算完 7 道,失败 4 道。
|
||||
- 主机时钟:账本时间比 VM 的 date 快约 6 分钟,看时间对不上时注意。
|
||||
|
||||
**Why:** 用户问"会不会自己改图、进度、有没有卡死、数据是不是更像人用 agent"。
|
||||
**How to apply:**
|
||||
- 看 05 停摆先查 s600 账号的 attempts.http_status。
|
||||
- 想看 429 正文,broker 要先把它记下来;不要拿密钥自己去打接口。
|
||||
|
||||
相关:[[v18-own-inputs-20261015]] [[v17-autonomy-check-20261015]]
|
||||
@ -1,30 +0,0 @@
|
||||
---
|
||||
name: check-20261017-0215
|
||||
description: 10-17 02:15 底层复查:v18+ 134 条的流程漏斗(导图 105、真改图约 10、板上自测 6、正式评测 20、通过 3);平台独立复测怎么做、漏了速度;重启后 5 小时很稳,只剩 Qwen 续跑必死(context-mask 有内存状态,续跑时请求对不上)
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: e00ddcbc-d4da-4f81-a8f7-1fd7eeb8e8df
|
||||
modified: 2026-10-16T18:24:07.553Z
|
||||
---
|
||||
|
||||
10-16 19:20 用户为断电取消 36 条(POWEROFF_20261016),Mac mini 20:26 重启,20:57 三个 collector 用 v22 配置起来(05 是 s600-r05-v22c)。仍走 Wi‑Fi,网关变成 192.168.1.1。
|
||||
|
||||
**v18+ 134 条漏斗(02:15,按 terminal_result.json 命令 + workspace 脚本 + stages 表)**
|
||||
- 用开发环境 114;导图 105(72 条用了平台的 harness.tools.export_onnx,92 条有 torch.onnx.export,有重叠);原图做校准 105;开发环境本地编译 98;本地精度诊断 101。
|
||||
- 真改交付图约 10 条:maxvit.t×2 改写 einsum、depth.anything.v2.large、efficientvit.b1、mask2former 改 einsum、05 fcn.resnet50 和 deeplabv3plus.resnet50 拆卷积、04 yolov8n.seg 改 proto、efficientnet.b0 拆 swish、retinanet.r50 拆 FPN(失败)。swin.b、两条 depth 的 make_node 是探针或库文件,不算。
|
||||
- 板上自测只有 6 条(重启后 25 次命令、19 次成功,用的人在变多)。
|
||||
- 正式编译提交 49、成功 32、HBM 30、上板 22、评测跑完 20、PASS 3。状态:取消 78、在跑 37、系统出错 12、预算用完 4、PASS 3。
|
||||
- 3 条 PASS 都在 05(S600):efficientnet.b7 = 平台导出脚本 + 默认配置 + per_channel,一次编过;fcn.resnet50 = 自写导出、PyTorch 层拆卷积、开发环境小样本试编;deeplabv3plus.resnet50 = ONNX 拆卷积 + 本地编译诊断 + 2 次正式编译,但板上 493 ms、BPU 2%,自己的静态估计 26 ms,平台参考(默认配置静态估计)38.8 ms。三条都没用板上自测。
|
||||
- 训练数据:05 SFT 101(v16–17 只有 5)、06 164(v16–17 只有 2),v18+ 0 条。
|
||||
|
||||
**平台独立复测**:verify.verify_rollout 只读平台阶段(06-hbm/07-board/08-eval)的测量,HBM sha256 在 G1/G3/G4 绑定;agent 的开发环境、板上自测结果标 UNVERIFIED、不计。agent 不调用正式阶段就是 UNVERIFIED,平台不自己补跑。缺口:G4 速度只记录(verdict na);速度参考是 hb_model_info 静态估计,不含落 CPU 的算子,所以抓不到 deeplab 这种情况。要在板上测默认配置的延迟才行。
|
||||
|
||||
**重启后 5.3 小时**:健康检查一直 ok;GLM 请求 4072 次,只有 1 次中断(已续上);阶段 98 次成功、3 次失败(conditional.detr 编译挂死、pspnet.r101 容器被杀、1 次 own_input_invalid 不计次数);开发命令 949 条,系统原因的失败基本消失(执行器退出 0 次,超时 12 次)。
|
||||
**唯一新的系统 bug**:01:24 04 两条 Qwen(efficientnet.b0 在拆 swish、retinanet.r50 在拆 FPN,各跑了约 4.5 小时)续跑 4 秒就以 INFRA_ERROR 结束。链条:00:53 一个 Qwen 请求流式空闲超时 → 整个 Qwen 组暂停 → 01:24 自动放弃那次请求并续跑 → 续跑重发的请求被 broker 拒,返回 409 EXECUTION_HAS_UNRESOLVED_GENERATION(quota._begin 要求重发内容和原来逐字节相同)→ 结束。根因是 agent/context-mask.js 的 `cut` 存在进程内存里(只在超限时移动),续跑是新进程,cut 从 0 重算,省略的位置不同,请求内容就变了。只影响开了省略的 04;10-13 以来本地还能查到的续跑:GLM 9 次全成功,Qwen 5 成 3 败。
|
||||
- 最简修法:让 context-mask 只由消息列表决定(按每个 assistant 消息前的前缀把原来的逐步省略重放一遍,得到同一个 cut),约 10 行;agent 进程启动和续跑时加载,热部署即可,不用重启 collector。还没改,等用户定。
|
||||
|
||||
**Why:** 用户重启后问:数据是否走完整条链路、平台是否自己复测、系统稳没稳、底层错误和最简修法。
|
||||
**How to apply:** 复查用同一口径(terminal_result.json 的 target=board 是板上自测;workspace 小文件在归档后仍在本地,events.jsonl 归档后会被删)。DB 里有 417 条 9 月旧批次的 PAUSED_INFRA,不计入并发,可以忽略。
|
||||
|
||||
相关:[[stability-sweep-20261016]] [[v18-own-inputs-20261015]] [[verifier-audit-20261012]] [[check-20261016-1700]]
|
||||
@ -1,40 +0,0 @@
|
||||
---
|
||||
name: compile-server-reclaim
|
||||
description: model-compile 编译机要被回收(10-08 告知);主控实际占用、必须跟着搬的东西、Mac 不能当主控的原因
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 1b3cd916-3faa-4ef7-be14-c3fb07244493
|
||||
modified: 2026-10-08T08:25:37.022Z
|
||||
---
|
||||
|
||||
2026-10-08 用户告知:model-compile(120.48.151.164,244 核 / 1.4 TB,x86)要被回收。这才是"编译上 Salty、主控放 Mac mini"方案的真实动机,提速不是目的。
|
||||
|
||||
10-08 实测:
|
||||
- 主控很轻:3 个采集器约 1 核、7.8 GB;21 个 agent(42 个进程)约 2.9 GB、几乎不占 CPU。
|
||||
- 吃算力的是编译:hb_compile 138 个进程,其中 77 个属于 bench 轨迹,其余是同机器上别人的实验(tune1.py、local_val.py;web_review.py 一个进程就占 124 GB 内存)。
|
||||
- 本机盘 /home 上 bench 必须搬走的东西:`/home/ming01.wang/bench-state`(sqlite 账本和锁,131 MB)、`/home/ming01.wang/toolchains`(32 GB x86 环境)、生产配置 `/home/ming01.wang/s100p-glm-8way-20260928-2124/`。代码、轨迹、缓存都在网盘 /mnt/data(JuiceFS robotwin-archive)上。
|
||||
- 只能在 Linux 上跑的部分:agent 沙箱是 `unshare -Urmpf --mount-proc`(scripts/jail_setup.py),进程管理读 /proc、boot_id、pidfd。06-hbm 用 hb_verifier/hb_model_info(地平线 x86 工具链),占轨迹总时长 16%。
|
||||
- 不止 05 要 x86:02 基线、03 导出、04 校准数据、08 评测主机侧都用 `paths.model_python(job)`(网盘 cache/venvs 里的 x86 模型环境),05/06 用地平线工具链。只有 07(板端测速只是 ssh)和 agent/采集器不依赖 x86。
|
||||
- 模型接口:GLM 走公网 open.bigmodel.cn,Qwen 走 ai-api.d-robotics.cc。
|
||||
- 用户 10-08 的想法:Mac mini 只做任务下发,集群只提供执行环境。我的建议:能要到小云主机(8 核 32 GB)就用它当主控,因为它和网盘、集群在同一个机房,省掉数据搬运。
|
||||
- Mac mini:`ssh amos@192.168.66.251`(10-08 已装本机 id_ed25519 公钥,不要用密码),工作目录 `/Users/amos/work`(10-08 为空)。
|
||||
M6 12 核、**16 GB 内存**、剩余 121 GB 磁盘,macOS 27.0(arm64),没装虚拟机工具和 node。
|
||||
电源设置:sleep=1、autorestart=0(断电后不会自动开机)。改电源设置属于系统设置,要用户自己执行。
|
||||
10-08 实测能连通:GLM、Qwen、板卡 40009 端口、编译机 22 端口。
|
||||
- 我当时操作用的 Mac 是 Mac16,1(MacBook Pro M4,16 GB),不是那台 Mac mini。
|
||||
|
||||
**Why:** 换机器时最容易漏的是本机盘上的账本和工具链,以及"只把 05 搬走"不够:06 也离不开 x86 工具链。
|
||||
**How to apply:** 讨论新主控放哪时,先问能否拿到同机房的 Linux 机器;搬之前按 [[bench-restart-hazards]] 正常排空(只用 SIGTERM)。相关:[[rollout-time-breakdown-20261008]]
|
||||
|
||||
**2026-10-08 实际回收**(19:13–21:02 之间,ssh 主机密钥三种全换):没跑 `migration-meta-20260928/stop-time-refresh.sh`,所以本机盘上的东西只有 9-28 22:17 的 tar 备份(`/mnt/data/ming01.wang/migration-meta-20260928/home-tars/`:bench-state.tar、toolchains.tar 33G、s100p-glm-8way-*.tar、.local.tar 含额度账本)。9-28 的 bench-state 比 r20260929-0x 批次还早,**这几个批次的账本没有备份**;今天写的 raise-concurrency-20261008 配置也没备份。
|
||||
新机器 `ming01.wang@36.144.20.182`:112 核、944 GB、Ubuntu 22.04,JuiceFS 已挂 `/mnt/data`,`/home` 盘只剩 23 GB(放不下 32 GB 工具链),unshare 沙箱可用、python3.10 在、没有 node。**新机器上 ming01.wang 是 uid 1006,网盘文件属 uid 1004(新机器上叫 xiaoyan.zhao)**:bench 目录、批次目录全部不可写,`.env.local` 和每条 rollout.json 都读不了。网盘上的记录文件数(04=119、05=214、06=319)与回收前一致。
|
||||
**更正(2026-10-11 实测)**:网盘 bench/runs 下的文件现在属 uid 1006(36 上的 ming01.wang)、组 1004,36 上能读能写,rollout.json、agent/events.jsonl、HBM 都读得到。120 已于 10-11 关机清理,旧轨迹以后从 36 读。
|
||||
|
||||
**2026-10-09 捞回**:老机器重装后仍是 `root@120.48.151.164`(我用本机 `~/.ssh/id_ed25519_codex_180_76_192_183` 登录,用户自己 ssh-copy-id 装的),原 home 整盘在 `/home/cce/ming01.wang`(属 uid 1004)。账本 jobs.sqlite / quota.sqlite 记到 10-08 20:47,完整性 OK。已拷到新机器暂存区 `~/restore-20261009`(sha256 一致):bench-state、.local/state/bench-shared、三个配置目录、resume-glm-20261002-104649 日志、.ssh、toolchains/node。
|
||||
- toolchains 只有 node 在 9-28 后变过(多了 mcp-remote),其余 bench-venv 7.5G / oe-3.7.0 13G / oellm2 11G 可从网盘 `migration-meta-20260928/home-tars/toolchains.tar` 解;四块都在配置里,沙箱会全挂载,缺一不可。venv 的 base 都是 /usr/bin python3.10。
|
||||
- `paths.work_root=/home/ming01.wang/bench-work` 是每条轨迹的编译临时目录,老机器上 444G,不用搬(续跑不依赖),但新机器需要大盘装它(重模型一条 10–17G)。
|
||||
- 停机时在跑的 37 条(04=5、05=16、06=16)账本里仍是 RUNNING,启动前要先处理。
|
||||
|
||||
**更正(10-09)**:网盘上 `bench/cache`、`bench/data`、`bench/state` 都是指向 `/home/ming01.wang/bench-{cache,data,state}` 的软链接,**缓存和数据其实在本机盘**,不在网盘。老机器上:bench-cache 415G(baseline 160、onnx 88、weights 81、adapter-probes 80、venvs 1G),9-28 备份(374G,19:43 打包)之后新增 397 个文件 71G(baseline 49G + onnx 22G,可由 02/03 阶段自动重算,约 12.3+1 小时串行累计);bench-data 46G,备份后零改动,网盘备份即最新。LLM 网关 `.local/state/litellm-formal-gateway` 备份后改动 231G(不属 bench,用户定)。
|
||||
`cache_contract` 锁的是模型 Python 路径和导出器版本,不是缓存内容;路径用 abspath 比对、不解软链接,所以大目录可放网盘、原路径放软链接。开 Salty 后只有 05(PTQ)上云、06 读 Salty 回执;01/02/03/04/08 仍在主控跑,需要 bench-venv + 缓存 + 数据(约 380G+)。
|
||||
@ -1,39 +0,0 @@
|
||||
---
|
||||
name: delivery-package-v1
|
||||
description: 给内部评测同事的第一版交付包:位置、冻结点、口径、已知缺口
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: cf15f913-cbc4-4a5a-9046-1031104e17bb
|
||||
modified: 2026-09-19T07:48:01.108Z
|
||||
---
|
||||
|
||||
第一版评测交付包在服务器 `/mnt/data/ming01.wang/val-model-compile`,13 GB / 3932 个文件。
|
||||
本地构建副本在 `/home/ming01.wang/delivery-20260918`(未删,待用户决定)。
|
||||
|
||||
**冻结点:2026-09-18 17:18:27,62 条 `rollouts.verdict_terminal='PASS'`,40 个模型。**
|
||||
冻结当天池子还在跑,次日已涨到 87 条——出第二版时这个数还会变,必须重新冻结快照。
|
||||
|
||||
关键口径(出第二版仍适用):
|
||||
- 唯一正确的"完成"口径是 `rollouts.verdict_terminal='PASS'`。
|
||||
`state='TRANSFERRED'` 那 597 条**一次都没跑过**,是任务槽位转交的墓碑标记,
|
||||
按它打包会打出几百个空目录。
|
||||
- `completion.workflow_completed=true` 比 PASS 多出来的那些是"跑通但掉点超容差"。
|
||||
- `jobs.state='PASS'` 是 oracle 标准答案流水线,不是 agent 轨迹。
|
||||
- `jobs` 表按基准批次记账,n100 派生批次没有独立行——62 条里 26 条查不到,
|
||||
查询一律走 `rollouts` 表。
|
||||
|
||||
打包时踩过、第二版要直接避开的坑:
|
||||
- `bin/bench`(711) 和 `harness/cli.py`(600) 权限不对外,拷完必须 `chmod a+r`,否则同事用不了。
|
||||
- `stages/*/meta.json` 必须带(62 条共 443 个,543 KB)。有它同事没板卡也能离线重算出
|
||||
PASS;没有它 62 条全变 UNVERIFIED。
|
||||
- 必须按工作区拷,不能 `git archive`——大量适配器没提交。
|
||||
- 缓存目录要按需挑:baseline 全量 59 GB(大头是 G2 参考张量,G2 不参与判定,不带)、
|
||||
onnx 27 GB、calib 16 GB,实际只用到其中 12.4 GB。
|
||||
- 排除 `.env.local`、`.sync/`、`.git/`、`__pycache__`、`docs/archive/`。
|
||||
`.env.local.example` 和 `.gitignore` 可以带(模板是空的)。
|
||||
|
||||
已知缺口(已写进包内说明):分割/姿态/光流/深度/特征/占据/双目 7 类零样本;
|
||||
实例分割仅 1 条且其轨迹因版本过滤未导出;4 条合格轨迹不在数据集里(轨迹只覆盖 58/62)。
|
||||
|
||||
相关:[[prefer-subtraction-over-addition]] [[bench-restart-hazards]]
|
||||
@ -1,16 +0,0 @@
|
||||
---
|
||||
name: git-mainline-20261009
|
||||
description: 10-09 仓库主线整理:main 改为生产分支、旧 main 打标签、Salty 单独分支;本地工作区还剩未提交的优雅退出/板卡改动
|
||||
metadata:
|
||||
type: project
|
||||
---
|
||||
|
||||
仓库 `bench/`(gitea `ming01.wang/data-gen-bpu`)。2026-10-09 用户决定:生产分支 `production/s100p-infra-fix-20260928` 就是完整主线,旧 `main`(9-16 起分叉、基本没用)不要了。
|
||||
- 网盘上在跑的代码 = `c2c52ed`(gitea 生产分支也已是它)。旧 main 留标签 `archive/main-20260917`(12ef41c);`main` 改指 `c2c52ed`(10-09 已用 `--force-with-lease=main:c44d3f3` 推到 gitea,标签和 feature 分支也已推)。
|
||||
- 旧 main 的 8 个真改动里,watchdog 按线程判 D 状态、06-hbm 回收中间 ONNX 已核实在 c2c52ed 里。
|
||||
- Salty 远程编译单独成分支 `feature/salty-20261009`(45b1262,基于 c2c52ed),默认关闭(需 `execution.salty.enabled: true` 且 stages 含 05-compile)。合到生产代码时补了:编译上限只放行"已记录过 submit"的 Salty 找回(`salty_stage.dispatch_recorded`)。
|
||||
- 本地工作区仍在旧的 10663a8 上,留着未提交的"进程优雅退出(owned_process/namespace_owner)+ 板卡侧"改动;其中 `board_guard.py` 与生产分支 f315496 的隔离释放策略相反,合之前要用户定。
|
||||
- 在新机器上跑单测:`PYTHONPATH=. timeout 900 python3.10 scripts/test_x.py`;`test_agent_first`、`test_board_pipeline` 在干净的 c2c52ed 上也失败(环境未装好),不是回归。
|
||||
|
||||
**Why:** 以后问"main 怎么变了""Salty 在哪个分支""本地那堆改动是什么"时直接查这里。
|
||||
**How to apply:** 新工作从 `main` 开分支;动本地工作区那批非 Salty 改动前先问用户。相关:[[compile-server-reclaim]]
|
||||
@ -1,19 +0,0 @@
|
||||
---
|
||||
name: glm-second-account-20261008
|
||||
description: 10-08 加第二个 GLM 账号、三个批次换代提并发的现场:配置位置、接续脚本、目标路数、账号对应关系
|
||||
metadata:
|
||||
type: project
|
||||
---
|
||||
|
||||
2026-10-08 用户要求提采集并发。事实:
|
||||
- 05(S600,`S600_API_KEY`)和 06(S100P,`TEACHER_API_KEY`)在 GLM 那边是**同一个账号 A**;`S100P_API_KEY`是**新账号 B**,之前从没用过,只能走 coding 套餐接口。
|
||||
- 换代前 24 小时两个 GLM 组共约 5470 次请求、零次 429;8 路时在途请求 76% 时间为 0、峰值 8。账号额度不是瓶颈,瓶颈是每个 collector 只开 8 路、全局上限 24。
|
||||
- 做法(用户选的方案 B):06 改用账号 B(新账号组 `glm-coding-account-b`,规则照抄老组),并发 8→16;05 留在账号 A,8→16;04(Qwen)配置不变、仍 5 路,只为抬全局上限重启。三份配置的全局上限都改成 40。
|
||||
- 新配置和接续脚本在远端 `/home/ming01.wang/raise-concurrency-20261008/`;`relaunch-<run>.log` 记录每个批次何时排空、何时起新进程。日志仍追加到 `/home/ming01.wang/resume-glm-20261002-104649/restored-r20260929-0X-fix-20261003.log`。
|
||||
- 16:26 SIGTERM;30 分钟内 05/06 一条没跑完,按用户要求不等了:16:58 起取消全部 21 条在跑轨迹(预计只损失约 4 条 PASS),17:14–17:17 三个新进程起来(04=5 路、05=16 路、06=16 路,06 走 `glm-coding-account-b`)。
|
||||
|
||||
- 18:07 核对:05、06 各 16 路满载,06 只走 `glm-coding-account-b`,零 429。04 在 17:44–18:00 被另一个会话("qwen-3.8-27B评估")接管:取消了新 04 的 5 条,重启为只跑 `benchmarks/qwen-sft-v1.json` 的 37 个任务并改了 Qwen 超时和账本规则。动 04 前先确认那个会话是否还在用它。
|
||||
- 19:04 核对:换代 1.8 小时 05/06 一条没结束。原因是 32 条同时开工、正好排到最重的一批模型(vit.h.14、dab.detr 等编译历史中位 1.5–4 小时);编译本身没变慢(换代后完成的编译中位 31 分 vs 之前 28 分)。编译槽位 24/24 满,7 条在排队、最长等 38 分钟,负载 130/244;用户同意后 19:08 把 05/06 配置的 `pools.cpu.workers` 改为 32(原子替换,collection_v1 不变);只对之后新起的编译/HBM 进程生效,之前已排队的 3 个仍按 24 等。查编译耗时:`stages` 表里成功的状态是 `OK` 不是 `DONE`。
|
||||
|
||||
**Why:** 下次再问"并发怎么变了""换回去"时,要知道动了哪些、老配置原样留在原处。
|
||||
**How to apply:** 回退就用原目录 `/home/ming01.wang/s100p-glm-8way-20260928-2124/` 和 `/home/ming01.wang/qwen-autofill-20260928/` 的老配置照同样流程换代。相关:[[bench-restart-hazards]] [[compile-server-reclaim]]
|
||||
@ -1,40 +0,0 @@
|
||||
---
|
||||
name: hbm-spotcheck-optional-20261014
|
||||
description: 10-14 06-hbm 的 hbm/bc 模拟器抽查改成 agent 按需开启、只作诊断;默认不跑已部署,配置校验与说明等下次重启;还有 DNS 续跑也等重启
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 2b85e929-b893-42c3-a7a0-3a72bda14705
|
||||
modified: 2026-10-14T04:48:46.602Z
|
||||
---
|
||||
|
||||
**问题**:06-hbm 以前每次必跑 hbm/bc 模拟器抽查,用 x86 模拟器跑 2 张校准图。
|
||||
- 本地跑时,它占这一步 95% 的时间(中位 7 分钟)。
|
||||
- Salty 上要 1–2 小时;hb_verifier 只要约 40 秒。
|
||||
- 10-12 17:16 到 10-14,Salty 上撞 2 小时上限 103 次,约 206 小时。
|
||||
|
||||
它从没单独发现过问题:历史 388 次抽查报出 13 次不一致,hb_verifier 都已先判掉;08-eval 还会在板上跑真 HBM。
|
||||
|
||||
**用户的决定**:不加大时限,也不按参数量设时限。把抽查改成 agent 自主判断能不能用,代码要通用。
|
||||
- 不按参数量设:模拟器耗时主要看计算量和输入分辨率,不看参数量。
|
||||
- 不加大时限:时间照样花掉、占着 Salty 核,模型每重编一次就要再生成一次 HBM,会吃掉 24 小时总时限。
|
||||
|
||||
**做法**(f74fe00,叠在 ba143fa 上):
|
||||
- 阶段配置新增 `verification_parameters.hbm_bc_samples`,取值 0–16,默认 0。
|
||||
- agent 用 `bench stage run hbm --config x.yaml` 开启。
|
||||
- 结果写在 G1.hbm_bc_cos_min,**不改 verifier_ok**。否则开了只会被扣分,agent 就永远不会开。
|
||||
- 适用于任何有 .bc 的后端。
|
||||
- 编译配置只取三段(input、calibration、compiler 参数),所以同一个 yaml 拿去编译不受影响。
|
||||
|
||||
**上线情况**:
|
||||
- 已部署:s06_hbm.py(10-14 12:48),默认不跑抽查。这个文件随阶段打包进容器,部署即生效。
|
||||
- **等下次重启**:
|
||||
- stageworker.py 的配置校验和 agent/SYSTEM.md 的说明(f74fe00)。说明不能先上线,否则 agent 按说明提交会被旧校验拒绝(invalid_request)。
|
||||
- quota.py 的 DNS_FAILED / DNS_TIMEOUT 可以续跑(e4e1513)。06 的 fcn.resnet101、depth.anything.v1.large 从 10-14 凌晨起一直停着,等重启后续上。
|
||||
|
||||
**Why:** 用户要"合理方案",不是简单加大时限;而且要通用。
|
||||
**How to apply:**
|
||||
- 下次重启要部署 stageworker.py、quota.py、agent/SYSTEM.md。部署前先和分支比对 md5。
|
||||
- 看 06 用时要分 Salty 和本地。Salty 上的文件时间是解包时刻,不能拿来分段,要看日志里的时间戳。
|
||||
|
||||
相关:[[infra-leaks-20261013]] [[salty-container-threads-20261013]] [[bench-restart-hazards]]
|
||||
@ -1,31 +0,0 @@
|
||||
---
|
||||
name: hbm-step-bottleneck-20261014
|
||||
description: 10-14 发现 06-hbm 成了新瓶颈:HBM≡BC 模拟器抽查在 Salty 上 1–2 小时,一夜 45 次撞 2 小时上限;DNS_FAILED 暂停不在自动续跑名单里
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 24818155-52a9-4a37-a413-b2ac9f163981
|
||||
modified: 2026-10-14T04:13:47.996Z
|
||||
---
|
||||
|
||||
2026-10-14 12:00 复查 10-13 20:41 重启后 15 小时:
|
||||
|
||||
- **06-hbm 撞时限 45 次,浪费约 89 小时**,占所有超时的大头。05(S600 nash-p)成功时的中位耗时也要 48 分钟,06 是 16 分钟。
|
||||
- 根因:s06_hbm._hbm_bc_spotcheck 用 x86 模拟器跑 HBM 推 2 张图。旧编译机上 swin.s 这一步就占 36/37 分钟;Salty 容器只有 4 核,要 1–2 小时以上。
|
||||
- 这个抽查没有实际作用:历史 401 次里报出 13 次不一致,这 13 次 hb_verifier 的余弦都已经低于 0.97,早被它判掉了。07/08 也会在板上跑真 HBM。
|
||||
- 模型撞时限后会反复重试同一步。比如 05 efficientvit.b2 kk2 连撞 6 次,烧掉 12 小时。
|
||||
- 超时的作业日志拿不回来,只能从 Salty 远端目录里的 usage.log 看。
|
||||
- **DNS_FAILED / DNS_TIMEOUT 暂停不会自动续跑**:
|
||||
- quota._resume_readiness 的可续跑原因名单里没有这两项,但 DNS 失败发生在发送之前,重放是安全的。
|
||||
- 06 有 fcn.resnet101 kk1 从 03:01 起、depth.anything.v1.large kk2 从 06:01 起一直停着。
|
||||
- 根因还是 Clash 的 DNS,见 [[mac-mini-clash-global-proxy-20261013]]。
|
||||
- 已确认正常的部分:
|
||||
- CONNECT_FAILED、REQUEST_OUTCOME_UNKNOWN 暂停后 1–30 分钟内自动续上(共 9 次)。
|
||||
- 孤儿容器清理在工作。
|
||||
- CPU 名额没有泄漏。
|
||||
|
||||
**Why:** 修完 [[infra-leaks-20261013]] 后吞吐回升,06-hbm 接着成了下一个卡点。
|
||||
**How to apply:**
|
||||
- 提速先砍 HBM 抽查,或把它降到 1 张,不要先加时限。
|
||||
- 改 s06_hbm 是阶段代码,新起的步骤马上生效。
|
||||
- 改 quota 可续跑名单要等重启调度程序才生效。
|
||||
@ -1,60 +0,0 @@
|
||||
---
|
||||
name: infra-leaks-20261013
|
||||
description: 10-13 查出四个基建问题:CPU 名额被超时步骤锁死(32 剩 6)、超时后 Salty 容器不停、续跑必死、领题后重复查额度把题作废
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 2b85e929-b893-42c3-a7a0-3a72bda14705
|
||||
modified: 2026-10-13T14:25:52.406Z
|
||||
---
|
||||
|
||||
2026-10-13 11:00 核查 Codex 报的"新增 51 UNVERIFIED、21 INFRA_ERROR、25 条在等阶段队列",几乎全是系统问题:
|
||||
|
||||
1. **CPU 名额泄漏**:state/locks/cpu-N.lease.json 由步骤进程退出时删(scheduler._release)。步骤超时(PHASE_EXECUTION_DEADLINE_EXCEEDED)被看门狗终止时来不及删,_try_slots 见到凭证就跳过这个名额。scheduler.reconcile_leases 只在 recovery.reconcile_run 里调(调度程序启动时),而且只放已结束(TERMINAL)的轨迹。10-12 18:07 起累计 26/32 个被锁死,只剩 6 个,排队步骤最长等 3 小时。pool_usage() 只数 flock,看不出这个问题,要看 lease 文件的持有进程是否还活着。
|
||||
2. **超时后远端容器不停**:本地步骤被杀,Salty 容器继续跑。10-13 11:00 有 7 个,最久跑了 7.5 小时,约 34 核。查法:salty/remote.json 的 job_id 在 salty ps 里是 running,但同目录 meta.json 已经写了结果。
|
||||
3. **暂停后续跑必死**:续跑时 stageworker.scan 重新核对每个旧步骤的回执。同一条轨迹每次编译都写同一个文件(05-compile-work/<job>.hbm、stages/04-calibrate/compile.yaml),旧回执核对时 ValueError("completed artifact changed"),接着 OPERATION_STATE_UNKNOWN 隔离整条轨迹。只要编译过两次以上,续跑后几秒就判 INFRA_ERROR(10-12 晚到 10-13 共 19 条)。reconcile_completed 里已经有"已结算就跳过"的判断,scan/_execute 没有。
|
||||
4. **领题后重复查额度**:pipeline 领题前调用 runner.check(execute=True),查不到额度会把题留在队列;rollout.run_rollout:317 领到之后又查一次,quota_snapshot 只等 0.5 秒。00:25–00:51 机器忙时读超时,QuotaError: QUOTA_SOURCE_UNAVAILABLE,题直接记成 UNVERIFIED,根本没开始跑(05 有 43 条,06 有 7 条)。
|
||||
|
||||
**已做(10-13 11:00–11:30,用户同意)**:
|
||||
- 停掉 7 个无人等结果的 Salty 容器(只停,没删作业和个人空间目录)。
|
||||
- 回收 26 个名额:把 lease 文件挪到 configs/config-backup-20261013/leases/。20 秒内 32 个名额又全部用上。
|
||||
- 59 条从没跑过的 UNVERIFIED(05 有 45 条,06 有 14 条)改回 QUEUED,并记了注释 REQUEUED_20261013。在跑的调度程序把它们记在已领过的名单里,下次重启后才会重新领。
|
||||
- 7093c32(11:11 部署,新起的步骤进程马上生效):
|
||||
- scheduler._lease_settled:占用者已死、该步骤已 DONE、清理已确认,就接手这个名额。板卡名额不在此列。
|
||||
- salty_remote 提交容器时加 `timeout <阶段执行时限>`。镜像里有 /usr/bin/timeout,实测到时限退出码 124。
|
||||
- ec1a58e(**未部署,等这批跑完、重启时一起上**):
|
||||
- stageworker._execute:已结算的操作直接沿用旧结果,不再重新核对。
|
||||
- rollout.run_rollout:领到题后不再查额度。
|
||||
|
||||
**10-13 20:28–20:41 重启(用户选"今晚取消在跑的直接重启")**:
|
||||
- 脚本和清单在 configs/restart-20261013/:
|
||||
- stop.sh:存旧环境,SIGTERM,并行取消。
|
||||
- start.sh:部署,补号,用 withenv.sh 照抄旧环境重启。
|
||||
- 取消了 43 条:37 条在跑,加上 6 条不可能续跑的暂停轨迹(账本里已放弃,或续跑次数用完)。06 的 5 条可续跑的暂停轨迹保留。
|
||||
- 05 yolov8s kk2 的取消命令卡在账本收尾 13 分钟。它的执行记录已经结束,所以用 SIGTERM 结束了这个命令。
|
||||
- 部署 ec1a58e(D、E)和 8a1ce8e:
|
||||
- 额度账本读取时限 0.5 秒改为 5 秒,领题前和续跑前两处检查都改了。下午续跑失败的真因其实是这个,不是"名额被抢"。
|
||||
- INCOMPLETE_GENERATION 可以续跑。
|
||||
- 补号:42 条取消的各补一个新 kk;202 条一步没跑就被作废的 UNVERIFIED 放回 QUEUED(有些在上午放回后又被作废了一次)。
|
||||
- 没重启 salty_offload:它当时正在跑一个导出作业,等它空闲时再重启,这样导出作业也能带上线程上限。
|
||||
- 测试注意:VM 里有代理环境变量,跑 scripts/test_quota 要 `env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY`,否则本机测试服务器全回 502。test_collection 的 test_real_database_lock_* 本来就时好时坏。
|
||||
|
||||
**10-13 21:53 核对,前三项都已修好**:
|
||||
- 11:11 后又有 36 次步骤超时,CPU 名额没有一个被死进程锁住;等名额的中位时间从 105–127 分钟降到 0。
|
||||
- 重启后 5 条续跑全部存活。
|
||||
- 读额度账本超时 0 次。
|
||||
- 漏洞 2 的修复只管超时,不管取消:20:28 重启取消的轨迹留下 11 个容器白跑,另有 1 个是 11:11 修复前提交、没有时限的 gmflow。
|
||||
- 已停掉 9 个,另外 3 个已经自己结束。
|
||||
- ad51e86:salty_offload 每轮开头调用 reap_orphans。本地这一步已有 meta.json、远端还在 running/pending 的就停;已 stopped 的用 sr._cleanup 删除作业和临时目录;failed 的保留。
|
||||
- 部署后由 configs/restart-20261013/restart-offload.sh 等旧进程空闲再重启它。
|
||||
- 踩坑:
|
||||
- VM 里 Python 的 time.sleep 显示的 wchan 是 do_select,不是 nanosleep。
|
||||
- pkill -f 会匹配到命令行里含同名字符串的 ssh/bash 自身,把自己杀掉。
|
||||
|
||||
**Why:** 用户以为是模型能力问题,实际上吞吐被系统卡住,数据被系统作废。
|
||||
**How to apply:**
|
||||
- 看吞吐先查 lease 文件,看持有进程是否还活着,不要只看 pool_usage。
|
||||
- 第 3、4 条的代码跑在调度程序进程里,要重启才生效。
|
||||
- 删 lease 文件和停 Salty 作业要先问用户。
|
||||
|
||||
相关:[[salty-container-threads-20261013]] [[selfheal-and-alert-20261012]] [[bench-restart-hazards]]
|
||||
@ -1,19 +0,0 @@
|
||||
---
|
||||
name: mac-mini-clash-global-proxy-20261013
|
||||
description: 10-13 查明:Mac mini 上 Clash Verge 全局模式+TUN,采集的全部流量(Qwen 自建服务、Salty、GLM)都绕美国洛杉矶 JMS 节点;断线、Salty 网络错误、请求慢的共同根因
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 6c848f33-e8d9-4981-ade2-dc77987005d7
|
||||
modified: 2026-10-13T08:13:56.815Z
|
||||
---
|
||||
|
||||
2026-10-13 16:30 查 Qwen 卡住的原因时发现:
|
||||
- VM 里采集进程环境 http(s)_proxy=192.168.5.2:7897(Mac mini 上的 Clash Verge / mihomo),没有 no_proxy。
|
||||
- Clash 是 **mode: global + TUN 开启**(verge.yaml enable_tun_mode: true,enable_system_proxy: true),GLOBAL 选的是 JMS 节点(日志 10-09 21:39 起就是),规则末尾 MATCH,JMS,没有 GEOIP,CN 直连规则。
|
||||
- 实测:VM 走代理、VM "直连"、Mac mini 本机直连,出口 IP 全是 **美国加州洛杉矶 IT7 网络**。对国内 Qwen 服务器一个小请求要 1.5–2.7 秒。
|
||||
- Clash 每 8 分钟更新一次订阅(profiles.yaml update_interval: 8),每次都 unset/set system DNS,10-13 一天 121 次;12:13 有一次"直接更新失败"。
|
||||
- 10-13 Salty API 网络错误集中在 12:10–12:50(245 次)和 14:00–14:10(132 次),与 Qwen 请求断在半路(12:20、12:53、13:54、14:25)、GLM CONNECT_FAILED 同时发生。
|
||||
|
||||
**Why:** 这是断线、Salty 网络错误、请求慢的共同根因(之前我以为是 Wi-Fi,不对)。**更正(10-14)**:10-13 Qwen 停摆的主因还有另一会话查出的系统漏洞——CPU 名额被超时步骤锁死(我当时看到的"等 Salty 容器几十分钟"其实是等本地名额)、续跑必死、读额度账本 0.5 秒超时,见 [[infra-leaks-20261013]];10-13 20:41 已修。代理问题到 10-14 中午仍未改。
|
||||
**How to apply:** 改 Clash 属于用户本机网络设置,我不能动,让用户在 Clash Verge 里改:要么 GLOBAL 选 DIRECT,要么切规则模式并给 114.111.18.115、*.d-robotics.cc、open.bigmodel.cn、板卡网段加 DIRECT 规则;订阅自动更新间隔调大。Mac mini 上还跑着 ChatGPT/Codex,可能需要境外代理,所以推荐规则模式而不是关掉。只设 no_proxy 不够:TUN 仍会接管。改完用 VM 里 `curl --noproxy '*' http://cip.cc` 确认出口在国内。相关:[[selfheal-and-alert-20261012]] [[mini-salty-controller-20261009]] [[qwen-ability-failures-20261013]]
|
||||
@ -1,14 +0,0 @@
|
||||
---
|
||||
name: memory-synced-to-repo
|
||||
description: 记忆要同步到仓库 docs/memory/(feature/salty-20261009 分支);入口 docs/memory/AGENTS.md 写了创建和存放规则
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: feedback
|
||||
originSessionId: 0f0f2556-2514-40e5-8ef7-f2c4e67c5701
|
||||
modified: 2026-10-11T05:22:28.020Z
|
||||
---
|
||||
|
||||
2026-10-11 用户要求把记忆推到远端仓库:放在 docs 目录下,用 AGENTS.md 引导,并写清楚怎么创建、怎么存放。现在仓库里的 `docs/memory/` 就是共享记忆库(Codex 等别的 agent 也读),入口是 `docs/memory/AGENTS.md`(规则和索引),根目录 AGENTS.md 里有指向它的说明。
|
||||
|
||||
**Why:** 用户同时在用 Codex 等别的 agent(10-11 起监控交给了 Codex),经验不能只留在我本机。
|
||||
**How to apply:** 每次新增或修改记忆后,把文件同步到工作树的 docs/memory/,更新 AGENTS.md 的索引,提交到 feature/salty-20261009 并推送。推之前先 grep 一遍,确认没有密钥或密钥片段。相关:[[git-mainline-20261009]]
|
||||
@ -1,75 +0,0 @@
|
||||
---
|
||||
name: mini-salty-controller-20261009
|
||||
description: 10-09 起 Mac mini 主控 + 每个阶段都在 Salty 容器里跑(salty_remote);各文件位置、为什么这样设计、还缺什么
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: a9e0f820-63b5-4eba-bcf3-24ad6f20af42
|
||||
modified: 2026-10-12T03:03:09.463Z
|
||||
---
|
||||
|
||||
10-09 用户要求把 04/05/06 三个批次搬到"Mac mini 主控 + Salty 执行 + 板卡验证",并让我直接打通。
|
||||
|
||||
**设计**:Mac mini 只跑 agent、调度、账本、配额、锁;agent 每次 `bench stage run`(01–08 全部)都在 Salty 起一个容器跑完整的 Stage.run(含查缓存),再把阶段目录(≤32MB 的文件)和板卡回执取回。代码:feature/salty-20261009 上的 harness/salty_remote.py + salty_remote_worker.py,钩子在 stages/base.py Stage.run 开头;配置键 execution.salty_remote。旧的 05-only salty_stage 保留但不启用。
|
||||
**Why:** 缓存/数据只在 Salty 个人空间,Mac mini 看不到;只远程跑命令不行,查缓存也必须在容器里。Salty 不能在新命名空间挂 /proc(09-30 实测),所以 agent 沙箱只能在 Mac mini;剩余题都不带 recipe,不需要容器里的沙箱。
|
||||
**How to apply:** 容器里用软链接复刻老编译机路径:/home/ming01.wang/{toolchains,outputs}→/data/personal/bench-runtime/home/…,{bench-cache,bench-data}→/data/personal/bench/{cache,data},/mnt/data/ming01.wang/model-compile-hbdk/bench 是真目录(BENCH_ROOT 会 resolve,不能是软链),runs→/data/personal/bench-root/runs。Mac mini 虚拟机里同一路径也是真目录。
|
||||
|
||||
位置(Mac mini 10-14 20:2x 起从外面连用 `ssh -p 443 amos@180.76.192.183`(公网转发,Mac mini 内网仍是 192.168.66.251,本机不在同一网段时内网地址连不上),报警脚本写成 ssh://amos@180.76.192.183:443;10-12 起内网是 amos@192.168.66.251;10-10 22:30 至 10-12 是 192.168.1.3;地址随重启/换网会变,连不上先问用户;仍走 Wi‑Fi,网线口 en0 空着):
|
||||
- Mac mini 虚拟机:bench 根 /mnt/data/ming01.wang/model-compile-hbdk/bench(state/ 是 10-08 20:47 账本,sha 2d583fb0…),配额 ~/.local/state/bench-shared/,配置 /mnt/data/ming01.wang/configs/mini/*.yaml,板卡私钥/指纹/端口 ~/.bench-board/,key 录入脚本 /mnt/data/ming01.wang/configs/set-keys.sh(用户自己跑)
|
||||
- 个人空间:personal:/bench-runtime/(三套环境 tar、board.env、ssh/、controller/ 账本备份);personal:/bench/ 归"salty迁移"会话,它负责解压 _tars
|
||||
- 120 临时目录 /root/bench-runtime-20261009/(迁完要删);36 临时 ~/runs-pack-20261009.*、~/runs-essentials-*.tar.gz
|
||||
- 本地 worktree:scratchpad/bench-salty(feature/salty-20261009),用完 `git worktree remove`
|
||||
|
||||
板卡:8 块都用 rdk_sandbox 私钥直连,指纹已固定;没用用户给的密码。rdk-sandbox(slot0)=40010。
|
||||
网络:36→本机 Mac 只有约 60KB/s;Mac mini 从个人空间下载约 1.3MB/s,大件走个人空间中转。
|
||||
|
||||
10-10 开跑后的教训(代码都在 feature/salty-20261009,0cf3c20/949c5da 等):
|
||||
- 编译工作目录必须在每个容器都是本地 /tmp/bench-work(base.work_dir 会 resolve 软链,04 写进 compile.yaml 的路径要和 05 一致);05/06 之间经 personal:/bench-root/work 交接。
|
||||
- 主控要在本地复核 HBM(编译回执哈希、最终判定读 rollouts/NN/artifacts/*.hbm),所以 05 的 HBM 和新写的 artifacts 必须取回;06 用 copy2 保留旧 mtime,要按快照比对取回。
|
||||
- Mac mini 走 Wi‑Fi 会网络抖动:salty describe 失败要重试(只读),大模型请求会变 PAUSED_INFRA(采集 --watch 自动恢复)。
|
||||
- Salty 没有个人额度上限(10-11 复现:我们同时占 42 核/128G 没排队;平台方也确认总额度不缺、无"每人 3 个")。之前说的"约 25 核上限""两小时自动失败"都是我猜错的,已收回。`salty ps` 的状态会滞后(作业日志已开跑,ps 还显示 pending),短暂 pending 可能只是显示慢。编译用 6 核(运行中改 cpu_by_stage 只影响之后新起的 stagerun)。
|
||||
- 作废名额的补法:旧行保留,db.rollout_upsert 新 kk + annotate 写原因;残留远程作业用 salty stop/rm/storage rm -r 清。
|
||||
- Mac mini 关机/换网会打断在跑轨迹(重启后 PAUSED_INFRA 且隔离,resume 不可用,只能取消+补回)。
|
||||
|
||||
10-10 下午的教训(6a013d6):
|
||||
- 356fdaa 起 Salty 传输层可重试(连续失败 30 分钟才算失败),上线后 20 次网络错误全被吸收;之前的 INFRA_ERROR 全是旧传输层的 OPERATION_STATE_UNKNOWN。
|
||||
- 容器异常退出时主控补写的 meta.json 必须带 ctx.extra["operation_identity"],否则 stageworker 报 completed attempt metadata mismatch → 轨迹 PAUSED_INFRA 且续跑也过不去,只能作废补号(6a013d6 已修)。
|
||||
- `bench execution cancel` 要等满轨迹自己的 cleanup_deadline_s(执行级 1800 秒),有操作停在 DISPATCHED 时就会等满 30 分钟,然后标"清理未确认"——不是卡死。
|
||||
- 大模型请求断网(CONNECT_FAILED/REQUEST_OUTCOME_UNKNOWN)是设计上的"先暂停、不重试",整组账号一起暂停,靠 collect --watch 自动续(每条最多 3 次)。根治靠网络。
|
||||
**更正(2026-10-12):只有 REQUEST_OUTCOME_UNKNOWN 和额度等待超时会自动续;CONNECT_FAILED 永远不会自动恢复(quota.py _resume_readiness),见 [[salty-throughput-drop-20261012]]。**
|
||||
- Mini 整机会掉线约 1 分钟(10-10 16:00 实测 ping 不通);Wi‑Fi 是 5GHz 160MHz(信道 40),可能撞雷达避让。
|
||||
- 04 里 iseg yolo11s.seg/yolo11x.seg/yolov8m.seg 的 kk0 PAUSED 是 10-08 旧行、不在 37 题清单里,不用管。
|
||||
- 容器"无声死亡"(REMOTE_WORKER_FAILED,Salty 原因为空、日志为空):大模型编译(unimatch.flow ×4、mask2former.swin.b ×2、sea.raft、ocrnet)和 depth.anything 的 06-hbm(8G,2 分钟内死)。Salty 容器是 cgroup v2 且 memory.oom.group=1:任一子进程超内存就整个容器被杀、原因留空;worker 本身不往 stdout 打字,所以日志为空分不出死法。往 /tmp 写满约 39G 也会被驱逐(节点盘 99G 共用),但 06-hbm 死时 /tmp 还剩 54G,主因是内存。10-10 19:40 起 05-compile 32Gi、06-hbm 16Gi;dd20d19 起容器每 15 秒把内存/磁盘/最大文件记到 personal:/bench-stage-*/usage.log,失败时附进 stderr.log。
|
||||
- 10-10 19:45 Mac mini 整机意外停机(无关机记录、无 panic),20:32 才重新开机;虚拟机不会自己起来,且因 ha.pid/vz.pid 的进程号被别的系统进程复用而显示 Broken。恢复:确认没有 limactl/VZ 进程后,把 ~/.lima/bench/{ha.pid,vz.pid,ha.sock,default_*.sock} 挪到 ~/lima-stale-20261010/,再 `limactl start bench`(别用 stop --force,会按旧进程号发信号)。
|
||||
- 重启后上次开机的所有执行都会被隔离(OWNERSHIP_UNKNOWN / SIDE_EFFECT_UNKNOWN),这是有测试守着的设计,不改。处理:Salty 上孤儿作业全部 stop/rm、目录 rm -r;`bench execution reconcile` 列出非 TERMINAL 的;并行 `execution cancel`(有阶段在跑的几秒结束,只是暂停的要等满 30 分钟清理期限);configs/requeue-after-crash-20261010.py 补新 kk;旧开机的 lease 归档到 state/locks-stale-*;再起采集。reconcile 的 ready=False 来自历史上 TERMINAL+QUARANTINED 的旧执行,不挡采集启动。
|
||||
- 监控脚本要报"连不上",否则掉线一小时都是静默(scratchpad/watch-prod.sh 已改)。
|
||||
- 重启恢复已做成脚本:configs/recover-after-reboot.sh "<原因>"(先手动 salty stop/rm 孤儿作业和 bench-stage 目录,再后台跑它;它会 reconcile→并行 cancel→requeue-cancelled.py 补 kk→归档旧开机 lease)。10-10 22:34 第三次重启用它恢复,8 条作废补回。
|
||||
- 按需加内存(df64aa8):默认编译 16Gi、06-hbm 8Gi,同一阶段容器 failed 过几次就翻几倍(上限 64Gi);统一给 32Gi 时共享队列只放得下 1 个编译。10-10 晚上队列很挤,6 个 16Gi 编译全排队,采集降到 05/06 各 2、04 一路。
|
||||
- 容器无声死亡已坐实是内存超限:10-11 01:04 UTC retinanet.r101 编译(salty-f4b63fd8…,6核16G)死前 15 秒内存冲到 15.5G;复现作业 bench-repro-oom-20261011(salty-de281d92…)同样 failed、原因 "-"。平台侧要做的是 describe 原因里带 OOMKilled/Evicted/退出码。665bbef 起失败作业不再 rm,留给平台取证(要手动清)。
|
||||
- 开不到 16/16/5 的真原因在 Mac mini(10GB 虚拟机),不在 Salty:① 调度程序内存涨到 5GB,10-11 07:08 05 的调度程序被内核 OOM 杀——Pi 每个 token 写一条 message_update,带两份整条消息(message、assistantMessageEvent.partial),events.jsonl 每条轨迹 300–800MB,budget_used 每次 read_text 整个文件。f1f44c9 修:Pi 不再写这两份快照(下游只用 type/delta),budget_used 逐行读。② 同一原因磁盘一天涨 16G(盘 78G)。③ 修完后每条轨迹在主控约 210MB(Pi 约 150MB),37 路约 8.7GB,太紧;10-11 起 05/06/04 = 12/12/4,要上 16/16/5 需把虚拟机内存调到 12GB(要用户同意,且需重启虚拟机)。
|
||||
- 调度程序的 Python 代码只在启动时加载:改 harness/*.py 后,正在跑的调度程序仍是旧代码,必须排空重启(configs/restart-after-drain-20261011.sh <旧pid> <配置> <run> <并发> [job清单])。
|
||||
- 10-11 用户要求"尽量用别处资源,Mac mini 只留最终数据集":configs/archive-finished.py 常驻(日志 logs/archive-finished.log),轨迹结束满 30 分钟后打包到 personal:/bench-root/archive/<run>/<job>/<kk>.tgz(容器产物在 personal:/bench-root/runs 已有同大小副本的不进包),核对后只删本地 .bc/spot npz/onnx 等。**坑:调度程序每结束一条轨迹就跑导出器(refresh_run_dataset),它重读每条轨迹的 agent/(SFT 原文)并按 sha256 重验 artifacts/<job>.hbm;删了它们导出会把 PASS 变 UNVERIFIED、SFT 变 0(10-11 12:00 踩过,已用 restore-agent.py / restore-hbm.py 从个人空间取回)。这两样必须留在本地,除非导出改到 Salty 上跑。**
|
||||
- 10-11 10:22 虚拟机内存 10→12GiB(备份 ~/lima-bench-lima.yaml.before-12g-20261011;宿主机 16G、已用 3.4G 交换,别再加);之后 06/04/05 = 16/5/16。正常 limactl stop 时调度程序把在跑轨迹记成 INFRA_ERROR(不是隔离),recover 脚本不会补它们,要按 finished_at 那一分钟单独补号。
|
||||
- c8a93c9 内存按题记:同一 job 任一轨迹该阶段 failed 过,就从当时内存(meta.salty_memory)的两倍起,上限 128Gi。
|
||||
- 10-11 用户定:①导出挪到 Salty;②hb_verifier 没测就崩溃算"未验证"(2018b62,只对之后的轨迹生效,yolo11n.seg 那条历史 FAIL 不改);③监控交给 Codex,我不再挂监控;④120 已关机清理。
|
||||
- 导出挪到 Salty(scripts/salty_offload.py,常驻在 Mac mini 上,取代 configs/archive-finished.py):每 10 分钟把结束满 30 分钟(UNVERIFIED 24 小时)、执行 TERMINAL、PASS 已审完的轨迹打包到 personal:/bench-root/archive/<run>/<job>/<kk>.tgz(之后的改动另传 <kk>.u<时间>.tgz,不覆盖旧包),然后删本地所有 >1MB 文件;有新包时最多每小时起一个 Salty 作业(salty_remote_worker.export)把包解到 personal:/bench-root/runs、跑 export/to_agentrl.py,数据集留在 personal:/bench-root/datasets/,再拷回 Mac mini 的 bench/datasets-final/(这是最终数据集)。调度程序自己每条结束后的本地导出(bench/datasets/)保留:它负责在文件还在本地时发现并审核新的 PASS,搬运要等它审完。
|
||||
- Salty 上导出的坑:容器里 runs 不能是指向个人空间的软链接。verify.py 会 resolve 轨迹目录,并拒绝走软链接的预测文件,路径一对不上,G3 就变"未验证"、审核回执的绑定也跟着变(第一次实测 SFT 0 条)。现在 export() 在容器本地建真实的轨迹和阶段目录,agent/、artifacts/ 用软链接;解包不解 workspace(它是大头,导出不读)。修完后和 Mac mini 本地导出逐条一致(3/3/2)。最终数据集在 bench/datasets-final/,Salty 上一份在 personal:/bench-root/datasets/。
|
||||
- 数据集版本门槛:旧网盘上 04/05/06 数据集的 min_agent_version 是 12;Mac mini 上的数据集是新建的,默认用了当前的 15,迁移前的 PASS 有 260 条(版本 12/14)会被直接丢掉。10-11 已用 `to_agentrl.py --min-agent-version 12` 改回 12(本地三份 filter_history 有记录;Salty 上的 manifest 删了,按本地的重新播种)。新建数据集时先查原批次的门槛。
|
||||
- 10-11 迁移前的 291 条 PASS(04/05/06 早于 10-09 开始的)已从 36 经 Mac mini 中转到 personal:/bench-root/archive/(13:28–16:46,零失败;不带 workspace、.bc/.onnx、06-hbm 抽检 npz;中转脚本已删)。
|
||||
- 10-11 补审:Salty 导出列出的"没审过/审核未知"44 条,用 configs/review-oldera-20261011.py 逐条取回 → 审 → 回执作为补丁包传回 → 立刻删本地(旧格式对话最大 650MB,读一条要 1.3G 内存,一次全取回会让三个调度程序的本地导出一起重读、撑爆内存)。结果 33 过、7 不过、4 未知:3 条是请求体 68–77 万字节超过审核账号 tpm 70 万的本地估算(字节当 token 算,改 collection_v1 要排空重启,等用户定),1 条是 10-08 旧服务器审到一半留下的 STARTED 回执(按设计不自动重发)。审核模型偶尔回格式不对(INVALID_REVIEW_SCHEMA 等),重试一次就过。
|
||||
- 10-11 19:32 最终数据集 SFT 04/05/06 = 30/76/136(共 242;旧网盘 10-08 是 177)。迁移前 291 条 PASS 全部有判定:进 SFT 234、审核不过 46、未知 3、轨迹不完整 8。一次性脚本(中转、补审、restore-*)都已删。
|
||||
- 10-12 用户确认审核模型 deepseek v4 flash 有 1M 上下文:三份配置新增额度组 deepseek-review-account-1m(规则照抄、tpm 70万→210万,覆盖 content_review 200 万字节的请求上限),reviewer 改走新组(原配置备份 configs/config-backup-20261012/)。额度账本按组名锁规则(同名改字段报 QUOTA_POLICY_CONFLICT),所以只能新开组。在跑的调度程序配置是启动时读的(lru_cache),下次重启才走新组;在那之前它们审长轨迹仍可能得 UNKNOWN(REQUEST_EXCEEDS_INTERNAL_TPM),用 scripts/salty_review_again.py 补(先停 salty_offload.py)。
|
||||
- 补审工具 scripts/salty_review_again.py RUN CONFIG [--release-started] [--only ...]:逐条从存档取回 → 审(有 COMPLETE UNKNOWN 就按重试走)→ 回执作补丁包传回 → 删本地。STARTED 回执的坑:06 shufflenet.v2.x1.5 的 10-08 请求其实完成了(账本 generation COMPLETE、HTTP 200),是写回执时崩了;按原执行号重审会被账本当作"已完成"直接不发。正确放行是把 STARTED 记成 COMPLETE UNKNOWN(OPERATOR_RELEASED_STARTED_RECEIPT,原件进 history/),走重试换新执行号(工具 --release-started 就这么做,用户 10-12 批准)。
|
||||
- 10-12 补审结果:04 yolov8m、mspn.r50,05 simplebaseline.r101、xfeat,06 shufflenet.v2.x1.5 通过;04 mobilenet.v3.large 不通过。迁移前 291 条 PASS 已无"未知"。
|
||||
- 36 上 ~/restore-20261009 只留 .ssh(120 的两把私钥)、qwen-autofill/s100p-glm-8way 旧配置、resume-glm 日志(别处没有);账本/额度/raise-concurrency/node 已核对个人空间 controller-state.tar.gz 和 Mac mini 有完整副本后删除(备份只缺空锁文件)。~/restore-logs 是别的会话恢复家目录的核对记录,不是我的,别删。
|
||||
- 导出的第二个坑(df51a7c):个人空间 runs 里还有容器替主控已放弃的尝试写的文件(xfeat 06-hbm a00 主控判超时,容器继续跑完写了 stdout.log),照搬会让门证据多出文件、审核绑定对不上。现在解包时把成员清单写进 .unpacked-* 标记,导出目录只按清单搭;HBM 从个人空间链接。
|
||||
- 10-12 03:40 最终 SFT 32/79/139(共 250);迁移前 291 条:SFT 237、审核不过 46、轨迹不完整 8;迁移后新跑 PASS 17 条:SFT 13、审核不过 4。没有"未审/未知"。
|
||||
- 每次 Salty 导出 45–75 分钟(搭目录约 30 分钟、个人空间读取时快时慢,05/06 各十几分钟),常驻程序等导出时不搬运;导出只在有新存档时发生,最多每小时一次。想更快就把 salty_offload.py 的导出间隔调大或减少阶段文件拷贝,目前没必要。
|
||||
- 10-12 10:06 用户直接重启 Mac mini,没先停虚拟机,虚拟机被强停(last -x 里上次开机没有关机记录)。在跑的 37 条轨迹的 execution.json 恰好在那一秒(10:06:37)写入,结果变成全零或半截,尽管 atomic_json 有 fsync。reconcile、recover-rollout 诊断读到它们都会抛 JSONDecodeError,恢复脚本因此第一步就失败。采集程序本地的 06 数据集文件(datasets/*-training/*.jsonl)也被清零,但它每次刷新都整份重写,会自己恢复;datasets-final 没受影响。sqlite(jobs、quota)quick_check 正常。
|
||||
处理办法:
|
||||
1. 先清 Salty 孤儿作业:在跑的 stop+rm,已成功没取回的 rm,再 storage rm -r 对应的 bench-stage 目录;失败的作业留着。
|
||||
2. 把坏的 execution.json 改名为 execution.json.corrupt-<日期> 留档。recover-rollout 只接受没有 execution.json 的轨迹。
|
||||
3. 执行 `bench recover-rollout --run R --job J --kk K --apply --reason ... --confirm-owner-dead --confirm-work-stopped`,旧号会变成 ABANDONED,同时补一个 QUEUED 新号。
|
||||
4. 再跑 configs/recover-after-reboot.sh,处理状态完好但没跑完的 3 条和旧开机留下的锁。
|
||||
清单和日志:logs/corrupt-execution-1012.txt、logs/recover-rollout-1012.log、logs/orphans-1012.log。
|
||||
**以后重启 Mac mini 之前先 `limactl stop bench`**,让虚拟机正常关机:在跑的轨迹会记成 INFRA_ERROR、文件不会写坏,但要按 finished_at 那一分钟单独补号。
|
||||
@ -1,96 +0,0 @@
|
||||
---
|
||||
name: net-resilience-board-deadline-20261014
|
||||
description: 10-14 网络小故障拖垮轨迹的三处修复(原地重发/免费续跑/结果不明可续跑,等重启)、板卡清理没确认的根因与修复(已部署)、下次重启要上的完整清单
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 2b85e929-b893-42c3-a7a0-3a72bda14705
|
||||
modified: 2026-10-14T07:55:38.311Z
|
||||
---
|
||||
|
||||
**网络(f83fa39,等重启)**
|
||||
|
||||
GLM 请求失败率约 0.3%,常成堆出现(10-14 12 点那一小时有 14 次连不上)。以前一次失败就让整条轨迹暂停,续跑 3 次的额度被网络故障用完后就卡死;回复中途断开的请求还会被自动放弃,同时取消整条执行(36 小时里 16 次)。
|
||||
|
||||
- A:broker 遇到确定一个字节都没发出去的失败(RETRY_UNSENT:CONNECT_FAILED / DNS_*),在原地等 cooldown_initial_s 后重发。每次把"等待 + 一次连接时限"计入 wait_spent,保证客户端超时的上界不变。
|
||||
- B:只有最近一次请求可能已送达(最后一个 attempt 的 sent ≠ 0),续跑才扣次数。账本的 readiness 返回 counted;Execution.resume(counted=) 只在 counted 时检查上限、增加计数;pipeline 不再按原始次数预先过滤。
|
||||
- C:reconcile 自动放弃时(resumable=True),把这一轮改成 PAUSED / REQUEST_OUTCOME_UNKNOWN,可以续跑并计次数,不再取消执行。人工放弃仍然永久隔离。续跑资格检查忽略已放弃的 attempt。
|
||||
- 测试:test_quota 新增 2 个;test_quota_auto_abandon 改了("自己那一轮永不重放"的旧规定被有意推翻);test_execution 增加不扣次数的续跑。
|
||||
|
||||
**板卡清理没确认(8a14b41,10-14 15:50 已部署)**
|
||||
|
||||
10-12 到 10-14 共 18 条:
|
||||
- 12 条是 07-board 卡满 1 小时、一张回执都没有。同一模型平时 0.2–2.6 分钟就完成,所以是真卡死。
|
||||
- 3 条是重启时主动取消。
|
||||
- 其余是回执 UNKNOWN。
|
||||
|
||||
板上证据(rdk-sandbox-40007,10-14 14:35 那次):
|
||||
- 模型 2 分钟传完,测量程序 14:37:40 启动,BPU 加载后内核日志再无动静;
|
||||
- guard-result 写的是 host_disconnected(容器被时限结束时)。
|
||||
- 板子负载约 6,可能有别人在用。卡住的具体原因:测量进度日志随容器一起丢了,查不到。
|
||||
|
||||
系统问题:board_perf.bench 的时限,本地执行时会收进阶段截止时间,Salty 容器里没有执行状态,固定 3600 秒,比 07-board 的 1 小时阶段时限还长。结果容器先被结束,回执带不回来,只能判 BOARD_CLEANUP_UNCONFIRMED,整条轨迹作废。
|
||||
|
||||
修法:
|
||||
- salty_remote 打包时把阶段截止时刻(墙钟)放进 BENCH_PHASE_DEADLINE;
|
||||
- board_perf 预留 600 秒交回结果,到时作为普通 FAIL 结束,agent 可以重试。
|
||||
- VM 和 Salty 时钟偏差约 1 秒。已核对部署后的打包里带着截止时刻。
|
||||
|
||||
还没做:板上进度停滞检测(例如 10 分钟没有 board_progress 就提前结束)。08-eval 每个样本批次可能好几分钟,窗口不好定,先不做。
|
||||
|
||||
**下次重启要部署的完整清单**(部署前和分支比对 md5):
|
||||
- harness/stageworker.py(f74fe00,hbm 抽查的配置校验)
|
||||
- agent/SYSTEM.md(f74fe00,hbm 抽查说明)
|
||||
- harness/quota.py(e4e1513 DNS 可续跑 + f83fa39 A/B/C)
|
||||
- harness/execution.py、harness/rollout.py、harness/pipeline.py(f83fa39)
|
||||
- 智能体自主性(a92510d,见 [[agent-autonomy-audit-20261014]]):
|
||||
- harness/stageworker.py、terminal.py、salty_dev.py(新文件)、salty_remote_worker.py、stages/s04_calibrate.py;
|
||||
- bin/stageclient.py、agent/SYSTEM.md、agent/VERSION。
|
||||
- **s04 和 worker 不能提前部署**:它们随阶段打包、部署就生效,而 s04 依赖新的 stageworker。
|
||||
- 三份新配置都要加:
|
||||
- salty_remote.stages 加 terminal;
|
||||
- cpu_by_stage.terminal: 8,memory_by_stage.terminal: 32Gi;
|
||||
- terminal.enabled: true;
|
||||
- pools.dev.workers: 16(三份要一致)。
|
||||
- SYSTEM.md 三个批次共用。04 不开的话,Qwen 会读到 bench dev 却用不了,所以要用户定 04 是否一起开。
|
||||
|
||||
**10-14 18:27–19:00 已重启上线(用户定:速度门槛 1.5 倍、04 一起开、马上重启)**
|
||||
- 脚本和清单在 configs/restart-20261014/:
|
||||
- stop.sh:SIGTERM 后最多等 30 分钟,让已在上板、评测的轨迹跑完;随后按新续跑规则用 targets.py 选出要取消的,并行取消;
|
||||
- start.sh:先备份,再部署 26 个文件,补号,用 -v16 配置启动;
|
||||
- restart-offload.sh:等 offload 空闲后用新配置重启它。
|
||||
- 结果:
|
||||
- 18:58 取消 35 条,其中 34 条在跑、1 条暂停但账本已作废;保留 7 条可续跑的暂停轨迹,重启后全部续上(仍是 v15)。等待那 30 分钟里有 3 条自然结束(1 PASS)。
|
||||
- 19:00 三个 collector 起来,1 分钟内开始领题。
|
||||
- 部署后和分支 9b7544a 逐文件比对 md5 一致。只有 scripts/bench_health_alert.sh 有意没部署:它是别的会话的飞书报警脚本。
|
||||
- 新配置:configs/mini/*-v16.yaml,在原配置基础上:
|
||||
- salty_remote.stages 加 terminal(4 核、16Gi);
|
||||
- terminal.enabled: true;
|
||||
- pools.dev.workers: 8。
|
||||
- 踩坑:
|
||||
- stop.sh 第一版用 grep 匹配 /proc/PID/cmdline,参数之间是 \0,匹配不上,安全中止,什么都没做;要先 tr 成空格。
|
||||
- Mac 上没有 timeout 命令。
|
||||
- g.sh 起 setsid 后台进程时 ssh 不会返回,要另开一条命令看日志。
|
||||
- Qwen 组因取消时有一条请求结果不明,进入 RECONCILING,04 要等约 31 分钟(request_total_s 1800 + 60)才领题;GLM 两个组 RECOVERING。
|
||||
|
||||
**重启后头 40 分钟(19:00–19:40)**
|
||||
- 恢复情况:
|
||||
- 05/06 各 16 条、04 有 5 条在跑;保留的暂停轨迹全部续上。新轨迹都是 v16(提示词里有 bench dev 和速度要求)。
|
||||
- 04 等 Qwen 组对账到约 19:31 才领题。
|
||||
- 19:29 S600 组有一条 GLM 回复中途断开(REQUEST_CONNECTION_INTERRUPTED)。整组 RECONCILING 约 20 分钟,05 有 9 条暂停;19:40 按 C 方案自动作废那一条请求,暂停的全部续上。断线的根因仍是 Clash 走洛杉矶。
|
||||
- bench dev 使用:
|
||||
- 40 分钟 91 次:74 次成功,14 次是 agent 自己的命令出错,3 次容器 OOM(vit.h.14、deeplabv3plus 在 16Gi 里跑 hb_compile)。中位 78 秒。
|
||||
- agent 拿它查平台 ONNX 和缓存:VM 上的 bench/cache 几乎是空的,缓存只在个人空间。
|
||||
- Salty 上 bench dev 同时只有 3–8 个,没有排队。
|
||||
- 当场修掉的两处(都不用重启):
|
||||
- 67d306d:offload 的孤儿清理和作业自己的清理撞车,stop 返回 404 后会重试 30 分钟,这期间归档和导出都停着。改为跳过已清理的作业,404 当作已停。offload 已在 19:17 重启。
|
||||
- d565796:bench dev 的内存取本模型正式编译需要量的下限(OOM 过就翻倍),失败时把容器的内存记录附进 stderr。terminal 每次都是新进程,部署后马上生效。
|
||||
- 还没做:
|
||||
- 速度门槛的强制判定:打算放到导出时按参考延迟过滤,不用重启。
|
||||
- 补参考延迟:每题用默认配置编译并上板测一次,要占 Salty 和板卡时间,等用户同意。
|
||||
- 开发环境大文件的清理。
|
||||
|
||||
**Why:** 用户要把这套系统当以后 RSI 的基础设施。原则:能在底层重试的不让上层知道;系统故障不扣模型的次数;结果不明就核对后续跑,不整条作废。
|
||||
**How to apply:** 看到新的"暂停没恢复"报警,先用 resume_readiness 看 counted 和 reason,再判断是不是这几类。
|
||||
|
||||
相关:[[infra-leaks-20261013]] [[hbm-spotcheck-optional-20261014]] [[mac-mini-clash-global-proxy-20261013]] [[bench-restart-hazards]]
|
||||
@ -1,11 +0,0 @@
|
||||
---
|
||||
name: never-edit-running-script
|
||||
description: 不要原地改正在运行的 shell 脚本;要改就写新文件或"新文件+改名替换"
|
||||
metadata:
|
||||
type: feedback
|
||||
---
|
||||
|
||||
不要原地修改正在执行的 bash 脚本(open(p,"w") / sed -i 都会改同一个 inode)。bash 边读边执行,运行中的进程会在下一段按旧字节偏移去读已变化的内容,行为不可预测。
|
||||
|
||||
**Why:** 2026-10-09 我在 120 上原地改了正在跑的 `/root/salty-migrate-20261009/migrate.sh` 的 up(),"salty迁移"会话指出后只能结束主 bash,改用新脚本 phase2.sh / verify.sh 接手,我的改动也就作废了。
|
||||
**How to apply:** 脚本已在运行时,改动写进新文件(或写好后 `mv` 替换,旧进程继续持有旧 inode),并明确下一轮用哪个文件。相关:[[salty-personal-migration-20261009]]
|
||||
@ -1,84 +0,0 @@
|
||||
---
|
||||
name: overnight-check-20261015
|
||||
description: 10-15 早上复查:dev 槽 8→37 已改;S600 板盘满(失败会话目录留板不删);暂停名额被新题抢;全 fp16 落 CPU 仍判过;4 条自带图/校准提交
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: cadbef09-8744-472b-81b1-a1ff37b0a735
|
||||
modified: 2026-10-15T01:41:51.665Z
|
||||
---
|
||||
|
||||
10-15 07:50–08:10 复查(v16 上线约 13 小时,50 条 v16 轨迹)。
|
||||
|
||||
- dev 槽 8→37:三份 -v16 配置一起改,07:50 生效,备份在 config-backup-20261014/before-restart-20261014/*.before-dev37。
|
||||
- paths.config() 有 lru_cache(09-22 合并带进来的),每个进程只读一次配置。改 workers 只对新起的进程生效;已在排队的还按旧槽数等。scheduler.acquire 里「每轮重读 config」的注释已过时。
|
||||
- 之前 8 槽被 agent 自设 3–12 小时的长编译占住,一夜 199 条 dev 命令排到超时(STAGE_EXECUTOR_EXIT),04–05 点比成功的还多。
|
||||
- 底层问题:
|
||||
- S600 板 /userdata 两次写满(01:31–03:25、06:50–06:56),05 的 07-board/08-eval 报「No space left on device」却记成 unknown,像 agent 的错。都自己恢复了。
|
||||
- 根因:board_perf.py 的 close() 在失败时把板上会话目录「留作诊断」,从不删。日志里 10-13 起留了 9 个(S600 上 3 个:dedode.v2、gmflow、yolo11m.pose)。
|
||||
- 再加上 solo.r50 一次评测输出 12 GB。
|
||||
- 板卡密钥只在 Salty 容器里,主控上看不了板子剩余空间。
|
||||
- 已修(a263374,10-15 09:41 热部署,阶段容器每次打包现读,不用重启):
|
||||
- 清理已确认的失败会话照样删;
|
||||
- 清理没确认的只留空目录和回执;
|
||||
- 每个新会话先删 6 小时没人动的 eval-* 并在阶段 stderr 记「removed stale board sessions」;
|
||||
- No space left on device → disk_full(瞬态)。
|
||||
- 不能直接删「活」目录:board_guard 的 finally 先往会话目录写 guard-result.json,再更新 /userdata/bench/.board-inference.quarantine.json。目录没了就抛异常,标记卡在 PENDING,这块板以后的推理都会被拦。test_failed_guard_retains_evidence_without_erasing_live_directory 守的就是这个。
|
||||
- 上线后核实(09:41–10:40):板上测试和评测全部 OK,没有磁盘满。3 块板的清扫共删了 31 个旧目录(S100P rdk-sandbox 18 个,两块 S600 各 10 和 3 个),比日志里看到的 9 个多,多出来的应该是容器被杀留下的。
|
||||
- reward.py 只对 board_offline 免罚,disk_full 还没加;这是判分改动,跟导出那批一起等用户。
|
||||
- 板上评测分批(f56ac2f)11:08 热部署后出事,11:45:38 回退(分支上 2b02417 撤销):
|
||||
- harness/eval/<任务>.py 的 sha256 在建 run 时冻进 job.task_contract,每个阶段开头都核对(stages/base.py:261)。
|
||||
- 改了 seg/mde/flow/feat/coco_extra.py 之后,这些任务的所有阶段都报 frozen task implementation changed。
|
||||
- 板卡阶段在派发前失败、没有回执,会被判 BOARD_CLEANUP_UNCONFIRMED,整条轨迹作废。
|
||||
- 损失三条:04 solov2.r50、05 deeplabv3plus.resnet50(自带图那条)、06 yolov8m.seg(阶段全失败后判 FAIL)。另有 3 条在跑的白白失败了几次阶段。没有编译失败,不占编译次数。
|
||||
- 规则:任务评测文件不能直接热改;要么建新 run,要么把精确的「旧指纹 → 新指纹」写进 harness/eval/__init__.py 的 EQUIVALENT_SOURCES(只限结果完全等价的改写)。common.py、board.py、run.py、board_perf.py、errors.py、pipeline.py、adapters/ 不在指纹里。
|
||||
- 另一个坑:板卡阶段派发前的任何 harness 异常都会被当成清理未确认,害死整条轨迹,应该写 not_started 回执。还没修。
|
||||
- 断线暂停的位子(10-15 11:32 用户同意换回):
|
||||
- 取消了 11:21 新开的 4 条。deeplabv3plus.resnet101、fcn.resnet50 11:33 续上,deeplabv3plus.resnet50 11:37 续上,efficientvit.seg.b1 还在等。
|
||||
- 11:33 又有 2 个位子被新题抢了。根因是 pipeline.py:813:同一轮里有一条就绪,新题就会跟着进来。
|
||||
- f191baf(_claimable:有暂停在等时这一轮只续跑;额度组 PAUSED/RECONCILING 一律算在等)11:44 已部署,collector 重启后生效。
|
||||
- collector 重启会杀掉它管的在跑轨迹(配额管理在进程里),不能随手重启。
|
||||
- deeplabv3plus.resnet101(05)11:34 PASS:第一条真自带图的通过。
|
||||
- G3 mIoU 32.53→31.80,降 2.24%,门槛 5%;G2 0.986 没过但不是必选;板上 506 ms。
|
||||
- 「暂停名额被抢」又发生:00:56–01:11 GLM 断线,05 停了 12 条。01:17 一起续跑,6 条因为额度还没恢复(RESUME_NOT_READY: WAITING_QUOTA)又被暂停,空出的位子给了 6 道新题。两条 efficientvit.seg.b1/b3 停了 7 小时还在等。修要改 collector、要重启。
|
||||
- 离线归档:06 zoedepth.n kk2 的包 23:00–00:12 一直传不上(网络 body error),跳过了。
|
||||
- Salty 上留着 327 个失败、94 个成功的 stage 作业没清(317 个在重启前)。不占 CPU,可能占个人空间。
|
||||
- collector 无 traceback;VM 负载 15、内存剩 4G、盘 60%。
|
||||
- 自主性:
|
||||
- 4 条编译用了自己的东西,都过了 I/O 检查:
|
||||
- maxvit.t 把 Einsum 改成 MatMul 的图;
|
||||
- deeplabv3plus.r101 把 2560→512 的 1×1 卷积拆成 5 个绕编译器 bug;
|
||||
- vit.h.14 自做校准集(两次)。
|
||||
- 还没有一条走到结论。
|
||||
- 12/50 写了产图脚本,9 条跑通。
|
||||
- 速度门槛只在提示里:05 efficientvit.seg.b0 最终全 float16,板上 4.1 秒一帧、BPU 利用率 0.003%,照样 PASS。导出时按速度过滤还没做。
|
||||
|
||||
- 板卡派发前失败误判(383710c,10-15 12:20 热部署):
|
||||
- base.Stage.run 本地执行完、board 池、非 SKIPPED、回执目录为空时,补一张 not_started(VERIFIED)回执。
|
||||
- 依据:board_perf.bench 在第一条板上命令前就写 UNKNOWN 回执。控制端包装 Salty 的那层拿不到这个证明,不补。
|
||||
- 待办总表(10-15 中午,按生效方式分):
|
||||
- 热部署已完成:板上会话目录清扫、disk_full、dev 不可见评测集、dev 去重复上传、dev 37 槽、派发前误判。
|
||||
- 要重启 collector:暂停名额修复 f191baf(已部署文件)。重启会断在跑轨迹,等计划窗口。
|
||||
- ~~要新建 run:评测分批 f56ac2f~~ 15:55 已不建新 run 上线(6ab088d 逐对批准的等价改写,见 [[restart-20261015]])。
|
||||
- 要新 agent 版本:SYSTEM.md 删「评测数据只读可见」;速度门槛(用户暂缓)。
|
||||
- 外部:Clash 走洛杉矶导致断线(用户侧)。
|
||||
- 遗留债:
|
||||
- salty_remote.py:294 失败作业和目录永不清(327 个);
|
||||
- s04 自带图大于 2 GB 的外部权重不哈希;
|
||||
- paths.config 有 lru_cache,scheduler 注释过时;
|
||||
- zoedepth.n 归档包上传失败;
|
||||
- agent 在长 dev 编译上同步干等几小时;
|
||||
- reward 里 disk_full 没免罚。
|
||||
- 热部署前必查:
|
||||
- 改的文件在不在 task_contract 指纹里(harness/eval/<任务>.py);
|
||||
- 在不在常驻进程里(collector 不重载);
|
||||
- VM 和 Salty 容器各跑一遍相关测试,base 和 new 对比。
|
||||
|
||||
**Why:** 用户要这套当 SFT 数据和以后 RSI 的底座:底层错误不能算到 agent 头上,速度兜底不能当通过。
|
||||
**How to apply:**
|
||||
- 下次重启要带上:
|
||||
- 续跑优先于新认领;
|
||||
- 额度没好时别把位子让出去。
|
||||
- 清 Salty 旧作业要先问用户。板上残留目录由会话进入时的 6 小时清扫处理,用户 10-15 已同意。
|
||||
|
||||
相关:[[agent-autonomy-audit-20261014]] [[net-resilience-board-deadline-20261014]] [[selfheal-and-alert-20261012]]
|
||||
@ -1,15 +0,0 @@
|
||||
---
|
||||
name: plain-language-no-jargon
|
||||
description: 汇报要用大白话,不要用英文行话或比喻性术语(金丝雀、canary 等)
|
||||
metadata:
|
||||
type: feedback
|
||||
---
|
||||
|
||||
用户原话:「你要让我看的话,那你就说点人话,好不好?全是一些黑话,什么金丝雀,什么鬼?你说的那些我完全听不懂。」
|
||||
|
||||
**Why:** 用户是项目负责人,关心的是「能不能开始收集数据了」这类结果,不是实现术语。
|
||||
行话让他无法判断进展,等于汇报无效。同时他对长篇分析而不出结果非常不满。
|
||||
|
||||
**How to apply:** 用「先拿一个试,确认好用了再动其他的」代替「金丝雀」;用「本机硬盘 / 网盘」代替
|
||||
「local disk / JuiceFS」。先给结论和数字,再给理由。能用一张小表说清就别写段落。
|
||||
承认没做好的地方要直接说,不要辩解。参见 [[prefer-subtraction-over-addition]]。
|
||||
@ -1,32 +0,0 @@
|
||||
---
|
||||
name: poweroff-20261016
|
||||
description: 10-16 19:29 Mac mini 断电前停机(取消 36 条、停 Salty 35 个、limactl stop)→ 20:57 开机后补号、原命令/环境拉起 5 个进程、垃圾已清;暂停轨迹跨开机续不了要一起取消;现成脚本和坑
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: c8e679af-e7a3-4129-a0e4-e1821fa37ce1
|
||||
modified: 2026-10-16T13:01:01.011Z
|
||||
---
|
||||
|
||||
10-16 19:29 用户要给 Mac mini 断电;20:50 开机后让我把 VM、进程、任务按原进度拉起并清垃圾。
|
||||
|
||||
**停(19:29–19:31)**:VM 里 /mnt/data/ming01.wang/configs/poweroff-20261016/stop.sh——存 3 个 collector + offload + speed_ref 的 environ/cmdline/日志/cwd → SIGTERM → 用 restart-20261016/targets.py 选出 36 条立即取消(不等上板/评测)→ 34 CANCELLED、2 INFRA_ERROR;04 的 5 条可续跑暂停保留。collector 自己退出、没卡导出。Salty 上 bench-* running/pending 35 个 `salty stop`。`limactl stop bench` 正常关机。
|
||||
|
||||
**起(20:57)**:同目录 start.sh——挪走旧开机的 lease → requeue.py 补号 36 → withenv 按存档原命令拉起 05/06/04/offload/speed_ref(env 自带 BENCH_CONFIG:04 qwen-s100p-r04-v22、05 s600-r05-v22c、06 s100p-r06-v22)。几秒内就开始领题,21:00 是 04 5/5(+5 暂停)、05 16/16、06 16/16。
|
||||
|
||||
**新踩的坑(下次停/起照做)**
|
||||
- Mac mini 开机时 launchd cc.d-robotics.bench-controller.vm(~/work/controller/start-existing-bench-vm.py,只 RunAtLoad)会自动把 VM 拉起,但不起任何进程。
|
||||
- 取消正在跑 dev 命令的轨迹会留下 state/locks/dev-*.lease.json(这次 24 个)。_lease_settled 只认 DONE+VERIFIED,所以不会自动释放、会一直占 dev 槽。重启后要把 holder.boot_id 不是本次开机的 lease 挪走(start.sh 里有)。
|
||||
- speed_ref 被中断的题会留下 runs/zz-speedref/<job>/rollouts/00/stages/ref/salty/bundle.tar,再跑时 FileExistsError;异常路径不写 FAILED,所以删掉 stages/ref 和 _dev-sync.* 就能重算。
|
||||
- offload 一启动就会把 04/05/06 停掉的 Salty 作业连目录一起回收(35 个里回收了 31 个);speed_ref 和导出的作业不归它管,要手工 `salty rm` + `salty storage rm -r personal:/<名字>`(cleanup.sh)。
|
||||
|
||||
**更正(10-16 22:45 体检发现):可续跑的暂停轨迹跨开机续不了**
|
||||
- 断电时保留的 04 两条暂停(dpt.hybrid #5、simplebaseline.r50 #5)开机后被标成 OWNERSHIP_UNKNOWN 隔离,健康检查每 10 分钟报"暂停 N 分钟没恢复"。
|
||||
- 原因:execution.process_status 对不是本次开机的进程一律返回 UNKNOWN(同一主机名也一样),reconcile 就隔离;Execution.resume 遇到开机变了直接拒(BOOT_CHANGED_RECONCILE_REQUIRED),deadline_monotonic 是按开机计时的,跨开机也没意义。
|
||||
- 22:48 用 fix-stuck-04.sh 取消并补号;取消要等清理旧开机的子进程到 cleanup_deadline_s(1800 秒)才返回。
|
||||
- 以后断电/重启 VM:可续跑的暂停也要一起取消(targets.py 里 KEEP 的那些),否则开机后会卡死。代码上可以让"同主机、不同开机"的进程算已死,但要重启 collector 才生效,等下次统一上线。
|
||||
|
||||
**清理(用户要求)**:Salty 剩下 4 个作业和目录、两道 speed_ref 的残留、挪走的 24 个 lease、含密钥的 env-* 存档、jobs.sqlite 备份和临时文件都删了;目录里只留脚本、cmd/log/cwd、名单。
|
||||
|
||||
**Why:** 10-12 用户直接重启没停 VM,37 条 execution.json 写坏;这次先取消再关 VM,文件和两个 sqlite 都完好(quick_check ok)。
|
||||
**How to apply:** 再断电就复制这个目录的 stop.sh/start.sh/cleanup.sh 改 PID 和名单用;见 [[bench-restart-hazards]] [[mini-salty-controller-20261009]]。
|
||||
@ -1,17 +0,0 @@
|
||||
---
|
||||
name: prefer-subtraction-over-addition
|
||||
description: 修复问题时优先删代码/消除不一致,而不是新增扩展点、配置项或抽象层
|
||||
metadata:
|
||||
type: feedback
|
||||
---
|
||||
|
||||
用户明确要求:「不一定只关注于做加法,而且要做减法。其实更主要的是要做减法,我不喜欢就是你来不来就做加法。」
|
||||
同时强调鲁棒性、优雅性、简洁性。
|
||||
|
||||
**Why:** 给一个本不该存在的默认值打补丁(例如新增只有一个子类使用的钩子),是在掩盖问题而不是消除它;
|
||||
代码库会因此越积越重。2026-09-18 修 JuiceFS 编译卡死时,第一版方案是给 `Stage` 加 `run_cwd()` 钩子,
|
||||
被用户否掉;改成删掉 `cwd=stage_dir` 这个从没人复核过的默认值,净 -10 行、0 新增抽象,反而更对。
|
||||
|
||||
**How to apply:** 提方案前先问「能不能靠删掉某个东西来修」。诚实报告增删行数与新增抽象数——用户会核对。
|
||||
把顺带发现的死代码、永真守卫、过时配置一并清掉,但不要让范围膨胀到另一个 bug 上。
|
||||
参见 [[plain-language-no-jargon]]。
|
||||
@ -1,74 +0,0 @@
|
||||
---
|
||||
name: quota-unknown-autorecover-fix
|
||||
description: 请求不明自动恢复(2f54d40)+板卡死锁自动释放(80a8e24);04/05/06 在跑;板卡核对/释放记录位置
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 64b486b6-4970-404f-9b9a-46e9ca4808d3
|
||||
modified: 2026-10-03T14:30:37.255Z
|
||||
---
|
||||
|
||||
2026-10-03 已推送到 gitea `production/s100p-infra-fix-20260928`:`2eb29ca`(现场原有的 recovery 隔离修改)、
|
||||
`2f54d40`(请求结果不明时自动恢复 + 单条任务只隔离、采集器不退出)。
|
||||
服务器上没有 git 身份和推送凭证:提交时用 `-c user.name=ming01.wang -c user.email=ming01.wang@d-robotics.cc`,
|
||||
推送时在服务器上打 bundle,传回 Mac 再推(Mac 有 gitea 凭证)。
|
||||
|
||||
- 06:21:06 用新代码重启,运行正常。
|
||||
- 04/05:21:5x 发了 SIGTERM 排空,由 `/home/ming01.wang/bench-relaunch-20261003/relaunch.sh` 等旧进程退出后在原批次上重启
|
||||
(新日志 `resume-glm-20261002-104649/restored-<run>-fix-20261003.log`)。
|
||||
之后 `bench-relaunch-cleanup.sh` 删除重启脚本目录,结果写入 `/home/ming01.wang/bench-relaunch-20261003.result`。
|
||||
**下次先看这个 .result 文件,确认 04/05 已经重启。**
|
||||
- 04/05 当时实际写日志的位置是 `/home/ming01.wang/sampling-recovery-20261003-1125/{qwen,s600}.log`,
|
||||
而不是 resume-glm 目录里的旧日志。
|
||||
- UniMatch r20260929-06#00:经人工核对(583 个进程全部已死,板端评估回执 VERIFIED),数据库结束为 INFRA_ERROR,
|
||||
释放了 agent-7、cpu-4、cpu-6 三个槽位;核对记录在 rollout 目录的 `operator-release.json`。
|
||||
|
||||
坑:经 ssh 单引号传 heredoc 脚本时,脚本里的引号和中文会出问题,写成文件 scp 过去再执行;
|
||||
`pkill -f`、`pgrep -f` 会匹配到自己所在的 `bash -c`,等进程退出时按 PID 用 `kill -0`。
|
||||
网盘上 `git status --ignored` 慢到不可用。重启要点见 [[bench-restart-hazards]]。
|
||||
|
||||
2026-10-04 00:2x–00:48 进展:按用户要求取消了 04/05 剩余任务。取消后旧采集器卡在退出前的数据导出上,
|
||||
确认"运行中"记录为 0 条后用 kill -9 结束。05 于 00:21 重启。04 缺 `QWEN_EVAL_API_KEY`,用户提供后已写入
|
||||
`.env.local`,00:39 重启。Qwen 账户自动解除暂停,试探请求成功,进入恢复中。
|
||||
并发:06、05 各 8 路(GLM-5.3),04 为 5 路(Qwen3.8-27B),全局 Agent 槽位 24。
|
||||
04 的 RetinaNet-R50 是自己的请求结果不明(INCOMPLETE_GENERATION),按设计永远停在暂停状态。
|
||||
|
||||
2026-10-04 08:00 板卡锁死事故:8 块可用板卡全部被已结束任务占着(占用进程已死、清理回执缺失),
|
||||
从 10-03 23:19 起没有任务能上板,排队满 4 小时报 QUEUE_DEADLINE_EXCEEDED。逐块核对(进程死、任务已结束、板上只有 sshd)后释放,
|
||||
核对记录在各 rollout 目录 `operator-release-board-<slot>.json`。修复 `80a8e24` 已推送:`scheduler.reconcile_leases` 遇到这种情况时
|
||||
会用 ssh 检查板卡,确认空闲后自动释放,并写入 `board-release-<slot>.json`。
|
||||
旧采集器没有重启;临时用 `~/board-reconcile-bridge.sh` 每 10 分钟跑一次,三个旧 PID 都退出后它自行退出并删除自身,日志写到 `~/bench-board-reconcile-bridge.log`。
|
||||
板卡 ssh 时断时连的原因:板端 sshd 达到 MaxStartups 上限("3 of 3-10 startups")。另外 `~/.ssh/known_hosts` 里 S600 的主机指纹已过期,
|
||||
连板卡要用密钥目录下的 known_hosts。
|
||||
|
||||
2026-10-04 09:00 板卡链路三处根因修复(均已推送,上板步骤每次都是新进程,无需重启采集器即生效):
|
||||
- `f315496` 板端 `/userdata/bench/.board-inference.quarantine.json`:guard 被杀后这个标记会永久留在板上,之后每次评估都返回 78
|
||||
(40009 因此被锁 1.5 天)。现在若板上已没有任何进程带着该次评估的标识,guard 会自动清掉它。
|
||||
- `f61d2ef` ssh 在登录前就被断开(kex_exchange_identification)时,`board_perf.run`/`guarded_run` 最多重放两次。
|
||||
- 板端 sshd 的 MaxStartups 改为 32:30:100(09-24 改过 40009/40007,本次补齐 40011/40015/40016 和 BOARD_HOST)。
|
||||
原因是公网端口常年挂着 15–35 个扫描器的未登录连接。每块板的备份和记录在 `/root/bench-ssh-recovery-20261004/`。
|
||||
- 不需要 S600 密码;用 harness 的密钥,加上密钥目录下的 known_hosts 即可。`~/.ssh/known_hosts` 里的旧指纹 harness 不会读取,不要拿它判断板卡状态。
|
||||
- 06 那 3 条暂停(yolov8l.seg、两个 DA-v2-large):阶段其实已结束,是回执校验误判。修复 `44f643d` 已推送:
|
||||
嵌套进程组校验和 owned_processes 统一走 `execution.scope_cleaned`;排队超时、从未拿到板卡的上板阶段不再要求板卡回执。
|
||||
这 3 条用新代码补登记阶段结果后已结算为 CANCELLED(清理确认),释放 agent-12/18/20。
|
||||
注意:隔离状态是单向的,处于"清理未确认"的暂停任务只能结算,不能恢复。
|
||||
|
||||
2026-10-04 10:03 用户要求立即重启让修复全部生效:取消了正在跑的 21 条(结案为 CANCELLED,原因 USER_APPROVED_IMMEDIATE_RESTART_20261004);三个采集器在原批次上重启,06 于 10:12、04 于 10:17、05 于 10:23 重启(运行的是 44f643d),10:38 恢复满并发 5+8+8。过渡用的 bridge 脚本已停止并删除。
|
||||
|
||||
2026-10-04 `4332f43`:编译次数成为每条轨迹的主预算,最多 3 次;`106a96a` 应用户要求改为 5 次(VERSION 14)(`stagerun.MAX_COMPILE_ATTEMPTS`,改它要同步改 e2e.md)。
|
||||
计数来源是 stages 表里 05-compile 中 lane!='queued' 的行,续跑不会清零。第 4 次提交编译直接返回 COMPILE_BUDGET_EXHAUSTED。stagerun 每个阶段都是新进程,所以不用重启就已生效;提示词只对新开始的轨迹生效。
|
||||
历史数据:142 条 PASS 中有 19 条(13%)用了 4 次以上编译;如果上限设成 4,只损失 4 条(约 3%)。
|
||||
提示词模板、SYSTEM.md、VERSION 每次准备新任务时都从磁盘读取,stagerun 每个阶段都是新进程,所以改这几处不用重启采集器;只有改采集器进程里的 Python 代码才需要重启。
|
||||
|
||||
2026-10-08 `ea0335b`:04 的 Qwen 账户从 10-07 08:26 起停在 PAUSED 达 15 小时。原因是自动作废那个结果不明的请求时,账户里另有一个正常请求还在进行,于是 probe_allowed=False,之后再也没人检查。
|
||||
现在作废、确认结束、reconcile 三处共用 `_probe_if_settled`:只要账户里没有未结束的请求,就放行一次试探。
|
||||
00:00 在原批次上重启(取消了 16 条正在跑的任务,04 暂停的 5 条从检查点续跑)。00:15 又出现一次结果不明,账户按新逻辑自动恢复。
|
||||
新采集器启动耗时变长:05 约 25 分钟,06 约 38 分钟。
|
||||
|
||||
2026-10-08 两处系统终止修复(已推送):
|
||||
- `5617f57`:bash 没写超时或超时超过上限时,pi_session.js 在执行前自动设成"系统上限减 60 秒"(约 14 分钟),超时只让这一条命令失败,轨迹继续;落盘的模型调用参数不变。SYSTEM.md 加了说明(VERSION 15)。每条轨迹都会新起 pi 进程,所以不用重启就生效,04 也已生效。
|
||||
- `e2bdd24`:之前说的"GLM 回复不完整"68 条是误判。其中 57 条是本地配额代理返回 413 REQUEST_EXCEEDS_INTERNAL_TPM:代理按字节数估算 token(约是真实值的 3.7 倍),单次请求超过 tpm 就直接拒绝,所以上下文一过约 13.5 万 token 轨迹就死;另外 11 条是重启期间代理不可用。05/06 两个 GLM 组的 tpm 从 50 万调到 200 万(和 Qwen 组一样)。生产配置(/home/ming01.wang/s100p-glm-8way-20260928-2124/s100p.yaml、s600.yaml)、账本 quota_groups.policy 和仓库 config.yaml 默认值都已同步修改。
|
||||
注意:采集器运行期间改配置文件会触发 RUNTIME_CONFIG_CHANGED;账本里存的策略和配置不一致会触发 QUOTA_POLICY_CONFLICT。所以必须先停采集器,再同时改配置和账本。
|
||||
- 07:38 按老规矩只重启了 05/06:取消 16 条(原因 USER_APPROVED_RESTART_20261008_GLM_TPM,其中 3 条已在校验,按实际结果结案)。06 于 07:46、05 于 07:51 在原批次上重启;04 没有动。
|
||||
|
||||
2026-10-08 `10b070e` 改了"连坐"规则(pipeline._collection_snapshot 的 SQL):同一任务里有一条被系统原因终止(命令超时、基础设施错误、取消、丢失、暂停后过期),不再挡住其余排队的轨迹。仍然会挡的只有两种:预算用完(BUDGET_EXHAUSTED),以及 verdict.timeout_reason=EXECUTION_DEADLINE_EXCEEDED(真正干满 24 小时)。正在跑或暂停中的同伴仍然一条一条排。用户确认的分类:系统原因 379 条放开,7 条继续挡。"导出必然失败"这类 agent 自己判 FAIL 的,本来就不挡,用户决定先不加规则。改之前被挡 1107 条(04 267、05 409、06 431),改后放出 1086 条。只重启了 06(09:58,取消 8 条);04/05 等下次重启时生效。重启时有一个取消命令卡在网盘 IO 上 20 分钟,轨迹其实早已结束,kill 掉之后脚本继续。
|
||||
@ -1,29 +0,0 @@
|
||||
---
|
||||
name: qwen-ability-failures-20261013
|
||||
description: 训练前 Qwen 真正因能力做不出的题清单与原因(对照 GLM 怎么做成的);训练前后对比应盯这组题,而不是已通过的题
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 6c848f33-e8d9-4981-ade2-dc77987005d7
|
||||
modified: 2026-10-13T03:09:29.425Z
|
||||
---
|
||||
|
||||
2026-10-13 用户明确:对比训练前后,要看的是**训练前 Qwen 因模型能力没通过的题**,已通过的题看不出提升。
|
||||
|
||||
**10-14 12:30 更新:能力失败共 6 道**(34/37 有结果、过 28;系统失败不计,depth.anything.v1.small 的 FAIL 是 06-hbm 超时,已剔除)。新增三道:
|
||||
- retinanet.r50.fpn(k4,3 次编译):板上 mAP 达标(+0.3%),G1 hb_verifier 余弦 0.66<0.99(P6/P7 头)。想调校准时把 per_channel/node_config/calibration_type 写进 compiler_parameters、自造 calib/quantization 段,stageworker.snapshot_config 白名单拒收(报错只给段名),Qwen 误判"平台不让改校准"。GLM-06 过 1。
|
||||
- simplebaseline.r50(k4,5 次编译):逐步把骨干改 int16,oks_ap 只掉 0.7%,但附加约束 keypoint_mean_shift_px_max=2 实测 2.33。GLM-06 过 1 败 1。
|
||||
- efficientvit.seg.b1(k3,5 次编译):int8 mIoU 5.1,全 int16 24.7(要 ≥29.1);Qwen 归因板端运行时、未验证。GLM-06 过 2,各 2 次编译。
|
||||
共同短板第三条:把自己的问题归到平台上。
|
||||
|
||||
**明确能力失败(verdict FAIL、系统正常)**:
|
||||
- mobilenet.v3.small(04 k1,5 次编译用完,板端 top1 2%):默认 int8 → verifier 0.983;之后尾部 int16 更差、尾部 run_on_cpu 无效且板端输出坏掉;第 4、5 次把 `activation.asymmetric` 等写成 hmct 不认的格式直接崩溃,白烧两次。全程 0 次本地试验。GLM-06 三条全过:k0 本地做了 28 次 hmct/ORT 试验找敏感节点,9 个节点 int16 后过(4 次编译);k1 本地试出全 int16 后 2 次过(全 int16 是验证器漏洞,见 [[verifier-audit-20261012]])。
|
||||
- solov2.r50(04 k0,10-07,1 次编译、mask_ap 掉 7.4% 就交失败;会话没迁过来,看不到为何停)。GLM-06 k0:首编掉 10.1%,本地 9 次试验定位 head/neck 里 GroupNorm 分解链,neck+heads int16 后掉 2.0% 过(2 次编译)。
|
||||
- yolov8n.seg(04 k1,5 次编译,mask_ap 掉 21.6%):Qwen 正确定位到 proto/mask 分支,但 int16 混合更差、run_on_cpu 被 HzSwish 转换挡住。GLM 也从没过过——是老师也做不到的上限题。
|
||||
|
||||
**很可能是能力问题(10-08 前 24 小时到期时精度还差很远,04 旧轨迹)**:YOLO 检测一族 det--yolov8n(掉 42%)、yolov8s(39%)、yolov5s(40%)、yolo11x(38%)、yolo11n(6 次编译仍掉 5.6%);flow--sea.raft 崩坏(GLM 也全败)。convnext.base 旧轨迹掉 9.6% 但后来过了,不算。其余 04 的 DEADLINE_EXCEEDED 大多 0–3 次编译、没有精度证据,多半是当时系统慢。
|
||||
|
||||
**能力差距的具体表现**:难题上 GLM 先在自己 workspace 里用 hmct API / onnxruntime 本地试验选参数,再花正式编译;Qwen 直接拿正式编译次数去猜,还会写错配置格式。全体会话里"做过本地试验"的比例两边差不多(Qwen 14/38、GLM 32/97),差别只出现在难题上。
|
||||
|
||||
**Why:** SFT 用 GLM 轨迹,能学到的正是"先本地诊断再编译、写对配置"这类做法;训练后 Qwen 在这组题上通过、或编译次数减少,才说明训练有效。
|
||||
**How to apply:** 训练前后对比报告以这组题为主;老师也做不到的(yolov8n.seg、sea.raft)单列为上限题。用户要扩题组时,把"很可能"那几道 YOLO 检测题给训练前 Qwen 各补跑 2 条确认。相关:[[qwen-sft-benchmark]] [[verifier-audit-20261012]]
|
||||
@ -1,19 +0,0 @@
|
||||
---
|
||||
name: qwen-nvfp4-service-speed-20261012
|
||||
description: 10-12 实测 04 用的 Qwen 服务(114.111.18.115:18080 NVFP4):单请求快、但一次只处理 1 个请求;04 开 5 路时服务只忙 15%,不是瓶颈
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: cbd0a330-0959-4662-b4f7-9df1eaf61d31
|
||||
modified: 2026-10-12T07:42:08.439Z
|
||||
---
|
||||
|
||||
2026-10-12 15:40 实测 04 的 Qwen 服务(`http://114.111.18.115:18080/v1`,模型 qwen3.8-27b-nvfp4,配置 /mnt/data/ming01.wang/configs/mini/qwen-s100p-r04.yaml,额度组 qwen38-nvfp4-selfhost-11411118115-18080,max_concurrency 5):
|
||||
- 单个请求:输出约 96 token/秒;1.5 万 token 的全新输入首字 3.2 秒。前缀缓存命中 96%,04 实际请求平均输入 6.4 万 token、输出 583 token,首字平均 1.9 秒。
|
||||
- 账本(attempts 表)里 04 实际请求:中位 5.6–6 秒、p90 17–22 秒(旧原版服务中位 39 秒,GLM 3.7 秒)。
|
||||
- **服务一次只处理 1 个请求**:同时发 4/8/16 个,总速度都停在约 65 token/秒,首字等待线性涨到 15/28/48 秒(最慢 88 秒);/metrics 里 running 始终 ≤1,排队原因是 capacity。显存够同时跑约 10 个满长度请求,所以多半是启动参数 `--max-num-seqs 1`。我没有这台机器的登录权限,要改得找服务负责人。
|
||||
- 但现在它不是瓶颈:04 开 5 路,最近 6 小时 383 个请求,服务只忙 15%;每条轨迹花在等模型上的时间约 3–5%,其余是编译/HBM/上板等工具。
|
||||
- 10-08 18:00 到 10-12 15:35,37 题基准共开 44 条,有效结果只有 12 条(10 PASS、2 FAIL),27 条被取消/作废/系统错误(主要是 Mac mini 重启和换代),5 条在跑。
|
||||
|
||||
**Why:** 用户以为 04 "一题一题做"、服务太慢。实际是 5 路并行,慢在单条轨迹要几小时、加上大量轨迹被系统打断。
|
||||
**How to apply:** 04 想提速,先在全局上限 40 内把 04 提到 8 路(16+16+8,不用动 05/06);超过约 15 路、或服务忙碌超过一半时,才需要让服务负责人调大并发。用户可以自己挑题,重启 04 时换 `--job` 清单即可(configs/restart-after-drain-20261011.sh)。相关:[[qwen-sft-benchmark]] [[mini-salty-controller-20261009]] [[bench-restart-hazards]]
|
||||
@ -1,39 +0,0 @@
|
||||
---
|
||||
name: qwen-sft-benchmark
|
||||
description: 用来比较原版 Qwen3.8-27B 和 GLM-5.3 SFT 版的 benchmark qwen-sft-v1:37 道题、规则、04 的跑法、训练数据污染情况
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 61038635-ef70-484b-8f19-363102d54e44
|
||||
modified: 2026-10-12T12:07:05.163Z
|
||||
---
|
||||
|
||||
2026-10-08 建立 benchmark qwen-sft-v1,题单冻结在 `bench/benchmarks/qwen-sft-v1.json`(提交 c2c52ed)。
|
||||
- 共 37 道 S100P 题:
|
||||
- 对照 4 道:GLM 一次编译就过。
|
||||
- 中等 26 道:GLM 要编译 2–9 次才通过。
|
||||
- 难题探针 7 道:姿态 mspn.r50;深度 depth.anything.v1.small、v2.small、dpt.hybrid;实例分割 yolov8n.seg、yolo11m.seg;语义分割 fast.scnn。
|
||||
- 规则:每道题 2 条;只计 2026-10-08 07:30 之后开始的轨迹;被系统打断的不算模型失败,要补跑。指标分档报:通过率、编译次数、轮次、工具报错率、扣掉等模型回复后的用时。
|
||||
- 原版 Qwen 的跑法:04 现在用的配置是 `/home/ming01.wang/raise-concurrency-20261008/qwen-s100p-r04.yaml`(全局上限 40)。10-08 18:0x 用它重启,加 37 个 `--job` 只跑这些题。Qwen 配额组的三个超时放宽为 first_byte 600、stream_idle 300、request_total 1800,新旧两份配置和账本策略三处都已改成一样。17:50 那次我误用了旧配置路径重启,18:0x 已纠正,期间没丢任务。
|
||||
- SFT 版:同一套系统、同样的题,只换模型地址,单独开一个批次。Qwen 服务现在部署的是原版(`/data/qwen38-27b/models/Qwen3.8-27B`,max_model_len 262144)。
|
||||
- 训练数据污染:`datasets/*/sft.jsonl` 里,37 道题大多已有 GLM 轨迹。9-29 之前的批次就包含 solov2、yolo11l.seg、fsaf、yolo11n、yolox.l、mobilenet.v3.large、regnet.y.400mf/1.6gf、vit.b.32;retinanet.r50.fpn 哪份都没有。导出文件里还有 42 条 Qwen 自己的轨迹也被标成了可训练。
|
||||
- 跟用户约定:旧的 Qwen 结果不删,只是不计入;"Qwen 一遍就能过"的文档已写成 Claude Doc。
|
||||
|
||||
**Why:** 用户用 GLM-5.3 的轨迹给 Qwen 做 SFT。在说明书写得很细的简单题上,两者同题打平,必须专门挑题才能看出微调前后的差别。
|
||||
**How to apply:**
|
||||
- 等用户告诉我训练实际用了哪份数据,再把题分成"训练里见过"和"没见过"两部分。
|
||||
- 原版和 SFT 版必须用同一个说明书版本(VERSION 15)在相近时间跑;系统再改动就要重测原版。
|
||||
- 10-08 17:00 有另一方用 "raise-concurrency-20261008" 把三个采集器都重启过,05/06 并发改成了 16。重启前先确认没有别人同时在操作。
|
||||
|
||||
相关:[[success-rate-breakdown-20261008]] [[quota-unknown-autorecover-fix]] [[bench-restart-hazards]]
|
||||
|
||||
**更正(2026-10-12):04 的 Qwen 轨迹只做训练前基准,不进 SFT。** 用户明确说过:Qwen 产生的数据不是用来做 SFT 的。衡量 Qwen 只看最终判定(PASS / FAIL),不要拿内容审核打回率或"进 SFT 的条数"当 Qwen 的指标。datasets-final/r20260929-04-training 里那 32 条 Qwen 轨迹不要混进训练数据。训练后的 Qwen 要用同样的部署方式(10-10 起 04 连的是 NVFP4 服务,配置备份 qwen-s100p-r04.before-nvfp4.yaml)、同样的 37 道题来跑,前后才可比。用户不想细分 Qwen 原版和 NVFP4,汇报时合在一起算。
|
||||
|
||||
**10-13 10:50 进度**:昨晚新过 4 题(depth.anything.v2.small 难题、yolo11l.seg、simplebaseline.r152、pidnet.s,各编译 1–2 次),无新失败;合计 27 题有结果、过 24 败 3,10 题没结果。系统问题(不算 Qwen):① 06-hbm 阶段时限 2 小时(config lifecycle.phases.06-hbm.execution_deadline_s=7200),depth.anything.v1.small 的 06-hbm 每次超 2 小时被杀,同一轨迹连杀 6 次;dpt.hybrid 超时后板端也超时→INFRA_ERROR;v2.small 用了 83 分钟刚好过。时限存在每条轨迹的 execution.json 里,改配置只影响新轨迹,config 每次现读、不用重启。② 10-13 10:14 网络断一下,efficientvit.seg.b1 停在 REQUEST_CONNECTION_INTERRUPTED:请求已发出、结果不明的不自动重发(quota.py 设计),会停到 24 小时过期。
|
||||
|
||||
**规则更正(10-12 20:10 用户决定):04 在 10-08 07:30 之前的 PASS/FAIL 也算。** 题单文件 count_rollouts_started_after 已改成 2026-09-29(提交 1e0b849,VM 上已同步,没有代码读它)。按新口径训练前 Qwen:23 题有结果、过 20 败 3(对照 3/3、中等 14/16、难题 3/4);败的三道都是真失败:mobilenet.v3.small、yolov8n.seg、solov2(mask_ap 掉 7.4%,只编译 1 次就收手)。两边都过的 20 题,编译平均 2.2 vs GLM-06 3.0,轮次中位 58 vs 61.5。这样调度"有旧结果的排后面"就和规则一致了,只剩 14 题没结果。下面 20:00 那段是旧口径。
|
||||
|
||||
**10-12 20:00 进度(训练后模型还没接,只有训练前基准)**:按规则(10-08 07:30 后)13 题有结果,过 11 败 2(对照 2/2、中等 7/8、难题 2/3);同题 11 道对比 GLM-06,轮次中位 56 vs 57、编译平均 2.4 vs 3.0,基本打平。10-08 前 04 还有 10 题的旧结果(过 9:densenet121、两道 regnet、fsaf、rtmdet、三道 yolox、yolo11m.seg;败 1:solov2),不计入但可参考,合起来 23 题过 20。从没出过结果的 14 题,5 条在跑(三道深度估计、yolo11l.seg、simplebaseline.r152,17:16/18:25 起)。**调度按"已完成次数"排队且算上 10-08 前的旧结果**(pipeline.py `_todo` 排序 `completed>0` 优先级最低),所以有旧结果的 9 题会排到最后才重跑。Qwen 的 tokens 计数口径和 GLM 不同(大一两个数量级),别拿来比。
|
||||
- 能拉开训练前后差距的只剩约 8 题:mobilenet.v3.small、yolov8n.seg、solov2、三道深度估计、两道 simplebaseline;GLM 自己也没过过 depth.anything.v1.small、dpt.hybrid、yolov8n.seg。其余题训练前已接近满分,只能比编译次数和轮次。
|
||||
|
||||
**10-12 15:40 进度**:37 题里只有 12 题有有效结果(对照 1/4、中等 8/26、难题 3/7),Qwen 过 10、败 2;25 题一条有效结果都没有(被重启/取消/作废打断),5 条在跑(都是中等题)。有结果、GLM 也做过的 11 题:Qwen 过 10,GLM 全过;同题编译次数两边差不多(2–5 次)。Qwen 的 2 个失败都是真失败,5 次编译用完仍没达标:mobilenet.v3.small(输出相似度 0.983,门槛 0.99;GLM 06 三条全过,各编译 2–4 次)、yolov8n.seg(mask_ap 掉 21.6%,容差 5%;GLM 从没跑完过)。最可能拉开差距的 3 道深度估计和 2 道 simplebaseline 姿态一条都没跑完,GLM 历史上在深度估计 3 题上也只过过 1 条。编译上限在 harness/stagerun.py 的 MAX_COMPILE_ATTEMPTS=5。
|
||||
@ -1,14 +0,0 @@
|
||||
---
|
||||
name: reply-in-chinese
|
||||
description: "回复用户一律用中文(10-16 用户看到英文回复后要求\"转为中文\")"
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: feedback
|
||||
originSessionId: c8e679af-e7a3-4129-a0e4-e1821fa37ce1
|
||||
modified: 2026-10-16T14:33:35.172Z
|
||||
---
|
||||
|
||||
2026-10-16 我用英文汇报断电停机/重启结果,用户回"转为中文"。
|
||||
|
||||
**Why:** 用户一直用中文沟通,英文汇报读着费劲。
|
||||
**How to apply:** 所有回复(包括汇报、提问、表格)都用中文;命令、路径、代码保持原样。配合 [[plain-language-no-jargon]]。
|
||||
@ -1,77 +0,0 @@
|
||||
---
|
||||
name: restart-20261015
|
||||
description: 10-15 下午重启前清查出的底层 bug(回执超时、暂停吃掉 24 小时、400 永久暂停、账本忙让出名额、评测一次 12 GB)、修法、v17/判分 v0.8,以及重启做法
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: cadbef09-8744-472b-81b1-a1ff37b0a735
|
||||
modified: 2026-10-15T06:39:25.114Z
|
||||
---
|
||||
|
||||
10-15 13:00–14:00 清查 10-14 19:00 重启以来所有结束的轨迹、阶段、dev 命令、collector/offload 日志、额度事件,用户要求"重启前查清所有已暴露的 bug"。
|
||||
|
||||
**新发现并修复(分支 feature/salty-20261009)**
|
||||
- 4c98f9f 板上清理回执读取:原来 15 秒超时、不重试,比 ssh 自己的 20 秒连接超时还短;板上已跑完并回收也会判清理未确认、整条作废。历史 9 次(8 次 TimeoutExpired),10-15 12:44 05 fcn.resnet50 r02。改为 60 秒、最多 3 次。
|
||||
- bf88f78 暂停期间 24 小时照算:10-14 三条 v15 轨迹(06 fcn.resnet101、depth.anything.v1.large、dpt.hybrid)被断线停了 7–16 小时,续上只剩不到 1 小时,编译/评测做到一半被截断。改为 execution.paused_credit:暂停时判超时扣除暂停时长,续跑时补回,累计最多补一个预算(防止永远续不上的暂停轨迹永不结束)。
|
||||
- c2ceff5:
|
||||
- 模型服务返回 400(Qwen 上下文超出,76 万 token)也被当成可续跑暂停,但续跑资格永远不收,挂到 24 小时(04 retinanet.r101)。历史 8 次全是 Qwen。现在 400 直接结束(INFRA_UPSTREAM_REJECTED)。
|
||||
- resume_readiness 在额度库忙时返回 QUOTA_SOURCE_UNAVAILABLE,f191baf 不算"在等",名额会被新题拿走;现在算在等。
|
||||
- b3be5cb 只取 f56ac2f 里 board.py/common.py/run.py(不在 task_contract 指纹里):每次上板按实测字节 ≤1 GiB 分批、G2 余弦按组算。05 solo.r50 G2 一批 32 张 12 GB 撑爆 8Gi 容器、回执带不回来、整条作废。v17 配置另把 08-eval 内存 8Gi→16Gi(coco_extra 一次 16 张仍在冻结文件里)。
|
||||
- 41eb1d2(用户定):agent v17,提示词删「评测数据只读可见」(VM 上 bench/data 为空,dev 只挂校准集);判分 v0.8,板上 disk_full 与 board_offline 一样免罚(export/reward.py,导出时生效)。
|
||||
- v17 配置(用户定):max_turns 240→400,token 上限不变;08-eval 16Gi。configs/mini/*-v17.yaml。
|
||||
|
||||
**查过、不是 bug**
|
||||
- offload 大包上传失败(convnext.large、zoedepth.n 等)后来都重试成功,ARCHIVED.json 在。
|
||||
- 阶段 OOM 后内存自动翻倍(salty_remote._memory,上限 128Gi)在起作用;depth.anything.v1.large 的 06-hbm 到 53 GB 还不够,是模型太大。
|
||||
- dev 的 STAGE_EXECUTOR_EXIT 07:50 改 37 槽后归零;之后的 TERMINAL_DEADLINE_EXCEEDED 是 agent 自设超时。
|
||||
- 07-board stalled 两次:S600 40015/40017 加载模型卡住,按瞬态重试;conditional.detr.r50 是 hbdk 编译挂死/崩溃,模型问题。
|
||||
|
||||
**已知限制,以后做**
|
||||
- Salty 容器里 onnxruntime 线程没限(只限了 OMP/MKL/OPENBLAS),大模型编译常超 4 小时阶段时限(vit.h.14、dab.detr、dpt.hybrid、conditional.detr、solo.r50)。
|
||||
- 板上进度停滞检测(加载卡住要等满 ~49 分钟)。
|
||||
- 分割/深度/光流/特征一次多张上板:各任务文件在指纹里,等新建 run 再上 f56ac2f 其余部分。
|
||||
- personal 空间有 425 个 bench-stage-*、28 个 bench-dev-* 目录没清,删要用户同意。
|
||||
|
||||
**重启做法**:configs/restart-20261015/(stop.sh、targets.py、requeue.py、start.sh、mkconfigs.py、deploy/ 18 个文件,md5 与分支 96d72b2 一致)。
|
||||
- 测试:VM 全量 base/new 对比(72 个文件)+ Salty 容器全量对比(VM 没有 numpy)。只有固定 v0.7 的两处测试不同,已跟到 v0.8。
|
||||
- VM 上跑测试要 `env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY`,否则 test_quota 的本地假上游走代理报 502(36 个假失败)。
|
||||
- test_collection、test_explicit_batch 在 base 上也时过时不过,是老的不稳定测试。
|
||||
- 过程:14:18 SIGTERM;等 3 条上板/评测的跑完(14:35);取消 35 条(34 在跑 + retinanet 的 400 暂停),补号 35;旧 collector 14:37 自己退出(没卡在导出)。14:37:18 用 -v17 起来,几秒内就开始领题(以前要 10–20 分钟)。
|
||||
- 06 的 GLM 组因取消打断一条在途请求进入 RECONCILING,约 21 分钟后自动恢复。
|
||||
- 启动时打印的「已隔离」清单(05 24 条、06 28 条、04 9 条)与上次启动完全相同,都是 10-10/10-12 的旧账。
|
||||
- 14:39 核实:05 16 条、04 5 条新轨迹全是 v17、max_turns 400;部署文件 md5 与分支一致;collector 环境里是 -v17 配置。
|
||||
|
||||
**15:00–16:10 追加(用户:不要新建批次;没进 SFT 的可以清;继续改进),全部热部署、没重启 collector**
|
||||
- 评测分批不建新批次(6ab088d,15:55 上线):
|
||||
- harness/eval/__init__.py 加 EQUIVALENT_SOURCES(精确的「冻结指纹 → 现指纹」对)和 contract_holds;阶段检查(stages/base.py)、基线检查(s02_baseline.validate_baseline)、expand 比对(pipeline._check_frozen_job)都走它。
|
||||
- 只批准 f56ac2f 的 5 个任务文件(seg/mde/flow/feat/coco_extra)。依据:上板逐个样本推理(board_bench.infer_inputs),每张的前后处理和指标不变。
|
||||
- 批次的冻结记录不动;基线缓存按冻结指纹寻址(baseline_identity 用 job 里的合同),照常命中。
|
||||
- 冻结检查在每个阶段新起的 stagerun 进程和容器里跑,不在 collector 里,所以能热部署:先上规则、5 秒后再上任务文件。
|
||||
- 上线后:iseg/pose 阶段都放行,没有 frozen 报错;iseg yolov8n.seg 评测 9 次上板 7.8 分钟。
|
||||
- 漏洞:harness/adapters/*.py 不在指纹里。顶层姿态和 mmdet 实例分割的适配器(topdown_pose_common、topdown_pose_batch8、rtmpose_common、mspn_common、rtmo_common、mmdet_iseg_common、mmdet_solo_common)仍然一张图一次上板:simplebaseline.r152 用了 176 次、37 分钟。要改 7 个适配器的循环,并在容器里做新旧对比,下一步再做。
|
||||
- 板上加载卡住 10 分钟就按 stalled 结束(b7577de,board_perf.LOAD_STALL_S)。
|
||||
- Salty 容器 onnxruntime 限线程(85ae175):
|
||||
- 实测 4 核容器 resnet50:默认 125 线程 320–500 ms,限 4 线程 22 ms;hmct 校准写死 10 线程,约 35–48 ms。
|
||||
- harness/container_site/sitecustomize.py 在 onnxruntime 第一次 import 时把 InferenceSession 限到 OMP_NUM_THREADS。
|
||||
- worker 把这个目录放进阶段和 dev 的 PYTHONPATH。盖掉的镜像 sitecustomize 只装 apport 钩子。
|
||||
- Salty 失败作业不再留(5bd0f1a):阶段失败拷完日志就删;offload.reap_orphans 也删已结束的失败作业。offload 用 configs/ops-20261015/restart-offload3.sh 等空闲时重启(原命令、原 -v16 配置,导出口径不变)。
|
||||
- 一次性清理(用户同意):删 442 个已结束的 bench-stage/bench-dev 作业和目录,外加 7 个空挂目录。只删主控端那次尝试已结束的;名字不是 bench-stage-/bench-dev- 的不动(bench-repro-pending、untar 探针、失败的 export)。脚本 /tmp/saltyclean-amos.py,日志 logs/ops-20261015/saltyclean.log。
|
||||
- 测试:VM 全量(去掉代理变量)和 Salty 全量,最终版 5bd0f1a 对比 a526af7,只多了新测试,其余一致。
|
||||
- 自己踩的坑:两组 Salty 测试共用 /tmp/saltytest-<label> 目录,一组结束时删掉了另一组的,要用不同 label。
|
||||
|
||||
- 19:18 核实(热部署后 3 小时 23 分):
|
||||
- 66 次阶段尝试全部 OK/SKIPPED,冻结检查误拦 0 次(iseg 24、pose 15)。
|
||||
- 一次性清理删了 410 个作业、411 个目录、3 个空挂目录;32 个删除失败,多半是被同时清理的进程先删了。19:18 时 Salty 上只剩 1 个失败的阶段作业、11 个阶段目录,没有再攒。
|
||||
- offload 16:06 已按原命令重启。
|
||||
- 姿态 simplebaseline.r152 三次评测仍是 176 次上板、35–37 分钟(适配器逐张路径)。
|
||||
|
||||
- 19:24 堵 dev 泄露(172a174,热部署):
|
||||
- [[sft-sample-review-20261015]] 发现 agent 能经 bench-cache/adapter-probes/*/private-source 读到 harness/registry 私有副本和参考配置。
|
||||
- dev 原来挂整个 personal:/bench/cache,还能看到 dataset-downloads(带标注原始数据集)和 baseline。
|
||||
- 现在只挂 onnx/weights/sources/venvs;v16/v17 的 dev 命令只用过 sources(7) 和 weights(1)。容器内实测三处都不可见,onnx/hmct 正常。
|
||||
- VM 侧 bench/cache 只有 weights,没有这些副本。
|
||||
|
||||
**Why:** 用户原则:底层错误不能算到 agent 头上,系统故障不扣模型的时间和次数。
|
||||
**How to apply:** 看到 DEADLINE_EXCEEDED / BOARD_CLEANUP_UNCONFIRMED / 长期 PAUSED_INFRA,先查是不是这几类;task_contract 只哈希 harness/eval/<任务>.py。
|
||||
|
||||
相关:[[overnight-check-20261015]] [[bench-restart-hazards]] [[net-resilience-board-deadline-20261014]]
|
||||
@ -1,25 +0,0 @@
|
||||
---
|
||||
name: rl-readiness-20261017
|
||||
description: 10-17 03:30 评估能否当强化学习环境:能用,但瓶颈是慢不是稳;收尾 5 件(速度进奖励、Qwen 省略改成确定性、冻结版本、挑有好有坏的题、Qwen 推理服务扩容并记模型版本),之后不再无边界优化
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: e00ddcbc-d4da-4f81-a8f7-1fd7eeb8e8df
|
||||
modified: 2026-10-16T19:20:54.154Z
|
||||
---
|
||||
|
||||
用户问:想把它做成 RL rollout 环境,是不是还不稳定、要不要放弃优化。我的判断:能用,不建议放弃,也不建议继续无边界优化;做完收尾清单就冻结。
|
||||
|
||||
**依据(10-17 03:30 数据)**
|
||||
- 稳定:20:57 重启后 6 小时健康检查全 ok,正式步骤 101 次失败 3 次,大模型请求 4000+ 次断 1 次;不算取消,系统出错占已结束轨迹 10-13 约 37%,10-14、10-15 约 13–16%,10-16 约 27%(含 Qwen 上下文超长和断电)。系统原因结束的轨迹都单独标记;导出时只有 PASS/FAIL 进 reward_learning(to_agentrl.py:823),所以不稳只伤吞吐、不污染奖励。
|
||||
- 慢:10-10 以来 agent 自己结束的轨迹,通过的中位 3.0h(04)/4.7h(05)/6.4h(06),失败 6–11h,预算用完 13–15h,超时 24h;每天 agent 自己结束的 5–73 条(多数 30–50)。Qwen(要训的模型)只开 5 路,服务一次只处理 1 个请求。
|
||||
- 学习信号:有 ≥2 个结果的题里,结果有好有坏的共 74 道(04 9、05 14、06 32,按 job 合并后 74),中位轨迹 3.8h;其中编译中位 ≤30 分钟的 31 道。全过或全不过的题对组内比较没有信号。注意多是 GLM 和旧规则下的历史。
|
||||
- 奖励 v0.8(export/reward.py):G1 1 分 + G2 1 分 + G3 2 分 + 成本最多扣 1 分 + 超时扣 1 分;速度 0 分;只编译成功就拿 1 分。
|
||||
- context-mask.js 的省略只发生在发送时,会话文件存原文:Qwen 训练数据和模型实际看到的不一致;和续跑必死是同一个根因(见 [[check-20261017-0215]])。
|
||||
|
||||
**收尾清单(都没动,等用户定)**:①速度进判定和奖励(板上实测默认配置作参考)并统一奖励口径;②context-mask 改成只由消息决定;③冻结一个训练专用版本;④从 74 道题起步挑题;⑤Qwen 推理服务支持并发、每条记模型版本。然后小规模试跑 30 题×4–8 条,测一步多久、多少题有信号,再决定在线 RL 还是批量采样+筛选训练。
|
||||
|
||||
**Why:** 用户在决定还要不要继续投入。
|
||||
**How to apply:** 用户再提"再优化一下"时,先对照这份清单,清单外的改动默认不做;冻结后只修会把系统原因算错的 bug。
|
||||
|
||||
相关:[[check-20261017-0215]] [[stability-sweep-20261016]] [[verifier-audit-20261012]]
|
||||
@ -1,20 +0,0 @@
|
||||
---
|
||||
name: rollout-time-breakdown-20261008
|
||||
description: "10-08 实测:轨迹时间花在哪、编译机 CPU 余量、04 时长重复计算——判断\"编译上云\"值不值的依据"
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 1b3cd916-3faa-4ef7-be14-c3fb07244493
|
||||
modified: 2026-10-08T05:21:23.910Z
|
||||
---
|
||||
|
||||
2026-10-08 13:20 在 model-compile(120.48.151.164,244 核 / 1.4 TB)上实测,04/05/06 三个批次,608 条已结束轨迹:
|
||||
|
||||
- 轨迹总时长按阶段:05-compile 33%,06-hbm 16%,08-eval 7%,07-board 1%,其余约 43% 是 agent 思考和杂项。
|
||||
- **stages 表里 04-calibrate 的时长是重复计算**:受保护模式下 04 的结束时间跟着 05 一起登记,1270 对里有 1085 对两者结束时间相差不到 2 分钟。看板上 04 占 34% 是假象。
|
||||
- 05 拿到 cpu 槽位后的编译时长:中位 29 分钟,p90 102 分钟(OMP_NUM_THREADS=8)。
|
||||
- 从提交到真正开始编译(04 合并配置 + 排队):中位 0.5 分钟,p90 18 分钟。05 从没出现过排队超时。
|
||||
- 当时机器 CPU 占用约 31%(空闲 67%,负载 82–89),cpu 槽位 13/24,板卡占用 0。
|
||||
|
||||
**Why:** 用户 10-08 提出把 05-compile 搬到 Salty 集群(他人写的方案,要求 Salty 作业生命周期、冻结输入、epoch 对账等)。数据显示目前编译机既不缺 CPU 也不排队,搬上云不会提高吞吐,单次编译反而多了传输和拉镜像的时间。
|
||||
**How to apply:** 再讨论扩并发或编译上云时,先重新采这几项数;只有目标并发大约超过当前的两倍、编译机 CPU 接近打满时,上云才有收益。相关:[[success-rate-breakdown-20261008]] [[prefer-subtraction-over-addition]]
|
||||
@ -1,36 +0,0 @@
|
||||
---
|
||||
name: salty-container-threads-20261013
|
||||
description: Salty 容器看到整机 244 核、只给 4–6 核,torch/numpy/onnxruntime 按 244 开线程被限流,评测慢 25 倍;10-13 已加线程上限,onnxruntime 还没处理
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 977b3506-bd83-4ae1-9791-f48a3caf1e3d
|
||||
modified: 2026-10-13T02:44:48.465Z
|
||||
---
|
||||
|
||||
2026-10-12 晚发现,同一个模型在 Salty 容器里有几步突然变得很慢。
|
||||
- 评测:rtmdet 6 分钟变成 160–173 分钟。rtmdet.l 那次,上板推理前三批几分钟就跑完,之后容器在本地计算那一步停了 107 分钟。
|
||||
- 生成 HBM:yolo11s 5 分钟变成 36 分钟,efficientvit.seg 15 分钟变成 80 分钟。
|
||||
- 编译:多数慢 1–1.5 倍,个别慢 2.5 倍。
|
||||
|
||||
stage 的 meta.duration_s 是容器自己计的时,不含排队,所以慢在容器里面。
|
||||
|
||||
**原因(测试容器实测,4 核、8G、cpu-shared 队列)**:
|
||||
- 容器里 os.cpu_count() 和亲和性都是 244(整机),cgroup cpu.max 是 400000/100000,也就是只给 4 核。整机负载约 73,机器本身不忙。
|
||||
- torch 默认开 122 线程,几乎每个 100ms 周期都被限流。1536×1536 矩阵乘每秒 2.6 次,set_num_threads(4) 后每秒 42 次,快 16 倍。
|
||||
- numpy:默认每秒 31 次,设 OMP/MKL/OPENBLAS_NUM_THREADS=4 后每秒 146–158 次,快 5 倍。
|
||||
- onnxruntime 1.19:默认开约 185 个线程。它不认 OMP_NUM_THREADS 这些环境变量,也不认进程亲和性。只有在 SessionOptions 里写 intra_op_num_threads=4 才快 2.2 倍(每秒约 300 次变成约 680 次)。
|
||||
|
||||
**已做(a7c454c,10-13 10:43 部署)**:salty_remote._submit 提交容器时加 `-e OMP_NUM_THREADS/MKL_NUM_THREADS/OPENBLAS_NUM_THREADS=分到的核数`。每个阶段都是新起的进程,部署后马上生效,不用重启调度程序。已核对:10:43 以后提交的新容器 describe 里带着这三个变量。salty_offload.py 常驻进程还是旧代码,它起的导出作业要等它重启才带上。
|
||||
|
||||
**没做**:onnxruntime 的线程数。我们自己的代码里只有 harness/tools/recipe_model.py 一处用它。主要用户是厂商工具链:oe-3.7.0 里 hb_compile 校准时用原始模型推理。工具链只在 personal:/bench-runtime 里,虚拟机里看不到源码。要处理,得在容器启动时加一个自动加载的脚本(sitecustomize),把 InferenceSession 默认的 intra_op_num_threads 设成核数,或者找厂商的参数。
|
||||
|
||||
**Why:** 用户发现迁到 Salty 后出结果极慢,怀疑系统有问题。这是一个跟题难不难无关、实打实的系统问题。
|
||||
**How to apply:**
|
||||
- 比较部署前后各阶段用时,要按同一个模型配对比。
|
||||
- 以后任何在容器里跑的程序,线程数都按 cgroup 的 cpu.max 定,别按 cpu_count。
|
||||
- 测试容器的写法:`salty run <镜像> --cpu 4 --memory 8Gi -- /bin/sh -c "ln -sfn /data/personal/bench-runtime/home/{toolchains,outputs} ... && exec <venv>/bin/python <脚本>"`,脚本先 storage cp 到个人空间。用完 salty rm 并 storage rm -r。
|
||||
|
||||
另:10-12 晚 Salty 共享队列一度排队 16 个作业,最久 45 分钟,我们最多同时跑约 22 个作业、116 核;10-13 上午不再排队。注意 usage.log 里的时间是北京时间,salty describe 的创建时间是 UTC。
|
||||
|
||||
相关:[[salty-throughput-drop-20261012]] [[selfheal-and-alert-20261012]] [[mini-salty-controller-20261009]]
|
||||
@ -1,38 +0,0 @@
|
||||
---
|
||||
name: salty-personal-migration-20261009
|
||||
description: 10-09 起把老机器 bench-cache/bench-data 传进 Salty 个人空间 personal:/bench/;脚本位置、速度、打包规则、Salty 存储的限制
|
||||
metadata:
|
||||
type: project
|
||||
---
|
||||
|
||||
用户 10-09 定:主控用 Mac mini(Lima `bench` arm64 VM,另一会话在搭,见 outputs/mac-mini-controller-20261008),编译用 Salty,板子验证;任务数据放 **Salty 个人空间**。
|
||||
- 目标:`personal:/bench/cache`、`personal:/bench/data`(容器内 `/data/personal/bench/...`);带软链接或小文件极多的 `pool`、`venvs`、`sources`、`adapter-probes` 打成 8G 切片的 tar 放 `personal:/bench/_tars/`(附 .sha256),以后要在 Salty 作业里解开。`bench-data/bench-data` 是自指软链接,未传。
|
||||
- 老机器 120 上:salty 0.2.1 装在 `/root/.local/bin`,root 已登录(用户网页批准;迁移完应让用户在网页撤销)。脚本 `/root/salty-migrate-20261009/migrate.sh`(可重跑,done.txt 记完成单元),日志 migrate.log,文件清单 manifest.tsv(100532 条,用于核对)。10-09 16:56 启动,646 个目录单元 + tar 切片。
|
||||
- 实测:单路 12.4 MB/s,6 路合计约 20 MB/s,全量约 460G 需 6–7 小时;单文件最大 12.7G,已设 `--timeout 14400`。
|
||||
- Salty 存储限制:`salty storage cp` 源只能是本地路径或 personal:/shared:/;服务端后台复制只在 Salty 两空间之间,**不能直接从 JuiceFS 网盘拉**;拒绝传软链接;远端目标结尾带 `/` 报非法;`salty run` 无挂卷选项,容器只挂 /data/personal 与 /data/shared;文档没写个人空间配额。
|
||||
- 跑任务的真正卡点不在上传:Salty 05 编译自带打包输入;01–04/08 需 x86 模型环境,arm64 Mac mini 跑不了,要改成在 Salty 上执行(未开发)。
|
||||
- **坑**:`salty storage cp 目录 远端` 在远端已存在同名目录时会把源**整个套进去一层**(远端/名字/名字/…)。失败重传前必须先 `salty storage rm -r` 自己的半截目录(migrate.sh 的 up() 已加)。服务端复制(空间之间)实测 1.89G 不到 3 秒,约 700MB/s,只是源必须已在 Salty。
|
||||
- 10-09 17:43 weights/501f… 遇网络错误失败,17:45 有**另一个会话**用 ssh root 登 120 手工补传,造成嵌套;我 17:54 删除重传纠正。动迁移前先确认没别的会话在同时操作。
|
||||
- **10-09 18:0x 起迁移由"salty迁移"会话统一负责**(对方要求,我已同意并停掉自己的后台检查):本会话不要再补传、删除、移动 personal:/bench/ 下文件,也不改 120 上的迁移目录;发现问题发消息给"salty迁移"。
|
||||
|
||||
**Why:** 下次接着核对上传、解 tar、或问"数据在哪/还差多少"时直接用。
|
||||
**How to apply:** 看进度:ssh 120 读 migrate.log 和 done.txt;失败就重跑 migrate.sh。传完用 manifest.tsv 核对后才让用户交回 120。相关:[[compile-server-reclaim]] [[git-mainline-20261009]]
|
||||
|
||||
**10-09 17:43 踩坑**:一个权重单元上传 493MB 文件时网络错误;我整目录补传时远端目录已存在,结果多套一层(`<N>/<N>/`),而且 CLI 报"已上传 2 个文件"但大文件在哪层都列不出来。改为单文件写明目标路径补传后正常。
|
||||
- 补传目录单元:远端目录已存在时只拷内容(源写成 `目录/.`,先 `mkdir -p` 目标),别重跑整个 migrate.sh 原样补;要另写补传脚本(不要原地改正在跑的脚本,bash 边跑边读文件)。
|
||||
- **CLI 报成功不等于在**:传完必须按 manifest.tsv 逐文件核对路径和大小,通过后才让用户交回 120。
|
||||
|
||||
**10-09 18:1x 接管与改道**:当时有两个会话同时在处理这次迁移("数据采集进度"和"salty迁移"),已约定由"salty迁移"会话统一负责。对方 17:55 **原地改了正在运行的 migrate.sh**(同一个 inode),主 bash 跑完第一轮后会按旧偏移读改过的文件,行为不可预测;我只结束了主 bash(pid 948591),第一轮 xargs(949300)继续传。后续由新文件接手:`phase2.sh`(等 949300 退出 → 失败单元逐文件补齐、只传不删 → 上传 tar 分片和 .sha256 → 调 verify.sh),`verify.sh`(逐层 `salty storage ls`,按 manifest.tsv 比对路径和大小,结果写 verify-report.txt,日志里出现"核对结束")。migrate.sh 不再运行。
|
||||
- 以后:正在运行的 bash 脚本绝不原地改;要改就另写新文件。
|
||||
|
||||
**解压待办(10-09 约定)**:"数据采集进度"会话要把 04/05/06 改成 Mac mini 主控 + 每阶段都在 Salty 容器里跑,容器里把 /home/ming01.wang/bench-cache→/data/personal/bench/cache、bench-data→/data/personal/bench/data 软链。它请我在 **verify 通过后** 用 Salty 作业解压 _tars,并在完成后发消息告诉它(它在那之前不让作业读写 personal:/bench/)。
|
||||
- 已备好 120 上的 `/root/salty-migrate-20261009/extract.sh`(未上传、未提交):用法 `extract.sh 包名 目标父目录 顶层名 应有文件数 应有链接数`;先 sha256sum -c,目标已存在就停。镜像用 `registry.d-robotics.cc/algo/bench-runtime@sha256:c530f4ff…`。
|
||||
- 应有数:data-pool 62913 文件/0 链接 → /data/personal/bench/data/pool;cache-venvs 11959/5、cache-sources 7132/1 → /data/personal/bench/cache/;cache-adapter-probes 6916/16 最后单独解。_tars 先不删,等用户定。
|
||||
- 对方另在 personal:/bench-runtime/ 上传三套环境 tar(约 22G,用同一个 120 登录),与我无关。
|
||||
|
||||
**结果(10-09 23:48 核对 / 10-10 解压)**:直接上传 11589 个文件,缺 4(2 个 .gitkeep 空占位、solo.r50--coco2017 的 2 个 g2_fp32.npz,后者由 02-baseline 用到时重算),大小不符 0、多出 0;18 个 tar 分片和校验文件全部一致。解压脚本在 `personal:/bench/_tools/extract.sh`;10-10 从本机提交两个 Salty 作业:`salty-835ccd4a-…`(venvs、sources、pool)和 `salty-fe327bed-…`(adapter-probes),4 CPU / 8Gi,镜像 bench-runtime@c530f4ff…。解完要发消息告诉"数据采集进度"会话。
|
||||
- **Salty 单文件上传上限**:在 7.3GiB(成功)到 12.7GiB(约 30 秒必失败,报 request or response body error)之间,确切值未知;大文件切到 8GiB 以内(8GiB 分片实测都成功)。solo.r50 两个 g2_fp32.npz 内容相同(sha256 ce06383e…,12727502206 字节,我已在 120 独立核实),由"数据采集进度"会话切 4G 分片传到 personal:/bench-runtime/solo-g2/ 后起作业拼回原位;约定只新建这两个路径、目标存在就停、拼完核对 sha256 和字节数后告诉我。
|
||||
- 10-10 04:4x:venvs/sources/adapter-probes 解完并逐文件核对通过(作业 salty-cb102738…,26029 项全对,合计 91564045377 字节),已通知"数据采集进度";solo.r50 两文件由对方补齐、我核过大小。pool 仍在解(作业一 salty-835ccd4a…,6.3 万小文件很慢),解完用 `personal:/bench/_tools/compare.sh data/pool` 核对后再通知对方。
|
||||
- **/data/personal 上别用 du 判断大小**:目录 st_size 是下层合计(类 CephFS),du -sb 会层层重复计算(venvs 显示 7.7G 实为 0.98G);按 find -printf %s 文件大小相加。工具 compare.sh + manifest-packed.tsv 在 personal:/bench/_tools/。
|
||||
- **10-10 05:1x 迁移完成**:pool 解完(salty-835ccd4a… succeeded)并逐文件核对通过(salty-c8d9e4f9…:62913 项全对,10877004191 字节)。personal:/bench/ 全部就绪:直接上传 11589 个文件只缺 2 个 .gitkeep;打包部分 88942 项全对。已通知"数据采集进度",我不再写 personal:/bench/。待用户:交回 120、网页撤销 120 的 salty 授权、决定 _tars(约 97G)删不删。
|
||||
- 小坑:salty run 回显"作业已创建:salty-…"是中文冒号无空格,脚本里取 ID 要用 `grep -o 'salty-[0-9a-f-]*'`,别按空格切。
|
||||
@ -1,23 +0,0 @@
|
||||
---
|
||||
name: salty-throughput-drop-20261012
|
||||
description: 10-12 清查:迁到 Salty 后通过数掉到 1/20,主因是剩下的全是重题(同模型在 Salty 不更慢);系统问题是整批重启、容器被杀占编译次数、连不上 GLM 永不自动恢复、metric3d 从没跑通、Mac mini Wi-Fi
|
||||
metadata:
|
||||
type: project
|
||||
---
|
||||
|
||||
2026-10-12 18:00 查 jobs.sqlite(04/05/06):
|
||||
- 每天 PASS:10-06 约 37、10-08 68(05 27 + 06 38 + 04 3);迁移后 10-10 4、10-11 13、10-12 3。端到端能跑通(迁移后共 20 条 PASS),但效率约为原来的十分之一。
|
||||
- 整批丢轨迹 114 条:10-09 迁移作废 37,10-10 重启取消 40,10-12 10:06 Mac mini 强制重启作废 37。
|
||||
- 阶段成功率(迁移前 10-05..08 → 迁移后 10-10..12):05-compile 84%→44%(容器被杀 REMOTE_WORKER_FAILED 93、执行器退出 15、超 4 小时 28、排队超时 10);06-hbm 80%→53%;07/08 没变差。
|
||||
- **更正(2026-10-12 16:15):同一个模型在 Salty 上并不更慢**。同题配对:编译时长比值中位 1.04,06-hbm 0.54、08-eval 0.50;容器排队+取回每步约 1.5 分钟。"中位 31→52 分钟"是题变重了。迁移前 06 跑的 104 题里 62 题早已 PASS(多是分类补采样,编译中位 25 分钟);迁移后 98 题 0 题 PASS 过(深度估计/光流/实例分割,编译中位 56 分钟)。05 同理(32/100 → 2/96,34→58 分钟)。每 100 路·小时 PASS:06 17.9→0.8,05 6.7→0.5,主因是题,不是 Salty。
|
||||
- **连不上 GLM 永不自动恢复**:harness/quota.py `_resume_readiness` 只放行 QUOTA_WAIT_BUDGET_EXHAUSTED / REQUEST_OUTCOME_UNKNOWN,CONNECT_FAILED(请求根本没发出去)被判 RESUME_NOT_ELIGIBLE。10-12 05 有 11 条因此停住(11:47–12:41 连续连不上 + 16:03 断网),auto_resume_count 0,不修会在 24h 期限判超时。broker 在调度程序进程里,改了要重启。
|
||||
- 06 的 glm-coding-account-b 只要有一个请求结果不明就整组 RECONCILING,16 路全停,直到本地时限过了自动放弃(10-12 有 5 次;16:03 断网后又停)。
|
||||
- Mac mini 10-12 15:50–16:05 断网 15 分钟(没重启);虚拟机时钟会漂 2–3 分钟。导出正常:datasets-final SFT 32/79/139 14:4x 刷新过,offload 日志只有断网时的网络错误。
|
||||
- 编译次数上限检查在 stagerun 里,每次 `bench stage run` 都是新进程,改 compile_attempts_used 不用重启就生效(之前说要重启是错的)。
|
||||
- **5 次编译上限把系统失败也算进去**:harness/stagerun.py compile_attempts_used 只排除 lane='queued'。10-10 以来 175 条有编译的轨迹里 73 条至少被系统吃掉 1 次机会,30 条 ≥2 次;用满 5 次的 19 条里 13 条含系统失败,被记成模型 FAIL(depth.anything.v2 metric 系列、flowformer、unimatch 等),数据被污染。
|
||||
- metric3d / metric3dv2(05)在任何机器上都没过过 02/03:适配器 harness/adapters/metric3d_common.py 重复导入 torchvision,报 "already a kernel registered ... roi_align",被包成 "Failed to find function"。10-10 以来 02/03 的 59 次失败里 47 次是它,agent 一条轨迹重试导出到 a14。
|
||||
- Salty 作业列表 123 failed / 5 succeeded 是假象:665bbef 起失败作业留着给平台取证,成功的取回后就删了。
|
||||
- Qwen 的 mobilenet.v3.small 失败核实为模型自己的配置错误(node_config 里写了不支持的 activation),不是系统问题。
|
||||
|
||||
**Why:** 用户怀疑系统有问题,确实有。
|
||||
**How to apply:** 修复优先级:①编译上限只数真正跑完的编译(排除 REMOTE_WORKER_FAILED、STAGE_EXECUTOR_EXIT),并把被污染的 FAIL 补号重跑(不用重启);②CONNECT_FAILED 加进可自动恢复原因(要重启);③metric3d 先从 05 清单拿掉;④Mac mini 插网线;⑤重启 Mac mini 前先 `limactl stop bench`。比较迁移前后吞吐时必须按题配对或按题难度分层。相关:[[mini-salty-controller-20261009]] [[bench-restart-hazards]] [[qwen-sft-benchmark]]
|
||||
@ -1,39 +0,0 @@
|
||||
---
|
||||
name: selfheal-and-alert-20261012
|
||||
description: 10-12 修复与自愈:系统失败不占编译次数、连不上可自动续、暂停轨迹不被抢名额(12787a2/8d39d2b);本机每 10 分钟健康检查弹通知;排空重启脚本;metric3d 撤出队列
|
||||
metadata:
|
||||
type: project
|
||||
---
|
||||
|
||||
2026-10-12 用户要求"都修好,出问题能自己修或者提醒我"。改动在 feature/salty-20261009(12787a2、8d39d2b),已部署到 Mac mini:
|
||||
- harness/stagerun.py:容器被杀(REMOTE_WORKER_FAILED)、执行器退出(STAGE_EXECUTOR_EXIT)不计入 5 次编译,每条轨迹最多免 3 次(FREE_INFRA_COMPILES)。stagerun 每次是新进程,部署即生效。提示词 e2e.md 没改(保持 VERSION 15,Qwen 前后可比)。
|
||||
- harness/quota.py `_resume_readiness`:CONNECT_FAILED、REQUEST_DEADLINE_BEFORE_SEND(_not_sent 证明没发出去)允许自动续。
|
||||
- harness/pipeline.py:有暂停轨迹只差额度(resume_pending)时,空出的名额不给新轨迹。之前新轨迹总抢在它就绪前,06 segformer.b5 因此饿了 4 小时。
|
||||
- 后两条在调度程序进程里,要重启才生效。**10-12 17:16 已按用户要求立即重启(不再排空)**:取消在跑和暂停的 46 条(42 CANCELLED、4 INFRA_ERROR,原因 USER_RESTART_20261012,清单 logs/cancel-1012-restart.txt),各补一个新 kk;Salty 上 34 个残留 bench-stage 作业 stop/rm/storage rm(logs/orphans-1012-restart.log,bench-export 没动);新调度程序 05=16、06=16、04=5(pid 95662/95665/95675,日志 logs/collect-*-1012-171614.log)。用户要"在原来的基础上跑",没改题单优先级、没降到 14 路。之前的排空方案(作废):16:43 起 scripts/drain_restart.sh(pid 90713,日志 /mnt/data/ming01.wang/logs/drain-restart-1012.log)在排空 04/05/06:旧进程跑完后按新代码重启,05/06 改 14 路(给训练后 Qwen 批次留 5 路),04 仍 5 路 + qwen-sft-v1-jobs.txt。旧进程无子进程且无校验中轨迹满 10 分钟就 kill -9(卡在导出)。
|
||||
- **坑**:`bench execution resume` 在原主人进程还活着时报 OWNER_NOT_RELEASABLE(execution.py resume),排空期间不能手动接手暂停轨迹,只能等旧进程退出由新进程续。drain_restart.sh 第一版因 `set -u` 下空关联数组报错直接退出,已修;重跑要加 NO_TERM=1,别重复发 SIGTERM。
|
||||
- metric3d / metric3dv2:05/06 排队的 19 条改为 CANCELLED(annotation METRIC3D_ADAPTER_BROKEN_20261012),在跑/暂停的 4 条已 execution cancel。04 里的 16 条 QUEUED 不在 37 题清单、不会被领,没动。修好适配器后按原 job 补新 kk。
|
||||
- 健康检查:scripts/health_check.py(虚拟机里只读,有问题每条一行);本机 ~/.bench-health/bench_health_alert.sh 由 launchd `cc.d-robotics.bench-health`(~/Library/LaunchAgents/)每 10 分钟跑,弹 macOS 通知(同样问题 2 小时提醒一次,恢复也通知),记录 ~/.bench-health/{log,alerts.log}。查:Mac mini 连不上(连续 2 次)、虚拟机没运行、调度程序没在跑、暂停超 1 小时、3 小时没新步骤、1 小时内容器失败 ≥8 或轨迹被中断 ≥5、账号 PAUSED/RECONCILING 超 45 分钟、磁盘 <10G、内存 <1G。脚本顶部 MINI(Mac mini 地址)和 RUNS(批次)要随现场改。
|
||||
- 没做自动修的:Mac mini 重启/断网/调度程序崩溃,只报警(恢复要清 Salty 孤儿作业,不适合无人值守)。飞书机器人(本机 cc-connect-feishu 等)没用来报警:会以用户身份发消息,要用户点头。
|
||||
|
||||
**Why:** 用户不想再出现"暂停没人管、系统失败算模型失败、掉线没人知道"。
|
||||
**How to apply:** 加训练后 Qwen 批次时:新开 run、跑 37 题、5 路,并把 run id 加进 ~/.bench-health/bench_health_alert.sh 的 RUNS(仓库 scripts/ 同步)。相关:[[salty-throughput-drop-20261012]] [[bench-restart-hazards]] [[mini-salty-controller-20261009]]
|
||||
|
||||
10-12 17:00 补充:
|
||||
- 用户说的"小金"=本机 lark-cli 的机器人(飞书应用 cli_aa870f4a38f91bd3,后台名"飞书群答疑助手"),5-27 Codex 会话里用户截图的"小金那条消息"就是它发的。已知可用的发法:`lark-cli im +messages-send --as bot --user-id <open_id>`;用户本人 汪明 open_id ou_cc299d6a3684a6c45d56d91f7127b185。报警脚本 FEISHU_TO 10-12 17:1x 经用户同意已填上并发过确认消息(发送失败会记进 alerts.log)。launchd 环境里 PATH 由脚本自己 export,lark-cli 能用;测试时别用 `source <(...)`(macOS bash 3.2 不支持)。
|
||||
- 排空期间旧调度程序不领新题、也不续跑任何暂停的轨迹(resume 在 pipeline 的 not stopped() 里),暂停轨迹会越积越多——这是用户 17:00 看到 11 项报警的原因。
|
||||
- 排空不会卡死在"核对中":每个等额度的请求都会先调 broker.reconcile()(quota.py 请求循环),不靠调度程序主循环。
|
||||
- 06 的 glm-coding-account-b 10-12 16:03、16:24、16:46 连续三次请求结果不明(INCOMPLETE_GENERATION / REQUEST_CONNECTION_INTERRUPTED,流中途断),每次整组停约 21 分钟(request_total_s 1200 + window 60)。05 同一 GLM 地址表现为 CONNECT_FAILED。根子是 Mac mini Wi-Fi(17:00 又掉线约 1 分钟)。
|
||||
- 用户要让 Codex 值守,交接提示词给的是:读 docs/memory/AGENTS.md 四篇 → 每 30 分钟跑 health_check.py、看 drain-restart-1012.log → 只读排查和补起没起来的调度程序可自己做,取消/杀进程/重启/改配置/删文件/停 Salty 作业先问。
|
||||
|
||||
10-12 22:00 补充(375ce4d,已部署,stagerun 每次新进程,马上生效):
|
||||
- 编译超过阶段时限(PHASE_EXECUTION_DEADLINE_EXCEEDED)、被看门狗判卡死(stalled,一小时没日志没 CPU)也不扣编译次数,和容器被杀、执行器退出合计每条最多免 3 次(stagerun.INFRA_COMPILE_ERRORS)。
|
||||
- depthpro:05 的 kk1–3、06 的 kk3 共 4 条 QUEUED 改成 CANCELLED(annotation DEPTHPRO_EXPORT_BROKEN_20261012)。三个批次里导出原始模型 8 次全失败、从没通过过。04 的 3 条不在 37 题清单里,不会被领到,没动。
|
||||
- 还没修:暂停轨迹的名额只在它"只差额度"时才保留。05 upernet.swin.b 18:01 因 CONNECT_FAILED 暂停时还不能续,名额马上被新轨迹(convnext.large)占走;之后能续了却没空位,干等 3 个多小时。修法:pipeline._resume_snapshot 对还有续跑次数、没被隔离的暂停轨迹一律算 pending。要重启才生效,等下次排空重启。
|
||||
- 姿态题 2 像素规则、校验工具比对不一致(上板精度合格)照旧判 FAIL,等用户定。
|
||||
- 8 块板子 10-12 22:00 逐块直连正常(4–8 秒);17:16 后上板 11 次、评测 12 次全部成功。
|
||||
|
||||
**10-13 补充**:自动续跑每条最多 3 次(lifecycle.max_auto_resumes=3),用完就停到 24 小时过期;05 夜里 02–04 点 GLM S600 账号多次"结果不明",yolov8x.seg/ocrnet/pspnet 三条用完 3 次,报警文字却写"暂停 N 分钟没恢复",有误导。upernet.swin.b 只用了 2 次也没续上,原因未查清。"请求断在半路"(REQUEST_CONNECTION_INTERRUPTED)按设计不自动续。
|
||||
|
||||
**10-13 14:50 Qwen 批次停摆的原因**:10-13 08:19 之后 04 没有新结果,13:42 后连新阶段都没有。链条:① Mac mini 仍走 Wi-Fi(en1,网线口 en0 inactive),12:20–14:25 断了约 5 次,GLM 同时刻也有 CONNECT_FAILED,Qwen 服务端 abort/error/preemption 都是 0 → 是网络不是服务端。② GLM 多数断在发出前(CONNECT_FAILED 能自动续);Qwen 服务一次只处理 1 个请求、每个请求流式输出好几分钟,断在半路(REQUEST_CONNECTION_INTERRUPTED / STREAM_IDLE_TIMEOUT)→ broker 把该 execution 记 cancelled=1、generation ABANDONED_UNKNOWN,永不重发,这条轨迹卡死到 24h。③ 每次断都让 Qwen 组 RECONCILING 约 21 分钟,其他在跑的 Qwen 轨迹跟着暂停。④ pipeline.py 我加的"有待续轨迹时空位不领新题"让剩下 3 个空位也闲着。⑤ 在跑的 2 条:depth v1 卡在 06-hbm 2 小时超时循环,convnext.large 在 06-hbm。
|
||||
|
||||
**更正(10-14)**:上面 10-13 14:50 那段把"步骤等容器"当成 Salty 排队、把续跑失败归到我的空位改动,都不准;真因见 [[infra-leaks-20261013]](名额泄漏、续跑核对旧回执、账本读取 0.5 秒超时),10-13 20:41 已修。活跃配置是 configs/mini/*.yaml(BENCH_CONFIG),不是 bench/config.yaml;06-hbm 时限按执行记录里的 config 看(仍是 7200)。
|
||||
@ -1,58 +0,0 @@
|
||||
---
|
||||
name: sft-autonomy-tally-20261016
|
||||
description: "10-16 17:40 按 collector 统计 PASS/SFT/通过题数,以及\"不只写配置\"的条数:GLM 310 条 PASS 里 71% 只写配置,真正用自己改的图编出成品的只有 4 条(3 条进 SFT);v18+ 还没有通过"
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 94b62e94-8a51-48f6-a116-dccb712c9118
|
||||
modified: 2026-10-16T09:45:04.546Z
|
||||
---
|
||||
|
||||
10-16 17:40 统计(数据来自 DB 和 datasets-final 的 sft/teacher/retained.jsonl 消息记录)。
|
||||
|
||||
| | PASS 条数 | 通过的题 | SFT |
|
||||
|---|---|---|---|
|
||||
| 04 Qwen | 77 | 59 | 54(按约定不进训练) |
|
||||
| 05 GLM S600 | 117 | 92 | 101 |
|
||||
| 06 GLM S100P | 193 | 112 | 164 |
|
||||
|
||||
**按做法分(PASS / SFT)**
|
||||
- 只写配置:
|
||||
- 04:53 / 36
|
||||
- 05:78 / 65
|
||||
- 06:142 / 122
|
||||
- GLM 合计约 71%。
|
||||
- 做过本地试验(输入仍用平台的):
|
||||
- 04:22 / 17
|
||||
- 05:36 / 33
|
||||
- 06:47 / 39
|
||||
- 大多是 v12/v14(旧编译机上有工具);v15 迁到 VM 后几乎没有;v16 起靠 dev 恢复。
|
||||
- 写过并运行改图或导图代码:
|
||||
- 04:2 / 1
|
||||
- 05:3 / 3
|
||||
- 06:4 / 3
|
||||
- **正式编译的成品真用了自己的图或校准(逐条核对编译配置):4 条,3 条进 SFT**:
|
||||
- 05 deeplabv3plus.resnet101 k2:自己改的图 + 自做的 32 张校准;
|
||||
- 05 fcn.resnet101 k2:拆卷积的图;
|
||||
- 06 deeplabv3plus.resnet50 k3:拆卷积的图;
|
||||
- 06 fcn.resnet50 k3:拆卷积的图,没进 SFT。
|
||||
- 05 yolo11x(v12)虽然写了改图代码,但有绕守卫、在预算外本地编译的问题,见 [[sft-sample-review-20261015]]。
|
||||
|
||||
**v18+(平台不给图、必须自己导)**:还没有一条 PASS。数据集里有 4 条 Qwen FAIL、1 条 UNVERIFIED,37 条在跑或有判定。
|
||||
|
||||
**v16–v17(10-14 19:00 至 10-15 21:43:有 dev 和速度要求,可自带图,平台仍给图)**
|
||||
- GLM 共 11 条 PASS(05 有 7 条,06 有 4 条),7 条进 SFT。其中 3 条是成品用自己改的图编的。
|
||||
- Qwen 共 8 条 PASS,5 条进 SFT。
|
||||
- 没进 SFT 的通过轨迹,原因都是内容审查 fail 或 unknown,也就是报告和实测结果对不上,或者还没审。
|
||||
- 用户说"新规则"时要先问清是 v16–17 还是 v18+。
|
||||
|
||||
**口径坑**:
|
||||
- 按消息文本搜 yaml 里的 `onnx_model:`、`cal_data_dir` 会误报,模板文字里就有。要以 stages/_queue/05-compile-*/config.yaml 为准。
|
||||
- 归档到个人空间的轨迹在 VM 上没有目录,只能看消息。
|
||||
|
||||
**Why:** 用户要知道"不再只是写配置"的轨迹有多少、有几条通过并进了 SFT。
|
||||
**How to apply:**
|
||||
- 下次复查用同一口径比较,看 v18+ 通过后"成品用自己的图"这一档能不能涨起来。
|
||||
- 脚本思路:消息里的工具调用做特征,再用磁盘上的编译配置核对。
|
||||
|
||||
相关:[[check-20261016-1700]] [[sft-sample-review-20261015]] [[v18-own-inputs-20261015]]
|
||||
@ -1,39 +0,0 @@
|
||||
---
|
||||
name: sft-sample-review-20261015
|
||||
description: 10-15 抽最新三条 SFT 细读 + 全量 315 条统计:真通过但决策含量低;两条是 Qwen 自己的(不该进训练);版本混杂、过长、含评测题
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 8ceff15d-eec5-43fd-994d-f4fd6b293526
|
||||
modified: 2026-10-15T07:06:27.533Z
|
||||
---
|
||||
|
||||
用户问"最新三条 SFT 数据真的有效吗"。最终数据集在 Mac mini 虚拟机 `bench/datasets-final/r20260929-0{4,5,6}-training/sft.jsonl`(本机没有;`ssh -p 443 amos@180.76.192.183` 后 `limactl shell bench`)。10-15 15:00 共 315 条(04=52 全是 Qwen,05=100、06=163 全是 GLM),193 个不同的题,每题最多 5 条。
|
||||
|
||||
**最新三条(按结束时间)**:05 deeplabv3plus.resnet101(GLM,126 轮,编译 3 次,16.6 小时)、04 superpoint(Qwen,59 轮,1 次编译)、04 foveabox.r50.fpn(Qwen,56 轮,1 次编译)。
|
||||
- 都是真通过、没作弊,板端评测来自平台;最后报告基本如实。
|
||||
- foveabox 的 compile.yaml 就是系统提示里的模板原样;superpoint 用平台默认配置。两条都是"一次过",答案在提示词里,决策含量近零。
|
||||
- 只有 deeplabv3plus 有真诊断:首次编译量化崩(余弦 0.16),逐节点定位到 ASPP 后 3×3 conv(2560→512),CPU/int16 对照实验,图手术拆成 5 个 conv,编译 #2 容器内存超限被杀,改 kl+32 张校准后 #3 过。速度结论只用了静态估计(38.1ms vs 50.7ms),板上实测 506ms 没和默认模型对比。
|
||||
- 三条 G2(数值余弦)都没过:0.9894 / 0.958 / 0.9856(只记录、不挡通过)。
|
||||
|
||||
**全量统计**:
|
||||
- 版本混杂:v12=110、v14=102、v15=95、v16 只有 8;v16 才有 bench dev 和速度要求,10-15 14:37 又换 v17。最新三条在 8 条里,不能代表整体。
|
||||
- 上下文长:中位 5.1 万 token,39% 超 6.4 万,8% 超 12.8 万,最长 22.7 万;GLM 轨迹的 usage 把缓存命中单列(input+cacheRead 才是真上下文)。thinking 占大头(GLM deeplab 13.8 万字符)。
|
||||
- 编译次数(粗数):GLM 263 条里 62% 一次、25% ≥3 次;真有迭代的主要在这 1/4。
|
||||
- 全部 G3 都是 pass-screen(100 张),G2 失败 136/315(43%);沿用 [[verifier-audit-20261012]] 的结论。
|
||||
- 工具报错 3/60、7/66、10/125:平台坑(terminal_exec "Unknown protected stage handle"、onnx_model/cal_data_dir 必须在 workspace 内、dev 里 mkdir 不同步)会被学成绕路。
|
||||
|
||||
**训练集问题**:
|
||||
1. 04 的 52 条 Qwen 自己的轨迹被导出器标成可训练,但用户定过 Qwen 数据不进 SFT([[qwen-sft-benchmark]])。训练前要剔掉。
|
||||
2. 37 道基准题里 29 道在 06 里有 GLM 轨迹(43 条),污染早已知,仍存在;训练实际用哪份数据还没告诉我。
|
||||
|
||||
**Why:** 用户想知道这批数据能不能指望它提升 Qwen;Qwen 短板是配置写错、不先本地诊断、归咎平台([[qwen-ability-failures-20261013]]),对应的示范只在少数有迭代的 GLM 轨迹里。
|
||||
**How to apply:** 回答"数据有没有用"时按"有没有真决策点"分,不按条数;建议按(GLM、编译≥2 次、配置偏离模板/默认、有诊断动作)筛子集,统一提示版本,先量上下文长度再定训练窗口;真正验证只能靠 37 题前后对照。未同步到仓库 docs/memory(本机 bench 检出在别的分支且有未提交改动)。
|
||||
|
||||
## 第二轮(10-15 晚):GLM 全量按"整条链路"筛
|
||||
用户要的好数据 = 像人把活交给 Agent、Agent 自己扛整条链路(不是只写 compile.yaml)。263 条 GLM 先用特征打分,读 21 条候选 + 3 条低分对照(6 个子 agent 分读,我核实了私有源码读取、yolo11x 钻空子、solov2/yolox.x 的关键点)。清单在 `outputs/glm-sft-review-20261015/selection.json`。
|
||||
- **没有一条符合最理想的样子**:263 条里 0 条自己导图(导出一直是平台阶段),自带图/校准的也极少;只有 v16 的 5 条带 bench dev。我的"自带图/校准"自动标签有误(rtmdet.l、upernet 其实没有),以子 agent 读原文为准。
|
||||
- 可用(评分≥4)约 10 条,约占 4%:yolox.x、solov2、efficientvit.seg.b1(kk0、kk1)、fcn.resnet101、deeplabv3plus.resnet101、rtmdet.l、efficientnet.b1、foveabox(nash-m kk0/kk1)。其中只有 yolox.x、fcn、deeplab 是"精准修复且速度代价小",其余多靠全 int16 / fp16 落 CPU 兜底(板上 98ms 对静态 9ms、685ms 对 6.5ms),v16 的 1.5 倍速度门槛下会不合格。
|
||||
- **要剔除**:GLM 8 条(v12×7、v15×1)读过 `cache/adapter-probes/*/private-source`(评测器/阶段源码/参考配置),其中 regnet.y.800mf 直接抄了参考配置;平台漏洞:agent 能经缓存目录读到评判代码。另 yolo11x 用 `python -m horizon_tc_ui.hb_compile` 绕守卫并在预算外本地编译约 14 次;mobilenet.v3.small 用评测集 100 张选配置。
|
||||
- 最自主的 v16 轨迹没进 SFT:teacher.jsonl 里 GLM v16 共 27 条,用开发环境最多的 rtmdet.s(66 次)、rtmdet.tiny(59)、efficientvit.seg.b1 nash-p(43)等都 TIMEOUT/FAIL,只有 5 条 PASS 进 SFT。v17(400 轮)下午 14:37 起在跑,好数据更可能出在那里。
|
||||
- 这批 sft.jsonl 是活的:行号会随新增而变,引用请用 id。
|
||||
@ -1,28 +0,0 @@
|
||||
---
|
||||
name: simplify-agent-rules-20261017
|
||||
description: "10-17 核对:把压缩交还 Pi 不是纯减法(台账改写报错并锁住执行、入口不等 Pi 续跑、导出不认压缩、总开关关着);提示词 SYSTEM.md 89 行 + e2e.md 27 行逐条分\"留事实/删规矩/交给系统\""
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: e52f9d49-06fe-4649-af69-fee51d593c88
|
||||
modified: 2026-10-17T08:03:54.758Z
|
||||
---
|
||||
|
||||
用户方向(10-17):严格管结果、不管过程;少给 agent 定规矩,让模型自己想办法。只列清单、没动系统;计划并入冻结训练版本一次上线。
|
||||
|
||||
**Pi 压缩核实(Mac mini 上 Pi 0.73.1 源码 agent-session.js)**
|
||||
- Pi 在一个长任务里只在“上下文超长报错”时压缩(agent_end 后 `_checkCompaction` 的 overflow 分支),压完用 `setTimeout(continue)` 自动续跑;每次成功回复后重置,可多次压缩。按阈值主动压缩只在整个 prompt 结束后才有,长任务里等于没有。
|
||||
- 要改的 5 处:①settings.json.tmpl `compaction.enabled:false`;②quota.py do_POST 把上游 400 换成 503 `CONTEXT_TOO_LONG`,Pi 的 isContextOverflow 不认(要原样回 400 + 上游原文,vLLM "maximum context length is N tokens" 能认);③同一处 `_pause` 把这次请求挂成暂停,锁住整个执行,Pi 的压缩请求和续跑都会 409,要改成直接记 FAILED;④pi_session.js 等完事件队列就存档退出,Pi 排的续跑来不及跑,要等它跑完;⑤export/trajectory.py 只从 message_end 重建消息、不认 compaction 事件,不改的话训练数据还是原文、和模型看到的不一致。删掉 context-mask.js 和 04 配置的 context_mask_chars: 450000。
|
||||
- 风险:Qwen 256k 下压缩请求本身接近窗口,上线前要实测一次;GLM 上下文满以后也会改成压缩(现在算预算用完),所以算新版本。
|
||||
- 结论:不做 10 行临时修复(04 Qwen 只做训练前基准),直接在冻结版本里换。
|
||||
|
||||
**提示词核对(运行中的 agent/VERSION 22)**
|
||||
- 留(事实/验收):任务和必选门、编译 5 次上限、job.json 为准、只认平台测量、控制机没工具链要用 bench dev、单条命令约 14 分钟、大文件只留在开发环境、图的输入输出要和平台一致、阶段命令清单、等待用 wait_handle、校准图片位置、板上 hbm_runtime 用法、如实报告。
|
||||
- 删(规矩/教学/系统已执行):不要 sleep>10 分钟、不要 find /、没有 rg、legacy 沙箱说明、stage wait 别轮询(直接删 stage wait 命令更省)、compile.yaml 旋钮的教学注释(kl/mix 首选、per_channel、挑最差节点提 int16)、速度一段的 speed_ref 机制和“不要自己编默认配置”、harness.tools.export_onnx 的提及(72 条照抄)、hbm_bc_samples 选项、512 MiB/40 子目录长说明(系统拒绝时报原因)、“terminal 命令模式”那段、结束说明里的 G1/标准误细则、e2e.md 和 SYSTEM.md 的重复段。
|
||||
- 交给系统:export 阶段“第一次编译前要跑过”改为编译时自动补;速度进判分后平台自己在板上测默认配置,提示词只留 1.5 倍一句。
|
||||
- 旋钮教学删掉可能让通过率降,要拿新版本数据对比。repair.md 现在批次没用到,不动。
|
||||
|
||||
**10-17 已实现(v23,未提交、未上线)**:工作树 scratchpad/wt(分支 simplify-20261017,基于 origin/feature/salty-20261009 e3697f0)。台账超长记 FAILED 并回 400 context_length_exceeded;入口压缩后编号 +1、等 Pi 续跑完、compaction_end 带 bench_compaction.kept_messages;导出输出 compactions(at/kept/summary_message);删 context-mask.js 和配置透传;models.json contextWindow 改 1000 万(否则暂停退出前 Pi 按 128k 估算先压缩,续跑请求对不上);stage status 与 wait 合一;SYSTEM.md 89→64 行、e2e.md 27→20 行,规矩全删(旋钮字段表保留为事实,文档里 node_config 写法和实际不同)、e2e.md 保留“## 输入和可复用缓存”和“预算:”两个配方改写要用的标记。真实 Pi 测试在 Mac mini /tmp/simplify-test 通过(超长→摘要→新编号重发→完成→导出校验通过)。已知:Pi 拆分轮次时两个摘要请求并发,第二个会被台账 409(只在预算提醒消息落在切点前时发生),失败就按上下文满结束。上线要重启三个采集程序,04 配置里的 context_mask_chars 届时删掉。
|
||||
|
||||
**Why:** 用户认为规则太多限制模型能力,且这几天的 bug 多出在“替 agent 安排流程”的代码上。
|
||||
**How to apply:** 做冻结版本时按这份清单删;以后不再往提示词加规矩,只修把系统故障算到 agent 头上的错误。相关:[[rl-readiness-20261017]] [[check-20261017-0215]] [[v17-autonomy-check-20261015]]
|
||||
@ -1,65 +0,0 @@
|
||||
---
|
||||
name: stability-sweep-20261016
|
||||
description: "10-16 18:00 全面体检(为\"建成稳定系统\"):最大问题是生产批次同时当试验场(3 天 8 个提示版本、每次改动要重启、重启就取消在跑的),近 7 天 40–93% 的轨迹因系统原因结束、约 4000 轨迹小时白费;其余缺口按层列表"
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 94b62e94-8a51-48f6-a116-dccb712c9118
|
||||
modified: 2026-10-16T10:55:46.194Z
|
||||
---
|
||||
|
||||
10-16 18:00 体检时的数字和事实:
|
||||
|
||||
- **系统原因结束的比例**(CANCELLED / INFRA / UNVERIFIED / LOST / ABANDONED),10-09 到 10-16 每天分别是 100%、93%、52%、89%、58%、40%、54%、65%。合计约 3960 轨迹小时,其中取消占大头。
|
||||
- **版本和重启太频繁**:
|
||||
- 10-14 到 10-16 三天里起了 v15 到 v22 共 8 个提示版本;
|
||||
- 10-12 以来,三个 collector 的日志里分别有 5–6 次启动记录;
|
||||
- configs 下有 7 个 restart/ops 目录、14 份 mini 配置、5 个备份目录。
|
||||
- **部署一致**:VM 上 401 个代码文件与分支 HEAD 一致,只有 bench_health_alert.sh 有意没部署。
|
||||
- **没有自动化保障**:
|
||||
- 没有 CI,77 个测试文件靠手跑;
|
||||
- VM 和主机上都没有 cron/timer,监控靠 Codex 会话和人工复查。
|
||||
- **主机**:
|
||||
- 默认路由走 Wi‑Fi(en1),网线口空着;
|
||||
- Clash Verge 在跑,collector 环境里仍有代理变量;
|
||||
- launchd 里 cc.d-robotics.bench-controller.vm 的上次退出码是 75;
|
||||
- 断电后能不能自动拉起 collector 没有验证过;
|
||||
- 密钥以权限 400 的 env 文件存在多个 restart 目录里,共 12 份以上。
|
||||
- **VM 资源**:
|
||||
- 内存 11G、负载 30、盘 81%(剩 16G);
|
||||
- 1.5 小时涨了 4G,其中 3G 是 offload 在 /tmp 打的归档包,上传失败在重试;
|
||||
- /tmp 里还留着 10-13 的 salty-archive 临时目录;
|
||||
- 有一个 1.3G 的 dev snapshot 残留;
|
||||
- quota.sqlite 已到 144MB,还在长。
|
||||
- **48 小时外部依赖**:
|
||||
- GLM 失败率约 0.3%(429 一共 10 次,DNS 失败 15 次);
|
||||
- Qwen 被拒 400 共 10 次;
|
||||
- 阶段失败里大量记成 unknown:07-board 16 次,08-eval 8 次;
|
||||
- dev 命令 3568 次,其中执行器退出 206 次、超时 157 次;
|
||||
- Salty 当时 9 个作业排队。
|
||||
- **数据口径**:
|
||||
- 导出时没有速度过滤;
|
||||
- 验证器不看精度构成;
|
||||
- 04 的 Qwen 被标成 teacher,所以 54 条进了 SFT;
|
||||
- SFT 里混着 v12–v17;
|
||||
- dev 里能读到 harness 源码和评测路径,有 10 条去找过评测集;
|
||||
- 37 道基准题里有 29 道 GLM 训练数据里也有。
|
||||
|
||||
**10-16 19:00 逐条复核,更正上面几处(用户要求解释每条、判断做法是否合理)**
|
||||
- "没有独立监控"是错的:本机 MacBook 的 launchd `cc.d-robotics.bench-health` 每 10 分钟跑 ~/.bench-health/bench_health_alert.sh,发飞书。盲区:
|
||||
- health_check.py 只查账号 PAUSED/RECONCILING;05 被 429 时账号状态是 COOLING,16:00–17:22 没报。
|
||||
- 15:33–16:14 误报"虚拟机没在运行":ssh 的 "remote port forwarding failed for listen port 7897" 警告混进输出第一行被当成状态(虚拟机 10-12 10:17 起没重启过),这 50 分钟其他检查全被挡住。
|
||||
- "约 4000 轨迹小时"是建号到结束的钟点数(含排队、暂停、74 条 ABANDONED 的 1720 小时)。agent 实际干活(wall_s):取消 922 + 系统出错 637 + 未验证 99 ≈ 1660 小时,占 10-09 以来 agent 总时长约 42%。按 10-14 以来同时长轨迹的通过率估,取消的 245 条约值 65 条通过(偏高,被取消的多是难题);同期实际通过 104 条。245 条取消里 149 条编译过至少一次。
|
||||
- 盘 19:00 是 68%、剩 25G(当时 81% 是归档包上传重试的临时高点);/tmp 有 737 个 bench-jail-* 残留目录约 1G。
|
||||
- Clash 已从全局改成规则模式,但只有 bigmodel.cn、36.144.20.182:22、向日葵直连,其余 MATCH,JMS;VM `curl --noproxy '*' cip.cc` 出口仍是洛杉矶;订阅仍 8 分钟更新;en0 仍 inactive。
|
||||
- 07-board 的 unknown 48 小时 11 次全在 10-15 01–12 点(板子盘满,a263374 已修),10-16 0 次;但各阶段合计 unknown 约 47 次(01-fetch 10、05 6、06 6、08 5…)。
|
||||
- 429 只记了 RATE_LIMIT,response_headers 为空,正文没存;账本里 429 共 18 次。
|
||||
- datasets-final 按批次分目录,sft.jsonl 每条带 model 和 agent_config_version:Qwen 进 SFT、版本混杂都不用改代码,训练时不用 04 的文件、按字段筛即可。
|
||||
- 复核后建议去掉:配置热加载、托管 CI、开机自动拉起采集、扩 VM 内存、量化比例门槛(速度过滤已覆盖全 int16)、按版本分数据集、账本清理、账号用量预警、板卡定时巡检、日报;无损重启和加核先观察/先测。
|
||||
|
||||
**Why:** 用户要把它建成稳定系统,问最大问题和还差什么。
|
||||
**How to apply:**
|
||||
- 先上低风险的告警和清理,再上发布纪律(冻结版本、小批量试跑、无损重启),最后加固判分。
|
||||
- 改判分、改提示词、重启、删东西都要用户拍板。
|
||||
|
||||
相关:[[v18-zero-pass-why-20261016]] [[sft-autonomy-tally-20261016]] [[bench-restart-hazards]] [[mac-mini-clash-global-proxy-20261013]] [[verifier-audit-20261012]]
|
||||
@ -1,20 +0,0 @@
|
||||
---
|
||||
name: stop-06-20261017
|
||||
description: 10-17 11:30 06 的 GLM 账号 B 从 06:29 起一直 429,用户说 key 没额度了、让停 06;只 SIGTERM 不取消,原命令/环境存在 configs/stop-06-20261017
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 912a804a-505b-4f45-ac78-d189a02a99ab
|
||||
modified: 2026-10-17T03:30:49.519Z
|
||||
---
|
||||
|
||||
- 10-17 06:28:57 起 glm-coding-account-b 每次请求都 429(RATE_LIMIT,正文没存),到 11:26 仍是;05 的 glm-coding-s600-account-c 一直正常。用户确认"那个 key 没有额度了"。
|
||||
- 11:30:04 给 06 collector(pid 1671)发 SIGTERM,没有取消任何轨迹。当时 06:14 条 PAUSED_INFRA(等额度)、2 条 RUNNING,进程等这 2 条结束或暂停后才会退出。
|
||||
- 拉起用的东西在 VM 的 /mnt/data/ming01.wang/configs/stop-06-20261017/(700 目录,文件 600):cmdline.txt、cwd.txt、logs.txt(日志追加到 logs/collect-s100p-r06-1012-171614.log)、environ.bin(含密钥,用完要删)。BENCH_CONFIG=configs/mini/s100p-r06-v22.yaml。
|
||||
- 之后两种走法:
|
||||
- 账号 B 额度恢复:同一次 VM 开机内按存档原命令/环境重启 06,暂停的会自动续。
|
||||
- 换新 key:要用新组名写新配置(老组规则不能改,见 [[bench-restart-hazards]]),暂停的 14 条续不了,要取消补号。
|
||||
- 健康检查的 RUNS 里还有 06,collector 退出后会报"调度程序没在跑"、暂停超 1 小时也会报。
|
||||
|
||||
**Why:** 用户不想再等这个账号。
|
||||
**How to apply:** 恢复 06 前先问用户用哪个 key;换 key 时取消暂停的再补号。相关 [[check-20261016-1700]] [[poweroff-20261016]]
|
||||
@ -1,28 +0,0 @@
|
||||
---
|
||||
name: success-rate-breakdown-20261008
|
||||
description: 04/05/06 成功率低的原因、GLM 和 Qwen 为什么看不出差距(同题对比打平、考卷区分不出)、各类失败的条数
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 27581662-7413-4d12-9a88-979ce883a5f0
|
||||
modified: 2026-10-07T22:10:15.554Z
|
||||
---
|
||||
|
||||
2026-10-08 分析(04/05/06,排除取消,共 590 条,PASS 249 条,占 42%):
|
||||
- agent 自己走完、但精度真的超出容差的,只有 2 条。其余失败的构成:
|
||||
- 被系统中途终止约 200 条:单条命令超 900s 81 条;"模型回复不完整"68 条(实为本地 413 上下文过长 57 条 + 代理不可用 11 条,见 [[quota-unknown-autorecover-fix]]);板卡清理/请求状态不明 53 条。
|
||||
- "暂停后过期" 70 条:请求状态不明导致暂停,一直停到 24h 被判超时,并没有真正在跑。之前说的"满 24 小时 72 条"其实就是这 70 条,真正跑满 24 小时的只有 3 条。
|
||||
- 判分和评测问题约 31 条:
|
||||
- pose 要求关键点平均偏移不超过 2px(registry/gates.yaml:59),算全部 17 个关节、不做置信度过滤;有 7 条 OKS 精度在容差内,只因偏移 2.4–8.4px 判失败。PASS 的那几条偏移在 1.0–1.3px。
|
||||
- rtmpose 的 11 条全部 UNVERIFIED,原因是 "fixed crop official_model mismatch",评测器验证不了。
|
||||
- 阶段超时 13 条,导出失败 3 条。
|
||||
- GLM 和 Qwen 的对比:
|
||||
- 同任务、同轨迹序号、两边都真正跑了的 56 对:Qwen 30 条 PASS,GLM 33 条。两边都自己走完的 28 对:26 比 27,基本打平。
|
||||
- 重叠的题几乎全是 cls/det,默认流程一次编译就能过,属于天花板效应。
|
||||
- Qwen 还没轮到 pose/mde/seg。早先"Qwen 自己走完的成功率 92%"是幸存者偏差,不能拿来和 GLM 比。
|
||||
- Qwen 自建服务没有前缀缓存,单次请求中位数 39s,GLM 是 3.7s。Qwen 用时长、请求状态不明多,主要是服务的原因,不是模型本身。
|
||||
**Why:** 用户怀疑 GLM-5.3 应该明显强于 Qwen-3.8-27B。结论是现在这张考卷区分不出两者:简单题两边都满分,难题两边都被系统砍掉或被判分规则卡住。
|
||||
**How to apply:**
|
||||
- 比较模型时只用同题配对、并且两边都真正跑过的轨迹;先把"暂停后过期"和"系统终止"剔掉。
|
||||
- pose 的 2px 规则和 rtmpose 评测器属于判分标准的改动,要用户拍板。
|
||||
相关:[[quota-unknown-autorecover-fix]]
|
||||
@ -1,49 +0,0 @@
|
||||
---
|
||||
name: v16-quality-20261015
|
||||
description: 10-15 09:30 对比 v16 与 v15 轨迹:dev 用得多、兜底配置少、慢 1.5 倍的通过少,但样本只有 8 条通过;240 轮上限开始卡最自主的轨迹;吞吐暂时降到约 1/4
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 24818155-52a9-4a37-a413-b2ac9f163981
|
||||
modified: 2026-10-15T01:43:46.343Z
|
||||
---
|
||||
|
||||
10-15 09:30 复查。v16 上线 14.5 小时,共 53 条轨迹,有判定的 12 条(8 过)。
|
||||
|
||||
**自主性(v16 vs v15)**
|
||||
- 用了 bench dev 的有 52/53 条,共 993 条命令,成功 614 条。v15 迁到 VM 后,171 条里只有 5 条做过本地试验。
|
||||
- 17 条在 dev 里改图或导出;48 条做过编译或推理试验。
|
||||
- 3 条提交了自带的图或校准:06 maxvit.t(Einsum 改写)、05 deeplabv3plus.r101(拆 1×1 卷积)、06 vit.h.14(自做校准)。
|
||||
- 通过轨迹的最终配置:
|
||||
- v16(8 条):5 条是默认配置一次过,2 条按节点调,1 条全 fp16 兜底(05 efficientvit.seg.b0,板上慢 110 倍仍 PASS)。
|
||||
- v15:06 有 25/46 全 int16;05 有 31/46 全 fp16;按节点调的只有 4 条和 1 条。
|
||||
- 比平台默认慢 1.5 倍以上的通过:v16 1/8;v15 的 06 是 17/39,04 是 11/25。
|
||||
- 说明:先跑完的都是简单题,v16 样本太小,下不了结论。
|
||||
|
||||
**代价**
|
||||
- **240 轮上限开始卡人**:06 已结束的 v16 轨迹里,有 3 条 BUDGET_EXHAUSTED,都是跑满 240 轮。当时 token 只用了 24–51 万,上限 80 万。
|
||||
- 每条 bench dev 命令算一轮。vit.h.14 自做校准后就被截掉了。
|
||||
- v15 的预算用完都是 token 到 87 万左右(中位 108 轮)。
|
||||
- 当时在跑的 v16 轨迹,工具调用已有 150–207 次的不少。
|
||||
- max_turns 在 configs/mini/*-v16.yaml 的 rollouts 段。collector 里有配置缓存,改了大概率要等重启。
|
||||
- **吞吐**:v16 上线后 14.5 小时只有 8 条 PASS。前一段 22 小时(10-13 20:41 到 10-14 19:00)有 44 条。原因有三:
|
||||
- 重启后所有轨迹从头跑;
|
||||
- 夜里 dev 只有 8 个槽,198 条命令排到执行器超时。07:50 改到 37 个后,只剩 2 条;
|
||||
- 每条 dev 命令要 75 秒左右。
|
||||
|
||||
**Qwen(04)在 v16 下:3 过 3 败,以前是 41 过 8 败**
|
||||
- 失败的 3 道(mspn.r50、simcc.resnet50、yolo11m.seg),以前都是靠"全模型 int16"通过的,板上比默认配置慢 2.8 倍、1.5 倍、3.4 倍。
|
||||
- v16 提示里有速度要求,Qwen 不再用这种兜底,改成按节点调,结果没调过:
|
||||
- 两道 pose 题差在"关键点平均偏移 ≤2 像素"这条附加约束;
|
||||
- yolo11m.seg 的 mask_ap 从 46 掉到 16。
|
||||
- mspn 的 25 条 dev 命令里,16 条死在夜里 dev 排队(07:50 已修)。
|
||||
- 结论:这是考卷变难了,不是 Qwen 变差。训练前基准大多在 v15 下测的,**训练后要在 v16 下比,基准也得在 v16 下重测**。04 按 5 路并行,37 道全测一遍约 4 天。
|
||||
|
||||
**Why:** 用户要的是能体现自主性的 SFT 数据;兜底配置和速度兜底不该算好轨迹。
|
||||
**How to apply:**
|
||||
- 下次重启考虑把 max_turns 提到约 400,token 上限不动。要用户拍板。
|
||||
- 导出时按速度过滤(不用重启)还没做。
|
||||
- 等难题跑完再重新对比 v16 和 v15。
|
||||
- 访问路径:本机不在 192.168.66.x 网段时,用 `ssh -p 443 amos@180.76.192.183`。
|
||||
|
||||
相关:[[agent-autonomy-audit-20261014]] [[overnight-check-20261015]]
|
||||
@ -1,18 +0,0 @@
|
||||
---
|
||||
name: v17-autonomy-check-20261015
|
||||
description: 10-15 19:40 查 v17 是否真做到"整条链路 agent 自己扛":32 条 GLM 跑 5 小时 0 条结束;全用开发环境,但从零自己导图 0、自备校准 0、只有 1 条图手术;72% 时间在开发环境,9 条各花约 4 小时编默认配置做速度自查
|
||||
metadata:
|
||||
type: project
|
||||
---
|
||||
|
||||
用户问:拿到权重/代码/工具链/验证集/板卡后,agent 是否已能自己导图、准备校准、本地诊断、编译、上板、验证并长时间跑完。10-15 19:40 统计 v17(14:37 起)的 32 条 GLM(05 16 条、06 16 条)。
|
||||
|
||||
- 32 条全部还在跑,0 条结束;轮数用了 30–134/400。
|
||||
- 都用了 bench dev(每条 3–40 次);但 32 条都先跑平台导出(03-export),从零自己导图 0 条;正式配置里自带 cal_data_dir 0 条。
|
||||
- 唯一用自己图的:06 deeplabv3.resnet50 r03,在平台图上把 ASPP 后 2560→512 卷积拆开(fixed_split.onnx,浮点 cos=1.0),正式编译后在上板。
|
||||
- 正式编译过 11/32;评测 6 次(yolov8n、yolox.x、yolo 系列 seg)量化精度都远低于浮点(如 yolov8n 26.3 对 40.4 mAP),还在迭代。
|
||||
- 时间:共 154 小时,开发环境 111 小时(72%,其中开发环境里编译 61 小时),正式阶段 15 小时(10%)。9 条(gma、dpt.hybrid、dpt.large、unidepth.convnext.l、flowformer++、raft.large、depth.anything.v1.large/v2.base、segformer.b2)各花约 4 小时在开发环境用默认配置编一次,是提示词里 1.5 倍速度自查要求的;大模型在 4–6 核开发容器里要 4 小时以上。
|
||||
- 设计上的限制:看不到板卡(只能走正式阶段)、看不到评测集(只能 08-eval)、平台导出永远兜底("不写就用平台导出的图")。
|
||||
|
||||
**Why:** 用户要的是"整条链路由 agent 自己扛"的轨迹;当前只是"允许",没有"做到"。
|
||||
**How to apply:** 提议过(未定):平台按题预算默认配置的延迟写进题目信息、删掉"自己编一次默认";是否去掉平台导出/校准兜底由用户定。v17 第一批结果要几小时后才有。相关 [[agent-autonomy-audit-20261014]] [[sft-sample-review-20261015]] [[restart-20261015]]
|
||||
@ -1,68 +0,0 @@
|
||||
---
|
||||
name: v18-own-inputs-20261015
|
||||
description: v18–v21(10-15/16):agent 自己导图、做校准(v21 起给训练集原图)、速度参考平台算、bench dev --board 板上自测(板卡容器里直接跑,S100P/S600 都开);dev 排队不算命令时间(待重启);stage wait 10 分钟
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: dfb570b8-7fd2-43b1-9acf-f30e5447c88b
|
||||
modified: 2026-10-16T08:00:47.044Z
|
||||
---
|
||||
|
||||
用户定(10-15 晚):A 平台预算默认配置延迟、删"自己编一次默认";B 去掉平台导图和校准兜底;热部署。
|
||||
|
||||
**怎么做的(分支 1f8a7e2、2ed2f5f;21:43 上线,部署脚本 configs/ops-20261015/v18/deploy.sh,备份 config-backup-20261015/before-v18-2143)**
|
||||
- 只对新轨迹生效:rollout.json 的 agent_config_version >= 18(stagerun.OWN_INPUTS_FROM)。stagerun._ctx_from_rollout 把 own_inputs_required 放进 ctx.extra,04 和 bench dev 都从这里读;v17 在跑的轨迹规则不变。
|
||||
- 04:v18 的 compile.yaml 必须写 model_parameters.onnx_model 和 calibration_parameters.cal_data_dir,缺了 own_input_invalid(编译前拒、不占次数)。没 --config 也会拒。
|
||||
- bench dev(worker spec own_inputs):不挂 cache/onnx,data/calib 换成 data/calib-raw。home/outputs 里只有 openmmlab 环境和 pose 人框,没有平台图。
|
||||
- calib-raw:原始训练图大多没迁过来(个人空间只有 ADE20K、HPatches 的 zip),所以是把平台 128 张 npy 反归一化成模型输入尺寸的 8 位 PNG(personal:/bench/data/calib-raw/<同 calib 路径>/png/,多帧是 _0/_1)。62 套全转好,抽看颜色正常。反归一化按候选(声明的 recipe、imagenet、0-1、0-255、[-1,1])挑落在 0–255 且像照片的那个。缺点:缩放/裁剪已经做了,agent 只需要做归一化、通道、布局。
|
||||
- 速度参考:scripts/speed_ref.py 在 dev 容器里用默认配置编一次,hb_model_info 估计延迟写 bench/agent/speed_ref/<job_id>.json(jail 只绑 job.json 单个文件,看不到 job 目录;agent/ 目录是绑定的、实时可见),path-guard 放行这个目录。
|
||||
- 一次开 8 路时 Salty 满了,agent 自己的 dev 作业开始排队,已停掉删掉。改成 configs/ops-20261015/speedref-loop.sh:每 10 分钟只给有 v18 轨迹的题编,3 路、6 核、32Gi;失败的在 runs/zz-speedref/<job>/FAILED 记一笔不重试。
|
||||
- 已有:mobilenet.v2 0.182 ms、yolov8n(S600) 0.8214 ms;每题 18–40 分钟。
|
||||
- 提示词 v18:SYSTEM.md / prompts/e2e.md 删平台图和校准位置、删"自己编默认配置";04 Qwen 也共用这套提示词,新 Qwen 轨迹同样是 v18。
|
||||
- 测试:Salty 全量 base/new 一致(73 个文件),VM 上在沙箱里失败的那些也逐个对比一致;guard-core.test.js 加了两条。
|
||||
|
||||
**10-16 夜间(v18 跑了 3 小时、11 条)**
|
||||
- agent 真的在自己导图、做校准(04 yolo11l.seg 自己的图+校准编过、上板、评测 39.0/48.1)。
|
||||
- 问题:提示词说"把 /data/calib/ 换成 /data/calib-raw/",agent 去 /data/calib-raw 找,每条花 5–8 次 dev 命令。00:55 改成 dev 环境变量 $BENCH_CALIB_IMAGES(salty_dev.calib_images),实测指向 128 张。
|
||||
- speed_ref 按轮跑时一轮等最慢的那道,改 --follow 滚动 3 路(configs/ops-20261016/speedref-follow.sh)。
|
||||
- dev 命令 TERMINAL_DEADLINE_EXCEEDED:有的轨迹 dev 容器按编译历史要 64Gi,Salty 排队超过 agent 给的超时,连 ls 都超时;没修(本地时限在 collector 里定,热改不了)。
|
||||
|
||||
**板上自测 v19(10-16 01:18 上线,16a9c50)**
|
||||
- 用户同意开。代码原有(terminal target=board),但 collector 里的请求检查只认 development 且不能热换,所以 `bench dev --board` 在命令前加标记 `#bench:board\n`,由新进程里的 terminal.run(board_request)转成板上请求。只对 v18+、nash-m 开放;上传当前目录(本地 + 只在开发环境里的大文件,salty_dev.download_dir 从个人空间拉),一次最多 30 分钟、4 GiB,跑完删 VM 上的副本(VM 盘只剩 25G)。
|
||||
- 板上隔离实测:S100P(rdk-sandbox,容器)要三处改 terminal_jail:系统目录递归绑定(/usr 下有只读主机挂载,用户命名空间里非递归绑定 EINVAL)、加 /host_lib 和只读 /sys(BPU 运行库读核心类型)、透传 LD_LIBRARY_PATH(/usr/hobot/lib)。之后隔离环境里 shadow、/userdata、/root 不可见、无网络,mobilenet.v2 在 BPU 上 0.79 ms,回执确认清理;端到端(hbm 只在开发环境、脚本在本地)也通过。
|
||||
- S600(s600-40015/40017)是 Docker 默认 seccomp,unshare 任何命名空间都 EPERM,要板卡主机改容器启动参数才行,等用户/板卡负责人定。
|
||||
- 探测脚本在会话 scratchpad(boardprobe.py、e2eboard.py),用 configs/restart-20261015/withenv.sh + env-06/05 拿板卡密钥,不打印。
|
||||
|
||||
**10-16 早上(用户定:板卡容器本身就是隔离、跑完删目录;原图去 36 找;排队不算命令时间)**
|
||||
- v20(04:52,22bf19d):板上自测去掉命名空间隔离,命令在板卡容器的会话目录里直接跑(board_guard 管进程和板卡锁),$BENCH_OUT 拉回,会话目录照旧删;S100P 0.75 ms、S600 0.97 ms,板上目录确认已删。所有 v18+ 题都能用。
|
||||
- v21(06:36,dbb717f):calib-raw/<set>/images/ 是训练集原图(原尺寸)+ images.json(顺序、来源、姿态题人框);v18–v20 的轨迹仍用 png/。来源:36 的 JuiceFS 很慢(小文件约 70 秒、并行也只有 1.6 MB/s),所以 COCO 走 images.cocodataset.org(字节一致)、ImageNet 把 4 个 parquet 分片并行分块拉到本地再取同一批行、KITTI/Cityscapes 并行读(zip 按偏移读+CRC)、NYU 从官网下 .mat(sha 与平台一致,索引从 1 开始)、ADE20K/HPatches 从 pool。62 套全过,NYU 每张 rgb 哈希与平台一致。临时文件已清。
|
||||
- b329385:prepare 给走 Salty 的 dev 请求多 3 小时排队余量,命令 timeout 从容器启动算;prepare 在 collector 里,**下次重启才生效**。
|
||||
- e543d7b:bench stage wait 一次最多等 10 分钟(原 60 秒)。v18 的 efficientvit.b0 400 轮里 302 轮是轮询,没编译就 BUDGET_EXHAUSTED。
|
||||
- 截至 06:40:v18 11 条、v19 6 条、v20 2 条;板上自测还没人用;速度参考只算完 3 道(S600 大深度模型一道约 3.5 小时)。
|
||||
|
||||
**10-16 下午修复(用户定:A 批热上、B 批写完直接重启、GLM token 2M、可删残留)**
|
||||
- A 批 v22(21fce5e,15:36 热部署,脚本 configs/ops-20261016/v22/deploy.sh,备份 config-backup-20261016/before-v22-1536):
|
||||
bench stage wait 两个入口都只看一眼不睡、指向 terminal_exec wait_handle;板上自测先 salty storage ls 量大小(>512 MiB/40 个子目录拒绝)、
|
||||
取回后到时限也拒绝,拒绝原因作为命令结果回给 agent,snapshot/dev-files 任何路径都删(删了 2.5 GB 残留);
|
||||
speed_ref 编不出来写 latency_ms_est=null + status unavailable,已给 4 道补上;SYSTEM.md 三处;VERSION 22。
|
||||
在跑的 speed_ref --follow 进程还是旧代码(新失败只写 FAILED),等它手上 3 道编完再重启它。
|
||||
- B 批(b248a28、f2322ac,重启上线,脚本 configs/restart-20261016/,配置 configs/mini/*-v22.yaml):
|
||||
quota 400 且正文说上下文超长 → CONTEXT_TOO_LONG(不续)→ rollout CONTEXT_BUDGET_EXHAUSTED(结局 BUDGET_EXHAUSTED);
|
||||
broker 内部异常打日志(类型/消息/位置),没发出的 generation 记 BROKER_ERROR 可续;连接被断的请求只占账号 120 秒(UNKNOWN_HOLD_S);
|
||||
agent/context-mask.js + rollouts.context_mask_chars=450000(只 04 Qwen):发出时省去最早的工具输出、降到 60%,会话文件不动——
|
||||
Pi 自带 compaction 只在两次 prompt 之间触发(agent_end),单个长任务里永远不触发,所以没用它。GLM max_tokens 2M、不压缩。
|
||||
- 测试:VM 全量 base/new 对比只有已知不稳定的(explicit_batch、collection、watchdog_stop、execution 的 0.1 秒计时);
|
||||
test_pi_session 要设 BENCH_TEST_NODE=/home/amos.guest/.local/lib/salty-controller/runtime/bin/node 才不跳过,真 SDK 下掩码测试通过。
|
||||
- 经 Mac mini 的 ssh 长命令约 5 分钟会被断,远端轮询每次 ≤4 分钟。
|
||||
|
||||
**10-16 15:00 体检**
|
||||
- 板上自测 4 次败 3:agent 在整个工作区(1.2 GB)跑 --board,从个人空间拉回要 40 分钟,超过命令 25 分钟时限,报 exit 2 无原因,VM 留下 2.5 GB snapshot(失败路径不删)。成功那次是 29 MB 小目录、2 分钟。
|
||||
- broker 内部异常(503 BROKER_UNAVAILABLE,异常被吞不记日志)在 rollout.py 判成 INFRA_INCOMPLETE_GENERATION 直接结束(05 hrnet.w32 v21 白跑 2.9h),应可续。
|
||||
- 断网一次(14:34–14:46)→ 05 GLM 组和 Qwen 组各卡 RECONCILING 到未知请求的期限(约 20–27 分钟),组里所有轨迹 PAUSED(自动恢复)。
|
||||
- GLM 真正卡住的是 max_tokens 800k(未命中缓存的输入+输出),约 90–150 轮就用完,400 轮上限摸不到;在跑的有 3 条到 88–96%。
|
||||
- 速度参考 18 小时只算完 3 道、败 4 道(S600 大深度模型默认配置 6 核 10 小时编不完);32 道 v18+ 题 29 道没有参考,agent 又开始自己编默认配置(maxvit.t)。
|
||||
- 14 小时 750 条 dev 命令里 42 条排队没开跑就超时(b329385 等重启);agent 常在 dev 里 4 核跑 4–10 小时的 hb_compile。
|
||||
- Qwen 当天 7 条因上下文超 256k 结束(v18+ 4、v17 3)。
|
||||
|
||||
**Why:** 用户要"整条链路 agent 自己扛"的轨迹;见 [[v17-autonomy-check-20261015]]。
|
||||
**How to apply:** 看 v18 效果时比 v17:自带图/校准是否真自己做(不是 cp)、开发环境里编默认配置的时间是否消失、编译被 own_input_invalid 拒的次数;calib-raw 的局限要告诉用户。相关 [[restart-20261015]] [[agent-autonomy-audit-20261014]]
|
||||
@ -1,58 +0,0 @@
|
||||
---
|
||||
name: v18-zero-pass-why-20261016
|
||||
description: 10-16 18:00 查 v18+(自己导图)90 条 0 通过的原因:42 条被今天两次重启取消、82/90 是从没通过过的难题、走到评测的都差精度、预算/上下文/4 小时编译上限;值得注意的行为:10 条想找评测集做本地验证(被沙箱挡住),28 条照搬平台导出脚本
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 94b62e94-8a51-48f6-a116-dccb712c9118
|
||||
modified: 2026-10-16T09:55:21.565Z
|
||||
---
|
||||
|
||||
10-16 18:00 统计 v18+ 共 90 条轨迹(GLM 75,Qwen 15)。
|
||||
|
||||
**0 通过的原因(按影响大小)**
|
||||
1. **重启取消 42 条**:
|
||||
- 16:20 全量重启,取消 26 条;
|
||||
- 17:21 05 换 GLM key 再重启,取消 16 条。
|
||||
- 这 42 条里,34 条还没走到正式编译。
|
||||
2. **题太难**:82/90 是任何版本都没通过过的题。按类型:深度 26、检测 16、大分类 15、姿态 11、分割 10、光流 7、实例分割 5。
|
||||
3. **走到评测的 9 条都差精度**:
|
||||
- Qwen:yolo11l.seg、yolov8n.seg、efficientvit.seg.b1、depth.anything.v2.small;
|
||||
- GLM:vkitti.small、convnext.small 和 efficientnet.b0 的 verifier 不过;
|
||||
- vkitti.base 板上评测卡住。
|
||||
4. **预算和时限**:
|
||||
- GLM 80 万 token 用完(efficientnet.v2.l、convnext.small);
|
||||
- efficientvit.b0 有 302 轮在轮询 stage wait,已修;
|
||||
- Qwen 上下文超 256k 有 5 条;
|
||||
- S600 大深度模型(zoedepth.n/nk、hypersim.large)正式编译超 4 小时阶段时限,4 次。
|
||||
5. **开发环境里耗时长**:不少轨迹先在 dev 里 4 核跑 2–6 小时的 hb_compile,dev 累计 5–14 小时还没正式编译。agent 没有办法终止自己卡住的 dev 命令。
|
||||
6. 05 的 S600 账号 16:00–17:22 一直返回 429,整批停摆。
|
||||
|
||||
**值得注意的行为**
|
||||
- **想找评测集做本地验证:10 条**(GLM 6、Qwen 4)。
|
||||
- 它们照着 harness 源码里的路径找 /home/ming01.wang/bench-data/t1/<数据集>/n100-s20260914、bench/data/pool、cache/baseline。
|
||||
- 想做的事:先在本地算精度,再花正式次数。比如 hypersim.small 自己写了 proxy_acc 代理精度评估。
|
||||
- 沙箱全部挡住:dev 里看不到这些路径。
|
||||
- 背后的真实需求:缺一份 agent 能用、又不是考题的验证集。
|
||||
- **"自己导图"多半照搬平台导出脚本**:
|
||||
- 28 条读了或复制了 harness/tools/export_onnx.py,有的写着"platform-identical recipe";
|
||||
- 自己写 torch.onnx.export 的有 27 条,两者有重叠。
|
||||
- harness 源码在 dev 里可见($BENCH_ROOT/harness),评测器的预处理也能读到(eval/common.py),有的就照它来做校准。
|
||||
- **像工程师的做法**:
|
||||
- 逐层对比精度:GLM 10 条、Qwen 8 条,例如 swin.b 用 HB_ONNXRuntime 沿网络深度追踪余弦;
|
||||
- 微基准:造小探针模型测 int8/int16 的开销,4 条,vkitti.base 还试了板上实测;
|
||||
- 读工具链手册(.horizon/skills):37 条;
|
||||
- 等价改写:efficientvit.b1 在线性注意力里缩放 relu(q)/relu(k),yolov8n.seg 把 SiLU 换成 tanh 形式。
|
||||
- **越界尝试**:06 convnext.small 试过一次 `sudo ls` 看缓存权重。
|
||||
|
||||
**Why:** 用户问第二版为什么一条都没过,以及改完以后有没有让人惊讶的操作。
|
||||
**How to apply:**
|
||||
- 想让 v18+ 出通过,可以考虑:
|
||||
- 少重启;
|
||||
- 给 agent 一份不是考题的验证子集;
|
||||
- dev 命令要能取消;
|
||||
- 大模型 dev 容器给更多核;
|
||||
- 重排队列,别只剩难题。
|
||||
- 这些都要用户拍板。
|
||||
|
||||
相关:[[sft-autonomy-tally-20261016]] [[check-20261016-1700]] [[v18-own-inputs-20261015]]
|
||||
@ -1,28 +0,0 @@
|
||||
---
|
||||
name: verifier-audit-20261012
|
||||
description: 10-12 核查 Qwen 训练前 20 个通过:成绩是真的、没抄没改;但验证器不查是否真 int8(全 int16 也算过)、只抽 100 张图;Qwen 与 GLM 打平是因为系统包办了大半
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 6c848f33-e8d9-4981-ade2-dc77987005d7
|
||||
modified: 2026-10-12T12:29:53.734Z
|
||||
---
|
||||
|
||||
2026-10-12 用户不信 Qwen 训练前过 20/23、平均编译 2.2 次,要求查验证器。结论:
|
||||
|
||||
**成绩是真的**(04 的 20 个通过 + 3 个失败逐条查):
|
||||
- 全部有板端实测(source=board, n=100),自己按 fp32/quant 重算掉点与 verdict 一致;FP32 基线与 GLM-06 完全相同。
|
||||
- HBM sha256 全库无重复,没抄别人的模型文件。
|
||||
- 会话还在盘上的 13 条(10-08 后的;10-08 前 9 条会话已归档没翻):没读别的 run/rollout、没碰 oracle、没写 stages/verdict/claim、没绕开 `bench stage run compile` 直接调 hb_compile;Agent 发起编译次数 = stages 表计数。
|
||||
- 验证器会判失败:全库 G3 fail 88 次;Qwen 三个失败都是真的(mobilenet.v3.small top1 67→2、yolov8n.seg 掉 21.6%、solov2 掉 7.4%)。
|
||||
|
||||
**为什么像"很厉害"**:导出、FP32 基线、上板、评测全是 harness 做的,Agent 主要写 compile.yaml,系统提示(agent/prompts 里 append-system)直接给了一份模板。mspn、simcc、midas、yolox.l 四题 Qwen 与 GLM 板端成绩小数点后 16 位相同——配置几乎一样、编译器结果确定。盘上 10 份 Qwen 通过配置里 6 份就是模板默认值。
|
||||
|
||||
**验证器两处偏松(对 GLM、Qwen 一视同仁)**:
|
||||
1. 不检查是否真 int8:job.json gates 只要 G1(有 hbm、march、输入格式、hb_verifier 余弦≥0.99、日志无 ERROR)+ G3(板端 100 张、相对掉点≤5%);G4 速度只记录。`quant_config.model_config.all_node_type: int16` 照样通过。Qwen efficientnet.b1、mobilenet.v3.large(1.85ms vs GLM 1.31ms)是全 int16 过的;GLM-06 yolo11m.seg 也是。盘上能查到的通过配置:Qwen 2/10、GLM 2/6 全 int16。SFT 用的 GLM 轨迹里也有这招。
|
||||
2. 只抽 100 张(verdict 'pass-screen'):3–4 张差异在噪声内,如 regnet.y.400mf 78→75 算过,efficientnet.b1 量化后反比 FP32 高 3 张。
|
||||
|
||||
**逐图核对(同日 20:30,页面 https://claude.ai/artifact/T4CcvQHeBFFZFrhzqU32A8)**:5 道分类题从板端逐图预测重数,答对张数与报告完全一致;8 位模型与原始模型同答案 95–100/100,GLM 与 Qwen 之间 96–100/100;ConvNeXt-Base 有 14 张三者错得一样,量化只改了 3 张。Qwen 坏掉的 mobilenet.v3.small 只对 2 张、答案集中在几个类。取数方法:分类的 `08-eval/a*/predictions.jsonl`(sample_id 末尾是真实类别号,取 eval.json 的 quant 与 verdict 一致的那个 attempt);原始逐图预测在 FP32 基线缓存 metrics.json 的 `_pred`(顺序同 predictions.jsonl),基线目录从 08-eval/meta.json 的 `--baseline` 参数取,在 personal:/bench/cache/ 下;测试图 personal:/bench/data/t1/<数据集>/n100-s20260914/20260914.tar;类名在 personal:/bench-runtime/home/toolchains/bench-venv/lib/python3.10/site-packages/torchvision/models/_meta.py。检测/分割/姿态/深度只存总分、不存逐图结果。`salty storage cp` 本地目标要写绝对路径,相对路径会静默失败。
|
||||
|
||||
**Why:** 用户要靠训练前后对比给训练方向反馈;验证器偏松 + 系统包办会让两边都接近满分,看不出差别。
|
||||
**How to apply:** 汇报通过率时说明这两处偏松。若用户决定收紧(禁全 int16/float、或加速度门槛),旧结果可按 compile.yaml + 已记录延迟重判,不必重跑;归档的配置在 personal:/bench-root/archive/<run>/<job>/<kk>.tgz。相关:[[qwen-sft-benchmark]] [[success-rate-breakdown-20261008]]
|
||||
@ -1,7 +1,7 @@
|
||||
"""reward v0.8:只按本次真实门证据、成本、内容审核和超时计算。
|
||||
"""reward v0.7:只按本次真实门证据、成本、内容审核和超时计算。
|
||||
|
||||
不读取 Oracle 对照或 BLOCKED 正确性标签。BLOCKED 的已测阶段仍保留分数,
|
||||
但导出器不将未证实的 BLOCKED 纳入奖励训练。板卡不可用(连不上或板上磁盘满)仍按既有规则免超时罚分。
|
||||
但导出器不将未证实的 BLOCKED 纳入奖励训练。板卡不可用仍按既有规则免超时罚分。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@ -9,22 +9,21 @@ import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
RECIPE_VERSION = "v0.8"
|
||||
RECIPE_VERSION = "v0.7"
|
||||
BOARD_STAGES = ("07-board", "08-eval")
|
||||
BOARD_UNAVAILABLE = ("board_offline", "disk_full") # v0.8: 10-15 S600 /userdata filled by kept sessions
|
||||
PASSING = {"pass", "pass-screen", "pass-weak"}
|
||||
W = {"G1": 1.0, "G2": 1.0, "G3": 2.0, "cost_turns": 0.5, "cost_tokens": 0.5,
|
||||
"timeout": 1.0}
|
||||
|
||||
|
||||
def board_unavailable(verdict: dict) -> bool:
|
||||
"""板卡不可用导致 G2/G3 没法测:G3 unverified,且 07-board 或 08-eval 的最后一次尝试以 board_offline / disk_full 失败。
|
||||
"""板卡不可用导致 G2/G3 没法测:G3 unverified,且 07-board 或 08-eval 的最后一次尝试以 board_offline 失败。
|
||||
verdict.stages 只记最后一次 attempt(verify._latest_meta),所以"最后一次"就是这条轨迹结束时板子的状态。"""
|
||||
g3 = (verdict.get("gates") or {}).get("G3") or {}
|
||||
if g3.get("verdict") != "unverified":
|
||||
return False
|
||||
stages = verdict.get("stages") or {}
|
||||
return any((stages.get(s) or {}).get("error_class") in BOARD_UNAVAILABLE for s in BOARD_STAGES)
|
||||
return any((stages.get(s) or {}).get("error_class") == "board_offline" for s in BOARD_STAGES)
|
||||
|
||||
|
||||
def compute(verdict: dict, max_turns: int = 40, max_tokens: int = 200000) -> dict:
|
||||
|
||||
@ -202,7 +202,7 @@ class MaterializationTests(unittest.TestCase):
|
||||
def test_timeout_keeps_verified_stage_rewards(self):
|
||||
rd = self.add(execution="TIMEOUT")
|
||||
reward = compute(json.loads((rd / "verdict.json").read_text()), 60, 100)
|
||||
self.assertEqual(reward["recipe"], "v0.8")
|
||||
self.assertEqual(reward["recipe"], "v0.7")
|
||||
self.assertEqual(reward["terms"]["G3"], 2.0)
|
||||
self.assertEqual(reward["terms"]["timeout"], -1.0)
|
||||
manifest = ex.materialize(["r"], "data")
|
||||
|
||||
@ -661,7 +661,7 @@ def _write_dataset(a, work: Path, ds_id: str) -> dict:
|
||||
"include_timeout": True, "purpose": a.purpose, "min_reward": a.min_reward},
|
||||
"stats": stats, "files": file_info,
|
||||
"notes": ["pipeline.jsonl contains historical deterministic evidence only for purpose=all; SFT/RL export reads Agent evidence only.",
|
||||
f"Verdict v4 and reward {RECIPE_VERSION} use only this rollout evidence; Oracle references are history only.",
|
||||
"Verdict v4 and reward v0.7 use only this rollout evidence; Oracle references are history only.",
|
||||
"SFT/reward candidates are labels; no train split, framework conversion, or online RL readiness is implied.",
|
||||
"Writers hold .export.lock; manifest is published last. Readers can lock or verify file hashes for a coherent refresh."]}
|
||||
(work / "manifest.json").write_text(json.dumps(manifest, indent=2, ensure_ascii=False))
|
||||
|
||||
@ -25,10 +25,7 @@ TERMINAL_STATES = frozenset({"DONE", "FAILED", "FAIL", "TIMEOUT", "CANCELLED", "
|
||||
"BUDGET_EXHAUSTED", "INTERRUPTED", "COMPLETED", "ERROR", "DEADLINE_EXCEEDED",
|
||||
"INFRA_ERROR", "AGENT_ERROR", "LOST"})
|
||||
OPERATIONS = frozenset({"session", "agent_start", "turn_start", "auto_retry_start", "auto_retry_end",
|
||||
"tool_execution_start", "tool_execution_update", "tool_execution_end", "compaction_start"})
|
||||
# Pi 0.73.1 core/messages.js: how a compaction summary reaches the model.
|
||||
COMPACTION_PREFIX = "The conversation history before this point was compacted into the following summary:\n\n<summary>\n"
|
||||
COMPACTION_SUFFIX = "\n</summary>"
|
||||
"tool_execution_start", "tool_execution_update", "tool_execution_end"})
|
||||
SUPPORTED_UPDATES = frozenset(f"{kind}_{suffix}" for kind in ("text", "thinking", "toolcall")
|
||||
for suffix in ("start", "delta", "end"))
|
||||
|
||||
@ -144,7 +141,7 @@ def inventory(source: FrozenSource) -> dict:
|
||||
errors.append("UNSUPPORTED_SOURCE")
|
||||
if metadata.get("state") not in TERMINAL_STATES:
|
||||
errors.append("SOURCE_NOT_TERMINAL")
|
||||
messages, operations, segment, turn, compactions = [], [], [], [], []
|
||||
messages, operations, segment, turn = [], [], [], []
|
||||
pending = None
|
||||
ended = False
|
||||
sdk_headers = []
|
||||
@ -272,17 +269,6 @@ def inventory(source: FrozenSource) -> dict:
|
||||
add(payload, observation, event_type="unresolved_mirror")
|
||||
if len(payloads) != len(candidates):
|
||||
errors.append("OBSERVATION_UNACCOUNTED")
|
||||
elif kind == "compaction_end":
|
||||
# Only a compaction Pi resumed after changed what the model saw; failed ones end the rollout.
|
||||
operations.append({"type": kind, "observation_id": location, "payload_sha256": payload_sha(event)})
|
||||
if event.get("willRetry"):
|
||||
kept = (event.get("bench_compaction") or {}).get("kept_messages")
|
||||
summary = (event.get("result") or {}).get("summary")
|
||||
if not isinstance(kept, int) or kept < 0 or not isinstance(summary, str):
|
||||
errors.append("UNSUPPORTED_SOURCE")
|
||||
else:
|
||||
compactions.append({"before": {item["message_id"] for item in messages}, "kept": kept,
|
||||
"summary": summary, "observation_id": location})
|
||||
elif kind == "agent_end":
|
||||
payloads = event.get("messages")
|
||||
if not segment and isinstance(payloads, list):
|
||||
@ -360,17 +346,7 @@ def inventory(source: FrozenSource) -> dict:
|
||||
retained.append(item)
|
||||
if msg.get("role") == "assistant" and msg.get("stopReason") in ("error", "aborted"):
|
||||
errors.append("ERROR_CONTENT_RETAINED")
|
||||
# messages[at:] came after the compaction; the model then saw messages[0] (system), the summary message and
|
||||
# messages[at - kept:at]. The full history stays in messages.
|
||||
views = []
|
||||
for record in compactions:
|
||||
at = sum(item["message_id"] in record["before"] or item["event_type"] == "system_snapshot" for item in retained)
|
||||
if record["kept"] > at - 1:
|
||||
errors.append("COMPACTION_VIEW_UNKNOWN")
|
||||
views.append({"at": at, "kept": record["kept"], "observation_id": record["observation_id"],
|
||||
"summary_message": {"role": "user", "content": [
|
||||
{"type": "text", "text": COMPACTION_PREFIX + record["summary"] + COMPACTION_SUFFIX}]}})
|
||||
return {"messages": retained, "excluded": excluded, "operations": operations, "compactions": views,
|
||||
return {"messages": retained, "excluded": excluded, "operations": operations,
|
||||
"errors": sorted(set(errors)), "tools": tools, "agent_end": ended, "context": context}
|
||||
|
||||
|
||||
@ -398,14 +374,14 @@ def closure(messages):
|
||||
return {"pass": not any(metrics.values()), **metrics}
|
||||
|
||||
|
||||
def _view(items, tools, compactions):
|
||||
def _view(items, tools):
|
||||
# Pi-native blocks preserve thinking, tool args/results and unknown fields.
|
||||
return redact_preserving({"messages": [item["payload"] for item in items], "tools": tools, "compactions": compactions})
|
||||
return redact_preserving({"messages": [item["payload"] for item in items], "tools": tools})
|
||||
|
||||
|
||||
def reconstruct(source: FrozenSource) -> dict:
|
||||
scanned = inventory(source)
|
||||
view, redaction = _view(scanned["messages"], scanned["tools"], scanned["compactions"])
|
||||
view, redaction = _view(scanned["messages"], scanned["tools"])
|
||||
lineage = [{k: item[k] for k in ("message_id", "observations", "event_type")} for item in scanned["messages"]]
|
||||
excluded = [{k: item[k] for k in ("message_id", "observations", "event_type", "reason")} for item in scanned["excluded"]]
|
||||
result = {**view, "format": "pi-native-messages/1", "rules_version": RULES_VERSION,
|
||||
@ -653,7 +629,7 @@ def validate(source: FrozenSource, output: dict) -> dict:
|
||||
wanted_observations.append(f"{source.manifest['agent/system.md']['sha256']}:0:/system")
|
||||
supplied = output.get("lineage", []) + output.get("excluded", [])
|
||||
observed = [obs["observation_id"] for item in supplied for obs in item.get("observations", [])]
|
||||
expected_view, redaction = _view(scanned["messages"], scanned["tools"], scanned["compactions"])
|
||||
expected_view, redaction = _view(scanned["messages"], scanned["tools"])
|
||||
expected_lineage = [{k: item[k] for k in ("message_id", "observations", "event_type")} for item in scanned["messages"]]
|
||||
expected_excluded = [{k: item[k] for k in ("message_id", "observations", "event_type", "reason")} for item in scanned["excluded"]]
|
||||
counts = Counter(actual)
|
||||
@ -668,8 +644,7 @@ def validate(source: FrozenSource, output: dict) -> dict:
|
||||
"expected": len(wanted_observations), "actual": len(observed)},
|
||||
"VALID_MESSAGE_COVERAGE": {"pass": covered == len(expected) > 0 and not (set(actual) - set(expected))
|
||||
and len(actual) == len(expected), "numerator": covered, "denominator": len(expected)},
|
||||
"PAYLOAD_COVERAGE": {"pass": output.get("messages") == expected_view["messages"] and output.get("tools") == expected_view["tools"]
|
||||
and output.get("compactions") == expected_view["compactions"]},
|
||||
"PAYLOAD_COVERAGE": {"pass": output.get("messages") == expected_view["messages"] and output.get("tools") == expected_view["tools"]},
|
||||
"IDENTITY_DEDUP": {"pass": output.get("lineage") == expected_lineage and output.get("excluded") == expected_excluded
|
||||
and all(ident in native_ids for ident in direct_order)
|
||||
and not ({"IDENTITY_AMBIGUOUS", "IDENTITY_PAYLOAD_CONFLICT"} & set(scanned["errors"]))},
|
||||
|
||||
@ -10,7 +10,8 @@
|
||||
bench report [--run r…]
|
||||
bench verify <rollout_dir>
|
||||
bench stage run <stage> [--config path] (rollout 内,agent 用)
|
||||
bench stage status <handle> (只看一眼;等待用 terminal_exec wait_handle)
|
||||
bench stage wait <handle> [--poll 60]
|
||||
bench stage status <handle>
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@ -401,8 +402,10 @@ def cmd_stage(a) -> int:
|
||||
if a.op == "run":
|
||||
stagerun.submit(a.stage, a.config, recipe=a.recipe, submission=a.submission)
|
||||
return 0
|
||||
if a.op in ("wait", "status"): # wait: old name, kept so older prompts' habits still get the hint
|
||||
if a.op == "wait":
|
||||
return stagerun.wait(a.stage, a.poll)
|
||||
if a.op == "status":
|
||||
return stagerun.status(a.stage)
|
||||
raise SystemExit("stage op: run | wait | status")
|
||||
|
||||
|
||||
|
||||
@ -51,8 +51,7 @@ def preflight(policy):
|
||||
if not paths.secret(endpoint['api_key_env']):
|
||||
raise ValueError('CREDENTIAL_UNAVAILABLE: ' + alias)
|
||||
broker() # Validate the persistent shared policy before claim; never send a request.
|
||||
# 0.5 s (the default) timed out whenever the controller was busy (10-13: ~17 rollouts/hour lost).
|
||||
quota = quota_snapshot(Path(value['quota_ledger']), timeout_s=5)
|
||||
quota = quota_snapshot(Path(value['quota_ledger']))
|
||||
group = next((row for row in quota.get('groups', [])
|
||||
if row['group_id'] == value['credential_groups'][policy]), None)
|
||||
if quota['status'] != 'OBSERVED' or group is None:
|
||||
@ -207,8 +206,7 @@ def freeze_checkpoint(rd, execution):
|
||||
path = Path(checkpoint['session_path'])
|
||||
if not path.resolve().is_relative_to((rd / 'workspace/.pi-sessions').resolve()):
|
||||
raise ValueError('CHECKPOINT_PATH_INVALID')
|
||||
# A long rollout's session passes 1 MiB (10-15..10-17: 17 rollouts, 4 ended as INFRA_ERROR); no size cap here.
|
||||
data = config_bytes(rd / 'workspace', str(path), limit=None)
|
||||
data = config_bytes(rd / 'workspace', str(path))
|
||||
if hashlib.sha256(data).hexdigest() != checkpoint.get('sha256'):
|
||||
raise ValueError('CHECKPOINT_CHANGED')
|
||||
# Binary session snapshots require fsync, like JSON lifecycle records.
|
||||
@ -253,9 +251,7 @@ def budget_used(rd):
|
||||
from export.trajectory import _empty_content
|
||||
path = Path(rd) / 'agent/events.jsonl'
|
||||
if path.exists():
|
||||
with path.open(errors='replace') as stream: # events.jsonl reaches hundreds of MB; never read it whole
|
||||
lines = [line for line in stream if '"turn_end"' in line]
|
||||
for line in lines:
|
||||
for line in path.read_text().splitlines():
|
||||
try:
|
||||
event = json.loads(line)
|
||||
except ValueError:
|
||||
|
||||
@ -1,44 +0,0 @@
|
||||
"""Imported by every Python started in a Salty container (the worker puts this directory on PYTHONPATH).
|
||||
|
||||
The container sees every host CPU (244) but gets a quota of OMP_NUM_THREADS. onnxruntime sizes its default pool from
|
||||
the CPU count (125 threads) and ran ~15x slower: 10-15 probe, resnet50 on 4 CPUs, 320-500 ms per run by default vs
|
||||
22 ms at 4 threads (hmct's fixed 10 threads: ~35 ms). Cap every session at the quota, only once onnxruntime is imported.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
|
||||
def _cap(module, limit):
|
||||
original = module.InferenceSession.__init__
|
||||
|
||||
def __init__(self, path_or_bytes, sess_options=None, *args, **kwargs):
|
||||
sess_options = sess_options or module.SessionOptions()
|
||||
if not 0 < sess_options.intra_op_num_threads <= limit:
|
||||
sess_options.intra_op_num_threads = limit
|
||||
original(self, path_or_bytes, sess_options, *args, **kwargs)
|
||||
|
||||
module.InferenceSession.__init__ = __init__
|
||||
|
||||
|
||||
class _OnImport:
|
||||
def __init__(self, limit):
|
||||
self.limit = limit
|
||||
|
||||
def find_spec(self, name, path=None, target=None):
|
||||
if name != "onnxruntime":
|
||||
return None
|
||||
sys.meta_path.remove(self)
|
||||
import importlib.util
|
||||
spec = importlib.util.find_spec(name)
|
||||
if spec is not None and spec.loader is not None:
|
||||
run = spec.loader.exec_module
|
||||
|
||||
def exec_module(module):
|
||||
run(module)
|
||||
_cap(module, self.limit)
|
||||
spec.loader.exec_module = exec_module
|
||||
return spec
|
||||
|
||||
|
||||
if os.environ.get("OMP_NUM_THREADS", "").isdigit() and int(os.environ["OMP_NUM_THREADS"]) > 0:
|
||||
sys.meta_path.insert(0, _OnImport(int(os.environ["OMP_NUM_THREADS"])))
|
||||
@ -9,13 +9,12 @@ import re
|
||||
DETERMINISTIC = ["unsupported_op", "shape_infer", "export_fail", "calib_nan", "verifier_mismatch", "config_error",
|
||||
# 门失败(gates.classify_gate_failure)
|
||||
"accuracy_collapse", "accuracy_drop", "numeric_mismatch", "march_mismatch", "perf_over_budget"]
|
||||
TRANSIENT = ["oom", "board_offline", "board_busy", "disk_full", "stalled"]
|
||||
TRANSIENT = ["oom", "board_offline", "board_busy", "stalled"]
|
||||
ALL = DETERMINISTIC + TRANSIENT + ["unknown"]
|
||||
|
||||
# (error_class, compiled regex)。写规则时对着真实 stderr 写,别猜。
|
||||
RULES: list[tuple[str, re.Pattern]] = [
|
||||
("board_busy", re.compile(r'"event":\s*"board_busy"|board busy after .*refusing contended', re.I)),
|
||||
("disk_full", re.compile(r"No space left on device")), # 10-15 S600 /userdata: was "unknown", read as the agent's fault
|
||||
("config_error", re.compile(r"Key '[^']+' error:|Wrong key '|Missing key|yaml.*(invalid|error)|schema.*error", re.I)), # hb_compile yaml 校验不过:harness/agent 配置写错
|
||||
("oom", re.compile(r"CUDA out of memory|OutOfMemoryError|std::bad_alloc|Cannot allocate memory|MemoryError|Killed process .* out of memory", re.I)),
|
||||
("board_offline", re.compile(r"Connection (refused|timed out|reset by peer)|No route to host|ssh: connect to host|board .*offline|hrt_.*: failed to connect", re.I)),
|
||||
|
||||
@ -20,25 +20,6 @@ TASK_MODULES = {
|
||||
}
|
||||
|
||||
|
||||
# A frozen task module may be replaced inside a running run only by a rewrite approved here, pair by pair, as giving
|
||||
# identical per-sample outputs and metrics; anything else still needs a new run. 10-15 (f56ac2f): board calls batched
|
||||
# by measured bytes instead of one image per call; the board still infers sample by sample (board_bench.infer_inputs).
|
||||
EQUIVALENT_SOURCES = {
|
||||
("95f5b4a5682feaf08920c7beb3a0455931f7d7cab270f6ba2d298a3989025c94", "f348981b3be85712b7c3874036e1ac5b1a8f4ab625f500467f7ea62586653969"), # seg
|
||||
("3feef7cd074a9e09a0ef98c68bd8d5db09c44b0fb3f0588cfd9185c84237788a", "dca658bfc25d906d5a155d51efdfe9300d9ff7fd1430c715574d64bd61177dd3"), # mde
|
||||
("5bdd6fd229cd3cf4accb0190e74920d71fc4a5d6a0d0aced5872ca84f7983bb3", "ca570f710ecd4428d02b6b81765de51a4d3b692754c6630d1462c5bc5d1073cd"), # flow
|
||||
("210f0ea503018b17ed4cb98fe79f82a0a761a80c6308821c020ca7fdc5aab32e", "9a321c31f29a6fd73d4072ac281fd6a5d00dbf5f88f0fc15bc96c6fe092fde32"), # feat
|
||||
("599cf728e77f3d6ffc8e678457a5b703d4e0de615ead12498275f37142db24d7", "505d37ffc245dc095e7d0d4ec215ed28e4f0105d542e660c5e3ff9ba32bdd9da"), # coco_extra
|
||||
}
|
||||
|
||||
|
||||
def contract_holds(frozen: dict, current: dict) -> bool:
|
||||
"""The current module still implements the frozen contract: identical, or an approved rewrite of the same module."""
|
||||
return current == frozen or (
|
||||
{**current, "source_sha256": None} == {**frozen, "source_sha256": None}
|
||||
and (frozen.get("source_sha256"), current.get("source_sha256")) in EQUIVALENT_SOURCES)
|
||||
|
||||
|
||||
def get(task: str):
|
||||
"""One dispatch point. A task module owns inputs and both metric paths."""
|
||||
import importlib
|
||||
|
||||
@ -15,7 +15,6 @@ class BoardRunner:
|
||||
"""每次评测传一次 HBM,以小块 NPZ 传输入/输出,退出时清理临时文件。"""
|
||||
|
||||
CHUNK = 32
|
||||
MAX_BYTES = 1 << 30 # inputs + outputs per board call (10-15: one 32-sample solo.r50 call was 12 GB, filled an S600)
|
||||
|
||||
def __init__(self, hbm: str, runtime_dir: str = "/userdata/bench"):
|
||||
self.hbm = str(hbm)
|
||||
@ -32,12 +31,6 @@ class BoardRunner:
|
||||
self._session: RemoteSession | None = None
|
||||
self._closed = False
|
||||
self._samples = 0
|
||||
self._sample_bytes = 0
|
||||
|
||||
def group_size(self) -> int:
|
||||
"""Samples per board call: one until a sample's inputs + outputs are measured, then as many as fit
|
||||
MAX_BYTES. Evaluators group by it too, so one group's outputs also fit in memory."""
|
||||
return max(1, min(self.CHUNK, self.MAX_BYTES // self._sample_bytes)) if self._sample_bytes else 1
|
||||
|
||||
def __enter__(self):
|
||||
if self._closed:
|
||||
@ -59,10 +52,8 @@ class BoardRunner:
|
||||
results: list[list[np.ndarray]] = []
|
||||
fin, fout = self.tmp / "inputs.npz", self.tmp / "outputs.npz"
|
||||
try:
|
||||
start = 0
|
||||
while start < len(xs):
|
||||
chunk = xs[start:start + self.group_size()]
|
||||
start += len(chunk)
|
||||
for start in range(0, len(xs), self.CHUNK):
|
||||
chunk = xs[start:start + self.CHUNK]
|
||||
for x in chunk:
|
||||
if x.dtype != np.float32:
|
||||
raise ValueError(f"board evaluation requires float32 inputs, found {x.dtype}")
|
||||
@ -74,7 +65,6 @@ class BoardRunner:
|
||||
if info.get("samples") != len(chunk):
|
||||
raise RuntimeError("board returned an unexpected sample count")
|
||||
session.download("outputs.npz", fout)
|
||||
self._sample_bytes = max(self._sample_bytes, (fin.stat().st_size + fout.stat().st_size) // len(chunk))
|
||||
output_count = len(info.get("outputs", []))
|
||||
if output_count == 0:
|
||||
raise RuntimeError("board returned no model outputs")
|
||||
|
||||
@ -8,7 +8,7 @@ from __future__ import annotations
|
||||
import json
|
||||
import numpy as np
|
||||
|
||||
from .common import groups, letterbox, run_batched
|
||||
from .common import letterbox, run_batched
|
||||
from .det import CONF, IOU, MAX_DET, _nms
|
||||
|
||||
METRICS = {"iseg": "mask_ap", "pose": "oks_ap"}
|
||||
@ -66,7 +66,8 @@ def predict(runner, t1, input_spec, task, pose_evidence=None, model=""):
|
||||
from pycocotools import mask as mask_utils
|
||||
size = int(input_spec.split("x")[-1])
|
||||
predictions = []
|
||||
for items in groups(runner, t1.items, 16):
|
||||
for offset in range(0, len(t1.items), 16):
|
||||
items = t1.items[offset:offset + 16]
|
||||
images = [t1.image(it) for it in items]
|
||||
pre = [letterbox(im, size) for im in images]
|
||||
outputs = run_batched(runner, [v[0] for v in pre])
|
||||
|
||||
@ -163,16 +163,6 @@ def run_batched(runner, xs: list[np.ndarray]) -> list[list[np.ndarray]]:
|
||||
return [runner(x) for x in xs]
|
||||
|
||||
|
||||
def groups(runner, items: list, size: int = 1):
|
||||
"""评测器每次交给 run_batched 的样本组:板卡 runner 按实测字节定组大小(一组输出也放得进内存),
|
||||
其他 runner 用 size。10-15 前分割/深度/光流/特征逐张上板,每张约 10 秒连接开销。"""
|
||||
start = 0
|
||||
while start < len(items):
|
||||
n = runner.group_size() if hasattr(runner, "group_size") else size
|
||||
yield items[start:start + n]
|
||||
start += n
|
||||
|
||||
|
||||
class RecipeRunner:
|
||||
"""A data-only proxy: model callbacks never enter the scoring process."""
|
||||
|
||||
|
||||
@ -97,15 +97,13 @@ def auc(errors, thresholds=(1, 3, 5)):
|
||||
|
||||
def evaluate(runner, t1, input_spec, model):
|
||||
if t1.manifest.get('dataset') != 'hpatches': raise ValueError('HPatches T1 required')
|
||||
from .common import groups, run_batched
|
||||
cache = {}; sizes = {}; samples = []
|
||||
for names in groups(runner, list(dict.fromkeys(item[f] for item in t1.items for f in ('file', 'file2')))):
|
||||
images = [t1.image({'file': name}) for name in names]
|
||||
outs = run_batched(runner, [common.preprocess(image, input_spec, model) for image in images])
|
||||
for name, image, out in zip(names, images, outs, strict=True):
|
||||
sizes[name] = image.size
|
||||
cache[name] = common.decode(model, out)
|
||||
for item in t1.items:
|
||||
for field in ('file', 'file2'):
|
||||
name = item[field]
|
||||
if name not in cache:
|
||||
image = t1.image({'file': name}); sizes[name] = image.size
|
||||
cache[name] = common.decode(model, runner(common.preprocess(image, input_spec, model)))
|
||||
samples.append({'id': item['id'], **pair_metrics(cache[item['file']], cache[item['file2']],
|
||||
item['H_0to1'], sizes[item['file']], sizes[item['file2']])})
|
||||
values = auc([s['corner_error_px'] for s in samples])
|
||||
|
||||
@ -101,13 +101,12 @@ def aggregate(samples: list[dict]) -> dict:
|
||||
|
||||
|
||||
def evaluate(runner, t1, input_spec: str, model: str) -> tuple[dict, list[dict]]:
|
||||
from .common import groups, run_batched
|
||||
samples = []
|
||||
for items in groups(runner, t1.items):
|
||||
for item, out in zip(items, run_batched(runner, [inputs(t1, item, input_spec, model) for item in items]), strict=True):
|
||||
truth, valid = ground_truth(t1, item)
|
||||
restored = restore_flow(out, input_spec, truth.shape[-2:])
|
||||
samples.append({"id": item["id"], **sample_metrics(restored, truth, valid)})
|
||||
for item in t1.items:
|
||||
truth, valid = ground_truth(t1, item)
|
||||
out = runner(inputs(t1, item, input_spec, model))
|
||||
restored = restore_flow(out, input_spec, truth.shape[-2:])
|
||||
samples.append({"id": item["id"], **sample_metrics(restored, truth, valid)})
|
||||
return aggregate(samples), samples
|
||||
|
||||
|
||||
|
||||
@ -152,13 +152,11 @@ def evaluate(runner, t1, input_spec: str, model: str) -> tuple[dict, list[dict]]
|
||||
raise ValueError("MDE evaluation input recipe mismatch")
|
||||
if t1.manifest.get("evaluation_protocol") != PROTOCOL:
|
||||
raise ValueError("T1 must freeze the explicit relative-depth evaluation protocol")
|
||||
from .common import groups, run_batched
|
||||
samples = []
|
||||
for items in groups(runner, t1.items):
|
||||
for item, out in zip(items, run_batched(runner, [inputs(t1, item, input_spec, model) for item in items]), strict=True):
|
||||
depth, mask = ground_truth(t1, item)
|
||||
prediction = restore_prediction(out, depth.shape, model)
|
||||
samples.append({"id": item["id"], **aligned_metrics(prediction, depth, mask)})
|
||||
for item in t1.items:
|
||||
depth, mask = ground_truth(t1, item)
|
||||
prediction = restore_prediction(runner(inputs(t1, item, input_spec, model)), depth.shape, model)
|
||||
samples.append({"id": item["id"], **aligned_metrics(prediction, depth, mask)})
|
||||
if not samples:
|
||||
raise ValueError("empty depth evaluation")
|
||||
metrics = {key: float(np.mean([item[key] for item in samples])) for key in ("absrel", "rmse", "delta1")}
|
||||
|
||||
@ -177,17 +177,15 @@ def numeric_diagnostic(runner, bdir: Path, calib_source: str) -> dict:
|
||||
prepared = runner.prepare([images[key] for key in keys])
|
||||
else:
|
||||
prepared = [xs[k] for k in keys]
|
||||
from .common import groups
|
||||
for group in groups(runner, list(range(len(keys))), len(keys)):
|
||||
results = run_batched(runner, [prepared[j] for j in group])
|
||||
if len(results) != len(group):
|
||||
raise ValueError("G2 runner changed sample count")
|
||||
for j, outs in zip(group, results):
|
||||
for i, o in enumerate(outs):
|
||||
ref_key = f"{keys[j]}/out{i}"
|
||||
if ref_key not in fp or fp[ref_key].shape != o.shape:
|
||||
return {"cos_min": None, "reason": "G2 output shapes differ"}
|
||||
per_out.setdefault(f"out{i}", []).append(cosine(fp[ref_key], o))
|
||||
results = run_batched(runner, prepared)
|
||||
if len(results) != len(keys):
|
||||
raise ValueError("G2 runner changed sample count")
|
||||
for k, outs in zip(keys, results):
|
||||
for i, o in enumerate(outs):
|
||||
ref_key = f"{k}/out{i}"
|
||||
if ref_key not in fp or fp[ref_key].shape != o.shape:
|
||||
return {"cos_min": None, "reason": "G2 output shapes differ"}
|
||||
per_out.setdefault(f"out{i}", []).append(cosine(fp[ref_key], o))
|
||||
cos_per_output = {k: float(min(v)) for k, v in per_out.items()}
|
||||
return {"cos_min": min(cos_per_output.values()) if cos_per_output else None,
|
||||
"cos_per_output": cos_per_output, "n": len(keys), "source": calib_source}
|
||||
|
||||
@ -126,9 +126,10 @@ def aggregate(samples: list[dict]) -> dict:
|
||||
def run_fp32(runner, t1, input_spec: str, out: dict, resize: int = 0) -> None:
|
||||
adapter, dataset, classes = _recipe(t1, out["model"], runner)
|
||||
samples = []
|
||||
from .common import groups, run_batched
|
||||
from .common import run_batched
|
||||
size = getattr(runner, "CHUNK", 16) if hasattr(runner, "recipe_adapter") else 1
|
||||
for items in groups(runner, t1.items, size):
|
||||
for start in range(0, len(t1.items), size):
|
||||
items = t1.items[start:start + size]
|
||||
xs = (runner.prepare([t1.image(item) for item in items]) if hasattr(runner, "recipe_adapter")
|
||||
else [inputs(t1, item, input_spec, out["model"]) for item in items])
|
||||
outputs = run_batched(runner, xs)
|
||||
|
||||
@ -218,16 +218,6 @@ def resume_policy(state):
|
||||
raise ExecutionStopped('RESUME_TOOL_POLICY_MISSING')
|
||||
|
||||
|
||||
def paused_credit(state):
|
||||
"""Time paused for infrastructure (quota, network) is not the agent's: it is handed back on resume, at most one
|
||||
execution budget in total so a paused rollout that can never resume still ends. 10-14: three rollouts paused
|
||||
7–16 h by GLM outages resumed with < 1 h left and were cut off mid-compile or mid-eval."""
|
||||
paused = state.get('paused_at') if state['execution_state'] == 'PAUSED_INFRA' else None
|
||||
if not paused:
|
||||
return 0.0
|
||||
return max(0.0, min(time.time() - paused, state['config']['execution_deadline_s'] - state.get('paused_s', 0.0)))
|
||||
|
||||
|
||||
def current():
|
||||
return getattr(_local, 'execution', None)
|
||||
|
||||
@ -309,7 +299,7 @@ class Execution:
|
||||
return state.get('stop_reason') or state['execution_state']
|
||||
if state.get('boot_id') != boot_id():
|
||||
return 'BOOT_CHANGED_RECONCILE_REQUIRED'
|
||||
if time.monotonic() >= state['deadline_monotonic'] + paused_credit(state):
|
||||
if time.monotonic() >= state['deadline_monotonic']:
|
||||
return 'EXECUTION_DEADLINE_EXCEEDED'
|
||||
owner_status = process_status(state['owner'])
|
||||
if owner_status == 'DEAD':
|
||||
@ -568,7 +558,7 @@ class Execution:
|
||||
checkpoint=checkpoint, paused_at=time.time())
|
||||
return self._save(state, 'EXECUTION_PAUSED', reason=reason)
|
||||
|
||||
def resume(self, *, automatic=True, counted=True):
|
||||
def resume(self, *, automatic=True):
|
||||
with self.locked():
|
||||
state = self.snapshot()
|
||||
if self.epoch is not None and self.epoch != state['epoch']:
|
||||
@ -580,10 +570,6 @@ class Execution:
|
||||
raise ExecutionStopped('SIDE_EFFECT_UNKNOWN')
|
||||
if state.get('boot_id') != boot_id():
|
||||
raise ExecutionStopped('BOOT_CHANGED_RECONCILE_REQUIRED')
|
||||
credit = paused_credit(state)
|
||||
state['deadline_monotonic'] += credit
|
||||
state['paused_s'] = state.get('paused_s', 0.0) + credit
|
||||
state.pop('paused_at', None)
|
||||
if time.monotonic() >= state['deadline_monotonic']:
|
||||
raise ExecutionStopped('EXECUTION_DEADLINE_EXCEEDED')
|
||||
self._checkpoint(state.get('checkpoint'))
|
||||
@ -596,12 +582,12 @@ class Execution:
|
||||
if members != [] or pending:
|
||||
raise ExecutionStopped('SIDE_EFFECT_UNKNOWN' if members is None else 'WORK_STILL_ACTIVE')
|
||||
count = state.get('auto_resume_count', 0)
|
||||
if automatic and counted and count >= state['config'].get('max_auto_resumes', 3):
|
||||
if automatic and count >= state['config'].get('max_auto_resumes', 3):
|
||||
raise ExecutionStopped('AUTO_RESUME_BUDGET_EXHAUSTED')
|
||||
state['epoch'] += 1
|
||||
self.epoch = state['epoch']
|
||||
state.update(owner=process_identity(os.getpid()), execution_state='ACTIVE', stop_reason=None,
|
||||
auto_resume_count=count + int(automatic and counted), resumed_at=time.time(), activity='WAITING_QUOTA')
|
||||
auto_resume_count=count + int(automatic), resumed_at=time.time(), activity='WAITING_QUOTA')
|
||||
return self._save(state, 'EXECUTION_RESUMED', automatic=automatic)
|
||||
|
||||
def cleanup(self):
|
||||
|
||||
@ -364,9 +364,6 @@ def _check_frozen_job(path: Path, requested: dict) -> None:
|
||||
if "toolchain" not in old and tc.get("profile") is None and tc.get("version") == "legacy-config":
|
||||
new.pop("toolchain", None)
|
||||
contract = new.get("task_contract") or {}
|
||||
from .eval import contract_holds
|
||||
if old.get("task_contract") and contract_holds(old["task_contract"], contract):
|
||||
new["task_contract"] = contract = old["task_contract"] # an approved rewrite keeps the frozen contract
|
||||
legacy_modules = {task: f"harness.eval.{task}" for task in ("cls", "det", "flow", "seg", "mde", "feat")}
|
||||
legacy_modules.update(pose="harness.eval.coco_extra", iseg="harness.eval.coco_extra", stereo=None, occ=None)
|
||||
if "task_contract" not in old and new.get("task") in legacy_modules and contract.get("module") == legacy_modules[new["task"]]:
|
||||
@ -563,7 +560,7 @@ def _resume_snapshot(run_id, job_ids=None, modes=None):
|
||||
cancel=bool(state.get('cancel_requested')), timeout_s=0)
|
||||
continue
|
||||
if (state['execution_state'] != 'PAUSED_INFRA' or state['cleanup_state'] == 'QUARANTINED'
|
||||
or state.get('cancel_requested')):
|
||||
or state.get('cancel_requested') or state.get('auto_resume_count', 0) >= state['config'].get('max_auto_resumes', 3)):
|
||||
continue
|
||||
try:
|
||||
resume_policy(state)
|
||||
@ -572,19 +569,11 @@ def _resume_snapshot(run_id, job_ids=None, modes=None):
|
||||
readiness = broker().resume_readiness(state['execution_id'])
|
||||
if readiness['eligible']:
|
||||
ready.append({**row, '_resume': True, 'runner': 'pi', 'job_state': 'RESUME'})
|
||||
elif (readiness.get('reason') in ('REQUEST_OUTCOME_UNKNOWN', 'QUOTA_SOURCE_UNAVAILABLE') # ledger busy: unknown, keep the slot
|
||||
or (readiness.get('quota_group') or {}).get('state') in ('PAUSED', 'RECONCILING') # any cause, e.g. 10-15 00:56 connection cut
|
||||
or readiness.get('not_before') is not None and readiness.get('reason') in ('COOLING', 'QUOTA_UNAVAILABLE', 'WAITING_QUOTA')):
|
||||
elif readiness.get('reason') == 'REQUEST_OUTCOME_UNKNOWN' or readiness.get('not_before') is not None and readiness.get('reason') in ('COOLING', 'QUOTA_UNAVAILABLE', 'WAITING_QUOTA'):
|
||||
pending = True
|
||||
return ready, pending
|
||||
|
||||
|
||||
def _claimable(resumes, resume_pending, queued):
|
||||
"""有暂停的轨迹在等(已就绪或只差额度)时,这一轮只续跑、不领新题,空出的名额留给它们;否则它们会一直饿到 24h 过期。
|
||||
10-15 11:21、11:33:一条续上的同一轮里新题跟着进来,把还在等额度的几条的名额占了。"""
|
||||
return resumes if (resumes or resume_pending) else queued
|
||||
|
||||
|
||||
def collect_rollouts(run_id: str, job_ids: list[str] | None = None, modes: list[str] | None = None,
|
||||
parallel: int | None = None, dry_run: bool = False, limit: int | None = None,
|
||||
log=_stamped, coverage: str = "matrix", watch: bool = False,
|
||||
@ -820,7 +809,7 @@ def collect_rollouts(run_id: str, job_ids: list[str] | None = None, modes: list[
|
||||
# A fully paused group has no request left to drive its own recovery.
|
||||
collection_runtime._broker.reconcile()
|
||||
resumes, resume_pending = _resume_snapshot(run_id, job_ids, modes)
|
||||
ready = _claimable(resumes, resume_pending, ready)
|
||||
ready = resumes + ready
|
||||
pending = pending or resume_pending
|
||||
active_jobs = {r["job_id"] for r in futures.values()}
|
||||
free = n - len(futures)
|
||||
@ -1122,11 +1111,6 @@ def run_oracle_job(run_id: str, job_id: str, dry_run: bool = False, log=_stamped
|
||||
for S in (ORDER[:3] if prepare_only else ORDER):
|
||||
stage = S()
|
||||
attempt, lane = 0, "normal"
|
||||
from . import salty_stage
|
||||
if salty_stage.selected(ctx, stage.name):
|
||||
pending = salty_stage.pending_attempt(ctx.stage_root)
|
||||
if pending is not None:
|
||||
attempt = pending
|
||||
while True:
|
||||
ctx.attempt, ctx.lane = attempt, lane
|
||||
with db.connect() as con:
|
||||
@ -1140,8 +1124,6 @@ def run_oracle_job(run_id: str, job_id: str, dry_run: bool = False, log=_stamped
|
||||
with scheduler.acquire_stage(ctx, stage.pool) as slot:
|
||||
ctx.extra["slot"] = slot
|
||||
out = stage.run(ctx)
|
||||
if out.error_class == "SIDE_EFFECT_UNKNOWN" or out.meta.get("cleanup_state") in {"QUARANTINED", "PENDING", "UNKNOWN"}:
|
||||
out.state, out.error_class = "FAIL", "SIDE_EFFECT_UNKNOWN"
|
||||
with db.connect() as con:
|
||||
db.stage_finish(con, run_id, job_id, stage.name, attempt, out.state, out.exit_code, out.error_class, -1)
|
||||
log(f"[{job_id}] {stage.name} a{attempt:02d} {lane}: {out.state} ec={out.error_class} {out.duration_s:.1f}s")
|
||||
@ -1156,15 +1138,6 @@ def run_oracle_job(run_id: str, job_id: str, dry_run: bool = False, log=_stamped
|
||||
if calib_meta:
|
||||
ctx.extra["calib_source"] = calib_meta.get("calib_source")
|
||||
break
|
||||
# Unknown side effects cannot be converted into a fresh attempt,
|
||||
# even when a lower layer returned the legacy STALLED state.
|
||||
if out.error_class == "SIDE_EFFECT_UNKNOWN" or out.meta.get("cleanup_state") in {"QUARANTINED", "PENDING", "UNKNOWN"}:
|
||||
out.state, out.error_class = "FAIL", "SIDE_EFFECT_UNKNOWN"
|
||||
rec.update(state=out.state, error_class=out.error_class)
|
||||
from . import execution
|
||||
if execution.current():
|
||||
execution.current().quarantine("SIDE_EFFECT_UNKNOWN")
|
||||
break
|
||||
# 失败:决定重试
|
||||
if out.state == "STALLED" and attempt < cfg["retry"]["stalled"]:
|
||||
attempt, lane = attempt + 1, "long"; continue
|
||||
@ -1173,12 +1146,6 @@ def run_oracle_job(run_id: str, job_id: str, dry_run: bool = False, log=_stamped
|
||||
break
|
||||
if out.state not in ("OK", "SKIPPED"):
|
||||
ec = out.error_class or ("stalled" if out.state == "STALLED" else "unknown")
|
||||
if ec == "SIDE_EFFECT_UNKNOWN" or out.meta.get("cleanup_state") in {"QUARANTINED", "PENDING", "UNKNOWN"}:
|
||||
# A board's ordinary failed measurement may be nonblocking.
|
||||
# Unknown cleanup/remote effects must stop the entire pipeline.
|
||||
terminal, error_class, stage_failed = "UNVERIFIED", "SIDE_EFFECT_UNKNOWN", stage.name
|
||||
result["cleanup_state"] = "QUARANTINED"
|
||||
break
|
||||
if stage.name == "07-board":
|
||||
result["measurements"]["G4"] = None # unverified,不阻塞
|
||||
result["stages"][stage.name]["note"] = "board failed after retries → G4 unverified"
|
||||
|
||||
119
harness/quota.py
119
harness/quota.py
@ -20,31 +20,15 @@ import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import secrets
|
||||
import socket
|
||||
import sqlite3
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import traceback
|
||||
from urllib.parse import urlsplit, quote
|
||||
import uuid
|
||||
|
||||
|
||||
# Failures proved to have sent nothing: wait and resend inside the broker instead of pausing the rollout.
|
||||
RETRY_UNSENT = ('CONNECT_FAILED', 'DNS_FAILED', 'DNS_TIMEOUT')
|
||||
# A 400 saying the request outgrew the model's window (vLLM "maximum context length", GLM "exceeds max length"):
|
||||
# the agent used up its context, not an infrastructure fault. The generation fails (it fences nothing) and Pi gets a
|
||||
# 400 its isContextOverflow() recognizes, so Pi compacts and continues; same request again would fail, never resumed.
|
||||
CONTEXT_OVERFLOW_BODY = b'{"error":{"code":"context_length_exceeded","message":"context_length_exceeded"}}'
|
||||
CONTEXT_FULL = re.compile(rb'maximum context|context length|context window|exceeds? (the )?max(imum)? length|prompt is too long', re.I)
|
||||
# How long a request whose connection dropped holds its account (it used to hold until the request's own
|
||||
# deadline, ~20-27 min: 10-16 14:35 one drop paused every 05 GLM and 04 Qwen rollout). Most servers stop
|
||||
# generating when the client goes away. ponytail: fixed for every group, a policy field when one needs longer.
|
||||
UNKNOWN_HOLD_S = 120
|
||||
|
||||
|
||||
class QuotaError(Exception):
|
||||
def __init__(self, reason: str, status: int = 503):
|
||||
self.reason, self.status = reason, status
|
||||
@ -504,13 +488,7 @@ class Broker:
|
||||
'message':exc.reason}}).encode()
|
||||
except (ValueError, socket.timeout):
|
||||
status, content_type, payload = 400, 'application/json', b'{"error":{"code":"INVALID_LOCAL_REQUEST"}}'
|
||||
except Exception as exc:
|
||||
# 10-16 05 hrnet.w32: swallowed with no trace and the rollout ended. Type, message and place only
|
||||
# (never the request); a generation that sent nothing resumes like any other unsent one.
|
||||
where = traceback.extract_tb(exc.__traceback__)[-1:]
|
||||
print(f'BROKER_UNAVAILABLE {type(exc).__name__}: {str(exc)[:200]} at '
|
||||
f'{[(f.filename.rsplit("/", 1)[-1], f.lineno) for f in where]}', file=sys.stderr, flush=True)
|
||||
broker.pause_unsent(self.headers.get('X-Bench-Generation-Id'))
|
||||
except Exception:
|
||||
status, content_type, payload = 503, 'application/json', b'{"error":{"code":"BROKER_UNAVAILABLE"}}'
|
||||
try:
|
||||
self.send_response(status)
|
||||
@ -582,27 +560,6 @@ class Broker:
|
||||
if owner == execution_id:
|
||||
self._interrupt(conn)
|
||||
|
||||
def pause_unsent(self, generation):
|
||||
"""After an internal broker error: a generation none of whose attempts may have reached the provider is
|
||||
paused as BROKER_ERROR, which resumes like a connect failure. Anything possibly sent is left as it is."""
|
||||
if not generation:
|
||||
return
|
||||
try:
|
||||
with self._db() as con:
|
||||
row = con.execute('SELECT e.group_id FROM generations g JOIN executions e ON e.execution_id=g.execution_id '
|
||||
'WHERE g.generation_id=?', (generation,)).fetchone()
|
||||
if not row:
|
||||
return
|
||||
with self._group_lock(row['group_id']), self._db() as con:
|
||||
con.execute('BEGIN IMMEDIATE')
|
||||
state = con.execute('SELECT state FROM generations WHERE generation_id=?', (generation,)).fetchone()['state']
|
||||
maybe_sent = con.execute('SELECT 1 FROM attempts WHERE generation_id=? AND (sent IS NOT 0 OR ended_at IS NULL)',
|
||||
(generation,)).fetchone()
|
||||
if state in ('QUEUED', 'RESUMABLE') and not maybe_sent:
|
||||
self._pause(con, generation, 'BROKER_ERROR')
|
||||
except sqlite3.Error:
|
||||
pass # the rollout then ends as before
|
||||
|
||||
def reconcile(self):
|
||||
"""Unknown transport end is a quarantine, not a lease-based refund.
|
||||
|
||||
@ -627,7 +584,7 @@ class Broker:
|
||||
self._event(con, group, 'UNKNOWN_SETTLED_PROBE_ALLOWED')
|
||||
for attempt_id in stale: # outside the group flock: abandon_unknown takes it itself
|
||||
try:
|
||||
self.abandon_unknown(attempt_id, operator_reason='AUTO: local deadline + token window elapsed', resumable=True)
|
||||
self.abandon_unknown(attempt_id, operator_reason='AUTO: local deadline + token window elapsed')
|
||||
except QuotaError:
|
||||
pass # another broker settled it first
|
||||
|
||||
@ -643,14 +600,11 @@ class Broker:
|
||||
con.execute("UPDATE quota_groups SET state='COOLING',epoch=epoch+1,target=1,reason=NULL,cooldown_until=MAX(cooldown_until,?),updated_at=? WHERE group_id=?", (now, now, group))
|
||||
return True
|
||||
|
||||
def abandon_unknown(self, attempt_id: str, *, operator_reason: str, resumable: bool = False):
|
||||
def abandon_unknown(self, attempt_id: str, *, operator_reason: str):
|
||||
"""Explicit operator release of local capacity, never provider-end proof.
|
||||
|
||||
The attempt stays unknown for audit (usage and ended_at untouched). An operator
|
||||
release fences the generation forever. reconcile() calls this with resumable=True
|
||||
once a token window has passed: the provider can no longer charge that window, so
|
||||
the turn may be replayed (one request's tokens, counted against the resume budget)
|
||||
instead of losing the rollout (10-13/14: 16 rollouts lost this way).
|
||||
The old generation is fenced forever. Original usage and ended_at stay
|
||||
untouched. reconcile() also calls this once a token window has passed.
|
||||
"""
|
||||
if not isinstance(operator_reason, str) or not operator_reason.strip():
|
||||
raise ValueError('explicit operator reason required')
|
||||
@ -671,19 +625,15 @@ class Broker:
|
||||
execution_id = con.execute('SELECT execution_id FROM generations WHERE generation_id=?',
|
||||
(generation,)).fetchone()[0]
|
||||
con.execute("UPDATE attempts SET state='ABANDONED_UNKNOWN' WHERE attempt_id=?", (attempt_id,))
|
||||
if resumable:
|
||||
con.execute("UPDATE generations SET state='PAUSED',reason='REQUEST_OUTCOME_UNKNOWN',updated_at=? WHERE generation_id=?", (now, generation))
|
||||
else:
|
||||
con.execute("UPDATE generations SET state='ABANDONED_UNKNOWN',reason='OPERATOR_ABANDONED_UNKNOWN',updated_at=? WHERE generation_id=?", (now, generation))
|
||||
con.execute('UPDATE executions SET cancelled=1 WHERE execution_id=?', (execution_id,))
|
||||
con.execute("UPDATE generations SET state='ABANDONED_UNKNOWN',reason='OPERATOR_ABANDONED_UNKNOWN',updated_at=? WHERE generation_id=?", (now, generation))
|
||||
con.execute('UPDATE executions SET cancelled=1 WHERE execution_id=?', (execution_id,))
|
||||
probe_allowed = self._probe_if_settled(con, group)
|
||||
self._event(con, group, 'OPERATOR_ABANDONED_UNKNOWN', generation,
|
||||
attempt_id=attempt_id, operator_reason=operator_reason,
|
||||
released_at=now, provider_end_confirmed=False,
|
||||
original_reserved_tokens=row['reserved_tokens'],
|
||||
original_charged_tokens=row['charged_tokens'], probe_allowed=bool(probe_allowed), resumable=resumable)
|
||||
if not resumable:
|
||||
self._cancel_requested.setdefault(execution_id, threading.Event()).set()
|
||||
original_charged_tokens=row['charged_tokens'], probe_allowed=bool(probe_allowed))
|
||||
self._cancel_requested.setdefault(execution_id, threading.Event()).set()
|
||||
return True
|
||||
|
||||
def confirm_end(self, attempt_id: str, receipt_id: str):
|
||||
@ -718,19 +668,13 @@ class Broker:
|
||||
if not gen:
|
||||
return result | {'reason':'NO_PAUSED_GENERATION'}
|
||||
result['generation_id']=gen['generation_id']
|
||||
# Only a resume after a possibly delivered request spends the budget; connect/DNS failures and
|
||||
# waiting behind another request's unknown end sent nothing (10-14: 4 rollouts stuck at 3/3 on those).
|
||||
last=con.execute('SELECT sent FROM attempts WHERE generation_id=? ORDER BY started_at DESC LIMIT 1',(gen['generation_id'],)).fetchone()
|
||||
result['counted']=last is not None and last['sent'] != 0
|
||||
policy=self.groups[ex['group_id']]
|
||||
if (ex['cancelled'] or gen['state']!='PAUSED'
|
||||
# CONNECT_FAILED / REQUEST_DEADLINE_BEFORE_SEND / DNS_*: nothing was sent, replay is safe.
|
||||
# INCOMPLETE_GENERATION: a cut-off reply whose end the ledger recorded (the open-attempt check below still applies).
|
||||
or gen['reason'] not in ('QUOTA_WAIT_BUDGET_EXHAUSTED','REQUEST_OUTCOME_UNKNOWN','CONNECT_FAILED','REQUEST_DEADLINE_BEFORE_SEND','INCOMPLETE_GENERATION','DNS_FAILED','DNS_TIMEOUT','BROKER_ERROR')
|
||||
or gen['reason'] not in ('QUOTA_WAIT_BUDGET_EXHAUSTED','REQUEST_OUTCOME_UNKNOWN')
|
||||
# A generation whose own request end is unknown never replays.
|
||||
or con.execute("SELECT 1 FROM attempts WHERE generation_id=? AND ended_at IS NULL AND state <> 'ABANDONED_UNKNOWN'",(gen['generation_id'],)).fetchone()):
|
||||
or con.execute('SELECT 1 FROM attempts WHERE generation_id=? AND ended_at IS NULL',(gen['generation_id'],)).fetchone()):
|
||||
return result | {'reason':'RESUME_NOT_ELIGIBLE'}
|
||||
if (result['counted'] and ex['resumes']>=policy.max_auto_resumes) or gen['attempts']>=policy.max_attempts:
|
||||
if ex['resumes']>=policy.max_auto_resumes or gen['attempts']>=policy.max_attempts:
|
||||
return result | {'reason':'RESUME_BUDGET_EXHAUSTED'}
|
||||
if gen['reserved_tokens']<=0:
|
||||
return result | {'reason':'RESERVATION_METADATA_MISSING'}
|
||||
@ -756,7 +700,7 @@ class Broker:
|
||||
def resume_readiness(self, execution_id: str):
|
||||
"""Bounded read-only hint, never a promise of future HTTP admission."""
|
||||
with self._db() as con:
|
||||
deadline=time.monotonic()+5 # .5 s failed under load; paused rollouts bounced back for hours
|
||||
deadline=time.monotonic()+.5
|
||||
con.set_progress_handler(lambda:int(time.monotonic()>=deadline),1000)
|
||||
try:
|
||||
return self._resume_readiness(con,execution_id,self._now())
|
||||
@ -776,7 +720,7 @@ class Broker:
|
||||
readiness=self._resume_readiness(con,execution_id,self._now())
|
||||
if not readiness['eligible'] or readiness['generation_id']!=generation_id:
|
||||
raise QuotaError(readiness['reason'] if not readiness['eligible'] else 'GENERATION_MISMATCH',409)
|
||||
con.execute('UPDATE executions SET resumes=resumes+? WHERE execution_id=?',(int(readiness['counted']),execution_id))
|
||||
con.execute('UPDATE executions SET resumes=resumes+1 WHERE execution_id=?',(execution_id,))
|
||||
con.execute("UPDATE generations SET state='RESUMABLE',wait_since=? WHERE generation_id=?",(self._now(),generation_id))
|
||||
|
||||
def _begin(self, token, body, requested_id):
|
||||
@ -969,8 +913,6 @@ class Broker:
|
||||
('FINISHED' if known_end else 'UNKNOWN',now if known_end else None,status,reason,json.dumps(_allowed_headers(headers)),usage if usage is not None else row['reserved_tokens'],'REPORTED' if usage is not None else 'ESTIMATED',attempt))
|
||||
con.execute('UPDATE attempts SET transport_reason=? WHERE attempt_id=?',(transport_reason,attempt))
|
||||
if not known_end:
|
||||
if transport_reason=='REQUEST_CONNECTION_INTERRUPTED':
|
||||
con.execute('UPDATE attempts SET deadline=MIN(deadline,?) WHERE attempt_id=?',(now+UNKNOWN_HOLD_S,attempt))
|
||||
self._pause(con,generation,'CANCELLED' if ex['cancelled'] else reason)
|
||||
con.execute("UPDATE quota_groups SET state='RECONCILING',reason='REQUEST_OUTCOME_UNKNOWN',updated_at=? WHERE group_id=? AND NOT (state='PAUSED' AND reason IS NOT NULL AND reason!='REQUEST_OUTCOME_UNKNOWN')",(now,group))
|
||||
elif usage is not None and usage>row['reserved_tokens']:
|
||||
@ -992,8 +934,6 @@ class Broker:
|
||||
con.execute('UPDATE quota_groups SET state=?,target=?,recovery_at=?,successes=0,failures=0,updated_at=? WHERE group_id=?',('OPEN' if target==policy.max_concurrency else 'RECOVERING',target,now,now,group))
|
||||
else:
|
||||
con.execute('UPDATE quota_groups SET successes=? WHERE group_id=?',(successes,group))
|
||||
elif reason=='CONTEXT_TOO_LONG':
|
||||
con.execute("UPDATE generations SET state='FAILED',reason=?,updated_at=? WHERE generation_id=?",(reason,now,generation))
|
||||
elif reason in ('RATE_LIMIT','UPSTREAM_TRANSIENT'):
|
||||
if reason!='RATE_LIMIT' or row['http_status']!=429:
|
||||
self._cool(con,group,policy,headers,row['epoch'])
|
||||
@ -1102,37 +1042,12 @@ class Broker:
|
||||
watcher = threading.Thread(target=watch_client, daemon=True)
|
||||
watcher.start()
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
return self._generate(scope, upstream, policy, generation, data, body, reserve, client_connected)
|
||||
except QuotaError as exc:
|
||||
if exc.reason not in RETRY_UNSENT or not self._wait_unsent(upstream.quota_group, generation, policy):
|
||||
raise
|
||||
return self._generate(scope, upstream, policy, generation, data, body, reserve, client_connected)
|
||||
finally:
|
||||
stopped.set()
|
||||
if watcher is not None:
|
||||
watcher.join()
|
||||
|
||||
def _wait_unsent(self, group, generation, policy):
|
||||
"""Nothing reached the provider: wait here and resend rather than pause the whole rollout.
|
||||
|
||||
Each retry is charged to the execution's queue budget (the wait plus one
|
||||
connect/DNS deadline), so the client timeout bound holds; once the budget
|
||||
is spent the generation stays paused as before.
|
||||
"""
|
||||
if self._closing.wait(policy.cooldown_initial_s):
|
||||
return False
|
||||
charge = policy.cooldown_initial_s + policy.connect_timeout_s
|
||||
with self._group_lock(group), self._db() as con:
|
||||
con.execute('BEGIN IMMEDIATE')
|
||||
row = con.execute('SELECT g.state,g.execution_id,e.wait_spent,e.cancelled FROM generations g JOIN executions e ON e.execution_id=g.execution_id WHERE g.generation_id=?', (generation,)).fetchone()
|
||||
if row['state'] != 'PAUSED' or row['cancelled'] or row['wait_spent'] + charge > policy.wait_budget_s:
|
||||
return False
|
||||
con.execute('UPDATE executions SET wait_spent=wait_spent+? WHERE execution_id=?', (charge, row['execution_id']))
|
||||
con.execute("UPDATE generations SET state='QUEUED',updated_at=? WHERE generation_id=?", (self._now(), generation))
|
||||
self._event(con, group, 'UNSENT_RETRY', generation)
|
||||
return True
|
||||
|
||||
def _generate(self, scope, upstream, policy, generation, data, body, reserve, client_connected):
|
||||
url=urlsplit(upstream.base_url)
|
||||
started=time.monotonic()
|
||||
@ -1167,15 +1082,13 @@ class Broker:
|
||||
elif status is not None and status>=500:
|
||||
reason='UPSTREAM_TRANSIENT'
|
||||
elif status is None or not 200<=status<300:
|
||||
reason='CONTEXT_TOO_LONG' if status==400 and CONTEXT_FULL.search(payload or b'') else 'UPSTREAM_REJECTED'
|
||||
reason='UPSTREAM_REJECTED'
|
||||
else:
|
||||
reason=validate_completion(payload,bool(data.get('stream')))
|
||||
usage=_usage(payload,bool(data.get('stream'))) if reason=='COMPLETE' else None
|
||||
self._finish(upstream.quota_group,generation,attempt,policy,reason,known_end=known_end,status=status,headers=headers,usage=usage,transport_reason=error)
|
||||
if reason in ('RATE_LIMIT','UPSTREAM_TRANSIENT') and known_end:
|
||||
continue
|
||||
if reason=='CONTEXT_TOO_LONG':
|
||||
return 400,'application/json',CONTEXT_OVERFLOW_BODY
|
||||
if reason!='COMPLETE':
|
||||
raise QuotaError(reason)
|
||||
with self._db() as con:
|
||||
|
||||
@ -102,10 +102,11 @@ def _recipe_system_prompt(system: str) -> str:
|
||||
section = """## 共享资源与正式验收
|
||||
按任务提示为当前操作提交必要文件:`bench stage run <stage> --recipe recipe.json` 冻结新版本,`--submission SUBMISSION_HANDLE` 复用那次提交。探索次序由你决定;已有 ONNX 可先 export,再试 compile。编译使用 `--config compile.yaml`,文件位于 workspace。
|
||||
共享工具链、板卡和正式评分通过原 stage 工具,使用结果前调用 `terminal_exec({"wait_handle": "<handle>"})`,一次等待原阶段完成。日志和 meta 保存真实执行与失败;按返回的产物路径使用结果。固定任务目标、评测数据与评分阈值不可修改,claim 不能代替独立验收。
|
||||
本任务使用私有 HOME;workspace 内可 cd、使用变量并安装私有依赖。共享工具链保持只读。
|
||||
|
||||
"""
|
||||
return before + section + end + after
|
||||
result = before + section + end + after
|
||||
return result.replace("现有 Pi legacy 沙箱仍保留 shell 路径限制;工具报限制时按其说明使用明确路径。不要为私有环境安装修改共享工具链。",
|
||||
"本任务使用私有 HOME;workspace 内可 cd、使用变量并安装私有依赖。共享工具链保持只读。")
|
||||
|
||||
|
||||
def setup_models_json(force: bool = False) -> Path:
|
||||
@ -313,9 +314,7 @@ def pi_command(rd: Path, prompt: str, mode: str = "e2e") -> list[str]:
|
||||
def run_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str, dry_run: bool = False,
|
||||
*, _claimed: bool = False, runner: str = "pi") -> dict:
|
||||
from .runners import get as get_runner
|
||||
# The collector checked quota right before claiming; a second 0.5 s ledger read here turned busy
|
||||
# moments into UNVERIFIED rollouts that never ran (10-13: 59).
|
||||
get_runner(runner).check(policy, execute=not dry_run and not _claimed)
|
||||
get_runner(runner).check(policy, execute=not dry_run)
|
||||
if not dry_run and runner == "pi":
|
||||
from .collection_runtime import config
|
||||
value, _ = config()
|
||||
@ -424,7 +423,7 @@ def resume_rollout(run_id: str, job_id: str, kk: int) -> dict:
|
||||
readiness = runtime.broker().resume_readiness(state['execution_id'])
|
||||
if not readiness['eligible']:
|
||||
raise ValueError('RESUME_NOT_READY: ' + str(readiness.get('reason')))
|
||||
execution.resume(counted=readiness.get('counted', True))
|
||||
execution.resume()
|
||||
try:
|
||||
runtime.broker().resume(state['execution_id'], checkpoint['generation_id'],
|
||||
checkpoint_complete=checkpoint.get('resumable', False), side_effects_known=True)
|
||||
@ -450,12 +449,6 @@ def resume_rollout(run_id: str, job_id: str, kk: int) -> dict:
|
||||
raise
|
||||
|
||||
|
||||
def _last_generation_reason(execution):
|
||||
from . import collection_runtime as runtime
|
||||
pending = (runtime.broker().execution_status(execution.snapshot()['execution_id']) or {}).get('generations', [])
|
||||
return pending[-1].get('reason') if pending else None
|
||||
|
||||
|
||||
def _execute_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str, dry_run: bool, runner: str = "pi", *, resume=False) -> dict:
|
||||
from .runners import get as get_runner
|
||||
adapter = get_runner(runner)
|
||||
@ -553,10 +546,7 @@ def _execute_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str,
|
||||
quota_state = runtime.broker().execution_status(execution.snapshot()['execution_id'])
|
||||
pending = (quota_state or {}).get('generations', [])
|
||||
reason = pending[-1].get('reason') if pending else 'INCOMPLETE_GENERATION'
|
||||
# A 400 is the provider refusing this exact request (e.g. the context outgrew the model): no resume can accept
|
||||
# it, so a pause only held the rollout until its deadline (10-15 04 retinanet.r101: 760k-token request).
|
||||
if (reason not in ('UPSTREAM_REJECTED', 'CONTEXT_TOO_LONG') and checkpoint.get('resumable') and checkpoint.get('phase') == 'PAUSED_INFRA'
|
||||
and checkpoint.get('closure') and pending and pending[-1]['state'] == 'PAUSED'):
|
||||
if checkpoint.get('resumable') and checkpoint.get('phase') == 'PAUSED_INFRA' and checkpoint.get('closure') and pending and pending[-1]['state'] == 'PAUSED':
|
||||
execution.pause(reason, checkpoint)
|
||||
rj = json.loads((rd / 'rollout.json').read_text())
|
||||
rj.update(state='PAUSED_INFRA', stop_reason=reason, exit_code=r.exit_code)
|
||||
@ -575,10 +565,9 @@ def _execute_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str,
|
||||
from .verify import _cost
|
||||
cost = runtime.budget_used(rd) if execution else _cost(rd)
|
||||
budget_exhausted = cost["turns"] >= rj["max_turns"] or cost["tokens"] >= rj["max_tokens"]
|
||||
context_full = bool(execution) and r.exit_code == 76 and _last_generation_reason(execution) == 'CONTEXT_TOO_LONG'
|
||||
if budget_exhausted or context_full:
|
||||
if budget_exhausted:
|
||||
rj["state"] = "TIMEOUT"
|
||||
rj["timeout_reason"] = "context window full" if context_full else "budget (events verified)"
|
||||
rj["timeout_reason"] = "budget (events verified)"
|
||||
if r.state == "STALLED" or r.exit_code == 75:
|
||||
rj["state"] = "TIMEOUT"; rj.setdefault("timeout_reason", r.stalled_reason or "budget")
|
||||
elif rj.get("state") != "TIMEOUT":
|
||||
@ -591,9 +580,9 @@ def _execute_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str,
|
||||
stop_reason = execution.snapshot().get('stop_reason') or (
|
||||
'BUDGET_EXHAUSTED' if budget_exhausted or r.exit_code == 75 else r.stalled_reason)
|
||||
if r.exit_code == 76 and not stop_reason:
|
||||
# The agent filled the model's window: its own budget ran out, like turns or tokens (outcome BUDGET_EXHAUSTED).
|
||||
last = _last_generation_reason(execution)
|
||||
stop_reason = 'CONTEXT_BUDGET_EXHAUSTED' if last == 'CONTEXT_TOO_LONG' else 'INFRA_' + (last or 'INCOMPLETE_GENERATION')
|
||||
quota_state = runtime.broker().execution_status(execution.snapshot()['execution_id']) or {}
|
||||
pending = quota_state.get('generations', [])
|
||||
stop_reason = 'INFRA_' + ((pending[-1].get('reason') if pending else None) or 'INCOMPLETE_GENERATION')
|
||||
# A resumable quota pause returned above. Every terminal path must fence
|
||||
# the broker BEFORE publishing a finished rollout, including SDK errors.
|
||||
runtime.broker().cancel(execution.snapshot()['execution_id'])
|
||||
|
||||
@ -1,776 +0,0 @@
|
||||
"""Durable, single-submission boundary for an already authorized Salty 0.2 CLI.
|
||||
|
||||
This module never installs/logs in, builds an image, stops jobs, reads credentials
|
||||
or publishes to shared storage. A caller supplies a credential-free bundle and
|
||||
worker plus an immutable, explicit image. Local timeouts leave the CLI process
|
||||
running and quarantine the operation. Receipt recovery never submits again.
|
||||
|
||||
prepare() creates an exclusive receipt directory and snapshots its two inputs.
|
||||
submit(), poll(), fetch() and cleanup() serialize through the receipt lock. The
|
||||
remote directory is derived solely from the receipt UUID, never caller text.
|
||||
Job completion only describes transport state; the caller must verify returned
|
||||
worker results before treating a stage as successful.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
import errno
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path, PurePosixPath
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
from typing import Callable, Mapping, Sequence
|
||||
import uuid
|
||||
|
||||
|
||||
SCHEMA = "bench-salty-transport.v1"
|
||||
JOB_PATTERN = r"salty-[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"
|
||||
TERMINAL = {"succeeded", "failed"}
|
||||
SAFE_ENV = {
|
||||
"OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS",
|
||||
"NUMEXPR_NUM_THREADS", "PYTHONDONTWRITEBYTECODE", "PYTHONUNBUFFERED",
|
||||
}
|
||||
PLAN_FIELDS = ("attempt_uuid", "job_name", "remote_directory", "container_root", "image", "salty",
|
||||
"cpu", "memory", "command", "env", "payloads")
|
||||
|
||||
|
||||
class TransportError(RuntimeError):
|
||||
"""No automatic retry or destructive action was performed."""
|
||||
|
||||
|
||||
class AmbiguousOperation(TransportError):
|
||||
"""The exact operation may have taken effect; retain and reconcile it."""
|
||||
|
||||
|
||||
class FormatError(TransportError):
|
||||
"""The audited CLI format or exact identity did not match."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CommandResult:
|
||||
exit_code: int | None
|
||||
pid: int | None = None
|
||||
timed_out: bool = False
|
||||
process_left_running: bool = False
|
||||
birth_identity: dict | None = None
|
||||
|
||||
|
||||
Runner = Callable[[Sequence[str], Path, Path, float], CommandResult]
|
||||
|
||||
|
||||
def _cli_birth_identity(pid: int) -> dict:
|
||||
"""Read the owned Linux child's birth metadata; never inspect argv/env."""
|
||||
identity = {"pid": pid, "status": "UNVERIFIED"}
|
||||
try:
|
||||
raw = Path("/proc/%d/stat" % pid).read_text()
|
||||
fields = raw[raw.rfind(")") + 2:].split()
|
||||
if int(fields[1]) != os.getpid():
|
||||
return identity
|
||||
identity.update(status="KNOWN", start_ticks=int(fields[19]),
|
||||
boot_id=Path("/proc/sys/kernel/random/boot_id").read_text().strip(),
|
||||
ppid=int(fields[1]))
|
||||
except (OSError, ValueError, IndexError):
|
||||
pass
|
||||
return identity
|
||||
|
||||
|
||||
class _CliProcessUnknown(RuntimeError):
|
||||
def __init__(self, result: CommandResult):
|
||||
self.process_result = result
|
||||
super().__init__("owned CLI process outcome unknown")
|
||||
|
||||
|
||||
class CliRunner:
|
||||
"""File-backed subprocess output; a deadline never terminates a process."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._unresolved: dict[int, subprocess.Popen] = {}
|
||||
|
||||
def __call__(self, argv: Sequence[str], stdout: Path, stderr: Path,
|
||||
timeout: float, *, on_started=None) -> CommandResult:
|
||||
# Do not inspect or serialize the inherited environment. Salty manages
|
||||
# its own existing login. No env-file or credential options are used.
|
||||
with stdout.open("xb") as out, stderr.open("xb") as err:
|
||||
os.chmod(stdout, 0o600)
|
||||
os.chmod(stderr, 0o600)
|
||||
process = subprocess.Popen(list(argv), stdin=subprocess.DEVNULL,
|
||||
stdout=out, stderr=err)
|
||||
# Retain the handle before any callback or interrupt can fail.
|
||||
self._unresolved[process.pid] = process
|
||||
identity = _cli_birth_identity(process.pid)
|
||||
try:
|
||||
if on_started is not None:
|
||||
on_started(process.pid, identity)
|
||||
started = time.monotonic()
|
||||
while process.poll() is None:
|
||||
if time.monotonic() - started >= timeout:
|
||||
return CommandResult(None, process.pid, True, True, identity)
|
||||
time.sleep(min(0.1, timeout))
|
||||
self._unresolved.pop(process.pid, None)
|
||||
return CommandResult(process.returncode, process.pid, birth_identity=identity)
|
||||
except BaseException as error:
|
||||
raise _CliProcessUnknown(CommandResult(None, process.pid, False, True, identity)) from error
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _atomic_json(path: Path, value: object) -> None:
|
||||
temporary = path.with_name(path.name + "." + uuid.uuid4().hex + ".tmp")
|
||||
try:
|
||||
with temporary.open("x", encoding="utf-8") as stream:
|
||||
os.chmod(temporary, 0o600)
|
||||
json.dump(value, stream, indent=2, sort_keys=True, ensure_ascii=False)
|
||||
stream.write("\n")
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temporary, path)
|
||||
descriptor = os.open(path.parent, os.O_RDONLY)
|
||||
try:
|
||||
os.fsync(descriptor)
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
finally:
|
||||
if temporary.exists():
|
||||
temporary.unlink()
|
||||
|
||||
|
||||
def validate_image(image: str) -> str:
|
||||
"""Require an explicit registry/repository and a SHA256 digest, no tag."""
|
||||
if not isinstance(image, str) or "@sha256:" not in image:
|
||||
raise ValueError("Salty image requires registry/repository@sha256:<64 lowercase hex>")
|
||||
repository, digest = image.rsplit("@sha256:", 1)
|
||||
if not re.fullmatch(r"[0-9a-f]{64}", digest) or "/" not in repository:
|
||||
raise ValueError("invalid immutable image reference")
|
||||
registry, name = repository.split("/", 1)
|
||||
if not re.fullmatch(r"[a-z0-9][a-z0-9.-]*(?::[0-9]{1,5})?", registry):
|
||||
raise ValueError("invalid explicit image registry")
|
||||
if "." not in registry and ":" not in registry and registry != "localhost":
|
||||
raise ValueError("image registry must be explicit")
|
||||
if not re.fullmatch(r"[a-z0-9]+(?:[._-][a-z0-9]+)*(?:/[a-z0-9]+(?:[._-][a-z0-9]+)*)*", name):
|
||||
raise ValueError("image repository must not contain tags, credentials or whitespace")
|
||||
return image
|
||||
|
||||
|
||||
def _safe_env(env: Mapping[str, str] | None) -> dict[str, str]:
|
||||
result = {}
|
||||
for key, value in (env or {}).items():
|
||||
if key not in SAFE_ENV or not isinstance(value, str):
|
||||
raise ValueError("only explicit non-sensitive numeric worker environment settings are allowed")
|
||||
if not re.fullmatch(r"[0-9]{1,3}", value):
|
||||
raise ValueError("worker environment values must be literal numbers")
|
||||
if key in {"PYTHONDONTWRITEBYTECODE", "PYTHONUNBUFFERED"}:
|
||||
valid = value in {"0", "1"}
|
||||
else:
|
||||
valid = 1 <= int(value) <= 256
|
||||
if not valid:
|
||||
raise ValueError("worker environment value outside allowed range")
|
||||
result[key] = value
|
||||
return result
|
||||
|
||||
|
||||
def _plan_hash(receipt: dict) -> str:
|
||||
raw = json.dumps({key: receipt[key] for key in PLAN_FIELDS}, sort_keys=True,
|
||||
separators=(",", ":"), ensure_ascii=False).encode("utf-8")
|
||||
return hashlib.sha256(raw).hexdigest()
|
||||
|
||||
|
||||
def parse_describe(output: str, job_id: str) -> str:
|
||||
"""Salty 0.2's evidenced Chinese text contract; drift fails closed."""
|
||||
identities = re.findall(r"^ID:\s*(" + JOB_PATTERN + r")\s*$", output, re.MULTILINE)
|
||||
states = re.findall(r"^状态:\s*([a-z]+)\s*$", output, re.MULTILINE)
|
||||
if identities != [job_id] or len(states) != 1:
|
||||
raise FormatError("describe format or precise job identity mismatch")
|
||||
if states[0] not in {"pending", "running", *TERMINAL}:
|
||||
raise FormatError("unknown Salty job state")
|
||||
return states[0]
|
||||
|
||||
|
||||
def _parse_submission(output: str) -> str:
|
||||
identities = set(re.findall(r"(?<![\w-])" + JOB_PATTERN + r"(?![\w-])", output))
|
||||
if len(identities) != 1:
|
||||
raise FormatError("submission did not return one unique full Salty job ID")
|
||||
return identities.pop()
|
||||
|
||||
|
||||
class SaltyClient:
|
||||
def __init__(self, receipt_dir: Path | str, *, runner: Runner | None = None,
|
||||
timeout_seconds: float = 180) -> None:
|
||||
if not math.isfinite(timeout_seconds) or timeout_seconds <= 0:
|
||||
raise ValueError("CLI deadline must be positive")
|
||||
raw = Path(receipt_dir).absolute()
|
||||
if raw.is_symlink() or not raw.is_dir():
|
||||
raise ValueError("receipt directory must be an existing regular directory")
|
||||
self.directory = raw.resolve()
|
||||
self.receipt_path = self.directory / "receipt.json"
|
||||
self.runner = runner or CliRunner()
|
||||
self.timeout_seconds = timeout_seconds
|
||||
self._read()
|
||||
|
||||
@classmethod
|
||||
def prepare(cls, receipt_dir: Path | str, *, image: str, bundle: Path | str,
|
||||
worker: Path | str, command: Sequence[str],
|
||||
env: Mapping[str, str] | None = None, cpu: int | None = None,
|
||||
memory: str | None = None, salty: str = "salty",
|
||||
runner: Runner | None = None, timeout_seconds: float = 180) -> "SaltyClient":
|
||||
image = validate_image(image)
|
||||
safe_env = _safe_env(env)
|
||||
if cpu is not None and (isinstance(cpu, bool) or not isinstance(cpu, int) or not 1 <= cpu <= 64):
|
||||
raise ValueError("CPU must be an explicit integer from 1 to 64")
|
||||
if memory is not None and (not isinstance(memory, str) or not re.fullmatch(r"[1-9][0-9]{0,5}(?:Mi|Gi)", memory)):
|
||||
raise ValueError("memory must be an explicit Mi/Gi quantity")
|
||||
if not isinstance(salty, str) or not salty or "\x00" in salty:
|
||||
raise ValueError("Salty executable must be a literal path/name")
|
||||
if isinstance(command, (str, bytes)) or not command:
|
||||
raise ValueError("worker command must be a nonempty argv list")
|
||||
if any(not isinstance(token, str) or not token or "\x00" in token for token in command):
|
||||
raise ValueError("worker command argv contains an invalid literal")
|
||||
sources = {"bundle.tar": Path(bundle).absolute(), "worker.py": Path(worker).absolute()}
|
||||
for source in sources.values():
|
||||
if source.is_symlink() or not source.is_file() or source.name.startswith(".env"):
|
||||
raise ValueError("payload sources must be caller-provided regular, non-credential files")
|
||||
if not math.isfinite(timeout_seconds) or timeout_seconds <= 0:
|
||||
raise ValueError("CLI deadline must be positive")
|
||||
directory = Path(receipt_dir).absolute()
|
||||
# Caller creates the parent. Never reuse an existing attempt directory.
|
||||
directory.mkdir(mode=0o700, parents=False, exist_ok=False)
|
||||
directory = directory.resolve()
|
||||
identifier = str(uuid.uuid4())
|
||||
name = "bench-stage-" + identifier
|
||||
container_root = "/data/personal/" + name
|
||||
payload = directory / "payload"
|
||||
payload.mkdir(mode=0o700)
|
||||
snapshots = {}
|
||||
try:
|
||||
for filename, source in sources.items():
|
||||
before = _sha256(source)
|
||||
destination = payload / filename
|
||||
with source.open("rb") as stream, destination.open("xb") as target:
|
||||
shutil.copyfileobj(stream, target)
|
||||
target.flush()
|
||||
os.fsync(target.fileno())
|
||||
os.chmod(destination, 0o600)
|
||||
after = _sha256(destination)
|
||||
if before != after or _sha256(source) != before:
|
||||
raise ValueError("payload changed during immutable snapshot creation")
|
||||
snapshots[filename] = {"sha256": after, "bytes": destination.stat().st_size}
|
||||
receipt = {
|
||||
"schema": SCHEMA, "attempt_uuid": identifier, "job_name": name,
|
||||
"remote_directory": "personal:/" + name, "container_root": container_root,
|
||||
"image": image, "salty": salty, "cpu": cpu, "memory": memory,
|
||||
"command": [token.replace("{container_root}", container_root) for token in command],
|
||||
"env": safe_env, "payloads": snapshots, "phase": "PREPARED",
|
||||
"submission_state": "NOT_SENT", "job_id": None, "job_state": None,
|
||||
"commands": [], "downloads": [], "cleanup": {}, "created_at": time.time(),
|
||||
}
|
||||
receipt["plan_sha256"] = _plan_hash(receipt)
|
||||
_atomic_json(directory / "receipt.json", receipt)
|
||||
except BaseException:
|
||||
# No remote action exists yet. Remove only this exclusive directory.
|
||||
shutil.rmtree(directory)
|
||||
raise
|
||||
return cls(directory, runner=runner, timeout_seconds=timeout_seconds)
|
||||
|
||||
def _read(self) -> dict:
|
||||
if self.receipt_path.is_symlink() or not self.receipt_path.is_file():
|
||||
raise ValueError("transport receipt must be a regular file")
|
||||
receipt = json.loads(self.receipt_path.read_text(encoding="utf-8"))
|
||||
if receipt.get("schema") != SCHEMA:
|
||||
raise ValueError("unknown Salty transport receipt schema")
|
||||
identifier = receipt.get("attempt_uuid")
|
||||
try:
|
||||
valid_uuid = isinstance(identifier, str) and str(uuid.UUID(identifier)) == identifier
|
||||
except ValueError:
|
||||
valid_uuid = False
|
||||
if not valid_uuid:
|
||||
raise ValueError("invalid transport UUID")
|
||||
name = "bench-stage-" + identifier
|
||||
if (receipt.get("job_name") != name or receipt.get("remote_directory") != "personal:/" + name
|
||||
or receipt.get("container_root") != "/data/personal/" + name):
|
||||
raise ValueError("receipt remote path/ownership guard mismatch")
|
||||
job_id = receipt.get("job_id")
|
||||
if job_id is not None and (not isinstance(job_id, str) or not re.fullmatch(JOB_PATTERN, job_id)):
|
||||
raise ValueError("receipt requires a complete precise Salty job ID")
|
||||
validate_image(receipt["image"])
|
||||
_safe_env(receipt.get("env"))
|
||||
if _plan_hash(receipt) != receipt.get("plan_sha256"):
|
||||
raise ValueError("immutable transport plan changed on receipt recovery")
|
||||
payloads = receipt.get("payloads")
|
||||
if not isinstance(payloads, dict) or set(payloads) != {"bundle.tar", "worker.py"}:
|
||||
raise ValueError("receipt payload identity guard mismatch")
|
||||
for filename, metadata in payloads.items():
|
||||
if (not isinstance(metadata, dict) or not re.fullmatch(r"[0-9a-f]{64}", metadata.get("sha256", ""))
|
||||
or not isinstance(metadata.get("bytes"), int) or metadata["bytes"] < 0):
|
||||
raise ValueError("invalid immutable payload receipt")
|
||||
# After authorized local discard (including an unreconciled mkdir),
|
||||
# no future job/upload can use these snapshots. A completed remote
|
||||
# cleanup may also have partly removed them after a crash.
|
||||
if not (receipt.get("cleanup", {}).get("storage_removed") or
|
||||
receipt.get("cleanup", {}).get("local_payload_removal_started")):
|
||||
payload_root = self.directory / "payload"
|
||||
path = payload_root / filename
|
||||
if (payload_root.is_symlink() or path.is_symlink() or not path.is_file()
|
||||
or path.stat().st_size != metadata["bytes"] or _sha256(path) != metadata["sha256"]):
|
||||
raise ValueError("immutable payload/source hash differs on receipt recovery")
|
||||
if job_id is not None and not receipt.get("cleanup", {}).get("job_removed"):
|
||||
submissions = [entry for entry in receipt["commands"] if entry["label"] == "submit"]
|
||||
if len(submissions) != 1:
|
||||
raise ValueError("known job must retain its unique successful submission receipt")
|
||||
entry = submissions[0]
|
||||
filename = entry.get("stdout")
|
||||
if not isinstance(filename, str) or Path(filename).name != filename:
|
||||
raise ValueError("submission recovery output path guard failed")
|
||||
output = self.directory / filename
|
||||
if (entry.get("state") != "COMPLETE" or entry.get("exit_code") != 0
|
||||
or output.is_symlink() or not output.is_file()
|
||||
or _sha256(output) != entry.get("stdout_sha256")
|
||||
or _parse_submission(output.read_text(encoding="utf-8")) != job_id):
|
||||
raise ValueError("known job ID differs from completed exact submission evidence")
|
||||
return receipt
|
||||
|
||||
@contextmanager
|
||||
def _lock(self):
|
||||
lock_path = self.directory / "receipt.lock"
|
||||
if lock_path.is_symlink():
|
||||
raise ValueError("receipt lock cannot be a symlink")
|
||||
descriptor = os.open(lock_path, os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600)
|
||||
try:
|
||||
try:
|
||||
fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError as exc:
|
||||
raise TransportError("another caller owns this transport receipt") from exc
|
||||
yield self._read()
|
||||
finally:
|
||||
fcntl.flock(descriptor, fcntl.LOCK_UN)
|
||||
os.close(descriptor)
|
||||
|
||||
def _save(self, receipt: dict) -> None:
|
||||
receipt["updated_at"] = time.time()
|
||||
_atomic_json(self.receipt_path, receipt)
|
||||
|
||||
@property
|
||||
def receipt(self) -> dict:
|
||||
with self._lock() as receipt:
|
||||
return receipt
|
||||
|
||||
@property
|
||||
def container_root(self) -> str:
|
||||
return self.receipt["container_root"]
|
||||
|
||||
def _execute(self, receipt: dict, arguments: Sequence[str], label: str) -> str:
|
||||
index = len(receipt["commands"])
|
||||
stdout = self.directory / ("%03d-%s.stdout.log" % (index, label))
|
||||
stderr = self.directory / ("%03d-%s.stderr.log" % (index, label))
|
||||
entry = {"label": label, "argv": [receipt["salty"], *arguments],
|
||||
"stdout": stdout.name, "stderr": stderr.name,
|
||||
"state": "STARTED", "started_at": time.time()}
|
||||
receipt["commands"].append(entry)
|
||||
# Persist before starting every CLI operation, especially run/rm/cp.
|
||||
self._save(receipt)
|
||||
try:
|
||||
if isinstance(self.runner, CliRunner):
|
||||
def started(pid, identity):
|
||||
entry.update(local_pid=pid, local_birth_identity=identity,
|
||||
process_left_running=True)
|
||||
self._save(receipt)
|
||||
result = self.runner(entry["argv"], stdout, stderr, self.timeout_seconds,
|
||||
on_started=started)
|
||||
else:
|
||||
result = self.runner(entry["argv"], stdout, stderr, self.timeout_seconds)
|
||||
except BaseException as exc:
|
||||
entry["state"] = "UNKNOWN"
|
||||
entry["exception_type"] = type(exc).__name__
|
||||
process = getattr(exc, "process_result", None)
|
||||
if isinstance(process, CommandResult):
|
||||
entry.update(local_pid=process.pid, local_birth_identity=process.birth_identity,
|
||||
process_left_running=True)
|
||||
self._save(receipt)
|
||||
raise AmbiguousOperation("CLI operation outcome unknown; no automatic retry: " + label) from exc
|
||||
entry.update(exit_code=result.exit_code, local_pid=result.pid, timed_out=result.timed_out,
|
||||
process_left_running=result.process_left_running, ended_at=time.time())
|
||||
if result.birth_identity is not None:
|
||||
entry["local_birth_identity"] = result.birth_identity
|
||||
for name, path in (("stdout", stdout), ("stderr", stderr)):
|
||||
if path.is_file() and not path.is_symlink() and not result.process_left_running:
|
||||
entry[name + "_sha256"] = _sha256(path)
|
||||
if result.timed_out or result.process_left_running or result.exit_code is None:
|
||||
entry["state"] = "UNKNOWN"
|
||||
self._save(receipt)
|
||||
raise AmbiguousOperation("CLI deadline/unknown outcome; process left untouched: " + label)
|
||||
entry["state"] = "COMPLETE" if result.exit_code == 0 else "FAILED"
|
||||
self._save(receipt)
|
||||
if result.exit_code != 0:
|
||||
raise TransportError("CLI %s failed (%s); retained receipt, no retry" % (label, result.exit_code))
|
||||
if stdout.is_symlink() or not stdout.is_file():
|
||||
raise FormatError("CLI runner did not produce regular output")
|
||||
return stdout.read_text(encoding="utf-8", errors="replace")
|
||||
|
||||
def submit(self) -> dict:
|
||||
with self._lock() as receipt:
|
||||
if receipt["job_id"] is not None:
|
||||
return receipt # Recovery uses its precise existing identity.
|
||||
if receipt["cleanup"]:
|
||||
raise TransportError("cleanup already started; payload disposal cannot be followed by submission")
|
||||
if receipt["phase"] != "PREPARED" or receipt["submission_state"] != "NOT_SENT":
|
||||
raise AmbiguousOperation("attempt already started; never resubmit an unknown or partial attempt")
|
||||
receipt["phase"] = "PREPARING"
|
||||
self._save(receipt)
|
||||
try:
|
||||
version = self._execute(receipt, ["version"], "version").strip()
|
||||
if version != "salty 0.2.0":
|
||||
raise FormatError("expected audited Salty 0.2.0 CLI")
|
||||
remote = receipt["remote_directory"]
|
||||
self._execute(receipt, ["storage", "mkdir", remote], "mkdir-exclusive")
|
||||
for filename, metadata in receipt["payloads"].items():
|
||||
path = self.directory / "payload" / filename
|
||||
if path.is_symlink() or not path.is_file() or _sha256(path) != metadata["sha256"]:
|
||||
raise ValueError("immutable local payload mismatch")
|
||||
self._execute(receipt, ["storage", "cp", str(path), remote + "/" + filename],
|
||||
"upload-" + filename.replace(".", "-"))
|
||||
receipt["phase"] = "SUBMITTING"
|
||||
receipt["submission_state"] = "UNKNOWN"
|
||||
self._save(receipt)
|
||||
arguments = ["run", receipt["image"], "--name", receipt["job_name"]]
|
||||
if receipt["cpu"] is not None:
|
||||
arguments.extend(["--cpu", str(receipt["cpu"])])
|
||||
if receipt["memory"] is not None:
|
||||
arguments.extend(["--memory", receipt["memory"]])
|
||||
for key, value in sorted(receipt["env"].items()):
|
||||
arguments.extend(["-e", key + "=" + value])
|
||||
submitted = self._execute(receipt, [*arguments, "--", *receipt["command"]], "submit")
|
||||
receipt["job_id"] = _parse_submission(submitted)
|
||||
receipt["submission_state"] = "KNOWN"
|
||||
receipt["phase"] = "SUBMITTED"
|
||||
self._save(receipt)
|
||||
return receipt
|
||||
except BaseException:
|
||||
receipt["phase"] = ("SUBMISSION_UNKNOWN" if receipt["submission_state"] == "UNKNOWN"
|
||||
else "PREPARATION_FAILED")
|
||||
self._save(receipt)
|
||||
raise
|
||||
|
||||
def recover_recorded_submission(self) -> dict:
|
||||
"""Recover a crash after recorded successful run, never discover by name.
|
||||
|
||||
Only the persisted completed command's exact output may supply the ID.
|
||||
An unresolved process, nonzero exit, missing/changed output or format
|
||||
drift needs operator reconciliation; no job lookup or submit follows.
|
||||
"""
|
||||
with self._lock() as receipt:
|
||||
if receipt["job_id"] is not None:
|
||||
return receipt
|
||||
submissions = [entry for entry in receipt["commands"] if entry["label"] == "submit"]
|
||||
if len(submissions) != 1:
|
||||
raise AmbiguousOperation("no unique recorded submission to recover")
|
||||
entry = submissions[0]
|
||||
if entry.get("state") != "COMPLETE" or entry.get("exit_code") != 0:
|
||||
raise AmbiguousOperation("submission exit remains unknown or failed")
|
||||
output = self.directory / entry["stdout"]
|
||||
if (output.is_symlink() or not output.is_file()
|
||||
or _sha256(output) != entry.get("stdout_sha256")):
|
||||
raise FormatError("recorded submission output is missing or changed")
|
||||
receipt["job_id"] = _parse_submission(output.read_text(encoding="utf-8"))
|
||||
receipt["submission_state"] = "KNOWN"
|
||||
receipt["phase"] = "SUBMITTED"
|
||||
self._save(receipt)
|
||||
return receipt
|
||||
|
||||
def _describe(self, receipt: dict) -> str:
|
||||
job_id = receipt["job_id"]
|
||||
if job_id is None:
|
||||
raise AmbiguousOperation("no known precise job ID; cannot poll or clean up")
|
||||
output = self._execute(receipt, ["describe", job_id], "describe")
|
||||
state = parse_describe(output, job_id)
|
||||
previous = receipt.get("job_state")
|
||||
if previous in TERMINAL and state != previous:
|
||||
raise FormatError("terminal job state changed unexpectedly")
|
||||
receipt["job_state"] = state
|
||||
receipt["phase"] = "TERMINAL" if state in TERMINAL else "SUBMITTED"
|
||||
self._save(receipt)
|
||||
return state
|
||||
|
||||
def poll(self) -> dict:
|
||||
with self._lock() as receipt:
|
||||
if receipt["phase"] == "CLEANED":
|
||||
return receipt
|
||||
if receipt["cleanup"].get("job_removed"):
|
||||
raise TransportError("job already deleted during partial cleanup; resume cleanup explicitly")
|
||||
self._describe(receipt)
|
||||
return receipt
|
||||
|
||||
def fetch(self, remote_relative: str, local_destination: Path | str) -> dict:
|
||||
"""Download one new file from this attempt after exact terminal status.
|
||||
|
||||
The CLI writes only a unique file inside this receipt. A regular file
|
||||
and its hash are recorded before an atomic, no-replace hard link
|
||||
publishes it. Cross-filesystem destinations are explicitly rejected;
|
||||
there is no copy/replace fallback. Caller verifies the returned bytes.
|
||||
Any partial/unknown transfer remains quarantined for reconciliation.
|
||||
"""
|
||||
if not isinstance(remote_relative, str) or not re.fullmatch(r"[A-Za-z0-9_./-]+", remote_relative):
|
||||
raise ValueError("download requires a canonical result file inside this attempt")
|
||||
relative = PurePosixPath(remote_relative)
|
||||
if (not remote_relative or relative.is_absolute()
|
||||
or ".." in relative.parts or relative.as_posix() != remote_relative
|
||||
or remote_relative in {".", "bundle.tar", "worker.py"}):
|
||||
raise ValueError("download requires a canonical result file inside this attempt")
|
||||
destination = Path(local_destination).absolute()
|
||||
if destination.exists() or destination.is_symlink() or not destination.parent.is_dir():
|
||||
raise ValueError("download destination must be a new file under an existing directory")
|
||||
destination = destination.parent.resolve() / destination.name
|
||||
with self._lock() as receipt:
|
||||
if receipt["cleanup"] or receipt["phase"] == "CLEANED":
|
||||
raise TransportError("cannot fetch after cleanup has started")
|
||||
if any(item["remote_relative"] == remote_relative or item["destination"] == str(destination)
|
||||
for item in receipt["downloads"]):
|
||||
raise AmbiguousOperation("download already attempted; inspect exact destination, never overwrite")
|
||||
state = self._describe(receipt)
|
||||
if state not in TERMINAL:
|
||||
raise TransportError("job is active; cannot fetch final results")
|
||||
temporary = self.directory / ("download-%03d-%s.part" %
|
||||
(len(receipt["downloads"]), uuid.uuid4().hex))
|
||||
download = {"remote_relative": remote_relative, "destination": str(destination),
|
||||
"temporary": temporary.name, "state": "STARTED"}
|
||||
receipt["downloads"].append(download)
|
||||
self._save(receipt)
|
||||
try:
|
||||
self._execute(receipt, ["storage", "cp", receipt["remote_directory"] + "/" + remote_relative,
|
||||
str(temporary)], "download")
|
||||
if temporary.is_symlink() or not temporary.is_file():
|
||||
raise FormatError("download did not produce a regular result file")
|
||||
download.update(state="DOWNLOADED", sha256=_sha256(temporary),
|
||||
bytes=temporary.stat().st_size)
|
||||
self._save(receipt)
|
||||
try:
|
||||
# The kernel arbitrates different receipts publishing the
|
||||
# same name. EEXIST must never fall back to an overwrite.
|
||||
os.link(temporary, destination, follow_symlinks=False)
|
||||
except OSError as exc:
|
||||
download.update(state="PUBLISH_FAILED", publication_errno=exc.errno)
|
||||
self._save(receipt)
|
||||
temporary.unlink()
|
||||
download["temporary_removed"] = True
|
||||
self._save(receipt)
|
||||
reason = ("destination already exists; preserved its bytes" if exc.errno == errno.EEXIST else
|
||||
"destination crosses filesystems; no overwrite/copy fallback" if exc.errno == errno.EXDEV else
|
||||
"atomic no-replace publication failed")
|
||||
raise TransportError(reason) from exc
|
||||
download["state"] = "COMPLETE"
|
||||
self._save(receipt)
|
||||
temporary.unlink()
|
||||
download["temporary_removed"] = True
|
||||
self._save(receipt)
|
||||
return dict(download)
|
||||
except BaseException:
|
||||
if download["state"] not in {"COMPLETE", "PUBLISH_FAILED"}:
|
||||
download["state"] = "UNKNOWN"
|
||||
self._save(receipt)
|
||||
raise
|
||||
|
||||
def _remove_local_payload(self, receipt: dict) -> None:
|
||||
cleanup = receipt["cleanup"]
|
||||
if cleanup.get("local_payload_removed"):
|
||||
return
|
||||
# A crash partway through unlinking must still allow receipt recovery.
|
||||
cleanup["local_payload_removal_started"] = True
|
||||
self._save(receipt)
|
||||
payload = self.directory / "payload"
|
||||
if payload.is_symlink():
|
||||
raise ValueError("payload cleanup refuses symlink")
|
||||
if payload.exists():
|
||||
for filename in ("bundle.tar", "worker.py"):
|
||||
path = payload / filename
|
||||
if path.exists() or path.is_symlink():
|
||||
if path.is_symlink() or not path.is_file():
|
||||
raise ValueError("payload cleanup requires regular owned snapshots")
|
||||
path.unlink()
|
||||
payload.rmdir()
|
||||
cleanup["local_payload_removed"] = True
|
||||
self._save(receipt)
|
||||
|
||||
def cleanup(self, *, authorized: bool = False) -> dict:
|
||||
"""Remove only this known terminal job and UUID directory, once authorized.
|
||||
|
||||
Never-submitted preparation can also be discarded: no submit record,
|
||||
no unresolved CLI process, and remote removal only after a verified
|
||||
successful exclusive mkdir. A failed/unknown mkdir never grants
|
||||
ownership or proves absence: retain an unreconciled diagnostic receipt.
|
||||
The caller preserves verified stage outputs before requesting cleanup.
|
||||
A partially completed/unknown destructive CLI operation is never blindly
|
||||
repeated; completed recorded steps can safely continue after restart.
|
||||
Receipt plus hashes remain; payload snapshots and finished CLI logs go.
|
||||
"""
|
||||
if authorized is not True:
|
||||
raise TransportError("cleanup requires explicit caller authorization for this attempt")
|
||||
with self._lock() as receipt:
|
||||
if receipt["phase"] == "CLEANED":
|
||||
return receipt
|
||||
if any(item.get("state") not in {"COMPLETE", "PUBLISH_FAILED"} for item in receipt["downloads"]):
|
||||
raise AmbiguousOperation("unresolved download must be reconciled before cleanup")
|
||||
cleanup = receipt["cleanup"]
|
||||
if receipt["job_id"] is None:
|
||||
if receipt["submission_state"] != "NOT_SENT" or any(
|
||||
entry.get("label") == "submit" or entry.get("argv", [None, None])[1:2] == ["run"]
|
||||
for entry in receipt["commands"]):
|
||||
raise AmbiguousOperation("submission may exist; no cleanup without its precise known job")
|
||||
mkdir_entries = [entry for entry in receipt["commands"] if entry.get("label") == "mkdir-exclusive"]
|
||||
if len(mkdir_entries) > 1:
|
||||
raise ValueError("preparation has multiple exclusive-directory operations")
|
||||
if mkdir_entries and mkdir_entries[0].get("state") != "COMPLETE":
|
||||
entry = mkdir_entries[0]
|
||||
expected = [receipt["salty"], "storage", "mkdir", receipt["remote_directory"]]
|
||||
if entry.get("argv") != expected:
|
||||
raise ValueError("exclusive directory ownership receipt mismatch")
|
||||
# Even a nonzero exit may follow successful server creation.
|
||||
# Never delete a directory whose ownership is unconfirmed.
|
||||
cleanup.update(authorized=True, job_absent=True, job_removed=False,
|
||||
never_submitted=True, remote_owned=False,
|
||||
storage_removed=False, remote_cleanup_unknown=True)
|
||||
cleanup.pop("storage_not_created_by_attempt", None)
|
||||
receipt["phase"] = "UNRECONCILED"
|
||||
self._save(receipt)
|
||||
# mkdir does not read payloads. Retain its logs (including
|
||||
# a still-running writer), while discarding unused snapshots.
|
||||
if not any(other is not entry and (other.get("state") in {"STARTED", "UNKNOWN"}
|
||||
or other.get("process_left_running"))
|
||||
for other in receipt["commands"]):
|
||||
self._remove_local_payload(receipt)
|
||||
for other in receipt["commands"]:
|
||||
if other is entry:
|
||||
continue
|
||||
for key in ("stdout", "stderr"):
|
||||
filename = other[key]
|
||||
if Path(filename).name != filename:
|
||||
raise ValueError("operation log cleanup path guard failed")
|
||||
path = self.directory / filename
|
||||
if path.is_symlink():
|
||||
raise ValueError("log cleanup refuses symlink")
|
||||
if path.is_file():
|
||||
path.unlink()
|
||||
other["logs_removed"] = True
|
||||
self._save(receipt)
|
||||
raise AmbiguousOperation("remote mkdir outcome/ownership is unreconciled; diagnostic receipt retained")
|
||||
# Includes version: an unresolved local process can still write
|
||||
# its log. A timed-out/unknown mutation can continue after us.
|
||||
if any(entry.get("state") in {"STARTED", "UNKNOWN"} or entry.get("process_left_running")
|
||||
for entry in receipt["commands"]):
|
||||
raise AmbiguousOperation("unresolved preparation CLI process; retain exact owned files")
|
||||
remote_owned = bool(cleanup.get("remote_owned"))
|
||||
already_removed = cleanup.get("never_submitted") and cleanup.get("storage_removed")
|
||||
if not already_removed and mkdir_entries and mkdir_entries[0].get("state") == "COMPLETE":
|
||||
entry = mkdir_entries[0]
|
||||
expected = [receipt["salty"], "storage", "mkdir", receipt["remote_directory"]]
|
||||
if entry.get("argv") != expected or entry.get("exit_code") != 0:
|
||||
raise ValueError("exclusive directory ownership receipt mismatch")
|
||||
for key in ("stdout", "stderr"):
|
||||
filename = entry.get(key)
|
||||
if not isinstance(filename, str) or Path(filename).name != filename:
|
||||
raise ValueError("exclusive-directory output path guard failed")
|
||||
output = self.directory / filename
|
||||
if (output.is_symlink() or not output.is_file()
|
||||
or _sha256(output) != entry.get(key + "_sha256")):
|
||||
raise FormatError("exclusive directory ownership evidence missing or changed")
|
||||
remote_owned = True
|
||||
cleanup.update(authorized=True, job_absent=True, job_removed=False,
|
||||
never_submitted=True, remote_owned=remote_owned)
|
||||
if not remote_owned:
|
||||
cleanup["storage_removed"] = True
|
||||
cleanup["storage_not_created_by_attempt"] = True
|
||||
cleanup["remote_cleanup_unknown"] = False
|
||||
self._save(receipt)
|
||||
if cleanup.get("operation_state") == "UNKNOWN":
|
||||
# A crash after _execute persisted rc=0 but before the step flag
|
||||
# need not repeat rm. Verify the completed precise command/logs
|
||||
# and finish only its local receipt bookkeeping.
|
||||
label = cleanup.get("operation")
|
||||
expected = ([receipt["salty"], "rm", receipt["job_id"]]
|
||||
if label == "remove-terminal-job" else
|
||||
[receipt["salty"], "storage", "rm", "-r", receipt["remote_directory"]]
|
||||
if label == "remove-owned-directory" else None)
|
||||
entry = receipt["commands"][-1] if receipt["commands"] else {}
|
||||
if (expected is None or entry.get("label") != label or entry.get("argv") != expected
|
||||
or entry.get("state") != "COMPLETE" or entry.get("exit_code") != 0):
|
||||
raise AmbiguousOperation("cleanup operation outcome unknown; never repeat blindly")
|
||||
for key in ("stdout", "stderr"):
|
||||
filename = entry.get(key)
|
||||
if not isinstance(filename, str) or Path(filename).name != filename:
|
||||
raise ValueError("cleanup recovery log path guard failed")
|
||||
path = self.directory / filename
|
||||
if (path.is_symlink() or not path.is_file()
|
||||
or _sha256(path) != entry.get(key + "_sha256")):
|
||||
raise FormatError("completed cleanup command evidence missing or changed")
|
||||
cleanup["job_removed" if label == "remove-terminal-job" else "storage_removed"] = True
|
||||
cleanup["operation_state"] = "COMPLETE"
|
||||
self._save(receipt)
|
||||
# A prior STARTED operation can still own the directory or write a
|
||||
# log even after a new describe confirms job completion. Refuse all
|
||||
# remote deletion before touching those materials, not afterward.
|
||||
if receipt["job_id"] is not None and any(
|
||||
entry.get("state") in {"STARTED", "UNKNOWN"}
|
||||
or entry.get("process_left_running") for entry in receipt["commands"]):
|
||||
raise AmbiguousOperation("unresolved CLI operation; retain job, directory and logs")
|
||||
if not (cleanup.get("job_removed") or cleanup.get("job_absent")):
|
||||
state = self._describe(receipt)
|
||||
if state not in TERMINAL:
|
||||
raise TransportError("pending/running job cannot be stopped or deleted")
|
||||
# Persist unknown before rm. A crash cannot replay deletion.
|
||||
cleanup.update(operation_state="UNKNOWN", operation="remove-terminal-job",
|
||||
authorized=True, terminal_state=state)
|
||||
self._save(receipt)
|
||||
self._execute(receipt, ["rm", receipt["job_id"]], "remove-terminal-job")
|
||||
cleanup.update(operation_state="COMPLETE", job_removed=True)
|
||||
self._save(receipt)
|
||||
if not cleanup.get("storage_removed"):
|
||||
cleanup.update(operation_state="UNKNOWN", operation="remove-owned-directory")
|
||||
self._save(receipt)
|
||||
self._execute(receipt, ["storage", "rm", "-r", receipt["remote_directory"]],
|
||||
"remove-owned-directory")
|
||||
cleanup.update(operation_state="COMPLETE", storage_removed=True)
|
||||
self._save(receipt)
|
||||
unresolved = [entry for entry in receipt["commands"]
|
||||
if entry.get("process_left_running") or entry.get("state") in {"STARTED", "UNKNOWN"}]
|
||||
if unresolved:
|
||||
raise AmbiguousOperation("unresolved local CLI process/logs retained; cleanup not complete")
|
||||
# These paths are derived internally. No caller-owned result file is
|
||||
# removed, and never recurse through symlinks in the receipt tree.
|
||||
self._remove_local_payload(receipt)
|
||||
for download in receipt["downloads"]:
|
||||
if download.get("temporary_removed") or "temporary" not in download:
|
||||
continue
|
||||
filename = download["temporary"]
|
||||
if not isinstance(filename, str) or not re.fullmatch(r"download-[0-9]{3,}-[0-9a-f]{32}\.part", filename):
|
||||
raise ValueError("download temporary cleanup path guard failed")
|
||||
path = self.directory / filename
|
||||
if path.is_symlink() or (path.exists() and not path.is_file()):
|
||||
raise ValueError("download temporary cleanup requires regular owned file")
|
||||
if path.exists():
|
||||
path.unlink()
|
||||
download["temporary_removed"] = True
|
||||
self._save(receipt)
|
||||
for entry in receipt["commands"]:
|
||||
for key in ("stdout", "stderr"):
|
||||
filename = entry[key]
|
||||
if Path(filename).name != filename:
|
||||
raise ValueError("operation log cleanup path guard failed")
|
||||
path = self.directory / filename
|
||||
if path.is_symlink():
|
||||
raise ValueError("log cleanup refuses symlink")
|
||||
if path.is_file():
|
||||
path.unlink()
|
||||
entry["logs_removed"] = True
|
||||
receipt["phase"] = "CLEANED"
|
||||
cleanup.update(local_temporary_removed=True, completed_at=time.time())
|
||||
self._save(receipt)
|
||||
return receipt
|
||||
@ -1,254 +0,0 @@
|
||||
"""`bench dev`: one agent command in a Salty container that has the toolchain (hb_compile, hmct,
|
||||
onnxruntime, torch). The agent's bash runs on the controller VM, which has none of it.
|
||||
|
||||
The workspace lives twice: the controller copy the agent edits, and the development copy on personal
|
||||
storage (the same absolute path inside containers, so 04/05 read the graph and calibration made here).
|
||||
Before a command, files the agent changed locally since the last sync go up; afterwards, files the
|
||||
command changed come back if small, and big ones (graphs, calibration sets) stay remote and are listed.
|
||||
The worker sandboxes the command (salty_remote_worker.dev): only toolchains, caches and data read-only
|
||||
plus this workspace are visible, no capabilities, no board credentials.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import fcntl
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import tarfile
|
||||
import time
|
||||
import uuid
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path, PurePosixPath
|
||||
from types import SimpleNamespace
|
||||
|
||||
from . import paths, salty_remote
|
||||
from .salty_client import TransportError
|
||||
from .watchdog import RunResult
|
||||
|
||||
EXCLUDED = {'.pi', '.pi-sessions', '.agent-home', '.bench-session.json', '.env', '.env.local', 'guard.log'}
|
||||
UPLOAD_LIMIT = 256 << 20 # the Mac mini uplink is ~1 MB/s; big inputs are made inside bench dev
|
||||
GRACE_S = 300
|
||||
FETCH_MARGIN_S = 120 # the command ends this long before the request deadline so its results still come back
|
||||
|
||||
|
||||
def selected(cfg=None) -> bool:
|
||||
policy = ((cfg or paths.config()).get('execution') or {}).get('salty_remote') or {}
|
||||
return policy.get('enabled') is True and 'terminal' in policy.get('stages', [])
|
||||
|
||||
|
||||
def local_files(ws: Path) -> dict:
|
||||
files = {}
|
||||
for root, dirs, names in os.walk(ws):
|
||||
top = Path(root) == ws
|
||||
dirs[:] = [d for d in dirs if not (top and d in EXCLUDED) and not os.path.islink(os.path.join(root, d))]
|
||||
for name in names:
|
||||
p = Path(root) / name
|
||||
if (top and name in EXCLUDED) or p.is_symlink() or not p.is_file():
|
||||
continue
|
||||
st = p.stat()
|
||||
files[p.relative_to(ws).as_posix()] = [st.st_size, st.st_mtime_ns]
|
||||
return files
|
||||
|
||||
|
||||
@contextmanager
|
||||
def synced(rd: Path):
|
||||
"""{relative path: [size, mtime_ns]} of the controller copy as last synced; changes since are uploads.
|
||||
Held only around reads/updates, so dev commands of one rollout still run in parallel."""
|
||||
path = rd / 'stages' / '_dev-sync.json'
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with (rd / 'stages' / '_dev-sync.lock').open('a') as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||
state = json.loads(path.read_text()) if path.exists() else {}
|
||||
yield state
|
||||
tmp = path.with_suffix('.tmp')
|
||||
tmp.write_text(json.dumps(state))
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
def bundle(path: Path, ws: Path, uploads: list, request: dict) -> None:
|
||||
with tarfile.open(path, 'x') as tar:
|
||||
data = json.dumps(request).encode()
|
||||
info = tarfile.TarInfo('request.json'); info.size = len(data)
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
for rel in uploads:
|
||||
tar.add(ws / rel, 'ws/' + rel, recursive=False)
|
||||
for name in salty_remote.CODE_DIRS: # harness tools (export_onnx, adapters) for the agent to reuse
|
||||
if (paths.BENCH_ROOT / name).is_dir():
|
||||
tar.add(paths.BENCH_ROOT / name, f'code/{name}',
|
||||
filter=lambda m: None if '__pycache__' in m.name or m.name.endswith('.pyc') else m)
|
||||
|
||||
|
||||
def install(ws: Path, archive: Path, result: dict) -> list:
|
||||
"""Changed small files back into the controller copy; only regular files under the workspace."""
|
||||
data = archive.read_bytes()
|
||||
if hashlib.sha256(data).hexdigest() != result['archive']['sha256']:
|
||||
raise ValueError('dev output archive bytes differ from its receipt')
|
||||
installed = []
|
||||
with tarfile.open(fileobj=io.BytesIO(data)) as tar:
|
||||
for member in tar.getmembers():
|
||||
rel = PurePosixPath(member.name)
|
||||
if not member.isfile() or rel.is_absolute() or '..' in rel.parts or rel.parts[0] in EXCLUDED:
|
||||
raise ValueError('unsafe dev output member: ' + member.name)
|
||||
body = tar.extractfile(member).read()
|
||||
if hashlib.sha256(body).hexdigest() != result['outputs'][member.name]['sha256']:
|
||||
raise ValueError('dev output bytes fail verification: ' + member.name)
|
||||
target = ws / rel
|
||||
if any(p.is_symlink() for p in [target, *target.parents] if p.is_relative_to(ws) and p != ws):
|
||||
raise ValueError('dev output would follow a workspace symlink: ' + member.name)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = target.with_name(target.name + '.dev-tmp')
|
||||
tmp.write_bytes(body)
|
||||
os.replace(tmp, target)
|
||||
installed.append(member.name)
|
||||
return installed
|
||||
|
||||
|
||||
def _storage(rd: Path, rel: str):
|
||||
policy = salty_remote._policy(SimpleNamespace(cfg=paths.config()))
|
||||
cli = salty_remote._Cli(policy.get('salty', 'salty'), policy.get('outage_seconds', 1800), time.sleep, time.time, lambda m: None)
|
||||
remote = 'personal:/bench-root/runs/' + (rd / 'workspace').relative_to(paths.BENCH_ROOT / 'runs').as_posix()
|
||||
if rel not in ('', '.'):
|
||||
remote += '/' + PurePosixPath(rel).as_posix()
|
||||
return cli, remote
|
||||
|
||||
|
||||
def remote_size(rd: Path, rel: str, cap: int, max_dirs: int = 40) -> int | None:
|
||||
"""Bytes under the development workspace's <rel>, or None once past cap bytes or max_dirs directories
|
||||
(one `salty storage ls` per directory, ~3 s each)."""
|
||||
cli, root = _storage(rd, rel)
|
||||
todo, total = [root], 0
|
||||
while todo:
|
||||
if max_dirs <= 0:
|
||||
return None
|
||||
max_dirs -= 1
|
||||
here = todo.pop()
|
||||
_, out, _ = cli.call('storage', 'ls', here, ok_404=True)
|
||||
for line in (out or '').splitlines()[1:]: # header: 名称 类型 大小 修改时间
|
||||
name, kind, size = (line.split('\t') + ['', '', ''])[:3]
|
||||
if kind == '目录':
|
||||
todo.append(here + '/' + name)
|
||||
elif size.isdigit():
|
||||
total += int(size)
|
||||
if total > cap:
|
||||
return None
|
||||
return total
|
||||
|
||||
|
||||
def download_dir(rd: Path, rel: str, destination: Path) -> None:
|
||||
"""The development workspace's copy of <rel> (files over the copy-back limit only exist there); a directory
|
||||
never synced up gives an empty destination."""
|
||||
cli, remote = _storage(rd, rel)
|
||||
part = destination.with_name(destination.name + '.part')
|
||||
def fresh():
|
||||
shutil.rmtree(part, ignore_errors=True); part.mkdir(parents=True)
|
||||
rc, _, err = cli.call('storage', 'cp', remote, str(part), timeout=1800, ok_404=True, before=fresh)
|
||||
got = part / PurePosixPath(remote).name # `salty storage cp <dir> <existing dir>` lands in <existing dir>/<name>
|
||||
if rc == 0 and got.is_dir():
|
||||
os.replace(got, destination)
|
||||
else:
|
||||
destination.mkdir(parents=True)
|
||||
shutil.rmtree(part)
|
||||
|
||||
|
||||
def calib_images(rd: Path, job: dict) -> str:
|
||||
"""The job's calibration images for agent v18+ (calib-raw mirrors calib; 10-15 night: told only to
|
||||
"replace /data/calib/", agents searched /data/calib-raw and spent 5-8 commands finding it). From v21 the
|
||||
original training images (images/, from 36's JuiceFS); v18-v20 rollouts keep the resized png/ they started with."""
|
||||
version = int(json.loads((rd / 'rollout.json').read_text()).get('agent_config_version', 0))
|
||||
return job['data']['calib']['path'].replace('/data/calib/', '/data/calib-raw/', 1) + ('/images' if version >= 21 else '/png')
|
||||
|
||||
|
||||
def execute(ctx, hd: Path, req: dict, deadline: float, stdout: Path, stderr: Path) -> RunResult:
|
||||
"""Run req['command'] remotely; returns a watchdog-shaped result for terminal.run."""
|
||||
started = time.time()
|
||||
rd = ctx.stage_root.parent
|
||||
ws = rd / 'workspace'
|
||||
adapter = hd / 'salty'
|
||||
adapter.mkdir(exist_ok=True)
|
||||
policy = salty_remote._policy(SimpleNamespace(cfg=paths.config()))
|
||||
|
||||
def log(message):
|
||||
with (adapter / 'transport.log').open('a') as f:
|
||||
f.write(f"{time.strftime('%F %T')} {message}\n")
|
||||
cli = salty_remote._Cli(policy.get('salty', 'salty'), policy.get('outage_seconds', 1800), time.sleep, time.time, log)
|
||||
state = {'name': 'bench-dev-' + str(uuid.uuid4()), 'job_id': None}
|
||||
state['remote'] = 'personal:/' + state['name']
|
||||
remaining = lambda: deadline - time.monotonic()
|
||||
|
||||
def done(st, code, reason=None, cleanup='VERIFIED'):
|
||||
stdout.touch(); stderr.touch()
|
||||
return RunResult(st, code, time.time() - started, stdout, stderr, stalled_reason=reason, cleanup_state=cleanup)
|
||||
|
||||
with synced(rd) as manifest:
|
||||
local = local_files(ws)
|
||||
uploads = sorted(rel for rel, st in local.items() if manifest.get(rel) != st)
|
||||
total = sum(local[rel][0] for rel in uploads)
|
||||
if total > UPLOAD_LIMIT:
|
||||
stderr.write_text(f'bench dev: {total >> 20} MiB of local changes to upload (limit {UPLOAD_LIMIT >> 20} MiB); '
|
||||
'make big files inside bench dev instead\n')
|
||||
return done('FAIL', 2)
|
||||
request = {'identity': {'stage': 'terminal', 'operation_id': hd.name},
|
||||
'dev': {'workspace': str(ws), 'command': req['command'], 'cwd': req.get('cwd', '.'),
|
||||
'deadline_wall': time.time() + remaining() - FETCH_MARGIN_S,
|
||||
'timeout': req.get('timeout'), # the command's own time, counted from container start
|
||||
'own_inputs': bool(ctx.extra.get('own_inputs_required')),
|
||||
'calib_images': calib_images(rd, ctx.job) if ctx.extra.get('own_inputs_required') else None,
|
||||
'cpu': str((policy.get('cpu_by_stage') or {}).get('terminal', policy.get('cpu', 4)))}}
|
||||
bundle(adapter / 'bundle.tar', ws, uploads, request)
|
||||
shutil.copyfile(salty_remote.WORKER, adapter / 'worker.py')
|
||||
try:
|
||||
# The heaviest dev use is compiling this model: give it at least what its formal compiles needed
|
||||
# (salty_remote._memory doubles after an OOM; 10-14: vit.h.14 died at 16Gi in bench dev).
|
||||
memory = f"{max(int(str(m).removesuffix('Gi')) for m in ((policy.get('memory_by_stage') or {}).get('terminal', policy.get('memory', '16Gi')), salty_remote._memory(policy, SimpleNamespace(name='05-compile'), ctx)))}Gi"
|
||||
state['job_id'] = salty_remote._submit(cli, state, policy, SimpleNamespace(name='terminal'), adapter, memory)
|
||||
(adapter / 'remote.json').write_text(json.dumps(state))
|
||||
with synced(rd) as manifest: # the remote copy has them now, whatever the command does
|
||||
manifest.update({rel: local[rel] for rel in uploads})
|
||||
job_state = None
|
||||
while job_state not in ('succeeded', 'failed', 'stopped'):
|
||||
if remaining() < -GRACE_S: # the worker's own timeout should have ended it already
|
||||
cli.call('stop', state['job_id'], ok_404=True)
|
||||
time.sleep(5 if time.time() - started < 120 else 30) # short commands answer fast; long ones poll gently
|
||||
out = cli.call('describe', state['job_id'])[1]
|
||||
job_state = (re.search(r'^状态:\s*(\w+)', out, re.MULTILINE) or [None, None])[1]
|
||||
if job_state != 'succeeded':
|
||||
salty_remote._remote_logs(policy, state['job_id'], stderr)
|
||||
# A container OOM gives no reason; its memory log shows it (the agent cannot raise the limit itself).
|
||||
if cli.once(['storage', 'cp', state['remote'] + '/usage.log', str(adapter / 'usage.log')])[0] == 0:
|
||||
with stderr.open('a') as f:
|
||||
f.write(f'\n[bench dev container: {memory} memory; usage every 15s — a peak near the limit means it ran out of memory]\n'
|
||||
+ (adapter / 'usage.log').read_text()[-4000:])
|
||||
return done('FAIL', None, 'TERMINAL_DEADLINE_EXCEEDED' if remaining() < 0 else 'REMOTE_WORKER_FAILED')
|
||||
for f in ('result.json', 'outputs.tar', 'stdout.log', 'stderr.log'):
|
||||
salty_remote._fetch(cli, f"{state['remote']}/result/{f}", adapter / f)
|
||||
result = json.loads((adapter / 'result.json').read_text())
|
||||
if result.get('identity') != request['identity']:
|
||||
raise ValueError('remote result belongs to another dev command')
|
||||
installed = install(ws, adapter / 'outputs.tar', result)
|
||||
with synced(rd) as manifest:
|
||||
for rel in installed:
|
||||
st = (ws / rel).stat()
|
||||
manifest[rel] = [st.st_size, st.st_mtime_ns]
|
||||
shutil.copyfile(adapter / 'stdout.log', stdout)
|
||||
shutil.copyfile(adapter / 'stderr.log', stderr)
|
||||
with stdout.open('a') as f:
|
||||
if installed:
|
||||
f.write(f"\n[bench dev] copied back to the workspace: {', '.join(installed[:50])}\n")
|
||||
if result.get('remote_only'):
|
||||
f.write('[bench dev] kept only in the development workspace (too big to copy back; usable from bench dev '
|
||||
'and compile.yaml): ' + ', '.join(f'{r} ({s >> 20} MiB)' for r, s in result['remote_only'][:50]) + '\n')
|
||||
code = result.get('exit_code')
|
||||
return done('OK' if code == 0 else 'FAIL', code, 'TERMINAL_DEADLINE_EXCEEDED' if code == 124 else None)
|
||||
except (TransportError, ValueError, OSError) as exc:
|
||||
with stderr.open('a') as f:
|
||||
f.write(f'\n[bench dev] {type(exc).__name__}: {exc}\n')
|
||||
# The container ends by its own timeout and writes only to this workspace.
|
||||
return done('FAIL', None, 'REMOTE_UNAVAILABLE', 'REMOTE_BOUNDED')
|
||||
finally:
|
||||
if state['job_id']:
|
||||
salty_remote._cleanup(cli, state)
|
||||
for f in ('bundle.tar', 'outputs.tar', 'worker.py'):
|
||||
(adapter / f).unlink(missing_ok=True)
|
||||
@ -1,326 +0,0 @@
|
||||
"""Run one whole stage attempt in a Salty container (opt-in).
|
||||
|
||||
Configuration: execution.salty_remote = {enabled: true, stages: [...], image: REGISTRY/...@sha256:...,
|
||||
salty: /path/to/salty, cpu: 2, cpu_by_stage: {05-compile: 8}, memory: 8Gi, memory_by_stage: {05-compile: 16Gi},
|
||||
wait_seconds: 21600, poll_seconds: 30, outage_seconds: 1800}.
|
||||
|
||||
The controller keeps the agent, ledger, quota and slot/board locks. The container
|
||||
(salty_remote_worker.py) recreates the old compile-server paths over personal
|
||||
storage, so frozen job paths and caches resolve unchanged. Only the stage
|
||||
directory, board receipts, new artifacts and 05's HBM come back to the controller.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
import shutil
|
||||
import re
|
||||
from pathlib import Path, PurePosixPath
|
||||
import subprocess
|
||||
import tarfile
|
||||
import time
|
||||
|
||||
import yaml
|
||||
|
||||
from . import execution, paths
|
||||
from .salty_client import TransportError, JOB_PATTERN
|
||||
|
||||
WORKER = Path(__file__).with_name("salty_remote_worker.py")
|
||||
CONTAINER_PYTHON = "/home/ming01.wang/toolchains/bench-venv/bin/python"
|
||||
BOOT = ("mkdir -p /home/ming01.wang && ln -sfn /data/personal/bench-runtime/home/toolchains /home/ming01.wang/toolchains"
|
||||
" && exec " + CONTAINER_PYTHON + " -B {container_root}/worker.py {container_root}")
|
||||
CODE_DIRS = ("harness", "registry", "scripts", "bin", "export")
|
||||
|
||||
|
||||
def selected(ctx, name: str) -> bool:
|
||||
if os.environ.get("BENCH_SALTY_WORKER"):
|
||||
return False
|
||||
policy = ((ctx.cfg.get("execution") or {}).get("salty_remote")) or {}
|
||||
return policy.get("enabled") is True and name in policy.get("stages", [])
|
||||
|
||||
|
||||
def _policy(ctx) -> dict:
|
||||
policy = dict(ctx.cfg["execution"]["salty_remote"])
|
||||
if "@sha256:" not in policy.get("image", ""):
|
||||
raise ValueError("salty_remote.image must be pinned by digest")
|
||||
return policy
|
||||
|
||||
|
||||
def _work_dir(stage, ctx) -> Path | None:
|
||||
"""05 leaves its HBM in the compile work dir (the completion receipt hashes it there); 06 reads that dir."""
|
||||
from .stages.base import work_dir
|
||||
return work_dir(ctx) if stage.name in ("05-compile", "06-hbm") else None
|
||||
|
||||
|
||||
def _bundle(path: Path, stage, ctx) -> None:
|
||||
config = yaml.safe_load(paths.CONFIG_PATH.read_text())
|
||||
config["paths"]["harness_python"] = CONTAINER_PYTHON
|
||||
job_json = ctx.job_dir / "job.json"
|
||||
files = {str(job_json): job_json}
|
||||
if ctx.agent_config:
|
||||
files[str(ctx.agent_config)] = ctx.agent_config
|
||||
links = {}
|
||||
if (ctx.job_dir / "data").is_symlink():
|
||||
links[str(ctx.job_dir / "data")] = os.readlink(ctx.job_dir / "data")
|
||||
env = {key: os.environ[key] for key in ("BENCH_BOARD_RECEIPT_DIR", "BENCH_OPERATION_ID") if key in os.environ}
|
||||
current = execution.current()
|
||||
phase = current.snapshot().get("phases", {}).get(execution.current_operation()) if current else None
|
||||
if phase and phase.get("execution_deadline_monotonic"):
|
||||
# The container has no execution state; board_perf caps its board call with this wall-clock end.
|
||||
env["BENCH_PHASE_DEADLINE"] = str(time.time() + phase["execution_deadline_monotonic"] - time.monotonic())
|
||||
request = {"identity": {"run_id": ctx.run_id, "job_id": ctx.job["job_id"], "rollout": ctx.rollout,
|
||||
"stage": stage.name, "attempt": ctx.attempt},
|
||||
"run_id": ctx.run_id, "stage": stage.name, "attempt": ctx.attempt, "lane": ctx.lane,
|
||||
"rollout": ctx.rollout, "job_json": str(job_json), "job_dir": str(ctx.job_dir),
|
||||
"stage_root": str(ctx.stage_root), "artifacts": str(ctx.artifacts),
|
||||
"agent_config": str(ctx.agent_config) if ctx.agent_config else None,
|
||||
"extra": json.loads(json.dumps(ctx.extra, default=str)), "env": env, "links": links,
|
||||
"work_dir": str(_work_dir(stage, ctx)) if _work_dir(stage, ctx) else None,
|
||||
"files": {destination: f"files/{i}" for i, destination in enumerate(files)}}
|
||||
root = paths.BENCH_ROOT
|
||||
with tarfile.open(path, "x") as tar:
|
||||
for name, data in (("request.json", json.dumps(request).encode()),
|
||||
("run-config.yaml", yaml.safe_dump(config, allow_unicode=True).encode())):
|
||||
info = tarfile.TarInfo(name)
|
||||
info.size = len(data)
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
for i, source in enumerate(files.values()):
|
||||
tar.add(source, f"files/{i}")
|
||||
for name in CODE_DIRS:
|
||||
if (root / name).is_dir():
|
||||
tar.add(root / name, f"code/{name}",
|
||||
filter=lambda m: None if "__pycache__" in m.name or m.name.endswith(".pyc") else m)
|
||||
|
||||
|
||||
def _install(result: dict, archive: Path, roots: list[Path]) -> None:
|
||||
"""Members are absolute paths without the leading slash; only the job dir and the compile work dir are writable."""
|
||||
data = archive.read_bytes()
|
||||
if hashlib.sha256(data).hexdigest() != result["archive"]["sha256"]:
|
||||
raise ValueError("stage output archive bytes differ from its receipt")
|
||||
with tarfile.open(fileobj=io.BytesIO(data)) as tar:
|
||||
members = tar.getmembers()
|
||||
if {m.name for m in members} != set(result["outputs"]):
|
||||
raise ValueError("stage output archive members differ from receipt")
|
||||
for member in members:
|
||||
rel = PurePosixPath(member.name)
|
||||
target = Path("/", rel)
|
||||
if (not member.isfile() or rel.is_absolute() or ".." in rel.parts
|
||||
or not any(target.is_relative_to(root) for root in roots)):
|
||||
raise ValueError("unsafe stage output member: " + member.name)
|
||||
body = tar.extractfile(member).read()
|
||||
if hashlib.sha256(body).hexdigest() != result["outputs"][member.name]["sha256"]:
|
||||
raise ValueError("stage output bytes fail verification: " + member.name)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = target.with_name(target.name + ".salty-tmp")
|
||||
tmp.write_bytes(body)
|
||||
os.replace(tmp, target)
|
||||
|
||||
|
||||
def _not_found(err: str) -> bool:
|
||||
return "not_found" in err or "404" in err
|
||||
|
||||
|
||||
class _Cli:
|
||||
"""Salty CLI with retries. Every step used here is safe to repeat: the job name is unique per
|
||||
attempt (submit looks it up by name first), uploads overwrite this attempt's own directory,
|
||||
describe/download only read, and rm of a missing object is success. A step fails only after
|
||||
`outage` seconds of consecutive errors (the Mac mini's link drops requests in bursts)."""
|
||||
|
||||
def __init__(self, salty, outage, sleep, clock, log):
|
||||
self.salty, self.outage, self.sleep, self.clock, self.log = salty, outage, sleep, clock, log
|
||||
|
||||
def once(self, args, timeout=120):
|
||||
proc = subprocess.Popen([self.salty, *args], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
try:
|
||||
out, err = proc.communicate(timeout=timeout)
|
||||
return proc.returncode, out, err
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.terminate() # never SIGKILL; a CLI ignoring SIGTERM is left alone and the step retried
|
||||
try:
|
||||
proc.communicate(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
pass
|
||||
return None, "", "timeout"
|
||||
|
||||
def call(self, *args, timeout=120, ok_404=False, before=None):
|
||||
failing_since = None
|
||||
while True:
|
||||
if before:
|
||||
before()
|
||||
rc, out, err = self.once(args, timeout)
|
||||
if rc == 0 or (ok_404 and rc is not None and _not_found(err)):
|
||||
return rc, out, err
|
||||
failing_since = failing_since or self.clock()
|
||||
self.log(f"salty {' '.join(args[:2])} failed: {err.strip()[-200:]}")
|
||||
if self.clock() - failing_since >= self.outage:
|
||||
raise TransportError(f"salty {' '.join(args[:2])} kept failing for {self.outage}s: {err.strip()[-200:]}")
|
||||
self.sleep(min(60, 5 + (self.clock() - failing_since) / 4))
|
||||
|
||||
|
||||
def _memory(policy, stage, ctx) -> str:
|
||||
"""Containers run with memory.oom.group=1: an OOM kills the whole job with no reason given. The need is the
|
||||
model's, so a failed (not stopped) job of this stage in any rollout of any job of this model (other runs and
|
||||
marches too) sets the floor: twice the memory it had, cap 128Gi (near the old compile server). Only the
|
||||
models that need it hold more of the queue."""
|
||||
base = int((policy.get("memory_by_stage") or {}).get(stage.name, policy.get("memory", "16Gi")).removesuffix("Gi"))
|
||||
gi = base
|
||||
model = "--".join(ctx.job_dir.name.split("--")[:2]) # <task>--<model>--<march>--...
|
||||
for meta in ctx.job_dir.parent.parent.glob(f"*/{model}--*/rollouts/*/stages/{stage.name}/a*/meta.json"):
|
||||
m = json.loads(meta.read_text())
|
||||
if m.get("error_class") == "REMOTE_WORKER_FAILED" and m.get("salty_job_state") == "failed":
|
||||
gi = max(gi, 2 * int(str(m.get("salty_memory") or f"{base}Gi").removesuffix("Gi")))
|
||||
return f"{min(gi, 128)}Gi"
|
||||
|
||||
|
||||
def _submit(cli, state, policy, stage, adapter, memory) -> str:
|
||||
remote, name = state["remote"], state["name"]
|
||||
cli.call("storage", "mkdir", "-p", remote)
|
||||
for f in ("bundle.tar", "worker.py"):
|
||||
cli.call("storage", "cp", str(adapter / f), f"{remote}/{f}", timeout=1800)
|
||||
cpu = str((policy.get("cpu_by_stage") or {}).get(stage.name, policy.get("cpu", 4)))
|
||||
# The container sees every host CPU (244) but is throttled to `cpu`; torch/numpy size their thread
|
||||
# pools from the CPU count and then spend most of the time throttled (10-12 probe: 16x slower).
|
||||
caps = [arg for var in ("OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS")
|
||||
for arg in ("-e", f"{var}={max(1, int(float(cpu)))}")]
|
||||
boot = BOOT.replace("{container_root}", "/data/personal/" + name)
|
||||
current = execution.current()
|
||||
if current:
|
||||
# The phase deadline kills only the local side; without this the container ran on for
|
||||
# hours with nobody waiting for it (10-13: 7 such jobs, ~34 cores).
|
||||
boot = boot.replace("&& exec ", f"&& exec timeout {int(current.policy(stage.name)['execution_deadline_s'])} ", 1)
|
||||
run = ["run", policy["image"], "--name", name, "-e", "PYTHONUNBUFFERED=1", *caps,
|
||||
"--cpu", cpu,
|
||||
"--memory", memory,
|
||||
"--", "/bin/sh", "-c", boot]
|
||||
failing_since = None
|
||||
while True: # a lost reply must not create a second job: look the unique name up before every (re)submit
|
||||
rc, out, err = cli.call("describe", name, ok_404=True)
|
||||
if rc == 0:
|
||||
return re.search(JOB_PATTERN, out).group(0)
|
||||
rc, out, err = cli.once(run)
|
||||
if rc == 0 and re.search(JOB_PATTERN, out):
|
||||
return re.search(JOB_PATTERN, out).group(0)
|
||||
failing_since = failing_since or cli.clock()
|
||||
cli.log(f"salty run failed: {err.strip()[-200:]}")
|
||||
if cli.clock() - failing_since >= cli.outage:
|
||||
raise TransportError("salty run kept failing: " + err.strip()[-200:])
|
||||
cli.sleep(10)
|
||||
|
||||
|
||||
def _wait(cli, job_id, policy, started, clock, sleep) -> str:
|
||||
while True:
|
||||
out = cli.call("describe", job_id)[1]
|
||||
state = (re.search(r"^状态:\s*(\w+)", out, re.MULTILINE) or [None, None])[1]
|
||||
if state in ("succeeded", "failed", "stopped"):
|
||||
return state
|
||||
if clock() - started >= policy.get("wait_seconds", 21600):
|
||||
raise TransportError("stage wait budget reached; Salty job retained")
|
||||
sleep(policy.get("poll_seconds", 30))
|
||||
|
||||
|
||||
def _fetch(cli, remote_file: str, destination: Path) -> None:
|
||||
part = destination.with_name(destination.name + ".part")
|
||||
cli.call("storage", "cp", remote_file, str(part), timeout=1800, before=lambda: part.unlink(missing_ok=True))
|
||||
os.replace(part, destination)
|
||||
|
||||
|
||||
def _cleanup(cli, state) -> None:
|
||||
"""Garbage only: the stage result is already installed, so a lasting outage just leaves the job/dir behind."""
|
||||
try:
|
||||
cli.call("rm", state["job_id"], ok_404=True)
|
||||
cli.call("storage", "rm", "-r", state["remote"], ok_404=True)
|
||||
state["cleaned"] = True
|
||||
except TransportError as exc:
|
||||
cli.log(f"cleanup deferred: {exc}")
|
||||
|
||||
|
||||
def run(stage, ctx, *, sleep=time.sleep, clock=time.time):
|
||||
"""An attempt interrupted on the controller is not resumed (stagerun opens a new attempt); its Salty job
|
||||
finishes alone and salty_offload.reap_orphans removes it and personal:/bench-stage-<uuid> once it ends."""
|
||||
out = _run(stage, ctx, sleep=sleep, clock=clock)
|
||||
sd = stage.stage_dir(ctx)
|
||||
if not (sd / "meta.json").exists(): # failures before the container wrote its own meta
|
||||
from .execution import atomic_json
|
||||
# Same identity fields as Stage.run, or the stage worker rejects the receipt (metadata mismatch).
|
||||
meta = {"stage": stage.name, "attempt": ctx.attempt, "rollout": ctx.rollout,
|
||||
**ctx.extra.get("operation_identity", {}),
|
||||
"state": out.state, "exit_code": out.exit_code, "error_class": out.error_class,
|
||||
"duration_s": round(out.duration_s, 3), "finished_at": time.time(),
|
||||
"measurements": out.measurements, **out.meta}
|
||||
atomic_json(sd / "meta.json", meta)
|
||||
atomic_json(ctx.stage_root / stage.name / "meta.json", meta)
|
||||
return out
|
||||
|
||||
|
||||
def _run(stage, ctx, *, sleep, clock):
|
||||
from .stages.base import StageOutcome
|
||||
from .execution import atomic_json
|
||||
started = clock()
|
||||
sd = stage.stage_dir(ctx)
|
||||
adapter = sd / "salty"
|
||||
adapter.mkdir(exist_ok=True)
|
||||
policy = _policy(ctx)
|
||||
|
||||
def log(message):
|
||||
with (adapter / "transport.log").open("a") as f:
|
||||
f.write(f"{time.strftime('%F %T')} {message}\n")
|
||||
cli = _Cli(policy.get("salty", "salty"), policy.get("outage_seconds", 1800), sleep, clock, log)
|
||||
state_file = adapter / "remote.json"
|
||||
state = json.loads(state_file.read_text()) if state_file.exists() else None
|
||||
try:
|
||||
if state is None:
|
||||
_bundle(adapter / "bundle.tar", stage, ctx)
|
||||
shutil.copyfile(WORKER, adapter / "worker.py")
|
||||
name = "bench-stage-" + str(uuid.uuid4())
|
||||
state = {"name": name, "remote": "personal:/" + name, "job_id": None}
|
||||
atomic_json(state_file, state)
|
||||
if not state["job_id"]:
|
||||
state["memory"] = _memory(policy, stage, ctx)
|
||||
state["job_id"] = _submit(cli, state, policy, stage, adapter, state["memory"])
|
||||
atomic_json(state_file, state)
|
||||
job_state = _wait(cli, state["job_id"], policy, started, clock, sleep)
|
||||
meta = {"executor": "salty", "salty_job_id": state["job_id"], "salty_memory": state.get("memory")}
|
||||
if job_state != "succeeded":
|
||||
# The worker never reached a result: no stage outcome exists, board receipts decide quarantine.
|
||||
_remote_logs(policy, state["job_id"], sd / "stderr.log")
|
||||
if cli.once(["storage", "cp", state["remote"] + "/usage.log", str(adapter / "usage.log")])[0] == 0:
|
||||
with (sd / "stderr.log").open("a") as f:
|
||||
f.write("\n[container memory and disk, every 15s]\n" + (adapter / "usage.log").read_text()[-20000:])
|
||||
# Logs, describe (OOM/eviction reason) and the memory log are in stderr.log now; keeping the job and its
|
||||
# personal:/bench-stage-* dir only piled up (10-15: 327 failed jobs, nobody looked at them).
|
||||
_cleanup(cli, state)
|
||||
atomic_json(state_file, state)
|
||||
return StageOutcome("FAIL", 1, "REMOTE_WORKER_FAILED", clock() - started, dict(meta, salty_job_state=job_state))
|
||||
_fetch(cli, state["remote"] + "/result/result.json", adapter / "result.json")
|
||||
result = json.loads((adapter / "result.json").read_text())
|
||||
if result.get("identity", {}).get("attempt") != ctx.attempt or result["identity"]["stage"] != stage.name:
|
||||
raise ValueError("remote result belongs to another stage attempt")
|
||||
_fetch(cli, state["remote"] + "/result/outputs.tar", adapter / "outputs.tar")
|
||||
_install(result, adapter / "outputs.tar", [p for p in (ctx.job_dir, _work_dir(stage, ctx)) if p])
|
||||
_cleanup(cli, state)
|
||||
atomic_json(state_file, state)
|
||||
for name in ("bundle.tar", "outputs.tar", "worker.py"):
|
||||
(adapter / name).unlink(missing_ok=True)
|
||||
ctx.extra.update({k: v for k, v in result["extra"].items() if k not in ctx.extra})
|
||||
o = result["outcome"]
|
||||
return StageOutcome(o["state"], o["exit_code"], o["error_class"], o["duration_s"], dict(o["meta"], **meta),
|
||||
o["measurements"] or {})
|
||||
except (TransportError, ValueError, OSError) as exc:
|
||||
# An ordinary stage failure the agent can retry; board stages still quarantine on missing receipts.
|
||||
with (sd / "stderr.log").open("a") as f:
|
||||
f.write(f"\n[salty remote] {type(exc).__name__}: {exc}\n")
|
||||
return StageOutcome("FAIL", None, "REMOTE_UNAVAILABLE", clock() - started,
|
||||
{"executor": "salty", "salty_job_id": (state or {}).get("job_id"),
|
||||
"salty_remote": (state or {}).get("remote")})
|
||||
|
||||
|
||||
def _remote_logs(policy: dict, job_id: str | None, destination: Path) -> None:
|
||||
if not job_id:
|
||||
return
|
||||
cli = _Cli(policy.get("salty", "salty"), 0, None, None, None)
|
||||
rc, out, err = cli.once(["logs", job_id])
|
||||
rc, desc, derr = cli.once(["describe", job_id]) # the reason (OOM, eviction) lives here; the job is removed next
|
||||
destination.write_text(out[-200000:] + err[-20000:] + "\n[salty describe]\n" + desc + derr)
|
||||
@ -1,383 +0,0 @@
|
||||
"""Container side of salty_remote: rebuild the old compile-server paths, run one stage, pack its directory.
|
||||
|
||||
Usage (inside the Salty job, under the bench-venv interpreter): worker.py <container_root>
|
||||
<container_root>/bundle.tar holds request.json, run-config.yaml, code/ and files/.
|
||||
Writes <container_root>/result/{result.json,outputs.tar}.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import threading
|
||||
import time
|
||||
|
||||
HOME = Path("/home/ming01.wang")
|
||||
BENCH = Path("/mnt/data/ming01.wang/model-compile-hbdk/bench")
|
||||
# harness/container_site/sitecustomize.py caps onnxruntime at the container's CPU quota (it shadows the image's own
|
||||
# sitecustomize, which only installs Ubuntu's apport crash hook).
|
||||
CONTAINER_PYTHONPATH = f"{BENCH / 'harness/container_site'}{os.pathsep}{BENCH}"
|
||||
PERSONAL = Path("/data/personal")
|
||||
RUNTIME = PERSONAL / "bench-runtime"
|
||||
LOCAL_WORK = Path("/tmp/bench-work")
|
||||
PERSISTENT_WORK = PERSONAL / "bench-root/work"
|
||||
UNUSED_ONNX = ("*_original_float_model.onnx", "*_optimized_float_model.onnx", "*_calibrated_model.onnx") # no reader (s06_hbm)
|
||||
|
||||
|
||||
def _link(path: Path, target: Path) -> None:
|
||||
if path.is_symlink() or path.exists():
|
||||
if path.is_symlink() and os.readlink(path) == str(target):
|
||||
return
|
||||
raise RuntimeError(f"layout path already exists: {path}")
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.symlink_to(target)
|
||||
|
||||
|
||||
def layout(stage: str) -> None:
|
||||
"""Frozen job paths point at /home/ming01.wang/... and the old bench root; keep them byte-identical."""
|
||||
# Always local: hb_compile/ld.lld crawl on JuiceFS, and base.work_dir() resolves this link, so every
|
||||
# container must agree on /tmp/bench-work (04 writes it into compile.yaml, 05 compiles there).
|
||||
LOCAL_WORK.mkdir(parents=True, exist_ok=True)
|
||||
for name, target in (("toolchains", RUNTIME / "home/toolchains"), ("outputs", RUNTIME / "home/outputs"),
|
||||
("bench-cache", PERSONAL / "bench/cache"), ("bench-data", PERSONAL / "bench/data"),
|
||||
("bench-work", LOCAL_WORK)):
|
||||
_link(HOME / name, target)
|
||||
(PERSONAL / "bench-root/runs").mkdir(parents=True, exist_ok=True)
|
||||
links = [("cache", PERSONAL / "bench/cache"), ("data", PERSONAL / "bench/data"), (".env.local", RUNTIME / "board.env")]
|
||||
if stage != "export": # export() builds real rollout dirs instead
|
||||
links.append(("runs", PERSONAL / "bench-root/runs"))
|
||||
for name, target in links:
|
||||
_link(BENCH / name, target)
|
||||
(BENCH / "state").mkdir(exist_ok=True)
|
||||
cuda_links()
|
||||
ssh = Path("/root/.ssh")
|
||||
ssh.mkdir(mode=0o700, exist_ok=True)
|
||||
for name in ("rdk_sandbox", "known_hosts"):
|
||||
if not (ssh / name).exists():
|
||||
shutil.copyfile(RUNTIME / "ssh" / name, ssh / name)
|
||||
os.chmod(ssh / name, 0o600)
|
||||
|
||||
|
||||
def cuda_links() -> None:
|
||||
# PTQ's libhbdnn.so loads CUDA 12 / cuDNN 9 from here (installed on the old server); bench-venv's torch wheels carry them.
|
||||
cuda = Path("/usr/local/cuda/lib64")
|
||||
cuda.mkdir(parents=True, exist_ok=True)
|
||||
for lib in (HOME / "toolchains/bench-venv/lib/python3.10/site-packages/nvidia").glob("*/lib/*.so*"):
|
||||
if not (cuda / lib.name).exists():
|
||||
(cuda / lib.name).symlink_to(lib)
|
||||
|
||||
|
||||
def unpack(bundle: Path) -> dict:
|
||||
with tarfile.open(bundle) as tar:
|
||||
request = json.load(tar.extractfile("request.json"))
|
||||
for member in tar.getmembers():
|
||||
if not member.isfile() or ".." in Path(member.name).parts or member.name.startswith("/"):
|
||||
continue
|
||||
if member.name.startswith("code/"):
|
||||
target = BENCH / member.name[len("code/"):]
|
||||
elif member.name == "run-config.yaml":
|
||||
target = BENCH / "run-config.yaml"
|
||||
else:
|
||||
continue
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_bytes(tar.extractfile(member).read())
|
||||
for destination, member in request["files"].items():
|
||||
data = tar.extractfile(member).read()
|
||||
path = Path(destination)
|
||||
if path.exists() and path.read_bytes() == data:
|
||||
continue
|
||||
if path.name == "job.json" and path.exists():
|
||||
raise RuntimeError("job.json differs from the controller copy")
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(data)
|
||||
for path, target in request.get("links", {}).items():
|
||||
_link(Path(path), Path(target))
|
||||
return request
|
||||
|
||||
|
||||
def run_stage(request: dict) -> dict:
|
||||
os.environ.update(request["env"], BENCH_ROOT=str(BENCH), BENCH_CONFIG=str(BENCH / "run-config.yaml"),
|
||||
BENCH_STATE_DIR=str(BENCH / "state"), HOME=str(HOME), BENCH_SALTY_WORKER="1",
|
||||
PYTHONPATH=CONTAINER_PYTHONPATH)
|
||||
sys.path.insert(0, str(BENCH))
|
||||
from harness.stages import BY_NAME, StageContext
|
||||
ctx = StageContext(run_id=request["run_id"], job=json.loads(Path(request["job_json"]).read_text()),
|
||||
job_dir=Path(request["job_dir"]), stage_root=Path(request["stage_root"]),
|
||||
artifacts=Path(request["artifacts"]), rollout=request["rollout"], attempt=request["attempt"],
|
||||
lane=request["lane"], agent_config=Path(request["agent_config"]) if request["agent_config"] else None,
|
||||
extra=request["extra"])
|
||||
out = BY_NAME[request["stage"]]().run(ctx)
|
||||
return {"outcome": {"state": out.state, "exit_code": out.exit_code, "error_class": out.error_class,
|
||||
"duration_s": out.duration_s, "meta": out.meta, "measurements": out.measurements},
|
||||
"extra": ctx.extra}
|
||||
|
||||
|
||||
def snapshot(directory: Path) -> dict:
|
||||
"""Artifacts are copied with copy2 (source mtime kept), so new/changed files are found by comparing stats."""
|
||||
return {p: (p.stat().st_size, p.stat().st_mtime_ns) for p in directory.rglob("*")
|
||||
if p.is_file() and not p.is_symlink()} if directory.is_dir() else {}
|
||||
|
||||
|
||||
def collect(request: dict, result_dir: Path, before: dict) -> dict:
|
||||
"""Everything the controller re-reads: stage-level files, this attempt's directory, board receipts,
|
||||
artifacts written during this stage (HBM/BC) and, for 05, the HBM in its work dir (receipts hash it).
|
||||
Members are absolute paths without the leading slash; the controller only installs under allowed roots."""
|
||||
stage_dir = Path(request["stage_root"]) / request["stage"]
|
||||
files = [p for p in stage_dir.iterdir() if p.is_file()] if stage_dir.is_dir() else []
|
||||
for directory in (stage_dir / f"a{request['attempt']:02d}", Path(request["env"].get("BENCH_BOARD_RECEIPT_DIR", "/nonexistent"))):
|
||||
if directory.is_dir():
|
||||
files += [p for p in directory.rglob("*") if p.is_file() and not p.is_symlink()]
|
||||
files += [p for p, stat in snapshot(Path(request["artifacts"])).items() if before.get(p) != stat]
|
||||
if request["stage"] == "05-compile" and request.get("work_dir") and Path(request["work_dir"]).is_dir():
|
||||
files += list(Path(request["work_dir"]).rglob("*.hbm"))
|
||||
outputs = {}
|
||||
result_dir.mkdir(parents=True, exist_ok=True)
|
||||
with tarfile.open(result_dir / "outputs.tar", "w") as tar:
|
||||
for path in sorted(set(files)):
|
||||
name = str(path).lstrip("/")
|
||||
data = path.read_bytes()
|
||||
info = tarfile.TarInfo(name)
|
||||
info.size, info.mode = len(data), 0o644
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
outputs[name] = {"sha256": hashlib.sha256(data).hexdigest(), "bytes": len(data)}
|
||||
archive = (result_dir / "outputs.tar").read_bytes()
|
||||
return {"outputs": outputs, "archive": {"sha256": hashlib.sha256(archive).hexdigest(), "bytes": len(archive)}}
|
||||
|
||||
|
||||
def _memory() -> str:
|
||||
cg = Path("/sys/fs/cgroup")
|
||||
for current, peak in (("memory.current", "memory.peak"), ("memory/memory.usage_in_bytes", "memory/memory.max_usage_in_bytes")):
|
||||
if (cg / current).exists():
|
||||
read = lambda n: int((cg / n).read_text()) >> 20 if (cg / n).exists() else -1
|
||||
return f"mem={read(current)}M peak={read(peak)}M"
|
||||
return "mem=?"
|
||||
|
||||
|
||||
def usage_log(path: Path, every: float = 15) -> None:
|
||||
"""Memory, free /tmp space and the largest local files, kept on personal storage: a container
|
||||
killed for OOM or evicted for node disk pressure leaves no reason and no logs, this file survives."""
|
||||
while True:
|
||||
files = []
|
||||
for p in Path("/tmp").rglob("*"):
|
||||
try:
|
||||
if p.is_file() and not p.is_symlink():
|
||||
files.append((p.stat().st_size, p))
|
||||
except OSError:
|
||||
pass
|
||||
top = ", ".join(f"{p}={size >> 20}M" for size, p in sorted(files, reverse=True)[:5])
|
||||
with path.open("a") as f:
|
||||
f.write(f"{time.strftime('%T')} {_memory()} tmp_free={shutil.disk_usage('/tmp').free >> 30}G "
|
||||
f"tmp_files={sum(s for s, _ in files) >> 30}G top: {top}\n")
|
||||
time.sleep(every)
|
||||
|
||||
|
||||
def export(request: dict, root: Path) -> None:
|
||||
"""scripts/salty_offload.py's job: unpack each archived rollout (the controller's files) into personal runs,
|
||||
then run the controller's exporter over a local tree of exactly those files.
|
||||
An archive is <kk>.tgz plus <kk>.u<time>.tgz overlays of later changes, unpacked in name order; each one's
|
||||
marker lists the members it unpacked."""
|
||||
runs = PERSONAL / "bench-root/runs"
|
||||
for tgz in sorted((PERSONAL / "bench-root/archive").glob("*/*/*.tgz")):
|
||||
rel = (tgz.parent.parent.name, tgz.parent.name, "rollouts", tgz.name.split(".")[0])
|
||||
st = tgz.stat()
|
||||
marker = runs.joinpath(*rel) / f".unpacked-{tgz.name}-{st.st_size}-{st.st_mtime_ns}"
|
||||
if marker.exists() and marker.stat().st_size: # an empty marker predates member lists: unpack again
|
||||
continue
|
||||
try:
|
||||
with tarfile.open(tgz) as tar:
|
||||
# workspace/ stays in the archive: the exporter never reads it (verify.py) and it is the bulk.
|
||||
members = [m for m in tar.getmembers() if m.isfile() and Path(m.name).parts[:4] == rel
|
||||
and ".." not in Path(m.name).parts and Path(m.name).parts[4:5] != ("workspace",)]
|
||||
tar.extractall(runs, members=members)
|
||||
except (tarfile.TarError, EOFError, OSError) as exc: # still uploading: the next export takes it
|
||||
print(f"skip {tgz}: {type(exc).__name__}: {exc}", flush=True)
|
||||
continue
|
||||
marker.parent.mkdir(parents=True, exist_ok=True)
|
||||
marker.write_text("\n".join(m.name for m in members) + "\n")
|
||||
# Only the archived files: personal runs also holds what containers wrote for attempts the controller had
|
||||
# already abandoned (xfeat 06-hbm a00 ran on past its deadline), and those would change the gate evidence.
|
||||
# verify.py resolves the rollout dir and refuses symlinked prediction sources and terminal_result.json, so
|
||||
# dirs, stage files and the rollout's own files are real copies; the rest is linked (no JuiceFS read).
|
||||
for run in request["export"]:
|
||||
for rd in sorted((runs / run).glob("*/rollouts/*")):
|
||||
names = {n for m in rd.glob(".unpacked-*") for n in m.read_text().splitlines() if n}
|
||||
for name in names:
|
||||
source, target, parts = runs / name, BENCH / "runs" / name, Path(name).parts
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
if parts[4] == "stages" or len(parts) == 5:
|
||||
shutil.copyfile(source, target)
|
||||
elif not target.is_symlink():
|
||||
target.symlink_to(source)
|
||||
for hbm in (rd / "artifacts").glob("*.hbm") if names else (): # left out of the archive as mirrored
|
||||
target = BENCH / "runs" / hbm.relative_to(runs)
|
||||
if not target.exists() and not target.is_symlink():
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.symlink_to(hbm)
|
||||
(PERSONAL / "bench-root/datasets").mkdir(exist_ok=True)
|
||||
_link(BENCH / "datasets", PERSONAL / "bench-root/datasets") # kept between runs: filters and history carry over
|
||||
env = dict(os.environ, BENCH_ROOT=str(BENCH), BENCH_STATE_DIR=str(BENCH / "state"), HOME=str(HOME),
|
||||
PYTHONPATH=str(BENCH))
|
||||
done = {}
|
||||
for run, config in request["export"].items():
|
||||
dataset = BENCH / "datasets" / f"{run}-training"
|
||||
seed = BENCH / "export-seed" / f"{run}.json"
|
||||
if not (dataset / "manifest.json").exists() and seed.exists(): # first run: the controller's filters
|
||||
dataset.mkdir(exist_ok=True)
|
||||
shutil.copyfile(seed, dataset / "manifest.json")
|
||||
done[run] = subprocess.run([sys.executable, "-B", "export/to_agentrl.py", "--runs", run, "--dataset-id",
|
||||
dataset.name, "--refresh"], cwd=BENCH, env=dict(env, BENCH_CONFIG=config),
|
||||
stdout=subprocess.DEVNULL).returncode
|
||||
(root / "result").mkdir(exist_ok=True)
|
||||
with tarfile.open(root / "result/datasets.tar.gz", "w:gz") as tar:
|
||||
for run, rc in done.items():
|
||||
if rc == 0:
|
||||
for f in sorted((BENCH / "datasets" / f"{run}-training").glob("*.json*")):
|
||||
tar.add(f, f"{run}-training/{f.name}")
|
||||
(root / "result/result.json").write_text(json.dumps({"identity": request["identity"], "exit_codes": done}))
|
||||
|
||||
|
||||
NOBODY = 65534
|
||||
DEV_COPY_BACK = 16 << 20 # per file; bigger outputs (graphs, calibration sets) stay on personal storage
|
||||
DEV_COPY_TOTAL = 64 << 20 # the controller downloads at ~1 MB/s
|
||||
DEV_LOG_TAIL = 4 << 20
|
||||
# Inside a user+mount namespace as uid nobody: cover personal storage, re-mount only what the agent may see,
|
||||
# then drop every capability so the cover cannot be unmounted (10-14 probe: without the drop, umount
|
||||
# revealed all of personal storage). /proc/<pid>/root of outside processes is not reachable from there.
|
||||
SANDBOX = r"""set -e
|
||||
i=0
|
||||
for src in $DEV_MOUNTS; do i=$((i+1)); mkdir -p /tmp/devsbx/$i; mount -n --rbind "$src" /tmp/devsbx/$i; done
|
||||
mount -n -t tmpfs none /data/personal
|
||||
i=0
|
||||
for src in $DEV_MOUNTS; do i=$((i+1)); mkdir -p "$src"; mount -n --move /tmp/devsbx/$i "$src"; done
|
||||
cd "$DEV_WORKSPACE/$DEV_CWD"
|
||||
exec setpriv --bounding-set=-all --inh-caps=-all --no-new-privs timeout -k 30 "$DEV_TIMEOUT" bash -c "$DEV_COMMAND"
|
||||
"""
|
||||
|
||||
|
||||
def dev(root: Path) -> None:
|
||||
"""bench dev (harness/salty_dev.py): the agent's command with the toolchain, sandboxed; workspace changes
|
||||
are reported back. Read-only by ownership: everything visible but the workspace belongs to root."""
|
||||
with tarfile.open(root / "bundle.tar") as tar:
|
||||
request = json.load(tar.extractfile("request.json"))
|
||||
spec = request["dev"]
|
||||
for name, target in (("toolchains", RUNTIME / "home/toolchains"), ("outputs", RUNTIME / "home/outputs"),
|
||||
("bench-cache", PERSONAL / "bench/cache"), ("bench-data", PERSONAL / "bench/data")):
|
||||
_link(HOME / name, target)
|
||||
for name, target in (("cache", PERSONAL / "bench/cache"), ("data", PERSONAL / "bench/data"),
|
||||
("runs", PERSONAL / "bench-root/runs")):
|
||||
_link(BENCH / name, target)
|
||||
cuda_links() # hb_compile / hmct need them; board keys and .env.local are deliberately not laid out
|
||||
ws = Path(spec["workspace"]).resolve() # BENCH/runs/... -> this rollout's personal-storage copy
|
||||
if not ws.is_relative_to(PERSONAL / "bench-root/runs") or ws.name != "workspace":
|
||||
raise RuntimeError("dev workspace must be a rollout workspace on personal storage")
|
||||
ws.mkdir(parents=True, exist_ok=True)
|
||||
os.chown(ws, NOBODY, NOBODY)
|
||||
for member in tar.getmembers():
|
||||
rel = Path(member.name)
|
||||
if not member.isfile() or ".." in rel.parts or rel.is_absolute():
|
||||
continue
|
||||
if rel.parts[0] == "code":
|
||||
target = BENCH / Path(*rel.parts[1:])
|
||||
elif rel.parts[0] == "ws":
|
||||
target = ws / Path(*rel.parts[1:])
|
||||
for parent in reversed([p for p in target.parents if p.is_relative_to(ws) and p != ws]):
|
||||
if not parent.exists():
|
||||
parent.mkdir(); os.chown(parent, NOBODY, NOBODY)
|
||||
else:
|
||||
continue
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
if target.is_symlink():
|
||||
target.unlink()
|
||||
target.write_bytes(tar.extractfile(member).read())
|
||||
if rel.parts[0] == "ws":
|
||||
os.chown(target, NOBODY, NOBODY)
|
||||
home = Path("/tmp/devhome"); home.mkdir(exist_ok=True); os.chown(home, NOBODY, NOBODY)
|
||||
# Only calibration data: bench/data/t1 (graded evaluation set, with labels) and pool must stay unseen,
|
||||
# or an agent can pick configs or calibrate on the answers (10-14: efficientvit.b3 read t1 in bench dev).
|
||||
# Likewise only the cache a model is built from: adapter-probes holds private harness/registry copies with
|
||||
# reference configs (8 GLM rollouts read them, regnet.y.800mf copied one), dataset-downloads the labelled
|
||||
# datasets T1 is drawn from, baseline the evaluation images' FP32 outputs.
|
||||
# From agent v18 (spec["own_inputs"]) the graph and calibration set are the agent's: no platform ONNX, and
|
||||
# the calibration training images unpreprocessed (calib-raw/<dataset>) instead of the platform's npy.
|
||||
own = spec.get("own_inputs")
|
||||
cache = [PERSONAL / "bench/cache" / name for name in ("weights", "sources", "venvs", *(() if own else ("onnx",)))]
|
||||
calib = PERSONAL / "bench/data" / ("calib-raw" if own else "calib")
|
||||
mounts = [RUNTIME / "home/toolchains", RUNTIME / "home/outputs", *cache, calib, ws]
|
||||
tc, cpu = HOME / "toolchains", spec["cpu"]
|
||||
env = {"PATH": f"{tc}/oe-3.7.0/venv/bin:{tc}/bench-venv/bin:/usr/local/bin:/usr/bin:/bin", "HOME": str(home),
|
||||
"LANG": "C.UTF-8", "PYTHONPATH": CONTAINER_PYTHONPATH, "BENCH_ROOT": str(BENCH), "PYTHONUNBUFFERED": "1",
|
||||
"OMP_NUM_THREADS": cpu, "MKL_NUM_THREADS": cpu, "OPENBLAS_NUM_THREADS": cpu,
|
||||
"DEV_MOUNTS": " ".join(str(p) for p in mounts if p.is_dir()), "DEV_WORKSPACE": str(ws),
|
||||
"DEV_CWD": spec.get("cwd", "."), "DEV_COMMAND": spec["command"],
|
||||
**({"BENCH_CALIB_IMAGES": spec["calib_images"]} if spec.get("calib_images") else {}),
|
||||
# Wall clock: container start-up must not eat into the time the results need to come back.
|
||||
"DEV_TIMEOUT": str(max(1, int(min(spec["deadline_wall"] - time.time(), spec.get("timeout") or float("inf")))))}
|
||||
before = snapshot(ws)
|
||||
out_dir = root / "result"
|
||||
out_dir.mkdir(exist_ok=True)
|
||||
with (out_dir / "stdout.log").open("wb") as out, (out_dir / "stderr.log").open("wb") as err:
|
||||
code = subprocess.run(["setpriv", f"--reuid={NOBODY}", f"--regid={NOBODY}", "--clear-groups",
|
||||
"unshare", "-Urm", "sh", "-c", SANDBOX], stdout=out, stderr=err, env=env).returncode
|
||||
for name in ("stdout.log", "stderr.log"):
|
||||
path = out_dir / name
|
||||
if path.stat().st_size > DEV_LOG_TAIL:
|
||||
with path.open("rb") as f:
|
||||
f.seek(-DEV_LOG_TAIL, os.SEEK_END)
|
||||
tail = f.read()
|
||||
path.write_bytes(b"[earlier output truncated]\n" + tail)
|
||||
changed = sorted((p, st[0]) for p, st in snapshot(ws).items() if before.get(p) != st)
|
||||
outputs, remote_only, total = {}, [], 0
|
||||
with tarfile.open(out_dir / "outputs.tar", "w") as tar:
|
||||
for path, size in changed:
|
||||
rel = path.relative_to(ws).as_posix()
|
||||
if size > DEV_COPY_BACK or total + size > DEV_COPY_TOTAL:
|
||||
remote_only.append((rel, size)); continue
|
||||
data = path.read_bytes()
|
||||
info = tarfile.TarInfo(rel)
|
||||
info.size, info.mode = len(data), 0o644
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
outputs[rel] = {"sha256": hashlib.sha256(data).hexdigest(), "bytes": len(data)}
|
||||
total += size
|
||||
archive = (out_dir / "outputs.tar").read_bytes()
|
||||
(out_dir / "result.json").write_text(json.dumps({
|
||||
"identity": request["identity"], "exit_code": code, "outputs": outputs, "remote_only": remote_only,
|
||||
"archive": {"sha256": hashlib.sha256(archive).hexdigest(), "bytes": len(archive)}}))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
root = Path(sys.argv[1])
|
||||
threading.Thread(target=usage_log, args=(root / "usage.log",), daemon=True).start()
|
||||
request = json.load(tarfile.open(root / "bundle.tar").extractfile("request.json"))
|
||||
if "dev" in request:
|
||||
dev(root)
|
||||
return 0
|
||||
layout(request["stage"])
|
||||
request = unpack(root / "bundle.tar")
|
||||
if "export" in request:
|
||||
export(request, root)
|
||||
return 0
|
||||
# 06 reads 05's work dir in a later container: persist it after 05/06, restore it before 06.
|
||||
rel = Path(request["work_dir"]).relative_to(HOME / "bench-work") if request.get("work_dir") else None
|
||||
if rel and request["stage"] == "06-hbm" and (PERSISTENT_WORK / rel).is_dir():
|
||||
shutil.copytree(PERSISTENT_WORK / rel, LOCAL_WORK / rel, symlinks=True, dirs_exist_ok=True)
|
||||
before = snapshot(Path(request["artifacts"]))
|
||||
result = run_stage(request)
|
||||
if rel and request["stage"] in ("05-compile", "06-hbm"):
|
||||
shutil.copytree(LOCAL_WORK / rel, PERSISTENT_WORK / rel, symlinks=True, dirs_exist_ok=True,
|
||||
ignore=shutil.ignore_patterns(*UNUSED_ONNX))
|
||||
if request["stage"] == "06-hbm": # s06 drops *_model.onnx after its verifier; mirror that
|
||||
for f in (PERSISTENT_WORK / rel).rglob("*_model.onnx"):
|
||||
f.unlink()
|
||||
result.update(collect(request, root / "result", before), identity=request["identity"])
|
||||
(root / "result/result.json").write_text(json.dumps(result, default=str))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@ -1,509 +0,0 @@
|
||||
"""Opt-in Stage adapter for one immutable PTQ compilation attempt on Salty.
|
||||
|
||||
Configuration: execution.salty = {enabled: true, stages: [05-compile],
|
||||
image: REGISTRY/OWNER/IMAGE@sha256:..., image_source_sha256: ...,
|
||||
image_runtime_contract_sha256: ..., verifier_sha256: ...,
|
||||
salty: /absolute/path/to/salty, wait_seconds: 3600, poll_seconds: 10,
|
||||
cleanup_terminal: true}. CPU/memory remain Salty defaults unless specified.
|
||||
|
||||
Only a normalized compiler configuration, one ONNX and calibration .npy bytes
|
||||
are uploaded. Job configuration, credentials, source tree, state and board locks
|
||||
never enter the bundle. Receipt existence always wins over submission intent.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import contextmanager
|
||||
import fcntl
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import tarfile
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
import yaml
|
||||
|
||||
from . import salty_stage_worker as protocol
|
||||
from .salty_client import SaltyClient, AmbiguousOperation, TransportError, validate_image
|
||||
|
||||
|
||||
def selected(ctx, stage):
|
||||
policy = ((getattr(ctx, "cfg", None) or {}).get("execution") or {}).get("salty") or {}
|
||||
if policy.get("enabled") is not True or stage not in policy.get("stages", []):
|
||||
return False
|
||||
if stage != "05-compile" or ctx.job.get("backend") != "ptq":
|
||||
raise ValueError("Salty stage routing currently supports 05-compile PTQ only")
|
||||
return True
|
||||
|
||||
|
||||
def pending_attempt(stage_root):
|
||||
"""Return the original unresolved compile attempt for trusted driver recovery.
|
||||
|
||||
A caller must resume this attempt, not assign len(previous_attempts). A
|
||||
missing/partial transport is unresolved too; it is never a new submission.
|
||||
"""
|
||||
pending = []
|
||||
for directory in sorted((Path(stage_root) / "05-compile").glob("a*")):
|
||||
if not re.fullmatch(r"a[0-9]{2,}", directory.name):
|
||||
continue
|
||||
plan = directory / "salty/plan.json"
|
||||
if plan.exists():
|
||||
protocol.regular(plan)
|
||||
value = json.loads(plan.read_text())
|
||||
# Byte verification does not certify job/storage deletion. Older
|
||||
# plans have no cleanup policy field, so require reconciliation.
|
||||
unresolved = (value.get("state") != "VERIFIED" or
|
||||
(value.get("cleanup_required", True) and value.get("cleanup_complete") is not True))
|
||||
transport = directory / "salty/transport/receipt.json"
|
||||
if not unresolved:
|
||||
protocol.regular(transport)
|
||||
receipt = json.loads(transport.read_text())
|
||||
unresolved = (any(command.get("state") in {"STARTED", "UNKNOWN"} or command.get("process_left_running")
|
||||
for command in receipt.get("commands", [])) or
|
||||
any(download.get("state") not in {"COMPLETE", "PUBLISH_FAILED"}
|
||||
for download in receipt.get("downloads", [])) or
|
||||
(value.get("cleanup_required", True) and receipt.get("phase") != "CLEANED"))
|
||||
if unresolved:
|
||||
pending.append(int(directory.name[1:]))
|
||||
elif (directory / "salty/transport").exists() or (directory / "salty/inputs.tar").exists():
|
||||
pending.append(int(directory.name[1:]))
|
||||
if len(pending) > 1:
|
||||
raise AmbiguousOperation("multiple unresolved Salty attempts; operator reconciliation required")
|
||||
return pending[0] if pending else None
|
||||
|
||||
|
||||
def dispatch_recorded(stage_root, attempt):
|
||||
"""True once this attempt's submit CLI was recorded (or a job ID is known).
|
||||
|
||||
Only that is dispatch evidence: a prepared bundle or a bare phase field is
|
||||
not, so a spent compile budget still refuses it. An unreadable receipt is
|
||||
never treated as undispatched.
|
||||
"""
|
||||
for directory in (Path(stage_root) / "05-compile").glob("a*"):
|
||||
if re.fullmatch(r"a[0-9]{2,}", directory.name) and int(directory.name[1:]) == attempt:
|
||||
transport = directory / "salty/transport/receipt.json"
|
||||
if not transport.exists():
|
||||
return False
|
||||
try:
|
||||
protocol.regular(transport)
|
||||
receipt = json.loads(transport.read_text())
|
||||
except (OSError, ValueError):
|
||||
return True
|
||||
return receipt.get("job_id") is not None or any(
|
||||
command.get("label") == "submit" for command in receipt.get("commands", []))
|
||||
return False
|
||||
|
||||
|
||||
def _policy(ctx):
|
||||
value = dict(ctx.cfg["execution"]["salty"])
|
||||
allowed = {"enabled", "stages", "image", "image_source_sha256", "image_runtime_contract_sha256",
|
||||
"verifier_sha256", "salty", "cpu", "memory", "wait_seconds", "poll_seconds", "cleanup_terminal"}
|
||||
if set(value) - allowed or value.get("stages") != ["05-compile"]:
|
||||
raise ValueError("unsupported Salty stage configuration")
|
||||
value["image"] = validate_image(value["image"])
|
||||
for key in ("image_source_sha256", "image_runtime_contract_sha256", "verifier_sha256"):
|
||||
if not isinstance(value.get(key), str) or not protocol.HEX.fullmatch(value[key]):
|
||||
raise ValueError("Salty route requires frozen " + key)
|
||||
for key, default in (("wait_seconds", 3600), ("poll_seconds", 10)):
|
||||
number = value.setdefault(key, default)
|
||||
if isinstance(number, bool) or not isinstance(number, (float, int)) or not math.isfinite(number) or number <= 0:
|
||||
raise ValueError("invalid Salty wait/poll budget")
|
||||
if value["poll_seconds"] > 60:
|
||||
raise ValueError("Salty polls must be at most 60 seconds")
|
||||
if type(value.setdefault("cleanup_terminal", True)) is not bool:
|
||||
raise ValueError("cleanup_terminal must be explicit boolean")
|
||||
return value
|
||||
|
||||
|
||||
def _atomic(path, value):
|
||||
from .execution import atomic_json
|
||||
atomic_json(path, value)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _lock(directory):
|
||||
for part in (directory, *directory.parents):
|
||||
if part.is_symlink():
|
||||
raise ValueError("stage receipt path contains symlink")
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
path = directory / "stage.lock"
|
||||
if path.is_symlink():
|
||||
raise ValueError("stage lock symlink")
|
||||
fd = os.open(path, os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600)
|
||||
try:
|
||||
try:
|
||||
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError as exc:
|
||||
raise TransportError("another caller owns this Salty stage attempt") from exc
|
||||
yield
|
||||
finally:
|
||||
fcntl.flock(fd, fcntl.LOCK_UN)
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def _identity(ctx):
|
||||
return {"run_id": ctx.run_id, "job_id": ctx.job["job_id"], "rollout": ctx.rollout, "attempt": ctx.attempt,
|
||||
"task": ctx.job["task"], "backend": ctx.job["backend"], "march": ctx.job["march"],
|
||||
"qcfg": ctx.job["qcfg"], "recipe_submission": (ctx.job.get("recipe") or {}).get("submission_sha256")}
|
||||
|
||||
|
||||
def _request(ctx, policy):
|
||||
from .stages.s04_calibrate import compile_yaml_path
|
||||
file = compile_yaml_path(ctx)
|
||||
if protocol.regular(file).st_size > 1 << 20:
|
||||
raise ValueError("oversized compile configuration")
|
||||
config = yaml.safe_load(file.read_text())
|
||||
if not isinstance(config, dict) or set(config) != set(protocol.CONFIG_FIELDS):
|
||||
raise ValueError("unsupported compile configuration")
|
||||
config = json.loads(json.dumps(config))
|
||||
model = config["model_parameters"]
|
||||
onnx = Path(model.pop("onnx_model"))
|
||||
cwd = model.pop("working_dir")
|
||||
from .stages.base import work_dir
|
||||
if cwd != str(work_dir(ctx)):
|
||||
raise ValueError("compile configuration belongs to another work directory")
|
||||
calibration = Path(config["calibration_parameters"].pop("cal_data_dir"))
|
||||
if not onnx.is_absolute() or not calibration.is_absolute() or onnx.suffix != ".onnx":
|
||||
raise ValueError("local compile inputs require explicit absolute paths")
|
||||
protocol.regular(onnx)
|
||||
for part in (calibration, *calibration.parents):
|
||||
if part.is_symlink():
|
||||
raise ValueError("calibration path contains symlink")
|
||||
if not calibration.is_dir():
|
||||
raise ValueError("calibration directory missing")
|
||||
entries = sorted(calibration.iterdir())
|
||||
# The generated calibration directory is .npy-only; a legacy manifest is
|
||||
# deliberately ignored, not packed. Any other non-.npy object is rejected.
|
||||
sources = {"input/model.onnx": onnx}
|
||||
npys = []
|
||||
for path in entries:
|
||||
if path.name == "manifest.json" and not path.is_symlink() and path.is_file():
|
||||
continue
|
||||
if path.suffix != ".npy":
|
||||
raise ValueError("unexpected object in calibration input directory")
|
||||
protocol.regular(path)
|
||||
npys.append(path)
|
||||
if not 1 <= len(npys) <= protocol.MAX_FILES:
|
||||
raise ValueError("calibration sample count outside bounds")
|
||||
sources.update({"input/calibration/%06d.npy" % i: file for i, file in enumerate(npys)})
|
||||
inputs, total = {}, 0
|
||||
for name, file in sources.items():
|
||||
if any(part.startswith(".env") or part in {".ssh", "state"} for part in file.parts):
|
||||
raise ValueError("credential/state path cannot be a compile input")
|
||||
size = protocol.regular(file).st_size
|
||||
total += size
|
||||
if size <= 0 or total > protocol.MAX_TOTAL:
|
||||
raise ValueError("empty/oversized compile input")
|
||||
inputs[name] = {"sha256": protocol.digest(file), "bytes": size}
|
||||
worker = Path(protocol.__file__).absolute()
|
||||
root = Path(__file__).parent
|
||||
controller_files = [root / name for name in ("salty_stage.py", "salty_client.py", "stageworker.py", "stagerun.py",
|
||||
"pipeline.py", "stages/base.py", "stages/s05_compile.py", "stages/s06_hbm.py")]
|
||||
code = {file.name: protocol.digest(file) for file in controller_files}
|
||||
request = {"schema": protocol.SCHEMA, "identity": _identity(ctx), "image": policy["image"],
|
||||
"image_source_sha256": policy["image_source_sha256"],
|
||||
"image_runtime_contract_sha256": policy["image_runtime_contract_sha256"],
|
||||
"verifier_sha256": policy["verifier_sha256"], "worker_sha256": protocol.digest(worker),
|
||||
"controller_code_sha256": hashlib.sha256(protocol.canonical(code)).hexdigest(),
|
||||
"config": protocol.config_parameters(config, ctx.job["march"], ctx.job["job_id"]), "inputs": inputs}
|
||||
return request, sources, worker
|
||||
|
||||
|
||||
def _bundle(path, request, sources):
|
||||
with tarfile.open(path, "x:") as packed:
|
||||
data = protocol.canonical(request)
|
||||
info = tarfile.TarInfo("request.json")
|
||||
info.size, info.mode, info.mtime = len(data), 0o600, 0
|
||||
packed.addfile(info, io.BytesIO(data))
|
||||
for name, source in sources.items():
|
||||
info = tarfile.TarInfo(name)
|
||||
info.size, info.mode, info.mtime = request["inputs"][name]["bytes"], 0o600, 0
|
||||
with protocol.open_regular(source) as stream:
|
||||
packed.addfile(info, stream)
|
||||
if protocol.digest(source) != request["inputs"][name]["sha256"]:
|
||||
raise ValueError("compile input changed during freeze")
|
||||
protocol.read_manifest(path)
|
||||
|
||||
|
||||
def _fetch(client, remote, destination):
|
||||
receipt = client.receipt
|
||||
prior = [row for row in receipt["downloads"] if row["remote_relative"] == remote]
|
||||
if prior:
|
||||
if (len(prior) != 1 or prior[0]["state"] != "COMPLETE" or prior[0]["destination"] != str(destination)
|
||||
or not destination.is_file() or destination.is_symlink()
|
||||
or destination.stat().st_size != prior[0]["bytes"] or protocol.digest(destination) != prior[0]["sha256"]):
|
||||
raise AmbiguousOperation("previous stage download is incomplete or changed; no repeat")
|
||||
return
|
||||
client.fetch(remote, destination)
|
||||
|
||||
|
||||
def _verified_result(path, request, bundle_sha):
|
||||
if protocol.regular(path).st_size > 2 << 20:
|
||||
raise ValueError("oversized stage result receipt")
|
||||
result = json.loads(path.read_text())
|
||||
for key in ("schema", "identity", "image", "image_source_sha256", "image_runtime_contract_sha256",
|
||||
"worker_sha256", "controller_code_sha256", "verifier_sha256"):
|
||||
if result.get(key) != request[key]:
|
||||
raise ValueError("remote result identity differs: " + key)
|
||||
if (result.get("request_sha256") != hashlib.sha256(protocol.canonical(request)).hexdigest()
|
||||
or result.get("bundle_sha256") != bundle_sha or result.get("state") not in {"OK", "FAIL"}):
|
||||
raise ValueError("remote result input/state identity mismatch")
|
||||
if type(result.get("exit_code")) is not int or result["exit_code"] < 0:
|
||||
raise ValueError("invalid remote compile exit status")
|
||||
for key in ("model_info_exit_code", "verifier_exit_code"):
|
||||
code = result.get(key)
|
||||
if code is not None and (type(code) is not int or code < 0):
|
||||
raise ValueError("invalid remote verification exit status")
|
||||
if result["state"] == "OK" and (result["exit_code"] != 0 or result.get("model_info_exit_code") != 0
|
||||
or result.get("march_read") != request["identity"]["march"]):
|
||||
raise ValueError("claimed successful compile lacks target/tool evidence")
|
||||
files = result.get("outputs")
|
||||
if not isinstance(files, dict) or len(files) > protocol.MAX_FILES:
|
||||
raise ValueError("invalid output manifest")
|
||||
total = 0
|
||||
for name, row in files.items():
|
||||
protocol.relative(name)
|
||||
if (Path(name).suffix not in {".hbm", ".bc", ".onnx", ".log"} or not isinstance(row, dict)
|
||||
or set(row) != {"sha256", "bytes"} or not protocol.HEX.fullmatch(row.get("sha256", ""))
|
||||
or type(row.get("bytes")) is not int or row["bytes"] < 0):
|
||||
raise ValueError("invalid stage output descriptor")
|
||||
total += row["bytes"]
|
||||
if total > protocol.MAX_TOTAL:
|
||||
raise ValueError("oversized compile results")
|
||||
archive = result.get("archive") or {}
|
||||
if (set(archive) != {"sha256", "bytes"} or not protocol.HEX.fullmatch(archive.get("sha256", ""))
|
||||
or type(archive.get("bytes")) is not int or not 0 < archive["bytes"] <= protocol.MAX_TOTAL + (16 << 20)):
|
||||
raise ValueError("invalid result archive identity")
|
||||
if result["state"] == "OK" and (len([name for name in files if name.endswith(".hbm")]) != 1
|
||||
or not any(name.endswith("quantized_model.bc") for name in files)
|
||||
or not any(name.endswith("_ptq_model.onnx") for name in files)):
|
||||
raise ValueError("successful PTQ result requires HBM, BC and quantized ONNX")
|
||||
return result
|
||||
|
||||
|
||||
def _install(archive, result, destination):
|
||||
protocol.regular(archive)
|
||||
if archive.stat().st_size != result["archive"]["bytes"] or protocol.digest(archive) != result["archive"]["sha256"]:
|
||||
raise ValueError("downloaded result archive bytes differ")
|
||||
destination.mkdir(parents=True, exist_ok=True)
|
||||
for part in (destination, *destination.parents):
|
||||
if part.is_symlink():
|
||||
raise ValueError("result destination contains symlink")
|
||||
with tempfile.TemporaryDirectory(prefix=".salty-output-", dir=destination) as temporary:
|
||||
staging = Path(temporary)
|
||||
with tarfile.open(archive, "r:") as packed:
|
||||
members = []
|
||||
for member in packed:
|
||||
members.append(member)
|
||||
if len(members) > protocol.MAX_FILES:
|
||||
raise ValueError("too many output members")
|
||||
if len(members) != len(result["outputs"]) or {m.name for m in members} != set(result["outputs"]):
|
||||
raise ValueError("output archive members differ from receipt")
|
||||
for member in members:
|
||||
protocol.relative(member.name)
|
||||
row = result["outputs"][member.name]
|
||||
if not member.isfile() or member.size != row["bytes"]:
|
||||
raise ValueError("special/changed stage output member")
|
||||
path = staging / member.name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with packed.extractfile(member) as source, path.open("xb") as out:
|
||||
shutil.copyfileobj(source, out)
|
||||
if protocol.digest(path) != row["sha256"]:
|
||||
raise ValueError("stage output bytes fail verification")
|
||||
# Validate every output before publishing any byte. Existing identical
|
||||
# bytes are crash-recovery only; an existing different file is preserved.
|
||||
for name, row in result["outputs"].items():
|
||||
path = destination / name
|
||||
if path.exists() or path.is_symlink():
|
||||
if protocol.regular(path).st_size != row["bytes"] or protocol.digest(path) != row["sha256"]:
|
||||
raise ValueError("existing compile output differs; no overwrite")
|
||||
for name in result["outputs"]:
|
||||
source, path = staging / name, destination / name
|
||||
if not path.exists():
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
for part in (path.parent, *path.parent.parents):
|
||||
if part.is_symlink():
|
||||
raise ValueError("result publication parent symlink")
|
||||
os.link(source, path, follow_symlinks=False)
|
||||
source.unlink()
|
||||
|
||||
|
||||
def _check_installed(result, destination):
|
||||
for name, row in result["outputs"].items():
|
||||
file = destination / str(protocol.relative(name))
|
||||
if protocol.regular(file).st_size != row["bytes"] or protocol.digest(file) != row["sha256"]:
|
||||
raise ValueError("installed compile result changed")
|
||||
|
||||
|
||||
def run(stage, ctx, stage_dir, *, runner=None, sleep=time.sleep, clock=time.monotonic):
|
||||
from .stages.base import StageOutcome, work_dir
|
||||
started = clock()
|
||||
adapter = stage_dir / "salty"
|
||||
with _lock(adapter):
|
||||
try:
|
||||
policy = _policy(ctx)
|
||||
previous = pending_attempt(ctx.stage_root)
|
||||
if previous is not None and previous != ctx.attempt:
|
||||
raise AmbiguousOperation("prior Salty attempt unresolved; resume attempt %d, never submit anew" % previous)
|
||||
ctx.extra.pop("salty_work_dir", None)
|
||||
request, sources, worker = _request(ctx, policy)
|
||||
output_directory = stage_dir / "salty" / "work"
|
||||
plan_file = adapter / "plan.json"
|
||||
bundle = adapter / "inputs.tar"
|
||||
if plan_file.exists():
|
||||
protocol.regular(plan_file)
|
||||
plan = json.loads(plan_file.read_text())
|
||||
if plan.get("request") != request or plan.get("work_dir") != str(output_directory):
|
||||
raise ValueError("current task/input/code identity differs from saved stage attempt")
|
||||
if plan.get("cleanup_required", True) != policy["cleanup_terminal"]:
|
||||
raise ValueError("saved Salty attempt cleanup policy differs; reconcile original policy")
|
||||
else:
|
||||
if (adapter / "transport").exists() or bundle.exists():
|
||||
raise AmbiguousOperation("partial stage preparation; preserve exact receipt and reconcile")
|
||||
_bundle(bundle, request, sources)
|
||||
plan = {"schema": protocol.SCHEMA, "request": request, "bundle_sha256": protocol.digest(bundle),
|
||||
"work_dir": str(output_directory), "state": "PREPARED",
|
||||
"cleanup_required": policy["cleanup_terminal"]}
|
||||
_atomic(plan_file, plan)
|
||||
transport = adapter / "transport"
|
||||
if transport.exists():
|
||||
client = SaltyClient(transport, runner=runner)
|
||||
if client.receipt["image"] != request["image"]:
|
||||
raise ValueError("stage/transport image identity differs")
|
||||
else:
|
||||
if plan["state"] != "PREPARED" or protocol.digest(bundle) != plan["bundle_sha256"]:
|
||||
raise AmbiguousOperation("stage transport missing after dispatch; never resubmit")
|
||||
client = SaltyClient.prepare(transport, image=request["image"], bundle=bundle, worker=worker,
|
||||
command=["/opt/bench/ptq/bin/python", "-B", "{container_root}/worker.py",
|
||||
"--bundle", "{container_root}/bundle.tar", "--bundle-sha256", plan["bundle_sha256"],
|
||||
"--result-dir", "{container_root}/result"],
|
||||
env={"PYTHONDONTWRITEBYTECODE": "1", "PYTHONUNBUFFERED": "1",
|
||||
"OMP_NUM_THREADS": "1", "MKL_NUM_THREADS": "1", "OPENBLAS_NUM_THREADS": "1"},
|
||||
salty=policy.get("salty", "salty"), cpu=policy.get("cpu"), memory=policy.get("memory"), runner=runner)
|
||||
receipt = client.receipt
|
||||
if receipt["phase"] == "PREPARED":
|
||||
client.submit()
|
||||
elif receipt["job_id"] is None:
|
||||
client.recover_recorded_submission()
|
||||
plan.update(state="RESULT_VERIFIED" if "verified_result" in plan else "DISPATCHED", job_id=client.receipt["job_id"])
|
||||
_atomic(plan_file, plan)
|
||||
if "verified_result" not in plan:
|
||||
while client.poll()["job_state"] not in {"succeeded", "failed"}:
|
||||
if clock() - started >= policy["wait_seconds"]:
|
||||
raise AmbiguousOperation("stage wait budget reached; existing Salty job retained for continuation")
|
||||
sleep(min(policy["poll_seconds"], max(0.001, policy["wait_seconds"] - (clock() - started))))
|
||||
result_file = adapter / "result.json"
|
||||
_fetch(client, "result/result.json", result_file)
|
||||
result = _verified_result(result_file, request, plan["bundle_sha256"])
|
||||
archive = adapter / "outputs.tar"
|
||||
_fetch(client, "result/outputs.tar", archive)
|
||||
_install(archive, result, output_directory)
|
||||
plan.update(state="RESULT_VERIFIED", verified_result=result, result_sha256=protocol.digest(result_file))
|
||||
_atomic(plan_file, plan)
|
||||
else:
|
||||
result = plan["verified_result"]
|
||||
verified = _verified_result(adapter / "result.json", request, plan["bundle_sha256"])
|
||||
if verified != result or protocol.digest(adapter / "result.json") != plan["result_sha256"]:
|
||||
raise ValueError("persisted verified result changed")
|
||||
_check_installed(result, output_directory)
|
||||
if policy["cleanup_terminal"]:
|
||||
client.cleanup(authorized=True)
|
||||
for file in (bundle, adapter / "outputs.tar"):
|
||||
if file.exists():
|
||||
protocol.regular(file)
|
||||
file.unlink()
|
||||
plan["cleanup_complete"] = True
|
||||
plan["state"] = "VERIFIED"
|
||||
_atomic(plan_file, plan)
|
||||
for kind in ("stdout", "stderr"):
|
||||
source = output_directory / ("compile." + kind + ".log")
|
||||
if source.exists():
|
||||
shutil.copyfile(source, stage_dir / (kind + ".log"))
|
||||
meta = {"executor": "salty", "salty_job_id": plan["job_id"], "salty_receipt": str(adapter / "plan.json"),
|
||||
"image": request["image"], "input_identity_sha256": hashlib.sha256(protocol.canonical(request)).hexdigest(),
|
||||
"work_dir": str(output_directory), "cleanup_complete": plan.get("cleanup_complete", False)}
|
||||
ctx.extra["salty_work_dir"] = str(output_directory)
|
||||
from .errors import classify
|
||||
error_class = None
|
||||
if result["state"] != "OK":
|
||||
error_class = classify((stage_dir / "stderr.log").read_text(errors="replace"),
|
||||
(stage_dir / "stdout.log").read_text(errors="replace"), result["exit_code"])
|
||||
return StageOutcome(result["state"], result["exit_code"], error_class,
|
||||
clock() - started, meta)
|
||||
except (AmbiguousOperation, TransportError) as exc:
|
||||
(stage_dir / "stderr.log").write_text(str(exc) + "\n")
|
||||
# Legacy pipeline retries STALLED with a new attempt. FAIL plus
|
||||
# SIDE_EFFECT_UNKNOWN preserves the lifecycle quarantine boundary.
|
||||
return StageOutcome("FAIL", None, "SIDE_EFFECT_UNKNOWN", clock() - started,
|
||||
{"executor": "salty", "salty_receipt": str(adapter / "plan.json"),
|
||||
"cleanup_state": "QUARANTINED", "reconciliation_required": True})
|
||||
|
||||
|
||||
def hbm(ctx, stage_dir):
|
||||
"""06-hbm consumes verified 05 receipts and original parsers, never x86 tools."""
|
||||
from .stages.base import StageOutcome, work_dir, recipe_receipt, recipe_output
|
||||
from .stages import s06_hbm as local
|
||||
attempts = sorted((ctx.stage_root / "05-compile").glob("a*"))
|
||||
if not attempts:
|
||||
return None
|
||||
file = attempts[-1] / "salty/plan.json"
|
||||
if not file.exists():
|
||||
return None
|
||||
if ctx.job.get("recipe"):
|
||||
parent = recipe_receipt(ctx, "05-compile")
|
||||
file = ctx.stage_root / "05-compile" / ("a%02d" % parent["attempt"]) / "salty/plan.json"
|
||||
protocol.regular(file)
|
||||
plan = json.loads(file.read_text())
|
||||
if plan.get("state") != "VERIFIED" or not plan.get("verified_result"):
|
||||
raise ValueError("latest remote compilation is unverified; 06-hbm cannot use an older attempt")
|
||||
request, result = plan["request"], plan["verified_result"]
|
||||
identity = _identity(ctx)
|
||||
identity["attempt"] = request["identity"]["attempt"]
|
||||
if request["identity"] != identity or result["state"] != "OK":
|
||||
raise ValueError("06-hbm requires successful 05 receipt for this exact task")
|
||||
verified = _verified_result(file.parent / "result.json", request, plan["bundle_sha256"])
|
||||
if verified != result or protocol.digest(file.parent / "result.json") != plan["result_sha256"]:
|
||||
raise ValueError("compile verification receipt changed")
|
||||
wd = Path(plan["work_dir"])
|
||||
_check_installed(result, wd)
|
||||
hbms = [wd / name for name in result["outputs"] if name.endswith(".hbm")]
|
||||
bcs = [wd / name for name in result["outputs"] if name.endswith("quantized_model.bc")]
|
||||
if len(hbms) != 1 or len(bcs) != 1:
|
||||
raise ValueError("06-hbm requires one verified HBM and BC")
|
||||
ctx.artifacts.mkdir(parents=True, exist_ok=True)
|
||||
dst, bc = local.hbm_path(ctx), local.bc_path(ctx)
|
||||
for source, target in ((hbms[0], dst), (bcs[0], bc)):
|
||||
if target.exists():
|
||||
if protocol.digest(target) != protocol.digest(source):
|
||||
raise ValueError("artifact exists with different bytes; no overwrite")
|
||||
else:
|
||||
with source.open("rb") as stream, target.open("xb") as out:
|
||||
shutil.copyfileobj(stream, out)
|
||||
info = "\n".join((wd / ("model_info." + kind + ".log")).read_text(errors="replace") for kind in ("stdout", "stderr"))
|
||||
io_fields = local._parse_io(info)
|
||||
verifier = "\n".join((wd / ("verifier." + kind + ".log")).read_text(errors="replace") for kind in ("stdout", "stderr")) if result.get("verifier_exit_code") is not None else ""
|
||||
cosine = local._verifier_output_cosine(verifier)
|
||||
compile_text = "\n".join((wd / ("compile." + kind + ".log")).read_text(errors="replace") for kind in ("stdout", "stderr"))
|
||||
measurements = {"hbm_exists": True, "hbm_sha256": protocol.digest(dst), "model_info_ok": result["model_info_exit_code"] == 0,
|
||||
"march_read": local._find_march(info), "march_ok": local._find_march(info) == ctx.job["march"],
|
||||
"io": io_fields, "io_ok": local._check_io(io_fields, ctx.job["model_spec"]["input_spec"]),
|
||||
"static_perf": local._parse_static_perf(info), "verifier_rc": result.get("verifier_exit_code"),
|
||||
"verifier_cosine": cosine, "verifier_ok": result.get("verifier_exit_code") == 0 and (cosine is None or cosine >= ctx.job["gates"]["G1"].get("verifier_cosine_threshold", 0.99)),
|
||||
"hbm_bc_cos_min": None, "hbm_bc_spotcheck": "not run; board HBM verification remains required",
|
||||
"log_has_error": bool(re.search(r"\bERROR\b", compile_text)), "hmct_output_cosine": local._hmct_output_cosine(compile_text)}
|
||||
for name in ("model_info.stdout.log", "model_info.stderr.log", "verifier.stdout.log", "verifier.stderr.log"):
|
||||
if (wd / name).exists():
|
||||
shutil.copyfile(wd / name, stage_dir / name)
|
||||
(stage_dir / "stdout.log").write_text(json.dumps(measurements, indent=2) + "\n")
|
||||
(stage_dir / "stderr.log").touch()
|
||||
meta = {"hbm": str(dst), "hbm_sha256": protocol.digest(dst), "source_hbm": str(hbms[0]),
|
||||
"size": dst.stat().st_size, "work_dir": str(wd), "executor": "salty-receipt",
|
||||
"salty_job_id": plan["job_id"], "compile_receipt_sha256": protocol.digest(file)}
|
||||
if ctx.job.get("recipe"):
|
||||
meta["recipe_outputs"] = {"hbm": recipe_output(dst), "bc": recipe_output(bc)}
|
||||
return StageOutcome("OK", 0, None, 0, meta, {"G1": measurements})
|
||||
@ -1,327 +0,0 @@
|
||||
"""Standalone, credential-free PTQ worker uploaded with one frozen input bundle.
|
||||
|
||||
Only its CLI runs a compiler. Importing this module performs no runtime work.
|
||||
No shell, timeout, process signalling, board access, or production state exists here.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from contextlib import contextmanager
|
||||
import hashlib
|
||||
import importlib.metadata
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path, PurePosixPath
|
||||
import re
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
SCHEMA = "bench-salty-ptq-stage.v1"
|
||||
HEX = re.compile(r"[0-9a-f]{64}")
|
||||
MAX_FILES = 4096
|
||||
MAX_TOTAL = 8 * 1024 ** 3
|
||||
CONFIG_FIELDS = {
|
||||
"model_parameters": {"march", "output_model_file_prefix", "layer_out_dump"},
|
||||
"input_parameters": {"input_name", "input_shape", "input_type_rt", "input_type_train",
|
||||
"input_layout_rt", "input_layout_train", "mean_value", "scale_value", "input_batch"},
|
||||
"calibration_parameters": {"calibration_type", "num_bin", "max_percentile", "per_channel", "quant_config"},
|
||||
"compiler_parameters": {"compile_mode", "optimize_level", "debug", "jobs"},
|
||||
}
|
||||
|
||||
|
||||
def canonical(value):
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode()
|
||||
|
||||
|
||||
def digest(path):
|
||||
h = hashlib.sha256()
|
||||
with open_regular(path) as stream:
|
||||
for block in iter(lambda: stream.read(1 << 20), b""):
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def open_regular(path):
|
||||
"""Walk with openat/O_NOFOLLOW so a racing leaf/parent cannot redirect IO."""
|
||||
path = Path(os.path.abspath(path))
|
||||
descriptor = os.open("/", os.O_RDONLY | os.O_DIRECTORY)
|
||||
try:
|
||||
for part in path.parts[1:-1]:
|
||||
next_fd = os.open(part, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, dir_fd=descriptor)
|
||||
os.close(descriptor)
|
||||
descriptor = next_fd
|
||||
fd = os.open(path.name, os.O_RDONLY | os.O_NOFOLLOW, dir_fd=descriptor)
|
||||
try:
|
||||
info = os.fstat(fd)
|
||||
if not stat.S_ISREG(info.st_mode) or info.st_nlink != 1:
|
||||
raise ValueError("input/output must be an independent regular file")
|
||||
with os.fdopen(fd, "rb", closefd=False) as stream:
|
||||
yield stream
|
||||
finally:
|
||||
os.close(fd)
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
|
||||
|
||||
def relative(name):
|
||||
if not isinstance(name, str) or not re.fullmatch(r"[A-Za-z0-9_./-]+", name):
|
||||
raise ValueError("noncanonical bundle/output name")
|
||||
p = PurePosixPath(name)
|
||||
if p.is_absolute() or any(part.startswith(".") for part in p.parts) or p.as_posix() != name or not p.parts:
|
||||
raise ValueError("bundle/output path traversal")
|
||||
return p
|
||||
|
||||
|
||||
def regular(path):
|
||||
path = Path(os.path.abspath(path))
|
||||
for part in (path, *path.parents):
|
||||
if part.is_symlink():
|
||||
raise ValueError("symlink in input/output path")
|
||||
info = path.stat()
|
||||
if not stat.S_ISREG(info.st_mode) or info.st_nlink != 1:
|
||||
raise ValueError("input/output must be an independent regular file")
|
||||
return info
|
||||
|
||||
|
||||
def config_parameters(config, march, prefix):
|
||||
if not isinstance(config, dict) or set(config) != set(CONFIG_FIELDS):
|
||||
raise ValueError("unsupported PTQ configuration sections")
|
||||
clean = {}
|
||||
for section, fields in CONFIG_FIELDS.items():
|
||||
values = config[section]
|
||||
if not isinstance(values, dict) or set(values) - fields:
|
||||
raise ValueError("unsupported PTQ parameters: " + section)
|
||||
clean[section] = values
|
||||
def check(value, depth=0):
|
||||
if depth > 12:
|
||||
raise ValueError("PTQ parameter nesting too deep")
|
||||
if isinstance(value, dict):
|
||||
for key, val in value.items():
|
||||
if (not isinstance(key, str) or not re.fullmatch(r"[A-Za-z0-9_.:/-]{1,200}", key)
|
||||
or any(word in key.lower() for word in ("credential", "secret", "password", "token", "api_key"))):
|
||||
raise ValueError("unsafe PTQ parameter key")
|
||||
check(val, depth + 1)
|
||||
elif isinstance(value, list):
|
||||
if len(value) > 1024:
|
||||
raise ValueError("PTQ parameter list too long")
|
||||
for val in value:
|
||||
check(val, depth + 1)
|
||||
elif isinstance(value, str):
|
||||
if (len(value) > 4096 or "\x00" in value or "://" in value or ".." in value
|
||||
or value.startswith(("/", "~")) or "PRIVATE KEY" in value):
|
||||
raise ValueError("external path/credential-like PTQ parameter")
|
||||
elif value is not None and type(value) not in (bool, int, float):
|
||||
raise ValueError("unsupported PTQ parameter value")
|
||||
elif isinstance(value, float) and not math.isfinite(value):
|
||||
raise ValueError("nonfinite PTQ parameter")
|
||||
check(clean)
|
||||
if march not in {"nash-e", "nash-m", "nash-p"} or not re.fullmatch(r"[A-Za-z0-9_-]{1,128}", prefix):
|
||||
raise ValueError("invalid march/output prefix")
|
||||
if (clean["model_parameters"].get("march") != march
|
||||
or clean["model_parameters"].get("output_model_file_prefix") != prefix):
|
||||
raise ValueError("PTQ configuration target/task mismatch")
|
||||
if clean["model_parameters"].get("layer_out_dump", False) is not False:
|
||||
raise ValueError("remote stage does not return layer dumps")
|
||||
# Resource use is controlled by the outer job, never agent-supplied compiler concurrency.
|
||||
clean = json.loads(json.dumps(clean))
|
||||
clean["compiler_parameters"]["jobs"] = 1
|
||||
return clean
|
||||
|
||||
|
||||
def read_manifest(bundle):
|
||||
with tarfile.open(bundle, "r:") as packed:
|
||||
members = []
|
||||
for member in packed:
|
||||
members.append(member)
|
||||
if len(members) > MAX_FILES + 2:
|
||||
raise ValueError("too many input members")
|
||||
names = [member.name for member in members]
|
||||
if len(set(names)) != len(names) or "request.json" not in names:
|
||||
raise ValueError("duplicate or missing input manifest")
|
||||
for member in members:
|
||||
relative(member.name)
|
||||
if not member.isfile() or not 0 <= member.size <= MAX_TOTAL:
|
||||
raise ValueError("input archive contains special/oversized members")
|
||||
member = packed.getmember("request.json")
|
||||
if member.size > 2 << 20:
|
||||
raise ValueError("oversized input manifest")
|
||||
request = json.loads(packed.extractfile(member).read())
|
||||
if request.get("schema") != SCHEMA or set(request) != {
|
||||
"schema", "identity", "image", "image_source_sha256", "image_runtime_contract_sha256",
|
||||
"worker_sha256", "controller_code_sha256", "verifier_sha256", "config", "inputs"}:
|
||||
raise ValueError("invalid input manifest schema/fields")
|
||||
for key in ("image_source_sha256", "image_runtime_contract_sha256", "worker_sha256", "controller_code_sha256", "verifier_sha256"):
|
||||
if not isinstance(request[key], str) or not HEX.fullmatch(request[key]):
|
||||
raise ValueError("invalid code/runtime identity")
|
||||
if not isinstance(request["image"], str) or not re.fullmatch(r"[A-Za-z0-9._:/-]+@sha256:[0-9a-f]{64}", request["image"]):
|
||||
raise ValueError("image digest required")
|
||||
identity = request["identity"]
|
||||
if not isinstance(identity, dict) or set(identity) != {
|
||||
"run_id", "job_id", "rollout", "attempt", "task", "backend", "march", "qcfg", "recipe_submission"}:
|
||||
raise ValueError("invalid task identity")
|
||||
for key in ("run_id", "job_id", "task", "qcfg"):
|
||||
if not isinstance(identity[key], str) or not re.fullmatch(r"[A-Za-z0-9_-]{1,128}", identity[key]):
|
||||
raise ValueError("invalid task identity field")
|
||||
if (identity["backend"] != "ptq" or type(identity["attempt"]) is not int or identity["attempt"] < 0
|
||||
or type(identity["rollout"]) is not int or identity["rollout"] < -1
|
||||
or (identity["recipe_submission"] is not None and not HEX.fullmatch(identity["recipe_submission"]))):
|
||||
raise ValueError("unsupported stage identity")
|
||||
request["config"] = config_parameters(request["config"], identity["march"], identity["job_id"])
|
||||
inputs = request["inputs"]
|
||||
if not isinstance(inputs, dict) or "input/model.onnx" not in inputs:
|
||||
raise ValueError("missing ONNX input")
|
||||
if not 2 <= len(inputs) <= MAX_FILES + 1 or set(names) != {"request.json", *inputs}:
|
||||
raise ValueError("input archive/manifest members differ")
|
||||
if sum(packed.getmember(name).size for name in inputs) > MAX_TOTAL:
|
||||
raise ValueError("oversized input bundle")
|
||||
for name, row in inputs.items():
|
||||
if name != "input/model.onnx" and not re.fullmatch(r"input/calibration/[0-9]{6}\.npy", name):
|
||||
raise ValueError("unapproved model/calibration input name")
|
||||
if (not isinstance(row, dict) or set(row) != {"sha256", "bytes"}
|
||||
or not isinstance(row["sha256"], str) or not HEX.fullmatch(row["sha256"])
|
||||
or type(row["bytes"]) is not int or row["bytes"] <= 0
|
||||
or packed.getmember(name).size != row["bytes"]):
|
||||
raise ValueError("input size/hash descriptor mismatch")
|
||||
return request
|
||||
|
||||
|
||||
def unpack_inputs(bundle, request, destination):
|
||||
with tarfile.open(bundle, "r:") as packed:
|
||||
for name, row in request["inputs"].items():
|
||||
target = destination / str(relative(name))
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
h, count = hashlib.sha256(), 0
|
||||
with packed.extractfile(name) as source, target.open("xb") as out:
|
||||
for chunk in iter(lambda: source.read(1 << 20), b""):
|
||||
count += len(chunk)
|
||||
h.update(chunk)
|
||||
out.write(chunk)
|
||||
if count != row["bytes"] or h.hexdigest() != row["sha256"]:
|
||||
raise ValueError("input bytes differ from frozen manifest")
|
||||
|
||||
|
||||
def runtime_identity(request):
|
||||
root = Path("/opt/bench")
|
||||
release = json.loads((root / "RELEASE_MANIFEST.json").read_text())
|
||||
caps = json.loads((root / "COMPILE_CAPABILITIES.json").read_text())
|
||||
contract = caps["runtime_contract"]
|
||||
if (release["source_files_sha256"] != request["image_source_sha256"]
|
||||
or hashlib.sha256(canonical(contract)).hexdigest() != request["image_runtime_contract_sha256"]
|
||||
or release["runtime_contract_sha256"] != request["image_runtime_contract_sha256"]
|
||||
or caps.get("toolchain_prefix") != "/opt/bench/ptq"):
|
||||
raise ValueError("image source/toolchain identity changed")
|
||||
if ".".join(map(str, sys.version_info[:3])) != contract["python_version"]:
|
||||
raise ValueError("runtime Python version changed")
|
||||
for package, version in contract["packages"].items():
|
||||
if importlib.metadata.version(package) != version:
|
||||
raise ValueError("runtime package changed: " + package)
|
||||
for name, row in contract["commands"].items():
|
||||
if digest(row["path"]) != row["sha256"]:
|
||||
raise ValueError("runtime compiler command changed: " + name)
|
||||
if digest("/usr/local/bin/hb_verifier") != request["verifier_sha256"]:
|
||||
raise ValueError("runtime verifier command changed")
|
||||
return contract
|
||||
|
||||
|
||||
def execute(bundle, result_dir, expected_bundle_sha256):
|
||||
regular(bundle)
|
||||
if digest(bundle) != expected_bundle_sha256:
|
||||
raise ValueError("uploaded bundle identity changed")
|
||||
request = read_manifest(bundle)
|
||||
if digest(__file__) != request["worker_sha256"]:
|
||||
raise ValueError("uploaded worker code changed")
|
||||
contract = runtime_identity(request)
|
||||
result_dir.mkdir(parents=False, exist_ok=False)
|
||||
started = time.time()
|
||||
# Temporary compiler intermediates are isolated from the personal mount.
|
||||
with tempfile.TemporaryDirectory(prefix="bench-salty-ptq-") as name:
|
||||
temp = Path(name)
|
||||
unpack_inputs(bundle, request, temp)
|
||||
import onnx
|
||||
import yaml
|
||||
model = onnx.load(str(temp / "input/model.onnx"), load_external_data=False)
|
||||
if any(tensor.external_data or tensor.data_location == onnx.TensorProto.EXTERNAL
|
||||
for tensor in onnx.external_data_helper._get_all_tensors(model)):
|
||||
raise ValueError("external ONNX tensor references are unsupported")
|
||||
work = temp / "work"
|
||||
work.mkdir()
|
||||
config = request["config"]
|
||||
config["model_parameters"].update(onnx_model=str(temp / "input/model.onnx"), working_dir=str(work))
|
||||
config["calibration_parameters"]["cal_data_dir"] = str(temp / "input/calibration")
|
||||
config_file = temp / "compile.yaml"
|
||||
config_file.write_text(yaml.safe_dump(config, sort_keys=False))
|
||||
env = {"PATH": "/usr/local/bin:/usr/bin:/bin", "HOME": str(temp), "LANG": "C.UTF-8",
|
||||
"PYTHONDONTWRITEBYTECODE": "1", "PYTHONUNBUFFERED": "1",
|
||||
**{key: "1" for key in ("OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS", "NUMEXPR_NUM_THREADS")}}
|
||||
def run(argv, label):
|
||||
with (work / (label + ".stdout.log")).open("xb") as stdout, (work / (label + ".stderr.log")).open("xb") as stderr:
|
||||
return subprocess.Popen(argv, cwd=work, env=env, stdout=stdout, stderr=stderr).wait()
|
||||
rc = run([contract["commands"]["hb_compile"]["path"], "-c", str(config_file)], "compile")
|
||||
hbms = sorted(work.rglob("*.hbm"))
|
||||
info_rc, march_read = None, None
|
||||
if rc == 0 and len(hbms) == 1 and regular(hbms[0]).st_size > 0:
|
||||
info_rc = run([contract["commands"]["hb_model_info"]["path"], str(hbms[0])], "model_info")
|
||||
text = "\n".join((work / ("model_info." + kind + ".log")).read_text(errors="replace") for kind in ("stdout", "stderr"))
|
||||
targets = set(re.findall(r"\bnash-[emp]\b", text))
|
||||
if len(targets) == 1:
|
||||
march_read = targets.pop()
|
||||
bcs = sorted(work.rglob("*quantized_model.bc"))
|
||||
onnx_outputs = sorted(work.rglob("*_ptq_model.onnx"))
|
||||
verifier_rc = None
|
||||
if rc == 0 and len(bcs) == 1 and len(onnx_outputs) == 1:
|
||||
sample = sorted((temp / "input/calibration").glob("*.npy"))[0]
|
||||
verifier_rc = run(["/usr/local/bin/hb_verifier", "-m", str(onnx_outputs[0]) + "," + str(bcs[0]),
|
||||
"-i", str(sample)], "verifier")
|
||||
selected = set(hbms)
|
||||
selected.update(work.rglob("*quantized_model.bc"))
|
||||
selected.update(work.rglob("*_ptq_model.onnx"))
|
||||
selected.update(work.glob("*.log"))
|
||||
outputs, total = {}, 0
|
||||
for path in sorted(selected):
|
||||
row = {"sha256": digest(path), "bytes": regular(path).st_size}
|
||||
total += row["bytes"]
|
||||
if total > MAX_TOTAL:
|
||||
raise ValueError("oversized compile results")
|
||||
outputs[path.relative_to(work).as_posix()] = row
|
||||
archive = result_dir / "outputs.tar"
|
||||
with tarfile.open(archive, "x:") as packed:
|
||||
for path in sorted(selected):
|
||||
relative(path.relative_to(work).as_posix())
|
||||
packed.add(path, arcname=path.relative_to(work).as_posix(), recursive=False)
|
||||
ok = rc == 0 and info_rc == 0 and march_read == request["identity"]["march"] and len(hbms) == 1
|
||||
result = {"schema": SCHEMA, "request_sha256": hashlib.sha256(canonical(read_manifest(bundle))).hexdigest(),
|
||||
"bundle_sha256": expected_bundle_sha256, "identity": request["identity"], "image": request["image"],
|
||||
"worker_sha256": request["worker_sha256"], "controller_code_sha256": request["controller_code_sha256"],
|
||||
"image_source_sha256": request["image_source_sha256"],
|
||||
"image_runtime_contract_sha256": request["image_runtime_contract_sha256"],
|
||||
"verifier_sha256": request["verifier_sha256"], "verifier_exit_code": verifier_rc,
|
||||
"state": "OK" if ok else "FAIL", "exit_code": rc, "model_info_exit_code": info_rc,
|
||||
"march_read": march_read, "duration_s": time.time() - started, "outputs": outputs,
|
||||
"archive": {"sha256": digest(archive), "bytes": archive.stat().st_size}}
|
||||
(result_dir / "result.json").write_text(json.dumps(result, indent=2) + "\n")
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--bundle", type=Path, required=True)
|
||||
parser.add_argument("--bundle-sha256", required=True)
|
||||
parser.add_argument("--result-dir", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
if not HEX.fullmatch(args.bundle_sha256):
|
||||
raise ValueError("invalid bundle SHA")
|
||||
parent = args.result_dir.parent
|
||||
if (not re.fullmatch(r"/data/personal/bench-stage-[0-9a-f-]{36}", str(parent))
|
||||
or args.result_dir.name != "result" or args.bundle != parent / "bundle.tar"):
|
||||
raise ValueError("remote attempt/result ownership guard failed")
|
||||
return execute(args.bundle, args.result_dir, args.bundle_sha256)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@ -52,7 +52,7 @@ def _try_slots(pool: str, order: list[int]):
|
||||
try:
|
||||
fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
lease = d / f"{pool}-{i}.lease.json"
|
||||
if lease.exists() and not (pool != "board" and _lease_settled(lease)):
|
||||
if lease.exists():
|
||||
# A flock disappearing after a crash is not proof of remote/local cleanup.
|
||||
fcntl.flock(f, fcntl.LOCK_UN)
|
||||
f.close()
|
||||
@ -76,19 +76,6 @@ def _try_slots(pool: str, order: list[int]):
|
||||
return None, None
|
||||
|
||||
|
||||
def _lease_settled(lease: Path) -> bool:
|
||||
"""A stage killed at its phase deadline dies without releasing its slot; the stage worker has
|
||||
since recorded its process cleanup as verified. Before 10-13 these leaked until a collector
|
||||
restart (26 of 32 cpu slots in one night)."""
|
||||
try:
|
||||
x = json.loads(lease.read_text())
|
||||
status = json.loads((Path(x["execution_dir"]) / "stages" / "_queue" / x["operation_id"] / "status.json").read_text())
|
||||
except (OSError, ValueError, KeyError, TypeError):
|
||||
return False
|
||||
return (execution.process_status(x["holder"]) == "DEAD" and status.get("state") == "DONE"
|
||||
and status.get("cleanup_state") == "VERIFIED")
|
||||
|
||||
|
||||
def _queue_budget(timeout_s):
|
||||
current = execution.current()
|
||||
if timeout_s is None:
|
||||
@ -166,7 +153,7 @@ def acquire(pool: str, poll_s: float = 2.0, lane: str = "normal", march: str | N
|
||||
started, timeout_s = _queue_budget(timeout_s)
|
||||
while fh is None:
|
||||
_check_wait(started, timeout_s, stop_check)
|
||||
# paths.config() 每个进程只读一次(lru_cache):改 workers / reserve_for_agent 只对之后新起的进程生效
|
||||
# 每轮重读 config:排队中的进程也能吃到 workers / reserve_for_agent 的改动
|
||||
# (踩过:cpu 8→10 之后,已在排队的 agent stagerun 还盯着旧的 8 个槽,白等)
|
||||
n, reserve = pool_size(pool), pool_reserve(pool)
|
||||
order = (list(range(reserve)) + list(range(reserve, n))) if lane == "agent" else list(range(reserve, n))
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
agent 不在 bash 里直接跑 hb_compile。它提交:
|
||||
bench stage run compile --config workspace/compile.yaml → 打印 handle
|
||||
terminal_exec({"wait_handle": <handle>}) → 一次等到结束(bench stage wait 只看一眼)
|
||||
bench stage wait <handle> --poll 60 → 最多等待60秒;完成打印结果,未完成返回2并提示继续等待
|
||||
harness 在 agent 进程外执行阶段:按 backend 选 venv、拿资源池、看门狗、日志落 rollout/stages/。
|
||||
四个好处:编译时长与 pi 的 bash 超时脱钩;venv 不可能选错;预算与看门狗一视同仁;日志位置统一。
|
||||
|
||||
@ -21,7 +21,6 @@ import uuid
|
||||
from pathlib import Path
|
||||
|
||||
from . import paths, db, scheduler, execution as lifecycle
|
||||
from .salty_client import AmbiguousOperation
|
||||
from .stages import BY_NAME, AGENT_RUNNABLE, StageContext, short
|
||||
|
||||
|
||||
@ -32,25 +31,12 @@ def rollout_dir() -> Path:
|
||||
return Path(d).resolve()
|
||||
|
||||
|
||||
OWN_INPUTS_FROM = 18 # agent/VERSION from which the graph and the calibration set must be the agent's own
|
||||
|
||||
|
||||
def own_inputs_required(rd: Path) -> bool:
|
||||
"""Rollouts started under agent v18+: no platform graph or calibration fallback (04 rejects, bench dev hides them).
|
||||
Earlier rollouts keep the rules they started with."""
|
||||
try:
|
||||
return int(json.loads((rd / "rollout.json").read_text()).get("agent_config_version", 0)) >= OWN_INPUTS_FROM
|
||||
except (OSError, ValueError):
|
||||
return False
|
||||
|
||||
|
||||
def _ctx_from_rollout(rd: Path, agent_config: Path | None) -> StageContext:
|
||||
jd = rd.parent.parent
|
||||
job = json.loads((jd / "job.json").read_text())
|
||||
kk = int(rd.name)
|
||||
return StageContext(run_id=job["run_id"], job=job, job_dir=jd, stage_root=rd / "stages",
|
||||
artifacts=rd / "workspace" / "artifacts", rollout=kk, agent_config=agent_config,
|
||||
extra={"own_inputs_required": own_inputs_required(rd)})
|
||||
artifacts=rd / "workspace" / "artifacts", rollout=kk, agent_config=agent_config)
|
||||
|
||||
|
||||
def submit(stage: str, config: str | None, poll_hint: int = 60, *, recipe=None, submission=None) -> str:
|
||||
@ -101,7 +87,7 @@ def submit(stage: str, config: str | None, poll_hint: int = 60, *, recipe=None,
|
||||
if execution:
|
||||
execution.register_locked(state, proc, "stage", scope_token)
|
||||
print(hid)
|
||||
print('# wait: terminal_exec(' + json.dumps({'wait_handle': hid}) + ')')
|
||||
print(f"# wait: bench stage wait {hid} --poll {poll_hint}")
|
||||
return hid
|
||||
|
||||
|
||||
@ -121,7 +107,6 @@ def _completion_status(ctx, name, out):
|
||||
outputs = {'compile_yaml': recipe_output(ctx.stage_root / name / 'compile.yaml')}
|
||||
return dict(state='DONE', stage=name, result=out.state, exit_code=out.exit_code,
|
||||
error_class=out.error_class, duration_s=round(out.duration_s, 1), attempt=ctx.attempt,
|
||||
cleanup_state=(getattr(out, 'meta', None) or {}).get('cleanup_state', 'VERIFIED'),
|
||||
stdout=str(sd / 'stdout.log'), stderr=str(sd / 'stderr.log'), meta=str(sd / 'meta.json'),
|
||||
measurements=out.measurements, finished_at=meta['finished_at'],
|
||||
meta_sha256=hashlib.sha256(raw).hexdigest(), outputs=outputs, db_registered=False,
|
||||
@ -129,20 +114,13 @@ def _completion_status(ctx, name, out):
|
||||
|
||||
|
||||
MAX_COMPILE_ATTEMPTS = 5 # 每条轨迹的主预算;改这里要同步改 agent/prompts/e2e.md 里的次数
|
||||
# 容器被杀(多是内存超限,下次自动加倍)、执行器退出、超过阶段时限、看门狗判卡死(一小时没日志没 CPU):
|
||||
# 系统的错,不扣模型的次数;每条轨迹最多免这么多次,防止一个怎么都编不过的模型无限重试。
|
||||
FREE_INFRA_COMPILES = 3
|
||||
INFRA_COMPILE_ERRORS = ('REMOTE_WORKER_FAILED', 'STAGE_EXECUTOR_EXIT', 'PHASE_EXECUTION_DEADLINE_EXCEEDED', 'stalled')
|
||||
|
||||
|
||||
def compile_attempts_used(con, run_id, job_id, rollout) -> int:
|
||||
"""Compiles that got a slot; a compile that timed out in the queue never ran and is free.
|
||||
Read from the stage ledger, so resume/recovery cannot reset it."""
|
||||
total, infra = con.execute(
|
||||
f"SELECT COUNT(*), COALESCE(SUM(error_class IN ({','.join('?' * len(INFRA_COMPILE_ERRORS))})),0) FROM stages "
|
||||
"WHERE run_id=? AND job_id=? AND rollout=? AND stage='05-compile' AND lane!='queued'",
|
||||
(*INFRA_COMPILE_ERRORS, run_id, job_id, rollout)).fetchone()
|
||||
return total - min(infra, FREE_INFRA_COMPILES)
|
||||
return con.execute("SELECT COUNT(*) FROM stages WHERE run_id=? AND job_id=? AND rollout=? "
|
||||
"AND stage='05-compile' AND lane!='queued'", (run_id, job_id, rollout)).fetchone()[0]
|
||||
|
||||
|
||||
def _exec_impl(hd: Path, *, rd=None, publish_status=None, protected=False) -> int:
|
||||
@ -197,16 +175,6 @@ def _exec_impl(hd: Path, *, rd=None, publish_status=None, protected=False) -> in
|
||||
# attempt = 该 rollout 内此阶段已提交次数(agent 自己的重试是同一条轨迹)
|
||||
prev = sorted((rd / "stages" / stage.name).glob("a*")) if (rd / "stages" / stage.name).exists() else []
|
||||
ctx.attempt = len(prev)
|
||||
pending_salty = None
|
||||
from . import salty_stage
|
||||
if salty_stage.selected(ctx, stage.name):
|
||||
pending_salty = salty_stage.pending_attempt(ctx.stage_root)
|
||||
if pending_salty is not None:
|
||||
ctx.attempt = pending_salty # Continue the exact durable job ID, never a new attempt.
|
||||
if ctx.agent_config:
|
||||
configs = sorted((ctx.stage_root / '04-calibrate').glob('a*/agent_config.yaml'))
|
||||
if not configs or hashlib.sha256(configs[-1].read_bytes()).hexdigest() != hashlib.sha256(ctx.agent_config.read_bytes()).hexdigest():
|
||||
raise AmbiguousOperation('unresolved Salty attempt cannot accept new compiler overrides')
|
||||
# 05-compile 一律走 long 车道(stall_s 3600)。hbdk4 的 BPU 编译阶段会长时间不打日志:
|
||||
# 实测 convnext.tiny 编译 3342s 才成功,而 900s 的 stall 判定把同一个 job 的另外两条 rollout
|
||||
# 在正常编译中途 SIGTERM 掉了(记成 error_class=stalled,白烧一条轨迹)。
|
||||
@ -217,10 +185,7 @@ def _exec_impl(hd: Path, *, rd=None, publish_status=None, protected=False) -> in
|
||||
limit = MAX_COMPILE_ATTEMPTS
|
||||
with db.connect() as con:
|
||||
used = compile_attempts_used(con, ctx.run_id, ctx.job["job_id"], ctx.rollout)
|
||||
# 已记录过提交的 Salty 编译只做找回、绝不重提,它早已计入 used,不受上限阻拦;
|
||||
# 只准备好、从未提交的仍按上限拒绝。
|
||||
recovering = pending_salty is not None and salty_stage.dispatch_recorded(ctx.stage_root, pending_salty)
|
||||
if used >= limit and not recovering:
|
||||
if used >= limit:
|
||||
put_status({"state": "DONE", "result": "FAIL", "exit_code": 2, "error_class": "COMPILE_BUDGET_EXHAUSTED",
|
||||
"duration_s": 0, "stdout": "", "stderr": "", "meta": "", "finished_at": time.time(),
|
||||
"note": f"本轨迹已启动 {used}/{limit} 次编译,不再编译;用已有产物完成 board/eval 并如实报告"})
|
||||
@ -231,7 +196,7 @@ def _exec_impl(hd: Path, *, rd=None, publish_status=None, protected=False) -> in
|
||||
# (v4/v5 两条 canary 都踩了)。04 对 PTQ 只是"合并 yaml + 落盘"(2 s),这里内联补跑:
|
||||
# 带了 --config 就按新配置重新合并;没带且 04 没跑过也补一次默认合并。
|
||||
yp = rd / "stages" / "04-calibrate" / "compile.yaml"
|
||||
if pending_salty is None and ctx.job.get("backend") == "ptq" and (ctx.job.get("recipe") or ctx.agent_config or not yp.exists()):
|
||||
if ctx.job.get("backend") == "ptq" and (ctx.job.get("recipe") or ctx.agent_config or not yp.exists()):
|
||||
cal = BY_NAME["04-calibrate"]()
|
||||
cprev = sorted((rd / "stages" / cal.name).glob("a*")) if (rd / "stages" / cal.name).exists() else []
|
||||
cctx = _ctx_from_rollout(rd, ctx.agent_config)
|
||||
@ -301,7 +266,6 @@ def _exec_impl(hd: Path, *, rd=None, publish_status=None, protected=False) -> in
|
||||
lifecycle.current().quarantine("BOARD_CLEANUP_UNCONFIRMED")
|
||||
sd = rd / "stages" / stage.name / f"a{ctx.attempt:02d}"
|
||||
status = {"state": "DONE", "result": out.state, "exit_code": out.exit_code, "error_class": out.error_class,
|
||||
"cleanup_state": (getattr(out, 'meta', None) or {}).get('cleanup_state', 'VERIFIED'),
|
||||
"duration_s": round(out.duration_s, 1), "attempt": ctx.attempt,
|
||||
"board_name": (ctx.extra.get("board") or {}).get("name"),
|
||||
"stdout": str(sd / "stdout.log"), "stderr": str(sd / "stderr.log"), "meta": str(sd / "meta.json"),
|
||||
@ -323,38 +287,51 @@ def _exec(hd: Path, *, rd=None, publish_status=None, protected=False) -> int:
|
||||
try:
|
||||
with execution.scope(req["stage"], hd.name) if execution else nullcontext():
|
||||
return _exec_impl(hd, rd=rd, publish_status=publish_status, protected=protected)
|
||||
except (lifecycle.ExecutionStopped, AmbiguousOperation) as exc:
|
||||
unknown = isinstance(exc, AmbiguousOperation)
|
||||
status = {"state": "DONE", "result": "FAIL", "exit_code": 2,
|
||||
"error_class": "SIDE_EFFECT_UNKNOWN" if unknown else str(exc),
|
||||
except lifecycle.ExecutionStopped as exc:
|
||||
status = {"state": "DONE", "result": "FAIL", "exit_code": 2, "error_class": str(exc),
|
||||
"duration_s": 0, "stdout": "", "stderr": "", "meta": "", "finished_at": time.time()}
|
||||
if unknown:
|
||||
status.update(cleanup_state='QUARANTINED', reconciliation_required=True)
|
||||
with execution.mutation() if execution else nullcontext():
|
||||
lifecycle.atomic_json(hd / "status.json", status)
|
||||
if publish_status:
|
||||
publish_status(status)
|
||||
if unknown and execution:
|
||||
execution.quarantine('SIDE_EFFECT_UNKNOWN')
|
||||
return 2
|
||||
|
||||
|
||||
def wait(handle: str, poll: int = 60) -> int:
|
||||
"""不再轮询,看一眼就返回:结束了打印结果;没结束返回 2,并指向 terminal_exec wait_handle(一次等到阶段结束、不占轮数)。
|
||||
10-16:efficientvit.b0 400 轮里 307 轮是 bench stage wait,每次还往上下文里加一段输出。"""
|
||||
def wait(handle: str, poll: int = 60, quiet: bool = False, max_wait_s: float = 60) -> int:
|
||||
"""单次最多等 60 秒;返回 2 表示仍在排队/运行,阶段继续执行。"""
|
||||
if not 0 < max_wait_s <= 60:
|
||||
raise ValueError("stage wait budget must be in (0, 60] seconds")
|
||||
hd = rollout_dir() / "handles" / handle
|
||||
if not hd.exists():
|
||||
raise SystemExit(f"handle 不存在:{handle}")
|
||||
st = json.loads((hd / "status.json").read_text())
|
||||
if st.get("state") != "DONE":
|
||||
print(f"{handle}: {st.get('state')}"
|
||||
f"{'(阶段计算已结束,等待数据库登记;无需重跑)' if st.get('registration') == 'PENDING' else ''}"
|
||||
f"。不要反复查:调用 terminal_exec({json.dumps({'wait_handle': handle})}),它一次等到这个阶段结束,不占轮数。")
|
||||
return 2
|
||||
t0 = time.time()
|
||||
deadline = time.monotonic() + max_wait_s
|
||||
while True:
|
||||
st = json.loads((hd / "status.json").read_text())
|
||||
if st.get("state") == "DONE":
|
||||
break
|
||||
if not quiet:
|
||||
since = st.get("started_at") or st.get("queued_at") or t0
|
||||
print(f"[{time.strftime('%H:%M:%S')}] {handle}: {st.get('state')}"
|
||||
f"{'(等 ' + st.get('pool', '?') + ' 池槽位)' if st.get('state') == 'QUEUED' else ''}"
|
||||
f"{'(阶段计算已结束,等待数据库登记;无需重跑)' if st.get('registration') == 'PENDING' else ''}"
|
||||
f" 已 {int(time.time() - since)}s", flush=True)
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
print(f"(已等 {int(max_wait_s - remaining)}s 仍未结束;这不是失败,再调一次 bench stage wait {handle} 继续等)")
|
||||
return 2
|
||||
time.sleep(min(max(1, poll), remaining))
|
||||
_print_status(st)
|
||||
return 0 if st["result"] in ("OK", "SKIPPED") else 1
|
||||
|
||||
|
||||
def status(handle: str) -> int:
|
||||
hd = rollout_dir() / "handles" / handle
|
||||
st = json.loads((hd / "status.json").read_text())
|
||||
_print_status(st)
|
||||
return 0
|
||||
|
||||
|
||||
def _print_status(st: dict) -> None:
|
||||
if st.get("state") != "DONE":
|
||||
print(json.dumps(st)); return
|
||||
|
||||
@ -54,8 +54,6 @@ def work_dir(ctx: StageContext) -> Path:
|
||||
相等跳过自拷日志(horizon_tc_ui/hb_compile.py:50),不等会在编译成功后删掉日志再抛错。
|
||||
叶子名暂留 05-compile-work:在飞 job 已按此路径写完,上移一层会让 s06_hbm 的 hb_compile.log 落空。
|
||||
"""
|
||||
if ctx.extra.get("salty_work_dir"):
|
||||
return Path(ctx.extra["salty_work_dir"])
|
||||
rel = f"{ctx.run_id}/{ctx.job['job_id']}" + (f"/r{ctx.rollout:02d}" if ctx.rollout >= 0 else "/oracle")
|
||||
d = paths.expand(paths.config()["paths"]["work_root"]) / rel / "05-compile-work"
|
||||
if ctx.job.get("recipe"):
|
||||
@ -233,13 +231,8 @@ class Stage:
|
||||
return d
|
||||
|
||||
def run(self, ctx: StageContext) -> StageOutcome:
|
||||
from .. import salty_remote
|
||||
if salty_remote.selected(ctx, self.name):
|
||||
return salty_remote.run(self, ctx)
|
||||
sd = self.stage_dir(ctx)
|
||||
from .. import salty_stage
|
||||
remote = salty_stage.selected(ctx, self.name)
|
||||
if ctx.job.get("recipe") and (sd / "meta.json").exists() and not (remote and (sd / "salty/plan.json").exists()):
|
||||
if ctx.job.get("recipe") and (sd / "meta.json").exists():
|
||||
raise ValueError("recipe stage attempt already exists; submit a new attempt")
|
||||
t0 = time.time()
|
||||
meta = {"stage": self.name, "level": self.level, "pool": self.pool, "attempt": ctx.attempt,
|
||||
@ -256,8 +249,8 @@ class Stage:
|
||||
meta["board"] = {k: ctx.extra["board"].get(k) for k in ("name", "slot", "march", "board_type")}
|
||||
try:
|
||||
if ctx.job.get("task_contract"):
|
||||
from ..eval import contract_holds, task_contract
|
||||
if not contract_holds(ctx.job["task_contract"], task_contract(ctx.job["task"])):
|
||||
from ..eval import task_contract
|
||||
if task_contract(ctx.job["task"]) != ctx.job["task_contract"]:
|
||||
raise ValueError("frozen task implementation changed; create a new run")
|
||||
if ctx.job.get("toolchain"):
|
||||
from ..toolchains import validate
|
||||
@ -276,12 +269,8 @@ class Stage:
|
||||
would = f"<command unavailable in dry-run: {e}>"
|
||||
out = StageOutcome("OK", 0, None, 0.0, {"dry_run": True, "would_run": would})
|
||||
else:
|
||||
if remote:
|
||||
ctx.extra.pop("salty_work_dir", None)
|
||||
cmd = self.command(ctx)
|
||||
if remote:
|
||||
out = salty_stage.run(self, ctx, sd)
|
||||
elif cmd is None:
|
||||
if cmd is None:
|
||||
out = self.native(ctx, sd)
|
||||
else:
|
||||
wd = ctx.watchdog_params()
|
||||
@ -306,23 +295,7 @@ class Stage:
|
||||
except Exception as e: # harness 自身的 bug 也要落盘,不能吞
|
||||
with (sd / "stderr.log").open("a") as log:
|
||||
log.write(f"\n[harness exception] {type(e).__name__}: {e}\n")
|
||||
uncertain_error = getattr(e, "error_class", None) == "SIDE_EFFECT_UNKNOWN" or getattr(e, "cleanup_state", None) == "QUARANTINED"
|
||||
unresolved = uncertain_error or remote and (sd / "salty/transport").exists()
|
||||
out = StageOutcome("FAIL", -1, "SIDE_EFFECT_UNKNOWN" if unresolved else "unknown", time.time() - t0,
|
||||
{"harness_exception": repr(e), **({"cleanup_state": "QUARANTINED",
|
||||
**({"salty_receipt": str(sd / "salty/plan.json")} if remote else {}),
|
||||
**({"process_receipt": str(e.receipt_path)} if hasattr(e, "receipt_path") else {})} if unresolved else {})})
|
||||
if out.error_class == "SIDE_EFFECT_UNKNOWN" or out.meta.get("cleanup_state") == "QUARANTINED":
|
||||
from .. import execution
|
||||
if execution.current():
|
||||
execution.current().quarantine("SIDE_EFFECT_UNKNOWN")
|
||||
elif (self.pool == "board" and not remote and out.state != "SKIPPED" and os.environ.get("BENCH_BOARD_RECEIPT_DIR")
|
||||
and not any(Path(os.environ["BENCH_BOARD_RECEIPT_DIR"]).glob("*.json"))):
|
||||
# A board session writes an UNKNOWN receipt before its first board command (board_perf.bench) and this
|
||||
# stage's processes are gone now: no receipt proves the board was never touched. Without one, a failure
|
||||
# before dispatch read as BOARD_CLEANUP_UNCONFIRMED and ended the rollout (10-15: two lost that way).
|
||||
from ..tools.board_perf import not_started
|
||||
not_started("STAGE_ENDED_BEFORE_BOARD_DISPATCH")
|
||||
out = StageOutcome("FAIL", -1, "unknown", time.time() - t0, {"harness_exception": repr(e)})
|
||||
out.duration_s = out.duration_s or (time.time() - t0)
|
||||
meta.update({"state": out.state, "exit_code": out.exit_code, "error_class": out.error_class,
|
||||
"duration_s": round(out.duration_s, 3), "finished_at": time.time(),
|
||||
|
||||
@ -29,8 +29,8 @@ def baseline_identity(job: dict) -> dict | None:
|
||||
|
||||
def validate_baseline(job: dict) -> None:
|
||||
if paths.cache_contract(job, validate=True) is not None:
|
||||
from ..eval import contract_holds, task_contract
|
||||
if not contract_holds(job.get("task_contract") or {}, task_contract(job["task"])):
|
||||
from ..eval import task_contract
|
||||
if task_contract(job["task"]) != job.get("task_contract"):
|
||||
raise ValueError("frozen task implementation changed; create a new run")
|
||||
|
||||
|
||||
|
||||
@ -5,7 +5,6 @@ rollout 里 agent 提交的 --config 在这里与 harness 钉死项合并。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import shlex
|
||||
from pathlib import Path
|
||||
@ -108,22 +107,10 @@ class Calibrate(Stage):
|
||||
|
||||
meta = {"calib_dir": str(cd), "calib_npy_dir": str(calib_npy_dir(cd)), "calib_source": source,
|
||||
"calib_n": manifest.get("count"), "calib_manifest_sha": manifest.get("sha256")}
|
||||
own = {}
|
||||
if (agent_overrides or ctx.extra.get("own_inputs_required")) and not job.get("recipe") and job["backend"] == "ptq":
|
||||
try:
|
||||
own = own_inputs(ctx, agent_overrides or {}, onnx)
|
||||
except (OSError, ValueError) as exc:
|
||||
(stage_dir / "stderr.log").write_text(f"{exc}\n")
|
||||
(stage_dir / "stdout.log").touch()
|
||||
return StageOutcome("FAIL", 2, "own_input_invalid", 0.0, {"reason": str(exc)[:500]})
|
||||
onnx = own.get("onnx", onnx)
|
||||
meta.update(own.get("meta", {}))
|
||||
if job["backend"] == "ptq":
|
||||
cfg = ptq.compile_config(job, onnx, calib_npy_dir(cd), work_dir(ctx), agent_overrides)
|
||||
if job.get("recipe") or not own.get("calib"):
|
||||
if job.get("recipe"):
|
||||
cfg["calibration_parameters"]["cal_data_dir"] = str(calib_npy_dir(cd))
|
||||
else:
|
||||
cfg["calibration_parameters"]["cal_data_dir"] = str(own["calib"])
|
||||
yp = stage_dir / "compile.yaml" if job.get("recipe") else compile_yaml_path(ctx)
|
||||
yp.parent.mkdir(parents=True, exist_ok=True)
|
||||
ptq.write_compile_config(cfg, yp)
|
||||
@ -138,70 +125,6 @@ class Calibrate(Stage):
|
||||
return StageOutcome("OK", 0, None, 0.0, meta, {"calib": meta})
|
||||
|
||||
|
||||
def io_signature(path: Path) -> list:
|
||||
"""Graph inputs (initializers excluded) and outputs as (element type, shape): what the platform's
|
||||
baseline, board runner and evaluator rely on. Names are free."""
|
||||
import onnx
|
||||
graph = onnx.load(str(path), load_external_data=False).graph
|
||||
weights = {t.name for t in graph.initializer}
|
||||
def sig(values):
|
||||
return [(v.type.tensor_type.elem_type,
|
||||
[d.dim_value if d.HasField("dim_value") else d.dim_param for d in v.type.tensor_type.shape.dim])
|
||||
for v in values]
|
||||
return [sig(v for v in graph.input if v.name not in weights), sig(graph.output)]
|
||||
|
||||
|
||||
def own_inputs(ctx: StageContext, overrides: dict, platform_onnx: Path | None) -> dict:
|
||||
"""The agent's own graph / calibration set, named in compile.yaml as workspace paths and made in
|
||||
`bench dev` (the development environment shares this workspace). Same I/O as the platform graph keeps
|
||||
baseline, board and evaluation unchanged."""
|
||||
ws = (ctx.stage_root.parent / "workspace").resolve()
|
||||
out, meta = {}, {}
|
||||
if ctx.extra.get("own_inputs_required"):
|
||||
need = [f"{section}.{key}" for section, key in (("model_parameters", "onnx_model"), ("calibration_parameters", "cal_data_dir"))
|
||||
if not (overrides.get(section) or {}).get(key)]
|
||||
if need:
|
||||
raise ValueError("no platform graph or calibration set to fall back on: compile.yaml must name your own "
|
||||
+ " and ".join(need) + " (workspace paths, made in bench dev)")
|
||||
|
||||
def resolve(rel: str) -> Path:
|
||||
p = (ws / rel).resolve()
|
||||
if not p.is_relative_to(ws):
|
||||
raise ValueError(f"{rel} is outside the workspace")
|
||||
return p
|
||||
|
||||
rel = (overrides.get("model_parameters") or {}).get("onnx_model")
|
||||
if rel:
|
||||
path = resolve(rel)
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(f"model_parameters.onnx_model: {rel} not found in the development workspace "
|
||||
"(files made in bench dev are there; local edits sync on the next bench dev)")
|
||||
if not (platform_onnx and platform_onnx.exists()):
|
||||
raise ValueError("run `bench stage run export` first: your graph is checked against the platform graph's inputs/outputs")
|
||||
mine, ref = io_signature(path), io_signature(platform_onnx)
|
||||
if mine != ref:
|
||||
raise ValueError(f"model_parameters.onnx_model inputs/outputs {mine} differ from the platform graph {ref}; "
|
||||
"keep their count, order, element type and shape so evaluation stays comparable")
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as f:
|
||||
for chunk in iter(lambda: f.read(1 << 20), b""):
|
||||
digest.update(chunk)
|
||||
# ponytail: external-data tensors (graphs > 2 GB) are not hashed; the main file pins the structure.
|
||||
out["onnx"] = path
|
||||
meta["graph"] = {"source": "workspace", "path": rel, "sha256": digest.hexdigest()}
|
||||
rel = (overrides.get("calibration_parameters") or {}).get("cal_data_dir")
|
||||
if rel:
|
||||
path = resolve(rel)
|
||||
n = sum(1 for _ in path.glob("*.npy")) if path.is_dir() else 0
|
||||
if not n:
|
||||
raise ValueError(f"calibration_parameters.cal_data_dir: {rel} has no .npy files in the development workspace")
|
||||
out["calib"] = path
|
||||
meta.update(calib_source="workspace", calib_npy_dir=str(path), calib_n=n,
|
||||
calibration={"source": "workspace", "path": rel, "count": n})
|
||||
out["meta"] = meta
|
||||
return out
|
||||
|
||||
|
||||
def _make_synthetic(out: Path, input_spec: str, n: int) -> None:
|
||||
"""占据类第三级回退:输入范围内的随机张量。G2 结果只作参考(pass-weak)。"""
|
||||
import hashlib
|
||||
|
||||
@ -12,8 +12,6 @@ import re
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from .. import watchdog
|
||||
from ..backends import get as backend
|
||||
from ..registry import boards
|
||||
@ -49,10 +47,6 @@ class Hbm(Stage):
|
||||
pool = "cpu"
|
||||
|
||||
def native(self, ctx: StageContext, stage_dir: Path) -> StageOutcome:
|
||||
from .. import salty_stage
|
||||
remote = salty_stage.hbm(ctx, stage_dir)
|
||||
if remote is not None:
|
||||
return remote
|
||||
job = ctx.job
|
||||
wd = work_dir(ctx)
|
||||
cands = ([recipe_artifact(ctx, "05-compile", "hbm")] if job.get("recipe") else
|
||||
@ -107,22 +101,18 @@ class Hbm(Stage):
|
||||
m["verifier_cosine"] = cos
|
||||
m["verifier_ok"] = (rv.state == "OK") and (cos is None or cos >= thr)
|
||||
m["verifier_rc"] = rv.exit_code
|
||||
if rv.state != "OK" and cos is None:
|
||||
# hb_verifier died before printing its table ("launch function crashed", rc 255): nothing was
|
||||
# compared, so consistency is unknown (G1 unverified), not a mismatch.
|
||||
m["verifier_ok"] = None
|
||||
m["verifier_note"] = f"hb_verifier {rv.state} rc={rv.exit_code} before measuring"
|
||||
else:
|
||||
m["verifier_ok"] = None
|
||||
m["verifier_note"] = "ptq_model.onnx or calib sample missing"
|
||||
else:
|
||||
m["verifier_ok"] = True # QAT 的一致性在 08-eval 的 G2 里判
|
||||
|
||||
# hbm ≡ bc 模拟器对比:agent 在配置里要了才跑,只报数不判分(10-14 起)。x86 模拟器每张图在 Salty 上要几十分钟,
|
||||
# 以前每次必跑 2 张,撞 2 小时上限 103 次;388 次报出的 13 次不一致 hb_verifier 都已先判掉;08-eval 在板上跑真 HBM。
|
||||
n = ((yaml.safe_load(ctx.agent_config.read_text()) or {}) if ctx.agent_config else {}).get("verification_parameters", {}).get("hbm_bc_samples", 0)
|
||||
m["hbm_bc_cos_min"] = _hbm_bc_spotcheck(ctx, stage_dir, dst, bc_path(ctx), be, wdp, n) if n and bc else None
|
||||
m["hbm_bc_spotcheck"] = f"{n} samples, diagnostic only" if n and bc else "not requested; board HBM verification remains required"
|
||||
# ---- hbm ≡ bc 抽查(编译一致性诊断;08-eval 另跑板端 HBM)
|
||||
if bc and job["backend"] == "ptq":
|
||||
m["hbm_bc_cos_min"] = _hbm_bc_spotcheck(ctx, stage_dir, dst, bc_path(ctx), be, wdp)
|
||||
if m["hbm_bc_cos_min"] is not None and m["hbm_bc_cos_min"] < 0.999:
|
||||
m["verifier_ok"] = False
|
||||
m["verifier_note"] = f"hbm vs bc cosine {m['hbm_bc_cos_min']:.6f} < 0.999"
|
||||
|
||||
# ---- 编译日志扫 ERROR(只看 hb_compile 自己的日志级别,不看 INFO 里的字样)
|
||||
clog = ""
|
||||
@ -153,7 +143,7 @@ class Hbm(Stage):
|
||||
"toolchain": job.get("toolchain"), "size": dst.stat().st_size, "work_dir": str(wd), **output_meta}, {"G1": m})
|
||||
|
||||
|
||||
def _hbm_bc_spotcheck(ctx: StageContext, stage_dir: Path, hbm: Path, bc: Path, be, wdp: dict, n: int) -> float | None:
|
||||
def _hbm_bc_spotcheck(ctx: StageContext, stage_dir: Path, hbm: Path, bc: Path, be, wdp: dict, n: int = 2) -> float | None:
|
||||
"""同 n 张校准样本分别过 hbm(run_sim) 与 bc(HBIR run),返回逐输出余弦的最小值。任何一步失败返回 None(记录,不判)。"""
|
||||
try:
|
||||
import numpy as np
|
||||
@ -169,7 +159,7 @@ def _hbm_bc_spotcheck(ctx: StageContext, stage_dir: Path, hbm: Path, bc: Path, b
|
||||
fo = stage_dir / f"spot_{tag}.npz"
|
||||
r = watchdog.run([str(be.python(ctx.job)), "-m", "harness.tools.infer_hbm", "--model", str(model), "--inputs", str(fin), "--out", str(fo)],
|
||||
cwd=work_dir(ctx), stdout=stage_dir / f"spot_{tag}.log", stderr=stage_dir / f"spot_{tag}.err.log",
|
||||
env=Hbm().env(ctx), stall_s=wdp["stall_s"], poll_s=5)
|
||||
env=self_env(ctx), stall_s=wdp["stall_s"], poll_s=5)
|
||||
if r.state != "OK":
|
||||
return None
|
||||
outs[tag] = np.load(fo)
|
||||
@ -184,6 +174,10 @@ def _hbm_bc_spotcheck(ctx: StageContext, stage_dir: Path, hbm: Path, bc: Path, b
|
||||
return None
|
||||
|
||||
|
||||
def self_env(ctx: StageContext) -> dict:
|
||||
return Hbm().env(ctx)
|
||||
|
||||
|
||||
def _find_march(info: str) -> str | None:
|
||||
low = info.lower()
|
||||
strings = boards().get("march_strings", {})
|
||||
|
||||
@ -19,20 +19,17 @@ from . import stagerun, execution as lifecycle, watchdog
|
||||
from .stages import AGENT_RUNNABLE
|
||||
|
||||
|
||||
def read_regular(parent_fd, name, limit=1024 * 1024):
|
||||
def read_regular(parent_fd, name):
|
||||
fd = os.open(name, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=parent_fd)
|
||||
try:
|
||||
if not stat.S_ISREG(os.fstat(fd).st_mode) or (limit is not None and os.fstat(fd).st_size > limit):
|
||||
if not stat.S_ISREG(os.fstat(fd).st_mode) or os.fstat(fd).st_size > 1024 * 1024:
|
||||
raise ValueError('ticket/config must be a regular file <= 1 MiB')
|
||||
if limit is not None:
|
||||
return os.read(fd, limit + 1)
|
||||
with os.fdopen(os.dup(fd), 'rb') as stream:
|
||||
return stream.read()
|
||||
return os.read(fd, 1024 * 1024 + 1)
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def config_bytes(ws, value, limit=1024 * 1024):
|
||||
def config_bytes(ws, value):
|
||||
p = Path(value)
|
||||
rel = p.relative_to(ws)
|
||||
if not rel.parts or '..' in rel.parts:
|
||||
@ -42,54 +39,29 @@ def config_bytes(ws, value, limit=1024 * 1024):
|
||||
for part in rel.parts[:-1]:
|
||||
nxt = os.open(part, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, dir_fd=fd)
|
||||
os.close(fd); fd = nxt
|
||||
return read_regular(fd, rel.name, limit)
|
||||
return read_regular(fd, rel.name)
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def workspace_relative(ws, value, field):
|
||||
"""A path the agent names inside its workspace, kept relative: 04 resolves it in the development
|
||||
environment, where files made by `bench dev` live (they may not exist on the controller)."""
|
||||
p = Path(value) if isinstance(value, str) and value else None
|
||||
if p is not None and p.is_absolute():
|
||||
try:
|
||||
p = p.relative_to(ws)
|
||||
except ValueError:
|
||||
p = None
|
||||
if p is None or not p.parts or '..' in p.parts:
|
||||
raise ValueError(f'{field} must be a path inside the workspace')
|
||||
return p.as_posix()
|
||||
|
||||
|
||||
def snapshot_config(ws, value):
|
||||
"""Keep path-bearing options from smuggling outer-worker file access."""
|
||||
cfg = yaml.safe_load(config_bytes(ws, value)) or {}
|
||||
allowed = {
|
||||
'input_parameters': {'input_type_rt', 'input_type_train', 'input_layout_train', 'input_layout_rt',
|
||||
'input_name', 'input_shape', 'mean_value', 'scale_value', 'std_value'},
|
||||
'calibration_parameters': {'calibration_type', 'max_percentile', 'per_channel', 'cal_data_dir',
|
||||
'calibration_parameters': {'calibration_type', 'max_percentile', 'per_channel',
|
||||
'run_on_cpu', 'run_on_bpu', 'optimization', 'quant_config'},
|
||||
'compiler_parameters': {'compile_mode', 'optimize_level', 'jobs', 'debug'},
|
||||
'verification_parameters': {'hbm_bc_samples'},
|
||||
}
|
||||
if not isinstance(cfg, dict):
|
||||
raise ValueError('config must be a mapping')
|
||||
for section, values in cfg.items():
|
||||
if section == 'model_parameters':
|
||||
# The graph is the agent's (04 checks its inputs/outputs against the platform graph);
|
||||
# march, working_dir and output names stay pinned, so other keys are ignored as before.
|
||||
if isinstance(values, dict) and 'onnx_model' in values:
|
||||
values['onnx_model'] = workspace_relative(ws, values['onnx_model'], 'model_parameters.onnx_model')
|
||||
continue
|
||||
continue # Existing backend ignores/pins this section.
|
||||
if section not in allowed or not isinstance(values, dict) or set(values)-allowed[section]:
|
||||
extra = sorted(set(values) - allowed[section]) if section in allowed and isinstance(values, dict) else []
|
||||
raise ValueError('unsupported config keys: ' + str(section) + (f' {extra}' if extra else ''))
|
||||
n = (cfg.get('verification_parameters') or {}).get('hbm_bc_samples', 0)
|
||||
if type(n) is not int or not 0 <= n <= 16:
|
||||
raise ValueError('verification_parameters.hbm_bc_samples must be an integer in [0, 16]')
|
||||
raise ValueError('unsupported config keys: ' + str(section))
|
||||
cal = cfg.get('calibration_parameters', {})
|
||||
if 'cal_data_dir' in cal:
|
||||
cal['cal_data_dir'] = workspace_relative(ws, cal['cal_data_dir'], 'calibration_parameters.cal_data_dir')
|
||||
quant = cal.get('quant_config')
|
||||
if isinstance(quant, str):
|
||||
p = Path(quant)
|
||||
@ -251,14 +223,6 @@ class Worker:
|
||||
status = queue / 'status.json'
|
||||
if status.exists() and json.loads(status.read_text()).get('state') == 'DONE':
|
||||
result = json.loads(status.read_text())
|
||||
ledger = self.execution.snapshot().get('operations', {}).get(hid) if self.execution else None
|
||||
if (result.get('error_class') != 'SIDE_EFFECT_UNKNOWN' and result.get('db_registered') is not False
|
||||
and previous.get('state') == 'DONE' and ledger == previous):
|
||||
# Settled before a pause (same skip as reconcile_completed). A later attempt of the
|
||||
# rollout may overwrite its outputs (one .hbm path per rollout), so re-proving them
|
||||
# on resume quarantined every multi-compile rollout (10-12/13: 19 rollouts).
|
||||
self._publish(fd, result)
|
||||
return
|
||||
if result.get('error_class') == 'SIDE_EFFECT_UNKNOWN':
|
||||
if self.execution:
|
||||
self.execution.quarantine('SIDE_EFFECT_UNKNOWN')
|
||||
|
||||
@ -19,39 +19,6 @@ from . import execution as lifecycle, paths, scheduler, watchdog
|
||||
from .recipe_runtime import regular_bytes
|
||||
|
||||
VERSION = 'bench.terminal/1'
|
||||
# `bench dev --board` (bin/stageclient.py) marks the command: the collectors' request check (prepare) predates
|
||||
# board commands and only admits the development target, while run() executes in a fresh process.
|
||||
BOARD_MARK = '#bench:board\n'
|
||||
BOARD_MAX_S = 1800 # the board is shared with formal board/eval stages
|
||||
BOARD_MAX_BYTES = 512 << 20 # dev-only files come back from personal storage at ~0.5 MB/s (10-16: 1.2 GB took 40 min)
|
||||
DEV_QUEUE_S = 3 * 3600 # bench dev: waiting for a dev slot / a Salty container is not the command's own time
|
||||
|
||||
|
||||
class BoardRefused(ValueError):
|
||||
"""A board command that cannot run; the message goes back to the agent as the command's result."""
|
||||
|
||||
|
||||
def board_request(ctx, rd, hd, req):
|
||||
"""A development request marked for the board: the requesting directory (local view plus the files
|
||||
that exist only in the development workspace, e.g. a compiled HBM) is what goes up."""
|
||||
if not ctx.extra.get('own_inputs_required'):
|
||||
raise BoardRefused('BOARD_TERMINAL_UNAVAILABLE: board commands are open to agent v18+ tasks')
|
||||
from . import salty_dev
|
||||
small = '新建一个只放 .hbm、脚本和几个输入的小目录,cd 进去再运行 bench dev --board'
|
||||
# Size first: 10-16 three of four self-tests ran from the whole workspace (1.2 GB), spent 40 min fetching
|
||||
# it, passed the deadline and failed with no reason given.
|
||||
if salty_dev.remote_size(rd, req['cwd'], BOARD_MAX_BYTES) is None:
|
||||
raise BoardRefused(f'BOARD_INPUT_TOO_LARGE: {req["cwd"]} 在开发环境里超过 {BOARD_MAX_BYTES >> 20} MiB(或子目录超过 40 个),'
|
||||
f'从开发环境取回约 0.5 MB/s,时限内传不完;{small}')
|
||||
started = time.monotonic()
|
||||
remote = hd / 'dev-files'
|
||||
salty_dev.download_dir(rd, req['cwd'], remote)
|
||||
if time.monotonic() >= req['deadline_monotonic'] - 60:
|
||||
total = sum(f.stat().st_size for f in remote.rglob('*') if f.is_file())
|
||||
raise BoardRefused(f'BOARD_TRANSFER_TOO_SLOW: 从开发环境取回 {req["cwd"]}({total >> 20} MiB)用了 '
|
||||
f'{(time.monotonic() - started) / 60:.0f} 分钟,这条命令的时限已到;{small}')
|
||||
return dict(req, target='board', command=req['command'][len(BOARD_MARK):], source=req['cwd'], cwd='.',
|
||||
deadline_monotonic=min(req['deadline_monotonic'], time.monotonic() + BOARD_MAX_S)), remote
|
||||
|
||||
|
||||
def relative(value, field):
|
||||
@ -87,21 +54,12 @@ def prepare(rd, queue, request):
|
||||
if not (rd / 'execution.json').is_file():
|
||||
raise ValueError('terminal requires a managed execution')
|
||||
metadata = json.loads((rd / 'rollout.json').read_text())
|
||||
from . import salty_dev
|
||||
if req['target'] == 'development' and salty_dev.selected():
|
||||
# bench dev: the Salty container's sandbox is the isolation (salty_remote_worker.dev); commands only,
|
||||
# formal results still come from the stages.
|
||||
if req['evaluate'] or req['source'] != '.':
|
||||
raise ValueError('bench dev runs commands only; evaluation and HBM results come from the stages')
|
||||
elif metadata.get('natural_agent_version') != 'natural-agent-v1' or metadata.get('jail_policy') != 'private-home':
|
||||
if metadata.get('natural_agent_version') != 'natural-agent-v1' or metadata.get('jail_policy') != 'private-home':
|
||||
raise ValueError('terminal requires a new task with natural-agent-v1 private-home isolation')
|
||||
if paths.config().get('terminal', {}).get('enabled') is not True:
|
||||
raise ValueError('managed terminal disabled until isolation acceptance')
|
||||
state = lifecycle.Execution(rd).snapshot()
|
||||
# The command's timeout starts when its container does (salty_remote_worker.dev); the request also covers the
|
||||
# queue (10-15: 64Gi dev containers waited in Salty past a 300 s timeout, so even `ls` failed).
|
||||
waiting = DEV_QUEUE_S if req['target'] == 'development' and salty_dev.selected() else 0
|
||||
req['deadline_monotonic'] = min(time.monotonic() + timeout + waiting, state['deadline_monotonic'])
|
||||
req['deadline_monotonic'] = min(time.monotonic() + timeout, state['deadline_monotonic'])
|
||||
if req['evaluate']:
|
||||
# Accepting a new formal candidate invalidates earlier stage success even
|
||||
# if preparation or the executor crashes before producing any prediction.
|
||||
@ -214,34 +172,12 @@ def score_predictions(raw, labels):
|
||||
|
||||
|
||||
def run(hd, rd, publish):
|
||||
req = json.loads((hd / 'request.json').read_text())
|
||||
board = req['target'] == 'board' or req['command'].startswith(BOARD_MARK)
|
||||
try:
|
||||
return _run(hd, rd, publish, req)
|
||||
except BoardRefused as exc:
|
||||
(hd / 'command.stderr.log').write_text(str(exc) + '\n')
|
||||
publish({'state': 'DONE', 'result': 'FAIL', 'exit_code': 2, 'error_class': str(exc).split(':', 1)[0],
|
||||
'duration_s': 0, 'stdout': '', 'stderr': str(hd / 'command.stderr.log'), 'meta': '',
|
||||
'stderr_tail': str(exc), 'cleanup_state': 'VERIFIED', 'measurements': {},
|
||||
'model_execution_source': 'UNVERIFIED', 'finished_at': time.time()})
|
||||
return 0
|
||||
finally:
|
||||
if board: # copies of HBMs and inputs would pile up on the controller (10-16: 2.5 GB after two failures)
|
||||
for name in ('snapshot', 'dev-files', 'dev-files.part'):
|
||||
shutil.rmtree(hd / name, ignore_errors=True)
|
||||
for name in ('board-input.tar', 'board-output.tar'): # output/ keeps the extracted files
|
||||
(hd / name).unlink(missing_ok=True)
|
||||
|
||||
|
||||
def _run(hd, rd, publish, req):
|
||||
from .stagerun import _ctx_from_rollout
|
||||
req = json.loads((hd / 'request.json').read_text())
|
||||
execution = lifecycle.current()
|
||||
if execution is None:
|
||||
raise RuntimeError('terminal requires execution ownership')
|
||||
ctx = _ctx_from_rollout(rd, None)
|
||||
dev_files = None
|
||||
if req['target'] == 'development' and req['command'].startswith(BOARD_MARK):
|
||||
req, dev_files = board_request(ctx, rd, hd, req)
|
||||
output, inputs = hd / 'output', hd / 'input'
|
||||
output.mkdir(); inputs.mkdir()
|
||||
formal = req['evaluate'] or req['target'] == 'board'
|
||||
@ -249,20 +185,6 @@ def _run(hd, rd, publish, req):
|
||||
manifest = None
|
||||
if formal:
|
||||
manifest = freeze(workspace / req['source'], hd / 'snapshot')
|
||||
if dev_files is not None:
|
||||
# Local edits win; big outputs that stayed in the development workspace fill the gaps.
|
||||
for f in sorted(dev_files.rglob('*')):
|
||||
target = hd / 'snapshot' / f.relative_to(dev_files)
|
||||
if f.is_file() and not f.is_symlink() and not target.exists():
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(f, target)
|
||||
digest = hashlib.sha256()
|
||||
with target.open('rb') as stream:
|
||||
for chunk in iter(lambda: stream.read(1 << 20), b''):
|
||||
digest.update(chunk)
|
||||
manifest[str(f.relative_to(dev_files))] = {'sha256': digest.hexdigest(), 'bytes': target.stat().st_size,
|
||||
'from': 'development'}
|
||||
shutil.rmtree(dev_files)
|
||||
workspace = hd / 'snapshot'
|
||||
lifecycle.atomic_json(hd / 'submission.json', manifest)
|
||||
elif req['source'] != '.':
|
||||
@ -281,8 +203,7 @@ def _run(hd, rd, publish, req):
|
||||
(hd / 'jail').mkdir()
|
||||
lifecycle.atomic_json(hd / 'jail.json', config)
|
||||
stdout, stderr = hd / 'command.stdout.log', hd / 'command.stderr.log'
|
||||
# A separate dev pool, when configured, keeps long agent experiments from holding compile slots.
|
||||
pool = 'board' if req['target'] == 'board' else 'dev' if 'dev' in paths.config().get('pools', {}) else 'cpu'
|
||||
pool = 'board' if req['target'] == 'board' else 'cpu'
|
||||
publish({'state': 'QUEUED', 'pool': pool, 'queued_at': time.time()})
|
||||
started = time.time()
|
||||
try:
|
||||
@ -290,12 +211,9 @@ def _run(hd, rd, publish, req):
|
||||
if time.monotonic() >= deadline:
|
||||
raise lifecycle.ExecutionStopped('TERMINAL_DEADLINE_EXCEEDED')
|
||||
publish({'state': 'RUNNING', 'pool': pool, 'started_at': time.time()})
|
||||
from . import salty_dev
|
||||
if req['target'] == 'board':
|
||||
from .terminal_board import execute
|
||||
result = execute(ctx, hd, config, deadline) # run() drops the copies; submission.json keeps the hashes
|
||||
elif salty_dev.selected():
|
||||
result = salty_dev.execute(ctx, hd, req, deadline, stdout, stderr)
|
||||
result = execute(ctx, hd, config, deadline)
|
||||
else:
|
||||
command = ['unshare', '-Urmpf' + ('n' if formal else ''), '--mount-proc', '--kill-child=KILL',
|
||||
paths.config()['paths'].get('harness_python', sys.executable),
|
||||
@ -304,7 +222,7 @@ def _run(hd, rd, publish, req):
|
||||
env={'PATH': '/usr/bin:/bin', 'LANG': 'C.UTF-8'}, execution=execution,
|
||||
role='terminal-command', hard_ceiling_s=max(.01, deadline - time.monotonic()), poll_s=.1,
|
||||
stop_check=lambda: 'TERMINAL_DEADLINE_EXCEEDED' if time.monotonic() >= deadline else None)
|
||||
if result.cleanup_state not in ('VERIFIED', 'REMOTE_BOUNDED'):
|
||||
if result.cleanup_state != 'VERIFIED':
|
||||
execution.quarantine('TERMINAL_CLEANUP_UNCONFIRMED')
|
||||
except BaseException:
|
||||
# Board helper quarantines inside the lease if a remote receipt is unknown.
|
||||
|
||||
@ -1,8 +1,7 @@
|
||||
"""Finite board transport for candidate commands, never substitutes board_bench.
|
||||
|
||||
The existing board guard owns remote descendants and the physical device lock. The command runs directly in
|
||||
the board's container (the boards are test devices; the S600 containers forbid namespaces anyway, 10-16) in
|
||||
this session's directory, which is removed afterwards; $BENCH_OUT is the directory brought back.
|
||||
The existing board guard owns remote descendants and the physical device lock.
|
||||
Arbitrary commands are admitted only after a namespace capability probe succeeds.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
@ -13,7 +12,7 @@ import time
|
||||
import uuid
|
||||
from contextlib import contextmanager
|
||||
|
||||
from . import execution as lifecycle, watchdog
|
||||
from . import execution as lifecycle, paths, watchdog
|
||||
from .stages.s07_board import board_env
|
||||
from .tools.board_perf import RemoteSession, HERE, guarded_run, run as transport
|
||||
|
||||
@ -56,7 +55,8 @@ class TerminalSession(RemoteSession):
|
||||
def __enter__(self):
|
||||
self.active = True
|
||||
self.setup(self.execute, ['mkdir', '-p', '--', self.directory])
|
||||
for source in (HERE / 'board_guard.py', HERE / 'board_probe.py', HERE.parent / 'process_reaper.py'):
|
||||
for source in (HERE / 'board_guard.py', HERE / 'board_probe.py',
|
||||
HERE.parent / 'process_reaper.py', paths.BENCH_ROOT / 'scripts/terminal_jail.py'):
|
||||
self.setup(self.upload, source, source.name)
|
||||
return self
|
||||
|
||||
@ -79,6 +79,12 @@ def execute(ctx, hd, config, deadline):
|
||||
session = TerminalSession(ctx.extra['board'].get('runtime_dir', '/userdata/bench'), deadline)
|
||||
except (ValueError, SystemExit):
|
||||
return not_started('BOARD_CONNECTION_CONFIGURATION_MISSING')
|
||||
# Read-only capability probe; no candidate process or remote job directory.
|
||||
try:
|
||||
session.execute(['unshare', '-Urmpfn', '--mount-proc', '--kill-child=KILL',
|
||||
'sh', '-c', 'command -v setpriv >/dev/null'], timeout=max(.01, min(20, deadline - time.monotonic())))
|
||||
except (Exception, SystemExit):
|
||||
return not_started('BOARD_ISOLATION_UNAVAILABLE')
|
||||
op = uuid.uuid4().hex
|
||||
dispatched = False
|
||||
receipt = {'operation_id': op, 'guard_version': 4, 'cleanup_state': 'UNKNOWN'}
|
||||
@ -92,11 +98,18 @@ def execute(ctx, hd, config, deadline):
|
||||
session.execute(['python3', '-c',
|
||||
'import tarfile,sys,pathlib; p=pathlib.Path(sys.argv[1]); '
|
||||
'tarfile.open(p/"input.tar").extractall(p, filter="data"); '
|
||||
'(p/"output").mkdir()', session.directory])
|
||||
'(p/"output").mkdir(); (p/"jail").mkdir()', session.directory])
|
||||
board_config = dict(config, jailroot=session.path('jail'), workspace=session.path('workspace'),
|
||||
output=session.path('output'), input=session.path('input'),
|
||||
runtimes=[], path='/usr/local/bin:/usr/bin:/bin',
|
||||
devices=['/dev/bpu', '/dev/bpu_core0', '/dev/bpu_core1', '/dev/bpu_core2', '/dev/bpu_core3', '/dev/ion'])
|
||||
board_config['timeout'] = max(1, int(session.remaining()))
|
||||
lifecycle.atomic_json(hd / 'board-jail.json', board_config)
|
||||
session.setup(session.upload, hd / 'board-jail.json', 'jail.json')
|
||||
command = ['python3', '-u', session.path('board_guard.py'), '--directory', session.directory,
|
||||
'--operation-id', op, '--timeout', str(session.remaining()), '--',
|
||||
'env', 'BENCH_OUT=' + session.path('output'), 'bash', '-c',
|
||||
f"cd {shlex.quote(session.path('workspace'))}/{shlex.quote(config['cwd'])} && {config['command']}"]
|
||||
'unshare', '-Urmpfn', '--mount-proc', '--kill-child=KILL', 'python3',
|
||||
session.path('terminal_jail.py'), session.path('jail.json')]
|
||||
timeout = session.remaining()
|
||||
dispatched = True
|
||||
try:
|
||||
@ -118,7 +131,7 @@ def execute(ctx, hd, config, deadline):
|
||||
receipt['error'] = type(exc).__name__
|
||||
lifecycle.atomic_json(hd / 'board-receipts' / (op + '.json'), receipt)
|
||||
if receipt['cleanup_state'] != 'VERIFIED':
|
||||
session.cleanup_unconfirmed = True
|
||||
session.failed_bench = True
|
||||
current.quarantine('BOARD_CLEANUP_UNCONFIRMED')
|
||||
raise RuntimeError('BOARD_CLEANUP_UNCONFIRMED')
|
||||
# Download only candidate-generated regular files after all children exit.
|
||||
@ -145,7 +158,7 @@ def execute(ctx, hd, config, deadline):
|
||||
proof='not_started', reason='BOARD_SETUP_FAILED')
|
||||
lifecycle.atomic_json(hd / 'board-receipts' / (op + '.json'), receipt)
|
||||
elif receipt['cleanup_state'] != 'VERIFIED':
|
||||
session.cleanup_unconfirmed = True
|
||||
session.failed_bench = True
|
||||
current.quarantine('BOARD_CLEANUP_UNCONFIRMED')
|
||||
raise
|
||||
finally:
|
||||
|
||||
@ -85,11 +85,6 @@ class _PreAuthDrop(Exception):
|
||||
pass
|
||||
|
||||
|
||||
# Loading a model takes seconds (S600 ~4 s). 10-15: two S600 loads hung until the guard's ~49 min deadline; end such a
|
||||
# call early as a retryable stall. Only the load phase is bounded: inference on slow models can legitimately take long.
|
||||
LOAD_STALL_S = 600
|
||||
|
||||
|
||||
def guarded_run(cmd: list[str], timeout: float) -> str:
|
||||
for attempt in range(3):
|
||||
try:
|
||||
@ -110,10 +105,9 @@ def _guarded_once(cmd: list[str], timeout: float) -> str:
|
||||
execution = from_environment()
|
||||
with execution.admission() if execution else nullcontext():
|
||||
proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
output, errors, partial = bytearray(), bytearray(), b''
|
||||
output, errors = bytearray(), bytearray()
|
||||
deadline = time.monotonic() + timeout + 15 # board deadline includes its cleanup acknowledgement
|
||||
next_lease = 0.0
|
||||
loading = None
|
||||
try:
|
||||
with selectors.DefaultSelector() as selector:
|
||||
selector.register(proc.stdout, selectors.EVENT_READ, 'stdout')
|
||||
@ -122,10 +116,6 @@ def _guarded_once(cmd: list[str], timeout: float) -> str:
|
||||
now = time.monotonic()
|
||||
if now > deadline:
|
||||
raise subprocess.TimeoutExpired(cmd, timeout)
|
||||
if loading is not None and now - loading > LOAD_STALL_S:
|
||||
print(f"__BENCH_WATCHDOG_STALLED__ board model load: no load_done after {LOAD_STALL_S} s",
|
||||
file=sys.stderr, flush=True)
|
||||
raise SystemExit(124)
|
||||
if now >= next_lease and proc.poll() is None:
|
||||
try:
|
||||
proc.stdin.write(b'lease\n'); proc.stdin.flush()
|
||||
@ -139,11 +129,6 @@ def _guarded_once(cmd: list[str], timeout: float) -> str:
|
||||
elif key.data == 'stderr':
|
||||
sys.stderr.write(data.decode(errors='replace')); sys.stderr.flush()
|
||||
errors.extend(data); del errors[:-4096]
|
||||
*lines, partial = (partial + data).split(b'\n')
|
||||
partial = partial[-4096:]
|
||||
for line in lines:
|
||||
if line.startswith(b'{"board_progress": "load_'):
|
||||
loading = time.monotonic() if b'"load_start"' in line else None
|
||||
else:
|
||||
output.extend(data)
|
||||
# Only the final JSON and a bounded diagnostic tail are needed.
|
||||
@ -181,7 +166,7 @@ def not_started(reason, directory=None):
|
||||
|
||||
|
||||
class RemoteSession:
|
||||
"""一次评测拥有一个临时目录。HBM/脚本只传一次,退出时清理自己的目录;进入时顺手删 6 小时没人动过的旧会话目录。"""
|
||||
"""一次评测拥有一个临时目录。HBM/脚本只传一次,退出时只清理自己的文件。"""
|
||||
|
||||
def __init__(self, hbm: str | Path, runtime_dir: str = "/userdata/bench"):
|
||||
self.hbm = Path(hbm)
|
||||
@ -189,21 +174,12 @@ class RemoteSession:
|
||||
self.scp_opts = ["-P" if o == "-p" else o for o in self.opts]
|
||||
self.directory = runtime_dir.rstrip("/") + "/eval-" + uuid.uuid4().hex
|
||||
self.active = False
|
||||
self.cleanup_unconfirmed = False
|
||||
self.failed_bench = False
|
||||
|
||||
def __enter__(self):
|
||||
if self.active:
|
||||
return self
|
||||
self.active = True
|
||||
try:
|
||||
# Sessions whose container was killed never reach close(). A live session touches its directory
|
||||
# around every board call (each ≤ 1 h), so 6 h untouched means nobody owns it.
|
||||
swept = self.execute(["find", self.directory.rsplit("/", 1)[0], "-maxdepth", "1", "-name", "eval-*", "-mmin", "+360",
|
||||
"-print", "-exec", "rm", "-rf", "--", "{}", "+"], timeout=120, cleanup=True)
|
||||
if swept.strip():
|
||||
print(f"removed stale board sessions: {' '.join(swept.split())}", file=sys.stderr, flush=True)
|
||||
except (Exception, SystemExit) as e:
|
||||
print(f"WARN stale board session sweep failed: {e}", file=sys.stderr)
|
||||
try:
|
||||
self.setup(self.execute, ["mkdir", "-p", "--", self.directory])
|
||||
for source, name in ((self.hbm, "model.hbm"), (HERE / "board_bench.py", "board_bench.py"),
|
||||
@ -256,19 +232,12 @@ class RemoteSession:
|
||||
# Recompute after uploads; the managed phase may already be partly spent.
|
||||
from harness.execution import from_environment, current_operation
|
||||
execution = from_environment()
|
||||
deadlines = []
|
||||
if execution:
|
||||
state = execution.snapshot()
|
||||
phase = state.get("phases", {}).get(current_operation(), {})
|
||||
deadlines = [state["deadline_monotonic"]]
|
||||
if phase.get("execution_deadline_monotonic") is not None:
|
||||
deadlines.append(phase["execution_deadline_monotonic"])
|
||||
if os.environ.get("BENCH_PHASE_DEADLINE"):
|
||||
# Salty containers carry no execution state: end before the phase does, leaving time to hand the
|
||||
# result and cleanup receipt back. 10-12/14: 12 board hangs outlived the phase, no receipt came
|
||||
# back and each rollout was lost as BOARD_CLEANUP_UNCONFIRMED instead of a retryable FAIL.
|
||||
deadlines.append(time.monotonic() + float(os.environ["BENCH_PHASE_DEADLINE"]) - time.time() - 540)
|
||||
if deadlines:
|
||||
timeout = min(timeout, min(deadlines) - time.monotonic() - 60)
|
||||
if timeout <= 0:
|
||||
self.not_started("BOARD_CLEANUP_BUDGET_INSUFFICIENT")
|
||||
@ -292,29 +261,26 @@ class RemoteSession:
|
||||
info = result_json(guarded_run(["ssh", *self.opts, self.target, shlex.join(command)], timeout))
|
||||
from harness.board_identity import validate
|
||||
validate(info, os.environ.get("BENCH_EXPECT_BOARD_TYPE", ""), os.environ.get("BENCH_EXPECT_UCP", ""))
|
||||
self.failed_bench = False
|
||||
return info
|
||||
except BaseException:
|
||||
self.failed_bench = True
|
||||
raise
|
||||
finally:
|
||||
# 15 s with no retry was below ssh's own 20 s ConnectTimeout: one slow handshake lost a rollout whose
|
||||
# board run had finished and been reaped (10-12..10-15: 9 times, 8 TimeoutExpired, 1 ssh exit).
|
||||
for attempt in range(3):
|
||||
try:
|
||||
actual = json.loads(self.execute(["cat", "--", self.path("guard-result.json")], timeout=60, cleanup=True))
|
||||
if actual.get("operation_id") != operation_id or actual.get("guard_version") != 4:
|
||||
raise ValueError("board cleanup receipt identity mismatch")
|
||||
if (actual.get("all_children_reaped") is not True or
|
||||
actual.get("proof") not in ("waitpid:ECHILD", "not_started")):
|
||||
raise ValueError("board cleanup receipt has no kernel proof")
|
||||
receipt.update(actual)
|
||||
receipt.pop("receipt_error", None)
|
||||
break
|
||||
except (Exception, SystemExit) as exc:
|
||||
receipt["receipt_error"] = type(exc).__name__
|
||||
if attempt < 2:
|
||||
time.sleep(10)
|
||||
try:
|
||||
actual = json.loads(self.execute(["cat", "--", self.path("guard-result.json")], timeout=15, cleanup=True))
|
||||
if actual.get("operation_id") != operation_id or actual.get("guard_version") != 4:
|
||||
raise ValueError("board cleanup receipt identity mismatch")
|
||||
if (actual.get("all_children_reaped") is not True or
|
||||
actual.get("proof") not in ("waitpid:ECHILD", "not_started")):
|
||||
raise ValueError("board cleanup receipt has no kernel proof")
|
||||
receipt.update(actual)
|
||||
except (Exception, SystemExit) as exc:
|
||||
receipt["receipt_error"] = type(exc).__name__
|
||||
if destination:
|
||||
atomic_json(Path(destination) / (operation_id + ".json"), receipt)
|
||||
if receipt["cleanup_state"] != "VERIFIED":
|
||||
self.cleanup_unconfirmed = True
|
||||
self.failed_bench = True
|
||||
if sys.exc_info()[0] is None:
|
||||
raise RuntimeError("BOARD_CLEANUP_UNCONFIRMED")
|
||||
|
||||
@ -324,15 +290,11 @@ class RemoteSession:
|
||||
def close(self):
|
||||
if self.active:
|
||||
self.active = False
|
||||
command = ["rm", "-rf", "--", self.directory]
|
||||
if self.cleanup_unconfirmed:
|
||||
# Its guard may still run and writes its receipt here, then the board quarantine marker: keep the
|
||||
# directory and the receipt, drop the rest (10-15: kept sessions filled the S600s' /userdata).
|
||||
print(f"board session cleanup unconfirmed, directory kept: {self.directory}", file=sys.stderr, flush=True)
|
||||
command = ["find", self.directory, "-mindepth", "1", "-maxdepth", "1", "!", "-name", "*guard-result.json",
|
||||
"-exec", "rm", "-rf", "--", "{}", "+"]
|
||||
if self.failed_bench:
|
||||
print(f"board failed session retained for diagnosis: {self.directory}", file=sys.stderr, flush=True)
|
||||
return
|
||||
try:
|
||||
self.execute(command, timeout=60, cleanup=True)
|
||||
self.execute(["rm", "-rf", "--", self.directory], timeout=60, cleanup=True)
|
||||
except (Exception, SystemExit) as e:
|
||||
print(f"WARN board temporary directory cleanup failed: {self.directory}: {e}", file=sys.stderr)
|
||||
|
||||
|
||||
@ -1,39 +0,0 @@
|
||||
#!/bin/bash
|
||||
# 本机每 10 分钟(launchd)查一次 Mac mini 上的采集;有问题弹系统通知,同样的问题每 2 小时再提醒一次,恢复了也通知。
|
||||
# 安装:cp 到 ~/.bench-health/,plist 放 ~/Library/LaunchAgents/cc.d-robotics.bench-health.plist。记录在 ~/.bench-health/log。
|
||||
MINI=amos@192.168.66.251 # Mac mini 换了地址改这里
|
||||
RUNS="r20260929-04 r20260929-05 r20260929-06" # 加新批次时加到这里
|
||||
FEISHU_TO="ou_cc299d6a3684a6c45d56d91f7127b185" # 填飞书 open_id 就同时用小金(飞书群答疑助手)私聊提醒
|
||||
export PATH=$HOME/.npm-global/bin:/usr/local/bin:/opt/homebrew/bin:/usr/bin:/bin
|
||||
D=~/.bench-health; mkdir -p "$D"
|
||||
notify() {
|
||||
osascript -e 'on run argv' -e 'display notification (item 1 of argv) with title (item 2 of argv) sound name "Glass"' -e 'end run' "$1" "$2"
|
||||
if [ -n "$FEISHU_TO" ] && ! lark-cli im +messages-send --as bot --user-id "$FEISHU_TO" --text "【$2】"$'\n'"$1" > /dev/null 2>&1; then
|
||||
echo "$(date '+%m-%d %H:%M') 飞书发送失败" >> "$D/alerts.log"
|
||||
fi
|
||||
printf '%s %s\n%s\n' "$(date '+%m-%d %H:%M')" "$2" "$1" >> "$D/alerts.log"
|
||||
}
|
||||
if ! ssh -o ConnectTimeout=15 -o BatchMode=yes "$MINI" true 2>/dev/null; then
|
||||
fails=$(( $(cat "$D/ssh_fails" 2>/dev/null || echo 0) + 1 )); echo $fails > "$D/ssh_fails"
|
||||
[ $fails -ge 2 ] && problems="Mac mini($MINI)连不上,已 $((fails * 10)) 分钟。Wi-Fi 掉了、换了地址或关机了" # Wi-Fi 抖一下不报
|
||||
else
|
||||
rm -f "$D/ssh_fails"
|
||||
out=$(ssh -o ConnectTimeout=15 -o BatchMode=yes "$MINI" \
|
||||
"export PATH=/opt/homebrew/bin:\$PATH; limactl list --format '{{.Status}}' bench && limactl shell bench -- python3 /mnt/data/ming01.wang/model-compile-hbdk/bench/scripts/health_check.py $RUNS" 2>&1)
|
||||
rc=$?
|
||||
status=$(head -1 <<<"$out")
|
||||
if [ "$status" != Running ]; then problems="bench 虚拟机没在运行($status),Mac mini 可能重启过,需要恢复"
|
||||
elif [ $rc != 0 ]; then problems="健康检查本身出错:$(tail -1 <<<"$out")"
|
||||
else problems=$(tail -n +2 <<<"$out"); fi
|
||||
fi
|
||||
key=$(tr -d '0-9' <<<"$problems") # 去掉数字,分钟数变了不算新问题
|
||||
now=$(date +%s); last=$(cat "$D/last_t" 2>/dev/null || echo 0)
|
||||
if [ -n "$problems" ]; then
|
||||
if [ "$key" != "$(cat "$D/last_key" 2>/dev/null)" ] || [ $((now - last)) -ge 7200 ]; then
|
||||
notify "$problems" "采集报警($(wc -l <<<"$problems" | tr -d ' ') 项)"
|
||||
echo "$key" > "$D/last_key"; echo $now > "$D/last_t"
|
||||
fi
|
||||
elif [ -s "$D/last_key" ]; then
|
||||
notify "之前报的问题都已消失" "采集恢复正常"; rm -f "$D/last_key"
|
||||
fi
|
||||
echo "$(date '+%m-%d %H:%M') ${problems:-ok}" | head -1 >> "$D/log"
|
||||
@ -1,34 +0,0 @@
|
||||
#!/bin/bash
|
||||
# 排空旧调度程序后换新代码重启,不丢在跑的轨迹。在 Mac mini 的 bench 虚拟机里后台跑:
|
||||
# setsid nohup scripts/drain_restart.sh "<run> <旧pid> <配置名> <新并发> [题单文件]" ... >> logs/drain-restart.log 2>&1 &
|
||||
# 暂停的轨迹归旧进程所有(OWNER_NOT_RELEASABLE),旧进程退出后由新进程自动续跑,不用手动 resume。
|
||||
# 旧进程没有子进程、也没有校验中的轨迹满 10 分钟,说明卡在最后的导出上,按 bench-restart-hazards 第 5 条 kill -9。
|
||||
set -u
|
||||
B=/mnt/data/ming01.wang/model-compile-hbdk/bench; C=/mnt/data/ming01.wang/configs
|
||||
export BENCH_ROOT=$B BENCH_STATE_DIR=$B/state
|
||||
export PATH=/home/amos.guest/.local/lib/salty-controller/python/bin:/home/amos.guest/.local/lib/salty-controller/runtime/bin:/home/amos.guest/.local/bin:/usr/local/bin:/usr/bin:/bin
|
||||
log() { echo "$(date '+%m-%d %H:%M:%S') $*"; }
|
||||
SPECS=("$@"); declare -A idle=() done_=()
|
||||
# 已经发过 SIGTERM 时用 NO_TERM=1 重跑本脚本:第二次信号可能让调度程序直接退出、丢掉在跑的轨迹。
|
||||
[ -z "${NO_TERM:-}" ] && for s in "${SPECS[@]}"; do set -- $s; kill -TERM "$2" && log "SIGTERM $1 pid $2,开始排空"; done
|
||||
while [ ${#done_[@]} -lt ${#SPECS[@]} ]; do
|
||||
for s in "${SPECS[@]}"; do
|
||||
set -- $s; run=$1 pid=$2 cfg=$3 n=$4 jobs=${5:-}
|
||||
[ -n "${done_[$run]:-}" ] && continue
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
busy=$(python3 -c "import sqlite3,sys; print(sqlite3.connect('file:$B/state/jobs.sqlite?mode=ro',uri=True).execute(\"SELECT COUNT(*) FROM rollouts WHERE run_id=? AND state IN ('VERIFYING','CLEANING')\",(sys.argv[1],)).fetchone()[0])" "$run")
|
||||
if [ -z "$(pgrep -P "$pid")" ] && [ "$busy" = 0 ]; then idle[$run]=$(( ${idle[$run]:-0} + 1 )); else idle[$run]=0; fi
|
||||
if [ "${idle[$run]}" -ge 10 ]; then
|
||||
log "$run 旧进程 $pid 已无在跑轨迹 10 分钟仍未退出(卡在导出),kill -9"
|
||||
kill -9 "$pid"; sleep 5; rm -rf "$B/datasets/$run-training"/.refresh-*
|
||||
fi
|
||||
continue
|
||||
fi
|
||||
args="--run $run --modes e2e --parallel $n --watch"
|
||||
[ -n "$jobs" ] && args="$args $(sed 's/^/--job /' "$jobs" | tr '\n' ' ')"
|
||||
log "$run 旧进程已退出,起新进程:$($C/run-collect.sh "$cfg" $args)"
|
||||
done_[$run]=1
|
||||
done
|
||||
sleep 60
|
||||
done
|
||||
log "全部重启完成"
|
||||
@ -1,61 +0,0 @@
|
||||
"""Mac mini 主控的健康检查:有问题每条打印一行,健康时什么都不打印。只读。
|
||||
用法(bench 虚拟机里):python3 scripts/health_check.py RUN [RUN ...]
|
||||
由本机 scripts/bench_health_alert.sh 每 10 分钟经 ssh 调用,有输出就弹通知。"""
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
B = os.environ.get("BENCH_ROOT", "/mnt/data/ming01.wang/model-compile-hbdk/bench")
|
||||
QUOTA = os.path.expanduser("~/.local/state/bench-shared/quota.sqlite")
|
||||
INFRA = ("REMOTE_WORKER_FAILED", "STAGE_EXECUTOR_EXIT", "QUEUE_DEADLINE_EXCEEDED")
|
||||
|
||||
|
||||
def problems(runs, now):
|
||||
ps = subprocess.run(["ps", "-eo", "args"], capture_output=True, text=True).stdout
|
||||
for run in runs:
|
||||
if f"collect --run {run} " not in ps:
|
||||
yield f"{run} 的调度程序没在跑"
|
||||
c = sqlite3.connect(f"file:{B}/state/jobs.sqlite?mode=ro", uri=True, timeout=30)
|
||||
q = ",".join("?" * len(runs))
|
||||
for run, job, kk in c.execute(f"SELECT run_id, job_id, kk FROM rollouts WHERE state='PAUSED_INFRA' AND run_id IN ({q})", runs):
|
||||
try:
|
||||
e = json.load(open(f"{B}/runs/{run}/{job}/rollouts/{kk:02d}/execution.json"))
|
||||
except (OSError, ValueError):
|
||||
continue
|
||||
paused = now - (e.get("paused_at") or 0)
|
||||
if 3600 < paused < 86400: # 正常的暂停几分钟内就会自动续;超过一天的是老问题,不反复报
|
||||
yield f"{run[-2:]} {job} #{kk} 暂停 {paused // 60:.0f} 分钟没恢复({e.get('stop_reason')})"
|
||||
for run in runs:
|
||||
alive = c.execute("SELECT COUNT(*) FROM rollouts WHERE run_id=? AND state='RUNNING'", (run,)).fetchone()[0]
|
||||
last = c.execute("SELECT MAX(started_at) FROM stages WHERE run_id=?", (run,)).fetchone()[0] or 0
|
||||
if alive and now - last > 3 * 3600:
|
||||
yield f"{run} 有 {alive} 条在跑,但 {(now - last) / 3600:.0f} 小时没有开始任何新步骤,调度程序可能卡住了"
|
||||
n = c.execute(f"SELECT COUNT(*) FROM stages WHERE run_id IN ({q}) AND state='FAIL' AND started_at>?"
|
||||
f" AND error_class IN ({','.join('?' * len(INFRA))})", (*runs, now - 3600, *INFRA)).fetchone()[0]
|
||||
if n >= 8:
|
||||
yield f"最近 1 小时 Salty 容器失败 {n} 次"
|
||||
n = c.execute(f"SELECT COUNT(*) FROM rollouts WHERE run_id IN ({q}) AND finished_at>?"
|
||||
" AND state IN ('INFRA_ERROR','ABANDONED','LOST','CANCELLED')", (*runs, now - 3600)).fetchone()[0]
|
||||
if n >= 5:
|
||||
yield f"最近 1 小时有 {n} 条轨迹被系统中断"
|
||||
qc = sqlite3.connect(f"file:{QUOTA}?mode=ro", uri=True, timeout=30)
|
||||
for group, state, updated in qc.execute(
|
||||
"SELECT group_id, state, updated_at FROM quota_groups WHERE state IN ('PAUSED','RECONCILING')"
|
||||
" AND group_id IN (SELECT group_id FROM attempts WHERE started_at>?)", (now - 86400,)):
|
||||
if now - updated > 2700: # 请求结果不明时本来就会整组核对约半小时,超过 45 分钟才算卡住
|
||||
yield f"模型账号 {group} 处于 {state} 已 {(now - updated) / 60:.0f} 分钟,用这个账号的轨迹都发不出请求"
|
||||
free = shutil.disk_usage(B).free / 2**30
|
||||
if free < 10:
|
||||
yield f"虚拟机磁盘只剩 {free:.0f} GB"
|
||||
mem = int(next(l.split()[1] for l in open("/proc/meminfo") if l.startswith("MemAvailable"))) // 1024
|
||||
if mem < 1024:
|
||||
yield f"虚拟机可用内存只剩 {mem} MB,调度程序可能被系统杀掉"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
for line in problems(sys.argv[1:], time.time()):
|
||||
print(line)
|
||||
@ -1,272 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Move finished rollouts to Salty personal space and build the datasets there; the controller keeps
|
||||
only small files and the final datasets.
|
||||
|
||||
Every 10 minutes:
|
||||
1. archive: a rollout that ended over 30 min ago (UNVERIFIED: 24 h, it can still be re-verified), whose
|
||||
execution is TERMINAL and, if PASS, whose content review is settled, goes to
|
||||
personal:/bench-root/archive/<run>/<job>/<kk>.tgz. Stage products already in personal:/bench-root/runs
|
||||
with the same name and size are left out. Later changes go to <kk>.u<time>.tgz overlays; an archive is
|
||||
never rewritten. Then every local file over 1 MB is deleted.
|
||||
2. export: at most hourly and only after new archives, one Salty job unpacks the archives into personal
|
||||
runs and runs export/to_agentrl.py there (salty_remote_worker.export); the datasets stay in
|
||||
personal:/bench-root/datasets/ and are copied to bench/datasets-final/.
|
||||
The collectors keep their own per-rollout export into bench/datasets/: it finds and reviews new PASS
|
||||
rollouts while their files are still here, which is what step 1 waits for.
|
||||
|
||||
Usage: salty_offload.py RUN=CONFIG [RUN=CONFIG ...] [--once] [--export-now]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
PRODUCTS = (".hbm", ".bc", ".npz", ".onnx")
|
||||
BIG = 1 << 20
|
||||
|
||||
|
||||
def log(message):
|
||||
print(f"{time.strftime('%F %T')} {message}", flush=True)
|
||||
|
||||
|
||||
def remote_sizes(cli, remote_dir):
|
||||
rc, out, _ = cli.once(["storage", "ls", remote_dir])
|
||||
rows = (line.split("\t") for line in out.splitlines()[1:]) if rc == 0 else ()
|
||||
return {f[0]: int(f[2]) for f in rows if len(f) > 2 and f[1] == "文件"}
|
||||
|
||||
|
||||
def settled(rd, reasons):
|
||||
"""PASS leaves once the collector's export has judged it: a finished review, or excluded for another reason."""
|
||||
for receipt in (rd / "content-reviews").glob("*.json"):
|
||||
with contextlib.suppress(ValueError, OSError):
|
||||
if json.loads(receipt.read_text()).get("state") == "COMPLETE":
|
||||
return True
|
||||
return reasons is not None and not any(r.endswith("content-review-unreviewed") for r in reasons)
|
||||
|
||||
|
||||
def archive(cli, paths, rd):
|
||||
"""Upload what changed since the last archive of this rollout, then free its local bulk."""
|
||||
marker = rd / "ARCHIVED.json"
|
||||
info = json.loads(marker.read_text()) if marker.exists() else {}
|
||||
started = time.time()
|
||||
files = [p for p in rd.rglob("*") if p.is_file() and not p.is_symlink() and p != marker]
|
||||
changed = [p for p in files if p.stat().st_mtime > info.get("at", 0)]
|
||||
if changed:
|
||||
mirrored, listings = set(), {}
|
||||
for p in changed:
|
||||
if p.suffix in PRODUCTS and p.stat().st_size > BIG:
|
||||
d = str(p.parent.relative_to(paths.RUNS))
|
||||
listings.setdefault(d, remote_sizes(cli, f"personal:/bench-root/runs/{d}"))
|
||||
if listings[d].get(p.name) == p.stat().st_size:
|
||||
mirrored.add(p)
|
||||
run, job, kk = rd.parts[-4], rd.parts[-3], rd.name
|
||||
name = f"{kk}.tgz" if not info else f"{kk}.u{int(started)}.tgz"
|
||||
remote = f"personal:/bench-root/archive/{run}/{job}/{name}"
|
||||
with tempfile.TemporaryDirectory(prefix="salty-archive-") as tmp:
|
||||
tgz = Path(tmp) / name
|
||||
with tarfile.open(tgz, "w:gz") as tar:
|
||||
for p in changed:
|
||||
if p not in mirrored:
|
||||
tar.add(p, str(p.relative_to(paths.RUNS)))
|
||||
cli.call("storage", "mkdir", "-p", remote.rsplit("/", 1)[0])
|
||||
cli.call("storage", "cp", str(tgz), remote, timeout=3600)
|
||||
size = tgz.stat().st_size
|
||||
if remote_sizes(cli, remote.rsplit("/", 1)[0]).get(name) != size:
|
||||
raise RuntimeError(f"uploaded {name} size differs")
|
||||
entry = {"archive": remote, "bytes": size, "sha256": hashlib.sha256(tgz.read_bytes()).hexdigest(),
|
||||
"mirrored": sorted(str(p.relative_to(paths.RUNS)) for p in mirrored)}
|
||||
info = {**entry, **info, "at": started}
|
||||
if name != f"{kk}.tgz":
|
||||
info["overlays"] = [*info.get("overlays", []), entry]
|
||||
marker.write_text(json.dumps(info, ensure_ascii=False, indent=1))
|
||||
freed = 0
|
||||
for p in files:
|
||||
if p.exists() and p.stat().st_size > BIG:
|
||||
freed += p.stat().st_size
|
||||
p.unlink()
|
||||
return len(changed), freed
|
||||
|
||||
|
||||
def archive_all(cli, paths, runs, terminal):
|
||||
con = sqlite3.connect(f"file:{paths.DB_PATH}?mode=ro", uri=True)
|
||||
rows = con.execute(f"SELECT run_id, job_id, kk, state, finished_at FROM rollouts WHERE run_id IN "
|
||||
f"({','.join('?' * len(runs))})", tuple(runs)).fetchall()
|
||||
con.close()
|
||||
reasons = {}
|
||||
for run in runs:
|
||||
with contextlib.suppress(OSError):
|
||||
for line in (paths.DATASETS / f"{run}-training/disposition.jsonl").read_text().splitlines():
|
||||
item = json.loads(line)
|
||||
reasons[(run, item["job_id"], item["rollout"])] = item["reasons"]
|
||||
uploaded = 0
|
||||
for run, job, kk, state, finished in rows:
|
||||
rd = paths.RUNS / run / job / "rollouts" / f"{kk:02d}"
|
||||
delay = 86400 if state == "UNVERIFIED" else 1800
|
||||
if state not in terminal or not finished or finished > time.time() - delay or not rd.is_dir():
|
||||
continue
|
||||
marker = rd / "ARCHIVED.json"
|
||||
at = json.loads(marker.read_text())["at"] if marker.exists() else None
|
||||
if at is not None and all(p.stat().st_size <= BIG and p.stat().st_mtime <= at for p in rd.rglob("*")
|
||||
if p.is_file() and not p.is_symlink() and p != marker):
|
||||
continue # archived, freed and unchanged
|
||||
execution = rd / "execution.json"
|
||||
if execution.exists() and json.loads(execution.read_text()).get("execution_state") != "TERMINAL":
|
||||
continue
|
||||
if state == "PASS" and not settled(rd, reasons.get((run, job, kk))):
|
||||
continue
|
||||
try:
|
||||
changed, freed = archive(cli, paths, rd)
|
||||
uploaded += bool(changed)
|
||||
log(f"{run} {job} {kk}: archived {changed} files, freed {freed >> 20}MB")
|
||||
except Exception as exc: # next pass retries; nothing local was deleted before the upload was checked
|
||||
log(f"{run} {job} {kk}: SKIP {type(exc).__name__}: {exc}")
|
||||
return uploaded
|
||||
|
||||
|
||||
def reap_orphans(cli, paths, sr, runs):
|
||||
"""A Salty job whose controller attempt already ended (phase deadline, cancel, crash) has no reader:
|
||||
stop it, and once it has ended remove it and its dir like salty_remote removes its own jobs (their logs and
|
||||
describe are already in the attempt's stderr.log). 10-13: 12 such jobs held ~64 cores while ours queued."""
|
||||
rc, out, _ = cli.once(["ps"])
|
||||
if rc:
|
||||
return
|
||||
status = {f[0]: f[2] for f in (line.split("\t") for line in out.splitlines()[1:]) if len(f) > 2}
|
||||
for run in runs:
|
||||
attempts = (paths.RUNS / run).glob("*/rollouts/*/stages/*/a*/salty/remote.json")
|
||||
# bench dev commands (harness/salty_dev.py): ended when their ticket is DONE; nothing to keep for forensics.
|
||||
dev = (paths.RUNS / run).glob("*/rollouts/*/stages/_queue/terminal-*/salty/remote.json")
|
||||
for p in [*attempts, *dev]:
|
||||
state = json.loads(p.read_text())
|
||||
if state.get("cleaned"): # its own run already removed it; ps can still list it for a moment
|
||||
continue
|
||||
s = status.get(state.get("job_id"))
|
||||
is_dev = p.parent.parent.name.startswith("terminal-")
|
||||
if is_dev:
|
||||
try:
|
||||
ended = json.loads((p.parent.parent / "status.json").read_text()).get("state") == "DONE"
|
||||
except (OSError, ValueError):
|
||||
ended = False
|
||||
else:
|
||||
ended = (p.parent.parent / "meta.json").exists()
|
||||
if s not in ("running", "pending", "stopped", "succeeded", "failed") or not ended:
|
||||
continue
|
||||
if s in ("stopped", "succeeded", "failed"):
|
||||
sr._cleanup(cli, state)
|
||||
else:
|
||||
cli.call("stop", state["job_id"], ok_404=True) # 404: gone meanwhile, nothing to stop (10-14: 30 min of retries)
|
||||
log(f"{state['job_id']} {s}, its attempt {p.parent.parent.relative_to(paths.RUNS)} already ended: "
|
||||
+ ("removed" if s in ("stopped", "succeeded", "failed") else "stop requested"))
|
||||
|
||||
|
||||
def export(cli, paths, sr, policy, runs):
|
||||
name = "bench-export-" + str(uuid.uuid4())
|
||||
bench = paths.BENCH_ROOT
|
||||
with tempfile.TemporaryDirectory(prefix="salty-export-") as tmp:
|
||||
work = Path(tmp)
|
||||
with contextlib.closing(sqlite3.connect(paths.DB_PATH)) as src, \
|
||||
contextlib.closing(sqlite3.connect(work / "jobs.sqlite")) as dst:
|
||||
src.backup(dst)
|
||||
files, configs = {str(bench / "state/jobs.sqlite"): work / "jobs.sqlite"}, {}
|
||||
with contextlib.closing(sqlite3.connect(work / "jobs.sqlite")) as con:
|
||||
for run, config in runs.items():
|
||||
configs[run] = str(bench / "export-configs" / f"{run}.yaml")
|
||||
files[configs[run]] = Path(config)
|
||||
seed = paths.DATASETS / f"{run}-training/manifest.json"
|
||||
if seed.exists():
|
||||
files[str(bench / "export-seed" / f"{run}.json")] = seed
|
||||
for (job,) in con.execute("SELECT job_id FROM jobs WHERE run_id=?", (run,)):
|
||||
if (paths.RUNS / run / job / "job.json").exists():
|
||||
files[str(paths.RUNS / run / job / "job.json")] = paths.RUNS / run / job / "job.json"
|
||||
request = {"identity": {"export": name}, "stage": "export", "export": configs, "env": {}, "links": {},
|
||||
"files": {destination: f"files/{i}" for i, destination in enumerate(files)}}
|
||||
data = json.dumps(request).encode()
|
||||
with tarfile.open(work / "bundle.tar", "w:gz") as tar: # the worker opens it with tarfile's autodetect
|
||||
info = tarfile.TarInfo("request.json")
|
||||
info.size = len(data)
|
||||
tar.addfile(info, io.BytesIO(data))
|
||||
for i, source in enumerate(files.values()):
|
||||
tar.add(source, f"files/{i}")
|
||||
for d in (*sr.CODE_DIRS, "agent"):
|
||||
tar.add(bench / d, f"code/{d}", filter=lambda m: None if "__pycache__" in m.name else m)
|
||||
shutil.copyfile(sr.WORKER, work / "worker.py")
|
||||
state = {"name": name, "remote": "personal:/" + name}
|
||||
state["job_id"] = sr._submit(cli, state, policy, SimpleNamespace(name="export"), work,
|
||||
policy.get("export_memory", "32Gi"))
|
||||
log(f"export job {state['job_id']} submitted")
|
||||
job_state = sr._wait(cli, state["job_id"], policy, time.time(), time.time, time.sleep)
|
||||
if job_state != "succeeded":
|
||||
sr._remote_logs(policy, state["job_id"], work / "remote.log")
|
||||
log(f"export job {job_state}, kept for forensics:\n{(work / 'remote.log').read_text()[-4000:]}")
|
||||
return False
|
||||
sr._fetch(cli, state["remote"] + "/result/result.json", work / "result.json")
|
||||
sr._fetch(cli, state["remote"] + "/result/datasets.tar.gz", work / "datasets.tar.gz")
|
||||
codes = json.loads((work / "result.json").read_text())["exit_codes"]
|
||||
with tarfile.open(work / "datasets.tar.gz") as tar:
|
||||
tar.extractall(work / "out", members=[m for m in tar.getmembers() if m.isfile() and ".." not in m.name
|
||||
and not m.name.startswith("/")])
|
||||
final = bench / "datasets-final"
|
||||
final.mkdir(exist_ok=True)
|
||||
for run, rc in codes.items():
|
||||
if rc != 0:
|
||||
log(f"export of {run} failed (exit {rc}); its last good dataset stays")
|
||||
continue
|
||||
target, old = final / f"{run}-training", final / f".{run}-training.old"
|
||||
shutil.rmtree(old, ignore_errors=True)
|
||||
if target.exists():
|
||||
target.rename(old)
|
||||
(work / "out" / f"{run}-training").rename(target)
|
||||
shutil.rmtree(old, ignore_errors=True)
|
||||
manifest = json.loads((target / "manifest.json").read_text())
|
||||
log(f"{run}: sft {manifest['stats']['sft_eligible']} of {manifest['stats']['scanned']} scanned")
|
||||
sr._cleanup(cli, state)
|
||||
return all(rc == 0 for rc in codes.values())
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("runs", nargs="+", metavar="RUN=CONFIG")
|
||||
ap.add_argument("--once", action="store_true", help="one pass, then exit")
|
||||
ap.add_argument("--export-now", action="store_true", help="export on the first pass even if nothing new was archived")
|
||||
a = ap.parse_args()
|
||||
runs = dict(item.split("=", 1) for item in a.runs)
|
||||
os.environ["BENCH_CONFIG"] = next(iter(runs.values())) # salty CLI and image come from the first config
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
from harness import paths, salty_remote as sr
|
||||
from export.to_agentrl import DB_TERMINAL_STATES
|
||||
policy = dict(paths.config()["execution"]["salty_remote"])
|
||||
if "@sha256:" not in policy.get("image", ""):
|
||||
raise SystemExit("execution.salty_remote.image must be pinned by digest")
|
||||
cli = sr._Cli(policy.get("salty", "salty"), policy.get("outage_seconds", 1800), time.sleep, time.time, log)
|
||||
pending, last_export = a.export_now, 0.0
|
||||
while True:
|
||||
try:
|
||||
reap_orphans(cli, paths, sr, runs)
|
||||
except Exception as exc: # retried next pass
|
||||
log(f"reap failed: {type(exc).__name__}: {exc}")
|
||||
pending = archive_all(cli, paths, runs, DB_TERMINAL_STATES) > 0 or pending
|
||||
if pending and time.time() - last_export >= 3600:
|
||||
last_export = time.time()
|
||||
try:
|
||||
pending = not export(cli, paths, sr, policy, runs)
|
||||
except Exception as exc: # retried next hour; the last good datasets stay in place
|
||||
log(f"export failed: {type(exc).__name__}: {exc}")
|
||||
if a.once:
|
||||
return 0
|
||||
time.sleep(600)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@ -1,101 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Review again the PASS rollouts the last Salty export (bench/datasets-final) left unreviewed or UNKNOWN
|
||||
whose files are no longer on the controller (salty_offload.py freed them).
|
||||
|
||||
One at a time: fetch <kk>.tgz from personal:/bench-root/archive (and the HBM from personal runs if the
|
||||
archive left it out), review with the configured reviewer, upload what the review wrote as an overlay and
|
||||
free the files again. Old-format events reach 650 MB, and every collector's export would re-read whatever
|
||||
sits here, so nothing stays longer than its own review. Stop salty_offload.py first: it would free a
|
||||
rollout in the middle of its review.
|
||||
|
||||
--release-started: a receipt left STARTED by a crash never authorizes a new request (the old one may have
|
||||
been sent and answered); with this flag the operator accepts a second request: the STARTED bytes go to
|
||||
content-reviews/history/ and the receipt becomes COMPLETE UNKNOWN (OPERATOR_RELEASED_STARTED_RECEIPT), which
|
||||
the normal UNKNOWN retry replaces under a new execution id.
|
||||
|
||||
Usage: salty_review_again.py RUN CONFIG [--release-started] [--only RUN/JOB/rollouts/KK,...] [--dry-run]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
WANT = {"content-review-unreviewed", "sft:content-review-unreviewed", "content-review-unknown", "sft:content-review-unknown"}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
run, config, flags = sys.argv[1], sys.argv[2], sys.argv[3:]
|
||||
if subprocess.run(["pgrep", "-f", r"^python[0-9.]* .*salty_offload\.py"], capture_output=True).returncode == 0:
|
||||
raise SystemExit("stop salty_offload.py first")
|
||||
os.environ["BENCH_CONFIG"] = config
|
||||
sys.path[:0] = [str(Path(__file__).resolve().parent.parent), str(Path(__file__).resolve().parent)]
|
||||
from harness import paths, salty_remote as sr
|
||||
from export.content_review import review_once
|
||||
import salty_offload
|
||||
|
||||
only = set(flags[flags.index("--only") + 1].split(",")) if "--only" in flags else None
|
||||
policy = paths.config()["execution"]["salty_remote"]
|
||||
cli = sr._Cli(policy.get("salty", "salty"), 1800, time.sleep, time.time, salty_offload.log)
|
||||
todo = [(d["source"], d["reasons"][0]) for d in map(json.loads, open(paths.BENCH_ROOT / f"datasets-final/{run}-training/disposition.jsonl"))
|
||||
if d["reasons"] and set(d["reasons"]) <= WANT and (only is None or d["source"].removeprefix("runs/") in only)]
|
||||
salty_offload.log(f"{run}: {len(todo)} to review again")
|
||||
for source, reason in todo:
|
||||
rd = paths.BENCH_ROOT / source
|
||||
job, kk = rd.parent.parent.name, rd.name
|
||||
if "--dry-run" in flags or (rd / "agent/events.jsonl").exists():
|
||||
salty_offload.log(f"{'would review' if '--dry-run' in flags else 'still local, skip'} {source} {reason}")
|
||||
continue
|
||||
remote = f"personal:/bench-root/archive/{run}/{job}/{kk}.tgz"
|
||||
tgz = Path(f"/tmp/review-again-{job}-{kk}.tgz")
|
||||
try:
|
||||
cli.call("storage", "cp", remote, str(tgz), timeout=3600)
|
||||
with tarfile.open(tgz) as tar:
|
||||
tar.extractall(paths.RUNS, members=[m for m in tar.getmembers() if m.isfile()
|
||||
and Path(m.name).parts[:4] == rd.relative_to(paths.RUNS).parts])
|
||||
hbm = rd / "artifacts" / f"{job}.hbm"
|
||||
if not hbm.exists():
|
||||
cli.call("storage", "cp", f"personal:/bench-root/runs/{rd.relative_to(paths.RUNS)}/artifacts/{hbm.name}",
|
||||
str(hbm), timeout=3600)
|
||||
os.utime(hbm, (0, 0))
|
||||
# Restored files keep old mtimes: only what the review writes after `at` goes into the overlay.
|
||||
marker = rd / "ARCHIVED.json"
|
||||
info = json.loads(marker.read_text()) if marker.exists() else {"archive": remote}
|
||||
marker.write_text(json.dumps({**info, "at": time.time()}, ensure_ascii=False, indent=1))
|
||||
if "--release-started" in flags:
|
||||
for receipt in (rd / "content-reviews").glob("*.json"):
|
||||
data = receipt.read_bytes()
|
||||
record = json.loads(data)
|
||||
if record.get("state") == "STARTED":
|
||||
# Close it as a COMPLETE UNKNOWN the operator released: the retry below then gets a new
|
||||
# execution id. Reusing the old one replays the ledger's finished generation, no new request.
|
||||
history = receipt.parent / "history" / (hashlib.sha256(data).hexdigest() + ".json")
|
||||
history.parent.mkdir(mode=0o700, exist_ok=True)
|
||||
history.write_bytes(data)
|
||||
receipt.write_text(json.dumps(dict(record, state="COMPLETE", decision="UNKNOWN", finished_at=time.time(),
|
||||
reason="OPERATOR_RELEASED_STARTED_RECEIPT", issues=[]), ensure_ascii=False))
|
||||
salty_offload.log(f"{source}: released STARTED receipt {receipt.name}")
|
||||
try:
|
||||
result = review_once(rd, managed=True, retry_unknown=True) # also an UNKNOWN written after the export
|
||||
except ValueError as exc:
|
||||
if "REVIEW_RETRY_REQUIRES_COMPLETE_UNKNOWN" not in str(exc):
|
||||
raise
|
||||
result = review_once(rd, managed=True) # no COMPLETE UNKNOWN receipt: a first review
|
||||
salty_offload.log(f"{source}: {result.get('state')} {result.get('decision')} {(result.get('reason') or '')[:120]}")
|
||||
except Exception as exc:
|
||||
salty_offload.log(f"{source}: FAILED {type(exc).__name__}: {str(exc)[:300]}")
|
||||
finally:
|
||||
tgz.unlink(missing_ok=True)
|
||||
if (rd / "ARCHIVED.json").exists():
|
||||
changed, freed = salty_offload.archive(cli, paths, rd)
|
||||
salty_offload.log(f"{source}: overlay {changed} files, freed {freed >> 20}MB")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@ -1,156 +0,0 @@
|
||||
"""Speed reference per job: compile the platform's default configuration once (in a bench dev container,
|
||||
outside any rollout) and write the compiler's latency estimate to agent/speed_ref/<job_id>.json: the agent's jail shows bench/agent
|
||||
read-only and live (not the job directory), so a reference computed after a rollout started still reaches it.
|
||||
|
||||
The agent compares its own model against it (SYSTEM.md 速度要求) instead of compiling the default itself
|
||||
(10-15: 9 of 32 v17 rollouts spent ~4 h each doing that). Jobs that already have the file are skipped.
|
||||
|
||||
BENCH_CONFIG=<a run config> python scripts/speed_ref.py r20260929-04 r20260929-05 r20260929-06 [--active --follow] [--workers 3]
|
||||
|
||||
--active: only jobs a v18+ rollout is on. Salty is shared with the live rollouts (10-15 20:40: their own dev
|
||||
jobs queued once 8 of these were added), so references follow demand instead of compiling all 418 up front.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import shlex
|
||||
import sys
|
||||
import time
|
||||
import traceback
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import yaml
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
from harness import paths, salty_dev # noqa: E402
|
||||
from harness.backends import ptq # noqa: E402
|
||||
from harness.stages.s01_fetch import weights_dir # noqa: E402
|
||||
from harness.stages.s03_export import onnx_path # noqa: E402
|
||||
from harness.stages.s06_hbm import _parse_static_perf # noqa: E402
|
||||
from harness.stagerun import own_inputs_required # noqa: E402
|
||||
|
||||
DEADLINE_S = 10 * 3600
|
||||
|
||||
|
||||
def command(job: dict, ws: Path) -> str:
|
||||
onnx = onnx_path(job)
|
||||
weights = weights_dir(job["model"]) # its files are on personal storage, listed in the container
|
||||
if onnx is None or weights is None:
|
||||
raise ValueError("weights not fetched yet")
|
||||
mine = ws / "model.onnx"
|
||||
calib = Path(job["data"]["calib"]["path"]) / "npy_f32"
|
||||
cfg = ptq.compile_config(job, Path("model.onnx"), calib, Path("ref"), None)
|
||||
(ws / "ref.yaml").write_text(yaml.safe_dump(cfg, sort_keys=False))
|
||||
export = shlex.join(ptq.export_cmd(job, mine, Path("WEIGHTS"))).replace("WEIGHTS", '"$W"')
|
||||
export = f"W=$(ls -d {shlex.quote(str(weights))}/* | grep -v /meta.json | head -1); {export}"
|
||||
return (f"set -e; if [ -f {shlex.quote(str(onnx))} ]; then ln -sfn {shlex.quote(str(onnx))} model.onnx; "
|
||||
f"else {export} > export.log 2>&1; fi; "
|
||||
"hb_compile -c ref.yaml > ref_compile.log 2>&1 || { tail -40 ref_compile.log; exit 1; }; "
|
||||
"hb_model_info ref/*.hbm > ref_info.log 2>&1; " # the dev PATH's OE toolchain, as the agent uses it
|
||||
"sha256sum ref/*.hbm > ref_hbm.sha256; tail -20 ref_info.log; rm -rf ref model.onnx") # keep personal storage small
|
||||
|
||||
|
||||
OUT = paths.AGENT / "speed_ref"
|
||||
|
||||
|
||||
def one(job_dir: Path) -> str:
|
||||
job = json.loads((job_dir / "job.json").read_text())
|
||||
jid = job["job_id"]
|
||||
rd = paths.BENCH_ROOT / "runs" / "zz-speedref" / jid / "rollouts" / "00"
|
||||
ws = rd / "workspace"
|
||||
ws.mkdir(parents=True, exist_ok=True)
|
||||
(rd / "stages").mkdir(exist_ok=True)
|
||||
(rd.parent.parent / "job.json").write_text(json.dumps(job))
|
||||
hd = rd / "stages" / "ref"
|
||||
hd.mkdir(exist_ok=True)
|
||||
ctx = SimpleNamespace(stage_root=rd / "stages", job=job, job_dir=rd.parent.parent, extra={})
|
||||
result = salty_dev.execute(ctx, hd, {"command": command(job, ws), "cwd": "."},
|
||||
time.monotonic() + DEADLINE_S, hd / "out.log", hd / "err.log")
|
||||
info = (ws / "ref_info.log").read_text(errors="ignore") if (ws / "ref_info.log").exists() else ""
|
||||
perf = _parse_static_perf(info)
|
||||
if result.state != "OK" or "latency_ms_est" not in perf:
|
||||
(rd.parent.parent / "FAILED").write_text((hd / "out.log").read_text(errors="ignore")[-4000:]) # not retried
|
||||
unavailable(job, f"{DEADLINE_S // 3600} 小时内没编完" if result.exit_code == 124 else f"编译失败,退出码 {result.exit_code}")
|
||||
return f"{jid}\tFAIL {result.state} {result.exit_code}\t{(hd / 'out.log').read_text(errors='ignore')[-300:]!r}"
|
||||
sha = (ws / "ref_hbm.sha256").read_text().split()[0] if (ws / "ref_hbm.sha256").exists() else None
|
||||
ref = {"schema": "bench.speed_ref/1", "job_id": jid, "march": job["march"], "qcfg": job["qcfg"],
|
||||
"config": "platform default (compile.yaml with no overrides)", **perf, "hbm_sha256": sha,
|
||||
"measured": "hb_model_info static estimate, not a board run", "computed_at": time.time()}
|
||||
publish(jid, ref)
|
||||
return f"{jid}\tOK\t{perf['latency_ms_est']} ms"
|
||||
|
||||
|
||||
def publish(jid: str, ref: dict) -> None:
|
||||
OUT.mkdir(exist_ok=True)
|
||||
tmp = OUT / f".{jid}.json.tmp"
|
||||
tmp.write_text(json.dumps(ref, indent=2, ensure_ascii=False))
|
||||
tmp.replace(OUT / f"{jid}.json")
|
||||
|
||||
|
||||
def unavailable(job: dict, why: str) -> None:
|
||||
"""The default configuration did not compile here (10-16: big S600 depth models, >10 h on 6 cores): say so,
|
||||
so agents stop waiting for the file or compiling the default themselves."""
|
||||
publish(job["job_id"], {"schema": "bench.speed_ref/1", "job_id": job["job_id"], "march": job["march"], "qcfg": job["qcfg"],
|
||||
"latency_ms_est": None, "status": "unavailable",
|
||||
"note": "平台默认配置在这道题上没编出来(" + why + "),本题没有速度参考值:不用再等,也不要自己编默认配置;"
|
||||
"只要别整模型提到 int16/fp16、别让大段算子落到 CPU,并用 board 阶段实测确认。",
|
||||
"computed_at": time.time()})
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("runs", nargs="+")
|
||||
ap.add_argument("--workers", type=int, default=3)
|
||||
ap.add_argument("--cpu", default="6")
|
||||
ap.add_argument("--only", action="append", help="job ids")
|
||||
ap.add_argument("--active", action="store_true", help="only jobs with a rollout under agent v18+")
|
||||
ap.add_argument("--skip", action="append", default=[], help="job ids (e.g. one another run is doing)")
|
||||
ap.add_argument("--follow", action="store_true", help="keep scanning every 2 min; at most --workers at a time")
|
||||
a = ap.parse_args()
|
||||
cfg = paths.config()
|
||||
cfg["execution"]["salty_remote"].setdefault("cpu_by_stage", {})["terminal"] = int(a.cpu)
|
||||
cfg["execution"]["salty_remote"].setdefault("memory_by_stage", {})["terminal"] = "32Gi"
|
||||
failed = paths.BENCH_ROOT / "runs" / "zz-speedref"
|
||||
|
||||
def pending() -> tuple[list[str], dict[str, Path]]:
|
||||
jobs: dict[str, Path] = {}
|
||||
started: dict[str, int] = {}
|
||||
for run in a.runs:
|
||||
for jd in sorted((paths.BENCH_ROOT / "runs" / run).iterdir()):
|
||||
if (jd / "job.json").exists() and (not a.only or jd.name in a.only) and (
|
||||
not a.active or any(own_inputs_required(r) for r in (jd / "rollouts").glob("*"))):
|
||||
jobs.setdefault(jd.name, jd) # same job id = same model, march and qcfg in every run
|
||||
started[jd.name] = started.get(jd.name, 0) + len(list((jd / "rollouts").glob("*")))
|
||||
for j in jobs: # failures recorded before the unavailable file existed (or by an older process)
|
||||
if (failed / j / "FAILED").exists() and not (OUT / f"{j}.json").exists():
|
||||
unavailable(json.loads((jobs[j] / "job.json").read_text()), "编译失败或超时")
|
||||
todo = sorted((j for j in jobs if not (OUT / f"{j}.json").exists() and j not in a.skip),
|
||||
key=lambda j: started.get(j, 0)) # untouched jobs first: new rollouts go there
|
||||
return todo, jobs
|
||||
|
||||
def safe(jid, jd):
|
||||
try:
|
||||
line = one(jd)
|
||||
except Exception as exc:
|
||||
line = f"{jid}\tERROR {type(exc).__name__}: {exc}\t{traceback.format_exc(limit=2)[-300:]!r}"
|
||||
print(time.strftime("%F %T"), line, flush=True)
|
||||
|
||||
running: dict[str, object] = {}
|
||||
with ThreadPoolExecutor(a.workers) as pool:
|
||||
while True:
|
||||
running = {j: f for j, f in running.items() if not f.done()}
|
||||
todo, jobs = pending()
|
||||
for jid in [j for j in todo if j not in running][:a.workers - len(running)]:
|
||||
print(time.strftime("%F %T"), "start", jid, flush=True)
|
||||
running[jid] = pool.submit(safe, jid, jobs[jid])
|
||||
if not a.follow and not running:
|
||||
break
|
||||
time.sleep(120 if a.follow else 5) # --follow: v18 rollouts keep starting on new jobs
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user