baseline 2026-09-15: P1-P3 landed; first PASS trajectory; collaboration rules (CONTRACT §12)

This commit is contained in:
bench 2026-09-15 08:10:26 +08:00
commit 29ef645b58
83 changed files with 7566 additions and 0 deletions

9
.env.local.example Normal file
View File

@ -0,0 +1,9 @@
# cp .env.local.example .env.local && chmod 600 .env.local
# harness 启动时 source;agent 进程永远拿不到这些变量
TEACHER_API_KEY=
STUDENT_API_KEY=
# 板卡:只给 harness 的 07-board 用;优先密钥登录
BOARD_HOST=rdk-sandbox.d-robotics.cc
BOARD_PORT=40010
BOARD_USER=
BOARD_SSH_KEY=~/.ssh/rdk_sandbox

12
.gitignore vendored Normal file
View File

@ -0,0 +1,12 @@
.sync/
__pycache__/
runs/
state/
data/pool/
data/t1/
data/calib/
cache/
datasets/
reports/*.html
.env.local
agent/models.json

144
CONTRACT.md Normal file
View File

@ -0,0 +1,144 @@
# bench/ 契约(CONTRACT v0.4 落地版)
本文件是《BPU 量化轨迹工厂 设计方案 v0.4》的可执行摘要。代码与本文件冲突时,以本文件为准并修代码;本文件与设计方案冲突时,先改设计方案再改这里。
## 0. 三条原则
1. **harness 是 oracle。** 每个 job 先由确定性流水线跑出 ground truth(`oracle/result.json`)。agent 的表现永远相对 oracle 衡量。
2. **agent 只有一个可写区。** `rollouts/<kk>/workspace/`。`job.json`、`harness/eval/`、`data/`、`cache/` 全部只读。agent 说什么不算数,verdict 才算数。
3. **Append-only。** `run_id` 冻结;换工具链、改配置、修 bug 都开新 run。`manifest.json` 记两套 venv 的 `pip freeze` sha256,job 启动前比对,不一致拒跑。状态只在 `state/jobs.sqlite`,文件系统只存内容。
## 1. 目录
```
bench/
├── CONTRACT.md README.md config.yaml requirements.txt
├── bin/bench # CLI 入口
├── registry/ # 输入 · 不可变 · git 管理
│ ├── models.csv matrix.yaml gates.yaml budgets.yaml datasets.yaml boards.yaml
├── data/ # 只读 · agent 永不下载
│ ├── pool/<dataset>/<version>/ # 全量 + manifest.json
│ ├── t1/<dataset>/<seed>.tar # smoke 子集,单包
│ └── calib/<task>/<input_spec>/ # npy,与 val 零交集
├── cache/{weights,onnx}/<sha256>/ cache/baseline/<model>--<dataset>/
├── harness/ # L1 + L3
│ ├── pipeline.py scheduler.py watchdog.py registry.py db.py errors.py gates.py
│ ├── stages/ backends/{ptq,qat}.py eval/
├── agent/ # L2 pi 配置
│ ├── SYSTEM.md settings.json.tmpl models.json.tmpl prompts/ extensions/
├── export/{to_agentrl,redact,reward}.py
├── scripts/ # 一次性:入池、Excel 转换、环境修补、板卡探测
├── state/jobs.sqlite reports/ datasets/<dataset_id>/
└── runs/<run_id>/
├── manifest.json
└── <job_id>/
├── job.json # 只读 · 输入契约
├── oracle/stages/<NN>-<name>/{stdout.log,stderr.log,exit_code,meta.json}
├── oracle/artifacts/<job_id>.{onnx,bc,hbm}
├── oracle/result.json
└── rollouts/<kk>/{workspace/, agent/{session.jsonl,events.jsonl}, claim.json, verdict.json}
```
## 2. 命名
| 标识 | 格式 | 示例 |
|---|---|---|
| run_id | `r<YYYYMMDD>-<seq>` | `r20260914-01` |
| job_id | `<task>--<model>--<march>--<backend>--<qcfg>` | `det--yolov8n--nash-e--ptq--int8` |
| task | 10 类固定 | `cls det seg iseg pose mde stereo flow feat occ` |
| march | `nash-e / nash-m / nash-p` | s100→e · s100p→m · s600→p |
| qcfg | 段内 `a-z0-9.` | `int8` · `fp16int8` |
| rollout | `<job_id>/rollouts/<kk>` | `…/rollouts/03` |
| stage | `<NN>-<name>` | `01-fetch 02-baseline 03-export 04-calibrate 05-compile 06-hbm 07-board 08-eval` |
| dataset_id | `ds<YYYYMMDD>-<seq>` | `ds20260914-01` |
段内字符集 `[a-z0-9.]`,段间 `--`。job_id 既是目录名也是 join key。
## 3. 状态机
**job(由 oracle 决定)**:`PENDING → ORACLE_RUNNING → PASS | FAIL | UNVERIFIED`
**rollout(由 verdict 决定)**:`QUEUED → RUNNING → VERIFYING → PASS | FAIL | BLOCKED | TIMEOUT`
- PASS = 必选门全部 pass。
- 门无法判定(数据缺、板未接)→ 该门 `unverified`,job `UNVERIFIED`。
- `na`(本轮设计上不评)≠ `unverified`(本该评没评成)。na 不阻塞 PASS。
- BLOCKED 判定:oracle 同阶段同类 error_class 失败 → 正确;oracle 通过 → 误判。
## 4. 四道门
| 门 | 必选 | 判定 |
|---|---|---|
| G1 compile | 是 | .hbm 存在;`hb_model_info` 的 march / 输入 dtype / layout 与 job.json 一致;`hb_verifier` 通过;日志无 ERROR |
| G2 numeric | 是 | 校准外样本逐输出余弦 ≥ 阈值(默认 0.99;occ 0.97;feat 逐输出);只报 min |
| G3 task | 是 | 量化实测 − FP32 基线在容差内;kind 必须显式 `abs_pp` 或 `rel_pct`;T1 上记 `pass-screen`,T2 上才 `pass` |
| G4 perf | 否 | 仅 job.json 有 `latency_budget_ms` 才判;mean 与 p95 分开存 |
## 5. 数据三档
| 档 | 用途 | 规模 |
|---|---|---|
| T0 calib | PTQ 校准 | 128–512,train 抽,与 val 零交集 |
| T1 smoke | 默认门 | cls 2000 · det/iseg/pose 1000 · seg 500 · 其余全量 |
| T2 full | 交付数 | 完整 val,开新 run |
复制入池,不软链;`chmod -R a-w`;`manifest.json` 记 sha256/数量/来源/日期。T1 打成单 tar 避开 JuiceFS 小文件。
## 6. 调度与看门狗
- 池:`gpu 4 · cpu 8 · board 1 · agent 3`。板卡严格串行带锁。
- **没有固定超时。** STALLED = 日志 `stall_s`(900 s)不增长 且 进程树 CPU < 5%。硬上限 24 h 只是保险。
- 重阶段(compile / board / eval)由 harness 执行。agent 走 `bench stage run` / `bench stage wait`,不在 bash 里跑 hb_compile。
- rollout 预算 = turns + tokens;墙钟 12 h 仅防挂死。
## 7. 重试与轨迹边界
| 层 | 谁 | 触发 | 轨迹 |
|---|---|---|---|
| oracle 阶段 | harness | 瞬态 error_class | 不是轨迹 |
| rollout 内 | agent | 自己改配置重提 | 同一条轨迹 |
| 新 rollout | harness | matrix k | 全新轨迹,永不串联 |
| repair | harness | oracle FAIL | 全新轨迹,输入是 oracle 现场 |
## 8. error_class
`unsupported_op shape_infer export_fail calib_nan verifier_mismatch`(确定性,不重试)
`oom board_offline stalled`(瞬态,重试 ≤ 2 / ≤ 2 / 1)
`unknown`(进人工分类队列)
## 9. reward(v0,离线算,配方版本化)
```
r = 1.0[G1] + 1.0[G2] + 2.0[G3]
r += 1.0·clip((Δ_agent − Δ_oracle)/|tol|, −1, 1)
r += 0.5·clip((lat_oracle − lat_agent)/lat_oracle, −1, 1)
r −= 0.5·turns/max_turns + 0.5·tokens/max_tokens
r −= 2.0[claim ≠ verdict]
BLOCKED: +1.0 若 oracle 同类失败 else −2.0 ;TIMEOUT: −1.0
G3 = na 时其 2.0 权重按比例分给 G1/G2,精度相对项取 0
```
## 10. 两套 venv
| backend | venv | 关键版本 |
|---|---|---|
| ptq | `~/toolchains/oe-3.7.0/venv` | hbdk4 4.7.5 · horizon_tc_ui 3.5.3 · hmct 2.6.5 |
| qat | `~/toolchains/oellm2/venv` | hbdk4 4.12.3a2 · horizon_plugin_pytorch 3.3.5 |
harness 按 backend 选解释器。**禁止把 backend 判断交给 agent。**
## 11. 下游边界
`~/data_collect` 零耦合。只搬清洗规则 R1–R8 与 redact,落在 `export/`,不 import。
## 12. 协作规则(2026-09-15 起,两个会话同时改一棵树踩过一次)
- **只有一个写者能改一层。** 分层:`harness/`(oracle 流水线、阶段、门、调度)· `agent/`(pi 配置、扩展、提示词)· `export/`(清洗、脱敏、reward)· `registry/`(清单与阈值,改了就是新 run)· `scripts/`。谁在改哪层写进下面的表;别的会话对那层只读。
- **改动只发生在本地副本,机器上不直接改代码。** 流程:改 → `git commit`(`scripts/sync.sh` 会自动提交)→ `scripts/sync.sh` 推送。sync 带守卫:远端自上次推送后被动过就拒绝,先合并再推。
- `registry/models.csv` 是在机器上由 `excel_to_models.py` 生成的,方向反过来:`scripts/sync.sh --pull`。
- 机器上的 `runs/ state/ data/ cache/ datasets/` 是数据,不同步、不手改;改判走 `bench rejudge`,补测走 `bench board-backfill`。
- 正在跑的调度进程(`bench run` / `bench collect`)加载的是启动时的代码;改了 harness 只对新进程生效,子进程工具(`harness.tools.*`、`harness.eval.run`)立刻生效——改评测口径必须同时升 `EVALUATOR_VERSION`。
| 层 | 当前写者 |
|---|---|
| harness/ agent/ export/ scripts/ registry/ | 本会话(Cowork · bench 主线) |
| 其他会话 | 只读;发现问题写成 issue 贴过来,不直接改 |

100
README.md Normal file
View File

@ -0,0 +1,100 @@
# bench/ — BPU 量化轨迹工厂 · P1 骨架
设计:`../BPU量化轨迹工厂-设计方案.html`(v0.4)。契约:`CONTRACT.md`。本 README 只讲怎么把它放到机器上跑起来。
## 0. 上机
```bash
# 本地 → 机器(不带 runs/ state/ data/ cache/,它们在机器上生成)
rsync -av --exclude runs --exclude state --exclude 'data/pool' --exclude 'data/t1' --exclude 'data/calib' \
--exclude cache --exclude datasets --exclude .env.local \
bench/ ming01.wang@120.48.151.164:/mnt/data/ming01.wang/model-compile-hbdk/bench/
# .env.local 单独传(600),不要进 git
scp bench/.env.local ming01.wang@120.48.151.164:/mnt/data/ming01.wang/model-compile-hbdk/bench/.env.local
```
机器上:
```bash
cd /mnt/data/ming01.wang/model-compile-hbdk/bench
export PATH=$PWD/bin:$PATH
# harness 只要 pyyaml;系统 python3 没有就用 PTQ venv 的解释器
python3 -c 'import yaml' || export BENCH_PYTHON=~/toolchains/oe-3.7.0/venv/bin/python
# JuiceFS 是 FUSE:小文件写入实测 ~0.5 s/个。state/(sqlite+锁)、data/(5 万张图)、cache/ 都放本地盘,软链过去
for d in state data cache; do mkdir -p /home/$USER/bench-$d; [ -L $d ] || { rm -rf $d; ln -s /home/$USER/bench-$d $d; }; done
# runs/(每 job 几十个文件)留在 JuiceFS 没问题
```
## 1. P1 顺序(每步都能单独重跑)
| # | 命令 | 产出 |
|---|---|---|
| 1 | `scripts/fix_env.sh` | `.env.oe-package` 的 `PYTHON_BIN` 指向真实 QAT venv(备份 `.bak-日期`) |
| 2 | `python scripts/excel_to_models.py <清单.xlsx> --dry-run` → 改 `COLMAP` → 去掉 `--dry-run` | `registry/models.csv`(338 行) |
| 3 | `source .env.local && scripts/board_probe.sh` | 沙箱板型 → 填 `registry/boards.yaml` 的 `board_type` / `march` |
| 4 | `~/toolchains/oe-3.7.0/venv/bin/python scripts/ingest_pool.py --dataset imagenet1k --dataset coco2017` | `data/pool/…/manifest.json`,只读 |
| 5 | `… scripts/make_t1.py --dataset imagenet1k --dataset coco2017` | `data/t1/<ds>/20260914.tar` |
| 6 | `… scripts/make_calib.py --task cls --input-spec 1x3x224x224` / `--task det --input-spec 1x3x640x640` | `data/calib/…` |
| 7 | `bench preflight` | 0 errors 才往下 |
| 8 | `bench new-run --note "P2 slice"` → `bench expand` | `runs/r…/<job_id>/job.json` × 6,`state/jobs.sqlite` |
| 9 | `bench run --dry-run` | 状态机空转一遍,不碰工具链 |
步骤 4–6 用 PTQ venv 的 python(要 pyarrow / pillow / numpy)。ImageNet 首选 HF parquet 14 片;tar 是备用源,入池时会 `tar -tzf | wc -l` 校验。
## 2. P2 切片(真跑)
```bash
bench run --job cls--resnet18--nash-e--ptq--int8 --parallel 1
bench show cls--resnet18--nash-e--ptq--int8 # 阶段时间线、stderr 尾巴
bench run --job det--yolov8n--nash-e--ptq--int8 --parallel 1
bench status; bench status --fail; bench report # reports/<run_id>.html
```
P2 实测已核对(2026-09-15,resnet18 / nash-e / PTQ):
- `hb_compile 3.5.3` 接受我们的 yaml;`norm_type`、`cal_data_type` 已弃用(去掉了);**cal_data_dir 下只能有 .npy**(它把目录里所有文件当样本读,manifest.json 会炸)→ 校准集放 `<calib>/npy_f32/`,manifest 在上一级。
- `hb_model_info`:I/O 表与 march 打在 stderr,静态 perf 估算(FPS / latency / DDR)打在 stdout;两者都解析,静态估算记进 `G1.static_perf`(不是板端实测)。
- `hb_verifier`:`-m ptq_model.onnx,quantized_model.bc -i <npy>`,逐层余弦表,rc=0;取最后一行(模型输出)余弦对照 G2 阈值。
- x86 跑量化模型:`HB_HBIRRuntime(quantized_model.bc).run()` **0.04 s/张,与 `HB_HBMRuntime(hbm).run_sim()` 逐位一致**(后者 ~20 s/张)。08-eval 全量走 .bc,06-hbm 用 2 张抽查 hbm ≡ bc。`ptq_model.onnx` 走 HB_ONNXRuntime 只是伪量化(cos≈0.9995),不用它出数。oellm2 里的 `hbm_infer` 是板端 RPC,不是模拟器。
- **HB_HBIRRuntime.run 返回的数组引用内部缓冲区**,下一次 run 原地覆盖——必须 `np.array(v, copy=True)`。踩过:一版 08-eval 的 top-1 算成 2.6%。
- resnet18 FP32 在 T1(2000 张分层)top-1 = 68.15(官方全量 69.76,2k 子集 SE≈1 pp,合理);hb_compile 校准余弦 0.999。
仍待核对:`tools/board_perf.py` 解析 `hrt_model_exec perf`(要板);`eval/det.py` 的 YOLOv8 后处理口径(P2 第二个切片)。核对完把 `VERIFIED = True`。
## 3. P3 闭环(agent)
```bash
bench agent setup # ~/.pi/agent/models.json(600),key 来自 .env.local
bench rollout cls--resnet18--nash-e--ptq--int8 --kk 0 --mode e2e --policy teacher --dry-run # 只准备目录、渲染 prompt/settings
bench rollout cls--resnet18--nash-e--ptq--int8 --kk 0 --mode e2e --policy teacher # 真起 pi
python export/to_agentrl.py --runs r20260914-01 # datasets/ds…/teacher.jsonl
```
P3 要核对(代码里标了 `P3`):pi 的 `--append-system-prompt` / `--tools` 参数;`agent/extensions/*.ts` 用到的 ExtensionAPI 形状(`pi.on("tool_call")` 返回 `{block, reason}`、`registerTool`、`turn_end`、`ctx.abort`);`models.json` 的 provider 字段名;`bench stage wait` 会不会被 pi 的 bash 超时掐断(不行就 `--poll 60` 改成 agent 反复调 `stage status`)。
## 4. agent 在 rollout 里能用什么
```
bench stage run calibrate|compile|hbm|board|eval [--config workspace/x.yaml] → handle
bench stage wait <handle> [--poll 60]
bench stage status <handle>
```
march / 输出前缀 / 工作目录被 harness 钉死;`--config` 必须在 `workspace/` 内;`fetch/baseline/export` 不允许 agent 提交。写文件只能在 `workspace/`(path-guard + 文件权限双保险)。凭证:LLM key 在 `~/.pi/agent/models.json`,板卡凭证只进 `07-board` 子进程。
## 5. 实现相对 v0.4 的补充(下次改文档时并入)
- **机器是共享的**(2026-09-14 实测):244 核 / 1.4 TB / **8 张 32 GB 卡**(不是 4 张),4/5/7 被别人跑满、6 占 29 GB;12 个登录用户。`config.yaml pools.gpu.devices: [0,1,2,3]` 钉死只用空闲卡,槽位 → `CUDA_VISIBLE_DEVICES`。
- 资源池是跨进程的(`state/locks/<pool>-<i>.lock` flock):oracle 调度进程、agent 的 `bench stage run`、多个 `bench run` 共用同一组槽位;board 池 1 槽 = 严格串行。model-level 阶段按模型加锁,三个 march 并行时只有一个真跑 01–03。job 认领是原子的(UPDATE … WHERE state='PENDING'),两个调度进程不会重跑同一个 job。
- 新 run 的 manifest 记 `harness_code_sha256`;harness 代码变了只告警,venv / registry / EVALUATOR_VERSION 变了拒跑(实测拦下过一次:改 config.yaml 的 venv 列表后 `bench run` 拒绝,开了新 run)。
- 沙箱板只有一块、一种 march:其他 march 的 job 07-board 直接 SKIPPED → G4 unverified,不阻塞。`board_perf.py` 开测前探测板上有无别人的推理进程 / BPU ratio,最多等 10 分钟,仍忙则照测但标 `contended: true`,reward 的延时相对项忽略这类样本。
- `job.json` 与 mode 无关:一个 job_id 一份,`rollout_plans` 列出 e2e/repair 计划;oracle 与所有 rollout 共用。
- sqlite 默认 TRUNCATE 日志(WAL 在 FUSE 上会 disk I/O error);`BENCH_STATE_DIR` 可把 state 放本地盘。
- 校准集来源:本机 ImageNet / COCO 只有 val,`make_calib.py` 退到 val-holdout(排除 T1)并在 manifest 记 `source`;出 T2 交付数前必须换 train。
- 07-board 失败重试后仍失败 → G4 unverified、继续 08-eval;08-eval 失败 → job UNVERIFIED 带 error_class;01–06 失败 → FAIL。
- claim 缺席(agent 没调 `claim` 工具)→ claim.json 记 `ABSENT`,reward −0.25;claim=PASS 但没有可核验产物 → verdict FAIL 并计 claim≠verdict。
- BLOCKED 正确性:claim.error_class 与 oracle 同类(`unknown` 不算同类)且阶段一致(claim 未填阶段则不比阶段)。
- reward 里相对精度项对 lower-is-better 指标已在 gates 层翻转成"越大越好",配方不用区分。
## 6. 目录里什么进 git
`registry/ harness/ agent/ export/ scripts/ bin/ config.yaml CONTRACT.md README.md`。`runs/ datasets/ state/ data/{pool,t1,calib} cache/ .env.local reports/*.html` 不进(见 `.gitignore`)。

37
agent/SYSTEM.md Normal file
View File

@ -0,0 +1,37 @@
# bench agent · 系统提示(追加在 pi 默认系统提示之后)
你在一个受控的量化上板环境里工作。规则不是建议,违反会被工具层拒绝或被判负。
## 你能写的地方
只有当前目录(`workspace/`)。其他一切只读:`../job.json`、`../../oracle/`(如果存在)、`data/`、`cache/`、`harness/`。越界写入会被 path-guard 拒绝并记录。
## 重阶段不在 bash 里跑
编译、上板、评测由 harness 执行。你提交、你等待、你看日志:
```
bench stage run calibrate # 生成编译配置(也可先看 job.json 自己写一份)
bench stage run compile --config workspace/compile.yaml
bench stage wait <handle> --poll 60
bench stage run hbm # 收产物 + hb_model_info + hb_verifier
bench stage run board # 板端 perf(可选,有板才跑)
bench stage run eval # G2 / G3
```
`--config` 必须在 `workspace/` 内。march、输出前缀、工作目录由 harness 钉死,改了也不生效。不要自己调用 `hb_compile`、`hb_mapper`、`hrt_model_exec`、`ssh`——它们要么不在你的环境里,要么会被拒。
## 同一方法最多重试 1 次
编译报错后,改配置再提交是允许的、被期待的。但同一套配置原样重提第三次没有意义。换方法:改校准参数、改输入类型、改算子、拆模型。三次不同方法仍失败 → 报 BLOCKED。
## 结束前必须调用 `claim` 工具
```
claim(terminal="PASS"|"FAIL"|"BLOCKED", stage=..., error_class=..., summary="…", artifacts=[…])
```
**预算规则**:剩 5 轮时你会收到一条预算提醒——收到后无论进展如何,先调用 `claim` 汇报当前状态(不确定就报 FAIL 并写清到了哪一步),再决定要不要继续。预算耗尽后除 `claim` 以外的工具都会被拒绝;没有 claim 就结束的轨迹被记为 claim 缺席,另扣分。
你的 claim 会与 harness 独立核验的 verdict 比对。claim=PASS 而 verdict=FAIL 是最重的扣分项,比老实报 FAIL 差得多。不确定就报 FAIL 并说明。BLOCKED 只在你确认是工具链/算子层面无法完成时使用,并给出 error_class(unsupported_op / shape_infer / calib_nan / verifier_mismatch 之一)。
## 不要做的事
不下载数据集或权重(都在只读区);不改 `job.json`;不猜指标——指标只来自 `bench stage run eval` 的输出;不在 bash 里 `sleep` 等编译,用 `bench stage wait`。
## 工具链参考
`.horizon/skills/` 下的 skill 是这条工具链的文档。`oe-doc <query>` 可以查 OE 文档。两套 venv 的选择由 harness 按 job.json 的 backend 决定,你不需要也不能选。

View File

@ -0,0 +1,97 @@
/**
* budget-kill — turns 与 tokens 是真正的预算(agent 自己能控制的量)。三段式,先软后硬:
*
* 1. 预警(剩 WARN_AHEAD 轮):塞一条 steer 消息告诉 agent"预算快没了,先 claim"。
* 2. 耗尽(turns ≥ MAX 或 tokens ≥ MAX):写 rollout.json state=TIMEOUT;此后 **除 claim 外的所有工具调用都被 block**,
* 模型只剩 claim 可用 → 调了 claim 或干脆停手 → 会话自然走到 agent_end → JSON 流完整 → 轨迹可导出。
* 3. 兜底(turns ≥ MAX + GRACE_TURNS):模型还在原地打转 → process.exit(75),harness 按 TIMEOUT 处理。
*
* 第一版在 turn_end 直接 ctx.abort():实测 pi 0.73.1 只掐断在途请求,进程继续下一轮,第一条轨迹跑到 turn 41 也没停;
* 而立刻 process.exit 会丢 agent_end,导出端拿不到线性化的 messages[]——所以硬退出只能是最后一手。
*
* 预算从 env 读(BENCH_MAX_TURNS / BENCH_MAX_TOKENS),由 rollout 驱动器按 config.yaml 设置。
*/
import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
import * as fs from "node:fs";
import * as path from "node:path";
const ROLLOUT = process.env.BENCH_ROLLOUT_DIR ?? "";
const MAX_TURNS = Number(process.env.BENCH_MAX_TURNS ?? 40);
const MAX_TOKENS = Number(process.env.BENCH_MAX_TOKENS ?? 200000);
const WARN_AHEAD = Number(process.env.BENCH_WARN_AHEAD_TURNS ?? 5);
const GRACE_TURNS = Number(process.env.BENCH_GRACE_TURNS ?? 3);
let turns = 0;
let tokens = 0;
let exhausted: string | null = null;
let warned = false;
function writeRollout(patch: Record<string, unknown>) {
if (!ROLLOUT) return;
const p = path.resolve(ROLLOUT, "rollout.json");
let rec: any = {};
try { rec = JSON.parse(fs.readFileSync(p, "utf8")); } catch { /* 新建 */ }
Object.assign(rec, patch, { turns, tokens });
fs.writeFileSync(p, JSON.stringify(rec, null, 2));
}
function usageOf(msg: any): number {
const u = msg?.usage;
if (!u) return 0;
return Number(u.input ?? u.input_tokens ?? 0) + Number(u.output ?? u.output_tokens ?? 0)
+ Number(u.cacheRead ?? 0) + Number(u.cacheWrite ?? 0);
}
export default function (pi: ExtensionAPI) {
// 2. 耗尽后只放行 claim
pi.on("tool_call", async (event: any) => {
if (!exhausted) return undefined;
const name: string = event.toolName ?? event.tool ?? "";
if (name === "claim") return undefined;
return { block: true, reason: `预算已耗尽(${exhausted})。现在只能调用 claim 汇报最终状态;不确定就报 FAIL 并说明。` };
});
pi.on("turn_end", async (event: any, ctx: any) => {
turns += 1;
tokens += usageOf(event?.message);
const left = MAX_TURNS - turns;
// 1. 预警
if (!warned && !exhausted && (left <= WARN_AHEAD || tokens >= MAX_TOKENS * 0.85)) {
warned = true;
writeRollout({ budget_warned_at_turn: turns });
try {
pi.sendMessage(
{ customType: "bench-budget", content: `【预算提醒】剩 ${Math.max(left, 0)} 轮 / ${Math.max(MAX_TOKENS - tokens, 0)} tokens。无论进展如何,请现在就调用 claim 汇报当前状态(不确定就报 FAIL 并说明到哪一步),之后再继续。`, display: true },
{ triggerTurn: false, deliverAs: "steer" },
);
} catch { /* sendMessage 不可用就只靠 SYSTEM.md 里的规则 */ }
}
// 2. 耗尽
if (!exhausted) {
if (turns >= MAX_TURNS) exhausted = `max_turns ${MAX_TURNS}`;
else if (tokens >= MAX_TOKENS) exhausted = `max_tokens ${MAX_TOKENS}`;
if (exhausted) {
writeRollout({ state: "TIMEOUT", timeout_reason: exhausted, timed_out_at: Date.now() / 1000 });
try {
pi.sendMessage(
{ customType: "bench-budget", content: `【预算耗尽:${exhausted}】除 claim 以外的工具已被禁用。请立即调用 claim,然后结束。`, display: true },
{ triggerTurn: false, deliverAs: "steer" },
);
} catch { /* ignore */ }
}
}
// 3. 兜底:耗尽后还赖着不走
if (exhausted && turns >= MAX_TURNS + GRACE_TURNS) {
writeRollout({ hard_killed_at_turn: turns });
try { ctx?.abort?.(); } catch { /* ignore */ }
setTimeout(() => process.exit(75), 2000).unref?.();
}
});
pi.on("agent_end", async () => {
writeRollout({ agent_end_turns: turns, agent_end_tokens: tokens });
});
}

View File

@ -0,0 +1,52 @@
/**
* claim-card — 强制结构化汇报。注册 `claim` 工具,写 <rollout>/claim.json。
*
* claim 是 agent 自报,不可信;L3 用 verdict 核验并比对。agent_end 时若没调用过 claim,
* 写一份 { terminal: "ABSENT" },reward 小额扣分(见 export/reward.py)。
*
* P3 核对:pi.registerTool 的签名与 Type(typebox)导入路径。
*/
import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
import { Type } from "@sinclair/typebox";
import * as fs from "node:fs";
import * as path from "node:path";
const ROLLOUT = process.env.BENCH_ROLLOUT_DIR ?? "";
const CLAIM = ROLLOUT ? path.resolve(ROLLOUT, "claim.json") : path.resolve(process.cwd(), "claim.json");
let claimed = false;
export default function (pi: ExtensionAPI) {
pi.registerTool({
name: "claim",
label: "claim",
description:
"结束前必须调用一次。汇报你认为的最终状态。会与 harness 独立核验的 verdict 比对;claim=PASS 而 verdict=FAIL 是最重扣分项。",
parameters: Type.Object({
terminal: Type.Union([Type.Literal("PASS"), Type.Literal("FAIL"), Type.Literal("BLOCKED")]),
stage: Type.Optional(Type.String({ description: "失败/阻塞发生的阶段,如 05-compile" })),
error_class: Type.Optional(
Type.Union([
Type.Literal("unsupported_op"), Type.Literal("shape_infer"), Type.Literal("export_fail"),
Type.Literal("calib_nan"), Type.Literal("verifier_mismatch"), Type.Literal("unknown"),
]),
),
summary: Type.String({ description: "你做了什么、为什么是这个结论,3–8 句" }),
artifacts: Type.Optional(Type.Array(Type.String(), { description: "workspace 内的关键产物相对路径" })),
metrics_seen: Type.Optional(Type.Record(Type.String(), Type.Number(), { description: "你从 bench stage eval 输出里看到的数字(不是你估计的)" })),
}),
async execute(_id: string, params: any) {
const rec = { ...params, claimed_at: Date.now() / 1000, schema: "bench.claim/1" };
fs.writeFileSync(CLAIM, JSON.stringify(rec, null, 2)); // 最后一次 claim 为准(预算提醒后可能先 claim 一次再继续)
try { fs.appendFileSync(CLAIM.replace(/\.json$/, "s.jsonl"), JSON.stringify(rec) + "\n"); } catch { /* 历史可选 */ }
claimed = true;
return { content: [{ type: "text", text: `claim 已记录:${params.terminal}` }], details: rec };
},
});
pi.on("agent_end", async () => {
if (!claimed && !fs.existsSync(CLAIM)) {
fs.writeFileSync(CLAIM, JSON.stringify({ terminal: "ABSENT", schema: "bench.claim/1", claimed_at: Date.now() / 1000 }, null, 2));
}
});
}

View File

@ -0,0 +1,33 @@
/**
* oe-doc — 把 oe-mcp 的文档检索包成 pi 工具(pi 不支持 MCP)。
*
* 实现方式:子进程调用 `bench/agent/oe-doc.sh <query>`,它再去调 oe-mcp 的 CLI 或直接 grep .horizon/docs。
* 只读;输出截断到 8k 字符。P3 把 oe-doc.sh 接到真实的 oe-mcp 入口。
*/
import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
import { Type } from "@sinclair/typebox";
import { execFileSync } from "node:child_process";
import * as path from "node:path";
const BENCH_ROOT = process.env.BENCH_ROOT ?? path.resolve(__dirname, "..", "..");
const SCRIPT = path.resolve(BENCH_ROOT, "agent", "oe-doc.sh");
export default function (pi: ExtensionAPI) {
pi.registerTool({
name: "oe_doc",
label: "oe_doc",
description: "查 OpenExplorer / hbdk / horizon_tc_ui 文档。输入关键词或报错片段,返回相关文档段落。",
parameters: Type.Object({ query: Type.String(), max_chars: Type.Optional(Type.Number({ default: 8000 })) }),
async execute(_id: string, params: any) {
let out = "";
try {
out = execFileSync(SCRIPT, [params.query], { encoding: "utf8", timeout: 60_000, maxBuffer: 4 << 20 });
} catch (e: any) {
out = `oe-doc 失败: ${e?.message ?? e}`;
}
const max = Number(params.max_chars ?? 8000);
if (out.length > max) out = out.slice(0, max) + `\n…(截断,共 ${out.length} 字符)`;
return { content: [{ type: "text", text: out }], details: { query: params.query, chars: out.length } };
},
});
}

View File

@ -0,0 +1,125 @@
/**
* path-guard — agent 只有一个可写区:<rollout>/workspace/。
*
* 拦截 write / edit 的目标路径;拦截 bash 里明显的越界写入(重定向、cp/mv/rm/tee/chmod 指向 workspace 之外)。
* bash 的静态检查不可能完备,第二道保险是文件系统权限:harness 把 job.json / data / cache / eval 设为只读。
* 每次拒绝写一行到 <rollout>/agent/guard.log,L4 导出时可统计"越界尝试次数"。
*
* API 形状对照 pi-coding-agent 的 ExtensionAPI(P3 按安装版本核对:pi.on("tool_call") 返回 { block, reason })。
*/
import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
import * as fs from "node:fs";
import * as path from "node:path";
const ROLLOUT = process.env.BENCH_ROLLOUT_DIR ?? "";
const WORKSPACE = ROLLOUT ? path.resolve(ROLLOUT, "workspace") : process.cwd();
const GUARD_LOG = ROLLOUT ? path.resolve(ROLLOUT, "agent", "guard.log") : "";
function inside(p: string): boolean {
const abs = path.resolve(WORKSPACE, p);
return abs === WORKSPACE || abs.startsWith(WORKSPACE + path.sep);
}
function log(kind: string, detail: string) {
if (!GUARD_LOG) return;
try {
fs.mkdirSync(path.dirname(GUARD_LOG), { recursive: true });
fs.appendFileSync(GUARD_LOG, JSON.stringify({ ts: Date.now() / 1000, kind, detail }) + "\n");
} catch { /* 日志失败不影响拒绝 */ }
}
// bash 静态规则:命中即拒。故意保守——误杀一个合法命令的代价远小于放过一次越界写。
// 第一版把 `2>/dev/null` 也当成了"重定向到绝对路径",第一条真实轨迹里 10 次调用被拒 4 次,全是误杀 → 重定向单独判目标。
// 被禁的**命令**:只在"作为命令被调用"的位置匹配(行首 / ; | && 之后 / env 赋值之后),
// 否则 `ps aux | grep hb_compile`、`grep -r hb_compile logs/` 这类只读排查也会被拒——第一条轨迹里真的被拒了。
const FORBIDDEN_CMDS = ["hb_compile", "hb_mapper", "hb_verifier", "hrt_model_exec", "hb_perf",
"ssh", "scp", "rsync", "wget", "curl", "pip", "pip3", "conda", "npm"];
const CMD_POSITION = new RegExp(
String.raw`(?:^|[;&|]\s*|\$\(|` + "`" + String.raw`)\s*(?:[A-Za-z_][A-Za-z0-9_]*=\S*\s+)*(?:sudo\s+|env\s+|nohup\s+|time\s+)*(` +
FORBIDDEN_CMDS.join("|") + String.raw`)\b`);
const WRITE_PATTERNS: RegExp[] = [
/\bsleep\s+[0-9]{3,}/, // 用 bench stage wait,不要 sleep 等编译
/\bgit\s+push\b/,
];
// 写文件的命令:destLast 的最后一个参数才是目的地(cp ../job.json . 是合法的"读进来");
// 其余命令的每个非选项参数都是目标。/tmp 放行(scratch),/dev/null 放行。
const DEST_LAST = new Set(["cp", "mv", "ln", "install", "rsync"]);
const ALL_ARGS = new Set(["rm", "rmdir", "tee", "touch", "mkdir", "chmod", "chown", "truncate", "dd", "shred"]);
function writeTargets(segment: string): string[] {
const toks = segment.trim().split(/\s+/).filter(Boolean);
let i = 0;
while (i < toks.length && /^[A-Za-z_][A-Za-z0-9_]*=/.test(toks[i])) i++; // 跳过 env 赋值
while (i < toks.length && ["sudo", "env", "nohup", "time"].includes(toks[i])) i++;
const cmd = (toks[i] ?? "").replace(/^.*\//, "");
const args = toks.slice(i + 1).filter((t) => !t.startsWith("-")).map((t) => t.replace(/^['"]|['"]$/g, ""));
if (DEST_LAST.has(cmd)) return args.length ? [args[args.length - 1]] : [];
if (ALL_ARGS.has(cmd)) return args;
return [];
}
function badWriteTarget(cmd: string): string | null {
for (const seg of cmd.split(/;|\|\||&&|\||&/)) {
for (const t of writeTargets(seg)) {
if (t.startsWith("/tmp/") || t === "/dev/null") continue;
if (!inside(t)) return t;
}
}
return null;
}
const REDIRECT_OK = new Set(["/dev/null", "/dev/stderr", "/dev/stdout"]);
/** 找出所有输出重定向目标(> >> 2> &>),有一个落在 workspace 外就拒。相对路径以 cwd=workspace 解析;`cd` 过的命令无法追踪,靠文件权限兜底。 */
function badRedirect(cmd: string): string | null {
const re = /(?:^|[\s;|&(])(?:\d?>>?|&>)\s*['"]?([^\s'"|;&)]+)/g;
let m: RegExpExecArray | null;
while ((m = re.exec(cmd)) !== null) {
const target = m[1];
if (target.startsWith("&")) continue; // 2>&1
if (REDIRECT_OK.has(target)) continue;
if (target.startsWith("/proc/self/fd/")) continue;
if (!inside(target)) return target;
}
return null;
}
export default function (pi: ExtensionAPI) {
pi.on("tool_call", async (event: any, _ctx: any) => {
const name: string = event.toolName ?? event.tool ?? "";
const input: any = event.input ?? {};
if (name === "write" || name === "edit") {
const p: string = input.path ?? input.file_path ?? "";
if (!p || !inside(p)) {
log("deny-" + name, p);
return { block: true, reason: `path-guard: 只能写 workspace/ 内的文件(${p})` };
}
}
if (name === "bash") {
const cmd: string = input.command ?? "";
const bad = badRedirect(cmd);
if (bad) {
log("deny-bash-redirect", cmd.slice(0, 500));
return { block: true, reason: `path-guard: 重定向到 workspace 之外(${bad})。写文件只在 workspace/。` };
}
const forbidden = CMD_POSITION.exec(cmd);
if (forbidden) {
log("deny-bash-cmd", cmd.slice(0, 500));
return { block: true, reason: `path-guard: 不要直接调用 ${forbidden[1]}。编译/上板/评测走 bench stage run(它会选对 venv、拿资源池、落日志)。只读地 grep 这个词是允许的。` };
}
const badTarget = badWriteTarget(cmd);
if (badTarget) {
log("deny-bash-write", cmd.slice(0, 500));
return { block: true, reason: `path-guard: 写到 workspace 之外(${badTarget})。读外面的文件可以,写只能在 workspace/(或 /tmp)。` };
}
for (const rx of WRITE_PATTERNS) {
if (rx.test(cmd)) {
log("deny-bash", cmd.slice(0, 500));
return { block: true, reason: `path-guard: 命令被拒(${rx})。重阶段用 bench stage run,写文件只在 workspace/。` };
}
}
}
return undefined;
});
}

37
agent/models.json.tmpl Normal file
View File

@ -0,0 +1,37 @@
{
"_comment": "渲染到 ~/.pi/agent/models.json(chmod 600)。{TEACHER_API_KEY}/{STUDENT_*} 由 `bench agent setup` 从 bench/.env.local 填。字段名对照 pi 版本文档(P3 核对:providers.<name>.{baseUrl, api, apiKey, models[]})。",
"providers": {
"dr-gateway": {
"baseUrl": "https://ai-api.d-robotics.cc/v1",
"api": "openai-completions",
"apiKey": "{TEACHER_API_KEY}",
"models": [
{
"id": "glm-5.3-flash",
"name": "GLM 5.3 Flash (teacher)",
"reasoning": false,
"input": ["text"],
"contextWindow": 128000,
"maxTokens": 16384,
"cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }
}
]
},
"student-vllm": {
"baseUrl": "{STUDENT_BASE_URL}",
"api": "openai-completions",
"apiKey": "{STUDENT_API_KEY}",
"models": [
{
"id": "{STUDENT_MODEL}",
"name": "student (vLLM)",
"reasoning": false,
"input": ["text"],
"contextWindow": 65536,
"maxTokens": 8192,
"cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }
}
]
}
}
}

14
agent/oe-doc.sh Executable file
View File

@ -0,0 +1,14 @@
#!/usr/bin/env bash
# oe-doc.sh <query> — 只读文档检索。P3 接到 oe-mcp 的 CLI;现在退化为在 .horizon 里 grep。
set -uo pipefail
Q="${1:-}"
[ -z "$Q" ] && { echo "usage: oe-doc.sh <query>"; exit 2; }
ROOT="${BENCH_ROOT:-$(cd "$(dirname "$0")/.." && pwd)}"
HZ="$ROOT/../.horizon"
if command -v oe-mcp-cli >/dev/null 2>&1; then
exec oe-mcp-cli search "$Q"
fi
grep -rIl --include='*.md' -i -- "$Q" "$HZ" 2>/dev/null | head -8 | while read -r f; do
echo "=== $f"
grep -n -i -B2 -A8 -- "$Q" "$f" | head -60
done

18
agent/prompts/e2e-occ.md Normal file
View File

@ -0,0 +1,18 @@
# 任务:把 {model}(3D 占据)编译到 {march}({backend}),本轮目标是编译通过 + 板端可运行,不评精度
job:`{job_id}` run:`{run_id}`
job.json:`{job_json}`(只读) 你的可写区:当前目录
## 本轮门
- G1 compile:`.hbm` 产出、march = {march}、`hb_verifier` 通过
- G2 numeric:余弦 ≥ {g2_threshold}(校准来源为 synthetic 时只作参考)
- G3:本轮不评(na),不要尝试跑评测
- G4:板端 perf 照常测,这是本轮最有价值的数字
## 校准输入
harness 按 repo_demo → nuscenes_128 → synthetic 三级回退,`bench stage run calibrate` 的输出会告诉你用了哪级。你可以在 workspace 里自己准备更好的校准 npy 并在 compile.yaml 里指向它(必须在 workspace 内)。
## 预期
BEV pooling / 体素化 / 稀疏卷积类算子在 {march} 上很可能不支持。`.horizon` 没有 occ 专用 skill。如果三种不同方法(改导出、拆算子、改输入规格)都过不了编译,报 BLOCKED 并给出 error_class 与具体算子名——这本身就是有价值的结果。
预算:{max_turns} 轮、{max_tokens} tokens;剩 5 轮会收到提醒,那时先 claim。

21
agent/prompts/e2e.md Normal file
View File

@ -0,0 +1,21 @@
# 任务:把 {model}({task})量化并编译到 {march}({backend} / {qcfg}),通过全部必选门
job:`{job_id}` run:`{run_id}`
job.json:`{job_json}`(只读) 数据:`{data_dir}`(只读) 你的可写区:当前目录
## 目标
1. G1 compile:产出 `.hbm`,march = {march},输入 {input_spec},`hb_verifier` 通过
2. G2 numeric:校准外样本逐输出余弦 ≥ {g2_threshold}
3. G3 task:{g3_metric} 在 T1 上相对 FP32 基线退化不超过 {g3_tol}({g3_kind})
4. G4 perf:只记录,不判
## 你手上有什么
- FP32 ONNX:`{onnx}`(03-export 已产出)
- FP32 基线:`{baseline_dir}/metrics.json`
- 校准集:`{calib_dir}`({calib_n} 个 npy,与 val 零交集)
- T1 评测子集:`{t1_tar}`({t1_n} 张)
## 流程
先读 job.json 和 `.horizon/skills/` 里与 {task} / {backend} 相关的 skill,再决定编译配置。提交 `bench stage run compile --config workspace/<你的>.yaml`,`bench stage wait`。失败就读日志改配置;同一方法最多重试 1 次。编译过了跑 `bench stage run hbm`,再 `bench stage run eval`。最后调用 `claim`。
预算:{max_turns} 轮、{max_tokens} tokens。等待编译不消耗预算。剩 5 轮会收到提醒——那时先 claim,别等到耗尽。

15
agent/prompts/repair.md Normal file
View File

@ -0,0 +1,15 @@
# 任务:修复一次失败的量化编译
job:`{job_id}` run:`{run_id}` 目标 {march} / {backend} / {qcfg}
job.json:`{job_json}`(只读) 你的可写区:当前目录
## 失败现场(来自确定性流水线 oracle,不是别人的尝试)
- 失败阶段:`{oracle_stage_failed}`
- error_class:`{oracle_error_class}`
- 日志:`{oracle_stage_dir}/stderr.log`、`stdout.log`(只读)
- oracle 用的编译配置:`{oracle_compile_yaml}`(只读,复制到 workspace 再改)
## 要做的
读懂失败原因,改配置或改模型导出方式,重新提交 `bench stage run compile --config workspace/<你的>.yaml`,直到 G1 通过,然后跑 `hbm` 与 `eval`。同一方法最多重试 1 次;三种方法都不行 → BLOCKED,说明你认为 oracle 的失败是否为工具链固有限制。
最后调用 `claim`。预算:{max_turns} 轮、{max_tokens} tokens;剩 5 轮会收到提醒,那时先 claim。

17
agent/settings.json.tmpl Normal file
View File

@ -0,0 +1,17 @@
{
"_comment": "渲染到 <rollout>/workspace/.pi/settings.json。{PROJECT_ROOT}/{BENCH_ROOT}/{MODEL} 由 harness/rollout.py 替换。compaction 关:训练数据里出现 CompactionSummaryMessage 是脏的,上下文靠 max_turns 控制。",
"defaultProvider": "dr-gateway",
"defaultModel": "{MODEL}",
"compaction": { "enabled": false },
"skills": [
"{PROJECT_ROOT}/.horizon/skills/*/*",
"{PROJECT_ROOT}/.horizon/skills/horizon-router"
],
"extensions": [
"{BENCH_ROOT}/agent/extensions/path-guard.ts",
"{BENCH_ROOT}/agent/extensions/claim-card.ts",
"{BENCH_ROOT}/agent/extensions/budget-kill.ts",
"{BENCH_ROOT}/agent/extensions/oe-doc.ts"
],
"tools": ["read", "edit", "write", "bash", "grep"]
}

8
bin/bench Executable file
View File

@ -0,0 +1,8 @@
#!/usr/bin/env bash
# bench CLI 入口。harness 只依赖 pyyaml;工具链在各自 venv 里由 harness 子进程调用。
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export BENCH_ROOT="${BENCH_ROOT:-$(dirname "$HERE")}"
export PYTHONPATH="$BENCH_ROOT${PYTHONPATH:+:$PYTHONPATH}"
PY="${BENCH_PYTHON:-python3}"
exec "$PY" -m harness.cli "$@"

81
config.yaml Normal file
View File

@ -0,0 +1,81 @@
# bench/config.yaml — 运行时配置(不进 job.json;job.json 只快照与结果相关的东西)
version: 1
paths:
# 项目根;harness 默认取本文件所在目录的上一级
project_root: null
# 两套 venv。harness 按 backend 选,agent 无权改
venv:
ptq: ~/toolchains/oe-3.7.0/venv # hb_compile / hb_model_info / hb_verifier(无 pip,不装东西)
qat: ~/toolchains/oellm2/venv # plugin + hbdk4 4.12 + hbm_infer(不装东西)
bench: ~/toolchains/bench-venv # 我们自己的:torch/torchvision/ultralytics/pycocotools/pyarrow — 权重、导 ONNX、FP32 基线、评测打分、数据入池
# harness 自身用的解释器(只需 pyyaml),默认 python3
harness_python: python3
# hb_compile 的 working_dir 根目录(中间产物几百 MB/个)。必须放本地盘:
# 实测 JuiceFS 写 1 MB=360 ms vs 本地盘 3 ms,大模型编译会卡在 D 状态被看门狗误判挂死。
# 留空则退回 runs/<run>/<job>/oracle/stages/05-compile/work(JuiceFS 上,只适合小规模)。
work_root: /home/ming01.wang/bench-work
# .horizon skill pack(只读)
horizon: ../.horizon
# agent 运行时(scripts/setup_agent.sh → ~/toolchains/node;pi 是 node 脚本,node 必须在 PATH 或这里给全路径)
pi_bin: ~/toolchains/node/bin/pi
node_bin: ~/toolchains/node/bin/node
pools:
# 机器是共享的:2026-09-14 实测 8 张卡,4/5/7 被别人跑满、6 占了 29 GB;0–3 空闲。
# 槽位 i → devices[i] → CUDA_VISIBLE_DEVICES。别人的卡不碰。
# reserve_for_agent:只给 rollout 的 `bench stage run` 用的槽位,oracle 拿不到。
# 实测(2026-09-15):两个 oracle 调度进程把 cpu 8/8 占满,第一条 rollout 的编译在队列里饿了 60 分钟。
gpu: { workers: 4, devices: [0, 1, 2, 3], reserve_for_agent: 1 } # baseline / eval / qat-calibrate
cpu: { workers: 8, reserve_for_agent: 2 } # export / hb_compile
board: { workers: 1 } # 一块板 = 1,严格串行,带锁(不预留:agent 与 oracle 都得排队)
agent: { workers: 3, rpm: 60 } # 受 LLM 端点限速
watchdog:
stall_s: 900 # 日志不增长 且 进程树 CPU < cpu_idle_pct 持续 stall_s → STALLED
cpu_idle_pct: 5
poll_s: 30
hard_ceiling_s: 86400 # 24h 纯保险
soft_alert: p95x3 # 超过历史 p95 × 3 只告警,不杀
perf:
warmup: 20
runs: 200
stats: [mean, p95]
rollouts:
k: 4
max_turns: 40
max_tokens: 200000
hard_ceiling_s: 43200 # 12h 只防 agent 进程挂死
# --tools 是白名单,同时作用于内置工具与 extension 注册的工具。
# 踩过(2026-09-15 第一条轨迹):claim / oe_doc 没列进来 → claim-card 注册了工具但被白名单关掉,
# agent 全程无法 claim,213 次 bash 调用一次 claim 都没有。extension 工具必须显式列出。
tools: [read, edit, write, bash, grep, claim, oe_doc]
retry:
transient: 2 # oom / board_offline
stalled: 1 # 进 long 车道
deterministic: 0
scheduling:
priority_order: [A, B, C]
c_lane: low # C 类走独立低优先级车道
estimate_source: history_then_params
data:
default_tier: t1
calib_n_default: 128
# LLM 端点。key 不写这里,写 bench/.env.local(gitignore)
endpoints:
teacher:
provider: openai-compatible
base_url: https://ai-api.d-robotics.cc/v1
model: glm-5.3-flash
api_key_env: TEACHER_API_KEY
student:
provider: openai-compatible
base_url: null # vLLM 起来后填
model: null
api_key_env: STUDENT_API_KEY

0
data/.gitkeep Normal file
View File

115
export/redact.py Normal file
View File

@ -0,0 +1,115 @@
"""脱敏。规则移植自 data_collect/redact_archive.py 的思路,不 import 那边任何东西。
agent 的 bash 输出会进轨迹:用户目录、主机名、IP、密钥、token、邮箱都要抹。
同一条轨迹内占位符稳定(同一个用户名每次映射到同一个 <USER_1>),保证轨迹可读。
"""
from __future__ import annotations
import json
import os
import re
import sys
from pathlib import Path
RULES: list[tuple[str, re.Pattern, str]] = [
("api_key", re.compile(r"\bsk-[A-Za-z0-9_\-]{10,}\b"), "<API_KEY>"),
("hf_token", re.compile(r"\bhf_[A-Za-z0-9]{20,}\b"), "<HF_TOKEN>"),
("bearer", re.compile(r"(?i)(bearer\s+)[A-Za-z0-9._\-]{16,}"), r"\1<TOKEN>"),
("kv_secret", re.compile(r"(?i)\b((?:api[_-]?key|token|secret|passw(?:or)?d|authorization)\s*[=:]\s*)['\"]?[^\s'\"]{6,}"), r"\1<REDACTED>"),
("ssh_target", re.compile(r"\b[a-z][a-z0-9._-]{0,31}@(?:\d{1,3}(?:\.\d{1,3}){3}|[a-z0-9.-]+\.[a-z]{2,})\b"), "<SSH_TARGET>"),
("email", re.compile(r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b"), "<EMAIL>"),
("ipv4", re.compile(r"\b(?!(?:0|127|10|192\.168)\.)\d{1,3}(?:\.\d{1,3}){3}\b"), "<IP>"),
("hostname_dr", re.compile(r"\b[a-z0-9-]+\.d-robotics\.cc\b"), "<HOST>"),
("instance", re.compile(r"\binstance-[a-z0-9]{6,}\b"), "<INSTANCE>"),
]
USER_DIR = re.compile(r"(/mnt/data/|/home/)([A-Za-z0-9][A-Za-z0-9._-]{1,31})(?=/|\b)")
class Redactor:
def __init__(self):
self.users: dict[str, str] = {}
self.counts: dict[str, int] = {}
def _user(self, m: re.Match) -> str:
name = m.group(2)
if name not in self.users:
self.users[name] = f"<USER_{len(self.users) + 1}>"
self.counts["user_dir"] = self.counts.get("user_dir", 0) + 1
return m.group(1) + self.users[name]
def text(self, s: str) -> str:
if not isinstance(s, str) or not s:
return s
s = USER_DIR.sub(self._user, s)
for name, rx, rep in RULES:
s, n = rx.subn(rep, s)
if n:
self.counts[name] = self.counts.get(name, 0) + n
return s
def obj(self, o):
if isinstance(o, str):
return self.text(o)
if isinstance(o, list):
return [self.obj(x) for x in o]
if isinstance(o, dict):
return {k: self.obj(v) for k, v in o.items()}
return o
# ---- 第二遍:裸用户名
# 第一遍只抹路径里的用户名(/mnt/data/<u>、/home/<u>)。但 agent 的 bash 输出里到处是
# `ls -la` 的属主/属组列("drwxr-xr-x 4 ming01.wang ming01.wang")、`whoami`、进程表的 USER 列——
# 那些不是路径,第一遍抓不到。实测第一条轨迹:路径替换 64 处,裸用户名仍残留 62 处。
def bare_user_regex(self, extra: set[str] | None = None) -> re.Pattern | None:
names = set(self.users) | (extra or set())
names = {n for n in names if len(n) >= 3 and not n.isdigit()}
if not names:
return None
return re.compile(r"(?<![A-Za-z0-9._@-])(" + "|".join(re.escape(n) for n in sorted(names, key=len, reverse=True)) + r")(?![A-Za-z0-9-])")
def bare_pass(self, o, rx: re.Pattern):
def sub(s: str) -> str:
def rep(m: re.Match) -> str:
name = m.group(1)
if name not in self.users:
self.users[name] = f"<USER_{len(self.users) + 1}>"
self.counts["user_bare"] = self.counts.get("user_bare", 0) + 1
return self.users[name]
return rx.sub(rep, s)
if isinstance(o, str):
return sub(o)
if isinstance(o, list):
return [self.bare_pass(x, rx) for x in o]
if isinstance(o, dict):
return {k: self.bare_pass(v, rx) for k, v in o.items()}
return o
def local_usernames() -> set[str]:
"""本机可能出现的用户名:当前用户 + /home 下的目录名。用于裸用户名替换的种子。"""
out = set()
for k in ("USER", "LOGNAME"):
v = os.environ.get(k)
if v:
out.add(v)
try:
out |= {p.name for p in Path("/home").iterdir() if p.is_dir()}
except OSError:
pass
return {n for n in out if len(n) >= 3}
def redact_messages(messages: list[dict]) -> tuple[list[dict], dict]:
r = Redactor()
out = r.obj(messages)
rx = r.bare_user_regex(local_usernames())
if rx is not None:
out = r.bare_pass(out, rx)
return out, {"counts": r.counts, "users": len(r.users)}
if __name__ == "__main__":
for p in sys.argv[1:]:
d = json.loads(Path(p).read_text())
out, stats = redact_messages(d if isinstance(d, list) else [d])
print(json.dumps({"file": p, **stats}, ensure_ascii=False))

79
export/reward.py Normal file
View File

@ -0,0 +1,79 @@
"""reward v0 配方。离线算,只读 verdict.json;配方版本化,可全量重打分。
r = 1.0·[G1] + 1.0·[G2] + 2.0·[G3] 绝对门
r += 1.0·clip((Δ_agent − Δ_oracle)/|tol|, −1, +1) 相对 oracle 的精度
r += 0.5·clip((lat_oracle − lat_agent)/lat_oracle, −1, +1) 相对 oracle 的延时
r −= 0.5·turns/max_turns + 0.5·tokens/max_tokens 代价
r −= 2.0·[claim ≠ verdict] 说谎
r −= 0.25·[claim 缺席]
BLOCKED: +1.0 若 oracle 同阶段同类失败;否则 −2.0
TIMEOUT: −1.0
G3 = na:其 2.0 权重按比例分给 G1/G2(各 2.0),精度相对项取 0
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
RECIPE_VERSION = "v0.1"
PASSING = {"pass", "pass-screen", "pass-weak"}
W = {"G1": 1.0, "G2": 1.0, "G3": 2.0, "acc_rel": 1.0, "lat_rel": 0.5, "cost_turns": 0.5, "cost_tokens": 0.5,
"lie": 2.0, "claim_absent": 0.25, "blocked_ok": 1.0, "blocked_bad": 2.0, "timeout": 1.0}
def clip(x: float, lo: float = -1.0, hi: float = 1.0) -> float:
return max(lo, min(hi, x))
def compute(verdict: dict, max_turns: int = 40, max_tokens: int = 200000) -> dict:
terms: dict[str, float] = {}
g = verdict.get("gates") or {}
terminal = verdict.get("terminal")
claim = verdict.get("claim") or {}
oref = verdict.get("oracle_ref") or {}
cost = verdict.get("cost") or {}
if terminal == "TIMEOUT":
terms["timeout"] = -W["timeout"]
elif terminal == "BLOCKED":
terms["blocked"] = W["blocked_ok"] if verdict.get("blocked_correct") else -W["blocked_bad"]
else:
g3_na = (g.get("G3") or {}).get("verdict") == "na"
w1, w2, w3 = (2.0, 2.0, 0.0) if g3_na else (W["G1"], W["G2"], W["G3"])
terms["G1"] = w1 * float((g.get("G1") or {}).get("verdict") in PASSING)
terms["G2"] = w2 * float((g.get("G2") or {}).get("verdict") in PASSING)
terms["G3"] = w3 * float((g.get("G3") or {}).get("verdict") in PASSING)
# 相对 oracle 的精度(都用同一 kind 的 delta;lower-is-better 已在 gates.g3 里翻转成"越大越好")
g3 = g.get("G3") or {}
if not g3_na and g3.get("delta") is not None and oref.get("G3_delta") is not None and g3.get("tolerance"):
terms["acc_rel"] = W["acc_rel"] * clip((g3["delta"] - oref["G3_delta"]) / abs(g3["tolerance"]))
else:
terms["acc_rel"] = 0.0
g4 = g.get("G4") or {}
if g4.get("latency_ms_mean") and oref.get("latency_ms_mean") and not g4.get("contended"):
terms["lat_rel"] = W["lat_rel"] * clip((oref["latency_ms_mean"] - g4["latency_ms_mean"]) / oref["latency_ms_mean"])
else:
terms["lat_rel"] = 0.0
turns, tokens = cost.get("turns") or 0, cost.get("tokens") or 0
if cost.get("tokens_reported") is False or tokens == 0:
# 端点不回 usage(dr-gateway 实测全 0):代价项只按 turns 算,权重合并,别把"没报"当成"很省"
terms["cost"] = -((W["cost_turns"] + W["cost_tokens"]) * min(1.0, turns / max_turns))
terms_note = "tokens unavailable"
else:
terms["cost"] = -(W["cost_turns"] * min(1.0, turns / max_turns) + W["cost_tokens"] * min(1.0, tokens / max_tokens))
terms_note = None
if claim.get("terminal") == "ABSENT" or not claim:
terms["claim_absent"] = -W["claim_absent"]
elif verdict.get("claim_consistent") is False:
terms["lie"] = -W["lie"]
r = sum(terms.values())
return {"reward": round(r, 4), "terms": {k: round(v, 4) for k, v in terms.items()}, "recipe": RECIPE_VERSION,
"terminal": terminal, "review": bool(verdict.get("review")), "note": terms_note}
if __name__ == "__main__":
for p in sys.argv[1:]:
v = json.loads(Path(p).read_text())
print(p, json.dumps(compute(v), ensure_ascii=False))

259
export/to_agentrl.py Normal file
View File

@ -0,0 +1,259 @@
"""L4 数据物化:扫描 rollouts → 脱敏、清洗(R1–R8)、线性化 → 与 verdict / oracle 配对 → reward → datasets/<dataset_id>/。
python export/to_agentrl.py --runs r20260914-01 [r…] [--policy teacher] [--include-timeout]
清洗规则(移植自 data_collect/clean_for_agentrl.py 的经验,独立实现):
R1 双方言归一:OpenAI / Anthropic 形状 → {role, content, tool_calls?, tool_call_id?}
R2 剥离 thinking / reasoning 块
R3 system 只在首条且只有一条(多条合并)
R4 截断悬空尾轮:结尾的 assistant tool_call 没有对应 tool 结果 → 连同该 assistant 一起去掉
R5 去空消息
R6 合并相邻同 role 的纯文本消息
R7 tool 结果截断(头 6k + 尾 2k 字符)
R8 丢弃:无 assistant 消息,或 turns < 2
轨迹源:events.jsonl 最后一条 agent_end.messages[](已线性化);session.jsonl 是树,只做备份。
"""
from __future__ import annotations
import argparse
import hashlib
import json
import sys
import time
from pathlib import Path
HERE = Path(__file__).resolve().parent
BENCH = HERE.parent
sys.path.insert(0, str(HERE))
from redact import redact_messages # noqa: E402
from reward import compute as compute_reward, RECIPE_VERSION # noqa: E402
TOOL_HEAD, TOOL_TAIL = 6000, 2000
# ---------------------------------------------------------------- 轨迹提取
def system_prompt(rd: Path) -> tuple[str, str]:
"""取这条轨迹的系统提示 → (text, source)。
pi 的 --append-system-prompt 内容**不出现在** agent_end.messages[] 里(实测 0.73.1):
导出时若不补回来,SFT 样本就只有 user 任务 + assistant 动作,没有"只能写 workspace / 必须 claim"这些规则,
等于训练一个看不见规则的策略。优先读 rollout 自己的快照,其次按 cmd.json 里的路径读当前文件(会漂移,故记 source)。
"""
snap = rd / "agent" / "system.md"
if snap.exists():
return snap.read_text(errors="replace"), "snapshot"
cmd = rd / "agent" / "cmd.json"
if cmd.exists():
try:
argv = json.loads(cmd.read_text())["cmd"]
i = argv.index("--append-system-prompt")
f = Path(argv[i + 1])
if f.exists():
return f.read_text(errors="replace"), "live-file"
except (ValueError, KeyError, IndexError, OSError):
pass
return "", "missing"
def load_messages(rd: Path) -> list[dict] | None:
ev = rd / "agent" / "events.jsonl"
if not ev.exists():
return None
last = None
for line in ev.read_text(errors="replace").splitlines():
try:
e = json.loads(line)
except json.JSONDecodeError:
continue
if e.get("type") == "agent_end" and isinstance(e.get("messages"), list):
last = e["messages"]
return last
# ---------------------------------------------------------------- R1–R8
def _text_of(content) -> str:
if isinstance(content, str):
return content
if isinstance(content, list):
parts = []
for c in content:
if isinstance(c, dict):
t = c.get("type")
if t in ("text", "output_text") and c.get("text"):
parts.append(c["text"])
elif t in ("thinking", "reasoning", "redacted_thinking"):
continue # R2
elif isinstance(c, str):
parts.append(c)
return "\n".join(parts)
return "" if content is None else str(content)
def r1_normalize(m: dict) -> list[dict]:
"""一条原始消息 → 0..n 条归一消息(Anthropic 的 tool_use/tool_result 混在 content 里要拆)。"""
role = m.get("role")
content = m.get("content")
out: list[dict] = []
if role in ("toolResult", "tool_result", "tool"):
return [{"role": "tool", "tool_call_id": m.get("toolCallId") or m.get("tool_call_id"),
"content": _text_of(content) or json.dumps(m.get("output") or m.get("result") or "", ensure_ascii=False)}]
if role == "assistant":
tool_calls = []
if isinstance(content, list):
for c in content:
if isinstance(c, dict) and c.get("type") in ("toolCall", "tool_use", "function_call"):
tool_calls.append({"id": c.get("id"), "name": c.get("name"),
"arguments": c.get("arguments") if "arguments" in c else c.get("input")})
for tc in m.get("tool_calls") or []:
fn = tc.get("function") or {}
tool_calls.append({"id": tc.get("id"), "name": fn.get("name") or tc.get("name"),
"arguments": fn.get("arguments") if fn else tc.get("arguments")})
msg = {"role": "assistant", "content": _text_of(content)}
if tool_calls:
msg["tool_calls"] = tool_calls
return [msg]
if role == "custom":
# pi 扩展塞进会话的消息(如 budget-kill 的预算提醒):模型是当 user 输入看到的,训练样本也要有
t = _text_of(content)
return [{"role": "user", "content": f"[{m.get('customType', 'extension')}] {t}"}] if t else []
if role in ("user", "system"):
if isinstance(content, list) and any(isinstance(c, dict) and c.get("type") == "tool_result" for c in content):
for c in content:
if isinstance(c, dict) and c.get("type") == "tool_result":
out.append({"role": "tool", "tool_call_id": c.get("tool_use_id"), "content": _text_of(c.get("content"))})
else:
t = _text_of([c])
if t:
out.append({"role": "user", "content": t})
return out
return [{"role": role, "content": _text_of(content)}]
return []
def clean(messages: list[dict]) -> tuple[list[dict], dict]:
stats = {"in": len(messages)}
msgs: list[dict] = []
for m in messages:
msgs.extend(r1_normalize(m))
# R3 system 合并到首条
systems = [m["content"] for m in msgs if m["role"] == "system"]
msgs = [m for m in msgs if m["role"] != "system"]
if systems:
msgs.insert(0, {"role": "system", "content": "\n\n".join(s for s in systems if s)})
# R5 去空
msgs = [m for m in msgs if m.get("content") or m.get("tool_calls")]
# R7 tool 结果截断
for m in msgs:
if m["role"] == "tool" and len(m["content"]) > TOOL_HEAD + TOOL_TAIL:
m["content"] = m["content"][:TOOL_HEAD] + f"\n…[truncated {len(m['content']) - TOOL_HEAD - TOOL_TAIL} chars]…\n" + m["content"][-TOOL_TAIL:]
stats["r7_truncated"] = stats.get("r7_truncated", 0) + 1
# R6 合并相邻同 role 纯文本
merged: list[dict] = []
for m in msgs:
if merged and m["role"] == merged[-1]["role"] and m["role"] in ("user", "assistant") \
and not m.get("tool_calls") and not merged[-1].get("tool_calls"):
merged[-1]["content"] += "\n" + m["content"]
stats["r6_merged"] = stats.get("r6_merged", 0) + 1
else:
merged.append(m)
msgs = merged
# R4 悬空尾轮
while msgs and msgs[-1]["role"] == "assistant" and msgs[-1].get("tool_calls"):
msgs.pop(); stats["r4_dangling"] = stats.get("r4_dangling", 0) + 1
while msgs and msgs[-1]["role"] in ("user", "tool") and len(msgs) > 2:
msgs.pop(); stats["r4_trailing_user"] = stats.get("r4_trailing_user", 0) + 1
# R8
n_assist = sum(m["role"] == "assistant" for m in msgs)
stats.update({"out": len(msgs), "assistant_turns": n_assist, "drop": n_assist == 0 or len(msgs) < 3})
return msgs, stats
# ---------------------------------------------------------------- 主流程
def scan(run_ids: list[str]) -> list[Path]:
out = []
for rid in run_ids:
for v in (BENCH / "runs" / rid).glob("*/rollouts/*/verdict.json"):
out.append(v.parent)
return sorted(out)
def next_dataset_id() -> str:
d = time.strftime("%Y%m%d")
existing = sorted((BENCH / "datasets").glob(f"ds{d}-*"))
return f"ds{d}-{len(existing) + 1:02d}"
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--runs", nargs="+", required=True)
ap.add_argument("--policy", choices=["teacher", "student"])
ap.add_argument("--include-timeout", action="store_true")
ap.add_argument("--max-turns", type=int, default=40)
ap.add_argument("--max-tokens", type=int, default=200000)
ap.add_argument("--dataset-id")
a = ap.parse_args()
ds_id = a.dataset_id or next_dataset_id()
out_dir = BENCH / "datasets" / ds_id
out_dir.mkdir(parents=True, exist_ok=True)
files = {"teacher": open(out_dir / "teacher.jsonl", "w"), "student": open(out_dir / "student.jsonl", "w")}
stats = {"scanned": 0, "written": 0, "dropped": {}, "review": 0, "by_terminal": {}}
for rd in scan(a.runs):
stats["scanned"] += 1
verdict = json.loads((rd / "verdict.json").read_text())
rj = json.loads((rd / "rollout.json").read_text()) if (rd / "rollout.json").exists() else {}
policy = verdict.get("policy") or rj.get("policy") or "teacher"
if a.policy and policy != a.policy:
continue
if verdict["terminal"] == "TIMEOUT" and not a.include_timeout:
stats["dropped"]["timeout"] = stats["dropped"].get("timeout", 0) + 1; continue
raw = load_messages(rd)
if not raw:
stats["dropped"]["no_trajectory"] = stats["dropped"].get("no_trajectory", 0) + 1; continue
msgs, cst = clean(raw)
sys_text, sys_src = system_prompt(rd)
if sys_text:
if msgs and msgs[0]["role"] == "system":
msgs[0]["content"] = sys_text + "\n\n" + msgs[0]["content"]
else:
msgs.insert(0, {"role": "system", "content": sys_text})
cst["system_source"] = sys_src
if cst["drop"]:
stats["dropped"]["r8"] = stats["dropped"].get("r8", 0) + 1; continue
msgs, rst = redact_messages(msgs)
rw = compute_reward(verdict, a.max_turns, a.max_tokens)
jd = rd.parent.parent
job = json.loads((jd / "job.json").read_text())
rec = {
"id": hashlib.sha1(f"{verdict['run_id']}/{verdict['job_id']}/{verdict['rollout']}".encode()).hexdigest()[:16],
"run_id": verdict["run_id"], "job_id": verdict["job_id"], "rollout": verdict["rollout"],
"mode": verdict.get("mode") or rj.get("mode"), "policy": policy, "model": rj.get("model"),
"task": job["task"], "march": job["march"], "backend": job["backend"],
"messages": msgs,
"verdict": {k: verdict[k] for k in ("terminal", "failed_gate", "claim_consistent", "blocked_correct", "review") if k in verdict},
"gates": {g: (verdict["gates"].get(g) or {}).get("verdict") for g in ("G1", "G2", "G3", "G4")},
"oracle_ref": verdict.get("oracle_ref"),
"reward": rw["reward"], "reward_terms": rw["terms"], "reward_recipe": rw["recipe"],
"cost": verdict.get("cost"),
"system_source": sys_src,
"system_sha1": hashlib.sha1(sys_text.encode()).hexdigest()[:12] if sys_text else None,
"clean_stats": cst, "redact_stats": rst,
}
files[policy].write(json.dumps(rec, ensure_ascii=False) + "\n")
stats["written"] += 1
stats["review"] += int(bool(verdict.get("review")))
stats["by_terminal"][verdict["terminal"]] = stats["by_terminal"].get(verdict["terminal"], 0) + 1
for f in files.values():
f.close()
manifest = {"dataset_id": ds_id, "runs": a.runs, "reward_recipe": RECIPE_VERSION, "created_at": time.time(),
"filters": {"policy": a.policy, "include_timeout": a.include_timeout}, "stats": stats}
(out_dir / "manifest.json").write_text(json.dumps(manifest, indent=2, ensure_ascii=False))
print(json.dumps(manifest, ensure_ascii=False))
return 0
if __name__ == "__main__":
sys.exit(main())

17
harness/__init__.py Normal file
View File

@ -0,0 +1,17 @@
"""bench harness — L1 oracle 流水线 + L3 verdict。
模块:
paths 根目录、config.yaml、.env.local
registry registry/* 加载、matrix 展开、job_id、job.json 快照
db state/jobs.sqlite 唯一真相
errors stderr → error_class
watchdog 无固定超时的子进程执行器
gates G1–G4 → verdict
stages 01–08,每阶段一个子目录落 stdout/stderr/exit/meta
backends ptq / qat:按 backend 选 venv,agent 无权改
scheduler 资源池、板卡锁、耗时估算
pipeline oracle 主流程 + preflight
stagerun agent 可用的 `bench stage run / wait`
cli `bench` 命令
"""
__version__ = "0.4.0"

View File

@ -0,0 +1,2 @@
"""custom 模型的构建适配器:每个模型一个 <model_id>.py,提供 build(weights: Path, input_spec: str) -> nn.Module。
非 torchvision / ultralytics / timm 的模型走这里。"""

View File

@ -0,0 +1,12 @@
"""按 backend 选 venv 与命令模板。harness 的职责;agent 无权改。"""
from __future__ import annotations
from . import ptq, qat
BACKENDS = {"ptq": ptq, "qat": qat}
def get(backend: str):
if backend not in BACKENDS:
raise KeyError(f"未知 backend: {backend}")
return BACKENDS[backend]

111
harness/backends/ptq.py Normal file
View File

@ -0,0 +1,111 @@
"""PTQ 后端:~/toolchains/oe-3.7.0/venv(hbdk4 4.7.5 · horizon_tc_ui 3.5.3 · hmct 2.6.5)。
流程:ONNX → hb_compile(含校准)→ .hbm → hb_model_info / hb_verifier。
字段名对照 OE 3.7 horizon_tc_ui 文档;P2 切片时逐项核对,核对完把 VERIFIED 改 True。
"""
from __future__ import annotations
from pathlib import Path
import yaml
from .. import paths
NAME = "ptq"
VERIFIED = False # P2:hb_compile yaml 字段与工具名核对后置 True
TOOLS = {
"compile": "hb_compile",
"model_info": "hb_model_info",
"verifier": "hb_verifier",
"perf": "hb_perf",
}
def python() -> Path:
return paths.venv_python(NAME)
def tool(name: str) -> str:
return str(paths.venv_bin(NAME, TOOLS[name]))
def export_cmd(job: dict, onnx_out: Path, weights: Path) -> list[str]:
return [str(paths.venv_python("bench")), "-m", "harness.tools.export_onnx", # 导 ONNX 用 bench venv(torch+torchvision+ultralytics)
"--model", job["model"], "--source", job["model_spec"]["source"],
"--weights", str(weights), "--input-spec", job["model_spec"]["input_spec"],
"--out", str(onnx_out)]
def compile_config(job: dict, onnx: Path, calib_dir: Path, work_dir: Path, agent_overrides: dict | None = None) -> dict:
"""hb_compile 的 yaml。agent 在 rollout 里可以提交自己的 yaml(bench stage run --config),
但 march / working_dir / output prefix 由 harness 钉死,覆盖不了。"""
prefix = job["job_id"]
cfg = {
"model_parameters": {
"onnx_model": str(onnx),
"march": job["march"],
"working_dir": str(work_dir),
"output_model_file_prefix": prefix,
"layer_out_dump": False,
},
"input_parameters": {
# 实测 hb_compile 3.5.3:norm_type / cal_data_type 已弃用(由 mean/scale 与 npy 决定);featuremap 输入让预处理留在主机侧,口径与评测器一致
"input_type_rt": "featuremap",
"input_type_train": "featuremap",
"input_layout_train": "NCHW",
},
"calibration_parameters": {
"cal_data_dir": str(calib_dir), # 只含 .npy 的目录,后缀 _f32(OE 约定)
"calibration_type": "default",
},
"compiler_parameters": {
"compile_mode": "latency",
"optimize_level": "O2",
"debug": False,
"jobs": 8,
},
}
if job["qcfg"] == "fp16int8":
# hmct quant_config(schema:calibration_parameters.quant_config: str|dict)。实测 3.5.3 不接受 model_parameters.quant_config。
# 定义 fp16int8 = 主体 int8 + 敏感算子与模型输出 float16(nash-p 有 fp16 算力)。qtype 可选:int8/int16/float16/…
cfg["calibration_parameters"]["quant_config"] = {
"model_config": {"all_node_type": "int8", "model_output_type": "float16"},
"op_config": {op: {"qtype": "float16"} for op in ("Softmax", "LayerNormalization", "Gelu", "Erf", "Exp", "Log", "Div", "Pow")},
}
if agent_overrides:
# 允许 agent 改:input_parameters / calibration_parameters / compiler_parameters / quant_config
for sect in ("input_parameters", "calibration_parameters", "compiler_parameters"):
cfg[sect].update(agent_overrides.get(sect) or {})
if agent_overrides.get("calibration_parameters", {}).get("quant_config") is not None:
cfg["calibration_parameters"]["quant_config"] = agent_overrides["calibration_parameters"]["quant_config"]
# 钉死项
cfg["model_parameters"]["march"] = job["march"]
cfg["model_parameters"]["working_dir"] = str(work_dir)
cfg["model_parameters"]["output_model_file_prefix"] = prefix
cfg["model_parameters"]["onnx_model"] = str(onnx)
return cfg
def write_compile_config(cfg: dict, path: Path) -> Path:
path.write_text(yaml.safe_dump(cfg, allow_unicode=True, sort_keys=False))
return path
def compile_cmd(yaml_path: Path) -> list[str]:
return [tool("compile"), "-c", str(yaml_path)]
def model_info_cmd(hbm: Path) -> list[str]:
return [tool("model_info"), str(hbm)]
def verifier_cmd(ptq_onnx: Path, bc: Path, sample_npy: Path) -> list[str]:
"""hb_verifier 3.5.3:-m 逗号分隔的 onnx/bc/hbm,-i 输入文件。hbm 需要板 ip,x86 侧比 ptq_model.onnx vs quantized_model.bc。
输出逐层余弦表,rc=0;G1 取最后一行(模型输出)的余弦。"""
return [tool("verifier"), "-m", f"{ptq_onnx},{bc}", "-i", str(sample_npy)]
def infer_cmd(hbm: Path, inputs_npz: Path, out_npz: Path) -> list[str]:
"""x86 侧用 hbm 推理拿输出张量(G2 与 G3 用)。P2 确认 PTQ venv 里是 hbm_infer 还是 hb_model_infer。"""
return [str(python()), "-m", "harness.tools.infer_hbm", "--hbm", str(hbm), "--inputs", str(inputs_npz), "--out", str(out_npz)]

52
harness/backends/qat.py Normal file
View File

@ -0,0 +1,52 @@
"""QAT 后端:~/toolchains/oellm2/venv(hbdk4 4.12.3a2 · horizon_plugin_pytorch 3.3.5 · hbm_infer 3.15.3)。
第一轮只做 calibration-only(plugin 的 calib 模式),不做训练。P4 之后再接。
流程:torch 模型 → plugin prepare(calib) → export hbir → hbdk4 compile → .hbm。
"""
from __future__ import annotations
from pathlib import Path
from .. import paths
NAME = "qat"
VERIFIED = False
TOOLS = {
"model_info": "hb_model_info", # P4 确认 hbdk4 4.12 里的工具名
"verifier": "hb_verifier",
}
def python() -> Path:
return paths.venv_python(NAME)
def tool(name: str) -> str:
return str(paths.venv_bin(NAME, TOOLS[name]))
def export_cmd(job: dict, onnx_out: Path, weights: Path) -> list[str]:
# QAT 路径不需要 ONNX,但 03-export 阶段统一产出一份 ONNX 供 FP32 基线对齐
return [str(paths.venv_python("bench")), "-m", "harness.tools.export_onnx", # 导 ONNX 用 bench venv(torch+torchvision+ultralytics)
"--model", job["model"], "--source", job["model_spec"]["source"],
"--weights", str(weights), "--input-spec", job["model_spec"]["input_spec"],
"--out", str(onnx_out)]
def calib_compile_cmd(job: dict, weights: Path, calib_dir: Path, work_dir: Path, agent_config: Path | None) -> list[str]:
cmd = [str(python()), "-m", "harness.tools.qat_calib_compile",
"--model", job["model"], "--weights", str(weights), "--march", job["march"],
"--calib", str(calib_dir), "--out", str(work_dir / f"{job['job_id']}.hbm"),
"--input-spec", job["model_spec"]["input_spec"]]
if agent_config:
cmd += ["--config", str(agent_config)]
return cmd
def model_info_cmd(hbm: Path) -> list[str]:
return [tool("model_info"), str(hbm)]
def infer_cmd(hbm: Path, inputs_npz: Path, out_npz: Path) -> list[str]:
return [str(python()), "-m", "harness.tools.infer_hbm", "--hbm", str(hbm), "--inputs", str(inputs_npz), "--out", str(out_npz)]

258
harness/cli.py Normal file
View File

@ -0,0 +1,258 @@
"""`bench` 命令。
bench preflight [--modes oracle,e2e]
bench new-run [--note ...]
bench expand [--run r…] [--modes …]
bench run [--run r…] [--job id …] [--dry-run] [--parallel N]
bench status [--run r…] [--fail]
bench show <job_id> [--run r…]
bench annotate <job_id> --note "…"
bench report [--run r…]
bench verify <rollout_dir>
bench stage run <stage> [--config path] (rollout 内,agent 用)
bench stage wait <handle> [--poll 60]
bench stage status <handle>
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
from . import paths, db
def _run_id(con, given: str | None) -> str:
rid = given or db.latest_run_id(con)
if not rid:
raise SystemExit("没有 run:先 bench new-run")
return rid
def cmd_preflight(a) -> int:
from .pipeline import preflight
errs, warns = preflight(a.modes.split(",") if a.modes else None, oracle_only=a.oracle_only)
for w in warns:
print(f"WARN {w}")
for e in errs:
print(f"ERROR {e}")
print(f"preflight: {len(errs)} errors, {len(warns)} warnings")
return 1 if errs else 0
def cmd_new_run(a) -> int:
from .pipeline import new_run
print(new_run(a.note or ""))
return 0
def cmd_expand(a) -> int:
from .pipeline import expand
with db.connect() as con:
rid = _run_id(con, a.run)
print(json.dumps({"run_id": rid, **expand(rid, a.modes.split(",") if a.modes else None, a.marches.split(",") if a.marches else None)}))
return 0
def cmd_run(a) -> int:
from .pipeline import run_oracle
with db.connect() as con:
rid = _run_id(con, a.run)
s = run_oracle(rid, a.job or None, dry_run=a.dry_run, parallel=a.parallel)
print(json.dumps({"run_id": rid, **s}))
return 0
def cmd_status(a) -> int:
with db.connect() as con:
rid = _run_id(con, a.run)
rows = db.status_counts(con, rid)
print(f"run {rid}")
if a.fail:
for r in db.error_class_counts(con, rid):
print(f" {r['n']:4d} {r['error_class']:<20} {r['stage_failed'] or ''}")
for j in db.list_jobs(con, rid, state="FAIL"):
print(f" FAIL {j['job_id']:<50} {j['error_class'] or '':<18} {j['stage_failed'] or ''}")
return 0
agg: dict = {}
for r in rows:
agg.setdefault(r["state"], {}).setdefault(r["task"], {})[r["march"]] = r["n"]
for st, tasks in agg.items():
total = sum(sum(m.values()) for m in tasks.values())
print(f" {st:<15} {total:5d}")
for t, m in tasks.items():
print(f" {t:<8} " + " ".join(f"{k}={v}" for k, v in m.items()))
ro = con.execute("SELECT state, COUNT(*) n FROM rollouts WHERE run_id=? GROUP BY state", (rid,)).fetchall()
if ro:
print(" rollouts: " + " ".join(f"{r['state']}={r['n']}" for r in ro))
from .scheduler import pool_usage, pool_size
from .registry import status_counts
print(" pools: " + " ".join(f"{p}={n}/{pool_size(p)}" for p, n in pool_usage().items()))
print(" registry: " + " ".join(f"{k}={v}" for k, v in status_counts().items()) + " (只有 ready 会被展开)")
return 0
def cmd_show(a) -> int:
with db.connect() as con:
rid = _run_id(con, a.run)
j = db.get_job(con, rid, a.job_id)
if not j:
raise SystemExit(f"{rid} 里没有 {a.job_id}")
print(f"{a.job_id} [{rid}] state={j['state']} ec={j['error_class']} stage={j['stage_failed']} est={j['estimate_s']}")
st = con.execute("SELECT stage, attempt, lane, state, exit_code, error_class, duration_s FROM stages"
" WHERE run_id=? AND job_id=? AND rollout=-1 ORDER BY stage, attempt", (rid, a.job_id)).fetchall()
for s in st:
print(f" {s['stage']:<14} a{s['attempt']:02d} {s['lane']:<6} {s['state']:<8} exit={s['exit_code']!s:<4} {s['error_class'] or '':<18} {s['duration_s'] or 0:8.1f}s")
jd = paths.RUNS / rid / a.job_id
if j["stage_failed"]:
for p in sorted((jd / "oracle" / "stages" / j["stage_failed"]).glob("a*/stderr.log")):
print(f"--- {p} (tail 50) ---")
print("\n".join(p.read_text(errors="replace").splitlines()[-50:]))
rs = db.list_rollouts(con, rid, a.job_id)
if rs:
print("rollouts:")
for r in rs:
print(f" {r['kk']:02d} {r['mode']:<6} {r['policy'] or '':<8} {r['state']:<10} claim={r['claim_terminal']} consistent={r['claim_consistent']} review={r['review']} turns={r['turns']} tokens={r['tokens']}")
notes = con.execute("SELECT ts, note FROM annotations WHERE run_id=? AND job_id=? ORDER BY ts", (rid, a.job_id)).fetchall()
for n in notes:
print(f" note: {n['note']}")
return 0
def cmd_annotate(a) -> int:
with db.connect() as con:
rid = _run_id(con, a.run)
db.annotate(con, rid, a.job_id, a.note)
print("ok")
return 0
def cmd_report(a) -> int:
from .report import render
with db.connect() as con:
rid = _run_id(con, a.run)
print(render(rid))
return 0
def cmd_watch(a) -> int:
from .watch import watch
with db.connect() as con:
rid = _run_id(con, a.run)
return watch(rid, a.interval, a.once)
def cmd_reset_stale(a) -> int:
from .pipeline import reset_stale
with db.connect() as con:
rid = _run_id(con, a.run)
print(json.dumps(reset_stale(rid, a.force)))
return 0
def cmd_board_backfill(a) -> int:
from .pipeline import board_backfill
with db.connect() as con:
rid = _run_id(con, a.run)
print(json.dumps(board_backfill(rid, a.job or None), ensure_ascii=False))
return 0
def cmd_rejudge(a) -> int:
from .pipeline import rejudge
with db.connect() as con:
rid = _run_id(con, a.run)
ids = [a.job_id] if a.job_id not in ("all", "--all") else [j["job_id"] for j in db.list_jobs(con, rid, mode="oracle") if j["state"] in ("PASS", "FAIL", "UNVERIFIED")]
from collections import Counter
c = Counter()
for jid in ids:
r = rejudge(rid, jid)
c[(r["terminal"], r.get("failed_gate"))] += 1
if a.job_id not in ("all", "--all"):
print(json.dumps(r, ensure_ascii=False))
if a.job_id in ("all", "--all"):
print(json.dumps({"rejudged": len(ids), "by_terminal_gate": {f"{k[0]}/{k[1]}": v for k, v in c.items()}}, ensure_ascii=False))
return 0
def cmd_verify(a) -> int:
from .verify import verify_rollout
v = verify_rollout(Path(a.rollout_dir))
print(json.dumps({k: v[k] for k in ("job_id", "rollout", "terminal", "failed_gate", "claim_consistent", "review")}))
return 0
def cmd_collect(a) -> int:
from .pipeline import collect_rollouts
with db.connect() as con:
rid = _run_id(con, a.run)
s = collect_rollouts(rid, a.job or None, a.modes.split(",") if a.modes else None,
parallel=a.parallel, dry_run=a.dry_run, limit=a.limit)
print(json.dumps({"run_id": rid, **s}, ensure_ascii=False))
return 0
def cmd_rollout(a) -> int:
from .rollout import run_rollout
with db.connect() as con:
rid = _run_id(con, a.run)
print(json.dumps(run_rollout(rid, a.job_id, a.kk, a.mode, a.policy, dry_run=a.dry_run), ensure_ascii=False))
return 0
def cmd_agent(a) -> int:
from .rollout import setup_models_json
if a.op == "setup":
print(setup_models_json(force=a.force))
return 0
raise SystemExit("agent op: setup")
def cmd_stage(a) -> int:
from . import stagerun
if a.op == "run":
stagerun.submit(a.stage, a.config)
return 0
if a.op == "wait":
return stagerun.wait(a.stage, a.poll)
if a.op == "status":
return stagerun.status(a.stage)
raise SystemExit("stage op: run | wait | status")
def main(argv=None) -> int:
p = argparse.ArgumentParser(prog="bench")
sp = p.add_subparsers(dest="cmd", required=True)
x = sp.add_parser("preflight"); x.add_argument("--modes"); x.add_argument("--oracle-only", action="store_true", help="只跑 benchmark 那一半时放行 agent 运行时缺失"); x.set_defaults(f=cmd_preflight)
x = sp.add_parser("new-run"); x.add_argument("--note"); x.set_defaults(f=cmd_new_run)
x = sp.add_parser("expand"); x.add_argument("--run"); x.add_argument("--modes"); x.add_argument("--marches", help="只展开这些 march,逗号分隔"); x.set_defaults(f=cmd_expand)
x = sp.add_parser("run"); x.add_argument("--run"); x.add_argument("--job", action="append")
x.add_argument("--dry-run", action="store_true"); x.add_argument("--parallel", type=int); x.set_defaults(f=cmd_run)
x = sp.add_parser("status"); x.add_argument("--run"); x.add_argument("--fail", action="store_true"); x.set_defaults(f=cmd_status)
x = sp.add_parser("show"); x.add_argument("job_id"); x.add_argument("--run"); x.set_defaults(f=cmd_show)
x = sp.add_parser("annotate"); x.add_argument("job_id"); x.add_argument("--note", required=True); x.add_argument("--run"); x.set_defaults(f=cmd_annotate)
x = sp.add_parser("report"); x.add_argument("--run"); x.set_defaults(f=cmd_report)
x = sp.add_parser("verify"); x.add_argument("rollout_dir"); x.set_defaults(f=cmd_verify)
x = sp.add_parser("rejudge"); x.add_argument("job_id", help="job_id 或 all(该 run 所有已结束的 job)"); x.add_argument("--run"); x.set_defaults(f=cmd_rejudge)
x = sp.add_parser("board-backfill"); x.add_argument("--run"); x.add_argument("--job", action="append"); x.set_defaults(f=cmd_board_backfill)
x = sp.add_parser("watch"); x.add_argument("--run"); x.add_argument("--interval", type=int, default=30); x.add_argument("--once", action="store_true"); x.set_defaults(f=cmd_watch)
x = sp.add_parser("reset-stale"); x.add_argument("--run"); x.add_argument("--force", action="store_true"); x.set_defaults(f=cmd_reset_stale)
x = sp.add_parser("collect"); x.add_argument("--run"); x.add_argument("--job", action="append")
x.add_argument("--modes", help="e2e,repair"); x.add_argument("--parallel", type=int)
x.add_argument("--limit", type=int, help="先采 N 条试水"); x.add_argument("--dry-run", action="store_true")
x.set_defaults(f=cmd_collect)
x = sp.add_parser("rollout"); x.add_argument("job_id"); x.add_argument("--kk", type=int, default=0)
x.add_argument("--mode", choices=["e2e", "repair"], default="e2e"); x.add_argument("--policy", choices=["teacher", "student"], default="teacher")
x.add_argument("--run"); x.add_argument("--dry-run", action="store_true"); x.set_defaults(f=cmd_rollout)
x = sp.add_parser("agent"); x.add_argument("op", choices=["setup"]); x.add_argument("--force", action="store_true"); x.set_defaults(f=cmd_agent)
x = sp.add_parser("stage"); x.add_argument("op", choices=["run", "wait", "status"]); x.add_argument("stage")
x.add_argument("--config"); x.add_argument("--poll", type=int, default=60); x.set_defaults(f=cmd_stage)
a = p.parse_args(argv)
return a.f(a)
if __name__ == "__main__":
sys.exit(main())

238
harness/db.py Normal file
View File

@ -0,0 +1,238 @@
"""state/jobs.sqlite — 状态、error_class、耗时历史的唯一真相。文件系统只存内容。"""
from __future__ import annotations
import json
import os
import sqlite3
import time
from contextlib import contextmanager
from pathlib import Path
from .paths import DB_PATH, STATE
SCHEMA = """
CREATE TABLE IF NOT EXISTS runs (
run_id TEXT PRIMARY KEY,
created_at REAL NOT NULL,
manifest TEXT NOT NULL, -- json:venv pip-freeze sha、板固件、registry 指纹、数据 manifest sha
note TEXT
);
CREATE TABLE IF NOT EXISTS jobs (
run_id TEXT NOT NULL,
job_id TEXT NOT NULL,
task TEXT, model TEXT, march TEXT, backend TEXT, qcfg TEXT, priority TEXT,
mode TEXT NOT NULL, -- oracle | e2e | repair
policy TEXT, -- teacher | student | NULL
k INTEGER DEFAULT 1,
state TEXT NOT NULL, -- PENDING ORACLE_RUNNING PASS FAIL UNVERIFIED
error_class TEXT,
stage_failed TEXT,
estimate_s REAL,
started_at REAL, finished_at REAL,
result TEXT, -- oracle/result.json 副本
note TEXT,
PRIMARY KEY (run_id, job_id, mode, policy)
);
CREATE TABLE IF NOT EXISTS stages (
run_id TEXT NOT NULL, job_id TEXT NOT NULL,
rollout INTEGER NOT NULL DEFAULT -1, -- -1 = oracle
stage TEXT NOT NULL,
attempt INTEGER NOT NULL DEFAULT 0,
lane TEXT DEFAULT 'normal', -- normal | long
state TEXT NOT NULL, -- RUNNING OK FAIL STALLED SKIPPED
exit_code INTEGER,
error_class TEXT,
started_at REAL, finished_at REAL, duration_s REAL,
PRIMARY KEY (run_id, job_id, rollout, stage, attempt)
);
CREATE TABLE IF NOT EXISTS rollouts (
run_id TEXT NOT NULL, job_id TEXT NOT NULL, kk INTEGER NOT NULL,
mode TEXT, policy TEXT,
state TEXT NOT NULL, -- QUEUED RUNNING VERIFYING PASS FAIL BLOCKED TIMEOUT
claim_terminal TEXT,
verdict_terminal TEXT,
claim_consistent INTEGER,
review INTEGER DEFAULT 0, -- oracle FAIL / rollout PASS → 1,进人工抽查
turns INTEGER, tokens INTEGER, wall_s REAL,
verdict TEXT,
started_at REAL, finished_at REAL,
PRIMARY KEY (run_id, job_id, kk)
);
CREATE TABLE IF NOT EXISTS annotations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
run_id TEXT, job_id TEXT, ts REAL, note TEXT
);
CREATE INDEX IF NOT EXISTS idx_jobs_state ON jobs(run_id, state);
CREATE INDEX IF NOT EXISTS idx_stages_hist ON stages(job_id, stage, state);
"""
def _open(path: Path) -> sqlite3.Connection:
con = sqlite3.connect(path, timeout=60)
con.row_factory = sqlite3.Row
con.execute("PRAGMA busy_timeout=60000")
return con
@contextmanager
def connect(path: Path = DB_PATH):
STATE.mkdir(parents=True, exist_ok=True)
con = _open(path)
# 默认 TRUNCATE:WAL 在 FUSE(JuiceFS 等)上会 disk I/O error(-shm 需要 mmap)。
# state/ 放本地盘(BENCH_STATE_DIR=/home/<user>/bench-state)时可设 BENCH_SQLITE_JOURNAL=WAL。
jm = os.environ.get("BENCH_SQLITE_JOURNAL", "TRUNCATE")
con.execute(f"PRAGMA journal_mode={jm}")
con.executescript(SCHEMA)
try:
yield con
con.commit()
finally:
con.close()
# ---------------------------------------------------------------- runs
def create_run(con, run_id: str, manifest: dict, note: str = "") -> None:
if con.execute("SELECT 1 FROM runs WHERE run_id=?", (run_id,)).fetchone():
raise RuntimeError(f"run_id 已存在(append-only,不允许覆盖): {run_id}")
con.execute("INSERT INTO runs VALUES (?,?,?,?)",
(run_id, time.time(), json.dumps(manifest, ensure_ascii=False), note))
def get_run(con, run_id: str) -> dict | None:
r = con.execute("SELECT * FROM runs WHERE run_id=?", (run_id,)).fetchone()
return dict(r) if r else None
def latest_run_id(con) -> str | None:
r = con.execute("SELECT run_id FROM runs ORDER BY created_at DESC LIMIT 1").fetchone()
return r[0] if r else None
def next_run_id(con, date_str: str) -> str:
rows = con.execute("SELECT run_id FROM runs WHERE run_id LIKE ?", (f"r{date_str}-%",)).fetchall()
seq = 1 + max([int(r[0].rsplit("-", 1)[1]) for r in rows] or [0])
return f"r{date_str}-{seq:02d}"
# ---------------------------------------------------------------- jobs
def upsert_job(con, run_id: str, spec_row: dict, estimate_s: float | None) -> bool:
"""展开幂等:已存在的 (run_id, job_id, mode, policy) 不动。返回是否新插入。"""
key = (run_id, spec_row["job_id"], spec_row["mode"], spec_row.get("policy"))
if con.execute("SELECT 1 FROM jobs WHERE run_id=? AND job_id=? AND mode=? AND policy IS ?", key).fetchone():
return False
con.execute(
"INSERT INTO jobs (run_id, job_id, task, model, march, backend, qcfg, priority, mode, policy, k, state, estimate_s)"
" VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)",
(run_id, spec_row["job_id"], spec_row["task"], spec_row["model"], spec_row["march"],
spec_row["backend"], spec_row["qcfg"], spec_row["priority"], spec_row["mode"],
spec_row.get("policy"), spec_row.get("k", 1), "PENDING", estimate_s))
return True
def claim_job(con, run_id: str, job_id: str, mode: str = "oracle", force: bool = False) -> bool:
"""原子认领:只有 PENDING 的 job 能被置成 ORACLE_RUNNING。两个 `bench run` 进程同时挑到同一个 job 时只有一个赢。
force=True 允许重跑已结束的 job(只给 --job 显式指定时用;结果仍是原地覆盖 result.json,慎用)。"""
allowed = "('PENDING')" if not force else "('PENDING','PASS','FAIL','UNVERIFIED')"
cur = con.execute(f"UPDATE jobs SET state='ORACLE_RUNNING', started_at=? WHERE run_id=? AND job_id=? AND mode=? AND state IN {allowed}",
(time.time(), run_id, job_id, mode))
con.commit()
return cur.rowcount == 1
def set_job_state(con, run_id: str, job_id: str, state: str, *, mode: str = "oracle",
error_class: str | None = None, stage_failed: str | None = None,
result: dict | None = None) -> None:
now = time.time()
if state == "ORACLE_RUNNING":
con.execute("UPDATE jobs SET state=?, started_at=? WHERE run_id=? AND job_id=? AND mode=?",
(state, now, run_id, job_id, mode))
else:
con.execute("UPDATE jobs SET state=?, finished_at=?, error_class=?, stage_failed=?, result=?"
" WHERE run_id=? AND job_id=? AND mode=?",
(state, now, error_class, stage_failed,
json.dumps(result, ensure_ascii=False) if result else None, run_id, job_id, mode))
def get_job(con, run_id: str, job_id: str, mode: str = "oracle") -> dict | None:
r = con.execute("SELECT * FROM jobs WHERE run_id=? AND job_id=? AND mode=?", (run_id, job_id, mode)).fetchone()
return dict(r) if r else None
def list_jobs(con, run_id: str, mode: str | None = None, state: str | None = None) -> list[dict]:
q, args = "SELECT * FROM jobs WHERE run_id=?", [run_id]
if mode:
q += " AND mode=?"; args.append(mode)
if state:
q += " AND state=?"; args.append(state)
q += " ORDER BY CASE priority WHEN 'A' THEN 0 WHEN 'B' THEN 1 ELSE 2 END, estimate_s, job_id"
return [dict(r) for r in con.execute(q, args)]
# ---------------------------------------------------------------- stages
def stage_start(con, run_id, job_id, stage, attempt=0, rollout=-1, lane="normal") -> None:
con.execute("INSERT OR REPLACE INTO stages (run_id, job_id, rollout, stage, attempt, lane, state, started_at)"
" VALUES (?,?,?,?,?,?,'RUNNING',?)", (run_id, job_id, rollout, stage, attempt, lane, time.time()))
def stage_finish(con, run_id, job_id, stage, attempt, state, exit_code=None, error_class=None, rollout=-1) -> None:
now = time.time()
r = con.execute("SELECT started_at FROM stages WHERE run_id=? AND job_id=? AND rollout=? AND stage=? AND attempt=?",
(run_id, job_id, rollout, stage, attempt)).fetchone()
started = r[0] if r and r[0] else now
con.execute("UPDATE stages SET state=?, exit_code=?, error_class=?, finished_at=?, duration_s=?"
" WHERE run_id=? AND job_id=? AND rollout=? AND stage=? AND attempt=?",
(state, exit_code, error_class, now, now - started, run_id, job_id, rollout, stage, attempt))
def stage_history(con, job_id: str, stage: str) -> list[float]:
"""同 job_id(跨 run)同阶段的历史成功耗时,用于估算。"""
return [r[0] for r in con.execute(
"SELECT duration_s FROM stages WHERE job_id=? AND stage=? AND state='OK' AND rollout=-1 AND duration_s IS NOT NULL",
(job_id, stage))]
def stage_history_by_model(con, model: str, stage: str) -> list[float]:
return [r[0] for r in con.execute(
"SELECT s.duration_s FROM stages s JOIN jobs j ON s.run_id=j.run_id AND s.job_id=j.job_id"
" WHERE j.model=? AND s.stage=? AND s.state='OK' AND s.rollout=-1 AND s.duration_s IS NOT NULL",
(model, stage))]
# ---------------------------------------------------------------- rollouts
def rollout_upsert(con, run_id, job_id, kk, mode, policy, state="QUEUED") -> None:
con.execute("INSERT OR IGNORE INTO rollouts (run_id, job_id, kk, mode, policy, state) VALUES (?,?,?,?,?,?)",
(run_id, job_id, kk, mode, policy, state))
def rollout_set(con, run_id, job_id, kk, **fields) -> None:
if not fields:
return
cols = ", ".join(f"{k}=?" for k in fields)
con.execute(f"UPDATE rollouts SET {cols} WHERE run_id=? AND job_id=? AND kk=?",
(*fields.values(), run_id, job_id, kk))
def list_rollouts(con, run_id, job_id) -> list[dict]:
return [dict(r) for r in con.execute("SELECT * FROM rollouts WHERE run_id=? AND job_id=? ORDER BY kk", (run_id, job_id))]
# ---------------------------------------------------------------- annotations / status
def annotate(con, run_id, job_id, note) -> None:
con.execute("INSERT INTO annotations (run_id, job_id, ts, note) VALUES (?,?,?,?)", (run_id, job_id, time.time(), note))
def status_counts(con, run_id: str, mode: str = "oracle") -> list[dict]:
return [dict(r) for r in con.execute(
"SELECT state, task, march, COUNT(*) n FROM jobs WHERE run_id=? AND mode=? GROUP BY state, task, march ORDER BY state, task, march",
(run_id, mode))]
def error_class_counts(con, run_id: str) -> list[dict]:
return [dict(r) for r in con.execute(
"SELECT COALESCE(error_class,'(none)') error_class, stage_failed, COUNT(*) n FROM jobs WHERE run_id=? AND state='FAIL'"
" GROUP BY error_class, stage_failed ORDER BY n DESC", (run_id,))]

46
harness/errors.py Normal file
View File

@ -0,0 +1,46 @@
"""stderr → error_class。正则规则库,unknown 进人工分类队列后在这里补规则。
规则顺序即优先级:先匹配到的赢。瞬态类在 TRANSIENT 里,决定重试策略。
"""
from __future__ import annotations
import re
DETERMINISTIC = ["unsupported_op", "shape_infer", "export_fail", "calib_nan", "verifier_mismatch", "config_error",
# 门失败(gates.classify_gate_failure)
"accuracy_collapse", "accuracy_drop", "numeric_mismatch", "march_mismatch", "perf_over_budget"]
TRANSIENT = ["oom", "board_offline", "stalled"]
ALL = DETERMINISTIC + TRANSIENT + ["unknown"]
# (error_class, compiled regex)。写规则时对着真实 stderr 写,别猜。
RULES: list[tuple[str, re.Pattern]] = [
("config_error", re.compile(r"Key '[^']+' error:|Wrong key '|Missing key|yaml.*(invalid|error)|schema.*error", re.I)), # hb_compile yaml 校验不过:harness/agent 配置写错
("oom", re.compile(r"CUDA out of memory|OutOfMemoryError|std::bad_alloc|Cannot allocate memory|MemoryError|Killed process .* out of memory", re.I)),
("board_offline", re.compile(r"Connection (refused|timed out|reset by peer)|No route to host|ssh: connect to host|board .*offline|hrt_.*: failed to connect", re.I)),
("unsupported_op", re.compile(r"(unsupported|not supported|Unsupported) (op|operator|node|type|layer)|No support for op|op .* is not supported on (nash|bpu)|不支持的算子", re.I)),
("shape_infer", re.compile(r"shape infer|ShapeInference|dynamic (shape|axis|dim)|Cannot infer shape|shape mismatch|unknown dimension", re.I)),
("export_fail", re.compile(r"torch\.onnx|onnx export|symbolic_trace|fx\.trace|TracerWarning: .*failed|Exporting the operator .* is not supported|torch\.export", re.I)),
("calib_nan", re.compile(r"\bnan\b.*(calib|quant|scale)|(calib|quant|scale).*\bnan\b|overflow encountered|inf.*threshold|threshold.*inf", re.I)),
("verifier_mismatch", re.compile(r"hb_verifier.*(fail|mismatch)|cosine.*(below|less than|<)|consistency check failed|verify.*mismatch", re.I)),
]
STALLED_MARK = "__BENCH_WATCHDOG_STALLED__"
def classify(stderr_text: str, stdout_text: str = "", exit_code: int | None = None) -> str:
text = (stderr_text or "") + "\n" + (stdout_text or "")
if STALLED_MARK in text:
return "stalled"
for cls, rx in RULES:
if rx.search(text):
return cls
return "unknown"
def is_transient(error_class: str) -> bool:
return error_class in TRANSIENT
def same_class(a: str | None, b: str | None) -> bool:
"""BLOCKED 判定用:同类错误。unknown 不与任何东西同类(避免误判为正确 BLOCKED)。"""
return bool(a) and a == b and a != "unknown"

11
harness/eval/__init__.py Normal file
View File

@ -0,0 +1,11 @@
"""逐任务评测器。agent 不可写;verdict 的唯一依据。FP32 基线与量化实测走同一份代码、同一子集、同一预处理。
EVALUATOR_VERSION 任何口径变化都要升;job.json 钉死它,基线缓存按它失效。
"""
EVALUATOR_VERSION = "0.4.0"
TASK_MODULES = {
"cls": "harness.eval.cls",
"det": "harness.eval.det",
# P4+:seg iseg pose mde stereo flow feat occ
}

57
harness/eval/cls.py Normal file
View File

@ -0,0 +1,57 @@
"""T01 分类:Top-1(主)/ Top-5(记录)。ImageNet-1K val T1=2000。
配对 delta 的标准误由不一致对决定:SE(Δ) ≈ sqrt((p10 + p01)/n)·100 pp。"""
from __future__ import annotations
import numpy as np
from .common import T1, preprocess_cls, run_batched
METRIC = "top1"
BATCH = 256
def _size(input_spec: str) -> int:
return int(input_spec.split("x")[-1])
def predict(runner, t1: T1, input_spec: str, items: list[dict] | None = None, resize: int = 0) -> tuple[np.ndarray, np.ndarray]:
"""返回 (top5_pred[n,5], labels[n])。resize=0 → size*256/224(经典 224/256);torchvision 各权重的 resize 由 models.csv 给。"""
items = items or t1.items
size = _size(input_spec)
resize = resize or round(size * 256 / 224)
preds, labels = [], []
for i in range(0, len(items), BATCH):
chunk = items[i:i + BATCH]
xs = [preprocess_cls(t1.image(it), size=size, resize=resize) for it in chunk]
for it, outs in zip(chunk, run_batched(runner, xs)):
logits = outs[0].reshape(-1)
preds.append(np.argsort(-logits)[:5])
labels.append(int(it["label"]))
return np.stack(preds), np.asarray(labels)
def metrics(top5: np.ndarray, labels: np.ndarray) -> dict:
top1 = float((top5[:, 0] == labels).mean() * 100)
t5 = float(np.any(top5 == labels[:, None], axis=1).mean() * 100)
return {"top1": top1, "top5": t5, "n": int(len(labels))}
def paired_se(top5_a: np.ndarray, top5_b: np.ndarray, labels: np.ndarray) -> float:
ca, cb = top5_a[:, 0] == labels, top5_b[:, 0] == labels
p_disc = float((ca != cb).mean())
return float(np.sqrt(p_disc / len(labels)) * 100)
def run_fp32(runner, t1: T1, input_spec: str, out: dict, resize: int = 0) -> None:
top5, labels = predict(runner, t1, input_spec, resize=resize)
out["metrics"] = metrics(top5, labels)
out["_pred"] = top5.tolist()
def run_quant(runner, t1: T1, input_spec: str, baseline: dict, out: dict, resize: int = 0) -> None:
top5, labels = predict(runner, t1, input_spec, resize=resize)
m = metrics(top5, labels)
se = paired_se(np.asarray(baseline["_pred"]), top5, labels) if baseline.get("_pred") else None
out["metrics"] = m
out["G3"] = {"metric": METRIC, "fp32": baseline["metrics"]["top1"], "quant": m["top1"],
"tier": "t1", "n": m["n"], "se": se, "also": {"top5": m["top5"], "fp32_top5": baseline["metrics"]["top5"]}}

153
harness/eval/common.py Normal file
View File

@ -0,0 +1,153 @@
"""T1 tar 读取、预处理、推理后端抽象。所有任务共用。"""
from __future__ import annotations
import hashlib
import io
import json
import tarfile
from dataclasses import dataclass
from pathlib import Path
import numpy as np
IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
@dataclass
class T1:
manifest: dict
tar: tarfile.TarFile
@classmethod
def open(cls, path: str | Path, expect_sha: str | None = None) -> "T1":
p = Path(path)
if expect_sha and expect_sha != "None":
h = hashlib.sha256()
with open(p, "rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
h.update(chunk)
if h.hexdigest() != expect_sha:
raise RuntimeError(f"T1 sha 不一致: {p} {h.hexdigest()[:12]} != {expect_sha[:12]}")
t = tarfile.open(p, "r")
man = json.loads(t.extractfile("manifest.json").read())
return cls(man, t)
@property
def items(self) -> list[dict]:
return self.manifest["items"]
def image(self, item: dict):
from PIL import Image
data = self.tar.extractfile(item["file"]).read()
return Image.open(io.BytesIO(data)).convert("RGB")
def g2_subset(self, n: int, seed: int = 20260914) -> list[dict]:
rng = np.random.default_rng(seed)
idx = rng.permutation(len(self.items))[:n]
return [self.items[i] for i in sorted(idx)]
# ---------------------------------------------------------------- 预处理(口径钉死,fp32 与 quant 同一份)
def preprocess_cls(img, size: int = 224, resize: int = 256) -> np.ndarray:
from PIL import Image
w, h = img.size
s = resize / min(w, h)
img = img.resize((max(1, round(w * s)), max(1, round(h * s))), Image.BILINEAR)
w, h = img.size
l, t = (w - size) // 2, (h - size) // 2
img = img.crop((l, t, l + size, t + size))
x = np.asarray(img, dtype=np.float32) / 255.0
x = (x - IMAGENET_MEAN) / IMAGENET_STD
return x.transpose(2, 0, 1)[None] # 1x3xHxW
def letterbox(img, size: int = 640):
"""YOLO 风格 letterbox。返回 (1x3xSxS float32 [0,1], scale, (padx, pady))。"""
from PIL import Image
w, h = img.size
s = size / max(w, h)
nw, nh = round(w * s), round(h * s)
img = img.resize((nw, nh), Image.BILINEAR)
canvas = Image.new("RGB", (size, size), (114, 114, 114))
px, py = (size - nw) // 2, (size - nh) // 2
canvas.paste(img, (px, py))
x = np.asarray(canvas, dtype=np.float32) / 255.0
return x.transpose(2, 0, 1)[None], s, (px, py)
# ---------------------------------------------------------------- 推理后端
class TorchRunner:
def __init__(self, model, source, weights, input_spec):
import torch
from harness.tools.export_onnx import load_model
self.torch = torch
self.m = load_model(model, source, Path(weights), input_spec)
self.dev = "cuda" if torch.cuda.is_available() else "cpu"
self.m.to(self.dev)
def __call__(self, x: np.ndarray) -> list[np.ndarray]:
with self.torch.no_grad():
out = self.m(self.torch.from_numpy(x).to(self.dev))
outs = out if isinstance(out, (list, tuple)) else [out]
flat = []
for o in outs:
if isinstance(o, (list, tuple)):
flat.extend(o)
else:
flat.append(o)
return [o.detach().float().cpu().numpy() for o in flat if hasattr(o, "detach")]
class HbmRunner:
"""x86 侧 hbm 推理。hbm 运行时(hbm_infer)在 runtime venv 里,本进程在 bench venv 里,
所以按块起子进程:一块 64 张 → npz → runtime venv 的 harness.tools.infer_hbm → npz。
单张调用也走这条路(慢),评测器应该用 batch()。"""
CHUNK = 64
def __init__(self, hbm: str, runtime_python: str, workdir: str | None = None):
import subprocess, tempfile
self.hbm = str(hbm)
self.py = runtime_python
self.tmp = Path(workdir or tempfile.mkdtemp(prefix="hbm-eval-"))
self.subprocess = subprocess
self._n = 0
def batch(self, xs: list[np.ndarray]) -> list[list[np.ndarray]]:
outs: list[list[np.ndarray]] = []
for i in range(0, len(xs), self.CHUNK):
chunk = xs[i:i + self.CHUNK]
self._n += 1
fin, fout = self.tmp / f"in{self._n:04d}.npz", self.tmp / f"out{self._n:04d}.npz"
np.savez(fin, **{f"s{j:04d}": x for j, x in enumerate(chunk)})
env = dict(**__import__("os").environ)
env["PYTHONPATH"] = str(Path(__file__).resolve().parents[2])
r = self.subprocess.run([self.py, "-m", "harness.tools.infer_hbm", "--model", self.hbm,
"--inputs", str(fin), "--out", str(fout)], capture_output=True, text=True, env=env)
if r.returncode != 0:
raise RuntimeError(f"infer_hbm failed rc={r.returncode}: {r.stderr[-2000:]}")
o = np.load(fout)
for j in range(len(chunk)):
keys = sorted(k for k in o.files if k.startswith(f"s{j:04d}/"))
outs.append([o[k] for k in keys])
fin.unlink(missing_ok=True); fout.unlink(missing_ok=True)
return outs
def __call__(self, x: np.ndarray) -> list[np.ndarray]:
return self.batch([x])[0]
def run_batched(runner, xs: list[np.ndarray]) -> list[list[np.ndarray]]:
"""TorchRunner 逐张、HbmRunner 分块——评测器只认这个入口。"""
if hasattr(runner, "batch"):
return runner.batch(xs)
return [runner(x) for x in xs]
def cosine(a: np.ndarray, b: np.ndarray) -> float:
a = a.astype(np.float64).ravel(); b = b.astype(np.float64).ravel()
na, nb = np.linalg.norm(a), np.linalg.norm(b)
return float(a @ b / (na * nb)) if na and nb else 0.0

101
harness/eval/det.py Normal file
View File

@ -0,0 +1,101 @@
"""T02 检测:box mAP@[.5:.95](主)/ AP50(记录)。COCO val2017 T1=1000,pycocotools。
口径钉死:letterbox 640、conf 0.001、NMS IoU 0.7、每图最多 300 框、pycocotools 版本记进 metrics。
后处理默认按 YOLOv8 输出 [1, 4+nc, N];其他头在 adapters 里给 decode()。
"""
from __future__ import annotations
import json
import tempfile
from pathlib import Path
import numpy as np
from .common import T1, letterbox, run_batched
METRIC = "map_50_95"
CONF, IOU, MAX_DET = 0.001, 0.7, 300
def _nms(boxes: np.ndarray, scores: np.ndarray, iou: float) -> np.ndarray:
try:
import torch, torchvision
keep = torchvision.ops.nms(torch.from_numpy(boxes).float(), torch.from_numpy(scores).float(), iou)
return keep.numpy()
except ImportError:
order = scores.argsort()[::-1]
keep = []
while order.size:
i = order[0]; keep.append(i)
xx1 = np.maximum(boxes[i, 0], boxes[order[1:], 0]); yy1 = np.maximum(boxes[i, 1], boxes[order[1:], 1])
xx2 = np.minimum(boxes[i, 2], boxes[order[1:], 2]); yy2 = np.minimum(boxes[i, 3], boxes[order[1:], 3])
inter = np.clip(xx2 - xx1, 0, None) * np.clip(yy2 - yy1, 0, None)
a = (boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1])
o = inter / (a[i] + a[order[1:]] - inter + 1e-9)
order = order[1:][o < iou]
return np.asarray(keep)
def decode_yolov8(out: np.ndarray) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
"""[1, 4+nc, N] → boxes xyxy(letterbox 坐标), scores, cls"""
p = out[0].T # N x (4+nc)
xywh, cls_scores = p[:, :4], p[:, 4:]
cls = cls_scores.argmax(1); score = cls_scores.max(1)
m = score > CONF
xywh, cls, score = xywh[m], cls[m], score[m]
boxes = np.stack([xywh[:, 0] - xywh[:, 2] / 2, xywh[:, 1] - xywh[:, 3] / 2,
xywh[:, 0] + xywh[:, 2] / 2, xywh[:, 1] + xywh[:, 3] / 2], 1)
# 按类偏移做 class-aware NMS
off = boxes + cls[:, None] * 4096
keep = _nms(off, score, IOU)[:MAX_DET]
return boxes[keep], score[keep], cls[keep]
BATCH = 64
def predict(runner, t1: T1, input_spec: str, decode=decode_yolov8) -> list[dict]:
size = int(input_spec.split("x")[-1])
cat_map = t1.manifest.get("cat_map") # 模型类别 idx → coco category_id
dets = []
items = t1.items
for i in range(0, len(items), BATCH):
chunk = items[i:i + BATCH]
pre = [letterbox(t1.image(it), size) for it in chunk]
for it, (x, s, (px, py)), outs in zip(chunk, pre, run_batched(runner, [p[0] for p in pre])):
boxes, scores, cls = decode(outs[0])
boxes = (boxes - np.array([px, py, px, py])) / s
for b, sc, c in zip(boxes, scores, cls):
cid = cat_map[str(int(c))] if cat_map else int(c)
dets.append({"image_id": it["id"], "category_id": cid, "score": float(sc),
"bbox": [float(b[0]), float(b[1]), float(b[2] - b[0]), float(b[3] - b[1])]})
return dets
def coco_eval(t1: T1, dets: list[dict]) -> dict:
from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval
import pycocotools
ann = json.loads(t1.tar.extractfile("annotations.json").read())
with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as f:
json.dump(ann, f); gt_path = f.name
gt = COCO(gt_path)
if not dets:
return {"map_50_95": 0.0, "ap50": 0.0, "n": len(t1.items), "pycocotools": getattr(pycocotools, "__version__", "?")}
dt = gt.loadRes(dets)
e = COCOeval(gt, dt, "bbox")
e.params.imgIds = [it["id"] for it in t1.items]
e.evaluate(); e.accumulate(); e.summarize()
return {"map_50_95": float(e.stats[0] * 100), "ap50": float(e.stats[1] * 100), "n": len(t1.items),
"pycocotools": getattr(pycocotools, "__version__", "?")}
def run_fp32(runner, t1: T1, input_spec: str, out: dict, resize: int = 0) -> None:
out["metrics"] = coco_eval(t1, predict(runner, t1, input_spec))
def run_quant(runner, t1: T1, input_spec: str, baseline: dict, out: dict, resize: int = 0) -> None:
m = coco_eval(t1, predict(runner, t1, input_spec))
out["metrics"] = m
out["G3"] = {"metric": METRIC, "fp32": baseline["metrics"]["map_50_95"], "quant": m["map_50_95"],
"tier": "t1", "n": m["n"], "se": None, "also": {"ap50": m["ap50"], "fp32_ap50": baseline["metrics"]["ap50"]}}

104
harness/eval/run.py Normal file
View File

@ -0,0 +1,104 @@
"""评测入口(在 backend venv 里跑)。
--mode fp32 : 02-baseline。T1 上算指标 + 抽 G2 样本保存输入与 FP32 输出。→ <out>/metrics.json, g2_inputs.npz, g2_fp32.npz
--mode quant: 08-eval。hbm 在 G2 样本上算余弦、在 T1 上算指标、与基线配对。→ <out>(eval.json)
"""
from __future__ import annotations
import argparse
import importlib
import json
import sys
from pathlib import Path
import numpy as np
from . import EVALUATOR_VERSION, TASK_MODULES
from .common import T1, TorchRunner, HbmRunner, cosine, preprocess_cls, letterbox, run_batched
def g2_inputs(task: str, t1: T1, input_spec: str, n: int, resize: int = 0) -> dict[str, np.ndarray]:
size = int(input_spec.split("x")[-1])
xs = {}
for i, it in enumerate(t1.g2_subset(n)):
img = t1.image(it)
if task == "cls":
xs[f"s{i:03d}"] = preprocess_cls(img, size=size, resize=resize or round(size * 256 / 224))
else:
xs[f"s{i:03d}"] = letterbox(img, size)[0]
return xs
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--mode", choices=["fp32", "quant"], required=True)
ap.add_argument("--task", required=True)
ap.add_argument("--model", required=True)
ap.add_argument("--source")
ap.add_argument("--weights")
ap.add_argument("--hbm")
ap.add_argument("--onnx")
ap.add_argument("--backend")
ap.add_argument("--input-spec", required=True)
ap.add_argument("--resize", type=int, default=0, help="cls:resize 短边(torchvision 默认权重各不同);0 = size*256/224")
ap.add_argument("--t1", required=True)
ap.add_argument("--t1-sha")
ap.add_argument("--baseline")
ap.add_argument("--g2-samples", type=int, default=32)
ap.add_argument("--calib-source", default="t0")
ap.add_argument("--runtime-python", help="hbm 运行时所在 venv 的 python(quant 模式必填)")
ap.add_argument("--evaluator-version", default=EVALUATOR_VERSION)
ap.add_argument("--out", required=True)
a = ap.parse_args()
if a.evaluator_version != EVALUATOR_VERSION:
print(f"evaluator version mismatch: job wants {a.evaluator_version}, code is {EVALUATOR_VERSION}", file=sys.stderr)
return 66
if a.task not in TASK_MODULES:
print(f"task {a.task} evaluator not implemented (P4+)", file=sys.stderr)
return 65
mod = importlib.import_module(TASK_MODULES[a.task])
t1 = T1.open(a.t1, a.t1_sha)
out: dict = {"task": a.task, "model": a.model, "evaluator_version": EVALUATOR_VERSION,
"t1_manifest_sha": t1.manifest.get("sha256"), "n": len(t1.items)}
if a.mode == "fp32":
runner = TorchRunner(a.model, a.source, a.weights, a.input_spec)
od = Path(a.out); od.mkdir(parents=True, exist_ok=True)
xs = g2_inputs(a.task, t1, a.input_spec, a.g2_samples, a.resize)
np.savez(od / "g2_inputs.npz", **xs)
np.savez(od / "g2_fp32.npz", **{f"{k}/out{i}": o for k, x in xs.items() for i, o in enumerate(runner(x))})
mod.run_fp32(runner, t1, a.input_spec, out, resize=a.resize)
(od / "metrics.json").write_text(json.dumps(out, ensure_ascii=False, indent=2))
print(json.dumps(out["metrics"]))
return 0
# quant
if not a.runtime_python:
print("--runtime-python required in quant mode", file=sys.stderr); return 2
runner = HbmRunner(a.hbm, a.runtime_python)
bdir = Path(a.baseline)
baseline = json.loads((bdir / "metrics.json").read_text())
if baseline.get("t1_manifest_sha") != t1.manifest.get("sha256"):
print("baseline T1 sha != current T1 sha:基线与实测子集不一致,delta 不可比", file=sys.stderr)
return 67
xs = np.load(bdir / "g2_inputs.npz")
fp = np.load(bdir / "g2_fp32.npz")
per_out: dict[str, list[float]] = {}
keys = list(xs.files)
for k, outs in zip(keys, run_batched(runner, [xs[k] for k in keys])):
for i, o in enumerate(outs):
ref = fp[f"{k}/out{i}"]
per_out.setdefault(f"out{i}", []).append(cosine(ref, o))
cos_per_output = {k: float(min(v)) for k, v in per_out.items()}
out["G2"] = {"cos_min": min(cos_per_output.values()) if cos_per_output else None,
"cos_per_output": cos_per_output, "n": len(xs.files), "source": a.calib_source}
mod.run_quant(runner, t1, a.input_spec, baseline, out, resize=a.resize)
Path(a.out).parent.mkdir(parents=True, exist_ok=True)
Path(a.out).write_text(json.dumps(out, ensure_ascii=False, indent=2))
print(json.dumps({"G2": out["G2"], "G3": out.get("G3")}))
return 0
if __name__ == "__main__":
sys.exit(main())

175
harness/gates.py Normal file
View File

@ -0,0 +1,175 @@
"""四道门:原始测量值 + job.json 门快照 → 逐门 verdict + terminal。
verdict 取值:pass | pass-screen(T1 放宽)| pass-weak(synthetic 校准源)| fail | unverified | na
terminal:PASS | FAIL | UNVERIFIED
oracle 与 rollout 走同一份代码;这里不读文件、不跑工具,纯函数,便于人肉核对。
"""
from __future__ import annotations
import math
PASSING = {"pass", "pass-screen", "pass-weak"}
LOWER_IS_BETTER = {"absrel", "rmse", "epe", "epe2d", "d1_all", "fl_all"}
def g1(m: dict | None, cfg: dict) -> dict:
if not m:
return {"verdict": "unverified", "reason": "no measurement"}
checks = {
"hbm_exists": m.get("hbm_exists"),
"march_ok": m.get("march_ok") if cfg.get("check_march", True) else True,
"io_ok": m.get("io_ok") if cfg.get("check_input_dtype_layout", True) else True,
"verifier_ok": m.get("verifier_ok") if cfg.get("run_hb_verifier", True) else True,
"log_clean": (not m.get("log_has_error")) if m.get("log_has_error") is not None else None,
}
if any(v is False for v in checks.values()):
failed = [k for k, v in checks.items() if v is False]
return {"verdict": "fail", "failed": failed, **checks, "march_read": m.get("march_read")}
if any(v is None for v in checks.values()):
return {"verdict": "unverified", "missing": [k for k, v in checks.items() if v is None], **checks}
return {"verdict": "pass", **checks, "march_read": m.get("march_read")}
def g2(m: dict | None, cfg: dict) -> dict:
if not m or (m.get("cos_min") is None and not m.get("cos_per_output")):
return {"verdict": "unverified", "reason": "no measurement"}
per = cfg.get("per_output")
if per:
outs = m.get("cos_per_output") or {}
results, ok = {}, True
for name, thr in per.items():
v = outs.get(name)
if v is None:
return {"verdict": "unverified", "reason": f"output {name} missing", "cos_per_output": outs}
results[name] = {"cos": v, "threshold": thr, "ok": v >= thr}
ok = ok and v >= thr
cos_min = min(outs.values()) if outs else None
verdict = "pass" if ok else "fail"
out = {"verdict": verdict, "cos_min": cos_min, "per_output": results}
else:
thr = cfg["threshold"]
cos_min = m["cos_min"]
out = {"verdict": "pass" if cos_min >= thr else "fail", "cos_min": cos_min, "threshold": thr}
if out["verdict"] == "pass" and m.get("source") == "synthetic":
out["verdict"] = "pass-weak"
out["source"] = m.get("source", "t1")
out["n_samples"] = m.get("n")
return out
def g3(m: dict | None, cfg: dict) -> dict:
if cfg.get("verdict") == "na" or not cfg.get("required", True):
return {"verdict": "na", "reason": cfg.get("reason", "gate not required for this task")}
if not m or m.get("fp32") is None or m.get("quant") is None:
return {"verdict": "unverified", "reason": "no measurement", "metric": cfg.get("metric")}
kind = cfg.get("kind")
if kind not in ("abs_pp", "rel_pct"):
raise ValueError(f"G3 kind 必须显式 abs_pp|rel_pct,得到 {kind!r}")
metric = m.get("metric") or cfg.get("metric")
metric_name = metric if isinstance(metric, str) else (metric[0] if metric else "")
fp32, quant, tol = float(m["fp32"]), float(m["quant"]), float(cfg["tol"])
lower_better = metric_name in LOWER_IS_BETTER
if kind == "abs_pp":
delta = quant - fp32
if lower_better:
delta = -delta
eff_tol = tol
if m.get("tier") == "t1" and m.get("se"):
eff_tol = tol - cfg.get("t1_screen_widen_se", 2) * float(m["se"])
ok = delta >= eff_tol
else: # rel_pct:劣化百分比
if fp32 == 0:
return {"verdict": "unverified", "reason": "fp32 baseline is 0"}
degr = (quant - fp32) / abs(fp32) * 100.0
if not lower_better:
degr = -degr
delta = degr
eff_tol = tol
if m.get("tier") == "t1" and m.get("se"):
eff_tol = tol + cfg.get("t1_screen_widen_se", 2) * float(m["se"])
ok = degr <= eff_tol
verdict = ("pass-screen" if m.get("tier") == "t1" else "pass") if ok else "fail"
return {"verdict": verdict, "metric": metric_name, "kind": kind, "fp32": fp32, "quant": quant,
"delta": delta, "tolerance": tol, "effective_tolerance": eff_tol, "tier": m.get("tier"),
"n": m.get("n"), "se": m.get("se")}
def g4(m: dict | None, cfg: dict) -> dict:
rec = {k: (m or {}).get(k) for k in cfg.get("record", [])}
rec["contended"] = bool((m or {}).get("contended"))
budget = cfg.get("latency_budget_ms")
if not budget:
return {"verdict": "na", **rec}
if not m or m.get("latency_ms_mean") is None:
return {"verdict": "unverified", **rec}
return {"verdict": "pass" if m["latency_ms_mean"] <= budget else "fail", "budget_ms": budget, **rec}
def evaluate(measurements: dict, gates_cfg: dict) -> dict:
"""返回 {"gates": {...}, "terminal": ...}。required 里的门决定 terminal。"""
res = {
"G1": g1(measurements.get("G1"), gates_cfg["G1"]),
"G2": g2(measurements.get("G2"), gates_cfg["G2"]),
"G3": g3(measurements.get("G3"), gates_cfg["G3"]),
"G4": g4(measurements.get("G4"), gates_cfg["G4"]),
}
required = list(gates_cfg.get("required", ["G1", "G2", "G3"]))
if gates_cfg["G4"].get("latency_budget_ms") and "G4" not in required:
required.append("G4")
verdicts = [res[g]["verdict"] for g in required]
if any(v == "fail" for v in verdicts):
terminal = "FAIL"
elif any(v == "unverified" for v in verdicts):
terminal = "UNVERIFIED"
else:
terminal = "PASS"
failed_gate = next((g for g in required if res[g]["verdict"] == "fail"), None)
return {"gates": res, "required": required, "terminal": terminal, "failed_gate": failed_gate}
def classify_gate_failure(ev: dict) -> str | None:
"""终态 FAIL 但没有阶段崩:从门给一个 error_class(喂 repair 队列与 BLOCKED 判定)。
march_mismatch G1 march 不符(作弊/配错)
verifier_mismatch G1 hb_verifier / hbm≡bc 不一致
accuracy_collapse G3 掉点 ≥ 10×容差 或 量化后精度接近随机(PTQ 校准崩了,如 EfficientNet/MobileNetV3 默认 int8)
accuracy_drop G3 掉点超容差但没崩
numeric_mismatch 只有 G2 余弦不达标
perf_over_budget 只有 G4 超预算
"""
g = ev["gates"]
if g["G1"]["verdict"] == "fail":
failed = g["G1"].get("failed", [])
if "march_ok" in failed:
return "march_mismatch"
if "verifier_ok" in failed and g["G3"].get("verdict") != "fail":
return "verifier_mismatch"
if g["G3"]["verdict"] == "fail":
d, tol = g["G3"].get("delta"), g["G3"].get("tolerance")
q, fp = g["G3"].get("quant"), g["G3"].get("fp32")
if d is not None and tol and (abs(d) >= 10 * abs(tol) or (fp and q is not None and q < 0.2 * fp)):
return "accuracy_collapse"
return "accuracy_drop"
if g["G1"]["verdict"] == "fail" and "verifier_ok" in g["G1"].get("failed", []):
return "verifier_mismatch"
if g["G2"]["verdict"] == "fail":
return "numeric_mismatch"
if g["G4"]["verdict"] == "fail":
return "perf_over_budget"
return None
def cosine(a, b) -> float:
"""逐输出余弦。a, b 为同 shape 的一维可迭代(numpy 数组会先 ravel)。"""
try:
import numpy as np
a = np.asarray(a, dtype=np.float64).ravel()
b = np.asarray(b, dtype=np.float64).ravel()
na, nb = np.linalg.norm(a), np.linalg.norm(b)
if na == 0 or nb == 0:
return 0.0
return float(a @ b / (na * nb))
except ImportError:
dot = sum(x * y for x, y in zip(a, b))
na = math.sqrt(sum(x * x for x in a))
nb = math.sqrt(sum(y * y for y in b))
return dot / (na * nb) if na and nb else 0.0

85
harness/paths.py Normal file
View File

@ -0,0 +1,85 @@
"""根目录、config.yaml、.env.local。所有路径解析只在这里做。"""
from __future__ import annotations
import os
from functools import lru_cache
from pathlib import Path
import yaml
BENCH_ROOT = Path(os.environ.get("BENCH_ROOT") or Path(__file__).resolve().parent.parent)
PROJECT_ROOT = BENCH_ROOT.parent
REGISTRY = BENCH_ROOT / "registry"
DATA = BENCH_ROOT / "data"
CACHE = BENCH_ROOT / "cache"
STATE = Path(os.environ["BENCH_STATE_DIR"]) if os.environ.get("BENCH_STATE_DIR") else BENCH_ROOT / "state"
RUNS = BENCH_ROOT / "runs"
REPORTS = BENCH_ROOT / "reports"
DATASETS = BENCH_ROOT / "datasets"
AGENT = BENCH_ROOT / "agent"
DB_PATH = STATE / "jobs.sqlite"
STAGE_NAMES = [
"01-fetch", "02-baseline", "03-export", # model-level → cache/
"04-calibrate", "05-compile", "06-hbm", "07-board", "08-eval", # target-level → job dir
]
MODEL_LEVEL = STAGE_NAMES[:3]
TARGET_LEVEL = STAGE_NAMES[3:]
TASKS = ["cls", "det", "seg", "iseg", "pose", "mde", "stereo", "flow", "feat", "occ"]
MARCHES = ["nash-e", "nash-m", "nash-p"]
def expand(p: str | os.PathLike | None, base: Path = BENCH_ROOT) -> Path | None:
if p is None:
return None
p = os.path.expandvars(os.path.expanduser(str(p)))
q = Path(p)
return q if q.is_absolute() else (base / q).resolve()
@lru_cache(maxsize=1)
def config() -> dict:
with open(BENCH_ROOT / "config.yaml", encoding="utf-8") as f:
cfg = yaml.safe_load(f) or {}
cfg.setdefault("paths", {})
if not cfg["paths"].get("project_root"):
cfg["paths"]["project_root"] = str(PROJECT_ROOT)
return cfg
def venv_python(backend: str) -> Path:
"""按 backend 选解释器。这是 harness 的职责,agent 无权改。"""
v = config()["paths"]["venv"].get(backend)
if not v:
raise KeyError(f"config.yaml paths.venv 没有 backend={backend}")
return expand(v) / "bin" / "python"
def venv_bin(backend: str, tool: str) -> Path:
return expand(config()["paths"]["venv"][backend]) / "bin" / tool
@lru_cache(maxsize=1)
def load_env_local() -> dict:
"""读取 bench/.env.local。只给 harness 用;rollout 的 agent 子进程环境由 stagerun/rollout 单独构造,永远不含这些。"""
env = {}
p = BENCH_ROOT / ".env.local"
if p.exists():
for line in p.read_text(encoding="utf-8").splitlines():
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
k, v = line.split("=", 1)
env[k.strip()] = v.strip().strip('"').strip("'")
return env
def secret(name: str) -> str | None:
return os.environ.get(name) or load_env_local().get(name)
def ensure_dirs() -> None:
for d in (STATE, RUNS, REPORTS, DATASETS, DATA / "pool", DATA / "t1", DATA / "calib",
CACHE / "weights", CACHE / "onnx", CACHE / "baseline"):
d.mkdir(parents=True, exist_ok=True)

516
harness/pipeline.py Normal file
View File

@ -0,0 +1,516 @@
"""oracle 主流程:preflight → new-run → expand → run。
run:每 job 顺序跑 01–08;每阶段拿对应资源池;瞬态重试;STALLED 进 long 车道;
产出 oracle/result.json;状态写 jobs.sqlite。--dry-run 走完整状态机但不调工具链。
"""
from __future__ import annotations
import hashlib
import json
import os
import shutil
import subprocess
import sys
import time
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from . import paths, db, gates, scheduler
from .errors import is_transient
from .eval import EVALUATOR_VERSION
from .registry import (JobSpec, expand_matrix, build_job_json, datasets, models, model_row,
registry_fingerprint, matrix)
from .stages import ORDER, StageContext
def _stamped(msg: str) -> None:
print(time.strftime("%H:%M:%S"), msg, flush=True)
# ---------------------------------------------------------------- preflight
def preflight(modes: list[str] | None = None, oracle_only: bool = False) -> tuple[list[str], list[str]]:
"""返回 (errors, warnings)。errors 非空 → 不允许 run。
agent 运行时(pi/node)缺失:matrix 里 e2e/repair 开着就是 ERROR——这个工厂的产品是轨迹,不是 benchmark;
只想跑 oracle 时显式 --oracle-only。(踩过:之前只 WARN,start_run 一路跑下去,跑了一夜零轨迹。)"""
errs, warns = [], []
cfg = paths.config()
# 1. registry 自洽
try:
rows = models()
specs = expand_matrix(modes)
except Exception as e:
return [f"registry: {e}"], warns
if not specs:
errs.append("matrix 展开为空")
# 2. venv
for be, v in cfg["paths"]["venv"].items():
py = paths.expand(v) / "bin" / "python"
if not py.exists():
(errs if be in {s.backend for s in specs} else warns).append(f"venv[{be}] 缺: {py}")
# 3. .horizon / pi / node
hz = paths.expand(cfg["paths"]["horizon"])
if not hz.exists():
warns.append(f".horizon 不在 {hz}(agent 模式需要)")
# pi 缺失对 oracle 无所谓,但只要 matrix 里开了 e2e/repair,就意味着"一条轨迹也采不到" → 必须是 ERROR。
# 踩过:pi 没装,preflight 只 WARN,全量 run 跑了几小时只产出 benchmark,rollouts 全停在 QUEUED。
pi = cfg["paths"].get("pi_bin") or "pi"
pi_missing = shutil.which(pi) is None and not Path(os.path.expanduser(pi)).exists()
if pi_missing:
(errs if (any(s.mode != "oracle" for s in specs) and not oracle_only) else warns).append(
f"pi 不在 PATH:{pi}(matrix 里 e2e/repair 开着 → 轨迹采不到;scripts/install_pi.sh)")
# 4. 数据:展开后需要的每个数据集的 pool / t1 / calib
need: dict[tuple, dict] = {}
for s in specs:
r = model_row(s.model)
need[(r.get("dataset"), s.task, r["input_spec"])] = r
dsets = datasets().get("datasets") or {}
for (ds, task, ispec), r in sorted(need.items(), key=lambda kv: str(kv[0])):
if not ds:
errs.append(f"models.csv {r['model_id']}: dataset 为空")
continue
if ds not in dsets:
errs.append(f"datasets.yaml 没有 {ds}({r['model_id']} 需要)")
continue
d = dsets[ds]
ver = d.get("version", "")
pool = paths.DATA / "pool" / ds / ver
if not (pool / "manifest.json").exists():
errs.append(f"pool 缺 {ds}/{ver}(scripts/ingest_pool.py --dataset {ds})")
t1n = (d.get("t1") or {}).get("n")
if task != "occ" and t1n not in (0, None):
t1 = paths.DATA / "t1" / ds / f"{(d.get('t1') or {}).get('seed', 0)}.tar"
if not t1.exists():
errs.append(f"T1 缺 {t1}(scripts/make_t1.py --dataset {ds})")
calib = paths.DATA / "calib" / task / ispec
if not (calib / "manifest.json").exists():
if task == "occ":
warns.append(f"calib 缺 {calib}:occ 将按阶梯回退,最终 synthetic")
else:
errs.append(f"calib 缺 {calib}(scripts/make_calib.py --task {task} --input-spec {ispec})")
# 5. 凭证
if not paths.secret("BOARD_SSH_KEY"):
warns.append("BOARD_SSH_KEY 未设:07-board 将 unverified")
if any(s.mode != "oracle" for s in specs) and not paths.secret("TEACHER_API_KEY"):
errs.append("TEACHER_API_KEY 未设,但 matrix 里 e2e/repair 是开的:轨迹一条也采不到")
# 6. .env.oe-package 死路径(agent 会读它)
envf = paths.PROJECT_ROOT / ".horizon" / ".env.oe-package"
if not envf.exists():
envf = paths.PROJECT_ROOT / ".env.oe-package"
if envf.exists():
for line in envf.read_text(errors="replace").splitlines():
if line.startswith("PYTHON_BIN="):
p = Path(os.path.expanduser(line.split("=", 1)[1].strip().strip('"')))
if not p.is_absolute():
p = paths.PROJECT_ROOT / p
if not p.exists():
errs.append(f".env.oe-package PYTHON_BIN 指向不存在的路径 {p}(scripts/fix_env.sh)")
return errs, warns
# ---------------------------------------------------------------- run / manifest
def _pip_freeze_sha(backend: str) -> str | None:
py = paths.venv_python(backend)
if not py.exists():
return None
try:
r = subprocess.run([str(py), "-m", "pip", "freeze"], capture_output=True, text=True, timeout=120)
if r.returncode == 0 and r.stdout.strip():
return hashlib.sha256(r.stdout.encode()).hexdigest()
except Exception:
pass
# PTQ venv 没有 pip:退到 site-packages 里 dist-info 目录名的哈希,效果等价
sp = sorted(p.name for p in (py.parent.parent / "lib").glob("python*/site-packages/*.dist-info"))
return hashlib.sha256("\n".join(sp).encode()).hexdigest() if sp else None
def harness_code_sha() -> str:
"""harness/ + export/ 全部 .py 的内容哈希。进 manifest;变了只告警(评测口径由 EVALUATOR_VERSION 单独钉)。"""
h = hashlib.sha256()
for p in sorted(list((paths.BENCH_ROOT / "harness").rglob("*.py")) + list((paths.BENCH_ROOT / "export").rglob("*.py"))):
h.update(str(p.relative_to(paths.BENCH_ROOT)).encode()); h.update(p.read_bytes())
return h.hexdigest()
def build_manifest() -> dict:
cfg = paths.config()
return {
"created_at": time.time(),
"evaluator_version": EVALUATOR_VERSION,
"harness_code_sha256": harness_code_sha(),
"registry_fingerprint": registry_fingerprint(),
"venv": {be: {"path": str(paths.expand(v)), "pip_freeze_sha256": _pip_freeze_sha(be)}
for be, v in cfg["paths"]["venv"].items()},
"board": {"host": paths.secret("BOARD_HOST"), "ucp": None, "board_type": None},
"host": os.uname().nodename,
}
def new_run(note: str = "") -> str:
paths.ensure_dirs()
with db.connect() as con:
run_id = db.next_run_id(con, time.strftime("%Y%m%d"))
man = build_manifest()
db.create_run(con, run_id, man, note)
rd = paths.RUNS / run_id
rd.mkdir(parents=True, exist_ok=True)
(rd / "manifest.json").write_text(json.dumps(man, indent=2, ensure_ascii=False))
return run_id
def check_manifest(run_id: str) -> list[str]:
"""job 启动前比对:两套 venv 的 pip freeze 哈希、registry 指纹。不一致拒跑。"""
man = json.loads((paths.RUNS / run_id / "manifest.json").read_text())
now = build_manifest()
bad = []
if man["registry_fingerprint"] != now["registry_fingerprint"]:
bad.append("registry 改了:开新 run")
for be, v in man["venv"].items():
if v["pip_freeze_sha256"] and v["pip_freeze_sha256"] != now["venv"].get(be, {}).get("pip_freeze_sha256"):
bad.append(f"venv[{be}] 变了:开新 run")
if man.get("evaluator_version") != now["evaluator_version"]:
bad.append("EVALUATOR_VERSION 变了:开新 run")
if man.get("harness_code_sha256") and man["harness_code_sha256"] != now["harness_code_sha256"]:
print(f"WARN harness 代码与 run {run_id} 创建时不同({man['harness_code_sha256'][:8]} → {now['harness_code_sha256'][:8]});"
"结果口径若受影响请开新 run", file=sys.stderr)
return bad
def expand(run_id: str, modes: list[str] | None = None, marches: list[str] | None = None) -> dict:
specs = expand_matrix(modes)
if marches:
specs = [s for s in specs if s.march in marches] # 只展开部分 march(如补跑 nash-p),不改 registry
# 按 job_id 聚合 rollout 计划
plans: dict[str, list[dict]] = {}
first: dict[str, JobSpec] = {}
for s in specs:
first.setdefault(s.job_id, s)
if s.mode != "oracle":
plans.setdefault(s.job_id, []).append({"mode": s.mode, "policy": s.policy, "k": s.k})
n_new = 0
with db.connect() as con:
for s in specs:
est = scheduler.estimate_job(con, s.model, s.task)
n_new += db.upsert_job(con, run_id, s.to_row(), est)
for i in range(s.k if s.mode != "oracle" else 0):
db.rollout_upsert(con, run_id, s.job_id, i, s.mode, s.policy)
n_dirs = 0
for jid, s in first.items():
jd = paths.RUNS / run_id / jid
jj = jd / "job.json"
if jj.exists():
continue
jd.mkdir(parents=True, exist_ok=True)
(jd / "oracle" / "stages").mkdir(parents=True, exist_ok=True)
(jd / "oracle" / "artifacts").mkdir(parents=True, exist_ok=True)
jj.write_text(json.dumps(build_job_json(s, run_id, EVALUATOR_VERSION, plans.get(jid)), indent=2, ensure_ascii=False))
os.chmod(jj, 0o444)
# data -> 软链只是给 agent 引用方便;那条"一个 job 一个目录"约束产出,不约束输入
link = jd / "data"
if not link.exists():
try:
link.symlink_to(paths.DATA)
except OSError:
pass
n_dirs += 1
return {"jobs_total": len(specs), "jobs_new": n_new, "job_dirs_new": n_dirs, "job_ids": len(first)}
# ---------------------------------------------------------------- rollouts(轨迹采集主循环)
def collect_rollouts(run_id: str, job_ids: list[str] | None = None, modes: list[str] | None = None,
parallel: int | None = None, dry_run: bool = False, limit: int | None = None,
log=_stamped) -> dict:
"""把这个 run 里所有 QUEUED 的 rollout 跑掉 —— 这是"采轨迹"的入口,oracle 的 run_oracle 只产 benchmark。
前提:该 job 的 oracle 已结束(01–03 的 cache 齐了;repair 还要求 oracle FAIL)。
并发受 agent 池约束(config.yaml pools.agent.workers),与 oracle 共用同一组跨进程槽位锁。
每个 rollout 结束即 verify → verdict.json;轨迹落在 rollouts/<kk>/agent/events.jsonl。
"""
from .rollout import run_rollout, setup_models_json
mx = matrix()
repair_ecs = set(((mx.get("modes") or {}).get("repair") or {}).get("error_classes") or [])
if not dry_run:
setup_models_json()
with db.connect() as con:
rows = [dict(r) for r in con.execute(
"SELECT r.*, j.state AS job_state, j.error_class AS job_ec FROM rollouts r"
" LEFT JOIN jobs j ON j.run_id=r.run_id AND j.job_id=r.job_id AND j.mode='oracle'"
" WHERE r.run_id=? AND r.state='QUEUED' ORDER BY r.job_id, r.kk", (run_id,))]
todo, skipped = [], {}
for r in rows:
if job_ids and r["job_id"] not in job_ids:
continue
if modes and r["mode"] not in modes:
continue
if r["job_state"] not in ("PASS", "FAIL", "UNVERIFIED"):
skipped["oracle 未完成"] = skipped.get("oracle 未完成", 0) + 1
continue
if r["mode"] == "repair":
if r["job_state"] != "FAIL":
skipped["repair 但 oracle 没失败"] = skipped.get("repair 但 oracle 没失败", 0) + 1
continue
if repair_ecs and r["job_ec"] not in repair_ecs:
skipped[f"repair 的 error_class 不在采集范围({r['job_ec']})"] = skipped.get(
f"repair 的 error_class 不在采集范围({r['job_ec']})", 0) + 1
continue
todo.append(r)
if limit:
todo = todo[:limit]
log(f"rollouts: {len(todo)} 条待采,跳过 {sum(skipped.values())}({skipped or '-'})")
def one(r: dict) -> dict:
try:
with scheduler.acquire("agent"):
return run_rollout(run_id, r["job_id"], r["kk"], r["mode"], r["policy"], dry_run=dry_run)
except SystemExit as e: # 前置缺失等:记下来,不要让整批停
log(f"[{r['job_id']}#{r['kk']:02d}] 跳过:{e}")
return {"terminal": "SKIPPED", "reason": str(e)}
except Exception as e: # noqa: BLE001
log(f"[{r['job_id']}#{r['kk']:02d}] 异常:{type(e).__name__}: {e}")
return {"terminal": "ERROR", "reason": repr(e)}
n = parallel or paths.config()["pools"]["agent"]["workers"]
summary: dict = {}
with ThreadPoolExecutor(max_workers=n) as ex:
for res in ex.map(one, todo):
t = res.get("terminal", "?")
summary[t] = summary.get(t, 0) + 1
if t not in ("SKIPPED", "ERROR"):
log(f" → {res.get('rollout_dir', '')} {t} claim_consistent={res.get('claim_consistent')} review={res.get('review')}")
log(f"rollouts done: {summary}")
return summary
# ---------------------------------------------------------------- rejudge
def rejudge(run_id: str, job_id: str) -> dict:
"""不重跑,只从 oracle/stages 的测量值重新过四道门、重写 result.json 与 jobs.sqlite。
用途:harness 汇总 bug 修复后重判;门阈值以 job.json 快照为准,不会偷偷换阈值。"""
from .verify import collect_measurements
job, jd = _load_job(run_id, job_id)
rp = jd / "oracle" / "result.json"
old = json.loads(rp.read_text()) if rp.exists() else {}
meas, summ = collect_measurements(jd, jd / "oracle" / "stages")
if not meas.get("G4") and (old.get("measurements") or {}).get("G4"):
meas["G4"] = old["measurements"]["G4"] # 补测写进来的 G4 不能被 07 的旧 SKIPPED meta 冲掉
ev = gates.evaluate(meas, job["gates"])
terminal = ev["terminal"]
if old.get("stage_failed") and old.get("terminal") in ("FAIL", "UNVERIFIED") and old["stage_failed"] != "08-eval":
terminal = old["terminal"] # 阶段本身没跑成的,不因为重判变 PASS
ec = old.get("error_class") if old.get("stage_failed") else (gates.classify_gate_failure(ev) if terminal == "FAIL" else None)
result = {**old, "measurements": meas, "gates": ev["gates"], "required": ev["required"], "terminal": terminal,
"failed_gate": ev["failed_gate"], "error_class": ec, "rejudged_at": time.time(), "stage_summary": summ}
rp.write_text(json.dumps(result, indent=2, ensure_ascii=False, default=str))
with db.connect() as con:
db.set_job_state(con, run_id, job_id, terminal, error_class=result.get("error_class"),
stage_failed=result.get("stage_failed"), result=result)
return {"job_id": job_id, "terminal": terminal, "failed_gate": ev["failed_gate"],
"gates": {g: v["verdict"] for g, v in ev["gates"].items()}}
# ---------------------------------------------------------------- board backfill
def board_backfill(run_id: str, job_ids: list[str] | None = None, log=_stamped) -> dict:
"""给已结束但没有 G4 的 job 补板端实测(job.json 展开时 boards.yaml 还没填板型的情况)。
只补 march 与沙箱板一致的 job;跑 07-board 到新的 attempt 目录,然后 rejudge。"""
from .registry import board_for_march
from .stages import Board
with db.connect() as con:
jobs = db.list_jobs(con, run_id, mode="oracle")
done, skipped = 0, {}
for j in jobs:
if j["state"] not in ("PASS", "FAIL", "UNVERIFIED") or (job_ids and j["job_id"] not in job_ids):
continue
board = board_for_march(j["march"])
if not board:
skipped["no_board_for_march"] = skipped.get("no_board_for_march", 0) + 1; continue
job, jd = _load_job(run_id, j["job_id"])
hbm = jd / "oracle" / "artifacts" / f"{j['job_id']}.hbm"
if not hbm.exists():
skipped["no_hbm"] = skipped.get("no_hbm", 0) + 1; continue
res = json.loads((jd / "oracle" / "result.json").read_text()) if (jd / "oracle" / "result.json").exists() else {}
if (res.get("measurements") or {}).get("G4"):
skipped["has_G4"] = skipped.get("has_G4", 0) + 1; continue
job = dict(job); job["board"] = board # 运行时注入板信息;job.json 本身不改
stage = Board()
prev = sorted((jd / "oracle" / "stages" / stage.name).glob("a*")) if (jd / "oracle" / "stages" / stage.name).exists() else []
ctx = StageContext(run_id=run_id, job=job, job_dir=jd, stage_root=jd / "oracle" / "stages",
artifacts=jd / "oracle" / "artifacts", rollout=-1, attempt=len(prev))
with db.connect() as con:
db.stage_start(con, run_id, j["job_id"], stage.name, ctx.attempt, -1, "backfill")
with scheduler.acquire("board") as slot:
ctx.extra["slot"] = slot
out = stage.run(ctx)
with db.connect() as con:
db.stage_finish(con, run_id, j["job_id"], stage.name, ctx.attempt, out.state, out.exit_code, out.error_class, -1)
if out.state == "OK" and out.measurements.get("G4"):
# 把 G4 写进 result.measurements,再重判
res.setdefault("measurements", {})["G4"] = out.measurements["G4"]
res["board_backfilled_at"] = time.time()
(jd / "oracle" / "result.json").write_text(json.dumps(res, indent=2, ensure_ascii=False, default=str))
rejudge(run_id, j["job_id"])
g4 = out.measurements["G4"]
log(f"[{j['job_id']}] G4 backfilled: {g4.get('latency_ms_mean'):.3f} ms mean / {g4.get('latency_ms_p95'):.3f} p95")
done += 1
else:
log(f"[{j['job_id']}] board stage {out.state} ec={out.error_class}")
skipped["board_fail"] = skipped.get("board_fail", 0) + 1
return {"backfilled": done, "skipped": skipped}
# ---------------------------------------------------------------- oracle
def _load_job(run_id: str, job_id: str) -> tuple[dict, Path]:
jd = paths.RUNS / run_id / job_id
return json.loads((jd / "job.json").read_text()), jd
def run_oracle_job(run_id: str, job_id: str, dry_run: bool = False, log=_stamped, force: bool = False) -> dict:
job, jd = _load_job(run_id, job_id)
cfg = paths.config()
with db.connect() as con:
if not db.claim_job(con, run_id, job_id, force=force):
log(f"[{job_id}] 已被别的进程认领或已完成,跳过")
return {"terminal": "SKIPPED", "job_id": job_id}
ctx = StageContext(run_id=run_id, job=job, job_dir=jd, stage_root=jd / "oracle" / "stages",
artifacts=jd / "oracle" / "artifacts", rollout=-1, dry_run=dry_run)
result = {"schema": "bench.result/1", "run_id": run_id, "job_id": job_id, "round": 1,
"stages": {}, "measurements": {}, "started_at": time.time()}
terminal, error_class, stage_failed = None, None, None
for S in ORDER:
stage = S()
attempt, lane = 0, "normal"
while True:
ctx.attempt, ctx.lane = attempt, lane
with db.connect() as con:
db.stage_start(con, run_id, job_id, stage.name, attempt, -1, lane)
if stage.level == "model":
# 同一模型的三个 march job 并行:只有一个真跑 01–03,其余进锁后看到缓存 → SKIPPED
with scheduler.model_lock(job["model"]), scheduler.acquire(stage.pool) as slot:
ctx.extra["slot"] = slot
out = stage.run(ctx)
else:
with scheduler.acquire(stage.pool) as slot:
ctx.extra["slot"] = slot
out = stage.run(ctx)
with db.connect() as con:
db.stage_finish(con, run_id, job_id, stage.name, attempt, out.state, out.exit_code, out.error_class, -1)
log(f"[{job_id}] {stage.name} a{attempt:02d} {lane}: {out.state} ec={out.error_class} {out.duration_s:.1f}s")
rec = result["stages"].setdefault(stage.name, {"attempts": []})
rec["attempts"].append({"attempt": attempt, "lane": lane, "state": out.state, "exit_code": out.exit_code,
"error_class": out.error_class, "duration_s": round(out.duration_s, 2)})
rec.update({"state": out.state, "error_class": out.error_class})
if out.state in ("OK", "SKIPPED"):
for k, v in (out.measurements or {}).items():
result["measurements"][k] = v
calib_meta = (out.measurements or {}).get("calib")
if calib_meta:
ctx.extra["calib_source"] = calib_meta.get("calib_source")
break
# 失败:决定重试
if out.state == "STALLED" and attempt < cfg["retry"]["stalled"]:
attempt, lane = attempt + 1, "long"; continue
if out.state == "FAIL" and is_transient(out.error_class or "") and attempt < cfg["retry"]["transient"]:
attempt += 1; continue
break
if out.state not in ("OK", "SKIPPED"):
ec = out.error_class or ("stalled" if out.state == "STALLED" else "unknown")
if stage.name == "07-board":
result["measurements"]["G4"] = None # unverified,不阻塞
result["stages"][stage.name]["note"] = "board failed after retries → G4 unverified"
continue
if stage.name == "08-eval":
terminal, error_class, stage_failed = "UNVERIFIED", ec, stage.name
break
# 01–06 失败 = G1 不可能过
result["measurements"]["G1"] = {"hbm_exists": False}
terminal, error_class, stage_failed = "FAIL", ec, stage.name
break
ev = gates.evaluate(result["measurements"], job["gates"])
if not error_class and (terminal or ev["terminal"]) == "FAIL":
error_class = gates.classify_gate_failure(ev)
result.update({"gates": ev["gates"], "required": ev["required"],
"terminal": terminal or ev["terminal"], "failed_gate": ev["failed_gate"],
"error_class": error_class, "stage_failed": stage_failed, "finished_at": time.time()})
if dry_run:
result["terminal"] = "UNVERIFIED"; result["dry_run"] = True
(jd / "oracle" / "result.json").write_text(json.dumps(result, indent=2, ensure_ascii=False, default=str))
with db.connect() as con:
db.set_job_state(con, run_id, job_id, result["terminal"], error_class=error_class,
stage_failed=stage_failed, result=result)
log(f"[{job_id}] → {result['terminal']} ec={error_class} stage={stage_failed}")
return result
def scheduler_alive() -> bool:
"""有没有 bench run 进程活着(持有 state/locks/run-*.lock)。"""
import fcntl
for p in (paths.STATE / "locks").glob("run-*.lock"):
try:
f = open(p, "w")
try:
fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
fcntl.flock(f, fcntl.LOCK_UN)
except BlockingIOError:
return True
finally:
f.close()
except OSError:
continue
return False
def reset_stale(run_id: str, force: bool = False) -> dict:
"""调度进程崩了/被杀后,ORACLE_RUNNING 的 job 回 PENDING,RUNNING 的 stage 记 ABORTED。有活着的调度进程时拒绝(--force 除外)。"""
if scheduler_alive() and not force:
raise SystemExit("还有 bench run 进程活着,拒绝 reset(--force 可强制)")
with db.connect() as con:
n_jobs = con.execute("UPDATE jobs SET state='PENDING', started_at=NULL WHERE run_id=? AND state='ORACLE_RUNNING'", (run_id,)).rowcount
n_st = con.execute("UPDATE stages SET state='ABORTED', finished_at=? WHERE run_id=? AND state='RUNNING'", (time.time(), run_id)).rowcount
return {"jobs_reset": n_jobs, "stages_aborted": n_st}
def run_oracle(run_id: str, job_ids: list[str] | None = None, dry_run: bool = False,
parallel: int | None = None, log=_stamped) -> dict:
import fcntl
(paths.STATE / "locks").mkdir(parents=True, exist_ok=True)
lockf = open(paths.STATE / "locks" / f"run-{os.getpid()}.lock", "w")
fcntl.flock(lockf, fcntl.LOCK_EX)
if not dry_run:
bad = check_manifest(run_id)
if bad:
raise SystemExit("manifest 不一致,拒跑:" + "; ".join(bad))
with db.connect() as con:
jobs = db.list_jobs(con, run_id, mode="oracle")
if job_ids:
jobs = [j for j in jobs if j["job_id"] in job_ids] # 显式指定:允许重跑(force)
else:
jobs = [j for j in jobs if j["state"] == "PENDING"]
jobs = scheduler.order_jobs(jobs)
n = parallel or paths.config()["pools"]["cpu"]["workers"]
summary = {"PASS": 0, "FAIL": 0, "UNVERIFIED": 0}
log(f"run {run_id}: {len(jobs)} jobs, parallel={n}")
with ThreadPoolExecutor(max_workers=n) as ex:
for r in ex.map(lambda j: run_oracle_job(run_id, j["job_id"], dry_run, log, force=bool(job_ids)), jobs):
summary[r["terminal"]] = summary.get(r["terminal"], 0) + 1
log(f"run {run_id} done: {summary}")
try:
from .report import render
render(run_id)
except Exception as e: # noqa: BLE001
log(f"report failed: {e}")
fcntl.flock(lockf, fcntl.LOCK_UN); lockf.close()
try:
(paths.STATE / "locks" / f"run-{os.getpid()}.lock").unlink()
except OSError:
pass
return summary

303
harness/registry.py Normal file
View File

@ -0,0 +1,303 @@
"""registry/* 加载、校验、matrix 展开、job_id、job.json 快照。
展开是纯函数:同一份 registry 永远展开出同一组 job。
"""
from __future__ import annotations
import csv
import hashlib
import json
import re
from dataclasses import dataclass, field, asdict
from functools import lru_cache
from pathlib import Path
import yaml
from . import paths
from .paths import REGISTRY, TASKS, MARCHES
SEG_RE = re.compile(r"^[a-z0-9.]+$")
JOB_ID_RE = re.compile(r"^[a-z0-9.]+--[a-z0-9.]+--nash-[emp]--[a-z0-9.]+--[a-z0-9.]+$")
# march 段本身含 '-'(nash-e),是唯一允许段内出现单横线的段
def _yaml(name: str) -> dict:
with open(REGISTRY / name, encoding="utf-8") as f:
return yaml.safe_load(f) or {}
@lru_cache(maxsize=None)
def models() -> list[dict]:
rows = []
with open(REGISTRY / "models.csv", encoding="utf-8", newline="") as f:
for r in csv.DictReader(f):
r = {k.strip(): (v or "").strip() for k, v in r.items()}
if not r.get("model_id"):
continue
rows.append(r)
validate_models(rows)
return rows
def validate_models(rows: list[dict]) -> None:
seen = set()
for r in rows:
mid = r["model_id"]
if not SEG_RE.match(mid):
raise ValueError(f"models.csv model_id 非法(只允许 a-z0-9.): {mid!r}")
if mid in seen:
raise ValueError(f"models.csv model_id 重复: {mid}")
seen.add(mid)
if r["task"] not in TASKS:
raise ValueError(f"models.csv {mid}: task={r['task']!r} 不在 {TASKS}")
if r["priority"] not in ("A", "B", "C"):
raise ValueError(f"models.csv {mid}: priority={r['priority']!r}")
if not re.match(r"^\d+x\d+x\d+x\d+$", r.get("input_spec", "")):
raise ValueError(f"models.csv {mid}: input_spec={r.get('input_spec')!r} 需形如 1x3x224x224")
@lru_cache(maxsize=None)
def matrix() -> dict:
return _yaml("matrix.yaml")
@lru_cache(maxsize=None)
def gates() -> dict:
return _yaml("gates.yaml")
@lru_cache(maxsize=None)
def budgets() -> dict:
return _yaml("budgets.yaml")
@lru_cache(maxsize=None)
def datasets() -> dict:
return _yaml("datasets.yaml")
@lru_cache(maxsize=None)
def boards() -> dict:
return _yaml("boards.yaml")
def registry_fingerprint() -> str:
h = hashlib.sha256()
for name in sorted(p.name for p in REGISTRY.iterdir() if p.is_file()):
h.update(name.encode())
h.update((REGISTRY / name).read_bytes())
return h.hexdigest()
# ---------------------------------------------------------------- job_id
def make_job_id(task: str, model: str, march: str, backend: str, qcfg: str) -> str:
for seg in (task, model, backend, qcfg):
if not SEG_RE.match(seg):
raise ValueError(f"job_id 段非法: {seg!r}")
if march not in MARCHES:
raise ValueError(f"march 非法: {march!r}")
jid = f"{task}--{model}--{march}--{backend}--{qcfg}"
assert JOB_ID_RE.match(jid), jid
return jid
def parse_job_id(jid: str) -> dict:
if not JOB_ID_RE.match(jid):
raise ValueError(f"job_id 非法: {jid!r}")
task, model, march, backend, qcfg = jid.split("--")
return dict(task=task, model=model, march=march, backend=backend, qcfg=qcfg)
# ---------------------------------------------------------------- expand
@dataclass
class JobSpec:
job_id: str
task: str
model: str
march: str
backend: str
qcfg: str
priority: str
mode: str # oracle | e2e | repair
policy: str | None = None # teacher | student | None(oracle)
k: int = 1
data_tier: str = "t1"
gates_required: list[str] = field(default_factory=lambda: ["G1", "G2", "G3"])
overrides: dict = field(default_factory=dict)
def to_row(self) -> dict:
d = asdict(self)
d["gates_required"] = json.dumps(self.gates_required)
d["overrides"] = json.dumps(self.overrides, ensure_ascii=False)
return d
def _select(row: dict, sel: dict | None) -> bool:
# status 默认只要 ready;matrix 里 select.status 可放宽(如 [ready, needs_eval] 做 compile-only)
allowed = (sel or {}).get("status", ["ready"])
if (row.get("status") or "ready") not in allowed:
return False
if not sel:
return True
if "priority" in sel and row["priority"] not in sel["priority"]:
return False
if "tasks" in sel and row["task"] not in sel["tasks"]:
return False
if "models" in sel and row["model_id"] not in sel["models"]:
return False
return True
def _apply_overrides(row: dict, mx: dict) -> dict:
out: dict = {}
for ov in mx.get("overrides", []) or []:
m = ov.get("match", {})
if all(row.get(k if k != "model" else "model_id") == v for k, v in m.items()):
out.update({k: v for k, v in ov.items() if k != "match"})
return out
def expand_matrix(modes: list[str] | None = None) -> list[JobSpec]:
"""把 registry 展开成 JobSpec 列表。repair 模式的 job 在 oracle FAIL 后由 pipeline 动态追加,这里不展开。"""
mx = matrix()
rows = models()
marches_all = mx.get("marches", MARCHES)
backends_all = mx.get("backends", ["ptq"])
qcfg_by_march = mx.get("qcfg_by_march", {})
tier = mx.get("data_tier", "t1")
out: list[JobSpec] = []
for mode, mcfg in (mx.get("modes") or {}).items():
if modes and mode not in modes:
continue
if not mcfg.get("enabled", True) or mcfg.get("when") == "oracle_fail":
continue
marches = marches_all if mcfg.get("marches", "all") == "all" else mcfg["marches"]
backends = backends_all if mcfg.get("backends", "all") == "all" else mcfg["backends"]
policies = mcfg.get("policies") or [None]
k = int(mcfg.get("k", 1))
for row in rows:
if not _select(row, mcfg.get("select")):
continue
ov = _apply_overrides(row, mx)
for march in marches:
qcfg = ov.get("qcfg") or qcfg_by_march.get(march, "int8")
for backend in backends:
for policy in policies:
jid = make_job_id(row["task"], row["model_id"], march, backend, qcfg)
out.append(JobSpec(
job_id=jid, task=row["task"], model=row["model_id"], march=march,
backend=backend, qcfg=qcfg, priority=row["priority"], mode=mode,
policy=policy, k=k, data_tier=tier,
gates_required=list(ov.get("gates", ["G1", "G2", "G3"])),
overrides=ov,
))
return out
# ---------------------------------------------------------------- job.json
def status_counts() -> dict:
from collections import Counter
return dict(Counter((r.get("status") or "ready") for r in models()))
def model_row(model_id: str) -> dict:
for r in models():
if r["model_id"] == model_id:
return r
raise KeyError(model_id)
def dataset_for(task: str, model_id: str) -> str:
r = model_row(model_id)
return r.get("dataset") or ""
def board_for_march(march: str) -> dict | None:
b = boards()
for bname, bcfg in (b.get("boards") or {}).items():
if bcfg.get("march") == march:
return {"name": bname, **bcfg}
return None
def gate_snapshot(spec: JobSpec) -> dict:
g = gates()
task = spec.task
g2_thr = spec.overrides.get("g2_threshold") or g["G2"].get("overrides", {}).get(task)
if isinstance(g2_thr, dict): # feat:逐输出
g2 = {"threshold": None, "per_output": g2_thr, "n_samples": g["G2"]["n_samples"]}
else:
g2 = {"threshold": g2_thr or g["G2"]["default_threshold"], "n_samples": g["G2"]["n_samples"]}
g3_task = dict(g["G3"]["tasks"].get(task, {}))
g3 = {"required": "G3" in spec.gates_required, **g3_task,
"t1_screen_widen_se": g["G3"].get("t1_screen_widen_se", 2)}
if spec.overrides.get("g3"):
g3.update(spec.overrides["g3"])
return {
"required": spec.gates_required,
"G1": g["G1"],
"G2": g2,
"G3": g3,
"G4": {"mandatory": False, "latency_budget_ms": spec.overrides.get("latency_budget_ms"),
"record": g["G4"]["record"]},
}
def data_snapshot(spec: JobSpec) -> dict:
"""job.json.data:calib / t1 / t2 的 pool 路径与 manifest sha。缺的记 null,preflight 会报。"""
row = model_row(spec.model)
ds_name = row.get("dataset") or ""
ds = (datasets().get("datasets") or {}).get(ds_name, {})
ver = ds.get("version", "")
pool = paths.DATA / "pool" / ds_name / ver
t1_spec = ds.get("t1", {}) or {}
seed = t1_spec.get("seed", 0)
t1_tar = paths.DATA / "t1" / ds_name / f"{seed}.tar"
calib_dir = paths.DATA / "calib" / spec.task / row["input_spec"]
def sha_of(p: Path, key: str = "sha256") -> str | None:
m = p / "manifest.json"
if m.exists():
try:
return json.loads(m.read_text()).get(key)
except Exception:
return None
return None
return {
"tier": spec.data_tier,
"dataset": ds_name,
"version": ver,
"calib": {"path": str(calib_dir), "n": (ds.get("calib") or {}).get("n", 128),
"manifest_sha": sha_of(calib_dir), "source": None},
"t1": {"path": str(t1_tar), "n": t1_spec.get("n"), "seed": seed,
"manifest_sha": sha_of(t1_tar.parent) if t1_tar.parent.exists() else None, # tar 文件 sha(T1.open 校验)
"content_sha": sha_of(t1_tar.parent, "content_sha256") if t1_tar.parent.exists() else None}, # 清单内容 sha(基线缓存键)
"t2": {"path": str(pool), "manifest_sha": sha_of(pool)},
}
def build_job_json(spec: JobSpec, run_id: str, evaluator_version: str, plans: list[dict] | None = None) -> dict:
"""job.json 与 mode 无关:一个 job_id 一份;oracle 与所有 rollout 共用。rollout 计划单独列。"""
row = model_row(spec.model)
board = board_for_march(spec.march)
return {
"schema": "bench.job/1",
"run_id": run_id,
"job_id": spec.job_id,
"task": spec.task, "model": spec.model, "march": spec.march,
"backend": spec.backend, "qcfg": spec.qcfg, "priority": spec.priority,
"rollout_plans": plans or [], # [{mode, policy, k}]
"model_spec": row,
"board": board,
"data": data_snapshot(spec),
"gates": gate_snapshot(spec),
"calib_source_ladder": spec.overrides.get("calib_source_ladder"),
"prompt": spec.overrides.get("prompt"),
"evaluator_version": evaluator_version,
"registry_fingerprint": registry_fingerprint(),
}

59
harness/report.py Normal file
View File

@ -0,0 +1,59 @@
"""bench report:从 jobs.sqlite 生成静态 HTML 到 reports/<run_id>.html,给不上机器的人看。"""
from __future__ import annotations
import html
import time
from . import paths, db
CSS = """
body{font:14px/1.5 system-ui,sans-serif;margin:24px;color:#111;background:#fafafa}
h1{font-size:20px}h2{font-size:16px;margin-top:28px}
table{border-collapse:collapse;background:#fff;font-variant-numeric:tabular-nums}
th,td{border:1px solid #ddd;padding:4px 8px;text-align:left;font-size:13px}
th{background:#f0f0f0}code{font-family:ui-monospace,monospace;font-size:12px}
.PASS{color:#2f6b46}.FAIL{color:#9e3223}.UNVERIFIED{color:#8a6516}
"""
def render(run_id: str) -> str:
with db.connect() as con:
run = db.get_run(con, run_id)
jobs = db.list_jobs(con, run_id, mode="oracle")
ec = db.error_class_counts(con, run_id)
ro = con.execute("SELECT * FROM rollouts WHERE run_id=? ORDER BY job_id, kk", (run_id,)).fetchall()
by_state: dict = {}
for j in jobs:
by_state[j["state"]] = by_state.get(j["state"], 0) + 1
out = [f"<!doctype html><meta charset=utf-8><title>bench {run_id}</title><style>{CSS}</style>",
f"<h1>bench · {html.escape(run_id)}</h1>",
f"<p>generated {time.strftime('%Y-%m-%d %H:%M:%S')} · jobs {len(jobs)} · "
+ " · ".join(f"<span class={s}>{s} {n}</span>" for s, n in by_state.items()) + "</p>"]
if ec:
out.append("<h2>failures by error_class</h2><table><tr><th>error_class</th><th>stage</th><th>n</th></tr>")
out += [f"<tr><td>{html.escape(str(r['error_class']))}</td><td>{html.escape(str(r['stage_failed'] or ''))}</td><td>{r['n']}</td></tr>" for r in ec]
out.append("</table>")
out.append("<h2>jobs</h2><table><tr><th>job_id</th><th>pri</th><th>state</th><th>error_class</th><th>stage</th><th>est s</th><th>G3 Δ</th><th>lat ms</th></tr>")
import json
for j in jobs:
r = json.loads(j["result"]) if j.get("result") else {}
g = r.get("gates") or {}
d = (g.get("G3") or {}).get("delta")
lat = (g.get("G4") or {}).get("latency_ms_mean")
out.append(f"<tr><td><code>{html.escape(j['job_id'])}</code></td><td>{j['priority']}</td>"
f"<td class={j['state']}>{j['state']}</td><td>{html.escape(j['error_class'] or '')}</td>"
f"<td>{html.escape(j['stage_failed'] or '')}</td><td>{int(j['estimate_s'] or 0)}</td>"
f"<td>{'' if d is None else f'{d:+.2f}'}</td><td>{'' if lat is None else f'{lat:.2f}'}</td></tr>")
out.append("</table>")
if ro:
out.append("<h2>rollouts</h2><table><tr><th>job_id</th><th>kk</th><th>mode</th><th>policy</th><th>state</th><th>claim</th><th>consistent</th><th>review</th><th>turns</th><th>tokens</th></tr>")
for r in ro:
out.append(f"<tr><td><code>{html.escape(r['job_id'])}</code></td><td>{r['kk']:02d}</td><td>{r['mode']}</td><td>{r['policy'] or ''}</td>"
f"<td class={r['state']}>{r['state']}</td><td>{r['claim_terminal'] or ''}</td><td>{r['claim_consistent']}</td>"
f"<td>{'⚠' if r['review'] else ''}</td><td>{r['turns'] or ''}</td><td>{r['tokens'] or ''}</td></tr>")
out.append("</table>")
html_text = "\n".join(out)
paths.REPORTS.mkdir(parents=True, exist_ok=True)
p = paths.REPORTS / f"{run_id}.html"
p.write_text(html_text)
return str(p)

165
harness/rollout.py Normal file
View File

@ -0,0 +1,165 @@
"""rollout 驱动器:起一个 pi 进程对一个 job 作答一次,然后独立核验。
bench rollout <job_id> --kk 0 --mode e2e --policy teacher
bench agent setup # 从 .env.local 渲染 ~/.pi/agent/models.json
每次 rollout 一个目录、全新上下文、看不到其他 rollout。agent 进程环境里没有任何凭证;
LLM key 在 ~/.pi/agent/models.json(600),板卡凭证只在 harness 的 07-board 子进程里。
"""
from __future__ import annotations
import json
import os
import shutil
import time
from pathlib import Path
from . import paths, db, watchdog
from .registry import gates as gates_cfg
from .stages.s02_baseline import baseline_dir
from .stages.s03_export import onnx_path
from .verify import verify_rollout
def rollout_dir(run_id: str, job_id: str, kk: int) -> Path:
return paths.RUNS / run_id / job_id / "rollouts" / f"{kk:02d}"
def _render(tmpl: str, **kw) -> str:
out = tmpl
for k, v in kw.items():
out = out.replace("{" + k + "}", "" if v is None else str(v))
return out
def setup_models_json(force: bool = False) -> Path:
"""渲染 ~/.pi/agent/models.json。key 只从 .env.local 来。"""
dst = Path(os.path.expanduser("~/.pi/agent/models.json"))
if dst.exists() and not force:
return dst
cfg = paths.config()["endpoints"]
tmpl = (paths.AGENT / "models.json.tmpl").read_text()
text = _render(tmpl,
TEACHER_API_KEY=paths.secret(cfg["teacher"]["api_key_env"]) or "",
STUDENT_BASE_URL=cfg["student"].get("base_url") or "http://127.0.0.1:8000/v1",
STUDENT_API_KEY=paths.secret(cfg["student"]["api_key_env"]) or "EMPTY",
STUDENT_MODEL=cfg["student"].get("model") or "student")
dst.parent.mkdir(parents=True, exist_ok=True)
dst.write_text(text)
os.chmod(dst, 0o600)
return dst
def prepare(run_id: str, job_id: str, kk: int, mode: str, policy: str) -> tuple[Path, dict, str]:
jd = paths.RUNS / run_id / job_id
job = json.loads((jd / "job.json").read_text())
rd = rollout_dir(run_id, job_id, kk)
if (rd / "rollout.json").exists():
raise SystemExit(f"rollout 已存在(append-only):{rd}。想再答一次就用下一个 kk。")
for d in ("workspace", "agent", "stages", "handles", "workspace/.pi"):
(rd / d).mkdir(parents=True, exist_ok=True)
# 前置:model-level 产物必须已在 cache 里(由 oracle 的 01–03 产出)
onnx = onnx_path(job)
bdir = baseline_dir(job)
missing = [str(p) for p in (onnx, bdir / "metrics.json") if not p or not p.exists()]
oracle = jd / "oracle" / "result.json"
if mode == "repair" and not oracle.exists():
missing.append(f"{oracle}(repair 需要 oracle 失败现场)")
if missing:
raise SystemExit("rollout 前置缺失,先跑 oracle(至少 01–03):\n " + "\n ".join(missing))
cfg = paths.config()
ep = cfg["endpoints"][policy]
settings = _render((paths.AGENT / "settings.json.tmpl").read_text(),
PROJECT_ROOT=str(paths.PROJECT_ROOT), BENCH_ROOT=str(paths.BENCH_ROOT), MODEL=ep["model"])
sj = json.loads(settings)
sj.pop("_comment", None)
sj["defaultProvider"] = "dr-gateway" if policy == "teacher" else "student-vllm"
sj["tools"] = cfg["rollouts"]["tools"]
(rd / "workspace" / ".pi" / "settings.json").write_text(json.dumps(sj, indent=2, ensure_ascii=False))
# prompt
tname = job.get("prompt") or (f"{mode}.md")
tmpl = (paths.AGENT / "prompts" / tname).read_text()
g = job["gates"]
ores = json.loads(oracle.read_text()) if oracle.exists() else {}
o_stage = ores.get("stage_failed") or ""
prompt = _render(
tmpl, job_id=job_id, run_id=run_id, model=job["model"], task=job["task"], march=job["march"],
backend=job["backend"], qcfg=job["qcfg"], job_json=str(jd / "job.json"), data_dir=str(jd / "data"),
input_spec=job["model_spec"]["input_spec"], onnx=str(onnx), baseline_dir=str(bdir),
calib_dir=job["data"]["calib"]["path"], calib_n=job["data"]["calib"]["n"],
t1_tar=job["data"]["t1"]["path"], t1_n=job["data"]["t1"]["n"],
g2_threshold=g["G2"].get("threshold"), g3_metric=g["G3"].get("metric"), g3_tol=g["G3"].get("tol"),
g3_kind=g["G3"].get("kind"), max_turns=cfg["rollouts"]["max_turns"], max_tokens=cfg["rollouts"]["max_tokens"],
oracle_stage_failed=o_stage, oracle_error_class=ores.get("error_class"),
oracle_stage_dir=str(jd / "oracle" / "stages" / o_stage) if o_stage else "",
oracle_compile_yaml=str(jd / "oracle" / "stages" / "04-calibrate" / "compile.yaml"),
)
(rd / "agent" / "prompt.md").write_text(prompt)
# 系统提示快照:pi 的 --append-system-prompt 内容不会进 agent_end.messages[],
# 导出时要靠它补回 SFT 样本的 system 消息;SYSTEM.md 以后改了也不影响已采的轨迹。
shutil.copy2(paths.AGENT / "SYSTEM.md", rd / "agent" / "system.md")
(rd / "rollout.json").write_text(json.dumps({"schema": "bench.rollout/1", "run_id": run_id, "job_id": job_id,
"kk": kk, "mode": mode, "policy": policy, "model": ep["model"],
"state": "RUNNING", "started_at": time.time()}, indent=2))
with db.connect() as con:
db.rollout_upsert(con, run_id, job_id, kk, mode, policy)
db.rollout_set(con, run_id, job_id, kk, state="RUNNING", started_at=time.time())
return rd, job, prompt
def agent_env(rd: Path) -> dict:
"""agent 进程环境:只有路径与预算,没有凭证。"""
cfg = paths.config()
path_parts = [str(paths.BENCH_ROOT / "bin")]
if cfg["paths"].get("node_bin"):
path_parts.append(str(Path(paths.expand(cfg["paths"]["node_bin"])).parent))
path_parts.append(os.environ.get("PATH", "/usr/bin:/bin"))
return {
"PATH": ":".join(path_parts),
"HOME": os.environ.get("HOME", "/tmp"),
"LANG": "C.UTF-8",
"TERM": "dumb",
"BENCH_ROOT": str(paths.BENCH_ROOT),
"BENCH_ROLLOUT_DIR": str(rd),
"BENCH_MAX_TURNS": str(cfg["rollouts"]["max_turns"]),
"BENCH_MAX_TOKENS": str(cfg["rollouts"]["max_tokens"]),
"PYTHONPATH": str(paths.BENCH_ROOT),
}
def pi_command(rd: Path, prompt: str) -> list[str]:
cfg = paths.config()
pi = str(paths.expand(cfg["paths"].get("pi_bin") or "pi")) if "/" in (cfg["paths"].get("pi_bin") or "") else (cfg["paths"].get("pi_bin") or "pi")
# 实测 pi-coding-agent 0.73.1(pi --help):--mode json 单独就是非交互 JSON 事件流;没有 -a 这个参数;
# --append-system-prompt 接受文本或文件路径;--tools 是逗号分隔白名单;extensions/skills 由 workspace/.pi/settings.json 发现。
return [pi, "--mode", "json", "--session-dir", str(rd / "agent"),
"--append-system-prompt", str(rd / "agent" / "system.md"), # 用 rollout 内的快照,与导出补回的 system 消息逐字一致
"--tools", ",".join(cfg["rollouts"]["tools"]),
prompt]
def run_rollout(run_id: str, job_id: str, kk: int, mode: str, policy: str, dry_run: bool = False) -> dict:
setup_models_json()
rd, job, prompt = prepare(run_id, job_id, kk, mode, policy)
cmd = pi_command(rd, prompt)
(rd / "agent" / "cmd.json").write_text(json.dumps({"cmd": cmd, "env_keys": sorted(agent_env(rd))}, indent=2, ensure_ascii=False))
if dry_run:
print(f"[dry-run] rollout 已准备:{rd}\n cwd={rd / 'workspace'}\n cmd={' '.join(cmd[:8])} …")
return {"rollout_dir": str(rd), "dry_run": True}
cfg = paths.config()["rollouts"]
r = watchdog.run(cmd, cwd=rd / "workspace", stdout=rd / "agent" / "events.jsonl", stderr=rd / "agent" / "pi.stderr.log",
env=agent_env(rd), stall_s=cfg["hard_ceiling_s"], poll_s=30, hard_ceiling_s=cfg["hard_ceiling_s"])
rj = json.loads((rd / "rollout.json").read_text())
if r.state == "STALLED" or r.exit_code == 75:
rj["state"] = "TIMEOUT"; rj.setdefault("timeout_reason", r.stalled_reason or "budget")
elif rj.get("state") != "TIMEOUT":
rj["state"] = "DONE"
rj.update({"exit_code": r.exit_code, "wall_s": round(r.duration_s, 1), "finished_at": time.time()})
(rd / "rollout.json").write_text(json.dumps(rj, indent=2))
with db.connect() as con:
db.rollout_set(con, run_id, job_id, kk, state="VERIFYING")
v = verify_rollout(rd)
return {"rollout_dir": str(rd), "terminal": v["terminal"], "claim_consistent": v["claim_consistent"], "review": v["review"]}

150
harness/scheduler.py Normal file
View File

@ -0,0 +1,150 @@
"""资源池、板卡锁、耗时估算、调度顺序。
资源池是**跨进程**的:每个池 N 个槽位,每个槽位一个 flock 文件(state/locks/<pool>-<i>.lock)。
oracle 流水线、agent 的 `bench stage run`、多个并行的 `bench run` 都通过同一组锁文件分配槽位,
所以不会因为开了两个进程就把 cpu 池当成 16 个、把板子当成两块。
gpu 池的槽位号即 GPU 号(CUDA_VISIBLE_DEVICES)。board 池只有 1 个槽 = 严格串行。
estimate 只用于排序与告警,永远不用于杀进程。
注意:锁文件所在的 state 目录必须是所有进程共用的同一个(BENCH_STATE_DIR 要一致)。
"""
from __future__ import annotations
import fcntl
import os
import time
from contextlib import contextmanager
from pathlib import Path
from . import paths
from .registry import budgets, model_row
def _lock_dir() -> Path:
d = paths.STATE / "locks"
d.mkdir(parents=True, exist_ok=True)
return d
def pool_size(pool: str) -> int:
return int(paths.config()["pools"].get(pool, {}).get("workers", 1))
def pool_reserve(pool: str) -> int:
"""给 agent 预留的槽位数:oracle 拿不到这些槽,rollout 的 `bench stage run` 优先拿。
实测(2026-09-15):两个 oracle 调度进程把 cpu 池 8/8 占满,第一条 rollout 的 05-compile 在队列里饿了 60 分钟。
轨迹是这个工厂的产品,benchmark 只是副产品——不能让 oracle 把 agent 饿死。"""
n = pool_size(pool)
r = int(paths.config()["pools"].get(pool, {}).get("reserve_for_agent", 0) or 0)
return max(0, min(r, n - 1))
def _try_slots(pool: str, order: list[int]):
d = _lock_dir()
for i in order:
f = open(d / f"{pool}-{i}.lock", "w")
try:
fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
f.write(f"{os.getpid()} {time.time()}\n"); f.flush()
return f, i
except BlockingIOError:
f.close()
return None, None
@contextmanager
def acquire(pool: str, poll_s: float = 2.0, lane: str = "normal"):
"""拿一个池槽位(跨进程)。yield 槽位号(gpu 池 = 槽位 → devices[slot])。
lane="normal"(oracle)只用 [reserve, n);lane="agent"(rollout 的 stage)可用全部,且先抢保留槽 [0, reserve)。
"""
n, reserve = pool_size(pool), pool_reserve(pool)
order = (list(range(reserve)) + list(range(reserve, n))) if lane == "agent" else list(range(reserve, n))
fh = slot = None
while fh is None:
fh, slot = _try_slots(pool, order)
if fh is None:
time.sleep(poll_s)
try:
yield slot
finally:
fcntl.flock(fh, fcntl.LOCK_UN)
fh.close()
@contextmanager
def model_lock(model: str):
"""model-level 阶段(01–03)按模型互斥:同一模型的三个 march job 并行时只有一个真跑,其余看到缓存 SKIPPED。"""
f = open(_lock_dir() / f"model-{model}.lock", "w")
fcntl.flock(f, fcntl.LOCK_EX)
try:
yield
finally:
fcntl.flock(f, fcntl.LOCK_UN)
f.close()
def pool_usage() -> dict[str, int]:
"""每个池当前被占的槽位数(bench status 用)。"""
out = {}
d = _lock_dir()
for pool in paths.config()["pools"]:
busy = 0
for i in range(pool_size(pool)):
p = d / f"{pool}-{i}.lock"
if not p.exists():
continue
f = open(p, "w")
try:
fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
fcntl.flock(f, fcntl.LOCK_UN)
except BlockingIOError:
busy += 1
finally:
f.close()
out[pool] = busy
return out
# ---------------------------------------------------------------- 估算
def _params_m(model: str) -> float | None:
r = model_row(model)
try:
return float(r.get("params_m") or "")
except ValueError:
return None
def estimate_stage(con, model: str, task: str, stage: str) -> float:
from .db import stage_history_by_model
hist = stage_history_by_model(con, model, stage)
if len(hist) >= 2:
hist = sorted(hist)
return hist[min(len(hist) - 1, int(0.95 * len(hist)))]
if len(hist) == 1:
return hist[0]
b = budgets()
pm = _params_m(model) or 30.0
key = stage.split("-", 1)[1]
row = next((r for r in b["by_params_m"] if r["max"] is None or pm <= r["max"]), b["by_params_m"][-1])
base = row.get(key, 300)
return base * b["task_multiplier"].get(task, 1.0)
def estimate_job(con, model: str, task: str) -> float:
return sum(estimate_stage(con, model, task, s) for s in ("05-compile", "07-board", "08-eval"))
def p95_alert_threshold(con, model: str, stage: str) -> float | None:
from .db import stage_history_by_model
hist = stage_history_by_model(con, model, stage)
if len(hist) < 3:
return None
hist = sorted(hist)
return 3 * hist[int(0.95 * (len(hist) - 1))]
def order_jobs(jobs: list[dict]) -> list[dict]:
"""A/B/C 优先级,同级内估算短的先;C 类单独排最后(低优先级车道)。"""
pri = {"A": 0, "B": 1, "C": 2}
return sorted(jobs, key=lambda j: (pri.get(j.get("priority"), 3), j.get("estimate_s") or 1e9, j["job_id"]))

161
harness/stagerun.py Normal file
View File

@ -0,0 +1,161 @@
"""agent 可用的 `bench stage run / wait / status`。
agent 不在 bash 里直接跑 hb_compile。它提交:
bench stage run compile --config workspace/compile.yaml → 打印 handle
bench stage wait <handle> --poll 60 → 阻塞到完成,打印结果与日志位置
harness 在 agent 进程外执行阶段:按 backend 选 venv、拿资源池、看门狗、日志落 rollout/stages/。
四个好处:编译时长与 pi 的 bash 超时脱钩;venv 不可能选错;预算与看门狗一视同仁;日志位置统一。
rollout 上下文由 env BENCH_ROLLOUT_DIR 给定(rollout 驱动器设置);agent 无法指向别的 job。
"""
from __future__ import annotations
import json
import os
import subprocess
import sys
import time
import uuid
from pathlib import Path
from . import paths, db, scheduler
from .stages import BY_NAME, AGENT_RUNNABLE, StageContext, short
def rollout_dir() -> Path:
d = os.environ.get("BENCH_ROLLOUT_DIR")
if not d:
raise SystemExit("BENCH_ROLLOUT_DIR 未设:bench stage 只能在 rollout 内使用")
return Path(d).resolve()
def _ctx_from_rollout(rd: Path, agent_config: Path | None) -> StageContext:
jd = rd.parent.parent
job = json.loads((jd / "job.json").read_text())
kk = int(rd.name)
return StageContext(run_id=job["run_id"], job=job, job_dir=jd, stage_root=rd / "stages",
artifacts=rd / "workspace" / "artifacts", rollout=kk, agent_config=agent_config)
def submit(stage: str, config: str | None, poll_hint: int = 60) -> str:
rd = rollout_dir()
name = short(stage)
if name not in AGENT_RUNNABLE:
raise SystemExit(f"阶段 {name} 不允许由 agent 提交(可用:{sorted(AGENT_RUNNABLE)})")
cfg_path = None
if config:
cfg_path = Path(config).resolve()
ws = (rd / "workspace").resolve()
if ws not in cfg_path.parents:
raise SystemExit(f"--config 必须在 workspace/ 内:{cfg_path}")
if not cfg_path.exists():
raise SystemExit(f"--config 不存在:{cfg_path}")
hid = f"{name}-{time.strftime('%H%M%S')}-{uuid.uuid4().hex[:6]}"
hd = rd / "handles" / hid
hd.mkdir(parents=True, exist_ok=True)
(hd / "request.json").write_text(json.dumps({"stage": name, "config": str(cfg_path) if cfg_path else None,
"submitted_at": time.time()}, indent=2))
(hd / "status.json").write_text(json.dumps({"state": "QUEUED"}))
# 脱离 agent 进程;环境里去掉凭证(07-board 自己从 .env.local 取)
env = {k: v for k, v in os.environ.items() if not k.endswith(("_API_KEY", "_TOKEN"))}
env["BENCH_ROOT"] = str(paths.BENCH_ROOT)
env["PYTHONPATH"] = str(paths.BENCH_ROOT)
py = paths.config()["paths"].get("harness_python", "python3")
with open(hd / "runner.log", "ab") as lf:
subprocess.Popen([py, "-m", "harness.stagerun", "--exec", str(hd)], cwd=str(paths.BENCH_ROOT), env=env,
stdout=lf, stderr=subprocess.STDOUT, start_new_session=True)
print(hid)
print(f"# wait: bench stage wait {hid} --poll {poll_hint}")
return hid
def _exec(hd: Path) -> int:
rd = hd.parent.parent
req = json.loads((hd / "request.json").read_text())
ctx = _ctx_from_rollout(rd, Path(req["config"]) if req.get("config") else None)
S = BY_NAME[req["stage"]]
stage = S()
# attempt = 该 rollout 内此阶段已提交次数(agent 自己的重试是同一条轨迹)
prev = sorted((rd / "stages" / stage.name).glob("a*")) if (rd / "stages" / stage.name).exists() else []
ctx.attempt = len(prev)
# 06-hbm 需要 04 的 calib_source 传给 08
cm = rd / "stages" / "04-calibrate" / "meta.json"
if cm.exists():
try:
ctx.extra["calib_source"] = json.loads(cm.read_text()).get("calib_source")
except Exception:
pass
# QUEUED = 已提交、在等池槽位;RUNNING = 真的在跑。两者分开,否则看板上分不出"排队 60 分钟"和"编译 60 分钟"。
(hd / "status.json").write_text(json.dumps({"state": "QUEUED", "pool": stage.pool, "attempt": ctx.attempt,
"queued_at": time.time()}))
with db.connect() as con:
db.stage_start(con, ctx.run_id, ctx.job["job_id"], stage.name, ctx.attempt, ctx.rollout, lane="queued")
with scheduler.acquire(stage.pool, lane="agent") as slot: # agent 优先拿保留槽,不被 oracle 饿死
(hd / "status.json").write_text(json.dumps({"state": "RUNNING", "pool": stage.pool, "slot": slot,
"attempt": ctx.attempt, "started_at": time.time()}))
with db.connect() as con: # 拿到槽位才算真的开始跑:看板据此区分"排队"与"在算"
con.execute("UPDATE stages SET lane='agent', started_at=? WHERE run_id=? AND job_id=? AND rollout=? AND stage=? AND attempt=?",
(time.time(), ctx.run_id, ctx.job["job_id"], ctx.rollout, stage.name, ctx.attempt))
ctx.extra["slot"] = slot
out = stage.run(ctx)
with db.connect() as con:
db.stage_finish(con, ctx.run_id, ctx.job["job_id"], stage.name, ctx.attempt, out.state, out.exit_code, out.error_class, ctx.rollout)
sd = rd / "stages" / stage.name / f"a{ctx.attempt:02d}"
status = {"state": "DONE", "result": out.state, "exit_code": out.exit_code, "error_class": out.error_class,
"duration_s": round(out.duration_s, 1), "attempt": ctx.attempt,
"stdout": str(sd / "stdout.log"), "stderr": str(sd / "stderr.log"), "meta": str(sd / "meta.json"),
"measurements": out.measurements, "finished_at": time.time()}
(hd / "status.json").write_text(json.dumps(status, indent=2, ensure_ascii=False, default=str))
return 0
def wait(handle: str, poll: int = 60, quiet: bool = False, max_wait_s: int | None = None) -> int:
"""阻塞到该阶段结束。每轮打一行心跳(排队中 / 运行中 + 已耗时),这样 agent 侧看得见进展,
pi 的 bash 工具也不会因为"长时间无输出"而误判。max_wait_s 到点就返回 2 并提示继续等(不是失败)。"""
hd = rollout_dir() / "handles" / handle
if not hd.exists():
raise SystemExit(f"handle 不存在:{handle}")
t0 = time.time()
while True:
st = json.loads((hd / "status.json").read_text())
if st.get("state") == "DONE":
break
if not quiet:
since = st.get("started_at") or st.get("queued_at") or t0
print(f"[{time.strftime('%H:%M:%S')}] {handle}: {st.get('state')}"
f"{'(等 ' + st.get('pool', '?') + ' 池槽位)' if st.get('state') == 'QUEUED' else ''}"
f" 已 {int(time.time() - since)}s", flush=True)
if max_wait_s and time.time() - t0 > max_wait_s:
print(f"(已等 {int(time.time() - t0)}s 仍未结束;这不是失败,再调一次 bench stage wait {handle} 继续等)")
return 2
time.sleep(poll)
_print_status(st)
return 0 if st["result"] in ("OK", "SKIPPED") else 1
def status(handle: str) -> int:
hd = rollout_dir() / "handles" / handle
st = json.loads((hd / "status.json").read_text())
_print_status(st)
return 0
def _print_status(st: dict) -> None:
if st.get("state") != "DONE":
print(json.dumps(st)); return
print(f"result: {st['result']} exit_code: {st['exit_code']} error_class: {st['error_class']} {st['duration_s']}s")
print(f"stdout: {st['stdout']}\nstderr: {st['stderr']}\nmeta: {st['meta']}")
if st.get("measurements"):
print("measurements: " + json.dumps(st["measurements"], ensure_ascii=False))
if st["result"] != "OK":
try:
tail = Path(st["stderr"]).read_text(errors="replace").splitlines()[-30:]
print("--- stderr tail ---"); print("\n".join(tail))
except OSError:
pass
if __name__ == "__main__":
if len(sys.argv) == 3 and sys.argv[1] == "--exec":
sys.exit(_exec(Path(sys.argv[2])))
sys.exit(2)

View File

@ -0,0 +1,25 @@
"""01–03 model-level(进 cache/,每模型一次);04–08 target-level(每 job 一遍)。"""
from .base import Stage, StageContext, StageOutcome
from .s01_fetch import Fetch
from .s02_baseline import Baseline
from .s03_export import Export
from .s04_calibrate import Calibrate
from .s05_compile import Compile
from .s06_hbm import Hbm
from .s07_board import Board
from .s08_eval import Eval
ORDER: list[type[Stage]] = [Fetch, Baseline, Export, Calibrate, Compile, Hbm, Board, Eval]
BY_NAME: dict[str, type[Stage]] = {s.name: s for s in ORDER}
# agent 通过 `bench stage run` 可提交的阶段(重阶段由 harness 执行)
AGENT_RUNNABLE = {"04-calibrate", "05-compile", "06-hbm", "07-board", "08-eval"}
def short(name: str) -> str:
"""'compile' → '05-compile';已带前缀则原样。"""
if name in BY_NAME:
return name
for k in BY_NAME:
if k.split("-", 1)[1] == name:
return k
raise KeyError(f"未知阶段: {name}")

149
harness/stages/base.py Normal file
View File

@ -0,0 +1,149 @@
"""Stage 基类:每阶段独立子目录,落 stdout / stderr / exit_code / meta.json / 耗时。
oracle 与 rollout 的 `bench stage run` 走同一份 Stage 代码,区别只在 ctx.stage_root 与 ctx.rollout。
"""
from __future__ import annotations
import json
import os
import time
from dataclasses import dataclass, field
from pathlib import Path
from .. import paths, watchdog
from ..errors import classify
@dataclass
class StageContext:
run_id: str
job: dict # job.json
job_dir: Path # runs/<run_id>/<job_id>
stage_root: Path # …/oracle/stages 或 …/rollouts/kk/stages
artifacts: Path # …/oracle/artifacts 或 …/rollouts/kk/workspace/artifacts
rollout: int = -1 # -1 = oracle
attempt: int = 0
lane: str = "normal"
dry_run: bool = False
agent_config: Path | None = None # rollout 提交的配置(workspace/x.yaml),oracle 为 None
extra: dict = field(default_factory=dict)
@property
def cfg(self) -> dict:
return paths.config()
def watchdog_params(self) -> dict:
wd = dict(self.cfg["watchdog"])
from ..registry import budgets
ov = (budgets().get("watchdog_overrides") or {}).get(self.job["task"]) or {}
wd.update(ov)
if self.lane == "long":
wd["stall_s"] = max(wd["stall_s"], 3600)
return wd
@dataclass
class StageOutcome:
state: str # OK | FAIL | STALLED | SKIPPED
exit_code: int | None = None
error_class: str | None = None
duration_s: float = 0.0
meta: dict = field(default_factory=dict)
measurements: dict = field(default_factory=dict) # 贡献给 gates.evaluate 的片段,如 {"G1": {...}}
class Stage:
name: str = "00-none"
level: str = "target" # model | target
pool: str = "cpu" # gpu | cpu | board
retry_transient: bool = True
# ------------------------------------------------------------ 子类实现
def cached(self, ctx: StageContext) -> dict | None:
"""model-level 阶段:已有缓存则返回 meta(跳过执行)。"""
return None
def command(self, ctx: StageContext) -> list[str] | None:
"""返回要执行的命令;None 表示本阶段是纯 python(重写 native)。"""
return None
def env(self, ctx: StageContext) -> dict:
"""子进程环境。默认继承但去掉凭证;07-board 自己加板卡变量。"""
env = {k: v for k, v in os.environ.items() if not k.endswith(("_API_KEY", "_TOKEN")) and not k.startswith("BOARD_")}
env["BENCH_ROOT"] = str(paths.BENCH_ROOT)
env["BENCH_JOB_DIR"] = str(ctx.job_dir)
env["PYTHONPATH"] = str(paths.BENCH_ROOT) + (":" + env["PYTHONPATH"] if env.get("PYTHONPATH") else "")
slot = ctx.extra.get("slot")
if self.pool == "gpu" and slot is not None:
devices = paths.config()["pools"]["gpu"].get("devices") or list(range(paths.config()["pools"]["gpu"]["workers"]))
env["CUDA_VISIBLE_DEVICES"] = str(devices[slot % len(devices)]) # 槽位 → 指定物理卡
elif self.pool != "gpu":
env["CUDA_VISIBLE_DEVICES"] = "" # cpu / board 阶段不碰 GPU
return env
def native(self, ctx: StageContext, stage_dir: Path) -> StageOutcome:
raise NotImplementedError
def measure(self, ctx: StageContext, stage_dir: Path, outcome: StageOutcome) -> dict:
"""执行完成后提取测量值片段。默认无。"""
return {}
# ------------------------------------------------------------ 框架
def stage_dir(self, ctx: StageContext) -> Path:
d = ctx.stage_root / self.name / f"a{ctx.attempt:02d}"
d.mkdir(parents=True, exist_ok=True)
return d
def run(self, ctx: StageContext) -> StageOutcome:
sd = self.stage_dir(ctx)
t0 = time.time()
meta = {"stage": self.name, "level": self.level, "pool": self.pool, "attempt": ctx.attempt,
"lane": ctx.lane, "rollout": ctx.rollout, "started_at": t0, "dry_run": ctx.dry_run}
try:
c = self.cached(ctx)
if c is not None and not ctx.dry_run:
out = StageOutcome("SKIPPED", 0, None, 0.0, {"cached": c})
elif ctx.dry_run:
try:
would = self.command(ctx)
except Exception as e: # 上游缓存不存在时 command() 会抛;dry-run 只记录
would = f"<command unavailable in dry-run: {e}>"
out = StageOutcome("OK", 0, None, 0.0, {"dry_run": True, "would_run": would})
else:
cmd = self.command(ctx)
if cmd is None:
out = self.native(ctx, sd)
else:
wd = ctx.watchdog_params()
r = watchdog.run(cmd, cwd=sd, stdout=sd / "stdout.log", stderr=sd / "stderr.log",
env=self.env(ctx), stall_s=wd["stall_s"], cpu_idle_pct=wd["cpu_idle_pct"],
poll_s=wd["poll_s"], hard_ceiling_s=wd["hard_ceiling_s"])
ec = None
if r.state != "OK":
ec = classify(_tail(sd / "stderr.log"), _tail(sd / "stdout.log"), r.exit_code)
out = StageOutcome(r.state, r.exit_code, ec, r.duration_s,
{"cmd": cmd, "stalled": r.stalled_reason,
"io_blocked_s": getattr(r, "io_blocked_s", 0.0)})
if out.state == "OK":
# native() 可能已经放了测量值(06-hbm 的 G1);measure() 只在有返回时覆盖。踩过:这里曾把 G1 抹成 {} → job UNVERIFIED
out.measurements = self.measure(ctx, sd, out) or out.measurements or {}
except Exception as e: # harness 自身的 bug 也要落盘,不能吞
(sd / "stderr.log").open("a").write(f"\n[harness exception] {type(e).__name__}: {e}\n")
out = StageOutcome("FAIL", -1, "unknown", time.time() - t0, {"harness_exception": repr(e)})
out.duration_s = out.duration_s or (time.time() - t0)
meta.update({"state": out.state, "exit_code": out.exit_code, "error_class": out.error_class,
"duration_s": round(out.duration_s, 3), "finished_at": time.time(),
"measurements": out.measurements, **out.meta})
(sd / "exit_code").write_text(str(out.exit_code))
(sd / "meta.json").write_text(json.dumps(meta, ensure_ascii=False, indent=2, default=str))
# 阶段级 meta.json = 最新 attempt 的副本
(ctx.stage_root / self.name / "meta.json").write_text(json.dumps(meta, ensure_ascii=False, indent=2, default=str))
return out
def _tail(p: Path, n: int = 400) -> str:
try:
lines = p.read_text(errors="replace").splitlines()
return "\n".join(lines[-n:])
except OSError:
return ""

View File

@ -0,0 +1,65 @@
"""01-fetch:权重 → cache/weights/<sha256>/。content-addressed,跨 run 复用。"""
from __future__ import annotations
import hashlib
import json
import shutil
from pathlib import Path
from .. import paths
from .base import Stage, StageContext, StageOutcome
INDEX = paths.CACHE / "weights" / "index.json"
def _index() -> dict:
return json.loads(INDEX.read_text()) if INDEX.exists() else {}
def weights_dir(model_id: str) -> Path | None:
sha = _index().get(model_id, {}).get("sha256")
return (paths.CACHE / "weights" / sha) if sha else None
def weights_file(model_id: str) -> Path | None:
d = weights_dir(model_id)
if not d:
return None
files = [p for p in d.iterdir() if p.name != "meta.json"]
return files[0] if files else None
class Fetch(Stage):
name = "01-fetch"
level = "model"
pool = "cpu"
def cached(self, ctx: StageContext):
d = weights_dir(ctx.job["model"])
return {"weights_dir": str(d)} if d and d.exists() else None
def command(self, ctx: StageContext):
tmp = paths.CACHE / "weights" / f"_tmp-{ctx.job['model']}"
tmp.mkdir(parents=True, exist_ok=True)
# torchvision:// 与 hf:// 需要 torch/hf 环境 → bench venv;http(s) 直接下
return [str(paths.venv_python("bench")), "-m", "harness.tools.fetch_weights",
"--model", ctx.job["model"], "--uri", ctx.job["model_spec"]["weights"], "--out-dir", str(tmp)]
def measure(self, ctx: StageContext, stage_dir: Path, outcome: StageOutcome) -> dict:
tmp = paths.CACHE / "weights" / f"_tmp-{ctx.job['model']}"
files = [p for p in tmp.iterdir() if p.is_file()]
if not files:
raise RuntimeError("fetch 未产出文件")
f = files[0]
sha = hashlib.sha256(f.read_bytes()).hexdigest()
dst = paths.CACHE / "weights" / sha
if not dst.exists():
dst.mkdir(parents=True)
shutil.move(str(f), dst / f.name)
(dst / "meta.json").write_text(json.dumps({"model": ctx.job["model"], "uri": ctx.job["model_spec"]["weights"],
"sha256": sha, "size": (dst / f.name).stat().st_size}, indent=2))
shutil.rmtree(tmp, ignore_errors=True)
idx = _index()
idx[ctx.job["model"]] = {"sha256": sha, "file": f.name}
INDEX.write_text(json.dumps(idx, indent=2, ensure_ascii=False))
return {"weights_sha256": sha}

View File

@ -0,0 +1,48 @@
"""02-baseline:FP32 指标(T1)+ 逐输出张量样本(G2 用)→ cache/baseline/<model>--<dataset>/。
基线自跑,不采信官方数。评测器与 08-eval 完全同一份代码(harness/eval/)。"""
from __future__ import annotations
import json
from pathlib import Path
from .. import paths
from .base import Stage, StageContext, StageOutcome
from .s01_fetch import weights_file
def baseline_dir(job: dict) -> Path:
return paths.CACHE / "baseline" / f"{job['model']}--{job['data']['dataset']}"
class Baseline(Stage):
name = "02-baseline"
level = "model"
pool = "gpu"
def cached(self, ctx: StageContext):
d = baseline_dir(ctx.job)
m = d / "metrics.json"
if m.exists():
mj = json.loads(m.read_text())
# 基线必须与当前 T1 子集、评测器版本一致,否则 delta 不可比
if mj.get("t1_manifest_sha") == ctx.job["data"]["t1"].get("content_sha") and \
mj.get("evaluator_version") == ctx.job["evaluator_version"]:
return {"baseline_dir": str(d), "metrics": mj.get("metrics")}
return None
def command(self, ctx: StageContext):
job = ctx.job
out = baseline_dir(job)
out.mkdir(parents=True, exist_ok=True)
return [str(paths.venv_python("bench")), "-m", "harness.eval.run",
"--mode", "fp32", "--task", job["task"], "--model", job["model"],
"--source", job["model_spec"]["source"], "--weights", str(weights_file(job["model"])),
"--input-spec", job["model_spec"]["input_spec"], "--resize", str(job["model_spec"].get("resize") or 0),
"--t1", job["data"]["t1"]["path"], "--t1-sha", str(job["data"]["t1"]["manifest_sha"]),
"--g2-samples", str(job["gates"]["G2"]["n_samples"]),
"--evaluator-version", job["evaluator_version"],
"--out", str(out)]
def measure(self, ctx, stage_dir, outcome) -> dict:
m = json.loads((baseline_dir(ctx.job) / "metrics.json").read_text())
return {"baseline": m.get("metrics")}

View File

@ -0,0 +1,52 @@
"""03-export:权重 → ONNX → cache/onnx/<weights_sha>/<model>.onnx。export_fail 是 model-level:一次影响该模型所有 job。"""
from __future__ import annotations
import hashlib
import json
from pathlib import Path
from .. import paths
from ..backends import get as backend
from .base import Stage, StageContext, StageOutcome
from .s01_fetch import weights_file, _index
def onnx_path(job: dict) -> Path | None:
sha = _index().get(job["model"], {}).get("sha256")
return (paths.CACHE / "onnx" / sha / f"{job['model']}.onnx") if sha else None
class Export(Stage):
name = "03-export"
level = "model"
pool = "cpu"
def cached(self, ctx: StageContext):
p = onnx_path(ctx.job)
if not (p and p.exists()):
return None
# 导出口径版本要一致,否则重导(踩过:ultralytics 旧导出留下的 ONNX 被新 run 直接复用)
from ..tools.export_onnx import EXPORTER_VERSION
meta = p.parent / "meta.json"
try:
if json.loads(meta.read_text()).get("exporter_version") != EXPORTER_VERSION:
return None
except Exception:
return None
return {"onnx": str(p)}
def command(self, ctx: StageContext):
p = onnx_path(ctx.job)
if p is None:
raise RuntimeError("01-fetch 未登记权重 sha,无法定位 onnx 缓存目录")
p.parent.mkdir(parents=True, exist_ok=True)
return backend(ctx.job["backend"]).export_cmd(ctx.job, p, weights_file(ctx.job["model"]))
def measure(self, ctx, stage_dir, outcome) -> dict:
p = onnx_path(ctx.job)
sha = hashlib.sha256(p.read_bytes()).hexdigest()
from ..tools.export_onnx import EXPORTER_VERSION
(p.parent / "meta.json").write_text(json.dumps({"model": ctx.job["model"], "onnx_sha256": sha,
"input_spec": ctx.job["model_spec"]["input_spec"],
"exporter_version": EXPORTER_VERSION}, indent=2))
return {"onnx_sha256": sha}

View File

@ -0,0 +1,109 @@
"""04-calibrate:确认校准集就位、按 occ 阶梯选来源、生成编译配置(PTQ 的 hb_compile yaml)。
纯 python,不跑工具。rollout 里 agent 提交的 --config 在这里与 harness 钉死项合并。
"""
from __future__ import annotations
import json
from pathlib import Path
import yaml
from .. import paths
from ..backends import ptq
from .base import Stage, StageContext, StageOutcome
from .s03_export import onnx_path
def calib_dir(job: dict) -> Path:
return Path(job["data"]["calib"]["path"])
def calib_npy_dir(cd: Path) -> Path:
"""hb_compile 的 cal_data_dir:只含 .npy 的子目录(manifest.json 在上一级)。"""
return cd / "npy_f32" if (cd / "npy_f32").exists() else cd
def compile_yaml_path(ctx: StageContext) -> Path:
return ctx.stage_root / "04-calibrate" / "compile.yaml"
def work_dir(ctx: StageContext) -> Path:
"""hb_compile 的 working_dir:中间产物(几个 ONNX + bc + hbm,几百 MB)都写这里。
默认放本地盘(config.yaml paths.work_root),不放 runs/。实测 2026-09-15:
JuiceFS 写 1 MB 要 360 ms,本地盘 3 ms(慢 118 倍),大模型编译会长时间卡在 D 状态,
看起来就像挂死——EfficientNet-B4/B5 因此被看门狗误杀。最终产物由 06-hbm 收进 runs/ 的 artifacts/。
work_root 为空则退回 runs/ 内(单机小规模时够用)。
"""
root = paths.config()["paths"].get("work_root")
if not root:
return ctx.stage_root / "05-compile" / "work"
rel = f"{ctx.run_id}/{ctx.job['job_id']}" + (f"/r{ctx.rollout:02d}" if ctx.rollout >= 0 else "/oracle")
return paths.expand(root) / rel / "05-compile-work"
class Calibrate(Stage):
name = "04-calibrate"
level = "target"
pool = "cpu"
def native(self, ctx: StageContext, stage_dir: Path) -> StageOutcome:
job = ctx.job
cd = calib_dir(job)
source = "t0"
ladder = job.get("calib_source_ladder")
if ladder:
# occ:repo_demo → nuscenes_128 → synthetic,逐级回退并记录用了哪级
for lvl in ladder:
cand = cd.parent / f"{cd.name}--{lvl}"
if cand.exists() and (cand / "manifest.json").exists():
cd, source = cand, lvl
break
else:
cd, source = cd.parent / f"{cd.name}--synthetic", "synthetic"
if not cd.exists():
cd.mkdir(parents=True, exist_ok=True)
_make_synthetic(cd, job["model_spec"]["input_spec"], job["data"]["calib"]["n"])
if not (cd / "manifest.json").exists():
(stage_dir / "stderr.log").write_text(f"calib manifest missing: {cd}\n")
return StageOutcome("FAIL", 2, "unknown", 0.0, {"reason": "calib_missing", "calib_dir": str(cd)})
manifest = json.loads((cd / "manifest.json").read_text())
agent_overrides = None
if ctx.agent_config and ctx.agent_config.exists():
agent_overrides = yaml.safe_load(ctx.agent_config.read_text()) or {}
(stage_dir / "agent_config.yaml").write_text(ctx.agent_config.read_text())
meta = {"calib_dir": str(cd), "calib_npy_dir": str(calib_npy_dir(cd)), "calib_source": source,
"calib_n": manifest.get("count"), "calib_manifest_sha": manifest.get("sha256")}
if job["backend"] == "ptq":
cfg = ptq.compile_config(job, onnx_path(job), calib_npy_dir(cd), work_dir(ctx), agent_overrides)
yp = compile_yaml_path(ctx)
yp.parent.mkdir(parents=True, exist_ok=True)
ptq.write_compile_config(cfg, yp)
meta["compile_yaml"] = str(yp)
(stage_dir / "stdout.log").write_text(json.dumps(meta, indent=2, ensure_ascii=False) + "\n")
(stage_dir / "stderr.log").touch()
return StageOutcome("OK", 0, None, 0.0, meta, {"calib": meta})
def _make_synthetic(out: Path, input_spec: str, n: int) -> None:
"""占据类第三级回退:输入范围内的随机张量。G2 结果只作参考(pass-weak)。"""
import hashlib
try:
import numpy as np
shape = tuple(int(x) for x in input_spec.split("x"))
rng = np.random.default_rng(20260914)
h = hashlib.sha256()
(out / "npy_f32").mkdir(exist_ok=True)
for i in range(n):
arr = rng.uniform(-1, 1, size=shape).astype("float32")
p = out / "npy_f32" / f"{i:05d}.npy"
np.save(p, arr)
h.update(p.read_bytes())
(out / "manifest.json").write_text(json.dumps({"count": n, "sha256": h.hexdigest(), "source": "synthetic",
"input_spec": input_spec}, indent=2))
except ImportError:
(out / "manifest.json").write_text(json.dumps({"count": 0, "sha256": None, "source": "synthetic",
"error": "numpy unavailable"}, indent=2))

View File

@ -0,0 +1,27 @@
"""05-compile:hb_compile(PTQ)/ plugin calib + hbdk4 compile(QAT)。CPU 密集,只有看门狗,没有超时。"""
from __future__ import annotations
from pathlib import Path
from ..backends import ptq, qat
from .base import Stage, StageContext
from .s01_fetch import weights_file
from .s04_calibrate import compile_yaml_path, work_dir, calib_dir
class Compile(Stage):
name = "05-compile"
level = "target"
pool = "cpu"
def command(self, ctx: StageContext):
wd = work_dir(ctx)
wd.mkdir(parents=True, exist_ok=True)
if ctx.job["backend"] == "ptq":
return ptq.compile_cmd(compile_yaml_path(ctx))
return qat.calib_compile_cmd(ctx.job, weights_file(ctx.job["model"]), calib_dir(ctx.job), wd, ctx.agent_config)
def env(self, ctx: StageContext) -> dict:
env = super().env(ctx)
env["OMP_NUM_THREADS"] = "8"
return env

212
harness/stages/s06_hbm.py Normal file
View File

@ -0,0 +1,212 @@
"""06-hbm:收产物、hb_model_info 读 march / 输入规格 / 静态性能估算、hb_verifier 一致性、编译日志扫 ERROR → G1 测量值。
产物命名由 job_id 决定:artifacts/<job_id>.hbm(.bc 也留一份)。march 不一致直接 fail——agent 最容易作弊的口子。
实测(hb_model_info 3.5.3):I/O 表与 march 打在 stderr(logging),静态 perf 估算(FPS / latency / DDR)打在 stdout。
实测(hb_verifier 3.5.3):-m ptq_model.onnx,quantized_model.bc -i <npy>,逐层余弦表,最后一行是模型输出。
"""
from __future__ import annotations
import json
import re
import shutil
from pathlib import Path
from .. import watchdog
from ..backends import get as backend
from ..registry import boards
from .base import Stage, StageContext, StageOutcome
from .s04_calibrate import work_dir, calib_npy_dir
ANSI = re.compile(r"\x1b\[[0-9;]*m")
def hbm_path(ctx: StageContext) -> Path:
return ctx.artifacts / f"{ctx.job['job_id']}.hbm"
def bc_path(ctx: StageContext) -> Path:
return ctx.artifacts / f"{ctx.job['job_id']}.bc"
def _read(p: Path) -> str:
return ANSI.sub("", p.read_text(errors="replace")) if p.exists() else ""
class Hbm(Stage):
name = "06-hbm"
level = "target"
pool = "cpu"
def native(self, ctx: StageContext, stage_dir: Path) -> StageOutcome:
job = ctx.job
wd = work_dir(ctx)
cands = sorted(wd.rglob("*.hbm"), key=lambda p: p.stat().st_mtime, reverse=True) if wd.exists() else []
m: dict = {"hbm_exists": bool(cands)}
if not cands:
(stage_dir / "stderr.log").write_text(f"no .hbm under {wd}\n")
return StageOutcome("FAIL", 2, "unknown", 0.0, {"reason": "no_hbm"}, {"G1": m})
ctx.artifacts.mkdir(parents=True, exist_ok=True)
dst = hbm_path(ctx)
shutil.copy2(cands[0], dst)
bc = next(iter(sorted(wd.rglob("*quantized_model.bc"))), None) or next(iter(sorted(wd.rglob("*.bc"))), None)
if bc:
shutil.copy2(bc, ctx.artifacts / f"{job['job_id']}.bc")
be = backend(job["backend"])
wdp = ctx.watchdog_params()
prefix = job["job_id"]
# ---- hb_model_info:stdout=静态 perf,stderr=I/O 表 + march
r = watchdog.run(be.model_info_cmd(dst), cwd=stage_dir, stdout=stage_dir / "model_info.log",
stderr=stage_dir / "model_info.err.log", env=self.env(ctx), stall_s=wdp["stall_s"], poll_s=5)
info = _read(stage_dir / "model_info.log") + "\n" + _read(stage_dir / "model_info.err.log")
m["model_info_ok"] = r.state == "OK"
m["march_read"] = _find_march(info)
m["march_ok"] = (m["march_read"] == job["march"]) if m["march_read"] else None
io = _parse_io(info)
m["io"] = io
m["io_ok"] = _check_io(io, job["model_spec"]["input_spec"]) if io else None
m["static_perf"] = _parse_static_perf(info)
# ---- hb_verifier(PTQ:ptq_model.onnx vs quantized_model.bc,一张校准外样本)
if job["backend"] == "ptq" and bc:
ptq_onnx = next(iter(sorted(wd.rglob("*_ptq_model.onnx"))), None)
sample = next(iter(sorted(calib_npy_dir(Path(job["data"]["calib"]["path"])).glob("*.npy"))), None)
if ptq_onnx and sample:
rv = watchdog.run(be.verifier_cmd(ptq_onnx, bc, sample), cwd=stage_dir,
stdout=stage_dir / "verifier.log", stderr=stage_dir / "verifier.err.log",
env=self.env(ctx), stall_s=wdp["stall_s"], poll_s=10)
vtext = _read(stage_dir / "verifier.log") + "\n" + _read(stage_dir / "verifier.err.log")
cos = _verifier_output_cosine(vtext)
thr = job["gates"]["G2"].get("threshold") or 0.99
m["verifier_cosine"] = cos
m["verifier_ok"] = (rv.state == "OK") and (cos is None or cos >= thr)
m["verifier_rc"] = rv.exit_code
else:
m["verifier_ok"] = None
m["verifier_note"] = "ptq_model.onnx or calib sample missing"
else:
m["verifier_ok"] = True # QAT 的一致性在 08-eval 的 G2 里判
# ---- hbm ≡ bc 抽查(08-eval 全量跑的是 .bc;这里用 2 张确认 hbm 的 run_sim 与 bc 逐位一致)
if bc and job["backend"] == "ptq":
m["hbm_bc_cos_min"] = _hbm_bc_spotcheck(ctx, stage_dir, dst, ctx.artifacts / f"{job['job_id']}.bc", be, wdp)
if m["hbm_bc_cos_min"] is not None and m["hbm_bc_cos_min"] < 0.999:
m["verifier_ok"] = False
m["verifier_note"] = f"hbm vs bc cosine {m['hbm_bc_cos_min']:.6f} < 0.999"
# ---- 编译日志扫 ERROR(只看 hb_compile 自己的日志级别,不看 INFO 里的字样)
clog = ""
for f in sorted((ctx.stage_root / "05-compile").glob("a*/stderr.log")):
clog += _read(f)
clog += _read(wd / "hb_compile.log")
# 上一次尝试的 ERROR 不算:只看最后一次 "Start hb_compile" 之后
last = clog.rfind("Start hb_compile")
tail = clog[last:] if last >= 0 else clog
m["log_has_error"] = bool(re.search(r"\bERROR\b", tail))
m["hmct_output_cosine"] = _hmct_output_cosine(tail) # hmct 校准阶段自己报的模型输出余弦(校准/量化),PTQ 崩了这里先看到
(stage_dir / "stdout.log").write_text(json.dumps(m, indent=2, ensure_ascii=False) + "\n")
(stage_dir / "stderr.log").touch()
# work 目录可能在本地盘(config paths.work_root),把它记下来,事后还能找到中间产物
return StageOutcome("OK", 0, None, 0.0,
{"hbm": str(dst), "size": dst.stat().st_size, "work_dir": str(wd)}, {"G1": m})
def _hbm_bc_spotcheck(ctx: StageContext, stage_dir: Path, hbm: Path, bc: Path, be, wdp: dict, n: int = 2) -> float | None:
"""同 n 张校准样本分别过 hbm(run_sim) 与 bc(HBIR run),返回逐输出余弦的最小值。任何一步失败返回 None(记录,不判)。"""
try:
import numpy as np
npys = sorted(calib_npy_dir(Path(ctx.job["data"]["calib"]["path"])).glob("*.npy"))[:n]
if not npys:
return None
fin = stage_dir / "spot_in.npz"
np.savez(fin, **{f"s{i:04d}": np.load(p) for i, p in enumerate(npys)})
outs = {}
for tag, model in (("hbm", hbm), ("bc", bc)):
fo = stage_dir / f"spot_{tag}.npz"
r = watchdog.run([str(be.python()), "-m", "harness.tools.infer_hbm", "--model", str(model), "--inputs", str(fin), "--out", str(fo)],
cwd=stage_dir, stdout=stage_dir / f"spot_{tag}.log", stderr=stage_dir / f"spot_{tag}.err.log",
env=self_env(ctx), stall_s=wdp["stall_s"], poll_s=5)
if r.state != "OK":
return None
outs[tag] = np.load(fo)
cos_min = 1.0
for k in outs["hbm"].files:
a = outs["hbm"][k].astype(np.float64).ravel(); b = outs["bc"][k].astype(np.float64).ravel()
na, nb = np.linalg.norm(a), np.linalg.norm(b)
cos_min = min(cos_min, float(a @ b / (na * nb)) if na and nb else 0.0)
fin.unlink(missing_ok=True)
return cos_min
except Exception:
return None
def self_env(ctx: StageContext) -> dict:
return Hbm().env(ctx)
def _find_march(info: str) -> str | None:
low = info.lower()
strings = boards().get("march_strings", {})
mm = re.search(r"march\s*[:=]?\s*\"?([a-z0-9-]+)", low)
if mm and mm.group(1).startswith("nash"):
return mm.group(1)
for march, names in strings.items():
if any(n.lower() in low for n in names):
return march
return None
def _parse_io(info: str) -> dict:
"""解析 'NAME TYPE SHAPE DATA_TYPE' 表:input input [1, 3, 224, 224] FLOAT32"""
io = {"inputs": [], "outputs": []}
for line in info.splitlines():
mm = re.search(r"(\S+)\s+(input|output)\s+\[([0-9,\s]+)\]\s+([A-Z0-9_]+)", line)
if mm:
io["inputs" if mm.group(2) == "input" else "outputs"].append(
{"name": mm.group(1), "shape": [int(x) for x in mm.group(3).split(",")], "dtype": mm.group(4)})
return io if io["inputs"] or io["outputs"] else {}
def _check_io(io: dict, input_spec: str) -> bool | None:
dims = [int(x) for x in input_spec.split("x")]
ins = io.get("inputs") or []
if not ins:
return None
return any(i["shape"] == dims for i in ins)
def _parse_static_perf(info: str) -> dict:
"""hb_model_info stdout:FPS=2318.51, latency = 431.3 us, DDR = 12709504 bytes(编译器静态估算,不是板端实测)"""
d = {}
mm = re.search(r"FPS\s*=\s*([\d.]+)", info)
if mm: d["fps_est"] = float(mm.group(1))
mm = re.search(r"latency\s*=\s*([\d.]+)\s*(us|ms)", info)
if mm: d["latency_ms_est"] = float(mm.group(1)) / (1000.0 if mm.group(2) == "us" else 1.0)
mm = re.search(r"DDR\s*=\s*(\d+)\s*bytes", info)
if mm: d["ddr_bytes_est"] = int(mm.group(1))
return d
def _hmct_output_cosine(text: str) -> dict | None:
"""hb_compile 日志末尾的表:| TensorName | Calibrated Cosine | Quantized Cosine |"""
i = text.rfind("Calibrated Cosine | Quantized Cosine")
if i < 0:
return None
rows = re.findall(r"\|\s*(\S+)\s*\|\s*([0-9.\-]+)\s*\|\s*([0-9.\-]+)\s*\|", text[i:])
if not rows:
return None
return {name: {"calibrated": float(c), "quantized": float(q)} for name, c, q in rows if _isnum(c) and _isnum(q)}
def _isnum(s: str) -> bool:
try:
float(s); return True
except ValueError:
return False
def _verifier_output_cosine(text: str) -> float | None:
"""逐层余弦表最后一行(模型输出)的余弦;找不到返回 None(只靠 rc 判)。"""
rows = re.findall(r"\|\s*\S+\s*\|\s*\S+\s*\|\s*([0-9.]+)\s*\|", text)
return float(rows[-1]) if rows else None

View File

@ -0,0 +1,54 @@
"""07-board:推 hbm 上板,hrt_model_exec perf → G4(mean 与 p95 分开存,永不互推)。
唯一带板卡凭证的阶段;凭证只进这个子进程的 env,agent 进程永远拿不到。
板卡 worker=1,锁在 scheduler 里。board_offline 归瞬态,重试后仍失败 → G4 unverified。
"""
from __future__ import annotations
import json
from pathlib import Path
from .. import paths
from .base import Stage, StageContext, StageOutcome
from .s06_hbm import hbm_path
class Board(Stage):
name = "07-board"
level = "target"
pool = "board"
def command(self, ctx: StageContext):
if not ctx.job.get("board"):
return None # → native:没有这个 march 的板 → G4 unverified,不阻塞
perf = ctx.cfg["perf"]
board = ctx.job.get("board") or {}
return [ctx.cfg["paths"].get("harness_python", "python3"), "-m", "harness.tools.board_perf",
"--hbm", str(hbm_path(ctx)), "--job-id", ctx.job["job_id"],
"--warmup", str(perf["warmup"]), "--runs", str(perf["runs"]),
"--runtime-dir", board.get("runtime_dir", "/userdata/bench"),
"--out", str(ctx.stage_root / self.name / "perf.json")]
def native(self, ctx: StageContext, stage_dir: Path) -> StageOutcome:
msg = f"no board registered for march {ctx.job['march']} (registry/boards.yaml) → G4 unverified"
(stage_dir / "stdout.log").write_text(msg + "\n"); (stage_dir / "stderr.log").touch()
return StageOutcome("SKIPPED", 0, None, 0.0, {"reason": "no_board_for_march"}, {"G4": None})
def env(self, ctx: StageContext) -> dict:
env = super().env(ctx)
for k in ("BOARD_HOST", "BOARD_PORT", "BOARD_USER", "BOARD_SSH_KEY"):
v = paths.secret(k)
if v:
env[k] = v
return env
def measure(self, ctx, stage_dir, outcome) -> dict:
p = ctx.stage_root / self.name / "perf.json"
if not p.exists():
return {"G4": None}
d = json.loads(p.read_text())
return {"G4": {"latency_ms_mean": d.get("latency_ms_mean"), "latency_ms_p95": d.get("latency_ms_p95"),
"latency_ms_p50": d.get("latency_ms_p50"), "fps_single_thread": d.get("fps_single_thread"),
"bpu_util": d.get("bpu_util"), "ddr_bw_mbps": d.get("ddr_bw_mbps"),
"board_type": d.get("board_type"), "ucp": d.get("ucp"),
"contended": d.get("contended", False)}} # 有人抢板 → 数字可疑,reward 的 lat_rel 应忽略

View File

@ -0,0 +1,42 @@
"""08-eval:量化模型在 T1 上跑同一评测器 → G2(余弦,校准外样本)+ G3(任务指标 − FP32 基线)。
评测器在 harness/eval/,agent 不可写,是 verdict 的唯一依据。
"""
from __future__ import annotations
import json
from pathlib import Path
from .. import paths
from .base import Stage, StageContext, StageOutcome
from .s02_baseline import baseline_dir
from .s06_hbm import hbm_path, bc_path
from .s03_export import onnx_path
class Eval(Stage):
name = "08-eval"
level = "target"
pool = "gpu"
def command(self, ctx: StageContext):
job = ctx.job
out = ctx.stage_root / self.name / "eval.json"
return [str(paths.venv_python("bench")), "-m", "harness.eval.run",
"--runtime-python", str(paths.venv_python(job["backend"])), # hbm 用产出它的工具链 venv 跑 x86 模拟
"--mode", "quant", "--task", job["task"], "--model", job["model"],
"--hbm", str(bc_path(ctx) if bc_path(ctx).exists() else hbm_path(ctx)), "--onnx", str(onnx_path(job)),
"--backend", job["backend"], "--input-spec", job["model_spec"]["input_spec"], "--resize", str(job["model_spec"].get("resize") or 0),
"--t1", job["data"]["t1"]["path"], "--t1-sha", str(job["data"]["t1"]["manifest_sha"]),
"--baseline", str(baseline_dir(job)),
"--g2-samples", str(job["gates"]["G2"]["n_samples"]),
"--calib-source", (ctx.extra.get("calib_source") or "t0"),
"--evaluator-version", job["evaluator_version"],
"--out", str(out)]
def measure(self, ctx, stage_dir, outcome) -> dict:
p = ctx.stage_root / self.name / "eval.json"
if not p.exists():
return {}
d = json.loads(p.read_text())
return {"G2": d.get("G2"), "G3": d.get("G3")}

View File

@ -0,0 +1 @@
"""在工具链 venv 里执行的脚本(需要 torch / hbdk / hf 等)。harness 主进程只依赖 pyyaml,靠子进程调这些。"""

View File

@ -0,0 +1,120 @@
"""在板子上跑:加载 hbm,单线程连续推理,输出时延统计 + BPU 占用采样(JSON)。
python3 board_bench.py --hbm x.hbm --warmup 20 --runs 200
实测 RDK S100P 沙箱(UCP 3.13.6,Ubuntu 22.04 容器):没有 hrt_model_exec CLI,只有 python 包 hbm_runtime:
m = hbm_runtime.HB_HBMRuntime(path); name = m.model_names[0]
m.input_names[name] / m.output_names[name] / m.input_shapes[name][input]
outs = m.run({name: {input: np.ndarray}})[name]
只依赖 numpy + hbm_runtime,板上原生有。stderr 上 UCP 的日志噪声不管,结果只看 stdout 最后一行 JSON。
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import threading
import time
def read_ratio() -> float | None:
for p in ("/sys/class/bpu/bpu_core0/ratio", "/sys/class/bpu/bpu/ratio"):
try:
return float(open(p).read().strip())
except (OSError, ValueError):
continue
return None
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--hbm", required=True)
ap.add_argument("--warmup", type=int, default=20)
ap.add_argument("--runs", type=int, default=200)
ap.add_argument("--input-npy", help="可选:真实输入 .npy(默认随机)")
a = ap.parse_args()
import numpy as np
import hbm_runtime
t_load = time.time()
m = hbm_runtime.HB_HBMRuntime(a.hbm)
name = m.model_names[0]
in_names = list(m.input_names[name])
out_names = list(m.output_names[name])
shapes = m.input_shapes[name]
types = getattr(m, "input_types", {}).get(name, {}) if hasattr(m, "input_types") else {}
load_s = time.time() - t_load
def make_input(n):
shp = tuple(int(x) for x in shapes[n])
dt = str(types.get(n, "")).lower()
if a.input_npy:
arr = np.load(a.input_npy)
return np.ascontiguousarray(arr.reshape(shp)) if arr.size == int(np.prod(shp)) else arr
if "int8" in dt or "uint8" in dt:
return np.random.randint(0, 255, size=shp, dtype=np.uint8 if "uint" in dt else np.int8)
return np.random.uniform(-1, 1, size=shp).astype(np.float32)
feed = {name: {n: make_input(n) for n in in_names}}
ratios: list[float] = []
stop = threading.Event()
def sampler():
while not stop.is_set():
r = read_ratio()
if r is not None:
ratios.append(r)
time.sleep(0.05)
for _ in range(a.warmup):
m.run(feed)
th = threading.Thread(target=sampler, daemon=True); th.start()
lat = []
t0 = time.time()
for _ in range(a.runs):
t = time.perf_counter()
m.run(feed)
lat.append((time.perf_counter() - t) * 1000.0)
wall = time.time() - t0
stop.set(); th.join(timeout=1)
lat_np = np.asarray(lat)
out = {
"model": name, "inputs": {n: list(map(int, shapes[n])) for n in in_names}, "outputs": out_names,
"input_types": {n: str(types.get(n, "?")) for n in in_names},
"latency_ms_mean": float(lat_np.mean()), "latency_ms_p50": float(np.percentile(lat_np, 50)),
"latency_ms_p95": float(np.percentile(lat_np, 95)), "latency_ms_min": float(lat_np.min()),
"fps_single_thread": a.runs / wall, "warmup": a.warmup, "runs": a.runs, "load_s": load_s,
"bpu_util": (float(np.mean(ratios)) / 100.0) if ratios else None, "bpu_ratio_samples": len(ratios),
"ucp": _ucp_version(), "board_type": _board_type(), "hostname": os.uname().nodename,
}
print(json.dumps(out))
return 0
def _ucp_version() -> str | None:
for p in ("/etc/ucp_version",):
try:
return open(p).read().strip()
except OSError:
pass
try:
import subprocess
o = subprocess.run(["python3", "-c", "import hbm_runtime"], capture_output=True, text=True, timeout=20)
import re
mm = re.search(r"UCP version = ([\d.]+)", o.stdout + o.stderr)
return mm.group(1) if mm else None
except Exception:
return None
def _board_type() -> str | None:
try:
return open("/proc/device-tree/model", "rb").read().decode(errors="replace").strip("\x00 \n")
except OSError:
return None
if __name__ == "__main__":
sys.exit(main())

View File

@ -0,0 +1,90 @@
"""07-board 子进程:scp hbm + board_bench.py 上板 → 板上 python3 跑 → 解析 JSON → perf.json。
凭证只从本进程 env 取(BOARD_HOST/PORT/USER/SSH_KEY),密钥登录。
实测 RDK S100P 沙箱:无 hrt_model_exec,用 hbm_runtime python 包计时(见 board_bench.py)。
板是共享的:开测前探测板上有没有别人的推理进程 / BPU ratio,最多等 --busy-wait-s,仍忙照测但标 contended。
"""
from __future__ import annotations
import argparse
import json
import os
import subprocess
import sys
import time
from pathlib import Path
HERE = Path(__file__).resolve().parent
def ssh_base() -> tuple[list[str], str]:
host, port, user, key = (os.environ.get(k) for k in ("BOARD_HOST", "BOARD_PORT", "BOARD_USER", "BOARD_SSH_KEY"))
user = user or "root"
if not (host and key):
print("BOARD_HOST/BOARD_SSH_KEY missing (see bench/.env.local)", file=sys.stderr)
raise SystemExit(4)
key = os.path.expanduser(key)
return ["-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=accept-new", "-o", "ConnectTimeout=20",
"-i", key, "-p", port or "22"], f"{user}@{host}"
def run(cmd: list[str], timeout: int = 1800) -> str:
r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
if r.returncode != 0:
sys.stderr.write(r.stderr[-4000:])
raise SystemExit(r.returncode) # ssh 连不上 → errors.classify 认出 board_offline
return r.stdout
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--hbm", required=True)
ap.add_argument("--job-id", required=True)
ap.add_argument("--warmup", type=int, default=20)
ap.add_argument("--runs", type=int, default=200)
ap.add_argument("--runtime-dir", default="/userdata/bench")
ap.add_argument("--busy-wait-s", type=int, default=600)
ap.add_argument("--out", required=True)
a = ap.parse_args()
opts, target = ssh_base()
scp_opts = [o if o != "-p" else "-P" for o in opts]
rdir = a.runtime_dir
remote_hbm = f"{rdir}/{a.job_id}.hbm"
run(["ssh", *opts, target, f"mkdir -p {rdir}"])
# 共享板:探测别人的推理进程 / BPU 占用
contended, waited = False, 0
while True:
probe = run(["ssh", *opts, target,
"pgrep -f 'board_bench|hrt_model_exec|hbm_runtime|dnn' | grep -v $$ | wc -l; "
"cat /sys/class/bpu/bpu_core0/ratio 2>/dev/null || cat /sys/class/bpu/bpu/ratio 2>/dev/null || echo 0"]).split()
others = int(probe[0]) if probe else 0
ratio = float(probe[1]) if len(probe) > 1 else 0.0
busy = others > 0 or ratio > 5
if not busy or waited >= a.busy_wait_s:
contended = busy
break
time.sleep(20); waited += 20
if contended:
print(f"WARN board busy after {waited}s (procs={others}, ratio={ratio}); measuring anyway, flag contended", file=sys.stderr)
run(["scp", *scp_opts, "-q", a.hbm, f"{target}:{remote_hbm}"], timeout=1800)
run(["scp", *scp_opts, "-q", str(HERE / "board_bench.py"), f"{target}:{rdir}/board_bench.py"])
out_txt = run(["ssh", *opts, target,
f"cd {rdir} && python3 board_bench.py --hbm {remote_hbm} --warmup {a.warmup} --runs {a.runs} 2>/dev/null | tail -1"],
timeout=3600)
run(["ssh", *opts, target, f"rm -f {remote_hbm}"])
try:
d = json.loads(out_txt.strip().splitlines()[-1])
except Exception:
sys.stderr.write("board_bench produced no JSON:\n" + out_txt[-2000:])
return 5
d.update({"contended": contended, "busy_waited_s": waited, "measured_at": time.time()})
Path(a.out).parent.mkdir(parents=True, exist_ok=True)
Path(a.out).write_text(json.dumps(d, indent=2, ensure_ascii=False))
print(json.dumps({k: d.get(k) for k in ("latency_ms_mean", "latency_ms_p95", "bpu_util", "board_type", "ucp", "contended")}))
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@ -0,0 +1,81 @@
"""03-export 子进程:权重 → ONNX(静态 shape,opset 由 OE 3.7 要求决定,默认 11)。
按 source 分派:torchvision / ultralytics / timm / hf / custom(custom 走 adapters/<model>.py)。
export_fail 是 model-level,一次影响该模型所有 job。
"""
from __future__ import annotations
import argparse
import importlib
import sys
from pathlib import Path
OPSET = 11 # 实测 OE 3.7 接受 opset 11
EXPORTER_VERSION = "2" # 导出口径版本:变了缓存作废。v2 = ultralytics 走官方 exporter;torchvision 走 TorchScript exporter + arch 名映射
def load_model(model: str, source: str, weights: Path, input_spec: str):
import torch
if source == "torchvision":
import torchvision.models as tvm
arch = model.replace(".", "_") # model_id 里 '.' 是 builder 名的 '_'(job_id 段不允许下划线以外的分隔)
m = tvm.get_model(arch, weights=None)
m.load_state_dict(torch.load(weights, map_location="cpu"))
return m.eval()
if source in ("ultralytics", "github") and weights.suffix == ".pt":
from ultralytics import YOLO
y = YOLO(str(weights))
# ultralytics 自带 export;这里只取 nn.Module 走统一路径,保证 03-export 口径一致
return y.model.eval().float()
if source == "timm":
import timm
m = timm.create_model(model, pretrained=False)
m.load_state_dict(torch.load(weights, map_location="cpu"))
return m.eval()
# custom:bench/harness/adapters/<model>.py 里 def build(weights: Path, input_spec: str) -> nn.Module
mod = importlib.import_module(f"harness.adapters.{model.replace('.', '_').replace('-', '_')}")
return mod.build(weights, input_spec).eval()
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--model", required=True)
ap.add_argument("--source", required=True)
ap.add_argument("--weights", required=True)
ap.add_argument("--input-spec", required=True)
ap.add_argument("--out", required=True)
ap.add_argument("--opset", type=int, default=OPSET)
a = ap.parse_args()
import torch
shape = tuple(int(x) for x in a.input_spec.split("x"))
out = Path(a.out)
out.parent.mkdir(parents=True, exist_ok=True)
if a.source == "ultralytics" or (Path(a.weights).suffix == ".pt" and a.source in ("github", "ultralytics")):
# 直接 torch.onnx.export(nn.Module) 会把 Detect 头里的 make_anchors/Cast 留在图里 → hb_compile 'hbir.cast_type Unsupported type'
# 官方 exporter 把 anchors / strides 常量化,输出 [1, 4+nc, N],与 eval/det.py 的解码一致
from ultralytics import YOLO
y = YOLO(str(a.weights))
exported = y.export(format="onnx", opset=a.opset, imgsz=shape[-1], simplify=True, dynamic=False, half=False)
import shutil
shutil.move(str(exported), str(out))
import onnx
onnx.checker.check_model(str(out))
print(out)
return 0
m = load_model(a.model, a.source, Path(a.weights), a.input_spec)
x = torch.randn(*shape)
kw = dict(opset_version=a.opset, input_names=["input"], do_constant_folding=True, dynamic_axes=None)
with torch.no_grad():
try:
# torch ≥ 2.9 默认 dynamo 导出器(要求 opset ≥ 18,图形态也不同);OE 3.7 走经典 TorchScript 导出器
torch.onnx.export(m, x, str(out), dynamo=False, **kw)
except TypeError:
torch.onnx.export(m, x, str(out), **kw)
import onnx
onnx.checker.check_model(str(out))
print(out)
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@ -0,0 +1,85 @@
"""01-fetch 子进程:把 models.csv 的 weights URI 落成一个文件。
支持:
https://… 直接下载(带断点续传)
torchvision://<arch>/<WEIGHTS_ENUM> 例 torchvision://resnet18/IMAGENET1K_V1 → 保存 state_dict .pth
hf://<repo>/<file> huggingface hub
file:///abs/path 本机已有
"""
from __future__ import annotations
import argparse
import shutil
import sys
import urllib.request
from pathlib import Path
def fetch_http(url: str, out: Path) -> Path:
tmp = out.with_suffix(out.suffix + ".part")
pos = tmp.stat().st_size if tmp.exists() else 0
req = urllib.request.Request(url, headers={"Range": f"bytes={pos}-"} if pos else {})
with urllib.request.urlopen(req, timeout=60) as r, open(tmp, "ab") as f:
shutil.copyfileobj(r, f, length=1 << 20)
tmp.rename(out)
return out
def fetch_torchvision(arch: str, enum: str, out: Path) -> Path:
import torch, torchvision.models as tvm
w = getattr(tvm.get_model_weights(arch), enum) # enum 可以是 DEFAULT
model = tvm.get_model(arch, weights=w)
torch.save(model.state_dict(), out)
return out
def fetch_hf(repo: str, filename: str, out: Path) -> Path:
from huggingface_hub import hf_hub_download
p = hf_hub_download(repo_id=repo, filename=filename)
shutil.copy2(p, out)
return out
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--model", required=True)
ap.add_argument("--uri", required=True)
ap.add_argument("--out-dir", required=True)
a = ap.parse_args()
out_dir = Path(a.out_dir)
out_dir.mkdir(parents=True, exist_ok=True)
uri = a.uri
if uri.startswith("ultralytics://"):
# 权重名自动下载(GitHub release assets);YOLO() 会把 .pt 放到 cwd
import os
wname = uri[len("ultralytics://"):]
os.chdir(out_dir)
from ultralytics import YOLO
y = YOLO(wname)
src = Path(getattr(y, "ckpt_path", None) or wname)
if not src.exists():
src = next(out_dir.glob("*.pt"))
p = out_dir / src.name
if src.resolve() != p.resolve():
shutil.copy2(src, p)
elif uri.startswith("torchvision://"):
arch, enum = uri[len("torchvision://"):].split("/", 1)
p = fetch_torchvision(arch, enum, out_dir / f"{a.model}.pth")
elif uri.startswith("hf://"):
repo, fn = uri[len("hf://"):].split("/", 2)[:2] if uri.count("/") == 3 else uri[len("hf://"):].rsplit("/", 1)
p = fetch_hf(repo, fn, out_dir / Path(fn).name)
elif uri.startswith("file://"):
src = Path(uri[len("file://"):])
p = out_dir / src.name
shutil.copy2(src, p)
elif uri.startswith(("http://", "https://")):
p = fetch_http(uri, out_dir / Path(uri.split("?")[0]).name)
else:
print(f"unknown weights uri: {uri}", file=sys.stderr)
return 2
print(p)
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@ -0,0 +1,65 @@
"""x86 侧量化模型推理:inputs.npz → outputs.npz。在产出该模型的工具链 venv 里跑。
实测(2026-09-15,PTQ venv horizon_tc_ui 3.5.3,resnet18/nash-e):
.bc → HB_HBIRRuntime(bc).run(output_names, input_info) 0.04 s/张,与 hbm 的 run_sim **逐位一致**(cos=1.0, maxabs=0)
.hbm → HB_HBMRuntime(hbm).run_sim(output_names, input_info) ~20 s/张(慢,只用来抽查 hbm ≡ bc)
ptq_model.onnx → HB_ONNXRuntime(伪量化,cos≈0.9995,不逐位)——不用它出数
所以 G3 全量走 .bc,06-hbm 用 2 张抽查 hbm ≡ bc。QAT 产物走 hbdk4(P4 核对)。
都不可用就非零退出,让上游记 unverified 而不是假 pass。
"""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
def _runner(model: Path):
errs = []
suf = model.suffix.lower()
try:
if suf == ".bc":
from horizon_tc_ui.hb_hbirruntime import HB_HBIRRuntime # type: ignore
rt = HB_HBIRRuntime(str(model))
names, outs = list(rt.input_names), list(rt.output_names)
return lambda x: dict(zip(outs, rt.run(output_names=outs, input_info={names[0]: x})))
if suf == ".hbm":
from horizon_tc_ui.hb_hbmruntime import HB_HBMRuntime # type: ignore
rt = HB_HBMRuntime(str(model))
names, outs = list(rt.input_names), list(rt.output_names)
return lambda x: dict(zip(outs, rt.run_sim(output_names=outs, input_info={names[0]: x})))
except Exception as e: # noqa: BLE001
errs.append(f"horizon_tc_ui runtime: {type(e).__name__}: {e}")
try:
from hbdk4.compiler import load # type: ignore # QAT / hbdk4 4.12 路径,P4 核对
m = load(str(model))
fn = m.functions[0]
return lambda x: fn.feed([x])
except Exception as e: # noqa: BLE001
errs.append(f"hbdk4.load: {type(e).__name__}: {e}")
print("no runtime available for %s in this venv:\n %s" % (model, "\n ".join(errs)), file=sys.stderr)
raise SystemExit(3)
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--model", "--hbm", dest="model", required=True, help=".bc(快,逐位等价)或 .hbm(慢)")
ap.add_argument("--inputs", required=True, help="npz: 每个 key 一个样本,shape 与 job 一致(含 batch 维)")
ap.add_argument("--out", required=True)
a = ap.parse_args()
import numpy as np
run = _runner(Path(a.model))
ins = np.load(a.inputs)
outs = {}
for k in ins.files:
res = run(np.ascontiguousarray(ins[k], dtype=np.float32))
items = list(res.items()) if isinstance(res, dict) else list(enumerate(res))
for i, (_, v) in enumerate(items):
# 必须深拷贝:HB_HBIRRuntime.run 返回的数组引用内部缓冲区,下一次 run 会原地覆盖(实测踩坑:两张图输出变成同一份)
outs[f"{k}/out{i}"] = np.array(v, copy=True)
np.savez(a.out, **outs)
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@ -0,0 +1,16 @@
"""QAT 后端 05-compile 子进程(oellm2 venv):plugin calibration-only → hbir → hbdk4 compile → .hbm。
P4 之后实现。现在只占位:非零退出并写清楚原因,让 job 记 FAIL:unknown 而不是假 PASS。
"""
from __future__ import annotations
import sys
def main() -> int:
print("qat backend not implemented yet (P4). See bench/harness/tools/qat_calib_compile.py", file=sys.stderr)
return 64
if __name__ == "__main__":
sys.exit(main())

152
harness/verify.py Normal file
View File

@ -0,0 +1,152 @@
"""L3:对一次 rollout 独立核验 → verdict.json。reward 的唯一来源。
输入:rollout/stages/*/meta.json(harness 自己跑的,agent 改不了)、claim.json(agent 自报,不可信)、
oracle/result.json(参照系)、agent/events.jsonl(成本)。
不读 workspace 里 agent 写的任何"结果"。
"""
from __future__ import annotations
import json
import time
from pathlib import Path
from . import db, gates
from .errors import same_class
def _latest_meta(stage_root: Path, name: str) -> dict | None:
p = stage_root / name / "meta.json"
return json.loads(p.read_text()) if p.exists() else None
def collect_measurements(rd: Path, stage_root: Path | None = None) -> tuple[dict, dict]:
"""从 stages/<NN>/meta.json 汇总测量值;返回 (measurements, stage_summary)。
rollout 传 rollout 目录;oracle 传 stage_root=…/oracle/stages。06-hbm 的 G1 若 meta 里没有,回退读它的 stdout.log(就是那份 JSON)。"""
root = stage_root or (rd / "stages")
meas, summ = {}, {}
for name in ("04-calibrate", "05-compile", "06-hbm", "07-board", "08-eval"):
m = _latest_meta(root, name)
if not m:
continue
summ[name] = {"state": m.get("state"), "error_class": m.get("error_class"), "attempts": m.get("attempt", 0) + 1}
if m.get("state") in ("OK", "SKIPPED"):
for k, v in (m.get("measurements") or {}).items():
meas[k] = v
if name == "06-hbm" and "G1" not in meas:
so = root / name / f"a{m.get('attempt', 0):02d}" / "stdout.log"
try:
meas["G1"] = json.loads(so.read_text())
except Exception:
pass
if "06-hbm" not in summ or summ["06-hbm"]["state"] != "OK":
meas.setdefault("G1", {"hbm_exists": False})
return meas, summ
def _cost(rd: Path) -> dict:
"""成本只从 events.jsonl 数:turns = turn_end 事件数(别数 message_end,那会重复计三次)。
实测(2026-09-15,dr-gateway / glm-5.3-flash):网关回的 usage 全是 0。
所以 tokens 可能整体不可用 → 记 tokens_reported=false,reward 的代价项据此退化为只按 turns 算,
否则"没报 token"会被误读成"这条轨迹很省"。
"""
ev = rd / "agent" / "events.jsonl"
turns = tokens = tool_calls = 0
t0 = t1 = None
if ev.exists():
with ev.open(errors="replace") as f:
for line in f:
try:
e = json.loads(line)
except json.JSONDecodeError:
continue
ts = e.get("timestamp") or e.get("ts") or (e.get("message") or {}).get("timestamp")
if isinstance(ts, (int, float)):
t0 = t0 or ts; t1 = ts
typ = e.get("type", "")
if typ == "turn_end":
turns += 1
u = ((e.get("message") or {}).get("usage")) or e.get("usage")
if isinstance(u, dict):
tokens += int(u.get("input", 0) or u.get("input_tokens", 0) or 0)
tokens += int(u.get("output", 0) or u.get("output_tokens", 0) or 0)
elif typ == "tool_execution_start":
tool_calls += 1
wall = None
if t0 and t1:
try:
wall = (float(t1) - float(t0)) / 1000.0 if float(t1) > 1e11 else float(t1) - float(t0)
except (TypeError, ValueError):
wall = None
return {"turns": turns, "tokens": tokens, "tokens_reported": tokens > 0,
"tool_calls": tool_calls, "wall_s": round(wall, 1) if wall else None}
def verify_rollout(rd: Path, write: bool = True) -> dict:
rd = rd.resolve()
jd = rd.parent.parent
job = json.loads((jd / "job.json").read_text())
kk = int(rd.name)
oracle = json.loads((jd / "oracle" / "result.json").read_text()) if (jd / "oracle" / "result.json").exists() else None
claim = json.loads((rd / "claim.json").read_text()) if (rd / "claim.json").exists() else None
rmeta = json.loads((rd / "rollout.json").read_text()) if (rd / "rollout.json").exists() else {}
meas, summ = collect_measurements(rd)
ev = gates.evaluate(meas, job["gates"])
terminal = ev["terminal"]
# agent 自报 BLOCKED / TIMEOUT 的处理
blocked_correct = None
if rmeta.get("state") == "TIMEOUT":
terminal = "TIMEOUT"
elif claim and claim.get("terminal") == "BLOCKED":
terminal = "BLOCKED"
o_ec = (oracle or {}).get("error_class")
o_stage = (oracle or {}).get("stage_failed")
blocked_correct = bool(oracle) and oracle.get("terminal") == "FAIL" and \
same_class(claim.get("error_class"), o_ec) and (claim.get("stage") in (None, o_stage))
elif terminal == "UNVERIFIED" and (claim or {}).get("terminal") == "PASS":
terminal = "FAIL" # 说通过但没有可核验的 hbm/eval → 按 FAIL 记,claim≠verdict 另扣
claim_consistent = None
if claim:
claim_consistent = (claim.get("terminal") == terminal) if terminal in ("PASS", "FAIL", "BLOCKED") else None
g3 = ev["gates"]["G3"]
o_g3 = ((oracle or {}).get("gates") or {}).get("G3") or {}
o_g4 = ((oracle or {}).get("gates") or {}).get("G4") or {}
review = bool(oracle) and oracle.get("terminal") == "FAIL" and terminal == "PASS"
v_ec = gates.classify_gate_failure(ev) if terminal == "FAIL" else None
for st in summ.values():
if st.get("state") not in ("OK", "SKIPPED") and st.get("error_class"):
v_ec = st["error_class"]; break
verdict = {
"schema": "bench.verdict/1",
"error_class": v_ec,
"job_id": job["job_id"], "run_id": job["run_id"], "rollout": kk,
"mode": rmeta.get("mode"), "policy": rmeta.get("policy"),
"gates": ev["gates"], "required": ev["required"],
"terminal": terminal,
"failed_gate": ev["failed_gate"],
"stages": summ,
"claim": claim, "claim_consistent": claim_consistent,
"blocked_correct": blocked_correct,
"oracle_ref": {"terminal": (oracle or {}).get("terminal"), "error_class": (oracle or {}).get("error_class"),
"stage_failed": (oracle or {}).get("stage_failed"),
"G3_delta": o_g3.get("delta"), "latency_ms_mean": o_g4.get("latency_ms_mean")} if oracle else None,
"review": review,
"cost": _cost(rd),
"verified_at": time.time(),
}
if write:
(rd / "verdict.json").write_text(json.dumps(verdict, indent=2, ensure_ascii=False, default=str))
with db.connect() as con:
db.rollout_upsert(con, job["run_id"], job["job_id"], kk, rmeta.get("mode"), rmeta.get("policy"))
db.rollout_set(con, job["run_id"], job["job_id"], kk, state=terminal,
claim_terminal=(claim or {}).get("terminal"), verdict_terminal=terminal,
claim_consistent=None if claim_consistent is None else int(claim_consistent),
review=int(review), turns=verdict["cost"]["turns"], tokens=verdict["cost"]["tokens"],
wall_s=verdict["cost"]["wall_s"], verdict=json.dumps(verdict, default=str),
finished_at=time.time())
return verdict

117
harness/watch.py Normal file
View File

@ -0,0 +1,117 @@
"""bench watch — 终端实时看板:在跑什么、跑了多久、日志多久没动、谁可能卡死、还剩多少。
bench watch [--run r…] [--interval 30] [--once]
"卡死"的判断口径与看门狗一致但更早预警:
日志 idle > 5 min → ⚠ 提示
日志 idle > stall_s × 2/3 → 🔥 看门狗即将介入(stall_s 默认 900 s,且要求 CPU 也空闲才真杀)
看门狗真正杀掉的会以 STALLED 出现在"最近完成"里,error_class=stalled。
"""
from __future__ import annotations
import os
import sys
import time
from pathlib import Path
from . import paths, db
from .scheduler import pool_usage, pool_size, estimate_stage
def _fmt(s: float | None) -> str:
if s is None:
return "-"
s = int(s)
return f"{s // 3600}h{(s % 3600) // 60:02d}m" if s >= 3600 else (f"{s // 60}m{s % 60:02d}s" if s >= 60 else f"{s}s")
def _log_age(run_id: str, job_id: str, rollout: int, stage: str, attempt: int) -> float | None:
jd = paths.RUNS / run_id / job_id
root = jd / "oracle" / "stages" if rollout < 0 else jd / "rollouts" / f"{rollout:02d}" / "stages"
sd = root / stage / f"a{attempt:02d}"
latest = 0.0
for p in list(sd.glob("*.log")) + list((sd.parent / "work").glob("*.log")) + [sd]:
try:
latest = max(latest, p.stat().st_mtime)
except OSError:
pass
return (time.time() - latest) if latest else None
def snapshot(run_id: str) -> str:
cfg = paths.config()
stall_s = int(cfg["watchdog"]["stall_s"])
lines = []
with db.connect() as con:
run = db.get_run(con, run_id)
jobs = db.list_jobs(con, run_id, mode="oracle")
counts = {}
for j in jobs:
counts[j["state"]] = counts.get(j["state"], 0) + 1
running = con.execute("SELECT * FROM stages WHERE run_id=? AND state='RUNNING' ORDER BY started_at", (run_id,)).fetchall()
done = con.execute("SELECT * FROM stages WHERE run_id=? AND state!='RUNNING' AND finished_at IS NOT NULL ORDER BY finished_at DESC LIMIT 8", (run_id,)).fetchall()
ec = db.error_class_counts(con, run_id)
# 完成的 job 平均耗时 → ETA
durs = [ (j["finished_at"] - j["started_at"]) for j in jobs if j.get("finished_at") and j.get("started_at") and j["state"] in ("PASS", "FAIL", "UNVERIFIED")]
from .registry import status_counts
try:
sc = status_counts()
except Exception:
sc = {}
now = time.strftime("%Y-%m-%d %H:%M:%S")
lines.append(f"bench watch · run {run_id} · {now} · {run.get('note','') if run else ''}")
lines.append(" jobs: " + " ".join(f"{k}={v}" for k, v in sorted(counts.items())) + f" (registry: {', '.join(f'{k}={v}' for k, v in sc.items())})")
lines.append(" pools: " + " ".join(f"{p}={n}/{pool_size(p)}" for p, n in pool_usage().items()))
pending = counts.get("PENDING", 0)
if durs and pending:
avg = sum(durs) / len(durs)
par = max(1, pool_usage().get("cpu", 1) or cfg["pools"]["cpu"]["workers"])
lines.append(f" ETA: 剩 {pending} 个 job × 平均 {_fmt(avg)} / 并行 {par} ≈ {_fmt(pending * avg / par)}(已完成 {len(durs)} 个)")
lines.append("")
lines.append(f" RUNNING ({len(running)})")
lines.append(f" {'job_id':<48} {'stage':<13} {'att':<4} {'elapsed':>8} {'est':>7} {'log idle':>9} flag")
for s in running:
el = time.time() - (s["started_at"] or time.time())
age = _log_age(run_id, s["job_id"], s["rollout"], s["stage"], s["attempt"])
try:
with db.connect() as con:
jrow = db.get_job(con, run_id, s["job_id"])
est = estimate_stage(con, jrow["model"], jrow["task"], s["stage"]) if jrow else None
except Exception:
est = None
flag = ""
if s["lane"] == "queued":
# rollout 提交的阶段还在等池槽位:不是卡死,别报看门狗
lines.append(f" {s['job_id']:<48} {s['stage']:<13} a{s['attempt']:02d}r{s['rollout']:02d} {_fmt(el):>8} {'-':>7} {'-':>9} ⏸ 排队等槽位")
continue
if age is not None and age > stall_s * 2 / 3:
flag = "🔥 看门狗即将介入"
elif age is not None and age > 300:
flag = "⚠ 日志 5 min 没动"
elif est and el > est * 3:
flag = "⏳ 超历史 p95×3(只告警)"
tag = f"r{s['rollout']:02d}" if s["rollout"] >= 0 else " "
lines.append(f" {s['job_id']:<48} {s['stage']:<13} a{s['attempt']:02d}{tag} {_fmt(el):>8} {_fmt(est):>7} {_fmt(age):>9} {flag}")
lines.append("")
lines.append(" 最近完成")
for s in done:
lines.append(f" {s['job_id']:<48} {s['stage']:<13} {s['state']:<8} {s['error_class'] or '':<16} {_fmt(s['duration_s']):>8}")
if ec:
lines.append("")
lines.append(" FAIL by error_class: " + " ".join(f"{r['error_class']}@{r['stage_failed']}={r['n']}" for r in ec))
return "\n".join(lines)
def watch(run_id: str, interval: int = 30, once: bool = False) -> int:
while True:
text = snapshot(run_id)
if not once:
os.system("clear" if os.name != "nt" else "cls")
print(text)
if once:
return 0
print(f"\n (每 {interval}s 刷新 · Ctrl-C 退出 · bench show <job_id> 看细节)")
try:
time.sleep(interval)
except KeyboardInterrupt:
return 0

139
harness/watchdog.py Normal file
View File

@ -0,0 +1,139 @@
"""无固定超时的子进程执行器。
STALLED 的唯一定义:stdout+stderr 文件 stall_s 秒不增长 且 进程树 CPU 使用率 < cpu_idle_pct%。
真正在算的编译,跑 6 小时也不会被杀。hard_ceiling_s 是纯保险。
"""
from __future__ import annotations
import os
import signal
import subprocess
import time
from dataclasses import dataclass
from pathlib import Path
from .errors import STALLED_MARK
CLK_TCK = os.sysconf("SC_CLK_TCK") if hasattr(os, "sysconf") else 100
@dataclass
class RunResult:
state: str # OK | FAIL | STALLED
exit_code: int | None
duration_s: float
stdout: Path
stderr: Path
stalled_reason: str | None = None
io_blocked_s: float = 0.0 # 期间最长的连续 I/O 阻塞(D 状态)时长,进 meta.json 供诊断
def _proc_tree_stat(root: int) -> tuple[int, int] | None:
"""Linux: (root 及其全部后代的 utime+stime 总和 ticks, 处于 D 状态的进程数)。非 Linux 返回 None。
D = 不可中断睡眠 = 卡在 I/O。实测(2026-09-15)JuiceFS 写 1 MB 要 360 ms(本地盘 3 ms),
hbdk 编译大模型时会长时间待在 D 里:CPU ticks 不涨、日志不涨 —— 与"真挂死"在表象上完全一样。
EfficientNet-B4/B5 因此被误判 STALLED 杀了两次(900s + 3600s)。所以 D 状态要单独看,不能算空闲。
"""
proc = Path("/proc")
if not proc.exists():
return None
children: dict[int, list[int]] = {}
ticks: dict[int, int] = {}
state: dict[int, str] = {}
for p in proc.iterdir():
if not p.name.isdigit():
continue
try:
stat = (p / "stat").read_text()
except OSError:
continue
# comm 可能含空格/括号:从最后一个 ')' 之后切
rest = stat[stat.rfind(")") + 2:].split()
pid = int(p.name)
state[pid] = rest[0]
ppid = int(rest[1])
ticks[pid] = int(rest[11]) + int(rest[12])
children.setdefault(ppid, []).append(pid)
total, d_count, stack = 0, 0, [root]
while stack:
pid = stack.pop()
total += ticks.get(pid, 0)
if state.get(pid) == "D":
d_count += 1
stack.extend(children.get(pid, []))
return total, d_count
def _kill_tree(proc: subprocess.Popen) -> None:
try:
os.killpg(proc.pid, signal.SIGTERM)
except ProcessLookupError:
return
for _ in range(20):
if proc.poll() is not None:
return
time.sleep(0.5)
try:
os.killpg(proc.pid, signal.SIGKILL)
except ProcessLookupError:
pass
def run(cmd: list[str] | str, *, cwd: Path, stdout: Path, stderr: Path, env: dict | None = None,
stall_s: int = 900, cpu_idle_pct: float = 5.0, poll_s: int = 30,
hard_ceiling_s: int = 86400, shell: bool = False, io_stall_s: int = 21600) -> RunResult:
stdout.parent.mkdir(parents=True, exist_ok=True)
t0 = time.time()
with open(stdout, "ab") as fo, open(stderr, "ab") as fe:
proc = subprocess.Popen(cmd, cwd=str(cwd), stdout=fo, stderr=fe, env=env,
shell=shell, start_new_session=True)
last_size = -1
st = _proc_tree_stat(proc.pid)
last_ticks = st[0] if st else None
stalled_for = io_blocked_for = max_io_blocked = 0.0
stalled_reason = None
while True:
rc = proc.poll()
if rc is not None:
break
time.sleep(poll_s)
now = time.time()
if now - t0 > hard_ceiling_s:
stalled_reason = f"hard_ceiling {hard_ceiling_s}s"
break
size = stdout.stat().st_size + stderr.stat().st_size
st = _proc_tree_stat(proc.pid)
ticks, d_count = (st if st else (None, 0))
cpu_busy = True
if ticks is not None and last_ticks is not None:
cpu_pct = 100.0 * (ticks - last_ticks) / CLK_TCK / max(poll_s, 1)
cpu_busy = cpu_pct >= cpu_idle_pct
last_ticks = ticks
if size == last_size and not cpu_busy:
if d_count > 0:
# 卡在 I/O(JuiceFS 常见),不是挂死:计入 io_blocked,不累加 stall
io_blocked_for += poll_s
max_io_blocked = max(max_io_blocked, io_blocked_for)
stalled_for = 0.0
else:
stalled_for += poll_s
io_blocked_for = 0.0
else:
stalled_for = io_blocked_for = 0.0
last_size = size
if stalled_for >= stall_s:
stalled_reason = f"log unchanged and cpu idle for {int(stalled_for)}s"
break
if io_blocked_for >= io_stall_s:
stalled_reason = f"blocked on I/O (D state) for {int(io_blocked_for)}s"
break
if stalled_reason:
_kill_tree(proc)
fe.write(f"\n{STALLED_MARK} {stalled_reason}\n".encode())
fe.flush()
return RunResult("STALLED", proc.poll(), time.time() - t0, stdout, stderr, stalled_reason,
round(max_io_blocked, 1))
rc = proc.returncode
return RunResult("OK" if rc == 0 else "FAIL", rc, time.time() - t0, stdout, stderr,
None, round(max_io_blocked, 1))

26
registry/boards.yaml Normal file
View File

@ -0,0 +1,26 @@
# registry/boards.yaml — 板型 ↔ march 映射与板卡接入。凭证不在这里(.env.local)。
version: 1
march_by_board:
s100: nash-e
s100p: nash-m
s600: nash-p
boards:
rdk-sandbox:
host_env: BOARD_HOST # rdk-sandbox.d-robotics.cc
port_env: BOARD_PORT # 40010
user_env: BOARD_USER
ssh_key_env: BOARD_SSH_KEY # 口令已泄露于明文对话,改密钥后轮换
board_type: s100p # 2026-09-15 实测 /proc/device-tree/model = "D-Robotics RDK S100P V1P0"
march: nash-m
ucp_version: "3.13.6" # 实测(.env.oe-package 里写的 3.15.2 不对);Ubuntu 22.04 容器,6×A78AE,1 个 BPU core,21 GB
runtime_dir: /userdata/bench # 板上工作目录
tools:
perf: python3 hbm_runtime # 沙箱没有 hrt_model_exec;用 hbm_runtime 包计时(harness/tools/board_bench.py)
# 主机侧 march 校验用的字符串(G1 check_march 比对 hb_model_info 输出)
march_strings:
nash-e: ["nash-e", "Nash-E"]
nash-m: ["nash-m", "Nash-M"]
nash-p: ["nash-p", "Nash-P"]

28
registry/budgets.yaml Normal file
View File

@ -0,0 +1,28 @@
# registry/budgets.yaml — 逐任务类耗时估算(只用于调度排序与 soft_alert,不用于杀进程)
# 首轮估算按参数量分档;jobs.sqlite 有实测后以实测 p95 为准(harness/scheduler.py::estimate)
version: 1
# 按参数量分档的首轮估算,单位秒
by_params_m:
- { max: 10, compile: 600, board: 120, eval: 300 }
- { max: 50, compile: 1800, board: 180, eval: 600 }
- { max: 200, compile: 5400, board: 300, eval: 1200 }
- { max: null, compile: 14400, board: 600, eval: 2400 }
# 任务类乘子(占据/光流/局部特征算子特殊,编译慢、失败率高)
task_multiplier:
cls: 1.0
det: 1.2
seg: 1.5
iseg: 1.5
pose: 1.2
mde: 1.5
stereo: 2.0
flow: 2.5
feat: 2.0
occ: 3.0
# 看门狗参数可按任务覆盖;不写则用 config.yaml watchdog
watchdog_overrides:
occ: { stall_s: 1800 }
flow: { stall_s: 1800 }

153
registry/datasets.yaml Normal file
View File

@ -0,0 +1,153 @@
# registry/datasets.yaml — 每个数据集的有序来源、版本、T0/T1 子集规格
# 来源按顺序尝试:local → hf → oss → github → official。scripts/ingest_pool.py 读本文件。
# sha256 字段在首次入池后由 ingest 写回 data/pool/<ds>/<ver>/manifest.json;这里的 expect_* 是入池校验用的硬约束。
version: 1
pool_root: data/pool
local_root: /mnt/data # 本机 JuiceFS 根
datasets:
imagenet1k:
version: val-2012
tasks: [cls]
expect_count: 50000
sources:
# ① 首选:HF parquet 14 片(已在本机,避开 tar 完整性风险)
- kind: local_parquet
path: chao01.wu/Data/HuggingFace/ImageNet_HF/imagenet/data
glob: "validation-*.parquet"
expect_shards: 14
# ② 备用:tar(旁边有 .tmp 残片,需 tar -tzf | wc -l == 50000)
- kind: local_tar
path: chao01.wu/Data/Datasets/ImageNet/val_images.tar.gz
verify: "tar -tzf {path} | wc -l"
expect: 50000
- kind: hf
repo: ILSVRC/imagenet-1k
split: validation
t1: { n: 2000, seed: 20260914, stratify: label }
calib: { n: 128, from: train, seed: 20260914, input_spec_default: 1x3x224x224,
train_parquet: chao01.wu/Data/HuggingFace/ImageNet_HF/imagenet/data, train_glob: "train-*.parquet" }
coco2017:
version: val2017
tasks: [det, iseg, pose]
expect_count: 5000
sources:
- kind: local_dir
path: chao01.wu/Data/Datasets/COCO2017/val2017
annotations:
- chao01.wu/Data/Datasets/COCO2017/annotations_trainval2017.zip
- kind: local_zip # 单文件顺序读,比逐文件 rsync 快 10 倍以上:--prefer local_zip
path: guannan.he/coco2017/val2017.zip
annotations:
- chao01.wu/Data/Datasets/COCO2017/annotations_trainval2017.zip
- kind: hf
repo: detection-datasets/coco
- kind: official
url: http://images.cocodataset.org/zips/val2017.zip # 26 KB/s,等于不通
t1: { n: 1000, seed: 20260914, stratify: none }
calib: { n: 128, from: train2017, seed: 20260914, input_spec_default: 1x3x640x640,
train_dir: chao01.wu/Data/Datasets/COCO2017/train2017 } # 本机只有 200 张 train2017,够 128
cityscapes:
version: val-fine
tasks: [seg]
expect_count: 500
sources:
- kind: local_zip
path: guannan.he/CityScapes/leftImg8bit_trainvaltest.zip
extract_only: "*/val/*"
extra:
- { path: guannan.he/CityScapes/gtFine_trainvaltest.zip, extract_only: "*/val/*" }
t1: { n: 500, seed: 20260914 }
calib: { n: 128, from: train, seed: 20260914 }
ade20k:
version: val-2016
tasks: [seg]
expect_count: 2000
sources:
- kind: hf
repo: 1aurent/ADE20K
split: validation
t1: { n: 500, seed: 20260914 }
calib: { n: 128, from: train, seed: 20260914 }
lvis:
version: v1-val
tasks: [iseg]
images_from: coco2017 # 图片复用 COCO,只需标注
sources:
- kind: github
url: https://dl.fbaipublicfiles.com/lvis/lvis_v1_val.json.zip
- kind: hf
repo: lvis/lvis
t1: { n: 1000, seed: 20260914 }
kitti2015:
version: data_scene_flow
tasks: [stereo, flow]
expect_count: 200
sources:
- kind: local_dir
path: qi.xiong/Dataset/StereoDataset/KITTI/data_scene_flow
t1: { n: 200, seed: 20260914 } # 全量
calib: { n: 128, from: training, seed: 20260914 }
note: stereo 与 flow 共用一份,manifest 分开
eth3d:
version: two-view
tasks: [stereo]
sources:
- kind: local_dir
path: qi.xiong/Dataset/StereoDataset/eth3d
t1: { n: all }
nyuv2:
version: test-654
tasks: [mde]
expect_count: 654
sources:
- kind: hf
repo: sayakpaul/nyu_depth_v2
split: validation
t1: { n: 654 }
sintel:
version: train-final
tasks: [flow]
expect_count: 1041
sources:
- kind: hf
repo: null # 待定:找到 HF 镜像后填
- kind: official
url: http://files.is.tue.mpg.de/sintel/MPI-Sintel-complete.zip
t1: { n: all }
hpatches:
version: sequences-release
tasks: [feat]
expect_count: 116
sources:
- kind: github
url: https://huggingface.co/datasets/cvg/hpatches-sequences # 待核实
- kind: official
url: http://icvl.ee.ic.ac.uk/vbalnt/hpatches/hpatches-sequences-release.tar.gz
t1: { n: all }
nuscenes:
version: full-v1.0
tasks: [occ]
sources:
- kind: local_dir
path: qi.xiong/Dataset/AutoDriveDataset/nuScenes/nuScenes_Full
t1: { n: 0 } # 第一轮不评 G3
calib: { n: 128, from: samples, seed: 20260914, note: "occ 第一轮只取 128 帧做校准" }
occ3d_nuscenes:
version: labels-v1
tasks: [occ]
round: 2 # 第二轮再入池
sources:
- kind: github
url: https://github.com/Tsinghua-MARS-Lab/Occ3D

100
registry/gates.yaml Normal file
View File

@ -0,0 +1,100 @@
# registry/gates.yaml — 四道门阈值。随 run 快照进 job.json。
# 容差 kind 必须显式:abs_pp(绝对百分点,Δ = quant − fp32)| rel_pct(相对劣化 %,(quant−fp32)/fp32)
# 来源列:交付指标文档 S* 编号;"proposed" = 本方案提议,第一个 run 后校准
version: 1
G1:
require_hbm: true
check_march: true
check_input_dtype_layout: true
run_hb_verifier: true
log_must_not_contain: ["ERROR"]
G2:
default_threshold: 0.99
report: min # 只报 min,不报均值
n_samples: 32 # 校准外样本,从 T1 另抽
overrides:
# batch-1(r20260915-03)校准:ShuffleNetV2 ×0.5/1.0/1.5、MobileNetV2 top-1 只掉 0.55–1.5 pp(G3 合格),
# 但 1000 类 logits 的 32 样本 min 余弦 0.93–0.98。0.99 来自特征匹配场景,对 logits 太严 → cls 放到 0.95。
# det 保持 0.99:yolov8n 余弦 0.9993 而 mAP 掉 15,G2 对检测头输出没有判别力,真正的判官是 G3。
cls: 0.95
occ: 0.97
feat: # 逐输出
desc: 0.99
heatmap: 0.92
G3:
# T1 上 tol 自动放宽 tol − 2·SE,记 pass-screen;T2 才 pass
t1_screen_widen_se: 2
tasks:
cls:
metric: top1
dataset: imagenet1k
kind: abs_pp
tol: -1.0
also_record: [top5]
source: "S9 MobileNetv2 −0.73 pp"
det:
metric: map_50_95
dataset: coco2017
kind: abs_pp
tol: -1.5
also_record: [ap50]
source: "S2 DOSODv2 目标差 1.3 / 1.0"
seg:
metric: miou
dataset: [ade20k, cityscapes]
kind: abs_pp
tol: -1.0
also_record: [per_class_iou]
source: "S9 Unet −0.44 pp"
iseg:
metric: mask_ap
dataset: [coco2017, lvis]
kind: abs_pp
tol: -1.5
source: "proposed (借用 T02)"
pose:
metric: oks_ap
dataset: coco2017
kind: abs_pp
tol: -1.5
extra: { keypoint_mean_shift_px_max: 2 }
source: "S11 1–2 px 可接受;AP 借用"
mde:
metric: absrel
dataset: [kitti_eigen, nyuv2]
kind: rel_pct
tol: 10
extra: { delta1_abs_pp_tol: -1.0, scale_alignment: required }
source: "S12 仅 0.90 相关性;proposed"
stereo:
metric: [epe, d1_all]
dataset: kitti2015
kind: rel_pct
tol: 15
source: "S4 SOW 量化允许 ≤ 15%"
flow:
metric: [epe2d, fl_all]
dataset: [sintel, kitti2015]
kind: rel_pct
tol: 15
source: "proposed (借用 T07)"
feat:
metric: hpatches_homography_auc
dataset: hpatches
kind: abs_pp
tol: -2.0
source: "S14 DFeature 0.9909 / 0.9282"
occ:
metric: [occ_iou, sem_miou]
dataset: occ3d_nuscenes
kind: abs_pp
tol: -1.5
source: "S15 无阈值;proposed"
G4:
mandatory: false
judge_only_if: latency_budget_ms # job.json 声明才判
record: [latency_ms_mean, latency_ms_p95, bpu_util, ddr_bw_mbps]

54
registry/matrix.yaml Normal file
View File

@ -0,0 +1,54 @@
# registry/matrix.yaml — march × backend × qcfg × mode × policy 的组合与裁剪
# 展开规则见 harness/registry.py::expand_matrix。展开结果写 state/jobs.sqlite,展开是幂等的。
version: 1
marches: [nash-e, nash-m, nash-p]
# 默认 qcfg:e/m 走 int8,p 走 fp16+int8 混合(段内只允许 a-z0-9. → 写作 fp16int8)
qcfg_by_march:
nash-e: int8
nash-m: int8
nash-p: fp16int8
backends: [ptq] # qat 在 P4 之后加
# 默认数据档;T2 只在专门的 run 里打开
data_tier: t1
modes:
# oracle:全 march × backend × 全模型。每 job 一份 ground truth。
oracle:
enabled: true
select: { priority: [A, B, C] }
marches: all
backends: all
# e2e:agent 从零走完全流程。按 (model, backend) 只采主 march,避免轨迹重复。
e2e:
enabled: true
select: { priority: [A], tasks: [cls, det] } # P3 先 cls/det;P4 扩到每类 6–10 个
marches: [nash-e]
backends: [ptq]
policies: [teacher]
k: 4 # 独立 rollout 数,构成 GRPO 组
# repair:给 oracle 的失败现场。只吃 oracle 的 FAIL,不吃 rollout 的。
repair:
enabled: true
when: oracle_fail
error_classes: [unsupported_op, shape_infer, calib_nan, verifier_mismatch, accuracy_collapse, accuracy_drop, numeric_mismatch]
marches: [nash-e]
backends: [ptq]
policies: [teacher]
k: 2
# 逐任务/逐模型覆盖。按顺序应用,后者覆盖前者。
overrides:
- match: { task: occ }
gates: [G1, G2, G4]
g3: { verdict: na, reason: round1-compile-only }
g2_threshold: 0.97
calib_source_ladder: [repo_demo, nuscenes_128, synthetic]
prompt: e2e-occ.md
- match: { task: feat }
g2_per_output: true

339
registry/models.csv Normal file
View File

@ -0,0 +1,339 @@
model_id,name,task,priority,family,source,repo,weights,input_spec,resize,dataset,params_m,official_metric,status,notes,excel_id
resnet18,ResNet-18,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://resnet18/IMAGENET1K_V1,1x3x224x224,256,imagenet1k,11.69,69.758,ready,TorchVision 预训练权重,可由 API 自动下载,1
resnet34,ResNet-34,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://resnet34/IMAGENET1K_V1,1x3x224x224,256,imagenet1k,21.8,73.314,ready,TorchVision 预训练权重,可由 API 自动下载,2
resnet50,ResNet-50,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://resnet50/IMAGENET1K_V2,1x3x224x224,232,imagenet1k,25.56,80.858,ready,TorchVision 预训练权重,可由 API 自动下载,3
resnet101,ResNet-101,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://resnet101/IMAGENET1K_V2,1x3x224x224,232,imagenet1k,44.55,81.886,ready,TorchVision 预训练权重,可由 API 自动下载,4
resnet152,ResNet-152,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://resnet152/IMAGENET1K_V2,1x3x224x224,232,imagenet1k,60.19,82.284,ready,TorchVision 预训练权重,可由 API 自动下载,5
resnext50.32x4d,ResNeXt-50-32x4d,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://resnext50_32x4d/IMAGENET1K_V2,1x3x224x224,232,imagenet1k,25.03,81.198,ready,TorchVision 预训练权重,可由 API 自动下载,6
resnext101.32x8d,ResNeXt-101-32x8d,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://resnext101_32x8d/IMAGENET1K_V2,1x3x224x224,232,imagenet1k,88.79,82.834,ready,TorchVision 预训练权重,可由 API 自动下载,7
wide.resnet50.2,Wide-ResNet-50-2,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://wide_resnet50_2/IMAGENET1K_V2,1x3x224x224,232,imagenet1k,68.88,81.602,ready,TorchVision 预训练权重,可由 API 自动下载,8
wide.resnet101.2,Wide-ResNet-101-2,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://wide_resnet101_2/IMAGENET1K_V2,1x3x224x224,232,imagenet1k,126.89,82.51,ready,TorchVision 预训练权重,可由 API 自动下载,9
densenet121,DenseNet-121,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://densenet121/IMAGENET1K_V1,1x3x224x224,256,imagenet1k,7.98,74.434,ready,TorchVision 预训练权重,可由 API 自动下载,10
densenet161,DenseNet-161,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://densenet161/IMAGENET1K_V1,1x3x224x224,256,imagenet1k,28.68,77.138,ready,TorchVision 预训练权重,可由 API 自动下载,11
densenet169,DenseNet-169,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://densenet169/IMAGENET1K_V1,1x3x224x224,256,imagenet1k,14.15,75.6,ready,TorchVision 预训练权重,可由 API 自动下载,12
densenet201,DenseNet-201,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://densenet201/IMAGENET1K_V1,1x3x224x224,256,imagenet1k,20.01,76.896,ready,TorchVision 预训练权重,可由 API 自动下载,13
mobilenet.v2,MobileNetV2,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://mobilenet_v2/IMAGENET1K_V2,1x3x224x224,232,imagenet1k,3.5,72.154,ready,TorchVision 预训练权重,可由 API 自动下载,14
mobilenet.v3.small,MobileNetV3-Small,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://mobilenet_v3_small/IMAGENET1K_V1,1x3x224x224,256,imagenet1k,2.54,67.668,ready,TorchVision 预训练权重,可由 API 自动下载,15
mobilenet.v3.large,MobileNetV3-Large,cls,C,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://mobilenet_v3_large/IMAGENET1K_V2,1x3x224x224,232,imagenet1k,5.48,75.274,ready,TorchVision 预训练权重,可由 API 自动下载,16
shufflenet.v2.x0.5,ShuffleNetV2-x0.5,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://shufflenet_v2_x0_5/IMAGENET1K_V1,1x3x224x224,256,imagenet1k,1.37,60.552,ready,TorchVision 预训练权重,可由 API 自动下载,17
shufflenet.v2.x1.0,ShuffleNetV2-x1.0,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://shufflenet_v2_x1_0/IMAGENET1K_V1,1x3x224x224,256,imagenet1k,2.28,69.362,ready,TorchVision 预训练权重,可由 API 自动下载,18
shufflenet.v2.x1.5,ShuffleNetV2-x1.5,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://shufflenet_v2_x1_5/IMAGENET1K_V1,1x3x224x224,232,imagenet1k,3.5,72.996,ready,TorchVision 预训练权重,可由 API 自动下载,19
shufflenet.v2.x2.0,ShuffleNetV2-x2.0,cls,C,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://shufflenet_v2_x2_0/IMAGENET1K_V1,1x3x224x224,232,imagenet1k,7.39,76.23,ready,TorchVision 预训练权重,可由 API 自动下载,20
efficientnet.b0,EfficientNet-B0,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://efficientnet_b0/IMAGENET1K_V1,1x3x224x224,256,imagenet1k,5.29,77.692,ready,TorchVision 预训练权重,可由 API 自动下载,21
efficientnet.b1,EfficientNet-B1,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://efficientnet_b1/IMAGENET1K_V2,1x3x240x240,255,imagenet1k,7.79,79.838,ready,TorchVision 预训练权重,可由 API 自动下载,22
efficientnet.b2,EfficientNet-B2,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://efficientnet_b2/IMAGENET1K_V1,1x3x288x288,288,imagenet1k,9.11,80.608,ready,TorchVision 预训练权重,可由 API 自动下载,23
efficientnet.b3,EfficientNet-B3,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://efficientnet_b3/IMAGENET1K_V1,1x3x300x300,320,imagenet1k,12.23,82.008,ready,TorchVision 预训练权重,可由 API 自动下载,24
efficientnet.b4,EfficientNet-B4,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://efficientnet_b4/IMAGENET1K_V1,1x3x380x380,384,imagenet1k,19.34,83.384,ready,TorchVision 预训练权重,可由 API 自动下载,25
efficientnet.b5,EfficientNet-B5,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://efficientnet_b5/IMAGENET1K_V1,1x3x456x456,456,imagenet1k,30.39,83.444,ready,TorchVision 预训练权重,可由 API 自动下载,26
efficientnet.b6,EfficientNet-B6,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://efficientnet_b6/IMAGENET1K_V1,1x3x528x528,528,imagenet1k,43.04,84.008,ready,TorchVision 预训练权重,可由 API 自动下载,27
efficientnet.b7,EfficientNet-B7,cls,C,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://efficientnet_b7/IMAGENET1K_V1,1x3x600x600,600,imagenet1k,66.35,84.122,ready,TorchVision 预训练权重,可由 API 自动下载,28
efficientnet.v2.s,EfficientNetV2-S,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://efficientnet_v2_s/IMAGENET1K_V1,1x3x384x384,384,imagenet1k,21.46,84.228,ready,TorchVision 预训练权重,可由 API 自动下载,29
efficientnet.v2.m,EfficientNetV2-M,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://efficientnet_v2_m/IMAGENET1K_V1,1x3x480x480,480,imagenet1k,54.14,85.112,ready,TorchVision 预训练权重,可由 API 自动下载,30
efficientnet.v2.l,EfficientNetV2-L,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://efficientnet_v2_l/IMAGENET1K_V1,1x3x480x480,480,imagenet1k,118.52,85.808,ready,TorchVision 预训练权重,可由 API 自动下载,31
convnext.tiny,ConvNeXt-Tiny,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://convnext_tiny/IMAGENET1K_V1,1x3x224x224,236,imagenet1k,28.59,82.52,ready,TorchVision 预训练权重,可由 API 自动下载,32
convnext.small,ConvNeXt-Small,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://convnext_small/IMAGENET1K_V1,1x3x224x224,230,imagenet1k,50.22,83.616,ready,TorchVision 预训练权重,可由 API 自动下载,33
convnext.base,ConvNeXt-Base,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://convnext_base/IMAGENET1K_V1,1x3x224x224,232,imagenet1k,88.59,84.062,ready,TorchVision 预训练权重,可由 API 自动下载,34
convnext.large,ConvNeXt-Large,cls,C,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://convnext_large/IMAGENET1K_V1,1x3x224x224,232,imagenet1k,197.77,84.414,ready,TorchVision 预训练权重,可由 API 自动下载,35
swin.t,Swin-T,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://swin_t/IMAGENET1K_V1,1x3x224x224,232,imagenet1k,28.29,81.474,ready,TorchVision 预训练权重,可由 API 自动下载,36
swin.s,Swin-S,cls,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://swin_s/IMAGENET1K_V1,1x3x224x224,246,imagenet1k,49.61,83.196,ready,TorchVision 预训练权重,可由 API 自动下载,37
swin.b,Swin-B,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://swin_b/IMAGENET1K_V1,1x3x224x224,238,imagenet1k,87.77,83.582,ready,TorchVision 预训练权重,可由 API 自动下载,38
vit.b.16,ViT-B/16,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://vit_b_16/IMAGENET1K_V1,1x3x224x224,256,imagenet1k,86.57,81.072,ready,TorchVision 预训练权重,可由 API 自动下载,39
vit.b.32,ViT-B/32,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://vit_b_32/IMAGENET1K_V1,1x3x224x224,256,imagenet1k,88.22,75.912,ready,TorchVision 预训练权重,可由 API 自动下载,40
vit.l.16,ViT-L/16,cls,C,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://vit_l_16/IMAGENET1K_V1,1x3x224x224,242,imagenet1k,304.33,79.662,ready,TorchVision 预训练权重,可由 API 自动下载,41
vit.l.32,ViT-L/32,cls,C,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://vit_l_32/IMAGENET1K_V1,1x3x224x224,256,imagenet1k,306.54,76.972,ready,TorchVision 预训练权重,可由 API 自动下载,42
vit.h.14,ViT-H/14,cls,C,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://vit_h_14/IMAGENET1K_SWAG_E2E_V1,1x3x518x518,518,imagenet1k,633.47,88.552,ready,TorchVision 预训练权重,可由 API 自动下载,43
maxvit.t,MaxViT-T,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://maxvit_t/IMAGENET1K_V1,1x3x224x224,224,imagenet1k,30.92,83.7,ready,TorchVision 预训练权重,可由 API 自动下载,44
regnet.y.400mf,RegNetY-400MF,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://regnet_y_400mf/IMAGENET1K_V2,1x3x224x224,232,imagenet1k,4.34,75.804,ready,TorchVision 预训练权重,可由 API 自动下载,45
regnet.y.800mf,RegNetY-800MF,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://regnet_y_800mf/IMAGENET1K_V2,1x3x224x224,232,imagenet1k,6.43,78.828,ready,TorchVision 预训练权重,可由 API 自动下载,46
regnet.y.1.6gf,RegNetY-1.6GF,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://regnet_y_1_6gf/IMAGENET1K_V2,1x3x224x224,232,imagenet1k,11.2,80.876,ready,TorchVision 预训练权重,可由 API 自动下载,47
regnet.y.3.2gf,RegNetY-3.2GF,cls,B,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://regnet_y_3_2gf/IMAGENET1K_V2,1x3x224x224,232,imagenet1k,19.44,81.982,ready,TorchVision 预训练权重,可由 API 自动下载,48
efficientvit.b0,EfficientViT-B0,cls,A,mit-han-lab/efficientvit https:,custom,https://github.com/mit-han-lab/efficientvit,https://github.com/mit-han-lab/efficientvit,1x3x224x224,,imagenet1k,,,needs_adapter,官方仓库含预训练权重/模型入口,49
efficientvit.b1,EfficientViT-B1,cls,A,mit-han-lab/efficientvit https:,custom,https://github.com/mit-han-lab/efficientvit,https://github.com/mit-han-lab/efficientvit,1x3x224x224,,imagenet1k,,,needs_adapter,官方仓库含预训练权重/模型入口,50
efficientvit.b2,EfficientViT-B2,cls,A,mit-han-lab/efficientvit https:,custom,https://github.com/mit-han-lab/efficientvit,https://github.com/mit-han-lab/efficientvit,1x3x224x224,,imagenet1k,,,needs_adapter,官方仓库含预训练权重/模型入口,51
efficientvit.b3,EfficientViT-B3,cls,A,mit-han-lab/efficientvit https:,custom,https://github.com/mit-han-lab/efficientvit,https://github.com/mit-han-lab/efficientvit,1x3x224x224,,imagenet1k,,,needs_adapter,官方仓库含预训练权重/模型入口,52
faster.r.cnn.r50.fpn,Faster R-CNN R50-FPN,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,53
faster.r.cnn.r101.fpn,Faster R-CNN R101-FPN,det,C,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,54
cascade.r.cnn.r50.fpn,Cascade R-CNN R50-FPN,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,55
cascade.r.cnn.r101.fpn,Cascade R-CNN R101-FPN,det,C,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,56
retinanet.r50.fpn,RetinaNet R50-FPN,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,57
retinanet.r101.fpn,RetinaNet R101-FPN,det,C,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,58
fcos.r50.fpn,FCOS R50-FPN,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,59
atss.r50.fpn,ATSS R50-FPN,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,60
gfl.r50.fpn,GFL R50-FPN,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,61
vfnet.r50.fpn,VFNet R50-FPN,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,62
foveabox.r50.fpn,FoveaBox R50-FPN,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,63
fsaf.r50.fpn,FSAF R50-FPN,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,64
reppoints.r50.fpn,RepPoints R50-FPN,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,65
cornernet.hourglass.104,CornerNet Hourglass-104,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,66
ssd300.vgg16,SSD300-VGG16,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,67
ssdlite320.mobilenetv3.large,SSDLite320-MobileNetV3-Large,det,C,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,68
yolov3.darknet53,YOLOv3-Darknet53,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,69
yolox.tiny,YOLOX-Tiny,det,A,megvii-basedetection/yolox https:,custom,https://github.com/Megvii-BaseDetection/YOLOX,https://github.com/Megvii-BaseDetection/YOLOX,1x3x640x640,,coco2017,,,needs_adapter,官方 release/model zoo,70
yolox.s,YOLOX-S,det,A,megvii-basedetection/yolox https:,custom,https://github.com/Megvii-BaseDetection/YOLOX,https://github.com/Megvii-BaseDetection/YOLOX,1x3x640x640,,coco2017,,,needs_adapter,官方 release/model zoo,71
yolox.m,YOLOX-M,det,B,megvii-basedetection/yolox https:,custom,https://github.com/Megvii-BaseDetection/YOLOX,https://github.com/Megvii-BaseDetection/YOLOX,1x3x640x640,,coco2017,,,needs_adapter,官方 release/model zoo,72
yolox.l,YOLOX-L,det,B,megvii-basedetection/yolox https:,custom,https://github.com/Megvii-BaseDetection/YOLOX,https://github.com/Megvii-BaseDetection/YOLOX,1x3x640x640,,coco2017,,,needs_adapter,官方 release/model zoo,73
yolox.x,YOLOX-X,det,B,megvii-basedetection/yolox https:,custom,https://github.com/Megvii-BaseDetection/YOLOX,https://github.com/Megvii-BaseDetection/YOLOX,1x3x640x640,,coco2017,,,needs_adapter,官方 release/model zoo,74
rtmdet.tiny,RTMDet-Tiny,det,A,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,75
rtmdet.s,RTMDet-S,det,A,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,76
rtmdet.m,RTMDet-M,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,77
rtmdet.l,RTMDet-L,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,78
rtmdet.x,RTMDet-X,det,C,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,79
yolov5n,YOLOv5n,det,A,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov5n.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,80
yolov5s,YOLOv5s,det,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov5s.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,81
yolov5m,YOLOv5m,det,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov5m.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,82
yolov5l,YOLOv5l,det,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov5l.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,83
yolov5x,YOLOv5x,det,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov5x.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,84
yolov8n,YOLOv8n,det,A,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov8n.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,85
yolov8s,YOLOv8s,det,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov8s.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,86
yolov8m,YOLOv8m,det,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov8m.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,87
yolov8l,YOLOv8l,det,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov8l.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,88
yolov8x,YOLOv8x,det,C,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov8x.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,89
yolo11n,YOLO11n,det,A,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolo11n.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,90
yolo11s,YOLO11s,det,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolo11s.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,91
yolo11m,YOLO11m,det,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolo11m.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,92
yolo11l,YOLO11l,det,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolo11l.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,93
yolo11x,YOLO11x,det,C,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolo11x.pt,1x3x640x640,,coco2017,,,ready,Ultralytics 权重名可自动下载,94
detr.r50,DETR-R50,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,95
deformable.detr.r50,Deformable-DETR-R50,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,96
conditional.detr.r50,Conditional-DETR-R50,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,97
dab.detr.r50,DAB-DETR-R50,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,98
dino.r50,DINO-R50,det,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection model zoo,99
rt.detr.r18,RT-DETR-R18,det,A,lyuwenyu/rt-detr https:,custom,https://github.com/lyuwenyu/RT-DETR,https://github.com/lyuwenyu/RT-DETR,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoint/model zoo,100
rt.detr.r50,RT-DETR-R50,det,B,lyuwenyu/rt-detr https:,custom,https://github.com/lyuwenyu/RT-DETR,https://github.com/lyuwenyu/RT-DETR,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoint/model zoo,101
rt.detrv2.r18,RT-DETRv2-R18,det,A,lyuwenyu/rt-detr https:,custom,https://github.com/lyuwenyu/RT-DETR,https://github.com/lyuwenyu/RT-DETR,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoint/model zoo,102
rt.detrv2.r50,RT-DETRv2-R50,det,B,lyuwenyu/rt-detr https:,custom,https://github.com/lyuwenyu/RT-DETR,https://github.com/lyuwenyu/RT-DETR,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoint/model zoo,103
fcn.resnet50,FCN-ResNet50,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,104
fcn.resnet101,FCN-ResNet101,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,105
deeplabv3.resnet50,DeepLabV3-ResNet50,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,106
deeplabv3.resnet101,DeepLabV3-ResNet101,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,107
deeplabv3.mobilenetv3.large,DeepLabV3-MobileNetV3-Large,seg,C,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,108
deeplabv3plus.resnet50,DeepLabV3+-ResNet50,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,109
deeplabv3plus.resnet101,DeepLabV3+-ResNet101,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,110
pspnet.resnet50,PSPNet-ResNet50,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,111
pspnet.resnet101,PSPNet-ResNet101,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,112
upernet.resnet50,UPerNet-ResNet50,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,113
upernet.swin.t,UPerNet-Swin-T,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,114
upernet.swin.s,UPerNet-Swin-S,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,115
upernet.swin.b,UPerNet-Swin-B,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,116
segformer.b0,SegFormer-B0,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,117
segformer.b1,SegFormer-B1,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,118
segformer.b2,SegFormer-B2,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,119
segformer.b3,SegFormer-B3,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,120
segformer.b4,SegFormer-B4,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,121
segformer.b5,SegFormer-B5,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,122
mask2former.swin.t,Mask2Former-Swin-T,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,123
mask2former.swin.s,Mask2Former-Swin-S,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,124
mask2former.swin.b,Mask2Former-Swin-B,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,125
ddrnet.23.slim,DDRNet-23-Slim,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,126
ddrnet.23,DDRNet-23,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,127
ddrnet.39,DDRNet-39,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,128
pidnet.s,PIDNet-S,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,129
pidnet.m,PIDNet-M,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,130
pidnet.l,PIDNet-L,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,131
bisenetv1.r18,BiSeNetV1-R18,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,132
bisenetv2,BiSeNetV2,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,133
stdc1.seg,STDC1-Seg,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,134
stdc2.seg,STDC2-Seg,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,135
fast.scnn,Fast-SCNN,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,136
hrnet.w18,HRNet-W18,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,137
hrnet.w48,HRNet-W48,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,138
ocrnet.hrnet.w18,OCRNet-HRNet-W18,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,139
ocrnet.hrnet.w48,OCRNet-HRNet-W48,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,140
segnext.tiny,SegNeXt-Tiny,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,141
segnext.small,SegNeXt-Small,seg,A,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,142
segnext.base,SegNeXt-Base,seg,B,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,143
segnext.large,SegNeXt-Large,seg,C,mmsegmentation,openmmlab,https://github.com/open-mmlab/mmsegmentation,https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml,1x3x512x1024,,ade20k,,,needs_adapter,MMSegmentation model zoo,144
efficientvit.seg.b0,EfficientViT-Seg-B0,seg,A,mit-han-lab/efficientvit https:,custom,https://github.com/mit-han-lab/efficientvit,https://github.com/mit-han-lab/efficientvit,1x3x512x1024,,ade20k,,,needs_adapter,官方 EfficientViT segmentation 权重,145
efficientvit.seg.b1,EfficientViT-Seg-B1,seg,A,mit-han-lab/efficientvit https:,custom,https://github.com/mit-han-lab/efficientvit,https://github.com/mit-han-lab/efficientvit,1x3x512x1024,,ade20k,,,needs_adapter,官方 EfficientViT segmentation 权重,146
efficientvit.seg.b2,EfficientViT-Seg-B2,seg,A,mit-han-lab/efficientvit https:,custom,https://github.com/mit-han-lab/efficientvit,https://github.com/mit-han-lab/efficientvit,1x3x512x1024,,ade20k,,,needs_adapter,官方 EfficientViT segmentation 权重,147
efficientvit.seg.b3,EfficientViT-Seg-B3,seg,A,mit-han-lab/efficientvit https:,custom,https://github.com/mit-han-lab/efficientvit,https://github.com/mit-han-lab/efficientvit,1x3x512x1024,,ade20k,,,needs_adapter,官方 EfficientViT segmentation 权重,148
mask.r.cnn.r50.fpn,Mask R-CNN R50-FPN,iseg,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection instance segmentation model zoo,149
mask.r.cnn.r101.fpn,Mask R-CNN R101-FPN,iseg,C,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection instance segmentation model zoo,150
cascade.mask.r.cnn.r50.fpn,Cascade Mask R-CNN R50-FPN,iseg,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection instance segmentation model zoo,151
cascade.mask.r.cnn.r101.fpn,Cascade Mask R-CNN R101-FPN,iseg,C,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection instance segmentation model zoo,152
htc.r50.fpn,HTC R50-FPN,iseg,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection instance segmentation model zoo,153
htc.r101.fpn,HTC R101-FPN,iseg,C,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection instance segmentation model zoo,154
solo.r50,SOLO R50,iseg,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection instance segmentation model zoo,155
solov2.r50,SOLOv2 R50,iseg,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection instance segmentation model zoo,156
condinst.r50,CondInst R50,iseg,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection instance segmentation model zoo,157
queryinst.r50,QueryInst R50,iseg,B,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection instance segmentation model zoo,158
mask2former.swin.t.instance,Mask2Former-Swin-T-Instance,iseg,A,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection instance segmentation model zoo,159
mask2former.swin.s.instance,Mask2Former-Swin-S-Instance,iseg,A,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection instance segmentation model zoo,160
mask2former.swin.b.instance,Mask2Former-Swin-B-Instance,iseg,A,mmdetection,openmmlab,https://github.com/open-mmlab/mmdetection,https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md,1x3x640x640,,coco2017,,,needs_adapter,MMDetection instance segmentation model zoo,161
yolov8n.seg,YOLOv8n-seg,iseg,A,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov8n-seg.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics 自动/官方下载,162
yolov8s.seg,YOLOv8s-seg,iseg,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov8s-seg.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics 自动/官方下载,163
yolov8m.seg,YOLOv8m-seg,iseg,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov8m-seg.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics 自动/官方下载,164
yolov8l.seg,YOLOv8l-seg,iseg,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov8l-seg.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics 自动/官方下载,165
yolov8x.seg,YOLOv8x-seg,iseg,C,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolov8x-seg.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics 自动/官方下载,166
yolo11n.seg,YOLO11n-seg,iseg,A,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolo11n-seg.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics 自动/官方下载,167
yolo11s.seg,YOLO11s-seg,iseg,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolo11s-seg.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics 自动/官方下载,168
yolo11m.seg,YOLO11m-seg,iseg,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolo11m-seg.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics 自动/官方下载,169
yolo11l.seg,YOLO11l-seg,iseg,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolo11l-seg.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics 自动/官方下载,170
yolo11x.seg,YOLO11x-seg,iseg,C,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolo11x-seg.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics 自动/官方下载,171
sam.vit.b,SAM-ViT-B,iseg,B,facebookresearch/segment-anything https:,custom,https://github.com/facebookresearch/segment-anything,https://github.com/facebookresearch/segment-anything,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoints,172
sam.vit.l,SAM-ViT-L,iseg,C,facebookresearch/segment-anything https:,custom,https://github.com/facebookresearch/segment-anything,https://github.com/facebookresearch/segment-anything,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoints,173
sam.vit.h,SAM-ViT-H,iseg,C,facebookresearch/segment-anything https:,custom,https://github.com/facebookresearch/segment-anything,https://github.com/facebookresearch/segment-anything,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoints,174
sam2.1.hiera.tiny,SAM2.1-Hiera-Tiny,iseg,A,facebookresearch/sam2 https:,custom,https://github.com/facebookresearch/sam2,https://github.com/facebookresearch/sam2,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoints,175
sam2.1.hiera.small,SAM2.1-Hiera-Small,iseg,A,facebookresearch/sam2 https:,custom,https://github.com/facebookresearch/sam2,https://github.com/facebookresearch/sam2,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoints,176
sam2.1.hiera.baseplus,SAM2.1-Hiera-Base+,iseg,B,facebookresearch/sam2 https:,custom,https://github.com/facebookresearch/sam2,https://github.com/facebookresearch/sam2,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoints,177
sam2.1.hiera.large,SAM2.1-Hiera-Large,iseg,C,facebookresearch/sam2 https:,custom,https://github.com/facebookresearch/sam2,https://github.com/facebookresearch/sam2,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoints,178
mobilesam,MobileSAM,iseg,A,chaoningzhang/mobilesam https:,custom,https://github.com/ChaoningZhang/MobileSAM,https://github.com/ChaoningZhang/MobileSAM,1x3x640x640,,coco2017,,,needs_adapter,官方权重,179
edgesam,EdgeSAM,iseg,B,chongzhou96/edgesam https:,custom,https://github.com/chongzhou96/EdgeSAM,https://github.com/chongzhou96/EdgeSAM,1x3x640x640,,coco2017,,,needs_adapter,官方权重/ONNX,180
efficientsam.ti,EfficientSAM-Ti,iseg,A,yformer/efficientsam https:,custom,https://github.com/yformer/EfficientSAM,https://github.com/yformer/EfficientSAM,1x3x640x640,,coco2017,,,needs_adapter,weights/ 目录含 checkpoint,181
efficientsam.s,EfficientSAM-S,iseg,A,yformer/efficientsam https:,custom,https://github.com/yformer/EfficientSAM,https://github.com/yformer/EfficientSAM,1x3x640x640,,coco2017,,,needs_adapter,weights/ 目录含 checkpoint,182
efficientvit.sam.l0,EfficientViT-SAM-L0,iseg,A,mit-han-lab/efficientvit https:,custom,https://github.com/mit-han-lab/efficientvit,https://github.com/mit-han-lab/efficientvit/tree/master/applications/efficientvit_sam,1x3x640x640,,coco2017,,,needs_adapter,官方 pretrained models,183
efficientvit.sam.l1,EfficientViT-SAM-L1,iseg,A,mit-han-lab/efficientvit https:,custom,https://github.com/mit-han-lab/efficientvit,https://github.com/mit-han-lab/efficientvit/tree/master/applications/efficientvit_sam,1x3x640x640,,coco2017,,,needs_adapter,官方 pretrained models,184
efficientvit.sam.l2,EfficientViT-SAM-L2,iseg,A,mit-han-lab/efficientvit https:,custom,https://github.com/mit-han-lab/efficientvit,https://github.com/mit-han-lab/efficientvit/tree/master/applications/efficientvit_sam,1x3x640x640,,coco2017,,,needs_adapter,官方 pretrained models,185
fastsam.s,FastSAM-s,iseg,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://FastSAM-s.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics 自动/官方下载,186
fastsam.x,FastSAM-x,iseg,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://FastSAM-x.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics 自动/官方下载,187
hq.sam.vit.b,HQ-SAM-ViT-B,iseg,A,syscv/sam-hq https:,custom,https://github.com/SysCV/sam-hq,https://github.com/SysCV/sam-hq,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoints,188
hq.sam.vit.l,HQ-SAM-ViT-L,iseg,C,syscv/sam-hq https:,custom,https://github.com/SysCV/sam-hq,https://github.com/SysCV/sam-hq,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoints,189
hq.sam.vit.h,HQ-SAM-ViT-H,iseg,C,syscv/sam-hq https:,custom,https://github.com/SysCV/sam-hq,https://github.com/SysCV/sam-hq,1x3x640x640,,coco2017,,,needs_adapter,官方 checkpoints,190
repvit.sam,RepViT-SAM,iseg,A,thu-mig/repvit https:,custom,https://github.com/THU-MIG/RepViT,https://github.com/THU-MIG/RepViT,1x3x640x640,,coco2017,,,needs_adapter,RepViT-SAM 权重入口,191
tinysam,TinySAM,iseg,A,xinghaochen/tinysam https:,custom,https://github.com/xinghaochen/TinySAM,https://github.com/xinghaochen/TinySAM,1x3x640x640,,coco2017,,,needs_adapter,官方权重,192
slimsam.50,SlimSAM-50,iseg,B,czg1225/slimsam https:,custom,https://github.com/czg1225/SlimSAM,https://github.com/czg1225/SlimSAM,1x3x640x640,,coco2017,,,needs_adapter,SlimSAM-50/77 官方权重,193
slimsam.77,SlimSAM-77,iseg,B,czg1225/slimsam https:,custom,https://github.com/czg1225/SlimSAM,https://github.com/czg1225/SlimSAM,1x3x640x640,,coco2017,,,needs_adapter,SlimSAM-50/77 官方权重,194
efficienttam.ti,EfficientTAM-Ti,iseg,A,yformer/efficienttam https:,custom,https://github.com/yformer/EfficientTAM,https://github.com/yformer/EfficientTAM,1x3x640x640,,coco2017,,,needs_adapter,checkpoints/download_checkpoints.sh,195
efficienttam.s,EfficientTAM-S,iseg,A,yformer/efficienttam https:,custom,https://github.com/yformer/EfficientTAM,https://github.com/yformer/EfficientTAM,1x3x640x640,,coco2017,,,needs_adapter,checkpoints/download_checkpoints.sh,196
simplebaseline.r50,SimpleBaseline-R50,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,197
simplebaseline.r101,SimpleBaseline-R101,pose,C,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,198
simplebaseline.r152,SimpleBaseline-R152,pose,C,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,199
hrnet.w32,HRNet-W32,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,200
hrnet.w48.2,HRNet-W48,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,201
litehrnet.18,LiteHRNet-18,pose,A,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,202
litehrnet.30,LiteHRNet-30,pose,A,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,203
rtmpose.tiny,RTMPose-Tiny,pose,A,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,204
rtmpose.s,RTMPose-S,pose,A,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,205
rtmpose.m,RTMPose-M,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,206
rtmpose.l,RTMPose-L,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,207
rtmpose.x,RTMPose-X,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,208
vitpose.s,ViTPose-S,pose,A,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,209
vitpose.b,ViTPose-B,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,210
vitpose.l,ViTPose-L,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,211
vitpose.h,ViTPose-H,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,212
mspn.r50,MSPN-R50,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,213
mspn.r101,MSPN-R101,pose,C,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,214
higherhrnet.w32,HigherHRNet-W32,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,215
higherhrnet.w48,HigherHRNet-W48,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,216
dekr.w32,DEKR-W32,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,217
dekr.w48,DEKR-W48,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,218
simcc.resnet50,SimCC-ResNet50,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,219
rtmo.s,RTMO-S,pose,A,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,220
rtmo.m,RTMO-M,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,221
rtmo.l,RTMO-L,pose,B,mmpose,openmmlab,https://github.com/open-mmlab/mmpose,https://github.com/open-mmlab/mmpose/blob/main/model-index.yml,1x3x640x640,,coco2017,,,needs_adapter,MMPose model zoo,222
yolo11n.pose,YOLO11n-pose,pose,A,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolo11n-pose.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics pose weights,223
yolo11s.pose,YOLO11s-pose,pose,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolo11s-pose.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics pose weights,224
yolo11m.pose,YOLO11m-pose,pose,B,ultralytics,ultralytics,https://docs.ultralytics.com/models/,ultralytics://yolo11m-pose.pt,1x3x640x640,,coco2017,,,needs_eval,Ultralytics pose weights,225
openpose.body25,OpenPose-Body25,pose,B,cmu-perceptual-computing-lab/openpose https:,custom,https://github.com/CMU-Perceptual-Computing-Lab/openpose,https://github.com/CMU-Perceptual-Computing-Lab/openpose,1x3x640x640,,coco2017,,,needs_adapter,官方模型下载脚本/文档,226
midas.small,MiDaS-Small,mde,A,isl-org/midas https:,custom,https://github.com/isl-org/MiDaS,https://github.com/isl-org/MiDaS,1x3x518x518,,kitti2015,,,needs_adapter,官方 model zoo,227
dpt.hybrid,DPT-Hybrid,mde,B,isl-org/midas https:,custom,https://github.com/isl-org/MiDaS,https://github.com/isl-org/MiDaS,1x3x518x518,,kitti2015,,,needs_adapter,官方 model zoo,228
dpt.large,DPT-Large,mde,C,isl-org/midas https:,custom,https://github.com/isl-org/MiDaS,https://github.com/isl-org/MiDaS,1x3x518x518,,kitti2015,,,needs_adapter,官方 model zoo,229
depth.anything.v1.small,Depth-Anything-V1-Small,mde,A,liheyoung/depth-anything https:,custom,https://github.com/LiheYoung/Depth-Anything,https://github.com/LiheYoung/Depth-Anything,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoints/HF,230
depth.anything.v1.base,Depth-Anything-V1-Base,mde,B,liheyoung/depth-anything https:,custom,https://github.com/LiheYoung/Depth-Anything,https://github.com/LiheYoung/Depth-Anything,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoints/HF,231
depth.anything.v1.large,Depth-Anything-V1-Large,mde,C,liheyoung/depth-anything https:,custom,https://github.com/LiheYoung/Depth-Anything,https://github.com/LiheYoung/Depth-Anything,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoints/HF,232
depth.anything.v2.small,Depth-Anything-V2-Small,mde,A,depthanything/depth-anything-v2 https:,custom,https://github.com/DepthAnything/Depth-Anything-V2,https://github.com/DepthAnything/Depth-Anything-V2,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoints/HF,233
depth.anything.v2.base,Depth-Anything-V2-Base,mde,B,depthanything/depth-anything-v2 https:,custom,https://github.com/DepthAnything/Depth-Anything-V2,https://github.com/DepthAnything/Depth-Anything-V2,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoints/HF,234
depth.anything.v2.large,Depth-Anything-V2-Large,mde,C,depthanything/depth-anything-v2 https:,custom,https://github.com/DepthAnything/Depth-Anything-V2,https://github.com/DepthAnything/Depth-Anything-V2,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoints/HF,235
depth.anything.v2.metric.hypersim.small,Depth-Anything-V2-Metric-Hypersim-Small,mde,A,depthanything/depth-anything-v2 https:,custom,https://github.com/DepthAnything/Depth-Anything-V2,https://github.com/DepthAnything/Depth-Anything-V2,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoints/HF,236
depth.anything.v2.metric.hypersim.base,Depth-Anything-V2-Metric-Hypersim-Base,mde,B,depthanything/depth-anything-v2 https:,custom,https://github.com/DepthAnything/Depth-Anything-V2,https://github.com/DepthAnything/Depth-Anything-V2,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoints/HF,237
depth.anything.v2.metric.hypersim.large,Depth-Anything-V2-Metric-Hypersim-Large,mde,C,depthanything/depth-anything-v2 https:,custom,https://github.com/DepthAnything/Depth-Anything-V2,https://github.com/DepthAnything/Depth-Anything-V2,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoints/HF,238
depth.anything.v2.metric.vkitti.small,Depth-Anything-V2-Metric-VKITTI-Small,mde,A,depthanything/depth-anything-v2 https:,custom,https://github.com/DepthAnything/Depth-Anything-V2,https://github.com/DepthAnything/Depth-Anything-V2,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoints/HF,239
depth.anything.v2.metric.vkitti.base,Depth-Anything-V2-Metric-VKITTI-Base,mde,B,depthanything/depth-anything-v2 https:,custom,https://github.com/DepthAnything/Depth-Anything-V2,https://github.com/DepthAnything/Depth-Anything-V2,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoints/HF,240
depth.anything.v2.metric.vkitti.large,Depth-Anything-V2-Metric-VKITTI-Large,mde,C,depthanything/depth-anything-v2 https:,custom,https://github.com/DepthAnything/Depth-Anything-V2,https://github.com/DepthAnything/Depth-Anything-V2,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoints/HF,241
zoedepth.n,ZoeDepth-N,mde,B,isl-org/zoedepth https:,custom,https://github.com/isl-org/ZoeDepth,https://github.com/isl-org/ZoeDepth,1x3x518x518,,kitti2015,,,needs_adapter,官方 torch hub/checkpoints,242
zoedepth.k,ZoeDepth-K,mde,B,isl-org/zoedepth https:,custom,https://github.com/isl-org/ZoeDepth,https://github.com/isl-org/ZoeDepth,1x3x518x518,,kitti2015,,,needs_adapter,官方 torch hub/checkpoints,243
zoedepth.nk,ZoeDepth-NK,mde,B,isl-org/zoedepth https:,custom,https://github.com/isl-org/ZoeDepth,https://github.com/isl-org/ZoeDepth,1x3x518x518,,kitti2015,,,needs_adapter,官方 torch hub/checkpoints,244
metric3dv2.vit.small,Metric3Dv2-ViT-Small,mde,A,yvanyin/metric3d https:,custom,https://github.com/YvanYin/Metric3D,https://github.com/YvanYin/Metric3D,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoint / torch.hub / ONNX,245
metric3dv2.vit.large,Metric3Dv2-ViT-Large,mde,C,yvanyin/metric3d https:,custom,https://github.com/YvanYin/Metric3D,https://github.com/YvanYin/Metric3D,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoint / torch.hub / ONNX,246
metric3dv2.vit.giant2,Metric3Dv2-ViT-Giant2,mde,C,yvanyin/metric3d https:,custom,https://github.com/YvanYin/Metric3D,https://github.com/YvanYin/Metric3D,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoint / torch.hub / ONNX,247
metric3d.v1.convnext.tiny,Metric3D-v1-ConvNeXt-Tiny,mde,A,yvanyin/metric3d https:,custom,https://github.com/YvanYin/Metric3D,https://github.com/YvanYin/Metric3D,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoint / torch.hub / ONNX,248
metric3d.v1.convnext.large,Metric3D-v1-ConvNeXt-Large,mde,C,yvanyin/metric3d https:,custom,https://github.com/YvanYin/Metric3D,https://github.com/YvanYin/Metric3D,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoint / torch.hub / ONNX,249
unidepth.v1.convnext.l,UniDepth-V1-ConvNeXt-L,mde,B,lpiccinelli-eth/unidepth https:,custom,https://github.com/lpiccinelli-eth/UniDepth,https://huggingface.co/lpiccinelli,1x3x518x518,,kitti2015,,,needs_adapter,官方 Hugging Face model zoo,250
unidepth.v1.vit.l,UniDepth-V1-ViT-L,mde,C,lpiccinelli-eth/unidepth https:,custom,https://github.com/lpiccinelli-eth/UniDepth,https://huggingface.co/lpiccinelli,1x3x518x518,,kitti2015,,,needs_adapter,官方 Hugging Face model zoo,251
unidepth.v2.vit.s,UniDepth-V2-ViT-S,mde,A,lpiccinelli-eth/unidepth https:,custom,https://github.com/lpiccinelli-eth/UniDepth,https://huggingface.co/lpiccinelli,1x3x518x518,,kitti2015,,,needs_adapter,官方 Hugging Face model zoo,252
unidepth.v2.vit.b,UniDepth-V2-ViT-B,mde,B,lpiccinelli-eth/unidepth https:,custom,https://github.com/lpiccinelli-eth/UniDepth,https://huggingface.co/lpiccinelli,1x3x518x518,,kitti2015,,,needs_adapter,官方 Hugging Face model zoo,253
unidepth.v2.vit.l,UniDepth-V2-ViT-L,mde,C,lpiccinelli-eth/unidepth https:,custom,https://github.com/lpiccinelli-eth/UniDepth,https://huggingface.co/lpiccinelli,1x3x518x518,,kitti2015,,,needs_adapter,官方 Hugging Face model zoo,254
depthpro,DepthPro,mde,B,apple/ml-depth-pro https:,custom,https://github.com/apple/ml-depth-pro,https://github.com/apple/ml-depth-pro,1x3x518x518,,kitti2015,,,needs_adapter,官方 checkpoint,255
psmnet,PSMNet,stereo,A,jiarenchang/psmnet https:,custom,https://github.com/JiaRenChang/PSMNet,https://github.com/JiaRenChang/PSMNet,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,256
gwcnet,GwcNet,stereo,A,xy-guo/gwcnet https:,custom,https://github.com/xy-guo/GwcNet,https://github.com/xy-guo/GwcNet,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,257
ganet,GANet,stereo,A,feihuzhang/ganet https:,custom,https://github.com/feihuzhang/GANet,https://github.com/feihuzhang/GANet,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,258
aanet,AANet,stereo,A,haofeixu/aanet https:,custom,https://github.com/haofeixu/aanet,https://github.com/haofeixu/aanet,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,259
leastereo,LEAStereo,stereo,A,xueliancheng/leastereo https:,custom,https://github.com/XuelianCheng/LEAStereo,https://github.com/XuelianCheng/LEAStereo,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,260
anynet,AnyNet,stereo,A,mileyan/anynet https:,custom,https://github.com/mileyan/AnyNet,https://github.com/mileyan/AnyNet,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,261
hitnet,HITNet,stereo,A,google-research/google-research,custom,https://github.com/google-research/google-research/tree/master/hitnet,https://github.com/google-research/google-research/tree/master/hitnet,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,262
mobilestereonet.2d,MobileStereoNet-2D,stereo,A,cogsys-tuebingen/mobilestereonet https:,custom,https://github.com/cogsys-tuebingen/mobilestereonet,https://github.com/cogsys-tuebingen/mobilestereonet,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,263
mobilestereonet.3d,MobileStereoNet-3D,stereo,A,cogsys-tuebingen/mobilestereonet https:,custom,https://github.com/cogsys-tuebingen/mobilestereonet,https://github.com/cogsys-tuebingen/mobilestereonet,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,264
raft.stereo,RAFT-Stereo,stereo,A,princeton-vl/raft-stereo https:,custom,https://github.com/princeton-vl/RAFT-Stereo,https://github.com/princeton-vl/RAFT-Stereo,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,265
iraftstereo.rvc,iRAFTStereo-RVC,stereo,A,princeton-vl/raft-stereo https:,custom,https://github.com/princeton-vl/RAFT-Stereo,https://github.com/princeton-vl/RAFT-Stereo,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,266
crestereo,CREStereo,stereo,A,megvii-research/crestereo https:,custom,https://github.com/megvii-research/CREStereo,https://github.com/megvii-research/CREStereo,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,267
igev.stereo,IGEV-Stereo,stereo,A,gangweix/igev https:,custom,https://github.com/gangweiX/IGEV,https://github.com/gangweiX/IGEV,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,268
unimatch.stereo,UniMatch-Stereo,stereo,A,autonomousvision/unimatch https:,custom,https://github.com/autonomousvision/unimatch,https://github.com/autonomousvision/unimatch,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,269
sttr,STTR,stereo,A,mli0603/stereo-transformer https:,custom,https://github.com/mli0603/stereo-transformer,https://github.com/mli0603/stereo-transformer,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,270
foundationstereo,FoundationStereo,stereo,C,nvlabs/foundationstereo https:,custom,https://github.com/NVlabs/FoundationStereo,https://github.com/NVlabs/FoundationStereo,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,271
fast.foundationstereo,Fast-FoundationStereo,stereo,C,nvlabs/fast-foundationstereo https:,custom,https://github.com/NVlabs/Fast-FoundationStereo,https://github.com/NVlabs/Fast-FoundationStereo,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,272
waft.stereo,WAFT-Stereo,stereo,A,princeton-vl/waft-stereo https:,custom,https://github.com/princeton-vl/WAFT-Stereo,https://github.com/princeton-vl/WAFT-Stereo,1x3x384x1248,,kitti2015,,,needs_adapter,官方仓库;按 README 下载 checkpoint,273
flownet2,FlowNet2,flow,A,nvidia/flownet2-pytorch https:,custom,https://github.com/NVIDIA/flownet2-pytorch,https://github.com/NVIDIA/flownet2-pytorch,1x3x384x1024,,kitti2015,,,needs_adapter,官方仓库/model zoo,274
spynet,SpyNet,flow,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,https://docs.pytorch.org/vision/main/models,1x3x384x1024,,kitti2015,,,needs_adapter,官方仓库/model zoo,275
pwc.net,PWC-Net,flow,A,nvlabs/pwc-net https:,custom,https://github.com/NVlabs/PWC-Net,https://github.com/NVlabs/PWC-Net,1x3x384x1024,,kitti2015,,,needs_adapter,官方仓库/model zoo,276
liteflownet2,LiteFlowNet2,flow,A,twhui/liteflownet2 https:,custom,https://github.com/twhui/LiteFlowNet2,https://github.com/twhui/LiteFlowNet2,1x3x384x1024,,kitti2015,,,needs_adapter,官方仓库/model zoo,277
raft.large,RAFT-Large,flow,C,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://raft_large/C_T_SKHT_V2,1x3x384x1024,256,kitti2015,5.26,,needs_adapter,官方仓库/model zoo,278
raft.small,RAFT-Small,flow,A,torchvision,torchvision,https://docs.pytorch.org/vision/main/models,torchvision://raft_small/C_T_V2,1x3x384x1024,256,kitti2015,0.99,,needs_adapter,官方仓库/model zoo,279
gma,GMA,flow,A,zacjiang/gma https:,custom,https://github.com/zacjiang/GMA,https://github.com/zacjiang/GMA,1x3x384x1024,,kitti2015,,,needs_adapter,官方仓库/model zoo,280
gmflow,GMFlow,flow,A,haofeixu/gmflow https:,custom,https://github.com/haofeixu/gmflow,https://github.com/haofeixu/gmflow,1x3x384x1024,,kitti2015,,,needs_adapter,官方仓库/model zoo,281
flowformer,FlowFormer,flow,A,drinkingcoder/flowformer-official https:,custom,https://github.com/drinkingcoder/FlowFormer-Official,https://github.com/drinkingcoder/FlowFormer-Official,1x3x384x1024,,kitti2015,,,needs_adapter,官方仓库/model zoo,282
flowformerplusplus,FlowFormer++,flow,C,xiaoyushi97/flowformerplusplus https:,custom,https://github.com/XiaoyuShi97/FlowFormerPlusPlus,https://github.com/XiaoyuShi97/FlowFormerPlusPlus,1x3x384x1024,,kitti2015,,,needs_adapter,官方仓库/model zoo,283
unimatch.flow,UniMatch-Flow,flow,A,autonomousvision/unimatch https:,custom,https://github.com/autonomousvision/unimatch,https://github.com/autonomousvision/unimatch,1x3x384x1024,,kitti2015,,,needs_adapter,官方仓库/model zoo,284
sea.raft,SEA-RAFT,flow,A,princeton-vl/sea-raft https:,custom,https://github.com/princeton-vl/SEA-RAFT,https://github.com/princeton-vl/SEA-RAFT,1x3x384x1024,,kitti2015,,,needs_adapter,官方仓库/model zoo,285
memflow,MemFlow,flow,A,dqiaole/memflow https:,custom,https://github.com/DQiaole/MemFlow,https://github.com/DQiaole/MemFlow,1x3x384x1024,,kitti2015,,,needs_adapter,官方仓库/model zoo,286
flow1d,Flow1D,flow,A,haofeixu/flow1d https:,custom,https://github.com/haofeixu/flow1d,https://github.com/haofeixu/flow1d,1x3x384x1024,,kitti2015,,,needs_adapter,官方仓库/model zoo,287
sift,SIFT,feat,A,opencv,opencv,https://docs.opencv.org/4.x/,https://docs.opencv.org/4.x/,1x1x480x640,,hpatches,,,skip,经典算子,非神经网络,不量化,288
orb,ORB,feat,A,opencv,opencv,https://docs.opencv.org/4.x/,https://docs.opencv.org/4.x/,1x1x480x640,,hpatches,,,skip,经典算子,非神经网络,不量化,289
akaze,AKAZE,feat,A,opencv,opencv,https://docs.opencv.org/4.x/,https://docs.opencv.org/4.x/,1x1x480x640,,hpatches,,,skip,经典算子,非神经网络,不量化,290
superpoint,SuperPoint,feat,A,magicleap/supergluepretrainednetwork https:,custom,https://github.com/magicleap/SuperGluePretrainedNetwork,https://github.com/magicleap/SuperGluePretrainedNetwork,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,291
disk,DISK,feat,A,cvg/lightglue https:,custom,https://github.com/cvg/LightGlue,https://github.com/cvg/LightGlue,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,292
aliked,ALIKED,feat,A,shiaoming/aliked https:,custom,https://github.com/Shiaoming/ALIKED,https://github.com/Shiaoming/ALIKED,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,293
xfeat,XFeat,feat,A,verlab/accelerated_features https:,custom,https://github.com/verlab/accelerated_features,https://github.com/verlab/accelerated_features,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,294
d2.net,D2-Net,feat,A,mihaidusmanu/d2-net https:,custom,https://github.com/mihaidusmanu/d2-net,https://github.com/mihaidusmanu/d2-net,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,295
r2d2,R2D2,feat,A,naver/r2d2 https:,custom,https://github.com/naver/r2d2,https://github.com/naver/r2d2,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,296
aslfeat,ASLFeat,feat,A,lzx551402/aslfeat https:,custom,https://github.com/lzx551402/ASLFeat,https://github.com/lzx551402/ASLFeat,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,297
keynetplushardnet,KeyNet+HardNet,feat,A,axelbarroso/key.net https:,custom,https://github.com/axelBarroso/Key.Net,https://github.com/axelBarroso/Key.Net,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,298
dedode.v2,DeDoDe-v2,feat,A,parskatt/dedode https:,custom,https://github.com/Parskatt/DeDoDe,https://github.com/Parskatt/DeDoDe,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,299
superglue.indoor,SuperGlue-Indoor,feat,A,magicleap/supergluepretrainednetwork https:,custom,https://github.com/magicleap/SuperGluePretrainedNetwork,https://github.com/magicleap/SuperGluePretrainedNetwork,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,300
superglue.outdoor,SuperGlue-Outdoor,feat,A,magicleap/supergluepretrainednetwork https:,custom,https://github.com/magicleap/SuperGluePretrainedNetwork,https://github.com/magicleap/SuperGluePretrainedNetwork,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,301
lightglue.superpoint,LightGlue-SuperPoint,feat,A,cvg/lightglue https:,custom,https://github.com/cvg/LightGlue,https://github.com/cvg/LightGlue,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,302
lightglue.disk,LightGlue-DISK,feat,A,cvg/lightglue https:,custom,https://github.com/cvg/LightGlue,https://github.com/cvg/LightGlue,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,303
lightglue.aliked,LightGlue-ALIKED,feat,A,cvg/lightglue https:,custom,https://github.com/cvg/LightGlue,https://github.com/cvg/LightGlue,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,304
lightglue.sift,LightGlue-SIFT,feat,A,cvg/lightglue https:,custom,https://github.com/cvg/LightGlue,https://github.com/cvg/LightGlue,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,305
loftr.indoor,LoFTR-Indoor,feat,A,zju3dv/loftr https:,custom,https://github.com/zju3dv/LoFTR,https://github.com/zju3dv/LoFTR,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,306
loftr.outdoor,LoFTR-Outdoor,feat,A,zju3dv/loftr https:,custom,https://github.com/zju3dv/LoFTR,https://github.com/zju3dv/LoFTR,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,307
matchformer.indoor,MatchFormer-Indoor,feat,A,jamycheung/matchformer https:,custom,https://github.com/jamycheung/MatchFormer,https://github.com/jamycheung/MatchFormer,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,308
matchformer.outdoor,MatchFormer-Outdoor,feat,A,jamycheung/matchformer https:,custom,https://github.com/jamycheung/MatchFormer,https://github.com/jamycheung/MatchFormer,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,309
aspanformer.indoor,ASpanFormer-Indoor,feat,A,apple/ml-aspanformer https:,custom,https://github.com/apple/ml-aspanformer,https://github.com/apple/ml-aspanformer,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,310
aspanformer.outdoor,ASpanFormer-Outdoor,feat,A,apple/ml-aspanformer https:,custom,https://github.com/apple/ml-aspanformer,https://github.com/apple/ml-aspanformer,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,311
dkm,DKM,feat,A,parskatt/dkm https:,custom,https://github.com/Parskatt/DKM,https://github.com/Parskatt/DKM,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,312
roma,RoMa,feat,A,parskatt/roma https:,custom,https://github.com/Parskatt/RoMa,https://github.com/Parskatt/RoMa,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,313
gluestick,GlueStick,feat,A,cvg/gluestick https:,custom,https://github.com/cvg/GlueStick,https://github.com/cvg/GlueStick,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,314
xfeatpluslighterglue,XFeat+LighterGlue,feat,A,verlab/accelerated_features https:,custom,https://github.com/verlab/accelerated_features,https://github.com/verlab/accelerated_features,1x1x480x640,,hpatches,,,needs_adapter,官方权重/仓库,315
monoscene.semantickitti,MonoScene-SemanticKITTI,occ,A,astra-vision/monoscene https:,custom,https://github.com/astra-vision/MonoScene,https://github.com/astra-vision/MonoScene,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,316
monoscene.nyuv2,MonoScene-NYUv2,occ,A,astra-vision/monoscene https:,custom,https://github.com/astra-vision/MonoScene,https://github.com/astra-vision/MonoScene,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,317
lmscnet.semantickitti,LMSCNet-SemanticKITTI,occ,A,astra-vision/lmscnet https:,custom,https://github.com/astra-vision/LMSCNet,https://github.com/astra-vision/LMSCNet,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,318
tpvformer,TPVFormer,occ,A,wzzheng/tpvformer https:,custom,https://github.com/wzzheng/TPVFormer,https://github.com/wzzheng/TPVFormer,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,319
voxformer.s,VoxFormer-S,occ,A,nvlabs/voxformer https:,custom,https://github.com/NVlabs/VoxFormer,https://github.com/NVlabs/VoxFormer,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,320
voxformer.t,VoxFormer-T,occ,A,nvlabs/voxformer https:,custom,https://github.com/NVlabs/VoxFormer,https://github.com/NVlabs/VoxFormer,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,321
voxformer.t.3d,VoxFormer-T-3D,occ,A,nvlabs/voxformer https:,custom,https://github.com/NVlabs/VoxFormer,https://github.com/NVlabs/VoxFormer,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,322
occformer,OccFormer,occ,A,zhangyp15/occformer https:,custom,https://github.com/zhangyp15/OccFormer,https://github.com/zhangyp15/OccFormer,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,323
surroundocc,SurroundOcc,occ,A,weiyithu/surroundocc https:,custom,https://github.com/weiyithu/SurroundOcc,https://github.com/weiyithu/SurroundOcc,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,324
occnet.r50,OccNet-R50,occ,A,opendrivelab/occnet https:,custom,https://github.com/OpenDriveLab/OccNet,https://github.com/OpenDriveLab/OccNet,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,325
flashocc.m0,FlashOcc-M0,occ,A,yzichen/flashocc https:,custom,https://github.com/Yzichen/FlashOCC,https://github.com/Yzichen/FlashOCC,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,326
flashocc.m1,FlashOcc-M1,occ,A,yzichen/flashocc https:,custom,https://github.com/Yzichen/FlashOCC,https://github.com/Yzichen/FlashOCC,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,327
panoptic.flashocc,Panoptic-FlashOcc,occ,A,yzichen/flashocc https:,custom,https://github.com/Yzichen/FlashOCC,https://github.com/Yzichen/FlashOCC,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,328
sparseocc.vision.sjtu,SparseOcc-VISION-SJTU,occ,A,vision-sjtu/sparseocc https:,custom,https://github.com/VISION-SJTU/SparseOcc,https://github.com/VISION-SJTU/SparseOcc,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,329
sparseocc.mcg.nju,SparseOcc-MCG-NJU,occ,A,mcg-nju/sparseocc https:,custom,https://github.com/MCG-NJU/SparseOcc,https://github.com/MCG-NJU/SparseOcc,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,330
renderocc,RenderOcc,occ,A,pmj110119/renderocc https:,custom,https://github.com/pmj110119/RenderOcc,https://github.com/pmj110119/RenderOcc,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,331
selfocc,SelfOcc,occ,A,huang-yh/selfocc https:,custom,https://github.com/huang-yh/SelfOcc,https://github.com/huang-yh/SelfOcc,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,332
gaussianformer,GaussianFormer,occ,A,huang-yh/gaussianformer https:,custom,https://github.com/huang-yh/GaussianFormer,https://github.com/huang-yh/GaussianFormer,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,333
gaussianformer.2,GaussianFormer-2,occ,C,huang-yh/gaussianformer https:,custom,https://github.com/huang-yh/GaussianFormer,https://github.com/huang-yh/GaussianFormer,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,334
gaussianocc,GaussianOcc,occ,A,ganwanshui/gaussianocc https:,custom,https://github.com/GANWANSHUI/GaussianOcc,https://github.com/GANWANSHUI/GaussianOcc,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,335
ctf.occ,CTF-Occ,occ,A,tsinghua-mars-lab/occ3d https:,custom,https://github.com/Tsinghua-MARS-Lab/Occ3D,https://github.com/Tsinghua-MARS-Lab/Occ3D,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,336
bevdetocc,BEVDetOCC,occ,A,huangjunjie2017/bevdet https:,custom,https://github.com/HuangJunJie2017/BEVDet,https://github.com/HuangJunJie2017/BEVDet,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,337
bevdetocc.4d.stereo,BEVDetOCC-4D-Stereo,occ,A,yzichen/flashocc https:,custom,https://github.com/Yzichen/FlashOCC,https://github.com/Yzichen/FlashOCC,1x3x256x704,,kitti2015,,,needs_adapter,官方仓库/model zoo,338
1 model_id name task priority family source repo weights input_spec resize dataset params_m official_metric status notes excel_id
2 resnet18 ResNet-18 cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://resnet18/IMAGENET1K_V1 1x3x224x224 256 imagenet1k 11.69 69.758 ready TorchVision 预训练权重,可由 API 自动下载 1
3 resnet34 ResNet-34 cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://resnet34/IMAGENET1K_V1 1x3x224x224 256 imagenet1k 21.8 73.314 ready TorchVision 预训练权重,可由 API 自动下载 2
4 resnet50 ResNet-50 cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://resnet50/IMAGENET1K_V2 1x3x224x224 232 imagenet1k 25.56 80.858 ready TorchVision 预训练权重,可由 API 自动下载 3
5 resnet101 ResNet-101 cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://resnet101/IMAGENET1K_V2 1x3x224x224 232 imagenet1k 44.55 81.886 ready TorchVision 预训练权重,可由 API 自动下载 4
6 resnet152 ResNet-152 cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://resnet152/IMAGENET1K_V2 1x3x224x224 232 imagenet1k 60.19 82.284 ready TorchVision 预训练权重,可由 API 自动下载 5
7 resnext50.32x4d ResNeXt-50-32x4d cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://resnext50_32x4d/IMAGENET1K_V2 1x3x224x224 232 imagenet1k 25.03 81.198 ready TorchVision 预训练权重,可由 API 自动下载 6
8 resnext101.32x8d ResNeXt-101-32x8d cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://resnext101_32x8d/IMAGENET1K_V2 1x3x224x224 232 imagenet1k 88.79 82.834 ready TorchVision 预训练权重,可由 API 自动下载 7
9 wide.resnet50.2 Wide-ResNet-50-2 cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://wide_resnet50_2/IMAGENET1K_V2 1x3x224x224 232 imagenet1k 68.88 81.602 ready TorchVision 预训练权重,可由 API 自动下载 8
10 wide.resnet101.2 Wide-ResNet-101-2 cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://wide_resnet101_2/IMAGENET1K_V2 1x3x224x224 232 imagenet1k 126.89 82.51 ready TorchVision 预训练权重,可由 API 自动下载 9
11 densenet121 DenseNet-121 cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://densenet121/IMAGENET1K_V1 1x3x224x224 256 imagenet1k 7.98 74.434 ready TorchVision 预训练权重,可由 API 自动下载 10
12 densenet161 DenseNet-161 cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://densenet161/IMAGENET1K_V1 1x3x224x224 256 imagenet1k 28.68 77.138 ready TorchVision 预训练权重,可由 API 自动下载 11
13 densenet169 DenseNet-169 cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://densenet169/IMAGENET1K_V1 1x3x224x224 256 imagenet1k 14.15 75.6 ready TorchVision 预训练权重,可由 API 自动下载 12
14 densenet201 DenseNet-201 cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://densenet201/IMAGENET1K_V1 1x3x224x224 256 imagenet1k 20.01 76.896 ready TorchVision 预训练权重,可由 API 自动下载 13
15 mobilenet.v2 MobileNetV2 cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://mobilenet_v2/IMAGENET1K_V2 1x3x224x224 232 imagenet1k 3.5 72.154 ready TorchVision 预训练权重,可由 API 自动下载 14
16 mobilenet.v3.small MobileNetV3-Small cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://mobilenet_v3_small/IMAGENET1K_V1 1x3x224x224 256 imagenet1k 2.54 67.668 ready TorchVision 预训练权重,可由 API 自动下载 15
17 mobilenet.v3.large MobileNetV3-Large cls C torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://mobilenet_v3_large/IMAGENET1K_V2 1x3x224x224 232 imagenet1k 5.48 75.274 ready TorchVision 预训练权重,可由 API 自动下载 16
18 shufflenet.v2.x0.5 ShuffleNetV2-x0.5 cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://shufflenet_v2_x0_5/IMAGENET1K_V1 1x3x224x224 256 imagenet1k 1.37 60.552 ready TorchVision 预训练权重,可由 API 自动下载 17
19 shufflenet.v2.x1.0 ShuffleNetV2-x1.0 cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://shufflenet_v2_x1_0/IMAGENET1K_V1 1x3x224x224 256 imagenet1k 2.28 69.362 ready TorchVision 预训练权重,可由 API 自动下载 18
20 shufflenet.v2.x1.5 ShuffleNetV2-x1.5 cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://shufflenet_v2_x1_5/IMAGENET1K_V1 1x3x224x224 232 imagenet1k 3.5 72.996 ready TorchVision 预训练权重,可由 API 自动下载 19
21 shufflenet.v2.x2.0 ShuffleNetV2-x2.0 cls C torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://shufflenet_v2_x2_0/IMAGENET1K_V1 1x3x224x224 232 imagenet1k 7.39 76.23 ready TorchVision 预训练权重,可由 API 自动下载 20
22 efficientnet.b0 EfficientNet-B0 cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://efficientnet_b0/IMAGENET1K_V1 1x3x224x224 256 imagenet1k 5.29 77.692 ready TorchVision 预训练权重,可由 API 自动下载 21
23 efficientnet.b1 EfficientNet-B1 cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://efficientnet_b1/IMAGENET1K_V2 1x3x240x240 255 imagenet1k 7.79 79.838 ready TorchVision 预训练权重,可由 API 自动下载 22
24 efficientnet.b2 EfficientNet-B2 cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://efficientnet_b2/IMAGENET1K_V1 1x3x288x288 288 imagenet1k 9.11 80.608 ready TorchVision 预训练权重,可由 API 自动下载 23
25 efficientnet.b3 EfficientNet-B3 cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://efficientnet_b3/IMAGENET1K_V1 1x3x300x300 320 imagenet1k 12.23 82.008 ready TorchVision 预训练权重,可由 API 自动下载 24
26 efficientnet.b4 EfficientNet-B4 cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://efficientnet_b4/IMAGENET1K_V1 1x3x380x380 384 imagenet1k 19.34 83.384 ready TorchVision 预训练权重,可由 API 自动下载 25
27 efficientnet.b5 EfficientNet-B5 cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://efficientnet_b5/IMAGENET1K_V1 1x3x456x456 456 imagenet1k 30.39 83.444 ready TorchVision 预训练权重,可由 API 自动下载 26
28 efficientnet.b6 EfficientNet-B6 cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://efficientnet_b6/IMAGENET1K_V1 1x3x528x528 528 imagenet1k 43.04 84.008 ready TorchVision 预训练权重,可由 API 自动下载 27
29 efficientnet.b7 EfficientNet-B7 cls C torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://efficientnet_b7/IMAGENET1K_V1 1x3x600x600 600 imagenet1k 66.35 84.122 ready TorchVision 预训练权重,可由 API 自动下载 28
30 efficientnet.v2.s EfficientNetV2-S cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://efficientnet_v2_s/IMAGENET1K_V1 1x3x384x384 384 imagenet1k 21.46 84.228 ready TorchVision 预训练权重,可由 API 自动下载 29
31 efficientnet.v2.m EfficientNetV2-M cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://efficientnet_v2_m/IMAGENET1K_V1 1x3x480x480 480 imagenet1k 54.14 85.112 ready TorchVision 预训练权重,可由 API 自动下载 30
32 efficientnet.v2.l EfficientNetV2-L cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://efficientnet_v2_l/IMAGENET1K_V1 1x3x480x480 480 imagenet1k 118.52 85.808 ready TorchVision 预训练权重,可由 API 自动下载 31
33 convnext.tiny ConvNeXt-Tiny cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://convnext_tiny/IMAGENET1K_V1 1x3x224x224 236 imagenet1k 28.59 82.52 ready TorchVision 预训练权重,可由 API 自动下载 32
34 convnext.small ConvNeXt-Small cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://convnext_small/IMAGENET1K_V1 1x3x224x224 230 imagenet1k 50.22 83.616 ready TorchVision 预训练权重,可由 API 自动下载 33
35 convnext.base ConvNeXt-Base cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://convnext_base/IMAGENET1K_V1 1x3x224x224 232 imagenet1k 88.59 84.062 ready TorchVision 预训练权重,可由 API 自动下载 34
36 convnext.large ConvNeXt-Large cls C torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://convnext_large/IMAGENET1K_V1 1x3x224x224 232 imagenet1k 197.77 84.414 ready TorchVision 预训练权重,可由 API 自动下载 35
37 swin.t Swin-T cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://swin_t/IMAGENET1K_V1 1x3x224x224 232 imagenet1k 28.29 81.474 ready TorchVision 预训练权重,可由 API 自动下载 36
38 swin.s Swin-S cls A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://swin_s/IMAGENET1K_V1 1x3x224x224 246 imagenet1k 49.61 83.196 ready TorchVision 预训练权重,可由 API 自动下载 37
39 swin.b Swin-B cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://swin_b/IMAGENET1K_V1 1x3x224x224 238 imagenet1k 87.77 83.582 ready TorchVision 预训练权重,可由 API 自动下载 38
40 vit.b.16 ViT-B/16 cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://vit_b_16/IMAGENET1K_V1 1x3x224x224 256 imagenet1k 86.57 81.072 ready TorchVision 预训练权重,可由 API 自动下载 39
41 vit.b.32 ViT-B/32 cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://vit_b_32/IMAGENET1K_V1 1x3x224x224 256 imagenet1k 88.22 75.912 ready TorchVision 预训练权重,可由 API 自动下载 40
42 vit.l.16 ViT-L/16 cls C torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://vit_l_16/IMAGENET1K_V1 1x3x224x224 242 imagenet1k 304.33 79.662 ready TorchVision 预训练权重,可由 API 自动下载 41
43 vit.l.32 ViT-L/32 cls C torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://vit_l_32/IMAGENET1K_V1 1x3x224x224 256 imagenet1k 306.54 76.972 ready TorchVision 预训练权重,可由 API 自动下载 42
44 vit.h.14 ViT-H/14 cls C torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://vit_h_14/IMAGENET1K_SWAG_E2E_V1 1x3x518x518 518 imagenet1k 633.47 88.552 ready TorchVision 预训练权重,可由 API 自动下载 43
45 maxvit.t MaxViT-T cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://maxvit_t/IMAGENET1K_V1 1x3x224x224 224 imagenet1k 30.92 83.7 ready TorchVision 预训练权重,可由 API 自动下载 44
46 regnet.y.400mf RegNetY-400MF cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://regnet_y_400mf/IMAGENET1K_V2 1x3x224x224 232 imagenet1k 4.34 75.804 ready TorchVision 预训练权重,可由 API 自动下载 45
47 regnet.y.800mf RegNetY-800MF cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://regnet_y_800mf/IMAGENET1K_V2 1x3x224x224 232 imagenet1k 6.43 78.828 ready TorchVision 预训练权重,可由 API 自动下载 46
48 regnet.y.1.6gf RegNetY-1.6GF cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://regnet_y_1_6gf/IMAGENET1K_V2 1x3x224x224 232 imagenet1k 11.2 80.876 ready TorchVision 预训练权重,可由 API 自动下载 47
49 regnet.y.3.2gf RegNetY-3.2GF cls B torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://regnet_y_3_2gf/IMAGENET1K_V2 1x3x224x224 232 imagenet1k 19.44 81.982 ready TorchVision 预训练权重,可由 API 自动下载 48
50 efficientvit.b0 EfficientViT-B0 cls A mit-han-lab/efficientvit https: custom https://github.com/mit-han-lab/efficientvit https://github.com/mit-han-lab/efficientvit 1x3x224x224 imagenet1k needs_adapter 官方仓库含预训练权重/模型入口 49
51 efficientvit.b1 EfficientViT-B1 cls A mit-han-lab/efficientvit https: custom https://github.com/mit-han-lab/efficientvit https://github.com/mit-han-lab/efficientvit 1x3x224x224 imagenet1k needs_adapter 官方仓库含预训练权重/模型入口 50
52 efficientvit.b2 EfficientViT-B2 cls A mit-han-lab/efficientvit https: custom https://github.com/mit-han-lab/efficientvit https://github.com/mit-han-lab/efficientvit 1x3x224x224 imagenet1k needs_adapter 官方仓库含预训练权重/模型入口 51
53 efficientvit.b3 EfficientViT-B3 cls A mit-han-lab/efficientvit https: custom https://github.com/mit-han-lab/efficientvit https://github.com/mit-han-lab/efficientvit 1x3x224x224 imagenet1k needs_adapter 官方仓库含预训练权重/模型入口 52
54 faster.r.cnn.r50.fpn Faster R-CNN R50-FPN det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 53
55 faster.r.cnn.r101.fpn Faster R-CNN R101-FPN det C mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 54
56 cascade.r.cnn.r50.fpn Cascade R-CNN R50-FPN det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 55
57 cascade.r.cnn.r101.fpn Cascade R-CNN R101-FPN det C mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 56
58 retinanet.r50.fpn RetinaNet R50-FPN det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 57
59 retinanet.r101.fpn RetinaNet R101-FPN det C mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 58
60 fcos.r50.fpn FCOS R50-FPN det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 59
61 atss.r50.fpn ATSS R50-FPN det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 60
62 gfl.r50.fpn GFL R50-FPN det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 61
63 vfnet.r50.fpn VFNet R50-FPN det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 62
64 foveabox.r50.fpn FoveaBox R50-FPN det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 63
65 fsaf.r50.fpn FSAF R50-FPN det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 64
66 reppoints.r50.fpn RepPoints R50-FPN det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 65
67 cornernet.hourglass.104 CornerNet Hourglass-104 det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 66
68 ssd300.vgg16 SSD300-VGG16 det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 67
69 ssdlite320.mobilenetv3.large SSDLite320-MobileNetV3-Large det C mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 68
70 yolov3.darknet53 YOLOv3-Darknet53 det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 69
71 yolox.tiny YOLOX-Tiny det A megvii-basedetection/yolox https: custom https://github.com/Megvii-BaseDetection/YOLOX https://github.com/Megvii-BaseDetection/YOLOX 1x3x640x640 coco2017 needs_adapter 官方 release/model zoo 70
72 yolox.s YOLOX-S det A megvii-basedetection/yolox https: custom https://github.com/Megvii-BaseDetection/YOLOX https://github.com/Megvii-BaseDetection/YOLOX 1x3x640x640 coco2017 needs_adapter 官方 release/model zoo 71
73 yolox.m YOLOX-M det B megvii-basedetection/yolox https: custom https://github.com/Megvii-BaseDetection/YOLOX https://github.com/Megvii-BaseDetection/YOLOX 1x3x640x640 coco2017 needs_adapter 官方 release/model zoo 72
74 yolox.l YOLOX-L det B megvii-basedetection/yolox https: custom https://github.com/Megvii-BaseDetection/YOLOX https://github.com/Megvii-BaseDetection/YOLOX 1x3x640x640 coco2017 needs_adapter 官方 release/model zoo 73
75 yolox.x YOLOX-X det B megvii-basedetection/yolox https: custom https://github.com/Megvii-BaseDetection/YOLOX https://github.com/Megvii-BaseDetection/YOLOX 1x3x640x640 coco2017 needs_adapter 官方 release/model zoo 74
76 rtmdet.tiny RTMDet-Tiny det A mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 75
77 rtmdet.s RTMDet-S det A mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 76
78 rtmdet.m RTMDet-M det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 77
79 rtmdet.l RTMDet-L det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 78
80 rtmdet.x RTMDet-X det C mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 79
81 yolov5n YOLOv5n det A ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov5n.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 80
82 yolov5s YOLOv5s det B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov5s.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 81
83 yolov5m YOLOv5m det B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov5m.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 82
84 yolov5l YOLOv5l det B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov5l.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 83
85 yolov5x YOLOv5x det B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov5x.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 84
86 yolov8n YOLOv8n det A ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov8n.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 85
87 yolov8s YOLOv8s det B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov8s.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 86
88 yolov8m YOLOv8m det B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov8m.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 87
89 yolov8l YOLOv8l det B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov8l.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 88
90 yolov8x YOLOv8x det C ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov8x.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 89
91 yolo11n YOLO11n det A ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolo11n.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 90
92 yolo11s YOLO11s det B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolo11s.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 91
93 yolo11m YOLO11m det B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolo11m.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 92
94 yolo11l YOLO11l det B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolo11l.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 93
95 yolo11x YOLO11x det C ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolo11x.pt 1x3x640x640 coco2017 ready Ultralytics 权重名可自动下载 94
96 detr.r50 DETR-R50 det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 95
97 deformable.detr.r50 Deformable-DETR-R50 det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 96
98 conditional.detr.r50 Conditional-DETR-R50 det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 97
99 dab.detr.r50 DAB-DETR-R50 det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 98
100 dino.r50 DINO-R50 det B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection model zoo 99
101 rt.detr.r18 RT-DETR-R18 det A lyuwenyu/rt-detr https: custom https://github.com/lyuwenyu/RT-DETR https://github.com/lyuwenyu/RT-DETR 1x3x640x640 coco2017 needs_adapter 官方 checkpoint/model zoo 100
102 rt.detr.r50 RT-DETR-R50 det B lyuwenyu/rt-detr https: custom https://github.com/lyuwenyu/RT-DETR https://github.com/lyuwenyu/RT-DETR 1x3x640x640 coco2017 needs_adapter 官方 checkpoint/model zoo 101
103 rt.detrv2.r18 RT-DETRv2-R18 det A lyuwenyu/rt-detr https: custom https://github.com/lyuwenyu/RT-DETR https://github.com/lyuwenyu/RT-DETR 1x3x640x640 coco2017 needs_adapter 官方 checkpoint/model zoo 102
104 rt.detrv2.r50 RT-DETRv2-R50 det B lyuwenyu/rt-detr https: custom https://github.com/lyuwenyu/RT-DETR https://github.com/lyuwenyu/RT-DETR 1x3x640x640 coco2017 needs_adapter 官方 checkpoint/model zoo 103
105 fcn.resnet50 FCN-ResNet50 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 104
106 fcn.resnet101 FCN-ResNet101 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 105
107 deeplabv3.resnet50 DeepLabV3-ResNet50 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 106
108 deeplabv3.resnet101 DeepLabV3-ResNet101 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 107
109 deeplabv3.mobilenetv3.large DeepLabV3-MobileNetV3-Large seg C mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 108
110 deeplabv3plus.resnet50 DeepLabV3+-ResNet50 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 109
111 deeplabv3plus.resnet101 DeepLabV3+-ResNet101 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 110
112 pspnet.resnet50 PSPNet-ResNet50 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 111
113 pspnet.resnet101 PSPNet-ResNet101 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 112
114 upernet.resnet50 UPerNet-ResNet50 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 113
115 upernet.swin.t UPerNet-Swin-T seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 114
116 upernet.swin.s UPerNet-Swin-S seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 115
117 upernet.swin.b UPerNet-Swin-B seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 116
118 segformer.b0 SegFormer-B0 seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 117
119 segformer.b1 SegFormer-B1 seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 118
120 segformer.b2 SegFormer-B2 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 119
121 segformer.b3 SegFormer-B3 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 120
122 segformer.b4 SegFormer-B4 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 121
123 segformer.b5 SegFormer-B5 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 122
124 mask2former.swin.t Mask2Former-Swin-T seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 123
125 mask2former.swin.s Mask2Former-Swin-S seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 124
126 mask2former.swin.b Mask2Former-Swin-B seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 125
127 ddrnet.23.slim DDRNet-23-Slim seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 126
128 ddrnet.23 DDRNet-23 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 127
129 ddrnet.39 DDRNet-39 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 128
130 pidnet.s PIDNet-S seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 129
131 pidnet.m PIDNet-M seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 130
132 pidnet.l PIDNet-L seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 131
133 bisenetv1.r18 BiSeNetV1-R18 seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 132
134 bisenetv2 BiSeNetV2 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 133
135 stdc1.seg STDC1-Seg seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 134
136 stdc2.seg STDC2-Seg seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 135
137 fast.scnn Fast-SCNN seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 136
138 hrnet.w18 HRNet-W18 seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 137
139 hrnet.w48 HRNet-W48 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 138
140 ocrnet.hrnet.w18 OCRNet-HRNet-W18 seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 139
141 ocrnet.hrnet.w48 OCRNet-HRNet-W48 seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 140
142 segnext.tiny SegNeXt-Tiny seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 141
143 segnext.small SegNeXt-Small seg A mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 142
144 segnext.base SegNeXt-Base seg B mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 143
145 segnext.large SegNeXt-Large seg C mmsegmentation openmmlab https://github.com/open-mmlab/mmsegmentation https://github.com/open-mmlab/mmsegmentation/blob/main/model-index.yml 1x3x512x1024 ade20k needs_adapter MMSegmentation model zoo 144
146 efficientvit.seg.b0 EfficientViT-Seg-B0 seg A mit-han-lab/efficientvit https: custom https://github.com/mit-han-lab/efficientvit https://github.com/mit-han-lab/efficientvit 1x3x512x1024 ade20k needs_adapter 官方 EfficientViT segmentation 权重 145
147 efficientvit.seg.b1 EfficientViT-Seg-B1 seg A mit-han-lab/efficientvit https: custom https://github.com/mit-han-lab/efficientvit https://github.com/mit-han-lab/efficientvit 1x3x512x1024 ade20k needs_adapter 官方 EfficientViT segmentation 权重 146
148 efficientvit.seg.b2 EfficientViT-Seg-B2 seg A mit-han-lab/efficientvit https: custom https://github.com/mit-han-lab/efficientvit https://github.com/mit-han-lab/efficientvit 1x3x512x1024 ade20k needs_adapter 官方 EfficientViT segmentation 权重 147
149 efficientvit.seg.b3 EfficientViT-Seg-B3 seg A mit-han-lab/efficientvit https: custom https://github.com/mit-han-lab/efficientvit https://github.com/mit-han-lab/efficientvit 1x3x512x1024 ade20k needs_adapter 官方 EfficientViT segmentation 权重 148
150 mask.r.cnn.r50.fpn Mask R-CNN R50-FPN iseg B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection instance segmentation model zoo 149
151 mask.r.cnn.r101.fpn Mask R-CNN R101-FPN iseg C mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection instance segmentation model zoo 150
152 cascade.mask.r.cnn.r50.fpn Cascade Mask R-CNN R50-FPN iseg B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection instance segmentation model zoo 151
153 cascade.mask.r.cnn.r101.fpn Cascade Mask R-CNN R101-FPN iseg C mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection instance segmentation model zoo 152
154 htc.r50.fpn HTC R50-FPN iseg B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection instance segmentation model zoo 153
155 htc.r101.fpn HTC R101-FPN iseg C mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection instance segmentation model zoo 154
156 solo.r50 SOLO R50 iseg B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection instance segmentation model zoo 155
157 solov2.r50 SOLOv2 R50 iseg B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection instance segmentation model zoo 156
158 condinst.r50 CondInst R50 iseg B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection instance segmentation model zoo 157
159 queryinst.r50 QueryInst R50 iseg B mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection instance segmentation model zoo 158
160 mask2former.swin.t.instance Mask2Former-Swin-T-Instance iseg A mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection instance segmentation model zoo 159
161 mask2former.swin.s.instance Mask2Former-Swin-S-Instance iseg A mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection instance segmentation model zoo 160
162 mask2former.swin.b.instance Mask2Former-Swin-B-Instance iseg A mmdetection openmmlab https://github.com/open-mmlab/mmdetection https://github.com/open-mmlab/mmdetection/blob/main/docs/en/model_zoo.md 1x3x640x640 coco2017 needs_adapter MMDetection instance segmentation model zoo 161
163 yolov8n.seg YOLOv8n-seg iseg A ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov8n-seg.pt 1x3x640x640 coco2017 needs_eval Ultralytics 自动/官方下载 162
164 yolov8s.seg YOLOv8s-seg iseg B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov8s-seg.pt 1x3x640x640 coco2017 needs_eval Ultralytics 自动/官方下载 163
165 yolov8m.seg YOLOv8m-seg iseg B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov8m-seg.pt 1x3x640x640 coco2017 needs_eval Ultralytics 自动/官方下载 164
166 yolov8l.seg YOLOv8l-seg iseg B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov8l-seg.pt 1x3x640x640 coco2017 needs_eval Ultralytics 自动/官方下载 165
167 yolov8x.seg YOLOv8x-seg iseg C ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolov8x-seg.pt 1x3x640x640 coco2017 needs_eval Ultralytics 自动/官方下载 166
168 yolo11n.seg YOLO11n-seg iseg A ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolo11n-seg.pt 1x3x640x640 coco2017 needs_eval Ultralytics 自动/官方下载 167
169 yolo11s.seg YOLO11s-seg iseg B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolo11s-seg.pt 1x3x640x640 coco2017 needs_eval Ultralytics 自动/官方下载 168
170 yolo11m.seg YOLO11m-seg iseg B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolo11m-seg.pt 1x3x640x640 coco2017 needs_eval Ultralytics 自动/官方下载 169
171 yolo11l.seg YOLO11l-seg iseg B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolo11l-seg.pt 1x3x640x640 coco2017 needs_eval Ultralytics 自动/官方下载 170
172 yolo11x.seg YOLO11x-seg iseg C ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolo11x-seg.pt 1x3x640x640 coco2017 needs_eval Ultralytics 自动/官方下载 171
173 sam.vit.b SAM-ViT-B iseg B facebookresearch/segment-anything https: custom https://github.com/facebookresearch/segment-anything https://github.com/facebookresearch/segment-anything 1x3x640x640 coco2017 needs_adapter 官方 checkpoints 172
174 sam.vit.l SAM-ViT-L iseg C facebookresearch/segment-anything https: custom https://github.com/facebookresearch/segment-anything https://github.com/facebookresearch/segment-anything 1x3x640x640 coco2017 needs_adapter 官方 checkpoints 173
175 sam.vit.h SAM-ViT-H iseg C facebookresearch/segment-anything https: custom https://github.com/facebookresearch/segment-anything https://github.com/facebookresearch/segment-anything 1x3x640x640 coco2017 needs_adapter 官方 checkpoints 174
176 sam2.1.hiera.tiny SAM2.1-Hiera-Tiny iseg A facebookresearch/sam2 https: custom https://github.com/facebookresearch/sam2 https://github.com/facebookresearch/sam2 1x3x640x640 coco2017 needs_adapter 官方 checkpoints 175
177 sam2.1.hiera.small SAM2.1-Hiera-Small iseg A facebookresearch/sam2 https: custom https://github.com/facebookresearch/sam2 https://github.com/facebookresearch/sam2 1x3x640x640 coco2017 needs_adapter 官方 checkpoints 176
178 sam2.1.hiera.baseplus SAM2.1-Hiera-Base+ iseg B facebookresearch/sam2 https: custom https://github.com/facebookresearch/sam2 https://github.com/facebookresearch/sam2 1x3x640x640 coco2017 needs_adapter 官方 checkpoints 177
179 sam2.1.hiera.large SAM2.1-Hiera-Large iseg C facebookresearch/sam2 https: custom https://github.com/facebookresearch/sam2 https://github.com/facebookresearch/sam2 1x3x640x640 coco2017 needs_adapter 官方 checkpoints 178
180 mobilesam MobileSAM iseg A chaoningzhang/mobilesam https: custom https://github.com/ChaoningZhang/MobileSAM https://github.com/ChaoningZhang/MobileSAM 1x3x640x640 coco2017 needs_adapter 官方权重 179
181 edgesam EdgeSAM iseg B chongzhou96/edgesam https: custom https://github.com/chongzhou96/EdgeSAM https://github.com/chongzhou96/EdgeSAM 1x3x640x640 coco2017 needs_adapter 官方权重/ONNX 180
182 efficientsam.ti EfficientSAM-Ti iseg A yformer/efficientsam https: custom https://github.com/yformer/EfficientSAM https://github.com/yformer/EfficientSAM 1x3x640x640 coco2017 needs_adapter weights/ 目录含 checkpoint 181
183 efficientsam.s EfficientSAM-S iseg A yformer/efficientsam https: custom https://github.com/yformer/EfficientSAM https://github.com/yformer/EfficientSAM 1x3x640x640 coco2017 needs_adapter weights/ 目录含 checkpoint 182
184 efficientvit.sam.l0 EfficientViT-SAM-L0 iseg A mit-han-lab/efficientvit https: custom https://github.com/mit-han-lab/efficientvit https://github.com/mit-han-lab/efficientvit/tree/master/applications/efficientvit_sam 1x3x640x640 coco2017 needs_adapter 官方 pretrained models 183
185 efficientvit.sam.l1 EfficientViT-SAM-L1 iseg A mit-han-lab/efficientvit https: custom https://github.com/mit-han-lab/efficientvit https://github.com/mit-han-lab/efficientvit/tree/master/applications/efficientvit_sam 1x3x640x640 coco2017 needs_adapter 官方 pretrained models 184
186 efficientvit.sam.l2 EfficientViT-SAM-L2 iseg A mit-han-lab/efficientvit https: custom https://github.com/mit-han-lab/efficientvit https://github.com/mit-han-lab/efficientvit/tree/master/applications/efficientvit_sam 1x3x640x640 coco2017 needs_adapter 官方 pretrained models 185
187 fastsam.s FastSAM-s iseg B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://FastSAM-s.pt 1x3x640x640 coco2017 needs_eval Ultralytics 自动/官方下载 186
188 fastsam.x FastSAM-x iseg B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://FastSAM-x.pt 1x3x640x640 coco2017 needs_eval Ultralytics 自动/官方下载 187
189 hq.sam.vit.b HQ-SAM-ViT-B iseg A syscv/sam-hq https: custom https://github.com/SysCV/sam-hq https://github.com/SysCV/sam-hq 1x3x640x640 coco2017 needs_adapter 官方 checkpoints 188
190 hq.sam.vit.l HQ-SAM-ViT-L iseg C syscv/sam-hq https: custom https://github.com/SysCV/sam-hq https://github.com/SysCV/sam-hq 1x3x640x640 coco2017 needs_adapter 官方 checkpoints 189
191 hq.sam.vit.h HQ-SAM-ViT-H iseg C syscv/sam-hq https: custom https://github.com/SysCV/sam-hq https://github.com/SysCV/sam-hq 1x3x640x640 coco2017 needs_adapter 官方 checkpoints 190
192 repvit.sam RepViT-SAM iseg A thu-mig/repvit https: custom https://github.com/THU-MIG/RepViT https://github.com/THU-MIG/RepViT 1x3x640x640 coco2017 needs_adapter RepViT-SAM 权重入口 191
193 tinysam TinySAM iseg A xinghaochen/tinysam https: custom https://github.com/xinghaochen/TinySAM https://github.com/xinghaochen/TinySAM 1x3x640x640 coco2017 needs_adapter 官方权重 192
194 slimsam.50 SlimSAM-50 iseg B czg1225/slimsam https: custom https://github.com/czg1225/SlimSAM https://github.com/czg1225/SlimSAM 1x3x640x640 coco2017 needs_adapter SlimSAM-50/77 官方权重 193
195 slimsam.77 SlimSAM-77 iseg B czg1225/slimsam https: custom https://github.com/czg1225/SlimSAM https://github.com/czg1225/SlimSAM 1x3x640x640 coco2017 needs_adapter SlimSAM-50/77 官方权重 194
196 efficienttam.ti EfficientTAM-Ti iseg A yformer/efficienttam https: custom https://github.com/yformer/EfficientTAM https://github.com/yformer/EfficientTAM 1x3x640x640 coco2017 needs_adapter checkpoints/download_checkpoints.sh 195
197 efficienttam.s EfficientTAM-S iseg A yformer/efficienttam https: custom https://github.com/yformer/EfficientTAM https://github.com/yformer/EfficientTAM 1x3x640x640 coco2017 needs_adapter checkpoints/download_checkpoints.sh 196
198 simplebaseline.r50 SimpleBaseline-R50 pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 197
199 simplebaseline.r101 SimpleBaseline-R101 pose C mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 198
200 simplebaseline.r152 SimpleBaseline-R152 pose C mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 199
201 hrnet.w32 HRNet-W32 pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 200
202 hrnet.w48.2 HRNet-W48 pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 201
203 litehrnet.18 LiteHRNet-18 pose A mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 202
204 litehrnet.30 LiteHRNet-30 pose A mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 203
205 rtmpose.tiny RTMPose-Tiny pose A mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 204
206 rtmpose.s RTMPose-S pose A mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 205
207 rtmpose.m RTMPose-M pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 206
208 rtmpose.l RTMPose-L pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 207
209 rtmpose.x RTMPose-X pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 208
210 vitpose.s ViTPose-S pose A mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 209
211 vitpose.b ViTPose-B pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 210
212 vitpose.l ViTPose-L pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 211
213 vitpose.h ViTPose-H pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 212
214 mspn.r50 MSPN-R50 pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 213
215 mspn.r101 MSPN-R101 pose C mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 214
216 higherhrnet.w32 HigherHRNet-W32 pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 215
217 higherhrnet.w48 HigherHRNet-W48 pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 216
218 dekr.w32 DEKR-W32 pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 217
219 dekr.w48 DEKR-W48 pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 218
220 simcc.resnet50 SimCC-ResNet50 pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 219
221 rtmo.s RTMO-S pose A mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 220
222 rtmo.m RTMO-M pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 221
223 rtmo.l RTMO-L pose B mmpose openmmlab https://github.com/open-mmlab/mmpose https://github.com/open-mmlab/mmpose/blob/main/model-index.yml 1x3x640x640 coco2017 needs_adapter MMPose model zoo 222
224 yolo11n.pose YOLO11n-pose pose A ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolo11n-pose.pt 1x3x640x640 coco2017 needs_eval Ultralytics pose weights 223
225 yolo11s.pose YOLO11s-pose pose B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolo11s-pose.pt 1x3x640x640 coco2017 needs_eval Ultralytics pose weights 224
226 yolo11m.pose YOLO11m-pose pose B ultralytics ultralytics https://docs.ultralytics.com/models/ ultralytics://yolo11m-pose.pt 1x3x640x640 coco2017 needs_eval Ultralytics pose weights 225
227 openpose.body25 OpenPose-Body25 pose B cmu-perceptual-computing-lab/openpose https: custom https://github.com/CMU-Perceptual-Computing-Lab/openpose https://github.com/CMU-Perceptual-Computing-Lab/openpose 1x3x640x640 coco2017 needs_adapter 官方模型下载脚本/文档 226
228 midas.small MiDaS-Small mde A isl-org/midas https: custom https://github.com/isl-org/MiDaS https://github.com/isl-org/MiDaS 1x3x518x518 kitti2015 needs_adapter 官方 model zoo 227
229 dpt.hybrid DPT-Hybrid mde B isl-org/midas https: custom https://github.com/isl-org/MiDaS https://github.com/isl-org/MiDaS 1x3x518x518 kitti2015 needs_adapter 官方 model zoo 228
230 dpt.large DPT-Large mde C isl-org/midas https: custom https://github.com/isl-org/MiDaS https://github.com/isl-org/MiDaS 1x3x518x518 kitti2015 needs_adapter 官方 model zoo 229
231 depth.anything.v1.small Depth-Anything-V1-Small mde A liheyoung/depth-anything https: custom https://github.com/LiheYoung/Depth-Anything https://github.com/LiheYoung/Depth-Anything 1x3x518x518 kitti2015 needs_adapter 官方 checkpoints/HF 230
232 depth.anything.v1.base Depth-Anything-V1-Base mde B liheyoung/depth-anything https: custom https://github.com/LiheYoung/Depth-Anything https://github.com/LiheYoung/Depth-Anything 1x3x518x518 kitti2015 needs_adapter 官方 checkpoints/HF 231
233 depth.anything.v1.large Depth-Anything-V1-Large mde C liheyoung/depth-anything https: custom https://github.com/LiheYoung/Depth-Anything https://github.com/LiheYoung/Depth-Anything 1x3x518x518 kitti2015 needs_adapter 官方 checkpoints/HF 232
234 depth.anything.v2.small Depth-Anything-V2-Small mde A depthanything/depth-anything-v2 https: custom https://github.com/DepthAnything/Depth-Anything-V2 https://github.com/DepthAnything/Depth-Anything-V2 1x3x518x518 kitti2015 needs_adapter 官方 checkpoints/HF 233
235 depth.anything.v2.base Depth-Anything-V2-Base mde B depthanything/depth-anything-v2 https: custom https://github.com/DepthAnything/Depth-Anything-V2 https://github.com/DepthAnything/Depth-Anything-V2 1x3x518x518 kitti2015 needs_adapter 官方 checkpoints/HF 234
236 depth.anything.v2.large Depth-Anything-V2-Large mde C depthanything/depth-anything-v2 https: custom https://github.com/DepthAnything/Depth-Anything-V2 https://github.com/DepthAnything/Depth-Anything-V2 1x3x518x518 kitti2015 needs_adapter 官方 checkpoints/HF 235
237 depth.anything.v2.metric.hypersim.small Depth-Anything-V2-Metric-Hypersim-Small mde A depthanything/depth-anything-v2 https: custom https://github.com/DepthAnything/Depth-Anything-V2 https://github.com/DepthAnything/Depth-Anything-V2 1x3x518x518 kitti2015 needs_adapter 官方 checkpoints/HF 236
238 depth.anything.v2.metric.hypersim.base Depth-Anything-V2-Metric-Hypersim-Base mde B depthanything/depth-anything-v2 https: custom https://github.com/DepthAnything/Depth-Anything-V2 https://github.com/DepthAnything/Depth-Anything-V2 1x3x518x518 kitti2015 needs_adapter 官方 checkpoints/HF 237
239 depth.anything.v2.metric.hypersim.large Depth-Anything-V2-Metric-Hypersim-Large mde C depthanything/depth-anything-v2 https: custom https://github.com/DepthAnything/Depth-Anything-V2 https://github.com/DepthAnything/Depth-Anything-V2 1x3x518x518 kitti2015 needs_adapter 官方 checkpoints/HF 238
240 depth.anything.v2.metric.vkitti.small Depth-Anything-V2-Metric-VKITTI-Small mde A depthanything/depth-anything-v2 https: custom https://github.com/DepthAnything/Depth-Anything-V2 https://github.com/DepthAnything/Depth-Anything-V2 1x3x518x518 kitti2015 needs_adapter 官方 checkpoints/HF 239
241 depth.anything.v2.metric.vkitti.base Depth-Anything-V2-Metric-VKITTI-Base mde B depthanything/depth-anything-v2 https: custom https://github.com/DepthAnything/Depth-Anything-V2 https://github.com/DepthAnything/Depth-Anything-V2 1x3x518x518 kitti2015 needs_adapter 官方 checkpoints/HF 240
242 depth.anything.v2.metric.vkitti.large Depth-Anything-V2-Metric-VKITTI-Large mde C depthanything/depth-anything-v2 https: custom https://github.com/DepthAnything/Depth-Anything-V2 https://github.com/DepthAnything/Depth-Anything-V2 1x3x518x518 kitti2015 needs_adapter 官方 checkpoints/HF 241
243 zoedepth.n ZoeDepth-N mde B isl-org/zoedepth https: custom https://github.com/isl-org/ZoeDepth https://github.com/isl-org/ZoeDepth 1x3x518x518 kitti2015 needs_adapter 官方 torch hub/checkpoints 242
244 zoedepth.k ZoeDepth-K mde B isl-org/zoedepth https: custom https://github.com/isl-org/ZoeDepth https://github.com/isl-org/ZoeDepth 1x3x518x518 kitti2015 needs_adapter 官方 torch hub/checkpoints 243
245 zoedepth.nk ZoeDepth-NK mde B isl-org/zoedepth https: custom https://github.com/isl-org/ZoeDepth https://github.com/isl-org/ZoeDepth 1x3x518x518 kitti2015 needs_adapter 官方 torch hub/checkpoints 244
246 metric3dv2.vit.small Metric3Dv2-ViT-Small mde A yvanyin/metric3d https: custom https://github.com/YvanYin/Metric3D https://github.com/YvanYin/Metric3D 1x3x518x518 kitti2015 needs_adapter 官方 checkpoint / torch.hub / ONNX 245
247 metric3dv2.vit.large Metric3Dv2-ViT-Large mde C yvanyin/metric3d https: custom https://github.com/YvanYin/Metric3D https://github.com/YvanYin/Metric3D 1x3x518x518 kitti2015 needs_adapter 官方 checkpoint / torch.hub / ONNX 246
248 metric3dv2.vit.giant2 Metric3Dv2-ViT-Giant2 mde C yvanyin/metric3d https: custom https://github.com/YvanYin/Metric3D https://github.com/YvanYin/Metric3D 1x3x518x518 kitti2015 needs_adapter 官方 checkpoint / torch.hub / ONNX 247
249 metric3d.v1.convnext.tiny Metric3D-v1-ConvNeXt-Tiny mde A yvanyin/metric3d https: custom https://github.com/YvanYin/Metric3D https://github.com/YvanYin/Metric3D 1x3x518x518 kitti2015 needs_adapter 官方 checkpoint / torch.hub / ONNX 248
250 metric3d.v1.convnext.large Metric3D-v1-ConvNeXt-Large mde C yvanyin/metric3d https: custom https://github.com/YvanYin/Metric3D https://github.com/YvanYin/Metric3D 1x3x518x518 kitti2015 needs_adapter 官方 checkpoint / torch.hub / ONNX 249
251 unidepth.v1.convnext.l UniDepth-V1-ConvNeXt-L mde B lpiccinelli-eth/unidepth https: custom https://github.com/lpiccinelli-eth/UniDepth https://huggingface.co/lpiccinelli 1x3x518x518 kitti2015 needs_adapter 官方 Hugging Face model zoo 250
252 unidepth.v1.vit.l UniDepth-V1-ViT-L mde C lpiccinelli-eth/unidepth https: custom https://github.com/lpiccinelli-eth/UniDepth https://huggingface.co/lpiccinelli 1x3x518x518 kitti2015 needs_adapter 官方 Hugging Face model zoo 251
253 unidepth.v2.vit.s UniDepth-V2-ViT-S mde A lpiccinelli-eth/unidepth https: custom https://github.com/lpiccinelli-eth/UniDepth https://huggingface.co/lpiccinelli 1x3x518x518 kitti2015 needs_adapter 官方 Hugging Face model zoo 252
254 unidepth.v2.vit.b UniDepth-V2-ViT-B mde B lpiccinelli-eth/unidepth https: custom https://github.com/lpiccinelli-eth/UniDepth https://huggingface.co/lpiccinelli 1x3x518x518 kitti2015 needs_adapter 官方 Hugging Face model zoo 253
255 unidepth.v2.vit.l UniDepth-V2-ViT-L mde C lpiccinelli-eth/unidepth https: custom https://github.com/lpiccinelli-eth/UniDepth https://huggingface.co/lpiccinelli 1x3x518x518 kitti2015 needs_adapter 官方 Hugging Face model zoo 254
256 depthpro DepthPro mde B apple/ml-depth-pro https: custom https://github.com/apple/ml-depth-pro https://github.com/apple/ml-depth-pro 1x3x518x518 kitti2015 needs_adapter 官方 checkpoint 255
257 psmnet PSMNet stereo A jiarenchang/psmnet https: custom https://github.com/JiaRenChang/PSMNet https://github.com/JiaRenChang/PSMNet 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 256
258 gwcnet GwcNet stereo A xy-guo/gwcnet https: custom https://github.com/xy-guo/GwcNet https://github.com/xy-guo/GwcNet 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 257
259 ganet GANet stereo A feihuzhang/ganet https: custom https://github.com/feihuzhang/GANet https://github.com/feihuzhang/GANet 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 258
260 aanet AANet stereo A haofeixu/aanet https: custom https://github.com/haofeixu/aanet https://github.com/haofeixu/aanet 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 259
261 leastereo LEAStereo stereo A xueliancheng/leastereo https: custom https://github.com/XuelianCheng/LEAStereo https://github.com/XuelianCheng/LEAStereo 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 260
262 anynet AnyNet stereo A mileyan/anynet https: custom https://github.com/mileyan/AnyNet https://github.com/mileyan/AnyNet 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 261
263 hitnet HITNet stereo A google-research/google-research custom https://github.com/google-research/google-research/tree/master/hitnet https://github.com/google-research/google-research/tree/master/hitnet 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 262
264 mobilestereonet.2d MobileStereoNet-2D stereo A cogsys-tuebingen/mobilestereonet https: custom https://github.com/cogsys-tuebingen/mobilestereonet https://github.com/cogsys-tuebingen/mobilestereonet 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 263
265 mobilestereonet.3d MobileStereoNet-3D stereo A cogsys-tuebingen/mobilestereonet https: custom https://github.com/cogsys-tuebingen/mobilestereonet https://github.com/cogsys-tuebingen/mobilestereonet 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 264
266 raft.stereo RAFT-Stereo stereo A princeton-vl/raft-stereo https: custom https://github.com/princeton-vl/RAFT-Stereo https://github.com/princeton-vl/RAFT-Stereo 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 265
267 iraftstereo.rvc iRAFTStereo-RVC stereo A princeton-vl/raft-stereo https: custom https://github.com/princeton-vl/RAFT-Stereo https://github.com/princeton-vl/RAFT-Stereo 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 266
268 crestereo CREStereo stereo A megvii-research/crestereo https: custom https://github.com/megvii-research/CREStereo https://github.com/megvii-research/CREStereo 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 267
269 igev.stereo IGEV-Stereo stereo A gangweix/igev https: custom https://github.com/gangweiX/IGEV https://github.com/gangweiX/IGEV 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 268
270 unimatch.stereo UniMatch-Stereo stereo A autonomousvision/unimatch https: custom https://github.com/autonomousvision/unimatch https://github.com/autonomousvision/unimatch 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 269
271 sttr STTR stereo A mli0603/stereo-transformer https: custom https://github.com/mli0603/stereo-transformer https://github.com/mli0603/stereo-transformer 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 270
272 foundationstereo FoundationStereo stereo C nvlabs/foundationstereo https: custom https://github.com/NVlabs/FoundationStereo https://github.com/NVlabs/FoundationStereo 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 271
273 fast.foundationstereo Fast-FoundationStereo stereo C nvlabs/fast-foundationstereo https: custom https://github.com/NVlabs/Fast-FoundationStereo https://github.com/NVlabs/Fast-FoundationStereo 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 272
274 waft.stereo WAFT-Stereo stereo A princeton-vl/waft-stereo https: custom https://github.com/princeton-vl/WAFT-Stereo https://github.com/princeton-vl/WAFT-Stereo 1x3x384x1248 kitti2015 needs_adapter 官方仓库;按 README 下载 checkpoint 273
275 flownet2 FlowNet2 flow A nvidia/flownet2-pytorch https: custom https://github.com/NVIDIA/flownet2-pytorch https://github.com/NVIDIA/flownet2-pytorch 1x3x384x1024 kitti2015 needs_adapter 官方仓库/model zoo 274
276 spynet SpyNet flow A torchvision torchvision https://docs.pytorch.org/vision/main/models https://docs.pytorch.org/vision/main/models 1x3x384x1024 kitti2015 needs_adapter 官方仓库/model zoo 275
277 pwc.net PWC-Net flow A nvlabs/pwc-net https: custom https://github.com/NVlabs/PWC-Net https://github.com/NVlabs/PWC-Net 1x3x384x1024 kitti2015 needs_adapter 官方仓库/model zoo 276
278 liteflownet2 LiteFlowNet2 flow A twhui/liteflownet2 https: custom https://github.com/twhui/LiteFlowNet2 https://github.com/twhui/LiteFlowNet2 1x3x384x1024 kitti2015 needs_adapter 官方仓库/model zoo 277
279 raft.large RAFT-Large flow C torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://raft_large/C_T_SKHT_V2 1x3x384x1024 256 kitti2015 5.26 needs_adapter 官方仓库/model zoo 278
280 raft.small RAFT-Small flow A torchvision torchvision https://docs.pytorch.org/vision/main/models torchvision://raft_small/C_T_V2 1x3x384x1024 256 kitti2015 0.99 needs_adapter 官方仓库/model zoo 279
281 gma GMA flow A zacjiang/gma https: custom https://github.com/zacjiang/GMA https://github.com/zacjiang/GMA 1x3x384x1024 kitti2015 needs_adapter 官方仓库/model zoo 280
282 gmflow GMFlow flow A haofeixu/gmflow https: custom https://github.com/haofeixu/gmflow https://github.com/haofeixu/gmflow 1x3x384x1024 kitti2015 needs_adapter 官方仓库/model zoo 281
283 flowformer FlowFormer flow A drinkingcoder/flowformer-official https: custom https://github.com/drinkingcoder/FlowFormer-Official https://github.com/drinkingcoder/FlowFormer-Official 1x3x384x1024 kitti2015 needs_adapter 官方仓库/model zoo 282
284 flowformerplusplus FlowFormer++ flow C xiaoyushi97/flowformerplusplus https: custom https://github.com/XiaoyuShi97/FlowFormerPlusPlus https://github.com/XiaoyuShi97/FlowFormerPlusPlus 1x3x384x1024 kitti2015 needs_adapter 官方仓库/model zoo 283
285 unimatch.flow UniMatch-Flow flow A autonomousvision/unimatch https: custom https://github.com/autonomousvision/unimatch https://github.com/autonomousvision/unimatch 1x3x384x1024 kitti2015 needs_adapter 官方仓库/model zoo 284
286 sea.raft SEA-RAFT flow A princeton-vl/sea-raft https: custom https://github.com/princeton-vl/SEA-RAFT https://github.com/princeton-vl/SEA-RAFT 1x3x384x1024 kitti2015 needs_adapter 官方仓库/model zoo 285
287 memflow MemFlow flow A dqiaole/memflow https: custom https://github.com/DQiaole/MemFlow https://github.com/DQiaole/MemFlow 1x3x384x1024 kitti2015 needs_adapter 官方仓库/model zoo 286
288 flow1d Flow1D flow A haofeixu/flow1d https: custom https://github.com/haofeixu/flow1d https://github.com/haofeixu/flow1d 1x3x384x1024 kitti2015 needs_adapter 官方仓库/model zoo 287
289 sift SIFT feat A opencv opencv https://docs.opencv.org/4.x/ https://docs.opencv.org/4.x/ 1x1x480x640 hpatches skip 经典算子,非神经网络,不量化 288
290 orb ORB feat A opencv opencv https://docs.opencv.org/4.x/ https://docs.opencv.org/4.x/ 1x1x480x640 hpatches skip 经典算子,非神经网络,不量化 289
291 akaze AKAZE feat A opencv opencv https://docs.opencv.org/4.x/ https://docs.opencv.org/4.x/ 1x1x480x640 hpatches skip 经典算子,非神经网络,不量化 290
292 superpoint SuperPoint feat A magicleap/supergluepretrainednetwork https: custom https://github.com/magicleap/SuperGluePretrainedNetwork https://github.com/magicleap/SuperGluePretrainedNetwork 1x1x480x640 hpatches needs_adapter 官方权重/仓库 291
293 disk DISK feat A cvg/lightglue https: custom https://github.com/cvg/LightGlue https://github.com/cvg/LightGlue 1x1x480x640 hpatches needs_adapter 官方权重/仓库 292
294 aliked ALIKED feat A shiaoming/aliked https: custom https://github.com/Shiaoming/ALIKED https://github.com/Shiaoming/ALIKED 1x1x480x640 hpatches needs_adapter 官方权重/仓库 293
295 xfeat XFeat feat A verlab/accelerated_features https: custom https://github.com/verlab/accelerated_features https://github.com/verlab/accelerated_features 1x1x480x640 hpatches needs_adapter 官方权重/仓库 294
296 d2.net D2-Net feat A mihaidusmanu/d2-net https: custom https://github.com/mihaidusmanu/d2-net https://github.com/mihaidusmanu/d2-net 1x1x480x640 hpatches needs_adapter 官方权重/仓库 295
297 r2d2 R2D2 feat A naver/r2d2 https: custom https://github.com/naver/r2d2 https://github.com/naver/r2d2 1x1x480x640 hpatches needs_adapter 官方权重/仓库 296
298 aslfeat ASLFeat feat A lzx551402/aslfeat https: custom https://github.com/lzx551402/ASLFeat https://github.com/lzx551402/ASLFeat 1x1x480x640 hpatches needs_adapter 官方权重/仓库 297
299 keynetplushardnet KeyNet+HardNet feat A axelbarroso/key.net https: custom https://github.com/axelBarroso/Key.Net https://github.com/axelBarroso/Key.Net 1x1x480x640 hpatches needs_adapter 官方权重/仓库 298
300 dedode.v2 DeDoDe-v2 feat A parskatt/dedode https: custom https://github.com/Parskatt/DeDoDe https://github.com/Parskatt/DeDoDe 1x1x480x640 hpatches needs_adapter 官方权重/仓库 299
301 superglue.indoor SuperGlue-Indoor feat A magicleap/supergluepretrainednetwork https: custom https://github.com/magicleap/SuperGluePretrainedNetwork https://github.com/magicleap/SuperGluePretrainedNetwork 1x1x480x640 hpatches needs_adapter 官方权重/仓库 300
302 superglue.outdoor SuperGlue-Outdoor feat A magicleap/supergluepretrainednetwork https: custom https://github.com/magicleap/SuperGluePretrainedNetwork https://github.com/magicleap/SuperGluePretrainedNetwork 1x1x480x640 hpatches needs_adapter 官方权重/仓库 301
303 lightglue.superpoint LightGlue-SuperPoint feat A cvg/lightglue https: custom https://github.com/cvg/LightGlue https://github.com/cvg/LightGlue 1x1x480x640 hpatches needs_adapter 官方权重/仓库 302
304 lightglue.disk LightGlue-DISK feat A cvg/lightglue https: custom https://github.com/cvg/LightGlue https://github.com/cvg/LightGlue 1x1x480x640 hpatches needs_adapter 官方权重/仓库 303
305 lightglue.aliked LightGlue-ALIKED feat A cvg/lightglue https: custom https://github.com/cvg/LightGlue https://github.com/cvg/LightGlue 1x1x480x640 hpatches needs_adapter 官方权重/仓库 304
306 lightglue.sift LightGlue-SIFT feat A cvg/lightglue https: custom https://github.com/cvg/LightGlue https://github.com/cvg/LightGlue 1x1x480x640 hpatches needs_adapter 官方权重/仓库 305
307 loftr.indoor LoFTR-Indoor feat A zju3dv/loftr https: custom https://github.com/zju3dv/LoFTR https://github.com/zju3dv/LoFTR 1x1x480x640 hpatches needs_adapter 官方权重/仓库 306
308 loftr.outdoor LoFTR-Outdoor feat A zju3dv/loftr https: custom https://github.com/zju3dv/LoFTR https://github.com/zju3dv/LoFTR 1x1x480x640 hpatches needs_adapter 官方权重/仓库 307
309 matchformer.indoor MatchFormer-Indoor feat A jamycheung/matchformer https: custom https://github.com/jamycheung/MatchFormer https://github.com/jamycheung/MatchFormer 1x1x480x640 hpatches needs_adapter 官方权重/仓库 308
310 matchformer.outdoor MatchFormer-Outdoor feat A jamycheung/matchformer https: custom https://github.com/jamycheung/MatchFormer https://github.com/jamycheung/MatchFormer 1x1x480x640 hpatches needs_adapter 官方权重/仓库 309
311 aspanformer.indoor ASpanFormer-Indoor feat A apple/ml-aspanformer https: custom https://github.com/apple/ml-aspanformer https://github.com/apple/ml-aspanformer 1x1x480x640 hpatches needs_adapter 官方权重/仓库 310
312 aspanformer.outdoor ASpanFormer-Outdoor feat A apple/ml-aspanformer https: custom https://github.com/apple/ml-aspanformer https://github.com/apple/ml-aspanformer 1x1x480x640 hpatches needs_adapter 官方权重/仓库 311
313 dkm DKM feat A parskatt/dkm https: custom https://github.com/Parskatt/DKM https://github.com/Parskatt/DKM 1x1x480x640 hpatches needs_adapter 官方权重/仓库 312
314 roma RoMa feat A parskatt/roma https: custom https://github.com/Parskatt/RoMa https://github.com/Parskatt/RoMa 1x1x480x640 hpatches needs_adapter 官方权重/仓库 313
315 gluestick GlueStick feat A cvg/gluestick https: custom https://github.com/cvg/GlueStick https://github.com/cvg/GlueStick 1x1x480x640 hpatches needs_adapter 官方权重/仓库 314
316 xfeatpluslighterglue XFeat+LighterGlue feat A verlab/accelerated_features https: custom https://github.com/verlab/accelerated_features https://github.com/verlab/accelerated_features 1x1x480x640 hpatches needs_adapter 官方权重/仓库 315
317 monoscene.semantickitti MonoScene-SemanticKITTI occ A astra-vision/monoscene https: custom https://github.com/astra-vision/MonoScene https://github.com/astra-vision/MonoScene 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 316
318 monoscene.nyuv2 MonoScene-NYUv2 occ A astra-vision/monoscene https: custom https://github.com/astra-vision/MonoScene https://github.com/astra-vision/MonoScene 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 317
319 lmscnet.semantickitti LMSCNet-SemanticKITTI occ A astra-vision/lmscnet https: custom https://github.com/astra-vision/LMSCNet https://github.com/astra-vision/LMSCNet 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 318
320 tpvformer TPVFormer occ A wzzheng/tpvformer https: custom https://github.com/wzzheng/TPVFormer https://github.com/wzzheng/TPVFormer 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 319
321 voxformer.s VoxFormer-S occ A nvlabs/voxformer https: custom https://github.com/NVlabs/VoxFormer https://github.com/NVlabs/VoxFormer 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 320
322 voxformer.t VoxFormer-T occ A nvlabs/voxformer https: custom https://github.com/NVlabs/VoxFormer https://github.com/NVlabs/VoxFormer 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 321
323 voxformer.t.3d VoxFormer-T-3D occ A nvlabs/voxformer https: custom https://github.com/NVlabs/VoxFormer https://github.com/NVlabs/VoxFormer 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 322
324 occformer OccFormer occ A zhangyp15/occformer https: custom https://github.com/zhangyp15/OccFormer https://github.com/zhangyp15/OccFormer 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 323
325 surroundocc SurroundOcc occ A weiyithu/surroundocc https: custom https://github.com/weiyithu/SurroundOcc https://github.com/weiyithu/SurroundOcc 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 324
326 occnet.r50 OccNet-R50 occ A opendrivelab/occnet https: custom https://github.com/OpenDriveLab/OccNet https://github.com/OpenDriveLab/OccNet 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 325
327 flashocc.m0 FlashOcc-M0 occ A yzichen/flashocc https: custom https://github.com/Yzichen/FlashOCC https://github.com/Yzichen/FlashOCC 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 326
328 flashocc.m1 FlashOcc-M1 occ A yzichen/flashocc https: custom https://github.com/Yzichen/FlashOCC https://github.com/Yzichen/FlashOCC 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 327
329 panoptic.flashocc Panoptic-FlashOcc occ A yzichen/flashocc https: custom https://github.com/Yzichen/FlashOCC https://github.com/Yzichen/FlashOCC 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 328
330 sparseocc.vision.sjtu SparseOcc-VISION-SJTU occ A vision-sjtu/sparseocc https: custom https://github.com/VISION-SJTU/SparseOcc https://github.com/VISION-SJTU/SparseOcc 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 329
331 sparseocc.mcg.nju SparseOcc-MCG-NJU occ A mcg-nju/sparseocc https: custom https://github.com/MCG-NJU/SparseOcc https://github.com/MCG-NJU/SparseOcc 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 330
332 renderocc RenderOcc occ A pmj110119/renderocc https: custom https://github.com/pmj110119/RenderOcc https://github.com/pmj110119/RenderOcc 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 331
333 selfocc SelfOcc occ A huang-yh/selfocc https: custom https://github.com/huang-yh/SelfOcc https://github.com/huang-yh/SelfOcc 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 332
334 gaussianformer GaussianFormer occ A huang-yh/gaussianformer https: custom https://github.com/huang-yh/GaussianFormer https://github.com/huang-yh/GaussianFormer 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 333
335 gaussianformer.2 GaussianFormer-2 occ C huang-yh/gaussianformer https: custom https://github.com/huang-yh/GaussianFormer https://github.com/huang-yh/GaussianFormer 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 334
336 gaussianocc GaussianOcc occ A ganwanshui/gaussianocc https: custom https://github.com/GANWANSHUI/GaussianOcc https://github.com/GANWANSHUI/GaussianOcc 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 335
337 ctf.occ CTF-Occ occ A tsinghua-mars-lab/occ3d https: custom https://github.com/Tsinghua-MARS-Lab/Occ3D https://github.com/Tsinghua-MARS-Lab/Occ3D 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 336
338 bevdetocc BEVDetOCC occ A huangjunjie2017/bevdet https: custom https://github.com/HuangJunJie2017/BEVDet https://github.com/HuangJunJie2017/BEVDet 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 337
339 bevdetocc.4d.stereo BEVDetOCC-4D-Stereo occ A yzichen/flashocc https: custom https://github.com/Yzichen/FlashOCC https://github.com/Yzichen/FlashOCC 1x3x256x704 kitti2015 needs_adapter 官方仓库/model zoo 338

0
reports/.gitkeep Normal file
View File

2
requirements.txt Normal file
View File

@ -0,0 +1,2 @@
# harness 自身只依赖 pyyaml;工具链在各自 venv 里
pyyaml>=6.0

24
scripts/_probe.py Normal file
View File

@ -0,0 +1,24 @@
"""一次性诊断:轨迹到底有没有、agent 侧依赖齐不齐。scp 到机器上跑,不进 git 主流程。"""
import glob
import os
import shutil
import sqlite3
DB = "/home/ming01.wang/bench-state/jobs.sqlite"
R = "/mnt/data/ming01.wang/model-compile-hbdk/bench"
c = sqlite3.connect(DB)
c.execute("PRAGMA busy_timeout=30000")
print("rollouts by state:", dict(c.execute("select state,count(*) from rollouts group by 1").fetchall()))
print("rollouts executed (turns not null):", c.execute("select count(*) from rollouts where turns is not null").fetchone()[0])
print("verdict rows:", c.execute("select count(*) from rollouts where verdict is not null").fetchone()[0])
print("jobs by state:", dict(c.execute("select state,count(*) from jobs group by 1").fetchall()))
print("datasets/:", os.listdir(R + "/datasets") if os.path.isdir(R + "/datasets") else "-")
print("rollout dirs on disk:", len(glob.glob(R + "/runs/*/*/rollouts/*")))
print("events.jsonl on disk:", len(glob.glob(R + "/runs/*/*/rollouts/*/agent/events.jsonl")))
for b in ("pi", "node", "npm", "npx"):
print(f" {b} -> {shutil.which(b) or 'NOT FOUND'}")
od = os.path.expanduser("~/.opencode/bin")
print("~/.opencode/bin:", os.listdir(od) if os.path.isdir(od) else "-")
pi = os.path.expanduser("~/.pi")
print("~/.pi:", os.listdir(pi) if os.path.isdir(pi) else "NOT FOUND")

23
scripts/board_probe.sh Executable file
View File

@ -0,0 +1,23 @@
#!/usr/bin/env bash
# 确认沙箱板是哪块(s100 / s100p / s600 → nash-e / nash-m / nash-p),并把结果写进 registry/boards.yaml 提示。
# 用法:source bench/.env.local; scripts/board_probe.sh
# 第一次:先生成密钥并用口令 ssh-copy-id 一次(口令已泄露于明文对话,之后请轮换):
# ssh-keygen -t ed25519 -f ~/.ssh/rdk_sandbox -N ''
# ssh-copy-id -i ~/.ssh/rdk_sandbox.pub -p 40010 <user>@rdk-sandbox.d-robotics.cc
set -uo pipefail
: "${BOARD_HOST:?source bench/.env.local}"; : "${BOARD_PORT:=22}"; : "${BOARD_USER:?}"; : "${BOARD_SSH_KEY:=~/.ssh/rdk_sandbox}"
KEY="${BOARD_SSH_KEY/#\~/$HOME}"
SSH="ssh -o BatchMode=yes -o ConnectTimeout=15 -o StrictHostKeyChecking=accept-new -i $KEY -p $BOARD_PORT $BOARD_USER@$BOARD_HOST"
echo "== device-tree model"; $SSH 'cat /proc/device-tree/model 2>/dev/null; echo'
echo "== hrut_somstatus"; $SSH 'hrut_somstatus 2>/dev/null | head -20 || echo n/a'
echo "== ucp / hrt version"; $SSH 'cat /etc/ucp_version 2>/dev/null; hrt_model_exec --version 2>/dev/null | head -2; hrut_bpuprofile --version 2>/dev/null | head -1'
echo "== cpuinfo / mem"; $SSH 'grep -m1 -i "model name\|hardware" /proc/cpuinfo; nproc; free -m | head -2'
echo "== bpu nodes"; $SSH 'ls /sys/class/bpu* /dev/bpu* 2>/dev/null; cat /sys/class/bpu*/bpu*/ratio 2>/dev/null'
echo "== disk"; $SSH 'df -h /userdata 2>/dev/null || df -h / | tail -1'
MODEL="$($SSH 'cat /proc/device-tree/model 2>/dev/null' | tr -d '\0' | tr 'A-Z' 'a-z')"
case "$MODEL" in
*s100p*|*nash-m*|*nashm*) echo ">> 推断 s100p → march nash-m" ;;
*s100*|*nash-e*|*nashe*) echo ">> 推断 s100 → march nash-e" ;;
*s600*|*nash-p*|*nashp*) echo ">> 推断 s600 → march nash-p" ;;
*) echo ">> 无法从 '$MODEL' 推断;看 hrut_somstatus 输出后手动填 registry/boards.yaml 的 board_type/march" ;;
esac

240
scripts/excel_to_models.py Normal file
View File

@ -0,0 +1,240 @@
"""《视觉ModelZoo_10类_不含VLM_338个候选模型.xlsx》→ registry/models.csv。
~/toolchains/bench-venv/bin/python scripts/excel_to_models.py registry/models_源清单.xlsx [--dry-run]
sheet「模型清单」列:ID | 任务分类 | 模型/有效变体 | 首选评测集 | 补充评测集 | 指标 | 官方仓库/模型库 | 权重下载位置 | 下载说明 | 端侧优先级 | 数据集入口
每行给一个 status,决定它现在能不能自动跑:
ready harness 现在就能端到端跑:torchvision 分类(权重 API 自动下)、ultralytics 检测(权重名自动下)
needs_eval 能取权重、能导出,但评测器还没写(ultralytics seg / pose;P4)
needs_adapter 需要模型仓库专用的取权重/导出适配器(mmdet / mmseg / mmpose / 各家 GitHub 仓库;P4–P5)
skip 不是神经网络(SIFT / ORB / AKAZE),不量化
matrix.yaml 默认只展开 status=ready。其余留在 registry 里,bench status 会报"未开始:needs_adapter N"。
torchvision 的 builder 名与默认权重的预处理尺寸从 torchvision 本身查(所以要在 bench venv 里跑)。
"""
from __future__ import annotations
import argparse
import csv
import re
import sys
from collections import Counter
from pathlib import Path
TASK_MAP = [ # 顺序即优先级:先匹配更具体的
("iseg", ["实例分割", "提示分割"]),
("stereo", ["双目", "视差", "立体"]),
("occ", ["占据", "占用", "occupancy"]),
("flow", ["光流"]),
("feat", ["局部特征", "特征提取", "匹配"]),
("pose", ["关键点", "姿态"]),
("mde", ["单目深度", "深度"]),
("seg", ["语义分割", "分割"]),
("det", ["目标检测", "检测"]),
("cls", ["分类"]),
]
DATASET_MAP = [("imagenet1k", ["imagenet"]), ("coco2017", ["coco"]), ("ade20k", ["ade20k"]), ("cityscapes", ["cityscapes"]),
("lvis", ["lvis"]), ("kitti2015", ["kitti 2015", "kitti2015", "kitti"]), ("nyuv2", ["nyu"]), ("sintel", ["sintel"]),
("hpatches", ["hpatches"]), ("occ3d_nuscenes", ["occ3d", "nuscenes"]), ("eth3d", ["eth3d"]), ("semantickitti", ["semantickitti"])]
DEFAULT_INPUT = {"cls": "1x3x224x224", "det": "1x3x640x640", "iseg": "1x3x640x640", "pose": "1x3x640x640",
"seg": "1x3x512x1024", "mde": "1x3x518x518", "stereo": "1x3x384x1248", "flow": "1x3x384x1024",
"feat": "1x1x480x640", "occ": "1x3x256x704"}
CLASSICAL = {"sift", "orb", "akaze"}
def norm(s) -> str:
return re.sub(r"\s+", " ", str(s or "")).strip()
def slug(s: str) -> str:
s = s.lower().replace("+", "plus").replace("/", "-").replace(" ", "-").replace("_", "-")
s = re.sub(r"[^a-z0-9.\-]", "", s)
s = re.sub(r"-+", "-", s).strip("-.")
return s.replace("-", ".") # job_id 段内不允许 '-'
def map_task(v: str) -> str | None:
for t, keys in TASK_MAP:
if any(k in v for k in keys):
return t
return None
def map_dataset(v: str, task: str) -> str:
low = v.lower()
for ds, keys in DATASET_MAP:
if any(k in low for k in keys):
return ds
return {"cls": "imagenet1k", "det": "coco2017", "iseg": "coco2017", "pose": "coco2017", "seg": "ade20k",
"mde": "nyuv2", "stereo": "kitti2015", "flow": "sintel", "feat": "hpatches", "occ": "occ3d_nuscenes"}[task]
def map_priority(v: str) -> str:
return v.strip()[:1].upper() if v.strip()[:1].upper() in "ABC" else "C"
def source_of(repo: str) -> tuple[str, str]:
"""→ (source, family)"""
r = repo.lower()
if "pytorch.org/vision" in r or "torchvision" in r:
return "torchvision", "torchvision"
if "ultralytics" in r:
return "ultralytics", "ultralytics"
for fam in ("mmsegmentation", "mmdetection", "mmpose", "mmdepth", "mmflow"):
if fam in r:
return "openmmlab", fam
if "opencv" in r:
return "opencv", "opencv"
m = re.search(r"github\.com/([^/]+/[^/]+)", r)
if m:
return "custom", m.group(1)
m = re.search(r"huggingface\.co/([^/]+/[^/]+)", r)
if m:
return "hf", m.group(1)
return "custom", r[:40]
# ---------------------------------------------------------------- torchvision 解析
_TV_MODELS: list[str] | None = None
def tv_models() -> list[str]:
global _TV_MODELS
if _TV_MODELS is None:
try:
import torchvision.models as tvm
_TV_MODELS = sorted(tvm.list_models())
except Exception as e: # noqa: BLE001
print(f"WARN torchvision 不可用({e}),builder 名靠规则猜", file=sys.stderr)
_TV_MODELS = []
return _TV_MODELS
def tv_builder(display: str) -> str | None:
"""'ResNeXt-50-32x4d' → 'resnext50_32x4d';'ViT-B/16' → 'vit_b_16';'ShuffleNetV2-x1.0' → 'shufflenet_v2_x1_0'"""
key = re.sub(r"[^a-z0-9]", "", display.lower())
cands = {re.sub(r"[^a-z0-9]", "", m): m for m in tv_models()}
if key in cands:
return cands[key]
# 常见写法差异
alt = key.replace("mobilenetv2", "mobilenetv2").replace("wideresnet", "wideresnet")
for k, m in cands.items():
if k == alt:
return m
# 'regnety400mf' → 'regnet_y_400mf';'maxvitt' → 'maxvit_t'
for k, m in cands.items():
if k.replace("_", "") == key:
return m
return None
def tv_meta(builder: str) -> dict:
"""DEFAULT 权重的 resize / crop / 参数量。"""
try:
import torchvision.models as tvm
w = tvm.get_model_weights(builder).DEFAULT
t = w.transforms()
crop = getattr(t, "crop_size", [224])
resize = getattr(t, "resize_size", [256])
crop = crop[0] if isinstance(crop, (list, tuple)) else crop
resize = resize[0] if isinstance(resize, (list, tuple)) else resize
return {"weights": f"torchvision://{builder}/{w.name}", "crop": int(crop), "resize": int(resize),
"params_m": round(w.meta.get("num_params", 0) / 1e6, 2) if w.meta.get("num_params") else "",
"official": w.meta.get("_metrics", {}).get("ImageNet-1K", {}).get("acc@1", "")}
except Exception as e: # noqa: BLE001
return {"weights": f"torchvision://{builder}/DEFAULT", "crop": 224, "resize": 256, "params_m": "", "official": "",
"note": f"tv_meta failed: {e}"}
# ---------------------------------------------------------------- 主流程
def convert(xlsx: Path) -> tuple[list[dict], list[str]]:
import openpyxl
wb = openpyxl.load_workbook(xlsx, read_only=True, data_only=True)
ws = wb["模型清单"] if "模型清单" in wb.sheetnames else wb.worksheets[0]
rows = list(ws.iter_rows(values_only=True))
hdr = [norm(c) for c in rows[0]]
col = {name: hdr.index(name) for name in hdr}
need = ["任务分类", "模型/有效变体", "首选评测集", "官方仓库/模型库", "权重下载位置", "下载说明", "端侧优先级"]
for n in need:
if n not in col:
raise SystemExit(f"Excel 缺列 {n!r},现有:{hdr}")
out, warns, seen = [], [], Counter()
for r in rows[1:]:
if not any(c not in (None, "") for c in r):
continue
g = lambda name: norm(r[col[name]])
name = g("模型/有效变体")
task = map_task(g("任务分类"))
if not task:
warns.append(f"{name}: 任务 {g('任务分类')!r} 无法映射"); continue
source, family = source_of(g("官方仓库/模型库") + " " + g("权重下载位置"))
mid = slug(name)
seen[mid] += 1
if seen[mid] > 1:
mid = f"{mid}.{seen[mid]}"
rec = {"model_id": mid, "name": name, "task": task, "priority": map_priority(g("端侧优先级")),
"family": family, "source": source, "repo": g("官方仓库/模型库"), "weights": g("权重下载位置"),
"input_spec": DEFAULT_INPUT[task], "resize": "", "dataset": map_dataset(g("首选评测集"), task),
"params_m": "", "official_metric": "", "status": "needs_adapter",
"notes": g("下载说明"), "excel_id": norm(r[col["ID"]]) if "ID" in col else ""}
low = name.lower()
if source == "torchvision":
b = tv_builder(name)
if not b:
warns.append(f"{name}: torchvision 里找不到 builder"); rec["status"] = "needs_adapter"
else:
m = tv_meta(b)
rec.update({"model_id": b.replace("_", "."), "weights": m["weights"], "params_m": m["params_m"],
"official_metric": m["official"], "resize": m["resize"]})
if task == "cls":
rec["input_spec"] = f"1x3x{m['crop']}x{m['crop']}"
rec["status"] = "ready"
else:
rec["status"] = "needs_adapter" # RAFT/SpyNet 双输入
if m.get("note"):
warns.append(f"{name}: {m['note']}")
elif source == "ultralytics":
wname = low.replace("yolo11", "yolo11").replace(" ", "") + ".pt" # yolov8n / yolo11n-seg / fastsam-s
if wname.startswith("fastsam"):
wname = "FastSAM-" + wname.split("-", 1)[1]
rec["weights"] = f"ultralytics://{wname}"
rec["status"] = "ready" if task == "det" else "needs_eval"
elif source == "opencv" or any(k == re.sub(r"[^a-z]", "", low) for k in CLASSICAL):
rec["status"] = "skip"; rec["notes"] = "经典算子,非神经网络,不量化"
out.append(rec)
return out, warns
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("xlsx")
ap.add_argument("--out", default=str(Path(__file__).resolve().parent.parent / "registry" / "models.csv"))
ap.add_argument("--dry-run", action="store_true")
a = ap.parse_args()
rows, warns = convert(Path(a.xlsx))
for w in warns:
print("WARN", w, file=sys.stderr)
print("rows:", len(rows), file=sys.stderr)
print("task:", dict(Counter(r["task"] for r in rows)), file=sys.stderr)
print("priority:", dict(Counter(r["priority"] for r in rows)), file=sys.stderr)
print("status:", dict(Counter(r["status"] for r in rows)), file=sys.stderr)
print("family:", dict(Counter(r["family"] for r in rows).most_common(12)), file=sys.stderr)
ready = [r for r in rows if r["status"] == "ready"]
print(f"ready ({len(ready)}):", ", ".join(r["model_id"] for r in ready), file=sys.stderr)
dup = [k for k, v in Counter(r["model_id"] for r in rows).items() if v > 1]
if dup:
raise SystemExit(f"model_id 重复:{dup}")
if a.dry_run:
return 0
fields = list(rows[0].keys())
with open(a.out, "w", newline="", encoding="utf-8") as f:
w = csv.DictWriter(f, fieldnames=fields); w.writeheader(); w.writerows(rows)
print(f"wrote {len(rows)} → {a.out}", file=sys.stderr)
return 0
if __name__ == "__main__":
sys.exit(main())

23
scripts/fix_env.sh Executable file
View File

@ -0,0 +1,23 @@
#!/usr/bin/env bash
# 修 .env.oe-package 里的死路径:PYTHON_BIN 指向 <项目根>/oellm2/venv/bin/python,实际 QAT venv 在 ~/toolchains/oellm2/venv。
# agent 按规则会读这个文件,所以 P1 第一件事。备份为 .env.oe-package.bak-<日期>。
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
F="$ROOT/.horizon/.env.oe-package"; [ -f "$F" ] || F="$ROOT/.env.oe-package"
[ -f "$F" ] || { echo "没有 .env.oe-package(找了 .horizon/ 与项目根)"; exit 1; }
NEW="${1:-$HOME/toolchains/oellm2/venv/bin/python}"
[ -x "$NEW" ] || { echo "目标解释器不存在: $NEW"; exit 2; }
cp -n "$F" "$F.bak-$(date +%Y%m%d)" || true
echo "before:"; grep -n '^PYTHON_BIN=' "$F" || echo " (no PYTHON_BIN line)"
if grep -q '^PYTHON_BIN=' "$F"; then
sed -i.tmp "s|^PYTHON_BIN=.*|PYTHON_BIN=$NEW|" "$F" && rm -f "$F.tmp"
else
echo "PYTHON_BIN=$NEW" >> "$F"
fi
echo "after:"; grep -n '^PYTHON_BIN=' "$F"
# 顺手核对所有以 / 或 ~ 开头的路径值
echo "--- 其他路径值核对 ---"
grep -E '^[A-Z_]+=(/|~|\$HOME)' "$F" | while IFS='=' read -r k v; do
p="${v/#\~/$HOME}"; p="${p/#\$HOME/$HOME}"
[ -e "$p" ] && echo "ok $k=$v" || echo "MISS $k=$v"
done

49
scripts/gate_summary.py Normal file
View File

@ -0,0 +1,49 @@
"""每个已结束 job 的四道门一行摘要(看 FAIL 到底败在哪一门、差多少)。
python3 scripts/gate_summary.py [--run r…] [--state FAIL]
"""
from __future__ import annotations
import argparse
import json
import sqlite3
import sys
from collections import Counter
from pathlib import Path
BENCH = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(BENCH))
from harness import paths # noqa: E402
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--run", action="append")
ap.add_argument("--state", default=None)
a = ap.parse_args()
con = sqlite3.connect(paths.DB_PATH)
runs = a.run or [con.execute("select run_id from runs order by created_at desc limit 1").fetchone()[0]]
q = "select run_id, job_id, state, error_class, stage_failed, result from jobs where mode='oracle' and run_id in (%s)" % ",".join("?" * len(runs))
args = list(runs)
if a.state:
q += " and state=?"; args.append(a.state)
q += " and state in ('PASS','FAIL','UNVERIFIED') order by run_id, job_id"
fg = Counter()
for rid, jid, st, ec, sf, res in con.execute(q, args):
r = json.loads(res) if res else {}
g = r.get("gates", {})
g1, g2, g3, g4 = (g.get(k, {}) for k in ("G1", "G2", "G3", "G4"))
fg[(st, r.get("failed_gate") or (sf and f"stage:{sf}") or "-")] += 1
f1 = f"/{','.join(g1.get('failed', []))}" if g1.get("failed") else (f"/miss:{','.join(g1.get('missing', []))}" if g1.get("missing") else "")
cos = g2.get("cos_min")
print(f"{rid} {jid:<46} {st:<10} fg={r.get('failed_gate') or '-':<3} ec={ec or '-':<14} "
f"G1={g1.get('verdict', '-')}{f1} G2={g2.get('verdict', '-')}{'' if cos is None else f'({cos:.4f})'} "
f"G3={g3.get('verdict', '-')}"
+ ("" if g3.get("fp32") is None else f"(fp32={g3['fp32']:.2f} q={g3['quant']:.2f} d={g3['delta']:+.2f} tol={g3.get('effective_tolerance', 0):.2f})")
+ (f" lat_est={g1.get('static_perf', {}).get('latency_ms_est')}ms" if g1.get("static_perf", {}).get("latency_ms_est") else ""))
print("\nsummary:", dict(fg))
return 0
if __name__ == "__main__":
sys.exit(main())

223
scripts/ingest_pool.py Normal file
View File

@ -0,0 +1,223 @@
"""数据入池:按 registry/datasets.yaml 的有序来源,把数据集复制进 data/pool/<ds>/<version>/,写 manifest.json,只读。
python scripts/ingest_pool.py --dataset imagenet1k [--dataset coco2017 …] [--all-local]
python scripts/ingest_pool.py --dataset imagenet1k --verify # 只校验已入池的 manifest
复制,不软链——别人的目录随时可能被清。入池后 chmod -R a-w。
**data/ 必须在本地盘**:JuiceFS 小文件写入实测 ~0.5 s/文件,5 万张要 7 小时;本地盘几分钟。
bench/data -> /home/<user>/bench-data 软链(README §0)。"复制不软链"说的是不要软链到别人的目录。
pool 布局(各任务评测器依赖):
imagenet1k/<ver>/images/<idx>_<label>.JPEG + labels.json {file: label}
coco2017/<ver>/val2017/*.jpg + annotations/instances_val2017.json, person_keypoints_val2017.json
cityscapes/<ver>/leftImg8bit/val/… + gtFine/val/…
kitti2015/<ver>/training/{image_2,image_3,disp_occ_0,flow_occ}/…
"""
from __future__ import annotations
import argparse
import fnmatch
import hashlib
import json
import os
import shutil
import subprocess
import sys
import tarfile
import time
import zipfile
from pathlib import Path
import yaml
BENCH = Path(__file__).resolve().parent.parent
REG = yaml.safe_load((BENCH / "registry" / "datasets.yaml").read_text())
LOCAL_ROOT = Path(REG.get("local_root", "/mnt/data"))
POOL = BENCH / REG.get("pool_root", "data/pool")
def sha256_tree(root: Path) -> tuple[str, int]:
"""确定性:按相对路径排序,hash(path + 文件 sha256)。返回 (sha, 文件数)。"""
h = hashlib.sha256()
n = 0
for p in sorted(x for x in root.rglob("*") if x.is_file() and x.name != "manifest.json"):
rel = str(p.relative_to(root)).encode()
fh = hashlib.sha256()
with open(p, "rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
fh.update(chunk)
h.update(rel); h.update(fh.digest()); n += 1
return h.hexdigest(), n
def write_manifest(dst: Path, ds: str, ver: str, source: dict, extra: dict | None = None) -> dict:
sha, n = sha256_tree(dst)
man = {"dataset": ds, "version": ver, "sha256": sha, "count": n, "source": source,
"ingested_at": time.strftime("%Y-%m-%d %H:%M:%S"), **(extra or {})}
(dst / "manifest.json").write_text(json.dumps(man, indent=2, ensure_ascii=False))
subprocess.run(["chmod", "-R", "a-w", str(dst)], check=False)
return man
# ---------------------------------------------------------------- sources
def src_local_parquet(src: dict, dst: Path) -> dict:
"""HF parquet(image bytes + label)→ images/ + labels.json。imagenet 专用。"""
import io
try:
import pyarrow.parquet as pq
from PIL import Image
except ImportError:
sys.exit("需要 pyarrow + pillow:在任一 venv 里跑本脚本,或 pip install pyarrow pillow")
root = LOCAL_ROOT / src["path"]
shards = sorted(root.glob(src["glob"]))
if src.get("expect_shards") and len(shards) != src["expect_shards"]:
raise RuntimeError(f"parquet 片数 {len(shards)} != {src['expect_shards']}:{root}")
(dst / "images").mkdir(parents=True, exist_ok=True)
labels, idx = {}, 0
for sh in shards:
t = pq.read_table(sh)
cols = t.column_names
img_col = "image" if "image" in cols else cols[0]
lab_col = "label" if "label" in cols else cols[1]
for img, lab in zip(t.column(img_col).to_pylist(), t.column(lab_col).to_pylist()):
data = img.get("bytes") if isinstance(img, dict) else img
if data is None and isinstance(img, dict) and img.get("path"):
data = Path(img["path"]).read_bytes()
fn = f"{idx:06d}_{int(lab):04d}.JPEG"
# 原样落盘(parquet 里就是原始 JPEG 字节):不重编码,不改变像素;评测器 Image.open().convert("RGB") 统一处理 CMYK/PNG 伪装
(dst / "images" / fn).write_bytes(data)
labels[fn] = int(lab); idx += 1
print(f" {sh.name}: 累计 {idx}", file=sys.stderr)
(dst / "labels.json").write_text(json.dumps(labels))
return {"kind": "local_parquet", "path": str(root), "shards": len(shards), "images": idx}
def src_local_tar(src: dict, dst: Path) -> dict:
p = LOCAL_ROOT / src["path"]
print(f" 校验 {p} …", file=sys.stderr)
n = int(subprocess.run(f"tar -tzf '{p}' | grep -c -v '/$'", shell=True, capture_output=True, text=True).stdout.strip() or 0)
if src.get("expect") and n != src["expect"]:
raise RuntimeError(f"tar 条目 {n} != {src['expect']}(旁边有 .tmp 残片,可能下载不完整)")
(dst / "images").mkdir(parents=True, exist_ok=True)
with tarfile.open(p) as t:
t.extractall(dst / "images")
return {"kind": "local_tar", "path": str(p), "entries": n, "note": "labels.json 需从 classes.py / 官方 devkit 生成"}
def _annotations(src: dict, dst: Path) -> None:
for ann in src.get("annotations") or []:
ap = LOCAL_ROOT / ann
if ap.suffix == ".zip":
with zipfile.ZipFile(ap) as z:
for m in z.namelist():
if m.endswith(".json") and "val2017" in m:
z.extract(m, dst)
else:
shutil.copy2(ap, dst / ap.name)
def src_local_dir(src: dict, dst: Path) -> dict:
"""注意:JuiceFS 上按文件 rsync 实测只有 5–10 文件/s;有 zip 就优先 zip(--prefer local_zip)。"""
p = LOCAL_ROOT / src["path"]
if not p.exists():
raise FileNotFoundError(p)
tgt = dst / p.name
print(f" 复制 {p} → {tgt}", file=sys.stderr)
subprocess.run(["rsync", "-a", f"{p}/", f"{tgt}/"], check=True)
_annotations(src, dst)
return {"kind": "local_dir", "path": str(p), "annotations": src.get("annotations")}
def src_local_zip(src: dict, dst: Path) -> dict:
items = [src] + list(src.get("extra") or [])
done = []
for it in items:
p = LOCAL_ROOT / it["path"]
pat = it.get("extract_only")
print(f" 解压 {p} ({pat or 'all'})", file=sys.stderr)
with zipfile.ZipFile(p) as z:
names = [m for m in z.namelist() if not pat or fnmatch.fnmatch(m, pat)]
for m in names:
z.extract(m, dst)
done.append({"path": str(p), "n": len(names)})
_annotations(src, dst)
return {"kind": "local_zip", "items": done}
def src_hf(src: dict, dst: Path) -> dict:
if not src.get("repo"):
raise RuntimeError("hf 源没有 repo")
print(f" hf download {src['repo']} → {dst}", file=sys.stderr)
subprocess.run(["hf", "download", src["repo"], "--repo-type", "dataset", "--local-dir", str(dst)], check=True)
return {"kind": "hf", "repo": src["repo"]}
HANDLERS = {"local_parquet": src_local_parquet, "local_tar": src_local_tar, "local_dir": src_local_dir,
"local_zip": src_local_zip, "hf": src_hf}
def ingest(ds: str, force: bool = False, prefer: str | None = None) -> None:
d = REG["datasets"][ds]
ver = d.get("version", "v1")
dst = POOL / ds / ver
if (dst / "manifest.json").exists() and not force:
print(f"{ds}/{ver} 已入池(--force 重做)", file=sys.stderr); return
if dst.exists():
subprocess.run(["chmod", "-R", "u+w", str(dst)], check=False)
shutil.rmtree(dst)
dst.mkdir(parents=True)
errors = []
sources = sorted(d["sources"], key=lambda s: 0 if s["kind"] == prefer else 1) if prefer else d["sources"]
for src in sources:
h = HANDLERS.get(src["kind"])
if not h:
errors.append(f"{src['kind']}: 未实现(github/official 请手动下载到 {dst} 后 --verify)"); continue
try:
print(f"[{ds}] 尝试来源 {src['kind']} …", file=sys.stderr)
info = h(src, dst)
man = write_manifest(dst, ds, ver, info)
if d.get("expect_count") and man["count"] < d["expect_count"]:
print(f" WARN count {man['count']} < expect {d['expect_count']}", file=sys.stderr)
print(f"[{ds}] 入池完成 sha={man['sha256'][:12]} count={man['count']}", file=sys.stderr)
return
except Exception as e:
errors.append(f"{src['kind']}: {e}")
print(f" 失败:{e}", file=sys.stderr)
for x in dst.iterdir():
shutil.rmtree(x) if x.is_dir() else x.unlink()
raise SystemExit(f"[{ds}] 所有来源失败:\n " + "\n ".join(errors))
def verify(ds: str) -> bool:
d = REG["datasets"][ds]
dst = POOL / ds / d.get("version", "v1")
m = json.loads((dst / "manifest.json").read_text())
sha, n = sha256_tree(dst)
ok = sha == m["sha256"] and n == m["count"]
print(f"{ds}: {'OK' if ok else 'MISMATCH'} sha={sha[:12]} count={n}")
return ok
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--dataset", action="append", default=[])
ap.add_argument("--all-local", action="store_true", help="所有第一来源是 local_* 的数据集")
ap.add_argument("--verify", action="store_true")
ap.add_argument("--force", action="store_true")
ap.add_argument("--prefer", help="优先尝试的来源 kind,如 local_zip")
a = ap.parse_args()
names = list(a.dataset)
if a.all_local:
names += [k for k, v in REG["datasets"].items() if v["sources"][0]["kind"].startswith("local_") and k not in names]
if not names:
ap.error("--dataset 或 --all-local")
for ds in names:
if a.verify:
verify(ds)
else:
ingest(ds, a.force, a.prefer)
return 0
if __name__ == "__main__":
sys.exit(main())

149
scripts/make_calib.py Normal file
View File

@ -0,0 +1,149 @@
"""T0 校准集:data/calib/<task>/<input_spec>/NNNNN.npy + manifest.json。同规格模型共用。
python scripts/make_calib.py --task cls --input-spec 1x3x224x224 [--n 128] [--from auto|train|val-holdout]
来源优先级(auto):
1. datasets.yaml calib.train_parquet(HF train-*.parquet,固定 seed 选片、选行)
2. datasets.yaml calib.train_dir(本机 train 图片目录)
3. pool 里带 "train" 的图片
4. val-holdout:从 val 里抽 *不在 T1 里* 的图 —— 与 T2 全量有交集,manifest 记 source,出交付数前必须换 train
预处理与评测器完全同一份代码(harness/eval/common.py),layout NCHW,含 batch 维。
"""
from __future__ import annotations
import argparse
import hashlib
import io
import json
import random
import sys
import tarfile
from pathlib import Path
import yaml
BENCH = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(BENCH))
REG = yaml.safe_load((BENCH / "registry" / "datasets.yaml").read_text())
POOL = BENCH / REG.get("pool_root", "data/pool")
LOCAL = Path(REG.get("local_root", "/mnt/data"))
TASK_DATASET = {"cls": "imagenet1k", "det": "coco2017", "iseg": "coco2017", "pose": "coco2017", "seg": "cityscapes",
"mde": "nyuv2", "stereo": "kitti2015", "flow": "kitti2015", "feat": "hpatches", "occ": "nuscenes"}
EXTS = {".jpg", ".jpeg", ".png", ".JPEG", ".JPG"}
def list_images(root: Path, split_hint: str = "") -> list[Path]:
return sorted(p for p in root.rglob("*") if p.suffix in EXTS and split_hint in str(p.relative_to(root)).lower())
def t1_files(ds: str) -> set[str]:
d = REG["datasets"][ds]
tar = BENCH / "data" / "t1" / ds / f"{(d.get('t1') or {}).get('seed', 20260914)}.tar"
if not tar.exists():
return set()
with tarfile.open(tar) as t:
man = json.loads(t.extractfile("manifest.json").read())
return {it["src"] for it in man["items"]}
def from_train_parquet(cfg: dict, n: int, seed: int):
"""返回 (PIL 图列表, 描述列表)。固定 seed 选 4 片、每片均匀抽。"""
import pyarrow.parquet as pq
from PIL import Image
root = LOCAL / cfg["train_parquet"]
shards = sorted(root.glob(cfg.get("train_glob", "train-*.parquet")))
if not shards:
raise FileNotFoundError(f"no train parquet under {root}")
rng = random.Random(seed)
picked = sorted(rng.sample(shards, min(4, len(shards))))
per = -(-n // len(picked))
imgs, desc = [], []
for sh in picked:
pf = pq.ParquetFile(sh)
total = pf.metadata.num_rows
rows = sorted(rng.sample(range(total), min(per, total)))
names = pf.schema_arrow.names # 顶层列名(HF: image{bytes,path}, label)
img_col = "image" if "image" in names else names[0]
col = pf.read(columns=[img_col]).column(0)
for r in rows:
v = col[r].as_py()
data = v.get("bytes") if isinstance(v, dict) else v
imgs.append(Image.open(io.BytesIO(data)).convert("RGB")); desc.append(f"{sh.name}#{r}")
if len(imgs) >= n:
return imgs, desc
return imgs, desc
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--task", required=True)
ap.add_argument("--input-spec", required=True)
ap.add_argument("--n", type=int, default=128)
ap.add_argument("--from", dest="src", choices=["auto", "train", "val-holdout"], default="auto")
ap.add_argument("--seed", type=int, default=20260914)
a = ap.parse_args()
import numpy as np
from PIL import Image
from harness.eval.common import preprocess_cls, letterbox
ds = TASK_DATASET[a.task]
d = REG["datasets"][ds]
ccfg = d.get("calib") or {}
pool = POOL / ds / d.get("version", "v1")
shape = tuple(int(x) for x in a.input_spec.split("x"))
rng = random.Random(a.seed)
imgs, desc, source = [], [], None
if a.src in ("auto", "train"):
if ccfg.get("train_parquet") and (LOCAL / ccfg["train_parquet"]).exists():
try:
imgs, desc = from_train_parquet(ccfg, a.n, a.seed); source = "train-parquet"
except Exception as e:
print(f"WARN train_parquet 失败:{e}", file=sys.stderr)
if not imgs and ccfg.get("train_dir") and (LOCAL / ccfg["train_dir"]).exists():
files = list_images(LOCAL / ccfg["train_dir"])
if len(files) >= a.n:
rng.shuffle(files); files = sorted(files[:a.n])
imgs = [Image.open(p).convert("RGB") for p in files]; desc = [str(p) for p in files]; source = "train-dir"
if not imgs:
files = list_images(pool, "train")
if len(files) >= a.n:
rng.shuffle(files); files = sorted(files[:a.n])
imgs = [Image.open(p).convert("RGB") for p in files]; desc = [str(p.relative_to(pool)) for p in files]; source = "train"
if not imgs:
if a.src == "train":
raise SystemExit("没有可用的 train 来源")
excl = t1_files(ds)
files = [p for p in list_images(pool) if str(p.relative_to(pool)) not in excl]
if len(files) < a.n:
raise SystemExit(f"val-holdout 可用图片 {len(files)} < {a.n}")
rng.shuffle(files); files = sorted(files[:a.n])
imgs = [Image.open(p).convert("RGB") for p in files]; desc = [str(p.relative_to(pool)) for p in files]; source = "val-holdout"
print(f"WARN: {ds} 没有 train 来源,用 val-holdout(排除 T1 的 {len(excl)} 张)。出 T2 交付数前须换 train。", file=sys.stderr)
out = BENCH / "data" / "calib" / a.task / a.input_spec
npy_dir = out / "npy_f32" # hb_compile 读 cal_data_dir 下的全部文件 → manifest.json 不能放同一目录;_f32 后缀是 OE 约定
if out.exists():
import shutil; shutil.rmtree(out)
npy_dir.mkdir(parents=True, exist_ok=True)
h = hashlib.sha256()
for i, img in enumerate(imgs[:a.n]):
if a.task == "cls":
x = preprocess_cls(img, size=shape[-1], resize=round(shape[-1] * 256 / 224))
else:
x = letterbox(img, shape[-1])[0]
assert x.shape == shape, (x.shape, shape)
fp = npy_dir / f"{i:05d}.npy"
np.save(fp, x.astype(np.float32))
h.update(fp.read_bytes())
(out / "manifest.json").write_text(json.dumps({
"task": a.task, "input_spec": a.input_spec, "count": len(imgs[:a.n]), "seed": a.seed, "source": source,
"dataset": ds, "sha256": h.hexdigest(), "files": desc[:a.n],
"preprocess": "harness.eval.common (same as evaluator)", "layout": "NCHW", "dtype": "float32", "npy_dir": "npy_f32"}, indent=2))
print(f"{out} n={len(imgs[:a.n])} source={source} sha={h.hexdigest()[:12]}")
return 0
if __name__ == "__main__":
sys.exit(main())

120
scripts/make_t1.py Normal file
View File

@ -0,0 +1,120 @@
"""T1 smoke 子集:从 pool 固定 seed 抽样 → data/t1/<ds>/<seed>.tar(单包,避开 JuiceFS 小文件)。
python scripts/make_t1.py --dataset imagenet1k [--dataset coco2017]
tar 内:manifest.json(items 清单 + 内容 sha)、images/…、(det/iseg/pose)annotations.json(COCO 子集)。
tar 确定性:条目按名排序、mtime=0、uid/gid=0 → 同一 pool 同一 seed 产出同一 sha。
外部 data/t1/<ds>/manifest.json 记 tar 文件 sha256(job.json 钉的就是它)。
"""
from __future__ import annotations
import argparse
import hashlib
import io
import json
import random
import sys
import tarfile
from collections import defaultdict
from pathlib import Path
import yaml
BENCH = Path(__file__).resolve().parent.parent
REG = yaml.safe_load((BENCH / "registry" / "datasets.yaml").read_text())
POOL = BENCH / REG.get("pool_root", "data/pool")
T1 = BENCH / "data" / "t1"
def _add(tar: tarfile.TarFile, name: str, data: bytes) -> None:
ti = tarfile.TarInfo(name)
ti.size = len(data); ti.mtime = 0; ti.uid = ti.gid = 0; ti.uname = ti.gname = ""
tar.addfile(ti, io.BytesIO(data))
def items_imagenet(pool: Path, n: int, seed: int, stratify: str) -> list[dict]:
labels = json.loads((pool / "labels.json").read_text())
by = defaultdict(list)
for fn, lab in labels.items():
by[lab].append(fn)
rng = random.Random(seed)
items = []
if stratify == "label":
per = max(1, n // len(by))
for lab in sorted(by):
fs = sorted(by[lab]); rng.shuffle(fs)
items += [{"id": f, "file": f"images/{f}", "src": f"images/{f}", "label": lab} for f in fs[:per]]
else:
fs = sorted(labels); rng.shuffle(fs)
items = [{"id": f, "file": f"images/{f}", "src": f"images/{f}", "label": labels[f]} for f in fs[:n]]
return items[:n]
def items_coco(pool: Path, n: int, seed: int) -> tuple[list[dict], dict]:
ann_p = next(pool.rglob("instances_val2017.json"))
ann = json.loads(ann_p.read_text())
imgs = sorted(ann["images"], key=lambda x: x["id"])
rng = random.Random(seed); rng.shuffle(imgs)
sel = sorted(imgs[:n], key=lambda x: x["id"])
ids = {im["id"] for im in sel}
sub = {"images": sel, "annotations": [a for a in ann["annotations"] if a["image_id"] in ids],
"categories": ann["categories"], "info": ann.get("info", {}), "licenses": ann.get("licenses", [])}
# 关键点标注也带上(pose 共用)
kp = next(pool.rglob("person_keypoints_val2017.json"), None)
if kp:
k = json.loads(kp.read_text())
sub["keypoints"] = {"annotations": [a for a in k["annotations"] if a["image_id"] in ids], "categories": k["categories"]}
img_dir = next(pool.rglob("val2017"))
items = [{"id": im["id"], "file": f"images/{im['file_name']}", "src": str((img_dir / im["file_name"]).relative_to(pool))} for im in sel]
cat_map = {str(i): c["id"] for i, c in enumerate(sorted(ann["categories"], key=lambda c: c["id"]))}
return items, {"annotations": sub, "cat_map": cat_map}
def build(ds: str) -> Path:
d = REG["datasets"][ds]
ver = d.get("version", "v1")
pool = POOL / ds / ver
pm = json.loads((pool / "manifest.json").read_text())
t1 = d.get("t1") or {}
n, seed = t1.get("n"), t1.get("seed", 20260914)
extra = {}
if ds == "imagenet1k":
items = items_imagenet(pool, n, seed, t1.get("stratify", "label"))
elif ds == "coco2017":
items, extra = items_coco(pool, n, seed)
else:
# 通用:pool 下全部文件(n=all)或随机 n 个;标注交给各任务评测器按 src 定位
files = sorted(str(p.relative_to(pool)) for p in pool.rglob("*") if p.is_file() and p.name != "manifest.json")
if n not in (None, "all", 0):
random.Random(seed).shuffle(files); files = sorted(files[:n])
items = [{"id": f, "file": f"files/{f}", "src": f} for f in files]
content_sha = hashlib.sha256(json.dumps([i["src"] for i in items]).encode() + pm["sha256"].encode()).hexdigest()
man = {"dataset": ds, "version": ver, "seed": seed, "n": len(items), "pool_sha256": pm["sha256"],
"sha256": content_sha, "items": items, **({"cat_map": extra["cat_map"]} if "cat_map" in extra else {})}
out = T1 / ds / f"{seed}.tar"
out.parent.mkdir(parents=True, exist_ok=True)
with tarfile.open(out, "w") as tar:
_add(tar, "manifest.json", json.dumps(man, ensure_ascii=False).encode())
if "annotations" in extra:
_add(tar, "annotations.json", json.dumps(extra["annotations"]).encode())
for it in items:
_add(tar, it["file"], (pool / it["src"]).read_bytes())
fsha = hashlib.sha256(out.read_bytes()).hexdigest()
(out.parent / "manifest.json").write_text(json.dumps({"dataset": ds, "seed": seed, "n": len(items), "tar": out.name,
"sha256": fsha, "content_sha256": content_sha,
"pool_sha256": pm["sha256"]}, indent=2))
print(f"{ds}: {out} n={len(items)} sha={fsha[:12]}")
return out
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--dataset", action="append", required=True)
a = ap.parse_args()
for ds in a.dataset:
build(ds)
return 0
if __name__ == "__main__":
sys.exit(main())

41
scripts/setup_agent.sh Normal file
View File

@ -0,0 +1,41 @@
#!/usr/bin/env bash
# 安装/校验 agent 运行时:node(~/toolchains/node)+ pi(@mariozechner/pi-coding-agent)。幂等。
# scripts/setup_agent.sh 安装缺的、打印版本
# scripts/setup_agent.sh --check 只检查
# node 用官方 tarball 解到 ~/toolchains/node(机器上没有 root 也没有 nvm);pi 用 npm -g,prefix 指到 ~/toolchains/node。
set -uo pipefail
NODE_DIR="$HOME/toolchains/node"
NODE_VER="${NODE_VER:-v22.12.0}"
CHECK=0; [ "${1:-}" = "--check" ] && CHECK=1
export PATH="$NODE_DIR/bin:$PATH"
echo "== node"
if ! command -v node >/dev/null 2>&1; then
[ $CHECK = 1 ] && { echo "node: 缺($NODE_DIR/bin/node)"; exit 1; }
mkdir -p "$HOME/toolchains"
URL="https://nodejs.org/dist/$NODE_VER/node-$NODE_VER-linux-x64.tar.xz"
echo "下载 $URL"
curl -fsSL "$URL" -o /tmp/node.tar.xz || { echo "node 下载失败"; exit 2; }
rm -rf "$NODE_DIR" && mkdir -p "$NODE_DIR" && tar -xJf /tmp/node.tar.xz -C "$NODE_DIR" --strip-components=1
fi
node --version; npm --version
echo "== pi(@mariozechner/pi-coding-agent)"
# 装错过 @mariozechner/pi(那是管 vLLM pod 的);清掉
if npm ls -g --depth=0 2>/dev/null | grep -q "@mariozechner/pi@"; then
echo "卸载装错的 @mariozechner/pi"; [ $CHECK = 1 ] || npm uninstall -g @mariozechner/pi >/dev/null 2>&1
fi
if ! command -v pi >/dev/null 2>&1 || ! pi --version >/dev/null 2>&1; then
[ $CHECK = 1 ] && { echo "pi: 缺"; exit 1; }
npm install -g @mariozechner/pi-coding-agent 2>&1 | tail -2
fi
echo "pi: $(command -v pi) version: $(pi --version 2>&1 | head -1)"
npm ls -g --depth=0 2>/dev/null | grep mariozechner
echo "== pi 包里的文档(extension API / CLI 参数核对用)"
PKG="$(npm root -g)/@mariozechner/pi-coding-agent"
ls "$PKG" 2>/dev/null | head; ls "$PKG/docs" 2>/dev/null | head -30
echo "== 写 config.yaml 的 pi_bin / node_bin 提示"
echo " paths.pi_bin: $(command -v pi)"
echo " paths.node_bin: $NODE_DIR/bin/node"

28
scripts/start_run.sh Executable file
View File

@ -0,0 +1,28 @@
#!/usr/bin/env bash
# 一键启动 / 续跑 oracle 全量:
# scripts/start_run.sh --note "P4 batch 1" [--parallel 8] 新 run:preflight → new-run → expand → 后台 run
# scripts/start_run.sh --resume [--parallel 8] 续跑最新 run 的 PENDING(调度进程崩了先 reset-stale)
# 之后:bench watch(实时看板)· bench status · bench status --fail · bench show <job_id> · bench report
set -euo pipefail
HERE="$(cd "$(dirname "$0")/.." && pwd)"; cd "$HERE"; export PATH="$HERE/bin:$PATH"
NOTE=""; PAR=""; RESUME=0
while [ $# -gt 0 ]; do case "$1" in
--note) NOTE="$2"; shift 2;; --parallel) PAR="--parallel $2"; shift 2;; --resume) RESUME=1; shift;;
*) echo "unknown arg $1"; exit 2;; esac; done
if [ "$RESUME" = 1 ]; then
RID=$(python3 -c "import sys; sys.path.insert(0,'.'); from harness import db
with db.connect() as c: print(db.latest_run_id(c) or '')")
[ -n "$RID" ] || { echo "没有 run 可续跑"; exit 1; }
bench reset-stale --run "$RID" || true
else
bench preflight || { echo "preflight 有 ERROR,先修"; exit 1; }
RID=$(bench new-run --note "${NOTE:-start_run $(date +%F)}")
bench expand --run "$RID"
fi
LOG="state/run-$RID.log"
setsid nohup bench run --run "$RID" $PAR >> "$LOG" 2>&1 < /dev/null & disown
echo "run $RID 已在后台启动(pid $!)"
echo " 实时看板: bench watch --run $RID"
echo " 调度日志: tail -f $LOG"
echo " 一眼汇总: bench status --run $RID ;失败清单 bench status --fail"
echo " 静态报告: bench report --run $RID → reports/$RID.html"

50
scripts/sync.sh Executable file
View File

@ -0,0 +1,50 @@
#!/usr/bin/env bash
# 本地 → 机器 的唯一同步入口。带"远端被人直接改过"的守卫,防止两个会话互相覆盖。
# scripts/sync.sh 推代码(harness/ agent/ export/ scripts/ bin/ registry/ config.yaml *.md),先查远端是否被动过
# scripts/sync.sh --pull 把远端的 registry/models.csv 等"在机器上生成的 registry 文件"拉回本地
# scripts/sync.sh --force 远端有未知改动也照推(会覆盖!只在你确认那些改动已经进本地时用)
# 规则(CONTRACT.md §12):只在本地改、git commit、sync;不在机器上直接改代码。远端改动会让下一次 sync 拒绝。
set -euo pipefail
HERE="$(cd "$(dirname "$0")/.." && pwd)"; cd "$HERE"
REMOTE="${BENCH_REMOTE:-ming01.wang@120.48.151.164}"
RDIR="${BENCH_REMOTE_DIR:-/mnt/data/ming01.wang/model-compile-hbdk/bench}"
STAMP=".sync/last_pushed.md5"
TRACK='-name *.py -o -name *.ts -o -name *.md -o -name *.yaml -o -name *.sh -o -name *.tmpl -o -name *.csv -o -name bench'
PRUNE='-path ./runs -o -path ./data -o -path ./cache -o -path ./state -o -path ./datasets -o -path ./reports -o -name __pycache__ -o -path ./.sync -o -path ./.git'
FORCE=0; PULL=0
for a in "$@"; do case "$a" in --force) FORCE=1;; --pull) PULL=1;; *) echo "unknown $a"; exit 2;; esac; done
manifest_local() { find . \( $PRUNE \) -prune -o -type f \( $TRACK \) -print | sort | xargs md5sum; }
manifest_remote() { ssh -o BatchMode=yes -o ConnectTimeout=10 "$REMOTE" "cd $RDIR && find . \\( $PRUNE \\) -prune -o -type f \\( $TRACK \\) -print | sort | xargs md5sum"; }
if [ $PULL = 1 ]; then
rsync -az "$REMOTE:$RDIR/registry/models.csv" registry/models.csv
echo "pulled registry/models.csv"; exit 0
fi
mkdir -p .sync
if [ -f "$STAMP" ] && [ $FORCE = 0 ]; then
echo "检查远端是否被直接改过…"
manifest_remote > .sync/remote_now.md5
# 远端现在 vs 上次推送时:不同 = 有人在机器上直接改了(或另一个会话推过)
if ! diff -q <(sort "$STAMP") <(sort .sync/remote_now.md5) >/dev/null; then
echo "!! 远端自上次 sync 后有改动,拒绝推送。差异:"
diff <(sort "$STAMP") <(sort .sync/remote_now.md5) | grep '^[<>]' | awk '{print $1, $3}' | sort -u -k2 | head -30
echo "先把这些改动合进本地(scp 回来 / 对比),或者确认可以覆盖后 --force。"
exit 3
fi
fi
if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
if ! git diff --quiet || ! git diff --cached --quiet || [ -n "$(git ls-files --others --exclude-standard)" ]; then
git add -A && git commit -qm "sync $(date +%F_%T) $(whoami)@$(hostname -s)" && echo "committed $(git rev-parse --short HEAD)"
fi
fi
find . -name __pycache__ -type d -prune -exec rm -rf {} + 2>/dev/null || true
rsync -az --delete-excluded --exclude '__pycache__' --exclude '.sync' --exclude '.git' \
harness agent export scripts bin registry/matrix.yaml registry/gates.yaml registry/budgets.yaml registry/datasets.yaml registry/boards.yaml \
config.yaml CONTRACT.md README.md requirements.txt "$REMOTE:$RDIR/" 2>&1 | tail -1
# registry/models.csv 是机器上生成的(excel_to_models),只在 --pull 方向走
ssh -o BatchMode=yes "$REMOTE" "cd $RDIR && chmod +x bin/bench scripts/*.sh agent/*.sh 2>/dev/null; find . -name __pycache__ -type d -prune -exec rm -rf {} + 2>/dev/null; true"
manifest_remote > "$STAMP"
echo "synced → $REMOTE:$RDIR ($(wc -l < "$STAMP") tracked files; stamp saved)"

119
scripts/test_path_guard.py Normal file
View File

@ -0,0 +1,119 @@
"""path-guard 规则的 Python 平行实现 + 用例。改 agent/extensions/path-guard.ts 的正则时同步改这里并跑一遍。
python3 scripts/test_path_guard.py
为什么需要:guard 只在真跑 rollout 时才生效,一条轨迹几十分钟,靠它试错太慢。
第一条真实轨迹里 10 次工具调用被误杀 4 次(`2>/dev/null` 被当成写绝对路径、`grep hb_compile` 被当成调用 hb_compile),
这些都是本文件里的用例。
"""
from __future__ import annotations
import re
import sys
FORBIDDEN = ["hb_compile", "hb_mapper", "hb_verifier", "hrt_model_exec", "hb_perf",
"ssh", "scp", "rsync", "wget", "curl", "pip", "pip3", "conda", "npm"]
CMD_POSITION = re.compile(r"(?:^|[;&|]\s*|\$\(|`)\s*(?:[A-Za-z_][A-Za-z0-9_]*=\S*\s+)*"
r"(?:sudo\s+|env\s+|nohup\s+|time\s+)*(" + "|".join(FORBIDDEN) + r")\b")
REDIRECT = re.compile(r"(?:^|[\s;|&(])(?:\d?>>?|&>)\s*['\"]?([^\s'\"|;&)]+)")
REDIRECT_OK = {"/dev/null", "/dev/stderr", "/dev/stdout"}
WRITE_PATTERNS = [re.compile(r"\bsleep\s+[0-9]{3,}"), re.compile(r"\bgit\s+push\b")]
DEST_LAST = {"cp", "mv", "ln", "install", "rsync"} # 最后一个参数才是目的地
ALL_ARGS = {"rm", "rmdir", "tee", "touch", "mkdir", "chmod", "chown", "truncate", "dd", "shred"}
PREFIX = {"sudo", "env", "nohup", "time"}
def _outside(t: str) -> bool:
"""近似 TS 里的 !inside(path.resolve(WORKSPACE, t)):绝对路径、或 ../ 跳出 workspace。"""
if t.startswith("/tmp/") or t == "/dev/null":
return False
if t.startswith("/") or t.startswith("~"):
return True
depth = 0
for part in t.split("/"):
if part == "..":
depth -= 1
elif part not in ("", "."):
depth += 1
if depth < 0:
return True
return False
def _write_targets(seg: str) -> list[str]:
toks = seg.split()
i = 0
while i < len(toks) and re.match(r"^[A-Za-z_][A-Za-z0-9_]*=", toks[i]):
i += 1
while i < len(toks) and toks[i] in PREFIX:
i += 1
cmd = toks[i].rsplit("/", 1)[-1] if i < len(toks) else ""
args = [t.strip("'\"") for t in toks[i + 1:] if not t.startswith("-")]
if cmd in DEST_LAST:
return args[-1:] if args else []
if cmd in ALL_ARGS:
return args
return []
def blocked(cmd: str) -> str | None:
for m in REDIRECT.finditer(cmd):
t = m.group(1)
if t.startswith("&") or t in REDIRECT_OK or t.startswith("/proc/self/fd/"):
continue
if _outside(t):
return f"redirect:{t}"
m = CMD_POSITION.search(cmd)
if m:
return f"cmd:{m.group(1)}"
for seg in re.split(r";|\|\||&&|\||&", cmd):
for t in _write_targets(seg):
if _outside(t):
return f"write:{t}"
for rx in WRITE_PATTERNS:
if rx.search(cmd):
return f"rule:{rx.pattern[:20]}"
return None
CASES: list[tuple[str, bool]] = [
# (命令, 是否应当被拒)
("ls -la .. && cat ../rollout.json", False),
("ls ../stages/04-calibrate && cat ../stages/04-calibrate/a00/meta.json 2>/dev/null", False),
("ps aux | grep -iE 'hb_compile|hbm|hmct' | grep -v grep | head", False), # 只读排查,不是调用
("grep -rn hb_compile ../stages/05-compile/a00/stderr.log | tail -5", False),
("bench stage run compile --config workspace/compile.yaml", False),
("bench stage wait 05-compile-050600-5a2e6d --poll 120 2>&1 | tail -30", False),
("python3 -c 'import yaml;print(1)' 2>/dev/null", False),
("cat ../job.json | python3 -m json.tool | head -40", False),
("echo 'model_parameters: {}' > compile.yaml", False), # 写 workspace 内,允许
("hb_compile -c compile.yaml", True), # 直接调工具链
("cd /tmp && hb_compile -c x.yaml", True),
("OMP_NUM_THREADS=8 hb_compile -c x.yaml", True),
("nohup hrt_model_exec perf --model_file x.hbm &", True),
("ssh root@board 'ls /userdata'", True),
("pip install onnxsim", True),
("echo hack >> /etc/passwd", True),
("cp ../job.json /tmp/x", False), # /tmp 放行
("cp ../job.json .", False), # 把只读文件读进 workspace
("cp ../stages/04-calibrate/compile.yaml my.yaml", False),
("cp workspace.yaml ../../oracle/artifacts/x.yaml", True), # 往 job 目录外写
("rm -rf ../oracle", True),
("mkdir -p artifacts && touch artifacts/x", False),
("sleep 3600", True), # 用 bench stage wait
]
def main() -> int:
bad = 0
for cmd, want in CASES:
got = blocked(cmd)
ok = bool(got) == want
bad += not ok
print(f"{'ok ' if ok else 'FAIL'} block={str(bool(got)):5} want={str(want):5} {got or '':<22} {cmd}")
print(f"\n{len(CASES) - bad}/{len(CASES)} passed")
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())

77
scripts/trace.py Normal file
View File

@ -0,0 +1,77 @@
"""看一条 rollout 轨迹在干什么:工具调用序列 + 结果摘要 + claim/verdict。
python3 scripts/trace.py runs/<run>/<job>/rollouts/00 [--full]
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
def text_of(content) -> str:
if isinstance(content, str):
return content
out = []
for c in content or []:
if isinstance(c, dict) and c.get("type") == "text":
out.append(c.get("text", ""))
return "".join(out)
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("rollout_dir")
ap.add_argument("--full", action="store_true")
ap.add_argument("--width", type=int, default=220)
a = ap.parse_args()
rd = Path(a.rollout_dir)
ev = rd / "agent" / "events.jsonl"
if not ev.exists():
print("no events.jsonl"); return 1
n = turns = 0
w = a.width if not a.full else 10_000
for line in ev.open(errors="replace"):
try:
e = json.loads(line)
except json.JSONDecodeError:
continue
t = e.get("type")
if t == "turn_end":
turns += 1
msg = e.get("message") or {}
txt = text_of(msg.get("content"))
if txt.strip():
print(f"--- turn {turns} assistant: {txt.strip()[:w]}")
elif t == "tool_execution_start":
n += 1
args = e.get("args") or {}
short = args.get("command") or args.get("path") or json.dumps(args, ensure_ascii=False)
print(f"[{n}] {e.get('toolName')}: {str(short)[:w]}")
elif t == "tool_execution_end":
r = e.get("result") or {}
txt = text_of(r.get("content") if isinstance(r, dict) else r)
flag = "ERR " if e.get("isError") else ""
print(f" -> {flag}{txt.strip()[:w].replace(chr(10), ' | ')}")
elif t == "agent_end":
print(f"=== agent_end: {len(e.get('messages', []))} messages, {turns} turns, {n} tool calls")
elif t in ("auto_retry_start", "auto_retry_end", "compaction_start"):
print(f"!!! {t}: {json.dumps(e, ensure_ascii=False)[:w]}")
for f in ("claim.json", "verdict.json", "rollout.json"):
p = rd / f
if p.exists():
d = json.loads(p.read_text())
keep = {k: d.get(k) for k in ("terminal", "state", "error_class", "summary", "failed_gate", "claim_consistent", "review", "exit_code", "wall_s", "timeout_reason") if k in d}
print(f"{f}: {json.dumps(keep, ensure_ascii=False)}")
g = rd / "agent" / "guard.log"
if g.exists():
lines = g.read_text().splitlines()
print(f"guard.log: {len(lines)} 次越界拦截")
for l in lines[-3:]:
print(" ", l[:w])
return 0
if __name__ == "__main__":
sys.exit(main())