pre-grind snapshot 2026-09-19: batches 1-7 adapters/probes/recipes + eval additive changes

This commit is contained in:
bench 2026-09-19 17:52:03 +08:00
parent 5b1284ec8c
commit d309109f2b
240 changed files with 23005 additions and 564 deletions

91
CONTINUE.md Normal file
View File

@ -0,0 +1,91 @@
# 剩余任务切到 100 张
旧批次保存已经发生的执行,新批次只接剩下的任务。流程是:**生成独立评测包 → 预览清单 → 停止领取并收尾 → 更新清单并应用 → 单条试跑 → 恢复并发**。
普通流水线和 Agent 仍使用原来的 `bench run`、`bench collect`。不全量 `expand`,不使用 `collect --coverage all`。注册表默认值和校准集不改;500 张复核按需另做,这里不加入自动升级机制。
## 1. 先准备,旧批次继续运行
在有原始数据池的机器上生成固定子集。ImageNet 随机选类,每类取一张;COCO 固定随机抽图。
```bash
python scripts/make_t1.py --dataset imagenet1k --dataset coco2017 --n 100 --seed 20260914
```
新包在 `data/t1/<dataset>/n100-s20260914/`,原包保留。同一份清单用于 FP32 和板端 INT8;新缓存按内容哈希和评测器版本分开存放。
下例的 `OLD_RUN`、`NEW_RUN` 和 `JOB_ID` 都替换成清单里的实际值。下面的 `plan` 只读数据库和文件,不领取任务、不创建 run、不停止进程:
```bash
bench continue plan --run OLD_RUN \
--t1 data/t1/imagenet1k/n100-s20260914/manifest.json \
--t1 data/t1/coco2017/n100-s20260914/manifest.json > preview.json
```
只提供剩余任务涉及的数据集;缺少某个数据集的替换包会报错。默认生成新的 run ID,也可用 `--to NEW_RUN` 指定。清单含新旧任务、原采样序号、模式、策略、评测包身份及阻塞原因。服务器仍在执行时出现 `blockers` 是正常的,最终清单在收尾后重新生成。
> `bench reset-stale` 必须跑在 `plan` **之前**,不能夹在 `plan` 和 `apply` 中间:plan 把源 run 的 stages 全行打进 `source_sha256`,reset-stale 改任何一行 stage 状态都会让已生成的清单在 apply 时被拒(fail-closed,报 `source changed`),只能重新 plan。
`counts` 分四项:
| 项目 | 含义 |
|---|---|
| `oracle` | 从未开始的普通任务 |
| `agent_queued` | 已经排队、从未开始的 Agent 采样 |
| `agent_due_repairs` | 新清单固定为 0,不再按 Oracle 失败补任务 |
| `conditional_repairs` | 新清单固定为 0;旧清单保留,但不再激活 |
成功和失败都消耗原采样次数。计划 4 条、完成 1 条就只接余下 3 条,并保留它们的原 `kk`,不从零重新编号。首轮优先顺序及 `collect --first-pass` 也计入旧批次的已启动、已完成记录。
## 2. 工具验证完成后,停止领取并完成交接
```bash
bench drain --run OLD_RUN --all
bench status --run OLD_RUN
```
`--all` 同时设置普通流水线和 Agent 的停止领取标记。等已经领取的整个任务及核验结束、两个调度进程退出;不等全部待办跑完。运行中的旧进程必须已支持 drain,不能靠同步新代码改变它的内存逻辑。
收尾后,用第 1 步的 `plan` 命令重新输出 `handoff.json`,核对清单,然后:
```bash
bench continue apply handoff.json
bench continue check --run NEW_RUN
```
应用时再次校验评测包、源任务状态及源注册表指纹。过期清单、还在运行、曾执行过又重置成待办的记录都会阻止交接。成功后源待办记为 `TRANSFERRED`,源 run 不再允许启动或重新铺队列;完成的结果、日志和轨迹保留原处。重复应用同一清单只核对结果,不重复创建任务。
工具只处理一次从原始批次到接续批次的交接;不提供连续多级迁移框架。数据库事务提交前的文件准备失败,可用同一清单重试。目标目录保存 `continuation.json` 作为交接回执。
## 3. 先试一条,再恢复并发
如果选中的任务还没跑普通流水线,先执行它;如果只剩 Agent,跳过这一条命令:
```bash
bench run --run NEW_RUN --job JOB_ID --parallel 1
bench collect --run NEW_RUN --job JOB_ID --parallel 1 --limit 1
bench export --run NEW_RUN
bench continue check --run NEW_RUN
```
确认 FP32 和 INT8 的包哈希一致、实际评测 `n=100`、结果及 Agent 训练记录包含 `eval_set`,再按原资源配置恢复:
```bash
bench run --run NEW_RUN
bench collect --run NEW_RUN --watch
```
两条恢复命令分别在现有终端/进程管理方式下启动;不新增守护服务。`check` 比较具体任务及采样序号,不只比较总数。试跑后待办数会减少,但“待办 + 执行中 + 已结束”仍与交接数量对应。历史 QUEUED repair 保留编号但不再领取,须与可执行的 e2e 待办分开统计。
已有任务快照中的 `oracle_ref` 保留历史身份;新执行与判定不读取旧 Oracle。Agent 按需调用已有输入准备阶段,缓存按评测集区分,不重跑整条普通流水线。新交接只保留实际存在的采样行,不补 Oracle 失败驱动的 repair。
100 张保留现有 `pass-screen` 筛查口径和原精度门槛。奖励 v0.6 只使用本次实测,不再与 Oracle 比较。训练记录保留原 run、job、采样序号和评测集身份,训练端按 `eval_set` 区分可比较的组。
## 本地验证范围
单元与集成测试使用临时 SQLite、合成评测包和模拟执行器,验证抽样、任务守恒、交接恢复、调度、缓存命令及导出。它们不能替代真实 100 张图片上的 FP32/INT8 试跑,也不能替代 Linux 板端和 Agent 沙箱验证。
## 已知未修问题(从 PLAN.md §7 迁入)
- **events.jsonl 在爆**:convnext.small #0 才 7 轮就 6.0 MB,其中 5.53 MB 是 1548 条 `message_update`(流式 token 增量)。落盘时过滤掉,或跑完压成 `.raw.zst` 只留结构事件。124 条 rollout 就是几个 GB,而 audit / 导出每次都要全文扫。
- **heartbeat 健康时零输出**,无法区分"都健康"和"它什么都没看到"。健康时也打一行。

View File

@ -4,9 +4,19 @@
2026-09-16 按当前目标收敛:默认运行 CSV 中 ready 模型的 S100P(`nash-m`)量化、编译、上板与精度评测,并为这些模型采集真实 Agent 轨迹。`collect --coverage all` 为本 run 的 oracle 模型补齐 e2e 计划:只去掉 matrix 的 priority 限制,保留任务类型、march、backend、policy 和 k;默认 `matrix` 仍使用原选择范围。已结束的流程和 Agent 轨迹都必须导出为数据,成功、精度未达标、预算耗尽分别标注,导出不能阻塞上板。
2026-09-16 全 338 项覆盖补充:保持原 CSV 标签和历史 run。新批次可用 `expand --model` 显式选择模型,
前置检查依据实际权重入口、构建/导出代码、评测器、校准、数据与附加门;新增 Ultralytics 任务还必须有真实
checkpoint/ONNX/指标对照证据,缺项即拒绝入队。具名批次只放宽 priority/task/status 选择,保留原 march、
backend、policy、k 和精度门。任务快照记录 admission 来源,不能把 admission 或 probe 等同于编译/上板通过。
`collect --first-pass` 每个 job 只领取首条,重复采样仍保持 QUEUED,不改 kk、不补造轨迹。
`coverage --all-runs` 总表包含全部 CSV 行,独立列出前置阻塞、实际阶段、原始响应中的 teacher 身份和训练导出;
传统算法仍保留条目并说明 PTQ 不适用。FastSAM 的类别语义未补齐前不能入队。
Pose 须有真实导出的关键点对应证据;板端同时判定 OKS AP 与平均原图坐标偏移 ≤2px。
对应由同图人框 IoU≥0.5 一对一匹配确定,统计全部17点,保留未匹配记录;无匹配或缺证据为 UNVERIFIED。
## 0. 三条原则
1. **harness 是 oracle。** 每个 job 先由确定性流水线跑出 ground truth(`oracle/result.json`)。agent 的表现永远相对 oracle 衡量。
1. **Agent 独立完成任务。** GLM-5.3 可直接领取 e2e,自行调用输入准备、量化、编译和上板工具;`bench run` 可选地预热 01–03 缓存。harness 分配资源、核验本次 HBM 的实测精度并保存轨迹。完整 Oracle 仅用显式 `bench run --oracle` 作历史对照或排查,不参与 Agent 执行、成功判定、奖励或 SFT 筛选。
2. **agent 只有一个可写区。** `rollouts/<kk>/workspace/`。`job.json`、`harness/eval/`、`data/`、`cache/` 全部只读。agent 说什么不算数,verdict 才算数。
3. **Append-only。** `run_id` 冻结;换工具链、改配置、修 bug 都开新 run。`manifest.json` 记两套 venv 的 `pip freeze` sha256,job 启动前比对,不一致拒跑。状态只在 `state/jobs.sqlite`,文件系统只存内容。
@ -58,7 +68,8 @@ bench/
## 3. 状态机
**job(由 oracle 决定)**:`PENDING → ORACLE_RUNNING → PASS | FAIL | UNVERIFIED`
**job 输入准备**:`PENDING → ORACLE_RUNNING → PREPARED | FAIL | UNVERIFIED`(沿用现有运行态名称)。
`PREPARED` 只表示 01–03 成功,写 `oracle/preparation.json`,不产生 Oracle verdict,也不代表适配或精度通过。显式 `--oracle` 可从 PENDING/PREPARED 跑完整 01–08,写 `oracle/result.json`,终态仍为 PASS/FAIL/UNVERIFIED。默认准备命令不覆盖已结束任务。
**rollout(由实际核验结果决定)**:`QUEUED → RUNNING → VERIFYING → PASS | FAIL | BLOCKED | UNVERIFIED`
Agent 执行是否结束/超预算独立记录在 `verdict.execution_terminal`、`timeout_reason`,不再用 `TIMEOUT` 覆盖已经核验的结果。`completion` 分别记录编译、HBM、真实上板、精度评测是否完成及精度是否达标。预算用尽但必选门已通过仍为 PASS;没做到的阶段为未完成,不凭预算耗尽推断模型失败。旧 TIMEOUT 可用 `bench verify` 按原 job 门槛重核,无需重编。
@ -66,7 +77,7 @@ Agent 执行是否结束/超预算独立记录在 `verdict.execution_terminal`
- PASS = 必选门全部 pass。
- 门无法判定(数据缺、板未接)→ 该门 `unverified`,job `UNVERIFIED`。
- `na`(本轮设计上不评)≠ `unverified`(本该评没评成)。na 不阻塞 PASS。
- BLOCKED 判定:oracle 同阶段同类 error_class 失败 → 正确;oracle 通过 → 误判。
- BLOCKED 记录 Agent 的阻塞报告和本次实测。失败日志不能证明任务不可解,`blocked_correct` 不作推断;不进入成功 SFT 或奖励学习候选。
## 4. 四道门
@ -97,9 +108,10 @@ Agent 执行是否结束/超预算独立记录在 `verdict.execution_terminal`
- 板卡登记使用固定且唯一的 `slot`,对应 `board-<slot>.lock`;现有板保留 slot 0 以兼容在途旧进程。新增板使用新槽位,删除/禁用板卡后不复用槽位;重复接入同一地址不能增加容量。只把实际独立、运行库兼容的物理板加入池。新增/禁用板卡是调度资源变更,不重写冻结的 job 或已生成 HBM。新调度进程等待时重读板卡登记;旧进程仍需自然收尾后更换。
- 新 run 的任务指纹不包含板卡设备列表,但仍包含 march 映射与其他 registry 内容。旧 run 扩板前把原始 `boards.yaml` 保存为 `board_registry.initial.yaml`;续跑时用该快照验证旧整体指纹,证明除板卡设备列表外的输入未变,不能跳过模型、数据、门槛、工具链校验。
- **没有固定超时。** STALLED = 日志 `stall_s`(900 s)不增长 且 进程树 CPU < 5%。硬上限 24 h 只是保险。
- 板端 RPC 的原有单次 3600 秒上限由独立 `board_guard` 在板上执行;SSH stdin 租约每 5 秒刷新,断连或 30 秒失联回收本次进程组。守护进程不加载模型,避免原生运行库阻塞清理。板端互斥锁由守护进程和推理子进程共同持有,未清空不能交给新推理。租约存活日志与样本/迭代进度分开记录,不能把心跳当作推理进展。主机阶段锁只代表调度占用,不能代替板端进程核验。
- 重阶段(compile / board / eval)由 harness 执行。agent 走 `bench stage run` / `bench stage wait`,不在 bash 里跑 hb_compile。
- rollout 预算 = turns + tokens;墙钟 12 h 仅防挂死。
- Agent 动态领取任务:先覆盖尚无完整轨迹的不同模型,同等覆盖下优先 oracle PASS 和预计耗时较短的模型,再做重复采样;同一 job 同时最多一条 Agent 轨迹。每次有空位都重新读取队列,`collect --watch` 跟随尚未结束的 oracle,避免新完成的模型一直排在旧批次后面。
- Agent 动态领取任务:先覆盖尚无完整轨迹的不同模型,同等覆盖下优先预计耗时较短的模型,再做重复采样;同一 job 同时最多一条 Agent 轨迹。每次有空位都重新读取队列,`collect --watch` 等待输入准备和在途任务,避免新完成的模型一直排在旧批次后面。
- 同一 run 只运行一个新 collector;任务领取原子化。SIGTERM 或 `state/drain-collect-<run>` 停止领取,已开始的任务自然收尾。导出在独立线程刷新,不占 Agent 名额。
## 7. 重试与轨迹边界
@ -109,7 +121,7 @@ Agent 执行是否结束/超预算独立记录在 `verdict.execution_terminal`
| oracle 阶段 | harness | 瞬态 error_class | 不是轨迹 |
| rollout 内 | agent | 自己改配置重提 | 同一条轨迹 |
| 新 rollout | harness | matrix k | 全新轨迹,永不串联 |
| repair | harness | oracle FAIL | 全新轨迹,输入是 oracle 现场 |
| 历史 repair | 不再生成或领取 | 保留既有编号和结果 | 有 Oracle 现场的修复案例,仅保留分析用途 |
显式扩大已有 run 的 Agent 覆盖时,仅追加缺少的计划和范围说明 annotation;编号接在该 job 的最大 kk 之后,重复执行不重复添加。冻结的 job、已执行轨迹、oracle 结果及评测门槛不变。确定性 oracle 的 PASS 不代表该模型已有 Agent 训练轨迹。
@ -126,19 +138,17 @@ bash 里 `cd/pushd/popd` 一律拒(唯一例外 `cd <自己的 workspace>`,
## 8. error_class
`unsupported_op shape_infer export_fail calib_nan verifier_mismatch`(确定性,不重试)
`oom board_offline stalled`(瞬态,重试 ≤ 2 / ≤ 2 / 1)
`oom board_offline board_busy stalled`(瞬态,前三者重试 ≤ 2,stalled 重试 ≤ 1)
`unknown`(进人工分类队列)
## 9. reward(v0.4,离线算,配方版本化)
## 9. reward(v0.6,离线算,配方版本化)
```
r = 1.0[G1] + 1.0[G2] + 2.0[G3]
r += 1.0·clip(accuracy_improvement/|tol|, −1, 1)
r += 0.5·clip((lat_oracle − lat_agent)/lat_oracle, −1, 1)
r −= 0.5·turns/max_turns + 0.5·tokens/max_tokens
r −= 2.0[claim ≠ verdict]
BLOCKED: +1.0 若 oracle 同类失败 else −2.0 ;TIMEOUT: −1.0
G3 = na 时其 2.0 权重按比例分给 G1/G2,精度相对项取 0
TIMEOUT: −1.0;BLOCKED 无特殊奖惩,仍保存本次真实阶段得分
G3 = na 时其 2.0 权重按比例分给 G1/G2
板卡不可用(G3 unverified 且最后一次 07-board/08-eval 为 board_offline):G2/G3 权重取 0,G1 保持 1.0 不加分,
免 TIMEOUT 罚分,reward 输出带 unverifiable="board_offline"(v0.4,2026-09-16)
```
@ -146,7 +156,7 @@ G3 = na 时其 2.0 权重按比例分给 G1/G2,精度相对项取 0
板卡不可用的轨迹在导出器里仍按 UNVERIFIED 排除在 RL/SFT 候选之外;回收它们的办法是 `bench board-backfill --run R --rollout JOB:KK`
用其现成 HBM 补跑 07/08 再 verify,而不是改分。
`abs_pp` 的 accuracy_improvement = Δ_agent − Δ_oracle;`rel_pct` 的 Δ 表示劣化,故取 Δ_oracle − Δ_agent。不同 kind 不互比。
不再计算相对 Oracle 的精度或延时奖励。G3 仍比较同一评测集上的 FP32 与本次 INT8,沿用任务冻结的门槛。
TIMEOUT 扣分根据执行终止原因独立相加,已完成的 G1/G2/G3 成果照常得分,不再整条清零。
## 10. 两套 venv
@ -164,13 +174,15 @@ harness 按 backend 选解释器。**禁止把 backend 判断交给 agent。**
数据默认按 `purpose=all` 保留已结束的成功、失败及部分轨迹;SFT 是其中的用途筛选,不能代表全部数据量。Agent 的 `teacher.jsonl` / `student.jsonl` 只含真实对话、工具调用和观测,缺失内容不得编造。`pipeline.jsonl` 保留确定性流程的实际输入、阶段动作与测量,明确标记其来源,不能计成模型生成的 Agent 轨迹。每个 run 使用同一导出目录,显式刷新,按稳定 ID 去重;导出失败报错并可补导,不回滚已完成任务。
`bench.verdict/3` 不读取 Oracle;旧判定在导出时只读重核,源 verdict、数据库、消息和产物不改。SFT 候选要求本次实测 PASS、声明一致、完整对话和工具定义,并保留污染注释与版本过滤。Oracle FAIL 不再触发待复核;历史 repair 保留原模式,仅供分析。导出是候选数据,仍需训练端分组、去重和独立评测。
## 12. 协作规则(2026-09-15 起,两个会话同时改一棵树踩过一次)
- **只有一个写者能改一层。** 分层:`harness/`(oracle 流水线、阶段、门、调度)· `agent/`(pi 配置、扩展、提示词)· `export/`(清洗、脱敏、reward)· `registry/`(清单与阈值,改了就是新 run)· `scripts/`。谁在改哪层写进下面的表;别的会话对那层只读。
- **改动只发生在本地副本,机器上不直接改代码。** 流程:改 → `git commit`(`scripts/sync.sh` 会自动提交)→ `scripts/sync.sh` 推送。sync 带守卫:远端自上次推送后被动过就拒绝,先合并再推。
- `registry/models.csv` 是在机器上由 `excel_to_models.py` 生成的,方向反过来:`scripts/sync.sh --pull`。
- 机器上的 `runs/ state/ data/ cache/ datasets/` 是数据,不同步、不手改;改判走 `bench rejudge`,补测走 `bench board-backfill`。
- 正在跑的调度进程(`bench run` / `bench collect`)加载的是启动时的代码;改了 harness 只对新进程生效,子进程工具(`harness.tools.*`、`harness.eval.run`)立刻生效——改评测口径必须同时升 `EVALUATOR_VERSION`。
- 正在跑的调度进程(`bench run` / `bench collect`)加载的是启动时的代码;改了 harness 只对新进程生效,子进程工具(`harness.tools.*`、`harness.eval.run`)立刻生效——改评测口径必须同时升 `EVALUATOR_VERSION`。**换代必须 `bench drain` 等它跑完再起新进程,只 sync 文件不算换代**(踩过:一个调度进程跑了 13 小时的旧代码)。
| 层 | 当前写者 |
|---|---|

View File

@ -5,20 +5,13 @@
## 一、它在做什么(一段话)
把 338 个视觉模型逐个量化编译到 BPU(nash-e/m/p),每个"模型×芯片×配置"叫一个 **job**。每个 job 先由 **harness** 自己跑一遍标准流程(叫 **oracle**,产出"标准答案"和四道门的判定),然后让 **agent**(一个调大模型的 pi 进程)在看不到答案的情况下**自己做一遍**,harness 独立核验它做得对不对。agent 做一遍的过程记录下来就是一条 **rollout(轨迹)**,带着 harness 给的可核验分数,这就是训练数据。
把 338 个视觉模型逐个量化编译到 BPU,每个“模型×芯片×配置”叫一个 **job**。GLM-5.3 Agent 自行准备输入、量化、编译和上板;harness 管理资源并用本次真实产物核验。一次执行的对话、工具调用和观测是一条 **rollout(轨迹)**。
```
registry/models.csv ──▶ bench new-run/expand ──▶ jobs(PENDING)
│
bench run(后台调度) ──▶ oracle:01 fetch → 02 baseline → 03 export → 04 calibrate → 05 compile → 06 hbm → 07 board → 08 eval
│ 四道门 G1 编译 / G2 余弦 / G3 任务指标 / G4 上板延时
▼
bench collect(后台采集) ──▶ 每个 job k 条 rollout:pi 进程在 <job>/rollouts/<kk>/workspace 里作答
│ → verify_rollout 独立核验 → verdict.json(reward 的唯一来源)
▼
scripts/audit_rollouts.py ──▶ 哪些轨迹干净、能进 RL/SFT
export/to_agentrl.py ──▶ 训练格式
```
流程:**领取任务 → Agent 执行 → 本次实测判定 → 导出并筛选训练候选**。
`bench run` 可以预热权重、FP32 基线和 ONNX 缓存。完整 Oracle 仅供显式对照,不是任务前置,也不参与成功判定、奖励或 SFT 筛选。FP32 仍是 INT8 精度的比较基线,两者使用同一评测集。
旧 Oracle、旧 repair 和已有结果保留。历史 repair 已获得失败现场,适合分析修复过程,不能当成独立完成任务的示范;尚未开始的旧 repair 保留编号但不再领取。
## 二、你能控制的东西(改这些文件,其他都是派生的)
@ -48,7 +41,7 @@ python3 scripts/trace.py <rollout_dir> 一条轨迹逐轮回放:a
bench report --run <run> 静态 HTML 报告(reports/<run>.html)
```
启动/续跑:`scripts/start_run.sh --note "..."`(oracle)、`bench collect --run <run> [--job …] [--limit N]`(采轨迹)。
启动/续跑:`scripts/start_run.sh --note "..."`(准备与采集)、`bench collect --run <run> [--job …] [--limit N]`(采轨迹)。
换代码/改配置要重启调度器时:`bench drain --run <run>`(跑完手里的不再认领新 job)→ 等它退出 → `scripts/start_run.sh --resume`。不要 kill 正在编译的进程。
注意常驻进程里的代码是启动那一刻的:改了 harness/ 或 config.yaml,正在跑的 `bench run` / `bench collect` 不会变,新起的才会。
两条自检:`node agent/extensions/guard-core.test.js`(边界判定单测)、`scripts/guard_smoke.sh`(起真 pi 进程试图越界,应全部被拒)。
@ -67,10 +60,10 @@ bench report --run <run> 静态 HTML 报告(reports/<run>.html
| `old_version(vX<Y)` | 用旧一版的提示词/边界跑的 | 导出默认过滤,留档不删 |
| `claim_absent` / `claim_mismatch` | 没交 claim / 自述与核验不一致 | 可以当负样本(reward 已扣),SFT 不收 |
| `harness/toolchain_read×N` | 去读评判代码或工具链源码 | 不算污染,但说明提示词没把它劝住,预算会被这个烧掉 |
| 干净 + `RL ✓` | 版本合格、没越界、没被误杀 | 进 GRPO 组 |
| `SFT ✓` | RL ✓ 且 verdict=PASS 且 claim 一致 | 进 SFT |
| 干净 + `RL ✓` | 初步审计合格 | 仍需核验结果与训练分组,不代表已可直接训练 |
| `SFT ✓` | 独立执行、实测 PASS、声明一致、对话和工具记录完整 | 成功 SFT 候选 |
想亲手看一条:`trace.py` 回放里盯这几个词——`oracle`、`rollouts/0X`(不是自己的编号)、`harness/`、`site-packages`、`path-guard:`(被拒的那一下)、`claim`(最后有没有交)、`bench stage run`(真正干活的提交)。一条健康的 e2e 轨迹长这样:读 job.json → 写 compile.yaml → `stage run calibrate/compile` → `stage wait` → 看日志改配置(≤2 轮) → `stage run hbm` → `stage run eval` → `claim`。前 3 步之外还在 `ls`/`cat` 各种目录的,就是在浪费预算。
想亲手看一条:`trace.py` 回放里盯这几个词——`oracle`、`rollouts/0X`(不是自己的编号)、`harness/`、`site-packages`、`path-guard:`(被拒的那一下)、`claim`(最后有没有交)、`bench stage run`(真正干活的提交)。常见 e2e 流程是:读 job.json → 按需准备输入 → 写配置 → 编译并检查日志 → 检查 HBM → 上板评测 → claim。具体步骤由 Agent 根据观测决定。
## 五、失败的轨迹要删吗
@ -105,4 +98,4 @@ v5 已把阶段执行移到 jail 外部 worker,state/其他 job 与评判代
命令客户端仅写 handles,pi 自身输出写 workspace 后由外部驱动收存。旧常驻 collect 不会热加载新驱动,后续采集需新进程。
导出默认最低版本5并读取污染/误杀 annotations,旧9份数据集已新增 VOID 标记。
`bench heartbeat --run <run>` 正常无输出退出0,异常退出1;`bench watch` 非TTY仅输出一次。
完整实测、审计前后对比和未覆盖项见 [REPORT-codex.md](REPORT-codex.md)。本节取代上面 v4 的“当前”状态描述。
完整实测、审计前后对比和未覆盖项见 [REPORT-codex.md](docs/archive/REPORT-codex.md)。本节取代上面 v4 的“当前”状态描述。

View File

@ -1,8 +1,41 @@
# bench/ — 视觉模型量化与上板评测
设计:`../BPU量化轨迹工厂-设计方案.html`(v0.4)。契约:`CONTRACT.md`。本 README 只讲怎么把它放到机器上跑起来。
## 338 模型分批覆盖
主流程按 CSV 中已就绪的模型运行:FP32 基线 → 导出 → 校准、编译 → S100P(`nash-m`)上板 → T1 精度评测。新任务默认要求任务指标相对 FP32 劣化不超过 5%,改善也通过;G2 余弦用于诊断,性能只记录。Agent / RLVR 采集按需单独启动。
`bench coverage --all-runs --csv reports/model-coverage.csv` 维护原 CSV 全部 338 项;
相邻 JSON 保存所有 run/job/kk、逐阶段记录、真实轨迹路径、teacher 身份和训练导出关联。
静态前置齐备、FP32/ONNX 对照通过、量化编译、真实上板、任务精度通过分别记录,不能互相替代。
SIFT/ORB/AKAZE 仍占总表三行,按传统算法记不适用 PTQ,不虚构 HBM 或 Agent 轨迹。
新接入模型先核查具体权重、构建/导出、输入、校准、标注和评测门;不修改原 CSV 标签或旧 run。
Ultralytics COCO 分割可用 `scripts/probe_coco_adapters.py --model yolov8n.seg` 做真实权重、
ONNX 数值与 COCO 指标对照。Pose 另存原图坐标关键点对应及偏移,必须通过真实导出对照才能入队;
板端 G3 同时要求 OKS AP 和平均偏移 ≤2px,缺几何证据则 UNVERIFIED。probe 不是板端验收。
FastSAM 的类别语义仍属阻塞。EfficientViT 构建器文件存在不代表依赖、预处理和导出已验证。
通过前置核查的具名批次沿用现有命令:
```bash
bench new-run --note '338 coverage: validated model batch'
bench expand --run RUN --modes oracle,e2e --marches nash-m --model yolov8n.seg
bench run --run RUN --parallel 2
bench collect --run RUN --modes e2e --first-pass --watch --parallel 2
```
显式 `--model` 放宽该批的 priority/task/status 选择,但保留 march/backend/qcfg、teacher、e2e k=4、
冻结精度门槛及原采样编号;不再生成 Oracle repair,缺真实 probe 或其他前置时整批拒绝。
`--first-pass` 只消费每个 job 的首条,原重复采样仍保持 QUEUED;等可执行模型完成首轮再用普通 collect 接续。
当前旧 collector 若需切换,只设置其 `state/drain-collect-RUN`,等在途任务与导出自然完成再启新进程。
不能直接 kill 或用新代码同步冒充旧进程已加载首轮策略。
创建 `state/coverage-master.enabled` 后,新进程每次训练导出会刷新总表;旧进程可沿用
`bench export --run RUN --follow` 接入相同刷新。总表失败会显式报错,不回滚已完成任务和训练数据。
设计:[BPU 量化轨迹工厂设计方案](docs/BPU量化轨迹工厂-设计方案.html)(v0.4)。契约:`CONTRACT.md`。方法论、闭环说明和技术分享讲稿见 [项目文档](docs/README.md)。本 README 只讲怎么把它放到机器上跑起来。
主流程:GLM-5.3 领取任务,按需准备输入、校准、编译、上板和评测 → harness 按本次实测独立判定 → 导出原始轨迹并筛选 SFT 候选。`bench run` 可选地预热权重、FP32 和 ONNX 缓存;`--oracle` 只作显式对照。Oracle 不参与 Agent 判定、奖励或 SFT 筛选,也不再生成 repair。历史文件和排队编号保留,旧 repair 不再领取。新任务仍要求相对 FP32 劣化不超过 5%,改善也通过;历史任务沿用冻结门槛。
已有批次只把剩余任务切到 100 张测试集,见 [剩余任务交接](CONTINUE.md)。使用交接清单保留原采样序号,不全量展开或重新补采。
## 0. 上机
@ -33,20 +66,20 @@ for d in state data cache; do mkdir -p /home/$USER/bench-$d; [ -L $d ] || { rm -
| 4 | `~/toolchains/oe-3.7.0/venv/bin/python scripts/ingest_pool.py --dataset imagenet1k --dataset coco2017` | `data/pool/…/manifest.json`,只读 |
| 5 | `… scripts/make_t1.py --dataset imagenet1k --dataset coco2017` | `data/t1/<ds>/20260914.tar` |
| 6 | `… scripts/make_calib.py --task cls --input-spec 1x3x224x224` / `--task det --input-spec 1x3x640x640` | `data/calib/…` |
| 7 | `bench preflight --modes oracle --oracle-only` | 0 errors 才往下;不要求 agent 环境 |
| 8 | `bench new-run --note "S100P"` → `bench expand --modes oracle --marches nash-m` | 每个就绪模型的 `runs/r…/<job_id>/job.json`,`state/jobs.sqlite` |
| 7 | `bench preflight --modes oracle,e2e` | 0 errors 才往下;包含 Agent 环境检查 |
| 8 | `bench new-run --note "S100P"` → `bench expand --modes oracle,e2e --marches nash-m` | 每个就绪模型的 `runs/r…/<job_id>/job.json`,`state/jobs.sqlite` |
| 9 | `bench run --dry-run` | 状态机空转一遍,不碰工具链 |
步骤 4–6 用 PTQ venv 的 python(要 pyarrow / pillow / numpy)。ImageNet 首选 HF parquet 14 片;tar 是备用源,入池时会 `tar -tzf | wc -l` 校验。
## 2. P2 切片(真跑)
环境和数据准备好后,`scripts/start_run.sh --note "S100P" --parallel 2` 会检查环境、展开 nash-m oracle 并后台运行;`--resume` 续跑已有 run。其他板型需先在 `registry/matrix.yaml` 的 `marches` 中加入对应值,再用 `bench expand --marches ...` 选择。
环境和数据准备好后,`scripts/start_run.sh --note "S100P" --parallel 2` 会检查环境、展开 nash-m 输入准备和 Agent 计划,后台准备 01–03 并按首轮优先采集;`--resume` 续跑已有 run。采样次数、GLM-5.3 身份和精度门槛不变。其他板型需先在 `registry/matrix.yaml` 的 `marches` 中加入对应值,再用 `bench expand --marches ...` 选择。
```bash
bench run --job cls--resnet18--nash-m--ptq--int8 --parallel 1
bench run --oracle --job cls--resnet18--nash-m--ptq--int8 --parallel 1 # 仅需要完整对照时
bench show cls--resnet18--nash-m--ptq--int8 # 阶段时间线、stderr 尾巴
bench run --job det--yolov8n--nash-m--ptq--int8 --parallel 1
bench run --oracle --job det--yolov8n--nash-m--ptq--int8 --parallel 1
bench status; bench status --fail; bench report # reports/<run_id>.html
```
@ -112,15 +145,15 @@ march / 输出前缀 / 工作目录被 harness 钉死;`--config` 必须在 `wo
- **机器是共享的**(2026-09-14 实测):244 核 / 1.4 TB / **8 张 32 GB 卡**(不是 4 张),4/5/7 被别人跑满、6 占 29 GB;12 个登录用户。`config.yaml pools.gpu.devices: [0,1,2,3]` 钉死只用空闲卡,槽位 → `CUDA_VISIBLE_DEVICES`。
- 资源池是跨进程的(`state/locks/<pool>-<i>.lock` flock):oracle 调度进程、agent 的 `bench stage run`、多个 `bench run` 共用同一组槽位;board 池 1 槽 = 严格串行。model-level 阶段按模型加锁,三个 march 并行时只有一个真跑 01–03。job 认领是原子的(UPDATE … WHERE state='PENDING'),两个调度进程不会重跑同一个 job。
- 新 run 的 manifest 记 `harness_code_sha256`;harness 代码变了只告警,venv / registry / EVALUATOR_VERSION 变了拒跑(实测拦下过一次:改 config.yaml 的 venv 列表后 `bench run` 拒绝,开了新 run)。
- 沙箱板是 S100P(`nash-m`);历史 job 未记录 board 时,阶段按 march 查询当前板卡配置。没有匹配板卡时无法完成板端 G3,结果保持 UNVERIFIED。`board_perf.py` 开测前探测板上有无别人的推理进程 / BPU ratio,最多等 10 分钟,仍忙则照测但标 `contended: true`,reward 的延时相对项忽略这类样本。
- 沙箱板是 S100P(`nash-m`);历史 job 未记录 board 时,阶段按 march 查询当前板卡配置。没有匹配板卡时无法完成板端 G3,结果保持 UNVERIFIED。`board_perf.py` 开测前探测板上推理进程 / BPU ratio,最多等 10 分钟,仍忙则以 `board_busy` 拒绝测量。07/08 使用独立板端守护进程,持有板端互斥锁,并在 SSH 断连、30 秒未收到租约或原有单次 RPC 3600 秒上限到达时回收推理进程组。详见 [板端进程生命周期](docs/board-process-lifecycle.md)。
- `job.json` 与 mode 无关:一个 job_id 一份,`rollout_plans` 列出 e2e/repair 计划;oracle 与所有 rollout 共用。
- sqlite 默认 TRUNCATE 日志(WAL 在 FUSE 上会 disk I/O error);`BENCH_STATE_DIR` 可把 state 放本地盘。
- 校准集来源:本机 ImageNet / COCO 只有 val,`make_calib.py` 退到 val-holdout(排除 T1)并在 manifest 记 `source`;出 T2 交付数前必须换 train。
- 07-board 失败重试后仍失败 → G4 unverified、继续 08-eval;08-eval 失败 → job UNVERIFIED 带 error_class;01–06 失败 → FAIL。
- claim 缺席(agent 没调 `claim` 工具)→ claim.json 记 `ABSENT`,reward −0.25;claim=PASS 但没有可核验产物 → verdict FAIL 并计 claim≠verdict。
- BLOCKED 正确性:claim.error_class 与 oracle 同类(`unknown` 不算同类)且阶段一致(claim 未填阶段则不比阶段)。
- reward 的精度相对项按 G3 的 `kind` 解释方向:`rel_pct` 是劣化百分比,越小越好;`abs_pp` 是改善百分点,越大越好。
- BLOCKED 仅记录阻塞报告,不根据 Oracle 推断正确性,不进入成功 SFT 或奖励学习候选。
- reward v0.6 只看本次实测门、成本和声明,不与 Oracle 比较。导出时只读重核旧判定,保留源结果;历史 repair 单列为分析案例。
## 6. 目录里什么进 git

View File

@ -1,6 +1,6 @@
# bench agent · 系统提示(追加在 pi 默认系统提示之后)
你在一个受控的量化上板环境里工作。规则不是建议,违反会被工具层拒绝或被判负。
你负责从权重到导出、FP32 基线、量化、编译、上板和精度评测的完整任务。harness 提供执行工具、资源池与独立核验,不会预先替你跑通模型。有效缓存可以复用。规则不是建议,违反会被工具层拒绝或被判负。
## 你能写的地方
只有当前目录(`workspace/`)。其他一切只读。越界写入会被 path-guard 拒绝并记录。
@ -49,9 +49,12 @@ compiler_parameters:
诊断信号在哪:`bench stage wait` 结束后给你 stdout/stderr 路径;hb_compile 日志末尾有逐节点 `Calibrated Cosine / Quantized Cosine` 表和模型输出余弦。余弦偏低时可检查敏感节点、尝试 `node_config` 提到 int16;最终任务精度以 G3 实测与 job 中的阈值为准。
## 重阶段不在 bash 里跑
编译、上板、评测由 harness 执行。你提交、你等待、你看日志:
导出、基线、编译、上板和评测均由你提交工具,harness 执行并保存日志:
```
bench stage run fetch # 获取或复用正式权重
bench stage run export # 导出 ONNX;有效缓存直接复用
bench stage run baseline # 当前 T1 的 FP32 参考;同子集缓存直接复用
bench stage run calibrate # 生成编译配置(也可先看 job.json 自己写一份)
bench stage run compile --config compile.yaml # cwd 就是 workspace/,相对路径按 workspace 解析(写 workspace/compile.yaml 也认)
bench stage wait <handle> --poll 60
@ -73,10 +76,10 @@ claim(terminal="PASS"|"FAIL"|"BLOCKED", stage=..., error_class=..., summary="…
```
**预算规则**:剩 5 轮时你会收到一条预算提醒——收到后无论进展如何,先调用 `claim` 汇报当前状态(不确定就报 FAIL 并写清到了哪一步),再决定要不要继续。达到轮数上限会立即结束进程,没有额外 grace 轮;请提前 claim。token 预算耗尽后只允许 claim,仍受轮数硬上限约束。没有 claim 就结束的轨迹被记为 claim 缺席,另扣分。
你的 claim 会与 harness 独立核验的 verdict 比对。claim=PASS 而 verdict=FAIL 是最重的扣分项,比老实报 FAIL 差得多。不确定就报 FAIL 并说明。BLOCKED 只在你确认是工具链/算子层面无法完成时使用,并给出 error_class(unsupported_op / shape_infer / calib_nan / verifier_mismatch 之一)。
你的 claim 会与 harness 独立核验的 verdict 比对。claim=PASS 而 verdict=FAIL 是最重的扣分项,比老实报 FAIL 差得多。不确定就报 FAIL 并说明。BLOCKED 用于有实际错误证据的工具、适配器或材料缺口;给出具体阶段、错误和所需材料。无法归入工具已有错误类别时用 unknown,不能把缺权重或缺适配器说成 unsupported_op。
## 不要做的事
不下载数据集或权重(都在只读区);不改 `job.json`;不猜指标——指标只来自 `bench stage run eval` 的输出;不在 bash 里 `sleep` 等编译,用 `bench stage wait`。
不直接写共享数据或权重缓存,权重获取用 `bench stage run fetch`;不改 `job.json`;不猜指标——指标只来自阶段工具的实测输出;不在 bash 里 `sleep` 等编译,用 `bench stage wait`。权重、适配器或数据缺失时如实报告具体缺口;未尝试编译就不能声称工具链不支持该模型。
## 工具链参考
`.horizon/skills/` 下的 skill 是这条工具链的文档。`oe-doc <query>` 可以查 OE 文档。两套 venv 的选择由 harness 按 job.json 的 backend 决定,你不需要也不能选。

View File

@ -1 +1 @@
6
7

View File

@ -1,4 +1,4 @@
# 任务:把 {model}({task})量化并编译到 {march}({backend} / {qcfg}),通过全部必选门
# 任务:从模型权重出发,将 {model}({task})部署到 {march}({backend} / {qcfg}),通过全部必选门
job:`{job_id}` run:`{run_id}`
job.json:`{job_json}`(只读) 数据:`{data_dir}`(只读) 你的可写区:当前目录
@ -11,13 +11,16 @@ job.json:`{job_json}`(只读) 数据:`{data_dir}`(只读) 你的
必选项以 job.json 的 `gates.required` 为准。
## 你手上有什么
- FP32 ONNX:`{onnx}`(03-export 已产出)
- FP32 基线:`{baseline_dir}/metrics.json`
- 校准集:`{calib_dir}`({calib_n} 个 npy,与 val 零交集)
## 输入和可复用缓存
- 权重来源、模型身份和输入约定:见 job.json 的 model_spec
- ONNX 缓存位置:`{onnx}`(可能尚未生成)
- FP32 基线缓存位置:`{baseline_dir}/metrics.json`(可能尚未生成或属于旧子集)
- 校准集位置:`{calib_dir}`(目标 {calib_n} 个 npy,与 val 零交集)
- T1 评测子集:`{t1_tar}`({t1_n} 张)
## 流程
先读 job.json;系统提示里有 compile.yaml 全部可改的旋钮,不必再读 harness 或工具链源码。默认配置(calibration_type: default,全 int8)先跑一次拿到基线,看日志里的逐节点余弦再决定调什么。提交 `bench stage run compile --config <你的>.yaml`(cwd 就是 workspace/,不要再加 workspace/ 前缀),`bench stage wait`。失败就读日志改配置;同一方法最多重试 1 次。编译过了依次跑 `bench stage run hbm`、`bench stage run board`、`bench stage run eval`,每步等待完成;性能记录失败仍继续评测。最后调用 `claim`,缺少板端精度结果时不能报 PASS。
先读 job.json,然后依次提交 `bench stage run fetch`、`bench stage run export`、`bench stage run baseline`,每步用 `bench stage wait` 等待并检查结果。这些阶段由你发起并记入本条轨迹;有效缓存会直接返回 SKIPPED,不重复计算。没有人预先替你判断能否量化成功。
输入准备完成后,由你选择量化方案并提交 `bench stage run compile --config <你的>.yaml`(cwd 就是 workspace/),再 `bench stage wait`。系统提示里列有 compile.yaml 可改参数。失败就读日志改方案;同一方法最多重试 1 次。编译通过后依次提交 `hbm`、`board`、`eval`,每步等待完成;性能记录失败仍继续评测。遇到缺权重、缺适配器或缺评测工具,明确记录实际错误和缺失材料,不把它写成量化失败。最后调用 `claim`,缺少板端精度结果时不能报 PASS。
预算:{max_turns} 轮、{max_tokens} tokens。等待编译不消耗预算。剩 5 轮会收到提醒——那时先 claim,别等到耗尽。

View File

@ -9,7 +9,7 @@ import time
import uuid
from pathlib import Path
NAMES = {n.split('-', 1)[1]: n for n in ('04-calibrate', '05-compile', '06-hbm', '07-board', '08-eval')}
NAMES = {n.split('-', 1)[1]: n for n in ('01-fetch', '02-baseline', '03-export', '04-calibrate', '05-compile', '06-hbm', '07-board', '08-eval')}
def rollout_dir():
return Path(os.environ['BENCH_ROLLOUT_DIR'])

View File

@ -39,6 +39,12 @@ watchdog:
cpu_idle_pct: 5
poll_s: 30
hard_ceiling_s: 86400 # 24h 纯保险
io_stall_s: 21600 # 树内持续有 D 线程(卡在 I/O)这么久 → 也判定异常。
# 别往下调:线程级 D 检测上线前这条是死代码(d_count 恒 0),
# 上线后它第一次生效,收紧会把"链接期正在写盘"换个理由再杀一次。
no_progress_ceiling_s: 28800 # 日志与 CPU 都不动这么久 → 兜底,免疫 D 抖动。
# 必须 > io_stall_s,否则它抢先命中、盖掉更准确的 STALLED 原因。
# 四条阈值单调:stall_s(long 3600) < io_stall_s < 本条 < hard_ceiling_s
soft_alert: p95x3 # 超过历史 p95 × 3 只告警,不杀
perf:
@ -77,7 +83,7 @@ data:
endpoints:
teacher:
provider: openai-compatible
base_url: https://ai-api.d-robotics.cc/v1
base_url: https://open.bigmodel.cn/api/coding/paas/v4
model: glm-5.3
api_key_env: TEACHER_API_KEY
student:

View File

@ -0,0 +1,475 @@
# 嵌入式大模型后训练 · 第一阶段汇报讲稿
面向:技术团队 / 同事(领导在场) | 完整版约 30 分钟,速用版 3 分钟 | 数字截面:2026-09-16 21:03 运行态快照 + 09-17 设计变更
> 用法:每节先是"讲什么"(可以直接念),后面是"提示"(只给你自己看:数字来源、措辞纪律、被追问时怎么接)。
>
> 全篇措辞原则:讲进展用具体数字,讲未完成用"在建 / 已排期",讲经验用"这一类工程的共性规律"。不评价任何既往决策,不做"早期做错了"的归因——第一阶段的目标本来就是打通,现在是按计划进入第二阶段。
---
# 速用 · 三分钟口头版
> 临时汇报、例会同步、走廊被问到,用这一版。可以直接念,念完约 3 分钟。
**背景。** 前期我们调研了嵌入式方向上能做的题目,可选的其实不少。我们的判断是,第一个锚点要选一条能自己闭环的——判据能被程序判定、验证比求解便宜、失败原因能分类,这三条同时成立,才撑得起大量采数据。BPU 模型量化上板这条线三条都满足,和楠总对齐之后,方向就定在这里了。
**起点。** 方向定下来,这个阶段的重点就是把上板这件事的数据积累起来。起点是三样现成的东西:HBDK 4 工具链、S100P 板子,以及我们整理的 338 个公开视觉模型清单——其中已完成接入的 63 个(分类 48、检测 15),其余的在逐个做导出适配和评测器补齐。
**做了什么。** 搭了两套东西。
一套是**数据收集系统**:agent 在隔离环境里拿到一个模型,自己决定怎么量化、怎么编译,harness 负责执行工具、分配板卡、独立核验它做出来的产物,全过程留痕。agent 这一层用的是 pi,teacher 模型是 GLM-5.3。
另一套是**板端评测环境**,这块的投入产出比超出预期:板子上只做一件事——跑张量。读图、预处理、后处理、算指标全部留在主机侧,标注和评测代码永远不上板。好处是加一类新任务板端零改动,而且 FP32 基线和量化模型吃同一份输入、走同一个评测器,比较才有意义。顺带一个意外收获:**板端推理比服务器模拟快 45 倍**——所以"必须上真板"不是成本,是提速。
**这里有一个重要的设计调整,上周做的。** 原本每个模型要先用确定性流水线跑一遍产出标准答案,再让 agent 做。实测下来,这一遍吃掉了三分之二的板时,而它一条轨迹也不产。所以我们把它改成了 **agent 主导**:默认只准备输入——权重、FP32 基线、ONNX 导出,这三步结果可缓存复用;输入齐备 agent 就直接上手。完整的确定性流水线保留成一个显式开关,只在做对照和排查问题的时候跑。**精度判据一点没放松**,因为参照物从来就是 FP32 基线,不是默认配置的量化结果——判的还是板端实测相对 FP32 劣化不超过 5%。
**现在的状态,三个数。**
第一,**板端终判打通了**。精度数字来自 S100P 真实推理,不是服务器模拟——这是这个阶段最关键的一步,在此之前所有精度都是模拟出来的,不能作为交付依据。目前已有 18 个模型完成板端实测并通过,这是第一批可以对外讲的上板基准。
第二,**agent 轨迹开始产出**。已结束 34 条,其中判定通过 22 条,覆盖 14 个不同模型。作为对比,上一个截面这个数是 3 条,而且全部集中在同一个最简单的模型上——那时候只能说明流程能走通,现在能说明 agent 确实在解决问题。最好的一条,板端 top-1 相对 FP32 只掉了 0.06%。
第三,**出现了第一条修复模式的成功轨迹**:确定性流水线做不出来的模型,agent 调出来了。明细还在核验,核完它就是我们第一条黄金样例。
**资源。** 现在是 4 块板加一台 8 卡机器。板卡是这条线的咽喉——严格串行,一块板同时只能跑一个任务,利用率已经到 93%,基本是满的。刚才那个设计调整把三分之二的板时让给了采集,等于产能翻了两倍多,但这是一次性红利,规模再往上走还是要靠扩容。一轮完整实验需要多少板时是能算出来的〔填入下一节算好的数〕。所以后续主要的诉求是板卡扩容,我这边也会去协调。
**下一步三件事。** 把 agent 单次实验从一个多小时压到 5 分钟以内,方案已经验证过;接学生模型,跑通在线训练闭环;把覆盖从 63 个模型往更多任务类扩,分割和姿态估计的适配已经在做了。
---
# 速用 · 数字卡片
> 汇报前存手机或打印一页。**打✱的项汇报当天早上要重新取一次**,它们会随采集推进变化。
| 会被问 | 直接答 |
|---|---|
| 模型清单多大 | 338 个公开视觉模型,10 个任务类;已完成接入 63 个(分类 48、检测 15),其余在补导出适配和评测器 |
| 板端跑通了吗 ✱ | 通了。已有 18 个模型完成板端实测并通过 |
| 采了多少轨迹 ✱ | 已结束 34 条(通过 22 / 失败 8 / 未验证 4),另有在跑和排队的 |
| 覆盖多少模型 ✱ | 通过的 22 条覆盖 14 个不同模型,最简单的 resnet18 只占 4 条 |
| 最好的一条结果 | regnet.y.800mf 板端 top-1 77.85 对 FP32 77.90,相对劣化 0.06% |
| 精度判据是什么 | 板端实测的任务指标,相对 FP32 基线劣化 ≤ 5%,改善也算通过 |
| 一次上板多久 | 性能测试 31 秒;完整精度评测(2000 张)约 8 分钟 |
| 板子忙不忙 | 12.6 小时窗口内利用率 93% |
| 板端比模拟快多少 | 45 倍(0.889 毫秒 vs 40 毫秒一张) |
| 编译要多久 | 单次 35 到 118 分钟;其中量化只占 107 秒,BPU 编译占 78 分钟,比例 1:44 |
| 测试集为什么 2000 张 | 配对比较下差值标准误约 0.5 个百分点;500 张时约 1 个点,和容差同量级会误判 |
| 隔离安不安全 | 内核级隔离,红队 11 种路径逃逸全部挡住 |
| 现在还跑标准流水线吗 | 默认不跑了。只准备输入(权重 / FP32 基线 / ONNX),agent 直接上手;完整流水线保留成显式对照开关 |
---
# 速用 · 资源诉求怎么提
> "能扩到更多会更好一些"这种说法领导没法决策。给算法、给数、给不扩的后果,三样都要有。
**算法**(当场能写在白板上):
```
一轮实验的板时 = 题目数 × 每题采样条数 × 每条里的板端测量次数 × 单次测量耗时
```
**现有实测单价**:单次完整精度评测约 8 分钟,性能测试 31 秒,板卡严格串行。
**代入现在的规模**(63 个模型 × 4 条采样,每条按 2 次板端评测算):约 500 次测量 × 8 分钟 ≈ 67 小时,**1 块板要 2.8 天,4 块板约 17 小时**。
**怎么讲**:现在 4 块板对应的节奏是一天一轮。上周那次设计调整已经把三分之二的板时从"跑标准答案"让给了"采轨迹",这是一次性红利,已经吃掉了。接下来采集规模按计划往上走——覆盖从 63 个模型扩到更多任务类、每题采样数从 4 提到 8——板时是线性涨的,届时一轮就变成三四天。这是训练迭代速度的直接上限,所以板卡扩容不是效率优化,是决定我们多久能验证一次假设。
**两个配套动作一起提**(显得诉求是算过的,不是单纯要东西):
- 两阶段编译落地后,agent 单次实验从一个多小时压到 5 分钟以内,同样的板子能多跑几倍任务——先把现有资源的效率榨出来,再谈扩容。
- 板卡要按物理身份登记、一块板一把锁。同一地址的多个容器不能算成额外容量,否则排队系统以为有多块板,两个任务同时推上去互相干扰,测出来的延时全废。
---
# 速用 · 资料清单
**A. 手边必备**:上面的数字卡片,一页纸。
**B. 投屏随时能打开**(按最可能被问的顺序排好标签页):
1. `BPU量化轨迹工厂-方法论.html` —— 被问"整条线怎么运转",打开 §0 流程图
2. `BPU量化轨迹工厂-闭环方法论.html` —— 被问"怎么算走通了",打开 §5 分辨力四例表
3. `CPU算子优化Agent-立项说明.html` —— 被问"下一个方向",打开 §5 缺口总表
4. 本讲稿 —— 被问细节时按节号跳
> ⚠️ 前两份 HTML 是按"确定性流水线先出标准答案"那一版设计写的,和现在的 agent 主导不一致。投屏前要么先说一句"这是上一版设计,第 X 节讲的调整就是针对这里",要么抽空把两页的对应段落更新掉。**别让听众自己发现不一致。**
**C. 被追问才调的证据**:
| 问题 | 文件 |
|---|---|
| 板端利用率、故障统计、补测结果 | `outputs/plan-verify-20260916/REVIEW.md` |
| 板卡连通性、已有产物清单 | `outputs/s100p-gap-20260916/` |
| 当前运行态计数 | `outputs/glm53-endpoint-switch/runtime-after.json` |
| 四道门口径、agent 主导的新原则 | `bench/CONTRACT.md` §0 和 §4 |
| 338 项覆盖总表 | `bench coverage --all-runs --csv reports/model-coverage.csv` |
**D. 汇报前必须自查的四件事**:
1. **板卡登记状态。** 容量等于"注册表里启用的板数",不是物理板数。最近一次记录里启用的只有 1 块,另一块因 BPU 打不开被停用。物理上现在有 4 块,要确认是否都已登记并启用——否则说"4 块板"但调度容量是 1,会被追问。
2. **修复模式那条轨迹的明细。** verdict、claim 一致性、审计洁净度三项,核完才能讲"黄金样例"。
3. **重新取一次计数。** 数字卡片里打✱的会变,用当天早上的数。
4. **repair 的触发条件。** 代码里 repair 仍然挂在"oracle 失败"上。默认不跑完整流水线之后,repair 默认也就不触发了——确认一下现在是靠显式 `--oracle` 补现场,还是已经改成吃 agent 自己的失败。这个点很可能被问到(见问答第 3 条)。
---
## 0. 开场(1 分钟)
**讲什么**
今天汇报三段:我们的主线是什么、为什么第一个锚点选在 BPU 部署这条链路上、这条链路现在跑出了什么。
先给结论:**主线是训练一个能干嵌入式工程活的大模型**。这件事成不成,不取决于我们用多大的底座模型,取决于一件更基础的事——我们能不能把"做得好不好"变成一个机器判得准的数字。判不出来就没有训练信号,再大的模型也只是在猜;判得不准,训练就是在拟合噪声。
所以第一阶段我们没有直接去训模型,而是先把这个"判得准的数字"造出来。现在它造出来了,并且已经在产出真实轨迹。
**提示**
- 开场不报通过率,第 6 节会专门讲为什么通过率不是指标。
- "不取决于用多大的模型"是整场的主心骨,后面每节都回扣它。
---
## 1. 主线:我们要训的是什么模型(4 分钟)
**讲什么**
我们的定位是做行业的底座——供应商逻辑。放到模型上,就是训一个**嵌入式领域的工程 agent**:客户把模型交给我们要往板上部署,中间那一串工程活——量化配置怎么调、算子不支持怎么办、落到 CPU 上的算子怎么优化、端到端延时怎么压——现在靠工程师一个个手工做,我们要让模型接手。
这类模型和通用对话模型的训练方式不一样,差别就在监督信号:
| | 通用对话 / 创意类 | 我们这类嵌入式工程任务 |
|---|---|---|
| 对错谁判 | 没有客观判据,要训一个奖励模型当裁判 | 程序判定:返回码、精度掉点、板端延时 |
| 主要风险 | 长期和"裁判被钻空子"缠斗 | 判据本身要造准,造准了就不会被钻 |
| 数据来源 | 人工标注 / 偏好对 | 真实执行留下的轨迹,自带可核验结果 |
这是我们的结构性优势,不用白不用。但要兑现它,前提是先有一个**可机器判定的验证器**,而且要有分辨力——不只是能盖章通过,还要能把该拒的拒掉。
所以训练路径是这样一条,顺序不能换:
1. **造验证器**:把工程任务变成有判据的题,判定完全由程序做。
2. **采轨迹**:让 agent 在隔离环境里做题,执行过程和核验结果一起落盘。
3. **评测与打分**:同一套代码独立核验,核验结果就是训练信号。
4. **训练**:先用做对且诚实的轨迹做 SFT 打底,再接 RLVR 做在线强化。
5. **回到 1**:模型变强之后,题目难度和判据跟着升级。
第一阶段做的是第 1 到第 3 步,第 4 步的数据和接口已经备好。
**提示**
- 这段是领导最关心的"我们到底在干什么",讲慢一点。
- 表格不要念完,念第一列和最后一列的对比,重点落在"我们不在那个约束里"。
- 被问"什么时候开训":答案在第 9 节,判据是"分辨力四例各有一例被正确落位",不是"攒够多少条"。
---
## 2. 锚点:为什么第一个选 BPU 部署这条链路(3 分钟)
**讲什么**
嵌入式的工程任务很多,我们前期接触到的也不少。第一个锚点选在 BPU 部署链路上,理由很实在:**它是目前唯一一条能自己闭环的**。
判断一个任务适不适合做成训练场,有三条硬标准,三条同时成立才行:
| 标准 | 含义 | BPU 部署链路满足吗 |
|---|---|---|
| 判据可机器化 | 对错由程序判,不靠人评、不靠模型当裁判 | 满足:返回码、逐输出余弦、Top-1 / mAP 差值,全是数值比较 |
| 验证远比求解便宜 | 跑一遍很贵、判一次很便宜,才撑得起大量采轨迹 | 满足:跑一个模型要编译加上板评测,判一次只是读一份结果文件 |
| 失败模式是有限集 | 失败能分类,才能区分"判断错"与"工具链真做不到" | 满足:确定性五类、瞬态三类,其余进人工队列 |
第三条最容易被忽略,但它决定了"正确地放弃"这种能力能不能被学到。没有失败分类,agent 说"这个做不了"我们无法判真假,只能一律当失败处理。
除了这三条,还有现实理由:模型清单是现成的、数据集是现成的、板卡是现成的——环境搭起来就能跑,不需要先等别的条件成熟。
这条链路上有两个落点,一个在跑、一个在立项:
- **落点一:量化编译(在跑)。** 模型往板上走,量化配置怎么调才能既过精度又跑得快。这是本次汇报的主体。
- **落点二:CPU fallback 算子优化(立项中)。** BPU 支持不了的算子会落到 CPU 上,往往吃掉端到端延时的大头。骨架可以直接复用,判据要按新任务重做——量化任务判的是精度,算子优化判的是正确性和速度,判据怎么定要重新设计。立项说明已经写好,等第一个落点的数据链跑顺再启动。
**提示**
- 说"第一个锚点",不说"只能做这个",留出横向扩展余地。
- 落点二一句话带过,有兴趣的会单独找你。
- 被问"为什么不先做更有业务价值的":答"这条链路本身就是业务交付——跑出来的就是模型在 S100P 上的上板基准数据,这是要交付的东西;训练数据是同一遍执行的副产物,不额外花钱"。这句能解释"为什么不是纯投入"。
---
## 3. 怎么做:三步走(5 分钟)
**讲什么**
方法上就三步:**搭环境 → 跑任务采数据 → 评测**。每一步里真正花时间的地方我说一下。
### 第一步:搭环境
不是装几个包的意思,是把"可复现"做成机械规则:
- 数据集复制入池、设只读、写清单(记哈希 / 数量 / 来源 / 日期),agent 永远不下载。三个理由:可复现(线上数据集版本会飘)、防作弊(agent 会挑好算的子集)、轨迹干净(网络波动不该进训练数据)。
- 资源分池:GPU、CPU、板卡、agent 各自设并发,跨进程文件锁。板卡严格串行,一块板一个槽位——用一个并发数管所有会把 GPU 堵死。
- 两套工具链两套虚拟环境,启动前比对依赖哈希,不一致直接拒跑而不是告警——否则"只追加不覆盖"就是句空话。
- agent 的作答区隔离在内核层,隔离起不来就拒绝启动,不静默降级。
### 第二步:跑任务采数据
这里上周做了一次**设计调整,是这个阶段最重要的一次决策**。
原来的做法是每个模型先由确定性流水线完整跑一遍产出标准答案,再让 agent 做。实测下来这一遍代价很大:在一个 12.6 小时的观测窗口里,标准答案占了 7.9 小时板时,agent 采集只占 3.8 小时——**三分之二的板时花在一条轨迹也不产出的事情上**,而板卡是整条线的咽喉。
所以改成 **agent 主导**:默认只做输入准备——取权重、跑 FP32 基线、导出 ONNX,这三步的结果可缓存、可复用;输入齐备 agent 就直接领任务上手。完整的确定性流水线保留成一个显式开关,需要做对照或者排查问题的时候单独跑。
这个调整有一条边界要说清楚,因为它是最容易被误解的地方:**精度判据没有因此变松**。G3 判的是"板端实测的任务指标相对 **FP32 基线** 劣化不超过 5%"——参照物从来就是 FP32,不是默认配置的量化结果。FP32 基线属于输入准备的一部分,照常产出。所以去掉的只是"默认配置能做到多少"这个参考点,判定本身一点没动。
顺带一个收益:交付口径其实变好了。原来的基准数字回答的是"默认配置能不能上板",现在回答的是"这个模型在 S100P 上到底能做到什么水平"——后者才是客户真正想知道的。
### 第三步:评测
**这一层的投入产出比超出预期,值得多说两句。**
核心设计是一条:**板上只跑张量,别的都不上板。**
```
主机:读图 → 预处理 → 输入张量
↓ 传张量
板子:跑推理 ← 板上只做这一件事
↑ 回传输出张量
主机:后处理 → 算指标 → 判门
```
标注、评测器、后处理、指标计算永远留在主机侧。这带来四个好处:
1. **加新任务类板端零改动。** 不管是分割还是姿态估计,板子看到的都只是张量。我们现在正在补分割和姿态的适配,板端一行代码没动。
2. **评测口径只有一份。** FP32 基线和量化模型吃同一份输入、走同一个评测器、比同一批图,比较才有意义——不会出现"板上一套、主机一套"。
3. **板上不用装依赖。** 不需要 pycocotools、不需要各种数据集工具包,接一块新板的成本很低。
4. **意外收获:板端比服务器模拟快 45 倍。** 单张 0.889 毫秒对 40 毫秒。所以"必须上真板"不是成本,是提速。
评测这层还有两件事是刻意做的:
- **数据分三档,规模是算出来的不是拍的。** 校准集、日常筛查集、交付全量集。日常筛查集分类任务用 2000 张——因为 500 张评 Top-1 单次标准误约 1.9 个百分点,分辨不出 1 个点的掉点;而 FP32 和量化是配对比较,差值标准误由结论不一致的图决定,2000 张时约 0.5 个点。所以 2000 张上只记"筛过了",交付数字要用完整验证集。
- **入队门槛严格,探测不等于验收。** 新接入的模型要先核查权重入口、导出代码、评测器、校准数据、标注和附加门,缺一项就拒绝入队。姿态估计还额外要求板端同时判 OKS AP 和平均关键点偏移不超过 2 像素。有一个模型的类别语义没对齐,现在还卡在门外没让进——这是故意的。
最后,评测这层的验收标准要特别说一句:**看分辨力,不看通过数**。一个只会盖章通过的验证器,和一个真能分辨好坏的验证器,从成功样本上看是一模一样的。所以每一步的签收都要包含反向对照——故意造一个错的、造一个说谎的,确认系统把它们挑出来。
**提示**
- 三步的标题要报清楚,细节可按时间裁。
- **第二步那个设计调整要主动讲,不要等人问。** 理由有三:一是已经写进契约的第一条原则,是有记录的决策不是漂移;二是文件夹里那几份 HTML 还是旧设计,不说清楚会被发现不一致;三是"我们发现三分之二板时花在不产数据的事情上,砍掉了"本身是个漂亮的决策故事,藏起来可惜。
- 讲的时候重音放在"判据没变松"这句上,这是唯一可能被质疑的点,主动堵住。
- 第三步是这次新增的重点,领导对"投入产出比高"这种话敏感,把 45 倍和"板端零改动"这两个具体的点说出来。
---
## 4. 这条线长什么样:五个构件(4 分钟)
**讲什么**
具体到实现,是一条单向链,五个构件,职责是硬边界:**出题的不作答,作答的不判分,判分的不看自述。**
1. **冻结的任务单元。** 一道题 = 任务类型 × 模型 × 芯片 × 后端 × 量化配置,拼成一个编号,比如 `cls--resnet18--nash-m--ptq--int8`。它既是磁盘上的目录名,也是所有表的关联键。换工具链、改阈值、修 bug 一律开新批次,老批次留着对照。这样任何一条轨迹永远能追溯到它当时面对的确切环境。
2. **可复用的输入准备。** 取权重、跑 FP32 基线、导出 ONNX。这三步的产物按内容哈希缓存,同一个模型在不同批次之间复用,不重复付钱。**要强调一点:准备产物是"可复用的输入证据",不是"标准答案"**——它不含任何量化结果,不参与判定。完整的确定性流水线是一个显式开关,做对照和排查时才跑。
3. **隔离的作答区。** agent 只有一个可写目录。隔离靠内核的命名空间机制,把兄弟轨迹、凭证、评测代码从文件系统里摘掉,再加进程隔离和权限清空。
4. **独立核验。** 四道门:编译、数值、任务指标、板端延时。其中编译和任务指标是必选门,数值余弦降为诊断,延时只记录。判定完全不读 agent 的自述,自述只用来和核验结果比对,不一致重罚——训练集里出现一次谎报成功,就是在教它谎报。
5. **离线打分和分级导出。** 打分是纯函数、带版本号,改配方可以对历史数据全量重打分,不用重跑任何实验。导出分两级:强化学习收正负例(要的是区分度),监督微调只收"做对了且诚实"的。
还有一条贯穿的原则:**每条轨迹彼此独立,永不串联。** 第 k+1 条看得见第 k 条,组内就不独立了,组内比较失去意义——而组内比较是整个训练方法的地基。所以"读了同一道题别人的作答"在审计里直接判不可用,和"读了评测代码"同等对待。
**提示**
- 构件 2 是这次改动最大的一块,措辞已经按新设计写好,照念即可。"可复用的输入证据,不是标准答案"这句是契约里的原话,很准确。
- 四道门的现行必选/可选:编译和任务指标必选,余弦诊断,延时记录。直接讲现行的,不铺垫历史。
- "出题的不作答,作答的不判分,判分的不看自述"这三句念出来效果好。
---
## 5. 路线与目标收敛(2 分钟)
**讲什么**
实施顺序是严格有序的:地基 → 一条切片打穿全流程并人肉核对每个数字 → 闭环(正常和修复各跑通一次,并能构造出"应该被拒"的例子)→ 横向铺一批压测资源池 → 全量。
第二步为什么要人肉核对:因为后面所有打分都建立在核验结果上,核验可信是整个计分体系的前提。这一步省掉,后面采的所有数据都建在沙子上。
采集分两条线,成功指标不同:一条是覆盖交付,求覆盖率;一条是训练数据,精选题目、每题多条采样,求分辨力。两条线按阶段分开调度,互不挤占。
目标板已经收敛到一块:**S100P**。配套口径也定了:精度必须板端实测,不认服务器模拟;任务指标相对 FP32 劣化 ≤ 5%;延时记录但不设硬门;不允许靠把算子挪下 BPU 来换精度;其他板型暂时无板,只判编译、精度标未验证。
这次收敛省下的算力是实打实的:任务数从两百多降到 63 个完整加编译检查,算力省掉三分之二。加上第 3 节讲的 agent 主导那次调整,同样的板子现在能产出的轨迹是原来的三倍多。
**提示**
- 措辞纪律:说"收敛",不说"之前太散"。收敛是阶段性动作,是好事。
- 最后那句"三倍多"是两次调整的叠加效果(省 2/3 任务数 + 省 2/3 板时),是这次汇报里效率提升最有力的一句,别漏。
---
## 6. 跑到哪了:第一阶段战果(8 分钟,重点)
**讲什么**
**6.1 板端终判接通了。** 这是本阶段最关键的一步。精度评测现在走真实板端推理,主机预处理、板上跑张量、主机后处理算指标。在此之前所有精度都是服务器模拟出来的,不能作为交付依据。性能测试 31 秒,完整精度评测(2000 张)约 8 分钟。
**6.2 第一批可交付的上板基准。** 目前已有 18 个模型完成板端实测并通过,精度按相对 FP32 劣化 5% 判。这是第一批可以对外讲的数字。
**6.3 agent 轨迹开始产出。** 已结束 34 条:通过 22、失败 8、未验证 4,另有在跑和排队的。
**6.4 核心验收信号达成。** 上一阶段定的验收是"非最简单模型的通过轨迹 ≥ 5 条"——当时是 0,三条通过全集中在 resnet18 上,说明系统只证明了"流程能走通",还没证明"agent 能解决问题"。现在 22 条通过覆盖 14 个不同模型,resnet18 只占 4 条,其余覆盖 resnet 系列、resnext、wide-resnet、mobilenet、efficientnet 四个规格、regnet 三个规格、shufflenet 四个规格。
**6.5 修复模式跑出第一条成功轨迹。** 确定性流水线做不出来的模型,agent 调出来了——这正是我们最想采到的样本形态。明细核验还在做,核完可以定为第一条黄金样例。
**6.6 板卡抖动的数据回收。** 中途板子的网络网关有一段拒连,几十次板端评测失败,两条轨迹跑满预算也没测成。处理方式不是改分数,而是加了一个补测入口:用 agent 自己编出来的产物补跑板端两步再重新核验。两条都是真实通过——最好的一条板端 top-1 相对 FP32 只掉 0.06%,是全批次最小的;另一条掉 3.69%,也在门内。同时打分规则补了一条:板卡不可用时只免罚、不加分,并标明不可验证的原因。
**6.7 效率上的两次调整。** 目标板收敛省掉三分之二任务数;agent 主导省掉三分之二板时。两项叠加,同样的硬件现在能产出的轨迹是原来的三倍多。
**6.8 覆盖在往外扩。** 338 项的覆盖总表已经建起来,逐阶段记录:静态前置齐备、FP32 对照通过、量化编译、真实上板、任务精度通过分别记录,不能互相替代。分割和姿态估计的适配在做,姿态的板端判据已经定死——同时看 OKS AP 和平均关键点偏移 ≤ 2 像素。传统算法那几项如实标"不适用 PTQ",不虚构产物。
**6.9 隔离与数据洁净度经得起查。** 内核级隔离做过红队:手写请求打了 11 种路径逃逸,11 种全部挡住。审计脚本对每条轨迹给机械理由,不合格的不进数据集;历史数据集全部加作废标记留档、不删除——轨迹是复盘证据,删了同样的问题要重踩一次才知道。
**提示(措辞纪律)**
- 6.5 **只讲到"状态是通过、明细在核"**,不要说"核心假设已验证"。措辞用"这正是我们想要的形态,核完就是黄金样例"。
- 6.5 还有一个坑:修复模式的输入是"确定性流水线的失败现场",而默认已经不跑它了。如果被追问"那以后还怎么产生修复任务",照第 10 节第 3 条答,不要临场编。
- 不要引用早期那组 143 / 83 / 60 / 38 的统计。如果领导记得并问起,答"那是上一批不同板型的统计口径,这次全部切到板端终判后重新起的截面,两组不可比"。
- 6.2 的 18 个,如果被问"这 18 个是 agent 跑出来的还是流水线跑出来的"——如实说是 09-16 那批完整流水线跑的,现在这批是 agent 在产出;两种都是板端实测,判据一样。**这个问题大概率会问,提前想好一句话。**
- 板端利用率 93% 这个数字为后面的板卡诉求做铺垫,一定要说到。
---
## 7. 这一类工程的共性规律(5 分钟)
**讲什么**
这些不是我们独有的问题,是所有"长时间运行 + 多进程 + 数值判定"系统的共性规律。我们把它们变成了规矩写进契约,后面接手的人不用再花时间。
1. **正在跑的代码不等于磁盘上的代码。** 常驻调度进程加载的是它启动那一刻的代码和配置,缓存还会把旧配置留在内存里。所以"改好了"指的是文件,不是正在跑的东西。规矩:改了流水线必须让旧进程收尾退出再用新代码起新进程,不能只同步文件;进程启动时自证代码指纹,不一致拒绝领新任务。
2. **隔离要做在内核层,不做在字符串层。** 解析 agent 写的命令做白名单,是一类永远追不平的防线——每一版都会出现新写法。改成"让它看不见"之后这类问题结构性地消失了。命令解析那层降级成教学层,只负责给一句人话的拒绝理由、省 agent 的预算。
3. **数值相似度是过程量,任务指标才是结论。** 多分支输出不能展平算一个整体余弦——检测模型的输出里框坐标量级大、类别分数在 0 到 1 之间,整体余弦会被前者主导。实测过一个:整体余弦最低 0.9976,而类别分支只有 0.36,对应 mAP 从 46 掉到 26。规矩:多语义输出逐分支判;余弦降为诊断,不当必选门。
4. **容差必须显式声明方向和单位。** "相对劣化百分比"越小越好,"绝对百分点改善"越大越好,符号相反。独立复现过:同样是掉 2.5%,容差写 -5 判失败、写 5 判通过。规矩:每条容差显式声明类型,评测器按类型算方向;百分点和百分比不相加。
5. **预算耗尽不等于失败。** 已经产出并核验过的结果照常记账,不能因为超预算把整条成果清零——否则一条做得很好但没来得及收尾的轨迹会被记成彻底失败,而那往往是最有训练价值的一条。规矩:执行终止原因和产物判定分开记,互不覆盖。
6. **工具链层面两个具体的坑,写出来给后面的人省时间。** 运行时返回的数组引用的是内部缓冲区,下次调用会原地覆盖,必须显式拷贝——有一版评测因此把 Top-1 算成了 2.6%。校准数据目录下只能放指定格式文件,工具会把目录里所有文件当样本读,清单文件放进去会炸。
**提示**
- 整体是"经验沉淀"的口吻,不是"我们犯了什么错"。每条落点都是"变成了规矩",不是"造成了损失"。
- 不要提具体进程号、不要说某批任务作废了多少个——那是内部复盘细节,汇报讲规矩就够。
- 时间紧就讲 1、2、3,这三条最能说明"为什么这套系统要这么设计"。
---
## 8. 在建项(2 分钟)
**讲什么**
排在第二阶段的工作,都已定方案:
1. **两阶段编译**(最大的提速项)。一次编译里量化占 107 秒、BPU 编译占 78 分钟,比例 1 比 44;而 agent 迭代要看的精度信号在量化结束那一刻就有了。拆开之后单次实验从 35 到 118 分钟压到 5 分钟以内。工具链原生支持这个拆分,已在一个模型上实测过等价性。落地前要先固定两件事:编译参数完整转发,候选产物用摘要关联成一份清单。
2. **修复模式的触发方式**。目前修复任务的输入是确定性流水线的失败现场,而默认已经不跑它了。要改成吃 agent 自己的失败现场,或者对失败的模型显式补跑一次对照来生成现场。这条直接关系到能不能持续产出黄金样本,优先级高。
3. **自述时序的处理**(约 1 小时)。板子断开时 agent 按预算提醒先写了失败、补测后结论变成通过,这种时序差目前会被记成不一致。方案已定:标为待复核而不扣分,复核前不进训练候选。
4. **判定的进一步拆分**。编译成功、输入输出契约、产物一致性目前共用一个字段,要拆开各自存;输入输出校验目前只比形状,要补数据类型和排布。
5. **打分锚点的补充**。原来的相对打分项以默认配置为参照,现在这个参照点按需才有。好消息是强化学习本身是组内比较——同一道题多条采样,模型难度天然抵消,这部分影响不大;但跨题比较和监督微调的绝对门槛还需要一个锚,方案是用 FP32 基线加门槛,必要时对少量代表性模型保留对照。
6. **学生策略接入**。目前全部是教师模型的轨迹,在线强化学习的闭环还没跑过。接入后第一次跑的目标不是涨分,是确认同一道题的多条采样之间确实存在可区分的好坏。
7. **覆盖扩展**。分割和姿态估计的适配在做,按"模型数除以边际成本"排序——这两类复用已有的图像和缓存,边际成本最低。
8. **板卡健康检查与扩容**。目前只看配置里的启用标记,没有连通性探测。加上探测和退避之后,板卡数量直接决定实验周期。
**提示**
- 口吻是"已排期的工作",不是"欠账"。每条带方案和量级,显得可控。
- 第 2 条和第 5 条是 agent 主导那次调整带出来的配套工作,主动列出来,说明这次改动是想清楚了才做的、连副作用都盘过——这比不提要好得多。
- 第 8 条是唯一隐含资源诉求的点。领导接话再给数,不要主动伸手要。
---
## 9. 下一步与阶段验收(2 分钟)
**讲什么**
第二阶段按这个顺序推进,不并行:
1. 补齐修复模式的触发方式和打分锚点——这两条是上次调整的配套,先收尾。
2. 两阶段编译落地。先在一个分类切片上验证参数、产物关联、板端数字一致,再扩到复杂模型和检测模型。
3. 判定拆分和显式重判接口:历史批次按冻结规则判,新批次按新规则判,两套口径不混。
4. 接学生策略,跑通一次在线采样,看同一道题多条采样之间打分有没有区分度。
5. 规模化。
阶段验收我建议用两个信号,都不是通过数:
- **分辨力四例各有一例被正确落位**:做对了、做错了、正确放弃、超预算——尤其是后两类"应当被拒"的例子。这是方法论走通的签收点。
- **可用轨迹比例**:版本合格、没越界、没被误判的轨迹占比。这才是这条流水线的产能指标。
通过数受模型难度、门槛松紧、工具链成熟度共同影响,想让它涨把阈值放松就行——所以它不能当项目指标。
**提示**
- 最后一句要说,但语气放平,说成"这是我们给自己定的纪律",不是"别拿通过率考核我们"。
- 需要对外时间点时可以说:两阶段编译落地后采集效率提升一个数量级,届时给出第一版训练数据集的规模和分辨力指标。
---
## 10. 问答预案
**1. "不跑确定性流水线了,判定会不会变松?"**
不会。精度门判的是"板端实测的任务指标相对 **FP32 基线** 劣化不超过 5%",参照物从来就是 FP32,不是默认配置的量化结果。FP32 基线属于输入准备,照常产出。去掉的只是"默认配置能做到多少"这个参考点,四道门一个没动,板端实测一步没省。
**2. "那怎么知道一个模型是 agent 做不好,还是我们的流水线本身有问题?"**
三条:一是完整流水线保留成显式开关,怀疑的时候单独跑一次对照,几十分钟出结果;二是同一道题有多条独立采样,如果多条以完全相同的方式失败,那更可能是环境问题不是 agent 问题;三是失败有分类,环境类和判断类的错误码是分开的。另外这也正是我们保留这个开关而不是删掉的原因。
**3. "修复模式的输入是流水线的失败现场,不跑了以后怎么产生修复任务?"**
这是这次调整带出来的配套工作,已经排进第二阶段第一项。两个方案:改成吃 agent 自己的失败现场,或者对失败的模型显式补跑一次对照生成现场。倾向前者——agent 自己的失败现场对训练更有价值,因为那才是它真实会遇到的情况。
**4. "什么时候能真正训出模型?"**
数据侧的接口和格式已经备好,轨迹在持续产出。开训的判据不是攒够多少条,而是分辨力四例各有一例被正确落位——因为只产出正样本证明不了验证器有分辨力,用没有分辨力的数据去训,花了算力学到的是测量误差。这四例里还差两类,在第二阶段第一项里。
**5. "通过率是不是偏低?是不是说明模型大多上不了板?"**
不能这么推。失败里混着配置层级的问题、进程终止、评测口径三类,都和模型本身能不能上 BPU 无关,要给这个结论得先把这几类剔干净。现在能确定说的是:通过的那些,精度数字是板端实测、按 5% 相对劣化判的,这部分可以直接作为交付基准。
**6. "agent 到底比默认配置强在哪?"**
有明确的局部证据:一个分类模型默认量化掉 8.9 个百分点,agent 通过调位宽和校准参数做到掉 0.1 个百分点,诊断链条完整——定位误差累积位置、做过对照实验。同时我们也主动给它加了约束:如果延时完全不设限,"整网提到 int16"就是一行代码的通用解,任务会失去难度。所以位宽构成要进打分,对照要做三组:默认、全 int16、agent 方案。
**7. "评测这块投入大吗?"**
不大,而且设计上是一次投入长期复用。板子上只跑张量,读图、预处理、后处理、算指标全在主机侧,所以加一类新任务板端零改动——我们现在补分割和姿态,板端一行没动。板上也不用装各种数据集工具包,接新板成本很低。顺带板端推理比服务器模拟快 45 倍,上真板反而是提速。
**8. "隔离真的安全吗?数据会不会泄漏?"**
内核级隔离加进程隔离加权限清空,红队 11 种路径逃逸全部挡住。唯一藏不住的是大模型的访问凭证——agent 进程自己要读它,同进程的命令行就一定读得到,这是结构性限制,靠定期轮换兜底。另外审计层还有一道:即使读到了,内容指纹会命中,那条轨迹直接判不可用。
**9. "这套东西只能用在量化上吗?"**
骨架和任务无关,可以迁出去,但有三条前提:判据可机器化、验证远比求解便宜、失败模式是有限集。CPU 算子优化那条满足,立项说明已经写了;判据要按新任务重做,这是迁移时最容易犯也最贵的错。
**10. "这件事对外能讲的价值是什么?"**
四样产出:一是后训练出来的模型;二是带打分和失败分类的数据集;三是这套基准和流水线——换一块板、换一类任务,别的组能直接接着用;四是业务数字,端到端延时降低多少、覆盖多少客户模型。按我们做底座的定位,第三样的战略价值不低于第一样:模型会过时,基准不会,而且基准定义了整个行业用什么标准衡量这件事。
---
## 附:数字来源
| 讲稿里的数字 | 来源 | 截面 |
|---|---|---|
| agent 主导、输入准备只做 01–03、完整流水线改显式开关 | `bench/CONTRACT.md` §0 第一条原则 | 09-17 |
| oracle 占板 7.92 h vs rollout 3.83 h、利用率 93%、板端故障统计 | `outputs/plan-verify-20260916/REVIEW.md` | 09-16 15:26 |
| 轨迹计数、板端通过数、端点切换 | `outputs/glm53-endpoint-switch/runtime-after.json` | 09-16 21:03 |
| 0.06% / 3.69% 补测结果 | `outputs/plan-verify-20260916/evidence/backfill_result.jsonl` | 09-16 16:35 |
| 板端 31 s / 8 min、45 倍、2000 张统计依据 | `bench/PLAN.md` §4、`bench/README.md` | 09-15 |
| 107 s : 78 min、两阶段等价性 | `bench/SPEC-two-phase-compile.md` | 09-15 |
| 覆盖总表、姿态判据、入队门槛 | `bench/README.md` 头部、`bench/CONTRACT.md` 头部 | 09-17 |
| 红队 11/11、审计规则、历史数据集作废 | `bench/REPORT-codex.md` | 09-15 |
| 余弦 0.9976 对 0.36、分辨力四例 | `BPU量化轨迹工厂-闭环方法论.html` | 09-15 |
| 三条适用前提、CPU 算子优化落点 | `CPU算子优化Agent-立项说明.html` | 09-15 |
**引用纪律**:旧截面的统计不能当现在的结论,引用前先确认截面时间;通过率不能当"模型能不能上 BPU"的结论。
**文档一致性提醒**:`方法论.html` 和 `闭环方法论.html` 仍是"确定性流水线先出标准答案"那一版设计,与现行的 agent 主导不一致。投屏前先说明,或抽空更新对应段落。

View File

@ -0,0 +1,498 @@
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>把工程任务做成训练数据</title>
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500;600&family=IBM+Plex+Sans:wght@400;500;600;700&family=IBM+Plex+Serif:ital,wght@0,400;0,500;1,400&display=swap">
<style>
:root {
--paper:#EEF2F4; --surface:#FFFFFF; --surface-sunk:#E3EAEE;
--ink:#11181D; --ink-soft:#495861; --ink-faint:#7C8C95;
--rule:#C9D6DC; --rule-soft:#DCE6EA;
--accent:#0B6E8A; --accent-soft:#DCEDF3;
--pass:#2F6B46; --fail:#9E3223; --hold:#8A6516;
--pass-bg:#E0EDE5; --fail-bg:#F6E2DE; --hold-bg:#F5EBD6;
--f-display:"IBM Plex Sans","PingFang SC","Hiragino Sans GB","Microsoft YaHei",system-ui,sans-serif;
--f-body:"IBM Plex Serif","Songti SC","SimSun",Georgia,serif;
--f-mono:"IBM Plex Mono",ui-monospace,"SF Mono",Menlo,monospace;
}
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) {
--paper:#0E1418; --surface:#161E24; --surface-sunk:#0A1014;
--ink:#E4EDF1; --ink-soft:#A3B4BD; --ink-faint:#6E828C;
--rule:#2A3840; --rule-soft:#1F2B32;
--accent:#4FB6D4; --accent-soft:#12303B;
--pass:#6FBE8C; --fail:#E08A78; --hold:#D9B25F;
--pass-bg:#142A1E; --fail-bg:#2E1A15; --hold-bg:#2A2311;
}
}
:root[data-theme="dark"] {
--paper:#0E1418; --surface:#161E24; --surface-sunk:#0A1014;
--ink:#E4EDF1; --ink-soft:#A3B4BD; --ink-faint:#6E828C;
--rule:#2A3840; --rule-soft:#1F2B32;
--accent:#4FB6D4; --accent-soft:#12303B;
--pass:#6FBE8C; --fail:#E08A78; --hold:#D9B25F;
--pass-bg:#142A1E; --fail-bg:#2E1A15; --hold-bg:#2A2311;
}
* { box-sizing: border-box; }
body { background:var(--paper); color:var(--ink); font-family:var(--f-body); font-size:16px; line-height:1.75; -webkit-font-smoothing:antialiased; }
.wrap { max-width:1000px; margin:0 auto; padding-block:56px 110px; padding-inline:24px; display:flex; flex-direction:column; gap:52px; }
.masthead { display:flex; flex-direction:column; gap:16px; }
.eyebrow { font-family:var(--f-mono); font-size:11.5px; letter-spacing:.16em; text-transform:uppercase; color:var(--accent); font-weight:600; }
h1 { font-family:var(--f-display); font-weight:700; font-size:clamp(30px,5vw,44px); line-height:1.12; letter-spacing:-.022em; margin:0; text-wrap:balance; }
.standfirst { font-size:18px; color:var(--ink-soft); max-width:62ch; margin:0; }
.facts { display:grid; grid-template-columns:repeat(auto-fit,minmax(150px,1fr)); border-top:1px solid var(--rule); border-bottom:1px solid var(--rule); margin:0; }
.fact { padding:13px 18px 13px 0; border-right:1px solid var(--rule-soft); }
.fact:last-child { border-right:0; }
.fact dt { font-family:var(--f-mono); font-size:10.5px; letter-spacing:.1em; text-transform:uppercase; color:var(--ink-faint); margin:0 0 3px; }
.fact dd { margin:0; font-family:var(--f-display); font-weight:600; font-size:15px; font-variant-numeric:tabular-nums; }
section { display:flex; flex-direction:column; gap:22px; }
h2 { font-family:var(--f-display); font-weight:600; font-size:23px; letter-spacing:-.012em; margin:0; padding-bottom:10px; border-bottom:2px solid var(--ink); display:flex; align-items:baseline; gap:14px; text-wrap:balance; }
h2 .num { font-family:var(--f-mono); font-size:12px; font-weight:500; color:var(--accent); letter-spacing:.08em; flex:none; }
h3 { font-family:var(--f-display); font-weight:600; font-size:16.5px; margin:0; letter-spacing:-.005em; text-wrap:balance; }
p { margin:0; max-width:70ch; }
.lede { font-size:17px; }
strong { font-weight:600; color:var(--ink); }
em { font-style:italic; color:var(--ink-soft); }
code { font-family:var(--f-mono); font-size:.84em; background:var(--surface-sunk); padding:1px 5px; border-radius:2px; color:var(--ink); overflow-wrap:anywhere; }
.rule-list { display:flex; flex-direction:column; }
.rule { display:grid; grid-template-columns:40px 1fr; gap:18px; padding:20px 0; border-bottom:1px solid var(--rule-soft); align-items:start; }
.rule:first-child { border-top:1px solid var(--rule-soft); }
.rule .n { font-family:var(--f-mono); font-size:15px; font-weight:600; color:var(--accent); padding-top:2px; font-variant-numeric:tabular-nums; }
.rule .b { display:flex; flex-direction:column; gap:9px; min-width:0; }
.rule .b p { font-size:15.5px; color:var(--ink-soft); }
.why { font-size:14.5px; color:var(--ink-soft); border-left:2px solid var(--hold); background:var(--hold-bg); padding:9px 13px; }
.why b { font-family:var(--f-mono); font-size:11px; letter-spacing:.08em; text-transform:uppercase; color:var(--hold); font-weight:600; display:block; margin-bottom:2px; }
.how { font-family:var(--f-mono); font-size:12.5px; color:var(--ink); background:var(--accent-soft); padding:9px 13px; border-left:2px solid var(--accent); line-height:1.65; overflow-wrap:anywhere; }
.flow { background:var(--surface); border:1px solid var(--rule); padding:18px 20px; overflow-x:auto; }
.flow pre { margin:0; font-family:var(--f-mono); font-size:12.5px; line-height:1.85; color:var(--ink-soft); white-space:pre; }
.tablewrap { overflow-x:auto; border:1px solid var(--rule); background:var(--surface); }
table { border-collapse:collapse; width:100%; font-size:14.5px; font-family:var(--f-display); }
th, td { text-align:left; padding:10px 14px; border-bottom:1px solid var(--rule-soft); vertical-align:top; }
thead th { font-family:var(--f-mono); font-size:10.5px; letter-spacing:.1em; text-transform:uppercase; color:var(--ink-faint); font-weight:600; border-bottom:1px solid var(--rule); white-space:nowrap; }
tbody tr:last-child td { border-bottom:0; }
td.num-cell { font-variant-numeric:tabular-nums; text-align:right; white-space:nowrap; }
.pill { font-family:var(--f-mono); font-size:10.5px; font-weight:600; letter-spacing:.06em; text-transform:uppercase; padding:2px 7px; border-radius:2px; white-space:nowrap; display:inline-block; }
.pill.ok { background:var(--pass-bg); color:var(--pass); }
.pill.no { background:var(--fail-bg); color:var(--fail); }
.pill.part { background:var(--hold-bg); color:var(--hold); }
.callout { background:var(--surface); border:1px solid var(--rule); border-top:3px solid var(--accent); padding:20px 22px; display:flex; flex-direction:column; gap:10px; }
.callout h3 { font-size:17.5px; }
.note { font-size:14px; color:var(--ink-faint); max-width:74ch; }
footer { border-top:1px solid var(--rule); padding-top:18px; display:flex; flex-direction:column; gap:6px; }
footer p { font-size:13.5px; color:var(--ink-faint); max-width:76ch; }
@media (max-width:620px) { .rule { grid-template-columns:1fr; gap:8px; } }
</style>
</head>
<body>
<div class="wrap">
<header class="masthead">
<div class="eyebrow">方法论 · BPU 量化轨迹工厂</div>
<h1>把工程任务做成训练数据</h1>
<p class="standfirst">这套东西从头到尾怎么做、每一步要注意什么。四个部分:任务怎么定、资源怎么摆、实验怎么跑、最后交付什么。每一条都是已经做过的,后面附着当初为什么这么定。</p>
<dl class="facts">
<div class="fact"><dt>规模</dt><dd>338 模型 / 10 类</dd></div>
<div class="fact"><dt>已就绪</dt><dd>63(cls 48 · det 15)</dd></div>
<div class="fact"><dt>主 run 已结束</dt><dd>168 oracle</dd></div>
<div class="fact"><dt>目标板</dt><dd>S100P · nash-m</dd></div>
<div class="fact"><dt>数据去向</dt><dd>RLVR · SFT</dd></div>
</dl>
</header>
<section>
<h2><span class="num">§0</span>整条线长什么样</h2>
<p class="lede">一句话:<strong>把 338 个模型的量化上板,逐个变成一道有标准答案的题;让 agent 在看不见答案的情况下做一遍;用同一套代码独立核验它做得对不对;核验结果就是训练信号。</strong></p>
<div class="flow">
<pre>registry/models.csv ──▶ new-run / expand ──▶ jobs(PENDING)
│
bench run(后台调度)──▶ oracle 八阶段:01 fetch → 02 baseline → 03 export
│ → 04 calibrate → 05 compile
│ → 06 hbm → 07 board → 08 eval
│ 四道门:G1 编译 · G2 余弦 · G3 任务指标 · G4 延时
▼
bench collect(后台采集)──▶ 每个 job k 条 rollout:agent 在自己的 workspace 里作答
│ → 独立核验 → verdict.json(reward 的唯一来源)
▼
audit_rollouts.py ──▶ 哪些轨迹干净、能进 RL / SFT
export/to_agentrl.py ──▶ datasets/&lt;dataset_id&gt;/</pre>
</div>
<p>这条线上每一环的职责是硬边界,不能交叉:<strong>出题的不作答,作答的不判分,判分的不看自述。</strong>下面四节就是沿着这条线,把「怎么做、为什么这么做」写下来。</p>
</section>
<section>
<h2><span class="num">§1</span>任务:怎么把一个工程活变成一道能训练的题</h2>
<p>这一节回答的是:什么算一道题、什么算做对了、做不出来怎么办。</p>
<div class="rule-list">
<div class="rule">
<div class="n">1</div>
<div class="b">
<h3>一道题 = 一个冻结的组合,题号既是目录名也是关联键</h3>
<p>一道题叫一个 <strong>job</strong>:任务类型 × 模型 × 芯片 × 后端 × 量化配置,五段拼成 <code>job_id</code>,比如 <code>det--yolov8n--nash-e--ptq--int8</code>。它同时是磁盘上的目录名,和所有表里的关联键。</p>
<div class="why"><b>为什么</b>后面要把轨迹、判定、奖励、数据集四类东西对起来。只要题号是唯一的、不变的,这四样永远能 join 回去;题号一变,历史数据就全断了。同理 run 也有号(<code>r20260914-01</code>),工具链指纹不进题号——两套 hbdk4 塞进一个号里,号本身就会说谎。</div>
</div>
</div>
<div class="rule">
<div class="n">2</div>
<div class="b">
<h3>先由确定性流水线跑出标准答案,再让 agent 做</h3>
<p>每道题先由 harness 自己走一遍八个阶段,产出 <code>oracle/result.json</code>,这就是标准答案。agent 之后做的每一次,都相对它衡量。</p>
<div class="why"><b>为什么</b>同一份标准答案有三个用途,少一个都不行:① <strong>对错的依据</strong>;② <strong>相对基准</strong>——奖励里的精度和延时都算相对 oracle 的差,因为绝对延时是模型自身的属性,不是 agent 的功劳;③ <strong>「做不到」的裁判</strong>——agent 说这道题做不了的时候,看 oracle 是不是在同一阶段以同类错误失败。不需要人判,也不需要另一个模型来判。</div>
</div>
</div>
<div class="rule">
<div class="n">3</div>
<div class="b">
<h3>题的状态和答的状态分开记,永不互相覆盖</h3>
<p>job 的状态由 oracle 决定(<code>PASS / FAIL / UNVERIFIED</code>),rollout 的状态由核验结果决定(<code>PASS / FAIL / BLOCKED / UNVERIFIED</code>)。一道题只有一份标准答案,每次作答各有一份判定。</p>
<div class="why"><b>为什么</b>塞进一个字段就会出数据事故:要么标准答案被某次作答覆盖,要么两种 PASS 混成一个。而且<strong>「oracle 做不出来、agent 做出来了」是最值钱的样本</strong>——它要么是黄金数据,要么说明我们的流水线有 bug。这类自动打 <code>review</code> 标记进人工抽查,查完反过来修 harness,形成回路。</div>
</div>
</div>
<div class="rule">
<div class="n">4</div>
<div class="b">
<h3>判据分成四道门,必选和可选分清楚</h3>
<p>G1 编译(产物在不在、芯片型号和输入格式对不对、日志有没有 ERROR)· G2 数值(量化前后逐输出余弦)· G3 任务指标(同一子集、同一预处理、同一评测器下相对 FP32 的掉点)· G4 板端延时。必选门全过才算 PASS。</p>
<div class="why"><b>为什么</b>门是<strong>绝对判断</strong>——这个模型能不能交付;奖励是<strong>相对判断</strong>——agent 比标准答案做得好还是差。两件事都要,但不能混成一个数。</div>
<div class="how">最容易出事的一条:G1 里的芯片型号不一致直接判死。这是 agent 最容易钻的口子——换个芯片编译当然更容易过。</div>
</div>
</div>
<div class="rule">
<div class="n">5</div>
<div class="b">
<h3>「这轮不评」和「本该评没评成」必须分开记</h3>
<p>门有两种非结果:<code>na</code> 是本轮设计上就不评(比如 3D 占据第一轮只编译不评精度),不阻塞 PASS;<code>unverified</code> 是本该评但没评成(数据缺、板没接),阻塞 PASS。</p>
<div class="why"><b>为什么</b>这是交付文档里「未判定 ≠ 不适用」的直接映射。两者混成一个「没数」,对外就会说成「这个模型不行」,而实际上只是我们没测。</div>
</div>
</div>
<div class="rule">
<div class="n">6</div>
<div class="b">
<h3>失败必须有名字,因为它要同时喂三个地方</h3>
<p>每个失败由正则分类器从 stderr 打一个标签,存进数据库和结果文件。确定性的五类:<code>unsupported_op · shape_infer · export_fail · calib_nan · verifier_mismatch</code>;瞬态的三类:<code>oom · board_offline · stalled</code>;没匹配上的进 <code>unknown</code> 人工队列,规则库随之增长。</p>
<div class="why"><b>为什么要分类</b>不是为了好看,是因为这个标签同时被三处用到:① <strong>决定要不要重试</strong>——确定性的不重试(重试一百次也一样),瞬态的重试;② <strong>喂给 repair 模式的提示词</strong>——修复任务要告诉 agent 现场是什么错;③ <strong>判定「做不到」是真是假</strong>——agent 报 BLOCKED 时,比对 oracle 是不是同阶段同类错误。<strong>没有这三处用途,确实不用分类;有了这三处,不分类就没法做。</strong></div>
</div>
</div>
<div class="rule">
<div class="n">7</div>
<div class="b">
<h3>难度不是事先定的,是 oracle 跑完自己浮出来的</h3>
<p>顺序是:先把所有就绪模型的 oracle 铺一遍 → 得到失败率分布和错误分类分布 → 难度自然分成三层:默认配置就过的、改配置能救的、工具链真做不到的。<strong>采轨迹的重点是中间那层,但这一层只能等 oracle 跑完才知道是谁。</strong></p>
<div class="why"><b>为什么</b>第一轮预期失败率就是高的,尤其局部特征、光流、3D 占据这三类——当初就写明了「这是数据,不是事故」。事先按模型大小或者主观印象去猜哪道题难,猜不准;而 oracle 本来就要全铺一遍(它是 benchmark 交付本身),难度分层是这一遍的<em>副产品</em>,不额外花钱。</div>
</div>
</div>
</div>
</section>
<section>
<h2><span class="num">§2</span>资源:机器、数据、板子怎么摆</h2>
<p>这一节回答的是:算力怎么分、数据从哪来、慢的东西怎么办。</p>
<div class="rule-list">
<div class="rule">
<div class="n">1</div>
<div class="b">
<h3>资源分池,一个并发数不能管所有</h3>
<p>四个池各自设并发:GPU(基线、评测)· CPU(导出、编译)· board(板卡)· agent(受 LLM 端点限速)。池是跨进程的文件锁,调度进程、agent 提交的阶段、多个 <code>bench run</code> 共用同一组槽位。</p>
<div class="why"><b>为什么</b>用一个数字管所有,板端会把 GPU 堵死——板卡是严格串行的,一块板就是一个槽位,而 GPU 阶段本可以并行好几路。</div>
</div>
</div>
<div class="rule">
<div class="n">2</div>
<div class="b">
<h3>板卡按物理身份登记,一块板一把锁,槽位不复用</h3>
<p>每块独立物理板登记唯一槽位、芯片型号、运行库路径和各自的连接信息;容量等于启用的板数。停用的板标记禁用,槽位不回收再用。同一地址、同一物理板的多个容器,不能登记成额外容量。</p>
<div class="why"><b>为什么</b>容量是用来排队的。把同一块板登记两次,排队系统会以为有两块,两个任务同时推上去互相干扰,测出来的延时全废。</div>
</div>
</div>
<div class="rule">
<div class="n">3</div>
<div class="b">
<h3>数据复制入池、设只读、写清单;agent 永远不下载</h3>
<p>数据集从机器上别人的目录<strong>复制</strong>进自己的池子(不是软链),复制完 <code>chmod -R a-w</code>,写 <code>manifest.json</code> 记样本清单的 sha256、数量、来源路径、日期。</p>
<div class="why"><b>为什么复制不软链</b>别人的目录随时可能被清掉。几十 GB 的冗余换来的是零耦合。<b style="margin-top:6px">为什么 agent 不下载</b>三条:可复现(线上数据集版本会飘)、防作弊(agent 会挑好算的子集)、轨迹干净(网络波动不该进训练数据)。<br>另外实测过:官方 COCO 源 26 KB/s,815 MB 要下 9 小时,而同样的东西 HF 是 8 MB/s、阿里云 OSS 是 43.8 MB/s。<strong>照着任务清单里给的官方入口下载是下不动的</strong>,所以每个数据集声明有序多源,逐个试、成功后校验 sha256。</div>
</div>
</div>
<div class="rule">
<div class="n">4</div>
<div class="b">
<h3>数据分三档,用途不同,规模是算出来的</h3>
<div class="tablewrap">
<table>
<thead><tr><th>档</th><th>用途</th><th>规模</th></tr></thead>
<tbody>
<tr><td><strong>T0 calib</strong></td><td>量化校准</td><td>128–512 张,从 train 抽,与验证集零交集</td></tr>
<tr><td><strong>T1 smoke</strong></td><td>日常<em>筛</em></td><td>cls 2000 · det/iseg/pose 1000 · seg 500</td></tr>
<tr><td><strong>T2 full</strong></td><td>交付<em>判</em></td><td>完整验证集,开新 run</td></tr>
</tbody>
</table>
</div>
<div class="why"><b>为什么 cls 是 2000 不是 500</b>这是算出来的,不是拍的:500 张评 Top-1,单次测量的标准误约 1.9 个百分点,根本分辨不出 1 个点的掉点。但 FP32 和量化是<strong>配对</strong>比较——同一批图,误差高度相关,差值的标准误由「结论不一致的那些图」决定,2000 张时约 0.5 个点,够用;500 张时约 1 个点,和容差同量级,会误判。<br><strong>所以 T1 上只能记 <code>pass-screen</code>(筛过了),T2 上才是 <code>pass</code>(判过了)。</strong>对训练信号来说 T1 够用,对交付数字来说必须 T2。</div>
</div>
</div>
<div class="rule">
<div class="n">5</div>
<div class="b">
<h3>没有固定超时,只有看门狗</h3>
<p>判定卡死的条件是两个同时成立:<strong>日志 15 分钟不增长,且整个进程树 CPU 空闲</strong>。24 小时硬上限只是保险,正常永远碰不到。编译走单独的长车道(1 小时)。</p>
<div class="why"><b>为什么</b>不同模型的编译时长差两个数量级——实测 convnext.small 118 分钟、efficientnet.b4 74 分钟、resnet50 61 分钟。固定超时一定会误杀慢模型,而误杀产生的失败样本<strong>不是 agent 的错,却会被当成 agent 的失败学进去</strong>。<br>判据本身也有坑:CPU 采样必须递归覆盖所有后代进程(编译的实际计算在孙子进程里),D 状态(等 I/O)不能算作空闲。</div>
</div>
</div>
<div class="rule">
<div class="n">6</div>
<div class="b">
<h3>重活由 harness 执行,agent 只提交和等待</h3>
<p>agent 不在自己的 bash 里跑编译,而是 <code>bench stage run compile --config workspace/x.yaml</code> 提交,拿一个句柄,再 <code>bench stage wait</code> 等结果。它的 bash 仍然可用,用来看日志、改配置、跑小脚本。</p>
<div class="why"><b>四个好处</b>① 编译时长和 agent 工具的超时脱钩;② 后端和虚拟环境的选择不可能被 agent 选错(两套工具链版本完全不同,禁止把这个判断交给 agent);③ 预算和看门狗对 oracle 与 agent 一视同仁;④ 日志位置统一,回放的时候找得到。</div>
</div>
</div>
<div class="rule">
<div class="n">7</div>
<div class="b">
<h3>便宜的信号用来迭代,贵的信号用来终判</h3>
<p>同一个编译命令里,<strong>量化 107 秒,BPU 编译 78 分钟,比例 1:44</strong>(resnet50 实测日志时间戳)。而 agent 迭代要看的精度信号,在量化结束那一刻就有了。所以把这两步拆开:agent 用量化后的中间产物快速试,确认可行了才付那一次昂贵的 BPU 编译,产出板端权威数字。</p>
<div class="why"><b>实测依据</b>resnet18 两阶段:量化 23 秒 + 编译 73 秒,产出的 .hbm <strong>字节数与一次性编译完全一致</strong>,所以拆分没有正确性风险。<br>另一头:x86 上跑中间产物 40.5 毫秒一张,板端 BPU 0.889 毫秒一张,<strong>板端快 45 倍</strong>——但要用板端就得先付编译的钱。所以分工是:便宜的当快速筛,板端当终判,权威数字只认板端。</div>
</div>
</div>
</div>
</section>
<section>
<h2><span class="num">§3</span>实验方案:怎么跑,怎么保证跑出来的东西可信</h2>
<p>这一节是踩坑最多的一节。可信度不是靠小心,是靠几条机械规则。</p>
<div class="rule-list">
<div class="rule">
<div class="n">1</div>
<div class="b">
<h3>只追加,不覆盖:口径变了就开新 run</h3>
<p>换工具链、改配置、修 bug 之后重跑,一律开新 run,老 run 留着对照。run 的清单文件记录两套虚拟环境 <code>pip freeze</code> 的 sha256,每个任务启动前比对,<strong>不一致直接拒跑</strong>。状态只存在数据库里,文件系统只存内容。</p>
<div class="why"><b>为什么要拒跑而不是告警</b>否则「只追加」是句空话——同一个 run 里混着两套工具链跑出来的结果,事后没有任何办法区分。</div>
</div>
</div>
<div class="rule">
<div class="n">2</div>
<div class="b">
<h3>正在跑的代码 ≠ 磁盘上的代码</h3>
<p>常驻调度进程加载的是<strong>它启动那一刻</strong>的代码和配置。改完 harness 必须让旧进程收尾退出(<code>bench drain</code>,不要 kill 正在编译的进程),再用新代码起新进程。</p>
<div class="why"><b>踩过的坑</b>一个调度进程连续跑了 23 小时多的旧代码,期间生成的编译配置一直是修好之前的写法;磁盘上的文件早就改对了,看文件完全看不出问题。<strong>「改好了」指的是文件,不是正在跑的东西。</strong>缓存也会把旧配置留在内存里。</div>
</div>
</div>
<div class="rule">
<div class="n">3</div>
<div class="b">
<h3>agent 只有一个可写区,隔离必须在内核层</h3>
<p>agent 的工作目录只有 <code>rollouts/&lt;kk&gt;/workspace/</code>,标准答案、评测代码、别的作答、凭证全在外面。隔离靠 mount namespace 把这些东西<strong>从文件系统里摘掉</strong>,加上 PID 命名空间和权限清空;jail 起不来就拒绝启动,不静默降级。</p>
<div class="why"><b>踩过的坑</b>之前那版隔离是解析 agent 写的 shell 命令做白名单,<strong>连续四版被绕过</strong>:换行后的 <code>cd</code>、<code>$IFS</code>、路径里拼 <code>$VAR/</code>、在自己目录里做个软链接指回去。这类防线永远追不平所有写法。改成「让它看不见」之后,这一类 bug 就没有了——看不见的东西,再多绕法也没用。<br>命令解析那层降级成教学层:只负责给一句人话的拒绝理由,省 agent 的预算。<br><b style="margin-top:6px">藏不住的一样东西</b>LLM 凭证。agent 进程自己要读它,同进程的 bash 就一定读得到。这是结构性限制,靠定期轮换 key 兜底,别指望隔离。</div>
</div>
</div>
<div class="rule">
<div class="n">4</div>
<div class="b">
<h3>agent 说的不算数,独立核验才算数</h3>
<p>判定完全不读 agent 的自述。它交的自述只用来和核验结果比对,比对结果单独记一个字段。自述与核验不一致要重罚,而且这类样本绝不进监督训练集。</p>
<div class="why"><b>为什么</b>训练集里出现一次「谎报成功」,就是在教它谎报。没交自述只小额扣分——不报比报错轻,但不能没有代价。<br>还有一条同类的:<strong>编译返回 0 不等于验收通过。</strong>编译日志里的百分比、校准余弦、静态性能估算都不能代替四道门。</div>
</div>
</div>
<div class="rule">
<div class="n">5</div>
<div class="b">
<h3>每条轨迹彼此独立,永不串联</h3>
<p>三层重试,边界是硬的:① harness 对瞬态错误的重试<strong>不算轨迹</strong>(没有 agent 参与);② agent 在一条轨迹里看到报错自己改配置重提,<strong>是同一条轨迹</strong>——这正是我们要采的诊断过程;③ 新的 rollout 是<strong>全新轨迹</strong>,全新上下文,不知道其他作答存在;④ 修复模式只吃 oracle 的失败现场,不吃别人作答的失败。</p>
<div class="why"><b>为什么</b>组内比较是整个训练方法的地基。第 k+1 条看得见第 k 条,组内就不独立了,比较失去意义。所以「读了同一道题别的作答」在审计里直接判不可用,和「读了标准答案」同等对待。</div>
</div>
</div>
<div class="rule">
<div class="n">6</div>
<div class="b">
<h3>失败的轨迹不删,过滤发生在导出那一刻</h3>
<p>磁盘上留什么,和数据集里收什么,是两件事。轨迹全留,导出时按版本号、污染标注和资格函数筛。改了提示词、边界或扩展,就把 agent 版本号 +1,旧版本的轨迹自然被过滤掉,不需要删。</p>
<div class="why"><b>为什么</b>三条:① 轨迹是复盘证据——好几个 bug 都是靠原始轨迹定位的,删了同样的坑要重踩一次才知道;② 数据洁净度由导出时的规则保证,和磁盘上留不留无关;③ 版本号是用来追溯的,它记着这条轨迹是在哪一版规则下跑的。<br>同理,job 和 run 也不删,历史数据集作废也是加作废标记(连同原清单的哈希),不是删除。</div>
</div>
</div>
<div class="rule">
<div class="n">7</div>
<div class="b">
<h3>每条轨迹能不能用,由审计逐条给理由</h3>
<p>审计脚本每条轨迹输出一行,理由是机械判据:读了标准答案、读了别人的作答、被看门狗误杀、旧版本、没交自述、自述与核验不符、去读评判代码或工具链、内容指纹命中、碰了状态库、读了别的题。干净且版本合格的标 <code>RL ✓</code>,再加上核验通过且自述一致才标 <code>SFT ✓</code>。</p>
<div class="why"><b>注意一个反例</b>做内容指纹匹配时,agent 自己合法拿到的阶段结果可能和标准答案数值完全相同(因为本来就该相同),这类不能算泄漏证据,否则正常轨迹会被误报。判泄漏要看<em>来源</em>,不能只看数值撞上了。</div>
</div>
</div>
<div class="rule">
<div class="n">8</div>
<div class="b">
<h3>先打穿一条切片,人肉核对每一个数,再扩量</h3>
<p>实施顺序是:地基(骨架、契约、数据入池、预检全绿)→ <strong>一条 oracle 切片打穿八个阶段,人工核对每一个数字</strong> → 闭环(端到端和修复各跑通一次)→ 横向铺一批压测资源池 → 全量。</p>
<div class="why"><b>为什么人肉核对</b>因为后面所有奖励都是相对 oracle 算的——<strong>oracle 可信是整个计分体系的前提</strong>。这一步省掉,后面采的所有数据都建在沙子上。</div>
</div>
</div>
<div class="rule">
<div class="n">9</div>
<div class="b">
<h3>假定 oracle 自己也可能是错的,留三道防线</h3>
<p>① 四道门是绝对判断,不依赖 oracle;② 相对项全部 clip 在有界区间,单条错值不会把奖励打飞;③ 「oracle 失败而 agent 成功」的样本强制人工抽查,查完反过来修 harness。</p>
<div class="why"><b>为什么</b>harness 的 bug 会系统性地把奖励打歪,而且不容易被发现——它不像崩溃,它安静地产出看起来合理的数字。</div>
</div>
</div>
<div class="rule">
<div class="n">10</div>
<div class="b">
<h3>改判定口径必须同时升版本号并重判</h3>
<p>评测器改了口径就升评测器版本;门的阈值改了就重跑判定(历史任务按它冻结时的阈值重判,新任务用新阈值)。两者都不做,数据库里就会同时存在两套口径的数字,而且分不出来。</p>
<div class="why"><b>踩过的坑</b>某个阈值改了之后没有重判,一批任务就一直按旧阈值挂着失败。<br>还有一类更隐蔽的:<strong>容差的方向和单位写反</strong>。「相对劣化百分比」越小越好,「绝对百分点改善」越大越好,两者符号相反;独立复现过一次——同样是掉 2.5%,容差写 <code>-5</code> 判失败、写 <code>5</code> 判通过。<strong>每条容差必须显式声明 kind,评测器按 kind 计算,不允许省略。</strong>另外百分点和百分比不能相加,统计误差的放宽单位也要跟着换算。</div>
</div>
</div>
<div class="rule">
<div class="n">11</div>
<div class="b">
<h3>指标要和被测对象的结构匹配</h3>
<p>多分支输出不能展平成一个整体余弦。检测模型的输出里同时有框坐标(量级大)和类别分数(0–1),整体余弦会被前者主导。实测过一个:<strong>整体余弦最低 0.9976,而类别分支只有 0.36</strong>。</p>
<div class="why"><b>怎么办</b>凡是多语义输出,逐分支判、逐分支记。这条同时提醒一件事:<em>不要拿一个高余弦就推断精度没问题</em>——数值相似度是过程量,任务指标才是结论。</div>
</div>
</div>
<div class="rule">
<div class="n">12</div>
<div class="b">
<h3>一层只能有一个写者</h3>
<p>代码按层划分归属:流水线与门、agent 配置、导出、清单与阈值、脚本。谁在改哪层要写明,别人对那层只读。改动只在本地副本做,通过同步脚本推上机器;同步脚本带守卫,机器上被人动过就拒绝推送。</p>
<div class="why"><b>踩过的坑</b>两个会话同时改一棵树,互相覆盖过一次;清单文件还丢过两次。机器上的运行数据(runs、state、data、cache)一律不手改——改判走重判命令,补测走补测命令。</div>
</div>
</div>
</div>
</section>
<section>
<h2><span class="num">§4</span>产出:交付什么,怎么算做成了</h2>
<p>这条线上其实同时长出四样东西,四样都是产出,验收方式各不相同。</p>
<div class="tablewrap">
<table>
<thead><tr><th>产出</th><th>是什么</th><th>怎么算做成了</th></tr></thead>
<tbody>
<tr><td><strong>① 量化交付数据</strong></td><td>每个模型能不能上板、掉多少点、板上多快——oracle 这一遍本身就是 benchmark</td><td>四道门有明确判定;T2 上的数字才叫交付数</td></tr>
<tr><td><strong>② 训练数据</strong></td><td>带可核验分数的 agent 轨迹(教师、学生分开落)</td><td>看干净可用的比例,不看通过数</td></tr>
<tr><td><strong>③ 这套流水线本身</strong></td><td>目录契约、四道门、核验与导出——换个任务也能用</td><td>换一类任务不用改环境契约</td></tr>
<tr><td><strong>④ 复盘证据</strong></td><td>每次失败的分类、每条轨迹的回放</td><td>任何结论都能追到原始记录</td></tr>
</tbody>
</table>
</div>
<div class="rule-list">
<div class="rule">
<div class="n">1</div>
<div class="b">
<h3>奖励离线算,纯函数,配方带版本号</h3>
<p>核验结果文件里只存<strong>原始测量值</strong>,奖励在导出阶段离线算。配方是纯函数、带版本号,改了配方可以对历史数据全量重打分,不用重跑任何实验。</p>
<div class="why"><b>为什么</b>配方一定会改好几版。如果奖励在跑的时候就写死进结果里,每改一次配方就要重跑一次实验——那是几个星期的机器时间。</div>
</div>
</div>
<div class="rule">
<div class="n">2</div>
<div class="b">
<h3>导出资格分两级:强化学习收正负例,监督训练只收做对且诚实的</h3>
<p>默认保留全部已结束的终态(成功、精度不达标、预算耗尽分别标注)。<strong>能不能进监督集是一层筛选,不代表数据总量。</strong>自述与核验不一致、需要人工复核这两条,是优先级最高的硬排除。</p>
<div class="why"><b>注意</b>预算耗尽不等于失败。已经产出并核验过的门结果照常记账,不能因为超预算把整条成果清零——否则一条做得很好但没来得及收尾的轨迹,会被记成彻底失败。</div>
</div>
</div>
<div class="rule">
<div class="n">3</div>
<div class="b">
<h3>每个数字都要带着证据边界一起流转</h3>
<p>必须显式区分的几组:<code>pass-screen</code>(筛子上过了)≠ <code>pass</code>(完整验证过);工具的静态性能估算 ≠ 板端实测;服务器上的模拟运行 ≠ 物理板端运行;用随机输入测的延时 ≠ 用真实数据集测的精度。</p>
<div class="why"><b>为什么</b>弱证据会在传递过程中逐级升格成强结论——第一手的人知道那是筛出来的,传到第三个人就变成「这个模型通过了」。这条不产出代码,但它决定这批数据将来能不能被别人信。</div>
</div>
</div>
<div class="rule">
<div class="n">4</div>
<div class="b">
<h3>验收看分辨力,不看通过数</h3>
<p>通过数受模型难度、门槛松紧、工具链成熟度共同影响——想让它涨,把阈值放松就行。真正的验收是:给这套系统四类终态(做对了、做错了、正确放弃、超预算),它能不能<strong>把四类都正确落位,包括把该拒的拒掉</strong>。</p>
<div class="why"><b>为什么</b>一个只会盖章通过的验证器,和一个真能分辨好坏的验证器,<em>从成功样本上看是一模一样的</em>。所以只产出正样本,证明不了任何事。</div>
</div>
</div>
<div class="rule">
<div class="n">5</div>
<div class="b">
<h3>教师和学生是两种数据,分开采、分开落</h3>
<p>教师模型的轨迹加核验筛选,用于监督训练和拒绝采样;学生模型(自己的模型)的轨迹用于在线强化学习。提示词模板相同,在矩阵里按策略维度分开采,导出时落到不同数据集。</p>
<div class="why"><b>还有一条</b>真正做在线强化学习时,需要采样时的对数概率,而现在这套 agent 进程只有对话记录。到那一步,驱动层会换成训练框架自己的循环。<strong>这套方案的持久资产是环境契约——目录结构、四道门、核验文件格式——不是当前这个 agent 进程。</strong></div>
</div>
</div>
</div>
</section>
<section>
<h2><span class="num">§5</span>这套方法论现在走到哪一步</h2>
<p>截至 2026-09-16 的核验快照,用来说明这套方法论走到了哪一步,也用来提醒哪些数字是旧口径。</p>
<div class="tablewrap">
<table>
<thead><tr><th>环节</th><th>状态</th><th>说明</th></tr></thead>
<tbody>
<tr><td>目录契约与命名</td><td><span class="pill ok">已落地</span></td><td>job / run / rollout / 阶段 / 数据集五套编号都在用</td></tr>
<tr><td>oracle 八阶段</td><td><span class="pill ok">已落地</span></td><td>主 run 已结束 168 个(3 通过 / 165 失败),另有在跑和排队的</td></tr>
<tr><td>四道门与核验</td><td><span class="pill part">在修口径</span></td><td>容差方向与单位、多分支指标、一致性检查的拆分正在改</td></tr>
<tr><td>内核级隔离</td><td><span class="pill ok">已落地</span></td><td>外部执行、只读产物、票据校验均已实现并有回归</td></tr>
<tr><td>轨迹审计与导出</td><td><span class="pill ok">已落地</span></td><td>按版本、污染标注、资格函数三重过滤;旧数据集已作废留档</td></tr>
<tr><td>板端终判</td><td><span class="pill part">接线中</span></td><td>板卡可登录;37 份 nash-m 产物中 9 个有板端性能;板端精度路径尚未接上</td></tr>
<tr><td>两阶段编译</td><td><span class="pill part">计划内 · 待改</span></td><td>方案已验证等价(字节一致),代码仍走一次性完整编译</td></tr>
<tr><td>修复模式 / 学生策略</td><td><span class="pill part">计划内 · 待启用</span></td><td>数据库里全部是教师的端到端轨迹</td></tr>
<tr><td>任务覆盖</td><td><span class="pill part">63 / 338</span></td><td>其余 257 个需要导出适配;评测器目前只有分类和检测两种</td></tr>
</tbody>
</table>
</div>
<div class="callout">
<h3>引用数字时的两条纪律</h3>
<p><strong>一、旧截面的统计不能当现在的结论。</strong>之前复盘里的 143 / 83 / 60 / 38 都是更早的截面和口径,现在的数是 168 个已结束 oracle。引用前先确认截面时间。</p>
<p><strong>二、通过率不能当作「模型本身能不能上 BPU」的结论。</strong>这批失败里混着配置层级的错误、看门狗终止、评测口径问题——都和模型本身能不能上板无关。要给这个结论,得先把这几类剔干净。</p>
</div>
</section>
<footer>
<p><strong>来源</strong>:本文的每一条都出自 <code>bench/</code> 已有的文档与实现——<code>CONTRACT.md</code>(契约 v0.4 落地版)、<code>HANDBOOK.md</code>(掌控手册)、<code>README.md</code>、<code>SPEC-two-phase-compile.md</code>(两阶段编译实测)、<code>REPORT-codex.md</code>(v5 隔离与导出修复报告)、《BPU 量化轨迹工厂 设计方案 v0.4》,以及 2026-09-16 的两份核验(方案审查、S100P 现状核验)。</p>
<p><strong>证据边界</strong>:§5 的状态是 2026-09-16 01:17–01:48 的远端快照,不是实时;本文没有重跑任何编译、推理或评测。文中实测数字(编译耗时比、两阶段字节一致、推理速度、外网带宽、统计标准误)均引自上述文档中已记录的实测,未重新验证。</p>
</footer>
</div>
</body>
</html>

View File

@ -0,0 +1,790 @@
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>html{color-scheme:light dark}body{margin:0}img{max-width:100%}[hidden]{display:none!important}</style>
<title>BPU 量化轨迹工厂</title>
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500;600&family=IBM+Plex+Sans:wght@400;500;600;700&family=IBM+Plex+Serif:ital,wght@0,400;0,500;1,400&display=swap">
<style>
:root {
--paper: #EEF2F4;
--surface: #FFFFFF;
--surface-sunk: #E3EAEE;
--ink: #11181D;
--ink-soft: #495861;
--ink-faint: #7C8C95;
--rule: #C9D6DC;
--rule-soft: #DCE6EA;
--accent: #0B6E8A;
--accent-soft: #DCEDF3;
--pass: #2F6B46;
--fail: #9E3223;
--hold: #8A6516;
--pass-bg: #E0EDE5;
--fail-bg: #F6E2DE;
--hold-bg: #F5EBD6;
--f-display: "IBM Plex Sans", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", system-ui, sans-serif;
--f-body: "IBM Plex Serif", "Songti SC", "SimSun", Georgia, serif;
--f-mono: "IBM Plex Mono", ui-monospace, "SF Mono", Menlo, monospace;
}
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) {
--paper: #0E1418; --surface: #161E24; --surface-sunk: #0A1014;
--ink: #E4EDF1; --ink-soft: #A3B4BD; --ink-faint: #6E828C;
--rule: #2A3840; --rule-soft: #1F2B32;
--accent: #4FB6D4; --accent-soft: #12303B;
--pass: #6FBE8C; --fail: #E08A78; --hold: #D9B25F;
--pass-bg: #142A1E; --fail-bg: #2E1A15; --hold-bg: #2A2311;
}
}
:root[data-theme="dark"] {
--paper: #0E1418; --surface: #161E24; --surface-sunk: #0A1014;
--ink: #E4EDF1; --ink-soft: #A3B4BD; --ink-faint: #6E828C;
--rule: #2A3840; --rule-soft: #1F2B32;
--accent: #4FB6D4; --accent-soft: #12303B;
--pass: #6FBE8C; --fail: #E08A78; --hold: #D9B25F;
--pass-bg: #142A1E; --fail-bg: #2E1A15; --hold-bg: #2A2311;
}
* { box-sizing: border-box; }
body { background: var(--paper); color: var(--ink); font-family: var(--f-body); font-size: 16px; line-height: 1.72; -webkit-font-smoothing: antialiased; }
.wrap { max-width: 1000px; margin: 0 auto; padding: 56px 28px 120px; display: flex; flex-direction: column; gap: 52px; }
.masthead { display: flex; flex-direction: column; gap: 18px; }
.eyebrow { font-family: var(--f-mono); font-size: 11.5px; letter-spacing: .16em; text-transform: uppercase; color: var(--accent); font-weight: 600; }
h1 { font-family: var(--f-display); font-weight: 700; font-size: clamp(30px, 5vw, 46px); line-height: 1.12; letter-spacing: -.022em; margin: 0; text-wrap: balance; }
.standfirst { font-size: 18px; color: var(--ink-soft); max-width: 62ch; margin: 0; }
.facts { display: grid; grid-template-columns: repeat(auto-fit, minmax(148px, 1fr)); gap: 0; border-top: 1px solid var(--rule); border-bottom: 1px solid var(--rule); margin: 0; }
.fact { padding: 14px 18px 14px 0; border-right: 1px solid var(--rule-soft); }
.fact:last-child { border-right: 0; }
.fact dt { font-family: var(--f-mono); font-size: 10.5px; letter-spacing: .1em; text-transform: uppercase; color: var(--ink-faint); margin: 0 0 3px; }
.fact dd { margin: 0; font-family: var(--f-display); font-weight: 600; font-size: 15px; font-variant-numeric: tabular-nums; }
section { display: flex; flex-direction: column; gap: 20px; }
h2 { font-family: var(--f-display); font-weight: 600; font-size: 23px; letter-spacing: -.012em; margin: 0; padding-bottom: 10px; border-bottom: 2px solid var(--ink); display: flex; align-items: baseline; gap: 14px; text-wrap: balance; }
h2 .num { font-family: var(--f-mono); font-size: 12px; font-weight: 500; color: var(--accent); letter-spacing: .08em; flex: none; }
h3 { font-family: var(--f-display); font-weight: 600; font-size: 16px; margin: 0; letter-spacing: -.005em; }
p { margin: 0; max-width: 68ch; }
.lede { font-size: 17px; }
strong { font-weight: 600; color: var(--ink); }
em { font-style: italic; color: var(--ink-soft); }
code { font-family: var(--f-mono); font-size: .86em; background: var(--surface-sunk); padding: 1px 5px; border-radius: 2px; color: var(--ink); }
a { color: var(--accent); text-decoration-thickness: 1px; text-underline-offset: 2px; }
a:focus-visible { outline: 2px solid var(--accent); outline-offset: 3px; }
.principles { display: flex; flex-direction: column; }
.principle { display: grid; grid-template-columns: 34px 1fr; gap: 16px; padding: 18px 0; border-bottom: 1px solid var(--rule-soft); align-items: start; }
.principle:first-child { border-top: 1px solid var(--rule-soft); }
.principle .mark { font-family: var(--f-mono); font-size: 12px; font-weight: 600; color: var(--accent); padding-top: 4px; }
.principle h3 { margin-bottom: 5px; }
.principle p { color: var(--ink-soft); font-size: 15.5px; }
.stack { display: flex; flex-direction: column; gap: 8px; }
.layer { display: grid; grid-template-columns: 58px 1fr; background: var(--surface); border: 1px solid var(--rule); border-left: 3px solid var(--accent); }
.layer.is-agent { border-left-color: var(--hold); }
.layer .tag { font-family: var(--f-mono); font-size: 12px; font-weight: 600; color: var(--accent); padding: 14px 0 14px 14px; letter-spacing: .04em; }
.layer.is-agent .tag { color: var(--hold); }
.layer .lbody { padding: 13px 16px 15px 4px; display: flex; flex-direction: column; gap: 5px; min-width: 0; }
.layer .lbody h3 { font-size: 15.5px; }
.layer .lbody p { font-size: 14.5px; color: var(--ink-soft); line-height: 1.6; }
.layer .lbody .meta { font-family: var(--f-mono); font-size: 11.5px; color: var(--ink-faint); line-height: 1.55; overflow-x: auto; white-space: nowrap; }
.code { background: var(--surface-sunk); border: 1px solid var(--rule-soft); border-radius: 3px; padding: 16px 18px; overflow-x: auto; margin: 0; }
.code pre { margin: 0; font-family: var(--f-mono); font-size: 12.5px; line-height: 1.68; white-space: pre; color: var(--ink); }
.code .c { color: var(--ink-faint); }
.code .k { color: var(--accent); font-weight: 500; }
.code .n { color: var(--fail); }
.code .g { color: var(--pass); }
.code figcaption { font-family: var(--f-mono); font-size: 10.5px; letter-spacing: .1em; text-transform: uppercase; color: var(--ink-faint); margin-bottom: 10px; }
.fsm { width: 100%; height: auto; display: block; min-width: 760px; }
.fsm text { font-family: var(--f-mono); font-size: 11px; fill: var(--ink); }
.fsm text.small { font-size: 9.5px; fill: var(--ink-faint); }
.fsm text.lane { font-size: 10px; fill: var(--accent); font-weight: 600; letter-spacing: .08em; }
.fsm .box { fill: var(--surface); stroke: var(--rule); stroke-width: 1; }
.fsm .box-pass { fill: var(--pass-bg); stroke: var(--pass); }
.fsm .box-fail { fill: var(--fail-bg); stroke: var(--fail); }
.fsm .box-hold { fill: var(--hold-bg); stroke: var(--hold); }
.fsm .edge { stroke: var(--ink-faint); stroke-width: 1.2; fill: none; }
.fsm .edge-em { stroke: var(--accent); stroke-width: 1.6; fill: none; }
.fsm .lane-rule { stroke: var(--rule-soft); stroke-width: 1; stroke-dasharray: 3 4; }
.tablewrap { overflow-x: auto; border: 1px solid var(--rule); border-radius: 3px; background: var(--surface); }
table { border-collapse: collapse; width: 100%; font-size: 14px; min-width: 560px; }
th { text-align: left; font-family: var(--f-mono); font-size: 10.5px; letter-spacing: .1em; text-transform: uppercase; color: var(--ink-faint); font-weight: 600; padding: 11px 14px; border-bottom: 1px solid var(--rule); white-space: nowrap; }
td { padding: 11px 14px; border-bottom: 1px solid var(--rule-soft); vertical-align: top; color: var(--ink-soft); font-variant-numeric: tabular-nums; }
tr:last-child td { border-bottom: 0; }
td:first-child { color: var(--ink); font-family: var(--f-mono); font-size: 12.5px; white-space: nowrap; }
table.gates td:nth-child(2) { color: var(--ink); }
table.gates td:nth-child(3) { font-family: var(--f-mono); font-size: 12.5px; white-space: nowrap; color: var(--ink); }
.note { border-left: 3px solid var(--accent); background: var(--accent-soft); padding: 14px 18px; display: flex; flex-direction: column; gap: 6px; }
.note.warn { border-left-color: var(--fail); background: var(--fail-bg); }
.note.fix { border-left-color: var(--hold); background: var(--hold-bg); }
.note .label { font-family: var(--f-mono); font-size: 10.5px; letter-spacing: .1em; text-transform: uppercase; font-weight: 600; color: var(--accent); }
.note.warn .label { color: var(--fail); }
.note.fix .label { color: var(--hold); }
.note p { font-size: 15px; max-width: none; }
.items { display: flex; flex-direction: column; }
.item { display: flex; flex-direction: column; gap: 4px; padding: 16px 0; border-bottom: 1px solid var(--rule-soft); }
.item:first-child { border-top: 1px solid var(--rule-soft); }
.item .head { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; }
.pill { font-family: var(--f-mono); font-size: 10px; letter-spacing: .08em; text-transform: uppercase; font-weight: 600; padding: 2px 7px; border-radius: 2px; white-space: nowrap; }
.pill.block { background: var(--fail-bg); color: var(--fail); }
.pill.dec { background: var(--hold-bg); color: var(--hold); }
.pill.ok { background: var(--pass-bg); color: var(--pass); }
.item p { font-size: 15px; color: var(--ink-soft); }
.phases { display: grid; grid-template-columns: repeat(auto-fit, minmax(210px, 1fr)); gap: 1px; background: var(--rule); border: 1px solid var(--rule); }
.phase { background: var(--surface); padding: 16px 16px 18px; display: flex; flex-direction: column; gap: 6px; }
.phase .n { font-family: var(--f-mono); font-size: 11px; font-weight: 600; color: var(--accent); letter-spacing: .08em; }
.phase h3 { font-size: 15px; }
.phase p { font-size: 14px; color: var(--ink-soft); line-height: 1.6; }
.qa { display: flex; flex-direction: column; }
.qa > div { display: grid; grid-template-columns: 26px 1fr; gap: 14px; padding: 14px 0; border-bottom: 1px solid var(--rule-soft); }
.qa > div:first-child { border-top: 1px solid var(--rule-soft); }
.qa .q { font-family: var(--f-mono); font-size: 12px; font-weight: 600; color: var(--accent); padding-top: 3px; }
.qa h3 { font-size: 15.5px; margin-bottom: 4px; }
.qa p { font-size: 15px; color: var(--ink-soft); }
footer { border-top: 1px solid var(--rule); padding-top: 20px; font-family: var(--f-mono); font-size: 11.5px; color: var(--ink-faint); line-height: 1.8; }
@media (max-width: 640px) {
.wrap { padding: 36px 18px 80px; gap: 40px; }
.fact { border-right: 0; border-bottom: 1px solid var(--rule-soft); }
.layer { grid-template-columns: 46px 1fr; }
}
@media (prefers-reduced-motion: reduce) { * { animation: none !important; transition: none !important; } }
</style>
</head>
<body>
<div class="wrap">
<header class="masthead">
<div class="eyebrow">model-compile-hbdk / bench · 设计方案 v0.4</div>
<h1>BPU 量化轨迹工厂</h1>
<p class="standfirst">确定性 harness 作为 oracle 跑完 338 个视觉模型的量化上板,agent 在受控子集上端到端执行,独立核验给出可验证奖励。一个 job 一个目录,agent 只有一个可写区,产物零散落。</p>
<dl class="facts">
<div class="fact"><dt>模型</dt><dd>338 / 10 类</dd></div>
<div class="fact"><dt>优先级</dt><dd>A 166 · B 128 · C 44</dd></div>
<div class="fact"><dt>目标板</dt><dd>s100 · s100p · s600</dd></div>
<div class="fact"><dt>march</dt><dd>nash-e · m · p</dd></div>
<div class="fact"><dt>后端</dt><dd>PTQ + QAT</dd></div>
<div class="fact"><dt>训练目标</dt><dd>RLVR</dd></div>
</dl>
</header>
<section>
<h2><span class="num">§0</span>v0.4 改了什么</h2>
<p class="lede">v0.3 → v0.4 的四处实质变化,全部来自机器实测或对 v0.3 的追问:</p>
<div class="qa">
<div><div class="q">A</div><div><h3>验证集层重写:大部分数据集已在 /mnt/data 上,外网带宽分化极大</h3><p>ImageNet val、COCO 2017、Cityscapes、KITTI、ETH3D、SceneFlow、nuScenes 都在别的用户目录里,world-readable。官方 COCO 源实测 26 KB/s 等于不通,HF 8 MB/s、OSS 44 MB/s。数据分 T0 calib / T1 smoke / T2 full 三档,job 默认只跑 T1。§7</p></div></div>
<div><div class="q">B</div><div><h3>固定超时废除,改看门狗;重编译一律由 harness 执行</h3><p>不同模型编译时长差两个数量级,固定超时会误杀。现在只在「日志不增长且 CPU 空闲 15 分钟」时判 STALLED。agent 通过 <code>bench stage run</code> 提交编译、<code>bench stage wait</code> 等结果,不在 bash 里直接跑 hb_compile。§8</p></div></div>
<div><div class="q">C</div><div><h3>明确「失败之后会怎样」与「重试在哪条轨迹里」</h3><p>错误分类表、<code>bench status</code> 可见性、三层重试的轨迹边界(oracle 瞬态重试不是轨迹;rollout 内自愈是同一条;新 rollout 是全新轨迹,永不串联)。§9</p></div></div>
<div><div class="q">D</div><div><h3>3D 占据 23 个模型第一轮只编译不评精度的具体协议</h3><p>G3 显式记 <code>na</code> 而非 <code>unverified</code>,reward 重归一化,校准输入来源分级。§6 末尾</p></div></div>
</div>
<p>v0.3 的变化(harness 改 oracle、claim/verdict 分离、四道门、stage 分层)保留,不再重复。</p>
</section>
<section>
<h2><span class="num">§1</span>三条不可违反的原则</h2>
<div class="principles">
<div class="principle">
<div class="mark">01</div>
<div>
<h3>harness 是 oracle,不是 agent 的替身</h3>
<p>每个 job 先由确定性流水线跑出 ground truth:能不能编译、精度掉多少、板端多快。这份结果既是 benchmark 数据,也是给 agent 打分的参照系。agent 的表现永远相对 oracle 衡量——绝对延时是模型的属性,不是 agent 的功劳。</p>
</div>
</div>
<div class="principle">
<div class="mark">02</div>
<div>
<h3>agent 只有一个可写区,一切可信输入只读</h3>
<p><code>job.json</code>、评测代码、校准集、验证集、FP32 基线全部在 agent 可写区之外。agent 写 <code>workspace/</code>,越界写入由 extension 拒绝。它说什么不算数,harness 用干净代码重新验它产出的 <code>.hbm</code> 才算数。</p>
</div>
</div>
<div class="principle">
<div class="mark">03</div>
<div>
<h3>Append-only:重跑即新 run,永不原地覆盖</h3>
<p><code>run_id</code> 冻结。换工具链、改配置、修 bug 后重跑,都开新 run。manifest 记录两套 venv 的 pip freeze 哈希,job 启动前比对,不一致拒跑——否则"append-only"是空话。状态只在 <code>jobs.sqlite</code>,文件系统只存内容。</p>
</div>
</div>
</div>
</section>
<section>
<h2><span class="num">§2</span>五层架构</h2>
<div class="stack">
<div class="layer">
<div class="tag">L0</div>
<div class="lbody">
<h3>清单、矩阵与门限</h3>
<p><code>models.csv</code>(338 行,不可变)、<code>matrix.yaml</code>(march × backend × qcfg × mode × policy 的组合与裁剪)、<code>gates.yaml</code>(逐任务通过容差)、<code>budgets.yaml</code>(逐任务类超时)。展开后写入 <code>jobs.sqlite</code>。</p>
<div class="meta">mode: oracle | e2e | repair · policy: teacher | student</div>
</div>
</div>
<div class="layer">
<div class="tag">L1</div>
<div class="lbody">
<h3>oracle harness(无 agent)</h3>
<p>model-level 阶段跑一次进缓存;target-level 阶段每 job 一遍。每阶段独立子目录,落 stdout / stderr / exit code / meta.json / 耗时。产出 <code>oracle/result.json</code>。</p>
<div class="meta">model-level: fetch → baseline → export | target-level: calibrate → compile → hbm → board → eval</div>
</div>
</div>
<div class="layer is-agent">
<div class="tag">L2</div>
<div class="lbody">
<h3>policy driver(pi)</h3>
<p>e2e 模式:给自然语言任务、job.json、只读数据路径,agent 自己走完全流程。repair 模式:给 oracle 的失败现场。每次尝试一个 rollout 目录,K 次独立 rollout 构成 GRPO 的组。</p>
<div class="meta">pi --mode json -a --session-dir &lt;rollout&gt;/agent "&lt;task&gt;" &gt; events.jsonl</div>
</div>
</div>
<div class="layer">
<div class="tag">L3</div>
<div class="lbody">
<h3>四道门与 verdict</h3>
<p>对 agent 产出的 <code>.hbm</code>,用与 oracle 完全相同的只读评测代码重跑 G1–G4,写 <code>verdict.json</code>。与 <code>claim.json</code> 比对,记录是否一致。</p>
<div class="meta">G1 compile · G2 numeric · G3 task · G4 perf(optional)</div>
</div>
</div>
<div class="layer">
<div class="tag">L4</div>
<div class="lbody">
<h3>数据物化</h3>
<p>扫描 rollouts,脱敏、清洗、线性化轨迹,与 verdict 和 oracle 配对,按 reward 配方打分,输出到 <code>datasets/&lt;dataset_id&gt;/</code>。reward 配方版本化,可全量重打分。</p>
<div class="meta">→ (prompt, trajectory, verdict, oracle_ref, reward) · 兼容 DemyAgent SFT</div>
</div>
</div>
</div>
</section>
<section>
<h2><span class="num">§3</span>目录契约</h2>
<p>所有新增物只进 <code>bench/</code>,<code>.horizon/</code> 只读。与 v0.2 相比新增 <code>cache/</code>、<code>data/</code>,job 内拆 <code>oracle/</code> 与 <code>rollouts/</code>。</p>
<figure class="code">
<figcaption>/mnt/data/ming01.wang/model-compile-hbdk</figcaption>
<pre><span class="c">├── AGENTS.md # 已有 · pi 向上遍历时加载</span>
<span class="c">├── .horizon/ # 已有 · <span class="n">只读</span></span>
<span class="c">├── .agents/skills -> .horizon/skills # 新增软链 · pi 的 skill 发现路径</span>
└── <span class="k">bench/</span>
├── CONTRACT.md <span class="c"># 本文档落地版</span>
├── config.yaml <span class="c"># 资源池并发、端点、板卡地址</span>
├── registry/ <span class="c"># 输入 · 不可变 · git 管理</span>
│ ├── models.csv
│ ├── matrix.yaml <span class="c"># march × backend × qcfg × mode × policy</span>
│ ├── gates.yaml <span class="c"># 逐任务通过容差(§6)</span>
│ ├── budgets.yaml <span class="c"># 逐任务类估算与看门狗参数(§8)</span>
│ ├── datasets.yaml <span class="c"># 每个数据集的有序来源、sha、T1 子集 seed(§7)</span>
│ └── boards.yaml <span class="c"># s100→nash-e · s100p→nash-m · s600→nash-p</span>
├── <span class="k">data/</span> <span class="c"># <span class="n">只读</span> · 从本机其他目录复制入池 · agent 永不下载(§7)</span>
│ ├── pool/&lt;dataset&gt;/&lt;version&gt;/ <span class="c"># 全量 + manifest.json(sha, 数量, 来源路径)</span>
│ ├── t1/&lt;dataset&gt;/&lt;seed&gt;.tar <span class="c"># smoke 子集打包,避开 JuiceFS 小文件</span>
│ └── calib/&lt;task&gt;/&lt;input_spec&gt;/ <span class="c"># 固定 seed 从 train 抽的 npy,与 val 零交集</span>
├── <span class="k">cache/</span> <span class="c"># model-level 产物 · content-addressed · 只读</span>
│ ├── weights/&lt;sha256&gt;/
│ ├── onnx/&lt;sha256&gt;/
│ └── baseline/&lt;model&gt;--&lt;dataset&gt;/ <span class="c"># FP32 指标 + 逐输出张量样本</span>
├── harness/ <span class="c"># L1 + L3 代码</span>
│ ├── pipeline.py scheduler.py
│ ├── stages/ backends/{ptq,qat}/ adapters/
│ └── eval/ <span class="c"># 逐任务评测器 · agent 不可写 · verdict 唯一依据</span>
├── agent/ <span class="c"># L2 pi 配置</span>
│ ├── SYSTEM.md settings.json <span class="c"># compaction 关闭</span>
│ ├── prompts/e2e.md repair.md <span class="c"># 任务描述模板</span>
│ └── extensions/ <span class="c"># path-guard · claim-card · oe-doc(CLI) · budget-kill</span>
├── export/ <span class="c"># L4 · 规则移植自 data_collect,<span class="n">不 import</span></span>
│ ├── to_agentrl.py redact.py reward.py
├── state/jobs.sqlite <span class="c"># 状态、error_class、耗时历史的唯一真相</span>
├── reports/ <span class="c"># bench report 生成的静态看板,按 run_id</span>
├── <span class="k">datasets/</span>ds20260914-01/ <span class="c"># 一次导出一个版本 · 记录覆盖的 run 与 reward 配方版本</span>
└── <span class="k">runs/</span>r20260914-01/
├── manifest.json <span class="c"># 两套 venv 的 pip freeze 哈希 · 板卡固件 · 数据 manifest sha</span>
└── <span class="k">det--yolov8n--nash-e--ptq--int8/</span>
├── job.json <span class="c"># <span class="n">只读</span> · 输入契约 · 含数据路径与 gate 阈值快照</span>
├── <span class="k">oracle/</span> <span class="c"># harness 确定性一遍 → ground truth</span>
│ ├── stages/04-calibrate/ 05-compile/ 06-hbm/ 07-board/ 08-eval/
│ ├── artifacts/ <span class="c"># .bc .hbm · 命名由 job_id 决定</span>
│ └── result.json <span class="c"># 四道门结果 + 原始测量值</span>
└── <span class="k">rollouts/</span>00/ 01/ … <span class="c"># 每次 agent 尝试一个目录</span>
├── <span class="g">workspace/</span> <span class="c"># <span class="g">agent 唯一可写区</span></span>
├── agent/session.jsonl events.jsonl
├── claim.json <span class="c"># agent 自报 · 不可信</span>
└── verdict.json <span class="c"># L3 独立核验 · reward 唯一来源</span></pre>
</figure>
<div class="note warn">
<div class="label">两个 venv 必须隔离</div>
<p>PTQ 用 <code>~/toolchains/oe-3.7.0/venv</code>(hbdk4 <strong>4.7.5</strong> + horizon_tc_ui 3.5.3 + hmct 2.6.5),QAT 用 <code>~/toolchains/oellm2/venv</code>(hbdk4 <strong>4.12.3a2.dev</strong> + horizon_plugin_pytorch 3.3.5)。harness 按 backend 选解释器,<strong>禁止把 backend 判断交给 agent</strong>。板端 UCP 3.15.2 可加载两套产物。</p>
</div>
</section>
<section>
<h2><span class="num">§4</span>命名规范</h2>
<div class="tablewrap">
<table>
<thead><tr><th>标识</th><th>格式</th><th>示例</th><th>说明</th></tr></thead>
<tbody>
<tr><td>run_id</td><td>r&lt;YYYYMMDD&gt;-&lt;seq&gt;</td><td>r20260914-01</td><td>工具链指纹在 manifest.json,不进 id——两套 hbdk4 塞一个会说谎</td></tr>
<tr><td>job_id</td><td>&lt;task&gt;--&lt;model&gt;--&lt;march&gt;--&lt;backend&gt;--&lt;qcfg&gt;</td><td>det--yolov8n--nash-e--ptq--int8</td><td>五段全小写,段内 <code>a-z0-9.</code>,段间双横线。既是目录名也是 join key</td></tr>
<tr><td>task 前缀</td><td>10 类固定缩写</td><td>cls det seg iseg pose mde stereo flow feat occ</td><td>对应交付文档 T01–T10,不允许新增</td></tr>
<tr><td>march</td><td>板型映射</td><td>s100→nash-e · s100p→nash-m · s600→nash-p</td><td>写死在 <code>boards.yaml</code>;默认 qcfg:e/m 走 int8,p 走 fp16+int8</td></tr>
<tr><td>rollout</td><td>&lt;job_id&gt;/rollouts/&lt;kk&gt;</td><td>…/rollouts/03</td><td>两位序号,K 次独立 rollout 构成一组</td></tr>
<tr><td>artifact</td><td>&lt;job_id&gt;.&lt;ext&gt;</td><td>det--yolov8n--nash-e--ptq--int8.hbm</td><td>单独拿出来也不撞名、不丢身份</td></tr>
<tr><td>stage 目录</td><td>&lt;NN&gt;-&lt;name&gt;</td><td>05-compile</td><td>01–03 是 model-level(缓存),04–08 是 target-level</td></tr>
<tr><td>dataset_id</td><td>ds&lt;YYYYMMDD&gt;-&lt;seq&gt;</td><td>ds20260914-01</td><td>记录覆盖的 run 列表与 reward 配方版本</td></tr>
</tbody>
</table>
</div>
</section>
<section>
<h2><span class="num">§5</span>状态机:job 与 rollout 分开</h2>
<p>job 的状态由 oracle 决定;rollout 的状态由 verdict 决定。两者互不覆盖,一个 job 可以 oracle FAIL 而某次 rollout PASS——那正是最有价值的样本。</p>
<div class="code" style="padding:22px 18px;">
<svg class="fsm" viewBox="0 0 920 330" role="img" aria-label="上排 job 状态机:PENDING 到 ORACLE_RUNNING,分流到 PASS、FAIL 或 UNVERIFIED。下排 rollout 状态机:QUEUED 到 RUNNING 到 VERIFYING,分流到 PASS、FAIL、BLOCKED 或 TIMEOUT。">
<defs>
<marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="var(--ink-faint)"/></marker>
</defs>
<text class="lane" x="14" y="26">JOB · 由 oracle 决定</text>
<rect class="box" x="14" y="48" width="96" height="36" rx="3"/><text x="62" y="71" text-anchor="middle">PENDING</text>
<rect class="box" x="160" y="48" width="150" height="36" rx="3"/><text x="235" y="71" text-anchor="middle">ORACLE_RUNNING</text>
<rect class="box box-pass" x="380" y="14" width="88" height="36" rx="3"/><text x="424" y="37" text-anchor="middle">PASS</text>
<rect class="box box-fail" x="380" y="66" width="88" height="36" rx="3"/><text x="468" y="89" text-anchor="end" dx="-14">FAIL</text>
<rect class="box box-hold" x="380" y="118" width="120" height="36" rx="3"/><text x="440" y="141" text-anchor="middle">UNVERIFIED</text>
<path class="edge" d="M 110 66 L 154 66" marker-end="url(#ah)"/>
<path class="edge" d="M 310 60 L 345 60 L 345 32 L 374 32" marker-end="url(#ah)"/>
<path class="edge" d="M 310 66 L 345 66 L 345 84 L 374 84" marker-end="url(#ah)"/>
<path class="edge" d="M 310 72 L 345 72 L 345 136 L 374 136" marker-end="url(#ah)"/>
<text class="small" x="520" y="37">四道必选门全过</text>
<text class="small" x="520" y="89">任一必选门 fail</text>
<text class="small" x="520" y="141">数据缺失、板未接 → 门 = 未判定,不是 PASS</text>
<line class="lane-rule" x1="14" y1="182" x2="906" y2="182"/>
<text class="lane" x="14" y="206">ROLLOUT · 由 verdict 决定 · 每个 job 可有 K 个</text>
<rect class="box" x="14" y="228" width="96" height="36" rx="3"/><text x="62" y="251" text-anchor="middle">QUEUED</text>
<rect class="box box-hold" x="160" y="228" width="110" height="36" rx="3"/><text x="215" y="251" text-anchor="middle">RUNNING</text>
<rect class="box box-hold" x="320" y="228" width="110" height="36" rx="3"/><text x="375" y="251" text-anchor="middle">VERIFYING</text>
<rect class="box box-pass" x="500" y="196" width="80" height="32" rx="3"/><text x="540" y="217" text-anchor="middle">PASS</text>
<rect class="box box-fail" x="500" y="236" width="80" height="32" rx="3"/><text x="540" y="257" text-anchor="middle">FAIL</text>
<rect class="box" x="500" y="276" width="96" height="32" rx="3"/><text x="548" y="297" text-anchor="middle">BLOCKED</text>
<rect class="box box-fail" x="620" y="276" width="96" height="32" rx="3"/><text x="668" y="297" text-anchor="middle">TIMEOUT</text>
<path class="edge" d="M 110 246 L 154 246" marker-end="url(#ah)"/>
<path class="edge" d="M 270 246 L 314 246" marker-end="url(#ah)"/>
<path class="edge" d="M 430 240 L 465 240 L 465 212 L 494 212" marker-end="url(#ah)"/>
<path class="edge" d="M 430 246 L 465 246 L 465 252 L 494 252" marker-end="url(#ah)"/>
<path class="edge" d="M 430 252 L 465 252 L 465 292 L 494 292" marker-end="url(#ah)"/>
<path class="edge" d="M 215 264 L 215 292 L 614 292" marker-end="url(#ah)"/>
<text class="small" x="600" y="217">claim 与 verdict 一致性另记</text>
<text class="small" x="600" y="257">含 claim=PASS 而 verdict=FAIL 的说谎样本</text>
<text class="small" x="730" y="297">预算耗尽,外部 kill</text>
</svg>
</div>
<div class="note">
<div class="label">BLOCKED 怎么判,现在有机械规则了</div>
<p>agent 报 BLOCKED 时,看同一 job 的 oracle:oracle 也在同一阶段以同类错误失败 → agent 判断正确,正分;oracle 通过了 → agent 是误判或放弃,负分。不需要人判,也不需要 LLM 判。</p>
</div>
<h3>job 与 rollout 为什么要分开——用一个例子说</h3>
<p>job 是「一个模型 × 一块板 × 一种后端 × 一种量化配置」这个<strong>问题</strong>本身,比如 <code>det--yolov8n--nash-e--ptq--int8</code>。rollout 是 agent 对这个问题的<strong>一次作答</strong>。一个问题可以被答很多次:K 次 e2e、若干次 repair、teacher 答几次、student 答几次。所以:</p>
<div class="qa">
<div><div class="q">→</div><div><h3>job 只有一个 oracle 结果,rollout 每次一个 verdict</h3><p>oracle 是确定性流水线跑出来的,跑一百次都一样,所以只存一份;它回答「这个问题的标准答案是什么」。verdict 每个 rollout 一份,回答「这次作答对不对、比标准答案好还是差」。把两者塞进一个状态机,要么 oracle 被某次 rollout 覆盖,要么 rollout 的 PASS/FAIL 和 job 的 PASS/FAIL 混成一个字段,都是数据事故。</p></div></div>
<div><div class="q">→</div><div><h3>oracle FAIL 而 rollout PASS:最值钱,也最需要复核</h3><p>这种样本意味着 agent 做到了确定性流水线没做到的事——通常是它换了导出方式、改了不支持的算子、调了校准策略。它要么是黄金样本,要么是 harness 有 bug(oracle 本该能过)。所以这类 rollout 自动打上 <code>review</code> 标记,进人工抽查队列,抽查通过后 oracle 的实现也要相应修正,形成 harness 自身的改进回路。</p></div></div>
<div><div class="q">→</div><div><h3>oracle PASS 而 rollout FAIL:正常,但要看它错在哪</h3><p>这是 agent 能力不足的样本,reward 为负。verdict 记录它在哪道门失败、error_class 是什么,这决定了它在 SFT 里是负样本还是被丢弃(例如超预算的 TIMEOUT 通常丢弃,说谎的 claim≠verdict 通常保留作对比对)。</p></div></div>
<div><div class="q">→</div><div><h3>同一 job 的 oracle 和所有 rollout 共用同一份数据、同一段评测代码</h3><p>否则 delta 不可比。<code>job.json</code> 里钉死 eval 子集 manifest sha 和评测器版本,任何一方跑的时候都校验。</p></div></div>
</div>
<div class="note warn">
<div class="label">一个隐患:oracle 本身可能是错的</div>
<p>相对 oracle 计分的前提是 oracle 可信。harness 的 bug 会系统性地把 reward 打歪。三道防线:四道门是绝对判断不依赖 oracle;相对项经过 clip 有界;oracle FAIL / rollout PASS 的样本强制人工抽查。P2 垂直切片阶段必须人肉核对 oracle 的每一个数。</p>
</div>
</section>
<section>
<h2><span class="num">§6</span>什么算通过:四道门</h2>
<p class="lede">通过是绝对判断(这个量化模型能不能交付),reward 是相对判断(agent 比 oracle 做得好还是差)。两者都需要,不能混。门的阈值住在 <code>registry/gates.yaml</code>,随 run 快照进 <code>job.json</code>。</p>
<div class="tablewrap">
<table>
<thead><tr><th>门</th><th>检查什么</th><th>必选</th><th>判定依据</th></tr></thead>
<tbody>
<tr><td>G1 compile</td><td>.hbm 存在;<code>hb_model_info</code> 读出的 march、输入 dtype/layout 与 job.json 一致;<code>hb_verifier</code> 通过;编译日志无 ERROR</td><td>是</td><td>全部满足 → pass。march 不一致直接 fail——这是 agent 最容易"作弊"的口子</td></tr>
<tr><td>G2 numeric</td><td>量化模型 vs FP32 在固定 N 张校准外样本上,逐输出余弦相似度</td><td>是</td><td>所有输出 ≥ 阈值(默认 0.99,T09 按输出单列)。只报 min,不报均值</td></tr>
<tr><td>G3 task</td><td>任务指标:量化实测 − FP32 基线(同一评测器、同一子集、同一预处理)</td><td>是</td><td>退化在逐任务容差内。基线来自 <code>cache/baseline/</code> 自跑,不采信官方数</td></tr>
<tr><td>G4 perf</td><td>板端单线程纯推理时延 mean 与 P95;BPU 利用率;DDR 带宽</td><td>否</td><td>仅当 job.json 声明 <code>latency_budget_ms</code> 才判;否则只记录。mean 与 P95 分开存,永不互推</td></tr>
</tbody>
</table>
</div>
<p><strong>PASS = 必选门全部 pass。</strong>任一必选门因数据缺失、板未接而无法判定 → 该门记 <code>unverified</code>,job 状态为 UNVERIFIED 而不是 PASS。<code>unverified</code> 与 <code>na</code>(该任务明确不用此门)分开记——这是交付文档"未判定 ≠ 不适用"的直接映射。</p>
<h3>G3 逐任务容差(建议默认值,可在 gates.yaml 覆盖)</h3>
<p>容差来源标在最后一列。凡标"借用"的,交付文档里没有找到该任务的阈值记录,属于本文提议,须在第一个 run 后校准。</p>
<div class="tablewrap">
<table class="gates">
<thead><tr><th>task</th><th>主指标(评测集)</th><th>量化容差</th><th>必须固定的口径</th><th>来源</th></tr></thead>
<tbody>
<tr><td>cls T01</td><td>Top-1(ImageNet-1K val);Top-5 只记录</td><td>Δ ≥ −1.0 pp</td><td>resize / crop / 归一化、类别映射;多标签不适用</td><td>S9 MobileNetv2 −0.73 pp</td></tr>
<tr><td>det T02</td><td>box mAP@[.5:.95](COCO val2017);AP50 附</td><td>Δ ≥ −1.5 mAP 点</td><td>IoU 匹配、NMS 阈值、pycocotools 版本;开放词汇另固定词表与 embedding</td><td>S2 DOSODv2 目标差 1.3 / 1.0</td></tr>
<tr><td>seg T03</td><td>mIoU(ADE20K val / Cityscapes val);附各类 IoU</td><td>Δ ≥ −1.0 pp</td><td>ignore 标签、背景是否参与、输出分辨率与 resize</td><td>S9 Unet −0.44 pp</td></tr>
<tr><td>iseg T04</td><td>mask AP(COCO/LVIS);提示分割用固定协议 mIoU</td><td>Δ ≥ −1.5</td><td>掩码 IoU 阈值、每图最大实例数;提示:点/框生成规则、多 mask 选择</td><td>借用 T02</td></tr>
<tr><td>pose T05</td><td>OKS AP(COCO keypoints val)</td><td>Δ ≥ −1.5;关键点均偏移 ≤ 2 px</td><td>sigma、可见性、ROI 对齐到原图还是输入图</td><td>S11 1–2 px 可接受;AP 借用</td></tr>
<tr><td>mde T06</td><td>AbsRel / RMSE / δ1(KITTI Eigen / NYUv2)</td><td>AbsRel 相对劣化 ≤ 10%;δ1 Δ ≥ −1.0 pp</td><td><strong>必须声明尺度对齐方式</strong>;相对深度先对齐再算;非正值处理</td><td>S12 仅有 0.90 相关性;阈值为提议</td></tr>
<tr><td>stereo T07</td><td>EPE / D1-all(KITTI 2015)</td><td>每项相对劣化 ≤ 15%</td><td>逐帧均值口径;有效像素范围;A99 / Infinity 只在客户集上算</td><td>S4 SOW:量化允许 ≤ 15%</td></tr>
<tr><td>flow T08</td><td>EPE(2D) / Fl-all(Sintel / KITTI 2015)</td><td>相对劣化 ≤ 15%</td><td>输出上采样与位移缩放;遮挡/无效像素。<strong>光流 EPE ≠ 双目 EPE</strong>,分开存</td><td>借用 T07</td></tr>
<tr><td>feat T09</td><td>逐输出余弦相似度;HPatches 单应 AUC</td><td>desc ≥ 0.99;heatmap ≥ 0.92;AUC Δ ≥ −2 pp</td><td>提取与匹配分开评;选点规则与点数</td><td>S14 DFeature 0.9909 / 0.9282</td></tr>
<tr><td>occ T10</td><td>占据 IoU / 语义 mIoU(Occ3D-nuScenes)</td><td>Δ ≥ −1.5 pp</td><td>体素大小、空间范围、类别集合、可见性 mask</td><td>S15 无阈值;借用</td></tr>
</tbody>
</table>
</div>
<div class="note fix">
<div class="label">pp 与相对 % 不是一回事</div>
<p>mAP 32.5 → 31.2 是"下降 1.3 个 mAP 点",不是"相对下降 1.3%"。gates.yaml 里每条容差都显式写 <code>kind: abs_pp | rel_pct</code>,评测器按 kind 计算,不允许省略。</p>
</div>
<h3>任务特例:3D 占据 23 个模型,第一轮只编译不评精度</h3>
<p>理由:Occ3D 评测依赖 nuScenes 多相机输入 + 标定 + 占据标签,预处理逐模型不同,是所有任务里最重的;而这 23 个模型的价值主要在算子兼容性(BEV pooling、体素化、稀疏卷积),编译与板端两维就能采到。nuScenes Full 已在机器上,第二轮再补 Occ3D 标签。具体做法:</p>
<div class="tablewrap">
<table>
<thead><tr><th>环节</th><th>第一轮做法</th><th>写进哪里</th></tr></thead>
<tbody>
<tr><td>matrix</td><td><code>task: occ</code> 的 job 声明 <code>gates: [G1, G2, G4]</code>,G3 显式 <code>na</code>,reason <code>round1-compile-only</code></td><td>matrix.yaml · job.json</td></tr>
<tr><td>校准输入</td><td>三级来源,逐级回退并记录用了哪级:① 模型仓库自带的 demo 样本或 nuScenes 多相机加载器;② 从 nuScenes Full 抽 128 帧、按模型输入规格 resize;③ 输入范围内的合成随机张量。第 ③ 级在 job.json 记 <code>calib.source: synthetic</code>,G2 结果只作参考</td><td>04-calibrate/meta.json</td></tr>
<tr><td>G2 数值门</td><td>用与校准同级来源的另一批样本算余弦,阈值放宽到 0.97;synthetic 来源时 G2 记 <code>pass-weak</code>,不阻塞 PASS 但进 verdict</td><td>gates.yaml occ 段</td></tr>
<tr><td>G4 板端</td><td>照常测 latency / BPU util / DDR,这是第一轮 occ 最有价值的数字</td><td>07-board</td></tr>
<tr><td>PASS 定义</td><td>G1 ∧ G2(含 pass-weak);job 状态 PASS,但 result.json 里 <code>round: 1</code>,第二轮跑 G3 时开新 run,不覆盖</td><td>jobs.sqlite</td></tr>
<tr><td>reward</td><td>reward.py 遇 G3=na 时把 G3 的 2.0 权重按比例分给 G1/G2,相对 oracle 的精度项取 0;不因缺门而整体缩水</td><td>export/reward.py</td></tr>
<tr><td>agent 提示</td><td>e2e prompt 模板明说「本轮目标:编译通过 + 板端可运行;不评精度」。<code>.horizon</code> 没有 occ 专用 skill,预期 BLOCKED 比例高,这是正常的、有价值的</td><td>agent/prompts/e2e-occ.md</td></tr>
</tbody>
</table>
</div>
<p><code>na</code> 与 <code>unverified</code> 在这里的区别很关键:na 是「这一轮设计上就不评」,job 可以 PASS;unverified 是「本该评但没评成」,job 不能 PASS。第二轮补 Occ3D 标签后,occ 的 G3 从 na 变回必选,那是新 run。</p>
</section>
<section>
<h2><span class="num">§7</span>数据集层:机器上已有什么,缺什么,怎么分档</h2>
<p class="lede">agent 永远不下载数据。可复现(版本会飘)、防作弊(agent 会挑好算的子集)、轨迹干净(网络噪声不进训练数据)。这一节的两条实测事实改写了 v0.3 的假设。</p>
<h3>事实一:外网带宽分化极大,官方源等于不通</h3>
<div class="tablewrap">
<table>
<thead><tr><th>源</th><th>实测</th><th>含义</th></tr></thead>
<tbody>
<tr><td>aliyun OSS</td><td>43.8 MB/s</td><td>OE 包来源,最快</td></tr>
<tr><td>huggingface.co</td><td>8.0 MB/s</td><td>直连 0.32 s,<code>hf</code> CLI 已装,不用换镜像</td></tr>
<tr><td>github releases</td><td>5.1 MB/s</td><td>可用</td></tr>
<tr><td>images.cocodataset.org</td><td><strong>26 KB/s</strong></td><td>815 MB 要 9 小时。Excel 的"数据集入口"列给的全是这类官方站点,<strong>不能照着下</strong></td></tr>
</tbody>
</table>
</div>
<p>所以 <code>datasets.yaml</code> 里每个数据集声明<strong>有序的多个源</strong>:本地 → HF → OSS 镜像 → GitHub → 官方。逐个尝试,成功后校验 sha256。</p>
<h3>事实二:大部分数据集已经在 /mnt/data 上</h3>
<p>JuiceFS 951 TB 可用,写入 482 MB/s。以下都在别的用户目录里,world-readable,实测 <code>ls</code> 通。</p>
<div class="tablewrap">
<table>
<thead><tr><th>数据集</th><th>位置(/mnt/data/…)</th><th>内容</th><th>入 pool 时要做的</th></tr></thead>
<tbody>
<tr><td>ImageNet-1K val</td><td>chao01.wu/Data/Datasets/ImageNet/val_images.tar.gz(6.67 GB);HuggingFace/ImageNet_HF/imagenet/data/validation-*.parquet(14 片全)</td><td>50,000 张 + classes.py</td><td>校验 tar 完整性(旁边有个 .tmp 残片,说明下过两次);解压到 pool;parquet 作第二来源</td></tr>
<tr><td>COCO 2017</td><td>chao01.wu/Data/Datasets/COCO2017/val2017/(已解压)+ annotations_trainval2017.zip;guannan.he/coco2017/val2017.zip</td><td>5,000 张 + instances / person_keypoints 标注</td><td>复制 val2017 与两份 json;det · iseg · pose 三类共用</td></tr>
<tr><td>Cityscapes</td><td>guannan.he/CityScapes/{leftImg8bit,gtFine}_trainvaltest.zip</td><td>val 500 张 + gtFine labelIds</td><td>只解压 <code>*/val/</code>,其余不动</td></tr>
<tr><td>KITTI 2012 / 2015</td><td>qi.xiong/Dataset/StereoDataset/KITTI/data_scene_flow/</td><td>stereo 视差 GT + flow GT,各 200 帧 train</td><td>stereo 与 flow 两个任务共用一份,manifest 分开</td></tr>
<tr><td>ETH3D two-view</td><td>qi.xiong/Dataset/StereoDataset/eth3d/</td><td>双目训练/测试对</td><td>stereo 补充集</td></tr>
<tr><td>SceneFlow · InStereo2K · FallingThings</td><td>guannan.he/SceneFlow/,qi.xiong/…/StereoDataset/</td><td>合成双目</td><td>备用,第一轮不用</td></tr>
<tr><td>nuScenes Full</td><td>qi.xiong/Dataset/AutoDriveDataset/nuScenes/nuScenes_Full</td><td>多相机原始数据</td><td>occ 第一轮只取 128 帧做校准;第二轮补 Occ3D 标签</td></tr>
<tr><td>bdd100k</td><td>guannan.he/bdd100k/</td><td>驾驶场景 det / seg</td><td>备用</td></tr>
</tbody>
</table>
</div>
<div class="note">
<div class="label">复制,不软链</div>
<p>别人的目录随时可能被清。全部<strong>复制</strong>进 <code>bench/data/pool/&lt;dataset&gt;/&lt;version&gt;/</code>,复制完 <code>chmod -R a-w</code>,写 manifest.json(样本清单 sha256、数量、来源路径、复制日期)。951 TB 面前,几十 GB 的冗余换来的是与他人目录的零耦合。</p>
</div>
<h3>仍需下载的(合计约 15 GB,HF 速率半小时)</h3>
<div class="tablewrap">
<table>
<thead><tr><th>数据集</th><th>任务</th><th>规模</th><th>首选源</th></tr></thead>
<tbody>
<tr><td>ADE20K val</td><td>seg</td><td>2,000 张,~1 GB</td><td>HF</td></tr>
<tr><td>LVIS v1 val 标注</td><td>iseg · 开放词汇</td><td>json ~1 GB,图片复用 COCO</td><td>HF / GitHub</td></tr>
<tr><td>NYUv2 test</td><td>mde</td><td>654 张,~4 GB</td><td>HF</td></tr>
<tr><td>MPI-Sintel train</td><td>flow</td><td>1,041 ×2,~5 GB</td><td>HF 镜像 → 官方</td></tr>
<tr><td>HPatches</td><td>feat</td><td>116 seq,~4 GB</td><td>GitHub / HF</td></tr>
<tr><td>Occ3D-nuScenes 标签</td><td>occ 第二轮</td><td>数 GB</td><td>GitHub release</td></tr>
</tbody>
</table>
</div>
<h3>三档数据,job 默认只跑 T1</h3>
<div class="tablewrap">
<table>
<thead><tr><th>档</th><th>用途</th><th>规模</th><th>来源</th><th>谁用</th></tr></thead>
<tbody>
<tr><td>T0 calib</td><td>PTQ 校准;QAT calibration-only</td><td>128–512 张,按任务</td><td>train 划分固定 seed 抽,与 val 零交集</td><td>每个 job 必用</td></tr>
<tr><td>T1 smoke</td><td>G2 数值门 + G3 任务门的<strong>筛</strong></td><td>cls 2,000 · det/iseg/pose 1,000 · seg 500 · 其余全量(本就 ≤ 1k)</td><td>val 固定 seed 分层抽,清单入 git</td><td>每个 job 默认</td></tr>
<tr><td>T2 full</td><td>正式数字,G3 的<strong>判</strong></td><td>完整 val</td><td>pool 全量</td><td>只给要出交付数的模型;开新 run</td></tr>
</tbody>
</table>
</div>
<div class="note fix">
<div class="label">T1 的统计边界</div>
<p>500 张评 Top-1,单次测量标准误约 1.9 pp——分辨不出 1 pp 的掉点。但 FP32 与 INT8 是<strong>配对</strong>比较,同一批图,误差高度相关,delta 的标准误由不一致对决定,通常 ~5%:2,000 张时约 0.5 pp,够用;500 张时约 1 pp,与 −1.0 pp 容差同量级,会误判。所以 cls 的 T1 取 2,000 而不是 500,且 T1 上的 G3 容差自动放宽 <code>tol − 2·SE</code>,记 <code>pass-screen</code>;T2 才是 <code>pass</code>。对 reward 来说 T1 的信号足够;对交付来说 T2 才算数。</p>
</div>
<div class="qa">
<div><div class="q">→</div><div><h3>校准集细则</h3><p>存 npy,shape 含 batch 维,layout 与 job 输入规格一致,按 <code>&lt;task&gt;/&lt;input_spec&gt;</code> 组织,同规格模型共用。G2 的数值对比样本从 T1 里另抽,不能用校准图自己验自己。hb_compile 的校准张数影响编译时长,默认 128。</p></div></div>
<div><div class="q">→</div><div><h3>注入方式</h3><p><code>job.json.data</code> 写明 calib / T1 / T2 的 pool 路径、version、manifest sha。目录对 agent 只读(权限 + path-guard 双保险)。job 目录里只放一条 <code>data -&gt;</code> 软链方便 agent 引用——这不违反"一个 job 一个目录",那条原则约束产出,不约束输入。</p></div></div>
<div><div class="q">→</div><div><h3>小文件与 JuiceFS</h3><p>JuiceFS 是 FUSE,<code>max_read=128K</code>,几万张小 JPEG 随机读会慢。T1 子集(≤ 2k 文件)打包成单个 tar 或 npz 放在 pool 里,评测器直接读包;T2 全量走原目录。/home 是本地盘还剩 1.2 TB,热子集可选镜像一份,不是必须。</p></div></div>
<div><div class="q">→</div><div><h3>preflight</h3><p><code>pipeline.py --preflight</code> 在跑任何 job 前把 matrix 展开后需要的全部数据集查一遍:pool 里有没有、manifest sha 对不对、T0/T1 子集生成了没有。缺什么一次性报出来。跑到 08-eval 才发现没数据,浪费的是前面七个阶段。</p></div></div>
</div>
</section>
<section>
<h2><span class="num">§8</span>调度、预算与看门狗</h2>
<p>并行数按资源池分开设,一个数字管所有会让板端把 GPU 堵死。耗时不均不靠固定超时解决——<strong>固定超时会误杀慢模型</strong>,v0.3 这一条是错的。</p>
<figure class="code">
<figcaption>bench/config.yaml(节选)</figcaption>
<pre><span class="k">pools</span>:
gpu: { workers: 4, gpu_per_worker: 1 } <span class="c"># baseline / eval / qat-calibrate</span>
cpu: { workers: 8 } <span class="c"># export / hb_compile(CPU 密集)</span>
board: { workers: 1, target: s100 } <span class="c"># 一块板 = 1,严格串行,带锁</span>
agent: { workers: 3, rpm: 60 } <span class="c"># 受 LLM 端点限速</span>
<span class="k">watchdog</span>:
stall_s: 900 <span class="c"># 日志不增长 且 进程树 CPU < 5% 持续 15 min → STALLED</span>
hard_ceiling_s: 86400 <span class="c"># 24h 纯保险,正常永远碰不到</span>
soft_alert: p95x3 <span class="c"># 超过历史 p95 × 3 只告警,不杀</span>
<span class="k">perf</span>: { warmup: 20, runs: 200, stats: [mean, p95] }
<span class="k">rollouts</span>: { k: 4, max_turns: 40, max_tokens: 200000, hard_ceiling_s: 43200 }
<span class="c"># 真正的预算是 turns 和 tokens;墙钟只是 12h 保险</span>
<span class="k">retry</span>: { transient: 2, deterministic: 0 }</pre>
</figure>
<div class="qa">
<div><div class="q">→</div><div><h3>看门狗而不是超时</h3><p>一个阶段只在「日志 15 分钟不增长 且 进程树 CPU 空闲」时才被判 STALLED——真正在算的编译,哪怕跑 6 小时也不会被杀。STALLED 归 transient,重试一次进 <code>long</code> 车道(只有看门狗、没有软预算),再 STALLED 才记 <code>FAIL:stalled</code>。历史 p95 × 3 只用来告警和调度估算,不用来杀。</p></div></div>
<div><div class="q">→</div><div><h3>所有重阶段由 harness 执行,agent 只提交和等待</h3><p>agent 不在 bash 里直接跑 <code>hb_compile</code>。它调 <code>bench stage run compile --job … --config workspace/x.yaml</code>,harness 把任务丢进 cpu 池、按 backend 选对 venv、日志落 stage 目录,返回一个 handle;agent 用 <code>bench stage wait &lt;handle&gt;</code> 拿结果。四个好处:编译时长与 pi 的 bash 工具超时脱钩;backend/venv 选择不可能被 agent 选错;预算与看门狗对 oracle 和 rollout 一视同仁;日志位置统一。agent 的 bash 仍然可用,用来看日志、改配置、跑小脚本。</p></div></div>
<div><div class="q">→</div><div><h3>调度顺序</h3><p>先按 A / B / C 优先级,同级内按估算耗时短作业优先。C 类 44 个走独立低优先级车道,永远不堵 A 类。首轮估算用参数量,之后用 <code>jobs.sqlite</code> 里同模型同阶段的实测耗时——每个 job 跑完都回填。</p></div></div>
<div><div class="q">→</div><div><h3>板卡串行</h3><p><code>07-board</code> 只有一个 worker,拿锁才能推 hbm。板掉线归 transient,重试后仍失败 → 该 job 的 G4 记 unverified,不阻塞其他阶段。</p></div></div>
<div><div class="q">→</div><div><h3>rollout 预算</h3><p>turns 和 tokens 是真正的预算——它们是 agent 自己能控制的量。墙钟 12h 只防 agent 进程本身挂死。因为编译等待发生在 harness 侧,一个 rollout 等一次 3 小时的编译不会吃掉它的预算。触顶由 <code>budget-kill</code> extension 结束会话,状态 TIMEOUT,轨迹保留。</p></div></div>
</div>
</section>
<section>
<h2><span class="num">§9</span>失败之后会怎样:可见性、重试、轨迹边界</h2>
<p class="lede">没量化成功的模型不是丢了,是一个带分类的数据点。这一节回答三件事:在哪看、重试几次、重试算不算同一条轨迹。</p>
<h3>在哪看</h3>
<div class="tablewrap">
<table>
<thead><tr><th>命令</th><th>给什么</th></tr></thead>
<tbody>
<tr><td>bench status</td><td>按 state × task × march 的计数表;<code>--fail</code> 只列失败,附 error_class 分布</td></tr>
<tr><td>bench show &lt;job_id&gt;</td><td>阶段时间线、每阶段 exit code 与耗时、error_class、stderr 末 50 行、rollouts 的 verdict 一览</td></tr>
<tr><td>bench annotate &lt;job_id&gt; --note</td><td>人工备注("已知 hbdk 4.7.5 不支持 DeformConv"),只记录不改 verdict</td></tr>
<tr><td>bench report</td><td>从 jobs.sqlite 生成静态 HTML 到 <code>bench/reports/&lt;run_id&gt;.html</code>,给不上机器的人看</td></tr>
</tbody>
</table>
</div>
<h3>error_class:失败必须有名字</h3>
<p>每个 FAIL 由正则分类器从 stderr 打一个标签,存进 jobs.sqlite 和 result.json。它同时喂三个地方:看板、repair 模式的 prompt、BLOCKED 判定里的"同类错误"。</p>
<div class="tablewrap">
<table>
<thead><tr><th>error_class</th><th>含义</th><th>性质</th><th>处置</th></tr></thead>
<tbody>
<tr><td>unsupported_op</td><td>算子不在 march 支持列表</td><td>确定性</td><td>不重试;进 repair 队列;最有价值的一类</td></tr>
<tr><td>shape_infer</td><td>动态 shape / 维度推导失败</td><td>确定性</td><td>不重试;进 repair 队列</td></tr>
<tr><td>export_fail</td><td>ONNX 导出或 fx trace 失败</td><td>确定性</td><td>不重试;model-level,一次影响该模型所有 job</td></tr>
<tr><td>calib_nan</td><td>校准出 NaN / 溢出</td><td>确定性</td><td>不重试;进 repair 队列</td></tr>
<tr><td>verifier_mismatch</td><td>hb_verifier 或 G2 数值不一致</td><td>确定性</td><td>不重试;进 repair 队列</td></tr>
<tr><td>oom</td><td>GPU / 主机内存不足</td><td>瞬态</td><td>重试 ≤ 2,第二次降并发</td></tr>
<tr><td>board_offline</td><td>板子失联</td><td>瞬态</td><td>重试 ≤ 2;仍失败 → G4 unverified</td></tr>
<tr><td>stalled</td><td>看门狗判定挂死</td><td>瞬态</td><td>重试 1 次进 long 车道</td></tr>
<tr><td>unknown</td><td>没匹配到规则</td><td>—</td><td>不重试;进人工分类队列,规则库随之增长</td></tr>
</tbody>
</table>
</div>
<h3>重试在哪一层,是不是同一条轨迹</h3>
<div class="tablewrap">
<table>
<thead><tr><th>层</th><th>谁重试</th><th>触发</th><th>轨迹归属</th><th>次数</th></tr></thead>
<tbody>
<tr><td>oracle 阶段</td><td>harness</td><td>瞬态 error_class</td><td><strong>不是轨迹</strong>,没有 agent 参与;meta.json 记 attempt 号</td><td>≤ 2</td></tr>
<tr><td>rollout 内</td><td>agent 自己</td><td>它看到编译报错,改配置再提交</td><td><strong>同一条轨迹</strong>。这就是要采的诊断过程;HORIZON.md 的"同一方法最多重试 1 次"在这里作为 prompt 约束生效</td><td>受 max_turns 约束</td></tr>
<tr><td>新 rollout</td><td>harness 起新进程</td><td>matrix 声明 k 次</td><td><strong>全新轨迹</strong>:全新上下文,同一初始状态,不知道其他 rollout 存在</td><td>k</td></tr>
<tr><td>repair 模式</td><td>harness 起新进程</td><td>oracle FAIL</td><td>全新轨迹,输入是 <strong>oracle</strong> 的失败现场,不是某次 rollout 的</td><td>按 matrix</td></tr>
</tbody>
</table>
</div>
<div class="note warn">
<div class="label">三条硬边界</div>
<p><strong>不串联</strong>:rollout k+1 永远看不到 rollout k 的历史,否则轨迹不独立,GRPO 的组内比较失去意义。<strong>不续跑</strong>:TIMEOUT 的 rollout 是一条完整轨迹,reward 记负;想要更长就在下一个 run 调大 max_turns。<strong>repair 不吃 rollout 的失败</strong>:只吃 oracle 的,保证 repair 轨迹之间也独立。</p>
</div>
<h3>没量化成功的模型最后会怎样</h3>
<p>oracle FAIL 的 job 状态就停在 FAIL,带 error_class,不影响其他 job。它会:出现在 <code>bench status --fail</code>;按 matrix 进入 repair 队列生成轨迹;被 BLOCKED 规则引用。人可以 annotate 它。它不会被自动重跑——重跑意味着新 run,那是人的决定。第一轮预期 FAIL 率不低,尤其局部特征、光流、占据三类;这是数据,不是事故。</p>
</section>
<section>
<h2><span class="num">§10</span>pi 作为 policy driver</h2>
<p>pi 是可替换的一层。<code>job.json</code>、<code>claim.json</code>、<code>verdict.json</code> 与 driver 无关,将来换成 verl 的 agent loop 或 RPC 模式下的工具执行器,环境契约不变。</p>
<div class="tablewrap">
<table>
<thead><tr><th>要点</th><th>做法</th></tr></thead>
<tbody>
<tr><td>启动命令</td><td><code>pi --mode json -a --session-dir &lt;rollout&gt;/agent "&lt;task&gt;" &gt; events.jsonl</code>。<strong>不是</strong> <code>-p --mode json</code>,两者是互斥模式;<code>-a</code> 让非交互模式加载项目级 extension</td></tr>
<tr><td>轨迹源</td><td>用 events.jsonl 最后一条 <code>agent_end.messages[]</code>,它已是线性化的完整对话;session.jsonl 是树,只做备份</td></tr>
<tr><td>compaction</td><td><code>.pi/settings.json</code> 里 <code>"compaction": {"enabled": false}</code>(已核对是正式设置项)。训练数据里出现 <code>CompactionSummaryMessage</code> 是脏的;上下文靠 max_turns 控制</td></tr>
<tr><td>skills 发现</td><td>不用软链。<code>.pi/settings.json</code> 的 <code>skills</code> 数组支持 glob:<code>"skills": ["../.horizon/skills/*/*", "../.horizon/skills/horizon-router"]</code>,路径相对 <code>.pi/</code> 解析。嵌套两层的问题由 glob 直接解决</td></tr>
<tr><td>重阶段</td><td>agent 不直接跑 hb_compile / 板端推理,通过 <code>bench stage run / wait</code> 提交给 harness(§8)。pi bash 工具的默认超时<strong>未查到文档</strong>,P1 实测;但因为编译已不在 bash 里跑,这个值只影响小命令</td></tr>
<tr><td>工具白名单</td><td><code>--tools read,edit,write,bash,grep</code>;write/edit 受 path-guard 限制在 workspace/</td></tr>
<tr><td>extensions</td><td><code>path-guard</code>(越界拒绝)· <code>claim-card</code>(强制结构化汇报,写 claim.json;未调用则 claim 缺席,小额扣分)· <code>oe-doc</code>(把 oe-mcp 包成 CLI,pi 不支持 MCP)· <code>budget-kill</code></td></tr>
<tr><td>模型端点</td><td><code>policy: teacher</code> 走 Claude/GPT 直连;<code>policy: student</code> 走 <code>~/.pi/agent/models.json</code> 指向你的 vLLM。matrix 里显式选,不混采</td></tr>
<tr><td>环境变量</td><td>agent 的 bash 只暴露 job 目录路径与 <code>PI_SESSION_ID</code>。<strong>板卡凭证不进 env</strong>,板端推送由 harness 在 agent 进程外完成</td></tr>
</tbody>
</table>
</div>
</section>
<section>
<h2><span class="num">§11</span>RLVR:verdict 定 reward,相对 oracle 计分</h2>
<p>result.json / verdict.json 只存原始测量值,reward 在 L4 离线算,配方版本化在 <code>export/reward.py</code>。</p>
<figure class="code">
<figcaption>rollouts/03/verdict.json</figcaption>
<pre>{
<span class="k">"job_id"</span>: "det--yolov8n--nash-e--ptq--int8", <span class="k">"rollout"</span>: 3, <span class="k">"mode"</span>: "e2e", <span class="k">"policy"</span>: "teacher",
<span class="k">"gates"</span>: {
"G1": { "verdict": "pass", "march_ok": true, "verifier_ok": true },
"G2": { "verdict": "pass", "cos_min": 0.9961, "threshold": 0.99 },
"G3": { "verdict": "pass", "metric": "mAP@[.5:.95]", "fp32": 0.372, "quant": 0.361,
"delta": -0.011, "kind": "abs_pp", "tolerance": -0.015 },
"G4": { "verdict": "na", "latency_ms_mean": 4.21, "latency_ms_p95": 4.60, "bpu_util": 0.78 }
},
<span class="k">"terminal"</span>: "PASS",
<span class="k">"claim_consistent"</span>: true,
<span class="k">"oracle_ref"</span>: { "terminal": "PASS", "G3_delta": -0.013, "latency_ms_mean": 4.35 },
<span class="k">"cost"</span>: { "turns": 17, "tokens": 91233, "wall_s": 612 }
}</pre>
</figure>
<figure class="code">
<figcaption>export/reward.py · v0 配方(示意,权重待第一批数据后校准)</figcaption>
<pre>r = 1.0·[G1 pass] + 1.0·[G2 pass] + 2.0·[G3 pass] <span class="c"># 绝对门</span>
r += 1.0·clip((Δ_agent − Δ_oracle) / |tolerance|, −1, +1) <span class="c"># 相对 oracle 的精度</span>
r += 0.5·clip((lat_oracle − lat_agent) / lat_oracle, −1, +1) <span class="c"># 相对 oracle 的延时</span>
r −= 0.5·(turns / max_turns) + 0.5·(tokens / max_tokens) <span class="c"># 代价</span>
r −= 2.0·[claim ≠ verdict] <span class="c"># 说谎</span>
BLOCKED: +1.0 若 oracle 同阶段同类失败;否则 −2.0
TIMEOUT: −1.0</pre>
</figure>
<div class="qa">
<div><div class="q">→</div><div><h3>teacher 与 student 是两种数据</h3><p>teacher(Claude/GPT)轨迹 + verdict 筛选 → SFT / 拒绝采样微调。student(你的模型)轨迹 → on-policy。两者 prompt 模板相同,matrix 里 policy 维度分开采,导出时分开落 dataset。</p></div></div>
<div><div class="q">→</div><div><h3>真 on-policy GRPO 时,rollout 驱动者会变</h3><p>GRPO 需要 rollout 时的 logprobs,pi session 只有 messages。到那一步,verl/AgentRL 自己的循环做 rollout,pi 退化为 RPC 模式的工具执行器或被替换。本方案的持久资产是环境契约(§3 目录 + §6 门 + verdict.json),不是 pi。</p></div></div>
<div><div class="q">→</div><div><h3>采样去重</h3><p>同一模型跑 nash-e/m/p 的 e2e 轨迹高度重复。e2e 模式按 (model, backend) 只采主 march(默认 nash-e),全 march 只在 oracle 模式铺。</p></div></div>
</div>
</section>
<section>
<h2><span class="num">§12</span>下游边界:搬规则,不搬代码</h2>
<p>数据全部留在本项目。<code>~/data_collect</code> 与 <code>data-litellm-1.0</code> 零耦合。</p>
<div class="tablewrap">
<table>
<thead><tr><th>data_collect 的东西</th><th>处置</th><th>原因</th></tr></thead>
<tbody>
<tr><td>rebuild / probe / backfill / consumer</td><td>不要</td><td>网关 raw 日志重建会话边界;pi 的 agent_end 已是完整线性对话</td></tr>
<tr><td>judge_sessions.py</td><td>不要</td><td>LLM 主观打分。我们有 verdict,不需要也不应该让模型判</td></tr>
<tr><td>clean_for_agentrl.py</td><td><strong>只搬规则 R1–R8</strong></td><td>双方言归一、thinking 剥离、system 首条、悬空尾轮截断——踩过的坑</td></tr>
<tr><td>redact_archive.py</td><td><strong>搬</strong>(v0.2 漏了)</td><td>agent 的 bash 输出会进轨迹,路径、主机名、任何凭证都要脱敏</td></tr>
</tbody>
</table>
</div>
<p>落地为 <code>bench/export/{to_agentrl,redact,reward}.py</code>:独立脚本,不 import 那边任何东西。那套代码 <code>ROOT</code> 硬编码指向 litellm 目录,一 import 两个项目的路径与数据版本就缠死。</p>
</section>
<section>
<h2><span class="num">§13</span>待决与阻塞项</h2>
<div class="items">
<div class="item">
<div class="head"><span class="pill ok">已解除</span><h3>验证集落盘(v0.3 的头号阻塞)</h3></div>
<p>ImageNet val、COCO、Cityscapes、KITTI、ETH3D、nuScenes 都在 /mnt/data 上(§7 表)。剩 ADE20K / LVIS 标注 / NYUv2 / Sintel / HPatches 约 15 GB 走 HF,半小时。HF token 不再需要。</p>
</div>
<div class="item">
<div class="head"><span class="pill dec">待验证</span><h3>ImageNet val tar 的完整性</h3></div>
<p><code>val_images.tar.gz</code> 旁边躺着一个 1.5 GB 的 <code>.tmp</code> 残片,说明下过两次。入 pool 前 <code>tar -tzf | wc -l</code> 必须等于 50000,或者直接用 HF parquet 那 14 片作首选源,二者取其一校验通过即可。</p>
</div>
<div class="item">
<div class="head"><span class="pill block">要修</span><h3>.env.oe-package 里的死路径</h3></div>
<p><code>PYTHON_BIN</code> 指向 <code>&lt;项目根&gt;/oellm2/venv/bin/python</code>,项目根下没有 <code>oellm2/</code>——QAT venv 实际在 <code>~/toolchains/oellm2/venv</code>。agent 按规则会读这个文件。P1 第一件事。</p>
</div>
<div class="item">
<div class="head"><span class="pill dec">待验证</span><h3>pi bash 工具的默认超时</h3></div>
<p>settings.md 里没有这一项。因为重编译已改由 harness 执行,它只影响 agent 跑的小命令;但 <code>bench stage wait</code> 本身是长阻塞,需要确认它不会被 bash 工具掐断——不行就改成轮询式 <code>wait --poll 60</code>。</p>
</div>
<div class="item">
<div class="head"><span class="pill dec">待确认</span><h3>rdk-sandbox 是哪块板</h3></div>
<p>映射已定:s100 → nash-e,s100p → nash-m,s600 → nash-p。沙箱那块具体是哪个,板上 <code>hrut_somstatus</code> 或 <code>/proc/device-tree/model</code> 看一眼。改密钥登录,口令用完轮换。</p>
</div>
<div class="item">
<div class="head"><span class="pill dec">待定</span><h3>teacher 模型选谁、student 端点在哪</h3></div>
<p>决定第一批数据的性质。建议 P3 先用 teacher 跑通闭环,student 端点等 vLLM 起来再接。</p>
</div>
<div class="item">
<div class="head"><span class="pill ok">已解除</span><h3>pi 对嵌套 skills 目录的发现</h3></div>
<p>settings.json 的 <code>skills</code> 数组支持 glob,<code>"../.horizon/skills/*/*"</code> 直接覆盖两层嵌套,不需要软链,也不需要逐个 <code>--skill</code>。</p>
</div>
<div class="item">
<div class="head"><span class="pill ok">已解除</span><h3>PTQ 工具链就位</h3></div>
<p>OE 3.7.0 s100/s600 包已装,TinyNet → hb_compile → .hbm 实测跑通,HBRT4 加载校验通过。338 个公开权重走标准 PTQ。</p>
</div>
<div class="item">
<div class="head"><span class="pill dec">留意</span><h3>Board-Agent 三个中间产物已被删</h3></div>
<p><code>model-codegen-info.json</code>、<code>qconfig_setting/</code>、<code>tmp_1_original_float_perf.json</code>,可再生。记在这里防误判。</p>
</div>
</div>
</section>
<section>
<h2><span class="num">§14</span>实施顺序</h2>
<div class="phases">
<div class="phase">
<div class="n">P1 · 地基</div>
<h3>骨架、契约、数据入池</h3>
<p>建 <code>bench/</code>、写 CONTRACT.md、修死路径、Excel → models.csv、写 gates/budgets/boards/datasets.yaml、装 pi + settings glob 验证 skills、<strong>COCO / ImageNet / Cityscapes / KITTI 从本机复制入池并校验</strong>,其余 15 GB 走 HF,<code>--preflight</code> 全绿。产出:空转的 pipeline.py。</p>
</div>
<div class="phase">
<div class="n">P2 · oracle 切片</div>
<h3>yolov8n + resnet18 各打穿八阶段</h3>
<p>det 用 COCO T1,cls 用 ImageNet T1(现在都在本机)。PTQ、nash-e。<strong>人肉核对 oracle 的每一个数</strong>——这是相对 oracle 计分的信任基础。产出:两份 oracle/result.json,看门狗与 bench stage CLI 跑通。</p>
</div>
<div class="phase">
<div class="n">P3 · 闭环</div>
<h3>e2e + repair 各跑通一次</h3>
<p>teacher 模型。e2e 跑 yolov8n;repair 拿一个 oracle 必失败的模型。验证 path-guard、claim/verdict 分离、budget-kill、导出脱敏。产出:第一批带 reward 的轨迹。</p>
</div>
<div class="phase">
<div class="n">P4 · 横向</div>
<h3>TorchVision 51 + COCO 系 ~60</h3>
<p>oracle 全 march;e2e 每类 6–10 个。压测资源池、板卡锁、耗时估算回填。产出:第一个完整 run、失败率分布、gates 容差校准。</p>
</div>
<div class="phase">
<div class="n">P5 · 全量</div>
<h3>A 类 166 → B → C</h3>
<p>局部特征 28、3D 占据 23、光流 14 算子特殊,是失败重灾区也是轨迹金矿,放最后。C 类走低优先级车道。</p>
</div>
</div>
</section>
<footer>
环境指纹 · instance-mzr1k6rw · 4 × RTX 5090 · .horizon skill pack v0.2.0<br>
PTQ 后端 ~/toolchains/oe-3.7.0/venv · horizon_tc_ui 3.5.3 · hmct 2.6.5 · hbdk4 4.7.5<br>
QAT 后端 ~/toolchains/oellm2/venv · horizon_plugin_pytorch 3.3.5+cu128.torch280 · hbdk4 4.12.3a2.dev+cd3ae44 · hbm_infer 3.15.3<br>
板端 rdk-sandbox · UCP 3.15.2 · s100→nash-e · s100p→nash-m · s600→nash-p<br>
存储 JuiceFS:robotwin-archive 951 TB 可用 · /home 本地 1.2 TB · 带宽 OSS 43.8 / HF 8.0 / GitHub 5.1 MB/s · COCO 官方 26 KB/s<br>
G3 容差来源 · 视觉模型交付指标说明 2026-09-14 · S2 S4 S9 S11 S12 S14 S15
</footer>
</div>
</body>
</html>

View File

@ -0,0 +1,460 @@
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>html{color-scheme:light dark}body{margin:0}img{max-width:100%}[hidden]{display:none!important}</style>
<title>闭环走通的判据</title>
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500;600&family=IBM+Plex+Sans:wght@400;500;600;700&family=IBM+Plex+Serif:ital,wght@0,400;0,500;1,400&display=swap">
<style>
:root {
--paper: #EEF2F4;
--surface: #FFFFFF;
--surface-sunk: #E3EAEE;
--ink: #11181D;
--ink-soft: #495861;
--ink-faint: #7C8C95;
--rule: #C9D6DC;
--rule-soft: #DCE6EA;
--accent: #0B6E8A;
--accent-soft: #DCEDF3;
--pass: #2F6B46;
--fail: #9E3223;
--hold: #8A6516;
--pass-bg: #E0EDE5;
--fail-bg: #F6E2DE;
--hold-bg: #F5EBD6;
--f-display: "IBM Plex Sans", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", system-ui, sans-serif;
--f-body: "IBM Plex Serif", "Songti SC", "SimSun", Georgia, serif;
--f-mono: "IBM Plex Mono", ui-monospace, "SF Mono", Menlo, monospace;
}
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) {
--paper: #0E1418; --surface: #161E24; --surface-sunk: #0A1014;
--ink: #E4EDF1; --ink-soft: #A3B4BD; --ink-faint: #6E828C;
--rule: #2A3840; --rule-soft: #1F2B32;
--accent: #4FB6D4; --accent-soft: #12303B;
--pass: #6FBE8C; --fail: #E08A78; --hold: #D9B25F;
--pass-bg: #142A1E; --fail-bg: #2E1A15; --hold-bg: #2A2311;
}
}
:root[data-theme="dark"] {
--paper: #0E1418; --surface: #161E24; --surface-sunk: #0A1014;
--ink: #E4EDF1; --ink-soft: #A3B4BD; --ink-faint: #6E828C;
--rule: #2A3840; --rule-soft: #1F2B32;
--accent: #4FB6D4; --accent-soft: #12303B;
--pass: #6FBE8C; --fail: #E08A78; --hold: #D9B25F;
--pass-bg: #142A1E; --fail-bg: #2E1A15; --hold-bg: #2A2311;
}
* { box-sizing: border-box; }
body { background: var(--paper); color: var(--ink); font-family: var(--f-body); font-size: 16px; line-height: 1.72; -webkit-font-smoothing: antialiased; }
.wrap { max-width: 1000px; margin: 0 auto; padding-block: 56px 110px; padding-inline: 24px; display: flex; flex-direction: column; gap: 52px; }
.masthead { display: flex; flex-direction: column; gap: 16px; }
.eyebrow { font-family: var(--f-mono); font-size: 11.5px; letter-spacing: .16em; text-transform: uppercase; color: var(--accent); font-weight: 600; }
h1 { font-family: var(--f-display); font-weight: 700; font-size: clamp(30px, 5vw, 44px); line-height: 1.12; letter-spacing: -.022em; margin: 0; text-wrap: balance; }
.standfirst { font-size: 18px; color: var(--ink-soft); max-width: 60ch; margin: 0; }
.facts { display: grid; grid-template-columns: repeat(auto-fit, minmax(140px, 1fr)); border-top: 1px solid var(--rule); border-bottom: 1px solid var(--rule); margin: 0; }
.fact { padding: 13px 18px 13px 0; border-right: 1px solid var(--rule-soft); }
.fact:last-child { border-right: 0; }
.fact dt { font-family: var(--f-mono); font-size: 10.5px; letter-spacing: .1em; text-transform: uppercase; color: var(--ink-faint); margin: 0 0 3px; }
.fact dd { margin: 0; font-family: var(--f-display); font-weight: 600; font-size: 15px; font-variant-numeric: tabular-nums; }
section { display: flex; flex-direction: column; gap: 20px; }
h2 { font-family: var(--f-display); font-weight: 600; font-size: 23px; letter-spacing: -.012em; margin: 0; padding-bottom: 10px; border-bottom: 2px solid var(--ink); display: flex; align-items: baseline; gap: 14px; text-wrap: balance; }
h2 .num { font-family: var(--f-mono); font-size: 12px; font-weight: 500; color: var(--accent); letter-spacing: .08em; flex: none; }
h3 { font-family: var(--f-display); font-weight: 600; font-size: 16px; margin: 0; letter-spacing: -.005em; text-wrap: balance; }
p { margin: 0; max-width: 66ch; }
.lede { font-size: 17px; }
strong { font-weight: 600; color: var(--ink); }
em { font-style: italic; color: var(--ink-soft); }
code { font-family: var(--f-mono); font-size: .84em; background: var(--surface-sunk); padding: 1px 5px; border-radius: 2px; color: var(--ink); overflow-wrap: anywhere; }
.stack { display: flex; flex-direction: column; gap: 8px; }
.layer { display: grid; grid-template-columns: 62px 1fr; background: var(--surface); border: 1px solid var(--rule); border-left: 3px solid var(--accent); }
.layer.is-agent { border-left-color: var(--hold); }
.layer .tag { font-family: var(--f-mono); font-size: 11.5px; font-weight: 600; color: var(--accent); padding: 15px 0 14px 14px; letter-spacing: .04em; }
.layer.is-agent .tag { color: var(--hold); }
.layer .lbody { padding: 13px 18px 15px 4px; display: flex; flex-direction: column; gap: 5px; min-width: 0; }
.layer .lbody h3 { font-size: 15.5px; }
.layer .lbody p { font-size: 14.5px; color: var(--ink-soft); }
.inv { display: flex; flex-direction: column; }
.inv-row { display: grid; grid-template-columns: 46px 1fr; gap: 16px; padding: 17px 0; border-bottom: 1px solid var(--rule-soft); align-items: start; }
.inv-row:first-child { border-top: 1px solid var(--rule-soft); }
.inv-row .mark { font-family: var(--f-mono); font-size: 12px; font-weight: 600; color: var(--accent); padding-top: 4px; letter-spacing: .04em; }
.inv-row .body { display: flex; flex-direction: column; gap: 6px; min-width: 0; }
.inv-row p { font-size: 15px; color: var(--ink-soft); }
.counter { font-size: 14px; color: var(--ink-soft); border-left: 2px solid var(--fail); padding-left: 12px; background: var(--fail-bg); padding-block: 7px; padding-right: 12px; }
.counter b { font-family: var(--f-mono); font-size: 11.5px; letter-spacing: .08em; text-transform: uppercase; color: var(--fail); font-weight: 600; display: block; margin-bottom: 1px; }
.tablewrap { overflow-x: auto; border: 1px solid var(--rule); background: var(--surface); }
table { border-collapse: collapse; width: 100%; font-size: 14.5px; font-family: var(--f-display); }
th, td { text-align: left; padding: 10px 14px; border-bottom: 1px solid var(--rule-soft); vertical-align: top; }
thead th { font-family: var(--f-mono); font-size: 10.5px; letter-spacing: .1em; text-transform: uppercase; color: var(--ink-faint); font-weight: 600; border-bottom: 1px solid var(--rule); white-space: nowrap; }
tbody tr:last-child td { border-bottom: 0; }
td.num-cell { font-variant-numeric: tabular-nums; text-align: right; white-space: nowrap; }
.pill { font-family: var(--f-mono); font-size: 10.5px; font-weight: 600; letter-spacing: .06em; text-transform: uppercase; padding: 2px 7px; border-radius: 2px; white-space: nowrap; display: inline-block; }
.pill.ok { background: var(--pass-bg); color: var(--pass); }
.pill.no { background: var(--fail-bg); color: var(--fail); }
.pill.part { background: var(--hold-bg); color: var(--hold); }
.checklist { display: flex; flex-direction: column; gap: 7px; }
.check { display: grid; grid-template-columns: 54px 1fr auto; gap: 14px; align-items: start; background: var(--surface); border: 1px solid var(--rule); padding: 12px 14px; }
.check .id { font-family: var(--f-mono); font-size: 12px; font-weight: 600; color: var(--accent); }
.check .txt { display: flex; flex-direction: column; gap: 3px; min-width: 0; }
.check .txt p { font-size: 14px; color: var(--ink-soft); }
.check .txt h3 { font-size: 14.5px; }
.seq { display: flex; flex-direction: column; }
.step { display: grid; grid-template-columns: 34px 1fr; gap: 16px; padding: 16px 0; border-bottom: 1px solid var(--rule-soft); }
.step:first-child { border-top: 1px solid var(--rule-soft); }
.step .n { font-family: var(--f-mono); font-size: 13px; font-weight: 600; color: var(--ink-faint); padding-top: 2px; font-variant-numeric: tabular-nums; }
.step .sbody { display: flex; flex-direction: column; gap: 7px; min-width: 0; }
.step .sbody p { font-size: 14.5px; color: var(--ink-soft); }
.accept { font-family: var(--f-mono); font-size: 12.5px; color: var(--ink); background: var(--accent-soft); padding: 7px 11px; border-left: 2px solid var(--accent); overflow-wrap: anywhere; line-height: 1.6; }
.callout { background: var(--surface); border: 1px solid var(--rule); border-top: 3px solid var(--accent); padding: 20px 22px; display: flex; flex-direction: column; gap: 9px; }
.callout h3 { font-size: 17px; }
.note { font-size: 14px; color: var(--ink-faint); max-width: 70ch; }
footer { border-top: 1px solid var(--rule); padding-top: 18px; display: flex; flex-direction: column; gap: 6px; }
footer p { font-size: 13.5px; color: var(--ink-faint); max-width: 74ch; }
@media (max-width: 620px) {
.layer { grid-template-columns: 1fr; }
.layer .tag { padding: 12px 14px 0; }
.layer .lbody { padding: 6px 14px 14px; }
.check { grid-template-columns: 1fr; gap: 6px; }
.inv-row { grid-template-columns: 1fr; gap: 6px; }
}
</style>
</head>
<body>
<div class="wrap">
<header class="masthead">
<div class="eyebrow">方法论 · BPU 量化轨迹工厂配套</div>
<h1>闭环走通的判据</h1>
<p class="standfirst">把「垂直工程任务 → 可训练轨迹」沉淀成可复用范式。走通的标准不是产出一条 PASS 样本,而是四类终态都能被正确落位。</p>
<dl class="facts">
<div class="fact"><dt>主批次终态</dt><dd>108</dd></div>
<div class="fact"><dt>oracle PASS</dt><dd>3</dd></div>
<div class="fact"><dt>可用轨迹</dt><dd>0 / 11</dd></div>
<div class="fact"><dt>repair 入队</dt><dd>0</dd></div>
<div class="fact"><dt>导出样本</dt><dd>1(tools 0)</dd></div>
</dl>
</header>
<section>
<h2><span class="num">§1</span>先说结论</h2>
<p class="lede">这套东西方法论上是成立的,而且已经把最难的那一半做完了——<strong>可机器判定的验证器</strong>(四道门 + oracle 相对基准)和<strong>内核级隔离</strong>都已落地。没走通的是另一半:<strong>闭环的最后三个接头是断的</strong>——FAIL 不进 repair、采集缺失不报错、运行进程与磁盘代码不一致。这三处都不是难题,是接线问题。</p>
<div class="callout">
<h3>「走通」不等于「跑出一条正样本」</h3>
<p>今天已经有 1 条 PASS 并导出成 teacher 样本了。但它不能证明闭环成立:它的 <code>tools</code> 数是 0,<code>sft_eligible</code> 仍被标成 1——说明<em>验证器在该拦的地方没有拦</em>。一个只会盖 PASS 的验证器和一个坏掉的验证器,从正样本上看不出区别。</p>
<p><strong>闭环成立的判据只有一个:给它四类终态(PASS / FAIL / BLOCKED / TIMEOUT),它能把四类都正确落位,包括把该拒的样本拒掉。</strong>这是本页第 5 节 MVL 清单的全部依据。</p>
</div>
</section>
<section>
<h2><span class="num">§2</span>范式:什么任务能这样工业化</h2>
<p>这套做法可以从量化编译迁出去,但有前提。三条同时成立才适用:</p>
<div class="tablewrap">
<table>
<thead><tr><th>前提</th><th>含义</th><th>量化编译任务为何满足</th></tr></thead>
<tbody>
<tr><td><strong>判据可机器化</strong></td><td>对错由程序判定,不靠人评、不靠模型当裁判</td><td>返回码、逐输出余弦、Top1 / mAP 差值,全部是数值比较</td></tr>
<tr><td><strong>验证远比求解便宜</strong></td><td>跑一遍很贵、判一次很便宜,才撑得起大量采轨迹</td><td>一个 job 的 oracle 要编译加评测;判四道门只是读 <code>result.json</code></td></tr>
<tr><td><strong>失败模式是有限集</strong></td><td>失败能分类,才能区分「agent 判断错」与「工具链真做不到」</td><td><code>error_class</code> 确定性五类、瞬态三类,其余归 <code>unknown</code> 进人工队列</td></tr>
</tbody>
</table>
</div>
<p class="note">第三条最容易被忽略,但它决定了 BLOCKED 这一类终态能不能存在。没有失败分类,agent 说「这个做不了」就无法判真假,只能当失败处理——正确放弃的能力就永远学不到。</p>
</section>
<section>
<h2><span class="num">§3</span>五个构件,缺一不闭合</h2>
<p>范式本身是一条单向链:<strong>冻结输入 → oracle 出标准答案 → 隔离下作答 → 独立核验 → 分级导出</strong>。每一环的职责是硬边界,不能交叉。</p>
<div class="stack">
<div class="layer">
<div class="tag">L0</div>
<div class="lbody">
<h3>冻结的任务单元</h3>
<p>job = 模型 × march × backend × qcfg,<code>job_id</code> 既是目录名也是 join key。run 口径(venv、registry、门阈值)任一变化就开新 run,不原地改。这让任何一条轨迹永远能追溯到它当时面对的确切世界。</p>
</div>
</div>
<div class="layer">
<div class="tag">L1</div>
<div class="lbody">
<h3>确定性 oracle —— 一份产出,三种用途</h3>
<p>这是整个范式的杠杆点。同一份 oracle 同时是:①ground truth(agent 做得对不对);②相对基准(reward 里精度与延时都是相对 oracle 的差,不是绝对值);③BLOCKED 的裁判(agent 说做不到时,看 oracle 是不是在同阶段以同类 <code>error_class</code> 失败)。省掉 oracle 就同时失去这三样。</p>
</div>
</div>
<div class="layer is-agent">
<div class="tag">L2</div>
<div class="lbody">
<h3>隔离的作答区</h3>
<p>agent 只有 <code>rollouts/&lt;kk&gt;/workspace/</code> 一个可写区。隔离靠 <code>unshare -rm</code> 的 mount namespace 把答案、兄弟轨迹、凭证从文件系统里摘掉;解析 shell 字符串的 guard 退化为教学层,只负责给一句人话的拒绝理由、省 agent 的预算。</p>
</div>
</div>
<div class="layer">
<div class="tag">L3</div>
<div class="lbody">
<h3>独立核验的 verdict</h3>
<p>四道门:G1 编译形式、G2 数值余弦、G3 任务指标、G4 板端延时(非必选)。判定完全不读 agent 的自述,<code>claim</code> 只用来和 verdict 比对。<code>na</code>(本轮设计上不评)与 <code>unverified</code>(本该评没评成)严格区分——前者不阻塞 PASS,后者阻塞。</p>
</div>
</div>
<div class="layer">
<div class="tag">L4</div>
<div class="lbody">
<h3>离线 reward + 分级导出资格</h3>
<p>reward 只读 <code>verdict.json</code>,纯函数、配方版本化,可对历史全量重打分。导出资格分两级:RL 收正负例(要的是区分度),SFT 只收「做对了且诚实」——<code>claim ≠ verdict</code> 和 <code>review=true</code> 是两条优先级最高的硬排除。</p>
</div>
</div>
</div>
</section>
<section>
<h2><span class="num">§4</span>八条不变量,每条都有今天的反例</h2>
<p>这些不是设计偏好,是今天这批任务用 105 个 FAIL 换来的。每条后面附的是它被违反时实际发生了什么。</p>
<div class="inv">
<div class="inv-row">
<div class="mark">I1</div>
<div class="body">
<h3>运行中的版本必须可证明等于磁盘上的版本</h3>
<p>常驻调度进程加载的是它<em>启动那一刻</em>的代码与配置。「改好了」指的是文件,不是正在跑的东西。进程启动时必须自证代码指纹,与 run manifest 不一致就拒绝领新任务——而不是打印一行 WARN。</p>
<div class="counter"><b>反例 · 37 例编译失败</b>oracle 调度 PID 3776582 于 01:41:58 启动,<code>harness/backends/ptq.py</code> 于 01:46:29 修好;进程没重启,下午新生成的 YAML 仍把 <code>quant_config</code> 放在 <code>model_parameters</code> 下。<code>paths.config()</code> 的 <code>lru_cache</code> 让旧配置也常驻内存。<code>check_manifest()</code> 对代码漂移只告警,没有拒跑。</div>
</div>
</div>
<div class="inv-row">
<div class="mark">I2</div>
<div class="body">
<h3>隔离必须在内核层,不能在字符串层</h3>
<p>解析 agent 写的 shell 命令做白名单,是一类永远追不平的防线——每一版都会出现新写法。真正的隔离是让进程<em>看不见</em>。看不见的东西,再多绕法也没用。</p>
<div class="counter"><b>反例 · guard 连续四版被穿透</b>换行后的 <code>cd</code>、<code>$IFS</code>、<code>$VAR/</code> 路径拼接、workspace 内软链回 <code>../../../oracle</code>——四种手法各穿透一版。v5 起改为 mount namespace 摘除加 capability drop,guard 降级为教学层。</div>
</div>
</div>
<div class="inv-row">
<div class="mark">I3</div>
<div class="body">
<h3>采集链任何一环缺失,必须硬失败</h3>
<p>数据管道里的静默降级比崩溃危险得多:崩溃会被发现,降级会安静地产出「看起来完整」的训练数据。凡是训练必需的字段,缺失即拒绝启动或拒绝导出。</p>
<div class="counter"><b>反例 · 唯一一条导出样本工具定义为空</b><code>rollout.py</code> 起 <code>dump_tools.js</code> 时没补 Node bin 的 PATH → <code>npm: not found</code> → 非零退出被忽略 → 不生成 <code>tools.json</code> → 导出回退也失败、返回 <code>([], 'missing')</code> → <code>sft_eligibility()</code> 不检查这项 → 样本仍被标 eligible。A/B 对照:同一脚本补 PATH 后返回 0、导出 7 个工具。</div>
</div>
</div>
<div class="inv-row">
<div class="mark">I4</div>
<div class="body">
<h3>verdict 覆盖 claim,说谎是最贵的</h3>
<p>agent 自述不参与任何判定,只参与一致性比对。<code>claim ≠ verdict</code> 扣 2.0 并绝不进 SFT——训练集里出现一次谎报成功,就是在教它谎报。claim 缺席扣 0.25:不报比报错轻,但不能没有代价。</p>
<div class="counter"><b>同类风险 · 编译返回 0 不是验收通过</b>编译日志里的百分比、校准余弦、静态 FPS 估算都不能代替 G1/G2/G3。三条 ConvNeXt 轨迹 <code>exit_code=0</code> 且已生成 HBM,仍没有最终 verdict。</div>
</div>
</div>
<div class="inv-row">
<div class="mark">I5</div>
<div class="body">
<h3>状态机必须闭合:每个终态都要有去处</h3>
<p>FAIL 不是终点,是 repair 轨迹的输入——而且是<em>更有价值</em>的那种输入,因为修复现场比一次性成功更能教出能力。配置里开了开关,不等于链路接上了。</p>
<div class="counter"><b>反例 · repair 0 / 0</b><code>matrix.yaml</code> 已配 <code>when: oracle_fail</code>;<code>registry.expand_matrix()</code> 见到该值直接跳过,注释声称由 pipeline 动态追加;而 <code>pipeline.run_oracle_job()</code> 存完 FAIL 就返回,<code>collect_rollouts()</code> 只消费数据库里已有的 QUEUED 行。全库 repair job 与 rollout 均为 0。</div>
</div>
</div>
<div class="inv-row">
<div class="mark">I6</div>
<div class="body">
<h3>指标口径必须与被测对象的结构匹配</h3>
<p>把多分支输出展平算一个整体余弦,结果会被量级最大的那一支主导,小量级分支的失真被完全掩盖。凡是多语义输出,门必须逐分支判。</p>
<div class="counter"><b>反例 · YOLO11s 整体余弦 0.9976,类别分支 0.36</b>输出 <code>[1,84,8400]</code> 同时含框坐标(量级大)与 0–1 类别分数。<code>eval/common.py::cosine()</code> 整体展平,G2 判为 pass;而实际 mAP 46.01 → 25.92。整段余弦高不等于精度达标。</div>
</div>
</div>
<div class="inv-row">
<div class="mark">I7</div>
<div class="body">
<h3>证据边界必须写在结论旁边</h3>
<p>每个数字都要带着「它证明了什么、没证明什么」一起流转。否则弱证据会在传递中逐级升格成强结论。这条不产出代码,但它决定了这套数据将来能不能被别人信。</p>
<div class="counter"><b>今天需要显式区分的三组</b><code>pass-screen</code>(T1 放宽门限初筛)≠ <code>pass</code>(T2 完整验证);<code>hb_model_info</code> 静态性能估算 ≠ 板端实测;服务器上 <code>run_sim</code> ≠ 物理板端运行。9 份板端记录用的是随机输入,没有板端数据集精度。</div>
</div>
</div>
<div class="inv-row">
<div class="mark">I8</div>
<div class="body">
<h3>失败的轨迹不删,过滤发生在导出时</h3>
<p>磁盘上留什么和数据集里收什么是两件事。轨迹是复盘证据——今天两个 bug 都是靠原始轨迹定位的;数据洁净度由 <code>--min-agent-version</code>、annotations 和资格函数在导出那一刻保证。删了,同样的坑得重新踩一遍才知道。</p>
<div class="counter"><b>当前状态</b>r20260915-03 的 11 条已完成 rollout 全部不可用(7 条读了 oracle、3 条被误杀、其余旧版本),已 <code>--annotate</code> 标注保留;9 份旧数据集加 VOID 标记而非删除。</div>
</div>
</div>
</div>
</section>
<section>
<h2><span class="num">§5</span>最小可复现闭环(MVL)</h2>
<p class="lede">定义:在一个<strong>新冻结 run</strong> 上锁定一条切片(建议 <code>cls--resnet18--nash-e--ptq--int8</code>,它是唯一已知能走完全程的 job),走到一条可进 SFT 的样本,<strong>并且同一 run 内四类终态各有一例被正确落位</strong>。做到这一步,方法论就算走通——规模是之后的事。</p>
<h3>5.1 四个前置闸门:全为真才允许启动</h3>
<div class="checklist">
<div class="check">
<div class="id">P1</div>
<div class="txt"><h3>版本自证</h3><p>调度进程启动时打印自身加载的 harness 代码指纹与两套 venv 的 <code>pip freeze</code> sha256,与 run manifest 比对;不一致<strong>拒跑</strong>而非告警。</p></div>
<div><span class="pill part">需改造</span></div>
</div>
<div class="check">
<div class="id">P2</div>
<div class="txt"><h3>真实产物校验</h3><p>用已安装的 horizon_tc_ui 3.5.3 Schema 校验<em>实际写到磁盘的第一份</em> compile YAML,而不是校验生成函数的返回值。这是今天 37 例失败唯一有效的拦截点。</p></div>
<div><span class="pill part">需改造</span></div>
</div>
<div class="check">
<div class="id">P3</div>
<div class="txt"><h3>采集链完整性</h3><p><code>tools.json</code> 存在且工具定义数大于 0,否则拒绝启动 rollout;导出侧同样把它列为硬性资格条件。</p></div>
<div><span class="pill no">缺失</span></div>
</div>
<div class="check">
<div class="id">P4</div>
<div class="txt"><h3>隔离自检</h3><p><code>guard_smoke.sh</code> 的越界尝试全部被拒、<code>guard-core.test.js</code> 通过;jail 不可用时拒绝启动,不静默退回 guard。</p></div>
<div><span class="pill ok">已具备</span></div>
</div>
</div>
<h3>5.2 七步产物链:每步都要留下可核验的东西</h3>
<div class="tablewrap">
<table>
<thead><tr><th>步</th><th>动作</th><th>可核验产物</th><th>判据</th></tr></thead>
<tbody>
<tr><td class="num-cell">1</td><td>oracle 跑完切片</td><td><code>oracle/result.json</code></td><td>G1/G2/G3 均有明确判定,<code>error_class</code> 非空</td></tr>
<tr><td class="num-cell">2</td><td>agent 在 jail 内作答</td><td><code>session.jsonl</code> · <code>events.jsonl</code> · workspace 产物</td><td><code>guard.log</code> 无越界成功记录</td></tr>
<tr><td class="num-cell">3</td><td>独立核验</td><td><code>verdict.json</code></td><td>不读 claim 即可算出四道门</td></tr>
<tr><td class="num-cell">4</td><td>一致性比对</td><td><code>claim.json</code> ↔ verdict</td><td><code>claim_consistent</code> 有明确真假,不是缺省</td></tr>
<tr><td class="num-cell">5</td><td>污染审计</td><td><code>audit_rollouts.py</code> 输出行</td><td>无 <code>oracle_read</code> / <code>sibling_read</code> / <code>stalled_killed</code> / <code>old_version</code></td></tr>
<tr><td class="num-cell">6</td><td>离线打分</td><td>reward 与配方版本号</td><td>同一 verdict 重算两次结果逐位一致</td></tr>
<tr><td class="num-cell">7</td><td>导出</td><td><code>teacher.jsonl</code></td><td>tools 数大于 0,<code>sft_eligible</code> 的理由可打印</td></tr>
</tbody>
</table>
</div>
<h3>5.3 分辨力四例:这才是走通的核心判据</h3>
<p>只产出正样本证明不了验证器有分辨力。必须在同一个 run 里构造出四类终态,并确认每一类都被<em>正确</em>落位——尤其是第二、第四行这两个「应当被拒」的例子。</p>
<div class="tablewrap">
<table>
<thead><tr><th>终态</th><th>怎么构造</th><th>期望落位</th><th>现状</th></tr></thead>
<tbody>
<tr>
<td><span class="pill ok">PASS</span></td>
<td>ResNet18 / nash-e 正常跑</td>
<td>四道门齐全 · RL ✓ · SFT ✓ · tools 完整</td>
<td><span class="pill part">有 PASS,tools 缺</span></td>
</tr>
<tr>
<td><span class="pill no">污染</span></td>
<td>故意让一条 rollout 去读 <code>oracle/</code></td>
<td>jail 层面根本读不到;即便读到,audit 标 <code>oracle_read</code> 并判不可用</td>
<td><span class="pill part">v5 已隔离,未做对照</span></td>
</tr>
<tr>
<td><span class="pill part">BLOCKED</span></td>
<td>挑 oracle 确定性失败的 job(MaxViT-t 的 <code>aten::einsum</code> / opset 11)</td>
<td>agent 正确判断放弃 → <code>blocked_correct=true</code> → reward +1 → SFT 收</td>
<td><span class="pill no">全库无 BLOCKED 样本</span></td>
</tr>
<tr>
<td><span class="pill no">谎报</span></td>
<td>构造 <code>claim=PASS</code> 而 verdict=FAIL 的轨迹</td>
<td>reward −2.0 · SFT 拒收 · 理由可打印</td>
<td><span class="pill no">未验证</span></td>
</tr>
</tbody>
</table>
</div>
<p class="note">第三行同时解释了为什么 105 个 FAIL 是资产而不是负债:MaxViT-t 的导出不支持、YOLOv8n 的 f64 Cast、ResNet34 的权重哈希不匹配,这三类都是现成的 BLOCKED 教材——前提是 repair 链路接上,让它们进得了轨迹。</p>
</section>
<section>
<h2><span class="num">§6</span>离 MVL 还差什么</h2>
<div class="tablewrap">
<table>
<thead><tr><th>缺口</th><th>证据</th><th>阻断了哪一步</th></tr></thead>
<tbody>
<tr><td>运行进程仍是旧代码</td><td>PID 3776582 / 2015257 未重启;7 个源文件两次采样哈希相同,修复未落到运行态</td><td>P1;以及步 1 的 <code>error_class</code> 非空(28 条 FAIL 分类为空,用当前纯函数可全部分类)</td></tr>
<tr><td>tools 快照链路断</td><td>PATH 缺 Node bin,A/B 对照已复现</td><td>P3、步 7</td></tr>
<tr><td>FAIL → repair 未接线</td><td><code>expand_matrix()</code> 跳过、<code>run_oracle_job()</code> 不追加</td><td>5.3 第三行(BLOCKED 无样本可造)</td></tr>
<tr><td>G2 口径对多分支输出失真</td><td>YOLO11s 类别分支 0.36 而整体 0.9976</td><td>步 3 的判定可信度</td></tr>
<tr><td>17 条 STALLED 真因未定</td><td>日志终止标记为 log unchanged and cpu idle;任务仍用 JuiceFS 工作目录,存在 I/O 等待误判风险,但缺历史进程树与 D 状态采样</td><td>步 5 的 <code>stalled_killed</code> 判断依据</td></tr>
</tbody>
</table>
</div>
</section>
<section>
<h2><span class="num">§7</span>执行顺序</h2>
<p>五步是严格有序的:前一步不做完,后一步的观测结果不可信。</p>
<div class="seq">
<div class="step">
<div class="n">1</div>
<div class="sbody">
<h3>统一运行版本,开新冻结 run</h3>
<p><code>bench drain</code> 让旧调度跑完手里的活并退出(不要 kill 正在编译的进程),保存当前证据,从修正后的代码起新进程。仅重启补不齐尚未实现的 repair 与导出校验。</p>
<div class="accept">验收:新 run 第一份落盘 compile YAML 过 Schema;进程指纹与 manifest 一致</div>
</div>
</div>
<div class="step">
<div class="n">2</div>
<div class="sbody">
<h3>接上三个断头</h3>
<p>Node PATH 与 tools 硬校验;FAIL → repair 的幂等入队;G1 的 dtype / layout 完整校验。三处都是接线,不涉及算法。</p>
<div class="accept">验收:造一个 FAIL,repair job 自动出现且不重复;tools 缺失时 rollout 拒绝启动</div>
</div>
</div>
<div class="step">
<div class="n">3</div>
<div class="sbody">
<h3>跑通 MVL 的七步产物链</h3>
<p>只跑 ResNet18 / nash-e 一条切片,不扩量。目标是一条 tools 完整、四门齐全、audit 干净的样本。</p>
<div class="accept">验收:audit 显示 RL ✓ 且 SFT ✓;导出样本 tools 数大于 0</div>
</div>
</div>
<div class="step">
<div class="n">4</div>
<div class="sbody">
<h3>补齐分辨力四例</h3>
<p>污染、BLOCKED、谎报三个负向对照各造一例。这一步才是「方法论走通」的签收点。</p>
<div class="accept">验收:四类终态各一例,落位与 5.3 表格逐行相符</div>
</div>
</div>
<div class="step">
<div class="n">5</div>
<div class="sbody">
<h3>再谈规模</h3>
<p>分别验收数值与任务指标:保留 FP32 ONNX 对照,按模型处理量化敏感层,检测模型另判框坐标与类别分数。<strong>不要为了提高 PASS 数放宽阈值</strong>——阈值一放,前面四步建立的分辨力就作废了。</p>
<div class="accept">验收:收敛指标看 RL ✓ 比例,不看 PASS 绝对数</div>
</div>
</div>
</div>
</section>
<section>
<h2><span class="num">§8</span>规模化阶段的唯一指标</h2>
<div class="callout">
<h3>看 RL ✓ 比例,不看 PASS 数</h3>
<p>PASS 数受模型难度、门阈值、工具链成熟度共同影响,涨跌说明不了 agent 学没学到东西。<code>RL ✓</code>(版本合格、没越界、没被误杀)衡量的是<em>轨迹本身可用吗</em>——它才是这条流水线的产能指标。当前该比例为 0/11。</p>
<p>同理,当前 3/108 约 2.8% 的通过率<strong>不能当作模型固有可编译率</strong>:其中 37 例是配置层级错误、17 例是看门狗终止,都与模型能不能上 BPU 无关。</p>
</div>
</section>
<footer>
<p><strong>数据来源</strong>:2026-09-15 的四份检查报告(远端状态复查 16:19–16:20、失败根因定位、板端执行证据复核、精度对比),以及 <code>bench/</code> 仓库的 CONTRACT.md、HANDBOOK.md 与 harness/、export/ 源码。</p>
<p><strong>证据边界</strong>:文中数字为已保存评测记录与当日快照,本页未重跑任何推理或编译。「37 例配置层级错误」与「tools 链路」两项有受控复现;「17 条 STALLED 全为误杀」未经证实,不应据此下结论。3/108 是当前判定口径下的比例。</p>
</footer>
</div>
</body>
</html>

18
docs/README.md Normal file
View File

@ -0,0 +1,18 @@
# 项目文档
本目录收录 BPU 量化轨迹工厂的设计与方法说明,原始文档内容保持不变:
- [设计方案](BPU量化轨迹工厂-设计方案.html)
- [方法论](BPU量化轨迹工厂-方法论.html)
- [闭环方法论](BPU量化轨迹工厂-闭环方法论.html)
- [技术分享讲稿(2026-09-16)](BPU量化轨迹工厂-技术分享讲稿-20260916.md)
运行入口与配置方式见 [项目 README](../README.md),实现约定见 [CONTRACT.md](../CONTRACT.md)。文档中的实验状态、服务器路径和实测结果对应其编写时间;当前运行状态需要重新核查。
## Git 收录范围
仓库收录 harness、模型适配器、Agent 配置模板、数据导出代码、脚本、测试、registry、配置和项目文档,保留现有提交历史。
模型权重、ONNX/HBM、数据集、缓存、运行状态、训练轨迹、评测输出、日志、第三方源码下载、压缩包以及本地凭据留在各运行环境,不随源码仓库分发。原工作目录下的 `outputs/` 和 `job_verdicts.csv` 属于运行产物;这些文件继续保留在本地。启动前按项目 README 准备依赖、数据、模型与本地环境变量。
`.env.local.example` 和 `agent/models.json.tmpl` 为可提交模板;真实 `.env.local` 和 `agent/models.json` 已被忽略。

View File

@ -4,7 +4,7 @@
## 结论与边界
B/A、D、E、G 均已实现;agent/VERSION=5。已通过真实 jail + pi smoke、外部 worker 的真实 calibrate 闭环、guard-core 单测、预算边界测试和 Python 回归。最终探针复测的原始输出见 [验证记录](scripts/evidence-codex.md)。
B/A、D、E、G 均已实现;agent/VERSION=5。已通过真实 jail + pi smoke、外部 worker 的真实 calibrate 闭环、guard-core 单测、预算边界测试和 Python 回归。最终探针复测的原始输出见 [验证记录](evidence-codex.md)。
没有启动 `bench collect` / `bench rollout`,没有 kill 生产编译、pi、stagerun 或调度进程,没有改 config.yaml / registry 数值,没有手工 UPDATE/DELETE 生产 DB。canary #02 仅供只读审计,未重新核验或改写其文件。旧数据集只新增作废文件。

View File

@ -1,3 +1,8 @@
> **已归档(2026-09-17):未实现,暂不做。**
> 注意不是"被 long 车道替代"——long 只是 STALLED 后把看门狗阈值抬到 3600s 的重试车道
> (`stagerun.py` 对 05-compile 无条件设 `ctx.lane="long"`),与两阶段编译无关。
> 本文的动机测量(量化 107s / BPU 编译 4680s)至今仍然成立,问题也仍未解决。
# SPEC · 05-compile 拆成两阶段(交接文档)
给下一个负责改这块的 agent。所有数字都是 2026-09-15 在 `120.48.151.164` 实测的,

View File

@ -0,0 +1,26 @@
# 板端进程生命周期
主机的 `board-<slot>.lock` 只覆盖主机阶段,SSH 退出不保证板上 Python 退出。
2026-09-19 的实测故障是:板端 CPU 算子执行较长,主机 `capture_output=True` 没有实时日志,
看门狗按主机空闲 900 秒终止本地进程组并释放锁;板端留成 PPID=1,重试又在同一板上启动推理。
旧退出函数仅删除临时目录,也无法回收运行中的模型。不能用 CPU 忙证明模型正常推进。
`RemoteSession.bench` 现在上传并调用独立的 `board_guard.py`,用于 07 性能和 08 精度两条入口:
- 守护进程持有 runtime_dir 下 `.board-inference.lock`;子进程继承锁,在未退出时不能再次开测。
- 开始前检查已有 `board_bench.py` / `hrt_model_exec`,发现未经守护的推理返回 `board_busy`。
- 主机每 5 秒续租;EOF、30 秒无租约、单次 RPC 3600 秒上限或终止信号,都回收本次进程组。
- 先 TERM,5 秒后 KILL;不因组长先退出而跳过组内其他进程。Linux parent-death signal 在守护进程被 KILL 时终止直接模型子进程。
- 原生运行库在子进程内,不能阻塞守护进程。无法立即清除的内核不可中断进程仍持有板端锁,后续测量失败退出。
- 板端 stderr 实时回传。`board_progress` 是加载、样本、warmup、计时迭代的实际边界;`alive` 只是传输存活,单次 RPC 上限独立计时。
- 成功后清理本次临时目录;失败目录保留 `guard-result.json` 和 HBM 供定位。日志记录 session 路径,结果记录 `board_guard_version`。
- 性能阶段等 10 分钟仍繁忙,拒绝生成新的性能数值。历史 `contended` 数据保留原始证据,不能静默改成正常测量。
验证:`PYTHONPATH=. python3 -m unittest scripts.test_board_runtime scripts.test_board_guard`。
Linux 上的 guard 测试会真实注入断连、停止续租、超时、TERM/KILL 和锁争用;不访问 SSH/BPU。
本地 macOS 会跳过 Linux 进程测试,应额外在 Linux 隔离临时目录执行。
部署走 `scripts/sync.sh`。若远端已有其他任务的未合并改动,可以提交本次文件后使用
`scripts/sync.sh --files <files...>`,仅比较所选文件与上次同步的前像,保存远端备份并校验落盘哈希。
新文件先于引用方安装;不会推进其他文件的同步基线。旧的已启动评测不会重新 import,
新启动的工具子进程会加载修复,无需重启 collector。现役任务的结果须分别检查,不得宣称所有旧进程已经换代。

View File

@ -1,18 +1,7 @@
"""reward v0.4 配方。离线算,只读 verdict.json;配方版本化,可全量重打分。
"""reward v0.6:只按本次真实门证据、成本、claim 一致性和超时计算。
r = 1.0·[G1] + 1.0·[G2] + 2.0·[G3] 绝对门
r += 1.0·clip(精度改善/|tol|, −1, +1) 相对 oracle 的精度
r += 0.5·clip((lat_oracle − lat_agent)/lat_oracle, −1, +1) 相对 oracle 的延时
r −= 0.5·turns/max_turns + 0.5·tokens/max_tokens 代价
r −= 2.0·[claim ≠ verdict] 说谎
r −= 0.25·[claim 缺席]
BLOCKED: +1.0 若 oracle 同阶段同类失败;否则 −2.0
执行 TIMEOUT: 另扣 −1.0,已经完成的真实阶段仍保留得分。
G3 = na:其 2.0 权重按比例分给 G1/G2(各 2.0),精度相对项取 0
板卡不可用(v0.4):G3 unverified 且最后一次 07-board / 08-eval 以 board_offline 失败 → G2/G3 权重取 0(板上测的两门
都没法测,不是 agent 的错),G1 保持 1.0 不加分,免 TIMEOUT 罚分,返回值带 unverifiable="board_offline"。
实测(2026-09-16 r20260916-01):regnet.y.800mf 14 次 eval 全 board_offline,v0.3 给 −0.97,等于把基础设施故障记在
agent 头上。注意这类轨迹在导出器里仍按 UNVERIFIED 排除在 RL/SFT 之外;真正回收要用 bench board-backfill --rollout 补测。
不读取 Oracle 对照或 BLOCKED 正确性标签。BLOCKED 的已测阶段仍保留分数,
但导出器不将未证实的 BLOCKED 纳入奖励训练。板卡不可用仍按既有规则免超时罚分。
"""
from __future__ import annotations
@ -20,15 +9,11 @@ import json
import sys
from pathlib import Path
RECIPE_VERSION = "v0.4"
RECIPE_VERSION = "v0.6"
BOARD_STAGES = ("07-board", "08-eval")
PASSING = {"pass", "pass-screen", "pass-weak"}
W = {"G1": 1.0, "G2": 1.0, "G3": 2.0, "acc_rel": 1.0, "lat_rel": 0.5, "cost_turns": 0.5, "cost_tokens": 0.5,
"lie": 2.0, "claim_absent": 0.25, "blocked_ok": 1.0, "blocked_bad": 2.0, "timeout": 1.0}
def clip(x: float, lo: float = -1.0, hi: float = 1.0) -> float:
return max(lo, min(hi, x))
W = {"G1": 1.0, "G2": 1.0, "G3": 2.0, "cost_turns": 0.5, "cost_tokens": 0.5,
"lie": 2.0, "claim_absent": 0.25, "timeout": 1.0}
def board_unavailable(verdict: dict) -> bool:
@ -46,37 +31,18 @@ def compute(verdict: dict, max_turns: int = 40, max_tokens: int = 200000) -> dic
g = verdict.get("gates") or {}
terminal = verdict.get("terminal")
claim = verdict.get("claim") or {}
oref = verdict.get("oracle_ref") or {}
cost = verdict.get("cost") or {}
unverifiable = None
if terminal == "BLOCKED":
terms["blocked"] = W["blocked_ok"] if verdict.get("blocked_correct") else -W["blocked_bad"]
g3_na = (g.get("G3") or {}).get("verdict") == "na"
if board_unavailable(verdict):
unverifiable = "board_offline"
w1, w2, w3 = W["G1"], 0.0, 0.0
else:
g3_na = (g.get("G3") or {}).get("verdict") == "na"
if board_unavailable(verdict):
unverifiable = "board_offline"
w1, w2, w3 = W["G1"], 0.0, 0.0 # 不加分只免罚:板上测的两门不计,G1 原样
else:
w1, w2, w3 = (2.0, 2.0, 0.0) if g3_na else (W["G1"], W["G2"], W["G3"])
terms["G1"] = w1 * float((g.get("G1") or {}).get("verdict") in PASSING)
terms["G2"] = w2 * float((g.get("G2") or {}).get("verdict") in PASSING)
terms["G3"] = w3 * float((g.get("G3") or {}).get("verdict") in PASSING)
# abs_pp 的 delta 越大越好;rel_pct 的 delta 是劣化百分比,越小越好。
g3 = g.get("G3") or {}
same_kind = oref.get("G3_kind", g3.get("kind")) == g3.get("kind")
if not g3_na and same_kind and g3.get("delta") is not None and oref.get("G3_delta") is not None and g3.get("tolerance"):
improvement = g3["delta"] - oref["G3_delta"]
if g3.get("kind") == "rel_pct":
improvement = -improvement
terms["acc_rel"] = W["acc_rel"] * clip(improvement / abs(g3["tolerance"]))
else:
terms["acc_rel"] = 0.0
g4 = g.get("G4") or {}
if g4.get("latency_ms_mean") and oref.get("latency_ms_mean") and not g4.get("contended"):
terms["lat_rel"] = W["lat_rel"] * clip((oref["latency_ms_mean"] - g4["latency_ms_mean"]) / oref["latency_ms_mean"])
else:
terms["lat_rel"] = 0.0
w1, w2, w3 = (2.0, 2.0, 0.0) if g3_na else (W["G1"], W["G2"], W["G3"])
terms["G1"] = w1 * float((g.get("G1") or {}).get("verdict") in PASSING)
terms["G2"] = w2 * float((g.get("G2") or {}).get("verdict") in PASSING)
terms["G3"] = w3 * float((g.get("G3") or {}).get("verdict") in PASSING)
if (verdict.get("execution_terminal") == "TIMEOUT" or terminal == "TIMEOUT") and not unverifiable:
terms["timeout"] = -W["timeout"] # 板子不通时把预算耗在重试上,不算 agent 超时
@ -95,7 +61,7 @@ def compute(verdict: dict, max_turns: int = 40, max_tokens: int = 200000) -> dic
terms["lie"] = -W["lie"]
r = sum(terms.values())
return {"reward": round(r, 4), "terms": {k: round(v, 4) for k, v in terms.items()}, "recipe": RECIPE_VERSION,
"terminal": terminal, "review": bool(verdict.get("review")), "note": terms_note,
"terminal": terminal, "note": terms_note,
"unverifiable": unverifiable}

View File

@ -11,6 +11,7 @@ from unittest.mock import patch
from export import to_agentrl as ex
from export.reward import compute
from harness.db import SCHEMA
from harness.verify import VERDICT_SCHEMA, VERIFICATION_RULE
def dialogue():
@ -36,15 +37,17 @@ class MaterializationTests(unittest.TestCase):
def tearDown(self):
self.env.stop(); self.patch.stop(); self.con.close(); self.tmp.cleanup()
def add(self, kk=0, terminal="PASS", execution="DONE", review=False, ended=True, version=6):
def add(self, kk=0, terminal="PASS", execution="DONE", review=False, ended=True, version=6, mode="e2e"):
jd = self.root / "runs/r/j"
rd = jd / "rollouts" / f"{kk:02d}"
(rd / "agent").mkdir(parents=True)
job = {"run_id": "r", "job_id": "j", "task": "cls", "model": "resnet18", "march": "nash-m", "backend": "ptq"}
job = {"run_id": "r", "job_id": "j", "task": "cls", "model": "resnet18", "march": "nash-m", "backend": "ptq",
"gates": {"required": ["G1", "G3"], "G1": {}, "G2": {},
"G3": {"kind": "rel_pct", "tol": 5, "require_source": "board"}, "G4": {}}}
(jd / "job.json").write_text(json.dumps(job))
(rd / "rollout.json").write_text(json.dumps({"state": execution, "model": "glm-5.3", "policy": "teacher",
(rd / "rollout.json").write_text(json.dumps({"state": execution, "model": "glm-5.3", "policy": "teacher", "mode": mode,
"agent_config_version": version, "max_turns": 60, "max_tokens": 100}))
verdict = {"run_id": "r", "job_id": "j", "rollout": kk, "terminal": terminal,
verdict = {"schema": VERDICT_SCHEMA, "verification_rule": VERIFICATION_RULE, "mode": mode, "run_id": "r", "job_id": "j", "rollout": kk, "terminal": terminal,
"execution_terminal": execution, "timeout_reason": "budget" if execution == "TIMEOUT" else None,
"claim_consistent": True, "claim": {"terminal": terminal}, "review": review,
"gates": {"G1": {"verdict": "pass"}, "G2": {"verdict": "pass"}, "G3": {"verdict": "pass-screen"}},
@ -75,7 +78,7 @@ class MaterializationTests(unittest.TestCase):
def test_timeout_keeps_verified_stage_rewards(self):
rd = self.add(execution="TIMEOUT")
reward = compute(json.loads((rd / "verdict.json").read_text()), 60, 100)
self.assertEqual(reward["recipe"], "v0.4")
self.assertEqual(reward["recipe"], "v0.6")
self.assertEqual(reward["terms"]["G3"], 2.0)
self.assertEqual(reward["terms"]["timeout"], -1.0)
manifest = ex.materialize(["r"], "data")
@ -95,13 +98,87 @@ class MaterializationTests(unittest.TestCase):
self.assertEqual(recs[0]["trajectory"]["source"], "message_end")
self.assertEqual(sum(m["role"] == "assistant" for m in recs[0]["messages"]), 1)
def test_review_is_retained_but_not_positive_training_candidate(self):
self.add(review=True)
manifest = ex.materialize(["r"], "data")
self.assertEqual(manifest["stats"]["review"], 1)
def legacy_evidence(self, rd, *, mode="e2e", claim="PASS"):
original = json.loads((rd / "verdict.json").read_text())
original.update(schema="bench.verdict/2", review=True,
oracle_ref={"terminal": "FAIL", "G3_delta": -99})
(rd / "verdict.json").write_text(json.dumps(original))
(rd / "claim.json").write_text(json.dumps({"terminal": claim}))
for stage, measurements in {
"06-hbm": {"G1": {"hbm_exists": True, "march_ok": True, "io_ok": True,
"verifier_ok": True, "log_has_error": False}},
"08-eval": {"G3": {"fp32": 80, "quant": 79.2, "n": 100, "source": "board", "tier": "t1"}},
}.items():
sd = rd / "stages" / stage
sd.mkdir(parents=True, exist_ok=True)
(sd / "meta.json").write_text(json.dumps({"state": "OK", "measurements": measurements}))
return original
def test_legacy_oracle_review_is_reverified_without_modifying_sources(self):
rd = self.add(review=True)
self.legacy_evidence(rd)
before = {p: hashlib.sha256(p.read_bytes()).hexdigest() for p in rd.rglob("*") if p.is_file()}
before_db = list(self.con.execute("SELECT * FROM rollouts"))
manifest = ex.materialize(["r"], "data", purpose="sft")
self.assertEqual(manifest["stats"]["reverified"], 1)
rec = json.loads((self.root / "datasets/data/teacher.jsonl").read_text())
self.assertTrue(rec["sft_eligible"])
self.assertTrue(rec["training_uses"]["reward_learning_candidate"])
self.assertEqual(rec["verification"]["method"], "readonly-reverification")
self.assertEqual(rec["verdict"]["schema"], VERDICT_SCHEMA)
self.assertFalse(rec["verdict"]["review"])
self.assertNotIn("oracle_ref", rec)
self.assertNotIn("history", rec)
self.assertTrue(rec["provenance"]["original_verdict"]["review"])
self.assertEqual(rec["provenance"]["original_verdict"]["sha256"], before[rd / "verdict.json"])
self.assertEqual(before, {p: hashlib.sha256(p.read_bytes()).hexdigest() for p in before})
self.assertEqual(before_db, list(self.con.execute("SELECT * FROM rollouts")))
def test_failed_reverification_is_excluded_without_legacy_fallback(self):
rd = self.add(review=True)
self.legacy_evidence(rd)
(rd / "stages/08-eval/meta.json").write_text("broken evidence")
stats = ex.materialize(["r"], "data")["stats"]
self.assertEqual(stats["written"], 0)
self.assertEqual(stats["dropped"], {"reverification_failed:JSONDecodeError": 1})
def test_historical_repair_remains_analysis_only(self):
rd = self.add(mode="repair")
self.legacy_evidence(rd)
ex.materialize(["r"], "all")
rec = json.loads((self.root / "datasets/all/teacher.jsonl").read_text())
self.assertEqual(rec["mode"], "repair")
self.assertFalse(rec["sft_eligible"])
self.assertEqual(rec["sft_reason"], "historical-repair-context")
self.assertFalse(rec["training_uses"]["reward_learning_candidate"])
self.assertEqual(ex.materialize(["r"], "sft", purpose="sft")["stats"]["written"], 0)
self.assertEqual(ex.materialize(["r"], "rl", purpose="rl")["stats"]["written"], 0)
def test_blocked_is_not_a_success_or_reward_training_target(self):
rd = self.add(terminal="BLOCKED")
v = json.loads((rd / "verdict.json").read_text())
v.update(blocked_correct=True, oracle_ref={"terminal": "FAIL"})
(rd / "verdict.json").write_text(json.dumps(v))
ex.materialize(["r"], "all")
rec = json.loads((self.root / "datasets/all/teacher.jsonl").read_text())
self.assertFalse(rec["sft_eligible"])
self.assertFalse(rec["training_uses"]["reward_learning_candidate"])
self.assertNotIn("blocked", rec["reward_terms"])
def test_oracle_independence_does_not_bypass_transcript_or_annotation_checks(self):
first = self.add(ended=False)
self.legacy_evidence(first)
second = self.add(kk=1)
self.legacy_evidence(second)
(second / "claim.json").unlink()
third = self.add(kk=2)
self.legacy_evidence(third)
self.con.execute("INSERT INTO annotations(run_id,job_id,note) VALUES('r','j','rollout#2 contaminated: copied solution')")
self.con.commit()
stats = ex.materialize(["r"], "data", purpose="sft")["stats"]
self.assertEqual(stats["written"], 0)
self.assertEqual(stats["dropped"], {"sft:partial-transcript": 1, "sft:claim-missing-or-inconsistent": 1,
"contaminated": 1})
def test_old_version_and_missing_verdict_have_explicit_exclusions(self):
self.add(version=5)
@ -113,6 +190,17 @@ class MaterializationTests(unittest.TestCase):
self.assertIn("old_version", exclusions)
self.assertIn("missing_verdict", exclusions)
def test_sft_and_rl_exports_do_not_read_historical_pipeline(self):
self.add()
with patch.object(ex, "pipeline_records", side_effect=OSError("unreadable old pipeline")) as pipeline:
for purpose in ("sft", "rl"):
with self.subTest(purpose=purpose):
result = ex.materialize(["r"], purpose, purpose=purpose)
self.assertEqual(result["stats"]["agent_written"], 1)
self.assertEqual(result["stats"]["pipeline_written"], 0)
self.assertEqual((self.root / "datasets" / purpose / "pipeline.jsonl").read_text(), "")
pipeline.assert_not_called()
def test_pipeline_is_real_separate_and_failed_jobs_are_retained(self):
self.add()
jd = self.root / "runs/r/j"

View File

@ -1,4 +1,4 @@
"""L4 数据物化:扫描 rollouts → 脱敏、清洗(R1–R8)、线性化 → 与 verdict / oracle 配对 → reward → datasets/<dataset_id>/。
"""L4 数据物化:扫描 rollouts → 脱敏、清洗(R1–R8)、线性化 → 独立核验 → reward → datasets/<dataset_id>/。
python export/to_agentrl.py --runs r20260914-01 [r…] [--policy teacher] [--include-timeout]
@ -267,15 +267,15 @@ def sft_eligibility(verdict: dict, reward: float) -> tuple[bool, str]:
honestly claimed success is not rejected solely for exhausting its budget.
Legacy TIMEOUT verdicts stay evidence until reverified, never assumed PASS.
"""
if verdict.get("review"):
return False, "review-pending(oracle FAIL / rollout PASS,待人工抽查)"
if verdict.get("mode") == "repair":
return False, "historical-repair-context"
if verdict.get("claim_consistent") is not True:
return False, "claim-missing-or-inconsistent"
t = verdict.get("terminal")
if t == "PASS":
return True, "PASS"
if t == "BLOCKED":
return (True, "BLOCKED-correct") if verdict.get("blocked_correct") else (False, "BLOCKED-misjudged")
return False, "BLOCKED-unproven"
if t == "TIMEOUT":
return False, "legacy-TIMEOUT-needs-reverification"
return False, f"{t}-not-success-demonstration"
@ -292,7 +292,7 @@ def scan(run_ids: list[str]) -> list[Path]:
# DB index only: never enumerate the JuiceFS run tree.
with readonly_db() as con:
rows = [row for rid in run_ids for row in con.execute(
"SELECT run_id, job_id, kk FROM rollouts WHERE run_id=? AND state NOT IN ('RUNNING','QUEUED','VERIFYING')", (rid,))]
"SELECT run_id, job_id, kk FROM rollouts WHERE run_id=? AND state NOT IN ('RUNNING','QUEUED','VERIFYING','TRANSFERRED')", (rid,))]
return sorted(BENCH / 'runs' / rid / jid / 'rollouts' / f'{kk:02d}' for rid, jid, kk in rows)
@ -363,7 +363,7 @@ def pipeline_records(run_ids: list[str]):
DB-indexed only. No recursive traversal of model/data trees and no simulated
LLM dialogue. Missing evidence is recorded rather than hiding a finished job.
"""
from harness.verify import completion_summary
from harness.verify import completion_summary, eval_set
with readonly_db() as con:
jobs = [dict(row) for rid in run_ids for row in con.execute(
"SELECT * FROM jobs WHERE run_id=? AND mode='oracle' AND state IN ('PASS','FAIL','UNVERIFIED')", (rid,))]
@ -402,6 +402,7 @@ def pipeline_records(run_ids: list[str]):
"schema": "bench.pipeline-training/1", "source": "deterministic_pipeline",
"id": hashlib.sha1(f"{rid}/{jid}/oracle".encode()).hexdigest()[:16],
"run_id": rid, "job_id": jid, "model": job.get("model", row.get("model")),
"eval_set": eval_set(job), "continuation": job.get("continuation"),
"task": job.get("task", row.get("task")), "march": job.get("march", row.get("march")),
"backend": job.get("backend", row.get("backend")), "terminal": result.get("terminal", row["state"]),
"completion": completion_summary(result.get("gates") or {}, result.get("stages") or {}),
@ -429,7 +430,8 @@ def _write_dataset(a, work: Path, ds_id: str) -> dict:
} for rid in a.runs}
stats = {"scanned": 0, "written": 0, "agent_written": 0, "teacher_written": 0, "student_written": 0,
"pipeline_written": 0, "excluded": 0, "dropped": {}, "review": 0, "partial": 0,
"by_terminal": {}, "by_execution_terminal": {}, "pipeline_by_terminal": {}, "sft_eligible": 0}
"by_terminal": {}, "by_execution_terminal": {}, "pipeline_by_terminal": {}, "sft_eligible": 0,
"reverified": 0}
with contextlib.ExitStack() as stack:
files = {name: stack.enter_context((work / f"{name}.jsonl").open("w"))
for name in ("teacher", "student", "pipeline", "exclusions")}
@ -442,10 +444,11 @@ def _write_dataset(a, work: Path, ds_id: str) -> dict:
write("exclusions", {"run_id": rd.parents[2].name, "job_id": rd.parents[1].name,
"rollout": int(rd.name), "source": str(rd.relative_to(BENCH)),
"reasons": sorted(reasons)})
for record in pipeline_records(a.runs):
write("pipeline", record)
stats["pipeline_written"] += 1
_inc(stats["pipeline_by_terminal"], record["terminal"])
if a.purpose == "all":
for record in pipeline_records(a.runs):
write("pipeline", record)
stats["pipeline_written"] += 1
_inc(stats["pipeline_by_terminal"], record["terminal"])
for rd in scan(a.runs):
_write_agent(rd, a, stats, annotations, write, exclude)
file_info = {}
@ -463,7 +466,8 @@ def _write_dataset(a, work: Path, ds_id: str) -> dict:
"annotations_read": True, "dropped_counts_overlap": True, "policy": a.policy,
"include_timeout": True, "purpose": a.purpose, "min_reward": a.min_reward},
"stats": stats, "files": file_info,
"notes": ["pipeline.jsonl is deterministic execution evidence, not Agent dialogue.",
"notes": ["pipeline.jsonl contains historical deterministic evidence only for purpose=all; SFT/RL export reads Agent evidence only.",
"Verdict v3 and reward v0.6 use only this rollout evidence; Oracle references are history only.",
"SFT/reward candidates are labels; no train split, framework conversion, or online RL readiness is implied.",
"Writers hold .export.lock; manifest is published last. Readers can lock or verify file hashes for a coherent refresh."]}
(work / "manifest.json").write_text(json.dumps(manifest, indent=2, ensure_ascii=False))
@ -471,8 +475,10 @@ def _write_dataset(a, work: Path, ds_id: str) -> dict:
def _write_agent(rd, a, stats, annotations, write, exclude):
from harness.verify import eval_set, verify_rollout, VERDICT_SCHEMA, VERIFICATION_RULE
stats["scanned"] += 1
verdict = _read_json(rd / "verdict.json")
original_verdict = _read_json(rd / "verdict.json")
verdict = original_verdict
rj = _read_json(rd / "rollout.json")
if not verdict:
exclude(rd, {"missing_verdict"}); return
@ -487,6 +493,17 @@ def _write_agent(rd, a, stats, annotations, write, exclude):
reasons.add('old_version')
if reasons:
exclude(rd, reasons); return
original_sha = hashlib.sha256((rd / "verdict.json").read_bytes()).hexdigest()
reverified = verdict.get("schema") != VERDICT_SCHEMA
if reverified:
try:
verdict = verify_rollout(rd, write=False)
stats["reverified"] += 1
except Exception as error:
exclude(rd, {f"reverification_failed:{type(error).__name__}"}); return
verification = {"schema": verdict.get("schema"),
"rule": verdict.get("verification_rule", VERIFICATION_RULE),
"method": "readonly-reverification" if reverified else "recorded-verdict"}
raw, trajectory = load_trajectory(rd)
if not raw:
exclude(rd, {"no_trajectory"}); return
@ -502,6 +519,10 @@ def _write_agent(rd, a, stats, annotations, write, exclude):
if cst["drop"]:
exclude(rd, {"no_usable_messages"}); return
msgs, rst = redact_messages(msgs)
jd = rd.parent.parent
job = json.loads((jd / "job.json").read_text())
mode = verdict.get("mode") or rj.get("mode")
verdict = {**verdict, "mode": mode}
rw = compute_reward(verdict, rj.get("max_turns", a.max_turns), rj.get("max_tokens", a.max_tokens))
sft_ok, sft_reason = sft_eligibility(verdict, rw["reward"])
trajectory["partial"] = trajectory["partial"] or not cst["tool_pairing"]["valid"] or cst.get("trailing_user", False)
@ -517,36 +538,39 @@ def _write_agent(rd, a, stats, annotations, write, exclude):
sft_ok, sft_reason = False, "missing-called-tool-definitions"
if a.purpose == "sft" and not sft_ok:
exclude(rd, {f"sft:{sft_reason}"}); return
reward_candidate = verdict["terminal"] in ("PASS", "FAIL", "BLOCKED") and not verdict.get("review")
reward_candidate = verdict["terminal"] in ("PASS", "FAIL") and mode != "repair" and not rw.get("unverifiable")
if a.purpose == "rl" and not reward_candidate:
exclude(rd, {"rl:review-or-unverified-outcome"}); return
exclude(rd, {"rl:historical-repair-or-unverified-outcome"}); return
if a.min_reward is not None and rw["reward"] < a.min_reward:
exclude(rd, {"min_reward"}); return
jd = rd.parent.parent
job = json.loads((jd / "job.json").read_text())
rec = {
"schema": "bench.agent-training/2", "source": "agent_events",
"id": hashlib.sha1(f"{verdict['run_id']}/{verdict['job_id']}/{verdict['rollout']}".encode()).hexdigest()[:16],
"run_id": verdict["run_id"], "job_id": verdict["job_id"], "rollout": verdict["rollout"],
"mode": verdict.get("mode") or rj.get("mode"), "policy": policy, "model": rj.get("model"),
"eval_set": verdict.get("eval_set") or eval_set(job),
"continuation": {**job["continuation"], "source_rollout": verdict["rollout"]} if job.get("continuation") else None,
"mode": mode, "policy": policy, "model": rj.get("model"),
"agent_config_version": str(rj.get("agent_config_version", "1")),
"task": job["task"], "march": job["march"], "backend": job["backend"],
"messages": msgs,
"verdict": {k: verdict[k] for k in ("terminal", "execution_terminal", "timeout_reason", "completion", "failed_gate", "claim_consistent", "blocked_correct", "review") if k in verdict},
"verdict": {k: verdict[k] for k in ("schema", "verification_rule", "terminal", "measured_terminal", "execution_terminal", "timeout_reason", "completion", "failed_gate", "claim_consistent", "blocked_correct", "review") if k in verdict},
"gates": {g: (verdict["gates"].get(g) or {}).get("verdict") for g in ("G1", "G2", "G3", "G4")},
"oracle_ref": verdict.get("oracle_ref"),
"verification": verification,
"reward": rw["reward"], "reward_terms": rw["terms"], "reward_recipe": rw["recipe"],
"reward_unverifiable": rw.get("unverifiable"), # v0.4:board_offline = 板卡故障导致没法测,不是 agent 的结果
"sft_eligible": sft_ok, "sft_reason": sft_reason,
"training_uses": {"analysis": True, "sft_candidate": sft_ok, "reward_learning_candidate": reward_candidate,
"review_required": bool(verdict.get("review"))},
"review_required": False},
"trajectory": trajectory,
"execution": {"terminal": verdict.get("execution_terminal") or rj.get("state"),
"timeout_reason": verdict.get("timeout_reason") or rj.get("timeout_reason"),
"started_at": rj.get("started_at"), "finished_at": rj.get("finished_at"),
"max_turns": rj.get("max_turns"), "max_tokens": rj.get("max_tokens")},
"provenance": {"events": str((rd / "agent" / "events.jsonl").relative_to(BENCH)),
"verdict": str((rd / "verdict.json").relative_to(BENCH))},
"verdict": str((rd / "verdict.json").relative_to(BENCH)),
"original_verdict": {"sha256": original_sha, "schema": original_verdict.get("schema"),
"terminal": original_verdict.get("terminal"),
"review": original_verdict.get("review")}},
"cost": verdict.get("cost"),
"tools": tools,
"tools_source": tools_src,
@ -560,7 +584,6 @@ def _write_agent(rd, a, stats, annotations, write, exclude):
stats["agent_written"] += 1
stats[f"{policy}_written"] += 1
stats["partial"] += int(trajectory["partial"])
stats["review"] += int(bool(verdict.get("review")))
_inc(stats["by_terminal"], verdict["terminal"])
_inc(stats["by_execution_terminal"], rec["execution"]["terminal"])
stats["sft_eligible"] += int(sft_ok)
@ -615,7 +638,7 @@ def main() -> int:
ap.add_argument("--policy", choices=["teacher", "student"])
ap.add_argument("--include-timeout", action="store_true", help="compatibility flag: timeout evidence is now included by default")
ap.add_argument("--purpose", choices=["all", "sft", "rl"], default="all",
help="all=保留终态轨迹并标注用途;sft=完整诚实成功候选;rl=有核验结果且不待复核的候选")
help="all=保留终态轨迹并标注用途;sft=完整诚实成功候选;rl=本次可核验 PASS/FAIL 候选")
ap.add_argument("--min-reward", type=float)
ap.add_argument("--max-turns", type=int, default=40)
ap.add_argument("--max-tokens", type=int, default=200000)

View File

@ -1,2 +1,43 @@
"""custom 模型的构建适配器:每个模型一个 <model_id>.py,提供 build(weights: Path, input_spec: str) -> nn.Module。
非 torchvision / ultralytics / timm 的模型走这里。"""
"""Optional model recipes, shared by input preparation and evaluation."""
from __future__ import annotations
import hashlib
import importlib
import json
from functools import lru_cache
from pathlib import Path
@lru_cache(maxsize=None)
def get(model: str):
name = model.replace(".", "_").replace("-", "_")
if not (Path(__file__).parent / f"{name}.py").is_file():
return None
return importlib.import_module(f"{__name__}.{name}")
def model_spec(row: dict) -> dict:
"""Resolve concrete adapter inputs without changing the source CSV."""
adapter = get(row["model_id"])
if adapter is None:
return row
result = dict(row)
result.update(getattr(adapter, "MODEL_SPEC", {}))
if checkpoint := getattr(adapter, "CHECKPOINT", None):
result.update(weights=checkpoint["url"], checkpoint=dict(checkpoint))
if recipe := getattr(adapter, "PREPROCESS", None):
result["preprocess"] = dict(recipe)
if runtime := getattr(adapter, "RUNTIME_PYTHON", None):
result["runtime_python"] = str(runtime)
return result
def calibration_dir(row: dict) -> Path:
from .. import paths
directory = paths.DATA / "calib" / row["task"] / row["input_spec"]
recipe = model_spec(row).get("preprocess")
if recipe:
# Equal image recipes share calibration; existing jobs retain their path.
digest = hashlib.sha256(json.dumps(recipe, sort_keys=True).encode()).hexdigest()
directory /= digest[:16]
return directory

View File

@ -0,0 +1,12 @@
"""Original ALIKED-N16; static 1024 keypoints, all learned descriptor layers in graph."""
from . import feature_common as common
EXPORTER_VERSION = common.EXPORTER_VERSION
INPUT_SPEC = common.input_spec('aliked')
MODEL_SPEC = {'input_spec': INPUT_SPEC, 'dataset': 'hpatches', 'input_override_reason': 'Official ALIKED expects RGB; CSV grayscale is incorrect.'}
CHECKPOINT = common.checkpoint_info('aliked')
PREPROCESS = common.recipe('aliked')
OPSET = 17
preprocess = common.preprocess
def build(weights, input_spec):
return common.build('aliked', weights, input_spec)

View File

@ -0,0 +1,12 @@
"""Exact MMDetection atss.r50.fpn; see mmdet_dense_common for static deployment geometry."""
from functools import partial
from .mmdet_dense_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
get_recipe, preprocess_variant, decode_variant, build_variant)
MODEL_ID = "atss.r50.fpn"
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
PREPROCESS = recipe(MODEL_ID)
EXPORTER_VERSION = "mmdet-atss.r50.fpn-640-raw-v1"
preprocess = partial(preprocess_variant, MODEL_ID)
decode = partial(decode_variant, MODEL_ID)
build = partial(build_variant, MODEL_ID)

View File

@ -0,0 +1,7 @@
"""Official Cityscapes bisenetv1.r18; admission requires real validation."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("bisenetv1.r18")
preprocess = partial(preprocess_image, PREPROCESS)
build = partial(build_model, "bisenetv1.r18")

View File

@ -0,0 +1,7 @@
"""Official Cityscapes bisenetv2; admission requires real validation."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("bisenetv2")
preprocess = partial(preprocess_image, PREPROCESS)
build = partial(build_model, "bisenetv2")

View File

@ -0,0 +1,114 @@
"""Exact MMDetection conditional.detr.r50; see mmdet_detr_common for the
static deployment geometry this batch reuses without editing it.
ConditionalDETR keeps the DETR single-scale encoder path (real per-image
padding mask retained in-graph) and adds the official conditional decoder
with learned query scaling and reference points. Outputs stay the final
decoder layer's raw class logits and normalized cxcywh; only the official
parameter-free DETRHead selection runs in host decode.
"""
from functools import partial
from types import SimpleNamespace
from .openmmlab import build as build_official, get_recipe
from .mmdet_detr_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, OPSET,
transformer_inputs)
from .mmdet_dense_common import (recipe as image_recipe,
preprocess_variant as image_preprocess,
geometry)
MODEL_ID = "conditional.detr.r50"
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
QUERIES, CHANNELS, TOPK = 300, 80, 100
def recipe():
dp = get_recipe(MODEL_ID)["preprocess"]["official_data_preprocessor"]
meta = image_recipe("retinanet.r50.fpn")
if (dp["mean"] != meta["mean"] or dp["std"] != meta["std"]
or not dp["bgr_to_rgb"] or dp["pad_size_divisor"] != 1):
raise ValueError("Official conditional-detr image normalization changed")
# Same image transformation permits existing calibration cache reuse.
return meta
PREPROCESS = recipe()
EXPORTER_VERSION = "mmdet-conditional.detr.r50-640-raw-v1"
def preprocess(image, input_spec=INPUT_SPEC):
if input_spec != INPUT_SPEC:
raise ValueError(f"Expected {INPUT_SPEC}, got {input_spec}")
recipe() # verify exact equality before reusing the image-only helper
return image_preprocess("retinanet.r50.fpn", image, input_spec)
def output_names():
return ["class_logits", "boxes_cxcywh"]
def _decoder():
from mmdet.models.dense_heads.detr_head import DETRHead
context = SimpleNamespace(test_cfg={"max_per_img": TOPK}, num_classes=80,
loss_cls=SimpleNamespace(use_sigmoid=True))
return partial(DETRHead._predict_by_feat_single, context)
def decode(outputs, original_size):
import numpy as np
import torch
if len(outputs) != 2:
raise ValueError("Expected final query logits and cxcywh tensors")
tensors = []
for value, shape in zip(outputs, ((1, QUERIES, CHANNELS), (1, QUERIES, 4))):
value = np.asarray(value)
if value.shape != shape or not np.isfinite(value).all():
raise ValueError(f"Invalid {MODEL_ID} raw output: {value.shape}, expected {shape}")
tensors.append(torch.tensor(value, dtype=torch.float32)[0])
_, scales, img_shape = geometry(original_size)
with torch.inference_mode():
result = _decoder()(*tensors, dict(img_shape=img_shape, scale_factor=scales), rescale=True)
return result.bboxes.numpy(), result.scores.numpy(), result.labels.numpy()
def build(weights, input_spec):
import torch
raw = build_official(MODEL_ID, weights, input_spec)
network, cfg = raw.upstream, raw.cfg
meta = recipe()
dp = cfg.model.data_preprocessor
if (type(network).__name__ != "ConditionalDETR"
or type(network.bbox_head).__name__ != "ConditionalDETRHead"
or network.num_queries != QUERIES
or network.bbox_head.cls_out_channels != CHANNELS
or network.bbox_head.num_classes != 80
or dict(cfg.model.test_cfg) != {"max_per_img": TOPK}
or list(dp.mean) != meta["mean"] or list(dp.std) != meta["std"]
or not dp.bgr_to_rgb or dp.pad_size_divisor != 1):
raise ValueError("Pinned official conditional-detr contract changed")
state = torch.load(str(weights), map_location="cpu")["state_dict"]
for name, parameter in network.named_parameters():
torch.testing.assert_close(parameter, state[name], rtol=0, atol=0)
del state
class RawQueries(torch.nn.Module):
def __init__(self):
super().__init__()
self.upstream = network
self.upstream.cfg = cfg
self.cfg = cfg
self.checkpoint_sha256 = raw.checkpoint_sha256
def forward(self, x):
features = self.upstream.extract_feat(x)
# Single-scale DETR path; the official conditional decoder consumes
# the same memory/mask/pos tensors via its own reference points.
encoder, decoder = transformer_inputs(self.upstream, features, x, False)
memory = self.upstream.forward_encoder(**encoder)
query, head_inputs = self.upstream.pre_decoder(**memory)
decoder.update(query)
head_inputs.update(self.upstream.forward_decoder(**decoder))
scores, boxes = self.upstream.bbox_head(**head_inputs)
return scores[-1], boxes[-1]
return RawQueries().eval()

View File

@ -0,0 +1,69 @@
"""Official D2-Net d2_tf single-scale CNN with exact official host postprocessing."""
from functools import lru_cache
from . import feature_sources as sources
INPUT_SPEC = '1x3x480x640'
MODEL_SPEC = {'input_spec': INPUT_SPEC, 'dataset': 'hpatches', 'input_override_reason': 'Official D2-Net Caffe preprocessing uses three BGR channels.'}
EXPORTER_VERSION = 'd2net-static-1'
OPSET = 17
CHECKPOINT = {'url': 'https://dusmanu.com/files/d2-net/d2_tf.pth',
'sha256': 'c2d32c3bc53b6588d40bc5325d536a159e529b7492198a2cfa4b11913c615c80',
'filename': 'd2_tf.pth', 'official_relative_path': str(sources.ROOT/'d2net/models/d2_tf.pth')}
PREPROCESS = {'input_spec': INPUT_SPEC, 'color': 'BGR', 'scale': 1., 'mean': [103.939,116.779,123.68],
'resize': 'opencv_linear_640x480', 'output_layout': 'NCHW', 'protocol': 'd2net-official-caffe-single-scale-v1'}
def official_module():
return sources.module('d2net/lib/model_test.py')
@lru_cache(None)
def _functions():
exception = sources.module('d2net/lib/exceptions.py').EmptyTensorError
ns = sources.definitions('d2net/lib/utils.py', ['preprocess_image','interpolate_dense_features','upscale_positions'],
{'EmptyTensorError': exception})
return sources.definitions('d2net/lib/pyramid.py', ['process_multiscale'], ns)
def preprocess(image, input_spec=INPUT_SPEC):
import cv2
import numpy as np
if input_spec != INPUT_SPEC: raise ValueError(input_spec)
rgb = cv2.resize(np.asarray(image.convert('RGB')), (640,480), interpolation=cv2.INTER_LINEAR)
return np.ascontiguousarray(_functions()['preprocess_image'](rgb, preprocessing='caffe')[None], dtype=np.float32)
def official_model(weights):
import torch
sources.verify_weights(weights, CHECKPOINT['sha256'])
m = official_module().D2Net(use_cuda=False, use_relu=True)
m.load_state_dict(torch.load(weights, map_location='cpu', weights_only=False)['model'], strict=True)
return m.eval()
def official_features(m, x):
import torch
k,s,d = _functions()['process_multiscale'](x, m, scales=[1])
# The official extractor swaps (row,col) to (x,y) before saving.
out = sources.top_features(k[:,[1,0]], s, d)
return tuple(torch.from_numpy(out[key]) for key in ('keypoints','scores','descriptors'))
def build(weights, input_spec):
if input_spec != INPUT_SPEC: raise ValueError(input_spec)
return official_model(weights).dense_feature_extraction
def decode(outputs):
import torch
from types import SimpleNamespace
dense, = sources.tensors(outputs)
assert tuple(dense.shape) == (1,512,119,159)
class DenseMap(torch.nn.Module):
num_channels = 512
def forward(self, x): return dense
module = official_module()
decoder = SimpleNamespace(dense_feature_extraction=DenseMap(), detection=module.HardDetectionModule(),
localization=module.HandcraftedLocalizationModule())
with torch.no_grad(): k,s,d = official_features(decoder, torch.zeros(1,3,480,640))
return {'keypoints': k.numpy(), 'scores': s.numpy(), 'descriptors': d.numpy()}

View File

@ -0,0 +1,12 @@
"""Exact MMDetection dab.detr.r50; see mmdet_detr_common for static deployment geometry."""
from functools import partial
from .mmdet_detr_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, OPSET, recipe,
get_recipe, preprocess_variant, decode_variant, build_variant)
MODEL_ID = "dab.detr.r50"
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
PREPROCESS = recipe(MODEL_ID)
EXPORTER_VERSION = "mmdet-dab.detr.r50-640-raw-v1"
preprocess = partial(preprocess_variant, MODEL_ID)
decode = partial(decode_variant, MODEL_ID)
build = partial(build_variant, MODEL_ID)

View File

@ -0,0 +1,7 @@
"""Official Cityscapes ddrnet.23; exact static benchmark recipe."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("ddrnet.23")
preprocess = partial(preprocess_image, PREPROCESS)
build = partial(build_model, "ddrnet.23")

View File

@ -0,0 +1,7 @@
"""Official Cityscapes ddrnet.23.slim; exact static benchmark recipe."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("ddrnet.23.slim")
preprocess = partial(preprocess_image, PREPROCESS)
build = partial(build_model, "ddrnet.23.slim")

View File

@ -0,0 +1,74 @@
"""Official DeDoDe v2 detector-L + descriptor-B; both learned nets in one graph."""
from . import feature_sources3 as sources
from .feature_common import MAX_KEYPOINTS
INPUT_SPEC='1x3x480x640'
MODEL_SPEC={'input_spec':INPUT_SPEC,'dataset':'hpatches',
'input_override_reason':'Official DeDoDe RGB ImageNet-normalized input; detector-L v2 plus descriptor-B.'}
EXPORTER_VERSION='dedode-v2-l-b-static-1'
OPSET=17
CHECKPOINT={'url':'https://github.com/Parskatt/DeDoDe/releases/download/v2/dedode_detector_L_v2.pth',
'sha256':'4113809dd9e0367af013a45fc2255a6b243ff241cd06520d17a65d9e231bdc17',
'filename':'dedode_detector_L_v2.pth',
'official_relative_path':str(sources.ROOT/'dedode/weights/dedode_detector_L_v2.pth')}
DESCRIPTOR_CHECKPOINT={'url':'https://github.com/Parskatt/DeDoDe/releases/download/dedode_pretrained_models/dedode_descriptor_B.pth',
'sha256':'8eccfc270ec990ced60cd54a434411a47c4c504de13586f596d042e005b3022b',
'path':str(sources.ROOT/'dedode/weights/dedode_descriptor_B.pth')}
PREPROCESS={'input_spec':INPUT_SPEC,'color':'RGB','scale':1/255,'mean':[.485,.456,.406],
'std':[.229,.224,.225],'resize':'opencv_linear_640x480','output_layout':'NCHW',
'protocol':'dedode-v2-official-rgb-l-b-static-v1'}
def preprocess(image,input_spec=INPUT_SPEC):
import cv2
import numpy as np
if input_spec!=INPUT_SPEC:raise ValueError(input_spec)
x=cv2.resize(np.asarray(image.convert('RGB')),(640,480),interpolation=cv2.INTER_LINEAR).astype(np.float64)/255
x=(x-np.asarray(PREPROCESS['mean']))/np.asarray(PREPROCESS['std'])
return np.ascontiguousarray(x.transpose(2,0,1)[None],dtype=np.float32)
def official_model(weights):
import torch
sources.check(weights,CHECKPOINT['sha256'])
sources.check(DESCRIPTOR_CHECKPOINT['path'],DESCRIPTOR_CHECKPOINT['sha256'])
module=sources.package('dedode','DeDoDe')
detector=module.dedode_detector_L(device='cpu',weights=torch.load(weights,map_location='cpu',weights_only=True),remove_borders=False)
descriptor=module.dedode_descriptor_B(device='cpu',weights=torch.load(DESCRIPTOR_CHECKPOINT['path'],map_location='cpu',weights_only=True))
nets=torch.nn.ModuleDict({'detector':detector,'descriptor':descriptor}).eval()
# The benchmark is FP32; CPU upstream autocast would be disabled anyway.
for m in nets.modules():
if hasattr(m,'amp'):m.amp=False
return nets
def official_features(m,x):
import torch.nn.functional as F
module=sources.package('dedode','DeDoDe.utils')
result=m['detector'].detect({'image':x},num_keypoints=MAX_KEYPOINTS)
desc=m['descriptor'].describe_keypoints({'image':x},result['keypoints'])['descriptions']
return module.to_pixel_coords(result['keypoints'],480,640)[0].cpu(),result['confidence'][0].cpu(),F.normalize(desc[0],dim=-1).cpu()
def build(weights,input_spec):
import torch
if input_spec!=INPUT_SPEC:raise ValueError(input_spec)
class Graph(torch.nn.Module):
def __init__(self):super().__init__();self.nets=official_model(weights)
def forward(self,x):
return self.nets['detector']({'image':x})['keypoint_logits'],self.nets['descriptor']({'image':x})['description_grid']
return Graph().eval()
def decode(outputs):
import numpy as np
import torch
import torch.nn.functional as F
logits,descriptors=[torch.as_tensor(np.asarray(o,np.float32)) for o in outputs]
assert logits.shape==(1,1,480,640) and descriptors.shape==(1,256,480,640)
assert torch.isfinite(logits).all() and torch.isfinite(descriptors).all()
module=sources.package('dedode','DeDoDe.utils')
# Exact detector.detect / descriptor.describe_keypoints operations; explicit
# CPU map device allows host decoding even when a server GPU is visible.
probability=logits.reshape(1,-1).softmax(-1).reshape(1,1,480*640).sum(1)
k,s=module.sample_keypoints(probability.reshape(1,480,640),device=logits.device,
use_nms=False,sample_topk=True,num_samples=MAX_KEYPOINTS,return_scoremap=True,
sharpen=False,upsample=False,increase_coverage=True,remove_borders=False)
d=F.grid_sample(descriptors,k[:,None],mode='bilinear',align_corners=False)[:,:,0].mT
return {'keypoints':module.to_pixel_coords(k,480,640)[0].numpy(),
'scores':s[0].numpy(),'descriptors':F.normalize(d[0],dim=-1).numpy()}

View File

@ -0,0 +1,7 @@
"""Exact official ADE20K DeepLabV3 ResNet101; admission requires actual validation."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("deeplabv3.resnet101")
preprocess = partial(preprocess_image, PREPROCESS)
build = partial(build_model, "deeplabv3.resnet101")

View File

@ -0,0 +1,7 @@
"""Exact official ADE20K DeepLabV3 ResNet50; admission requires actual validation."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("deeplabv3.resnet50")
preprocess = partial(preprocess_image, PREPROCESS)
build = partial(build_model, "deeplabv3.resnet50")

View File

@ -0,0 +1,7 @@
"""Exact official ADE20K DeepLabV3+ ResNet101; admission requires actual validation."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("deeplabv3plus.resnet101")
preprocess = partial(preprocess_image, PREPROCESS)
build = partial(build_model, "deeplabv3plus.resnet101")

View File

@ -0,0 +1,7 @@
"""Exact official ADE20K DeepLabV3+ ResNet50; admission requires actual validation."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("deeplabv3plus.resnet50")
preprocess = partial(preprocess_image, PREPROCESS)
build = partial(build_model, "deeplabv3plus.resnet50")

View File

@ -0,0 +1,12 @@
"""Exact MMDetection deformable.detr.r50; see mmdet_detr_common for static deployment geometry."""
from functools import partial
from .mmdet_detr_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, OPSET, recipe,
get_recipe, preprocess_variant, decode_variant, build_variant)
MODEL_ID = "deformable.detr.r50"
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
PREPROCESS = recipe(MODEL_ID)
EXPORTER_VERSION = "mmdet-deformable.detr.r50-640-raw-v1"
preprocess = partial(preprocess_variant, MODEL_ID)
decode = partial(decode_variant, MODEL_ID)
build = partial(build_variant, MODEL_ID)

View File

@ -0,0 +1,10 @@
"""Official Depth Anything V1 base: relative inverse depth."""
from pathlib import Path
from .depth_anything_v1_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET, PREPROCESS,
RESTORE_ALIGN_CORNERS, build_variant, checkpoint_info, preprocess)
CHECKPOINT = checkpoint_info("base")
def build(weights: Path, input_spec: str):
return build_variant("base", weights, input_spec)

View File

@ -0,0 +1,87 @@
"""Official relative Depth Anything V1; pinned source and strict checkpoints."""
from __future__ import annotations
import hashlib
import importlib
import os
from pathlib import Path
import sys
from .depth_anything_v2_common import INPUT_SPEC, preprocess
UPSTREAM_COMMIT = "1d03336771fe09c5398ffdd211441e33941a97dc"
SOURCE_TREE_SHA256 = "0196533fdd23b8031f85b7ea542bd0400a4c362916b16b2e5ea91678144f65b6"
RUNTIME_ROOT = Path.home() / "bench-cache/sources/depth-anything-v1" / UPSTREAM_COMMIT
MODEL_SPEC = {"source": "custom", "dataset": "nyuv2", "input_spec": INPUT_SPEC, "resize": 0}
OPSET = 17
EXPORTER_VERSION = "depth-anything-v1-relative-v1-opset17-static518"
RESTORE_ALIGN_CORNERS = False
PREPROCESS = {
"id": "depth-anything-v1-rgb-static518-v1",
"layout": "NCHW", "dtype": "float32", "color": "RGB",
"resize": "OpenCV INTER_CUBIC to 518x518; keep_aspect_ratio=False for static deployment",
"normalize": "RGB/255; mean=[0.485,0.456,0.406]; std=[0.229,0.224,0.225]",
"output": "N,H,W nonnegative affine-invariant inverse depth; NOT metric depth",
"restore": "bilinear to original H,W, align_corners=False; no min/max image normalization",
"source_commit": UPSTREAM_COMMIT,
}
_MODELS = {
"small": {"encoder": "vits", "features": 64, "out_channels": [48, 96, 192, 384],
"revision": "1bee667edbfec7f9b345abaffec8c9c2bd5293c2",
"sha256": "ba0ab5365610467de4fea1ffe64b4ae3a756561a4df9e11bd7369205241f2082"},
"base": {"encoder": "vitb", "features": 128, "out_channels": [96, 192, 384, 768],
"revision": "a7dff65359777209cb99e1361df8a76144f87e49",
"sha256": "d25d4cf2b50e63198c42e684abb10b9d242714cec0fe3e2eb8f7da41fada25b3"},
"large": {"encoder": "vitl", "features": 256, "out_channels": [256, 512, 1024, 1024],
"revision": "973948530e4e4f4afd6d1913f670d9f96071dcaa",
"sha256": "6fda5199f0043e91e1c6117df00313db57c630d8ed0306070b43da1ecfffbd9d"},
}
def checkpoint_info(variant: str) -> dict:
info = _MODELS[variant]
repo = f"LiheYoung/depth_anything_{info['encoder']}14"
return {"url": f"https://huggingface.co/{repo}/resolve/{info['revision']}/pytorch_model.bin",
"sha256": info["sha256"], "revision": info["revision"], "output_type": "relative_inverse_depth"}
def source_root() -> Path:
root = Path(os.environ.get("BENCH_DEPTH_ANYTHING_V1_SOURCE", str(RUNTIME_ROOT))).resolve()
digest = hashlib.sha256()
files = sorted(list((root / "depth_anything").rglob("*.py")) + list((root / "torchhub").rglob("*.py")))
for path in files:
digest.update(str(path.relative_to(root)).encode() + b"\0" + path.read_bytes())
if digest.hexdigest() != SOURCE_TREE_SHA256:
raise RuntimeError(f"missing or changed official Depth Anything V1 source {UPSTREAM_COMMIT}: {root}")
return root
def build_variant(variant: str, weights: Path, input_spec: str):
import torch
if input_spec != INPUT_SPEC:
raise ValueError(f"expected fixed {INPUT_SPEC}, got {input_spec}")
root = source_root()
weights = Path(weights).resolve()
config = _MODELS[variant]
digest = hashlib.sha256()
with weights.open("rb") as stream:
for block in iter(lambda: stream.read(1 << 20), b""):
digest.update(block)
if digest.hexdigest() != config["sha256"]:
raise ValueError(f"wrong official relative Depth Anything V1 {variant} checkpoint")
if str(root) not in sys.path:
sys.path.insert(0, str(root))
module = importlib.import_module("depth_anything.dpt")
if Path(module.__file__).resolve() != root / "depth_anything/dpt.py":
raise RuntimeError("a different Depth Anything V1 source is already imported")
# Upstream localhub resolves its bundled DINOv2 relative to the checkout.
# The harness builds each model in a separate worker process.
previous = Path.cwd()
try:
os.chdir(root)
model = module.DepthAnything({key: config[key] for key in ("encoder", "features", "out_channels")})
finally:
os.chdir(previous)
model.load_state_dict(torch.load(weights, map_location="cpu", weights_only=True), strict=True)
return model.eval()

View File

@ -0,0 +1,10 @@
"""Official Depth Anything V1 large: relative inverse depth."""
from pathlib import Path
from .depth_anything_v1_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET, PREPROCESS,
RESTORE_ALIGN_CORNERS, build_variant, checkpoint_info, preprocess)
CHECKPOINT = checkpoint_info("large")
def build(weights: Path, input_spec: str):
return build_variant("large", weights, input_spec)

View File

@ -0,0 +1,10 @@
"""Official Depth Anything V1 small: relative inverse depth."""
from pathlib import Path
from .depth_anything_v1_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET, PREPROCESS,
RESTORE_ALIGN_CORNERS, build_variant, checkpoint_info, preprocess)
CHECKPOINT = checkpoint_info("small")
def build(weights: Path, input_spec: str):
return build_variant("small", weights, input_spec)

View File

@ -0,0 +1,10 @@
"""Official Depth Anything V2 base: affine-invariant inverse depth."""
from pathlib import Path
from .depth_anything_v2_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET, PREPROCESS,
build_variant, checkpoint_info, preprocess)
CHECKPOINT = checkpoint_info("base")
def build(weights: Path, input_spec: str):
return build_variant("base", weights, input_spec)

View File

@ -0,0 +1,94 @@
"""Pinned official Depth Anything V2 relative inverse-depth models.
The original source stays in a private checkout. No timm/transformers install,
replacement architecture, metric-depth checkpoint, or synthetic weights.
"""
from __future__ import annotations
import hashlib
import importlib
import os
from pathlib import Path
import sys
UPSTREAM_COMMIT = "a561b849ebae10a6f5ef49e26c83cbbcd36c71bf"
SOURCE_TREE_SHA256 = "92af3e54e49dac0216ac039d7234c81082ec902329691e7aa57d30b2219f612b"
RUNTIME_ROOT = Path.home() / "bench-cache/sources/depth-anything-v2" / UPSTREAM_COMMIT
INPUT_SPEC = "1x3x518x518"
MODEL_SPEC = {"source": "custom", "dataset": "nyuv2", "input_spec": INPUT_SPEC, "resize": 0}
OPSET = 17
EXPORTER_VERSION = "depth-anything-v2-relative-v1-opset17-static518"
PREPROCESS = {
"id": "depth-anything-v2-rgb-static518-v1",
"layout": "NCHW", "dtype": "float32", "color": "RGB",
"resize": "OpenCV INTER_CUBIC to 518x518; keep_aspect_ratio=False for static deployment",
"normalize": "RGB/255; mean=[0.485,0.456,0.406]; std=[0.229,0.224,0.225]",
"output": "N,H,W nonnegative affine-invariant inverse depth; NOT metric depth",
"restore": "bilinear to original H,W, align_corners=True; no min/max image normalization",
"source_commit": UPSTREAM_COMMIT,
}
_MODELS = {
"small": {"encoder": "vits", "features": 64, "out_channels": [48, 96, 192, 384],
"revision": "03876f8651c73a60fe4c2c48294e09fcb6838fcf",
"sha256": "715fade13be8f229f8a70cc02066f656f2423a59effd0579197bbf57860e1378"},
"base": {"encoder": "vitb", "features": 128, "out_channels": [96, 192, 384, 768],
"revision": "a4e71a6c2ce52fe50df0f212066b0d4a87be9b5e",
"sha256": "0d2b7002e62d39d655571c371333340bd88f67ab95050c03591555aa05645328"},
"large": {"encoder": "vitl", "features": 256, "out_channels": [256, 512, 1024, 1024],
"revision": "cbbb86a30ce19b5684b7a05155dc7e6cbc7685b9",
"sha256": "a7ea19fa0ed99244e67b624c72b8580b7e9553043245905be58796a608eb9345"},
}
def checkpoint_info(variant: str) -> dict:
info = _MODELS[variant]
filename = f"depth_anything_v2_{info['encoder']}.pth"
repo = f"depth-anything/Depth-Anything-V2-{variant.capitalize()}"
return {"url": f"https://huggingface.co/{repo}/resolve/{info['revision']}/{filename}",
"sha256": info["sha256"], "revision": info["revision"], "output_type": "relative_inverse_depth"}
def source_root() -> Path:
root = Path(os.environ.get("BENCH_DEPTH_ANYTHING_V2_SOURCE", str(RUNTIME_ROOT))).resolve()
digest = hashlib.sha256()
for path in sorted((root / "depth_anything_v2").rglob("*.py")):
digest.update(str(path.relative_to(root)).encode() + b"\0" + path.read_bytes())
if digest.hexdigest() != SOURCE_TREE_SHA256:
raise RuntimeError(f"missing or changed official Depth Anything V2 source {UPSTREAM_COMMIT}: {root}")
return root
def preprocess(image, input_spec: str = INPUT_SPEC):
import cv2
import numpy as np
if input_spec != INPUT_SPEC:
raise ValueError(f"Depth Anything V2 recipe is fixed at {INPUT_SPEC}, got {input_spec}")
rgb = np.asarray(image.convert("RGB"), dtype=np.float64) / 255.0
rgb = cv2.resize(rgb, (518, 518), interpolation=cv2.INTER_CUBIC)
rgb = (rgb - np.array([.485, .456, .406])) / np.array([.229, .224, .225])
return np.ascontiguousarray(rgb.transpose(2, 0, 1)[None], dtype=np.float32)
def build_variant(variant: str, weights: Path, input_spec: str):
import torch
if input_spec != INPUT_SPEC:
raise ValueError(f"expected fixed {INPUT_SPEC}, got {input_spec}")
root = source_root()
if str(root) not in sys.path:
sys.path.insert(0, str(root))
module = importlib.import_module("depth_anything_v2.dpt")
if Path(module.__file__).resolve() != root / "depth_anything_v2/dpt.py":
raise RuntimeError("a different Depth Anything V2 source is already imported")
digest = hashlib.sha256()
with Path(weights).open("rb") as stream:
for block in iter(lambda: stream.read(1 << 20), b""):
digest.update(block)
config = _MODELS[variant]
if digest.hexdigest() != config["sha256"]:
raise ValueError(f"wrong official relative Depth Anything V2 {variant} checkpoint")
model = module.DepthAnythingV2(**{key: config[key] for key in ("encoder", "features", "out_channels")})
model.load_state_dict(torch.load(weights, map_location="cpu", weights_only=True), strict=True)
return model.eval()

View File

@ -0,0 +1,10 @@
"""Official Depth Anything V2 large: affine-invariant inverse depth."""
from pathlib import Path
from .depth_anything_v2_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET, PREPROCESS,
build_variant, checkpoint_info, preprocess)
CHECKPOINT = checkpoint_info("large")
def build(weights: Path, input_spec: str):
return build_variant("large", weights, input_spec)

View File

@ -0,0 +1,120 @@
"""Pinned official Depth Anything V2 metric-depth models (Hypersim / VKITTI).
Same pinned upstream checkout as the relative V2 adapters; the metric head
lives in metric_depth/ of that tree. Sigmoid head times max_depth gives
nonnegative metric depth in metres. Static 518x518 deployment resize with the
official normalization; restore is the author's bilinear align_corners=True.
"""
from __future__ import annotations
import hashlib
import importlib
import os
from pathlib import Path
import sys
UPSTREAM_COMMIT = "a561b849ebae10a6f5ef49e26c83cbbcd36c71bf"
# sha256 over "<rel>\0<bytes>" of every *.py under metric_depth/ of the pinned V2 checkout.
SOURCE_TREE_SHA256 = "5c2a223435082664012aa38ae11f3810738a4b6cba1efbb3c09a0405b8f6f17d"
RUNTIME_ROOT = Path.home() / "bench-cache/sources/depth-anything-v2" / UPSTREAM_COMMIT
INPUT_SPEC = "1x3x518x518"
MODEL_SPEC = {"source": "custom", "dataset": "nyuv2", "input_spec": INPUT_SPEC, "resize": 0}
OPSET = 17
EXPORTER_VERSION = "depth-anything-v2-metric-v1-opset17-static518"
RESTORE_ALIGN_CORNERS = True
PREPROCESS = {
"id": "depth-anything-v2-metric-rgb-static518-v1",
"layout": "NCHW", "dtype": "float32", "color": "RGB",
"resize": "OpenCV INTER_CUBIC to 518x518; keep_aspect_ratio=False for static deployment",
"normalize": "RGB/255; mean=[0.485,0.456,0.406]; std=[0.229,0.224,0.225]",
"output": "N,H,W nonnegative metric depth in metres (Sigmoid head times max_depth)",
"restore": "bilinear to original H,W, align_corners=True; no min/max image normalization",
"source_commit": UPSTREAM_COMMIT,
}
_ENCODERS = {
"small": {"encoder": "vits", "features": 64, "out_channels": [48, 96, 192, 384]},
"base": {"encoder": "vitb", "features": 128, "out_channels": [96, 192, 384, 768]},
"large": {"encoder": "vitl", "features": 256, "out_channels": [256, 512, 1024, 1024]},
}
_DOMAINS = {
"hypersim": {"max_depth": 20.0, "dataset": "Hypersim (indoor)", "repo_tag": "Hypersim"},
"vkitti": {"max_depth": 80.0, "dataset": "VKITTI (outdoor)", "repo_tag": "VKITTI"},
}
_MODELS = {
("hypersim", "small"): ("3bc65d4e14a6786a61acec16453c50e12bf5f338",
"b782898d8a3e8be1f639de33837ed85e9b4b73e40f8f5e5cd99067588d722545"),
("hypersim", "base"): ("4ebcd7fd0858cbb953451619ca943ea39ec08e00",
"9dc9e274c5eff55c6daf27b660c0ced0eca4e8593a6da90cdcb04d2b4d3f3fa2"),
("hypersim", "large"): ("79720800638389a78b2defc92caa885104f69974",
"6f82ff2bc543ac02ddff4aa31fa363676a8305dd3ccf04e80e2af115a044cb6d"),
("vkitti", "small"): ("c725b8589bdf6ab04072cab74c0467830db80d6d",
"9203e538d35255c90dda4b7fedb47ff33fe725497bcca3b1e53b3a65ee63f0cb"),
("vkitti", "base"): ("e96fd0c8cedf1825a55b9c61309f754982948df0",
"4dad67a7cc10b462bca48e6b8569c762b8eb3c1adada170a3851a6d3ba37bb3e"),
("vkitti", "large"): ("070e97e4b80e317ec1d03c19927304f4091a180b",
"239b1054a369e66da2576e9a118d6d7c12d90dc8ebe609579a9a09cd8e05fe38"),
}
def checkpoint_info(domain: str, variant: str) -> dict:
revision, sha256 = _MODELS[(domain, variant)]
encoder = _ENCODERS[variant]["encoder"]
filename = f"depth_anything_v2_metric_{domain}_{encoder}.pth"
tag = _DOMAINS[domain]["repo_tag"]
repo = f"depth-anything/Depth-Anything-V2-Metric-{tag}-{variant.capitalize()}"
return {"url": f"https://huggingface.co/{repo}/resolve/{revision}/{filename}",
"sha256": sha256, "revision": revision, "max_depth": _DOMAINS[domain]["max_depth"],
"trained_on": _DOMAINS[domain]["dataset"], "output_type": "metric_depth_m"}
def source_root() -> Path:
root = Path(os.environ.get("BENCH_DEPTH_ANYTHING_V2_METRIC_SOURCE", str(RUNTIME_ROOT))).resolve()
digest = hashlib.sha256()
files = sorted(path for path in (root / "metric_depth").rglob("*.py")
if "__pycache__" not in path.parts)
if not files:
raise RuntimeError(f"missing official Depth Anything V2 metric source {UPSTREAM_COMMIT}: {root}")
for path in files:
digest.update(str(path.relative_to(root)).encode() + b"\0" + path.read_bytes())
if digest.hexdigest() != SOURCE_TREE_SHA256:
raise RuntimeError(f"missing or changed official Depth Anything V2 metric source {UPSTREAM_COMMIT}: {root}")
return root
def preprocess(image, input_spec: str = INPUT_SPEC):
import cv2
import numpy as np
if input_spec != INPUT_SPEC:
raise ValueError(f"Depth Anything V2 metric recipe is fixed at {INPUT_SPEC}, got {input_spec}")
rgb = np.asarray(image.convert("RGB"), dtype=np.float64) / 255.0
rgb = cv2.resize(rgb, (518, 518), interpolation=cv2.INTER_CUBIC)
rgb = (rgb - np.array([.485, .456, .406])) / np.array([.229, .224, .225])
return np.ascontiguousarray(rgb.transpose(2, 0, 1)[None], dtype=np.float32)
def build_variant(domain: str, variant: str, weights: Path, input_spec: str):
import torch
if input_spec != INPUT_SPEC:
raise ValueError(f"expected fixed {INPUT_SPEC}, got {input_spec}")
root = source_root()
config = _MODELS[(domain, variant)]
digest = hashlib.sha256()
with Path(weights).open("rb") as stream:
for block in iter(lambda: stream.read(1 << 20), b""):
digest.update(block)
if digest.hexdigest() != config[1]:
raise ValueError(f"wrong official metric Depth Anything V2 {domain}-{variant} checkpoint")
if str(root) not in sys.path:
sys.path.insert(0, str(root))
module = importlib.import_module("metric_depth.depth_anything_v2.dpt")
if Path(module.__file__).resolve() != root / "metric_depth/depth_anything_v2/dpt.py":
raise RuntimeError("a different Depth Anything V2 metric source is already imported")
model = module.DepthAnythingV2(**{key: value for key, value in _ENCODERS[variant].items()
if key in ("encoder", "features", "out_channels")},
max_depth=_DOMAINS[domain]["max_depth"])
model.load_state_dict(torch.load(weights, map_location="cpu", weights_only=True), strict=True)
return model.eval()

View File

@ -0,0 +1,12 @@
"""Official metric Depth Anything V2 Hypersim-base: metric depth, static 518."""
from pathlib import Path
from .depth_anything_v2_metric_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET,
PREPROCESS, RESTORE_ALIGN_CORNERS, build_variant,
checkpoint_info, preprocess)
CHECKPOINT = checkpoint_info("hypersim", "base")
def build(weights: Path, input_spec: str):
return build_variant("hypersim", "base", weights, input_spec)

View File

@ -0,0 +1,12 @@
"""Official metric Depth Anything V2 Hypersim-large: metric depth, static 518."""
from pathlib import Path
from .depth_anything_v2_metric_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET,
PREPROCESS, RESTORE_ALIGN_CORNERS, build_variant,
checkpoint_info, preprocess)
CHECKPOINT = checkpoint_info("hypersim", "large")
def build(weights: Path, input_spec: str):
return build_variant("hypersim", "large", weights, input_spec)

View File

@ -0,0 +1,12 @@
"""Official metric Depth Anything V2 Hypersim-small: metric depth, static 518."""
from pathlib import Path
from .depth_anything_v2_metric_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET,
PREPROCESS, RESTORE_ALIGN_CORNERS, build_variant,
checkpoint_info, preprocess)
CHECKPOINT = checkpoint_info("hypersim", "small")
def build(weights: Path, input_spec: str):
return build_variant("hypersim", "small", weights, input_spec)

View File

@ -0,0 +1,12 @@
"""Official metric Depth Anything V2 VKITTI-base: metric depth, static 518."""
from pathlib import Path
from .depth_anything_v2_metric_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET,
PREPROCESS, RESTORE_ALIGN_CORNERS, build_variant,
checkpoint_info, preprocess)
CHECKPOINT = checkpoint_info("vkitti", "base")
def build(weights: Path, input_spec: str):
return build_variant("vkitti", "base", weights, input_spec)

View File

@ -0,0 +1,12 @@
"""Official metric Depth Anything V2 VKITTI-large: metric depth, static 518."""
from pathlib import Path
from .depth_anything_v2_metric_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET,
PREPROCESS, RESTORE_ALIGN_CORNERS, build_variant,
checkpoint_info, preprocess)
CHECKPOINT = checkpoint_info("vkitti", "large")
def build(weights: Path, input_spec: str):
return build_variant("vkitti", "large", weights, input_spec)

View File

@ -0,0 +1,12 @@
"""Official metric Depth Anything V2 VKITTI-small: metric depth, static 518."""
from pathlib import Path
from .depth_anything_v2_metric_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET,
PREPROCESS, RESTORE_ALIGN_CORNERS, build_variant,
checkpoint_info, preprocess)
CHECKPOINT = checkpoint_info("vkitti", "small")
def build(weights: Path, input_spec: str):
return build_variant("vkitti", "small", weights, input_spec)

View File

@ -0,0 +1,10 @@
"""Official Depth Anything V2 small: affine-invariant inverse depth."""
from pathlib import Path
from .depth_anything_v2_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET, PREPROCESS,
build_variant, checkpoint_info, preprocess)
CHECKPOINT = checkpoint_info("small")
def build(weights: Path, input_spec: str):
return build_variant("small", weights, input_spec)

View File

@ -0,0 +1,12 @@
"""Exact MMDetection detr.r50; see mmdet_detr_common for static deployment geometry."""
from functools import partial
from .mmdet_detr_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, OPSET, recipe,
get_recipe, preprocess_variant, decode_variant, build_variant)
MODEL_ID = "detr.r50"
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
PREPROCESS = recipe(MODEL_ID)
EXPORTER_VERSION = "mmdet-detr.r50-640-raw-v1"
preprocess = partial(preprocess_variant, MODEL_ID)
decode = partial(decode_variant, MODEL_ID)
build = partial(build_variant, MODEL_ID)

12
harness/adapters/disk.py Normal file
View File

@ -0,0 +1,12 @@
"""Original EPFL DISK depth checkpoint/Unet; official host NMS and descriptor lookup."""
from . import feature_common as common
EXPORTER_VERSION = common.EXPORTER_VERSION
INPUT_SPEC = common.input_spec('disk')
MODEL_SPEC = {'input_spec': INPUT_SPEC, 'dataset': 'hpatches', 'input_override_reason': 'Official DISK expects RGB; CSV grayscale is incorrect.'}
CHECKPOINT = common.checkpoint_info('disk')
PREPROCESS = common.recipe('disk')
OPSET = 17
preprocess = common.preprocess
def build(weights, input_spec):
return common.build('disk', weights, input_spec)

View File

@ -0,0 +1,19 @@
"""Official MiDaS v3 DPT-Hybrid: relative inverse depth at static 384x384."""
from pathlib import Path
from .midas_common import (EXPORTER_VERSION, INPUT_SPECS, MODEL_SPECS, OPSET, PREPROCESSES,
RESTORE_ALIGN_CORNERS, RESTORE_INTERPOLATION, build_variant,
checkpoint_info, preprocess_for)
CHECKPOINT = checkpoint_info("hybrid")
INPUT_SPEC = INPUT_SPECS["hybrid"]
MODEL_SPEC = MODEL_SPECS["hybrid"]
PREPROCESS = PREPROCESSES["hybrid"]
def build(weights: Path, input_spec: str):
return build_variant("hybrid", weights, input_spec)
def preprocess(image, input_spec: str = INPUT_SPEC):
return preprocess_for(image, input_spec, "hybrid")

View File

@ -0,0 +1,19 @@
"""Official MiDaS v3 DPT-Large: relative inverse depth at static 384x384."""
from pathlib import Path
from .midas_common import (EXPORTER_VERSION, INPUT_SPECS, MODEL_SPECS, OPSET, PREPROCESSES,
RESTORE_ALIGN_CORNERS, RESTORE_INTERPOLATION, build_variant,
checkpoint_info, preprocess_for)
CHECKPOINT = checkpoint_info("large")
INPUT_SPEC = INPUT_SPECS["large"]
MODEL_SPEC = MODEL_SPECS["large"]
PREPROCESS = PREPROCESSES["large"]
def build(weights: Path, input_spec: str):
return build_variant("large", weights, input_spec)
def preprocess(image, input_spec: str = INPUT_SPEC):
return preprocess_for(image, input_spec, "large")

View File

@ -0,0 +1,10 @@
"""CSV efficientvit.b0: MIT Han Lab ImageNet-1K B0, official r224 checkpoint."""
from pathlib import Path
from .efficientvit_common import PREPROCESS, build_variant, checkpoint_info, preprocess
CHECKPOINT = checkpoint_info("b0")
def build(weights: Path, input_spec: str):
return build_variant("b0", weights, input_spec)

View File

@ -0,0 +1,10 @@
"""CSV efficientvit.b1: MIT Han Lab ImageNet-1K B1, official r224 checkpoint."""
from pathlib import Path
from .efficientvit_common import PREPROCESS, build_variant, checkpoint_info, preprocess
CHECKPOINT = checkpoint_info("b1")
def build(weights: Path, input_spec: str):
return build_variant("b1", weights, input_spec)

View File

@ -0,0 +1,10 @@
"""CSV efficientvit.b2: MIT Han Lab ImageNet-1K B2, official r224 checkpoint."""
from pathlib import Path
from .efficientvit_common import PREPROCESS, build_variant, checkpoint_info, preprocess
CHECKPOINT = checkpoint_info("b2")
def build(weights: Path, input_spec: str):
return build_variant("b2", weights, input_spec)

View File

@ -0,0 +1,10 @@
"""CSV efficientvit.b3: MIT Han Lab ImageNet-1K B3, official r224 checkpoint."""
from pathlib import Path
from .efficientvit_common import PREPROCESS, build_variant, checkpoint_info, preprocess
CHECKPOINT = checkpoint_info("b3")
def build(weights: Path, input_spec: str):
return build_variant("b3", weights, input_spec)

View File

@ -0,0 +1,163 @@
"""MIT Han Lab EfficientViT ImageNet classifiers, using the official factory.
No downloads, package installation, key remapping, or architecture fallback occur
here. The pinned official package and its dependencies live in a private cache;
the shared venv is unchanged. Missing files remain errors, not readiness evidence.
"""
from __future__ import annotations
import hashlib
import importlib
import importlib.util
import sys
from pathlib import Path
UPSTREAM_REPOSITORY = "https://github.com/mit-han-lab/efficientvit"
# The original classification release avoids later SAM/DC_AE eager imports.
# Its unmodified factory strictly accepts all four official r224 checkpoints.
UPSTREAM_COMMIT = "b3ed56d97ac8ff0cc6add7ad2533c42a7488d736"
RUNTIME_ROOT = Path.home() / "bench-cache/sources/efficientvit-cls"
CHECKPOINT_REPOSITORY = "han-cai/efficientvit-cls"
CHECKPOINT_REVISION = "df3d006c2567f9e322b03731f20fe4405a1ab090"
INPUT_SPEC = "1x3x224x224"
# Hashes are the HF LFS object IDs, also verified against downloaded bytes.
_CHECKPOINTS = {
"b0": (13741687, "feebc4d3ebf610673c8ed1d92b21fa2c2e57661868173a37126b734ca5dc3771"),
"b1": (36575137, "bf8798aa03ed2bba21fc8d58f43ca9a92096f3f03bd700e1df1dcde11559f499"),
"b2": (97581671, "6ccfecc65460856cb6415f2356deb8b90e0216960f661f5055aef173f8ca421d"),
"b3": (194992573, "3c5be1a11a8a41b3ac4e30e5bff8bf99065c98b4b2d586e1803fd7571f76fb0d"),
}
# Check the actual architecture, factory and loader, not just a package name.
# This pins core source identity; it is not a full environment attestation.
_SOURCE_SHA256 = {
"__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"cls_model_zoo.py": "c5e21fa447add578968ce6ed179c6e877aa3414b46f1d21af45e947610030599",
"models/efficientvit/__init__.py": "6ec6831645b992d01f479174f55df2c026a05adf099a9eebb45b9690c84b8051",
"models/efficientvit/cls.py": "41534fcd771aa42d9db2cb95d7580dfdf5ac5d54b57b2df8ac2b35dd4be7414e",
"models/efficientvit/backbone.py": "f30d4460b3c2000421fc3421d1cbb3d501233e13fa4f80f9df7bbc7ed5664863",
"models/nn/ops.py": "a5c177a1d3f9d1d8c85647c57d01295506f84ba53b763b0ba77095be34e174e0",
"models/nn/norm.py": "45d34b3b5002c8d76557ed3017c02ccfc31723f2d84efd2f0cd7074706132cf5",
"models/nn/act.py": "7b71a6626dd7ebd7bf0cba283f3f3f5ec3da6ebec0d7c04b2fbafa1969abd7f8",
"models/utils/network.py": "db63b048a48c1edd47e6f401d0cc7d96dfc388b7e37f568acdef5c7a83acc5c4",
}
PREPROCESS = {
"input_spec": INPUT_SPEC,
"color": "RGB",
"crop_ratio": 0.95,
"resize_shorter": 236, # ceil(224 / 0.95)
"interpolation": "bicubic",
"center_crop": 224,
"scale": 1.0 / 255.0,
"mean": (0.485, 0.456, 0.406),
"std": (0.229, 0.224, 0.225),
"output_layout": "NCHW",
}
def checkpoint_info(variant: str) -> dict:
"""Return an immutable-revision HTTPS URI suitable for existing fetch_http."""
if variant not in _CHECKPOINTS:
raise ValueError(f"Unsupported MIT EfficientViT classification variant: {variant!r}")
size, sha256 = _CHECKPOINTS[variant]
filename = f"efficientvit_{variant}_r224.pt"
return {
"model_id": f"efficientvit.{variant}",
"factory_name": f"{variant}-r224",
"filename": filename,
"url": f"https://huggingface.co/{CHECKPOINT_REPOSITORY}/resolve/{CHECKPOINT_REVISION}/{filename}",
"size": size,
"sha256": sha256,
}
def _sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
for chunk in iter(lambda: stream.read(1 << 20), b""):
digest.update(chunk)
return digest.hexdigest()
def verify_checkpoint(variant: str, weights: Path) -> dict:
"""Reject HTML, wrong resolutions/variants, and converted or random weights."""
info = checkpoint_info(variant)
weights = Path(weights)
if not weights.is_file():
raise FileNotFoundError(f"Official checkpoint required: {weights}; source: {info['url']}")
if weights.stat().st_size != info["size"] or _sha256(weights) != info["sha256"]:
raise ValueError(
f"Checkpoint identity mismatch for {info['model_id']}: {weights}; "
f"expected {info['size']} bytes, SHA256 {info['sha256']} from {info['url']}"
)
return info
def _check_input_spec(input_spec: str) -> None:
if input_spec != INPUT_SPEC:
raise ValueError(f"This adapter pins the official r224 recipe: expected {INPUT_SPEC}, got {input_spec!r}")
def _official_factory():
source = RUNTIME_ROOT / f"efficientvit-{UPSTREAM_COMMIT}"
root = source / "efficientvit"
for relative, expected in _SOURCE_SHA256.items():
path = root / relative
if not path.is_file() or _sha256(path) != expected:
raise RuntimeError(
f"Unverified EfficientViT source: {path}; expected core source from "
f"{UPSTREAM_REPOSITORY}/tree/{UPSTREAM_COMMIT} (SHA256 {expected})."
)
# Only this process sees the private dependencies. Restore the import path
# after upstream's imports; never change PYTHONPATH or the shared venv.
original_path = sys.path[:]
try:
sys.path[:0] = [str(source), str(RUNTIME_ROOT / "deps")]
spec = importlib.util.find_spec("efficientvit")
if spec is None or spec.origin is None or Path(spec.origin).resolve() != (root / "__init__.py").resolve():
raise RuntimeError("A different EfficientViT package is already loaded; use a fresh worker.")
module = importlib.import_module("efficientvit.cls_model_zoo")
return module.create_cls_model
except ImportError as exc:
raise RuntimeError(
f"Official EfficientViT import failed ({exc}). Prepare pinned private dependencies "
f"under {RUNTIME_ROOT / 'deps'}. No fallback is used."
) from exc
finally:
sys.path[:] = original_path
def build_variant(variant: str, weights: Path, input_spec: str):
"""Load a hash-verified checkpoint via upstream's strict state_dict loader.
Upstream keeps norm epsilon at 1e-5 for the B series and reads the nested
state_dict with torch.load(map_location='cpu'), then load_state_dict(strict=True).
"""
_check_input_spec(input_spec)
info = verify_checkpoint(variant, weights)
factory = _official_factory()
model = factory(
name=info["factory_name"], pretrained=True, weight_url=str(Path(weights).resolve())
)
return model.eval()
def preprocess(image, input_spec: str = INPUT_SPEC):
"""Official PIL/torchvision evaluation transform, returned as float32 NCHW.
Shared by baseline, calibration, G2 and quantized evaluation through the
harness adapter hook; it is not an in-model normalization wrapper.
"""
_check_input_spec(input_spec)
from torchvision import transforms
from torchvision.transforms.functional import InterpolationMode
transform = transforms.Compose([
transforms.Resize(236, interpolation=InterpolationMode.BICUBIC),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize(mean=PREPROCESS["mean"], std=PREPROCESS["std"]),
])
return transform(image.convert("RGB")).unsqueeze(0).numpy()

View File

@ -0,0 +1,14 @@
"""Official MIT EfficientViT segmentation B0; readiness requires a real probe."""
from pathlib import Path
from .efficientvit_seg_common import (
build_variant, checkpoint_info, model_spec, preprocess, preprocess_recipe,
)
CHECKPOINT = checkpoint_info("b0")
MODEL_SPEC = model_spec("b0")
PREPROCESS = preprocess_recipe("b0")
def build(weights: Path, input_spec: str):
return build_variant("b0", weights, input_spec)

View File

@ -0,0 +1,14 @@
"""Official MIT EfficientViT segmentation B1; readiness requires a real probe."""
from pathlib import Path
from .efficientvit_seg_common import (
build_variant, checkpoint_info, model_spec, preprocess, preprocess_recipe,
)
CHECKPOINT = checkpoint_info("b1")
MODEL_SPEC = model_spec("b1")
PREPROCESS = preprocess_recipe("b1")
def build(weights: Path, input_spec: str):
return build_variant("b1", weights, input_spec)

View File

@ -0,0 +1,14 @@
"""Official MIT EfficientViT segmentation B2; readiness requires a real probe."""
from pathlib import Path
from .efficientvit_seg_common import (
build_variant, checkpoint_info, model_spec, preprocess, preprocess_recipe,
)
CHECKPOINT = checkpoint_info("b2")
MODEL_SPEC = model_spec("b2")
PREPROCESS = preprocess_recipe("b2")
def build(weights: Path, input_spec: str):
return build_variant("b2", weights, input_spec)

View File

@ -0,0 +1,14 @@
"""Official MIT EfficientViT segmentation B3; readiness requires a real probe."""
from pathlib import Path
from .efficientvit_seg_common import (
build_variant, checkpoint_info, model_spec, preprocess, preprocess_recipe,
)
CHECKPOINT = checkpoint_info("b3")
MODEL_SPEC = model_spec("b3")
PREPROCESS = preprocess_recipe("b3")
def build(weights: Path, input_spec: str):
return build_variant("b3", weights, input_spec)

View File

@ -0,0 +1,94 @@
"""Official MIT EfficientViT segmentation heads and fixed board input recipe.
Reuse the unmodified 2023 package/private deps prepared for classification.
Weights are immutable-revision official releases, never classification weights.
The bench keeps its static 512x1024 input for both datasets; this is not the
published full-resolution Cityscapes / variable-aspect ADE20K evaluation.
"""
from __future__ import annotations
import importlib
from pathlib import Path
from .efficientvit_common import RUNTIME_ROOT, UPSTREAM_COMMIT, _official_factory, _sha256
SOURCE_COMMIT = "b3ed56d97ac8ff0cc6add7ad2533c42a7488d736"
CHECKPOINT_REPOSITORY = "han-cai/efficientvit-seg"
CHECKPOINT_REVISION = "cf3ccaf9cbaf670a2cb283612773546824ca0aa1"
INPUT_SPEC = "1x3x512x1024"
_SOURCE_SHA256 = {
"seg_model_zoo.py": "8cd5e99445dbf8c1270c37f93f6f4eca0ac3fefe66f263f325e0e23d50222922",
"models/efficientvit/seg.py": "a279c103c57f467de1eb93c7db3c170872a48faaf618aff58ada1af6bd5fd8bb",
}
_CHECKPOINTS = {
"b0": ("cityscapes", 19, 2924013, "923d6fdd5e93640cc0c2f3f213764f34e80b477cd98a6b294d870ea6df5acc50"),
"b1": ("ade20k", 150, 19339726, "59cca6a7ff4d1a6494b7f12072c6a68fee40ef401e971f26af01a2e0f3c8f5e7"),
"b2": ("ade20k", 150, 61436227, "fabfa7163e35d14888690969cd852069f399137be39697bfe5165f03ea76b510"),
"b3": ("ade20k", 150, 158594800, "fd2706e63ae9d425eb022bce26e10cebb56edd347644003f32f3ac0c9c3855a2"),
}
def checkpoint_info(variant: str) -> dict:
dataset, classes, size, sha256 = _CHECKPOINTS[variant]
filename = f"efficientvit_seg_{variant}_{dataset}.pt"
return {"model_id": f"efficientvit.seg.{variant}", "factory_name": variant,
"dataset": dataset, "classes": classes, "filename": filename,
"url": f"https://huggingface.co/{CHECKPOINT_REPOSITORY}/resolve/{CHECKPOINT_REVISION}/{filename}",
"size": size, "sha256": sha256, "source_commit": SOURCE_COMMIT}
def model_spec(variant: str) -> dict:
info = checkpoint_info(variant)
result = {"dataset": info["dataset"], "input_spec": INPUT_SPEC}
if variant == "b0":
result["dataset_override_reason"] = (
"CSV says ADE20K; official B0 segmentation factory/weights only support Cityscapes (19 classes)."
)
return result
def preprocess_recipe(variant: str) -> dict:
return {"input_spec": INPUT_SPEC, "dataset": checkpoint_info(variant)["dataset"],
"color": "RGB", "resize": "fixed_shape", "interpolation": "opencv_inter_cubic",
"scale": 1.0 / 255.0, "mean": [0.485, 0.456, 0.406], "std": [0.229, 0.224, 0.225],
"output_layout": "NCHW", "logits_resize": "bicubic", "align_corners": False,
"protocol": "bench-static-resize-v1; logits resized to original label size before argmax"}
def input_shape(input_spec: str) -> tuple[int, ...]:
if input_spec != INPUT_SPEC:
raise ValueError(f"EfficientViT segmentation recipe requires {INPUT_SPEC}, got {input_spec!r}")
return tuple(int(n) for n in input_spec.split("x"))
def preprocess(image, input_spec: str = INPUT_SPEC):
"""Official cv2 cubic + float32 ImageNet normalization at the bench shape."""
import cv2
import numpy as np
_, _, h, w = input_shape(input_spec)
rgb = np.asarray(image.convert("RGB"))
if rgb.shape[:2] != (h, w):
rgb = cv2.resize(rgb, (w, h), interpolation=cv2.INTER_CUBIC)
x = rgb.astype(np.float32) / np.float32(255.0)
x = (x - np.asarray((0.485, 0.456, 0.406), np.float32)) / np.asarray((0.229, 0.224, 0.225), np.float32)
return np.ascontiguousarray(x.transpose(2, 0, 1)[None])
def build_variant(variant: str, weights: Path, input_spec: str):
input_shape(input_spec)
info, weights = checkpoint_info(variant), Path(weights)
if not weights.is_file() or weights.stat().st_size != info["size"] or _sha256(weights) != info["sha256"]:
raise ValueError(f"Official segmentation checkpoint identity mismatch: {weights}; expected {info}")
if UPSTREAM_COMMIT != SOURCE_COMMIT:
raise RuntimeError("EfficientViT segmentation requires the pinned 2023 shared source")
root = RUNTIME_ROOT / f"efficientvit-{SOURCE_COMMIT}" / "efficientvit"
# Check the segmentation code before the shared factory's eager imports.
for relative, expected in _SOURCE_SHA256.items():
path = root / relative
if not path.is_file() or _sha256(path) != expected:
raise RuntimeError(f"Unverified official segmentation source: {path}")
_official_factory() # Verify shared core; load private dependencies; restore sys.path.
factory = importlib.import_module("efficientvit.seg_model_zoo").create_seg_model
return factory(name=variant, dataset=info["dataset"], pretrained=True,
weight_url=str(weights.resolve())).eval()

View File

@ -0,0 +1,13 @@
"""Official Cityscapes Fast-SCNN with exact static pooling during export."""
from functools import partial
from .mmseg_common import (metadata, preprocess_image, build_model,
lower_static_pools, RUNTIME_PYTHON, EXPORTER_VERSION)
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("fast.scnn")
preprocess = partial(preprocess_image, PREPROCESS)
def build(weights, input_spec):
network = build_model("fast.scnn", weights, input_spec)
lower_static_pools(network.upstream.backbone)
return network

View File

@ -0,0 +1,7 @@
"""Exact official ADE20K FCN ResNet101; admission requires actual validation."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("fcn.resnet101")
preprocess = partial(preprocess_image, PREPROCESS)
build = partial(build_model, "fcn.resnet101")

View File

@ -0,0 +1,7 @@
"""Exact official ADE20K FCN ResNet50; admission requires actual validation."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("fcn.resnet50")
preprocess = partial(preprocess_image, PREPROCESS)
build = partial(build_model, "fcn.resnet50")

View File

@ -0,0 +1,12 @@
"""Exact MMDetection fcos.r50.fpn; see mmdet_dense_common for static deployment geometry."""
from functools import partial
from .mmdet_dense_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
get_recipe, preprocess_variant, decode_variant, build_variant)
MODEL_ID = "fcos.r50.fpn"
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
PREPROCESS = recipe(MODEL_ID)
EXPORTER_VERSION = "mmdet-fcos.r50.fpn-640-raw-v1"
preprocess = partial(preprocess_variant, MODEL_ID)
decode = partial(decode_variant, MODEL_ID)
build = partial(build_variant, MODEL_ID)

View File

@ -0,0 +1,215 @@
"""Pinned, unmodified feature sources. Static learned graphs; explicit host decoding."""
from __future__ import annotations
import hashlib
import importlib
import importlib.util
import json
import os
from functools import lru_cache
from pathlib import Path
import sys
ROOT = Path(os.environ.get('BENCH_FEATURE_SOURCES', '/home/ming01.wang/bench-cache/sources/feature'))
MAX_KEYPOINTS = 1024
EXPORTER_VERSION = 'feature-static-1'
_CHECKPOINTS = {
'superpoint': ('superglue/models/weights/superpoint_v1.pth', '52b6708629640ca883673b5d5c097c4ddad37d8048b33f09c8ca0d69db12c40e',
'https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/models/weights/superpoint_v1.pth'),
'disk': ('disk/depth-save.pth', '9c2ee4ded238892dfa51569941372601e35e4a74aa6f84ea80053d2ab1c07abe',
'https://raw.githubusercontent.com/cvlab-epfl/disk/8dc6d4d65b47210f629e21ad1fbfb56944b11696/depth-save.pth'),
'aliked': ('aliked/models/aliked-n16.pth', '5be8704840ed662d9d8c561bf7279c222092674e7eb05fd0feab94899e9d82f2',
'https://raw.githubusercontent.com/Shiaoming/ALIKED/683d7c65197395c0b3f01ebe76e1084a27e73a65/models/aliked-n16.pth'),
}
def checkpoint_info(model):
rel, digest, url = _CHECKPOINTS[model]
return {'url': url, 'sha256': digest, 'filename': Path(rel).name, 'official_relative_path': rel}
def input_spec(model):
if model in ('xfeat', 'd2.net', 'r2d2', 'dedode.v2', 'keynetplushardnet'):
from . import get
return get(model).INPUT_SPEC
return '1x1x480x640' if model == 'superpoint' else '1x3x480x640'
def recipe(model):
return {'input_spec': input_spec(model), 'color': 'GRAY' if model == 'superpoint' else 'RGB',
'resize': 'opencv_linear_640x480', 'scale': 1 / 255, 'output_layout': 'NCHW',
'protocol': 'feature-static-640x480-v1', 'gray': 'opencv_RGB2GRAY_uint8'}
def preprocess(image, spec, model=None):
if model in ('xfeat', 'd2.net', 'r2d2', 'dedode.v2', 'keynetplushardnet'):
from . import get
return get(model).preprocess(image, spec)
import cv2
import numpy as np
if spec not in ('1x1x480x640', '1x3x480x640'): raise ValueError(spec)
x = np.asarray(image.convert('RGB'))
if spec.startswith('1x1x'): x = cv2.cvtColor(x, cv2.COLOR_RGB2GRAY)
x = cv2.resize(x, (640, 480), interpolation=cv2.INTER_LINEAR).astype(np.float32) / 255
return np.ascontiguousarray(x[None, None] if x.ndim == 2 else x.transpose(2, 0, 1)[None])
@lru_cache(None)
def official_module(model):
lock = json.loads((ROOT / 'source-lock.json').read_text())
prefix = {'superpoint': 'superglue/', 'disk': 'disk/', 'aliked': 'aliked/'}[model]
for rel, expected in lock['files'].items():
if rel.startswith(prefix) and hashlib.sha256((ROOT / rel).read_bytes()).hexdigest() != expected:
raise ValueError(f'upstream source modified: {rel}')
if model == 'superpoint':
spec = importlib.util.spec_from_file_location('_bench_official_superpoint', ROOT / 'superglue/models/superpoint.py')
module = importlib.util.module_from_spec(spec); spec.loader.exec_module(module)
return module
roots = [ROOT / model]
if model == 'disk': roots += [ROOT / 'disk/submodules' / n for n in ('unets', 'torch-dimcheck', 'torch-localize')]
# Optional pure-Python dependencies live here, never in the shared venv.
roots.append(ROOT / 'deps')
old = sys.path[:]
try:
sys.path[:0] = [str(p) for p in roots]
name = 'disk' if model == 'disk' else 'nets.aliked'
mod = importlib.import_module(name)
if not Path(mod.__file__).resolve().is_relative_to((ROOT / model).resolve()):
raise RuntimeError(f'upstream module namespace collision: {mod.__file__}')
return mod
finally:
sys.path[:] = old
def official_model(model, weights):
import torch
weights = Path(weights)
if hashlib.sha256(weights.read_bytes()).hexdigest() != checkpoint_info(model)['sha256']:
raise ValueError('official checkpoint SHA mismatch')
module = official_module(model)
state = torch.load(weights, map_location='cpu', weights_only=False)
if model == 'superpoint':
m = module.SuperPoint({'max_keypoints': MAX_KEYPOINTS})
elif model == 'disk':
m = module.DISK(); state = state['extractor']
else:
m = module.ALIKED(model_name='aliked-n16', device='cpu', top_k=MAX_KEYPOINTS, load_pretrained=False)
m.load_state_dict(state, strict=True)
return m.eval()
def official_features(model, m, x):
"""Reference forward without ALIKED's CUDA timing-only calls on CPU."""
if model == 'superpoint':
d = m({'image': x})
return d['keypoints'][0], d['scores'][0], d['descriptors'][0].T
if model == 'disk':
d = m.features(x, kind='nms', n=MAX_KEYPOINTS, window_size=5, cutoff=0.)[0]
return d.kp, d.kp_logp, d.desc
feature, score = m.extract_dense_map(x)
keypoints, scores, _ = m.dkd(score)
descriptors, _ = m.desc_head(feature, keypoints)
wh = x.new_tensor([x.shape[-1] - 1, x.shape[-2] - 1])
return (keypoints[0] + 1) * wh / 2, scores[0], descriptors[0]
def replace_deformable(m):
"""Exact torchvision DCNv1 sampling expressed as standard ONNX GridSample+Conv.
Only ALIKED's ungrouped, unmodulated 3x3 stride-1 operator is supported.
This changes the wrapper instance, never the downloaded source or reference.
"""
import torch
from torch import nn
from torch.nn import functional as F
class StaticDeform(nn.Module):
def __init__(self, source):
super().__init__()
assert not source.mask and source.regular_conv.kernel_size == (3, 3)
assert source.regular_conv.stride == (1, 1) and source.regular_conv.groups == 1
self.offset_conv = source.offset_conv; self.regular_conv = source.regular_conv
def forward(self, x):
# The adapter contract is static NCHW; freeze these dimensions so
# ONNX knows the 1x1 convolution kernel shape after sample stacking.
b, c, h, w = (int(v) for v in x.shape)
offset = self.offset_conv(x).clamp(-max(h, w) / 4., max(h, w) / 4.)
yy, xx = torch.meshgrid(torch.arange(h, device=x.device, dtype=x.dtype),
torch.arange(w, device=x.device, dtype=x.dtype), indexing='ij')
samples = []
for k in range(9):
y = yy + (k // 3 - 1) + offset[:, 2 * k]
z = xx + (k % 3 - 1) + offset[:, 2 * k + 1]
grid = torch.stack((2 * z / (w - 1) - 1, 2 * y / (h - 1) - 1), -1)
samples.append(F.grid_sample(x, grid, align_corners=True, padding_mode='zeros'))
sampled = torch.stack(samples, 2).reshape(b, c * 9, h, w)
weights = self.regular_conv.weight.flatten(1).unsqueeze(-1).unsqueeze(-1)
return F.conv2d(sampled, weights, self.regular_conv.bias)
for name, child in list(m.named_children()):
if child.__class__.__name__ == 'DeformableConv2d': setattr(m, name, StaticDeform(child))
else: replace_deformable(child)
def build(model, weights, spec):
import torch
from torch import nn
from torch.nn import functional as F
if spec != input_spec(model): raise ValueError(f'{model} requires {input_spec(model)}, got {spec}')
original = official_model(model, weights)
class Graph(nn.Module):
def __init__(self, m):
super().__init__(); self.m = m
def forward(self, x):
m = self.m
if model == 'disk': return m.unet(x)
if model == 'aliked':
keypoints, scores, descriptors = official_features(model, m, x)
return (keypoints.reshape(1, MAX_KEYPOINTS, 2),
scores.reshape(1, MAX_KEYPOINTS), descriptors.reshape(1, MAX_KEYPOINTS, 128))
z = m.relu(m.conv1a(x)); z = m.relu(m.conv1b(z)); z = m.pool(z)
z = m.relu(m.conv2a(z)); z = m.relu(m.conv2b(z)); z = m.pool(z)
z = m.relu(m.conv3a(z)); z = m.relu(m.conv3b(z)); z = m.pool(z)
z = m.relu(m.conv4a(z)); z = m.relu(m.conv4b(z))
scores = F.softmax(m.convPb(m.relu(m.convPa(z))), 1)[:, :-1]
b, _, h, w = scores.shape
scores = scores.permute(0, 2, 3, 1).reshape(b, h, w, 8, 8)
scores = scores.permute(0, 1, 3, 2, 4).reshape(b, h * 8, w * 8)
desc = F.normalize(m.convDb(m.relu(m.convDa(z))), p=2, dim=1)
return scores, desc
if model == 'aliked': replace_deformable(original)
return Graph(original).eval()
def decode(model, outputs):
"""The same pinned official postprocessing for eager, ORT and board outputs."""
if model in ('xfeat', 'd2.net', 'r2d2', 'dedode.v2', 'keynetplushardnet'):
from . import get
return get(model).decode(outputs)
import numpy as np
import torch
module = official_module(model)
tensors = [torch.as_tensor(np.asarray(o, dtype=np.float32)) for o in outputs]
if any(not torch.isfinite(t).all() for t in tensors): raise ValueError('nonfinite feature outputs')
if model == 'superpoint':
scores, descriptors = tensors
assert scores.shape == (1, 480, 640) and descriptors.shape == (1, 256, 60, 80)
scores = module.simple_nms(scores, 4)[0]
k = torch.nonzero(scores > .005); s = scores[tuple(k.T)]
k, s = module.remove_borders(k, s, 4, 480, 640)
k, s = module.top_k_keypoints(k, s, MAX_KEYPOINTS)
k = torch.flip(k, [1]).float()
d = module.sample_descriptors(k[None], descriptors, 8)[0].T
elif model == 'disk':
assert tensors[0].shape == (1, 129, 480, 640)
from disk.model.detector import Detector
kp = Detector().nms(tensors[0][:, 128:], n=MAX_KEYPOINTS, window_size=5, cutoff=0.)[0]
f = kp.merge_with_descriptors(tensors[0][0, :128]); k, s, d = f.kp, f.kp_logp, f.desc
else:
k, s, d = [t[0] for t in tensors]
assert k.shape == (MAX_KEYPOINTS, 2) and d.shape == (MAX_KEYPOINTS, 128)
return {'keypoints': k.numpy(), 'scores': s.numpy(), 'descriptors': d.numpy()}

View File

@ -0,0 +1,79 @@
"""Verified official sources for the second feature batch; no shared recipe changes."""
import ast
import hashlib
import importlib.util
import json
import os
from functools import lru_cache
from pathlib import Path
import sys
ROOT = Path(os.environ.get('BENCH_FEATURE_BATCH2_SOURCES', '/home/ming01.wang/bench-cache/sources/feature-batch2'))
@lru_cache(None)
def verify_sources(family):
manifest = json.loads((ROOT/'source-lock.json').read_text())
for rel, digest in manifest['files'].items():
if rel.startswith(family+'/') and hashlib.sha256((ROOT/rel).read_bytes()).hexdigest() != digest:
raise ValueError(f'official source mismatch: {rel}')
def verify_weights(path, expected):
if hashlib.sha256(Path(path).read_bytes()).hexdigest() != expected:
raise ValueError(f'checkpoint identity mismatch: {path}')
@lru_cache(None)
def module(relative):
verify_sources(relative.split('/')[0])
name = '_bench_feature2_' + relative.replace('/', '_').replace('.', '_')
spec = importlib.util.spec_from_file_location(name, ROOT/relative)
value = importlib.util.module_from_spec(spec)
before = sys.path[:]
try:
sys.path.insert(0, str(ROOT/relative.split('/')[0]))
spec.loader.exec_module(value)
finally:
sys.path[:] = before
return value
def definitions(relative, names, namespace=None):
"""Import exact upstream definitions without unrelated training/UI imports.
Original function/class/assignment AST nodes are compiled unchanged. This
does not reimplement their numerical logic or edit the upstream files.
"""
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
verify_sources(relative.split('/')[0])
body = []
for node in ast.parse((ROOT/relative).read_text()).body:
if isinstance(node, (ast.FunctionDef, ast.ClassDef)) and node.name in names:
body.append(node)
elif isinstance(node, ast.Assign) and any(isinstance(t, ast.Name) and t.id in names for t in node.targets):
body.append(node)
assert len(body) == len(names), (relative, names)
scope = {'np': np, 'torch': torch, 'nn': nn, 'F': F, **(namespace or {})}
exec(compile(ast.Module(body=body, type_ignores=[]), str(ROOT/relative), 'exec'), scope)
return scope
def tensors(outputs):
import numpy as np
import torch
result = [torch.as_tensor(np.asarray(v, np.float32)) for v in outputs]
if any(not torch.isfinite(v).all() for v in result): raise ValueError('nonfinite feature maps')
return result
def top_features(keypoints, scores, descriptors):
import numpy as np
from .feature_common import MAX_KEYPOINTS
indices = np.asarray(scores).argsort()[-MAX_KEYPOINTS:]
return {'keypoints': np.asarray(keypoints, np.float32)[indices, :2],
'scores': np.asarray(scores, np.float32)[indices],
'descriptors': np.asarray(descriptors, np.float32)[indices]}

View File

@ -0,0 +1,39 @@
"""Private batch3 official source loader; never used from production during probes."""
import hashlib
import importlib
import importlib.util
import json
from functools import lru_cache
from pathlib import Path
import sys
ROOT=Path('/home/ming01.wang/bench-cache/sources/feature-batch3')
@lru_cache(None)
def verify(family):
record=next(r for r in json.loads((ROOT/'source-lock.json').read_text()) if r['name']==family)
for relative,digest in record['files'].items():
if hashlib.sha256((ROOT/family/relative).read_bytes()).hexdigest()!=digest:
raise ValueError(f'official source changed: {family}/{relative}')
def check(path,digest):
if hashlib.sha256(Path(path).read_bytes()).hexdigest()!=digest:raise ValueError(f'weight mismatch: {path}')
@lru_cache(None)
def package(family,name):
verify(family)
before=sys.path[:]
try:
sys.path[:0]=[str(ROOT/family),str(ROOT/'deps')]
module=importlib.import_module(name)
if not Path(module.__file__).resolve().is_relative_to((ROOT/family).resolve()):
raise RuntimeError(f'official namespace collision: {module.__file__}')
return module
finally:sys.path[:]=before
@lru_cache(None)
def module(family,relative):
verify(family)
spec=importlib.util.spec_from_file_location('_official3_'+family+'_'+relative.replace('/','_'),ROOT/family/relative)
result=importlib.util.module_from_spec(spec);spec.loader.exec_module(result)
return result

View File

@ -0,0 +1,12 @@
"""Exact MMDetection foveabox.r50.fpn; see mmdet_dense_common for static deployment geometry."""
from functools import partial
from .mmdet_dense_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
get_recipe, preprocess_variant, decode_variant, build_variant)
MODEL_ID = "foveabox.r50.fpn"
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
PREPROCESS = recipe(MODEL_ID)
EXPORTER_VERSION = "mmdet-foveabox.r50.fpn-640-raw-v1"
preprocess = partial(preprocess_variant, MODEL_ID)
decode = partial(decode_variant, MODEL_ID)
build = partial(build_variant, MODEL_ID)

View File

@ -0,0 +1,12 @@
"""Exact MMDetection fsaf.r50.fpn; see mmdet_dense_common for static deployment geometry."""
from functools import partial
from .mmdet_dense_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
get_recipe, preprocess_variant, decode_variant, build_variant)
MODEL_ID = "fsaf.r50.fpn"
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
PREPROCESS = recipe(MODEL_ID)
EXPORTER_VERSION = "mmdet-fsaf.r50.fpn-640-raw-v1"
preprocess = partial(preprocess_variant, MODEL_ID)
decode = partial(decode_variant, MODEL_ID)
build = partial(build_variant, MODEL_ID)

View File

@ -0,0 +1,12 @@
"""Exact MMDetection gfl.r50.fpn; see mmdet_dense_common for static deployment geometry."""
from functools import partial
from .mmdet_dense_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
get_recipe, preprocess_variant, decode_variant, build_variant)
MODEL_ID = "gfl.r50.fpn"
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
PREPROCESS = recipe(MODEL_ID)
EXPORTER_VERSION = "mmdet-gfl.r50.fpn-640-raw-v1"
preprocess = partial(preprocess_variant, MODEL_ID)
decode = partial(decode_variant, MODEL_ID)
build = partial(build_variant, MODEL_ID)

View File

@ -0,0 +1,7 @@
"""Official Cityscapes hrnet.w18; exact static benchmark recipe."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("hrnet.w18")
preprocess = partial(preprocess_image, PREPROCESS)
build = partial(build_model, "hrnet.w18")

View File

@ -0,0 +1,9 @@
"""Official hrnet.w32 COCO17 at 256x192."""
from functools import partial
from .topdown_pose_common import (metadata, build_model, predict as _predict, decode as _decode,
preprocess, g2_input, PREPROCESS, RUNTIME_PYTHON, EXPORTER_VERSION)
CHECKPOINT, MODEL_SPEC = metadata('hrnet.w32')
build = partial(build_model, 'hrnet.w32')
predict = partial(_predict, 'hrnet.w32')
decode = partial(_decode, 'hrnet.w32')

View File

@ -0,0 +1,7 @@
"""Official Cityscapes hrnet.w48; exact static benchmark recipe."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("hrnet.w48")
preprocess = partial(preprocess_image, PREPROCESS)
build = partial(build_model, "hrnet.w48")

View File

@ -0,0 +1,9 @@
"""Official hrnet.w48.2 COCO17 at 256x192."""
from functools import partial
from .topdown_pose_common import (metadata, build_model, predict as _predict, decode as _decode,
preprocess, g2_input, PREPROCESS, RUNTIME_PYTHON, EXPORTER_VERSION)
CHECKPOINT, MODEL_SPEC = metadata('hrnet.w48.2')
build = partial(build_model, 'hrnet.w48.2')
predict = partial(_predict, 'hrnet.w48.2')
decode = partial(_decode, 'hrnet.w48.2')

View File

@ -0,0 +1,129 @@
"""Author PyTorch Key.Net plus original HardNet++; no HyNet substitution."""
from . import feature_sources3 as sources
from .feature_common import MAX_KEYPOINTS, preprocess as gray_preprocess
INPUT_SPEC='1x1x480x640'
MODEL_SPEC={'input_spec':INPUT_SPEC,'dataset':'hpatches'}
EXPORTER_VERSION='keynet-hardnetpp-static-1'
OPSET=17
CHECKPOINT={'url':'https://raw.githubusercontent.com/axelBarroso/Key.Net-Pytorch/74fe0a75998445bb47b74b642b764543b25dc90a/model/weights/keynet_pytorch.pth',
'sha256':'59d3a46d57b439513acf531877fdfaa95eb550ccd10ec462e23d7c1f7a86d52a',
'filename':'keynet_pytorch.pth','official_relative_path':str(sources.ROOT/'keynet-pytorch/model/weights/keynet_pytorch.pth')}
DESCRIPTOR_CHECKPOINT={'url':'https://raw.githubusercontent.com/axelBarroso/Key.Net/24808fb5bd7376e33f43e9f8f1029f4ef1a7ce3c/keyNet/pretrained_nets/HardNet%2B%2B.pth',
'path':str(sources.ROOT/'keynet/keyNet/pretrained_nets/HardNet++.pth'),
'sha256':'9fe80a335fe72d1225fdd1ae0943418b12489f95014bd51dddc9819d6ffa33c7'}
PREPROCESS={'input_spec':INPUT_SPEC,'color':'GRAY','gray':'opencv_RGB2GRAY_uint8','scale':1/255,
'resize':'opencv_linear_640x480','output_layout':'NCHW',
'protocol':'keynet-author-pytorch-hardnetpp-single-scale-s22-v1'}
OFFICIAL_REFERENCE_POLICY = {
'selection': 'official positive NMS15, top1024; equal scores use ascending raster index',
'tie_rule': 'ONNX TopK standard; official torch.sort is unstable for equal scores',
'patches': 'official extract_patch, s_mult22, PS32, grayscale255, align_cornersFalse',
'descriptor': 'official HardNet++ class and weights, no HyNet substitution',
}
def preprocess(image,input_spec=INPUT_SPEC):
if input_spec!=INPUT_SPEC:raise ValueError(input_spec)
return gray_preprocess(image,input_spec)
def official_model(weights):
import torch
sources.check(weights,CHECKPOINT['sha256'])
sources.check(DESCRIPTOR_CHECKPOINT['path'],DESCRIPTOR_CHECKPOINT['sha256'])
network=sources.package('keynet-pytorch','model.network')
detector=network.KeyNet({'num_filters':8,'num_levels':3,'kernel_size':5})
detector.load_state_dict(torch.load(weights,map_location='cpu',weights_only=False)['state_dict'],strict=True)
descriptor=sources.module('keynet','keyNet/model/hardnet_pytorch.py').HardNet()
descriptor.load_state_dict(torch.load(DESCRIPTOR_CHECKPOINT['path'],map_location='cpu',weights_only=False)['state_dict'],strict=True)
return torch.nn.ModuleDict({'detector':detector,'descriptor':descriptor}).eval()
def official_features(m,x):
import torch
module=sources.package('keynet-pytorch','model.extraction_tools')
score=module.remove_borders(m['detector'](x),15)
yx=module.NonMaxSuppression(nms_size=15)(score)
values=score[0,0,yx[0],yx[1]]
# Preserve official NMS/candidate ordering; specify only its undefined tie order.
order=torch.argsort(values,descending=True,stable=True)[:MAX_KEYPOINTS]
k=torch.stack([yx[1,order],yx[0,order]],-1).float();s=values[order]
n=len(k)
laf=module.to_laf(k[None],22*torch.ones(1,n,1,1),torch.zeros(1,n,1))
patches=module.extract_patch(255*x,laf,PS=32,normalize_lafs_before_extraction=True)[0]
d=torch.cat([m['descriptor'](p) for p in patches.split(1000)],0)
return k,s,d
def build(weights,input_spec):
import torch
import torch.nn.functional as F
if input_spec!=INPUT_SPEC:raise ValueError(input_spec)
utils=sources.package('keynet-pytorch','model.kornia_tools.utils')
class StaticHandcrafted(torch.nn.Module):
def __init__(self):
super().__init__()
# Original Kornia 0.1.4 Sobel kernel, fixed C=1 eliminates dynamic repeat.
self.register_buffer('kernel',utils.kornia.filters.SpatialGradient().kernel[None,None].flip(-3))
def gradient(self,x):
x=F.pad(x,[1,1,1,1],'replicate')[:,:,None]
return F.conv3d(F.pad(x,[0,0,0,0,1,1]),self.kernel)
def forward(self,x):
sobel=self.gradient(x);dx,dy=sobel[:,:,0],sobel[:,:,1]
sobel_dx=self.gradient(dx);dxx,dxy=sobel_dx[:,:,0],sobel_dx[:,:,1]
dyy=self.gradient(dy)[:,:,1]
return torch.cat([dx,dy,dx**2.,dy**2.,dx*dy,dxy,dxy**2.,dxx,dyy,dxx*dyy],1)
class StableTopK(torch.autograd.Function):
@staticmethod
def forward(ctx,values):
indices=torch.argsort(values,dim=1,descending=True,stable=True)[:,:MAX_KEYPOINTS]
return values.gather(1,indices),indices
@staticmethod
def symbolic(g,values):
# ONNX TopK explicitly breaks ties by the lower original index.
k=g.op('Constant',value_t=torch.tensor([MAX_KEYPOINTS],dtype=torch.long))
return g.op('TopK',values,k,axis_i=1,largest_i=1,sorted_i=1,outputs=2)
class Graph(torch.nn.Module):
def __init__(self):
super().__init__();self.nets=official_model(weights)
self.nets['detector'].feature_extractor.hc_block=StaticHandcrafted()
self.register_buffer('pyramid_kernel',utils._get_pyramid_gaussian_kernel()[None])
border=torch.zeros(1,1,480,640);border[:,:,15:-15,15:-15]=1
self.register_buffer('border',border)
unit=F.affine_grid(torch.tensor([[[1.,0.,0.],[0.,1.,0.]]]),[1,1,32,32],align_corners=False)
self.register_buffer('patch_grid',torch.cat([unit,torch.ones(1,32,32,1)],-1).reshape(1,1024,3))
def forward(self,x):
# Same three internal pyramid levels as official KeyNet; fixed grayscale shapes.
detector=self.nets['detector'];level=x;features=[detector.feature_extractor(level)]
for h,w in ((400,533),(333,444)):
level=F.interpolate(F.conv2d(F.pad(level,[2,2,2,2],'reflect'),self.pyramid_kernel),
size=(h,w),mode='bilinear',align_corners=False)
features.append(F.interpolate(detector.feature_extractor(level),size=(480,640),mode='bilinear',align_corners=False))
scores=detector.last_conv(torch.cat(features,1))*self.border
scores=scores*(scores==F.max_pool2d(scores,15,1,7)).to(scores.dtype)
s,ids=StableTopK.apply(scores.flatten(1))
k=torch.stack((ids%640,torch.div(ids,640,rounding_mode='floor')),-1).float()
laf=utils.laf_from_center_scale_ori(k,22*torch.ones(1,MAX_KEYPOINTS,1,1,device=x.device),torch.zeros(1,MAX_KEYPOINTS,1,device=x.device))
# At s=22, PS=32, every patch has official pyramid index 0.
# Precompute only the constant affine base grid; keep official LAF
# normalization and its floating-point operation order.
theta=utils.denormalize_laf(utils.normalize_laf(laf,x),x).reshape(MAX_KEYPOINTS,2,3)
grid=torch.bmm(self.patch_grid.expand(MAX_KEYPOINTS,-1,-1),theta.transpose(1,2))
grid=torch.stack((2*grid[:,:,0]/640-1,2*grid[:,:,1]/480-1),-1)
# One tiled grid avoids materializing 1024 copies of the input.
patches=F.grid_sample(255*x,grid.reshape(1,MAX_KEYPOINTS*32,32,2),
padding_mode='border',align_corners=False).reshape(MAX_KEYPOINTS,1,32,32)
d=self.nets['descriptor'](patches)
return k.reshape(1,MAX_KEYPOINTS,2),s.reshape(1,MAX_KEYPOINTS),d[None]
return Graph().eval()
def decode(outputs):
import numpy as np
k,s,d=[np.asarray(o,np.float32)[0] for o in outputs]
assert k.shape==(MAX_KEYPOINTS,2) and s.shape==(MAX_KEYPOINTS,) and d.shape==(MAX_KEYPOINTS,128)
assert all(np.isfinite(v).all() for v in (k,s,d))
valid=s>0
return {'keypoints':k[valid],'scores':s[valid],'descriptors':d[valid]}

View File

@ -0,0 +1,22 @@
"""Exact MMDetection mask2former.swin.b.instance; see mmdet_iseg_common.
Backed by the official swin-b COCO-panoptic config and checkpoint (the model
zoo publishes no instance-only swin-b variant); instance outputs are the 80
thing classes, which mmdet orders exactly like CocoDataset.
"""
from functools import partial
from .mmdet_iseg_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
get_recipe, preprocess_variant, decode_variant,
build_variant, predict_variant, g2_input_variant)
MODEL_ID = "mask2former.swin.b.instance"
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
PREPROCESS = recipe(MODEL_ID)
EXPORTER_VERSION = "mmdet-mask2former-640-raw-v1"
OPSET = 16 # ONNX GridSample for the multi-scale deformable attention layers.
preprocess = partial(preprocess_variant, MODEL_ID)
decode = partial(decode_variant, MODEL_ID)
build = partial(build_variant, MODEL_ID)
predict = partial(predict_variant, MODEL_ID)
g2_input = partial(g2_input_variant, MODEL_ID)

View File

@ -0,0 +1,17 @@
"""Exact MMDetection mask2former.swin.s.instance; see mmdet_iseg_common."""
from functools import partial
from .mmdet_iseg_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
get_recipe, preprocess_variant, decode_variant,
build_variant, predict_variant, g2_input_variant)
MODEL_ID = "mask2former.swin.s.instance"
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
PREPROCESS = recipe(MODEL_ID)
EXPORTER_VERSION = "mmdet-mask2former-640-raw-v1"
OPSET = 16 # ONNX GridSample for the multi-scale deformable attention layers.
preprocess = partial(preprocess_variant, MODEL_ID)
decode = partial(decode_variant, MODEL_ID)
build = partial(build_variant, MODEL_ID)
predict = partial(predict_variant, MODEL_ID)
g2_input = partial(g2_input_variant, MODEL_ID)

View File

@ -0,0 +1,17 @@
"""Exact MMDetection mask2former.swin.t.instance; see mmdet_iseg_common."""
from functools import partial
from .mmdet_iseg_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
get_recipe, preprocess_variant, decode_variant,
build_variant, predict_variant, g2_input_variant)
MODEL_ID = "mask2former.swin.t.instance"
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
PREPROCESS = recipe(MODEL_ID)
EXPORTER_VERSION = "mmdet-mask2former-640-raw-v1"
OPSET = 16 # ONNX GridSample for the multi-scale deformable attention layers.
preprocess = partial(preprocess_variant, MODEL_ID)
decode = partial(decode_variant, MODEL_ID)
build = partial(build_variant, MODEL_ID)
predict = partial(predict_variant, MODEL_ID)
g2_input = partial(g2_input_variant, MODEL_ID)

View File

@ -0,0 +1,242 @@
"""Pinned official MiDaS relative-depth models (v2.1 small, v3 DPT hybrid/large).
The pinned private runtime tree keeps the official MiDaS source, the timm
0.6.12 it requires and the gen-efficientnet backbone code, with only the
local-hub patch from the recorded receipt. No package install, replacement
architecture or synthetic weights. Outputs are nonnegative relative inverse
depth (disparity); restore follows the author's bicubic resize.
"""
from __future__ import annotations
import hashlib
import os
from pathlib import Path
import sys
UPSTREAM_COMMIT = "454597711a62eabcbf7d1e89f3fb9f569051ac9b"
# sha256 over "<rel>\0<bytes>" of every *.py in the pinned runtime tree
# (midas + gen-efficientnet + timm), matching source_receipt.json.
SOURCE_TREE_SHA256 = "b27d6846712df86b72264c1df091d0693b4834155cfcc1d30777a359904d8b82"
RUNTIME_ROOT = Path.home() / "bench-cache/sources/midas" / UPSTREAM_COMMIT
OPSET = 17
EXPORTER_VERSION = "midas-relative-v1-opset17-static"
RESTORE_INTERPOLATION = "bicubic"
RESTORE_ALIGN_CORNERS = False
INPUT_SPECS = {
"small": "1x3x256x256",
"hybrid": "1x3x384x384",
"large": "1x3x384x384",
}
MODEL_SPECS = {
"small": {"source": "custom", "dataset": "nyuv2", "input_spec": INPUT_SPECS["small"], "resize": 0},
"hybrid": {"source": "custom", "dataset": "nyuv2", "input_spec": INPUT_SPECS["hybrid"], "resize": 0},
"large": {"source": "custom", "dataset": "nyuv2", "input_spec": INPUT_SPECS["large"], "resize": 0},
}
PREPROCESSES = {
"small": {
"id": "midas-v21-small-rgb-static256-v1",
"layout": "NCHW", "dtype": "float32", "color": "RGB",
"resize": "OpenCV INTER_CUBIC to 256x256; keep_aspect_ratio=False for static deployment",
"normalize": "RGB/255; mean=[0.485,0.456,0.406]; std=[0.229,0.224,0.225]",
"output": "N,H,W nonnegative relative inverse depth (disparity); NOT metric depth",
"restore": "bicubic to original H,W, align_corners=False; no min/max image normalization",
"source_commit": UPSTREAM_COMMIT,
},
"hybrid": {
"id": "midas-v3-hybrid-rgb-static384-v1",
"layout": "NCHW", "dtype": "float32", "color": "RGB",
"resize": "OpenCV INTER_CUBIC to 384x384; keep_aspect_ratio=False for static deployment",
"normalize": "RGB/255; mean=[0.5,0.5,0.5]; std=[0.5,0.5,0.5]",
"output": "N,H,W nonnegative relative inverse depth (disparity); NOT metric depth",
"restore": "bicubic to original H,W, align_corners=False; no min/max image normalization",
"source_commit": UPSTREAM_COMMIT,
},
"large": {
"id": "midas-v3-large-rgb-static384-v1",
"layout": "NCHW", "dtype": "float32", "color": "RGB",
"resize": "OpenCV INTER_CUBIC to 384x384; keep_aspect_ratio=False for static deployment",
"normalize": "RGB/255; mean=[0.5,0.5,0.5]; std=[0.5,0.5,0.5]",
"output": "N,H,W nonnegative relative inverse depth (disparity); NOT metric depth",
"restore": "bicubic to original H,W, align_corners=False; no min/max image normalization",
"source_commit": UPSTREAM_COMMIT,
},
}
_CHECKPOINTS = {
"small": {"url": "https://github.com/isl-org/MiDaS/releases/download/v2_1/midas_v21_small_256.pt",
"sha256": "70d6b9c891758c67f974a6097fb0c608c7ee67fb81ac3e5588847d5596d56fca",
"release": "v2_1", "output_type": "relative_inverse_depth"},
"hybrid": {"url": "https://github.com/isl-org/MiDaS/releases/download/v3/dpt_hybrid_384.pt",
"sha256": "501f0c75b3bca7daec6b3682c5054c09b366765aef6fa3a09d03a5cb4b230853",
"release": "v3", "output_type": "relative_inverse_depth"},
"large": {"url": "https://github.com/isl-org/MiDaS/releases/download/v3/dpt_large_384.pt",
"sha256": "2f21e586477d90cb9624c7eef5df7891edca49a1c4795ee2cb631fd4daa6ca69",
"release": "v3", "output_type": "relative_inverse_depth"},
}
def checkpoint_info(variant: str) -> dict:
return dict(_CHECKPOINTS[variant])
def file_sha256(path: Path) -> str:
digest = hashlib.sha256()
with Path(path).open("rb") as stream:
for block in iter(lambda: stream.read(1 << 20), b""):
digest.update(block)
return digest.hexdigest()
def source_root() -> Path:
root = Path(os.environ.get("BENCH_MIDAS_SOURCE", str(RUNTIME_ROOT))).resolve()
digest = hashlib.sha256()
files = sorted(root.rglob("*.py"))
for path in files:
digest.update(str(path.relative_to(root)).encode() + b"\0" + path.read_bytes())
if digest.hexdigest() != SOURCE_TREE_SHA256:
raise RuntimeError(f"missing or changed pinned MiDaS runtime source {UPSTREAM_COMMIT}: {root}")
return root
def preprocess_for(image, input_spec: str, variant: str):
"""Static deployment resize; per-variant normalization from midas.model_loader.
The v3 DPT checkpoints train with mean=std=0.5, the v2.1 small checkpoint
with ImageNet statistics; both follow the official loader branches.
"""
import cv2
import numpy as np
size = int(INPUT_SPECS[variant].split("x")[-1])
if input_spec != INPUT_SPECS[variant]:
raise ValueError(f"MiDaS {variant} recipe is fixed at {INPUT_SPECS[variant]}, got {input_spec}")
mean, std = (([.5, .5, .5], [.5, .5, .5]) if variant != "small"
else ([.485, .456, .406], [.229, .224, .225]))
rgb = np.asarray(image.convert("RGB"), dtype=np.float64) / 255.0
rgb = cv2.resize(rgb, (size, size), interpolation=cv2.INTER_CUBIC)
rgb = (rgb - np.array(mean)) / np.array(std)
return np.ascontiguousarray(rgb.transpose(2, 0, 1)[None], dtype=np.float32)
def _static_forward_flex(self, x):
"""Official midas backbones/vit.py forward_flex with python-int grid sizes.
x.shape entries become Tensors under torch.onnx tracing (torch >= 2.x) and
then break _resize_pos_embed's F.interpolate and nn.Unflatten validation.
The pinned deployment resolution is static, so the ints are constants.
"""
import torch
h, w = int(x.shape[2]), int(x.shape[3])
pos_embed = self._resize_pos_embed(self.pos_embed, h // self.patch_size[1], w // self.patch_size[0])
b = x.shape[0]
if hasattr(self.patch_embed, "backbone"):
x = self.patch_embed.backbone(x)
if isinstance(x, (list, tuple)):
x = x[-1]
x = self.patch_embed.proj(x).flatten(2).transpose(1, 2)
if getattr(self, "dist_token", None) is not None:
cls_tokens = self.cls_token.expand(b, -1, -1)
dist_token = self.dist_token.expand(b, -1, -1)
x = torch.cat((cls_tokens, dist_token, x), dim=1)
else:
if self.no_embed_class:
x = x + pos_embed
cls_tokens = self.cls_token.expand(b, -1, -1)
x = torch.cat((cls_tokens, x), dim=1)
if not self.no_embed_class:
x = x + pos_embed
x = self.pos_drop(x)
for blk in self.blocks:
x = blk(x)
x = self.norm(x)
return x
def _static_forward_vit(pretrained, x):
"""Official midas backbones/utils.py forward_adapted_unflatten, int sizes."""
import torch
import torch.nn as nn
h, w = int(x.shape[2]), int(x.shape[3])
glob = pretrained.model.forward_flex(x)
layer_1 = pretrained.activations["1"]
layer_2 = pretrained.activations["2"]
layer_3 = pretrained.activations["3"]
layer_4 = pretrained.activations["4"]
layer_1 = pretrained.act_postprocess1[0:2](layer_1)
layer_2 = pretrained.act_postprocess2[0:2](layer_2)
layer_3 = pretrained.act_postprocess3[0:2](layer_3)
layer_4 = pretrained.act_postprocess4[0:2](layer_4)
unflatten = nn.Sequential(
nn.Unflatten(2, torch.Size([
h // pretrained.model.patch_size[1],
w // pretrained.model.patch_size[0],
])))
if layer_1.ndim == 3:
layer_1 = unflatten(layer_1)
if layer_2.ndim == 3:
layer_2 = unflatten(layer_2)
if layer_3.ndim == 3:
layer_3 = unflatten(layer_3)
if layer_4.ndim == 3:
layer_4 = unflatten(layer_4)
layer_1 = pretrained.act_postprocess1[3: len(pretrained.act_postprocess1)](layer_1)
layer_2 = pretrained.act_postprocess2[3: len(pretrained.act_postprocess2)](layer_2)
layer_3 = pretrained.act_postprocess3[3: len(pretrained.act_postprocess3)](layer_3)
layer_4 = pretrained.act_postprocess4[3: len(pretrained.act_postprocess4)](layer_4)
return layer_1, layer_2, layer_3, layer_4
def _install_static_vit_path(model, input_spec: str):
"""Swap the ViT flex path for the int-size twins and prove bitwise equality.
The official modules, weights and math are untouched: the twins only turn
the traced Tensor shape entries into python ints for the fixed deployment
resolution. A fixed deterministic input verifies the swapped forward stays
bitwise identical to the untouched official forward before returning.
"""
import types
import torch
size = int(input_spec.split("x")[-1])
generator = torch.Generator().manual_seed(20260918)
probe = torch.rand((1, 3, size, size), generator=generator)
with torch.inference_mode():
official = model(probe).clone()
model.pretrained.model.forward_flex = types.MethodType(_static_forward_flex, model.pretrained.model)
model.forward_transformer = _static_forward_vit
with torch.inference_mode():
replaced = model(probe)
if not torch.equal(official, replaced):
raise RuntimeError("static-grid ViT path is not bitwise identical to the official forward")
return model
def build_variant(variant: str, weights: Path, input_spec: str):
import torch
if input_spec != INPUT_SPECS[variant]:
raise ValueError(f"expected fixed {INPUT_SPECS[variant]}, got {input_spec}")
root = source_root()
if file_sha256(weights) != _CHECKPOINTS[variant]["sha256"]:
raise ValueError(f"wrong official MiDaS {variant} checkpoint")
if str(root) not in sys.path:
sys.path.insert(0, str(root))
# Constructing with the checkpoint path keeps use_pretrained=False: the
# official ImageNet backbone download never runs and the strict load is
# performed by MiDaS BaseModel.load itself.
if variant == "small":
from midas.midas_net_custom import MidasNet_small
model = MidasNet_small(Path(weights), features=64, backbone="efficientnet_lite3",
exportable=True, non_negative=True, blocks={"expand": True})
else:
from midas.dpt_depth import DPTDepthModel
backbone = "vitl16_384" if variant == "large" else "vitb_rn50_384"
model = DPTDepthModel(path=str(weights), backbone=backbone, non_negative=True)
_install_static_vit_path(model, input_spec)
return model.eval()

View File

@ -0,0 +1,19 @@
"""Official MiDaS v2.1 small: relative inverse depth at static 256x256."""
from pathlib import Path
from .midas_common import (EXPORTER_VERSION, INPUT_SPECS, MODEL_SPECS, OPSET, PREPROCESSES,
RESTORE_ALIGN_CORNERS, RESTORE_INTERPOLATION, build_variant,
checkpoint_info, preprocess_for)
CHECKPOINT = checkpoint_info("small")
INPUT_SPEC = INPUT_SPECS["small"]
MODEL_SPEC = MODEL_SPECS["small"]
PREPROCESS = PREPROCESSES["small"]
def build(weights: Path, input_spec: str):
return build_variant("small", weights, input_spec)
def preprocess(image, input_spec: str = INPUT_SPEC):
return preprocess_for(image, input_spec, "small")

View File

@ -0,0 +1,222 @@
"""Pinned MMDetection dense detectors at the CSV's static 640 deployment size.
Only deployment geometry differs from the upstream 1333x800 test pipeline:
keep-ratio resize to 640, then normalized zero padding at bottom/right to 640.
Weights, normalization, raw heads and official postprocessing are unchanged.
Heavy imports remain inside runtime functions for lightweight recipe discovery.
"""
from functools import lru_cache
from types import SimpleNamespace
from .openmmlab import build as build_official, get_recipe
RUNTIME_PYTHON = "/home/ming01.wang/outputs/full-coverage-20260917/openmmlab/venv/bin/python"
INPUT_SPEC = "1x3x640x640"
MODEL_SPEC = {"source": "custom", "dataset": "coco2017", "input_spec": INPUT_SPEC}
MODELS = ("retinanet.r50.fpn", "retinanet.r101.fpn", "fcos.r50.fpn",
"atss.r50.fpn", "gfl.r50.fpn", "vfnet.r50.fpn",
"foveabox.r50.fpn", "fsaf.r50.fpn", "reppoints.r50.fpn")
HEADS = {"retinanet": ("RetinaHead", (720, 36)), "fcos": ("FCOSHead", (80, 4, 1)),
"atss": ("ATSSHead", (80, 4, 1)), "gfl": ("GFLHead", (80, 68)),
"vfnet": ("VFNetHead", (80, 4)), "foveabox": ("FoveaHead", (80, 4)),
"fsaf": ("FSAFHead", (80, 4)), "reppoints": ("RepPointsHead", (80, 4))}
STRIDES = (8, 16, 32, 64, 128)
TEST_CFG = {"nms_pre": 1000, "min_bbox_size": 0, "score_thr": .05,
"nms": {"type": "nms", "iou_threshold": .5}, "max_per_img": 100}
def test_config(family):
from copy import deepcopy
cfg = deepcopy(TEST_CFG)
if family in ("atss", "gfl", "vfnet"):
cfg["nms"]["iou_threshold"] = .6
if family == "foveabox":
del cfg["min_bbox_size"]
return cfg
def recipe(model):
if model not in MODELS:
raise ValueError(model)
dp = get_recipe(model)["preprocess"]["official_data_preprocessor"]
return {"recipe": "mmdet-dense-static640-v1", "input_spec": INPUT_SPEC,
"color": "RGB" if dp["bgr_to_rgb"] else "BGR",
"mean": dp["mean"], "std": dp["std"],
"resize": {"scale": [640, 640], "keep_ratio": True,
"rounding": "int(value+0.5)", "backend": "cv2", "interpolation": "bilinear"},
"pad": {"size": [640, 640], "placement": "bottom_right", "value": 0,
"stage": "after_normalization"}, "layout": "NCHW", "dtype": "float32"}
def geometry(original_size):
w, h = map(int, original_size)
if min(w, h) <= 0:
raise ValueError(original_size)
scale = 640 / max(w, h)
nw, nh = int(w * scale + .5), int(h * scale + .5)
if min(nw, nh) < 1:
raise ValueError("Image collapses under official keep-ratio resize")
# Clipping is to resized image geometry, not the normalized padded canvas.
return (nw, nh), (nw / w, nh / h), (nh, nw)
def preprocess_variant(model, image, input_spec=INPUT_SPEC):
if input_spec != INPUT_SPEC:
raise ValueError(f"Expected {INPUT_SPEC}, got {input_spec}")
import mmcv
import numpy as np
meta = recipe(model)
bgr = np.asarray(image.convert("RGB"))[:, :, ::-1].copy()
x = mmcv.imrescale(bgr, (640, 640), interpolation="bilinear", backend="cv2")
if meta["color"] == "RGB":
x = x[:, :, ::-1]
x = np.ascontiguousarray(x.transpose(2, 0, 1), dtype=np.float32)
x = (x - np.asarray(meta["mean"], dtype=np.float32)[:, None, None]) / np.asarray(meta["std"], dtype=np.float32)[:, None, None]
out = np.zeros((1, 3, 640, 640), dtype=np.float32)
out[0, :, :x.shape[1], :x.shape[2]] = x
return out
@lru_cache(maxsize=8)
def _decoder(family):
# The official BaseDenseHead owns filtering, centerness, clipping, rescaling
# and class-aware NMS. This context has no learned layers or checkpoint.
from mmengine.config import ConfigDict
from mmdet.models.dense_heads.base_dense_head import BaseDenseHead
from mmdet.models.task_modules.prior_generators import AnchorGenerator, MlvlPointGenerator
from mmdet.models.task_modules.coders import DeltaXYWHBBoxCoder, DistancePointBBoxCoder, TBLRBBoxCoder
class Decoder(BaseDenseHead):
def loss_by_feat(self, **kwargs):
raise NotImplementedError("Inference-only decoder")
if family == "gfl":
from mmdet.models.dense_heads.gfl_head import GFLHead, Integral
Decoder._predict_by_feat_single = GFLHead._predict_by_feat_single
Decoder.anchor_center = GFLHead.anchor_center
elif family in ("foveabox", "reppoints"):
from mmdet.models.dense_heads import FoveaHead, RepPointsHead
official = FoveaHead if family == "foveabox" else RepPointsHead
Decoder._predict_by_feat_single = official._predict_by_feat_single
Decoder._bbox_decode = official._bbox_decode
decoder = Decoder()
if family in ("fcos", "vfnet", "foveabox", "reppoints"):
decoder.prior_generator = MlvlPointGenerator(STRIDES,
offset=.5 if family in ("fcos", "foveabox") else 0.)
else:
retina = family == "retinanet"
decoder.prior_generator = AnchorGenerator(strides=STRIDES,
ratios=[.5, 1., 2.] if retina else [1.],
octave_base_scale=4 if retina else (1 if family == "fsaf" else 8),
scales_per_octave=3 if retina else 1)
if family in ("retinanet", "atss"):
decoder.bbox_coder = DeltaXYWHBBoxCoder(target_means=[0., 0., 0., 0.],
target_stds=[.1, .1, .2, .2] if family == "atss" else [1., 1., 1., 1.])
elif family == "fsaf":
decoder.bbox_coder = TBLRBBoxCoder(normalizer=4.0)
else:
decoder.bbox_coder = DistancePointBBoxCoder()
if family == "gfl":
decoder.integral = Integral(16)
elif family == "foveabox":
decoder.strides = list(STRIDES)
decoder.base_edge_list = [16, 32, 64, 128, 256]
elif family == "reppoints":
decoder.point_strides = list(STRIDES)
decoder.cls_out_channels = 80
decoder.use_sigmoid_cls = True
decoder.loss_cls = SimpleNamespace(custom_cls_channels=False)
decoder.test_cfg = ConfigDict(test_config(family))
return decoder
def output_names(model):
groups = ("cls", "bbox", "centerness") if model.split(".")[0] in ("fcos", "atss") else ("cls", "bbox")
return [f"{group}_s{s}" for group in groups for s in STRIDES]
def decode_variant(model, outputs, original_size):
import numpy as np
import torch
family = model.split(".")[0]
channels = HEADS[family][1]
if len(outputs) != len(channels) * 5:
raise ValueError(f"{model}: wrong raw head count {len(outputs)}")
raw = []
for i, value in enumerate(outputs):
expected = (1, channels[i // 5], 640 // STRIDES[i % 5], 640 // STRIDES[i % 5])
array = np.asarray(value)
if array.shape != expected or not np.isfinite(array).all():
raise ValueError(f"{model}: invalid raw head {i}: {array.shape}, expected {expected}")
raw.append(torch.as_tensor(np.ascontiguousarray(array), dtype=torch.float32))
_, scales, img_shape = geometry(original_size)
meta = dict(img_shape=img_shape, ori_shape=tuple(reversed(original_size)),
pad_shape=(640, 640), batch_input_shape=(640, 640), scale_factor=scales)
with torch.inference_mode():
result = _decoder(family).predict_by_feat(
raw[:5], raw[5:10], score_factors=raw[10:] if len(channels) == 3 else None,
batch_img_metas=[meta], rescale=True)[0]
return result.bboxes.numpy(), result.scores.numpy(), result.labels.numpy()
def build_variant(model, weights, input_spec):
import torch
raw = build_official(model, weights, input_spec)
cfg, head = raw.cfg, raw.upstream.bbox_head
meta = recipe(model)
dp = cfg.model.data_preprocessor
family = model.split(".")[0]
if (dict(cfg.model.test_cfg) != test_config(family) or head.num_classes != 80
or head.__class__.__name__ != HEADS[family][0]
or not head.use_sigmoid_cls or head.cls_out_channels != 80
or list(dp.mean) != meta["mean"] or list(dp.std) != meta["std"]
or dp.bgr_to_rgb != (meta["color"] == "RGB") or dp.pad_size_divisor != 32):
raise ValueError("Pinned official dense detector contract changed")
decoder = _decoder(family)
# Compare actual upstream geometry helpers, including defaults not in config.
for name in ("strides", "num_base_priors"):
if getattr(head.prior_generator, name) != getattr(decoder.prior_generator, name):
raise ValueError(f"Official prior {name} differs")
sizes = [(640 // s, 640 // s) for s in STRIDES]
for expected, actual in zip(head.prior_generator.grid_priors(sizes, device="cpu"),
decoder.prior_generator.grid_priors(sizes, device="cpu")):
torch.testing.assert_close(actual, expected, rtol=0, atol=0)
if head.bbox_coder.__dict__ != decoder.bbox_coder.__dict__:
raise ValueError("Official bbox coder differs")
if family == "gfl":
torch.testing.assert_close(head.integral.project, decoder.integral.project, rtol=0, atol=0)
elif family == "foveabox":
if (list(head.strides) != decoder.strides or list(head.base_edge_list) != decoder.base_edge_list
or head.with_deform):
raise ValueError("Official Fovea geometry differs")
elif family == "reppoints":
if (head.point_strides != decoder.point_strides or head.num_points != 9
or head.transform_method != "moment"):
raise ValueError("Official RepPoints geometry differs")
if family in ("vfnet", "reppoints"):
# VFNet's upstream head overrides its state-load hook; verify every
# learned parameter against the checkpoint before wrapping export ops.
state = torch.load(str(weights), map_location="cpu")["state_dict"]
for name, parameter in raw.upstream.named_parameters():
torch.testing.assert_close(parameter, state[name], rtol=0, atol=0)
del state
from .vfnet_deformconv import enable_export
if family == "reppoints":
enable_export(head, ("reppoints_cls_conv", "reppoints_pts_refine_conv"))
else:
enable_export(head)
class RawHeads(torch.nn.Module):
def __init__(self):
super().__init__()
self.upstream = raw.upstream
self.upstream.cfg = cfg
self.cfg = cfg
self.checkpoint_sha256 = raw.checkpoint_sha256
def forward(self, x):
return tuple(value for level in self.upstream._forward(x) for value in level)
return RawHeads().eval()

View File

@ -0,0 +1,141 @@
"""Exact pinned MMDetection DETR models, including their complete transformers.
Static640 follows the CSV deployment protocol. Padding masks are derived inside
the graph from normalized-zero padding, never frozen to the export image shape.
Only parameter-free official box selection and geometry run in host decode.
"""
from functools import lru_cache
from types import SimpleNamespace
from .openmmlab import build as build_official, get_recipe
from .mmdet_dense_common import RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, geometry
MODELS = ("detr.r50", "deformable.detr.r50", "dab.detr.r50")
OPSET = 16 # Standard ONNX GridSample for genuine multi-scale deformable attention.
CONTRACTS = {"detr.r50": ("DETR", "DETRHead", 100, 81, 100),
"deformable.detr.r50": ("DeformableDETR", "DeformableDETRHead", 300, 80, 100),
"dab.detr.r50": ("DABDETR", "DABDETRHead", 300, 80, 300)}
def recipe(model):
if model not in MODELS:
raise ValueError(model)
from .mmdet_dense_common import recipe as image_recipe
meta = image_recipe("retinanet.r50.fpn")
dp = get_recipe(model)["preprocess"]["official_data_preprocessor"]
if (dp["mean"] != meta["mean"] or dp["std"] != meta["std"]
or not dp["bgr_to_rgb"] or dp["pad_size_divisor"] != 1):
raise ValueError("Official DETR image normalization changed")
# Same image transformation permits existing calibration cache reuse.
return meta
def preprocess_variant(model, image, input_spec=INPUT_SPEC):
from .mmdet_dense_common import preprocess_variant as image_preprocess
recipe(model) # verify exact equality before reusing the image-only helper
return image_preprocess("retinanet.r50.fpn", image, input_spec)
def output_names(model):
if model not in MODELS:
raise ValueError(model)
return ["class_logits", "boxes_cxcywh"]
def padding_mask(x):
# Resized RGB pixels are uint8 and all official channel means are fractional:
# a real pixel cannot be normalized zero. Zero is exclusively padding.
return (x == 0).all(dim=1)
def transformer_inputs(network, features, x, deformable):
"""Official pre_transformer math with the real per-image mask retained."""
import torch
from torch.nn import functional as F
mask = padding_mask(x).to(x.dtype)
masks = [F.interpolate(mask[:, None], size=f.shape[-2:]).to(torch.bool)[:, 0]
for f in features]
positions = [network.positional_encoding(m) for m in masks]
flatten = lambda f: f.flatten(2).transpose(1, 2)
if not deformable:
feat, pos, mask = flatten(features[-1]), flatten(positions[-1]), masks[-1].flatten(1)
return dict(feat=feat, feat_mask=mask, feat_pos=pos), dict(memory_mask=mask, memory_pos=pos)
shapes = torch.tensor([list(f.shape[-2:]) for f in features], dtype=torch.long, device=x.device)
starts = torch.cat((shapes.new_zeros(1), shapes.prod(1).cumsum(0)[:-1]))
valid = torch.stack([network.get_valid_ratio(m) for m in masks], 1)
mask = torch.cat([m.flatten(1) for m in masks], 1)
pos = torch.cat([flatten(p) + network.level_embed[i].view(1, 1, -1)
for i, p in enumerate(positions)], 1)
extra = dict(spatial_shapes=shapes, level_start_index=starts, valid_ratios=valid)
return (dict(feat=torch.cat([flatten(f) for f in features], 1), feat_mask=mask, feat_pos=pos, **extra),
dict(memory_mask=mask, **extra))
@lru_cache(maxsize=3)
def _decoder(model):
from mmdet.models.dense_heads.detr_head import DETRHead
_, _, _, channels, topk = CONTRACTS[model]
context = SimpleNamespace(test_cfg={"max_per_img": topk}, num_classes=80,
loss_cls=SimpleNamespace(use_sigmoid=channels == 80))
from functools import partial
return partial(DETRHead._predict_by_feat_single, context)
def decode_variant(model, outputs, original_size):
import numpy as np
import torch
_, _, queries, channels, _ = CONTRACTS[model]
if len(outputs) != 2:
raise ValueError("Expected final query logits and cxcywh tensors")
tensors = []
for value, shape in zip(outputs, ((1, queries, channels), (1, queries, 4))):
value = np.asarray(value)
if value.shape != shape or not np.isfinite(value).all():
raise ValueError(f"Invalid {model} raw output: {value.shape}, expected {shape}")
tensors.append(torch.tensor(value, dtype=torch.float32)[0])
_, scales, img_shape = geometry(original_size)
with torch.inference_mode():
result = _decoder(model)(*tensors, dict(img_shape=img_shape, scale_factor=scales), rescale=True)
return result.bboxes.numpy(), result.scores.numpy(), result.labels.numpy()
def build_variant(model, weights, input_spec):
import torch
raw = build_official(model, weights, input_spec)
network, cfg = raw.upstream, raw.cfg
kind, head, queries, channels, topk = CONTRACTS[model]
meta = recipe(model)
dp = cfg.model.data_preprocessor
if (type(network).__name__ != kind or type(network.bbox_head).__name__ != head
or network.num_queries != queries or network.bbox_head.cls_out_channels != channels
or network.bbox_head.num_classes != 80 or dict(cfg.model.test_cfg) != {"max_per_img": topk}
or list(dp.mean) != meta["mean"] or list(dp.std) != meta["std"]
or not dp.bgr_to_rgb or dp.pad_size_divisor != 1):
raise ValueError("Pinned official DETR contract changed")
deformable = kind == "DeformableDETR"
if deformable and (network.with_box_refine or network.as_two_stage or network.num_feature_levels != 4):
raise ValueError("Wrong Deformable-DETR variant")
state = torch.load(str(weights), map_location="cpu")["state_dict"]
for name, parameter in network.named_parameters():
torch.testing.assert_close(parameter, state[name], rtol=0, atol=0)
del state
class RawQueries(torch.nn.Module):
def __init__(self):
super().__init__()
self.upstream = network
self.upstream.cfg = cfg
self.cfg = cfg
self.checkpoint_sha256 = raw.checkpoint_sha256
def forward(self, x):
features = self.upstream.extract_feat(x)
encoder, decoder = transformer_inputs(self.upstream, features, x, deformable)
memory = self.upstream.forward_encoder(**encoder)
query, head_inputs = self.upstream.pre_decoder(**memory)
decoder.update(query)
head_inputs.update(self.upstream.forward_decoder(**decoder))
scores, boxes = self.upstream.bbox_head(**head_inputs)
return scores[-1], boxes[-1]
return RawQueries().eval()

View File

@ -0,0 +1,290 @@
"""Pinned MMDetection query-based instance segmentation at the CSV static 640 size.
Mask2Former COCO instance checkpoints: the graph returns the final decoder
layer's query classification logits and per-query mask logits over the
stride-4 mask feature map. The official MaskFormerHead.predict upsampling and
the parameter-free MaskFormerFusionHead.instance_postprocess run in host decode
on those two raw tensors, exactly like the dense-detector decoders.
Deployment geometry follows mmdet_dense_common (upstream 1333x800 keep-ratio
test resize pinned to 640, normalized zero padding at bottom/right), so the
image recipe and calibration digests stay shared. Heavy imports stay inside
runtime functions for lightweight recipe discovery.
"""
from functools import lru_cache
from .openmmlab import get_recipe, load_config, verify_checkpoint
from .mmdet_dense_common import RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, geometry
MODELS = ("mask2former.swin.t.instance", "mask2former.swin.s.instance",
"mask2former.swin.b.instance")
# stage-3 blocks, patch-embed channels, queries, classes, stuff classes and
# the official test_cfg flavor, per resolved official config.
CONTRACTS = {"mask2former.swin.t.instance": (6, 96, 100, 80, 0, "instance"),
"mask2former.swin.s.instance": (18, 96, 100, 80, 0, "instance"),
"mask2former.swin.b.instance": (18, 128, 100, 133, 53, "panoptic")}
TEST_CFGS = {"instance": {"panoptic_on": False, "semantic_on": False, "instance_on": True,
"max_per_image": 100, "iou_thr": 0.8, "filter_low_score": True},
"panoptic": {"panoptic_on": True, "semantic_on": False, "instance_on": True,
"max_per_image": 100, "iou_thr": 0.8, "filter_low_score": True}}
# The COCO instance configs load to float32 before resizing; the swin-b
# panoptic config's test pipeline keeps the default uint8 load.
F32_LOAD = {"mask2former.swin.t.instance", "mask2former.swin.s.instance"}
def recipe(model):
"""Image recipe variant: instance configs load to float32 before resizing."""
if model not in MODELS:
raise ValueError(model)
from .mmdet_dense_common import recipe as image_recipe
meta = image_recipe("retinanet.r50.fpn")
dp = get_recipe(model)["preprocess"]["official_data_preprocessor"]
if (dp["mean"] != meta["mean"] or dp["std"] != meta["std"]
or not dp["bgr_to_rgb"] or dp["pad_size_divisor"] != 32):
raise ValueError("Official Mask2Former image normalization changed")
if model in F32_LOAD:
return {**meta, "recipe": "mmdet-mask2former-static640-f32-v1",
"load_dtype": "float32_before_resize"}
return meta
def preprocess_variant(model, image, input_spec=INPUT_SPEC):
if input_spec != INPUT_SPEC:
raise ValueError(f"Expected {INPUT_SPEC}, got {input_spec}")
meta = recipe(model)
if model not in F32_LOAD:
from .mmdet_dense_common import preprocess_variant as image_preprocess
return image_preprocess("retinanet.r50.fpn", image, input_spec)
import mmcv
import numpy as np
# Official pipeline: LoadImageFromFile(to_float32=True) -> Resize -> normalize.
bgr = np.asarray(image.convert("RGB"))[:, :, ::-1].copy().astype(np.float32)
x = mmcv.imrescale(bgr, (640, 640), interpolation="bilinear", backend="cv2")
if meta["color"] == "RGB":
x = x[:, :, ::-1]
x = np.ascontiguousarray(x.transpose(2, 0, 1), dtype=np.float32)
x = (x - np.asarray(meta["mean"], dtype=np.float32)[:, None, None]) / np.asarray(meta["std"], dtype=np.float32)[:, None, None]
out = np.zeros((1, 3, 640, 640), dtype=np.float32)
out[0, :, :x.shape[1], :x.shape[2]] = x
return out
def output_names(model):
if model not in MODELS:
raise ValueError(model)
return ["cls_logits", "mask_logits"]
DEBUG_OUTPUT_NAMES = ["feat3", "mask_features", "msm0", "msm1", "msm2", "cls_logits", "mask_logits"]
def build_debug_variant(model, weights, input_spec):
"""Same official forward with pre-decoder tensors exposed for the probe.
The Mask2Former transformer decoder derives boolean attention masks from
interpolated mask logits (sigmoid < 0.5). Torch/ONNX fp reassociation
(~1e-4) can flip borderline bits, and each flip changes decoder outputs
discretely (measured: flipping 0.001% of the bits moves class logits by
5.2 and mask logits by 112). Raw-output cosine therefore only gates the
continuous backbone/pixel-decoder segment; decoder equivalence is judged
on decoded predictions and the COCO metric.
"""
network = build_variant(model, weights, input_spec)
upstream, head = network.upstream, network.upstream.panoptic_head
import torch
class DebugTaps(torch.nn.Module):
def __init__(self):
super().__init__()
self.raw = network
def forward(self, x):
feats = upstream.extract_feat(x)
mask_features, multi_scale_memorys = head.pixel_decoder(feats)
batch_size = x.shape[0]
decoder_inputs, decoder_positional_encodings = [], []
for i in range(head.num_transformer_feat_level):
decoder_input = head.decoder_input_projs[i](multi_scale_memorys[i])
decoder_input = decoder_input.flatten(2).permute(0, 2, 1)
decoder_input = decoder_input + head.level_embed.weight[i].view(1, 1, -1)
mask = decoder_input.new_zeros(
(batch_size,) + multi_scale_memorys[i].shape[-2:], dtype=torch.bool)
positional = head.decoder_positional_encoding(mask).flatten(2).permute(0, 2, 1)
decoder_inputs.append(decoder_input)
decoder_positional_encodings.append(positional)
query_feat = head.query_feat.weight.unsqueeze(0).repeat((batch_size, 1, 1))
query_embed = head.query_embed.weight.unsqueeze(0).repeat((batch_size, 1, 1))
cls_pred, mask_pred, attn_mask = head._forward_head(
query_feat, mask_features, multi_scale_memorys[0].shape[-2:])
for i in range(head.num_transformer_decoder_layers):
level_idx = i % head.num_transformer_feat_level
mask_sum = (attn_mask.sum(-1) != attn_mask.shape[-1]).unsqueeze(-1)
attn_mask = attn_mask & mask_sum
layer = head.transformer_decoder.layers[i]
query_feat = layer(
query=query_feat, key=decoder_inputs[level_idx],
value=decoder_inputs[level_idx], query_pos=query_embed,
key_pos=decoder_positional_encodings[level_idx],
cross_attn_mask=attn_mask, query_key_padding_mask=None,
key_padding_mask=None)
cls_pred, mask_pred, attn_mask = head._forward_head(
query_feat, mask_features,
multi_scale_memorys[(i + 1) % head.num_transformer_feat_level].shape[-2:])
return (feats[-1], mask_features, multi_scale_memorys[0],
multi_scale_memorys[1], multi_scale_memorys[2], cls_pred, mask_pred)
return DebugTaps().eval()
@lru_cache(maxsize=3)
def _decoder(model):
"""Official parameter-free instance postprocess; no learned weights here."""
from mmdet.models.seg_heads.panoptic_fusion_heads.maskformer_fusion_head import MaskFormerFusionHead
from mmengine.config import ConfigDict
_, _, _, classes, stuff, flavor = CONTRACTS[model]
decoder = MaskFormerFusionHead.__new__(MaskFormerFusionHead)
decoder.num_classes = classes
decoder.num_things_classes = classes - stuff
decoder.num_stuff_classes = stuff
decoder.test_cfg = ConfigDict(TEST_CFGS[flavor])
return decoder
def decode_variant(model, outputs, original_size):
"""Official predict path: upsample -> crop -> rescale -> instance_postprocess."""
import numpy as np
import torch
from torch.nn import functional as F
if model not in MODELS:
raise ValueError(model)
queries, classes = CONTRACTS[model][2], CONTRACTS[model][3]
if len(outputs) != 2:
raise ValueError(f"{model}: expected query logits and mask logits")
cls_logits, mask_logits = outputs
cls_logits = np.asarray(cls_logits)
mask_logits = np.asarray(mask_logits)
if cls_logits.shape != (1, queries, classes + 1) or not np.isfinite(cls_logits).all():
raise ValueError(f"{model}: invalid cls logits {cls_logits.shape}")
if (mask_logits.ndim != 4 or mask_logits.shape[:2] != (1, queries)
or not np.isfinite(mask_logits).all()):
raise ValueError(f"{model}: invalid mask logits {mask_logits.shape}")
(nw, nh), scales, img_shape = geometry(original_size)
size = int(INPUT_SPEC.split("x")[-1])
with torch.inference_mode():
# MaskFormerHead.predict: upsample to batch_input_shape first.
masks = F.interpolate(torch.from_numpy(mask_logits), size=(size, size),
mode="bilinear", align_corners=False)
# MaskFormerFusionHead.predict: remove padding, then rescale.
masks = masks[0, ..., :nh, :nw]
ori_h, ori_w = int(original_size[1]), int(original_size[0])
masks = F.interpolate(masks[:, None], size=(ori_h, ori_w),
mode="bilinear", align_corners=False)[:, 0]
result = _decoder(model).instance_postprocess(
torch.from_numpy(cls_logits)[0], masks)
return (result.bboxes.numpy(), result.scores.numpy(), result.labels.numpy(),
result.masks.numpy())
def g2_input_variant(model, t1, item, input_spec):
"""Diagnostic-input hook for harness/eval/run.py g2_inputs."""
return preprocess_variant(model, t1.image(item), input_spec)
def predict_variant(model, runner, t1, input_spec, pose_evidence=None):
"""Shared harness/eval/coco_extra.py hook: COCO segm RLE predictions."""
import numpy as np
from pycocotools import mask as mask_utils
from ..eval.common import run_batched
if pose_evidence is not None:
raise ValueError(f"{model}: iseg predict takes no pose evidence")
cat_map = t1.manifest["cat_map"]
predictions = []
for offset in range(0, len(t1.items), 16):
items = t1.items[offset:offset + 16]
images = [t1.image(it) for it in items]
xs = [preprocess_variant(model, image, input_spec) for image in images]
outputs = run_batched(runner, xs)
for it, image, outs in zip(items, images, outputs):
_, scores, labels, masks = decode_variant(model, outs, image.size)
for mask, score, label in zip(masks, scores, labels):
rle = mask_utils.encode(np.asfortranarray(mask.astype(np.uint8)))
rle["counts"] = rle["counts"].decode("ascii")
predictions.append({"image_id": it["id"],
"category_id": cat_map[str(int(label))],
"score": float(score), "segmentation": rle})
return predictions
def build_variant(model, weights, input_spec):
"""Construct the official Mask2Former and strictly load its checkpoint.
MaskFormer has no bbox_head, so the shared openmmlab tensor route cannot
serve it; this builder uses the same public recipe/config/checkpoint
helpers and the same strict-load rules as openmmlab.build.
"""
import importlib
import torch
from mmengine.registry import init_default_scope
from mmengine.runner import load_checkpoint
if model not in MODELS or input_spec != INPUT_SPEC:
raise ValueError(f"{model}: expected {INPUT_SPEC}")
model_recipe = get_recipe(model)
if model_recipe["resolution_status"] != "resolved_catalog":
raise ValueError(f"Unresolved recipe {model}: {model_recipe['blockers']}")
if input_spec != model_recipe["input_spec"]:
raise ValueError(f"{model}: recipe input {model_recipe['input_spec']}, got {input_spec}")
sha = verify_checkpoint(model_recipe, weights)
cfg = load_config(model)
importlib.import_module("mmdet.utils").register_all_modules(init_default_scope=False)
init_default_scope("mmdet")
registry = importlib.import_module("mmdet.registry")
network = registry.MODELS.build(cfg.model)
# MMEngine marks normalization buffers non-persistent; official checkpoints
# keep them. Preserve every key and verify before restoring persistence.
state = torch.load(str(weights), map_location="cpu")["state_dict"]
preprocessor = network.data_preprocessor
for name in ("mean", "std"):
key = f"data_preprocessor.{name}"
if key in state and name in preprocessor._non_persistent_buffers_set:
expected = getattr(preprocessor, name)
if not torch.equal(state[key], expected):
raise ValueError(f"Checkpoint {key} disagrees with the official config")
preprocessor.register_buffer(name, expected, persistent=True)
load_checkpoint(network, str(weights), map_location="cpu", strict=True)
network.cpu().eval()
depth3, embed, queries, classes, stuff, flavor = CONTRACTS[model]
head = network.panoptic_head
dp = cfg.model.data_preprocessor
meta = recipe(model)
if (type(network).__name__ != "Mask2Former" or type(head).__name__ != "Mask2FormerHead"
or type(network.backbone).__name__ != "SwinTransformer"
or [len(stage.blocks) for stage in network.backbone.stages] != [2, 2, depth3, 2]
or network.backbone.patch_embed.projection.out_channels != embed
or head.num_queries != queries or head.num_classes != classes
or head.num_things_classes != classes - stuff or head.num_stuff_classes != stuff
or dict(cfg.model.test_cfg) != TEST_CFGS[flavor]
or list(dp.mean) != meta["mean"] or list(dp.std) != meta["std"]
or not dp.bgr_to_rgb or dp.pad_size_divisor != 32):
raise ValueError("Pinned official Mask2Former contract changed")
# Verify every learned parameter against the checkpoint bytes.
for name, parameter in network.named_parameters():
torch.testing.assert_close(parameter, state[name], rtol=0, atol=0)
del state
class RawQueries(torch.nn.Module):
def __init__(self):
super().__init__()
self.upstream = network
self.upstream.cfg = cfg
self.cfg = cfg
self.checkpoint_sha256 = sha
def forward(self, x):
# Mask2FormerHead.forward is metadata-free: batch size from x.
cls_pred_list, mask_pred_list = self.upstream.panoptic_head(
self.upstream.extract_feat(x), None)
return cls_pred_list[-1], mask_pred_list[-1]
return RawQueries().eval()

View File

@ -0,0 +1,209 @@
"""Pinned MMDetection SOLO/SOLOv2 instance segmentation at static 640.
Both heads emit dense static tensors (per-grid mask/kernel predictions, class
scores and a unified mask feature map); the official matrix-NMS decode and the
dynamic mask convolutions are parameter-free and run in host decode, exactly
like the dense-detector decoders. SOLO result bboxes are upstream placeholders
(all zeros); masks carry the real geometry.
The test pipeline keeps the default uint8 load (no to_float32), so the image
recipe and calibration digest stay shared with the dense detectors.
"""
from functools import lru_cache
from .openmmlab import build as build_official, get_recipe
from .mmdet_dense_common import RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, geometry
MODELS = ("solo.r50", "solov2.r50")
GRIDS = (40, 36, 24, 16, 12)
STRIDES = (8, 8, 16, 32, 32)
TEST_CFG = {"nms_pre": 500, "score_thr": .1, "mask_thr": .5, "filter_thr": .05,
"kernel": "gaussian", "sigma": 2.0, "max_per_img": 100}
def recipe(model):
"""Same image recipe as the dense detectors once normalization is verified."""
if model not in MODELS:
raise ValueError(model)
from .mmdet_dense_common import recipe as image_recipe
meta = image_recipe("retinanet.r50.fpn")
dp = get_recipe(model)["preprocess"]["official_data_preprocessor"]
if (dp["mean"] != meta["mean"] or dp["std"] != meta["std"]
or not dp["bgr_to_rgb"] or dp["pad_size_divisor"] != 32):
raise ValueError("Official SOLO image normalization changed")
return meta
def preprocess_variant(model, image, input_spec=INPUT_SPEC):
from .mmdet_dense_common import preprocess_variant as image_preprocess
recipe(model) # verify exact equality before reusing the image-only helper
return image_preprocess("retinanet.r50.fpn", image, input_spec)
def output_names(model):
if model not in MODELS:
raise ValueError(model)
first = "mask" if model == "solo.r50" else "kernel"
names = [f"{first}_g{g}" for g in GRIDS] + [f"cls_g{g}" for g in GRIDS]
if model == "solov2.r50":
names.append("mask_feats")
return names
DEBUG_OUTPUT_NAMES = [f"fpn{i}" for i in range(5)]
def build_debug_variant(model, weights, input_spec):
"""Backbone+FPN taps: the continuous segment gated at .9999 cosine.
The SOLO heads stack seven conv+GroupNorm+ReLU blocks over generated
coordinate grids; fp reassociation across frameworks flips near-zero
pre-activations and shifts head logits by a few 1e-3 (measured, identical
at opset 11 and 16). Head equivalence is therefore judged on decoded
predictions and the COCO metric, not on raw logits.
"""
network = build_variant(model, weights, input_spec)
import torch
class NeckTaps(torch.nn.Module):
def __init__(self):
super().__init__()
self.upstream = network.upstream
def forward(self, x):
return tuple(self.upstream.extract_feat(x))
return NeckTaps().eval()
@lru_cache(maxsize=2)
def _decoder(model):
"""Official parameter-free SOLO decode context; no learned weights."""
from mmengine.config import ConfigDict
if model == "solo.r50":
from mmdet.models.dense_heads.solo_head import SOLOHead
head = SOLOHead.__new__(SOLOHead)
head.cls_out_channels = 80
else:
from mmdet.models.dense_heads.solov2_head import SOLOV2Head
head = SOLOV2Head.__new__(SOLOV2Head)
head.cls_out_channels = 80
head.kernel_out_channels = 256
head.dynamic_conv_size = 1
head.mask_stride = 4
head.num_grids = list(GRIDS)
head.num_levels = len(GRIDS)
head.strides = list(STRIDES)
head.test_cfg = ConfigDict(TEST_CFG)
return head
def decode_variant(model, outputs, original_size):
"""Official predict_by_feat on the raw dense head tensors."""
import numpy as np
import torch
if model not in MODELS:
raise ValueError(model)
count = 11 if model == "solov2.r50" else 10
if len(outputs) != count:
raise ValueError(f"{model}: wrong raw head count {len(outputs)}")
tensors = []
for index, value in enumerate(outputs):
array = np.asarray(value)
if array.ndim != 4 or not np.isfinite(array).all():
raise ValueError(f"{model}: invalid raw head {index}: {array.shape}")
tensors.append(torch.as_tensor(np.ascontiguousarray(array), dtype=torch.float32))
for level, grid in enumerate(GRIDS):
if tensors[5 + level].shape[1] != 80 or tensors[5 + level].shape[2:] != (grid, grid):
raise ValueError(f"{model}: unexpected class head {level}: {tensors[5 + level].shape}")
(nw, nh), scales, img_shape = geometry(original_size)
meta = dict(img_shape=img_shape, ori_shape=(int(original_size[1]), int(original_size[0])),
pad_shape=(640, 640), batch_input_shape=(640, 640), scale_factor=scales)
with torch.inference_mode():
if model == "solo.r50":
result = _decoder(model).predict_by_feat(tensors[:5], tensors[5:10], [meta])[0]
else:
result = _decoder(model).predict_by_feat(tensors[:5], tensors[5:10], tensors[10], [meta])[0]
return (result.bboxes.numpy(), result.scores.numpy(), result.labels.numpy(),
result.masks.numpy())
def g2_input_variant(model, t1, item, input_spec):
"""Diagnostic-input hook for harness/eval/run.py g2_inputs."""
return preprocess_variant(model, t1.image(item), input_spec)
def predict_variant(model, runner, t1, input_spec, pose_evidence=None):
"""Shared harness/eval/coco_extra.py hook: COCO segm RLE predictions."""
import numpy as np
from pycocotools import mask as mask_utils
from ..eval.common import run_batched
if pose_evidence is not None:
raise ValueError(f"{model}: iseg predict takes no pose evidence")
cat_map = t1.manifest["cat_map"]
predictions = []
for offset in range(0, len(t1.items), 16):
items = t1.items[offset:offset + 16]
images = [t1.image(it) for it in items]
xs = [preprocess_variant(model, image, input_spec) for image in images]
outputs = run_batched(runner, xs)
for it, image, outs in zip(items, images, outputs):
_, scores, labels, masks = decode_variant(model, outs, image.size)
for mask, score, label in zip(masks, scores, labels):
rle = mask_utils.encode(np.asfortranarray(mask.astype(np.uint8)))
rle["counts"] = rle["counts"].decode("ascii")
predictions.append({"image_id": it["id"],
"category_id": cat_map[str(int(label))],
"score": float(score), "segmentation": rle})
return predictions
def build_variant(model, weights, input_spec):
"""Construct the official SOLO model and strictly load its checkpoint."""
import torch
if model not in MODELS or input_spec != INPUT_SPEC:
raise ValueError(f"{model}: expected {INPUT_SPEC}")
raw = build_official(model, weights, input_spec)
network, cfg = raw.upstream, raw.cfg
meta = recipe(model)
dp = cfg.model.data_preprocessor
kind, head_kind = (("SOLO", "SOLOHead") if model == "solo.r50"
else ("SOLOv2", "SOLOV2Head"))
head = network.mask_head
if (type(network).__name__ != kind or type(head).__name__ != head_kind
or network.with_bbox or head.num_classes != 80
or list(head.strides) != list(STRIDES) or list(head.num_grids) != list(GRIDS)
or dict(head.test_cfg) != TEST_CFG
or list(dp.mean) != meta["mean"] or list(dp.std) != meta["std"]
or not dp.bgr_to_rgb or dp.pad_size_divisor != 32):
raise ValueError("Pinned official SOLO contract changed")
if model == "solov2.r50" and (head.kernel_out_channels != 256
or head.dynamic_conv_size != 1 or head.mask_stride != 4):
raise ValueError("Pinned official SOLOv2 mask geometry changed")
# Verify every learned parameter against the checkpoint bytes.
state = torch.load(str(weights), map_location="cpu")["state_dict"]
for name, parameter in network.named_parameters():
torch.testing.assert_close(parameter, state[name], rtol=0, atol=0)
del state
class RawHeads(torch.nn.Module):
def __init__(self):
super().__init__()
self.upstream = network
self.upstream.cfg = cfg
self.cfg = cfg
self.checkpoint_sha256 = raw.checkpoint_sha256
def forward(self, x):
outs = self.upstream.extract_feat(x)
results = self.upstream.mask_head.forward(outs)
flat = []
for group in results:
if isinstance(group, (list, tuple)):
flat.extend(group)
else:
flat.append(group)
return tuple(flat)
return RawHeads().eval()

View File

@ -0,0 +1,83 @@
"""Reuse pinned MMSeg models with an explicit static-image benchmark recipe."""
from pathlib import Path
from .openmmlab import build as official_build, get_recipe
RUNTIME_PYTHON = "/home/ming01.wang/outputs/full-coverage-20260917/openmmlab/venv/bin/python"
EXPORTER_VERSION = "mmseg-static-raw-logits-v1"
def lower_static_pools(network):
"""Export exact adaptive-pooling bins for the fixed deployment shape."""
import torch
class StaticPool(torch.nn.AdaptiveAvgPool2d):
def forward(self, x):
if not torch.onnx.is_in_onnx_export():
return super().forward(x)
oh, ow = (self.output_size, self.output_size) if isinstance(self.output_size, int) else self.output_size
h, w = map(int, x.shape[-2:])
if h % oh == 0 and w % ow == 0:
return super().forward(x)
rows = []
for i in range(oh):
top, bottom = i * h // oh, ((i + 1) * h + oh - 1) // oh
columns = []
for j in range(ow):
left, right = j * w // ow, ((j + 1) * w + ow - 1) // ow
columns.append(x[..., top:bottom, left:right].mean((-2, -1), keepdim=True))
rows.append(torch.cat(columns, dim=-1))
return torch.cat(rows, dim=-2)
for name, child in list(network.named_children()):
if isinstance(child, torch.nn.AdaptiveAvgPool2d):
setattr(network, name, StaticPool(child.output_size))
else:
lower_static_pools(child)
def metadata(model: str):
recipe = get_recipe(model)
if recipe["task"] != "seg" or recipe["source"]["checkout"] != "mmsegmentation":
raise ValueError(f"Not an MMSeg semantic segmentation recipe: {model}")
data = recipe["preprocess"]["official_data_preprocessor"]
_, _, h, w = map(int, recipe["input_spec"].split("x"))
checkpoint = dict(recipe["checkpoint"], classes={"ade20k": 150, "cityscapes": 19}[recipe["dataset"]])
spec = {"source": "custom", "dataset": recipe["dataset"], "input_spec": recipe["input_spec"]}
preprocess = {"recipe": "mmseg-static-resize-v1", "input_spec": recipe["input_spec"],
"dataset": recipe["dataset"], "color": "RGB" if data.get("bgr_to_rgb") else "BGR",
"mean": data["mean"], "std": data["std"], "scale": 1.0,
"layout": "NCHW", "dtype": "float32", "resize": [h, w],
"interpolation": "opencv_inter_linear", "logits_resize": "bilinear",
"align_corners": False, "logits_intermediate_size": [h, w],
"protocol": "bench static resize; whole-image logits; original label coordinates"}
return checkpoint, spec, preprocess
def preprocess_image(recipe: dict, image, input_spec: str):
import cv2
import numpy as np
if input_spec != recipe["input_spec"]:
raise ValueError(f"MMSeg input differs from verified recipe: {input_spec}")
h, w = recipe["resize"]
x = np.asarray(image.convert("RGB"))
if x.shape[:2] != (h, w):
x = cv2.resize(x, (w, h), interpolation=cv2.INTER_LINEAR)
if recipe["color"] == "BGR":
x = x[:, :, ::-1]
x = (x.astype(np.float32) - np.asarray(recipe["mean"], np.float32)) / np.asarray(recipe["std"], np.float32)
return np.ascontiguousarray(x.transpose(2, 0, 1)[None])
def build_model(model: str, weights: Path, input_spec: str):
network = official_build(model, weights, input_spec)
head = network.upstream.decode_head
checkpoint, _, recipe = metadata(model)
if head.align_corners != recipe["align_corners"] or head.num_classes != checkpoint["classes"]:
raise ValueError("Official head differs from the frozen label/resize recipe")
if network.upstream.test_cfg.mode != "whole":
raise ValueError("This adapter requires official whole-image inference")
if model.startswith(("pspnet.", "upernet.")):
lower_static_pools(network.upstream.decode_head)
return network

View File

@ -0,0 +1,17 @@
"""Official ADE20K OCRNet-HRNet18 with both learned cascade heads."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, RUNTIME_PYTHON, EXPORTER_VERSION
from .openmmlab import build as official_build
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("ocrnet.hrnet.w18")
preprocess = partial(preprocess_image, PREPROCESS)
def build(weights, input_spec):
network = official_build("ocrnet.hrnet.w18", weights, input_spec)
heads = network.upstream.decode_head
if (len(heads) != 2 or any(head.align_corners is not False or
head.num_classes != CHECKPOINT["classes"] for head in heads)
or network.upstream.test_cfg.mode != "whole"):
raise ValueError("Official OCRNet cascade differs from its frozen geometry")
return network

View File

@ -0,0 +1,17 @@
"""Official ADE20K OCRNet-HRNet48 with both learned cascade heads."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, RUNTIME_PYTHON, EXPORTER_VERSION
from .openmmlab import build as official_build
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("ocrnet.hrnet.w48")
preprocess = partial(preprocess_image, PREPROCESS)
def build(weights, input_spec):
network = official_build("ocrnet.hrnet.w48", weights, input_spec)
heads = network.upstream.decode_head
if (len(heads) != 2 or any(head.align_corners is not False or
head.num_classes != CHECKPOINT["classes"] for head in heads)
or network.upstream.test_cfg.mode != "whole"):
raise ValueError("Official OCRNet cascade differs from its frozen geometry")
return network

View File

@ -0,0 +1,184 @@
"""Pinned official OpenMMLab construction; catalog resolution is not admission.
The existing harness may call get_recipe(model) without importing PyTorch or
OpenMMLab. build() runs only in an explicitly provisioned isolated environment.
It never downloads, installs, substitutes architectures, or changes the registry.
The returned module consumes ALREADY PREPROCESSED float NCHW tensors and returns
upstream raw head tensors. Official geometry/decoding and board evaluation remain
separate gates; a successful build/export does not satisfy them.
"""
from __future__ import annotations
import hashlib
import importlib
import importlib.util
from functools import lru_cache
from copy import deepcopy
import yaml
from pathlib import Path
import subprocess
_RECIPES = Path(__file__).with_name("openmmlab_recipes.yaml")
_PACKAGES = {"mmdetection": "mmdet", "mmsegmentation": "mmseg", "mmpose": "mmpose"}
@lru_cache(maxsize=1)
def _catalog() -> dict:
return yaml.safe_load(_RECIPES.read_text())["recipes"]
def get_recipe(model: str) -> dict:
"""Return a detached recipe from the once-loaded catalog."""
return deepcopy(_catalog()[model])
def _sha256(path: Path) -> str:
h = hashlib.sha256()
with path.open("rb") as stream:
for chunk in iter(lambda: stream.read(1 << 20), b""):
h.update(chunk)
return h.hexdigest()
def _source(recipe: dict, source_root: Path | None = None) -> tuple[Path, str]:
source = recipe["source"]
checkout = source["checkout"]
if checkout not in _PACKAGES:
raise RuntimeError(f"{checkout} requires its separately recorded legacy environment/adapter")
package = _PACKAGES[checkout]
# Use the actual imported official package, or the caller's pinned checkout.
spec = importlib.util.find_spec(package)
if spec is None or spec.origin is None:
raise RuntimeError(f"Missing {package}; provision the isolated recipe environment first")
installed_root = Path(spec.origin).resolve().parent.parent
root = Path(source_root).resolve() if source_root else installed_root
if installed_root != root:
raise RuntimeError(f"Imported {package} comes from {installed_root}, not requested {root}")
revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=root, text=True).strip()
dirty = subprocess.check_output(
["git", "status", "--porcelain", "--untracked-files=no"], cwd=root, text=True
).strip()
if revision != source["revision"] or dirty:
raise RuntimeError(f"Unverified {package} checkout: expected clean {source['revision']}; got {revision}")
config = root / source["config"]
if _sha256(config) != source["config_sha256"]:
raise RuntimeError(f"Config hash mismatch: {config}")
return config, package
def verify_checkpoint(recipe: dict, weights: Path) -> str:
"""Require the full digest recorded from an official-download receipt.
Filename tokens are provenance only: some official MMPose files retain
an old token after checkpoint conversion and do not match their SHA256.
"""
weights = Path(weights)
if not weights.is_file():
raise FileNotFoundError(f"Official checkpoint required: {recipe['checkpoint']['url']}")
sha = _sha256(weights)
info = recipe["checkpoint"]
expected = info.get("sha256")
if not expected or len(expected) != 64:
raise RuntimeError("Recipe lacks a verifiable checkpoint digest; record the official download hash first")
if sha != expected:
raise ValueError(f"Checkpoint identity mismatch: expected {expected}, got {sha}")
return sha
def load_config(model: str, *, source_root: Path | None = None):
"""Parse official config and scoped bases from the verified source checkout.
Editable installs can lack the wheel-only .mim resources. Keep MMEngine's
parser and scope handling; resolve same-package files in the pinned repo.
"""
config_path, package = _source(get_recipe(model), source_root)
from mmengine.config import Config
from unittest.mock import patch
config_root = Path(importlib.util.find_spec(package).origin).resolve().parents[1] / "configs"
original = Config._get_cfg_path
def resolve(base, filename):
if base.startswith(f"{package}::"):
candidate = (config_root / base.split("::", 1)[1]).resolve()
if candidate.is_relative_to(config_root) and candidate.is_file():
return str(candidate), package
return original(base, filename)
with patch.object(Config, "_get_cfg_path", staticmethod(resolve)):
return Config.fromfile(str(config_path))
def build(model: str, weights: Path, input_spec: str, *, source_root: Path | None = None):
"""Construct the exact official model and strictly load its complete weights.
No tensor/key remapping or non-strict fallback is provided. Upstream load
hooks handle their own historical format migrations; MMEngine's documented
DDP ``module.`` prefix removal is retained. Evidence records strict=True.
Input normalization, affine geometry and output decoding are NOT implicit.
"""
recipe = get_recipe(model)
if recipe["resolution_status"] != "resolved_catalog":
raise ValueError(f"Unresolved recipe {model}: {recipe['blockers']}")
sha = verify_checkpoint(recipe, weights)
shape = tuple(int(v) for v in input_spec.split("x"))
if len(shape) != 4 or shape[0:2] != (1, 3) or min(shape) <= 0:
raise ValueError(f"Expected static batch-one NCHW RGB/BGR input, got {input_spec}")
expected = recipe.get("input_spec")
if not expected:
raise RuntimeError(f"{model}: static probe input contract has not been resolved")
if input_spec != expected:
raise ValueError(f"{model}: recipe input {expected}, got {input_spec}; no silent shape substitution")
import torch
from mmengine.runner import load_checkpoint
from mmengine.registry import init_default_scope
cfg = load_config(model, source_root=source_root)
package = _PACKAGES[recipe["source"]["checkout"]]
utils = importlib.import_module(f"{package}.utils")
utils.register_all_modules(init_default_scope=False)
init_default_scope(package)
registry = importlib.import_module(f"{package}.registry")
network = registry.MODELS.build(cfg.model)
# MMEngine now marks normalization buffers non-persistent; older official
# checkpoints saved them. Preserve every key and strictly compare its value
# with this recipe before restoring persistence. Never drop learned keys.
state = torch.load(str(weights), map_location="cpu")["state_dict"]
preprocessor = network.data_preprocessor
for name in ("mean", "std"):
key = f"data_preprocessor.{name}"
# MMPose's official migration hook removes legacy non-learned norm keys.
if package != "mmpose" and key in state and name in preprocessor._non_persistent_buffers_set:
expected_buffer = getattr(preprocessor, name)
if not torch.equal(state[key], expected_buffer):
raise ValueError(f"Checkpoint {key} disagrees with the official config")
preprocessor.register_buffer(name, expected_buffer, persistent=True)
del state
# Do not call init_weights(): the complete trained checkpoint is mandatory.
load_checkpoint(network, str(weights), map_location="cpu", strict=True)
network.cpu().eval()
class RawTensorModel(torch.nn.Module):
def __init__(self):
super().__init__()
self.upstream = network
self.openmmlab_recipe = recipe
self.checkpoint_sha256 = sha
self.cfg = cfg
def forward(self, x):
# Two-stage detector/transformer heads require image metadata even
# in tensor mode. It describes this already-preprocessed tensor,
# NOT a claim about original-image resize or coordinate recovery.
samples = None
if package == "mmdet":
from mmdet.structures import DetDataSample
samples = [DetDataSample(metainfo=dict(
img_shape=shape[-2:], ori_shape=shape[-2:],
pad_shape=shape[-2:], batch_input_shape=shape[-2:],
scale_factor=(1., 1.)
))]
return self.upstream(x, samples, mode="tensor")
return RawTensorModel().eval()

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,17 @@
"""Official Cityscapes PIDNet-L preserving its align_corners=True geometry."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, RUNTIME_PYTHON, EXPORTER_VERSION
from .openmmlab import build as official_build
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("pidnet.l")
PREPROCESS["align_corners"] = True
preprocess = partial(preprocess_image, PREPROCESS)
def build(weights, input_spec):
network = official_build("pidnet.l", weights, input_spec)
head = network.upstream.decode_head
if (head.align_corners is not True or head.num_classes != CHECKPOINT["classes"]
or network.upstream.test_cfg.mode != "whole"):
raise ValueError("Official PIDNet-L head differs from its frozen geometry")
return network

View File

@ -0,0 +1,17 @@
"""Official Cityscapes PIDNet-M preserving its align_corners=True geometry."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, RUNTIME_PYTHON, EXPORTER_VERSION
from .openmmlab import build as official_build
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("pidnet.m")
PREPROCESS["align_corners"] = True
preprocess = partial(preprocess_image, PREPROCESS)
def build(weights, input_spec):
network = official_build("pidnet.m", weights, input_spec)
head = network.upstream.decode_head
if (head.align_corners is not True or head.num_classes != CHECKPOINT["classes"]
or network.upstream.test_cfg.mode != "whole"):
raise ValueError("Official PIDNet-M head differs from its frozen geometry")
return network

View File

@ -0,0 +1,17 @@
"""Official Cityscapes PIDNet-S preserving its align_corners=True geometry."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, RUNTIME_PYTHON, EXPORTER_VERSION
from .openmmlab import build as official_build
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("pidnet.s")
PREPROCESS["align_corners"] = True
preprocess = partial(preprocess_image, PREPROCESS)
def build(weights, input_spec):
network = official_build("pidnet.s", weights, input_spec)
head = network.upstream.decode_head
if (head.align_corners is not True or head.num_classes != CHECKPOINT["classes"]
or network.upstream.test_cfg.mode != "whole"):
raise ValueError("Official PIDNet-S head differs from its frozen geometry")
return network

View File

@ -0,0 +1,7 @@
"""Official ADE20K pspnet.resnet101; admission requires real validation."""
from functools import partial
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("pspnet.resnet101")
preprocess = partial(preprocess_image, PREPROCESS)
build = partial(build_model, "pspnet.resnet101")

Some files were not shown because too many files have changed in this diff Show More