pre-grind snapshot 2026-09-19: batches 1-7 adapters/probes/recipes + eval additive changes
This commit is contained in:
parent
5b1284ec8c
commit
d309109f2b
91
CONTINUE.md
Normal file
91
CONTINUE.md
Normal file
@ -0,0 +1,91 @@
|
||||
# 剩余任务切到 100 张
|
||||
|
||||
旧批次保存已经发生的执行,新批次只接剩下的任务。流程是:**生成独立评测包 → 预览清单 → 停止领取并收尾 → 更新清单并应用 → 单条试跑 → 恢复并发**。
|
||||
|
||||
普通流水线和 Agent 仍使用原来的 `bench run`、`bench collect`。不全量 `expand`,不使用 `collect --coverage all`。注册表默认值和校准集不改;500 张复核按需另做,这里不加入自动升级机制。
|
||||
|
||||
## 1. 先准备,旧批次继续运行
|
||||
|
||||
在有原始数据池的机器上生成固定子集。ImageNet 随机选类,每类取一张;COCO 固定随机抽图。
|
||||
|
||||
```bash
|
||||
python scripts/make_t1.py --dataset imagenet1k --dataset coco2017 --n 100 --seed 20260914
|
||||
```
|
||||
|
||||
新包在 `data/t1/<dataset>/n100-s20260914/`,原包保留。同一份清单用于 FP32 和板端 INT8;新缓存按内容哈希和评测器版本分开存放。
|
||||
|
||||
下例的 `OLD_RUN`、`NEW_RUN` 和 `JOB_ID` 都替换成清单里的实际值。下面的 `plan` 只读数据库和文件,不领取任务、不创建 run、不停止进程:
|
||||
|
||||
```bash
|
||||
bench continue plan --run OLD_RUN \
|
||||
--t1 data/t1/imagenet1k/n100-s20260914/manifest.json \
|
||||
--t1 data/t1/coco2017/n100-s20260914/manifest.json > preview.json
|
||||
```
|
||||
|
||||
只提供剩余任务涉及的数据集;缺少某个数据集的替换包会报错。默认生成新的 run ID,也可用 `--to NEW_RUN` 指定。清单含新旧任务、原采样序号、模式、策略、评测包身份及阻塞原因。服务器仍在执行时出现 `blockers` 是正常的,最终清单在收尾后重新生成。
|
||||
|
||||
> `bench reset-stale` 必须跑在 `plan` **之前**,不能夹在 `plan` 和 `apply` 中间:plan 把源 run 的 stages 全行打进 `source_sha256`,reset-stale 改任何一行 stage 状态都会让已生成的清单在 apply 时被拒(fail-closed,报 `source changed`),只能重新 plan。
|
||||
|
||||
`counts` 分四项:
|
||||
|
||||
| 项目 | 含义 |
|
||||
|---|---|
|
||||
| `oracle` | 从未开始的普通任务 |
|
||||
| `agent_queued` | 已经排队、从未开始的 Agent 采样 |
|
||||
| `agent_due_repairs` | 新清单固定为 0,不再按 Oracle 失败补任务 |
|
||||
| `conditional_repairs` | 新清单固定为 0;旧清单保留,但不再激活 |
|
||||
|
||||
成功和失败都消耗原采样次数。计划 4 条、完成 1 条就只接余下 3 条,并保留它们的原 `kk`,不从零重新编号。首轮优先顺序及 `collect --first-pass` 也计入旧批次的已启动、已完成记录。
|
||||
|
||||
## 2. 工具验证完成后,停止领取并完成交接
|
||||
|
||||
```bash
|
||||
bench drain --run OLD_RUN --all
|
||||
bench status --run OLD_RUN
|
||||
```
|
||||
|
||||
`--all` 同时设置普通流水线和 Agent 的停止领取标记。等已经领取的整个任务及核验结束、两个调度进程退出;不等全部待办跑完。运行中的旧进程必须已支持 drain,不能靠同步新代码改变它的内存逻辑。
|
||||
|
||||
收尾后,用第 1 步的 `plan` 命令重新输出 `handoff.json`,核对清单,然后:
|
||||
|
||||
```bash
|
||||
bench continue apply handoff.json
|
||||
bench continue check --run NEW_RUN
|
||||
```
|
||||
|
||||
应用时再次校验评测包、源任务状态及源注册表指纹。过期清单、还在运行、曾执行过又重置成待办的记录都会阻止交接。成功后源待办记为 `TRANSFERRED`,源 run 不再允许启动或重新铺队列;完成的结果、日志和轨迹保留原处。重复应用同一清单只核对结果,不重复创建任务。
|
||||
|
||||
工具只处理一次从原始批次到接续批次的交接;不提供连续多级迁移框架。数据库事务提交前的文件准备失败,可用同一清单重试。目标目录保存 `continuation.json` 作为交接回执。
|
||||
|
||||
## 3. 先试一条,再恢复并发
|
||||
|
||||
如果选中的任务还没跑普通流水线,先执行它;如果只剩 Agent,跳过这一条命令:
|
||||
|
||||
```bash
|
||||
bench run --run NEW_RUN --job JOB_ID --parallel 1
|
||||
bench collect --run NEW_RUN --job JOB_ID --parallel 1 --limit 1
|
||||
bench export --run NEW_RUN
|
||||
bench continue check --run NEW_RUN
|
||||
```
|
||||
|
||||
确认 FP32 和 INT8 的包哈希一致、实际评测 `n=100`、结果及 Agent 训练记录包含 `eval_set`,再按原资源配置恢复:
|
||||
|
||||
```bash
|
||||
bench run --run NEW_RUN
|
||||
bench collect --run NEW_RUN --watch
|
||||
```
|
||||
|
||||
两条恢复命令分别在现有终端/进程管理方式下启动;不新增守护服务。`check` 比较具体任务及采样序号,不只比较总数。试跑后待办数会减少,但“待办 + 执行中 + 已结束”仍与交接数量对应。历史 QUEUED repair 保留编号但不再领取,须与可执行的 e2e 待办分开统计。
|
||||
|
||||
已有任务快照中的 `oracle_ref` 保留历史身份;新执行与判定不读取旧 Oracle。Agent 按需调用已有输入准备阶段,缓存按评测集区分,不重跑整条普通流水线。新交接只保留实际存在的采样行,不补 Oracle 失败驱动的 repair。
|
||||
|
||||
100 张保留现有 `pass-screen` 筛查口径和原精度门槛。奖励 v0.6 只使用本次实测,不再与 Oracle 比较。训练记录保留原 run、job、采样序号和评测集身份,训练端按 `eval_set` 区分可比较的组。
|
||||
|
||||
## 本地验证范围
|
||||
|
||||
单元与集成测试使用临时 SQLite、合成评测包和模拟执行器,验证抽样、任务守恒、交接恢复、调度、缓存命令及导出。它们不能替代真实 100 张图片上的 FP32/INT8 试跑,也不能替代 Linux 板端和 Agent 沙箱验证。
|
||||
|
||||
## 已知未修问题(从 PLAN.md §7 迁入)
|
||||
|
||||
- **events.jsonl 在爆**:convnext.small #0 才 7 轮就 6.0 MB,其中 5.53 MB 是 1548 条 `message_update`(流式 token 增量)。落盘时过滤掉,或跑完压成 `.raw.zst` 只留结构事件。124 条 rollout 就是几个 GB,而 audit / 导出每次都要全文扫。
|
||||
- **heartbeat 健康时零输出**,无法区分"都健康"和"它什么都没看到"。健康时也打一行。
|
||||
38
CONTRACT.md
38
CONTRACT.md
@ -4,9 +4,19 @@
|
||||
|
||||
2026-09-16 按当前目标收敛:默认运行 CSV 中 ready 模型的 S100P(`nash-m`)量化、编译、上板与精度评测,并为这些模型采集真实 Agent 轨迹。`collect --coverage all` 为本 run 的 oracle 模型补齐 e2e 计划:只去掉 matrix 的 priority 限制,保留任务类型、march、backend、policy 和 k;默认 `matrix` 仍使用原选择范围。已结束的流程和 Agent 轨迹都必须导出为数据,成功、精度未达标、预算耗尽分别标注,导出不能阻塞上板。
|
||||
|
||||
2026-09-16 全 338 项覆盖补充:保持原 CSV 标签和历史 run。新批次可用 `expand --model` 显式选择模型,
|
||||
前置检查依据实际权重入口、构建/导出代码、评测器、校准、数据与附加门;新增 Ultralytics 任务还必须有真实
|
||||
checkpoint/ONNX/指标对照证据,缺项即拒绝入队。具名批次只放宽 priority/task/status 选择,保留原 march、
|
||||
backend、policy、k 和精度门。任务快照记录 admission 来源,不能把 admission 或 probe 等同于编译/上板通过。
|
||||
`collect --first-pass` 每个 job 只领取首条,重复采样仍保持 QUEUED,不改 kk、不补造轨迹。
|
||||
`coverage --all-runs` 总表包含全部 CSV 行,独立列出前置阻塞、实际阶段、原始响应中的 teacher 身份和训练导出;
|
||||
传统算法仍保留条目并说明 PTQ 不适用。FastSAM 的类别语义未补齐前不能入队。
|
||||
Pose 须有真实导出的关键点对应证据;板端同时判定 OKS AP 与平均原图坐标偏移 ≤2px。
|
||||
对应由同图人框 IoU≥0.5 一对一匹配确定,统计全部17点,保留未匹配记录;无匹配或缺证据为 UNVERIFIED。
|
||||
|
||||
## 0. 三条原则
|
||||
|
||||
1. **harness 是 oracle。** 每个 job 先由确定性流水线跑出 ground truth(`oracle/result.json`)。agent 的表现永远相对 oracle 衡量。
|
||||
1. **Agent 独立完成任务。** GLM-5.3 可直接领取 e2e,自行调用输入准备、量化、编译和上板工具;`bench run` 可选地预热 01–03 缓存。harness 分配资源、核验本次 HBM 的实测精度并保存轨迹。完整 Oracle 仅用显式 `bench run --oracle` 作历史对照或排查,不参与 Agent 执行、成功判定、奖励或 SFT 筛选。
|
||||
2. **agent 只有一个可写区。** `rollouts/<kk>/workspace/`。`job.json`、`harness/eval/`、`data/`、`cache/` 全部只读。agent 说什么不算数,verdict 才算数。
|
||||
3. **Append-only。** `run_id` 冻结;换工具链、改配置、修 bug 都开新 run。`manifest.json` 记两套 venv 的 `pip freeze` sha256,job 启动前比对,不一致拒跑。状态只在 `state/jobs.sqlite`,文件系统只存内容。
|
||||
|
||||
@ -58,7 +68,8 @@ bench/
|
||||
|
||||
## 3. 状态机
|
||||
|
||||
**job(由 oracle 决定)**:`PENDING → ORACLE_RUNNING → PASS | FAIL | UNVERIFIED`
|
||||
**job 输入准备**:`PENDING → ORACLE_RUNNING → PREPARED | FAIL | UNVERIFIED`(沿用现有运行态名称)。
|
||||
`PREPARED` 只表示 01–03 成功,写 `oracle/preparation.json`,不产生 Oracle verdict,也不代表适配或精度通过。显式 `--oracle` 可从 PENDING/PREPARED 跑完整 01–08,写 `oracle/result.json`,终态仍为 PASS/FAIL/UNVERIFIED。默认准备命令不覆盖已结束任务。
|
||||
**rollout(由实际核验结果决定)**:`QUEUED → RUNNING → VERIFYING → PASS | FAIL | BLOCKED | UNVERIFIED`
|
||||
|
||||
Agent 执行是否结束/超预算独立记录在 `verdict.execution_terminal`、`timeout_reason`,不再用 `TIMEOUT` 覆盖已经核验的结果。`completion` 分别记录编译、HBM、真实上板、精度评测是否完成及精度是否达标。预算用尽但必选门已通过仍为 PASS;没做到的阶段为未完成,不凭预算耗尽推断模型失败。旧 TIMEOUT 可用 `bench verify` 按原 job 门槛重核,无需重编。
|
||||
@ -66,7 +77,7 @@ Agent 执行是否结束/超预算独立记录在 `verdict.execution_terminal`
|
||||
- PASS = 必选门全部 pass。
|
||||
- 门无法判定(数据缺、板未接)→ 该门 `unverified`,job `UNVERIFIED`。
|
||||
- `na`(本轮设计上不评)≠ `unverified`(本该评没评成)。na 不阻塞 PASS。
|
||||
- BLOCKED 判定:oracle 同阶段同类 error_class 失败 → 正确;oracle 通过 → 误判。
|
||||
- BLOCKED 记录 Agent 的阻塞报告和本次实测。失败日志不能证明任务不可解,`blocked_correct` 不作推断;不进入成功 SFT 或奖励学习候选。
|
||||
|
||||
## 4. 四道门
|
||||
|
||||
@ -97,9 +108,10 @@ Agent 执行是否结束/超预算独立记录在 `verdict.execution_terminal`
|
||||
- 板卡登记使用固定且唯一的 `slot`,对应 `board-<slot>.lock`;现有板保留 slot 0 以兼容在途旧进程。新增板使用新槽位,删除/禁用板卡后不复用槽位;重复接入同一地址不能增加容量。只把实际独立、运行库兼容的物理板加入池。新增/禁用板卡是调度资源变更,不重写冻结的 job 或已生成 HBM。新调度进程等待时重读板卡登记;旧进程仍需自然收尾后更换。
|
||||
- 新 run 的任务指纹不包含板卡设备列表,但仍包含 march 映射与其他 registry 内容。旧 run 扩板前把原始 `boards.yaml` 保存为 `board_registry.initial.yaml`;续跑时用该快照验证旧整体指纹,证明除板卡设备列表外的输入未变,不能跳过模型、数据、门槛、工具链校验。
|
||||
- **没有固定超时。** STALLED = 日志 `stall_s`(900 s)不增长 且 进程树 CPU < 5%。硬上限 24 h 只是保险。
|
||||
- 板端 RPC 的原有单次 3600 秒上限由独立 `board_guard` 在板上执行;SSH stdin 租约每 5 秒刷新,断连或 30 秒失联回收本次进程组。守护进程不加载模型,避免原生运行库阻塞清理。板端互斥锁由守护进程和推理子进程共同持有,未清空不能交给新推理。租约存活日志与样本/迭代进度分开记录,不能把心跳当作推理进展。主机阶段锁只代表调度占用,不能代替板端进程核验。
|
||||
- 重阶段(compile / board / eval)由 harness 执行。agent 走 `bench stage run` / `bench stage wait`,不在 bash 里跑 hb_compile。
|
||||
- rollout 预算 = turns + tokens;墙钟 12 h 仅防挂死。
|
||||
- Agent 动态领取任务:先覆盖尚无完整轨迹的不同模型,同等覆盖下优先 oracle PASS 和预计耗时较短的模型,再做重复采样;同一 job 同时最多一条 Agent 轨迹。每次有空位都重新读取队列,`collect --watch` 跟随尚未结束的 oracle,避免新完成的模型一直排在旧批次后面。
|
||||
- Agent 动态领取任务:先覆盖尚无完整轨迹的不同模型,同等覆盖下优先预计耗时较短的模型,再做重复采样;同一 job 同时最多一条 Agent 轨迹。每次有空位都重新读取队列,`collect --watch` 等待输入准备和在途任务,避免新完成的模型一直排在旧批次后面。
|
||||
- 同一 run 只运行一个新 collector;任务领取原子化。SIGTERM 或 `state/drain-collect-<run>` 停止领取,已开始的任务自然收尾。导出在独立线程刷新,不占 Agent 名额。
|
||||
|
||||
## 7. 重试与轨迹边界
|
||||
@ -109,7 +121,7 @@ Agent 执行是否结束/超预算独立记录在 `verdict.execution_terminal`
|
||||
| oracle 阶段 | harness | 瞬态 error_class | 不是轨迹 |
|
||||
| rollout 内 | agent | 自己改配置重提 | 同一条轨迹 |
|
||||
| 新 rollout | harness | matrix k | 全新轨迹,永不串联 |
|
||||
| repair | harness | oracle FAIL | 全新轨迹,输入是 oracle 现场 |
|
||||
| 历史 repair | 不再生成或领取 | 保留既有编号和结果 | 有 Oracle 现场的修复案例,仅保留分析用途 |
|
||||
|
||||
显式扩大已有 run 的 Agent 覆盖时,仅追加缺少的计划和范围说明 annotation;编号接在该 job 的最大 kk 之后,重复执行不重复添加。冻结的 job、已执行轨迹、oracle 结果及评测门槛不变。确定性 oracle 的 PASS 不代表该模型已有 Agent 训练轨迹。
|
||||
|
||||
@ -126,19 +138,17 @@ bash 里 `cd/pushd/popd` 一律拒(唯一例外 `cd <自己的 workspace>`,
|
||||
## 8. error_class
|
||||
|
||||
`unsupported_op shape_infer export_fail calib_nan verifier_mismatch`(确定性,不重试)
|
||||
`oom board_offline stalled`(瞬态,重试 ≤ 2 / ≤ 2 / 1)
|
||||
`oom board_offline board_busy stalled`(瞬态,前三者重试 ≤ 2,stalled 重试 ≤ 1)
|
||||
`unknown`(进人工分类队列)
|
||||
|
||||
## 9. reward(v0.4,离线算,配方版本化)
|
||||
## 9. reward(v0.6,离线算,配方版本化)
|
||||
|
||||
```
|
||||
r = 1.0[G1] + 1.0[G2] + 2.0[G3]
|
||||
r += 1.0·clip(accuracy_improvement/|tol|, −1, 1)
|
||||
r += 0.5·clip((lat_oracle − lat_agent)/lat_oracle, −1, 1)
|
||||
r −= 0.5·turns/max_turns + 0.5·tokens/max_tokens
|
||||
r −= 2.0[claim ≠ verdict]
|
||||
BLOCKED: +1.0 若 oracle 同类失败 else −2.0 ;TIMEOUT: −1.0
|
||||
G3 = na 时其 2.0 权重按比例分给 G1/G2,精度相对项取 0
|
||||
TIMEOUT: −1.0;BLOCKED 无特殊奖惩,仍保存本次真实阶段得分
|
||||
G3 = na 时其 2.0 权重按比例分给 G1/G2
|
||||
板卡不可用(G3 unverified 且最后一次 07-board/08-eval 为 board_offline):G2/G3 权重取 0,G1 保持 1.0 不加分,
|
||||
免 TIMEOUT 罚分,reward 输出带 unverifiable="board_offline"(v0.4,2026-09-16)
|
||||
```
|
||||
@ -146,7 +156,7 @@ G3 = na 时其 2.0 权重按比例分给 G1/G2,精度相对项取 0
|
||||
板卡不可用的轨迹在导出器里仍按 UNVERIFIED 排除在 RL/SFT 候选之外;回收它们的办法是 `bench board-backfill --run R --rollout JOB:KK`
|
||||
用其现成 HBM 补跑 07/08 再 verify,而不是改分。
|
||||
|
||||
`abs_pp` 的 accuracy_improvement = Δ_agent − Δ_oracle;`rel_pct` 的 Δ 表示劣化,故取 Δ_oracle − Δ_agent。不同 kind 不互比。
|
||||
不再计算相对 Oracle 的精度或延时奖励。G3 仍比较同一评测集上的 FP32 与本次 INT8,沿用任务冻结的门槛。
|
||||
TIMEOUT 扣分根据执行终止原因独立相加,已完成的 G1/G2/G3 成果照常得分,不再整条清零。
|
||||
|
||||
## 10. 两套 venv
|
||||
@ -164,13 +174,15 @@ harness 按 backend 选解释器。**禁止把 backend 判断交给 agent。**
|
||||
|
||||
数据默认按 `purpose=all` 保留已结束的成功、失败及部分轨迹;SFT 是其中的用途筛选,不能代表全部数据量。Agent 的 `teacher.jsonl` / `student.jsonl` 只含真实对话、工具调用和观测,缺失内容不得编造。`pipeline.jsonl` 保留确定性流程的实际输入、阶段动作与测量,明确标记其来源,不能计成模型生成的 Agent 轨迹。每个 run 使用同一导出目录,显式刷新,按稳定 ID 去重;导出失败报错并可补导,不回滚已完成任务。
|
||||
|
||||
`bench.verdict/3` 不读取 Oracle;旧判定在导出时只读重核,源 verdict、数据库、消息和产物不改。SFT 候选要求本次实测 PASS、声明一致、完整对话和工具定义,并保留污染注释与版本过滤。Oracle FAIL 不再触发待复核;历史 repair 保留原模式,仅供分析。导出是候选数据,仍需训练端分组、去重和独立评测。
|
||||
|
||||
## 12. 协作规则(2026-09-15 起,两个会话同时改一棵树踩过一次)
|
||||
|
||||
- **只有一个写者能改一层。** 分层:`harness/`(oracle 流水线、阶段、门、调度)· `agent/`(pi 配置、扩展、提示词)· `export/`(清洗、脱敏、reward)· `registry/`(清单与阈值,改了就是新 run)· `scripts/`。谁在改哪层写进下面的表;别的会话对那层只读。
|
||||
- **改动只发生在本地副本,机器上不直接改代码。** 流程:改 → `git commit`(`scripts/sync.sh` 会自动提交)→ `scripts/sync.sh` 推送。sync 带守卫:远端自上次推送后被动过就拒绝,先合并再推。
|
||||
- `registry/models.csv` 是在机器上由 `excel_to_models.py` 生成的,方向反过来:`scripts/sync.sh --pull`。
|
||||
- 机器上的 `runs/ state/ data/ cache/ datasets/` 是数据,不同步、不手改;改判走 `bench rejudge`,补测走 `bench board-backfill`。
|
||||
- 正在跑的调度进程(`bench run` / `bench collect`)加载的是启动时的代码;改了 harness 只对新进程生效,子进程工具(`harness.tools.*`、`harness.eval.run`)立刻生效——改评测口径必须同时升 `EVALUATOR_VERSION`。
|
||||
- 正在跑的调度进程(`bench run` / `bench collect`)加载的是启动时的代码;改了 harness 只对新进程生效,子进程工具(`harness.tools.*`、`harness.eval.run`)立刻生效——改评测口径必须同时升 `EVALUATOR_VERSION`。**换代必须 `bench drain` 等它跑完再起新进程,只 sync 文件不算换代**(踩过:一个调度进程跑了 13 小时的旧代码)。
|
||||
|
||||
| 层 | 当前写者 |
|
||||
|---|---|
|
||||
|
||||
29
HANDBOOK.md
29
HANDBOOK.md
@ -5,20 +5,13 @@
|
||||
|
||||
## 一、它在做什么(一段话)
|
||||
|
||||
把 338 个视觉模型逐个量化编译到 BPU(nash-e/m/p),每个"模型×芯片×配置"叫一个 **job**。每个 job 先由 **harness** 自己跑一遍标准流程(叫 **oracle**,产出"标准答案"和四道门的判定),然后让 **agent**(一个调大模型的 pi 进程)在看不到答案的情况下**自己做一遍**,harness 独立核验它做得对不对。agent 做一遍的过程记录下来就是一条 **rollout(轨迹)**,带着 harness 给的可核验分数,这就是训练数据。
|
||||
把 338 个视觉模型逐个量化编译到 BPU,每个“模型×芯片×配置”叫一个 **job**。GLM-5.3 Agent 自行准备输入、量化、编译和上板;harness 管理资源并用本次真实产物核验。一次执行的对话、工具调用和观测是一条 **rollout(轨迹)**。
|
||||
|
||||
```
|
||||
registry/models.csv ──▶ bench new-run/expand ──▶ jobs(PENDING)
|
||||
│
|
||||
bench run(后台调度) ──▶ oracle:01 fetch → 02 baseline → 03 export → 04 calibrate → 05 compile → 06 hbm → 07 board → 08 eval
|
||||
│ 四道门 G1 编译 / G2 余弦 / G3 任务指标 / G4 上板延时
|
||||
▼
|
||||
bench collect(后台采集) ──▶ 每个 job k 条 rollout:pi 进程在 <job>/rollouts/<kk>/workspace 里作答
|
||||
│ → verify_rollout 独立核验 → verdict.json(reward 的唯一来源)
|
||||
▼
|
||||
scripts/audit_rollouts.py ──▶ 哪些轨迹干净、能进 RL/SFT
|
||||
export/to_agentrl.py ──▶ 训练格式
|
||||
```
|
||||
流程:**领取任务 → Agent 执行 → 本次实测判定 → 导出并筛选训练候选**。
|
||||
|
||||
`bench run` 可以预热权重、FP32 基线和 ONNX 缓存。完整 Oracle 仅供显式对照,不是任务前置,也不参与成功判定、奖励或 SFT 筛选。FP32 仍是 INT8 精度的比较基线,两者使用同一评测集。
|
||||
|
||||
旧 Oracle、旧 repair 和已有结果保留。历史 repair 已获得失败现场,适合分析修复过程,不能当成独立完成任务的示范;尚未开始的旧 repair 保留编号但不再领取。
|
||||
|
||||
## 二、你能控制的东西(改这些文件,其他都是派生的)
|
||||
|
||||
@ -48,7 +41,7 @@ python3 scripts/trace.py <rollout_dir> 一条轨迹逐轮回放:a
|
||||
bench report --run <run> 静态 HTML 报告(reports/<run>.html)
|
||||
```
|
||||
|
||||
启动/续跑:`scripts/start_run.sh --note "..."`(oracle)、`bench collect --run <run> [--job …] [--limit N]`(采轨迹)。
|
||||
启动/续跑:`scripts/start_run.sh --note "..."`(准备与采集)、`bench collect --run <run> [--job …] [--limit N]`(采轨迹)。
|
||||
换代码/改配置要重启调度器时:`bench drain --run <run>`(跑完手里的不再认领新 job)→ 等它退出 → `scripts/start_run.sh --resume`。不要 kill 正在编译的进程。
|
||||
注意常驻进程里的代码是启动那一刻的:改了 harness/ 或 config.yaml,正在跑的 `bench run` / `bench collect` 不会变,新起的才会。
|
||||
两条自检:`node agent/extensions/guard-core.test.js`(边界判定单测)、`scripts/guard_smoke.sh`(起真 pi 进程试图越界,应全部被拒)。
|
||||
@ -67,10 +60,10 @@ bench report --run <run> 静态 HTML 报告(reports/<run>.html
|
||||
| `old_version(vX<Y)` | 用旧一版的提示词/边界跑的 | 导出默认过滤,留档不删 |
|
||||
| `claim_absent` / `claim_mismatch` | 没交 claim / 自述与核验不一致 | 可以当负样本(reward 已扣),SFT 不收 |
|
||||
| `harness/toolchain_read×N` | 去读评判代码或工具链源码 | 不算污染,但说明提示词没把它劝住,预算会被这个烧掉 |
|
||||
| 干净 + `RL ✓` | 版本合格、没越界、没被误杀 | 进 GRPO 组 |
|
||||
| `SFT ✓` | RL ✓ 且 verdict=PASS 且 claim 一致 | 进 SFT |
|
||||
| 干净 + `RL ✓` | 初步审计合格 | 仍需核验结果与训练分组,不代表已可直接训练 |
|
||||
| `SFT ✓` | 独立执行、实测 PASS、声明一致、对话和工具记录完整 | 成功 SFT 候选 |
|
||||
|
||||
想亲手看一条:`trace.py` 回放里盯这几个词——`oracle`、`rollouts/0X`(不是自己的编号)、`harness/`、`site-packages`、`path-guard:`(被拒的那一下)、`claim`(最后有没有交)、`bench stage run`(真正干活的提交)。一条健康的 e2e 轨迹长这样:读 job.json → 写 compile.yaml → `stage run calibrate/compile` → `stage wait` → 看日志改配置(≤2 轮) → `stage run hbm` → `stage run eval` → `claim`。前 3 步之外还在 `ls`/`cat` 各种目录的,就是在浪费预算。
|
||||
想亲手看一条:`trace.py` 回放里盯这几个词——`oracle`、`rollouts/0X`(不是自己的编号)、`harness/`、`site-packages`、`path-guard:`(被拒的那一下)、`claim`(最后有没有交)、`bench stage run`(真正干活的提交)。常见 e2e 流程是:读 job.json → 按需准备输入 → 写配置 → 编译并检查日志 → 检查 HBM → 上板评测 → claim。具体步骤由 Agent 根据观测决定。
|
||||
|
||||
## 五、失败的轨迹要删吗
|
||||
|
||||
@ -105,4 +98,4 @@ v5 已把阶段执行移到 jail 外部 worker,state/其他 job 与评判代
|
||||
命令客户端仅写 handles,pi 自身输出写 workspace 后由外部驱动收存。旧常驻 collect 不会热加载新驱动,后续采集需新进程。
|
||||
导出默认最低版本5并读取污染/误杀 annotations,旧9份数据集已新增 VOID 标记。
|
||||
`bench heartbeat --run <run>` 正常无输出退出0,异常退出1;`bench watch` 非TTY仅输出一次。
|
||||
完整实测、审计前后对比和未覆盖项见 [REPORT-codex.md](REPORT-codex.md)。本节取代上面 v4 的“当前”状态描述。
|
||||
完整实测、审计前后对比和未覆盖项见 [REPORT-codex.md](docs/archive/REPORT-codex.md)。本节取代上面 v4 的“当前”状态描述。
|
||||
|
||||
53
README.md
53
README.md
@ -1,8 +1,41 @@
|
||||
# bench/ — 视觉模型量化与上板评测
|
||||
|
||||
设计:`../BPU量化轨迹工厂-设计方案.html`(v0.4)。契约:`CONTRACT.md`。本 README 只讲怎么把它放到机器上跑起来。
|
||||
## 338 模型分批覆盖
|
||||
|
||||
主流程按 CSV 中已就绪的模型运行:FP32 基线 → 导出 → 校准、编译 → S100P(`nash-m`)上板 → T1 精度评测。新任务默认要求任务指标相对 FP32 劣化不超过 5%,改善也通过;G2 余弦用于诊断,性能只记录。Agent / RLVR 采集按需单独启动。
|
||||
`bench coverage --all-runs --csv reports/model-coverage.csv` 维护原 CSV 全部 338 项;
|
||||
相邻 JSON 保存所有 run/job/kk、逐阶段记录、真实轨迹路径、teacher 身份和训练导出关联。
|
||||
静态前置齐备、FP32/ONNX 对照通过、量化编译、真实上板、任务精度通过分别记录,不能互相替代。
|
||||
SIFT/ORB/AKAZE 仍占总表三行,按传统算法记不适用 PTQ,不虚构 HBM 或 Agent 轨迹。
|
||||
|
||||
新接入模型先核查具体权重、构建/导出、输入、校准、标注和评测门;不修改原 CSV 标签或旧 run。
|
||||
Ultralytics COCO 分割可用 `scripts/probe_coco_adapters.py --model yolov8n.seg` 做真实权重、
|
||||
ONNX 数值与 COCO 指标对照。Pose 另存原图坐标关键点对应及偏移,必须通过真实导出对照才能入队;
|
||||
板端 G3 同时要求 OKS AP 和平均偏移 ≤2px,缺几何证据则 UNVERIFIED。probe 不是板端验收。
|
||||
FastSAM 的类别语义仍属阻塞。EfficientViT 构建器文件存在不代表依赖、预处理和导出已验证。
|
||||
|
||||
通过前置核查的具名批次沿用现有命令:
|
||||
|
||||
```bash
|
||||
bench new-run --note '338 coverage: validated model batch'
|
||||
bench expand --run RUN --modes oracle,e2e --marches nash-m --model yolov8n.seg
|
||||
bench run --run RUN --parallel 2
|
||||
bench collect --run RUN --modes e2e --first-pass --watch --parallel 2
|
||||
```
|
||||
|
||||
显式 `--model` 放宽该批的 priority/task/status 选择,但保留 march/backend/qcfg、teacher、e2e k=4、
|
||||
冻结精度门槛及原采样编号;不再生成 Oracle repair,缺真实 probe 或其他前置时整批拒绝。
|
||||
`--first-pass` 只消费每个 job 的首条,原重复采样仍保持 QUEUED;等可执行模型完成首轮再用普通 collect 接续。
|
||||
当前旧 collector 若需切换,只设置其 `state/drain-collect-RUN`,等在途任务与导出自然完成再启新进程。
|
||||
不能直接 kill 或用新代码同步冒充旧进程已加载首轮策略。
|
||||
|
||||
创建 `state/coverage-master.enabled` 后,新进程每次训练导出会刷新总表;旧进程可沿用
|
||||
`bench export --run RUN --follow` 接入相同刷新。总表失败会显式报错,不回滚已完成任务和训练数据。
|
||||
|
||||
设计:[BPU 量化轨迹工厂设计方案](docs/BPU量化轨迹工厂-设计方案.html)(v0.4)。契约:`CONTRACT.md`。方法论、闭环说明和技术分享讲稿见 [项目文档](docs/README.md)。本 README 只讲怎么把它放到机器上跑起来。
|
||||
|
||||
主流程:GLM-5.3 领取任务,按需准备输入、校准、编译、上板和评测 → harness 按本次实测独立判定 → 导出原始轨迹并筛选 SFT 候选。`bench run` 可选地预热权重、FP32 和 ONNX 缓存;`--oracle` 只作显式对照。Oracle 不参与 Agent 判定、奖励或 SFT 筛选,也不再生成 repair。历史文件和排队编号保留,旧 repair 不再领取。新任务仍要求相对 FP32 劣化不超过 5%,改善也通过;历史任务沿用冻结门槛。
|
||||
|
||||
已有批次只把剩余任务切到 100 张测试集,见 [剩余任务交接](CONTINUE.md)。使用交接清单保留原采样序号,不全量展开或重新补采。
|
||||
|
||||
## 0. 上机
|
||||
|
||||
@ -33,20 +66,20 @@ for d in state data cache; do mkdir -p /home/$USER/bench-$d; [ -L $d ] || { rm -
|
||||
| 4 | `~/toolchains/oe-3.7.0/venv/bin/python scripts/ingest_pool.py --dataset imagenet1k --dataset coco2017` | `data/pool/…/manifest.json`,只读 |
|
||||
| 5 | `… scripts/make_t1.py --dataset imagenet1k --dataset coco2017` | `data/t1/<ds>/20260914.tar` |
|
||||
| 6 | `… scripts/make_calib.py --task cls --input-spec 1x3x224x224` / `--task det --input-spec 1x3x640x640` | `data/calib/…` |
|
||||
| 7 | `bench preflight --modes oracle --oracle-only` | 0 errors 才往下;不要求 agent 环境 |
|
||||
| 8 | `bench new-run --note "S100P"` → `bench expand --modes oracle --marches nash-m` | 每个就绪模型的 `runs/r…/<job_id>/job.json`,`state/jobs.sqlite` |
|
||||
| 7 | `bench preflight --modes oracle,e2e` | 0 errors 才往下;包含 Agent 环境检查 |
|
||||
| 8 | `bench new-run --note "S100P"` → `bench expand --modes oracle,e2e --marches nash-m` | 每个就绪模型的 `runs/r…/<job_id>/job.json`,`state/jobs.sqlite` |
|
||||
| 9 | `bench run --dry-run` | 状态机空转一遍,不碰工具链 |
|
||||
|
||||
步骤 4–6 用 PTQ venv 的 python(要 pyarrow / pillow / numpy)。ImageNet 首选 HF parquet 14 片;tar 是备用源,入池时会 `tar -tzf | wc -l` 校验。
|
||||
|
||||
## 2. P2 切片(真跑)
|
||||
|
||||
环境和数据准备好后,`scripts/start_run.sh --note "S100P" --parallel 2` 会检查环境、展开 nash-m oracle 并后台运行;`--resume` 续跑已有 run。其他板型需先在 `registry/matrix.yaml` 的 `marches` 中加入对应值,再用 `bench expand --marches ...` 选择。
|
||||
环境和数据准备好后,`scripts/start_run.sh --note "S100P" --parallel 2` 会检查环境、展开 nash-m 输入准备和 Agent 计划,后台准备 01–03 并按首轮优先采集;`--resume` 续跑已有 run。采样次数、GLM-5.3 身份和精度门槛不变。其他板型需先在 `registry/matrix.yaml` 的 `marches` 中加入对应值,再用 `bench expand --marches ...` 选择。
|
||||
|
||||
```bash
|
||||
bench run --job cls--resnet18--nash-m--ptq--int8 --parallel 1
|
||||
bench run --oracle --job cls--resnet18--nash-m--ptq--int8 --parallel 1 # 仅需要完整对照时
|
||||
bench show cls--resnet18--nash-m--ptq--int8 # 阶段时间线、stderr 尾巴
|
||||
bench run --job det--yolov8n--nash-m--ptq--int8 --parallel 1
|
||||
bench run --oracle --job det--yolov8n--nash-m--ptq--int8 --parallel 1
|
||||
bench status; bench status --fail; bench report # reports/<run_id>.html
|
||||
```
|
||||
|
||||
@ -112,15 +145,15 @@ march / 输出前缀 / 工作目录被 harness 钉死;`--config` 必须在 `wo
|
||||
- **机器是共享的**(2026-09-14 实测):244 核 / 1.4 TB / **8 张 32 GB 卡**(不是 4 张),4/5/7 被别人跑满、6 占 29 GB;12 个登录用户。`config.yaml pools.gpu.devices: [0,1,2,3]` 钉死只用空闲卡,槽位 → `CUDA_VISIBLE_DEVICES`。
|
||||
- 资源池是跨进程的(`state/locks/<pool>-<i>.lock` flock):oracle 调度进程、agent 的 `bench stage run`、多个 `bench run` 共用同一组槽位;board 池 1 槽 = 严格串行。model-level 阶段按模型加锁,三个 march 并行时只有一个真跑 01–03。job 认领是原子的(UPDATE … WHERE state='PENDING'),两个调度进程不会重跑同一个 job。
|
||||
- 新 run 的 manifest 记 `harness_code_sha256`;harness 代码变了只告警,venv / registry / EVALUATOR_VERSION 变了拒跑(实测拦下过一次:改 config.yaml 的 venv 列表后 `bench run` 拒绝,开了新 run)。
|
||||
- 沙箱板是 S100P(`nash-m`);历史 job 未记录 board 时,阶段按 march 查询当前板卡配置。没有匹配板卡时无法完成板端 G3,结果保持 UNVERIFIED。`board_perf.py` 开测前探测板上有无别人的推理进程 / BPU ratio,最多等 10 分钟,仍忙则照测但标 `contended: true`,reward 的延时相对项忽略这类样本。
|
||||
- 沙箱板是 S100P(`nash-m`);历史 job 未记录 board 时,阶段按 march 查询当前板卡配置。没有匹配板卡时无法完成板端 G3,结果保持 UNVERIFIED。`board_perf.py` 开测前探测板上推理进程 / BPU ratio,最多等 10 分钟,仍忙则以 `board_busy` 拒绝测量。07/08 使用独立板端守护进程,持有板端互斥锁,并在 SSH 断连、30 秒未收到租约或原有单次 RPC 3600 秒上限到达时回收推理进程组。详见 [板端进程生命周期](docs/board-process-lifecycle.md)。
|
||||
|
||||
- `job.json` 与 mode 无关:一个 job_id 一份,`rollout_plans` 列出 e2e/repair 计划;oracle 与所有 rollout 共用。
|
||||
- sqlite 默认 TRUNCATE 日志(WAL 在 FUSE 上会 disk I/O error);`BENCH_STATE_DIR` 可把 state 放本地盘。
|
||||
- 校准集来源:本机 ImageNet / COCO 只有 val,`make_calib.py` 退到 val-holdout(排除 T1)并在 manifest 记 `source`;出 T2 交付数前必须换 train。
|
||||
- 07-board 失败重试后仍失败 → G4 unverified、继续 08-eval;08-eval 失败 → job UNVERIFIED 带 error_class;01–06 失败 → FAIL。
|
||||
- claim 缺席(agent 没调 `claim` 工具)→ claim.json 记 `ABSENT`,reward −0.25;claim=PASS 但没有可核验产物 → verdict FAIL 并计 claim≠verdict。
|
||||
- BLOCKED 正确性:claim.error_class 与 oracle 同类(`unknown` 不算同类)且阶段一致(claim 未填阶段则不比阶段)。
|
||||
- reward 的精度相对项按 G3 的 `kind` 解释方向:`rel_pct` 是劣化百分比,越小越好;`abs_pp` 是改善百分点,越大越好。
|
||||
- BLOCKED 仅记录阻塞报告,不根据 Oracle 推断正确性,不进入成功 SFT 或奖励学习候选。
|
||||
- reward v0.6 只看本次实测门、成本和声明,不与 Oracle 比较。导出时只读重核旧判定,保留源结果;历史 repair 单列为分析案例。
|
||||
|
||||
## 6. 目录里什么进 git
|
||||
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# bench agent · 系统提示(追加在 pi 默认系统提示之后)
|
||||
|
||||
你在一个受控的量化上板环境里工作。规则不是建议,违反会被工具层拒绝或被判负。
|
||||
你负责从权重到导出、FP32 基线、量化、编译、上板和精度评测的完整任务。harness 提供执行工具、资源池与独立核验,不会预先替你跑通模型。有效缓存可以复用。规则不是建议,违反会被工具层拒绝或被判负。
|
||||
|
||||
## 你能写的地方
|
||||
只有当前目录(`workspace/`)。其他一切只读。越界写入会被 path-guard 拒绝并记录。
|
||||
@ -49,9 +49,12 @@ compiler_parameters:
|
||||
诊断信号在哪:`bench stage wait` 结束后给你 stdout/stderr 路径;hb_compile 日志末尾有逐节点 `Calibrated Cosine / Quantized Cosine` 表和模型输出余弦。余弦偏低时可检查敏感节点、尝试 `node_config` 提到 int16;最终任务精度以 G3 实测与 job 中的阈值为准。
|
||||
|
||||
## 重阶段不在 bash 里跑
|
||||
编译、上板、评测由 harness 执行。你提交、你等待、你看日志:
|
||||
导出、基线、编译、上板和评测均由你提交工具,harness 执行并保存日志:
|
||||
|
||||
```
|
||||
bench stage run fetch # 获取或复用正式权重
|
||||
bench stage run export # 导出 ONNX;有效缓存直接复用
|
||||
bench stage run baseline # 当前 T1 的 FP32 参考;同子集缓存直接复用
|
||||
bench stage run calibrate # 生成编译配置(也可先看 job.json 自己写一份)
|
||||
bench stage run compile --config compile.yaml # cwd 就是 workspace/,相对路径按 workspace 解析(写 workspace/compile.yaml 也认)
|
||||
bench stage wait <handle> --poll 60
|
||||
@ -73,10 +76,10 @@ claim(terminal="PASS"|"FAIL"|"BLOCKED", stage=..., error_class=..., summary="…
|
||||
```
|
||||
**预算规则**:剩 5 轮时你会收到一条预算提醒——收到后无论进展如何,先调用 `claim` 汇报当前状态(不确定就报 FAIL 并写清到了哪一步),再决定要不要继续。达到轮数上限会立即结束进程,没有额外 grace 轮;请提前 claim。token 预算耗尽后只允许 claim,仍受轮数硬上限约束。没有 claim 就结束的轨迹被记为 claim 缺席,另扣分。
|
||||
|
||||
你的 claim 会与 harness 独立核验的 verdict 比对。claim=PASS 而 verdict=FAIL 是最重的扣分项,比老实报 FAIL 差得多。不确定就报 FAIL 并说明。BLOCKED 只在你确认是工具链/算子层面无法完成时使用,并给出 error_class(unsupported_op / shape_infer / calib_nan / verifier_mismatch 之一)。
|
||||
你的 claim 会与 harness 独立核验的 verdict 比对。claim=PASS 而 verdict=FAIL 是最重的扣分项,比老实报 FAIL 差得多。不确定就报 FAIL 并说明。BLOCKED 用于有实际错误证据的工具、适配器或材料缺口;给出具体阶段、错误和所需材料。无法归入工具已有错误类别时用 unknown,不能把缺权重或缺适配器说成 unsupported_op。
|
||||
|
||||
## 不要做的事
|
||||
不下载数据集或权重(都在只读区);不改 `job.json`;不猜指标——指标只来自 `bench stage run eval` 的输出;不在 bash 里 `sleep` 等编译,用 `bench stage wait`。
|
||||
不直接写共享数据或权重缓存,权重获取用 `bench stage run fetch`;不改 `job.json`;不猜指标——指标只来自阶段工具的实测输出;不在 bash 里 `sleep` 等编译,用 `bench stage wait`。权重、适配器或数据缺失时如实报告具体缺口;未尝试编译就不能声称工具链不支持该模型。
|
||||
|
||||
## 工具链参考
|
||||
`.horizon/skills/` 下的 skill 是这条工具链的文档。`oe-doc <query>` 可以查 OE 文档。两套 venv 的选择由 harness 按 job.json 的 backend 决定,你不需要也不能选。
|
||||
|
||||
@ -1 +1 @@
|
||||
6
|
||||
7
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
# 任务:把 {model}({task})量化并编译到 {march}({backend} / {qcfg}),通过全部必选门
|
||||
# 任务:从模型权重出发,将 {model}({task})部署到 {march}({backend} / {qcfg}),通过全部必选门
|
||||
|
||||
job:`{job_id}` run:`{run_id}`
|
||||
job.json:`{job_json}`(只读) 数据:`{data_dir}`(只读) 你的可写区:当前目录
|
||||
@ -11,13 +11,16 @@ job.json:`{job_json}`(只读) 数据:`{data_dir}`(只读) 你的
|
||||
|
||||
必选项以 job.json 的 `gates.required` 为准。
|
||||
|
||||
## 你手上有什么
|
||||
- FP32 ONNX:`{onnx}`(03-export 已产出)
|
||||
- FP32 基线:`{baseline_dir}/metrics.json`
|
||||
- 校准集:`{calib_dir}`({calib_n} 个 npy,与 val 零交集)
|
||||
## 输入和可复用缓存
|
||||
- 权重来源、模型身份和输入约定:见 job.json 的 model_spec
|
||||
- ONNX 缓存位置:`{onnx}`(可能尚未生成)
|
||||
- FP32 基线缓存位置:`{baseline_dir}/metrics.json`(可能尚未生成或属于旧子集)
|
||||
- 校准集位置:`{calib_dir}`(目标 {calib_n} 个 npy,与 val 零交集)
|
||||
- T1 评测子集:`{t1_tar}`({t1_n} 张)
|
||||
|
||||
## 流程
|
||||
先读 job.json;系统提示里有 compile.yaml 全部可改的旋钮,不必再读 harness 或工具链源码。默认配置(calibration_type: default,全 int8)先跑一次拿到基线,看日志里的逐节点余弦再决定调什么。提交 `bench stage run compile --config <你的>.yaml`(cwd 就是 workspace/,不要再加 workspace/ 前缀),`bench stage wait`。失败就读日志改配置;同一方法最多重试 1 次。编译过了依次跑 `bench stage run hbm`、`bench stage run board`、`bench stage run eval`,每步等待完成;性能记录失败仍继续评测。最后调用 `claim`,缺少板端精度结果时不能报 PASS。
|
||||
先读 job.json,然后依次提交 `bench stage run fetch`、`bench stage run export`、`bench stage run baseline`,每步用 `bench stage wait` 等待并检查结果。这些阶段由你发起并记入本条轨迹;有效缓存会直接返回 SKIPPED,不重复计算。没有人预先替你判断能否量化成功。
|
||||
|
||||
输入准备完成后,由你选择量化方案并提交 `bench stage run compile --config <你的>.yaml`(cwd 就是 workspace/),再 `bench stage wait`。系统提示里列有 compile.yaml 可改参数。失败就读日志改方案;同一方法最多重试 1 次。编译通过后依次提交 `hbm`、`board`、`eval`,每步等待完成;性能记录失败仍继续评测。遇到缺权重、缺适配器或缺评测工具,明确记录实际错误和缺失材料,不把它写成量化失败。最后调用 `claim`,缺少板端精度结果时不能报 PASS。
|
||||
|
||||
预算:{max_turns} 轮、{max_tokens} tokens。等待编译不消耗预算。剩 5 轮会收到提醒——那时先 claim,别等到耗尽。
|
||||
|
||||
@ -9,7 +9,7 @@ import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
NAMES = {n.split('-', 1)[1]: n for n in ('04-calibrate', '05-compile', '06-hbm', '07-board', '08-eval')}
|
||||
NAMES = {n.split('-', 1)[1]: n for n in ('01-fetch', '02-baseline', '03-export', '04-calibrate', '05-compile', '06-hbm', '07-board', '08-eval')}
|
||||
|
||||
def rollout_dir():
|
||||
return Path(os.environ['BENCH_ROLLOUT_DIR'])
|
||||
|
||||
@ -39,6 +39,12 @@ watchdog:
|
||||
cpu_idle_pct: 5
|
||||
poll_s: 30
|
||||
hard_ceiling_s: 86400 # 24h 纯保险
|
||||
io_stall_s: 21600 # 树内持续有 D 线程(卡在 I/O)这么久 → 也判定异常。
|
||||
# 别往下调:线程级 D 检测上线前这条是死代码(d_count 恒 0),
|
||||
# 上线后它第一次生效,收紧会把"链接期正在写盘"换个理由再杀一次。
|
||||
no_progress_ceiling_s: 28800 # 日志与 CPU 都不动这么久 → 兜底,免疫 D 抖动。
|
||||
# 必须 > io_stall_s,否则它抢先命中、盖掉更准确的 STALLED 原因。
|
||||
# 四条阈值单调:stall_s(long 3600) < io_stall_s < 本条 < hard_ceiling_s
|
||||
soft_alert: p95x3 # 超过历史 p95 × 3 只告警,不杀
|
||||
|
||||
perf:
|
||||
@ -77,7 +83,7 @@ data:
|
||||
endpoints:
|
||||
teacher:
|
||||
provider: openai-compatible
|
||||
base_url: https://ai-api.d-robotics.cc/v1
|
||||
base_url: https://open.bigmodel.cn/api/coding/paas/v4
|
||||
model: glm-5.3
|
||||
api_key_env: TEACHER_API_KEY
|
||||
student:
|
||||
|
||||
475
docs/BPU量化轨迹工厂-技术分享讲稿-20260916.md
Normal file
475
docs/BPU量化轨迹工厂-技术分享讲稿-20260916.md
Normal file
@ -0,0 +1,475 @@
|
||||
# 嵌入式大模型后训练 · 第一阶段汇报讲稿
|
||||
|
||||
面向:技术团队 / 同事(领导在场) | 完整版约 30 分钟,速用版 3 分钟 | 数字截面:2026-09-16 21:03 运行态快照 + 09-17 设计变更
|
||||
|
||||
> 用法:每节先是"讲什么"(可以直接念),后面是"提示"(只给你自己看:数字来源、措辞纪律、被追问时怎么接)。
|
||||
>
|
||||
> 全篇措辞原则:讲进展用具体数字,讲未完成用"在建 / 已排期",讲经验用"这一类工程的共性规律"。不评价任何既往决策,不做"早期做错了"的归因——第一阶段的目标本来就是打通,现在是按计划进入第二阶段。
|
||||
|
||||
---
|
||||
|
||||
# 速用 · 三分钟口头版
|
||||
|
||||
> 临时汇报、例会同步、走廊被问到,用这一版。可以直接念,念完约 3 分钟。
|
||||
|
||||
**背景。** 前期我们调研了嵌入式方向上能做的题目,可选的其实不少。我们的判断是,第一个锚点要选一条能自己闭环的——判据能被程序判定、验证比求解便宜、失败原因能分类,这三条同时成立,才撑得起大量采数据。BPU 模型量化上板这条线三条都满足,和楠总对齐之后,方向就定在这里了。
|
||||
|
||||
**起点。** 方向定下来,这个阶段的重点就是把上板这件事的数据积累起来。起点是三样现成的东西:HBDK 4 工具链、S100P 板子,以及我们整理的 338 个公开视觉模型清单——其中已完成接入的 63 个(分类 48、检测 15),其余的在逐个做导出适配和评测器补齐。
|
||||
|
||||
**做了什么。** 搭了两套东西。
|
||||
|
||||
一套是**数据收集系统**:agent 在隔离环境里拿到一个模型,自己决定怎么量化、怎么编译,harness 负责执行工具、分配板卡、独立核验它做出来的产物,全过程留痕。agent 这一层用的是 pi,teacher 模型是 GLM-5.3。
|
||||
|
||||
另一套是**板端评测环境**,这块的投入产出比超出预期:板子上只做一件事——跑张量。读图、预处理、后处理、算指标全部留在主机侧,标注和评测代码永远不上板。好处是加一类新任务板端零改动,而且 FP32 基线和量化模型吃同一份输入、走同一个评测器,比较才有意义。顺带一个意外收获:**板端推理比服务器模拟快 45 倍**——所以"必须上真板"不是成本,是提速。
|
||||
|
||||
**这里有一个重要的设计调整,上周做的。** 原本每个模型要先用确定性流水线跑一遍产出标准答案,再让 agent 做。实测下来,这一遍吃掉了三分之二的板时,而它一条轨迹也不产。所以我们把它改成了 **agent 主导**:默认只准备输入——权重、FP32 基线、ONNX 导出,这三步结果可缓存复用;输入齐备 agent 就直接上手。完整的确定性流水线保留成一个显式开关,只在做对照和排查问题的时候跑。**精度判据一点没放松**,因为参照物从来就是 FP32 基线,不是默认配置的量化结果——判的还是板端实测相对 FP32 劣化不超过 5%。
|
||||
|
||||
**现在的状态,三个数。**
|
||||
|
||||
第一,**板端终判打通了**。精度数字来自 S100P 真实推理,不是服务器模拟——这是这个阶段最关键的一步,在此之前所有精度都是模拟出来的,不能作为交付依据。目前已有 18 个模型完成板端实测并通过,这是第一批可以对外讲的上板基准。
|
||||
|
||||
第二,**agent 轨迹开始产出**。已结束 34 条,其中判定通过 22 条,覆盖 14 个不同模型。作为对比,上一个截面这个数是 3 条,而且全部集中在同一个最简单的模型上——那时候只能说明流程能走通,现在能说明 agent 确实在解决问题。最好的一条,板端 top-1 相对 FP32 只掉了 0.06%。
|
||||
|
||||
第三,**出现了第一条修复模式的成功轨迹**:确定性流水线做不出来的模型,agent 调出来了。明细还在核验,核完它就是我们第一条黄金样例。
|
||||
|
||||
**资源。** 现在是 4 块板加一台 8 卡机器。板卡是这条线的咽喉——严格串行,一块板同时只能跑一个任务,利用率已经到 93%,基本是满的。刚才那个设计调整把三分之二的板时让给了采集,等于产能翻了两倍多,但这是一次性红利,规模再往上走还是要靠扩容。一轮完整实验需要多少板时是能算出来的〔填入下一节算好的数〕。所以后续主要的诉求是板卡扩容,我这边也会去协调。
|
||||
|
||||
**下一步三件事。** 把 agent 单次实验从一个多小时压到 5 分钟以内,方案已经验证过;接学生模型,跑通在线训练闭环;把覆盖从 63 个模型往更多任务类扩,分割和姿态估计的适配已经在做了。
|
||||
|
||||
---
|
||||
|
||||
# 速用 · 数字卡片
|
||||
|
||||
> 汇报前存手机或打印一页。**打✱的项汇报当天早上要重新取一次**,它们会随采集推进变化。
|
||||
|
||||
| 会被问 | 直接答 |
|
||||
|---|---|
|
||||
| 模型清单多大 | 338 个公开视觉模型,10 个任务类;已完成接入 63 个(分类 48、检测 15),其余在补导出适配和评测器 |
|
||||
| 板端跑通了吗 ✱ | 通了。已有 18 个模型完成板端实测并通过 |
|
||||
| 采了多少轨迹 ✱ | 已结束 34 条(通过 22 / 失败 8 / 未验证 4),另有在跑和排队的 |
|
||||
| 覆盖多少模型 ✱ | 通过的 22 条覆盖 14 个不同模型,最简单的 resnet18 只占 4 条 |
|
||||
| 最好的一条结果 | regnet.y.800mf 板端 top-1 77.85 对 FP32 77.90,相对劣化 0.06% |
|
||||
| 精度判据是什么 | 板端实测的任务指标,相对 FP32 基线劣化 ≤ 5%,改善也算通过 |
|
||||
| 一次上板多久 | 性能测试 31 秒;完整精度评测(2000 张)约 8 分钟 |
|
||||
| 板子忙不忙 | 12.6 小时窗口内利用率 93% |
|
||||
| 板端比模拟快多少 | 45 倍(0.889 毫秒 vs 40 毫秒一张) |
|
||||
| 编译要多久 | 单次 35 到 118 分钟;其中量化只占 107 秒,BPU 编译占 78 分钟,比例 1:44 |
|
||||
| 测试集为什么 2000 张 | 配对比较下差值标准误约 0.5 个百分点;500 张时约 1 个点,和容差同量级会误判 |
|
||||
| 隔离安不安全 | 内核级隔离,红队 11 种路径逃逸全部挡住 |
|
||||
| 现在还跑标准流水线吗 | 默认不跑了。只准备输入(权重 / FP32 基线 / ONNX),agent 直接上手;完整流水线保留成显式对照开关 |
|
||||
|
||||
---
|
||||
|
||||
# 速用 · 资源诉求怎么提
|
||||
|
||||
> "能扩到更多会更好一些"这种说法领导没法决策。给算法、给数、给不扩的后果,三样都要有。
|
||||
|
||||
**算法**(当场能写在白板上):
|
||||
|
||||
```
|
||||
一轮实验的板时 = 题目数 × 每题采样条数 × 每条里的板端测量次数 × 单次测量耗时
|
||||
```
|
||||
|
||||
**现有实测单价**:单次完整精度评测约 8 分钟,性能测试 31 秒,板卡严格串行。
|
||||
|
||||
**代入现在的规模**(63 个模型 × 4 条采样,每条按 2 次板端评测算):约 500 次测量 × 8 分钟 ≈ 67 小时,**1 块板要 2.8 天,4 块板约 17 小时**。
|
||||
|
||||
**怎么讲**:现在 4 块板对应的节奏是一天一轮。上周那次设计调整已经把三分之二的板时从"跑标准答案"让给了"采轨迹",这是一次性红利,已经吃掉了。接下来采集规模按计划往上走——覆盖从 63 个模型扩到更多任务类、每题采样数从 4 提到 8——板时是线性涨的,届时一轮就变成三四天。这是训练迭代速度的直接上限,所以板卡扩容不是效率优化,是决定我们多久能验证一次假设。
|
||||
|
||||
**两个配套动作一起提**(显得诉求是算过的,不是单纯要东西):
|
||||
- 两阶段编译落地后,agent 单次实验从一个多小时压到 5 分钟以内,同样的板子能多跑几倍任务——先把现有资源的效率榨出来,再谈扩容。
|
||||
- 板卡要按物理身份登记、一块板一把锁。同一地址的多个容器不能算成额外容量,否则排队系统以为有多块板,两个任务同时推上去互相干扰,测出来的延时全废。
|
||||
|
||||
---
|
||||
|
||||
# 速用 · 资料清单
|
||||
|
||||
**A. 手边必备**:上面的数字卡片,一页纸。
|
||||
|
||||
**B. 投屏随时能打开**(按最可能被问的顺序排好标签页):
|
||||
|
||||
1. `BPU量化轨迹工厂-方法论.html` —— 被问"整条线怎么运转",打开 §0 流程图
|
||||
2. `BPU量化轨迹工厂-闭环方法论.html` —— 被问"怎么算走通了",打开 §5 分辨力四例表
|
||||
3. `CPU算子优化Agent-立项说明.html` —— 被问"下一个方向",打开 §5 缺口总表
|
||||
4. 本讲稿 —— 被问细节时按节号跳
|
||||
|
||||
> ⚠️ 前两份 HTML 是按"确定性流水线先出标准答案"那一版设计写的,和现在的 agent 主导不一致。投屏前要么先说一句"这是上一版设计,第 X 节讲的调整就是针对这里",要么抽空把两页的对应段落更新掉。**别让听众自己发现不一致。**
|
||||
|
||||
**C. 被追问才调的证据**:
|
||||
|
||||
| 问题 | 文件 |
|
||||
|---|---|
|
||||
| 板端利用率、故障统计、补测结果 | `outputs/plan-verify-20260916/REVIEW.md` |
|
||||
| 板卡连通性、已有产物清单 | `outputs/s100p-gap-20260916/` |
|
||||
| 当前运行态计数 | `outputs/glm53-endpoint-switch/runtime-after.json` |
|
||||
| 四道门口径、agent 主导的新原则 | `bench/CONTRACT.md` §0 和 §4 |
|
||||
| 338 项覆盖总表 | `bench coverage --all-runs --csv reports/model-coverage.csv` |
|
||||
|
||||
**D. 汇报前必须自查的四件事**:
|
||||
|
||||
1. **板卡登记状态。** 容量等于"注册表里启用的板数",不是物理板数。最近一次记录里启用的只有 1 块,另一块因 BPU 打不开被停用。物理上现在有 4 块,要确认是否都已登记并启用——否则说"4 块板"但调度容量是 1,会被追问。
|
||||
2. **修复模式那条轨迹的明细。** verdict、claim 一致性、审计洁净度三项,核完才能讲"黄金样例"。
|
||||
3. **重新取一次计数。** 数字卡片里打✱的会变,用当天早上的数。
|
||||
4. **repair 的触发条件。** 代码里 repair 仍然挂在"oracle 失败"上。默认不跑完整流水线之后,repair 默认也就不触发了——确认一下现在是靠显式 `--oracle` 补现场,还是已经改成吃 agent 自己的失败。这个点很可能被问到(见问答第 3 条)。
|
||||
|
||||
---
|
||||
|
||||
## 0. 开场(1 分钟)
|
||||
|
||||
**讲什么**
|
||||
|
||||
今天汇报三段:我们的主线是什么、为什么第一个锚点选在 BPU 部署这条链路上、这条链路现在跑出了什么。
|
||||
|
||||
先给结论:**主线是训练一个能干嵌入式工程活的大模型**。这件事成不成,不取决于我们用多大的底座模型,取决于一件更基础的事——我们能不能把"做得好不好"变成一个机器判得准的数字。判不出来就没有训练信号,再大的模型也只是在猜;判得不准,训练就是在拟合噪声。
|
||||
|
||||
所以第一阶段我们没有直接去训模型,而是先把这个"判得准的数字"造出来。现在它造出来了,并且已经在产出真实轨迹。
|
||||
|
||||
**提示**
|
||||
|
||||
- 开场不报通过率,第 6 节会专门讲为什么通过率不是指标。
|
||||
- "不取决于用多大的模型"是整场的主心骨,后面每节都回扣它。
|
||||
|
||||
---
|
||||
|
||||
## 1. 主线:我们要训的是什么模型(4 分钟)
|
||||
|
||||
**讲什么**
|
||||
|
||||
我们的定位是做行业的底座——供应商逻辑。放到模型上,就是训一个**嵌入式领域的工程 agent**:客户把模型交给我们要往板上部署,中间那一串工程活——量化配置怎么调、算子不支持怎么办、落到 CPU 上的算子怎么优化、端到端延时怎么压——现在靠工程师一个个手工做,我们要让模型接手。
|
||||
|
||||
这类模型和通用对话模型的训练方式不一样,差别就在监督信号:
|
||||
|
||||
| | 通用对话 / 创意类 | 我们这类嵌入式工程任务 |
|
||||
|---|---|---|
|
||||
| 对错谁判 | 没有客观判据,要训一个奖励模型当裁判 | 程序判定:返回码、精度掉点、板端延时 |
|
||||
| 主要风险 | 长期和"裁判被钻空子"缠斗 | 判据本身要造准,造准了就不会被钻 |
|
||||
| 数据来源 | 人工标注 / 偏好对 | 真实执行留下的轨迹,自带可核验结果 |
|
||||
|
||||
这是我们的结构性优势,不用白不用。但要兑现它,前提是先有一个**可机器判定的验证器**,而且要有分辨力——不只是能盖章通过,还要能把该拒的拒掉。
|
||||
|
||||
所以训练路径是这样一条,顺序不能换:
|
||||
|
||||
1. **造验证器**:把工程任务变成有判据的题,判定完全由程序做。
|
||||
2. **采轨迹**:让 agent 在隔离环境里做题,执行过程和核验结果一起落盘。
|
||||
3. **评测与打分**:同一套代码独立核验,核验结果就是训练信号。
|
||||
4. **训练**:先用做对且诚实的轨迹做 SFT 打底,再接 RLVR 做在线强化。
|
||||
5. **回到 1**:模型变强之后,题目难度和判据跟着升级。
|
||||
|
||||
第一阶段做的是第 1 到第 3 步,第 4 步的数据和接口已经备好。
|
||||
|
||||
**提示**
|
||||
|
||||
- 这段是领导最关心的"我们到底在干什么",讲慢一点。
|
||||
- 表格不要念完,念第一列和最后一列的对比,重点落在"我们不在那个约束里"。
|
||||
- 被问"什么时候开训":答案在第 9 节,判据是"分辨力四例各有一例被正确落位",不是"攒够多少条"。
|
||||
|
||||
---
|
||||
|
||||
## 2. 锚点:为什么第一个选 BPU 部署这条链路(3 分钟)
|
||||
|
||||
**讲什么**
|
||||
|
||||
嵌入式的工程任务很多,我们前期接触到的也不少。第一个锚点选在 BPU 部署链路上,理由很实在:**它是目前唯一一条能自己闭环的**。
|
||||
|
||||
判断一个任务适不适合做成训练场,有三条硬标准,三条同时成立才行:
|
||||
|
||||
| 标准 | 含义 | BPU 部署链路满足吗 |
|
||||
|---|---|---|
|
||||
| 判据可机器化 | 对错由程序判,不靠人评、不靠模型当裁判 | 满足:返回码、逐输出余弦、Top-1 / mAP 差值,全是数值比较 |
|
||||
| 验证远比求解便宜 | 跑一遍很贵、判一次很便宜,才撑得起大量采轨迹 | 满足:跑一个模型要编译加上板评测,判一次只是读一份结果文件 |
|
||||
| 失败模式是有限集 | 失败能分类,才能区分"判断错"与"工具链真做不到" | 满足:确定性五类、瞬态三类,其余进人工队列 |
|
||||
|
||||
第三条最容易被忽略,但它决定了"正确地放弃"这种能力能不能被学到。没有失败分类,agent 说"这个做不了"我们无法判真假,只能一律当失败处理。
|
||||
|
||||
除了这三条,还有现实理由:模型清单是现成的、数据集是现成的、板卡是现成的——环境搭起来就能跑,不需要先等别的条件成熟。
|
||||
|
||||
这条链路上有两个落点,一个在跑、一个在立项:
|
||||
|
||||
- **落点一:量化编译(在跑)。** 模型往板上走,量化配置怎么调才能既过精度又跑得快。这是本次汇报的主体。
|
||||
- **落点二:CPU fallback 算子优化(立项中)。** BPU 支持不了的算子会落到 CPU 上,往往吃掉端到端延时的大头。骨架可以直接复用,判据要按新任务重做——量化任务判的是精度,算子优化判的是正确性和速度,判据怎么定要重新设计。立项说明已经写好,等第一个落点的数据链跑顺再启动。
|
||||
|
||||
**提示**
|
||||
|
||||
- 说"第一个锚点",不说"只能做这个",留出横向扩展余地。
|
||||
- 落点二一句话带过,有兴趣的会单独找你。
|
||||
- 被问"为什么不先做更有业务价值的":答"这条链路本身就是业务交付——跑出来的就是模型在 S100P 上的上板基准数据,这是要交付的东西;训练数据是同一遍执行的副产物,不额外花钱"。这句能解释"为什么不是纯投入"。
|
||||
|
||||
---
|
||||
|
||||
## 3. 怎么做:三步走(5 分钟)
|
||||
|
||||
**讲什么**
|
||||
|
||||
方法上就三步:**搭环境 → 跑任务采数据 → 评测**。每一步里真正花时间的地方我说一下。
|
||||
|
||||
### 第一步:搭环境
|
||||
|
||||
不是装几个包的意思,是把"可复现"做成机械规则:
|
||||
|
||||
- 数据集复制入池、设只读、写清单(记哈希 / 数量 / 来源 / 日期),agent 永远不下载。三个理由:可复现(线上数据集版本会飘)、防作弊(agent 会挑好算的子集)、轨迹干净(网络波动不该进训练数据)。
|
||||
- 资源分池:GPU、CPU、板卡、agent 各自设并发,跨进程文件锁。板卡严格串行,一块板一个槽位——用一个并发数管所有会把 GPU 堵死。
|
||||
- 两套工具链两套虚拟环境,启动前比对依赖哈希,不一致直接拒跑而不是告警——否则"只追加不覆盖"就是句空话。
|
||||
- agent 的作答区隔离在内核层,隔离起不来就拒绝启动,不静默降级。
|
||||
|
||||
### 第二步:跑任务采数据
|
||||
|
||||
这里上周做了一次**设计调整,是这个阶段最重要的一次决策**。
|
||||
|
||||
原来的做法是每个模型先由确定性流水线完整跑一遍产出标准答案,再让 agent 做。实测下来这一遍代价很大:在一个 12.6 小时的观测窗口里,标准答案占了 7.9 小时板时,agent 采集只占 3.8 小时——**三分之二的板时花在一条轨迹也不产出的事情上**,而板卡是整条线的咽喉。
|
||||
|
||||
所以改成 **agent 主导**:默认只做输入准备——取权重、跑 FP32 基线、导出 ONNX,这三步的结果可缓存、可复用;输入齐备 agent 就直接领任务上手。完整的确定性流水线保留成一个显式开关,需要做对照或者排查问题的时候单独跑。
|
||||
|
||||
这个调整有一条边界要说清楚,因为它是最容易被误解的地方:**精度判据没有因此变松**。G3 判的是"板端实测的任务指标相对 **FP32 基线** 劣化不超过 5%"——参照物从来就是 FP32,不是默认配置的量化结果。FP32 基线属于输入准备的一部分,照常产出。所以去掉的只是"默认配置能做到多少"这个参考点,判定本身一点没动。
|
||||
|
||||
顺带一个收益:交付口径其实变好了。原来的基准数字回答的是"默认配置能不能上板",现在回答的是"这个模型在 S100P 上到底能做到什么水平"——后者才是客户真正想知道的。
|
||||
|
||||
### 第三步:评测
|
||||
|
||||
**这一层的投入产出比超出预期,值得多说两句。**
|
||||
|
||||
核心设计是一条:**板上只跑张量,别的都不上板。**
|
||||
|
||||
```
|
||||
主机:读图 → 预处理 → 输入张量
|
||||
↓ 传张量
|
||||
板子:跑推理 ← 板上只做这一件事
|
||||
↑ 回传输出张量
|
||||
主机:后处理 → 算指标 → 判门
|
||||
```
|
||||
|
||||
标注、评测器、后处理、指标计算永远留在主机侧。这带来四个好处:
|
||||
|
||||
1. **加新任务类板端零改动。** 不管是分割还是姿态估计,板子看到的都只是张量。我们现在正在补分割和姿态的适配,板端一行代码没动。
|
||||
2. **评测口径只有一份。** FP32 基线和量化模型吃同一份输入、走同一个评测器、比同一批图,比较才有意义——不会出现"板上一套、主机一套"。
|
||||
3. **板上不用装依赖。** 不需要 pycocotools、不需要各种数据集工具包,接一块新板的成本很低。
|
||||
4. **意外收获:板端比服务器模拟快 45 倍。** 单张 0.889 毫秒对 40 毫秒。所以"必须上真板"不是成本,是提速。
|
||||
|
||||
评测这层还有两件事是刻意做的:
|
||||
|
||||
- **数据分三档,规模是算出来的不是拍的。** 校准集、日常筛查集、交付全量集。日常筛查集分类任务用 2000 张——因为 500 张评 Top-1 单次标准误约 1.9 个百分点,分辨不出 1 个点的掉点;而 FP32 和量化是配对比较,差值标准误由结论不一致的图决定,2000 张时约 0.5 个点。所以 2000 张上只记"筛过了",交付数字要用完整验证集。
|
||||
- **入队门槛严格,探测不等于验收。** 新接入的模型要先核查权重入口、导出代码、评测器、校准数据、标注和附加门,缺一项就拒绝入队。姿态估计还额外要求板端同时判 OKS AP 和平均关键点偏移不超过 2 像素。有一个模型的类别语义没对齐,现在还卡在门外没让进——这是故意的。
|
||||
|
||||
最后,评测这层的验收标准要特别说一句:**看分辨力,不看通过数**。一个只会盖章通过的验证器,和一个真能分辨好坏的验证器,从成功样本上看是一模一样的。所以每一步的签收都要包含反向对照——故意造一个错的、造一个说谎的,确认系统把它们挑出来。
|
||||
|
||||
**提示**
|
||||
|
||||
- 三步的标题要报清楚,细节可按时间裁。
|
||||
- **第二步那个设计调整要主动讲,不要等人问。** 理由有三:一是已经写进契约的第一条原则,是有记录的决策不是漂移;二是文件夹里那几份 HTML 还是旧设计,不说清楚会被发现不一致;三是"我们发现三分之二板时花在不产数据的事情上,砍掉了"本身是个漂亮的决策故事,藏起来可惜。
|
||||
- 讲的时候重音放在"判据没变松"这句上,这是唯一可能被质疑的点,主动堵住。
|
||||
- 第三步是这次新增的重点,领导对"投入产出比高"这种话敏感,把 45 倍和"板端零改动"这两个具体的点说出来。
|
||||
|
||||
---
|
||||
|
||||
## 4. 这条线长什么样:五个构件(4 分钟)
|
||||
|
||||
**讲什么**
|
||||
|
||||
具体到实现,是一条单向链,五个构件,职责是硬边界:**出题的不作答,作答的不判分,判分的不看自述。**
|
||||
|
||||
1. **冻结的任务单元。** 一道题 = 任务类型 × 模型 × 芯片 × 后端 × 量化配置,拼成一个编号,比如 `cls--resnet18--nash-m--ptq--int8`。它既是磁盘上的目录名,也是所有表的关联键。换工具链、改阈值、修 bug 一律开新批次,老批次留着对照。这样任何一条轨迹永远能追溯到它当时面对的确切环境。
|
||||
|
||||
2. **可复用的输入准备。** 取权重、跑 FP32 基线、导出 ONNX。这三步的产物按内容哈希缓存,同一个模型在不同批次之间复用,不重复付钱。**要强调一点:准备产物是"可复用的输入证据",不是"标准答案"**——它不含任何量化结果,不参与判定。完整的确定性流水线是一个显式开关,做对照和排查时才跑。
|
||||
|
||||
3. **隔离的作答区。** agent 只有一个可写目录。隔离靠内核的命名空间机制,把兄弟轨迹、凭证、评测代码从文件系统里摘掉,再加进程隔离和权限清空。
|
||||
|
||||
4. **独立核验。** 四道门:编译、数值、任务指标、板端延时。其中编译和任务指标是必选门,数值余弦降为诊断,延时只记录。判定完全不读 agent 的自述,自述只用来和核验结果比对,不一致重罚——训练集里出现一次谎报成功,就是在教它谎报。
|
||||
|
||||
5. **离线打分和分级导出。** 打分是纯函数、带版本号,改配方可以对历史数据全量重打分,不用重跑任何实验。导出分两级:强化学习收正负例(要的是区分度),监督微调只收"做对了且诚实"的。
|
||||
|
||||
还有一条贯穿的原则:**每条轨迹彼此独立,永不串联。** 第 k+1 条看得见第 k 条,组内就不独立了,组内比较失去意义——而组内比较是整个训练方法的地基。所以"读了同一道题别人的作答"在审计里直接判不可用,和"读了评测代码"同等对待。
|
||||
|
||||
**提示**
|
||||
|
||||
- 构件 2 是这次改动最大的一块,措辞已经按新设计写好,照念即可。"可复用的输入证据,不是标准答案"这句是契约里的原话,很准确。
|
||||
- 四道门的现行必选/可选:编译和任务指标必选,余弦诊断,延时记录。直接讲现行的,不铺垫历史。
|
||||
- "出题的不作答,作答的不判分,判分的不看自述"这三句念出来效果好。
|
||||
|
||||
---
|
||||
|
||||
## 5. 路线与目标收敛(2 分钟)
|
||||
|
||||
**讲什么**
|
||||
|
||||
实施顺序是严格有序的:地基 → 一条切片打穿全流程并人肉核对每个数字 → 闭环(正常和修复各跑通一次,并能构造出"应该被拒"的例子)→ 横向铺一批压测资源池 → 全量。
|
||||
|
||||
第二步为什么要人肉核对:因为后面所有打分都建立在核验结果上,核验可信是整个计分体系的前提。这一步省掉,后面采的所有数据都建在沙子上。
|
||||
|
||||
采集分两条线,成功指标不同:一条是覆盖交付,求覆盖率;一条是训练数据,精选题目、每题多条采样,求分辨力。两条线按阶段分开调度,互不挤占。
|
||||
|
||||
目标板已经收敛到一块:**S100P**。配套口径也定了:精度必须板端实测,不认服务器模拟;任务指标相对 FP32 劣化 ≤ 5%;延时记录但不设硬门;不允许靠把算子挪下 BPU 来换精度;其他板型暂时无板,只判编译、精度标未验证。
|
||||
|
||||
这次收敛省下的算力是实打实的:任务数从两百多降到 63 个完整加编译检查,算力省掉三分之二。加上第 3 节讲的 agent 主导那次调整,同样的板子现在能产出的轨迹是原来的三倍多。
|
||||
|
||||
**提示**
|
||||
|
||||
- 措辞纪律:说"收敛",不说"之前太散"。收敛是阶段性动作,是好事。
|
||||
- 最后那句"三倍多"是两次调整的叠加效果(省 2/3 任务数 + 省 2/3 板时),是这次汇报里效率提升最有力的一句,别漏。
|
||||
|
||||
---
|
||||
|
||||
## 6. 跑到哪了:第一阶段战果(8 分钟,重点)
|
||||
|
||||
**讲什么**
|
||||
|
||||
**6.1 板端终判接通了。** 这是本阶段最关键的一步。精度评测现在走真实板端推理,主机预处理、板上跑张量、主机后处理算指标。在此之前所有精度都是服务器模拟出来的,不能作为交付依据。性能测试 31 秒,完整精度评测(2000 张)约 8 分钟。
|
||||
|
||||
**6.2 第一批可交付的上板基准。** 目前已有 18 个模型完成板端实测并通过,精度按相对 FP32 劣化 5% 判。这是第一批可以对外讲的数字。
|
||||
|
||||
**6.3 agent 轨迹开始产出。** 已结束 34 条:通过 22、失败 8、未验证 4,另有在跑和排队的。
|
||||
|
||||
**6.4 核心验收信号达成。** 上一阶段定的验收是"非最简单模型的通过轨迹 ≥ 5 条"——当时是 0,三条通过全集中在 resnet18 上,说明系统只证明了"流程能走通",还没证明"agent 能解决问题"。现在 22 条通过覆盖 14 个不同模型,resnet18 只占 4 条,其余覆盖 resnet 系列、resnext、wide-resnet、mobilenet、efficientnet 四个规格、regnet 三个规格、shufflenet 四个规格。
|
||||
|
||||
**6.5 修复模式跑出第一条成功轨迹。** 确定性流水线做不出来的模型,agent 调出来了——这正是我们最想采到的样本形态。明细核验还在做,核完可以定为第一条黄金样例。
|
||||
|
||||
**6.6 板卡抖动的数据回收。** 中途板子的网络网关有一段拒连,几十次板端评测失败,两条轨迹跑满预算也没测成。处理方式不是改分数,而是加了一个补测入口:用 agent 自己编出来的产物补跑板端两步再重新核验。两条都是真实通过——最好的一条板端 top-1 相对 FP32 只掉 0.06%,是全批次最小的;另一条掉 3.69%,也在门内。同时打分规则补了一条:板卡不可用时只免罚、不加分,并标明不可验证的原因。
|
||||
|
||||
**6.7 效率上的两次调整。** 目标板收敛省掉三分之二任务数;agent 主导省掉三分之二板时。两项叠加,同样的硬件现在能产出的轨迹是原来的三倍多。
|
||||
|
||||
**6.8 覆盖在往外扩。** 338 项的覆盖总表已经建起来,逐阶段记录:静态前置齐备、FP32 对照通过、量化编译、真实上板、任务精度通过分别记录,不能互相替代。分割和姿态估计的适配在做,姿态的板端判据已经定死——同时看 OKS AP 和平均关键点偏移 ≤ 2 像素。传统算法那几项如实标"不适用 PTQ",不虚构产物。
|
||||
|
||||
**6.9 隔离与数据洁净度经得起查。** 内核级隔离做过红队:手写请求打了 11 种路径逃逸,11 种全部挡住。审计脚本对每条轨迹给机械理由,不合格的不进数据集;历史数据集全部加作废标记留档、不删除——轨迹是复盘证据,删了同样的问题要重踩一次才知道。
|
||||
|
||||
**提示(措辞纪律)**
|
||||
|
||||
- 6.5 **只讲到"状态是通过、明细在核"**,不要说"核心假设已验证"。措辞用"这正是我们想要的形态,核完就是黄金样例"。
|
||||
- 6.5 还有一个坑:修复模式的输入是"确定性流水线的失败现场",而默认已经不跑它了。如果被追问"那以后还怎么产生修复任务",照第 10 节第 3 条答,不要临场编。
|
||||
- 不要引用早期那组 143 / 83 / 60 / 38 的统计。如果领导记得并问起,答"那是上一批不同板型的统计口径,这次全部切到板端终判后重新起的截面,两组不可比"。
|
||||
- 6.2 的 18 个,如果被问"这 18 个是 agent 跑出来的还是流水线跑出来的"——如实说是 09-16 那批完整流水线跑的,现在这批是 agent 在产出;两种都是板端实测,判据一样。**这个问题大概率会问,提前想好一句话。**
|
||||
- 板端利用率 93% 这个数字为后面的板卡诉求做铺垫,一定要说到。
|
||||
|
||||
---
|
||||
|
||||
## 7. 这一类工程的共性规律(5 分钟)
|
||||
|
||||
**讲什么**
|
||||
|
||||
这些不是我们独有的问题,是所有"长时间运行 + 多进程 + 数值判定"系统的共性规律。我们把它们变成了规矩写进契约,后面接手的人不用再花时间。
|
||||
|
||||
1. **正在跑的代码不等于磁盘上的代码。** 常驻调度进程加载的是它启动那一刻的代码和配置,缓存还会把旧配置留在内存里。所以"改好了"指的是文件,不是正在跑的东西。规矩:改了流水线必须让旧进程收尾退出再用新代码起新进程,不能只同步文件;进程启动时自证代码指纹,不一致拒绝领新任务。
|
||||
|
||||
2. **隔离要做在内核层,不做在字符串层。** 解析 agent 写的命令做白名单,是一类永远追不平的防线——每一版都会出现新写法。改成"让它看不见"之后这类问题结构性地消失了。命令解析那层降级成教学层,只负责给一句人话的拒绝理由、省 agent 的预算。
|
||||
|
||||
3. **数值相似度是过程量,任务指标才是结论。** 多分支输出不能展平算一个整体余弦——检测模型的输出里框坐标量级大、类别分数在 0 到 1 之间,整体余弦会被前者主导。实测过一个:整体余弦最低 0.9976,而类别分支只有 0.36,对应 mAP 从 46 掉到 26。规矩:多语义输出逐分支判;余弦降为诊断,不当必选门。
|
||||
|
||||
4. **容差必须显式声明方向和单位。** "相对劣化百分比"越小越好,"绝对百分点改善"越大越好,符号相反。独立复现过:同样是掉 2.5%,容差写 -5 判失败、写 5 判通过。规矩:每条容差显式声明类型,评测器按类型算方向;百分点和百分比不相加。
|
||||
|
||||
5. **预算耗尽不等于失败。** 已经产出并核验过的结果照常记账,不能因为超预算把整条成果清零——否则一条做得很好但没来得及收尾的轨迹会被记成彻底失败,而那往往是最有训练价值的一条。规矩:执行终止原因和产物判定分开记,互不覆盖。
|
||||
|
||||
6. **工具链层面两个具体的坑,写出来给后面的人省时间。** 运行时返回的数组引用的是内部缓冲区,下次调用会原地覆盖,必须显式拷贝——有一版评测因此把 Top-1 算成了 2.6%。校准数据目录下只能放指定格式文件,工具会把目录里所有文件当样本读,清单文件放进去会炸。
|
||||
|
||||
**提示**
|
||||
|
||||
- 整体是"经验沉淀"的口吻,不是"我们犯了什么错"。每条落点都是"变成了规矩",不是"造成了损失"。
|
||||
- 不要提具体进程号、不要说某批任务作废了多少个——那是内部复盘细节,汇报讲规矩就够。
|
||||
- 时间紧就讲 1、2、3,这三条最能说明"为什么这套系统要这么设计"。
|
||||
|
||||
---
|
||||
|
||||
## 8. 在建项(2 分钟)
|
||||
|
||||
**讲什么**
|
||||
|
||||
排在第二阶段的工作,都已定方案:
|
||||
|
||||
1. **两阶段编译**(最大的提速项)。一次编译里量化占 107 秒、BPU 编译占 78 分钟,比例 1 比 44;而 agent 迭代要看的精度信号在量化结束那一刻就有了。拆开之后单次实验从 35 到 118 分钟压到 5 分钟以内。工具链原生支持这个拆分,已在一个模型上实测过等价性。落地前要先固定两件事:编译参数完整转发,候选产物用摘要关联成一份清单。
|
||||
|
||||
2. **修复模式的触发方式**。目前修复任务的输入是确定性流水线的失败现场,而默认已经不跑它了。要改成吃 agent 自己的失败现场,或者对失败的模型显式补跑一次对照来生成现场。这条直接关系到能不能持续产出黄金样本,优先级高。
|
||||
|
||||
3. **自述时序的处理**(约 1 小时)。板子断开时 agent 按预算提醒先写了失败、补测后结论变成通过,这种时序差目前会被记成不一致。方案已定:标为待复核而不扣分,复核前不进训练候选。
|
||||
|
||||
4. **判定的进一步拆分**。编译成功、输入输出契约、产物一致性目前共用一个字段,要拆开各自存;输入输出校验目前只比形状,要补数据类型和排布。
|
||||
|
||||
5. **打分锚点的补充**。原来的相对打分项以默认配置为参照,现在这个参照点按需才有。好消息是强化学习本身是组内比较——同一道题多条采样,模型难度天然抵消,这部分影响不大;但跨题比较和监督微调的绝对门槛还需要一个锚,方案是用 FP32 基线加门槛,必要时对少量代表性模型保留对照。
|
||||
|
||||
6. **学生策略接入**。目前全部是教师模型的轨迹,在线强化学习的闭环还没跑过。接入后第一次跑的目标不是涨分,是确认同一道题的多条采样之间确实存在可区分的好坏。
|
||||
|
||||
7. **覆盖扩展**。分割和姿态估计的适配在做,按"模型数除以边际成本"排序——这两类复用已有的图像和缓存,边际成本最低。
|
||||
|
||||
8. **板卡健康检查与扩容**。目前只看配置里的启用标记,没有连通性探测。加上探测和退避之后,板卡数量直接决定实验周期。
|
||||
|
||||
**提示**
|
||||
|
||||
- 口吻是"已排期的工作",不是"欠账"。每条带方案和量级,显得可控。
|
||||
- 第 2 条和第 5 条是 agent 主导那次调整带出来的配套工作,主动列出来,说明这次改动是想清楚了才做的、连副作用都盘过——这比不提要好得多。
|
||||
- 第 8 条是唯一隐含资源诉求的点。领导接话再给数,不要主动伸手要。
|
||||
|
||||
---
|
||||
|
||||
## 9. 下一步与阶段验收(2 分钟)
|
||||
|
||||
**讲什么**
|
||||
|
||||
第二阶段按这个顺序推进,不并行:
|
||||
|
||||
1. 补齐修复模式的触发方式和打分锚点——这两条是上次调整的配套,先收尾。
|
||||
2. 两阶段编译落地。先在一个分类切片上验证参数、产物关联、板端数字一致,再扩到复杂模型和检测模型。
|
||||
3. 判定拆分和显式重判接口:历史批次按冻结规则判,新批次按新规则判,两套口径不混。
|
||||
4. 接学生策略,跑通一次在线采样,看同一道题多条采样之间打分有没有区分度。
|
||||
5. 规模化。
|
||||
|
||||
阶段验收我建议用两个信号,都不是通过数:
|
||||
|
||||
- **分辨力四例各有一例被正确落位**:做对了、做错了、正确放弃、超预算——尤其是后两类"应当被拒"的例子。这是方法论走通的签收点。
|
||||
- **可用轨迹比例**:版本合格、没越界、没被误判的轨迹占比。这才是这条流水线的产能指标。
|
||||
|
||||
通过数受模型难度、门槛松紧、工具链成熟度共同影响,想让它涨把阈值放松就行——所以它不能当项目指标。
|
||||
|
||||
**提示**
|
||||
|
||||
- 最后一句要说,但语气放平,说成"这是我们给自己定的纪律",不是"别拿通过率考核我们"。
|
||||
- 需要对外时间点时可以说:两阶段编译落地后采集效率提升一个数量级,届时给出第一版训练数据集的规模和分辨力指标。
|
||||
|
||||
---
|
||||
|
||||
## 10. 问答预案
|
||||
|
||||
**1. "不跑确定性流水线了,判定会不会变松?"**
|
||||
不会。精度门判的是"板端实测的任务指标相对 **FP32 基线** 劣化不超过 5%",参照物从来就是 FP32,不是默认配置的量化结果。FP32 基线属于输入准备,照常产出。去掉的只是"默认配置能做到多少"这个参考点,四道门一个没动,板端实测一步没省。
|
||||
|
||||
**2. "那怎么知道一个模型是 agent 做不好,还是我们的流水线本身有问题?"**
|
||||
三条:一是完整流水线保留成显式开关,怀疑的时候单独跑一次对照,几十分钟出结果;二是同一道题有多条独立采样,如果多条以完全相同的方式失败,那更可能是环境问题不是 agent 问题;三是失败有分类,环境类和判断类的错误码是分开的。另外这也正是我们保留这个开关而不是删掉的原因。
|
||||
|
||||
**3. "修复模式的输入是流水线的失败现场,不跑了以后怎么产生修复任务?"**
|
||||
这是这次调整带出来的配套工作,已经排进第二阶段第一项。两个方案:改成吃 agent 自己的失败现场,或者对失败的模型显式补跑一次对照生成现场。倾向前者——agent 自己的失败现场对训练更有价值,因为那才是它真实会遇到的情况。
|
||||
|
||||
**4. "什么时候能真正训出模型?"**
|
||||
数据侧的接口和格式已经备好,轨迹在持续产出。开训的判据不是攒够多少条,而是分辨力四例各有一例被正确落位——因为只产出正样本证明不了验证器有分辨力,用没有分辨力的数据去训,花了算力学到的是测量误差。这四例里还差两类,在第二阶段第一项里。
|
||||
|
||||
**5. "通过率是不是偏低?是不是说明模型大多上不了板?"**
|
||||
不能这么推。失败里混着配置层级的问题、进程终止、评测口径三类,都和模型本身能不能上 BPU 无关,要给这个结论得先把这几类剔干净。现在能确定说的是:通过的那些,精度数字是板端实测、按 5% 相对劣化判的,这部分可以直接作为交付基准。
|
||||
|
||||
**6. "agent 到底比默认配置强在哪?"**
|
||||
有明确的局部证据:一个分类模型默认量化掉 8.9 个百分点,agent 通过调位宽和校准参数做到掉 0.1 个百分点,诊断链条完整——定位误差累积位置、做过对照实验。同时我们也主动给它加了约束:如果延时完全不设限,"整网提到 int16"就是一行代码的通用解,任务会失去难度。所以位宽构成要进打分,对照要做三组:默认、全 int16、agent 方案。
|
||||
|
||||
**7. "评测这块投入大吗?"**
|
||||
不大,而且设计上是一次投入长期复用。板子上只跑张量,读图、预处理、后处理、算指标全在主机侧,所以加一类新任务板端零改动——我们现在补分割和姿态,板端一行没动。板上也不用装各种数据集工具包,接新板成本很低。顺带板端推理比服务器模拟快 45 倍,上真板反而是提速。
|
||||
|
||||
**8. "隔离真的安全吗?数据会不会泄漏?"**
|
||||
内核级隔离加进程隔离加权限清空,红队 11 种路径逃逸全部挡住。唯一藏不住的是大模型的访问凭证——agent 进程自己要读它,同进程的命令行就一定读得到,这是结构性限制,靠定期轮换兜底。另外审计层还有一道:即使读到了,内容指纹会命中,那条轨迹直接判不可用。
|
||||
|
||||
**9. "这套东西只能用在量化上吗?"**
|
||||
骨架和任务无关,可以迁出去,但有三条前提:判据可机器化、验证远比求解便宜、失败模式是有限集。CPU 算子优化那条满足,立项说明已经写了;判据要按新任务重做,这是迁移时最容易犯也最贵的错。
|
||||
|
||||
**10. "这件事对外能讲的价值是什么?"**
|
||||
四样产出:一是后训练出来的模型;二是带打分和失败分类的数据集;三是这套基准和流水线——换一块板、换一类任务,别的组能直接接着用;四是业务数字,端到端延时降低多少、覆盖多少客户模型。按我们做底座的定位,第三样的战略价值不低于第一样:模型会过时,基准不会,而且基准定义了整个行业用什么标准衡量这件事。
|
||||
|
||||
---
|
||||
|
||||
## 附:数字来源
|
||||
|
||||
| 讲稿里的数字 | 来源 | 截面 |
|
||||
|---|---|---|
|
||||
| agent 主导、输入准备只做 01–03、完整流水线改显式开关 | `bench/CONTRACT.md` §0 第一条原则 | 09-17 |
|
||||
| oracle 占板 7.92 h vs rollout 3.83 h、利用率 93%、板端故障统计 | `outputs/plan-verify-20260916/REVIEW.md` | 09-16 15:26 |
|
||||
| 轨迹计数、板端通过数、端点切换 | `outputs/glm53-endpoint-switch/runtime-after.json` | 09-16 21:03 |
|
||||
| 0.06% / 3.69% 补测结果 | `outputs/plan-verify-20260916/evidence/backfill_result.jsonl` | 09-16 16:35 |
|
||||
| 板端 31 s / 8 min、45 倍、2000 张统计依据 | `bench/PLAN.md` §4、`bench/README.md` | 09-15 |
|
||||
| 107 s : 78 min、两阶段等价性 | `bench/SPEC-two-phase-compile.md` | 09-15 |
|
||||
| 覆盖总表、姿态判据、入队门槛 | `bench/README.md` 头部、`bench/CONTRACT.md` 头部 | 09-17 |
|
||||
| 红队 11/11、审计规则、历史数据集作废 | `bench/REPORT-codex.md` | 09-15 |
|
||||
| 余弦 0.9976 对 0.36、分辨力四例 | `BPU量化轨迹工厂-闭环方法论.html` | 09-15 |
|
||||
| 三条适用前提、CPU 算子优化落点 | `CPU算子优化Agent-立项说明.html` | 09-15 |
|
||||
|
||||
**引用纪律**:旧截面的统计不能当现在的结论,引用前先确认截面时间;通过率不能当"模型能不能上 BPU"的结论。
|
||||
|
||||
**文档一致性提醒**:`方法论.html` 和 `闭环方法论.html` 仍是"确定性流水线先出标准答案"那一版设计,与现行的 agent 主导不一致。投屏前先说明,或抽空更新对应段落。
|
||||
498
docs/BPU量化轨迹工厂-方法论.html
Normal file
498
docs/BPU量化轨迹工厂-方法论.html
Normal file
@ -0,0 +1,498 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>把工程任务做成训练数据</title>
|
||||
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500;600&family=IBM+Plex+Sans:wght@400;500;600;700&family=IBM+Plex+Serif:ital,wght@0,400;0,500;1,400&display=swap">
|
||||
<style>
|
||||
:root {
|
||||
--paper:#EEF2F4; --surface:#FFFFFF; --surface-sunk:#E3EAEE;
|
||||
--ink:#11181D; --ink-soft:#495861; --ink-faint:#7C8C95;
|
||||
--rule:#C9D6DC; --rule-soft:#DCE6EA;
|
||||
--accent:#0B6E8A; --accent-soft:#DCEDF3;
|
||||
--pass:#2F6B46; --fail:#9E3223; --hold:#8A6516;
|
||||
--pass-bg:#E0EDE5; --fail-bg:#F6E2DE; --hold-bg:#F5EBD6;
|
||||
--f-display:"IBM Plex Sans","PingFang SC","Hiragino Sans GB","Microsoft YaHei",system-ui,sans-serif;
|
||||
--f-body:"IBM Plex Serif","Songti SC","SimSun",Georgia,serif;
|
||||
--f-mono:"IBM Plex Mono",ui-monospace,"SF Mono",Menlo,monospace;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root:not([data-theme="light"]) {
|
||||
--paper:#0E1418; --surface:#161E24; --surface-sunk:#0A1014;
|
||||
--ink:#E4EDF1; --ink-soft:#A3B4BD; --ink-faint:#6E828C;
|
||||
--rule:#2A3840; --rule-soft:#1F2B32;
|
||||
--accent:#4FB6D4; --accent-soft:#12303B;
|
||||
--pass:#6FBE8C; --fail:#E08A78; --hold:#D9B25F;
|
||||
--pass-bg:#142A1E; --fail-bg:#2E1A15; --hold-bg:#2A2311;
|
||||
}
|
||||
}
|
||||
:root[data-theme="dark"] {
|
||||
--paper:#0E1418; --surface:#161E24; --surface-sunk:#0A1014;
|
||||
--ink:#E4EDF1; --ink-soft:#A3B4BD; --ink-faint:#6E828C;
|
||||
--rule:#2A3840; --rule-soft:#1F2B32;
|
||||
--accent:#4FB6D4; --accent-soft:#12303B;
|
||||
--pass:#6FBE8C; --fail:#E08A78; --hold:#D9B25F;
|
||||
--pass-bg:#142A1E; --fail-bg:#2E1A15; --hold-bg:#2A2311;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body { background:var(--paper); color:var(--ink); font-family:var(--f-body); font-size:16px; line-height:1.75; -webkit-font-smoothing:antialiased; }
|
||||
.wrap { max-width:1000px; margin:0 auto; padding-block:56px 110px; padding-inline:24px; display:flex; flex-direction:column; gap:52px; }
|
||||
|
||||
.masthead { display:flex; flex-direction:column; gap:16px; }
|
||||
.eyebrow { font-family:var(--f-mono); font-size:11.5px; letter-spacing:.16em; text-transform:uppercase; color:var(--accent); font-weight:600; }
|
||||
h1 { font-family:var(--f-display); font-weight:700; font-size:clamp(30px,5vw,44px); line-height:1.12; letter-spacing:-.022em; margin:0; text-wrap:balance; }
|
||||
.standfirst { font-size:18px; color:var(--ink-soft); max-width:62ch; margin:0; }
|
||||
.facts { display:grid; grid-template-columns:repeat(auto-fit,minmax(150px,1fr)); border-top:1px solid var(--rule); border-bottom:1px solid var(--rule); margin:0; }
|
||||
.fact { padding:13px 18px 13px 0; border-right:1px solid var(--rule-soft); }
|
||||
.fact:last-child { border-right:0; }
|
||||
.fact dt { font-family:var(--f-mono); font-size:10.5px; letter-spacing:.1em; text-transform:uppercase; color:var(--ink-faint); margin:0 0 3px; }
|
||||
.fact dd { margin:0; font-family:var(--f-display); font-weight:600; font-size:15px; font-variant-numeric:tabular-nums; }
|
||||
|
||||
section { display:flex; flex-direction:column; gap:22px; }
|
||||
h2 { font-family:var(--f-display); font-weight:600; font-size:23px; letter-spacing:-.012em; margin:0; padding-bottom:10px; border-bottom:2px solid var(--ink); display:flex; align-items:baseline; gap:14px; text-wrap:balance; }
|
||||
h2 .num { font-family:var(--f-mono); font-size:12px; font-weight:500; color:var(--accent); letter-spacing:.08em; flex:none; }
|
||||
h3 { font-family:var(--f-display); font-weight:600; font-size:16.5px; margin:0; letter-spacing:-.005em; text-wrap:balance; }
|
||||
p { margin:0; max-width:70ch; }
|
||||
.lede { font-size:17px; }
|
||||
strong { font-weight:600; color:var(--ink); }
|
||||
em { font-style:italic; color:var(--ink-soft); }
|
||||
code { font-family:var(--f-mono); font-size:.84em; background:var(--surface-sunk); padding:1px 5px; border-radius:2px; color:var(--ink); overflow-wrap:anywhere; }
|
||||
|
||||
.rule-list { display:flex; flex-direction:column; }
|
||||
.rule { display:grid; grid-template-columns:40px 1fr; gap:18px; padding:20px 0; border-bottom:1px solid var(--rule-soft); align-items:start; }
|
||||
.rule:first-child { border-top:1px solid var(--rule-soft); }
|
||||
.rule .n { font-family:var(--f-mono); font-size:15px; font-weight:600; color:var(--accent); padding-top:2px; font-variant-numeric:tabular-nums; }
|
||||
.rule .b { display:flex; flex-direction:column; gap:9px; min-width:0; }
|
||||
.rule .b p { font-size:15.5px; color:var(--ink-soft); }
|
||||
.why { font-size:14.5px; color:var(--ink-soft); border-left:2px solid var(--hold); background:var(--hold-bg); padding:9px 13px; }
|
||||
.why b { font-family:var(--f-mono); font-size:11px; letter-spacing:.08em; text-transform:uppercase; color:var(--hold); font-weight:600; display:block; margin-bottom:2px; }
|
||||
.how { font-family:var(--f-mono); font-size:12.5px; color:var(--ink); background:var(--accent-soft); padding:9px 13px; border-left:2px solid var(--accent); line-height:1.65; overflow-wrap:anywhere; }
|
||||
|
||||
.flow { background:var(--surface); border:1px solid var(--rule); padding:18px 20px; overflow-x:auto; }
|
||||
.flow pre { margin:0; font-family:var(--f-mono); font-size:12.5px; line-height:1.85; color:var(--ink-soft); white-space:pre; }
|
||||
|
||||
.tablewrap { overflow-x:auto; border:1px solid var(--rule); background:var(--surface); }
|
||||
table { border-collapse:collapse; width:100%; font-size:14.5px; font-family:var(--f-display); }
|
||||
th, td { text-align:left; padding:10px 14px; border-bottom:1px solid var(--rule-soft); vertical-align:top; }
|
||||
thead th { font-family:var(--f-mono); font-size:10.5px; letter-spacing:.1em; text-transform:uppercase; color:var(--ink-faint); font-weight:600; border-bottom:1px solid var(--rule); white-space:nowrap; }
|
||||
tbody tr:last-child td { border-bottom:0; }
|
||||
td.num-cell { font-variant-numeric:tabular-nums; text-align:right; white-space:nowrap; }
|
||||
|
||||
.pill { font-family:var(--f-mono); font-size:10.5px; font-weight:600; letter-spacing:.06em; text-transform:uppercase; padding:2px 7px; border-radius:2px; white-space:nowrap; display:inline-block; }
|
||||
.pill.ok { background:var(--pass-bg); color:var(--pass); }
|
||||
.pill.no { background:var(--fail-bg); color:var(--fail); }
|
||||
.pill.part { background:var(--hold-bg); color:var(--hold); }
|
||||
|
||||
.callout { background:var(--surface); border:1px solid var(--rule); border-top:3px solid var(--accent); padding:20px 22px; display:flex; flex-direction:column; gap:10px; }
|
||||
.callout h3 { font-size:17.5px; }
|
||||
|
||||
.note { font-size:14px; color:var(--ink-faint); max-width:74ch; }
|
||||
footer { border-top:1px solid var(--rule); padding-top:18px; display:flex; flex-direction:column; gap:6px; }
|
||||
footer p { font-size:13.5px; color:var(--ink-faint); max-width:76ch; }
|
||||
|
||||
@media (max-width:620px) { .rule { grid-template-columns:1fr; gap:8px; } }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
|
||||
<header class="masthead">
|
||||
<div class="eyebrow">方法论 · BPU 量化轨迹工厂</div>
|
||||
<h1>把工程任务做成训练数据</h1>
|
||||
<p class="standfirst">这套东西从头到尾怎么做、每一步要注意什么。四个部分:任务怎么定、资源怎么摆、实验怎么跑、最后交付什么。每一条都是已经做过的,后面附着当初为什么这么定。</p>
|
||||
<dl class="facts">
|
||||
<div class="fact"><dt>规模</dt><dd>338 模型 / 10 类</dd></div>
|
||||
<div class="fact"><dt>已就绪</dt><dd>63(cls 48 · det 15)</dd></div>
|
||||
<div class="fact"><dt>主 run 已结束</dt><dd>168 oracle</dd></div>
|
||||
<div class="fact"><dt>目标板</dt><dd>S100P · nash-m</dd></div>
|
||||
<div class="fact"><dt>数据去向</dt><dd>RLVR · SFT</dd></div>
|
||||
</dl>
|
||||
</header>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§0</span>整条线长什么样</h2>
|
||||
<p class="lede">一句话:<strong>把 338 个模型的量化上板,逐个变成一道有标准答案的题;让 agent 在看不见答案的情况下做一遍;用同一套代码独立核验它做得对不对;核验结果就是训练信号。</strong></p>
|
||||
<div class="flow">
|
||||
<pre>registry/models.csv ──▶ new-run / expand ──▶ jobs(PENDING)
|
||||
│
|
||||
bench run(后台调度)──▶ oracle 八阶段:01 fetch → 02 baseline → 03 export
|
||||
│ → 04 calibrate → 05 compile
|
||||
│ → 06 hbm → 07 board → 08 eval
|
||||
│ 四道门:G1 编译 · G2 余弦 · G3 任务指标 · G4 延时
|
||||
▼
|
||||
bench collect(后台采集)──▶ 每个 job k 条 rollout:agent 在自己的 workspace 里作答
|
||||
│ → 独立核验 → verdict.json(reward 的唯一来源)
|
||||
▼
|
||||
audit_rollouts.py ──▶ 哪些轨迹干净、能进 RL / SFT
|
||||
export/to_agentrl.py ──▶ datasets/<dataset_id>/</pre>
|
||||
</div>
|
||||
<p>这条线上每一环的职责是硬边界,不能交叉:<strong>出题的不作答,作答的不判分,判分的不看自述。</strong>下面四节就是沿着这条线,把「怎么做、为什么这么做」写下来。</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§1</span>任务:怎么把一个工程活变成一道能训练的题</h2>
|
||||
<p>这一节回答的是:什么算一道题、什么算做对了、做不出来怎么办。</p>
|
||||
<div class="rule-list">
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">1</div>
|
||||
<div class="b">
|
||||
<h3>一道题 = 一个冻结的组合,题号既是目录名也是关联键</h3>
|
||||
<p>一道题叫一个 <strong>job</strong>:任务类型 × 模型 × 芯片 × 后端 × 量化配置,五段拼成 <code>job_id</code>,比如 <code>det--yolov8n--nash-e--ptq--int8</code>。它同时是磁盘上的目录名,和所有表里的关联键。</p>
|
||||
<div class="why"><b>为什么</b>后面要把轨迹、判定、奖励、数据集四类东西对起来。只要题号是唯一的、不变的,这四样永远能 join 回去;题号一变,历史数据就全断了。同理 run 也有号(<code>r20260914-01</code>),工具链指纹不进题号——两套 hbdk4 塞进一个号里,号本身就会说谎。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">2</div>
|
||||
<div class="b">
|
||||
<h3>先由确定性流水线跑出标准答案,再让 agent 做</h3>
|
||||
<p>每道题先由 harness 自己走一遍八个阶段,产出 <code>oracle/result.json</code>,这就是标准答案。agent 之后做的每一次,都相对它衡量。</p>
|
||||
<div class="why"><b>为什么</b>同一份标准答案有三个用途,少一个都不行:① <strong>对错的依据</strong>;② <strong>相对基准</strong>——奖励里的精度和延时都算相对 oracle 的差,因为绝对延时是模型自身的属性,不是 agent 的功劳;③ <strong>「做不到」的裁判</strong>——agent 说这道题做不了的时候,看 oracle 是不是在同一阶段以同类错误失败。不需要人判,也不需要另一个模型来判。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">3</div>
|
||||
<div class="b">
|
||||
<h3>题的状态和答的状态分开记,永不互相覆盖</h3>
|
||||
<p>job 的状态由 oracle 决定(<code>PASS / FAIL / UNVERIFIED</code>),rollout 的状态由核验结果决定(<code>PASS / FAIL / BLOCKED / UNVERIFIED</code>)。一道题只有一份标准答案,每次作答各有一份判定。</p>
|
||||
<div class="why"><b>为什么</b>塞进一个字段就会出数据事故:要么标准答案被某次作答覆盖,要么两种 PASS 混成一个。而且<strong>「oracle 做不出来、agent 做出来了」是最值钱的样本</strong>——它要么是黄金数据,要么说明我们的流水线有 bug。这类自动打 <code>review</code> 标记进人工抽查,查完反过来修 harness,形成回路。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">4</div>
|
||||
<div class="b">
|
||||
<h3>判据分成四道门,必选和可选分清楚</h3>
|
||||
<p>G1 编译(产物在不在、芯片型号和输入格式对不对、日志有没有 ERROR)· G2 数值(量化前后逐输出余弦)· G3 任务指标(同一子集、同一预处理、同一评测器下相对 FP32 的掉点)· G4 板端延时。必选门全过才算 PASS。</p>
|
||||
<div class="why"><b>为什么</b>门是<strong>绝对判断</strong>——这个模型能不能交付;奖励是<strong>相对判断</strong>——agent 比标准答案做得好还是差。两件事都要,但不能混成一个数。</div>
|
||||
<div class="how">最容易出事的一条:G1 里的芯片型号不一致直接判死。这是 agent 最容易钻的口子——换个芯片编译当然更容易过。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">5</div>
|
||||
<div class="b">
|
||||
<h3>「这轮不评」和「本该评没评成」必须分开记</h3>
|
||||
<p>门有两种非结果:<code>na</code> 是本轮设计上就不评(比如 3D 占据第一轮只编译不评精度),不阻塞 PASS;<code>unverified</code> 是本该评但没评成(数据缺、板没接),阻塞 PASS。</p>
|
||||
<div class="why"><b>为什么</b>这是交付文档里「未判定 ≠ 不适用」的直接映射。两者混成一个「没数」,对外就会说成「这个模型不行」,而实际上只是我们没测。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">6</div>
|
||||
<div class="b">
|
||||
<h3>失败必须有名字,因为它要同时喂三个地方</h3>
|
||||
<p>每个失败由正则分类器从 stderr 打一个标签,存进数据库和结果文件。确定性的五类:<code>unsupported_op · shape_infer · export_fail · calib_nan · verifier_mismatch</code>;瞬态的三类:<code>oom · board_offline · stalled</code>;没匹配上的进 <code>unknown</code> 人工队列,规则库随之增长。</p>
|
||||
<div class="why"><b>为什么要分类</b>不是为了好看,是因为这个标签同时被三处用到:① <strong>决定要不要重试</strong>——确定性的不重试(重试一百次也一样),瞬态的重试;② <strong>喂给 repair 模式的提示词</strong>——修复任务要告诉 agent 现场是什么错;③ <strong>判定「做不到」是真是假</strong>——agent 报 BLOCKED 时,比对 oracle 是不是同阶段同类错误。<strong>没有这三处用途,确实不用分类;有了这三处,不分类就没法做。</strong></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">7</div>
|
||||
<div class="b">
|
||||
<h3>难度不是事先定的,是 oracle 跑完自己浮出来的</h3>
|
||||
<p>顺序是:先把所有就绪模型的 oracle 铺一遍 → 得到失败率分布和错误分类分布 → 难度自然分成三层:默认配置就过的、改配置能救的、工具链真做不到的。<strong>采轨迹的重点是中间那层,但这一层只能等 oracle 跑完才知道是谁。</strong></p>
|
||||
<div class="why"><b>为什么</b>第一轮预期失败率就是高的,尤其局部特征、光流、3D 占据这三类——当初就写明了「这是数据,不是事故」。事先按模型大小或者主观印象去猜哪道题难,猜不准;而 oracle 本来就要全铺一遍(它是 benchmark 交付本身),难度分层是这一遍的<em>副产品</em>,不额外花钱。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§2</span>资源:机器、数据、板子怎么摆</h2>
|
||||
<p>这一节回答的是:算力怎么分、数据从哪来、慢的东西怎么办。</p>
|
||||
<div class="rule-list">
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">1</div>
|
||||
<div class="b">
|
||||
<h3>资源分池,一个并发数不能管所有</h3>
|
||||
<p>四个池各自设并发:GPU(基线、评测)· CPU(导出、编译)· board(板卡)· agent(受 LLM 端点限速)。池是跨进程的文件锁,调度进程、agent 提交的阶段、多个 <code>bench run</code> 共用同一组槽位。</p>
|
||||
<div class="why"><b>为什么</b>用一个数字管所有,板端会把 GPU 堵死——板卡是严格串行的,一块板就是一个槽位,而 GPU 阶段本可以并行好几路。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">2</div>
|
||||
<div class="b">
|
||||
<h3>板卡按物理身份登记,一块板一把锁,槽位不复用</h3>
|
||||
<p>每块独立物理板登记唯一槽位、芯片型号、运行库路径和各自的连接信息;容量等于启用的板数。停用的板标记禁用,槽位不回收再用。同一地址、同一物理板的多个容器,不能登记成额外容量。</p>
|
||||
<div class="why"><b>为什么</b>容量是用来排队的。把同一块板登记两次,排队系统会以为有两块,两个任务同时推上去互相干扰,测出来的延时全废。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">3</div>
|
||||
<div class="b">
|
||||
<h3>数据复制入池、设只读、写清单;agent 永远不下载</h3>
|
||||
<p>数据集从机器上别人的目录<strong>复制</strong>进自己的池子(不是软链),复制完 <code>chmod -R a-w</code>,写 <code>manifest.json</code> 记样本清单的 sha256、数量、来源路径、日期。</p>
|
||||
<div class="why"><b>为什么复制不软链</b>别人的目录随时可能被清掉。几十 GB 的冗余换来的是零耦合。<b style="margin-top:6px">为什么 agent 不下载</b>三条:可复现(线上数据集版本会飘)、防作弊(agent 会挑好算的子集)、轨迹干净(网络波动不该进训练数据)。<br>另外实测过:官方 COCO 源 26 KB/s,815 MB 要下 9 小时,而同样的东西 HF 是 8 MB/s、阿里云 OSS 是 43.8 MB/s。<strong>照着任务清单里给的官方入口下载是下不动的</strong>,所以每个数据集声明有序多源,逐个试、成功后校验 sha256。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">4</div>
|
||||
<div class="b">
|
||||
<h3>数据分三档,用途不同,规模是算出来的</h3>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>档</th><th>用途</th><th>规模</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><strong>T0 calib</strong></td><td>量化校准</td><td>128–512 张,从 train 抽,与验证集零交集</td></tr>
|
||||
<tr><td><strong>T1 smoke</strong></td><td>日常<em>筛</em></td><td>cls 2000 · det/iseg/pose 1000 · seg 500</td></tr>
|
||||
<tr><td><strong>T2 full</strong></td><td>交付<em>判</em></td><td>完整验证集,开新 run</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="why"><b>为什么 cls 是 2000 不是 500</b>这是算出来的,不是拍的:500 张评 Top-1,单次测量的标准误约 1.9 个百分点,根本分辨不出 1 个点的掉点。但 FP32 和量化是<strong>配对</strong>比较——同一批图,误差高度相关,差值的标准误由「结论不一致的那些图」决定,2000 张时约 0.5 个点,够用;500 张时约 1 个点,和容差同量级,会误判。<br><strong>所以 T1 上只能记 <code>pass-screen</code>(筛过了),T2 上才是 <code>pass</code>(判过了)。</strong>对训练信号来说 T1 够用,对交付数字来说必须 T2。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">5</div>
|
||||
<div class="b">
|
||||
<h3>没有固定超时,只有看门狗</h3>
|
||||
<p>判定卡死的条件是两个同时成立:<strong>日志 15 分钟不增长,且整个进程树 CPU 空闲</strong>。24 小时硬上限只是保险,正常永远碰不到。编译走单独的长车道(1 小时)。</p>
|
||||
<div class="why"><b>为什么</b>不同模型的编译时长差两个数量级——实测 convnext.small 118 分钟、efficientnet.b4 74 分钟、resnet50 61 分钟。固定超时一定会误杀慢模型,而误杀产生的失败样本<strong>不是 agent 的错,却会被当成 agent 的失败学进去</strong>。<br>判据本身也有坑:CPU 采样必须递归覆盖所有后代进程(编译的实际计算在孙子进程里),D 状态(等 I/O)不能算作空闲。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">6</div>
|
||||
<div class="b">
|
||||
<h3>重活由 harness 执行,agent 只提交和等待</h3>
|
||||
<p>agent 不在自己的 bash 里跑编译,而是 <code>bench stage run compile --config workspace/x.yaml</code> 提交,拿一个句柄,再 <code>bench stage wait</code> 等结果。它的 bash 仍然可用,用来看日志、改配置、跑小脚本。</p>
|
||||
<div class="why"><b>四个好处</b>① 编译时长和 agent 工具的超时脱钩;② 后端和虚拟环境的选择不可能被 agent 选错(两套工具链版本完全不同,禁止把这个判断交给 agent);③ 预算和看门狗对 oracle 与 agent 一视同仁;④ 日志位置统一,回放的时候找得到。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">7</div>
|
||||
<div class="b">
|
||||
<h3>便宜的信号用来迭代,贵的信号用来终判</h3>
|
||||
<p>同一个编译命令里,<strong>量化 107 秒,BPU 编译 78 分钟,比例 1:44</strong>(resnet50 实测日志时间戳)。而 agent 迭代要看的精度信号,在量化结束那一刻就有了。所以把这两步拆开:agent 用量化后的中间产物快速试,确认可行了才付那一次昂贵的 BPU 编译,产出板端权威数字。</p>
|
||||
<div class="why"><b>实测依据</b>resnet18 两阶段:量化 23 秒 + 编译 73 秒,产出的 .hbm <strong>字节数与一次性编译完全一致</strong>,所以拆分没有正确性风险。<br>另一头:x86 上跑中间产物 40.5 毫秒一张,板端 BPU 0.889 毫秒一张,<strong>板端快 45 倍</strong>——但要用板端就得先付编译的钱。所以分工是:便宜的当快速筛,板端当终判,权威数字只认板端。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§3</span>实验方案:怎么跑,怎么保证跑出来的东西可信</h2>
|
||||
<p>这一节是踩坑最多的一节。可信度不是靠小心,是靠几条机械规则。</p>
|
||||
<div class="rule-list">
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">1</div>
|
||||
<div class="b">
|
||||
<h3>只追加,不覆盖:口径变了就开新 run</h3>
|
||||
<p>换工具链、改配置、修 bug 之后重跑,一律开新 run,老 run 留着对照。run 的清单文件记录两套虚拟环境 <code>pip freeze</code> 的 sha256,每个任务启动前比对,<strong>不一致直接拒跑</strong>。状态只存在数据库里,文件系统只存内容。</p>
|
||||
<div class="why"><b>为什么要拒跑而不是告警</b>否则「只追加」是句空话——同一个 run 里混着两套工具链跑出来的结果,事后没有任何办法区分。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">2</div>
|
||||
<div class="b">
|
||||
<h3>正在跑的代码 ≠ 磁盘上的代码</h3>
|
||||
<p>常驻调度进程加载的是<strong>它启动那一刻</strong>的代码和配置。改完 harness 必须让旧进程收尾退出(<code>bench drain</code>,不要 kill 正在编译的进程),再用新代码起新进程。</p>
|
||||
<div class="why"><b>踩过的坑</b>一个调度进程连续跑了 23 小时多的旧代码,期间生成的编译配置一直是修好之前的写法;磁盘上的文件早就改对了,看文件完全看不出问题。<strong>「改好了」指的是文件,不是正在跑的东西。</strong>缓存也会把旧配置留在内存里。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">3</div>
|
||||
<div class="b">
|
||||
<h3>agent 只有一个可写区,隔离必须在内核层</h3>
|
||||
<p>agent 的工作目录只有 <code>rollouts/<kk>/workspace/</code>,标准答案、评测代码、别的作答、凭证全在外面。隔离靠 mount namespace 把这些东西<strong>从文件系统里摘掉</strong>,加上 PID 命名空间和权限清空;jail 起不来就拒绝启动,不静默降级。</p>
|
||||
<div class="why"><b>踩过的坑</b>之前那版隔离是解析 agent 写的 shell 命令做白名单,<strong>连续四版被绕过</strong>:换行后的 <code>cd</code>、<code>$IFS</code>、路径里拼 <code>$VAR/</code>、在自己目录里做个软链接指回去。这类防线永远追不平所有写法。改成「让它看不见」之后,这一类 bug 就没有了——看不见的东西,再多绕法也没用。<br>命令解析那层降级成教学层:只负责给一句人话的拒绝理由,省 agent 的预算。<br><b style="margin-top:6px">藏不住的一样东西</b>LLM 凭证。agent 进程自己要读它,同进程的 bash 就一定读得到。这是结构性限制,靠定期轮换 key 兜底,别指望隔离。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">4</div>
|
||||
<div class="b">
|
||||
<h3>agent 说的不算数,独立核验才算数</h3>
|
||||
<p>判定完全不读 agent 的自述。它交的自述只用来和核验结果比对,比对结果单独记一个字段。自述与核验不一致要重罚,而且这类样本绝不进监督训练集。</p>
|
||||
<div class="why"><b>为什么</b>训练集里出现一次「谎报成功」,就是在教它谎报。没交自述只小额扣分——不报比报错轻,但不能没有代价。<br>还有一条同类的:<strong>编译返回 0 不等于验收通过。</strong>编译日志里的百分比、校准余弦、静态性能估算都不能代替四道门。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">5</div>
|
||||
<div class="b">
|
||||
<h3>每条轨迹彼此独立,永不串联</h3>
|
||||
<p>三层重试,边界是硬的:① harness 对瞬态错误的重试<strong>不算轨迹</strong>(没有 agent 参与);② agent 在一条轨迹里看到报错自己改配置重提,<strong>是同一条轨迹</strong>——这正是我们要采的诊断过程;③ 新的 rollout 是<strong>全新轨迹</strong>,全新上下文,不知道其他作答存在;④ 修复模式只吃 oracle 的失败现场,不吃别人作答的失败。</p>
|
||||
<div class="why"><b>为什么</b>组内比较是整个训练方法的地基。第 k+1 条看得见第 k 条,组内就不独立了,比较失去意义。所以「读了同一道题别的作答」在审计里直接判不可用,和「读了标准答案」同等对待。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">6</div>
|
||||
<div class="b">
|
||||
<h3>失败的轨迹不删,过滤发生在导出那一刻</h3>
|
||||
<p>磁盘上留什么,和数据集里收什么,是两件事。轨迹全留,导出时按版本号、污染标注和资格函数筛。改了提示词、边界或扩展,就把 agent 版本号 +1,旧版本的轨迹自然被过滤掉,不需要删。</p>
|
||||
<div class="why"><b>为什么</b>三条:① 轨迹是复盘证据——好几个 bug 都是靠原始轨迹定位的,删了同样的坑要重踩一次才知道;② 数据洁净度由导出时的规则保证,和磁盘上留不留无关;③ 版本号是用来追溯的,它记着这条轨迹是在哪一版规则下跑的。<br>同理,job 和 run 也不删,历史数据集作废也是加作废标记(连同原清单的哈希),不是删除。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">7</div>
|
||||
<div class="b">
|
||||
<h3>每条轨迹能不能用,由审计逐条给理由</h3>
|
||||
<p>审计脚本每条轨迹输出一行,理由是机械判据:读了标准答案、读了别人的作答、被看门狗误杀、旧版本、没交自述、自述与核验不符、去读评判代码或工具链、内容指纹命中、碰了状态库、读了别的题。干净且版本合格的标 <code>RL ✓</code>,再加上核验通过且自述一致才标 <code>SFT ✓</code>。</p>
|
||||
<div class="why"><b>注意一个反例</b>做内容指纹匹配时,agent 自己合法拿到的阶段结果可能和标准答案数值完全相同(因为本来就该相同),这类不能算泄漏证据,否则正常轨迹会被误报。判泄漏要看<em>来源</em>,不能只看数值撞上了。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">8</div>
|
||||
<div class="b">
|
||||
<h3>先打穿一条切片,人肉核对每一个数,再扩量</h3>
|
||||
<p>实施顺序是:地基(骨架、契约、数据入池、预检全绿)→ <strong>一条 oracle 切片打穿八个阶段,人工核对每一个数字</strong> → 闭环(端到端和修复各跑通一次)→ 横向铺一批压测资源池 → 全量。</p>
|
||||
<div class="why"><b>为什么人肉核对</b>因为后面所有奖励都是相对 oracle 算的——<strong>oracle 可信是整个计分体系的前提</strong>。这一步省掉,后面采的所有数据都建在沙子上。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">9</div>
|
||||
<div class="b">
|
||||
<h3>假定 oracle 自己也可能是错的,留三道防线</h3>
|
||||
<p>① 四道门是绝对判断,不依赖 oracle;② 相对项全部 clip 在有界区间,单条错值不会把奖励打飞;③ 「oracle 失败而 agent 成功」的样本强制人工抽查,查完反过来修 harness。</p>
|
||||
<div class="why"><b>为什么</b>harness 的 bug 会系统性地把奖励打歪,而且不容易被发现——它不像崩溃,它安静地产出看起来合理的数字。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">10</div>
|
||||
<div class="b">
|
||||
<h3>改判定口径必须同时升版本号并重判</h3>
|
||||
<p>评测器改了口径就升评测器版本;门的阈值改了就重跑判定(历史任务按它冻结时的阈值重判,新任务用新阈值)。两者都不做,数据库里就会同时存在两套口径的数字,而且分不出来。</p>
|
||||
<div class="why"><b>踩过的坑</b>某个阈值改了之后没有重判,一批任务就一直按旧阈值挂着失败。<br>还有一类更隐蔽的:<strong>容差的方向和单位写反</strong>。「相对劣化百分比」越小越好,「绝对百分点改善」越大越好,两者符号相反;独立复现过一次——同样是掉 2.5%,容差写 <code>-5</code> 判失败、写 <code>5</code> 判通过。<strong>每条容差必须显式声明 kind,评测器按 kind 计算,不允许省略。</strong>另外百分点和百分比不能相加,统计误差的放宽单位也要跟着换算。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">11</div>
|
||||
<div class="b">
|
||||
<h3>指标要和被测对象的结构匹配</h3>
|
||||
<p>多分支输出不能展平成一个整体余弦。检测模型的输出里同时有框坐标(量级大)和类别分数(0–1),整体余弦会被前者主导。实测过一个:<strong>整体余弦最低 0.9976,而类别分支只有 0.36</strong>。</p>
|
||||
<div class="why"><b>怎么办</b>凡是多语义输出,逐分支判、逐分支记。这条同时提醒一件事:<em>不要拿一个高余弦就推断精度没问题</em>——数值相似度是过程量,任务指标才是结论。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">12</div>
|
||||
<div class="b">
|
||||
<h3>一层只能有一个写者</h3>
|
||||
<p>代码按层划分归属:流水线与门、agent 配置、导出、清单与阈值、脚本。谁在改哪层要写明,别人对那层只读。改动只在本地副本做,通过同步脚本推上机器;同步脚本带守卫,机器上被人动过就拒绝推送。</p>
|
||||
<div class="why"><b>踩过的坑</b>两个会话同时改一棵树,互相覆盖过一次;清单文件还丢过两次。机器上的运行数据(runs、state、data、cache)一律不手改——改判走重判命令,补测走补测命令。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§4</span>产出:交付什么,怎么算做成了</h2>
|
||||
<p>这条线上其实同时长出四样东西,四样都是产出,验收方式各不相同。</p>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>产出</th><th>是什么</th><th>怎么算做成了</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><strong>① 量化交付数据</strong></td><td>每个模型能不能上板、掉多少点、板上多快——oracle 这一遍本身就是 benchmark</td><td>四道门有明确判定;T2 上的数字才叫交付数</td></tr>
|
||||
<tr><td><strong>② 训练数据</strong></td><td>带可核验分数的 agent 轨迹(教师、学生分开落)</td><td>看干净可用的比例,不看通过数</td></tr>
|
||||
<tr><td><strong>③ 这套流水线本身</strong></td><td>目录契约、四道门、核验与导出——换个任务也能用</td><td>换一类任务不用改环境契约</td></tr>
|
||||
<tr><td><strong>④ 复盘证据</strong></td><td>每次失败的分类、每条轨迹的回放</td><td>任何结论都能追到原始记录</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="rule-list">
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">1</div>
|
||||
<div class="b">
|
||||
<h3>奖励离线算,纯函数,配方带版本号</h3>
|
||||
<p>核验结果文件里只存<strong>原始测量值</strong>,奖励在导出阶段离线算。配方是纯函数、带版本号,改了配方可以对历史数据全量重打分,不用重跑任何实验。</p>
|
||||
<div class="why"><b>为什么</b>配方一定会改好几版。如果奖励在跑的时候就写死进结果里,每改一次配方就要重跑一次实验——那是几个星期的机器时间。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">2</div>
|
||||
<div class="b">
|
||||
<h3>导出资格分两级:强化学习收正负例,监督训练只收做对且诚实的</h3>
|
||||
<p>默认保留全部已结束的终态(成功、精度不达标、预算耗尽分别标注)。<strong>能不能进监督集是一层筛选,不代表数据总量。</strong>自述与核验不一致、需要人工复核这两条,是优先级最高的硬排除。</p>
|
||||
<div class="why"><b>注意</b>预算耗尽不等于失败。已经产出并核验过的门结果照常记账,不能因为超预算把整条成果清零——否则一条做得很好但没来得及收尾的轨迹,会被记成彻底失败。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">3</div>
|
||||
<div class="b">
|
||||
<h3>每个数字都要带着证据边界一起流转</h3>
|
||||
<p>必须显式区分的几组:<code>pass-screen</code>(筛子上过了)≠ <code>pass</code>(完整验证过);工具的静态性能估算 ≠ 板端实测;服务器上的模拟运行 ≠ 物理板端运行;用随机输入测的延时 ≠ 用真实数据集测的精度。</p>
|
||||
<div class="why"><b>为什么</b>弱证据会在传递过程中逐级升格成强结论——第一手的人知道那是筛出来的,传到第三个人就变成「这个模型通过了」。这条不产出代码,但它决定这批数据将来能不能被别人信。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">4</div>
|
||||
<div class="b">
|
||||
<h3>验收看分辨力,不看通过数</h3>
|
||||
<p>通过数受模型难度、门槛松紧、工具链成熟度共同影响——想让它涨,把阈值放松就行。真正的验收是:给这套系统四类终态(做对了、做错了、正确放弃、超预算),它能不能<strong>把四类都正确落位,包括把该拒的拒掉</strong>。</p>
|
||||
<div class="why"><b>为什么</b>一个只会盖章通过的验证器,和一个真能分辨好坏的验证器,<em>从成功样本上看是一模一样的</em>。所以只产出正样本,证明不了任何事。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="rule">
|
||||
<div class="n">5</div>
|
||||
<div class="b">
|
||||
<h3>教师和学生是两种数据,分开采、分开落</h3>
|
||||
<p>教师模型的轨迹加核验筛选,用于监督训练和拒绝采样;学生模型(自己的模型)的轨迹用于在线强化学习。提示词模板相同,在矩阵里按策略维度分开采,导出时落到不同数据集。</p>
|
||||
<div class="why"><b>还有一条</b>真正做在线强化学习时,需要采样时的对数概率,而现在这套 agent 进程只有对话记录。到那一步,驱动层会换成训练框架自己的循环。<strong>这套方案的持久资产是环境契约——目录结构、四道门、核验文件格式——不是当前这个 agent 进程。</strong></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§5</span>这套方法论现在走到哪一步</h2>
|
||||
<p>截至 2026-09-16 的核验快照,用来说明这套方法论走到了哪一步,也用来提醒哪些数字是旧口径。</p>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>环节</th><th>状态</th><th>说明</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>目录契约与命名</td><td><span class="pill ok">已落地</span></td><td>job / run / rollout / 阶段 / 数据集五套编号都在用</td></tr>
|
||||
<tr><td>oracle 八阶段</td><td><span class="pill ok">已落地</span></td><td>主 run 已结束 168 个(3 通过 / 165 失败),另有在跑和排队的</td></tr>
|
||||
<tr><td>四道门与核验</td><td><span class="pill part">在修口径</span></td><td>容差方向与单位、多分支指标、一致性检查的拆分正在改</td></tr>
|
||||
<tr><td>内核级隔离</td><td><span class="pill ok">已落地</span></td><td>外部执行、只读产物、票据校验均已实现并有回归</td></tr>
|
||||
<tr><td>轨迹审计与导出</td><td><span class="pill ok">已落地</span></td><td>按版本、污染标注、资格函数三重过滤;旧数据集已作废留档</td></tr>
|
||||
<tr><td>板端终判</td><td><span class="pill part">接线中</span></td><td>板卡可登录;37 份 nash-m 产物中 9 个有板端性能;板端精度路径尚未接上</td></tr>
|
||||
<tr><td>两阶段编译</td><td><span class="pill part">计划内 · 待改</span></td><td>方案已验证等价(字节一致),代码仍走一次性完整编译</td></tr>
|
||||
<tr><td>修复模式 / 学生策略</td><td><span class="pill part">计划内 · 待启用</span></td><td>数据库里全部是教师的端到端轨迹</td></tr>
|
||||
<tr><td>任务覆盖</td><td><span class="pill part">63 / 338</span></td><td>其余 257 个需要导出适配;评测器目前只有分类和检测两种</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="callout">
|
||||
<h3>引用数字时的两条纪律</h3>
|
||||
<p><strong>一、旧截面的统计不能当现在的结论。</strong>之前复盘里的 143 / 83 / 60 / 38 都是更早的截面和口径,现在的数是 168 个已结束 oracle。引用前先确认截面时间。</p>
|
||||
<p><strong>二、通过率不能当作「模型本身能不能上 BPU」的结论。</strong>这批失败里混着配置层级的错误、看门狗终止、评测口径问题——都和模型本身能不能上板无关。要给这个结论,得先把这几类剔干净。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<footer>
|
||||
<p><strong>来源</strong>:本文的每一条都出自 <code>bench/</code> 已有的文档与实现——<code>CONTRACT.md</code>(契约 v0.4 落地版)、<code>HANDBOOK.md</code>(掌控手册)、<code>README.md</code>、<code>SPEC-two-phase-compile.md</code>(两阶段编译实测)、<code>REPORT-codex.md</code>(v5 隔离与导出修复报告)、《BPU 量化轨迹工厂 设计方案 v0.4》,以及 2026-09-16 的两份核验(方案审查、S100P 现状核验)。</p>
|
||||
<p><strong>证据边界</strong>:§5 的状态是 2026-09-16 01:17–01:48 的远端快照,不是实时;本文没有重跑任何编译、推理或评测。文中实测数字(编译耗时比、两阶段字节一致、推理速度、外网带宽、统计标准误)均引自上述文档中已记录的实测,未重新验证。</p>
|
||||
</footer>
|
||||
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
790
docs/BPU量化轨迹工厂-设计方案.html
Normal file
790
docs/BPU量化轨迹工厂-设计方案.html
Normal file
@ -0,0 +1,790 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<style>html{color-scheme:light dark}body{margin:0}img{max-width:100%}[hidden]{display:none!important}</style>
|
||||
<title>BPU 量化轨迹工厂</title>
|
||||
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500;600&family=IBM+Plex+Sans:wght@400;500;600;700&family=IBM+Plex+Serif:ital,wght@0,400;0,500;1,400&display=swap">
|
||||
<style>
|
||||
:root {
|
||||
--paper: #EEF2F4;
|
||||
--surface: #FFFFFF;
|
||||
--surface-sunk: #E3EAEE;
|
||||
--ink: #11181D;
|
||||
--ink-soft: #495861;
|
||||
--ink-faint: #7C8C95;
|
||||
--rule: #C9D6DC;
|
||||
--rule-soft: #DCE6EA;
|
||||
--accent: #0B6E8A;
|
||||
--accent-soft: #DCEDF3;
|
||||
--pass: #2F6B46;
|
||||
--fail: #9E3223;
|
||||
--hold: #8A6516;
|
||||
--pass-bg: #E0EDE5;
|
||||
--fail-bg: #F6E2DE;
|
||||
--hold-bg: #F5EBD6;
|
||||
|
||||
--f-display: "IBM Plex Sans", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", system-ui, sans-serif;
|
||||
--f-body: "IBM Plex Serif", "Songti SC", "SimSun", Georgia, serif;
|
||||
--f-mono: "IBM Plex Mono", ui-monospace, "SF Mono", Menlo, monospace;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root:not([data-theme="light"]) {
|
||||
--paper: #0E1418; --surface: #161E24; --surface-sunk: #0A1014;
|
||||
--ink: #E4EDF1; --ink-soft: #A3B4BD; --ink-faint: #6E828C;
|
||||
--rule: #2A3840; --rule-soft: #1F2B32;
|
||||
--accent: #4FB6D4; --accent-soft: #12303B;
|
||||
--pass: #6FBE8C; --fail: #E08A78; --hold: #D9B25F;
|
||||
--pass-bg: #142A1E; --fail-bg: #2E1A15; --hold-bg: #2A2311;
|
||||
}
|
||||
}
|
||||
:root[data-theme="dark"] {
|
||||
--paper: #0E1418; --surface: #161E24; --surface-sunk: #0A1014;
|
||||
--ink: #E4EDF1; --ink-soft: #A3B4BD; --ink-faint: #6E828C;
|
||||
--rule: #2A3840; --rule-soft: #1F2B32;
|
||||
--accent: #4FB6D4; --accent-soft: #12303B;
|
||||
--pass: #6FBE8C; --fail: #E08A78; --hold: #D9B25F;
|
||||
--pass-bg: #142A1E; --fail-bg: #2E1A15; --hold-bg: #2A2311;
|
||||
}
|
||||
|
||||
* { box-sizing: border-box; }
|
||||
body { background: var(--paper); color: var(--ink); font-family: var(--f-body); font-size: 16px; line-height: 1.72; -webkit-font-smoothing: antialiased; }
|
||||
.wrap { max-width: 1000px; margin: 0 auto; padding: 56px 28px 120px; display: flex; flex-direction: column; gap: 52px; }
|
||||
|
||||
.masthead { display: flex; flex-direction: column; gap: 18px; }
|
||||
.eyebrow { font-family: var(--f-mono); font-size: 11.5px; letter-spacing: .16em; text-transform: uppercase; color: var(--accent); font-weight: 600; }
|
||||
h1 { font-family: var(--f-display); font-weight: 700; font-size: clamp(30px, 5vw, 46px); line-height: 1.12; letter-spacing: -.022em; margin: 0; text-wrap: balance; }
|
||||
.standfirst { font-size: 18px; color: var(--ink-soft); max-width: 62ch; margin: 0; }
|
||||
.facts { display: grid; grid-template-columns: repeat(auto-fit, minmax(148px, 1fr)); gap: 0; border-top: 1px solid var(--rule); border-bottom: 1px solid var(--rule); margin: 0; }
|
||||
.fact { padding: 14px 18px 14px 0; border-right: 1px solid var(--rule-soft); }
|
||||
.fact:last-child { border-right: 0; }
|
||||
.fact dt { font-family: var(--f-mono); font-size: 10.5px; letter-spacing: .1em; text-transform: uppercase; color: var(--ink-faint); margin: 0 0 3px; }
|
||||
.fact dd { margin: 0; font-family: var(--f-display); font-weight: 600; font-size: 15px; font-variant-numeric: tabular-nums; }
|
||||
|
||||
section { display: flex; flex-direction: column; gap: 20px; }
|
||||
h2 { font-family: var(--f-display); font-weight: 600; font-size: 23px; letter-spacing: -.012em; margin: 0; padding-bottom: 10px; border-bottom: 2px solid var(--ink); display: flex; align-items: baseline; gap: 14px; text-wrap: balance; }
|
||||
h2 .num { font-family: var(--f-mono); font-size: 12px; font-weight: 500; color: var(--accent); letter-spacing: .08em; flex: none; }
|
||||
h3 { font-family: var(--f-display); font-weight: 600; font-size: 16px; margin: 0; letter-spacing: -.005em; }
|
||||
p { margin: 0; max-width: 68ch; }
|
||||
.lede { font-size: 17px; }
|
||||
strong { font-weight: 600; color: var(--ink); }
|
||||
em { font-style: italic; color: var(--ink-soft); }
|
||||
code { font-family: var(--f-mono); font-size: .86em; background: var(--surface-sunk); padding: 1px 5px; border-radius: 2px; color: var(--ink); }
|
||||
a { color: var(--accent); text-decoration-thickness: 1px; text-underline-offset: 2px; }
|
||||
a:focus-visible { outline: 2px solid var(--accent); outline-offset: 3px; }
|
||||
|
||||
.principles { display: flex; flex-direction: column; }
|
||||
.principle { display: grid; grid-template-columns: 34px 1fr; gap: 16px; padding: 18px 0; border-bottom: 1px solid var(--rule-soft); align-items: start; }
|
||||
.principle:first-child { border-top: 1px solid var(--rule-soft); }
|
||||
.principle .mark { font-family: var(--f-mono); font-size: 12px; font-weight: 600; color: var(--accent); padding-top: 4px; }
|
||||
.principle h3 { margin-bottom: 5px; }
|
||||
.principle p { color: var(--ink-soft); font-size: 15.5px; }
|
||||
|
||||
.stack { display: flex; flex-direction: column; gap: 8px; }
|
||||
.layer { display: grid; grid-template-columns: 58px 1fr; background: var(--surface); border: 1px solid var(--rule); border-left: 3px solid var(--accent); }
|
||||
.layer.is-agent { border-left-color: var(--hold); }
|
||||
.layer .tag { font-family: var(--f-mono); font-size: 12px; font-weight: 600; color: var(--accent); padding: 14px 0 14px 14px; letter-spacing: .04em; }
|
||||
.layer.is-agent .tag { color: var(--hold); }
|
||||
.layer .lbody { padding: 13px 16px 15px 4px; display: flex; flex-direction: column; gap: 5px; min-width: 0; }
|
||||
.layer .lbody h3 { font-size: 15.5px; }
|
||||
.layer .lbody p { font-size: 14.5px; color: var(--ink-soft); line-height: 1.6; }
|
||||
.layer .lbody .meta { font-family: var(--f-mono); font-size: 11.5px; color: var(--ink-faint); line-height: 1.55; overflow-x: auto; white-space: nowrap; }
|
||||
|
||||
.code { background: var(--surface-sunk); border: 1px solid var(--rule-soft); border-radius: 3px; padding: 16px 18px; overflow-x: auto; margin: 0; }
|
||||
.code pre { margin: 0; font-family: var(--f-mono); font-size: 12.5px; line-height: 1.68; white-space: pre; color: var(--ink); }
|
||||
.code .c { color: var(--ink-faint); }
|
||||
.code .k { color: var(--accent); font-weight: 500; }
|
||||
.code .n { color: var(--fail); }
|
||||
.code .g { color: var(--pass); }
|
||||
.code figcaption { font-family: var(--f-mono); font-size: 10.5px; letter-spacing: .1em; text-transform: uppercase; color: var(--ink-faint); margin-bottom: 10px; }
|
||||
|
||||
.fsm { width: 100%; height: auto; display: block; min-width: 760px; }
|
||||
.fsm text { font-family: var(--f-mono); font-size: 11px; fill: var(--ink); }
|
||||
.fsm text.small { font-size: 9.5px; fill: var(--ink-faint); }
|
||||
.fsm text.lane { font-size: 10px; fill: var(--accent); font-weight: 600; letter-spacing: .08em; }
|
||||
.fsm .box { fill: var(--surface); stroke: var(--rule); stroke-width: 1; }
|
||||
.fsm .box-pass { fill: var(--pass-bg); stroke: var(--pass); }
|
||||
.fsm .box-fail { fill: var(--fail-bg); stroke: var(--fail); }
|
||||
.fsm .box-hold { fill: var(--hold-bg); stroke: var(--hold); }
|
||||
.fsm .edge { stroke: var(--ink-faint); stroke-width: 1.2; fill: none; }
|
||||
.fsm .edge-em { stroke: var(--accent); stroke-width: 1.6; fill: none; }
|
||||
.fsm .lane-rule { stroke: var(--rule-soft); stroke-width: 1; stroke-dasharray: 3 4; }
|
||||
|
||||
.tablewrap { overflow-x: auto; border: 1px solid var(--rule); border-radius: 3px; background: var(--surface); }
|
||||
table { border-collapse: collapse; width: 100%; font-size: 14px; min-width: 560px; }
|
||||
th { text-align: left; font-family: var(--f-mono); font-size: 10.5px; letter-spacing: .1em; text-transform: uppercase; color: var(--ink-faint); font-weight: 600; padding: 11px 14px; border-bottom: 1px solid var(--rule); white-space: nowrap; }
|
||||
td { padding: 11px 14px; border-bottom: 1px solid var(--rule-soft); vertical-align: top; color: var(--ink-soft); font-variant-numeric: tabular-nums; }
|
||||
tr:last-child td { border-bottom: 0; }
|
||||
td:first-child { color: var(--ink); font-family: var(--f-mono); font-size: 12.5px; white-space: nowrap; }
|
||||
table.gates td:nth-child(2) { color: var(--ink); }
|
||||
table.gates td:nth-child(3) { font-family: var(--f-mono); font-size: 12.5px; white-space: nowrap; color: var(--ink); }
|
||||
|
||||
.note { border-left: 3px solid var(--accent); background: var(--accent-soft); padding: 14px 18px; display: flex; flex-direction: column; gap: 6px; }
|
||||
.note.warn { border-left-color: var(--fail); background: var(--fail-bg); }
|
||||
.note.fix { border-left-color: var(--hold); background: var(--hold-bg); }
|
||||
.note .label { font-family: var(--f-mono); font-size: 10.5px; letter-spacing: .1em; text-transform: uppercase; font-weight: 600; color: var(--accent); }
|
||||
.note.warn .label { color: var(--fail); }
|
||||
.note.fix .label { color: var(--hold); }
|
||||
.note p { font-size: 15px; max-width: none; }
|
||||
|
||||
.items { display: flex; flex-direction: column; }
|
||||
.item { display: flex; flex-direction: column; gap: 4px; padding: 16px 0; border-bottom: 1px solid var(--rule-soft); }
|
||||
.item:first-child { border-top: 1px solid var(--rule-soft); }
|
||||
.item .head { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; }
|
||||
.pill { font-family: var(--f-mono); font-size: 10px; letter-spacing: .08em; text-transform: uppercase; font-weight: 600; padding: 2px 7px; border-radius: 2px; white-space: nowrap; }
|
||||
.pill.block { background: var(--fail-bg); color: var(--fail); }
|
||||
.pill.dec { background: var(--hold-bg); color: var(--hold); }
|
||||
.pill.ok { background: var(--pass-bg); color: var(--pass); }
|
||||
.item p { font-size: 15px; color: var(--ink-soft); }
|
||||
|
||||
.phases { display: grid; grid-template-columns: repeat(auto-fit, minmax(210px, 1fr)); gap: 1px; background: var(--rule); border: 1px solid var(--rule); }
|
||||
.phase { background: var(--surface); padding: 16px 16px 18px; display: flex; flex-direction: column; gap: 6px; }
|
||||
.phase .n { font-family: var(--f-mono); font-size: 11px; font-weight: 600; color: var(--accent); letter-spacing: .08em; }
|
||||
.phase h3 { font-size: 15px; }
|
||||
.phase p { font-size: 14px; color: var(--ink-soft); line-height: 1.6; }
|
||||
|
||||
.qa { display: flex; flex-direction: column; }
|
||||
.qa > div { display: grid; grid-template-columns: 26px 1fr; gap: 14px; padding: 14px 0; border-bottom: 1px solid var(--rule-soft); }
|
||||
.qa > div:first-child { border-top: 1px solid var(--rule-soft); }
|
||||
.qa .q { font-family: var(--f-mono); font-size: 12px; font-weight: 600; color: var(--accent); padding-top: 3px; }
|
||||
.qa h3 { font-size: 15.5px; margin-bottom: 4px; }
|
||||
.qa p { font-size: 15px; color: var(--ink-soft); }
|
||||
|
||||
footer { border-top: 1px solid var(--rule); padding-top: 20px; font-family: var(--f-mono); font-size: 11.5px; color: var(--ink-faint); line-height: 1.8; }
|
||||
|
||||
@media (max-width: 640px) {
|
||||
.wrap { padding: 36px 18px 80px; gap: 40px; }
|
||||
.fact { border-right: 0; border-bottom: 1px solid var(--rule-soft); }
|
||||
.layer { grid-template-columns: 46px 1fr; }
|
||||
}
|
||||
@media (prefers-reduced-motion: reduce) { * { animation: none !important; transition: none !important; } }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
|
||||
<header class="masthead">
|
||||
<div class="eyebrow">model-compile-hbdk / bench · 设计方案 v0.4</div>
|
||||
<h1>BPU 量化轨迹工厂</h1>
|
||||
<p class="standfirst">确定性 harness 作为 oracle 跑完 338 个视觉模型的量化上板,agent 在受控子集上端到端执行,独立核验给出可验证奖励。一个 job 一个目录,agent 只有一个可写区,产物零散落。</p>
|
||||
<dl class="facts">
|
||||
<div class="fact"><dt>模型</dt><dd>338 / 10 类</dd></div>
|
||||
<div class="fact"><dt>优先级</dt><dd>A 166 · B 128 · C 44</dd></div>
|
||||
<div class="fact"><dt>目标板</dt><dd>s100 · s100p · s600</dd></div>
|
||||
<div class="fact"><dt>march</dt><dd>nash-e · m · p</dd></div>
|
||||
<div class="fact"><dt>后端</dt><dd>PTQ + QAT</dd></div>
|
||||
<div class="fact"><dt>训练目标</dt><dd>RLVR</dd></div>
|
||||
</dl>
|
||||
</header>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§0</span>v0.4 改了什么</h2>
|
||||
<p class="lede">v0.3 → v0.4 的四处实质变化,全部来自机器实测或对 v0.3 的追问:</p>
|
||||
<div class="qa">
|
||||
<div><div class="q">A</div><div><h3>验证集层重写:大部分数据集已在 /mnt/data 上,外网带宽分化极大</h3><p>ImageNet val、COCO 2017、Cityscapes、KITTI、ETH3D、SceneFlow、nuScenes 都在别的用户目录里,world-readable。官方 COCO 源实测 26 KB/s 等于不通,HF 8 MB/s、OSS 44 MB/s。数据分 T0 calib / T1 smoke / T2 full 三档,job 默认只跑 T1。§7</p></div></div>
|
||||
<div><div class="q">B</div><div><h3>固定超时废除,改看门狗;重编译一律由 harness 执行</h3><p>不同模型编译时长差两个数量级,固定超时会误杀。现在只在「日志不增长且 CPU 空闲 15 分钟」时判 STALLED。agent 通过 <code>bench stage run</code> 提交编译、<code>bench stage wait</code> 等结果,不在 bash 里直接跑 hb_compile。§8</p></div></div>
|
||||
<div><div class="q">C</div><div><h3>明确「失败之后会怎样」与「重试在哪条轨迹里」</h3><p>错误分类表、<code>bench status</code> 可见性、三层重试的轨迹边界(oracle 瞬态重试不是轨迹;rollout 内自愈是同一条;新 rollout 是全新轨迹,永不串联)。§9</p></div></div>
|
||||
<div><div class="q">D</div><div><h3>3D 占据 23 个模型第一轮只编译不评精度的具体协议</h3><p>G3 显式记 <code>na</code> 而非 <code>unverified</code>,reward 重归一化,校准输入来源分级。§6 末尾</p></div></div>
|
||||
</div>
|
||||
<p>v0.3 的变化(harness 改 oracle、claim/verdict 分离、四道门、stage 分层)保留,不再重复。</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§1</span>三条不可违反的原则</h2>
|
||||
<div class="principles">
|
||||
<div class="principle">
|
||||
<div class="mark">01</div>
|
||||
<div>
|
||||
<h3>harness 是 oracle,不是 agent 的替身</h3>
|
||||
<p>每个 job 先由确定性流水线跑出 ground truth:能不能编译、精度掉多少、板端多快。这份结果既是 benchmark 数据,也是给 agent 打分的参照系。agent 的表现永远相对 oracle 衡量——绝对延时是模型的属性,不是 agent 的功劳。</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="principle">
|
||||
<div class="mark">02</div>
|
||||
<div>
|
||||
<h3>agent 只有一个可写区,一切可信输入只读</h3>
|
||||
<p><code>job.json</code>、评测代码、校准集、验证集、FP32 基线全部在 agent 可写区之外。agent 写 <code>workspace/</code>,越界写入由 extension 拒绝。它说什么不算数,harness 用干净代码重新验它产出的 <code>.hbm</code> 才算数。</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="principle">
|
||||
<div class="mark">03</div>
|
||||
<div>
|
||||
<h3>Append-only:重跑即新 run,永不原地覆盖</h3>
|
||||
<p><code>run_id</code> 冻结。换工具链、改配置、修 bug 后重跑,都开新 run。manifest 记录两套 venv 的 pip freeze 哈希,job 启动前比对,不一致拒跑——否则"append-only"是空话。状态只在 <code>jobs.sqlite</code>,文件系统只存内容。</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§2</span>五层架构</h2>
|
||||
<div class="stack">
|
||||
<div class="layer">
|
||||
<div class="tag">L0</div>
|
||||
<div class="lbody">
|
||||
<h3>清单、矩阵与门限</h3>
|
||||
<p><code>models.csv</code>(338 行,不可变)、<code>matrix.yaml</code>(march × backend × qcfg × mode × policy 的组合与裁剪)、<code>gates.yaml</code>(逐任务通过容差)、<code>budgets.yaml</code>(逐任务类超时)。展开后写入 <code>jobs.sqlite</code>。</p>
|
||||
<div class="meta">mode: oracle | e2e | repair · policy: teacher | student</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="layer">
|
||||
<div class="tag">L1</div>
|
||||
<div class="lbody">
|
||||
<h3>oracle harness(无 agent)</h3>
|
||||
<p>model-level 阶段跑一次进缓存;target-level 阶段每 job 一遍。每阶段独立子目录,落 stdout / stderr / exit code / meta.json / 耗时。产出 <code>oracle/result.json</code>。</p>
|
||||
<div class="meta">model-level: fetch → baseline → export | target-level: calibrate → compile → hbm → board → eval</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="layer is-agent">
|
||||
<div class="tag">L2</div>
|
||||
<div class="lbody">
|
||||
<h3>policy driver(pi)</h3>
|
||||
<p>e2e 模式:给自然语言任务、job.json、只读数据路径,agent 自己走完全流程。repair 模式:给 oracle 的失败现场。每次尝试一个 rollout 目录,K 次独立 rollout 构成 GRPO 的组。</p>
|
||||
<div class="meta">pi --mode json -a --session-dir <rollout>/agent "<task>" > events.jsonl</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="layer">
|
||||
<div class="tag">L3</div>
|
||||
<div class="lbody">
|
||||
<h3>四道门与 verdict</h3>
|
||||
<p>对 agent 产出的 <code>.hbm</code>,用与 oracle 完全相同的只读评测代码重跑 G1–G4,写 <code>verdict.json</code>。与 <code>claim.json</code> 比对,记录是否一致。</p>
|
||||
<div class="meta">G1 compile · G2 numeric · G3 task · G4 perf(optional)</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="layer">
|
||||
<div class="tag">L4</div>
|
||||
<div class="lbody">
|
||||
<h3>数据物化</h3>
|
||||
<p>扫描 rollouts,脱敏、清洗、线性化轨迹,与 verdict 和 oracle 配对,按 reward 配方打分,输出到 <code>datasets/<dataset_id>/</code>。reward 配方版本化,可全量重打分。</p>
|
||||
<div class="meta">→ (prompt, trajectory, verdict, oracle_ref, reward) · 兼容 DemyAgent SFT</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§3</span>目录契约</h2>
|
||||
<p>所有新增物只进 <code>bench/</code>,<code>.horizon/</code> 只读。与 v0.2 相比新增 <code>cache/</code>、<code>data/</code>,job 内拆 <code>oracle/</code> 与 <code>rollouts/</code>。</p>
|
||||
<figure class="code">
|
||||
<figcaption>/mnt/data/ming01.wang/model-compile-hbdk</figcaption>
|
||||
<pre><span class="c">├── AGENTS.md # 已有 · pi 向上遍历时加载</span>
|
||||
<span class="c">├── .horizon/ # 已有 · <span class="n">只读</span></span>
|
||||
<span class="c">├── .agents/skills -> .horizon/skills # 新增软链 · pi 的 skill 发现路径</span>
|
||||
└── <span class="k">bench/</span>
|
||||
├── CONTRACT.md <span class="c"># 本文档落地版</span>
|
||||
├── config.yaml <span class="c"># 资源池并发、端点、板卡地址</span>
|
||||
├── registry/ <span class="c"># 输入 · 不可变 · git 管理</span>
|
||||
│ ├── models.csv
|
||||
│ ├── matrix.yaml <span class="c"># march × backend × qcfg × mode × policy</span>
|
||||
│ ├── gates.yaml <span class="c"># 逐任务通过容差(§6)</span>
|
||||
│ ├── budgets.yaml <span class="c"># 逐任务类估算与看门狗参数(§8)</span>
|
||||
│ ├── datasets.yaml <span class="c"># 每个数据集的有序来源、sha、T1 子集 seed(§7)</span>
|
||||
│ └── boards.yaml <span class="c"># s100→nash-e · s100p→nash-m · s600→nash-p</span>
|
||||
├── <span class="k">data/</span> <span class="c"># <span class="n">只读</span> · 从本机其他目录复制入池 · agent 永不下载(§7)</span>
|
||||
│ ├── pool/<dataset>/<version>/ <span class="c"># 全量 + manifest.json(sha, 数量, 来源路径)</span>
|
||||
│ ├── t1/<dataset>/<seed>.tar <span class="c"># smoke 子集打包,避开 JuiceFS 小文件</span>
|
||||
│ └── calib/<task>/<input_spec>/ <span class="c"># 固定 seed 从 train 抽的 npy,与 val 零交集</span>
|
||||
├── <span class="k">cache/</span> <span class="c"># model-level 产物 · content-addressed · 只读</span>
|
||||
│ ├── weights/<sha256>/
|
||||
│ ├── onnx/<sha256>/
|
||||
│ └── baseline/<model>--<dataset>/ <span class="c"># FP32 指标 + 逐输出张量样本</span>
|
||||
├── harness/ <span class="c"># L1 + L3 代码</span>
|
||||
│ ├── pipeline.py scheduler.py
|
||||
│ ├── stages/ backends/{ptq,qat}/ adapters/
|
||||
│ └── eval/ <span class="c"># 逐任务评测器 · agent 不可写 · verdict 唯一依据</span>
|
||||
├── agent/ <span class="c"># L2 pi 配置</span>
|
||||
│ ├── SYSTEM.md settings.json <span class="c"># compaction 关闭</span>
|
||||
│ ├── prompts/e2e.md repair.md <span class="c"># 任务描述模板</span>
|
||||
│ └── extensions/ <span class="c"># path-guard · claim-card · oe-doc(CLI) · budget-kill</span>
|
||||
├── export/ <span class="c"># L4 · 规则移植自 data_collect,<span class="n">不 import</span></span>
|
||||
│ ├── to_agentrl.py redact.py reward.py
|
||||
├── state/jobs.sqlite <span class="c"># 状态、error_class、耗时历史的唯一真相</span>
|
||||
├── reports/ <span class="c"># bench report 生成的静态看板,按 run_id</span>
|
||||
├── <span class="k">datasets/</span>ds20260914-01/ <span class="c"># 一次导出一个版本 · 记录覆盖的 run 与 reward 配方版本</span>
|
||||
└── <span class="k">runs/</span>r20260914-01/
|
||||
├── manifest.json <span class="c"># 两套 venv 的 pip freeze 哈希 · 板卡固件 · 数据 manifest sha</span>
|
||||
└── <span class="k">det--yolov8n--nash-e--ptq--int8/</span>
|
||||
├── job.json <span class="c"># <span class="n">只读</span> · 输入契约 · 含数据路径与 gate 阈值快照</span>
|
||||
├── <span class="k">oracle/</span> <span class="c"># harness 确定性一遍 → ground truth</span>
|
||||
│ ├── stages/04-calibrate/ 05-compile/ 06-hbm/ 07-board/ 08-eval/
|
||||
│ ├── artifacts/ <span class="c"># .bc .hbm · 命名由 job_id 决定</span>
|
||||
│ └── result.json <span class="c"># 四道门结果 + 原始测量值</span>
|
||||
└── <span class="k">rollouts/</span>00/ 01/ … <span class="c"># 每次 agent 尝试一个目录</span>
|
||||
├── <span class="g">workspace/</span> <span class="c"># <span class="g">agent 唯一可写区</span></span>
|
||||
├── agent/session.jsonl events.jsonl
|
||||
├── claim.json <span class="c"># agent 自报 · 不可信</span>
|
||||
└── verdict.json <span class="c"># L3 独立核验 · reward 唯一来源</span></pre>
|
||||
</figure>
|
||||
<div class="note warn">
|
||||
<div class="label">两个 venv 必须隔离</div>
|
||||
<p>PTQ 用 <code>~/toolchains/oe-3.7.0/venv</code>(hbdk4 <strong>4.7.5</strong> + horizon_tc_ui 3.5.3 + hmct 2.6.5),QAT 用 <code>~/toolchains/oellm2/venv</code>(hbdk4 <strong>4.12.3a2.dev</strong> + horizon_plugin_pytorch 3.3.5)。harness 按 backend 选解释器,<strong>禁止把 backend 判断交给 agent</strong>。板端 UCP 3.15.2 可加载两套产物。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§4</span>命名规范</h2>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>标识</th><th>格式</th><th>示例</th><th>说明</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>run_id</td><td>r<YYYYMMDD>-<seq></td><td>r20260914-01</td><td>工具链指纹在 manifest.json,不进 id——两套 hbdk4 塞一个会说谎</td></tr>
|
||||
<tr><td>job_id</td><td><task>--<model>--<march>--<backend>--<qcfg></td><td>det--yolov8n--nash-e--ptq--int8</td><td>五段全小写,段内 <code>a-z0-9.</code>,段间双横线。既是目录名也是 join key</td></tr>
|
||||
<tr><td>task 前缀</td><td>10 类固定缩写</td><td>cls det seg iseg pose mde stereo flow feat occ</td><td>对应交付文档 T01–T10,不允许新增</td></tr>
|
||||
<tr><td>march</td><td>板型映射</td><td>s100→nash-e · s100p→nash-m · s600→nash-p</td><td>写死在 <code>boards.yaml</code>;默认 qcfg:e/m 走 int8,p 走 fp16+int8</td></tr>
|
||||
<tr><td>rollout</td><td><job_id>/rollouts/<kk></td><td>…/rollouts/03</td><td>两位序号,K 次独立 rollout 构成一组</td></tr>
|
||||
<tr><td>artifact</td><td><job_id>.<ext></td><td>det--yolov8n--nash-e--ptq--int8.hbm</td><td>单独拿出来也不撞名、不丢身份</td></tr>
|
||||
<tr><td>stage 目录</td><td><NN>-<name></td><td>05-compile</td><td>01–03 是 model-level(缓存),04–08 是 target-level</td></tr>
|
||||
<tr><td>dataset_id</td><td>ds<YYYYMMDD>-<seq></td><td>ds20260914-01</td><td>记录覆盖的 run 列表与 reward 配方版本</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§5</span>状态机:job 与 rollout 分开</h2>
|
||||
<p>job 的状态由 oracle 决定;rollout 的状态由 verdict 决定。两者互不覆盖,一个 job 可以 oracle FAIL 而某次 rollout PASS——那正是最有价值的样本。</p>
|
||||
<div class="code" style="padding:22px 18px;">
|
||||
<svg class="fsm" viewBox="0 0 920 330" role="img" aria-label="上排 job 状态机:PENDING 到 ORACLE_RUNNING,分流到 PASS、FAIL 或 UNVERIFIED。下排 rollout 状态机:QUEUED 到 RUNNING 到 VERIFYING,分流到 PASS、FAIL、BLOCKED 或 TIMEOUT。">
|
||||
<defs>
|
||||
<marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="var(--ink-faint)"/></marker>
|
||||
</defs>
|
||||
<text class="lane" x="14" y="26">JOB · 由 oracle 决定</text>
|
||||
<rect class="box" x="14" y="48" width="96" height="36" rx="3"/><text x="62" y="71" text-anchor="middle">PENDING</text>
|
||||
<rect class="box" x="160" y="48" width="150" height="36" rx="3"/><text x="235" y="71" text-anchor="middle">ORACLE_RUNNING</text>
|
||||
<rect class="box box-pass" x="380" y="14" width="88" height="36" rx="3"/><text x="424" y="37" text-anchor="middle">PASS</text>
|
||||
<rect class="box box-fail" x="380" y="66" width="88" height="36" rx="3"/><text x="468" y="89" text-anchor="end" dx="-14">FAIL</text>
|
||||
<rect class="box box-hold" x="380" y="118" width="120" height="36" rx="3"/><text x="440" y="141" text-anchor="middle">UNVERIFIED</text>
|
||||
<path class="edge" d="M 110 66 L 154 66" marker-end="url(#ah)"/>
|
||||
<path class="edge" d="M 310 60 L 345 60 L 345 32 L 374 32" marker-end="url(#ah)"/>
|
||||
<path class="edge" d="M 310 66 L 345 66 L 345 84 L 374 84" marker-end="url(#ah)"/>
|
||||
<path class="edge" d="M 310 72 L 345 72 L 345 136 L 374 136" marker-end="url(#ah)"/>
|
||||
<text class="small" x="520" y="37">四道必选门全过</text>
|
||||
<text class="small" x="520" y="89">任一必选门 fail</text>
|
||||
<text class="small" x="520" y="141">数据缺失、板未接 → 门 = 未判定,不是 PASS</text>
|
||||
|
||||
<line class="lane-rule" x1="14" y1="182" x2="906" y2="182"/>
|
||||
<text class="lane" x="14" y="206">ROLLOUT · 由 verdict 决定 · 每个 job 可有 K 个</text>
|
||||
<rect class="box" x="14" y="228" width="96" height="36" rx="3"/><text x="62" y="251" text-anchor="middle">QUEUED</text>
|
||||
<rect class="box box-hold" x="160" y="228" width="110" height="36" rx="3"/><text x="215" y="251" text-anchor="middle">RUNNING</text>
|
||||
<rect class="box box-hold" x="320" y="228" width="110" height="36" rx="3"/><text x="375" y="251" text-anchor="middle">VERIFYING</text>
|
||||
<rect class="box box-pass" x="500" y="196" width="80" height="32" rx="3"/><text x="540" y="217" text-anchor="middle">PASS</text>
|
||||
<rect class="box box-fail" x="500" y="236" width="80" height="32" rx="3"/><text x="540" y="257" text-anchor="middle">FAIL</text>
|
||||
<rect class="box" x="500" y="276" width="96" height="32" rx="3"/><text x="548" y="297" text-anchor="middle">BLOCKED</text>
|
||||
<rect class="box box-fail" x="620" y="276" width="96" height="32" rx="3"/><text x="668" y="297" text-anchor="middle">TIMEOUT</text>
|
||||
<path class="edge" d="M 110 246 L 154 246" marker-end="url(#ah)"/>
|
||||
<path class="edge" d="M 270 246 L 314 246" marker-end="url(#ah)"/>
|
||||
<path class="edge" d="M 430 240 L 465 240 L 465 212 L 494 212" marker-end="url(#ah)"/>
|
||||
<path class="edge" d="M 430 246 L 465 246 L 465 252 L 494 252" marker-end="url(#ah)"/>
|
||||
<path class="edge" d="M 430 252 L 465 252 L 465 292 L 494 292" marker-end="url(#ah)"/>
|
||||
<path class="edge" d="M 215 264 L 215 292 L 614 292" marker-end="url(#ah)"/>
|
||||
<text class="small" x="600" y="217">claim 与 verdict 一致性另记</text>
|
||||
<text class="small" x="600" y="257">含 claim=PASS 而 verdict=FAIL 的说谎样本</text>
|
||||
<text class="small" x="730" y="297">预算耗尽,外部 kill</text>
|
||||
</svg>
|
||||
</div>
|
||||
<div class="note">
|
||||
<div class="label">BLOCKED 怎么判,现在有机械规则了</div>
|
||||
<p>agent 报 BLOCKED 时,看同一 job 的 oracle:oracle 也在同一阶段以同类错误失败 → agent 判断正确,正分;oracle 通过了 → agent 是误判或放弃,负分。不需要人判,也不需要 LLM 判。</p>
|
||||
</div>
|
||||
|
||||
<h3>job 与 rollout 为什么要分开——用一个例子说</h3>
|
||||
<p>job 是「一个模型 × 一块板 × 一种后端 × 一种量化配置」这个<strong>问题</strong>本身,比如 <code>det--yolov8n--nash-e--ptq--int8</code>。rollout 是 agent 对这个问题的<strong>一次作答</strong>。一个问题可以被答很多次:K 次 e2e、若干次 repair、teacher 答几次、student 答几次。所以:</p>
|
||||
<div class="qa">
|
||||
<div><div class="q">→</div><div><h3>job 只有一个 oracle 结果,rollout 每次一个 verdict</h3><p>oracle 是确定性流水线跑出来的,跑一百次都一样,所以只存一份;它回答「这个问题的标准答案是什么」。verdict 每个 rollout 一份,回答「这次作答对不对、比标准答案好还是差」。把两者塞进一个状态机,要么 oracle 被某次 rollout 覆盖,要么 rollout 的 PASS/FAIL 和 job 的 PASS/FAIL 混成一个字段,都是数据事故。</p></div></div>
|
||||
<div><div class="q">→</div><div><h3>oracle FAIL 而 rollout PASS:最值钱,也最需要复核</h3><p>这种样本意味着 agent 做到了确定性流水线没做到的事——通常是它换了导出方式、改了不支持的算子、调了校准策略。它要么是黄金样本,要么是 harness 有 bug(oracle 本该能过)。所以这类 rollout 自动打上 <code>review</code> 标记,进人工抽查队列,抽查通过后 oracle 的实现也要相应修正,形成 harness 自身的改进回路。</p></div></div>
|
||||
<div><div class="q">→</div><div><h3>oracle PASS 而 rollout FAIL:正常,但要看它错在哪</h3><p>这是 agent 能力不足的样本,reward 为负。verdict 记录它在哪道门失败、error_class 是什么,这决定了它在 SFT 里是负样本还是被丢弃(例如超预算的 TIMEOUT 通常丢弃,说谎的 claim≠verdict 通常保留作对比对)。</p></div></div>
|
||||
<div><div class="q">→</div><div><h3>同一 job 的 oracle 和所有 rollout 共用同一份数据、同一段评测代码</h3><p>否则 delta 不可比。<code>job.json</code> 里钉死 eval 子集 manifest sha 和评测器版本,任何一方跑的时候都校验。</p></div></div>
|
||||
</div>
|
||||
<div class="note warn">
|
||||
<div class="label">一个隐患:oracle 本身可能是错的</div>
|
||||
<p>相对 oracle 计分的前提是 oracle 可信。harness 的 bug 会系统性地把 reward 打歪。三道防线:四道门是绝对判断不依赖 oracle;相对项经过 clip 有界;oracle FAIL / rollout PASS 的样本强制人工抽查。P2 垂直切片阶段必须人肉核对 oracle 的每一个数。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§6</span>什么算通过:四道门</h2>
|
||||
<p class="lede">通过是绝对判断(这个量化模型能不能交付),reward 是相对判断(agent 比 oracle 做得好还是差)。两者都需要,不能混。门的阈值住在 <code>registry/gates.yaml</code>,随 run 快照进 <code>job.json</code>。</p>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>门</th><th>检查什么</th><th>必选</th><th>判定依据</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>G1 compile</td><td>.hbm 存在;<code>hb_model_info</code> 读出的 march、输入 dtype/layout 与 job.json 一致;<code>hb_verifier</code> 通过;编译日志无 ERROR</td><td>是</td><td>全部满足 → pass。march 不一致直接 fail——这是 agent 最容易"作弊"的口子</td></tr>
|
||||
<tr><td>G2 numeric</td><td>量化模型 vs FP32 在固定 N 张校准外样本上,逐输出余弦相似度</td><td>是</td><td>所有输出 ≥ 阈值(默认 0.99,T09 按输出单列)。只报 min,不报均值</td></tr>
|
||||
<tr><td>G3 task</td><td>任务指标:量化实测 − FP32 基线(同一评测器、同一子集、同一预处理)</td><td>是</td><td>退化在逐任务容差内。基线来自 <code>cache/baseline/</code> 自跑,不采信官方数</td></tr>
|
||||
<tr><td>G4 perf</td><td>板端单线程纯推理时延 mean 与 P95;BPU 利用率;DDR 带宽</td><td>否</td><td>仅当 job.json 声明 <code>latency_budget_ms</code> 才判;否则只记录。mean 与 P95 分开存,永不互推</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<p><strong>PASS = 必选门全部 pass。</strong>任一必选门因数据缺失、板未接而无法判定 → 该门记 <code>unverified</code>,job 状态为 UNVERIFIED 而不是 PASS。<code>unverified</code> 与 <code>na</code>(该任务明确不用此门)分开记——这是交付文档"未判定 ≠ 不适用"的直接映射。</p>
|
||||
|
||||
<h3>G3 逐任务容差(建议默认值,可在 gates.yaml 覆盖)</h3>
|
||||
<p>容差来源标在最后一列。凡标"借用"的,交付文档里没有找到该任务的阈值记录,属于本文提议,须在第一个 run 后校准。</p>
|
||||
<div class="tablewrap">
|
||||
<table class="gates">
|
||||
<thead><tr><th>task</th><th>主指标(评测集)</th><th>量化容差</th><th>必须固定的口径</th><th>来源</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>cls T01</td><td>Top-1(ImageNet-1K val);Top-5 只记录</td><td>Δ ≥ −1.0 pp</td><td>resize / crop / 归一化、类别映射;多标签不适用</td><td>S9 MobileNetv2 −0.73 pp</td></tr>
|
||||
<tr><td>det T02</td><td>box mAP@[.5:.95](COCO val2017);AP50 附</td><td>Δ ≥ −1.5 mAP 点</td><td>IoU 匹配、NMS 阈值、pycocotools 版本;开放词汇另固定词表与 embedding</td><td>S2 DOSODv2 目标差 1.3 / 1.0</td></tr>
|
||||
<tr><td>seg T03</td><td>mIoU(ADE20K val / Cityscapes val);附各类 IoU</td><td>Δ ≥ −1.0 pp</td><td>ignore 标签、背景是否参与、输出分辨率与 resize</td><td>S9 Unet −0.44 pp</td></tr>
|
||||
<tr><td>iseg T04</td><td>mask AP(COCO/LVIS);提示分割用固定协议 mIoU</td><td>Δ ≥ −1.5</td><td>掩码 IoU 阈值、每图最大实例数;提示:点/框生成规则、多 mask 选择</td><td>借用 T02</td></tr>
|
||||
<tr><td>pose T05</td><td>OKS AP(COCO keypoints val)</td><td>Δ ≥ −1.5;关键点均偏移 ≤ 2 px</td><td>sigma、可见性、ROI 对齐到原图还是输入图</td><td>S11 1–2 px 可接受;AP 借用</td></tr>
|
||||
<tr><td>mde T06</td><td>AbsRel / RMSE / δ1(KITTI Eigen / NYUv2)</td><td>AbsRel 相对劣化 ≤ 10%;δ1 Δ ≥ −1.0 pp</td><td><strong>必须声明尺度对齐方式</strong>;相对深度先对齐再算;非正值处理</td><td>S12 仅有 0.90 相关性;阈值为提议</td></tr>
|
||||
<tr><td>stereo T07</td><td>EPE / D1-all(KITTI 2015)</td><td>每项相对劣化 ≤ 15%</td><td>逐帧均值口径;有效像素范围;A99 / Infinity 只在客户集上算</td><td>S4 SOW:量化允许 ≤ 15%</td></tr>
|
||||
<tr><td>flow T08</td><td>EPE(2D) / Fl-all(Sintel / KITTI 2015)</td><td>相对劣化 ≤ 15%</td><td>输出上采样与位移缩放;遮挡/无效像素。<strong>光流 EPE ≠ 双目 EPE</strong>,分开存</td><td>借用 T07</td></tr>
|
||||
<tr><td>feat T09</td><td>逐输出余弦相似度;HPatches 单应 AUC</td><td>desc ≥ 0.99;heatmap ≥ 0.92;AUC Δ ≥ −2 pp</td><td>提取与匹配分开评;选点规则与点数</td><td>S14 DFeature 0.9909 / 0.9282</td></tr>
|
||||
<tr><td>occ T10</td><td>占据 IoU / 语义 mIoU(Occ3D-nuScenes)</td><td>Δ ≥ −1.5 pp</td><td>体素大小、空间范围、类别集合、可见性 mask</td><td>S15 无阈值;借用</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="note fix">
|
||||
<div class="label">pp 与相对 % 不是一回事</div>
|
||||
<p>mAP 32.5 → 31.2 是"下降 1.3 个 mAP 点",不是"相对下降 1.3%"。gates.yaml 里每条容差都显式写 <code>kind: abs_pp | rel_pct</code>,评测器按 kind 计算,不允许省略。</p>
|
||||
</div>
|
||||
|
||||
<h3>任务特例:3D 占据 23 个模型,第一轮只编译不评精度</h3>
|
||||
<p>理由:Occ3D 评测依赖 nuScenes 多相机输入 + 标定 + 占据标签,预处理逐模型不同,是所有任务里最重的;而这 23 个模型的价值主要在算子兼容性(BEV pooling、体素化、稀疏卷积),编译与板端两维就能采到。nuScenes Full 已在机器上,第二轮再补 Occ3D 标签。具体做法:</p>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>环节</th><th>第一轮做法</th><th>写进哪里</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>matrix</td><td><code>task: occ</code> 的 job 声明 <code>gates: [G1, G2, G4]</code>,G3 显式 <code>na</code>,reason <code>round1-compile-only</code></td><td>matrix.yaml · job.json</td></tr>
|
||||
<tr><td>校准输入</td><td>三级来源,逐级回退并记录用了哪级:① 模型仓库自带的 demo 样本或 nuScenes 多相机加载器;② 从 nuScenes Full 抽 128 帧、按模型输入规格 resize;③ 输入范围内的合成随机张量。第 ③ 级在 job.json 记 <code>calib.source: synthetic</code>,G2 结果只作参考</td><td>04-calibrate/meta.json</td></tr>
|
||||
<tr><td>G2 数值门</td><td>用与校准同级来源的另一批样本算余弦,阈值放宽到 0.97;synthetic 来源时 G2 记 <code>pass-weak</code>,不阻塞 PASS 但进 verdict</td><td>gates.yaml occ 段</td></tr>
|
||||
<tr><td>G4 板端</td><td>照常测 latency / BPU util / DDR,这是第一轮 occ 最有价值的数字</td><td>07-board</td></tr>
|
||||
<tr><td>PASS 定义</td><td>G1 ∧ G2(含 pass-weak);job 状态 PASS,但 result.json 里 <code>round: 1</code>,第二轮跑 G3 时开新 run,不覆盖</td><td>jobs.sqlite</td></tr>
|
||||
<tr><td>reward</td><td>reward.py 遇 G3=na 时把 G3 的 2.0 权重按比例分给 G1/G2,相对 oracle 的精度项取 0;不因缺门而整体缩水</td><td>export/reward.py</td></tr>
|
||||
<tr><td>agent 提示</td><td>e2e prompt 模板明说「本轮目标:编译通过 + 板端可运行;不评精度」。<code>.horizon</code> 没有 occ 专用 skill,预期 BLOCKED 比例高,这是正常的、有价值的</td><td>agent/prompts/e2e-occ.md</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<p><code>na</code> 与 <code>unverified</code> 在这里的区别很关键:na 是「这一轮设计上就不评」,job 可以 PASS;unverified 是「本该评但没评成」,job 不能 PASS。第二轮补 Occ3D 标签后,occ 的 G3 从 na 变回必选,那是新 run。</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§7</span>数据集层:机器上已有什么,缺什么,怎么分档</h2>
|
||||
<p class="lede">agent 永远不下载数据。可复现(版本会飘)、防作弊(agent 会挑好算的子集)、轨迹干净(网络噪声不进训练数据)。这一节的两条实测事实改写了 v0.3 的假设。</p>
|
||||
|
||||
<h3>事实一:外网带宽分化极大,官方源等于不通</h3>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>源</th><th>实测</th><th>含义</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>aliyun OSS</td><td>43.8 MB/s</td><td>OE 包来源,最快</td></tr>
|
||||
<tr><td>huggingface.co</td><td>8.0 MB/s</td><td>直连 0.32 s,<code>hf</code> CLI 已装,不用换镜像</td></tr>
|
||||
<tr><td>github releases</td><td>5.1 MB/s</td><td>可用</td></tr>
|
||||
<tr><td>images.cocodataset.org</td><td><strong>26 KB/s</strong></td><td>815 MB 要 9 小时。Excel 的"数据集入口"列给的全是这类官方站点,<strong>不能照着下</strong></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<p>所以 <code>datasets.yaml</code> 里每个数据集声明<strong>有序的多个源</strong>:本地 → HF → OSS 镜像 → GitHub → 官方。逐个尝试,成功后校验 sha256。</p>
|
||||
|
||||
<h3>事实二:大部分数据集已经在 /mnt/data 上</h3>
|
||||
<p>JuiceFS 951 TB 可用,写入 482 MB/s。以下都在别的用户目录里,world-readable,实测 <code>ls</code> 通。</p>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>数据集</th><th>位置(/mnt/data/…)</th><th>内容</th><th>入 pool 时要做的</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>ImageNet-1K val</td><td>chao01.wu/Data/Datasets/ImageNet/val_images.tar.gz(6.67 GB);HuggingFace/ImageNet_HF/imagenet/data/validation-*.parquet(14 片全)</td><td>50,000 张 + classes.py</td><td>校验 tar 完整性(旁边有个 .tmp 残片,说明下过两次);解压到 pool;parquet 作第二来源</td></tr>
|
||||
<tr><td>COCO 2017</td><td>chao01.wu/Data/Datasets/COCO2017/val2017/(已解压)+ annotations_trainval2017.zip;guannan.he/coco2017/val2017.zip</td><td>5,000 张 + instances / person_keypoints 标注</td><td>复制 val2017 与两份 json;det · iseg · pose 三类共用</td></tr>
|
||||
<tr><td>Cityscapes</td><td>guannan.he/CityScapes/{leftImg8bit,gtFine}_trainvaltest.zip</td><td>val 500 张 + gtFine labelIds</td><td>只解压 <code>*/val/</code>,其余不动</td></tr>
|
||||
<tr><td>KITTI 2012 / 2015</td><td>qi.xiong/Dataset/StereoDataset/KITTI/data_scene_flow/</td><td>stereo 视差 GT + flow GT,各 200 帧 train</td><td>stereo 与 flow 两个任务共用一份,manifest 分开</td></tr>
|
||||
<tr><td>ETH3D two-view</td><td>qi.xiong/Dataset/StereoDataset/eth3d/</td><td>双目训练/测试对</td><td>stereo 补充集</td></tr>
|
||||
<tr><td>SceneFlow · InStereo2K · FallingThings</td><td>guannan.he/SceneFlow/,qi.xiong/…/StereoDataset/</td><td>合成双目</td><td>备用,第一轮不用</td></tr>
|
||||
<tr><td>nuScenes Full</td><td>qi.xiong/Dataset/AutoDriveDataset/nuScenes/nuScenes_Full</td><td>多相机原始数据</td><td>occ 第一轮只取 128 帧做校准;第二轮补 Occ3D 标签</td></tr>
|
||||
<tr><td>bdd100k</td><td>guannan.he/bdd100k/</td><td>驾驶场景 det / seg</td><td>备用</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="note">
|
||||
<div class="label">复制,不软链</div>
|
||||
<p>别人的目录随时可能被清。全部<strong>复制</strong>进 <code>bench/data/pool/<dataset>/<version>/</code>,复制完 <code>chmod -R a-w</code>,写 manifest.json(样本清单 sha256、数量、来源路径、复制日期)。951 TB 面前,几十 GB 的冗余换来的是与他人目录的零耦合。</p>
|
||||
</div>
|
||||
|
||||
<h3>仍需下载的(合计约 15 GB,HF 速率半小时)</h3>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>数据集</th><th>任务</th><th>规模</th><th>首选源</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>ADE20K val</td><td>seg</td><td>2,000 张,~1 GB</td><td>HF</td></tr>
|
||||
<tr><td>LVIS v1 val 标注</td><td>iseg · 开放词汇</td><td>json ~1 GB,图片复用 COCO</td><td>HF / GitHub</td></tr>
|
||||
<tr><td>NYUv2 test</td><td>mde</td><td>654 张,~4 GB</td><td>HF</td></tr>
|
||||
<tr><td>MPI-Sintel train</td><td>flow</td><td>1,041 ×2,~5 GB</td><td>HF 镜像 → 官方</td></tr>
|
||||
<tr><td>HPatches</td><td>feat</td><td>116 seq,~4 GB</td><td>GitHub / HF</td></tr>
|
||||
<tr><td>Occ3D-nuScenes 标签</td><td>occ 第二轮</td><td>数 GB</td><td>GitHub release</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h3>三档数据,job 默认只跑 T1</h3>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>档</th><th>用途</th><th>规模</th><th>来源</th><th>谁用</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>T0 calib</td><td>PTQ 校准;QAT calibration-only</td><td>128–512 张,按任务</td><td>train 划分固定 seed 抽,与 val 零交集</td><td>每个 job 必用</td></tr>
|
||||
<tr><td>T1 smoke</td><td>G2 数值门 + G3 任务门的<strong>筛</strong></td><td>cls 2,000 · det/iseg/pose 1,000 · seg 500 · 其余全量(本就 ≤ 1k)</td><td>val 固定 seed 分层抽,清单入 git</td><td>每个 job 默认</td></tr>
|
||||
<tr><td>T2 full</td><td>正式数字,G3 的<strong>判</strong></td><td>完整 val</td><td>pool 全量</td><td>只给要出交付数的模型;开新 run</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="note fix">
|
||||
<div class="label">T1 的统计边界</div>
|
||||
<p>500 张评 Top-1,单次测量标准误约 1.9 pp——分辨不出 1 pp 的掉点。但 FP32 与 INT8 是<strong>配对</strong>比较,同一批图,误差高度相关,delta 的标准误由不一致对决定,通常 ~5%:2,000 张时约 0.5 pp,够用;500 张时约 1 pp,与 −1.0 pp 容差同量级,会误判。所以 cls 的 T1 取 2,000 而不是 500,且 T1 上的 G3 容差自动放宽 <code>tol − 2·SE</code>,记 <code>pass-screen</code>;T2 才是 <code>pass</code>。对 reward 来说 T1 的信号足够;对交付来说 T2 才算数。</p>
|
||||
</div>
|
||||
|
||||
<div class="qa">
|
||||
<div><div class="q">→</div><div><h3>校准集细则</h3><p>存 npy,shape 含 batch 维,layout 与 job 输入规格一致,按 <code><task>/<input_spec></code> 组织,同规格模型共用。G2 的数值对比样本从 T1 里另抽,不能用校准图自己验自己。hb_compile 的校准张数影响编译时长,默认 128。</p></div></div>
|
||||
<div><div class="q">→</div><div><h3>注入方式</h3><p><code>job.json.data</code> 写明 calib / T1 / T2 的 pool 路径、version、manifest sha。目录对 agent 只读(权限 + path-guard 双保险)。job 目录里只放一条 <code>data -></code> 软链方便 agent 引用——这不违反"一个 job 一个目录",那条原则约束产出,不约束输入。</p></div></div>
|
||||
<div><div class="q">→</div><div><h3>小文件与 JuiceFS</h3><p>JuiceFS 是 FUSE,<code>max_read=128K</code>,几万张小 JPEG 随机读会慢。T1 子集(≤ 2k 文件)打包成单个 tar 或 npz 放在 pool 里,评测器直接读包;T2 全量走原目录。/home 是本地盘还剩 1.2 TB,热子集可选镜像一份,不是必须。</p></div></div>
|
||||
<div><div class="q">→</div><div><h3>preflight</h3><p><code>pipeline.py --preflight</code> 在跑任何 job 前把 matrix 展开后需要的全部数据集查一遍:pool 里有没有、manifest sha 对不对、T0/T1 子集生成了没有。缺什么一次性报出来。跑到 08-eval 才发现没数据,浪费的是前面七个阶段。</p></div></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§8</span>调度、预算与看门狗</h2>
|
||||
<p>并行数按资源池分开设,一个数字管所有会让板端把 GPU 堵死。耗时不均不靠固定超时解决——<strong>固定超时会误杀慢模型</strong>,v0.3 这一条是错的。</p>
|
||||
<figure class="code">
|
||||
<figcaption>bench/config.yaml(节选)</figcaption>
|
||||
<pre><span class="k">pools</span>:
|
||||
gpu: { workers: 4, gpu_per_worker: 1 } <span class="c"># baseline / eval / qat-calibrate</span>
|
||||
cpu: { workers: 8 } <span class="c"># export / hb_compile(CPU 密集)</span>
|
||||
board: { workers: 1, target: s100 } <span class="c"># 一块板 = 1,严格串行,带锁</span>
|
||||
agent: { workers: 3, rpm: 60 } <span class="c"># 受 LLM 端点限速</span>
|
||||
|
||||
<span class="k">watchdog</span>:
|
||||
stall_s: 900 <span class="c"># 日志不增长 且 进程树 CPU < 5% 持续 15 min → STALLED</span>
|
||||
hard_ceiling_s: 86400 <span class="c"># 24h 纯保险,正常永远碰不到</span>
|
||||
soft_alert: p95x3 <span class="c"># 超过历史 p95 × 3 只告警,不杀</span>
|
||||
|
||||
<span class="k">perf</span>: { warmup: 20, runs: 200, stats: [mean, p95] }
|
||||
|
||||
<span class="k">rollouts</span>: { k: 4, max_turns: 40, max_tokens: 200000, hard_ceiling_s: 43200 }
|
||||
<span class="c"># 真正的预算是 turns 和 tokens;墙钟只是 12h 保险</span>
|
||||
<span class="k">retry</span>: { transient: 2, deterministic: 0 }</pre>
|
||||
</figure>
|
||||
<div class="qa">
|
||||
<div><div class="q">→</div><div><h3>看门狗而不是超时</h3><p>一个阶段只在「日志 15 分钟不增长 且 进程树 CPU 空闲」时才被判 STALLED——真正在算的编译,哪怕跑 6 小时也不会被杀。STALLED 归 transient,重试一次进 <code>long</code> 车道(只有看门狗、没有软预算),再 STALLED 才记 <code>FAIL:stalled</code>。历史 p95 × 3 只用来告警和调度估算,不用来杀。</p></div></div>
|
||||
<div><div class="q">→</div><div><h3>所有重阶段由 harness 执行,agent 只提交和等待</h3><p>agent 不在 bash 里直接跑 <code>hb_compile</code>。它调 <code>bench stage run compile --job … --config workspace/x.yaml</code>,harness 把任务丢进 cpu 池、按 backend 选对 venv、日志落 stage 目录,返回一个 handle;agent 用 <code>bench stage wait <handle></code> 拿结果。四个好处:编译时长与 pi 的 bash 工具超时脱钩;backend/venv 选择不可能被 agent 选错;预算与看门狗对 oracle 和 rollout 一视同仁;日志位置统一。agent 的 bash 仍然可用,用来看日志、改配置、跑小脚本。</p></div></div>
|
||||
<div><div class="q">→</div><div><h3>调度顺序</h3><p>先按 A / B / C 优先级,同级内按估算耗时短作业优先。C 类 44 个走独立低优先级车道,永远不堵 A 类。首轮估算用参数量,之后用 <code>jobs.sqlite</code> 里同模型同阶段的实测耗时——每个 job 跑完都回填。</p></div></div>
|
||||
<div><div class="q">→</div><div><h3>板卡串行</h3><p><code>07-board</code> 只有一个 worker,拿锁才能推 hbm。板掉线归 transient,重试后仍失败 → 该 job 的 G4 记 unverified,不阻塞其他阶段。</p></div></div>
|
||||
<div><div class="q">→</div><div><h3>rollout 预算</h3><p>turns 和 tokens 是真正的预算——它们是 agent 自己能控制的量。墙钟 12h 只防 agent 进程本身挂死。因为编译等待发生在 harness 侧,一个 rollout 等一次 3 小时的编译不会吃掉它的预算。触顶由 <code>budget-kill</code> extension 结束会话,状态 TIMEOUT,轨迹保留。</p></div></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§9</span>失败之后会怎样:可见性、重试、轨迹边界</h2>
|
||||
<p class="lede">没量化成功的模型不是丢了,是一个带分类的数据点。这一节回答三件事:在哪看、重试几次、重试算不算同一条轨迹。</p>
|
||||
|
||||
<h3>在哪看</h3>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>命令</th><th>给什么</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>bench status</td><td>按 state × task × march 的计数表;<code>--fail</code> 只列失败,附 error_class 分布</td></tr>
|
||||
<tr><td>bench show <job_id></td><td>阶段时间线、每阶段 exit code 与耗时、error_class、stderr 末 50 行、rollouts 的 verdict 一览</td></tr>
|
||||
<tr><td>bench annotate <job_id> --note</td><td>人工备注("已知 hbdk 4.7.5 不支持 DeformConv"),只记录不改 verdict</td></tr>
|
||||
<tr><td>bench report</td><td>从 jobs.sqlite 生成静态 HTML 到 <code>bench/reports/<run_id>.html</code>,给不上机器的人看</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h3>error_class:失败必须有名字</h3>
|
||||
<p>每个 FAIL 由正则分类器从 stderr 打一个标签,存进 jobs.sqlite 和 result.json。它同时喂三个地方:看板、repair 模式的 prompt、BLOCKED 判定里的"同类错误"。</p>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>error_class</th><th>含义</th><th>性质</th><th>处置</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>unsupported_op</td><td>算子不在 march 支持列表</td><td>确定性</td><td>不重试;进 repair 队列;最有价值的一类</td></tr>
|
||||
<tr><td>shape_infer</td><td>动态 shape / 维度推导失败</td><td>确定性</td><td>不重试;进 repair 队列</td></tr>
|
||||
<tr><td>export_fail</td><td>ONNX 导出或 fx trace 失败</td><td>确定性</td><td>不重试;model-level,一次影响该模型所有 job</td></tr>
|
||||
<tr><td>calib_nan</td><td>校准出 NaN / 溢出</td><td>确定性</td><td>不重试;进 repair 队列</td></tr>
|
||||
<tr><td>verifier_mismatch</td><td>hb_verifier 或 G2 数值不一致</td><td>确定性</td><td>不重试;进 repair 队列</td></tr>
|
||||
<tr><td>oom</td><td>GPU / 主机内存不足</td><td>瞬态</td><td>重试 ≤ 2,第二次降并发</td></tr>
|
||||
<tr><td>board_offline</td><td>板子失联</td><td>瞬态</td><td>重试 ≤ 2;仍失败 → G4 unverified</td></tr>
|
||||
<tr><td>stalled</td><td>看门狗判定挂死</td><td>瞬态</td><td>重试 1 次进 long 车道</td></tr>
|
||||
<tr><td>unknown</td><td>没匹配到规则</td><td>—</td><td>不重试;进人工分类队列,规则库随之增长</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h3>重试在哪一层,是不是同一条轨迹</h3>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>层</th><th>谁重试</th><th>触发</th><th>轨迹归属</th><th>次数</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>oracle 阶段</td><td>harness</td><td>瞬态 error_class</td><td><strong>不是轨迹</strong>,没有 agent 参与;meta.json 记 attempt 号</td><td>≤ 2</td></tr>
|
||||
<tr><td>rollout 内</td><td>agent 自己</td><td>它看到编译报错,改配置再提交</td><td><strong>同一条轨迹</strong>。这就是要采的诊断过程;HORIZON.md 的"同一方法最多重试 1 次"在这里作为 prompt 约束生效</td><td>受 max_turns 约束</td></tr>
|
||||
<tr><td>新 rollout</td><td>harness 起新进程</td><td>matrix 声明 k 次</td><td><strong>全新轨迹</strong>:全新上下文,同一初始状态,不知道其他 rollout 存在</td><td>k</td></tr>
|
||||
<tr><td>repair 模式</td><td>harness 起新进程</td><td>oracle FAIL</td><td>全新轨迹,输入是 <strong>oracle</strong> 的失败现场,不是某次 rollout 的</td><td>按 matrix</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="note warn">
|
||||
<div class="label">三条硬边界</div>
|
||||
<p><strong>不串联</strong>:rollout k+1 永远看不到 rollout k 的历史,否则轨迹不独立,GRPO 的组内比较失去意义。<strong>不续跑</strong>:TIMEOUT 的 rollout 是一条完整轨迹,reward 记负;想要更长就在下一个 run 调大 max_turns。<strong>repair 不吃 rollout 的失败</strong>:只吃 oracle 的,保证 repair 轨迹之间也独立。</p>
|
||||
</div>
|
||||
|
||||
<h3>没量化成功的模型最后会怎样</h3>
|
||||
<p>oracle FAIL 的 job 状态就停在 FAIL,带 error_class,不影响其他 job。它会:出现在 <code>bench status --fail</code>;按 matrix 进入 repair 队列生成轨迹;被 BLOCKED 规则引用。人可以 annotate 它。它不会被自动重跑——重跑意味着新 run,那是人的决定。第一轮预期 FAIL 率不低,尤其局部特征、光流、占据三类;这是数据,不是事故。</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§10</span>pi 作为 policy driver</h2>
|
||||
<p>pi 是可替换的一层。<code>job.json</code>、<code>claim.json</code>、<code>verdict.json</code> 与 driver 无关,将来换成 verl 的 agent loop 或 RPC 模式下的工具执行器,环境契约不变。</p>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>要点</th><th>做法</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>启动命令</td><td><code>pi --mode json -a --session-dir <rollout>/agent "<task>" > events.jsonl</code>。<strong>不是</strong> <code>-p --mode json</code>,两者是互斥模式;<code>-a</code> 让非交互模式加载项目级 extension</td></tr>
|
||||
<tr><td>轨迹源</td><td>用 events.jsonl 最后一条 <code>agent_end.messages[]</code>,它已是线性化的完整对话;session.jsonl 是树,只做备份</td></tr>
|
||||
<tr><td>compaction</td><td><code>.pi/settings.json</code> 里 <code>"compaction": {"enabled": false}</code>(已核对是正式设置项)。训练数据里出现 <code>CompactionSummaryMessage</code> 是脏的;上下文靠 max_turns 控制</td></tr>
|
||||
<tr><td>skills 发现</td><td>不用软链。<code>.pi/settings.json</code> 的 <code>skills</code> 数组支持 glob:<code>"skills": ["../.horizon/skills/*/*", "../.horizon/skills/horizon-router"]</code>,路径相对 <code>.pi/</code> 解析。嵌套两层的问题由 glob 直接解决</td></tr>
|
||||
<tr><td>重阶段</td><td>agent 不直接跑 hb_compile / 板端推理,通过 <code>bench stage run / wait</code> 提交给 harness(§8)。pi bash 工具的默认超时<strong>未查到文档</strong>,P1 实测;但因为编译已不在 bash 里跑,这个值只影响小命令</td></tr>
|
||||
<tr><td>工具白名单</td><td><code>--tools read,edit,write,bash,grep</code>;write/edit 受 path-guard 限制在 workspace/</td></tr>
|
||||
<tr><td>extensions</td><td><code>path-guard</code>(越界拒绝)· <code>claim-card</code>(强制结构化汇报,写 claim.json;未调用则 claim 缺席,小额扣分)· <code>oe-doc</code>(把 oe-mcp 包成 CLI,pi 不支持 MCP)· <code>budget-kill</code></td></tr>
|
||||
<tr><td>模型端点</td><td><code>policy: teacher</code> 走 Claude/GPT 直连;<code>policy: student</code> 走 <code>~/.pi/agent/models.json</code> 指向你的 vLLM。matrix 里显式选,不混采</td></tr>
|
||||
<tr><td>环境变量</td><td>agent 的 bash 只暴露 job 目录路径与 <code>PI_SESSION_ID</code>。<strong>板卡凭证不进 env</strong>,板端推送由 harness 在 agent 进程外完成</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§11</span>RLVR:verdict 定 reward,相对 oracle 计分</h2>
|
||||
<p>result.json / verdict.json 只存原始测量值,reward 在 L4 离线算,配方版本化在 <code>export/reward.py</code>。</p>
|
||||
<figure class="code">
|
||||
<figcaption>rollouts/03/verdict.json</figcaption>
|
||||
<pre>{
|
||||
<span class="k">"job_id"</span>: "det--yolov8n--nash-e--ptq--int8", <span class="k">"rollout"</span>: 3, <span class="k">"mode"</span>: "e2e", <span class="k">"policy"</span>: "teacher",
|
||||
<span class="k">"gates"</span>: {
|
||||
"G1": { "verdict": "pass", "march_ok": true, "verifier_ok": true },
|
||||
"G2": { "verdict": "pass", "cos_min": 0.9961, "threshold": 0.99 },
|
||||
"G3": { "verdict": "pass", "metric": "mAP@[.5:.95]", "fp32": 0.372, "quant": 0.361,
|
||||
"delta": -0.011, "kind": "abs_pp", "tolerance": -0.015 },
|
||||
"G4": { "verdict": "na", "latency_ms_mean": 4.21, "latency_ms_p95": 4.60, "bpu_util": 0.78 }
|
||||
},
|
||||
<span class="k">"terminal"</span>: "PASS",
|
||||
<span class="k">"claim_consistent"</span>: true,
|
||||
<span class="k">"oracle_ref"</span>: { "terminal": "PASS", "G3_delta": -0.013, "latency_ms_mean": 4.35 },
|
||||
<span class="k">"cost"</span>: { "turns": 17, "tokens": 91233, "wall_s": 612 }
|
||||
}</pre>
|
||||
</figure>
|
||||
<figure class="code">
|
||||
<figcaption>export/reward.py · v0 配方(示意,权重待第一批数据后校准)</figcaption>
|
||||
<pre>r = 1.0·[G1 pass] + 1.0·[G2 pass] + 2.0·[G3 pass] <span class="c"># 绝对门</span>
|
||||
r += 1.0·clip((Δ_agent − Δ_oracle) / |tolerance|, −1, +1) <span class="c"># 相对 oracle 的精度</span>
|
||||
r += 0.5·clip((lat_oracle − lat_agent) / lat_oracle, −1, +1) <span class="c"># 相对 oracle 的延时</span>
|
||||
r −= 0.5·(turns / max_turns) + 0.5·(tokens / max_tokens) <span class="c"># 代价</span>
|
||||
r −= 2.0·[claim ≠ verdict] <span class="c"># 说谎</span>
|
||||
BLOCKED: +1.0 若 oracle 同阶段同类失败;否则 −2.0
|
||||
TIMEOUT: −1.0</pre>
|
||||
</figure>
|
||||
<div class="qa">
|
||||
<div><div class="q">→</div><div><h3>teacher 与 student 是两种数据</h3><p>teacher(Claude/GPT)轨迹 + verdict 筛选 → SFT / 拒绝采样微调。student(你的模型)轨迹 → on-policy。两者 prompt 模板相同,matrix 里 policy 维度分开采,导出时分开落 dataset。</p></div></div>
|
||||
<div><div class="q">→</div><div><h3>真 on-policy GRPO 时,rollout 驱动者会变</h3><p>GRPO 需要 rollout 时的 logprobs,pi session 只有 messages。到那一步,verl/AgentRL 自己的循环做 rollout,pi 退化为 RPC 模式的工具执行器或被替换。本方案的持久资产是环境契约(§3 目录 + §6 门 + verdict.json),不是 pi。</p></div></div>
|
||||
<div><div class="q">→</div><div><h3>采样去重</h3><p>同一模型跑 nash-e/m/p 的 e2e 轨迹高度重复。e2e 模式按 (model, backend) 只采主 march(默认 nash-e),全 march 只在 oracle 模式铺。</p></div></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§12</span>下游边界:搬规则,不搬代码</h2>
|
||||
<p>数据全部留在本项目。<code>~/data_collect</code> 与 <code>data-litellm-1.0</code> 零耦合。</p>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>data_collect 的东西</th><th>处置</th><th>原因</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>rebuild / probe / backfill / consumer</td><td>不要</td><td>网关 raw 日志重建会话边界;pi 的 agent_end 已是完整线性对话</td></tr>
|
||||
<tr><td>judge_sessions.py</td><td>不要</td><td>LLM 主观打分。我们有 verdict,不需要也不应该让模型判</td></tr>
|
||||
<tr><td>clean_for_agentrl.py</td><td><strong>只搬规则 R1–R8</strong></td><td>双方言归一、thinking 剥离、system 首条、悬空尾轮截断——踩过的坑</td></tr>
|
||||
<tr><td>redact_archive.py</td><td><strong>搬</strong>(v0.2 漏了)</td><td>agent 的 bash 输出会进轨迹,路径、主机名、任何凭证都要脱敏</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<p>落地为 <code>bench/export/{to_agentrl,redact,reward}.py</code>:独立脚本,不 import 那边任何东西。那套代码 <code>ROOT</code> 硬编码指向 litellm 目录,一 import 两个项目的路径与数据版本就缠死。</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§13</span>待决与阻塞项</h2>
|
||||
<div class="items">
|
||||
<div class="item">
|
||||
<div class="head"><span class="pill ok">已解除</span><h3>验证集落盘(v0.3 的头号阻塞)</h3></div>
|
||||
<p>ImageNet val、COCO、Cityscapes、KITTI、ETH3D、nuScenes 都在 /mnt/data 上(§7 表)。剩 ADE20K / LVIS 标注 / NYUv2 / Sintel / HPatches 约 15 GB 走 HF,半小时。HF token 不再需要。</p>
|
||||
</div>
|
||||
<div class="item">
|
||||
<div class="head"><span class="pill dec">待验证</span><h3>ImageNet val tar 的完整性</h3></div>
|
||||
<p><code>val_images.tar.gz</code> 旁边躺着一个 1.5 GB 的 <code>.tmp</code> 残片,说明下过两次。入 pool 前 <code>tar -tzf | wc -l</code> 必须等于 50000,或者直接用 HF parquet 那 14 片作首选源,二者取其一校验通过即可。</p>
|
||||
</div>
|
||||
<div class="item">
|
||||
<div class="head"><span class="pill block">要修</span><h3>.env.oe-package 里的死路径</h3></div>
|
||||
<p><code>PYTHON_BIN</code> 指向 <code><项目根>/oellm2/venv/bin/python</code>,项目根下没有 <code>oellm2/</code>——QAT venv 实际在 <code>~/toolchains/oellm2/venv</code>。agent 按规则会读这个文件。P1 第一件事。</p>
|
||||
</div>
|
||||
<div class="item">
|
||||
<div class="head"><span class="pill dec">待验证</span><h3>pi bash 工具的默认超时</h3></div>
|
||||
<p>settings.md 里没有这一项。因为重编译已改由 harness 执行,它只影响 agent 跑的小命令;但 <code>bench stage wait</code> 本身是长阻塞,需要确认它不会被 bash 工具掐断——不行就改成轮询式 <code>wait --poll 60</code>。</p>
|
||||
</div>
|
||||
<div class="item">
|
||||
<div class="head"><span class="pill dec">待确认</span><h3>rdk-sandbox 是哪块板</h3></div>
|
||||
<p>映射已定:s100 → nash-e,s100p → nash-m,s600 → nash-p。沙箱那块具体是哪个,板上 <code>hrut_somstatus</code> 或 <code>/proc/device-tree/model</code> 看一眼。改密钥登录,口令用完轮换。</p>
|
||||
</div>
|
||||
<div class="item">
|
||||
<div class="head"><span class="pill dec">待定</span><h3>teacher 模型选谁、student 端点在哪</h3></div>
|
||||
<p>决定第一批数据的性质。建议 P3 先用 teacher 跑通闭环,student 端点等 vLLM 起来再接。</p>
|
||||
</div>
|
||||
<div class="item">
|
||||
<div class="head"><span class="pill ok">已解除</span><h3>pi 对嵌套 skills 目录的发现</h3></div>
|
||||
<p>settings.json 的 <code>skills</code> 数组支持 glob,<code>"../.horizon/skills/*/*"</code> 直接覆盖两层嵌套,不需要软链,也不需要逐个 <code>--skill</code>。</p>
|
||||
</div>
|
||||
<div class="item">
|
||||
<div class="head"><span class="pill ok">已解除</span><h3>PTQ 工具链就位</h3></div>
|
||||
<p>OE 3.7.0 s100/s600 包已装,TinyNet → hb_compile → .hbm 实测跑通,HBRT4 加载校验通过。338 个公开权重走标准 PTQ。</p>
|
||||
</div>
|
||||
<div class="item">
|
||||
<div class="head"><span class="pill dec">留意</span><h3>Board-Agent 三个中间产物已被删</h3></div>
|
||||
<p><code>model-codegen-info.json</code>、<code>qconfig_setting/</code>、<code>tmp_1_original_float_perf.json</code>,可再生。记在这里防误判。</p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§14</span>实施顺序</h2>
|
||||
<div class="phases">
|
||||
<div class="phase">
|
||||
<div class="n">P1 · 地基</div>
|
||||
<h3>骨架、契约、数据入池</h3>
|
||||
<p>建 <code>bench/</code>、写 CONTRACT.md、修死路径、Excel → models.csv、写 gates/budgets/boards/datasets.yaml、装 pi + settings glob 验证 skills、<strong>COCO / ImageNet / Cityscapes / KITTI 从本机复制入池并校验</strong>,其余 15 GB 走 HF,<code>--preflight</code> 全绿。产出:空转的 pipeline.py。</p>
|
||||
</div>
|
||||
<div class="phase">
|
||||
<div class="n">P2 · oracle 切片</div>
|
||||
<h3>yolov8n + resnet18 各打穿八阶段</h3>
|
||||
<p>det 用 COCO T1,cls 用 ImageNet T1(现在都在本机)。PTQ、nash-e。<strong>人肉核对 oracle 的每一个数</strong>——这是相对 oracle 计分的信任基础。产出:两份 oracle/result.json,看门狗与 bench stage CLI 跑通。</p>
|
||||
</div>
|
||||
<div class="phase">
|
||||
<div class="n">P3 · 闭环</div>
|
||||
<h3>e2e + repair 各跑通一次</h3>
|
||||
<p>teacher 模型。e2e 跑 yolov8n;repair 拿一个 oracle 必失败的模型。验证 path-guard、claim/verdict 分离、budget-kill、导出脱敏。产出:第一批带 reward 的轨迹。</p>
|
||||
</div>
|
||||
<div class="phase">
|
||||
<div class="n">P4 · 横向</div>
|
||||
<h3>TorchVision 51 + COCO 系 ~60</h3>
|
||||
<p>oracle 全 march;e2e 每类 6–10 个。压测资源池、板卡锁、耗时估算回填。产出:第一个完整 run、失败率分布、gates 容差校准。</p>
|
||||
</div>
|
||||
<div class="phase">
|
||||
<div class="n">P5 · 全量</div>
|
||||
<h3>A 类 166 → B → C</h3>
|
||||
<p>局部特征 28、3D 占据 23、光流 14 算子特殊,是失败重灾区也是轨迹金矿,放最后。C 类走低优先级车道。</p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<footer>
|
||||
环境指纹 · instance-mzr1k6rw · 4 × RTX 5090 · .horizon skill pack v0.2.0<br>
|
||||
PTQ 后端 ~/toolchains/oe-3.7.0/venv · horizon_tc_ui 3.5.3 · hmct 2.6.5 · hbdk4 4.7.5<br>
|
||||
QAT 后端 ~/toolchains/oellm2/venv · horizon_plugin_pytorch 3.3.5+cu128.torch280 · hbdk4 4.12.3a2.dev+cd3ae44 · hbm_infer 3.15.3<br>
|
||||
板端 rdk-sandbox · UCP 3.15.2 · s100→nash-e · s100p→nash-m · s600→nash-p<br>
|
||||
存储 JuiceFS:robotwin-archive 951 TB 可用 · /home 本地 1.2 TB · 带宽 OSS 43.8 / HF 8.0 / GitHub 5.1 MB/s · COCO 官方 26 KB/s<br>
|
||||
G3 容差来源 · 视觉模型交付指标说明 2026-09-14 · S2 S4 S9 S11 S12 S14 S15
|
||||
</footer>
|
||||
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
460
docs/BPU量化轨迹工厂-闭环方法论.html
Normal file
460
docs/BPU量化轨迹工厂-闭环方法论.html
Normal file
@ -0,0 +1,460 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<style>html{color-scheme:light dark}body{margin:0}img{max-width:100%}[hidden]{display:none!important}</style>
|
||||
<title>闭环走通的判据</title>
|
||||
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500;600&family=IBM+Plex+Sans:wght@400;500;600;700&family=IBM+Plex+Serif:ital,wght@0,400;0,500;1,400&display=swap">
|
||||
<style>
|
||||
:root {
|
||||
--paper: #EEF2F4;
|
||||
--surface: #FFFFFF;
|
||||
--surface-sunk: #E3EAEE;
|
||||
--ink: #11181D;
|
||||
--ink-soft: #495861;
|
||||
--ink-faint: #7C8C95;
|
||||
--rule: #C9D6DC;
|
||||
--rule-soft: #DCE6EA;
|
||||
--accent: #0B6E8A;
|
||||
--accent-soft: #DCEDF3;
|
||||
--pass: #2F6B46;
|
||||
--fail: #9E3223;
|
||||
--hold: #8A6516;
|
||||
--pass-bg: #E0EDE5;
|
||||
--fail-bg: #F6E2DE;
|
||||
--hold-bg: #F5EBD6;
|
||||
|
||||
--f-display: "IBM Plex Sans", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", system-ui, sans-serif;
|
||||
--f-body: "IBM Plex Serif", "Songti SC", "SimSun", Georgia, serif;
|
||||
--f-mono: "IBM Plex Mono", ui-monospace, "SF Mono", Menlo, monospace;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root:not([data-theme="light"]) {
|
||||
--paper: #0E1418; --surface: #161E24; --surface-sunk: #0A1014;
|
||||
--ink: #E4EDF1; --ink-soft: #A3B4BD; --ink-faint: #6E828C;
|
||||
--rule: #2A3840; --rule-soft: #1F2B32;
|
||||
--accent: #4FB6D4; --accent-soft: #12303B;
|
||||
--pass: #6FBE8C; --fail: #E08A78; --hold: #D9B25F;
|
||||
--pass-bg: #142A1E; --fail-bg: #2E1A15; --hold-bg: #2A2311;
|
||||
}
|
||||
}
|
||||
:root[data-theme="dark"] {
|
||||
--paper: #0E1418; --surface: #161E24; --surface-sunk: #0A1014;
|
||||
--ink: #E4EDF1; --ink-soft: #A3B4BD; --ink-faint: #6E828C;
|
||||
--rule: #2A3840; --rule-soft: #1F2B32;
|
||||
--accent: #4FB6D4; --accent-soft: #12303B;
|
||||
--pass: #6FBE8C; --fail: #E08A78; --hold: #D9B25F;
|
||||
--pass-bg: #142A1E; --fail-bg: #2E1A15; --hold-bg: #2A2311;
|
||||
}
|
||||
|
||||
* { box-sizing: border-box; }
|
||||
body { background: var(--paper); color: var(--ink); font-family: var(--f-body); font-size: 16px; line-height: 1.72; -webkit-font-smoothing: antialiased; }
|
||||
.wrap { max-width: 1000px; margin: 0 auto; padding-block: 56px 110px; padding-inline: 24px; display: flex; flex-direction: column; gap: 52px; }
|
||||
|
||||
.masthead { display: flex; flex-direction: column; gap: 16px; }
|
||||
.eyebrow { font-family: var(--f-mono); font-size: 11.5px; letter-spacing: .16em; text-transform: uppercase; color: var(--accent); font-weight: 600; }
|
||||
h1 { font-family: var(--f-display); font-weight: 700; font-size: clamp(30px, 5vw, 44px); line-height: 1.12; letter-spacing: -.022em; margin: 0; text-wrap: balance; }
|
||||
.standfirst { font-size: 18px; color: var(--ink-soft); max-width: 60ch; margin: 0; }
|
||||
.facts { display: grid; grid-template-columns: repeat(auto-fit, minmax(140px, 1fr)); border-top: 1px solid var(--rule); border-bottom: 1px solid var(--rule); margin: 0; }
|
||||
.fact { padding: 13px 18px 13px 0; border-right: 1px solid var(--rule-soft); }
|
||||
.fact:last-child { border-right: 0; }
|
||||
.fact dt { font-family: var(--f-mono); font-size: 10.5px; letter-spacing: .1em; text-transform: uppercase; color: var(--ink-faint); margin: 0 0 3px; }
|
||||
.fact dd { margin: 0; font-family: var(--f-display); font-weight: 600; font-size: 15px; font-variant-numeric: tabular-nums; }
|
||||
|
||||
section { display: flex; flex-direction: column; gap: 20px; }
|
||||
h2 { font-family: var(--f-display); font-weight: 600; font-size: 23px; letter-spacing: -.012em; margin: 0; padding-bottom: 10px; border-bottom: 2px solid var(--ink); display: flex; align-items: baseline; gap: 14px; text-wrap: balance; }
|
||||
h2 .num { font-family: var(--f-mono); font-size: 12px; font-weight: 500; color: var(--accent); letter-spacing: .08em; flex: none; }
|
||||
h3 { font-family: var(--f-display); font-weight: 600; font-size: 16px; margin: 0; letter-spacing: -.005em; text-wrap: balance; }
|
||||
p { margin: 0; max-width: 66ch; }
|
||||
.lede { font-size: 17px; }
|
||||
strong { font-weight: 600; color: var(--ink); }
|
||||
em { font-style: italic; color: var(--ink-soft); }
|
||||
code { font-family: var(--f-mono); font-size: .84em; background: var(--surface-sunk); padding: 1px 5px; border-radius: 2px; color: var(--ink); overflow-wrap: anywhere; }
|
||||
|
||||
.stack { display: flex; flex-direction: column; gap: 8px; }
|
||||
.layer { display: grid; grid-template-columns: 62px 1fr; background: var(--surface); border: 1px solid var(--rule); border-left: 3px solid var(--accent); }
|
||||
.layer.is-agent { border-left-color: var(--hold); }
|
||||
.layer .tag { font-family: var(--f-mono); font-size: 11.5px; font-weight: 600; color: var(--accent); padding: 15px 0 14px 14px; letter-spacing: .04em; }
|
||||
.layer.is-agent .tag { color: var(--hold); }
|
||||
.layer .lbody { padding: 13px 18px 15px 4px; display: flex; flex-direction: column; gap: 5px; min-width: 0; }
|
||||
.layer .lbody h3 { font-size: 15.5px; }
|
||||
.layer .lbody p { font-size: 14.5px; color: var(--ink-soft); }
|
||||
|
||||
.inv { display: flex; flex-direction: column; }
|
||||
.inv-row { display: grid; grid-template-columns: 46px 1fr; gap: 16px; padding: 17px 0; border-bottom: 1px solid var(--rule-soft); align-items: start; }
|
||||
.inv-row:first-child { border-top: 1px solid var(--rule-soft); }
|
||||
.inv-row .mark { font-family: var(--f-mono); font-size: 12px; font-weight: 600; color: var(--accent); padding-top: 4px; letter-spacing: .04em; }
|
||||
.inv-row .body { display: flex; flex-direction: column; gap: 6px; min-width: 0; }
|
||||
.inv-row p { font-size: 15px; color: var(--ink-soft); }
|
||||
.counter { font-size: 14px; color: var(--ink-soft); border-left: 2px solid var(--fail); padding-left: 12px; background: var(--fail-bg); padding-block: 7px; padding-right: 12px; }
|
||||
.counter b { font-family: var(--f-mono); font-size: 11.5px; letter-spacing: .08em; text-transform: uppercase; color: var(--fail); font-weight: 600; display: block; margin-bottom: 1px; }
|
||||
|
||||
.tablewrap { overflow-x: auto; border: 1px solid var(--rule); background: var(--surface); }
|
||||
table { border-collapse: collapse; width: 100%; font-size: 14.5px; font-family: var(--f-display); }
|
||||
th, td { text-align: left; padding: 10px 14px; border-bottom: 1px solid var(--rule-soft); vertical-align: top; }
|
||||
thead th { font-family: var(--f-mono); font-size: 10.5px; letter-spacing: .1em; text-transform: uppercase; color: var(--ink-faint); font-weight: 600; border-bottom: 1px solid var(--rule); white-space: nowrap; }
|
||||
tbody tr:last-child td { border-bottom: 0; }
|
||||
td.num-cell { font-variant-numeric: tabular-nums; text-align: right; white-space: nowrap; }
|
||||
|
||||
.pill { font-family: var(--f-mono); font-size: 10.5px; font-weight: 600; letter-spacing: .06em; text-transform: uppercase; padding: 2px 7px; border-radius: 2px; white-space: nowrap; display: inline-block; }
|
||||
.pill.ok { background: var(--pass-bg); color: var(--pass); }
|
||||
.pill.no { background: var(--fail-bg); color: var(--fail); }
|
||||
.pill.part { background: var(--hold-bg); color: var(--hold); }
|
||||
|
||||
.checklist { display: flex; flex-direction: column; gap: 7px; }
|
||||
.check { display: grid; grid-template-columns: 54px 1fr auto; gap: 14px; align-items: start; background: var(--surface); border: 1px solid var(--rule); padding: 12px 14px; }
|
||||
.check .id { font-family: var(--f-mono); font-size: 12px; font-weight: 600; color: var(--accent); }
|
||||
.check .txt { display: flex; flex-direction: column; gap: 3px; min-width: 0; }
|
||||
.check .txt p { font-size: 14px; color: var(--ink-soft); }
|
||||
.check .txt h3 { font-size: 14.5px; }
|
||||
|
||||
.seq { display: flex; flex-direction: column; }
|
||||
.step { display: grid; grid-template-columns: 34px 1fr; gap: 16px; padding: 16px 0; border-bottom: 1px solid var(--rule-soft); }
|
||||
.step:first-child { border-top: 1px solid var(--rule-soft); }
|
||||
.step .n { font-family: var(--f-mono); font-size: 13px; font-weight: 600; color: var(--ink-faint); padding-top: 2px; font-variant-numeric: tabular-nums; }
|
||||
.step .sbody { display: flex; flex-direction: column; gap: 7px; min-width: 0; }
|
||||
.step .sbody p { font-size: 14.5px; color: var(--ink-soft); }
|
||||
.accept { font-family: var(--f-mono); font-size: 12.5px; color: var(--ink); background: var(--accent-soft); padding: 7px 11px; border-left: 2px solid var(--accent); overflow-wrap: anywhere; line-height: 1.6; }
|
||||
|
||||
.callout { background: var(--surface); border: 1px solid var(--rule); border-top: 3px solid var(--accent); padding: 20px 22px; display: flex; flex-direction: column; gap: 9px; }
|
||||
.callout h3 { font-size: 17px; }
|
||||
|
||||
.note { font-size: 14px; color: var(--ink-faint); max-width: 70ch; }
|
||||
footer { border-top: 1px solid var(--rule); padding-top: 18px; display: flex; flex-direction: column; gap: 6px; }
|
||||
footer p { font-size: 13.5px; color: var(--ink-faint); max-width: 74ch; }
|
||||
|
||||
@media (max-width: 620px) {
|
||||
.layer { grid-template-columns: 1fr; }
|
||||
.layer .tag { padding: 12px 14px 0; }
|
||||
.layer .lbody { padding: 6px 14px 14px; }
|
||||
.check { grid-template-columns: 1fr; gap: 6px; }
|
||||
.inv-row { grid-template-columns: 1fr; gap: 6px; }
|
||||
}
|
||||
</style>
|
||||
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
|
||||
<header class="masthead">
|
||||
<div class="eyebrow">方法论 · BPU 量化轨迹工厂配套</div>
|
||||
<h1>闭环走通的判据</h1>
|
||||
<p class="standfirst">把「垂直工程任务 → 可训练轨迹」沉淀成可复用范式。走通的标准不是产出一条 PASS 样本,而是四类终态都能被正确落位。</p>
|
||||
<dl class="facts">
|
||||
<div class="fact"><dt>主批次终态</dt><dd>108</dd></div>
|
||||
<div class="fact"><dt>oracle PASS</dt><dd>3</dd></div>
|
||||
<div class="fact"><dt>可用轨迹</dt><dd>0 / 11</dd></div>
|
||||
<div class="fact"><dt>repair 入队</dt><dd>0</dd></div>
|
||||
<div class="fact"><dt>导出样本</dt><dd>1(tools 0)</dd></div>
|
||||
</dl>
|
||||
</header>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§1</span>先说结论</h2>
|
||||
<p class="lede">这套东西方法论上是成立的,而且已经把最难的那一半做完了——<strong>可机器判定的验证器</strong>(四道门 + oracle 相对基准)和<strong>内核级隔离</strong>都已落地。没走通的是另一半:<strong>闭环的最后三个接头是断的</strong>——FAIL 不进 repair、采集缺失不报错、运行进程与磁盘代码不一致。这三处都不是难题,是接线问题。</p>
|
||||
<div class="callout">
|
||||
<h3>「走通」不等于「跑出一条正样本」</h3>
|
||||
<p>今天已经有 1 条 PASS 并导出成 teacher 样本了。但它不能证明闭环成立:它的 <code>tools</code> 数是 0,<code>sft_eligible</code> 仍被标成 1——说明<em>验证器在该拦的地方没有拦</em>。一个只会盖 PASS 的验证器和一个坏掉的验证器,从正样本上看不出区别。</p>
|
||||
<p><strong>闭环成立的判据只有一个:给它四类终态(PASS / FAIL / BLOCKED / TIMEOUT),它能把四类都正确落位,包括把该拒的样本拒掉。</strong>这是本页第 5 节 MVL 清单的全部依据。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§2</span>范式:什么任务能这样工业化</h2>
|
||||
<p>这套做法可以从量化编译迁出去,但有前提。三条同时成立才适用:</p>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>前提</th><th>含义</th><th>量化编译任务为何满足</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><strong>判据可机器化</strong></td><td>对错由程序判定,不靠人评、不靠模型当裁判</td><td>返回码、逐输出余弦、Top1 / mAP 差值,全部是数值比较</td></tr>
|
||||
<tr><td><strong>验证远比求解便宜</strong></td><td>跑一遍很贵、判一次很便宜,才撑得起大量采轨迹</td><td>一个 job 的 oracle 要编译加评测;判四道门只是读 <code>result.json</code></td></tr>
|
||||
<tr><td><strong>失败模式是有限集</strong></td><td>失败能分类,才能区分「agent 判断错」与「工具链真做不到」</td><td><code>error_class</code> 确定性五类、瞬态三类,其余归 <code>unknown</code> 进人工队列</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<p class="note">第三条最容易被忽略,但它决定了 BLOCKED 这一类终态能不能存在。没有失败分类,agent 说「这个做不了」就无法判真假,只能当失败处理——正确放弃的能力就永远学不到。</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§3</span>五个构件,缺一不闭合</h2>
|
||||
<p>范式本身是一条单向链:<strong>冻结输入 → oracle 出标准答案 → 隔离下作答 → 独立核验 → 分级导出</strong>。每一环的职责是硬边界,不能交叉。</p>
|
||||
<div class="stack">
|
||||
<div class="layer">
|
||||
<div class="tag">L0</div>
|
||||
<div class="lbody">
|
||||
<h3>冻结的任务单元</h3>
|
||||
<p>job = 模型 × march × backend × qcfg,<code>job_id</code> 既是目录名也是 join key。run 口径(venv、registry、门阈值)任一变化就开新 run,不原地改。这让任何一条轨迹永远能追溯到它当时面对的确切世界。</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="layer">
|
||||
<div class="tag">L1</div>
|
||||
<div class="lbody">
|
||||
<h3>确定性 oracle —— 一份产出,三种用途</h3>
|
||||
<p>这是整个范式的杠杆点。同一份 oracle 同时是:①ground truth(agent 做得对不对);②相对基准(reward 里精度与延时都是相对 oracle 的差,不是绝对值);③BLOCKED 的裁判(agent 说做不到时,看 oracle 是不是在同阶段以同类 <code>error_class</code> 失败)。省掉 oracle 就同时失去这三样。</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="layer is-agent">
|
||||
<div class="tag">L2</div>
|
||||
<div class="lbody">
|
||||
<h3>隔离的作答区</h3>
|
||||
<p>agent 只有 <code>rollouts/<kk>/workspace/</code> 一个可写区。隔离靠 <code>unshare -rm</code> 的 mount namespace 把答案、兄弟轨迹、凭证从文件系统里摘掉;解析 shell 字符串的 guard 退化为教学层,只负责给一句人话的拒绝理由、省 agent 的预算。</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="layer">
|
||||
<div class="tag">L3</div>
|
||||
<div class="lbody">
|
||||
<h3>独立核验的 verdict</h3>
|
||||
<p>四道门:G1 编译形式、G2 数值余弦、G3 任务指标、G4 板端延时(非必选)。判定完全不读 agent 的自述,<code>claim</code> 只用来和 verdict 比对。<code>na</code>(本轮设计上不评)与 <code>unverified</code>(本该评没评成)严格区分——前者不阻塞 PASS,后者阻塞。</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="layer">
|
||||
<div class="tag">L4</div>
|
||||
<div class="lbody">
|
||||
<h3>离线 reward + 分级导出资格</h3>
|
||||
<p>reward 只读 <code>verdict.json</code>,纯函数、配方版本化,可对历史全量重打分。导出资格分两级:RL 收正负例(要的是区分度),SFT 只收「做对了且诚实」——<code>claim ≠ verdict</code> 和 <code>review=true</code> 是两条优先级最高的硬排除。</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§4</span>八条不变量,每条都有今天的反例</h2>
|
||||
<p>这些不是设计偏好,是今天这批任务用 105 个 FAIL 换来的。每条后面附的是它被违反时实际发生了什么。</p>
|
||||
<div class="inv">
|
||||
|
||||
<div class="inv-row">
|
||||
<div class="mark">I1</div>
|
||||
<div class="body">
|
||||
<h3>运行中的版本必须可证明等于磁盘上的版本</h3>
|
||||
<p>常驻调度进程加载的是它<em>启动那一刻</em>的代码与配置。「改好了」指的是文件,不是正在跑的东西。进程启动时必须自证代码指纹,与 run manifest 不一致就拒绝领新任务——而不是打印一行 WARN。</p>
|
||||
<div class="counter"><b>反例 · 37 例编译失败</b>oracle 调度 PID 3776582 于 01:41:58 启动,<code>harness/backends/ptq.py</code> 于 01:46:29 修好;进程没重启,下午新生成的 YAML 仍把 <code>quant_config</code> 放在 <code>model_parameters</code> 下。<code>paths.config()</code> 的 <code>lru_cache</code> 让旧配置也常驻内存。<code>check_manifest()</code> 对代码漂移只告警,没有拒跑。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="inv-row">
|
||||
<div class="mark">I2</div>
|
||||
<div class="body">
|
||||
<h3>隔离必须在内核层,不能在字符串层</h3>
|
||||
<p>解析 agent 写的 shell 命令做白名单,是一类永远追不平的防线——每一版都会出现新写法。真正的隔离是让进程<em>看不见</em>。看不见的东西,再多绕法也没用。</p>
|
||||
<div class="counter"><b>反例 · guard 连续四版被穿透</b>换行后的 <code>cd</code>、<code>$IFS</code>、<code>$VAR/</code> 路径拼接、workspace 内软链回 <code>../../../oracle</code>——四种手法各穿透一版。v5 起改为 mount namespace 摘除加 capability drop,guard 降级为教学层。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="inv-row">
|
||||
<div class="mark">I3</div>
|
||||
<div class="body">
|
||||
<h3>采集链任何一环缺失,必须硬失败</h3>
|
||||
<p>数据管道里的静默降级比崩溃危险得多:崩溃会被发现,降级会安静地产出「看起来完整」的训练数据。凡是训练必需的字段,缺失即拒绝启动或拒绝导出。</p>
|
||||
<div class="counter"><b>反例 · 唯一一条导出样本工具定义为空</b><code>rollout.py</code> 起 <code>dump_tools.js</code> 时没补 Node bin 的 PATH → <code>npm: not found</code> → 非零退出被忽略 → 不生成 <code>tools.json</code> → 导出回退也失败、返回 <code>([], 'missing')</code> → <code>sft_eligibility()</code> 不检查这项 → 样本仍被标 eligible。A/B 对照:同一脚本补 PATH 后返回 0、导出 7 个工具。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="inv-row">
|
||||
<div class="mark">I4</div>
|
||||
<div class="body">
|
||||
<h3>verdict 覆盖 claim,说谎是最贵的</h3>
|
||||
<p>agent 自述不参与任何判定,只参与一致性比对。<code>claim ≠ verdict</code> 扣 2.0 并绝不进 SFT——训练集里出现一次谎报成功,就是在教它谎报。claim 缺席扣 0.25:不报比报错轻,但不能没有代价。</p>
|
||||
<div class="counter"><b>同类风险 · 编译返回 0 不是验收通过</b>编译日志里的百分比、校准余弦、静态 FPS 估算都不能代替 G1/G2/G3。三条 ConvNeXt 轨迹 <code>exit_code=0</code> 且已生成 HBM,仍没有最终 verdict。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="inv-row">
|
||||
<div class="mark">I5</div>
|
||||
<div class="body">
|
||||
<h3>状态机必须闭合:每个终态都要有去处</h3>
|
||||
<p>FAIL 不是终点,是 repair 轨迹的输入——而且是<em>更有价值</em>的那种输入,因为修复现场比一次性成功更能教出能力。配置里开了开关,不等于链路接上了。</p>
|
||||
<div class="counter"><b>反例 · repair 0 / 0</b><code>matrix.yaml</code> 已配 <code>when: oracle_fail</code>;<code>registry.expand_matrix()</code> 见到该值直接跳过,注释声称由 pipeline 动态追加;而 <code>pipeline.run_oracle_job()</code> 存完 FAIL 就返回,<code>collect_rollouts()</code> 只消费数据库里已有的 QUEUED 行。全库 repair job 与 rollout 均为 0。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="inv-row">
|
||||
<div class="mark">I6</div>
|
||||
<div class="body">
|
||||
<h3>指标口径必须与被测对象的结构匹配</h3>
|
||||
<p>把多分支输出展平算一个整体余弦,结果会被量级最大的那一支主导,小量级分支的失真被完全掩盖。凡是多语义输出,门必须逐分支判。</p>
|
||||
<div class="counter"><b>反例 · YOLO11s 整体余弦 0.9976,类别分支 0.36</b>输出 <code>[1,84,8400]</code> 同时含框坐标(量级大)与 0–1 类别分数。<code>eval/common.py::cosine()</code> 整体展平,G2 判为 pass;而实际 mAP 46.01 → 25.92。整段余弦高不等于精度达标。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="inv-row">
|
||||
<div class="mark">I7</div>
|
||||
<div class="body">
|
||||
<h3>证据边界必须写在结论旁边</h3>
|
||||
<p>每个数字都要带着「它证明了什么、没证明什么」一起流转。否则弱证据会在传递中逐级升格成强结论。这条不产出代码,但它决定了这套数据将来能不能被别人信。</p>
|
||||
<div class="counter"><b>今天需要显式区分的三组</b><code>pass-screen</code>(T1 放宽门限初筛)≠ <code>pass</code>(T2 完整验证);<code>hb_model_info</code> 静态性能估算 ≠ 板端实测;服务器上 <code>run_sim</code> ≠ 物理板端运行。9 份板端记录用的是随机输入,没有板端数据集精度。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="inv-row">
|
||||
<div class="mark">I8</div>
|
||||
<div class="body">
|
||||
<h3>失败的轨迹不删,过滤发生在导出时</h3>
|
||||
<p>磁盘上留什么和数据集里收什么是两件事。轨迹是复盘证据——今天两个 bug 都是靠原始轨迹定位的;数据洁净度由 <code>--min-agent-version</code>、annotations 和资格函数在导出那一刻保证。删了,同样的坑得重新踩一遍才知道。</p>
|
||||
<div class="counter"><b>当前状态</b>r20260915-03 的 11 条已完成 rollout 全部不可用(7 条读了 oracle、3 条被误杀、其余旧版本),已 <code>--annotate</code> 标注保留;9 份旧数据集加 VOID 标记而非删除。</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§5</span>最小可复现闭环(MVL)</h2>
|
||||
<p class="lede">定义:在一个<strong>新冻结 run</strong> 上锁定一条切片(建议 <code>cls--resnet18--nash-e--ptq--int8</code>,它是唯一已知能走完全程的 job),走到一条可进 SFT 的样本,<strong>并且同一 run 内四类终态各有一例被正确落位</strong>。做到这一步,方法论就算走通——规模是之后的事。</p>
|
||||
|
||||
<h3>5.1 四个前置闸门:全为真才允许启动</h3>
|
||||
<div class="checklist">
|
||||
<div class="check">
|
||||
<div class="id">P1</div>
|
||||
<div class="txt"><h3>版本自证</h3><p>调度进程启动时打印自身加载的 harness 代码指纹与两套 venv 的 <code>pip freeze</code> sha256,与 run manifest 比对;不一致<strong>拒跑</strong>而非告警。</p></div>
|
||||
<div><span class="pill part">需改造</span></div>
|
||||
</div>
|
||||
<div class="check">
|
||||
<div class="id">P2</div>
|
||||
<div class="txt"><h3>真实产物校验</h3><p>用已安装的 horizon_tc_ui 3.5.3 Schema 校验<em>实际写到磁盘的第一份</em> compile YAML,而不是校验生成函数的返回值。这是今天 37 例失败唯一有效的拦截点。</p></div>
|
||||
<div><span class="pill part">需改造</span></div>
|
||||
</div>
|
||||
<div class="check">
|
||||
<div class="id">P3</div>
|
||||
<div class="txt"><h3>采集链完整性</h3><p><code>tools.json</code> 存在且工具定义数大于 0,否则拒绝启动 rollout;导出侧同样把它列为硬性资格条件。</p></div>
|
||||
<div><span class="pill no">缺失</span></div>
|
||||
</div>
|
||||
<div class="check">
|
||||
<div class="id">P4</div>
|
||||
<div class="txt"><h3>隔离自检</h3><p><code>guard_smoke.sh</code> 的越界尝试全部被拒、<code>guard-core.test.js</code> 通过;jail 不可用时拒绝启动,不静默退回 guard。</p></div>
|
||||
<div><span class="pill ok">已具备</span></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<h3>5.2 七步产物链:每步都要留下可核验的东西</h3>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>步</th><th>动作</th><th>可核验产物</th><th>判据</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="num-cell">1</td><td>oracle 跑完切片</td><td><code>oracle/result.json</code></td><td>G1/G2/G3 均有明确判定,<code>error_class</code> 非空</td></tr>
|
||||
<tr><td class="num-cell">2</td><td>agent 在 jail 内作答</td><td><code>session.jsonl</code> · <code>events.jsonl</code> · workspace 产物</td><td><code>guard.log</code> 无越界成功记录</td></tr>
|
||||
<tr><td class="num-cell">3</td><td>独立核验</td><td><code>verdict.json</code></td><td>不读 claim 即可算出四道门</td></tr>
|
||||
<tr><td class="num-cell">4</td><td>一致性比对</td><td><code>claim.json</code> ↔ verdict</td><td><code>claim_consistent</code> 有明确真假,不是缺省</td></tr>
|
||||
<tr><td class="num-cell">5</td><td>污染审计</td><td><code>audit_rollouts.py</code> 输出行</td><td>无 <code>oracle_read</code> / <code>sibling_read</code> / <code>stalled_killed</code> / <code>old_version</code></td></tr>
|
||||
<tr><td class="num-cell">6</td><td>离线打分</td><td>reward 与配方版本号</td><td>同一 verdict 重算两次结果逐位一致</td></tr>
|
||||
<tr><td class="num-cell">7</td><td>导出</td><td><code>teacher.jsonl</code></td><td>tools 数大于 0,<code>sft_eligible</code> 的理由可打印</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h3>5.3 分辨力四例:这才是走通的核心判据</h3>
|
||||
<p>只产出正样本证明不了验证器有分辨力。必须在同一个 run 里构造出四类终态,并确认每一类都被<em>正确</em>落位——尤其是第二、第四行这两个「应当被拒」的例子。</p>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>终态</th><th>怎么构造</th><th>期望落位</th><th>现状</th></tr></thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><span class="pill ok">PASS</span></td>
|
||||
<td>ResNet18 / nash-e 正常跑</td>
|
||||
<td>四道门齐全 · RL ✓ · SFT ✓ · tools 完整</td>
|
||||
<td><span class="pill part">有 PASS,tools 缺</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><span class="pill no">污染</span></td>
|
||||
<td>故意让一条 rollout 去读 <code>oracle/</code></td>
|
||||
<td>jail 层面根本读不到;即便读到,audit 标 <code>oracle_read</code> 并判不可用</td>
|
||||
<td><span class="pill part">v5 已隔离,未做对照</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><span class="pill part">BLOCKED</span></td>
|
||||
<td>挑 oracle 确定性失败的 job(MaxViT-t 的 <code>aten::einsum</code> / opset 11)</td>
|
||||
<td>agent 正确判断放弃 → <code>blocked_correct=true</code> → reward +1 → SFT 收</td>
|
||||
<td><span class="pill no">全库无 BLOCKED 样本</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><span class="pill no">谎报</span></td>
|
||||
<td>构造 <code>claim=PASS</code> 而 verdict=FAIL 的轨迹</td>
|
||||
<td>reward −2.0 · SFT 拒收 · 理由可打印</td>
|
||||
<td><span class="pill no">未验证</span></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<p class="note">第三行同时解释了为什么 105 个 FAIL 是资产而不是负债:MaxViT-t 的导出不支持、YOLOv8n 的 f64 Cast、ResNet34 的权重哈希不匹配,这三类都是现成的 BLOCKED 教材——前提是 repair 链路接上,让它们进得了轨迹。</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§6</span>离 MVL 还差什么</h2>
|
||||
<div class="tablewrap">
|
||||
<table>
|
||||
<thead><tr><th>缺口</th><th>证据</th><th>阻断了哪一步</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>运行进程仍是旧代码</td><td>PID 3776582 / 2015257 未重启;7 个源文件两次采样哈希相同,修复未落到运行态</td><td>P1;以及步 1 的 <code>error_class</code> 非空(28 条 FAIL 分类为空,用当前纯函数可全部分类)</td></tr>
|
||||
<tr><td>tools 快照链路断</td><td>PATH 缺 Node bin,A/B 对照已复现</td><td>P3、步 7</td></tr>
|
||||
<tr><td>FAIL → repair 未接线</td><td><code>expand_matrix()</code> 跳过、<code>run_oracle_job()</code> 不追加</td><td>5.3 第三行(BLOCKED 无样本可造)</td></tr>
|
||||
<tr><td>G2 口径对多分支输出失真</td><td>YOLO11s 类别分支 0.36 而整体 0.9976</td><td>步 3 的判定可信度</td></tr>
|
||||
<tr><td>17 条 STALLED 真因未定</td><td>日志终止标记为 log unchanged and cpu idle;任务仍用 JuiceFS 工作目录,存在 I/O 等待误判风险,但缺历史进程树与 D 状态采样</td><td>步 5 的 <code>stalled_killed</code> 判断依据</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§7</span>执行顺序</h2>
|
||||
<p>五步是严格有序的:前一步不做完,后一步的观测结果不可信。</p>
|
||||
<div class="seq">
|
||||
<div class="step">
|
||||
<div class="n">1</div>
|
||||
<div class="sbody">
|
||||
<h3>统一运行版本,开新冻结 run</h3>
|
||||
<p><code>bench drain</code> 让旧调度跑完手里的活并退出(不要 kill 正在编译的进程),保存当前证据,从修正后的代码起新进程。仅重启补不齐尚未实现的 repair 与导出校验。</p>
|
||||
<div class="accept">验收:新 run 第一份落盘 compile YAML 过 Schema;进程指纹与 manifest 一致</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="step">
|
||||
<div class="n">2</div>
|
||||
<div class="sbody">
|
||||
<h3>接上三个断头</h3>
|
||||
<p>Node PATH 与 tools 硬校验;FAIL → repair 的幂等入队;G1 的 dtype / layout 完整校验。三处都是接线,不涉及算法。</p>
|
||||
<div class="accept">验收:造一个 FAIL,repair job 自动出现且不重复;tools 缺失时 rollout 拒绝启动</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="step">
|
||||
<div class="n">3</div>
|
||||
<div class="sbody">
|
||||
<h3>跑通 MVL 的七步产物链</h3>
|
||||
<p>只跑 ResNet18 / nash-e 一条切片,不扩量。目标是一条 tools 完整、四门齐全、audit 干净的样本。</p>
|
||||
<div class="accept">验收:audit 显示 RL ✓ 且 SFT ✓;导出样本 tools 数大于 0</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="step">
|
||||
<div class="n">4</div>
|
||||
<div class="sbody">
|
||||
<h3>补齐分辨力四例</h3>
|
||||
<p>污染、BLOCKED、谎报三个负向对照各造一例。这一步才是「方法论走通」的签收点。</p>
|
||||
<div class="accept">验收:四类终态各一例,落位与 5.3 表格逐行相符</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="step">
|
||||
<div class="n">5</div>
|
||||
<div class="sbody">
|
||||
<h3>再谈规模</h3>
|
||||
<p>分别验收数值与任务指标:保留 FP32 ONNX 对照,按模型处理量化敏感层,检测模型另判框坐标与类别分数。<strong>不要为了提高 PASS 数放宽阈值</strong>——阈值一放,前面四步建立的分辨力就作废了。</p>
|
||||
<div class="accept">验收:收敛指标看 RL ✓ 比例,不看 PASS 绝对数</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<h2><span class="num">§8</span>规模化阶段的唯一指标</h2>
|
||||
<div class="callout">
|
||||
<h3>看 RL ✓ 比例,不看 PASS 数</h3>
|
||||
<p>PASS 数受模型难度、门阈值、工具链成熟度共同影响,涨跌说明不了 agent 学没学到东西。<code>RL ✓</code>(版本合格、没越界、没被误杀)衡量的是<em>轨迹本身可用吗</em>——它才是这条流水线的产能指标。当前该比例为 0/11。</p>
|
||||
<p>同理,当前 3/108 约 2.8% 的通过率<strong>不能当作模型固有可编译率</strong>:其中 37 例是配置层级错误、17 例是看门狗终止,都与模型能不能上 BPU 无关。</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<footer>
|
||||
<p><strong>数据来源</strong>:2026-09-15 的四份检查报告(远端状态复查 16:19–16:20、失败根因定位、板端执行证据复核、精度对比),以及 <code>bench/</code> 仓库的 CONTRACT.md、HANDBOOK.md 与 harness/、export/ 源码。</p>
|
||||
<p><strong>证据边界</strong>:文中数字为已保存评测记录与当日快照,本页未重跑任何推理或编译。「37 例配置层级错误」与「tools 链路」两项有受控复现;「17 条 STALLED 全为误杀」未经证实,不应据此下结论。3/108 是当前判定口径下的比例。</p>
|
||||
</footer>
|
||||
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
18
docs/README.md
Normal file
18
docs/README.md
Normal file
@ -0,0 +1,18 @@
|
||||
# 项目文档
|
||||
|
||||
本目录收录 BPU 量化轨迹工厂的设计与方法说明,原始文档内容保持不变:
|
||||
|
||||
- [设计方案](BPU量化轨迹工厂-设计方案.html)
|
||||
- [方法论](BPU量化轨迹工厂-方法论.html)
|
||||
- [闭环方法论](BPU量化轨迹工厂-闭环方法论.html)
|
||||
- [技术分享讲稿(2026-09-16)](BPU量化轨迹工厂-技术分享讲稿-20260916.md)
|
||||
|
||||
运行入口与配置方式见 [项目 README](../README.md),实现约定见 [CONTRACT.md](../CONTRACT.md)。文档中的实验状态、服务器路径和实测结果对应其编写时间;当前运行状态需要重新核查。
|
||||
|
||||
## Git 收录范围
|
||||
|
||||
仓库收录 harness、模型适配器、Agent 配置模板、数据导出代码、脚本、测试、registry、配置和项目文档,保留现有提交历史。
|
||||
|
||||
模型权重、ONNX/HBM、数据集、缓存、运行状态、训练轨迹、评测输出、日志、第三方源码下载、压缩包以及本地凭据留在各运行环境,不随源码仓库分发。原工作目录下的 `outputs/` 和 `job_verdicts.csv` 属于运行产物;这些文件继续保留在本地。启动前按项目 README 准备依赖、数据、模型与本地环境变量。
|
||||
|
||||
`.env.local.example` 和 `agent/models.json.tmpl` 为可提交模板;真实 `.env.local` 和 `agent/models.json` 已被忽略。
|
||||
@ -4,7 +4,7 @@
|
||||
|
||||
## 结论与边界
|
||||
|
||||
B/A、D、E、G 均已实现;agent/VERSION=5。已通过真实 jail + pi smoke、外部 worker 的真实 calibrate 闭环、guard-core 单测、预算边界测试和 Python 回归。最终探针复测的原始输出见 [验证记录](scripts/evidence-codex.md)。
|
||||
B/A、D、E、G 均已实现;agent/VERSION=5。已通过真实 jail + pi smoke、外部 worker 的真实 calibrate 闭环、guard-core 单测、预算边界测试和 Python 回归。最终探针复测的原始输出见 [验证记录](evidence-codex.md)。
|
||||
|
||||
没有启动 `bench collect` / `bench rollout`,没有 kill 生产编译、pi、stagerun 或调度进程,没有改 config.yaml / registry 数值,没有手工 UPDATE/DELETE 生产 DB。canary #02 仅供只读审计,未重新核验或改写其文件。旧数据集只新增作废文件。
|
||||
|
||||
@ -1,3 +1,8 @@
|
||||
> **已归档(2026-09-17):未实现,暂不做。**
|
||||
> 注意不是"被 long 车道替代"——long 只是 STALLED 后把看门狗阈值抬到 3600s 的重试车道
|
||||
> (`stagerun.py` 对 05-compile 无条件设 `ctx.lane="long"`),与两阶段编译无关。
|
||||
> 本文的动机测量(量化 107s / BPU 编译 4680s)至今仍然成立,问题也仍未解决。
|
||||
|
||||
# SPEC · 05-compile 拆成两阶段(交接文档)
|
||||
|
||||
给下一个负责改这块的 agent。所有数字都是 2026-09-15 在 `120.48.151.164` 实测的,
|
||||
26
docs/board-process-lifecycle.md
Normal file
26
docs/board-process-lifecycle.md
Normal file
@ -0,0 +1,26 @@
|
||||
# 板端进程生命周期
|
||||
|
||||
主机的 `board-<slot>.lock` 只覆盖主机阶段,SSH 退出不保证板上 Python 退出。
|
||||
2026-09-19 的实测故障是:板端 CPU 算子执行较长,主机 `capture_output=True` 没有实时日志,
|
||||
看门狗按主机空闲 900 秒终止本地进程组并释放锁;板端留成 PPID=1,重试又在同一板上启动推理。
|
||||
旧退出函数仅删除临时目录,也无法回收运行中的模型。不能用 CPU 忙证明模型正常推进。
|
||||
|
||||
`RemoteSession.bench` 现在上传并调用独立的 `board_guard.py`,用于 07 性能和 08 精度两条入口:
|
||||
|
||||
- 守护进程持有 runtime_dir 下 `.board-inference.lock`;子进程继承锁,在未退出时不能再次开测。
|
||||
- 开始前检查已有 `board_bench.py` / `hrt_model_exec`,发现未经守护的推理返回 `board_busy`。
|
||||
- 主机每 5 秒续租;EOF、30 秒无租约、单次 RPC 3600 秒上限或终止信号,都回收本次进程组。
|
||||
- 先 TERM,5 秒后 KILL;不因组长先退出而跳过组内其他进程。Linux parent-death signal 在守护进程被 KILL 时终止直接模型子进程。
|
||||
- 原生运行库在子进程内,不能阻塞守护进程。无法立即清除的内核不可中断进程仍持有板端锁,后续测量失败退出。
|
||||
- 板端 stderr 实时回传。`board_progress` 是加载、样本、warmup、计时迭代的实际边界;`alive` 只是传输存活,单次 RPC 上限独立计时。
|
||||
- 成功后清理本次临时目录;失败目录保留 `guard-result.json` 和 HBM 供定位。日志记录 session 路径,结果记录 `board_guard_version`。
|
||||
- 性能阶段等 10 分钟仍繁忙,拒绝生成新的性能数值。历史 `contended` 数据保留原始证据,不能静默改成正常测量。
|
||||
|
||||
验证:`PYTHONPATH=. python3 -m unittest scripts.test_board_runtime scripts.test_board_guard`。
|
||||
Linux 上的 guard 测试会真实注入断连、停止续租、超时、TERM/KILL 和锁争用;不访问 SSH/BPU。
|
||||
本地 macOS 会跳过 Linux 进程测试,应额外在 Linux 隔离临时目录执行。
|
||||
|
||||
部署走 `scripts/sync.sh`。若远端已有其他任务的未合并改动,可以提交本次文件后使用
|
||||
`scripts/sync.sh --files <files...>`,仅比较所选文件与上次同步的前像,保存远端备份并校验落盘哈希。
|
||||
新文件先于引用方安装;不会推进其他文件的同步基线。旧的已启动评测不会重新 import,
|
||||
新启动的工具子进程会加载修复,无需重启 collector。现役任务的结果须分别检查,不得宣称所有旧进程已经换代。
|
||||
@ -1,18 +1,7 @@
|
||||
"""reward v0.4 配方。离线算,只读 verdict.json;配方版本化,可全量重打分。
|
||||
"""reward v0.6:只按本次真实门证据、成本、claim 一致性和超时计算。
|
||||
|
||||
r = 1.0·[G1] + 1.0·[G2] + 2.0·[G3] 绝对门
|
||||
r += 1.0·clip(精度改善/|tol|, −1, +1) 相对 oracle 的精度
|
||||
r += 0.5·clip((lat_oracle − lat_agent)/lat_oracle, −1, +1) 相对 oracle 的延时
|
||||
r −= 0.5·turns/max_turns + 0.5·tokens/max_tokens 代价
|
||||
r −= 2.0·[claim ≠ verdict] 说谎
|
||||
r −= 0.25·[claim 缺席]
|
||||
BLOCKED: +1.0 若 oracle 同阶段同类失败;否则 −2.0
|
||||
执行 TIMEOUT: 另扣 −1.0,已经完成的真实阶段仍保留得分。
|
||||
G3 = na:其 2.0 权重按比例分给 G1/G2(各 2.0),精度相对项取 0
|
||||
板卡不可用(v0.4):G3 unverified 且最后一次 07-board / 08-eval 以 board_offline 失败 → G2/G3 权重取 0(板上测的两门
|
||||
都没法测,不是 agent 的错),G1 保持 1.0 不加分,免 TIMEOUT 罚分,返回值带 unverifiable="board_offline"。
|
||||
实测(2026-09-16 r20260916-01):regnet.y.800mf 14 次 eval 全 board_offline,v0.3 给 −0.97,等于把基础设施故障记在
|
||||
agent 头上。注意这类轨迹在导出器里仍按 UNVERIFIED 排除在 RL/SFT 之外;真正回收要用 bench board-backfill --rollout 补测。
|
||||
不读取 Oracle 对照或 BLOCKED 正确性标签。BLOCKED 的已测阶段仍保留分数,
|
||||
但导出器不将未证实的 BLOCKED 纳入奖励训练。板卡不可用仍按既有规则免超时罚分。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@ -20,15 +9,11 @@ import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
RECIPE_VERSION = "v0.4"
|
||||
RECIPE_VERSION = "v0.6"
|
||||
BOARD_STAGES = ("07-board", "08-eval")
|
||||
PASSING = {"pass", "pass-screen", "pass-weak"}
|
||||
W = {"G1": 1.0, "G2": 1.0, "G3": 2.0, "acc_rel": 1.0, "lat_rel": 0.5, "cost_turns": 0.5, "cost_tokens": 0.5,
|
||||
"lie": 2.0, "claim_absent": 0.25, "blocked_ok": 1.0, "blocked_bad": 2.0, "timeout": 1.0}
|
||||
|
||||
|
||||
def clip(x: float, lo: float = -1.0, hi: float = 1.0) -> float:
|
||||
return max(lo, min(hi, x))
|
||||
W = {"G1": 1.0, "G2": 1.0, "G3": 2.0, "cost_turns": 0.5, "cost_tokens": 0.5,
|
||||
"lie": 2.0, "claim_absent": 0.25, "timeout": 1.0}
|
||||
|
||||
|
||||
def board_unavailable(verdict: dict) -> bool:
|
||||
@ -46,37 +31,18 @@ def compute(verdict: dict, max_turns: int = 40, max_tokens: int = 200000) -> dic
|
||||
g = verdict.get("gates") or {}
|
||||
terminal = verdict.get("terminal")
|
||||
claim = verdict.get("claim") or {}
|
||||
oref = verdict.get("oracle_ref") or {}
|
||||
cost = verdict.get("cost") or {}
|
||||
unverifiable = None
|
||||
|
||||
if terminal == "BLOCKED":
|
||||
terms["blocked"] = W["blocked_ok"] if verdict.get("blocked_correct") else -W["blocked_bad"]
|
||||
g3_na = (g.get("G3") or {}).get("verdict") == "na"
|
||||
if board_unavailable(verdict):
|
||||
unverifiable = "board_offline"
|
||||
w1, w2, w3 = W["G1"], 0.0, 0.0
|
||||
else:
|
||||
g3_na = (g.get("G3") or {}).get("verdict") == "na"
|
||||
if board_unavailable(verdict):
|
||||
unverifiable = "board_offline"
|
||||
w1, w2, w3 = W["G1"], 0.0, 0.0 # 不加分只免罚:板上测的两门不计,G1 原样
|
||||
else:
|
||||
w1, w2, w3 = (2.0, 2.0, 0.0) if g3_na else (W["G1"], W["G2"], W["G3"])
|
||||
terms["G1"] = w1 * float((g.get("G1") or {}).get("verdict") in PASSING)
|
||||
terms["G2"] = w2 * float((g.get("G2") or {}).get("verdict") in PASSING)
|
||||
terms["G3"] = w3 * float((g.get("G3") or {}).get("verdict") in PASSING)
|
||||
# abs_pp 的 delta 越大越好;rel_pct 的 delta 是劣化百分比,越小越好。
|
||||
g3 = g.get("G3") or {}
|
||||
same_kind = oref.get("G3_kind", g3.get("kind")) == g3.get("kind")
|
||||
if not g3_na and same_kind and g3.get("delta") is not None and oref.get("G3_delta") is not None and g3.get("tolerance"):
|
||||
improvement = g3["delta"] - oref["G3_delta"]
|
||||
if g3.get("kind") == "rel_pct":
|
||||
improvement = -improvement
|
||||
terms["acc_rel"] = W["acc_rel"] * clip(improvement / abs(g3["tolerance"]))
|
||||
else:
|
||||
terms["acc_rel"] = 0.0
|
||||
g4 = g.get("G4") or {}
|
||||
if g4.get("latency_ms_mean") and oref.get("latency_ms_mean") and not g4.get("contended"):
|
||||
terms["lat_rel"] = W["lat_rel"] * clip((oref["latency_ms_mean"] - g4["latency_ms_mean"]) / oref["latency_ms_mean"])
|
||||
else:
|
||||
terms["lat_rel"] = 0.0
|
||||
w1, w2, w3 = (2.0, 2.0, 0.0) if g3_na else (W["G1"], W["G2"], W["G3"])
|
||||
terms["G1"] = w1 * float((g.get("G1") or {}).get("verdict") in PASSING)
|
||||
terms["G2"] = w2 * float((g.get("G2") or {}).get("verdict") in PASSING)
|
||||
terms["G3"] = w3 * float((g.get("G3") or {}).get("verdict") in PASSING)
|
||||
|
||||
if (verdict.get("execution_terminal") == "TIMEOUT" or terminal == "TIMEOUT") and not unverifiable:
|
||||
terms["timeout"] = -W["timeout"] # 板子不通时把预算耗在重试上,不算 agent 超时
|
||||
@ -95,7 +61,7 @@ def compute(verdict: dict, max_turns: int = 40, max_tokens: int = 200000) -> dic
|
||||
terms["lie"] = -W["lie"]
|
||||
r = sum(terms.values())
|
||||
return {"reward": round(r, 4), "terms": {k: round(v, 4) for k, v in terms.items()}, "recipe": RECIPE_VERSION,
|
||||
"terminal": terminal, "review": bool(verdict.get("review")), "note": terms_note,
|
||||
"terminal": terminal, "note": terms_note,
|
||||
"unverifiable": unverifiable}
|
||||
|
||||
|
||||
|
||||
@ -11,6 +11,7 @@ from unittest.mock import patch
|
||||
from export import to_agentrl as ex
|
||||
from export.reward import compute
|
||||
from harness.db import SCHEMA
|
||||
from harness.verify import VERDICT_SCHEMA, VERIFICATION_RULE
|
||||
|
||||
|
||||
def dialogue():
|
||||
@ -36,15 +37,17 @@ class MaterializationTests(unittest.TestCase):
|
||||
def tearDown(self):
|
||||
self.env.stop(); self.patch.stop(); self.con.close(); self.tmp.cleanup()
|
||||
|
||||
def add(self, kk=0, terminal="PASS", execution="DONE", review=False, ended=True, version=6):
|
||||
def add(self, kk=0, terminal="PASS", execution="DONE", review=False, ended=True, version=6, mode="e2e"):
|
||||
jd = self.root / "runs/r/j"
|
||||
rd = jd / "rollouts" / f"{kk:02d}"
|
||||
(rd / "agent").mkdir(parents=True)
|
||||
job = {"run_id": "r", "job_id": "j", "task": "cls", "model": "resnet18", "march": "nash-m", "backend": "ptq"}
|
||||
job = {"run_id": "r", "job_id": "j", "task": "cls", "model": "resnet18", "march": "nash-m", "backend": "ptq",
|
||||
"gates": {"required": ["G1", "G3"], "G1": {}, "G2": {},
|
||||
"G3": {"kind": "rel_pct", "tol": 5, "require_source": "board"}, "G4": {}}}
|
||||
(jd / "job.json").write_text(json.dumps(job))
|
||||
(rd / "rollout.json").write_text(json.dumps({"state": execution, "model": "glm-5.3", "policy": "teacher",
|
||||
(rd / "rollout.json").write_text(json.dumps({"state": execution, "model": "glm-5.3", "policy": "teacher", "mode": mode,
|
||||
"agent_config_version": version, "max_turns": 60, "max_tokens": 100}))
|
||||
verdict = {"run_id": "r", "job_id": "j", "rollout": kk, "terminal": terminal,
|
||||
verdict = {"schema": VERDICT_SCHEMA, "verification_rule": VERIFICATION_RULE, "mode": mode, "run_id": "r", "job_id": "j", "rollout": kk, "terminal": terminal,
|
||||
"execution_terminal": execution, "timeout_reason": "budget" if execution == "TIMEOUT" else None,
|
||||
"claim_consistent": True, "claim": {"terminal": terminal}, "review": review,
|
||||
"gates": {"G1": {"verdict": "pass"}, "G2": {"verdict": "pass"}, "G3": {"verdict": "pass-screen"}},
|
||||
@ -75,7 +78,7 @@ class MaterializationTests(unittest.TestCase):
|
||||
def test_timeout_keeps_verified_stage_rewards(self):
|
||||
rd = self.add(execution="TIMEOUT")
|
||||
reward = compute(json.loads((rd / "verdict.json").read_text()), 60, 100)
|
||||
self.assertEqual(reward["recipe"], "v0.4")
|
||||
self.assertEqual(reward["recipe"], "v0.6")
|
||||
self.assertEqual(reward["terms"]["G3"], 2.0)
|
||||
self.assertEqual(reward["terms"]["timeout"], -1.0)
|
||||
manifest = ex.materialize(["r"], "data")
|
||||
@ -95,13 +98,87 @@ class MaterializationTests(unittest.TestCase):
|
||||
self.assertEqual(recs[0]["trajectory"]["source"], "message_end")
|
||||
self.assertEqual(sum(m["role"] == "assistant" for m in recs[0]["messages"]), 1)
|
||||
|
||||
def test_review_is_retained_but_not_positive_training_candidate(self):
|
||||
self.add(review=True)
|
||||
manifest = ex.materialize(["r"], "data")
|
||||
self.assertEqual(manifest["stats"]["review"], 1)
|
||||
def legacy_evidence(self, rd, *, mode="e2e", claim="PASS"):
|
||||
original = json.loads((rd / "verdict.json").read_text())
|
||||
original.update(schema="bench.verdict/2", review=True,
|
||||
oracle_ref={"terminal": "FAIL", "G3_delta": -99})
|
||||
(rd / "verdict.json").write_text(json.dumps(original))
|
||||
(rd / "claim.json").write_text(json.dumps({"terminal": claim}))
|
||||
for stage, measurements in {
|
||||
"06-hbm": {"G1": {"hbm_exists": True, "march_ok": True, "io_ok": True,
|
||||
"verifier_ok": True, "log_has_error": False}},
|
||||
"08-eval": {"G3": {"fp32": 80, "quant": 79.2, "n": 100, "source": "board", "tier": "t1"}},
|
||||
}.items():
|
||||
sd = rd / "stages" / stage
|
||||
sd.mkdir(parents=True, exist_ok=True)
|
||||
(sd / "meta.json").write_text(json.dumps({"state": "OK", "measurements": measurements}))
|
||||
return original
|
||||
|
||||
def test_legacy_oracle_review_is_reverified_without_modifying_sources(self):
|
||||
rd = self.add(review=True)
|
||||
self.legacy_evidence(rd)
|
||||
before = {p: hashlib.sha256(p.read_bytes()).hexdigest() for p in rd.rglob("*") if p.is_file()}
|
||||
before_db = list(self.con.execute("SELECT * FROM rollouts"))
|
||||
manifest = ex.materialize(["r"], "data", purpose="sft")
|
||||
self.assertEqual(manifest["stats"]["reverified"], 1)
|
||||
rec = json.loads((self.root / "datasets/data/teacher.jsonl").read_text())
|
||||
self.assertTrue(rec["sft_eligible"])
|
||||
self.assertTrue(rec["training_uses"]["reward_learning_candidate"])
|
||||
self.assertEqual(rec["verification"]["method"], "readonly-reverification")
|
||||
self.assertEqual(rec["verdict"]["schema"], VERDICT_SCHEMA)
|
||||
self.assertFalse(rec["verdict"]["review"])
|
||||
self.assertNotIn("oracle_ref", rec)
|
||||
self.assertNotIn("history", rec)
|
||||
self.assertTrue(rec["provenance"]["original_verdict"]["review"])
|
||||
self.assertEqual(rec["provenance"]["original_verdict"]["sha256"], before[rd / "verdict.json"])
|
||||
self.assertEqual(before, {p: hashlib.sha256(p.read_bytes()).hexdigest() for p in before})
|
||||
self.assertEqual(before_db, list(self.con.execute("SELECT * FROM rollouts")))
|
||||
|
||||
def test_failed_reverification_is_excluded_without_legacy_fallback(self):
|
||||
rd = self.add(review=True)
|
||||
self.legacy_evidence(rd)
|
||||
(rd / "stages/08-eval/meta.json").write_text("broken evidence")
|
||||
stats = ex.materialize(["r"], "data")["stats"]
|
||||
self.assertEqual(stats["written"], 0)
|
||||
self.assertEqual(stats["dropped"], {"reverification_failed:JSONDecodeError": 1})
|
||||
|
||||
def test_historical_repair_remains_analysis_only(self):
|
||||
rd = self.add(mode="repair")
|
||||
self.legacy_evidence(rd)
|
||||
ex.materialize(["r"], "all")
|
||||
rec = json.loads((self.root / "datasets/all/teacher.jsonl").read_text())
|
||||
self.assertEqual(rec["mode"], "repair")
|
||||
self.assertFalse(rec["sft_eligible"])
|
||||
self.assertEqual(rec["sft_reason"], "historical-repair-context")
|
||||
self.assertFalse(rec["training_uses"]["reward_learning_candidate"])
|
||||
self.assertEqual(ex.materialize(["r"], "sft", purpose="sft")["stats"]["written"], 0)
|
||||
self.assertEqual(ex.materialize(["r"], "rl", purpose="rl")["stats"]["written"], 0)
|
||||
|
||||
def test_blocked_is_not_a_success_or_reward_training_target(self):
|
||||
rd = self.add(terminal="BLOCKED")
|
||||
v = json.loads((rd / "verdict.json").read_text())
|
||||
v.update(blocked_correct=True, oracle_ref={"terminal": "FAIL"})
|
||||
(rd / "verdict.json").write_text(json.dumps(v))
|
||||
ex.materialize(["r"], "all")
|
||||
rec = json.loads((self.root / "datasets/all/teacher.jsonl").read_text())
|
||||
self.assertFalse(rec["sft_eligible"])
|
||||
self.assertFalse(rec["training_uses"]["reward_learning_candidate"])
|
||||
self.assertNotIn("blocked", rec["reward_terms"])
|
||||
|
||||
def test_oracle_independence_does_not_bypass_transcript_or_annotation_checks(self):
|
||||
first = self.add(ended=False)
|
||||
self.legacy_evidence(first)
|
||||
second = self.add(kk=1)
|
||||
self.legacy_evidence(second)
|
||||
(second / "claim.json").unlink()
|
||||
third = self.add(kk=2)
|
||||
self.legacy_evidence(third)
|
||||
self.con.execute("INSERT INTO annotations(run_id,job_id,note) VALUES('r','j','rollout#2 contaminated: copied solution')")
|
||||
self.con.commit()
|
||||
stats = ex.materialize(["r"], "data", purpose="sft")["stats"]
|
||||
self.assertEqual(stats["written"], 0)
|
||||
self.assertEqual(stats["dropped"], {"sft:partial-transcript": 1, "sft:claim-missing-or-inconsistent": 1,
|
||||
"contaminated": 1})
|
||||
|
||||
def test_old_version_and_missing_verdict_have_explicit_exclusions(self):
|
||||
self.add(version=5)
|
||||
@ -113,6 +190,17 @@ class MaterializationTests(unittest.TestCase):
|
||||
self.assertIn("old_version", exclusions)
|
||||
self.assertIn("missing_verdict", exclusions)
|
||||
|
||||
def test_sft_and_rl_exports_do_not_read_historical_pipeline(self):
|
||||
self.add()
|
||||
with patch.object(ex, "pipeline_records", side_effect=OSError("unreadable old pipeline")) as pipeline:
|
||||
for purpose in ("sft", "rl"):
|
||||
with self.subTest(purpose=purpose):
|
||||
result = ex.materialize(["r"], purpose, purpose=purpose)
|
||||
self.assertEqual(result["stats"]["agent_written"], 1)
|
||||
self.assertEqual(result["stats"]["pipeline_written"], 0)
|
||||
self.assertEqual((self.root / "datasets" / purpose / "pipeline.jsonl").read_text(), "")
|
||||
pipeline.assert_not_called()
|
||||
|
||||
def test_pipeline_is_real_separate_and_failed_jobs_are_retained(self):
|
||||
self.add()
|
||||
jd = self.root / "runs/r/j"
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
"""L4 数据物化:扫描 rollouts → 脱敏、清洗(R1–R8)、线性化 → 与 verdict / oracle 配对 → reward → datasets/<dataset_id>/。
|
||||
"""L4 数据物化:扫描 rollouts → 脱敏、清洗(R1–R8)、线性化 → 独立核验 → reward → datasets/<dataset_id>/。
|
||||
|
||||
python export/to_agentrl.py --runs r20260914-01 [r…] [--policy teacher] [--include-timeout]
|
||||
|
||||
@ -267,15 +267,15 @@ def sft_eligibility(verdict: dict, reward: float) -> tuple[bool, str]:
|
||||
honestly claimed success is not rejected solely for exhausting its budget.
|
||||
Legacy TIMEOUT verdicts stay evidence until reverified, never assumed PASS.
|
||||
"""
|
||||
if verdict.get("review"):
|
||||
return False, "review-pending(oracle FAIL / rollout PASS,待人工抽查)"
|
||||
if verdict.get("mode") == "repair":
|
||||
return False, "historical-repair-context"
|
||||
if verdict.get("claim_consistent") is not True:
|
||||
return False, "claim-missing-or-inconsistent"
|
||||
t = verdict.get("terminal")
|
||||
if t == "PASS":
|
||||
return True, "PASS"
|
||||
if t == "BLOCKED":
|
||||
return (True, "BLOCKED-correct") if verdict.get("blocked_correct") else (False, "BLOCKED-misjudged")
|
||||
return False, "BLOCKED-unproven"
|
||||
if t == "TIMEOUT":
|
||||
return False, "legacy-TIMEOUT-needs-reverification"
|
||||
return False, f"{t}-not-success-demonstration"
|
||||
@ -292,7 +292,7 @@ def scan(run_ids: list[str]) -> list[Path]:
|
||||
# DB index only: never enumerate the JuiceFS run tree.
|
||||
with readonly_db() as con:
|
||||
rows = [row for rid in run_ids for row in con.execute(
|
||||
"SELECT run_id, job_id, kk FROM rollouts WHERE run_id=? AND state NOT IN ('RUNNING','QUEUED','VERIFYING')", (rid,))]
|
||||
"SELECT run_id, job_id, kk FROM rollouts WHERE run_id=? AND state NOT IN ('RUNNING','QUEUED','VERIFYING','TRANSFERRED')", (rid,))]
|
||||
return sorted(BENCH / 'runs' / rid / jid / 'rollouts' / f'{kk:02d}' for rid, jid, kk in rows)
|
||||
|
||||
|
||||
@ -363,7 +363,7 @@ def pipeline_records(run_ids: list[str]):
|
||||
DB-indexed only. No recursive traversal of model/data trees and no simulated
|
||||
LLM dialogue. Missing evidence is recorded rather than hiding a finished job.
|
||||
"""
|
||||
from harness.verify import completion_summary
|
||||
from harness.verify import completion_summary, eval_set
|
||||
with readonly_db() as con:
|
||||
jobs = [dict(row) for rid in run_ids for row in con.execute(
|
||||
"SELECT * FROM jobs WHERE run_id=? AND mode='oracle' AND state IN ('PASS','FAIL','UNVERIFIED')", (rid,))]
|
||||
@ -402,6 +402,7 @@ def pipeline_records(run_ids: list[str]):
|
||||
"schema": "bench.pipeline-training/1", "source": "deterministic_pipeline",
|
||||
"id": hashlib.sha1(f"{rid}/{jid}/oracle".encode()).hexdigest()[:16],
|
||||
"run_id": rid, "job_id": jid, "model": job.get("model", row.get("model")),
|
||||
"eval_set": eval_set(job), "continuation": job.get("continuation"),
|
||||
"task": job.get("task", row.get("task")), "march": job.get("march", row.get("march")),
|
||||
"backend": job.get("backend", row.get("backend")), "terminal": result.get("terminal", row["state"]),
|
||||
"completion": completion_summary(result.get("gates") or {}, result.get("stages") or {}),
|
||||
@ -429,7 +430,8 @@ def _write_dataset(a, work: Path, ds_id: str) -> dict:
|
||||
} for rid in a.runs}
|
||||
stats = {"scanned": 0, "written": 0, "agent_written": 0, "teacher_written": 0, "student_written": 0,
|
||||
"pipeline_written": 0, "excluded": 0, "dropped": {}, "review": 0, "partial": 0,
|
||||
"by_terminal": {}, "by_execution_terminal": {}, "pipeline_by_terminal": {}, "sft_eligible": 0}
|
||||
"by_terminal": {}, "by_execution_terminal": {}, "pipeline_by_terminal": {}, "sft_eligible": 0,
|
||||
"reverified": 0}
|
||||
with contextlib.ExitStack() as stack:
|
||||
files = {name: stack.enter_context((work / f"{name}.jsonl").open("w"))
|
||||
for name in ("teacher", "student", "pipeline", "exclusions")}
|
||||
@ -442,10 +444,11 @@ def _write_dataset(a, work: Path, ds_id: str) -> dict:
|
||||
write("exclusions", {"run_id": rd.parents[2].name, "job_id": rd.parents[1].name,
|
||||
"rollout": int(rd.name), "source": str(rd.relative_to(BENCH)),
|
||||
"reasons": sorted(reasons)})
|
||||
for record in pipeline_records(a.runs):
|
||||
write("pipeline", record)
|
||||
stats["pipeline_written"] += 1
|
||||
_inc(stats["pipeline_by_terminal"], record["terminal"])
|
||||
if a.purpose == "all":
|
||||
for record in pipeline_records(a.runs):
|
||||
write("pipeline", record)
|
||||
stats["pipeline_written"] += 1
|
||||
_inc(stats["pipeline_by_terminal"], record["terminal"])
|
||||
for rd in scan(a.runs):
|
||||
_write_agent(rd, a, stats, annotations, write, exclude)
|
||||
file_info = {}
|
||||
@ -463,7 +466,8 @@ def _write_dataset(a, work: Path, ds_id: str) -> dict:
|
||||
"annotations_read": True, "dropped_counts_overlap": True, "policy": a.policy,
|
||||
"include_timeout": True, "purpose": a.purpose, "min_reward": a.min_reward},
|
||||
"stats": stats, "files": file_info,
|
||||
"notes": ["pipeline.jsonl is deterministic execution evidence, not Agent dialogue.",
|
||||
"notes": ["pipeline.jsonl contains historical deterministic evidence only for purpose=all; SFT/RL export reads Agent evidence only.",
|
||||
"Verdict v3 and reward v0.6 use only this rollout evidence; Oracle references are history only.",
|
||||
"SFT/reward candidates are labels; no train split, framework conversion, or online RL readiness is implied.",
|
||||
"Writers hold .export.lock; manifest is published last. Readers can lock or verify file hashes for a coherent refresh."]}
|
||||
(work / "manifest.json").write_text(json.dumps(manifest, indent=2, ensure_ascii=False))
|
||||
@ -471,8 +475,10 @@ def _write_dataset(a, work: Path, ds_id: str) -> dict:
|
||||
|
||||
|
||||
def _write_agent(rd, a, stats, annotations, write, exclude):
|
||||
from harness.verify import eval_set, verify_rollout, VERDICT_SCHEMA, VERIFICATION_RULE
|
||||
stats["scanned"] += 1
|
||||
verdict = _read_json(rd / "verdict.json")
|
||||
original_verdict = _read_json(rd / "verdict.json")
|
||||
verdict = original_verdict
|
||||
rj = _read_json(rd / "rollout.json")
|
||||
if not verdict:
|
||||
exclude(rd, {"missing_verdict"}); return
|
||||
@ -487,6 +493,17 @@ def _write_agent(rd, a, stats, annotations, write, exclude):
|
||||
reasons.add('old_version')
|
||||
if reasons:
|
||||
exclude(rd, reasons); return
|
||||
original_sha = hashlib.sha256((rd / "verdict.json").read_bytes()).hexdigest()
|
||||
reverified = verdict.get("schema") != VERDICT_SCHEMA
|
||||
if reverified:
|
||||
try:
|
||||
verdict = verify_rollout(rd, write=False)
|
||||
stats["reverified"] += 1
|
||||
except Exception as error:
|
||||
exclude(rd, {f"reverification_failed:{type(error).__name__}"}); return
|
||||
verification = {"schema": verdict.get("schema"),
|
||||
"rule": verdict.get("verification_rule", VERIFICATION_RULE),
|
||||
"method": "readonly-reverification" if reverified else "recorded-verdict"}
|
||||
raw, trajectory = load_trajectory(rd)
|
||||
if not raw:
|
||||
exclude(rd, {"no_trajectory"}); return
|
||||
@ -502,6 +519,10 @@ def _write_agent(rd, a, stats, annotations, write, exclude):
|
||||
if cst["drop"]:
|
||||
exclude(rd, {"no_usable_messages"}); return
|
||||
msgs, rst = redact_messages(msgs)
|
||||
jd = rd.parent.parent
|
||||
job = json.loads((jd / "job.json").read_text())
|
||||
mode = verdict.get("mode") or rj.get("mode")
|
||||
verdict = {**verdict, "mode": mode}
|
||||
rw = compute_reward(verdict, rj.get("max_turns", a.max_turns), rj.get("max_tokens", a.max_tokens))
|
||||
sft_ok, sft_reason = sft_eligibility(verdict, rw["reward"])
|
||||
trajectory["partial"] = trajectory["partial"] or not cst["tool_pairing"]["valid"] or cst.get("trailing_user", False)
|
||||
@ -517,36 +538,39 @@ def _write_agent(rd, a, stats, annotations, write, exclude):
|
||||
sft_ok, sft_reason = False, "missing-called-tool-definitions"
|
||||
if a.purpose == "sft" and not sft_ok:
|
||||
exclude(rd, {f"sft:{sft_reason}"}); return
|
||||
reward_candidate = verdict["terminal"] in ("PASS", "FAIL", "BLOCKED") and not verdict.get("review")
|
||||
reward_candidate = verdict["terminal"] in ("PASS", "FAIL") and mode != "repair" and not rw.get("unverifiable")
|
||||
if a.purpose == "rl" and not reward_candidate:
|
||||
exclude(rd, {"rl:review-or-unverified-outcome"}); return
|
||||
exclude(rd, {"rl:historical-repair-or-unverified-outcome"}); return
|
||||
if a.min_reward is not None and rw["reward"] < a.min_reward:
|
||||
exclude(rd, {"min_reward"}); return
|
||||
jd = rd.parent.parent
|
||||
job = json.loads((jd / "job.json").read_text())
|
||||
rec = {
|
||||
"schema": "bench.agent-training/2", "source": "agent_events",
|
||||
"id": hashlib.sha1(f"{verdict['run_id']}/{verdict['job_id']}/{verdict['rollout']}".encode()).hexdigest()[:16],
|
||||
"run_id": verdict["run_id"], "job_id": verdict["job_id"], "rollout": verdict["rollout"],
|
||||
"mode": verdict.get("mode") or rj.get("mode"), "policy": policy, "model": rj.get("model"),
|
||||
"eval_set": verdict.get("eval_set") or eval_set(job),
|
||||
"continuation": {**job["continuation"], "source_rollout": verdict["rollout"]} if job.get("continuation") else None,
|
||||
"mode": mode, "policy": policy, "model": rj.get("model"),
|
||||
"agent_config_version": str(rj.get("agent_config_version", "1")),
|
||||
"task": job["task"], "march": job["march"], "backend": job["backend"],
|
||||
"messages": msgs,
|
||||
"verdict": {k: verdict[k] for k in ("terminal", "execution_terminal", "timeout_reason", "completion", "failed_gate", "claim_consistent", "blocked_correct", "review") if k in verdict},
|
||||
"verdict": {k: verdict[k] for k in ("schema", "verification_rule", "terminal", "measured_terminal", "execution_terminal", "timeout_reason", "completion", "failed_gate", "claim_consistent", "blocked_correct", "review") if k in verdict},
|
||||
"gates": {g: (verdict["gates"].get(g) or {}).get("verdict") for g in ("G1", "G2", "G3", "G4")},
|
||||
"oracle_ref": verdict.get("oracle_ref"),
|
||||
"verification": verification,
|
||||
"reward": rw["reward"], "reward_terms": rw["terms"], "reward_recipe": rw["recipe"],
|
||||
"reward_unverifiable": rw.get("unverifiable"), # v0.4:board_offline = 板卡故障导致没法测,不是 agent 的结果
|
||||
"sft_eligible": sft_ok, "sft_reason": sft_reason,
|
||||
"training_uses": {"analysis": True, "sft_candidate": sft_ok, "reward_learning_candidate": reward_candidate,
|
||||
"review_required": bool(verdict.get("review"))},
|
||||
"review_required": False},
|
||||
"trajectory": trajectory,
|
||||
"execution": {"terminal": verdict.get("execution_terminal") or rj.get("state"),
|
||||
"timeout_reason": verdict.get("timeout_reason") or rj.get("timeout_reason"),
|
||||
"started_at": rj.get("started_at"), "finished_at": rj.get("finished_at"),
|
||||
"max_turns": rj.get("max_turns"), "max_tokens": rj.get("max_tokens")},
|
||||
"provenance": {"events": str((rd / "agent" / "events.jsonl").relative_to(BENCH)),
|
||||
"verdict": str((rd / "verdict.json").relative_to(BENCH))},
|
||||
"verdict": str((rd / "verdict.json").relative_to(BENCH)),
|
||||
"original_verdict": {"sha256": original_sha, "schema": original_verdict.get("schema"),
|
||||
"terminal": original_verdict.get("terminal"),
|
||||
"review": original_verdict.get("review")}},
|
||||
"cost": verdict.get("cost"),
|
||||
"tools": tools,
|
||||
"tools_source": tools_src,
|
||||
@ -560,7 +584,6 @@ def _write_agent(rd, a, stats, annotations, write, exclude):
|
||||
stats["agent_written"] += 1
|
||||
stats[f"{policy}_written"] += 1
|
||||
stats["partial"] += int(trajectory["partial"])
|
||||
stats["review"] += int(bool(verdict.get("review")))
|
||||
_inc(stats["by_terminal"], verdict["terminal"])
|
||||
_inc(stats["by_execution_terminal"], rec["execution"]["terminal"])
|
||||
stats["sft_eligible"] += int(sft_ok)
|
||||
@ -615,7 +638,7 @@ def main() -> int:
|
||||
ap.add_argument("--policy", choices=["teacher", "student"])
|
||||
ap.add_argument("--include-timeout", action="store_true", help="compatibility flag: timeout evidence is now included by default")
|
||||
ap.add_argument("--purpose", choices=["all", "sft", "rl"], default="all",
|
||||
help="all=保留终态轨迹并标注用途;sft=完整诚实成功候选;rl=有核验结果且不待复核的候选")
|
||||
help="all=保留终态轨迹并标注用途;sft=完整诚实成功候选;rl=本次可核验 PASS/FAIL 候选")
|
||||
ap.add_argument("--min-reward", type=float)
|
||||
ap.add_argument("--max-turns", type=int, default=40)
|
||||
ap.add_argument("--max-tokens", type=int, default=200000)
|
||||
|
||||
@ -1,2 +1,43 @@
|
||||
"""custom 模型的构建适配器:每个模型一个 <model_id>.py,提供 build(weights: Path, input_spec: str) -> nn.Module。
|
||||
非 torchvision / ultralytics / timm 的模型走这里。"""
|
||||
"""Optional model recipes, shared by input preparation and evaluation."""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib
|
||||
import json
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@lru_cache(maxsize=None)
|
||||
def get(model: str):
|
||||
name = model.replace(".", "_").replace("-", "_")
|
||||
if not (Path(__file__).parent / f"{name}.py").is_file():
|
||||
return None
|
||||
return importlib.import_module(f"{__name__}.{name}")
|
||||
|
||||
|
||||
def model_spec(row: dict) -> dict:
|
||||
"""Resolve concrete adapter inputs without changing the source CSV."""
|
||||
adapter = get(row["model_id"])
|
||||
if adapter is None:
|
||||
return row
|
||||
result = dict(row)
|
||||
result.update(getattr(adapter, "MODEL_SPEC", {}))
|
||||
if checkpoint := getattr(adapter, "CHECKPOINT", None):
|
||||
result.update(weights=checkpoint["url"], checkpoint=dict(checkpoint))
|
||||
if recipe := getattr(adapter, "PREPROCESS", None):
|
||||
result["preprocess"] = dict(recipe)
|
||||
if runtime := getattr(adapter, "RUNTIME_PYTHON", None):
|
||||
result["runtime_python"] = str(runtime)
|
||||
return result
|
||||
|
||||
|
||||
def calibration_dir(row: dict) -> Path:
|
||||
from .. import paths
|
||||
directory = paths.DATA / "calib" / row["task"] / row["input_spec"]
|
||||
recipe = model_spec(row).get("preprocess")
|
||||
if recipe:
|
||||
# Equal image recipes share calibration; existing jobs retain their path.
|
||||
digest = hashlib.sha256(json.dumps(recipe, sort_keys=True).encode()).hexdigest()
|
||||
directory /= digest[:16]
|
||||
return directory
|
||||
|
||||
12
harness/adapters/aliked.py
Normal file
12
harness/adapters/aliked.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Original ALIKED-N16; static 1024 keypoints, all learned descriptor layers in graph."""
|
||||
from . import feature_common as common
|
||||
EXPORTER_VERSION = common.EXPORTER_VERSION
|
||||
INPUT_SPEC = common.input_spec('aliked')
|
||||
MODEL_SPEC = {'input_spec': INPUT_SPEC, 'dataset': 'hpatches', 'input_override_reason': 'Official ALIKED expects RGB; CSV grayscale is incorrect.'}
|
||||
CHECKPOINT = common.checkpoint_info('aliked')
|
||||
PREPROCESS = common.recipe('aliked')
|
||||
OPSET = 17
|
||||
preprocess = common.preprocess
|
||||
|
||||
def build(weights, input_spec):
|
||||
return common.build('aliked', weights, input_spec)
|
||||
12
harness/adapters/atss_r50_fpn.py
Normal file
12
harness/adapters/atss_r50_fpn.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Exact MMDetection atss.r50.fpn; see mmdet_dense_common for static deployment geometry."""
|
||||
from functools import partial
|
||||
from .mmdet_dense_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
|
||||
get_recipe, preprocess_variant, decode_variant, build_variant)
|
||||
|
||||
MODEL_ID = "atss.r50.fpn"
|
||||
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
|
||||
PREPROCESS = recipe(MODEL_ID)
|
||||
EXPORTER_VERSION = "mmdet-atss.r50.fpn-640-raw-v1"
|
||||
preprocess = partial(preprocess_variant, MODEL_ID)
|
||||
decode = partial(decode_variant, MODEL_ID)
|
||||
build = partial(build_variant, MODEL_ID)
|
||||
7
harness/adapters/bisenetv1_r18.py
Normal file
7
harness/adapters/bisenetv1_r18.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""Official Cityscapes bisenetv1.r18; admission requires real validation."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("bisenetv1.r18")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
build = partial(build_model, "bisenetv1.r18")
|
||||
7
harness/adapters/bisenetv2.py
Normal file
7
harness/adapters/bisenetv2.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""Official Cityscapes bisenetv2; admission requires real validation."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("bisenetv2")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
build = partial(build_model, "bisenetv2")
|
||||
114
harness/adapters/conditional_detr_r50.py
Normal file
114
harness/adapters/conditional_detr_r50.py
Normal file
@ -0,0 +1,114 @@
|
||||
"""Exact MMDetection conditional.detr.r50; see mmdet_detr_common for the
|
||||
static deployment geometry this batch reuses without editing it.
|
||||
|
||||
ConditionalDETR keeps the DETR single-scale encoder path (real per-image
|
||||
padding mask retained in-graph) and adds the official conditional decoder
|
||||
with learned query scaling and reference points. Outputs stay the final
|
||||
decoder layer's raw class logits and normalized cxcywh; only the official
|
||||
parameter-free DETRHead selection runs in host decode.
|
||||
"""
|
||||
from functools import partial
|
||||
from types import SimpleNamespace
|
||||
|
||||
from .openmmlab import build as build_official, get_recipe
|
||||
from .mmdet_detr_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, OPSET,
|
||||
transformer_inputs)
|
||||
from .mmdet_dense_common import (recipe as image_recipe,
|
||||
preprocess_variant as image_preprocess,
|
||||
geometry)
|
||||
|
||||
MODEL_ID = "conditional.detr.r50"
|
||||
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
|
||||
QUERIES, CHANNELS, TOPK = 300, 80, 100
|
||||
|
||||
|
||||
def recipe():
|
||||
dp = get_recipe(MODEL_ID)["preprocess"]["official_data_preprocessor"]
|
||||
meta = image_recipe("retinanet.r50.fpn")
|
||||
if (dp["mean"] != meta["mean"] or dp["std"] != meta["std"]
|
||||
or not dp["bgr_to_rgb"] or dp["pad_size_divisor"] != 1):
|
||||
raise ValueError("Official conditional-detr image normalization changed")
|
||||
# Same image transformation permits existing calibration cache reuse.
|
||||
return meta
|
||||
|
||||
|
||||
PREPROCESS = recipe()
|
||||
EXPORTER_VERSION = "mmdet-conditional.detr.r50-640-raw-v1"
|
||||
|
||||
|
||||
def preprocess(image, input_spec=INPUT_SPEC):
|
||||
if input_spec != INPUT_SPEC:
|
||||
raise ValueError(f"Expected {INPUT_SPEC}, got {input_spec}")
|
||||
recipe() # verify exact equality before reusing the image-only helper
|
||||
return image_preprocess("retinanet.r50.fpn", image, input_spec)
|
||||
|
||||
|
||||
def output_names():
|
||||
return ["class_logits", "boxes_cxcywh"]
|
||||
|
||||
|
||||
def _decoder():
|
||||
from mmdet.models.dense_heads.detr_head import DETRHead
|
||||
context = SimpleNamespace(test_cfg={"max_per_img": TOPK}, num_classes=80,
|
||||
loss_cls=SimpleNamespace(use_sigmoid=True))
|
||||
return partial(DETRHead._predict_by_feat_single, context)
|
||||
|
||||
|
||||
def decode(outputs, original_size):
|
||||
import numpy as np
|
||||
import torch
|
||||
if len(outputs) != 2:
|
||||
raise ValueError("Expected final query logits and cxcywh tensors")
|
||||
tensors = []
|
||||
for value, shape in zip(outputs, ((1, QUERIES, CHANNELS), (1, QUERIES, 4))):
|
||||
value = np.asarray(value)
|
||||
if value.shape != shape or not np.isfinite(value).all():
|
||||
raise ValueError(f"Invalid {MODEL_ID} raw output: {value.shape}, expected {shape}")
|
||||
tensors.append(torch.tensor(value, dtype=torch.float32)[0])
|
||||
_, scales, img_shape = geometry(original_size)
|
||||
with torch.inference_mode():
|
||||
result = _decoder()(*tensors, dict(img_shape=img_shape, scale_factor=scales), rescale=True)
|
||||
return result.bboxes.numpy(), result.scores.numpy(), result.labels.numpy()
|
||||
|
||||
|
||||
def build(weights, input_spec):
|
||||
import torch
|
||||
raw = build_official(MODEL_ID, weights, input_spec)
|
||||
network, cfg = raw.upstream, raw.cfg
|
||||
meta = recipe()
|
||||
dp = cfg.model.data_preprocessor
|
||||
if (type(network).__name__ != "ConditionalDETR"
|
||||
or type(network.bbox_head).__name__ != "ConditionalDETRHead"
|
||||
or network.num_queries != QUERIES
|
||||
or network.bbox_head.cls_out_channels != CHANNELS
|
||||
or network.bbox_head.num_classes != 80
|
||||
or dict(cfg.model.test_cfg) != {"max_per_img": TOPK}
|
||||
or list(dp.mean) != meta["mean"] or list(dp.std) != meta["std"]
|
||||
or not dp.bgr_to_rgb or dp.pad_size_divisor != 1):
|
||||
raise ValueError("Pinned official conditional-detr contract changed")
|
||||
state = torch.load(str(weights), map_location="cpu")["state_dict"]
|
||||
for name, parameter in network.named_parameters():
|
||||
torch.testing.assert_close(parameter, state[name], rtol=0, atol=0)
|
||||
del state
|
||||
|
||||
class RawQueries(torch.nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.upstream = network
|
||||
self.upstream.cfg = cfg
|
||||
self.cfg = cfg
|
||||
self.checkpoint_sha256 = raw.checkpoint_sha256
|
||||
|
||||
def forward(self, x):
|
||||
features = self.upstream.extract_feat(x)
|
||||
# Single-scale DETR path; the official conditional decoder consumes
|
||||
# the same memory/mask/pos tensors via its own reference points.
|
||||
encoder, decoder = transformer_inputs(self.upstream, features, x, False)
|
||||
memory = self.upstream.forward_encoder(**encoder)
|
||||
query, head_inputs = self.upstream.pre_decoder(**memory)
|
||||
decoder.update(query)
|
||||
head_inputs.update(self.upstream.forward_decoder(**decoder))
|
||||
scores, boxes = self.upstream.bbox_head(**head_inputs)
|
||||
return scores[-1], boxes[-1]
|
||||
|
||||
return RawQueries().eval()
|
||||
69
harness/adapters/d2_net.py
Normal file
69
harness/adapters/d2_net.py
Normal file
@ -0,0 +1,69 @@
|
||||
"""Official D2-Net d2_tf single-scale CNN with exact official host postprocessing."""
|
||||
from functools import lru_cache
|
||||
from . import feature_sources as sources
|
||||
|
||||
INPUT_SPEC = '1x3x480x640'
|
||||
MODEL_SPEC = {'input_spec': INPUT_SPEC, 'dataset': 'hpatches', 'input_override_reason': 'Official D2-Net Caffe preprocessing uses three BGR channels.'}
|
||||
EXPORTER_VERSION = 'd2net-static-1'
|
||||
OPSET = 17
|
||||
CHECKPOINT = {'url': 'https://dusmanu.com/files/d2-net/d2_tf.pth',
|
||||
'sha256': 'c2d32c3bc53b6588d40bc5325d536a159e529b7492198a2cfa4b11913c615c80',
|
||||
'filename': 'd2_tf.pth', 'official_relative_path': str(sources.ROOT/'d2net/models/d2_tf.pth')}
|
||||
PREPROCESS = {'input_spec': INPUT_SPEC, 'color': 'BGR', 'scale': 1., 'mean': [103.939,116.779,123.68],
|
||||
'resize': 'opencv_linear_640x480', 'output_layout': 'NCHW', 'protocol': 'd2net-official-caffe-single-scale-v1'}
|
||||
|
||||
|
||||
def official_module():
|
||||
return sources.module('d2net/lib/model_test.py')
|
||||
|
||||
|
||||
@lru_cache(None)
|
||||
def _functions():
|
||||
exception = sources.module('d2net/lib/exceptions.py').EmptyTensorError
|
||||
ns = sources.definitions('d2net/lib/utils.py', ['preprocess_image','interpolate_dense_features','upscale_positions'],
|
||||
{'EmptyTensorError': exception})
|
||||
return sources.definitions('d2net/lib/pyramid.py', ['process_multiscale'], ns)
|
||||
|
||||
|
||||
def preprocess(image, input_spec=INPUT_SPEC):
|
||||
import cv2
|
||||
import numpy as np
|
||||
if input_spec != INPUT_SPEC: raise ValueError(input_spec)
|
||||
rgb = cv2.resize(np.asarray(image.convert('RGB')), (640,480), interpolation=cv2.INTER_LINEAR)
|
||||
return np.ascontiguousarray(_functions()['preprocess_image'](rgb, preprocessing='caffe')[None], dtype=np.float32)
|
||||
|
||||
|
||||
def official_model(weights):
|
||||
import torch
|
||||
sources.verify_weights(weights, CHECKPOINT['sha256'])
|
||||
m = official_module().D2Net(use_cuda=False, use_relu=True)
|
||||
m.load_state_dict(torch.load(weights, map_location='cpu', weights_only=False)['model'], strict=True)
|
||||
return m.eval()
|
||||
|
||||
|
||||
def official_features(m, x):
|
||||
import torch
|
||||
k,s,d = _functions()['process_multiscale'](x, m, scales=[1])
|
||||
# The official extractor swaps (row,col) to (x,y) before saving.
|
||||
out = sources.top_features(k[:,[1,0]], s, d)
|
||||
return tuple(torch.from_numpy(out[key]) for key in ('keypoints','scores','descriptors'))
|
||||
|
||||
|
||||
def build(weights, input_spec):
|
||||
if input_spec != INPUT_SPEC: raise ValueError(input_spec)
|
||||
return official_model(weights).dense_feature_extraction
|
||||
|
||||
|
||||
def decode(outputs):
|
||||
import torch
|
||||
from types import SimpleNamespace
|
||||
dense, = sources.tensors(outputs)
|
||||
assert tuple(dense.shape) == (1,512,119,159)
|
||||
class DenseMap(torch.nn.Module):
|
||||
num_channels = 512
|
||||
def forward(self, x): return dense
|
||||
module = official_module()
|
||||
decoder = SimpleNamespace(dense_feature_extraction=DenseMap(), detection=module.HardDetectionModule(),
|
||||
localization=module.HandcraftedLocalizationModule())
|
||||
with torch.no_grad(): k,s,d = official_features(decoder, torch.zeros(1,3,480,640))
|
||||
return {'keypoints': k.numpy(), 'scores': s.numpy(), 'descriptors': d.numpy()}
|
||||
12
harness/adapters/dab_detr_r50.py
Normal file
12
harness/adapters/dab_detr_r50.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Exact MMDetection dab.detr.r50; see mmdet_detr_common for static deployment geometry."""
|
||||
from functools import partial
|
||||
from .mmdet_detr_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, OPSET, recipe,
|
||||
get_recipe, preprocess_variant, decode_variant, build_variant)
|
||||
|
||||
MODEL_ID = "dab.detr.r50"
|
||||
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
|
||||
PREPROCESS = recipe(MODEL_ID)
|
||||
EXPORTER_VERSION = "mmdet-dab.detr.r50-640-raw-v1"
|
||||
preprocess = partial(preprocess_variant, MODEL_ID)
|
||||
decode = partial(decode_variant, MODEL_ID)
|
||||
build = partial(build_variant, MODEL_ID)
|
||||
7
harness/adapters/ddrnet_23.py
Normal file
7
harness/adapters/ddrnet_23.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""Official Cityscapes ddrnet.23; exact static benchmark recipe."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("ddrnet.23")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
build = partial(build_model, "ddrnet.23")
|
||||
7
harness/adapters/ddrnet_23_slim.py
Normal file
7
harness/adapters/ddrnet_23_slim.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""Official Cityscapes ddrnet.23.slim; exact static benchmark recipe."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("ddrnet.23.slim")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
build = partial(build_model, "ddrnet.23.slim")
|
||||
74
harness/adapters/dedode_v2.py
Normal file
74
harness/adapters/dedode_v2.py
Normal file
@ -0,0 +1,74 @@
|
||||
"""Official DeDoDe v2 detector-L + descriptor-B; both learned nets in one graph."""
|
||||
from . import feature_sources3 as sources
|
||||
from .feature_common import MAX_KEYPOINTS
|
||||
|
||||
INPUT_SPEC='1x3x480x640'
|
||||
MODEL_SPEC={'input_spec':INPUT_SPEC,'dataset':'hpatches',
|
||||
'input_override_reason':'Official DeDoDe RGB ImageNet-normalized input; detector-L v2 plus descriptor-B.'}
|
||||
EXPORTER_VERSION='dedode-v2-l-b-static-1'
|
||||
OPSET=17
|
||||
CHECKPOINT={'url':'https://github.com/Parskatt/DeDoDe/releases/download/v2/dedode_detector_L_v2.pth',
|
||||
'sha256':'4113809dd9e0367af013a45fc2255a6b243ff241cd06520d17a65d9e231bdc17',
|
||||
'filename':'dedode_detector_L_v2.pth',
|
||||
'official_relative_path':str(sources.ROOT/'dedode/weights/dedode_detector_L_v2.pth')}
|
||||
DESCRIPTOR_CHECKPOINT={'url':'https://github.com/Parskatt/DeDoDe/releases/download/dedode_pretrained_models/dedode_descriptor_B.pth',
|
||||
'sha256':'8eccfc270ec990ced60cd54a434411a47c4c504de13586f596d042e005b3022b',
|
||||
'path':str(sources.ROOT/'dedode/weights/dedode_descriptor_B.pth')}
|
||||
PREPROCESS={'input_spec':INPUT_SPEC,'color':'RGB','scale':1/255,'mean':[.485,.456,.406],
|
||||
'std':[.229,.224,.225],'resize':'opencv_linear_640x480','output_layout':'NCHW',
|
||||
'protocol':'dedode-v2-official-rgb-l-b-static-v1'}
|
||||
|
||||
def preprocess(image,input_spec=INPUT_SPEC):
|
||||
import cv2
|
||||
import numpy as np
|
||||
if input_spec!=INPUT_SPEC:raise ValueError(input_spec)
|
||||
x=cv2.resize(np.asarray(image.convert('RGB')),(640,480),interpolation=cv2.INTER_LINEAR).astype(np.float64)/255
|
||||
x=(x-np.asarray(PREPROCESS['mean']))/np.asarray(PREPROCESS['std'])
|
||||
return np.ascontiguousarray(x.transpose(2,0,1)[None],dtype=np.float32)
|
||||
|
||||
def official_model(weights):
|
||||
import torch
|
||||
sources.check(weights,CHECKPOINT['sha256'])
|
||||
sources.check(DESCRIPTOR_CHECKPOINT['path'],DESCRIPTOR_CHECKPOINT['sha256'])
|
||||
module=sources.package('dedode','DeDoDe')
|
||||
detector=module.dedode_detector_L(device='cpu',weights=torch.load(weights,map_location='cpu',weights_only=True),remove_borders=False)
|
||||
descriptor=module.dedode_descriptor_B(device='cpu',weights=torch.load(DESCRIPTOR_CHECKPOINT['path'],map_location='cpu',weights_only=True))
|
||||
nets=torch.nn.ModuleDict({'detector':detector,'descriptor':descriptor}).eval()
|
||||
# The benchmark is FP32; CPU upstream autocast would be disabled anyway.
|
||||
for m in nets.modules():
|
||||
if hasattr(m,'amp'):m.amp=False
|
||||
return nets
|
||||
|
||||
def official_features(m,x):
|
||||
import torch.nn.functional as F
|
||||
module=sources.package('dedode','DeDoDe.utils')
|
||||
result=m['detector'].detect({'image':x},num_keypoints=MAX_KEYPOINTS)
|
||||
desc=m['descriptor'].describe_keypoints({'image':x},result['keypoints'])['descriptions']
|
||||
return module.to_pixel_coords(result['keypoints'],480,640)[0].cpu(),result['confidence'][0].cpu(),F.normalize(desc[0],dim=-1).cpu()
|
||||
|
||||
def build(weights,input_spec):
|
||||
import torch
|
||||
if input_spec!=INPUT_SPEC:raise ValueError(input_spec)
|
||||
class Graph(torch.nn.Module):
|
||||
def __init__(self):super().__init__();self.nets=official_model(weights)
|
||||
def forward(self,x):
|
||||
return self.nets['detector']({'image':x})['keypoint_logits'],self.nets['descriptor']({'image':x})['description_grid']
|
||||
return Graph().eval()
|
||||
|
||||
def decode(outputs):
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
logits,descriptors=[torch.as_tensor(np.asarray(o,np.float32)) for o in outputs]
|
||||
assert logits.shape==(1,1,480,640) and descriptors.shape==(1,256,480,640)
|
||||
assert torch.isfinite(logits).all() and torch.isfinite(descriptors).all()
|
||||
module=sources.package('dedode','DeDoDe.utils')
|
||||
# Exact detector.detect / descriptor.describe_keypoints operations; explicit
|
||||
# CPU map device allows host decoding even when a server GPU is visible.
|
||||
probability=logits.reshape(1,-1).softmax(-1).reshape(1,1,480*640).sum(1)
|
||||
k,s=module.sample_keypoints(probability.reshape(1,480,640),device=logits.device,
|
||||
use_nms=False,sample_topk=True,num_samples=MAX_KEYPOINTS,return_scoremap=True,
|
||||
sharpen=False,upsample=False,increase_coverage=True,remove_borders=False)
|
||||
d=F.grid_sample(descriptors,k[:,None],mode='bilinear',align_corners=False)[:,:,0].mT
|
||||
return {'keypoints':module.to_pixel_coords(k,480,640)[0].numpy(),
|
||||
'scores':s[0].numpy(),'descriptors':F.normalize(d[0],dim=-1).numpy()}
|
||||
7
harness/adapters/deeplabv3_resnet101.py
Normal file
7
harness/adapters/deeplabv3_resnet101.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""Exact official ADE20K DeepLabV3 ResNet101; admission requires actual validation."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("deeplabv3.resnet101")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
build = partial(build_model, "deeplabv3.resnet101")
|
||||
7
harness/adapters/deeplabv3_resnet50.py
Normal file
7
harness/adapters/deeplabv3_resnet50.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""Exact official ADE20K DeepLabV3 ResNet50; admission requires actual validation."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("deeplabv3.resnet50")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
build = partial(build_model, "deeplabv3.resnet50")
|
||||
7
harness/adapters/deeplabv3plus_resnet101.py
Normal file
7
harness/adapters/deeplabv3plus_resnet101.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""Exact official ADE20K DeepLabV3+ ResNet101; admission requires actual validation."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("deeplabv3plus.resnet101")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
build = partial(build_model, "deeplabv3plus.resnet101")
|
||||
7
harness/adapters/deeplabv3plus_resnet50.py
Normal file
7
harness/adapters/deeplabv3plus_resnet50.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""Exact official ADE20K DeepLabV3+ ResNet50; admission requires actual validation."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("deeplabv3plus.resnet50")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
build = partial(build_model, "deeplabv3plus.resnet50")
|
||||
12
harness/adapters/deformable_detr_r50.py
Normal file
12
harness/adapters/deformable_detr_r50.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Exact MMDetection deformable.detr.r50; see mmdet_detr_common for static deployment geometry."""
|
||||
from functools import partial
|
||||
from .mmdet_detr_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, OPSET, recipe,
|
||||
get_recipe, preprocess_variant, decode_variant, build_variant)
|
||||
|
||||
MODEL_ID = "deformable.detr.r50"
|
||||
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
|
||||
PREPROCESS = recipe(MODEL_ID)
|
||||
EXPORTER_VERSION = "mmdet-deformable.detr.r50-640-raw-v1"
|
||||
preprocess = partial(preprocess_variant, MODEL_ID)
|
||||
decode = partial(decode_variant, MODEL_ID)
|
||||
build = partial(build_variant, MODEL_ID)
|
||||
10
harness/adapters/depth_anything_v1_base.py
Normal file
10
harness/adapters/depth_anything_v1_base.py
Normal file
@ -0,0 +1,10 @@
|
||||
"""Official Depth Anything V1 base: relative inverse depth."""
|
||||
from pathlib import Path
|
||||
from .depth_anything_v1_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET, PREPROCESS,
|
||||
RESTORE_ALIGN_CORNERS, build_variant, checkpoint_info, preprocess)
|
||||
|
||||
CHECKPOINT = checkpoint_info("base")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("base", weights, input_spec)
|
||||
87
harness/adapters/depth_anything_v1_common.py
Normal file
87
harness/adapters/depth_anything_v1_common.py
Normal file
@ -0,0 +1,87 @@
|
||||
"""Official relative Depth Anything V1; pinned source and strict checkpoints."""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
from .depth_anything_v2_common import INPUT_SPEC, preprocess
|
||||
|
||||
UPSTREAM_COMMIT = "1d03336771fe09c5398ffdd211441e33941a97dc"
|
||||
SOURCE_TREE_SHA256 = "0196533fdd23b8031f85b7ea542bd0400a4c362916b16b2e5ea91678144f65b6"
|
||||
RUNTIME_ROOT = Path.home() / "bench-cache/sources/depth-anything-v1" / UPSTREAM_COMMIT
|
||||
MODEL_SPEC = {"source": "custom", "dataset": "nyuv2", "input_spec": INPUT_SPEC, "resize": 0}
|
||||
OPSET = 17
|
||||
EXPORTER_VERSION = "depth-anything-v1-relative-v1-opset17-static518"
|
||||
RESTORE_ALIGN_CORNERS = False
|
||||
PREPROCESS = {
|
||||
"id": "depth-anything-v1-rgb-static518-v1",
|
||||
"layout": "NCHW", "dtype": "float32", "color": "RGB",
|
||||
"resize": "OpenCV INTER_CUBIC to 518x518; keep_aspect_ratio=False for static deployment",
|
||||
"normalize": "RGB/255; mean=[0.485,0.456,0.406]; std=[0.229,0.224,0.225]",
|
||||
"output": "N,H,W nonnegative affine-invariant inverse depth; NOT metric depth",
|
||||
"restore": "bilinear to original H,W, align_corners=False; no min/max image normalization",
|
||||
"source_commit": UPSTREAM_COMMIT,
|
||||
}
|
||||
_MODELS = {
|
||||
"small": {"encoder": "vits", "features": 64, "out_channels": [48, 96, 192, 384],
|
||||
"revision": "1bee667edbfec7f9b345abaffec8c9c2bd5293c2",
|
||||
"sha256": "ba0ab5365610467de4fea1ffe64b4ae3a756561a4df9e11bd7369205241f2082"},
|
||||
"base": {"encoder": "vitb", "features": 128, "out_channels": [96, 192, 384, 768],
|
||||
"revision": "a7dff65359777209cb99e1361df8a76144f87e49",
|
||||
"sha256": "d25d4cf2b50e63198c42e684abb10b9d242714cec0fe3e2eb8f7da41fada25b3"},
|
||||
"large": {"encoder": "vitl", "features": 256, "out_channels": [256, 512, 1024, 1024],
|
||||
"revision": "973948530e4e4f4afd6d1913f670d9f96071dcaa",
|
||||
"sha256": "6fda5199f0043e91e1c6117df00313db57c630d8ed0306070b43da1ecfffbd9d"},
|
||||
}
|
||||
|
||||
|
||||
def checkpoint_info(variant: str) -> dict:
|
||||
info = _MODELS[variant]
|
||||
repo = f"LiheYoung/depth_anything_{info['encoder']}14"
|
||||
return {"url": f"https://huggingface.co/{repo}/resolve/{info['revision']}/pytorch_model.bin",
|
||||
"sha256": info["sha256"], "revision": info["revision"], "output_type": "relative_inverse_depth"}
|
||||
|
||||
|
||||
def source_root() -> Path:
|
||||
root = Path(os.environ.get("BENCH_DEPTH_ANYTHING_V1_SOURCE", str(RUNTIME_ROOT))).resolve()
|
||||
digest = hashlib.sha256()
|
||||
files = sorted(list((root / "depth_anything").rglob("*.py")) + list((root / "torchhub").rglob("*.py")))
|
||||
for path in files:
|
||||
digest.update(str(path.relative_to(root)).encode() + b"\0" + path.read_bytes())
|
||||
if digest.hexdigest() != SOURCE_TREE_SHA256:
|
||||
raise RuntimeError(f"missing or changed official Depth Anything V1 source {UPSTREAM_COMMIT}: {root}")
|
||||
return root
|
||||
|
||||
|
||||
def build_variant(variant: str, weights: Path, input_spec: str):
|
||||
import torch
|
||||
|
||||
if input_spec != INPUT_SPEC:
|
||||
raise ValueError(f"expected fixed {INPUT_SPEC}, got {input_spec}")
|
||||
root = source_root()
|
||||
weights = Path(weights).resolve()
|
||||
config = _MODELS[variant]
|
||||
digest = hashlib.sha256()
|
||||
with weights.open("rb") as stream:
|
||||
for block in iter(lambda: stream.read(1 << 20), b""):
|
||||
digest.update(block)
|
||||
if digest.hexdigest() != config["sha256"]:
|
||||
raise ValueError(f"wrong official relative Depth Anything V1 {variant} checkpoint")
|
||||
if str(root) not in sys.path:
|
||||
sys.path.insert(0, str(root))
|
||||
module = importlib.import_module("depth_anything.dpt")
|
||||
if Path(module.__file__).resolve() != root / "depth_anything/dpt.py":
|
||||
raise RuntimeError("a different Depth Anything V1 source is already imported")
|
||||
# Upstream localhub resolves its bundled DINOv2 relative to the checkout.
|
||||
# The harness builds each model in a separate worker process.
|
||||
previous = Path.cwd()
|
||||
try:
|
||||
os.chdir(root)
|
||||
model = module.DepthAnything({key: config[key] for key in ("encoder", "features", "out_channels")})
|
||||
finally:
|
||||
os.chdir(previous)
|
||||
model.load_state_dict(torch.load(weights, map_location="cpu", weights_only=True), strict=True)
|
||||
return model.eval()
|
||||
10
harness/adapters/depth_anything_v1_large.py
Normal file
10
harness/adapters/depth_anything_v1_large.py
Normal file
@ -0,0 +1,10 @@
|
||||
"""Official Depth Anything V1 large: relative inverse depth."""
|
||||
from pathlib import Path
|
||||
from .depth_anything_v1_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET, PREPROCESS,
|
||||
RESTORE_ALIGN_CORNERS, build_variant, checkpoint_info, preprocess)
|
||||
|
||||
CHECKPOINT = checkpoint_info("large")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("large", weights, input_spec)
|
||||
10
harness/adapters/depth_anything_v1_small.py
Normal file
10
harness/adapters/depth_anything_v1_small.py
Normal file
@ -0,0 +1,10 @@
|
||||
"""Official Depth Anything V1 small: relative inverse depth."""
|
||||
from pathlib import Path
|
||||
from .depth_anything_v1_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET, PREPROCESS,
|
||||
RESTORE_ALIGN_CORNERS, build_variant, checkpoint_info, preprocess)
|
||||
|
||||
CHECKPOINT = checkpoint_info("small")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("small", weights, input_spec)
|
||||
10
harness/adapters/depth_anything_v2_base.py
Normal file
10
harness/adapters/depth_anything_v2_base.py
Normal file
@ -0,0 +1,10 @@
|
||||
"""Official Depth Anything V2 base: affine-invariant inverse depth."""
|
||||
from pathlib import Path
|
||||
from .depth_anything_v2_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET, PREPROCESS,
|
||||
build_variant, checkpoint_info, preprocess)
|
||||
|
||||
CHECKPOINT = checkpoint_info("base")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("base", weights, input_spec)
|
||||
94
harness/adapters/depth_anything_v2_common.py
Normal file
94
harness/adapters/depth_anything_v2_common.py
Normal file
@ -0,0 +1,94 @@
|
||||
"""Pinned official Depth Anything V2 relative inverse-depth models.
|
||||
|
||||
The original source stays in a private checkout. No timm/transformers install,
|
||||
replacement architecture, metric-depth checkpoint, or synthetic weights.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
UPSTREAM_COMMIT = "a561b849ebae10a6f5ef49e26c83cbbcd36c71bf"
|
||||
SOURCE_TREE_SHA256 = "92af3e54e49dac0216ac039d7234c81082ec902329691e7aa57d30b2219f612b"
|
||||
RUNTIME_ROOT = Path.home() / "bench-cache/sources/depth-anything-v2" / UPSTREAM_COMMIT
|
||||
INPUT_SPEC = "1x3x518x518"
|
||||
MODEL_SPEC = {"source": "custom", "dataset": "nyuv2", "input_spec": INPUT_SPEC, "resize": 0}
|
||||
OPSET = 17
|
||||
EXPORTER_VERSION = "depth-anything-v2-relative-v1-opset17-static518"
|
||||
PREPROCESS = {
|
||||
"id": "depth-anything-v2-rgb-static518-v1",
|
||||
"layout": "NCHW", "dtype": "float32", "color": "RGB",
|
||||
"resize": "OpenCV INTER_CUBIC to 518x518; keep_aspect_ratio=False for static deployment",
|
||||
"normalize": "RGB/255; mean=[0.485,0.456,0.406]; std=[0.229,0.224,0.225]",
|
||||
"output": "N,H,W nonnegative affine-invariant inverse depth; NOT metric depth",
|
||||
"restore": "bilinear to original H,W, align_corners=True; no min/max image normalization",
|
||||
"source_commit": UPSTREAM_COMMIT,
|
||||
}
|
||||
|
||||
_MODELS = {
|
||||
"small": {"encoder": "vits", "features": 64, "out_channels": [48, 96, 192, 384],
|
||||
"revision": "03876f8651c73a60fe4c2c48294e09fcb6838fcf",
|
||||
"sha256": "715fade13be8f229f8a70cc02066f656f2423a59effd0579197bbf57860e1378"},
|
||||
"base": {"encoder": "vitb", "features": 128, "out_channels": [96, 192, 384, 768],
|
||||
"revision": "a4e71a6c2ce52fe50df0f212066b0d4a87be9b5e",
|
||||
"sha256": "0d2b7002e62d39d655571c371333340bd88f67ab95050c03591555aa05645328"},
|
||||
"large": {"encoder": "vitl", "features": 256, "out_channels": [256, 512, 1024, 1024],
|
||||
"revision": "cbbb86a30ce19b5684b7a05155dc7e6cbc7685b9",
|
||||
"sha256": "a7ea19fa0ed99244e67b624c72b8580b7e9553043245905be58796a608eb9345"},
|
||||
}
|
||||
|
||||
|
||||
def checkpoint_info(variant: str) -> dict:
|
||||
info = _MODELS[variant]
|
||||
filename = f"depth_anything_v2_{info['encoder']}.pth"
|
||||
repo = f"depth-anything/Depth-Anything-V2-{variant.capitalize()}"
|
||||
return {"url": f"https://huggingface.co/{repo}/resolve/{info['revision']}/{filename}",
|
||||
"sha256": info["sha256"], "revision": info["revision"], "output_type": "relative_inverse_depth"}
|
||||
|
||||
|
||||
def source_root() -> Path:
|
||||
root = Path(os.environ.get("BENCH_DEPTH_ANYTHING_V2_SOURCE", str(RUNTIME_ROOT))).resolve()
|
||||
digest = hashlib.sha256()
|
||||
for path in sorted((root / "depth_anything_v2").rglob("*.py")):
|
||||
digest.update(str(path.relative_to(root)).encode() + b"\0" + path.read_bytes())
|
||||
if digest.hexdigest() != SOURCE_TREE_SHA256:
|
||||
raise RuntimeError(f"missing or changed official Depth Anything V2 source {UPSTREAM_COMMIT}: {root}")
|
||||
return root
|
||||
|
||||
|
||||
def preprocess(image, input_spec: str = INPUT_SPEC):
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
if input_spec != INPUT_SPEC:
|
||||
raise ValueError(f"Depth Anything V2 recipe is fixed at {INPUT_SPEC}, got {input_spec}")
|
||||
rgb = np.asarray(image.convert("RGB"), dtype=np.float64) / 255.0
|
||||
rgb = cv2.resize(rgb, (518, 518), interpolation=cv2.INTER_CUBIC)
|
||||
rgb = (rgb - np.array([.485, .456, .406])) / np.array([.229, .224, .225])
|
||||
return np.ascontiguousarray(rgb.transpose(2, 0, 1)[None], dtype=np.float32)
|
||||
|
||||
|
||||
def build_variant(variant: str, weights: Path, input_spec: str):
|
||||
import torch
|
||||
|
||||
if input_spec != INPUT_SPEC:
|
||||
raise ValueError(f"expected fixed {INPUT_SPEC}, got {input_spec}")
|
||||
root = source_root()
|
||||
if str(root) not in sys.path:
|
||||
sys.path.insert(0, str(root))
|
||||
module = importlib.import_module("depth_anything_v2.dpt")
|
||||
if Path(module.__file__).resolve() != root / "depth_anything_v2/dpt.py":
|
||||
raise RuntimeError("a different Depth Anything V2 source is already imported")
|
||||
digest = hashlib.sha256()
|
||||
with Path(weights).open("rb") as stream:
|
||||
for block in iter(lambda: stream.read(1 << 20), b""):
|
||||
digest.update(block)
|
||||
config = _MODELS[variant]
|
||||
if digest.hexdigest() != config["sha256"]:
|
||||
raise ValueError(f"wrong official relative Depth Anything V2 {variant} checkpoint")
|
||||
model = module.DepthAnythingV2(**{key: config[key] for key in ("encoder", "features", "out_channels")})
|
||||
model.load_state_dict(torch.load(weights, map_location="cpu", weights_only=True), strict=True)
|
||||
return model.eval()
|
||||
10
harness/adapters/depth_anything_v2_large.py
Normal file
10
harness/adapters/depth_anything_v2_large.py
Normal file
@ -0,0 +1,10 @@
|
||||
"""Official Depth Anything V2 large: affine-invariant inverse depth."""
|
||||
from pathlib import Path
|
||||
from .depth_anything_v2_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET, PREPROCESS,
|
||||
build_variant, checkpoint_info, preprocess)
|
||||
|
||||
CHECKPOINT = checkpoint_info("large")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("large", weights, input_spec)
|
||||
120
harness/adapters/depth_anything_v2_metric_common.py
Normal file
120
harness/adapters/depth_anything_v2_metric_common.py
Normal file
@ -0,0 +1,120 @@
|
||||
"""Pinned official Depth Anything V2 metric-depth models (Hypersim / VKITTI).
|
||||
|
||||
Same pinned upstream checkout as the relative V2 adapters; the metric head
|
||||
lives in metric_depth/ of that tree. Sigmoid head times max_depth gives
|
||||
nonnegative metric depth in metres. Static 518x518 deployment resize with the
|
||||
official normalization; restore is the author's bilinear align_corners=True.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
UPSTREAM_COMMIT = "a561b849ebae10a6f5ef49e26c83cbbcd36c71bf"
|
||||
# sha256 over "<rel>\0<bytes>" of every *.py under metric_depth/ of the pinned V2 checkout.
|
||||
SOURCE_TREE_SHA256 = "5c2a223435082664012aa38ae11f3810738a4b6cba1efbb3c09a0405b8f6f17d"
|
||||
RUNTIME_ROOT = Path.home() / "bench-cache/sources/depth-anything-v2" / UPSTREAM_COMMIT
|
||||
INPUT_SPEC = "1x3x518x518"
|
||||
MODEL_SPEC = {"source": "custom", "dataset": "nyuv2", "input_spec": INPUT_SPEC, "resize": 0}
|
||||
OPSET = 17
|
||||
EXPORTER_VERSION = "depth-anything-v2-metric-v1-opset17-static518"
|
||||
RESTORE_ALIGN_CORNERS = True
|
||||
PREPROCESS = {
|
||||
"id": "depth-anything-v2-metric-rgb-static518-v1",
|
||||
"layout": "NCHW", "dtype": "float32", "color": "RGB",
|
||||
"resize": "OpenCV INTER_CUBIC to 518x518; keep_aspect_ratio=False for static deployment",
|
||||
"normalize": "RGB/255; mean=[0.485,0.456,0.406]; std=[0.229,0.224,0.225]",
|
||||
"output": "N,H,W nonnegative metric depth in metres (Sigmoid head times max_depth)",
|
||||
"restore": "bilinear to original H,W, align_corners=True; no min/max image normalization",
|
||||
"source_commit": UPSTREAM_COMMIT,
|
||||
}
|
||||
|
||||
_ENCODERS = {
|
||||
"small": {"encoder": "vits", "features": 64, "out_channels": [48, 96, 192, 384]},
|
||||
"base": {"encoder": "vitb", "features": 128, "out_channels": [96, 192, 384, 768]},
|
||||
"large": {"encoder": "vitl", "features": 256, "out_channels": [256, 512, 1024, 1024]},
|
||||
}
|
||||
_DOMAINS = {
|
||||
"hypersim": {"max_depth": 20.0, "dataset": "Hypersim (indoor)", "repo_tag": "Hypersim"},
|
||||
"vkitti": {"max_depth": 80.0, "dataset": "VKITTI (outdoor)", "repo_tag": "VKITTI"},
|
||||
}
|
||||
|
||||
_MODELS = {
|
||||
("hypersim", "small"): ("3bc65d4e14a6786a61acec16453c50e12bf5f338",
|
||||
"b782898d8a3e8be1f639de33837ed85e9b4b73e40f8f5e5cd99067588d722545"),
|
||||
("hypersim", "base"): ("4ebcd7fd0858cbb953451619ca943ea39ec08e00",
|
||||
"9dc9e274c5eff55c6daf27b660c0ced0eca4e8593a6da90cdcb04d2b4d3f3fa2"),
|
||||
("hypersim", "large"): ("79720800638389a78b2defc92caa885104f69974",
|
||||
"6f82ff2bc543ac02ddff4aa31fa363676a8305dd3ccf04e80e2af115a044cb6d"),
|
||||
("vkitti", "small"): ("c725b8589bdf6ab04072cab74c0467830db80d6d",
|
||||
"9203e538d35255c90dda4b7fedb47ff33fe725497bcca3b1e53b3a65ee63f0cb"),
|
||||
("vkitti", "base"): ("e96fd0c8cedf1825a55b9c61309f754982948df0",
|
||||
"4dad67a7cc10b462bca48e6b8569c762b8eb3c1adada170a3851a6d3ba37bb3e"),
|
||||
("vkitti", "large"): ("070e97e4b80e317ec1d03c19927304f4091a180b",
|
||||
"239b1054a369e66da2576e9a118d6d7c12d90dc8ebe609579a9a09cd8e05fe38"),
|
||||
}
|
||||
|
||||
|
||||
def checkpoint_info(domain: str, variant: str) -> dict:
|
||||
revision, sha256 = _MODELS[(domain, variant)]
|
||||
encoder = _ENCODERS[variant]["encoder"]
|
||||
filename = f"depth_anything_v2_metric_{domain}_{encoder}.pth"
|
||||
tag = _DOMAINS[domain]["repo_tag"]
|
||||
repo = f"depth-anything/Depth-Anything-V2-Metric-{tag}-{variant.capitalize()}"
|
||||
return {"url": f"https://huggingface.co/{repo}/resolve/{revision}/{filename}",
|
||||
"sha256": sha256, "revision": revision, "max_depth": _DOMAINS[domain]["max_depth"],
|
||||
"trained_on": _DOMAINS[domain]["dataset"], "output_type": "metric_depth_m"}
|
||||
|
||||
|
||||
def source_root() -> Path:
|
||||
root = Path(os.environ.get("BENCH_DEPTH_ANYTHING_V2_METRIC_SOURCE", str(RUNTIME_ROOT))).resolve()
|
||||
digest = hashlib.sha256()
|
||||
files = sorted(path for path in (root / "metric_depth").rglob("*.py")
|
||||
if "__pycache__" not in path.parts)
|
||||
if not files:
|
||||
raise RuntimeError(f"missing official Depth Anything V2 metric source {UPSTREAM_COMMIT}: {root}")
|
||||
for path in files:
|
||||
digest.update(str(path.relative_to(root)).encode() + b"\0" + path.read_bytes())
|
||||
if digest.hexdigest() != SOURCE_TREE_SHA256:
|
||||
raise RuntimeError(f"missing or changed official Depth Anything V2 metric source {UPSTREAM_COMMIT}: {root}")
|
||||
return root
|
||||
|
||||
|
||||
def preprocess(image, input_spec: str = INPUT_SPEC):
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
if input_spec != INPUT_SPEC:
|
||||
raise ValueError(f"Depth Anything V2 metric recipe is fixed at {INPUT_SPEC}, got {input_spec}")
|
||||
rgb = np.asarray(image.convert("RGB"), dtype=np.float64) / 255.0
|
||||
rgb = cv2.resize(rgb, (518, 518), interpolation=cv2.INTER_CUBIC)
|
||||
rgb = (rgb - np.array([.485, .456, .406])) / np.array([.229, .224, .225])
|
||||
return np.ascontiguousarray(rgb.transpose(2, 0, 1)[None], dtype=np.float32)
|
||||
|
||||
|
||||
def build_variant(domain: str, variant: str, weights: Path, input_spec: str):
|
||||
import torch
|
||||
|
||||
if input_spec != INPUT_SPEC:
|
||||
raise ValueError(f"expected fixed {INPUT_SPEC}, got {input_spec}")
|
||||
root = source_root()
|
||||
config = _MODELS[(domain, variant)]
|
||||
digest = hashlib.sha256()
|
||||
with Path(weights).open("rb") as stream:
|
||||
for block in iter(lambda: stream.read(1 << 20), b""):
|
||||
digest.update(block)
|
||||
if digest.hexdigest() != config[1]:
|
||||
raise ValueError(f"wrong official metric Depth Anything V2 {domain}-{variant} checkpoint")
|
||||
if str(root) not in sys.path:
|
||||
sys.path.insert(0, str(root))
|
||||
module = importlib.import_module("metric_depth.depth_anything_v2.dpt")
|
||||
if Path(module.__file__).resolve() != root / "metric_depth/depth_anything_v2/dpt.py":
|
||||
raise RuntimeError("a different Depth Anything V2 metric source is already imported")
|
||||
model = module.DepthAnythingV2(**{key: value for key, value in _ENCODERS[variant].items()
|
||||
if key in ("encoder", "features", "out_channels")},
|
||||
max_depth=_DOMAINS[domain]["max_depth"])
|
||||
model.load_state_dict(torch.load(weights, map_location="cpu", weights_only=True), strict=True)
|
||||
return model.eval()
|
||||
12
harness/adapters/depth_anything_v2_metric_hypersim_base.py
Normal file
12
harness/adapters/depth_anything_v2_metric_hypersim_base.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Official metric Depth Anything V2 Hypersim-base: metric depth, static 518."""
|
||||
from pathlib import Path
|
||||
|
||||
from .depth_anything_v2_metric_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET,
|
||||
PREPROCESS, RESTORE_ALIGN_CORNERS, build_variant,
|
||||
checkpoint_info, preprocess)
|
||||
|
||||
CHECKPOINT = checkpoint_info("hypersim", "base")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("hypersim", "base", weights, input_spec)
|
||||
12
harness/adapters/depth_anything_v2_metric_hypersim_large.py
Normal file
12
harness/adapters/depth_anything_v2_metric_hypersim_large.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Official metric Depth Anything V2 Hypersim-large: metric depth, static 518."""
|
||||
from pathlib import Path
|
||||
|
||||
from .depth_anything_v2_metric_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET,
|
||||
PREPROCESS, RESTORE_ALIGN_CORNERS, build_variant,
|
||||
checkpoint_info, preprocess)
|
||||
|
||||
CHECKPOINT = checkpoint_info("hypersim", "large")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("hypersim", "large", weights, input_spec)
|
||||
12
harness/adapters/depth_anything_v2_metric_hypersim_small.py
Normal file
12
harness/adapters/depth_anything_v2_metric_hypersim_small.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Official metric Depth Anything V2 Hypersim-small: metric depth, static 518."""
|
||||
from pathlib import Path
|
||||
|
||||
from .depth_anything_v2_metric_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET,
|
||||
PREPROCESS, RESTORE_ALIGN_CORNERS, build_variant,
|
||||
checkpoint_info, preprocess)
|
||||
|
||||
CHECKPOINT = checkpoint_info("hypersim", "small")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("hypersim", "small", weights, input_spec)
|
||||
12
harness/adapters/depth_anything_v2_metric_vkitti_base.py
Normal file
12
harness/adapters/depth_anything_v2_metric_vkitti_base.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Official metric Depth Anything V2 VKITTI-base: metric depth, static 518."""
|
||||
from pathlib import Path
|
||||
|
||||
from .depth_anything_v2_metric_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET,
|
||||
PREPROCESS, RESTORE_ALIGN_CORNERS, build_variant,
|
||||
checkpoint_info, preprocess)
|
||||
|
||||
CHECKPOINT = checkpoint_info("vkitti", "base")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("vkitti", "base", weights, input_spec)
|
||||
12
harness/adapters/depth_anything_v2_metric_vkitti_large.py
Normal file
12
harness/adapters/depth_anything_v2_metric_vkitti_large.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Official metric Depth Anything V2 VKITTI-large: metric depth, static 518."""
|
||||
from pathlib import Path
|
||||
|
||||
from .depth_anything_v2_metric_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET,
|
||||
PREPROCESS, RESTORE_ALIGN_CORNERS, build_variant,
|
||||
checkpoint_info, preprocess)
|
||||
|
||||
CHECKPOINT = checkpoint_info("vkitti", "large")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("vkitti", "large", weights, input_spec)
|
||||
12
harness/adapters/depth_anything_v2_metric_vkitti_small.py
Normal file
12
harness/adapters/depth_anything_v2_metric_vkitti_small.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Official metric Depth Anything V2 VKITTI-small: metric depth, static 518."""
|
||||
from pathlib import Path
|
||||
|
||||
from .depth_anything_v2_metric_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET,
|
||||
PREPROCESS, RESTORE_ALIGN_CORNERS, build_variant,
|
||||
checkpoint_info, preprocess)
|
||||
|
||||
CHECKPOINT = checkpoint_info("vkitti", "small")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("vkitti", "small", weights, input_spec)
|
||||
10
harness/adapters/depth_anything_v2_small.py
Normal file
10
harness/adapters/depth_anything_v2_small.py
Normal file
@ -0,0 +1,10 @@
|
||||
"""Official Depth Anything V2 small: affine-invariant inverse depth."""
|
||||
from pathlib import Path
|
||||
from .depth_anything_v2_common import (EXPORTER_VERSION, INPUT_SPEC, MODEL_SPEC, OPSET, PREPROCESS,
|
||||
build_variant, checkpoint_info, preprocess)
|
||||
|
||||
CHECKPOINT = checkpoint_info("small")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("small", weights, input_spec)
|
||||
12
harness/adapters/detr_r50.py
Normal file
12
harness/adapters/detr_r50.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Exact MMDetection detr.r50; see mmdet_detr_common for static deployment geometry."""
|
||||
from functools import partial
|
||||
from .mmdet_detr_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, OPSET, recipe,
|
||||
get_recipe, preprocess_variant, decode_variant, build_variant)
|
||||
|
||||
MODEL_ID = "detr.r50"
|
||||
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
|
||||
PREPROCESS = recipe(MODEL_ID)
|
||||
EXPORTER_VERSION = "mmdet-detr.r50-640-raw-v1"
|
||||
preprocess = partial(preprocess_variant, MODEL_ID)
|
||||
decode = partial(decode_variant, MODEL_ID)
|
||||
build = partial(build_variant, MODEL_ID)
|
||||
12
harness/adapters/disk.py
Normal file
12
harness/adapters/disk.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Original EPFL DISK depth checkpoint/Unet; official host NMS and descriptor lookup."""
|
||||
from . import feature_common as common
|
||||
EXPORTER_VERSION = common.EXPORTER_VERSION
|
||||
INPUT_SPEC = common.input_spec('disk')
|
||||
MODEL_SPEC = {'input_spec': INPUT_SPEC, 'dataset': 'hpatches', 'input_override_reason': 'Official DISK expects RGB; CSV grayscale is incorrect.'}
|
||||
CHECKPOINT = common.checkpoint_info('disk')
|
||||
PREPROCESS = common.recipe('disk')
|
||||
OPSET = 17
|
||||
preprocess = common.preprocess
|
||||
|
||||
def build(weights, input_spec):
|
||||
return common.build('disk', weights, input_spec)
|
||||
19
harness/adapters/dpt_hybrid.py
Normal file
19
harness/adapters/dpt_hybrid.py
Normal file
@ -0,0 +1,19 @@
|
||||
"""Official MiDaS v3 DPT-Hybrid: relative inverse depth at static 384x384."""
|
||||
from pathlib import Path
|
||||
|
||||
from .midas_common import (EXPORTER_VERSION, INPUT_SPECS, MODEL_SPECS, OPSET, PREPROCESSES,
|
||||
RESTORE_ALIGN_CORNERS, RESTORE_INTERPOLATION, build_variant,
|
||||
checkpoint_info, preprocess_for)
|
||||
|
||||
CHECKPOINT = checkpoint_info("hybrid")
|
||||
INPUT_SPEC = INPUT_SPECS["hybrid"]
|
||||
MODEL_SPEC = MODEL_SPECS["hybrid"]
|
||||
PREPROCESS = PREPROCESSES["hybrid"]
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("hybrid", weights, input_spec)
|
||||
|
||||
|
||||
def preprocess(image, input_spec: str = INPUT_SPEC):
|
||||
return preprocess_for(image, input_spec, "hybrid")
|
||||
19
harness/adapters/dpt_large.py
Normal file
19
harness/adapters/dpt_large.py
Normal file
@ -0,0 +1,19 @@
|
||||
"""Official MiDaS v3 DPT-Large: relative inverse depth at static 384x384."""
|
||||
from pathlib import Path
|
||||
|
||||
from .midas_common import (EXPORTER_VERSION, INPUT_SPECS, MODEL_SPECS, OPSET, PREPROCESSES,
|
||||
RESTORE_ALIGN_CORNERS, RESTORE_INTERPOLATION, build_variant,
|
||||
checkpoint_info, preprocess_for)
|
||||
|
||||
CHECKPOINT = checkpoint_info("large")
|
||||
INPUT_SPEC = INPUT_SPECS["large"]
|
||||
MODEL_SPEC = MODEL_SPECS["large"]
|
||||
PREPROCESS = PREPROCESSES["large"]
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("large", weights, input_spec)
|
||||
|
||||
|
||||
def preprocess(image, input_spec: str = INPUT_SPEC):
|
||||
return preprocess_for(image, input_spec, "large")
|
||||
10
harness/adapters/efficientvit_b0.py
Normal file
10
harness/adapters/efficientvit_b0.py
Normal file
@ -0,0 +1,10 @@
|
||||
"""CSV efficientvit.b0: MIT Han Lab ImageNet-1K B0, official r224 checkpoint."""
|
||||
from pathlib import Path
|
||||
|
||||
from .efficientvit_common import PREPROCESS, build_variant, checkpoint_info, preprocess
|
||||
|
||||
CHECKPOINT = checkpoint_info("b0")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("b0", weights, input_spec)
|
||||
10
harness/adapters/efficientvit_b1.py
Normal file
10
harness/adapters/efficientvit_b1.py
Normal file
@ -0,0 +1,10 @@
|
||||
"""CSV efficientvit.b1: MIT Han Lab ImageNet-1K B1, official r224 checkpoint."""
|
||||
from pathlib import Path
|
||||
|
||||
from .efficientvit_common import PREPROCESS, build_variant, checkpoint_info, preprocess
|
||||
|
||||
CHECKPOINT = checkpoint_info("b1")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("b1", weights, input_spec)
|
||||
10
harness/adapters/efficientvit_b2.py
Normal file
10
harness/adapters/efficientvit_b2.py
Normal file
@ -0,0 +1,10 @@
|
||||
"""CSV efficientvit.b2: MIT Han Lab ImageNet-1K B2, official r224 checkpoint."""
|
||||
from pathlib import Path
|
||||
|
||||
from .efficientvit_common import PREPROCESS, build_variant, checkpoint_info, preprocess
|
||||
|
||||
CHECKPOINT = checkpoint_info("b2")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("b2", weights, input_spec)
|
||||
10
harness/adapters/efficientvit_b3.py
Normal file
10
harness/adapters/efficientvit_b3.py
Normal file
@ -0,0 +1,10 @@
|
||||
"""CSV efficientvit.b3: MIT Han Lab ImageNet-1K B3, official r224 checkpoint."""
|
||||
from pathlib import Path
|
||||
|
||||
from .efficientvit_common import PREPROCESS, build_variant, checkpoint_info, preprocess
|
||||
|
||||
CHECKPOINT = checkpoint_info("b3")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("b3", weights, input_spec)
|
||||
163
harness/adapters/efficientvit_common.py
Normal file
163
harness/adapters/efficientvit_common.py
Normal file
@ -0,0 +1,163 @@
|
||||
"""MIT Han Lab EfficientViT ImageNet classifiers, using the official factory.
|
||||
|
||||
No downloads, package installation, key remapping, or architecture fallback occur
|
||||
here. The pinned official package and its dependencies live in a private cache;
|
||||
the shared venv is unchanged. Missing files remain errors, not readiness evidence.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
UPSTREAM_REPOSITORY = "https://github.com/mit-han-lab/efficientvit"
|
||||
# The original classification release avoids later SAM/DC_AE eager imports.
|
||||
# Its unmodified factory strictly accepts all four official r224 checkpoints.
|
||||
UPSTREAM_COMMIT = "b3ed56d97ac8ff0cc6add7ad2533c42a7488d736"
|
||||
RUNTIME_ROOT = Path.home() / "bench-cache/sources/efficientvit-cls"
|
||||
CHECKPOINT_REPOSITORY = "han-cai/efficientvit-cls"
|
||||
CHECKPOINT_REVISION = "df3d006c2567f9e322b03731f20fe4405a1ab090"
|
||||
INPUT_SPEC = "1x3x224x224"
|
||||
|
||||
# Hashes are the HF LFS object IDs, also verified against downloaded bytes.
|
||||
_CHECKPOINTS = {
|
||||
"b0": (13741687, "feebc4d3ebf610673c8ed1d92b21fa2c2e57661868173a37126b734ca5dc3771"),
|
||||
"b1": (36575137, "bf8798aa03ed2bba21fc8d58f43ca9a92096f3f03bd700e1df1dcde11559f499"),
|
||||
"b2": (97581671, "6ccfecc65460856cb6415f2356deb8b90e0216960f661f5055aef173f8ca421d"),
|
||||
"b3": (194992573, "3c5be1a11a8a41b3ac4e30e5bff8bf99065c98b4b2d586e1803fd7571f76fb0d"),
|
||||
}
|
||||
|
||||
# Check the actual architecture, factory and loader, not just a package name.
|
||||
# This pins core source identity; it is not a full environment attestation.
|
||||
_SOURCE_SHA256 = {
|
||||
"__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
||||
"cls_model_zoo.py": "c5e21fa447add578968ce6ed179c6e877aa3414b46f1d21af45e947610030599",
|
||||
"models/efficientvit/__init__.py": "6ec6831645b992d01f479174f55df2c026a05adf099a9eebb45b9690c84b8051",
|
||||
"models/efficientvit/cls.py": "41534fcd771aa42d9db2cb95d7580dfdf5ac5d54b57b2df8ac2b35dd4be7414e",
|
||||
"models/efficientvit/backbone.py": "f30d4460b3c2000421fc3421d1cbb3d501233e13fa4f80f9df7bbc7ed5664863",
|
||||
"models/nn/ops.py": "a5c177a1d3f9d1d8c85647c57d01295506f84ba53b763b0ba77095be34e174e0",
|
||||
"models/nn/norm.py": "45d34b3b5002c8d76557ed3017c02ccfc31723f2d84efd2f0cd7074706132cf5",
|
||||
"models/nn/act.py": "7b71a6626dd7ebd7bf0cba283f3f3f5ec3da6ebec0d7c04b2fbafa1969abd7f8",
|
||||
"models/utils/network.py": "db63b048a48c1edd47e6f401d0cc7d96dfc388b7e37f568acdef5c7a83acc5c4",
|
||||
}
|
||||
|
||||
PREPROCESS = {
|
||||
"input_spec": INPUT_SPEC,
|
||||
"color": "RGB",
|
||||
"crop_ratio": 0.95,
|
||||
"resize_shorter": 236, # ceil(224 / 0.95)
|
||||
"interpolation": "bicubic",
|
||||
"center_crop": 224,
|
||||
"scale": 1.0 / 255.0,
|
||||
"mean": (0.485, 0.456, 0.406),
|
||||
"std": (0.229, 0.224, 0.225),
|
||||
"output_layout": "NCHW",
|
||||
}
|
||||
|
||||
|
||||
def checkpoint_info(variant: str) -> dict:
|
||||
"""Return an immutable-revision HTTPS URI suitable for existing fetch_http."""
|
||||
if variant not in _CHECKPOINTS:
|
||||
raise ValueError(f"Unsupported MIT EfficientViT classification variant: {variant!r}")
|
||||
size, sha256 = _CHECKPOINTS[variant]
|
||||
filename = f"efficientvit_{variant}_r224.pt"
|
||||
return {
|
||||
"model_id": f"efficientvit.{variant}",
|
||||
"factory_name": f"{variant}-r224",
|
||||
"filename": filename,
|
||||
"url": f"https://huggingface.co/{CHECKPOINT_REPOSITORY}/resolve/{CHECKPOINT_REVISION}/{filename}",
|
||||
"size": size,
|
||||
"sha256": sha256,
|
||||
}
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1 << 20), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def verify_checkpoint(variant: str, weights: Path) -> dict:
|
||||
"""Reject HTML, wrong resolutions/variants, and converted or random weights."""
|
||||
info = checkpoint_info(variant)
|
||||
weights = Path(weights)
|
||||
if not weights.is_file():
|
||||
raise FileNotFoundError(f"Official checkpoint required: {weights}; source: {info['url']}")
|
||||
if weights.stat().st_size != info["size"] or _sha256(weights) != info["sha256"]:
|
||||
raise ValueError(
|
||||
f"Checkpoint identity mismatch for {info['model_id']}: {weights}; "
|
||||
f"expected {info['size']} bytes, SHA256 {info['sha256']} from {info['url']}"
|
||||
)
|
||||
return info
|
||||
|
||||
|
||||
def _check_input_spec(input_spec: str) -> None:
|
||||
if input_spec != INPUT_SPEC:
|
||||
raise ValueError(f"This adapter pins the official r224 recipe: expected {INPUT_SPEC}, got {input_spec!r}")
|
||||
|
||||
|
||||
def _official_factory():
|
||||
source = RUNTIME_ROOT / f"efficientvit-{UPSTREAM_COMMIT}"
|
||||
root = source / "efficientvit"
|
||||
for relative, expected in _SOURCE_SHA256.items():
|
||||
path = root / relative
|
||||
if not path.is_file() or _sha256(path) != expected:
|
||||
raise RuntimeError(
|
||||
f"Unverified EfficientViT source: {path}; expected core source from "
|
||||
f"{UPSTREAM_REPOSITORY}/tree/{UPSTREAM_COMMIT} (SHA256 {expected})."
|
||||
)
|
||||
# Only this process sees the private dependencies. Restore the import path
|
||||
# after upstream's imports; never change PYTHONPATH or the shared venv.
|
||||
original_path = sys.path[:]
|
||||
try:
|
||||
sys.path[:0] = [str(source), str(RUNTIME_ROOT / "deps")]
|
||||
spec = importlib.util.find_spec("efficientvit")
|
||||
if spec is None or spec.origin is None or Path(spec.origin).resolve() != (root / "__init__.py").resolve():
|
||||
raise RuntimeError("A different EfficientViT package is already loaded; use a fresh worker.")
|
||||
module = importlib.import_module("efficientvit.cls_model_zoo")
|
||||
return module.create_cls_model
|
||||
except ImportError as exc:
|
||||
raise RuntimeError(
|
||||
f"Official EfficientViT import failed ({exc}). Prepare pinned private dependencies "
|
||||
f"under {RUNTIME_ROOT / 'deps'}. No fallback is used."
|
||||
) from exc
|
||||
finally:
|
||||
sys.path[:] = original_path
|
||||
|
||||
|
||||
def build_variant(variant: str, weights: Path, input_spec: str):
|
||||
"""Load a hash-verified checkpoint via upstream's strict state_dict loader.
|
||||
|
||||
Upstream keeps norm epsilon at 1e-5 for the B series and reads the nested
|
||||
state_dict with torch.load(map_location='cpu'), then load_state_dict(strict=True).
|
||||
"""
|
||||
_check_input_spec(input_spec)
|
||||
info = verify_checkpoint(variant, weights)
|
||||
factory = _official_factory()
|
||||
model = factory(
|
||||
name=info["factory_name"], pretrained=True, weight_url=str(Path(weights).resolve())
|
||||
)
|
||||
return model.eval()
|
||||
|
||||
|
||||
def preprocess(image, input_spec: str = INPUT_SPEC):
|
||||
"""Official PIL/torchvision evaluation transform, returned as float32 NCHW.
|
||||
|
||||
Shared by baseline, calibration, G2 and quantized evaluation through the
|
||||
harness adapter hook; it is not an in-model normalization wrapper.
|
||||
"""
|
||||
_check_input_spec(input_spec)
|
||||
from torchvision import transforms
|
||||
from torchvision.transforms.functional import InterpolationMode
|
||||
|
||||
transform = transforms.Compose([
|
||||
transforms.Resize(236, interpolation=InterpolationMode.BICUBIC),
|
||||
transforms.CenterCrop(224),
|
||||
transforms.ToTensor(),
|
||||
transforms.Normalize(mean=PREPROCESS["mean"], std=PREPROCESS["std"]),
|
||||
])
|
||||
return transform(image.convert("RGB")).unsqueeze(0).numpy()
|
||||
14
harness/adapters/efficientvit_seg_b0.py
Normal file
14
harness/adapters/efficientvit_seg_b0.py
Normal file
@ -0,0 +1,14 @@
|
||||
"""Official MIT EfficientViT segmentation B0; readiness requires a real probe."""
|
||||
from pathlib import Path
|
||||
|
||||
from .efficientvit_seg_common import (
|
||||
build_variant, checkpoint_info, model_spec, preprocess, preprocess_recipe,
|
||||
)
|
||||
|
||||
CHECKPOINT = checkpoint_info("b0")
|
||||
MODEL_SPEC = model_spec("b0")
|
||||
PREPROCESS = preprocess_recipe("b0")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("b0", weights, input_spec)
|
||||
14
harness/adapters/efficientvit_seg_b1.py
Normal file
14
harness/adapters/efficientvit_seg_b1.py
Normal file
@ -0,0 +1,14 @@
|
||||
"""Official MIT EfficientViT segmentation B1; readiness requires a real probe."""
|
||||
from pathlib import Path
|
||||
|
||||
from .efficientvit_seg_common import (
|
||||
build_variant, checkpoint_info, model_spec, preprocess, preprocess_recipe,
|
||||
)
|
||||
|
||||
CHECKPOINT = checkpoint_info("b1")
|
||||
MODEL_SPEC = model_spec("b1")
|
||||
PREPROCESS = preprocess_recipe("b1")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("b1", weights, input_spec)
|
||||
14
harness/adapters/efficientvit_seg_b2.py
Normal file
14
harness/adapters/efficientvit_seg_b2.py
Normal file
@ -0,0 +1,14 @@
|
||||
"""Official MIT EfficientViT segmentation B2; readiness requires a real probe."""
|
||||
from pathlib import Path
|
||||
|
||||
from .efficientvit_seg_common import (
|
||||
build_variant, checkpoint_info, model_spec, preprocess, preprocess_recipe,
|
||||
)
|
||||
|
||||
CHECKPOINT = checkpoint_info("b2")
|
||||
MODEL_SPEC = model_spec("b2")
|
||||
PREPROCESS = preprocess_recipe("b2")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("b2", weights, input_spec)
|
||||
14
harness/adapters/efficientvit_seg_b3.py
Normal file
14
harness/adapters/efficientvit_seg_b3.py
Normal file
@ -0,0 +1,14 @@
|
||||
"""Official MIT EfficientViT segmentation B3; readiness requires a real probe."""
|
||||
from pathlib import Path
|
||||
|
||||
from .efficientvit_seg_common import (
|
||||
build_variant, checkpoint_info, model_spec, preprocess, preprocess_recipe,
|
||||
)
|
||||
|
||||
CHECKPOINT = checkpoint_info("b3")
|
||||
MODEL_SPEC = model_spec("b3")
|
||||
PREPROCESS = preprocess_recipe("b3")
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("b3", weights, input_spec)
|
||||
94
harness/adapters/efficientvit_seg_common.py
Normal file
94
harness/adapters/efficientvit_seg_common.py
Normal file
@ -0,0 +1,94 @@
|
||||
"""Official MIT EfficientViT segmentation heads and fixed board input recipe.
|
||||
|
||||
Reuse the unmodified 2023 package/private deps prepared for classification.
|
||||
Weights are immutable-revision official releases, never classification weights.
|
||||
The bench keeps its static 512x1024 input for both datasets; this is not the
|
||||
published full-resolution Cityscapes / variable-aspect ADE20K evaluation.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
from pathlib import Path
|
||||
|
||||
from .efficientvit_common import RUNTIME_ROOT, UPSTREAM_COMMIT, _official_factory, _sha256
|
||||
|
||||
SOURCE_COMMIT = "b3ed56d97ac8ff0cc6add7ad2533c42a7488d736"
|
||||
CHECKPOINT_REPOSITORY = "han-cai/efficientvit-seg"
|
||||
CHECKPOINT_REVISION = "cf3ccaf9cbaf670a2cb283612773546824ca0aa1"
|
||||
INPUT_SPEC = "1x3x512x1024"
|
||||
_SOURCE_SHA256 = {
|
||||
"seg_model_zoo.py": "8cd5e99445dbf8c1270c37f93f6f4eca0ac3fefe66f263f325e0e23d50222922",
|
||||
"models/efficientvit/seg.py": "a279c103c57f467de1eb93c7db3c170872a48faaf618aff58ada1af6bd5fd8bb",
|
||||
}
|
||||
_CHECKPOINTS = {
|
||||
"b0": ("cityscapes", 19, 2924013, "923d6fdd5e93640cc0c2f3f213764f34e80b477cd98a6b294d870ea6df5acc50"),
|
||||
"b1": ("ade20k", 150, 19339726, "59cca6a7ff4d1a6494b7f12072c6a68fee40ef401e971f26af01a2e0f3c8f5e7"),
|
||||
"b2": ("ade20k", 150, 61436227, "fabfa7163e35d14888690969cd852069f399137be39697bfe5165f03ea76b510"),
|
||||
"b3": ("ade20k", 150, 158594800, "fd2706e63ae9d425eb022bce26e10cebb56edd347644003f32f3ac0c9c3855a2"),
|
||||
}
|
||||
|
||||
|
||||
def checkpoint_info(variant: str) -> dict:
|
||||
dataset, classes, size, sha256 = _CHECKPOINTS[variant]
|
||||
filename = f"efficientvit_seg_{variant}_{dataset}.pt"
|
||||
return {"model_id": f"efficientvit.seg.{variant}", "factory_name": variant,
|
||||
"dataset": dataset, "classes": classes, "filename": filename,
|
||||
"url": f"https://huggingface.co/{CHECKPOINT_REPOSITORY}/resolve/{CHECKPOINT_REVISION}/{filename}",
|
||||
"size": size, "sha256": sha256, "source_commit": SOURCE_COMMIT}
|
||||
|
||||
|
||||
def model_spec(variant: str) -> dict:
|
||||
info = checkpoint_info(variant)
|
||||
result = {"dataset": info["dataset"], "input_spec": INPUT_SPEC}
|
||||
if variant == "b0":
|
||||
result["dataset_override_reason"] = (
|
||||
"CSV says ADE20K; official B0 segmentation factory/weights only support Cityscapes (19 classes)."
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def preprocess_recipe(variant: str) -> dict:
|
||||
return {"input_spec": INPUT_SPEC, "dataset": checkpoint_info(variant)["dataset"],
|
||||
"color": "RGB", "resize": "fixed_shape", "interpolation": "opencv_inter_cubic",
|
||||
"scale": 1.0 / 255.0, "mean": [0.485, 0.456, 0.406], "std": [0.229, 0.224, 0.225],
|
||||
"output_layout": "NCHW", "logits_resize": "bicubic", "align_corners": False,
|
||||
"protocol": "bench-static-resize-v1; logits resized to original label size before argmax"}
|
||||
|
||||
|
||||
def input_shape(input_spec: str) -> tuple[int, ...]:
|
||||
if input_spec != INPUT_SPEC:
|
||||
raise ValueError(f"EfficientViT segmentation recipe requires {INPUT_SPEC}, got {input_spec!r}")
|
||||
return tuple(int(n) for n in input_spec.split("x"))
|
||||
|
||||
|
||||
def preprocess(image, input_spec: str = INPUT_SPEC):
|
||||
"""Official cv2 cubic + float32 ImageNet normalization at the bench shape."""
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
_, _, h, w = input_shape(input_spec)
|
||||
rgb = np.asarray(image.convert("RGB"))
|
||||
if rgb.shape[:2] != (h, w):
|
||||
rgb = cv2.resize(rgb, (w, h), interpolation=cv2.INTER_CUBIC)
|
||||
x = rgb.astype(np.float32) / np.float32(255.0)
|
||||
x = (x - np.asarray((0.485, 0.456, 0.406), np.float32)) / np.asarray((0.229, 0.224, 0.225), np.float32)
|
||||
return np.ascontiguousarray(x.transpose(2, 0, 1)[None])
|
||||
|
||||
|
||||
def build_variant(variant: str, weights: Path, input_spec: str):
|
||||
input_shape(input_spec)
|
||||
info, weights = checkpoint_info(variant), Path(weights)
|
||||
if not weights.is_file() or weights.stat().st_size != info["size"] or _sha256(weights) != info["sha256"]:
|
||||
raise ValueError(f"Official segmentation checkpoint identity mismatch: {weights}; expected {info}")
|
||||
if UPSTREAM_COMMIT != SOURCE_COMMIT:
|
||||
raise RuntimeError("EfficientViT segmentation requires the pinned 2023 shared source")
|
||||
root = RUNTIME_ROOT / f"efficientvit-{SOURCE_COMMIT}" / "efficientvit"
|
||||
# Check the segmentation code before the shared factory's eager imports.
|
||||
for relative, expected in _SOURCE_SHA256.items():
|
||||
path = root / relative
|
||||
if not path.is_file() or _sha256(path) != expected:
|
||||
raise RuntimeError(f"Unverified official segmentation source: {path}")
|
||||
_official_factory() # Verify shared core; load private dependencies; restore sys.path.
|
||||
factory = importlib.import_module("efficientvit.seg_model_zoo").create_seg_model
|
||||
return factory(name=variant, dataset=info["dataset"], pretrained=True,
|
||||
weight_url=str(weights.resolve())).eval()
|
||||
13
harness/adapters/fast_scnn.py
Normal file
13
harness/adapters/fast_scnn.py
Normal file
@ -0,0 +1,13 @@
|
||||
"""Official Cityscapes Fast-SCNN with exact static pooling during export."""
|
||||
from functools import partial
|
||||
from .mmseg_common import (metadata, preprocess_image, build_model,
|
||||
lower_static_pools, RUNTIME_PYTHON, EXPORTER_VERSION)
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("fast.scnn")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
|
||||
|
||||
def build(weights, input_spec):
|
||||
network = build_model("fast.scnn", weights, input_spec)
|
||||
lower_static_pools(network.upstream.backbone)
|
||||
return network
|
||||
7
harness/adapters/fcn_resnet101.py
Normal file
7
harness/adapters/fcn_resnet101.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""Exact official ADE20K FCN ResNet101; admission requires actual validation."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("fcn.resnet101")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
build = partial(build_model, "fcn.resnet101")
|
||||
7
harness/adapters/fcn_resnet50.py
Normal file
7
harness/adapters/fcn_resnet50.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""Exact official ADE20K FCN ResNet50; admission requires actual validation."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("fcn.resnet50")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
build = partial(build_model, "fcn.resnet50")
|
||||
12
harness/adapters/fcos_r50_fpn.py
Normal file
12
harness/adapters/fcos_r50_fpn.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Exact MMDetection fcos.r50.fpn; see mmdet_dense_common for static deployment geometry."""
|
||||
from functools import partial
|
||||
from .mmdet_dense_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
|
||||
get_recipe, preprocess_variant, decode_variant, build_variant)
|
||||
|
||||
MODEL_ID = "fcos.r50.fpn"
|
||||
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
|
||||
PREPROCESS = recipe(MODEL_ID)
|
||||
EXPORTER_VERSION = "mmdet-fcos.r50.fpn-640-raw-v1"
|
||||
preprocess = partial(preprocess_variant, MODEL_ID)
|
||||
decode = partial(decode_variant, MODEL_ID)
|
||||
build = partial(build_variant, MODEL_ID)
|
||||
215
harness/adapters/feature_common.py
Normal file
215
harness/adapters/feature_common.py
Normal file
@ -0,0 +1,215 @@
|
||||
"""Pinned, unmodified feature sources. Static learned graphs; explicit host decoding."""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
ROOT = Path(os.environ.get('BENCH_FEATURE_SOURCES', '/home/ming01.wang/bench-cache/sources/feature'))
|
||||
MAX_KEYPOINTS = 1024
|
||||
EXPORTER_VERSION = 'feature-static-1'
|
||||
_CHECKPOINTS = {
|
||||
'superpoint': ('superglue/models/weights/superpoint_v1.pth', '52b6708629640ca883673b5d5c097c4ddad37d8048b33f09c8ca0d69db12c40e',
|
||||
'https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/ddcf11f42e7e0732a0c4607648f9448ea8d73590/models/weights/superpoint_v1.pth'),
|
||||
'disk': ('disk/depth-save.pth', '9c2ee4ded238892dfa51569941372601e35e4a74aa6f84ea80053d2ab1c07abe',
|
||||
'https://raw.githubusercontent.com/cvlab-epfl/disk/8dc6d4d65b47210f629e21ad1fbfb56944b11696/depth-save.pth'),
|
||||
'aliked': ('aliked/models/aliked-n16.pth', '5be8704840ed662d9d8c561bf7279c222092674e7eb05fd0feab94899e9d82f2',
|
||||
'https://raw.githubusercontent.com/Shiaoming/ALIKED/683d7c65197395c0b3f01ebe76e1084a27e73a65/models/aliked-n16.pth'),
|
||||
}
|
||||
|
||||
|
||||
def checkpoint_info(model):
|
||||
rel, digest, url = _CHECKPOINTS[model]
|
||||
return {'url': url, 'sha256': digest, 'filename': Path(rel).name, 'official_relative_path': rel}
|
||||
|
||||
|
||||
def input_spec(model):
|
||||
if model in ('xfeat', 'd2.net', 'r2d2', 'dedode.v2', 'keynetplushardnet'):
|
||||
from . import get
|
||||
return get(model).INPUT_SPEC
|
||||
return '1x1x480x640' if model == 'superpoint' else '1x3x480x640'
|
||||
|
||||
|
||||
def recipe(model):
|
||||
return {'input_spec': input_spec(model), 'color': 'GRAY' if model == 'superpoint' else 'RGB',
|
||||
'resize': 'opencv_linear_640x480', 'scale': 1 / 255, 'output_layout': 'NCHW',
|
||||
'protocol': 'feature-static-640x480-v1', 'gray': 'opencv_RGB2GRAY_uint8'}
|
||||
|
||||
|
||||
def preprocess(image, spec, model=None):
|
||||
if model in ('xfeat', 'd2.net', 'r2d2', 'dedode.v2', 'keynetplushardnet'):
|
||||
from . import get
|
||||
return get(model).preprocess(image, spec)
|
||||
import cv2
|
||||
import numpy as np
|
||||
if spec not in ('1x1x480x640', '1x3x480x640'): raise ValueError(spec)
|
||||
x = np.asarray(image.convert('RGB'))
|
||||
if spec.startswith('1x1x'): x = cv2.cvtColor(x, cv2.COLOR_RGB2GRAY)
|
||||
x = cv2.resize(x, (640, 480), interpolation=cv2.INTER_LINEAR).astype(np.float32) / 255
|
||||
return np.ascontiguousarray(x[None, None] if x.ndim == 2 else x.transpose(2, 0, 1)[None])
|
||||
|
||||
|
||||
@lru_cache(None)
|
||||
def official_module(model):
|
||||
lock = json.loads((ROOT / 'source-lock.json').read_text())
|
||||
prefix = {'superpoint': 'superglue/', 'disk': 'disk/', 'aliked': 'aliked/'}[model]
|
||||
for rel, expected in lock['files'].items():
|
||||
if rel.startswith(prefix) and hashlib.sha256((ROOT / rel).read_bytes()).hexdigest() != expected:
|
||||
raise ValueError(f'upstream source modified: {rel}')
|
||||
if model == 'superpoint':
|
||||
spec = importlib.util.spec_from_file_location('_bench_official_superpoint', ROOT / 'superglue/models/superpoint.py')
|
||||
module = importlib.util.module_from_spec(spec); spec.loader.exec_module(module)
|
||||
return module
|
||||
roots = [ROOT / model]
|
||||
if model == 'disk': roots += [ROOT / 'disk/submodules' / n for n in ('unets', 'torch-dimcheck', 'torch-localize')]
|
||||
# Optional pure-Python dependencies live here, never in the shared venv.
|
||||
roots.append(ROOT / 'deps')
|
||||
old = sys.path[:]
|
||||
try:
|
||||
sys.path[:0] = [str(p) for p in roots]
|
||||
name = 'disk' if model == 'disk' else 'nets.aliked'
|
||||
mod = importlib.import_module(name)
|
||||
if not Path(mod.__file__).resolve().is_relative_to((ROOT / model).resolve()):
|
||||
raise RuntimeError(f'upstream module namespace collision: {mod.__file__}')
|
||||
return mod
|
||||
finally:
|
||||
sys.path[:] = old
|
||||
|
||||
|
||||
def official_model(model, weights):
|
||||
import torch
|
||||
weights = Path(weights)
|
||||
if hashlib.sha256(weights.read_bytes()).hexdigest() != checkpoint_info(model)['sha256']:
|
||||
raise ValueError('official checkpoint SHA mismatch')
|
||||
module = official_module(model)
|
||||
state = torch.load(weights, map_location='cpu', weights_only=False)
|
||||
if model == 'superpoint':
|
||||
m = module.SuperPoint({'max_keypoints': MAX_KEYPOINTS})
|
||||
elif model == 'disk':
|
||||
m = module.DISK(); state = state['extractor']
|
||||
else:
|
||||
m = module.ALIKED(model_name='aliked-n16', device='cpu', top_k=MAX_KEYPOINTS, load_pretrained=False)
|
||||
m.load_state_dict(state, strict=True)
|
||||
return m.eval()
|
||||
|
||||
|
||||
def official_features(model, m, x):
|
||||
"""Reference forward without ALIKED's CUDA timing-only calls on CPU."""
|
||||
if model == 'superpoint':
|
||||
d = m({'image': x})
|
||||
return d['keypoints'][0], d['scores'][0], d['descriptors'][0].T
|
||||
if model == 'disk':
|
||||
d = m.features(x, kind='nms', n=MAX_KEYPOINTS, window_size=5, cutoff=0.)[0]
|
||||
return d.kp, d.kp_logp, d.desc
|
||||
feature, score = m.extract_dense_map(x)
|
||||
keypoints, scores, _ = m.dkd(score)
|
||||
descriptors, _ = m.desc_head(feature, keypoints)
|
||||
wh = x.new_tensor([x.shape[-1] - 1, x.shape[-2] - 1])
|
||||
return (keypoints[0] + 1) * wh / 2, scores[0], descriptors[0]
|
||||
|
||||
|
||||
def replace_deformable(m):
|
||||
"""Exact torchvision DCNv1 sampling expressed as standard ONNX GridSample+Conv.
|
||||
|
||||
Only ALIKED's ungrouped, unmodulated 3x3 stride-1 operator is supported.
|
||||
This changes the wrapper instance, never the downloaded source or reference.
|
||||
"""
|
||||
import torch
|
||||
from torch import nn
|
||||
from torch.nn import functional as F
|
||||
|
||||
class StaticDeform(nn.Module):
|
||||
def __init__(self, source):
|
||||
super().__init__()
|
||||
assert not source.mask and source.regular_conv.kernel_size == (3, 3)
|
||||
assert source.regular_conv.stride == (1, 1) and source.regular_conv.groups == 1
|
||||
self.offset_conv = source.offset_conv; self.regular_conv = source.regular_conv
|
||||
|
||||
def forward(self, x):
|
||||
# The adapter contract is static NCHW; freeze these dimensions so
|
||||
# ONNX knows the 1x1 convolution kernel shape after sample stacking.
|
||||
b, c, h, w = (int(v) for v in x.shape)
|
||||
offset = self.offset_conv(x).clamp(-max(h, w) / 4., max(h, w) / 4.)
|
||||
yy, xx = torch.meshgrid(torch.arange(h, device=x.device, dtype=x.dtype),
|
||||
torch.arange(w, device=x.device, dtype=x.dtype), indexing='ij')
|
||||
samples = []
|
||||
for k in range(9):
|
||||
y = yy + (k // 3 - 1) + offset[:, 2 * k]
|
||||
z = xx + (k % 3 - 1) + offset[:, 2 * k + 1]
|
||||
grid = torch.stack((2 * z / (w - 1) - 1, 2 * y / (h - 1) - 1), -1)
|
||||
samples.append(F.grid_sample(x, grid, align_corners=True, padding_mode='zeros'))
|
||||
sampled = torch.stack(samples, 2).reshape(b, c * 9, h, w)
|
||||
weights = self.regular_conv.weight.flatten(1).unsqueeze(-1).unsqueeze(-1)
|
||||
return F.conv2d(sampled, weights, self.regular_conv.bias)
|
||||
|
||||
for name, child in list(m.named_children()):
|
||||
if child.__class__.__name__ == 'DeformableConv2d': setattr(m, name, StaticDeform(child))
|
||||
else: replace_deformable(child)
|
||||
|
||||
|
||||
def build(model, weights, spec):
|
||||
import torch
|
||||
from torch import nn
|
||||
from torch.nn import functional as F
|
||||
if spec != input_spec(model): raise ValueError(f'{model} requires {input_spec(model)}, got {spec}')
|
||||
original = official_model(model, weights)
|
||||
|
||||
class Graph(nn.Module):
|
||||
def __init__(self, m):
|
||||
super().__init__(); self.m = m
|
||||
|
||||
def forward(self, x):
|
||||
m = self.m
|
||||
if model == 'disk': return m.unet(x)
|
||||
if model == 'aliked':
|
||||
keypoints, scores, descriptors = official_features(model, m, x)
|
||||
return (keypoints.reshape(1, MAX_KEYPOINTS, 2),
|
||||
scores.reshape(1, MAX_KEYPOINTS), descriptors.reshape(1, MAX_KEYPOINTS, 128))
|
||||
z = m.relu(m.conv1a(x)); z = m.relu(m.conv1b(z)); z = m.pool(z)
|
||||
z = m.relu(m.conv2a(z)); z = m.relu(m.conv2b(z)); z = m.pool(z)
|
||||
z = m.relu(m.conv3a(z)); z = m.relu(m.conv3b(z)); z = m.pool(z)
|
||||
z = m.relu(m.conv4a(z)); z = m.relu(m.conv4b(z))
|
||||
scores = F.softmax(m.convPb(m.relu(m.convPa(z))), 1)[:, :-1]
|
||||
b, _, h, w = scores.shape
|
||||
scores = scores.permute(0, 2, 3, 1).reshape(b, h, w, 8, 8)
|
||||
scores = scores.permute(0, 1, 3, 2, 4).reshape(b, h * 8, w * 8)
|
||||
desc = F.normalize(m.convDb(m.relu(m.convDa(z))), p=2, dim=1)
|
||||
return scores, desc
|
||||
|
||||
if model == 'aliked': replace_deformable(original)
|
||||
return Graph(original).eval()
|
||||
|
||||
|
||||
def decode(model, outputs):
|
||||
"""The same pinned official postprocessing for eager, ORT and board outputs."""
|
||||
if model in ('xfeat', 'd2.net', 'r2d2', 'dedode.v2', 'keynetplushardnet'):
|
||||
from . import get
|
||||
return get(model).decode(outputs)
|
||||
import numpy as np
|
||||
import torch
|
||||
module = official_module(model)
|
||||
tensors = [torch.as_tensor(np.asarray(o, dtype=np.float32)) for o in outputs]
|
||||
if any(not torch.isfinite(t).all() for t in tensors): raise ValueError('nonfinite feature outputs')
|
||||
if model == 'superpoint':
|
||||
scores, descriptors = tensors
|
||||
assert scores.shape == (1, 480, 640) and descriptors.shape == (1, 256, 60, 80)
|
||||
scores = module.simple_nms(scores, 4)[0]
|
||||
k = torch.nonzero(scores > .005); s = scores[tuple(k.T)]
|
||||
k, s = module.remove_borders(k, s, 4, 480, 640)
|
||||
k, s = module.top_k_keypoints(k, s, MAX_KEYPOINTS)
|
||||
k = torch.flip(k, [1]).float()
|
||||
d = module.sample_descriptors(k[None], descriptors, 8)[0].T
|
||||
elif model == 'disk':
|
||||
assert tensors[0].shape == (1, 129, 480, 640)
|
||||
from disk.model.detector import Detector
|
||||
kp = Detector().nms(tensors[0][:, 128:], n=MAX_KEYPOINTS, window_size=5, cutoff=0.)[0]
|
||||
f = kp.merge_with_descriptors(tensors[0][0, :128]); k, s, d = f.kp, f.kp_logp, f.desc
|
||||
else:
|
||||
k, s, d = [t[0] for t in tensors]
|
||||
assert k.shape == (MAX_KEYPOINTS, 2) and d.shape == (MAX_KEYPOINTS, 128)
|
||||
return {'keypoints': k.numpy(), 'scores': s.numpy(), 'descriptors': d.numpy()}
|
||||
79
harness/adapters/feature_sources.py
Normal file
79
harness/adapters/feature_sources.py
Normal file
@ -0,0 +1,79 @@
|
||||
"""Verified official sources for the second feature batch; no shared recipe changes."""
|
||||
import ast
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
ROOT = Path(os.environ.get('BENCH_FEATURE_BATCH2_SOURCES', '/home/ming01.wang/bench-cache/sources/feature-batch2'))
|
||||
|
||||
|
||||
@lru_cache(None)
|
||||
def verify_sources(family):
|
||||
manifest = json.loads((ROOT/'source-lock.json').read_text())
|
||||
for rel, digest in manifest['files'].items():
|
||||
if rel.startswith(family+'/') and hashlib.sha256((ROOT/rel).read_bytes()).hexdigest() != digest:
|
||||
raise ValueError(f'official source mismatch: {rel}')
|
||||
|
||||
|
||||
def verify_weights(path, expected):
|
||||
if hashlib.sha256(Path(path).read_bytes()).hexdigest() != expected:
|
||||
raise ValueError(f'checkpoint identity mismatch: {path}')
|
||||
|
||||
|
||||
@lru_cache(None)
|
||||
def module(relative):
|
||||
verify_sources(relative.split('/')[0])
|
||||
name = '_bench_feature2_' + relative.replace('/', '_').replace('.', '_')
|
||||
spec = importlib.util.spec_from_file_location(name, ROOT/relative)
|
||||
value = importlib.util.module_from_spec(spec)
|
||||
before = sys.path[:]
|
||||
try:
|
||||
sys.path.insert(0, str(ROOT/relative.split('/')[0]))
|
||||
spec.loader.exec_module(value)
|
||||
finally:
|
||||
sys.path[:] = before
|
||||
return value
|
||||
|
||||
|
||||
def definitions(relative, names, namespace=None):
|
||||
"""Import exact upstream definitions without unrelated training/UI imports.
|
||||
|
||||
Original function/class/assignment AST nodes are compiled unchanged. This
|
||||
does not reimplement their numerical logic or edit the upstream files.
|
||||
"""
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
verify_sources(relative.split('/')[0])
|
||||
body = []
|
||||
for node in ast.parse((ROOT/relative).read_text()).body:
|
||||
if isinstance(node, (ast.FunctionDef, ast.ClassDef)) and node.name in names:
|
||||
body.append(node)
|
||||
elif isinstance(node, ast.Assign) and any(isinstance(t, ast.Name) and t.id in names for t in node.targets):
|
||||
body.append(node)
|
||||
assert len(body) == len(names), (relative, names)
|
||||
scope = {'np': np, 'torch': torch, 'nn': nn, 'F': F, **(namespace or {})}
|
||||
exec(compile(ast.Module(body=body, type_ignores=[]), str(ROOT/relative), 'exec'), scope)
|
||||
return scope
|
||||
|
||||
|
||||
def tensors(outputs):
|
||||
import numpy as np
|
||||
import torch
|
||||
result = [torch.as_tensor(np.asarray(v, np.float32)) for v in outputs]
|
||||
if any(not torch.isfinite(v).all() for v in result): raise ValueError('nonfinite feature maps')
|
||||
return result
|
||||
|
||||
|
||||
def top_features(keypoints, scores, descriptors):
|
||||
import numpy as np
|
||||
from .feature_common import MAX_KEYPOINTS
|
||||
indices = np.asarray(scores).argsort()[-MAX_KEYPOINTS:]
|
||||
return {'keypoints': np.asarray(keypoints, np.float32)[indices, :2],
|
||||
'scores': np.asarray(scores, np.float32)[indices],
|
||||
'descriptors': np.asarray(descriptors, np.float32)[indices]}
|
||||
39
harness/adapters/feature_sources3.py
Normal file
39
harness/adapters/feature_sources3.py
Normal file
@ -0,0 +1,39 @@
|
||||
"""Private batch3 official source loader; never used from production during probes."""
|
||||
import hashlib
|
||||
import importlib
|
||||
import importlib.util
|
||||
import json
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
ROOT=Path('/home/ming01.wang/bench-cache/sources/feature-batch3')
|
||||
|
||||
@lru_cache(None)
|
||||
def verify(family):
|
||||
record=next(r for r in json.loads((ROOT/'source-lock.json').read_text()) if r['name']==family)
|
||||
for relative,digest in record['files'].items():
|
||||
if hashlib.sha256((ROOT/family/relative).read_bytes()).hexdigest()!=digest:
|
||||
raise ValueError(f'official source changed: {family}/{relative}')
|
||||
|
||||
def check(path,digest):
|
||||
if hashlib.sha256(Path(path).read_bytes()).hexdigest()!=digest:raise ValueError(f'weight mismatch: {path}')
|
||||
|
||||
@lru_cache(None)
|
||||
def package(family,name):
|
||||
verify(family)
|
||||
before=sys.path[:]
|
||||
try:
|
||||
sys.path[:0]=[str(ROOT/family),str(ROOT/'deps')]
|
||||
module=importlib.import_module(name)
|
||||
if not Path(module.__file__).resolve().is_relative_to((ROOT/family).resolve()):
|
||||
raise RuntimeError(f'official namespace collision: {module.__file__}')
|
||||
return module
|
||||
finally:sys.path[:]=before
|
||||
|
||||
@lru_cache(None)
|
||||
def module(family,relative):
|
||||
verify(family)
|
||||
spec=importlib.util.spec_from_file_location('_official3_'+family+'_'+relative.replace('/','_'),ROOT/family/relative)
|
||||
result=importlib.util.module_from_spec(spec);spec.loader.exec_module(result)
|
||||
return result
|
||||
12
harness/adapters/foveabox_r50_fpn.py
Normal file
12
harness/adapters/foveabox_r50_fpn.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Exact MMDetection foveabox.r50.fpn; see mmdet_dense_common for static deployment geometry."""
|
||||
from functools import partial
|
||||
from .mmdet_dense_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
|
||||
get_recipe, preprocess_variant, decode_variant, build_variant)
|
||||
|
||||
MODEL_ID = "foveabox.r50.fpn"
|
||||
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
|
||||
PREPROCESS = recipe(MODEL_ID)
|
||||
EXPORTER_VERSION = "mmdet-foveabox.r50.fpn-640-raw-v1"
|
||||
preprocess = partial(preprocess_variant, MODEL_ID)
|
||||
decode = partial(decode_variant, MODEL_ID)
|
||||
build = partial(build_variant, MODEL_ID)
|
||||
12
harness/adapters/fsaf_r50_fpn.py
Normal file
12
harness/adapters/fsaf_r50_fpn.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Exact MMDetection fsaf.r50.fpn; see mmdet_dense_common for static deployment geometry."""
|
||||
from functools import partial
|
||||
from .mmdet_dense_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
|
||||
get_recipe, preprocess_variant, decode_variant, build_variant)
|
||||
|
||||
MODEL_ID = "fsaf.r50.fpn"
|
||||
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
|
||||
PREPROCESS = recipe(MODEL_ID)
|
||||
EXPORTER_VERSION = "mmdet-fsaf.r50.fpn-640-raw-v1"
|
||||
preprocess = partial(preprocess_variant, MODEL_ID)
|
||||
decode = partial(decode_variant, MODEL_ID)
|
||||
build = partial(build_variant, MODEL_ID)
|
||||
12
harness/adapters/gfl_r50_fpn.py
Normal file
12
harness/adapters/gfl_r50_fpn.py
Normal file
@ -0,0 +1,12 @@
|
||||
"""Exact MMDetection gfl.r50.fpn; see mmdet_dense_common for static deployment geometry."""
|
||||
from functools import partial
|
||||
from .mmdet_dense_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
|
||||
get_recipe, preprocess_variant, decode_variant, build_variant)
|
||||
|
||||
MODEL_ID = "gfl.r50.fpn"
|
||||
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
|
||||
PREPROCESS = recipe(MODEL_ID)
|
||||
EXPORTER_VERSION = "mmdet-gfl.r50.fpn-640-raw-v1"
|
||||
preprocess = partial(preprocess_variant, MODEL_ID)
|
||||
decode = partial(decode_variant, MODEL_ID)
|
||||
build = partial(build_variant, MODEL_ID)
|
||||
7
harness/adapters/hrnet_w18.py
Normal file
7
harness/adapters/hrnet_w18.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""Official Cityscapes hrnet.w18; exact static benchmark recipe."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("hrnet.w18")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
build = partial(build_model, "hrnet.w18")
|
||||
9
harness/adapters/hrnet_w32.py
Normal file
9
harness/adapters/hrnet_w32.py
Normal file
@ -0,0 +1,9 @@
|
||||
"""Official hrnet.w32 COCO17 at 256x192."""
|
||||
from functools import partial
|
||||
from .topdown_pose_common import (metadata, build_model, predict as _predict, decode as _decode,
|
||||
preprocess, g2_input, PREPROCESS, RUNTIME_PYTHON, EXPORTER_VERSION)
|
||||
|
||||
CHECKPOINT, MODEL_SPEC = metadata('hrnet.w32')
|
||||
build = partial(build_model, 'hrnet.w32')
|
||||
predict = partial(_predict, 'hrnet.w32')
|
||||
decode = partial(_decode, 'hrnet.w32')
|
||||
7
harness/adapters/hrnet_w48.py
Normal file
7
harness/adapters/hrnet_w48.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""Official Cityscapes hrnet.w48; exact static benchmark recipe."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("hrnet.w48")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
build = partial(build_model, "hrnet.w48")
|
||||
9
harness/adapters/hrnet_w48_2.py
Normal file
9
harness/adapters/hrnet_w48_2.py
Normal file
@ -0,0 +1,9 @@
|
||||
"""Official hrnet.w48.2 COCO17 at 256x192."""
|
||||
from functools import partial
|
||||
from .topdown_pose_common import (metadata, build_model, predict as _predict, decode as _decode,
|
||||
preprocess, g2_input, PREPROCESS, RUNTIME_PYTHON, EXPORTER_VERSION)
|
||||
|
||||
CHECKPOINT, MODEL_SPEC = metadata('hrnet.w48.2')
|
||||
build = partial(build_model, 'hrnet.w48.2')
|
||||
predict = partial(_predict, 'hrnet.w48.2')
|
||||
decode = partial(_decode, 'hrnet.w48.2')
|
||||
129
harness/adapters/keynetplushardnet.py
Normal file
129
harness/adapters/keynetplushardnet.py
Normal file
@ -0,0 +1,129 @@
|
||||
"""Author PyTorch Key.Net plus original HardNet++; no HyNet substitution."""
|
||||
from . import feature_sources3 as sources
|
||||
from .feature_common import MAX_KEYPOINTS, preprocess as gray_preprocess
|
||||
|
||||
INPUT_SPEC='1x1x480x640'
|
||||
MODEL_SPEC={'input_spec':INPUT_SPEC,'dataset':'hpatches'}
|
||||
EXPORTER_VERSION='keynet-hardnetpp-static-1'
|
||||
OPSET=17
|
||||
CHECKPOINT={'url':'https://raw.githubusercontent.com/axelBarroso/Key.Net-Pytorch/74fe0a75998445bb47b74b642b764543b25dc90a/model/weights/keynet_pytorch.pth',
|
||||
'sha256':'59d3a46d57b439513acf531877fdfaa95eb550ccd10ec462e23d7c1f7a86d52a',
|
||||
'filename':'keynet_pytorch.pth','official_relative_path':str(sources.ROOT/'keynet-pytorch/model/weights/keynet_pytorch.pth')}
|
||||
DESCRIPTOR_CHECKPOINT={'url':'https://raw.githubusercontent.com/axelBarroso/Key.Net/24808fb5bd7376e33f43e9f8f1029f4ef1a7ce3c/keyNet/pretrained_nets/HardNet%2B%2B.pth',
|
||||
'path':str(sources.ROOT/'keynet/keyNet/pretrained_nets/HardNet++.pth'),
|
||||
'sha256':'9fe80a335fe72d1225fdd1ae0943418b12489f95014bd51dddc9819d6ffa33c7'}
|
||||
PREPROCESS={'input_spec':INPUT_SPEC,'color':'GRAY','gray':'opencv_RGB2GRAY_uint8','scale':1/255,
|
||||
'resize':'opencv_linear_640x480','output_layout':'NCHW',
|
||||
'protocol':'keynet-author-pytorch-hardnetpp-single-scale-s22-v1'}
|
||||
OFFICIAL_REFERENCE_POLICY = {
|
||||
'selection': 'official positive NMS15, top1024; equal scores use ascending raster index',
|
||||
'tie_rule': 'ONNX TopK standard; official torch.sort is unstable for equal scores',
|
||||
'patches': 'official extract_patch, s_mult22, PS32, grayscale255, align_cornersFalse',
|
||||
'descriptor': 'official HardNet++ class and weights, no HyNet substitution',
|
||||
}
|
||||
|
||||
def preprocess(image,input_spec=INPUT_SPEC):
|
||||
if input_spec!=INPUT_SPEC:raise ValueError(input_spec)
|
||||
return gray_preprocess(image,input_spec)
|
||||
|
||||
def official_model(weights):
|
||||
import torch
|
||||
sources.check(weights,CHECKPOINT['sha256'])
|
||||
sources.check(DESCRIPTOR_CHECKPOINT['path'],DESCRIPTOR_CHECKPOINT['sha256'])
|
||||
network=sources.package('keynet-pytorch','model.network')
|
||||
detector=network.KeyNet({'num_filters':8,'num_levels':3,'kernel_size':5})
|
||||
detector.load_state_dict(torch.load(weights,map_location='cpu',weights_only=False)['state_dict'],strict=True)
|
||||
descriptor=sources.module('keynet','keyNet/model/hardnet_pytorch.py').HardNet()
|
||||
descriptor.load_state_dict(torch.load(DESCRIPTOR_CHECKPOINT['path'],map_location='cpu',weights_only=False)['state_dict'],strict=True)
|
||||
return torch.nn.ModuleDict({'detector':detector,'descriptor':descriptor}).eval()
|
||||
|
||||
def official_features(m,x):
|
||||
import torch
|
||||
module=sources.package('keynet-pytorch','model.extraction_tools')
|
||||
score=module.remove_borders(m['detector'](x),15)
|
||||
yx=module.NonMaxSuppression(nms_size=15)(score)
|
||||
values=score[0,0,yx[0],yx[1]]
|
||||
# Preserve official NMS/candidate ordering; specify only its undefined tie order.
|
||||
order=torch.argsort(values,descending=True,stable=True)[:MAX_KEYPOINTS]
|
||||
k=torch.stack([yx[1,order],yx[0,order]],-1).float();s=values[order]
|
||||
n=len(k)
|
||||
laf=module.to_laf(k[None],22*torch.ones(1,n,1,1),torch.zeros(1,n,1))
|
||||
patches=module.extract_patch(255*x,laf,PS=32,normalize_lafs_before_extraction=True)[0]
|
||||
d=torch.cat([m['descriptor'](p) for p in patches.split(1000)],0)
|
||||
return k,s,d
|
||||
|
||||
def build(weights,input_spec):
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
if input_spec!=INPUT_SPEC:raise ValueError(input_spec)
|
||||
utils=sources.package('keynet-pytorch','model.kornia_tools.utils')
|
||||
class StaticHandcrafted(torch.nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
# Original Kornia 0.1.4 Sobel kernel, fixed C=1 eliminates dynamic repeat.
|
||||
self.register_buffer('kernel',utils.kornia.filters.SpatialGradient().kernel[None,None].flip(-3))
|
||||
|
||||
def gradient(self,x):
|
||||
x=F.pad(x,[1,1,1,1],'replicate')[:,:,None]
|
||||
return F.conv3d(F.pad(x,[0,0,0,0,1,1]),self.kernel)
|
||||
|
||||
def forward(self,x):
|
||||
sobel=self.gradient(x);dx,dy=sobel[:,:,0],sobel[:,:,1]
|
||||
sobel_dx=self.gradient(dx);dxx,dxy=sobel_dx[:,:,0],sobel_dx[:,:,1]
|
||||
dyy=self.gradient(dy)[:,:,1]
|
||||
return torch.cat([dx,dy,dx**2.,dy**2.,dx*dy,dxy,dxy**2.,dxx,dyy,dxx*dyy],1)
|
||||
|
||||
class StableTopK(torch.autograd.Function):
|
||||
@staticmethod
|
||||
def forward(ctx,values):
|
||||
indices=torch.argsort(values,dim=1,descending=True,stable=True)[:,:MAX_KEYPOINTS]
|
||||
return values.gather(1,indices),indices
|
||||
|
||||
@staticmethod
|
||||
def symbolic(g,values):
|
||||
# ONNX TopK explicitly breaks ties by the lower original index.
|
||||
k=g.op('Constant',value_t=torch.tensor([MAX_KEYPOINTS],dtype=torch.long))
|
||||
return g.op('TopK',values,k,axis_i=1,largest_i=1,sorted_i=1,outputs=2)
|
||||
|
||||
class Graph(torch.nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__();self.nets=official_model(weights)
|
||||
self.nets['detector'].feature_extractor.hc_block=StaticHandcrafted()
|
||||
self.register_buffer('pyramid_kernel',utils._get_pyramid_gaussian_kernel()[None])
|
||||
border=torch.zeros(1,1,480,640);border[:,:,15:-15,15:-15]=1
|
||||
self.register_buffer('border',border)
|
||||
unit=F.affine_grid(torch.tensor([[[1.,0.,0.],[0.,1.,0.]]]),[1,1,32,32],align_corners=False)
|
||||
self.register_buffer('patch_grid',torch.cat([unit,torch.ones(1,32,32,1)],-1).reshape(1,1024,3))
|
||||
|
||||
def forward(self,x):
|
||||
# Same three internal pyramid levels as official KeyNet; fixed grayscale shapes.
|
||||
detector=self.nets['detector'];level=x;features=[detector.feature_extractor(level)]
|
||||
for h,w in ((400,533),(333,444)):
|
||||
level=F.interpolate(F.conv2d(F.pad(level,[2,2,2,2],'reflect'),self.pyramid_kernel),
|
||||
size=(h,w),mode='bilinear',align_corners=False)
|
||||
features.append(F.interpolate(detector.feature_extractor(level),size=(480,640),mode='bilinear',align_corners=False))
|
||||
scores=detector.last_conv(torch.cat(features,1))*self.border
|
||||
scores=scores*(scores==F.max_pool2d(scores,15,1,7)).to(scores.dtype)
|
||||
s,ids=StableTopK.apply(scores.flatten(1))
|
||||
k=torch.stack((ids%640,torch.div(ids,640,rounding_mode='floor')),-1).float()
|
||||
laf=utils.laf_from_center_scale_ori(k,22*torch.ones(1,MAX_KEYPOINTS,1,1,device=x.device),torch.zeros(1,MAX_KEYPOINTS,1,device=x.device))
|
||||
# At s=22, PS=32, every patch has official pyramid index 0.
|
||||
# Precompute only the constant affine base grid; keep official LAF
|
||||
# normalization and its floating-point operation order.
|
||||
theta=utils.denormalize_laf(utils.normalize_laf(laf,x),x).reshape(MAX_KEYPOINTS,2,3)
|
||||
grid=torch.bmm(self.patch_grid.expand(MAX_KEYPOINTS,-1,-1),theta.transpose(1,2))
|
||||
grid=torch.stack((2*grid[:,:,0]/640-1,2*grid[:,:,1]/480-1),-1)
|
||||
# One tiled grid avoids materializing 1024 copies of the input.
|
||||
patches=F.grid_sample(255*x,grid.reshape(1,MAX_KEYPOINTS*32,32,2),
|
||||
padding_mode='border',align_corners=False).reshape(MAX_KEYPOINTS,1,32,32)
|
||||
d=self.nets['descriptor'](patches)
|
||||
return k.reshape(1,MAX_KEYPOINTS,2),s.reshape(1,MAX_KEYPOINTS),d[None]
|
||||
return Graph().eval()
|
||||
|
||||
def decode(outputs):
|
||||
import numpy as np
|
||||
k,s,d=[np.asarray(o,np.float32)[0] for o in outputs]
|
||||
assert k.shape==(MAX_KEYPOINTS,2) and s.shape==(MAX_KEYPOINTS,) and d.shape==(MAX_KEYPOINTS,128)
|
||||
assert all(np.isfinite(v).all() for v in (k,s,d))
|
||||
valid=s>0
|
||||
return {'keypoints':k[valid],'scores':s[valid],'descriptors':d[valid]}
|
||||
22
harness/adapters/mask2former_swin_b_instance.py
Normal file
22
harness/adapters/mask2former_swin_b_instance.py
Normal file
@ -0,0 +1,22 @@
|
||||
"""Exact MMDetection mask2former.swin.b.instance; see mmdet_iseg_common.
|
||||
|
||||
Backed by the official swin-b COCO-panoptic config and checkpoint (the model
|
||||
zoo publishes no instance-only swin-b variant); instance outputs are the 80
|
||||
thing classes, which mmdet orders exactly like CocoDataset.
|
||||
"""
|
||||
from functools import partial
|
||||
|
||||
from .mmdet_iseg_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
|
||||
get_recipe, preprocess_variant, decode_variant,
|
||||
build_variant, predict_variant, g2_input_variant)
|
||||
|
||||
MODEL_ID = "mask2former.swin.b.instance"
|
||||
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
|
||||
PREPROCESS = recipe(MODEL_ID)
|
||||
EXPORTER_VERSION = "mmdet-mask2former-640-raw-v1"
|
||||
OPSET = 16 # ONNX GridSample for the multi-scale deformable attention layers.
|
||||
preprocess = partial(preprocess_variant, MODEL_ID)
|
||||
decode = partial(decode_variant, MODEL_ID)
|
||||
build = partial(build_variant, MODEL_ID)
|
||||
predict = partial(predict_variant, MODEL_ID)
|
||||
g2_input = partial(g2_input_variant, MODEL_ID)
|
||||
17
harness/adapters/mask2former_swin_s_instance.py
Normal file
17
harness/adapters/mask2former_swin_s_instance.py
Normal file
@ -0,0 +1,17 @@
|
||||
"""Exact MMDetection mask2former.swin.s.instance; see mmdet_iseg_common."""
|
||||
from functools import partial
|
||||
|
||||
from .mmdet_iseg_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
|
||||
get_recipe, preprocess_variant, decode_variant,
|
||||
build_variant, predict_variant, g2_input_variant)
|
||||
|
||||
MODEL_ID = "mask2former.swin.s.instance"
|
||||
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
|
||||
PREPROCESS = recipe(MODEL_ID)
|
||||
EXPORTER_VERSION = "mmdet-mask2former-640-raw-v1"
|
||||
OPSET = 16 # ONNX GridSample for the multi-scale deformable attention layers.
|
||||
preprocess = partial(preprocess_variant, MODEL_ID)
|
||||
decode = partial(decode_variant, MODEL_ID)
|
||||
build = partial(build_variant, MODEL_ID)
|
||||
predict = partial(predict_variant, MODEL_ID)
|
||||
g2_input = partial(g2_input_variant, MODEL_ID)
|
||||
17
harness/adapters/mask2former_swin_t_instance.py
Normal file
17
harness/adapters/mask2former_swin_t_instance.py
Normal file
@ -0,0 +1,17 @@
|
||||
"""Exact MMDetection mask2former.swin.t.instance; see mmdet_iseg_common."""
|
||||
from functools import partial
|
||||
|
||||
from .mmdet_iseg_common import (RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, recipe,
|
||||
get_recipe, preprocess_variant, decode_variant,
|
||||
build_variant, predict_variant, g2_input_variant)
|
||||
|
||||
MODEL_ID = "mask2former.swin.t.instance"
|
||||
CHECKPOINT = get_recipe(MODEL_ID)["checkpoint"]
|
||||
PREPROCESS = recipe(MODEL_ID)
|
||||
EXPORTER_VERSION = "mmdet-mask2former-640-raw-v1"
|
||||
OPSET = 16 # ONNX GridSample for the multi-scale deformable attention layers.
|
||||
preprocess = partial(preprocess_variant, MODEL_ID)
|
||||
decode = partial(decode_variant, MODEL_ID)
|
||||
build = partial(build_variant, MODEL_ID)
|
||||
predict = partial(predict_variant, MODEL_ID)
|
||||
g2_input = partial(g2_input_variant, MODEL_ID)
|
||||
242
harness/adapters/midas_common.py
Normal file
242
harness/adapters/midas_common.py
Normal file
@ -0,0 +1,242 @@
|
||||
"""Pinned official MiDaS relative-depth models (v2.1 small, v3 DPT hybrid/large).
|
||||
|
||||
The pinned private runtime tree keeps the official MiDaS source, the timm
|
||||
0.6.12 it requires and the gen-efficientnet backbone code, with only the
|
||||
local-hub patch from the recorded receipt. No package install, replacement
|
||||
architecture or synthetic weights. Outputs are nonnegative relative inverse
|
||||
depth (disparity); restore follows the author's bicubic resize.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
UPSTREAM_COMMIT = "454597711a62eabcbf7d1e89f3fb9f569051ac9b"
|
||||
# sha256 over "<rel>\0<bytes>" of every *.py in the pinned runtime tree
|
||||
# (midas + gen-efficientnet + timm), matching source_receipt.json.
|
||||
SOURCE_TREE_SHA256 = "b27d6846712df86b72264c1df091d0693b4834155cfcc1d30777a359904d8b82"
|
||||
RUNTIME_ROOT = Path.home() / "bench-cache/sources/midas" / UPSTREAM_COMMIT
|
||||
OPSET = 17
|
||||
EXPORTER_VERSION = "midas-relative-v1-opset17-static"
|
||||
RESTORE_INTERPOLATION = "bicubic"
|
||||
RESTORE_ALIGN_CORNERS = False
|
||||
|
||||
INPUT_SPECS = {
|
||||
"small": "1x3x256x256",
|
||||
"hybrid": "1x3x384x384",
|
||||
"large": "1x3x384x384",
|
||||
}
|
||||
MODEL_SPECS = {
|
||||
"small": {"source": "custom", "dataset": "nyuv2", "input_spec": INPUT_SPECS["small"], "resize": 0},
|
||||
"hybrid": {"source": "custom", "dataset": "nyuv2", "input_spec": INPUT_SPECS["hybrid"], "resize": 0},
|
||||
"large": {"source": "custom", "dataset": "nyuv2", "input_spec": INPUT_SPECS["large"], "resize": 0},
|
||||
}
|
||||
PREPROCESSES = {
|
||||
"small": {
|
||||
"id": "midas-v21-small-rgb-static256-v1",
|
||||
"layout": "NCHW", "dtype": "float32", "color": "RGB",
|
||||
"resize": "OpenCV INTER_CUBIC to 256x256; keep_aspect_ratio=False for static deployment",
|
||||
"normalize": "RGB/255; mean=[0.485,0.456,0.406]; std=[0.229,0.224,0.225]",
|
||||
"output": "N,H,W nonnegative relative inverse depth (disparity); NOT metric depth",
|
||||
"restore": "bicubic to original H,W, align_corners=False; no min/max image normalization",
|
||||
"source_commit": UPSTREAM_COMMIT,
|
||||
},
|
||||
"hybrid": {
|
||||
"id": "midas-v3-hybrid-rgb-static384-v1",
|
||||
"layout": "NCHW", "dtype": "float32", "color": "RGB",
|
||||
"resize": "OpenCV INTER_CUBIC to 384x384; keep_aspect_ratio=False for static deployment",
|
||||
"normalize": "RGB/255; mean=[0.5,0.5,0.5]; std=[0.5,0.5,0.5]",
|
||||
"output": "N,H,W nonnegative relative inverse depth (disparity); NOT metric depth",
|
||||
"restore": "bicubic to original H,W, align_corners=False; no min/max image normalization",
|
||||
"source_commit": UPSTREAM_COMMIT,
|
||||
},
|
||||
"large": {
|
||||
"id": "midas-v3-large-rgb-static384-v1",
|
||||
"layout": "NCHW", "dtype": "float32", "color": "RGB",
|
||||
"resize": "OpenCV INTER_CUBIC to 384x384; keep_aspect_ratio=False for static deployment",
|
||||
"normalize": "RGB/255; mean=[0.5,0.5,0.5]; std=[0.5,0.5,0.5]",
|
||||
"output": "N,H,W nonnegative relative inverse depth (disparity); NOT metric depth",
|
||||
"restore": "bicubic to original H,W, align_corners=False; no min/max image normalization",
|
||||
"source_commit": UPSTREAM_COMMIT,
|
||||
},
|
||||
}
|
||||
|
||||
_CHECKPOINTS = {
|
||||
"small": {"url": "https://github.com/isl-org/MiDaS/releases/download/v2_1/midas_v21_small_256.pt",
|
||||
"sha256": "70d6b9c891758c67f974a6097fb0c608c7ee67fb81ac3e5588847d5596d56fca",
|
||||
"release": "v2_1", "output_type": "relative_inverse_depth"},
|
||||
"hybrid": {"url": "https://github.com/isl-org/MiDaS/releases/download/v3/dpt_hybrid_384.pt",
|
||||
"sha256": "501f0c75b3bca7daec6b3682c5054c09b366765aef6fa3a09d03a5cb4b230853",
|
||||
"release": "v3", "output_type": "relative_inverse_depth"},
|
||||
"large": {"url": "https://github.com/isl-org/MiDaS/releases/download/v3/dpt_large_384.pt",
|
||||
"sha256": "2f21e586477d90cb9624c7eef5df7891edca49a1c4795ee2cb631fd4daa6ca69",
|
||||
"release": "v3", "output_type": "relative_inverse_depth"},
|
||||
}
|
||||
|
||||
|
||||
def checkpoint_info(variant: str) -> dict:
|
||||
return dict(_CHECKPOINTS[variant])
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with Path(path).open("rb") as stream:
|
||||
for block in iter(lambda: stream.read(1 << 20), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def source_root() -> Path:
|
||||
root = Path(os.environ.get("BENCH_MIDAS_SOURCE", str(RUNTIME_ROOT))).resolve()
|
||||
digest = hashlib.sha256()
|
||||
files = sorted(root.rglob("*.py"))
|
||||
for path in files:
|
||||
digest.update(str(path.relative_to(root)).encode() + b"\0" + path.read_bytes())
|
||||
if digest.hexdigest() != SOURCE_TREE_SHA256:
|
||||
raise RuntimeError(f"missing or changed pinned MiDaS runtime source {UPSTREAM_COMMIT}: {root}")
|
||||
return root
|
||||
|
||||
|
||||
def preprocess_for(image, input_spec: str, variant: str):
|
||||
"""Static deployment resize; per-variant normalization from midas.model_loader.
|
||||
|
||||
The v3 DPT checkpoints train with mean=std=0.5, the v2.1 small checkpoint
|
||||
with ImageNet statistics; both follow the official loader branches.
|
||||
"""
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
size = int(INPUT_SPECS[variant].split("x")[-1])
|
||||
if input_spec != INPUT_SPECS[variant]:
|
||||
raise ValueError(f"MiDaS {variant} recipe is fixed at {INPUT_SPECS[variant]}, got {input_spec}")
|
||||
mean, std = (([.5, .5, .5], [.5, .5, .5]) if variant != "small"
|
||||
else ([.485, .456, .406], [.229, .224, .225]))
|
||||
rgb = np.asarray(image.convert("RGB"), dtype=np.float64) / 255.0
|
||||
rgb = cv2.resize(rgb, (size, size), interpolation=cv2.INTER_CUBIC)
|
||||
rgb = (rgb - np.array(mean)) / np.array(std)
|
||||
return np.ascontiguousarray(rgb.transpose(2, 0, 1)[None], dtype=np.float32)
|
||||
|
||||
|
||||
def _static_forward_flex(self, x):
|
||||
"""Official midas backbones/vit.py forward_flex with python-int grid sizes.
|
||||
|
||||
x.shape entries become Tensors under torch.onnx tracing (torch >= 2.x) and
|
||||
then break _resize_pos_embed's F.interpolate and nn.Unflatten validation.
|
||||
The pinned deployment resolution is static, so the ints are constants.
|
||||
"""
|
||||
import torch
|
||||
|
||||
h, w = int(x.shape[2]), int(x.shape[3])
|
||||
pos_embed = self._resize_pos_embed(self.pos_embed, h // self.patch_size[1], w // self.patch_size[0])
|
||||
b = x.shape[0]
|
||||
if hasattr(self.patch_embed, "backbone"):
|
||||
x = self.patch_embed.backbone(x)
|
||||
if isinstance(x, (list, tuple)):
|
||||
x = x[-1]
|
||||
x = self.patch_embed.proj(x).flatten(2).transpose(1, 2)
|
||||
if getattr(self, "dist_token", None) is not None:
|
||||
cls_tokens = self.cls_token.expand(b, -1, -1)
|
||||
dist_token = self.dist_token.expand(b, -1, -1)
|
||||
x = torch.cat((cls_tokens, dist_token, x), dim=1)
|
||||
else:
|
||||
if self.no_embed_class:
|
||||
x = x + pos_embed
|
||||
cls_tokens = self.cls_token.expand(b, -1, -1)
|
||||
x = torch.cat((cls_tokens, x), dim=1)
|
||||
if not self.no_embed_class:
|
||||
x = x + pos_embed
|
||||
x = self.pos_drop(x)
|
||||
for blk in self.blocks:
|
||||
x = blk(x)
|
||||
x = self.norm(x)
|
||||
return x
|
||||
|
||||
|
||||
def _static_forward_vit(pretrained, x):
|
||||
"""Official midas backbones/utils.py forward_adapted_unflatten, int sizes."""
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
|
||||
h, w = int(x.shape[2]), int(x.shape[3])
|
||||
glob = pretrained.model.forward_flex(x)
|
||||
layer_1 = pretrained.activations["1"]
|
||||
layer_2 = pretrained.activations["2"]
|
||||
layer_3 = pretrained.activations["3"]
|
||||
layer_4 = pretrained.activations["4"]
|
||||
layer_1 = pretrained.act_postprocess1[0:2](layer_1)
|
||||
layer_2 = pretrained.act_postprocess2[0:2](layer_2)
|
||||
layer_3 = pretrained.act_postprocess3[0:2](layer_3)
|
||||
layer_4 = pretrained.act_postprocess4[0:2](layer_4)
|
||||
unflatten = nn.Sequential(
|
||||
nn.Unflatten(2, torch.Size([
|
||||
h // pretrained.model.patch_size[1],
|
||||
w // pretrained.model.patch_size[0],
|
||||
])))
|
||||
if layer_1.ndim == 3:
|
||||
layer_1 = unflatten(layer_1)
|
||||
if layer_2.ndim == 3:
|
||||
layer_2 = unflatten(layer_2)
|
||||
if layer_3.ndim == 3:
|
||||
layer_3 = unflatten(layer_3)
|
||||
if layer_4.ndim == 3:
|
||||
layer_4 = unflatten(layer_4)
|
||||
layer_1 = pretrained.act_postprocess1[3: len(pretrained.act_postprocess1)](layer_1)
|
||||
layer_2 = pretrained.act_postprocess2[3: len(pretrained.act_postprocess2)](layer_2)
|
||||
layer_3 = pretrained.act_postprocess3[3: len(pretrained.act_postprocess3)](layer_3)
|
||||
layer_4 = pretrained.act_postprocess4[3: len(pretrained.act_postprocess4)](layer_4)
|
||||
return layer_1, layer_2, layer_3, layer_4
|
||||
|
||||
|
||||
def _install_static_vit_path(model, input_spec: str):
|
||||
"""Swap the ViT flex path for the int-size twins and prove bitwise equality.
|
||||
|
||||
The official modules, weights and math are untouched: the twins only turn
|
||||
the traced Tensor shape entries into python ints for the fixed deployment
|
||||
resolution. A fixed deterministic input verifies the swapped forward stays
|
||||
bitwise identical to the untouched official forward before returning.
|
||||
"""
|
||||
import types
|
||||
|
||||
import torch
|
||||
|
||||
size = int(input_spec.split("x")[-1])
|
||||
generator = torch.Generator().manual_seed(20260918)
|
||||
probe = torch.rand((1, 3, size, size), generator=generator)
|
||||
with torch.inference_mode():
|
||||
official = model(probe).clone()
|
||||
model.pretrained.model.forward_flex = types.MethodType(_static_forward_flex, model.pretrained.model)
|
||||
model.forward_transformer = _static_forward_vit
|
||||
with torch.inference_mode():
|
||||
replaced = model(probe)
|
||||
if not torch.equal(official, replaced):
|
||||
raise RuntimeError("static-grid ViT path is not bitwise identical to the official forward")
|
||||
return model
|
||||
|
||||
|
||||
def build_variant(variant: str, weights: Path, input_spec: str):
|
||||
import torch
|
||||
|
||||
if input_spec != INPUT_SPECS[variant]:
|
||||
raise ValueError(f"expected fixed {INPUT_SPECS[variant]}, got {input_spec}")
|
||||
root = source_root()
|
||||
if file_sha256(weights) != _CHECKPOINTS[variant]["sha256"]:
|
||||
raise ValueError(f"wrong official MiDaS {variant} checkpoint")
|
||||
if str(root) not in sys.path:
|
||||
sys.path.insert(0, str(root))
|
||||
# Constructing with the checkpoint path keeps use_pretrained=False: the
|
||||
# official ImageNet backbone download never runs and the strict load is
|
||||
# performed by MiDaS BaseModel.load itself.
|
||||
if variant == "small":
|
||||
from midas.midas_net_custom import MidasNet_small
|
||||
|
||||
model = MidasNet_small(Path(weights), features=64, backbone="efficientnet_lite3",
|
||||
exportable=True, non_negative=True, blocks={"expand": True})
|
||||
else:
|
||||
from midas.dpt_depth import DPTDepthModel
|
||||
|
||||
backbone = "vitl16_384" if variant == "large" else "vitb_rn50_384"
|
||||
model = DPTDepthModel(path=str(weights), backbone=backbone, non_negative=True)
|
||||
_install_static_vit_path(model, input_spec)
|
||||
return model.eval()
|
||||
19
harness/adapters/midas_small.py
Normal file
19
harness/adapters/midas_small.py
Normal file
@ -0,0 +1,19 @@
|
||||
"""Official MiDaS v2.1 small: relative inverse depth at static 256x256."""
|
||||
from pathlib import Path
|
||||
|
||||
from .midas_common import (EXPORTER_VERSION, INPUT_SPECS, MODEL_SPECS, OPSET, PREPROCESSES,
|
||||
RESTORE_ALIGN_CORNERS, RESTORE_INTERPOLATION, build_variant,
|
||||
checkpoint_info, preprocess_for)
|
||||
|
||||
CHECKPOINT = checkpoint_info("small")
|
||||
INPUT_SPEC = INPUT_SPECS["small"]
|
||||
MODEL_SPEC = MODEL_SPECS["small"]
|
||||
PREPROCESS = PREPROCESSES["small"]
|
||||
|
||||
|
||||
def build(weights: Path, input_spec: str):
|
||||
return build_variant("small", weights, input_spec)
|
||||
|
||||
|
||||
def preprocess(image, input_spec: str = INPUT_SPEC):
|
||||
return preprocess_for(image, input_spec, "small")
|
||||
222
harness/adapters/mmdet_dense_common.py
Normal file
222
harness/adapters/mmdet_dense_common.py
Normal file
@ -0,0 +1,222 @@
|
||||
"""Pinned MMDetection dense detectors at the CSV's static 640 deployment size.
|
||||
|
||||
Only deployment geometry differs from the upstream 1333x800 test pipeline:
|
||||
keep-ratio resize to 640, then normalized zero padding at bottom/right to 640.
|
||||
Weights, normalization, raw heads and official postprocessing are unchanged.
|
||||
Heavy imports remain inside runtime functions for lightweight recipe discovery.
|
||||
"""
|
||||
from functools import lru_cache
|
||||
from types import SimpleNamespace
|
||||
|
||||
from .openmmlab import build as build_official, get_recipe
|
||||
|
||||
RUNTIME_PYTHON = "/home/ming01.wang/outputs/full-coverage-20260917/openmmlab/venv/bin/python"
|
||||
INPUT_SPEC = "1x3x640x640"
|
||||
MODEL_SPEC = {"source": "custom", "dataset": "coco2017", "input_spec": INPUT_SPEC}
|
||||
MODELS = ("retinanet.r50.fpn", "retinanet.r101.fpn", "fcos.r50.fpn",
|
||||
"atss.r50.fpn", "gfl.r50.fpn", "vfnet.r50.fpn",
|
||||
"foveabox.r50.fpn", "fsaf.r50.fpn", "reppoints.r50.fpn")
|
||||
HEADS = {"retinanet": ("RetinaHead", (720, 36)), "fcos": ("FCOSHead", (80, 4, 1)),
|
||||
"atss": ("ATSSHead", (80, 4, 1)), "gfl": ("GFLHead", (80, 68)),
|
||||
"vfnet": ("VFNetHead", (80, 4)), "foveabox": ("FoveaHead", (80, 4)),
|
||||
"fsaf": ("FSAFHead", (80, 4)), "reppoints": ("RepPointsHead", (80, 4))}
|
||||
STRIDES = (8, 16, 32, 64, 128)
|
||||
TEST_CFG = {"nms_pre": 1000, "min_bbox_size": 0, "score_thr": .05,
|
||||
"nms": {"type": "nms", "iou_threshold": .5}, "max_per_img": 100}
|
||||
|
||||
|
||||
def test_config(family):
|
||||
from copy import deepcopy
|
||||
cfg = deepcopy(TEST_CFG)
|
||||
if family in ("atss", "gfl", "vfnet"):
|
||||
cfg["nms"]["iou_threshold"] = .6
|
||||
if family == "foveabox":
|
||||
del cfg["min_bbox_size"]
|
||||
return cfg
|
||||
|
||||
|
||||
def recipe(model):
|
||||
if model not in MODELS:
|
||||
raise ValueError(model)
|
||||
dp = get_recipe(model)["preprocess"]["official_data_preprocessor"]
|
||||
return {"recipe": "mmdet-dense-static640-v1", "input_spec": INPUT_SPEC,
|
||||
"color": "RGB" if dp["bgr_to_rgb"] else "BGR",
|
||||
"mean": dp["mean"], "std": dp["std"],
|
||||
"resize": {"scale": [640, 640], "keep_ratio": True,
|
||||
"rounding": "int(value+0.5)", "backend": "cv2", "interpolation": "bilinear"},
|
||||
"pad": {"size": [640, 640], "placement": "bottom_right", "value": 0,
|
||||
"stage": "after_normalization"}, "layout": "NCHW", "dtype": "float32"}
|
||||
|
||||
|
||||
def geometry(original_size):
|
||||
w, h = map(int, original_size)
|
||||
if min(w, h) <= 0:
|
||||
raise ValueError(original_size)
|
||||
scale = 640 / max(w, h)
|
||||
nw, nh = int(w * scale + .5), int(h * scale + .5)
|
||||
if min(nw, nh) < 1:
|
||||
raise ValueError("Image collapses under official keep-ratio resize")
|
||||
# Clipping is to resized image geometry, not the normalized padded canvas.
|
||||
return (nw, nh), (nw / w, nh / h), (nh, nw)
|
||||
|
||||
|
||||
def preprocess_variant(model, image, input_spec=INPUT_SPEC):
|
||||
if input_spec != INPUT_SPEC:
|
||||
raise ValueError(f"Expected {INPUT_SPEC}, got {input_spec}")
|
||||
import mmcv
|
||||
import numpy as np
|
||||
meta = recipe(model)
|
||||
bgr = np.asarray(image.convert("RGB"))[:, :, ::-1].copy()
|
||||
x = mmcv.imrescale(bgr, (640, 640), interpolation="bilinear", backend="cv2")
|
||||
if meta["color"] == "RGB":
|
||||
x = x[:, :, ::-1]
|
||||
x = np.ascontiguousarray(x.transpose(2, 0, 1), dtype=np.float32)
|
||||
x = (x - np.asarray(meta["mean"], dtype=np.float32)[:, None, None]) / np.asarray(meta["std"], dtype=np.float32)[:, None, None]
|
||||
out = np.zeros((1, 3, 640, 640), dtype=np.float32)
|
||||
out[0, :, :x.shape[1], :x.shape[2]] = x
|
||||
return out
|
||||
|
||||
|
||||
@lru_cache(maxsize=8)
|
||||
def _decoder(family):
|
||||
# The official BaseDenseHead owns filtering, centerness, clipping, rescaling
|
||||
# and class-aware NMS. This context has no learned layers or checkpoint.
|
||||
from mmengine.config import ConfigDict
|
||||
from mmdet.models.dense_heads.base_dense_head import BaseDenseHead
|
||||
from mmdet.models.task_modules.prior_generators import AnchorGenerator, MlvlPointGenerator
|
||||
from mmdet.models.task_modules.coders import DeltaXYWHBBoxCoder, DistancePointBBoxCoder, TBLRBBoxCoder
|
||||
|
||||
class Decoder(BaseDenseHead):
|
||||
def loss_by_feat(self, **kwargs):
|
||||
raise NotImplementedError("Inference-only decoder")
|
||||
|
||||
if family == "gfl":
|
||||
from mmdet.models.dense_heads.gfl_head import GFLHead, Integral
|
||||
Decoder._predict_by_feat_single = GFLHead._predict_by_feat_single
|
||||
Decoder.anchor_center = GFLHead.anchor_center
|
||||
elif family in ("foveabox", "reppoints"):
|
||||
from mmdet.models.dense_heads import FoveaHead, RepPointsHead
|
||||
official = FoveaHead if family == "foveabox" else RepPointsHead
|
||||
Decoder._predict_by_feat_single = official._predict_by_feat_single
|
||||
Decoder._bbox_decode = official._bbox_decode
|
||||
decoder = Decoder()
|
||||
if family in ("fcos", "vfnet", "foveabox", "reppoints"):
|
||||
decoder.prior_generator = MlvlPointGenerator(STRIDES,
|
||||
offset=.5 if family in ("fcos", "foveabox") else 0.)
|
||||
else:
|
||||
retina = family == "retinanet"
|
||||
decoder.prior_generator = AnchorGenerator(strides=STRIDES,
|
||||
ratios=[.5, 1., 2.] if retina else [1.],
|
||||
octave_base_scale=4 if retina else (1 if family == "fsaf" else 8),
|
||||
scales_per_octave=3 if retina else 1)
|
||||
if family in ("retinanet", "atss"):
|
||||
decoder.bbox_coder = DeltaXYWHBBoxCoder(target_means=[0., 0., 0., 0.],
|
||||
target_stds=[.1, .1, .2, .2] if family == "atss" else [1., 1., 1., 1.])
|
||||
elif family == "fsaf":
|
||||
decoder.bbox_coder = TBLRBBoxCoder(normalizer=4.0)
|
||||
else:
|
||||
decoder.bbox_coder = DistancePointBBoxCoder()
|
||||
if family == "gfl":
|
||||
decoder.integral = Integral(16)
|
||||
elif family == "foveabox":
|
||||
decoder.strides = list(STRIDES)
|
||||
decoder.base_edge_list = [16, 32, 64, 128, 256]
|
||||
elif family == "reppoints":
|
||||
decoder.point_strides = list(STRIDES)
|
||||
decoder.cls_out_channels = 80
|
||||
decoder.use_sigmoid_cls = True
|
||||
decoder.loss_cls = SimpleNamespace(custom_cls_channels=False)
|
||||
decoder.test_cfg = ConfigDict(test_config(family))
|
||||
return decoder
|
||||
|
||||
|
||||
def output_names(model):
|
||||
groups = ("cls", "bbox", "centerness") if model.split(".")[0] in ("fcos", "atss") else ("cls", "bbox")
|
||||
return [f"{group}_s{s}" for group in groups for s in STRIDES]
|
||||
|
||||
|
||||
def decode_variant(model, outputs, original_size):
|
||||
import numpy as np
|
||||
import torch
|
||||
family = model.split(".")[0]
|
||||
channels = HEADS[family][1]
|
||||
if len(outputs) != len(channels) * 5:
|
||||
raise ValueError(f"{model}: wrong raw head count {len(outputs)}")
|
||||
raw = []
|
||||
for i, value in enumerate(outputs):
|
||||
expected = (1, channels[i // 5], 640 // STRIDES[i % 5], 640 // STRIDES[i % 5])
|
||||
array = np.asarray(value)
|
||||
if array.shape != expected or not np.isfinite(array).all():
|
||||
raise ValueError(f"{model}: invalid raw head {i}: {array.shape}, expected {expected}")
|
||||
raw.append(torch.as_tensor(np.ascontiguousarray(array), dtype=torch.float32))
|
||||
_, scales, img_shape = geometry(original_size)
|
||||
meta = dict(img_shape=img_shape, ori_shape=tuple(reversed(original_size)),
|
||||
pad_shape=(640, 640), batch_input_shape=(640, 640), scale_factor=scales)
|
||||
with torch.inference_mode():
|
||||
result = _decoder(family).predict_by_feat(
|
||||
raw[:5], raw[5:10], score_factors=raw[10:] if len(channels) == 3 else None,
|
||||
batch_img_metas=[meta], rescale=True)[0]
|
||||
return result.bboxes.numpy(), result.scores.numpy(), result.labels.numpy()
|
||||
|
||||
|
||||
def build_variant(model, weights, input_spec):
|
||||
import torch
|
||||
raw = build_official(model, weights, input_spec)
|
||||
cfg, head = raw.cfg, raw.upstream.bbox_head
|
||||
meta = recipe(model)
|
||||
dp = cfg.model.data_preprocessor
|
||||
family = model.split(".")[0]
|
||||
if (dict(cfg.model.test_cfg) != test_config(family) or head.num_classes != 80
|
||||
or head.__class__.__name__ != HEADS[family][0]
|
||||
or not head.use_sigmoid_cls or head.cls_out_channels != 80
|
||||
or list(dp.mean) != meta["mean"] or list(dp.std) != meta["std"]
|
||||
or dp.bgr_to_rgb != (meta["color"] == "RGB") or dp.pad_size_divisor != 32):
|
||||
raise ValueError("Pinned official dense detector contract changed")
|
||||
decoder = _decoder(family)
|
||||
# Compare actual upstream geometry helpers, including defaults not in config.
|
||||
for name in ("strides", "num_base_priors"):
|
||||
if getattr(head.prior_generator, name) != getattr(decoder.prior_generator, name):
|
||||
raise ValueError(f"Official prior {name} differs")
|
||||
sizes = [(640 // s, 640 // s) for s in STRIDES]
|
||||
for expected, actual in zip(head.prior_generator.grid_priors(sizes, device="cpu"),
|
||||
decoder.prior_generator.grid_priors(sizes, device="cpu")):
|
||||
torch.testing.assert_close(actual, expected, rtol=0, atol=0)
|
||||
if head.bbox_coder.__dict__ != decoder.bbox_coder.__dict__:
|
||||
raise ValueError("Official bbox coder differs")
|
||||
|
||||
if family == "gfl":
|
||||
torch.testing.assert_close(head.integral.project, decoder.integral.project, rtol=0, atol=0)
|
||||
elif family == "foveabox":
|
||||
if (list(head.strides) != decoder.strides or list(head.base_edge_list) != decoder.base_edge_list
|
||||
or head.with_deform):
|
||||
raise ValueError("Official Fovea geometry differs")
|
||||
elif family == "reppoints":
|
||||
if (head.point_strides != decoder.point_strides or head.num_points != 9
|
||||
or head.transform_method != "moment"):
|
||||
raise ValueError("Official RepPoints geometry differs")
|
||||
|
||||
if family in ("vfnet", "reppoints"):
|
||||
# VFNet's upstream head overrides its state-load hook; verify every
|
||||
# learned parameter against the checkpoint before wrapping export ops.
|
||||
state = torch.load(str(weights), map_location="cpu")["state_dict"]
|
||||
for name, parameter in raw.upstream.named_parameters():
|
||||
torch.testing.assert_close(parameter, state[name], rtol=0, atol=0)
|
||||
del state
|
||||
from .vfnet_deformconv import enable_export
|
||||
if family == "reppoints":
|
||||
enable_export(head, ("reppoints_cls_conv", "reppoints_pts_refine_conv"))
|
||||
else:
|
||||
enable_export(head)
|
||||
|
||||
class RawHeads(torch.nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.upstream = raw.upstream
|
||||
self.upstream.cfg = cfg
|
||||
self.cfg = cfg
|
||||
self.checkpoint_sha256 = raw.checkpoint_sha256
|
||||
|
||||
def forward(self, x):
|
||||
return tuple(value for level in self.upstream._forward(x) for value in level)
|
||||
|
||||
return RawHeads().eval()
|
||||
141
harness/adapters/mmdet_detr_common.py
Normal file
141
harness/adapters/mmdet_detr_common.py
Normal file
@ -0,0 +1,141 @@
|
||||
"""Exact pinned MMDetection DETR models, including their complete transformers.
|
||||
|
||||
Static640 follows the CSV deployment protocol. Padding masks are derived inside
|
||||
the graph from normalized-zero padding, never frozen to the export image shape.
|
||||
Only parameter-free official box selection and geometry run in host decode.
|
||||
"""
|
||||
from functools import lru_cache
|
||||
from types import SimpleNamespace
|
||||
|
||||
from .openmmlab import build as build_official, get_recipe
|
||||
from .mmdet_dense_common import RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, geometry
|
||||
|
||||
MODELS = ("detr.r50", "deformable.detr.r50", "dab.detr.r50")
|
||||
OPSET = 16 # Standard ONNX GridSample for genuine multi-scale deformable attention.
|
||||
CONTRACTS = {"detr.r50": ("DETR", "DETRHead", 100, 81, 100),
|
||||
"deformable.detr.r50": ("DeformableDETR", "DeformableDETRHead", 300, 80, 100),
|
||||
"dab.detr.r50": ("DABDETR", "DABDETRHead", 300, 80, 300)}
|
||||
|
||||
|
||||
def recipe(model):
|
||||
if model not in MODELS:
|
||||
raise ValueError(model)
|
||||
from .mmdet_dense_common import recipe as image_recipe
|
||||
meta = image_recipe("retinanet.r50.fpn")
|
||||
dp = get_recipe(model)["preprocess"]["official_data_preprocessor"]
|
||||
if (dp["mean"] != meta["mean"] or dp["std"] != meta["std"]
|
||||
or not dp["bgr_to_rgb"] or dp["pad_size_divisor"] != 1):
|
||||
raise ValueError("Official DETR image normalization changed")
|
||||
# Same image transformation permits existing calibration cache reuse.
|
||||
return meta
|
||||
|
||||
|
||||
def preprocess_variant(model, image, input_spec=INPUT_SPEC):
|
||||
from .mmdet_dense_common import preprocess_variant as image_preprocess
|
||||
recipe(model) # verify exact equality before reusing the image-only helper
|
||||
return image_preprocess("retinanet.r50.fpn", image, input_spec)
|
||||
|
||||
|
||||
def output_names(model):
|
||||
if model not in MODELS:
|
||||
raise ValueError(model)
|
||||
return ["class_logits", "boxes_cxcywh"]
|
||||
|
||||
|
||||
def padding_mask(x):
|
||||
# Resized RGB pixels are uint8 and all official channel means are fractional:
|
||||
# a real pixel cannot be normalized zero. Zero is exclusively padding.
|
||||
return (x == 0).all(dim=1)
|
||||
|
||||
|
||||
def transformer_inputs(network, features, x, deformable):
|
||||
"""Official pre_transformer math with the real per-image mask retained."""
|
||||
import torch
|
||||
from torch.nn import functional as F
|
||||
mask = padding_mask(x).to(x.dtype)
|
||||
masks = [F.interpolate(mask[:, None], size=f.shape[-2:]).to(torch.bool)[:, 0]
|
||||
for f in features]
|
||||
positions = [network.positional_encoding(m) for m in masks]
|
||||
flatten = lambda f: f.flatten(2).transpose(1, 2)
|
||||
if not deformable:
|
||||
feat, pos, mask = flatten(features[-1]), flatten(positions[-1]), masks[-1].flatten(1)
|
||||
return dict(feat=feat, feat_mask=mask, feat_pos=pos), dict(memory_mask=mask, memory_pos=pos)
|
||||
shapes = torch.tensor([list(f.shape[-2:]) for f in features], dtype=torch.long, device=x.device)
|
||||
starts = torch.cat((shapes.new_zeros(1), shapes.prod(1).cumsum(0)[:-1]))
|
||||
valid = torch.stack([network.get_valid_ratio(m) for m in masks], 1)
|
||||
mask = torch.cat([m.flatten(1) for m in masks], 1)
|
||||
pos = torch.cat([flatten(p) + network.level_embed[i].view(1, 1, -1)
|
||||
for i, p in enumerate(positions)], 1)
|
||||
extra = dict(spatial_shapes=shapes, level_start_index=starts, valid_ratios=valid)
|
||||
return (dict(feat=torch.cat([flatten(f) for f in features], 1), feat_mask=mask, feat_pos=pos, **extra),
|
||||
dict(memory_mask=mask, **extra))
|
||||
|
||||
|
||||
@lru_cache(maxsize=3)
|
||||
def _decoder(model):
|
||||
from mmdet.models.dense_heads.detr_head import DETRHead
|
||||
_, _, _, channels, topk = CONTRACTS[model]
|
||||
context = SimpleNamespace(test_cfg={"max_per_img": topk}, num_classes=80,
|
||||
loss_cls=SimpleNamespace(use_sigmoid=channels == 80))
|
||||
from functools import partial
|
||||
return partial(DETRHead._predict_by_feat_single, context)
|
||||
|
||||
|
||||
def decode_variant(model, outputs, original_size):
|
||||
import numpy as np
|
||||
import torch
|
||||
_, _, queries, channels, _ = CONTRACTS[model]
|
||||
if len(outputs) != 2:
|
||||
raise ValueError("Expected final query logits and cxcywh tensors")
|
||||
tensors = []
|
||||
for value, shape in zip(outputs, ((1, queries, channels), (1, queries, 4))):
|
||||
value = np.asarray(value)
|
||||
if value.shape != shape or not np.isfinite(value).all():
|
||||
raise ValueError(f"Invalid {model} raw output: {value.shape}, expected {shape}")
|
||||
tensors.append(torch.tensor(value, dtype=torch.float32)[0])
|
||||
_, scales, img_shape = geometry(original_size)
|
||||
with torch.inference_mode():
|
||||
result = _decoder(model)(*tensors, dict(img_shape=img_shape, scale_factor=scales), rescale=True)
|
||||
return result.bboxes.numpy(), result.scores.numpy(), result.labels.numpy()
|
||||
|
||||
|
||||
def build_variant(model, weights, input_spec):
|
||||
import torch
|
||||
raw = build_official(model, weights, input_spec)
|
||||
network, cfg = raw.upstream, raw.cfg
|
||||
kind, head, queries, channels, topk = CONTRACTS[model]
|
||||
meta = recipe(model)
|
||||
dp = cfg.model.data_preprocessor
|
||||
if (type(network).__name__ != kind or type(network.bbox_head).__name__ != head
|
||||
or network.num_queries != queries or network.bbox_head.cls_out_channels != channels
|
||||
or network.bbox_head.num_classes != 80 or dict(cfg.model.test_cfg) != {"max_per_img": topk}
|
||||
or list(dp.mean) != meta["mean"] or list(dp.std) != meta["std"]
|
||||
or not dp.bgr_to_rgb or dp.pad_size_divisor != 1):
|
||||
raise ValueError("Pinned official DETR contract changed")
|
||||
deformable = kind == "DeformableDETR"
|
||||
if deformable and (network.with_box_refine or network.as_two_stage or network.num_feature_levels != 4):
|
||||
raise ValueError("Wrong Deformable-DETR variant")
|
||||
state = torch.load(str(weights), map_location="cpu")["state_dict"]
|
||||
for name, parameter in network.named_parameters():
|
||||
torch.testing.assert_close(parameter, state[name], rtol=0, atol=0)
|
||||
del state
|
||||
|
||||
class RawQueries(torch.nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.upstream = network
|
||||
self.upstream.cfg = cfg
|
||||
self.cfg = cfg
|
||||
self.checkpoint_sha256 = raw.checkpoint_sha256
|
||||
|
||||
def forward(self, x):
|
||||
features = self.upstream.extract_feat(x)
|
||||
encoder, decoder = transformer_inputs(self.upstream, features, x, deformable)
|
||||
memory = self.upstream.forward_encoder(**encoder)
|
||||
query, head_inputs = self.upstream.pre_decoder(**memory)
|
||||
decoder.update(query)
|
||||
head_inputs.update(self.upstream.forward_decoder(**decoder))
|
||||
scores, boxes = self.upstream.bbox_head(**head_inputs)
|
||||
return scores[-1], boxes[-1]
|
||||
|
||||
return RawQueries().eval()
|
||||
290
harness/adapters/mmdet_iseg_common.py
Normal file
290
harness/adapters/mmdet_iseg_common.py
Normal file
@ -0,0 +1,290 @@
|
||||
"""Pinned MMDetection query-based instance segmentation at the CSV static 640 size.
|
||||
|
||||
Mask2Former COCO instance checkpoints: the graph returns the final decoder
|
||||
layer's query classification logits and per-query mask logits over the
|
||||
stride-4 mask feature map. The official MaskFormerHead.predict upsampling and
|
||||
the parameter-free MaskFormerFusionHead.instance_postprocess run in host decode
|
||||
on those two raw tensors, exactly like the dense-detector decoders.
|
||||
|
||||
Deployment geometry follows mmdet_dense_common (upstream 1333x800 keep-ratio
|
||||
test resize pinned to 640, normalized zero padding at bottom/right), so the
|
||||
image recipe and calibration digests stay shared. Heavy imports stay inside
|
||||
runtime functions for lightweight recipe discovery.
|
||||
"""
|
||||
from functools import lru_cache
|
||||
|
||||
from .openmmlab import get_recipe, load_config, verify_checkpoint
|
||||
from .mmdet_dense_common import RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, geometry
|
||||
|
||||
MODELS = ("mask2former.swin.t.instance", "mask2former.swin.s.instance",
|
||||
"mask2former.swin.b.instance")
|
||||
# stage-3 blocks, patch-embed channels, queries, classes, stuff classes and
|
||||
# the official test_cfg flavor, per resolved official config.
|
||||
CONTRACTS = {"mask2former.swin.t.instance": (6, 96, 100, 80, 0, "instance"),
|
||||
"mask2former.swin.s.instance": (18, 96, 100, 80, 0, "instance"),
|
||||
"mask2former.swin.b.instance": (18, 128, 100, 133, 53, "panoptic")}
|
||||
TEST_CFGS = {"instance": {"panoptic_on": False, "semantic_on": False, "instance_on": True,
|
||||
"max_per_image": 100, "iou_thr": 0.8, "filter_low_score": True},
|
||||
"panoptic": {"panoptic_on": True, "semantic_on": False, "instance_on": True,
|
||||
"max_per_image": 100, "iou_thr": 0.8, "filter_low_score": True}}
|
||||
# The COCO instance configs load to float32 before resizing; the swin-b
|
||||
# panoptic config's test pipeline keeps the default uint8 load.
|
||||
F32_LOAD = {"mask2former.swin.t.instance", "mask2former.swin.s.instance"}
|
||||
|
||||
|
||||
def recipe(model):
|
||||
"""Image recipe variant: instance configs load to float32 before resizing."""
|
||||
if model not in MODELS:
|
||||
raise ValueError(model)
|
||||
from .mmdet_dense_common import recipe as image_recipe
|
||||
meta = image_recipe("retinanet.r50.fpn")
|
||||
dp = get_recipe(model)["preprocess"]["official_data_preprocessor"]
|
||||
if (dp["mean"] != meta["mean"] or dp["std"] != meta["std"]
|
||||
or not dp["bgr_to_rgb"] or dp["pad_size_divisor"] != 32):
|
||||
raise ValueError("Official Mask2Former image normalization changed")
|
||||
if model in F32_LOAD:
|
||||
return {**meta, "recipe": "mmdet-mask2former-static640-f32-v1",
|
||||
"load_dtype": "float32_before_resize"}
|
||||
return meta
|
||||
|
||||
|
||||
def preprocess_variant(model, image, input_spec=INPUT_SPEC):
|
||||
if input_spec != INPUT_SPEC:
|
||||
raise ValueError(f"Expected {INPUT_SPEC}, got {input_spec}")
|
||||
meta = recipe(model)
|
||||
if model not in F32_LOAD:
|
||||
from .mmdet_dense_common import preprocess_variant as image_preprocess
|
||||
return image_preprocess("retinanet.r50.fpn", image, input_spec)
|
||||
import mmcv
|
||||
import numpy as np
|
||||
# Official pipeline: LoadImageFromFile(to_float32=True) -> Resize -> normalize.
|
||||
bgr = np.asarray(image.convert("RGB"))[:, :, ::-1].copy().astype(np.float32)
|
||||
x = mmcv.imrescale(bgr, (640, 640), interpolation="bilinear", backend="cv2")
|
||||
if meta["color"] == "RGB":
|
||||
x = x[:, :, ::-1]
|
||||
x = np.ascontiguousarray(x.transpose(2, 0, 1), dtype=np.float32)
|
||||
x = (x - np.asarray(meta["mean"], dtype=np.float32)[:, None, None]) / np.asarray(meta["std"], dtype=np.float32)[:, None, None]
|
||||
out = np.zeros((1, 3, 640, 640), dtype=np.float32)
|
||||
out[0, :, :x.shape[1], :x.shape[2]] = x
|
||||
return out
|
||||
|
||||
|
||||
def output_names(model):
|
||||
if model not in MODELS:
|
||||
raise ValueError(model)
|
||||
return ["cls_logits", "mask_logits"]
|
||||
|
||||
|
||||
DEBUG_OUTPUT_NAMES = ["feat3", "mask_features", "msm0", "msm1", "msm2", "cls_logits", "mask_logits"]
|
||||
|
||||
|
||||
def build_debug_variant(model, weights, input_spec):
|
||||
"""Same official forward with pre-decoder tensors exposed for the probe.
|
||||
|
||||
The Mask2Former transformer decoder derives boolean attention masks from
|
||||
interpolated mask logits (sigmoid < 0.5). Torch/ONNX fp reassociation
|
||||
(~1e-4) can flip borderline bits, and each flip changes decoder outputs
|
||||
discretely (measured: flipping 0.001% of the bits moves class logits by
|
||||
5.2 and mask logits by 112). Raw-output cosine therefore only gates the
|
||||
continuous backbone/pixel-decoder segment; decoder equivalence is judged
|
||||
on decoded predictions and the COCO metric.
|
||||
"""
|
||||
network = build_variant(model, weights, input_spec)
|
||||
upstream, head = network.upstream, network.upstream.panoptic_head
|
||||
import torch
|
||||
|
||||
class DebugTaps(torch.nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.raw = network
|
||||
|
||||
def forward(self, x):
|
||||
feats = upstream.extract_feat(x)
|
||||
mask_features, multi_scale_memorys = head.pixel_decoder(feats)
|
||||
batch_size = x.shape[0]
|
||||
decoder_inputs, decoder_positional_encodings = [], []
|
||||
for i in range(head.num_transformer_feat_level):
|
||||
decoder_input = head.decoder_input_projs[i](multi_scale_memorys[i])
|
||||
decoder_input = decoder_input.flatten(2).permute(0, 2, 1)
|
||||
decoder_input = decoder_input + head.level_embed.weight[i].view(1, 1, -1)
|
||||
mask = decoder_input.new_zeros(
|
||||
(batch_size,) + multi_scale_memorys[i].shape[-2:], dtype=torch.bool)
|
||||
positional = head.decoder_positional_encoding(mask).flatten(2).permute(0, 2, 1)
|
||||
decoder_inputs.append(decoder_input)
|
||||
decoder_positional_encodings.append(positional)
|
||||
query_feat = head.query_feat.weight.unsqueeze(0).repeat((batch_size, 1, 1))
|
||||
query_embed = head.query_embed.weight.unsqueeze(0).repeat((batch_size, 1, 1))
|
||||
cls_pred, mask_pred, attn_mask = head._forward_head(
|
||||
query_feat, mask_features, multi_scale_memorys[0].shape[-2:])
|
||||
for i in range(head.num_transformer_decoder_layers):
|
||||
level_idx = i % head.num_transformer_feat_level
|
||||
mask_sum = (attn_mask.sum(-1) != attn_mask.shape[-1]).unsqueeze(-1)
|
||||
attn_mask = attn_mask & mask_sum
|
||||
layer = head.transformer_decoder.layers[i]
|
||||
query_feat = layer(
|
||||
query=query_feat, key=decoder_inputs[level_idx],
|
||||
value=decoder_inputs[level_idx], query_pos=query_embed,
|
||||
key_pos=decoder_positional_encodings[level_idx],
|
||||
cross_attn_mask=attn_mask, query_key_padding_mask=None,
|
||||
key_padding_mask=None)
|
||||
cls_pred, mask_pred, attn_mask = head._forward_head(
|
||||
query_feat, mask_features,
|
||||
multi_scale_memorys[(i + 1) % head.num_transformer_feat_level].shape[-2:])
|
||||
return (feats[-1], mask_features, multi_scale_memorys[0],
|
||||
multi_scale_memorys[1], multi_scale_memorys[2], cls_pred, mask_pred)
|
||||
|
||||
return DebugTaps().eval()
|
||||
|
||||
|
||||
@lru_cache(maxsize=3)
|
||||
def _decoder(model):
|
||||
"""Official parameter-free instance postprocess; no learned weights here."""
|
||||
from mmdet.models.seg_heads.panoptic_fusion_heads.maskformer_fusion_head import MaskFormerFusionHead
|
||||
from mmengine.config import ConfigDict
|
||||
_, _, _, classes, stuff, flavor = CONTRACTS[model]
|
||||
decoder = MaskFormerFusionHead.__new__(MaskFormerFusionHead)
|
||||
decoder.num_classes = classes
|
||||
decoder.num_things_classes = classes - stuff
|
||||
decoder.num_stuff_classes = stuff
|
||||
decoder.test_cfg = ConfigDict(TEST_CFGS[flavor])
|
||||
return decoder
|
||||
|
||||
|
||||
def decode_variant(model, outputs, original_size):
|
||||
"""Official predict path: upsample -> crop -> rescale -> instance_postprocess."""
|
||||
import numpy as np
|
||||
import torch
|
||||
from torch.nn import functional as F
|
||||
if model not in MODELS:
|
||||
raise ValueError(model)
|
||||
queries, classes = CONTRACTS[model][2], CONTRACTS[model][3]
|
||||
if len(outputs) != 2:
|
||||
raise ValueError(f"{model}: expected query logits and mask logits")
|
||||
cls_logits, mask_logits = outputs
|
||||
cls_logits = np.asarray(cls_logits)
|
||||
mask_logits = np.asarray(mask_logits)
|
||||
if cls_logits.shape != (1, queries, classes + 1) or not np.isfinite(cls_logits).all():
|
||||
raise ValueError(f"{model}: invalid cls logits {cls_logits.shape}")
|
||||
if (mask_logits.ndim != 4 or mask_logits.shape[:2] != (1, queries)
|
||||
or not np.isfinite(mask_logits).all()):
|
||||
raise ValueError(f"{model}: invalid mask logits {mask_logits.shape}")
|
||||
(nw, nh), scales, img_shape = geometry(original_size)
|
||||
size = int(INPUT_SPEC.split("x")[-1])
|
||||
with torch.inference_mode():
|
||||
# MaskFormerHead.predict: upsample to batch_input_shape first.
|
||||
masks = F.interpolate(torch.from_numpy(mask_logits), size=(size, size),
|
||||
mode="bilinear", align_corners=False)
|
||||
# MaskFormerFusionHead.predict: remove padding, then rescale.
|
||||
masks = masks[0, ..., :nh, :nw]
|
||||
ori_h, ori_w = int(original_size[1]), int(original_size[0])
|
||||
masks = F.interpolate(masks[:, None], size=(ori_h, ori_w),
|
||||
mode="bilinear", align_corners=False)[:, 0]
|
||||
result = _decoder(model).instance_postprocess(
|
||||
torch.from_numpy(cls_logits)[0], masks)
|
||||
return (result.bboxes.numpy(), result.scores.numpy(), result.labels.numpy(),
|
||||
result.masks.numpy())
|
||||
|
||||
|
||||
def g2_input_variant(model, t1, item, input_spec):
|
||||
"""Diagnostic-input hook for harness/eval/run.py g2_inputs."""
|
||||
return preprocess_variant(model, t1.image(item), input_spec)
|
||||
|
||||
|
||||
def predict_variant(model, runner, t1, input_spec, pose_evidence=None):
|
||||
"""Shared harness/eval/coco_extra.py hook: COCO segm RLE predictions."""
|
||||
import numpy as np
|
||||
from pycocotools import mask as mask_utils
|
||||
from ..eval.common import run_batched
|
||||
if pose_evidence is not None:
|
||||
raise ValueError(f"{model}: iseg predict takes no pose evidence")
|
||||
cat_map = t1.manifest["cat_map"]
|
||||
predictions = []
|
||||
for offset in range(0, len(t1.items), 16):
|
||||
items = t1.items[offset:offset + 16]
|
||||
images = [t1.image(it) for it in items]
|
||||
xs = [preprocess_variant(model, image, input_spec) for image in images]
|
||||
outputs = run_batched(runner, xs)
|
||||
for it, image, outs in zip(items, images, outputs):
|
||||
_, scores, labels, masks = decode_variant(model, outs, image.size)
|
||||
for mask, score, label in zip(masks, scores, labels):
|
||||
rle = mask_utils.encode(np.asfortranarray(mask.astype(np.uint8)))
|
||||
rle["counts"] = rle["counts"].decode("ascii")
|
||||
predictions.append({"image_id": it["id"],
|
||||
"category_id": cat_map[str(int(label))],
|
||||
"score": float(score), "segmentation": rle})
|
||||
return predictions
|
||||
|
||||
|
||||
def build_variant(model, weights, input_spec):
|
||||
"""Construct the official Mask2Former and strictly load its checkpoint.
|
||||
|
||||
MaskFormer has no bbox_head, so the shared openmmlab tensor route cannot
|
||||
serve it; this builder uses the same public recipe/config/checkpoint
|
||||
helpers and the same strict-load rules as openmmlab.build.
|
||||
"""
|
||||
import importlib
|
||||
import torch
|
||||
from mmengine.registry import init_default_scope
|
||||
from mmengine.runner import load_checkpoint
|
||||
|
||||
if model not in MODELS or input_spec != INPUT_SPEC:
|
||||
raise ValueError(f"{model}: expected {INPUT_SPEC}")
|
||||
model_recipe = get_recipe(model)
|
||||
if model_recipe["resolution_status"] != "resolved_catalog":
|
||||
raise ValueError(f"Unresolved recipe {model}: {model_recipe['blockers']}")
|
||||
if input_spec != model_recipe["input_spec"]:
|
||||
raise ValueError(f"{model}: recipe input {model_recipe['input_spec']}, got {input_spec}")
|
||||
sha = verify_checkpoint(model_recipe, weights)
|
||||
cfg = load_config(model)
|
||||
importlib.import_module("mmdet.utils").register_all_modules(init_default_scope=False)
|
||||
init_default_scope("mmdet")
|
||||
registry = importlib.import_module("mmdet.registry")
|
||||
network = registry.MODELS.build(cfg.model)
|
||||
# MMEngine marks normalization buffers non-persistent; official checkpoints
|
||||
# keep them. Preserve every key and verify before restoring persistence.
|
||||
state = torch.load(str(weights), map_location="cpu")["state_dict"]
|
||||
preprocessor = network.data_preprocessor
|
||||
for name in ("mean", "std"):
|
||||
key = f"data_preprocessor.{name}"
|
||||
if key in state and name in preprocessor._non_persistent_buffers_set:
|
||||
expected = getattr(preprocessor, name)
|
||||
if not torch.equal(state[key], expected):
|
||||
raise ValueError(f"Checkpoint {key} disagrees with the official config")
|
||||
preprocessor.register_buffer(name, expected, persistent=True)
|
||||
load_checkpoint(network, str(weights), map_location="cpu", strict=True)
|
||||
network.cpu().eval()
|
||||
|
||||
depth3, embed, queries, classes, stuff, flavor = CONTRACTS[model]
|
||||
head = network.panoptic_head
|
||||
dp = cfg.model.data_preprocessor
|
||||
meta = recipe(model)
|
||||
if (type(network).__name__ != "Mask2Former" or type(head).__name__ != "Mask2FormerHead"
|
||||
or type(network.backbone).__name__ != "SwinTransformer"
|
||||
or [len(stage.blocks) for stage in network.backbone.stages] != [2, 2, depth3, 2]
|
||||
or network.backbone.patch_embed.projection.out_channels != embed
|
||||
or head.num_queries != queries or head.num_classes != classes
|
||||
or head.num_things_classes != classes - stuff or head.num_stuff_classes != stuff
|
||||
or dict(cfg.model.test_cfg) != TEST_CFGS[flavor]
|
||||
or list(dp.mean) != meta["mean"] or list(dp.std) != meta["std"]
|
||||
or not dp.bgr_to_rgb or dp.pad_size_divisor != 32):
|
||||
raise ValueError("Pinned official Mask2Former contract changed")
|
||||
# Verify every learned parameter against the checkpoint bytes.
|
||||
for name, parameter in network.named_parameters():
|
||||
torch.testing.assert_close(parameter, state[name], rtol=0, atol=0)
|
||||
del state
|
||||
|
||||
class RawQueries(torch.nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.upstream = network
|
||||
self.upstream.cfg = cfg
|
||||
self.cfg = cfg
|
||||
self.checkpoint_sha256 = sha
|
||||
|
||||
def forward(self, x):
|
||||
# Mask2FormerHead.forward is metadata-free: batch size from x.
|
||||
cls_pred_list, mask_pred_list = self.upstream.panoptic_head(
|
||||
self.upstream.extract_feat(x), None)
|
||||
return cls_pred_list[-1], mask_pred_list[-1]
|
||||
|
||||
return RawQueries().eval()
|
||||
209
harness/adapters/mmdet_solo_common.py
Normal file
209
harness/adapters/mmdet_solo_common.py
Normal file
@ -0,0 +1,209 @@
|
||||
"""Pinned MMDetection SOLO/SOLOv2 instance segmentation at static 640.
|
||||
|
||||
Both heads emit dense static tensors (per-grid mask/kernel predictions, class
|
||||
scores and a unified mask feature map); the official matrix-NMS decode and the
|
||||
dynamic mask convolutions are parameter-free and run in host decode, exactly
|
||||
like the dense-detector decoders. SOLO result bboxes are upstream placeholders
|
||||
(all zeros); masks carry the real geometry.
|
||||
|
||||
The test pipeline keeps the default uint8 load (no to_float32), so the image
|
||||
recipe and calibration digest stay shared with the dense detectors.
|
||||
"""
|
||||
from functools import lru_cache
|
||||
|
||||
from .openmmlab import build as build_official, get_recipe
|
||||
from .mmdet_dense_common import RUNTIME_PYTHON, INPUT_SPEC, MODEL_SPEC, geometry
|
||||
|
||||
MODELS = ("solo.r50", "solov2.r50")
|
||||
GRIDS = (40, 36, 24, 16, 12)
|
||||
STRIDES = (8, 8, 16, 32, 32)
|
||||
TEST_CFG = {"nms_pre": 500, "score_thr": .1, "mask_thr": .5, "filter_thr": .05,
|
||||
"kernel": "gaussian", "sigma": 2.0, "max_per_img": 100}
|
||||
|
||||
|
||||
def recipe(model):
|
||||
"""Same image recipe as the dense detectors once normalization is verified."""
|
||||
if model not in MODELS:
|
||||
raise ValueError(model)
|
||||
from .mmdet_dense_common import recipe as image_recipe
|
||||
meta = image_recipe("retinanet.r50.fpn")
|
||||
dp = get_recipe(model)["preprocess"]["official_data_preprocessor"]
|
||||
if (dp["mean"] != meta["mean"] or dp["std"] != meta["std"]
|
||||
or not dp["bgr_to_rgb"] or dp["pad_size_divisor"] != 32):
|
||||
raise ValueError("Official SOLO image normalization changed")
|
||||
return meta
|
||||
|
||||
|
||||
def preprocess_variant(model, image, input_spec=INPUT_SPEC):
|
||||
from .mmdet_dense_common import preprocess_variant as image_preprocess
|
||||
recipe(model) # verify exact equality before reusing the image-only helper
|
||||
return image_preprocess("retinanet.r50.fpn", image, input_spec)
|
||||
|
||||
|
||||
def output_names(model):
|
||||
if model not in MODELS:
|
||||
raise ValueError(model)
|
||||
first = "mask" if model == "solo.r50" else "kernel"
|
||||
names = [f"{first}_g{g}" for g in GRIDS] + [f"cls_g{g}" for g in GRIDS]
|
||||
if model == "solov2.r50":
|
||||
names.append("mask_feats")
|
||||
return names
|
||||
|
||||
|
||||
DEBUG_OUTPUT_NAMES = [f"fpn{i}" for i in range(5)]
|
||||
|
||||
|
||||
def build_debug_variant(model, weights, input_spec):
|
||||
"""Backbone+FPN taps: the continuous segment gated at .9999 cosine.
|
||||
|
||||
The SOLO heads stack seven conv+GroupNorm+ReLU blocks over generated
|
||||
coordinate grids; fp reassociation across frameworks flips near-zero
|
||||
pre-activations and shifts head logits by a few 1e-3 (measured, identical
|
||||
at opset 11 and 16). Head equivalence is therefore judged on decoded
|
||||
predictions and the COCO metric, not on raw logits.
|
||||
"""
|
||||
network = build_variant(model, weights, input_spec)
|
||||
import torch
|
||||
|
||||
class NeckTaps(torch.nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.upstream = network.upstream
|
||||
|
||||
def forward(self, x):
|
||||
return tuple(self.upstream.extract_feat(x))
|
||||
|
||||
return NeckTaps().eval()
|
||||
|
||||
|
||||
@lru_cache(maxsize=2)
|
||||
def _decoder(model):
|
||||
"""Official parameter-free SOLO decode context; no learned weights."""
|
||||
from mmengine.config import ConfigDict
|
||||
if model == "solo.r50":
|
||||
from mmdet.models.dense_heads.solo_head import SOLOHead
|
||||
head = SOLOHead.__new__(SOLOHead)
|
||||
head.cls_out_channels = 80
|
||||
else:
|
||||
from mmdet.models.dense_heads.solov2_head import SOLOV2Head
|
||||
head = SOLOV2Head.__new__(SOLOV2Head)
|
||||
head.cls_out_channels = 80
|
||||
head.kernel_out_channels = 256
|
||||
head.dynamic_conv_size = 1
|
||||
head.mask_stride = 4
|
||||
head.num_grids = list(GRIDS)
|
||||
head.num_levels = len(GRIDS)
|
||||
head.strides = list(STRIDES)
|
||||
head.test_cfg = ConfigDict(TEST_CFG)
|
||||
return head
|
||||
|
||||
|
||||
def decode_variant(model, outputs, original_size):
|
||||
"""Official predict_by_feat on the raw dense head tensors."""
|
||||
import numpy as np
|
||||
import torch
|
||||
if model not in MODELS:
|
||||
raise ValueError(model)
|
||||
count = 11 if model == "solov2.r50" else 10
|
||||
if len(outputs) != count:
|
||||
raise ValueError(f"{model}: wrong raw head count {len(outputs)}")
|
||||
tensors = []
|
||||
for index, value in enumerate(outputs):
|
||||
array = np.asarray(value)
|
||||
if array.ndim != 4 or not np.isfinite(array).all():
|
||||
raise ValueError(f"{model}: invalid raw head {index}: {array.shape}")
|
||||
tensors.append(torch.as_tensor(np.ascontiguousarray(array), dtype=torch.float32))
|
||||
for level, grid in enumerate(GRIDS):
|
||||
if tensors[5 + level].shape[1] != 80 or tensors[5 + level].shape[2:] != (grid, grid):
|
||||
raise ValueError(f"{model}: unexpected class head {level}: {tensors[5 + level].shape}")
|
||||
(nw, nh), scales, img_shape = geometry(original_size)
|
||||
meta = dict(img_shape=img_shape, ori_shape=(int(original_size[1]), int(original_size[0])),
|
||||
pad_shape=(640, 640), batch_input_shape=(640, 640), scale_factor=scales)
|
||||
with torch.inference_mode():
|
||||
if model == "solo.r50":
|
||||
result = _decoder(model).predict_by_feat(tensors[:5], tensors[5:10], [meta])[0]
|
||||
else:
|
||||
result = _decoder(model).predict_by_feat(tensors[:5], tensors[5:10], tensors[10], [meta])[0]
|
||||
return (result.bboxes.numpy(), result.scores.numpy(), result.labels.numpy(),
|
||||
result.masks.numpy())
|
||||
|
||||
|
||||
def g2_input_variant(model, t1, item, input_spec):
|
||||
"""Diagnostic-input hook for harness/eval/run.py g2_inputs."""
|
||||
return preprocess_variant(model, t1.image(item), input_spec)
|
||||
|
||||
|
||||
def predict_variant(model, runner, t1, input_spec, pose_evidence=None):
|
||||
"""Shared harness/eval/coco_extra.py hook: COCO segm RLE predictions."""
|
||||
import numpy as np
|
||||
from pycocotools import mask as mask_utils
|
||||
from ..eval.common import run_batched
|
||||
if pose_evidence is not None:
|
||||
raise ValueError(f"{model}: iseg predict takes no pose evidence")
|
||||
cat_map = t1.manifest["cat_map"]
|
||||
predictions = []
|
||||
for offset in range(0, len(t1.items), 16):
|
||||
items = t1.items[offset:offset + 16]
|
||||
images = [t1.image(it) for it in items]
|
||||
xs = [preprocess_variant(model, image, input_spec) for image in images]
|
||||
outputs = run_batched(runner, xs)
|
||||
for it, image, outs in zip(items, images, outputs):
|
||||
_, scores, labels, masks = decode_variant(model, outs, image.size)
|
||||
for mask, score, label in zip(masks, scores, labels):
|
||||
rle = mask_utils.encode(np.asfortranarray(mask.astype(np.uint8)))
|
||||
rle["counts"] = rle["counts"].decode("ascii")
|
||||
predictions.append({"image_id": it["id"],
|
||||
"category_id": cat_map[str(int(label))],
|
||||
"score": float(score), "segmentation": rle})
|
||||
return predictions
|
||||
|
||||
|
||||
def build_variant(model, weights, input_spec):
|
||||
"""Construct the official SOLO model and strictly load its checkpoint."""
|
||||
import torch
|
||||
|
||||
if model not in MODELS or input_spec != INPUT_SPEC:
|
||||
raise ValueError(f"{model}: expected {INPUT_SPEC}")
|
||||
raw = build_official(model, weights, input_spec)
|
||||
network, cfg = raw.upstream, raw.cfg
|
||||
meta = recipe(model)
|
||||
dp = cfg.model.data_preprocessor
|
||||
kind, head_kind = (("SOLO", "SOLOHead") if model == "solo.r50"
|
||||
else ("SOLOv2", "SOLOV2Head"))
|
||||
head = network.mask_head
|
||||
if (type(network).__name__ != kind or type(head).__name__ != head_kind
|
||||
or network.with_bbox or head.num_classes != 80
|
||||
or list(head.strides) != list(STRIDES) or list(head.num_grids) != list(GRIDS)
|
||||
or dict(head.test_cfg) != TEST_CFG
|
||||
or list(dp.mean) != meta["mean"] or list(dp.std) != meta["std"]
|
||||
or not dp.bgr_to_rgb or dp.pad_size_divisor != 32):
|
||||
raise ValueError("Pinned official SOLO contract changed")
|
||||
if model == "solov2.r50" and (head.kernel_out_channels != 256
|
||||
or head.dynamic_conv_size != 1 or head.mask_stride != 4):
|
||||
raise ValueError("Pinned official SOLOv2 mask geometry changed")
|
||||
# Verify every learned parameter against the checkpoint bytes.
|
||||
state = torch.load(str(weights), map_location="cpu")["state_dict"]
|
||||
for name, parameter in network.named_parameters():
|
||||
torch.testing.assert_close(parameter, state[name], rtol=0, atol=0)
|
||||
del state
|
||||
|
||||
class RawHeads(torch.nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.upstream = network
|
||||
self.upstream.cfg = cfg
|
||||
self.cfg = cfg
|
||||
self.checkpoint_sha256 = raw.checkpoint_sha256
|
||||
|
||||
def forward(self, x):
|
||||
outs = self.upstream.extract_feat(x)
|
||||
results = self.upstream.mask_head.forward(outs)
|
||||
flat = []
|
||||
for group in results:
|
||||
if isinstance(group, (list, tuple)):
|
||||
flat.extend(group)
|
||||
else:
|
||||
flat.append(group)
|
||||
return tuple(flat)
|
||||
|
||||
return RawHeads().eval()
|
||||
83
harness/adapters/mmseg_common.py
Normal file
83
harness/adapters/mmseg_common.py
Normal file
@ -0,0 +1,83 @@
|
||||
"""Reuse pinned MMSeg models with an explicit static-image benchmark recipe."""
|
||||
from pathlib import Path
|
||||
|
||||
from .openmmlab import build as official_build, get_recipe
|
||||
|
||||
RUNTIME_PYTHON = "/home/ming01.wang/outputs/full-coverage-20260917/openmmlab/venv/bin/python"
|
||||
EXPORTER_VERSION = "mmseg-static-raw-logits-v1"
|
||||
|
||||
|
||||
def lower_static_pools(network):
|
||||
"""Export exact adaptive-pooling bins for the fixed deployment shape."""
|
||||
import torch
|
||||
|
||||
class StaticPool(torch.nn.AdaptiveAvgPool2d):
|
||||
def forward(self, x):
|
||||
if not torch.onnx.is_in_onnx_export():
|
||||
return super().forward(x)
|
||||
oh, ow = (self.output_size, self.output_size) if isinstance(self.output_size, int) else self.output_size
|
||||
h, w = map(int, x.shape[-2:])
|
||||
if h % oh == 0 and w % ow == 0:
|
||||
return super().forward(x)
|
||||
rows = []
|
||||
for i in range(oh):
|
||||
top, bottom = i * h // oh, ((i + 1) * h + oh - 1) // oh
|
||||
columns = []
|
||||
for j in range(ow):
|
||||
left, right = j * w // ow, ((j + 1) * w + ow - 1) // ow
|
||||
columns.append(x[..., top:bottom, left:right].mean((-2, -1), keepdim=True))
|
||||
rows.append(torch.cat(columns, dim=-1))
|
||||
return torch.cat(rows, dim=-2)
|
||||
|
||||
for name, child in list(network.named_children()):
|
||||
if isinstance(child, torch.nn.AdaptiveAvgPool2d):
|
||||
setattr(network, name, StaticPool(child.output_size))
|
||||
else:
|
||||
lower_static_pools(child)
|
||||
|
||||
|
||||
def metadata(model: str):
|
||||
recipe = get_recipe(model)
|
||||
if recipe["task"] != "seg" or recipe["source"]["checkout"] != "mmsegmentation":
|
||||
raise ValueError(f"Not an MMSeg semantic segmentation recipe: {model}")
|
||||
data = recipe["preprocess"]["official_data_preprocessor"]
|
||||
_, _, h, w = map(int, recipe["input_spec"].split("x"))
|
||||
checkpoint = dict(recipe["checkpoint"], classes={"ade20k": 150, "cityscapes": 19}[recipe["dataset"]])
|
||||
spec = {"source": "custom", "dataset": recipe["dataset"], "input_spec": recipe["input_spec"]}
|
||||
preprocess = {"recipe": "mmseg-static-resize-v1", "input_spec": recipe["input_spec"],
|
||||
"dataset": recipe["dataset"], "color": "RGB" if data.get("bgr_to_rgb") else "BGR",
|
||||
"mean": data["mean"], "std": data["std"], "scale": 1.0,
|
||||
"layout": "NCHW", "dtype": "float32", "resize": [h, w],
|
||||
"interpolation": "opencv_inter_linear", "logits_resize": "bilinear",
|
||||
"align_corners": False, "logits_intermediate_size": [h, w],
|
||||
"protocol": "bench static resize; whole-image logits; original label coordinates"}
|
||||
return checkpoint, spec, preprocess
|
||||
|
||||
|
||||
def preprocess_image(recipe: dict, image, input_spec: str):
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
if input_spec != recipe["input_spec"]:
|
||||
raise ValueError(f"MMSeg input differs from verified recipe: {input_spec}")
|
||||
h, w = recipe["resize"]
|
||||
x = np.asarray(image.convert("RGB"))
|
||||
if x.shape[:2] != (h, w):
|
||||
x = cv2.resize(x, (w, h), interpolation=cv2.INTER_LINEAR)
|
||||
if recipe["color"] == "BGR":
|
||||
x = x[:, :, ::-1]
|
||||
x = (x.astype(np.float32) - np.asarray(recipe["mean"], np.float32)) / np.asarray(recipe["std"], np.float32)
|
||||
return np.ascontiguousarray(x.transpose(2, 0, 1)[None])
|
||||
|
||||
|
||||
def build_model(model: str, weights: Path, input_spec: str):
|
||||
network = official_build(model, weights, input_spec)
|
||||
head = network.upstream.decode_head
|
||||
checkpoint, _, recipe = metadata(model)
|
||||
if head.align_corners != recipe["align_corners"] or head.num_classes != checkpoint["classes"]:
|
||||
raise ValueError("Official head differs from the frozen label/resize recipe")
|
||||
if network.upstream.test_cfg.mode != "whole":
|
||||
raise ValueError("This adapter requires official whole-image inference")
|
||||
if model.startswith(("pspnet.", "upernet.")):
|
||||
lower_static_pools(network.upstream.decode_head)
|
||||
return network
|
||||
17
harness/adapters/ocrnet_hrnet_w18.py
Normal file
17
harness/adapters/ocrnet_hrnet_w18.py
Normal file
@ -0,0 +1,17 @@
|
||||
"""Official ADE20K OCRNet-HRNet18 with both learned cascade heads."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
from .openmmlab import build as official_build
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("ocrnet.hrnet.w18")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
|
||||
|
||||
def build(weights, input_spec):
|
||||
network = official_build("ocrnet.hrnet.w18", weights, input_spec)
|
||||
heads = network.upstream.decode_head
|
||||
if (len(heads) != 2 or any(head.align_corners is not False or
|
||||
head.num_classes != CHECKPOINT["classes"] for head in heads)
|
||||
or network.upstream.test_cfg.mode != "whole"):
|
||||
raise ValueError("Official OCRNet cascade differs from its frozen geometry")
|
||||
return network
|
||||
17
harness/adapters/ocrnet_hrnet_w48.py
Normal file
17
harness/adapters/ocrnet_hrnet_w48.py
Normal file
@ -0,0 +1,17 @@
|
||||
"""Official ADE20K OCRNet-HRNet48 with both learned cascade heads."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
from .openmmlab import build as official_build
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("ocrnet.hrnet.w48")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
|
||||
|
||||
def build(weights, input_spec):
|
||||
network = official_build("ocrnet.hrnet.w48", weights, input_spec)
|
||||
heads = network.upstream.decode_head
|
||||
if (len(heads) != 2 or any(head.align_corners is not False or
|
||||
head.num_classes != CHECKPOINT["classes"] for head in heads)
|
||||
or network.upstream.test_cfg.mode != "whole"):
|
||||
raise ValueError("Official OCRNet cascade differs from its frozen geometry")
|
||||
return network
|
||||
184
harness/adapters/openmmlab.py
Normal file
184
harness/adapters/openmmlab.py
Normal file
@ -0,0 +1,184 @@
|
||||
"""Pinned official OpenMMLab construction; catalog resolution is not admission.
|
||||
|
||||
The existing harness may call get_recipe(model) without importing PyTorch or
|
||||
OpenMMLab. build() runs only in an explicitly provisioned isolated environment.
|
||||
It never downloads, installs, substitutes architectures, or changes the registry.
|
||||
The returned module consumes ALREADY PREPROCESSED float NCHW tensors and returns
|
||||
upstream raw head tensors. Official geometry/decoding and board evaluation remain
|
||||
separate gates; a successful build/export does not satisfy them.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib
|
||||
import importlib.util
|
||||
from functools import lru_cache
|
||||
from copy import deepcopy
|
||||
import yaml
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
|
||||
_RECIPES = Path(__file__).with_name("openmmlab_recipes.yaml")
|
||||
_PACKAGES = {"mmdetection": "mmdet", "mmsegmentation": "mmseg", "mmpose": "mmpose"}
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _catalog() -> dict:
|
||||
return yaml.safe_load(_RECIPES.read_text())["recipes"]
|
||||
|
||||
|
||||
def get_recipe(model: str) -> dict:
|
||||
"""Return a detached recipe from the once-loaded catalog."""
|
||||
return deepcopy(_catalog()[model])
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
h = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1 << 20), b""):
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def _source(recipe: dict, source_root: Path | None = None) -> tuple[Path, str]:
|
||||
source = recipe["source"]
|
||||
checkout = source["checkout"]
|
||||
if checkout not in _PACKAGES:
|
||||
raise RuntimeError(f"{checkout} requires its separately recorded legacy environment/adapter")
|
||||
package = _PACKAGES[checkout]
|
||||
# Use the actual imported official package, or the caller's pinned checkout.
|
||||
spec = importlib.util.find_spec(package)
|
||||
if spec is None or spec.origin is None:
|
||||
raise RuntimeError(f"Missing {package}; provision the isolated recipe environment first")
|
||||
installed_root = Path(spec.origin).resolve().parent.parent
|
||||
root = Path(source_root).resolve() if source_root else installed_root
|
||||
if installed_root != root:
|
||||
raise RuntimeError(f"Imported {package} comes from {installed_root}, not requested {root}")
|
||||
revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=root, text=True).strip()
|
||||
dirty = subprocess.check_output(
|
||||
["git", "status", "--porcelain", "--untracked-files=no"], cwd=root, text=True
|
||||
).strip()
|
||||
if revision != source["revision"] or dirty:
|
||||
raise RuntimeError(f"Unverified {package} checkout: expected clean {source['revision']}; got {revision}")
|
||||
config = root / source["config"]
|
||||
if _sha256(config) != source["config_sha256"]:
|
||||
raise RuntimeError(f"Config hash mismatch: {config}")
|
||||
return config, package
|
||||
|
||||
|
||||
def verify_checkpoint(recipe: dict, weights: Path) -> str:
|
||||
"""Require the full digest recorded from an official-download receipt.
|
||||
|
||||
Filename tokens are provenance only: some official MMPose files retain
|
||||
an old token after checkpoint conversion and do not match their SHA256.
|
||||
"""
|
||||
weights = Path(weights)
|
||||
if not weights.is_file():
|
||||
raise FileNotFoundError(f"Official checkpoint required: {recipe['checkpoint']['url']}")
|
||||
sha = _sha256(weights)
|
||||
info = recipe["checkpoint"]
|
||||
expected = info.get("sha256")
|
||||
if not expected or len(expected) != 64:
|
||||
raise RuntimeError("Recipe lacks a verifiable checkpoint digest; record the official download hash first")
|
||||
if sha != expected:
|
||||
raise ValueError(f"Checkpoint identity mismatch: expected {expected}, got {sha}")
|
||||
return sha
|
||||
|
||||
|
||||
def load_config(model: str, *, source_root: Path | None = None):
|
||||
"""Parse official config and scoped bases from the verified source checkout.
|
||||
|
||||
Editable installs can lack the wheel-only .mim resources. Keep MMEngine's
|
||||
parser and scope handling; resolve same-package files in the pinned repo.
|
||||
"""
|
||||
config_path, package = _source(get_recipe(model), source_root)
|
||||
from mmengine.config import Config
|
||||
from unittest.mock import patch
|
||||
|
||||
config_root = Path(importlib.util.find_spec(package).origin).resolve().parents[1] / "configs"
|
||||
original = Config._get_cfg_path
|
||||
|
||||
def resolve(base, filename):
|
||||
if base.startswith(f"{package}::"):
|
||||
candidate = (config_root / base.split("::", 1)[1]).resolve()
|
||||
if candidate.is_relative_to(config_root) and candidate.is_file():
|
||||
return str(candidate), package
|
||||
return original(base, filename)
|
||||
|
||||
with patch.object(Config, "_get_cfg_path", staticmethod(resolve)):
|
||||
return Config.fromfile(str(config_path))
|
||||
|
||||
|
||||
def build(model: str, weights: Path, input_spec: str, *, source_root: Path | None = None):
|
||||
"""Construct the exact official model and strictly load its complete weights.
|
||||
|
||||
No tensor/key remapping or non-strict fallback is provided. Upstream load
|
||||
hooks handle their own historical format migrations; MMEngine's documented
|
||||
DDP ``module.`` prefix removal is retained. Evidence records strict=True.
|
||||
Input normalization, affine geometry and output decoding are NOT implicit.
|
||||
"""
|
||||
recipe = get_recipe(model)
|
||||
if recipe["resolution_status"] != "resolved_catalog":
|
||||
raise ValueError(f"Unresolved recipe {model}: {recipe['blockers']}")
|
||||
sha = verify_checkpoint(recipe, weights)
|
||||
shape = tuple(int(v) for v in input_spec.split("x"))
|
||||
if len(shape) != 4 or shape[0:2] != (1, 3) or min(shape) <= 0:
|
||||
raise ValueError(f"Expected static batch-one NCHW RGB/BGR input, got {input_spec}")
|
||||
expected = recipe.get("input_spec")
|
||||
if not expected:
|
||||
raise RuntimeError(f"{model}: static probe input contract has not been resolved")
|
||||
if input_spec != expected:
|
||||
raise ValueError(f"{model}: recipe input {expected}, got {input_spec}; no silent shape substitution")
|
||||
|
||||
import torch
|
||||
from mmengine.runner import load_checkpoint
|
||||
from mmengine.registry import init_default_scope
|
||||
|
||||
cfg = load_config(model, source_root=source_root)
|
||||
package = _PACKAGES[recipe["source"]["checkout"]]
|
||||
utils = importlib.import_module(f"{package}.utils")
|
||||
utils.register_all_modules(init_default_scope=False)
|
||||
init_default_scope(package)
|
||||
registry = importlib.import_module(f"{package}.registry")
|
||||
network = registry.MODELS.build(cfg.model)
|
||||
# MMEngine now marks normalization buffers non-persistent; older official
|
||||
# checkpoints saved them. Preserve every key and strictly compare its value
|
||||
# with this recipe before restoring persistence. Never drop learned keys.
|
||||
state = torch.load(str(weights), map_location="cpu")["state_dict"]
|
||||
preprocessor = network.data_preprocessor
|
||||
for name in ("mean", "std"):
|
||||
key = f"data_preprocessor.{name}"
|
||||
# MMPose's official migration hook removes legacy non-learned norm keys.
|
||||
if package != "mmpose" and key in state and name in preprocessor._non_persistent_buffers_set:
|
||||
expected_buffer = getattr(preprocessor, name)
|
||||
if not torch.equal(state[key], expected_buffer):
|
||||
raise ValueError(f"Checkpoint {key} disagrees with the official config")
|
||||
preprocessor.register_buffer(name, expected_buffer, persistent=True)
|
||||
del state
|
||||
# Do not call init_weights(): the complete trained checkpoint is mandatory.
|
||||
load_checkpoint(network, str(weights), map_location="cpu", strict=True)
|
||||
network.cpu().eval()
|
||||
|
||||
class RawTensorModel(torch.nn.Module):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.upstream = network
|
||||
self.openmmlab_recipe = recipe
|
||||
self.checkpoint_sha256 = sha
|
||||
self.cfg = cfg
|
||||
|
||||
def forward(self, x):
|
||||
# Two-stage detector/transformer heads require image metadata even
|
||||
# in tensor mode. It describes this already-preprocessed tensor,
|
||||
# NOT a claim about original-image resize or coordinate recovery.
|
||||
samples = None
|
||||
if package == "mmdet":
|
||||
from mmdet.structures import DetDataSample
|
||||
samples = [DetDataSample(metainfo=dict(
|
||||
img_shape=shape[-2:], ori_shape=shape[-2:],
|
||||
pad_shape=shape[-2:], batch_input_shape=shape[-2:],
|
||||
scale_factor=(1., 1.)
|
||||
))]
|
||||
return self.upstream(x, samples, mode="tensor")
|
||||
|
||||
return RawTensorModel().eval()
|
||||
5097
harness/adapters/openmmlab_recipes.yaml
Normal file
5097
harness/adapters/openmmlab_recipes.yaml
Normal file
File diff suppressed because it is too large
Load Diff
17
harness/adapters/pidnet_l.py
Normal file
17
harness/adapters/pidnet_l.py
Normal file
@ -0,0 +1,17 @@
|
||||
"""Official Cityscapes PIDNet-L preserving its align_corners=True geometry."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
from .openmmlab import build as official_build
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("pidnet.l")
|
||||
PREPROCESS["align_corners"] = True
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
|
||||
|
||||
def build(weights, input_spec):
|
||||
network = official_build("pidnet.l", weights, input_spec)
|
||||
head = network.upstream.decode_head
|
||||
if (head.align_corners is not True or head.num_classes != CHECKPOINT["classes"]
|
||||
or network.upstream.test_cfg.mode != "whole"):
|
||||
raise ValueError("Official PIDNet-L head differs from its frozen geometry")
|
||||
return network
|
||||
17
harness/adapters/pidnet_m.py
Normal file
17
harness/adapters/pidnet_m.py
Normal file
@ -0,0 +1,17 @@
|
||||
"""Official Cityscapes PIDNet-M preserving its align_corners=True geometry."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
from .openmmlab import build as official_build
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("pidnet.m")
|
||||
PREPROCESS["align_corners"] = True
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
|
||||
|
||||
def build(weights, input_spec):
|
||||
network = official_build("pidnet.m", weights, input_spec)
|
||||
head = network.upstream.decode_head
|
||||
if (head.align_corners is not True or head.num_classes != CHECKPOINT["classes"]
|
||||
or network.upstream.test_cfg.mode != "whole"):
|
||||
raise ValueError("Official PIDNet-M head differs from its frozen geometry")
|
||||
return network
|
||||
17
harness/adapters/pidnet_s.py
Normal file
17
harness/adapters/pidnet_s.py
Normal file
@ -0,0 +1,17 @@
|
||||
"""Official Cityscapes PIDNet-S preserving its align_corners=True geometry."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
from .openmmlab import build as official_build
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("pidnet.s")
|
||||
PREPROCESS["align_corners"] = True
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
|
||||
|
||||
def build(weights, input_spec):
|
||||
network = official_build("pidnet.s", weights, input_spec)
|
||||
head = network.upstream.decode_head
|
||||
if (head.align_corners is not True or head.num_classes != CHECKPOINT["classes"]
|
||||
or network.upstream.test_cfg.mode != "whole"):
|
||||
raise ValueError("Official PIDNet-S head differs from its frozen geometry")
|
||||
return network
|
||||
7
harness/adapters/pspnet_resnet101.py
Normal file
7
harness/adapters/pspnet_resnet101.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""Official ADE20K pspnet.resnet101; admission requires real validation."""
|
||||
from functools import partial
|
||||
from .mmseg_common import metadata, preprocess_image, build_model, RUNTIME_PYTHON, EXPORTER_VERSION
|
||||
|
||||
CHECKPOINT, MODEL_SPEC, PREPROCESS = metadata("pspnet.resnet101")
|
||||
preprocess = partial(preprocess_image, PREPROCESS)
|
||||
build = partial(build_model, "pspnet.resnet101")
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user