droboticflow/docs/wjzh-operator-regression.md
2026-06-23 06:38:41 +08:00

356 lines
16 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# WJZH 真实数据算子回归
这个回归入口面向真实生产环境,不再使用临时 shell 适配脚本里的明文 `AK/SK`,而是优先复用系统里的 BOS `storage connection`。
入口脚本:
```bash
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_ID=<storage-connection-id> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='[{"id":"clarity-gate","field":"quality.clarity","operator":"gte","value":0.3,"samplePercent":100}]' \
python3 scripts/wjzh_operator_regression.py --preflight-only
```
## 回归内容
脚本会创建 1 个回归项目,并依次执行完整真实数据回归工作流:
1. `00-源数据目录推理`
2. `01-索引校验双上传`
3. `02-JSON工具链多分支`
4. `03-人工审核通过`
5. `04-人工审核拒绝`
6. `05-Mongo写入`
7. `06-中大型视频切片`
8. `07-超大视频全链路`
9. `08-多目录批量窗口化`
10. `09-Robotics真实概览`
11. `10-多目录多视频批量质检`
12. `11-多目录多视频批量全链路`
13. `12-增量嗅探循环处理`
数据源必须显式提供:
```text
SOURCE_PATH=bos://<bucket>/<input-prefix>
```
输出前缀必须显式提供:
```text
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix>
```
脚本不再内置任何真实 bucket、路径、连接名、视频文件名、设备号、日期范围、Robotics 数据路径、增量目录元数据、质检阈值、视频处理时长或抽检规则。执行者必须按目标环境传入 `SOURCE_PATH`、`UPLOAD_ROOT_PREFIX`、`STORAGE_CONNECTION_ID` 或 `STORAGE_CONNECTION_NAME`、`MEDIUM_VIDEO`、`HUGE_VIDEO`、`ROUTE_DEVICE_ID`、`FILTER_DATE_START`、`FILTER_DATE_END`、`LEROBOT_SOURCE_PATH`、`ROSBAG_SOURCE_PATH`、`INCREMENTAL_PLACE`、`INCREMENTAL_CAMERA_MODEL`、`INCREMENTAL_VIDEO_SUBDIR`、`QUALITY_MIN_CLARITY`、`QUALITY_MAX_SHAKE`、`QUALITY_MAX_HAND_OCCLUSION`、`HUGE_WINDOW_SECONDS`、`MEDIUM_WINDOW_SECONDS`、`MEDIUM_SEGMENT_DURATION_SECONDS`、`HUGE_SEGMENT_DURATION_SECONDS`、`BATCH_WINDOW_SECONDS`、`BATCH_QUALITY_WINDOW_SECONDS`、`BATCH_SEGMENT_DURATION_SECONDS`、`INCREMENTAL_WINDOW_SECONDS`、`INCREMENTAL_SEGMENT_DURATION_SECONDS`、`ANNOTATION_FRAME_INTERVAL_SECONDS`、`SAMPLING_RULES_JSON`。`AUDIT_VIDEO` 可单独指定;未指定时复用 `MEDIUM_VIDEO`。受保护部署还需要 `DROBOTICFLOW_API_TOKEN`,或 `DROBOTICFLOW_ACCEPTANCE_EMAIL` / `DROBOTICFLOW_ACCEPTANCE_PASSWORD` 组织管理员登录凭据。
`SAMPLING_RULES_JSON` 必须是非空 JSON 数组。每条规则必须包含 `id`、`field`、`operator`、`value`、`samplePercent`,且 `samplePercent` 必须是有限数字,取值范围是 `(0, 100]`;布尔值不会被当作 `1` / `0` 接受。`operator` 支持 `eq`、`ne`、`gt`、`gte`、`lt`、`lte`、`contains`、`in`、`exists`、`missing`。这些规则会写入 `07-超大视频全链路`、`11-多目录多视频批量全链路`、`12-增量嗅探循环处理` 以及多项目入口的 `11B-人工抽检采样`,脚本不会替调用方选择默认阈值。
`QUALITY_MIN_CLARITY`、`QUALITY_MAX_SHAKE`、`QUALITY_MAX_HAND_OCCLUSION` 必须是 `[0, 1]` 范围内的有限归一化阈值,会写入所有 `视频质检` 节点以及带日期过滤的 `数据过滤` 节点。调用方需要根据目标真实数据集选择阈值;脚本不会使用 `0.0/1.0/1.0` 这类宽松默认值作为验收标准,也不会接受超出归一化指标范围或 `NaN/Inf` 这类非有限值作为生产验收证据。
窗口、切片和标注 timing 配置必须显式提供。`HUGE_WINDOW_SECONDS`、`MEDIUM_WINDOW_SECONDS`、`MEDIUM_SEGMENT_DURATION_SECONDS`、`HUGE_SEGMENT_DURATION_SECONDS`、`BATCH_WINDOW_SECONDS`、`BATCH_QUALITY_WINDOW_SECONDS`、`BATCH_SEGMENT_DURATION_SECONDS`、`INCREMENTAL_WINDOW_SECONDS`、`INCREMENTAL_SEGMENT_DURATION_SECONDS` 必须是 exact safe positive integer,拒绝符号、前导零、小数、单位和 unsafe integer;`ANNOTATION_FRAME_INTERVAL_SECONDS` 必须是大于 0 的有限数字。这些值会写入 `Shell 命令执行` 的窗口采样命令、`视频切片` 的 `SEGMENT_DURATION_SECONDS` 和 `视频标注` 的 `FRAME_INTERVAL_SECONDS`。
## 生产前置
必须满足下面 4 项,否则脚本会在 preflight 或 runtime acceptance 阶段直接失败:
1. API / worker 已部署为当前仓库版本
关键标志:`/api/settings` 返回里应包含 `k8s.workspacePVCName`。
2. 共享工作区 PVC 已配置
至少要有:
- `WORKSPACE_PVC_NAME`
- `WORKSPACE_MOUNT_PATH`
3. 已存在可用的 BOS storage connection
连接需要指向 `SOURCE_PATH` 和 `UPLOAD_ROOT_PREFIX` 使用的 bucket。若两个路径跨 bucket,需要准备能访问两个 bucket 的连接,或拆分为相同 bucket 下的输入输出前缀。
4. Robotics 数据路径对 operator-runner 可见
`LEROBOT_SOURCE_PATH` 和 `ROSBAG_SOURCE_PATH` 必须是 Kubernetes Job 容器内可读取的真实路径,例如共享 PVC 下的 `/workspace/datasets/...`。
## 推荐执行顺序
先做纯配置预检:
```bash
API_BASE_URL=http://<api-host>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_ID=<storage-connection-id> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_regression.py --preflight-only
```
预检通过后,再做带运行时烟测的正式回归:
```bash
API_BASE_URL=http://<api-host>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_ID=<storage-connection-id> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_regression.py --delete-regression-projects
```
如果确实要先清空所有项目,再重建回归项目,显式加:
```bash
--delete-all-projects
```
这个选项是破坏性的,默认不会启用。
## 连接配置方式
### 方式 1:使用已有 storage connection
推荐:
```bash
API_BASE_URL=http://<api-host>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_ID=<storage-connection-id> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_regression.py --preflight-only
```
也可以按名称查找:
```bash
API_BASE_URL=http://<api-host>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_NAME=<storage-connection-name> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_regression.py --preflight-only
```
### 方式 2:由脚本创建 storage connection
仅当环境里还没有现成连接时使用。`BOS_ENDPOINT` 可留空,由 API 运行时默认值解析;如果需要固定特定地域端点,再额外设置 `BOS_ENDPOINT=https://<region>.bcebos.com`。
```bash
API_BASE_URL=http://<api-host>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_NAME=<storage-connection-name> \
CREATE_CONNECTION_IF_MISSING=1 \
BOS_BUCKET=<bucket> \
BOS_ACCESS_KEY_ID=<ak> \
BOS_SECRET_ACCESS_KEY=<sk> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_regression.py --preflight-only
```
## 多项目回归入口
`scripts/wjzh_operator_multi_project_regression.py` 会把基类回归拆到多个项目中执行,并额外覆盖 `真实数据源` 的 `SOURCE_TEXT` / `SOURCE_JSON` 两个入口。两个入口不再使用仓库内置样本,必须由调用方提供真实 payload:
```bash
WJZH_REGRESSION_SOURCE_TEXT_FILE=/path/to/real-source-text.txt \
WJZH_REGRESSION_SOURCE_JSON_FILE=/path/to/real-source-payload.json \
API_BASE_URL=http://<api-host>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_ID=<storage-connection-id> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_multi_project_regression.py --preflight-only
```
如果不想把 payload 放在文件中,也可以改用 `WJZH_REGRESSION_SOURCE_TEXT` 和 `WJZH_REGRESSION_SOURCE_JSON`。同一个 payload 只能选择环境变量或文件路径其中一种方式。
## 鉴权配置
- `DROBOTICFLOW_API_TOKEN`:用于未强制 session auth 的 token-protected API。
- `DROBOTICFLOW_ACCEPTANCE_EMAIL` / `DROBOTICFLOW_ACCEPTANCE_PASSWORD`:用于公开 session-auth 环境。脚本会登录、保存 cookie,并在写请求中发送 CSRF header。
- 如果 API 同时要求共享 token 和 session 登录,可以同时设置两组变量。
## 脚本做了什么
1. 校验 `/api/settings` 是否满足当前版本的数据面要求。
2. 校验并解析 BOS storage connection。
3. 校验回归所需内置算子是否齐全。
4. 先创建一条最小 `connectionId` 烟测工作流:
`BOS 增量扫描 -> X 平台数据集索引 -> 采集目录校验 -> 结果上传`
5. 只有烟测通过后,才会继续创建完整真实数据回归工作流。
## 如果脚本失败
重点看两类错误:
1. `settings.k8s.workspacePVCName is missing`
说明线上还没部署到当前仓库这版,或者共享 PVC 没配好。
2. `runtime-acceptance-failed`
说明 API 里虽然已经存了 `connectionId`,但 worker / operator-runner 在真实运行时没有正确解析 BOS 连接,通常是线上部署版本落后于当前代码。
## 验收标准
1. 运行时烟测工作流成功。
2. 正式回归工作流全部 `completed`。
3. 工作流定义里不再保存明文 `BOS_ACCESS_KEY_ID` / `BOS_SECRET_ACCESS_KEY`。
4. 结果写入 `UPLOAD_ROOT_PREFIX` 下独立时间戳前缀。
5. 所有抽检节点的 `RULES_JSON` 与调用方传入的 `SAMPLING_RULES_JSON` 一致。