droboticflow-new/docs/wjzh-operator-regression.md
2026-06-23 06:38:41 +08:00

16 KiB
Raw Permalink Blame History

WJZH 真实数据算子回归

这个回归入口面向真实生产环境,不再使用临时 shell 适配脚本里的明文 AK/SK,而是优先复用系统里的 BOS storage connection。

入口脚本:

SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_ID=<storage-connection-id> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='[{"id":"clarity-gate","field":"quality.clarity","operator":"gte","value":0.3,"samplePercent":100}]' \
python3 scripts/wjzh_operator_regression.py --preflight-only

回归内容

脚本会创建 1 个回归项目,并依次执行完整真实数据回归工作流:

  1. 00-源数据目录推理
  2. 01-索引校验双上传
  3. 02-JSON工具链多分支
  4. 03-人工审核通过
  5. 04-人工审核拒绝
  6. 05-Mongo写入
  7. 06-中大型视频切片
  8. 07-超大视频全链路
  9. 08-多目录批量窗口化
  10. 09-Robotics真实概览
  11. 10-多目录多视频批量质检
  12. 11-多目录多视频批量全链路
  13. 12-增量嗅探循环处理

数据源必须显式提供:

SOURCE_PATH=bos://<bucket>/<input-prefix>

输出前缀必须显式提供:

UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix>

脚本不再内置任何真实 bucket、路径、连接名、视频文件名、设备号、日期范围、Robotics 数据路径、增量目录元数据、质检阈值、视频处理时长或抽检规则。执行者必须按目标环境传入 SOURCE_PATH、UPLOAD_ROOT_PREFIX、STORAGE_CONNECTION_ID 或 STORAGE_CONNECTION_NAME、MEDIUM_VIDEO、HUGE_VIDEO、ROUTE_DEVICE_ID、FILTER_DATE_START、FILTER_DATE_END、LEROBOT_SOURCE_PATH、ROSBAG_SOURCE_PATH、INCREMENTAL_PLACE、INCREMENTAL_CAMERA_MODEL、INCREMENTAL_VIDEO_SUBDIR、QUALITY_MIN_CLARITY、QUALITY_MAX_SHAKE、QUALITY_MAX_HAND_OCCLUSION、HUGE_WINDOW_SECONDS、MEDIUM_WINDOW_SECONDS、MEDIUM_SEGMENT_DURATION_SECONDS、HUGE_SEGMENT_DURATION_SECONDS、BATCH_WINDOW_SECONDS、BATCH_QUALITY_WINDOW_SECONDS、BATCH_SEGMENT_DURATION_SECONDS、INCREMENTAL_WINDOW_SECONDS、INCREMENTAL_SEGMENT_DURATION_SECONDS、ANNOTATION_FRAME_INTERVAL_SECONDS、SAMPLING_RULES_JSON。AUDIT_VIDEO 可单独指定;未指定时复用 MEDIUM_VIDEO。受保护部署还需要 DROBOTICFLOW_API_TOKEN,或 DROBOTICFLOW_ACCEPTANCE_EMAIL / DROBOTICFLOW_ACCEPTANCE_PASSWORD 组织管理员登录凭据。

SAMPLING_RULES_JSON 必须是非空 JSON 数组。每条规则必须包含 id、field、operator、value、samplePercent,且 samplePercent 必须是有限数字,取值范围是 (0, 100];布尔值不会被当作 1 / 0 接受。operator 支持 eq、ne、gt、gte、lt、lte、contains、in、exists、missing。这些规则会写入 07-超大视频全链路、11-多目录多视频批量全链路、12-增量嗅探循环处理 以及多项目入口的 11B-人工抽检采样,脚本不会替调用方选择默认阈值。

QUALITY_MIN_CLARITY、QUALITY_MAX_SHAKE、QUALITY_MAX_HAND_OCCLUSION 必须是 [0, 1] 范围内的有限归一化阈值,会写入所有 视频质检 节点以及带日期过滤的 数据过滤 节点。调用方需要根据目标真实数据集选择阈值;脚本不会使用 0.0/1.0/1.0 这类宽松默认值作为验收标准,也不会接受超出归一化指标范围或 NaN/Inf 这类非有限值作为生产验收证据。

窗口、切片和标注 timing 配置必须显式提供。HUGE_WINDOW_SECONDS、MEDIUM_WINDOW_SECONDS、MEDIUM_SEGMENT_DURATION_SECONDS、HUGE_SEGMENT_DURATION_SECONDS、BATCH_WINDOW_SECONDS、BATCH_QUALITY_WINDOW_SECONDS、BATCH_SEGMENT_DURATION_SECONDS、INCREMENTAL_WINDOW_SECONDS、INCREMENTAL_SEGMENT_DURATION_SECONDS 必须是 exact safe positive integer,拒绝符号、前导零、小数、单位和 unsafe integer;ANNOTATION_FRAME_INTERVAL_SECONDS 必须是大于 0 的有限数字。这些值会写入 Shell 命令执行 的窗口采样命令、视频切片 的 SEGMENT_DURATION_SECONDS 和 视频标注 的 FRAME_INTERVAL_SECONDS。

生产前置

必须满足下面 4 项,否则脚本会在 preflight 或 runtime acceptance 阶段直接失败:

  1. API / worker 已部署为当前仓库版本 关键标志:/api/settings 返回里应包含 k8s.workspacePVCName。

  2. 共享工作区 PVC 已配置 至少要有:

    • WORKSPACE_PVC_NAME
    • WORKSPACE_MOUNT_PATH
  3. 已存在可用的 BOS storage connection 连接需要指向 SOURCE_PATH 和 UPLOAD_ROOT_PREFIX 使用的 bucket。若两个路径跨 bucket,需要准备能访问两个 bucket 的连接,或拆分为相同 bucket 下的输入输出前缀。

  4. Robotics 数据路径对 operator-runner 可见 LEROBOT_SOURCE_PATH 和 ROSBAG_SOURCE_PATH 必须是 Kubernetes Job 容器内可读取的真实路径,例如共享 PVC 下的 /workspace/datasets/...。

推荐执行顺序

先做纯配置预检:

API_BASE_URL=http://<api-host>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_ID=<storage-connection-id> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_regression.py --preflight-only

预检通过后,再做带运行时烟测的正式回归:

API_BASE_URL=http://<api-host>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_ID=<storage-connection-id> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_regression.py --delete-regression-projects

如果确实要先清空所有项目,再重建回归项目,显式加:

--delete-all-projects

这个选项是破坏性的,默认不会启用。

连接配置方式

方式 1:使用已有 storage connection

推荐:

API_BASE_URL=http://<api-host>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_ID=<storage-connection-id> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_regression.py --preflight-only

也可以按名称查找:

API_BASE_URL=http://<api-host>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_NAME=<storage-connection-name> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_regression.py --preflight-only

方式 2:由脚本创建 storage connection

仅当环境里还没有现成连接时使用。BOS_ENDPOINT 可留空,由 API 运行时默认值解析;如果需要固定特定地域端点,再额外设置 BOS_ENDPOINT=https://<region>.bcebos.com。

API_BASE_URL=http://<api-host>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_NAME=<storage-connection-name> \
CREATE_CONNECTION_IF_MISSING=1 \
BOS_BUCKET=<bucket> \
BOS_ACCESS_KEY_ID=<ak> \
BOS_SECRET_ACCESS_KEY=<sk> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_regression.py --preflight-only

多项目回归入口

scripts/wjzh_operator_multi_project_regression.py 会把基类回归拆到多个项目中执行,并额外覆盖 真实数据源 的 SOURCE_TEXT / SOURCE_JSON 两个入口。两个入口不再使用仓库内置样本,必须由调用方提供真实 payload:

WJZH_REGRESSION_SOURCE_TEXT_FILE=/path/to/real-source-text.txt \
WJZH_REGRESSION_SOURCE_JSON_FILE=/path/to/real-source-payload.json \
API_BASE_URL=http://<api-host>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_ID=<storage-connection-id> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_multi_project_regression.py --preflight-only

如果不想把 payload 放在文件中,也可以改用 WJZH_REGRESSION_SOURCE_TEXT 和 WJZH_REGRESSION_SOURCE_JSON。同一个 payload 只能选择环境变量或文件路径其中一种方式。

鉴权配置

  • DROBOTICFLOW_API_TOKEN:用于未强制 session auth 的 token-protected API。
  • DROBOTICFLOW_ACCEPTANCE_EMAIL / DROBOTICFLOW_ACCEPTANCE_PASSWORD:用于公开 session-auth 环境。脚本会登录、保存 cookie,并在写请求中发送 CSRF header。
  • 如果 API 同时要求共享 token 和 session 登录,可以同时设置两组变量。

脚本做了什么

  1. 校验 /api/settings 是否满足当前版本的数据面要求。
  2. 校验并解析 BOS storage connection。
  3. 校验回归所需内置算子是否齐全。
  4. 先创建一条最小 connectionId 烟测工作流: BOS 增量扫描 -> X 平台数据集索引 -> 采集目录校验 -> 结果上传
  5. 只有烟测通过后,才会继续创建完整真实数据回归工作流。

如果脚本失败

重点看两类错误:

  1. settings.k8s.workspacePVCName is missing 说明线上还没部署到当前仓库这版,或者共享 PVC 没配好。

  2. runtime-acceptance-failed 说明 API 里虽然已经存了 connectionId,但 worker / operator-runner 在真实运行时没有正确解析 BOS 连接,通常是线上部署版本落后于当前代码。

验收标准

  1. 运行时烟测工作流成功。
  2. 正式回归工作流全部 completed。
  3. 工作流定义里不再保存明文 BOS_ACCESS_KEY_ID / BOS_SECRET_ACCESS_KEY。
  4. 结果写入 UPLOAD_ROOT_PREFIX 下独立时间戳前缀。
  5. 所有抽检节点的 RULES_JSON 与调用方传入的 SAMPLING_RULES_JSON 一致。