SelfRising/RL/README.md

67 lines
3.8 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Reinforcement Learning
MuJoCo双轴起身环境、速度动作搜索、教师数据、BC/PPO训练与可复现评测。**主权重是诊断用BC43,尚未通过实机验收。**
## 文件与入口
| 文件/目录 | 用途 |
|---|---|
| [actor.npz](actor.npz)、[policy.py](policy.py) | 主BC43与独立NumPy推理,196维输入、2维速度请求 |
| [infer.py](infer.py)、[examples/](examples/) | 保存的可测观测做一次推理,不连接硬件 |
| [training/continuous_sg90_v3_20261113/stiction_domain/](training/continuous_sg90_v3_20261113/stiction_domain/) | 可编辑物理域、搜索、训练、评估和数据 |
| [training/continuous_sg90_v3_20261113/calibration/](training/continuous_sg90_v3_20261113/calibration/) | 实机轨迹提取、参数拟合与留出预测 |
| [reproduction/](reproduction/) | 完整封存权重、测试轨迹与教师数据,大JSON无损压缩 |
| [delivery/](delivery/) | 早期定位/速度策略的审阅快照 |
## 安装与推理
从仓库根目录运行:
```sh
python3.10 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
python -B RL/infer.py --observation RL/examples/first_observation.json
```
输入包含初始朝上向量、16帧IMU/加速度/实际命令/时间历史及经过时间,不包含真实关节角、接触、seed或方向标签;输出是归一化速度,不是目标角。
## 回放
```sh
python3 scripts/materialize_reproduction.py --out build/v3
python3 -B build/v3/verify_package.py
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 python -B build/v3/run_demo.py \
--sample build/v3/samples/primary_success.json --out build/success_replay
```
精确复现要求Python3.10.20与记录依赖。[完整复现步骤](../docs/REPRODUCING.md)包含失败样本、轨迹/RNG核对和平台限制。[已有8段视频](reproduction/v3/videos/README.md)可直接查看。
## 训练与评估
按“搜索→教师→训练→评测→导出”阅读,动作均为连续速度请求:
| 步骤 | 入口(v3训练目录内) | 输出/约束 |
|---|---|---|
| 仿真对齐 | calibration/extract.py、fit.py、stiction_diagnostic/ | 参数族与误差;当前校准未通过 |
| 搜索双轴动作 | stiction_domain/search.py、multidir_transfer.py | 上轴/下轴先动、同步、反转换支点等候选 |
| 生成教师 | search/snapshot_crossq_43episodes_v1/、datasets/multidir50/ | 成功见证、obs/action及物理seed分组 |
| BC及PPO | stiction_domain/train_crossq.py、train_multidir.py | 检查点与actor.npz,显式诊断模型族 |
| 冻结测试 | stiction_domain/evaluate.py | 同起点每方向成功/失败/排除和配对结果 |
| 导出和X5对齐 | train模块的export、[RDK](../RDK/README.md) | Torch/NumPy误差、文件身份、无PWM推理 |
```sh
python -m pip install -r requirements-training.txt
python -B RL/training/continuous_sg90_v3_20261113/stiction_domain/search.py --help
python -B RL/training/continuous_sg90_v3_20261113/stiction_domain/train_multidir.py --help
python -B RL/training/continuous_sg90_v3_20261113/stiction_domain/evaluate.py --help
```
原生加速器依赖平台,重编译后需新环境身份。accepted_models仍BLOCKED,不得跳过校准门或回退nominal。历史脚本名含crossq,实际导入SB3 PPO;主BC43为0个PPO环境步的监督导出。[训练限制和数据](../docs/REPRODUCING.md#3-训练代码与数据入口)
## 当前结果与下一步
主BC43严格仿真55/963;+Y为16/200、+X+Y为15/109。斜向配额不足,不能算充分全向验收。旧位置策略和早期速度策略条件不同,不能直接比较。[逐方向结果](../docs/RESULTS.md)
先处理[实机响应差异、新几何、杆位变化和限幅](../docs/OPTIMIZATION_HANDOFF.md),再衡量训练增益。教师成功、网络闭环成功与实机成功分别报告。