67 lines
3.8 KiB
Markdown
67 lines
3.8 KiB
Markdown
# Reinforcement Learning
|
||
|
||
MuJoCo双轴起身环境、速度动作搜索、教师数据、BC/PPO训练与可复现评测。**主权重是诊断用BC43,尚未通过实机验收。**
|
||
|
||
## 文件与入口
|
||
|
||
| 文件/目录 | 用途 |
|
||
|---|---|
|
||
| [actor.npz](actor.npz)、[policy.py](policy.py) | 主BC43与独立NumPy推理,196维输入、2维速度请求 |
|
||
| [infer.py](infer.py)、[examples/](examples/) | 保存的可测观测做一次推理,不连接硬件 |
|
||
| [training/continuous_sg90_v3_20261113/stiction_domain/](training/continuous_sg90_v3_20261113/stiction_domain/) | 可编辑物理域、搜索、训练、评估和数据 |
|
||
| [training/continuous_sg90_v3_20261113/calibration/](training/continuous_sg90_v3_20261113/calibration/) | 实机轨迹提取、参数拟合与留出预测 |
|
||
| [reproduction/](reproduction/) | 完整封存权重、测试轨迹与教师数据,大JSON无损压缩 |
|
||
| [delivery/](delivery/) | 早期定位/速度策略的审阅快照 |
|
||
|
||
## 安装与推理
|
||
|
||
从仓库根目录运行:
|
||
|
||
```sh
|
||
python3.10 -m venv .venv
|
||
. .venv/bin/activate
|
||
python -m pip install -r requirements.txt
|
||
python -B RL/infer.py --observation RL/examples/first_observation.json
|
||
```
|
||
|
||
输入包含初始朝上向量、16帧IMU/加速度/实际命令/时间历史及经过时间,不包含真实关节角、接触、seed或方向标签;输出是归一化速度,不是目标角。
|
||
|
||
## 回放
|
||
|
||
```sh
|
||
python3 scripts/materialize_reproduction.py --out build/v3
|
||
python3 -B build/v3/verify_package.py
|
||
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 python -B build/v3/run_demo.py \
|
||
--sample build/v3/samples/primary_success.json --out build/success_replay
|
||
```
|
||
|
||
精确复现要求Python3.10.20与记录依赖。[完整复现步骤](../docs/REPRODUCING.md)包含失败样本、轨迹/RNG核对和平台限制。[已有8段视频](reproduction/v3/videos/README.md)可直接查看。
|
||
|
||
## 训练与评估
|
||
|
||
按“搜索→教师→训练→评测→导出”阅读,动作均为连续速度请求:
|
||
|
||
| 步骤 | 入口(v3训练目录内) | 输出/约束 |
|
||
|---|---|---|
|
||
| 仿真对齐 | calibration/extract.py、fit.py、stiction_diagnostic/ | 参数族与误差;当前校准未通过 |
|
||
| 搜索双轴动作 | stiction_domain/search.py、multidir_transfer.py | 上轴/下轴先动、同步、反转换支点等候选 |
|
||
| 生成教师 | search/snapshot_crossq_43episodes_v1/、datasets/multidir50/ | 成功见证、obs/action及物理seed分组 |
|
||
| BC及PPO | stiction_domain/train_crossq.py、train_multidir.py | 检查点与actor.npz,显式诊断模型族 |
|
||
| 冻结测试 | stiction_domain/evaluate.py | 同起点每方向成功/失败/排除和配对结果 |
|
||
| 导出和X5对齐 | train模块的export、[RDK](../RDK/README.md) | Torch/NumPy误差、文件身份、无PWM推理 |
|
||
|
||
```sh
|
||
python -m pip install -r requirements-training.txt
|
||
python -B RL/training/continuous_sg90_v3_20261113/stiction_domain/search.py --help
|
||
python -B RL/training/continuous_sg90_v3_20261113/stiction_domain/train_multidir.py --help
|
||
python -B RL/training/continuous_sg90_v3_20261113/stiction_domain/evaluate.py --help
|
||
```
|
||
|
||
原生加速器依赖平台,重编译后需新环境身份。accepted_models仍BLOCKED,不得跳过校准门或回退nominal。历史脚本名含crossq,实际导入SB3 PPO;主BC43为0个PPO环境步的监督导出。[训练限制和数据](../docs/REPRODUCING.md#3-训练代码与数据入口)
|
||
|
||
## 当前结果与下一步
|
||
|
||
主BC43严格仿真55/963;+Y为16/200、+X+Y为15/109。斜向配额不足,不能算充分全向验收。旧位置策略和早期速度策略条件不同,不能直接比较。[逐方向结果](../docs/RESULTS.md)
|
||
|
||
先处理[实机响应差异、新几何、杆位变化和限幅](../docs/OPTIMIZATION_HANDOFF.md),再衡量训练增益。教师成功、网络闭环成功与实机成功分别报告。
|