MotrixSimWeb/README.md

285 lines
12 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# MotrixSim Universal Three.js Preview
This repository now contains a working universal-preview path:
```text
MotrixSim / MJCF scene
-> scene manifest
-> Three.js WebGL renderer
-> replay / polling / websocket state source
```
The first verified version supports primitives, OBJ/STL/GLTF/GLB mesh loading,
Z-up MotrixSim coordinate conversion, body-name state binding, desktop/mobile
responsive layout, MJCF `<include>` expansion, HTTP asset serving, headless
MotrixSim bridge controls, and basic Franka joint/gripper operation.
The current exporter/renderer path also covers higher-variance MJCF assets:
nested `<default>` inheritance for visible-vs-collision geoms, `assetdir` /
`meshdir` / `texturedir` resolution, file textures, builtin checker textures,
mesh scale, `fromto`, `xyaxes`, `zaxis`, inline convex meshes, and browser-side
mesh caching for repeated assets.
## Run Locally
```bash
npm install
npm run dev -- --port 5173
```
Open:
```text
http://127.0.0.1:5173/
```
For the real MotrixSim backend used by the G1 store scene, use the repeatable
dev helper instead of hand-running SSH commands:
```bash
# Print local tunnel and bridge health.
npm run real-backend:status
# Check local Node, SSH access, remote MotrixSim paths/imports, running bridge,
# and manifest grouping before a handoff or demo.
npm run real-backend:doctor
# Sync server/headless_motrix_bridge.py to Wuying, restart the active physical
# scene paused, ensure the 18090 tunnel exists, then run real backend checks.
npm run real-backend:up
# Run the product-readiness gate: real backend doctor, backend edit sync,
# Scene Builder compose / share-link restore / config export-import restore /
# desktop / mobile screenshots, Chinese-default UI, UI-downloaded acceptance
# package/report, external simulation labeling, backend-physics labeling, and
# object physics-tier labeling, plus pause/resume latency checks.
npm run product:acceptance
# Full handoff/release gate: typecheck, production build, gzip bundle budget,
# and production-preview product acceptance.
npm run product:release-check
```
Useful overrides:
```bash
WUYING_HOST=39.106.101.95
WUYING_USER=root
WUYING_SSH_KEY=/path/to/wuying_ed25519
MOTRIXSIM_REMOTE_SCENE=/tmp/motrixsim-web-composed/active_g1_store_scene.xml
MOTRIXSIM_LOCAL_PORT=18090
```
The first screen is now a Scene Builder for the backend-required product path.
The default `G1 + 货架商品后端物理场景` preset uses
`http://127.0.0.1:18090/manifest`; it loads one real MotrixSim MJCF runtime and
binds every visible shelf/product/robot object to `/state`.
When the matching bridge exposes `POST /compose`, Scene Builder object
Add/Remove/Clear operations regenerate the combined MJCF on the backend and
reload the live manifest. Those objects are real MotrixSim bodies with
free joints and collision geoms, not front-end-only Three.js meshes.
For physical store scenes, the bridge also publishes product-level
`editableGroup` metadata derived from real runtime body poses. Store shelf
frames, plates, and pre-stocked products are fixed environment bodies so they
keep their collision shape and do not avalanche when the simulator resumes.
Backend-compatible added props remain real dynamic bodies when the backend
adapter creates them. The current default product scene starts with no added
props; uploaded mesh assets are disabled until a backend adapter can turn them
into real MJCF bodies. Repositioning a fixed shelf layout should be handled by
rebuilding the composed scene, not by keeping the shelf as a free runtime body.
The UI, diagnostics panel, JSON acceptance package, and HTML acceptance report
now expose object physics tiers: backend dynamic bodies, backend fixed bodies,
backend state-only bodies, and visual-only objects. Product acceptance requires
`visual-only=0`.
To load a remote manifest:
```text
http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18083%2Fmanifest
```
## Verify
```bash
npm run build
npm run verify:exporter
npm run verify:preview
```
For a MotrixSim bridge with only primitive objects:
```bash
MIN_OBJECTS=2 MIN_MESHES=0 npm run verify:preview -- \
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18083%2Fmanifest' \
test-results/remote-motrix
```
For a larger robot scene with many mesh files, raise the verification timeout:
```bash
VERIFY_TIMEOUT_MS=45000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:preview -- \
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
test-results/robot-panda
```
For the backend control path:
```bash
VERIFY_TIMEOUT_MS=60000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:control -- \
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
test-results/control-panda
```
For the productized real backend compose and editing path on the G1 store scene:
```bash
npm run verify:real-backend -- \
'http://127.0.0.1:18090/manifest' \
test-results/real-backend-editing
```
This probe calls the live MotrixSim bridge, not a mock. It checks editable
entity groups in `/manifest`, calls real backend `/compose` for a controlled
legacy dynamic-body edit test, verifies the composed object bodies in `/state`,
writes a pose for one composed dynamic body, verifies fixed pre-stocked product
bodies reject `set_body_pose`, resumes the simulator, confirms fixed shelf
anchors do not drift, checks a lifted dynamic prop drops under backend physics,
and measures compact pause/resume control latency. After verification,
`npm run real-backend:up` restores the default stocked shelf scene with no added
test objects.
For handoff or acceptance, prefer the single product gate:
```bash
npm run product:acceptance
```
It writes `test-results/product-readiness/product-readiness-report.json` plus
Scene Builder compose, share-link restore, config import restore, desktop, and mobile screenshots, plus an auditable
`test-results/product-readiness/product-readiness-evidence.json` manifest with
artifact hashes and git provenance. The gate also clicks the in-page export
buttons, saves the downloaded builder config JSON, JSON acceptance package, and
HTML report, parses them, imports the builder config through a real browser file
chooser, opens the downloaded HTML report in Chromium, screenshots it, and
records their hashes. If `127.0.0.1:5173` is not already running,
it starts the Vite dev server for the duration of the check. It does not use
mock scene data: backend checks call the live MotrixSim bridge, and browser
checks load `http://127.0.0.1:18090/manifest` by default. The browser gate also
clicks the Scene Builder `Physical Scene` control in an isolated browser
context to prove the product UI triggers backend `/compose`, then checks the
runtime diagnostics panel for manifest/state/control endpoints, external
simulation mode, object bindings, asset health, and control latency. It also
clicks the product copy-link action, reads the browser clipboard, opens the
share URL in a clean browser context, and verifies that the restored scene uses
the same builder config, backend physics manifest, `/control` endpoint, and
object/body binding counts. It also exports the builder config, imports that
file in a clean browser context, and verifies the same backend physical scene is
restored.
The gate also validates the object physics-tier evidence. The product physical
scene must include backend fixed environment bodies and backend state-only robot
links while keeping `visual-only=0`. The product gate also validates builder
physics coverage: the default stocked shelf path has no added props, and any
uploaded OBJ asset without a backend adapter is blocked instead of being added
as a visual-only object.
For release handoff, `npm run product:release-check` first builds `dist/`,
checks the JavaScript gzip bundle budget, then forces the browser gate through
`vite preview` on an isolated localhost port instead of reusing the development
server. Release checks fail on warnings.
For demo and handoff details, see
[`docs/product-readiness.md`](docs/product-readiness.md). That document records
the real backend topology, acceptance criteria, current product gate baseline,
evidence paths, screenshot hashes, and known product boundaries.
For the Franka joint/gripper control path:
```bash
VERIFY_TIMEOUT_MS=70000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:joint -- \
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
test-results/joint-panda
```
For a large textured store scene on the remote MotrixSim bridge:
```bash
VERIFY_TIMEOUT_MS=120000 MIN_OBJECTS=500 MIN_MESHES=20 npm run verify:preview -- \
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18085%2Fmanifest' \
test-results/store-textured
```
For a Go2 quadruped robot scene:
```bash
VERIFY_TIMEOUT_MS=60000 MIN_OBJECTS=30 MIN_MESHES=20 npm run verify:preview -- \
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18086%2Fmanifest' \
test-results/go2-robot
```
## Headless MotrixSim Bridge
Run this on a Linux server with MotrixSim available:
```bash
python server/headless_motrix_bridge.py \
--scene /path/to/scene.xml \
--host 127.0.0.1 \
--port 18083 \
--hz 120
```
It exposes:
- `GET /manifest`: Three.js preview manifest generated from the XML scene.
- `GET /state`: live MotrixSim body pose state.
- `GET /assets/...`: mesh sidecar assets referenced by the manifest.
- `POST /control`: `pause`, `resume`, `reset`, `step`, `home`,
`set_joint`, `set_body_pose`, `set_body_poses`, and `gripper`.
- `POST /compose`: for supported physical presets, generate a new combined
MJCF scene from selected robot/scene/object choices, hot-swap the runtime,
and return the refreshed manifest.
- `GET /health`: service health.
No display server is required for this state-driven preview path.
## Current Boundary
This is a real Three.js/WebGL preview path, not the earlier Canvas 2.5D page.
It is not yet a perfect renderer for every MotrixSim feature. Unsupported or
private features still need explicit adapters or a server-side render/video
fallback.
Scene Builder product composition must stay connected to a MotrixSim backend
state source. Added objects are accepted only when a backend adapter creates
real MJCF bodies. The current default physical path is the stocked shelf scene on
port `18090`; assets without adapters are visible in the asset library but cannot
be inserted into the product physics scene.
## 页面内上传与组装(已支持)
- 左侧「场景上传 / 上传场景包」支持一次选择 `.json` manifest 和关联的 mesh / texture / bin 文件。manifest 至少需要包含 `version` 与 `objects`。
- 上传后会自动插入到场景下拉列表首位,并把 manifest 内相对路径(例如 `meshes/robot.obj`、`textures/base.png`)匹配到同批上传的本地文件,改写为浏览器可加载的 `blob:` URL。
- 左侧「资产库 / 上传资产」支持 `.gltf/.glb/.obj/.stl`,也可以同时选择 `.bin/.png/.jpg/.webp/.ktx2/.mtl` 等 sidecar 文件;`.gltf` 内部的 buffer / image 相对引用会同步改写。
- 已有后端适配的资产才能加入产品物理场景,并可继续调整位置(X/Y/Z 输入框)和通过左键拖拽编辑(暂停时可拖拽)。
- 未接入后端适配的上传 mesh 会显示“需后端适配”,添加按钮保持禁用,不会作为“仅视觉”物体混入产品验收场景。
- 上传面板支持拖放文件;「清空自定义」会清理上传资产;「清空场景」会移除当前上传场景并回到默认场景列表。
边界:本地上传 mesh 不会自动变成 MotrixSim 后端碰撞体。需要真实碰撞时,必须先通过后端适配把对象转成 MJCF body / geom;适配完成前产品模式不允许仅视觉加入。
The Franka Panda MJCF robot scene has been verified as a real robot asset
preview: 59 manifest objects, 58 loaded OBJ/STL meshes, and zero mesh fallback.
Joint-level operation is now wired for this scene through MotrixSim
`joint_names`, `link_names`, and link pose streaming. Broader task-level robot
actions still need scene-specific policy/action adapters.
Additional remote scenes verified:
- Store shelf scene: 569 objects, 568 loaded meshes, 17 textured materials, zero
fallback, desktop/mobile nonblank.
- Go2 quadruped scene: 34 objects, 33 loaded meshes, 12 joints exposed, zero
fallback, desktop/mobile nonblank.
- G1 + stocked shelf backend physical scene: served from bridge port 18090 with
shared backend collision bodies for the robot, shelves, and fixed stock
products. Store shelf/product groups are exposed for inspection, and product
acceptance requires `631/631` body bindings and `visual-only=0`.