285 lines
12 KiB
Markdown
285 lines
12 KiB
Markdown
# MotrixSim Universal Three.js Preview
|
||
|
||
This repository now contains a working universal-preview path:
|
||
|
||
```text
|
||
MotrixSim / MJCF scene
|
||
-> scene manifest
|
||
-> Three.js WebGL renderer
|
||
-> replay / polling / websocket state source
|
||
```
|
||
|
||
The first verified version supports primitives, OBJ/STL/GLTF/GLB mesh loading,
|
||
Z-up MotrixSim coordinate conversion, body-name state binding, desktop/mobile
|
||
responsive layout, MJCF `<include>` expansion, HTTP asset serving, headless
|
||
MotrixSim bridge controls, and basic Franka joint/gripper operation.
|
||
|
||
The current exporter/renderer path also covers higher-variance MJCF assets:
|
||
nested `<default>` inheritance for visible-vs-collision geoms, `assetdir` /
|
||
`meshdir` / `texturedir` resolution, file textures, builtin checker textures,
|
||
mesh scale, `fromto`, `xyaxes`, `zaxis`, inline convex meshes, and browser-side
|
||
mesh caching for repeated assets.
|
||
|
||
## Run Locally
|
||
|
||
```bash
|
||
npm install
|
||
npm run dev -- --port 5173
|
||
```
|
||
|
||
Open:
|
||
|
||
```text
|
||
http://127.0.0.1:5173/
|
||
```
|
||
|
||
For the real MotrixSim backend used by the G1 store scene, use the repeatable
|
||
dev helper instead of hand-running SSH commands:
|
||
|
||
```bash
|
||
# Print local tunnel and bridge health.
|
||
npm run real-backend:status
|
||
|
||
# Check local Node, SSH access, remote MotrixSim paths/imports, running bridge,
|
||
# and manifest grouping before a handoff or demo.
|
||
npm run real-backend:doctor
|
||
|
||
# Sync server/headless_motrix_bridge.py to Wuying, restart the active physical
|
||
# scene paused, ensure the 18090 tunnel exists, then run real backend checks.
|
||
npm run real-backend:up
|
||
|
||
# Run the product-readiness gate: real backend doctor, backend edit sync,
|
||
# Scene Builder compose / share-link restore / config export-import restore /
|
||
# desktop / mobile screenshots, Chinese-default UI, UI-downloaded acceptance
|
||
# package/report, external simulation labeling, backend-physics labeling, and
|
||
# object physics-tier labeling, plus pause/resume latency checks.
|
||
npm run product:acceptance
|
||
|
||
# Full handoff/release gate: typecheck, production build, gzip bundle budget,
|
||
# and production-preview product acceptance.
|
||
npm run product:release-check
|
||
```
|
||
|
||
Useful overrides:
|
||
|
||
```bash
|
||
WUYING_HOST=39.106.101.95
|
||
WUYING_USER=root
|
||
WUYING_SSH_KEY=/path/to/wuying_ed25519
|
||
MOTRIXSIM_REMOTE_SCENE=/tmp/motrixsim-web-composed/active_g1_store_scene.xml
|
||
MOTRIXSIM_LOCAL_PORT=18090
|
||
```
|
||
|
||
The first screen is now a Scene Builder for the backend-required product path.
|
||
The default `G1 + 货架商品后端物理场景` preset uses
|
||
`http://127.0.0.1:18090/manifest`; it loads one real MotrixSim MJCF runtime and
|
||
binds every visible shelf/product/robot object to `/state`.
|
||
When the matching bridge exposes `POST /compose`, Scene Builder object
|
||
Add/Remove/Clear operations regenerate the combined MJCF on the backend and
|
||
reload the live manifest. Those objects are real MotrixSim bodies with
|
||
free joints and collision geoms, not front-end-only Three.js meshes.
|
||
For physical store scenes, the bridge also publishes product-level
|
||
`editableGroup` metadata derived from real runtime body poses. Store shelf
|
||
frames, plates, and pre-stocked products are fixed environment bodies so they
|
||
keep their collision shape and do not avalanche when the simulator resumes.
|
||
Backend-compatible added props remain real dynamic bodies when the backend
|
||
adapter creates them. The current default product scene starts with no added
|
||
props; uploaded mesh assets are disabled until a backend adapter can turn them
|
||
into real MJCF bodies. Repositioning a fixed shelf layout should be handled by
|
||
rebuilding the composed scene, not by keeping the shelf as a free runtime body.
|
||
The UI, diagnostics panel, JSON acceptance package, and HTML acceptance report
|
||
now expose object physics tiers: backend dynamic bodies, backend fixed bodies,
|
||
backend state-only bodies, and visual-only objects. Product acceptance requires
|
||
`visual-only=0`.
|
||
|
||
To load a remote manifest:
|
||
|
||
```text
|
||
http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18083%2Fmanifest
|
||
```
|
||
|
||
## Verify
|
||
|
||
```bash
|
||
npm run build
|
||
npm run verify:exporter
|
||
npm run verify:preview
|
||
```
|
||
|
||
For a MotrixSim bridge with only primitive objects:
|
||
|
||
```bash
|
||
MIN_OBJECTS=2 MIN_MESHES=0 npm run verify:preview -- \
|
||
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18083%2Fmanifest' \
|
||
test-results/remote-motrix
|
||
```
|
||
|
||
For a larger robot scene with many mesh files, raise the verification timeout:
|
||
|
||
```bash
|
||
VERIFY_TIMEOUT_MS=45000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:preview -- \
|
||
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
|
||
test-results/robot-panda
|
||
```
|
||
|
||
For the backend control path:
|
||
|
||
```bash
|
||
VERIFY_TIMEOUT_MS=60000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:control -- \
|
||
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
|
||
test-results/control-panda
|
||
```
|
||
|
||
For the productized real backend compose and editing path on the G1 store scene:
|
||
|
||
```bash
|
||
npm run verify:real-backend -- \
|
||
'http://127.0.0.1:18090/manifest' \
|
||
test-results/real-backend-editing
|
||
```
|
||
|
||
This probe calls the live MotrixSim bridge, not a mock. It checks editable
|
||
entity groups in `/manifest`, calls real backend `/compose` for a controlled
|
||
legacy dynamic-body edit test, verifies the composed object bodies in `/state`,
|
||
writes a pose for one composed dynamic body, verifies fixed pre-stocked product
|
||
bodies reject `set_body_pose`, resumes the simulator, confirms fixed shelf
|
||
anchors do not drift, checks a lifted dynamic prop drops under backend physics,
|
||
and measures compact pause/resume control latency. After verification,
|
||
`npm run real-backend:up` restores the default stocked shelf scene with no added
|
||
test objects.
|
||
|
||
For handoff or acceptance, prefer the single product gate:
|
||
|
||
```bash
|
||
npm run product:acceptance
|
||
```
|
||
|
||
It writes `test-results/product-readiness/product-readiness-report.json` plus
|
||
Scene Builder compose, share-link restore, config import restore, desktop, and mobile screenshots, plus an auditable
|
||
`test-results/product-readiness/product-readiness-evidence.json` manifest with
|
||
artifact hashes and git provenance. The gate also clicks the in-page export
|
||
buttons, saves the downloaded builder config JSON, JSON acceptance package, and
|
||
HTML report, parses them, imports the builder config through a real browser file
|
||
chooser, opens the downloaded HTML report in Chromium, screenshots it, and
|
||
records their hashes. If `127.0.0.1:5173` is not already running,
|
||
it starts the Vite dev server for the duration of the check. It does not use
|
||
mock scene data: backend checks call the live MotrixSim bridge, and browser
|
||
checks load `http://127.0.0.1:18090/manifest` by default. The browser gate also
|
||
clicks the Scene Builder `Physical Scene` control in an isolated browser
|
||
context to prove the product UI triggers backend `/compose`, then checks the
|
||
runtime diagnostics panel for manifest/state/control endpoints, external
|
||
simulation mode, object bindings, asset health, and control latency. It also
|
||
clicks the product copy-link action, reads the browser clipboard, opens the
|
||
share URL in a clean browser context, and verifies that the restored scene uses
|
||
the same builder config, backend physics manifest, `/control` endpoint, and
|
||
object/body binding counts. It also exports the builder config, imports that
|
||
file in a clean browser context, and verifies the same backend physical scene is
|
||
restored.
|
||
The gate also validates the object physics-tier evidence. The product physical
|
||
scene must include backend fixed environment bodies and backend state-only robot
|
||
links while keeping `visual-only=0`. The product gate also validates builder
|
||
physics coverage: the default stocked shelf path has no added props, and any
|
||
uploaded OBJ asset without a backend adapter is blocked instead of being added
|
||
as a visual-only object.
|
||
|
||
For release handoff, `npm run product:release-check` first builds `dist/`,
|
||
checks the JavaScript gzip bundle budget, then forces the browser gate through
|
||
`vite preview` on an isolated localhost port instead of reusing the development
|
||
server. Release checks fail on warnings.
|
||
|
||
For demo and handoff details, see
|
||
[`docs/product-readiness.md`](docs/product-readiness.md). That document records
|
||
the real backend topology, acceptance criteria, current product gate baseline,
|
||
evidence paths, screenshot hashes, and known product boundaries.
|
||
|
||
For the Franka joint/gripper control path:
|
||
|
||
```bash
|
||
VERIFY_TIMEOUT_MS=70000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:joint -- \
|
||
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
|
||
test-results/joint-panda
|
||
```
|
||
|
||
For a large textured store scene on the remote MotrixSim bridge:
|
||
|
||
```bash
|
||
VERIFY_TIMEOUT_MS=120000 MIN_OBJECTS=500 MIN_MESHES=20 npm run verify:preview -- \
|
||
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18085%2Fmanifest' \
|
||
test-results/store-textured
|
||
```
|
||
|
||
For a Go2 quadruped robot scene:
|
||
|
||
```bash
|
||
VERIFY_TIMEOUT_MS=60000 MIN_OBJECTS=30 MIN_MESHES=20 npm run verify:preview -- \
|
||
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18086%2Fmanifest' \
|
||
test-results/go2-robot
|
||
```
|
||
|
||
## Headless MotrixSim Bridge
|
||
|
||
Run this on a Linux server with MotrixSim available:
|
||
|
||
```bash
|
||
python server/headless_motrix_bridge.py \
|
||
--scene /path/to/scene.xml \
|
||
--host 127.0.0.1 \
|
||
--port 18083 \
|
||
--hz 120
|
||
```
|
||
|
||
It exposes:
|
||
|
||
- `GET /manifest`: Three.js preview manifest generated from the XML scene.
|
||
- `GET /state`: live MotrixSim body pose state.
|
||
- `GET /assets/...`: mesh sidecar assets referenced by the manifest.
|
||
- `POST /control`: `pause`, `resume`, `reset`, `step`, `home`,
|
||
`set_joint`, `set_body_pose`, `set_body_poses`, and `gripper`.
|
||
- `POST /compose`: for supported physical presets, generate a new combined
|
||
MJCF scene from selected robot/scene/object choices, hot-swap the runtime,
|
||
and return the refreshed manifest.
|
||
- `GET /health`: service health.
|
||
|
||
No display server is required for this state-driven preview path.
|
||
|
||
## Current Boundary
|
||
|
||
This is a real Three.js/WebGL preview path, not the earlier Canvas 2.5D page.
|
||
It is not yet a perfect renderer for every MotrixSim feature. Unsupported or
|
||
private features still need explicit adapters or a server-side render/video
|
||
fallback.
|
||
|
||
Scene Builder product composition must stay connected to a MotrixSim backend
|
||
state source. Added objects are accepted only when a backend adapter creates
|
||
real MJCF bodies. The current default physical path is the stocked shelf scene on
|
||
port `18090`; assets without adapters are visible in the asset library but cannot
|
||
be inserted into the product physics scene.
|
||
|
||
## 页面内上传与组装(已支持)
|
||
|
||
- 左侧「场景上传 / 上传场景包」支持一次选择 `.json` manifest 和关联的 mesh / texture / bin 文件。manifest 至少需要包含 `version` 与 `objects`。
|
||
- 上传后会自动插入到场景下拉列表首位,并把 manifest 内相对路径(例如 `meshes/robot.obj`、`textures/base.png`)匹配到同批上传的本地文件,改写为浏览器可加载的 `blob:` URL。
|
||
- 左侧「资产库 / 上传资产」支持 `.gltf/.glb/.obj/.stl`,也可以同时选择 `.bin/.png/.jpg/.webp/.ktx2/.mtl` 等 sidecar 文件;`.gltf` 内部的 buffer / image 相对引用会同步改写。
|
||
- 已有后端适配的资产才能加入产品物理场景,并可继续调整位置(X/Y/Z 输入框)和通过左键拖拽编辑(暂停时可拖拽)。
|
||
- 未接入后端适配的上传 mesh 会显示“需后端适配”,添加按钮保持禁用,不会作为“仅视觉”物体混入产品验收场景。
|
||
- 上传面板支持拖放文件;「清空自定义」会清理上传资产;「清空场景」会移除当前上传场景并回到默认场景列表。
|
||
|
||
边界:本地上传 mesh 不会自动变成 MotrixSim 后端碰撞体。需要真实碰撞时,必须先通过后端适配把对象转成 MJCF body / geom;适配完成前产品模式不允许仅视觉加入。
|
||
|
||
The Franka Panda MJCF robot scene has been verified as a real robot asset
|
||
preview: 59 manifest objects, 58 loaded OBJ/STL meshes, and zero mesh fallback.
|
||
Joint-level operation is now wired for this scene through MotrixSim
|
||
`joint_names`, `link_names`, and link pose streaming. Broader task-level robot
|
||
actions still need scene-specific policy/action adapters.
|
||
|
||
Additional remote scenes verified:
|
||
|
||
- Store shelf scene: 569 objects, 568 loaded meshes, 17 textured materials, zero
|
||
fallback, desktop/mobile nonblank.
|
||
- Go2 quadruped scene: 34 objects, 33 loaded meshes, 12 joints exposed, zero
|
||
fallback, desktop/mobile nonblank.
|
||
- G1 + stocked shelf backend physical scene: served from bridge port 18090 with
|
||
shared backend collision bodies for the robot, shelves, and fixed stock
|
||
products. Store shelf/product groups are exposed for inspection, and product
|
||
acceptance requires `631/631` body bindings and `visual-only=0`.
|