MotrixSim Universal Three.js Preview
This repository now contains a working universal-preview path:
MotrixSim / MJCF scene
-> scene manifest
-> Three.js WebGL renderer
-> replay / polling / websocket state source
The first verified version supports primitives, OBJ/STL/GLTF/GLB mesh loading,
Z-up MotrixSim coordinate conversion, body-name state binding, desktop/mobile
responsive layout, MJCF <include> expansion, HTTP asset serving, headless
MotrixSim bridge controls, and basic Franka joint/gripper operation.
The current exporter/renderer path also covers higher-variance MJCF assets:
nested <default> inheritance for visible-vs-collision geoms, assetdir /
meshdir / texturedir resolution, file textures, builtin checker textures,
mesh scale, fromto, xyaxes, zaxis, inline convex meshes, and browser-side
mesh caching for repeated assets.
Run Locally
npm install
npm run dev -- --port 5173
Open:
http://127.0.0.1:5173/
For the real MotrixSim backend used by the G1 store scene, use the repeatable dev helper instead of hand-running SSH commands:
# Print local tunnel and bridge health.
npm run real-backend:status
# Check local Node, SSH access, remote MotrixSim paths/imports, running bridge,
# and manifest grouping before a handoff or demo.
npm run real-backend:doctor
# Sync server/headless_motrix_bridge.py to Wuying, restart the active physical
# scene paused, ensure the 18090 tunnel exists, then run real backend checks.
npm run real-backend:up
# Run the product-readiness gate: real backend doctor, backend edit sync,
# desktop/mobile browser screenshots, Chinese-default UI, external simulation
# labeling, backend-physics labeling, and pause/resume latency checks.
npm run product:acceptance
# Full handoff/release gate: typecheck, production build, gzip bundle budget,
# and production-preview product acceptance.
npm run product:release-check
Useful overrides:
WUYING_HOST=39.106.101.95
WUYING_USER=root
WUYING_SSH_KEY=/path/to/wuying_ed25519
MOTRIXSIM_REMOTE_SCENE=/tmp/motrixsim-web-composed/active_g1_store_scene.xml
MOTRIXSIM_LOCAL_PORT=18090
The first screen is now a Scene Builder. It can compose a base scene, one live
robot manifest, and front-end primitive objects into a single Three.js preview.
Use Build Preview for visual composition, or Physical Scene to load the
native Franka + cube MotrixSim scene where collision is computed by the backend.
The G1 + Store + blocks physics preset uses http://127.0.0.1:18090/manifest
and is a backend-composed MJCF scene, so G1, store shelves/products, and demo
objects share one MotrixSim physics world.
When the matching bridge exposes POST /compose, Scene Builder object
Add/Remove/Clear operations regenerate the combined MJCF on the backend and
reload the live manifest. Those objects are real MotrixSim bodies with
free joints and collision geoms, not front-end-only Three.js meshes.
For physical store scenes, the bridge also publishes product-level
editableGroup metadata derived from real runtime body poses. While paused,
dragging a shelf anchor writes a batched set_body_poses command back to
MotrixSim so the shelf and its products move together; dragging a product member
directly still edits that product's own real body pose.
To load a remote manifest:
http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18083%2Fmanifest
Verify
npm run build
npm run verify:exporter
npm run verify:preview
For a MotrixSim bridge with only primitive objects:
MIN_OBJECTS=2 MIN_MESHES=0 npm run verify:preview -- \
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18083%2Fmanifest' \
test-results/remote-motrix
For a larger robot scene with many mesh files, raise the verification timeout:
VERIFY_TIMEOUT_MS=45000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:preview -- \
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
test-results/robot-panda
For the backend control path:
VERIFY_TIMEOUT_MS=60000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:control -- \
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
test-results/control-panda
For the productized real backend editing path on the G1 store scene:
npm run verify:real-backend -- \
'http://127.0.0.1:18090/manifest' \
test-results/real-backend-editing
This probe calls the live MotrixSim bridge, not a mock. It checks editable
entity groups in /manifest, verifies paused /state, writes a no-op pose for
one member, writes the full shelf group through set_body_poses, and measures
compact pause/resume control latency.
For handoff or acceptance, prefer the single product gate:
npm run product:acceptance
It writes test-results/product-readiness/product-readiness-report.json plus
desktop/mobile screenshots. If 127.0.0.1:5173 is not already running, it
starts the Vite dev server for the duration of the check. It does not use mock
scene data: backend checks call the live MotrixSim bridge, and browser checks
load http://127.0.0.1:18090/manifest by default. The browser gate also
checks the runtime diagnostics panel for manifest/state/control endpoints,
external simulation mode, object bindings, asset health, and control latency.
For release handoff, npm run product:release-check first builds dist/,
checks the JavaScript gzip bundle budget, then forces the browser gate through
vite preview on an isolated localhost port instead of reusing the development
server. Release checks fail on warnings.
For demo and handoff details, see
docs/product-readiness.md. That document records
the real backend topology, acceptance criteria, current 22/22 baseline, evidence
paths, and known product boundaries.
For the Franka joint/gripper control path:
VERIFY_TIMEOUT_MS=70000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:joint -- \
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
test-results/joint-panda
For a large textured store scene on the remote MotrixSim bridge:
VERIFY_TIMEOUT_MS=120000 MIN_OBJECTS=500 MIN_MESHES=20 npm run verify:preview -- \
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18085%2Fmanifest' \
test-results/store-textured
For a Go2 quadruped robot scene:
VERIFY_TIMEOUT_MS=60000 MIN_OBJECTS=30 MIN_MESHES=20 npm run verify:preview -- \
'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18086%2Fmanifest' \
test-results/go2-robot
Headless MotrixSim Bridge
Run this on a Linux server with MotrixSim available:
python server/headless_motrix_bridge.py \
--scene /path/to/scene.xml \
--host 127.0.0.1 \
--port 18083 \
--hz 120
It exposes:
GET /manifest: Three.js preview manifest generated from the XML scene.GET /state: live MotrixSim body pose state.GET /assets/...: mesh sidecar assets referenced by the manifest.POST /control:pause,resume,reset,step,home,set_joint,set_body_pose,set_body_poses, andgripper.POST /compose: for supported physical presets, generate a new combined MJCF scene from selected robot/scene/object choices, hot-swap the runtime, and return the refreshed manifest.GET /health: service health.
No display server is required for this state-driven preview path.
Current Boundary
This is a real Three.js/WebGL preview path, not the earlier Canvas 2.5D page. It is not yet a perfect renderer for every MotrixSim feature. Unsupported or private features still need explicit adapters or a server-side render/video fallback.
Scene Builder visual composition does not create a new MotrixSim physics world.
Added front-end objects are preview-only unless they are sent through a physical
compose endpoint. The current physical compose path supports the
G1 + Store + blocks physics preset and the built-in primitive object set.
页面内上传与组装(已支持)
- 左侧「场景上传 / 上传场景包」支持一次选择
.jsonmanifest 和关联的 mesh / texture / bin 文件。manifest 至少需要包含version与objects。 - 上传后会自动插入到场景下拉列表首位,并把 manifest 内相对路径(例如
meshes/robot.obj、textures/base.png)匹配到同批上传的本地文件,改写为浏览器可加载的blob:URL。 - 左侧「资产库 / 上传资产」支持
.gltf/.glb/.obj/.stl,也可以同时选择.bin/.png/.jpg/.webp/.ktx2/.mtl等 sidecar 文件;.gltf内部的 buffer / image 相对引用会同步改写。 - 点击资产卡片会把该对象加入当前场景列表,可继续调整位置(X/Y/Z 输入框)和通过左键拖拽编辑(暂停时可拖拽)。
- 上传面板支持拖放文件;「清空自定义」会清理上传资产;「清空场景」会移除当前上传场景并回到默认场景列表。
边界:本地上传 mesh 默认是 Three.js 可视资产。它可以预览和摆放,但不会自动变成 MotrixSim 后端碰撞体。需要真实碰撞时,仍要通过后端 /compose 把对象转成 MJCF body / geom;当前后端物理组合只覆盖内置 primitive 对象。
The Franka Panda MJCF robot scene has been verified as a real robot asset
preview: 59 manifest objects, 58 loaded OBJ/STL meshes, and zero mesh fallback.
Joint-level operation is now wired for this scene through MotrixSim
joint_names, link_names, and link pose streaming. Broader task-level robot
actions still need scene-specific policy/action adapters.
Additional remote scenes verified:
- Store shelf scene: 569 objects, 568 loaded meshes, 17 textured materials, zero fallback, desktop/mobile nonblank.
- Go2 quadruped scene: 34 objects, 33 loaded meshes, 12 joints exposed, zero fallback, desktop/mobile nonblank.
- G1 + Store + demo blocks physical scene: served from bridge port 18090 with shared backend collision bodies for the robot, shelves, products, and demo primitives. Store shelf/product groups are exposed as editable entity groups so the web UI can synchronize paused layout edits back into the MotrixSim runtime instead of only moving Three.js visuals.