MotrixSim Universal Three.js Preview

This repository now contains a working universal-preview path:

MotrixSim / MJCF scene
  -> scene manifest
  -> Three.js WebGL renderer
  -> replay / polling / websocket state source

The first verified version supports primitives, OBJ/STL/GLTF/GLB mesh loading, Z-up MotrixSim coordinate conversion, body-name state binding, desktop/mobile responsive layout, MJCF <include> expansion, HTTP asset serving, headless MotrixSim bridge controls, and basic Franka joint/gripper operation.

The current exporter/renderer path also covers higher-variance MJCF assets: nested <default> inheritance for visible-vs-collision geoms, assetdir / meshdir / texturedir resolution, file textures, builtin checker textures, mesh scale, fromto, xyaxes, zaxis, inline convex meshes, and browser-side mesh caching for repeated assets.

Run Locally

npm install
npm run dev -- --port 5173

Open:

http://127.0.0.1:5173/

For the real MotrixSim backend used by the G1 store scene, use the repeatable dev helper instead of hand-running SSH commands:

# Print local tunnel and bridge health.
npm run real-backend:status

# Check local Node, SSH access, remote MotrixSim paths/imports, running bridge,
# and manifest grouping before a handoff or demo.
npm run real-backend:doctor

# Sync server/headless_motrix_bridge.py to Wuying, restart the active physical
# scene paused, ensure the 18090 tunnel exists, then run real backend checks.
npm run real-backend:up

# Run the product-readiness gate: real backend doctor, backend edit sync,
# desktop/mobile browser screenshots, Chinese-default UI, external simulation
# labeling, backend-physics labeling, and pause/resume latency checks.
npm run product:acceptance

Useful overrides:

WUYING_HOST=39.106.101.95
WUYING_USER=root
WUYING_SSH_KEY=/path/to/wuying_ed25519
MOTRIXSIM_REMOTE_SCENE=/tmp/motrixsim-web-composed/active_g1_store_scene.xml
MOTRIXSIM_LOCAL_PORT=18090

The first screen is now a Scene Builder. It can compose a base scene, one live robot manifest, and front-end primitive objects into a single Three.js preview. Use Build Preview for visual composition, or Physical Scene to load the native Franka + cube MotrixSim scene where collision is computed by the backend. The G1 + Store + blocks physics preset uses http://127.0.0.1:18090/manifest and is a backend-composed MJCF scene, so G1, store shelves/products, and demo objects share one MotrixSim physics world. When the matching bridge exposes POST /compose, Scene Builder object Add/Remove/Clear operations regenerate the combined MJCF on the backend and reload the live manifest. Those objects are real MotrixSim bodies with free joints and collision geoms, not front-end-only Three.js meshes. For physical store scenes, the bridge also publishes product-level editableGroup metadata derived from real runtime body poses. While paused, dragging a shelf anchor writes a batched set_body_poses command back to MotrixSim so the shelf and its products move together; dragging a product member directly still edits that product's own real body pose.

To load a remote manifest:

http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18083%2Fmanifest

Verify

npm run build
npm run verify:exporter
npm run verify:preview

For a MotrixSim bridge with only primitive objects:

MIN_OBJECTS=2 MIN_MESHES=0 npm run verify:preview -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18083%2Fmanifest' \
  test-results/remote-motrix

For a larger robot scene with many mesh files, raise the verification timeout:

VERIFY_TIMEOUT_MS=45000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:preview -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
  test-results/robot-panda

For the backend control path:

VERIFY_TIMEOUT_MS=60000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:control -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
  test-results/control-panda

For the productized real backend editing path on the G1 store scene:

npm run verify:real-backend -- \
  'http://127.0.0.1:18090/manifest' \
  test-results/real-backend-editing

This probe calls the live MotrixSim bridge, not a mock. It checks editable entity groups in /manifest, verifies paused /state, writes a no-op pose for one member, writes the full shelf group through set_body_poses, and measures compact pause/resume control latency.

For handoff or acceptance, prefer the single product gate:

npm run product:acceptance

It writes test-results/product-readiness/product-readiness-report.json plus desktop/mobile screenshots. If 127.0.0.1:5173 is not already running, it starts the Vite dev server for the duration of the check. It does not use mock scene data: backend checks call the live MotrixSim bridge, and browser checks load http://127.0.0.1:18090/manifest by default.

For the Franka joint/gripper control path:

VERIFY_TIMEOUT_MS=70000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:joint -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
  test-results/joint-panda

For a large textured store scene on the remote MotrixSim bridge:

VERIFY_TIMEOUT_MS=120000 MIN_OBJECTS=500 MIN_MESHES=20 npm run verify:preview -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18085%2Fmanifest' \
  test-results/store-textured

For a Go2 quadruped robot scene:

VERIFY_TIMEOUT_MS=60000 MIN_OBJECTS=30 MIN_MESHES=20 npm run verify:preview -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18086%2Fmanifest' \
  test-results/go2-robot

Headless MotrixSim Bridge

Run this on a Linux server with MotrixSim available:

python server/headless_motrix_bridge.py \
  --scene /path/to/scene.xml \
  --host 127.0.0.1 \
  --port 18083 \
  --hz 120

It exposes:

  • GET /manifest: Three.js preview manifest generated from the XML scene.
  • GET /state: live MotrixSim body pose state.
  • GET /assets/...: mesh sidecar assets referenced by the manifest.
  • POST /control: pause, resume, reset, step, home, set_joint, set_body_pose, set_body_poses, and gripper.
  • POST /compose: for supported physical presets, generate a new combined MJCF scene from selected robot/scene/object choices, hot-swap the runtime, and return the refreshed manifest.
  • GET /health: service health.

No display server is required for this state-driven preview path.

Current Boundary

This is a real Three.js/WebGL preview path, not the earlier Canvas 2.5D page. It is not yet a perfect renderer for every MotrixSim feature. Unsupported or private features still need explicit adapters or a server-side render/video fallback.

Scene Builder visual composition does not create a new MotrixSim physics world. Added front-end objects are preview-only unless they are sent through a physical compose endpoint. The current physical compose path supports the G1 + Store + blocks physics preset and the built-in primitive object set.

页面内上传与组装(已支持)

  • 左侧「场景上传 / 上传场景包」支持一次选择 .json manifest 和关联的 mesh / texture / bin 文件。manifest 至少需要包含 version 与 objects。
  • 上传后会自动插入到场景下拉列表首位,并把 manifest 内相对路径(例如 meshes/robot.obj、textures/base.png)匹配到同批上传的本地文件,改写为浏览器可加载的 blob: URL。
  • 左侧「资产库 / 上传资产」支持 .gltf/.glb/.obj/.stl,也可以同时选择 .bin/.png/.jpg/.webp/.ktx2/.mtl 等 sidecar 文件;.gltf 内部的 buffer / image 相对引用会同步改写。
  • 点击资产卡片会把该对象加入当前场景列表,可继续调整位置(X/Y/Z 输入框)和通过左键拖拽编辑(暂停时可拖拽)。
  • 上传面板支持拖放文件;「清空自定义」会清理上传资产;「清空场景」会移除当前上传场景并回到默认场景列表。

边界:本地上传 mesh 默认是 Three.js 可视资产。它可以预览和摆放,但不会自动变成 MotrixSim 后端碰撞体。需要真实碰撞时,仍要通过后端 /compose 把对象转成 MJCF body / geom;当前后端物理组合只覆盖内置 primitive 对象。

The Franka Panda MJCF robot scene has been verified as a real robot asset preview: 59 manifest objects, 58 loaded OBJ/STL meshes, and zero mesh fallback. Joint-level operation is now wired for this scene through MotrixSim joint_names, link_names, and link pose streaming. Broader task-level robot actions still need scene-specific policy/action adapters.

Additional remote scenes verified:

  • Store shelf scene: 569 objects, 568 loaded meshes, 17 textured materials, zero fallback, desktop/mobile nonblank.
  • Go2 quadruped scene: 34 objects, 33 loaded meshes, 12 joints exposed, zero fallback, desktop/mobile nonblank.
  • G1 + Store + demo blocks physical scene: served from bridge port 18090 with shared backend collision bodies for the robot, shelves, products, and demo primitives. Store shelf/product groups are exposed as editable entity groups so the web UI can synchronize paused layout edits back into the MotrixSim runtime instead of only moving Three.js visuals.
Description
No description provided
Readme 24 MiB
Languages
TypeScript 55.8%
JavaScript 26.3%
Python 15%
CSS 2.6%
Shell 0.2%