MotrixSim Universal Three.js Preview

This repository now contains a working universal-preview path:

MotrixSim / MJCF scene
  -> scene manifest
  -> Three.js WebGL renderer
  -> replay / polling / websocket state source

The first verified version supports primitives, OBJ/STL/GLTF/GLB mesh loading, Z-up MotrixSim coordinate conversion, body-name state binding, desktop/mobile responsive layout, MJCF <include> expansion, HTTP asset serving, headless MotrixSim bridge controls, and basic Franka joint/gripper operation.

The current exporter/renderer path also covers higher-variance MJCF assets: nested <default> inheritance for visible-vs-collision geoms, assetdir / meshdir / texturedir resolution, file textures, builtin checker textures, mesh scale, fromto, xyaxes, zaxis, inline convex meshes, and browser-side mesh caching for repeated assets.

Run Locally

npm install
npm run dev -- --port 5173

Open:

http://127.0.0.1:5173/

The first screen is now a Scene Builder. It can compose a base scene, one live robot manifest, and front-end primitive objects into a single Three.js preview. Use Build Preview for visual composition, or Physical Scene to load the native Franka + cube MotrixSim scene where collision is computed by the backend. The G1 + Store + blocks physics preset uses http://127.0.0.1:18090/manifest and is a backend-composed MJCF scene, so G1, store shelves/products, and demo objects share one MotrixSim physics world. When the matching bridge exposes POST /compose, Scene Builder object Add/Remove/Clear operations regenerate the combined MJCF on the backend and reload the live manifest. Those objects are real MotrixSim bodies with free joints and collision geoms, not front-end-only Three.js meshes.

To load a remote manifest:

http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18083%2Fmanifest

Verify

npm run build
npm run verify:exporter
npm run verify:preview

For a MotrixSim bridge with only primitive objects:

MIN_OBJECTS=2 MIN_MESHES=0 npm run verify:preview -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18083%2Fmanifest' \
  test-results/remote-motrix

For a larger robot scene with many mesh files, raise the verification timeout:

VERIFY_TIMEOUT_MS=45000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:preview -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
  test-results/robot-panda

For the backend control path:

VERIFY_TIMEOUT_MS=60000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:control -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
  test-results/control-panda

For the Franka joint/gripper control path:

VERIFY_TIMEOUT_MS=70000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:joint -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
  test-results/joint-panda

For a large textured store scene on the remote MotrixSim bridge:

VERIFY_TIMEOUT_MS=120000 MIN_OBJECTS=500 MIN_MESHES=20 npm run verify:preview -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18085%2Fmanifest' \
  test-results/store-textured

For a Go2 quadruped robot scene:

VERIFY_TIMEOUT_MS=60000 MIN_OBJECTS=30 MIN_MESHES=20 npm run verify:preview -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18086%2Fmanifest' \
  test-results/go2-robot

Headless MotrixSim Bridge

Run this on a Linux server with MotrixSim available:

python server/headless_motrix_bridge.py \
  --scene /path/to/scene.xml \
  --host 127.0.0.1 \
  --port 18083 \
  --hz 120

It exposes:

  • GET /manifest: Three.js preview manifest generated from the XML scene.
  • GET /state: live MotrixSim body pose state.
  • GET /assets/...: mesh sidecar assets referenced by the manifest.
  • POST /control: pause, resume, reset, step, home, set_joint, and gripper.
  • POST /compose: for supported physical presets, generate a new combined MJCF scene from selected robot/scene/object choices, hot-swap the runtime, and return the refreshed manifest.
  • GET /health: service health.

No display server is required for this state-driven preview path.

Current Boundary

This is a real Three.js/WebGL preview path, not the earlier Canvas 2.5D page. It is not yet a perfect renderer for every MotrixSim feature. Unsupported or private features still need explicit adapters or a server-side render/video fallback.

Scene Builder visual composition does not create a new MotrixSim physics world. Added front-end objects are preview-only unless they are sent through a physical compose endpoint. The current physical compose path supports the G1 + Store + blocks physics preset and the built-in primitive object set.

页面内上传与组装(已支持)

  • 左侧「场景上传 / 上传场景包」支持一次选择 .json manifest 和关联的 mesh / texture / bin 文件。manifest 至少需要包含 version 与 objects。
  • 上传后会自动插入到场景下拉列表首位,并把 manifest 内相对路径(例如 meshes/robot.obj、textures/base.png)匹配到同批上传的本地文件,改写为浏览器可加载的 blob: URL。
  • 左侧「资产库 / 上传资产」支持 .gltf/.glb/.obj/.stl,也可以同时选择 .bin/.png/.jpg/.webp/.ktx2/.mtl 等 sidecar 文件;.gltf 内部的 buffer / image 相对引用会同步改写。
  • 点击资产卡片会把该对象加入当前场景列表,可继续调整位置(X/Y/Z 输入框)和通过左键拖拽编辑(暂停时可拖拽)。
  • 上传面板支持拖放文件;「清空自定义」会清理上传资产;「清空场景」会移除当前上传场景并回到默认场景列表。

边界:本地上传 mesh 默认是 Three.js 可视资产。它可以预览和摆放,但不会自动变成 MotrixSim 后端碰撞体。需要真实碰撞时,仍要通过后端 /compose 把对象转成 MJCF body / geom;当前后端物理组合只覆盖内置 primitive 对象。

The Franka Panda MJCF robot scene has been verified as a real robot asset preview: 59 manifest objects, 58 loaded OBJ/STL meshes, and zero mesh fallback. Joint-level operation is now wired for this scene through MotrixSim joint_names, link_names, and link pose streaming. Broader task-level robot actions still need scene-specific policy/action adapters.

Additional remote scenes verified:

  • Store shelf scene: 569 objects, 568 loaded meshes, 17 textured materials, zero fallback, desktop/mobile nonblank.
  • Go2 quadruped scene: 34 objects, 33 loaded meshes, 12 joints exposed, zero fallback, desktop/mobile nonblank.
  • G1 + Store + demo blocks physical scene: served from bridge port 18090 with shared backend collision bodies for the robot, shelves, products, and demo primitives.
Description
No description provided
Readme 24 MiB
Languages
TypeScript 55.8%
JavaScript 26.3%
Python 15%
CSS 2.6%
Shell 0.2%