MotrixSimWeb/README.md

12 KiB
Raw Permalink Blame History

MotrixSim Universal Three.js Preview

This repository now contains a working universal-preview path:

MotrixSim / MJCF scene
  -> scene manifest
  -> Three.js WebGL renderer
  -> replay / polling / websocket state source

The first verified version supports primitives, OBJ/STL/GLTF/GLB mesh loading, Z-up MotrixSim coordinate conversion, body-name state binding, desktop/mobile responsive layout, MJCF <include> expansion, HTTP asset serving, headless MotrixSim bridge controls, and basic Franka joint/gripper operation.

The current exporter/renderer path also covers higher-variance MJCF assets: nested <default> inheritance for visible-vs-collision geoms, assetdir / meshdir / texturedir resolution, file textures, builtin checker textures, mesh scale, fromto, xyaxes, zaxis, inline convex meshes, and browser-side mesh caching for repeated assets.

Run Locally

npm install
npm run dev -- --port 5173

Open:

http://127.0.0.1:5173/

For the real MotrixSim backend used by the G1 store scene, use the repeatable dev helper instead of hand-running SSH commands:

# Print local tunnel and bridge health.
npm run real-backend:status

# Check local Node, SSH access, remote MotrixSim paths/imports, running bridge,
# and manifest grouping before a handoff or demo.
npm run real-backend:doctor

# Sync server/headless_motrix_bridge.py to Wuying, restart the active physical
# scene paused, ensure the 18090 tunnel exists, then run real backend checks.
npm run real-backend:up

# Run the product-readiness gate: real backend doctor, backend edit sync,
# Scene Builder compose / share-link restore / config export-import restore /
# desktop / mobile screenshots, Chinese-default UI, UI-downloaded acceptance
# package/report, external simulation labeling, backend-physics labeling, and
# object physics-tier labeling, plus pause/resume latency checks.
npm run product:acceptance

# Full handoff/release gate: typecheck, production build, gzip bundle budget,
# and production-preview product acceptance.
npm run product:release-check

Useful overrides:

WUYING_HOST=39.106.101.95
WUYING_USER=root
WUYING_SSH_KEY=/path/to/wuying_ed25519
MOTRIXSIM_REMOTE_SCENE=/tmp/motrixsim-web-composed/active_g1_store_scene.xml
MOTRIXSIM_LOCAL_PORT=18090

The first screen is now a Scene Builder for the backend-required product path. The default G1 + 货架商品后端物理场景 preset uses http://127.0.0.1:18090/manifest; it loads one real MotrixSim MJCF runtime and binds every visible shelf/product/robot object to /state. When the matching bridge exposes POST /compose, Scene Builder object Add/Remove/Clear operations regenerate the combined MJCF on the backend and reload the live manifest. Those objects are real MotrixSim bodies with free joints and collision geoms, not front-end-only Three.js meshes. For physical store scenes, the bridge also publishes product-level editableGroup metadata derived from real runtime body poses. Store shelf frames, plates, and pre-stocked products are fixed environment bodies so they keep their collision shape and do not avalanche when the simulator resumes. Backend-compatible added props remain real dynamic bodies when the backend adapter creates them. The current default product scene starts with no added props; uploaded mesh assets are disabled until a backend adapter can turn them into real MJCF bodies. Repositioning a fixed shelf layout should be handled by rebuilding the composed scene, not by keeping the shelf as a free runtime body. The UI, diagnostics panel, JSON acceptance package, and HTML acceptance report now expose object physics tiers: backend dynamic bodies, backend fixed bodies, backend state-only bodies, and visual-only objects. Product acceptance requires visual-only=0.

To load a remote manifest:

http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18083%2Fmanifest

Verify

npm run build
npm run verify:exporter
npm run verify:preview

For a MotrixSim bridge with only primitive objects:

MIN_OBJECTS=2 MIN_MESHES=0 npm run verify:preview -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18083%2Fmanifest' \
  test-results/remote-motrix

For a larger robot scene with many mesh files, raise the verification timeout:

VERIFY_TIMEOUT_MS=45000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:preview -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
  test-results/robot-panda

For the backend control path:

VERIFY_TIMEOUT_MS=60000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:control -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
  test-results/control-panda

For the productized real backend compose and editing path on the G1 store scene:

npm run verify:real-backend -- \
  'http://127.0.0.1:18090/manifest' \
  test-results/real-backend-editing

This probe calls the live MotrixSim bridge, not a mock. It checks editable entity groups in /manifest, calls real backend /compose for a controlled legacy dynamic-body edit test, verifies the composed object bodies in /state, writes a pose for one composed dynamic body, verifies fixed pre-stocked product bodies reject set_body_pose, resumes the simulator, confirms fixed shelf anchors do not drift, checks a lifted dynamic prop drops under backend physics, and measures compact pause/resume control latency. After verification, npm run real-backend:up restores the default stocked shelf scene with no added test objects.

For handoff or acceptance, prefer the single product gate:

npm run product:acceptance

It writes test-results/product-readiness/product-readiness-report.json plus Scene Builder compose, share-link restore, config import restore, desktop, and mobile screenshots, plus an auditable test-results/product-readiness/product-readiness-evidence.json manifest with artifact hashes and git provenance. The gate also clicks the in-page export buttons, saves the downloaded builder config JSON, JSON acceptance package, and HTML report, parses them, imports the builder config through a real browser file chooser, opens the downloaded HTML report in Chromium, screenshots it, and records their hashes. If 127.0.0.1:5173 is not already running, it starts the Vite dev server for the duration of the check. It does not use mock scene data: backend checks call the live MotrixSim bridge, and browser checks load http://127.0.0.1:18090/manifest by default. The browser gate also clicks the Scene Builder Physical Scene control in an isolated browser context to prove the product UI triggers backend /compose, then checks the runtime diagnostics panel for manifest/state/control endpoints, external simulation mode, object bindings, asset health, and control latency. It also clicks the product copy-link action, reads the browser clipboard, opens the share URL in a clean browser context, and verifies that the restored scene uses the same builder config, backend physics manifest, /control endpoint, and object/body binding counts. It also exports the builder config, imports that file in a clean browser context, and verifies the same backend physical scene is restored. The gate also validates the object physics-tier evidence. The product physical scene must include backend fixed environment bodies and backend state-only robot links while keeping visual-only=0. The product gate also validates builder physics coverage: the default stocked shelf path has no added props, and any uploaded OBJ asset without a backend adapter is blocked instead of being added as a visual-only object.

For release handoff, npm run product:release-check first builds dist/, checks the JavaScript gzip bundle budget, then forces the browser gate through vite preview on an isolated localhost port instead of reusing the development server. Release checks fail on warnings.

For demo and handoff details, see docs/product-readiness.md. That document records the real backend topology, acceptance criteria, current product gate baseline, evidence paths, screenshot hashes, and known product boundaries.

For the Franka joint/gripper control path:

VERIFY_TIMEOUT_MS=70000 MIN_OBJECTS=50 MIN_MESHES=40 npm run verify:joint -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18084%2Fmanifest' \
  test-results/joint-panda

For a large textured store scene on the remote MotrixSim bridge:

VERIFY_TIMEOUT_MS=120000 MIN_OBJECTS=500 MIN_MESHES=20 npm run verify:preview -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18085%2Fmanifest' \
  test-results/store-textured

For a Go2 quadruped robot scene:

VERIFY_TIMEOUT_MS=60000 MIN_OBJECTS=30 MIN_MESHES=20 npm run verify:preview -- \
  'http://127.0.0.1:5173/?manifest=http%3A%2F%2F127.0.0.1%3A18086%2Fmanifest' \
  test-results/go2-robot

Headless MotrixSim Bridge

Run this on a Linux server with MotrixSim available:

python server/headless_motrix_bridge.py \
  --scene /path/to/scene.xml \
  --host 127.0.0.1 \
  --port 18083 \
  --hz 120

It exposes:

  • GET /manifest: Three.js preview manifest generated from the XML scene.
  • GET /state: live MotrixSim body pose state.
  • GET /assets/...: mesh sidecar assets referenced by the manifest.
  • POST /control: pause, resume, reset, step, home, set_joint, set_body_pose, set_body_poses, and gripper.
  • POST /compose: for supported physical presets, generate a new combined MJCF scene from selected robot/scene/object choices, hot-swap the runtime, and return the refreshed manifest.
  • GET /health: service health.

No display server is required for this state-driven preview path.

Current Boundary

This is a real Three.js/WebGL preview path, not the earlier Canvas 2.5D page. It is not yet a perfect renderer for every MotrixSim feature. Unsupported or private features still need explicit adapters or a server-side render/video fallback.

Scene Builder product composition must stay connected to a MotrixSim backend state source. Added objects are accepted only when a backend adapter creates real MJCF bodies. The current default physical path is the stocked shelf scene on port 18090; assets without adapters are visible in the asset library but cannot be inserted into the product physics scene.

页面内上传与组装(已支持)

  • 左侧「场景上传 / 上传场景包」支持一次选择 .json manifest 和关联的 mesh / texture / bin 文件。manifest 至少需要包含 version 与 objects。
  • 上传后会自动插入到场景下拉列表首位,并把 manifest 内相对路径(例如 meshes/robot.obj、textures/base.png)匹配到同批上传的本地文件,改写为浏览器可加载的 blob: URL。
  • 左侧「资产库 / 上传资产」支持 .gltf/.glb/.obj/.stl,也可以同时选择 .bin/.png/.jpg/.webp/.ktx2/.mtl 等 sidecar 文件;.gltf 内部的 buffer / image 相对引用会同步改写。
  • 已有后端适配的资产才能加入产品物理场景,并可继续调整位置(X/Y/Z 输入框)和通过左键拖拽编辑(暂停时可拖拽)。
  • 未接入后端适配的上传 mesh 会显示“需后端适配”,添加按钮保持禁用,不会作为“仅视觉”物体混入产品验收场景。
  • 上传面板支持拖放文件;「清空自定义」会清理上传资产;「清空场景」会移除当前上传场景并回到默认场景列表。

边界:本地上传 mesh 不会自动变成 MotrixSim 后端碰撞体。需要真实碰撞时,必须先通过后端适配把对象转成 MJCF body / geom;适配完成前产品模式不允许仅视觉加入。

The Franka Panda MJCF robot scene has been verified as a real robot asset preview: 59 manifest objects, 58 loaded OBJ/STL meshes, and zero mesh fallback. Joint-level operation is now wired for this scene through MotrixSim joint_names, link_names, and link pose streaming. Broader task-level robot actions still need scene-specific policy/action adapters.

Additional remote scenes verified:

  • Store shelf scene: 569 objects, 568 loaded meshes, 17 textured materials, zero fallback, desktop/mobile nonblank.
  • Go2 quadruped scene: 34 objects, 33 loaded meshes, 12 joints exposed, zero fallback, desktop/mobile nonblank.
  • G1 + stocked shelf backend physical scene: served from bridge port 18090 with shared backend collision bodies for the robot, shelves, and fixed stock products. Store shelf/product groups are exposed for inspection, and product acceptance requires 631/631 body bindings and visual-only=0.