2026-04-30 10:47:00 +08:00
2026-04-29 09:40:29 +08:00

Community Real2Sim extension (real2sim branch). This is not the official paper release. Upstream now provides GS-Real2Sim. The paper title, acceptance notice and throughput claims below belong to the original authors, not to this extension. Our additions focus on asset packaging, texture references, preview tooling and failure analysis. See the release assessment and dependency license boundaries.

GS-Playground

A High-Throughput Photorealistic Simulator for Vision-Informed Robot Learning

Languages: English | 简体中文

Project Page arXiv Hugging Face RSS 2026

🎉 Accepted to RSS 2026 🎉

GS-Playground teaser

GS-Playground is a high-throughput photorealistic simulation framework for vision-informed robot learning. It couples a parallel robot physics engine with batch 3D Gaussian Splatting (3DGS) rendering, enabling large-scale visual reinforcement learning with real-world appearance, rigid-link visual synchronization, and sim-ready assets.

✨ Highlights

  • Photorealistic visual simulation: Batch 3DGS rendering for RGB and depth observations in robot learning loops.
  • High-throughput perception: The paper reports up to 10^4 FPS at 640 x 480 resolution with batch rendering and memory-efficient 3DGS assets.
  • Rigid-Link Gaussian Kinematics: 3DGS clusters are bound to simulator rigid bodies for temporally consistent robot and object motion.
  • Parallel physics engine: A velocity-impulse solver designed for stable contact-rich robot tasks and large time steps.
  • Real2Sim asset workflow: A pipeline for reconstructing photorealistic, physically consistent, memory-efficient scenes from real captures.
  • Multi-embodiment scope: Experiments cover locomotion, navigation, and manipulation, including quadrupeds, humanoids, and robot arms.

🗺️ Release Plan

The paper system is larger than this preview repository. Planned releases:

🧰 Environment Requirements

  • Linux x86_64. The preview package is intended for 64-bit Linux systems.
  • NVIDIA GPU with a recent Linux driver. The dependencies use the CUDA 12.8 PyTorch wheel; a local CUDA toolkit installation is not required, but the NVIDIA driver must be new enough for the bundled CUDA runtime. You can check driver visibility with nvidia-smi.
  • git for cloning the repository.
  • uv for dependency resolution, Python environment creation, and running the demos. No manual virtual env setup is required.

🛠️ Installation

Run all commands from this repository root.

# Skip this line if uv is already installed.
curl -LsSf https://astral.sh/uv/install.sh | sh

git clone https://github.com/discoverse-dev/gs_playground.git
cd gs_playground
uv sync

Dependency versions and platform markers are tracked in pyproject.toml and uv.lock.

🚀 Quick Start

Live Replay Demo

uv run python demo/live_demo/replay.py

Navigation Demo

uv run python demo/navigation/robot_locomotion.py --config configs/go2_scene1.json
uv run python demo/navigation/robot_locomotion.py --config configs/g1_scene1.json

The first navigation launch may take a while to load the robot policy and 3DGS assets. It is ready to use once the 3DGS view appears on the left.

Batch Rendering Benchmark

uv run jupyter nbconvert \
  --to notebook \
  --execute benchmark/mtx_batch_minimal.ipynb \
  --ExecutePreprocessor.cwd=benchmark \
  --output mtx_batch_minimal.executed.ipynb

Experimental Real2Sim Pipeline

This branch includes an experimental Real2Sim orchestration layer that follows the paper pipeline as closely as the public preview package allows:

  1. GroundingDINO detection.
  2. SAM/SAM2 instance segmentation.
  3. Visual mask-only instance de-duplication and confidence ranking.
  4. LaMa one-object-at-a-time mask expansion and sequential background inpainting.
  5. Optional Depth Anything V2 monocular depth estimation for the hybrid path.
  6. AnySplat scene-level 3DGS reconstruction through the official lhjiang/anysplat model/export path.
  7. SAM-3D object-level 3DGS, mesh, pose, and scale reconstruction through the official facebookresearch/sam-3d-objects quick-start API.
  8. Rendered SAM-3D object Gaussian depth/mask alignment against AnySplat scene depth, with mesh rendering only as a fallback.
  9. Speedy-Splat external pruning and GS-Playground scene packaging.

The large model implementations and checkpoints are not vendored in this repository. Configure them as external commands in configs/real2sim.example.json or through environment variables. configs/real2sim.external.example.json contains runnable adapter commands for the scripts under scripts/real2sim_external/. The exact stage artifact schema is documented in docs/real2sim.md.

  • REAL2SIM_SYNTHETIC_MULTIVIEW_COMMAND (optional single-image expansion)
  • REAL2SIM_GROUNDING_DINO_COMMAND
  • REAL2SIM_SAM2_COMMAND
  • REAL2SIM_INSTANCE_REFINEMENT_COMMAND
  • REAL2SIM_LAMA_COMMAND
  • REAL2SIM_DEPTH_COMMAND (optional)
  • REAL2SIM_ANYSPLAT_COMMAND
  • REAL2SIM_SAM3D_COMMAND
  • REAL2SIM_POSE_ALIGNMENT_COMMAND (optional)
  • REAL2SIM_SPEEDY_SPLAT_COMMAND (required when pruning.mode=speedy_splat)

Run a configured pipeline:

uv run python -m real2sim run \
  --input /path/to/rgb.png \
  --output /tmp/gsplayground_real2sim_scene \
  --prompt "all task-relevant rigid objects" \
  --backend auto \
  --config configs/real2sim.example.json

--backend auto uses configured external model commands when available, and falls back to the built-in single-image baseline otherwise. Use --backend external to require the GroundingDINO/SAM2/LaMa/AnySplat/SAM-3D chain, --backend hybrid to use external GroundingDINO/SAM2/LaMa/Depth artifacts with the built-in sim-ready 3D packaging fallback, or --backend builtin to force the deterministic built-in path.

For best visual quality, use a short video or nearby-view image set so AnySplat receives real multi-view evidence through {frames_dir}. A single RGB image can still run through the chain, but the scene reconstruction is limited by the single-view prior. Official SAM-3D Objects checkpoints are gated on Hugging Face and the upstream setup currently recommends an NVIDIA GPU with at least 32 GB VRAM.

When only one RGB image is available, --synthetic-multiview can call the optional image-edit adapter to generate an 8-view contact sheet, split it into frames, and pass those frames to AnySplat:

export REAL2SIM_IMAGE_EDIT_API_KEY=...
uv run python -m real2sim run \
  --input /path/to/rgb.png \
  --output /tmp/gsplayground_real2sim_scene \
  --prompt "all task-relevant rigid objects" \
  --backend external \
  --config configs/real2sim.external.example.json \
  --synthetic-multiview

This is a reconstruction-quality aid for single-image inputs. For strict paper-equivalent evaluation, prefer real nearby-view photos or video frames.

For paper-strict runs, keep pruning.mode set to speedy_splat and configure the external Speedy-Splat command. If that command is missing the pipeline fails instead of silently using the built-in opacity/area pruner.

Use --paper-strict when you want exact paper-equivalent behavior. It forces the external pipeline, verifies official AnySplat/SAM-3D/SAM2/Speedy-Splat runtime prerequisites, and fails before running if any required stage would fall back to an approximation:

uv run python -m real2sim check \
  --paper-strict \
  --backend external \
  --config configs/real2sim.external.example.json

The output package contains mjcf/scene.xml, 3dgs/*.ply, configs/<robot>_real2sim.json, and package_manifest.json. It can be loaded by the navigation demo:

uv run python demo/navigation/robot_locomotion.py \
  --config /tmp/gsplayground_real2sim_scene/configs/go2_real2sim.json

If you already have reconstructed assets, package them without running the model stages:

uv run python -m real2sim package-existing \
  --source /path/to/source.png \
  --scene-ply /path/to/point_cloud.ply \
  --scene-mesh /path/to/mesh.obj \
  --output /tmp/gsplayground_real2sim_scene

Dynamic object Gaussian clusters are linked to same-name MJCF bodies through the generated gaussian_links config, so the renderer can update them with Rigid-Link Gaussian Kinematics when the physics model exposes those bodies.

Optional Jupyter Kernel

uv run python -m ipykernel install \
  --user \
  --name gsplayground \
  --display-name "gsplayground"

GS-Playground builds on several components and prior systems from our ecosystem. They are not fully integrated into this preview repository yet; future releases will consolidate the relevant physics, rendering, sensing, and learning interfaces into the GS-Playground workflow described in the RSS 2026 paper.

  • Physics simulator: MotrixSim provides the robot physics backend behind the high-throughput contact-rich simulation stack.
  • State-based RL: MotrixLab contains state-based reinforcement learning infrastructure that will be connected to the GS-Playground training pipeline.
  • RLGK rendering: GaussianRenderer includes the Gaussian rendering components related to Rigid-Link Gaussian Kinematics.
  • Batch LiDAR: MuJoCo-LiDAR is our earlier batch LiDAR module; the GS-Playground sensor suite will integrate this line of work for navigation and locomotion tasks.
  • Previous-generation platform: DISCOVERSE is our earlier embodied simulation platform. GS-Playground can be viewed as a next-generation, photorealistic and high-throughput successor to DISCOVERSE.

QR Code

Add the assistant on WeChat to join the group. Please note in your request: gsp交流

📚 Citation

@article{jia2025discoverse,
      title={DISCOVERSE: Efficient Robot Simulation in Complex High-Fidelity Environments},
      author={Yufei Jia and Guangyu Wang and Yuhang Dong and Junzhe Wu and Yupei Zeng and Haonan Lin and Zifan Wang and Haizhou Ge and Weibin Gu and Chuxuan Li and Ziming Wang and Yunjie Cheng and Wei Sui and Ruqi Huang and Guyue Zhou},
      journal={arXiv preprint arXiv:2507.21981},
      year={2025},
      url={https://arxiv.org/abs/2507.21981}
}

@article{jia2026gsplayground,
      title={GS-Playground: A High-Throughput Photorealistic Simulator for Vision-Informed Robot Learning},
      author={Yufei Jia and Heng Zhang and Ziheng Zhang and Junzhe Wu and Mingrui Yu and Zifan Wang and Dixuan Jiang and Zheng Li and Chenyu Cao and Zhuoyuan Yu and Xun Yang and Haizhou Ge and Yuchi Zhang and Jiayuan Zhang and Zhenbiao Huang and Tianle Liu and Shenyu Chen and Jiacheng Wang and Bin Xie and Xuran Yao and Xiwa Deng and Guangyu Wang and Jinzhi Zhang and Lei Hao and Zhixing Chen and Yuxiang Chen and Anqi Wang and Hongyun Tian and Yiyi Yan and Zhanxiang Cao and Yizhou Jiang and Hanyang Shao and Yue Li and Lu Shi and Bokui Chen and Wei Sui and Hanqing Cui and Yusen Qin and Ruqi Huang and Lei Han and Tiancai Wang and Guyue Zhou},
      journal={arXiv preprint arXiv:2604.25459},
      year={2026},
      url={https://arxiv.org/abs/2604.25459}
}

For the GS-Playground physics engine component:

@software{motrixsim2026,
      title  = {MotrixSim: A Physics Simulation Engine for Robotics and Embodied AI},
      author = {{Motphys Team}},
      year   = {2026},
      url    = {https://motrixsim.readthedocs.io/},
      note   = {Python binary package}
}
Description
GS-Playground Real2Sim workbench
Readme MIT 113 MiB
Languages
Python 82.3%
Jupyter Notebook 16.8%
Shell 0.9%