Community Real2Sim extension (
real2simbranch). This is not the official paper release. Upstream now provides GS-Real2Sim. The paper title, acceptance notice and throughput claims below belong to the original authors, not to this extension. Our additions focus on asset packaging, texture references, preview tooling and failure analysis. See the release assessment and dependency license boundaries.
GS-Playground
A High-Throughput Photorealistic Simulator for Vision-Informed Robot Learning
Languages: English | 简体中文
🎉 Accepted to RSS 2026 🎉
GS-Playground is a high-throughput photorealistic simulation framework for vision-informed robot learning. It couples a parallel robot physics engine with batch 3D Gaussian Splatting (3DGS) rendering, enabling large-scale visual reinforcement learning with real-world appearance, rigid-link visual synchronization, and sim-ready assets.
✨ Highlights
- Photorealistic visual simulation: Batch 3DGS rendering for RGB and depth observations in robot learning loops.
- High-throughput perception: The paper reports up to
10^4FPS at640 x 480resolution with batch rendering and memory-efficient 3DGS assets. - Rigid-Link Gaussian Kinematics: 3DGS clusters are bound to simulator rigid bodies for temporally consistent robot and object motion.
- Parallel physics engine: A velocity-impulse solver designed for stable contact-rich robot tasks and large time steps.
- Real2Sim asset workflow: A pipeline for reconstructing photorealistic, physically consistent, memory-efficient scenes from real captures.
- Multi-embodiment scope: Experiments cover locomotion, navigation, and manipulation, including quadrupeds, humanoids, and robot arms.
🗺️ Release Plan
The paper system is larger than this preview repository. Planned releases:
- Core simulator API for batched robot simulation, synchronized 3DGS observations, RGB/depth cameras, contacts, and MJCF-compatible assets.
- Batch 3DGS renderer kernels, pruning utilities, memory-efficient asset loading, and multi-scene batching examples, GaussianRenderer .
- Benchmark suite for rendering throughput, physics stability, locomotion experiments from the RSS 2026 paper.
- Sensor modules for batch LiDAR examples (mesh temporary).
- Upstream reconstruction tools are now available in GS-Real2Sim; this branch remains an independent experimental integration.
- PPO and visual policy training scripts for vision-centric navigation and manipulation.
- Hugging Face release with compressed 3DGS assets, example scenes, robot assets, trained policies, and evaluation traces.
🧰 Environment Requirements
- Linux x86_64. The preview package is intended for 64-bit Linux systems.
- NVIDIA GPU with a recent Linux driver. The dependencies use the CUDA 12.8 PyTorch wheel; a local CUDA toolkit installation is not required, but the NVIDIA driver must be new enough for the bundled CUDA runtime. You can check driver visibility with
nvidia-smi. gitfor cloning the repository.uvfor dependency resolution, Python environment creation, and running the demos. No manual virtual env setup is required.
🛠️ Installation
Run all commands from this repository root.
# Skip this line if uv is already installed.
curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/discoverse-dev/gs_playground.git
cd gs_playground
uv sync
Dependency versions and platform markers are tracked in pyproject.toml and uv.lock.
🚀 Quick Start
Live Replay Demo
uv run python demo/live_demo/replay.py
Navigation Demo
uv run python demo/navigation/robot_locomotion.py --config configs/go2_scene1.json
uv run python demo/navigation/robot_locomotion.py --config configs/g1_scene1.json
The first navigation launch may take a while to load the robot policy and 3DGS assets. It is ready to use once the 3DGS view appears on the left.
Batch Rendering Benchmark
uv run jupyter nbconvert \
--to notebook \
--execute benchmark/mtx_batch_minimal.ipynb \
--ExecutePreprocessor.cwd=benchmark \
--output mtx_batch_minimal.executed.ipynb
Experimental Real2Sim Pipeline
This branch includes an experimental Real2Sim orchestration layer that follows the paper pipeline as closely as the public preview package allows:
- GroundingDINO detection.
- SAM/SAM2 instance segmentation.
- Visual mask-only instance de-duplication and confidence ranking.
- LaMa one-object-at-a-time mask expansion and sequential background inpainting.
- Optional Depth Anything V2 monocular depth estimation for the hybrid path.
- AnySplat scene-level 3DGS reconstruction through the official
lhjiang/anysplatmodel/export path. - SAM-3D object-level 3DGS, mesh, pose, and scale reconstruction through the
official
facebookresearch/sam-3d-objectsquick-start API. - Rendered SAM-3D object Gaussian depth/mask alignment against AnySplat scene depth, with mesh rendering only as a fallback.
- Speedy-Splat external pruning and GS-Playground scene packaging.
The large model implementations and checkpoints are not vendored in this
repository. Configure them as external commands in configs/real2sim.example.json
or through environment variables. configs/real2sim.external.example.json
contains runnable adapter commands for the scripts under
scripts/real2sim_external/. The exact stage artifact schema is documented in
docs/real2sim.md.
REAL2SIM_SYNTHETIC_MULTIVIEW_COMMAND(optional single-image expansion)REAL2SIM_GROUNDING_DINO_COMMANDREAL2SIM_SAM2_COMMANDREAL2SIM_INSTANCE_REFINEMENT_COMMANDREAL2SIM_LAMA_COMMANDREAL2SIM_DEPTH_COMMAND(optional)REAL2SIM_ANYSPLAT_COMMANDREAL2SIM_SAM3D_COMMANDREAL2SIM_POSE_ALIGNMENT_COMMAND(optional)REAL2SIM_SPEEDY_SPLAT_COMMAND(required whenpruning.mode=speedy_splat)
Run a configured pipeline:
uv run python -m real2sim run \
--input /path/to/rgb.png \
--output /tmp/gsplayground_real2sim_scene \
--prompt "all task-relevant rigid objects" \
--backend auto \
--config configs/real2sim.example.json
--backend auto uses configured external model commands when available, and
falls back to the built-in single-image baseline otherwise. Use
--backend external to require the GroundingDINO/SAM2/LaMa/AnySplat/SAM-3D
chain, --backend hybrid to use external GroundingDINO/SAM2/LaMa/Depth
artifacts with the built-in sim-ready 3D packaging fallback, or
--backend builtin to force the deterministic built-in path.
For best visual quality, use a short video or nearby-view image set so AnySplat
receives real multi-view evidence through {frames_dir}. A single RGB image can
still run through the chain, but the scene reconstruction is limited by the
single-view prior. Official SAM-3D Objects checkpoints are gated on Hugging Face
and the upstream setup currently recommends an NVIDIA GPU with at least 32 GB
VRAM.
When only one RGB image is available, --synthetic-multiview can call the
optional image-edit adapter to generate an 8-view contact sheet, split it into
frames, and pass those frames to AnySplat:
export REAL2SIM_IMAGE_EDIT_API_KEY=...
uv run python -m real2sim run \
--input /path/to/rgb.png \
--output /tmp/gsplayground_real2sim_scene \
--prompt "all task-relevant rigid objects" \
--backend external \
--config configs/real2sim.external.example.json \
--synthetic-multiview
This is a reconstruction-quality aid for single-image inputs. For strict paper-equivalent evaluation, prefer real nearby-view photos or video frames.
For paper-strict runs, keep pruning.mode set to speedy_splat and configure
the external Speedy-Splat command. If that command is missing the pipeline fails
instead of silently using the built-in opacity/area pruner.
Use --paper-strict when you want exact paper-equivalent behavior. It forces
the external pipeline, verifies official AnySplat/SAM-3D/SAM2/Speedy-Splat
runtime prerequisites, and fails before running if any required stage would
fall back to an approximation:
uv run python -m real2sim check \
--paper-strict \
--backend external \
--config configs/real2sim.external.example.json
The output package contains mjcf/scene.xml, 3dgs/*.ply,
configs/<robot>_real2sim.json, and package_manifest.json. It can be loaded by
the navigation demo:
uv run python demo/navigation/robot_locomotion.py \
--config /tmp/gsplayground_real2sim_scene/configs/go2_real2sim.json
If you already have reconstructed assets, package them without running the model stages:
uv run python -m real2sim package-existing \
--source /path/to/source.png \
--scene-ply /path/to/point_cloud.ply \
--scene-mesh /path/to/mesh.obj \
--output /tmp/gsplayground_real2sim_scene
Dynamic object Gaussian clusters are linked to same-name MJCF bodies through the
generated gaussian_links config, so the renderer can update them with
Rigid-Link Gaussian Kinematics when the physics model exposes those bodies.
Optional Jupyter Kernel
uv run python -m ipykernel install \
--user \
--name gsplayground \
--display-name "gsplayground"
🔗 Related Projects
GS-Playground builds on several components and prior systems from our ecosystem. They are not fully integrated into this preview repository yet; future releases will consolidate the relevant physics, rendering, sensing, and learning interfaces into the GS-Playground workflow described in the RSS 2026 paper.
- Physics simulator: MotrixSim provides the robot physics backend behind the high-throughput contact-rich simulation stack.
- State-based RL: MotrixLab contains state-based reinforcement learning infrastructure that will be connected to the GS-Playground training pipeline.
- RLGK rendering: GaussianRenderer includes the Gaussian rendering components related to Rigid-Link Gaussian Kinematics.
- Batch LiDAR: MuJoCo-LiDAR is our earlier batch LiDAR module; the GS-Playground sensor suite will integrate this line of work for navigation and locomotion tasks.
- Previous-generation platform: DISCOVERSE is our earlier embodied simulation platform. GS-Playground can be viewed as a next-generation, photorealistic and high-throughput successor to DISCOVERSE.
Add the assistant on WeChat to join the group. Please note in your request: gsp交流
📚 Citation
@article{jia2025discoverse,
title={DISCOVERSE: Efficient Robot Simulation in Complex High-Fidelity Environments},
author={Yufei Jia and Guangyu Wang and Yuhang Dong and Junzhe Wu and Yupei Zeng and Haonan Lin and Zifan Wang and Haizhou Ge and Weibin Gu and Chuxuan Li and Ziming Wang and Yunjie Cheng and Wei Sui and Ruqi Huang and Guyue Zhou},
journal={arXiv preprint arXiv:2507.21981},
year={2025},
url={https://arxiv.org/abs/2507.21981}
}
@article{jia2026gsplayground,
title={GS-Playground: A High-Throughput Photorealistic Simulator for Vision-Informed Robot Learning},
author={Yufei Jia and Heng Zhang and Ziheng Zhang and Junzhe Wu and Mingrui Yu and Zifan Wang and Dixuan Jiang and Zheng Li and Chenyu Cao and Zhuoyuan Yu and Xun Yang and Haizhou Ge and Yuchi Zhang and Jiayuan Zhang and Zhenbiao Huang and Tianle Liu and Shenyu Chen and Jiacheng Wang and Bin Xie and Xuran Yao and Xiwa Deng and Guangyu Wang and Jinzhi Zhang and Lei Hao and Zhixing Chen and Yuxiang Chen and Anqi Wang and Hongyun Tian and Yiyi Yan and Zhanxiang Cao and Yizhou Jiang and Hanyang Shao and Yue Li and Lu Shi and Bokui Chen and Wei Sui and Hanqing Cui and Yusen Qin and Ruqi Huang and Lei Han and Tiancai Wang and Guyue Zhou},
journal={arXiv preprint arXiv:2604.25459},
year={2026},
url={https://arxiv.org/abs/2604.25459}
}
For the GS-Playground physics engine component:
@software{motrixsim2026,
title = {MotrixSim: A Physics Simulation Engine for Robotics and Embodied AI},
author = {{Motphys Team}},
year = {2026},
url = {https://motrixsim.readthedocs.io/},
note = {Python binary package}
}

