Real2Sim External Model Adapters
These scripts connect the GS-Playground Real2Sim stage contract to public model implementations. They are optional: the built-in backend still runs without any of these dependencies.
Quality Upgrade Path
-
groundingdino_hf_detect.py- Purpose: replace heuristic connected components with open-vocabulary boxes.
- Source: official GroundingDINO / Grounded-SAM-2 Hugging Face path.
- Main dependency:
transformers,Pillow,torch. - Default model:
IDEA-Research/grounding-dino-tiny.
-
sam2_segment.py- Purpose: replace coarse masks with SAM2 box-prompted masks.
- Source: official
facebookresearch/sam2. - Required env:
REAL2SIM_SAM2_CHECKPOINT=/path/to/sam2.1_hiera_large.ptREAL2SIM_SAM2_MODEL_CFG=configs/sam2.1/sam2.1_hiera_l.yaml
-
instance_refine.py- Purpose: match the paper's visual-only instance de-duplication stage after SAM/SAM2.
- Method: composite detection/segmentation confidence, mask IoU, mask inclusion, and boundary overlap.
-
lama_inpaint.py- Purpose: replace median-fill background removal with sequential LaMa inpainting.
- Runtime used here:
simple-lama-inpainting, a lightweight LaMa wrapper listed by the official LaMa repository. - Method: one-object-at-a-time mask expansion and sequential background
update. Re-detection of newly exposed regions must be supplied by the
configured detection/segmentation loop for fully strict paper parity.
The strict config uses
REAL2SIM_GROUNDING_DINO_COMMANDandREAL2SIM_SAM2_COMMANDfor this nested loop.
-
depth_anything_depth.py- Purpose: replace flat pseudo-depth detail with a monocular Depth Anything V2 depth prior before building the fallback 3DGS assets.
- Main dependency:
transformers,Pillow,torch. - Default model:
depth-anything/Depth-Anything-V2-Small-hf.
-
anysplat_scene.py- Purpose: replace pseudo-depth scene Gaussians with AnySplat scene Gaussians.
- Source: official AnySplat repo and Hugging Face model
lhjiang/anysplat. - Output: official
export_plyscene Gaussians, decoder depth, camera metadata, and a depth-derived scene mesh. - Required env:
REAL2SIM_ANYSPLAT_ROOT=/path/to/AnySplat
- Best input: multiple nearby views or a short video, not a single repeated frame.
-
sam3d_objects_reconstruct.py- Purpose: replace object box meshes and object pseudo-Gaussians with SAM 3D Objects outputs.
- Source: official Meta
facebookresearch/sam-3d-objects. - Output: object Gaussian PLY from the SAM-3D
save_plyoutput, an exportable object mesh, and pose/scale metadata. The script fails if no mesh is returned unless--allow-fallback-meshis passed for debugging. - Required env:
REAL2SIM_SAM3D_ROOT=/path/to/sam-3d-objectsREAL2SIM_SAM3D_CONFIG=/path/to/checkpoints/hf/pipeline.yaml
- Checkpoints are gated on Hugging Face and require accepted access. The upstream setup currently recommends an NVIDIA GPU with at least 32 GB VRAM.
- Use
--cameras {cameras_json} --match-camera-preprocesswhen the scene was reconstructed by AnySplat, so SAM-3D receives the same aspect-preserving center crop and mask geometry as the scene camera. - Use
--canonicalize-gaussian-to-meshto rotate the object Gaussian PLY into the SAM-3D mesh canonical frame using only geometric extent axes. It updates positions, normals, and splat quaternions without source-image photometric scoring. - Use
--remove-video-overlayonly for screenshots/video thumbnails with a central play button; it inpaints detected bright, low-saturation UI pixels near the image center before SAM-3D inference.
-
depth_mask_pose_align.py- Purpose: align SAM-3D objects back into the AnySplat scene using rendered object Gaussian PLY depth/mask, scene depth, and camera intrinsics. Mesh rendering is used only as a fallback when the object Gaussian asset is unavailable or invalid.
- Output: an updated
{aligned_assets_json}manifest with refinedpose.position,pose.quat_wxyz, andpose.scalevalues plus alignment metadata including the selectedrender_source, candidate axis search label, mask IoU, centroid error, depth error, and quality summary. If the object Gaussian directory contains SAM-3D metadata showinggaussian_canonicalization.status=ok, the aligner keeps the canonical rotation and uses only a mask/depth position-scale seed instead of source-view axis search.
-
Speedy-Splat pruning
- Purpose: match the paper's 3DGS pruning stage.
- Configure
pruning.mode=speedy_splatand setpruning.external_commandorREAL2SIM_SPEEDY_SPLAT_COMMAND. - The pipeline fails if the command is missing in Speedy-Splat mode.
- The adapter keeps Speedy-Splat's official score computation, but applies an exact-count tie-safe prune mask so all-zero or heavily tied object scores cannot delete every Gaussian in a canonical local object asset.
Example
python -m real2sim run \
--backend hybrid \
--config configs/real2sim.external.example.json \
--input media/teaser.png \
--output /tmp/gs_real2sim_hybrid \
--prompt "robot. person. table. chair. object."
Use --backend external only after AnySplat and SAM 3D Objects are installed,
their checkpoints are available, and the machine meets the SAM-3D memory
requirements. Use python -m real2sim run ... --dry-run --backend hybrid to
check which stage commands are configured before launching a long model run.