feat(docs): Add mkdocs to manager docs. (#50)
This commit is contained in:
Xinjie
GitHub
parent
031ba17742
commit
7a391771ef
30
.github/workflows/deploy_docs.yml
vendored
Normal file
30
.github/workflows/deploy_docs.yml
vendored
Normal file
@ -0,0 +1,30 @@
|
||||
name: Deploy MkDocs Documentation
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
jobs:
|
||||
build-deploy:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pip install mkdocs-material
|
||||
pip install mkdocstrings[python]
|
||||
pip install mkdocs-git-revision-date-localized-plugin
|
||||
|
||||
- name: Deploy documentation
|
||||
run: mkdocs gh-deploy --force
|
||||
4
.gitignore
vendored
4
.gitignore
vendored
@ -60,3 +60,7 @@ scripts/tools/
|
||||
weights
|
||||
apps/sessions/
|
||||
apps/assets/
|
||||
|
||||
# Larger than 1MB
|
||||
docs/assets/real2sim_mujoco.gif
|
||||
docs/assets/scene3d.gif
|
||||
49
README.md
49
README.md
@ -15,7 +15,7 @@
|
||||
> ***EmbodiedGen*** is a generative engine to create diverse and interactive 3D worlds composed of high-quality 3D assets(mesh & 3DGS) with plausible physics, leveraging generative AI to address the challenges of generalization in embodied intelligence related research.
|
||||
> It composed of six key modules: `Image-to-3D`, `Text-to-3D`, `Texture Generation`, `Articulated Object Generation`, `Scene Generation` and `Layout Generation`.
|
||||
|
||||
<img src="apps/assets/overall.jpg" alt="Overall Framework" width="700"/>
|
||||
<img src="docs/assets/overall.jpg" alt="Overall Framework" width="700"/>
|
||||
|
||||
---
|
||||
|
||||
@ -76,7 +76,7 @@ Explore EmbodiedGen generated assets in [](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Image-to-3D) Generate physically plausible 3D asset URDF from single input image, offering high-quality support for digital twin systems.
|
||||
(HF space is a simplified demonstration. For the full functionality, please refer to `img3d-cli`.)
|
||||
|
||||
<img src="apps/assets/image_to_3d.jpg" alt="Image to 3D" width="700">
|
||||
<img src="docs/assets/image_to_3d.jpg" alt="Image to 3D" width="700">
|
||||
|
||||
### ☁️ Service
|
||||
Run the image-to-3D generation service locally.
|
||||
@ -91,8 +91,8 @@ CUDA_VISIBLE_DEVICES=0 nohup python apps/image_to_3d.py > /dev/null 2>&1 &
|
||||
### ⚡ API
|
||||
Generate physically plausible 3D assets from image input via the command-line API.
|
||||
```sh
|
||||
img3d-cli --image_path apps/assets/example_image/sample_04.jpg apps/assets/example_image/sample_19.jpg \
|
||||
--n_retry 2 --output_root outputs/imageto3d
|
||||
img3d-cli --image_path apps/assets/example_image/sample_00.jpg apps/assets/example_image/sample_01.jpg apps/assets/example_image/sample_19.jpg \
|
||||
--n_retry 1 --output_root outputs/imageto3d
|
||||
|
||||
# See result(.urdf/mesh.obj/mesh.glb/gs.ply) in ${output_root}/sample_xx/result
|
||||
```
|
||||
@ -104,7 +104,7 @@ img3d-cli --image_path apps/assets/example_image/sample_04.jpg apps/assets/examp
|
||||
|
||||
[](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Text-to-3D) Create 3D assets from text descriptions for a wide range of geometry and styles. (HF space is a simplified demonstration. For the full functionality, please refer to `text3d-cli`.)
|
||||
|
||||
<img src="apps/assets/text_to_3d.jpg" alt="Text to 3D" width="700">
|
||||
<img src="docs/assets/text_to_3d.jpg" alt="Text to 3D" width="700">
|
||||
|
||||
### ☁️ Service
|
||||
Deploy the text-to-3D generation service locally.
|
||||
@ -119,11 +119,11 @@ python apps/text_to_3d.py
|
||||
Text-to-image model based on SD3.5 Medium, English prompts only.
|
||||
Usage requires agreement to the [model license(click accept)](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium), models downloaded automatically.
|
||||
|
||||
For large-scale 3D assets generation, set `--n_pipe_retry=2` to ensure high end-to-end 3D asset usability through automatic quality check and retries. For more diverse results, do not set `--seed_img`.
|
||||
For large-scale 3D asset generation, set `--n_image_retry=4` `--n_asset_retry=3` `--n_pipe_retry=2`, slower but better, via automatic checking and retries. For more diverse results, omit `--seed_img`.
|
||||
|
||||
```sh
|
||||
text3d-cli --prompts "small bronze figurine of a lion" "A globe with wooden base" "wooden table with embroidery" \
|
||||
--n_image_retry 2 --n_asset_retry 2 --n_pipe_retry 1 --seed_img 0 \
|
||||
--n_image_retry 1 --n_asset_retry 1 --n_pipe_retry 1 --seed_img 0 \
|
||||
--output_root outputs/textto3d
|
||||
```
|
||||
|
||||
@ -142,7 +142,7 @@ ps: models with more permissive licenses found in `embodied_gen/models/image_com
|
||||
|
||||
[](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Texture-Gen) Generate visually rich textures for 3D mesh.
|
||||
|
||||
<img src="apps/assets/texture_gen.jpg" alt="Texture Gen" width="700">
|
||||
<img src="docs/assets/texture_gen.jpg" alt="Texture Gen" width="700">
|
||||
|
||||
|
||||
### ☁️ Service
|
||||
@ -167,7 +167,7 @@ texture-cli --mesh_path "apps/assets/example_texture/meshes/robot_text.obj" \
|
||||
|
||||
<h2 id="3d-scene-generation">🌍 3D Scene Generation</h2>
|
||||
|
||||
<img src="apps/assets/scene3d.gif" alt="scene3d" style="width: 600px;">
|
||||
<img src="docs/assets/scene3d.gif" alt="scene3d" style="width: 600px;">
|
||||
|
||||
### ⚡ API
|
||||
> Run `bash install.sh extra` to install additional requirements if you need to use `scene3d-cli`.
|
||||
@ -190,7 +190,7 @@ CUDA_VISIBLE_DEVICES=0 scene3d-cli \
|
||||
|
||||
🚧 *Coming Soon*
|
||||
|
||||
<img src="apps/assets/articulate.gif" alt="articulate" style="width: 500px;">
|
||||
<img src="docs/assets/articulate.gif" alt="articulate" style="width: 500px;">
|
||||
|
||||
|
||||
---
|
||||
@ -202,12 +202,12 @@ CUDA_VISIBLE_DEVICES=0 scene3d-cli \
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td><img src="apps/assets/layout1.gif" alt="layout1" width="320"/></td>
|
||||
<td><img src="apps/assets/layout2.gif" alt="layout2" width="320"/></td>
|
||||
<td><img src="docs/assets/layout1.gif" alt="layout1" width="320"/></td>
|
||||
<td><img src="docs/assets/layout2.gif" alt="layout2" width="320"/></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><img src="apps/assets/layout3.gif" alt="layout3" width="320"/></td>
|
||||
<td><img src="apps/assets/layout4.gif" alt="layout4" width="320"/></td>
|
||||
<td><img src="docs/assets/layout3.gif" alt="layout3" width="320"/></td>
|
||||
<td><img src="docs/assets/layout4.gif" alt="layout4" width="320"/></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
@ -225,8 +225,8 @@ layout-cli --task_descs "Place the pen in the mug on the desk" "Put the fruit on
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td><img src="apps/assets/Iscene_demo1.gif" alt="Iscene_demo1" width="234"/></td>
|
||||
<td><img src="apps/assets/Iscene_demo2.gif" alt="Iscene_demo2" width="350"/></td>
|
||||
<td><img src="docs/assets/Iscene_demo1.gif" alt="Iscene_demo1" width="234"/></td>
|
||||
<td><img src="docs/assets/Iscene_demo2.gif" alt="Iscene_demo2" width="350"/></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
@ -243,7 +243,8 @@ Using `compose_layout.py`, you can recompose the layout of the generated interac
|
||||
```sh
|
||||
python embodied_gen/scripts/compose_layout.py \
|
||||
--layout_path "outputs/layouts_gens/task_0000/layout.json" \
|
||||
--output_dir "outputs/layouts_gens/task_0000/recompose" --insert_robot
|
||||
--output_dir "outputs/layouts_gens/task_0000/recompose" \
|
||||
--insert_robot
|
||||
```
|
||||
|
||||
We provide `sim-cli`, that allows users to easily load generated layouts into an interactive 3D simulation using the SAPIEN engine (will support for more simulators in future updates).
|
||||
@ -257,8 +258,8 @@ Example: generate multiple parallel simulation envs with `gym.make` and record s
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td><img src="apps/assets/parallel_sim.gif" alt="parallel_sim1" width="290"/></td>
|
||||
<td><img src="apps/assets/parallel_sim2.gif" alt="parallel_sim2" width="290"/></td>
|
||||
<td><img src="docs/assets/parallel_sim.gif" alt="parallel_sim1" width="290"/></td>
|
||||
<td><img src="docs/assets/parallel_sim2.gif" alt="parallel_sim2" width="290"/></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
@ -271,7 +272,7 @@ python embodied_gen/scripts/parallel_sim.py \
|
||||
|
||||
### 🖼️ Real-to-Sim Digital Twin
|
||||
|
||||
<img src="apps/assets/real2sim_mujoco.gif" alt="real2sim_mujoco" width="400">
|
||||
<img src="docs/assets/real2sim_mujoco.gif" alt="real2sim_mujoco" width="400">
|
||||
|
||||
---
|
||||
|
||||
@ -284,11 +285,11 @@ Example in `tests/test_examples/test_asset_converter.py`.
|
||||
| Simulator | Conversion Class |
|
||||
|-----------|------------------|
|
||||
| [isaacsim](https://github.com/isaac-sim/IsaacSim) | MeshtoUSDConverter |
|
||||
| [mujoco](https://github.com/google-deepmind/mujoco) | MeshtoMJCFConverter |
|
||||
| [genesis](https://github.com/Genesis-Embodied-AI/Genesis) / [sapien](https://github.com/haosulab/SAPIEN) / [isaacgym](https://github.com/isaac-sim/IsaacGymEnvs) / [pybullet](https://github.com/bulletphysics/bullet3) | EmbodiedGen generated .urdf can be used directly |
|
||||
| [mujoco](https://github.com/google-deepmind/mujoco) / [genesis](https://github.com/Genesis-Embodied-AI/Genesis) | MeshtoMJCFConverter |
|
||||
| [sapien](https://github.com/haosulab/SAPIEN) / [isaacgym](https://github.com/isaac-sim/IsaacGymEnvs) / [pybullet](https://github.com/bulletphysics/bullet3) | EmbodiedGen generated .urdf can be used directly |
|
||||
|
||||
|
||||
<img src="apps/assets/simulators_collision.jpg" alt="simulators_collision" width="500">
|
||||
<img src="docs/assets/simulators_collision.jpg" alt="simulators_collision" width="500">
|
||||
|
||||
---
|
||||
|
||||
@ -296,6 +297,8 @@ Example in `tests/test_examples/test_asset_converter.py`.
|
||||
```sh
|
||||
pip install -e .[dev] && pre-commit install
|
||||
python -m pytest # Pass all unit-test are required.
|
||||
# mkdocs serve --dev-addr 0.0.0.0:8000
|
||||
# mkdocs gh-deploy --force
|
||||
```
|
||||
|
||||
## 📚 Citation
|
||||
|
||||
BIN
apps/assets/example_image/sample_00.jpg
Normal file
BIN
apps/assets/example_image/sample_00.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 82 KiB |
BIN
apps/assets/example_image/sample_05.jpg
Normal file
BIN
apps/assets/example_image/sample_05.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 71 KiB |
@ -16,3 +16,6 @@ Pick up the charger and move it slightly to the left
|
||||
Move the jar to the left side of the desk
|
||||
Pick the rubik's cube on the top of the desk
|
||||
Move the mug to the right
|
||||
Put the apples from table to the basket
|
||||
Put the oranges from table to the bowl
|
||||
Put the red cup on the tray on the table
|
||||
@ -1,11 +1,52 @@
|
||||
# Project EmbodiedGen
|
||||
#
|
||||
# Copyright (c) 2025 Horizon Robotics. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
# implied. See the License for the specific language governing
|
||||
# permissions and limitations under the License.
|
||||
|
||||
|
||||
import os
|
||||
|
||||
gradio_tmp_dir = os.path.join(
|
||||
os.path.dirname(os.path.abspath(__file__)), "gradio_cache"
|
||||
)
|
||||
os.makedirs(gradio_tmp_dir, exist_ok=True)
|
||||
os.environ["GRADIO_TEMP_DIR"] = gradio_tmp_dir
|
||||
|
||||
import shutil
|
||||
import uuid
|
||||
import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Tuple
|
||||
|
||||
import gradio as gr
|
||||
import pandas as pd
|
||||
import yaml
|
||||
from app_style import custom_theme, lighting_css
|
||||
|
||||
try:
|
||||
from embodied_gen.utils.gpt_clients import GPT_CLIENT as gpt_client
|
||||
|
||||
gpt_client.check_connection()
|
||||
GPT_AVAILABLE = True
|
||||
except Exception as e:
|
||||
gpt_client = None
|
||||
GPT_AVAILABLE = False
|
||||
print(
|
||||
f"Warning: GPT client could not be initialized. Search will be disabled. Error: {e}"
|
||||
)
|
||||
|
||||
|
||||
# --- Configuration & Data Loading ---
|
||||
VERSION = "v0.1.5"
|
||||
RUNNING_MODE = "local" # local or hf_remote
|
||||
@ -36,6 +77,7 @@ TMP_DIR = os.path.join(
|
||||
)
|
||||
os.makedirs(TMP_DIR, exist_ok=True)
|
||||
|
||||
|
||||
# --- Custom CSS for Styling ---
|
||||
css = """
|
||||
.gradio-container .gradio-group { box-shadow: 0 2px 4px rgba(0,0,0,0.05) !important; }
|
||||
@ -44,14 +86,43 @@ css = """
|
||||
|
||||
lighting_css = """
|
||||
<style>
|
||||
#lighter_mesh canvas {
|
||||
filter: brightness(2.2) !important;
|
||||
}
|
||||
#visual_mesh canvas { filter: brightness(2.2) !important; }
|
||||
#collision_mesh_a canvas, #collision_mesh_b canvas { filter: brightness(1.0) !important; }
|
||||
</style>
|
||||
"""
|
||||
|
||||
_prev_temp = {}
|
||||
|
||||
# --- Helper Functions ---
|
||||
|
||||
def _unique_path(
|
||||
src_path: str | None, session_hash: str, kind: str
|
||||
) -> str | None:
|
||||
"""Link/copy src to GRADIO_TEMP_DIR/session_hash with random filename. Always return a fresh URL."""
|
||||
if not src_path:
|
||||
return None
|
||||
tmp_root = (
|
||||
Path(os.environ.get("GRADIO_TEMP_DIR", "/tmp"))
|
||||
/ "model3d-cache"
|
||||
/ session_hash
|
||||
)
|
||||
tmp_root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# rolling cleanup for same kind
|
||||
prev = _prev_temp.get(session_hash, {})
|
||||
old = prev.get(kind)
|
||||
if old and old.exists():
|
||||
old.unlink()
|
||||
|
||||
ext = Path(src_path).suffix or ".glb"
|
||||
dst = tmp_root / f"{kind}-{uuid.uuid4().hex}{ext}"
|
||||
shutil.copy2(src_path, dst)
|
||||
|
||||
prev[kind] = dst
|
||||
_prev_temp[session_hash] = prev
|
||||
return str(dst)
|
||||
|
||||
|
||||
# --- Helper Functions (data filtering) ---
|
||||
def get_primary_categories():
|
||||
return sorted(df["primary_category"].dropna().unique())
|
||||
|
||||
@ -81,7 +152,7 @@ def get_categories(primary, secondary):
|
||||
|
||||
def get_assets(primary, secondary, category):
|
||||
if not primary or not secondary:
|
||||
return [], gr.update(interactive=False)
|
||||
return [], gr.update(interactive=False), pd.DataFrame()
|
||||
|
||||
subset = df[
|
||||
(df["primary_category"] == primary)
|
||||
@ -105,79 +176,211 @@ def get_assets(primary, secondary, category):
|
||||
else "https://dummyimage.com/512x512/cccccc/000000&text=No+Preview"
|
||||
)
|
||||
|
||||
return items, gr.update(interactive=True)
|
||||
return items, gr.update(interactive=True), subset
|
||||
|
||||
|
||||
def show_asset_from_gallery(
|
||||
evt: gr.SelectData, primary: str, secondary: str, category: str
|
||||
):
|
||||
index = evt.index
|
||||
subset = df[
|
||||
(df["primary_category"] == primary)
|
||||
& (df["secondary_category"] == secondary)
|
||||
]
|
||||
if category:
|
||||
subset = subset[subset["category"] == category]
|
||||
def search_assets(query: str, top_k: int):
|
||||
if not GPT_AVAILABLE or not query:
|
||||
gr.Warning(
|
||||
"GPT client is not available or query is empty. Cannot perform search."
|
||||
)
|
||||
return [], gr.update(interactive=False), pd.DataFrame()
|
||||
|
||||
est_type_text = "N/A"
|
||||
est_height_text = "N/A"
|
||||
est_mass_text = "N/A"
|
||||
est_mu_text = "N/A"
|
||||
gr.Info(f"Searching for assets matching: '{query}'...")
|
||||
|
||||
if index >= len(subset):
|
||||
return (
|
||||
None,
|
||||
"Error: Selection index is out of bounds.",
|
||||
None,
|
||||
None,
|
||||
est_type_text,
|
||||
est_height_text,
|
||||
est_mass_text,
|
||||
est_mu_text,
|
||||
keywords = query.split()
|
||||
keyword_filter = pd.Series([False] * len(df), index=df.index)
|
||||
for keyword in keywords:
|
||||
keyword_filter |= df['description'].str.contains(
|
||||
keyword, case=False, na=False
|
||||
)
|
||||
|
||||
row = subset.iloc[index]
|
||||
candidates = df[keyword_filter]
|
||||
|
||||
if len(candidates) > 100:
|
||||
candidates = candidates.head(100)
|
||||
|
||||
if candidates.empty:
|
||||
gr.Warning("No assets found matching the keywords.")
|
||||
return [], gr.update(interactive=True), pd.DataFrame()
|
||||
|
||||
try:
|
||||
descriptions = [
|
||||
f"{idx}: {desc}" for idx, desc in candidates['description'].items()
|
||||
]
|
||||
descriptions_text = "\n".join(descriptions)
|
||||
|
||||
prompt = f"""
|
||||
A user is searching for 3D assets with the query: "{query}".
|
||||
Below is a list of available assets, each with an ID and a description.
|
||||
Please evaluate how well each asset description matches the user's query and rate them on a scale from 0 to 10, where 10 is a perfect match.
|
||||
|
||||
Your task is to return a list of the top {top_k} asset IDs, sorted from the most relevant to the least relevant.
|
||||
The output format must be a simple comma-separated list of IDs, for example: "123,45,678". Do not add any other text.
|
||||
|
||||
Asset Descriptions:
|
||||
{descriptions_text}
|
||||
|
||||
User Query: "{query}"
|
||||
|
||||
Top {top_k} sorted asset IDs:
|
||||
"""
|
||||
response = gpt_client.query(prompt)
|
||||
sorted_ids_str = response.strip().split(',')
|
||||
sorted_ids = [
|
||||
int(id_str.strip())
|
||||
for id_str in sorted_ids_str
|
||||
if id_str.strip().isdigit()
|
||||
]
|
||||
top_assets = df.loc[sorted_ids].head(top_k)
|
||||
except Exception as e:
|
||||
gr.Error(f"An error occurred while using GPT for ranking: {e}")
|
||||
top_assets = candidates.head(top_k)
|
||||
|
||||
items = []
|
||||
for row in top_assets.itertuples():
|
||||
asset_dir = os.path.join(DATA_ROOT, row.asset_dir)
|
||||
video_path = None
|
||||
if pd.notna(row.asset_dir) and os.path.exists(asset_dir):
|
||||
for f in os.listdir(asset_dir):
|
||||
if f.lower().endswith(".mp4"):
|
||||
video_path = os.path.join(asset_dir, f)
|
||||
break
|
||||
items.append(
|
||||
video_path
|
||||
if video_path
|
||||
else "https://dummyimage.com/512x512/cccccc/000000&text=No+Preview"
|
||||
)
|
||||
|
||||
gr.Info(f"Found {len(items)} assets.")
|
||||
return items, gr.update(interactive=True), top_assets
|
||||
|
||||
|
||||
# --- Mesh extraction ---
|
||||
def _extract_mesh_paths(row) -> Tuple[str | None, str | None, str]:
|
||||
desc = row["description"]
|
||||
urdf_path = os.path.join(DATA_ROOT, row["urdf_path"])
|
||||
asset_dir = os.path.join(DATA_ROOT, row["asset_dir"])
|
||||
mesh_to_display = None
|
||||
visual_mesh_path = None
|
||||
collision_mesh_path = None
|
||||
|
||||
if pd.notna(urdf_path) and os.path.exists(urdf_path):
|
||||
try:
|
||||
tree = ET.parse(urdf_path)
|
||||
root = tree.getroot()
|
||||
|
||||
mesh_element = root.find('.//visual/geometry/mesh')
|
||||
if mesh_element is not None:
|
||||
mesh_filename = mesh_element.get('filename')
|
||||
if mesh_filename:
|
||||
glb_filename = os.path.splitext(mesh_filename)[0] + ".glb"
|
||||
visual_mesh_element = root.find('.//visual/geometry/mesh')
|
||||
if visual_mesh_element is not None:
|
||||
visual_mesh_filename = visual_mesh_element.get('filename')
|
||||
if visual_mesh_filename:
|
||||
glb_filename = (
|
||||
os.path.splitext(visual_mesh_filename)[0] + ".glb"
|
||||
)
|
||||
potential_path = os.path.join(asset_dir, glb_filename)
|
||||
if os.path.exists(potential_path):
|
||||
mesh_to_display = potential_path
|
||||
visual_mesh_path = potential_path
|
||||
|
||||
category_elem = root.find('.//extra_info/category')
|
||||
if category_elem is not None and category_elem.text:
|
||||
est_type_text = category_elem.text.strip()
|
||||
|
||||
height_elem = root.find('.//extra_info/real_height')
|
||||
if height_elem is not None and height_elem.text:
|
||||
est_height_text = height_elem.text.strip()
|
||||
|
||||
mass_elem = root.find('.//extra_info/min_mass')
|
||||
if mass_elem is not None and mass_elem.text:
|
||||
est_mass_text = mass_elem.text.strip()
|
||||
|
||||
mu_elem = root.find('.//collision/gazebo/mu2')
|
||||
if mu_elem is not None and mu_elem.text:
|
||||
est_mu_text = mu_elem.text.strip()
|
||||
collision_mesh_element = root.find('.//collision/geometry/mesh')
|
||||
if collision_mesh_element is not None:
|
||||
collision_mesh_filename = collision_mesh_element.get(
|
||||
'filename'
|
||||
)
|
||||
if collision_mesh_filename:
|
||||
potential_collision_path = os.path.join(
|
||||
asset_dir, collision_mesh_filename
|
||||
)
|
||||
if os.path.exists(potential_collision_path):
|
||||
collision_mesh_path = potential_collision_path
|
||||
|
||||
except ET.ParseError:
|
||||
desc = f"Error: Failed to parse URDF at {urdf_path}. {desc}"
|
||||
except Exception as e:
|
||||
desc = f"An error occurred while processing URDF: {str(e)}. {desc}"
|
||||
|
||||
return visual_mesh_path, collision_mesh_path, desc
|
||||
|
||||
|
||||
def show_asset_from_gallery(
|
||||
evt: gr.SelectData,
|
||||
primary: str,
|
||||
secondary: str,
|
||||
category: str,
|
||||
search_query: str,
|
||||
gallery_df: pd.DataFrame,
|
||||
):
|
||||
"""Parse the selected asset and return raw paths + metadata."""
|
||||
index = evt.index
|
||||
|
||||
if search_query and gallery_df is not None and not gallery_df.empty:
|
||||
subset = gallery_df
|
||||
else:
|
||||
if not primary or not secondary:
|
||||
return (
|
||||
None, # visual_path
|
||||
None, # collision_path
|
||||
"Error: Primary or secondary category not selected.",
|
||||
None, # asset_dir
|
||||
None, # urdf_path
|
||||
"N/A",
|
||||
"N/A",
|
||||
"N/A",
|
||||
"N/A",
|
||||
)
|
||||
|
||||
subset = df[
|
||||
(df["primary_category"] == primary)
|
||||
& (df["secondary_category"] == secondary)
|
||||
]
|
||||
if category:
|
||||
subset = subset[subset["category"] == category]
|
||||
|
||||
if subset.empty or index >= len(subset):
|
||||
return (
|
||||
None,
|
||||
None,
|
||||
"Error: Selection index is out of bounds or data is missing.",
|
||||
None,
|
||||
None,
|
||||
"N/A",
|
||||
"N/A",
|
||||
"N/A",
|
||||
"N/A",
|
||||
)
|
||||
|
||||
row = subset.iloc[index]
|
||||
visual_path, collision_path, desc = _extract_mesh_paths(row)
|
||||
|
||||
urdf_path = os.path.join(DATA_ROOT, row["urdf_path"])
|
||||
asset_dir = os.path.join(DATA_ROOT, row["asset_dir"])
|
||||
|
||||
# read extra info
|
||||
est_type_text = "N/A"
|
||||
est_height_text = "N/A"
|
||||
est_mass_text = "N/A"
|
||||
est_mu_text = "N/A"
|
||||
|
||||
if pd.notna(urdf_path) and os.path.exists(urdf_path):
|
||||
try:
|
||||
tree = ET.parse(urdf_path)
|
||||
root = tree.getroot()
|
||||
category_elem = root.find('.//extra_info/category')
|
||||
if category_elem is not None and category_elem.text:
|
||||
est_type_text = category_elem.text.strip()
|
||||
height_elem = root.find('.//extra_info/real_height')
|
||||
if height_elem is not None and height_elem.text:
|
||||
est_height_text = height_elem.text.strip()
|
||||
mass_elem = root.find('.//extra_info/min_mass')
|
||||
if mass_elem is not None and mass_elem.text:
|
||||
est_mass_text = mass_elem.text.strip()
|
||||
mu_elem = root.find('.//collision/gazebo/mu2')
|
||||
if mu_elem is not None and mu_elem.text:
|
||||
est_mu_text = mu_elem.text.strip()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return (
|
||||
gr.update(value=mesh_to_display),
|
||||
visual_path,
|
||||
collision_path,
|
||||
desc,
|
||||
asset_dir,
|
||||
urdf_path,
|
||||
@ -188,6 +391,56 @@ def show_asset_from_gallery(
|
||||
)
|
||||
|
||||
|
||||
def render_meshes(
|
||||
visual_path: str | None,
|
||||
collision_path: str | None,
|
||||
switch_viewer: bool,
|
||||
req: gr.Request,
|
||||
):
|
||||
session_hash = getattr(req, "session_hash", "default")
|
||||
|
||||
if switch_viewer:
|
||||
yield (
|
||||
gr.update(value=None),
|
||||
gr.update(value=None, visible=False),
|
||||
gr.update(value=None, visible=True),
|
||||
True,
|
||||
)
|
||||
else:
|
||||
yield (
|
||||
gr.update(value=None),
|
||||
gr.update(value=None, visible=True),
|
||||
gr.update(value=None, visible=False),
|
||||
True,
|
||||
)
|
||||
|
||||
visual_unique = (
|
||||
_unique_path(visual_path, session_hash, "visual")
|
||||
if visual_path
|
||||
else None
|
||||
)
|
||||
collision_unique = (
|
||||
_unique_path(collision_path, session_hash, "collision")
|
||||
if collision_path
|
||||
else None
|
||||
)
|
||||
|
||||
if switch_viewer:
|
||||
yield (
|
||||
gr.update(value=visual_unique),
|
||||
gr.update(value=None, visible=False),
|
||||
gr.update(value=collision_unique, visible=True),
|
||||
False,
|
||||
)
|
||||
else:
|
||||
yield (
|
||||
gr.update(value=visual_unique),
|
||||
gr.update(value=collision_unique, visible=True),
|
||||
gr.update(value=None, visible=False),
|
||||
True,
|
||||
)
|
||||
|
||||
|
||||
def create_asset_zip(asset_dir: str, req: gr.Request):
|
||||
user_dir = os.path.join(TMP_DIR, str(req.session_hash))
|
||||
os.makedirs(user_dir, exist_ok=True)
|
||||
@ -214,7 +467,7 @@ def end_session(req: gr.Request) -> None:
|
||||
shutil.rmtree(user_dir)
|
||||
|
||||
|
||||
# --- Gradio UI Definition ---
|
||||
# --- UI ---
|
||||
with gr.Blocks(
|
||||
theme=custom_theme,
|
||||
css=css,
|
||||
@ -256,9 +509,35 @@ with gr.Blocks(
|
||||
category_list = get_categories(primary_val, secondary_val)
|
||||
category_val = category_list[0] if category_list else None
|
||||
asset_folder = gr.State(value=None)
|
||||
gallery_df_state = gr.State()
|
||||
|
||||
switch_viewer_state = gr.State(value=False)
|
||||
|
||||
with gr.Row(equal_height=False):
|
||||
with gr.Column(scale=1, min_width=350):
|
||||
with gr.Group():
|
||||
gr.Markdown("### Search Asset with Descriptions")
|
||||
search_box = gr.Textbox(
|
||||
label="🔎 Enter your search query",
|
||||
placeholder="e.g., 'a red chair with four legs'",
|
||||
interactive=GPT_AVAILABLE,
|
||||
)
|
||||
top_k_slider = gr.Slider(
|
||||
minimum=1,
|
||||
maximum=50,
|
||||
value=10,
|
||||
step=1,
|
||||
label="Number of results",
|
||||
interactive=GPT_AVAILABLE,
|
||||
)
|
||||
search_button = gr.Button(
|
||||
"Search", variant="primary", interactive=GPT_AVAILABLE
|
||||
)
|
||||
if not GPT_AVAILABLE:
|
||||
gr.Markdown(
|
||||
"<p style='color: #ff4b4b;'>⚠️ GPT client not available. Search is disabled.</p>"
|
||||
)
|
||||
|
||||
with gr.Group():
|
||||
gr.Markdown("### Select Asset Category")
|
||||
primary = gr.Dropdown(
|
||||
@ -278,10 +557,11 @@ with gr.Blocks(
|
||||
)
|
||||
|
||||
with gr.Group():
|
||||
initial_assets, _, initial_df = get_assets(
|
||||
primary_val, secondary_val, category_val
|
||||
)
|
||||
gallery = gr.Gallery(
|
||||
value=get_assets(primary_val, secondary_val, category_val)[
|
||||
0
|
||||
],
|
||||
value=initial_assets,
|
||||
label="🖼️ Asset Previews",
|
||||
columns=3,
|
||||
height="auto",
|
||||
@ -292,14 +572,40 @@ with gr.Blocks(
|
||||
|
||||
with gr.Column(scale=2, min_width=500):
|
||||
with gr.Group():
|
||||
viewer = gr.Model3D(
|
||||
label="🧊 3D Model Viewer",
|
||||
height=500,
|
||||
clear_color=[0.95, 0.95, 0.95],
|
||||
elem_id="lighter_mesh",
|
||||
with gr.Tabs():
|
||||
with gr.TabItem("Visual Mesh") as t1:
|
||||
viewer = gr.Model3D(
|
||||
label="🧊 3D Model Viewer",
|
||||
height=500,
|
||||
clear_color=[0.95, 0.95, 0.95],
|
||||
elem_id="visual_mesh",
|
||||
)
|
||||
with gr.TabItem("Collision Mesh") as t2:
|
||||
collision_viewer_a = gr.Model3D(
|
||||
label="🧊 Collision Mesh",
|
||||
height=500,
|
||||
clear_color=[0.95, 0.95, 0.95],
|
||||
elem_id="collision_mesh_a",
|
||||
visible=True,
|
||||
)
|
||||
collision_viewer_b = gr.Model3D(
|
||||
label="🧊 Collision Mesh",
|
||||
height=500,
|
||||
clear_color=[0.95, 0.95, 0.95],
|
||||
elem_id="collision_mesh_b",
|
||||
visible=False,
|
||||
)
|
||||
|
||||
t1.select(
|
||||
fn=lambda: None,
|
||||
js="() => { window.dispatchEvent(new Event('resize')); }",
|
||||
)
|
||||
t2.select(
|
||||
fn=lambda: None,
|
||||
js="() => { window.dispatchEvent(new Event('resize')); }",
|
||||
)
|
||||
|
||||
with gr.Row():
|
||||
# TODO: Add more asset details if needed
|
||||
est_type_text = gr.Textbox(
|
||||
label="Asset category", interactive=False
|
||||
)
|
||||
@ -312,10 +618,11 @@ with gr.Blocks(
|
||||
est_mu_text = gr.Textbox(
|
||||
label="Friction coefficient", interactive=False
|
||||
)
|
||||
with gr.Accordion(label="Asset Details", open=False):
|
||||
with gr.Row():
|
||||
desc_box = gr.Textbox(
|
||||
label="📝 Asset Description", interactive=False
|
||||
)
|
||||
with gr.Accordion(label="Asset Details", open=False):
|
||||
urdf_file = gr.Textbox(
|
||||
label="URDF File Path", interactive=False, lines=2
|
||||
)
|
||||
@ -331,55 +638,64 @@ with gr.Blocks(
|
||||
interactive=False,
|
||||
)
|
||||
|
||||
search_button.click(
|
||||
fn=search_assets,
|
||||
inputs=[search_box, top_k_slider],
|
||||
outputs=[gallery, gallery, gallery_df_state],
|
||||
)
|
||||
search_box.submit(
|
||||
fn=search_assets,
|
||||
inputs=[search_box, top_k_slider],
|
||||
outputs=[gallery, gallery, gallery_df_state],
|
||||
)
|
||||
|
||||
def update_on_primary_change(p):
|
||||
s_choices = get_secondary_categories(p)
|
||||
initial_assets, gallery_update, initial_df = get_assets(p, None, None)
|
||||
return (
|
||||
gr.update(choices=s_choices, value=None),
|
||||
gr.update(choices=[], value=None),
|
||||
[],
|
||||
gr.update(interactive=False),
|
||||
initial_assets,
|
||||
gallery_update,
|
||||
initial_df,
|
||||
)
|
||||
|
||||
def update_on_secondary_change(p, s):
|
||||
c_choices = get_categories(p, s)
|
||||
return (
|
||||
gr.update(choices=c_choices, value=None),
|
||||
[],
|
||||
gr.update(interactive=False),
|
||||
)
|
||||
|
||||
def update_on_secondary_change(p, s):
|
||||
c_choices = get_categories(p, s)
|
||||
asset_previews, gallery_update = get_assets(p, s, None)
|
||||
asset_previews, gallery_update, gallery_df = get_assets(p, s, None)
|
||||
return (
|
||||
gr.update(choices=c_choices, value=None),
|
||||
asset_previews,
|
||||
gallery_update,
|
||||
gallery_df,
|
||||
)
|
||||
|
||||
def update_assets(p, s, c):
|
||||
asset_previews, gallery_update, gallery_df = get_assets(p, s, c)
|
||||
return asset_previews, gallery_update, gallery_df
|
||||
|
||||
primary.change(
|
||||
fn=update_on_primary_change,
|
||||
inputs=[primary],
|
||||
outputs=[secondary, category, gallery, gallery],
|
||||
outputs=[secondary, category, gallery, gallery, gallery_df_state],
|
||||
)
|
||||
|
||||
secondary.change(
|
||||
fn=update_on_secondary_change,
|
||||
inputs=[primary, secondary],
|
||||
outputs=[category, gallery, gallery],
|
||||
outputs=[category, gallery, gallery, gallery_df_state],
|
||||
)
|
||||
|
||||
category.change(
|
||||
fn=get_assets,
|
||||
fn=update_assets,
|
||||
inputs=[primary, secondary, category],
|
||||
outputs=[gallery, gallery],
|
||||
outputs=[gallery, gallery, gallery_df_state],
|
||||
)
|
||||
|
||||
gallery.select(
|
||||
fn=show_asset_from_gallery,
|
||||
inputs=[primary, secondary, category],
|
||||
inputs=[primary, secondary, category, search_box, gallery_df_state],
|
||||
outputs=[
|
||||
viewer,
|
||||
(visual_path_state := gr.State()),
|
||||
(collision_path_state := gr.State()),
|
||||
desc_box,
|
||||
asset_folder,
|
||||
urdf_file,
|
||||
@ -388,22 +704,23 @@ with gr.Blocks(
|
||||
est_mass_text,
|
||||
est_mu_text,
|
||||
],
|
||||
).then(
|
||||
fn=render_meshes,
|
||||
inputs=[visual_path_state, collision_path_state, switch_viewer_state],
|
||||
outputs=[
|
||||
viewer,
|
||||
collision_viewer_a,
|
||||
collision_viewer_b,
|
||||
switch_viewer_state,
|
||||
],
|
||||
).success(
|
||||
lambda: tuple(
|
||||
[
|
||||
gr.Button(interactive=True),
|
||||
gr.Button(interactive=False),
|
||||
]
|
||||
),
|
||||
lambda: (gr.Button(interactive=True), gr.Button(interactive=False)),
|
||||
outputs=[extract_btn, download_btn],
|
||||
)
|
||||
|
||||
extract_btn.click(
|
||||
fn=create_asset_zip, inputs=[asset_folder], outputs=[download_btn]
|
||||
).success(
|
||||
fn=lambda: gr.update(interactive=True),
|
||||
outputs=download_btn,
|
||||
)
|
||||
).success(fn=lambda: gr.update(interactive=True), outputs=download_btn)
|
||||
|
||||
demo.load(start_session)
|
||||
demo.unload(end_session)
|
||||
@ -411,7 +728,7 @@ with gr.Blocks(
|
||||
|
||||
if __name__ == "__main__":
|
||||
demo.launch(
|
||||
server_name="10.34.8.82",
|
||||
server_name="10.34.8.77",
|
||||
server_port=8088,
|
||||
allowed_paths=[
|
||||
"/horizon-bucket/robot_lab/datasets/embodiedgen/assets"
|
||||
|
||||
28
docs/acknowledgement.md
Normal file
28
docs/acknowledgement.md
Normal file
@ -0,0 +1,28 @@
|
||||
# 🙌 Acknowledgement
|
||||
|
||||
EmbodiedGen builds upon the following amazing projects and models:
|
||||
🌟 [Trellis](https://github.com/microsoft/TRELLIS) | 🌟 [Hunyuan-Delight](https://huggingface.co/tencent/Hunyuan3D-2/tree/main/hunyuan3d-delight-v2-0) | 🌟 [Segment Anything](https://github.com/facebookresearch/segment-anything) | 🌟 [Rembg](https://github.com/danielgatis/rembg) | 🌟 [RMBG-1.4](https://huggingface.co/briaai/RMBG-1.4) | 🌟 [Stable Diffusion x4](https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler) | 🌟 [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN) | 🌟 [Kolors](https://github.com/Kwai-Kolors/Kolors) | 🌟 [ChatGLM3](https://github.com/THUDM/ChatGLM3) | 🌟 [Aesthetic Score](http://captions.christoph-schuhmann.de/aesthetic_viz_laion_sac+logos+ava1-l14-linearMSE-en-2.37B.html) | 🌟 [Pano2Room](https://github.com/TrickyGo/Pano2Room) | 🌟 [Diffusion360](https://github.com/ArcherFMY/SD-T2I-360PanoImage) | 🌟 [Kaolin](https://github.com/NVIDIAGameWorks/kaolin) | 🌟 [diffusers](https://github.com/huggingface/diffusers) | 🌟 [gsplat](https://github.com/nerfstudio-project/gsplat) | 🌟 [QWEN-2.5VL](https://github.com/QwenLM/Qwen2.5-VL) | 🌟 [GPT4o](https://platform.openai.com/docs/models/gpt-4o) | 🌟 [SD3.5](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium) | 🌟 [ManiSkill](https://github.com/haosulab/ManiSkill)
|
||||
|
||||
---
|
||||
|
||||
## 📚 Citation
|
||||
|
||||
If you use EmbodiedGen in your research or projects, please cite:
|
||||
|
||||
```bibtex
|
||||
@misc{wang2025embodiedgengenerative3dworld,
|
||||
title={EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligence},
|
||||
author={Xinjie Wang and Liu Liu and Yu Cao and Ruiqi Wu and Wenkang Qin and Dehui Wang and Wei Sui and Zhizhong Su},
|
||||
year={2025},
|
||||
eprint={2506.10600},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.RO},
|
||||
url={https://arxiv.org/abs/2506.10600},
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚖️ License
|
||||
|
||||
This project is licensed under the [Apache License 2.0](LICENSE). See the `LICENSE` file for details.
|
||||
25
docs/api/data.md
Normal file
25
docs/api/data.md
Normal file
@ -0,0 +1,25 @@
|
||||
# Data API
|
||||
|
||||
::: embodied_gen.data.asset_converter
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.data.datasets
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.data.differentiable_render
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.data.mesh_operator
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.data.backproject_v2
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.data.convex_decomposer
|
||||
options:
|
||||
heading_level: 3
|
||||
7
docs/api/envs.md
Normal file
7
docs/api/envs.md
Normal file
@ -0,0 +1,7 @@
|
||||
# Envs API
|
||||
|
||||
Documentation for simulation environments and task definitions.
|
||||
|
||||
::: embodied_gen.envs.pick_embodiedgen
|
||||
options:
|
||||
heading_level: 3
|
||||
14
docs/api/index.md
Normal file
14
docs/api/index.md
Normal file
@ -0,0 +1,14 @@
|
||||
# API Reference
|
||||
|
||||
Welcome to the API reference for EmbodiedGen.
|
||||
|
||||
This section contains detailed documentation for all public modules, classes,
|
||||
and functions. Use the navigation on the left (or the list below) to
|
||||
browse the different components.
|
||||
|
||||
* [**Data API**](data.md): Tools for data processing, conversion, and rendering.
|
||||
* [**Envs API**](envs.md): Simulation environment definitions.
|
||||
* [**Models API**](models.md): The core generative models (Texture, 3DGS, Layout, etc.).
|
||||
* [**Trainer API**](trainer.md): PyTorch-Lightning style trainers for models.
|
||||
* [**Utilities API**](utils.md): Helper functions and configuration.
|
||||
* [**Validators API**](validators.md): Tools for checking and validating assets.
|
||||
33
docs/api/models.md
Normal file
33
docs/api/models.md
Normal file
@ -0,0 +1,33 @@
|
||||
# Models API
|
||||
|
||||
::: embodied_gen.models.texture_model
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.models.gs_model
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.models.layout
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.models.text_model
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.models.sr_model
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.models.segment_model
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.models.image_comm_model
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.models.delight_model
|
||||
options:
|
||||
heading_level: 3
|
||||
11
docs/api/trainer.md
Normal file
11
docs/api/trainer.md
Normal file
@ -0,0 +1,11 @@
|
||||
# Trainer API
|
||||
|
||||
This section covers the training pipelines for various models.
|
||||
|
||||
::: embodied_gen.trainer.gsplat_trainer
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.trainer.pono2mesh_trainer
|
||||
options:
|
||||
heading_level: 3
|
||||
47
docs/api/utils.md
Normal file
47
docs/api/utils.md
Normal file
@ -0,0 +1,47 @@
|
||||
# Utilities API
|
||||
|
||||
General-purpose utility functions, configuration, and helper classes.
|
||||
|
||||
::: embodied_gen.utils.config
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.utils.log
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.utils.enum
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.utils.geometry
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.utils.gaussian
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.utils.gpt_clients
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.utils.process_media
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.utils.simulation
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.utils.tags
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.utils.trender
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.utils.monkey_patches
|
||||
options:
|
||||
heading_level: 3
|
||||
15
docs/api/validators.md
Normal file
15
docs/api/validators.md
Normal file
@ -0,0 +1,15 @@
|
||||
# Validators API
|
||||
|
||||
Tools for asset validation, quality control, and conversion.
|
||||
|
||||
::: embodied_gen.validators.aesthetic_predictor
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.validators.quality_checkers
|
||||
options:
|
||||
heading_level: 3
|
||||
|
||||
::: embodied_gen.validators.urdf_convertor
|
||||
options:
|
||||
heading_level: 3
|
||||
|
Before Width: | Height: | Size: 471 KiB After Width: | Height: | Size: 471 KiB |
|
Before Width: | Height: | Size: 875 KiB After Width: | Height: | Size: 875 KiB |
|
Before Width: | Height: | Size: 809 KiB After Width: | Height: | Size: 809 KiB |
|
Before Width: | Height: | Size: 36 KiB After Width: | Height: | Size: 36 KiB |
|
Before Width: | Height: | Size: 684 KiB After Width: | Height: | Size: 684 KiB |
|
Before Width: | Height: | Size: 713 KiB After Width: | Height: | Size: 713 KiB |
|
Before Width: | Height: | Size: 642 KiB After Width: | Height: | Size: 642 KiB |
|
Before Width: | Height: | Size: 699 KiB After Width: | Height: | Size: 699 KiB |
|
Before Width: | Height: | Size: 236 KiB After Width: | Height: | Size: 236 KiB |
|
Before Width: | Height: | Size: 771 KiB After Width: | Height: | Size: 771 KiB |
|
Before Width: | Height: | Size: 653 KiB After Width: | Height: | Size: 653 KiB |
|
Before Width: | Height: | Size: 712 KiB After Width: | Height: | Size: 712 KiB |
|
Before Width: | Height: | Size: 2.8 MiB After Width: | Height: | Size: 2.8 MiB |
|
Before Width: | Height: | Size: 3.3 MiB After Width: | Height: | Size: 3.3 MiB |
|
Before Width: | Height: | Size: 926 KiB After Width: | Height: | Size: 926 KiB |
|
Before Width: | Height: | Size: 47 KiB After Width: | Height: | Size: 47 KiB |
|
Before Width: | Height: | Size: 76 KiB After Width: | Height: | Size: 76 KiB |
23
docs/index.md
Normal file
23
docs/index.md
Normal file
@ -0,0 +1,23 @@
|
||||
---
|
||||
hide:
|
||||
- navigation
|
||||
---
|
||||
|
||||
# 👋 Welcome to EmbodiedGen
|
||||
|
||||
[](https://horizonrobotics.github.io/robot_lab/embodied_gen/index.html)
|
||||
[](https://arxiv.org/abs/2506.10600)
|
||||
[](https://www.youtube.com/watch?v=rG4odybuJRk)
|
||||
[](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Image-to-3D)
|
||||
[](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Text-to-3D)
|
||||
[](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Texture-Gen)
|
||||
[](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Gallery-Explorer)
|
||||
[](https://mp.weixin.qq.com/s/HH1cPBhK2xcDbyCK4BBTbw)
|
||||
|
||||
*EmbodiedGen*: Towards a Generative 3D World Engine for Embodied Intelligence
|
||||
|
||||
<img src="assets/overall.jpg" alt="Overall Framework" width="700"/>
|
||||
|
||||
> ***EmbodiedGen*** is a generative engine to create diverse and interactive 3D worlds composed of high-quality 3D assets(mesh & 3DGS) with plausible physics, leveraging generative AI to address the challenges of generalization in embodied intelligence related research.
|
||||
|
||||
---
|
||||
38
docs/install.md
Normal file
38
docs/install.md
Normal file
@ -0,0 +1,38 @@
|
||||
---
|
||||
hide:
|
||||
- navigation
|
||||
---
|
||||
|
||||
## ✅ Setup Environment
|
||||
```sh
|
||||
git clone https://github.com/HorizonRobotics/EmbodiedGen.git
|
||||
cd EmbodiedGen
|
||||
git checkout v0.1.5
|
||||
git submodule update --init --recursive --progress
|
||||
conda create -n embodiedgen python=3.10.13 -y # recommended to use a new env.
|
||||
conda activate embodiedgen
|
||||
bash install.sh basic
|
||||
```
|
||||
|
||||
## ✅ Starting from Docker
|
||||
|
||||
We provide a pre-built Docker image on [Docker Hub](https://hub.docker.com/repository/docker/wangxinjie/embodiedgen) with a configured environment for your convenience. For more details, please refer to [Docker documentation](https://github.com/HorizonRobotics/EmbodiedGen/tree/master/docker).
|
||||
|
||||
> **Note:** Model checkpoints are not included in the image, they will be automatically downloaded on first run. You still need to set up the GPT Agent manually.
|
||||
|
||||
```sh
|
||||
IMAGE=wangxinjie/embodiedgen:env_v0.1.x
|
||||
CONTAINER=EmbodiedGen-docker-${USER}
|
||||
docker pull ${IMAGE}
|
||||
docker run -itd --shm-size="64g" --gpus all --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --privileged --net=host --name ${CONTAINER} ${IMAGE}
|
||||
docker exec -it ${CONTAINER} bash
|
||||
```
|
||||
|
||||
## ✅ Setup GPT Agent
|
||||
|
||||
Update the API key in file: `embodied_gen/utils/gpt_config.yaml`.
|
||||
|
||||
You can choose between two backends for the GPT agent:
|
||||
|
||||
- **`gpt-4o`** (Recommended) – Use this if you have access to **Azure OpenAI**.
|
||||
- **`qwen2.5-vl`** – An alternative with free usage via OpenRouter, apply a free key [here](https://openrouter.ai/settings/keys) and update `api_key` in `embodied_gen/utils/gpt_config.yaml` (50 free requests per day)
|
||||
97
docs/services/image_to_3d.md
Normal file
97
docs/services/image_to_3d.md
Normal file
@ -0,0 +1,97 @@
|
||||
<script type="module" src="https://unpkg.com/@google/model-viewer/dist/model-viewer.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.js"></script>
|
||||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.css">
|
||||
<script>
|
||||
document.addEventListener('DOMContentLoaded', function () {
|
||||
const swiper = new Swiper('.swiper1', {
|
||||
loop: true,
|
||||
slidesPerView: 3,
|
||||
spaceBetween: 20,
|
||||
navigation: {
|
||||
nextEl: '.swiper-button-next',
|
||||
prevEl: '.swiper-button-prev',
|
||||
},
|
||||
centeredSlides: false,
|
||||
noSwiping: true,
|
||||
noSwipingClass: 'swiper-no-swiping',
|
||||
watchSlidesProgress: true,
|
||||
});
|
||||
});
|
||||
</script>
|
||||
|
||||
|
||||
# 🖼️ Image-to-3D Service
|
||||
[](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Image-to-3D)
|
||||
|
||||
This service launches a web application to generate physically plausible 3D asset URDF from single input image, offering high-quality support for digital twin systems.
|
||||
|
||||
<div class="swiper swiper1" style="max-width: 1000px; margin: 20px auto; border-radius: 12px;">
|
||||
<div class="swiper-wrapper">
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/image/astronaut.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/image/robot_i.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/image/desk.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/image/chair.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/image/desk2.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
</div>
|
||||
<div class="swiper-button-prev"></div>
|
||||
<div class="swiper-button-next"></div>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## ☁️ Run the App Service
|
||||
|
||||
!!! note "Note"
|
||||
Gradio servive is a simplified demonstration. For the full functionality, please refer to [img3d-cli](../tutorials/image_to_3d.md).
|
||||
|
||||
Run the image-to-3D generation service locally. Models are automatically downloaded on first run, please be patient.
|
||||
|
||||
```sh
|
||||
# Run in foreground
|
||||
python apps/image_to_3d.py
|
||||
|
||||
# Or run in the background
|
||||
CUDA_VISIBLE_DEVICES=0 nohup python apps/image_to_3d.py > /dev/null 2>&1 &
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
!!! tip "Getting Started"
|
||||
- Try it directly online via our [Hugging Face Space](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Image-to-3D) — no installation required.
|
||||
- Explore EmbodiedGen generated sim-ready [Assets Gallery](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Gallery-Explorer).
|
||||
- For instructions on using the generated asset in any simulator, see [Any Simulators Tutorial](tutorials/any_simulators.md).
|
||||
46
docs/services/index.md
Normal file
46
docs/services/index.md
Normal file
@ -0,0 +1,46 @@
|
||||
# Interactive 3D Generation & Visualization Services
|
||||
|
||||
EmbodiedGen provides a suite of **interactive services** that transform images and text into **physically plausible, simulator-ready 3D assets**.
|
||||
Each service is optimized for visual quality, simulation compatibility, and scalability — making it easy to create, edit, and explore assets for **digital twin**, **robotic simulation**, and **AI embodiment** scenarios.
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Prerequisites
|
||||
|
||||
!!! tip "Prerequisites"
|
||||
Make sure to finish the [Installation Guide](../install.md) before launching any service. Missing dependencies will cause initialization errors. Model weights are automatically downloaded on first run.
|
||||
|
||||
---
|
||||
|
||||
## 🧩 Overview of Available Services
|
||||
|
||||
| Service | Description |
|
||||
|----------|--------------|
|
||||
| [🖼️ **Image to 3D**](image_to_3d.md) | Generate physically plausible 3D asset URDF from single input image, offering high-quality support for digital twin systems. |
|
||||
| [📝 **Text to 3D**](text_to_3d.md) | Generate physically plausible 3D assets from text descriptions for a wide range of geometry and styles. |
|
||||
| [🎨 **Texture Edit**](texture_edit.md) | Generate visually rich textures for existing 3D meshes. |
|
||||
| [📸 **Asset Gallery**](visualize_asset.md) | Explore and download EmbodiedGen All-Simulators-Ready Assets. |
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ How to Run Locally
|
||||
|
||||
!!! tip "Quick Start"
|
||||
Each service can be launched directly as a local Gradio app:
|
||||
```bash
|
||||
# Example: Run the Image-to-3D service
|
||||
python apps/image_to_3d.py
|
||||
```
|
||||
|
||||
Models are automatically downloaded on first run. For full CLI usage, please check the corresponding [tutorials](../tutorials/index.md).
|
||||
|
||||
---
|
||||
|
||||
## 🧭 Next Steps
|
||||
|
||||
- [📘 Tutorials](../tutorials/index.md) – Learn how to use EmbodiedGen in generating interactive 3D scenes for embodied intelligence.
|
||||
- [🧱 API Reference](../api/index.md) – Integrate EmbodiedGen code programmatically.
|
||||
|
||||
---
|
||||
|
||||
> 💡 *EmbodiedGen bridges the gap between AI-driven 3D generation and physically grounded simulation, enabling true embodiment for intelligent agents.*
|
||||
111
docs/services/text_to_3d.md
Normal file
111
docs/services/text_to_3d.md
Normal file
@ -0,0 +1,111 @@
|
||||
<script type="module" src="https://unpkg.com/@google/model-viewer/dist/model-viewer.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.js"></script>
|
||||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.css">
|
||||
<script>
|
||||
document.addEventListener('DOMContentLoaded', function () {
|
||||
const swiper = new Swiper('.swiper1', {
|
||||
loop: true,
|
||||
slidesPerView: 3,
|
||||
spaceBetween: 20,
|
||||
navigation: {
|
||||
nextEl: '.swiper-button-next',
|
||||
prevEl: '.swiper-button-prev',
|
||||
},
|
||||
centeredSlides: false,
|
||||
noSwiping: true,
|
||||
noSwipingClass: 'swiper-no-swiping',
|
||||
watchSlidesProgress: true,
|
||||
});
|
||||
});
|
||||
</script>
|
||||
|
||||
|
||||
# 📝 Text-to-3D Service
|
||||
|
||||
[](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Text-to-3D)
|
||||
|
||||
This service launches a web application to generate physically plausible 3D assets from text descriptions for a wide range of geometry and styles.
|
||||
|
||||
<div class="swiper swiper1" style="max-width: 1000px; margin: 20px auto; border-radius: 12px;">
|
||||
<div class="swiper-wrapper">
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/text/c2.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
background-color="#ffffff"
|
||||
style="display:block; width: 100%; height: 160px; border-radius: 12px;"
|
||||
>
|
||||
</model-viewer>
|
||||
<p style="text-align: center; margin-top: 8px; font-size: 14px;">"Antique brass key, intricate filigree"</p>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/text/c3.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
background-color="#ffffff"
|
||||
style="display:block; width: 100%; height: 160px;">
|
||||
</model-viewer>
|
||||
<p style="text-align: center; margin-top: 8px; font-size: 14px;">"Rusty old wrench, peeling paint"</p>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/text/c4.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
background-color="#ffffff"
|
||||
style="display:block; width: 100%; height: 160px;">
|
||||
</model-viewer>
|
||||
<p style="text-align: center; margin-top: 8px; font-size: 14px;">"Sleek black drone, red sensors"</p>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/text/c7.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
background-color="#ffffff"
|
||||
style="display:block; width: 100%; height: 160px;">
|
||||
</model-viewer>
|
||||
<p style="text-align: center; margin-top: 8px; font-size: 14px;">"Miniature screwdriver with bright orange handle"</p>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/text/c9.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
background-color="#ffffff"
|
||||
style="display:block; width: 100%; height: 160px;">
|
||||
</model-viewer>
|
||||
<p style="text-align: center; margin-top: 8px; font-size: 14px;">"European style wooden dressing table"</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="swiper-button-prev swiper1-prev"></div>
|
||||
<div class="swiper-button-next swiper1-next"></div>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## ☁️ Run the App Service
|
||||
|
||||
Create 3D assets from text descriptions for a wide range of geometry and styles.
|
||||
!!! note "Note"
|
||||
Gradio servive is a simplified demonstration. For the full functionality, please refer to [text3d-cli](../tutorials/text_to_3d.md).
|
||||
|
||||
|
||||
Text-to-image model based on the Kolors model, supporting Chinese and English prompts. Models downloaded automatically on first run, please be patient.
|
||||
|
||||
```sh
|
||||
# Run in foreground
|
||||
python apps/text_to_3d.py
|
||||
|
||||
# Or run in the background
|
||||
CUDA_VISIBLE_DEVICES=0 nohup python apps/text_to_3d.py > /dev/null 2>&1 &
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
!!! tip "Getting Started"
|
||||
- You can also try Text-to-3D instantly online via our [Hugging Face Space](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Text-to-3D) — no installation required.
|
||||
- Explore EmbodiedGen generated sim-ready [Assets Gallery](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Gallery-Explorer).
|
||||
- For instructions on using the generated asset in any simulator, see [Any Simulators Tutorial](tutorials/any_simulators.md).
|
||||
162
docs/services/texture_edit.md
Normal file
162
docs/services/texture_edit.md
Normal file
@ -0,0 +1,162 @@
|
||||
<script type="module" src="https://unpkg.com/@google/model-viewer/dist/model-viewer.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.js"></script>
|
||||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.css">
|
||||
<script>
|
||||
document.addEventListener('DOMContentLoaded', function () {
|
||||
const swiper = new Swiper('.swiper1', {
|
||||
loop: true,
|
||||
slidesPerView: 3,
|
||||
spaceBetween: 20,
|
||||
navigation: {
|
||||
nextEl: '.swiper-button-next',
|
||||
prevEl: '.swiper-button-prev',
|
||||
},
|
||||
centeredSlides: false,
|
||||
noSwiping: true,
|
||||
noSwipingClass: 'swiper-no-swiping',
|
||||
watchSlidesProgress: true,
|
||||
});
|
||||
});
|
||||
</script>
|
||||
|
||||
# 🎨 Texture Generation Service
|
||||
|
||||
[](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Texture-Gen)
|
||||
|
||||
This service launches a web application to generate visually rich textures for 3D mesh.
|
||||
|
||||
<div class="swiper swiper1" style="max-width: 1000px; margin: 20px auto; border-radius: 12px;">
|
||||
<div class="swiper-wrapper">
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit/hello2.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit/love4.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit/robot_china.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit/horse1.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit/horse2.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit/shoe_0_0.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit/shoe_0_3.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit/clock_num.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit/clock5.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit/vase1.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit/vase2.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit/drill1.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit/drill4.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
</div>
|
||||
<div class="swiper-button-prev"></div>
|
||||
<div class="swiper-button-next"></div>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## ☁️ Run the App Service
|
||||
|
||||
!!! note "Note"
|
||||
Gradio servive is a simplified demonstration. For the full functionality, please refer to [texture-cli](../tutorials/texture_edit.md).
|
||||
|
||||
Run the texture generation service locally. Models downloaded automatically on first run, see `download_kolors_weights`, `geo_cond_mv`.
|
||||
|
||||
|
||||
```sh
|
||||
# Run in foreground
|
||||
python apps/texture_edit.py
|
||||
|
||||
# Or run in the background
|
||||
CUDA_VISIBLE_DEVICES=0 nohup python apps/texture_edit.py > /dev/null 2>&1 &
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
!!! tip "Getting Started"
|
||||
- Try it directly online via our [Hugging Face Space](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Texture-Gen) — no installation required.
|
||||
- Explore EmbodiedGen generated sim-ready [Assets Gallery](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Gallery-Explorer).
|
||||
- For instructions on using the generated asset in any simulator, see [Any Simulators Tutorial](tutorials/any_simulators.md).
|
||||
8
docs/services/visualize_asset.md
Normal file
8
docs/services/visualize_asset.md
Normal file
@ -0,0 +1,8 @@
|
||||
# 📸 EmbodiedGen All-Simulators-Ready Assets Gallery
|
||||
|
||||
[](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Gallery-Explorer)
|
||||
|
||||
|
||||
!!! tip "Getting Started"
|
||||
- Explore EmbodiedGen generated sim-ready [Assets Gallery](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Gallery-Explorer).
|
||||
- For instructions on using the generated asset in any simulator, see [Any Simulators Tutorial](tutorials/any_simulators.md).
|
||||
22
docs/stylesheets/extra.css
Normal file
22
docs/stylesheets/extra.css
Normal file
@ -0,0 +1,22 @@
|
||||
/* Adjust the logo size */
|
||||
.md-header__button.md-logo {
|
||||
height: 4rem;
|
||||
padding: 0;
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
}
|
||||
.md-header__button.md-logo img {
|
||||
height: 4rem;
|
||||
width: auto;
|
||||
}
|
||||
|
||||
.md-typeset pre code {
|
||||
font-size: 0.7rem;
|
||||
/* line-height: 1.5; */
|
||||
font-family: "Fira Code", "JetBrains Mono", monospace;
|
||||
}
|
||||
|
||||
.md-typeset .admonition,
|
||||
.md-typeset details {
|
||||
font-size: 0.77rem;
|
||||
}
|
||||
63
docs/tutorials/any_simulators.md
Normal file
63
docs/tutorials/any_simulators.md
Normal file
@ -0,0 +1,63 @@
|
||||
# 🎮 Use EmbodiedGen in Any Simulator
|
||||
|
||||
Leverage **EmbodiedGen-generated assets** with *accurate physical collisions* and *consistent visual appearance* across major simulation engines — **IsaacSim**, **MuJoCo**, **Genesis**, **PyBullet**, **IsaacGym**, and **SAPIEN**.
|
||||
|
||||
!!! tip "Universal Compatibility"
|
||||
EmbodiedGen assets follow **standardized URDF semantics** with **physically consistent collision meshes**,
|
||||
enabling seamless loading across multiple simulation frameworks — no manual editing needed.
|
||||
|
||||
---
|
||||
|
||||
## 🧩 Supported Simulators
|
||||
|
||||
| Simulator | Conversion Class |
|
||||
|------------|------------------|
|
||||
| [IsaacSim](https://github.com/isaac-sim/IsaacSim) | `MeshtoUSDConverter` |
|
||||
| [MuJoCo](https://github.com/google-deepmind/mujoco) / [Genesis](https://github.com/Genesis-Embodied-AI/Genesis) | `MeshtoMJCFConverter` |
|
||||
| [SAPIEN](https://github.com/haosulab/SAPIEN) / [IsaacGym](https://github.com/isaac-sim/IsaacGymEnvs) / [PyBullet](https://github.com/bulletphysics/bullet3) | `.urdf` generated by EmbodiedGen can be used **directly** |
|
||||
|
||||
!!! note "Simulator Integration Overview"
|
||||
|
||||
This table summarizes the compatibility of EmbodiedGen assets with various simulators:
|
||||
|
||||
| Simulator | Supported Format | Notes |
|
||||
|-----------|-----------------|-------|
|
||||
| IsaacSim | USD / .usda | Use `MeshtoUSDConverter` to convert mesh to USD format. |
|
||||
| MuJoCo | MJCF (.xml) | Use `MeshtoMJCFConverter` for physics-ready assets. |
|
||||
| Genesis | MJCF (.xml) | Same as MuJoCo; fully compatible with Genesis scenes. |
|
||||
| SAPIEN | URDF (.urdf) | Can directly load EmbodiedGen `.urdf` assets. |
|
||||
| IsaacGym | URDF (.urdf) | Directly usable. |
|
||||
| PyBullet | URDF (.urdf) | Directly usable. |
|
||||
|
||||
---
|
||||
|
||||
|
||||
## 🧱 Example: Conversion to Target Simulator
|
||||
|
||||
```python
|
||||
from embodied_gen.data.asset_converter import SimAssetMapper, cvt_embodiedgen_asset_to_anysim
|
||||
from typing import Literal
|
||||
|
||||
simulator_name: Literal[
|
||||
"isaacsim",
|
||||
"isaacgym",
|
||||
"genesis",
|
||||
"pybullet",
|
||||
"sapien3",
|
||||
"mujoco",
|
||||
] = "mujoco"
|
||||
|
||||
dst_asset_path = cvt_embodiedgen_asset_to_anysim(
|
||||
urdf_files=[
|
||||
"path1_to_embodiedgen_asset/asset.urdf",
|
||||
"path2_to_embodiedgen_asset/asset.urdf",
|
||||
],
|
||||
target_type=SimAssetMapper[simulator_name],
|
||||
source_type=AssetType.MESH,
|
||||
overwrite=True,
|
||||
)
|
||||
```
|
||||
|
||||
<img src="assets/simulators_collision.jpg" alt="simulators_collision" width="800">
|
||||
|
||||
Collision and visualization mesh across simulators, showing consistent geometry and material fidelity.
|
||||
0
docs/tutorials/articulated_gen.md
Normal file
0
docs/tutorials/articulated_gen.md
Normal file
3
docs/tutorials/digital_twin.md
Normal file
3
docs/tutorials/digital_twin.md
Normal file
@ -0,0 +1,3 @@
|
||||
# Real-to-Sim Digital Twin Creation
|
||||
|
||||
<img src="assets/real2sim_mujoco.gif" alt="real2sim_mujoco" width="600">
|
||||
22
docs/tutorials/gym_env.md
Normal file
22
docs/tutorials/gym_env.md
Normal file
@ -0,0 +1,22 @@
|
||||
# Simulation in Parallel Envs
|
||||
|
||||
Generate multiple parallel simulation environments with `gym.make` and record sensor and trajectory data.
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Command-Line Usage
|
||||
|
||||
```sh
|
||||
python embodied_gen/scripts/parallel_sim.py \
|
||||
--layout_file "outputs/layouts_gen/task_0000/layout.json" \
|
||||
--output_dir "outputs/parallel_sim/task_0000" \
|
||||
--num_envs 16
|
||||
```
|
||||
|
||||
<div style="display: flex; justify-content: center; align-items: center; gap: 16px; margin: 16px 0;">
|
||||
<img src="assets/parallel_sim.gif" alt="parallel_sim1"
|
||||
style="width: 330px; max-width: 100%; border-radius: 12px; display: block;">
|
||||
<img src="assets/parallel_sim2.gif" alt="parallel_sim2"
|
||||
style="width: 330px; max-width: 100%; border-radius: 12px; display: block;">
|
||||
</div>
|
||||
|
||||
92
docs/tutorials/image_to_3d.md
Normal file
92
docs/tutorials/image_to_3d.md
Normal file
@ -0,0 +1,92 @@
|
||||
<script type="module" src="https://unpkg.com/@google/model-viewer/dist/model-viewer.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.js"></script>
|
||||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.css">
|
||||
<script>
|
||||
document.addEventListener('DOMContentLoaded', function () {
|
||||
const swiper = new Swiper('.swiper1', {
|
||||
loop: true,
|
||||
slidesPerView: 3,
|
||||
spaceBetween: 20,
|
||||
navigation: {
|
||||
nextEl: '.swiper-button-next',
|
||||
prevEl: '.swiper-button-prev',
|
||||
},
|
||||
centeredSlides: false,
|
||||
noSwiping: true,
|
||||
noSwipingClass: 'swiper-no-swiping',
|
||||
watchSlidesProgress: true,
|
||||
});
|
||||
});
|
||||
</script>
|
||||
|
||||
# 🖼️ Image-to-3D: Physically Plausible 3D Asset Generation
|
||||
|
||||
Generate **physically plausible 3D assets** from a single input image, supporting **digital twin** and **simulation environments**.
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Command-Line Usage
|
||||
|
||||
```bash
|
||||
img3d-cli --image_path apps/assets/example_image/sample_00.jpg \
|
||||
apps/assets/example_image/sample_01.jpg apps/assets/example_image/sample_19.jpg \
|
||||
--n_retry 1 --output_root outputs/imageto3d
|
||||
```
|
||||
|
||||
You will get the following results:
|
||||
|
||||
<div class="swiper swiper1" style="max-width: 1000px; margin: 20px auto; border-radius: 12px;">
|
||||
<div class="swiper-wrapper">
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/image2/sample_00.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/image2/sample_01.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/image2/sample_19.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
</div>
|
||||
<div class="swiper-button-prev"></div>
|
||||
<div class="swiper-button-next"></div>
|
||||
</div>
|
||||
|
||||
|
||||
The generated results are organized as follows:
|
||||
```sh
|
||||
outputs/imageto3d/sample_xx/result
|
||||
├── mesh
|
||||
│ ├── material_0.png
|
||||
│ ├── material.mtl
|
||||
│ ├── sample_xx_collision.ply
|
||||
│ ├── sample_xx.glb
|
||||
│ ├── sample_xx_gs.ply
|
||||
│ └── sample_xx.obj
|
||||
├── sample_xx.urdf
|
||||
└── video.mp4
|
||||
```
|
||||
|
||||
- `mesh/` → Geometry and texture files, including visual mesh, collision mesh and 3DGS.
|
||||
- `*.urdf` → Simulator-ready URDF with collision and visual meshes
|
||||
- `video.mp4` → Preview of the generated 3D asset
|
||||
|
||||
|
||||
!!! tip "Getting Started"
|
||||
- Try it directly online via our [Hugging Face Space](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Image-to-3D) — no installation required.
|
||||
- Explore EmbodiedGen generated sim-ready [Assets Gallery](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Gallery-Explorer).
|
||||
- For instructions on using the generated asset in any simulator, see [Any Simulators Tutorial](any_simulators.md).
|
||||
197
docs/tutorials/index.md
Normal file
197
docs/tutorials/index.md
Normal file
@ -0,0 +1,197 @@
|
||||
<script type="module" src="https://unpkg.com/@google/model-viewer/dist/model-viewer.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.js"></script>
|
||||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.css">
|
||||
<script>
|
||||
document.addEventListener('DOMContentLoaded', function () {
|
||||
const swiper = new Swiper('.swiper1', {
|
||||
loop: true,
|
||||
slidesPerView: 3,
|
||||
spaceBetween: 20,
|
||||
navigation: {
|
||||
nextEl: '.swiper-button-next',
|
||||
prevEl: '.swiper-button-prev',
|
||||
},
|
||||
centeredSlides: false,
|
||||
noSwiping: true,
|
||||
noSwipingClass: 'swiper-no-swiping',
|
||||
watchSlidesProgress: true,
|
||||
});
|
||||
});
|
||||
</script>
|
||||
|
||||
# Tutorials & Interface Usage
|
||||
|
||||
Welcome to the tutorials for `EmbodiedGen`. `EmbodiedGen` is a powerful toolset for generating 3D assets, textures, scenes, and interactive layouts ready for simulators and digital twin environments.
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Prerequisites
|
||||
|
||||
!!! tip "Prerequisites"
|
||||
Make sure to finish the [Installation Guide](../install.md) before starting tutorial. Missing dependencies will cause initialization errors. Model weights are automatically downloaded on first run.
|
||||
|
||||
---
|
||||
|
||||
## [🖼️ Image-to-3D](image_to_3d.md)
|
||||
|
||||
Generate **physically plausible 3D assets** from a single input image, supporting digital twin and simulation environments.
|
||||
|
||||
|
||||
<div class="swiper swiper1" style="max-width: 1000px; margin: 20px auto; border-radius: 12px;">
|
||||
<div class="swiper-wrapper">
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/image2/sample_00.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/image2/sample_01.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/image2/sample_19.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
</div>
|
||||
<div class="swiper-button-prev"></div>
|
||||
<div class="swiper-button-next"></div>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## [📝 Text-to-3D](text_to_3d.md)
|
||||
|
||||
Create **physically plausible 3D assets** from **text descriptions**, supporting a wide range of geometry, style, and material details.
|
||||
|
||||
|
||||
<div class="swiper swiper1" style="max-width: 1000px; margin: 20px auto; border-radius: 12px;">
|
||||
<div class="swiper-wrapper">
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/text2/sample3d_0.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
background-color="#ffffff"
|
||||
style="display:block; width: 100%; height: 160px; border-radius: 12px;"
|
||||
>
|
||||
</model-viewer>
|
||||
<p style="text-align: center; margin-top: 8px; font-size: 14px;">"small bronze figurine of a lion"</p>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/text2/sample3d_1.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
background-color="#ffffff"
|
||||
style="display:block; width: 100%; height: 160px;">
|
||||
</model-viewer>
|
||||
<p style="text-align: center; margin-top: 8px; font-size: 14px;">"A globe with wooden base"</p>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/text2/sample3d_2.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
background-color="#ffffff"
|
||||
style="display:block; width: 100%; height: 160px;">
|
||||
</model-viewer>
|
||||
<p style="text-align: center; margin-top: 8px; font-size: 14px;">"wooden table with embroidery"</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="swiper-button-prev"></div>
|
||||
<div class="swiper-button-next"></div>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## [🎨 Texture Generation](texture_gen.md)
|
||||
|
||||
Generate **high-quality textures** for 3D meshes using **text prompts**, supporting both Chinese and English, to enhance the visual appearance of existing 3D assets.
|
||||
|
||||
<div class="swiper swiper1" style="max-width: 1000px; margin: 20px auto; border-radius: 12px;">
|
||||
<div class="swiper-wrapper">
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit2/robot_text.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
camera-orbit="180deg auto auto"
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit2/horse.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
camera-orbit="90deg auto auto"
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
</div>
|
||||
<div class="swiper-button-prev"></div>
|
||||
<div class="swiper-button-next"></div>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## [🌍 3D Scene Generation](scene_gen.md)
|
||||
|
||||
Generate **physically consistent and visually coherent 3D environments** from text prompts. Typically used as **background** 3DGS scenes in simulators for efficient and photo-realistic rendering.
|
||||
|
||||
<img src="assets/scene3d.gif" style="width: 500px; max-width: 100%; border-radius: 12px; display: block; margin: 16px auto;">
|
||||
|
||||
---
|
||||
|
||||
## [🏞️ Layout Generation](layout_gen.md)
|
||||
|
||||
Generate diverse, physically realistic, and scalable **interactive 3D scenes** from natural language task descriptions, while also modeling the robot and manipulable objects.
|
||||
|
||||
<div align="center" style="display: grid; grid-template-columns: repeat(2, 1fr); gap: 16px; justify-items: center; margin: 20px 0;">
|
||||
<img src="assets/layout1.gif" alt="layout1" style="width: 400px; border-radius: 12px; display: block;">
|
||||
<img src="assets/layout2.gif" alt="layout2" style="width: 400px; border-radius: 12px; display: block;">
|
||||
<img src="assets/layout3.gif" alt="layout3" style="width: 400px; border-radius: 12px; display: block;">
|
||||
<img src="assets/Iscene_demo2.gif" alt="layout4" style="width: 400px; border-radius: 12px; display: block;">
|
||||
</div>
|
||||
|
||||
|
||||
---
|
||||
|
||||
## [🏎️ Parallel Simulation](gym_env.md)
|
||||
|
||||
Generate multiple **parallel simulation environments** with `gym.make` and record sensor and trajectory data.
|
||||
|
||||
<div style="display: flex; justify-content: center; align-items: center; gap: 16px; margin: 16px 0;">
|
||||
<img src="assets/parallel_sim.gif" alt="parallel_sim1"
|
||||
style="width: 330px; max-width: 100%; border-radius: 12px; display: block;">
|
||||
<img src="assets/parallel_sim2.gif" alt="parallel_sim2"
|
||||
style="width: 330px; max-width: 100%; border-radius: 12px; display: block;">
|
||||
</div>
|
||||
|
||||
|
||||
---
|
||||
|
||||
## [🎮 Use in Any Simulator](any_simulators.md)
|
||||
|
||||
Seamlessly use EmbodiedGen-generated assets in major simulators like **IsaacSim**, **MuJoCo**, **Genesis**, **PyBullet**, **IsaacGym**, and **SAPIEN**, featuring **accurate physical collisions** and **consistent visual appearance**.
|
||||
|
||||
<div align="center">
|
||||
<img src="assets/simulators_collision.jpg" alt="simulators_collision" style="width: 400px; max-width: 100%; border-radius: 12px; display: block; margin: 16px 0;">
|
||||
</div>
|
||||
|
||||
## [🔧 Real-to-Sim Digital Twin Creation](digital_twin.md)
|
||||
|
||||
<div align="center">
|
||||
<img src="assets/real2sim_mujoco.gif" alt="real2sim_mujoco" style="width: 400px; max-width: 100%; border-radius: 12px; display: block; margin: 16px 0;">
|
||||
</div>
|
||||
90
docs/tutorials/layout_gen.md
Normal file
90
docs/tutorials/layout_gen.md
Normal file
@ -0,0 +1,90 @@
|
||||
# 🏞️ Layout Generation — Interactive 3D Scenes
|
||||
|
||||
Layout Generation enables the generation of diverse, physically realistic, and scalable **interactive 3D scenes** directly from natural language task descriptions, while also modeling the robot's pose and relationships with manipulable objects. Target objects are randomly placed within the robot's reachable range, making the scenes readily usable for downstream simulation and reinforcement learning tasks in any mainstream simulator.
|
||||
|
||||
<div align="center" style="display: grid; grid-template-columns: repeat(2, 1fr); gap: 16px; justify-items: center; margin: 20px 0;">
|
||||
<img src="assets/layout1.gif" alt="layout1" style="width: 400px; border-radius: 12px; display: block;">
|
||||
<img src="assets/layout2.gif" alt="layout2" style="width: 400px; border-radius: 12px; display: block;">
|
||||
<img src="assets/layout3.gif" alt="layout3" style="width: 400px; border-radius: 12px; display: block;">
|
||||
<img src="assets/Iscene_demo2.gif" alt="layout4" style="width: 400px; border-radius: 12px; display: block;">
|
||||
</div>
|
||||
|
||||
!!! note "Model Requirement"
|
||||
The text-to-image model is based on `SD3.5 Medium`. Usage requires agreement to the [model license](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium).
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites — Prepare Background 3D Scenes
|
||||
|
||||
Before running `layout-cli`, you need to prepare background 3D scenes.
|
||||
You can either **generate your own** using the [`scene3d-cli`](scene_gen.md), or **download pre-generated backgrounds** for convenience.
|
||||
|
||||
Each scene takes approximately **30 minutes** to generate. For efficiency, we recommend pre-generating and listing them in `outputs/bg_scenes/scene_list.txt`.
|
||||
|
||||
```bash
|
||||
# Option 1: Download pre-generated backgrounds (~4 GB)
|
||||
hf download xinjjj/scene3d-bg --repo-type dataset --local-dir outputs
|
||||
|
||||
# Option 2: Download a larger background set (~14 GB)
|
||||
hf download xinjjj/EmbodiedGenRLv2-BG --repo-type dataset --local-dir outputs
|
||||
```
|
||||
|
||||
## Generate Interactive Layout Scenes
|
||||
|
||||
Use the `layout-cli` to create interactive 3D scenes based on task descriptions. Each layout generation takes approximately 30 minutes.
|
||||
|
||||
```sh
|
||||
layout-cli \
|
||||
--task_descs "Place the pen in the mug on the desk" \
|
||||
"Put the fruit on the table on the plate" \
|
||||
--bg_list "outputs/bg_scenes/scene_list.txt" \
|
||||
--output_root "outputs/layouts_gen" \
|
||||
--insert_robot
|
||||
```
|
||||
|
||||
You will get the following results:
|
||||
<div align="center" style="display: grid; grid-template-columns: repeat(2, 1fr); gap: 16px; justify-items: center; margin: 20px 0;">
|
||||
<img src="assets/Iscene_demo1.gif" alt="Iscene_demo1" style="width: 320px; border-radius: 12px; display: block;">
|
||||
<img src="assets/Iscene_demo2.gif" alt="Iscene_demo2" style="width: 320px; border-radius: 12px; display: block;">
|
||||
</div>
|
||||
|
||||
|
||||
### Batch Generation
|
||||
|
||||
You can also run multiple tasks via a task list file in the backend.
|
||||
|
||||
```sh
|
||||
CUDA_VISIBLE_DEVICES=0 nohup layout-cli \
|
||||
--task_descs "apps/assets/example_layout/task_list.txt" \
|
||||
--bg_list "outputs/bg_scenes/scene_list.txt" \
|
||||
--output_root "outputs/layouts_gens" \
|
||||
--insert_robot > layouts_gens.log &
|
||||
```
|
||||
|
||||
> 💡 Remove `--insert_robot` if you don’t need robot pose consideration in layout generation.
|
||||
|
||||
### Layout Randomization
|
||||
|
||||
Using `compose_layout.py`, you can **recompose the layout** of the generated interactive 3D scenes.
|
||||
|
||||
```sh
|
||||
python embodied_gen/scripts/compose_layout.py \
|
||||
--layout_path "outputs/layouts_gens/task_0000/layout.json" \
|
||||
--output_dir "outputs/layouts_gens/task_0000/recompose" \
|
||||
--insert_robot
|
||||
```
|
||||
|
||||
### Load Interactive 3D Scenes in Simulators
|
||||
|
||||
We provide `sim-cli`, that allows users to easily load generated layouts into an interactive 3D simulation using the SAPIEN engine.
|
||||
|
||||
```sh
|
||||
sim-cli --layout_path "outputs/layouts_gen/task_0000/layout.json" \
|
||||
--output_dir "outputs/layouts_gen/task_0000/sapien_render" --insert_robot
|
||||
```
|
||||
|
||||
!!! tip "Recommended Workflow"
|
||||
1. Generate or download background scenes using `scene3d-cli`.
|
||||
2. Create interactive layouts from task descriptions using `layout-cli`.
|
||||
3. Optionally recompose them using `compose_layout.py`.
|
||||
4. Load the final layouts into simulators with `sim-cli`.
|
||||
47
docs/tutorials/scene_gen.md
Normal file
47
docs/tutorials/scene_gen.md
Normal file
@ -0,0 +1,47 @@
|
||||
# 🌍 3D Scene Generation
|
||||
|
||||
Generate **physically consistent and visually coherent 3D environments** from text prompts. Typically used as **background** 3DGS scenes in simulators for efficient and photo-realistic rendering.
|
||||
|
||||
---
|
||||
|
||||
<img src="assets/scene3d.gif" style="width: 600px; border-radius: 12px; display: block; margin: 16px auto;">
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Command-Line Usage
|
||||
|
||||
> 💡 Run `bash install.sh extra` to install additional dependencies if you plan to use `scene3d-cli`.
|
||||
|
||||
It typically takes ~30 minutes per scene to generate both the colored mesh and 3D Gaussian Splat(3DGS) representation.
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 scene3d-cli \
|
||||
--prompts "Art studio with easel and canvas" \
|
||||
--output_dir outputs/bg_scenes/ \
|
||||
--seed 0 \
|
||||
--gs3d.max_steps 4000 \
|
||||
--disable_pano_check
|
||||
```
|
||||
|
||||
The generated results are organized as follows:
|
||||
```sh
|
||||
outputs/bg_scenes/scene_000
|
||||
├── gs_model.ply
|
||||
├── gsplat_cfg.yml
|
||||
├── mesh_model.ply
|
||||
├── pano_image.png
|
||||
├── prompt.txt
|
||||
└── video.mp4
|
||||
```
|
||||
|
||||
- `gs_model.ply` → Generated 3D scene in 3D Gaussian Splat representation.
|
||||
- `mesh_model.ply` → Color mesh representation of the generated scene.
|
||||
- `gsplat_cfg.yml` → Configuration file for 3DGS training and rendering parameters.
|
||||
- `pano_image.png` → Generated panoramic view image.
|
||||
- `prompt.txt` → Original scene generation prompt for traceability.
|
||||
- `video.mp4` → Preview RGB and depth preview of the generated 3D scene.
|
||||
|
||||
!!! note "Usage Notes"
|
||||
- `3D Scene Generation` produces background 3DGS scenes optimized for efficient rendering in simulation environments. We also provide hybrid rendering examples combining background 3DGS with foreground interactive assets, see the [example]()
|
||||
for details.
|
||||
- In Layout Generation, we further demonstrate task-desc-driven interactive 3D scene generation, building complete 3D scenes based on natural language task descriptions. See the [Layout Generation Guide](layout_gen.md).
|
||||
137
docs/tutorials/text_to_3d.md
Normal file
137
docs/tutorials/text_to_3d.md
Normal file
@ -0,0 +1,137 @@
|
||||
<script type="module" src="https://unpkg.com/@google/model-viewer/dist/model-viewer.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.js"></script>
|
||||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.css">
|
||||
<script>
|
||||
document.addEventListener('DOMContentLoaded', function () {
|
||||
const swiper = new Swiper('.swiper1', {
|
||||
loop: true,
|
||||
slidesPerView: 3,
|
||||
spaceBetween: 20,
|
||||
navigation: {
|
||||
nextEl: '.swiper-button-next',
|
||||
prevEl: '.swiper-button-prev',
|
||||
},
|
||||
centeredSlides: false,
|
||||
noSwiping: true,
|
||||
noSwipingClass: 'swiper-no-swiping',
|
||||
watchSlidesProgress: true,
|
||||
});
|
||||
});
|
||||
</script>
|
||||
|
||||
|
||||
# 📝 Text-to-3D: Generate 3D Assets from Text
|
||||
|
||||
Create **physically plausible 3D assets** from **text descriptions**, supporting a wide range of geometry, style, and material details.
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Command-Line Usage
|
||||
|
||||
**Basic CLI(recommend)**
|
||||
|
||||
Text-to-image model based on Stable Diffusion 3.5 Medium， English prompts only. Usage requires agreement to the [model license (click “Accept”)](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium).
|
||||
|
||||
```bash
|
||||
text3d-cli \
|
||||
--prompts "small bronze figurine of a lion" "A globe with wooden base" "wooden table with embroidery" \
|
||||
--n_image_retry 1 \
|
||||
--n_asset_retry 1 \
|
||||
--n_pipe_retry 1 \
|
||||
--seed_img 0 \
|
||||
--output_root outputs/textto3d
|
||||
```
|
||||
|
||||
- `--n_image_retry`: Number of retries per prompt for text-to-image generation
|
||||
- `--n_asset_retry`: Retry attempts for image-to-3D assets generation
|
||||
- `--n_pipe_retry`: Pipeline retry for end-to-end 3D asset quality check
|
||||
- `--seed_img`: Optional initial seed image for style guidance
|
||||
- `--output_root`: Directory to save generated assets
|
||||
|
||||
For large-scale 3D asset generation, set `--n_image_retry=4` `--n_asset_retry=3` `--n_pipe_retry=2`, slower but better, via automatic checking and retries. For more diverse results, omit `--seed_img`.
|
||||
|
||||
You will get the following results:
|
||||
|
||||
<div class="swiper swiper1" style="max-width: 1000px; margin: 20px auto; border-radius: 12px;">
|
||||
<div class="swiper-wrapper">
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/text2/sample3d_0.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
background-color="#ffffff"
|
||||
style="display:block; width: 100%; height: 160px; border-radius: 12px;"
|
||||
>
|
||||
</model-viewer>
|
||||
<p style="text-align: center; margin-top: 8px; font-size: 14px;">"small bronze figurine of a lion"</p>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/text2/sample3d_1.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
background-color="#ffffff"
|
||||
style="display:block; width: 100%; height: 160px;">
|
||||
</model-viewer>
|
||||
<p style="text-align: center; margin-top: 8px; font-size: 14px;">"A globe with wooden base"</p>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/text2/sample3d_2.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
background-color="#ffffff"
|
||||
style="display:block; width: 100%; height: 160px;">
|
||||
</model-viewer>
|
||||
<p style="text-align: center; margin-top: 8px; font-size: 14px;">"wooden table with embroidery"</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="swiper-button-prev swiper1-prev"></div>
|
||||
<div class="swiper-button-next swiper1-next"></div>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
|
||||
Kolors Model CLI (Supports Chinese & English Prompts):
|
||||
```bash
|
||||
bash embodied_gen/scripts/textto3d.sh \
|
||||
--prompts "small bronze figurine of a lion" "A globe with wooden base and latitude and longitude lines" "橙色电动手钻，有磨损细节" \
|
||||
--output_root outputs/textto3d_k
|
||||
```
|
||||
|
||||
> Models with more permissive licenses can be found in `embodied_gen/models/image_comm_model.py`.
|
||||
|
||||
|
||||
The generated results are organized as follows:
|
||||
```sh
|
||||
outputs/textto3d
|
||||
├── asset3d
|
||||
│ ├── sample3d_xx
|
||||
│ │ └── result
|
||||
│ │ ├── mesh
|
||||
│ │ │ ├── material_0.png
|
||||
│ │ │ ├── material.mtl
|
||||
│ │ │ ├── sample3d_xx_collision.obj
|
||||
│ │ │ ├── sample3d_xx.glb
|
||||
│ │ │ ├── sample3d_xx_gs.ply
|
||||
│ │ │ └── sample3d_xx.obj
|
||||
│ │ ├── sample3d_xx.urdf
|
||||
│ │ └── video.mp4
|
||||
└── images
|
||||
├── sample3d_xx.png
|
||||
├── sample3d_xx_raw.png
|
||||
```
|
||||
|
||||
- `mesh/` → 3D geometry and texture files for the asset, including visual mesh, collision mesh and 3DGS
|
||||
- `*.urdf` → Simulator-ready URDF including collision and visual meshes
|
||||
- `video.mp4` → Preview video of the generated 3D asset
|
||||
- `images/sample3d_xx.png` → Foreground-extracted image used for image-to-3D step
|
||||
- `images/sample3d_xx_raw.png` → Original generated image from the text-to-image step
|
||||
|
||||
---
|
||||
|
||||
!!! tip "Getting Started"
|
||||
- You can also try Text-to-3D instantly online via our [Hugging Face Space](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Text-to-3D) — no installation required.
|
||||
- Explore EmbodiedGen generated sim-ready [Assets Gallery](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Gallery-Explorer).
|
||||
- For instructions on using the generated asset in any simulator, see [Any Simulators Tutorial](any_simulators.md).
|
||||
78
docs/tutorials/texture_edit.md
Normal file
78
docs/tutorials/texture_edit.md
Normal file
@ -0,0 +1,78 @@
|
||||
<script type="module" src="https://unpkg.com/@google/model-viewer/dist/model-viewer.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.js"></script>
|
||||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swiper/swiper-bundle.min.css">
|
||||
<script>
|
||||
document.addEventListener('DOMContentLoaded', function () {
|
||||
const swiper = new Swiper('.swiper1', {
|
||||
loop: true,
|
||||
slidesPerView: 3,
|
||||
spaceBetween: 20,
|
||||
navigation: {
|
||||
nextEl: '.swiper-button-next',
|
||||
prevEl: '.swiper-button-prev',
|
||||
},
|
||||
centeredSlides: false,
|
||||
noSwiping: true,
|
||||
noSwipingClass: 'swiper-no-swiping',
|
||||
watchSlidesProgress: true,
|
||||
});
|
||||
});
|
||||
</script>
|
||||
|
||||
# 🎨 Texture Generation: Create Visually Rich Textures for 3D Meshes
|
||||
|
||||
Generate **high-quality textures** for 3D meshes using **text prompts**, supporting both **Chinese and English**. This allows you to enhance the visual appearance of existing 3D assets for simulation, visualization, or digital twin applications.
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Command-Line Usage
|
||||
|
||||
```bash
|
||||
texture-cli \
|
||||
--mesh_path "apps/assets/example_texture/meshes/robot_text.obj" \
|
||||
"apps/assets/example_texture/meshes/horse.obj" \
|
||||
--prompt "举着牌子的写实风格机器人，大眼睛，牌子上写着“Hello”的文字" \
|
||||
"A gray horse head with flying mane and brown eyes" \
|
||||
--output_root "outputs/texture_gen" \
|
||||
--seed 0
|
||||
```
|
||||
|
||||
- `--mesh_path` → Path(s) to input 3D mesh files
|
||||
- `--prompt` → Text prompt(s) describing desired texture/style for each mesh
|
||||
- `--output_root` → Directory to save textured meshes and related outputs
|
||||
- `--seed` → Random seed for reproducible texture generation
|
||||
|
||||
|
||||
You will get the following results:
|
||||
|
||||
<div class="swiper swiper1" style="max-width: 1000px; margin: 20px auto; border-radius: 12px;">
|
||||
<div class="swiper-wrapper">
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit2/robot_text.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
camera-orbit="180deg auto auto"
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
<div class="swiper-slide model-card">
|
||||
<model-viewer
|
||||
src="https://raw.githubusercontent.com/HochCC/ShowCase/main/edit2/horse.glb"
|
||||
auto-rotate
|
||||
camera-controls
|
||||
camera-orbit="90deg auto auto"
|
||||
style="display:block; width:100%; height:250px; background-color: #f8f8f8;">
|
||||
</model-viewer>
|
||||
</div>
|
||||
</div>
|
||||
<div class="swiper-button-prev"></div>
|
||||
<div class="swiper-button-next"></div>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
!!! tip "Getting Started"
|
||||
- Try it directly online via our [Hugging Face Space](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Texture-Gen) — no installation required.
|
||||
- Explore EmbodiedGen generated sim-ready [Assets Gallery](https://huggingface.co/spaces/HorizonRobotics/EmbodiedGen-Gallery-Explorer).
|
||||
- For instructions on using the generated asset in any simulator, see [Any Simulators Tutorial](any_simulators.md).
|
||||
@ -59,6 +59,7 @@ def cvt_embodiedgen_asset_to_anysim(
|
||||
urdf_files: list[str],
|
||||
target_type: AssetType,
|
||||
source_type: AssetType,
|
||||
overwrite: bool = False,
|
||||
**kwargs,
|
||||
) -> dict[str, str]:
|
||||
"""Convert URDF files generated by EmbodiedGen into the format required by all simulators.
|
||||
@ -76,6 +77,8 @@ def cvt_embodiedgen_asset_to_anysim(
|
||||
urdf_files (List[str]): List of URDF file paths to be converted.
|
||||
target_type (AssetType): The target asset type.
|
||||
source_type (AssetType): The source asset type.
|
||||
overwrite (bool): Whether to overwrite existing converted files.
|
||||
**kwargs: Additional keyword arguments for the converter.
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: A dictionary mapping the original URDF file path to the converted asset file path.
|
||||
@ -424,7 +427,8 @@ class MeshtoUSDConverter(AssetConverterBase):
|
||||
|
||||
DEFAULT_BIND_APIS = [
|
||||
"MaterialBindingAPI",
|
||||
"PhysicsMeshCollisionAPI",
|
||||
# "PhysicsMeshCollisionAPI",
|
||||
"PhysxDecompositionCollisionAPI",
|
||||
"PhysicsCollisionAPI",
|
||||
"PhysxCollisionAPI",
|
||||
"PhysicsMassAPI",
|
||||
@ -554,7 +558,8 @@ class MeshtoUSDConverter(AssetConverterBase):
|
||||
class PhysicsUSDAdder(MeshtoUSDConverter):
|
||||
DEFAULT_BIND_APIS = [
|
||||
"MaterialBindingAPI",
|
||||
"PhysicsMeshCollisionAPI",
|
||||
# "PhysicsMeshCollisionAPI",
|
||||
"PhysxDecompositionCollisionAPI",
|
||||
"PhysicsCollisionAPI",
|
||||
"PhysxCollisionAPI",
|
||||
"PhysicsRigidBodyAPI",
|
||||
@ -771,33 +776,33 @@ if __name__ == "__main__":
|
||||
for urdf_path, output_file in zip(urdf_paths, output_files):
|
||||
asset_converter.convert(urdf_path, output_file)
|
||||
|
||||
# urdf_path = "outputs/embodiedgen_assets/demo_assets/remote_control/result/remote_control.urdf"
|
||||
# output_file = "outputs/embodiedgen_assets/demo_assets/remote_control/usd/remote_control.usd"
|
||||
urdf_path = "outputs/embodiedgen_assets/demo_assets/remote_control/result/remote_control.urdf"
|
||||
output_file = "outputs/embodiedgen_assets/demo_assets/remote_control/usd/remote_control.usd"
|
||||
|
||||
# asset_converter = AssetConverterFactory.create(
|
||||
# target_type=AssetType.USD,
|
||||
# source_type=AssetType.URDF,
|
||||
# rotate_wxyz=(0.7071, 0.7071, 0, 0), # rotate 90 deg around the X-axis
|
||||
# )
|
||||
asset_converter = AssetConverterFactory.create(
|
||||
target_type=AssetType.USD,
|
||||
source_type=AssetType.URDF,
|
||||
rotate_wxyz=(0.7071, 0.7071, 0, 0), # rotate 90 deg around the X-axis
|
||||
)
|
||||
|
||||
# with asset_converter:
|
||||
# asset_converter.convert(urdf_path, output_file)
|
||||
with asset_converter:
|
||||
asset_converter.convert(urdf_path, output_file)
|
||||
|
||||
# # Convert infinigen urdf to mjcf
|
||||
# urdf_path = "/home/users/xinjie.wang/xinjie/infinigen/outputs/exports/kitchen_i_urdf/export_scene/scene.urdf"
|
||||
# output_file = "/home/users/xinjie.wang/xinjie/infinigen/outputs/exports/kitchen_i_urdf/mjcf/scene.xml"
|
||||
# asset_converter = AssetConverterFactory.create(
|
||||
# target_type=AssetType.MJCF,
|
||||
# source_type=AssetType.URDF,
|
||||
# keep_materials=["diffuse"],
|
||||
# )
|
||||
# with asset_converter:
|
||||
# asset_converter.convert(urdf_path, output_file)
|
||||
# Convert infinigen urdf to mjcf
|
||||
urdf_path = "/home/users/xinjie.wang/xinjie/infinigen/outputs/exports/kitchen_i_urdf/export_scene/scene.urdf"
|
||||
output_file = "/home/users/xinjie.wang/xinjie/infinigen/outputs/exports/kitchen_i_urdf/mjcf/scene.xml"
|
||||
asset_converter = AssetConverterFactory.create(
|
||||
target_type=AssetType.MJCF,
|
||||
source_type=AssetType.URDF,
|
||||
keep_materials=["diffuse"],
|
||||
)
|
||||
with asset_converter:
|
||||
asset_converter.convert(urdf_path, output_file)
|
||||
|
||||
# # Convert infinigen usdc to physics usdc
|
||||
# converter = PhysicsUSDAdder()
|
||||
# with converter:
|
||||
# converter.convert(
|
||||
# usd_path="/home/users/xinjie.wang/xinjie/infinigen/outputs/usdc/export_scene/export_scene.usdc",
|
||||
# output_file="/home/users/xinjie.wang/xinjie/infinigen/outputs/usdc_p3/export_scene/export_scene.usdc",
|
||||
# )
|
||||
# Convert infinigen usdc to physics usdc
|
||||
converter = PhysicsUSDAdder()
|
||||
with converter:
|
||||
converter.convert(
|
||||
usd_path="/home/users/xinjie.wang/xinjie/infinigen/outputs/usdc/export_scene/export_scene.usdc",
|
||||
output_file="/home/users/xinjie.wang/xinjie/infinigen/outputs/usdc_p3/export_scene/export_scene.usdc",
|
||||
)
|
||||
|
||||
@ -42,6 +42,56 @@ def build_texture_gen_pipe(
|
||||
ip_adapt_scale: float = 0,
|
||||
device: str = "cuda",
|
||||
) -> DiffusionPipeline:
|
||||
"""Build and initialize the Kolors + ControlNet (optional IP-Adapter) texture generation pipeline.
|
||||
|
||||
Loads Kolors tokenizer, text encoder (ChatGLM), VAE, UNet, scheduler and (optionally)
|
||||
a ControlNet checkpoint plus IP-Adapter vision encoder. If ``controlnet_ckpt`` is
|
||||
not provided, the default multi-view texture ControlNet weights are downloaded
|
||||
automatically from the hub. When ``ip_adapt_scale > 0`` an IP-Adapter vision
|
||||
encoder and its weights are also loaded and activated.
|
||||
|
||||
Args:
|
||||
base_ckpt_dir (str):
|
||||
Root directory where Kolors (and optionally Kolors-IP-Adapter-Plus) weights
|
||||
are or will be stored. Required subfolders: ``Kolors/{text_encoder,vae,unet,scheduler}``.
|
||||
controlnet_ckpt (str, optional):
|
||||
Directory containing a ControlNet checkpoint (safetensors). If ``None``,
|
||||
downloads the default ``texture_gen_mv_v1`` snapshot.
|
||||
ip_adapt_scale (float, optional):
|
||||
Strength (>=0) of IP-Adapter conditioning. Set >0 to enable IP-Adapter;
|
||||
typical values: 0.4-0.8. Default: 0 (disabled).
|
||||
device (str, optional):
|
||||
Target device to move the pipeline to (e.g. ``"cuda"``, ``"cuda:0"``, ``"cpu"``).
|
||||
Default: ``"cuda"``.
|
||||
|
||||
Returns:
|
||||
DiffusionPipeline: A configured
|
||||
``StableDiffusionXLControlNetImg2ImgPipeline`` ready for multi-view texture
|
||||
generation (with optional IP-Adapter support).
|
||||
|
||||
Example:
|
||||
Initialize pipeline with IP-Adapter enabled.
|
||||
```python
|
||||
from embodied_gen.models.texture_model import build_texture_gen_pipe
|
||||
ip_adapt_scale = 0.7
|
||||
PIPELINE = build_texture_gen_pipe(
|
||||
base_ckpt_dir="./weights",
|
||||
ip_adapt_scale=ip_adapt_scale,
|
||||
device="cuda",
|
||||
)
|
||||
PIPELINE.set_ip_adapter_scale([ip_adapt_scale])
|
||||
```
|
||||
Initialize pipeline without IP-Adapter.
|
||||
```python
|
||||
from embodied_gen.models.texture_model import build_texture_gen_pipe
|
||||
PIPELINE = build_texture_gen_pipe(
|
||||
base_ckpt_dir="./weights",
|
||||
ip_adapt_scale=0,
|
||||
device="cuda",
|
||||
)
|
||||
```
|
||||
"""
|
||||
|
||||
download_kolors_weights(f"{base_ckpt_dir}/Kolors")
|
||||
logger.info(f"Load Kolors weights...")
|
||||
tokenizer = ChatGLMTokenizer.from_pretrained(
|
||||
|
||||
110
mkdocs.yml
Normal file
110
mkdocs.yml
Normal file
@ -0,0 +1,110 @@
|
||||
site_name: Documentation
|
||||
site_url: https://horizonrobotics.github.io/EmbodiedGen/
|
||||
repo_name: "EmbodiedGen"
|
||||
repo_url: https://github.com/HorizonRobotics/EmbodiedGen
|
||||
copyright: "Copyright (c) 2025 Horizon Robotics"
|
||||
|
||||
nav:
|
||||
- 🏠 Home: index.md
|
||||
- 🚀 Installation: install.md
|
||||
- 🧩 Services:
|
||||
- Overview: services/index.md
|
||||
- Image-to-3D: services/image_to_3d.md
|
||||
- Text-to-3D: services/text_to_3d.md
|
||||
- Texture Generation: services/texture_edit.md
|
||||
- Asset Visualizer: services/visualize_asset.md
|
||||
- 📘 Tutorials:
|
||||
- Overview: tutorials/index.md
|
||||
- Image-to-3D: tutorials/image_to_3d.md
|
||||
- Text-to-3D: tutorials/text_to_3d.md
|
||||
- Texture Generation: tutorials/texture_edit.md
|
||||
# - Articulated Object Generation: tutorials/articulated_gen.md
|
||||
- 3D Scene Generation: tutorials/scene_gen.md
|
||||
- Interactive 3D Scenes: tutorials/layout_gen.md
|
||||
- Gym Parallel Envs: tutorials/gym_env.md
|
||||
- Any Simulators: tutorials/any_simulators.md
|
||||
- Digital Twin Creation: tutorials/digital_twin.md
|
||||
- 📚 API Reference:
|
||||
- Overview: api/index.md
|
||||
- Data: api/data.md
|
||||
- Envs: api/envs.md
|
||||
- Models: api/models.md
|
||||
- Trainer: api/trainer.md
|
||||
- Utilities: api/utils.md
|
||||
- Validators: api/validators.md
|
||||
- ✨ Acknowledgement: acknowledgement.md
|
||||
|
||||
extra:
|
||||
social:
|
||||
- icon: simple/huggingface
|
||||
link: https://huggingface.co/collections/HorizonRobotics/embodiedgen
|
||||
- icon: fontawesome/brands/github
|
||||
link: https://github.com/HorizonRobotics/EmbodiedGen
|
||||
- icon: simple/arxiv
|
||||
link: https://arxiv.org/abs/2506.10600
|
||||
- icon: fontawesome/solid/globe
|
||||
link: https://horizonrobotics.github.io/robot_lab/embodied_gen/index.html
|
||||
- icon: fontawesome/brands/youtube
|
||||
link: https://www.youtube.com/watch?v=rG4odybuJRk
|
||||
|
||||
theme:
|
||||
name: material
|
||||
language: en
|
||||
logo: assets/logo.png
|
||||
favicon: assets/logo.png
|
||||
icon:
|
||||
repo: fontawesome/brands/github
|
||||
features:
|
||||
- navigation.instant
|
||||
- navigation.instant.prefetch
|
||||
- navigation.instant.progress
|
||||
- navigation.path
|
||||
- navigation.tabs
|
||||
- navigation.top
|
||||
- search.highlight
|
||||
- content.code.copy
|
||||
- content.action.edit
|
||||
palette:
|
||||
- media: "(prefers-color-scheme: light)"
|
||||
scheme: default
|
||||
primary: brown
|
||||
accent: red
|
||||
toggle:
|
||||
icon: material/weather-sunny
|
||||
name: Switch to dark mode
|
||||
- media: "(prefers-color-scheme: dark)"
|
||||
scheme: slate
|
||||
primary: brown
|
||||
accent: red
|
||||
toggle:
|
||||
icon: material/weather-night
|
||||
name: Switch to light mode
|
||||
|
||||
plugins:
|
||||
- search
|
||||
- mkdocstrings:
|
||||
handlers:
|
||||
python:
|
||||
paths: [embodied_gen]
|
||||
options:
|
||||
show_signature_annotations: true
|
||||
separate_signature: true
|
||||
show_root_toc_entry: false
|
||||
docstring_style: google
|
||||
show_source: true
|
||||
merge_init_into_class: true
|
||||
show_inherited_members: true
|
||||
show_root_heading: true
|
||||
show_root_full_path: true
|
||||
|
||||
- git-revision-date-localized:
|
||||
enable_creation_date: true
|
||||
|
||||
extra_css:
|
||||
- stylesheets/extra.css
|
||||
|
||||
markdown_extensions:
|
||||
- pymdownx.highlight
|
||||
- pymdownx.superfences
|
||||
- admonition
|
||||
|
||||
@ -24,6 +24,10 @@ dev = [
|
||||
"isort",
|
||||
"pytest",
|
||||
"pytest-mock",
|
||||
"mkdocs",
|
||||
"mkdocs-material",
|
||||
"mkdocstrings[python]",
|
||||
"mkdocs-git-revision-date-localized-plugin",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
|
||||
@ -1,2 +1,2 @@
|
||||
[pycodestyle]
|
||||
ignore = E203,W503,E402,E501
|
||||
ignore = E203,W503,E402,E501,E251
|
||||
|
||||
@ -1,6 +1,13 @@
|
||||
from typing import Literal
|
||||
|
||||
import pytest
|
||||
from huggingface_hub import snapshot_download
|
||||
from embodied_gen.data.asset_converter import AssetConverterFactory, AssetType
|
||||
from embodied_gen.data.asset_converter import (
|
||||
AssetConverterFactory,
|
||||
AssetType,
|
||||
SimAssetMapper,
|
||||
cvt_embodiedgen_asset_to_anysim,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
@ -56,3 +63,23 @@ def test_MeshtoUSDConverter(data_dir):
|
||||
|
||||
assert output_file.exists(), f"Output not generated: {output_file}"
|
||||
assert output_file.stat().st_size > 0
|
||||
|
||||
|
||||
def test_cvt_embodiedgen_asset_to_anysim(
|
||||
simulator_name: Literal[
|
||||
"isaacsim",
|
||||
"isaacgym",
|
||||
"genesis",
|
||||
"pybullet",
|
||||
"sapien3",
|
||||
"mujoco",
|
||||
] = "mujoco",
|
||||
):
|
||||
dst_asset_path = cvt_embodiedgen_asset_to_anysim(
|
||||
urdf_files=[
|
||||
"outputs/embodiedgen_assets/demo_assets/remote_control2/result/remote_control.urdf",
|
||||
],
|
||||
target_type=SimAssetMapper[simulator_name],
|
||||
source_type=AssetType.MESH,
|
||||
overwrite=True,
|
||||
)
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user