RoboTwin Code Generation

Branch: gpt |

Installation

Notice: the model files of this branch are different from the main branch, which have different kinds of objects information !!!

Refer to INSTALLATION.md for installation instructions. It takes approximately 20 minutes to complete the installation.

Configure LLM API Key

Please configure the necessary API keys in the ./gpt_api/gpt_agent.py file. Additionally, if the LLM you are utilizing does not support integration with the OpenAI API, you may need to make corresponding adjustments to the generate() function.

Generate Your Task Code

1. Add Task Description

Add new task information, including the task name and natural language description, in ./gpt_api/task_info.py.

2. Add Basic Task Code

Add the basic code file ${task_name}.py in the ./envs/ directory, following the file structure as shown in BASIC_TASK_FILE.

3. Add Task Scene File

Add the scene configuration file ${task_name}.json in the ./task_config/scene_info/ directory, following the file structure as shown in SCENE_CONFIG.

4. Generate the Final Code

Run the following script to generate task code:

python task_generation.py ${task_name}

The generated code file will be ./envs/gpt_${task_name}.py. For example:

python task_generation.py apple_cabinet_storage

The generated code file will be ./envs/gpt_apple_cabinet_storage.py

5. Collect Data

Run the script:

bash run_task.sh ${task_name} {gpu_id}

To collect expert data for the relevant tasks, where ${task_name} corresponds to the task file name in the ./envs/ directory. For example:

bash run_task.sh gpt_apple_cabinet_storage 0

See TASK_CONFIG for data collection configurations.

Directory Structure

.
├── aloha_maniskill_sim            Robot URDF and SRDF files
├── Code-generator Document.md
├── data                           Directory for expert data collection
├── envs
│   ├── base_task.py               Base class for tasks
│   ├── TASK_NAME.py               Task scene generation and success determination
│   ├── gpt_TASK_NAME.py           Code file generated for the task
│   └── utils
├── gpt_api
│   ├── gpt_agent.py                
│   ├── __init__.py
│   ├── prompt.py
│   └── task_info.py               Task information
├── models
│   ├── MODEL_FILE                 Digital asset files
│   │   ├── base{id}.glb           Model files
│   │   └── model_data{id}.json    Model calibration data files
│   ├── ...
│   └── models.py
├── run_task.sh                    Script to run tasks
├── task_config
│   ├── scene_info                 Directory for each task scene data
│   ├── seeds                      Directory for random seeds of expert data for each task
│   ├── TASK_NAME.yml              Task data collection parameters
│   └── ...
├── task_generation.py             Code generation script for tasks
└── ...