Support IMX export and inference for Pose Estimation (#20196)

Signed-off-by: Francesco Mattioli <Francesco.mttl@gmail.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
Co-authored-by: Laughing-q <1185102784@qq.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: Ultralytics Assistant <135830346+UltralyticsAssistant@users.noreply.github.com>
Co-authored-by: Lakshantha Dissanayake <lakshantha@ultralytics.com>
Co-authored-by: Jing Qiu <61612323+Laughing-q@users.noreply.github.com>
This commit is contained in:
Francesco Mattioli 2025-07-15 06:42:58 +02:00 committed by GitHub
parent 121eaff005
commit 3d2f56dbb6
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 217 additions and 115 deletions

View File

@ -35,6 +35,13 @@ The IMX500 works with quantized models. Quantization makes models smaller and fa
**Before You Begin:** For best results, ensure your YOLO11 model is well-prepared for export by following our [Model Training Guide](https://docs.ultralytics.com/modes/train/), [Data Preparation Guide](https://docs.ultralytics.com/datasets/), and [Hyperparameter Tuning Guide](https://docs.ultralytics.com/guides/hyperparameter-tuning/).
## Supported Tasks
Currently, you can only export models that include the following tasks to IMX500 format.
- [Object detection](https://docs.ultralytics.com/tasks/detect/)
- [Pose estimation](https://docs.ultralytics.com/tasks/pose/)
## Usage Examples
Export an Ultralytics YOLO11 model to IMX500 format and run inference with the exported model.
@ -43,7 +50,7 @@ Export an Ultralytics YOLO11 model to IMX500 format and run inference with the e
Here we perform inference just to make sure the model works as expected. However, for deployment and inference on the Raspberry Pi AI Camera, please jump to [Using IMX500 Export in Deployment](#using-imx500-export-in-deployment) section.
!!! example
!!! example "Object Detection"
=== "Python"
@ -73,6 +80,36 @@ Export an Ultralytics YOLO11 model to IMX500 format and run inference with the e
yolo predict model=yolo11n_imx_model source='https://ultralytics.com/images/bus.jpg'
```
!!! example "Pose Estimation"
=== "Python"
```python
from ultralytics import YOLO
# Load a YOLO11n-pose PyTorch model
model = YOLO("yolo11n-pose.pt")
# Export the model
model.export(format="imx", data="coco8-pose.yaml") # exports with PTQ quantization by default
# Load the exported model
imx_model = YOLO("yolo11n-pose_imx_model")
# Run inference
results = imx_model("https://ultralytics.com/images/bus.jpg")
```
=== "CLI"
```bash
# Export a YOLO11n-pose PyTorch model to imx format with Post-Training Quantization (PTQ)
yolo export model=yolo11n-pose.pt format=imx data=coco8-pose.yaml
# Run inference with the exported model
yolo predict model=yolo11n-pose_imx_model source='https://ultralytics.com/images/bus.jpg'
```
!!! warning
The Ultralytics package installs additional export dependencies at runtime. The first time you run the export command, you may need to restart your console to ensure it works correctly.
@ -96,15 +133,31 @@ For more details about the export process, visit the [Ultralytics documentation
The export process will create an ONNX model for quantization validation, along with a directory named `<model-name>_imx_model`. This directory will include the `packerOut.zip` file, which is essential for packaging the model for deployment on the IMX500 hardware. Additionally, the `<model-name>_imx_model` folder will contain a text file (`labels.txt`) listing all the labels associated with the model.
```bash
yolo11n_imx_model
├── dnnParams.xml
├── labels.txt
├── packerOut.zip
├── yolo11n_imx.onnx
├── yolo11n_imx500_model_MemoryReport.json
└── yolo11n_imx500_model.pbtxt
```
!!! example "Folder Structure"
=== "Object Detection"
```bash
yolo11n_imx_model
├── dnnParams.xml
├── labels.txt
├── packerOut.zip
├── yolo11n_imx.onnx
├── yolo11n_imx500_model_MemoryReport.json
└── yolo11n_imx500_model.pbtxt
```
=== "Pose Estimation"
```bash
yolo11n-pose_imx_model
├── dnnParams.xml
├── labels.txt
├── packerOut.zip
├── yolo11n-pose_imx.onnx
├── yolo11n-pose_imx500_model_MemoryReport.json
└── yolo11n-pose_imx500_model.pbtxt
```
## Using IMX500 Export in Deployment
@ -131,69 +184,127 @@ Step 1: Open a terminal window and execute the following commands to update the
sudo apt update && sudo apt full-upgrade
```
Step 2: Install IMX500 firmware which is required to operate the IMX500 sensor along with a packager tool.
Step 2: Install IMX500 firmware which is required to operate the IMX500 sensor.
```bash
sudo apt install imx500-all imx500-tools
sudo apt install imx500-all
```
Step 3: Install prerequisites to run `picamera2` application. We will use this application later for the deployment process.
```bash
sudo apt install python3-opencv python3-munkres
```
Step 4: Reboot Raspberry Pi for the changes to take into effect
Step 3: Reboot Raspberry Pi for the changes to take into effect
```bash
sudo reboot
```
### Package Model and Deploy to AI Camera
After obtaining `packerOut.zip` from the IMX500 conversion process, you can pass this file into the packager tool to obtain an RPK file. This file can then be deployed directly to the AI Camera using `picamera2`.
Step 1: Package the model into RPK file
Step 4: Install [Aitrios Raspberry Pi application module library](https://github.com/SonySemiconductorSolutions/aitrios-rpi-application-module-library)
```bash
imx500-package -i path/to/packerOut.zip -o path/to/output/folder
pip install git+https://github.com/SonySemiconductorSolutions/aitrios-rpi-application-module-library.git
```
The above will generate a `network.rpk` file inside the specified output folder.
Step 2: Clone `picamera2` repository, install it and navigate to the imx500 examples
```bash
git clone https://github.com/raspberrypi/picamera2
cd picamera2
pip install -e . --break-system-packages
cd examples/imx500
```
Step 3: Run YOLO11 object detection, using the labels.txt file that has been generated during the IMX500 export.
```bash
python imx500_object_detection_demo.py --model path/to/network.rpk --fps 17 --bbox-normalization --ignore-dash-labels --bbox-order xy --labels path/to/labels.txt
```
Then you will be able to see live inference output as follows
<p align="center">
<img width="100%" src="https://github.com/ultralytics/assets/releases/download/v8.3.0/imx500-inference-rpi.avif" alt="Inference on Raspberry Pi AI Camera">
</p>
## Benchmarks
YOLOv8 and YOLO11n benchmarks below were run by the Ultralytics team on Raspberry Pi AI Camera with `imx` model format measuring speed and accuracy.
| Model | Format | Status | Size of `RPK` (MB) | mAP50-95(B) | Inference time (ms/im) |
| ------- | ------ | ------ | ------------------ | ----------- | ---------------------- |
| YOLOv8n | imx | ✅ | 3.1 | 0.433 | 58.82 |
| YOLO11n | imx | ✅ | 3.2 | 0.492 | 62.50 |
Step 5: Run YOLO11 object detection and pose estimation by using the below scripts which are available in [aitrios-rpi-application-module-library examples](https://github.com/SonySemiconductorSolutions/aitrios-rpi-application-module-library/tree/main/examples/aicam).
!!! note
Validation for the above benchmark was done using COCO128 dataset on a Raspberry Pi 5
Make sure to replace `model_file` and `labels.txt` directories according to your environment before running these scripts.
!!! example "Python Scripts"
=== "Object Detection"
```python
import numpy as np
from modlib.apps import Annotator
from modlib.devices import AiCamera
from modlib.models import COLOR_FORMAT, MODEL_TYPE, Model
from modlib.models.post_processors import pp_od_yolo_ultralytics
class Yolo(Model):
def __init__(self):
super().__init__(
model_file="yolo11n_imx_model/packerOut.zip", # replace with proper directory
model_type=MODEL_TYPE.CONVERTED,
color_format=COLOR_FORMAT.RGB,
preserve_aspect_ratio=False,
)
self.labels = np.genfromtxt(
"yolo11n_imx_model/labels.txt", # replace with proper directory
dtype=str,
delimiter="\n",
)
def post_process(self, output_tensors):
return pp_od_yolo_ultralytics(output_tensors)
device = AiCamera(frame_rate=16) # Optimal frame rate for maximum DPS of the Yolo model running on the AI Camera
model = Yolo()
device.deploy(model)
annotator = Annotator()
with device as stream:
for frame in stream:
detections = frame.detections[frame.detections.confidence > 0.55]
labels = [f"{model.labels[class_id]}: {score:0.2f}" for _, score, class_id, _ in detections]
annotator.annotate_boxes(frame, detections, labels=labels, alpha=0.3, corner_radius=10)
frame.display()
```
=== "Pose Estimation"
```python
from modlib.apps import Annotator
from modlib.devices import AiCamera
from modlib.models import COLOR_FORMAT, MODEL_TYPE, Model
from modlib.models.post_processors import pp_yolo_pose_ultralytics
class YoloPose(Model):
def __init__(self):
super().__init__(
model_file="yolo11n-pose_imx_model/packerOut.zip", # replace with proper directory
model_type=MODEL_TYPE.CONVERTED,
color_format=COLOR_FORMAT.RGB,
preserve_aspect_ratio=False,
)
def post_process(self, output_tensors):
return pp_yolo_pose_ultralytics(output_tensors)
device = AiCamera(frame_rate=17) # Optimal frame rate for maximum DPS of the Yolo-pose model running on the AI Camera
model = YoloPose()
device.deploy(model)
annotator = Annotator()
with device as stream:
for frame in stream:
detections = frame.detections[frame.detections.confidence > 0.4]
annotator.annotate_keypoints(frame, detections)
annotator.annotate_boxes(frame, detections, corner_length=20)
frame.display()
```
## Benchmarks
YOLOv8n, YOLO11n, YOLOv8n-pose and YOLO11n-pose benchmarks below were run by the Ultralytics team on Raspberry Pi AI Camera with `imx` model format measuring speed and accuracy.
| Model | Format | Status | Size of `packerOut.zip` (MB) | mAP50-95(B) | Inference time (ms/im) |
| ------------ | ------ | ------ | ---------------------------- | ----------- | ---------------------- |
| YOLOv8n | imx | ✅ | 2.1 | 0.470 | 58.79 |
| YOLO11n | imx | ✅ | 2.2 | 0.517 | 58.82 |
| YOLOv8n-pose | imx | ✅ | 2.0 | 0.687 | 58.79 |
| YOLO11n-pose | imx | ✅ | 2.1 | 0.788 | 62.50 |
!!! note
Validation for the above benchmarks were done using COCO128 dataset for detection models and COCO8-Pose dataset for pose estimation models
## What's Under the Hood?
@ -276,7 +387,7 @@ model = YOLO("yolo11n.pt")
model.export(format="imx") # Exports with PTQ quantization by default
```
The export process will create a directory containing the necessary files for deployment, including `packerOut.zip` which can be used with the IMX500 packager tool on Raspberry Pi.
The export process will create a directory containing the necessary files for deployment, including `packerOut.zip`.
### What are the key benefits of using the IMX500 format for edge AI deployment?
@ -300,8 +411,7 @@ Hardware:
Software:
- Raspberry Pi OS Bookworm
- IMX500 firmware and tools (`sudo apt install imx500-all imx500-tools`)
- Python packages for `picamera2` (`sudo apt install python3-opencv python3-munkres`)
- IMX500 firmware and tools (`sudo apt install imx500-all`)
### What performance can I expect from YOLO11 models on the IMX500?
@ -312,26 +422,3 @@ Based on Ultralytics benchmarks on Raspberry Pi AI Camera:
- Model size of only 3.2MB after quantization
This demonstrates that IMX500 format provides efficient real-time inference while maintaining good accuracy for edge AI applications.
### How do I package and deploy my exported model to the Raspberry Pi AI Camera?
After exporting to IMX500 format:
1. Use the packager tool to create an RPK file:
```bash
imx500-package -i path/to/packerOut.zip -o path/to/output/folder
```
2. Clone and install picamera2:
```bash
git clone https://github.com/raspberrypi/picamera2
cd picamera2 && pip install -e . --break-system-packages
```
3. Run inference using the generated RPK file:
```bash
python imx500_object_detection_demo.py --model path/to/network.rpk --fps 17 --bbox-normalization --labels path/to/labels.txt
```

View File

@ -142,7 +142,7 @@ def export_formats():
["PaddlePaddle", "paddle", "_paddle_model", True, True, ["batch"]],
["MNN", "mnn", ".mnn", True, True, ["batch", "half", "int8"]],
["NCNN", "ncnn", "_ncnn_model", True, True, ["batch", "half"]],
["IMX", "imx", "_imx_model", True, True, ["int8", "fraction"]],
["IMX", "imx", "_imx_model", True, True, ["int8", "fraction", "nms"]],
["RKNN", "rknn", "_rknn_model", False, False, ["batch", "name"]],
]
return dict(zip(["Format", "Argument", "Suffix", "CPU", "GPU", "Arguments"], zip(*x)))
@ -313,8 +313,11 @@ class Exporter:
if not self.args.int8:
LOGGER.warning("IMX export requires int8=True, setting int8=True.")
self.args.int8 = True
if model.task != "detect":
raise ValueError("IMX export only supported for detection models.")
if not self.args.nms:
LOGGER.warning("IMX export requires nms=True, setting nms=True.")
self.args.nms = True
if model.task not in {"detect", "pose"}:
raise ValueError("IMX export only supported for detection and pose estimation models.")
if not hasattr(model, "names"):
model.names = default_class_names()
model.names = check_class_names(model.names)
@ -428,7 +431,7 @@ class Exporter:
y = None
for _ in range(2): # dry runs
y = NMSModel(model, self.args)(im) if self.args.nms and not coreml else model(im)
y = NMSModel(model, self.args)(im) if self.args.nms and not (coreml or imx) else model(im)
if self.args.half and onnx and self.device.type != "cpu":
im, model = im.half(), model.half() # to FP16
@ -1166,15 +1169,13 @@ class Exporter:
)
if getattr(self.model, "end2end", False):
raise ValueError("IMX export is not supported for end2end models.")
check_requirements(
("model-compression-toolkit>=2.3.0,<2.4.1", "sony-custom-layers>=0.3.0", "edge-mdt-tpc>=1.1.0")
)
check_requirements(("model-compression-toolkit>=2.4.1", "sony-custom-layers>=0.3.0", "edge-mdt-tpc>=1.1.0"))
check_requirements("imx500-converter[pt]>=3.16.1") # Separate requirements for imx500-converter
import model_compression_toolkit as mct
import onnx
from edgemdt_tpc import get_target_platform_capabilities
from sony_custom_layers.pytorch import multiclass_nms
from sony_custom_layers.pytorch import multiclass_nms_with_indices
LOGGER.info(f"\n{prefix} starting export with model_compression_toolkit {mct.__version__}...")
@ -1198,13 +1199,23 @@ class Exporter:
bit_cfg = mct.core.BitWidthConfig()
if "C2PSA" in self.model.__str__(): # YOLO11
layer_names = ["sub", "mul_2", "add_14", "cat_21"]
weights_memory = 2585350.2439
n_layers = 238 # 238 layers for fused YOLO11n
if self.model.task == "detect":
layer_names = ["sub", "mul_2", "add_14", "cat_21"]
weights_memory = 2585350.2439
n_layers = 238 # 238 layers for fused YOLO11n
elif self.model.task == "pose":
layer_names = ["sub", "mul_2", "add_14", "cat_22", "cat_23", "mul_4", "add_15"]
weights_memory = 2437771.67
n_layers = 257 # 257 layers for fused YOLO11n-pose
else: # YOLOv8
layer_names = ["sub", "mul", "add_6", "cat_17"]
weights_memory = 2550540.8
n_layers = 168 # 168 layers for fused YOLOv8n
if self.model.task == "detect":
layer_names = ["sub", "mul", "add_6", "cat_17"]
weights_memory = 2550540.8
n_layers = 168 # 168 layers for fused YOLOv8n
elif self.model.task == "pose":
layer_names = ["add_7", "mul_2", "cat_19", "mul", "sub", "add_6", "cat_18"]
weights_memory = 2482451.85
n_layers = 187 # 187 layers for fused YOLO11n-pose
# Check if the model has the expected number of layers
if len(list(self.model.modules())) != n_layers:
@ -1251,6 +1262,7 @@ class Exporter:
score_threshold: float = 0.001,
iou_threshold: float = 0.7,
max_detections: int = 300,
task: str = "detect",
):
"""
Initialize NMSWrapper with PyTorch Module and NMS parameters.
@ -1260,34 +1272,40 @@ class Exporter:
score_threshold (float): Score threshold for non-maximum suppression.
iou_threshold (float): Intersection over union threshold for non-maximum suppression.
max_detections (int): The number of detections to return.
task (str): Task type, either 'detect' or 'pose'.
"""
super().__init__()
self.model = model
self.score_threshold = score_threshold
self.iou_threshold = iou_threshold
self.max_detections = max_detections
self.task = task
def forward(self, images):
"""Forward pass with model inference and NMS post-processing."""
# model inference
outputs = self.model(images)
boxes = outputs[0]
scores = outputs[1]
nms = multiclass_nms(
boxes, scores = outputs[0], outputs[1]
nms_outputs = multiclass_nms_with_indices(
boxes=boxes,
scores=scores,
score_threshold=self.score_threshold,
iou_threshold=self.iou_threshold,
max_detections=self.max_detections,
)
return nms
if self.task == "pose":
kpts = outputs[2] # (bs, max_detections, kpts 17*3)
out_kpts = torch.gather(kpts, 1, nms_outputs.indices.unsqueeze(-1).expand(-1, -1, kpts.size(-1)))
return nms_outputs.boxes, nms_outputs.scores, nms_outputs.labels, out_kpts
return nms_outputs
quant_model = NMSWrapper(
model=quant_model,
score_threshold=self.args.conf or 0.001,
iou_threshold=self.args.iou,
max_detections=self.args.max_det,
task=self.model.task,
).to(self.device)
f = Path(str(self.file).replace(self.file.suffix, "_imx_model"))

View File

@ -259,11 +259,7 @@ class AutoBackend(nn.Module):
session = onnxruntime.InferenceSession(w, providers=providers)
else:
check_requirements(
[
"model-compression-toolkit>=2.3.0,<2.4.1",
"sony-custom-layers[torch]>=0.3.0",
"onnxruntime-extensions",
]
["model-compression-toolkit>=2.4.1", "sony-custom-layers[torch]>=0.3.0", "onnxruntime-extensions"]
)
w = next(Path(w).glob("*.onnx"))
LOGGER.info(f"Loading {w} for ONNX IMX inference...")
@ -273,7 +269,6 @@ class AutoBackend(nn.Module):
session_options = mctq.get_ort_session_options()
session_options.enable_mem_reuse = False # fix the shape mismatch from onnxruntime
session = onnxruntime.InferenceSession(w, session_options, providers=["CPUExecutionProvider"])
task = "detect"
output_names = [x.name for x in session.get_outputs()]
metadata = session.get_modelmeta().custom_metadata_map
@ -674,8 +669,12 @@ class AutoBackend(nn.Module):
self.session.run_with_iobinding(self.io)
y = self.bindings
if self.imx:
# boxes, conf, cls
y = np.concatenate([y[0], y[1][:, :, None], y[2][:, :, None]], axis=-1)
if self.task == "detect":
# boxes, conf, cls
y = np.concatenate([y[0], y[1][:, :, None], y[2][:, :, None]], axis=-1)
elif self.task == "pose":
# boxes, conf, kpts
y = np.concatenate([y[0], y[1][:, :, None], y[2][:, :, None], y[3]], axis=-1)
# OpenVINO
elif self.xml:

View File

@ -178,14 +178,10 @@ class Detect(nn.Module):
grid_size = torch.tensor([grid_w, grid_h, grid_w, grid_h], device=box.device).reshape(1, 4, 1)
norm = self.strides / (self.stride[0] * grid_size)
dbox = self.decode_bboxes(self.dfl(box) * norm, self.anchors.unsqueeze(0) * norm[:, :2])
elif self.export and self.format == "imx":
dbox = self.decode_bboxes(
self.dfl(box) * self.strides, self.anchors.unsqueeze(0) * self.strides, xywh=False
)
return dbox.transpose(1, 2), cls.sigmoid().permute(0, 2, 1)
else:
dbox = self.decode_bboxes(self.dfl(box), self.anchors.unsqueeze(0)) * self.strides
if self.export and self.format == "imx":
return dbox.transpose(1, 2), cls.sigmoid().permute(0, 2, 1)
return torch.cat((dbox, cls.sigmoid()), 1)
def bias_init(self):
@ -384,6 +380,8 @@ class Pose(Detect):
if self.training:
return x, kpt
pred_kpt = self.kpts_decode(bs, kpt)
if self.export and self.format == "imx":
return (*x, pred_kpt.permute(0, 2, 1))
return torch.cat([x, pred_kpt], 1) if self.export else (torch.cat([x[0], pred_kpt], 1), (x[1], kpt))
def kpts_decode(self, bs: int, kpts: torch.Tensor) -> torch.Tensor: