Support IMX export and inference for Pose Estimation (#20196)
Signed-off-by: Francesco Mattioli <Francesco.mttl@gmail.com> Co-authored-by: UltralyticsAssistant <web@ultralytics.com> Co-authored-by: Laughing-q <1185102784@qq.com> Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> Co-authored-by: Ultralytics Assistant <135830346+UltralyticsAssistant@users.noreply.github.com> Co-authored-by: Lakshantha Dissanayake <lakshantha@ultralytics.com> Co-authored-by: Jing Qiu <61612323+Laughing-q@users.noreply.github.com>
This commit is contained in:
parent
121eaff005
commit
3d2f56dbb6
@ -35,6 +35,13 @@ The IMX500 works with quantized models. Quantization makes models smaller and fa
|
||||
|
||||
**Before You Begin:** For best results, ensure your YOLO11 model is well-prepared for export by following our [Model Training Guide](https://docs.ultralytics.com/modes/train/), [Data Preparation Guide](https://docs.ultralytics.com/datasets/), and [Hyperparameter Tuning Guide](https://docs.ultralytics.com/guides/hyperparameter-tuning/).
|
||||
|
||||
## Supported Tasks
|
||||
|
||||
Currently, you can only export models that include the following tasks to IMX500 format.
|
||||
|
||||
- [Object detection](https://docs.ultralytics.com/tasks/detect/)
|
||||
- [Pose estimation](https://docs.ultralytics.com/tasks/pose/)
|
||||
|
||||
## Usage Examples
|
||||
|
||||
Export an Ultralytics YOLO11 model to IMX500 format and run inference with the exported model.
|
||||
@ -43,7 +50,7 @@ Export an Ultralytics YOLO11 model to IMX500 format and run inference with the e
|
||||
|
||||
Here we perform inference just to make sure the model works as expected. However, for deployment and inference on the Raspberry Pi AI Camera, please jump to [Using IMX500 Export in Deployment](#using-imx500-export-in-deployment) section.
|
||||
|
||||
!!! example
|
||||
!!! example "Object Detection"
|
||||
|
||||
=== "Python"
|
||||
|
||||
@ -73,6 +80,36 @@ Export an Ultralytics YOLO11 model to IMX500 format and run inference with the e
|
||||
yolo predict model=yolo11n_imx_model source='https://ultralytics.com/images/bus.jpg'
|
||||
```
|
||||
|
||||
!!! example "Pose Estimation"
|
||||
|
||||
=== "Python"
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
|
||||
# Load a YOLO11n-pose PyTorch model
|
||||
model = YOLO("yolo11n-pose.pt")
|
||||
|
||||
# Export the model
|
||||
model.export(format="imx", data="coco8-pose.yaml") # exports with PTQ quantization by default
|
||||
|
||||
# Load the exported model
|
||||
imx_model = YOLO("yolo11n-pose_imx_model")
|
||||
|
||||
# Run inference
|
||||
results = imx_model("https://ultralytics.com/images/bus.jpg")
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
# Export a YOLO11n-pose PyTorch model to imx format with Post-Training Quantization (PTQ)
|
||||
yolo export model=yolo11n-pose.pt format=imx data=coco8-pose.yaml
|
||||
|
||||
# Run inference with the exported model
|
||||
yolo predict model=yolo11n-pose_imx_model source='https://ultralytics.com/images/bus.jpg'
|
||||
```
|
||||
|
||||
!!! warning
|
||||
|
||||
The Ultralytics package installs additional export dependencies at runtime. The first time you run the export command, you may need to restart your console to ensure it works correctly.
|
||||
@ -96,15 +133,31 @@ For more details about the export process, visit the [Ultralytics documentation
|
||||
|
||||
The export process will create an ONNX model for quantization validation, along with a directory named `<model-name>_imx_model`. This directory will include the `packerOut.zip` file, which is essential for packaging the model for deployment on the IMX500 hardware. Additionally, the `<model-name>_imx_model` folder will contain a text file (`labels.txt`) listing all the labels associated with the model.
|
||||
|
||||
```bash
|
||||
yolo11n_imx_model
|
||||
├── dnnParams.xml
|
||||
├── labels.txt
|
||||
├── packerOut.zip
|
||||
├── yolo11n_imx.onnx
|
||||
├── yolo11n_imx500_model_MemoryReport.json
|
||||
└── yolo11n_imx500_model.pbtxt
|
||||
```
|
||||
!!! example "Folder Structure"
|
||||
|
||||
=== "Object Detection"
|
||||
|
||||
```bash
|
||||
yolo11n_imx_model
|
||||
├── dnnParams.xml
|
||||
├── labels.txt
|
||||
├── packerOut.zip
|
||||
├── yolo11n_imx.onnx
|
||||
├── yolo11n_imx500_model_MemoryReport.json
|
||||
└── yolo11n_imx500_model.pbtxt
|
||||
```
|
||||
|
||||
=== "Pose Estimation"
|
||||
|
||||
```bash
|
||||
yolo11n-pose_imx_model
|
||||
├── dnnParams.xml
|
||||
├── labels.txt
|
||||
├── packerOut.zip
|
||||
├── yolo11n-pose_imx.onnx
|
||||
├── yolo11n-pose_imx500_model_MemoryReport.json
|
||||
└── yolo11n-pose_imx500_model.pbtxt
|
||||
```
|
||||
|
||||
## Using IMX500 Export in Deployment
|
||||
|
||||
@ -131,69 +184,127 @@ Step 1: Open a terminal window and execute the following commands to update the
|
||||
sudo apt update && sudo apt full-upgrade
|
||||
```
|
||||
|
||||
Step 2: Install IMX500 firmware which is required to operate the IMX500 sensor along with a packager tool.
|
||||
Step 2: Install IMX500 firmware which is required to operate the IMX500 sensor.
|
||||
|
||||
```bash
|
||||
sudo apt install imx500-all imx500-tools
|
||||
sudo apt install imx500-all
|
||||
```
|
||||
|
||||
Step 3: Install prerequisites to run `picamera2` application. We will use this application later for the deployment process.
|
||||
|
||||
```bash
|
||||
sudo apt install python3-opencv python3-munkres
|
||||
```
|
||||
|
||||
Step 4: Reboot Raspberry Pi for the changes to take into effect
|
||||
Step 3: Reboot Raspberry Pi for the changes to take into effect
|
||||
|
||||
```bash
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
### Package Model and Deploy to AI Camera
|
||||
|
||||
After obtaining `packerOut.zip` from the IMX500 conversion process, you can pass this file into the packager tool to obtain an RPK file. This file can then be deployed directly to the AI Camera using `picamera2`.
|
||||
|
||||
Step 1: Package the model into RPK file
|
||||
Step 4: Install [Aitrios Raspberry Pi application module library](https://github.com/SonySemiconductorSolutions/aitrios-rpi-application-module-library)
|
||||
|
||||
```bash
|
||||
imx500-package -i path/to/packerOut.zip -o path/to/output/folder
|
||||
pip install git+https://github.com/SonySemiconductorSolutions/aitrios-rpi-application-module-library.git
|
||||
```
|
||||
|
||||
The above will generate a `network.rpk` file inside the specified output folder.
|
||||
|
||||
Step 2: Clone `picamera2` repository, install it and navigate to the imx500 examples
|
||||
|
||||
```bash
|
||||
git clone https://github.com/raspberrypi/picamera2
|
||||
cd picamera2
|
||||
pip install -e . --break-system-packages
|
||||
cd examples/imx500
|
||||
```
|
||||
|
||||
Step 3: Run YOLO11 object detection, using the labels.txt file that has been generated during the IMX500 export.
|
||||
|
||||
```bash
|
||||
python imx500_object_detection_demo.py --model path/to/network.rpk --fps 17 --bbox-normalization --ignore-dash-labels --bbox-order xy --labels path/to/labels.txt
|
||||
```
|
||||
|
||||
Then you will be able to see live inference output as follows
|
||||
|
||||
<p align="center">
|
||||
<img width="100%" src="https://github.com/ultralytics/assets/releases/download/v8.3.0/imx500-inference-rpi.avif" alt="Inference on Raspberry Pi AI Camera">
|
||||
</p>
|
||||
|
||||
## Benchmarks
|
||||
|
||||
YOLOv8 and YOLO11n benchmarks below were run by the Ultralytics team on Raspberry Pi AI Camera with `imx` model format measuring speed and accuracy.
|
||||
|
||||
| Model | Format | Status | Size of `RPK` (MB) | mAP50-95(B) | Inference time (ms/im) |
|
||||
| ------- | ------ | ------ | ------------------ | ----------- | ---------------------- |
|
||||
| YOLOv8n | imx | ✅ | 3.1 | 0.433 | 58.82 |
|
||||
| YOLO11n | imx | ✅ | 3.2 | 0.492 | 62.50 |
|
||||
Step 5: Run YOLO11 object detection and pose estimation by using the below scripts which are available in [aitrios-rpi-application-module-library examples](https://github.com/SonySemiconductorSolutions/aitrios-rpi-application-module-library/tree/main/examples/aicam).
|
||||
|
||||
!!! note
|
||||
|
||||
Validation for the above benchmark was done using COCO128 dataset on a Raspberry Pi 5
|
||||
Make sure to replace `model_file` and `labels.txt` directories according to your environment before running these scripts.
|
||||
|
||||
!!! example "Python Scripts"
|
||||
|
||||
=== "Object Detection"
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
from modlib.apps import Annotator
|
||||
from modlib.devices import AiCamera
|
||||
from modlib.models import COLOR_FORMAT, MODEL_TYPE, Model
|
||||
from modlib.models.post_processors import pp_od_yolo_ultralytics
|
||||
|
||||
|
||||
class Yolo(Model):
|
||||
def __init__(self):
|
||||
super().__init__(
|
||||
model_file="yolo11n_imx_model/packerOut.zip", # replace with proper directory
|
||||
model_type=MODEL_TYPE.CONVERTED,
|
||||
color_format=COLOR_FORMAT.RGB,
|
||||
preserve_aspect_ratio=False,
|
||||
)
|
||||
|
||||
self.labels = np.genfromtxt(
|
||||
"yolo11n_imx_model/labels.txt", # replace with proper directory
|
||||
dtype=str,
|
||||
delimiter="\n",
|
||||
)
|
||||
|
||||
def post_process(self, output_tensors):
|
||||
return pp_od_yolo_ultralytics(output_tensors)
|
||||
|
||||
|
||||
device = AiCamera(frame_rate=16) # Optimal frame rate for maximum DPS of the Yolo model running on the AI Camera
|
||||
model = Yolo()
|
||||
device.deploy(model)
|
||||
|
||||
annotator = Annotator()
|
||||
|
||||
with device as stream:
|
||||
for frame in stream:
|
||||
detections = frame.detections[frame.detections.confidence > 0.55]
|
||||
labels = [f"{model.labels[class_id]}: {score:0.2f}" for _, score, class_id, _ in detections]
|
||||
|
||||
annotator.annotate_boxes(frame, detections, labels=labels, alpha=0.3, corner_radius=10)
|
||||
frame.display()
|
||||
```
|
||||
|
||||
=== "Pose Estimation"
|
||||
|
||||
```python
|
||||
from modlib.apps import Annotator
|
||||
from modlib.devices import AiCamera
|
||||
from modlib.models import COLOR_FORMAT, MODEL_TYPE, Model
|
||||
from modlib.models.post_processors import pp_yolo_pose_ultralytics
|
||||
|
||||
|
||||
class YoloPose(Model):
|
||||
def __init__(self):
|
||||
super().__init__(
|
||||
model_file="yolo11n-pose_imx_model/packerOut.zip", # replace with proper directory
|
||||
model_type=MODEL_TYPE.CONVERTED,
|
||||
color_format=COLOR_FORMAT.RGB,
|
||||
preserve_aspect_ratio=False,
|
||||
)
|
||||
|
||||
def post_process(self, output_tensors):
|
||||
return pp_yolo_pose_ultralytics(output_tensors)
|
||||
|
||||
|
||||
device = AiCamera(frame_rate=17) # Optimal frame rate for maximum DPS of the Yolo-pose model running on the AI Camera
|
||||
model = YoloPose()
|
||||
device.deploy(model)
|
||||
|
||||
annotator = Annotator()
|
||||
|
||||
with device as stream:
|
||||
for frame in stream:
|
||||
detections = frame.detections[frame.detections.confidence > 0.4]
|
||||
|
||||
annotator.annotate_keypoints(frame, detections)
|
||||
annotator.annotate_boxes(frame, detections, corner_length=20)
|
||||
frame.display()
|
||||
```
|
||||
|
||||
## Benchmarks
|
||||
|
||||
YOLOv8n, YOLO11n, YOLOv8n-pose and YOLO11n-pose benchmarks below were run by the Ultralytics team on Raspberry Pi AI Camera with `imx` model format measuring speed and accuracy.
|
||||
|
||||
| Model | Format | Status | Size of `packerOut.zip` (MB) | mAP50-95(B) | Inference time (ms/im) |
|
||||
| ------------ | ------ | ------ | ---------------------------- | ----------- | ---------------------- |
|
||||
| YOLOv8n | imx | ✅ | 2.1 | 0.470 | 58.79 |
|
||||
| YOLO11n | imx | ✅ | 2.2 | 0.517 | 58.82 |
|
||||
| YOLOv8n-pose | imx | ✅ | 2.0 | 0.687 | 58.79 |
|
||||
| YOLO11n-pose | imx | ✅ | 2.1 | 0.788 | 62.50 |
|
||||
|
||||
!!! note
|
||||
|
||||
Validation for the above benchmarks were done using COCO128 dataset for detection models and COCO8-Pose dataset for pose estimation models
|
||||
|
||||
## What's Under the Hood?
|
||||
|
||||
@ -276,7 +387,7 @@ model = YOLO("yolo11n.pt")
|
||||
model.export(format="imx") # Exports with PTQ quantization by default
|
||||
```
|
||||
|
||||
The export process will create a directory containing the necessary files for deployment, including `packerOut.zip` which can be used with the IMX500 packager tool on Raspberry Pi.
|
||||
The export process will create a directory containing the necessary files for deployment, including `packerOut.zip`.
|
||||
|
||||
### What are the key benefits of using the IMX500 format for edge AI deployment?
|
||||
|
||||
@ -300,8 +411,7 @@ Hardware:
|
||||
Software:
|
||||
|
||||
- Raspberry Pi OS Bookworm
|
||||
- IMX500 firmware and tools (`sudo apt install imx500-all imx500-tools`)
|
||||
- Python packages for `picamera2` (`sudo apt install python3-opencv python3-munkres`)
|
||||
- IMX500 firmware and tools (`sudo apt install imx500-all`)
|
||||
|
||||
### What performance can I expect from YOLO11 models on the IMX500?
|
||||
|
||||
@ -312,26 +422,3 @@ Based on Ultralytics benchmarks on Raspberry Pi AI Camera:
|
||||
- Model size of only 3.2MB after quantization
|
||||
|
||||
This demonstrates that IMX500 format provides efficient real-time inference while maintaining good accuracy for edge AI applications.
|
||||
|
||||
### How do I package and deploy my exported model to the Raspberry Pi AI Camera?
|
||||
|
||||
After exporting to IMX500 format:
|
||||
|
||||
1. Use the packager tool to create an RPK file:
|
||||
|
||||
```bash
|
||||
imx500-package -i path/to/packerOut.zip -o path/to/output/folder
|
||||
```
|
||||
|
||||
2. Clone and install picamera2:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/raspberrypi/picamera2
|
||||
cd picamera2 && pip install -e . --break-system-packages
|
||||
```
|
||||
|
||||
3. Run inference using the generated RPK file:
|
||||
|
||||
```bash
|
||||
python imx500_object_detection_demo.py --model path/to/network.rpk --fps 17 --bbox-normalization --labels path/to/labels.txt
|
||||
```
|
||||
|
||||
@ -142,7 +142,7 @@ def export_formats():
|
||||
["PaddlePaddle", "paddle", "_paddle_model", True, True, ["batch"]],
|
||||
["MNN", "mnn", ".mnn", True, True, ["batch", "half", "int8"]],
|
||||
["NCNN", "ncnn", "_ncnn_model", True, True, ["batch", "half"]],
|
||||
["IMX", "imx", "_imx_model", True, True, ["int8", "fraction"]],
|
||||
["IMX", "imx", "_imx_model", True, True, ["int8", "fraction", "nms"]],
|
||||
["RKNN", "rknn", "_rknn_model", False, False, ["batch", "name"]],
|
||||
]
|
||||
return dict(zip(["Format", "Argument", "Suffix", "CPU", "GPU", "Arguments"], zip(*x)))
|
||||
@ -313,8 +313,11 @@ class Exporter:
|
||||
if not self.args.int8:
|
||||
LOGGER.warning("IMX export requires int8=True, setting int8=True.")
|
||||
self.args.int8 = True
|
||||
if model.task != "detect":
|
||||
raise ValueError("IMX export only supported for detection models.")
|
||||
if not self.args.nms:
|
||||
LOGGER.warning("IMX export requires nms=True, setting nms=True.")
|
||||
self.args.nms = True
|
||||
if model.task not in {"detect", "pose"}:
|
||||
raise ValueError("IMX export only supported for detection and pose estimation models.")
|
||||
if not hasattr(model, "names"):
|
||||
model.names = default_class_names()
|
||||
model.names = check_class_names(model.names)
|
||||
@ -428,7 +431,7 @@ class Exporter:
|
||||
|
||||
y = None
|
||||
for _ in range(2): # dry runs
|
||||
y = NMSModel(model, self.args)(im) if self.args.nms and not coreml else model(im)
|
||||
y = NMSModel(model, self.args)(im) if self.args.nms and not (coreml or imx) else model(im)
|
||||
if self.args.half and onnx and self.device.type != "cpu":
|
||||
im, model = im.half(), model.half() # to FP16
|
||||
|
||||
@ -1166,15 +1169,13 @@ class Exporter:
|
||||
)
|
||||
if getattr(self.model, "end2end", False):
|
||||
raise ValueError("IMX export is not supported for end2end models.")
|
||||
check_requirements(
|
||||
("model-compression-toolkit>=2.3.0,<2.4.1", "sony-custom-layers>=0.3.0", "edge-mdt-tpc>=1.1.0")
|
||||
)
|
||||
check_requirements(("model-compression-toolkit>=2.4.1", "sony-custom-layers>=0.3.0", "edge-mdt-tpc>=1.1.0"))
|
||||
check_requirements("imx500-converter[pt]>=3.16.1") # Separate requirements for imx500-converter
|
||||
|
||||
import model_compression_toolkit as mct
|
||||
import onnx
|
||||
from edgemdt_tpc import get_target_platform_capabilities
|
||||
from sony_custom_layers.pytorch import multiclass_nms
|
||||
from sony_custom_layers.pytorch import multiclass_nms_with_indices
|
||||
|
||||
LOGGER.info(f"\n{prefix} starting export with model_compression_toolkit {mct.__version__}...")
|
||||
|
||||
@ -1198,13 +1199,23 @@ class Exporter:
|
||||
|
||||
bit_cfg = mct.core.BitWidthConfig()
|
||||
if "C2PSA" in self.model.__str__(): # YOLO11
|
||||
layer_names = ["sub", "mul_2", "add_14", "cat_21"]
|
||||
weights_memory = 2585350.2439
|
||||
n_layers = 238 # 238 layers for fused YOLO11n
|
||||
if self.model.task == "detect":
|
||||
layer_names = ["sub", "mul_2", "add_14", "cat_21"]
|
||||
weights_memory = 2585350.2439
|
||||
n_layers = 238 # 238 layers for fused YOLO11n
|
||||
elif self.model.task == "pose":
|
||||
layer_names = ["sub", "mul_2", "add_14", "cat_22", "cat_23", "mul_4", "add_15"]
|
||||
weights_memory = 2437771.67
|
||||
n_layers = 257 # 257 layers for fused YOLO11n-pose
|
||||
else: # YOLOv8
|
||||
layer_names = ["sub", "mul", "add_6", "cat_17"]
|
||||
weights_memory = 2550540.8
|
||||
n_layers = 168 # 168 layers for fused YOLOv8n
|
||||
if self.model.task == "detect":
|
||||
layer_names = ["sub", "mul", "add_6", "cat_17"]
|
||||
weights_memory = 2550540.8
|
||||
n_layers = 168 # 168 layers for fused YOLOv8n
|
||||
elif self.model.task == "pose":
|
||||
layer_names = ["add_7", "mul_2", "cat_19", "mul", "sub", "add_6", "cat_18"]
|
||||
weights_memory = 2482451.85
|
||||
n_layers = 187 # 187 layers for fused YOLO11n-pose
|
||||
|
||||
# Check if the model has the expected number of layers
|
||||
if len(list(self.model.modules())) != n_layers:
|
||||
@ -1251,6 +1262,7 @@ class Exporter:
|
||||
score_threshold: float = 0.001,
|
||||
iou_threshold: float = 0.7,
|
||||
max_detections: int = 300,
|
||||
task: str = "detect",
|
||||
):
|
||||
"""
|
||||
Initialize NMSWrapper with PyTorch Module and NMS parameters.
|
||||
@ -1260,34 +1272,40 @@ class Exporter:
|
||||
score_threshold (float): Score threshold for non-maximum suppression.
|
||||
iou_threshold (float): Intersection over union threshold for non-maximum suppression.
|
||||
max_detections (int): The number of detections to return.
|
||||
task (str): Task type, either 'detect' or 'pose'.
|
||||
"""
|
||||
super().__init__()
|
||||
self.model = model
|
||||
self.score_threshold = score_threshold
|
||||
self.iou_threshold = iou_threshold
|
||||
self.max_detections = max_detections
|
||||
self.task = task
|
||||
|
||||
def forward(self, images):
|
||||
"""Forward pass with model inference and NMS post-processing."""
|
||||
# model inference
|
||||
outputs = self.model(images)
|
||||
|
||||
boxes = outputs[0]
|
||||
scores = outputs[1]
|
||||
nms = multiclass_nms(
|
||||
boxes, scores = outputs[0], outputs[1]
|
||||
nms_outputs = multiclass_nms_with_indices(
|
||||
boxes=boxes,
|
||||
scores=scores,
|
||||
score_threshold=self.score_threshold,
|
||||
iou_threshold=self.iou_threshold,
|
||||
max_detections=self.max_detections,
|
||||
)
|
||||
return nms
|
||||
if self.task == "pose":
|
||||
kpts = outputs[2] # (bs, max_detections, kpts 17*3)
|
||||
out_kpts = torch.gather(kpts, 1, nms_outputs.indices.unsqueeze(-1).expand(-1, -1, kpts.size(-1)))
|
||||
return nms_outputs.boxes, nms_outputs.scores, nms_outputs.labels, out_kpts
|
||||
return nms_outputs
|
||||
|
||||
quant_model = NMSWrapper(
|
||||
model=quant_model,
|
||||
score_threshold=self.args.conf or 0.001,
|
||||
iou_threshold=self.args.iou,
|
||||
max_detections=self.args.max_det,
|
||||
task=self.model.task,
|
||||
).to(self.device)
|
||||
|
||||
f = Path(str(self.file).replace(self.file.suffix, "_imx_model"))
|
||||
|
||||
@ -259,11 +259,7 @@ class AutoBackend(nn.Module):
|
||||
session = onnxruntime.InferenceSession(w, providers=providers)
|
||||
else:
|
||||
check_requirements(
|
||||
[
|
||||
"model-compression-toolkit>=2.3.0,<2.4.1",
|
||||
"sony-custom-layers[torch]>=0.3.0",
|
||||
"onnxruntime-extensions",
|
||||
]
|
||||
["model-compression-toolkit>=2.4.1", "sony-custom-layers[torch]>=0.3.0", "onnxruntime-extensions"]
|
||||
)
|
||||
w = next(Path(w).glob("*.onnx"))
|
||||
LOGGER.info(f"Loading {w} for ONNX IMX inference...")
|
||||
@ -273,7 +269,6 @@ class AutoBackend(nn.Module):
|
||||
session_options = mctq.get_ort_session_options()
|
||||
session_options.enable_mem_reuse = False # fix the shape mismatch from onnxruntime
|
||||
session = onnxruntime.InferenceSession(w, session_options, providers=["CPUExecutionProvider"])
|
||||
task = "detect"
|
||||
|
||||
output_names = [x.name for x in session.get_outputs()]
|
||||
metadata = session.get_modelmeta().custom_metadata_map
|
||||
@ -674,8 +669,12 @@ class AutoBackend(nn.Module):
|
||||
self.session.run_with_iobinding(self.io)
|
||||
y = self.bindings
|
||||
if self.imx:
|
||||
# boxes, conf, cls
|
||||
y = np.concatenate([y[0], y[1][:, :, None], y[2][:, :, None]], axis=-1)
|
||||
if self.task == "detect":
|
||||
# boxes, conf, cls
|
||||
y = np.concatenate([y[0], y[1][:, :, None], y[2][:, :, None]], axis=-1)
|
||||
elif self.task == "pose":
|
||||
# boxes, conf, kpts
|
||||
y = np.concatenate([y[0], y[1][:, :, None], y[2][:, :, None], y[3]], axis=-1)
|
||||
|
||||
# OpenVINO
|
||||
elif self.xml:
|
||||
|
||||
@ -178,14 +178,10 @@ class Detect(nn.Module):
|
||||
grid_size = torch.tensor([grid_w, grid_h, grid_w, grid_h], device=box.device).reshape(1, 4, 1)
|
||||
norm = self.strides / (self.stride[0] * grid_size)
|
||||
dbox = self.decode_bboxes(self.dfl(box) * norm, self.anchors.unsqueeze(0) * norm[:, :2])
|
||||
elif self.export and self.format == "imx":
|
||||
dbox = self.decode_bboxes(
|
||||
self.dfl(box) * self.strides, self.anchors.unsqueeze(0) * self.strides, xywh=False
|
||||
)
|
||||
return dbox.transpose(1, 2), cls.sigmoid().permute(0, 2, 1)
|
||||
else:
|
||||
dbox = self.decode_bboxes(self.dfl(box), self.anchors.unsqueeze(0)) * self.strides
|
||||
|
||||
if self.export and self.format == "imx":
|
||||
return dbox.transpose(1, 2), cls.sigmoid().permute(0, 2, 1)
|
||||
return torch.cat((dbox, cls.sigmoid()), 1)
|
||||
|
||||
def bias_init(self):
|
||||
@ -384,6 +380,8 @@ class Pose(Detect):
|
||||
if self.training:
|
||||
return x, kpt
|
||||
pred_kpt = self.kpts_decode(bs, kpt)
|
||||
if self.export and self.format == "imx":
|
||||
return (*x, pred_kpt.permute(0, 2, 1))
|
||||
return torch.cat([x, pred_kpt], 1) if self.export else (torch.cat([x[0], pred_kpt], 1), (x[1], kpt))
|
||||
|
||||
def kpts_decode(self, bs: int, kpts: torch.Tensor) -> torch.Tensor:
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user