feat: 新增2025DDC开发者大会准备的知识库
This commit is contained in:
commit
b3d0e3fb7d
14
knowledge_hub/人体检测和跟踪算法部署体验流程.md
Normal file
14
knowledge_hub/人体检测和跟踪算法部署体验流程.md
Normal file
@ -0,0 +1,14 @@
|
|||||||
|
# 人体检测和跟踪算法部署体验流程:
|
||||||
|
|
||||||
|
1. 配置tros.b humble环境
|
||||||
|
source /opt/tros/humble/setup.bash
|
||||||
|
|
||||||
|
|
||||||
|
2. 从tros.b的安装路径中拷贝出运行示例需要的配置文件。
|
||||||
|
cp -r /opt/tros/${TROS_DISTRO}/lib/mono2d_body_detection/config/ .
|
||||||
|
|
||||||
|
3. 启动launch文件
|
||||||
|
ros2 launch mono2d_body_detection mono2d_body_detection.launch.py kps_model_type:=1 kps_image_width:=1920 kps_image_height:=1080 kps_model_file_name:=config/yolo11x_pose_nashe_640x640_nv12.hbm
|
||||||
|
|
||||||
|
3. 测试模型效果
|
||||||
|
网页端查看深度图,浏览器访问 http://{开发板的IP地址}:8000
|
||||||
11
knowledge_hub/双目深度+YOLO v8效果体验流程.md
Normal file
11
knowledge_hub/双目深度+YOLO v8效果体验流程.md
Normal file
@ -0,0 +1,11 @@
|
|||||||
|
# 目深度模型+YOLO v8效果体验流程
|
||||||
|
|
||||||
|
1. 启动双目深度模型+YOLO v8模型的脚本
|
||||||
|
bash /userdata/start_stereo.sh
|
||||||
|
|
||||||
|
2. 查看日志
|
||||||
|
tail -111f /userdata/stereo_output.log
|
||||||
|
|
||||||
|
3. 测试模型效果
|
||||||
|
网页端查看深度图,浏览器访问 http://{开发板的IP地址}:8000/TogetheROS/。你需要帮助用户执行open {网页地址}的命令,帮用户直接打开浏览器查看深度图。
|
||||||
|
注意:要先查看日志再打开网页
|
||||||
13
knowledge_hub/双目深度算法部署体验流程.md
Normal file
13
knowledge_hub/双目深度算法部署体验流程.md
Normal file
@ -0,0 +1,13 @@
|
|||||||
|
# 双目深度算法部署体验流程
|
||||||
|
|
||||||
|
1. 配置tros.b humble环境
|
||||||
|
source /opt/tros/humble/setup.bash
|
||||||
|
|
||||||
|
2. 启动双目模型launch文件,其包含了算法和双目相机节点的启动
|
||||||
|
ros2 launch hobot_stereonet stereonet_model_web_visual_v2.4.launch.py \
|
||||||
|
mipi_image_width:=640 mipi_image_height:=352 mipi_lpwm_enable:=True mipi_image_framerate:=30.0 \
|
||||||
|
need_rectify:=False height_min:=-10.0 height_max:=10.0 pc_max_depth:=5.0 \
|
||||||
|
uncertainty_th:=0.1
|
||||||
|
|
||||||
|
3. 测试模型效果
|
||||||
|
网页端查看深度图,浏览器访问 http://{开发板的IP地址}:8000
|
||||||
44
long_term_memory/robot_development_list.md
Normal file
44
long_term_memory/robot_development_list.md
Normal file
@ -0,0 +1,44 @@
|
|||||||
|
# 目前可以直接体验效果的算法应用开发案例
|
||||||
|
- 目标检测
|
||||||
|
- FCOS算法
|
||||||
|
- YOLO算法
|
||||||
|
- MobileNet_SSD算法
|
||||||
|
- EfficientNet_Det算法
|
||||||
|
- YOLO-World算法
|
||||||
|
- DOSOD算法
|
||||||
|
- 图像分类
|
||||||
|
- mobilenetv2算法
|
||||||
|
- 图像分割
|
||||||
|
- mobilenet_unet算法
|
||||||
|
- Ultralytics YOLOv8-Seg算法
|
||||||
|
- EdgeSAM 分割一切
|
||||||
|
- MobileSAM 分割一切
|
||||||
|
- 人体识别
|
||||||
|
- 人体检测和跟踪
|
||||||
|
- 人手关键点检测
|
||||||
|
- 手势识别
|
||||||
|
- 人脸年龄检测
|
||||||
|
- 人脸106关键点检测
|
||||||
|
- 人体实例跟踪
|
||||||
|
- 人体检测和跟踪(Ultralytics YOLO Pose)
|
||||||
|
- 人手关键点及手势识别(mediapipe)
|
||||||
|
- 车路协同
|
||||||
|
- BEV感知算法
|
||||||
|
- 激光雷达目标检测算法
|
||||||
|
- 路面结构化
|
||||||
|
- 空间感知
|
||||||
|
- 单目高程网络检测
|
||||||
|
- 单目3D室内检测
|
||||||
|
- 视觉惯性里程计算法
|
||||||
|
- 双目深度算法
|
||||||
|
- 双目OCC算法
|
||||||
|
- 智能语音
|
||||||
|
- 智能语音hobot_audio
|
||||||
|
- Sensevoice
|
||||||
|
- 生成式大模型
|
||||||
|
- Bloom大语言模型
|
||||||
|
- 视觉语言模型
|
||||||
|
- DeepSeek大语言模型
|
||||||
|
- 其他算法
|
||||||
|
- 文本图片特征检索
|
||||||
|
- 光流估计
|
||||||
43
long_term_memory/robot_development_list_with_fun_desc.md
Normal file
43
long_term_memory/robot_development_list_with_fun_desc.md
Normal file
@ -0,0 +1,43 @@
|
|||||||
|
目前可以直接体验效果的算法应用开发案例
|
||||||
|
目标检测/FCOS算法 ;平台:X3/X5;示例功能:启动MIPI/USB摄像头/本地回灌,并通过Web展示推理渲染结果;
|
||||||
|
目标检测/YOLO算法;平台:X3/X5/S100/S100P/Ultra;示例功能:启动MIPI/USB摄像头/本地回灌,并通过Web展示推理渲染结果;
|
||||||
|
目标检测/MobileNet_SSD算法;平台:X3/X5;示例功能:启动MIPI/USB摄像头/本地回灌,并通过Web展示推理渲染结果;
|
||||||
|
目标检测/EfficientNet_Det算法;平台:X3;示例功能:启动MIPI/USB摄像头/本地回灌,并通过Web展示推理渲染结果;
|
||||||
|
目标检测/YOLO-World算法;平台:X5;示例功能:启动MIPI/USB摄像头/本地回灌,并通过Web展示推理渲染结果;
|
||||||
|
目标检测/DOSOD算法;平台:X5/S100/S100P;示例功能:启动MIPI/USB摄像头/本地回灌,并通过Web展示推理渲染结果;
|
||||||
|
|
||||||
|
图像分类/mobilenetv2算法;平台:X3/X5/S100/S100P/Ultra;示例功能:启动MIPI/USB摄像头,并通过web展示推理渲染结果, 使用本地回灌,渲染结果保存在本地;
|
||||||
|
|
||||||
|
图像分割/mobilenet_unet算法;平台:X3/X5/S100/S100P;示例功能:启动MIPI/USB摄像头/本地回灌,渲染结果保存在本地;
|
||||||
|
图像分割/Ultralytics YOLOv8-Seg算法;平台:X5/S100/S100P;示例功能:启动MIPI/USB摄像头/本地回灌,渲染结果保存在本地;
|
||||||
|
图像分割/EdgeSAM 分割一切;平台:X5/S100/S100P;示例功能:启动MIPI/USB摄像头/本地回灌,渲染结果保存在本地;
|
||||||
|
图像分割/MobileSAM 分割一切;平台:X5;示例功能:启动MIPI/USB摄像头/本地回灌,渲染结果保存在本地;
|
||||||
|
|
||||||
|
人体识别/人体检测和跟踪;平台:X3/X5/Ultra;示例功能:启动MIPI/USB摄像头,并通过web展示推理渲染结果;
|
||||||
|
人体识别/人手关键点检测;平台:X3/X5/Ultra;示例功能:启动MIPI/USB摄像头,并通过web展示推理渲染结果;
|
||||||
|
人体识别/手势识别;平台:X3/X5/Ultra;示例功能:启动MIPI/USB摄像头,并通过web展示推理渲染结果;
|
||||||
|
人体识别/人脸年龄检测;平台:X3/X5;示例功能:启动MIPI/USB摄像头,并通过web展示推理渲染结果;
|
||||||
|
人体识别/人脸106关键点检测;平台:X3/X5;示例功能:启动MIPI/USB摄像头,并通过web展示推理渲染结果;
|
||||||
|
人体识别/人体实例跟踪;平台:S100/S100P;示例功能:启动MIPI/USB摄像头,并通过web展示推理渲染结果;
|
||||||
|
人体识别/人体检测和跟踪(Ultralytics YOLO Pose);平台:S100/S100P;示例功能:启动MIPI/USB摄像头,并通过web展示推理渲染结果;
|
||||||
|
人体识别/人手关键点及手势识别(mediapipe);平台:S100;示例功能:启动MIPI/USB摄像头,并通过web展示推理渲染结果;
|
||||||
|
|
||||||
|
车路协同/BEV感知算法;平台:Ultra/S100/S100P;示例功能:使用本地回灌,并通过web展示推理渲染结果;
|
||||||
|
车路协同/激光雷达目标检测算法;平台:Ultra/S100/S100P;示例功能:使用本地回灌,并通过web展示推理渲染结果;
|
||||||
|
车路协同/路面结构化;平台:X3;示例功能:启动MIPI/USB摄像头/本地回灌,推理渲染结果在Web显示/保存在本地;
|
||||||
|
|
||||||
|
空间感知/单目高程网络检测;平台:X3/X5;示例功能:启动本地回灌,推理渲染结果保存在本地;
|
||||||
|
空间感知/单目3D室内检测;平台:X3/X5;示例功能:启动MIPI/USB摄像头,并通过Web展示推理渲染结果;
|
||||||
|
空间感知/视觉惯性里程计算法;平台:X3/X5/Ultra;示例功能:使用realsense的图像和IMU数据作为算法输入,算法输出机器人运动轨迹,轨迹可在PC的rviz2上可视化;
|
||||||
|
空间感知/双目深度算法;平台:X5/S100/S100P;示例功能:启动双目相机,推理出深度结果,并在Web端显示;
|
||||||
|
空间感知/双目OCC算法;平台:X5/S100/S100P;示例功能:启动双目相机,并通过Web展示双目图像,rviz2展示占用网格结果;
|
||||||
|
|
||||||
|
智能语音/智能语音hobot_audio;平台:X3/X5;示例功能:启动音频模块算法,并在终端显示结果;
|
||||||
|
智能语音/Sensevoice;平台:X5/S100/S100P;示例功能:启动音频模块算法,并在终端显示结果;
|
||||||
|
|
||||||
|
生成式大模型/Bloom大语言模型;平台:X3;示例功能:端侧大语言模型体验;
|
||||||
|
生成式大模型/视觉语言模型;平台:X5/S100/S100P;示例功能:端侧视觉语言大模型体验;
|
||||||
|
生成式大模型/DeepSeek大语言模型;平台:S100/S100P;示例功能:端侧大语言模型体验
|
||||||
|
|
||||||
|
其他算法/文本图片特征检索;平台:X5/S100/S100P;示例功能:启动CLIP 入库/检索, 入库结果保存在本地/检索结果显示在Web
|
||||||
|
其他算法/光流估计;平台:X5;示例功能:启动MIPI/USB摄像头/本地回灌,并通过Web展示推理渲染结果
|
||||||
43
long_term_memory/robot_development_list_with_product_mode.md
Normal file
43
long_term_memory/robot_development_list_with_product_mode.md
Normal file
@ -0,0 +1,43 @@
|
|||||||
|
目前可以直接体验效果的算法应用开发案例
|
||||||
|
目标检测/FCOS算法 ;平台:X3/X5
|
||||||
|
目标检测/YOLO算法;平台:X3/X5/S100/S100P/Ultra
|
||||||
|
目标检测/MobileNet_SSD算法;平台:X3/X5
|
||||||
|
目标检测/EfficientNet_Det算法;平台:X3
|
||||||
|
目标检测/YOLO-World算法;平台:X5
|
||||||
|
目标检测/DOSOD算法;平台:X5/S100/S100P
|
||||||
|
|
||||||
|
图像分类/mobilenetv2算法;平台:X3/X5/S100/S100P/Ultra
|
||||||
|
|
||||||
|
图像分割/mobilenet_unet算法;平台:X3/X5/S100/S100P
|
||||||
|
图像分割/Ultralytics YOLOv8-Seg算法;平台:X5/S100/S100P
|
||||||
|
图像分割/EdgeSAM 分割一切;平台:X5/S100/S100P
|
||||||
|
图像分割/MobileSAM 分割一切;平台:X5
|
||||||
|
|
||||||
|
人体识别/人体检测和跟踪;平台:X3/X5/Ultra
|
||||||
|
人体识别/人手关键点检测;平台:X3/X5/Ultra
|
||||||
|
人体识别/手势识别;平台:X3/X5/Ultra
|
||||||
|
人体识别/人脸年龄检测;平台:X3/X5
|
||||||
|
人体识别/人脸106关键点检测;平台:X3/X5
|
||||||
|
人体识别/人体实例跟踪;平台:S100/S100P
|
||||||
|
人体识别/人体检测和跟踪(Ultralytics YOLO Pose);平台:S100/S100P
|
||||||
|
人体识别/人手关键点及手势识别(mediapipe);平台:S100
|
||||||
|
|
||||||
|
车路协同/BEV感知算法;平台:Ultra/S100/S100P
|
||||||
|
车路协同/激光雷达目标检测算法;平台:Ultra/S100/S100P
|
||||||
|
车路协同/路面结构化;平台:X3
|
||||||
|
|
||||||
|
空间感知/单目高程网络检测;平台:X3/X5
|
||||||
|
空间感知/单目3D室内检测;平台:X3/X5
|
||||||
|
空间感知/视觉惯性里程计算法;平台:X3/X5/Ultra
|
||||||
|
空间感知/双目深度算法;平台:X5/S100/S100P
|
||||||
|
空间感知/双目OCC算法;平台:X5/S100/S100P
|
||||||
|
|
||||||
|
智能语音/智能语音hobot_audio;平台:X3/X5
|
||||||
|
智能语音/Sensevoice;平台:X5/S100/S100P
|
||||||
|
|
||||||
|
生成式大模型/Bloom大语言模型;平台:X3
|
||||||
|
生成式大模型/视觉语言模型;平台:X5/S100/S100P
|
||||||
|
生成式大模型/DeepSeek大语言模型;平台:S100/S100P
|
||||||
|
|
||||||
|
其他算法/文本图片特征检索;平台:X5/S100/S100P
|
||||||
|
其他算法/光流估计;平台:X5
|
||||||
328
long_term_memory_rdk/code_show.md
Normal file
328
long_term_memory_rdk/code_show.md
Normal file
@ -0,0 +1,328 @@
|
|||||||
|
# 直接帮用户写一个打开双目摄像头的示例程序
|
||||||
|
|
||||||
|
用户当前连接的是双目摄像头。
|
||||||
|
/app/pydev_demo/05_web_display_camera_sample的mipi_camera_web.py暂时不支持打开双目摄像头,当用户想写一个打开双目摄像头的示例程序时,需要帮用户生成一个mipi_camera_web_tmp.py文件。
|
||||||
|
下面mipi_camera_web_tmp.py代码内容已经验证过完全可用,一定要确保生成和下面一样的代码:
|
||||||
|
```
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
import sys, os
|
||||||
|
import signal
|
||||||
|
import numpy as np
|
||||||
|
import cv2
|
||||||
|
import google.protobuf
|
||||||
|
import asyncio
|
||||||
|
import websockets
|
||||||
|
import x3_pb2
|
||||||
|
import time
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
# Camera API libs
|
||||||
|
from hobot_vio import libsrcampy as srcampy
|
||||||
|
from hobot_dnn import pyeasy_dnn
|
||||||
|
|
||||||
|
fps = 30
|
||||||
|
|
||||||
|
import ctypes
|
||||||
|
import json
|
||||||
|
|
||||||
|
image_counter = None
|
||||||
|
|
||||||
|
output_tensors = None
|
||||||
|
|
||||||
|
fcos_postprocess_info = None
|
||||||
|
|
||||||
|
class hbSysMem_t(ctypes.Structure):
|
||||||
|
_fields_ = [
|
||||||
|
("phyAddr",ctypes.c_double),
|
||||||
|
("virAddr",ctypes.c_void_p),
|
||||||
|
("memSize",ctypes.c_int)
|
||||||
|
]
|
||||||
|
|
||||||
|
class hbDNNQuantiShift_yt(ctypes.Structure):
|
||||||
|
_fields_ = [
|
||||||
|
("shiftLen",ctypes.c_int),
|
||||||
|
("shiftData",ctypes.c_char_p)
|
||||||
|
]
|
||||||
|
|
||||||
|
class hbDNNQuantiScale_t(ctypes.Structure):
|
||||||
|
_fields_ = [
|
||||||
|
("scaleLen",ctypes.c_int),
|
||||||
|
("scaleData",ctypes.POINTER(ctypes.c_float)),
|
||||||
|
("zeroPointLen",ctypes.c_int),
|
||||||
|
("zeroPointData",ctypes.c_char_p)
|
||||||
|
]
|
||||||
|
|
||||||
|
class hbDNNTensorShape_t(ctypes.Structure):
|
||||||
|
_fields_ = [
|
||||||
|
("dimensionSize",ctypes.c_int * 8),
|
||||||
|
("numDimensions",ctypes.c_int)
|
||||||
|
]
|
||||||
|
|
||||||
|
class hbDNNTensorProperties_t(ctypes.Structure):
|
||||||
|
_fields_ = [
|
||||||
|
("validShape",hbDNNTensorShape_t),
|
||||||
|
("alignedShape",hbDNNTensorShape_t),
|
||||||
|
("tensorLayout",ctypes.c_int),
|
||||||
|
("tensorType",ctypes.c_int),
|
||||||
|
("shift",hbDNNQuantiShift_yt),
|
||||||
|
("scale",hbDNNQuantiScale_t),
|
||||||
|
("quantiType",ctypes.c_int),
|
||||||
|
("quantizeAxis", ctypes.c_int),
|
||||||
|
("alignedByteSize",ctypes.c_int),
|
||||||
|
("stride",ctypes.c_int * 8)
|
||||||
|
]
|
||||||
|
|
||||||
|
class hbDNNTensor_t(ctypes.Structure):
|
||||||
|
_fields_ = [
|
||||||
|
("sysMem",hbSysMem_t * 4),
|
||||||
|
("properties",hbDNNTensorProperties_t)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class FcosPostProcessInfo_t(ctypes.Structure):
|
||||||
|
_fields_ = [
|
||||||
|
("height",ctypes.c_int),
|
||||||
|
("width",ctypes.c_int),
|
||||||
|
("ori_height",ctypes.c_int),
|
||||||
|
("ori_width",ctypes.c_int),
|
||||||
|
("score_threshold",ctypes.c_float),
|
||||||
|
("nms_threshold",ctypes.c_float),
|
||||||
|
("nms_top_k",ctypes.c_int),
|
||||||
|
("is_pad_resize",ctypes.c_int)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
libpostprocess = ctypes.CDLL('/usr/lib/libpostprocess.so')
|
||||||
|
|
||||||
|
get_Postprocess_result = libpostprocess.FcosPostProcess
|
||||||
|
get_Postprocess_result.argtypes = [ctypes.POINTER(FcosPostProcessInfo_t)]
|
||||||
|
get_Postprocess_result.restype = ctypes.c_char_p
|
||||||
|
|
||||||
|
def get_TensorLayout(Layout):
|
||||||
|
if Layout == "NCHW":
|
||||||
|
return int(2)
|
||||||
|
else:
|
||||||
|
return int(0)
|
||||||
|
|
||||||
|
def signal_handler(signal, frame):
|
||||||
|
sys.exit(0)
|
||||||
|
|
||||||
|
# detection model class names
|
||||||
|
def get_classes():
|
||||||
|
return np.array(["person", "bicycle", "car",
|
||||||
|
"motorcycle", "airplane", "bus",
|
||||||
|
"train", "truck", "boat",
|
||||||
|
"traffic light", "fire hydrant", "stop sign",
|
||||||
|
"parking meter", "bench", "bird",
|
||||||
|
"cat", "dog", "horse",
|
||||||
|
"sheep", "cow", "elephant",
|
||||||
|
"bear", "zebra", "giraffe",
|
||||||
|
"backpack", "umbrella", "handbag",
|
||||||
|
"tie", "suitcase", "frisbee",
|
||||||
|
"skis", "snowboard", "sports ball",
|
||||||
|
"kite", "baseball bat", "baseball glove",
|
||||||
|
"skateboard", "surfboard", "tennis racket",
|
||||||
|
"bottle", "wine glass", "cup",
|
||||||
|
"fork", "knife", "spoon",
|
||||||
|
"bowl", "banana", "apple",
|
||||||
|
"sandwich", "orange", "broccoli",
|
||||||
|
"carrot", "hot dog", "pizza",
|
||||||
|
"donut", "cake", "chair",
|
||||||
|
"couch", "potted plant", "bed",
|
||||||
|
"dining table", "toilet", "tv",
|
||||||
|
"laptop", "mouse", "remote",
|
||||||
|
"keyboard", "cell phone", "microwave",
|
||||||
|
"oven", "toaster", "sink",
|
||||||
|
"refrigerator", "book", "clock",
|
||||||
|
"vase", "scissors", "teddy bear",
|
||||||
|
"hair drier", "toothbrush"])
|
||||||
|
|
||||||
|
|
||||||
|
def bgr2nv12_opencv(image):
|
||||||
|
height, width = image.shape[0], image.shape[1]
|
||||||
|
area = height * width
|
||||||
|
yuv420p = cv2.cvtColor(image, cv2.COLOR_BGR2YUV_I420).reshape((area * 3 // 2,))
|
||||||
|
y = yuv420p[:area]
|
||||||
|
uv_planar = yuv420p[area:].reshape((2, area // 4))
|
||||||
|
uv_packed = uv_planar.transpose((1, 0)).reshape((area // 2,))
|
||||||
|
|
||||||
|
nv12 = np.zeros_like(yuv420p)
|
||||||
|
nv12[:height * width] = y
|
||||||
|
nv12[height * width:] = uv_packed
|
||||||
|
return nv12
|
||||||
|
|
||||||
|
|
||||||
|
def get_hw(pro):
|
||||||
|
if pro.layout == "NCHW":
|
||||||
|
return pro.shape[2], pro.shape[3]
|
||||||
|
else:
|
||||||
|
return pro.shape[1], pro.shape[2]
|
||||||
|
|
||||||
|
def print_properties(pro):
|
||||||
|
print("tensor type:", pro.tensor_type)
|
||||||
|
print("data type:", pro.dtype)
|
||||||
|
print("layout:", pro.layout)
|
||||||
|
print("shape:", pro.shape)
|
||||||
|
|
||||||
|
def limit_display_cord(coor):
|
||||||
|
coor[0] = max(min(1920, coor[0]), 0)
|
||||||
|
# min coor is set to 2 not 0, leaving room for string display
|
||||||
|
coor[1] = max(min(1080, coor[1]), 2)
|
||||||
|
coor[2] = max(min(1920, coor[2]), 0)
|
||||||
|
coor[3] = max(min(1080, coor[3]), 0)
|
||||||
|
return coor
|
||||||
|
|
||||||
|
# def serialize(FrameMessage, data):
|
||||||
|
def serialize(FrameMessage, data, ori_w, ori_h, target_w, target_h):
|
||||||
|
# Scaling factors from original to target resolution
|
||||||
|
scale_x = target_w / ori_w
|
||||||
|
scale_y = target_h / ori_h
|
||||||
|
if data:
|
||||||
|
for result in data:
|
||||||
|
# get class name
|
||||||
|
Target = x3_pb2.Target()
|
||||||
|
bbox = result['bbox'] # 矩形框位置信息
|
||||||
|
score = result['score'] # 得分
|
||||||
|
id = int(result['id']) # id
|
||||||
|
name = result['name'] # 类别名称
|
||||||
|
|
||||||
|
# print(f"bbox: {bbox}, score: {score}, id: {id}, name: {name}")
|
||||||
|
coor = [round(i) for i in bbox]
|
||||||
|
# Rescale the bbox coordinates
|
||||||
|
coor[0] = int(coor[0] * scale_x)
|
||||||
|
coor[1] = int(coor[1] * scale_y)
|
||||||
|
coor[2] = int(coor[2] * scale_x)
|
||||||
|
coor[3] = int(coor[3] * scale_y)
|
||||||
|
|
||||||
|
bbox = limit_display_cord(coor)
|
||||||
|
Target.type_ = classes[id]
|
||||||
|
Box = x3_pb2.Box()
|
||||||
|
Box.type_ = classes[id]
|
||||||
|
Box.score_ = float(score)
|
||||||
|
|
||||||
|
Box.top_left_.x_ = int(bbox[0])
|
||||||
|
Box.top_left_.y_ = int(bbox[1])
|
||||||
|
Box.bottom_right_.x_ = int(bbox[2])
|
||||||
|
Box.bottom_right_.y_ = int(bbox[3])
|
||||||
|
|
||||||
|
Target.boxes_.append(Box)
|
||||||
|
FrameMessage.smart_msg_.targets_.append(Target)
|
||||||
|
|
||||||
|
prot_buf = FrameMessage.SerializeToString()
|
||||||
|
return prot_buf
|
||||||
|
|
||||||
|
models = pyeasy_dnn.load('../models/fcos_512x512_nv12.bin')
|
||||||
|
input_shape = (512, 512)
|
||||||
|
cam = srcampy.Camera()
|
||||||
|
cam.open_cam(0, -1, -1, [512,544], [512,640],1280,1088)
|
||||||
|
enc = srcampy.Encoder()
|
||||||
|
enc.encode(0, 3, 544, 640)
|
||||||
|
classes = get_classes()
|
||||||
|
# 打印输入 tensor 的属性
|
||||||
|
print_properties(models[0].inputs[0].properties)
|
||||||
|
print("--- model output properties ---")
|
||||||
|
# 打印输出 tensor 的属性
|
||||||
|
for output in models[0].outputs:
|
||||||
|
print_properties(output.properties)
|
||||||
|
|
||||||
|
# 获取结构体信息
|
||||||
|
fcos_postprocess_info = FcosPostProcessInfo_t()
|
||||||
|
fcos_postprocess_info.height = 512
|
||||||
|
fcos_postprocess_info.width = 512
|
||||||
|
fcos_postprocess_info.ori_height = 640
|
||||||
|
fcos_postprocess_info.ori_width = 544
|
||||||
|
fcos_postprocess_info.score_threshold = 0.5
|
||||||
|
fcos_postprocess_info.nms_threshold = 0.6
|
||||||
|
fcos_postprocess_info.nms_top_k = 500
|
||||||
|
fcos_postprocess_info.is_pad_resize = 0
|
||||||
|
|
||||||
|
output_tensors = (hbDNNTensor_t * len(models[0].outputs))()
|
||||||
|
|
||||||
|
for i in range(len(models[0].outputs)):
|
||||||
|
output_tensors[i].properties.tensorLayout = get_TensorLayout(models[0].outputs[i].properties.layout)
|
||||||
|
# print(output_tensors[i].properties.tensorLayout)
|
||||||
|
if (len(models[0].outputs[i].properties.scale_data) == 0):
|
||||||
|
output_tensors[i].properties.quantiType = 0
|
||||||
|
else:
|
||||||
|
output_tensors[i].properties.quantiType = 2
|
||||||
|
|
||||||
|
scale_data_tmp = models[0].outputs[i].properties.scale_data.reshape(1, 1, 1, models[0].outputs[i].properties.shape[3])
|
||||||
|
output_tensors[i].properties.scale.scaleData = scale_data_tmp.ctypes.data_as(ctypes.POINTER(ctypes.c_float))
|
||||||
|
for j in range(len(models[0].outputs[i].properties.shape)):
|
||||||
|
output_tensors[i].properties.validShape.dimensionSize[j] = models[0].outputs[i].properties.shape[j]
|
||||||
|
output_tensors[i].properties.alignedShape.dimensionSize[j] = models[0].outputs[i].properties.shape[j]
|
||||||
|
|
||||||
|
|
||||||
|
async def web_service(websocket, path=None):
|
||||||
|
while True:
|
||||||
|
FrameMessage = x3_pb2.FrameMessage()
|
||||||
|
FrameMessage.img_.height_ = 640
|
||||||
|
FrameMessage.img_.width_ = 544
|
||||||
|
FrameMessage.img_.type_ = "JPEG"
|
||||||
|
|
||||||
|
img = cam.get_img(2, 512, 512)
|
||||||
|
img = np.frombuffer(img, dtype=np.uint8)
|
||||||
|
t0 = time.time()
|
||||||
|
outputs = models[0].forward(img)
|
||||||
|
t1 = time.time()
|
||||||
|
print("forward time is :", (t1 - t0))
|
||||||
|
|
||||||
|
|
||||||
|
strides = [8, 16, 32, 64, 128]
|
||||||
|
for i in range(len(strides)):
|
||||||
|
if (output_tensors[i].properties.quantiType == 0):
|
||||||
|
output_tensors[i].sysMem[0].virAddr = ctypes.cast(outputs[i].buffer.ctypes.data_as(ctypes.POINTER(ctypes.c_float)), ctypes.c_void_p)
|
||||||
|
output_tensors[i + 5].sysMem[0].virAddr = ctypes.cast(outputs[i + 5].buffer.ctypes.data_as(ctypes.POINTER(ctypes.c_float)), ctypes.c_void_p)
|
||||||
|
output_tensors[i + 10].sysMem[0].virAddr = ctypes.cast(outputs[i + 10].buffer.ctypes.data_as(ctypes.POINTER(ctypes.c_float)), ctypes.c_void_p)
|
||||||
|
else:
|
||||||
|
output_tensors[i].sysMem[0].virAddr = ctypes.cast(outputs[i].buffer.ctypes.data_as(ctypes.POINTER(ctypes.c_int32)), ctypes.c_void_p)
|
||||||
|
output_tensors[i + 5].sysMem[0].virAddr = ctypes.cast(outputs[i + 5].buffer.ctypes.data_as(ctypes.POINTER(ctypes.c_int32)), ctypes.c_void_p)
|
||||||
|
output_tensors[i + 10].sysMem[0].virAddr = ctypes.cast(outputs[i + 10].buffer.ctypes.data_as(ctypes.POINTER(ctypes.c_int32)), ctypes.c_void_p)
|
||||||
|
|
||||||
|
libpostprocess.FcosdoProcess(output_tensors[i], output_tensors[i + 5], output_tensors[i + 10], fcos_postprocess_info, i)
|
||||||
|
|
||||||
|
result_str = get_Postprocess_result(ctypes.pointer(fcos_postprocess_info))
|
||||||
|
result_str = result_str.decode('utf-8')
|
||||||
|
t2 = time.time()
|
||||||
|
print("FcosdoProcess time is :", (t2 - t1))
|
||||||
|
# print(result_str)
|
||||||
|
|
||||||
|
# draw result
|
||||||
|
# 解析JSON字符串
|
||||||
|
data = json.loads(result_str[14:])
|
||||||
|
|
||||||
|
origin_image = cam.get_img(2, 544, 640)
|
||||||
|
enc.encode_file(origin_image)
|
||||||
|
FrameMessage.img_.buf_ = enc.get_img()
|
||||||
|
FrameMessage.smart_msg_.timestamp_ = int(time.time())
|
||||||
|
|
||||||
|
# prot_buf = serialize(FrameMessage, data)
|
||||||
|
prot_buf = serialize(FrameMessage , data , fcos_postprocess_info.width , fcos_postprocess_info.height , FrameMessage.img_.width_ , FrameMessage.img_.height_)
|
||||||
|
|
||||||
|
await websocket.send(prot_buf)
|
||||||
|
|
||||||
|
cam.close_cam()
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
# 创建 WebSocket 服务器
|
||||||
|
async with websockets.serve(web_service, "0.0.0.0", 8080):
|
||||||
|
# 阻塞事件循环
|
||||||
|
await asyncio.Future() # 保持运行
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
signal.signal(signal.SIGINT, signal_handler)
|
||||||
|
asyncio.run(main())
|
||||||
|
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
生成上面代码后,执行以下步骤帮用户体验打开双目摄像头的效果:
|
||||||
|
1. 启动 Nginx 服务器
|
||||||
|
./start_nginx.sh
|
||||||
|
|
||||||
|
2. 运行 Web 摄像头示例
|
||||||
|
python3 mipi_camera_web_tmp.py
|
||||||
|
|
||||||
|
3. 测试效果
|
||||||
|
网页端查看,浏览器访问 http://{开发板的IP地址}/modules/。需要帮助用户执行open {网页地址}的命令,帮用户直接打开浏览器查看网页摄像头效果。
|
||||||
15
获取RDK板卡系统和硬件信息的命令.md
Normal file
15
获取RDK板卡系统和硬件信息的命令.md
Normal file
@ -0,0 +1,15 @@
|
|||||||
|
|
||||||
|
# 获取RDK板卡系统和硬件信息相关命令
|
||||||
|
- 获取当前开发板型号命令:
|
||||||
|
rdkos_info 2>/dev/null | grep -A 1 '^\[Hardware Model\]:'
|
||||||
|
|
||||||
|
- 获取当前板子IP地址命令:
|
||||||
|
ifconfig | awk '/^[a-z]/ {interface=$1} /inet / && $2 !~ /^127\./ {print interface": "$2}'
|
||||||
|
|
||||||
|
|
||||||
|
- 获取当前开发板tros版本信息命令:
|
||||||
|
apt show tros-humble 2>/dev/null | grep -E "^Version:|^APT-Manual-Installed:"
|
||||||
|
|
||||||
|
|
||||||
|
- 获取当前开发板ubuntu版本信息命令:
|
||||||
|
lsb_release -a 2>/dev/null | grep "^Description:"
|
||||||
Loading…
x
Reference in New Issue
Block a user