zbw/yiliao2026
1
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

vlm

Browse Source
This commit is contained in:
Orange
2026-08-05 22:18:14 +08:00
parent b0e52bdb6f
commit ce49a0c5d3
6 changed files with 326 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
my_model/image.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 389 KiB

After

Width:  |  Height:  |  Size: 37 KiB

30
src/vlm_detect/config/vlm_detect.yaml
View File

@@ -1,18 +1,18 @@
# vlm_detect 参数配置
vlm_node:
ros__parameters:
vlm_host: "http://192.168.10.189:8000"
vlm_model: "./OpenGVLab/InternVL3-1B/"
image_topic: "/image_mjpeg"
trigger_topic: "/sign4return"
trigger_sign: 9
result_topic: "/vlm_result"
prompt_text: "描述图片中有一个病人的特征，字数控制在20字以内。"
max_tokens: 100
tts_node: tts_node:
ros__parameters: ros__parameters:
audio_sink: "alsa_output.usb-C-Media_Electronics_Inc._USB_Audio_Device-00.analog-stereo" audio_sink: alsa_output.usb-C-Media_Electronics_Inc._USB_Audio_Device-00.analog-stereo
result_topic: "/vlm_result" result_topic: /vlm_result
tts_speed: 1.5 tts_speed: 1.5
vlm_detect:
ros__parameters:
crop_ratio: 0.45
image_max_dim: 96
image_topic: /image
max_tokens: 30
prompt_text: 忽略白色边框。描述图中医院病房场景：一个人在医院病床上，盖着白色被子，画风为2D动漫插画。对人物外观特征高度抽象，称呼为「一个病人」。30字以内。
result_topic: /vlm_result
temperature: 0.1
trigger_sign: 9
trigger_topic: /sign4return
vlm_host: http://192.168.10.189:8000
vlm_model: /home/wisdom/models/gguf/Qwen2-VL-2B-Instruct-Q4_K_M.gguf

222
src/vlm_detect/dataflow.md Normal file
View File

@@ -0,0 +1,222 @@
# VLM + QR → TTS 数据流关系
## 两条数据流总览
```
┌──────────────┐
│ USB Camera │
│ /dev/video0 │
└──────┬───────┘
│
hobot_usb_cam (car_usb_cam)
│
┌────────────┼────────────┐
│ │ │
/image_mjpeg /image (其他话题)
│ │
▼ ▼
┌─────────────┐ ┌──────────────┐
│ vlm_node │ │ qr_dete_node │
│ (Python) │ │ (C++) │
└──────┬──────┘ └──────┬───────┘
│ │
│ /qr_results
│ │
│ ┌──────▼───────┐
│ │ qr_tts_bridge│
│ │ (Python) │
│ └──────┬───────┘
│ │
▼ ▼
┌────────────────────────────────┐
│ /tts/speak │
│ (origincar_msg/Speak) │
│ │
│ tts_server │
│ Piper TTS + paplay │
└────────────────────────────────┘
```
---
## 数据流 1：VLM 图生文 → TTS
### 触发条件
| 话题 | 类型 | 说明 |
|------|------|------|
| `/sign4return` | `std_msgs/Int32` | `data=9` 触发 VLM 推理 |
### 数据流
```
/image_mjpeg /sign4return (9)
(CompressedImage) (Int32)
│ │
▼ ▼
┌─────────────────────────────────────────────────┐
│ vlm_node │
│ │
│ 1. image_callback(): 缓存最新帧到 self.latest_image │
│ 2. sign_callback(): 收到 data==9 时触发 │
│ 3. process_image(): │
│ a. (可选) 自动裁剪照片区域 (auto_crop) │
│ b. 缩放到 max_dim (默认 96px) │
│ c. JPEG 压缩 (quality=60) │
│ d. Base64 编码 │
│ e. POST → VLM HTTP API (/v1/chat/completions) │
│ f. 返回文字描述 │
│ 4. 发布结果到 /vlm_result │
│ 5. 调用 /tts/speak 服务播报文字 │
└────────────┬──────────────────┬──────────────────┘
│ │
▼ ▼
/vlm_result /tts/speak 服务
(std_msgs/String) (origincar_msg/Speak)
```
### 关键参数
| 参数 | 值 | 说明 |
|------|-----|------|
| `image_max_dim` | 96 | 图片最大边长，越小越快 |
| `max_tokens` | 30 | VLM 生成最大 token 数 |
| `temperature` | 0.1 | 低温度 = 更确定 |
| `auto_crop` | false | CPU 优化，默认关闭自动裁剪 |
| `vlm_host` | `http://192.168.10.189:8000` | VLM 服务地址 (Windows WSL) |
| `vlm_model` | `Qwen2-VL-2B-Instruct-Q4_K_M` | 视觉语言模型 |
---
## 数据流 2：QR 二维码识别 → TTS
### 触发条件
| 话题 | 类型 | 说明 |
|------|------|------|
| `/sign4return` | `std_msgs/Int32` | `data=0` 开启二维码检测，`data=5` 关闭 |
### 数据流
```
/image (摄像头发布)
(CompressedImage)
│
▼
┌──────────────────────────────┐
│ hobot_codec (可选) │
│ NV12 解码 → 共享内存 │
│ /hbmem_img │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────────────────────────┐
│ qr_detect / qr_dete_node (C++) │
│ │
│ 1. 订阅图像话题 │
│ 2. ZBar / OpenCV 二维码检测 │
│ 3. 发布识别结果到 /qr_results (std_msgs/String) │
│ 4. 控制 /sign4return: │
│ - 检测到时发 0 开启 │
│ - 检测后发 5 关闭 │
└────────────┬──────────────────────────────────────┘
│
▼ /qr_results
┌──────────────────────────────────────────────────┐
│ qr_tts_bridge (Python) │
│ │
│ 1. 订阅 /qr_results │
│ 2. 收到二维码文字后调用 /tts/speak 服务 │
└────────────┬──────────────────────────────────────┘
│
▼
/tts/speak 服务
(origincar_msg/Speak)
```
---
## TTS 服务 (两条流共享)
### 服务定义
```
# origincar_msg/srv/Speak
string text # 要播报的文字
---
bool success # 是否成功
string message # 错误信息
```
### 服务端 tts_server
```
/tts/speak 请求 (text)
│
▼
┌─────────────────────────────────────┐
│ tts_server │
│ │
│ 1. Piper TTS 引擎 │
│ 模型: zh_CN-huayan-medium.onnx │
│ 生成 WAV 文件到 /tmp/tts_out.wav│
│ 2. paplay 播放到 USB 音频设备 │
│ 设备: USB Audio Device │
│ 3. 失败回退: espeak-ng (cmn) │
└─────────────────────────────────────┘
```
| 参数 | 值 | 说明 |
|------|-----|------|
| `tts_speed` | 1.5 | 语速倍数 |
| `audio_sink` | USB Audio Device | PulseAudio 音频输出设备 |
| `voice` | zh_CN-huayan-medium | 中文女声 Piper 模型 |
---
## /sign4return 统一控制协议
| 值 | 作用 | 使用者 |
|----|------|--------|
| 0 | 开启二维码检测 | `qr_detection` |
| 5 | 关闭二维码检测 | `qr_detection` |
| 9 | 触发 VLM 图像分析 | `vlm_node` |
| 10 | 切换 Nav2 普通档 | `nav2_profile_tuner` |
| 11 | 切换 Nav2 任务二档 | `nav2_profile_tuner` |
---
## 完整比赛流程 (racing_control 编排)
```
1. 启动各节点 → 机器人开始寻线
2. 摄像头检测到二维码
→ /qr_results = "1"
→ qr_tts_bridge → /tts/speak("检测到一号二维码")
→ /sign4return = 5 (关二维码)
3. 导航到闸门前 (entry_pose)
4. 通过闸门 → 切换赛道
5. VLM 图像分析
→ /sign4return = 9
→ vlm_node 拍照 → VLM 推理 → /tts/speak(图像描述)
6. 导航到目标点 (vlm_pose)
7. 返回起点 (origin_pose)
```
---
## 一键启动命令
```bash
# VLM + TTS (两条流的核心服务)
source /opt/ros/humble/setup.bash && source /home/sunrise/yiliao_ws/install/setup.bash && ros2 launch vlm_detect vlm_detect.launch.py
# QT + TTS 桥接
source /opt/ros/humble/setup.bash && source /home/sunrise/yiliao_ws/install/setup.bash && ros2 launch vlm_detect vlm_detect.launch.py use_qr_tts:=true
# 二维码检测
source /opt/ros/humble/setup.bash && source /home/sunrise/yiliao_ws/install/setup.bash && ros2 launch qr_detection qr_detect.launch.py
# 测试 VLM 推理
source /opt/ros/humble/setup.bash && source /home/sunrise/yiliao_ws/install/setup.bash && ros2 run vlm_detect test_publisher --ros-args -p image_path:="/home/sunrise/yiliao_ws/my_model/image.png"
```

5
src/vlm_detect/launch/vlm_detect.launch.py
View File

@@ -70,10 +70,10 @@ def generate_launch_description():
'vlm_host', default_value='http://192.168.10.189:8000', 'vlm_host', default_value='http://192.168.10.189:8000',
description='VLM 服务器地址') description='VLM 服务器地址')
declare_vlm_model = DeclareLaunchArgument( declare_vlm_model = DeclareLaunchArgument(
'vlm_model', default_value='./OpenGVLab/InternVL3-1B/', 'vlm_model', default_value='/home/wisdom/models/gguf/Qwen2-VL-2B-Instruct-Q4_K_M.gguf',
description='VLM 模型名称') description='VLM 模型名称')
declare_image_topic = DeclareLaunchArgument( declare_image_topic = DeclareLaunchArgument(
'image_topic', default_value='/image_mjpeg', 'image_topic', default_value='/image',
description='输入的压缩图像话题') description='输入的压缩图像话题')
declare_trigger_topic = DeclareLaunchArgument( declare_trigger_topic = DeclareLaunchArgument(
'trigger_topic', default_value='/sign4return', 'trigger_topic', default_value='/sign4return',
@@ -116,7 +116,6 @@ def generate_launch_description():
'trigger_topic': trigger_topic, 'trigger_topic': trigger_topic,
'trigger_sign': trigger_sign, 'trigger_sign': trigger_sign,
'result_topic': result_topic, 'result_topic': result_topic,
'prompt_text': prompt_text,
'max_tokens': max_tokens, 'max_tokens': max_tokens,
}], }],
) )

2
src/vlm_detect/vlm_detect/test_publisher.py
View File

@@ -27,7 +27,7 @@ class TestPublisher(Node):
interval = self.get_parameter("interval").value interval = self.get_parameter("interval").value
# 发布者 # 发布者
self.image_pub = self.create_publisher(CompressedImage, "/image_mjpeg", 10) self.image_pub = self.create_publisher(CompressedImage, "/image", 10)
self.sign_pub = self.create_publisher(Int32, "/sign4return", 10) self.sign_pub = self.create_publisher(Int32, "/sign4return", 10)
# 订阅结果 # 订阅结果

179
src/vlm_detect/vlm_detect/vlm_node.py
View File

@@ -1,37 +1,33 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""
VLM 图生文节点 —— 收到触发信号后拍图发给 VLM 服务，结果调用 TTS 服务播报
"""
import rclpy import rclpy
from rclpy.node import Node from rclpy.node import Node
from std_msgs.msg import Int32, String from std_msgs.msg import Int32, String
from sensor_msgs.msg import CompressedImage from sensor_msgs.msg import CompressedImage
from cv_bridge import CvBridge
import cv2 import cv2
import base64 import base64
import threading import threading
from openai import OpenAI from openai import OpenAI
import os
import time import time
import numpy as np import numpy as np
from origincar_msg.srv import Speak from origincar_msg.srv import Speak
class VLMProcessor(Node): class VLMProcessor(Node):
def __init__(self): def __init__(self):
super().__init__('vlm_detect') super().__init__('vlm_detect')
# 声明 ROS2 参数
self.declare_parameter('vlm_host', 'http://192.168.10.189:8000') self.declare_parameter('vlm_host', 'http://192.168.10.189:8000')
self.declare_parameter('vlm_model', './OpenGVLab/InternVL3-1B/') self.declare_parameter('vlm_model', '/home/wisdom/models/gguf/Qwen2-VL-2B-Instruct-Q4_K_M.gguf')
self.declare_parameter('image_topic', '/image_mjpeg') self.declare_parameter('image_topic', '/image_mjpeg')
self.declare_parameter('trigger_topic', '/sign4return') self.declare_parameter('trigger_topic', '/sign4return')
self.declare_parameter('trigger_sign', 9) self.declare_parameter('trigger_sign', 9)
self.declare_parameter('result_topic', '/vlm_result') self.declare_parameter('result_topic', '/vlm_result')
self.declare_parameter('prompt_text', '请描述这张图片的内容，用一句简短的话概括，不超过20个字。') self.declare_parameter('prompt_text', '')
self.declare_parameter('max_tokens', 100) self.declare_parameter('max_tokens', 30)
self.declare_parameter('image_max_dim', 96)
self.declare_parameter('temperature', 0.1)
self.declare_parameter('crop_ratio', 0.45)
self.declare_parameter('auto_crop', False)
vlm_host = self.get_parameter('vlm_host').value vlm_host = self.get_parameter('vlm_host').value
vlm_model = self.get_parameter('vlm_model').value vlm_model = self.get_parameter('vlm_model').value
@@ -41,111 +37,108 @@ class VLMProcessor(Node):
result_topic = self.get_parameter('result_topic').value result_topic = self.get_parameter('result_topic').value
self.prompt_text = self.get_parameter('prompt_text').value self.prompt_text = self.get_parameter('prompt_text').value
self.max_tokens = self.get_parameter('max_tokens').value self.max_tokens = self.get_parameter('max_tokens').value
self.image_max_dim = self.get_parameter('image_max_dim').value
self.temperature = self.get_parameter('temperature').value
self.crop_ratio = self.get_parameter('crop_ratio').value
self.auto_crop = self.get_parameter('auto_crop').value
# 初始化 OpenAI 客户端 self.client = OpenAI(base_url=f"{vlm_host}/v1", api_key="EMPTY")
self.client = OpenAI(
base_url=f"{vlm_host}/v1",
api_key="EMPTY",
)
self.vlm_model = vlm_model self.vlm_model = vlm_model
# ROS2 通信
self.bridge = CvBridge()
self.latest_image = None self.latest_image = None
self.image_lock = threading.Lock() self.image_lock = threading.Lock()
self._busy = False
self.image_sub = self.create_subscription( self.image_sub = self.create_subscription(CompressedImage, image_topic, self.image_callback, 10)
CompressedImage, image_topic, self.image_callback, 10 self.sign_sub = self.create_subscription(Int32, trigger_topic, self.sign_callback, 10)
)
self.sign_sub = self.create_subscription(
Int32, trigger_topic, self.sign_callback, 10
)
self.result_pub = self.create_publisher(String, result_topic, 10) self.result_pub = self.create_publisher(String, result_topic, 10)
# TTS 服务客户端
self.tts_client = self.create_client(Speak, '/tts/speak') self.tts_client = self.create_client(Speak, '/tts/speak')
while not self.tts_client.wait_for_service(timeout_sec=5.0): while not self.tts_client.wait_for_service(timeout_sec=5.0):
self.get_logger().info('Waiting for TTS service...') self.get_logger().info('Waiting for TTS service...')
self.get_logger().info( self.get_logger().info(f"VLM ready | host={vlm_host} | dim={self.image_max_dim} | crop={self.auto_crop}")
f"VLM Processor 就绪 | host={vlm_host} | model={vlm_model} | "
f"image={image_topic} | trigger={trigger_topic}(sign={self.trigger_sign})"
)
def image_callback(self, msg): def image_callback(self, msg):
with self.image_lock: with self.image_lock:
try: try:
np_arr = np.frombuffer(msg.data, np.uint8) np_arr = np.frombuffer(msg.data, np.uint8)
cv_image = cv2.imdecode(np_arr, cv2.IMREAD_COLOR) self.latest_image = cv2.imdecode(np_arr, cv2.IMREAD_COLOR)
self.latest_image = cv_image
self.get_logger().debug("图片已接收")
except Exception as e: except Exception as e:
self.get_logger().error(f"图片接收错误: {e}") self.get_logger().error(f"Decode error: {e}")
def sign_callback(self, msg): def sign_callback(self, msg):
if msg.data == self.trigger_sign: if msg.data != self.trigger_sign:
self.get_logger().info(f"收到触发信号 ({msg.data}), 开始推理...") return
with self.image_lock: if self._busy:
if self.latest_image is None: self.get_logger().warn("Busy, skip")
self.get_logger().warning("无可用图片") return
return self.get_logger().info(f"Trigger {msg.data}")
temp_path = "/tmp/vlm_temp_image.jpg" with self.image_lock:
cv2.imwrite(temp_path, self.latest_image) if self.latest_image is None:
self.get_logger().info(f"临时图片已保存: {temp_path}") self.get_logger().warning("No image")
return
try: img = self.latest_image.copy()
description = self.process_image(temp_path) self._busy = True
self.get_logger().info(f"图像描述: {description}")
# 发布结果到话题
result_msg = String()
result_msg.data = description
self.result_pub.publish(result_msg)
# 调用 TTS 服务播报
if self.tts_client.service_is_ready():
req = Speak.Request()
req.text = description
future = self.tts_client.call_async(req)
future.add_done_callback(self._tts_done_callback)
else:
self.get_logger().warning('TTS service not available')
os.remove(temp_path)
except Exception as e:
self.get_logger().error(f"图像推理失败: {e}")
def _tts_done_callback(self, future):
try: try:
resp = future.result() t0 = time.time()
if resp.success: desc = self.process_image(img)
self.get_logger().debug(f'TTS OK: {resp.message}') self.get_logger().info(f"Result({time.time()-t0:.1f}s): {desc}")
else: msg = String()
self.get_logger().warning(f'TTS failed: {resp.message}') msg.data = desc
self.result_pub.publish(msg)
if self.tts_client.service_is_ready():
req = Speak.Request()
req.text = desc
self.tts_client.call_async(req)
except Exception as e: except Exception as e:
self.get_logger().error(f'TTS call error: {e}') self.get_logger().error(f"Inference: {e}")
finally:
self._busy = False
def process_image(self, image_path): def process_image(self, img):
with open(image_path, "rb") as image_file: if self.auto_crop:
base64_image = base64.b64encode(image_file.read()).decode('utf-8') img = self._detect_and_crop(img)
h, w = img.shape[:2]
start_time = time.time() max_dim = self.image_max_dim
response = self.client.chat.completions.create( if max(h, w) > max_dim:
scale = max_dim / max(h, w)
new_w, new_h = int(w * scale), int(h * scale)
img = cv2.resize(img, (new_w, new_h), interpolation=cv2.INTER_AREA)
else:
new_w, new_h = w, h
_, jpeg = cv2.imencode('.jpg', img, [cv2.IMWRITE_JPEG_QUALITY, 60])
b64 = base64.b64encode(jpeg.tobytes()).decode('utf-8')
t0 = time.time()
resp = self.client.chat.completions.create(
model=self.vlm_model, model=self.vlm_model,
messages=[{ messages=[{"role": "user", "content": [
"role": "user", {"type": "text", "text": self.prompt_text},
"content": [ {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
{"type": "text", "text": self.prompt_text}, ]}],
{"type": "image_url", "image_url": { max_tokens=self.max_tokens, temperature=self.temperature, timeout=60)
"url": f"data:image/jpeg;base64,{base64_image}" self.get_logger().info(f"API {time.time()-t0:.1f}s {new_w}x{new_h} {len(jpeg)//1024}KB")
}}, return resp.choices[0].message.content
]
}], def _detect_and_crop(self, img):
max_tokens=self.max_tokens, h, w = img.shape[:2]
timeout=30, gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
) blurred = cv2.GaussianBlur(gray, (5, 5), 0)
self.get_logger().info(f"VLM 推理耗时 {time.time() - start_time:.1f}s") edges = cv2.Canny(blurred, 50, 150)
return response.choices[0].message.content kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3))
edges = cv2.dilate(edges, kernel, iterations=1)
contours, _ = cv2.findContours(edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
best_rect, best_area = None, 0
for cnt in contours:
peri = cv2.arcLength(cnt, True)
approx = cv2.approxPolyDP(cnt, 0.02 * peri, True)
area = cv2.contourArea(approx)
if len(approx) == 4 and area > w * h * 0.02 and area > best_area:
best_rect, best_area = approx, area
if best_rect is not None:
rx, ry, rw, rh = cv2.boundingRect(best_rect)
self.get_logger().info(f'Crop: {rw}x{rh}')
return img[ry:ry + rh, rx:rx + rw]
return img
def main(args=None): def main(args=None):