Shiori

交换机是9790

感谢回复
4台机器架构一致，各使用一张管理网卡和两张IB网卡。
使用ray部署集群
主节点使用命令
ray start --head --port=6397 \
--node-ip-address=10.66.3.19 \
--dashboard-host=0.0.0.0 \
--num-gpus=8
网卡信息和环境变量见截图

一、软硬件信息
1.服务器厂家:
H3C

2.沐曦GPU型号：
C500

3.操作系统内核版本：
Ubuntu 22.04.3 LTS
5.15.0-119-generic

4.是否开启CPU虚拟化：
否

5.mx-smi回显：
mx-smi
mx-smi version: 2.2.6

=================== MetaX System Management Interface Log ===================
Timestamp : Tue Dec 2 17:28:48 2025

Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.2.6 Kernel Mode Driver Version: 2.16.0 |
| MACA Version: 3.0.0.8 BIOS Version: 1.26.1.0 |
|------------------------------------+---------------------+----------------------+
| GPU NAME Persistence-M | Bus-id | GPU-Util sGPU-M |
| Temp Pwr:Usage/Cap Perf | Memory-Usage | GPU-State |
|====================================+=====================+======================|
| 0 MetaX C500 Off | 0000:08:00.0 | 43% Native |
| 44C 145W / 350W P9 | 65102/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 1 MetaX C500 Off | 0000:09:00.0 | 42% Native |
| 44C 152W / 350W P9 | 65358/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 2 MetaX C500 Off | 0000:0e:00.0 | 43% Native |
| 45C 148W / 350W P9 | 65422/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 3 MetaX C500 Off | 0000:11:00.0 | 42% Native |
| 43C 147W / 350W P9 | 65102/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 4 MetaX C500 Off | 0000:32:00.0 | 42% Native |
| 44C 146W / 350W P9 | 65102/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 5 MetaX C500 Off | 0000:38:00.0 | 42% Native |
| 45C 145W / 350W P9 | 65422/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 6 MetaX C500 Off | 0000:3b:00.0 | 43% Native |
| 44C 145W / 350W P9 | 65358/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 7 MetaX C500 Off | 0000:3c:00.0 | 42% Native |
| 45C 146W / 350W P9 | 65102/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+

+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| 0 3419263 ray::RayWorkerW 64192 |
| 1 3419275 ray::RayWorkerW 64448 |
| 2 3419268 ray::RayWorkerW 64512 |
| 3 3419311 ray::RayWorkerW 64192 |
| 4 3419277 ray::RayWorkerW 64192 |
| 5 3419303 ray::RayWorkerW 64512 |
| 6 3419305 ray::RayWorkerW 64448 |
| 7 3419296 ray::RayWorkerW 64192 |
+---------------------------------------------------------------------------------+
二、问题现象
4机32卡推理DeepSeek-R1 671B模型，使用命令
vllm serve /mnt/DeepSeek-R1-0528-bf16 --served-model-name DeepSeek-R1 -pp 4 -tp 8 --trust-remote-code --distributed-executor-backend ray --dtype bfloat16 --port 16397 --max-model-len 4096 --gpu-memory-utilization 0.8

使用镜像
cr.metax-tech.com/public-ai-release/maca/vllm:maca.ai3.1.0.7-torch2.6-py310-ubuntu22.04-amd64

推理过程正常，接口测试正常。
接口长时间请求访问后，主节点显存一直有明显增加，直到打满后报OOM错误。从节点显存使用率一直正常。
需要帮忙排查一下原因或给点排查思路