一、软硬件信息
1. 服务器厂家:浪潮
2. 沐曦GPU型号:C500
3. 操作系统内核版本:Linux 10-7-182-232 6.14.0-27-generic #27~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 17:38:49 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
4. 是否开启CPU虚拟化:否
5. mx-smi回显:
Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.2.9 Kernel Mode Driver Version: 3.4.4 |
| MACA Version: 3.3.0.15 BIOS Version: 1.20.3.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C500 | 0 Off | 0000:08:00.0 | 60% Disabled |
| 159W / 350W | 47C P9 | 54052/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C500 | 1 Off | 0000:09:00.0 | 59% Disabled |
| 151W / 350W | 45C P9 | 54052/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C500 | 2 Off | 0000:0e:00.0 | 60% Disabled |
| 147W / 350W | 46C P9 | 54052/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C500 | 3 Off | 0000:11:00.0 | 60% Disabled |
| 149W / 350W | 47C P9 | 54052/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C500 | 4 Off | 0000:32:00.0 | 0% Enabled |
| 70W / 350W | 38C P9 | 5147/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C500 | 5 Off | 0000:38:00.0 | 0% Disabled |
| 75W / 350W | 40C P9 | 56795/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C500 | 6 Off | 0000:3b:00.0 | 0% Disabled |
| 54W / 350W | 40C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C500 | 7 Off | 0000:3c:00.0 | 0% Disabled |
| 71W / 350W | 41C P9 | 51355/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
6. docker info回显:
root@10-7-182-232:/home/srcb# docker version
Client: Docker Engine - Community
Version: 29.9.1
API version: 1.56
Go version: go1.26.9
Git commit: dfc520d
Built: Wed Oct 14 18:32:55 2026
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 29.9.1
API version: 1.56 (minimum version 1.40)
Go version: go1.26.9
Git commit: b26fb77
Built: Wed Oct 14 18:32:55 2026
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v2.4.2
GitCommit: b7a25261e029d68978bbf290df0a62fc819b3c51
runc:
Version: 1.5.2
GitCommit: v1.5.2-0-g29dd3dc2
docker-init:
Version: 0.19.0
GitCommit: de40ad0
7. 镜像版本:
cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.28.0-maca.ai3.8.2.401-torch2.10-py310-ubuntu22.04-amd64
8. 启动容器命令:
docker run -itd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --device=/dev/dri/renderD130 --device=/dev/dri/renderD131 --device=/dev/mxcd --group-add video -p 8790:8790 -p 8791:8791 -v /data/LLM:/workspace --shm-size=256g --name vllm-server --restart unless-stopped cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.28.0-maca.ai3.8.2.401-torch2.10-py310-ubuntu22.04-amd64 /bin/bash
9. 容器内执行命令:
vllm serve --model /workspace/models/Qwen3.8-27b \
--gpu-memory-utilization 0.8 \
--max-model-len 65536 \
--tensor-parallel-size 4 \
--dtype bfloat16 \
--port 8790 \
--max-num-seqs 32 \
--enable-auto-tool-choice \
--trust-remote-code \
--enable-prefix-caching \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_xml &
二、问题现象
做并发测试的时候,任务进入队列速度太慢,有很大延迟。并且推理速度不稳定,首token延迟较大。
并发测试命令 - 1 (各个请求都是长输入):
vllm bench serve \
--backend openai-chat \
--base-url http://localhost:8790 \
--model /workspace/models/Qwen3.8-27b \
--tokenizer /workspace/models/Qwen3.8-27b \
--endpoint /v1/chat/completions \
--dataset-name random \
--random-input-len 32000 \
--random-output-len 4000 \
--num-prompts 100 \
--max-concurrency 20 \
--request-rate inf \
--save-result
并发测试结果 - 1:
============ Serving Benchmark Result ============
Successful requests: 100
Failed requests: 0
Maximum request concurrency: 20
Benchmark duration (s): 1844.47
Total input tokens: 3282038
Total generated tokens: 400000
Request throughput (req/s): 0.05
Output token throughput (tok/s): 216.86
Peak output token throughput (tok/s): 480.00
Peak concurrent requests: 21.00
Total token throughput (tok/s): 1996.26
---------------Time to First Token----------------
Mean TTFT (ms): 31773.82
Median TTFT (ms): 13634.68
P99 TTFT (ms): 185621.46
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 83.90
Median TPOT (ms): 88.14
P99 TPOT (ms): 90.32
---------------Inter-token Latency----------------
Mean ITL (ms): 87.61
Median ITL (ms): 44.66
P99 ITL (ms): 512.34
==================================================
并发测试命令 - 2:
vllm bench serve \
--backend openai-chat \
--base-url http://localhost:8790 \
--model /workspace/models/Qwen3.8-27b \
--tokenizer /workspace/models/Qwen3.8-27b \
--endpoint /v1/chat/completions \
--dataset-name random \
--random-input-len 30000 \
--random-output-len 3000 \
--random-range-ratio '{"input":0.8,"output":0.5}' \
--num-prompts 100 \
--max-concurrency 20 \
--request-rate inf \
--save-result
并发测试结果 - 2:
============ Serving Benchmark Result ============
Successful requests: 100
Failed requests: 0
Maximum request concurrency: 20
Benchmark duration (s): 1373.31
Total input tokens: 3049723
Total generated tokens: 316646
Request throughput (req/s): 0.07
Output token throughput (tok/s): 230.57
Peak output token throughput (tok/s): 451.00
Peak concurrent requests: 22.00
Total token throughput (tok/s): 2451.28
---------------Time to First Token----------------
Mean TTFT (ms): 7566.73
Median TTFT (ms): 7089.44
P99 TTFT (ms): 18061.99
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 80.17
Median TPOT (ms): 81.81
P99 TPOT (ms): 110.74
---------------Inter-token Latency----------------
Mean ITL (ms): 86.56
Median ITL (ms): 45.83
P99 ITL (ms): 479.14
==================================================
具体log如下:
(APIServer pid=13) INFO 11-11 19:20:52 [loggers.py:310] Engine 000: Avg prompt throughput: 7140.5 tokens/s, Avg generation throughput: 5.6 tokens/s, Running: 5 reqs, Waiting: 15 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:21:02 [loggers.py:310] Engine 000: Avg prompt throughput: 3346.5 tokens/s, Avg generation throughput: 12.9 tokens/s, Running: 7 reqs, Waiting: 13 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:21:12 [loggers.py:310] Engine 000: Avg prompt throughput: 5141.3 tokens/s, Avg generation throughput: 15.3 tokens/s, Running: 9 reqs, Waiting: 11 reqs, GPU KV cache usage: 9.2%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:21:22 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.4 tokens/s, Running: 9 reqs, Waiting: 11 reqs, GPU KV cache usage: 10.9%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:21:32 [loggers.py:310] Engine 000: Avg prompt throughput: 4508.7 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 10 reqs, Waiting: 10 reqs, GPU KV cache usage: 12.8%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:21:42 [loggers.py:310] Engine 000: Avg prompt throughput: 3662.4 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 11 reqs, Waiting: 9 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:21:52 [loggers.py:310] Engine 000: Avg prompt throughput: 6237.0 tokens/s, Avg generation throughput: 25.3 tokens/s, Running: 13 reqs, Waiting: 7 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO: 172.17.0.1:46532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=13) INFO 11-11 19:22:02 [loggers.py:310] Engine 000: Avg prompt throughput: 3722.3 tokens/s, Avg generation throughput: 27.7 tokens/s, Running: 14 reqs, Waiting: 7 reqs, GPU KV cache usage: 18.9%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:22:12 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 29.9 tokens/s, Running: 14 reqs, Waiting: 7 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:22:22 [loggers.py:310] Engine 000: Avg prompt throughput: 5264.4 tokens/s, Avg generation throughput: 29.8 tokens/s, Running: 15 reqs, Waiting: 6 reqs, GPU KV cache usage: 22.5%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:22:32 [loggers.py:310] Engine 000: Avg prompt throughput: 4986.2 tokens/s, Avg generation throughput: 32.6 tokens/s, Running: 16 reqs, Waiting: 5 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:22:42 [loggers.py:310] Engine 000: Avg prompt throughput: 3517.0 tokens/s, Avg generation throughput: 34.9 tokens/s, Running: 17 reqs, Waiting: 4 reqs, GPU KV cache usage: 26.3%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:22:52 [loggers.py:310] Engine 000: Avg prompt throughput: 3640.0 tokens/s, Avg generation throughput: 37.2 tokens/s, Running: 18 reqs, Waiting: 3 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:23:02 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 37.4 tokens/s, Running: 18 reqs, Waiting: 3 reqs, GPU KV cache usage: 30.0%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:23:12 [loggers.py:310] Engine 000: Avg prompt throughput: 4106.6 tokens/s, Avg generation throughput: 37.8 tokens/s, Running: 19 reqs, Waiting: 2 reqs, GPU KV cache usage: 31.8%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:23:22 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 19 reqs, Waiting: 2 reqs, GPU KV cache usage: 31.8%, Prefix cache hit rate: 0.0%
(Worker_TP0 pid=1165) WARNING 11-11 19:23:26 [jit_monitor.py:141] Triton kernel JIT compilation during inference: _topk_topp_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=13) INFO 11-11 19:23:32 [loggers.py:310] Engine 000: Avg prompt throughput: 3298.1 tokens/s, Avg generation throughput: 26.0 tokens/s, Running: 21 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.1%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:23:42 [loggers.py:310] Engine 000: Avg prompt throughput: 5093.4 tokens/s, Avg generation throughput: 81.3 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.4%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:23:52 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 439.9 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.7%, Prefix cache hit rate: 0.0%
(APIServer pid=13) INFO 11-11 19:24:02 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 437.9 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.9%, Prefix cache hit rate: 0.0%