MetaX-Tech Developer Forum 论坛首页
  • 沐曦开发者
search
Sign in

Alviz

  • Members
  • Joined 2026年11月11日
  • message 帖子
  • forum 主题
  • favorite 关注者
  • favorite_border Follows
  • person_outline 详细信息

Alviz has posted 3 messages.

  • See post chevron_right
    Alviz
    Members
    沐曦C500 4卡部署Qwen3.8-27b,并发测试首token延迟较大(特别是长序列输入更加明显),且初始任务进入队列速度极慢,并未达到队列上限,就出现排队。 已解决 2026年11月20日 11:33

    请问沐曦vllm与SGLang的镜像支持想Nvidia卡支持的PD分离(Prefill-Decode Disaggregation)设置吗? PD分离是能解决这个问题的

  • See post chevron_right
    Alviz
    Members
    沐曦C500 4卡部署Qwen3.8-27b,并发测试首token延迟较大(特别是长序列输入更加明显),且初始任务进入队列速度极慢,并未达到队列上限,就出现排队。 已解决 2026年11月13日 11:41

    结果有一些缓解,但是延迟依然很大,主要是长序列输入,主要是还是初始任务进入队列慢:
    ============ Serving Benchmark Result ============
    Successful requests: 100
    Failed requests: 0
    Maximum request concurrency: 20
    Benchmark duration (s): 1644.61
    Total input tokens: 3205238
    Total generated tokens: 400000
    Request throughput (req/s): 0.06
    Output token throughput (tok/s): 243.22
    Peak output token throughput (tok/s): 399.00
    Peak concurrent requests: 22.00
    Total token throughput (tok/s): 2192.15
    ---------------Time to First Token----------------
    Mean TTFT (ms): 26064.01
    Median TTFT (ms): 14545.58
    P99 TTFT (ms): 130038.23
    -----Time per Output Token (excl. 1st token)------
    Mean TPOT (ms): 75.14
    Median TPOT (ms): 76.72
    P99 TPOT (ms): 81.82
    ---------------Inter-token Latency----------------
    Mean ITL (ms): 83.09
    Median ITL (ms): 52.16
    P99 ITL (ms): 345.90
    ==================================================

  • See post chevron_right
    Alviz
    Members
    沐曦C500 4卡部署Qwen3.8-27b,并发测试首token延迟较大(特别是长序列输入更加明显),且初始任务进入队列速度极慢,并未达到队列上限,就出现排队。 已解决 2026年11月11日 19:56

    一、软硬件信息
    1. 服务器厂家:浪潮
    2. 沐曦GPU型号:C500
    3. 操作系统内核版本:Linux 10-7-182-232 6.14.0-27-generic #27~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 17:38:49 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
    4. 是否开启CPU虚拟化:否
    5. mx-smi回显:
    Attached GPUs : 8
    +---------------------------------------------------------------------------------+
    | MX-SMI 2.2.9 Kernel Mode Driver Version: 3.4.4 |
    | MACA Version: 3.3.0.15 BIOS Version: 1.20.3.0 |
    |------------------+-----------------+---------------------+----------------------|
    | Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
    | Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
    |==================+=================+=====================+======================|
    | 0 MetaX C500 | 0 Off | 0000:08:00.0 | 60% Disabled |
    | 159W / 350W | 47C P9 | 54052/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 1 MetaX C500 | 1 Off | 0000:09:00.0 | 59% Disabled |
    | 151W / 350W | 45C P9 | 54052/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 2 MetaX C500 | 2 Off | 0000:0e:00.0 | 60% Disabled |
    | 147W / 350W | 46C P9 | 54052/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 3 MetaX C500 | 3 Off | 0000:11:00.0 | 60% Disabled |
    | 149W / 350W | 47C P9 | 54052/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 4 MetaX C500 | 4 Off | 0000:32:00.0 | 0% Enabled |
    | 70W / 350W | 38C P9 | 5147/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 5 MetaX C500 | 5 Off | 0000:38:00.0 | 0% Disabled |
    | 75W / 350W | 40C P9 | 56795/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 6 MetaX C500 | 6 Off | 0000:3b:00.0 | 0% Disabled |
    | 54W / 350W | 40C P0 | 858/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 7 MetaX C500 | 7 Off | 0000:3c:00.0 | 0% Disabled |
    | 71W / 350W | 41C P9 | 51355/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    6. docker info回显:
    root@10-7-182-232:/home/srcb# docker version
    Client: Docker Engine - Community
    Version: 29.9.1
    API version: 1.56
    Go version: go1.26.9
    Git commit: dfc520d
    Built: Wed Oct 14 18:32:55 2026
    OS/Arch: linux/amd64
    Context: default

    Server: Docker Engine - Community
    Engine:
    Version: 29.9.1
    API version: 1.56 (minimum version 1.40)
    Go version: go1.26.9
    Git commit: b26fb77
    Built: Wed Oct 14 18:32:55 2026
    OS/Arch: linux/amd64
    Experimental: false
    containerd:
    Version: v2.4.2
    GitCommit: b7a25261e029d68978bbf290df0a62fc819b3c51
    runc:
    Version: 1.5.2
    GitCommit: v1.5.2-0-g29dd3dc2
    docker-init:
    Version: 0.19.0
    GitCommit: de40ad0
    7. 镜像版本:
    cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.28.0-maca.ai3.8.2.401-torch2.10-py310-ubuntu22.04-amd64
    8. 启动容器命令:
    docker run -itd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 --device=/dev/dri/renderD130 --device=/dev/dri/renderD131 --device=/dev/mxcd --group-add video -p 8790:8790 -p 8791:8791 -v /data/LLM:/workspace --shm-size=256g --name vllm-server --restart unless-stopped cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.28.0-maca.ai3.8.2.401-torch2.10-py310-ubuntu22.04-amd64 /bin/bash
    9. 容器内执行命令:
    vllm serve --model /workspace/models/Qwen3.8-27b \
    --gpu-memory-utilization 0.8 \
    --max-model-len 65536 \
    --tensor-parallel-size 4 \
    --dtype bfloat16 \
    --port 8790 \
    --max-num-seqs 32 \
    --enable-auto-tool-choice \
    --trust-remote-code \
    --enable-prefix-caching \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_xml &

    二、问题现象
    做并发测试的时候,任务进入队列速度太慢,有很大延迟。并且推理速度不稳定,首token延迟较大。
    并发测试命令 - 1 (各个请求都是长输入):
    vllm bench serve \
    --backend openai-chat \
    --base-url http://localhost:8790 \
    --model /workspace/models/Qwen3.8-27b \
    --tokenizer /workspace/models/Qwen3.8-27b \
    --endpoint /v1/chat/completions \
    --dataset-name random \
    --random-input-len 32000 \
    --random-output-len 4000 \
    --num-prompts 100 \
    --max-concurrency 20 \
    --request-rate inf \
    --save-result

    并发测试结果 - 1:
    ============ Serving Benchmark Result ============
    Successful requests: 100
    Failed requests: 0
    Maximum request concurrency: 20
    Benchmark duration (s): 1844.47
    Total input tokens: 3282038
    Total generated tokens: 400000
    Request throughput (req/s): 0.05
    Output token throughput (tok/s): 216.86
    Peak output token throughput (tok/s): 480.00
    Peak concurrent requests: 21.00
    Total token throughput (tok/s): 1996.26
    ---------------Time to First Token----------------
    Mean TTFT (ms): 31773.82
    Median TTFT (ms): 13634.68
    P99 TTFT (ms): 185621.46
    -----Time per Output Token (excl. 1st token)------
    Mean TPOT (ms): 83.90
    Median TPOT (ms): 88.14
    P99 TPOT (ms): 90.32
    ---------------Inter-token Latency----------------
    Mean ITL (ms): 87.61
    Median ITL (ms): 44.66
    P99 ITL (ms): 512.34
    ==================================================

    并发测试命令 - 2:
    vllm bench serve \
    --backend openai-chat \
    --base-url http://localhost:8790 \
    --model /workspace/models/Qwen3.8-27b \
    --tokenizer /workspace/models/Qwen3.8-27b \
    --endpoint /v1/chat/completions \
    --dataset-name random \
    --random-input-len 30000 \
    --random-output-len 3000 \
    --random-range-ratio '{"input":0.8,"output":0.5}' \
    --num-prompts 100 \
    --max-concurrency 20 \
    --request-rate inf \
    --save-result

    并发测试结果 - 2:
    ============ Serving Benchmark Result ============
    Successful requests: 100
    Failed requests: 0
    Maximum request concurrency: 20
    Benchmark duration (s): 1373.31
    Total input tokens: 3049723
    Total generated tokens: 316646
    Request throughput (req/s): 0.07
    Output token throughput (tok/s): 230.57
    Peak output token throughput (tok/s): 451.00
    Peak concurrent requests: 22.00
    Total token throughput (tok/s): 2451.28
    ---------------Time to First Token----------------
    Mean TTFT (ms): 7566.73
    Median TTFT (ms): 7089.44
    P99 TTFT (ms): 18061.99
    -----Time per Output Token (excl. 1st token)------
    Mean TPOT (ms): 80.17
    Median TPOT (ms): 81.81
    P99 TPOT (ms): 110.74
    ---------------Inter-token Latency----------------
    Mean ITL (ms): 86.56
    Median ITL (ms): 45.83
    P99 ITL (ms): 479.14
    ==================================================

    具体log如下:
    (APIServer pid=13) INFO 11-11 19:20:52 [loggers.py:310] Engine 000: Avg prompt throughput: 7140.5 tokens/s, Avg generation throughput: 5.6 tokens/s, Running: 5 reqs, Waiting: 15 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:21:02 [loggers.py:310] Engine 000: Avg prompt throughput: 3346.5 tokens/s, Avg generation throughput: 12.9 tokens/s, Running: 7 reqs, Waiting: 13 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:21:12 [loggers.py:310] Engine 000: Avg prompt throughput: 5141.3 tokens/s, Avg generation throughput: 15.3 tokens/s, Running: 9 reqs, Waiting: 11 reqs, GPU KV cache usage: 9.2%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:21:22 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.4 tokens/s, Running: 9 reqs, Waiting: 11 reqs, GPU KV cache usage: 10.9%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:21:32 [loggers.py:310] Engine 000: Avg prompt throughput: 4508.7 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 10 reqs, Waiting: 10 reqs, GPU KV cache usage: 12.8%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:21:42 [loggers.py:310] Engine 000: Avg prompt throughput: 3662.4 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 11 reqs, Waiting: 9 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:21:52 [loggers.py:310] Engine 000: Avg prompt throughput: 6237.0 tokens/s, Avg generation throughput: 25.3 tokens/s, Running: 13 reqs, Waiting: 7 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO: 172.17.0.1:46532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
    (APIServer pid=13) INFO 11-11 19:22:02 [loggers.py:310] Engine 000: Avg prompt throughput: 3722.3 tokens/s, Avg generation throughput: 27.7 tokens/s, Running: 14 reqs, Waiting: 7 reqs, GPU KV cache usage: 18.9%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:22:12 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 29.9 tokens/s, Running: 14 reqs, Waiting: 7 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:22:22 [loggers.py:310] Engine 000: Avg prompt throughput: 5264.4 tokens/s, Avg generation throughput: 29.8 tokens/s, Running: 15 reqs, Waiting: 6 reqs, GPU KV cache usage: 22.5%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:22:32 [loggers.py:310] Engine 000: Avg prompt throughput: 4986.2 tokens/s, Avg generation throughput: 32.6 tokens/s, Running: 16 reqs, Waiting: 5 reqs, GPU KV cache usage: 24.4%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:22:42 [loggers.py:310] Engine 000: Avg prompt throughput: 3517.0 tokens/s, Avg generation throughput: 34.9 tokens/s, Running: 17 reqs, Waiting: 4 reqs, GPU KV cache usage: 26.3%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:22:52 [loggers.py:310] Engine 000: Avg prompt throughput: 3640.0 tokens/s, Avg generation throughput: 37.2 tokens/s, Running: 18 reqs, Waiting: 3 reqs, GPU KV cache usage: 28.3%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:23:02 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 37.4 tokens/s, Running: 18 reqs, Waiting: 3 reqs, GPU KV cache usage: 30.0%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:23:12 [loggers.py:310] Engine 000: Avg prompt throughput: 4106.6 tokens/s, Avg generation throughput: 37.8 tokens/s, Running: 19 reqs, Waiting: 2 reqs, GPU KV cache usage: 31.8%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:23:22 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 19 reqs, Waiting: 2 reqs, GPU KV cache usage: 31.8%, Prefix cache hit rate: 0.0%
    (Worker_TP0 pid=1165) WARNING 11-11 19:23:26 [jit_monitor.py:141] Triton kernel JIT compilation during inference: _topk_topp_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
    (APIServer pid=13) INFO 11-11 19:23:32 [loggers.py:310] Engine 000: Avg prompt throughput: 3298.1 tokens/s, Avg generation throughput: 26.0 tokens/s, Running: 21 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.1%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:23:42 [loggers.py:310] Engine 000: Avg prompt throughput: 5093.4 tokens/s, Avg generation throughput: 81.3 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.4%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:23:52 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 439.9 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.7%, Prefix cache hit rate: 0.0%
    (APIServer pid=13) INFO 11-11 19:24:02 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 437.9 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 34.9%, Prefix cache hit rate: 0.0%

  • 沐曦开发者论坛
powered by misago