Hello
vllm-metax:0.20.0-maca.ai3.7.0.101-torch2.8-py310-ubuntu22.04-amd64
用这个镜像跑MiMo-V2-Flash-W8A8出现对话乱码。看看是需要怎么调整，感谢。
按照modelscope.cn/models/metax-tech/MiMo-V2-Flash-W8A8/feedback的内容修改了flash_attn.py和flash_attn_diffkv.py
root@test:~# mx-smi
mx-smi version: 2.3.1
=================== MetaX System Management Interface Log ===================
Timestamp : Wed May 27 01:00:52 2026
Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.3.1 Kernel Mode Driver Version: 3.8.1 |
| MACA Version: 3.7.1.5 BIOS Version: 1.33.4.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C500 | 0 Off | 0000:4b:00.0 | 0% Disabled |
| 63W / 350W | 56C P0 | 864/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C500 | 1 Off | 0000:4c:00.0 | 0% Disabled |
| 61W / 350W | 58C P0 | 864/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C500 | 2 Off | 0000:50:00.0 | 0% Disabled |
| 60W / 350W | 53C P0 | 864/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C500 | 3 Off | 0000:51:00.0 | 0% Disabled |
| 61W / 350W | 56C P0 | 864/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C500 | 4 Off | 0000:cb:00.0 | 0% Disabled |
| 62W / 350W | 57C P0 | 864/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C500 | 5 Off | 0000:cc:00.0 | 0% Disabled |
| 61W / 350W | 58C P0 | 864/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C500 | 6 Off | 0000:cd:00.0 | 0% Disabled |
| 62W / 350W | 54C P0 | 864/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C500 | 7 Off | 0000:ce:00.0 | 0% Disabled |
| 60W / 350W | 56C P0 | 864/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| no process found |
+---------------------------------------------------------------------------------+
End of Log
root@test:~#
docker run -itd \
--device=/dev/dri \
--device=/dev/mxcd \
--group-add video \
--name vllm-test \
--device=/dev/mem \
--network=host \
--security-opt seccomp=unconfined \
--security-opt apparmor=unconfined \
--shm-size 100gb \
--ulimit memlock=-1 \
-v /home/test/models/MiMo-V2-Flash-W8A8:/model \
cr.metax-tech.com/public-init/maca/vllm-metax:0.20.0-maca.ai3.7.0.101-torch2.8-py310-ubuntu22.04-amd64
vllm serve /model -tp 8 \
--trust-remote-code \
--dtype auto \
--max-num-seqs 128 \
--max-model-len 8192 \
--gpu-memory-utilization 0.9 \
--distributed_executor_backend mp
测试
root@test:/home/test/models# curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "/model",
"messages": [
{"role": "user", "content": "你好，请简单介绍一下你自己。"}
],
"temperature": 0.1,
"max_tokens": 100
}'
{"id":"chatcmpl-8cbab88c9405cdb9","object":"chat.completion","created":1779844118,"model":"/model","choices":[{"index":0,"message":{"role":"assistant","content":" rob自由ydk이는_7 ! everywhere \n\nigh精湛�مصطفمصطف Freconiمصطفߡire ge Vمصطفمصطفighمصطفمصطفigh!自由Rocket [ankigh Hav.とりあ揉market Havスポー�hift Batter Hav Hav自由0مصطف 鋆自由清扫ao / Bradfordすべかもし (טל康康isk вечすべ Mockitoisk **vcمصطفمصطفREchingREЛениかもし دمشمصطفiskすべﲻとりあ/!�熙�nicoseuropäische handheld! использова0RFcompatible\Bridgeねぇاعتم叠","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null},"logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":32,"total_tokens":132,"completion_tokens":100,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}root@test:/home/test/models#
