一、软硬件信息
1.服务器厂家: 五舟
2.沐曦GPU型号：C500
3.操作系统内核版本：
Ubuntu 22.04， 5.15.0-119-generic
# uname -a
Linux MX-GPU-09 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
4.是否开启CPU虚拟化：
------------------------------宿主机上执行结果：-------------------------------------------------
mx-smi
mx-smi version: 2.3.1
=================== MetaX System Management Interface Log ===================
Timestamp : Fri Jun 26 11:23:36 2026
Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.3.1 Kernel Mode Driver Version: 3.8.1 |
| MACA Version: 3.7.1.5 BIOS Version: 1.22.3.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C500 | 0 N/A | 0000:17:00.0 | 0% Disabled |
| 39W / 350W | 43C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C500 | 1 N/A | 0000:18:00.0 | 0% Disabled |
| 44W / 350W | 50C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C500 | 2 N/A | 0000:19:00.0 | 0% Disabled |
| 43W / 350W | 53C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C500 | 3 N/A | 0000:1a:00.0 | 0% Disabled |
| 41W / 350W | 46C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C500 | 4 N/A | 0000:98:00.0 | 0% Disabled |
| 43W / 350W | 47C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C500 | 5 N/A | 0000:99:00.0 | 0% Disabled |
| 39W / 350W | 46C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C500 | 6 N/A | 0000:9a:00.0 | 0% Disabled |
| 39W / 350W | 44C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C500 | 7 N/A | 0000:9b:00.0 | 0% Disabled |
| 37W / 350W | 42C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| no process found |
+---------------------------------------------------------------------------------+
End of Log
------------------------------在docker镜像实例内：-------------------------------------------------
mx-smi
mx-smi version: 2.3.1
=================== MetaX System Management Interface Log ===================
Timestamp : Fri Jun 26 11:24:32 2026
Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.3.1 Kernel Mode Driver Version: 3.8.1 |
| MACA Version: 3.0.1028.20260227 BIOS Version: 1.22.3.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C500 | 0 N/A | 0000:17:00.0 | 0% Disabled |
| 39W / 350W | 44C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C500 | 1 N/A | 0000:18:00.0 | 0% Disabled |
| 45W / 350W | 51C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C500 | 2 N/A | 0000:19:00.0 | 0% Disabled |
| 44W / 350W | 53C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C500 | 3 N/A | 0000:1a:00.0 | 0% Disabled |
| 41W / 350W | 46C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C500 | 4 N/A | 0000:98:00.0 | 0% Disabled |
| 44W / 350W | 48C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C500 | 5 N/A | 0000:99:00.0 | 0% Disabled |
| 39W / 350W | 47C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C500 | 6 N/A | 0000:9a:00.0 | 0% Disabled |
| 39W / 350W | 45C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C500 | 7 N/A | 0000:9b:00.0 | 0% Disabled |
| 37W / 350W | 42C N/A | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| no process found |
+---------------------------------------------------------------------------------+
End of Log
6.docker info回显：
root@MX-GPU-09:/workspace# exit
exit
root@MX-GPU-09:~# docker info
Client: Docker Engine - Community
Version: 29.5.3
Context: default
Debug Mode: false
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 4
Server Version: 29.5.3
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 193637f7ee8ae5f5aa5248f49e7baa3e6164966e
runc version: v1.3.5-0-g488fc13e
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-119-generic
Operating System: Ubuntu 22.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 128
Total Memory: 1.968TiB
Name: MX-GPU-09
ID: 91391226-6df1-40c5-b876-9356b2510871
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Registry Mirrors:
docker.1ms.run/
Live Restore Enabled: false
Firewall Backend: iptables
EnableUserlandProxy: true
UserlandProxyPath: /usr/bin/docker-proxy
7.镜像版本：
pub-registry1.metax-tech.com/ai-opentest/master/maca/vllm-metax:0.15.0-maca.ai20260227-177-torch2.8-py312-ubuntu22.04-amd64
8.启动容器命令：
#docker run -itd \
-e RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1 \
-e RAY_EXPERIMENTAL_NOSET_MUSA_VISIBLE_DEVICES=1 \
-e RAY_EXPERIMENTAL_NOSET_MC_VISIBLE_DEVICES=1 \
-e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
-e VLLM_WORKER_MULTIPROC_METHOD=spawn \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
-v /data/models/Kimi-K2___6:/models/Kimi-K2.6 \
--name kimi-k26-metax \
--network=host \
--shm-size 100gb \
--ulimit memlock=-1 \
--device=/dev/mxcd \
--device=/dev/dri \
--group-add video \
--security-opt seccomp=unconfined \
--security-opt apparmor=unconfined \
pub-registry1.metax-tech.com/ai-opentest/master/maca/vllm-metax:0.15.0-maca.ai20260227-177-torch2.8-py312-ubuntu22.04-amd64
9.容器内执行命令：
Ray Head节点和worker节点
rm -rf ~/.cache/torch/kernels/
rm -rf ~/.cache/vllm/gpu_p2p_access_cache_*
Ray Head节点：
ray start --head --port=6379 --num-gpus=8
Ray worker节点：
ray start --address='10.105.113.24:6379' --num-gpus=8
Ray Head节点：
ray status
======== Autoscaler status: 2026-06-26 11:32:27.543635 ========
Node status
Active:
1 node_42a118a6527c171285c34cf6f687e264e9da981cd9f06c77d57cbdf4
1 node_2c05d0f8c3fd7fc5f49a4bea4581d0463622eb4397a67dddcfc10214
Pending:
(no pending nodes)
Recent failures:
(no failures)
Resources
Total Usage:
0.0/256.0 CPU
0.0/16.0 GPU
0B/3.75TiB memory
0B/190.00GiB object_store_memory
From request_resources:
(none)
Pending Demands:
(no resource demands)
#vllm serve /models/Kimi-K2.6 \
--tensor-parallel-size 8 \
--pipeline-parallel-size 2 \
--distributed-executor-backend ray \
--disable-custom-all-reduce \
--trust-remote-code \
--max-model-len 4096 \
--gpu-memory-utilization 0.80 \
--swap-space 16 \
--port 9000 \
--no-enable-prefix-caching \
--tool-call-parser kimi_k2 \
--reasoning-parser kimi_k2 \
--mm-processor-cache-gb 32
二、问题现象
两台物理机，OS为Ubuntu22.04，2TB内存一台，8张沐曦C500卡，物理机的版本信息如下：
OS：ubuntu 22.04
MX-SMI 2.3.1
Kernel Mode Driver Version: 3.8.1 |
MACA Version: 3.7.1.5
BIOS Version: 1.22.3.0
在docker里创建并运行vllm-metax:0.15.0-maca.ai20260227-177-torch2.8-py312-ubuntu22.04-amd64镜像的实例，镜像内版本信息如下：
MX-SMI 2.3.1
Kernel Mode Driver Version: 3.8.1
MACA Version: 3.0.1028.20260227
BIOS Version: 1.22.3.0
进入实例内启动kimi大模型失败，采用单卡启动就报OOM，选用单节点多卡、Ray集群两个节点多卡、MP方式两节点多卡启动大模型始终报CUDA driver error: invalid argument错误，请问各位专家，是否有办法解决？是否镜像内MACA版本跟宿主机不一致带来的问题？
具体日志见20260624-error-6.txt文件内容。
