jdeies

我看沐曦魔塔发布的模型大小254.74GB，也不可以？
单机8卡c500最大支持多大的模型？

是的，单机8卡

一、软硬件信息
1.服务器厂家:
H3C R5300 G6

2.沐曦GPU型号：
8卡C500

3.操作系统内核版本：
5.15.0-119-generic

4.是否开启CPU虚拟化：
开了

5.mx-smi回显：
mx-smi version: 2.2.6

=================== MetaX System Management Interface Log ===================
Timestamp : Fri May 29 08:56:21 2026

Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.2.6 Kernel Mode Driver Version: 2.16.0 |
| MACA Version: 3.0.0.8 BIOS Version: 1.26.1.0 |
|------------------------------------+---------------------+----------------------+
| GPU NAME Persistence-M | Bus-id | GPU-Util sGPU-M |
| Temp Pwr:Usage/Cap Perf | Memory-Usage | GPU-State |
|====================================+=====================+======================|
| 0 MetaX C500 Off | 0000:08:00.0 | 0% Native |
| 36C 78W / 350W P9 | 64230/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 1 MetaX C500 Off | 0000:09:00.0 | 0% Native |
| 36C 74W / 350W P9 | 64550/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 2 MetaX C500 Off | 0000:0e:00.0 | 0% Native |
| 36C 80W / 350W P9 | 64550/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 3 MetaX C500 Off | 0000:11:00.0 | 0% Native |
| 35C 75W / 350W P9 | 64230/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 4 MetaX C500 Off | 0000:32:00.0 | 0% Native |
| 36C 79W / 350W P9 | 64230/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 5 MetaX C500 Off | 0000:38:00.0 | 0% Native |
| 37C 75W / 350W P9 | 64550/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 6 MetaX C500 Off | 0000:3b:00.0 | 0% Native |
| 37C 80W / 350W P9 | 64550/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+
| 7 MetaX C500 Off | 0000:3c:00.0 | 0% Native |
| 38C 80W / 350W P9 | 64230/65536 MiB | Available |
+------------------------------------+---------------------+----------------------+

+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| 0 160829 python3.10 63360 |
| 1 160830 python3.10 63680 |
| 2 160831 python3.10 63680 |
| 3 160832 python3.10 63360 |
| 4 160833 python3.10 63360 |
| 5 160834 python3.10 63680 |
| 6 160835 python3.10 63680 |
| 7 160836 python3.10 63360 |
+---------------------------------------------------------------------------------+

6.docker info回显：
Client: Docker Engine - Community
Version: 28.5.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.29.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.40.2
Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 4
Running: 3
Paused: 0
Stopped: 1
Images: 12
Server Version: 28.5.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b98a3aace656320842a23f4a392a33f46af97866
runc version: v1.3.0-0-g4ca628d1
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-119-generic
Operating System: Ubuntu 22.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 160
Total Memory: 1.968TiB
Name: 10-66-1-220
ID: 34b4a18a-92e7-45f0-b1fc-0eddb89a247e
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false

7.镜像版本：
vllm-metax:0.19.0-maca.ai3.5.3.502-torch2.8-py312-ubuntu22.04-amd64

8.启动容器命令：
docker run -itd --device=/dev/dri --device=/dev/mxcd --group-add video --network=host --name deepseek-v4 --security-opt seccomp=unconfined --security-opt apparmor=unconfined --shm-size 100gb --ulimit memlock=-1 -v /share/DeepSeek-V4-Flash:/workspace/share cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.19.0-maca.ai3.5.3.502-torch2.8-py312-ubuntu22.04-amd64

9.容器内执行命令：
vllm serve /workspace/share/GLM-4.7 \
--tensor-parallel-size 8 \
--gpu-memory-utilization 0.8 \
--tool-call-parser glm47 \
--enable-auto-tool-choice --reasoning-parser glm45 \
--served-model-name GLM-4.7\
--trust-remote-code

二、问题现象
请描述详细的问题现象日志。若日志过长，请上传附件(txt格式）。
运行metax-tech/GLM-4.7-W8A8模型报下面错误

/opt/conda/lib/python3.12/site-packages/tvm_ffi/_optional_torch_c_dlpack.py:181: UserWarning: Failed to JIT torch c dlpack extension, EnvTensorAllocator will not be enabled.
We recommend installing via pip install torch-c-dlpack-ext
(APIServer pid=6934) /opt/conda/lib/python3.12/site-packages/compressed_tensors/quantization/quant_args.py:359: UserWarning: No observer is used for dynamic quant., setting to None
(APIServer pid=6934) warnings.warn(
(APIServer pid=6934) INFO 05-29 11:18:03 [vllm.py:790] Asynchronous scheduling is disabled.
(Worker pid=5460) INFO 05-29 11:10:37 [mccl.py:27] Found mccl from library libmccl.so
(Worker pid=5460) INFO 05-29 11:10:37 [pynccl.py:111] vLLM is using nccl==2.16.5
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] WorkerProc failed to start.
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] Traceback (most recent call last):
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 826, in worker_main
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] worker = WorkerProc(args, kwargs)
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] return func(args, kwargs)
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 605, in init
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.worker.init_device()
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 312, in init_device
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.worker.init_device() # type: ignore
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] return func(*args, kwargs)
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 263, in init_device
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] init_worker_distributed_environment(
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1043, in init_worker_distributed_environment
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ensure_model_parallel_initialized(
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1748, in ensure_model_parallel_initialized
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] initialize_model_parallel(
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1575, in initialize_model_parallel
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] _TP = init_model_parallel_group(
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1157, in init_model_parallel_group
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] return GroupCoordinator(
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 376, in init
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.device_communicator = device_comm_cls(
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm_metax/distributed/device_communicators/cuda_communicator.py", line 30, in init
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] super().init(
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 75, in init
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.pynccl_comm = PyNcclCommunicator(
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl.py", line 144, in init
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.all_reduce(data)
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl.py", line 177, in all_reduce
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.nccl.ncclAllReduce(
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm_metax/patch/distributed/pynccl_wrapper.py", line 348, in ncclAllReduce
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.NCCL_CHECK(
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm_metax/patch/distributed/pynccl_wrapper.py", line 292, in NCCL_CHECK
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] raise RuntimeError(f"NCCL error: {error_str}")
(Worker pid=5461) ERROR 05-29 11:10:39 [multiproc_executor.py:857] RuntimeError: NCCL error: internal error
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] WorkerProc failed to start.
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] Traceback (most recent call last):
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 826, in worker_main
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] worker = WorkerProc(args, kwargs)
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] return func(args, kwargs)
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 605, in init
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.worker.init_device()
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 312, in init_device
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.worker.init_device() # type: ignore
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] return func(*args, kwargs)
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 263, in init_device
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] init_worker_distributed_environment(
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1043, in init_worker_distributed_environment
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ensure_model_parallel_initialized(
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1748, in ensure_model_parallel_initialized
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] initialize_model_parallel(
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1575, in initialize_model_parallel
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] _TP = init_model_parallel_group(
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1157, in init_model_parallel_group
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] return GroupCoordinator(
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 376, in init
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.device_communicator = device_comm_cls(
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm_metax/distributed/device_communicators/cuda_communicator.py", line 30, in init
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] super().init(
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 75, in init
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.pynccl_comm = PyNcclCommunicator(
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl.py", line 144, in init
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.all_reduce(data)
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl.py", line 177, in all_reduce
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.nccl.ncclAllReduce(
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm_metax/patch/distributed/pynccl_wrapper.py", line 348, in ncclAllReduce
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.NCCL_CHECK(
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm_metax/patch/distributed/pynccl_wrapper.py", line 292, in NCCL_CHECK
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] raise RuntimeError(f"NCCL error: {error_str}")
(Worker pid=5465) ERROR 05-29 11:10:39 [multiproc_executor.py:857] RuntimeError: NCCL error: internal error
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] WorkerProc failed to start.
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] Traceback (most recent call last):
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 826, in worker_main
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] worker = WorkerProc(args, kwargs)
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] return func(args, kwargs)
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 605, in init
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.worker.init_device()
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 312, in init_device
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.worker.init_device() # type: ignore
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] return func(*args, kwargs)
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 263, in init_device
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] init_worker_distributed_environment(
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1043, in init_worker_distributed_environment
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ensure_model_parallel_initialized(
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1748, in ensure_model_parallel_initialized
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] initialize_model_parallel(
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1575, in initialize_model_parallel
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] _TP = init_model_parallel_group(
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1157, in init_model_parallel_group
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] return GroupCoordinator(
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 376, in init
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.device_communicator = device_comm_cls(
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm_metax/distributed/device_communicators/cuda_communicator.py", line 30, in init
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] super().init(
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 75, in init
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.pynccl_comm = PyNcclCommunicator(
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl.py", line 144, in init
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.all_reduce(data)
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl.py", line 177, in all_reduce
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.nccl.ncclAllReduce(
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm_metax/patch/distributed/pynccl_wrapper.py", line 348, in ncclAllReduce
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.NCCL_CHECK(
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm_metax/patch/distributed/pynccl_wrapper.py", line 292, in NCCL_CHECK
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] raise RuntimeError(f"NCCL error: {error_str}")
(Worker pid=5466) ERROR 05-29 11:10:39 [multiproc_executor.py:857] RuntimeError: NCCL error: internal error
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] WorkerProc failed to start.
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] Traceback (most recent call last):
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 826, in worker_main
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] worker = WorkerProc(args, kwargs)
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] return func(args, kwargs)
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 605, in init
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.worker.init_device()
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 312, in init_device
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.worker.init_device() # type: ignore
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] return func(*args, kwargs)
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 263, in init_device
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] init_worker_distributed_environment(
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1043, in init_worker_distributed_environment
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ensure_model_parallel_initialized(
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1748, in ensure_model_parallel_initialized
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] initialize_model_parallel(
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1575, in initialize_model_parallel
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] _TP = init_model_parallel_group(
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 1157, in init_model_parallel_group
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] return GroupCoordinator(
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/parallel_state.py", line 376, in init
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.device_communicator = device_comm_cls(
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm_metax/distributed/device_communicators/cuda_communicator.py", line 30, in init
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] super().init(
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 75, in init
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.pynccl_comm = PyNcclCommunicator(
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] ^^^^^^^^^^^^^^^^^^^
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl.py", line 144, in init
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.all_reduce(data)
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl.py", line 177, in all_reduce
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.nccl.ncclAllReduce(
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm_metax/patch/distributed/pynccl_wrapper.py", line 348, in ncclAllReduce
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] self.NCCL_CHECK(
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] File "/opt/conda/lib/python3.12/site-packages/vllm_metax/patch/distributed/pynccl_wrapper.py", line 292, in NCCL_CHECK
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] raise RuntimeError(f"NCCL error: {error_str}")
(Worker pid=5462) ERROR 05-29 11:10:39 [multiproc_executor.py:857] RuntimeError: NCCL error: internal error
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] EngineCore failed to start.
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] Traceback (most recent call last):
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1082, in run_engine_core
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] engine_core = EngineCoreProc(args, engine_index=dp_rank, kwargs)
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] return func(args, kwargs)
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 848, in init
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] super().init(
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 114, in init
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] self.model_executor = executor_class(vllm_config)
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 101, in init
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] super().init(vllm_config)
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] return func(*args, kwargs)
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 103, in init
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] self._init_executor()
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 190, in _init_executor
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 736, in wait_for_ready
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] raise e from None
(EngineCore pid=5111) ERROR 05-29 11:10:47 [core.py:1108] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.