我在六机上进行glm5.1 PD分离操作，启动decode节点报错，报错内容如文件
lishuai
- Members
- Joined 2025年12月19日
-
-
-
metax@metax-host-104:/opt/maca/samples/mccl_tests/perf$ bash mccl.sh 8
The test is all_reduce_perf, the maca version is /opt/maca-3.7.1
main_process = 7324
metax-host-104: Test CUDA failure common.cu:1349 'initialization error'
.. metax-host-104 pid 7325: Test failure common.cu:1271
metax-host-104: Test CUDA failure common.cu:1349 'initialization error'
.. metax-host-104 pid 7326: Test failure common.cu:1271
metax-host-104: Test CUDA failure common.cu:1349 'initialization error'
.. metax-host-104 pid 7327: Test failure common.cu:1271
metax-host-104: Test CUDA failure common.cu:1349 'initialization error'
.. metax-host-104 pid 7328: Test failure common.cu:1271
metax-host-104: Test CUDA failure common.cu:1349 'initialization error'
.. metax-host-104 pid 7329: Test failure common.cu:1271
metax-host-104: Test CUDA failure common.cu:1349 'initialization error'
.. metax-host-104 pid 7330: Test failure common.cu:1271
metax-host-104: Test CUDA failure common.cu:1349 'initialization error'
.. metax-host-104 pid 7331: Test failure common.cu:1271
===============================nThread 1 nGpus 1 minBytes 1024 maxBytes 1073741824 step: 2(factor) warmup iters: 5 iters: 10 agg iters: 1 validation: 1 graph: 0
Using devices
metax-host-104: Test CUDA failure common.cu:1349 'initialization error'
.. metax-host-104 pid 7324: Test failure common.cu:1271
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:Process name: [[25858,1],4]
Exit code: 2
metax@metax-host-104:/opt/maca/samples/mccl_tests/perf$ bash mccl.sh 2
The test is all_reduce_perf, the maca version is /opt/maca-3.7.1
main_process = 7378
metax-host-104: Test CUDA failure common.cu:1349 'initialization error'
.. metax-host-104 pid 7379: Test failure common.cu:1271
===============================nThread 1 nGpus 1 minBytes 1024 maxBytes 1073741824 step: 2(factor) warmup iters: 5 iters: 10 agg iters: 1 validation: 1 graph: 0
Using devices
metax-host-104: Test CUDA failure common.cu:1349 'initialization error'
.. metax-host-104 pid 7378: Test failure common.cu:1271
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:Process name: [[25940,1],0]
Exit code: 2
metax@metax-host-104:/opt/maca/samples/mccl_tests/perf$
单机mccl测试失败是怎么回事
mx-smi回显信息正常如下
metax@metax-host-104:/opt/maca/samples/mccl_tests/perf$ mx-smi
mx-smi version: 2.3.1=================== MetaX System Management Interface Log ===================
Timestamp : Wed Jun 17 10:13:47 2026Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.3.1 Kernel Mode Driver Version: 3.8.1 |
| MACA Version: 3.7.1.5 BIOS Version: 1.29.1.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C550 | 0 Off | 0000:2b:00.0 | 0% Disabled |
| 54W / 450W | 32C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C550 | 1 Off | 0000:3a:00.0 | 0% Disabled |
| 56W / 450W | 33C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C550 | 2 Off | 0000:4d:00.0 | 0% Disabled |
| 52W / 450W | 33C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C550 | 3 Off | 0000:5c:00.0 | 0% Disabled |
| 56W / 450W | 33C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C550 | 4 Off | 0000:aa:00.0 | 0% Disabled |
| 53W / 450W | 32C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C550 | 5 Off | 0000:ba:00.0 | 0% Disabled |
| 52W / 450W | 33C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C550 | 6 Off | 0000:ca:00.0 | 0% Disabled |
| 54W / 450W | 34C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C550 | 7 Off | 0000:da:00.0 | 0% Disabled |
| 53W / 450W | 33C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------++---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| no process found |
+---------------------------------------------------------------------------------+End of Log
metax@metax-host-104:/opt/maca/samples/mccl_tests/perf$ -
-
-
如果要在裸金属宿主机环境下进行模型部署，安装maca-vllm-metax-0.19.0-py310-3.5.3.502-linux-x86_64.tar.xz过后，官方的vllm还需要安装吗？
-
缺陷报告：MCCL 跨机 RoCE 内存注册失败 (ibv_reg_mr Invalid argument)
【环境信息】硬件： 双节点共 16 张沐曦 MetaX C500 (8卡/节点)，机头是两台H3c uniserver R5330 G7(C500*16)
网络： 40Gbps RoCE 网卡 (设备名 rocep6s0, rocep95s0)---这是ibstat显示的信息，网卡用的是400G的迈洛斯 cx7
软件： MACA 3.5.3 / MCCL 2.16.5【问题现象】
在使用 mpirun 执行跨机 all_reduce_perf 测试时，若开启 IB/RoCE 硬件加速，MCCL 在初始化阶段崩溃，报错：
MCCL WARN Call to ibv_reg_mr failed with error Invalid argument
【已完成的排查与隔离】
网络层正常： RoCE 高速网段（192.168.100.x）跨机 ping 测试延迟 < 0.1ms，双端 firewalld 与 SELinux 已关闭。
系统限制正常： 双端通过 MPI 验证 ulimit -l 均为 unlimited，排除 memlock 限制导致的问题。
iova2 降级测试： 增加 -x MCCL_IB_PCI_RELAXED_ORDERING=0 后，报错从 ibv_reg_mr_iova2 failed 退化为 ibv_reg_mr failed，依然返回 Invalid argument (retcode 2)。
TCP 降级对照组（核心证据）： 增加 -x MCCL_IB_DISABLE=1 强制走普通 TCP/Socket 通信后，测试完美通过（#wrong 0）。
mccl测试脚本内容：
[root@localhost /opt/maca/samples/mccl_tests/perf]# cat cluster.sh!/bin/bash
MACA_PATH="${MACA_PATH:-/opt/maca}"
HOST_IP=192.168.1.204:8,192.168.1.205:8
GPU_NUM=16TEST_DIR=$MACA_PATH/samples/mccl_tests/perf/mccl_perf
BENCH_NAMES="all_reduce_perf all_gather_perf reduce_scatter_perf sendrecv_perf alltoall_perf"
BENCH_NAMES="all_reduce_perf"
if [[ -z "$1" || -z "$2" || -z "$3" ]]; then
echo "Use the default ip addr. Run with parameters for custom ip addr, for example: bash cluster.sh ip_1:proc_count,ip_2:proc_count gpu_num test_name"
else
HOST_IP=$1
GPU_NUM=$2if [ "$3" = "all" ]; then
BENCH_NAMES="all_reduce_perf all_gather_perf reduce_scatter_perf sendrecv_perf alltoall_perf"
else
if [ -e "$TEST_DIR/$3" ]; then
BENCH_NAMES=$3
else
echo "$TEST_DIR/$3 dose not exist!"
exit 1
fi
fi
fiIP_MASK="$(echo "$HOST_IP" | cut -d. -f1-3).0/24"
IP_MASK="192.168.100.0/24"
IB_PORT=rocep6s0,rocep95s0PERF_ENV="-x FORCE_ACTIVE_WAIT=2"
LIB_PATH_ENV="-x MACA_PATH=${MACA_PATH} -x LD_LIBRARY_PATH=${MACA_PATH}/lib:/${MACA_PATH}/ompi/lib:/${MACA_PATH}/ucx/lib"ENV_VAR="-x MCCL_IB_HCA=${IB_PORT} -x MCCL_CROSS_NIC=1 ${PERF_ENV} ${LIB_PATH_ENV}"
ENV_VAR="-x MCCL_IB_HCA=rocep6s0,rocep95s0 -x MCCL_SOCKET_IFNAME=p50p1,p51p1 -x MCCL_CROSS_NIC=1 ${PERF_ENV} ${LIB_PATH_ENV} -x MCCL_IB_DISABLE=0"
MPI_PROCESS_NUM=${GPU_NUM}
MPI_RUN_OPT="--allow-run-as-root -mca btl_tcp_if_include ${IP_MASK} -mca oob_tcp_if_include ${IP_MASK} -mca pml ^ucx -mca osc ^ucx -mca btl ^openib"for BENCH in ${BENCH_NAMES}; do
echo -n "The test is ${BENCH}, the maca version is " && realpath ${MACA_PATH}
${MACA_PATH}/ompi/bin/mpirun -np ${MPI_PROCESS_NUM} ${MPI_RUN_OPT} -host ${HOST_IP} ${ENV_VAR} ${TEST_DIR}/${BENCH} -b 1K -e 1G -d float -f 2 -g 1 -n 10
done报错信息如附件内容
-
安装信息：python -m pip install paddle-metax-gpu==3.3.0 -i www.paddlepaddle.org.cn/packages/stable/maca/
模型运行脚本如上传文件：使用模型：
PP-OCRv5_server_det
PP-LCNet_x1_0_doc_ori
PP-LCNet_x1_0_textline_ori
PP-OCRv5_server_rec
UVDoc -
-
镜像：cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.11.0-maca.ai3.3.0.11-torch2.6-py310-ubuntu22.04-amd64
容器构建命令：docker run -it --restart always --device=/dev/dri --device=/dev/mxcd --group-add 44 --name GLM-4.1V-9B-Thinking --device=/dev/mem --network=host --security-opt seccomp=unconfined --security-opt apparmor=unconfined --shm-size '100gb' --ulimit memlock=-1 -v /mnt/data/models/GLM-4.1V-9B-Thinking:/mnt/data/models/GLM-4.1V-9B-Thinking cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.11.0-maca.ai3.3.0.11-torch2.6-py310-ubuntu22.04-amd64
服务启动命令：vllm serve /mnt/data/models/GLM-4.1V-9B-Thinking/ --trust-remote-code --dtype auto --max-model-len 4096 --gpu-memory-utilization 0.9 --served-model-name GLM-4.1V-9B-Thinking
报错：
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] EngineCore failed to start.
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] Traceback (most recent call last):
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 1057, in call_hf_processor
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] output = hf_processor(data,
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/transformers/models/glm4v/processing_glm4v.py", line 150, in call
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] videos_inputs = self.video_processor(videos=videos, output_kwargs["videos_kwargs"])
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/transformers/video_processing_utils.py", line 206, in call
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] return self.preprocess(videos, kwargs)
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/transformers/video_processing_utils.py", line 387, in preprocess
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] preprocessed_videos = self._preprocess(videos=videos, kwargs)
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/transformers/models/glm4v/video_processing_glm4v.py", line 177, in preprocess
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] resized_height, resized_width = smart_resize(
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/transformers/models/glm4v/image_processing_glm4v.py", line 59, in smart_resize
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] raise ValueError(f"t:{num_frames} must be larger than temporal_factor:{temporal_factor}")
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] ValueError: t:1 must be larger than temporal_factor:2
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708]
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] The above exception was the direct cause of the following exception:
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708]
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] Traceback (most recent call last):
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 699, in run_engine_core
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] engine_core = EngineCoreProc(args, kwargs)
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 498, in init
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] super().init(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 83, in init
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 54, in init
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] self._init_executor()
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 54, in _init_executor
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] self.collective_rpc("init_device")
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/utils/init.py", line 3122, in run_method
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] return func(args, **kwargs)
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 259, in init_device
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] self.worker.init_device() # type: ignore
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 201, in init_device
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] self.model_runner: GPUModelRunner = GPUModelRunner(
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 421, in init
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] self.mm_budget = MultiModalBudget(
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/utils.py", line 47, in init
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] max_tokens_by_modality = mm_registry \
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/multimodal/registry.py", line 167, in get_max_tokens_per_item_by_nonzero_modality
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] max_tokens_per_item = self.get_max_tokens_per_item_by_modality(
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/multimodal/registry.py", line 143, in get_max_tokens_per_item_by_modality
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] return profiler.get_mm_max_contiguous_tokens(
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/multimodal/profiling.py", line 282, in get_mm_max_contiguous_tokens
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] return self._get_mm_max_tokens(seq_len,
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/multimodal/profiling.py", line 262, in _get_mm_max_tokens
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] mm_inputs = self._get_dummy_mm_inputs(seq_len, mm_counts)
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/multimodal/profiling.py", line 173, in _get_dummy_mm_inputs
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] return self.processor.apply(
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 2036, in apply
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] ) = self._cached_apply_hf_processor(
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 1826, in _cached_apply_hf_processor
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] ) = self._apply_hf_processor_main(
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 1572, in _apply_hf_processor_main
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] mm_processed_data = self._apply_hf_processor_mm_only(
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 1529, in _apply_hf_processor_mm_only
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] , mm_processed_data, _ = self._apply_hf_processor_text_mm(
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 1456, in _apply_hf_processor_text_mm
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] processed_data = self._call_hf_processor(
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/glm4_1v.py", line 1207, in _call_hf_processor
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] video_outputs = super()._call_hf_processor(
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 1417, in _call_hf_processor
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] return self.info.ctx.call_hf_processor(
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] File "/opt/conda/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 1080, in call_hf_processor
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] raise ValueError(msg) from exc
(EngineCore_DP0 pid=137) ERROR 12-19 09:43:42 [core.py:708] ValueError: Failed to apply Glm4vProcessor on data={'text': '<|begin_of_video|><|video|><|end_of_video|>', 'videos': [[array([[[[255, 255, 255],