是否确定有存在这样的镜像，我再去渠道获取下
wanglifeng
- Members
- Joined 2025年9月16日
-
root@i-yTUxSCxNM:~# sudo ./metax-driver-3.9.0.14-deb-x86_64.run -- -f
Verifying archive integrity... 100% MD5 checksums are OK. All good.
Uncompressing MetaX driver installer 100%
uninstalling metax-linux
(Reading database ... 116825 files and directories currently installed.)
Removing metax-linux (3.10.14) ...
metax-linux package uninstalled successfully
Purging configuration files for metax-linux (3.10.14) ...
metax-linux package uninstalled successfully
uninstalling mxsmt
(Reading database ... 116705 files and directories currently installed.)
Removing mxsmt (3.9.0.14) ...
mxsmt package uninstalled successfully
Purging configuration files for mxsmt (3.9.0.14) ...
mxsmt package uninstalled successfully
dpkg: warning: while removing mxsmt, directory '/opt/mxdriver' not empty so not removed
uninstalling mxfw
(Reading database ... 116692 files and directories currently installed.)
Removing mxfw (3.9.0) ...
mxfw package uninstalled successfully
Purging configuration files for mxfw (3.9.0) ...
mxfw package uninstalled successfullydeb/mxfw_.deb
deb/mxsmt_.deb
deb/metax-linux[-_]*.deb
installing packages under /opt/mxdriver
installing deb/mxfw_3.9.0-14.all.deb
Selecting previously unselected package mxfw.
(Reading database ... 116583 files and directories currently installed.)
Preparing to unpack deb/mxfw_3.9.0-14.all.deb ...
Unpacking mxfw (3.9.0) ...
Setting up mxfw (3.9.0) ...
mxfw package installed successfully
installing deb/mxsmt_3.9.0.14.amd64.deb
Selecting previously unselected package mxsmt.
(Reading database ... 116692 files and directories currently installed.)
Preparing to unpack deb/mxsmt_3.9.0.14.amd64.deb ...
Unpacking mxsmt (3.9.0.14) ...
Setting up mxsmt (3.9.0.14) ...
mxsmt package installed successfully
Processing triggers for libc-bin (2.35-0ubuntu3.15) ...
installing deb/metax-linux_3.10.14-14_amd64.deb
Selecting previously unselected package metax-linux.
(Reading database ... 116705 files and directories currently installed.)
Preparing to unpack .../metax-linux_3.10.14-14_amd64.deb ...
Unpacking metax-linux (3.10.14) ...
Setting up metax-linux (3.10.14) ...
metax-linux package installed successfully -
按照官网文档，我安装了最新驱动metax-driver-3.9.0.14-deb-x86_64.run，但是maca的版本是unknow ,不管安装哪个版本都是unknow，是否是还需要安装除了驱动以外的，我看官方文档安装驱动也相当于安装了固件什么的
root@i-yTUxSCxNM:~# mx-smi
mx-smi version: 2.3.4=================== MetaX System Management Interface Log ===================
Timestamp : Mon Sep 21 05:58:13 2026Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.3.4 Kernel Mode Driver Version: 3.9.35 |
| MACA Version: unknown BIOS Version: 1.22.3.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C550 | 0 N/A | 0000:23:00.0 | 0% Disabled |
| 84W / 450W | 36C N/A | 860/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C550 | 1 N/A | 0000:26:00.0 | 0% Disabled |
| 84W / 450W | 35C N/A | 860/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C550 | 2 N/A | 0000:63:00.0 | 0% Disabled |
| 81W / 450W | 35C N/A | 860/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C550 | 3 N/A | 0000:66:00.0 | 0% Disabled |
| 79W / 450W | 37C N/A | 860/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C550 | 4 N/A | 0000:a3:00.0 | 0% Disabled |
| 85W / 450W | 38C N/A | 860/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C550 | 5 N/A | 0000:a4:00.0 | 0% Disabled |
| 92W / 450W | 36C N/A | 860/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C550 | 6 N/A | 0000:e3:00.0 | 0% Disabled |
| 86W / 450W | 35C N/A | 860/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C550 | 7 N/A | 0000:e4:00.0 | 0% Disabled |
| 89W / 450W | 38C N/A | 860/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+ -
-
结果:
MCCL INFO Connected all trees1024 256 float sum -1 47.37 0.02 0.04 0 44.43 0.02 0.04 0 2048 512 float sum -1 43.77 0.05 0.09 0 43.64 0.05 0.09 0 4096 1024 float sum -1 46.11 0.09 0.17 0 47.15 0.09 0.16 0 8192 2048 float sum -1 47.47 0.17 0.32 0 45.45 0.18 0.34 0 16384 4096 float sum -1 48.26 0.34 0.64 0 54.89 0.30 0.56 0 32768 8192 float sum -1 51.41 0.64 1.20 0 56.86 0.58 1.08 0 65536 16384 float sum -1 131.29 0.50 0.94 0 129.71 0.51 0.95 0 131072 32768 float sum -1 132.38 0.99 1.86 0 133.09 0.98 1.85 0 262144 65536 float sum -1 136.61 1.92 3.60 0 138.39 1.89 3.55 0 524288 131072 float sum -1 151.87 3.45 6.47 0 160.54 3.27 6.12 0 1048576 262144 float sum -1 190.91 5.49 10.30 0 175.86 5.96 11.18 0 2097152 524288 float sum -1 387.85 5.41 10.14 0 216.03 9.71 18.20 0 4194304 1048576 float sum -1 291.62 14.38 26.97 0 286.74 14.63 27.43 0 8388608 2097152 float sum -1 426.88 19.65 36.85 0 430.51 19.49 36.54 0 16777216 4194304 float sum -1 670.59 25.02 46.91 0 753.13 22.28 41.77 0 33554432 8388608 float sum -1 1503.81 22.31 41.84 0 1541.25 21.77 40.82 0 67108864 16777216 float sum -1 2721.40 24.66 46.24 0 2719.98 24.67 46.26 0134217728 33554432 float sum -1 5417.88 24.77 46.45 0 5400.53 24.85 46.60 0
268435456 67108864 float sum -1 10798.70 24.86 46.61 0 10757.76 24.95 46.79 0
536870912 134217728 float sum -1 21586.39 24.87 46.63 0 21569.71 24.89 46.67 0
1073741824 268435456 float sum -1 43066.86 24.93 46.75 0 43017.88 24.96 46.80 0
MCCL INFO comm 0x7f279eec4010 rank 14 nranks 16 cudaDev 6 busId e3000 - Destroy COMPLETE
458098:458098 [6] MCCL INFO comm 0x7f12d7f12010 rank 6 nranks 16 cudaDev 6 busId e3000 - Destroy COMPLETE
:458092 [0] MCCL INFO comm 0x7f7a5329c010 rank 0 nranks 16 cudaDev 0 busId 23000 - Destroy COMPLETE
[4] MCCL INFO comm 0x7fb504712010 rank 12 nranks 16 cudaDev 4 busId a3000 - Destroy COMPLETE
Out of bounds values : 0 OK
Avg bus bandwidth : 20.1137
脚本：
!/bin/bash
set -euo pipefail
MACA_PATH=/opt/maca
HOST_IP=${HOST_IP:-"机器信息"}
IP_MASK=${IP_MASK:-"172.16.1.0/24"}
GPU_NUM=${GPU_NUM:-64}
IB_PORT="mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7"
GID_INDEX=${GID_INDEX:-3}
TEST_DIR=/opt/maca/samples/mccl_tests/perf/mccl_perf
BENCH_NAMES=${BENCH_NAMES:-"all_reduce_perf"}
PERF_ENV="-x FORCE_ACTIVE_WAIT=2"
LIB_PATH_ENV="-x LD_LIBRARY_PATH=${MACA_PATH}/lib:${MACA_PATH}/ompi/lib"
ENV_VAR="\
-x OMPI_ALLOW_RUN_AS_ROOT=1 \
-x OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 \
-x MCCL_IB_HCA=${IB_PORT} \
-x MCCL_IB_GID_INDEX=${GID_INDEX} \
-x MCCL_SOCKET_IFNAME=eth10 \ #尝试增加多个mlx卡 通信也无效
-x MCCL_CROSS_NIC=0 \ # 0，1尝试也无效
${PERF_ENV} \
${LIB_PATH_ENV}"
MPI_PROCESS_NUM=${GPU_NUM}
MPI_RUN_OPT="\
-mca btl_tcp_if_include ${IP_MASK} \
-mca oob_tcp_if_include ${IP_MASK} \
-mca pml ^ucx \
-mca osc ^ucx \
-mca btl ^openib"
for BENCH in ${BENCH_NAMES}; do
echo "The test is ${BENCH}, the maca version is $(realpath ${MACA_PATH})"
echo "HOST_IP=${HOST_IP}"
echo "IP_MASK=${IP_MASK}"
echo "GPU_NUM=${GPU_NUM}"
echo "IB_PORT=${IB_PORT}"
echo "GID_INDEX=${GID_INDEX}"
${MACA_PATH}/ompi/bin/mpirun \
--allow-run-as-root \ -np ${MPI_PROCESS_NUM} \ ${MPI_RUN_OPT} \ -host ${HOST_IP} \ ${ENV_VAR} \ ${TEST_DIR}/${BENCH} \ -b 1K -e 1G -d float -f 2 -g 1 -n 10done
-
-
-
-
-
服务器：
h3c服务器
芯片：
c550
操作系统：
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="www.ubuntu.com/"
SUPPORT_URL="help.ubuntu.com/"
BUG_REPORT_URL="bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
启动的模型：qwen3-vl-235b
镜像启动：docker run \
--network=host \
--device /dev/dri:/dev/dri \
--device /dev/mxcd:/dev/mxcd \
--group-add video \
--runtime=runc \
--detach=true \
--shm-size 100gb \
--ulimit memlock=-1 \
-it \
cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.12.0-maca.ai3.3.0.204-torch2.8-py312-ubuntu22.04-amd64
模型启动命令：vllm serve qwen3-vl-235b-a22 --tensor-parallel-size 8 --max-model-len 1024 --enable-chunked-prefill --max-num-batched-tokens 2048 --trust-remote-code --gpu-memory-utilization 0.80 --mm-processor-cache-gb 0看样子是卡在多卡通信如何解决
-
mxc500 Ubuntu 22.04. docker 27.5.1
使用镜像megatron-lm:maca.ai3.0.0.5-torch2.4-py310-ubuntu22.04-amd64
启动命令
docker run -itd --device=/dev/dri --device=/dev/mxcd --group-add video -
-network=host --name megatron_lm --security-opt seccomp=unconfined --
security-opt apparmor=unconfined --shm-size 100gb --ulimit memlock=-1 -
v /data:/data
我理解我的问题和基础的环境均无关，是否是该镜像编译的时候少安装包了，导致镜像中执行脚本报错 -
在使用/workspace/Pai-Megatron-Patch/toolkits/model_checkpoints_convertor/qwen/进行模型转换时报错
转换的模型为qwen3-8b ,但是目前看导入包的时候就报错了
使用的镜像megatron-lm:maca.ai3.0.0.5-torch2.4-py310-ubuntu22.04-amd64
hf2mcore_qwen2_dense_and_moe_gqa.py
File "/workspace/Pai-Megatron-Patch/toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen2_dense_and_moe_gqa.py", line 12, in <module>
from transformers.modeling_utils import WEIGHTS_INDEX_NAME, WEIGHTS_NAME, shard_checkpoint, load_sharded_checkpoint
ImportError: cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' (/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py)
E0916 16:49:29.393000 140209256093504 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 407) of binary: /opt/conda/bin/python3.10