多个卡带宽太低了，即使只用一个网卡 结果也是一样的
wanglifeng
- Members
- Joined 2025年9月16日
-
-
结果:
MCCL INFO Connected all trees1024 256 float sum -1 47.37 0.02 0.04 0 44.43 0.02 0.04 0 2048 512 float sum -1 43.77 0.05 0.09 0 43.64 0.05 0.09 0 4096 1024 float sum -1 46.11 0.09 0.17 0 47.15 0.09 0.16 0 8192 2048 float sum -1 47.47 0.17 0.32 0 45.45 0.18 0.34 0 16384 4096 float sum -1 48.26 0.34 0.64 0 54.89 0.30 0.56 0 32768 8192 float sum -1 51.41 0.64 1.20 0 56.86 0.58 1.08 0 65536 16384 float sum -1 131.29 0.50 0.94 0 129.71 0.51 0.95 0 131072 32768 float sum -1 132.38 0.99 1.86 0 133.09 0.98 1.85 0 262144 65536 float sum -1 136.61 1.92 3.60 0 138.39 1.89 3.55 0 524288 131072 float sum -1 151.87 3.45 6.47 0 160.54 3.27 6.12 0 1048576 262144 float sum -1 190.91 5.49 10.30 0 175.86 5.96 11.18 0 2097152 524288 float sum -1 387.85 5.41 10.14 0 216.03 9.71 18.20 0 4194304 1048576 float sum -1 291.62 14.38 26.97 0 286.74 14.63 27.43 0 8388608 2097152 float sum -1 426.88 19.65 36.85 0 430.51 19.49 36.54 0 16777216 4194304 float sum -1 670.59 25.02 46.91 0 753.13 22.28 41.77 0 33554432 8388608 float sum -1 1503.81 22.31 41.84 0 1541.25 21.77 40.82 0 67108864 16777216 float sum -1 2721.40 24.66 46.24 0 2719.98 24.67 46.26 0134217728 33554432 float sum -1 5417.88 24.77 46.45 0 5400.53 24.85 46.60 0
268435456 67108864 float sum -1 10798.70 24.86 46.61 0 10757.76 24.95 46.79 0
536870912 134217728 float sum -1 21586.39 24.87 46.63 0 21569.71 24.89 46.67 0
1073741824 268435456 float sum -1 43066.86 24.93 46.75 0 43017.88 24.96 46.80 0
MCCL INFO comm 0x7f279eec4010 rank 14 nranks 16 cudaDev 6 busId e3000 - Destroy COMPLETE
458098:458098 [6] MCCL INFO comm 0x7f12d7f12010 rank 6 nranks 16 cudaDev 6 busId e3000 - Destroy COMPLETE
:458092 [0] MCCL INFO comm 0x7f7a5329c010 rank 0 nranks 16 cudaDev 0 busId 23000 - Destroy COMPLETE
[4] MCCL INFO comm 0x7fb504712010 rank 12 nranks 16 cudaDev 4 busId a3000 - Destroy COMPLETE
Out of bounds values : 0 OK
Avg bus bandwidth : 20.1137
脚本：
!/bin/bash
set -euo pipefail
MACA_PATH=/opt/maca
HOST_IP=${HOST_IP:-"机器信息"}
IP_MASK=${IP_MASK:-"172.16.1.0/24"}
GPU_NUM=${GPU_NUM:-64}
IB_PORT="mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7"
GID_INDEX=${GID_INDEX:-3}
TEST_DIR=/opt/maca/samples/mccl_tests/perf/mccl_perf
BENCH_NAMES=${BENCH_NAMES:-"all_reduce_perf"}
PERF_ENV="-x FORCE_ACTIVE_WAIT=2"
LIB_PATH_ENV="-x LD_LIBRARY_PATH=${MACA_PATH}/lib:${MACA_PATH}/ompi/lib"
ENV_VAR="\
-x OMPI_ALLOW_RUN_AS_ROOT=1 \
-x OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 \
-x MCCL_IB_HCA=${IB_PORT} \
-x MCCL_IB_GID_INDEX=${GID_INDEX} \
-x MCCL_SOCKET_IFNAME=eth10 \ #尝试增加多个mlx卡 通信也无效
-x MCCL_CROSS_NIC=0 \ # 0，1尝试也无效
${PERF_ENV} \
${LIB_PATH_ENV}"
MPI_PROCESS_NUM=${GPU_NUM}
MPI_RUN_OPT="\
-mca btl_tcp_if_include ${IP_MASK} \
-mca oob_tcp_if_include ${IP_MASK} \
-mca pml ^ucx \
-mca osc ^ucx \
-mca btl ^openib"
for BENCH in ${BENCH_NAMES}; do
echo "The test is ${BENCH}, the maca version is $(realpath ${MACA_PATH})"
echo "HOST_IP=${HOST_IP}"
echo "IP_MASK=${IP_MASK}"
echo "GPU_NUM=${GPU_NUM}"
echo "IB_PORT=${IB_PORT}"
echo "GID_INDEX=${GID_INDEX}"
${MACA_PATH}/ompi/bin/mpirun \
--allow-run-as-root \ -np ${MPI_PROCESS_NUM} \ ${MPI_RUN_OPT} \ -host ${HOST_IP} \ ${ENV_VAR} \ ${TEST_DIR}/${BENCH} \ -b 1K -e 1G -d float -f 2 -g 1 -n 10done
-
-
-
-
-
服务器：
h3c服务器
芯片：
c550
操作系统：
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="www.ubuntu.com/"
SUPPORT_URL="help.ubuntu.com/"
BUG_REPORT_URL="bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
启动的模型：qwen3-vl-235b
镜像启动：docker run \
--network=host \
--device /dev/dri:/dev/dri \
--device /dev/mxcd:/dev/mxcd \
--group-add video \
--runtime=runc \
--detach=true \
--shm-size 100gb \
--ulimit memlock=-1 \
-it \
cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.12.0-maca.ai3.3.0.204-torch2.8-py312-ubuntu22.04-amd64
模型启动命令：vllm serve qwen3-vl-235b-a22 --tensor-parallel-size 8 --max-model-len 1024 --enable-chunked-prefill --max-num-batched-tokens 2048 --trust-remote-code --gpu-memory-utilization 0.80 --mm-processor-cache-gb 0看样子是卡在多卡通信如何解决
-
mxc500 Ubuntu 22.04. docker 27.5.1
使用镜像megatron-lm:maca.ai3.0.0.5-torch2.4-py310-ubuntu22.04-amd64
启动命令
docker run -itd --device=/dev/dri --device=/dev/mxcd --group-add video -
-network=host --name megatron_lm --security-opt seccomp=unconfined --
security-opt apparmor=unconfined --shm-size 100gb --ulimit memlock=-1 -
v /data:/data
我理解我的问题和基础的环境均无关，是否是该镜像编译的时候少安装包了，导致镜像中执行脚本报错 -
在使用/workspace/Pai-Megatron-Patch/toolkits/model_checkpoints_convertor/qwen/进行模型转换时报错
转换的模型为qwen3-8b ,但是目前看导入包的时候就报错了
使用的镜像megatron-lm:maca.ai3.0.0.5-torch2.4-py310-ubuntu22.04-amd64
hf2mcore_qwen2_dense_and_moe_gqa.py
File "/workspace/Pai-Megatron-Patch/toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen2_dense_and_moe_gqa.py", line 12, in <module>
from transformers.modeling_utils import WEIGHTS_INDEX_NAME, WEIGHTS_NAME, shard_checkpoint, load_sharded_checkpoint
ImportError: cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' (/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py)
E0916 16:49:29.393000 140209256093504 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 407) of binary: /opt/conda/bin/python3.10