服务器信息：h3c
gpu：mx c550
cpu：海光
rdma网卡:8 * mlx
问题描述：
无论如何增加HCA高性能网卡，多机allreduce等最后带宽只打满了一张网卡，也就是全部都走了一个网卡，MCCL_CROSS_NIC也分别修改过0 1 没有效果
MCCL_IB_HCA=mlx5_0,mlx5_1,.....
rdma网络正常,，网卡间通信正常，所有机器的网卡1属于子网a，网卡2属于子网b，一一对应，调整了很多参数还是无法做到让allreduce测试走多个网卡
wanglifeng
- Members
- Joined 2025年9月16日
-
服务器：
h3c服务器
芯片：
c550
操作系统：
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="www.ubuntu.com/"
SUPPORT_URL="help.ubuntu.com/"
BUG_REPORT_URL="bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
启动的模型：qwen3-vl-235b
镜像启动：docker run \
--network=host \
--device /dev/dri:/dev/dri \
--device /dev/mxcd:/dev/mxcd \
--group-add video \
--runtime=runc \
--detach=true \
--shm-size 100gb \
--ulimit memlock=-1 \
-it \
cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.12.0-maca.ai3.3.0.204-torch2.8-py312-ubuntu22.04-amd64
模型启动命令：vllm serve qwen3-vl-235b-a22 --tensor-parallel-size 8 --max-model-len 1024 --enable-chunked-prefill --max-num-batched-tokens 2048 --trust-remote-code --gpu-memory-utilization 0.80 --mm-processor-cache-gb 0看样子是卡在多卡通信如何解决
-
在使用/workspace/Pai-Megatron-Patch/toolkits/model_checkpoints_convertor/qwen/进行模型转换时报错
转换的模型为qwen3-8b ,但是目前看导入包的时候就报错了
使用的镜像megatron-lm:maca.ai3.0.0.5-torch2.4-py310-ubuntu22.04-amd64
hf2mcore_qwen2_dense_and_moe_gqa.py
File "/workspace/Pai-Megatron-Patch/toolkits/model_checkpoints_convertor/qwen/hf2mcore_qwen2_dense_and_moe_gqa.py", line 12, in <module>
from transformers.modeling_utils import WEIGHTS_INDEX_NAME, WEIGHTS_NAME, shard_checkpoint, load_sharded_checkpoint
ImportError: cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' (/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py)
E0916 16:49:29.393000 140209256093504 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 407) of binary: /opt/conda/bin/python3.10