已按要求reboot后 还是一样的问题，无法成功
2026年7月30日
24
-
-
尊敬的开发者您好，请联系售后渠道定位此问题
-
我在不启动docker的情况下执行bash /opt/maca/samples/mccl_tests/perf/mccl.sh 1、bash /opt/maca/samples/mccl_tests/perf/mccl.sh 4、bash /opt/maca/samples/mccl_tests/perf/mccl.sh 8，日志如下 ，这个能说明是八卡互联通信存在问题吗？
app@muxi-gpu-server:~$ bash /opt/maca/samples/mccl_tests/perf/mccl.sh 1
The test is all_reduce_perf, the maca version is /opt/maca-3.8.0
main_process = 457322
===============================nThread 1 nGpus 1 minBytes 1024 maxBytes 1073741824 step: 2(factor) warmup iters: 5 iters: 10 agg iters: 1 validation: 1 graph: 0
Using devices
Rank 0 Pid 457322 on muxi-gpu-server device 0 [0x23] MetaX C550
┌----- out-of-place ------┐ ┌------ in-place -------┐
size count type redop root time algbw busbw #wrong time algbw busbw #wrong
(B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
1024 512 bfloat16 sum -1 5.85 0.18 0.00 0 0.28 3.69 0.00 0 2048 1024 bfloat16 sum -1 5.17 0.40 0.00 0 0.30 6.88 0.00 0 4096 2048 bfloat16 sum -1 5.52 0.74 0.00 0 0.27 15.03 0.00 0 8192 4096 bfloat16 sum -1 5.17 1.58 0.00 0 0.27 30.51 0.00 0 16384 8192 bfloat16 sum -1 5.40 3.03 0.00 0 0.27 60.57 0.00 0 32768 16384 bfloat16 sum -1 6.11 5.36 0.00 0 0.27 120.65 0.00 0 65536 32768 bfloat16 sum -1 6.30 10.40 0.00 0 0.28 237.88 0.00 0 131072 65536 bfloat16 sum -1 7.32 17.89 0.00 0 0.29 454.32 0.00 0 262144 131072 bfloat16 sum -1 9.32 28.12 0.00 0 0.27 954.99 0.00 0 524288 262144 bfloat16 sum -1 13.45 38.99 0.00 0 0.27 1945.41 0.00 0 1048576 524288 bfloat16 sum -1 22.11 47.42 0.00 0 0.27 3862.16 0.00 0 2097152 1048576 bfloat16 sum -1 39.14 53.58 0.00 0 0.27 7667.83 0.00 0 4194304 2097152 bfloat16 sum -1 28.19 148.78 0.00 0 0.29 14533.28 0.00 0 8388608 4194304 bfloat16 sum -1 35.12 238.88 0.00 0 0.28 30338.55 0.00 0 16777216 8388608 bfloat16 sum -1 46.05 364.34 0.00 0 0.28 59388.38 0.00 0 33554432 16777216 bfloat16 sum -1 72.91 460.20 0.00 0 0.28 118734.72 0.00 0 67108864 33554432 bfloat16 sum -1 115.19 582.59 0.00 0 0.28 243589.34 0.00 0134217728 67108864 bfloat16 sum -1 207.48 646.89 0.00 0 0.30 440780.72 0.00 0
268435456 134217728 bfloat16 sum -1 399.89 671.28 0.00 0 0.28 963861.60 0.00 0
536870912 268435456 bfloat16 sum -1 761.13 705.36 0.00 0 0.25 2187738.03 0.00 0
1073741824 536870912 bfloat16 sum -1 1495.97 717.76 0.00 0 0.21 5177154.41 0.00 0Out of bounds values : 0 OK
Avg bus bandwidth : 0
app@muxi-gpu-server:~$
-
就是这个性能测试脚本的作用是什么？bash /opt/maca/samples/mccl_tests/perf/mccl.sh 1 ，用一张卡能测试出数据，但是2、4、8 多张卡的时候都没有数据，这个能判断是多卡互联通信存在问题吗？
-
尊敬的开发者您好，八卡互联存在问题，请联系售后渠道
-
补充测试日志：app@muxi-gpu-server:~$ bash /opt/maca/samples/mccl_tests/perf/mccl.sh 1
The test is all_reduce_perf, the maca version is /opt/maca-3.8.0
main_process = 501528
===============================nThread 1 nGpus 1 minBytes 1024 maxBytes 1073741824 step: 2(factor) warmup iters: 5 iters: 10 agg iters: 1 validation: 1 graph: 0
Using devices
Rank 0 Pid 501528 on muxi-gpu-server device 0 [0x23] MetaX C550
┌----- out-of-place ------┐ ┌------ in-place -------┐
size count type redop root time algbw busbw #wrong time algbw busbw #wrong
(B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
1024 512 bfloat16 sum -1 5.33 0.19 0.00 0 0.28 3.70 0.00 0 2048 1024 bfloat16 sum -1 5.29 0.39 0.00 0 0.28 7.20 0.00 0 4096 2048 bfloat16 sum -1 5.34 0.77 0.00 0 0.27 15.20 0.00 0 8192 4096 bfloat16 sum -1 5.20 1.57 0.00 0 0.27 30.40 0.00 0 16384 8192 bfloat16 sum -1 5.32 3.08 0.00 0 0.27 60.79 0.00 0 32768 16384 bfloat16 sum -1 5.48 5.98 0.00 0 0.28 116.82 0.00 0 65536 32768 bfloat16 sum -1 6.09 10.76 0.00 0 0.27 244.08 0.00 0 131072 65536 bfloat16 sum -1 7.39 17.74 0.00 0 0.27 479.24 0.00 0 262144 131072 bfloat16 sum -1 9.21 28.45 0.00 0 0.28 951.52 0.00 0 524288 262144 bfloat16 sum -1 13.63 38.46 0.00 0 0.27 1923.99 0.00 0 1048576 524288 bfloat16 sum -1 22.20 47.23 0.00 0 0.27 3876.44 0.00 0 2097152 1048576 bfloat16 sum -1 39.87 52.61 0.00 0 0.28 7527.47 0.00 0 4194304 2097152 bfloat16 sum -1 29.72 141.15 0.00 0 0.27 15279.80 0.00 0 8388608 4194304 bfloat16 sum -1 34.49 243.21 0.00 0 0.27 30885.89 0.00 0 16777216 8388608 bfloat16 sum -1 46.04 364.44 0.00 0 0.27 61567.77 0.00 0 33554432 16777216 bfloat16 sum -1 68.91 486.96 0.00 0 0.27 123589.07 0.00 0 67108864 33554432 bfloat16 sum -1 115.29 582.11 0.00 0 0.27 249939.90 0.00 0134217728 67108864 bfloat16 sum -1 208.51 643.71 0.00 0 0.21 640963.36 0.00 0
268435456 134217728 bfloat16 sum -1 397.83 674.74 0.00 0 0.19 1440104.38 0.00 0
536870912 268435456 bfloat16 sum -1 758.84 707.49 0.00 0 0.19 2849633.29 0.00 0
1073741824 536870912 bfloat16 sum -1 1499.05 716.28 0.00 0 0.29 3683505.40 0.00 0Out of bounds values : 0 OK
Avg bus bandwidth : 0
app@muxi-gpu-server:~$ bash /opt/maca/samples/mccl_tests/perf/mccl.sh 2
The test is all_reduce_perf, the maca version is /opt/maca-3.8.0
main_process = 501623
===============================nThread 1 nGpus 1 minBytes 1024 maxBytes 1073741824 step: 2(factor) warmup iters: 5 iters: 10 agg iters: 1 validation: 1 graph: 0
Using devices
Rank 0 Pid 501623 on muxi-gpu-server device 0 [0x23] MetaX C550
Rank 1 Pid 501624 on muxi-gpu-server device 1 [0x26] MetaX C550
┌----- out-of-place ------┐ ┌------ in-place -------┐
size count type redop root time algbw busbw #wrong time algbw busbw #wrong
(B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
^C^Capp@muxi-gpu-server:~$ bash /opt/maca/samples/mccl_tests/perf/mccl.sh 8
The test is all_reduce_perf, the maca version is /opt/maca-3.8.0
main_process = 503189
===============================nThread 1 nGpus 1 minBytes 1024 maxBytes 1073741824 step: 2(factor) warmup iters: 5 iters: 10 agg iters: 1 validation: 1 graph: 0
Using devices
Rank 0 Pid 503189 on muxi-gpu-server device 0 [0x23] MetaX C550
Rank 1 Pid 503190 on muxi-gpu-server device 1 [0x26] MetaX C550
Rank 2 Pid 503191 on muxi-gpu-server device 2 [0x63] MetaX C550
Rank 3 Pid 503192 on muxi-gpu-server device 3 [0x66] MetaX C550
Rank 4 Pid 503193 on muxi-gpu-server device 4 [0xa3] MetaX C550
Rank 5 Pid 503194 on muxi-gpu-server device 5 [0xa4] MetaX C550
Rank 6 Pid 503195 on muxi-gpu-server device 6 [0xe3] MetaX C550
Rank 7 Pid 503196 on muxi-gpu-server device 7 [0xe4] MetaX C550
┌----- out-of-place ------┐ ┌------ in-place -------┐
size count type redop root time algbw busbw #wrong time algbw busbw #wrong
(B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
^C^Capp@muxi-gpu-server:~$ bash /opt/maca/samples/mccl_tests/perf/mccl.sh 4
The test is all_reduce_perf, the maca version is /opt/maca-3.8.0
main_process = 504027
===============================nThread 1 nGpus 1 minBytes 1024 maxBytes 1073741824 step: 2(factor) warmup iters: 5 iters: 10 agg iters: 1 validation: 1 graph: 0
Using devices
Rank 0 Pid 504027 on muxi-gpu-server device 0 [0x23] MetaX C550
Rank 1 Pid 504028 on muxi-gpu-server device 1 [0x26] MetaX C550
Rank 2 Pid 504029 on muxi-gpu-server device 2 [0x63] MetaX C550
Rank 3 Pid 504030 on muxi-gpu-server device 3 [0x66] MetaX C550
┌----- out-of-place ------┐ ┌------ in-place -------┐
size count type redop root time algbw busbw #wrong time algbw busbw #wrong
(B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
-
服务器未关闭ACS导致pcie逻辑隔离，卡间无法通信 。修改grub配置后系统 恢复正常通信
-
尊敬的开发者您好，感谢您的回复
-
- By shuai_chen on 2026年8月10日 10:16.