Posts | uncle4 | 沐曦开发者论坛

See post chevron_right

uncle4
Members

C500推理提问回答后返回为null 已解决 2026年6月1日 10:10

我的权重模型与魔塔给出的sha256进行了比对，没有差异，而且在developer.metax-tech.com/softnova/docker?chip_name=%E6%9B%A6%E4%BA%91C500%E7%B3%BB%E5%88%97&package_name=vllm-metax:0.19.0-maca.ai3.5.3.502-torch2.8-py312-ubuntu22.04-amd64 镜像下成功推理，并做出问答。
See post chevron_right

uncle4
Members

C500推理提问回答后返回为null 已解决 2026年5月29日 13:55

模型地址：modelscope.cn/models/unsloth/DeepSeek-V3-bf16/files
镜像地址：developer.metax-tech.com/softnova/docker-preview?chip_name=曦云C500系列&package_kind=AI&dimension=docker&deliver_type=分层包&release_type=preview
See post chevron_right

uncle4
Members

C500推理提问回答后返回为null 已解决 2026年5月29日 13:46

4台环境docker logs内容在这里，启动镜像和执行推理的脚本为deploy_deepseek_v3_vllm.txt，你需要我补充什么内容进行支持？
See post chevron_right

uncle4
Members

C500推理提问回答后返回为null 已解决 2026年5月29日 13:22

1.服务器厂家:

H3R5300G6

2.沐曦GPU型号：
C500

3.操作系统内核版本：

root@muxi-45:~# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.5 LTS"
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="www.ubuntu.com/"
SUPPORT_URL="help.ubuntu.com/"
BUG_REPORT_URL="bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
distribution_version=v0.3.205
firmware_version=v0.3.132
driver_version=v0.3.165
root@muxi-45:~#

4.是否开启CPU虚拟化：

KVM acceleration can be used

5.mx-smi回显：

=================== MetaX System Management Interface Log ===================
Timestamp : Thu May 28 02:22:37 2026

Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.3.1 Kernel Mode Driver Version: 3.8.23 |
| MACA Version: 3.7.0.38 BIOS Version: 1.33.4.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C500 | 0 Off | 0000:08:00.0 | 0% Disabled |
| 69W / 350W | 45C P9 | 58096/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C500 | 1 Off | 0000:09:00.0 | 0% Disabled |
| 73W / 350W | 47C P9 | 58736/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C500 | 2 Off | 0000:0e:00.0 | 0% Disabled |
| 76W / 350W | 45C P9 | 58736/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C500 | 3 Off | 0000:11:00.0 | 0% Disabled |
| 74W / 350W | 46C P9 | 58096/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C500 | 4 Off | 0000:32:00.0 | 0% Disabled |
| 71W / 350W | 45C P9 | 58096/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C500 | 5 Off | 0000:38:00.0 | 0% Disabled |
| 71W / 350W | 45C P9 | 58736/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C500 | 6 Off | 0000:3b:00.0 | 0% Disabled |
| 73W / 350W | 47C P9 | 58736/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C500 | 7 Off | 0000:3c:00.0 | 0% Disabled |
| 70W / 350W | 47C P9 | 58096/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+

+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| 0 73379 VLLM::Worker_PP 57216 |
| 1 73380 VLLM::Worker_PP 57856 |
| 2 73381 VLLM::Worker_PP 57856 |
| 3 73382 VLLM::Worker_PP 57216 |
| 4 73383 VLLM::Worker_PP 57216 |
| 5 73384 VLLM::Worker_PP 57856 |
| 6 73385 VLLM::Worker_PP 57856 |
| 7 73386 VLLM::Worker_PP 57216 |
+---------------------------------------------------------------------------------+

6.docker info回显：

root@muxi-45:~# docker info
Client:
Version: 29.1.3
Context: default
Debug Mode: false
Plugins:
trust: Manage trust on Docker images (Docker Inc.)
Version: 29.1.3
Path: /usr/libexec/docker/cli-plugins/docker-trust

Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 1
Server Version: 29.3.1
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 301b2dac98f15c27117da5c8af12118a041a31d9
runc version:
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-179-generic
Operating System: Ubuntu Core 24
OSType: linux
Architecture: x86_64
CPUs: 160
Total Memory: 1.968TiB
Name: muxi-45
ID: 865d1485-4d36-469e-bb69-ed557023934d
Docker Root Dir: /var/snap/docker/common/var-lib-docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
::1/128
Live Restore Enabled: false
Firewall Backend: iptables

7.镜像版本：

megatron-lm:0.14.0-maca.ai3.5.3.16-torch2.6-py310-ubuntu22.04-amd64

8.启动容器命令：

见附件

9.容器内执行命令：

见附件

二、问题现象

训练deepseekv3bf16，回答问题返回content为""
root@muxi-44:~# curl http://10.5.1.44:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model":"DeepSeek-V3-bf16",
"messages":[{"role":"user","content":"用一句话介绍你自己"}],
"temperature":0.7,
"max_tokens":1024
}'
{"id":"chatcmpl-a0288a58cc3cf02e","object":"chat.completion","created":1779905158,"model":"DeepSeek-V3-bf16","choices":[{"index":0,"message":{"role":"assistant","content":"","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null},"logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":7,"total_tokens":71,"completion_tokens":64,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids"

NODE 节点muxi-44:1073:9860 [6] MCCL INFO NET/IB: Dev 1 Port 1 qpn 472 mtu 5 GID 1 ((APIServer pid=1) INFO: 10.5.1.44:35530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 05-29 13:07:19 [loggers.py:271] Engine 000: Avg prompt throughput: 0.7 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-29 13:07:29 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-29 13:08:09 [loggers.py:271] Engine 000: Avg prompt throughput: 0.4 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-29 13:08:19 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-29 13:08:29 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-29 13:08:39 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-29 13:08:49 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 05-29 13:08:59 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
See post chevron_right

uncle4
Members

DeepSeekV3bf16 训练返回的 content 为 "" 已解决 2026年5月28日 11:42

抱歉我的描述错误。是推理。
See post chevron_right

uncle4
Members

DeepSeekV3bf16 训练返回的 content 为 "" 已解决 2026年5月28日 11:17

在附件中有容器启动和执行的命令
See post chevron_right

uncle4
Members

DeepSeekV3bf16 训练返回的 content 为 "" 已解决 2026年5月28日 10:38

modelscope.cn/models/unsloth/deepseek-V3-bf16/files
See post chevron_right

uncle4
Members

DeepSeekV3bf16 训练返回的 content 为 "" 已解决 2026年5月28日 10:29

1.服务器厂家:

H3R5300G6

2.沐曦GPU型号：
C500

3.操作系统内核版本：

root@muxi-45:~# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.5 LTS"
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="www.ubuntu.com/"
SUPPORT_URL="help.ubuntu.com/"
BUG_REPORT_URL="bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
distribution_version=v0.3.205
firmware_version=v0.3.132
driver_version=v0.3.165
root@muxi-45:~#

4.是否开启CPU虚拟化：

KVM acceleration can be used

5.mx-smi回显：

=================== MetaX System Management Interface Log ===================
Timestamp : Thu May 28 02:22:37 2026

Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.3.1 Kernel Mode Driver Version: 3.8.23 |
| MACA Version: 3.7.0.38 BIOS Version: 1.33.4.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C500 | 0 Off | 0000:08:00.0 | 0% Disabled |
| 69W / 350W | 45C P9 | 58096/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C500 | 1 Off | 0000:09:00.0 | 0% Disabled |
| 73W / 350W | 47C P9 | 58736/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C500 | 2 Off | 0000:0e:00.0 | 0% Disabled |
| 76W / 350W | 45C P9 | 58736/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C500 | 3 Off | 0000:11:00.0 | 0% Disabled |
| 74W / 350W | 46C P9 | 58096/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C500 | 4 Off | 0000:32:00.0 | 0% Disabled |
| 71W / 350W | 45C P9 | 58096/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C500 | 5 Off | 0000:38:00.0 | 0% Disabled |
| 71W / 350W | 45C P9 | 58736/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C500 | 6 Off | 0000:3b:00.0 | 0% Disabled |
| 73W / 350W | 47C P9 | 58736/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C500 | 7 Off | 0000:3c:00.0 | 0% Disabled |
| 70W / 350W | 47C P9 | 58096/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+

+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| 0 73379 VLLM::Worker_PP 57216 |
| 1 73380 VLLM::Worker_PP 57856 |
| 2 73381 VLLM::Worker_PP 57856 |
| 3 73382 VLLM::Worker_PP 57216 |
| 4 73383 VLLM::Worker_PP 57216 |
| 5 73384 VLLM::Worker_PP 57856 |
| 6 73385 VLLM::Worker_PP 57856 |
| 7 73386 VLLM::Worker_PP 57216 |
+---------------------------------------------------------------------------------+

6.docker info回显：

root@muxi-45:~# docker info
Client:
Version: 29.1.3
Context: default
Debug Mode: false
Plugins:
trust: Manage trust on Docker images (Docker Inc.)
Version: 29.1.3
Path: /usr/libexec/docker/cli-plugins/docker-trust

Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 1
Server Version: 29.3.1
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 301b2dac98f15c27117da5c8af12118a041a31d9
runc version:
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-179-generic
Operating System: Ubuntu Core 24
OSType: linux
Architecture: x86_64
CPUs: 160
Total Memory: 1.968TiB
Name: muxi-45
ID: 865d1485-4d36-469e-bb69-ed557023934d
Docker Root Dir: /var/snap/docker/common/var-lib-docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
::1/128
Live Restore Enabled: false
Firewall Backend: iptables

7.镜像版本：

megatron-lm:0.14.0-maca.ai3.5.3.16-torch2.6-py310-ubuntu22.04-amd64

8.启动容器命令：

见附件

9.容器内执行命令：

见附件

二、问题现象

训练deepseekv3bf16，回答问题返回content为""
root@muxi-44:~# curl http://10.5.1.44:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model":"DeepSeek-V3-bf16",
"messages":[{"role":"user","content":"用一句话介绍你自己"}],
"temperature":0,
"max_tokens":64
}'
{"id":"chatcmpl-a0288a58cc3cf02e","object":"chat.completion","created":1779905158,"model":"DeepSeek-V3-bf16","choices":[{"index":0,"message":{"role":"assistant","content":"","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null},"logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":7,"total_tokens":71,"completion_tokens":64,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids"

node0部分日志见附件
See post chevron_right

uncle4
Members

有没有适用MACA Driver 3.7.0.x的megatron-lm镜像。已解决 2026年5月25日 20:46

有没有适用MACA Driver 3.7.0.x的megatron-lm镜像。
See post chevron_right

uncle4
Members

沐曦C500是否支持GDRCOPY 已解决 2026年5月18日 15:08

在nccl test中，默认使用local read作为ncclIbIflush以确保GDR完成。
在不支持local read的情况下需要开启NCCL_GDRCOPY_ENABLE和NCCL_GDRCOPY_FLUSH_ENABLE，这需要依赖gdrcopy这个feature。请问C500是否支持？
See post chevron_right

uncle4
Members

/opt/maca/samples/mccl_tests/perf$ sudo bash mccl.sh 8 reduce_perf 已解决 2026年5月15日 11:59

通过图片看到，执行export MACA_VISIBLE_DEVICES=2,3,6,7 后再执行mccl.sh 通过mx-smi查看，不符合预期。
See post chevron_right

uncle4
Members

跨节点mccl出现#wrong 和Out of bounds values : 28 FAILED 已解决 2026年5月15日 11:58

一、软硬件信息
1.服务器厂家:H3C UniServer R5300 G6
2.沐曦GPU型号：C500
3.操作系统内核版本：
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.5 LTS"
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="www.ubuntu.com/"
SUPPORT_URL="help.ubuntu.com/"
BUG_REPORT_URL="bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
distribution_version=v0.3.205
firmware_version=v0.3.132
driver_version=v0.3.165
4.是否开启CPU虚拟化：
5.mx-smi回显：
mx-smi version: 2.3.1

=================== MetaX System Management Interface Log ===================
Timestamp : Wed May 13 10:06:54 2026

Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.3.1 Kernel Mode Driver Version: 3.8.23 |
| MACA Version: 3.7.0.38 BIOS Version: 1.33.4.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C500 | 0 Off | 0000:08:00.0 | 0% Disabled |
| 36W / 350W | 36C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C500 | 1 Off | 0000:09:00.0 | 0% Disabled |
| 39W / 350W | 38C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C500 | 2 Off | 0000:0e:00.0 | 0% Disabled |
| 44W / 350W | 38C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C500 | 3 Off | 0000:11:00.0 | 0% Disabled |
| 42W / 350W | 38C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C500 | 4 Off | 0000:32:00.0 | 0% Disabled |
| 38W / 350W | 37C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C500 | 5 Off | 0000:38:00.0 | 0% Disabled |
| 38W / 350W | 37C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C500 | 6 Off | 0000:3b:00.0 | 0% Disabled |
| 41W / 350W | 39C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C500 | 7 Off | 0000:3c:00.0 | 0% Disabled |
| 41W / 350W | 38C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+

+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| no process found |
+---------------------------------------------------------------------------------+

End of Log
二、问题现象
运行bash cluster.sh "10.5.1.44:1,10.5.1.45:1" 2 reduce_perf有如下文档回显。脚本也贴在附件中
See post chevron_right

uncle4
Members

/opt/maca/samples/mccl_tests/perf$ sudo bash mccl.sh 8 reduce_perf 已解决 2026年5月15日 11:42

@shuai_chen has written:

尊敬的开发者您好，按照实际GPU ID填写，export CUDA_VISIBLE_DEVICES=0,1,2,3，执行bash mccl.sh 2或者4，进行交叉验证，确认是哪几个GPU通信问题。确认后，请进行服务器下电重新插拔GPU尝试。

请确认，确定是CUDA_VISIBLE_DEVICES，而不是MACA_VISIBLE_DEVICES吗？两个参数是否有差别。
See post chevron_right

uncle4
Members

/opt/maca/samples/mccl_tests/perf$ sudo bash mccl.sh 8 reduce_perf 已解决 2026年5月15日 11:14

多次执行bash mccl.sh 2 未出现异常。多次bash mccl.sh 4 时概率出现
[muxi-44:20247] Process received signal
[muxi-44:20247] Signal: Segmentation fault (11)
[muxi-44:20247] Signal code: Address not mapped (1)
[muxi-44:20247] Failing at address: 0x2d20b170
[muxi-44:20247] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f3a2cbed520]
[muxi-44:20247] [ 1] /opt/maca/lib/libmccompiler.so(+0x94330)[0x7f3a2d494330]
[muxi-44:20247] [ 2] /opt/maca/lib/libmccompiler.so(+0x9caca)[0x7f3a2d49caca]
[muxi-44:20247] [ 3] /opt/maca/lib/libmccompiler.so(+0x82b7a)[0x7f3a2d482b7a]
[muxi-44:20247] [ 4] /opt/maca/lib/libmccompiler.so(+0x873b3)[0x7f3a2d4873b3]
[muxi-44:20247] [ 5] /opt/maca/lib/libmccompiler.so(+0x7a05a)[0x7f3a2d47a05a]
[muxi-44:20247] [ 6] /opt/maca/lib/libmccompiler.so(+0x6a9a7)[0x7f3a2d46a9a7]
[muxi-44:20247] [ 7] /opt/maca/lib/libmccl.so(+0x2a2f92)[0x7f3a2eca2f92]
[muxi-44:20247] [ 8] /lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0xb6)[0x7f3a2cbf0a56]
[muxi-44:20247] [ 9] /opt/maca/lib/libmccl.so(+0x4d483)[0x7f3a2ea4d483]
[muxi-44:20247] End of error message

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

mpirun noticed that process rank 0 with PID 0 on node muxi-44 exited on signal 11 (Segmentation fault).
See post chevron_right

uncle4
Members

/opt/maca/samples/mccl_tests/perf$ sudo bash mccl.sh 8 reduce_perf 已解决 2026年5月14日 15:04

执行bash mccl.sh 8 后多种回显
See post chevron_right

uncle4
Members

/opt/maca/samples/mccl_tests/perf$ sudo bash mccl.sh 8 reduce_perf 已解决 2026年5月14日 14:58

不同通信原语在环境执行时有多种异常回显，麻烦帮忙排查一下，报错原因。
See post chevron_right

uncle4
Members

/opt/maca/samples/mccl_tests/perf$ sudo bash mccl.sh 8 reduce_perf 已解决 2026年5月13日 18:05

一、软硬件信息
1.服务器厂家:H3C UniServer R5300 G6
2.沐曦GPU型号：C500
3.操作系统内核版本：
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.5 LTS"
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="www.ubuntu.com/"
SUPPORT_URL="help.ubuntu.com/"
BUG_REPORT_URL="bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
distribution_version=v0.3.205
firmware_version=v0.3.132
driver_version=v0.3.165
4.是否开启CPU虚拟化：
5.mx-smi回显：
mx-smi version: 2.3.1

=================== MetaX System Management Interface Log ===================
Timestamp : Wed May 13 10:06:54 2026

Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.3.1 Kernel Mode Driver Version: 3.8.23 |
| MACA Version: 3.7.0.38 BIOS Version: 1.33.4.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C500 | 0 Off | 0000:08:00.0 | 0% Disabled |
| 36W / 350W | 36C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C500 | 1 Off | 0000:09:00.0 | 0% Disabled |
| 39W / 350W | 38C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C500 | 2 Off | 0000:0e:00.0 | 0% Disabled |
| 44W / 350W | 38C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C500 | 3 Off | 0000:11:00.0 | 0% Disabled |
| 42W / 350W | 38C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C500 | 4 Off | 0000:32:00.0 | 0% Disabled |
| 38W / 350W | 37C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C500 | 5 Off | 0000:38:00.0 | 0% Disabled |
| 38W / 350W | 37C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C500 | 6 Off | 0000:3b:00.0 | 0% Disabled |
| 41W / 350W | 39C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C500 | 7 Off | 0000:3c:00.0 | 0% Disabled |
| 41W / 350W | 38C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+

+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| no process found |
+---------------------------------------------------------------------------------+

End of Log
二、问题现象

Out of bounds values : 0 OK

Avg bus bandwidth : 28.3128

[muxi-45:06515] *** Process received signal ***
[muxi-45:06515] Signal: Segmentation fault (11)
[muxi-45:06515] Signal code: Address not mapped (1)
[muxi-45:06515] Failing at address: 0x185e0008
[muxi-45:06515] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7ff9f71ed520]
[muxi-45:06515] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0xa275d)[0x7ff9f724d75d]
[muxi-45:06515] [ 2] /lib/x86_64-linux-gnu/libc.so.6(free+0x73)[0x7ff9f7250453]
[muxi-45:06515] [ 3] /opt/maca/lib/libmccompiler.so(+0x9422b)[0x7ff9f7a9422b]
[muxi-45:06515] [ 4] /opt/maca/lib/libmccompiler.so(+0x9caca)[0x7ff9f7a9caca]
[muxi-45:06515] [ 5] /opt/maca/lib/libmccompiler.so(+0x82b7a)[0x7ff9f7a82b7a]
[muxi-45:06515] [ 6] /opt/maca/lib/libmccompiler.so(+0x873b3)[0x7ff9f7a873b3]
[muxi-45:06515] [ 7] /opt/maca/lib/libmccompiler.so(+0x7a05a)[0x7ff9f7a7a05a]
[muxi-45:06515] [ 8] /opt/maca/lib/libmccompiler.so(+0x6a9a7)[0x7ff9f7a6a9a7]
[muxi-45:06515] [ 9] /opt/maca/lib/libmccl.so(+0x22f4f2)[0x7ff9f922f4f2]
[muxi-45:06515] [10] /lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0xb6)[0x7ff9f71f0a56]
[muxi-45:06515] [11] /opt/maca/lib/libmccl.so(+0x4d483)[0x7ff9f904d483]
[muxi-45:06515] *** End of error message ***

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

mpirun noticed that process rank 3 with PID 0 on node muxi-45 exited on signal 11 (Segmentation fault).
请问这个是什么问题，为什么会出现这样的错误。

uncle4

Out of bounds values : 0 OK

Avg bus bandwidth : 28.3128