一、软硬件信息
1.服务器厂家:国鑫
2.沐曦GPU型号：C500
3.操作系统内核版本：5.15.0-112-generic
4.是否开启CPU虚拟化：Virtualization features:
Virtualization: AMD-V
5.mx-smi回显：mx-smi version: 2.3.1
=================== MetaX System Management Interface Log ===================
Timestamp : Wed May 27 08:27:52 2026
Attached GPUs : 7
+---------------------------------------------------------------------------------+
| MX-SMI 2.3.1 Kernel Mode Driver Version: 3.8.1 |
| MACA Version: 3.7.1.5 BIOS Version: 1.33.4.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C500 | 0 Off | 0000:06:00.0 | 0% Disabled |
| 40W / 350W | 50C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C500 | 1 Off | 0000:07:00.0 | 0% Disabled |
| 32W / 350W | 48C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C500 | 2 Off | 0000:15:00.0 | 0% Disabled |
| 41W / 350W | 49C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C500 | 3 Off | 0000:17:00.0 | 0% Disabled |
| 37W / 350W | 48C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C500 | 4 Off | 0000:65:00.0 | 0% Disabled |
| 34W / 350W | 48C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C500 | 5 Off | 0000:66:00.0 | 0% Disabled |
| 37W / 350W | 49C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C500 | 6 Off | 0000:77:00.0 | 0% Disabled |
| 38W / 350W | 51C P0 | 858/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| no process found |
+---------------------------------------------------------------------------------+
二、问题现象
请描述详细的问题现象日志。若日志过长，请上传附件(txt格式）。
有一张卡报不可用mx-smi -L 显示 GPU#7 MXC500 0000:76:00.0 Not Available(Layout Node Not Found)，重新安装驱动和更新固件也没有用，还得麻烦帮忙看下
dmesg日志
[Wed May 27 08:16:41 2026] ERST: Error Record Serialization Table (ERST) support is initialized.
[Wed May 27 08:16:42 2026] RAS: Correctable Errors collector initialized.
[Wed May 27 08:16:43 2026] igb 0000:51:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed May 27 08:16:43 2026] igb 0000:51:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed May 27 08:17:27 2026] METAX.B7600.D0.SMP.ERROR EID (0000:76:00.0): 0x2106, wait SMP1 timeout
[Wed May 27 08:17:27 2026] METAX.B7600.D0.SMP.ERROR SMP1 start failed, enter mini mode, fw:1.31.1.0
[Wed May 27 08:17:32 2026] METAX.B7600.D0.SMP.ERROR SMP1: mxc500_mailbox_message_c2p_send_recv, 5000ms timeout! resp_val -2
[Wed May 27 08:17:32 2026] METAX.B7600.D0.SMP.ERROR SMP1 is inactive.
[Wed May 27 08:17:32 2026] MXCD.TOPO.ERROR <2211> get hbmecc error, -62
[Wed May 27 08:17:32 2026] MXCD.TOPO.ERROR <2234> get vbios version error, -22
[Wed May 27 08:19:11 2026] METAX.B7600.D0.SMI.ERROR vbios and driver are not compatible, <4138>, ret:-5
[Wed May 27 08:19:17 2026] METAX.B7600.D0.SMI.ERROR vbios and driver are not compatible, <4138>, ret:-5
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 516
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: It has been corrected by h/w and requires no further action
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: event severity: corrected
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: Error 0, type: corrected
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: fru_text: PcieError
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: section_type: PCIe error
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: port_type: 6, downstream switch port
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: version: 0.2
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: command: 0x0407, status: 0x0010
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: device_id: 0000:72:03.0
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: slot: 52
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: secondary_bus: 0x76
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: vendor_id: 0x1000, device_id: 0xc030
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: class_code: 060400
[Wed May 27 08:20:09 2026] {1}[Hardware Error]: bridge: secondary_status: 0x0000, control: 0x0002
[Wed May 27 08:20:09 2026] pcieport 0000:72:03.0: [ 0] RxErr
[Wed May 27 08:20:35 2026] METAX.B7600.D0.SMP.ERROR EID (0000:76:00.0): 0x2106, wait SMP0 timeout
[Wed May 27 08:20:35 2026] METAX.B7600.D0.SMP.ERROR xcore_chip smp0 sync failed, fw:1.33.4.0, ret:-62
[Wed May 27 08:20:35 2026] METAX.B7600.D0.BASE.ERROR sync with smp failed, -62
[Wed May 27 08:20:35 2026] METAX.BASE.ERROR GPU init error
[Wed May 27 08:20:35 2026] [drm:metax_pci_probe [metax]] ERROR metax load kms failed, -62
[Wed May 27 08:20:35 2026] METAX: probe of 0000:76:00.0 failed with error -62