一、软硬件信息
1.服务器厂家: 浪潮
2.沐曦GPU型号：MXC500
3.操作系统内核版本：Ubuntu 22.02 Linux 5.15.0-72-generic
4.pip list(简略):
torch 2.8.0+metax3.5.3.9
torchaudio 2.4.1+metax3.5.3.9
torchcodec 0.6.0+metax3.5.3.9
torchvision 0.15.1+metax3.5.3.9
vllm 0.19.0
vllm_metax 0.19.0+g933e92.d20260429.maca3.5.3.20.torch2.8
5.Docker 镜像版本：vllm-metax:0.19.0-maca.ai3.5.3.502-torch2.8-py312-ubuntu22.04-amd64
mx-smi回显：

root@node02:~# mx-smi
mx-smi  version: 2.2.3

=================== MetaX System Management Interface Log ===================
Timestamp                                         : Tue May 26 17:02:48 2026

Attached GPUs                                     : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.2.3                        Kernel Mode Driver Version: 2.14.6          |
| MACA Version: 2.32.0.6              BIOS Version: 1.24.3.0                      |
|------------------------------------+---------------------+----------------------+
| GPU         NAME                   | Bus-id              | GPU-Util             |
| Temp        Pwr:Usage/Cap          | Memory-Usage        |                      |
|====================================+=====================+======================|
| 0           MetaX C500             | 0000:0f:00.0        | 0%                   |
| 43C         76W / 350W             | 59952/65536 MiB     |                      |
+------------------------------------+---------------------+----------------------+
| 1           MetaX C500             | 0000:10:00.0        | 0%                   |
| 44C         78W / 350W             | 59952/65536 MiB     |                      |
+------------------------------------+---------------------+----------------------+
| 2           MetaX C500             | 0000:11:00.0        | 0%                   |
| 42C         61W / 350W             | 870/65536 MiB       |                      |
+------------------------------------+---------------------+----------------------+
| 3           MetaX C500             | 0000:13:00.0        | 0%                   |
| 40C         60W / 350W             | 870/65536 MiB       |                      |
+------------------------------------+---------------------+----------------------+
| 4           MetaX C500             | 0000:88:00.0        | 0%                   |
| 37C         58W / 350W             | 863/65536 MiB       |                      |
+------------------------------------+---------------------+----------------------+
| 5           MetaX C500             | 0000:89:00.0        | 0%                   |
| 39C         60W / 350W             | 863/65536 MiB       |                      |
+------------------------------------+---------------------+----------------------+
| 6           MetaX C500             | 0000:8a:00.0        | 0%                   |
| 40C         62W / 350W             | 863/65536 MiB       |                      |
+------------------------------------+---------------------+----------------------+
| 7           MetaX C500             | 0000:8b:00.0        | 0%                   |
| 40C         58W / 350W             | 863/65536 MiB       |                      |
+------------------------------------+---------------------+----------------------+

+---------------------------------------------------------------------------------+
| Process:                                                                        |
|  GPU                    PID         Process Name                 GPU Memory     |
|                                                                  Usage(MiB)     |
|=================================================================================|
|  0                    35623         VLLM::Worker                 2              |
|  0                   502514         VLLM::Worker_TP              59086          |
|  1                    35624         VLLM::Worker                 2              |
|  1                   502517         VLLM::Worker_TP              59086          |
|  2                    35625         VLLM::Worker                 6              |
|  3                    35626         VLLM::Worker                 6              |
+---------------------------------------------------------------------------------+

End of Log

在尝试 4 卡部署Qwen3.6-35B-A3B-W8A8时遇到跨卡通信问题：

(Worker pid=51607) INFO 05-26 16:58:38 [mccl.py:27] Found mccl from library libmccl.so
(Worker pid=51607) INFO 05-26 16:58:38 [pynccl.py:111] vLLM is using nccl==2.16.5
[16:58:49.492][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[16:58:49.492][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:61644 type:21. Retrying.
[16:58:59.728][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:61644 type:21. Retrying.
[16:58:59.728][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[16:59:09.972][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[16:59:09.972][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:61644 type:21. Retrying.
[16:59:20.212][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[16:59:20.212][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:61644 type:21. Retrying.
[16:59:30.452][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[16:59:30.452][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:61644 type:21. Retrying.
[16:59:40.688][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[16:59:40.692][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:61644 type:21. Retrying.
[16:59:50.932][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[16:59:50.932][MXKW][E]queues.c                :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:61644 type:21. Retrying.

裸金属内核日志如下：

root@node02:~# dmesg -T | grep -i err
[Wed Feb 11 15:12:48 2026] ACPI: Using IOAPIC for interrupt routing
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKA configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKA disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKB configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKB disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKC configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKC disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKD configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKD disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKE configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKE disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKF configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKF disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKG configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKG disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKH configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKH disabled
[Wed Feb 11 15:12:49 2026] pcieport 0000:36:01.0: pciehp: Slot #16 AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise- Interlock- NoCompl- IbPresDis- LLActRep+ (with Cmd Compl erratum)
[Wed Feb 11 15:12:49 2026] ERST: Error Record Serialization Table (ERST) support is initialized.
[Wed Feb 11 15:12:49 2026] RAS: Correctable Errors collector initialized.
[Wed Feb 11 15:12:50 2026] i801_smbus 0000:00:1f.4: SMBus using PCI interrupt
[Wed Feb 11 15:12:50 2026] igb 0000:37:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed Feb 11 15:12:50 2026] igb 0000:37:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed Feb 11 15:12:50 2026] igb 0000:37:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed Feb 11 15:12:50 2026] igb 0000:37:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed Feb 11 15:12:56 2026] EDAC MC0: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#0: DEV 0000:7e:0c.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC1: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#1: DEV 0000:7e:0d.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC2: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#2: DEV 0000:7e:0e.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC3: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#3: DEV 0000:7e:0f.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC4: Giving out device to module i10nm_edac controller Intel_10nm Socket#1 IMC#0: DEV 0000:fe:0c.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC5: Giving out device to module i10nm_edac controller Intel_10nm Socket#1 IMC#1: DEV 0000:fe:0d.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC6: Giving out device to module i10nm_edac controller Intel_10nm Socket#1 IMC#2: DEV 0000:fe:0e.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC7: Giving out device to module i10nm_edac controller Intel_10nm Socket#1 IMC#3: DEV 0000:fe:0f.0 (INTERRUPT)
[Wed Feb 11 22:19:11 2026] perf: interrupt took too long (2502 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
[Thu Feb 12 00:57:43 2026] perf: interrupt took too long (3143 > 3127), lowering kernel.perf_event_max_sample_rate to 63500
[Thu Feb 12 04:31:00 2026] perf: interrupt took too long (3939 > 3928), lowering kernel.perf_event_max_sample_rate to 50750
[Thu Feb 12 10:39:28 2026] perf: interrupt took too long (9058 > 4923), lowering kernel.perf_event_max_sample_rate to 22000
[Fri Feb 20 09:12:51 2026] hrtimer: interrupt took 14704 ns
[Tue Mar  3 21:17:10 2026] METAX.B1000.D0.MC.ERROR vram try to use (67272992 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:17:10 2026] METAX.B1000.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:17:10 2026] METAX.B1000.D0.MC.ERROR vram try to use (67242276 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:17:10 2026] METAX.B1000.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:17:10 2026] METAX.BF00.D0.MC.ERROR vram try to use (67764516 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:17:10 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:17:10 2026] METAX.BF00.D0.MC.ERROR vram try to use (67733796 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:17:10 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:20:33 2026] METAX.BF00.D0.MC.ERROR vram try to use (67132796 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:20:33 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:20:33 2026] METAX.BF00.D0.MC.ERROR vram try to use (67132796 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:20:33 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 22:51:21 2026] worker.io[1950624]: segfault at 560cf17d34b5 ip 00007fa175cbf13d sp 00007fa119ffa3e0 error 4 in _raylet.so[7fa174f06000+1a25000]
[Tue Mar  3 23:00:25 2026] traps: python3.10[2069691] general protection fault ip:7f6622857a69 sp:7ffe00ba4070 error:0 in _raylet.so[7f6621306000+1a25000]
[Tue Apr 14 10:59:43 2026] perf: interrupt took too long (15005 > 11322), lowering kernel.perf_event_max_sample_rate to 13250
[Wed Apr 15 13:14:37 2026] METAX.MC.ERROR -4 can not get user pages 0x55b3f6c68000 num_pages 0x1801 vm flags 0x8100073 gup flags 0x1
[Wed Apr 15 13:14:37 2026] METAX.MC.ERROR failed to get user pages, -4
[Wed Apr 15 13:14:37 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Wed Apr 15 13:14:37 2026] METAX.MC.ERROR -4 can not get user pages 0x55c628ea1000 num_pages 0x1801 vm flags 0x8100073 gup flags 0x1
[Wed Apr 15 13:14:37 2026] METAX.MC.ERROR failed to get user pages, -4
[Wed Apr 15 13:14:37 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 02:29:05 2026] METAX.BF00.D0.MC.ERROR vram try to use (68437304 kB) beyond total memory (67108864 kB), failed
[Fri May  8 02:29:05 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Fri May  8 04:24:36 2026] METAX.BF00.D0.MC.ERROR vram try to use (67449308 kB) beyond total memory (67108864 kB), failed
[Fri May  8 04:24:36 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Fri May  8 04:24:36 2026] METAX.BF00.D0.MC.ERROR vram try to use (67449304 kB) beyond total memory (67108864 kB), failed
[Fri May  8 04:24:36 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Fri May  8 10:03:36 2026] METAX.B8A00.D0.MC.ERROR vram try to use (68371796 kB) beyond total memory (67108864 kB), failed
[Fri May  8 10:03:36 2026] METAX.B8A00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7fa388fbc000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7fdcfad66000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f9d63b88000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f6318d73000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f46197f5000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f4fe57d4000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:24:16 2026] METAX.MC.ERROR -4 can not get user pages 0x7fef5e345000 num_pages 0x2000 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:24:16 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:24:16 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:25:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7fcd737bb000 num_pages 0x800 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:25:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:25:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:26:39 2026] METAX.MC.ERROR -4 can not get user pages 0x7f50c255b000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:26:39 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:26:39 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 17:13:53 2026] METAX.MC.ERROR -4 can not get user pages 0x7fb90cb3e000 num_pages 0x800 vm flags 0x8100073 gup flags 0x1
[Fri May  8 17:13:53 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 17:13:53 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 11:34:42 2026] pt_nccl_heartbt[4127564]: segfault at 4c0 ip 00007f71c1e8bc1d sp 00007f6e237fd630 error 4 in libc.so.6[7f71c1e6f000+195000]
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f980f9d0000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f847df5b000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f7bf0d3d000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f399e75b000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7fd121dac000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sun May 10 00:30:12 2026] METAX.MC.ERROR -4 can not get user pages 0x7f3267188000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Sun May 10 00:30:12 2026] METAX.MC.ERROR failed to get user pages, -4
[Sun May 10 00:30:12 2026] METAX.MC.ERROR -4 can not get user pages 0x7f299b9bc000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Sun May 10 00:30:12 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sun May 10 00:30:12 2026] METAX.MC.ERROR failed to get user pages, -4
[Sun May 10 00:30:12 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Mon May 18 22:55:38 2026] python3.7[3982761]: segfault at 0 ip 0000000000000000 sp 00007ffdb6f2ccc8 error 14 in python3.7[55708cfdd000+58000]
root@node02:~#

root@node02:~# dmesg -T | grep -i err
[Wed Feb 11 15:12:48 2026] ACPI: Using IOAPIC for interrupt routing
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKA configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKA disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKB configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKB disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKC configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKC disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKD configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKD disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKE configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKE disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKF configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKF disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKG configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKG disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKH configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKH disabled
[Wed Feb 11 15:12:49 2026] pcieport 0000:36:01.0: pciehp: Slot #16 AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise- Interlock- NoCompl- IbPresDis- LLActRep+ (with Cmd Compl erratum)
[Wed Feb 11 15:12:49 2026] ERST: Error Record Serialization Table (ERST) support is initialized.
[Wed Feb 11 15:12:49 2026] RAS: Correctable Errors collector initialized.
[Wed Feb 11 15:12:50 2026] i801_smbus 0000:00:1f.4: SMBus using PCI interrupt
[Wed Feb 11 15:12:50 2026] igb 0000:37:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed Feb 11 15:12:50 2026] igb 0000:37:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed Feb 11 15:12:50 2026] igb 0000:37:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed Feb 11 15:12:50 2026] igb 0000:37:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed Feb 11 15:12:56 2026] EDAC MC0: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#0: DEV 0000:7e:0c.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC1: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#1: DEV 0000:7e:0d.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC2: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#2: DEV 0000:7e:0e.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC3: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#3: DEV 0000:7e:0f.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC4: Giving out device to module i10nm_edac controller Intel_10nm Socket#1 IMC#0: DEV 0000:fe:0c.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC5: Giving out device to module i10nm_edac controller Intel_10nm Socket#1 IMC#1: DEV 0000:fe:0d.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC6: Giving out device to module i10nm_edac controller Intel_10nm Socket#1 IMC#2: DEV 0000:fe:0e.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC7: Giving out device to module i10nm_edac controller Intel_10nm Socket#1 IMC#3: DEV 0000:fe:0f.0 (INTERRUPT)
[Wed Feb 11 22:19:11 2026] perf: interrupt took too long (2502 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
[Thu Feb 12 00:57:43 2026] perf: interrupt took too long (3143 > 3127), lowering kernel.perf_event_max_sample_rate to 63500
[Thu Feb 12 04:31:00 2026] perf: interrupt took too long (3939 > 3928), lowering kernel.perf_event_max_sample_rate to 50750
[Thu Feb 12 10:39:28 2026] perf: interrupt took too long (9058 > 4923), lowering kernel.perf_event_max_sample_rate to 22000
[Fri Feb 20 09:12:51 2026] hrtimer: interrupt took 14704 ns
[Tue Mar  3 21:17:10 2026] METAX.B1000.D0.MC.ERROR vram try to use (67272992 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:17:10 2026] METAX.B1000.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:17:10 2026] METAX.B1000.D0.MC.ERROR vram try to use (67242276 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:17:10 2026] METAX.B1000.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:17:10 2026] METAX.BF00.D0.MC.ERROR vram try to use (67764516 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:17:10 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:17:10 2026] METAX.BF00.D0.MC.ERROR vram try to use (67733796 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:17:10 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:20:33 2026] METAX.BF00.D0.MC.ERROR vram try to use (67132796 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:20:33 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:20:33 2026] METAX.BF00.D0.MC.ERROR vram try to use (67132796 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:20:33 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 22:51:21 2026] worker.io[1950624]: segfault at 560cf17d34b5 ip 00007fa175cbf13d sp 00007fa119ffa3e0 error 4 in _raylet.so[7fa174f06000+1a25000]
[Tue Mar  3 23:00:25 2026] traps: python3.10[2069691] general protection fault ip:7f6622857a69 sp:7ffe00ba4070 error:0 in _raylet.so[7f6621306000+1a25000]
[Tue Apr 14 10:59:43 2026] perf: interrupt took too long (15005 > 11322), lowering kernel.perf_event_max_sample_rate to 13250
[Wed Apr 15 13:14:37 2026] METAX.MC.ERROR -4 can not get user pages 0x55b3f6c68000 num_pages 0x1801 vm flags 0x8100073 gup flags 0x1
[Wed Apr 15 13:14:37 2026] METAX.MC.ERROR failed to get user pages, -4
[Wed Apr 15 13:14:37 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Wed Apr 15 13:14:37 2026] METAX.MC.ERROR -4 can not get user pages 0x55c628ea1000 num_pages 0x1801 vm flags 0x8100073 gup flags 0x1
[Wed Apr 15 13:14:37 2026] METAX.MC.ERROR failed to get user pages, -4
[Wed Apr 15 13:14:37 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 02:29:05 2026] METAX.BF00.D0.MC.ERROR vram try to use (68437304 kB) beyond total memory (67108864 kB), failed
[Fri May  8 02:29:05 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Fri May  8 04:24:36 2026] METAX.BF00.D0.MC.ERROR vram try to use (67449308 kB) beyond total memory (67108864 kB), failed
[Fri May  8 04:24:36 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Fri May  8 04:24:36 2026] METAX.BF00.D0.MC.ERROR vram try to use (67449304 kB) beyond total memory (67108864 kB), failed
[Fri May  8 04:24:36 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Fri May  8 10:03:36 2026] METAX.B8A00.D0.MC.ERROR vram try to use (68371796 kB) beyond total memory (67108864 kB), failed
[Fri May  8 10:03:36 2026] METAX.B8A00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7fa388fbc000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7fdcfad66000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f9d63b88000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f6318d73000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f46197f5000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f4fe57d4000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:24:16 2026] METAX.MC.ERROR -4 can not get user pages 0x7fef5e345000 num_pages 0x2000 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:24:16 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:24:16 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:25:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7fcd737bb000 num_pages 0x800 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:25:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:25:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:26:39 2026] METAX.MC.ERROR -4 can not get user pages 0x7f50c255b000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:26:39 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:26:39 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 17:13:53 2026] METAX.MC.ERROR -4 can not get user pages 0x7fb90cb3e000 num_pages 0x800 vm flags 0x8100073 gup flags 0x1
[Fri May  8 17:13:53 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 17:13:53 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 11:34:42 2026] pt_nccl_heartbt[4127564]: segfault at 4c0 ip 00007f71c1e8bc1d sp 00007f6e237fd630 error 4 in libc.so.6[7f71c1e6f000+195000]
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f980f9d0000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f847df5b000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f7bf0d3d000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f399e75b000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7fd121dac000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sun May 10 00:30:12 2026] METAX.MC.ERROR -4 can not get user pages 0x7f3267188000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Sun May 10 00:30:12 2026] METAX.MC.ERROR failed to get user pages, -4
[Sun May 10 00:30:12 2026] METAX.MC.ERROR -4 can not get user pages 0x7f299b9bc000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Sun May 10 00:30:12 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sun May 10 00:30:12 2026] METAX.MC.ERROR failed to get user pages, -4
[Sun May 10 00:30:12 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Mon May 18 22:55:38 2026] python3.7[3982761]: segfault at 0 ip 0000000000000000 sp 00007ffdb6f2ccc8 error 14 in python3.7[55708cfdd000+58000]
root@node02:~#

同样的问题发生在 VLLM 同节点跨卡部署，这是内核日志请查收

如题，SGLang 官方文档部署要求 SGLang版本>=0.5.10。官方是否有对 SGLang0.5.10 做适配？

相关部分关键代码如下：
DDP 初始化（src/utils/distributed.py）

def setup_distributed(requested_device: str = "auto") -> DistributedContext:
    if not is_distributed_env():
        device = _resolve_single_process_device(requested_device)
        return DistributedContext(
            distributed=False, rank=0, local_rank=0, world_size=1, device=device,
        )
    rank = int(os.environ["RANK"])
    local_rank = int(os.environ["LOCAL_RANK"])
    world_size = int(os.environ["WORLD_SIZE"])
    if torch.cuda.is_available() and requested_device != "cpu":
        torch.cuda.set_device(local_rank)
        device = torch.device("cuda", local_rank)
        backend = "nccl"
    else:
        device = torch.device("cpu")
        backend = "gloo"
    if not dist.is_initialized():
        dist.init_process_group(backend=backend)
    return DistributedContext(
        distributed=True, rank=rank, local_rank=local_rank,
        world_size=world_size, device=device,
    )

训练入口 DDP 包装（scripts/train_stage2.py）

# Barrier: all ranks finish model loading before DDP wrapping
if distributed_context.is_distributed:
    dist.barrier()

if distributed_context.is_distributed:
    from torch.nn.parallel import DistributedDataParallel
    generator = DistributedDataParallel(
        generator,
        device_ids=[distributed_context.local_rank] if device.type == "cuda" else None,
        find_unused_parameters=False,
        broadcast_buffers=False,
    )
    discriminator = DistributedDataParallel(
        discriminator,
        device_ids=[distributed_context.local_rank] if device.type == "cuda" else None,
        find_unused_parameters=False,
        broadcast_buffers=False,
    )

# Barrier: DDP construction done
if distributed_context.is_distributed:
    dist.barrier()

训练循环中的 collective 操作（src/training/trainer_stage2.py）

# 梯度累积步完成后
if (step + 1) % gradient_accumulation_steps == 0:
    self.scaler.step(self.gen_optimizer)  # DDP backward 触发 allreduce
    self.optimizer.zero_grad()
    global_step += 1

    # 评估（内含 barrier + all_reduce）
    if global_step % eval_steps == 0:
        eval_metrics = self._evaluate()     # barrier → generate → all_reduce

def _evaluate(self) -> dict:
    if self.distributed_context.is_distributed:
        torch.distributed.barrier()        # 所有 rank 同步进入
    # ... 遍历验证集生成 + 打分 ...
    totals = distributed_sum(totals, self.distributed_context)  # all_reduce

崩溃日志摘要

模型加载后 4 个 rank 卡在 barrier，其余 rank 无法到达 → 训练不启动

Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt # ×4 条
[MXKW][E] ioctl create queue block timeout, gpu_id:51332 type:21. Retrying. # 持续重复

DDP collective 不匹配崩溃

[rank6]: RuntimeError: Detected mismatch between collectives on ranks.
Rank 6 is running collective: ALLREDUCE (backward pass)
Rank 0 is running collective: BARRIER (_validate)

启动命令

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc_per_node=8 \
  scripts/train_stage2.py \
  --config configs/training/stage2.yaml \
  --generator-config configs/model/generator.yaml \
  --discriminator-config configs/model/discriminator.yaml \
  --device cuda

一、软硬件信息
1.服务器厂家: 浪潮
2.沐曦GPU型号：MXC500
3.操作系统内核版本：Ubuntu 22.02 Linux 5.15.0-72-generic
mx-smi回显：
root@e5b68a9638cbo-0:/trdyun# mx-smi
mx-smi version: 2.2.12
=================== MetaX System Management Interface Log ===================
Timestamp : Sun May 10 00:26:55 2026
Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.2.12 Kernel Mode Driver Version: 2.14.6 |
| MACA Version: 3.5.3.20 BIOS Version: 1.24.3.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C500 | 0 Off | 0000:0f:00.0 | 0% Disabled |
| 76W / 350W | 43C P9 | 41245/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C500 | 1 Off | 0000:10:00.0 | 0% Disabled |
| 78W / 350W | 44C P9 | 41449/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C500 | 2 Off | 0000:11:00.0 | 0% Disabled |
| 79W / 350W | 47C P9 | 2855/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C500 | 3 Off | 0000:13:00.0 | 0% Disabled |
| 77W / 350W | 44C P9 | 41129/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C500 | 4 Off | 0000:88:00.0 | 0% Disabled |
| 75W / 350W | 41C P9 | 2535/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C500 | 5 Off | 0000:89:00.0 | 0% Disabled |
| 78W / 350W | 43C P9 | 41449/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C500 | 6 Off | 0000:8a:00.0 | 0% Disabled |
| 80W / 350W | 45C P9 | 2855/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C500 | 7 Off | 0000:8b:00.0 | 0% Disabled |
| 75W / 350W | 44C P9 | 2535/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+

+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| 0 402735 python3.12 40262 |
| 0 402736 python3.12 36 |
| 0 402738 python3.12 36 |
| 0 402740 python3.12 36 |
| 1 402736 python3.12 40582 |
| 2 402737 python3.12 1988 |
| 3 402738 python3.12 40262 |
| 4 402739 python3.12 1668 |
| 5 402740 python3.12 40582 |
| 6 402741 python3.12 1988 |
| 7 402742 python3.12 1668 |
+---------------------------------------------------------------------------------+

End of Log
二、问题现象
(上文省略)
Loading weights: 100%|██████████| 393/393 [00:00<00:00, 5234.61it/s, Materializing param=roberta.encoder.layer.23.output.dense.weight]
Loading weights: 100%|██████████| 393/393 [00:00<00:00, 6889.67it/s, Materializing param=roberta.encoder.layer.23.output.dense.weight]
Loading weights: 100%|██████████| 393/393 [00:00<00:00, 6808.91it/s, Materializing param=roberta.encoder.layer.23.output.dense.weight]
Loading weights: 100%|██████████| 393/393 [00:00<00:00, 6890.28it/s, Materializing param=roberta.encoder.layer.23.output.dense.weight]
Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt
Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt
Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt
Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt
[00:22:16.852][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:17.108][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:17.108][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:17.108][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:27.092][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:27.348][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:27.348][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:27.348][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:37.328][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:37.588][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:37.588][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:37.588][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
(下面日志内容同样省略)
相关依赖版本
torch 2.8.0+metax3.5.3.9
torchaudio 2.4.1+metax3.5.3.9
torchcodec 0.6.0+metax3.5.3.9
torchvision 0.15.1+metax3.5.3.9
transformers 5.2.0
业务：对抗训练文本改写模型 (GAN)，Generator (mT5-XL 3.7B + LoRA) + Discriminator (XLM-RoBERTa-large)，两个模型均用 DistributedDataParallel 包装
4 卡训练不稳定稳定，8 卡频繁卡死或崩溃。4 卡（GPU 4-7）可完整跑完多个 epoch，4卡（GPU0-3）以及8 卡时出现现象：
DDP collective 不匹配崩溃：[rank6] RuntimeError: Detected mismatch between collectives on ranks. Rank 6 is running ALLREDUCE (backward pass), but Rank 0 is running BARRIER (_validate)
模型加载后卡死：8 卡同时加载完 Discriminator 权重后，约 4 个 rank 停在加载完成日志处，其余 rank 无法到达 barrier，训练不启动
MXKW 驱动层超时频发：[MXKW][E]queues.c:826: [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.，每 10 秒重复一次，GPU 计算队列创建持续失败
已尝试的排查
降 num_workers（8→2，减少 CUDA 上下文压力）→ 无效
DDP 加 broadcast_buffers=False, find_unused_parameters=False → 无效
pkill -9 清理残留进程 → 无效
调用ms-smi -r对卡进行reset 也无效

我正在使用的就是w8a8的权重，没有额外的环境变量，你可以尝试一下

可以参考我的启动脚本，可以正常运行

镜像:vllm0.15.0_transformer5.2.0_py310
我们在 C500 上跑 MiMo-V2-Flash-W8A8（modelscope上沐曦上传的权重），在走 unified_attention 算子时触发 We only support head_dim 64 for S extra。
以下是启动脚本
vllm serve /mnt/model \
--host 0.0.0.0 \
--port 8080 \
--served-model-name mimo_v2_flash \
--tensor-parallel-size 8 \
--max-model-len 8192 \
--trust-remote-code \
--enforce-eager \
--gpu-memory-utilization 0.90

看到Meta官方做了GLM5.1的Day0适配，请问一下w8a8的量化权重文件有公开吗？

如题，报错如下
root@4n8mh6oeu78t9-0:/mnt/model# ./start.sh
INFO 04-06 18:49:28 [init.py:43] Available plugins for group vllm.platform_plugins:
INFO 04-06 18:49:28 [init.py:45] - metax -> vllm_metax:register
INFO 04-06 18:49:28 [init.py:48] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 04-06 18:49:28 [init.py:217] Platform plugin metax is activated
INFO 04-06 18:49:28 [envs.py:83] Plugin sets VLLM_USE_FLASHINFER_SAMPLER to False. Reason: flashinfer sampler are not supported on maca
INFO 04-06 18:49:28 [envs.py:83] Plugin sets VLLM_ENGINE_READY_TIMEOUT_S to 3600. Reason: set timeout to 3600s for model loading
INFO Print the version information of mcoplib during compilation.

Version info:Mcoplib_Version = '0.4.0'
Build_Maca_Version = '3.5.3.18'
GIT_BRANCH = 'HEAD'
GIT_COMMIT = 'fe3a7e2'
Vllm Op Version = 0.15.0
SGlang Op Version = 0.5.7 && 0.5.8

INFO Staring Check the current MACA version of the operating environment.

INFO: Release major.minor matching, successful:3.5.

WARNING 04-06 18:49:32 [init.py:86] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq.MacaAWQConfig'>.
WARNING 04-06 18:49:32 [init.py:86] The quantization method 'awq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq_marlin.MacaAWQMarlinConfig'>.
WARNING 04-06 18:49:32 [init.py:86] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.compressed_tensors.MacaCompressedTensorsConfig'>.
WARNING 04-06 18:49:32 [init.py:86] The quantization method 'gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq.MacaGPTQConfig'>.
WARNING 04-06 18:49:32 [init.py:86] The quantization method 'gptq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq_marlin.MacaGPTQMarlinConfig'>.
WARNING 04-06 18:49:32 [init.py:86] The quantization method 'moe_wna16' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.moe_wna16.MacaMoeWNA16Config'>.
WARNING 04-06 18:49:36 [registry.py:812] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_mtp:DeepSeekMTP.
WARNING 04-06 18:49:36 [registry.py:812] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV2ForCausalLM.
WARNING 04-06 18:49:36 [registry.py:812] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 04-06 18:49:36 [registry.py:812] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 04-06 18:49:36 [registry.py:812] Model architecture KimiK25ForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.kimi_k25:KimiK25ForConditionalGeneration.
(APIServer pid=317) INFO 04-06 18:49:36 [utils.py:325]
(APIServer pid=317) INFO 04-06 18:49:36 [utils.py:325] █ █ █▄ ▄█
(APIServer pid=317) INFO 04-06 18:49:36 [utils.py:325] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.15.0
(APIServer pid=317) INFO 04-06 18:49:36 [utils.py:325] █▄█▀ █ █ █ █ model /mnt/model
(APIServer pid=317) INFO 04-06 18:49:36 [utils.py:325] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=317) INFO 04-06 18:49:36 [utils.py:325]
(APIServer pid=317) INFO 04-06 18:49:36 [utils.py:261] non-default args: {'model_tag': '/mnt/model', 'api_server_count': 1, 'host': '0.0.0.0', 'port': 7878, 'enable_auto_tool_choice': True, 'tool_call_parser': 'minimax_m2', 'model': '/mnt/model', 'trust_remote_code': True, 'served_model_name': ['MiniMax-M2.5-196k'], 'reasoning_parser': 'minimax_m2_append_think', 'tensor_parallel_size': 8, 'enable_expert_parallel': True, 'max_num_seqs': 8, 'compilation_config': {'level': None, 'mode': None, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'splitting_ops': None, 'compile_mm_encoder': False, 'compile_sizes': None, 'compile_ranges_split_points': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.PIECEWISE: 1>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': None, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None}}
(APIServer pid=317) The argument trust_remote_code is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=317) The argument trust_remote_code is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=317) INFO 04-06 18:49:36 [model.py:541] Resolved architecture: MiniMaxM2ForCausalLM
(APIServer pid=317) ERROR 04-06 18:49:36 [repo_utils.py:47] Error retrieving safetensors: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/mnt/model'. Use repo_type argument if needed., retrying 1 of 2
(APIServer pid=317) ERROR 04-06 18:49:38 [repo_utils.py:45] Error retrieving safetensors: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/mnt/model'. Use repo_type argument if needed.
(APIServer pid=317) INFO 04-06 18:49:38 [model.py:1882] Downcasting torch.float32 to torch.bfloat16.
(APIServer pid=317) INFO 04-06 18:49:38 [model.py:1561] Using max model len 196608
(APIServer pid=317) Traceback (most recent call last):
(APIServer pid=317) File "/opt/conda/bin/vllm", line 8, in <module>
(APIServer pid=317) sys.exit(main())
(APIServer pid=317) File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=317) args.dispatch_function(args)
(APIServer pid=317) File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/cli/serve.py", line 111, in cmd
(APIServer pid=317) uvloop.run(run_server(args))
(APIServer pid=317) File "/opt/conda/lib/python3.10/site-packages/uvloop/init.py", line 69, in run
(APIServer pid=317) return loop.run_until_complete(wrapper())
(APIServer pid=317) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=317) File "/opt/conda/lib/python3.10/site-packages/uvloop/init.py", line 48, in wrapper
(APIServer pid=317) return await main
(APIServer pid=317) File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 919, in run_server
(APIServer pid=317) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=317) File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 938, in run_server_worker
(APIServer pid=317) async with build_async_engine_client(
(APIServer pid=317) File "/opt/conda/lib/python3.10/contextlib.py", line 199, in aenter
(APIServer pid=317) return await anext(self.gen)
(APIServer pid=317) File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 147, in build_async_engine_client
(APIServer pid=317) async with build_async_engine_client_from_engine_args(
(APIServer pid=317) File "/opt/conda/lib/python3.10/contextlib.py", line 199, in aenter
(APIServer pid=317) return await anext(self.gen)
(APIServer pid=317) File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 173, in build_async_engine_client_from_engine_args
(APIServer pid=317) vllm_config = engine_args.create_engine_config(usage_context=usage_context)
(APIServer pid=317) File "/opt/conda/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 1374, in create_engine_config
(APIServer pid=317) model_config = self.create_model_config()
(APIServer pid=317) File "/opt/conda/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 1228, in create_model_config
(APIServer pid=317) return ModelConfig(
(APIServer pid=317) File "/opt/conda/lib/python3.10/site-packages/pydantic/_internal/_dataclasses.py", line 121, in init
(APIServer pid=317) s.pydantic_validator.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
(APIServer pid=317) pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
(APIServer pid=317) Value error, fp8 quantization is currently not supported in maca. [type=value_error, input_value=ArgsKwargs((), {'model': ...rocessor_plugin': None}), input_type=ArgsKwargs]
(APIServer pid=317) For further information visit errors.pydantic.dev/2.12/v/value_error