MetaX C500 8 卡 DDP 训练跨 NUMA 集体通信异常

trdyun
2026年5月10日
chat_bubble_outline 4
  • link
    Members 10 posts
    2026年5月10日 00:32
    link

    一、软硬件信息
    1.服务器厂家: 浪潮
    2.沐曦GPU型号：MXC500
    3.操作系统内核版本：Ubuntu 22.02 Linux 5.15.0-72-generic
    mx-smi回显：
    root@e5b68a9638cbo-0:/trdyun# mx-smi
    mx-smi version: 2.2.12
    =================== MetaX System Management Interface Log ===================
    Timestamp : Sun May 10 00:26:55 2026
    Attached GPUs : 8
    +---------------------------------------------------------------------------------+
    | MX-SMI 2.2.12 Kernel Mode Driver Version: 2.14.6 |
    | MACA Version: 3.5.3.20 BIOS Version: 1.24.3.0 |
    |------------------+-----------------+---------------------+----------------------|
    | Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
    | Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
    |==================+=================+=====================+======================|
    | 0 MetaX C500 | 0 Off | 0000:0f:00.0 | 0% Disabled |
    | 76W / 350W | 43C P9 | 41245/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 1 MetaX C500 | 1 Off | 0000:10:00.0 | 0% Disabled |
    | 78W / 350W | 44C P9 | 41449/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 2 MetaX C500 | 2 Off | 0000:11:00.0 | 0% Disabled |
    | 79W / 350W | 47C P9 | 2855/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 3 MetaX C500 | 3 Off | 0000:13:00.0 | 0% Disabled |
    | 77W / 350W | 44C P9 | 41129/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 4 MetaX C500 | 4 Off | 0000:88:00.0 | 0% Disabled |
    | 75W / 350W | 41C P9 | 2535/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 5 MetaX C500 | 5 Off | 0000:89:00.0 | 0% Disabled |
    | 78W / 350W | 43C P9 | 41449/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 6 MetaX C500 | 6 Off | 0000:8a:00.0 | 0% Disabled |
    | 80W / 350W | 45C P9 | 2855/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 7 MetaX C500 | 7 Off | 0000:8b:00.0 | 0% Disabled |
    | 75W / 350W | 44C P9 | 2535/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+

    +---------------------------------------------------------------------------------+
    | Process: |
    | GPU PID Process Name GPU Memory |
    | Usage(MiB) |
    |=================================================================================|
    | 0 402735 python3.12 40262 |
    | 0 402736 python3.12 36 |
    | 0 402738 python3.12 36 |
    | 0 402740 python3.12 36 |
    | 1 402736 python3.12 40582 |
    | 2 402737 python3.12 1988 |
    | 3 402738 python3.12 40262 |
    | 4 402739 python3.12 1668 |
    | 5 402740 python3.12 40582 |
    | 6 402741 python3.12 1988 |
    | 7 402742 python3.12 1668 |
    +---------------------------------------------------------------------------------+

    End of Log
    二、问题现象
    (上文省略)
    Loading weights: 100%|██████████| 393/393 [00:00<00:00, 5234.61it/s, Materializing param=roberta.encoder.layer.23.output.dense.weight]
    Loading weights: 100%|██████████| 393/393 [00:00<00:00, 6889.67it/s, Materializing param=roberta.encoder.layer.23.output.dense.weight]
    Loading weights: 100%|██████████| 393/393 [00:00<00:00, 6808.91it/s, Materializing param=roberta.encoder.layer.23.output.dense.weight]
    Loading weights: 100%|██████████| 393/393 [00:00<00:00, 6890.28it/s, Materializing param=roberta.encoder.layer.23.output.dense.weight]
    Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt
    Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt
    Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt
    Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt
    [00:22:16.852][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
    [00:22:17.108][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
    [00:22:17.108][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
    [00:22:17.108][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
    [00:22:27.092][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
    [00:22:27.348][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
    [00:22:27.348][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
    [00:22:27.348][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
    [00:22:37.328][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
    [00:22:37.588][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
    [00:22:37.588][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
    [00:22:37.588][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
    (下面日志内容同样省略)
    相关依赖版本
    torch 2.8.0+metax3.5.3.9
    torchaudio 2.4.1+metax3.5.3.9
    torchcodec 0.6.0+metax3.5.3.9
    torchvision 0.15.1+metax3.5.3.9
    transformers 5.2.0
    业务：对抗训练文本改写模型 (GAN)，Generator (mT5-XL 3.7B + LoRA) + Discriminator (XLM-RoBERTa-large)，两个模型均用 DistributedDataParallel 包装
    4 卡训练不稳定稳定，8 卡频繁卡死或崩溃。4 卡（GPU 4-7）可完整跑完多个 epoch，4卡（GPU0-3）以及8 卡时出现现象：
    DDP collective 不匹配崩溃：[rank6] RuntimeError: Detected mismatch between collectives on ranks. Rank 6 is running ALLREDUCE (backward pass), but Rank 0 is running BARRIER (_validate)
    模型加载后卡死：8 卡同时加载完 Discriminator 权重后，约 4 个 rank 停在加载完成日志处，其余 rank 无法到达 barrier，训练不启动
    MXKW 驱动层超时频发：[MXKW][E]queues.c:826: [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.，每 10 秒重复一次，GPU 计算队列创建持续失败
    已尝试的排查
    降 num_workers（8→2，减少 CUDA 上下文压力）→ 无效
    DDP 加 broadcast_buffers=False, find_unused_parameters=False → 无效
    pkill -9 清理残留进程 → 无效
    调用ms-smi -r对卡进行reset 也无效

  • link
    Members 10 posts
    2026年5月10日 00:34
    link

    相关部分关键代码如下：
    DDP 初始化（src/utils/distributed.py）

    def setup_distributed(requested_device: str = "auto") -> DistributedContext:
    if not is_distributed_env():
        device = _resolve_single_process_device(requested_device)
        return DistributedContext(
            distributed=False, rank=0, local_rank=0, world_size=1, device=device,
        )
    rank = int(os.environ["RANK"])
    local_rank = int(os.environ["LOCAL_RANK"])
    world_size = int(os.environ["WORLD_SIZE"])
    if torch.cuda.is_available() and requested_device != "cpu":
        torch.cuda.set_device(local_rank)
        device = torch.device("cuda", local_rank)
        backend = "nccl"
    else:
        device = torch.device("cpu")
        backend = "gloo"
    if not dist.is_initialized():
        dist.init_process_group(backend=backend)
    return DistributedContext(
        distributed=True, rank=rank, local_rank=local_rank,
        world_size=world_size, device=device,
    )

    训练入口 DDP 包装（scripts/train_stage2.py）

    # Barrier: all ranks finish model loading before DDP wrapping
if distributed_context.is_distributed:
    dist.barrier()

if distributed_context.is_distributed:
    from torch.nn.parallel import DistributedDataParallel
    generator = DistributedDataParallel(
        generator,
        device_ids=[distributed_context.local_rank] if device.type == "cuda" else None,
        find_unused_parameters=False,
        broadcast_buffers=False,
    )
    discriminator = DistributedDataParallel(
        discriminator,
        device_ids=[distributed_context.local_rank] if device.type == "cuda" else None,
        find_unused_parameters=False,
        broadcast_buffers=False,
    )

# Barrier: DDP construction done
if distributed_context.is_distributed:
    dist.barrier()

    训练循环中的 collective 操作（src/training/trainer_stage2.py）

    # 梯度累积步完成后
if (step + 1) % gradient_accumulation_steps == 0:
    self.scaler.step(self.gen_optimizer)  # DDP backward 触发 allreduce
    self.optimizer.zero_grad()
    global_step += 1

    # 评估（内含 barrier + all_reduce）
    if global_step % eval_steps == 0:
        eval_metrics = self._evaluate()     # barrier → generate → all_reduce

def _evaluate(self) -> dict:
    if self.distributed_context.is_distributed:
        torch.distributed.barrier()        # 所有 rank 同步进入
    # ... 遍历验证集生成 + 打分 ...
    totals = distributed_sum(totals, self.distributed_context)  # all_reduce

    崩溃日志摘要

    模型加载后 4 个 rank 卡在 barrier，其余 rank 无法到达 → 训练不启动

    Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt # ×4 条
    [MXKW][E] ioctl create queue block timeout, gpu_id:51332 type:21. Retrying. # 持续重复

    DDP collective 不匹配崩溃

    [rank6]: RuntimeError: Detected mismatch between collectives on ranks.
    Rank 6 is running collective: ALLREDUCE (backward pass)
    Rank 0 is running collective: BARRIER (_validate)

    启动命令

    CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc_per_node=8 \
  scripts/train_stage2.py \
  --config configs/training/stage2.yaml \
  --generator-config configs/model/generator.yaml \
  --discriminator-config configs/model/discriminator.yaml \
  --device cuda
  • arrow_forward

    Thread has been moved from 产品&运维.

  • link
    Members 497 posts
    2026年5月11日 11:20
    link

    尊敬的开发者您好，请裸金属执行

    dmesg -T | grep -i err
  • arrow_forward

    Thread has been moved from 解决中.

  • link
    Members 10 posts
    2026年5月26日 17:02
    link 
    root@node02:~# dmesg -T | grep -i err
[Wed Feb 11 15:12:48 2026] ACPI: Using IOAPIC for interrupt routing
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKA configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKA disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKB configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKB disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKC configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKC disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKD configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKD disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKE configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKE disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKF configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKF disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKG configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKG disabled
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKH configured for IRQ 15
[Wed Feb 11 15:12:49 2026] ACPI: PCI: Interrupt link LNKH disabled
[Wed Feb 11 15:12:49 2026] pcieport 0000:36:01.0: pciehp: Slot #16 AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise- Interlock- NoCompl- IbPresDis- LLActRep+ (with Cmd Compl erratum)
[Wed Feb 11 15:12:49 2026] ERST: Error Record Serialization Table (ERST) support is initialized.
[Wed Feb 11 15:12:49 2026] RAS: Correctable Errors collector initialized.
[Wed Feb 11 15:12:50 2026] i801_smbus 0000:00:1f.4: SMBus using PCI interrupt
[Wed Feb 11 15:12:50 2026] igb 0000:37:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed Feb 11 15:12:50 2026] igb 0000:37:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed Feb 11 15:12:50 2026] igb 0000:37:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed Feb 11 15:12:50 2026] igb 0000:37:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[Wed Feb 11 15:12:56 2026] EDAC MC0: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#0: DEV 0000:7e:0c.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC1: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#1: DEV 0000:7e:0d.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC2: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#2: DEV 0000:7e:0e.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC3: Giving out device to module i10nm_edac controller Intel_10nm Socket#0 IMC#3: DEV 0000:7e:0f.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC4: Giving out device to module i10nm_edac controller Intel_10nm Socket#1 IMC#0: DEV 0000:fe:0c.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC5: Giving out device to module i10nm_edac controller Intel_10nm Socket#1 IMC#1: DEV 0000:fe:0d.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC6: Giving out device to module i10nm_edac controller Intel_10nm Socket#1 IMC#2: DEV 0000:fe:0e.0 (INTERRUPT)
[Wed Feb 11 15:12:56 2026] EDAC MC7: Giving out device to module i10nm_edac controller Intel_10nm Socket#1 IMC#3: DEV 0000:fe:0f.0 (INTERRUPT)
[Wed Feb 11 22:19:11 2026] perf: interrupt took too long (2502 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
[Thu Feb 12 00:57:43 2026] perf: interrupt took too long (3143 > 3127), lowering kernel.perf_event_max_sample_rate to 63500
[Thu Feb 12 04:31:00 2026] perf: interrupt took too long (3939 > 3928), lowering kernel.perf_event_max_sample_rate to 50750
[Thu Feb 12 10:39:28 2026] perf: interrupt took too long (9058 > 4923), lowering kernel.perf_event_max_sample_rate to 22000
[Fri Feb 20 09:12:51 2026] hrtimer: interrupt took 14704 ns
[Tue Mar  3 21:17:10 2026] METAX.B1000.D0.MC.ERROR vram try to use (67272992 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:17:10 2026] METAX.B1000.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:17:10 2026] METAX.B1000.D0.MC.ERROR vram try to use (67242276 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:17:10 2026] METAX.B1000.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:17:10 2026] METAX.BF00.D0.MC.ERROR vram try to use (67764516 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:17:10 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:17:10 2026] METAX.BF00.D0.MC.ERROR vram try to use (67733796 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:17:10 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:20:33 2026] METAX.BF00.D0.MC.ERROR vram try to use (67132796 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:20:33 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 21:20:33 2026] METAX.BF00.D0.MC.ERROR vram try to use (67132796 kB) beyond total memory (67108864 kB), failed
[Tue Mar  3 21:20:33 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Tue Mar  3 22:51:21 2026] worker.io[1950624]: segfault at 560cf17d34b5 ip 00007fa175cbf13d sp 00007fa119ffa3e0 error 4 in _raylet.so[7fa174f06000+1a25000]
[Tue Mar  3 23:00:25 2026] traps: python3.10[2069691] general protection fault ip:7f6622857a69 sp:7ffe00ba4070 error:0 in _raylet.so[7f6621306000+1a25000]
[Tue Apr 14 10:59:43 2026] perf: interrupt took too long (15005 > 11322), lowering kernel.perf_event_max_sample_rate to 13250
[Wed Apr 15 13:14:37 2026] METAX.MC.ERROR -4 can not get user pages 0x55b3f6c68000 num_pages 0x1801 vm flags 0x8100073 gup flags 0x1
[Wed Apr 15 13:14:37 2026] METAX.MC.ERROR failed to get user pages, -4
[Wed Apr 15 13:14:37 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Wed Apr 15 13:14:37 2026] METAX.MC.ERROR -4 can not get user pages 0x55c628ea1000 num_pages 0x1801 vm flags 0x8100073 gup flags 0x1
[Wed Apr 15 13:14:37 2026] METAX.MC.ERROR failed to get user pages, -4
[Wed Apr 15 13:14:37 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 02:29:05 2026] METAX.BF00.D0.MC.ERROR vram try to use (68437304 kB) beyond total memory (67108864 kB), failed
[Fri May  8 02:29:05 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Fri May  8 04:24:36 2026] METAX.BF00.D0.MC.ERROR vram try to use (67449308 kB) beyond total memory (67108864 kB), failed
[Fri May  8 04:24:36 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Fri May  8 04:24:36 2026] METAX.BF00.D0.MC.ERROR vram try to use (67449304 kB) beyond total memory (67108864 kB), failed
[Fri May  8 04:24:36 2026] METAX.BF00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Fri May  8 10:03:36 2026] METAX.B8A00.D0.MC.ERROR vram try to use (68371796 kB) beyond total memory (67108864 kB), failed
[Fri May  8 10:03:36 2026] METAX.B8A00.D0.MC.ERROR failed to create bo on domain VRAM, -12
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7fa388fbc000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7fdcfad66000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f9d63b88000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f6318d73000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f46197f5000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:22:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f4fe57d4000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:22:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:22:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:24:16 2026] METAX.MC.ERROR -4 can not get user pages 0x7fef5e345000 num_pages 0x2000 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:24:16 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:24:16 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:25:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7fcd737bb000 num_pages 0x800 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:25:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:25:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 16:26:39 2026] METAX.MC.ERROR -4 can not get user pages 0x7f50c255b000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Fri May  8 16:26:39 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 16:26:39 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Fri May  8 17:13:53 2026] METAX.MC.ERROR -4 can not get user pages 0x7fb90cb3e000 num_pages 0x800 vm flags 0x8100073 gup flags 0x1
[Fri May  8 17:13:53 2026] METAX.MC.ERROR failed to get user pages, -4
[Fri May  8 17:13:53 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 11:34:42 2026] pt_nccl_heartbt[4127564]: segfault at 4c0 ip 00007f71c1e8bc1d sp 00007f6e237fd630 error 4 in libc.so.6[7f71c1e6f000+195000]
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f980f9d0000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f847df5b000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f7bf0d3d000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7f399e75b000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR -4 can not get user pages 0x7fd121dac000 num_pages 0x2001 vm flags 0x8100073 gup flags 0x1
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sat May  9 18:12:36 2026] METAX.MC.ERROR failed to get user pages, -4
[Sat May  9 18:12:36 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sun May 10 00:30:12 2026] METAX.MC.ERROR -4 can not get user pages 0x7f3267188000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Sun May 10 00:30:12 2026] METAX.MC.ERROR failed to get user pages, -4
[Sun May 10 00:30:12 2026] METAX.MC.ERROR -4 can not get user pages 0x7f299b9bc000 num_pages 0x1001 vm flags 0x8100073 gup flags 0x1
[Sun May 10 00:30:12 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Sun May 10 00:30:12 2026] METAX.MC.ERROR failed to get user pages, -4
[Sun May 10 00:30:12 2026] METAX.BF00.D0.MC.ERROR init_user_pages failed, -4
[Mon May 18 22:55:38 2026] python3.7[3982761]: segfault at 0 ip 0000000000000000 sp 00007ffdb6f2ccc8 error 14 in python3.7[55708cfdd000+58000]
root@node02:~#

    同样的问题发生在 VLLM 同节点跨卡部署，这是内核日志请查收

  • link
    Members 497 posts
    2026年5月26日 17:06
    link

    尊敬的开发者您好，请关闭服务器，断掉电源线，重新插拔GPU尝试