一、软硬件信息
1.服务器厂家: 浪潮
2.沐曦GPU型号：MXC500
3.操作系统内核版本：Ubuntu 22.02 Linux 5.15.0-72-generic
mx-smi回显：
root@e5b68a9638cbo-0:/trdyun# mx-smi
mx-smi version: 2.2.12
=================== MetaX System Management Interface Log ===================
Timestamp : Sun May 10 00:26:55 2026
Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.2.12 Kernel Mode Driver Version: 2.14.6 |
| MACA Version: 3.5.3.20 BIOS Version: 1.24.3.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C500 | 0 Off | 0000:0f:00.0 | 0% Disabled |
| 76W / 350W | 43C P9 | 41245/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C500 | 1 Off | 0000:10:00.0 | 0% Disabled |
| 78W / 350W | 44C P9 | 41449/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C500 | 2 Off | 0000:11:00.0 | 0% Disabled |
| 79W / 350W | 47C P9 | 2855/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C500 | 3 Off | 0000:13:00.0 | 0% Disabled |
| 77W / 350W | 44C P9 | 41129/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C500 | 4 Off | 0000:88:00.0 | 0% Disabled |
| 75W / 350W | 41C P9 | 2535/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C500 | 5 Off | 0000:89:00.0 | 0% Disabled |
| 78W / 350W | 43C P9 | 41449/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C500 | 6 Off | 0000:8a:00.0 | 0% Disabled |
| 80W / 350W | 45C P9 | 2855/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C500 | 7 Off | 0000:8b:00.0 | 0% Disabled |
| 75W / 350W | 44C P9 | 2535/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+

End of Log
二、问题现象
(上文省略)
Loading weights: 100%|██████████| 393/393 [00:00<00:00, 5234.61it/s, Materializing param=roberta.encoder.layer.23.output.dense.weight]
Loading weights: 100%|██████████| 393/393 [00:00<00:00, 6889.67it/s, Materializing param=roberta.encoder.layer.23.output.dense.weight]
Loading weights: 100%|██████████| 393/393 [00:00<00:00, 6808.91it/s, Materializing param=roberta.encoder.layer.23.output.dense.weight]
Loading weights: 100%|██████████| 393/393 [00:00<00:00, 6890.28it/s, Materializing param=roberta.encoder.layer.23.output.dense.weight]
Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt
Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt
Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt
Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt
[00:22:16.852][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:17.108][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:17.108][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:17.108][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:27.092][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:27.348][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:27.348][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:27.348][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:37.328][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:37.588][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:37.588][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
[00:22:37.588][MXKW][E]queues.c :826 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.
(下面日志内容同样省略)
相关依赖版本
torch 2.8.0+metax3.5.3.9
torchaudio 2.4.1+metax3.5.3.9
torchcodec 0.6.0+metax3.5.3.9
torchvision 0.15.1+metax3.5.3.9
transformers 5.2.0
业务：对抗训练文本改写模型 (GAN)，Generator (mT5-XL 3.7B + LoRA) + Discriminator (XLM-RoBERTa-large)，两个模型均用 DistributedDataParallel 包装
4 卡训练不稳定稳定，8 卡频繁卡死或崩溃。4 卡（GPU 4-7）可完整跑完多个 epoch，4卡（GPU0-3）以及8 卡时出现现象：
DDP collective 不匹配崩溃：[rank6] RuntimeError: Detected mismatch between collectives on ranks. Rank 6 is running ALLREDUCE (backward pass), but Rank 0 is running BARRIER (_validate)
模型加载后卡死：8 卡同时加载完 Discriminator 权重后，约 4 个 rank 停在加载完成日志处，其余 rank 无法到达 barrier，训练不启动
MXKW 驱动层超时频发：[MXKW][E]queues.c:826: [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:51332 type:21. Retrying.，每 10 秒重复一次，GPU 计算队列创建持续失败
已尝试的排查
降 num_workers（8→2，减少 CUDA 上下文压力）→ 无效
DDP 加 broadcast_buffers=False, find_unused_parameters=False → 无效
pkill -9 清理残留进程 → 无效
调用ms-smi -r对卡进行reset 也无效

相关部分关键代码如下：
DDP 初始化（src/utils/distributed.py）

def setup_distributed(requested_device: str = "auto") -> DistributedContext:
    if not is_distributed_env():
        device = _resolve_single_process_device(requested_device)
        return DistributedContext(
            distributed=False, rank=0, local_rank=0, world_size=1, device=device,
        )
    rank = int(os.environ["RANK"])
    local_rank = int(os.environ["LOCAL_RANK"])
    world_size = int(os.environ["WORLD_SIZE"])
    if torch.cuda.is_available() and requested_device != "cpu":
        torch.cuda.set_device(local_rank)
        device = torch.device("cuda", local_rank)
        backend = "nccl"
    else:
        device = torch.device("cpu")
        backend = "gloo"
    if not dist.is_initialized():
        dist.init_process_group(backend=backend)
    return DistributedContext(
        distributed=True, rank=rank, local_rank=local_rank,
        world_size=world_size, device=device,
    )

训练入口 DDP 包装（scripts/train_stage2.py）

# Barrier: all ranks finish model loading before DDP wrapping
if distributed_context.is_distributed:
    dist.barrier()

if distributed_context.is_distributed:
    from torch.nn.parallel import DistributedDataParallel
    generator = DistributedDataParallel(
        generator,
        device_ids=[distributed_context.local_rank] if device.type == "cuda" else None,
        find_unused_parameters=False,
        broadcast_buffers=False,
    )
    discriminator = DistributedDataParallel(
        discriminator,
        device_ids=[distributed_context.local_rank] if device.type == "cuda" else None,
        find_unused_parameters=False,
        broadcast_buffers=False,
    )

# Barrier: DDP construction done
if distributed_context.is_distributed:
    dist.barrier()

训练循环中的 collective 操作（src/training/trainer_stage2.py）

# 梯度累积步完成后
if (step + 1) % gradient_accumulation_steps == 0:
    self.scaler.step(self.gen_optimizer)  # DDP backward 触发 allreduce
    self.optimizer.zero_grad()
    global_step += 1

    # 评估（内含 barrier + all_reduce）
    if global_step % eval_steps == 0:
        eval_metrics = self._evaluate()     # barrier → generate → all_reduce

def _evaluate(self) -> dict:
    if self.distributed_context.is_distributed:
        torch.distributed.barrier()        # 所有 rank 同步进入
    # ... 遍历验证集生成 + 打分 ...
    totals = distributed_sum(totals, self.distributed_context)  # all_reduce

崩溃日志摘要

模型加载后 4 个 rank 卡在 barrier，其余 rank 无法到达 → 训练不启动

Discriminator loaded from checkpoint: checkpoints/discriminator/best/best.pt # ×4 条
[MXKW][E] ioctl create queue block timeout, gpu_id:51332 type:21. Retrying. # 持续重复

DDP collective 不匹配崩溃

[rank6]: RuntimeError: Detected mismatch between collectives on ranks.
Rank 6 is running collective: ALLREDUCE (backward pass)
Rank 0 is running collective: BARRIER (_validate)

启动命令

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nproc_per_node=8 \
  scripts/train_stage2.py \
  --config configs/training/stage2.yaml \
  --generator-config configs/model/generator.yaml \
  --discriminator-config configs/model/discriminator.yaml \
  --device cuda

MetaX C500 8 卡 DDP 训练跨 NUMA 集体通信异常

模型加载后 4 个 rank 卡在 barrier，其余 rank 无法到达 → 训练不启动

DDP collective 不匹配崩溃