MetaX-Tech Developer Forum 论坛首页
  • 沐曦开发者
search
Sign in

forwardwu

  • Members
  • Joined 2026年10月28日
  • message 帖子
  • forum 主题
  • favorite 关注者
  • favorite_border Follows
  • person_outline 详细信息

forwardwu has posted 2 messages.

  • See post chevron_right
    forwardwu
    Members
    MetaX C500 训练create queue block timeout 报错 已解决 2026年10月28日 17:13

    好的,感谢,我联系一下问问

  • See post chevron_right
    forwardwu
    Members
    MetaX C500 训练create queue block timeout 报错 已解决 2026年10月28日 14:31

    一、软硬件信息
    1. 服务器厂家:新华三
    2. 沐曦GPU型号:MetaX C500X
    3. 操作系统内核版本:5.15.0-119-generic
    4. 是否开启CPU虚拟化:否
    5. mx-smi回显:
    mx-smi version: 2.2.9

    =================== MetaX System Management Interface Log ===================
    Timestamp : Wed Oct 28 13:53:26 2026

    Attached GPUs : 8
    +---------------------------------------------------------------------------------+
    | MX-SMI 2.2.9 Kernel Mode Driver Version: 3.4.4 |
    | MACA Version: 3.3.0.15 BIOS Version: 1.25.1.0 |
    |------------------+-----------------+---------------------+----------------------|
    | Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
    | Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
    |==================+=================+=====================+======================|
    | 0 MetaX C500X | 0 Off | 0000:08:00.0 | 31% Disabled |
    | 91W / 350W | 40C P9 | 9642/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 1 MetaX C500X | 1 Off | 0000:09:00.0 | 0% Disabled |
    | 81W / 350W | 38C P9 | 10152/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 2 MetaX C500X | 2 Off | 0000:0e:00.0 | 31% Disabled |
    | 91W / 350W | 39C P9 | 10152/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 3 MetaX C500X | 3 Off | 0000:11:00.0 | 31% Disabled |
    | 88W / 350W | 39C P9 | 10152/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 4 MetaX C500X | 4 Off | 0000:32:00.0 | 0% Disabled |
    | 85W / 350W | 38C P9 | 61257/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 5 MetaX C500X | 5 Off | 0000:38:00.0 | 0% Disabled |
    | 68W / 350W | 38C P0 | 874/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 6 MetaX C500X | 6 Off | 0000:3b:00.0 | 31% Disabled |
    | 90W / 350W | 39C P9 | 9512/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 7 MetaX C500X | 7 Off | 0000:3c:00.0 | 0% Disabled |
    | 82W / 350W | 39C P9 | 61257/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+

    +---------------------------------------------------------------------------------+
    | Process: |
    | GPU PID Process Name GPU Memory |
    | Usage(MiB) |
    |=================================================================================|
    | 0 2535659 python3.12 8644 |
    | 0 2535661 python3.12 38 |
    | 0 2535662 python3.12 38 |
    | 0 2535663 python3.12 38 |
    | 1 2535660 python3.12 9270 |
    | 1 2535661 python3.12 0 |
    | 1 2535662 python3.12 0 |
    | 1 2535663 python3.12 0 |
    | 2 2535661 python3.12 9270 |
    | 2 2535662 python3.12 0 |
    | 2 2535663 python3.12 0 |
    | 3 2535661 python3.12 0 |
    | 3 2535662 python3.12 9270 |
    | 3 2535663 python3.12 0 |
    | 4 1945445 VLLM::EngineCor 60382 |
    | 6 2535661 python3.12 0 |
    | 6 2535662 python3.12 0 |
    | 6 2535663 python3.12 8630 |
    | 7 2555249 VLLM::EngineCor 60382 |
    +---------------------------------------------------------------------------------+

    1. docker info回显:
    2. 镜像版本:msswift-4.4.3-maca.ai3.8.0.109-torch2.10-py312-ubuntu22.04-amd64
    3. 启动容器命令:
    4. 容器内执行命令:python main.py

    [2026-10-28T14:01:31+08:00] [INFO] cmd: ['swift', 'sft', '--model', '/mnt/xxx/xxx/Qwen3.6-27B', '--output_dir', '/mnt/xxx/qwen36_27b_sft_lora_muxi-Qwen3.6-2-steplatest', '--tuner_type', 'lora', '--per_device_train_batch_size', '6', '--gradient_accumulation_steps', '8', '--num_train_epochs', '3.0', '--learning_rate', '0.0003', '--gradient_checkpointing', 'true', '--report_to', 'none', '--save_strategy', 'epoch', '--logging_steps', '10', '--eval_strategy', 'steps', '--add_version', 'false', '--experts_impl', 'eager', '--per_device_eval_batch_size', '6', '--bf16', 'true', '--save_only_model', 'true', '--adapters', '/mnt/xxx/checkpoint-latest', '--template', 'qwen3_5', '--model_type', 'qwen3_5', '--attn_impl', 'flash_attention_2', '--warmup_ratio', '0.05', '--weight_decay', '0.01', '--deepspeed', '/mnt/xxx/ds.json', '--save_total_limit', '1', '--eval_steps', '50.0', '--dataset', '/mnt/xxx/wxq_plain_300_V1/300.jsonl', '--split_dataset_ratio', '0.1', '--max_length', '1024', '--lora_rank', '8', '--lora_alpha', '16', '--lora_dropout', '0.1']

    二、问题现象
    机器上只要运行的有其他服务,例如vllm,多卡微调的时候就会报错,报错信息在下方。但是另一台机器上启动一个同样的vllm,同样的微调任务则不会出现这种问题。
    微调报错信息如下:
    [14:04:48.293][MXKW][E]queues.c :844 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:9560 type:21. Retrying.

    dmesg -T | grep -i mxcd | tail -n 200
    [Wed Oct 28 13:59:36 2026] MXCD.B800.D0.RINGBUF.ERROR wait_ret failed, -110
    [Wed Oct 28 13:59:36 2026] MXCD.B800.D0.RINGBUF.ERROR type 0x0 create ringbuf failed, -110
    [Wed Oct 28 13:59:36 2026] MXCD.B800.D0.RINGBUF.INFO pasid 35690 DRC create ringbuf type 0 retry(by UMD), -110
    [Wed Oct 28 13:59:46 2026] MXCD.B800.D0.RINGBUF.ERROR wait_ret failed, -110
    [Wed Oct 28 13:59:46 2026] MXCD.B800.D0.RINGBUF.ERROR type 0x0 create ringbuf failed, -110
    [Wed Oct 28 13:59:46 2026] MXCD.B800.D0.RINGBUF.INFO pasid 35690 DRC create ringbuf type 0 retry(by UMD), -110
    [Wed Oct 28 13:59:56 2026] MXCD.B800.D0.RINGBUF.ERROR wait_ret failed, -110
    [Wed Oct 28 13:59:56 2026] MXCD.B800.D0.RINGBUF.ERROR type 0x0 create ringbuf failed, -110
    [Wed Oct 28 13:59:56 2026] MXCD.B800.D0.RINGBUF.INFO pasid 35690 DRC create ringbuf type 0 retry(by UMD), -110
    [Wed Oct 28 14:00:07 2026] MXCD.B800.D0.RINGBUF.ERROR wait_ret failed, -110
    [Wed Oct 28 14:00:07 2026] MXCD.B800.D0.RINGBUF.ERROR type 0x0 create ringbuf failed, -110
    [Wed Oct 28 14:00:07 2026] MXCD.B800.D0.RINGBUF.INFO pasid 35690 DRC create ringbuf type 0 retry(by UMD), -110
    [Wed Oct 28 14:00:17 2026] MXCD.B800.D0.RINGBUF.ERROR wait_ret failed, -110
    [Wed Oct 28 14:00:17 2026] MXCD.B800.D0.RINGBUF.ERROR type 0x0 create ringbuf failed, -110
    [Wed Oct 28 14:00:17 2026] MXCD.B800.D0.RINGBUF.INFO pasid 35690 DRC create ringbuf type 0 retry(by UMD), -110
    [Wed Oct 28 14:00:22 2026] MXCD.B800.D0.RINGBUF.ERROR wait_ret failed, -110
    [Wed Oct 28 14:00:22 2026] MXCD.B800.D0.RINGBUF.ERROR type 0x0 create ringbuf failed, -110
    [Wed Oct 28 14:00:23 2026] MXCD.B800.D0.RINGBUF.INFO pasid 35690 DRC create ringbuf type 0 retry(by UMD), -110

  • 沐曦开发者论坛
powered by misago