好的,感谢,我联系一下问问
好的,感谢,我联系一下问问
一、软硬件信息
1. 服务器厂家:新华三
2. 沐曦GPU型号:MetaX C500X
3. 操作系统内核版本:5.15.0-119-generic
4. 是否开启CPU虚拟化:否
5. mx-smi回显:
mx-smi version: 2.2.9
=================== MetaX System Management Interface Log ===================
Timestamp : Wed Oct 28 13:53:26 2026
Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.2.9 Kernel Mode Driver Version: 3.4.4 |
| MACA Version: 3.3.0.15 BIOS Version: 1.25.1.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C500X | 0 Off | 0000:08:00.0 | 31% Disabled |
| 91W / 350W | 40C P9 | 9642/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C500X | 1 Off | 0000:09:00.0 | 0% Disabled |
| 81W / 350W | 38C P9 | 10152/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C500X | 2 Off | 0000:0e:00.0 | 31% Disabled |
| 91W / 350W | 39C P9 | 10152/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C500X | 3 Off | 0000:11:00.0 | 31% Disabled |
| 88W / 350W | 39C P9 | 10152/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C500X | 4 Off | 0000:32:00.0 | 0% Disabled |
| 85W / 350W | 38C P9 | 61257/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C500X | 5 Off | 0000:38:00.0 | 0% Disabled |
| 68W / 350W | 38C P0 | 874/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C500X | 6 Off | 0000:3b:00.0 | 31% Disabled |
| 90W / 350W | 39C P9 | 9512/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C500X | 7 Off | 0000:3c:00.0 | 0% Disabled |
| 82W / 350W | 39C P9 | 61257/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| 0 2535659 python3.12 8644 |
| 0 2535661 python3.12 38 |
| 0 2535662 python3.12 38 |
| 0 2535663 python3.12 38 |
| 1 2535660 python3.12 9270 |
| 1 2535661 python3.12 0 |
| 1 2535662 python3.12 0 |
| 1 2535663 python3.12 0 |
| 2 2535661 python3.12 9270 |
| 2 2535662 python3.12 0 |
| 2 2535663 python3.12 0 |
| 3 2535661 python3.12 0 |
| 3 2535662 python3.12 9270 |
| 3 2535663 python3.12 0 |
| 4 1945445 VLLM::EngineCor 60382 |
| 6 2535661 python3.12 0 |
| 6 2535662 python3.12 0 |
| 6 2535663 python3.12 8630 |
| 7 2555249 VLLM::EngineCor 60382 |
+---------------------------------------------------------------------------------+
[2026-10-28T14:01:31+08:00] [INFO] cmd: ['swift', 'sft', '--model', '/mnt/xxx/xxx/Qwen3.6-27B', '--output_dir', '/mnt/xxx/qwen36_27b_sft_lora_muxi-Qwen3.6-2-steplatest', '--tuner_type', 'lora', '--per_device_train_batch_size', '6', '--gradient_accumulation_steps', '8', '--num_train_epochs', '3.0', '--learning_rate', '0.0003', '--gradient_checkpointing', 'true', '--report_to', 'none', '--save_strategy', 'epoch', '--logging_steps', '10', '--eval_strategy', 'steps', '--add_version', 'false', '--experts_impl', 'eager', '--per_device_eval_batch_size', '6', '--bf16', 'true', '--save_only_model', 'true', '--adapters', '/mnt/xxx/checkpoint-latest', '--template', 'qwen3_5', '--model_type', 'qwen3_5', '--attn_impl', 'flash_attention_2', '--warmup_ratio', '0.05', '--weight_decay', '0.01', '--deepspeed', '/mnt/xxx/ds.json', '--save_total_limit', '1', '--eval_steps', '50.0', '--dataset', '/mnt/xxx/wxq_plain_300_V1/300.jsonl', '--split_dataset_ratio', '0.1', '--max_length', '1024', '--lora_rank', '8', '--lora_alpha', '16', '--lora_dropout', '0.1']
二、问题现象
机器上只要运行的有其他服务,例如vllm,多卡微调的时候就会报错,报错信息在下方。但是另一台机器上启动一个同样的vllm,同样的微调任务则不会出现这种问题。
微调报错信息如下:
[14:04:48.293][MXKW][E]queues.c :844 : [mxkwCreateQueueBlock][Hint]ioctl create queue block timeout, gpu_id:9560 type:21. Retrying.
dmesg -T | grep -i mxcd | tail -n 200
[Wed Oct 28 13:59:36 2026] MXCD.B800.D0.RINGBUF.ERROR wait_ret failed, -110
[Wed Oct 28 13:59:36 2026] MXCD.B800.D0.RINGBUF.ERROR type 0x0 create ringbuf failed, -110
[Wed Oct 28 13:59:36 2026] MXCD.B800.D0.RINGBUF.INFO pasid 35690 DRC create ringbuf type 0 retry(by UMD), -110
[Wed Oct 28 13:59:46 2026] MXCD.B800.D0.RINGBUF.ERROR wait_ret failed, -110
[Wed Oct 28 13:59:46 2026] MXCD.B800.D0.RINGBUF.ERROR type 0x0 create ringbuf failed, -110
[Wed Oct 28 13:59:46 2026] MXCD.B800.D0.RINGBUF.INFO pasid 35690 DRC create ringbuf type 0 retry(by UMD), -110
[Wed Oct 28 13:59:56 2026] MXCD.B800.D0.RINGBUF.ERROR wait_ret failed, -110
[Wed Oct 28 13:59:56 2026] MXCD.B800.D0.RINGBUF.ERROR type 0x0 create ringbuf failed, -110
[Wed Oct 28 13:59:56 2026] MXCD.B800.D0.RINGBUF.INFO pasid 35690 DRC create ringbuf type 0 retry(by UMD), -110
[Wed Oct 28 14:00:07 2026] MXCD.B800.D0.RINGBUF.ERROR wait_ret failed, -110
[Wed Oct 28 14:00:07 2026] MXCD.B800.D0.RINGBUF.ERROR type 0x0 create ringbuf failed, -110
[Wed Oct 28 14:00:07 2026] MXCD.B800.D0.RINGBUF.INFO pasid 35690 DRC create ringbuf type 0 retry(by UMD), -110
[Wed Oct 28 14:00:17 2026] MXCD.B800.D0.RINGBUF.ERROR wait_ret failed, -110
[Wed Oct 28 14:00:17 2026] MXCD.B800.D0.RINGBUF.ERROR type 0x0 create ringbuf failed, -110
[Wed Oct 28 14:00:17 2026] MXCD.B800.D0.RINGBUF.INFO pasid 35690 DRC create ringbuf type 0 retry(by UMD), -110
[Wed Oct 28 14:00:22 2026] MXCD.B800.D0.RINGBUF.ERROR wait_ret failed, -110
[Wed Oct 28 14:00:22 2026] MXCD.B800.D0.RINGBUF.ERROR type 0x0 create ringbuf failed, -110
[Wed Oct 28 14:00:23 2026] MXCD.B800.D0.RINGBUF.INFO pasid 35690 DRC create ringbuf type 0 retry(by UMD), -110