FengKuangDeShiTou

有Qwen3.5-397B-A17B‌模型基于vllm引擎关于分布式部署（pd分离）的指导文档吗？另外是基于哪个镜像的，麻烦提供一个链接，谢谢

配套使用的镜像：vllm-metax:0.20.0-maca.ai3.7.0.107-torch2.8-py310-ubuntu22.04-amd64，启动命令： vllm serve /workspace/models/Qwen3.5-122B-A10B-W8A8 --host 0.0.0.0 --port 8088 --dtype bfloat16 -tp 2 -pp 1 -dp 4 --max-model-len 262144 --enable-expert-parallel --trust-remote-code --gpu-memory-utilization 0.9 --all2all-backend deepep_high_throughput，在单节点部署。请分析一下，是否支持deepep的功能

错误信息：(Worker_DP0_TP0_EP0 pid=25858) INFO 07-23 12:04:52 [backends.py:1069] Using cache directory: /root/.cache/vllm/torch_compile_cache/56c16fdb9d/rank_0_0/backbone for vLLM's torch.compile
(Worker_DP0_TP0_EP0 pid=25858) INFO 07-23 12:04:52 [backends.py:1128] Dynamo bytecode transform time: 9.05 s
(Worker_DP0_TP0_EP0 pid=25858) INFO 07-23 12:04:53 [fix_compilation_backend.py:223] Directly load the compiled graph(s) for compile range (1, 2048) from the cache, took 1.226 s
(Worker_DP0_TP0_EP0 pid=25858) INFO 07-23 12:05:07 [fused_moe.py:1335] User defined config folder: /opt/conda/lib/python3.10/site-packages/vllm_metax/model_executor/layers/fused_moe/configs/H=3072
(Worker_DP0_TP0_EP0 pid=25858) WARNING 07-23 12:05:07 [fused_moe.py:1366] Using default MoE config. Performance might be sub-optimal! Config file not found at /opt/conda/lib/python3.10/site-packages/v llm_metax/model_executor/layers/fused_moe/configs/H=3072/H=3072,E=32,N=1024,device_name=MXC550,dtype=int8_w8a8.json, /opt/conda/lib/python3.10/site-packages/vllm_metax/model_executor/layers/fused_moe/ configs/E=32,N=1024,device_name=MXC550,dtype=int8_w8a8.json
[12:05:08.059][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.059][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=2,queueID=1!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7ef6d8000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=1,queueID=1!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7ef4d8000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=2,queueID=1!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7f28d8000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=2,queueID=1!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7fbba8000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=2,queueID=1!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7f53b8000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1000: Node ID is 7, Wave ID is 32546
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.060][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=1,queueID=1!
[12:05:08.060][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7f31a0000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=2,queueID=1!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7f249c000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1000: Node ID is 5, Wave ID is 32589
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1000: Node ID is 4, Wave ID is 32687
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.060][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.060][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.060][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.060][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1000: Node ID is 3, Wave ID is 32549
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.060][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.060][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1000: Node ID is 8, Wave ID is 32536
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.060][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.060][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.061][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=2,queueID=1!
[12:05:08.061][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7f23e4000c70!
[12:05:08.061][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.061][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.061][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.061][MCR][E]mx_trapProcess.cpp :1000: Node ID is 9, Wave ID is 32535
[12:05:08.061][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.061][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.061][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.061][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.061][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.062][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.062][MCR][E]mx_trapProcess.cpp :1000: Node ID is 2, Wave ID is -2
[12:05:08.062][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.062][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.062][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.062][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.067][MCR][E]mx_trapProcess.cpp :1000: Node ID is 6, Wave ID is 32490
[12:05:08.067][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.067][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.067][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.068][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.071][MCR][E]mc_runtime_api.cpp :215 : 25858: [7efe146fa740] mcGetDevice: Returned mcErrorIllegalAddress
[12:05:08.072][MCR][E]mc_runtime_api.cpp :215 : 25858: [7efe146fa740] mcGetDevice: Returned mcErrorIllegalAddress
[12:05:08.075][MCR][E]mc_runtime_api.cpp :215 : 25858: [7efe146fa740] mcGetDevice: Returned mcErrorIllegalAddress
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] WorkerProc hit an exception.
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] Traceback (most recent call last):
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/tmp/torchinductor_root/kt/cktpgzzi3qihf2snuow3makty3juo3d2rzuerwbazphoytzemhta.py", line 949, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] buf17 = torch.ops.vllm.moe_forward_shared.default(buf16, buf16, buf16, None, 'from_forward_context')
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_ops.py", line 829, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._op(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 125, in _moe_forward_shared
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return layer.runner._forward_impl(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 722, in _forward_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] shared_output, hidden_states = self._apply_quant_method(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 470, in _apply_quant_method
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] fused_out = self.quant_method.apply(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm_metax/quant_config/compressed_tensors_moe/compressed_tensors_moe_w8 a8_int8.py", line 163, in apply
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self.moe_kernel.apply(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1614, in a pply
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self.impl.apply(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1408, in a pply
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._finalize(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1307, in _ finalize
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] self._maybe_apply_shared_experts(shared_experts_input)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1110, in _ maybe_apply_shared_experts
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] self.shared_experts.apply(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/shared_experts.py", line 176 , in apply
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] self._output[self._output_idx] = self._layer(shared_experts_input)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._call_impl(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return forward_call(*args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_moe.py", line 116, in forward
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] out = self.act_fn(gate_up)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._call_impl(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return forward_call(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/custom_op.py", line 136, in forward
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._forward_method(*args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 736, in compile_wrapper
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return fn(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1495, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._torchdynamo_orig_callable(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1272, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] result = self._inner_convert(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 629, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return _compile(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1164, in _compile
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] raise InternalTorchDynamoError(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1111, in _compile
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] guarded_code = compile_inner(code, one_graph, hooks, transform)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_utils_internal.py", line 97, in wrapper_function
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return function(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/dynamo/convert_frame.py", line 793, in compile_inner
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return _compile_inner(code, one_graph, hooks, transform)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 832, in _compile_inner
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] out_code = transform_code_object(code, transform)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1424, in transform_code object
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] transformations(instructions, code_options)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 288, in _fn
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] torch.cuda.set_rng_state(cuda_rng_state)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/cuda/random.py", line 79, in set_rng_state
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] _lazy_call(cb)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py", line 348, in _lazy_call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] callable()
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/cuda/random.py", line 75, in cb
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] idx = current_device()
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py", line 1146, in current_device
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return torch._C._cuda_getDevice()
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] torch._dynamo.exc.InternalTorchDynamoError: AcceleratorError: CUDA error: an illegal memory access was encountered
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962]
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962]
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962]
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] During handling of the above exception, another exception occurred:
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962]
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] Traceback (most recent call last):
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 957, in worker_busy_loop
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] output = func(*args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return func(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 370, in determine_available_memory
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] self.model_runner.profile_run()
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5848, in profile_run
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] hidden_states, last_hidden_states = self._dummy_run(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return func(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5537, in _dummy_run
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] outputs = self.model(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._call_impl(*args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return forward_call(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_5.py", line 695, in forward
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] hidden_states = self.language_model.model(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 638, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] output = TorchCompileWithNoGuardsWrapper.call(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/compilation/wrapper.py", line 197, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._call_with_optional_nvtx_range(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/compilation/wrapper.py", line 76, in _call_with_optional_nvtx_range
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return callable_fn(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 736, in compile_wrapper
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return fn(*args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_next.py", line 495, in forward
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] def forward(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return fn(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/compilation/caching.py", line 215, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self.optimized_call(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "<string>", line 221, in execution_fn
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/compilation/piecewise_backend.py", line 380, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return range_entry.runnable(args)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/compilation/compiler_interface.py", line 433, in compiled_graph_wra pper
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] graph_output = inductor_compiled_graph(args)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/standalone_compile.py", line 62, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._compiled_fn(args)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/standalone_compile.py", line 182, in <lambda>
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return CompiledArtifact(lambda args: compiled_fn(list(args)), None)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 384, in runtim e_wrapper
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] all_outs = call_func_at_runtime_with_args(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runt ime_with_args
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] out = normalize_as_list(f(args))
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 556, in wrappe r
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return compiled_fn(runtime_args)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 584, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self.current_callable(inputs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/utils.py", line 2720, in run
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] out = model(new_inputs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/tmp/torchinductor_root/kt/cktpgzzi3qihf2snuow3makty3juo3d2rzuerwbazphoytzemhta.py", line 906, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] with torch.cuda._DeviceGuard(0):
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py", line 525, in exit
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] self.idx = torch.cuda._maybe_exchange_device(self.prev_idx)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] torch.AcceleratorError: CUDA error: an illegal memory access was encountered
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.