8专家并行，docker是vllm-metax-0.23.0-maca.ai3.8.0.103-torch2.10-py312-ubuntu22.04-amd64.tar.xz，先是带CUDA graph的启动失败，报驱动层面的trace，index_elementwise_kernel越界；换eager启动以后，发一个请求出现：
(Worker_DP4_EP4 pid=237836) WARNING 08-19 10:28:15 [fused_moe.py:1379] Using default MoE config. Performance might be sub-optimal! Config file not found at /opt/conda/lib/python3.12/site-packages/vllm_metax/model_executor/layers/fused_moe/configs/H=2048/H=2048,E=32,N=512,device_name=MXC500.json, /opt/conda/lib/python3.12/site-packages/vllm_metax/model_executor/layers/fused_moe/configs/E=32,N=512,device_name=MXC500.json
[10:28:15.415][MXC][E]xnack(0x8): kernel causes atu address translation error
[10:28:15.415][MCR][E]mx_trapProcess.cpp :1405: call get queue logic id fail
[10:28:15.415][MCR][E]mx_trapProcess.cpp :130 : trapping pipeID=0,queueID=0!
[10:28:15.415][MCR][E]mx_trapProcess.cpp :131 : trapping virtualDevice=0x7f907c000c70!
[10:28:15.416][MCR][E]mx_trapProcess.cpp :998 : Node ID is 3, Wave ID is 32646
[10:28:15.416][MCR][E]mx_trapProcess.cpp :1021: debug info: 0x7c,0x00,0x00,0x10,0x16,0x2c,0x7f,0x8c,0x7c,0x00,0x00,0x10,0x14,0x24,0x7f,0x8c,
[10:28:15.416][MCR][E]mx_device.cpp :10571: trap:precise positioning.
[10:28:15.416][MCR][E]mx_device.cpp :10576: trapping: kernelName: _ZN7mctlass6KernelINS_4gemm6kernel11MacaGemmMoeINS1_11threadblock20MacaMoeMmaMultistageINS1_9GemmShapeILi16ELi128ELi128EEENS6_ILi16ELi32ELi128EEE15__maca_bfloat16NS_6layout8RowMajorES9_NSA_11ColumnMajorEiSB_Li2ELi1ELb1ELb0ENS_4arch4Sm80ELb0EbEENS_8epilogue11threadblock30MacaMoeGemmEpilogueDirectStoreIS7_S9_SB_S9_SC_Li1ENSG_6thread24MacaLinearCombinationMoeIfLi2EfS9_LNSJ_9ScaleType13ScaleBiasKindE11ELb0ELNS_15FloatRoundStyleE2EEES9_SB_iLi2ELi1ELb1ELb0EbEENS4_37GemmBatchedIdentityThreadblockSwizzleEbEEEEvNT_6ParamsE ,commandIndex: 6573 , trapType: Xnack Error/ATU Fault(0x8)
[10:28:15.416][MCR][E]mx_trapProcess.cpp :918 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[10:28:15.428][MCR][E]mc_runtime_api.cpp :215 : 237839: [7f9368735740] mcGetDevice: Returned mcErrorIllegalAddress
terminate called after throwing an instance of 'c10::AcceleratorError'
what(): CUDA error: an illegal memory access was encountered
Search for cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile withTORCH_USE_CUDA_DSA` to enable device-side assertions.
Yunxiao
- Members
- Joined 2026年4月21日
-
-
需要vllm-metax:0.18.0-maca.ai3.5.3.405-torch2.8-py310-ubuntu22.04-amd64
-
复制官方docker pull命令后：
docker login --username=xxx --password=xxx && sudo docker pull cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.18.0-maca.ai3.5.3.405-torch2.8-py310-ubuntu22.04-amd64
Error response from daemon: Head "cr.metax-tech.com/v2/public-ai-release/maca/vllm-metax/manifests/0.18.0-maca.ai3.5.3.405-torch2.8-py310-ubuntu22.04-amd64": unauthorized: unauthorized: invalid token claims, error: standard claim invalid: token is expired by 1h55m6.614827907s
如何解决？ -
-