• Members 17 posts
    2026年8月19日 10:38

    8专家并行,docker是vllm-metax-0.23.0-maca.ai3.8.0.103-torch2.10-py312-ubuntu22.04-amd64.tar.xz,先是带CUDA graph的启动失败,报驱动层面的trace,index_elementwise_kernel越界;换eager启动以后,发一个请求出现:
    (Worker_DP4_EP4 pid=237836) WARNING 08-19 10:28:15 [fused_moe.py:1379] Using default MoE config. Performance might be sub-optimal! Config file not found at /opt/conda/lib/python3.12/site-packages/vllm_metax/model_executor/layers/fused_moe/configs/H=2048/H=2048,E=32,N=512,device_name=MXC500.json, /opt/conda/lib/python3.12/site-packages/vllm_metax/model_executor/layers/fused_moe/configs/E=32,N=512,device_name=MXC500.json
    [10:28:15.415][MXC][E]xnack(0x8): kernel causes atu address translation error
    [10:28:15.415][MCR][E]mx_trapProcess.cpp :1405: call get queue logic id failqueuePhyId=0,queuephyId=0
    [10:28:15.415][MCR][E]mx_trapProcess.cpp :130 : trapping pipeID=0,queueID=0!
    [10:28:15.415][MCR][E]mx_trapProcess.cpp :131 : trapping virtualDevice=0x7f907c000c70!
    [10:28:15.416][MCR][E]mx_trapProcess.cpp :998 : Node ID is 3, Wave ID is 32646
    [10:28:15.416][MCR][E]mx_trapProcess.cpp :1021: debug info: 0x7c,0x00,0x00,0x10,0x16,0x2c,0x7f,0x8c,0x7c,0x00,0x00,0x10,0x14,0x24,0x7f,0x8c,
    [10:28:15.416][MCR][E]mx_device.cpp :10571: trap:precise positioning.
    [10:28:15.416][MCR][E]mx_device.cpp :10576: trapping: kernelName: _ZN7mctlass6KernelINS_4gemm6kernel11MacaGemmMoeINS1_11threadblock20MacaMoeMmaMultistageINS1_9GemmShapeILi16ELi128ELi128EEENS6_ILi16ELi32ELi128EEE15__maca_bfloat16NS_6layout8RowMajorES9_NSA_11ColumnMajorEiSB_Li2ELi1ELb1ELb0ENS_4arch4Sm80ELb0EbEENS_8epilogue11threadblock30MacaMoeGemmEpilogueDirectStoreIS7_S9_SB_S9_SC_Li1ENSG_6thread24MacaLinearCombinationMoeIfLi2EfS9_LNSJ_9ScaleType13ScaleBiasKindE11ELb0ELNS_15FloatRoundStyleE2EEES9_SB_iLi2ELi1ELb1ELb0EbEENS4_37GemmBatchedIdentityThreadblockSwizzleEbEEEEvNT_6ParamsE ,commandIndex: 6573 , trapType: Xnack Error/ATU Fault(0x8)
    [10:28:15.416][MCR][E]mx_trapProcess.cpp :918 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
    [10:28:15.428][MCR][E]mc_runtime_api.cpp :215 : 237839: [7f9368735740] mcGetDevice: Returned mcErrorIllegalAddress
    terminate called after throwing an instance of 'c10::AcceleratorError'
    what(): CUDA error: an illegal memory access was encountered
    Search for cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile withTORCH_USE_CUDA_DSA` to enable device-side assertions.

  • arrow_forward

    Thread has been moved from 产品&运维.

  • arrow_forward

    Thread has been moved from 解决中.