Yunxiao

8专家并行，docker是vllm-metax-0.23.0-maca.ai3.8.0.103-torch2.10-py312-ubuntu22.04-amd64.tar.xz，先是带CUDA graph的启动失败，报驱动层面的trace，index_elementwise_kernel越界；换eager启动以后，发一个请求出现：
(Worker_DP4_EP4 pid=237836) WARNING 08-19 10:28:15 [fused_moe.py:1379] Using default MoE config. Performance might be sub-optimal! Config file not found at /opt/conda/lib/python3.12/site-packages/vllm_metax/model_executor/layers/fused_moe/configs/H=2048/H=2048,E=32,N=512,device_name=MXC500.json, /opt/conda/lib/python3.12/site-packages/vllm_metax/model_executor/layers/fused_moe/configs/E=32,N=512,device_name=MXC500.json
[10:28:15.415][MXC][E]xnack(0x8): kernel causes atu address translation error
[10:28:15.415][MCR][E]mx_trapProcess.cpp :1405: call get queue logic id fail
[10:28:15.415][MCR][E]mx_trapProcess.cpp :130 : trapping pipeID=0,queueID=0!
[10:28:15.415][MCR][E]mx_trapProcess.cpp :131 : trapping virtualDevice=0x7f907c000c70!
[10:28:15.416][MCR][E]mx_trapProcess.cpp :998 : Node ID is 3, Wave ID is 32646
[10:28:15.416][MCR][E]mx_trapProcess.cpp :1021: debug info: 0x7c,0x00,0x00,0x10,0x16,0x2c,0x7f,0x8c,0x7c,0x00,0x00,0x10,0x14,0x24,0x7f,0x8c,
[10:28:15.416][MCR][E]mx_device.cpp :10571: trap:precise positioning.
[10:28:15.416][MCR][E]mx_device.cpp :10576: trapping: kernelName: _ZN7mctlass6KernelINS_4gemm6kernel11MacaGemmMoeINS1_11threadblock20MacaMoeMmaMultistageINS1_9GemmShapeILi16ELi128ELi128EEENS6_ILi16ELi32ELi128EEE15__maca_bfloat16NS_6layout8RowMajorES9_NSA_11ColumnMajorEiSB_Li2ELi1ELb1ELb0ENS_4arch4Sm80ELb0EbEENS_8epilogue11threadblock30MacaMoeGemmEpilogueDirectStoreIS7_S9_SB_S9_SC_Li1ENSG_6thread24MacaLinearCombinationMoeIfLi2EfS9_LNSJ_9ScaleType13ScaleBiasKindE11ELb0ELNS_15FloatRoundStyleE2EEES9_SB_iLi2ELi1ELb1ELb0EbEENS4_37GemmBatchedIdentityThreadblockSwizzleEbEEEEvNT_6ParamsE ,commandIndex: 6573 , trapType: Xnack Error/ATU Fault(0x8)
[10:28:15.416][MCR][E]mx_trapProcess.cpp :918 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[10:28:15.428][MCR][E]mc_runtime_api.cpp :215 : 237839: [7f9368735740] mcGetDevice: Returned mcErrorIllegalAddress
terminate called after throwing an instance of 'c10::AcceleratorError'
what(): CUDA error: an illegal memory access was encountered
Search for cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile withTORCH_USE_CUDA_DSA` to enable device-side assertions.

问题：C500 显存经ibv_reg_dmabuf_mr注册给mlx5网卡。GPU kernel写这块显存后，GPU的写停在L2域，网卡通过dma-buf P2P读HBM读到的是旧值。
提问：
1. C500 GPUDirect 是否支持网卡从显存 READ(从显存发送),还是只支持网卡写显存(接收)?
2. 若支持，有没有让GPU kernel的显存写对网卡P2P读可见的 API（L2→HBM flush/system-scope store/dma-buf 注册前的正确内存属性)?mcDeviceFlushGPUDirectRDMAWrites是接收侧,有无发送侧对应？

我现在有一个.cu文件，如何把他转换成MACA源代码比如.maca或者直接编译成可执行文件

需要vllm-metax:0.21.0-maca.ai3.7.1.106-torch2.8-py310-ubuntu22.04-amd64离线镜像包导出

目前想把C500的RDMA传输控制面从CPU改为由GPU kernel来做，需要GPU和MACA支持GDAKI / GPUDirect Async，请问目前是否支持，有相关文档吗？

谢谢，已发帖申请

需要vllm-metax:0.18.0-maca.ai3.5.3.405-torch2.8-py310-ubuntu22.04-amd64

能否优化一下呢？我重新复制docker命令以后并没有触发断点续传，是完全重新下载了，在时间范围内下载不完，这个错误就循环出现了，这是我刚刚复制命令后下载的：
Login Succeeded
0.18.0-maca.ai3.5.3.405-torch2.8-py310-ubuntu22.04-amd64: Pulling from public-ai-release/maca/vllm-metax
43f89b94cd7d: Already exists
f2b7c1644807: Pull complete
077977940afd: Pull complete
b1b8f989434a: Pull complete
84398257dc67: Downloading 6.488MB/94.22MB
443c93c96edb: Downloading 4.866MB/3.572GB
2f427e59f857: Downloading 1.229MB/3.485MB
b0c2e660298b: Waiting
0c1d591115bb: Waiting
30e855b71d68: Waiting
06830532ea96: Pulling fs layer
4f4fb700ef54: Waiting
afe16f59bac8: Waiting
1d0f707f6f52: Waiting
96b54463a433: Waiting
37bec76d27ba: Waiting

Login Succeeded
0.18.0-maca.ai3.5.3.405-torch2.8-py310-ubuntu22.04-amd64: Pulling from public-ai-release/maca/vllm-metax
43f89b94cd7d: Already exists
f2b7c1644807: Pull complete
077977940afd: Pull complete
b1b8f989434a: Pull complete
84398257dc67: Pull complete
443c93c96edb: Downloading 857MB/3.572GB
2f427e59f857: Download complete
b0c2e660298b: Downloading 826.1MB/2.967GB
0c1d591115bb: Download complete
30e855b71d68: Download complete
06830532ea96: Downloading
4f4fb700ef54: Waiting
afe16f59bac8: Waiting
1d0f707f6f52: Waiting
96b54463a433: Waiting
37bec76d27ba: Waiting
unauthorized: unauthorized: invalid token claims, error: standard claim invalid: token is expired by 31m9.653425596s
xxx@xxx:~$ sudo docker login --username=api-cr-cloud-lib-ai-ro@78575726 --password=xxx cr.metax-tech.com && sudo docker pull cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.18.0-maca.ai3.5.3.405-torch2.8-py310-ubuntu22.04-amd64
[sudo] xxx 的密码：
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get "cr.metax-tech.com/v2/": unauthorized: unauthorized: invalid token claims, error: standard claim invalid: token is expired by 1h15m45.733795048s

管理员你好：
刚刚又试了一次还是同样的报错信息：
xxx@xxx:~$ docker login --username=api-cr-cloud-lib-ai-ro@xxx cr.metax-tech.com && sudo docker pull cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.18.0-maca.ai3.5.3.405-torch2.8-py310-ubuntu22.04-amd64
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/dpu4/.docker/config.json.
Configure a credential helper to remove this warning. See
docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[sudo] xxx 的密码：
Error response from daemon: Head "cr.metax-tech.com/v2/public-ai-release/maca/vllm-metax/manifests/0.18.0-maca.ai3.5.3.405-torch2.8-py310-ubuntu22.04-amd64": unauthorized: unauthorized: invalid token claims, error: standard claim invalid: token is expired by 25h0m47.748002581s
我这边没有root权限，pull docker需要sudo (sudo docker pull)
我在login也用sudo以后 (sudo login && sudo docker pull) 报错消失，从头开始pulling:
Login Succeeded
0.18.0-maca.ai3.5.3.405-torch2.8-py310-ubuntu22.04-amd64: Pulling from public-ai-release/maca/vllm-metax
43f89b94cd7d: Already exists
f2b7c1644807: Pull complete
077977940afd: Pull complete
b1b8f989434a: Downloading 3.637MB/4.809MB
84398257dc67: Downloading 4.325MB/94.22MB
443c93c96edb: Downloading 3.785MB/3.572GB
2f427e59f857: Waiting
b0c2e660298b: Waiting
0c1d591115bb: Waiting
30e855b71d68: Waiting
06830532ea96: Pulling fs layer
4f4fb700ef54: Waiting
afe16f59bac8: Waiting
1d0f707f6f52: Waiting
96b54463a433: Waiting
37bec76d27ba: Waiting
根据下载速度观察，大概率在token valid的时间范围内没办法完整下载整个镜像，后续可能还是会报同样的错误，如果后续报错我会在这个帖子下发详细信息。
感谢回复。

管理员你好，我这边没有root权限，请问在vllm-metax-0.13.0的镜像中是否可以源码编译vllm-metax-0.18.0？我的源镜像是：vllm-metax:0.13.0-maca.ai3.3.0.303-torch2.8-py310-ubuntu22.04-amd64

重新复制以后是一模一样的命令，还是报token is expired的错误

复制官方docker pull命令后：
docker login --username=xxx --password=xxx && sudo docker pull cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.18.0-maca.ai3.5.3.405-torch2.8-py310-ubuntu22.04-amd64
Error response from daemon: Head "cr.metax-tech.com/v2/public-ai-release/maca/vllm-metax/manifests/0.18.0-maca.ai3.5.3.405-torch2.8-py310-ubuntu22.04-amd64": unauthorized: unauthorized: invalid token claims, error: standard claim invalid: token is expired by 1h55m6.614827907s
如何解决？

官网的教程是需要在windows安装（win-perf-kit工具包内），通过ssh连接服务器使用，请问有没有在linux中直接通过cli直接使用mcProfiler的方法或者相关的软件包安装？

请问在哪里查询相关的api信息？

GDR，GPUDirect RDMA

或者类似于NVIDIA nvidia-peermem的模块？