Posts | FengKuangDeShiTou | 沐曦开发者论坛

See post chevron_right

FengKuangDeShiTou
Members

有Qwen3.5-397B-A17B‌模型基于vllm引擎关于分布式部署（pd分离）的指导文档吗？已解决 2026年8月5日 15:33

文档和镜像还没有开源吗？
See post chevron_right

FengKuangDeShiTou
Members

有Qwen3.5-397B-A17B‌模型基于vllm引擎关于分布式部署（pd分离）的指导文档吗？已解决 2026年8月5日 15:10

有Qwen3.5-397B-A17B‌模型基于vllm引擎关于分布式部署（pd分离）的指导文档吗？另外是基于哪个镜像的，麻烦提供一个链接，谢谢
See post chevron_right

FengKuangDeShiTou
Members

求助：metax C550上使用vllm部署Qwen3.5-122B-A10B-W8A8模型的deepep功能报错已解决 2026年7月24日 16:13

另外麻烦确认下，文档中只有low-latency模式的实例，high_throughput模式是否支持？
See post chevron_right

FengKuangDeShiTou
Members

求助：metax C550上使用vllm部署Qwen3.5-122B-A10B-W8A8模型的deepep功能报错已解决 2026年7月24日 15:25

切到这个镜像了，也没有找到该目录，具体是在镜像的哪个目录下呢？
See post chevron_right

FengKuangDeShiTou
Members

求助：metax C550上使用vllm部署Qwen3.5-122B-A10B-W8A8模型的deepep功能报错已解决 2026年7月24日 15:02

deepep
├── low-latency.sh # 测试脚本
├── src
│ ├── internode.py # 多节点测试
│ ├── low_latency.py # 单节点测试
│ └── utils.py # 工具
└── tmux-config.sh # tmux 配置文件

请问文档中的这个deeppe脚本目录是在哪里呢？
See post chevron_right

FengKuangDeShiTou
Members

求助：metax C550上使用vllm部署Qwen3.5-122B-A10B-W8A8模型的deepep功能报错已解决 2026年7月24日 14:32

镜像： sglang:v0.5.12-deepseek-v4-rc1-maca.ai.3.7.1.110-torch2.8-py310-ubuntu22.04-amd64
脚本： gitee.com/metax-maca/DeepEP/blob/main/tests/test_intranode.py

使用了metax的deepep库的测试脚本，发现底层库报错了，请确认这个镜像中的deepep底层库是否正常？

错误信息：
[config] num_tokens=4096, hidden=7168, num_topk=8
[layout] Kernel performance: 0.178 ms

[testing] Running with BF16, without top-k (async=False, previous=False) ...[14:26:55.346][MXC][E]memory violation(0x4): [14:26:55.346][MXC][E]memory violation(0x4): memory access offset is negative, out of bounds, or misaligned in kernel
[14:26:55.346][MXC][E]memory violation(0x4): memory access offset is negative, out of bounds, or misaligned in kernel
[14:26:55.346][MXC][E]memory violation(0x4): memory access offset is negative, out of bounds, or misaligned in kernel
[14:26:55.346][MXC][E]memory violation(0x4): memory access offset is negative, out of bounds, or misaligned in kernel
[14:26:55.346][MXC][E]memory violation(0x4): memory access offset is negative, out of bounds, or misaligned in kernel
[14:26:55.346][MXC][E]memory violation(0x4): memory access offset is negative, out of bounds, or misaligned in kernel
[14:26:55.346][MXC][E]memory violation(0x4): memory access offset is negative, out of bounds, or misaligned in kernel
[14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_device.cpp [14:26:55.346][MCR][E]mx_device.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_device.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_device.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_device.cpp [14:26:55.346][MCR][E]mx_device.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_device.cpp [14:26:55.346][MCR][E]mx_device.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_device.cpp [14:26:55.346][MCR][E]mx_device.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.346][MCR][E]mx_trapProcess.cpp [14:26:55.347][MCR][E]mx_device.cpp [14:26:55.347][MCR][E]mx_device.cpp [14:26:55.347][MCR][E]mx_trapProcess.cpp [14:26:55.347][MCR][E]mx_trapProcess.cpp [14:26:55.347][MCR][E]mx_trapProcess.cpp [14:26:55.347][MCR][E]mx_trapProcess.cpp [14:26:55.347][MCR][E]mx_trapProcess.cpp [14:26:55.347][MCR][E]mx_trapProcess.cpp [14:26:55.347][MCR][E]mx_device.cpp [14:26:55.347][MCR][E]mx_device.cpp [14:26:55.347][MCR][E]mx_trapProcess.cpp [14:26:55.349][MCR][E]mx_trapProcess.cpp [14:26:55.353][MCR][E]mx_trapProcess.cpp [14:26:55.353][MCR][E]mx_trapProcess.cpp [14:26:55.353][MCR][E]mx_device.cpp [14:26:55.353][MCR][E]mx_device.cpp [14:26:55.353][MCR][E]mx_trapProcess.cpp [14:26:55.629][MCR][E]mc_runtime_api.cpp [14:26:55.629][MCR][E]mc_runtime_api.cpp memory access offset is negative, out of bounds, or misaligned in kernel
:128 : trapping pipeID=2,queueID=0!
:129 : trapping virtualDevice=0x7f1f78000c70!
:128 : trapping pipeID=2,queueID=0!
:129 : trapping virtualDevice=0x7f2efc000c70!
:128 : trapping pipeID=2,queueID=0!
:129 : trapping virtualDevice=0x7f36cc000c70!
:128 : trapping pipeID=2,queueID=0!
:129 : trapping virtualDevice=0x7f77dc000c70!
:128 : trapping pipeID=2,queueID=0!
:129 : trapping virtualDevice=0x7fb884000c70!
:1000: Node ID is 2, Wave ID is -2
:1023: debug info: 0x67,0x0e,0x28,0x0a,0x0e,0x01,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
:10101: trap:precise positioning.
:10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 112 , trapType: Xnack Error/ATU Fault(0x8)
:1000: Node ID is 7, Wave ID is -2
:1023: debug info: 0x7c,0x00,0x00,0x70,0x0c,0x18,0x7f,0x3e,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
:10101: trap:precise positioning.
:128 : trapping pipeID=2,queueID=0!
:128 : trapping pipeID=2,queueID=0!
:1000: Node ID is 8, Wave ID is -2
:10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 112 , trapType: Memory Violation(0x4)
:129 : trapping virtualDevice=0x7fdbe0000c70!
:1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
:129 : trapping virtualDevice=0x7f07e4000c70!
:10101: trap:precise positioning.
:10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 112 , trapType: Memory Violation(0x4)
:128 : trapping pipeID=2,queueID=0!
:129 : trapping virtualDevice=0x7fdfc8000c70!
:914 : Memory violation(0x4) exception happened in the shader, the mcruntime api will be disabled
:1000: Node ID is 6, Wave ID is -2
:1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
:10101: trap:precise positioning.
:10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 112 , trapType: Memory Violation(0x4)
:1000: Node ID is 5, Wave ID is -2
:1023: debug info: 0xd6,0x00,0x82,0x04,0x4f,0x01,0x00,0x00,0xe0,0x01,0x03,0x00,0x4e,0x00,0x00,0x00,
:10101: trap:precise positioning.
:10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 112 , trapType: Xnack Error/ATU Fault(0x8)
:1000: Node ID is 4, Wave ID is -2
:1023: debug info: 0x7c,0x00,0x00,0x73,0x0c,0x10,0x7f,0x3e,0xdc,0x00,0x00,0x60,0x22,0x14,0x7f,0x3e,
:10101: trap:precise positioning.
:10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 112 , trapType: Xnack Error/ATU Fault(0x8)
:914 : Memory violation(0x4) exception happened in the shader, the mcruntime api will be disabled
:914 : Memory violation(0x4) exception happened in the shader, the mcruntime api will be disabled
:914 : Memory violation(0x4) exception happened in the shader, the mcruntime api will be disabled
:914 : Memory violation(0x4) exception happened in the shader, the mcruntime api will be disabled
:1000: Node ID is 3, Wave ID is -2
:1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
:10101: trap:precise positioning.
:10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 112 , trapType: Memory Violation(0x4)
:914 : Memory violation(0x4) exception happened in the shader, the mcruntime api will be disabled
:914 : Memory violation(0x4) exception happened in the shader, the mcruntime api will be disabled
:1000: Node ID is 9, Wave ID is -2
:1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
:10101: trap:precise positioning.
:10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 112 , trapType: Memory Violation(0x4)
:914 : Memory violation(0x4) exception happened in the shader, the mcruntime api will be disabled
:1737: 1961 : [7f1d8cff9640] mcCtxGetCurrent: Returned mcErrorIllegalAddress
:215 : 1961 : [7f1d8cff9640] mcGetDevice: Returned mcErrorIllegalAddress

metax-host-084:1961:3388 [0] /workspace/out/Release/build/linux/x86_64/mccl/macaify/src/proxy.cc:1219 MCCL WARN [Proxy Service] MACA shut down, proxy exit!

[14:26:55.630][MCR][E]mc_runtime_api.cpp :1737: 1960 : [7f2d037fe640] mcCtxGetCurrent: Returned mcErrorIllegalAddress
[14:26:55.630][MCR][E]mc_runtime_api.cpp :215 : 1960 : [7f2d037fe640] mcGetDevice: Returned mcErrorIllegalAddress

metax-host-084:1960:3389 [0] /workspace/out/Release/build/linux/x86_64/mccl/macaify/src/proxy.cc:1219 MCCL WARN [Proxy Service] MACA shut down, proxy exit!

[14:26:55.631][MCR][E]mc_runtime_api.cpp :1737: 1962 : [7f05f5ffb640] mcCtxGetCurrent: Returned mcErrorIllegalAddress
[14:26:55.631][MCR][E]mc_runtime_api.cpp :1737: 1963 : [7fdddcff9640] mcCtxGetCurrent: Returned mcErrorIllegalAddress
[14:26:55.631][MCR][E]mc_runtime_api.cpp :215 : 1962 : [7f05f5ffb640] mcGetDevice: Returned mcErrorIllegalAddress

metax-host-084:1962:3393 [0] /workspace/out/Release/build/linux/x86_64/mccl/macaify/src/proxy.cc:1219 MCCL WARN [Proxy Service] MACA shut down, proxy exit!

[14:26:55.631][MCR][E]mc_runtime_api.cpp :215 : 1963 : [7fdddcff9640] mcGetDevice: Returned mcErrorIllegalAddress
[14:26:55.631][MCR][E]mc_runtime_api.cpp :1737: 1964 : [7fd9ea7fc640] mcCtxGetCurrent: Returned mcErrorIllegalAddress

metax-host-084:1963:3386 [0] /workspace/out/Release/build/linux/x86_64/mccl/macaify/src/proxy.cc:1219 MCCL WARN [Proxy Service] MACA shut down, proxy exit!

[14:26:55.631][MCR][E]mc_runtime_api.cpp :1737: 1967 : [7f34cb7fe640] mcCtxGetCurrent: Returned mcErrorIllegalAddress
[14:26:55.631][MCR][E]mc_runtime_api.cpp :215 : 1964 : [7fd9ea7fc640] mcGetDevice: Returned mcErrorIllegalAddress

metax-host-084:1964:3387 [0] /workspace/out/Release/build/linux/x86_64/mccl/macaify/src/proxy.cc:1219 MCCL WARN [Proxy Service] MACA shut down, proxy exit!

[14:26:55.631][MCR][E]mc_runtime_api.cpp :215 : 1967 : [7f34cb7fe640] mcGetDevice: Returned mcErrorIllegalAddress

metax-host-084:1967:3392 [0] /workspace/out/Release/build/linux/x86_64/mccl/macaify/src/proxy.cc:1219 MCCL WARN [Proxy Service] MACA shut down, proxy exit!

[14:26:55.632][MCR][E]mc_runtime_api.cpp :1737: 1965 : [7f75f0ff9640] mcCtxGetCurrent: Returned mcErrorIllegalAddress
[14:26:55.632][MCR][E]mc_runtime_api.cpp :215 : 1965 : [7f75f0ff9640] mcGetDevice: Returned mcErrorIllegalAddress

metax-host-084:1965:3391 [0] /workspace/out/Release/build/linux/x86_64/mccl/macaify/src/proxy.cc:1219 MCCL WARN [Proxy Service] MACA shut down, proxy exit!

[14:26:55.632][MCR][E]mc_runtime_api.cpp :1737: 1966 : [7fb683fff640] mcCtxGetCurrent: Returned mcErrorIllegalAddress
[14:26:55.632][MCR][E]mc_runtime_api.cpp :215 : 1966 : [7fb683fff640] mcGetDevice: Returned mcErrorIllegalAddress

metax-host-084:1966:3390 [0] /workspace/out/Release/build/linux/x86_64/mccl/macaify/src/proxy.cc:1219 MCCL WARN [Proxy Service] MACA shut down, proxy exit!

[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7f23e9044380] wait
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7f0c52044380] wait
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7f337d444380] wait
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7f23e9044380] wait
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7fe044244380] wait
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x563a1ab12620 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7fbcefe44380] wait
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7f3b31844380] wait
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7f7c4aa44380] wait
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x563a1a8bd7f0 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7f0c52044380] wait
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x55b45b957f10 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7f337d444380] wait
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x55d0bfbeeea0 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x55b45b957540 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7fe044244380] wait
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7f3b31844380] wait
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7fbcefe44380] wait
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7f7c4aa44380] wait
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x55d0bfe0e2b0 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x55b8fefd5c20 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x5560951beb50 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x55b38f1f3330 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x563fc8c3ae20 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7fe43cc44380] wait
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x55b8ff437b10 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x5560953e6920 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x55b38f601f70 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x563fc8c2dc60 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_signal.cpp :590 : Failed signal [0x7fe43cc44380] wait
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x5573668d9440 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mx_command.cpp :129 : command 0x5573668b5600 status invalid! Current status : 0,try to set status : -1
[14:26:56.345][MCR][E]mc_runtime_api.cpp :215 : 1965 : [7f7e22320740] mcGetDevice: Returned mcErrorIllegalAddress
[14:26:56.345][MCR][E]mc_runtime_api.cpp :215 : 1966 : [7fbec767d740] mcGetDevice: Returned mcErrorIllegalAddress
[14:26:56.345][MCR][E]mc_runtime_api.cpp :215 : 1961 : [7f25c0823740] mcGetDevice: Returned mcErrorIllegalAddress
terminate called after throwing an instance of 'c10::AcceleratorError'
terminate called after throwing an instance of 'c10::AcceleratorError'
[14:26:56.345][MCR][E]mc_runtime_api.cpp :215 : 1963 : [7fe614499740] mcGetDevice: Returned mcErrorIllegalAddress
terminate called after throwing an instance of 'c10::AcceleratorError'
[14:26:56.346][MCR][E]mc_runtime_api.cpp :215 : 1960 : [7f354b46a740] mcGetDevice: Returned mcErrorIllegalAddress
[14:26:56.346][MCR][E]mc_runtime_api.cpp :215 : 1962 : [7f0e297f8740] mcGetDevice: Returned mcErrorIllegalAddress
[14:26:56.346][MCR][E]mc_runtime_api.cpp :215 : 1967 : [7f3d091f0740] mcGetDevice: Returned mcErrorIllegalAddress
[14:26:56.346][MCR][E]mc_runtime_api.cpp :215 : 1964 : [7fe21bb17740] mcGetDevice: Returned mcErrorIllegalAddress
terminate called after throwing an instance of 'terminate called after throwing an instance of 'c10::AcceleratorErrorc10::AcceleratorError'
'
terminate called after throwing an instance of 'c10::AcceleratorError'
terminate called after throwing an instance of 'c10::AcceleratorError'
terminate called after throwing an instance of 'c10::AcceleratorError'
what(): CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /workspace/framework/mcPytorch/c10/cuda/CUDAException.cpp:42 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x88 (0x7fe612b928f8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x177d3 (0x7fe613b177d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::SetDevice(signed char, bool) + 0x72 (0x7fe613b5f6f2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x30207 (0x7fe613b30207 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0x7807b5 (0x7fe5ecb1c7b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x44d0f3d (0x7fe5f086cf3d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: at::_ops::_local_scalar_dense::redispatch(c10::DispatchKeySet, at::Tensor const&) + 0x6b (0x7fe5fe69ef4b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x4d63cf0 (0x7fe600e36cf0 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x4d63d70 (0x7fe600e36d70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::_ops::_local_scalar_dense::call(at::Tensor const&) + 0x167 (0x7fe5fe762a67 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::native::item(at::Tensor const&) + 0xd3 (0x7fe5fde5ea93 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x2e0168d (0x7fe5feed468d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #12: at::_ops::item::call(at::Tensor const&) + 0x167 (0x7fe5fe59fc77 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x5151f6 (0x7fe60df9b1f6 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #14: <unknown function> + 0x14357c (0x5572779da57c in /opt/conda/bin/python3)
frame #15: _PyEval_EvalFrameDefault + 0x5ebb (0x5572779cd3eb in /opt/conda/bin/python3)
frame #16: <unknown function> + 0x14c6e1 (0x5572779e36e1 in /opt/conda/bin/python3)
frame #17: <unknown function> + 0x1d99b6 (0x557277a709b6 in /opt/conda/bin/python3)
frame #18: PyObject_CallMethod + 0xcd (0x557277a7558d in /opt/conda/bin/python3)
frame #19: torch::handle_torch_function_no_python_arg_parser(c10::ArrayRef<_object>, _object, _object, char const, _object, char const, torch::TorchFunctionName) + 0x1a3c (0x7fe60e7346fc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #20: <unknown function> + 0xcaf0fd (0x7fe60e7350fd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #21: <unknown function> + 0x515165 (0x7fe60df9b165 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #22: <unknown function> + 0x14357c (0x5572779da57c in /opt/conda/bin/python3)
frame #23: _PyEval_EvalFrameDefault + 0x735 (0x5572779c7c65 in /opt/conda/bin/python3)
frame #24: _PyFunction_Vectorcall + 0x6f (0x5572779d7b1f in /opt/conda/bin/python3)
frame #25: _PyEval_EvalFrameDefault + 0x332 (0x5572779c7862 in /opt/conda/bin/python3)
frame #26: _PyFunction_Vectorcall + 0x6f (0x5572779d7b1f in /opt/conda/bin/python3)
frame #27: _PyEval_EvalFrameDefault + 0x332 (0x5572779c7862 in /opt/conda/bin/python3)
frame #28: _PyFunction_Vectorcall + 0x6f (0x5572779d7b1f in /opt/conda/bin/python3)
frame #29: _PyEval_EvalFrameDefault + 0x2ec3 (0x5572779ca3f3 in /opt/conda/bin/python3)
frame #30: _PyFunction_Vectorcall + 0x6f (0x5572779d7b1f in /opt/conda/bin/python3)
frame #31: _PyEval_EvalFrameDefault + 0x2ec3 (0x5572779ca3f3 in /opt/conda/bin/python3)
frame #32: _PyFunction_Vectorcall + 0x6f (0x5572779d7b1f in /opt/conda/bin/python3)
frame #33: _PyEval_EvalFrameDefault + 0x735 (0x5572779c7c65 in /opt/conda/bin/python3)
frame #34: _PyFunction_Vectorcall + 0x6f (0x5572779d7b1f in /opt/conda/bin/python3)
frame #35: _PyEval_EvalFrameDefault + 0x735 (0x5572779c7c65 in /opt/conda/bin/python3)
frame #36: _PyFunction_Vectorcall + 0x6f (0x5572779d7b1f in /opt/conda/bin/python3)
frame #37: _PyEval_EvalFrameDefault + 0x332 (0x5572779c7862 in /opt/conda/bin/python3)
frame #38: _PyFunction_Vectorcall + 0x6f (0x5572779d7b1f in /opt/conda/bin/python3)
frame #39: _PyEval_EvalFrameDefault + 0x13d0 (0x5572779c8900 in /opt/conda/bin/python3)
frame #40: <unknown function> + 0x1db6a2 (0x557277a726a2 in /opt/conda/bin/python3)
frame #41: PyEval_EvalCode + 0x87 (0x557277a725e7 in /opt/conda/bin/python3)
frame #42: <unknown function> + 0x20e3fc (0x557277aa53fc in /opt/conda/bin/python3)
frame #43: <unknown function> + 0x2092d4 (0x557277aa02d4 in /opt/conda/bin/python3)
frame #44: PyRun_StringFlags + 0x7d (0x557277a9846d in /opt/conda/bin/python3)
frame #45: PyRun_SimpleStringFlags + 0x3d (0x557277a9828d in /opt/conda/bin/python3)
frame #46: Py_RunMain + 0x272 (0x557277a97152 in /opt/conda/bin/python3)
frame #47: Py_BytesMain + 0x39 (0x557277a64dc9 in /opt/conda/bin/python3)
frame #48: <unknown function> + 0x29d90 (0x7fe6144c5d90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #49: __libc_start_main + 0x80 (0x7fe6144c5e40 in /lib/x86_64-linux-gnu/libc.so.6)
frame #50: <unknown function> + 0x1cdcc1 (0x557277a64cc1 in /opt/conda/bin/python3)
what():
CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /workspace/framework/mcPytorch/c10/cuda/CUDAException.cpp:42 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x88 (0x7f7e207928f8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x177d3 (0x7f7e20af67d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::SetDevice(signed char, bool) + 0x72 (0x7f7e20b3e6f2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x30207 (0x7f7e20b0f207 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0x7807b5 (0x7f7dfab1c7b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x44d0f3d (0x7f7dfe86cf3d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: at::_ops::_local_scalar_dense::redispatch(c10::DispatchKeySet, at::Tensor const&) + 0x6b (0x7f7e0c69ef4b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x4d63cf0 (0x7f7e0ee36cf0 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x4d63d70 (0x7f7e0ee36d70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::_ops::_local_scalar_dense::call(at::Tensor const&) + 0x167 (0x7f7e0c762a67 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::native::item(at::Tensor const&) + 0xd3 (0x7f7e0be5ea93 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x2e0168d (0x7f7e0ced468d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #12: at::_ops::item::call(at::Tensor const&) + 0x167 (0x7f7e0c59fc77 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x5151f6 (0x7f7e1bf9b1f6 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #14: <unknown function> + 0x14357c (0x555fa065757c in /opt/conda/bin/python3)
frame #15: _PyEval_EvalFrameDefault + 0x5ebb (0x555fa064a3eb in /opt/conda/bin/python3)
frame #16: <unknown function> + 0x14c6e1 (0x555fa06606e1 in /opt/conda/bin/python3)
frame #17: <unknown function> + 0x1d99b6 (0x555fa06ed9b6 in /opt/conda/bin/python3)
frame #18: PyObject_CallMethod + 0xcd (0x555fa06f258d in /opt/conda/bin/python3)
frame #19: torch::handle_torch_function_no_python_arg_parser(c10::ArrayRef<_object>, _object, _object, char const, _object, char const, torch::TorchFunctionName) + 0x1a3c (0x7f7e1c7346fc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #20: <unknown function> + 0xcaf0fd (0x7f7e1c7350fd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #21: <unknown function> + 0x515165 (0x7f7e1bf9b165 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #22: <unknown function> + 0x14357c (0x555fa065757c in /opt/conda/bin/python3)
frame #23: _PyEval_EvalFrameDefault + 0x735 (0x555fa0644c65 in /opt/conda/bin/python3)
frame #24: _PyFunction_Vectorcall + 0x6f (0x555fa0654b1f in /opt/conda/bin/python3)
frame #25: _PyEval_EvalFrameDefault + 0x332 (0x555fa0644862 in /opt/conda/bin/python3)
frame #26: _PyFunction_Vectorcall + 0x6f (0x555fa0654b1f in /opt/conda/bin/python3)
frame #27: _PyEval_EvalFrameDefault + 0x332 (0x555fa0644862 in /opt/conda/bin/python3)
frame #28: _PyFunction_Vectorcall + 0x6f (0x555fa0654b1f in /opt/conda/bin/python3)
frame #29: _PyEval_EvalFrameDefault + 0x2ec3 (0x555fa06473f3 in /opt/conda/bin/python3)
frame #30: _PyFunction_Vectorcall + 0x6f (0x555fa0654b1f in /opt/conda/bin/python3)
frame #31: _PyEval_EvalFrameDefault + 0x2ec3 (0x555fa06473f3 in /opt/conda/bin/python3)
frame #32: _PyFunction_Vectorcall + 0x6f (0x555fa0654b1f in /opt/conda/bin/python3)
frame #33: _PyEval_EvalFrameDefault + 0x735 (0x555fa0644c65 in /opt/conda/bin/python3)
frame #34: _PyFunction_Vectorcall + 0x6f (0x555fa0654b1f in /opt/conda/bin/python3)
frame #35: _PyEval_EvalFrameDefault + 0x735 (0x555fa0644c65 in /opt/conda/bin/python3)
frame #36: _PyFunction_Vectorcall + 0x6f (0x555fa0654b1f in /opt/conda/bin/python3)
frame #37: _PyEval_EvalFrameDefault + 0x332 (0x555fa0644862 in /opt/conda/bin/python3)
frame #38: _PyFunction_Vectorcall + 0x6f (0x555fa0654b1f in /opt/conda/bin/python3)
frame #39: _PyEval_EvalFrameDefault + 0x13d0 (0x555fa0645900 in /opt/conda/bin/python3)
frame #40: <unknown function> + 0x1db6a2 (0x555fa06ef6a2 in /opt/conda/bin/python3)
frame #41: PyEval_EvalCode + 0x87 (0x555fa06ef5e7 in /opt/conda/bin/python3)
frame #42: <unknown function> + 0x20e3fc (0x555fa07223fc in /opt/conda/bin/python3)
frame #43: <unknown function> + 0x2092d4 (0x555fa071d2d4 in /opt/conda/bin/python3)
frame #44: PyRun_StringFlags + 0x7d (0x555fa071546d in /opt/conda/bin/python3)
frame #45: PyRun_SimpleStringFlags + 0x3d (0x555fa071528d in /opt/conda/bin/python3)
frame #46: Py_RunMain + 0x272 (0x555fa0714152 in /opt/conda/bin/python3)
frame #47: Py_BytesMain + 0x39 (0x555fa06e1dc9 in /opt/conda/bin/python3)
frame #48: <unknown function> + 0x29d90 (0x7f7e2234cd90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #49: __libc_start_main + 0x80 (0x7f7e2234ce40 in /lib/x86_64-linux-gnu/libc.so.6)
frame #50: <unknown function> + 0x1cdcc1 (0x555fa06e1cc1 in /opt/conda/bin/python3)
what():
CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /workspace/framework/mcPytorch/c10/cuda/CUDAException.cpp:42 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x88 (0x7f25beb928f8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x177d3 (0x7f25bef657d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::SetDevice(signed char, bool) + 0x72 (0x7f25befad6f2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x30207 (0x7f25bef7e207 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0x7807b5 (0x7f2598f1c7b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x44d0f3d (0x7f259cc6cf3d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: at::_ops::_local_scalar_dense::redispatch(c10::DispatchKeySet, at::Tensor const&) + 0x6b (0x7f25aaa9ef4b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x4d63cf0 (0x7f25ad236cf0 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x4d63d70 (0x7f25ad236d70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::_ops::_local_scalar_dense::call(at::Tensor const&) + 0x167 (0x7f25aab62a67 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::native::item(at::Tensor const&) + 0xd3 (0x7f25aa25ea93 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x2e0168d (0x7f25ab2d468d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #12: at::_ops::item::call(at::Tensor const&) + 0x167 (0x7f25aa99fc77 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x5151f6 (0x7f25ba39b1f6 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #14: <unknown function> + 0x14357c (0x5639270e657c in /opt/conda/bin/python3)
frame #15: _PyEval_EvalFrameDefault + 0x5ebb (0x5639270d93eb in /opt/conda/bin/python3)
frame #16: <unknown function> + 0x14c6e1 (0x5639270ef6e1 in /opt/conda/bin/python3)
frame #17: <unknown function> + 0x1d99b6 (0x56392717c9b6 in /opt/conda/bin/python3)
frame #18: PyObject_CallMethod + 0xcd (0x56392718158d in /opt/conda/bin/python3)
frame #19: torch::handle_torch_function_no_python_arg_parser(c10::ArrayRef<_object>, _object, _object, char const, _object, char const, torch::TorchFunctionName) + 0x1a3c (0x7f25bab346fc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #20: <unknown function> + 0xcaf0fd (0x7f25bab350fd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #21: <unknown function> + 0x515165 (0x7f25ba39b165 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #22: <unknown function> + 0x14357c (0x5639270e657c in /opt/conda/bin/python3)
frame #23: _PyEval_EvalFrameDefault + 0x735 (0x5639270d3c65 in /opt/conda/bin/python3)
frame #24: _PyFunction_Vectorcall + 0x6f (0x5639270e3b1f in /opt/conda/bin/python3)
frame #25: _PyEval_EvalFrameDefault + 0x332 (0x5639270d3862 in /opt/conda/bin/python3)
frame #26: _PyFunction_Vectorcall + 0x6f (0x5639270e3b1f in /opt/conda/bin/python3)
frame #27: _PyEval_EvalFrameDefault + 0x332 (0x5639270d3862 in /opt/conda/bin/python3)
frame #28: _PyFunction_Vectorcall + 0x6f (0x5639270e3b1f in /opt/conda/bin/python3)
frame #29: _PyEval_EvalFrameDefault + 0x2ec3 (0x5639270d63f3 in /opt/conda/bin/python3)
frame #30: _PyFunction_Vectorcall + 0x6f (0x5639270e3b1f in /opt/conda/bin/python3)
frame #31: _PyEval_EvalFrameDefault + 0x2ec3 (0x5639270d63f3 in /opt/conda/bin/python3)
frame #32: _PyFunction_Vectorcall + 0x6f (0x5639270e3b1f in /opt/conda/bin/python3)
frame #33: _PyEval_EvalFrameDefault + 0x735 (0x5639270d3c65 in /opt/conda/bin/python3)
frame #34: _PyFunction_Vectorcall + 0x6f (0x5639270e3b1f in /opt/conda/bin/python3)
frame #35: _PyEval_EvalFrameDefault + 0x735 (0x5639270d3c65 in /opt/conda/bin/python3)
frame #36: _PyFunction_Vectorcall + 0x6f (0x5639270e3b1f in /opt/conda/bin/python3)
frame #37: _PyEval_EvalFrameDefault + 0x332 (0x5639270d3862 in /opt/conda/bin/python3)
frame #38: _PyFunction_Vectorcall + 0x6f (0x5639270e3b1f in /opt/conda/bin/python3)
frame #39: _PyEval_EvalFrameDefault + 0x13d0 (0x5639270d4900 in /opt/conda/bin/python3)
frame #40: <unknown function> + 0x1db6a2 (0x56392717e6a2 in /opt/conda/bin/python3)
frame #41: PyEval_EvalCode + 0x87 (0x56392717e5e7 in /opt/conda/bin/python3)
frame #42: <unknown function> + 0x20e3fc (0x5639271b13fc in /opt/conda/bin/python3)
frame #43: <unknown function> + 0x2092d4 (0x5639271ac2d4 in /opt/conda/bin/python3)
frame #44: PyRun_StringFlags + 0x7d (0x5639271a446d in /opt/conda/bin/python3)
frame #45: PyRun_SimpleStringFlags + 0x3d (0x5639271a428d in /opt/conda/bin/python3)
frame #46: Py_RunMain + 0x272 (0x5639271a3152 in /opt/conda/bin/python3)
frame #47: Py_BytesMain + 0x39 (0x563927170dc9 in /opt/conda/bin/python3)
frame #48: <unknown function> + 0x29d90 (0x7f25c084fd90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #49: __libc_start_main + 0x80 (0x7f25c084fe40 in /lib/x86_64-linux-gnu/libc.so.6)
frame #50: <unknown function> + 0x1cdcc1 (0x563927170cc1 in /opt/conda/bin/python3)

what(): CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /workspace/framework/mcPytorch/c10/cuda/CUDAException.cpp:42 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x88 (0x7f3d075928f8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x177d3 (0x7f3d079657d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::SetDevice(signed char, bool) + 0x72 (0x7f3d079ad6f2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x30207 (0x7f3d0797e207 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0x7807b5 (0x7f3ce191c7b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x44d0f3d (0x7f3ce566cf3d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: at::_ops::_local_scalar_dense::redispatch(c10::DispatchKeySet, at::Tensor const&) + 0x6b (0x7f3cf349ef4b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x4d63cf0 (0x7f3cf5c36cf0 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x4d63d70 (0x7f3cf5c36d70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::_ops::_local_scalar_dense::call(at::Tensor const&) + 0x167 (0x7f3cf3562a67 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::native::item(at::Tensor const&) + 0xd3 (0x7f3cf2c5ea93 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x2e0168d (0x7f3cf3cd468d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #12: at::_ops::item::call(at::Tensor const&) + 0x167 (0x7f3cf339fc77 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x5151f6 (0x7f3d02d9b1f6 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #14: <unknown function> + 0x14357c (0x563ed24b157c in /opt/conda/bin/python3)
frame #15: _PyEval_EvalFrameDefault + 0x5ebb (0x563ed24a43eb in /opt/conda/bin/python3)
frame #16: <unknown function> + 0x14c6e1 (0x563ed24ba6e1 in /opt/conda/bin/python3)
frame #17: <unknown function> + 0x1d99b6 (0x563ed25479b6 in /opt/conda/bin/python3)
frame #18: PyObject_CallMethod + 0xcd (0x563ed254c58d in /opt/conda/bin/python3)
frame #19: torch::handle_torch_function_no_python_arg_parser(c10::ArrayRef<_object>, _object, _object, char const, _object, char const, torch::TorchFunctionName) + 0x1a3c (0x7f3d035346fc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #20: <unknown function> + 0xcaf0fd (0x7f3d035350fd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #21: <unknown function> + 0x515165 (0x7f3d02d9b165 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #22: <unknown function> + 0x14357c (0x563ed24b157c in /opt/conda/bin/python3)
frame #23: _PyEval_EvalFrameDefault + 0x735 (0x563ed249ec65 in /opt/conda/bin/python3)
frame #24: _PyFunction_Vectorcall + 0x6f (0x563ed24aeb1f in /opt/conda/bin/python3)
frame #25: _PyEval_EvalFrameDefault + 0x332 (0x563ed249e862 in /opt/conda/bin/python3)
frame #26: _PyFunction_Vectorcall + 0x6f (0x563ed24aeb1f in /opt/conda/bin/python3)
frame #27: _PyEval_EvalFrameDefault + 0x332 (0x563ed249e862 in /opt/conda/bin/python3)
frame #28: _PyFunction_Vectorcall + 0x6f (0x563ed24aeb1f in /opt/conda/bin/python3)
frame #29: _PyEval_EvalFrameDefault + 0x2ec3 (0x563ed24a13f3 in /opt/conda/bin/python3)
frame #30: _PyFunction_Vectorcall + 0x6f (0x563ed24aeb1f in /opt/conda/bin/python3)
frame #31: _PyEval_EvalFrameDefault + 0x2ec3 (0x563ed24a13f3 in /opt/conda/bin/python3)
frame #32: _PyFunction_Vectorcall + 0x6f (0x563ed24aeb1f in /opt/conda/bin/python3)
frame #33: _PyEval_EvalFrameDefault + 0x735 (0x563ed249ec65 in /opt/conda/bin/python3)
frame #34: _PyFunction_Vectorcall + 0x6f (0x563ed24aeb1f in /opt/conda/bin/python3)
frame #35: _PyEval_EvalFrameDefault + 0x735 (0x563ed249ec65 in /opt/conda/bin/python3)
frame #36: _PyFunction_Vectorcall + 0x6f (0x563ed24aeb1f in /opt/conda/bin/python3)
frame #37: _PyEval_EvalFrameDefault + 0x332 (0x563ed249e862 in /opt/conda/bin/python3)
frame #38: _PyFunction_Vectorcall + 0x6f (0x563ed24aeb1f in /opt/conda/bin/python3)
frame #39: _PyEval_EvalFrameDefault + 0x13d0 (0x563ed249f900 in /opt/conda/bin/python3)
frame #40: <unknown function> + 0x1db6a2 (0x563ed25496a2 in /opt/conda/bin/python3)
frame #41: PyEval_EvalCode + 0x87 (0x563ed25495e7 in /opt/conda/bin/python3)
frame #42: <unknown function> + 0x20e3fc (0x563ed257c3fc in /opt/conda/bin/python3)
frame #43: <unknown function> + 0x2092d4 (0x563ed25772d4 in /opt/conda/bin/python3)
frame #44: PyRun_StringFlags + 0x7d (0x563ed256f46d in /opt/conda/bin/python3)
frame #45: PyRun_SimpleStringFlags + 0x3d (0x563ed256f28d in /opt/conda/bin/python3)
frame #46: Py_RunMain + 0x272 (0x563ed256e152 in /opt/conda/bin/python3)
frame #47: Py_BytesMain + 0x39 (0x563ed253bdc9 in /opt/conda/bin/python3)
frame #48: <unknown function> + 0x29d90 (0x7f3d0921cd90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #49: __libc_start_main + 0x80 (0x7f3d0921ce40 in /lib/x86_64-linux-gnu/libc.so.6)
frame #50: <unknown function> + 0x1cdcc1 (0x563ed253bcc1 in /opt/conda/bin/python3)

what(): CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /workspace/framework/mcPytorch/c10/cuda/CUDAException.cpp:42 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x88 (0x7fbec5d928f8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x177d3 (0x7fbec6cfb7d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::SetDevice(signed char, bool) + 0x72 (0x7fbec6d436f2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x30207 (0x7fbec6d14207 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0x7807b5 (0x7fbe9fd1c7b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x44d0f3d (0x7fbea3a6cf3d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: at::_ops::_local_scalar_dense::redispatch(c10::DispatchKeySet, at::Tensor const&) + 0x6b (0x7fbeb189ef4b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x4d63cf0 (0x7fbeb4036cf0 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x4d63d70 (0x7fbeb4036d70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::_ops::_local_scalar_dense::call(at::Tensor const&) + 0x167 (0x7fbeb1962a67 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::native::item(at::Tensor const&) + 0xd3 (0x7fbeb105ea93 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x2e0168d (0x7fbeb20d468d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #12: at::_ops::item::call(at::Tensor const&) + 0x167 (0x7fbeb179fc77 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x5151f6 (0x7fbec119b1f6 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #14: <unknown function> + 0x14357c (0x55b2a0eef57c in /opt/conda/bin/python3)
frame #15: _PyEval_EvalFrameDefault + 0x5ebb (0x55b2a0ee23eb in /opt/conda/bin/python3)
frame #16: <unknown function> + 0x14c6e1 (0x55b2a0ef86e1 in /opt/conda/bin/python3)
frame #17: <unknown function> + 0x1d99b6 (0x55b2a0f859b6 in /opt/conda/bin/python3)
frame #18: PyObject_CallMethod + 0xcd (0x55b2a0f8a58d in /opt/conda/bin/python3)
frame #19: torch::handle_torch_function_no_python_arg_parser(c10::ArrayRef<_object>, _object, _object, char const, _object, char const, torch::TorchFunctionName) + 0x1a3c (0x7fbec19346fc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #20: <unknown function> + 0xcaf0fd (0x7fbec19350fd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #21: <unknown function> + 0x515165 (0x7fbec119b165 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #22: <unknown function> + 0x14357c (0x55b2a0eef57c in /opt/conda/bin/python3)
frame #23: _PyEval_EvalFrameDefault + 0x735 (0x55b2a0edcc65 in /opt/conda/bin/python3)
frame #24: _PyFunction_Vectorcall + 0x6f (0x55b2a0eecb1f in /opt/conda/bin/python3)
frame #25: _PyEval_EvalFrameDefault + 0x332 (0x55b2a0edc862 in /opt/conda/bin/python3)
frame #26: _PyFunction_Vectorcall + 0x6f (0x55b2a0eecb1f in /opt/conda/bin/python3)
frame #27: _PyEval_EvalFrameDefault + 0x332 (0x55b2a0edc862 in /opt/conda/bin/python3)
frame #28: _PyFunction_Vectorcall + 0x6f (0x55b2a0eecb1f in /opt/conda/bin/python3)
frame #29: _PyEval_EvalFrameDefault + 0x2ec3 (0x55b2a0edf3f3 in /opt/conda/bin/python3)
frame #30: _PyFunction_Vectorcall + 0x6f (0x55b2a0eecb1f in /opt/conda/bin/python3)
frame #31: _PyEval_EvalFrameDefault + 0x2ec3 (0x55b2a0edf3f3 in /opt/conda/bin/python3)
frame #32: _PyFunction_Vectorcall + 0x6f (0x55b2a0eecb1f in /opt/conda/bin/python3)
frame #33: _PyEval_EvalFrameDefault + 0x735 (0x55b2a0edcc65 in /opt/conda/bin/python3)
frame #34: _PyFunction_Vectorcall + 0x6f (0x55b2a0eecb1f in /opt/conda/bin/python3)
frame #35: _PyEval_EvalFrameDefault + 0x735 (0x55b2a0edcc65 in /opt/conda/bin/python3)
frame #36: _PyFunction_Vectorcall + 0x6f (0x55b2a0eecb1f in /opt/conda/bin/python3)
frame #37: _PyEval_EvalFrameDefault + 0x332 (0x55b2a0edc862 in /opt/conda/bin/python3)
frame #38: _PyFunction_Vectorcall + 0x6f (0x55b2a0eecb1f in /opt/conda/bin/python3)
frame #39: _PyEval_EvalFrameDefault + 0x13d0 (0x55b2a0edd900 in /opt/conda/bin/python3)
frame #40: <unknown function> + 0x1db6a2 (0x55b2a0f876a2 in /opt/conda/bin/python3)
frame #41: PyEval_EvalCode + 0x87 (0x55b2a0f875e7 in /opt/conda/bin/python3)
frame #42: <unknown function> + 0x20e3fc (0x55b2a0fba3fc in /opt/conda/bin/python3)
frame #43: <unknown function> + 0x2092d4 (0x55b2a0fb52d4 in /opt/conda/bin/python3)
frame #44: PyRun_StringFlags + 0x7d (0x55b2a0fad46d in /opt/conda/bin/python3)
frame #45: PyRun_SimpleStringFlags + 0x3d (0x55b2a0fad28d in /opt/conda/bin/python3)
frame #46: Py_RunMain + 0x272 (0x55b2a0fac152 in /opt/conda/bin/python3)
frame #47: Py_BytesMain + 0x39 (0x55b2a0f79dc9 in /opt/conda/bin/python3)
frame #48: <unknown function> + 0x29d90 (0x7fbec76a9d90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #49: __libc_start_main + 0x80 (0x7fbec76a9e40 in /lib/x86_64-linux-gnu/libc.so.6)
frame #50: <unknown function> + 0x1cdcc1 (0x55b2a0f79cc1 in /opt/conda/bin/python3)

what(): CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /workspace/framework/mcPytorch/c10/cuda/CUDAException.cpp:42 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x88 (0x7f3549b928f8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x177d3 (0x7f354aae87d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::SetDevice(signed char, bool) + 0x72 (0x7f354ab306f2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x30207 (0x7f354ab01207 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0x7807b5 (0x7f3523b1c7b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x44d0f3d (0x7f352786cf3d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: at::_ops::_local_scalar_dense::redispatch(c10::DispatchKeySet, at::Tensor const&) + 0x6b (0x7f353569ef4b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x4d63cf0 (0x7f3537e36cf0 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x4d63d70 (0x7f3537e36d70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::_ops::_local_scalar_dense::call(at::Tensor const&) + 0x167 (0x7f3535762a67 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::native::item(at::Tensor const&) + 0xd3 (0x7f3534e5ea93 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x2e0168d (0x7f3535ed468d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #12: at::_ops::item::call(at::Tensor const&) + 0x167 (0x7f353559fc77 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x5151f6 (0x7f3544f9b1f6 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #14: <unknown function> + 0x14357c (0x55cfd135557c in /opt/conda/bin/python3)
frame #15: _PyEval_EvalFrameDefault + 0x5ebb (0x55cfd13483eb in /opt/conda/bin/python3)
frame #16: <unknown function> + 0x14c6e1 (0x55cfd135e6e1 in /opt/conda/bin/python3)
frame #17: <unknown function> + 0x1d99b6 (0x55cfd13eb9b6 in /opt/conda/bin/python3)
frame #18: PyObject_CallMethod + 0xcd (0x55cfd13f058d in /opt/conda/bin/python3)
frame #19: torch::handle_torch_function_no_python_arg_parser(c10::ArrayRef<_object>, _object, _object, char const, _object, char const, torch::TorchFunctionName) + 0x1a3c (0x7f35457346fc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #20: <unknown function> + 0xcaf0fd (0x7f35457350fd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #21: <unknown function> + 0x515165 (0x7f3544f9b165 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #22: <unknown function> + 0x14357c (0x55cfd135557c in /opt/conda/bin/python3)
frame #23: _PyEval_EvalFrameDefault + 0x735 (0x55cfd1342c65 in /opt/conda/bin/python3)
frame #24: _PyFunction_Vectorcall + 0x6f (0x55cfd1352b1f in /opt/conda/bin/python3)
frame #25: _PyEval_EvalFrameDefault + 0x332 (0x55cfd1342862 in /opt/conda/bin/python3)
frame #26: _PyFunction_Vectorcall + 0x6f (0x55cfd1352b1f in /opt/conda/bin/python3)
frame #27: _PyEval_EvalFrameDefault + 0x332 (0x55cfd1342862 in /opt/conda/bin/python3)
frame #28: _PyFunction_Vectorcall + 0x6f (0x55cfd1352b1f in /opt/conda/bin/python3)
frame #29: _PyEval_EvalFrameDefault + 0x2ec3 (0x55cfd13453f3 in /opt/conda/bin/python3)
frame #30: _PyFunction_Vectorcall + 0x6f (0x55cfd1352b1f in /opt/conda/bin/python3)
frame #31: _PyEval_EvalFrameDefault + 0x2ec3 (0x55cfd13453f3 in /opt/conda/bin/python3)
frame #32: _PyFunction_Vectorcall + 0x6f (0x55cfd1352b1f in /opt/conda/bin/python3)
frame #33: _PyEval_EvalFrameDefault + 0x735 (0x55cfd1342c65 in /opt/conda/bin/python3)
frame #34: _PyFunction_Vectorcall + 0x6f (0x55cfd1352b1f in /opt/conda/bin/python3)
frame #35: _PyEval_EvalFrameDefault + 0x735 (0x55cfd1342c65 in /opt/conda/bin/python3)
frame #36: _PyFunction_Vectorcall + 0x6f (0x55cfd1352b1f in /opt/conda/bin/python3)
frame #37: _PyEval_EvalFrameDefault + 0x332 (0x55cfd1342862 in /opt/conda/bin/python3)
frame #38: _PyFunction_Vectorcall + 0x6f (0x55cfd1352b1f in /opt/conda/bin/python3)
frame #39: _PyEval_EvalFrameDefault + 0x13d0 (0x55cfd1343900 in /opt/conda/bin/python3)
frame #40: <unknown function> + 0x1db6a2 (0x55cfd13ed6a2 in /opt/conda/bin/python3)
frame #41: PyEval_EvalCode + 0x87 (0x55cfd13ed5e7 in /opt/conda/bin/python3)
frame #42: <unknown function> + 0x20e3fc (0x55cfd14203fc in /opt/conda/bin/python3)
frame #43: <unknown function> + 0x2092d4 (0x55cfd141b2d4 in /opt/conda/bin/python3)
frame #44: PyRun_StringFlags + 0x7d (0x55cfd141346d in /opt/conda/bin/python3)
frame #45: PyRun_SimpleStringFlags + 0x3d (0x55cfd141328d in /opt/conda/bin/python3)
frame #46: Py_RunMain + 0x272 (0x55cfd1412152 in /opt/conda/bin/python3)
frame #47: Py_BytesMain + 0x39 (0x55cfd13dfdc9 in /opt/conda/bin/python3)
frame #48: <unknown function> + 0x29d90 (0x7f354b496d90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #49: __libc_start_main + 0x80 (0x7f354b496e40 in /lib/x86_64-linux-gnu/libc.so.6)
frame #50: <unknown function> + 0x1cdcc1 (0x55cfd13dfcc1 in /opt/conda/bin/python3)

what(): CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /workspace/framework/mcPytorch/c10/cuda/CUDAException.cpp:42 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x88 (0x7fe219f928f8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x177d3 (0x7fe21a2e97d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::SetDevice(signed char, bool) + 0x72 (0x7fe21a3316f2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x30207 (0x7fe21a302207 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0x7807b5 (0x7fe1f431c7b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x44d0f3d (0x7fe1f806cf3d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: at::_ops::_local_scalar_dense::redispatch(c10::DispatchKeySet, at::Tensor const&) + 0x6b (0x7fe205e9ef4b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x4d63cf0 (0x7fe208636cf0 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x4d63d70 (0x7fe208636d70 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::_ops::_local_scalar_dense::call(at::Tensor const&) + 0x167 (0x7fe205f62a67 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::native::item(at::Tensor const&) + 0xd3 (0x7fe20565ea93 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x2e0168d (0x7fe2066d468d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #12: at::_ops::item::call(at::Tensor const&) + 0x167 (0x7fe205d9fc77 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x5151f6 (0x7fe21579b1f6 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #14: <unknown function> + 0x14357c (0x55b80a50357c in /opt/conda/bin/python3)
frame #15: _PyEval_EvalFrameDefault + 0x5ebb (0x55b80a4f63eb in /opt/conda/bin/python3)
frame #16: <unknown function> + 0x14c6e1 (0x55b80a50c6e1 in /opt/conda/bin/python3)
frame #17: <unknown function> + 0x1d99b6 (0x55b80a5999b6 in /opt/conda/bin/python3)
frame #18: PyObject_CallMethod + 0xcd (0x55b80a59e58d in /opt/conda/bin/python3)
frame #19: torch::handle_torch_function_no_python_arg_parser(c10::ArrayRef<_object>, _object, _object, char const, _object, char const, torch::TorchFunctionName) + 0x1a3c (0x7fe215f346fc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #20: <unknown function> + 0xcaf0fd (0x7fe215f350fd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #21: <unknown function> + 0x515165 (0x7fe21579b165 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #22: <unknown function> + 0x14357c (0x55b80a50357c in /opt/conda/bin/python3)
frame #23: _PyEval_EvalFrameDefault + 0x735 (0x55b80a4f0c65 in /opt/conda/bin/python3)
frame #24: _PyFunction_Vectorcall + 0x6f (0x55b80a500b1f in /opt/conda/bin/python3)
frame #25: _PyEval_EvalFrameDefault + 0x332 (0x55b80a4f0862 in /opt/conda/bin/python3)
frame #26: _PyFunction_Vectorcall + 0x6f (0x55b80a500b1f in /opt/conda/bin/python3)
frame #27: _PyEval_EvalFrameDefault + 0x332 (0x55b80a4f0862 in /opt/conda/bin/python3)
frame #28: _PyFunction_Vectorcall + 0x6f (0x55b80a500b1f in /opt/conda/bin/python3)
frame #29: _PyEval_EvalFrameDefault + 0x2ec3 (0x55b80a4f33f3 in /opt/conda/bin/python3)
frame #30: _PyFunction_Vectorcall + 0x6f (0x55b80a500b1f in /opt/conda/bin/python3)
frame #31: _PyEval_EvalFrameDefault + 0x2ec3 (0x55b80a4f33f3 in /opt/conda/bin/python3)
frame #32: _PyFunction_Vectorcall + 0x6f (0x55b80a500b1f in /opt/conda/bin/python3)
frame #33: _PyEval_EvalFrameDefault + 0x735 (0x55b80a4f0c65 in /opt/conda/bin/python3)
frame #34: _PyFunction_Vectorcall + 0x6f (0x55b80a500b1f in /opt/conda/bin/python3)
frame #35: _PyEval_EvalFrameDefault + 0x735 (0x55b80a4f0c65 in /opt/conda/bin/python3)
frame #36: _PyFunction_Vectorcall + 0x6f (0x55b80a500b1f in /opt/conda/bin/python3)
frame #37: _PyEval_EvalFrameDefault + 0x332 (0x55b80a4f0862 in /opt/conda/bin/python3)
frame #38: _PyFunction_Vectorcall + 0x6f (0x55b80a500b1f in /opt/conda/bin/python3)
frame #39: _PyEval_EvalFrameDefault + 0x13d0 (0x55b80a4f1900 in /opt/conda/bin/python3)
frame #40: <unknown function> + 0x1db6a2 (0x55b80a59b6a2 in /opt/conda/bin/python3)
frame #41: PyEval_EvalCode + 0x87 (0x55b80a59b5e7 in /opt/conda/bin/python3)
frame #42: <unknown function> + 0x20e3fc (0x55b80a5ce3fc in /opt/conda/bin/python3)
frame #43: <unknown function> + 0x2092d4 (0x55b80a5c92d4 in /opt/conda/bin/python3)
frame #44: PyRun_StringFlags + 0x7d (0x55b80a5c146d in /opt/conda/bin/python3)
frame #45: PyRun_SimpleStringFlags + 0x3d (0x55b80a5c128d in /opt/conda/bin/python3)
frame #46: Py_RunMain + 0x272 (0x55b80a5c0152 in /opt/conda/bin/python3)
frame #47: Py_BytesMain + 0x39 (0x55b80a58ddc9 in /opt/conda/bin/python3)
frame #48: <unknown function> + 0x29d90 (0x7fe21bb43d90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #49: __libc_start_main + 0x80 (0x7fe21bb43e40 in /lib/x86_64-linux-gnu/libc.so.6)
frame #50: <unknown function> + 0x1cdcc1 (0x55b80a58dcc1 in /opt/conda/bin/python3)

what(): CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /workspace/framework/mcPytorch/c10/cuda/CUDAException.cpp:42 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x88 (0x7f0e27b928f8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x177d3 (0x7f0e27f657d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::SetDevice(signed char, bool) + 0x72 (0x7f0e27fad6f2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x30207 (0x7f0e27f7e207 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
See post chevron_right

FengKuangDeShiTou
Members

求助：metax C550上使用vllm部署Qwen3.5-122B-A10B-W8A8模型的deepep功能报错已解决 2026年7月24日 11:01

好的，sglang和vllm针对deepep的底层库是一样的吗？
See post chevron_right

FengKuangDeShiTou
Members

求助：metax C550上使用vllm部署Qwen3.5-122B-A10B-W8A8模型的deepep功能报错已解决 2026年7月24日 10:29

请问目前官网开源的镜像库中是否有支持deepep功能的？
See post chevron_right

FengKuangDeShiTou
Members

求助：metax C550上使用vllm部署Qwen3.5-122B-A10B-W8A8模型的deepep功能报错已解决 2026年7月24日 10:18

你好，能否提供一个deepep功能可用的vllm镜像库？
See post chevron_right

FengKuangDeShiTou
Members

求助：metax C550上使用vllm部署Qwen3.5-122B-A10B-W8A8模型的deepep功能报错已解决 2026年7月23日 20:17

配套使用的镜像：vllm-metax:0.20.0-maca.ai3.7.0.107-torch2.8-py310-ubuntu22.04-amd64，启动命令： vllm serve /workspace/models/Qwen3.5-122B-A10B-W8A8 --host 0.0.0.0 --port 8088 --dtype bfloat16 -tp 2 -pp 1 -dp 4 --max-model-len 262144 --enable-expert-parallel --trust-remote-code --gpu-memory-utilization 0.9 --all2all-backend deepep_high_throughput，在单节点部署。请分析一下，是否支持deepep的功能

错误信息：(Worker_DP0_TP0_EP0 pid=25858) INFO 07-23 12:04:52 [backends.py:1069] Using cache directory: /root/.cache/vllm/torch_compile_cache/56c16fdb9d/rank_0_0/backbone for vLLM's torch.compile
(Worker_DP0_TP0_EP0 pid=25858) INFO 07-23 12:04:52 [backends.py:1128] Dynamo bytecode transform time: 9.05 s
(Worker_DP0_TP0_EP0 pid=25858) INFO 07-23 12:04:53 [fix_compilation_backend.py:223] Directly load the compiled graph(s) for compile range (1, 2048) from the cache, took 1.226 s
(Worker_DP0_TP0_EP0 pid=25858) INFO 07-23 12:05:07 [fused_moe.py:1335] User defined config folder: /opt/conda/lib/python3.10/site-packages/vllm_metax/model_executor/layers/fused_moe/configs/H=3072
(Worker_DP0_TP0_EP0 pid=25858) WARNING 07-23 12:05:07 [fused_moe.py:1366] Using default MoE config. Performance might be sub-optimal! Config file not found at /opt/conda/lib/python3.10/site-packages/v llm_metax/model_executor/layers/fused_moe/configs/H=3072/H=3072,E=32,N=1024,device_name=MXC550,dtype=int8_w8a8.json, /opt/conda/lib/python3.10/site-packages/vllm_metax/model_executor/layers/fused_moe/ configs/E=32,N=1024,device_name=MXC550,dtype=int8_w8a8.json
[12:05:08.059][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.059][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MXC][E]xnack(0x8): kernel causes atu address translation error
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=2,queueID=1!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7ef6d8000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=1,queueID=1!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7ef4d8000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=2,queueID=1!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7f28d8000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=2,queueID=1!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7fbba8000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=2,queueID=1!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7f53b8000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1000: Node ID is 7, Wave ID is 32546
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.060][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=1,queueID=1!
[12:05:08.060][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7f31a0000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=2,queueID=1!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7f249c000c70!
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1000: Node ID is 5, Wave ID is 32589
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1000: Node ID is 4, Wave ID is 32687
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.060][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.060][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.060][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.060][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1000: Node ID is 3, Wave ID is 32549
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.060][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.060][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1000: Node ID is 8, Wave ID is 32536
[12:05:08.060][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.060][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.060][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.061][MCR][E]mx_trapProcess.cpp :128 : trapping pipeID=2,queueID=1!
[12:05:08.061][MCR][E]mx_trapProcess.cpp :129 : trapping virtualDevice=0x7f23e4000c70!
[12:05:08.061][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.061][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.061][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.061][MCR][E]mx_trapProcess.cpp :1000: Node ID is 9, Wave ID is 32535
[12:05:08.061][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.061][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.061][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.061][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.061][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.062][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.062][MCR][E]mx_trapProcess.cpp :1000: Node ID is 2, Wave ID is -2
[12:05:08.062][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.062][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.062][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.062][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.067][MCR][E]mx_trapProcess.cpp :1000: Node ID is 6, Wave ID is 32490
[12:05:08.067][MCR][E]mx_trapProcess.cpp :1023: debug info: 0xd1,0x00,0x14,0x14,0x7c,0x00,0x00,0x00,0xe0,0x23,0x00,0x00,0x40,0x00,0x00,0x00,
[12:05:08.067][MCR][E]mx_device.cpp :10101: trap:precise positioning.
[12:05:08.067][MCR][E]mx_device.cpp :10106: trapping: kernelName: _ZN7deep_ep9intranode7combineI15__maca_bfloat16Li8ELi768EEEvPT_PfPKS3_PKfPKiSB_SB_PiiiiiPPviii ,commandIndex: 9 , trapType: Xnack Error/ATU Fault(0x8)
[12:05:08.068][MCR][E]mx_trapProcess.cpp :914 : Xnack(0x8) exception happened in the shader, the mcruntime api will be disabled
[12:05:08.071][MCR][E]mc_runtime_api.cpp :215 : 25858: [7efe146fa740] mcGetDevice: Returned mcErrorIllegalAddress
[12:05:08.072][MCR][E]mc_runtime_api.cpp :215 : 25858: [7efe146fa740] mcGetDevice: Returned mcErrorIllegalAddress
[12:05:08.075][MCR][E]mc_runtime_api.cpp :215 : 25858: [7efe146fa740] mcGetDevice: Returned mcErrorIllegalAddress
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] WorkerProc hit an exception.
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] Traceback (most recent call last):
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/tmp/torchinductor_root/kt/cktpgzzi3qihf2snuow3makty3juo3d2rzuerwbazphoytzemhta.py", line 949, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] buf17 = torch.ops.vllm.moe_forward_shared.default(buf16, buf16, buf16, None, 'from_forward_context')
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_ops.py", line 829, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._op(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 125, in _moe_forward_shared
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return layer.runner._forward_impl(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 722, in _forward_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] shared_output, hidden_states = self._apply_quant_method(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 470, in _apply_quant_method
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] fused_out = self.quant_method.apply(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm_metax/quant_config/compressed_tensors_moe/compressed_tensors_moe_w8 a8_int8.py", line 163, in apply
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self.moe_kernel.apply(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1614, in a pply
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self.impl.apply(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1408, in a pply
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._finalize(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1307, in _ finalize
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] self._maybe_apply_shared_experts(shared_experts_input)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1110, in _ maybe_apply_shared_experts
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] self.shared_experts.apply(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/shared_experts.py", line 176 , in apply
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] self._output[self._output_idx] = self._layer(shared_experts_input)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._call_impl(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return forward_call(*args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_moe.py", line 116, in forward
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] out = self.act_fn(gate_up)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._call_impl(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return forward_call(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/custom_op.py", line 136, in forward
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._forward_method(*args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 736, in compile_wrapper
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return fn(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1495, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._torchdynamo_orig_callable(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1272, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] result = self._inner_convert(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 629, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return _compile(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1164, in _compile
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] raise InternalTorchDynamoError(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1111, in _compile
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] guarded_code = compile_inner(code, one_graph, hooks, transform)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_utils_internal.py", line 97, in wrapper_function
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return function(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/dynamo/convert_frame.py", line 793, in compile_inner
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return _compile_inner(code, one_graph, hooks, transform)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 832, in _compile_inner
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] out_code = transform_code_object(code, transform)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1424, in transform_code object
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] transformations(instructions, code_options)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 288, in _fn
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] torch.cuda.set_rng_state(cuda_rng_state)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/cuda/random.py", line 79, in set_rng_state
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] _lazy_call(cb)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py", line 348, in _lazy_call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] callable()
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/cuda/random.py", line 75, in cb
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] idx = current_device()
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py", line 1146, in current_device
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return torch._C._cuda_getDevice()
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] torch._dynamo.exc.InternalTorchDynamoError: AcceleratorError: CUDA error: an illegal memory access was encountered
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962]
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962]
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962]
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] During handling of the above exception, another exception occurred:
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962]
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] Traceback (most recent call last):
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 957, in worker_busy_loop
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] output = func(*args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return func(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 370, in determine_available_memory
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] self.model_runner.profile_run()
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5848, in profile_run
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] hidden_states, last_hidden_states = self._dummy_run(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return func(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5537, in _dummy_run
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] outputs = self.model(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._call_impl(*args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return forward_call(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_5.py", line 695, in forward
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] hidden_states = self.language_model.model(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 638, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] output = TorchCompileWithNoGuardsWrapper.call(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/compilation/wrapper.py", line 197, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._call_with_optional_nvtx_range(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/compilation/wrapper.py", line 76, in _call_with_optional_nvtx_range
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return callable_fn(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 736, in compile_wrapper
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return fn(*args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_next.py", line 495, in forward
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] def forward(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return fn(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/compilation/caching.py", line 215, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self.optimized_call(args, kwargs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "<string>", line 221, in execution_fn
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/compilation/piecewise_backend.py", line 380, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return range_entry.runnable(args)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/vllm/compilation/compiler_interface.py", line 433, in compiled_graph_wra pper
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] graph_output = inductor_compiled_graph(args)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/standalone_compile.py", line 62, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self._compiled_fn(args)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/standalone_compile.py", line 182, in <lambda>
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return CompiledArtifact(lambda args: compiled_fn(list(args)), None)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 384, in runtim e_wrapper
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] all_outs = call_func_at_runtime_with_args(
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runt ime_with_args
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] out = normalize_as_list(f(args))
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 556, in wrappe r
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return compiled_fn(runtime_args)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 584, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] return self.current_callable(inputs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/utils.py", line 2720, in run
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] out = model(new_inputs)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/tmp/torchinductor_root/kt/cktpgzzi3qihf2snuow3makty3juo3d2rzuerwbazphoytzemhta.py", line 906, in call
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] with torch.cuda._DeviceGuard(0):
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] File "/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py", line 525, in exit
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] self.idx = torch.cuda._maybe_exchange_device(self.prev_idx)
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] torch.AcceleratorError: CUDA error: an illegal memory access was encountered
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker_DP0_TP0_EP0 pid=25858) ERROR 07-23 12:05:08 [multiproc_executor.py:962] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.