dmesg报错

我在我的一台8卡C500服务器，执行dmesg -T | grep -i err
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO mxc500_shader_error_queue_handler, handle the int err only first time, hw_queue:0x3
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO mxc500_shader_error_queue_handler, handle the int err only first time, hw_queue:0x3
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO mxc500_shader_error_queue_handler, handle the int err only first time, hw_queue:0x3
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO mxc500_shader_error_queue_handler, handle the int err only first time, hw_queue:0x3
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO mxc500_shader_error_queue_handler, handle the int err only first time, hw_queue:0x3
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO mxc500_shader_error_queue_handler, handle the int err only first time, hw_queue:0x3
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B4F00.D0.XCORE.INFO EID (0000:4f:00.0): 0x2102, shader exception, pasid 33195 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO mxc500_shader_error_queue_handler, handle the int err only first time, hw_queue:0x3
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5000.D0.XCORE.INFO EID (0000:50:00.0): 0x2102, shader exception, pasid 33194 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3C00.D0.XCORE.INFO EID (0000:3c:00.0): 0x2102, shader exception, pasid 33198 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B2E00.D0.XCORE.INFO EID (0000:2e:00.0): 0x2102, shader exception, pasid 33199 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B3F00.D0.XCORE.INFO EID (0000:3f:00.0): 0x2102, shader exception, pasid 33193 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5F00.D0.XCORE.INFO EID (0000:5f:00.0): 0x2102, shader exception, pasid 33196 error_type mem_viol(0x4)
[日 6月 21 14:37:36 2026] METAX.B5E00.D0.XCORE.INFO EID (0000:5e:00.0): 0x2102, shader exception, pasid 33197 error_type mem_viol(0x4)

一直重复刷新，这是个什么错误？下面还有一堆，全是这样的

尊敬的开发者您好，请详细描述您的操作过程

启动模型，模型命令：
cat docker-compose-qwen122-ray.yaml
version: '3.8'

services:
qwen3.5-122b-a10b:
image: cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.19.0-maca.ai3.5.3.502-torch2.8-py310-kylinv11-amd64
container_name: qwen3.5-122b-a10b
network_mode: host
shm_size: 200gb
privileged: true
devices:
- /dev/dri
- /dev/mxcd
- /dev/mem
group_add:
- video
security_opt:
- seccomp=unconfined
- apparmor=unconfined
ulimits:
memlock: -1
environment:
- GLOO_SOCKET_IFNAME=enp184s0f1np1
- MCCL_IB_HCA=rocep75s0f1
- MACA_DIRECT_DISPATCH=1
- MACA_SMALL_PAGESIZE_ENABLE=1
volumes:
- /usr/local/:/usr/local/
- /mnt/data/models:/mnt/data/models
command: >
/bin/bash -c "ray start --address='192.168.100.236:6379' --num-gpus=8 && tail -f /dev/null"
restart: always

主节点236是没有硬件报错，就是因为ray集群一直崩溃，排查到237辅节点有这个报错的

[一 6月 22 15:40:29 2026] MXCD.B3C00.D0.RINGBUF.ERROR EID (0000:3c:00.0): 0x2201, destroy ce ringbuf failed, logic rb_id:3, pipe_id:1
[一 6月 22 15:40:29 2026] MXCD.B3C00.D0.RINGBUF.ERROR results in a decrease in the number of available ringbufs, ret 1
[一 6月 22 15:41:03 2026] METAX.B3C00.D0.XCORE.ERROR EID (0000:3c:00.0): 0x2103, kill queue timeout, pipe 2, queue 3
[一 6月 22 15:41:03 2026] METAX.B2D00.D0.XCORE.ERROR gpu_id:28956, ctrl:0x106000
[一 6月 22 15:41:03 2026] METAX.B2E00.D0.XCORE.ERROR gpu_id:32860, ctrl:0x106000
[一 6月 22 15:41:03 2026] METAX.B3C00.D0.XCORE.ERROR gpu_id:11728, ctrl:0x106000
[一 6月 22 15:41:03 2026] METAX.B3F00.D0.XCORE.ERROR gpu_id:56464, ctrl:0x106000
[一 6月 22 15:41:03 2026] METAX.B4F00.D0.XCORE.ERROR gpu_id:14517, ctrl:0x106000
[一 6月 22 15:41:03 2026] METAX.B5000.D0.XCORE.ERROR gpu_id:253, ctrl:0x106000
[一 6月 22 15:41:03 2026] METAX.B5E00.D0.XCORE.ERROR gpu_id:25721, ctrl:0x106000
[一 6月 22 15:41:03 2026] METAX.B5F00.D0.XCORE.ERROR gpu_id:62648, ctrl:0x106000

还有这个错误也一直在重复

尊敬的开发者您好，请停止ray服务尝试，ray后端已不适配，切换MP分布式后端

vllm哪个版本开始不适配的？

尊敬的开发者您好，大概今年发布的版本，vllm-MetaX不适配ray后端

我用的是0.17.0的版本，应该不是ray导致的吧，现在我换了mp后端了，还是会崩溃，等下次崩溃我把日志贴出来，之前很正常，这两周模型使用过程中一直崩溃，不存在大量请求或者异常请求

尊敬的开发者您好，请单机分别跑模型，然后再测双机MCCL，IB Write打流定位问题