之前部署一个模型,推理也没问题,但是升级显卡驱动到3.1.0.14,突然就不行了,推理启动有时候有问题,有时候能启动,但是不能推理,没有返回数据,有时候启动还报错,镜像版本是:cr.metax-tech.com/public-ai-release/maca/modelzoo.llm.vllm:maca.ai3.1.0.7-torch2.6-py310-ubuntu22.04-amd64, 会不会是因为镜像版本低于驱动版本造成的?如果是的话,去哪里下载最新的docker镜像:
[19:40:55.226][MXKW][E]queues.c :812 : [mxkwCreateQueueBlock]ioctl create queue block failed -1
[19:40:55.228][MXC][E]exception: DMAQueue create failed at mxkwCreateQueueBlock.
[19:40:55.229][MCR][E]mx_device.cpp :3544: Mxc copy from host to device failed with code 4104
[19:40:55.243][MXKW][E]queues.c :812 : [mxkwCreateQueueBlock]ioctl create queue block failed -1
[19:40:55.244][MXC][E]exception: DMAQueue create failed at mxkwCreateQueueBlock.
[19:40:55.244][MCR][E]mx_device.cpp :3637: Mxc copy from device to device failed with code 4104
[19:40:55.260][MXKW][E]queues.c :812 : [mxkwCreateQueueBlock]ioctl create queue block failed -1
[19:40:55.263][MXC][E]exception: DMAQueue create failed at mxkwCreateQueueBlock.
[19:40:55.263][MCR][E]mx_device.cpp :3544: Mxc copy from host to device failed with code 4104
[19:40:55.288][MXKW][E]queues.c :812 : [mxkwCreateQueueBlock]ioctl create queue block failed -1
[19:40:55.288][MXC][E]exception: DMAQueue create failed at mxkwCreateQueueBlock.
[19:40:55.289][MCR][E]mx_device.cpp :3544: Mxc copy from host to device failed with code 4104
[19:40:55.306][MXKW][E]queues.c :812 : [mxkwCreateQueueBlock]ioctl create queue block failed -1
[19:40:55.306][MCR][E]mx_device.cpp :1219: Device::acquireQueue: mxc_queue_acquire failed!
Traceback (most recent call last):
File "/opt/conda/bin/vllm", line 8, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/cli/main.py", line 54, in main
args.dispatch_function(args)
File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/cli/serve.py", line 52, in cmd
uvloop.run(run_server(args))
File "/opt/conda/lib/python3.10/site-packages/uvloop/init.py", line 82, in run
return loop.run_until_complete(wrapper())
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/opt/conda/lib/python3.10/site-packages/uvloop/init.py", line 61, in wrapper
return await main
File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 1791, in run_server
await run_server_worker(listen_address, sock, args, *uvicorn_kwargs)
File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 1811, in run_server_worker
async with build_async_engine_client(args, client_config) as engine_client:
File "/opt/conda/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 158, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/opt/conda/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 194, in build_async_engine_client_from_engine_args
async_llm = AsyncLLM.from_vllm_config(
File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 163, in from_vllm_config
return cls(
File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 117, in init
self.engine_core = EngineCoreClient.make_async_mp_client(
File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 98, in make_async_mp_client
return AsyncMPClient(client_args)
File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 677, in init
super().init(
File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 408, in init
with launch_core_engines(vllm_config, executor_class,
File "/opt/conda/lib/python3.10/contextlib.py", line 142, in exit
next(self.gen)
File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 697, in launch_core_engines
wait_for_engine_startup(
File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 750, in wait_for_engine_startup
raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_0': -11}
root@host:/workspace# mx-smi
mx-smi version: 2.2.8
=================== MetaX System Management Interface Log ===================
Timestamp : Thu Oct 23 19:43:59 2025
Attached GPUs : 1
+---------------------------------------------------------------------------------+
| MX-SMI 2.2.8 Kernel Mode Driver Version: 3.0.11 |
| MACA Version: 3.1.0.14 BIOS Version: unknown |
|------------------------------------+---------------------+----------------------+
| GPU NAME Persistence-M | Bus-id | GPU-Util sGPU-M |
| Temp Pwr:Usage/Cap Perf | Memory-Usage | GPU-State |
|====================================+=====================+======================|
| 0 MetaX C500 N/A | 0000:0c:00.0 | N/A Native |
| N/A NA / NA N/A | 858/65536 MiB | Not Available |
+------------------------------------+---------------------+----------------------+
+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| no process found |
+---------------------------------------------------------------------------------+