• Members 21 posts
    2025年10月23日 19:46

    之前部署一个模型,推理也没问题,但是升级显卡驱动到3.1.0.14,突然就不行了,推理启动有时候有问题,有时候能启动,但是不能推理,没有返回数据,有时候启动还报错,镜像版本是:cr.metax-tech.com/public-ai-release/maca/modelzoo.llm.vllm:maca.ai3.1.0.7-torch2.6-py310-ubuntu22.04-amd64, 会不会是因为镜像版本低于驱动版本造成的?如果是的话,去哪里下载最新的docker镜像:
    [19:40:55.226][MXKW][E]queues.c :812 : [mxkwCreateQueueBlock]ioctl create queue block failed -1
    [19:40:55.228][MXC][E]exception: DMAQueue create failed at mxkwCreateQueueBlock.
    [19:40:55.229][MCR][E]mx_device.cpp :3544: Mxc copy from host to device failed with code 4104
    [19:40:55.243][MXKW][E]queues.c :812 : [mxkwCreateQueueBlock]ioctl create queue block failed -1
    [19:40:55.244][MXC][E]exception: DMAQueue create failed at mxkwCreateQueueBlock.
    [19:40:55.244][MCR][E]mx_device.cpp :3637: Mxc copy from device to device failed with code 4104
    [19:40:55.260][MXKW][E]queues.c :812 : [mxkwCreateQueueBlock]ioctl create queue block failed -1
    [19:40:55.263][MXC][E]exception: DMAQueue create failed at mxkwCreateQueueBlock.
    [19:40:55.263][MCR][E]mx_device.cpp :3544: Mxc copy from host to device failed with code 4104
    [19:40:55.288][MXKW][E]queues.c :812 : [mxkwCreateQueueBlock]ioctl create queue block failed -1
    [19:40:55.288][MXC][E]exception: DMAQueue create failed at mxkwCreateQueueBlock.
    [19:40:55.289][MCR][E]mx_device.cpp :3544: Mxc copy from host to device failed with code 4104
    [19:40:55.306][MXKW][E]queues.c :812 : [mxkwCreateQueueBlock]ioctl create queue block failed -1
    [19:40:55.306][MCR][E]mx_device.cpp :1219: Device::acquireQueue: mxc_queue_acquire failed!
    Traceback (most recent call last):
    File "/opt/conda/bin/vllm", line 8, in <module>
    sys.exit(main())
    File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/cli/main.py", line 54, in main
    args.dispatch_function(args)
    File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/cli/serve.py", line 52, in cmd
    uvloop.run(run_server(args))
    File "/opt/conda/lib/python3.10/site-packages/uvloop/init.py", line 82, in run
    return loop.run_until_complete(wrapper())
    File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
    File "/opt/conda/lib/python3.10/site-packages/uvloop/init.py", line 61, in wrapper
    return await main
    File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 1791, in run_server
    await run_server_worker(listen_address, sock, args, *uvicorn_kwargs)
    File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 1811, in run_server_worker
    async with build_async_engine_client(args, client_config) as engine_client:
    File "/opt/conda/lib/python3.10/contextlib.py", line 199, in aenter
    return await anext(self.gen)
    File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 158, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
    File "/opt/conda/lib/python3.10/contextlib.py", line 199, in aenter
    return await anext(self.gen)
    File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 194, in build_async_engine_client_from_engine_args
    async_llm = AsyncLLM.from_vllm_config(
    File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 163, in from_vllm_config
    return cls(
    File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 117, in init
    self.engine_core = EngineCoreClient.make_async_mp_client(
    File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 98, in make_async_mp_client
    return AsyncMPClient(
    client_args)
    File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 677, in init
    super().init(
    File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 408, in init
    with launch_core_engines(vllm_config, executor_class,
    File "/opt/conda/lib/python3.10/contextlib.py", line 142, in exit
    next(self.gen)
    File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 697, in launch_core_engines
    wait_for_engine_startup(
    File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 750, in wait_for_engine_startup
    raise RuntimeError("Engine core initialization failed. "
    RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_0': -11}
    root@host:/workspace# mx-smi
    mx-smi version: 2.2.8

    =================== MetaX System Management Interface Log ===================
    Timestamp : Thu Oct 23 19:43:59 2025

    Attached GPUs : 1
    +---------------------------------------------------------------------------------+
    | MX-SMI 2.2.8 Kernel Mode Driver Version: 3.0.11 |
    | MACA Version: 3.1.0.14 BIOS Version: unknown |
    |------------------------------------+---------------------+----------------------+
    | GPU NAME Persistence-M | Bus-id | GPU-Util sGPU-M |
    | Temp Pwr:Usage/Cap Perf | Memory-Usage | GPU-State |
    |====================================+=====================+======================|
    | 0 MetaX C500 N/A | 0000:0c:00.0 | N/A Native |
    | N/A NA / NA N/A | 858/65536 MiB | Not Available |
    +------------------------------------+---------------------+----------------------+

    +---------------------------------------------------------------------------------+
    | Process: |
    | GPU PID Process Name GPU Memory |
    | Usage(MiB) |
    |=================================================================================|
    | no process found |
    +---------------------------------------------------------------------------------+

  • Members 139 posts
    2025年10月24日 11:07

    尊敬的开发者您好,根据mx-smi回显,您没有升级对应的固件,请您升级固件后再尝试。

  • arrow_forward

    Thread has been moved from 公共.

  • Members 21 posts
    2025年10月24日 12:16

    您好,请问,是从mx-smi哪里看出来我的固件有问题,我好下次知道怎么回事

  • Members 139 posts
    2025年10月24日 13:33

    尊敬的开发者您好,从mx-smi回显BIOS Version: unknown可以看出固件异常。

  • arrow_forward

    Thread has been moved from 解决中.