一、软硬件信息
1.服务器厂家: 未知
2.沐曦GPU型号:MX-SMI 2.3.4
3.操作系统内核版本:Ubuntu 24.04 (Linux 6.8.0-147-generic)
4.是否开启CPU虚拟化:是
5.mx-smi回显: mx-smi
mx-smi version: 2.3.4
=================== MetaX System Management Interface Log ===================
Timestamp : Fri Oct 23 02:23:55 2026
Attached GPUs : 2
+---------------------------------------------------------------------------------+
| MX-SMI 2.3.4 Kernel Mode Driver Version: 3.9.61 |
| MACA Version: 3.8.5.14 BIOS Version: 255.255.255.255* |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX N260 | 0 N/A | 0000:98:00.0 | N/A Disabled |
| NA / NA | N/A N/A | 668/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX N260 | 1 Off | 0000:a8:00.0 | 0% Disabled |
| 46W / 225W | 73C P0 | 668/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| no process found |
+---------------------------------------------------------------------------------+
End of Log
二、问题现象
请描述详细的问题现象日志。若日志过长,请上传附件(txt格式)。
驱动安装有问题以后,导致我无法使用muxi的显卡做测评和AI应用
depmod...
*****WARNING*****
5.15.0-161-generic dkms build was failed since the kernel headers does not install.
5.15.0-181-generic dkms build was failed since the kernel headers does not install.
5.15.0-185-generic dkms build was failed since the kernel headers does not install.
5.15.0-25-generic dkms build was failed since the kernel headers does not install.
6.8.0-124-generic dkms build was failed since the kernel headers does not install.
6.8.0-134-generic dkms build was failed since the kernel headers does not install.
6.8.0-136-generic dkms build was failed since the kernel headers does not install.
6.8.0-137-generic dkms build was failed since the kernel headers does not install.
6.8.0-138-generic dkms build was failed since the kernel headers does not install.
6.8.0-139-generic dkms build was failed since the kernel headers does not install.
6.8.0-146-generic dkms build was failed since the kernel headers does not install.
具体看附件
root@server:/workspace# CUDA_VISIBLE_DEVICES=0,1 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
VLLM_METAX_USE_FLASHINFER_SAMPLER=0 \
vllm serve /workspace/data/models/Qwen/Qwen2.5-7B-Instruct \
--tensor-parallel-size 2 \
--trust-remote-code \
--dtype bfloat16 \
--max-model-len 2048 \
--gpu-memory-utilization 0.85 \
--port 8000
INFO 10-22 15:09:10 [init.py:52] Available plugins for group vllm.platform_plugins:
INFO 10-22 15:09:10 [init.py:54] - metax -> vllm_metax:register
INFO 10-22 15:09:10 [init.py:57] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 10-22 15:09:10 [init.py:254] Platform plugin metax is activated
INFO 10-22 15:09:10 [envs.py:136] vllm-metax would try to sets VLLM_USE_FLASHINFER_SAMPLER default to False. Reason: controlled by VLLM_METAX_USE_FLASHINFER_SAMPLER; disabled by default
WARNING 10-22 15:09:10 [envs.py:145] Detected VLLM_USE_FLASHINFER_SAMPLER was manually set to 0.
WARNING 10-22 15:09:10 [envs.py:151] Not overriding VLLM_USE_FLASHINFER_SAMPLER to False because it was manually set. If you want to override it, set forced=True.
INFO 10-22 15:09:10 [envs.py:136] vllm-metax would try to sets VLLM_ENGINE_READY_TIMEOUT_S default to 7200. Reason: set timeout to 7200s for model loading
INFO 10-22 15:09:10 [envs.py:136] vllm-metax would try to sets VLLM_FLOAT32_MATMUL_PRECISION default to high. Reason: set float32 matmul precision to high for better performance on Maca platform
INFO 10-22 15:09:10 [envs.py:136] vllm-metax would try to sets VLLM_USE_V2_MODEL_RUNNER default to False. Reason: v2 model runner is still under development and not fully tested on Maca platform, disable it by default
INFO 10-22 15:09:10 [envs.py:136] vllm-metax would try to sets VLLM_USE_DEEP_GEMM_E8M0 default to False. Reason: USE_DEEP_GEMM_E8M0 default to false on Maca platform
INFO 10-22 15:09:10 [envs.py:136] vllm-metax would try to sets VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS default to False. Reason: disable CUDA graph memory estimate in the KV cache budget on Maca platform
Version info:Mcoplib_Version = '0.4.13'
Build_Maca_Version = '3.8.2.6'
Build_Torch_Version = '2.10.0+metax3.8.2.2'
Min_Compatibility_Maca_Version = '3.7.0'
GIT_BRANCH = 'HEAD'
GIT_COMMIT = 'cdc10b7'
Vllm Op Version = 0.28.0
SGlang Op Version = 0.5.17
INFO mcoplib version: None
INFO: Torch version compatibility check passed: build=2.10.0+metax3.8.2.2, runtime=2.10.0+metax3.8.2.2.
INFO: Release major.minor matching, successful:3.8.
INFO: MACA minimum compatibility check passed: 3.8.5 >= 3.7.0.
/opt/conda/lib/python3.12/site-packages/fla/utils.py:425: UserWarning: Triton is not supported on current platform, roll back to CPU.
warnings.warn(('Triton is not supported on current platform, roll back to CPU.'), stacklevel=1)
/opt/conda/lib/python3.12/site-packages/fla/utils.py:425: UserWarning: Triton is not supported on current platform, roll back to CPU.
warnings.warn(('Triton is not supported on current platform, roll back to CPU.'), stacklevel=1)
/opt/conda/lib/python3.12/site-packages/fla/utils.py:425: UserWarning: Triton is not supported on current platform, roll back to CPU.
warnings.warn(('Triton is not supported on current platform, roll back to CPU.'), stacklevel=1)
!!!!!!! Segfault encountered !!!!!!!
File "<unknown>", line 0, in ModuleManager::ModuleManager()
File "<unknown>", line 0, in GetModule()
File "<unknown>", line 0, in mctlassExHandleCreate
File "<unknown>", line 0, in void pybind11::detail::argument_loader<pybind11::detail::value_and_holder&>::call_impl<void, pybind11::detail::initimpl::constructor<>::execute<pybind11::class_<mctlassExInternalHandleWrapper>, , 0>(pybind11::class_<mctlassExInternalHandleWrapper>&)::{lambda(pybind11::detail::value_and_holder&)#1}&, 0ul, pybind11::detail::void_type>(pybind11::detail::initimpl::constructor<>::execute<pybind11::class_<mctlassExInternalHandleWrapper>, , 0>(pybind11::class_<mctlassExInternalHandleWrapper>&)::{lambda(pybind11::detail::value_and_holder&)#1}&, std::integer_sequence<unsigned long, 0ul>, pybind11::detail::void_type&&) &&
File "<unknown>", line 0, in pybind11::cpp_function::initialize<pybind11::detail::initimpl::constructor<>::execute<pybind11::class_<mctlassExInternalHandleWrapper>, , 0>(pybind11::class_<mctlassExInternalHandleWrapper>&)::{lambda(pybind11::detail::value_and_holder&)#1}, void, pybind11::detail::value_and_holder&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::class_<mctlassExInternalHandleWrapper>&&, void ()(pybind11::detail::value_and_holder&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&)::{lambda(pybind11::detail::function_call&)#1}::operator()(pybind11::detail::function_call&) const
File "<unknown>", line 0, in pybind11::cpp_function::initialize<pybind11::detail::initimpl::constructor<>::execute<pybind11::class_<mctlassExInternalHandleWrapper>, , 0>(pybind11::class_<mctlassExInternalHandleWrapper>&)::{lambda(pybind11::detail::value_and_holder&)#1}, void, pybind11::detail::value_and_holder&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::class_<mctlassExInternalHandleWrapper>&&, void ()(pybind11::detail::value_and_holder&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&)::{lambda(pybind11::detail::function_call&)#1}::invoke(pybind11::detail::function_call&)
File "<unknown>", line 0, in pybind11::cpp_function::dispatcher(_object, _object, _object*)
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 537, in cfunction_call
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 240, in _PyObject_MakeTpCall
File "/usr/local/src/conda/python-3.12.11/Objects/classobject.c", line 69, in method_vectorcall
File "/usr/local/src/conda/python-3.12.11/Objects/typeobject.c", line 9038, in slot_tp_init
File "/usr/local/src/conda/python-3.12.11/Objects/typeobject.c", line 1679, in type_call
File "<unknown>", line 0, in pybind11_meta_call
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 240, in _PyObject_MakeTpCall
File "Python/bytecodes.c", line 2715, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_ceval.h", line 89, in _PyEval_EvalFrame
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 1685, in _PyEval_Vector
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 419, in _PyFunction_Vectorcall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 133, in _PyObject_FastCallDictTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 508, in _PyObject_Call_Prepend
File "/usr/local/src/conda/python-3.12.11/Objects/typeobject.c", line 9035, in slot_tp_init
File "/usr/local/src/conda/python-3.12.11/Objects/typeobject.c", line 1679, in type_call
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 240, in _PyObject_MakeTpCall
File "Python/bytecodes.c", line 2715, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 2510, in import_name
File "Python/bytecodes.c", line 2144, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 2510, in import_name
File "Python/bytecodes.c", line 2144, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 2510, in import_name
File "Python/bytecodes.c", line 2144, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 2510, in import_name
File "Python/bytecodes.c", line 2144, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 2510, in import_name
File "Python/bytecodes.c", line 2144, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 276, in builtinimportimpl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 107, in builtin___import
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2945, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 2510, in import_name
File "Python/bytecodes.c", line 2144, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 276, in builtinimportimpl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 107, in builtinimport_
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2945, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 2510, in import_name
File "Python/bytecodes.c", line 2144, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 276, in builtin_importimpl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 107, in builtinimport__
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2945, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 2510, in import_name
File "Python/bytecodes.c", line 2144, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 276, in builtinimportimpl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 107, in builtinimport_
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2945, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 2510, in import_name
File "Python/bytecodes.c", line 2144, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 276, in builtin_importimpl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 107, in builtinimport__
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 276, in builtinimportimpl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 107, in builtinimport_
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 2510, in import_name
File "Python/bytecodes.c", line 2144, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 276, in builtin_importimpl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 107, in builtinimport__
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2945, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 2510, in import_name
File "Python/bytecodes.c", line 2144, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 276, in builtinimportimpl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 107, in builtinimport
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunctionvectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2945, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 2510, in import_name
File "Python/bytecodes.c", line 2144, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/bltinmodule.c", line 1093, in builtin_exec_impl
File "/usr/local/src/conda/python-3.12.11/Python/clinic/bltinmodule.c.h", line 543, in builtin_exec
File "/usr/local/src/conda/python-3.12.11/Objects/methodobject.c", line 438, in cfunction_vectorcall_FASTCALL_KEYWORDS
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 387, in PyCFunction_Call
File "Python/bytecodes.c", line 3263, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Include/internal/pycore_call.h", line 92, in _PyObject_VectorcallTstate
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 850, in object_vacall
File "/usr/local/src/conda/python-3.12.11/Objects/call.c", line 911, in PyObject_CallMethodObjArgs
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2793, in import_find_and_load
File "/usr/local/src/conda/python-3.12.11/Python/import.c", line 2876, in PyImport_ImportModuleLevelObject
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 2510, in import_name
File "Python/bytecodes.c", line 2144, in _PyEval_EvalFrameDefault
File "/usr/local/src/conda/python-3.12.11/Python/ceval.c", line 580, in PyEval_EvalCode
File "/usr/local/src/conda/python-3.12.11/Python/pythonrun.c", line 1757, in run_eval_code_obj
File "/usr/local/src/conda/python-3.12.11/Python/pythonrun.c", line 1778, in run_mod
File "/usr/local/src/conda/python-3.12.11/Python/pythonrun.c", line 1674, in pyrun_file
File "/usr/local/src/conda/python-3.12.11/Python/pythonrun.c", line 459, in _PyRun_SimpleFileObject
File "/usr/local/src/conda/python-3.12.11/Python/pythonrun.c", line 78, in _PyRun_AnyFileObject
File "/usr/local/src/conda/python-3.12.11/Modules/main.c", line 361, in pymain_run_file_obj
File "/us