MetaX C550 部署Qwen3.5-122B-A10B，8卡 无法部署成功，采用VLLM镜像部署 镜像cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.22.0-maca.ai3.8.0.5-torch2.10-py310-ubuntu22.04-amd64，参数--tensor-parallel-size=4 --trust-remote-code --dtype=bfloat16 --distributed-executor-backend=mp --gpu-memory-utilization=0.80 --max-model-len=8192 --max-num-batched-tokens=8192 --no-async-scheduling --mm-encoder-tp-mode=data --mm-processor-cache-type=shm --limit-mm-per-prompt={"image":5,"video":1} --skip-mm-profiling --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser=qwen3_coder --reasoning-parser=qwen3 --pipeline-parallel-size=1 --enforce-eager 然后一直报 (EngineCore pid=739) INFO 07-30 09:47:39 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=739) INFO 07-30 09:48:39 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization). 无法正常部署
app@muxi-gpu-server:~$ mx-smi
mx-smi version: 2.2.12
=================== MetaX System Management Interface Log ===================
Timestamp : Thu Jul 30 01:41:42 2026
Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.2.12 Kernel Mode Driver Version: 3.6.11 |
| MACA Version: 3.5.3.18 BIOS Version: 1.22.3.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C550 | 0 N/A | 0000:23:00.0 | 0% Disabled |
| NA / NA | 43C N/A | 860/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C550 | 1 N/A | 0000:26:00.0 | 0% Disabled |
| NA / NA | 39C N/A | 860/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C550 | 2 N/A | 0000:63:00.0 | 0% Disabled |
| NA / NA | 38C N/A | 860/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C550 | 3 N/A | 0000:66:00.0 | 0% Disabled |
| NA / NA | 45C N/A | 860/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C550 | 4 N/A | 0000:a3:00.0 | 4% Disabled |
| NA / NA | 47C N/A | 63719/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C550 | 5 N/A | 0000:a4:00.0 | 4% Disabled |
| NA / NA | 41C N/A | 63719/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C550 | 6 N/A | 0000:e3:00.0 | 4% Disabled |
| NA / NA | 41C N/A | 63719/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C550 | 7 N/A | 0000:e4:00.0 | 4% Disabled |
| NA / NA | 47C N/A | 63719/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| 4 2246419 VLLM::Worker_TP 62856 |
| 5 2246420 VLLM::Worker_TP 62856 |
| 6 2246421 VLLM::Worker_TP 62856 |
| 7 2246422 VLLM::Worker_TP 62856 |
+---------------------------------------------------------------------------------+
End of Log
app@muxi-gpu-server:~$
日志如下 :
2026-07-30 09:43:30.440035+08:00 - gpustack.worker.backends.base - INFO - Preparing model files...
2026-07-30 09:43:30.468723+08:00 - gpustack.worker.backends.base - INFO - Model files are ready.
2026-07-30 09:43:30.468842+08:00 - gpustack.worker.serve_manager - INFO - Provisioning model instance Qwen3.5-122B-A10B-Wh6Qo
2026-07-30 09:43:30.468892+08:00 - gpustack.worker.backends.custom - INFO - Starting custom backend model instance: Qwen3.5-122B-A10B-Wh6Qo
2026-07-30 09:43:30.469443+08:00 - gpustack.worker.backends.custom - INFO - Creating custom backend container workload: Qwen3.5-122B-A10B-Wh6Qo
2026-07-30 09:43:30.469533+08:00 - gpustack.worker.backends.custom - INFO - With image: cr.metax-tech.com/public-ai-release/maca/vllm-metax:0.22.0-maca.ai3.8.0.5-torch2.10-py310-ubuntu22.04-amd64, command: [/opt/conda/bin/vllm], arguments: [serve /data/models/Qwen3.5-122B-A10B --port 40036 --host 0.0.0.0 --served-model-name Qwen3.5-122B-A10B --tensor-parallel-size=4 --trust-remote-code --dtype=bfloat16 --distributed-executor-backend=mp --gpu-memory-utilization=0.80 --max-model-len=8192 --max-num-batched-tokens=8192 --no-async-scheduling --mm-encoder-tp-mode=data --mm-processor-cache-type=shm --limit-mm-per-prompt={"image":5,"video":1} --skip-mm-profiling --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser=qwen3_coder --reasoning-parser=qwen3 --pipeline-parallel-size=1 --enforce-eager], ports: [40036], envs(inconsistent input items mean unchangeable):
2026-07-30 09:43:30.520105+08:00 - gpustack.utils.config - INFO - Using system default registry 'quay.io'; image resolved to: quay.io/gpustack/runtime:pause
2026-07-30 09:43:30.520326+08:00 - gpustack.utils.config - INFO - Using system default registry 'quay.io'; image resolved to: quay.io/gpustack/runtime:health
2026-07-30 09:43:30.525039+08:00 - 95465 - gpustack_runtime.deployer.docker - INFO - Mirrored deployment enabled, using self Container ab59017db0ce for options mirroring
2026-07-30 09:43:31.905859+08:00 - gpustack.worker.backends.custom - INFO - Created custom backend container workload: Qwen3.5-122B-A10B-Wh6Qo
2026-07-30 09:43:31.906115+08:00 - gpustack.worker.serve_manager - INFO - Finished provisioning model instance Qwen3.5-122B-A10B-Wh6Qo
INFO 07-30 09:43:35 [init.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-30 09:43:35 [init.py:46] - metax -> vllm_metax:register
INFO 07-30 09:43:35 [init.py:49] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 07-30 09:43:35 [init.py:238] Platform plugin metax is activated
INFO 07-30 09:43:35 [envs.py:123] Plugin sets VLLM_USE_FLASHINFER_SAMPLER to False. Reason: flashinfer sampler are not supported on maca
INFO 07-30 09:43:35 [envs.py:123] Plugin sets VLLM_ENGINE_READY_TIMEOUT_S to 3600. Reason: set timeout to 3600s for model loading
INFO 07-30 09:43:35 [envs.py:123] Plugin sets VLLM_FLOAT32_MATMUL_PRECISION to high. Reason: set float32 matmul precision to high for better performance on Maca platform
INFO 07-30 09:43:35 [envs.py:123] Plugin sets VLLM_USE_V2_MODEL_RUNNER to False. Reason: v2 model runner is still under development and not fully tested on Maca platform, disable it by default
WARNING 07-30 09:43:37 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
INFO Print the version information of mcoplib during compilation.
Version info:Mcoplib_Version = '0.4.7'
Build_Maca_Version = '3.8.0.23'
GIT_BRANCH = 'HEAD'
GIT_COMMIT = '3c71b39'
Vllm Op Version = 0.22.0
SGlang Op Version = 0.5.12
INFO Staring Check the current MACA version of the operating environment.
INFO: Release major.minor matching, successful:3.8.
WARNING 07-30 09:43:56 [init.py:87] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq.MacaAWQConfig'>.
WARNING 07-30 09:43:56 [init.py:87] The quantization method 'awq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq_marlin.MacaAWQMarlinConfig'>.
WARNING 07-30 09:43:56 [init.py:87] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.compressed_tensors.MacaCompressedTensorsConfig'>.
WARNING 07-30 09:43:56 [init.py:87] The quantization method 'gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.auto_gptq.MacaAutoGPTQConfig'>.
WARNING 07-30 09:43:56 [init.py:87] The quantization method 'auto_gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.auto_gptq.MacaAutoGPTQConfig'>.
WARNING 07-30 09:43:56 [init.py:87] The quantization method 'moe_wna16' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.moe_wna16.MacaMoeWNA16Config'>.
WARNING 07-30 09:43:56 [registry.py:983] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_mtp:DeepSeekMTP.
WARNING 07-30 09:43:56 [registry.py:983] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV2ForCausalLM.
WARNING 07-30 09:43:56 [registry.py:983] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-30 09:43:56 [registry.py:983] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-30 09:43:56 [registry.py:983] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:GlmMoeDsaForCausalLM.
WARNING 07-30 09:43:56 [registry.py:983] Model architecture DeepseekV4ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4:DeepseekV4ForCausalLM.
WARNING 07-30 09:43:56 [registry.py:983] Model architecture DeepSeekV4MTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4_mtp:DeepSeekV4MTP.
WARNING 07-30 09:43:56 [registry.py:983] Model architecture Step3p5MTP is already registered, and will be overwritten by the new model class vllm_metax.models.step3p5_mtp:Step3p5MTP.
WARNING 07-30 09:43:56 [registry.py:983] Model architecture Qwen3OmniMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_omni_moe_thinker:Qwen3OmniMoeThinkerForConditionalGeneration.
WARNING 07-30 09:43:56 [registry.py:983] Model architecture MiMoV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2:MiMoV2ForCausalLM.
WARNING 07-30 09:43:56 [registry.py:983] Model architecture MiMoV2FlashForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2:MiMoV2FlashForCausalLM.
WARNING 07-30 09:43:56 [registry.py:983] Model architecture MiMoV2MTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2_mtp:MiMoV2MTP.
WARNING 07-30 09:43:56 [registry.py:983] Model architecture MiMoV2OmniMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2_mtp:MiMoV2OmniMTP.
(APIServer pid=1) INFO 07-30 09:43:56 [utils.py:344]
(APIServer pid=1) INFO 07-30 09:43:56 [utils.py:344] █ █ █▄ ▄█
(APIServer pid=1) INFO 07-30 09:43:56 [utils.py:344] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.22.0
(APIServer pid=1) INFO 07-30 09:43:56 [utils.py:344] █▄█▀ █ █ █ █ model /data/models/Qwen3.5-122B-A10B
(APIServer pid=1) INFO 07-30 09:43:56 [utils.py:344] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=1) INFO 07-30 09:43:56 [utils.py:344]
(APIServer pid=1) INFO 07-30 09:43:56 [utils.py:278] non-default args: {'model_tag': '/data/models/Qwen3.5-122B-A10B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 40036, 'model': '/data/models/Qwen3.5-122B-A10B', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_model_len': 8192, 'enforce_eager': True, 'served_model_name': ['Qwen3.5-122B-A10B'], 'reasoning_parser': 'qwen3', 'distributed_executor_backend': 'mp', 'tensor_parallel_size': 4, 'gpu_memory_utilization': 0.8, 'enable_prefix_caching': True, 'limit_mm_per_prompt': {'image': 5, 'video': 1}, 'mm_processor_cache_type': 'shm', 'mm_encoder_tp_mode': 'data', 'skip_mm_profiling': True, 'max_num_batched_tokens': 8192}
(APIServer pid=1) INFO 07-30 09:44:29 [model.py:617] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=1) INFO 07-30 09:44:29 [model.py:1752] Using max model len 8192
(APIServer pid=1) INFO 07-30 09:44:29 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=1) WARNING 07-30 09:44:29 [config.py:355] Mamba cache mode is set to 'align' for Qwen3_5MoeForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=1) INFO 07-30 09:44:29 [config.py:375] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=1) INFO 07-30 09:44:29 [vllm.py:977] Asynchronous scheduling is disabled.
(APIServer pid=1) WARNING 07-30 09:44:29 [vllm.py:1033] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(APIServer pid=1) WARNING 07-30 09:44:29 [vllm.py:1058] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(APIServer pid=1) INFO 07-30 09:44:29 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(APIServer pid=1) INFO 07-30 09:44:29 [vllm.py:1234] Cudagraph is disabled under eager mode
(APIServer pid=1) INFO 07-30 09:44:29 [envs.py:123] Plugin sets VLLM_TUNED_CONFIG_FOLDER to /opt/conda/lib/python3.10/site-packages/vllm_metax/model_executor/layers/fused_moe/configs/H=3072. Reason: set FusedMoE tuned config dir by hidden_size=3072
(APIServer pid=1) INFO 07-30 09:44:30 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
WARNING 07-30 09:44:44 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
INFO 07-30 09:44:44 [init.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-30 09:44:44 [init.py:46] - metax -> vllm_metax:register
INFO 07-30 09:44:44 [init.py:49] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 07-30 09:44:44 [init.py:238] Platform plugin metax is activated
INFO 07-30 09:44:44 [envs.py:123] Plugin sets VLLM_USE_FLASHINFER_SAMPLER to False. Reason: flashinfer sampler are not supported on maca
INFO 07-30 09:44:44 [envs.py:123] Plugin sets VLLM_ENGINE_READY_TIMEOUT_S to 3600. Reason: set timeout to 3600s for model loading
INFO 07-30 09:44:44 [envs.py:123] Plugin sets VLLM_FLOAT32_MATMUL_PRECISION to high. Reason: set float32 matmul precision to high for better performance on Maca platform
INFO 07-30 09:44:44 [envs.py:123] Plugin sets VLLM_USE_V2_MODEL_RUNNER to False. Reason: v2 model runner is still under development and not fully tested on Maca platform, disable it by default
(EngineCore pid=739) INFO Print the version information of mcoplib during compilation.
(EngineCore pid=739)
(EngineCore pid=739) Version info:Mcoplib_Version = '0.4.7'
(EngineCore pid=739) Build_Maca_Version = '3.8.0.23'
(EngineCore pid=739) GIT_BRANCH = 'HEAD'
(EngineCore pid=739) GIT_COMMIT = '3c71b39'
(EngineCore pid=739) Vllm Op Version = 0.22.0
(EngineCore pid=739) SGlang Op Version = 0.5.12
(EngineCore pid=739)
(EngineCore pid=739) INFO Staring Check the current MACA version of the operating environment.
(EngineCore pid=739)
(EngineCore pid=739) INFO: Release major.minor matching, successful:3.8.
(EngineCore pid=739)
(EngineCore pid=739) WARNING 07-30 09:45:01 [init.py:87] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq.MacaAWQConfig'>.
(EngineCore pid=739) WARNING 07-30 09:45:01 [init.py:87] The quantization method 'awq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq_marlin.MacaAWQMarlinConfig'>.
(EngineCore pid=739) WARNING 07-30 09:45:01 [init.py:87] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.compressed_tensors.MacaCompressedTensorsConfig'>.
(EngineCore pid=739) WARNING 07-30 09:45:01 [init.py:87] The quantization method 'gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.auto_gptq.MacaAutoGPTQConfig'>.
(EngineCore pid=739) WARNING 07-30 09:45:01 [init.py:87] The quantization method 'auto_gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.auto_gptq.MacaAutoGPTQConfig'>.
(EngineCore pid=739) WARNING 07-30 09:45:01 [init.py:87] The quantization method 'moe_wna16' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.moe_wna16.MacaMoeWNA16Config'>.
(EngineCore pid=739) WARNING 07-30 09:45:01 [registry.py:983] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_mtp:DeepSeekMTP.
(EngineCore pid=739) WARNING 07-30 09:45:01 [registry.py:983] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV2ForCausalLM.
(EngineCore pid=739) WARNING 07-30 09:45:01 [registry.py:983] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
(EngineCore pid=739) WARNING 07-30 09:45:01 [registry.py:983] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
(EngineCore pid=739) WARNING 07-30 09:45:01 [registry.py:983] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:GlmMoeDsaForCausalLM.
(EngineCore pid=739) WARNING 07-30 09:45:01 [registry.py:983] Model architecture DeepseekV4ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4:DeepseekV4ForCausalLM.
(EngineCore pid=739) WARNING 07-30 09:45:01 [registry.py:983] Model architecture DeepSeekV4MTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4_mtp:DeepSeekV4MTP.
(EngineCore pid=739) WARNING 07-30 09:45:01 [registry.py:983] Model architecture Step3p5MTP is already registered, and will be overwritten by the new model class vllm_metax.models.step3p5_mtp:Step3p5MTP.
(EngineCore pid=739) WARNING 07-30 09:45:01 [registry.py:983] Model architecture Qwen3OmniMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_omni_moe_thinker:Qwen3OmniMoeThinkerForConditionalGeneration.
(EngineCore pid=739) WARNING 07-30 09:45:01 [registry.py:983] Model architecture MiMoV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2:MiMoV2ForCausalLM.
(EngineCore pid=739) WARNING 07-30 09:45:01 [registry.py:983] Model architecture MiMoV2FlashForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2:MiMoV2FlashForCausalLM.
(EngineCore pid=739) WARNING 07-30 09:45:01 [registry.py:983] Model architecture MiMoV2MTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2_mtp:MiMoV2MTP.
(EngineCore pid=739) WARNING 07-30 09:45:01 [registry.py:983] Model architecture MiMoV2OmniMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2_mtp:MiMoV2OmniMTP.
(EngineCore pid=739) INFO 07-30 09:45:01 [core.py:112] Initializing a V1 LLM engine (v0.22.0) with config: model='/data/models/Qwen3.5-122B-A10B', speculative_config=None, tokenizer='/data/models/Qwen3.5-122B-A10B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.5-122B-A10B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=739) WARNING 07-30 09:45:01 [multiproc_executor.py:1029] Reducing Torch parallelism from 96 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=739) INFO 07-30 09:45:01 [multiproc_executor.py:139] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=173.30.14.19 (local), world_size=4, local_world_size=4
INFO 07-30 09:45:05 [init.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-30 09:45:05 [init.py:46] - metax -> vllm_metax:register
INFO 07-30 09:45:05 [init.py:49] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 07-30 09:45:05 [init.py:238] Platform plugin metax is activated
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_USE_FLASHINFER_SAMPLER to False. Reason: flashinfer sampler are not supported on maca
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_ENGINE_READY_TIMEOUT_S to 3600. Reason: set timeout to 3600s for model loading
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_FLOAT32_MATMUL_PRECISION to high. Reason: set float32 matmul precision to high for better performance on Maca platform
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_USE_V2_MODEL_RUNNER to False. Reason: v2 model runner is still under development and not fully tested on Maca platform, disable it by default
INFO 07-30 09:45:05 [init.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-30 09:45:05 [init.py:46] - metax -> vllm_metax:register
INFO 07-30 09:45:05 [init.py:49] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 07-30 09:45:05 [init.py:238] Platform plugin metax is activated
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_USE_FLASHINFER_SAMPLER to False. Reason: flashinfer sampler are not supported on maca
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_ENGINE_READY_TIMEOUT_S to 3600. Reason: set timeout to 3600s for model loading
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_FLOAT32_MATMUL_PRECISION to high. Reason: set float32 matmul precision to high for better performance on Maca platform
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_USE_V2_MODEL_RUNNER to False. Reason: v2 model runner is still under development and not fully tested on Maca platform, disable it by default
INFO 07-30 09:45:05 [init.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-30 09:45:05 [init.py:46] - metax -> vllm_metax:register
INFO 07-30 09:45:05 [init.py:49] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 07-30 09:45:05 [init.py:238] Platform plugin metax is activated
INFO 07-30 09:45:05 [init.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-30 09:45:05 [init.py:46] - metax -> vllm_metax:register
INFO 07-30 09:45:05 [init.py:49] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 07-30 09:45:05 [init.py:238] Platform plugin metax is activated
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_USE_FLASHINFER_SAMPLER to False. Reason: flashinfer sampler are not supported on maca
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_ENGINE_READY_TIMEOUT_S to 3600. Reason: set timeout to 3600s for model loading
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_FLOAT32_MATMUL_PRECISION to high. Reason: set float32 matmul precision to high for better performance on Maca platform
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_USE_V2_MODEL_RUNNER to False. Reason: v2 model runner is still under development and not fully tested on Maca platform, disable it by default
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_USE_FLASHINFER_SAMPLER to False. Reason: flashinfer sampler are not supported on maca
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_ENGINE_READY_TIMEOUT_S to 3600. Reason: set timeout to 3600s for model loading
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_FLOAT32_MATMUL_PRECISION to high. Reason: set float32 matmul precision to high for better performance on Maca platform
INFO 07-30 09:45:05 [envs.py:123] Plugin sets VLLM_USE_V2_MODEL_RUNNER to False. Reason: v2 model runner is still under development and not fully tested on Maca platform, disable it by default
WARNING 07-30 09:45:07 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
WARNING 07-30 09:45:07 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
WARNING 07-30 09:45:07 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
WARNING 07-30 09:45:07 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
INFO Print the version information of mcoplib during compilation.
Version info:Mcoplib_Version = '0.4.7'
Build_Maca_Version = '3.8.0.23'
GIT_BRANCH = 'HEAD'
GIT_COMMIT = '3c71b39'
Vllm Op Version = 0.22.0
SGlang Op Version = 0.5.12
INFO Staring Check the current MACA version of the operating environment.
INFO: Release major.minor matching, successful:3.8.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq.MacaAWQConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'awq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq_marlin.MacaAWQMarlinConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.compressed_tensors.MacaCompressedTensorsConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.auto_gptq.MacaAutoGPTQConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'auto_gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.auto_gptq.MacaAutoGPTQConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'moe_wna16' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.moe_wna16.MacaMoeWNA16Config'>.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_mtp:DeepSeekMTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV2ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:GlmMoeDsaForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV4ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4:DeepseekV4ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepSeekV4MTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4_mtp:DeepSeekV4MTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture Step3p5MTP is already registered, and will be overwritten by the new model class vllm_metax.models.step3p5_mtp:Step3p5MTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture Qwen3OmniMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_omni_moe_thinker:Qwen3OmniMoeThinkerForConditionalGeneration.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2:MiMoV2ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2FlashForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2:MiMoV2FlashForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2MTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2_mtp:MiMoV2MTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2OmniMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2_mtp:MiMoV2OmniMTP.
INFO Print the version information of mcoplib during compilation.
Version info:Mcoplib_Version = '0.4.7'
Build_Maca_Version = '3.8.0.23'
GIT_BRANCH = 'HEAD'
GIT_COMMIT = '3c71b39'
Vllm Op Version = 0.22.0
SGlang Op Version = 0.5.12
INFO Staring Check the current MACA version of the operating environment.
INFO: Release major.minor matching, successful:3.8.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq.MacaAWQConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'awq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq_marlin.MacaAWQMarlinConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.compressed_tensors.MacaCompressedTensorsConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.auto_gptq.MacaAutoGPTQConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'auto_gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.auto_gptq.MacaAutoGPTQConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'moe_wna16' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.moe_wna16.MacaMoeWNA16Config'>.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_mtp:DeepSeekMTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV2ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:GlmMoeDsaForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV4ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4:DeepseekV4ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepSeekV4MTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4_mtp:DeepSeekV4MTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture Step3p5MTP is already registered, and will be overwritten by the new model class vllm_metax.models.step3p5_mtp:Step3p5MTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture Qwen3OmniMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_omni_moe_thinker:Qwen3OmniMoeThinkerForConditionalGeneration.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2:MiMoV2ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2FlashForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2:MiMoV2FlashForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2MTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2_mtp:MiMoV2MTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2OmniMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2_mtp:MiMoV2OmniMTP.
INFO Print the version information of mcoplib during compilation.
Version info:Mcoplib_Version = '0.4.7'
Build_Maca_Version = '3.8.0.23'
GIT_BRANCH = 'HEAD'
GIT_COMMIT = '3c71b39'
Vllm Op Version = 0.22.0
SGlang Op Version = 0.5.12
INFO Staring Check the current MACA version of the operating environment.
INFO: Release major.minor matching, successful:3.8.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq.MacaAWQConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'awq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq_marlin.MacaAWQMarlinConfig'>.
INFO Print the version information of mcoplib during compilation.
Version info:Mcoplib_Version = '0.4.7'
Build_Maca_Version = '3.8.0.23'
GIT_BRANCH = 'HEAD'
GIT_COMMIT = '3c71b39'
Vllm Op Version = 0.22.0
SGlang Op Version = 0.5.12
INFO Staring Check the current MACA version of the operating environment.
INFO: Release major.minor matching, successful:3.8.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq.MacaAWQConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'awq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq_marlin.MacaAWQMarlinConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.compressed_tensors.MacaCompressedTensorsConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.auto_gptq.MacaAutoGPTQConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'auto_gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.auto_gptq.MacaAutoGPTQConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'moe_wna16' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.moe_wna16.MacaMoeWNA16Config'>.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_mtp:DeepSeekMTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV2ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:GlmMoeDsaForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV4ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4:DeepseekV4ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepSeekV4MTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4_mtp:DeepSeekV4MTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture Step3p5MTP is already registered, and will be overwritten by the new model class vllm_metax.models.step3p5_mtp:Step3p5MTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture Qwen3OmniMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_omni_moe_thinker:Qwen3OmniMoeThinkerForConditionalGeneration.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2:MiMoV2ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2FlashForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2:MiMoV2FlashForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2MTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2_mtp:MiMoV2MTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2OmniMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2_mtp:MiMoV2OmniMTP.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.compressed_tensors.MacaCompressedTensorsConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.auto_gptq.MacaAutoGPTQConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'auto_gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.auto_gptq.MacaAutoGPTQConfig'>.
WARNING 07-30 09:45:25 [init.py:87] The quantization method 'moe_wna16' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.moe_wna16.MacaMoeWNA16Config'>.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_mtp:DeepSeekMTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV2ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:GlmMoeDsaForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepseekV4ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4:DeepseekV4ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture DeepSeekV4MTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4_mtp:DeepSeekV4MTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture Step3p5MTP is already registered, and will be overwritten by the new model class vllm_metax.models.step3p5_mtp:Step3p5MTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture Qwen3OmniMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_omni_moe_thinker:Qwen3OmniMoeThinkerForConditionalGeneration.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2:MiMoV2ForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2FlashForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2:MiMoV2FlashForCausalLM.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2MTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2_mtp:MiMoV2MTP.
WARNING 07-30 09:45:25 [registry.py:983] Model architecture MiMoV2OmniMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.mimo_v2_mtp:MiMoV2OmniMTP.
(Worker pid=1009) INFO 07-30 09:45:26 [parallel_state.py:1422] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:45737 backend=nccl
(Worker pid=1010) INFO 07-30 09:45:26 [parallel_state.py:1422] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:45737 backend=nccl
(Worker pid=1007) INFO 07-30 09:45:26 [parallel_state.py:1422] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:45737 backend=nccl
(Worker pid=1008) INFO 07-30 09:45:26 [parallel_state.py:1422] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:45737 backend=nccl
(Worker pid=1007) Current Python version 3.10 is below the recommended 3.11 version. It is recommended to upgrade to Python 3.11 or higher for the best experience.
(Worker pid=1009) Current Python version 3.10 is below the recommended 3.11 version. It is recommended to upgrade to Python 3.11 or higher for the best experience.
(Worker pid=1008) Current Python version 3.10 is below the recommended 3.11 version. It is recommended to upgrade to Python 3.11 or higher for the best experience.
(Worker pid=1010) Current Python version 3.10 is below the recommended 3.11 version. It is recommended to upgrade to Python 3.11 or higher for the best experience.
(Worker pid=1010) torch.compile is not available in Python 3.10, using identity decorator instead
(Worker pid=1008) /opt/conda/lib/python3.10/site-packages/flashinfer/gdn_prefill.py:27: UserWarning: fla package is not installed, gdn_prefill module will not be available. Install via: pip install flash_linear_attention-.whl
(Worker pid=1008) warnings.warn(
(Worker pid=1007) /opt/conda/lib/python3.10/site-packages/flashinfer/gdn_prefill.py:27: UserWarning: fla package is not installed, gdn_prefill module will not be available. Install via: pip install flash_linear_attention-.whl
(Worker pid=1007) warnings.warn(
(Worker pid=1007) INFO 07-30 09:45:28 [mccl.py:27] Found mccl from library libmccl.so
(Worker pid=1007) INFO 07-30 09:45:28 [pynccl.py:113] vLLM is using nccl==2.16.5
(Worker pid=1009) /opt/conda/lib/python3.10/site-packages/flashinfer/gdn_prefill.py:27: UserWarning: fla package is not installed, gdn_prefill module will not be available. Install via: pip install flash_linear_attention-.whl
(Worker pid=1009) warnings.warn(
(Worker pid=1010) /opt/conda/lib/python3.10/site-packages/flashinfer/gdn_prefill.py:27: UserWarning: fla package is not installed, gdn_prefill module will not be available. Install via: pip install flash_linear_attention-.whl
(Worker pid=1010) warnings.warn(
(Worker pid=1007) INFO 07-30 09:45:29 [cuda_communicator.py:232] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=1007) INFO 07-30 09:45:30 [cuda_communicator.py:232] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=1007) INFO 07-30 09:45:30 [parallel_state.py:1735] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker_TP1 pid=1008) INFO 07-30 09:45:36 [platform.py:432] Valid backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']
(Worker_TP3 pid=1010) INFO 07-30 09:45:37 [platform.py:432] Valid backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']
(Worker_TP2 pid=1009) INFO 07-30 09:45:38 [platform.py:432] Valid backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']
(Worker_TP0 pid=1007) INFO 07-30 09:45:38 [gpu_model_runner.py:5037] Starting to load model /data/models/Qwen3.5-122B-A10B...
(Worker_TP0 pid=1007) INFO 07-30 09:45:39 [fa_utils.py:24] Using Maca version of flash attention, which only supports version 2.
(Worker_TP0 pid=1007) INFO 07-30 09:45:39 [mm_encoder_attention.py:76] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker_TP0 pid=1007) INFO 07-30 09:45:39 [mm_encoder_attention.py:79] Using AttentionBackendEnum.FLASH_ATTN for MacaMMEncoderAttention.
(Worker_TP0 pid=1007) INFO 07-30 09:45:39 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=None).
(Worker_TP0 pid=1007) INFO 07-30 09:45:39 [platform.py:420] Some attention backends are not valid for maca with AttentionSelectorConfig(head_size=256, dtype=torch.bfloat16, kv_cache_dtype=auto, block_size=None, use_mla=False, has_sink=False, use_sparse=False, use_mm_prefix=False, use_per_head_quant_scales=False, attn_type=decoder, use_non_causal=False, use_batch_invariant=False). Reasons: {TURBOQUANT: [kv_cache_dtype not supported]}.
(Worker_TP0 pid=1007) INFO 07-30 09:45:39 [platform.py:432] Valid backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']
(Worker_TP0 pid=1007) INFO 07-30 09:45:39 [platform.py:463] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(Worker_TP0 pid=1007) INFO 07-30 09:45:40 [weight_utils.py:922] Filesystem type for checkpoints: EXT4. Checkpoint size: 232.99 GiB. Available RAM: 913.38 GiB.
(Worker_TP0 pid=1007) INFO 07-30 09:45:40 [weight_utils.py:945] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 0% Completed | 0/39 [00:00<?, ?it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 3% Completed | 1/39 [00:00<00:25, 1.49it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 5% Completed | 2/39 [00:01<00:26, 1.39it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 8% Completed | 3/39 [00:02<00:25, 1.40it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 10% Completed | 4/39 [00:02<00:24, 1.42it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 13% Completed | 5/39 [00:03<00:24, 1.41it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 15% Completed | 6/39 [00:04<00:23, 1.42it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 18% Completed | 7/39 [00:04<00:21, 1.46it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 21% Completed | 8/39 [00:05<00:21, 1.44it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 23% Completed | 9/39 [00:06<00:20, 1.45it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 26% Completed | 10/39 [00:06<00:19, 1.47it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 28% Completed | 11/39 [00:07<00:18, 1.50it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 31% Completed | 12/39 [00:08<00:18, 1.47it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 33% Completed | 13/39 [00:08<00:17, 1.46it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 36% Completed | 14/39 [00:09<00:17, 1.44it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 38% Completed | 15/39 [00:10<00:17, 1.40it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 41% Completed | 16/39 [00:11<00:16, 1.38it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 44% Completed | 17/39 [00:11<00:16, 1.36it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 46% Completed | 18/39 [00:12<00:15, 1.33it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 49% Completed | 19/39 [00:13<00:15, 1.31it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 51% Completed | 20/39 [00:14<00:14, 1.29it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 54% Completed | 21/39 [00:15<00:14, 1.28it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 56% Completed | 22/39 [00:15<00:13, 1.27it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 59% Completed | 23/39 [00:16<00:12, 1.28it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 62% Completed | 24/39 [00:17<00:11, 1.26it/s]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 64% Completed | 25/39 [00:20<00:18, 1.30s/it]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 67% Completed | 26/39 [00:22<00:22, 1.76s/it]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 69% Completed | 27/39 [00:25<00:24, 2.08s/it]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 72% Completed | 28/39 [00:28<00:25, 2.28s/it]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 74% Completed | 29/39 [00:31<00:24, 2.44s/it]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 77% Completed | 30/39 [00:34<00:23, 2.59s/it]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 79% Completed | 31/39 [00:37<00:21, 2.68s/it]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 82% Completed | 32/39 [00:39<00:19, 2.73s/it]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 85% Completed | 33/39 [00:42<00:16, 2.77s/it]
(Worker_TP1 pid=1008) INFO 07-30 09:46:25 [interface.py:649] Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP1 pid=1008) INFO 07-30 09:46:25 [interface.py:673] Padding mamba page size by 1.34% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 87% Completed | 34/39 [00:45<00:13, 2.77s/it]
(Worker_TP3 pid=1010) INFO 07-30 09:46:28 [interface.py:649] Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP3 pid=1010) INFO 07-30 09:46:28 [interface.py:673] Padding mamba page size by 1.34% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 90% Completed | 35/39 [00:48<00:11, 2.81s/it]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 92% Completed | 36/39 [00:51<00:08, 2.76s/it]
(Worker_TP2 pid=1009) INFO 07-30 09:46:31 [interface.py:649] Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP2 pid=1009) INFO 07-30 09:46:31 [interface.py:673] Padding mamba page size by 1.34% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 95% Completed | 37/39 [00:53<00:05, 2.62s/it]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 97% Completed | 38/39 [00:56<00:02, 2.82s/it]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 100% Completed | 39/39 [00:57<00:00, 2.24s/it]
(Worker_TP0 pid=1007)
Loading safetensors checkpoint shards: 100% Completed | 39/39 [00:57<00:00, 1.48s/it]
(Worker_TP0 pid=1007)
(Worker_TP0 pid=1007) INFO 07-30 09:46:37 [default_loader.py:397] Loading weights took 57.60 seconds
(Worker_TP0 pid=1007) INFO 07-30 09:46:37 [unquantized.py:341] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=1007) INFO 07-30 09:46:38 [gpu_model_runner.py:5132] Model loading took 57.94 GiB memory and 58.184261 seconds
(Worker_TP0 pid=1007) INFO 07-30 09:46:38 [interface.py:649] Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size.
(Worker_TP0 pid=1007) INFO 07-30 09:46:38 [interface.py:673] Padding mamba page size by 1.34% to ensure that mamba page size and attention page size are exactly equal.
(Worker_TP1 pid=1008) INFO 07-30 09:46:39 [gpu_model_runner.py:6109] Skipping memory profiling for multimodal encoder and encoder cache.
(Worker_TP3 pid=1010) INFO 07-30 09:46:39 [gpu_model_runner.py:6109] Skipping memory profiling for multimodal encoder and encoder cache.
(Worker_TP2 pid=1009) INFO 07-30 09:46:39 [gpu_model_runner.py:6109] Skipping memory profiling for multimodal encoder and encoder cache.
(Worker_TP0 pid=1007) INFO 07-30 09:46:39 [gpu_model_runner.py:6109] Skipping memory profiling for multimodal encoder and encoder cache.
(EngineCore pid=739) INFO 07-30 09:47:39 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=739) INFO 07-30 09:48:39 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
