能不能提供一个基于k8s 1.20的版本？
gxwangit
- Members
- Joined 2026年7月13日
-
bash /opt/maca/samples/mccl_tests/perf/mccl.sh 8
The test is all_reduce_perf, the maca version is /opt/maca-3.7.0
main_process = 69813
===============================nThread 1 nGpus 1 minBytes 1024 maxBytes 1073741824 step: 2(factor) warmup iters: 5 iters: 10 agg iters: 1 validation: 1 graph: 0
Using devices
Rank 0 Pid 69813 on cloudos device 0 [0x23] MetaX C550
Rank 1 Pid 69814 on cloudos device 1 [0x26] MetaX C550
Rank 2 Pid 69815 on cloudos device 2 [0x63] MetaX C550
Rank 3 Pid 69816 on cloudos device 3 [0x66] MetaX C550
Rank 4 Pid 69817 on cloudos device 4 [0xa3] MetaX C550
Rank 5 Pid 69818 on cloudos device 5 [0xa4] MetaX C550
Rank 6 Pid 69819 on cloudos device 6 [0xe3] MetaX C550
Rank 7 Pid 69820 on cloudos device 7 [0xe4] MetaX C550
┌----- out-of-place ------┐ ┌------ in-place -------┐
size count type redop root time algbw busbw #wrong time algbw busbw #wrong
(B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
1024 512 bfloat16 sum -1 22.49 0.05 0.08 0 19.24 0.05 0.09 0 2048 1024 bfloat16 sum -1 19.91 0.10 0.18 0 16.02 0.13 0.22 0 4096 2048 bfloat16 sum -1 16.20 0.25 0.44 0 16.64 0.25 0.43 0 8192 4096 bfloat16 sum -1 16.41 0.50 0.87 0 15.92 0.51 0.90 0 16384 8192 bfloat16 sum -1 16.24 1.01 1.77 0 16.09 1.02 1.78 0 32768 16384 bfloat16 sum -1 13.03 2.51 4.40 0 12.81 2.56 4.48 0 65536 32768 bfloat16 sum -1 14.24 4.60 8.06 0 13.92 4.71 8.24 0 131072 65536 bfloat16 sum -1 25.80 5.08 8.89 0 23.94 5.47 9.58 0 262144 131072 bfloat16 sum -1 21.65 12.11 21.19 0 21.74 12.06 21.10 0 524288 262144 bfloat16 sum -1 25.16 20.84 36.47 0 24.72 21.21 37.12 0 1048576 524288 bfloat16 sum -1 25.53 41.07 71.87 0 25.48 41.16 72.02 0 2097152 1048576 bfloat16 sum -1 34.11 61.49 107.60 0 36.48 57.48 100.60 0 4194304 2097152 bfloat16 sum -1 55.32 75.82 132.68 0 54.95 76.33 133.58 0 8388608 4194304 bfloat16 sum -1 96.43 87.00 152.24 0 97.45 86.08 150.65 0 16777216 8388608 bfloat16 sum -1 155.22 108.08 189.15 0 156.93 106.91 187.09 0 33554432 16777216 bfloat16 sum -1 267.63 125.38 219.41 0 271.71 123.49 216.11 0 67108864 33554432 bfloat16 sum -1 484.47 138.52 242.41 0 489.52 137.09 239.91 0134217728 67108864 bfloat16 sum -1 904.91 148.32 259.56 0 904.97 148.31 259.54 0
268435456 134217728 bfloat16 sum -1 2200.85 121.97 213.45 0 2204.07 121.79 213.13 0
536870912 268435456 bfloat16 sum -1 4334.63 123.86 216.75 0 4330.16 123.98 216.97 0
1073741824 536870912 bfloat16 sum -1 8710.96 123.26 215.71 0 8753.79 122.66 214.66 0Out of bounds values : 0 OK
Avg bus bandwidth : 99.7952
-
沐曦卡是c550
安装device-plugin和gpu-label
/opt/ai-deployer/bin/helm install metax-operator -n kube-system metax-operator/
安装export
kubectl apply -f mx-exporter/deployment/namespace.yaml
kubectl apply -f mx-exporter/deployment/mx-exporter/mx-exporter-daemonset.yaml查看相关容器的状态
root@ubuntuh3c:/opt/metax-gpu# kubectl get pods -A |grep metax
kube-system metax-container-runtime-ztspk 1/1 Running 0 66m
kube-system metax-driver-5jl24 1/1 Running 0 66m
kube-system metax-gpu-device-bt55c 1/1 Running 0 65m
kube-system metax-gpu-label-4wpb7 1/1 Running 0 66m
kube-system metax-maca-nvs6b 1/1 Running 0 66m
kube-system metax-operator-b88c9fb87-bwmhp 1/1 Running 0 66m
kube-system metax-operator-pre-delete-jkjpp 0/1 Completed 0 68m
kube-system metax-pre-delete-gtwq8 0/1 Completed 0 4d4h
metax-monitor metax-mx-exporter-pwjwv 1/1 Running 0 94m
root@ubuntuh3c:/opt/metax-gpu# kubectl describe node 10.10.5.32
Capacity:
cpu: 160
ephemeral-storage: 920729380Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 2113444596Ki
metax-tech.com/gpu: 8
metax-tech.com/vfio-gpu: 0
pods: 200
Allocatable:
cpu: 160
ephemeral-storage: 848544195204
现在生成一个可用的pod之后 去跑vllm 单卡可以，多卡直接报错
vllm serve \
/ai/abc/models/Qwen3.5-35B-A3B \
-tp 4 \
--gpu-memory-utilization 0.9 \
--trust-remote-code \
--max-model-len 8192 \
--dtype=bfloat16 \
--max-num-batched-tokens 4096 \
--max-num-seqs 32 \
--no-enable-prefix-caching
INFO 07-10 11:04:17 [init.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-10 11:04:17 [init.py:46] - metax -> vllm_metax:register
INFO 07-10 11:04:17 [init.py:49] All plugins in this group will be loaded. SetVLLM_PLUGINSto control which plugins to load.
INFO 07-10 11:04:17 [init.py:238] Platform plugin metax is activated
INFO 07-10 11:04:17 [envs.py:116] Plugin sets VLLM_USE_FLASHINFER_SAMPLER to False. Reason: flashinfer sampler are not supported on maca
INFO 07-10 11:04:17 [envs.py:116] Plugin sets VLLM_ENGINE_READY_TIMEOUT_S to 3600. Reason: set timeout to 3600s for model loading
INFO 07-10 11:04:17 [envs.py:116] Plugin sets VLLM_FLOAT32_MATMUL_PRECISION to high. Reason: set float32 matmul precision to high for better performance on Maca platform
INFO 07-10 11:04:17 [envs.py:116] Plugin sets VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS to 0. Reason: the feature can't estimate cuda graph size correctly
INFO Print the version information of mcoplib during compilation.Version info:Mcoplib_Version = '0.4.5'
Build_Maca_Version = '3.7.0.37'
GIT_BRANCH = 'HEAD'
GIT_COMMIT = 'a9125e4'
Vllm Op Version = 0.20.0
SGlang Op Version = 0.5.10INFO Staring Check the current MACA version of the operating environment.
INFO: Release major.minor matching, successful:3.7.
INFO 07-10 11:04:23 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL.
INFO 07-10 11:04:23 [nixl_utils.py:32] NIXL is available
WARNING 07-10 11:04:49 [init.py:87] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq.MacaAWQConfig'>.
WARNING 07-10 11:04:49 [init.py:87] The quantization method 'awq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq_marlin.MacaAWQMarlinConfig'>.
WARNING 07-10 11:04:49 [init.py:87] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.compressed_tensors.MacaCompressedTensorsConfig'>.
WARNING 07-10 11:04:49 [init.py:87] The quantization method 'gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq.MacaGPTQConfig'>.
WARNING 07-10 11:04:49 [init.py:87] The quantization method 'gptq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq_marlin.MacaGPTQMarlinConfig'>.
WARNING 07-10 11:04:49 [init.py:87] The quantization method 'moe_wna16' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.moe_wna16.MacaMoeWNA16Config'>.
WARNING 07-10 11:04:49 [registry.py:945] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_mtp:DeepSeekMTP.
WARNING 07-10 11:04:49 [registry.py:945] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV2ForCausalLM.
WARNING 07-10 11:04:49 [registry.py:945] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-10 11:04:49 [registry.py:945] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-10 11:04:49 [registry.py:945] Model architecture KimiK25ForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.kimi_k25:KimiK25ForConditionalGeneration.
WARNING 07-10 11:04:49 [registry.py:945] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:GlmMoeDsaForCausalLM.
WARNING 07-10 11:04:49 [registry.py:945] Model architecture DeepseekV4ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4:DeepseekV4ForCausalLM.
WARNING 07-10 11:04:49 [registry.py:945] Model architecture MiniMaxM2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.minimax_m2:MiniMaxM2ForCausalLM.
WARNING 07-10 11:04:49 [registry.py:945] Model architecture Qwen3VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_vl:Qwen3VLForConditionalGeneration.
WARNING 07-10 11:04:49 [registry.py:945] Model architecture Qwen3VLMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_vl_moe:Qwen3VLMoeForConditionalGeneration.
(APIServer pid=147) INFO 07-10 11:04:50 [utils.py:299]
(APIServer pid=147) INFO 07-10 11:04:50 [utils.py:299] █ █ █▄ ▄█
(APIServer pid=147) INFO 07-10 11:04:50 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.20.0
(APIServer pid=147) INFO 07-10 11:04:50 [utils.py:299] █▄█▀ █ █ █ █ model /ai/abc/models/Qwen3.5-35B-A3B
(APIServer pid=147) INFO 07-10 11:04:50 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=147) INFO 07-10 11:04:50 [utils.py:299]
(APIServer pid=147) INFO 07-10 11:04:50 [utils.py:233] non-default args: {'model_tag': '/ai/abc/models/Qwen3.5-35B-A3B', 'model': '/ai/abc/models/Qwen3.5-35B-A3B', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_model_len': 8192, 'tensor_parallel_size': 4, 'gpu_memory_utilization': 0.9, 'enable_prefix_caching': False, 'max_num_batched_tokens': 4096, 'max_num_seqs': 32}
(APIServer pid=147) WARNING 07-10 11:04:50 [envs.py:1818] Unknown vLLM environment variable detected: VLLM_USE_METAX
(APIServer pid=147) INFO 07-10 11:05:38 [model.py:555] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=147) INFO 07-10 11:05:38 [model.py:1680] Using max model len 8192
(APIServer pid=147) INFO 07-10 11:05:38 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=4096.
(APIServer pid=147) INFO 07-10 11:05:38 [vllm.py:840] Asynchronous scheduling is disabled.
(APIServer pid=147) INFO 07-10 11:05:38 [kernel.py:205] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
(APIServer pid=147) INFO 07-10 11:05:38 [envs.py:116] Plugin sets VLLM_TUNED_CONFIG_FOLDER to /opt/conda/lib/python3.12/site-packages/vllm_metax/model_executor/layers/fused_moe/configs/H=2048. Reason: set FusedMoE tuned config dir by hidden_size=2048
(APIServer pid=147) [transformers]Qwen2VLImageProcessorFastis deprecated. TheFastsuffix for image processors has been removed; useQwen2VLImageProcessorinstead.
(APIServer pid=147) [transformers] Theuse_fastparameter is deprecated and will be removed in a future version. Usebackend="torchvision"instead ofuse_fast=True, orbackend="pil"instead ofuse_fast=False.
/opt/conda/lib/python3.12/site-packages/torchvision/datapoints/init.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: github.com/pytorch/vision/issues/6753, and you can also check out github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
/opt/conda/lib/python3.12/site-packages/torchvision/transforms/v2/init.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: github.com/pytorch/vision/issues/6753, and you can also check out github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)
INFO 07-10 11:05:55 [init.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-10 11:05:55 [init.py:46] - metax -> vllm_metax:register
INFO 07-10 11:05:55 [init.py:49] All plugins in this group will be loaded. SetVLLM_PLUGINSto control which plugins to load.
INFO 07-10 11:05:55 [init.py:238] Platform plugin metax is activated
INFO 07-10 11:05:55 [envs.py:116] Plugin sets VLLM_USE_FLASHINFER_SAMPLER to False. Reason: flashinfer sampler are not supported on maca
INFO 07-10 11:05:55 [envs.py:116] Plugin sets VLLM_ENGINE_READY_TIMEOUT_S to 3600. Reason: set timeout to 3600s for model loading
INFO 07-10 11:05:55 [envs.py:116] Plugin sets VLLM_FLOAT32_MATMUL_PRECISION to high. Reason: set float32 matmul precision to high for better performance on Maca platform
INFO 07-10 11:05:55 [envs.py:116] Plugin sets VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS to 0. Reason: the feature can't estimate cuda graph size correctly
(EngineCore pid=661) INFO Print the version information of mcoplib during compilation.
(EngineCore pid=661)
(EngineCore pid=661) Version info:Mcoplib_Version = '0.4.5'
(EngineCore pid=661) Build_Maca_Version = '3.7.0.37'
(EngineCore pid=661) GIT_BRANCH = 'HEAD'
(EngineCore pid=661) GIT_COMMIT = 'a9125e4'
(EngineCore pid=661) Vllm Op Version = 0.20.0
(EngineCore pid=661) SGlang Op Version = 0.5.10
(EngineCore pid=661)
(EngineCore pid=661) INFO Staring Check the current MACA version of the operating environment.
(EngineCore pid=661)
(EngineCore pid=661) INFO: Release major.minor matching, successful:3.7.
(EngineCore pid=661)
(EngineCore pid=661) INFO 07-10 11:05:58 [nixl_utils.py:32] NIXL is available
(EngineCore pid=661) WARNING 07-10 11:06:26 [init.py:87] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq.MacaAWQConfig'>.
(EngineCore pid=661) WARNING 07-10 11:06:26 [init.py:87] The quantization method 'awq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq_marlin.MacaAWQMarlinConfig'>.
(EngineCore pid=661) WARNING 07-10 11:06:26 [init.py:87] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.compressed_tensors.MacaCompressedTensorsConfig'>.
(EngineCore pid=661) WARNING 07-10 11:06:26 [init.py:87] The quantization method 'gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq.MacaGPTQConfig'>.
(EngineCore pid=661) WARNING 07-10 11:06:26 [init.py:87] The quantization method 'gptq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq_marlin.MacaGPTQMarlinConfig'>.
(EngineCore pid=661) WARNING 07-10 11:06:26 [init.py:87] The quantization method 'moe_wna16' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.moe_wna16.MacaMoeWNA16Config'>.
(EngineCore pid=661) WARNING 07-10 11:06:27 [registry.py:945] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_mtp:DeepSeekMTP.
(EngineCore pid=661) WARNING 07-10 11:06:27 [registry.py:945] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV2ForCausalLM.
(EngineCore pid=661) WARNING 07-10 11:06:27 [registry.py:945] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
(EngineCore pid=661) WARNING 07-10 11:06:27 [registry.py:945] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
(EngineCore pid=661) WARNING 07-10 11:06:27 [registry.py:945] Model architecture KimiK25ForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.kimi_k25:KimiK25ForConditionalGeneration.
(EngineCore pid=661) WARNING 07-10 11:06:27 [registry.py:945] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:GlmMoeDsaForCausalLM.
(EngineCore pid=661) WARNING 07-10 11:06:27 [registry.py:945] Model architecture DeepseekV4ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4:DeepseekV4ForCausalLM.
(EngineCore pid=661) WARNING 07-10 11:06:27 [registry.py:945] Model architecture MiniMaxM2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.minimax_m2:MiniMaxM2ForCausalLM.
(EngineCore pid=661) WARNING 07-10 11:06:27 [registry.py:945] Model architecture Qwen3VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_vl:Qwen3VLForConditionalGeneration.
(EngineCore pid=661) WARNING 07-10 11:06:27 [registry.py:945] Model architecture Qwen3VLMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_vl_moe:Qwen3VLMoeForConditionalGeneration.
(EngineCore pid=661) INFO 07-10 11:06:27 [core.py:109] Initializing a V1 LLM engine (v0.20.0) with config: model='/ai/abc/models/Qwen3.5-35B-A3B', speculative_config=None, tokenizer='/ai/abc/models/Qwen3.5-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/ai/abc/models/Qwen3.5-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::mx_sparse_attn_indexer', 'vllm::mx_sparse_attn_indexer_bf16', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto')
(EngineCore pid=661) WARNING 07-10 11:06:27 [multiproc_executor.py:1029] Reducing Torch parallelism from 255 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=661) INFO 07-10 11:06:27 [multiproc_executor.py:139] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=162.30.0.113 (local), world_size=4, local_world_size=4
INFO 07-10 11:06:30 [init.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-10 11:06:30 [init.py:46] - metax -> vllm_metax:register
INFO 07-10 11:06:30 [init.py:49] All plugins in this group will be loaded. SetVLLM_PLUGINSto control which plugins to load.
INFO 07-10 11:06:30 [init.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-10 11:06:30 [init.py:46] - metax -> vllm_metax:register
INFO 07-10 11:06:30 [init.py:49] All plugins in this group will be loaded. SetVLLM_PLUGINSto control which plugins to load.
INFO 07-10 11:06:30 [init.py:238] Platform plugin metax is activated
INFO 07-10 11:06:30 [init.py:238] Platform plugin metax is activated
INFO 07-10 11:06:30 [init.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-10 11:06:30 [init.py:46] - metax -> vllm_metax:register
INFO 07-10 11:06:30 [init.py:49] All plugins in this group will be loaded. SetVLLM_PLUGINSto control which plugins to load.
INFO 07-10 11:06:30 [init.py:238] Platform plugin metax is activated
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_USE_FLASHINFER_SAMPLER to False. Reason: flashinfer sampler are not supported on maca
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_ENGINE_READY_TIMEOUT_S to 3600. Reason: set timeout to 3600s for model loading
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_FLOAT32_MATMUL_PRECISION to high. Reason: set float32 matmul precision to high for better performance on Maca platform
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS to 0. Reason: the feature can't estimate cuda graph size correctly
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_USE_FLASHINFER_SAMPLER to False. Reason: flashinfer sampler are not supported on maca
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_ENGINE_READY_TIMEOUT_S to 3600. Reason: set timeout to 3600s for model loading
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_FLOAT32_MATMUL_PRECISION to high. Reason: set float32 matmul precision to high for better performance on Maca platform
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS to 0. Reason: the feature can't estimate cuda graph size correctly
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_USE_FLASHINFER_SAMPLER to False. Reason: flashinfer sampler are not supported on maca
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_ENGINE_READY_TIMEOUT_S to 3600. Reason: set timeout to 3600s for model loading
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_FLOAT32_MATMUL_PRECISION to high. Reason: set float32 matmul precision to high for better performance on Maca platform
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS to 0. Reason: the feature can't estimate cuda graph size correctly
INFO 07-10 11:06:30 [init.py:44] Available plugins for group vllm.platform_plugins:
INFO 07-10 11:06:30 [init.py:46] - metax -> vllm_metax:register
INFO 07-10 11:06:30 [init.py:49] All plugins in this group will be loaded. SetVLLM_PLUGINSto control which plugins to load.
INFO 07-10 11:06:30 [init.py:238] Platform plugin metax is activated
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_USE_FLASHINFER_SAMPLER to False. Reason: flashinfer sampler are not supported on maca
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_ENGINE_READY_TIMEOUT_S to 3600. Reason: set timeout to 3600s for model loading
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_FLOAT32_MATMUL_PRECISION to high. Reason: set float32 matmul precision to high for better performance on Maca platform
INFO 07-10 11:06:30 [envs.py:116] Plugin sets VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS to 0. Reason: the feature can't estimate cuda graph size correctly
INFO Print the version information of mcoplib during compilation.Version info:Mcoplib_Version = '0.4.5'
Build_Maca_Version = '3.7.0.37'
GIT_BRANCH = 'HEAD'
GIT_COMMIT = 'a9125e4'
Vllm Op Version = 0.20.0
SGlang Op Version = 0.5.10INFO Staring Check the current MACA version of the operating environment.
INFO: Release major.minor matching, successful:3.7.
INFO Print the version information of mcoplib during compilation.
Version info:Mcoplib_Version = '0.4.5'
Build_Maca_Version = '3.7.0.37'
GIT_BRANCH = 'HEAD'
GIT_COMMIT = 'a9125e4'
Vllm Op Version = 0.20.0
SGlang Op Version = 0.5.10INFO Staring Check the current MACA version of the operating environment.
INFO: Release major.minor matching, successful:3.7.
INFO Print the version information of mcoplib during compilation.
Version info:Mcoplib_Version = '0.4.5'
Build_Maca_Version = '3.7.0.37'
GIT_BRANCH = 'HEAD'
GIT_COMMIT = 'a9125e4'
Vllm Op Version = 0.20.0
SGlang Op Version = 0.5.10INFO Staring Check the current MACA version of the operating environment.
INFO: Release major.minor matching, successful:3.7.
INFO 07-10 11:06:36 [nixl_utils.py:32] NIXL is available
INFO Print the version information of mcoplib during compilation.Version info:Mcoplib_Version = '0.4.5'
Build_Maca_Version = '3.7.0.37'
GIT_BRANCH = 'HEAD'
GIT_COMMIT = 'a9125e4'
Vllm Op Version = 0.20.0
SGlang Op Version = 0.5.10INFO Staring Check the current MACA version of the operating environment.
INFO: Release major.minor matching, successful:3.7.
INFO 07-10 11:06:36 [nixl_utils.py:32] NIXL is available
INFO 07-10 11:06:36 [nixl_utils.py:32] NIXL is available
INFO 07-10 11:06:36 [nixl_utils.py:32] NIXL is available
WARNING 07-10 11:07:03 [init.py:87] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq.MacaAWQConfig'>.
WARNING 07-10 11:07:03 [init.py:87] The quantization method 'awq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq_marlin.MacaAWQMarlinConfig'>.
WARNING 07-10 11:07:04 [init.py:87] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.compressed_tensors.MacaCompressedTensorsConfig'>.
WARNING 07-10 11:07:04 [init.py:87] The quantization method 'gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq.MacaGPTQConfig'>.
WARNING 07-10 11:07:04 [init.py:87] The quantization method 'gptq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq_marlin.MacaGPTQMarlinConfig'>.
WARNING 07-10 11:07:04 [init.py:87] The quantization method 'moe_wna16' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.moe_wna16.MacaMoeWNA16Config'>.
WARNING 07-10 11:07:04 [registry.py:945] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_mtp:DeepSeekMTP.
WARNING 07-10 11:07:04 [registry.py:945] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV2ForCausalLM.
WARNING 07-10 11:07:04 [registry.py:945] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-10 11:07:04 [registry.py:945] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-10 11:07:04 [registry.py:945] Model architecture KimiK25ForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.kimi_k25:KimiK25ForConditionalGeneration.
WARNING 07-10 11:07:04 [registry.py:945] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:GlmMoeDsaForCausalLM.
WARNING 07-10 11:07:04 [registry.py:945] Model architecture DeepseekV4ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4:DeepseekV4ForCausalLM.
WARNING 07-10 11:07:04 [registry.py:945] Model architecture MiniMaxM2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.minimax_m2:MiniMaxM2ForCausalLM.
WARNING 07-10 11:07:04 [registry.py:945] Model architecture Qwen3VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_vl:Qwen3VLForConditionalGeneration.
WARNING 07-10 11:07:04 [registry.py:945] Model architecture Qwen3VLMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_vl_moe:Qwen3VLMoeForConditionalGeneration.
[transformers]Qwen2VLImageProcessorFastis deprecated. TheFastsuffix for image processors has been removed; useQwen2VLImageProcessorinstead.
WARNING 07-10 11:07:05 [init.py:87] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq.MacaAWQConfig'>.
WARNING 07-10 11:07:05 [init.py:87] The quantization method 'awq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq_marlin.MacaAWQMarlinConfig'>.
WARNING 07-10 11:07:05 [init.py:87] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.compressed_tensors.MacaCompressedTensorsConfig'>.
WARNING 07-10 11:07:05 [init.py:87] The quantization method 'gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq.MacaGPTQConfig'>.
WARNING 07-10 11:07:05 [init.py:87] The quantization method 'gptq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq_marlin.MacaGPTQMarlinConfig'>.
WARNING 07-10 11:07:05 [init.py:87] The quantization method 'moe_wna16' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.moe_wna16.MacaMoeWNA16Config'>.
WARNING 07-10 11:07:05 [init.py:87] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq.MacaAWQConfig'>.
WARNING 07-10 11:07:05 [init.py:87] The quantization method 'awq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq_marlin.MacaAWQMarlinConfig'>.
WARNING 07-10 11:07:05 [registry.py:945] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_mtp:DeepSeekMTP.
WARNING 07-10 11:07:05 [registry.py:945] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV2ForCausalLM.
WARNING 07-10 11:07:05 [registry.py:945] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-10 11:07:05 [registry.py:945] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-10 11:07:05 [registry.py:945] Model architecture KimiK25ForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.kimi_k25:KimiK25ForConditionalGeneration.
WARNING 07-10 11:07:05 [registry.py:945] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:GlmMoeDsaForCausalLM.
WARNING 07-10 11:07:05 [registry.py:945] Model architecture DeepseekV4ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4:DeepseekV4ForCausalLM.
WARNING 07-10 11:07:05 [registry.py:945] Model architecture MiniMaxM2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.minimax_m2:MiniMaxM2ForCausalLM.
WARNING 07-10 11:07:05 [registry.py:945] Model architecture Qwen3VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_vl:Qwen3VLForConditionalGeneration.
WARNING 07-10 11:07:05 [registry.py:945] Model architecture Qwen3VLMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_vl_moe:Qwen3VLMoeForConditionalGeneration.
[transformers]Qwen2VLImageProcessorFastis deprecated. TheFastsuffix for image processors has been removed; useQwen2VLImageProcessorinstead.
(Worker pid=902) INFO 07-10 11:07:05 [parallel_state.py:1402] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:59623 backend=nccl
WARNING 07-10 11:07:06 [init.py:87] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq.MacaAWQConfig'>.
WARNING 07-10 11:07:06 [init.py:87] The quantization method 'awq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.awq_marlin.MacaAWQMarlinConfig'>.
WARNING 07-10 11:07:06 [init.py:87] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.compressed_tensors.MacaCompressedTensorsConfig'>.
WARNING 07-10 11:07:06 [init.py:87] The quantization method 'gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq.MacaGPTQConfig'>.
WARNING 07-10 11:07:06 [init.py:87] The quantization method 'gptq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq_marlin.MacaGPTQMarlinConfig'>.
WARNING 07-10 11:07:06 [init.py:87] The quantization method 'moe_wna16' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.moe_wna16.MacaMoeWNA16Config'>.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_mtp:DeepSeekMTP.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV2ForCausalLM.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture KimiK25ForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.kimi_k25:KimiK25ForConditionalGeneration.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:GlmMoeDsaForCausalLM.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture DeepseekV4ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4:DeepseekV4ForCausalLM.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture MiniMaxM2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.minimax_m2:MiniMaxM2ForCausalLM.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture Qwen3VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_vl:Qwen3VLForConditionalGeneration.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture Qwen3VLMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_vl_moe:Qwen3VLMoeForConditionalGeneration.
[transformers]Qwen2VLImageProcessorFastis deprecated. TheFastsuffix for image processors has been removed; useQwen2VLImageProcessorinstead.
WARNING 07-10 11:07:06 [init.py:87] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.compressed_tensors.MacaCompressedTensorsConfig'>.
WARNING 07-10 11:07:06 [init.py:87] The quantization method 'gptq' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq.MacaGPTQConfig'>.
WARNING 07-10 11:07:06 [init.py:87] The quantization method 'gptq_marlin' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.gptq_marlin.MacaGPTQMarlinConfig'>.
WARNING 07-10 11:07:06 [init.py:87] The quantization method 'moe_wna16' already exists and will be overwritten by the quantization config <class 'vllm_metax.quant_config.moe_wna16.MacaMoeWNA16Config'>.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_mtp:DeepSeekMTP.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV2ForCausalLM.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:DeepseekV3ForCausalLM.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture KimiK25ForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.kimi_k25:KimiK25ForConditionalGeneration.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v2:GlmMoeDsaForCausalLM.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture DeepseekV4ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.deepseek_v4:DeepseekV4ForCausalLM.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture MiniMaxM2ForCausalLM is already registered, and will be overwritten by the new model class vllm_metax.models.minimax_m2:MiniMaxM2ForCausalLM.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture Qwen3VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_vl:Qwen3VLForConditionalGeneration.
WARNING 07-10 11:07:06 [registry.py:945] Model architecture Qwen3VLMoeForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_metax.models.qwen3_vl_moe:Qwen3VLMoeForConditionalGeneration.
[transformers]Qwen2VLImageProcessorFastis deprecated. TheFastsuffix for image processors has been removed; useQwen2VLImageProcessorinstead.
(Worker pid=900) INFO 07-10 11:07:07 [parallel_state.py:1402] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:59623 backend=nccl
(Worker pid=903) INFO 07-10 11:07:08 [parallel_state.py:1402] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:59623 backend=nccl
(Worker pid=901) INFO 07-10 11:07:08 [parallel_state.py:1402] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:59623 backend=nccl
(Worker pid=900) [rank0]:W0710 11:07:08.570000 900 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
(Worker pid=900) [rank0]:W0710 11:07:08.570000 900 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
(Worker pid=903) [rank3]:W0710 11:07:08.571000 903 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
(Worker pid=903) [rank3]:W0710 11:07:08.571000 903 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
(Worker pid=901) [rank1]:W0710 11:07:08.571000 901 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
(Worker pid=901) [rank1]:W0710 11:07:08.571000 901 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
(Worker pid=902) [rank2]:W0710 11:07:08.572000 902 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
(Worker pid=902) [rank2]:W0710 11:07:08.572000 902 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
(Worker pid=900) INFO 07-10 11:07:08 [mccl.py:27] Found mccl from library libmccl.so
(Worker pid=900) INFO 07-10 11:07:08 [pynccl.py:111] vLLM is using nccl==2.16.5
!!!!!!! Segfault encountered !!!!!!!!!!!!!! Segfault encountered !!!!!!!
!!!!!!! Segfault encountered !!!!!!!
!!!!!!! Segfault encountered !!!!!!!
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] EngineCore failed to start.
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] Traceback (most recent call last):
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] engine_core = EngineCoreProc(args, engine_index=dp_rank, kwargs)
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] return func(args, kwargs)
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 876, in init
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] super().init(
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 118, in init
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] self.model_executor = executor_class(vllm_config)
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 107, in init
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] super().init(vllm_config)
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] return func(*args, kwargs)
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 109, in init
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] self._init_executor()
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 200, in _init_executor
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 747, in wait_for_ready
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] raise e from None
(EngineCore pid=661) ERROR 07-10 11:07:14 [core.py:1136] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
(EngineCore pid=661) Process EngineCore:
(EngineCore pid=661) Traceback (most recent call last):
(EngineCore pid=661) File "/opt/conda/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=661) self.run()
(EngineCore pid=661) File "/opt/conda/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=661) self._target(self._args, self._kwargs)
(EngineCore pid=661) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1140, in run_engine_core
(EngineCore pid=661) raise e
(EngineCore pid=661) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core
(EngineCore pid=661) engine_core = EngineCoreProc(args, engine_index=dp_rank, kwargs)
(EngineCore pid=661) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=661) File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=661) return func(*args, kwargs)
(EngineCore pid=661) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=661) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 876, in init
(EngineCore pid=661) super().init(
(EngineCore pid=661) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 118, in init
(EngineCore pid=661) self.model_executor = executor_class(vllm_config)
(EngineCore pid=661) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=661) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 107, in init
(EngineCore pid=661) super().init(vllm_config)
(EngineCore pid=661) File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=661) return func(args, kwargs)
(EngineCore pid=661) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=661) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 109, in init
(EngineCore pid=661) self._init_executor()
(EngineCore pid=661) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 200, in _init_executor
(EngineCore pid=661) self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore pid=661) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=661) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 747, in wait_for_ready
(EngineCore pid=661) raise e from None
(EngineCore pid=661) Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
(APIServer pid=147) Traceback (most recent call last):
(APIServer pid=147) File "/opt/conda/bin/vllm", line 8, in <module>
(APIServer pid=147) sys.exit(main())
(APIServer pid=147) ^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=147) args.dispatch_function(args)
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=147) uvloop.run(run_server(args))
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/uvloop/init.py", line 96, in run
(APIServer pid=147) return asyncio.run(
(APIServer pid=147) ^^^^^^^^^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=147) return runner.run(main)
(APIServer pid=147) ^^^^^^^^^^^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=147) return self._loop.run_until_complete(task)
(APIServer pid=147) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=147) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/uvloop/__init.py", line 48, in wrapper
(APIServer pid=147) return await main
(APIServer pid=147) ^^^^^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 678, in run_server
(APIServer pid=147) await run_server_worker(listen_address, sock, args, uvicorn_kwargs)
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 692, in run_server_worker
(APIServer pid=147) async with build_async_engine_client(
(APIServer pid=147) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=147) return await anext(self.gen)
(APIServer pid=147) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=147) async with build_async_engine_client_from_engine_args(
(APIServer pid=147) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=147) return await anext(self.gen)
(APIServer pid=147) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=147) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=147) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=147) return cls(
(APIServer pid=147) ^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 146, in init
(APIServer pid=147) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=147) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=147) return func(args, kwargs)
(APIServer pid=147) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=147) return AsyncMPClient(client_args)
(APIServer pid=147) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=147) return func(args, kwargs)
(APIServer pid=147) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 900, in init
(APIServer pid=147) super().init(
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 535, in init
(APIServer pid=147) with launch_core_engines(
(APIServer pid=147) ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=147) File "/opt/conda/lib/python3.12/contextlib.py", line 144, in exit
(APIServer pid=147) next(self.gen)
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1119, in launch_core_engines
(APIServer pid=147) wait_for_engine_startup(
(APIServer pid=147) File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1178, in wait_for_engine_startup
(APIServer pid=147) raise RuntimeError(
(APIServer pid=147) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}