启动了,但是推理报错了: INFO: Started server process [8911]
(APIServer pid=8911) INFO: Waiting for application startup.
(APIServer pid=8911) INFO: Application startup complete.
(APIServer pid=8911) INFO 11-02 13:35:45 [tokenizer.py:24] IndexTTS2Tokenizer.init: vocab_file=/app/models/IndexTTS-2/bpe.model
(APIServer pid=8911) INFO 11-02 13:35:45 [tokenizer.py:27] IndexTTS2Tokenizer initialized successfully
(APIServer pid=8911) INFO 11-02 13:35:45 [serving_speech.py:3296] TTS speech request speech-813de70907d00c0b: text='你好,欢迎使用语音克隆测试。', model=indextts2
(APIServer pid=8911) WARNING 11-02 13:35:45 [base.py:313] Using None for EOS token id because tokenizer is not initialized
[13:35:58.553][MCR][E]mx_device.cpp :11000: Fail to allocate device memory
[13:35:58.554][MCR][E]mx_device.cpp :11186: Fail to allocate device memory
[13:35:58.554][MCR][E]mc_platform.cpp :589 : Fail to allocate device memory
[13:35:58.554][MCR][E]mc_runtime_api.cpp :349 : 9211 : [7fb4c907f740] mcLaunchKernel: Returned mcErrorInvalidValue
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.26.0) with config: model='/app/models/IndexTTS-2', speculative_config=None, tokenizer='gpt2', skip_tokenizer_init=True, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2560, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/app/models/IndexTTS-2, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto'),
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [dump_input.py:79] Dumping scheduler output for model execution: OmniSchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=speech-813de70907d00c0b-8cf12697,prompt_token_ids_len=64,prefill_token_ids_len=None,mm_features=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=10.0, temperature=0.8, top_p=0.8, top_k=30, min_p=0.0, seed=None, stop=[], stop_token_ids=[8193], bad_words=[], thinking_token_budget=None, include_stop_str_in_output=False, ignore_eos=False, max_tokens=1500, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, structured_outputs=None, extra_args=None),block_ids=([1, 2, 3, 4],),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[],num_computed_tokens=[],num_output_tokens=[]), num_scheduled_tokens={speech-813de70907d00c0b-8cf12697: 64}, total_num_scheduled_tokens=64, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0], finished_req_ids=[], free_encoder_mm_hashes=[], scheduled_encoder_input_stats=null, preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=null, ec_connector_metadata=null, new_block_ids_to_zero=null, kv_cache_block_copies=null, num_spec_tokens_to_schedule=0, finished_requests_needing_kv_transfer={}, pending_input_registrations=[])
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, num_skipped_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.0007135212272565417, iteration_details=None, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=None, kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] StageEngineCoreProc encountered a fatal error.
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] Traceback (most recent call last):
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/stage_engine_core_proc.py", line 183, in run_stage_core
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] engine_core.run_busy_loop()
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1364, in run_busy_loop
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] self._process_engine_step()
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1403, in _process_engine_step
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] outputs, model_executed = self.step_fn()
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 647, in step_with_batch_queue
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] exec_future = self.model_executor.execute_model(
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 120, in execute_model
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] output.result()
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return self.__get_result()
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] raise self._exception
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 98, in collective_rpc
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] result = run_method(self.driver_worker, method, args, kwargs)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return func(args, kwargs)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 351, in execute_model
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return self.worker.execute_model(scheduler_output)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return func(args, kwargs)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1147, in execute_model
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] output = self.model_runner.execute_model(
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return func(*args, kwargs)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/worker/gpu_ar_model_runner.py", line 1264, in execute_model
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ) = self._preprocess(scheduler_output, num_tokens_padded, intermediate_tensors)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/worker/gpu_model_runner.py", line 1774, in _preprocess
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] req_input_ids, req_embeds, update_dict = self.model.preprocess(
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/model_executor/models/indextts2/indextts2_talker.py", line 404, in preprocess
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return self._preprocess_prefill(input_ids, input_embeds, info_dict, span_len)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/model_executor/models/indextts2/indextts2_talker.py", line 500, in _preprocess_prefill
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] spk_cond_emb = wav2vec_extract(wav_16k, w2v_model, w2v_proc, device, self._w2v_stat)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/model_executor/models/indextts2/preprocess_utils.py", line 200, in wav2vec_extract
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] outputs = model(
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return self._call_impl(args, kwargs)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return forward_call(args, kwargs)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/utils/generic.py", line 1068, in wrapper
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] output = func(self, *args, kwargs)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/utils/output_capturing.py", line 262, in wrapper
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] outputs = func(self, args, kwargs)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/models/wav2vec2_bert/modeling_wav2vec2_bert.py", line 1027, in forward
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] encoder_outputs = self.encoder(
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return self._call_impl(args, kwargs)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return forward_call(*args, kwargs)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/models/wav2vec2_bert/modeling_wav2vec2_bert.py", line 527, in forward
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] attention_mask = create_bidirectional_mask(
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/masking_utils.py", line 1083, in create_bidirectional_mask
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] attention_mask = mask_interface(
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 171, in wrapped_func
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return func(args, *kwargs)
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/masking_utils.py", line 500, in sdpa_mask
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] if allow_is_bidirectional_skip and _ignore_bidirectional_mask_sdpa(padding_mask, kv_length, local_size):
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/masking_utils.py", line 330, in _ignore_bidirectional_mask_sdpa
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] and (padding_mask is None or padding_mask.all())
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] torch.AcceleratorError: CUDA error: invalid argument
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] Search for cudaErrorInvalidValue' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] Compile withTORCH_USE_CUDA_DSA` to enable device-side assertions.
(StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192]
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] [StagePool] _poll_stage_raw failed for stage-0 replica-0
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] Traceback (most recent call last):
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/stage_pool.py", line 1136, in poll_llm_raw_output
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] return await asyncio.wait_for(
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] ^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] File "/opt/conda/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] return await fut
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] ^^^^^^^^^
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/stage_pool.py", line 1083, in _poll_stage_raw
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] outputs = await client.get_output_async()
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1061, in get_output_async
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] raise self._format_exception(outputs) from None
(APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=8911) ERROR 11-02 13:35:58 [orchestrator.py:963] [Orchestrator] Stage-0 replica-0 is dead: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=8911) ERROR 11-02 13:35:58 [async_omni.py:926] [AsyncOmni] Engine dead: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=8911) ERROR 11-02 13:35:58 [async_omni.py:814] [AsyncOmni] Orchestrator error for req=speech-813de70907d00c0b-8cf12697 stage-0: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=8911) INFO 11-02 13:35:58 [async_omni.py:1074] [AsyncOmni] Aborted request(s) speech-813de70907d00c0b-8cf12697
(APIServer pid=8911) INFO 11-02 13:35:58 [async_omni.py:650] [AsyncOmni] Request speech-813de70907d00c0b-8cf12697 failed (input error): EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=8911) ERROR 11-02 13:35:58 [serving_speech.py:3875] [SpeechE2E] request_id=speech-813de70907d00c0b stream=False status=engine_error total_ms=14558.34
(APIServer pid=8911) ERROR 11-02 13:35:58 [api_server.py:337] EngineDeadError: orchestrator_alive=True, errored=True, request_id=speech-813de70907d00c0b, error_stage_id=0
(APIServer pid=8911) INFO: Shutting down
(APIServer pid=8911) INFO 11-02 13:35:58 [orchestrator.py:540] [Orchestrator] Engine dead during shutdown: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=8911) INFO 11-02 13:35:58 [orchestrator.py:2258] [Orchestrator] Shutting down all 2 client(s)
(APIServer pid=8911) INFO 11-02 13:35:58 [core_client.py:655] [shutdown] MPClient: start timeout=default
(APIServer pid=8911) INFO 11-02 13:35:58 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(APIServer pid=8911) INFO: 127.0.0.1:49990 - "POST /v1/audio/speech HTTP/1.1" 500 Internal Server Error
(APIServer pid=8911) INFO: Waiting for application shutdown.
(APIServer pid=8911) INFO: Application shutdown complete.
(APIServer pid=8911) INFO: Finished server process [8911]
(APIServer pid=8911) INFO 11-02 13:35:58 [omni_base.py:630] [AsyncOmni] Shutting down
(APIServer pid=8911) INFO 11-02 13:35:58 [async_omni_engine.py:1827] [AsyncOmniEngine] Shutting down Orchestrator
(APIServer pid=8911) INFO 11-02 13:36:01 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=8911) INFO 11-02 13:36:01 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=8911) INFO 11-02 13:36:01 [core_client.py:662] [shutdown] MPClient: complete
(APIServer pid=8911) INFO 11-02 13:36:01 [stage_pool.py:1237] [StagePool] Stage 0 replica 0 shut down
(APIServer pid=8911) INFO 11-02 13:36:01 [core_client.py:655] [shutdown] MPClient: start timeout=default
(APIServer pid=8911) WARNING 11-02 13:36:06 [utils.py:626] [shutdown] Process manager: force killing remaining processes count=1
(APIServer pid=8911) INFO 11-02 13:36:06 [stage_pool.py:1237] [StagePool] Stage 1 replica 0 shut down
/opt/conda/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '