• Members 30 posts
    2026年10月27日 15:50

    一、软硬件信息
    1. 服务器厂家:
    Manufacturer: Advantech
    Product Name: SVH-90
    2. 沐曦GPU型号:
    N260
    3. 操作系统内核版本:
    Ubuntu 22.04.5 LTS 5.15.0-119-generic
    4. 是否开启CPU虚拟化:
    开启
    5. mx-smi回显:
    mx-smi
    mx-smi version: 2.2.8

    =================== MetaX System Management Interface Log ===================
    Timestamp : Tue Oct 27 15:15:55 2026

    Attached GPUs : 6
    +---------------------------------------------------------------------------------+
    | MX-SMI 2.2.8 Kernel Mode Driver Version: 3.0.11 |
    | MACA Version: 3.1.0.14 BIOS Version: 1.27.5.0*
    6. docker info回显:
    docker info
    Client:
    Version: 29.1.3
    Context: default
    Debug Mode: false
    Plugins:
    trust: Manage trust on Docker images (Docker Inc.)
    Version: 29.1.3
    Path: /usr/libexec/docker/cli-plugins/docker-trust

    Server:
    Containers: 27
    Running: 26
    Paused: 0
    Stopped: 1
    Images: 17
    Server Version: 29.1.3
    Storage Driver: overlayfs
    driver-type: io.containerd.snapshotter.v1
    Logging Driver: json-file
    Cgroup Driver: systemd
    Cgroup Version: 2
    Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
    CDI spec directories:
    /etc/cdi
    /var/run/cdi
    Swarm: inactive
    Runtimes: runc io.containerd.runc.v2
    Default Runtime: runc
    Init Binary: docker-init
    containerd version:
    runc version:
    init version:
    Security Options:
    apparmor
    seccomp
    Profile: builtin
    cgroupns
    Kernel Version: 5.15.0-119-generic
    Operating System: Ubuntu 22.04.5 LTS
    OSType: linux
    Architecture: x86_64
    CPUs: 128
    Total Memory: 503.8GiB
    Name: ai-box
    ID: a70d6a29-c033-468e-863e-d9a0cfc43bed
    Docker Root Dir: /zhytm_data/docker
    Debug Mode: false
    Experimental: false
    Insecure Registries:
    172.18.5.249:8082
    ::1/128
    127.0.0.0/8
    Registry Mirrors:
    http://172.18.5.249:8082/artifactory/api/docker/docker-dev-virtual/
    docker.m.daocloud.io/
    Live Restore Enabled: false
    Firewall Backend: iptables

    1. 镜像版本:
      cr.metax-tech.com/public-ai-release/maca/vllm-omni-metax:0.26.0-maca.ai3.8.2.0-torch2.10-py312-ubuntu22.04-amd64
    2. 启动容器命令:
      docker run -d --name tts --privileged --network=host --ipc=host --device=/dev/dri --device=/dev/mxcd --group-add video --restart no --security-opt seccomp=unconfined --security-opt apparmor=unconfined --shm-size=100g -v /zhytm_data/models:/app/models -e MACA_VISIBLE_DEVICES=4 --ulimit memlock=-1 cr.metax-tech.com/public-ai-release/maca/vllm-omni-metax:0.26.0-maca.ai3.8.2.0-torch2.10-py312-ubuntu22.04-amd64 sleep infinity
    3. 容器内执行命令:
      vllm serve /app/models/IndexTTS-2 --omni --trust-remote-code --port 8092
      二、问题现象
      出错了,去缓存目录找w2v-bert-2.0了,不是我这个/app/models/IndexTTS-2 目录下,应该是/app/models/IndexTTS-2/hf_cache/w2v-bert-2.0,是需要加什么参数吗?
      Acquired flock prefetch lock for facebook/w2v-bert-2.0 at /root/.cache/huggingface/locks/vllm-omni-prefetch/facebook__w2v-bert-2.0.lock
      (StageEngineCoreProc_stage0_replica0 pid=1985) '[Errno 101] Network is unreachable' thrown while requesting HEAD huggingface.co/facebook/w2v-bert-2.0/resolve/main/processor_config.json
      (StageEngineCoreProc_stage0_replica0 pid=1985) Retrying in 1s [Retry 1/5].
      (StageEngineCoreProc_stage0_replica0 pid=1985) '[Errno 101] Network is unreachable' thrown while requesting HEAD huggingface.co/facebook/w2v-bert-2.0/resolve/main/processor_config.json
      (StageEngineCoreProc_stage0_replica0 pid=1985) Retrying in 2s [Retry 2/5].
      (StageEngineCoreProc_stage0_replica0 pid=1985) '[Errno 101] Network is unreachable' thrown while requesting HEAD huggingface.co/facebook/w2v-bert-2.0/resolve/main/processor_config.json
      (StageEngineCoreProc_stage0_replica0 pid=1985) Retrying in 4s [Retry 3/5].
      (StageEngineCoreProc_stage0_replica0 pid=1985) '[Errno 101] Network is unreachable' thrown while requesting HEAD huggingface.co/facebook/w2v-bert-2.0/resolve/main/processor_config.json
      (StageEngineCoreProc_stage0_replica0 pid=1985) Retrying in 8s [Retry 4/5].
      (StageEngineCoreProc_stage0_replica0 pid=1985) '[Errno 101] Network is unreachable' thrown while requesting HEAD huggingface.co/facebook/w2v-bert-2.0/resolve/main/processor_config.json
      (StageEngineCoreProc_stage0_replica0 pid=1985) Retrying in 8s [Retry 5/5].
  • Members 1029 posts
    2026年10月27日 16:03

    尊敬的开发者您好,请配置hugging face国内镜像代理下载

  • arrow_forward

    Thread has been moved from 公共.

  • arrow_forward

    Thread has been moved from 解决中.

  • Members 30 posts
    2026年11月2日 11:03

    设置完代理确实这里没问题了,但是后面又有新的错误了,错误如下:
    (StageEngineCoreProc_stage0_replica0 pid=6834) /opt/conda/lib/python3.12/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
    (StageEngineCoreProc_stage0_replica0 pid=6834) WeightNorm.apply(module, name, dim)
    (StageEngineCoreProc_stage0_replica0 pid=6834) WARNING 11-02 10:53:56 [preprocess_utils.py:45] Could not resolve IndexTTS2 asset semantic_codec.pth from /app/models/IndexTTS-2: /app/models/IndexTTS-2 does not appear to have a file named semantic_codec.pth. Checkout 'huggingface.co//app/models/IndexTTS-2/tree/main' for available files.
    (StageEngineCoreProc_stage0_replica0 pid=6834) WARNING 11-02 10:54:00 [preprocess_utils.py:45] Could not resolve IndexTTS2 asset campplus.pth from /app/models/IndexTTS-2: /app/models/IndexTTS-2 does not appear to have a file named campplus.pth. Checkout 'huggingface.co//app/models/IndexTTS-2/tree/main' for available files.
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:03 [indextts2_talker.py:1060] External models preloaded: Wav2Vec2, RepCodec, CAMPPlus
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:04 [default_loader.py:430] Loading weights took 1531091.77 seconds
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:06 [gpu_model_runner.py:5347] Model loading took 3.95 GiB memory and 44.034698 seconds
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:14 [backends.py:1094] Using cache directory: /root/.cache/vllm/torch_compile_cache/7ca1ef1aea/rank_0_0/backbone for vLLM's torch.compile
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:14 [backends.py:1155] Dynamo bytecode transform time: 6.11 s
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:22 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 512) from the cache, took 2.467 s
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:22 [decorators.py:311] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/c011cf7bd91869f61a67668773de6b212f665f9c57a8e9a74463d52d844a7eae/rank_0_0/model
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:22 [monitor.py:53] torch.compile took 14.08 s in total
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:24 [monitor.py:81] Initial profiling/warmup run took 2.51 s
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:27 [base.py:163] Available KV cache memory: 10.28 GiB (profiling fallback)
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:27 [kv_cache_utils.py:2177] GPU KV cache size: 89,808 tokens
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:27 [kv_cache_utils.py:2178] Maximum concurrency for 2,560 tokens per request: 35.08x
    (StageEngineCoreProc_stage0_replica0 pid=6834) WARNING 11-02 10:54:27 [base.py:188] [LLM Worker 0] Sleep Mode DISABLED.
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:27 [ll_bf16.py:30] cuteDSL (CUTLASS Python) not available, ll_bf16_gemm disabled
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:27 [cutedsl_warmup.py:96] Skipping CuTeDSL warmup on non-CUDA platform.
    (StageEngineCoreProc_stage0_replica0 pid=6834) INFO 11-02 10:54:27 [gpu_model_runner.py:6798] Rank 0: Torch profiler disabled for CUDA graph capture

    (APIServer pid=6534) WARNING 11-02 11:01:54 [async_omni_engine.py:475] [AsyncOmniEngine] Orchestrator startup timed out after 600s. Multi-stage deployments that initialize stages sequentially on one device or load checkpoints from slow storage may need larger --init-timeout and --stage-init-timeout values.
    (APIServer pid=6534) INFO 11-02 11:01:54 [async_omni_engine.py:1827] [AsyncOmniEngine] Shutting down Orchestrator
    (APIServer pid=6534) ERROR 11-02 11:02:24 [async_omni_engine.py:1856] [AsyncOmniEngine] Orchestrator did not stop within 30.0 seconds; continuing cleanup
    (APIServer pid=6534) WARNING 11-02 11:02:24 [async_omni_engine.py:1876] [AsyncOmniEngine] Deferring StageRuntime shutdown until the Orchestrator exits
    (APIServer pid=6534) Traceback (most recent call last):
    (APIServer pid=6534) File "/opt/conda/bin/vllm", line 8, in <module>
    (APIServer pid=6534) sys.exit(main())
    (APIServer pid=6534) ^^^^^^
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 55, in main
    (APIServer pid=6534) omni_main()
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/site-packages/vllm_omni/entrypoints/cli/main.py", line 72, in main
    (APIServer pid=6534) args.dispatch_function(args)
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/site-packages/vllm_omni/entrypoints/cli/serve.py", line 107, in cmd
    (APIServer pid=6534) uvloop.run(omni_run_server(args))
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/site-packages/uvloop/init.py", line 96, in run
    (APIServer pid=6534) return asyncio.run(
    (APIServer pid=6534) ^^^^^^^^^^^^^^
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/asyncio/runners.py", line 195, in run
    (APIServer pid=6534) return runner.run(main)
    (APIServer pid=6534) ^^^^^^^^^^^^^^^^
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/asyncio/runners.py", line 118, in run
    (APIServer pid=6534) return self._loop.run_until_complete(task)
    (APIServer pid=6534) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (APIServer pid=6534) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/site-packages/uvloop/__init
    .py", line 48, in wrapper
    (APIServer pid=6534) return await main
    (APIServer pid=6534) ^^^^^^^^^^
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/site-packages/vllm_omni/entrypoints/openai/api_server.py", line 458, in omni_run_server
    (APIServer pid=6534) await omni_run_server_worker(listen_address, sock, args, uvicorn_kwargs)
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/site-packages/vllm_omni/entrypoints/openai/api_server.py", line 485, in omni_run_server_worker
    (APIServer pid=6534) async with build_async_omni(
    (APIServer pid=6534) ^^^^^^^^^^^^^^^^^
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/contextlib.py", line 210, in aenter
    (APIServer pid=6534) return await anext(self.gen)
    (APIServer pid=6534) ^^^^^^^^^^^^^^^^^^^^^
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/site-packages/vllm_omni/entrypoints/openai/api_server.py", line 629, in build_async_omni
    (APIServer pid=6534) async with build_async_omni_from_stage_config(
    (APIServer pid=6534) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/contextlib.py", line 210, in aenter
    (APIServer pid=6534) return await anext(self.gen)
    (APIServer pid=6534) ^^^^^^^^^^^^^^^^^^^^^
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/site-packages/vllm_omni/entrypoints/openai/api_server.py", line 695, in build_async_omni_from_stage_config
    (APIServer pid=6534) async_omni = AsyncOmni(model=model,
    kwargs)
    (APIServer pid=6534) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/site-packages/vllm_omni/entrypoints/async_omni.py", line 147, in init
    (APIServer pid=6534) OmniBase.init(self, model=model, **kwargs)
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/site-packages/vllm_omni/entrypoints/omni_base.py", line 178, in init
    (APIServer pid=6534) self.engine = AsyncOmniEngine(
    (APIServer pid=6534) ^^^^^^^^^^^^^^^^
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/async_omni_engine.py", line 289, in init
    (APIServer pid=6534) self._wait_for_orchestrator_init(startup_future, startup_timeout)
    (APIServer pid=6534) File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/async_omni_engine.py", line 483, in _wait_for_orchestrator_init
    (APIServer pid=6534) raise TimeoutError(f"Orchestrator did not become ready within {startup_timeout}s")
    (APIServer pid=6534) TimeoutError: Orchestrator did not become ready within 600s
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] [StageRuntime] Stage initialization failed; shutting down 0 initialized client(s)
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] Traceback (most recent call last):
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/stage_runtime.py", line 240, in initialize
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] initialized_clients = self._initialize_stage_replicas(stage_plans, self._stage_init_timeout)
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/stage_runtime.py", line 504, in _initialize_stage_replicas
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] raise primary_exc
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/stage_runtime.py", line 465, in _init_group
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] client = self._initialize_replica(
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] ^^^^^^^^^^^^^^^^^^^^^^^^^
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/stage_runtime.py", line 530, in _initialize_replica
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] return self._initialize_local_llm_replica(plan, stage_init_timeout)
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/stage_runtime.py", line 577, in _initialize_local_llm_replica
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] with launch_stage_replica(
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] ^^^^^^^^^^^^^^^^^^^^^
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] File "/opt/conda/lib/python3.12/contextlib.py", line 144, in exit
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] next(self.gen)
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/stage_engine_startup.py", line 1127, in launch_stage_replica
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] wait_for_engine_startup(
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1272, in wait_for_engine_startup
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] raise RuntimeError(
    (APIServer pid=6534) ERROR 11-02 11:02:26 [stage_runtime.py:253] RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
    /opt/conda/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
    warnings.warn('resource_tracker: There appear to be %d '

  • Members 1029 posts
    2026年11月2日 11:23

    尊敬的开发者您好,请在vllm server后加--enforce-eager尝试

  • Members 30 posts
    2026年11月2日 13:37

    启动了,但是推理报错了: INFO: Started server process [8911]
    (APIServer pid=8911) INFO: Waiting for application startup.
    (APIServer pid=8911) INFO: Application startup complete.
    (APIServer pid=8911) INFO 11-02 13:35:45 [tokenizer.py:24] IndexTTS2Tokenizer.init: vocab_file=/app/models/IndexTTS-2/bpe.model
    (APIServer pid=8911) INFO 11-02 13:35:45 [tokenizer.py:27] IndexTTS2Tokenizer initialized successfully
    (APIServer pid=8911) INFO 11-02 13:35:45 [serving_speech.py:3296] TTS speech request speech-813de70907d00c0b: text='你好,欢迎使用语音克隆测试。', model=indextts2
    (APIServer pid=8911) WARNING 11-02 13:35:45 [base.py:313] Using None for EOS token id because tokenizer is not initialized
    [13:35:58.553][MCR][E]mx_device.cpp :11000: Fail to allocate device memory
    [13:35:58.554][MCR][E]mx_device.cpp :11186: Fail to allocate device memory
    [13:35:58.554][MCR][E]mc_platform.cpp :589 : Fail to allocate device memory
    [13:35:58.554][MCR][E]mc_runtime_api.cpp :349 : 9211 : [7fb4c907f740] mcLaunchKernel: Returned mcErrorInvalidValue
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.26.0) with config: model='/app/models/IndexTTS-2', speculative_config=None, tokenizer='gpt2', skip_tokenizer_init=True, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2560, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/app/models/IndexTTS-2, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto'),
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [dump_input.py:79] Dumping scheduler output for model execution: OmniSchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=speech-813de70907d00c0b-8cf12697,prompt_token_ids_len=64,prefill_token_ids_len=None,mm_features=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=10.0, temperature=0.8, top_p=0.8, top_k=30, min_p=0.0, seed=None, stop=[], stop_token_ids=[8193], bad_words=[], thinking_token_budget=None, include_stop_str_in_output=False, ignore_eos=False, max_tokens=1500, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, structured_outputs=None, extra_args=None),block_ids=([1, 2, 3, 4],),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[],num_computed_tokens=[],num_output_tokens=[]), num_scheduled_tokens={speech-813de70907d00c0b-8cf12697: 64}, total_num_scheduled_tokens=64, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0], finished_req_ids=[], free_encoder_mm_hashes=[], scheduled_encoder_input_stats=null, preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=null, ec_connector_metadata=null, new_block_ids_to_zero=null, kv_cache_block_copies=null, num_spec_tokens_to_schedule=0, finished_requests_needing_kv_transfer={}, pending_input_registrations=[])
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, num_skipped_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.0007135212272565417, iteration_details=None, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=None, kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] StageEngineCoreProc encountered a fatal error.
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] Traceback (most recent call last):
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/stage_engine_core_proc.py", line 183, in run_stage_core
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] engine_core.run_busy_loop()
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1364, in run_busy_loop
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] self._process_engine_step()
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1403, in _process_engine_step
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] outputs, model_executed = self.step_fn()
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 647, in step_with_batch_queue
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] exec_future = self.model_executor.execute_model(
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 120, in execute_model
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] output.result()
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return self.__get_result()
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] raise self._exception
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 98, in collective_rpc
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] result = run_method(self.driver_worker, method, args, kwargs)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 510, in run_method
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return func(args, kwargs)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 351, in execute_model
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return self.worker.execute_model(scheduler_output)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return func(
    args, kwargs)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1147, in execute_model
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] output = self.model_runner.execute_model(
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return func(*args,
    kwargs)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/worker/gpu_ar_model_runner.py", line 1264, in execute_model
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ) = self._preprocess(scheduler_output, num_tokens_padded, intermediate_tensors)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/worker/gpu_model_runner.py", line 1774, in _preprocess
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] req_input_ids, req_embeds, update_dict = self.model.preprocess(
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/model_executor/models/indextts2/indextts2_talker.py", line 404, in preprocess
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return self._preprocess_prefill(input_ids, input_embeds, info_dict, span_len)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/model_executor/models/indextts2/indextts2_talker.py", line 500, in _preprocess_prefill
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] spk_cond_emb = wav2vec_extract(wav_16k, w2v_model, w2v_proc, device, self._w2v_stat)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/model_executor/models/indextts2/preprocess_utils.py", line 200, in wav2vec_extract
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] outputs = model(
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return self._call_impl(args, kwargs)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return forward_call(
    args, kwargs)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/utils/generic.py", line 1068, in wrapper
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] output = func(self, *args,
    kwargs)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/utils/output_capturing.py", line 262, in wrapper
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] outputs = func(self, args, kwargs)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/models/wav2vec2_bert/modeling_wav2vec2_bert.py", line 1027, in forward
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] encoder_outputs = self.encoder(
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return self._call_impl(
    args, kwargs)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return forward_call(*args,
    kwargs)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/models/wav2vec2_bert/modeling_wav2vec2_bert.py", line 527, in forward
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] attention_mask = create_bidirectional_mask(
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/masking_utils.py", line 1083, in create_bidirectional_mask
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] attention_mask = mask_interface(
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 171, in wrapped_func
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] return func(args, *kwargs)
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/masking_utils.py", line 500, in sdpa_mask
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] if allow_is_bidirectional_skip and _ignore_bidirectional_mask_sdpa(padding_mask, kv_length, local_size):
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] File "/opt/conda/lib/python3.12/site-packages/transformers/masking_utils.py", line 330, in _ignore_bidirectional_mask_sdpa
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] and (padding_mask is None or padding_mask.all())
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] ^^^^^^^^^^^^^^^^^^
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] torch.AcceleratorError: CUDA error: invalid argument
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] Search for cudaErrorInvalidValue' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information. (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] For debugging consider passing CUDA_LAUNCH_BLOCKING=1 (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192] Compile withTORCH_USE_CUDA_DSA` to enable device-side assertions.
    (StageEngineCoreProc_stage0_replica0 pid=9211) ERROR 11-02 13:35:58 [stage_engine_core_proc.py:192]
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] [StagePool] _poll_stage_raw failed for stage-0 replica-0
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] Traceback (most recent call last):
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/stage_pool.py", line 1136, in poll_llm_raw_output
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] return await asyncio.wait_for(
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] ^^^^^^^^^^^^^^^^^^^^^^^
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] File "/opt/conda/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] return await fut
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] ^^^^^^^^^
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] File "/opt/conda/lib/python3.12/site-packages/vllm_omni/engine/stage_pool.py", line 1083, in _poll_stage_raw
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] outputs = await client.get_output_async()
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] File "/opt/conda/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1061, in get_output_async
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] raise self._format_exception(outputs) from None
    (APIServer pid=8911) ERROR 11-02 13:35:58 [stage_pool.py:1145] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
    (APIServer pid=8911) ERROR 11-02 13:35:58 [orchestrator.py:963] [Orchestrator] Stage-0 replica-0 is dead: EngineCore encountered an issue. See stack trace (above) for the root cause.
    (APIServer pid=8911) ERROR 11-02 13:35:58 [async_omni.py:926] [AsyncOmni] Engine dead: EngineCore encountered an issue. See stack trace (above) for the root cause.
    (APIServer pid=8911) ERROR 11-02 13:35:58 [async_omni.py:814] [AsyncOmni] Orchestrator error for req=speech-813de70907d00c0b-8cf12697 stage-0: EngineCore encountered an issue. See stack trace (above) for the root cause.
    (APIServer pid=8911) INFO 11-02 13:35:58 [async_omni.py:1074] [AsyncOmni] Aborted request(s) speech-813de70907d00c0b-8cf12697
    (APIServer pid=8911) INFO 11-02 13:35:58 [async_omni.py:650] [AsyncOmni] Request speech-813de70907d00c0b-8cf12697 failed (input error): EngineCore encountered an issue. See stack trace (above) for the root cause.
    (APIServer pid=8911) ERROR 11-02 13:35:58 [serving_speech.py:3875] [SpeechE2E] request_id=speech-813de70907d00c0b stream=False status=engine_error total_ms=14558.34
    (APIServer pid=8911) ERROR 11-02 13:35:58 [api_server.py:337] EngineDeadError: orchestrator_alive=True, errored=True, request_id=speech-813de70907d00c0b, error_stage_id=0
    (APIServer pid=8911) INFO: Shutting down
    (APIServer pid=8911) INFO 11-02 13:35:58 [orchestrator.py:540] [Orchestrator] Engine dead during shutdown: EngineCore encountered an issue. See stack trace (above) for the root cause.
    (APIServer pid=8911) INFO 11-02 13:35:58 [orchestrator.py:2258] [Orchestrator] Shutting down all 2 client(s)
    (APIServer pid=8911) INFO 11-02 13:35:58 [core_client.py:655] [shutdown] MPClient: start timeout=default
    (APIServer pid=8911) INFO 11-02 13:35:58 [core_client.py:657] [shutdown] MPClient: stopping engine manager
    (APIServer pid=8911) INFO: 127.0.0.1:49990 - "POST /v1/audio/speech HTTP/1.1" 500 Internal Server Error
    (APIServer pid=8911) INFO: Waiting for application shutdown.
    (APIServer pid=8911) INFO: Application shutdown complete.
    (APIServer pid=8911) INFO: Finished server process [8911]
    (APIServer pid=8911) INFO 11-02 13:35:58 [omni_base.py:630] [AsyncOmni] Shutting down
    (APIServer pid=8911) INFO 11-02 13:35:58 [async_omni_engine.py:1827] [AsyncOmniEngine] Shutting down Orchestrator
    (APIServer pid=8911) INFO 11-02 13:36:01 [core_client.py:659] [shutdown] MPClient: engine manager stopped
    (APIServer pid=8911) INFO 11-02 13:36:01 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
    (APIServer pid=8911) INFO 11-02 13:36:01 [core_client.py:662] [shutdown] MPClient: complete
    (APIServer pid=8911) INFO 11-02 13:36:01 [stage_pool.py:1237] [StagePool] Stage 0 replica 0 shut down
    (APIServer pid=8911) INFO 11-02 13:36:01 [core_client.py:655] [shutdown] MPClient: start timeout=default
    (APIServer pid=8911) WARNING 11-02 13:36:06 [utils.py:626] [shutdown] Process manager: force killing remaining processes count=1
    (APIServer pid=8911) INFO 11-02 13:36:06 [stage_pool.py:1237] [StagePool] Stage 1 replica 0 shut down
    /opt/conda/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
    warnings.warn('resource_tracker: There appear to be %d '

  • Members 1029 posts
    2026年11月2日 14:13

    尊敬的开发者您好,请宿主机执行

    dmesg -T | grep -i err
    
  • Members 30 posts
    2026年11月2日 16:48

    可以了,光有-e MACA_VISIBLE_DEVICES=4 不行,必须加上 -e CUDA_VISIBLE_DEVICES=4,不然默认就是0卡,0卡显存不够,但是推理完以后音频大小1M,就是一听啥内容都没有,滋滋的声音,和咋们这边有关系吗?

  • Members 1029 posts
    2026年11月2日 17:05

    尊敬的开发者您好,请多次请求,更换不同提示词尝试

  • Members 30 posts
    2026年11月2日 17:12

    确实多次请求了,也都是不一样的提示词,生成的都不行。图片显示的是生成信息

    image.png

    PNG, 209.0 KB, uploaded by Raining on 2026年11月2日.

  • Members 1029 posts
    2026年11月2日 17:48

    尊敬的开发者您好,请将日志、提示词和生成的文件传到附件

  • Members 1029 posts
    2026年11月3日 15:51

    尊敬的开发者您好,此版本存在bug,等待vllm-metax omini 0.28.0近期发布后重新测试