INFO: 10.66.3.19:36086 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO 12-02 17:28:34 [logger.py:41] Received request chatcmpl-9119253648c346dcadaef0d7d55136b3: prompt: '<｜begin▁of▁sentence｜><｜User｜>脖子淋巴癌早期症状都有什么样的表现？<｜Assistant｜>', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=0, min_p=0.0, seed=None, stop=['<|im_end|>'], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=4084, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None. INFO 12-02 17:28:34 [async_llm.py:269] Added request chatcmpl-9119253648c346dcadaef0d7d55136b3. INFO: 10.66.3.19:34628 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO 12-02 17:28:39 [logger.py:41] Received request chatcmpl-f3e65ff6f6be45158cb400469a8acbed: prompt: '<｜begin▁of▁sentence｜><｜User｜>来月经的时候和男接吻会有事情吗？ 来月经的时候和男朋友接吻会有事情吗？<｜Assistant｜>', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=0, min_p=0.0, seed=None, stop=['<|im_end|>'], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=4070, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None. INFO 12-02 17:28:39 [async_llm.py:269] Added request chatcmpl-f3e65ff6f6be45158cb400469a8acbed. INFO 12-02 17:28:42 [loggers.py:122] Engine 000: Avg prompt throughput: 5.6 tokens/s, Avg generation throughput: 352.0 tokens/s, Running: 38 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% INFO 12-02 17:28:52 [loggers.py:122] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 355.8 tokens/s, Running: 38 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 0.0% INFO: 10.66.3.19:55696 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO 12-02 17:28:56 [logger.py:41] Received request chatcmpl-fdfaae2ac28c4a31bbd309473921b04f: prompt: '<｜begin▁of▁sentence｜><｜User｜>浦原喜助!这四个字的日文怎么写!<｜Assistant｜>', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=0, min_p=0.0, seed=None, stop=['<|im_end|>'], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=4082, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None. INFO 12-02 17:28:56 [async_llm.py:269] Added request chatcmpl-fdfaae2ac28c4a31bbd309473921b04f. INFO: 10.66.3.19:37994 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO 12-02 17:28:57 [logger.py:41] Received request chatcmpl-3ace21c65beb48699f6f7760cd3d9fec: prompt: '<｜begin▁of▁sentence｜><｜User｜>下肢静脉曲张的治疗方法用什么?静脉曲张的预防?静脉曲张袜正品<｜Assistant｜>', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=0, min_p=0.0, seed=None, stop=['<|im_end|>'], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=4073, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None. INFO 12-02 17:28:57 [async_llm.py:269] Added request chatcmpl-3ace21c65beb48699f6f7760cd3d9fec. INFO 12-02 17:29:02 [loggers.py:122] Engine 000: Avg prompt throughput: 3.7 tokens/s, Avg generation throughput: 324.0 tokens/s, Running: 38 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.5%, Prefix cache hit rate: 0.0% (raylet) A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffffd0d6e32d27a569ddc2a9989702000000 Worker ID: a7623ea826205f083a31330f063a634dace191dbd1fa51d35cd9cdf6 Node ID: d0d6ce7502e5b178b1eae5d0644011e366d0a080f388b9480e45bfc7 Worker IP address: 10.66.3.19 Worker port: 10010 Worker PID: 11432 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors. (RayWorkerWrapper pid=11467) [17:26:57.225][MXKW][E]fmm.c :3376: [_fmm_map_to_xpu]Map to aperture ccx non-cachable failed. r=-1, n_success=0 map_n_device=8 [repeated 14x across cluster] (RayWorkerWrapper pid=11467) [17:26:57.225][MXKW][E]fmm.c :3443: [fmm_map_to_xpu]map to xpu failed, [17:26:57.225][MXKW][E]memory.c :829 : [mxkwMapMemoryToXPU]fail to map memory to xpu type 1, addr 0x7eb274000000 1056964608 bytes. ret = 0 [repeated 14x across cluster] 2025-12-02 17:29:07,478 INFO compiled_dag_node.py:2173 -- Tearing down compiled DAG 2025-12-02 17:29:07,478 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 03e384a2f6ceaa006e5a221a02000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 5c3f981cd607dddb1927899102000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, d0d6e32d27a569ddc2a9989702000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 35653a2635ad74c48e9c37a402000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, dbcd02af4f2632750af61f7f02000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, cb0970478029ea9603273a9602000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 954c47b60fdc7f574f15f0f102000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, a0f710e2ad0f67203cc2f38202000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 7a0105fbb0b8878b665b374b02000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, f9059f1e567bfa323c04921e02000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, e2f5cdf693e9230c194d80fb02000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 3be16639d9ea4d14e882215102000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 9ead9db31702a28b32925c0b02000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 6ef6620157e6924f46fab73b02000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 14ea202482c17d02080f24df02000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 3e04af7902d7c0a4ee0b322d02000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 78e0d1b83eb20078683c95c402000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 9ecf8b21fc0c5f47a1d5ccba02000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, edbe88b5c0c1c568965b0a9002000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 0ba57f5bc2140f23795900f602000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 6f71c4e352ba7e7206d8dfdd02000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 77a17e4aeea3c48d7a2b174202000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 583809e1fdf273bd3ec3af0602000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 117e154c177f7be135b190b502000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, dcbcdcde90badd23f8c44f1002000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, a5cbca48832fea837ac76a2302000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 78ac27f0d83f8b371f6e374f02000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 54b22de2a605fc076320744e02000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 6715f707030b4d21c647df0802000000) 2025-12-02 17:29:07,479 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, cadbc862f8bbfef4869ea6f802000000) 2025-12-02 17:29:07,480 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, 9dda2837c628d011fc38534802000000) 2025-12-02 17:29:07,480 INFO compiled_dag_node.py:2178 -- Cancelling compiled worker on actor: Actor(RayWorkerWrapper, c3321b9fbccbb72e48c38e4702000000) ERROR 12-02 17:29:07 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.10.0) with config: model='/mnt/DeepSeek-R1-0528-bf16', speculative_config=None, tokenizer='/mnt/DeepSeek-R1-0528-bf16', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=4, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=DeepSeek-R1, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":256,"local_cache_dir":null}, ERROR 12-02 17:29:07 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-be95c26b59624a22be203056cd584847', 'chatcmpl-27da4659af08471da9e28d5a2c11524a', 'chatcmpl-9119253648c346dcadaef0d7d55136b3', 'chatcmpl-fdfaae2ac28c4a31bbd309473921b04f'], resumed_from_preemption=[false, false, false, false], new_token_ids=[[19135], [2163], [39993], [10280]], new_block_ids=[[[]], [[]], [[]], [[]]], num_computed_tokens=[597, 488, 270, 63]), num_scheduled_tokens={chatcmpl-fdfaae2ac28c4a31bbd309473921b04f: 1, chatcmpl-9119253648c346dcadaef0d7d55136b3: 1, chatcmpl-be95c26b59624a22be203056cd584847: 1, chatcmpl-27da4659af08471da9e28d5a2c11524a: 1}, total_num_scheduled_tokens=4, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null) ERROR 12-02 17:29:07 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=38, num_waiting_reqs=0, kv_cache_usage=0.1751012145748988, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0) ERROR 12-02 17:29:07 [core.py:634] EngineCore encountered a fatal error. ERROR 12-02 17:29:07 [core.py:634] Traceback (most recent call last): ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 625, in run_engine_core ERROR 12-02 17:29:07 [core.py:634] engine_core.run_busy_loop() ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 652, in run_busy_loop ERROR 12-02 17:29:07 [core.py:634] self._process_engine_step() ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 677, in _process_engine_step ERROR 12-02 17:29:07 [core.py:634] outputs, model_executed = self.step_fn() ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 318, in step_with_batch_queue ERROR 12-02 17:29:07 [core.py:634] model_output = self.execute_model_with_error_logging( ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 253, in execute_model_with_error_logging ERROR 12-02 17:29:07 [core.py:634] raise err ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 244, in execute_model_with_error_logging ERROR 12-02 17:29:07 [core.py:634] return model_fn(scheduler_output) ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 319, in ERROR 12-02 17:29:07 [core.py:634] lambda _: future.result(), scheduler_output) ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/executor/ray_distributed_executor.py", line 37, in result ERROR 12-02 17:29:07 [core.py:634] return self.refs[0].get() ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/ray/experimental/compiled_dag_ref.py", line 145, in get ERROR 12-02 17:29:07 [core.py:634] raise execution_error from None ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/ray/experimental/compiled_dag_ref.py", line 136, in get ERROR 12-02 17:29:07 [core.py:634] ray.get(actor_execution_loop_refs, timeout=10) ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper ERROR 12-02 17:29:07 [core.py:634] return fn(*args, **kwargs) ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper ERROR 12-02 17:29:07 [core.py:634] return func(*args, **kwargs) ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/ray/_private/worker.py", line 2822, in get ERROR 12-02 17:29:07 [core.py:634] values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout) ERROR 12-02 17:29:07 [core.py:634] File "/opt/conda/lib/python3.10/site-packages/ray/_private/worker.py", line 932, in get_objects ERROR 12-02 17:29:07 [core.py:634] raise value ERROR 12-02 17:29:07 [core.py:634] ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task. ERROR 12-02 17:29:07 [core.py:634] class_name: RayWorkerWrapper ERROR 12-02 17:29:07 [core.py:634] actor_id: d0d6e32d27a569ddc2a9989702000000 ERROR 12-02 17:29:07 [core.py:634] pid: 11432 ERROR 12-02 17:29:07 [core.py:634] namespace: 9e2ff8af-d0d3-4bb7-8737-0b5e5f7aad55 ERROR 12-02 17:29:07 [core.py:634] ip: 10.66.3.19 ERROR 12-02 17:29:07 [core.py:634] The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors. INFO 12-02 17:29:07 [ray_distributed_executor.py:120] Shutting down Ray distributed executor. If you see error log from logging.cc regarding SIGTERM received, please ignore because this is the expected termination process in Ray. ERROR 12-02 17:29:07 [async_llm.py:416] AsyncLLM output_handler failed. ERROR 12-02 17:29:07 [async_llm.py:416] Traceback (most recent call last): ERROR 12-02 17:29:07 [async_llm.py:416] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 375, in output_handler ERROR 12-02 17:29:07 [async_llm.py:416] outputs = await engine_core.get_output_async() ERROR 12-02 17:29:07 [async_llm.py:416] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 751, in get_output_async ERROR 12-02 17:29:07 [async_llm.py:416] raise self._format_exception(outputs) from None ERROR 12-02 17:29:07 [async_llm.py:416] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-c4c12826b10448799492bc384a140c19 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-a327a46427904197b7c4bae1297e60e4 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-0be5658495b046ce99d1059750034527 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-4f4a7f0a4a334d3797f8e8b5f707a1b8 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-82b0e815ce5f4a8083a7bb4e1b95722f failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-9ac321ae1479470b81539349c3f412a0 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-55d3d0aac5d9467da3163b7ba04e7029 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-e5e16f3d92a74c9cb7a97bfd12390579 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-bc960b15d1ed40309250a04b251a45cc failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-7362e14598e1447aa39c6599c539b874 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-095d7b8976a94b72b5e6a92fe78e1586 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-e8a8ef42c8ba4e41a421a13a6d8374e4 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-5a4c71838692408cba9c8577da2e9811 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-08578e4f17854633aa7109b7eefdde94 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-7c74f34e354f463cabde161494864acd failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-bba4adc21ff44c0cb691710d02d2a1a7 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-87c20e3b6a7f49cfbe3f98ae9fc262ae failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-5dceafea55564f74bd0ae9418a5c8419 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-661f1ae2cf234428956c41aa66c75950 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-5c0e21528f6546a8b5f063060c6f3569 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-11d1af9ecc5d4ee9ae2d24f7a09f5f84 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-f48b7750e79d47b3b3eddcca372206ba failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-84e86db43d4f4a9197d5bd22db3e08a4 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-2d8eb22ebd1a42828ac54a976aa37b70 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-be95c26b59624a22be203056cd584847 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-250fa1a42019441dbbad6a29c8554359 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-e06c4ca1e3974d41afbdcfdbf4184139 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-27da4659af08471da9e28d5a2c11524a failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-e7d47c099bcb41ea8bc7f4c738a7b1af failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-6766e3375e2848f4b7008dfc0340ec64 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-342eb01fc72b45cc83b06d157083a7a0 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-c6b88eca112d497bbf882ed22866cac9 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-2e97d3eeec8647e183c3cda7c782b6f3 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-e21c1d680ba84361aa6e36ae02f5c9f4 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-9119253648c346dcadaef0d7d55136b3 failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-f3e65ff6f6be45158cb400469a8acbed failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-fdfaae2ac28c4a31bbd309473921b04f failed (engine dead). INFO 12-02 17:29:07 [async_llm.py:342] Request chatcmpl-3ace21c65beb48699f6f7760cd3d9fec failed (engine dead). INFO: 10.66.3.19:37322 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36970 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44668 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:47604 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:57696 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50138 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:57832 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37070 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:34368 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37202 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36718 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36528 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36922 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:41844 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50152 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37086 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:51072 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37198 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36908 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:40514 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37092 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:52982 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50168 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:38276 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:48766 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37212 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37052 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50024 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37346 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37422 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37164 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50010 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44762 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44634 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36086 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:34628 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:55696 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37994 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error (RayWorkerWrapper pid=11430) Destructing NCCL group on actor: Actor(RayWorkerWrapper, 03e384a2f6ceaa006e5a221a02000000) INFO: 10.66.3.19:50024 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37052 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37092 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36908 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37164 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37994 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44668 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36922 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44762 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:52982 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:38276 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:34628 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37070 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36086 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50152 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50138 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:48766 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37346 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:57696 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37198 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:57832 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37202 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44634 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37322 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36970 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37212 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37422 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:47604 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37086 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:41844 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36528 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50010 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50168 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:55696 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:34368 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:40514 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:51072 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36718 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50024 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37052 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:52982 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37092 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:38276 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36908 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:34628 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37164 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37994 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44668 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36922 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44762 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36086 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37070 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50152 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50138 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:48766 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37346 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:57696 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:57832 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44634 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37198 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37202 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37322 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36970 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37212 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37422 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:47604 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37086 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:41844 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36528 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50010 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50168 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:55696 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:34368 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:40514 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:51072 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36718 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50024 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:52982 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37052 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37092 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:38276 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36908 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:34628 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37164 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37994 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44668 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36922 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44762 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36086 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37070 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50152 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50138 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:48766 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37346 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:57696 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:57832 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44634 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: Shutting down INFO: 10.66.3.19:41844 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37322 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36970 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37212 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37422 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36528 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:47604 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37086 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37198 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37202 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50024 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50010 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:55696 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:50168 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:34368 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:40514 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:51072 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36718 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:52982 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37092 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:38276 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36908 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37052 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:34628 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37164 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:37994 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44668 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:36922 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 10.66.3.19:44762 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [100272] *** SIGTERM received at time=1764667747 on cpu 154 *** PC: @ 0x7fae02bdf117 (unknown) (unknown) @ 0x7fae02b90520 (unknown) (unknown) [2025-12-02 17:29:07,674 E 100546 100546] logging.cc:496: *** SIGTERM received at time=1764667747 on cpu 154 *** [2025-12-02 17:29:07,674 E 100546 100546] logging.cc:496: PC: @ 0x7fae02bdf117 (unknown) (unknown) [2025-12-02 17:29:07,674 E 100546 100546] logging.cc:496: @ 0x7fae02b90520 (unknown) (unknown) (raylet, ip=10.66.3.20) [2025-12-02 17:29:09,626 C 64832 64832] (raylet) experimental_mutable_object_provider.cc:153: Check failed: object_manager_->WriteAcquire(info.local_object_id, total_data_size, nullptr, total_metadata_size, info.num_readers, object_backing_store) Status not OK: ChannelError: Channel closed. (raylet, ip=10.66.3.20) *** StackTrace Information *** (raylet, ip=10.66.3.20) /opt/conda/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0xde786a) [0x55e39168e86a] ray::operator<<() (raylet, ip=10.66.3.20) /opt/conda/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0xde9c29) [0x55e391690c29] ray::RayLog::~RayLog() (raylet, ip=10.66.3.20) /opt/conda/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x4cb25d) [0x55e390d7225d] ray::core::experimental::MutableObjectProvider::HandlePushMutableObject() (raylet, ip=10.66.3.20) /opt/conda/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x29151e) [0x55e390b3851e] ray::raylet::NodeManager::HandlePushMutableObject() (raylet, ip=10.66.3.20) /opt/conda/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x2fc397) [0x55e390ba3397] ray::rpc::ServerCallImpl<>::HandleRequestImpl() (raylet, ip=10.66.3.20) /opt/conda/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x760648) [0x55e391007648] EventTracker::RecordExecution() (raylet, ip=10.66.3.20) /opt/conda/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x75b63e) [0x55e39100263e] std::_Function_handler<>::_M_invoke() (raylet, ip=10.66.3.20) /opt/conda/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x75bab6) [0x55e391002ab6] boost::asio::detail::completion_handler<>::do_complete() (raylet, ip=10.66.3.20) /opt/conda/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0xdc37db) [0x55e39166a7db] boost::asio::detail::scheduler::do_run_one() (raylet, ip=10.66.3.20) /opt/conda/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0xdc5d69) [0x55e39166cd69] boost::asio::detail::scheduler::run() (raylet, ip=10.66.3.20) /opt/conda/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0xdc6282) [0x55e39166d282] boost::asio::io_context::run() (raylet, ip=10.66.3.20) /opt/conda/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x1f461f) [0x55e390a9b61f] main (raylet, ip=10.66.3.20) /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fb211067d90] (raylet, ip=10.66.3.20) /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fb211067e40] __libc_start_main (raylet, ip=10.66.3.20) /opt/conda/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet(+0x24dd27) [0x55e390af4d27] (raylet, ip=10.66.3.20) (RayWorkerWrapper pid=11430) [2025-12-02 17:29:09,650 E 11430 113860] raylet_client.cc:364: Error pushing mutable object: RpcError: RPC Error message: Socket closed; RPC Error details: rpc_code: 14 (RayWorkerWrapper pid=11430) [2025-12-02 17:29:09,650 E 11430 113862] raylet_client.cc:364: Error pushing mutable object: RpcError: RPC Error message: Socket closed; RPC Error details: rpc_code: 14