Multi-thread loading shards: 100% Completed | 152/152 [00:20<00:00,  7.35it/s]
[2026-07-07 10:13:57 DP0 TP0 EP0] Load weight end. elapsed=42.33 s, type=GlmMoeDsaForCausalLM, quant=compressed-tensors, avail mem=17.08 GB, mem usage=42.95 GB.
[2026-07-07 10:13:59 DP2 TP2 EP2] Load weight end. elapsed=43.63 s, type=GlmMoeDsaForCausalLM, quant=compressed-tensors, avail mem=17.08 GB, mem usage=42.95 GB.
[2026-07-07 10:14:00 DP3 TP3 EP3] Load weight end. elapsed=44.76 s, type=GlmMoeDsaForCausalLM, quant=compressed-tensors, avail mem=17.18 GB, mem usage=42.95 GB.
[2026-07-07 10:14:01 DP6 TP6 EP6] Load weight end. elapsed=46.35 s, type=GlmMoeDsaForCausalLM, quant=compressed-tensors, avail mem=17.37 GB, mem usage=42.95 GB.
[2026-07-07 10:14:02 DP5 TP5 EP5] Load weight end. elapsed=46.45 s, type=GlmMoeDsaForCausalLM, quant=compressed-tensors, avail mem=17.08 GB, mem usage=42.95 GB.
[2026-07-07 10:14:02 DP7 TP7 EP7] Load weight end. elapsed=46.44 s, type=GlmMoeDsaForCausalLM, quant=compressed-tensors, avail mem=17.27 GB, mem usage=42.95 GB.
[2026-07-07 10:14:02 DP1 TP1 EP1] Load weight end. elapsed=46.87 s, type=GlmMoeDsaForCausalLM, quant=compressed-tensors, avail mem=17.08 GB, mem usage=42.95 GB.
[2026-07-07 10:14:02 DP4 TP4 EP4] Load weight end. elapsed=46.61 s, type=GlmMoeDsaForCausalLM, quant=compressed-tensors, avail mem=17.08 GB, mem usage=42.95 GB.
[2026-07-07 10:14:03 DP0 TP0 EP0] Using KV cache dtype: torch.bfloat16
[2026-07-07 10:14:03 DP0 TP0 EP0] KV Cache is allocated. #tokens: 58496, KV size: 5.99 GB
[2026-07-07 10:14:03 DP0 TP0 EP0] Memory pool end. avail mem=10.82 GB
[2026-07-07 10:14:03 DP2 TP2 EP2] KV Cache is allocated. #tokens: 58496, KV size: 5.99 GB
[2026-07-07 10:14:03 DP5 TP5 EP5] KV Cache is allocated. #tokens: 58496, KV size: 5.99 GB
[2026-07-07 10:14:03 DP3 TP3 EP3] KV Cache is allocated. #tokens: 58496, KV size: 5.99 GB
[2026-07-07 10:14:03 DP4 TP4 EP4] KV Cache is allocated. #tokens: 58496, KV size: 5.99 GB
[2026-07-07 10:14:03 DP7 TP7 EP7] KV Cache is allocated. #tokens: 58496, KV size: 5.99 GB
[2026-07-07 10:14:03 DP6 TP6 EP6] KV Cache is allocated. #tokens: 58496, KV size: 5.99 GB
[2026-07-07 10:14:03 DP2 TP2 EP2] Memory pool end. avail mem=10.82 GB
[2026-07-07 10:14:03 DP5 TP5 EP5] Memory pool end. avail mem=10.82 GB
[2026-07-07 10:14:03 DP4 TP4 EP4] Memory pool end. avail mem=10.82 GB
[2026-07-07 10:14:03 DP3 TP3 EP3] Memory pool end. avail mem=10.92 GB
[2026-07-07 10:14:03 DP7 TP7 EP7] Memory pool end. avail mem=11.02 GB
[2026-07-07 10:14:03 DP6 TP6 EP6] Memory pool end. avail mem=11.12 GB
[2026-07-07 10:14:03 DP1 TP1 EP1] KV Cache is allocated. #tokens: 58496, KV size: 5.99 GB
[2026-07-07 10:14:03 DP1 TP1 EP1] Memory pool end. avail mem=10.82 GB
[2026-07-07 10:14:03 DP0 TP0 EP0] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:03 DP2 TP2 EP2] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:04 DP6 TP6 EP6] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:04 DP1 TP1 EP1] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:04 DP3 TP3 EP3] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:04 DP5 TP5 EP5] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:04 DP7 TP7 EP7] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:04 DP4 TP4 EP4] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:07 DP7 TP7 EP7] Init torch distributed begin.
[2026-07-07 10:14:07 DP6 TP6 EP6] Init torch distributed begin.
[2026-07-07 10:14:07 DP1 TP1 EP1] Init torch distributed begin.
[2026-07-07 10:14:07 DP4 TP4 EP4] Init torch distributed begin.
[2026-07-07 10:14:07 DP5 TP5 EP5] Init torch distributed begin.
[2026-07-07 10:14:07 DP0 TP0 EP0] Chunked prefix cache is turned on.
[2026-07-07 10:14:07 DP0 TP0 EP0] Init torch distributed begin.
[2026-07-07 10:14:07 DP2 TP2 EP2] Init torch distributed begin.
[2026-07-07 10:14:07 DP3 TP3 EP3] Init torch distributed begin.
[2026-07-07 10:14:07 DP0 TP0 EP0] Init torch distributed ends. elapsed=0.19 s, mem usage=0.00 GB
[2026-07-07 10:14:07 DP0 TP0 EP0] Load weight begin. avail mem=10.22 GB
[2026-07-07 10:14:07 DP7 TP7 EP7] Init torch distributed ends. elapsed=0.31 s, mem usage=0.00 GB
[2026-07-07 10:14:07 DP5 TP5 EP5] Init torch distributed ends. elapsed=0.20 s, mem usage=0.00 GB
[2026-07-07 10:14:07 DP6 TP6 EP6] Init torch distributed ends. elapsed=0.30 s, mem usage=0.00 GB
[2026-07-07 10:14:07 DP3 TP3 EP3] Init torch distributed ends. elapsed=0.10 s, mem usage=0.00 GB
[2026-07-07 10:14:07 DP4 TP4 EP4] Init torch distributed ends. elapsed=0.24 s, mem usage=0.00 GB
[2026-07-07 10:14:07 DP2 TP2 EP2] Init torch distributed ends. elapsed=0.16 s, mem usage=0.00 GB
[2026-07-07 10:14:07 DP1 TP1 EP1] Init torch distributed ends. elapsed=0.26 s, mem usage=0.00 GB
[2026-07-07 10:14:07 DP7 TP7 EP7] Load weight begin. avail mem=10.41 GB
[2026-07-07 10:14:07 DP5 TP5 EP5] Load weight begin. avail mem=10.22 GB
[2026-07-07 10:14:07 DP4 TP4 EP4] Load weight begin. avail mem=10.22 GB
[2026-07-07 10:14:07 DP2 TP2 EP2] Load weight begin. avail mem=10.22 GB
[2026-07-07 10:14:07 DP1 TP1 EP1] Load weight begin. avail mem=10.22 GB
[2026-07-07 10:14:07 DP6 TP6 EP6] Load weight begin. avail mem=10.51 GB
[2026-07-07 10:14:07 DP3 TP3 EP3] Load weight begin. avail mem=10.32 GB
Multi-thread loading shards: 100% Completed | 152/152 [00:03<00:00, 39.71it/s]
[2026-07-07 10:14:11 DP2 TP2 EP2] Load weight end. elapsed=4.00 s, type=DeepseekV3ForCausalLMNextN, quant=compressed-tensors, avail mem=6.03 GB, mem usage=4.19 GB.
[2026-07-07 10:14:11 DP0 TP0 EP0] Load weight end. elapsed=4.02 s, type=DeepseekV3ForCausalLMNextN, quant=compressed-tensors, avail mem=6.03 GB, mem usage=4.19 GB.
[2026-07-07 10:14:11 DP1 TP1 EP1] Load weight end. elapsed=4.14 s, type=DeepseekV3ForCausalLMNextN, quant=compressed-tensors, avail mem=6.03 GB, mem usage=4.19 GB.
[2026-07-07 10:14:11 DP6 TP6 EP6] Load weight end. elapsed=4.14 s, type=DeepseekV3ForCausalLMNextN, quant=compressed-tensors, avail mem=6.32 GB, mem usage=4.19 GB.
[2026-07-07 10:14:11 DP7 TP7 EP7] Load weight end. elapsed=4.17 s, type=DeepseekV3ForCausalLMNextN, quant=compressed-tensors, avail mem=6.22 GB, mem usage=4.19 GB.
[2026-07-07 10:14:11 DP4 TP4 EP4] Load weight end. elapsed=4.19 s, type=DeepseekV3ForCausalLMNextN, quant=compressed-tensors, avail mem=6.03 GB, mem usage=4.19 GB.
[2026-07-07 10:14:11 DP3 TP3 EP3] Load weight end. elapsed=4.22 s, type=DeepseekV3ForCausalLMNextN, quant=compressed-tensors, avail mem=6.13 GB, mem usage=4.19 GB.
[2026-07-07 10:14:11 DP5 TP5 EP5] Load weight end. elapsed=4.29 s, type=DeepseekV3ForCausalLMNextN, quant=compressed-tensors, avail mem=6.03 GB, mem usage=4.19 GB.
[2026-07-07 10:14:11 DP0 TP0 EP0] Using KV cache dtype: torch.bfloat16
[2026-07-07 10:14:11 DP4 TP4 EP4] KV Cache is allocated. #tokens: 58496, KV size: 0.08 GB
[2026-07-07 10:14:11 DP5 TP5 EP5] KV Cache is allocated. #tokens: 58496, KV size: 0.08 GB
[2026-07-07 10:14:11 DP2 TP2 EP2] KV Cache is allocated. #tokens: 58496, KV size: 0.08 GB
[2026-07-07 10:14:11 DP1 TP1 EP1] KV Cache is allocated. #tokens: 58496, KV size: 0.08 GB
[2026-07-07 10:14:11 DP4 TP4 EP4] Memory pool end. avail mem=5.95 GB
[2026-07-07 10:14:11 DP5 TP5 EP5] Memory pool end. avail mem=5.95 GB
[2026-07-07 10:14:11 DP3 TP3 EP3] KV Cache is allocated. #tokens: 58496, KV size: 0.08 GB
[2026-07-07 10:14:11 DP2 TP2 EP2] Memory pool end. avail mem=5.95 GB
[2026-07-07 10:14:11 DP1 TP1 EP1] Memory pool end. avail mem=5.95 GB
[2026-07-07 10:14:11 DP6 TP6 EP6] KV Cache is allocated. #tokens: 58496, KV size: 0.08 GB
[2026-07-07 10:14:11 DP3 TP3 EP3] Memory pool end. avail mem=6.05 GB
[2026-07-07 10:14:11 DP7 TP7 EP7] KV Cache is allocated. #tokens: 58496, KV size: 0.08 GB
[2026-07-07 10:14:11 DP0 TP0 EP0] KV Cache is allocated. #tokens: 58496, KV size: 0.08 GB
[2026-07-07 10:14:11 DP6 TP6 EP6] Memory pool end. avail mem=6.24 GB
[2026-07-07 10:14:11 DP0 TP0 EP0] Memory pool end. avail mem=5.95 GB
[2026-07-07 10:14:11 DP7 TP7 EP7] Memory pool end. avail mem=6.14 GB
[2026-07-07 10:14:11 DP2 TP2 EP2] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:11 DP7 TP7 EP7] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:11 DP0 TP0 EP0] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:11 DP5 TP5 EP5] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:11 DP4 TP4 EP4] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:11 DP1 TP1 EP1] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:11 DP3 TP3 EP3] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:11 DP6 TP6 EP6] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set
[2026-07-07 10:14:15 DP0 TP0 EP0] max_total_num_tokens=58496, chunked_prefill_size=14336, max_prefill_tokens=16384, max_running_requests=32, context_len=65536, available_gpu_mem=9.42 GB
[2026-07-07 10:14:16] INFO:     Started server process [8931]
[2026-07-07 10:14:16] INFO:     Waiting for application startup.
[2026-07-07 10:14:16] Using default chat sampling params from model generation config: {'temperature': 1.0}
[2026-07-07 10:14:16] INFO:     Application startup complete.
[2026-07-07 10:14:16] INFO:     Uvicorn running on http://192.168.2.17:9293 (Press CTRL+C to quit)
[2026-07-07 10:14:17] INFO:     192.168.2.17:40160 - "GET /model_info HTTP/1.1" 200 OK
[2026-07-07 10:14:17] Start of pd disaggregation warmup ...
[rank5]:[W707 10:14:23.344075308 unwind.cpp:219] Warning: Unsupported unwinding pattern: Address not in range (function unwinderFor)
[rank6]:[W707 10:14:23.344082779 unwind.cpp:219] Warning: Unsupported unwinding pattern: Address not in range (function unwinderFor)
[rank4]:[W707 10:14:23.344083264 unwind.cpp:219] Warning: Unsupported unwinding pattern: Address not in range (function unwinderFor)
[rank3]:[W707 10:14:23.350862975 unwind.cpp:219] Warning: Unsupported unwinding pattern: Address not in range (function unwinderFor)
[rank1]:[W707 10:14:23.351891634 unwind.cpp:219] Warning: Unsupported unwinding pattern: Address not in range (function unwinderFor)
[rank0]:[W707 10:14:23.352304565 unwind.cpp:219] Warning: Unsupported unwinding pattern: Address not in range (function unwinderFor)
[rank7]:[W707 10:14:23.354911355 unwind.cpp:219] Warning: Unsupported unwinding pattern: Address not in range (function unwinderFor)
[rank2]:[W707 10:14:23.368236572 unwind.cpp:219] Warning: Unsupported unwinding pattern: Address not in range (function unwinderFor)
[rank4]:W0707 10:14:24.728000 9789 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank4]:W0707 10:14:24.728000 9789 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank5]:W0707 10:14:24.728000 9790 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank5]:W0707 10:14:24.728000 9790 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank4]:W0707 10:14:24.729000 9789 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank4]:W0707 10:14:24.729000 9789 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank5]:W0707 10:14:24.729000 9790 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank5]:W0707 10:14:24.729000 9790 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank6]:W0707 10:14:24.732000 9791 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank6]:W0707 10:14:24.732000 9791 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank6]:W0707 10:14:24.733000 9791 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank6]:W0707 10:14:24.733000 9791 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank7]:W0707 10:14:24.734000 9792 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank7]:W0707 10:14:24.734000 9792 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank7]:W0707 10:14:24.735000 9792 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank7]:W0707 10:14:24.735000 9792 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank0]:W0707 10:14:24.743000 9785 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank0]:W0707 10:14:24.743000 9785 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank4]:W0707 10:14:24.743000 9789 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank4]:W0707 10:14:24.743000 9789 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank5]:W0707 10:14:24.743000 9790 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank5]:W0707 10:14:24.743000 9790 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank3]:W0707 10:14:24.743000 9788 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank3]:W0707 10:14:24.743000 9788 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank1]:W0707 10:14:24.743000 9786 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank1]:W0707 10:14:24.743000 9786 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank0]:W0707 10:14:24.744000 9785 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank0]:W0707 10:14:24.744000 9785 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank4]:W0707 10:14:24.744000 9789 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank4]:W0707 10:14:24.744000 9789 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank5]:W0707 10:14:24.744000 9790 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank5]:W0707 10:14:24.744000 9790 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank2]:W0707 10:14:24.744000 9787 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank2]:W0707 10:14:24.744000 9787 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank3]:W0707 10:14:24.744000 9788 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank3]:W0707 10:14:24.744000 9788 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank1]:W0707 10:14:24.744000 9786 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank1]:W0707 10:14:24.744000 9786 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank2]:W0707 10:14:24.745000 9787 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank2]:W0707 10:14:24.745000 9787 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank6]:W0707 10:14:24.747000 9791 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank6]:W0707 10:14:24.747000 9791 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank6]:W0707 10:14:24.748000 9791 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank6]:W0707 10:14:24.748000 9791 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank7]:W0707 10:14:24.748000 9792 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank7]:W0707 10:14:24.748000 9792 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank7]:W0707 10:14:24.749000 9792 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank7]:W0707 10:14:24.749000 9792 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank1]:W0707 10:14:24.760000 9786 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank1]:W0707 10:14:24.760000 9786 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank3]:W0707 10:14:24.760000 9788 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank3]:W0707 10:14:24.760000 9788 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank1]:W0707 10:14:24.761000 9786 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank1]:W0707 10:14:24.761000 9786 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank3]:W0707 10:14:24.761000 9788 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank3]:W0707 10:14:24.761000 9788 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank0]:W0707 10:14:24.778000 9785 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank0]:W0707 10:14:24.778000 9785 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank0]:W0707 10:14:24.779000 9785 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank0]:W0707 10:14:24.779000 9785 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank2]:W0707 10:14:24.784000 9787 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank2]:W0707 10:14:24.784000 9787 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[rank2]:W0707 10:14:24.785000 9787 site-packages/torch/utils/cpp_extension.py:2527] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
[rank2]:W0707 10:14:24.785000 9787 site-packages/torch/utils/cpp_extension.py:2527] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[2026-07-07 10:14:24 DP7 TP7 EP7] Scheduler hit an exception: Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3621, in run_scheduler_process
    scheduler.run_event_loop()
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1304, in run_event_loop
    dispatch_event_loop(self)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3518, in dispatch_event_loop
    scheduler.event_loop_normal_disagg_decode()
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/disaggregation/decode.py", line 1155, in event_loop_normal_disagg_decode
    result = self.run_batch(batch)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 2729, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 335, in forward_batch_generation
    self.verify(batch, spec_info)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 746, in verify
    batch_result = self.target_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 469, in forward_batch_generation
    out = self.model_runner.forward(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2785, in forward
    output = self._forward_raw(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2898, in _forward_raw
    ret, can_run_graph = self.forward_extend(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2698, in forward_extend
    self.model.forward(
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2292, in forward
    hidden_states = self.model(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2101, in forward
    hidden_states, residual, topk_indices = layer(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 1790, in forward
    hidden_states = self.mlp(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 267, in forward
    x, _ = self.down_proj(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/linear.py", line 1509, in forward
    output_parallel = self.quant_method.apply(self, input_parallel, bias=bias_)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 971, in apply
    return scheme.apply_weights(layer, x, bias=bias)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py", line 176, in apply_weights
    x_scale = x[2]
IndexError: tuple index out of range

[2026-07-07 10:14:24 DP1 TP1 EP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3621, in run_scheduler_process
    scheduler.run_event_loop()
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1304, in run_event_loop
    dispatch_event_loop(self)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3518, in dispatch_event_loop
    scheduler.event_loop_normal_disagg_decode()
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/disaggregation/decode.py", line 1155, in event_loop_normal_disagg_decode
    result = self.run_batch(batch)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 2729, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 335, in forward_batch_generation
    self.verify(batch, spec_info)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 746, in verify
    batch_result = self.target_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 469, in forward_batch_generation
    out = self.model_runner.forward(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2785, in forward
    output = self._forward_raw(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2898, in _forward_raw
    ret, can_run_graph = self.forward_extend(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2698, in forward_extend
    self.model.forward(
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2292, in forward
    hidden_states = self.model(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2101, in forward
    hidden_states, residual, topk_indices = layer(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 1790, in forward
    hidden_states = self.mlp(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 267, in forward
    x, _ = self.down_proj(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/linear.py", line 1509, in forward
    output_parallel = self.quant_method.apply(self, input_parallel, bias=bias_)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 971, in apply
    return scheme.apply_weights(layer, x, bias=bias)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py", line 176, in apply_weights
    x_scale = x[2]
IndexError: tuple index out of range

[2026-07-07 10:14:24 DP0 TP0 EP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3621, in run_scheduler_process
    scheduler.run_event_loop()
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1304, in run_event_loop
    dispatch_event_loop(self)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3518, in dispatch_event_loop
    scheduler.event_loop_normal_disagg_decode()
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/disaggregation/decode.py", line 1155, in event_loop_normal_disagg_decode
    result = self.run_batch(batch)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 2729, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 335, in forward_batch_generation
    self.verify(batch, spec_info)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 746, in verify
    batch_result = self.target_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 469, in forward_batch_generation
    out = self.model_runner.forward(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2785, in forward
    output = self._forward_raw(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2898, in _forward_raw
    ret, can_run_graph = self.forward_extend(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2698, in forward_extend
    self.model.forward(
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2292, in forward
    hidden_states = self.model(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2101, in forward
    hidden_states, residual, topk_indices = layer(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 1790, in forward
    hidden_states = self.mlp(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 267, in forward
    x, _ = self.down_proj(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/linear.py", line 1509, in forward
    output_parallel = self.quant_method.apply(self, input_parallel, bias=bias_)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 971, in apply
    return scheme.apply_weights(layer, x, bias=bias)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py", line 176, in apply_weights
    x_scale = x[2]
IndexError: tuple index out of range

[2026-07-07 10:14:24 DP2 TP2 EP2] Scheduler hit an exception: Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3621, in run_scheduler_process
    scheduler.run_event_loop()
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1304, in run_event_loop
    dispatch_event_loop(self)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3518, in dispatch_event_loop
    scheduler.event_loop_normal_disagg_decode()
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/disaggregation/decode.py", line 1155, in event_loop_normal_disagg_decode
    result = self.run_batch(batch)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 2729, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 335, in forward_batch_generation
    self.verify(batch, spec_info)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 746, in verify
    batch_result = self.target_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 469, in forward_batch_generation
    out = self.model_runner.forward(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2785, in forward
    output = self._forward_raw(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2898, in _forward_raw
    ret, can_run_graph = self.forward_extend(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2698, in forward_extend
    self.model.forward(
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2292, in forward
    hidden_states = self.model(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2101, in forward
    hidden_states, residual, topk_indices = layer(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 1790, in forward
    hidden_states = self.mlp(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 267, in forward
    x, _ = self.down_proj(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/linear.py", line 1509, in forward
    output_parallel = self.quant_method.apply(self, input_parallel, bias=bias_)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 971, in apply
    return scheme.apply_weights(layer, x, bias=bias)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py", line 176, in apply_weights
    x_scale = x[2]
IndexError: tuple index out of range

[2026-07-07 10:14:24 DP4 TP4 EP4] Scheduler hit an exception: Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3621, in run_scheduler_process
    scheduler.run_event_loop()
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1304, in run_event_loop
    dispatch_event_loop(self)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3518, in dispatch_event_loop
    scheduler.event_loop_normal_disagg_decode()
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/disaggregation/decode.py", line 1155, in event_loop_normal_disagg_decode
    result = self.run_batch(batch)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 2729, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 335, in forward_batch_generation
    self.verify(batch, spec_info)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 746, in verify
    batch_result = self.target_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 469, in forward_batch_generation
    out = self.model_runner.forward(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2785, in forward
    output = self._forward_raw(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2898, in _forward_raw
    ret, can_run_graph = self.forward_extend(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2698, in forward_extend
    self.model.forward(
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2292, in forward
    hidden_states = self.model(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2101, in forward
    hidden_states, residual, topk_indices = layer(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 1790, in forward
    hidden_states = self.mlp(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 267, in forward
    x, _ = self.down_proj(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/linear.py", line 1509, in forward
    output_parallel = self.quant_method.apply(self, input_parallel, bias=bias_)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 971, in apply
    return scheme.apply_weights(layer, x, bias=bias)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py", line 176, in apply_weights
    x_scale = x[2]
IndexError: tuple index out of range

[2026-07-07 10:14:24 DP3 TP3 EP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3621, in run_scheduler_process
    scheduler.run_event_loop()
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1304, in run_event_loop
    dispatch_event_loop(self)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3518, in dispatch_event_loop
    scheduler.event_loop_normal_disagg_decode()
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/disaggregation/decode.py", line 1155, in event_loop_normal_disagg_decode
    result = self.run_batch(batch)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 2729, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 335, in forward_batch_generation
    self.verify(batch, spec_info)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 746, in verify
    batch_result = self.target_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 469, in forward_batch_generation
    out = self.model_runner.forward(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2785, in forward
    output = self._forward_raw(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2898, in _forward_raw
    ret, can_run_graph = self.forward_extend(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2698, in forward_extend
    self.model.forward(
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2292, in forward
    hidden_states = self.model(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2101, in forward
    hidden_states, residual, topk_indices = layer(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 1790, in forward
    hidden_states = self.mlp(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 267, in forward
    x, _ = self.down_proj(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/linear.py", line 1509, in forward
    output_parallel = self.quant_method.apply(self, input_parallel, bias=bias_)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 971, in apply
    return scheme.apply_weights(layer, x, bias=bias)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py", line 176, in apply_weights
    x_scale = x[2]
IndexError: tuple index out of range

[2026-07-07 10:14:25 DP5 TP5 EP5] Scheduler hit an exception: Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3621, in run_scheduler_process
    scheduler.run_event_loop()
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1304, in run_event_loop
    dispatch_event_loop(self)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3518, in dispatch_event_loop
    scheduler.event_loop_normal_disagg_decode()
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/disaggregation/decode.py", line 1155, in event_loop_normal_disagg_decode
    result = self.run_batch(batch)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 2729, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 335, in forward_batch_generation
    self.verify(batch, spec_info)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 746, in verify
    batch_result = self.target_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 469, in forward_batch_generation
    out = self.model_runner.forward(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2785, in forward
    output = self._forward_raw(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2898, in _forward_raw
    ret, can_run_graph = self.forward_extend(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2698, in forward_extend
    self.model.forward(
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2292, in forward
    hidden_states = self.model(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2101, in forward
    hidden_states, residual, topk_indices = layer(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 1790, in forward
    hidden_states = self.mlp(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 267, in forward
    x, _ = self.down_proj(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/linear.py", line 1509, in forward
    output_parallel = self.quant_method.apply(self, input_parallel, bias=bias_)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 971, in apply
    return scheme.apply_weights(layer, x, bias=bias)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py", line 176, in apply_weights
    x_scale = x[2]
IndexError: tuple index out of range

[2026-07-07 10:14:25 DP6 TP6 EP6] Scheduler hit an exception: Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3621, in run_scheduler_process
    scheduler.run_event_loop()
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1304, in run_event_loop
    dispatch_event_loop(self)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3518, in dispatch_event_loop
    scheduler.event_loop_normal_disagg_decode()
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/disaggregation/decode.py", line 1155, in event_loop_normal_disagg_decode
    result = self.run_batch(batch)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 2729, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 335, in forward_batch_generation
    self.verify(batch, spec_info)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 746, in verify
    batch_result = self.target_worker.forward_batch_generation(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 469, in forward_batch_generation
    out = self.model_runner.forward(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2785, in forward
    output = self._forward_raw(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2898, in _forward_raw
    ret, can_run_graph = self.forward_extend(
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 2698, in forward_extend
    self.model.forward(
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2292, in forward
    hidden_states = self.model(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 2101, in forward
    hidden_states, residual, topk_indices = layer(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 1790, in forward
    hidden_states = self.mlp(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/models/deepseek_v2.py", line 267, in forward
    x, _ = self.down_proj(
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/linear.py", line 1509, in forward
    output_parallel = self.quant_method.apply(self, input_parallel, bias=bias_)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 971, in apply
    return scheme.apply_weights(layer, x, bias=bias)
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py", line 176, in apply_weights
    x_scale = x[2]
IndexError: tuple index out of range


metax-host-017:9785:11680 [0] /workspace/out/Release/build/linux/x86_64/mccl/macaify/src/misc/socket.cc:38 MCCL WARN socketProgressOpt: Call to recv from 192.168.2.93<57953> failed : Connection reset by peer

metax-host-017:9790:11674 [0] /workspace/out/Release/build/linux/x86_64/mccl/macaify/src/misc/socket.cc:38 MCCL WARN socketProgressOpt: Call to recv from 192.168.2.18<40925> failed : Connection reset by peer
[2026-07-07 10:15:10] Subprocess scheduler_0 (pid=9133) crashed with exit code -3. Triggering SIGQUIT for cleanup...
[2026-07-07 10:15:10] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
./sglang/decode.sh: line 140:  8931 Killed                  python3 -m sglang.launch_server "${args[@]}"