Loading safetensors checkpoint shards:   0% Completed | 0/152 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   1% Completed | 1/152 [00:00<00:23,  6.30it/s]
Loading safetensors checkpoint shards:   1% Completed | 2/152 [00:00<00:19,  7.57it/s]
Loading safetensors checkpoint shards:   2% Completed | 3/152 [00:00<00:45,  3.29it/s]
Loading safetensors checkpoint shards:   3% Completed | 5/152 [00:00<00:26,  5.46it/s]
Loading safetensors checkpoint shards:   5% Completed | 7/152 [00:01<00:20,  6.99it/s]
Loading safetensors checkpoint shards:   6% Completed | 9/152 [00:01<00:19,  7.40it/s]
Loading safetensors checkpoint shards:   7% Completed | 11/152 [00:01<00:15,  8.82it/s]
Loading safetensors checkpoint shards:   9% Completed | 13/152 [00:01<00:13, 10.51it/s]
Loading safetensors checkpoint shards:  10% Completed | 15/152 [00:01<00:13, 10.13it/s]
Loading safetensors checkpoint shards:  11% Completed | 17/152 [00:02<00:12, 10.80it/s]
Loading safetensors checkpoint shards:  12% Completed | 19/152 [00:02<00:19,  6.86it/s]
Loading safetensors checkpoint shards:  14% Completed | 21/152 [00:02<00:16,  7.88it/s]
Loading safetensors checkpoint shards:  15% Completed | 23/152 [00:02<00:14,  9.10it/s]
Loading safetensors checkpoint shards:  16% Completed | 25/152 [00:02<00:11, 10.77it/s]
Loading safetensors checkpoint shards:  18% Completed | 27/152 [00:03<00:11, 11.29it/s]
Loading safetensors checkpoint shards:  19% Completed | 29/152 [00:03<00:10, 11.26it/s]
Loading safetensors checkpoint shards:  20% Completed | 31/152 [00:03<00:11, 10.79it/s]
Loading safetensors checkpoint shards:  22% Completed | 33/152 [00:03<00:10, 10.84it/s]
Loading safetensors checkpoint shards:  23% Completed | 35/152 [00:03<00:10, 11.60it/s]
Loading safetensors checkpoint shards:  24% Completed | 37/152 [00:04<00:10, 11.22it/s]
Loading safetensors checkpoint shards:  26% Completed | 39/152 [00:04<00:15,  7.06it/s]
Loading safetensors checkpoint shards:  27% Completed | 41/152 [00:04<00:13,  8.23it/s]
Loading safetensors checkpoint shards:  28% Completed | 43/152 [00:04<00:12,  8.79it/s]
Loading safetensors checkpoint shards:  30% Completed | 45/152 [00:05<00:12,  8.39it/s]
Loading safetensors checkpoint shards:  31% Completed | 47/152 [00:05<00:11,  9.44it/s]
Loading safetensors checkpoint shards:  32% Completed | 49/152 [00:05<00:10, 10.23it/s]
Loading safetensors checkpoint shards:  34% Completed | 51/152 [00:05<00:09, 10.90it/s]
Loading safetensors checkpoint shards:  35% Completed | 53/152 [00:05<00:09, 10.83it/s]
Loading safetensors checkpoint shards:  36% Completed | 55/152 [00:05<00:08, 11.32it/s]
Loading safetensors checkpoint shards:  38% Completed | 57/152 [00:06<00:07, 11.93it/s]
Loading safetensors checkpoint shards:  39% Completed | 59/152 [00:06<00:07, 12.62it/s]
Loading safetensors checkpoint shards:  40% Completed | 61/152 [00:06<00:13,  6.62it/s]
Loading safetensors checkpoint shards:  41% Completed | 63/152 [00:07<00:11,  7.90it/s]
Loading safetensors checkpoint shards:  43% Completed | 65/152 [00:07<00:09,  9.11it/s]
Loading safetensors checkpoint shards:  44% Completed | 67/152 [00:07<00:08, 10.24it/s]
Loading safetensors checkpoint shards:  45% Completed | 69/152 [00:07<00:07, 10.94it/s]
Loading safetensors checkpoint shards:  47% Completed | 71/152 [00:07<00:07, 11.09it/s]
Loading safetensors checkpoint shards:  48% Completed | 73/152 [00:07<00:06, 11.42it/s]
Loading safetensors checkpoint shards:  49% Completed | 75/152 [00:07<00:06, 12.55it/s]
Loading safetensors checkpoint shards:  51% Completed | 77/152 [00:08<00:05, 13.06it/s]
Loading safetensors checkpoint shards:  52% Completed | 79/152 [00:08<00:05, 12.59it/s]
Loading safetensors checkpoint shards:  53% Completed | 81/152 [00:08<00:05, 12.46it/s]
Loading safetensors checkpoint shards:  55% Completed | 83/152 [00:08<00:05, 13.78it/s]
Loading safetensors checkpoint shards:  56% Completed | 85/152 [00:08<00:04, 13.78it/s]
Loading safetensors checkpoint shards:  57% Completed | 87/152 [00:09<00:12,  5.22it/s]
Loading safetensors checkpoint shards:  59% Completed | 90/152 [00:09<00:08,  7.10it/s]
Loading safetensors checkpoint shards:  61% Completed | 92/152 [00:09<00:07,  8.20it/s]
Loading safetensors checkpoint shards:  62% Completed | 94/152 [00:10<00:06,  8.82it/s]
Loading safetensors checkpoint shards:  63% Completed | 96/152 [00:10<00:05, 10.25it/s]
Loading safetensors checkpoint shards:  64% Completed | 98/152 [00:10<00:05, 10.39it/s]
Loading safetensors checkpoint shards:  66% Completed | 100/152 [00:10<00:04, 11.49it/s]
Loading safetensors checkpoint shards:  67% Completed | 102/152 [00:10<00:04, 11.31it/s]
Loading safetensors checkpoint shards:  68% Completed | 104/152 [00:10<00:04, 11.08it/s]
Loading safetensors checkpoint shards:  70% Completed | 106/152 [00:11<00:05,  9.15it/s]
Loading safetensors checkpoint shards:  71% Completed | 108/152 [00:11<00:04, 10.10it/s]
Loading safetensors checkpoint shards:  72% Completed | 110/152 [00:11<00:03, 10.81it/s]
Loading safetensors checkpoint shards:  74% Completed | 112/152 [00:11<00:03, 11.36it/s]
Loading safetensors checkpoint shards:  75% Completed | 114/152 [00:11<00:03, 12.53it/s]
Loading safetensors checkpoint shards:  76% Completed | 116/152 [00:11<00:02, 12.90it/s]
Loading safetensors checkpoint shards:  78% Completed | 118/152 [00:12<00:02, 11.85it/s]
Loading safetensors checkpoint shards:  79% Completed | 120/152 [00:12<00:02, 11.54it/s]
Loading safetensors checkpoint shards:  80% Completed | 122/152 [00:13<00:05,  5.80it/s]
Loading safetensors checkpoint shards:  82% Completed | 124/152 [00:13<00:04,  6.80it/s]
Loading safetensors checkpoint shards:  83% Completed | 126/152 [00:13<00:03,  8.16it/s]
Loading safetensors checkpoint shards:  84% Completed | 128/152 [00:13<00:02,  8.96it/s]
Loading safetensors checkpoint shards:  86% Completed | 130/152 [00:13<00:02,  9.64it/s]
Loading safetensors checkpoint shards:  87% Completed | 132/152 [00:13<00:01, 10.77it/s]
Loading safetensors checkpoint shards:  88% Completed | 134/152 [00:14<00:01, 10.32it/s]
Loading safetensors checkpoint shards:  89% Completed | 136/152 [00:14<00:01, 11.51it/s]
Loading safetensors checkpoint shards:  91% Completed | 138/152 [00:14<00:01, 12.29it/s]
Loading safetensors checkpoint shards:  92% Completed | 140/152 [00:14<00:00, 12.16it/s]
Loading safetensors checkpoint shards:  93% Completed | 142/152 [00:14<00:00, 12.60it/s]
Loading safetensors checkpoint shards:  95% Completed | 144/152 [00:14<00:00, 12.61it/s]
Loading safetensors checkpoint shards:  96% Completed | 146/152 [00:14<00:00, 12.29it/s]
Loading safetensors checkpoint shards:  97% Completed | 148/152 [00:15<00:00, 12.18it/s]
Loading safetensors checkpoint shards:  99% Completed | 150/152 [00:15<00:00, 12.29it/s]
Loading safetensors checkpoint shards: 100% Completed | 152/152 [00:15<00:00, 12.07it/s]
Loading safetensors checkpoint shards: 100% Completed | 152/152 [00:15<00:00,  9.82it/s]

[2026-07-03 17:19:02 DP5 TP5 EP5] Load weight end. type=GlmMoeDsaForCausalLM, dtype=torch.bfloat16, avail mem=16.88 GB, mem usage=43.03 GB.
[2026-07-03 17:19:02 DP0 TP0 EP0] Load weight end. type=GlmMoeDsaForCausalLM, dtype=torch.bfloat16, avail mem=16.88 GB, mem usage=43.03 GB.
[2026-07-03 17:19:02 DP4 TP4 EP4] Load weight end. type=GlmMoeDsaForCausalLM, dtype=torch.bfloat16, avail mem=16.88 GB, mem usage=43.03 GB.
[2026-07-03 17:19:02 DP6 TP6 EP6] Load weight end. type=GlmMoeDsaForCausalLM, dtype=torch.bfloat16, avail mem=17.11 GB, mem usage=43.03 GB.
[2026-07-03 17:19:03 DP7 TP7 EP7] Load weight end. type=GlmMoeDsaForCausalLM, dtype=torch.bfloat16, avail mem=17.04 GB, mem usage=43.03 GB.
[2026-07-03 17:19:06 DP3 TP3 EP3] Load weight end. type=GlmMoeDsaForCausalLM, dtype=torch.bfloat16, avail mem=16.96 GB, mem usage=43.03 GB.
[2026-07-03 17:19:07 DP1 TP1 EP1] Load weight end. type=GlmMoeDsaForCausalLM, dtype=torch.bfloat16, avail mem=16.96 GB, mem usage=43.03 GB.
[2026-07-03 17:19:07 DP2 TP2 EP2] Load weight end. type=GlmMoeDsaForCausalLM, dtype=torch.bfloat16, avail mem=16.96 GB, mem usage=43.03 GB.
[2026-07-03 17:19:07 DP0 TP0 EP0] Using KV cache dtype: torch.bfloat16
[2026-07-03 17:19:07 DP0 TP0 EP0] KV Cache is allocated. #tokens: 57216, KV size: 5.86 GB
[2026-07-03 17:19:07 DP0 TP0 EP0] Memory pool end. avail mem=10.70 GB
[2026-07-03 17:19:07 DP1 TP1 EP1] KV Cache is allocated. #tokens: 57216, KV size: 5.86 GB
[2026-07-03 17:19:07 DP2 TP2 EP2] KV Cache is allocated. #tokens: 57216, KV size: 5.86 GB
[2026-07-03 17:19:07 DP5 TP5 EP5] KV Cache is allocated. #tokens: 57216, KV size: 5.86 GB
[2026-07-03 17:19:07 DP7 TP7 EP7] KV Cache is allocated. #tokens: 57216, KV size: 5.86 GB
[2026-07-03 17:19:07 DP1 TP1 EP1] Memory pool end. avail mem=10.78 GB
[2026-07-03 17:19:07 DP5 TP5 EP5] Memory pool end. avail mem=10.70 GB
[2026-07-03 17:19:07 DP2 TP2 EP2] Memory pool end. avail mem=10.78 GB
[2026-07-03 17:19:07 DP3 TP3 EP3] KV Cache is allocated. #tokens: 57216, KV size: 5.86 GB
[2026-07-03 17:19:07 DP7 TP7 EP7] Memory pool end. avail mem=10.86 GB
[2026-07-03 17:19:07 DP3 TP3 EP3] Memory pool end. avail mem=10.78 GB
[2026-07-03 17:19:07 DP6 TP6 EP6] KV Cache is allocated. #tokens: 57216, KV size: 5.86 GB
[2026-07-03 17:19:07 DP6 TP6 EP6] Memory pool end. avail mem=10.93 GB
[2026-07-03 17:19:07 DP4 TP4 EP4] KV Cache is allocated. #tokens: 57216, KV size: 5.86 GB
[2026-07-03 17:19:07 DP4 TP4 EP4] Memory pool end. avail mem=10.70 GB
[2026-07-03 17:19:07 DP7 TP7 EP7] Capture cuda graph begin. This can take up to several minutes. avail mem=10.30 GB
[2026-07-03 17:19:07 DP5 TP5 EP5] Capture cuda graph begin. This can take up to several minutes. avail mem=10.14 GB
[2026-07-03 17:19:07 DP4 TP4 EP4] Capture cuda graph begin. This can take up to several minutes. avail mem=10.14 GB
[2026-07-03 17:19:07 DP0 TP0 EP0] Capture cuda graph begin. This can take up to several minutes. avail mem=10.14 GB
[2026-07-03 17:19:07 DP0 TP0 EP0] Capture cuda graph bs [1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]
[2026-07-03 17:19:07 DP1 TP1 EP1] Capture cuda graph begin. This can take up to several minutes. avail mem=10.22 GB
[2026-07-03 17:19:07 DP2 TP2 EP2] Capture cuda graph begin. This can take up to several minutes. avail mem=10.22 GB
[2026-07-03 17:19:07 DP3 TP3 EP3] Capture cuda graph begin. This can take up to several minutes. avail mem=10.22 GB
[2026-07-03 17:19:07 DP6 TP6 EP6] Capture cuda graph begin. This can take up to several minutes. avail mem=10.37 GB
Capturing batches (bs=32 avail_mem=10.08 GB):   0%|                              | 0/20 [00:00<?, ?it/s]/data/xlu/hy3/mxshmem/src/transport/ibgda/mlx5/ibgda_mlx5.cpp:2310: non-zero status: -2 mlx5_devices_init failed 
/data/xlu/hy3/mxshmem/src/transport/transport.cpp:412: non-zero status: -2 mxshmem_ibgda_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:613: non-zero status: -2 mxshmemi_transport_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:733: non-zero status: -2 mxshmemi_common_init failed
/data/xlu/hy3/mxshmem/src/init/init_device.cpp:57: non-zero status: -2 mxshmem_internal_init_thread failed 

/data/xlu/glm5/mxshmem/out/include/mxshmem/host/mxshmem_api.h:85: non-zero status: -2, exiting due to error in mxshmemi_init_thread... /data/xlu/hy3/mxshmem/src/transport/ibgda/mlx5/ibgda_mlx5.cpp:2310: non-zero status: -2 mlx5_devices_init failed 
/data/xlu/hy3/mxshmem/src/transport/transport.cpp:412: non-zero status: -2 mxshmem_ibgda_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:613: non-zero status: -2 mxshmemi_transport_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:733: non-zero status: -2 mxshmemi_common_init failed
/data/xlu/hy3/mxshmem/src/init/init_device.cpp:57: non-zero status: -2 mxshmem_internal_init_thread failed 

/data/xlu/glm5/mxshmem/out/include/mxshmem/host/mxshmem_api.h:85: non-zero status: -2, exiting due to error in mxshmemi_init_thread... /data/xlu/hy3/mxshmem/src/transport/ibgda/mlx5/ibgda_mlx5.cpp:2310: non-zero status: -2 mlx5_devices_init failed 
/data/xlu/hy3/mxshmem/src/transport/transport.cpp:412: non-zero status: -2 mxshmem_ibgda_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:613: non-zero status: -2 mxshmemi_transport_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:733: non-zero status: -2 mxshmemi_common_init failed
/data/xlu/hy3/mxshmem/src/init/init_device.cpp:57: non-zero status: -2 mxshmem_internal_init_thread failed 

/data/xlu/glm5/mxshmem/out/include/mxshmem/host/mxshmem_api.h:85: non-zero status: -2, exiting due to error in mxshmemi_init_thread... /data/xlu/hy3/mxshmem/src/transport/ibgda/mlx5/ibgda_mlx5.cpp:2310: non-zero status: -2 mlx5_devices_init failed 
/data/xlu/hy3/mxshmem/src/transport/transport.cpp:412: non-zero status: -2 mxshmem_ibgda_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:613: non-zero status: -2 mxshmemi_transport_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:733: non-zero status: -2 mxshmemi_common_init failed
/data/xlu/hy3/mxshmem/src/init/init_device.cpp:57: non-zero status: -2 mxshmem_internal_init_thread failed 

/data/xlu/glm5/mxshmem/out/include/mxshmem/host/mxshmem_api.h:85: non-zero status: -2, exiting due to error in mxshmemi_init_thread... /data/xlu/hy3/mxshmem/src/transport/ibgda/mlx5/ibgda_mlx5.cpp:2310: non-zero status: -2 mlx5_devices_init failed 
/data/xlu/hy3/mxshmem/src/transport/transport.cpp:412: non-zero status: -2 mxshmem_ibgda_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:613: non-zero status: -2 mxshmemi_transport_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:733: non-zero status: -2 mxshmemi_common_init failed
/data/xlu/hy3/mxshmem/src/init/init_device.cpp:57: non-zero status: -2 mxshmem_internal_init_thread failed 

/data/xlu/glm5/mxshmem/out/include/mxshmem/host/mxshmem_api.h:85: non-zero status: -2, exiting due to error in mxshmemi_init_thread... /data/xlu/hy3/mxshmem/src/transport/ibgda/mlx5/ibgda_mlx5.cpp:2310: non-zero status: -2 mlx5_devices_init failed 
/data/xlu/hy3/mxshmem/src/transport/transport.cpp:412: non-zero status: -2 mxshmem_ibgda_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:613: non-zero status: -2 mxshmemi_transport_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:733: non-zero status: -2 mxshmemi_common_init failed
/data/xlu/hy3/mxshmem/src/init/init_device.cpp:57: non-zero status: -2 mxshmem_internal_init_thread failed 

/data/xlu/glm5/mxshmem/out/include/mxshmem/host/mxshmem_api.h:85: non-zero status: -2, exiting due to error in mxshmemi_init_thread... /data/xlu/hy3/mxshmem/src/transport/ibgda/mlx5/ibgda_mlx5.cpp:2310: non-zero status: -2 mlx5_devices_init failed 
/data/xlu/hy3/mxshmem/src/transport/transport.cpp:412: non-zero status: -2 mxshmem_ibgda_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:613: non-zero status: -2 mxshmemi_transport_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:733: non-zero status: -2 mxshmemi_common_init failed
/data/xlu/hy3/mxshmem/src/init/init_device.cpp:57: non-zero status: -2 mxshmem_internal_init_thread failed 

/data/xlu/glm5/mxshmem/out/include/mxshmem/host/mxshmem_api.h:85: non-zero status: -2, exiting due to error in mxshmemi_init_thread... /data/xlu/hy3/mxshmem/src/transport/ibgda/mlx5/ibgda_mlx5.cpp:2310: non-zero status: -2 mlx5_devices_init failed 
/data/xlu/hy3/mxshmem/src/transport/transport.cpp:412: non-zero status: -2 mxshmem_ibgda_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:613: non-zero status: -2 mxshmemi_transport_init failed
/data/xlu/hy3/mxshmem/src/init/init.cpp:733: non-zero status: -2 mxshmemi_common_init failed
/data/xlu/hy3/mxshmem/src/init/init_device.cpp:57: non-zero status: -2 mxshmem_internal_init_thread failed 

/data/xlu/glm5/mxshmem/out/include/mxshmem/host/mxshmem_api.h:85: non-zero status: -2, exiting due to error in mxshmemi_init_thread... [2026-07-03 17:19:21] DataParallelController hit an exception: Traceback (most recent call last):
  File "/workspace/sglang-260415-183721-3e30019/python/sglang/srt/managers/data_parallel_controller.py", line 616, in run_data_parallel_controller_process
    controller = DataParallelController(
  File "/workspace/sglang-260415-183721-3e30019/python/sglang/srt/managers/data_parallel_controller.py", line 196, in __init__
    self.launch_dp_attention_schedulers(server_args, port_args)
  File "/workspace/sglang-260415-183721-3e30019/python/sglang/srt/managers/data_parallel_controller.py", line 409, in launch_dp_attention_schedulers
    self.launch_tensor_parallel_group(
  File "/workspace/sglang-260415-183721-3e30019/python/sglang/srt/managers/data_parallel_controller.py", line 496, in launch_tensor_parallel_group
    scheduler_info.append(scheduler_pipe_readers[i].recv())
  File "/opt/conda/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/opt/conda/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/opt/conda/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

[2026-07-03 17:19:21] Received sigquit from a child process. It usually means the child failed.
./sglang/decode.sh: line 140: 30855 Killed                  python3 -m sglang.launch_server "${args[@]}"