沐曦C500 双机16卡 Ubuntu22.04 系统
运行容器：cr.metax-tech.com/public-init/maca/sglang:v0.5.8-deepseek-v4-rc3-maca.ai.3.7.1.9-torch2.8-py310-ubuntu22.04-amd64
MX-SMI 2.3.1 Kernel Mode Driver Version: 3.9.10 |
| MACA Version: 3.7.2.0 BIOS Version: 1.33.5.0
运行DeepSeek-V4-Flash-AWQ-W8A8 模型，rdma设备只能使用2个
File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 3032, in run_scheduler_process
scheduler = Scheduler(
File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 350, in __init__
self.init_model_worker()
File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 541, in init_model_worker
self.init_tp_model_worker()
File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 504, in init_tp_model_worker
self.tp_worker = TpModelWorker(
File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 246, in __init__
self._init_model_runner()
File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 329, in _init_model_runner
self._model_runner = ModelRunner(
File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 408, in __init__
self.initialize()
File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 488, in initialize
self.load_model()
File "/opt/conda/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 1110, in load_model
raise ValueError(
ValueError: TP rank 0 could finish the model loading, but there are other ranks that didn't finish loading. It is likely due to unexpected failures (e.g., OOM) or a slow node.
0] MCCL INFO /workspace/out/Release/build/linux/x86_64/mccl/macaify/src/misc/socket.cc:655 -> 2
sd:58212:60273 [0] MCCL INFO /workspace/out/Release/build/linux/x86_64/mccl/macaify/src/misc/socket.cc:678 -> 2
sd:58212:60273 [0] MCCL INFO /workspace/out/Release/build/linux/x86_64/mccl/macaify/src/ras/rasnet.cc:125 -> 2
需要哪些特殊的环境变量？
