感谢回复,补充完整信息如下。
一、环境信息
- GPU:曦云 C600(MXC600),单卡复现
- 宿主机驱动:Kernel Mode Driver 3.9.6,MACA 3.8.2.6(mx-smi 2.3.4)
- 镜像:cr.metax-tech.com/public-ai-release/maca/sglang:0.5.14-maca.ai3.8.2.122-torch2.10-py312-ubuntu22.04-amd64
- 架构 / OS:amd64 / Ubuntu 22.04 / Python 3.12
- 模型:Qwen3-8B(本地目录,非量化,bf16)
说明:我们线上用的是基于该官方镜像构建的封装镜像(只额外加了内网 pip 源、dumb-init 和 SDK,并在出问题的文件里补了这行 import 作为临时规避)。下面的复现是在封装镜像里删掉我们补的那行 import(即恢复官方文件原样,diff 仅此一行)后跑的,启动参数与线上默认参数完全一致。
二、现象
只要带 --enable-torch-compile 启动,服务在 decode CUDA graph capture 阶段抛 NameError,随后子进程被 kill,服务退出。去掉该参数可以正常启动。
三、根因
文件 /opt/conda/lib/python3.12/site-packages/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py:
- 第 410 行
set_torch_compile_config()里调用了monkey_patch_torch_compile() - 但文件里没有
from sglang.srt.utils.patch_torch import monkey_patch_torch_compile
即 0.5.14 发布包里这一行 import 缺失,属于打包问题,不是参数或环境配置问题。
四、K8S 启动 yaml(最小复现,Pod)
apiVersion: v1
kind: Pod
metadata:
name: repro-sglang-nameerror
namespace: default
spec:
containers:
- name: sglang
image: cr.metax-tech.com/public-ai-release/maca/sglang:0.5.14-maca.ai3.8.2.122-torch2.10-py312-ubuntu22.04-amd64
imagePullPolicy: IfNotPresent
command: ["/bin/bash", "-c", "sleep infinity"]
resources:
limits:
metax-tech.com/gpu: "1"
requests:
metax-tech.com/gpu: "1"
volumeMounts:
- name: models
mountPath: /models
- name: shm
mountPath: /dev/shm
volumes:
- name: models
hostPath:
path: /bata1/models/ws-qwen3-8b # 内含 Qwen/Qwen3-8B
type: Directory
- name: shm
emptyDir:
medium: Memory
sizeLimit: 16Gi
进入容器后执行的启动命令(与线上默认参数一致):
python3 -m sglang.launch_server \
--model-path /models/Qwen/Qwen3-8B \
--host 0.0.0.0 --port 8000 \
--trust-remote-code --enable-mixed-chunk \
--context-length 8192 \
--tensor-parallel-size 1 \
--chunked-prefill-size 1024 \
--cpu-offload-gb 1 \
--dp-size 1 --enable-dp-attention \
--tool-call-parser auto \
--mem-fraction-static 0.8 \
--enable-torch-compile --torch-compile-max-bs 16
五、完整启动日志
(server_args= 那一行超长,为便于阅读已省略,完整参数即上面的启动命令)
/opt/conda/lib/python3.12/site-packages/flashinfer/gdn_prefill.py:27: UserWarning: fla package is not installed, gdn_prefill module will not be available. Install via: pip install flash_linear_attention-*.whl
warnings.warn(
/opt/conda/lib/python3.12/site-packages/sglang/launch_server.py:61: UserWarning: 'python -m sglang.launch_server' is still supported, but 'sglang serve' is the recommended entrypoint.
Example: sglang serve --model-path <model> [options]
warnings.warn(
[2026-11-10 11:01:08] Attention backend not specified. Use flashinfer backend by default.
[2026-11-10 11:01:08] flashinfer.comm allreduce_fusion API is not available (No module named 'flashinfer.comm'), falling back to standard implementation
[2026-11-10 11:01:09] server_args=ServerArgs(...)
[2026-11-10 11:01:11] Auto-detected --tool-call-parser as 'qwen' from chat template
INFO Print the version information of mcoplib during compilation.
Version info:Mcoplib_Version = '0.4.10'
Build_Maca_Version = '3.8.0.25'
Min_Compatibility_Maca_Version = '3.7.0'
GIT_BRANCH = 'HEAD'
GIT_COMMIT = '9ca2367'
Vllm Op Version = 0.25.0
SGlang Op Version = 0.5.13
INFO Staring Check the current MACA version of the operating environment.
INFO: Release major.minor matching, successful:3.8.
INFO Staring Check the MACA minimum compatibility version of the operating environment.
INFO: MACA minimum compatibility check passed: 3.8.2 >= 3.7.0.
[2026-11-10 11:01:12] Using default HuggingFace chat template with detected content format: string
[2026-11-10 11:01:12] Auto-detected template features: reasoning_config=ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None), reasoning_parser=qwen3, tool_call_parser=qwen
flashinfer.comm allreduce_fusion API is not available (No module named 'flashinfer.comm'), falling back to standard implementation
[2026-11-10 11:01:36] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2026-11-10 11:01:38] DCP disabled, dcp_size=1, tp_size=1
[2026-11-10 11:01:38] Init torch distributed ends. elapsed=2.08 s, mem usage=0.01 GB
[2026-11-10 11:01:39] Load weight begin. avail mem=70.96 GB
Multi-thread loading shards: 100% Completed | 5/5 [00:14<00:00, 2.98s/it]
[2026-11-10 11:01:55] Load weight end. elapsed=16.34 s, type=Qwen3ForCausalLM, avail mem=56.70 GB, mem usage=14.26 GB.
[2026-11-10 11:01:57] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 309553, K size: 21.26 GB, V size: 21.26 GB
[2026-11-10 11:01:57] Memory pool end. avail mem=13.95 GB
[2026-11-10 11:01:57] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules).
[2026-11-10 11:01:57] Capture target decode CUDA graph begin. backend=full, num_tokens_per_bs=1, bs=[1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], avail mem=13.29 GB
[2026-11-10 11:01:57] Scheduler hit an exception: Traceback (most recent call last):
File "/opt/conda/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 4358, in run_scheduler_process
scheduler = Scheduler(
^^^^^^^^^^
File "/opt/conda/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 428, in __init__
self.init_model_worker()
File "/opt/conda/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 871, in init_model_worker
self.init_all_cuda_graphs()
File "/opt/conda/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 854, in init_all_cuda_graphs
self.tp_worker.init_cuda_graphs()
File "/opt/conda/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 351, in init_cuda_graphs
self.model_runner.init_cuda_graphs(
File "/opt/conda/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 961, in init_cuda_graphs
self.init_decode_cuda_graph()
File "/opt/conda/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 2671, in init_decode_cuda_graph
self.decode_cuda_graph_runner = graph_runners[self.device](self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/site-packages/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 613, in __init__
set_torch_compile_config()
File "/opt/conda/lib/python3.12/site-packages/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 410, in set_torch_compile_config
monkey_patch_torch_compile()
^^^^^^^^^^^^^^^^^^^^^^^^^^
NameError: name 'monkey_patch_torch_compile' is not defined
[2026-11-10 11:01:57] Received sigquit from a child process. It usually means the child failed.
[2026-11-10 11:01:57] kill_process_tree called: parent_pid=297, include_parent=True, pid=297
六、临时规避
- 不传
--enable-torch-compile可正常启动(我们实测启动约 98s,chat / stream 均正常,只是首次请求有 JIT 开销); - 或在
decode_cuda_graph_runner.py顶部补一行
from sglang.srt.utils.patch_torch import monkey_patch_torch_compile,我们目前就是这样规避的。
麻烦确认下这是否会在下个版本修复,以及有没有官方推荐的 workaround。
另外顺带问一个疑问:0.5.14 镜像里 mcoplib 打印的版本是 SGlang Op Version = 0.5.13(见上面日志),这个是否正常?谢谢。