• Members 2 posts
    2026年11月10日 09:10

    一、软硬件信息

    1. 服务器厂家:浪潮 Inspur CS5998H3(255 核 / 1007 GB 内存)
    2. 沐曦 GPU 型号:曦云 C600-A ×16(单卡 73728 MiB)
    3. 操作系统内核版本:Ubuntu 22.04.5 LTS,5.15.0-119-generic
    4. 是否开启 CPU 虚拟化:未开启
    5. mx-smi 回显:
    mx-smi  version: 2.3.4
    Kernel Mode Driver Version: 3.9.6
    MACA Version: 3.8.1.3
    BIOS Version: 2.3.5.0
    Attached GPUs: 16
    | 0   MetaX C600-A | 0           Off | 0000:06:00.0  | 0%          Disabled |
    | 143W / 850W      | 35C          P0 | 178/73728 MiB | Available            |
    
    1. docker info 回显:本机是 k3s 单机集群(k3s v1.34.3+k3s1,containerd v1.34.0-k3s2),镜像由 k8s 节点直接拉取,不走 docker daemon。

    2. 镜像版本:cr.metax-tech.com/public-ai-release/maca/sglang:0.5.14-maca.ai3.8.2.122-torch2.10-py312-ubuntu22.04-amd64(linux/amd64)

    3. 启动容器命令:实际是用 k8s Deployment 起单卡实例,关键片段如下

    image: cr.metax-tech.com/public-ai-release/maca/sglang:0.5.14-maca.ai3.8.2.122-torch2.10-py312-ubuntu22.04-amd64
    resources:
      limits:
        metax-tech.com/gpu: "1"
    env:
      - REPO_ID=Qwen/Qwen3-0.6B
      - ENGINE_VERSION=v0.5.14
      - ENGINE_ARGS=--context-length 8192 --tensor-parallel-size 1 --chunked-prefill-size 1024 --cpu-offload-gb 1 --dp-size 1 --enable-dp-attention --tool-call-parser auto --mem-fraction-static 0.8
    

    镜像自带的启动脚本在 ENGINE_VERSION >= 0.4.4 时会自动追加 --enable-torch-compile,等价于下面这条直接启动命令:

    docker run --rm -it --network host --ipc host \
      --device /dev/mxcd \
      -v /models/Qwen3-0.6B:/models/Qwen3-0.6B \
      cr.metax-tech.com/public-ai-release/maca/sglang:0.5.14-maca.ai3.8.2.122-torch2.10-py312-ubuntu22.04-amd64 \
      python3 -m sglang.launch_server \
        --model-path /models/Qwen3-0.6B \
        --host 0.0.0.0 --port 8000 \
        --tp 1 --mem-fraction-static 0.8 \
        --enable-torch-compile
    

    同一镜像去掉 --enable-torch-compile 可以正常起服务、正常出结果,所以问题只跟 torch compile 相关。

    1. 容器内执行命令与关键日志:
    File "/opt/conda/lib/python3.12/site-packages/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 410, in set_torch_compile_config
        monkey_patch_torch_compile()
    NameError: name 'monkey_patch_torch_compile' is not defined
    ...
    Received sigquit from a child process. ...
    

    之后进程退出、容器重启,服务起不来。

    二、问题现象与定位

    1. 现象:只要带 --enable-torch-compile 启动,在 decode CUDA graph 初始化阶段就直接抛 NameError 退出;不带该参数一切正常。

    2. 定位:镜像内 decode_cuda_graph_runner.py 有两处异常:

    $ grep -n "monkey_patch_torch_compile\|def set_torch_compile_config\|patch_torch" \
        /opt/conda/lib/python3.12/site-packages/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py
    39:from sglang.srt.compilation import torch_compile_decoration
    40:from sglang.srt.compilation.torch_compile_decoration import set_torch_compile_config
    397:def set_torch_compile_config():
    410:    monkey_patch_torch_compile()
    

    第 40 行已经从 torch_compile_decoration 导入了 set_torch_compile_config,但第 397 行又在本文件里重新定义了一份同名函数(覆盖了导入的版本),而这份本地实现第 410 行调用了 monkey_patch_torch_compile(),文件里却没有对应的 import。

    monkey_patch_torch_compile 本身是存在的,定义在 sglang/srt/utils/patch_torch.py:106:

    $ grep -n "def monkey_patch_torch_compile" /opt/conda/lib/python3.12/site-packages/sglang/srt/utils/patch_torch.py
    106:def monkey_patch_torch_compile():
    
    1. 与上游对比:上游 v0.5.14 的 decode_cuda_graph_runner.py 没有这段本地重复定义,直接用第 40 行导入的 set_torch_compile_config,因此不会触发该问题。看起来是沐曦分支合入本地改动时,把一份旧版函数体带进来了,漏掉了对应的 import。

    2. 临时规避(我们已经在自己的 Dockerfile 里加了,只是想知道官方是否会在后续版本修复):

    sed -i '/^from sglang.srt.compilation.torch_compile_decoration import set_torch_compile_config$/a from sglang.srt.utils.patch_torch import monkey_patch_torch_compile' \
        /opt/conda/lib/python3.12/site-packages/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py
    

    补上 import 后,同一镜像带 --enable-torch-compile 冷启动约 4 分钟(compile + CUDA graph 捕获约 3 分钟),The server is fired up and ready to roll!,/v1/models 与 /v1/chat/completions 均正常。

    1. 希望确认:
    2. 这是已知问题吗?后续 sglang 镜像版本(0.5.15 及以后)是否会修复?
    3. 0.5.14 镜像的 mcoplib 版本信息里打印的是 SGlang Op Version = 0.5.13,这是否会影响该版本的其他功能?

    附件:无

  • Members 1029 posts
    2026年11月10日 10:32

    尊敬的开发者您好,请提供K8S启动的yml文件详细脚本以及相关启动日志

  • Members 2 posts
    2026年11月10日 11:06

    感谢回复,补充完整信息如下。

    一、环境信息

    说明:我们线上用的是基于该官方镜像构建的封装镜像(只额外加了内网 pip 源、dumb-init 和 SDK,并在出问题的文件里补了这行 import 作为临时规避)。下面的复现是在封装镜像里删掉我们补的那行 import(即恢复官方文件原样,diff 仅此一行)后跑的,启动参数与线上默认参数完全一致。

    二、现象

    只要带 --enable-torch-compile 启动,服务在 decode CUDA graph capture 阶段抛 NameError,随后子进程被 kill,服务退出。去掉该参数可以正常启动。

    三、根因

    文件 /opt/conda/lib/python3.12/site-packages/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py:

    • 第 410 行 set_torch_compile_config() 里调用了 monkey_patch_torch_compile()
    • 但文件里没有 from sglang.srt.utils.patch_torch import monkey_patch_torch_compile

    即 0.5.14 发布包里这一行 import 缺失,属于打包问题,不是参数或环境配置问题。

    四、K8S 启动 yaml(最小复现,Pod)

    apiVersion: v1
    kind: Pod
    metadata:
      name: repro-sglang-nameerror
      namespace: default
    spec:
      containers:
      - name: sglang
        image: cr.metax-tech.com/public-ai-release/maca/sglang:0.5.14-maca.ai3.8.2.122-torch2.10-py312-ubuntu22.04-amd64
        imagePullPolicy: IfNotPresent
        command: ["/bin/bash", "-c", "sleep infinity"]
        resources:
          limits:
            metax-tech.com/gpu: "1"
          requests:
            metax-tech.com/gpu: "1"
        volumeMounts:
        - name: models
          mountPath: /models
        - name: shm
          mountPath: /dev/shm
      volumes:
      - name: models
        hostPath:
          path: /bata1/models/ws-qwen3-8b    # 内含 Qwen/Qwen3-8B
          type: Directory
      - name: shm
        emptyDir:
          medium: Memory
          sizeLimit: 16Gi
    

    进入容器后执行的启动命令(与线上默认参数一致):

    python3 -m sglang.launch_server \
      --model-path /models/Qwen/Qwen3-8B \
      --host 0.0.0.0 --port 8000 \
      --trust-remote-code --enable-mixed-chunk \
      --context-length 8192 \
      --tensor-parallel-size 1 \
      --chunked-prefill-size 1024 \
      --cpu-offload-gb 1 \
      --dp-size 1 --enable-dp-attention \
      --tool-call-parser auto \
      --mem-fraction-static 0.8 \
      --enable-torch-compile --torch-compile-max-bs 16
    

    五、完整启动日志

    (server_args= 那一行超长,为便于阅读已省略,完整参数即上面的启动命令)

    /opt/conda/lib/python3.12/site-packages/flashinfer/gdn_prefill.py:27: UserWarning: fla package is not installed, gdn_prefill module will not be available. Install via: pip install flash_linear_attention-*.whl
      warnings.warn(
    /opt/conda/lib/python3.12/site-packages/sglang/launch_server.py:61: UserWarning: 'python -m sglang.launch_server' is still supported, but 'sglang serve' is the recommended entrypoint.
      Example: sglang serve --model-path <model> [options]
      warnings.warn(
    [2026-11-10 11:01:08] Attention backend not specified. Use flashinfer backend by default.
    [2026-11-10 11:01:08] flashinfer.comm allreduce_fusion API is not available (No module named 'flashinfer.comm'), falling back to standard implementation
    [2026-11-10 11:01:09] server_args=ServerArgs(...)
    [2026-11-10 11:01:11] Auto-detected --tool-call-parser as 'qwen' from chat template
    INFO Print the version information of mcoplib during compilation.
    
    Version info:Mcoplib_Version = '0.4.10'
    Build_Maca_Version = '3.8.0.25'
    Min_Compatibility_Maca_Version = '3.7.0'
    GIT_BRANCH = 'HEAD'
    GIT_COMMIT = '9ca2367'
    Vllm Op Version = 0.25.0
    SGlang Op Version  = 0.5.13
    
    INFO Staring Check the current MACA version of the operating environment.
    
    INFO: Release major.minor matching,  successful:3.8.
    
    INFO Staring Check the MACA minimum compatibility version of the operating environment.
    
    INFO: MACA minimum compatibility check passed: 3.8.2 >= 3.7.0.
    
    [2026-11-10 11:01:12] Using default HuggingFace chat template with detected content format: string
    [2026-11-10 11:01:12] Auto-detected template features: reasoning_config=ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None), reasoning_parser=qwen3, tool_call_parser=qwen
    flashinfer.comm allreduce_fusion API is not available (No module named 'flashinfer.comm'), falling back to standard implementation
    [2026-11-10 11:01:36] Init torch distributed begin.
    [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
    [2026-11-10 11:01:38] DCP disabled, dcp_size=1, tp_size=1
    [2026-11-10 11:01:38] Init torch distributed ends. elapsed=2.08 s, mem usage=0.01 GB
    [2026-11-10 11:01:39] Load weight begin. avail mem=70.96 GB
    Multi-thread loading shards: 100% Completed | 5/5 [00:14<00:00,  2.98s/it]
    [2026-11-10 11:01:55] Load weight end. elapsed=16.34 s, type=Qwen3ForCausalLM, avail mem=56.70 GB, mem usage=14.26 GB.
    [2026-11-10 11:01:57] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 309553, K size: 21.26 GB, V size: 21.26 GB
    [2026-11-10 11:01:57] Memory pool end. avail mem=13.95 GB
    [2026-11-10 11:01:57] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules).
    [2026-11-10 11:01:57] Capture target decode CUDA graph begin. backend=full, num_tokens_per_bs=1, bs=[1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], avail mem=13.29 GB
    [2026-11-10 11:01:57] Scheduler hit an exception: Traceback (most recent call last):
      File "/opt/conda/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 4358, in run_scheduler_process
        scheduler = Scheduler(
                    ^^^^^^^^^^
      File "/opt/conda/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 428, in __init__
        self.init_model_worker()
      File "/opt/conda/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 871, in init_model_worker
        self.init_all_cuda_graphs()
      File "/opt/conda/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 854, in init_all_cuda_graphs
        self.tp_worker.init_cuda_graphs()
      File "/opt/conda/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 351, in init_cuda_graphs
        self.model_runner.init_cuda_graphs(
      File "/opt/conda/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 961, in init_cuda_graphs
        self.init_decode_cuda_graph()
      File "/opt/conda/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 2671, in init_decode_cuda_graph
        self.decode_cuda_graph_runner = graph_runners[self.device](self)
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.12/site-packages/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 613, in __init__
        set_torch_compile_config()
      File "/opt/conda/lib/python3.12/site-packages/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 410, in set_torch_compile_config
        monkey_patch_torch_compile()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    NameError: name 'monkey_patch_torch_compile' is not defined
    
    [2026-11-10 11:01:57] Received sigquit from a child process. It usually means the child failed.
    [2026-11-10 11:01:57] kill_process_tree called: parent_pid=297, include_parent=True, pid=297
    

    六、临时规避

    1. 不传 --enable-torch-compile 可正常启动(我们实测启动约 98s,chat / stream 均正常,只是首次请求有 JIT 开销);
    2. 或在 decode_cuda_graph_runner.py 顶部补一行
      from sglang.srt.utils.patch_torch import monkey_patch_torch_compile,我们目前就是这样规避的。

    麻烦确认下这是否会在下个版本修复,以及有没有官方推荐的 workaround。

    另外顺带问一个疑问:0.5.14 镜像里 mcoplib 打印的版本是 SGlang Op Version = 0.5.13(见上面日志),这个是否正常?谢谢。

  • Members 1029 posts
    2026年11月10日 14:43

    尊敬的开发者您好,感谢您的反馈,后续版本会修复。0.5.14 镜像里 mcoplib 打印的版本是 SGlang Op Version = 0.5.13正常情况,后续版本会修复。

  • Members 1029 posts
    2026年11月10日 16:51

    尊敬的开发者您好,
    1、当前规避方法是可以的
    2、下一个0.5.19发布的版本修复了这个问题
    3、mcoplib是沐曦算子通用算子库,这里显示的是最小兼容的sglang版本

  • arrow_forward

    Thread has been moved from 解决中.