MetaX-Tech Developer Forum 论坛首页
  • 沐曦开发者
search
Sign in

ZhaiLanSanMu

  • Members
  • Joined 2026年7月15日
  • message 帖子
  • forum 主题
  • favorite 关注者
  • favorite_border Follows
  • person_outline 详细信息

ZhaiLanSanMu has posted 6 messages.

  • See post chevron_right
    ZhaiLanSanMu
    Members
    gpu-label 无法将 C500 节点 metax.com/gpu.installed设置为true 已解决 2026年7月21日 16:21

    我把vendorID改成0x9999就正常了,感谢

  • See post chevron_right
    ZhaiLanSanMu
    Members
    gpu-label 无法将 C500 节点 metax.com/gpu.installed设置为true 已解决 2026年7月17日 15:28

    最先是报vender参数解析的问题:
    helm upgrade --install metax-operator ./packages/metax-operator-0.15.2.tgz -f ./values/metax-operator.yaml -n metax-operator
    Release "metax-operator" does not exist. Installing it now.
    I0717 14:51:05.514079 1811828 warnings.go:107] "Warning: spec.template.spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].preference.matchExpressions[0].key: node-role.kubernetes.io/master is use \"node-role.kubernetes.io/control-plane\" instead"
    Error: server-side apply failed for object /cluster-operator gpu.metax-tech.com/v1alpha1, Kind=ClusterOperator: ClusterOperator.gpu.metax-tech.com "cluster-operator" is invalid: [spec.vendor.charDev: Invalid value: "null": spec.vendor.charDev in body must be of type string: "null", spec.vendor.domain: Invalid value: "null": spec.vendor.domain in body must be of type string: "null", spec.vendor.driver: Invalid value: "null": spec.vendor.driver in body must be of type string: "null", spec.vendor.vendorID: Invalid value: "null": spec.vendor.vendorID in body must be of type string: "null", spec.vendor.virtDriver: Invalid value: "null": spec.vendor.virtDriver in body must be of type string: "null"]

    指定如下的vendor后,helm 部署成功,但是gpu-label无法识别 C500 卡:

    vendor:
    vendorID: "9999" # PCI Vendor ID (lspci shows Device [9999:4001])
    driver: "metax" # Kernel driver module (Kernel driver in use: METAX)
    virtDriver: "vfio-pci" # VFIO driver for GPU passthrough to KVM/QEMU VMs
    charDev: "mxcd" # Character device (/dev/mxcd - MetaX Control Device)
    domain: "metax.com" # K8s extended resource prefix (e.g., metax.com/gpu)

  • See post chevron_right
    ZhaiLanSanMu
    Members
    gpu-label 无法将 C500 节点 metax.com/gpu.installed设置为true 已解决 2026年7月17日 15:26

    就是参考的这个文档,出现上述问题。

  • See post chevron_right
    ZhaiLanSanMu
    Members
    gpu-label 无法将 C500 节点 metax.com/gpu.installed设置为true 已解决 2026年7月17日 15:06

    gpu scheduler安装后gpu label依然无法正常打标:

    kubectl get pods -n metax-operator
    NAME READY STATUS RESTARTS AGE
    metax-gpu-label-5l7cd 1/1 Running 0 5m8s
    metax-gpu-label-cgz9p 1/1 Running 0 4s
    metax-gpu-scheduler-bdcc887d5-l8rfx 2/2 Running 0 51s
    metax-operator-64cf4b48b8-7fbhx 1/1 Running 0 5m26s
    kubectl logs -f metax-gpu-label-cgz9p -n metax-operator
    Defaulted container "gpu-labelgen" out of: gpu-labelgen, init-log-dir (init)
    {"level":"info","ts":"2026-07-17T06:45:03Z","logger":"label-generator.entry","msg":"Starting manager"}
    {"level":"info","ts":"2026-07-17T06:45:03Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
    {"level":"info","ts":"2026-07-17T06:45:03Z","msg":"Starting EventSource","controller":"node","controllerGroup":"","controllerKind":"Node","source":"kind source: *v1.Node"}
    {"level":"info","ts":"2026-07-17T06:45:03Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
    {"level":"info","ts":"2026-07-17T06:45:03Z","msg":"Starting Controller","controller":"node","controllerGroup":"","controllerKind":"Node"}
    {"level":"info","ts":"2026-07-17T06:45:03Z","msg":"Starting workers","controller":"node","controllerGroup":"","controllerKind":"Node","worker count":1}
    {"level":"info","ts":"2026-07-17T06:45:03Z","logger":"label-generator.UpdateEvent","msg":"label changed","diff":{"metax.com/gpu.installed":"nil => false"}}
    ^C

  • See post chevron_right
    ZhaiLanSanMu
    Members
    gpu-label 无法将 C500 节点 metax.com/gpu.installed设置为true 已解决 2026年7月17日 11:48

    请问一下 0.15.2版本是暂时不支持volcano的方式部署吗?

  • See post chevron_right
    ZhaiLanSanMu
    Members
    gpu-label 无法将 C500 节点 metax.com/gpu.installed设置为true 已解决 2026年7月16日 17:17

    一、软硬件信息
    1.服务器厂家:
    2.沐曦GPU型号:C500
    3.操作系统内核版本:5.15.0-119-generic
    4.是否开启CPU虚拟化:
    lsmod | grep kvm
    kvm_intel 372736 0
    kvm 1036288 1 kvm_intel
    5.mx-smi回显:

    mx-smi
    mx-smi version: 2.3.1

    =================== MetaX System Management Interface Log ===================
    Timestamp : Thu Jul 16 17:12:01 2026

    Attached GPUs : 8
    +---------------------------------------------------------------------------------+
    | MX-SMI 2.3.1 Kernel Mode Driver Version: 3.9.10 |
    | MACA Version: 3.7.2.0 BIOS Version: 1.33.5.0 |
    |------------------+-----------------+---------------------+----------------------|
    | Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
    | Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
    |==================+=================+=====================+======================|
    | 0 MetaX C500 | 0 Off | 0000:06:00.0 | 0% Disabled |
    | 71W / 350W | 46C P9 | 61938/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 1 MetaX C500 | 1 Off | 0000:07:00.0 | 0% Disabled |
    | 74W / 350W | 46C P9 | 63240/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 2 MetaX C500 | 2 Off | 0000:08:00.0 | 0% Disabled |
    | 71W / 350W | 46C P9 | 63238/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 3 MetaX C500 | 3 Off | 0000:09:00.0 | 0% Disabled |
    | 75W / 350W | 45C P9 | 61940/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 4 MetaX C500 | 4 Off | 0000:95:00.0 | 0% Disabled |
    | 68W / 350W | 39C P9 | 61942/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 5 MetaX C500 | 5 Off | 0000:96:00.0 | 0% Disabled |
    | 70W / 350W | 39C P9 | 63238/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 6 MetaX C500 | 6 Off | 0000:97:00.0 | 0% Disabled |
    | 69W / 350W | 39C P9 | 63234/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+
    | 7 MetaX C500 | 7 Off | 0000:98:00.0 | 0% Disabled |
    | 70W / 350W | 40C P9 | 61940/65536 MiB | Available |
    +------------------+-----------------+---------------------+----------------------+

    +---------------------------------------------------------------------------------+
    | Process: |
    | GPU PID Process Name GPU Memory |
    | Usage(MiB) |
    |=================================================================================|
    | 0 673394 sglang::schedul 61054 |
    | 1 673395 sglang::schedul 62356 |
    | 2 673396 sglang::schedul 62354 |
    | 3 673397 sglang::schedul 61056 |
    | 4 673398 sglang::schedul 61058 |
    | 5 673399 sglang::schedul 62354 |
    | 6 673400 sglang::schedul 62350 |
    | 7 673401 sglang::schedul 61056 |
    +---------------------------------------------------------------------------------+

    End of Log

    二、问题现象

    C500节点无法被正确打标

    kubectl logs -f metax-gpu-label-27gfr -n metax-operator
    Defaulted container "gpu-labelgen" out of: gpu-labelgen, init-log-dir (init)
    {"level":"info","ts":"2026-07-16T08:48:19Z","logger":"label-generator.entry","msg":"Starting manager"}
    {"level":"info","ts":"2026-07-16T08:48:19Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
    {"level":"info","ts":"2026-07-16T08:48:19Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
    {"level":"info","ts":"2026-07-16T08:48:19Z","msg":"Starting EventSource","controller":"node","controllerGroup":"","controllerKind":"Node","source":"kind source: *v1.Node"}
    {"level":"info","ts":"2026-07-16T08:48:19Z","msg":"Starting Controller","controller":"node","controllerGroup":"","controllerKind":"Node"}
    {"level":"info","ts":"2026-07-16T08:48:19Z","msg":"Starting workers","controller":"node","controllerGroup":"","controllerKind":"Node","worker count":1}
    {"level":"info","ts":"2026-07-16T08:48:19Z","logger":"label-generator.UpdateEvent","msg":"label changed","diff":{"metax.com/gpu.installed":"nil => false"}}

    operator版本:
    NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
    metax-operator metax-operator 4 2026-07-15 11:42:51.857938402 +0800 CST deployed metax-operator-0.15.2 0.15.2

    helm value配置:

    metax-operator values

    Registry for container images

    registry: "hub.xxx.com/metax"

    Driver payload configuration

    driver:
    payload:
    version: 3.8.0.10-amd64

    Maca payload configuration

    maca:
    payload:
    registry: "hub.sudoinfotech.com/metax"
    images:
    - "maca:3.8.0.11-ubuntu22.04-amd64"

    vendor:
    vendorID: "9999"
    driver: "metax"
    virtDriver: "vfio-pci"
    charDev: "mxcd"
    domain: "metax.com"

    openshift:
    deploy: false

  • 沐曦开发者论坛
powered by misago