一、软硬件信息
1.服务器厂家: 联想
2.沐曦GPU型号：C500
3.操作系统内核版本：Linux master 5.15.0-72-generic #79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
4.是否开启CPU虚拟化：没有
5.mx-smi回显：正常
沐曦目前dra的支持方式有哪些？
沐曦的共享方式SRIOV sgpu shared-GPU
jasongideon
- Members
- Joined 2026年1月6日
-
一、软硬件信息
1.服务器厂家: 联想
2.沐曦GPU型号：C500
3.操作系统内核版本：Linux master 5.15.0-72-generic #79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
4.是否开启CPU虚拟化：没有
5.mx-smi回显：正常参照官方说明：developer.metax-tech.com/api/client/document/preview/1269/k8s/03_component.html#oj1rzffy91ep1
helm开启gpuScheduler.deploy=true， v2后，metax-gpu-scheduler，运行异常
gpu-aware {"level":"info","msg":"gpu-aware started. [protocal=https, port=:8080]","time":"2026-07-23T09:02:42Z"} kube-scheduler I0723 09:04:12.722489 1 serving.go:386] Generated self-signed cert in-memory kube-scheduler E0723 09:04:12.722722 1 run.go:72] "command failed" err="no kind \"KubeSchedulerConfiguration\" is registered for version \"kubescheduler.config.k8s.io/v1beta2\" in scheme \"pkg/scheduler/apis/config/scheme/scheme.go:29\"" stream closed: EOF for admin/metax-gpu-scheduler-667d4d85b5-fcpzr (kube-scheduler) -
一、软硬件信息
1.服务器厂家: 联想
2.沐曦GPU型号：C500
3.操作系统内核版本：Linux master 5.15.0-72-generic #79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
4.是否开启CPU虚拟化：没有
5.mx-smi回显：正常参照官方说明：developer.metax-tech.com/api/client/document/preview/1269/k8s/03_component.html#oj1rzffy91ep1
helm开启gpuScheduler.deploy=true， v2后，metax-gpu-scheduler不创建
helm安装cmd如下
helm install metax-operator ./metax-operator-0.15.2.tgz --create-namespace -n metax-operator --wait --set driver.payload.version=3.3.0.4-amd64 --set maca.payload.images={"maca-native:3.3.0.4-ubuntu22.04-amd64"} --set gpuScheduler.deploy=true --set gpuScheduler.version=v2operator log如下
/workspace/cloud/gpu-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:461 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /workspace/cloud/gpu-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /workspace/cloud/gpu-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296 2026-07-22T06:25:49Z info DaemonSet:metax-gpu-device not found in namespace:metax-operator {"error": "DaemonSet.apps \"metax-gpu-device\" not found"} 2026-07-22T06:25:49Z info ConfigMap:metax-device-config not found in namespace:metax-operator {"error": "ConfigMap \"metax-device-config\" not found"} 2026-07-22T06:25:49Z info CompUpdate:gpu-device resource updating plan {"ResourceSyncAction": {"ClusterRole":"Update","ClusterRoleBinding":"Update","ConfigMap/metax-device-config":"Update","DaemonSet":"Update","Role":"Update","RoleBinding":"Update","ServiceAccount":"Update"}, "ResourceDeployed": {"ClusterRole":false,"ClusterRoleBinding":false,"ConfigMap/metax-device-config":false,"DaemonSet":false,"Dependency":false,"Role":false,"RoleBinding":false,"ServiceAccount":false}} 2026-07-22T06:25:49Z info CompUpdate:gpu-device resource updating {"resourceKind": "RoleBinding", "syncAction": "Update"} 2026-07-22T06:25:49Z info CompUpdate:gpu-device resource updating {"resourceKind": "ServiceAccount", "syncAction": "Update"} 2026-07-22T06:25:49Z info CompUpdate:gpu-device resource updating {"resourceKind": "ClusterRole", "syncAction": "Update"} 2026-07-22T06:25:49Z info CompUpdate:gpu-device resource updating {"resourceKind": "ClusterRoleBinding", "syncAction": "Update"} 2026-07-22T06:25:49Z info CompUpdate:gpu-device resource updating {"resourceKind": "Role", "syncAction": "Update"} 2026-07-22T06:25:49Z info CompUpdate:gpu-device resource updating {"resourceKind": "DaemonSet", "syncAction": "Update"} 2026-07-22T06:25:49Z info CompUpdate:gpu-device resource updating {"resourceKind": "ConfigMap/metax-device-config", "syncAction": "Update"} 2026-07-22T06:25:49Z info CompUpdate:gpu-device resource updating result {"ResourceSyncAction": {"ClusterRole":"Done","ClusterRoleBinding":"Done","ConfigMap/metax-device-config":"Done","DaemonSet":"Done","Role":"Done","RoleBinding":"Update","ServiceAccount":"Done"}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"ConfigMap/metax-device-config":true,"DaemonSet":true,"Dependency":false,"Role":true,"RoleBinding":false,"ServiceAccount":true}} 2026-07-22T06:25:49Z error CompUpdate:gpu-device failed to update resource {"error": "comp:gpu-device, resource:RoleBinding, action:Update, error:rolebindings.rbac.authorization.k8s.io \"metax-gpu-device\" not found", "errorCauses": [{"error": "comp:gpu-device, resource:RoleBinding, action:Update, error:rolebindings.rbac.authorization.k8s.io \"metax-gpu-device\" not found"}]} metax-tech.com/cloud/metax-operator/controllers.CompUpdate /workspace/cloud/gpu-operator/controllers/resource_process.go:798 metax-tech.com/cloud/metax-operator/controllers.(*ClusterOperatorReconciler).runCompUpdate /workspace/cloud/gpu-operator/controllers/clusteroperator_controller.go:319 metax-tech.com/cloud/metax-operator/controllers.(*ClusterOperatorReconciler).syncComponents /workspace/cloud/gpu-operator/controllers/clusteroperator_controller.go:180 metax-tech.com/cloud/metax-operator/controllers.(*ClusterOperatorReconciler).Reconcile /workspace/cloud/gpu-operator/controllers/clusteroperator_controller.go:105 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /workspace/cloud/gpu-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:216 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /workspace/cloud/gpu-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:461 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /workspace/cloud/gpu-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:421 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /workspace/cloud/gpu-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:296 2026-07-22T06:25:49Z info Deployment:metax-gpu-scheduler not found in namespace:metax-operator {"error": "Deployment.apps \"metax-gpu-scheduler\" not found"} 2026-07-22T06:25:49Z info ConfigMap:metax-gpu-scheduler-config not found in namespace:metax-operator {"error": "ConfigMap \"metax-gpu-scheduler-config\" not found"} 2026-07-22T06:25:49Z info CompUpdate:gpu-scheduler resource updating plan {"ResourceSyncAction": {"ConfigMap/metax-gpu-scheduler-config":"Update","MutatingWebhookConfiguration":"Update","Service":"Update"}, "ResourceDeployed": {"ConfigMap/metax-gpu-scheduler-config":false,"Dependency":false,"Deployment":false,"MutatingWebhookConfiguration":false,"Service":false}} 2026-07-22T06:25:49Z info CompUpdate:gpu-scheduler resource updating {"resourceKind": "ConfigMap/metax-gpu-scheduler-config", "syncAction": "Update"} 2026-07-22T06:25:49Z info CompUpdate:gpu-scheduler resource updating {"resourceKind": "Service", "syncAction": "Update"} 2026-07-22T06:25:49Z info CompUpdate:gpu-scheduler resource updating {"resourceKind": "MutatingWebhookConfiguration", "syncAction": "Update"} 2026-07-22T06:25:49Z info CompUpdate:gpu-scheduler resource updating result {"ResourceSyncAction": {"ConfigMap/metax-gpu-scheduler-config":"Done","MutatingWebhookConfiguration":"Done","Service":"Done"}, "ResourceDeployed": {"ConfigMap/metax-gpu-scheduler-config":true,"Dependency":false,"Deployment":false,"MutatingWebhookConfiguration":true,"Service":true}} 2026-07-22T06:25:49Z info CompUpdate:openshift resource updating plan {"ResourceSyncAction": {}, "ResourceDeployed": {"ClusterRole":false,"ClusterRoleBinding":false,"Dependency":false,"SecurityContextConstraints":false}} 2026-07-22T06:25:49Z info CompUpdate:openshift resource updating result {"ResourceSyncAction": {}, "ResourceDeployed": {"ClusterRole":false,"ClusterRoleBinding":false,"Dependency":false,"SecurityContextConstraints":false}} 2026-07-22T06:25:49Z info DaemonSet:metax-topo-worker not found in namespace:metax-operator {"error": "DaemonSet.apps \"metax-topo-worker\" not found"} 2026-07-22T06:25:49Z info CompUpdate:topo-worker resource updating plan {"ResourceSyncAction": {"ClusterRole":"Update","ClusterRoleBinding":"Update","Service":"Update","ServiceAccount":"Update"}, "ResourceDeployed": {"ClusterRole":false,"ClusterRoleBinding":false,"DaemonSet":false,"Dependency":false,"Service":false,"ServiceAccount":false}} 2026-07-22T06:25:49Z info CompUpdate:topo-worker resource updating {"resourceKind": "Service", "syncAction": "Update"} 2026-07-22T06:25:49Z info CompUpdate:topo-worker resource updating {"resourceKind": "ServiceAccount", "syncAction": "Update"} 2026-07-22T06:25:49Z info CompUpdate:topo-worker resource updating {"resourceKind": "ClusterRole", "syncAction": "Update"} 2026-07-22T06:25:49Z info CompUpdate:topo-worker resource updating {"resourceKind": "ClusterRoleBinding", "syncAction": "Update"} 2026-07-22T06:25:50Z info CompUpdate:topo-worker resource updating result {"ResourceSyncAction": {"ClusterRole":"Done","ClusterRoleBinding":"Done","Service":"Done","ServiceAccount":"Done"}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"DaemonSet":false,"Dependency":false,"Service":true,"ServiceAccount":true}} 2026-07-22T06:25:50Z info ConfigMap:vendor-config not found in namespace:metax-operator {"error": "ConfigMap \"vendor-config\" not found"} 2026-07-22T06:25:50Z info CompUpdate:vendor resource updating plan {"ResourceSyncAction": {"ConfigMap/vendor-config":"Update"}, "ResourceDeployed": {"ConfigMap/vendor-config":false,"Dependency":false}} 2026-07-22T06:25:50Z info CompUpdate:vendor resource updating {"resourceKind": "ConfigMap/vendor-config", "syncAction": "Update"} 2026-07-22T06:25:50Z info CompUpdate:vendor resource updating result {"ResourceSyncAction": {"ConfigMap/vendor-config":"Done"}, "ResourceDeployed": {"ConfigMap/vendor-config":true,"Dependency":false}} 2026-07-22T06:25:50Z info DaemonSet:metax-data-exporter not found in namespace:metax-operator {"error": "DaemonSet.apps \"metax-data-exporter\" not found"} 2026-07-22T06:25:50Z info ConfigMap:metax-data-exporter-config not found in namespace:metax-operator {"error": "ConfigMap \"metax-data-exporter-config\" not found"} 2026-07-22T06:25:50Z info CompUpdate:data-exporter resource updating plan {"ResourceSyncAction": {"ConfigMap/metax-data-exporter-config":"Update","Service":"Update"}, "ResourceDeployed": {"ConfigMap/metax-data-exporter-config":false,"DaemonSet":false,"Dependency":false,"Service":false}} 2026-07-22T06:25:50Z info CompUpdate:data-exporter resource updating {"resourceKind": "Service", "syncAction": "Update"} 2026-07-22T06:25:50Z info CompUpdate:data-exporter resource updating {"resourceKind": "ConfigMap/metax-data-exporter-config", "syncAction": "Update"} 2026-07-22T06:25:50Z info CompUpdate:data-exporter resource updating result {"ResourceSyncAction": {"ConfigMap/metax-data-exporter-config":"Done","Service":"Done"}, "ResourceDeployed": {"ConfigMap/metax-data-exporter-config":true,"DaemonSet":false,"Dependency":false,"Service":true}} 2026-07-22T06:25:50Z info unknown field "spec.gpuScheduler" {"controller": "clusteroperator", "controllerGroup": "gpu.metax-tech.com", "controllerKind": "ClusterOperator", "ClusterOperator": {"name":"cluster-operator"}, "namespace": "", "name": "cluster-operator", "reconcileID": "0b90c8f1-044f-4682-94dc-a6dafb91ad02"} 2026-07-22T06:25:50Z info unknown field "spec.openshift" {"controller": "clusteroperator", "controllerGroup": "gpu.metax-tech.com", "controllerKind": "ClusterOperator", "ClusterOperator": {"name":"cluster-operator"}, "namespace": "", "name": "cluster-operator", "reconcileID": "0b90c8f1-044f-4682-94dc-a6dafb91ad02"} 2026-07-22T06:25:50Z info unknown field "spec.vendor" {"controller": "clusteroperator", "controllerGroup": "gpu.metax-tech.com", "controllerKind": "ClusterOperator", "ClusterOperator": {"name":"cluster-operator"}, "namespace": "", "name": "cluster-operator", "reconcileID": "0b90c8f1-044f-4682-94dc-a6dafb91ad02"} 2026-07-22T06:25:50Z info CompUpdate:gpu-label resource updating plan {"ResourceSyncAction": {"DaemonSet":"Update"}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"DaemonSet":true,"Dependency":false,"Role":true,"RoleBinding":true,"ServiceAccount":true}} 2026-07-22T06:25:50Z info CompUpdate:gpu-label resource updating {"resourceKind": "DaemonSet", "syncAction": "Update"} 2026-07-22T06:25:50Z info CompUpdate:gpu-label resource updating result {"ResourceSyncAction": {"DaemonSet":"Done"}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"DaemonSet":true,"Dependency":false,"Role":true,"RoleBinding":true,"ServiceAccount":true}} 2026-07-22T06:25:50Z info CompUpdate:container-runtime resource updating plan {"ResourceSyncAction": {"ClusterRoleBinding":"Update","DaemonSet":"Update"}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":false,"DaemonSet":true,"Dependency":false,"Role":true,"RoleBinding":true,"RuntimeClass":true,"ServiceAccount":true}} 2026-07-22T06:25:50Z info CompUpdate:container-runtime resource updating {"resourceKind": "ClusterRoleBinding", "syncAction": "Update"} 2026-07-22T06:25:50Z info CompUpdate:container-runtime resource updating {"resourceKind": "DaemonSet", "syncAction": "Update"} 2026-07-22T06:25:50Z info CompUpdate:container-runtime resource updating result {"ResourceSyncAction": {"ClusterRoleBinding":"Done","DaemonSet":"Done"}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"DaemonSet":true,"Dependency":false,"Role":true,"RoleBinding":true,"RuntimeClass":true,"ServiceAccount":true}} 2026-07-22T06:25:50Z info CompUpdate:driver resource updating plan {"ResourceSyncAction": {"DaemonSet":"Update"}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"ConfigMap/driver-config":true,"ConfigMap/metax-vfio-config":true,"DaemonSet":true,"Dependency":false,"Role":true,"RoleBinding":true,"ServiceAccount":true}} 2026-07-22T06:25:50Z info CompUpdate:driver resource updating {"resourceKind": "DaemonSet", "syncAction": "Update"} 2026-07-22T06:25:50Z info CompUpdate:driver resource updating result {"ResourceSyncAction": {"DaemonSet":"Done"}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"ConfigMap/driver-config":true,"ConfigMap/metax-vfio-config":true,"DaemonSet":true,"Dependency":false,"Role":true,"RoleBinding":true,"ServiceAccount":true}} 2026-07-22T06:25:50Z info DaemonSet:metax-data-exporter not found in namespace:metax-operator {"error": "DaemonSet.apps \"metax-data-exporter\" not found"} 2026-07-22T06:25:50Z info CompUpdate:data-exporter resource updating plan {"ResourceSyncAction": {}, "ResourceDeployed": {"ConfigMap/metax-data-exporter-config":true,"DaemonSet":false,"Dependency":false,"Service":true}} 2026-07-22T06:25:50Z info CompUpdate:data-exporter resource updating result {"ResourceSyncAction": {}, "ResourceDeployed": {"ConfigMap/metax-data-exporter-config":true,"DaemonSet":false,"Dependency":false,"Service":true}} 2026-07-22T06:25:50Z info DriverUpgrade ValidatingWebhook info {"name": "metax-upgrade-validating-webhook-configuration", "kind": "ValidatingWebhookConfiguration"} 2026-07-22T06:25:50Z info CompUpdate:driver-upgrade resource updating plan {"ResourceSyncAction": {"ValidatingWebhookConfiguration":"Update"}, "ResourceDeployed": {"Dependency":false,"ValidatingWebhookConfiguration":true}} 2026-07-22T06:25:50Z info CompUpdate:driver-upgrade resource updating {"resourceKind": "ValidatingWebhookConfiguration", "syncAction": "Update"} 2026-07-22T06:25:50Z info CompUpdate:driver-upgrade resource updating result {"ResourceSyncAction": {"ValidatingWebhookConfiguration":"Done"}, "ResourceDeployed": {"Dependency":false,"ValidatingWebhookConfiguration":true}} 2026-07-22T06:25:50Z info Deployment:metax-topo-master not found in namespace:metax-operator {"error": "Deployment.apps \"metax-topo-master\" not found"} 2026-07-22T06:25:50Z info CompUpdate:topo-master resource updating plan {"ResourceSyncAction": {}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"ConfigMap/metax-topo-master-config":true,"Dependency":false,"Deployment":false,"Secret":true,"Service":true,"ServiceAccount":true}} 2026-07-22T06:25:50Z info CompUpdate:topo-master resource updating result {"ResourceSyncAction": {}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"ConfigMap/metax-topo-master-config":true,"Dependency":false,"Deployment":false,"Secret":true,"Service":true,"ServiceAccount":true}} 2026-07-22T06:25:50Z info CompUpdate:maca resource updating plan {"ResourceSyncAction": {"DaemonSet":"Update","RoleBinding":"Update"}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"ConfigMap/maca-config":true,"DaemonSet":true,"Dependency":false,"Role":true,"RoleBinding":false,"ServiceAccount":true}} 2026-07-22T06:25:50Z info CompUpdate:maca resource updating {"resourceKind": "RoleBinding", "syncAction": "Update"} 2026-07-22T06:25:50Z info CompUpdate:maca resource updating {"resourceKind": "DaemonSet", "syncAction": "Update"} 2026-07-22T06:25:50Z info CompUpdate:maca resource updating result {"ResourceSyncAction": {"DaemonSet":"Done","RoleBinding":"Done"}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"ConfigMap/maca-config":true,"DaemonSet":true,"Dependency":false,"Role":true,"RoleBinding":true,"ServiceAccount":true}} 2026-07-22T06:25:50Z info CompUpdate:gpu-device resource updating plan {"ResourceSyncAction": {"DaemonSet":"Update","RoleBinding":"Update"}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"ConfigMap/metax-device-config":true,"DaemonSet":true,"Dependency":false,"Role":true,"RoleBinding":false,"ServiceAccount":true}} 2026-07-22T06:25:50Z info CompUpdate:gpu-device resource updating {"resourceKind": "RoleBinding", "syncAction": "Update"} 2026-07-22T06:25:50Z info CompUpdate:gpu-device resource updating {"resourceKind": "DaemonSet", "syncAction": "Update"} 2026-07-22T06:25:50Z info CompUpdate:gpu-device resource updating result {"ResourceSyncAction": {"DaemonSet":"Done","RoleBinding":"Done"}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"ConfigMap/metax-device-config":true,"DaemonSet":true,"Dependency":false,"Role":true,"RoleBinding":true,"ServiceAccount":true}} 2026-07-22T06:25:50Z info Deployment:metax-gpu-scheduler not found in namespace:metax-operator {"error": "Deployment.apps \"metax-gpu-scheduler\" not found"} 2026-07-22T06:25:50Z info CompUpdate:gpu-scheduler resource updating plan {"ResourceSyncAction": {}, "ResourceDeployed": {"ConfigMap/metax-gpu-scheduler-config":true,"Dependency":false,"Deployment":false,"MutatingWebhookConfiguration":true,"Service":true}} 2026-07-22T06:25:50Z info CompUpdate:gpu-scheduler resource updating result {"ResourceSyncAction": {}, "ResourceDeployed": {"ConfigMap/metax-gpu-scheduler-config":true,"Dependency":false,"Deployment":false,"MutatingWebhookConfiguration":true,"Service":true}} 2026-07-22T06:25:50Z info CompUpdate:openshift resource updating plan {"ResourceSyncAction": {}, "ResourceDeployed": {"ClusterRole":false,"ClusterRoleBinding":false,"Dependency":false,"SecurityContextConstraints":false}} 2026-07-22T06:25:50Z info CompUpdate:openshift resource updating result {"ResourceSyncAction": {}, "ResourceDeployed": {"ClusterRole":false,"ClusterRoleBinding":false,"Dependency":false,"SecurityContextConstraints":false}} 2026-07-22T06:25:50Z info DaemonSet:metax-topo-worker not found in namespace:metax-operator {"error": "DaemonSet.apps \"metax-topo-worker\" not found"} 2026-07-22T06:25:50Z info CompUpdate:topo-worker resource updating plan {"ResourceSyncAction": {}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"DaemonSet":false,"Dependency":false,"Service":true,"ServiceAccount":true}} 2026-07-22T06:25:50Z info CompUpdate:topo-worker resource updating result {"ResourceSyncAction": {}, "ResourceDeployed": {"ClusterRole":true,"ClusterRoleBinding":true,"DaemonSet":false,"Dependency":false,"Service":true,"ServiceAccount":true}} 2026-07-22T06:25:50Z info CompUpdate:vendor resource updating plan {"ResourceSyncAction": {}, "ResourceDeployed": {"ConfigMap/vendor-config":true,"Dependency":false}} 2026-07-22T06:25:50Z info CompUpdate:vendor resource updating result {"ResourceSyncAction": {}, "ResourceDeployed": {"ConfigMap/vendor-config":true,"Dependency":false}} -
一、软硬件信息
1.服务器厂家: 联想
2.沐曦GPU型号：C500
3.操作系统内核版本：Linux master 5.15.0-72-generic #79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
4.是否开启CPU虚拟化：没有
5.mx-smi回显：失败
二、问题现象
使用helm 部署metax-gpu-operator 0.14.0，安装metax-container-runtime步骤出现crash，报错如下。
沐曦的container runtime一直崩溃，重新部署operator也是，我更新到15.2也崩溃。
重启containerd、kubelet 、宿主机、都依然报错。
曾经有正常的时候，后面一直报错。 -
/* * 测试用例：演示沐曦GPU __ballot_sync bug及workaround验证 * * 问题：在warp 1中，线程59-63(lane_id 27-31)执行__ballot_sync时， * 即使predicate为真，返回0x00000000而不是正确的bitmask * */ #include <mc_runtime.h> #include <stdio.h> #define USE_METAX // 启用workaround // 原始的__ballot_sync实现（NVIDIA GPU正常，沐曦GPU有bug） __device__ uint32_t ballot_original(bool pred) { uint32_t FULL_WARP_ACTIVE_BMSK = 0xFFFFFFFF; return __ballot_sync(FULL_WARP_ACTIVE_BMSK, pred); } // Workaround：使用共享内存实现ballot（沐曦GPU专用） __device__ uint32_t ballot_workaround(bool pred) { uint32_t lane_id = threadIdx.x % 32; uint32_t thrdIdxInBlk = threadIdx.x + threadIdx.y * blockDim.x; uint32_t warp_id = thrdIdxInBlk / 32; __shared__ uint32_t s_warp_pred[512]; uint32_t idx = warp_id * 32 + lane_id; s_warp_pred[idx] = pred ? 1 : 0; __syncthreads(); uint32_t ballotResult = 0; for (int i = 0; i < 32; i++) { if (s_warp_pred[warp_id * 32 + i]) { ballotResult |= (1U << i); } } __syncthreads(); return ballotResult; } // 测试内核：重现生产环境中的bug场景 __global__ void test_ballot_bug( uint32_t* d_original_results, uint32_t* d_workaround_results, uint32_t nInfoBits ) { uint32_t thrdIdxInBlk = threadIdx.x + threadIdx.y * blockDim.x; uint32_t lane_id = threadIdx.x % 32; uint32_t warp_id = thrdIdxInBlk / 32; // 生产环境中的精确predicate逻辑 uint32_t N_MAX_INFO_BITS = 128; int16_t interleaverTblStartIdx = N_MAX_INFO_BITS - nInfoBits; bool pred = ((thrdIdxInBlk < N_MAX_INFO_BITS) && (thrdIdxInBlk >= interleaverTblStartIdx)) ? true : false; // 测试原始实现 uint32_t original_ballot = ballot_original(pred); // 测试workaround uint32_t workaround_ballot = ballot_workaround(pred); // 保存结果（每个warp只保存第一个线程的结果） if (lane_id == 0 && warp_id < 2) { // 只保存warp 0和warp 1 d_original_results[warp_id] = original_ballot; d_workaround_results[warp_id] = workaround_ballot; } // 打印调试信息 if (pred) { printf("threadIdx.x=%d, threadIdx.y=%d, thrdIdxInBlk=%d, lane_id=%d, warp_id=%d, pred=%d\n", threadIdx.x, threadIdx.y, thrdIdxInBlk, lane_id, warp_id, pred); printf(" original_ballot=0x%08x, workaround_ballot=0x%08x\n", original_ballot, workaround_ballot); } } // 验证结果的辅助函数 void verify_results(uint32_t* original, uint32_t* workaround, int nWarps, uint32_t expected_mask) { printf("\n========== 验证结果 ==========\n"); printf("预期bitmask (所有pred为true的线程): 0x%08x\n", expected_mask); printf("\n原始 __ballot_sync 结果:\n"); for (int i = 0; i < nWarps; i++) { printf(" Warp %d: 0x%08x %s\n", i, original[i], (original[i] == expected_mask) ? "✓ 正确" : "✗ 错误"); } printf("\nWorkaround 结果:\n"); for (int i = 0; i < nWarps; i++) { printf(" Warp %d: 0x%08x %s\n", i, workaround[i], (workaround[i] == expected_mask) ? "✓ 正确" : "✗ 错误"); } } int main() { // 测试参数：重现生产环境中的bug场景 uint32_t nInfoBits = 69; // 导致线程59-63的predicate为true uint32_t N_MAX_INFO_BITS = 128; printf("========== 测试配置 ==========\n"); printf("nInfoBits = %d, N_MAX_INFO_BITS = %d\n", nInfoBits, N_MAX_INFO_BITS); printf("Block dim: 32x16x1 (512 threads)\n"); printf("Grid dim: 1x1x1\n\n"); // 计算预期的bitmask // 线程59-63 (lane 27-31 in warp 1) 应该有 pred=true // 预期bitmask = 0xF8000000 (lanes 27-31 set) uint32_t expected_mask = 0xF8000000; // 分配设备内存 uint32_t *d_original, *d_workaround; uint32_t *h_original, *h_workaround; mcMalloc(&d_original, 2 * sizeof(uint32_t)); mcMalloc(&d_workaround, 2 * sizeof(uint32_t)); h_original = (uint32_t*)malloc(2 * sizeof(uint32_t)); h_workaround = (uint32_t*)malloc(2 * sizeof(uint32_t)); // 初始化 mcMemset(d_original, 0, 2 * sizeof(uint32_t)); mcMemset(d_workaround, 0, 2 * sizeof(uint32_t)); // 启动kernel printf("========== 启动Kernel ==========\n"); dim3 blockDim(32, 16, 1); // 32x16x1 = 512 threads dim3 gridDim(1, 1, 1); test_ballot_bug<<<gridDim, blockDim>>>(d_original, d_workaround, nInfoBits); // 同步并检查错误 mcError_t err = mcDeviceSynchronize(); if (err != mcSuccess) { printf("CUDA Error: %s\n", mcGetErrorString(err)); return -1; } // 复制结果回主机 mcMemcpy(h_original, d_original, 2 * sizeof(uint32_t), mcMemcpyDeviceToHost); mcMemcpy(h_workaround, d_workaround, 2 * sizeof(uint32_t), mcMemcpyDeviceToHost); // 验证结果 verify_results(h_original, h_workaround, 2, expected_mask); // 清理 mcFree(d_original); mcFree(d_workaround); free(h_original); free(h_workaround); printf("\n========== 测试完成 ==========\n"); return 0; }Warp 0, 2, 3的__ballot_sync工作正常
只有Warp 1的特定lane范围(27-31)受影响
Warp 1的lanes 27-31 (线程59-63)在predicate=true时，__ballot_sync返回0x00000000 -
#ifdef USE_METAX #include <maca_cooperative_groups.h> #include <maca_reduce.h> #include <maca_fp16.h> #include "math_utils.cuh" #include <mma.h> #include <assert.h> // for wmma using namespace mxmaca; #else #include <cooperative_groups.h> #include <cooperative_groups/reduce.h> #include "cuda_fp16.h" #include "math_utils.cuh" #include "mma.h" #include <assert.h> // for wmma using namespace nvcuda; #endif我这边在做国产化的适配，nvidia是mma.h和 namespace nvcude。请问沐曦对应的头文件在哪个path？哪个文件？namespace是mxmaca吗？
找到一堆mma.h，有没有例子？找的太费劲了
find /opt/maca-3.3.0/ -name *mma.h /opt/maca-3.3.0/include/mctlass/arch/mma.h /opt/maca-3.3.0/include/mctlass/arch/wmma.h /opt/maca-3.3.0/include/mctlass/gemm/warp/mma_tensor_op_wmma.h /opt/maca-3.3.0/include/mctlass/gemm/warp/mma_tensor_op_tile_iterator_wmma.h /opt/maca-3.3.0/include/mctlass/gemm/warp/mma.h /opt/maca-3.3.0/include/mctlass/gemm/thread/mma.h /opt/maca-3.3.0/include/mctlass/gemm/threadblock/maca_default_mma.h /opt/maca-3.3.0/include/mctlass/gemm/threadblock/maca_default_moe_mma.h /opt/maca-3.3.0/include/mctlass/gemm/threadblock/default_mma_core_wmma.h /opt/maca-3.3.0/include/mctlass/gemm/threadblock/maca_default_masked_grouped_mma.h /opt/maca-3.3.0/include/mctlass/gemm/threadblock/default_sparse_mma.h /opt/maca-3.3.0/include/mctlass/gemm/threadblock/maca_default_contiguous_grouped_mma.h /opt/maca-3.3.0/include/mctlass/gemm/threadblock/default_mma.h /opt/maca-3.3.0/include/mctlass/gemm/threadblock/maca_default_group_mma.h