4. 指标及标签说明

4.1. 指标说明

表 4.1 指标名称及说明

指标名称

所属类别

指标说明

mx_chip_hotspot_temp

Temperature

芯片内部最高温度，单位摄氏度

mx_chip_hbm_temp

Temperature

芯片HBM温度，单位摄氏度

mx_board_core_temp

Temperature

板卡温度，单位摄氏度

mx_optical_module_temp

Temperature

光模块温度，仅部分设备有光模块，如C500X

mx_vpue_usage

Usage

编码器利用率 （C600系列不支持）

mx_vpud_usage

Usage

解码器利用率 （C600系列不支持）

mx_memory_usage

Usage

HBM及系统内存利用率，指标标签 type 用于区分HBM和系统内存，vram代表HBM，xtt代表系统内存

mx_memory_total

Usage

HBM及系统内存总量，单位KB，指标标签 type 用于区分HBM和系统内存，vram代表HBM，xtt代表系统内存

mx_memory_used

Usage

HBM及系统内存使用量，单位KB，指标标签 type 用于区分HBM和系统内存，vram代表HBM，xtt代表系统内存

mx_gpu_usage

Usage

GPU利用率

mx_board_power

Power

功耗，单位毫瓦

mx_vpue_clock

Clock

编码器时钟频率，单位MHz （C600系列不支持）

mx_vpud_clock

Clock

解码时钟频率，单位MHz （C600系列不支持）

mx_mem_clock

Clock

HBM时钟频率，单位MHz

mx_gpu_clock

Clock

GPU时钟频率，单位MHz

mx_pcie_bw

PCIE

PCIe Tx和Rx吞吐量，单位MB/s

mx_pcie_speed

PCIE

设备每条lane上的PCIe传输速率，单位GT/s

mx_pcie_width

PCIE

设备带宽，表示PCIe lanes

mx_pcie_bridge_speed

PCIE

PCIe bridge传输速率，单位GT/s

mx_pcie_bridge_width

PCIE

PCIe bridge lanes

mx_eth_bw

ETH

ETH Tx和Rx吞吐量，单位MB/s

mx_mxlk_bw

MetaXLink

MetaXLink Tx和Rx吞吐量，单位MB/s

mx_mxlk_speed

MetaXLink

MetaXLink实时速率，单位GT/s

mx_mxlk_width

MetaXLink

MetaXLink lanes

mx_server_conn_status

MetaXLink

服务器MetaXLink整体状态，1为健康，0为有MetaXLink存在异常

mx_mxlk_aer_count

MetaXLink

MetaXLink AER计数，分UE和CE

mx_mxlk_traffic_total_bytes

MetaXLink

GPU驱动加载后MetaXLink传输总数据量，单位：Byte

mx_xcore_dpm_level

DPM

当前DPM等级

mx_gpu_state

Status

设备状态信息，0为不可用，1为可用。当为0时，标签 unavailable_reason 中会给出GPU不可用的原因

mx_clk_thr

Clocks Throttle Reasons

当前降频原因，0为无降频；降频原因二进制显示，从右往左依次占位为Idle，Application Limit，Over Power，Chip过温，VR过温，HBM过温，Thermal过温，PCC，Power Brake，DIDT，Low Usage，Other

例如，若数值为12，表示当前降频原因为Over Power及Chip过温

mx_sgpu_compute_quota

SGPU Usage

当前分给该子设备的算力占父设备的百分比，单位%（仅当启用sGPU且 default-counters.csv 中配置启用该指标时，默认禁用此指标）

mx_sgpu_usage

SGPU Usage

当前该子设备的使用率，单位%（仅当启用sGPU且 default-counters.csv 中配置启用该指标时，默认禁用此指标）

mx_sgpu_total_memory

SGPU Usage

当前分给该子设备的VRAM总量，单位KB（仅当启用sGPU且 default-counters.csv 中配置启用该指标时，默认禁用此指标）

mx_sgpu_used_memory

SGPU Usage

当前该子设备已使用的VRAM大小，单位KB（仅当启用sGPU且 default-counters.csv 中配置启用该指标时，默认禁用此指标）

mx_sgpu_free_memory

SGPU Usage

当前该子设备还可以使用的VRAM余量，单位KB（仅当启用sGPU且 default-counters.csv 中配置启用该指标时，默认禁用此指标）

mx_server_info

UUID

服务器UUID，local代表本机信息，remote代表与之互联的服务器信息

mx_topo_info

Topo

设备拓扑信息

mx_driver_log_errors

Log

驱动每个周期每个模块日志级别为ERROR 及 ALERT 的信息条数

mx_driver_eid_errors

Log

最新驱动EID错误信息，指标数值换算成十六进制后为EID错误码。错误详细信息，影响及常见处理方法参见《曦云® 系列通用GPU EID手册》

mx_sdk_eid_errors

Log

最新SDK EID错误信息及当前MXMACA SDK版本信息，指标数值换算成十六进制后为EID错误码。错误详细信息，影响及常见处理方法参见《曦云® 系列通用GPU EID手册》

mx_ecc_error_count

Error

ECC 错误的计数，包括sram和dram的CE（Correctable Errors）和UE（Uncorrectable Errors）个数，以及坏页数

mx_pci_event

Error

显示错误事件名称、PCIe错误类型及计数。类型分为 aer_ue （PCIe AER Uncorrectable Error）、aer_ce （PCIe AER Correctable Error）、synfld （PCIe Syncflood Error）、dbe （PCI device base error）以及 mmio （PCI mmio invalid）。标签 event_name 显示具体的事件信息

mx_ras_count

Error

显示IP RAS错误统计，若存在则会按照Corrected Error和Uncorrected Error分类统计数目

mx_ras_status

Status

显示IP RAS状态寄存器的值，获取到的数值为十进制，需转换成十六进制

4.2. 标签说明

表 4.2 标签说明

名称

含义

Hostname

主机名称

若mx-exporter在容器中运行，则为容器名称，若需修改，可在启动容器时传入 --hostname <hostname>

bios_version

BIOS版本信息

deviceId

设备号，即板卡号（通过 mx-smi -L 获取板卡ID信息）

driver_version

设备驱动版本信息

exported_container

正在使用设备的k8s pod的container名称，仅当启动mx-exporter容器传入 -v /var/lib/kubelet/pod-resources:/var/lib/kubelet/pod-resources 时可显示

exported_namespace

正在使用设备的pod所在k8s命名空间，仅当启动mx-exporter容器传入 -v /var/lib/kubelet/pod-resources:/var/lib/kubelet/pod-resources 时可显示

exported_pod

正在使用设备的k8s pod名称，仅当启动mx-exporter容器传入 -v /var/lib/kubelet/pod-resources:/var/lib/kubelet/pod-resources 时可显示

modelName

芯片型号

uuid

设备UUID

major/minor

子设备唯一编号（仅当启用sGPU且 default-counters.csv 中配置启用该指标时，默认禁用此指标）

sgpuID

在当前父设备下创建的子设备ID，范围为[0-15]（仅当启用sGPU且 default-counters.csv 中配置启用该指标时，默认禁用此指标）

4.3. 指标输出示例

使用以下命令启动mx-exporter：

docker run -d --device=/dev/dri -v /var/log:/host/var/log -v /var/lib/kubelet/pod-resources:/var/lib/kubelet/pod-resources --name=mx-exporter -p 0.0.0.0:8000:8000 <image ID> -mp=/host

执行 curl http://<host_ip>:8000/metrics 可得到以下内容：

# HELP mx_device_type Device type
# TYPE mx_device_type gauge
mx_device_type{deviceId="0",deviceType="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 1.0
# HELP mx_bios_ver Bios version
# TYPE mx_bios_ver gauge
mx_bios_ver{bios="1.7.4.0",deviceId="0"} 1.0
# HELP mx_driver_ver Driver version
# TYPE mx_driver_ver gauge
mx_driver_ver{deviceId="0",driver="2.3.0"} 1.0
# HELP mx_chip_hotspot_temp Chip hotspot temperature
# TYPE mx_chip_hotspot_temp gauge
mx_chip_hotspot_temp{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 36.25
# HELP mx_board_core_temp Board DrMOS Core temperature
# TYPE mx_board_core_temp gauge
mx_board_core_temp{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 35.0
# HELP mx_vpue_usage Vpue usage in percent
# TYPE mx_vpue_usage gauge
mx_vpue_usage{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
# HELP mx_vpud_usage Vpud usage in percent
# TYPE mx_vpud_usage gauge
mx_vpud_usage{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
# HELP mx_memory_usage HBM and system memory usage in percent
# TYPE mx_memory_usage gauge
mx_memory_usage{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="vram",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 1.2849688529968262
mx_memory_usage{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="xtt",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.001035634935770525
# HELP mx_memory_total Total HBM and system memory in KB
# TYPE mx_memory_total gauge
mx_memory_total{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="vram",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 6.7108864e+07
mx_memory_total{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="xtt",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 1.585114545e+09
# HELP mx_memory_used HBM and system memory used in KB
# TYPE mx_memory_used gauge
mx_memory_used{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="vram",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 862328.0
mx_memory_used{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="xtt",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 16416.0
# HELP mx_gpu_usage GPU uage in percent
# TYPE mx_gpu_usage gauge
mx_gpu_usage{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
# HELP mx_board_power Board power in milliwatt
# TYPE mx_board_power gauge
mx_board_power{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 55983.0
# HELP mx_vpue_clock Encoder clock frequency in MHz
# TYPE mx_vpue_clock gauge
mx_vpue_clock{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 535.0
# HELP mx_vpud_clock Decoder clock frequency in MHz
# TYPE mx_vpud_clock gauge
mx_vpud_clock{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 525.0
# HELP mx_mem_clock Memory clock frequency in MHz
# TYPE mx_mem_clock gauge
mx_mem_clock{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 1800.0
# HELP mx_gpu_clock GPU clock frequency in MHz
# TYPE mx_gpu_clock gauge
mx_gpu_clock{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 428.0
# HELP mx_pcie_bw Pcie Tx and Rx throughput in MB/s
# TYPE mx_pcie_bw gauge
mx_pcie_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_pcie_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
# HELP mx_pcie_speed Pcie current speed in GT/s
# TYPE mx_pcie_speed gauge
mx_pcie_speed{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 32.0
# HELP mx_pcie_width Pcie current lanes
# TYPE mx_pcie_width gauge
mx_pcie_width{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 16.0
# HELP mx_pcie_bridge_speed Pcie bridge current speed in GT/s
# TYPE mx_pcie_bridge_speed gauge
mx_pcie_bridge_speed{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 32.0
# HELP mx_pcie_bridge_width Pcie bridge current lanes
# TYPE mx_pcie_bridge_width gauge
mx_pcie_bridge_width{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 16.0
# HELP mx_mxlk_bw MetaXLink Tx and Rx throughput in MB/s
# TYPE mx_mxlk_bw gauge
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="1",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="2",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="3",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="4",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="5",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="6",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="7",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="1",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="2",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="3",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="4",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="5",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="6",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_bw{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="7",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
# HELP mx_mxlk_speed MetaXLink current link speed in GT/s
# TYPE mx_mxlk_speed gauge
mx_mxlk_speed{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="1",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_speed{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="2",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_speed{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="3",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_speed{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="4",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 32.0
mx_mxlk_speed{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="5",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 32.0
mx_mxlk_speed{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="6",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 32.0
mx_mxlk_speed{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="7",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
# HELP mx_mxlk_width MetaXLink current link width
# TYPE mx_mxlk_width gauge
mx_mxlk_width{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="1",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_width{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="2",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_width{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="3",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_width{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="4",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 16.0
mx_mxlk_width{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="5",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 16.0
mx_mxlk_width{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="6",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 16.0
mx_mxlk_width{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="7",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
# HELP mx_mxlk_traffic_total_bytes  MetaXLink traffic total in Bytes
# TYPE mx_mxlk_traffic_total_bytes gauge
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="1",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="2",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="3",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="4",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 2.1474920704e+010
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="5",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 2.147492064e+010
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="6",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 2.1474920684e+010
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="7",type="rx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="1",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="2",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="3",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="4",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 2.1474930688e+010
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="5",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 2.1474932352e+010
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="6",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 2.1474930688e+010
mx_mxlk_traffic_total_bytes{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="7",type="tx",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
# HELP mx_mxlk_aer_count MetaXLink aer count
# TYPE mx_mxlk_aer_count gauge
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="1",type="ce",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="2",type="ce",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="3",type="ce",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="4",type="ce",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="5",type="ce",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="6",type="ce",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="7",type="ce",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="1",type="ue",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="2",type="ue",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="3",type="ue",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="4",type="ue",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="5",type="ue",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="6",type="ue",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_mxlk_aer_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",mxlkId="7",type="ue",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
# HELP mx_xcore_dpm_level Dpm xcore performance level
# TYPE mx_xcore_dpm_level gauge
mx_xcore_dpm_level{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
# HELP mx_gpu_state GPU state: 0(not available) 1(available)
# TYPE mx_gpu_state gauge
mx_gpu_state{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 1.0
# HELP mx_clk_thr Current gpu clock throttling reason
# TYPE mx_clk_thr  gauge
mx_clk_thr{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
# HELP mx_ecc_error_count Total ECC error count
# TYPE mx_ecc_error_count gauge
mx_ecc_error_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="sram_ce",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_ecc_error_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="sram_ue",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_ecc_error_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="dram_ce",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_ecc_error_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="dram_ue",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_ecc_error_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="retired_page",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
# HELP mx_server_info Local server and its connected remote servers uuid info
# TYPE mx_server_info gauge
mx_server_info{Hostname="mxsrv003",kind="local",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 1.0
mx_server_info{Hostname="mxsrv003",kind="remote",uuid="GPU-1d54ca5a-b3d9-65df-0ffc-8ebe0d6347e7"} 1.0
# HELP mx_driver_log_errors Driver kernel log errors
# TYPE mx_driver_log_errors gauge
mx_driver_log_errors{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",error_level="ERROR",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",module="ATU",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 4.0
# HELP mx_driver_eid_errors Value of the latest driver EID error encountered
# TYPE mx_driver_eid_errors gauge
mx_driver_eid_errors{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",eid_info="shader exception, pasid 32769 error_type mem_viol(0x4)",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 8450.0
mx_driver_eid_errors{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",eid_info="atu 0x0 pde_base_addr 0x1000340000",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 8449.0
# HELP mx_sdk_eid_errors Value of the latest SDK EID error encountered
# TYPE mx_sdk_eid_errors gauge
mx_sdk_eid_errors{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",eid_info="Xnack Error/ATU Fault(0x8), check app kernel, _Z10trapKernelPfS_i",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",processId="415256",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 12548.0
# HELP mx_pci_event Driver pci event includes aer_ue/aer_ce/synfld/dbe/mmio
# TYPE mx_pci_event gauge
mx_pci_event{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",event_name="MC3 MCA FATAL",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",type="synfld",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 1.0
# HELP mx_ras_count Display value in the ras error counter registers
# TYPE mx_ras_count gauge
mx_ras_count{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="MCCTL0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 255.0
# HELP mx_ras_status Display data in status registers
# TYPE mx_ras_status gauge
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="PCIE reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 53248.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="MCCTL0 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 1.646208e+07
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="MCCTL1 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 1.4626816e+07
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="MCCTL2 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 8.33536e+06
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="MCCTL3 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 1.6723968e+07
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="SMP0 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 0.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="SMP1 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 4096.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="FUSE reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 8192.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="INT reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 36864.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="INT reg1",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="CCX0 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 49152.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="CCX0 reg1",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="CCX1 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 49152.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="CCX1 reg1",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="CCX2 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 49152.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="CCX2 reg1",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DHUB1 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 57344.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DHUB2 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 57344.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DHUB3 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 57344.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DHUB4 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 57344.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DHUB5 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 57344.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DHUB6 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 57344.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DHUB7 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 57344.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="VPUE0 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="VPUD0 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="VPUD1 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="VPUD2 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="VPUD3 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="VPUD4 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="VPUD5 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="VPUD6 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="VPUE0 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="HAG reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DMA0 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 36864.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DMA0 reg1",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DMA1 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 36864.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DMA1 reg1",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DMA2 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 36864.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DMA2 reg1",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DMA3 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 36864.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DMA3 reg1",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DMA4 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 36864.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="DMA4 reg1",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="ATH reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="ATUL20 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="ATUL20 reg1",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="ATUL21 reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="ATUL21 reg1",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 40960.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="XSC reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 28672.0
mx_ras_status{Hostname="mxsrv003",bios_version="1.7.4.0",deviceId="0",driver_version="2.3.0",exported_container="",exported_namespace="",exported_pod="",modelName="MXC500",register_name="CE reg0",uuid="GPU-1d34ca5a-b3d9-65df-0ffc-8ebe0d6347d9"} 28672.0