A standard host dashboard reports a quiet machine while the most expensive component in it does nothing, because GPU telemetry comes from somewhere else entirely. The useful metrics are the ones that separate a card doing work from a card waiting on data or quietly throttling. This covers collecting them, and the one alert that pays for the whole exercise.

Host monitoring tells you nothing about a GPU. CPU is idle, memory looks fine, load average is near zero, and the expensive part of the machine may be sitting at 4% utilisation or thermally throttling. GPU metrics come from a separate source and answer a different question.

The Question Worth Asking

For a GPU the useful question is not "is it up" but "is it earning its cost". A card at $2.26 an hour running at 20% utilisation is the most expensive idle resource in the estate, and nothing in a standard dashboard shows it.

So the metrics that matter are the ones that distinguish four states: genuinely busy, waiting on data, memory-constrained, and throttling. Each has a different fix, and telling them apart is the entire value of GPU monitoring.

MetricWhat it tells you
SM utilisationWhether the compute units are doing work at all
Memory used vs totalHow close you are to an out-of-memory failure
Memory bandwidth utilisationWhether you are compute-bound or memory-bound
Temperature and power drawWhether the card can sustain its clocks
Clock throttle reasonsWhy clocks dropped: thermal, power, or a cap
PCIe throughputWhether the data pipeline is the bottleneck
ECC error countsEarly warning of a failing card

Utilisation alone is the metric people watch and the most misleading one. It reports whether any kernel was resident, not whether the card was busy, so a job feeding the GPU badly can show high utilisation and low throughput.

nvidia-smi Is for Looking, Not Recording

nvidia-smi
nvidia-smi dmon -s pucvmet
nvidia-smi --query-gpu=utilization.gpu,memory.used,temperature.gpu,clocks_throttle_reasons.active \
  --format=csv -l 5
nvidia-smi -q -d PERFORMANCE | grep -A8 Clocks_Throttle

These are the right tools for a live look at one machine and the wrong basis for a monitoring system. There is no history, so a job that failed overnight leaves no evidence, and parsing the output into a metrics store is a script you will maintain forever.

The throttle query is worth knowing regardless. A card reporting SW Power Cap or HW Thermal Slowdown as active is delivering less than you paid for, and no utilisation graph reveals it.

DCGM Exporter Is the Real Answer

NVIDIA's Data Center GPU Manager exposes the full metric set, including the ones nvidia-smi summarises away, and its exporter presents them in Prometheus format on a scrape endpoint.

docker run -d --gpus all --rm -p 9400:9400 \
  --cap-add SYS_ADMIN \
  nvcr.io/nvidia/k8s/dcgm-exporter:latest
curl -s localhost:9400/metrics | grep -E '^DCGM_FI_DEV_(GPU_UTIL|FB_USED|GPU_TEMP)'

Then add it as one more scrape target. This is the part where existing infrastructure does the work, so do not stand up a second monitoring stack for it. Our walkthrough for running Prometheus and Grafana on Ubuntu covers the stack itself, the storage retention and the alerting channels; the GPU side is one job block:

scrape_configs:
  - job_name: dcgm
    static_configs:
      - targets: ['gpu-01:9400','gpu-02:9400']

DCGM also runs health checks that nvidia-smi does not surface, and on multi-GPU nodes it reports NVLink counters, which is how you find a training job whose inter-GPU transfers are the bottleneck rather than its compute.

The Alerts Worth Having

Most GPU dashboards are built and then never looked at, which means the value has to come from alerts. Four are worth the noise.

groups:
  - name: gpu
    rules:
      - alert: GpuMemoryNearLimit
        expr: DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) > 0.95
        for: 5m
      - alert: GpuThermalThrottle
        expr: DCGM_FI_DEV_GPU_TEMP > 85
        for: 10m
      - alert: GpuIdleButAllocated
        expr: avg_over_time(DCGM_FI_DEV_GPU_UTIL[2h]) < 10
        for: 2h
      - alert: GpuEccErrors
        expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[1h]) > 0

The third one is the alert nobody writes and the one that pays for the exercise. A card allocated to a project and averaging under 10% for two hours is money leaving the building, and it is invisible to every other alert you have.

The fourth is a hardware warning. Double-bit ECC errors are not a transient; a card producing them should be drained and replaced rather than watched.

Reading the Dashboard

Four patterns cover almost everything you will see, and each has one obvious response.

High utilisation, high memory, stable clocks. Working as intended. Leave it alone.

Low utilisation, high PCIe throughput. The GPU is waiting on data. The fix is in the data pipeline: more loader workers, prefetching, or faster storage. Adding GPUs here makes the problem worse and more expensive.

Sawtooth utilisation with gaps. Batches too small, or synchronisation between steps. Raise batch size or accumulate gradients before increasing capacity.

High utilisation, clocks below base, throttle reasons active. Thermal or power limited. This is a facility and chassis question, not a software one, and it caps everything above it.

In Kubernetes

On a cluster, the device plugin advertises GPUs as a schedulable resource and DCGM exporter runs as a DaemonSet, which gives you per-node metrics labelled with pod and namespace. That labelling is the point: it turns "GPU 2 is idle" into "this team's job is idle", which is the form an owner can act on.

Note that a GPU request in Kubernetes is integral and exclusive by default, so a pod requesting one GPU holds the whole card whether or not it uses it. That is exactly the condition the idle alert above catches. Our guide to Kubernetes pod troubleshooting covers the gap between what a pod requests and what it uses, which is the same gap in a more expensive form here.

The Infrastructure Underneath

Monitoring reveals whether a card is delivering what it should. Sustained clocks under load are a function of cooling and power, which is a facility property rather than a configuration one.

MassiveGRID's GPU instances run in Tier-4 facilities on HA clusters with dedicated bare-metal GPU passthrough and full CUDA core access, so a card is not shared and its thermal envelope is not somebody else's variable. An A100 40GB with 16 vCPUs and 120 GB of RAM is $2.26 per hour or $1,649 a month, an A100 80GB is $3.42 or $2,499, and an H100 80GB with 480 GB of DDR5 is $5.48 or $3,999, with CUDA drivers and ML frameworks pre-installed. Where you would rather somebody else watched the dashboards, NOC services provide the monitoring and response.

Instances can be ordered across a partner footprint of more than 700 datacenters in 85 metros, 30 countries and six continents, with auto-provisioning in New York, London, Frankfurt and Singapore.

Further Reading