Monitoring a Proxmox Cluster with Prometheus and Grafana

The conditions that take a cluster down are quorum instability, storage sitting degraded, a guest that failover could not restart, and a pool approaching its limit. A standard node dashboard shows none of them, because they are reported by three sources nobody installs by default. This covers collecting each and the alert thresholds that survive a maintenance window without being muted.

Generic host monitoring on a Proxmox cluster tells you the nodes are alive, which was never the question. The failures that matter are quorum wobbling, Ceph sitting degraded, a guest that HA could not restart, and a pool filling. None of those appear in CPU and memory graphs.

Four Layers, Four Sources

A cluster produces metrics at four levels and they come from different places, which is why a single exporter never covers it.

Most clusters have the first layer and nothing else, because node_exporter is what everybody installs. The three that produce actionable alerts are the other three.

This post covers the Proxmox-specific collection and alerting. For the Prometheus and Grafana stack itself, retention, dashboards and notification channels, our walkthrough for Prometheus and Grafana on Ubuntu covers it, and none of that changes here.

pve-exporter for the Cluster Layer

Proxmox has no Prometheus endpoint of its own. prometheus-pve-exporter queries the API and translates, and it runs anywhere with API access rather than on the nodes, which is the arrangement to prefer: an exporter installed on a node cannot report that the node is gone.

pveum user add prometheus@pve
pveum role add Monitoring -privs "VM.Audit Datastore.Audit Sys.Audit Sys.Modify"
pveum aclmod / -user prometheus@pve -role Monitoring
pveum user token add prometheus@pve exporter --privsep 0

# /etc/prometheus/pve.yml
default:
  user: prometheus@pve
  token_name: exporter
  token_value: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  verify_ssl: true

scrape_configs:
  - job_name: pve
    metrics_path: /pve
    params:
      cluster: ['1']
      node: ['1']
    static_configs:
      - targets: ['pve-01.example.com:8006','pve-02.example.com:8006']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: exporter-host:9221

That relabelling is the part that looks odd and is doing something specific: the target in your config is the Proxmox node, but the scrape goes to the exporter, which is told which node to query via the target parameter. Listing two nodes rather than one means cluster metrics keep arriving when one is down.

Give the token an audit role rather than administrator. Sys.Modify appears in the list above only because some cluster status endpoints require it; drop it if your exporter version does not need it.

Ceph Exports Itself

ceph mgr module enable prometheus
ceph config set mgr mgr/prometheus/server_port 9283
curl -s http://pve-01:9283/metrics | grep -E '^ceph_health_status|^ceph_osd_up'

One caveat that catches people: the endpoint is served by whichever manager is active, and that moves. Scrape all manager candidates rather than one address, and accept that the standbys return nothing, or put them behind a target list Prometheus can fail over across.

The Alerts That Earn Their Place

groups:
  - name: proxmox
    rules:
      - alert: ClusterQuorumLost
        expr: pve_quorate != 1
        for: 1m
      - alert: NodeOffline
        expr: pve_node_info{} * on(id) group_left pve_up == 0
        for: 2m
      - alert: CephNotHealthy
        expr: ceph_health_status != 0
        for: 15m
      - alert: CephOsdDown
        expr: ceph_osd_up == 0
        for: 5m
      - alert: CephPoolNearFull
        expr: ceph_pool_bytes_used / ceph_pool_max_avail > 0.75
        for: 30m
      - alert: GuestStoppedUnexpectedly
        expr: pve_up{id=~"qemu/.*"} == 0
        for: 5m
      - alert: NoRoomForNodeFailure
        expr: sum(pve_memory_usage_bytes{id=~"node/.*"})
              / sum(pve_memory_size_bytes{id=~"node/.*"}) > 0.66
        for: 1h

Three of those deserve a note.

The Ceph health alert waits fifteen minutes deliberately. Ceph goes to HEALTH_WARN during any normal rebalance, including one you caused by rebooting a node, so a tighter window produces alerts during every maintenance window and gets muted.

The pool alert fires at 75%, not 85%. Ceph's own warning is near 85% and writes stop at the full ratio. By the time Ceph complains, adding capacity is urgent; at 75% it is a purchase order. Our guide to cluster sizing covers why the practical ceiling is lower than the arithmetic.

The last alert is the one nobody writes. It says the cluster no longer has room to absorb a node failure, which is a condition that arrives gradually as guests are added and is invisible until the failure happens. On a three-node cluster the threshold is 66%; adjust it to your node count.

What to Monitor Inside Guests

pve-exporter reports what the hypervisor sees, which is not what the guest experiences. Two specific gaps.

Memory as reported by the hypervisor is what the guest was allocated and has touched, not what is free inside it. A guest at 95% by that measure may be fine, with the difference being page cache. Only an agent inside the guest knows.

And disk usage of a thin-provisioned volume is invisible from outside. A guest whose filesystem is full while its virtual disk has space allocated looks healthy to the hypervisor and is down to its users.

So run node_exporter inside guests you care about, and treat hypervisor metrics as the capacity view rather than the health view. The distinction is worth making explicit in your dashboards, because the two numbers disagree and somebody will eventually chase the discrepancy.

The Cheaper Option That Is Often Enough

Proxmox can push metrics to InfluxDB or Graphite natively, configured in the web interface with no exporter to maintain. Datacenter, then Metric Server.

If you already run InfluxDB, this is a five-minute job that covers the node and guest layers well. It does not give you Ceph detail or Prometheus-style alerting rules, so a cluster with Ceph is better served by the exporter route. A cluster without Ceph, monitored by one person, is not.

Choose based on what you already operate. A second monitoring stack stood up alongside an existing one is worse than a slightly less detailed view in the one people already look at.

Or Have Somebody Watch It

The hard part of cluster monitoring is not collection, it is that quorum loss at three in the morning needs a response within minutes and an alert with nobody reading it is a log entry.

MassiveGRID runs this monitoring on its own platform, where Proxmox high-availability clustering with automatic failover sits over Ceph storage replicating every block three times across independent NVMe drives, behind a 100% uptime SLA. NOC services provide the staffed monitoring and response for clusters you own, and Proxmox support from $99 per node per month up to $449 and custom tiers covers the alerting design, including the thresholds above tuned to your node count.

Infrastructure can be ordered across a partner footprint of more than 700 datacenters in 85 metros, 30 countries and six continents, with auto-provisioning in New York, London, Frankfurt and Singapore.