Proxmox Ceph Setup: Hyper-Converged Storage Step by Step

Ceph is what gives a Proxmox cluster shared storage without an external array, and it is the component most often built wrong, because it rewards a hardware layout that contradicts long-standing storage habit. No RAID controller, no write cache, and a network budget that looks excessive right up until the first rebuild. This guide covers the hardware, the network separation, the pool settings that decide durability, and how to break the cluster deliberately before it breaks itself.

The reason to accept that unfamiliarity is what Ceph removes. Without it, virtual machine disks sit on local storage, which rules out both live migration and HA restart: the surviving nodes cannot read a disk that exists only on the node that just failed. Ceph makes every disk readable from every node, and it does so without a SAN, a filer or a second vendor in the room.

MassiveGRID runs Ceph in production. Every block written to a MassiveGRID server is replicated three times across independent NVMe drives, on the same Proxmox clusters that carry the rest of the platform.

HA cloud storage · Our technology
Proxmox support — Ceph and storage management included from $249/node/month

What Ceph Is Doing

Four components matter for a Proxmox deployment. An OSD is one daemon per physical disk, and it owns that disk entirely. Monitors maintain the cluster map and must form their own quorum, so you run three or five, never two or four. Managers serve metrics and the dashboard, with one active and the rest standby. Pools group the data and carry the replication policy.

Placement groups sit between objects and disks. Every object hashes to a placement group, and each placement group maps to a set of OSDs according to the CRUSH map. That indirection is why adding a disk rebalances a fraction of the data rather than reshuffling everything.

Hardware That Suits Ceph

Three requirements, and each one is regularly ignored with predictable results.

Disks passed through directly. Ceph must see raw block devices. If you have a RAID controller, put it in HBA or JBOD mode so each disk appears individually. Building OSDs on RAID volumes stacks replication on top of parity, multiplying writes, and it hides disk failures from the layer whose entire job is handling them. Ceph recovers from a failed disk in minutes; a RAID controller that quietly masks a failing disk denies it the chance.

Enterprise SSDs with power-loss protection. This is not preference. Ceph issues frequent flushes to guarantee durability. A consumer SSD without a protected write buffer honours each flush by writing to flash, and sustained throughput collapses to a small fraction of the drive's rated figure. The symptom is a cluster that benchmarks fine for a moment and then crawls. Check the specification for power-loss protection before buying, because the difference is one or two orders of magnitude in write latency.

Network with real headroom. Every client write becomes three writes plus coordination, and a failed disk triggers a rebuild that saturates whatever you gave it. 10 GbE is the practical floor; 25 GbE or better is right for NVMe, which will otherwise sit idle waiting on the network.

Budget RAM generously. The default target is around 4 GB per OSD, and recovery is when memory pressure peaks, which is exactly when you cannot afford an OSD to be killed by the kernel.

Separate the Networks

Ceph uses two networks and Proxmox needs a third. Collapsing them is the second most common design error after the RAID controller.

The Ceph public network carries client traffic between VMs and OSDs. The Ceph cluster network carries replication and recovery between OSDs, which during a rebuild means saturation. Corosync needs its own low-jitter link, because Ceph recovery traffic on a shared interface will delay corosync messages and fence a healthy node.

A workable three-node layout: corosync on its own 1 GbE pair, Ceph public and cluster on 25 GbE, and VM traffic separate again. For exactly three nodes, a full-mesh configuration connects each node directly to the others and avoids buying a 25 GbE switch entirely.

Installing Ceph

Proxmox wraps deployment in pveceph. Run the install on every node:

pveceph install

Initialise the cluster from the first node, naming both networks explicitly:

pveceph init --network 10.30.0.0/24 --cluster-network 10.40.0.0/24

Create a monitor on three nodes and a manager on at least two:

# on each of three nodes
pveceph mon create

# on two nodes
pveceph mgr create

ceph -s

Then create one OSD per disk. Confirm the device is genuinely empty first, because this is destructive and the error is not recoverable:

lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,MODEL
pveceph osd create /dev/nvme1n1
pveceph osd create /dev/nvme2n1
ceph osd tree

Wait for HEALTH_OK before continuing. A cluster that never reaches it has a problem you want to understand now rather than after it holds data.

Pools, Replication and the min_size Trap

Create a pool for VM disks:

pveceph pool create vm-storage --size 3 --min_size 2 --pg_autoscale_mode on

Those two numbers decide the durability of everything you store, and the second one is where people get into trouble.

size 3 keeps three copies of every object on three different hosts. min_size 2 means writes continue while at least two copies are available, so you survive one host failing with no interruption.

Setting min_size 1 looks like extra resilience and is the single most dangerous configuration in Ceph. It permits writes when only one copy exists. Lose that disk before replication catches up and the data is gone, with no second copy to recover from. Leave it at 2.

Similarly, size 2 with min_size 2 blocks all writes when any host fails, and size 2 with min_size 1 risks the scenario above. Three copies is the configuration that is both available and safe, and it is what MassiveGRID runs.

Placement group autoscaling is on by default in current releases and should stay on. Manual tuning was necessary years ago and is now a way to introduce problems.

Attaching It to Proxmox

Add the pool as RBD storage for VM disks and containers:

pvesm add rbd ceph-vm --pool vm-storage --content images,rootdir
pvesm status

For shared files, ISO images, snippets and backups, add CephFS, which needs a metadata server:

pveceph mds create
pveceph fs create --name cephfs --pg_num 32

Once RBD is attached, live migration works between any nodes in the cluster and HA can restart a VM anywhere, because every node reads the same storage.

Verifying It Before You Trust It

Benchmark, then break it deliberately. Both matter, and the second matters more.

rados bench -p vm-storage 60 write --no-cleanup
rados bench -p vm-storage 60 rand
rados -p vm-storage cleanup

Then pull a disk, or mark an OSD out, and watch what happens:

ceph osd out osd.3
watch -n 2 ceph -s

Two things are worth observing. Whether client I/O stays usable while recovery runs, which tells you if the cluster network is adequate. And how long the rebuild takes, which is your exposure window: until it completes, one more failure in the wrong place means data loss. A cluster that takes eight hours to rebuild has an eight-hour window, and that is the number to design around.

Capacity Planning People Get Wrong

Three-way replication means usable capacity is raw divided by three. Nine 4 TB drives is 36 TB raw and 12 TB usable, and that is before headroom.

Ceph also needs free space to rebuild into. Plan to run below 70 percent, because at 85 percent Ceph raises a nearfull warning and at 95 percent it stops accepting writes entirely to protect itself. Filling a Ceph cluster is a serious incident, not a capacity notice, and recovering from it requires adding hardware under pressure.

So the realistic figure for nine 4 TB drives is closer to 8 TB of workable capacity. Size for that number, not the raw one.

Storage You Do Not Have to Commission

Ceph is excellent once correctly built and unforgiving of the shortcuts that look reasonable at purchase time. Consumer SSDs, a RAID controller left in RAID mode, a shared network, min_size 1: each of those is a cluster that works in testing and disappoints later.

MassiveGRID has operated Ceph in production for years, with three-way replication across independent enterprise NVMe drives on every cluster, in any of the 85+ metros we deploy into. That replication is included rather than an upgrade tier, and it sits under the same Proxmox HA clusters that provide automatic failover, so a disk failure is invisible and a node failure is a brief restart.

For clusters you run yourself, Proxmox support at $249 per node per month includes Ceph and storage management with a four-hour critical response SLA, and Proxmox Backup Server setup with it. The compute side of the same design is in our Proxmox HA cluster guide, and the mechanics of replication are covered in how triple replication protects your data.