Kubernetes Backup with Velero: Cluster and Volume Recovery

Two things get called a cluster backup and only one of them contains your data. Plans built on the wrong one restore a complete set of manifests attached to empty volumes, which is discovered at the worst possible time. This separates the two mechanisms, covers the flags that decide whether volumes are captured at all, and ends with a drill that costs nothing.

Backing up a cluster means two unrelated things, and conflating them is why so many Kubernetes recovery plans do not work. One is the API objects: deployments, services, config, secrets. The other is the data inside persistent volumes. Different mechanisms, different failure modes, and a plan covering only the first restores an empty application.

What etcd Snapshots Do and Do Not Cover

An etcd snapshot is a copy of the entire cluster state at one instant. It is the right tool for exactly one scenario: the control plane is gone and you are rebuilding the same cluster.

It is the wrong tool for everything else. You cannot restore one namespace from it, you cannot restore into a different cluster with a different certificate authority, and it contains no volume data at all. It is also all-or-nothing, so recovering an accidentally deleted deployment means reverting the whole cluster to an earlier state, undoing everything else that happened since.

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-$(date +%F).db

Take these anyway, on a schedule, and store them off the cluster. They are cheap and they are the only route back from a lost control plane. Our walkthrough for a highly available Kubernetes cluster covers keeping the control plane quorate so you need them less often.

What Velero Adds

Velero works at the API level. It queries the API server for objects matching a selector, writes them to object storage, and can restore them selectively into the same cluster or a different one.

That selectivity is the whole point. Four things become possible that etcd snapshots cannot do: restoring one namespace, restoring into a rebuilt or differently sized cluster, migrating workloads between clusters, and restoring a single object somebody deleted.

Installing It

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.x \
  --bucket k8s-backups \
  --secret-file ./credentials-velero \
  --backup-location-config region=eu-central-1,s3ForcePathStyle=true,s3Url=https://s3.example.com \
  --use-node-agent \
  --default-volumes-to-fs-backup

The AWS plugin speaks S3, so it works against any S3-compatible endpoint, which is what s3Url and s3ForcePathStyle are for. You do not need AWS to use it.

The last two flags are the ones worth understanding. --use-node-agent deploys a DaemonSet that reads volume contents directly, and --default-volumes-to-fs-backup makes that the default rather than opt-in per volume. Without them, Velero backs up your YAML and silently skips your data, which is the single most common way a Velero deployment turns out to be useless.

Two Ways to Capture Volumes

CSI volume snapshots. Velero asks the storage layer to snapshot the volume. Fast, efficient, and the snapshot usually lives in the same storage system, which means it does not protect against that system failing. Requires a CSI driver with snapshot support and a configured VolumeSnapshotClass.

File-system backup. The node agent reads the files and writes them to object storage. Slower and storage-agnostic, and the copy genuinely leaves the cluster. This is the one to default to, with CSI snapshots added where the storage supports them and you want the speed.

Both share one serious limitation: a running database's files captured mid-write produce a crash-consistent copy, which recovers by replaying a log and may not land where you expect. Hooks are the answer.

annotations:
  pre.hook.backup.velero.io/container: postgres
  pre.hook.backup.velero.io/command: '["/bin/sh","-c","pg_dump -U app app > /data/dump.sql"]'
  post.hook.backup.velero.io/command: '["/bin/sh","-c","rm -f /data/dump.sql"]'

An application-aware dump written into the volume before the backup runs is worth more than any amount of snapshot cleverness. Our guide to database backup strategy covers why, and what consistency actually requires per engine.

Schedules, and What to Exclude

velero schedule create daily-all \
  --schedule="0 2 * * *" --ttl 720h \
  --exclude-namespaces kube-system,velero

velero backup create pre-upgrade --include-namespaces prod --wait
velero backup describe pre-upgrade --details
velero backup logs pre-upgrade

--ttl is retention, and Velero deletes expired backups including their object storage contents, so it is the retention mechanism rather than a hint.

Exclude what you should not restore. Backing up kube-system and restoring it into a cluster that has its own version of those objects causes conflicts rather than recovery. Velero's own namespace is likewise a bad thing to restore over a running Velero.

What you must include and people forget: CustomResourceDefinitions, and any cluster-scoped RBAC your applications depend on. A namespace restored without its CRDs produces objects the API server rejects.

The Restore That Tests It

velero restore create --from-backup daily-all-20260909 \
  --include-namespaces prod --namespace-mappings prod:prod-restore
velero restore describe prod-restore-xyz --details
kubectl -n prod-restore get pods,pvc

Namespace mapping is the feature that makes drills cheap: restore alongside the live namespace rather than over it, verify, then delete. No maintenance window, no risk to production, and a real answer to how long a restore takes.

Three things a restore drill reliably finds. Secrets that were never in the backup because they are created by an operator rather than stored. Services whose cluster IPs cannot be recreated, which is fine but changes DNS. And init containers that fail because they expect state the volume backup did not include. Our guide to running recovery drills covers structuring these so they surface problems rather than confirming hopes.

What Belongs in Git Instead

A large share of what Velero backs up should not need backing up. If deployments, services and config are declared in Git and reconciled by a controller, recovering them is pointing the controller at a new cluster.

That narrows the backup problem to what Git cannot hold: persistent volume data, and secrets if they are not managed declaratively. Which is a much smaller and more tractable problem, and it is the argument for GitOps having a disaster recovery benefit rather than only a workflow one. Our guide to GitOps with Argo CD covers that side.

Keep Velero regardless. Git holds intent; it does not hold the state an operator generated, and it does not restore a namespace somebody deleted at 4pm on a Friday.

The Storage Both Halves Need

Velero needs object storage that is not the cluster, and volume snapshots need a CSI driver whose storage survives a node.

MassiveGRID's managed Kubernetes starts from $0.03474 per hour, about $25.37 a month, with resources billed in cloudlets of 128 MiB RAM and 400 MHz CPU, over Ceph storage that replicates every block three times across independent NVMe drives, with Proxmox high-availability clustering restarting a lost node's workloads automatically. Backup services provide the off-cluster destination with block-level incremental backups and AES-256 encryption at $0.01 per GB, on RAID10 in Tier-3 and Tier-4 datacenters with destinations in the US and EU.

Clusters and backup destinations can be ordered across a partner footprint of more than 700 datacenters in 85 metros, 30 countries and six continents, with auto-provisioning in New York, London, Frankfurt and Singapore, so the backup can sit in a different region from the cluster.