Build a 3-Node High-Availability Kubernetes Cluster

A highly available Kubernetes control plane is three nodes, one shared API endpoint and an etcd quorum, and the whole design turns on getting the endpoint right in the very first command. This guide covers why three rather than two or four, how to front the API servers, the containerd setting that breaks kubelets under memory pressure, and the two failure tests that prove the cluster behaves as designed.

Almost every failed attempt at this makes the same mistake, and it happens in the first command. If --control-plane-endpoint points at a node's own address rather than at a shared endpoint, you get a cluster with three control planes that cannot survive losing the first one, and discovering that requires rebuilding from scratch.

MassiveGRID infrastructure for Kubernetes: Proxmox HA cluster with automatic failover · Ceph 3x replicated NVMe storage · independent CPU, RAM and storage scaling · 85+ metros across 30+ countries, auto-provisioned in New York, London, Frankfurt and Singapore

Dedicated VPS for guaranteed cores under etcd
Managed Kubernetes — HA control plane pre-configured, from about $25.37/mo

What High Availability Means Here

Be precise about which failure you are buying protection against, because the answer changes the design.

Worker node failure needs no HA control plane at all. Kubernetes reschedules the pods onto surviving workers, which is ordinary behaviour with a single control plane.

Control plane failure is what this article addresses. With one control plane node, a failure leaves running pods running and nothing else: no scheduling, no scaling, no recovery from any subsequent failure, and no kubectl. The cluster is frozen in whatever state it was in.

The distinction matters because a single control plane with three workers is a legitimate design for many workloads, and it costs a third of what this does.

The Quorum Arithmetic

etcd holds cluster state and requires a majority of members to accept a write. That single rule produces the whole node-count table.

Two nodes tolerate nothing, which is worse than one node because you have doubled the chance of an event that stops the cluster. Four tolerate exactly what three do while costing more. Odd numbers only, and three is right for almost everyone.

Five is worth it when you need to survive losing two members, typically because control planes are spread across three failure domains and you want a domain outage plus a node fault covered.

Sizing and Topology

etcd is unusually sensitive to disk latency, because every write is fsynced before it is acknowledged. Slow or contended storage shows up as leader elections and API timeouts rather than as anything that mentions disks, which makes it a genuinely difficult symptom to diagnose. This is the strongest argument for guaranteed cores and NVMe under control plane nodes: shared, oversubscribed storage under etcd produces a cluster that works in testing and destabilises under load.

Stacked topology runs etcd on the control plane nodes, which is simpler and what most clusters use. External etcd on its own three nodes isolates the failure domains and doubles the machine count. Start stacked.

The API Endpoint

Every node and every kubectl talks to one address. That address cannot be a single control plane node, or you have reintroduced the single point of failure at the network layer while believing you removed it.

Two workable approaches. A virtual IP that floats between control plane nodes, using kube-vip or keepalived, which needs no extra machine and requires the provider to permit a shared address on the network. Or a load balancer in front of the three API servers, which is cleaner but is itself a component that must not be a single point of failure.

A minimal HAProxy configuration for the load balancer approach:

frontend k8s-api
    bind *:6443
    mode tcp
    option tcplog
    default_backend k8s-control-plane

backend k8s-control-plane
    mode tcp
    option tcp-check
    balance roundrobin
    server cp1 10.0.0.11:6443 check fall 3 rise 2
    server cp2 10.0.0.12:6443 check fall 3 rise 2
    server cp3 10.0.0.13:6443 check fall 3 rise 2

TCP mode, not HTTP. The API server speaks TLS and terminating it at the proxy breaks client certificate authentication, which is how nodes and kubelets identify themselves.

Preparing Every Node

Identical on all six machines. Swap off, kernel modules loaded, sysctls set, containerd installed:

swapoff -a
sed -i '/ swap / s/^/#/' /etc/fstab

cat <<'EOT' > /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOT
modprobe overlay && modprobe br_netfilter

cat <<'EOT' > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOT
sysctl --system

Then containerd, with the cgroup driver set to systemd. This is the second most common failure after the endpoint mistake: a containerd using the default cgroupfs driver against a systemd host produces kubelets that start and then fail unpredictably under memory pressure.

apt install -y containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
systemctl restart containerd

Initialising the First Control Plane

This is the command that decides whether the cluster is genuinely HA:

kubeadm init \
  --control-plane-endpoint "k8s-api.example.com:6443" \
  --upload-certs \
  --pod-network-cidr=10.244.0.0/16

--control-plane-endpoint must be the virtual IP or load balancer name. --upload-certs stores the control plane certificates in the cluster so the other two can collect them, which saves copying key material by hand.

Keep both join commands the output prints. They differ: one carries --control-plane and a certificate key, the other does not. Using the worker command on an intended control plane produces a cluster that looks right and has one control plane.

Install a CNI before expecting anything to schedule. Without it, nodes stay NotReady and CoreDNS sits pending, which is normal and alarms people:

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml
kubectl get nodes -w

Joining the Other Nodes

# control planes two and three
kubeadm join k8s-api.example.com:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane --certificate-key <key>

# workers
kubeadm join k8s-api.example.com:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

The certificate key uploaded by --upload-certs expires after two hours. Past that, regenerate it with kubeadm init phase upload-certs --upload-certs rather than hunting for why the join is rejected.

Confirm etcd has three healthy members, which is the actual test of whether this worked:

kubectl -n kube-system exec -it etcd-cp1 -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  endpoint status --cluster -w table

Test the Failure You Designed For

An untested HA cluster is an assumption. Kill a control plane node abruptly, not gracefully, because a clean shutdown exercises a different path from a power loss:

# on the node itself
echo b > /proc/sysrq-trigger

# from your workstation
kubectl get nodes
kubectl create deployment failover-test --image=nginx
kubectl get pods -w

The API should stay available and the deployment should be created. If kubectl hangs, the endpoint is pointing at the node you just killed.

Then do the test people skip: kill a second control plane node. The API should stop accepting writes, because two of three members are gone and quorum is lost. Confirming that it fails in the way you expect is as valuable as confirming that it survives one failure, because it tells you what an incident will look like.

Recovery from lost quorum needs an etcd snapshot, so take them on a schedule and store them off the cluster:

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).db \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key

The Certificate Expiry Nobody Plans For

kubeadm issues client and serving certificates valid for one year. They renew automatically when you upgrade the cluster, which means a cluster left untouched for twelve months stops working, and it stops working in a way that looks like a total outage rather than an expiry.

kubeadm certs check-expiration
kubeadm certs renew all       # then restart the control plane pods

Put the expiry date in a calendar the day you build the cluster. This is the single most common cause of a self-managed cluster failing long after anyone has touched it.

Mistakes That Cost a Rebuild

Or Let Someone Else Run the Control Plane

None of the above is difficult. It is, however, permanent: etcd snapshots, certificate rotation, version upgrades across three nodes in the right order, and a load balancer that itself needs to stay up. That is ongoing work, and it arrives on its own schedule rather than yours.

MassiveGRID managed Kubernetes ships the HA control plane already built, with redundant API servers, an etcd cluster and scheduler, Traefik or nginx ingress with TLS termination, horizontal and vertical pod autoscaling, and dynamic persistent volume provisioning. Billing is on the RAM and CPU your pods actually use rather than on server size, measured in cloudlets of 128 MiB and 400 MHz, from $0.03474 per hour or roughly $25.37 a month.

If you would rather build it, put the control plane nodes on Dedicated VPS instances so etcd gets guaranteed cores and uncontended NVMe. Underneath either choice sits a Proxmox high-availability cluster with automatic failover and Ceph storage replicating every block three times, in any of 85+ metros. For a lighter alternative, our k3s guide covers the same HA pattern with embedded etcd and far fewer parts.