Redis Cluster Day Two: What Happens on Node Failure, and How Scaling by Pairs Reshards Without Downtime

A cache that fails takes the application with it, which is why teams that depend on Redis eventually run it as a cluster. But a cluster is only as good as its operator's understanding of what it does under stress. This post follows an SRE through two events on a Redis Cluster running on MassiveGRID PaaS, which is built on Virtuozzo Application Platform, formerly Jelastic: a primary failing at night, and the cluster growing from six nodes to eight.

The cluster backs the API of a ticketing platform we will call Gatefold Events, storing rate-limit counters, seat-hold locks and cached event data. It is the Redis Cluster package: three primaries, three replicas, each on a separate physical host, with the option to scale in primary-replica pairs up to twelve nodes.

Event one: a primary dies at 02:14

The platform's hardware is not immune to failure; what matters is that the cluster is designed for it. At 02:14 the container hosting primary number two became unreachable. Redis Cluster's own health protocol handled it: the other nodes marked the primary as failing after the configured node timeout, its replica was promoted to primary for the same hash slots within seconds, and clients that used the cluster's MOVED redirection reconnected to the new primary on their next command. The write that was in flight at the moment of failure was lost, which is the documented trade-off of asynchronous replication; Gatefold's seat-hold logic tolerates this by treating a missing lock as expired.

The platform, meanwhile, restored the failed container on a healthy host. When it came back it rejoined the cluster as a replica of the newly promoted primary, and the cluster was back to three-plus-three. No one was paged because the SRE had set the alerting thresholds on things users feel, error rate and latency, and neither moved.

What you should still do in the morning

• Check the cluster's view of itself. From any node's Web SSH console, redis-cli -a the-password cluster nodes lists roles, slot ranges and states. The SRE confirms three primaries with contiguous slot coverage and three replicas, one per primary.
• Look at the platform's statistics for the affected node around 02:14: RAM, CPU and disk I/O per minute. The failure was host-level, not resource-level, which the flat graphs confirm.
• Consider whether roles should be rebalanced. After a failover a former replica is now a primary, which is fine; Redis Cluster does not care which physical node holds which role. The SRE leaves it.

Event two: growing from six nodes to eight

Three months later the platform's default load alert fires: one Redis node is above 80% of its cloudlet limit for RAM. Rate-limit counters have grown with a new event series. The SRE has two options. Vertical: raise the node's cloudlet limit (a cloudlet is 128 MiB of RAM plus 400 MHz of CPU) so the platform allocates more memory to it. Horizontal: add a primary-replica pair so the key space is spread across four primaries instead of three.

He raises the limit first, because it is instant and the platform's vertical scaling costs only what the node then uses. But the per-node CPU graph shows the same node is also busiest, because Redis is single-threaded per node and that node's slots hold the hottest counters. Horizontal is the real fix, and the package supports it in the way Redis Cluster requires: scaling is always by pairs, and when a primary is added the platform runs automatic resharding and rebalancing, migrating a share of the hash slots from the existing primaries to the new one while the cluster stays online. Slot migration in Redis Cluster is live: keys move slot by slot, clients that hit a moving slot get an ASK redirection and follow it, and no key is unavailable for more than the moment it takes to move.

He scales the layer from six to eight from the dashboard. Twelve minutes later the cluster has four primaries with roughly a quarter of the slots each and four replicas, and the hot node's RAM is at 55% of its limit.

Or let the triggers do it

The package can be installed with horizontal auto-scaling enabled, and its default triggers are worth knowing: add two nodes when CPU or memory across the cluster exceeds 70% for five minutes, up to twelve nodes; remove two when the load is below 40% for fifteen minutes, down to six. Scaling in also reshards, moving slots off the departing primary before it is removed. Gatefold's SRE enables it after the manual scale, with the ceiling at ten nodes, because event on-sales are exactly the kind of predictable-in-shape, unpredictable-in-timing load that triggers handle better than people at 02:14.

Persistence and backups for data you would miss

Gatefold's counters and locks are ephemeral, but the cached event catalogue is not free to rebuild. The nodes persist RDB snapshots and can enable AOF; the SRE turns on AOF with a one-second fsync on the primaries so a full-cluster restart loses at most a second. For an off-cluster copy he installs the Database Backup/Restore add-on, which supports Redis, and schedules a nightly dump to a backup storage node. It has never been used. That is what a good backup looks like.

Frequently Asked Questions

How long does failover take, and can I tune it?

Redis Cluster marks a primary as failed after cluster-node-timeout (15 seconds by default) plus the election, so promotion typically completes within 15 to 30 seconds; many deployments lower the timeout to 5 seconds. Edit redis.conf on the nodes through the platform's config editor and restart sequentially to apply.

Can I add just one node instead of a pair?

Not through the package's scaling, and you should not want to: a primary without a replica is a single point of failure for its slots. Scaling by pairs keeps every slot range replicated. If you need less capacity per pair, use a lower cloudlet limit per node instead.

Does resharding affect application latency?

Slightly and briefly. During slot migration, commands for keys in a moving slot may receive an ASK redirection, which cluster-aware clients follow transparently with one extra round trip. The migration is throttled, so throughput on the other slots is unaffected. Schedule manual scaling outside peaks anyway.