Multi-Region Redis for a Global Application: Local Reads in Three Cities, Replicas That Survive a Region

A cache is useful in proportion to how close it is. An application in Singapore reading from a Redis in London pays a round trip of roughly 180 milliseconds per command, which is slower than most database queries it was meant to avoid. Global applications therefore need their cache where their users are, and they need it to survive the loss of a region. This post follows an architect solving both on MassiveGRID PaaS, which is built on Virtuozzo Application Platform, formerly Jelastic, using the Multi-Region Redis Cluster package.

The application is a SaaS analytics dashboard we will call Meridian Metrics, with application environments in London, New York and Singapore serving customers in those regions. The Multi-Region Redis Cluster package deploys one Redis Cluster across at least three regions, in either a Maximum Performance topology (data sharded across the regions) or a Maximum Reliability topology (primaries in one region, replicas in the others), and configures the cross-region addressing for you.

The shape of the problem

Meridian caches rendered dashboard fragments and per-tenant configuration. Reads are constant and latency-sensitive; writes are occasional, when a tenant changes a setting or a fragment is invalidated. Its three application environments already live in three MassiveGRID regions, each a separate data centre, so the architecture question is where the cache lives relative to them. A single Redis in one region makes two of the three applications slow. Three independent caches make invalidation a distributed-systems project. A single Redis Cluster whose nodes are spread across the regions is the third option, and it is what the package builds.

Two topologies, and which one Meridian picked

The package offers two layouts, and the choice is the most important decision in the install dialog. Maximum Performance shards the key space across primaries in all of the selected regions, with replicas alongside, so every region owns a share of the data and the cluster scales horizontally. Maximum Reliability puts the primaries in the first region you list and replicas in each of the others, which is not scalable but survives the loss of one or even two data centres. Both give failover without downtime if a node fails; the difference is what happens when a whole region goes.

Meridian chooses Maximum Performance, because local reads in three cities are the goal and the dataset is small enough that a region's loss is survivable: Redis Cluster stays available while a majority of primaries are reachable and every unreachable primary has a reachable replica, and the cluster's replica migration gives an unreplicated primary a spare replica from one that has several. The architect's job is to make sure the layout has the spares. With three primaries she runs two replicas each (nine nodes), so any single region's loss leaves every slot range covered.

The installer takes care of what makes this painful by hand: nodes in different regions must announce addresses the others can reach, and the failover timers must tolerate inter-region latency. It also creates the cluster's environments in one environment group, optionally isolated, so the three regional environments are managed as a unit. If the clients live outside the platform, an option attaches public IPs to the nodes (three per region); Meridian's applications are on the platform, so it keeps everything on internal addresses.

Reading locally

By default Redis Cluster sends every command to the primary that owns the key's slot, which for a Singapore user reading a key owned by London's primary is the 180 millisecond trip the architect wants to avoid. Redis has the answer built in: replica reads. A client that issues READONLY on a connection to a replica may read from it, accepting that the replica can lag the primary by the replication delay. Meridian's dashboard fragments tolerate a second of staleness happily. The architect configures each application's client (ioredis in Node.js, with scaleReads set to prefer replicas nearest to it) so that reads go to whichever node in the local region holds the key, primary or replica, and writes go to the owning primary wherever it is.

The installer's success email lists every node with its region, and the architect gives each application environment the nodes in its own region as seeds, so the client discovers the topology from local nodes and prefers them for reads. With two replicas per primary spread across the other two regions, every slot range has a copy in every region, and no read has to cross an ocean; only writes to a remotely owned slot do.

Writes, and what cross-region replication costs

Writes go to the primary that owns the slot, which for a tenant in Singapore whose key hashes to London's slots is a cross-region write. Meridian handles this with a design choice rather than a platform feature: tenant configuration keys are written with a hash tag that pins each tenant's keys to slots owned by the primary in the tenant's home region. Writes are then local for the common case, replication to the remote replica is asynchronous and does not delay the client, and only the rare cross-tenant key crosses regions. Replication traffic between regions is small for a cache with few writes; the architect confirms it in the platform's per-node network statistics after a week.

Losing a region on purpose

Before going live the architect tests the failure the design is for. She stops the Singapore Redis nodes from the dashboard. The cluster's health protocol notices within the node timeout, promotes a replica of Singapore's slots in London or New York, and the Singapore application, whose local Redis nodes are gone, reconnects through its remaining seeds to the London and New York nodes. Reads from Singapore are slow during the outage, which is expected and far better than failing. She starts the nodes again; they rejoin as replicas, and after a few minutes the cluster is back to its intended layout. The test took twenty minutes and cost nothing beyond the running nodes.

Cost and sizing

The cluster is nine nodes across three regions: three primaries and two replicas each. Each node is small; Meridian's cache is a few hundred megabytes, so nodes idle at 2 to 4 cloudlets (a cloudlet is 128 MiB of RAM plus 400 MHz of CPU) under the platform's vertical scaling. Nine nodes at that size come to roughly 25 to 30 cloudlets an hour, or about $60 to $75 a month at MassiveGRID's published $0.003372 per cloudlet-hour before discounts, for a cache that is local in three cities and survives losing one. Inter-region replication traffic is billed as external traffic on the nodes and, for a read-heavy cache, is a minor line.

Frequently Asked Questions

Does cross-region replication make writes slower?

No. Redis replication is asynchronous, so a write is acknowledged by the primary as soon as it is applied locally; the replica in another region receives it milliseconds later. What is slower is a write from an application to a primary in another region, which the hash-tag design above avoids for the common case.

How many regions does the package need?

At least three, chosen from New York, London, Frankfurt and Singapore. Redis Cluster needs a majority of primaries reachable to keep serving, and three regions is the smallest layout in which losing one region leaves a majority. With the Maximum Reliability topology the first region you list hosts the primaries and the others hold replicas.

What about a multi-region database to go with it?

The marketplace has multi-region packages for MariaDB Galera, MySQL and PostgreSQL, built on the same idea of nodes in several regions with local reads. The PostgreSQL Multi-Region Cluster places a primary in one region and hot standbys in others with Pgpool-II in each region.