Cross-Region Failover with Traffic Distributor: Running an Active-Passive Web Application Across London and Frankfurt

Most high-availability designs stop at the building: separate hosts, replicated databases, load balancers, all in one data centre. A business-continuity plan that mentions floods or fibre cuts wants more, and the classic answer is a warm standby in another location with something in front that switches. This post follows an architect building that on MassiveGRID PaaS, which is built on Virtuozzo Application Platform, formerly Jelastic, using Traffic Distributor's Failover routing across two regions.

The application is a member portal for a professional association we will call the Institute of Chartered Surveyors of Wessex: a PHP application, a MariaDB database and uploaded documents, whose continuity plan requires the portal to return within fifteen minutes of losing its primary data centre. The Traffic Distributor add-on's Failover routing method sends all traffic to a primary backend and switches to the secondary only when the primary fails its health check.

The design in one diagram's worth of words

Two environments, one in London (active) and one in Frankfurt (passive), each a complete copy of the portal: PHP application layer, MariaDB, document storage. The database in Frankfurt is a replica of London's. Uploaded documents are synchronised from London to Frankfurt. In front, a Traffic Distributor environment with a public IP and the portal's domain, routing method Failover, primary backend London, secondary Frankfurt, health check every few seconds. On a normal day every request goes to London. When London stops answering, the distributor's health check fails and traffic goes to Frankfurt without anyone touching anything.

Where the distributor itself lives, and why that matters

A failover switch is useless if it dies with the thing it protects. The architect places the Traffic Distributor environment in Frankfurt, the passive region: if London is lost, the distributor is still up to notice and switch. The cost is a few milliseconds on every normal-day request, which travel to Frankfurt and back to London; for a member portal that is invisible. The distributor's two NGINX nodes give it redundancy within Frankfurt, and its public IP is the address the domain points at.

Data: the honest part of any failover design

The application layer fails over in seconds because it is stateless. The data does not, and the architect is precise about what happens. Database: Frankfurt's MariaDB replicates from London asynchronously, so at the moment of failure it is behind by the replication lag, normally under a second. On failover it is a read-only replica; the portal in Frankfurt therefore starts in a read-only mode the application already supports for maintenance (members can log in and read, not upload or pay) until an operator promotes the replica to primary and flips the flag. That promotion is a deliberate human step, because automatic promotion during a network partition between regions risks two primaries, and a fifteen-minute recovery objective leaves room for a human. Documents: the File Synchronization add-on keeps Frankfurt's storage a near-real-time copy of London's, so a document uploaded a minute before the failure is almost certainly there.

The recovery point objective, written into the continuity plan from these facts rather than from hope: up to one minute of database writes and uploads in the worst case. The recovery time objective: seconds to a read-only portal, under fifteen minutes to full service after promotion.

Pulling the plug

A failover design that has never failed over is a theory. The architect scheduled a test with the association's staff watching. At 18:00 she stopped the entire London environment from the dashboard. Within about ten seconds the distributor's health check failed three times and switched to Frankfurt; the portal reloaded in read-only mode with a banner. At 18:04 she promoted Frankfurt's MariaDB to primary from its Web SSH console and flipped the application's read-only flag; full service at 18:05. At 18:30 she started London again, which came back as it was, now out of date; she rebuilt London's database as a replica of Frankfurt, reversed the file synchronisation direction, and swapped the distributor's primary and secondary so Frankfurt was now active. The next quarter's test will swap them back. Total cost of the test: nothing beyond the environments that were already running.

What the standby costs on a normal day

Frankfurt's passive environment is a single PHP node, a MariaDB replica and a storage node, idling at about 12 cloudlets an hour (a cloudlet is 128 MiB of RAM plus 400 MHz of CPU) under vertical scaling because nothing is asking it questions; the distributor's two NGINX nodes are another 4 or so. Roughly 16 cloudlets an hour is about $39 a month before discounts at MassiveGRID's published $0.003372 per cloudlet-hour, plus one public IP and the storage node's disk. When Frankfurt becomes active its cloudlets rise to match the load, and London's fall. The continuity plan's previous option had been a contract for a standby server in another provider's facility, at a fixed monthly fee several times that figure, that nobody had ever tested.

Variations

For a database that must fail over without a human, the marketplace's multi-region database packages place primaries and replicas across regions with ProxySQL entry points in each and defined failover behaviour, and this blog's PostgreSQL and MariaDB multi-region posts cover the trade-offs. For an active-active layout where both regions serve users, Traffic Distributor's Round Robin or Sticky Sessions routing works across regions too, with the multi-region database underneath; the price is the cross-region latency on every write, which the architect judged wrong for a portal whose traffic is almost entirely from one country.

Frequently Asked Questions

How fast does Failover routing switch?

It switches when the health check's failure threshold is reached: with the default of checks every few seconds and three consecutive failures, about ten seconds. Tighten the interval for faster switching at the cost of more health-check traffic; loosen it to avoid flapping on a brief blip.

Does the distributor switch back automatically when the primary recovers?

Failover routing returns to the primary once it passes the health check's rise threshold. In the design above that is undesirable, because London's database is stale after Frankfurt was promoted; the architect avoids it by keeping London stopped until it has been rebuilt as a replica, then swapping the primary and secondary in the distributor deliberately.

Can I use DNS failover instead of a distributor?

You can, but DNS changes take minutes to propagate and some clients cache stubbornly, so recovery is slower and less predictable. The distributor switches at the HTTP layer in seconds with one stable IP. Combine them if you also want protection against the distributor's own region.