When a Database Node Dies at 2 AM: What Automatic Failover Actually Saves, in Money and Sleep

Every CTO has approved a single database server on the grounds that a cluster was overkill for now. The CTO of a payments-reconciliation startup we will call Ledgerline made that call in year one and revisited it in year two after a disk failure at 02:10 took the service down for four hours and cost a customer. This post is her before-and-after arithmetic for moving to the MySQL/MariaDB/Percona Cluster package on MassiveGRID PaaS, built on Virtuozzo Application Platform (formerly Jelastic), written for anyone making the same call.

The incident, priced

The failure itself was ordinary: a storage fault on the single database node. What made it expensive was everything after. The on-call engineer was paged at 02:14, spent forty minutes confirming the database was unrecoverable in place, restored the previous night's backup to a new server, replayed what could be replayed, and brought the service back at 06:05. Reconciliation jobs for three customers ran late; one, whose contract had a service-level clause, invoked it.

The CTO's conclusion was that the company had been running a recovery point objective of one day and a recovery time objective of four hours without ever agreeing to either.

What the cluster costs

The cluster package was sized as three MariaDB nodes in Galera mode plus the two ProxySQL nodes it installs as the entry point. MassiveGRID bills in cloudlets, 128 MiB of RAM plus 400 MHz of CPU each, at a published $0.003372 per cloudlet-hour, which is $2.46 per cloudlet per month. Each node is charged hourly for the greater of its peak RAM and average CPU, and reserved cloudlets carry a discount over dynamic ones.

The premium over the single server is on the order of $50 a month before volume and reserved-cloudlet discounts. The CTO's note in the margin: the service credit alone was more than two years of that premium.

What the cluster changes about 2 AM

The same disk fault on a Galera node produces a different night. The node stops answering; within seconds the two ProxySQL nodes mark it offline and route every query to the two healthy nodes, which hold identical committed data because Galera replicates synchronously. The application sees, at most, a retried query. Meanwhile the platform notices the degraded host and live-migrates the container to a healthy one within about thirty seconds, or restarts it if the storage is genuinely lost, and the node rejoins the cluster with a state transfer from a peer. A load alert emails the on-call engineer, who reads it in the morning.

The two ProxySQL nodes matter as much as the three database nodes: an entry point that is itself a single server would just move the 2 AM problem. The package deploys two so that the proxy layer survives a node loss too.

Replication is not backup

The CTO was careful not to over-claim. A replicated cluster protects against hardware and node failure. It replicates a bad migration or an accidental DELETE perfectly. So the redundancy budget includes the Database Backup/Restore add-on, taking dumps every six hours to a separate Backup Storage node with a retention of 28 copies, and a quarterly restore rehearsal into a cloned environment. That sets the recovery point objective for logical errors at six hours, a number the company can now write into contracts because it chose it.

The Database Cluster Recovery add-on covers the failure mode between the two: a cluster that is up but unhealthy, for example after a network partition leaves Galera without quorum. Its diagnostic reports the state of each node and its recovery action bootstraps from the node with the most advanced transaction sequence and rejoins the others, which is precisely the procedure an engineer would otherwise be performing from documentation at 2 AM.

The board slide

The CTO summarised the change in four lines for the board.

• Before: RTO about four hours, RPO about one day, both undocumented. Monthly database hosting about $20.
• After: RTO seconds for node failure and about fifteen minutes for a logical restore, RPO zero for node failure and six hours for logical errors, both written into customer contracts. Monthly database hosting about $72 before discounts, on infrastructure with a 100% uptime SLA.
• Break-even: the premium equals the cost of one incident roughly every five years. The company had one in its first eighteen months.
• Next: a second region. The multi-region Galera package would extend the same guarantees across two MassiveGRID regions, a conversation for when a customer asks about regional disaster recovery.

Frequently Asked Questions

Is three nodes really necessary, or would two do?

Galera needs a majority to keep quorum, so two nodes cannot safely decide which one is authoritative if they lose contact with each other. Three nodes tolerate one failure while continuing to accept writes. For asynchronous primary-secondary replication two nodes work, but failover is manual and the unreplicated tail of transactions can be lost.

What is the performance cost of synchronous replication?

Each commit waits for the write set to be certified by the other nodes, which within a single MassiveGRID region adds a small, consistent latency, typically single-digit milliseconds. Read performance improves because every node serves reads. For write-heavy workloads that cannot accept the latency, the asynchronous schemes remain available.

Does the platform's own SLA cover the database?

MassiveGRID's 100% uptime SLA covers the infrastructure: hosts, network and storage underneath the containers, with live migration away from degraded hardware. Availability of the database service itself depends on the topology you choose, which is why a clustered scheme is recommended for production data.