GlusterFS Replicated Volume on PaaS: Shared Storage for a Web Cluster That Survives Losing a Storage Node

A load-balanced web cluster removes the single point of failure from the application tier and, quietly, moves it to the storage tier: the one node that holds the uploaded files every application node mounts. GlusterFS is the answer the platform offers for that, a file system whose servers replicate every write. This post follows a DevOps engineer adopting the GlusterFS Replicated Volume package on MassiveGRID PaaS, which is built on Virtuozzo Application Platform, formerly Jelastic.

The cluster serves a recipe-and-community site for a food publisher we will call Saffron Table: four PHP application nodes behind NGINX, a MariaDB cluster, and 600 GB of user-uploaded photographs on a single shared storage node mounted into every PHP node over NFS. The GlusterFS Replicated Volume package deploys three storage nodes in a replicated volume, so every file exists on all of them, with mount details for native and NFS clients.

What a single storage node does and does not protect against

The platform's shared storage container is a good default: NFSv4, a large disk allowance, mount points from the dashboard, and the node itself is on its own physical host with live migration and platform restoration if it fails. What it does not do is keep serving while it is being restored. For Saffron Table, whose editors upload continuously and whose pages are mostly photographs, a storage node outage of several minutes is a site of broken images and failed uploads, and the engineer's incident log had two of those in a year.

How the replicated volume works

GlusterFS has no metadata server; every server (a brick) holds files directly, and the client talks to them. In a replicated volume each file is written to every brick before the write is acknowledged, so the three storage nodes hold identical copies. If a brick is unreachable, reads and writes continue against the others, and when it returns the volume's self-heal brings it up to date. Three bricks is the recommended minimum because it tolerates one failure and avoids split-brain decisions; two bricks plus an arbiter is the lighter alternative. The package configures the trusted pool, the volume (named data), replication and healing settings, and the nodes sit on separate physical hosts by the platform's anti-affinity.

Mounting from the PHP layer: native or NFS

Clients have two choices. The Gluster native (FUSE) client connects to all bricks, writes to all of them and fails over between them itself; it is the choice for failover and write-heavy workloads. NFS mounts a single brick's export and is faster for read-heavy workloads but depends on that brick. The platform's mount points UI on the PHP layer offers both when the server is a GlusterFS cluster. Saffron Table's engineer chooses the native client for the uploads directory, because failover is the whole reason for the change, and accepts the modest CPU cost the docs note for the FUSE client. Every PHP node, including ones added by auto-scaling, gets the same mount because it is defined on the layer.

Migrating 600 GB without a maintenance window

1. Install the GlusterFS package in the same environment group as the web cluster. Size each brick's disk at 1 TB for growth.
2. Mount the Gluster volume on every PHP node at a temporary path with the native client, alongside the existing NFS mount.
3. Copy the 600 GB from the NFS mount to the Gluster mount with rsync over the internal network, which is free of traffic charges. First pass takes an afternoon; the site keeps running on NFS.
4. In a short evening window: put the site in maintenance for two minutes, run a final rsync of the files changed since the first pass, swap the mount paths so the application's uploads directory is now the Gluster mount, lift maintenance.
5. Keep the old storage node for a week as the rollback, then delete it.

The engineer rehearsed steps two to four on a clone of the whole environment first, which was also how he learned that the application's image thumbnail cache wanted to be excluded from replication and left on each PHP node's local disk, because it is regenerable and write-heavy.

Pulling a brick

The test the change was for: with the site live and an editor uploading, the engineer stopped one of the three storage nodes from the dashboard. Uploads continued; pages loaded; the native client on the PHP nodes logged the brick as unreachable and carried on with two. Twelve minutes later he started the node again, and the volume's self-heal copied the files written in the meantime onto it, visible in the heal status from the node's Web SSH console. Nobody on the editorial team noticed, which he confirmed by asking.

Two limitations from the docs are worth knowing and did not matter here: Gluster shares do not work with sleeping (stopped) environments, and they pause during a live migration of a node, which is brief.

Cost, honestly

Three bricks each holding a full copy means three times the disk of one node, and three nodes' cloudlets instead of one. Saffron Table's bricks idle at 3 to 4 cloudlets each (a cloudlet is 128 MiB of RAM plus 400 MHz of CPU) and rise during heals, averaging about 12 cloudlets an hour for the storage tier, roughly $29 a month before discounts at MassiveGRID's published $0.003372 per cloudlet-hour, plus 3 TB of allocated disk against the previous 1 TB. The engineer's framing to the publisher: the extra disk is the price of two more copies of every photograph the readers have ever uploaded, held on three separate machines, and the two outages a year are gone.

Frequently Asked Questions

GlusterFS or the platform's shared storage container?

The single shared storage container suits most clusters and is simpler and cheaper. Choose the GlusterFS replicated volume when the storage itself must stay available through a node failure, or when write concurrency from many application nodes is high. The shared storage container can also be deployed as an auto-clustered set, which uses Gluster underneath.

Can environments outside the group mount the volume?

Yes. Mount by the bricks' addresses with the native client or use the NFS export, over the internal network for environments on the same account and region, or over public IPs with the container firewall opened for environments elsewhere.

How do I add capacity?

Enlarge each brick's disk from the dashboard; a replicated volume's capacity is one brick's size, so all bricks grow together. Adding bricks to a replicated volume increases the replica count, not capacity; for capacity beyond one disk, a distributed-replicated layout is the next step and is a manual Gluster configuration.