Deferring updates is a rational response to an asymmetric bet: a good outcome is invisible and a bad one costs your evening and possibly your data. The fix is not more courage but a procedure dull enough to repeat monthly, where the decisions were made once. This is that procedure, and the one property of database migrations that makes rollback harder than it appears.
The reason people put off upgrading self-hosted software is not laziness. It is that the downside is asymmetric: nothing visible improves when it works, and a bad one costs an evening and possibly data. The way out is a routine boring enough to run monthly, which means deciding a few things once rather than every time.
Pin First, or Nothing Else Helps
An application tracking a floating tag upgrades itself whenever a container is recreated, which can be a reboot, a host migration, or a routine restart. That means an unplanned upgrade at a moment nobody chose, and it makes every other practice here impossible.
# wrong: version changes whenever the container is recreated
image: example/app:latest
# right: the version changes when you change this line
image: example/app:2.14.3Pin the exact version, in a file that lives in version control. That file then answers the question "what were we running before it broke", which is otherwise a matter of memory. The same applies to package pins on a distribution install and to chart versions if you deploy with Helm.
An automatic-update tool with a floating tag is the worst combination available: unattended changes to software that holds your data, at times you did not pick.
Read the Right Two Things
Between your current version and the target there are usually several releases, and skimming the newest release notes misses everything in between.
Read the release notes for every version you are crossing, looking specifically for breaking changes, and read the upgrade or migration guide if the project publishes one separately. Many projects put the thing that will break you in a migration document that the release notes only link to.
Three specific things to look for. Whether the version supports upgrading directly from yours, because some projects require intermediate stops. Whether a bundled dependency's major version changed, since a database engine upgrade is a separate operation with its own risks. And whether any configuration key was renamed or removed, which produces a service that starts and quietly ignores half its settings.
The Migration That Only Goes One Way
This is the fact that makes application upgrades different from anything else you patch. When an upgrade runs a schema migration, the old binary can no longer read the new database. Rolling back the application without rolling back the data leaves you with software that will not start.
So rollback is not "run the old image again". It is restore the database to the pre-upgrade state, restore the application to the previous version, and accept losing anything written in between.
| Change type | Rollback cost |
|---|---|
| Application code only | Change the tag back, restart |
| Additive schema change | Usually safe to run the old version |
| Destructive schema change | Full restore; data since the backup is lost |
| Bundled database engine upgrade | Full restore; the data directory format changed |
| Search index or cache rebuild | Safe, but slow and CPU-heavy on restart |
Identify which row you are in before you start. It determines whether you need a maintenance window with users locked out, or whether you can do this on a Tuesday afternoon.
Rehearse on a Copy
The only staging environment worth having is one restored from a production backup. A staging instance with three test records exercises none of the paths that matter: the custom field somebody added, the plugin two versions behind, the table with four million rows whose migration takes ninety minutes.
Restore the most recent production backup onto a second instance, run the upgrade there, and time it. That timing is the number you put in the maintenance notice, and it is routinely several times what people guess. Then have somebody who uses the application daily click through their real workflows, because they will notice the report that now renders empty and you will not.
Spinning up a temporary instance for a rehearsal is cheap on per-resource pricing: at $2.87 per CPU core, $0.80 per GB of RAM and $0.01 per GB of SSD per month, a 4 vCPU, 8 GB, 100 GB rehearsal instance is $18.88 for a full month and a fraction of that for an afternoon. That is less than the cost of one failed production upgrade in anybody's time.
The Backup Immediately Before
Last night's backup is not the pre-upgrade backup. Take one at the start of the window, with the application stopped or in maintenance mode so nothing is written during it, and verify it completed rather than assuming.
Include everything the application needs to be whole: the database, the uploads or data directory, the configuration, and any secrets or key files stored outside both. A dump that restores the database and loses the encryption key restores nothing usable. Our guide to backup encryption and key recovery covers that last category, which is the one people omit.
Where the platform supports it, a snapshot of the whole instance immediately before is the fastest rollback available, since it returns the machine to a known state in one step. Treat it as a rollback mechanism for the window, not as a backup: our guide to immutable backups covers why a snapshot on the same storage is not a substitute.
A Sequence That Fits on a Card
Announce the window. Rehearse on a restored copy and time it. Note the current version and every customisation. Put the application in maintenance mode. Take and verify a backup, and a snapshot if available. Change the pinned version and apply. Run the migration command if the project has an explicit one. Bring it up and read the logs before opening it to users. Check the specific workflows the release notes touched. Leave maintenance mode. Record what you did and how long it took.
The last step is the one that gets dropped and the one that makes the next upgrade cheaper. A file with the version history, the timings and the surprises turns a monthly upgrade from an investigation into a checklist.
How Often
Monthly for patch releases, promptly for anything with a security advisory, and deliberately for major versions once the first point release exists. Waiting for x.y.1 rather than x.y.0 is not cynicism; it is letting other people find the packaging mistakes.
Falling more than two major versions behind is the position to avoid. Projects test upgrades from recent versions, upgrade paths get longer, and eventually the only route forward is a series of intermediate upgrades or a migration into a fresh install. The applications that become unmaintainable are almost always the ones nobody touched for two years.
The Platform That Makes This Cheap
Two things turn an upgrade from an event into a routine: a disposable rehearsal instance, and a rollback that does not depend on the upgrade going well.
MassiveGRID's Linux VPS is priced per resource at the rates above, so a rehearsal instance is sized to the job and destroyed afterwards. The platform runs Proxmox high-availability clustering with automatic failover over Ceph storage replicating every block three times across independent NVMe drives, with two free VPS snapshots included, and backup services add block-level incremental backups with AES-256 encryption at $0.01 per GB for the copies that need to survive the instance. DevOps support covers running the rehearsal and the window where nobody internally wants to own it.
Instances can be ordered across a partner footprint of more than 700 datacenters in 85 metros, 30 countries and six continents, with auto-provisioning in New York, London, Frankfurt and Singapore.