Preview Environments Done Properly: An Ephemeral Kubernetes Cluster per Pull Request

Preview environments are the feature every developer wants and every platform team dreads. The usual compromise is a namespace per pull request on a shared cluster, which works until the pull request changes the ingress controller, a CustomResourceDefinition or a cluster-wide operator, at which point one developer's experiment becomes everyone's outage. This post shows how a developer at a fintech we will call Kestrel Pay gets a whole Kubernetes cluster per pull request on MassiveGRID PaaS, built on Virtuozzo Application Platform (formerly Jelastic), and why the finance team stopped asking about the cost.

Why a namespace is not enough

Kestrel's platform repository defines more than application deployments. It carries the ingress configuration, network policies, a Kafka operator and the CRDs for the team's own reconciliation controller. A pull request that bumps the operator version cannot be tested in a namespace, because the operator is cluster-scoped. Historically those changes were tested on a permanent staging cluster, one at a time, with a queue in the team channel.

The fix is a cluster per pull request. On most clouds that is slow (VM-based nodes take minutes) and expensive (a control-plane fee per cluster). On MassiveGRID PaaS a Kubernetes cluster is an environment, and environments can be cloned in minutes and stopped when idle. That changes the economics enough to make the obvious design the practical one.

The golden cluster

The team maintains one environment called k8s-golden: a Kubernetes Cluster package installation with the current production Kubernetes version, the Traefik ingress, the monitoring stack and the team's baseline operators applied. It is kept stopped. A stopped environment on the platform is charged for disk only, so the golden cluster costs a few dollars a month to exist.

When a pull request opens, the CI pipeline calls the platform API to clone k8s-golden into an environment named after the pull request, pr-1842 for example, starts it, waits for the API server to answer, and applies the pull request's manifests with Helm. The clone carries every configuration the golden cluster had, because cloning copies the containers rather than re-running an installer. From the developer's point of view, a comment appears on the pull request with the preview URL and a kubeconfig download link a few minutes after they push.

# Simplified pipeline steps using the platform CLI
~/jelastic/environment/control/cloneenv --srcEnvName k8s-golden --dstEnvName pr-${PR_NUMBER}
~/jelastic/environment/control/startenv --envName pr-${PR_NUMBER}
# wait for the API server, fetch kubeconfig from the master node, then:
helm upgrade --install kestrel ./charts/kestrel -f values/preview.yaml \
  --set ingress.host=pr-${PR_NUMBER}.preview.example.com

The ingress host works immediately because the environment has the platform's built-in wildcard SSL on its own domain, and Traefik was already configured in the golden image. For a customer-style hostname the pipeline can attach a public IP and run the Let's Encrypt add-on, but for previews the platform domain is enough.

Sizing previews so they are cheap and honest

A preview cluster does not need production capacity, but it does need to behave like production. Kestrel sets the clone's workers to two nodes with a low reserved allocation and a dynamic ceiling of 16 cloudlets (2 GiB) each. Vertical scaling means a preview that is being load-tested grows into its ceiling for that hour and shrinks afterwards, and a preview nobody is clicking on idles near its reserved minimum. Every cloudlet is 128 MiB of RAM plus 400 MHz of CPU, billed hourly at MassiveGRID's published $0.003372, so an idle preview cluster with a small control plane and two quiet workers costs on the order of a few cents an hour.

The pipeline also installs the Start/Stop Scheduler add-on on every preview with a rule that stops the environment at 20:00 and starts it at 08:00. Developers can start a preview early from the dashboard if they are working late. Because stopped environments keep their IPs, hostnames and data, a preview started the next morning is exactly where it was left.

Tear-down and the two clusters that live longer

When the pull request merges or closes, a webhook deletes the environment. Nothing lingers, because the environment is the unit of everything: nodes, volumes, the shared storage container behind the NFS provisioner, the public IP if one was attached. There is no orphaned disk to hunt for a month later.

Two previews are exceptions. A pull request tagged perf gets a clone with production-sized worker ceilings and a horizontal auto-scaling trigger, so a load test shows how the platform adds workers under pressure. And the release-candidate branch gets a preview that is promoted rather than deleted: the team swaps its role with staging by moving the Traffic Distributor ratio in front of the two, which is the same blue-green mechanism they use for production.

What developers noticed

The first thing developers noticed was that reviews changed. A reviewer no longer read a Helm diff and imagined the result; they opened the preview URL, ran the smoke tests against a real ingress and a real operator, and approved with evidence. The second thing was that nobody queued for staging any more, because cluster-scoped changes had their own cluster.

The platform team noticed something else. Because every preview is a clone of the golden cluster, drift disappeared. When a Kubernetes upgrade is due, they upgrade k8s-golden in place from the dashboard, test it, and from that moment every new preview is on the new version. Production follows once a week of previews has passed. The golden environment's export, a JSON description of its topology and configuration, is committed to the platform repository, so the whole arrangement is reproducible on another region or another installation without lock-in.

Frequently Asked Questions

How long does cloning a Kubernetes environment take?

A few minutes in most cases. Containers are copied from the source environment, including the control plane's state, and a short freeze on the source may occur during memory migration, which is why the golden cluster is kept stopped rather than cloned while running. Large clusters with a lot of persistent data take longer.

Can previews reach shared services such as a staging database?

Yes, over the platform's internal network, which carries no traffic charge. Put the preview environments and the shared services in the same environment group, or use container firewall rules that reference a whole layer so they stay correct as previews come and go. Network isolation between groups blocks anything you did not explicitly allow.

Does the platform API cover everything the pipeline needs?

Environment creation, cloning, start and stop, scaling, add-on installation and deletion are all available through the REST API and the CLI, which mirrors the API's structure. Every action the dashboard performs has an API equivalent, and the CLI returns JSON that pipelines can parse.