Self-Hosted GitLab with Auto-Scaling CI Runners: Ending the Monday Morning Pipeline Queue

A CI system has two failure modes: pipelines that queue because there are not enough runners, and runners that sit idle all night because there are too many. Most self-hosted GitLab installations live with one or the other. This post shows how a DevOps engineer gets neither by running the GitLab package on MassiveGRID PaaS, which is built on Virtuozzo Application Platform, formerly Jelastic, and letting the runner layer scale with the queue.

The team is a 40-developer software company we will call Corvid Systems, moving off a GitLab instance on a single virtual machine that also hosted its only runner. The DevOps Lab - GitLab Server package deploys GitLab Community Edition with a container registry on one node and a separate, scalable layer of GitLab Runners that register themselves.

What the package actually installs

The package creates two system containers running Docker Engine CE. The first runs GitLab CE and its container registry from a Compose file, with GitLab's bundled PostgreSQL, Redis and NGINX inside. The second runs a GitLab Runner with the Docker executor, registered to the server automatically during installation. The certificates GitLab uses are shared with the runner layer over an NFS volume, so runners trust the registry and the server without manual steps. A set of add-ons is pre-installed on the server node: Let's Encrypt with a custom domain, SMTP settings for notification email, and an Update GitLab Server action that upgrades to the latest release.

The detail that matters for this story is the last one in the package description: automates the discovery of newly added Runners upon horizontal scaling. The runner layer is designed to be scaled out, and a new node registers itself to the server when it appears.

The Monday queue, measured

Before the move, Corvid's engineer pulled a month of pipeline data. The pattern was unambiguous: between 9:00 and 11:00 on weekdays, and again after lunch, twenty or more pipelines were pending at once, each waiting several minutes for the single runner. Between 19:00 and 07:00 the runner executed almost nothing. Nightly scheduled pipelines ran at 02:00 and finished by 02:40.

The obvious fix on a VM is more runners, which is more VMs running idle for fifteen hours a day. The platform's fix is a scaling trigger.

Scaling the runner layer on CPU

On the environment's Auto Horizontal Scaling settings the engineer adds two triggers on the runner layer. Scale out: add one node when the layer's average CPU exceeds 70% for 3 minutes, up to a maximum of six nodes. Scale in: remove one node when average CPU is below 20% for 15 minutes, down to a minimum of one. The platform evaluates the triggers every minute against the average across the whole layer, and the two directions are independent.

CPU is the right signal for CI. A runner executing a build pegs its cores; a runner waiting for a job uses almost nothing. The 15-minute scale-in delay means a runner survives the gap between two pipelines in the same merge request, and the 3-minute scale-out delay means a single 90-second job does not summon a new node.

The runner layer is scaled in stateless mode, which is exactly right here: each new node is created from the base image, runs the package's registration script, and appears in GitLab's runner list within a minute or two. When it is removed, GitLab marks it offline and the platform deletes the container. Nothing on a runner is worth preserving; the Docker executor pulls fresh images for every job, and caches live in the registry.

What a pipeline minute costs when runners are hourly

Each runner node is sized at 8 to 16 cloudlets (a cloudlet is 128 MiB of RAM plus 400 MHz of CPU), with the platform's vertical scaling allocating the actual amount used within that range by the hour. In a typical month Corvid's runner layer averages about 2.2 nodes across all hours, or roughly 26 cloudlets an hour when the vertical scaling is accounted for. At the published rate of $0.003372 per cloudlet-hour that is about $64 a month before discounts for a CI capacity that peaks at six parallel runners.

The GitLab server node itself needs more: GitLab CE is a large Ruby application and comfortably wants 4 to 8 GiB, so a 32 to 64 cloudlet limit with vertical scaling doing the work. That node runs all month at perhaps 40 cloudlets on average, around $100 before discounts. Compared with a hosted GitLab tier priced per user, 40 developers on a mid tier would run into the low thousands a month, and hosted compute minutes are metered on top. The self-hosted numbers above have no per-seat component at all.

Keeping the server node healthy

The server node is the one that cannot be stateless, so the engineer gives it the attention the runners do not need. The container registry stores images on the node's disk, which grows quickly with per-commit images, so a disk load alert at 80% is set and a registry garbage-collection job runs weekly. Backups use GitLab's own gitlab-rake gitlab:backup:create executed inside the GitLab container on a cron schedule, with the archive copied to a backup storage node; the runners hold nothing and are never backed up. The Update GitLab Server add-on is run monthly on a cloned environment first, and on production the following day, because GitLab upgrades are sequential by minor version and a clone is the cheapest place to learn that.

What changed for the developers

The Monday queue disappeared. Nobody at Corvid saw a runner being added or removed; they saw pipelines start when they pushed. The engineer sees it in the environment's scaling history and in a monthly bill that is lower than the two VMs the company was about to buy, because the runners now exist only when the developers do.

Frequently Asked Questions

Can I use my own runners, for example on a GPU server, alongside the auto-scaling ones?

Yes. Register any external runner to the GitLab server with a runner token and tag it (for example gpu). Jobs that need it select the tag; everything else lands on the auto-scaling layer. MassiveGRID's GPU dedicated servers are a common home for such a runner.

Does horizontal scaling work if I run the Docker executor with docker-in-docker builds?

Yes. Each runner node is a full Docker Engine, and the package's registration uses the Docker executor. Docker-in-docker and Kaniko-style image builds work as on any Docker host. New nodes carry the same configuration because it is applied at registration.

How does a scaled-in runner avoid killing a running job?

Set the scale-in trigger's delay long enough that a runner is idle for a while before it qualifies, and keep the minimum at one. GitLab Runner also handles the SIGTERM it receives on removal by finishing or failing the current job gracefully; a failed job can be retried. For long jobs, use a longer scale-in window.