Jenkins with Auto-Scaling Build Agents on PaaS: Retiring the Static Build Farm

Jenkins agents are the part of a CI system that nobody wants to own. They need capacity for the busiest hour, they idle the other twenty-three, and every one of them is a machine to patch. This post follows a DevOps engineer replacing a static build farm with the Jenkins DevOps Pack on MassiveGRID PaaS, which is built on Virtuozzo Application Platform, formerly Jelastic, where the agents are a layer that grows and shrinks with the queue.

The team is a 25-developer embedded-software company we will call Ferrous Dynamics, with long C++ builds and a Jenkins controller on a VM that also hosted four permanent agents. The Jenkins DevOps Pack deploys a Jenkins controller node and a layer of Java Engine worker nodes, two by default, that can be scaled from one to ten and register themselves with the controller.

What the pack installs, and the detail that matters

The package creates a Jenkins controller container with persistent storage for its home directory, plugins and job configuration, and a separate layer of worker containers running a Java engine, each connected to the controller as an agent. The controller ships with the plugins most pipelines need: Git, Pipeline, Docker, build tools and deployment plugins. The property the engineer cares about is in one line of the package description: workers can be scaled from one to ten nodes with auto-discovery and self-registration on the controller. A worker that the platform creates announces itself to Jenkins and starts taking jobs; a worker the platform removes disappears from the agent list. No SSH keys to distribute, no node definitions to hand-edit.

Measuring the old farm before replacing it

The engineer pulls a month of Jenkins build history. Four agents, each busy about 22% of the time on average, with the busy time concentrated between 09:00 and 17:00 on weekdays and a nightly full build at 01:00 that used two agents for forty minutes. The other 78% of agent-hours were idle VMs being patched, backed up and paid for. The peak, though, was real: on release days all four agents were saturated and developers waited.

Triggers for a build farm

On the worker layer she sets two horizontal scaling triggers. Scale out: add one node when the layer's average CPU exceeds 70% for 2 minutes, up to eight. Scale in: remove one node when average CPU is below 15% for 20 minutes, down to one. The platform evaluates the triggers every minute against the whole layer's average. The 20-minute scale-in window is deliberate: a C++ build has quiet phases (linking, test collection) and a shorter window would remove a node between two stages of the same job.

Agents are stateless in the scaling sense: the layer is scaled from the base image with the registration script, so a new worker arrives clean and identical. Build workspaces are ephemeral, which the team had wanted for years; the dependency cache that made the old agents "warm" moves to a Nexus Repository proxy in the same environment group, so a fresh agent pulls dependencies from the local network rather than the internet.

Sizing the controller, which is the stateful part

The controller is the one node that must not be treated as disposable. It holds job definitions, credentials (encrypted on its persistent disk), build history and plugins. The engineer gives it a cloudlet limit of 32 (a cloudlet is 128 MiB of RAM plus 400 MHz of CPU) so the JVM has room during plugin updates and history indexing, and lets vertical scaling allocate what it uses by the hour. Build history is pruned per job with the discard-old-builds option, so the disk does not grow forever, and a disk load alert at 80% covers the rest. Backups are a nightly archive of the Jenkins home directory to a backup storage node, taken by a scheduled job on the controller itself.

What a build costs now

Worker nodes are sized at 8 reserved and 16 dynamic cloudlets, matching the package defaults, so a saturated worker draws about 2 GiB and 6.4 GHz. Averaged over a month, the layer runs about 2.4 nodes, roughly 30 cloudlets an hour when vertical scaling is accounted for, plus the controller's average of about 18. At MassiveGRID's published $0.003372 per cloudlet-hour that is about $118 a month before discounts for the whole CI estate, against four VMs that had cost more than that and never scaled past four. Ferrous also stops the entire Jenkins environment on public holidays with the Start/Stop Scheduler, which bills a stopped environment for disk only.

Docker builds and the one gotcha

The workers have Docker available and the Docker Pipeline plugin is installed, so container image builds run on the agents as before. The gotcha the engineer hit: a job that pinned itself to a named agent with a label broke when that agent was scaled away. The fix is the point of the whole exercise: jobs target the label the layer's workers share, never an individual node, and Jenkins schedules across whatever workers exist at that minute.

Frequently Asked Questions

Can I keep a specialised agent, for example on a GPU server or Windows, alongside the scaling layer?

Yes. Register it with the controller the ordinary way and give it a distinct label. Jobs that need it request that label; everything else uses the auto-scaling layer's label. MassiveGRID's GPU dedicated servers are a common home for such an agent.

How does a scaled-in worker avoid killing a running build?

Keep the scale-in window long, as above, so a worker must be idle for a while before it qualifies. Jenkins marks a disconnected agent's build as failed, which can be retried; for long builds, set the scale-in delay longer than your longest stage.

Jenkins or GitLab CI on the platform?

Jenkins is a general automation server with a vast plugin catalogue and works with any Git host. The GitLab Server package bundles CI with repositories and a registry, with runners that scale the same way. Teams already on GitHub or Bitbucket, or with existing Jenkinsfiles, usually stay with Jenkins.