🦊 GitLab Runners Topologies: Pros and Cons
Tags: gitlab, devops, kubernetes, aws
Published: 2023-02-07 15:30:29

---
title: "\U0001F98A GitLab Runners Topologies: Pros and Cons"
tags:
  - gitlab
  - devops
  - kubernetes
  - aws
license: public-domain
cover_image: 'https://raw.githubusercontent.com/bcouetil/articles/main/images/gitlab/pipes/many-pipes-city-1.jpg'
published: true
id: 1320154
date: '2023-02-07T15:30:29Z'
---

<!-- (masterpiece, best quality), <lora:add_detail:.5>, sharp, intricate details,  ((<lora:ManyPipesAI:1> ManyPipesAI orange city)), orange pipelines -->

{%- # TOC start (generated with https://github.com/derlin/bitdowntoc) -%}

- [Initial thoughts](#initial-thoughts)
- [Time spent handling cache](#time-spent-handling-cache)
- [Topologies without autoscaling](#topologies-without-autoscaling)
  - [Classic single server](#classic-single-server)
  - [Multiple fixed servers](#multiple-fixed-servers)
  - [Kubernetes runner on fixed size cluster](#kubernetes-runner-on-fixed-size-cluster)
- [Topologies with autoscaling](#topologies-with-autoscaling)
  - [GitLab shared runners](#gitlab-shared-runners)
  - [Kubernetes runner on node autoscaling cluster](#kubernetes-runner-on-node-autoscaling-cluster)
  - [Docker machine on Amazon EC2](#docker-machine-on-amazon-ec2)
  - [Docker machine on AWS Fargate](#docker-machine-on-aws-fargate)
- [Wrapping up](#wrapping-up)
- [Further reading](#further-reading)

{%- # TOC end -%}

<!-- cyberpunk one for next article: mechanical orange fox, city, (best quality, masterpiece), cyberpunk -->

# Initial thoughts

GitLab runner can be installed virtually anywhere, on your local personal computer whatever the OS, even on a [Raspberry Pi](https://dev.to/zenika/installation-d-un-gitlab-runner-sur-un-raspberry-pi-en-10-minutes-3o11).

That being said, only a handful of topologies are really viable for professional use. Let's introduce them and elaborate on pros and cons.

_This blog post focuses only on container executors (Docker, Docker Machine, Kubernetes) which are broadly used. But it partially applies to other executors._

# Time spent handling cache

Cache is hell of a subject as a CICD Engineer, one for which you can loose a tremendous amount of time tweaking for performance.

To understand some pros and cons discussed in topologies, we should be aware of the whole caching process in GitLab. We assume here cases where jobs are all using cache and all pulling and pushing it.

When remote cache is not configured, the process is:

- unzip local cache from past jobs/pipelines (if any)
- run the job scripts
- zip produced cache for future jobs/pipelines

When remote cache is configured, the process is:

- download remote cache from past jobs/pipelines (if any)
- unzip cache (if downloaded)
- run the job scripts
- zip produced cache
- upload produced cache for future jobs/pipelines

Without autoscaling, when infrastructure is undersized, some jobs have also to wait for available hardware resources.

On autoscaling infrastructure, when current resources are not enough, some jobs have also to wait for new servers to spin up.

The [official documentation](https://docs.gitlab.com/ee/ci/caching/#good-caching-practices) is pretty straightforward ; for runners to work with caches efficiently, you must do one of the following:

- Use a single runner for all your jobs. At least the similar ones
- Use multiple runners that have distributed caching, where the cache is stored in S3 buckets. Shared runners on GitLab.com behave this way. These runners can be in autoscale mode, but they don’t have to be
- Use multiple runners with the same architecture and have these runners share a common network-mounted directory to store the cache. This directory should use NFS or something similar. These runners may be in autoscale mode

![ManyPipesAI orange city, orange pipelines](https://raw.githubusercontent.com/bcouetil/articles/main/images/gitlab/pipes/many-pipes-city-2.jpg)

# Topologies without autoscaling

No autoscaling means that physical resources are reserved upfront and are, by definition, limited. Several of those topologies are viable. And they all share some pros and cons. Some cons can be mitigated.

_Pros:_

- Simple topology
- Predictable cost
- No runtime consumed by spinning new servers

_Cons:_

- Hard limit capacity
  - Mitigators: better/faster/stronger server (vertical scalability) or more servers (horizontal scalability)
- Constant cost even when not used
  - Mitigator: Stopping servers outside working hours

## Classic single server

A single GitLab runner on a server, able to run multiple jobs locally.

_Pros:_

- Simple architecture
- Fastest GitLab and Docker cache (local)
- Easy to install and configure

_Cons:_

- Local cache is not guaranteed with multiple jobs in parallel
  - Mitigator: [Mounting the cache folder](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1151#note_1019634818)

<!-- - Best solution for
  - Teams not allowed to create/configure remote cache
  - Small to medium projects who need fastest pipelines (no remote cache, available local docker and GitLab cache) -->

## Multiple fixed servers

The single GitLab runner can be extended to multiple servers, each one having a separate installation of a GitLab runner.

_Pros:_

- some form of (manual) scalability
- ability to customize job target
- ability to mitigate cache reliability with runner tags

_Cons:_

- Scalability is not manual
  - Mitigator: infra-as-code
- Unreliable cache when remote cache is not configured
  - Mitigator: define tags on servers to guarantee local cache (one for Node jobs, one for Spring jobs, etc.)
  - Alternative mitigator: share a common network-mounted directory as stated in [GitLab documentation about caching](https://docs.gitlab.com/ee/ci/caching/#good-caching-practices)

<!-- - Best solution for
  - Rare occasions when a better/faster/stronger single server is not an option -->

## Kubernetes runner on fixed size cluster

[Kubernetes runner](https://docs.gitlab.com/runner/executors/kubernetes.html) is a highly viable way of leveraging unused physical resources on existing Kubernetes clusters.

_Pros:_

- Uses available resources on existing cluster

_Cons:_

- Can slow down applications on shared nodes
- Costly if the cluster is only used by GitLab runners
  - mitigators: use of preemptible nodes on Cloud providers

![ManyPipesAI orange city, orange pipelines](https://raw.githubusercontent.com/bcouetil/articles/main/images/gitlab/pipes/many-pipes-city-3.jpg)

# Topologies with autoscaling

Autoscaling means that physical resources are reserved when needed, with virtually no physical limit. Most of those topologies are viable. And, like for the previous category, they share some pros and cons. Some cons can be also mitigated.

_Pros:_

- Virtually no physical limit
- No cost when not used

_Cons:_

- Time consumed by spinning new servers
- Highly unreliable local GitLab and Docker cache

## GitLab shared runners

GitLab shared runner are there for everyone, fully managed by GiLab and consumed in the form of CICD minutes.

_Pros:_

- No maintenance
- Very reasonable price per minute

_Cons:_

- Notably slow
- No local cache

## Kubernetes runner on node autoscaling cluster

[Kubernetes runner](https://docs.gitlab.com/runner/executors/kubernetes.html) is a highly viable way of leveraging autoscaling capabilities on existing Kubernetes clusters.

_Pros:_

- Uses available resources on existing cluster
- At company scale, a dedicated autoscaling cluster becomes cost-efficient

_Cons:_

- Can slow down other applications on shared nodes
  - Mitigator: use pods/nodes affinity/anti-affinity and/or limit-ranges/quotas
- Spinning new nodes slows down the pipelines
  - Mitigator: [over-provision your cluster](https://learnk8s.io/kubernetes-autoscaling-strategies#overprovisioning-nodes-in-your-kubernetes-cluster) by one or two nodes

## Docker machine on Amazon EC2

GitLab can leverage AWS infrastructure to install [autoscaling GitLab Runner on AWS EC2](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/). The architecture is pretty simple: a tiny VM is responsible for spawning spot instances when needed.

![GitLab auto scaling runners on AWS spot instances](https://github.com/npalm/assets/raw/main/images/terraform-aws-gitlab-runner/runner-default.png)

It can be easily installed using infra-as-code with a [Terraform repo](https://github.com/npalm/terraform-aws-gitlab-runner/).

_Pros:_

- cost-efficient

_Cons:_

- Spinning new VMs slows down the pipelines
- Vendor dependant on AWS

## Docker machine on AWS Fargate

Similarly, GitLab can leverage AWS infrastructure to install [autoscaling GitLab CI on AWS Fargate](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws_fargate/). Architecture is straightforward, with the serverless capabilities of the Fargate service.

![Autoscaling GitLab CI on AWS Fargate](https://docs.gitlab.com/runner/configuration/img/runner_fargate_driver_ssh.png)

_Pros:_

- cost-efficient

_Cons:_

- Spinning new VMs slows down the pipelines
- Vendor dependant

# Wrapping up

What is the best topology ? As always, no silver bullet, it depends on a lot of constraints.

Here are opinionated recommendations for some scenarios:

- Any context on _gitlab.com_ where slow CICD is (really) not an issue:
  - **GitLab shared runners**
- GitLab CI local or remote training sessions, with 6 to 8 students working in parallel on individual projects (similar to [Zenika GitLab training](https://training.zenika.com/fr-fr/training/gitlabci/description)):
  - **Dedicated single cloud VM with enough disk IO**
- Structure with a single developers team, mono or multi-repo:
  - **Dedicated single cloud VM with enough disk IO**
- Multi-teams company with knowledge and access to Cloud managed Kubernetes infrastructure:
  - **Dedicated Kubernetes runner on node autoscaling cluster**
- Multi-teams company without Kubernetes knowledge:
  - **Docker machine on Amazon EC2** or **Per team dedicated single cloud VM with enough disk IO**
- Multi gigabytes jobs cache (Spring, Node, etc.):
  - **Dedicated single stateful cloud VM with enough disk IO and custom mounted local cache**

And you ? What are your favorite runners topologies ?

If you experienced other scenarios and/or have different opinions, please share in the comments 🤓

![ManyPipesAI orange city, orange pipelines](https://raw.githubusercontent.com/bcouetil/articles/main/images/gitlab/pipes/many-pipes-city-4.jpg)

_Illustrations generated locally by Automatic1111 using RevAnimated model with ManyPipesAI LoRA_

# Further reading

{% embed https://dev.to/zenika/gitlab-a-python-script-displaying-latest-pipelines-in-groups-projects-5b5a %}

{% embed https://dev.to/zenika/gitlab-a-python-script-calculating-dora-metrics-258o %}

{% embed https://dev.to/zenika/gitlab-ci-the-majestic-single-server-runner-1b5b %}

{% embed https://dev.to/zenika/gitlab-ci-yaml-modifications-tackling-the-feedback-loop-problem-4ib1 %}

{% embed https://dev.to/zenika/gitlab-ci-optimization-15-tips-for-faster-pipelines-55al %}

{% embed https://dev.to/zenika/gitlab-ci-10-best-practices-to-avoid-widespread-anti-patterns-2mb5 %}

{% embed https://dev.to/zenika/gitlab-pages-preview-the-no-compromise-hack-to-serve-per-branch-pages-5599 %}

{% embed https://dev.to/zenika/chatgpt-if-you-please-make-me-a-gitlab-jobs-attributes-sorter-3co3 %}