Deploying gitlab on Docker Swarm
Tags: showdev, gitlab, docker, swarm
Published: 2018-01-14 00:00:00

---
title: Deploying gitlab on Docker Swarm
published: true
date: '2018-01-14'
description: 
tags: showdev, gitlab, docker, swarm
---

**Update**: You may want to see the instructions [on the gitlab documentation](https://docs.gitlab.com/omnibus/settings/nginx.html).

Docker Swarm is a container orchestration system that is very easy to setup and to get started with, it is built in the Docker engine and can be set up in a few minutes. But deploying a service like Gitlab can be a bit tricky.

In this tutorial I will show how to deploy Gitlab on Docker Swarm while trying to mimic a production environment.

## But Kubernetes won the container orchestration war!

I have heard that since Docker announced support for Kubernetes and that Swarm will be discontinued, but I believe the two solutions will coexist (at least I hope so). Kubernetes is an excellent solution but it is not very easy to get started with, and setting up a new cluster can be quite challenging, specially if you have just a few servers available (remember that you need an etcd cluster too).

Kubernetes is still an excellent choice and my guess is that Swarm might become a façade for Kubernetes.

## Before we begin...

There are a few things that we will need before we can get to Gitlab:

- A proxy to forward the requests to the correct services
- Postgres and Redis
- Prometheus to collect metrics from Gitlab and Grafana to visualize these metrics
- An NFS4 share to store any persistent data we need

### The Proxy

For the proxy, we can use [Traefik](https://traefik.io/). We just need to create a new network on our Swarm cluster a start the Traefik service:

```sh
$ docker network create --driver overlay proxy
$ docker service create \
    --name traefik \
    --constraint=node.role==manager \
    --publish 80:80 --publish 8080:8080 \
    --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
    --network proxy \
    traefik \
    --docker \
    --docker.swarmmode \
    --docker.domain=localtest.me \
    --docker.watch \
    --web
```

We first created the `proxy` network so any service attached to this network will be reachable by Traefik. By mounting the Docker socket into the Traefik service, we allow it to watch for when new services are deployed and to reconfigure itself. The argument `--docker.domain=[localtest.me](http://readme.localtest.me/)` will instruct Traefik to proxy `{service_name}.localtest.me` to the corresponding service ([and you can override the default settings](https://docs.traefik.io/configuration/backends/docker/#labels-overriding-default-behaviour)).

Something to be noted is that the domain "[localtest.me](http://readme.localtest.me/)" is a service that redirects to `127.0.0.1`. Pretty handy since do not have to mess up with `/etc/hosts`.

### Setting up NFS

First we need to install the NFS server:

```sh
# On Debian/Ubuntu
apt-get install -y nfs-kernel-server

# On RedHat/CentOS
yum install -y nfs-utils

# Start the NFS server and enable it on startup
systemctl enable --now nfs-server
```

For the NFS share the will need a directory structure like the following:

    /srv/gitlab-swarm/
    ├── gitlab
    │   ├── config
    │   ├── data
    │   └── logs
    ├── grafana
    ├── postgres
    └── prometheus

We can create it using the following shell:

```sh
mkdir -p /srv/gitlab-swarm && \
mkdir -p /srv/gitlab-swarm/gitlab/{data,logs,config} && \
mkdir -p /srv/gitlab-swarm/postgres && \
mkdir -p /srv/gitlab-swarm/grafana && \
mkdir -p /srv/gitlab-swarm/prometheus && \
chmod -R 777 /srv/gitlab-swarm
```

Then we need to create the directory `/exports/gitlab-swarm` and mount `srv/gitlab-swarm` onto it (this is required for NFS version 4):

```sh
mkdir -p /exports/gitlab-swarm
mount --bind /srv/gitlab-swarm /exports/gitlab-swarm
```

Setup the NFS share by editing `/etc/exports`:

```
# /etc/exports

/exports/               *(rw,sync,fsid=0,crossmnt,no_subtree_check)
/exports/gitlab-swarm   *(rw,sync,no_root_squash,no_subtree_check)
```

Now we reload the NFS configuration:

```sh
exportfs -ra
```

**update 2018-08-18**
As [pointed out by Alexey Petushkov](https://dev.to/apetushkov/comment/4lg2), in order to make the nfs mounts persist after reboots, we need to add the following line to `/etc/fstab`:

```
/srv/gitlab-swarm/      /exports/gitlab-swarm/  none    bind
```

Remember that we need the NFS client installed on each node of the cluster:

```sh
# On Debian/Ubuntu
apt-get install -y nfs-common

# On RedHat/CentOS
yum install -y nfs-utils
```

To test if the NFS configuration is correct, we can try mounting the share:

```sh
mkdir /var/tmp/test-nfs && \
mount -t nfs4 127.0.0.1:/gitlab-swarm /var/tmp/test-nfs && \
grep nfs4 /proc/mounts | cut -d ' ' -f 1,2,3 && \
umount /var/tmp/test-nfs
```

Running the commands above should output this:

```
127.0.0.1:/gitlab-swarm /var/tmp/test-nfs nfs4
```

## Building the stack

### Configuration files

To deploy our Gitlab stack first we need to create the configuration files for the services we are deploying.

The first one is for Gitlab itself (gitlab.rb). The Docker image version we are using (10.3.3) is based on the omnibus installation and contains all the services needed for gitlab, including postgres, redis and prometheus. We will disable these services and set gitlab to look for them in other containers:

```ruby
# gitlab.rb

external_url 'http://gitlab.local'
registry_external_url 'http://registry.gitlab.local'

# Disable services
postgresql['enable'] = false
redis['enable'] = false
prometheus['enable'] = false
postgres_exporter['enable'] = false
redis_exporter['enable'] = false

# Postgres settings
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_encoding'] = "unicode"

# database service will be named "postgres" in the stack
gitlab_rails['db_host'] = "postgres" 
gitlab_rails['db_database'] = "gitlab"
gitlab_rails['db_username'] = "gitlab"
gitlab_rails['db_password'] = "gitlab"

# Redis settings
# redis service will be named "redis" in the stack
gitlab_rails['redis_host'] = "redis"

# Prometheus exporters
node_exporter['listen_address'] = '0.0.0.0:9100'
gitlab_monitor['listen_address'] = '0.0.0.0'
gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229"
```

The Prometheus configuration file to setup metrics collection from gitlab:

```yaml
# prometheus.yaml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # gitlab monitor
  - job_name: 'gitlab_monitor'
    static_configs:
      - targets: ['gitlab:9168']

  # gitlab sidekiq
  - job_name: 'gitlab_sidekiq'
    metrics_path: /sidekiq
    static_configs:
      - targets: ['gitlab:9168']

  # gitlab process
  - job_name: 'gitlab_process'
    metrics_path: /process
    static_configs:
      - targets: ['gitlab:9168']

  # gitlab pages
  - job_name: 'gitlab_pages'
    static_configs:
      - targets: ['gitlab:9235']

  # gitaly
  - job_name: gitaly
    static_configs:
      - targets: ['gitlab:9236']

  # gitlab workhorse
  - job_name: workhorse
    static_configs:
      - targets: ['gitlab:9229']
```

The Grafana configuration file:

```ini
[database]
path = "/data/grafana.db"

[session]
provider = "redis"
provider_config = "addr=redis:6379,prefix=grafana:"
```

### Services

Now we can start defining the services of our stack. Let's begin with Gitlab itself:

```yaml
# stack.yaml

version: "3.4"

services:
  gitlab:
    image: "gitlab/gitlab-ce:10.3.3-ce.0"
    volumes:
      - "gitlab_data:/var/opt/gitlab"
      - "gitlab_logs:/var/log/gitlab"
      - "gitlab_config:/etc/gitlab"
    ports:
      - "2222:22"
    configs:
      - source: "gitlab.rb"
        target: "/etc/gitlab/gitlab.rb"
    networks:
      - default
      - proxy
    deploy:
      labels:
        traefik.port: "80"
        traefik.frontend.rule: "Host:gitlab.localtest.me"
        traefik.docker.network: "proxy"

volumes:
  gitlab_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/gitlab/data"
  gitlab_logs:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/gitlab/logs"
  gitlab_config:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/gitlab/config"

configs:
  gitlab.rb:
    file: "./gitlab.rb"

networks:
  proxy:
    external: true
```

The most notable thing here is the volumes. [Docker supports mounting NFS volumes using the "local" driver](https://docs.docker.com/engine/reference/commandline/volume_create/#driver-specific-options). The volumes are mounting to `127.0.0.1` because the NFS server is running on the same machine as the my 1 node cluster.

Besides that, we are adding some labels on gitlab service to tell Traefik to use the hostname "gitlab.localtest.me" and port 80 when proxying to this service, and we also needed to tell traefik wich network to use to reach the gitlab service. We needed to set the two networks, "proxy" to be reached with traefik and "default" to copmmunicate with the other services on this stack. We also used a [swarm config](https://docs.docker.com/engine/swarm/configs/), which is like a [swarm secret](https://docs.docker.com/engine/swarm/secrets/), just not encrypted (you can use a secret if you want).

Since we disabled postgres, redis and prometheus in `gitlab.rb`, we need to setup those services too:

```yaml
# stack.yaml

services:
  gitlab:
    # ...
  redis:
    image: "redis:4.0.6-alpine"
  postgres:
    image: "postgres:10.1-alpine"
    volumes:
      - "postgres_data:/data"
    environment:
      POSTGRES_USER: gitlab
      POSTGRES_PASSWORD: gitlab
      PGDATA: /data
      POSTGRES_DB: gitlab
  prometheus:
    image: "prom/prometheus:v2.0.0"
    command: "--config.file=/prometheus.yaml --storage.tsdb.path /data"
    volumes:
      - "prometheus_data:/data"
    configs:
      - prometheus.yaml
    networks:
      - default
      - proxy
    deploy:
      labels:
        traefik.port: 9090
        traefik.frontend.rule: "Host:prometheus.localtest.me"
        traefik.docker.network: "proxy"

volumes:
  # ...
  postgres_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: :/gitlab-swarm/postgres
  prometheus_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/prometheus"
  
configs:
  gitlab.rb: # ...
  prometheus.yaml:
    file: "./prometheus.yaml"
```

Similarly to the gitlab service, we created NFS volumes to the Postgres and Prometheus services and added the Traefik labels and config file to Prometheus service.

Now to the Grafana service:

```yaml
# stack.yaml

services:
  # ...
  grafana:
    image: "grafana/grafana:4.6.3"
    environment:
      GF_PATHS_CONFIG: "/grafana.ini"
    configs:
      - grafana.ini
    volumes:
      - "grafana_data:/data"
    networks:
      - default
      - proxy
    deploy:
      labels:
        traefik.port: 3000
        traefik.frontend.rule: "Host:grafana.localtest.me"
        traefik.docker.network: "proxy"

volumes:
  # ...
  grafana_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/grafana"

configs:
  # ...
  grafana.ini:
    file: "./grafana.ini"
```

Putting it all together we will have stack like this:

```yaml
# stack.yaml

version: "3.4"

services:
  gitlab:
    image: "gitlab/gitlab-ce:10.3.3-ce.0"
    volumes:
      - "gitlab_data:/var/opt/gitlab"
      - "gitlab_logs:/var/log/gitlab"
      - "gitlab_config:/etc/gitlab"
    ports:
      - "2222:22"
    configs:
      - source: "gitlab.rb"
        target: "/etc/gitlab/gitlab.rb"
    networks:
      - default
      - proxy
    deploy:
      labels:
        traefik.port: 80
        traefik.frontend.rule: "Host:gitlab.localtest.me"
        traefik.docker.network: "proxy"
  redis:
    image: "redis:4.0.6-alpine"
  postgres:
    image: "postgres:10.1-alpine"
    volumes:
      - "postgres_data:/data"
    environment:
      POSTGRES_USER: "gitlab"
      POSTGRES_PASSWORD: "gitlab"
      PGDATA: "/data"
      POSTGRES_DB: "gitlab"
  prometheus:
    image: "prom/prometheus:v2.0.0"
    command: "--config.file=/prometheus.yaml --storage.tsdb.path /data"
    volumes:
      - "prometheus_data:/data"
    configs:
      - prometheus.yaml
    networks:
      - default
      - proxy
    deploy:
      labels:
        traefik.port: 9090
        traefik.frontend.rule: "Host:prometheus.localtest.me"
        traefik.docker.network: "proxy"
  grafana:
    image: grafana/grafana:4.6.3
    environment:
      GF_PATHS_CONFIG: "/grafana.ini"
    configs:
      - grafana.ini
    volumes:
      - "grafana_data:/data"
    networks:
      - default
      - proxy
    deploy:
      labels:
        traefik.port: 3000
        traefik.frontend.rule: "Host:grafana.localtest.me"
        traefik.docker.network: "proxy"

volumes:
  gitlab_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/gitlab/data"
  gitlab_logs:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/gitlab/logs"
  gitlab_config:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/gitlab/config"
  postgres_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/postgres"
  prometheus_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/prometheus"
  grafana_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/grafana"

configs:
  gitlab.rb:
    file: "./gitlab.rb"
  prometheus.yaml:
    file: "./prometheus.yaml"
  grafana.ini:
    file: "./grafana.ini"

networks:
  proxy:
    external: true
```

## Deploying the stack

To deploy our stack, we use the docker command line

    docker stack deploy -c stack.yaml gitlab

It may take some time, but when all the services are ready, you can access `localhost:8080` to see the Traefik dashboard:

![Traefik dashboard](https://thepracticaldev.s3.amazonaws.com/i/6jcnoe0l72ip3nvtb7h9.png)

And if you access `gitlab.localtest.me` you get to our Gitlab instance running on Docker Swarm!

![Gitlab first access](https://thepracticaldev.s3.amazonaws.com/i/oz6y875skyw5f5t5dfsm.png)

Going to `grafana.localtest.me`, you can login using "admin" for both username and password, click in "Add datasource" and create a new datasource of type "Prometheus" and with the url as "http://prometheus:9090":

![Grafana add datasource](https://thepracticaldev.s3.amazonaws.com/i/ygvaehik72t4hfj79ueb.png)

Then you got to "Menu (grafana icon) > Dashboards > Import" and import the [Gitlab-Monitor dashboard](https://grafana.com/dashboards/1575) using the datasource we just created and we will have some graphics of gitlab activity:

![Grafana dashboard gitlab monitor](https://thepracticaldev.s3.amazonaws.com/i/efmu0rbcmm8cavn32o3e.png)