How to Setup a Concourse CI Server
Tags: concourse, devops, linux, tutorial
Published: 2019-12-17 21:29:09

![concourse](https://user-images.githubusercontent.com/567298/71035636-8d37b900-2124-11ea-96e2-4bb9fd4c8194.png)

Concourse is a Pipeline Based Continuous Integration System written in Go.

## What is Concourse CI:

Concourse CI is a Continuous Integration Platform. Concourse enables you to construct pipelines with `yaml` configuration that can consist out of 3 core concepts that compose them:

- tasks
- resources
- jobs

For more information about this have a look at their [docs](https://concourse.ci/concepts.html)

## What will we be doing today

We will setup a Concourse Server on Ubuntu 18.04 and run the traditional `Hello, World` pipeline

## Setup the Server:

Concourse requires PostgresSQL for it's database (at least version 9.3) 

```bash
$ apt update && apt upgrade -y
$ apt install postgresql postgresql-contrib -y
$ systemctl enable postgresql
```

Create the Database and User for Concourse on your Postgres database:

```bash
$ sudo -u postgres createuser concourse
$ sudo -u postgres createdb --owner=concourse atc
```

Download the Concourse Server Archive:

```bash
$ export CONCOURSE_VERSION=5.7.2
$ wget https://github.com/concourse/concourse/releases/download/v${CONCOURSE_VERSION}/concourse-${CONCOURSE_VERSION}-linux-amd64.tgz
```

Extract the concourse archive:

```bash
$ tar -zxf concourse-${CONCOURSE_VERSION}-linux-amd64.tgz -C /usr/local
```
 
Create the encryption keys:

```bash
$ mkdir /etc/concourse
$ ssh-keygen -t rsa -q -N '' -f /etc/concourse/tsa_host_key
$ ssh-keygen -t rsa -q -N '' -f /etc/concourse/worker_key
$ ssh-keygen -t rsa -q -N '' -f /etc/concourse/session_signing_key
$ cp /etc/concourse/worker_key.pub /etc/concourse/authorized_worker_keys
```

Concourse Web Process Configuration:

```bash
$ EXT_IP=$(curl -s https://ip.ruan.dev)

$ cat > /etc/concourse/web_environment << EOF
CONCOURSE_ADD_LOCAL_USER=admin:admin
CONCOURSE_SESSION_SIGNING_KEY=/etc/concourse/session_signing_key
CONCOURSE_TSA_HOST_KEY=/etc/concourse/tsa_host_key
CONCOURSE_TSA_AUTHORIZED_KEYS=/etc/concourse/authorized_worker_keys
CONCOURSE_POSTGRES_HOST=127.0.0.1
CONCOURSE_POSTGRES_USER=concourse
CONCOURSE_POSTGRES_PASSWORD=concourse
CONCOURSE_POSTGRES_DATABASE=atc
CONCOURSE_MAIN_TEAM_LOCAL_USER=admin
CONCOURSE_EXTERNAL_URL=http://${EXT_IP}:8080
EOF
```

Concourse Worker Process Configuration:

```bash
$ cat > /etc/concourse/worker_environment << EOF
CONCOURSE_WORK_DIR=/var/lib/concourse
CONCOURSE_TSA_HOST=127.0.0.1:2222
CONCOURSE_TSA_PUBLIC_KEY=/etc/concourse/tsa_host_key.pub
CONCOURSE_TSA_WORKER_PRIVATE_KEY=/etc/concourse/worker_key
EOF
```

Create a Concourse user:

```bash
$ mkdir /var/lib/concourse
$ sudo adduser --system --group concourse
$ sudo chown -R concourse:concourse /etc/concourse /var/lib/concourse
$ sudo chmod 600 /etc/concourse/*_environment
```

Create SystemD Unit Files, first for the Web Service:

```bash
$ cat > /etc/systemd/system/concourse-web.service << EOF
[Unit]
Description=Concourse CI web process (ATC and TSA)
After=postgresql.service

[Service]
User=concourse
Restart=on-failure
EnvironmentFile=/etc/concourse/web_environment
ExecStart=/usr/local/concourse/bin/concourse web

[Install]
WantedBy=multi-user.target
EOF
```

Then the SystemD Unit File for the Worker Service:

```bash
$ cat  > /etc/systemd/system/concourse-worker.service << EOF
[Unit]
Description=Concourse CI worker process
After=concourse-web.service

[Service]
User=root
Restart=on-failure
EnvironmentFile=/etc/concourse/worker_environment
ExecStart=/usr/local/concourse/bin/concourse worker

[Install]
WantedBy=multi-user.target
EOF
```

Create a postgres password for the concourse user:

```bash
$ cd /home/concourse/
$ sudo -u concourse psql atc
atc=> ALTER USER concourse WITH PASSWORD 'concourse';
atc=> \q
```

Start and Enable the Services:

```bash
$ systemctl start concourse-web concourse-worker
$ systemctl enable concourse-web concourse-worker postgresql
$ systemctl status concourse-web concourse-worker

$ systemctl is-active concourse-worker concourse-web
active
active
```

The listening ports should more or less look like the following:

```bash
$ netstat -tulpn

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      11290/postgres
tcp        0      0 127.0.0.1:7776          0.0.0.0:*               LISTEN      13040/concourse
tcp        0      0 127.0.0.1:7777          0.0.0.0:*               LISTEN      13118/gdn
tcp        0      0 127.0.0.1:7787          0.0.0.0:*               LISTEN      13040/concourse
tcp        0      0 127.0.0.1:7788          0.0.0.0:*               LISTEN      13040/concourse
tcp        0      0 127.0.0.1:8079          0.0.0.0:*               LISTEN      13039/concourse
tcp6       0      0 :::41299                :::*                    LISTEN      13039/concourse
tcp6       0      0 :::8888                 :::*                    LISTEN      13040/concourse
tcp6       0      0 ::1:5432                :::*                    LISTEN      11290/postgres
tcp6       0      0 :::35323                :::*                    LISTEN      13039/concourse
tcp6       0      0 :::2222                 :::*                    LISTEN      13039/concourse
tcp6       0      0 :::8080                 :::*                    LISTEN      13039/concourse
udp        0      0 0.0.0.0:54582           0.0.0.0:*                           13118/gdn
```

## Client Side:

When you access the UI on the external ip on port 8080:

![image](https://user-images.githubusercontent.com/567298/71032496-244d4280-211e-11ea-8f28-5b5a0fff7ae0.png)

After logging in, you should see a screen with no pipelines configured:

![image](https://user-images.githubusercontent.com/567298/71032574-4941b580-211e-11ea-958c-c047ac718af7.png)

We require the fly cli client which is used to interact with concourse. On your local machine, download the fly archive, extract the archive and move the fly binary to your path. 

I will by using the client for Mac, for alternate versions have a look at their [releases](https://github.com/concourse/concourse/releases) page

```bash
$ export CONCOURSE_VERSION=5.7.2
$ wget https://github.com/concourse/concourse/releases/download/v${CONCOURSE_VERSION}/fly-${CONCOURSE_VERSION}-darwin-amd64.tgz
$ tar -xf fly-${CONCOURSE_VERSION}-darwin-amd64.tgz
$ sudo mv fly /usr/local/bin/fly
```

Next, we need to setup our Concourse Target by Authenticating against our Concourse Endpoint, lets setup our target with the name `ci`:

```bash
$ fly -t ci login -c http://${EXT_IP}:8080
logging in to team 'main'

navigate to the following URL in your browser:

  http://136.244.102.215:8080/login?fly_port=49722

or enter token manually:
```

Follow the URL and your target should be saved. Lets list our targets:

```bash
$ fly targets
name  url                        team  expiry
ci      http://136.244.102.215:8080  main  Wed, 18 Dec 2019 20:48:02 UTC
```

Listing Registered Workers:

```bash
$ fly -t ci workers
name  containers  platform  tags  team  state    version  age
demo  0           linux     none  none  running  2.2      16m1s
```

Listing Active Containers:

```bash
$ fly -t ci containers
handle  worker  pipeline  job  build #  build id  type  name  attempt
```

## Hello World Pipeline:

Let's create a basic pipeline, that will print out `Hello, World!`:

Our `hello-world.yml`

```yml
jobs:
- name: my-job
  plan:
  - task: say-hello
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: alpine
          tag: edge
      run:
        path: /bin/sh
        args:
        - -c
        - |
          echo "============="
          echo "Hello, World!"
          echo "============="
```

Applying the configuration to our pipeline:

```bash
$ fly -t ci set-pipeline -p yeeehaa -c hello-world.yml
jobs:
  job my-job has been added:
    name: my-job
    plan:
    - task: say-hello
      config:
        platform: linux
        image_resource:
          type: docker-image
          source:
            repository: alpine
            tag: edge
        run:
          path: /bin/sh
          args:
          - -c
          - |
            echo "============="
            echo "Hello, World!"
            echo "============="

apply configuration? [yN]: y
pipeline created!
you can view your pipeline here: http:////136.244.102.215:8080/teams/main/pipelines/yeeehaa

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command
  - click play next to the pipeline in the web ui
```

In the Web-UI we will see the pipeline that we created:

![image](https://user-images.githubusercontent.com/567298/71033463-01239280-2120-11ea-85f6-cc92ba406c50.png)

We can unpause the pipeline in the Web-UI, but since I like to do everything on cli as far as possible, I will unpause the pipeline via cli:

```bash
$ fly -t ci unpause-pipeline -p yeeehaa
unpaused 'yeeehaa'
```

Now our Pipeline is unpaused, but since we did not specify any triggers, we need to manually trigger the pipeline to run, you can either via the WebUI, select your pipeline which in this case will be named `yeeehaa` and then select the job, which will be `my-job` then hit the `+` sign, which will trigger the pipeline.

I will be using the cli:

```bash
$ fly -t ci trigger-job --job yeeehaa/my-job
started yeeehaa/my-job #1
```

Via the WebUI on `http://${EXT_IP}:8080/teams/main/pipelines/yeeehaa/jobs/my-job/builds/1` you should see the `Hello, World!` output.

If you are running Concourse on Ubuntu 18 and you ran into this [issue](https://github.com/concourse/concourse/issues/374):

![image](https://user-images.githubusercontent.com/567298/71034192-8c515800-2121-11ea-92c7-0082c11ccab4.png)

It's a DNS fix that you can apply by doing this:

```
$ sudo apt install resolvconf
$ echo -e "nameserver 8.8.4.4\nnameserver 8.8.8.8" > /etc/resolvconf/resolv.conf.d/head
$ sudo service resolvconf restart
```

We can trigger our pipeline via the cli, we also have the option to see the output, so let's trigger it again, but this time passing the `--watch` flag:

```bash
$ fly -t ci trigger-job --job yeeehaa/my-job --watch
started yeeehaa/my-job #5

initializing
running /bin/sh -c echo "============="
echo "Hello, World!"
echo "============="

=============
Hello, World!
=============
succeeded
```

Listing our Workers and Containers again:

```bash
$ fly -t ci workers
name   containers  platform  tags  team  state    version  age
demo   1           linux     none  none  running  2.2      45m32s

$ fly -t ci containers
handle                                worker            pipeline     job         handle                                worker  pipeline  job     build #  build id  type   name       attempt
e467775c-ab6e-413d-5823-45fc84e2a3fd  demo    yeeehaa   my-job  5        8         task   say-hello  n/a
e5908c49-0ca2-4482-558e-99da76c44a79  demo    none      none    none     none      check  none       n/a
```

## Resources:

If you feel like exploring more, check out their website and my other blogs for more concourse content:

- https://concourse-ci.org/
- https://sysadmins.co.za/tag/concourse/
- https://blog.ruanbekker.com/blog/categories/concourse/

## Thank You

Let me know what you think. If you liked my content, feel free to visit me at [ruan.dev](https://ruan.dev) or follow me on twitter at [@ruanbekker](https://twitter.com/ruanbekker)