How to setup Github Actions for Go + Postgres to run automated tests
Tags: go, github, devops, tutorial
Published: 2020-09-25 22:45:16

**Continuous integration (CI)** is one important part of the software development process where a shared code repository is continuously changing due to new work of a team member being integrated into it.

To ensure the high quality of the code and reduce potential errors, each integration is usually verified by an automated build and test process.

In this article, we will learn how to setup that process using **Github Action** to automatically build and run unit tests for our simple bank project, which is written in **Golang** and uses **PostgreSQL** as its main database.

{% youtube 3mzQRJY1GVE %}

Here's:

- Link to the [full series playlist on Youtube](https://bit.ly/backendmaster)
- And its [Github repository](https://github.com/techschool/simplebank)

## How Github Actions works

**Github Action** is a service offered by **Github** that has similar functionality as other CI tools like **Jenkins**, **Travis**, or **CircleCI**.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/erqirnwe4sl9eaapqu8k.png)

### Workflow

In order to use **Github Actions**, we must define a **workflow**. Workflow is basically an automated procedure that’s made up of one or more jobs. It can be triggered by 3 different ways:

- By an event that happens on the Github repository
- By setting a repetitive schedule
- Or manually clicking on the run workflow button on the repository UI.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/xj8xw9etxbztuz60mrho.png)

To create a workflow, we just need to add a `.yml` file to the `.github/workflows` folder in our repository. For example, this is a simple workflow file `ci.yml`:

```yaml
name: build-and-test

on:
  push:
    branches: [ master ]
  schedule:
    - cron:  '*/15 * * * *'

jobs:
  build:
    runs-on: ubuntu-latest
```

The name of this workflow is `build-and-test`. We can define how it will be triggered using the `on` keyword.

In this flow, there's an event that will trigger the workflow whenever a change is pushed to the master branch, and another scheduled trigger that will run the workflow every 15 minute.

Then we define the list of jobs to run in the `jobs` section of the workflow yaml file.

### Runner

In order to run the jobs, we must specify a runner for each of them. A runner is simply a server that listens for available jobs, and it will run only 1 job at a time.

We can use Github hosted runner directly, or specify our own self-hosted runner.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/8se40pvo751as9a9vef1.png)

The runners will run the jobs, then report the their progress, logs, and results back to Github, so we can easily check it on the UI of the repository.

We use the `run-on` keyword to specify the runner we want to use.

```yaml
jobs:
  build:
    runs-on: ubuntu-latest
```

In this example workflow, we’re using Github’s hosted runner for Ubuntu’s latest version.

### Job

Now let’s talk about **Job**. A job is a set of steps that will be executed on the same runner.

Normally all jobs in the workflow run in parallel, except when you have some jobs that depend on each other, then they will be run serially.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/aqzjfdtte7mlqsj5s1q1.png)

The jobs are listed inside the workflow under the `jobs` keyword.

```yaml
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v2
      - name: Build server
        run: ./build_server.sh
  test:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - run: ./test_server.sh
```

In this example, we have 2 jobs:

- The first one is `build`, which has 2 steps: check out code, and build server.
- The second job is `test`, which will run the tests of the application.

Here we use the `needs` keyword to say that the `test` job depends on the `build` job, so that it can only be run after our application is successfully built.

This `test` job only has 1 step that runs the `test_server.sh` script.

### Step

Steps are individual tasks that run serially, one after another within a job. A step can contain 1 or multiple actions.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/mkf84x65vk7wm0e8pkwh.png)

Action is basically a standalone command like the one that run the `test_server.sh` script that we’ve seen before. If a step contains multiple actions, they will be run serially.

An interesting thing about action is that it can be reused. So if someone has already written a github action that we need, we can actually use it in our workflow.

Let’s take a look at this example.

```yaml
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v2
      - name: Build server
        run: ./build_server.sh
```

Here we use the `steps` keyword to list out all steps we want to run in our job.

The first step is to check out the code from Github to our runner machine. To do that, we just use the Github actions `checkout@v2`, which has already been written by the Github action team.

The second step is to build our application server. In this case, we provide our own action, which is simply running the `build_server.sh` script that we’ve created in the repository.

And that’s it!

### Summary

Before jumping in to coding, let’s do a quick summary:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/sljvhybb3b32cgi3fy3e.png)

- We can trigger a workflow by 3 ways: event, scheduled, or manually.
- A workflow consists of one or multiple jobs.
- A job is composed of multiple steps.
- Each step can have 1 or more actions.
- All jobs inside a workflow normally run in parallel, unless they depend on each other, then in that case, they run serially.
- Each job will be run separately by a specific runner.
- The runners will report progress, logs, and results of the jobs back to github. And we can check them directly on Github repository’s UI.

## Setup a workflow for Golang and Postgres

Alright, now let’s learn how to setup a real workflow for our Golang application so that it can connect to Postgres, and run all the unit tests that we’ve written in previous lectures whenever new changes are pushed to Github.

### Use a template workflow

In our [simple bank repository](https://github.com/techschool/simplebank) on Github, let’s select the **Actions** tab.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ulep0p4w85q6k2pwh3cw.png)

Github knows that our project is written mainly in `Go`, so it suggests us to setup the workflow for `Go`. Let’s click this setup button.

As you can see, a new file `go.yml` is being created under the folder `.github/workflows` of our repository with this template:

```yaml
name: Go

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:

  build:
    name: Build
    runs-on: ubuntu-latest
    steps:

    - name: Set up Go 1.x
      uses: actions/setup-go@v2
      with:
        go-version: ^1.13
      id: go

    - name: Check out code into the Go module directory
      uses: actions/checkout@v2

    - name: Get dependencies
      run: |
        go get -v -t -d ./...
        if [ -f Gopkg.toml ]; then
            curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
            dep ensure
        fi

    - name: Build
      run: go build -v .

    - name: Test
      run: go test -v .

```

We can edit this file directly here using this Github editor. However, I prefer to add the file to our local repository first, then edit it locally with visual studio code before pushing to Github.

### Create a workflow yaml file

So let’s open our simple bank project folder in the terminal. I’m gonna create a new folder `.github/workflows`.

```bash
❯ cd ~/Projects/techschool/simplebank
❯ mkdir -p .github/workflows
```

Then create a new Yaml file for our workflow inside this folder. You can name it whatever you want, just make sure it has `yml` extension. For me, I’m just gonna use `ci.yml` to be simple.

```bash
❯ touch .github/workflows/ci.yml 
```

Now let’s open this project in visual studio code.

Here we can see the `ci.yml` file under `.github/workflows` folder. Let’s go back to **Github** and copy the `go.yml` file content, then paste it to our `ci.yml` file.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/o8u3fz94nkj4hlduaui3.png)

First we need to set a name for this workflow, for example: `ci-test`. This name will be displayed in our **Github** repository’s **Actions** page.

```yaml
name: ci-test
```

### Config trigger events

Then we define the events that can trigger this workflow. Normally we would want to run tests whenever there’s a change being pushed to the master branch, or when there’s a pull request to merge into the master branch.

```yaml
on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]
```

There are many other events that you can use. Please refer to the [Github Actions documentation](https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows) to know more about them.

### Setup jobs

Next we’re gonna setup the jobs. In this template that Github provides us, we have only 1 job.

Its name is `build`, and it runs on a Ubuntu runner. I think we should rename this job to `test` because that’s the main purpose of it.

```yaml
jobs:

  test:
    name: Test
    runs-on: ubuntu-latest
```

There are several steps in this job.

- Step 1: Install Go

    The first step is to setup or install Go into the runner. In this step, we just need to use the existing Github action called `setup-go@v2`.

    ```yaml
        steps:

        - name: Set up Go 1.x
        uses: actions/setup-go@v2
        with:
            go-version: ^1.15
        id: go
    ```

    We use the `with` keyword to provide input parameters to this action. In this case, we can ask it to use a specific version of Go, such as version `1.15`.

    The `id` field is just a unique identifier of this step. We might need it if we want to refer to this step in other context.

- Step 2: Checkout code

    The second step is to check out the code of this repository into the runner. To do that, we also reuse an existing action: `checkout@v2`.

    ```yaml
        - name: Check out code into the Go module directory
        uses: actions/checkout@v2
    ```

- Step 3: Get dependencies

    The next step is to get all the dependencies, or external packages that our project is using.

    ```yaml
        - name: Get dependencies
        run: |
            go get -v -t -d ./...
            if [ -f Gopkg.toml ]; then
                curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
                dep ensure
            fi
    ```

    In fact, we don’t need this step because go mod will automatically download missing libraries when we build the application or run the tests. So let's remove it!

    The build step is also not necessary because the application will be built automatically when we run go test.

    ```yaml
        - name: Build
        run: go build -v .
    ```

- Step 4: Run the tests

    So the last remaining step is to run our unit tests. We already have a `make test` command defined in the Makefile for this purpose. Therefore, all we have to do in this step is to call it:

    ```yaml
        - name: Test
        run: make test
    ```

### Push the workflow to Github

So we’re done with the first basic version of our CI workflow:

```yaml
name: ci-test

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:

  test:
    name: Test
    runs-on: ubuntu-latest

    steps:

    - name: Set up Go 1.x
      uses: actions/setup-go@v2
      with:
        go-version: ^1.15
      id: go

    - name: Check out code into the Go module directory
      uses: actions/checkout@v2

    - name: Test
      run: make test
```

It might not work yet because we haven’t setup the Postgres database. But let’s just push it to Github to see how it run:

```bash
❯ git status
❯ git add .
❯ git commit -m "init CI workflow"
```

Here we first run `git status` to check the status of our local repository. After that, we run `git add .` to add all new changes to the list of our commit.

Then run `git commit` with a message saying `init CI workflow` to commit it to our local repository. And finally, we run `git push origin master` to push this change to our remote repository on **Github**.

Now let's go back to our **Github** repository page and select **Actions** tab.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/cvfxenpybifrxjsi245k.png)

Now we can see our `ci-test` workflow here, and there’s a new run of it for our commit. When we open this run, we can see 1 job in progress: `Test`.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/xpdrymscd6ps8tmxtqlv.png)

All steps are listed on the right. The `Setup job`, `Setup Go`, and `Checkout code` steps are finished successfully because there’s a green tick in front of them. The `Test` step is still running because there’s a yellow circle before it.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/e4b89wg0qbvpucqj0eyg.png)

Now the `Test` step has finished, but it failed. We know that because of the red `x` icon next to it. 

This is expected, because as we’re seeing in the logs, the code cannot connect to port `5432` of Postgres, since we haven’t set it up in our workflow yet. So let’s do that now!

### Add Postgres service

Let’s search for `github action postgres`, and open this [official Github Action documentation page](https://docs.github.com/en/free-pro-team@latest/actions/guides/creating-postgresql-service-containers) about creating Postgres service containers.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/pda3tmedux9sp8nz128x.png)

Here in this section, we can see that Postgres is declared as an external service of this job. Let’s copy this block of code and paste it to our workflow file.

So we use the `services` keyword to specify a list of external services that we want to run together with our job. In this case, we only need 1 service, which is Postgres.

And since we’re using Postgres version 12 in our project, let’s set this docker image name to `postgres:12`. You can check out available versions and tags of this Postgres image on [Docker Hub](https://hub.docker.com/_/postgres).

```yaml
    services:
      postgres:
        image: postgres:12
        env:
          POSTGRES_USER: root
          POSTGRES_PASSWORD: secret
          POSTGRES_DB: simple_bank
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
```

Next we need to set some environment variables for the credentials to access the database.

If you still remember, we’re using `user = "root"`, `password = "secret"`, and `database = "simple_bank"` in our local Postgres container. So let’s set the same value here for our CI workflow.

The health check option is very important because it is used by the runner to check if **Postgres** has started successfully or not, so that it can know when to run the next steps in the workflow.

That’s great because we only want our tests to be run after **Postgres** is started. Otherwise, the tests will still fail because it cannot connect to the database, right?

### Add run migrations step

OK, now the Postgres service is defined, but in order for our tests to run successfully, we also need to run db migrations to create the correct database schema for our application.

So let’s define a new step here, after the check out code step. Its name will be `Run migrations`. And the only action it needs to do is to run `make migrateup`.

```yaml
    - name: Run migrations
      run: make migrateup
```

Alright, now let’s try to push this new workflow changes to Github to see what will happen.

OK, now in our repository’s Actions page, we can see a new run for our new commit. Here the Test job is still running. The job is set up successfully, and now it’s initializing the containers.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/et55klrcfdh5pvfosuby.png)

From the logs, we know that it’s still waiting for Postgres service to be ready. As soon as Postgres is up, all following steps are run immediately.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/k84v6mphh41y84itkui2.png)

Here we can see a log saying `Postgres service is healthy`. The Setup Go step is also successful. Then it checkout new the code.

Now the migrations step is failing because migrate is not found. We forgot to install the `golang-migrate` CLI tool to run the migrations.

### Install golang-migrate CLI

So let’s search for `golang migrate`, and open this [Github page documentation](https://github.com/golang-migrate/migrate/tree/master/cmd/migrate).

There are several options depending on the OS that you use. We’re using Ubuntu for our runner, so I’m gonna copy this curl command to download a pre-built binary of the migrate CLI.

```bash
❯ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/migrate.$platform-amd64.tar.gz | tar xvz
```

Now in the workflow, let’s add a new Step to `Install golang migrate`. Then in the run action, let’s paste in the curl command.

We have to set the correct URL for the version of migrate CLI and the platform that we want to use. So let’s click on this [Release downloads link](https://github.com/golang-migrate/migrate/releases).

The latest release is version `4.12.2`. And since our ubuntu runner is a linux platform, let’s copy the `migrate.linux-amd64.tar.gz` link address, then paste it to our curl command.

```yaml
    - name: Install golang-migrate
      run: curl -L https://github.com/golang-migrate/migrate/releases/download/v4.12.2/migrate.linux-amd64.tar.gz | tar xvz
```

It will download the zip file, and unzip it to give us the migrate binary named `migrate.linux-amd64`. Now in order for the migrate command to work, we have to move that binary to the `/usr/bin` folder.

So this step will include more than just 1 curl command. We use this vertical pipe `|` character here to specify a multi-line commands. Let's add this move command to the step:

```yaml
    - name: Install golang-migrate
      run: |
        curl -L https://github.com/golang-migrate/migrate/releases/download/v4.12.2/migrate.linux-amd64.tar.gz | tar xvz
        sudo mv migrate.linux-amd64 /usr/bin/
        which migrate
```

Note that only a superuser can change the content of the `/usr/bin` folder, so we have to run this command with `sudo`.

We also add 1 more command: `which migrate`, just to check if the migrate CLI binary is successfully installed and ready to be used in the runner or not.

Now let's commit the new change of our workflow and push it to **Github**. Then check our repository’s Actions page.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/bdi2gb8j9d697923jur2.png)

Now the job is still failing, but this time it fails at the `Install golang-migrate step`.

From the logs, we can say that the binary file was successfully downloaded. So it might fail because of the move command, or the which migrate command.

OK I know why! That’s because we’re just moving the file `migrate.linux-amd64` to `/usr/bin`, but we don’t rename it to `migrate`. So when we run `which migrate`, it cannot find any binary with that name.

All we have to do now is to add `migrate` to the end of the move command, so that the binary file is moved to `/usr/bin` with a new name: `migrate`.

```yaml
    - name: Install golang-migrate
      run: |
        curl -L https://github.com/golang-migrate/migrate/releases/download/v4.12.2/migrate.linux-amd64.tar.gz | tar xvz
        sudo mv migrate.linux-amd64 /usr/bin/migrate
        which migrate
```

This will ensure that when the `make migrateup` command is run, the correct `migrate` CLI binary will be used.

Alright, let’s add this new change, commit it, and push it to Github. Then go back to our Repository’s Action page to check the job’s status.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/iwmy4tdno5ptznj4be54.png)

It's still failing. However, this time, the `Install golang-migrate` step is successful. The step that fails is `Run migrations`.

And the reason is: it still cannot connect to port `5432` of our Postgres container. Why? We’ve already added **Postgres** to the services list right?

Well, yes! But we haven’t exposed its local port to the external host yet. That’s why our code still cannot connect to the port.

### Add port mapping to Postgres service

We can use the `ports` keyword to specify the ports that we want to expose to the external host, just like what we normally do in our docker compose file. Let’s add it to our CI workflow.

```yaml
    services:
      postgres:
        image: postgres:12
        env:
          POSTGRES_USER: root
          POSTGRES_PASSWORD: secret
          POSTGRES_DB: simple_bank
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
```

The default port `5432` of Postgres is now available to our job runner to access.

Let’s add this new change, commit it, and push the change to Github. Hopefully this time it will work.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/qarzhc6uetx92q4108cw.png)

It’s done! All green ticks. So finally our CI-test workflow runs successfully.

After all steps in our workflow are completed, Github do some clean up steps and stop the containers.

And that’s it! We have learned about continuous integration by writing our first Github Action workflow  to run Golang unit tests that need to connect to an external Postgres service.

Here's the complete workflow file `.github/workflows/ci.yml`:

```yaml
name: ci-test

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:

  test:
    name: Test
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:12
        env:
          POSTGRES_USER: root
          POSTGRES_PASSWORD: secret
          POSTGRES_DB: simple_bank
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:

    - name: Set up Go 1.x
      uses: actions/setup-go@v2
      with:
        go-version: ^1.15
      id: go

    - name: Check out code into the Go module directory
      uses: actions/checkout@v2

    - name: Install golang-migrate
      run: |
        curl -L https://github.com/golang-migrate/migrate/releases/download/v4.12.2/migrate.linux-amd64.tar.gz | tar xvz
        sudo mv migrate.linux-amd64 /usr/bin/migrate
        which migrate

    - name: Run migrations
      run: make migrateup

    - name: Test
      run: make test

```

There are a lot of more things that Github Action can do. I encourage you to check out its [official documentation](https://docs.github.com/en/free-pro-team@latest/actions) to learn more about them.

And that brings us to the end of this article. Thanks a lot for reading, and I will see you guys in the next lecture!

---

If you like the article, please [subscribe to our Youtube channel](http://bit.ly/techschoolguru) and [follow us on Twitter](https://twitter.com/techschoolguru) for more tutorials in the future.

---
If you want to join me on my current amazing team at Voodoo, check out [our job openings here](https://bit.ly/voodoojobs). Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.