Speeding up Go Modules for Docker and CI
Tags: go, modules, docker, ci
Published: 2019-08-05 15:19:53

---
title: Speeding up Go Modules for Docker and CI
published: true
canonical_url: https://evilmartians.com/chronicles/speeding-up-go-modules-for-docker-and-ci
tags: go, modules, docker, ci
---


Finally, the Golang world has a built-in, conventional dependency manager in the ecosystem: Go Modules. What began in Go 1.11 as an opt-in feature has become widely adopted by the community, and we are so close to Go 1.13 when Go Modules will be enabled by default. The delightful dilemma of choosing the “best” tool can be finally resolved.

I can’t help but mention two features which are very close to my heart:


– No more `$GOPATH` imprisonment! In my years of experience, I’d gotten used to storing everything I work on in `~/Projects/` and its subfolders somewhere in the home directory, no matter the programming language. So, being forced to keep my Golang stuff in another specific place and respect SCM url in the path was a real pain, and made routine `cd` operations feel like such a chore. No longer an issue!
– No more vendoring! Dependency updates don’t produce enormous PR diffs to read, and repositories are lighter. I can just remove the `vendor` folder from my source code and forget about it.

The migration to Go Modules is [pretty simple](https://github.com/golang/go/wiki/Modules#how-to-use-modules) and won’t take more than a couple of minutes, especially if you use [any of the supported package managers](https://tip.golang.org/pkg/cmd/go/internal/modconv/?m=all#pkg-variables) to migrate from.

```sh
$ go mod init
$ rm vendor/*
$ go test ./...
$ git add .
$ git commit
```

That’s pretty much it!

With Go Modules your dependencies are not a part of your source code anymore. The toolchain downloads them on its own, keeps modules up-to-date, and caches them locally in `$GOPATH/pkg/mod` for future use. That sounds perfect for when all of your processes occur in a stateful environment like a laptop, but what about stateless builds in your CI pipeline or Docker? Every now and again Go will download every item in your dependencies and waste your priceless time. Let’s fix that with some caching!

# Caching on CI

It’s such a common situation to cache dependencies between builds on CI that some of the services provide a simplified, ecosystem-specific syntax to make it easier. Alas, I haven’t found specific Go Modules caching on popular CIs yet, so let’s do it manually.

If you use TravisCI, it’s very straightforward. Just add those lines to your `.travis.yml` config and you’re all set:

```yaml
cache:
  directories:
    - $GOPATH/pkg/mod
```

Setting up dependency caching on my favorite CircleCI is a little more verbose. Wrap `go mod download` or your build step in the code below. Golang will take care of the missing dependencies and CircleCI will cache them between builds relying on the content of the `go.sum` file.

```yaml
      - restore_cache:
          keys:
            - go-modules-v1-{{ checksum "go.sum" }}
            - go-modules-v1
      # get dependencies here with `go mod download` or implicitly 
      # with `go build` or `go test`
      - save_cache:
          key: go-modules-v1-{{ checksum "go.sum" }}
          paths:
            - "/go/pkg/mod"
```

Here are the results of boosting of my little project on CircleCI:

Before:

```
`go test ./...` => 00:20s
```

After cache warm-up:

```
Restoring Cache => 00:03s
`go test ./...` => 00:06s
Saving Cache    => 00:00s
```

Not bad at all: 2x faster CI build, and for free.


# Caching in Docker

There are two completely different use cases for how we use Docker: for the development process to isolate an application and its environment, and for packing production builds.


## In Development

If you follow Test Driven Development (TDD) caching, Go Modules can significantly increase your development productivity. You definitely know how crucial it is to have as fast a test suite as possible.

Using Docker Compose, cache your modules in a separate volume, and see the performance boost. I saved 20 seconds. Not that bad for a small change!

Here is a minimal `docker-compose.yml`, simplified for the sake of brevity, with highlighted volumes changes:

```yaml
version: '3'
services:
  app:
    image: application:0.0.1-development
    build:
      context: .
      dockerfile: docker/development/Dockerfile
    volumes:
      - .:/app
      - go-modules:/go/pkg/mod # Put modules cache into a separate volume
  runner:
    <<: *app
  test:
    <<: *app
    command: go test ./...
volumes:
  go-modules: # Define the volume
```

## In production

For production builds, we can take advantage of the [layer caching](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#leverage-build-cache) power. Dependencies change less often than the code itself—let’s make it a separate step in your `Dockerfile` before the build phase.

```dockerfile
# `FROM` and other prerequisites here skipped for the sake of brevity
 
# Copy `go.mod` for definitions and `go.sum` to invalidate the next layer
# in case of a change in the dependencies
COPY go.mod go.sum ./
# Download dependencies
RUN go mod download

# `RUN go build ...` and further steps
```

# In a nutshell

Introducing Go Modules was an exciting moment and a significant relief to the Golang community. It brought us a lot of excellent features we had been waiting a long time for. Don’t hesitate to try Modules if you haven’t yet. It’s pretty easy to migrate to it, but don’t forget to change your CI or Docker settings to avoid downloading overhead and keep your builds blazing fast.