Ultimate Setup for Your Next Golang Project
Tags: go, docker, tutorial, makefile
Published: 2019-09-29 09:24:24

_Note: This was originally posted at [martinheinz.dev](https://martinheinz.dev/blog/5)_

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/m0bpux6obtyo8js8njqb.png)

For me, the biggest struggle when starting new project has always been trying to set the project up _"perfectly"_. I always try to use the best directory structure so everything is easy to find and imports work nicely, setup all commands so that I'm always one click/command away from desired action, find the best linter, formatter, testing framework for the language/library that I'm using...

The list goes on and it never gets to the point that I'm actually satisfied with the setup... except for this ultimate and best (IMHO) setup for _Golang_!

_Note: This setup works so well partly because it is based on existing projects which can be found [here](https://github.com/thockin/go-build-template) and [here](https://github.com/qiangxue/golang-restful-starter-kit)_.

_TL;DR: Here is my repository - <https://github.com/MartinHeinz/go-project-blueprint>_

## Directory Structure

First of all, let's go over directory structure of our [project](https://github.com/MartinHeinz/go-project-blueprint). There are few top-level files as well as 4 directories:

- `pkg` - Let's start simple - `pkg` is a _Go_ package that contains only global version string. This is substituted for actual version computed from commit hash during build.
- `config` - Next, there is configuration directory which holds files with all necessary environment variables. Any file type can be used, but I recommend _YAML_ files, as they are more readable.
- `build` - This directory contains all shell scripts needed to build and test your application as well as generate reports for code analysis tools.
- `cmd` - Actual source code! By convention, the source directory is named `cmd`, inside there is another one with name of the project - in this case `blueprint`. Next, inside this directory is a `main.go` that runs the whole application, along with it, there are all other source files divided into modules (more on that later).

_Note: From some feedback, I found out, that a lot of people prefer to use `internal` and `pkg` directories to house all their source code. I personally find it unnecessary and redundant, therefore I put everything into `cmd`, but each to their own._

Other than directories, there are also quite a few files and we will talk about those in following sections.

## Go Modules for Perfect Dependency Management

_Go_ projects use wide variety of dependency management strategies. However, since version 1.11 _Go_ has official dependency management solution called [_Go modules_](https://github.com/golang/go/wiki/Modules).
All our dependencies are listed in `go.mod` file, which can be found in root directory. This is how it might look:

```golang
module github.com/MartinHeinz/go-project-blueprint

go 1.12

require (
	github.com/spf13/viper v1.4.0
	github.com/stretchr/testify v1.4.0
)
```

You may ask _"How is the file populated with dependencies?"_. Well, it's pretty simple, all you need is one command:

```shell
go mod vendor
```

This command resets the main module's vendor directory to include all packages needed to build and test all of the module's packages based on the state of the `go.mod` files and Go source code. 

## Actual Source Code and Configuration

Now we're finally getting to source code. As mentioned above, the source code is divided into modules. Each module is a directory in source root. In each module there are source files along with their tests files, e.g.:

```
./cmd/
└── blueprint
    ├── apis <- Module
    │   ├── apis_test.go
    │   ├── user.go
    │   └── user_test.go
    ├── daos <- Module
    │   ├── user.go
    │   └── user_test.go
    ├── services <- Module
    │   ├── user.go
    │   └── user_test.go
    ├── config <- Module
    │       └── config.go
    └── main.go

```

This structure help with readability and maintainability, as it divides code into reasonable chunks, which are easier to traverse. As for the configuration, in this setup I use [Viper](https://github.com/spf13/viper), which is _Go_ configuration library, that can handle various formats, commandline flags, environment variables, etc.
So how do we use it (_Viper_) here? Let's have a look at `config` package:

```golang

var Config appConfig

type appConfig struct {
	// Example Variable, which is loaded in LoadConfig function
	ConfigVar string
}

// LoadConfig loads config from files
func LoadConfig(configPaths ...string) error {
	v := viper.New()
	v.SetConfigName("example")  // <- name of config file
	v.SetConfigType("yaml")
	v.SetEnvPrefix("blueprint")
	v.AutomaticEnv()
	for _, path := range configPaths {
		v.AddConfigPath(path)  // <- // path to look for the config file in
	}
	if err := v.ReadInConfig(); err != nil {
		return fmt.Errorf("failed to read the configuration file: %s", err)
	}
	return v.Unmarshal(&Config)
}

```

This package consist of a single file. It declares one `struct` that holds all config variables and has one function `LoadConfig` which, well, loads config. It takes path to config files, in our case, we would use path to `config` directory which resides in project root and contains our _YAML_ files (mentioned above). And how do we use it? We run it as first thing in `main.go`:

```golang
if err := config.LoadConfig("./config"); err != nil {
    panic(fmt.Errorf("invalid application configuration: %s", err))
}
```

## Simple and Fast Testing

The second most important thing after code itself? Quality tests. To be willing to write lots of good tests, you need a setup that will make it easy for you to do so. To achieve that we will use `Makefile` target called `test`, which collects and runs all tests in `cmd` subdirectories (all files with `_test.go` suffix). These tests are also cached, so they are ran only if there were some changes to relevant code. This is crucial as if the tests are too slow you will (most likely) eventually stop running and maintaining them. Besides unit testing, the `make test` also helps you maintain general code quality, as it also runs `gofmt` and `go vet` with every test run. `go fmt` forces you to format your code properly and `go vet` finds any suspicious code constructs using heuristics. Example output:

```console
foo@bar:~$ make test
Running tests:
ok  	github.com/MartinHeinz/go-project-blueprint/cmd/blueprint	(cached)
?   	github.com/MartinHeinz/go-project-blueprint/cmd/blueprint/config	[no test files]
?   	github.com/MartinHeinz/go-project-blueprint/pkg	[no test files]

Checking gofmt: FAIL - the following files need to be gofmt'ed:
    cmd/blueprint/main.go

Checking go vet: FAIL
# github.com/MartinHeinz/go-project-blueprint/cmd/blueprint
cmd/blueprint/main.go:19:7: assignment copies lock value to l: sync.Mutex

Makefile:157: recipe for target 'test' failed
make: *** [test] Error 1
```

## Always Running in Docker

People often say _"It works on my machine (and not in cloud)..."_, to avoid this we have simple solution - _always_ run in docker container. And when I say _always_ I really mean it - build in container, run in container, test in container. Actually I didn't mention it in previous section, but the `make test` really is _"just"_ `docker run`.

So, how does it work here? Let's start with `Dockerfiles` we have in root of the project - we have two of them one for testing (`test.Dockerfile`) and one running the application (`in.Dockerfile`):

- `test.Dockerfile` - In ideal world, we would have just one Dockerfile for both running and testing the application. However, there might be need for little adjustments in environment for when the tests are ran. That's why we have this image here - to allow us to install additional tools and libraries, in case our tests require it. As an example, let's assume that we have database that we are connecting to. We don't want to spin up whole _PostgreSQL_ server with every test run or be dependent some database running on host machine. So instead, we can use _SQLite_ in-memory database for our test runs. But, guess what? _SQLite_ binary requires [_CGO_](https://blog.golang.org/c-go-cgo). So, what do we do? We just install `gcc` and `g++`, flip the `CGO_ENABLED` flag and we are good to go.

- `in.Dockerfile` - If you look at this `Dockerfile` in the repository, it's just bunch of arguments and copying of config into image - so, what's going on in there? `in.Dockerfile` is used only from `Makefile`, where the arguments are populated, when we run `make container`. Now, it's time to look at the `Makefile` itself, which does all the `docker` stuff for us. &#x1F447;

## Tying it all together with Makefile

For a longest time, `Makefiles` seemed scary to me as I've only seen them used with `C` code, but they are not scary and can be used for so many things, including this project! Let's now explore _targets_ we have in our `Makefile`:

- `make build` - First in the workflow - application build - it builds binary executable in `bin` directory:

```makefile
@echo "making $(OUTBIN)"
  @docker run                                              \ # <- It's just a `docker run`
       -i                                                     \ #    command in disguise  
       --rm                                                   \ # <- Remove container when done
       -u $$(id -u):$$(id -g)                                 \ # <- Use current user
    -v $$(pwd):/src                                        \ # <- Mount source folder
    -w /src                                                \ # <- Set workdir
    -v $$(pwd)/.go/bin/$(OS)_$(ARCH):/go/bin               \ # <- Mount directories where
    -v $$(pwd)/.go/bin/$(OS)_$(ARCH):/go/bin/$(OS)_$(ARCH) \ #    binary will be outputted
    -v $$(pwd)/.go/cache:/.cache                           \
    --env HTTP_PROXY=$(HTTP_PROXY)                         \
    --env HTTPS_PROXY=$(HTTPS_PROXY)                       \
    $(BUILD_IMAGE)                                         \
    /bin/sh -c "                                           \ # <- Run build script
        ARCH=$(ARCH)                                       \ #    (Checks for presence
        OS=$(OS)                                           \ #    of arguments, sets
        VERSION=$(VERSION)                                 \ #    env vars and runs
        ./build/build.sh                                   \ #    `go install`)
    "
  @if ! cmp -s .go/$(OUTBIN) $(OUTBIN); then \ # <- If binaries have changed 
      mv .go/$(OUTBIN) $(OUTBIN);               \ #    move them from `.go` to `bin`
      date >$@;                                 \
  fi
```

- `make test` - Next one is testing - it once again uses `docker run` which is nearly identical, with only difference being the `test.sh` script (only relevant parts):

```shell
TARGETS=$(for d in "$@"; do echo ./$d/...; done)

go test -installsuffix "static" ${TARGETS} 2>&1

ERRS=$(find "$@" -type f -name \*.go | xargs gofmt -l 2>&1 || true)

ERRS=$(go vet ${TARGETS} 2>&1 || true)
```

The lines above are the important part of the file. First of them collects testing targets using path given as parameter. Second line runs the tests and prints output to std out. Remaining two lines run `go fmt` and `go vet` respectively, both collecting errors (if there are any) and printing them.

- `make container` - Now, the most important part - creating container that can be deployed:

```makefile
.container-$(DOTFILE_IMAGE): bin/$(OS)_$(ARCH)/$(BIN) in.Dockerfile
	@sed                                 \
	    -e 's|{ARG_BIN}|$(BIN)|g'        \
	    -e 's|{ARG_ARCH}|$(ARCH)|g'      \
	    -e 's|{ARG_OS}|$(OS)|g'          \
	    -e 's|{ARG_FROM}|$(BASEIMAGE)|g' \
	    in.Dockerfile > .dockerfile-$(OS)_$(ARCH)
	@docker build -t $(IMAGE):$(TAG) -t $(IMAGE):latest -f .dockerfile-$(OS)_$(ARCH) .
	@docker images -q $(IMAGE):$(TAG) > $@
```

Code for this target is pretty simple, it first substitutes variables in `in.Dockerfile` and then runs `docker build` to produce image with both _"dirty"_ and _"latest"_ tags. Finally it prints container name to standard output.

- `make push` - Next, when we have image, we need to store it somewhere, right? So, all that `make push` does is push image to _Docker_ registry.
- `make ci` - Another good use for `Makefile` is to leverage it inside our CI/CD pipeline (next section). This target is very similar to `make test` - it also runs all the tests, but on top of that, it also generates coverage reports which are then used as an input to code analysis tools.
- `make clean` - Lastly, if we want to clean-up our project, we can run `make clean`, which removes all files generated by previous targets.

I will omit the remaining ones as they are not needed for normal workflow or are just part of other _targets_.
    
## CI/CD for Ultimate Coding Experience

Last, but definitely not the least - _CI/CD_. With such a nice setup (if I say so myself), it would be a shame to omit some fancy pipeline, that can do tons of stuff for us, right? I won't go into too much detail about what is in the pipeline, because you can check it out yourself [here](https://github.com/MartinHeinz/go-project-blueprint/blob/master/.travis.yml) (I also included comments for pretty much every line, so everything is explained), but I want to point out few things:

This _Travis_ build uses _Matrix Build_ with 4 parallel jobs to speed up whole process
- The 4 parts (jobs) here are:
    - _Build and Test_ where we verify that application works as expected
    - _SonarCloud_ where we generate coverage reports and send them to _SonarCloud_ server
    - _CodeClimate_ - here, again as in previous one - we generate reports and send them, this time to _CodeClimate_ using their test reporter
    - _Push to Registry_ - finally, we push our container to _GitHub Registry_ (stay tuned for blog post on that!)
    
## Conclusion

I hope this post will help you in your future _Go_ coding adventures. If you want to see more details, go ahead and checkout the repository [here](https://github.com/MartinHeinz/go-project-blueprint). Also, if you have any feedback or ideas for improvements, don't hesitate and submit issue, fork the repo or just give a star, so I know it makes sense to work on it little more. &#x1F642;

In the next part we will look at how you can extend this blueprint to easily build RESTful APIs, test with in-memory database and setup swagger documentation (you can have a sneak peek in [`rest-api` branch in the repository](https://github.com/MartinHeinz/go-project-blueprint/tree/rest-api)).