A Primer To Go Modules
Tags: go
Published: 2020-04-01 13:00:48

![featured](https://dev-to-uploads.s3.amazonaws.com/i/fjzu5a36twnxxwmxy59l.jpg)*(Image from https://www.youtube.com/watch?v=NCAIhepJgr0)*

Go Modules was introduced in 2018 with Go version 1.11 and with the current (25th of February 2020) version 1.14 is considered ready for production use. It is fairly straight-forward and easy to use. This blog post gives some background on the historical transition to modules, what modules are and the advantages of using it.

# What are Go Packages?

Modules use the existing package management of Go without changing it fundamentally. To better understand them it is therefore necessary to first understand what packages are.

An important and fundamental part of high-quality software is code reuse embodied in the principle “Don't Repeat Yourself” — the DRY principle. You should never repeat the same code over and over again, but instead allow it to be reused as much as possible. This also increases the maintainability of code.

Packages help in organizing related Go source files together into an atomic unit. A package bundles functions and variables together and makes them available to the rest of a bigger project or to completely separate projects. This makes it easier to share code with other projects by allowing them to import and reuse it.

In this way Go encourages you to write small pieces of software components through packages, and compose your applications with these small packages.

Using packages has the following advantages:

* No name conflicts, since functions can have the same name while belonging to different packages.
* Allows reusing and sharing code among projects.
* Organizes related code together.
* Reduces compile times as packages are only re-compiled if their code is changed.

## Declaring a package

A package is not much more than a directory inside your Go workspace with one or more Go source files, or other Go packages.

![golang-package-illustration](https://dev-to-uploads.s3.amazonaws.com/i/uvngbdrnpovtq4zpg704.jpg)*(Image from https://www.callicoder.com/golang-packages/)*

Every Go source file must belong to a package, which is declared at the top of a file using the following syntax:

```golang
package <packagename>
```

All functions and variables defined in this source file then become part of the declared package.

You can export a member of your package by letting its declared name begin with a [capital letter](https://tour.golang.org/basics/3). Only then can other packages import and use it.

## Importing packages

To import packages in Go use the `import` syntax

```golang
import (
  "fmt"
  "time"
  "math"
  "math/rand"
)
```

The last element of the import path is the package name available for your project. For example, the package imported with `math/rand` is simply `rand` and its exported function [`Intn(n int)`](https://golang.org/pkg/math/rand/#Intn) can be invoked with e.g. `rand.Intn(100)`. The package is imported with `math/rand` because it is nested inside the `math` package as a subdirectory.

### Nesting

It is possible to nest a package inside another by creating a subdirectory.

```
$GOPATH
└── github.com
    └── bjarnemagnussen
        └── myproject
            ├── numbers          # Some Package
            └── strings          # Another Package
                └── greet        # Nested Package
                    └── texts.go
```

### The GOPATH

All import paths for non-builtin packages are relative to its `GOPATH`.

```golang
import (
  "github.com/bjarnemagnussen/myproject/numbers"
  "github.com/bjarnemagnussen/myproject/strings"  
  "github.com/bjarnemagnussen/myproject/strings/greet"
)
```

### Getting 3rd Party Packages

This makes it very easy to import third party packages. You can use the `go get` command to download third party packages from remote repositories.

```bash
$ go get -u github.com/jinzhu/gorm
```

The above command fetches the [`gorm`](https://github.com/jinzhu/gorm) package from Github and automatically stores it inside `GOPATH`. To use it import the package to any of the source files in your project with:

```golang
import "github.com/jinzhu/gorm"
```

### Package Alias

What happens if different packages have the same names? Package alias can resolve naming conflicts between packages, or give a short name to an imported package.

```golang
import (
  str "strings"   // Package Alias "str"
)
```

## Vendoring And Versioning

Go packages do not provide any builtin versioning system. The [Go FAQ](https://golang.org/doc/faq#get_version) suggested to create a local copy of a third party package that should stay frozen:

> If you're using an externally supplied package and worry that it might change in unexpected ways, the simplest solution is to copy it to your local repository. (This is the approach Google takes internally.) Store the copy under a new import path that identifies it as a local copy.

Over the years many different tools where proposed and developed around the idea of “vendoring” such as [`vend`](https://medium.com/@forEmil/dead-simple-dependencies-in-go-3b2cb856d200), [`gb`](http://www.getgb.io/) and [`dep`](https://github.com/golang/dep). Around 2016 a discussion arose to [tag versions](https://dave.cheney.net/2016/06/24/gophers-please-tag-your-releases) onto packages and prepare for a time when it would become valuable. The community agreed on `git` release tags with *semantic versioning*.

Semantic versioning divides releases into

* a major (breaking) version,
* a minor (feature) version,
* and a patch version.

The full version is prepended with a single “v” and looks like “v1.2.3”, which translates into a major version 1, with minor version 2 and patch version 3. When there is a bugfix the patch version is increased. If features are added that are backwards compatible the minor version is increased. For changes to the code that remove backwards compatibility the major version is increased.

Version tags are used by vendoring tools such as `dep` to define which commit state of a package to vendor. However, another approach was introduced in 2018 to completely eliminate vendoring and allow for project-based workflows instead of `GOPATH`: Go Modules.

# Go Modules

Today Go modules has replaced `dep` and followed up on the work of yet another dependency manager called [`vgo`](https://research.swtch.com/vgo-tour). Modules is a way of packaging software and is Go's new simplistic dependency management that even allows for reproducible builds.

Imagine you are developing a Go program with a number of dependencies such as package `A`. At the time of writing your program, package `A` works in a set way.

However, what happens when the maintainers of package `A` update their program to fix a bug or extend functionality? You might get lucky and their changes might not impact your application. However, you might be unlucky and these changes break your application.

This is where Go Modules comes in to save the day. By using modules, we can select the precise versions of a dependency that we wish to use and ensure that whenever we build our program, it always uses the specified versions.

## go.mod

A module is just a collection of Go packages stored in a directory with a `go.mod` file at its root. As an example the following go.mod file defines the module `github.com/my/thing`:

```
module github.com/my/thing

require (
    github.com/some/dependency v1.2.3
    github.com/another/dependency/v4 v4.0.0
)
```

There are four directives: `module`, `require`, `replace` and `exclude`. I refer to the excellent [Go Wiki](https://github.com/golang/go/wiki/Modules#modules) page to read about the last two directives `replace` and `exclude`. The `require` directive specifies the versions of the named packages that are used by this module as dependencies and will be revisited in more detail later.

A *module path* is defined with the `module` directive. All packages in a module share the module path as a common prefix to their import paths. In other words, the module path and relative path from the `go.mod` file to a package's directory decides its import path.

For example, if you are creating a module for a repository `github.com/my/thing` with two packages `foo` and `bar`, then the first line of the `go.mod` file would declare the module path with `module github.com/my/thing`. The corresponding directory structure could be:

```
thing
├── bar
|   └── bar.go
├── foo
|   └── foo.go
└── go.mod
```

The import paths for the `foo` and `bar` packages would then be `github.com/my/thing/foo` and `github.com/my/thing/bar`. Hence the package foo would be imported in a Go source file with:

```golang
import "github.com/my/thing/foo"
```

> Importing a module stored locally outside your `GOPATH` from another module can be easily achieved using a VCS (e.g. github.com) for the dependency.

Go can automatically take care of downloading and importing a dependency if it is using a VCS, regardless of where its working directory is stored on disk. Alternatively, the `replace` directive of `go.mod` can be used to point to the dependency's local directory, see [this answer](https://stackoverflow.com/questions/52328952/how-to-use-a-module-that-is-outside-of-gopath-in-another-module) on StackOverflow.

## Version Selection

With Go Modules, adding a new dependency to any of your Go source files will have most Go commands (such as `go build` and `go test`) automatically download the *highest* version of that new package and add it as a direct dependency to the `go.mod` file with the `require` directive.

For example, assume you create a new module from scratch with a `go.mod` file only consisting of:

```
module github.com/my/thing
```

Adding a new import to a dependency `github.com/some/mod`, which latest tagged release version is v1.2.3, and running `go build ./...` will automatically download the dependency and change your `go.mod` file to reflect the new dependency with its latest version:

```
module github.com/my/thing

require (
  github.com/some/mod v1.2.3
)
```

The module `github.com/some/mod` is now a dependency with an allowed version greater than v1.2.3 but less than v2, since the assumed semantic version means that v2 is incompatible with v1.

Go Modules uses a *minimal version selection* for *indirect* dependencies. That means that Go Modules selects the highest of the versions explicitly listed by a require directive in your module or any one of its dependencies. If your module depends on `A` that itself has a `require D v1.0.0` and your module also depends on `B` that has a `require D v1.1.1`, then Go Modules would select v1.1.1 of dependency `D`. This selection of v1.1.1 remains consistent even if some time later a v1.2.0 of `D` becomes available. This is in contrast to `go dep`, which always selects the highest compatible released version under the assumption of semantic versioning.

To see a list of the selected module versions (including indirect dependencies), you can use `go list -m all`.

If later an indirect dependency is removed, Go Modules will still keep track of the “latest not-greatest” version. In other words, if we were to remove from our module the dependency `B` containing a `require D v1.1.1` but keep dependency `A` with a `require D v1.0.0`, then Go Modules would not fallback to v1.0.0 but instead keep v1.1.1 of `D`.

Keeping track of dependency versions is done with a `go.sum` file.

## go.sum

The `go.sum` file is an opaque reliability meta data file and it should not be used to understand your dependencies. Any time your Go Module downloads a version of a dependency not yet seen, that version is tagged inside the `go.sum` file with its corresponding git commit hash acting as a cryptographic checksum.

The `go.sum` file retains checksums for dependency versions even after you stop using that particular dependency or version. This allows validation of the checksums if you later resume using something, providing additional safety. The `go.sum` will therefore frequently have more dependency listed than your `go.mod` file.

**Your module's `go.sum` file should always be committed along with your `go.mod` file.**

Committing `go.sum` has the following benefits:

* If someone clones your repository and downloads your dependencies, they will receive an error if there is any mismatch between their downloaded copies of your dependencies and the corresponding entries in your `go.sum`.
* In addition, `go mod verify` checks that the on-disk cached copies of module downloads still match the entries in `go.sum`.

## Activating And Using Modules

Enough of the theory! Let's see how we can add Go Modules to a project and use it.

![travlling-outside-gopath](https://dev-to-uploads.s3.amazonaws.com/i/a4y9ueodnv22m9v1c6sa.png)*(Image from https://blog.francium.tech/go-modules-go-project-set-up-without-gopath-1ae601a4e868)*

One of the main features of modules is to work on Go projects **outside of** `GOPATH`. We will therefore start by navigating to the root directory of some *imaginary* project not stored in the `GOPATH`

```
$ cd <project path outside $GOPATH/src>         # e.g., cd ~/projects/hello
```

1. Create the initial module definition by automatically writing it to the `go.mod` file with:

```
$ go mod init
```

`go mod init` will often be able to use auxiliary data (such as VCS meta-data) to automatically determine the appropriate module path. But if `go mod init` should state that it cannot automatically determine the module path, you can supply the module path as an optional argument with `go mod init github.com/my/thing`.

2. Build the module with

```
$ go build ./...
```

Building the module will automatically download all necessary dependencies and choose their latest versions. The direct dependencies will show up with the `require` directive inside `go.mod`, while both the direct and indirect dependencies are tagged to their commit hashes inside `go.sum`.

3. It is good practice to run `go mod tidy` before committing code to your repository.

This will ensure that the `go.mod` file contains the dependency requirements for all possible combinations of OS, architecture, and build tags, including for testing purposes. This will help others on your team and your CI/CD environments.

Running `go mod tidy` will also fix any inconsistency reflected by a missing `go.mod` file of a dependency of your module, e.g. because the dependency has not yet opted in to modules itself, or if its `go.mod` file is missing one or more of its dependencies, e.g. because the module author did not run `go mod tidy`. The missing transitive dependencies will be added to your module's requirements along with an `// indirect` comment to indicate that the dependency is not from a direct import within your module.

This behaviour is how modules provide 100% reproducible builds and tests by recording precise dependency information.

## Updating Dependencies

To update a specific dependency inside your module and all its indirect dependency you run as usually

```
$ go get -u github/some/package
```

This will also automatically change both `go.mod` and `go.sum` to reflect the new version(s).

To upgrade *all direct and indirect* dependencies to their latest versions do the following:

* Upgrading to latest minor or patch releases: `go get -u ./...` (and add `-t` to also upgrade test dependencies).
* Upgrading to latest patch releases only: `go get -u=patch ./...` (and add `-t` to also upgrade test dependencies).

## Conclusion

I may have skipped a lot of small details in this post. But the idea behind Go Modules is that it should often be possible to just magically work without configuration. However, knowing some background from where Go Modules comes helps in understanding the design choices. Today Go Modules should be used with *every* new project and can also easily be activated for existing ones.