Evaluating Go's Package Management and Module Systems
Tags: go, beginners
Published: 2020-07-08 16:39:49

*This article was originally written by [Ayooluwa Isaiah][1] on the [Honeybadger Developer Blog][2].*  

A package manager is a tool for automating the process of building your code, as well as downloading, updating, and removing project dependencies in a consistent manner. It can determine whether a specific version of a package is installed on a project, and then install or upgrade the package, typically from a remote host.

Package managers have been around for a long time. They were first used in operating systems and later used in programming environments. While there are differences between system-level and language-level package managers, there is also significant overlap in how they work.

In a language context, a package manager makes it easier to work with first-party and third-party libraries by helping you define and download project dependencies and pin down versions or version ranges so that you can, in theory, upgrade your dependencies without fear of breaking things. Most package managers also support a lock file through which they guarantee the reproducibility of builds in any environment.

Today, it is one of the most important things to consider when deciding whether to adopt a new programming language. This is evidenced by the fact that most mainstream languages have some sort of standard package management solution. For example, Ruby has RubyGems, Node.js has npm, and Rust has Cargo.

```bash
$ gem install dotenv
```

For a long time, the package management strategy in Go was inadequate, as there was no official way to version dependencies. Although third-party solutions did exist, integration with the language wasn't seamless, and adoption varied from project to project.

In 2018, the Go team finally introduced [Go modules](https://github.com/golang/go/wiki/Modules) with the aim of plugging this gaping hole in the ecosystem. The feature initially landed in Go 1.11 but wasn't enabled by default. To turn it on, you had to set the `$GO111MODULE` environmental variable to `on`. As of Go 1.14, the latest version at the time of writing, modules are now enabled by default and continue to see increasing adoption in the community.

This post will walk you through how Go modules work. We'll consider all the basic use cases where the knowledge of modules will help you get things done faster and help save hours of trial and error.

## The GOPATH

Before Go modules were introduced, projects had to be created inside the `$GOPATH`, which is an environmental variable that points to the directory where your Go workspace exists. This workspace is where Go manages your project files, dependencies, and installed binaries. The `GOPATH` was assumed to be `$HOME/go` on Unix systems and `%USERPROFILE%\go` on Windows by default.

Usage of `$GOPATH` mechanics proved to be inflexible and introduced a bit of a learning curve for Go beginners when it comes to setting up a development environment and understanding how the compiler manages dependencies for a project. In addition, versioning packages was not officially supported in the language (the way it is uses a `Gemfile` for a Ruby project, for example).

As of Go v1.13, `$GOPATH` mechanics is now largely irrelevant. While third-party dependencies are still placed in the `GOPATH` by default, you are now free to create a project anywhere in your filesystem, and vendoring and package versioning are now fully supported with the `go` tool.

In the next sections, we'll discuss how you can start using modules in your project and all the common use cases that you are likely to encounter. Make sure you have the [latest version](https://golang.org/dl/) of Go installed before proceeding with the rest of this article.

## Getting started with Modules

To initialize a project using Modules, enter the command below at the project root:

```bash
$ go mod init <module name>
```

The module name doubles as the import path, which allows internal imports to be resolved inside the module. It's also how other projects will import your package (if you're developing a library, for example). Idiomatically, this will be the URL of the repository hosting the code. Note that you don't need to have your project checked into version control or pushed to a remote repository before specifying a module name.

Suppose your project is called `example`; you can utilize modules in your project using the command below:

```bash
$ go mod init github.com/ayoisaiah/example
go: creating new go.mod: module github.com/ayoisaiah/example
```

The above command will create a `go.mod` file in the root of your project directory. This will contain the import path for the project and the Go version information. It's the `Gemfile` equivalent for Go.

```bash
$ cat go.mod
module github.com/ayoisaiah/example

go 1.14
```

## Installing dependencies

One of the main reasons Go modules were introduced was to make dependency management a lot easier. Adding a dependency to your project can be done using the `go get` command just as before:

```bash
$ go get github.com/joho/godotenv
```

You can target a specific branch:

```bash
$ go get github.com/joho/godotenv@master
```

Or a specific version:

```bash
$ go get github.com/joho/godotenv@v1.2.0
```

Or even a specific commit:

```bash
$ go get github.com/joho/godotenv@d6ee687
```

Your `go.mod` file should look like this now:

```bash
$ cat go.mod
module github.com/ayoisaiah/example

go 1.14

require github.com/joho/godotenv v1.3.1-0.20200301204615-d6ee6871f21d // indirect
```

The `// indirect` comment indicates that this package is not currently being used in the project. You may also see this comment when a package is an indirect dependency (that is a dependency of another dependency).

You can import and use the newly installed `godotenv` package by specifying its import path and using one of its exported methods.

```go
// main.go
package main

import (
	"github.com/joho/godotenv"
)

func main() {
	godotenv.Load()
}
```

At this point, you can run the `go mod tidy` command in your terminal to update the `go.mod` file. This command will remove unused dependencies in your project and add any missing ones (for example, if you import a third-party package to your project without fetching it first with `go get`).

Before releasing a new version of your project, and before each commit, you should run the `go mod tidy` command to ensure your module file is clean and accurate. This file possesses all the required information necessary for reproducible builds.

At this point, there also should be a `go.sum` file in your project root. It's not a lock file (like `Gemfile.lock`) but is maintained for the purpose of containing the expected [cryptographic hashes](https://golang.org/cmd/go/#hdr-Module_downloading_and_verification) of the content of specific module versions. You can think of it as additional verification to ensure that the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons.

```bash
$ cat go.sum
github.com/joho/godotenv v1.3.1-0.20200301204615-d6ee6871f21d h1:LRaxUhLYBFLUpSZk7X173VtzdRwPtu7HSs6avaT7lbU=
github.com/joho/godotenv v1.3.1-0.20200301204615-d6ee6871f21d/go.mod h1:7hK45KPybAkOC6peb+G5yklZfMxEjkZhHbwpqxOKXbg=
```

The recorded checksums are retained even if you stop using the module so that you can pick up right where you left off if you start using it again later. For this reason, make sure both your `go.mod` and `go.sum` files are checked into version control.

All downloaded modules are cached locally in your `$GOPATH/pkg/mod` directory by default. If you import a package to your project without downloading it first using `go get`, the latest tagged version of the module providing that package will be installed automatically and added to your `go.mod` file when you run `go build` or `go test` before your project is compiled.

You can see this in action by adding a new dependency to your project, such as this [color](https://github.com/gookit/color) package, and using it as shown below:

```go
package main

import (
	"github.com/joho/godotenv"
	"gopkg.in/gookit/color.v1"
)

func main() {
	godotenv.Load()

	color.Red.Println("This is color red!")
}
```

Running `go build` will fetch the package and add it to your `go.mod` file:

```bash
$ go build
go: finding module for package gopkg.in/gookit/color.v1
go: found gopkg.in/gookit/color.v1 in gopkg.in/gookit/color.v1 v1.1.6
$ cat go.mod
module github.com/ayoisaiah/example

go 1.14

require (
	github.com/joho/godotenv v1.3.1-0.20200301204615-d6ee6871f21d
	gopkg.in/gookit/color.v1 v1.1.6
)
```

## Updating dependencies

Go modules use the [Semantic Versioning](https://semver.org/) (Semver) system for versioning, which has three parts: major, minor, and patch. A package in version `1.2.3` has 1 as its major version, 2 as its minor version, and 3 as its patch version.

### Minor or patch versions

You can use `go get -u` or `go get -u=patch` to upgrade a package to its latest minor or patch version, respectively, but you can't do this for major version upgrades. This is because major version updates have different semantics for how they are published and maintained.

```bash
$ go get -u gopkg.in/gookit/color.v1
go: gopkg.in/gookit/color.v1 upgrade => v1.1.6
```

### Major versions

The convention for code opting into Go modules is to use a different module path for each new major version. Starting at `v2`, the path must end in the major version. For example, if the developer of `gotdotenv` makes a major version release, the module path will change to `[github.com/joho/godotenv/v2](http://github.com/joho/godotenv/v2)`, which is how you'll be able to upgrade to it. The original module path (`github.com/joho/godotenv`) will continue to refer to `v1` of the package.

For example, let's say we're building a CLI app using `v1` of this [cli package](https://github.com/urfave/cli/) in our project:

```go
package main

import (
	"os"

	"github.com/urfave/cli"
)

func main() {
	(&cli.App{}).Run(os.Args)
}
```

After building the project and running `go mod tidy`, our `go.mod` file should look like this:

```bash
$ go build
go: finding module for package github.com/urfave/cli
go: found github.com/urfave/cli in github.com/urfave/cli v1.22.4
$ go mod tidy
$ cat go.mod
module github.com/ayoisaiah/example

go 1.14

require github.com/urfave/cli v1.22.4
```

Let's say we want to upgrade to `v2` of the package. All you need to do is replace the `v1` import path with the `v2` import path, as shown below. You should obviously read the documentation for the package you're upgrading, so any necessary changes can be made to the code. In this example, the code stays the same.

```go
package main

import (
	"os"

	"github.com/urfave/cli/v2"
)

func main() {
	(&cli.App{}).Run(os.Args)
}
```

Running `go build` again will add the `v2` package to our import path, alongside `v1`:

```bash
$ cat go.mod
module github.com/ayoisaiah/example

go 1.14

require (
	github.com/urfave/cli v1.22.4
	github.com/urfave/cli/v2 v2.2.0
)
```

If you've completed the migration successfully (and all tests pass), you can run `go mod tidy` to clean up the now unused `v1` dependency from your `go.mod` file. This convention of using different module paths for major versions is known as [semantic import versions](https://research.swtch.com/vgo-import). Due to this convention, it's possible to use multiple versions of a package simultaneously, such as when performing an incremental migration in a large codebase.

## Removing dependencies

To delete a dependency from your project, all you need to do is remove all references to the package from your project, and then run `go mod tidy` on the command line to clean up your `go.mod` file. Remember that the cryptographic hash of a package's content will be retained in your `go.sum` file even after the package is removed.

## Vendoring dependencies

As mentioned earlier, all downloaded dependencies for a project are placed in the `$GOPATH/pkg/mod` directory by default. Vendoring is the act of making a copy of the third-party packages your project depends on and placing them in a `vendor` directory within your project. This is one way to ensure the stability of your production builds without having to rely on external services.

Here are some other benefits of vendoring:

- You'll be able to use `git diff` to see the changes when you update a dependency, and this history will be maintained in your git repo.
- If a package [suddenly disappears](https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/) from the internet, you are covered.

If you want to vendor your dependencies in Ruby, you may use the following command:

```bash
$ bundle package
```

Here's how vendoring is achieved in Go:

```bash
$ go mod tidy
$ go mod vendor
```

Running `go mod tidy` is essential to keep the list of dependencies listed in your `go.mod` file accurate before vendoring. The second command will create a `vendor` directory in your project root, and all the third-party dependencies (direct and indirect) required to build your project are copied over to the directory. This folder should be checked into version control.

As of Go 1.14, the `go` command will also verify that your project’s `vendor/modules.txt` file is consistent with its `go.mod` file. If not, you may get an error, such as the one shown below, when building your project:

```bash
$ go build
go: inconsistent vendoring in /home/ayo/Developer/demo/example:
        github.com/urfave/cli@v1.22.4: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/urfave/cli/v2@v2.2.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory
```

To fix this error, run `go mod tidy`, and then run `go mod vendor` so that everything is consistent again.

## Wrapping Things Up

In this article, we covered the most important concepts you need to know regarding Go modules before adopting them in your project. To recap,

- `go mod init` will initialize modules in your project.
- `go get` adds a new dependency, and you can use `go get -u` or `go get -u=patch` to upgrade a dependency to a new minor or patch version.
- `go mod tidy` cleans up unused dependencies or adds missing dependencies.
- `go mod vendor` copies all third-party dependencies to a `vendor` folder in your project root.

If you have any questions or opinions, I'd love to hear about it on [Twitter](https://twitter.com/ayisaiah). Thanks for reading, and happy coding!

[1]: https://www.honeybadger.io/blog/golang-go-package-management/#authorDetails
[2]: https://www.honeybadger.io/blog/golang-go-package-management/