The New Go Package Versioning for Dummies
Tags: go
Published: 2020-02-20 21:38:40

---
title: The New Go Package Versioning for Dummies
published: true
date: 2018-04-06 21:25:34 UTC
tags: go
canonical_url: https://www.skillscheck.co.uk/post/go-package-versioning-for-dummies/
---

Or perhaps that should be ‘_by_ Dummies’…

In February (2018) [Russ Cox](https://swtch.com/~rsc/), one of the people behind Go, published a series of blogs where he attempts to tackle one of the biggest problems in the Golang space right now. This is once again an attempt by me to understand what the bloody hell is going on in the world.

This blog is going to mostly rehash what he’s done, peppered with bits that I’ve picked up on my travels around the internet. The purpose isn’t only for your entertainment, it’s to help fix things in my mind.

Source material:

- [Russ’ Go Versioning Blog](https://research.swtch.com/vgo)
- [Versioning Proposal Blog](https://blog.golang.org/versioning-proposal)
- [The formal Go package versioning proposal](https://github.com/golang/proposal/blob/master/design/24301-versioned-go.md)

# What’s the problem

Let’s say you import a package `github.com/me/pkg` in your project. All is well until you decide to build your project on another machine and it no longer works. Your code didn’t change but running `go get githhub.com/me/pkg` got an entirely different and incompatible version of the package. There’s no way of specifying what version of the package you want. If the API changes or a bug is introduced, you have no protection. A change to `github.com/sartori/go.uuid` is a good example of this. [This issue](https://github.com/satori/go.uuid/issues/66) was raised following a commit that changed the return signature of some of the methods, breaking projects that imported the package. For a while there was no way of telling which version you had without looking at the code.

It gets worse when you consider that packages that your project imports, also import their own dependencies. Again you don’t control the version that `go` downloads for you.

And what happens if your dependencies import the same packages you do? Let’s take `go.uuid` as an example again. What if my project imports `go.uuid` and another import does too. If I found the API change and updated my code, what about other imported packages? I have to wait for the maintainers to discover the problem and fix it when they get around to it - if they ever do!

> **Quick note on versioning**
> 
> Believe it or not, there is a standard for what a version means. It’s called [semantic versioning](https://semver.org/). In a nutshell, your version is defined as `vX.Y.Z`. Let’s work through this from lowest to highest importance:
> 
> - Z -> Patch Version – change this to indicate a small changes such as bug fixes
> - Y -> Minor Version – change this for making backwards compatible changes such as adding new features
> - X -> Major Version – change this when you make incompatible API changes

# What was done to solve it

## Gopkg.in

Available via at [http://gopkg.in](http://gopkg.in), this service allows package developers to manage versions of their APIs. This service acts as a façade in front of Github, allowing developers to `go get` specific major versions of the package that you’d like to import. The Gopkg service will work out what the latest release by parsing git tags, making sure that `go get -u` will update your dependency.

This was one of the earliest solutions to the package versioning problem, but it never seemed to catch on. My feeling is that it should have been more popular than it is. As good as it is, there are a few problems with this approach:

- You can only use Github to host your code, so no private Bitbucket or corporate repos
- You can only get the latest release of a major version - no rolling back if you find a problem
- Your dependency relies on 2 external services and not just Github

## Lots of tools

Tools started to be developed to help the situation. There’s a fairly comprehensive list [here](https://github.com/golang/go/wiki/PackageManagementTools). By and large these tools allowed you to specify which version of a dependency you would like to download, even down to the specific commit. Some work by cloning repositories in to your `GOPATH`, others acutally provided full build solutions.

Some of the interesting ones were:

- [glide](https://glide.sh/) was for a long time the gold standard in Go package management, using a file to specify the versions of each dependency
- [Godep](https://github.com/tools/godep) saves the dependencies that you’ve already downloaded to allow you to commit the `vendor` directory
- [Govendor](https://github.com/kardianos/govendor) can fetch individual using the command line parameters and save them in `vendor` again

There is also an honourable mention for [gb](https://getgb.io/) which created a seperate `vendor` directory alongside `src` in `GOPATH` but also performed the build.

All of these tools attempted to solve the problem of package versioning in slightly different ways, with varying degrees of success. Ultimately though none of them were officially supported by the Go teama and suffered because of it.

## Vendor directory

Concensus had started to form around the need of the `vendor` directory. So in version 1.5 of Go brought official experimental support for the the `vendor` directory by setting the environment variable `GO15VENDOREXPERIMENT` to `1`. [This](https://medium.com/@freeformz/go-1-5-s-vendor-experiment-fd3e830f52c3) gives a good background but the essential bit is `go` would now prefer the `vendor` directory when resolving dependencies. Dependencies would be looked up as though `./vendor` were the same as `$GOPATH/src`. Packages that had their own `vendor` directory was still a problem that you had to rely on tools to solve at this point.

At version 1.7, the experiment was over and the `vendor` directory no longer sat behind an environment variable. Subsequent versions improved support by doing things like making sure that `go test ./...` didn’t navigate `vendor`.

The intention of the Go team was that making this directory available would allow the community to come to a concensus around the right solution but this didn’t appear.

## Dep

It became clear that just expecting a major change in functionality to spontaneously arrive from the community wasn’t going to work. They decided to put together a crack team to tackle the problem. The people that they chose had made important contributions, some being the authors of the packages managers above. The intention was that whatever tooling they came up with would, eventually, become a first class supported part of Go tooling.

They quickly arrived at the decision to create a new package management tool from scratch, learning all of the lessons of the tool that when before them. This tool was called `dep`. It allowed you to specify dependencies in the `Gopkg.toml` and `Gopkg.lock` which could be committed along with the contents of `vendor` if you so wish. It’s a great tool, it works out what your dependencies are and which versions are compatible.

But it can be slow. `dep ensure -v` will show you what it’s doing, sometimes that means it will clone a repository, build the latest tag, watch it fail and then work down the tags until something works. Sometimes not even this works.

## Why won’t these work?

There’s one problem with all of these approaches so far haven’t dealt with. What happens when you have 2 dependencies that rely on different versions of the same sub-dependency? Ordinarily this isn’t a problem, but what if this makes part of the API that you’re consuming?

# RSC’s solution

Russ essentially went back to basics and perhaps a little bit back in time and remembered an early proposition by the Go team that has been called the _import compatibility rule_. Russ has a great description on the [Go Blog](https://blog.golang.org/versioning-proposal).

> “If an old package and a new package have the same import path, the new package must be backwards compatible with the old package.”

Is there anything more to say than this? Perhaps there is…

What it means is that if you want to make breaking API changes, you need to contrive a new import path. This becomes what he calls [_semantic import versioning_](https://research.swtch.com/vgo-import). What this means is that if you have version 1 of your package, which in turn means putting the version of the API in the import path. `github.com/my/pkg` becomes `github.com/my/pkg/v1`. Following the _semantic versioning_ standard means that API compatibility is indicated using the Major version, which we use in the path.

When he did a little thinking about this Russ discovered a couple of interesting implications:

- If package developers keep to the semver standard, you can be confident that when you import something it will be compatible with EVERYTHING in your `GOPATH`
- If you decide to upgrade a package, it won’t surprise you by breaking your build, or business logic

Russ’s experience with Dep showed the algorithm used to resolve package versions could be very slow. He has developed a new algorithm, [_minimal version selection_](https://research.swtch.com/vgo-mvs) that used the concepts from above. It proved MUCH faster to resolve versions in the early prototype. It has the added benefit that it can use a single configuration file that can be edited both by humans and tools.

An fuller prototype of a version of the Go tool implementing all of this has been written, [`vgo`](https://godoc.org/golang.org/x/vgo).