Introducing go_test.mod
Tags: go
Published: 2021-09-24 21:12:52

---
title: Introducing go_test.mod
published: true
description: How to avoid meta test dependencies across Go modules
tags: golang
---

### How to avoid meta test dependencies across Go modules

Since the [Go v1.14](https://golang.org/doc/go1.1408) release, the `go` command now includes a little known flag called `-modfile` that can be used to manage multiple set of dependencies within the same repository.

The `-modfile` flag can also be really helpful to also manage better what are the dependencies that importers of your package end up bringing into their project, specially when dealing with test dependencies.

#### The problem

For example, let's say that we have the following couple of repos that do not have any dependencies:

```go
# github.com/wallyqs/go-mod-a
package hello

func Hello() string {
	return "Hello"
}
```

```go
# github.com/wallyqs/go-mod-b
package world

func World() string {
	return "World"
}
```

Though for testing purposes, they do depend on each other. We add `hello_test.go` first in this case in the `go-mod-a` repo:

```go
package hello

import (
	"testing"
	"github.com/wallyqs/go-mod-b"
)

func TestHello(t *testing.T) {
	got := Hello()
	expected := "Hello"
	if got != expected {
		t.Fatalf("Expected %v, got: %v", expected, got)
	}
}

func TestHelloWorld(t *testing.T) {
	got := fmt.Sprintf("%s %s", Hello(), world.World())
	expected := "Hello World"
	if got != expected {
		t.Fatalf("Expected %v, got: %v", expected, got)
	}
}
```

The result of this will be something like the following in the
`go.mod` and `go.sum` in the go-mod-a repo:

```sh
$ cat go.mod 
module github.com/wallyqs/go-mod-a

go 1.17

require github.com/wallyqs/go-mod-b v0.0.0-20210924195232-cc93d43aa245 // indirect

$ cat go.sum 
github.com/wallyqs/go-mod-b v0.0.0-20210924195232-cc93d43aa245 h1:vD21YUG9esVWngkuysL7tmR4l9EdwvaghEfdGNcBDVw=
github.com/wallyqs/go-mod-b v0.0.0-20210924195232-cc93d43aa245/go.mod h1:IMoYMTpn8BaoMzQP+9DeRp7bHQv/n+jAhQfguxXc9oM=
```

At this point, things look like this in the go-mod-a repo:

```sh
tree .
.
├── LICENSE
├── README.md
├── go.mod
├── go.sum
├── hello.go
└── hello_test.go
```

Now, if we add the same test to the go-mod-b, we will have a circular testing dependency which Go modules is actually fine with:

```go
package world

import (
	"fmt"
	"testing"
	"github.com/wallyqs/go-mod-a"
)

func TestWorld(t *testing.T) {
	got := World()
	expected := "World"
	if got != expected {
		t.Fatalf("Expected %v, got: %v", expected, got)
	}
}

func TestHelloWorld(t *testing.T) {
	got := fmt.Sprintf("%s %s", hello.Hello(), World())
	expected := "Hello World"
	if got != expected {
		t.Fatalf("Expected %v, got: %v", expected, got)
	}
}
```

```sh
$ ~/go/src/github.com/wallyqs/go-mod-b (main) $ go get github.com/wallyqs/go-mod-a
go: downloading github.com/wallyqs/go-mod-a v0.0.0-20210924201556-e67af2677460
go get: added github.com/wallyqs/go-mod-a v0.0.0-20210924201556-e67af2677460

$ ~/go/src/github.com/wallyqs/go-mod-b (main) $ go test ./... -v
=== RUN   TestWorld
--- PASS: TestWorld (0.00s)
=== RUN   TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
ok  	github.com/wallyqs/go-mod-b	0.014s
```

In the go-mod-b repo, now we will have something like this though:

```sh
$ tree .
.
├── LICENSE
├── README.md
├── go.mod
├── go.sum
├── world.go
└── world_test.go

$ cat go.sum 
github.com/wallyqs/go-mod-a v0.0.0-20210924201556-e67af2677460 h1:alim5qw73L9EfbCEF7TW3KDlbfWoMs580PtS4+1fF38=
github.com/wallyqs/go-mod-a v0.0.0-20210924201556-e67af2677460/go.mod h1:+86UH/9vUOVQadGIBaupYvn1FMXQIzXpXqmGtq28JPk=
github.com/wallyqs/go-mod-b v0.0.0-20210924195232-cc93d43aa245/go.mod h1:IMoYMTpn8BaoMzQP+9DeRp7bHQv/n+jAhQfguxXc9oM=
```

Notice how go-mod-b includes a *meta test dependency of itself*, if left as is now all importers of `go-mod-b` will be actually importing two different dependencies of the same module which is not really necessary.

Let's consider that this is now v0.1.0 version of both packages, so we then update our original test dependencies to be a tagged version:

```sh
$ go get github.com/wallyqs/go-mod-b@v0.1.0
go: downloading github.com/wallyqs/go-mod-b v0.1.0
go get: upgraded github.com/wallyqs/go-mod-b v0.0.0-20210924202312-2deb2c1c1be4 => v0.1.0

$ ~/go/src/github.com/wallyqs/go-mod-a (main) $ cat go.mod
module github.com/wallyqs/go-mod-a

go 1.17

require github.com/wallyqs/go-mod-b v0.1.0
```

Now after `go mod tidy` on the repo, the library will still be
bringing the last untagged version of itself as well:

```sh
$ cd ~/go/src/githubsrc/github.com/wallyqs/go-mod-a
$ cat go.sum
github.com/wallyqs/go-mod-a v0.0.0-20210924201556-e67af2677460/go.mod h1:+86UH/9vUOVQadGIBaupYvn1FMXQIzXpXqmGtq28JPk=
github.com/wallyqs/go-mod-b v0.0.0-20210924195232-cc93d43aa245/go.mod h1:IMoYMTpn8BaoMzQP+9DeRp7bHQv/n+jAhQfguxXc9oM=
github.com/wallyqs/go-mod-b v0.1.0 h1:3Qz5nQ0B98wlrwiXv7dEW2PhPfRqPB+dTmHPI08Dr/8=
github.com/wallyqs/go-mod-b v0.1.0/go.mod h1:BtD/yaPmz5Vf2OMu9+VcCNvR9I60VpeZTKvBQsUD5wg=
```

### How to solve this?

With the `-modfile` flag, we can better separate what dependencies are being imported if we change a bit how we use the testing dependencies. We will follow these steps then:

- Make a `test` folder

- Include the tests with test dependencies in the `test` folder.  This means that the `HelloWorld` test is now moved to its own test file like:

```go
// test/hello_test.go
package hello

import (
	"fmt"
	"testing"
	"github.com/wallyqs/go-mod-a"
	"github.com/wallyqs/go-mod-b"
)

func TestHelloWorld(t *testing.T) {
	got := fmt.Sprintf("%s %s", hello.Hello(), world.World())
	expected := "Hello World"
	if got != expected {
		t.Fatalf("Expected %v, got: %v", expected, got)
	}
}
```

```sh
$ tree ~/go/src/github.com/wallyqs/go-mod-a
├── LICENSE
├── README.md
├── go.mod
├── go_test.mod
├── go_test.sum
├── hello.go
├── hello_test.go
└── test
    └── hello_test.go
```

- Create a `go_test.mod` file:

```sh
cp go.mod go_test.mod && go mod tidy -modfile go_test.mod
```

- Then use `go test -modfile go_test.mod` to run the tests with the extra deps:

```sh
$ mkdir test
$ cp go.mod go_test.mod && go mod tidy -modfile go_test.mod

$ go test -modfile=go_test.mod ./... -v
=== RUN   TestHello
--- PASS: TestHello (0.00s)
PASS
ok  	github.com/wallyqs/go-mod-a	(cached)
=== RUN   TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
ok  	github.com/wallyqs/go-mod-a/test	(cached)
```

Next, to generate a minimal `go.mod` for your library without the testing dependencies, just test the package:

```sh
$ cd ~/go/src/github.com/wallyqs/go-mod-a
$ rm go.mod
$ go mod init
$ go test github.com/wallyqs/go-mod-a
```

To run the tests with the tests with extra dependencies you will need to use `go_test.mod` instead:

```sh
$ go test -modfile=go_test.mod -v github.com/wallyqs/go-mod-a/test
=== RUN   TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
```

And if you want to test all you can do:

```sh
$ go test ./... -modfile=go_test.mod -v
=== RUN   TestHello
--- PASS: TestHello (0.00s)
PASS
ok  	github.com/wallyqs/go-mod-a	(cached)
=== RUN   TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
ok  	github.com/wallyqs/go-mod-a/test	(cached)
```

#### Look, no dependencies!

Since both packages didn't actually depend on each other other than for the tests this will mean that the `go.mod` is just empty:

```sh
cd ~/go/src/github.com/wallyqs/go-mod-a (main) $ cat go.mod
module github.com/wallyqs/go-mod-a

go 1.17
```

*Bonus*: This will also mean that it does not depend on previous untagged versions of itself for testing!

```sh
cd ~/go/src/github.com/wallyqs/go-mod-a (main) $ cat go_test.mod 
module github.com/wallyqs/go-mod-a

go 1.17

require github.com/wallyqs/go-mod-b v0.1.0
```

#### Upgrading a test dependency

Let's say now that we tag the next version of `wallyqs/go-mod-a`, we can then upgrade our test dependency in the `wallyqs/go-mod-b` as follows:

```sh
$ cd ~/go/src/github.com/wallyqs/go-mod-b (main) $ 
go get -modfile=go_test.mod github.com/wallyqs/go-mod-a@v0.2.0
```

The `go_test.mod` as a result is just the latest tagged version:

```sh
head go*
==> go.mod <==
module github.com/wallyqs/go-mod-b

go 1.17

==> go_test.mod <==
module github.com/wallyqs/go-mod-b

go 1.17

require github.com/wallyqs/go-mod-a v0.2.0

==> go_test.sum <==
github.com/wallyqs/go-mod-a v0.2.0 h1:6o2qaAI3ssJhSx89cHvvY449+V3h/1Z09Z1ySoUPNwI=
github.com/wallyqs/go-mod-a v0.2.0/go.mod h1:N2dbiJxS0vlXnYKNv9J5JE6ULgOhcHA81/VkivvCr08=
```

#### Summary

If possible, avoid including test dependencies to your library and instead move them to a subfolder of our package.  Being able to do this is good for the ecosystem around your package since will mean less interactions with goproxy and simplified dependencies overall.

The example repos in this post can be found at:

- https://github.com/wallyqs/go-mod-a
- https://github.com/wallyqs/go-mod-b

We have been using this approach in the [nats.go](https://github.com/nats-io/nats.go) client for some time already where both the server and the client depend on each other for some of the tests.  As a result, the `go.mod` of the client is very small and users do not require to download dependencies of the server:

https://github.com/nats-io/nats.go/blob/main/go.mod

```go
module github.com/nats-io/nats.go

go 1.16

require (
	github.com/nats-io/nkeys v0.3.0
	github.com/nats-io/nuid v1.0.1
)
```

Hope this helps!

- Wally