Repository: go
Issue: cmd/go: implement lazy module loading in workspace mode to avoid `ambiguous import` errors
Labels: NeedsFix,GoCommand,modules

<!--
Please answer these questions before submitting your issue. Thanks!
-->

### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.20.3 linux/amd64
</pre>

Also tested on the latest:

<pre>
$ go version
go version go1.20.4 linux/amd64
</pre>


### Does this issue reproduce with the latest release?

Yes.

### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env
O111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/rpajak/.cache/go-build"
GOENV="/home/rpajak/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/rpajak/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/rpajak/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/rpajak/repos/opentelemetry-go-contrib/instrumentation/github.com/gin-gonic/gin/otelgin/go.mod"
GOWORK="/home/rpajak/repos/opentelemetry-go-contrib/go.work"
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3679794853=/tmp/go-build -gno-record-gcc-switches"
</pre></details>

### What did you do?

<!--
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on go.dev/play is best.
-->

I added `go.work` and it looks like it caused the `go build` tooling to differently handle dependency versions. Reference:  https://github.com/open-telemetry/opentelemetry-go-contrib/pull/3805

You can checkout this repository https://github.com/pellared/opentelemetry-go-contrib/tree/5a8a020149031cf531deada3c13e83d79c39ed3a and run: `go test ./instrumentation/github.com/gin-gonic/gin/otelgin/...` to reproduce the bug.

### What did you expect to see?

Build and tests are passing.

Using `go.work` does not affect the build

### What did you see instead?

```
~/repos/opentelemetry-go-contrib$ go test ./instrumentation/github.com/gin-gonic/gin/otelgin/...
../../go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/binding/msgpack.go:15:2: ambiguous import: found package github.com/ugorji/go/codec in multiple modules:
        github.com/ugorji/go v0.0.0-20171122102828-84cb69a8af83 (/home/rpajak/go/pkg/mod/github.com/ugorji/go@v0.0.0-20171122102828-84cb69a8af83/codec)
        github.com/ugorji/go/codec v1.2.9 (/home/rpajak/go/pkg/mod/github.com/ugorji/go/codec@v1.2.9)
```

I tried even to clear Go modules cache hoping that it will fix the issue. However, for some reason it keeps downloading `github.com/ugorji/go v0.0.0-20171122102828-84cb69a8af83` even though `github.com/ugorji/go/codec v1.2.9` is in `go.mod`:

```
~/repos/opentelemetry-go-contrib/instrumentation/github.com/gin-gonic/gin/otelgin$ go clean -modcache

~/repos/opentelemetry-go-contrib/instrumentation/github.com/gin-gonic/gin/otelgin$ go test
go: downloading github.com/gin-gonic/gin v1.8.2
go: downloading go.opentelemetry.io/otel v1.15.1
go: downloading github.com/stretchr/testify v1.8.2
go: downloading go.opentelemetry.io/otel/trace v1.15.1
go: downloading golang.org/x/net v0.9.0
go: downloading github.com/gin-contrib/sse v0.1.0
go: downloading github.com/mattn/go-isatty v0.0.17
go: downloading github.com/ugorji/go/codec v1.2.9
go: downloading github.com/go-playground/validator/v10 v10.11.1
go: downloading google.golang.org/protobuf v1.30.0
go: downloading github.com/pelletier/go-toml/v2 v2.0.6
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/sys v0.7.0
go: downloading github.com/go-logr/logr v1.2.4
go: downloading github.com/go-logr/stdr v1.2.2
go: downloading github.com/ugorji/go v0.0.0-20171122102828-84cb69a8af83
go: downloading golang.org/x/text v0.9.0
go: downloading github.com/go-playground/universal-translator v0.18.1
go: downloading golang.org/x/crypto v0.6.0
go: downloading github.com/leodido/go-urn v1.2.1
go: downloading github.com/pelletier/go-toml v1.9.5
go: downloading github.com/go-playground/locales v0.14.1
/home/rpajak/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/binding/msgpack.go:15:2: ambiguous import: found package github.com/ugorji/go/codec in multiple modules:
        github.com/ugorji/go v0.0.0-20171122102828-84cb69a8af83 (/home/rpajak/go/pkg/mod/github.com/ugorji/go@v0.0.0-20171122102828-84cb69a8af83/codec)
        github.com/ugorji/go/codec v1.2.9 (/home/rpajak/go/pkg/mod/github.com/ugorji/go/codec@v1.2.9)

~/repos/opentelemetry-go-contrib/instrumentation/github.com/gin-gonic/gin/otelgin$ go mod why github.com/ugorji/go
go: downloading k8s.io/apimachinery v0.27.1
go: downloading k8s.io/client-go v0.27.1
go: downloading github.com/pierrec/lz4/v4 v4.1.17
go: downloading honnef.co/go/tools v0.4.3
# github.com/ugorji/go
(main module does not need package github.com/ugorji/go)
```

I found a hacky workaround:

```
~/repos/opentelemetry-go-contrib/instrumentation/github.com/gin-gonic/gin/otelgin$ go get github.com/ugorji/go@v1.2.9
go: downloading github.com/ugorji/go v1.2.9
go: added github.com/ugorji/go v1.2.9

~/repos/opentelemetry-go-contrib/instrumentation/github.com/gin-gonic/gin/otelgin$ go test
PASS
ok      go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin    0.016s
```

However, `go mod tidy` correctly clears the `github.com/ugorji/go v1.2.9 // indirect` entry from `go.mod` and the issue is back again:

```
~/repos/opentelemetry-go-contrib/instrumentation/github.com/gin-gonic/gin/otelgin$ go mod tidy

~/repos/opentelemetry-go-contrib/instrumentation/github.com/gin-gonic/gin/otelgin$ go test
/home/rpajak/go/pkg/mod/github.com/gin-gonic/gin@v1.8.2/binding/msgpack.go:15:2: ambiguous import: found package github.com/ugorji/go/codec in multiple modules:
        github.com/ugorji/go v0.0.0-20171122102828-84cb69a8af83 (/home/rpajak/go/pkg/mod/github.com/ugorji/go@v0.0.0-20171122102828-84cb69a8af83/codec)
        github.com/ugorji/go/codec v1.2.9 (/home/rpajak/go/pkg/mod/github.com/ugorji/go/codec@v1.2.9)
````

I find this issue frightening. I am not sure if I can confidently use Go workspaces in monorepos.