Repository: go
Issue: cmd/go: spends a long time loading packages
Labels: OS-Windows,NeedsInvestigation,GoCommand

<!--
Please answer these questions before submitting your issue. Thanks!
-->

### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.20.4 windows/amd64
</pre>

### Does this issue reproduce with the latest release?
Yes


### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\a3367\AppData\Local\go-build
set GOENV=C:\Users\a3367\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\a3367\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\a3367\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.20.4
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\a3367\work\web\go.mod
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\a3367\AppData\Local\Temp\go-build1854760467=/tmp/go-build -gno-record-gcc-switches
</pre></details>

### What did you do?
I tried to build a program with the following `go.mod` and `go.sum`.

[gomod.zip](https://github.com/golang/go/files/11933269/gomod.zip)

and did a profiling of the build process with the following command:

```
go build -debug-trace="build.trace" .\cmd\server\GoServer.go
```

and got the following trace:

[build_trace.zip](https://github.com/golang/go/files/11933274/build_trace.zip)

then I visualized the trace in https://ui.perfetto.dev/

P.S.
I have asked on both go-nuts and golangbridge, but it seems that the community does not have an answer either.
https://groups.google.com/g/golang-nuts/c/72OsIXVcmCs
https://forum.golangbridge.org/t/very-slow-build-taking-lots-of-time/32149

### What did you expect to see?
I expect to see the build finish in 7 seconds, which is indicated by the section
`exec.Builder.Do` in the trace.


### What did you see instead?
I saw the build taking an entire 40 seconds!
with most the time spent in `load.PackagesAndErrors`.
A closer look suggests that the culprit might be in the following functions:

* modfetch.download
* modload.loadImport

Since I can build my program without internet connection, this rules out network issues.
Why is module loading taking so much time (there're only 122 lines in my go.sum)?
How can we reduce the time spent in module loading, or even better eliminate these loading operations?

For many many many years, the most exciting and unbeatable feature of Go is its consistently fast build times on all my programs.
I am kind of surprised that not only is building with the latest release slow, but most of the time is spent on loading modules, instead of actually compiling.

# Update 2023/07/07
As described in this [comment](https://github.com/golang/go/issues/61149#issuecomment-1624772612), the build can be made fast when building with `go mod vendor`. Under vendor mode, the build takes 10s, which is a 4x improvement over the original 40s.
However, there's still a mysterious 6s gap in the very beginning, and eliminating this do nothing gap would make build times drop to 4s, which is a 10x improvement!
However, I don't know what causes this gap neither how to remove it. Is this gap due to some sort of warm up or unnecessary checks during start up?