Repository: go
Issue: runtime: deadlock when running concurrent builds on MacOS
Labels: NeedsInvestigation,compiler/runtime

<!--
Please answer these questions before submitting your issue. Thanks!
-->

### What version of Go are you using (`go version`)?

<pre>
$ go version
1.20.2
</pre>

### Does this issue reproduce with the latest release?

Yes, this could be reproduce on `1.20.3` but cannot be reproduce on `1.19.8`

### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/sluongng/Library/Caches/go-build"
GOENV="/Users/sluongng/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/sluongng/work/golang/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/sluongng/work/golang"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/external/go_sdk"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/private/var/tmp/_bazel_sluongng/5c02d8a9b82454292bdfef7b6f9ac04e/external/go_sdk/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.2"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/sluongng/work/bazelbuild/rules_go/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/ly/gjsqx9wd0cv1ck7xdslrt6t00000gn/T/go-build51901763=/tmp/go-build -gno-record-gcc-switches -fno-common"
</pre></details>

### What did you do?

<!--
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on go.dev/play is best.
-->

In [Bazel's rules_go](https://github.com/bazelbuild/rules_go), we need to compile a helper binary using `go build ...` before building anything else.

However, due to different Bazel configurations existing in the same repo (i.e. CGO enable/disable), this same helper binary is built multiple times, once for each unique set of configurations.

When trying to build a test in rules_go with `bazel build //tests/core/go_path:copy_path`, the helper binary was scheduled by Bazel to be compiled 9 times in parallel. This resulted in 6 different `go build` processes running in parallel at once.

![image](https://user-images.githubusercontent.com/26684313/232322131-4e31ff23-3376-482a-a83f-6c21929407cf.png)

Each of these build process could spawn multiple `compile` subprocess as a result

![image](https://user-images.githubusercontent.com/26684313/232322180-922c1cfd-35ac-46b8-bd8c-32ee568b2806.png)

### What did you expect to see?

Build to complete without issue

### What did you see instead?

All these builds are stuck in a deadlock that completely freeze applications running on the same machine. Opening a new Chrome tab, or running `git config` would never finish.

Using Activity Monitor's sampling feature we could see call graphs like such for each `go build` process

![image](https://user-images.githubusercontent.com/26684313/232322308-526608ce-23d8-466f-94d0-59ef3656172a.png)
![image](https://user-images.githubusercontent.com/26684313/232322316-4bb134d9-274f-468c-9488-d34376b63aff.png)

It seems like there are 3 types of stack relating to `libsystem_kernel.dylib / libsystem_pthread.dylib`

```
    2430 Thread_44978   DispatchQueue_1: com.apple.main-thread  (serial)
    + 2430 ???  (in <unknown binary>)  [0x1358]
    +   2430 runtime.asmcgocall.abi0  (in go) + 124  [0x100b702ac]
    +     2430 runtime.syscall6.abi0  (in go) + 56  [0x100b71a98]
    +       2430 __wait4_nocancel  (in libsystem_kernel.dylib) + 8  [0x1a17a04f4]

    2430 Thread_44999
    + 2430 runtime.asmcgocall.abi0  (in go) + 201  [0x100b702f9]
    +   2430 runtime.pthread_cond_timedwait_relative_np_trampoline.abi0  (in go) + 28  [0x100b717ec]
    +     2430 _pthread_cond_wait  (in libsystem_pthread.dylib) + 1276  [0x1a17d45a0]
    +       2430 __psynch_cvwait  (in libsystem_kernel.dylib) + 8  [0x1a1797710]

    2430 Thread_45038
    + 2430 runtime.kevent_trampoline.abi0  (in go) + 40  [0x100b71518]
    +   2430 kevent  (in libsystem_kernel.dylib) + 8  [0x1a179a060]
```

The last stack seems to be waiting for some event that never finishes (?).

This bug could be reproduced consistently for me with go `1.20.2` and `1.20.3`. It's a bit painful though as I would need to restart my laptop each time. It would go away if I try to build the binary sequentially by limiting Bazel's parallelism, or if I were to downgrade the go version to `1.19.8`.

Here is some additional system info
```
> uname -a
Darwin Sons-Laptop.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64
```