Repository: go
Issue: runtime: performance degradation on tip that disappears when traced/profiled
Labels: NeedsInvestigation,compiler/runtime

### Go version

go version go1.22.4 darwin/arm64

### Output of `go env` in your module/workspace:

```shell
GO111MODULE='on'
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/joaks/Library/Caches/go-build'
GOENV='/Users/joaks/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/joaks/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/joaks/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.22.4/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.22.4/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.4'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/Users/joaks/go/src/scratch/123_benchmarking/standalone_json/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/xj/2wbc4_xn293gkz7_6rzxsz5w0000gn/T/go-build3952519913=/tmp/go-build -gno-record-gcc-switches -fno-common'
```


### What did you do?

While doing some initial testing in preparation for Go 1.23 in the coming months, we ran some of Uber’s open-source benchmarks using the tip of Go.

### What did you see happen?

We saw surprising degradations in a couple [Zap](https://github.com/uber-go/zap) benchmarks of around 20%. The most concerning of which was a simple test that measured concurrent calls to json.Marshal (nothing Zap-related). Here’s a simplified version of that benchmark:

```go
const _size = 5000

func BenchmarkEncodingJSONStrings(b *testing.B) {
	s := make([]string, _size)
	for i := 0; i < len(s); i++ {
		s[i] = fmt.Sprint("00000000-0000-0000-0000-0000000000", i)
	}

	b.ResetTimer()
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			if _, err := json.Marshal(s); err != nil {
				b.Fatal(err)
			}
		}
	})
}
```

To reproduce:
```
$ go version
go version go1.22.4 darwin/arm64
$ go test -count 30 -bench . | tee go1224.log

$ ~/go/src/github.com/golang/go3/bin/go version
go version devel go1.23-93bbf719a6 Wed Jun 5 17:30:16 2024 +0000 darwin/arm64
$ ~/go/src/github.com/golang/go3/bin/go test -count 30 -bench . | tee head.log

$ benchstat go1224.log head.log
goos: darwin
goarch: arm64
pkg: standalonejson
cpu: Apple M1 Max
                       │ go1224.log  │              head.log               │
                       │   sec/op    │   sec/op     vs base                │
EncodingJSONStrings-10   43.48µ ± 2%   51.90µ ± 2%  +19.38% (p=0.000 n=30)
```

Some notes about reproducing:
* The degradation only seems to manifest when running the benchmark in parallel. When parallelism is decreased, the magnitude of the degradation decreases, completely disappearing when run in serial.
* Myself and @sywhang were able to reproduce this on darwin-arm64. We don’t have an isolated linux environment, so we aren’t able to say for sure if it reproduced on linux - but based on anecdotal evidence it does seem to.
* This does not seem to be isolated to encoding/json. Several Zap benchmarks degraded due to the same commit:
  * ~20% - [BenchmarkZapJSON](https://github.com/uber-go/zap/blob/v1.27.0/zapcore/json_encoder_bench_test.go#L75) - benchmarks Zap’s JSON encoder
  * ~5% - [BenchmarkZapJSONFloat32AndComplex64](https://github.com/uber-go/zap/blob/v1.27.0/zapcore/json_encoder_bench_test.go#L46)
  * ~20% - [BenchmarkStandardJSON](https://github.com/uber-go/zap/blob/v1.27.0/zapcore/json_encoder_bench_test.go#L100) - benchmarks encoding/json.Marshal

**Our Investigation**

Using git bisect revealed that [this commit](https://github.com/golang/go/commit/5bba5b256ce287736da60372c5cf634395d7b1a3) is what is causing the degradation in the benchmarks.

This is a surprising result as the offending commit seems entirely unrelated to the benchmarks. We tried to see how this new code could be getting hit at all, and were unable to find a link via dlv breakpoints nor print statements in the runtime package.

We did try to collect CPU profiles while running the benchmarks to investigate further but interestingly, the degradation disappears when collecting a CPU profile.

```
$ ~/go/src/github.com/golang/go3/bin/go version
go version devel go1.23-93bbf719a6 Wed Jun 5 17:30:16 2024 +0000 darwin/arm64
$ ~/go/src/github.com/golang/go3/bin/go test -count 30 -cpuprofile headcpu.prof -bench . | tee head-prof.log
goos: darwin
goarch: arm64
pkg: standalonejson
cpu: Apple M1 Max
                       │ go1224.log  │         head-prof.log         │
                       │   sec/op    │   sec/op     vs base          │
EncodingJSONStrings-10   43.48µ ± 2%   43.31µ ± 1%  ~ (p=0.495 n=30)
```

Since this issue seems to be unrelated to Zap specifically, and reproduces with calls to encoding/json, we wanted to bring it up in case it points to a more general issue.

### What did you expect to see?

No practical degradation.