Repository: go
Issue: crypto/rsa: TLS requests are 2-3x slower in go1.20 with race detection enabled
Labels: Performance,NeedsInvestigation

### What version of Go are you using (`go version`)?

<pre>
$ go version

go version devel go1.20-bb0d8297d7 Mon Nov 28 19:38:37 2022 +0000 linux/amd64
and
go version devel go1.20-bb0d8297d7 Mon Nov 28 19:38:37 2022 +0000 darwin/arm64

</pre>

### Does this issue reproduce with the latest release?

No, introduced in go1.20 dev branch in https://go-review.googlesource.com/c/go/+/326012

### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/liggitt/.cache/go-build"
GOENV="/usr/local/google/home/liggitt/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/usr/local/google/home/liggitt/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/local/google/home/liggitt/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/google/home/liggitt/git/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/google/home/liggitt/git/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="devel go1.20-bb0d8297d7 Mon Nov 28 19:38:37 2022 +0000"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1952584495=/tmp/go-build -gno-record-gcc-switches"

and 

GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/liggitt/Library/Caches/go-build"
GOENV="/Users/liggitt/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/liggitt/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/liggitt/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/liggitt/git/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/liggitt/git/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="devel go1.20-bb0d8297d7 Mon Nov 28 19:38:37 2022 +0000"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/37/ns7gt60104scfm9fvg02p1jh00kjgb/T/go-build3981291129=/tmp/go-build -gno-record-gcc-switches -fno-common"

</pre></details>

### What did you do?

Ran project unit tests making TLS requests with race detection enabled (xref https://github.com/kubernetes/kubernetes/issues/114145#issuecomment-1330378146)

Minimal reproducer:

```go
package main

import (
	"net/http"
	"net/http/httptest"
	"strings"
	"testing"
	"time"
)

func TestRequest(t *testing.T) {
	transport, req := setup(t)
	start := time.Now()
	do(t, transport, req)
	t.Log(time.Since(start))
}

func BenchmarkRequest(b *testing.B) {
	transport, req := setup(b)
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		do(b, transport, req)
	}
}

func setup(t testing.TB) (*http.Transport, *http.Request) {
	data := []byte(strings.Repeat("a", 1024*1024)) // 1kb of data
	server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { w.Write(data) }))
	t.Cleanup(server.Close)

	transport := server.Client().Transport.(*http.Transport).Clone()
	transport.DisableKeepAlives = true // dial every time

	req, err := http.NewRequest("GET", server.URL, nil)
	if err != nil {
		t.Fatal(err)
	}

	return transport, req
}

func do(t testing.TB, transport *http.Transport, req *http.Request) {
	resp, err := transport.RoundTrip(req)
	if err != nil {
		t.Fatal(err)
	}
	defer resp.Body.Close()
	if resp.StatusCode != 200 {
		t.Fatal("unexpected status")
	}
}
```

run with
```shell
$go1.19.3 test main_test.go -race -bench . -count=20 > 1.19.3-race.txt
$go1.19.3 test main_test.go -bench . -count=20 > 1.19.3-norace.txt
$gotip test main_test.go -race -bench . -count=20 > head-race.txt
$gotip test main_test.go -bench . -count=20 > head-norace.txt
benchstat 1.19.3-race.txt head-race.txt
benchstat 1.19.3-norace.txt head-norace.txt
```

### What did you expect to see?

TLS request times are roughly equivalent to go1.19 regardless of race detection.

### What did you see instead?

to make the same request as with go1.19, go1.20 takes ~2x longer on linux/amd64, ~3x longer on darwin/arm64, and even longer on linux/ppc64le (I'm working on getting numbers with the minimal reproducer above)

linux/amd64:
```
name        old time/op  new time/op   delta
Request-24  8.92ms ± 3%  16.82ms ± 1%  +88.45%  (p=0.000 n=20+20)
```

darwin/arm64
```
name        old time/op  new time/op   delta
Request-10  9.32ms ± 5%  30.26ms ± 1%  +224.58%  (p=0.000 n=19+18)
```

This is with a simple request that only has the client verifying the server cert. When more is involved (mutual TLS or an RSA JWT verification), the slowdown is even more pronounced (3x - 7x slowdowns observed in https://github.com/kubernetes/kubernetes/issues/114145).

_Without_ race detection, the slowdown due to https://go-review.googlesource.com/c/go/+/326012 was not _as_ drastic, but was still milliseconds longer per request:

linux/amd64:
```
name        old time/op  new time/op  delta
Request-24  3.18ms ± 3%  4.02ms ± 3%  +26.58%  (p=0.000 n=19+19)
```

darwin/arm64
```
name        old time/op  new time/op  delta
Request-10  5.68ms ± 8%  7.72ms ±31%  +35.90%  (p=0.000 n=20+20)
```

cc @FiloSottile