Repository: go
Issue: crypto/tls: ClientHelloInfo.Conn field is nil (or return value of RemoteAddr())
Labels: NeedsInvestigation

<!--
Please answer these questions before submitting your issue. Thanks!
-->

### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.21rc3 linux/amd64
</pre>

### Does this issue reproduce with the latest release?
Reproducable with `go1.21rcX`
Not reproducable with `<go1.20.6`


### What operating system and processor architecture are you using (`go env`)?
`Debian 12.1 amd64`
<details><summary><code>go env</code> Output</summary><br><pre>
$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/hnrk/.cache/go-build'
GOENV='/home/hnrk/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/hnrk/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/hnrk/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/hnrk/go-bins/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/hnrk/go-bins/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21rc3'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1381339882=/tmp/go-build -gno-record-gcc-switches'
</pre></details>

### What did you do?

<!--
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on go.dev/play is best.
-->
Trying to run Caddy webserver (_`Version v2.7.0-beta.2.0.20230725185021-d7d16360d411 h1:Hq2Ph3i47imGFwMmyEb8g8ExG2G9ISJlQJ6R73ddb6E=`_) with go1.21rc3 as described [here](https://github.com/caddyserver/caddy/issues/5659#issue-1826376600) but it panics after some time..
<details><summary>Logs</summary><br><pre>
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: panic: runtime error: invalid memory address or nil pointer dereference
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x8d5d60]
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: goroutine 108749 [running]:
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: github.com/caddyserver/certmagic.(*Config).getCertDuringHandshake(0xc00137a5b0, {0x1e2ab90, 0x2ae9520}, _, _)
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]:         github.com/caddyserver/certmagic@v0.19.1/handshake.go:378 +0x1340
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: github.com/caddyserver/certmagic.(*Config).GetCertificateWithContext(0xc00137a5b0, {0x1e2ab90, 0x2ae9520}, 0xc00137a4e0)
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]:         github.com/caddyserver/certmagic@v0.19.1/handshake.go:84 +0xbc5
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: github.com/caddyserver/certmagic.(*Config).GetCertificate(...)
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]:         github.com/caddyserver/certmagic@v0.19.1/handshake.go:50
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: github.com/caddyserver/caddy/v2/modules/caddytls.(*ConnectionPolicy).buildStandardTLSConfig.func1(0xc00137a4e0)
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]:         github.com/caddyserver/caddy/v2@v2.7.0-beta.2.0.20230725185021-d7d16360d411/modules/caddytls/connpolicy.go:232 +>
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: crypto/tls.(*Config).getCertificate(0xc0008e8ea0, 0xc00137a4e0)
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]:         crypto/tls/common.go:1116 +0x3b
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: crypto/tls.(*serverHandshakeStateTLS13).pickCertificate(0xc0009d9bf8)
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]:         crypto/tls/handshake_server_tls13.go:435 +0x314
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc0009d9bf8)
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]:         crypto/tls/handshake_server_tls13.go:59 +0x53
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: crypto/tls.(*Conn).serverHandshake(0xc000774380, {0x1e2aae8, 0xc00146e230})
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]:         crypto/tls/handshake_server.go:53 +0x185
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: crypto/tls.(*Conn).handshakeContext(0xc000774380, {0x1e2aab0, 0xc0006fee40})
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]:         crypto/tls/conn.go:1547 +0x3d3
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: crypto/tls.(*Conn).HandshakeContext(0xc000afafd0?, {0x1e2aab0?, 0xc0006fee40?})
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]:         crypto/tls/conn.go:1487 +0x1d
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]: created by crypto/tls.(*QUICConn).Start in goroutine 109988
Jul 28 13:43:03 www.hnrk.io v2caddy[2690046]:         crypto/tls/quic.go:177 +0xc9
Jul 28 13:43:03 www.hnrk.io systemd[1]: v2caddy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 28 13:43:03 www.hnrk.io systemd[1]: v2caddy.service: Failed with result 'exit-code'.
Jul 28 13:43:03 www.hnrk.io systemd[1]: v2caddy.service: Consumed 2min 26.066s CPU time.
</pre></details>

It looks like "ClientHelloInfo.Conn field is nil (or maybe just the return value of RemoteAddr() is)" (Thanks @mholt)

### What did you expect to see?
Caddy webserver in operating state without any panics.


### What did you see instead?
Caddy webserver keeps panicking irregularly.