Repository: kubernetes
Issue: apiserver: data race
Labels: kind/bug,sig/api-machinery,sig/instrumentation,triage/accepted

### What happened?

When enabling race detection with (roughly) Kubernetes d89d5ab2680bc74fe4487ad71e514f4e0812d9ce, the following race was reported:

```
k8s.io/kubernetes/test/integration/etcd: TestOverlappingCustomResourceCustomResourceDefinition

==================
WARNING: DATA RACE
Read at 0x00c00e27c4f8 by goroutine 11833:
  k8s.io/kubernetes/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/handler.go:206 +0x19ae
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithLatencyTrackers.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/webhook_duration.go:57 +0x3b8
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x307
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withRequestReceivedTimestampWithClock.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/request_received_time.go:38 +0x227
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithMuxAndDiscoveryComplete.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/mux_discovery_complete.go:52 +0x224
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:74 +0x137
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withAuditInit.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit_init.go:63 +0x369
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x61
  net/http.serverHandler.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2936 +0x682
  net/http.initALPNRequest.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:3545 +0x36d
  net/http.(*initALPNRequest).ServeHTTP()
      <autogenerated>:1 +0x85
  net/http.Handler.ServeHTTP-fm()
      <autogenerated>:1 +0x75
  k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).runHandler()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2311 +0xec
  k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).processHeaders.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2025 +0x64

Previous write at 0x00c00e27c4f8 by goroutine 11834:
  k8s.io/kubernetes/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*bodyWrapper).Read()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/wrap.go:40 +0xa4
  io.(*LimitedReader).Read()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/io/io.go:477 +0xc6
  io.ReadAll()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/io/io.go:701 +0xf5
  io/ioutil.ReadAll()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/io/ioutil/ioutil.go:27 +0x1cd
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.limitedReadBody()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:382 +0x1c1
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.limitedReadBodyWithRecordMetric()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:393 +0x6a
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.UpdateResource.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/update.go:76 +0x4c7
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.restfulUpdateResource.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1294 +0xda
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:571 +0x36b
  k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/v3.(*Container).dispatch()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/v3/container.go:299 +0x92e
  k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/v3.(*Container).Dispatch()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/v3/container.go:204 +0x77a
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x74c
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP()
      <autogenerated>:1 +0x85
  k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.(*proxyHandler).ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:125 +0xf47
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:249 +0x5af
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:235 +0x68
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x959
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP()
      <autogenerated>:1 +0x85
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1.1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:103 +0x75
  runtime.deferreturn()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/runtime/panic.go:476 +0x32
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x619
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x243
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1.1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:103 +0x75
  runtime.deferreturn()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/runtime/panic.go:476 +0x32
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithPriorityAndFairness.func2.9()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/priority-and-fairness.go:292 +0x14e
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flowcontrol.(*configController).Handle.func2()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flowcontrol/apf_filter.go:193 +0x375
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flowcontrol.immediateRequest.Finish()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flowcontrol/apf_controller.go:968 +0x44
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flowcontrol.(*immediateRequest).Finish()
      <autogenerated>:1 +0x1d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flowcontrol.(*configController).Handle()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flowcontrol/apf_filter.go:180 +0xbb6
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithPriorityAndFairness.func2()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/priority-and-fairness.go:295 +0x13b5
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x243
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1.1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:103 +0x75
  runtime.deferreturn()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/runtime/panic.go:476 +0x32
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x2e7
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x243
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1.1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:103 +0x75
  runtime.deferreturn()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/runtime/panic.go:476 +0x32
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x243
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1.1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:103 +0x75
  runtime.deferreturn()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/runtime/panic.go:476 +0x32
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:105 +0xa81
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:94 +0x4b5
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:115 +0xdb

Goroutine 11833 (running) created at:
  k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).processHeaders()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2025 +0xdf5
  k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).processFrame()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:1530 +0x424
  k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).processFrameFromReader()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:1473 +0x2ed
  k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).serve()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:957 +0x15dc
  k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*Server).ServeConn()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:531 +0x179d
  k8s.io/kubernetes/vendor/golang.org/x/net/http2.ConfigureServer.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:321 +0x124
  net/http.(*conn).serve()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:1903 +0x1c41
  net/http.(*Server).Serve.func3()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:3089 +0x58

Goroutine 11834 (running) created at:
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:101 +0x351
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withRequestDeadline.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/request_deadline.go:100 +0x41b
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withWaitGroup.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:86 +0x287
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:35 +0x2e4
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0x186
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.withLogging.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:111 +0xb6
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/handler.go:204 +0x1992
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithLatencyTrackers.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/webhook_duration.go:57 +0x3b8
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x307
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withRequestReceivedTimestampWithClock.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/request_received_time.go:38 +0x227
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithMuxAndDiscoveryComplete.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/mux_discovery_complete.go:52 +0x224
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:74 +0x137
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withAuditInit.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit_init.go:63 +0x369
  net/http.HandlerFunc.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2122 +0x4d
  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x61
  net/http.serverHandler.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:2936 +0x682
  net/http.initALPNRequest.ServeHTTP()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/.gimme/versions/go1.20.2.linux.amd64/src/net/http/server.go:3545 +0x36d
  net/http.(*initALPNRequest).ServeHTTP()
      <autogenerated>:1 +0x85
  net/http.Handler.ServeHTTP-fm()
      <autogenerated>:1 +0x75
  k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).runHandler()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2311 +0xec
  k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).processHeaders.func1()
      /home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2025 +0x64
==================
```



### What did you expect to happen?

No race.

### How can we reproduce it (as minimally and precisely as possible)?

`go test -race ./test/integration/etcd` (but might only fail in the CI).

### Anything else we need to know?

The read occurs here:

https://github.com/kubernetes/kubernetes/blob/d89d5ab2680bc74fe4487ad71e514f4e0812d9ce/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/handler.go#L206

The write occurs here:

https://github.com/kubernetes/kubernetes/blob/d89d5ab2680bc74fe4487ad71e514f4e0812d9ce/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/wrap.go#L40

Note that the write occurs in a goroutine created by [apiserver/pkg/server/filters/timeout.go:101](https://github.com/kubernetes/kubernetes/blob/d89d5ab2680bc74fe4487ad71e514f4e0812d9ce/staging/src/k8s.io/apiserver/pkg/server/filters/timeout.go#L101).

My theory is that the otelhttp code assumes that handling the request is completed once `h.handler.ServeHTTP` has returned. Kubernetes seems to violate that assumption to implement timeout handling. I bet that in case of a timeout, the data stored in the wrapper gets read while it is still in use.

Not sure whether that is a bug in otelhttp (then locking has to be added there) or in Kubernetes.

/cc @damien

### Kubernetes version

master

### Cloud provider

none

### OS version

Linux

### Install tools

_No response_

### Container runtime (CRI) and version (if applicable)

_No response_

### Related plugins (CNI, CSI, ...) and versions (if applicable)

_No response_