Repository: go
Issue: net/http: HTTP/2 client sending RST_STREAM at more than 2500 transactions-per-second (TPS)
Labels: NeedsInvestigation

<!--
Please answer these questions before submitting your issue. Thanks!
-->

### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.18.1 windows/amd64
</pre>

### Does this issue reproduce with the latest release?
Yes


### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\AAA\AppData\Local\go-build
set GOENV=C:\Users\AAA\AppData\Roaming\go\env
set GOEXE=
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\AAA\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=linux
set GOPATH=C:\Users\AAA\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.18.1
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=0
set GOMOD=NUL
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\AAA~1\AppData\Local\Temp\go-build2512202176=/tmp/go-build -gno-record-gcc-switches
</pre></details>

### What did you do?

<!--
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on go.dev/play is best.
-->
I am using a non-TLS HTTP/2 go client program which sends more than 2500 POST requests per second to a Netty based HTTP/2 server. (Note : I am running this program on a Linux 64-bit machine. So basically my build envt is Windows, but the run envt is Linux). 

Each of the request is sent by a separate goroutine which in turn calls the following ```ChfConnection.SendUpdate()``` method spawned in another goroutine.

```go
func (connection *ChfConnection) SendUpdate(sessionId, messageBody string) (string, error) {
	totalN40RequestsTriggered.WithLabelValues("Update").Add(1)
	url := fmt.Sprintf("%s%s", connection.baseUrl, connection.updateUri)
	url = strings.ReplaceAll(url, "{ChargingDataRef}", sessionId)
	alog.Trace("POST ChargingDataRequest-Update =>\n%s", messageBody)
	resp, err := connection.send(messageBody, url, POST)
	if err != nil {
		return "", err
	}
	defer resp.Body.Close()
	respBody, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return "", err
	}

	alog.Trace("ChargingDataResponse-Update =>\n%s", string(respBody))

	if resp.StatusCode != http.StatusOK {
		return string(respBody), errors.New("HTTP request failed")
	}
	return string(respBody), nil
}

func (connection *ChfConnection) send(messageBody string, url string, reqType string) (*http.Response, error) {
	r := strings.NewReader(messageBody)
	req, err := http.NewRequest(reqType, url, r)
	if err != nil {
		return nil, err
	}
	return connection.do(req)
}

func (connection *ChfConnection) do(req *http.Request) (resp *http.Response, err error) {
	req.Header.Add("user-agent", "go-http-client")
	req.Header.Add("content-type", httpContentTypeApplicationJson)
	resp, err = connection.httpClient.Do(req)
	defer func() {
		if resp != nil {
			processedChargingRequestsCounter.WithLabelValues(fmt.Sprintf("%d", resp.StatusCode), req.Method).Add(1)
		}

		if resp == nil {
			unknownFailures.Inc()
		}
	}()
	return
}

```

When the rate of requests is as low as about 2000 per second, then I do not see my client sending RST_STREAM to the server.

However, when the rate is about 2500-3000 per second, I see the following in my program log file (after enabling ```GODEBUG="http2debug=2"``` ):
```
2022/05/10 21:59:12.827474 http2: Framer 0xc23f448000: wrote HEADERS flags=END_HEADERS stream=247467 len=65
2022/05/10 21:59:12.896018 http2: Framer 0xc23f448000: wrote DATA stream=247467 len=75 data="{\n\t\"invocationSequenceNumber\": 1,\n\t\"invocationTimeStamp\": \"2022-05-10T11:59"
2022/05/10 21:59:12.996612 RoundTrip failure: context deadline exceeded
2022/05/10 21:59:12.996692 http2: Framer 0xc23f448000: wrote RST_STREAM stream=247467 len=4 ErrCode=CANCEL
```

I am confused as to which context deadline is being exceeded ?

The client timeout that I have set in my HTTP client is 5 seconds whereas based on the log messages above, the time between the last part of the DATA frame and the time when I see the context deadline is just about 100 ms !!!

### What did you expect to see?
No such errors


### What did you see instead?
As mentioned above, I see Roundtrip timeout errors. This leads to message/request rate to be much lower (about 30% lower) than what I expect it to be.