Repository: go
Issue: x/net/http2: Stuck http requests
Labels: NeedsInvestigation

### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.18.7 linux/amd64
</pre>

### Does this issue reproduce with the latest release?
Haven't tried


### What did you do?

We're running a http server which uses a http client to proxy the request body to another server (using h2c) and return its response body, we've recently noticed there's sometimes a bunch of requests getting stuck waiting for a sync.Cond, unfortunately I do not have a reproducer (this works fine 99.9% of the time), but I do have goroutine stacktraces from both the servers. Hoping someone more familiar with the http code internals might offer some help, as from what I can tell from the stacktraces, the client is waiting for the server to do something and the server is waiting for the client (ie looks like a deadlock).

One thing to note is that only requests which are proxied directly (io.Copy from the server Request into the client Request) get stuck, other requests where the body is constructed in memory were never affected.

Here's stacktraces from the server doing the proxying (some lines containing private code trimmed):

```
goroutine profile: total 806
242 @ 0x43d156 0x435b37 0x467da9 0x4e1812 0x4e2b7a 0x4e2b68 0x548689 0x55b705 0x75591f 0x46d9c1
#	0x467da8	internal/poll.runtime_pollWait+0x88		runtime/netpoll.go:302
#	0x4e1811	internal/poll.(*pollDesc).wait+0x31		internal/poll/fd_poll_runtime.go:83
#	0x4e2b79	internal/poll.(*pollDesc).waitRead+0x259	internal/poll/fd_poll_runtime.go:88
#	0x4e2b67	internal/poll.(*FD).Read+0x247			internal/poll/fd_unix.go:167
#	0x548688	net.(*netFD).Read+0x28				net/fd_posix.go:55
#	0x55b704	net.(*conn).Read+0x44				net/net.go:183
#	0x75591e	net/http.(*connReader).backgroundRead+0x3e	net/http/server.go:672

242 @ 0x43d156 0x44cbf2 0x905589 0x904b7e 0x46d9c1
#	0x905588	golang.org/x/net/http2.(*clientStream).writeRequest+0x9c8	golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/transport.go:1355
#	0x904b7d	golang.org/x/net/http2.(*clientStream).doRequest+0x1d		golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/transport.go:1217

⬇️ these requests are stuck
201 @ 0x43d156 0x4698bd 0x46989d 0x475cec 0x8efa6b 0x90bf25 0x4d8cf2 0x920b32 0x920b0c 0x132afe2 0x1234c7b 0x12dccce 0x122b084 0x12f64f5 0x122c2fe 0x122a570 0x923e71 0x122e098 0x9702a5 0x96f872 0x12df80b 0x924e52 0x122dc3b 0x122f511 0x924c7c 0x76057b 0x75ba37 0x46d9c1
#	0x46989c	sync.runtime_notifyListWait+0x11c							runtime/sema.go:513
#	0x475ceb	sync.(*Cond).Wait+0x8b									sync/cond.go:56
#	0x8efa6a	golang.org/x/net/http2.(*pipe).Read+0xea						golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/pipe.go:76
#	0x90bf24	golang.org/x/net/http2.transportResponseBody.Read+0x84					golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/transport.go:2417
#	0x4d8cf1	io.copyBuffer+0x1b1									io/io.go:426
#	0x920b31	io.Copy+0x71										io/io.go:385
#	0x920b0b	github.com/labstack/echo/v4.(*context).Stream+0x4b					github.com/labstack/echo/v4@v4.5.0/context.go:561
#	0x132afe1	<private_pkg>.handleEndpoint+0xd81				private.go:234
...
#	0x924e51	github.com/labstack/echo/v4.(*Echo).ServeHTTP.func1+0x131				github.com/labstack/echo/v4@v4.5.0/echo.go:648
#	0x122dc3a	github.com/labstack/echo/v4/middleware.MethodOverrideWithConfig.func1.1+0xba		github.com/labstack/echo/v4@v4.5.0/middleware/method_override.go:65
#	0x122f510	github.com/labstack/echo/v4/middleware.RemoveTrailingSlashWithConfig.func1.1+0x210	github.com/labstack/echo/v4@v4.5.0/middleware/slash.go:118
#	0x924c7b	github.com/labstack/echo/v4.(*Echo).ServeHTTP+0x3bb					github.com/labstack/echo/v4@v4.5.0/echo.go:654
#	0x76057a	net/http.serverHandler.ServeHTTP+0x43a							net/http/server.go:2916
#	0x75ba36	net/http.(*conn).serve+0x5d6								net/http/server.go:1966

35 @ 0x43d156 0x4698bd 0x46989d 0x475cec 0x8efa6b 0x90bf25 0x570354 0x4d87ba 0x6f9105 0x6f90b7 0x6f8d1b 0x6f89fe 0x133580f 0x132bdb9 0x1234c7b 0x12dccce 0x122b084 0x12f64f5 0x122c2fe 0x122a570 0x923e71 0x122e098 0x9702a5 0x96f872 0x12df80b 0x924e52 0x122dc3b 0x122f511 0x924c7c 0x76057b 0x75ba37 0x46d9c1
#	0x46989c	sync.runtime_notifyListWait+0x11c							runtime/sema.go:513
#	0x475ceb	sync.(*Cond).Wait+0x8b									sync/cond.go:56
#	0x8efa6a	golang.org/x/net/http2.(*pipe).Read+0xea						golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/pipe.go:76
#	0x90bf24	golang.org/x/net/http2.transportResponseBody.Read+0x84					golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/transport.go:2417
#	0x570353	bufio.(*Reader).Read+0x1b3								bufio/bufio.go:236
#	0x4d87b9	io.ReadAtLeast+0x99									io/io.go:331
#	0x6f9104	io.ReadFull+0xa4									io/io.go:350
#	0x6f90b6	compress/gzip.(*Reader).readHeader+0x56							compress/gzip/gunzip.go:175
#	0x6f8d1a	compress/gzip.(*Reader).Reset+0x2ba							compress/gzip/gunzip.go:113
#	0x6f89fd	compress/gzip.NewReader+0x3d								compress/gzip/gunzip.go:94
#	0x133580e	<private_pkg>.createRequest+0x28e			private.go:1225
...
#	0x924e51	github.com/labstack/echo/v4.(*Echo).ServeHTTP.func1+0x131				github.com/labstack/echo/v4@v4.5.0/echo.go:648
#	0x122dc3a	github.com/labstack/echo/v4/middleware.MethodOverrideWithConfig.func1.1+0xba		github.com/labstack/echo/v4@v4.5.0/middleware/method_override.go:65
#	0x122f510	github.com/labstack/echo/v4/middleware.RemoveTrailingSlashWithConfig.func1.1+0x210	github.com/labstack/echo/v4@v4.5.0/middleware/slash.go:118
#	0x924c7b	github.com/labstack/echo/v4.(*Echo).ServeHTTP+0x3bb					github.com/labstack/echo/v4@v4.5.0/echo.go:654
#	0x76057a	net/http.serverHandler.ServeHTTP+0x43a							net/http/server.go:2916
#	0x75ba36	net/http.(*conn).serve+0x5d6								net/http/server.go:1966

17 @ 0x43d156 0x435b37 0x467da9 0x4e1812 0x4e2b7a 0x4e2b68 0x548689 0x55b705 0x755ded 0x56fd83 0x56fedd 0x757338 0x75b78b 0x46d9c1
#	0x467da8	internal/poll.runtime_pollWait+0x88		runtime/netpoll.go:302
#	0x4e1811	internal/poll.(*pollDesc).wait+0x31		internal/poll/fd_poll_runtime.go:83
#	0x4e2b79	internal/poll.(*pollDesc).waitRead+0x259	internal/poll/fd_poll_runtime.go:88
#	0x4e2b67	internal/poll.(*FD).Read+0x247			internal/poll/fd_unix.go:167
#	0x548688	net.(*netFD).Read+0x28				net/fd_posix.go:55
#	0x55b704	net.(*conn).Read+0x44				net/net.go:183
#	0x755dec	net/http.(*connReader).Read+0x16c		net/http/server.go:780
#	0x56fd82	bufio.(*Reader).fill+0x102			bufio/bufio.go:106
#	0x56fedc	bufio.(*Reader).Peek+0x5c			bufio/bufio.go:144
#	0x757337	net/http.(*conn).readRequest+0x1f7		net/http/server.go:985
#	0x75b78a	net/http.(*conn).serve+0x32a			net/http/server.go:1891

7 @ 0x43d156 0x44cbf2 0x778a95 0x46d9c1
#	0x778a94	net/http.(*persistConn).writeLoop+0xf4	net/http/transport.go:2392

6 @ 0x43d156 0x4698bd 0x46989d 0x475cec 0x8efa6b 0x90bf25 0x7df5df 0x7df1db 0x7dee38 0x11d5f5a 0x11d69c7 0x11f3a7d 0x132adae 0x1234c7b 0x12dccce 0x122b084 0x12f64f5 0x122c2fe 0x122a570 0x923e71 0x122e098 0x9702a5 0x96f872 0x12df80b 0x924e52 0x122dc3b 0x122f511 0x924c7c 0x76057b 0x75ba37 0x46d9c1
#	0x46989c	sync.runtime_notifyListWait+0x11c							runtime/sema.go:513
#	0x475ceb	sync.(*Cond).Wait+0x8b									sync/cond.go:56
#	0x8efa6a	golang.org/x/net/http2.(*pipe).Read+0xea						golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/pipe.go:76
#	0x90bf24	golang.org/x/net/http2.transportResponseBody.Read+0x84					golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/transport.go:2417
#	0x7df5de	encoding/json.(*Decoder).refill+0x17e							encoding/json/stream.go:165
#	0x7df1da	encoding/json.(*Decoder).readValue+0xba							encoding/json/stream.go:140
#	0x7dee37	encoding/json.(*Decoder).Decode+0x77							encoding/json/stream.go:63
#	0x11d5f59	<private_pkg>.doRequest+0x119				private.go:204
...
#	0x924e51	github.com/labstack/echo/v4.(*Echo).ServeHTTP.func1+0x131				github.com/labstack/echo/v4@v4.5.0/echo.go:648
#	0x122dc3a	github.com/labstack/echo/v4/middleware.MethodOverrideWithConfig.func1.1+0xba		github.com/labstack/echo/v4@v4.5.0/middleware/method_override.go:65
#	0x122f510	github.com/labstack/echo/v4/middleware.RemoveTrailingSlashWithConfig.func1.1+0x210	github.com/labstack/echo/v4@v4.5.0/middleware/slash.go:118
#	0x924c7b	github.com/labstack/echo/v4.(*Echo).ServeHTTP+0x3bb					github.com/labstack/echo/v4@v4.5.0/echo.go:654
#	0x76057a	net/http.serverHandler.ServeHTTP+0x43a							net/http/server.go:2916
#	0x75ba36	net/http.(*conn).serve+0x5d6								net/http/server.go:1966

4 @ 0x43d156 0x435b37 0x467da9 0x4e1812 0x4e2b7a 0x4e2b68 0x548689 0x55b705 0x570354 0x4d87ba 0x8e708e 0x8e7048 0x8e78f5 0x909f70 0x9094af 0x46d9c1
#	0x467da8	internal/poll.runtime_pollWait+0x88			runtime/netpoll.go:302
#	0x4e1811	internal/poll.(*pollDesc).wait+0x31			internal/poll/fd_poll_runtime.go:83
#	0x4e2b79	internal/poll.(*pollDesc).waitRead+0x259		internal/poll/fd_poll_runtime.go:88
#	0x4e2b67	internal/poll.(*FD).Read+0x247				internal/poll/fd_unix.go:167
#	0x548688	net.(*netFD).Read+0x28					net/fd_posix.go:55
#	0x55b704	net.(*conn).Read+0x44					net/net.go:183
#	0x570353	bufio.(*Reader).Read+0x1b3				bufio/bufio.go:236
#	0x4d87b9	io.ReadAtLeast+0x99					io/io.go:331
#	0x8e708d	io.ReadFull+0x6d					io/io.go:350
#	0x8e7047	golang.org/x/net/http2.readFrameHeader+0x27		golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/frame.go:237
#	0x8e78f4	golang.org/x/net/http2.(*Framer).ReadFrame+0x94		golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/frame.go:498
#	0x909f6f	golang.org/x/net/http2.(*clientConnReadLoop).run+0x12f	golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/transport.go:2134
#	0x9094ae	golang.org/x/net/http2.(*ClientConn).readLoop+0x6e	golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/transport.go:2030

4 @ 0x43d156 0x435b37 0x467da9 0x4e1812 0x4e2b7a 0x4e2b68 0x548689 0x55b705 0x775f8e 0x56fd83 0x56fedd 0x776dac 0x46d9c1
#	0x467da8	internal/poll.runtime_pollWait+0x88		runtime/netpoll.go:302
#	0x4e1811	internal/poll.(*pollDesc).wait+0x31		internal/poll/fd_poll_runtime.go:83
#	0x4e2b79	internal/poll.(*pollDesc).waitRead+0x259	internal/poll/fd_poll_runtime.go:88
#	0x4e2b67	internal/poll.(*FD).Read+0x247			internal/poll/fd_unix.go:167
#	0x548688	net.(*netFD).Read+0x28				net/fd_posix.go:55
#	0x55b704	net.(*conn).Read+0x44				net/net.go:183
#	0x775f8d	net/http.(*persistConn).Read+0x4d		net/http/transport.go:1929
#	0x56fd82	bufio.(*Reader).fill+0x102			bufio/bufio.go:106
#	0x56fedc	bufio.(*Reader).Peek+0x5c			bufio/bufio.go:144
#	0x776dab	net/http.(*persistConn).readLoop+0x1ab		net/http/transport.go:2093

3 @ 0x43d156 0x435b37 0x467da9 0x4e1812 0x4e2b7a 0x4e2b68 0x548689 0x55b705 0x6a2fdd 0x50e5b8 0x6a31c5 0x6a07d6 0x6a622f 0x6a6230 0x775f8e 0x56fd83 0x56fedd 0x776dac 0x46d9c1
#	0x467da8	internal/poll.runtime_pollWait+0x88		runtime/netpoll.go:302
#	0x4e1811	internal/poll.(*pollDesc).wait+0x31		internal/poll/fd_poll_runtime.go:83
#	0x4e2b79	internal/poll.(*pollDesc).waitRead+0x259	internal/poll/fd_poll_runtime.go:88
#	0x4e2b67	internal/poll.(*FD).Read+0x247			internal/poll/fd_unix.go:167
#	0x548688	net.(*netFD).Read+0x28				net/fd_posix.go:55
#	0x55b704	net.(*conn).Read+0x44				net/net.go:183
#	0x6a2fdc	crypto/tls.(*atLeastReader).Read+0x3c		crypto/tls/conn.go:785
#	0x50e5b7	bytes.(*Buffer).ReadFrom+0x97			bytes/buffer.go:204
#	0x6a31c4	crypto/tls.(*Conn).readFromUntil+0xe4		crypto/tls/conn.go:807
#	0x6a07d5	crypto/tls.(*Conn).readRecordOrCCS+0x115	crypto/tls/conn.go:614
#	0x6a622e	crypto/tls.(*Conn).readRecord+0x16e		crypto/tls/conn.go:582
#	0x6a622f	crypto/tls.(*Conn).Read+0x16f			crypto/tls/conn.go:1285
#	0x775f8d	net/http.(*persistConn).Read+0x4d		net/http/transport.go:1929
#	0x56fd82	bufio.(*Reader).fill+0x102			bufio/bufio.go:106
#	0x56fedc	bufio.(*Reader).Peek+0x5c			bufio/bufio.go:144
#	0x776dab	net/http.(*persistConn).readLoop+0x1ab		net/http/transport.go:2093
```

And here's stacktraces from the h2c server:
```
goroutine profile: total 287
⬇️ these are the stuck ones (as you can see this is after the http Handler returns, which seems to point to the client being in bad state
243 @ 0x43d0d6 0x44cb72 0xcba285 0xcc1e65 0xcc175e 0x565f02 0xcc260f 0xcc3318 0xcc0c79 0xcc09f7 0x46db41
#	0xcba284	golang.org/x/net/http2.(*serverConn).writeDataFromHandler+0x204	golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/server.go:1064
#	0xcc1e64	golang.org/x/net/http2.(*responseWriterState).writeChunk+0x4c4	golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/server.go:2589
#	0xcc175d	golang.org/x/net/http2.chunkWriter.Write+0x1d			golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/server.go:2469
#	0x565f01	bufio.(*Writer).Flush+0x61					bufio/bufio.go:628
#	0xcc260e	golang.org/x/net/http2.(*responseWriter).Flush+0x2e		golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/server.go:2669
#	0xcc3317	golang.org/x/net/http2.(*responseWriter).handlerDone+0x37	golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/server.go:2832
#	0xcc0c78	golang.org/x/net/http2.(*serverConn).runHandler.func1+0x218	golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/server.go:2253
#	0xcc09f6	golang.org/x/net/http2.(*serverConn).runHandler+0x96		golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/server.go:2257

4 @ 0x43d0d6 0x435ab7 0x467e49 0x4d8632 0x4d999a 0x4d9988 0x5c9f49 0x5dc685 0x6f95ad 0x564303 0x564eef 0x565147 0x685a19 0x6f4979 0x6f497a 0x6fab4a 0x6fef4b 0x46db41
#	0x467e48	internal/poll.runtime_pollWait+0x88		runtime/netpoll.go:302
#	0x4d8631	internal/poll.(*pollDesc).wait+0x31		internal/poll/fd_poll_runtime.go:83
#	0x4d9999	internal/poll.(*pollDesc).waitRead+0x259	internal/poll/fd_poll_runtime.go:88
#	0x4d9987	internal/poll.(*FD).Read+0x247			internal/poll/fd_unix.go:167
#	0x5c9f48	net.(*netFD).Read+0x28				net/fd_posix.go:55
#	0x5dc684	net.(*conn).Read+0x44				net/net.go:183
#	0x6f95ac	net/http.(*connReader).Read+0x16c		net/http/server.go:780
#	0x564302	bufio.(*Reader).fill+0x102			bufio/bufio.go:106
#	0x564eee	bufio.(*Reader).ReadSlice+0x2e			bufio/bufio.go:371
#	0x565146	bufio.(*Reader).ReadLine+0x26			bufio/bufio.go:400
#	0x685a18	net/textproto.(*Reader).readLineSlice+0x98	net/textproto/reader.go:57
#	0x6f4978	net/textproto.(*Reader).ReadLine+0x78		net/textproto/reader.go:38
#	0x6f4979	net/http.readRequest+0x79			net/http/request.go:1029
#	0x6fab49	net/http.(*conn).readRequest+0x249		net/http/server.go:988
#	0x6fef4a	net/http.(*conn).serve+0x32a			net/http/server.go:1891

4 @ 0x43d0d6 0x435ab7 0x467e49 0x4d8632 0x4d999a 0x4d9988 0x5c9f49 0x5dc685 0xdc0f84 0x4b11da 0xcac56e 0xcac528 0xcacdd5 0xcb7fb1 0x46db41
#	0x467e48	internal/poll.runtime_pollWait+0x88			runtime/netpoll.go:302
#	0x4d8631	internal/poll.(*pollDesc).wait+0x31			internal/poll/fd_poll_runtime.go:83
#	0x4d9999	internal/poll.(*pollDesc).waitRead+0x259		internal/poll/fd_poll_runtime.go:88
#	0x4d9987	internal/poll.(*FD).Read+0x247				internal/poll/fd_unix.go:167
#	0x5c9f48	net.(*netFD).Read+0x28					net/fd_posix.go:55
#	0x5dc684	net.(*conn).Read+0x44					net/net.go:183
#	0xdc0f83	golang.org/x/net/http2/h2c.(*bufConn).Read+0xa3		golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/h2c/h2c.go:214
#	0x4b11d9	io.ReadAtLeast+0x99					io/io.go:331
#	0xcac56d	io.ReadFull+0x6d					io/io.go:350
#	0xcac527	golang.org/x/net/http2.readFrameHeader+0x27		golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/frame.go:237
#	0xcacdd4	golang.org/x/net/http2.(*Framer).ReadFrame+0x94		golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/frame.go:498
#	0xcb7fb0	golang.org/x/net/http2.(*serverConn).readFrames+0x90	golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/server.go:786

4 @ 0x43d0d6 0x44cb72 0xcb8e6c 0xcb72ac 0xdc029d 0xdc9527 0x70074f 0x703d3b 0x6ff1f7 0x46db41
#	0xcb8e6b	golang.org/x/net/http2.(*serverConn).serve+0x88b		golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/server.go:900
#	0xcb72ab	golang.org/x/net/http2.(*Server).ServeConn+0xb8b		golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/server.go:504
#	0xdc029c	golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP+0x2bc		golang.org/x/net@v0.0.0-20220909164309-bea034e7d591/http2/h2c/h2c.go:88
#	0xdc9526	<private_pkg>.NewServer.func1+0x86	private.go:59
#	0x70074e	net/http.HandlerFunc.ServeHTTP+0x2e				net/http/server.go:2084
#	0x703d3a	net/http.serverHandler.ServeHTTP+0x43a				net/http/server.go:2916
#	0x6ff1f6	net/http.(*conn).serve+0x5d6					net/http/server.go:1966
```