Repository: go
Issue: net: interleaving sendfile and read calls are very slow
Labels: Performance,NeedsInvestigation

### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.16.2 linux/amd64
</pre>

### Does this issue reproduce with the latest release?

Yes.

### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.2"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/gocode/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2301853134=/tmp/go-build -gno-record-gcc-switches"

</pre></details>

### What did you do?

Calling `io.Copy` triggering `sendfile` multiple times on a socket with the same data size.

### What did you expect to see?

Each `io.Copy` call to take the same amount of time.

### What did you see instead?

The first `io.Copy` call is fast at around `500µs`. All following calls to `io.Copy` are slow at around `43ms`.

I have written a simple program to demonstrate the issue: https://gist.github.com/erikdubbelboer/3eebecc551245a3b1d5f06a9ef433654#file-bench-go

I have also included a C version of the server to show that implementing similar code in C doesn't have the issue: https://gist.github.com/erikdubbelboer/3eebecc551245a3b1d5f06a9ef433654#file-server-c

Output:
<pre>
$ go run bench.go
501.52µs       <-- fast
211.711399ms   <-- always super slow for some reason
48.073968ms    <-- still slow
43.86743ms
43.951452ms
47.955709ms
43.997812ms
43.916976ms
44.037359ms
48.032407ms
$ gcc -o server server.c
$ ./server &
[1] 44860
$ go run bench.go noserver
214.586µs
108.971µs
276.024µs
278.569µs
91.277µs
102.528µs
102.278µs
87.94µs
90.094µs
92.318µs
[1]+  Done  ./server
</pre>

Interesting is also that removing the 1 byte that is being send to the server first, and just having the server send all 10 files at once doesn't trigger the performance impact. Also not when replacing the reading of the 1 byte with a `time.Sleep(time.Millisecond)`. So it seems that something about the data coming from the other direction causes the followup sendfile calls to be slow.

The reason why I'm thinking this is related to `sendfile` is because when I use `io.Copy(writerOnly{conn}, fd)` with `writerOnly` from [`net/net.go`](https://github.com/golang/go/blob/0e85fd7561de869add933801c531bf25dee9561c/src/net/net.go#L647-L656) it doesn't affect performance. `writerOnly` as documented in the Go source doesn't implement `io.ReaderFrom` (like `net.TCPConn` does) and [doesn't trigger sendfile](https://github.com/golang/go/blob/98a902323f6406d39c068d60253a0872364041ac/src/net/tcpsock_posix.go#L52).

Not using `sendfile` at all is also fast:
```
$ go run bench-nosendfile.go 
324.237µs
165.15µs
154.409µs
173.415µs
158.407µs
146.023µs
144.038µs
140.452µs
159.669µs
156.783µs
```

This mostly impacts `net/http` file serving on keep-alive connections. This is why I included the 1 byte being send from the client to the server first as it imitates http behavior.