Repository: go
Issue: x/net/http2: steady performance degeneration seen with HTTP/2 over recent releases
Labels: Performance,NeedsInvestigation

Hello, I work on the full text search engine at couchbase, and we recently moved our scatter gather protocol to use x/net/http2 instead of net/http.

The go version we currently use is go-1.9.3.

In some testing in house .. I've noticed the query throughput over a full text index take a nose dive as I upgraded the x/net release version used. (These numbers are over a system that runs on centos7)

 | branch                            | query throughput |
 | ----------------------- | ----------------- |
 | release-branch.go1.6    | 27734 q/s             |
 | release-branch.go1.7    | 17037 q/s              |
 | release-branch.go1.8    | 15139 q/s              |
 | release-branch.go1.9    | 14836 q/s             |
 | release-branch.go1.10  | 9500 q/s               |
 | master                            | 8336 q/s               |

This is how the scatter-gather client is set up in our code using  x/net/http2's transport:

    transport2 := &http2.Transport{
		TLSClientConfig: clientTLSConfig,
    }
    cbft.Http2Client = &http.Client{Transport: transport2}

And here's how the server is setup, which also uses the x/net/netutil to set up a LimitListener: 

    listener, _ := net.Listen("tcp", bindHTTP)
    server := &http.Server{Addr: bindHTTP,
	    Handler:      routerInUse,
	    ReadTimeout:  httpReadTimeout,
	    WriteTimeout: httpWriteTimeout
    }
    ...
    keepAliveListener := tcpKeepAliveListener{listener.(*net.TCPListener)}
    limitListener := netutil.LimitListener(keepAliveListener, httpMaxConnections)
    tlsListener := tls.NewListener(limitListener, serverTLSConfig)
    ...
    _ = server.Serve(tlsListener)
 
**I was wondering if there're any known issues around this area, or if there're any additional settings that I'd have to tune in the later branches of x/net.**

I also tried writing up a simple server client benchmark unit test, based on your current tests to see if it'd point out anything obvious ..

```
func BenchmarkServerClient(b *testing.B) {
	b.ReportAllocs()

	const (
		itemSize = 1 << 10
		itemCount = 1000
	)

	st := newServerTester(b, func(w http.ResponseWriter, r *http.Request) {
		for i := 0; i < itemCount; i++ {
			_, err := w.Write(make([]byte, itemSize))
			if err != nil {
				return
			}
		}
	}, optOnlyServer)
	defer st.Close()

	tlsConfig := &tls.Config{
		InsecureSkipVerify: true,
	}

	tr := &Transport{TLSClientConfig: tlsConfig}
	defer tr.CloseIdleConnections()
	cl := &http.Client{Transport: tr}

	b.ResetTimer()

	for i := 0; i < b.N; i++ {
		resp, err := cl.Get(st.ts.URL)
		if err != nil {
			b.Fatal(err)
		}

		resp.Body.Close()
	}
}
```

And here're the results for the above benchmark test from my computer (on OSX 10.13.4):

```
release-branch.go1.6   	    BenchmarkServerClient-8   	    2000	    742842 ns/op	   18715 B/op	      64 allocs/op
release-branch.go1.7   	    BenchmarkServerClient-8   	    2000	    554580 ns/op	   20688 B/op	      83 allocs/op
release-branch.go1.8   	    BenchmarkServerClient-8   	    2000	    631001 ns/op	   20728 B/op	      84 allocs/op
release-branch.go1.9   	    BenchmarkServerClient-8   	    2000	    675503 ns/op	   19745 B/op	      87 allocs/op
release-branch.go1.10       BenchmarkServerClient-8   	    2000	    772618 ns/op	   19087 B/op	      89 allocs/op
master   	       	    BenchmarkServerClient-8   	    2000	    930152 ns/op	   19212 B/op	      89 allocs/op
```

So although I couldn't reproduce the issue accurately (with 1.6 as the anomaly here), there is some degeneration in time taken per op from 1.7 to master.

This is the go env output from my system:

```
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/abhinavdangeti/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/abhinavdangeti/Documents/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10.1/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10.1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/ln/vkg1npcj5tz0ps359y6fwmn80000gn/T/go-build266872403=/tmp/go-build -gno-record-gcc-switches -fno-common"
```

I'd appreciate any pointers or advise regarding what I'm observing here ..