Repository: go
Issue: runtime: performance degradation in high-load long running applications
Labels: NeedsInvestigation,compiler/runtime

### What version of Go are you using (`go version`)?

<pre>
$ go version
go version go1.12.5 linux/amd64
</pre>

### Does this issue reproduce with the latest release?

yes

### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/spin/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/.../go"
GOPROXY=""
GORACE=""
GOROOT="/.../1.12.5/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.12.5/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/q5/b4nzm8zj3g5gcjs4rhkpvczh0000gn/T/go-build533400238=/tmp/go-build -gno-record-gcc-switches -fno-common"
</pre></details>

### What did you do?

My company spotted serious performance degradation in one of high load services. Service handles about 100K TCP permanent connections and using complex processing for about 75K partners requests per second. When we tried to upgrade Go from 1.11.6 to 1.12.5 we noted that performance fell by a third. Service begins seriously misbehave after about 15-30 mins after start.

After investigation I found that performance degradation caused by commits `1678b2c5..873bd47d`. Commits `1678b2c5..457c8f4f` causes slight performance degradation. Performance of service builded with `873bd47d` and later is absolutely horrible as well as memory consumption. Also I noted that CPU load from `873bd47d` reduced from 100% to 67%. Last commit with acceptable performance is `6e9fb11b3`.

Very strange benchmark:

```
func BenchmarkWrite(b *testing.B) {
	f, err := os.Create("/dev/null")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	buf := make([]byte, 1024*100)
	b.ResetTimer()
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			for i:=0;i<100000;i++ {
				f.SetDeadline(time.Now().Add(5*time.Minute))
				f.Write(buf)
				f.SetDeadline(time.Time{})
				runtime.GC()
			}
		}
	})
	b.StopTimer()
}
```

Maybe not of all ops makes sense but the results speak for themselves. Benchmarks are executed with Go built from source `6e9fb11b..873bd47d`. Here is brief results. You can find details in attached archives.

`6e9fb11b` (OK):
```
$ CGO_ENABLED=0 $HOME/go.dev/bin/go test -run=XXX -bench=BenchmarkWrite ./...
goos: linux
goarch: amd64
pkg: ...
BenchmarkWrite-2   	       1	17839662490 ns/op	   93816 B/op	      61 allocs/op
PASS
```

`873bd47d` (degraded):
```
...
goos: linux
goarch: amd64
pkg: ...
BenchmarkWrite-2   	       1	41872237656 ns/op	 6708104 B/op	   68744 allocs/op
PASS
```

Detailed results:
[0-6e9fb11b.zip](https://github.com/golang/go/files/3245824/0-6e9fb11b.zip)
[1-1678b2c5.zip](https://github.com/golang/go/files/3245825/1-1678b2c5.zip)
[2-143b13ae.zip](https://github.com/golang/go/files/3245826/2-143b13ae.zip)
[3-457c8f4f.zip](https://github.com/golang/go/files/3245827/3-457c8f4f.zip)
[4-873bd47.zip](https://github.com/golang/go/files/3245828/4-873bd47.zip)

Referred commits:
- https://github.com/golang/go/commit/1678b2c580584fa1ea8de3c14df0d8d77b6f7387
- https://github.com/golang/go/commit/143b13ae82d81020dfa6db40818bef5a1f701c3f
- https://github.com/golang/go/commit/457c8f4fe9e4d45f97d0a3f3c4a80789c6616fd6
- https://github.com/golang/go/commit/873bd47dfb34ba4416d4df30180905250b91f137

### What did you expect to see?

Normal performance like in Go 1.11.6. 

### What did you see instead?

Big performance degradation with service built with Go 1.12+