Repository: go
Issue: runtime: stack growth during critical section harms performance
Labels: Performance,NeedsInvestigation,compiler/runtime

Hello,

[etcd](https://github.com/coreos/etcd) observed significant performance regression with Go 1.7.3.

Details follow or please visit https://github.com/coreos/etcd/issues/6876.

<br>

### What version of Go are you using (`go version`)?

Go 1.6.3 and Go 1.7.3


<br>

### What operating system and processor architecture are you using (`go env`)?

```bash
$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/gyuho/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build467209749=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
```

Google Cloud VM with 8 vCPUs, 16 GB memory, ubuntu-1610

> gcloud compute instances create test-machine --custom-cpu=8 --custom-memory=16 --image-family=ubuntu-1610 --image-project=ubuntu-os-cloud --boot-disk-size=150 --boot-disk-type="pd-ssd" --zone us-west1-a


<br>

### What did you do?

##### Build etcd binaries, others

Build etcd binaries in different Go versions with etcd commit https://github.com/coreos/etcd/commit/faeeb2fc7514c5caf7a9a0cc03ac9ee2ff94438b.

Go version is the only difference!

```bash
mkdir -p ${GOPATH}/src/github.com/coreos
cd ${GOPATH}/src/github.com/coreos
git clone https://github.com/coreos/etcd.git
cd ${GOPATH}/src/github.com/coreos/etcd
git reset --hard faeeb2fc7514c5caf7a9a0cc03ac9ee2ff94438b

./build
./bin/etcd --version

go build -v ./cmd/tools/benchmark
./benchmark help
```

Or

```bash
wget https://storage.googleapis.com/etcd/tip/etcd-v3.0.2-go1.6.3
wget https://storage.googleapis.com/etcd/tip/etcd-v3.0.2-go1.7.3
wget https://storage.googleapis.com/etcd/tip/etcdctl-master-go1.7.3
wget https://storage.googleapis.com/etcd/tip/benchmark-master-go1.7.3

sudo chmod +x ./etcd-v3.0.2-go1.6.3      && ./etcd-v3.0.2-go1.6.3 --version
sudo chmod +x ./etcd-v3.0.2-go1.7.3      && ./etcd-v3.0.2-go1.7.3 --version
sudo chmod +x ./etcdctl-master-go1.7.3   && ./etcdctl-master-go1.7.3 --version
sudo chmod +x ./benchmark-master-go1.7.3 && ./benchmark-master-go1.7.3 help
```


##### Run single-node cluster, benchmark

Go 1.6.3 + master branch Go 1.7.3 client

```bash
rm -rf test.etcd
./etcd-v3.0.2-go1.6.3 --name test \
    --listen-client-urls http://localhost:2379 \
    --advertise-client-urls http://localhost:2379 \
    --listen-peer-urls http://localhost:2380 \
    --initial-advertise-peer-urls http://localhost:2380 \
    --initial-cluster test=http://localhost:2380 \
    --initial-cluster-token test-token \
    --initial-cluster-state new \
    --enable-pprof

ETCDCTL_API=3 ./etcdctl-master-go1.7.3 --endpoints=localhost:2379 put foo bar
ETCDCTL_API=3 ./etcdctl-master-go1.7.3 --endpoints=localhost:2379 get foo --consistency=s
./benchmark-master-go1.7.3 --endpoints=localhost:2379 --conns=100 --clients=1000 range foo --consistency=s --total=200000
```

Go 1.7.3 + master branch Go 1.7.3 client

```bash
rm -rf test.etcd
./etcd-v3.0.2-go1.7.3 --name test \
    --listen-client-urls http://localhost:2379 \
    --advertise-client-urls http://localhost:2379 \
    --listen-peer-urls http://localhost:2380 \
    --initial-advertise-peer-urls http://localhost:2380 \
    --initial-cluster test=http://localhost:2380 \
    --initial-cluster-token test-token \
    --initial-cluster-state new \
    --enable-pprof

ETCDCTL_API=3 ./etcdctl-master-go1.7.3 --endpoints=localhost:2379 put foo bar
ETCDCTL_API=3 ./etcdctl-master-go1.7.3 --endpoints=localhost:2379 get foo --consistency=s
./benchmark-master-go1.7.3 --endpoints=localhost:2379 --conns=100 --clients=1000 range foo --consistency=s --total=200000
```


<br>

### What did you expect to see?

etcd with Go 1.6.3 shows:

```
Summary:
  Total:	55.4612 secs.
  Requests/sec:	36061.2548
```


<br>

### What did you see instead?

etcd with Go 1.7.3 is 2x slower:

```
Summary:
  Total:	105.3251 secs.
  Requests/sec:	18988.8279
```

We expected Go 1.7.3 to be comparable or better.

<br>

Here's our profiling data

- [etcd-v3.0.2-go1.6.3-2M-serializable-reads-00.pdf](https://github.com/golang/go/files/607171/etcd-v3.0.2-go1.6.3-2M-serializable-reads-00.pdf)
- [etcd-v3.0.2-go1.7.3-2M-serializable-reads-00.pdf](https://github.com/golang/go/files/607172/etcd-v3.0.2-go1.7.3-2M-serializable-reads-00.pdf)

Note: etcd uses old `golang.org/x/net/http2` but same behavior happens with latest `http2`; see https://github.com/coreos/etcd/issues/6876#issuecomment-262292981.

Could anybody help us debug this around Go runtime?
Please let me know if more information is needed.

Thanks a lot in advance!