Repository: kubernetes
Issue: Unrecognized internal errors in apiserver when etcd server is restarted
Labels: kind/bug,sig/api-machinery,lifecycle/frozen

<!-- This form is for bug reports and feature requests ONLY!

If you're looking for help check [Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes) and the [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/).

If the matter is security related, please disclose it privately via https://kubernetes.io/security/.
-->

**Is this a BUG REPORT or FEATURE REQUEST?**:

> Uncomment only one, leave it on its own line:
>
/kind bug
> /kind feature


**What happened**:
I have three machines dedicated to running etcd servers (as systemd services), seven machines dedicated to being kubernetes masters, and ten machines dedicated to being kubernetes worker nodes.  Additionally I configure the etcd server nodes to also be kube workers, to make it easy to gather Prometheus metrics from the etcd servers.  Lately I have been testing reboots of the etcd server machines.  See https://docs.google.com/document/d/1eXbdC5Hf-cwsWwCpM1jqTIJBsBOVBsUVqnBsx3vruHw for a run that went well.  Then I tried it with a higher load on the system.  See https://docs.google.com/document/d/15wh16AhMUpNAL9yORc75Ij0YC9RwXN8rYdhsmblq9UI for full details.  The load is generated by a simple client that creates, updates, and deletes ConfigMap objects.  In this case I ran it at 1000 Kube API operations per second.  During one burst of update operations, the client recorded  error responses to 19 out of the 150 updates in that burst (find the client's log in the data archive, find its schema in the Experiment 1 main page).  The burst happened just as the etcd machine was going down, and lasted for something under 60 milliseconds.  Three sorts of error messages were recorded: `grpc: the client connection is closing`, `rpc error: code = Unavailable desc = transport is closing`, and `rpc error: code = Unavailable desc = the server stops accepting new RPCs`.

**What you expected to happen**:
I expected the kube apiservers would internally handle the complications due to an etcd machine going down, not return an error to the kubenetes clients.

**How to reproduce it (as minimally and precisely as possible)**:
I do not yet have a lot of experience with this scenario, so am not sure what are the critical elements.  The most obvious elements are taking an etcd server machine down while the system is under high write load, in a multi-master configuration.

**Anything else we need to know?**:
Here is where to look in the apiserver logs:
```
root@mjstest1:~# egrep -c 'the client connection is closing|transport is closing|the server stops accepting new RPCs' 0613.0940.8148*server*stderr*log
0613.0940.8148-kube-apiserver-mjs-api-dal10s-a-stderr.log:10
0613.0940.8148-kube-apiserver-mjs-api-dal10s-b-stderr.log:0
0613.0940.8148-kube-apiserver-mjs-api-dal10s-c-stderr.log:14
0613.0940.8148-kube-apiserver-mjs-api-dal10s-d-stderr.log:0
0613.0940.8148-kube-apiserver-mjs-api-dal10s-e-stderr.log:0
0613.0940.8148-kube-apiserver-mjs-api-dal10s-f-stderr.log:14
0613.0940.8148-kube-apiserver-mjs-api-dal10s-g-stderr.log:0
```
Intriguingly, that shows 2*19 matches.

**Environment**:
- Kubernetes version (use `kubectl version`):
```
root@mjs-api-dal10s-c:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.4-1+cff305847e8117-dirty", GitCommit:"cff305847e811748a902ffed70cbc505b3667bd4", GitTreeState:"dirty", BuildDate:"2018-06-12T14:47:46Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
```
The server code is https://github.com/MikeSpreitzer/kubernetes/tree/fix-63608-b-on-1.10.4 , in the Docker container image `mspreitz/hyperkube-amd64:v1.10.4.1`.

- Cloud provider or hardware configuration: no Kube "cloud provider" used; hardware as mentioned above, physical for the etcd servers and kube masters, virtual (xen) for the plain kube workers.

- OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS on the etcd and kube servers

- Kernel (e.g. `uname -a`): `Linux mjs-api-dal10s-c 4.4.0-127-generic #153-Ubuntu SMP Sat May 19 10:58:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux`

- Install tools: private Ansible playbook that deploys etcd as systemd services and drives kubeadm to deploy Kubernetes

- etcd servers: release 3.3.7

- Others: Follow pointers from earlier pages to more details, including the client that drove the load