Repository: kubernetes
Issue: Admission webhooks affected by dead tcp connections
Labels: kind/bug,priority/important-soon,area/client-libraries,sig/api-machinery,area/admission-control,lifecycle/frozen

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->
**What happened**:

I was doing some tests regarding mutating webhook (especially for istio), and i found a strange behavior.

Stopping a worker instance with a pod of the webhook running on it seems to break the admission controller for this webhook during approximatively 15 minutes.

After some debugging, the admission controller seems to cache the pod ip associated with this webhook service.


kubectl get events are full of:
```
0s          Warning   FailedCreate             replicaset/sleep-1-d64b54564   Error creating: Internal error occurred: failed calling admission webhook "sidecar-injector.morven.me": Post https://sidecar-injector-webhook-svc.webhook.svc:443/mutate?timeout=30s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
```

Solutions to mitigate these errors:
 * Remove the label sidecar-injector=enabled from the namespace, wait the scheduling of the pods, add the label again, and delete each pod to reschedule the pods (and the mutating webhook associated with it)


The logs of kubernetes master are full of things like:
```
I0717 17:00:35.211387 1 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"sleep", UID:"d5faa2cf-a8ad-11e9-83cc-0a4693bb62ee", APIVersion:"apps/v1", ResourceVersion:"5094155", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: Internal error occurred: failed calling admission webhook "sidecar-injector.morven.me": Post https://sidecar-injector-webhook-svc.webhook.svc:443/inject?timeout=30s: read tcp 10.50.1.212:58344->10.50.0.77:443: read: connection timed out
```

The 10.50.0.77 ip matches the ip of the pod which was killed because of the shutdown of the node.



**What you expected to happen**:

Mutating webhook admission controller should invalidate the cache associated with the webhook once the first timeout occurred.


**How to reproduce it (as minimally and precisely as possible)**:

On a running kubernetes cluster:
  1. Install a mutating webhook: https://github.com/morvencao/kube-mutating-webhook-tutorial#deploy  + an application using this webhook; TLDR:
```
kubectl create namespace webhook

git clone https://github.com/morvencao/kube-mutating-webhook-tutorial;

cd kube-mutating-webhook-tutorial;

./deployment/webhook-create-signed-cert.sh \
    --service sidecar-injector-webhook-svc \
    --secret sidecar-injector-webhook-certs \
    --namespace webhook

cat deployment/mutatingwebhook.yaml | \
    deployment/webhook-patch-ca-bundle.sh > \
    deployment/mutatingwebhook-ca-bundle.yaml


# Add failurePolicy: Fail to the mutating webhook configuration to trigger the timeout error, and update the namespace:
head -n 14 deployment/mutatingwebhook-ca-bundle.yaml

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector-webhook-cfg
  labels:
    app: sidecar-injector
webhooks:
  - name: sidecar-injector.morven.me
    failurePolicy: Fail
    clientConfig:
      service:
        name: sidecar-injector-webhook-svc
        namespace: webhook
        path: "/mutate"
...


kns webhook;
kubectl create -f deployment/configmap.yaml
kubectl create -f deployment/deployment.yaml
kubectl create -f deployment/service.yaml
kubectl create -f deployment/mutatingwebhook-ca-bundle.yaml

kns default;
kubectl label namespace default sidecar-injector=enabled
kubectl create -f deployment/nginxconfigmap.yaml

cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        sidecar-injector-webhook.morven.me/inject: "yes"
      labels:
        app: sleep
    spec:
      containers:
      - name: sleep
        image: tutum/curl
        command: ["/bin/sleep","infinity"]
        imagePullPolicy:
EOF

```

At the end, you should have something like :

```
kubectl get pods -n default; kubectl get pods -n webhook
NAME                              READY   STATUS     RESTARTS   AGE
sleep-d64b54564-zfhcx             2/2     Running    0          10s
NAME                                                  READY   STATUS    RESTARTS   AGE
sidecar-injector-webhook-deployment-c89cb69b6-n8kff   1/1     Running   0          76s

```

  2. Shutdown the worker where the mutating webhook is running. (kubectl get pods -o wide to see the node)
  3. Wait the node to be down (take a look at the rescheduling of the mutating webhook; kubectl get pods -n webhook)
  4. Scale up the deployment (in this case, the "sleep" deployment) or add some pods within a namespace with the label sidecar-injector=enabled:

```
k scale deployment -n default sleep --replicas=5
```

  4. Nothing happens, except "timeout" errors:
```
kubectl get events -n default
1s          Warning   FailedCreate              ReplicaSet   Error creating: Internal error occurred: failed calling admission webhook "sidecar-injector.morven.me": Post https://sidecar-injector-webhook-svc.webhook.svc:443/mutate?timeout=30s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
...
```

You have to wait 15min to see the pods be scheduled.


**Anything else we need to know?**:

Related issues on istio side :
   * Automatic injection fails in AWS  (closed) : https://github.com/istio/istio/issues/14267
   * Istio sidecar injection fails on scale down in EKS (open) : https://github.com/istio/istio/issues/13762
   * Rolling upgrade of the worker nodes results in brief outage of application pods (open) : https://github.com/istio/istio/issues/13840


Doing a kubectl drain on the specific node instead of a shutdown seems to work.


**Environment**:
- Kubernetes version (use `kubectl version`):
```
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-26T00:04:52Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-eks-d69f1b", GitCommit:"d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7", GitTreeState:"clean", BuildDate:"2019-02-28T20:26:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
```
- Cloud provider or hardware configuration:

Aws, Kubernetes Version 1.12 Platform Version eks.2

- OS (e.g: `cat /etc/os-release`):
- Kernel (e.g. `uname -a`):
- Install tools:
- Network plugin and version (if this is a network-related bug):
- Others: