Repository: kubernetes
Issue: ValidatingWebhookConfiguration causes deployments to maintain two underlying RS at the same time
Labels: kind/bug,priority/important-soon,sig/scheduling,sig/apps,lifecycle/frozen

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!-->

**What happened**:
I have ValidationWebhookConfiguration that checks the deployments for the availability of `spec.template.spec.tolerations` and throws an error if it does not have `tolerations`. Thus enforcing deployments to have `tolerations`. I enforce `tolerations` on staging namespaces only in order to enforce deployments to run on pre-emptive nodes. 

After I add the `tolerations` because of the above enforcement(ie.update the deployment with tolerations). The deployment maintains two of its underlying ReplicaSet. Thus keeping the old pods and new pods as well. It is important to note that I cannot simply delete old RS by --force too, it is just recreated again.

When I remove the  ValidationWebhookConfiguration, I have no problems, I can update the deployments with tolerations(or for that matter any other key/map) and the deployment maintains only one RS i.e., it has only new pods. I can also delete the old RS(if needed).

**Please find below, the ValidationWebhookConfiguration that I am using:-**
```
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: deny-absence-of-tolerations
webhooks:
  - name: deny.absence.of.tolerations
    rules:
      - apiGroups: ["apps","extensions"]
        apiVersions: ["v1","v1beta1","v1beta2"]
        operations: [ "CREATE","UPDATE" ]
        #operations: [ "*" ]
        resources: ["deployments","replicasets"]
    namespaceSelector:
      matchExpressions:
      - key: namespace
        operator: In
        values:
        - test-namespace
    failurePolicy: Fail
    clientConfig:
      url: "https://deny_absence_of_tolerations"
      caBundle: cabundlevaluehere 
```


**And the core part of my backend server goes like this:-**

```
  var admissionResponse = {
    allowed: false
  };

  var found = false;

  if (!object.spec.template.spec.tolerations) {

      console.log("Workload is not using tolerations");

      admissionResponse.status = {
        status: 'Failure',
          message: "On Staging/Testing please use tolerations",
        reason: " Workload ( ie.,deployment ) Requirement Failed",
        code: 402
      };

      found = true;

  };

  if (!found) {
    admissionResponse.allowed = true;
  }

  var admissionReview = {
    response: admissionResponse
  };

  res.setHeader('Content-Type', 'application/json');
  res.send(JSON.stringify(admissionReview));
  res.status(200).end();
```


 

**What you expected to happen**:
Expected behaviour of deployment i.e., deployment should maintain only one RS.

**How to reproduce it (as minimally and precisely as possible)**:
   1).   First of all, have no validationWebhookConfiguration.
   2).  Create a deployment(without tolerations) in a namespace(lets call it :- test-namespace)
   3).  Add validationWebhookConfiguration now to test-namespace in order to enforce tolerations.
   4).  Now update the  deployment with tolerations and apply it.
   5). As mentioned above in detail, now the ValidationWebhookConfiguration causes deployments to maintain two underlying RS at the same time


**Anything else we need to know?**: A complete delete of deployment and applying the updated deployment does not cause the above problem. But deleting a complete deployment is a downtime to consider.

**Environment**:
- Kubernetes version (use `kubectl version`): 1.10.6-gke.9
- Cloud provider or hardware configuration: Google Kubernetes Engine
- OS (e.g. from /etc/os-release): COS
- Kernel (e.g. `uname -a`): 4.14.56+ #1 SMP x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux
- Install tools:
- Others:

@kubernetes/sig-scheduling-bug
@kubernetes/sig-apps-bug
<!-- DO NOT EDIT BELOW THIS LINE -->
/kind bug