Repository: kubernetes
Issue: existing toleration can not be modified when add annotation for daemonsetPod after enabling podtolerationrestriction
Labels: kind/bug,sig/api-machinery,triage/accepted

### What happened?

https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#taints-and-tolerations 
Although Daemon Pods respect [taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/), the following tolerations are added to DaemonSet Pods automatically according to the related features.

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podtolerationrestriction
podtolerationrestriction is enabled after the daemonset is enabled.

After enabling podtolerationrestriction, I cannot update a daemonset pod's annotation.

After some digging into the issue, we find it is caused by a toleration merge in podtolerationrestriction.

- There is a limitation of pod toleration update: you can add new toleration or update toleration tolerationSeconds.
- In this case, some tolerations are removed due to merging process. See reproduce steps below.

### What did you expect to happen?

If I update pod annotation and no toleration changes, the podtolerationrestriction plugin should not deny it for toleration change. 

There is also another solution, that we can merge the tolerations firstly when creating, so the podtolerationrestriction merge will not make any difference. However, this needs to recreate the pod as well.

### How can we reproduce it (as minimally and precisely as possible)?

**Step1: create a daemonset with NoSchedule Exists tolerations.**
If the daemonset is with one toleration `{"operator":"Exists","effect":"NoSchedule"}`,  the daemonset pod will be with tolerations like below.
```
"tolerations":[
    {"operator":"Exists","effect":"NoSchedule"},
    {"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute"},
    {"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute"},
    {"key":"node.kubernetes.io/disk-pressure","operator":"Exists","effect":"NoSchedule"},
    {"key":"node.kubernetes.io/memory-pressure","operator":"Exists","effect":"NoSchedule"},
    {"key":"node.kubernetes.io/pid-pressure","operator":"Exists","effect":"NoSchedule"},
    {"key":"node.kubernetes.io/unschedulable","operator":"Exists","effect":"NoSchedule"}
]
```

**Step2: enable podtolerationrestriction**
Then I enable admission controller [podtolerationrestriction](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podtolerationrestriction).

**Step 3: update the pod annotation**

> [root@dce-10-6-154-30 ~]# kubectl patch pod ansible-dao-2048-smktb -p '{"spec":{"anotations": {"a":"b"}}}'
> The Pod "ansible-dao-2048-smktb" is invalid: spec.tolerations: Forbidden: existing toleration can not be modified except its tolerationSeconds

The pod is a burstable pod and it will add TaintNodeMemoryPressure to the pod which already exists. It will trigger a toleration merge and the first ` {"operator":"Exists","effect":"NoSchedule"}` is already including the last four NoSchedule-Exists tolerations .

https://github.com/kubernetes/kubernetes/blob/5f1854ee57c7b5f36d8ae6fb865ca881764e5b07/plugin/pkg/admission/podtolerationrestriction/admission.go#L102-L108

### Anything else we need to know?

workaround: We delete/recreate the daemonset pod after enabling podtolerationrestriction.


### Kubernetes version

we can reproduce it in both v1.24 and v1.18.

### Cloud provider

vsphere

### OS version

Centos 7 

### Install tools

kubeadm & DaoCloud Enterprise.

### Container runtime (CRI) and version (if applicable)

docker or containerd

### Related plugins (CNI, CSI, ...) and versions (if applicable)

calico or parcel