Repository: kubernetes
Issue: Kubelet does not sync pod updates for static pods
Labels: kind/bug,priority/backlog,sig/node,triage/accepted

### What happened?

When a static pod is created on a node, it's mirror pod is created on the apiserver. Patching the mirror pod on the apiserver triggers HandlePodUpdate function in the kubelet:
https://github.com/kubernetes/kubernetes/blob/fcf5d23e6818ccfbc92f332af82f31661f8d4f88/pkg/kubelet/kubelet.go#L2491-L2502
However, handleMirrorPod sets up SyncPodUpdate for the original pod instead of the updated pod.
https://github.com/kubernetes/kubernetes/blob/fcf5d23e6818ccfbc92f332af82f31661f8d4f88/pkg/kubelet/kubelet.go#L2412-L2420
As a result, SyncPod does not really process the updates for static pods.


### What did you expect to happen?

Pod updates should be processed for static pods in the same manner as they are processed for normal pods.

### How can we reproduce it (as minimally and precisely as possible)?

Start local cluster w/ latest master (as of Mar 14, 2023):
```bash
# echo > /tmp/kubelet.log && DBG=1 ./hack/local-up-cluster.sh
...
...
+++ [0314 14:50:37] WARNING: linux/arm will no longer be built/shipped by default, please build it explicitly if needed.
+++ [0314 14:50:37]          support for linux/arm will be removed in a subsequent release.
make: Entering directory '/root/go/src/k8s.io/master_k8s'
+++ [0314 14:50:37] WARNING: linux/arm will no longer be built/shipped by default, please build it explicitly if needed.
+++ [0314 14:50:37]          support for linux/arm will be removed in a subsequent release.
go version go1.20.2 linux/arm64
+++ [0314 14:50:38] Building go targets for linux/arm64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
...
...
To start using your cluster, you can open up another terminal/tab and run:

  export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
  cluster/kubectl.sh

Alternatively, you can write to the default kubeconfig:

  export KUBERNETES_PROVIDER=local

  cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/server-ca.crt
  cluster/kubectl.sh config set-credentials myself --client-key=/var/run/kubernetes/client-admin.key --client-certificate=/var/run/kubernetes/client-admin.crt
  cluster/kubectl.sh config set-context local --cluster=local --user=myself
  cluster/kubectl.sh config use-context local
  cluster/kubectl.sh
```

Create static pod:
```bash
# cat <<EOF > /var/run/kubernetes/static-pods/bbox.yaml
apiVersion: v1
kind: Pod
metadata:
  name: bbox
spec:
  restartPolicy: OnFailure
  terminationGracePeriodSeconds: 2
  containers:
  - name: ctr
    image: busybox:1.33
    imagePullPolicy: IfNotPresent
    command: ["tail", "-f", "/dev/null"]
EOF
```

Verify image in container status of the static pod:
```bash
# ./cluster/kubectl.sh get pod bbox-127.0.0.1 -ojson | jq .status.containerStatuses[0].image
"docker.io/library/busybox:1.33"
```

Patch container image:
```bash
# ./cluster/kubectl.sh patch pod bbox-127.0.0.1 --patch '{"spec":{"containers":[{"name":"ctr", "image":"busybox:1.34"}]}}'
pod/bbox-127.0.0.1 patched
```

Check pod spec of static pod and verify image has been updated:
```bash
# ./cluster/kubectl.sh get pod bbox-127.0.0.1 -ojson | jq .spec.containers[0].image
"busybox:1.34"
```

After some time, check container status of static pod:
```bash
# ./cluster/kubectl.sh get pod bbox-127.0.0.1 -ojson | jq .status.containerStatuses[0].image
"docker.io/library/busybox:1.33"
``` 

Repeat exercise with normal pod to verify that container restarts and image has been updated.

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
# ./cluster/kubectl.sh version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.0-alpha.3.573+a9008b502d32a5", GitCommit:"a9008b502d32a5dbacf835adb040419bf31f5c04", GitTreeState:"clean", BuildDate:"2023-03-14T14:50:38Z", GoVersion:"go1.20.2", Compiler:"gc", Platform:"linux/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.0-alpha.3.573+a9008b502d32a5", GitCommit:"a9008b502d32a5dbacf835adb040419bf31f5c04", GitTreeState:"clean", BuildDate:"2023-03-14T14:50:38Z", GoVersion:"go1.20.2", Compiler:"gc", Platform:"linux/arm64"}
```
</details>


### Cloud provider

<details>
Local ubuntu VM on mac M1
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
# uname -a
Linux vbuild 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:48:57 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>