Repository: kubernetes
Issue: AppArmor fields dropped from higher-level workload spec (daemonset / replicaset / etc) if modified on skewed cluster
Labels: kind/support,sig/apps,sig/testing,needs-triage

### What happened?

1. Upgrade first node in 3-node cluster from 1.29 to 1.30
2. Modified a DaemonSet to migrate from `container.apparmor.security.beta.kubernetes.io` annotations to `appArmorProfile` field.
  Modification was done by running `helm upgrade` on the node that had been upgraded, with a chart that checks the apiserver minor version: https://github.com/cilium/cilium/blob/v1.16.3/install/kubernetes/cilium/templates/cilium-agent/daemonset.yaml#L87-L95
3. Daemonset controller running on skewed node (still on 1.29) patched the DaemonSet status subresource, and the `appArmorProfile` field was dropped from the resource
5. DaemonSet Pods crashloop due to missing profile

### What did you expect to happen?

Fields are not dropped from resources when cluster is in a supported version skew.

### How can we reproduce it (as minimally and precisely as possible)?

Attempt to add AppArmor fields to a controller-managed resource (daemonset/deployment/replicaset/etc) in a skewed 1.29/1.30 cluster. Specifically, the kube-controller-manager must still be down-level.

### Anything else we need to know?

This was observed while upgrading an RKE2 cluster using Cilium, whose chart migrates from annotations to the new field when it detects that it is talking to a 1.30 apiserver.

<details>

<summary>Audit logs showing daemonset controller status update dropping fields</summary>

`cat audit*.log | jq -s '. | sort_by(.stageTimestamp)[] | select((.verb != "get") and (.stage == "ResponseComplete") and (.responseObject.metadata.name == "cilium")) | {verb, requestURI, stageTimestamp, userAgent, "spec.template.spec.securityContext": .responseObject.spec.template.spec.securityContext, "spec.template.metadata.annotations": .responseObject.spec.template.metadata.annotations}'`

```json
{
  "verb": "create",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets?fieldManager=helm",
  "stageTimestamp": "2025-01-07T19:38:24.127192Z",
  "userAgent": "Helm/3.16.1",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/cilium-agent": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/clean-cilium-state": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/mount-cgroup": "unconfined",
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:38:24.177231Z",
  "userAgent": "kube-controller-manager/v1.29.11+rke2r1 (linux/amd64) kubernetes/960a2f0/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/cilium-agent": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/clean-cilium-state": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/mount-cgroup": "unconfined",
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:38:26.353795Z",
  "userAgent": "kube-controller-manager/v1.29.11+rke2r1 (linux/amd64) kubernetes/960a2f0/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/cilium-agent": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/clean-cilium-state": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/mount-cgroup": "unconfined",
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:38:41.303421Z",
  "userAgent": "kube-controller-manager/v1.29.11+rke2r1 (linux/amd64) kubernetes/960a2f0/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/cilium-agent": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/clean-cilium-state": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/mount-cgroup": "unconfined",
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:38:52.544272Z",
  "userAgent": "kube-controller-manager/v1.29.11+rke2r1 (linux/amd64) kubernetes/960a2f0/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/cilium-agent": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/clean-cilium-state": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/mount-cgroup": "unconfined",
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:39:23.015950Z",
  "userAgent": "kube-controller-manager/v1.29.11+rke2r1 (linux/amd64) kubernetes/960a2f0/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/cilium-agent": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/clean-cilium-state": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/mount-cgroup": "unconfined",
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:39:30.984065Z",
  "userAgent": "kube-controller-manager/v1.29.11+rke2r1 (linux/amd64) kubernetes/960a2f0/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/cilium-agent": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/clean-cilium-state": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/mount-cgroup": "unconfined",
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:44:12.727728Z",
  "userAgent": "kube-controller-manager/v1.29.11+rke2r1 (linux/amd64) kubernetes/960a2f0/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/cilium-agent": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/clean-cilium-state": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/mount-cgroup": "unconfined",
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:44:35.494079Z",
  "userAgent": "kube-controller-manager/v1.29.11+rke2r1 (linux/amd64) kubernetes/960a2f0/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/cilium-agent": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/clean-cilium-state": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/mount-cgroup": "unconfined",
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:44:48.598381Z",
  "userAgent": "kube-controller-manager/v1.29.11+rke2r1 (linux/amd64) kubernetes/960a2f0/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/cilium-agent": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/clean-cilium-state": "unconfined",
    "container.apparmor.security.beta.kubernetes.io/mount-cgroup": "unconfined",
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "patch",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium?fieldManager=helm",
  "stageTimestamp": "2025-01-07T19:44:59.558778Z",
  "userAgent": "Helm/3.16.1",
  "spec.template.spec.securityContext": {
    "appArmorProfile": {
      "type": "Unconfined"
    }
  },
  "spec.template.metadata.annotations": {
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:44:59.609019Z",
  "userAgent": "kube-controller-manager/v1.29.11+rke2r1 (linux/amd64) kubernetes/960a2f0/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:45:00.128220Z",
  "userAgent": "kube-controller-manager/v1.29.11+rke2r1 (linux/amd64) kubernetes/960a2f0/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:45:00.180536Z",
  "userAgent": "kube-controller-manager/v1.29.11+rke2r1 (linux/amd64) kubernetes/960a2f0/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:45:00.193503Z",
  "userAgent": "kube-controller-manager/v1.29.11+rke2r1 (linux/amd64) kubernetes/960a2f0/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:46:11.694538Z",
  "userAgent": "kube-controller-manager/v1.30.7+rke2r1 (linux/amd64) kubernetes/0c76c64/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
{
  "verb": "update",
  "requestURI": "/apis/apps/v1/namespaces/kube-system/daemonsets/cilium/status",
  "stageTimestamp": "2025-01-07T19:46:12.115811Z",
  "userAgent": "kube-controller-manager/v1.30.7+rke2r1 (linux/amd64) kubernetes/0c76c64/system:serviceaccount:kube-system:daemon-set-controller",
  "spec.template.spec.securityContext": {},
  "spec.template.metadata.annotations": {
    "prometheus.io/port": "9962",
    "prometheus.io/scrape": "true"
  }
}
```

</details>

### Kubernetes version

<details>

```console
root@rke2-server-1:/# kubectl version
Client Version: v1.30.7+rke2r1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.7+rke2r1
```

</details>


### Cloud provider

n/a

### OS version

Ubuntu 22.04

### Install tools

rke2

### Container runtime (CRI) and version (if applicable)

n/a

### Related plugins (CNI, CSI, ...) and versions (if applicable)

cilium