Repository: kubernetes
Issue: Garbage collector deletes DaemonSet after updating ownerReference and deleting previous owner
Labels: kind/bug,sig/api-machinery,help wanted,triage/accepted

### What happened?

In rare cases, I'm seeing the GC deleting a DaemonSet with a valid ownerReference. This issue looks like a time-of-check/time-of-use error in the garbage collector, although I'm unsure from the docs what behavior to expect.

I'm using a tool similar to `kubectl apply --prune`, called `synk`, except instead of using a label selector to prune old resources, it uses the Kubernetes garbage collector. However, I've found that some systems (single-node test clusters on VMs) break after a reboot, due to unexpected deletion of a DaemonSet. Strangely, I've not seen any other resources getting deleted (even DaemonSets in other manifests installed the same way), so I'm not sure what's special about this DaemonSet.

The order of events, when using this to install a manifest for kube-proxy, is as follows:

- setup (working as intended)
  - synk creates a new owner custom resource (the `ResourceSet`), kube-proxy.v1
  - synk creates the resources in the manifest, including the `kube-proxy` DaemonSet
- update (not always working)
  - synk creates a new owner custom resource (the `ResourceSet`), kube-proxy.v2
  - synk patches the `kube-proxy` DaemonSet to update its ownerReference to point to kube-proxy.v1
  - synk deletes the old owner, kube-proxy.v1
  - sometimes, the garbage collector deletes the DaemonSet, even though it still has a valid ownerReference

### What did you expect to happen?

The garbage collector should only delete resources with dangling ownerReferences. Alternatively, if it's impossible to handle the case of updated ownerReferences in general, the [GC docs](https://kubernetes.io/docs/concepts/architecture/garbage-collection/) should tell you how to update the ownerReference and wait for the GC to become aware of this, before deleting the obsolete owner (eg can the GC enforce some ceiling on the time between fetching the ownerReferences and checking them for existence?).



### How can we reproduce it (as minimally and precisely as possible)?

I don't have a repro I can share I'm afraid. It requires setting up a VM and rebooting it ~100 times. A colleague has tried triggering it by running the command that updates the ownerReference at the same time as restarting cluster services like etcd, kube-controller-manager, etc - but this doesn't seem to trigger the problem.

### Anything else we need to know?

Looking at the GC source code, I don't see any protection against the race between awareness of updated ownerReferences and awareness of the deletion of the old owner:

- on [line 512](https://github.com/kubernetes/kubernetes/blob/v1.26.0/pkg/controller/garbagecollector/garbagecollector.go#L512) it gets the "latest" ownerReferences
- on [line 544](https://github.com/kubernetes/kubernetes/blob/v1.26.0/pkg/controller/garbagecollector/garbagecollector.go#L544) it checks if these owners exist
- on [line 615](https://github.com/kubernetes/kubernetes/blob/v1.26.0/pkg/controller/garbagecollector/garbagecollector.go#L615) it deletes the object, without checking if the ownerReference has changed. I don't see any timeouts (and the code uses context.TODO()) so I guess there is an arbitrary period of time between time of check and time of use here.

I also found this quote from the `kubectl delete` docs, which suggests that the fix is not as simple as including the resourceVersion in the DELETE request. The [API concepts](https://kubernetes.io/docs/reference/using-api/api-concepts/) doc doesn't explictly state whether/how resourceVersion can affect deletes, so I'm not sure if this is just for `kubectl delete` or all deletion requests.

> Note that the delete command does NOT do resource version checks, so if someone submits an update to a resource right when you submit a delete, their update will be lost along with the rest of the resource.

Audit / GC logs seem to agree with this:

<details>

Here are the audit logs during the event (annotated by me)

```
$ grep -E "ResponseComplete.*(daemonsets|resourcesets).*kube-proxy" /var/log/kubernetes/audit/audit.log
# synk creates new owner
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"f897b357-4059-434c-9a87-35d992188846","stage":"ResponseComplete","requestURI":"/apis/apps.cloudrobotics.com/v1alpha1/resourcesets","verb":"create","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"synk/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"resourcesets","name":"kube-proxy.v19","apiGroup":"apps.cloudrobotics.com","apiVersion":"v1alpha1"},"responseStatus":{"metadata":{},"code":201},"requestObject":{"apiVersion":"apps.cloudrobotics.com/v1alpha1","kind":"ResourceSet","metadata":{"creationTimestamp":null,"labels":{"name":"kube-proxy"},"name":"kube-proxy.v19"},"spec":{"resources":[{"items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"ServiceAccount","version":"v1"},{"items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"ConfigMap","version":"v1"},{"group":"apps","items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"DaemonSet","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"name":"kubeadm:node-proxier"}],"kind":"ClusterRoleBinding","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"Role","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"RoleBinding","version":"v1"}]},"status":{"finishedAt":null,"phase":"Pending","startedAt":"2022-12-20T10:37:03Z"}},"responseObject":{"apiVersion":"apps.cloudrobotics.com/v1alpha1","kind":"ResourceSet","metadata":{"creationTimestamp":"2022-12-20T10:37:03Z","generation":1,"labels":{"name":"kube-proxy"},"managedFields":[{"apiVersion":"apps.cloudrobotics.com/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:name":{}}},"f:spec":{".":{},"f:resources":{}},"f:status":{".":{},"f:finishedAt":{},"f:phase":{},"f:startedAt":{}}},"manager":"synk","operation":"Update","time":"2022-12-20T10:37:03Z"}],"name":"kube-proxy.v19","resourceVersion":"52681","uid":"41aaee93-9741-4d79-8913-533e67015449"},"spec":{"resources":[{"items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"ServiceAccount","version":"v1"},{"items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"ConfigMap","version":"v1"},{"group":"apps","items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"DaemonSet","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"name":"kubeadm:node-proxier"}],"kind":"ClusterRoleBinding","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"Role","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"RoleBinding","version":"v1"}]},"status":{"finishedAt":null,"phase":"Pending","startedAt":"2022-12-20T10:37:03Z"}},"requestReceivedTimestamp":"2022-12-20T10:37:03.755514Z","stageTimestamp":"2022-12-20T10:37:03.759625Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
# GC gets DaemonSet before ownerReference is updated
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"c14d3646-9119-417e-8d7b-43e78dbec9aa","stage":"ResponseComplete","requestURI":"/apis/apps/v1/namespaces/kube-system/daemonsets/kube-proxy","verb":"get","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"synk/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"daemonsets","namespace":"kube-system","name":"kube-proxy","apiGroup":"apps","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"responseObject":{"kind":"DaemonSet","apiVersion":"apps/v1","metadata":{"name":"kube-proxy","namespace":"kube-system","uid":"d7956af1-c25f-4952-9ed7-6d6ac92f985f","resourceVersion":"52625","generation":1,"creationTimestamp":"2022-12-20T08:09:12Z","labels":{"k8s-app":"kube-proxy"},"annotations":{"deprecated.daemonset.template.generation":"1","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"DaemonSet\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-app\":\"kube-proxy\"},\"name\":\"kube-proxy\",\"namespace\":\"kube-system\",\"ownerReferences\":[{\"apiVersion\":\"apps.cloudrobotics.com/v1alpha1\",\"blockOwnerDeletion\":true,\"kind\":\"ResourceSet\",\"name\":\"kube-proxy.v18\",\"uid\":\"39e9d246-62db-4acd-8fe7-6105dcedea3e\"}]},\"spec\":{\"selector\":{\"matchLabels\":{\"k8s-app\":\"kube-proxy\"}},\"template\":{\"metadata\":{\"labels\":{\"k8s-app\":\"kube-proxy\"}},\"spec\":{\"containers\":[{\"command\":[\"/usr/local/bin/kube-proxy\",\"--config=/var/lib/kube-proxy/config.conf\",\"--hostname-override=$(NODE_NAME)\"],\"env\":[{\"name\":\"NODE_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"spec.nodeName\"}}}],\"image\":\"k8s.gcr.io/kube-proxy:v1.23.15\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"kube-proxy\",\"securityContext\":{\"privileged\":true},\"volumeMounts\":[{\"mountPath\":\"/var/lib/kube-proxy\",\"name\":\"kube-proxy\"},{\"mountPath\":\"/run/xtables.lock\",\"name\":\"xtables-lock\"},{\"mountPath\":\"/lib/modules\",\"name\":\"lib-modules\",\"readOnly\":true}]}],\"hostNetwork\":true,\"nodeSelector\":{\"kubernetes.io/os\":\"linux\"},\"priorityClassName\":\"system-node-critical\",\"serviceAccountName\":\"kube-proxy\",\"tolerations\":[{\"operator\":\"Exists\"}],\"volumes\":[{\"configMap\":{\"name\":\"kube-proxy\"},\"name\":\"kube-proxy\"},{\"hostPath\":{\"path\":\"/run/xtables.lock\",\"type\":\"FileOrCreate\"},\"name\":\"xtables-lock\"},{\"hostPath\":{\"path\":\"/lib/modules\"},\"name\":\"lib-modules\"}]}},\"updateStrategy\":{\"type\":\"RollingUpdate\"}}}\n"},"ownerReferences":[{"apiVersion":"apps.cloudrobotics.com/v1alpha1","kind":"ResourceSet","name":"kube-proxy.v18","uid":"39e9d246-62db-4acd-8fe7-6105dcedea3e","blockOwnerDeletion":true}],"managedFields":[{"manager":"synk","operation":"Update","apiVersion":"apps/v1","time":"2022-12-20T10:28:43Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:deprecated.daemonset.template.generation":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:k8s-app":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"39e9d246-62db-4acd-8fe7-6105dcedea3e\"}":{}}},"f:spec":{"f:revisionHistoryLimit":{},"f:selector":{},"f:template":{"f:metadata":{"f:labels":{".":{},"f:k8s-app":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"kube-proxy\"}":{".":{},"f:command":{},"f:env":{".":{},"k:{\"name\":\"NODE_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{}}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:securityContext":{".":{},"f:privileged":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/lib/modules\"}":{".":{},"f:mountPath":{},"f:name":{},"f:readOnly":{}},"k:{\"mountPath\":\"/run/xtables.lock\"}":{".":{},"f:mountPath":{},"f:name":{}},"k:{\"mountPath\":\"/var/lib/kube-proxy\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:hostNetwork":{},"f:nodeSelector":{},"f:priorityClassName":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:tolerations":{},"f:volumes":{".":{},"k:{\"name\":\"kube-proxy\"}":{".":{},"f:configMap":{".":{},"f:defaultMode":{},"f:name":{}},"f:name":{}},"k:{\"name\":\"lib-modules\"}":{".":{},"f:hostPath":{".":{},"f:path":{},"f:type":{}},"f:name":{}},"k:{\"name\":\"xtables-lock\"}":{".":{},"f:hostPath":{".":{},"f:path":{},"f:type":{}},"f:name":{}}}}},"f:updateStrategy":{"f:rollingUpdate":{".":{},"f:maxSurge":{},"f:maxUnavailable":{}},"f:type":{}}}}},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"apps/v1","time":"2022-12-20T10:37:00Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:currentNumberScheduled":{},"f:desiredNumberScheduled":{},"f:numberReady":{},"f:numberUnavailable":{},"f:observedGeneration":{},"f:updatedNumberScheduled":{}}},"subresource":"status"}]},"spec":{"selector":{"matchLabels":{"k8s-app":"kube-proxy"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"k8s-app":"kube-proxy"}},"spec":{"volumes":[{"name":"kube-proxy","configMap":{"name":"kube-proxy","defaultMode":420}},{"name":"xtables-lock","hostPath":{"path":"/run/xtables.lock","type":"FileOrCreate"}},{"name":"lib-modules","hostPath":{"path":"/lib/modules","type":""}}],"containers":[{"name":"kube-proxy","image":"k8s.gcr.io/kube-proxy:v1.23.15","command":["/usr/local/bin/kube-proxy","--config=/var/lib/kube-proxy/config.conf","--hostname-override=$(NODE_NAME)"],"env":[{"name":"NODE_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.nodeName"}}}],"resources":{},"volumeMounts":[{"name":"kube-proxy","mountPath":"/var/lib/kube-proxy"},{"name":"xtables-lock","mountPath":"/run/xtables.lock"},{"name":"lib-modules","readOnly":true,"mountPath":"/lib/modules"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"privileged":true}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"kubernetes.io/os":"linux"},"serviceAccountName":"kube-proxy","serviceAccount":"kube-proxy","hostNetwork":true,"securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"operator":"Exists"}],"priorityClassName":"system-node-critical"}},"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":1,"maxSurge":0}},"revisionHistoryLimit":10},"status":{"currentNumberScheduled":1,"numberMisscheduled":0,"desiredNumberScheduled":1,"numberReady":0,"observedGeneration":1,"updatedNumberScheduled":1,"numberUnavailable":1}},"requestReceivedTimestamp":"2022-12-20T10:37:03.799863Z","stageTimestamp":"2022-12-20T10:37:03.802276Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
# synk patches DaemonSet's ownerReference
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"b9ed71b6-a1a8-43d3-95c5-4a559301cc93","stage":"ResponseComplete","requestURI":"/apis/apps/v1/namespaces/kube-system/daemonsets/kube-proxy","verb":"patch","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"synk/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"daemonsets","namespace":"kube-system","name":"kube-proxy","apiGroup":"apps","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"metadata":{"$setElementOrder/ownerReferences":[{"uid":"41aaee93-9741-4d79-8913-533e67015449"}],"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"DaemonSet\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-app\":\"kube-proxy\"},\"name\":\"kube-proxy\",\"namespace\":\"kube-system\",\"ownerReferences\":[{\"apiVersion\":\"apps.cloudrobotics.com/v1alpha1\",\"blockOwnerDeletion\":true,\"kind\":\"ResourceSet\",\"name\":\"kube-proxy.v19\",\"uid\":\"41aaee93-9741-4d79-8913-533e67015449\"}]},\"spec\":{\"selector\":{\"matchLabels\":{\"k8s-app\":\"kube-proxy\"}},\"template\":{\"metadata\":{\"labels\":{\"k8s-app\":\"kube-proxy\"}},\"spec\":{\"containers\":[{\"command\":[\"/usr/local/bin/kube-proxy\",\"--config=/var/lib/kube-proxy/config.conf\",\"--hostname-override=$(NODE_NAME)\"],\"env\":[{\"name\":\"NODE_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"spec.nodeName\"}}}],\"image\":\"k8s.gcr.io/kube-proxy:v1.23.15\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"kube-proxy\",\"securityContext\":{\"privileged\":true},\"volumeMounts\":[{\"mountPath\":\"/var/lib/kube-proxy\",\"name\":\"kube-proxy\"},{\"mountPath\":\"/run/xtables.lock\",\"name\":\"xtables-lock\"},{\"mountPath\":\"/lib/modules\",\"name\":\"lib-modules\",\"readOnly\":true}]}],\"hostNetwork\":true,\"nodeSelector\":{\"kubernetes.io/os\":\"linux\"},\"priorityClassName\":\"system-node-critical\",\"serviceAccountName\":\"kube-proxy\",\"tolerations\":[{\"operator\":\"Exists\"}],\"volumes\":[{\"configMap\":{\"name\":\"kube-proxy\"},\"name\":\"kube-proxy\"},{\"hostPath\":{\"path\":\"/run/xtables.lock\",\"type\":\"FileOrCreate\"},\"name\":\"xtables-lock\"},{\"hostPath\":{\"path\":\"/lib/modules\"},\"name\":\"lib-modules\"}]}},\"updateStrategy\":{\"type\":\"RollingUpdate\"}}}\n"},"ownerReferences":[{"apiVersion":"apps.cloudrobotics.com/v1alpha1","blockOwnerDeletion":true,"kind":"ResourceSet","name":"kube-proxy.v19","uid":"41aaee93-9741-4d79-8913-533e67015449"},{"$patch":"delete","uid":"39e9d246-62db-4acd-8fe7-6105dcedea3e"}]}},"responseObject":{"kind":"DaemonSet","apiVersion":"apps/v1","metadata":{"name":"kube-proxy","namespace":"kube-system","uid":"d7956af1-c25f-4952-9ed7-6d6ac92f985f","resourceVersion":"52684","generation":1,"creationTimestamp":"2022-12-20T08:09:12Z","labels":{"k8s-app":"kube-proxy"},"annotations":{"deprecated.daemonset.template.generation":"1","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"DaemonSet\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-app\":\"kube-proxy\"},\"name\":\"kube-proxy\",\"namespace\":\"kube-system\",\"ownerReferences\":[{\"apiVersion\":\"apps.cloudrobotics.com/v1alpha1\",\"blockOwnerDeletion\":true,\"kind\":\"ResourceSet\",\"name\":\"kube-proxy.v19\",\"uid\":\"41aaee93-9741-4d79-8913-533e67015449\"}]},\"spec\":{\"selector\":{\"matchLabels\":{\"k8s-app\":\"kube-proxy\"}},\"template\":{\"metadata\":{\"labels\":{\"k8s-app\":\"kube-proxy\"}},\"spec\":{\"containers\":[{\"command\":[\"/usr/local/bin/kube-proxy\",\"--config=/var/lib/kube-proxy/config.conf\",\"--hostname-override=$(NODE_NAME)\"],\"env\":[{\"name\":\"NODE_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"spec.nodeName\"}}}],\"image\":\"k8s.gcr.io/kube-proxy:v1.23.15\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"kube-proxy\",\"securityContext\":{\"privileged\":true},\"volumeMounts\":[{\"mountPath\":\"/var/lib/kube-proxy\",\"name\":\"kube-proxy\"},{\"mountPath\":\"/run/xtables.lock\",\"name\":\"xtables-lock\"},{\"mountPath\":\"/lib/modules\",\"name\":\"lib-modules\",\"readOnly\":true}]}],\"hostNetwork\":true,\"nodeSelector\":{\"kubernetes.io/os\":\"linux\"},\"priorityClassName\":\"system-node-critical\",\"serviceAccountName\":\"kube-proxy\",\"tolerations\":[{\"operator\":\"Exists\"}],\"volumes\":[{\"configMap\":{\"name\":\"kube-proxy\"},\"name\":\"kube-proxy\"},{\"hostPath\":{\"path\":\"/run/xtables.lock\",\"type\":\"FileOrCreate\"},\"name\":\"xtables-lock\"},{\"hostPath\":{\"path\":\"/lib/modules\"},\"name\":\"lib-modules\"}]}},\"updateStrategy\":{\"type\":\"RollingUpdate\"}}}\n"},"ownerReferences":[{"apiVersion":"apps.cloudrobotics.com/v1alpha1","kind":"ResourceSet","name":"kube-proxy.v19","uid":"41aaee93-9741-4d79-8913-533e67015449","blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"apps/v1","time":"2022-12-20T10:37:00Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:currentNumberScheduled":{},"f:desiredNumberScheduled":{},"f:numberReady":{},"f:numberUnavailable":{},"f:observedGeneration":{},"f:updatedNumberScheduled":{}}},"subresource":"status"},{"manager":"synk","operation":"Update","apiVersion":"apps/v1","time":"2022-12-20T10:37:03Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:deprecated.daemonset.template.generation":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:k8s-app":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"41aaee93-9741-4d79-8913-533e67015449\"}":{}}},"f:spec":{"f:revisionHistoryLimit":{},"f:selector":{},"f:template":{"f:metadata":{"f:labels":{".":{},"f:k8s-app":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"kube-proxy\"}":{".":{},"f:command":{},"f:env":{".":{},"k:{\"name\":\"NODE_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{}}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:securityContext":{".":{},"f:privileged":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/lib/modules\"}":{".":{},"f:mountPath":{},"f:name":{},"f:readOnly":{}},"k:{\"mountPath\":\"/run/xtables.lock\"}":{".":{},"f:mountPath":{},"f:name":{}},"k:{\"mountPath\":\"/var/lib/kube-proxy\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:hostNetwork":{},"f:nodeSelector":{},"f:priorityClassName":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:tolerations":{},"f:volumes":{".":{},"k:{\"name\":\"kube-proxy\"}":{".":{},"f:configMap":{".":{},"f:defaultMode":{},"f:name":{}},"f:name":{}},"k:{\"name\":\"lib-modules\"}":{".":{},"f:hostPath":{".":{},"f:path":{},"f:type":{}},"f:name":{}},"k:{\"name\":\"xtables-lock\"}":{".":{},"f:hostPath":{".":{},"f:path":{},"f:type":{}},"f:name":{}}}}},"f:updateStrategy":{"f:rollingUpdate":{".":{},"f:maxSurge":{},"f:maxUnavailable":{}},"f:type":{}}}}}]},"spec":{"selector":{"matchLabels":{"k8s-app":"kube-proxy"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"k8s-app":"kube-proxy"}},"spec":{"volumes":[{"name":"kube-proxy","configMap":{"name":"kube-proxy","defaultMode":420}},{"name":"xtables-lock","hostPath":{"path":"/run/xtables.lock","type":"FileOrCreate"}},{"name":"lib-modules","hostPath":{"path":"/lib/modules","type":""}}],"containers":[{"name":"kube-proxy","image":"k8s.gcr.io/kube-proxy:v1.23.15","command":["/usr/local/bin/kube-proxy","--config=/var/lib/kube-proxy/config.conf","--hostname-override=$(NODE_NAME)"],"env":[{"name":"NODE_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.nodeName"}}}],"resources":{},"volumeMounts":[{"name":"kube-proxy","mountPath":"/var/lib/kube-proxy"},{"name":"xtables-lock","mountPath":"/run/xtables.lock"},{"name":"lib-modules","readOnly":true,"mountPath":"/lib/modules"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"privileged":true}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"kubernetes.io/os":"linux"},"serviceAccountName":"kube-proxy","serviceAccount":"kube-proxy","hostNetwork":true,"securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"operator":"Exists"}],"priorityClassName":"system-node-critical"}},"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":1,"maxSurge":0}},"revisionHistoryLimit":10},"status":{"currentNumberScheduled":1,"numberMisscheduled":0,"desiredNumberScheduled":1,"numberReady":0,"observedGeneration":1,"updatedNumberScheduled":1,"numberUnavailable":1}},"requestReceivedTimestamp":"2022-12-20T10:37:03.808502Z","stageTimestamp":"2022-12-20T10:37:03.817132Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
# synk updates status of new owner (I don't think this is relevant)
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"7361274e-2d1e-4532-8c34-22ec0e6fff7d","stage":"ResponseComplete","requestURI":"/apis/apps.cloudrobotics.com/v1alpha1/resourcesets/kube-proxy.v19","verb":"update","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"synk/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"resourcesets","name":"kube-proxy.v19","uid":"41aaee93-9741-4d79-8913-533e67015449","apiGroup":"apps.cloudrobotics.com","apiVersion":"v1alpha1","resourceVersion":"52681"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"apiVersion":"apps.cloudrobotics.com/v1alpha1","kind":"ResourceSet","metadata":{"creationTimestamp":"2022-12-20T10:37:03Z","generation":1,"labels":{"name":"kube-proxy"},"managedFields":[{"apiVersion":"apps.cloudrobotics.com/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:name":{}}},"f:spec":{".":{},"f:resources":{}},"f:status":{".":{},"f:finishedAt":{},"f:phase":{},"f:startedAt":{}}},"manager":"synk","operation":"Update","time":"2022-12-20T10:37:03Z"}],"name":"kube-proxy.v19","resourceVersion":"52681","uid":"41aaee93-9741-4d79-8913-533e67015449"},"spec":{"resources":[{"items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"ServiceAccount","version":"v1"},{"items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"ConfigMap","version":"v1"},{"group":"apps","items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"DaemonSet","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"name":"kubeadm:node-proxier"}],"kind":"ClusterRoleBinding","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"Role","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"RoleBinding","version":"v1"}]},"status":{"applied":[{"items":[{"action":"Update","name":"kube-proxy","namespace":"kube-system","uid":"2dc82b30-73c1-4021-a303-fb926cd10d43"}],"kind":"ServiceAccount","version":"v1"},{"items":[{"action":"Update","name":"kube-proxy","namespace":"kube-system","uid":"08e25636-baca-4244-9948-a19f083a2395"}],"kind":"ConfigMap","version":"v1"},{"group":"apps","items":[{"action":"Update","generation":1,"name":"kube-proxy","namespace":"kube-system","uid":"d7956af1-c25f-4952-9ed7-6d6ac92f985f"}],"kind":"DaemonSet","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"action":"Update","name":"kubeadm:node-proxier","uid":"f12b9ee0-59c0-4632-80e9-23f7dfbd46a6"}],"kind":"ClusterRoleBinding","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"action":"Update","name":"kube-proxy","namespace":"kube-system","uid":"9fc28fb0-b234-476b-9a55-51d38c6e6f91"}],"kind":"Role","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"action":"Update","name":"kube-proxy","namespace":"kube-system","uid":"c74c987f-3b91-423f-85c3-131c61f9620b"}],"kind":"RoleBinding","version":"v1"}],"finishedAt":"2022-12-20T10:37:03Z","phase":"Settled","startedAt":"2022-12-20T10:37:03Z"}},"responseObject":{"apiVersion":"apps.cloudrobotics.com/v1alpha1","kind":"ResourceSet","metadata":{"creationTimestamp":"2022-12-20T10:37:03Z","generation":2,"labels":{"name":"kube-proxy"},"managedFields":[{"apiVersion":"apps.cloudrobotics.com/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:name":{}}},"f:spec":{".":{},"f:resources":{}},"f:status":{".":{},"f:applied":{},"f:finishedAt":{},"f:phase":{},"f:startedAt":{}}},"manager":"synk","operation":"Update","time":"2022-12-20T10:37:03Z"}],"name":"kube-proxy.v19","resourceVersion":"52688","uid":"41aaee93-9741-4d79-8913-533e67015449"},"spec":{"resources":[{"items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"ServiceAccount","version":"v1"},{"items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"ConfigMap","version":"v1"},{"group":"apps","items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"DaemonSet","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"name":"kubeadm:node-proxier"}],"kind":"ClusterRoleBinding","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"Role","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"name":"kube-proxy","namespace":"kube-system"}],"kind":"RoleBinding","version":"v1"}]},"status":{"applied":[{"items":[{"action":"Update","name":"kube-proxy","namespace":"kube-system","uid":"2dc82b30-73c1-4021-a303-fb926cd10d43"}],"kind":"ServiceAccount","version":"v1"},{"items":[{"action":"Update","name":"kube-proxy","namespace":"kube-system","uid":"08e25636-baca-4244-9948-a19f083a2395"}],"kind":"ConfigMap","version":"v1"},{"group":"apps","items":[{"action":"Update","generation":1,"name":"kube-proxy","namespace":"kube-system","uid":"d7956af1-c25f-4952-9ed7-6d6ac92f985f"}],"kind":"DaemonSet","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"action":"Update","name":"kubeadm:node-proxier","uid":"f12b9ee0-59c0-4632-80e9-23f7dfbd46a6"}],"kind":"ClusterRoleBinding","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"action":"Update","name":"kube-proxy","namespace":"kube-system","uid":"9fc28fb0-b234-476b-9a55-51d38c6e6f91"}],"kind":"Role","version":"v1"},{"group":"rbac.authorization.k8s.io","items":[{"action":"Update","name":"kube-proxy","namespace":"kube-system","uid":"c74c987f-3b91-423f-85c3-131c61f9620b"}],"kind":"RoleBinding","version":"v1"}],"finishedAt":"2022-12-20T10:37:03Z","phase":"Settled","startedAt":"2022-12-20T10:37:03Z"}},"requestReceivedTimestamp":"2022-12-20T10:37:03.840184Z","stageTimestamp":"2022-12-20T10:37:03.845316Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
# synk deletes old owner
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"88122155-5101-4626-9663-1ff16cb1fac7","stage":"ResponseComplete","requestURI":"/apis/apps.cloudrobotics.com/v1alpha1/resourcesets/kube-proxy.v18","verb":"delete","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"synk/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"resourcesets","name":"kube-proxy.v18","apiGroup":"apps.cloudrobotics.com","apiVersion":"v1alpha1"},"responseStatus":{"metadata":{},"status":"Success","code":200},"responseObject":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Success","details":{"name":"kube-proxy.v18","group":"apps.cloudrobotics.com","kind":"resourcesets","uid":"39e9d246-62db-4acd-8fe7-6105dcedea3e"}},"requestReceivedTimestamp":"2022-12-20T10:37:03.897775Z","stageTimestamp":"2022-12-20T10:37:03.902829Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
# GC tries to get old owner, gets 404
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"0aac9c78-87ea-4503-9952-b73c6d69fa10","stage":"ResponseComplete","requestURI":"/apis/apps.cloudrobotics.com/v1alpha1/resourcesets/kube-proxy.v18","verb":"get","user":{"username":"system:serviceaccount:kube-system:generic-garbage-collector","uid":"79c88df8-eda6-46eb-aeb2-680d1d105620","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-controller-manager/v1.23.12 (linux/amd64) kubernetes/c693979/system:serviceaccount:kube-system:generic-garbage-collector","objectRef":{"resource":"resourcesets","name":"kube-proxy.v18","apiGroup":"apps.cloudrobotics.com","apiVersion":"v1alpha1"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"responseObject":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"resourcesets.apps.cloudrobotics.com \"kube-proxy.v18\" not found","reason":"NotFound","details":{"name":"kube-proxy.v18","group":"apps.cloudrobotics.com","kind":"resourcesets"},"code":404},"requestReceivedTimestamp":"2022-12-20T10:37:04.193988Z","stageTimestamp":"2022-12-20T10:37:04.199090Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"permissive-binding\" of ClusterRole \"cluster-admin\" to Group \"system:serviceaccounts\""}}
# GC mistakenly deletes DaemonSet
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"cc71750b-dbe3-4eaa-ac4e-6567b92fec72","stage":"ResponseComplete","requestURI":"/apis/apps/v1/namespaces/kube-system/daemonsets/kube-proxy","verb":"delete","user":{"username":"system:serviceaccount:kube-system:generic-garbage-collector","uid":"79c88df8-eda6-46eb-aeb2-680d1d105620","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-controller-manager/v1.23.12 (linux/amd64) kubernetes/c693979/system:serviceaccount:kube-system:generic-garbage-collector","objectRef":{"resource":"daemonsets","namespace":"kube-system","name":"kube-proxy","apiGroup":"apps","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Success","code":200},"requestObject":{"kind":"DeleteOptions","apiVersion":"v1","preconditions":{"uid":"d7956af1-c25f-4952-9ed7-6d6ac92f985f"},"propagationPolicy":"Background"},"responseObject":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Success","details":{"name":"kube-proxy","group":"apps","kind":"daemonsets","uid":"d7956af1-c25f-4952-9ed7-6d6ac92f985f"}},"requestReceivedTimestamp":"2022-12-20T10:37:04.691655Z","stageTimestamp":"2022-12-20T10:37:04.697375Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"permissive-binding\" of ClusterRole \"cluster-admin\" to Group \"system:serviceaccounts\""}}
# GC gets new owner (probably while processing another resource?)
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"5566b8c3-429c-40d7-b1ff-fb1118f6b6af","stage":"ResponseComplete","requestURI":"/apis/apps.cloudrobotics.com/v1alpha1/resourcesets/kube-proxy.v19","verb":"get","user":{"username":"system:serviceaccount:kube-system:generic-garbage-collector","uid":"79c88df8-eda6-46eb-aeb2-680d1d105620","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-controller-manager/v1.23.12 (linux/amd64) kubernetes/c693979/system:serviceaccount:kube-system:generic-garbage-collector","objectRef":{"resource":"resourcesets","name":"kube-proxy.v19","apiGroup":"apps.cloudrobotics.com","apiVersion":"v1alpha1"},"responseStatus":{"metadata":{},"code":200},"responseObject":{"kind":"PartialObjectMetadata","apiVersion":"meta.k8s.io/v1","metadata":{"name":"kube-proxy.v19","uid":"41aaee93-9741-4d79-8913-533e67015449","resourceVersion":"52688","generation":2,"creationTimestamp":"2022-12-20T10:37:03Z","labels":{"name":"kube-proxy"},"managedFields":[{"manager":"synk","operation":"Update","apiVersion":"apps.cloudrobotics.com/v1alpha1","time":"2022-12-20T10:37:03Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:name":{}}},"f:spec":{".":{},"f:resources":{}},"f:status":{".":{},"f:applied":{},"f:finishedAt":{},"f:phase":{},"f:startedAt":{}}}}]}},"requestReceivedTimestamp":"2022-12-20T10:37:08.841614Z","stageTimestamp":"2022-12-20T10:37:08.843671Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"permissive-binding\" of ClusterRole \"cluster-admin\" to Group \"system:serviceaccounts\""}}
# GC tries to get old owner again (not sure why, just here for completeness)
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"7177c495-4e39-403c-af25-9972e6c62683","stage":"ResponseComplete","requestURI":"/apis/apps.cloudrobotics.com/v1alpha1/resourcesets/kube-proxy.v18","verb":"get","user":{"username":"system:serviceaccount:kube-system:generic-garbage-collector","uid":"79c88df8-eda6-46eb-aeb2-680d1d105620","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-controller-manager/v1.23.12 (linux/amd64) kubernetes/c693979/system:serviceaccount:kube-system:generic-garbage-collector","objectRef":{"resource":"resourcesets","name":"kube-proxy.v18","apiGroup":"apps.cloudrobotics.com","apiVersion":"v1alpha1"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"responseObject":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"resourcesets.apps.cloudrobotics.com \"kube-proxy.v18\" not found","reason":"NotFound","details":{"name":"kube-proxy.v18","group":"apps.cloudrobotics.com","kind":"resourcesets"},"code":404},"requestReceivedTimestamp":"2022-12-20T10:37:09.291824Z","stageTimestamp":"2022-12-20T10:37:09.293876Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"permissive-binding\" of ClusterRole \"cluster-admin\" to Group \"system:serviceaccounts\""}}
# GC tries to get the DaemonSet that it already deleted (not sure why, just here for completeness)
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"6e326ece-f101-4a4a-8ab9-c35d1f476670","stage":"ResponseComplete","requestURI":"/apis/apps/v1/namespaces/kube-system/daemonsets/kube-proxy","verb":"get","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kubectl/v1.23.12 (linux/amd64) kubernetes/c693979","objectRef":{"resource":"daemonsets","namespace":"kube-system","name":"kube-proxy","apiGroup":"apps","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"responseObject":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"daemonsets.apps \"kube-proxy\" not found","reason":"NotFound","details":{"name":"kube-proxy","group":"apps","kind":"daemonsets"},"code":404},"requestReceivedTimestamp":"2022-12-20T10:42:53.339889Z","stageTimestamp":"2022-12-20T10:42:53.344024Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
```

And the GC verbose logs (`--vmodule=garbagecollector=9`):

```
$ journalctl -u kube-controller-manager --no-pager | grep kube-proxy
Dec 20 10:37:03 vm-rodrigoq-8687 kube-controller-manager[334]: I1220 10:37:03.193688     334 garbagecollector.go:468] "Processing object" object="kube-system/kube-proxy" objectUID=d7956af1-c25f-4952-9ed7-6d6ac92f985f kind="DaemonSet" virtual=false
Dec 20 10:37:03 vm-rodrigoq-8687 kube-controller-manager[334]: I1220 10:37:03.793418     334 garbagecollector.go:324] item &garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:v1.OwnerReference{APIVersion:"apps.cloudrobotics.com/v1alpha1", Kind:"ResourceSet", Name:"kube-proxy.v18", UID:"39e9d246-62db-4acd-8fe7-6105dcedea3e", Controller:(*bool)(nil), BlockOwnerDeletion:(*bool)(nil)}, Namespace:"kube-system"}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:1, readerWait:0}, dependents:map[*garbagecollector.node]struct {}{(*garbagecollector.node)(0xc001e860f0):struct {}{}, (*garbagecollector.node)(0xc00258c690):struct {}{}, (*garbagecollector.node)(0xc0028962d0):struct {}{}, (*garbagecollector.node)(0xc002898e10):struct {}{}, (*garbagecollector.node)(0xc002b6c3c0):struct {}{}, (*garbagecollector.node)(0xc00378e690):struct {}{}}, deletingDependents:false, deletingDependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, beingDeleted:false, beingDeletedLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, virtual:true, virtualLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, owners:[]v1.OwnerReference(nil)} no longer virtual in the graph, skipping attemptToDeleteItem on virtual node
Dec 20 10:37:04 vm-rodrigoq-8687 kube-controller-manager[334]: I1220 10:37:04.201049     334 garbagecollector.go:408] object d7956af1-c25f-4952-9ed7-6d6ac92f985f's owner apps.cloudrobotics.com/v1alpha1/ResourceSet, kube-proxy.v18 is not found
Dec 20 10:37:04 vm-rodrigoq-8687 kube-controller-manager[334]: I1220 10:37:04.201097     334 garbagecollector.go:515] classify references of [apps/v1/DaemonSet, namespace: kube-system, name: kube-proxy, uid: d7956af1-c25f-4952-9ed7-6d6ac92f985f].
Dec 20 10:37:04 vm-rodrigoq-8687 kube-controller-manager[334]: dangling: []v1.OwnerReference{v1.OwnerReference{APIVersion:"apps.cloudrobotics.com/v1alpha1", Kind:"ResourceSet", Name:"kube-proxy.v18", UID:"39e9d246-62db-4acd-8fe7-6105dcedea3e", Controller:(*bool)(nil), BlockOwnerDeletion:(*bool)(0xc0036a8407)}}
Dec 20 10:37:04 vm-rodrigoq-8687 kube-controller-manager[334]: I1220 10:37:04.201147     334 garbagecollector.go:580] "Deleting object" object="kube-system/kube-proxy" objectUID=d7956af1-c25f-4952-9ed7-6d6ac92f985f kind="DaemonSet" propagationPolicy=Background
# ^^^^ this is the mistaken deletion

# vvv here is a later log, this is what you would normally see of the DaemonSet after the ownerReference was updated
Dec 20 10:37:08 vm-rodrigoq-8687 kube-controller-manager[334]: I1220 10:37:08.844263     334 garbagecollector.go:519] object garbagecollector.objectReference{OwnerReference:v1.OwnerReference{APIVersion:"rbac.authorization.k8s.io/v1", Kind:"ClusterRoleBinding", Name:"kubeadm:node-proxier", UID:"f12b9ee0-59c0-4632-80e9-23f7dfbd46a6", Controller:(*bool)(nil), BlockOwnerDeletion:(*bool)(nil)}, Namespace:""} has at least one existing owner: []v1.OwnerReference{v1.OwnerReference{APIVersion:"apps.cloudrobotics.com/v1alpha1", Kind:"ResourceSet", Name:"kube-proxy.v19", UID:"41aaee93-9741-4d79-8913-533e67015449", Controller:(*bool)(nil), BlockOwnerDeletion:(*bool)(0xc00547e12e)}}, will not garbage collect
```
</details>

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.12", GitCommit:"c6939792865ef0f70f92006081690d77411c8ed5", GitTreeState:"clean", BuildDate:"2022-09-21T12:20:29Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.12", GitCommit:"c6939792865ef0f70f92006081690d77411c8ed5", GitTreeState:"clean", BuildDate:"2022-09-21T12:13:07Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
kubeadm-based single-node cluster running on GCP MV
</details>


### OS version

<details>

```console
# On Linux:
# This is a non-public fork of Google's Container-optimized OS.
$ uname -a
Linux vm-rodrigoq-8687 5.15.72-rt51+ #1 SMP PREEMPT_RT Mon Dec 19 17:24:11 CET 2022 x86_64 Intel(R) Xeon(R) CPU @ 2.20GHz GenuineIntel GNU/Linux
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>
```
$ containerd --version
containerd github.com/containerd/containerd 1.6.12 36ae9ea499e67e0aea79989e9478e307a1e45bee
```
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>