Repository: kubernetes
Issue: Unable to SSA remove array entry containing nested field with "foreign" owner
Labels: kind/bug,sig/api-machinery,triage/accepted

### What happened?

We are in the process of removing a long-time deprecated API version in one of our internal operators. We have done this process before, and are familiar with how it should be done. But since then, we have migrated all our code to use SSA. While this in general has been a pleasant experience, we hit a major issue when our CI/CD pipeline attempted to delete obsolete webhooks from our webhook configuration resources (using SSA). We have both validating and mutating webhooks. The same error was reported for both, and I will use the validating webhook in the following description.

When the pipeline attempts to SSA the webhook configuration, we got the following error:

````
Error from server (Invalid): ValidatingWebhookConfiguration.admissionregistration.k8s.io "application-operator-validating-webhook-configuration" is invalid: [webhooks[2].sideEffects: Required value: must specify one of None, NoneOnDryRun, webhooks[2].clientConfig: Required value: exactly one of url or service is required, webhooks[2].admissionReviewVersions: Required value: must specify one of v1, v1beta1]
````

Pretty cryptic error message, but I am pretty sure this is caused by "foreign" ownership to selected nested fields in the array item that the pipeline is trying to remove, making the item retain with invalid configuration and thus rejected by the SSA. Looking at the managed fields confirms my suspicion:

````yaml
managedFields:
    - manager: cert-manager-cainjector
      operation: Apply
      apiVersion: admissionregistration.k8s.io/v1
      time: '2024-09-03T08:00:49Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:webhooks':
          'k:{"name":"v1alpha1-stasjob-validator.stas.statnett.no"}':
            .: {}
            'f:clientConfig':
              'f:caBundle': {}
            'f:name': {}
          'k:{"name":"v1beta2-application-validator.stas.statnett.no"}':
            .: {}
            'f:clientConfig':
              'f:caBundle': {}
            'f:name': {}
          'k:{"name":"v1beta2-stasjob-validator.stas.statnett.no"}':
            .: {}
            'f:clientConfig':
              'f:caBundle': {}
            'f:name': {}
````

As we use cert-manager to inject the webhook CA bundle, it owns a single nested field in each array item as expected. Note: We have enabled the ServerSideApply feature gate in cert-manager, if that matters.

### What did you expect to happen?

I would expect the obsolete/removed webhook to be removed without error from the array of webhooks in the webhook configuration resource.

### How can we reproduce it (as minimally and precisely as possible)?

There are probably many ways to reproduce this issue. But to reproduce the issue in the same context as us, I would suggest the following:

1. Install cert-manager with mostly default options. We have enabled the c-m ServerSideApply feature gate, but I don't think it matters.
2. Add/update a ValidatingWebhookConfiguration containing at least two webhooks **using SSA**.
3. Create a cert-manager `Certificate` for your webhook.
4. Request injection of webhook CA bundle by cert-manager ca-injector adding an annotation to the VWC: `cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)`
5. Wait for cert-manager to reconcile the VWC (`caBundle` should be injected)
6. Attempt to remove one of the webhooks from the VMC, still **using SSA**:

### Anything else we need to know?

This issue was initially raised on sig-api-machinery Slack: https://kubernetes.slack.com/archives/C0EG7JC6T/p1728996348284339

It is also worth mentioning that a temporary rollback to use client-side apply (a.k.a. Update) in our pipeline appears as a workaround for this issue. We successfully used this approach to finalize our removal of the obsolete API version.

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.29.7
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.8+632b078
```

</details>


### Cloud provider

<details>
N/A OpenShift on VMWare
</details>


### OS version

<details>

N/A RCOS (OpenShift)

</details>


### Install tools

<details>
N/A
</details>


### Container runtime (CRI) and version (if applicable)

<details>
N/A
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
N/A
</details>