Repository: kubernetes
Issue: Cannot remove pod affinity field after a deployment has been created with the field set when using server side apply
Labels: kind/bug,sig/api-machinery,triage/accepted

### What happened?

We have a controller that manages objects on the cluster and uses server side apply.  For this specific case, we are interested in the deployment object which in the example below happens to be the gatekeeper deployment.

Step 1: Controller creates the deployment resource with pod anti affinity field set. 
- Created deployment : [create.txt](https://github.com/kubernetes/kubernetes/files/13275431/create.txt)

Step 2: Controller modifies the deployment object to remove the Pod affinity field and does a server side apply dry run `Patch` on the deployment resource. The dry run does **not** detect a diff. Ignoring the diff and applying the patch also doesn't result in the pod anti affinity field being removed. 
- Updated deployment + managed fields showing the anti affinity field still not removed: 
[updated_deployment.txt](https://github.com/kubernetes/kubernetes/files/13275506/updated_deployment.txt)

Also note that:
- Step 1 and Step2 use the same fieldOwner so there shouldn't be any conflicts here (and both dry run and apply don't show any conflicts). Step1 has managed-field operation `Update` and Step2 has managed-field operation `Apply`. 
- The controller also sets force ownership when applying the patch.

**NOTE:** If we Create the deployment without the affinity field set, then we can set/remove it with a Patch just fine. The problem only seems to occur when the deployment is created with the affinity field set and then we attempt to remove it with a patch. 

Other notes: 
- This seems to work with kubectl --server-side apply.  The only managed-field operation in this case is `Apply` See logs: 
[kubectl_ssa.txt](https://github.com/kubernetes/kubernetes/files/13286243/kubectl_ssa.txt)



The same behavior is also seen on pod tolerations. 

### What did you expect to happen?

Expected the pod anti affinity field to be removed from the live object when updating the deployment object with SSA to remove the affinity field.


### How can we reproduce it (as minimally and precisely as possible)?

1. [Create deployment](https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/client/interfaces.go#L66) with pod anti affinity field set.

2. [Patch deployment](https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/client/interfaces.go#L77
) with SSA to remove the anti affinity field. 
   - Set [Apply](https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/client/patch.go#L30) to use SSA
   - Set client options [FieldOwner](https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/client/options.go#L140) same as the field owner used to Create and [ForceOwnership](https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/client/options.go#L742). 



### Anything else we need to know?

_No response_

### Kubernetes version

</details>


### Cloud provider

<details>
GKE
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>