Repository: kubernetes
Issue: Server-side apply: migration from client-side apply leaves stuck fields in the object
Labels: kind/bug,sig/api-machinery,wg/api-expression,triage/accepted

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->

Migrating from client-side apply to server-side apply (as currently implemented in kubectl and documented [here](https://kubernetes.io/docs/reference/using-api/server-side-apply/)) leaves the objects in a somewhat corrupted state, and result in the resources forever being inconsistent with their expected state. Namely, removed fields that have been set in the initial client-side applied version stay in the resource forever and aren't dropped.

#### What happened:

If I apply a manifest using client-side apply and then switch to server-side apply with no changes, the operation succeeds. However, if I then delete a field from the manifest and run server-side apply again, the field stays in the resource instead of being removed

#### What you expected to happen:

I would obviously expect the field to be removed from the resource.

#### How to reproduce it (as minimally and precisely as possible):

1. Create a configmap in the cluster with client-side apply:

       cat <<EOF | kubectl apply -f -
       apiVersion: v1
       kind: ConfigMap
       metadata:
         name: test
       data:
         key: value
         legacy: unused
       EOF

2. Confirm that the configmap is fine:

       $ kubectl get configmap -o yaml test | egrep -A 3 '^data:'
       data:
         key: value
         legacy: unused
       kind: ConfigMap

3. Apply the same manifest with server-side apply:

       cat <<EOF | kubectl apply --server-side -f -
       apiVersion: v1
       kind: ConfigMap
       metadata:
         name: test
       data:
         key: value
         legacy: unused
       EOF

4. Confirm that it wasn't changed:

       $ kubectl get configmap -o yaml test | egrep -A 3 '^data:'
       data:
         key: value
         legacy: unused
       kind: ConfigMap

5. Remove one of the values from the configmap and apply again using server-side apply:

       cat <<EOF | kubectl apply --server-side -f -
       apiVersion: v1
       kind: ConfigMap
       metadata:
         name: test
       data:
         key: value
       EOF

6. Check the configmap, expecting the `legacy` value to be removed:

       $ kubectl get configmap -o yaml test | egrep -A 3 '^data:'
       data:
         key: value
         legacy: unused
       kind: ConfigMap

7. As you can see, the `legacy` value is still there.

#### Anything else we need to know?:

This happens because the client-side apply results are recorded in `managedFields` with an `Update` operation by the `kubectl-client-side-apply` field manager. When the resource is migrated to server-side apply, the results are instead tracked as `Apply` by the `kubectl` field manager. If there are no conflicts between the two fields (and naturally you wouldn't expect them to be there, since you're trying to convert the same resource to SSA), _both_ field managers will own the fields and the client-side apply won't get kicked out. If you then remove a field from the list of fields managed by SSA, the field will still be managed by the old `Update` from `kubectl-client-side-apply` and wouldn't be removed as a result.

#### Environment:
- Kubernetes version (use `kubectl version`):

      Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T21:51:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
      Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}