Repository: kubernetes
Issue: owned Custom Resources not garbage collected after CRD API version removal
Labels: kind/bug,sig/api-machinery,lifecycle/frozen,triage/accepted

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


#### What happened:

We are trying to deprecate and remove some CRD API versions and as part of that to ensure that users who have originally deployed Custom Resources at the legacy versions can upgrade.
For a Custom Resource X that is a dependent on another Custom Resource Y and they both have been originally created using a legacy API version `v1alpha2`, we observe that after an upgrade to a controller version where `v1alpha2` is no longer served, X is not garbage collected after Y has been deleted.

X after the upgrade:
```
apiVersion: cert-manager.io/v1
kind: CertificateRequest
metadata:
  annotations:
    cert-manager.io/certificate-name: test1
...
name: test1-4d8sc
  namespace: default
  ownerReferences:
  - apiVersion: cert-manager.io/v1alpha2
    blockOwnerDeletion: true
    controller: true
    kind: Certificate
    name: test1
    uid: a20001e7-4793-42bf-955a-a6216cbdc933
...
```

Y after the upgrade:
```
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
...
  name: test1
  namespace: default
  resourceVersion: "12276"
  uid: a20001e7-4793-42bf-955a-a6216cbdc933
```

In the above config `CertificateRequest.metadata.ownerReferences[0].apiVersion` is incorrect. I thought that perhaps this is the issue, so I tried editing it `v1alpha2` -> `v1` just before running the delete command. The edited `CertificateRequest` was still not deleted.
Does the `metadata.ownerReferences[0].apiVersion` matter- I seem to remember seeing somewhere on #sig-api-machinery Slack that it is not actually used?

#### What you expected to happen:

I expected that X would be deleted after Y is deleted as it happens normally with these resources.

#### How to reproduce it (as minimally and precisely as possible):
1. Deploy `cert-manager` version `v0.16` where the storage version for all CRDs is `v1alpha2`
2. Create an `Certificate` (Y) Custom Resource using the `v1alpha2` API. A `CertificateRequest` X will be created for Y with Y set as X's owner reference.
3. Upgrade to a newer version of `cert-manager` where the only storage version is `v1` (`v1alpha2` is still being served)
4. Upgrade stored versions of Custom Resources and `status.storedVersions` of CRDs as described in Kubernetes docs [here](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#upgrade-existing-objects-to-a-new-stored-version)
5. Upgrade to a version of `cert-manager` where `v1` is the only served version
6. Delete Y `kubectl delete certificate test1`
7. Observe that X is not deleted even after ~30 minutes, and `deletionTimestamp` is not set. 

#### Anything else we need to know?:

Please let me know how I can go about debugging this or any additional information I could provide?

#### Environment:
- Kubernetes version (use `kubectl version`):
on GKE:
```
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.6-gke.1000", GitCommit:"3ae0998c5052f420a17cb96bacf860ec5d6822a3", GitTreeState:"clean", BuildDate:"2021-04-29T09:17:16Z", GoVersion:"go1.15.10b5", Compiler:"gc", Platform:"linux/amd64"}
```
on Kind:
```
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-18T01:10:20Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
```

- Cloud provider or hardware configuration: GKE (1.20.6-gke.1000), Kind
- OS (e.g: `cat /etc/os-release`):
- Kernel (e.g. `uname -a`): `Linux irbe-XPS-13-9310 5.8.0-53-generic #60-Ubuntu SMP Thu May 6 07:46:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux`
- Install tools: kubectl, helm
- Network plugin and version (if this is a network-related bug):
- Others:

/cc @munnerz