Repository: kubernetes
Issue: Custom resource create might succeed after CRD deletion, leaving data in etcd
Labels: kind/bug,sig/api-machinery,help wanted,priority/important-longterm,area/custom-resources,lifecycle/frozen,triage/accepted

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->

Either `/priority important-longterm` or `/priority backlog`

#### What happened:

The chance for the race condition is low. There is a narrow window where CRs can be created after the CRD gets deleted:

1. CR create request hit API server; API server noted the request [isn't terminating yet](https://github.com/kubernetes/kubernetes/blob/2a05c786008f9d0592c0ea51dfb49124bc622100/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go#L316), went ahead to prepare the handler
2. CRD delete request hit API server; CRD got [marked as terminating](https://github.com/kubernetes/kubernetes/blob/2a05c786008f9d0592c0ea51dfb49124bc622100/staging/src/k8s.io/apiextensions-apiserver/pkg/registry/customresourcedefinition/etcd.go#L148-L149)
3. CRD finalizer controller deleted existing CRs, removed [CRD cleanup finalizer](https://github.com/kubernetes/kubernetes/blob/2a05c786008f9d0592c0ea51dfb49124bc622100/staging/src/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go#L125-L174); CRD got deleted from etcd
4. API server [served the CR create request](https://github.com/kubernetes/kubernetes/blob/2a05c786008f9d0592c0ea51dfb49124bc622100/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go#L380), wrote data to etcd
5. destruction of the CR etcd client happened after `#3`, and might [last a long time](https://github.com/kubernetes/kubernetes/blob/2a05c786008f9d0592c0ea51dfb49124bc622100/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go#L583-L584) to allow `#4` to happen-- as long as `#4` [registered itself to the in-flight requests](https://github.com/kubernetes/kubernetes/blob/2a05c786008f9d0592c0ea51dfb49124bc622100/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go#L379)

The window is narrow. It is more likely either
- `#4` happened before `#3`-- the created CR got cleaned up by CRD finalizer
- `#4` happened after `#5`-- the create request would get a 503 Service Unavailable "apiserver is shutting down"

#### What you expected to happen:

No CR data persists in etcd after CRD deletion. 

#### How to reproduce it (as minimally and precisely as possible):

- Add `time.Sleep` to make triggering the race easier: https://github.com/roycaihw/kubernetes/commit/7f4d6cbf7ac015edd5dc799c97ab5bc49f814f1c
- `kubectl apply -f crd.yaml` (example yaml: https://github.com/roycaihw/kubernetes/commit/0d3732cab99e79118eeafca0534e1c28dc8c67d9)
- `kubectl apply -f cr.yaml`; immediately `kubectl delete -f crd.yaml` in parallel
- See the CRD was deleted immediately, the CR got 201 CREATED 10s later
- Moreover, re-create the CRD and see the old CR still exists `kubectl apply -f crd.yaml && kubectl get crontabs`

#### Anything else we need to know?:

The narrow window becomes a concern if we increase the latency between `#1` and `#4`, which is what https://github.com/kubernetes/kubernetes/pull/96403 is doing. A mitigation is to read the latest CRD from cache after the increased latency and right before `#4`, to make sure the window is still small: https://github.com/kubernetes/kubernetes/pull/96403/commits/6ddc1eb1fb3c85a241a87ccf66c142be578ff4e3. The race still exist, but it isn't getting worse. 

#### Environment:
- Kubernetes version (use `kubectl version`): HEAD

/sig api-machinery