Repository: kubernetes
Issue: Namespace stuck in Terminating with foregroundDeletion finalizer despite no resources being present in the namespace
Labels: kind/bug,sig/api-machinery,triage/accepted

### What happened?

Intermittently, during our acceptance tests, we create and destroy namespaces and these namespaces sometimes get stuck in the Terminating state despite no resources existing in the namespace. Manually removing the `foregroundDeletion` finalizer allows the namespace to be deleted.

### What did you expect to happen?

Namespaces which contain no resources terminate when a deletion is requested.

### How can we reproduce it (as minimally and precisely as possible)?

Create/delete namespaces

### Anything else we need to know?

Command used (which produced no output) to verify no objects present in the namespace:

`kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n dev-fuse-namespace`

output of `kubectl get ns -o yaml dev-fuse-namespace` which is an example of a stuck namespace:

```
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2022-07-19T07:30:44Z"
  deletionTimestamp: "2022-07-19T07:30:55Z"
  finalizers:
  - foregroundDeletion
  labels:
    kubernetes.io/metadata.name: dev-fuse-namespace
  name: dev-fuse-namespace
  resourceVersion: "44884564"
  uid: a07518d6-69d1-4cd0-831d-521cdac16dca
spec: {}
status:
  conditions:
  - lastTransitionTime: "2022-07-19T07:31:01Z"
    message: All resources successfully discovered
    reason: ResourcesDiscovered
    status: "False"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2022-07-19T07:31:01Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2022-07-19T07:31:01Z"
    message: All content successfully deleted, may be waiting on finalization
    reason: ContentDeleted
    status: "False"
    type: NamespaceDeletionContentFailure
  - lastTransitionTime: "2022-07-19T07:31:40Z"
    message: All content successfully removed
    reason: ContentRemoved
    status: "False"
    type: NamespaceContentRemaining
  - lastTransitionTime: "2022-07-19T07:31:01Z"
    message: All content-preserving finalizers finished
    reason: ContentHasNoFinalizers
    status: "False"
    type: NamespaceFinalizersRemaining
  phase: Terminating
```

kube-controller-manager logs regarding the namespace:

```
I0719 07:30:55.571466       1 graph_builder.go:587] add [projectcalico.org/v3/Profile, namespace: , name: kns.dev-fuse-namespace, uid: a07518d6-69d1-4cd0-831d-521cdac16dca] to the attemptToDelete, because it's waiting for its dependents to be deleted
I0719 07:30:55.571543       1 garbagecollector.go:471] "Processing object" object="kns.dev-fuse-namespace" objectUID=a07518d6-69d1-4cd0-831d-521cdac16dca kind="Profile" virtual=false
I0719 07:30:55.614487       1 garbagecollector.go:590] remove DeleteDependents finalizer for item [projectcalico.org/v3/Profile, namespace: , name: kns.dev-fuse-namespace, uid: a07518d6-69d1-4cd0-831d-521cdac16dca]
E0719 07:31:01.127156       1 tokens_controller.go:262] error synchronizing serviceaccount dev-fuse-namespace/default: secrets "default-token-vvnpw" is forbidden: unable to create new content in namespace dev-fuse-namespace because it is being terminated
```


### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.10", GitCommit:"eae22ba6238096f5dec1ceb62766e97783f0ba2f", GitTreeState:"clean", BuildDate:"2022-05-24T12:56:35Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.11", GitCommit:"5824e3251d294d324320db85bf63a53eb0767af2", GitTreeState:"clean", BuildDate:"2022-06-16T05:33:55Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
```
</details>


### Cloud provider

<details>
AWS
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
Warning: Permanently added 'ip-10-121-12-134.us-west-2.compute.internal' (ECDSA) to the list of known hosts.
NAME="Rocky Linux"
VERSION="8.6 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.6 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8"
$ uname -a
Linux <redacted hostname> 4.18.0-372.9.1.el8.x86_64 #1 SMP Tue May 10 14:48:47 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>
Custom install using Terraform and home grown scripts.
</details>


### Container runtime (CRI) and version (if applicable)

<details>
crio 1.22.5
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
n/a
</details>