Repository: kubernetes
Issue: Failure cluster [6140da15...] e2e Flake: Finished pod is deleted before events can be retrieved
Labels: priority/important-soon,sig/node,kind/flake,kind/failing-test,triage/accepted

### Failure cluster [6140da152b810b30d99b](https://go.k8s.io/triage#6140da152b810b30d99b)

##### Error text:
```
[FAILED] timed out waiting for the condition
In [It] at: test/e2e/node/pods.go:276 @ 01/02/23 10:40:40.11

```
#### Recent failures:
[1/9/2023, 3:08:19 PM pr:pull-kubernetes-e2e-kind](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/114879/pull-kubernetes-e2e-kind/1612541338065047552)
[1/9/2023, 12:37:26 PM pr:pull-kubernetes-e2e-kind-ipv6](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/batch/pull-kubernetes-e2e-kind-ipv6/1612502921440661504)
[1/9/2023, 9:57:40 AM pr:pull-kubernetes-e2e-kind-ipv6](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/114846/pull-kubernetes-e2e-kind-ipv6/1612463069030518784)
[1/6/2023, 6:35:47 PM pr:pull-kubernetes-e2e-kind-ipv6](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/113744/pull-kubernetes-e2e-kind-ipv6/1611506307183939584)
[1/6/2023, 4:48:42 PM pr:pull-kubernetes-e2e-kind-ipv6](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/114883/pull-kubernetes-e2e-kind-ipv6/1611479334852235264)


/kind failing-test
/kind flake
<!-- If this is a flake, please add: /kind flake -->

/sig node

#### Additional Information

It looks like the [test](https://github.com/kubernetes/kubernetes/blob/2f6c4f5eab85d3f15cd80d21f4a0c353a8ceb10b/test/e2e/node/pods.go#L232) does the following:
1. [Creates a pod that has a container that immediately exits (command is `/bin/true`)](https://github.com/kubernetes/kubernetes/blob/2f6c4f5eab85d3f15cd80d21f4a0c353a8ceb10b/test/e2e/node/pods.go#L237-L265)
2. [Confirms that the pod has completed successfully](https://github.com/kubernetes/kubernetes/blob/2f6c4f5eab85d3f15cd80d21f4a0c353a8ceb10b/test/e2e/node/pods.go#L271)
3. [Attempts to read the events for the pod](https://github.com/kubernetes/kubernetes/blob/2f6c4f5eab85d3f15cd80d21f4a0c353a8ceb10b/test/e2e/node/pods.go#L276-L292)

When this test flakes/fails, it seems like steps 1 and 2 are successful, but it times out on step 3 because apparently the pod no longer exists.

This makes me wonder if, under certain load conditions, the pod GC is cleaning up the terminated pod before step 3 (or even step 2) can do its thing.

It [looks like the default value for `--terminated-pod-gc-threshold` is 12500](https://github.com/kubernetes/kubernetes/blob/e01ff1641c7321ac81fe5775f6ccb21aa6775c04/pkg/controller/podgc/config/v1alpha1/defaults.go#L34),

That seems high enough that it would be unlikely that the threshold would be reached (could there really be 12500 completed pods?), but I wonder if something in the e2e tests is lowering it.  I noticed [this line which sets `TERMINATED_POD_GC_THRESHOLD` to 100](https://github.com/kubernetes/kubernetes/blob/545835e28826d888b3d236f71fbbd88107d3b635/cluster/gce/config-test.sh#L163), but that is gce-specific.  Is it possible something similar is being set to lower the threshold for the e2e-kind tests?  I did a search but didn't see anywhere that it was being set.