Repository: kubernetes
Issue: Potential inconsistent behaviour with apiserver apiextensions (custom resources) and 404s
Labels: kind/bug,sig/api-machinery,triage/accepted

### What happened?

I'm working on a system the extends the kubernetes api server. 
This system consists of a kubernetes cluster, with several custom resource definitions and custom resources of each kind. 
These resources are consumed by _in-cluster_ and _external clients_. The difference between the two types of clients is that the in-cluster client uses a cached client for reads, while external clients use the REST API for reads as well as writes. 

Including the kubernetes components and other components that are deployed, the system has a few dozen in-cluster clients (including kubelets). The number of external clients is > 5000. 

Recently, there was an issue with all the k8s-apiservers of the cluster restarting at the same time. The in-cluster clients failed during this time window, eventually resulting in container restarts. And this was expected. 

The **potential inconsistent behaviour is in relation to the external clients**. Per system logs _(apiserver audit logs)_, around the time of the incident, the following two behaviours were observed
* external clients receiving a 5xx response, (expected)
* external clients receiving 404s for resources that exist. 

Most of the external clients got 5xx's, but there were four instances where the response was a 404. Even in these cases (404s), the clients received 200 responses within ~30 seconds of receiving a 404.

During the incident window, the restarts of the k8s apiservers were observed at different times over a ~30s window. No delay was observed in the container restarts, and each apiserver pod was back up and receiving traffic within 30-40 seconds (or sooner). 
* per the audit logs, the same external client received, 200, 404, 200, 404, 200, 200, 200, .... (at ~30s intervals). 
* the audit logs are missing any information regarding the apiserver instance that emitted the log. 
* it's likely that the different 404s were seen by the same external client (for the same resource), for responses from two different apiserver instances

### What did you expect to happen?

All external clients should've received 5xx response codes if apiserver was down. Otherwise, the response should have been a 200, (because the resource the client was requesting was known to exist in etcd).

### How can we reproduce it (as minimally and precisely as possible)?

* create k8s cluster. 
* add several custom resource definitions, and several resources of each kind
* (may or may not be relevant: etcd size is approximately 4 GB) 
* run a few clients that use the k8s REST API to query for different resources that were created earlier
* (overall rate of query ^ should be high, so that the intermittent inconsistency is observed). 
* shut down all api servers.
* (clients continue: start seeing 5xx)
* restart the api servers. 
* watch for 404s in client logs
* since clients are only requesting resources that are already created, 404 is not an expected response. 

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

_ignore client version, that's from my local system_

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.10", GitCommit:"a7a32748b5c60445c4c7ee904caf01b91f2dbb71", GitTreeState:"clean", BuildDate:"2022-02-16T11:18:16Z", GoVersion:"go1.16.14", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>

The system described in this issue is a kubernetes cluster deployed on Azure compute nodes. 

</details>


### OS version

<details>

```console
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ uname -a
Linux cp-vmss000001 5.15.0-1019-azure #24~20.04.1-Ubuntu SMP Tue Aug 23 15:52:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>