Repository: kubernetes
Issue: API server creates one etcd client per CRD
Labels: kind/bug,sig/api-machinery,triage/accepted,sig/etcd

### What happened?

Currently the API server is creating one etcd client per CRD. This causes it to use more memory and more TCP connections than it should need to. In https://github.com/kubernetes/kubernetes/issues/111477 we significantly reduced the amount of memory said clients use by updating them to share a single logger, but we should address the underlying issue.

I've heard various folks, including @aojea and @lavalamp mention that they've looked into addressing this but I couldn't immediately find a tracking issue. We were previously discussing this in https://github.com/kubernetes/kubernetes/issues/111476 which was closed by https://github.com/kubernetes/kubernetes/issues/111477, but there seems to be agreement that the "correct fix" is what is tracked by this issue.

### What did you expect to happen?

Presuming that a single etcd client can handle multiple requests at once, I'd expect the API server to use approximately one etcd client (i.e. one https://pkg.go.dev/go.etcd.io/etcd/client/v3#Client) per etcd cluster it's connected to. I say approximately because I see that additional clients are also used for things like health probes.

### How can we reproduce it (as minimally and precisely as possible)?

Create a kind cluster and deploy a few CRDs. 

```console
$ k get crd|wc -l
1878

# Enter the kind cluster's control plane container.
$ docker exec -it e092b6d27bf-withlogger-control-plane bash

# Find the API server's PID
root@e092b6d27bf-withlogger-control-plane:/# ps -C kube-apiserver
    PID TTY          TIME CMD
    493 ?        00:29:03 kube-apiserver

# Count how many etcd connections exist with the 1,878 CRDs loaded
# Column 3 is the remote host and port per https://www.kernel.org/doc/Documentation/networking/proc_net_tcp.txt
# 0100007F:094B is hex for 127.0.0.1:2379 (the etcd client port)
root@e092b6d27bf-withlogger-control-plane:/# cat /proc/493/net/tcp|awk '{ print $3 }'|grep '0100007F:094B'|wc -l
1937

# Back outside of the kind control plane container, delete all your CRDs
$ k delete crds --all

# Then back inside the container re-count the connections
root@e092b6d27bf-withlogger-control-plane:/# cat /proc/493/net/tcp|awk '{ print $3 }'|grep '0100007F:094B'|wc -l
60
```

Note that 1,937 - 60 = 1,877 - the number of CRDs we have loaded (the 1,878 above includes the header line).

### Anything else we need to know?

Based on my read of the code, I see roughly:

* All of /apis is handled by a *crdHandler
* When a request comes in it gets an instance of the crd (from an informer)
* Gets serving info from a map of CRDs - if there is none it...
* Calls customresource.NewStorage which creates a genericregistry.Store
* Calls its CompleteWithOptions method
* That calls its “Decorator”, which is a genericregistry.StorageWithCacher
* That calls generic.NewRawStorage which just wraps factory.Create
* factory.Create calls newETCD3Storage which calls newETCD3Client

### Kubernetes version

<details>

Commit e000a2edb754af398340c13e97c669bfccf3bc72 (current master)

</details>


### Cloud provider

<details>
N/A
</details>


### OS version

<details>

N/A

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>