Repository: kubernetes
Issue: Using Dial function with client-go kubernetes.NewForConfig triggers a memory leak
Labels: kind/bug,sig/api-machinery,triage/accepted

### What happened?

After we upgraded k8s.io/client-go package to v0.26.0, we noticed memory leaks in our components that access the Kubernetes API.  For some components we managed to remove the leak by rolling back client-go to 0.25.4, but for other components it creates a dependency hell.

After some profiling and debugging I managed to write a mockup code that is roughly based on our production code and reliably triggers the leak in client-go 0.26.0 to 0.27.3.  The code is uploaded to https://github.com/kublr/memory_leak_demo .  It includes the pprof endpoint so you can verify the leak.  The memory allocations happen in transport.(*tlsTransportCache)get method.

The leak is triggered by the presence of a custom Dial function in rest.Config object.  As I infer from the commits like https://github.com/kubernetes/kubernetes/pull/117311 , this usage is now considered incorrect, but we need some guidance or advice how to adapt our production code.


### What did you expect to happen?

We expected to use public interfaces without triggering memory leaks in the package. 

### How can we reproduce it (as minimally and precisely as possible)?

Compile and run the sample code in https://github.com/kublr/memory_leak_demo .  Using pprof you can observe growing amount of memory allocated in transport(*tlsTransportCache)get(), and using the debugger you can observe that every call of kubernetes.NewForConfig() creates a new entry in TLS cache.

### Anything else we need to know?

As I understand, the real culprit is line https://github.com/kubernetes/kubernetes/blob/fa78f28f0a011e9922cf12145dd602c5a63981e8/staging/src/k8s.io/client-go/rest/transport.go#L115 , where client-go creates a DialHolder structure for configs that do not have one, but have a Dial hook.

Because DialHolder pointer is added to the **copy** of the original config (at least in the code path originating from kubernetes.NewForConfig), the next calls to kubernetes.NewForConfig will result in the new DialHolder instances.

Because pointer to the DialHolder instance is a part of TLS cache key, this creates a sequence of unique map keys that produce useless cache entries (that will never be a cache hit), plus a memory leak.  

I reiterate that the problem is not a memory leak itself, but a broken cache with unique cache keys.  I think we would be better to mark configs with custom Dial hook as non-cacheable.

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.10", GitCommit:"e770bdbb87cccdc2daa790ecd69f40cf4df3cc9d", GitTreeState:"archive", BuildDate:"2023-05-19T08:42:33Z", GoVersion:"go1.19.9", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.9", GitCommit:"a1a87a0a2bcd605820920c6b0e618a8ab7d117d4", GitTreeState:"clean", BuildDate:"2023-04-12T12:08:36Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}
```
We also observed the leaks with k8s versions from 1.22 to 1.26
</details>


### Cloud provider

<details>
We observed the leak on vSphere, Azure and onprem installations.  Most likely all clouds are affected.
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ uname -a
Linux dvi-7696-1686743539-vsp1-master-0 5.4.0-80-generic #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Install tools

<details>
custom
</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>