Repository: kubernetes
Issue: API Server Keeps Using Open TCP Connections to Terminating Admission Controller Pods
Labels: kind/bug,sig/network,sig/api-machinery,triage/accepted

### What happened?

I am encountering an issue where the Kubernetes API server continues to use open TCP connections to an admission controller pod after it has been marked as unready and removed from the service endpoints.

### My Setup

- I have a `ValidatingWebhookConfiguration` pointing to my service `admission-controller`, which uses an `ignore` failure policy.
- The `admission-controller` deployment consists of multiple pods, each running an HTTP server. The service selector targets all pods in the deployment, leading to multiple endpoints.
- Upon pod termination, the goal is to perform a graceful shutdown by:
  - Marking the pod as unready.
  - Waiting until the pod is removed from the service endpoints.
  - Handling inflight messages before shutting down the pod.
 
### The Issue
Once a pod is marked as unready, it is correctly removed from the service endpoints and does not receive new connections—this is expected behavior.
However, the pod continues to receive HTTP requests over old open TCP connections. 

This leads to two major problems:

1. I cannot close the TCP connections on the server side because it may result in a potential loss of requests from the API server.
2. There is no defined deadline for when the connection gets closed on the client side, leaving it open indefinitely unless manually terminated.

Additionally, I discovered that the API server does not retry requests (and opens a new TCP connection) if it encounters a closed connection.

### Stress Test Results
I performed a test to further diagnose the issue:

1. Deployed one `admission-controller` pod in the deployment (intended to block all requests).
2. Ran the following command: 
   ```kubectl rollout restart deployment admission-controller```
   - A new pod was created, and the old pod began terminating.
   - The old pod closed the server as soon as it became unready.
4. Sent 1000 requests to create a pod to simulate stress.

#### Expected Result
No pods should be created, as the `admission-controller` should block all the requests.

#### Actual Result
Some pods were created during each run. There appears to be a small window of time where the cluster is unprotected, and the API server does not receive a response because of the failurePolicy (ignore). As a result, it proceeds with pod creation.

### Impact
This is particularly concerning because during that window where old TCP connections are still in use, the API server can bypass the admission controller, leading to potential security risks.

### What I’ve Tried
I attempted to gracefully handle the shutdown by waiting for connections to close naturally, but I am unable to define a clear deadline or force the API server to retry on connection closure.


Any assistance or guidance on how to address this issue would be greatly appreciated.

### What did you expect to happen?

1. The TCP connections used by the API server should have a reasonable time-to-live, allowing the pod to wait long enough to ensure all existing TCP connections are properly closed upon termination.
2. I expect the API server to have a built-in retry mechanism to handle such failures.

### How can we reproduce it (as minimally and precisely as possible)?

As mentioned above using the described setup and stressed tests.

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

$ kubectl version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.3-gke.1639000
```

Tested on both Minikube and GKE.</details>


### Cloud provider

<details>
Tested on both Minikube and GKE.
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>