Repository: kubernetes
Issue: APIService's status blocked on FailedDiscoveryCheck condition with 503
Labels: kind/bug,sig/network,triage/accepted

### What happened?

We meet a problem that the APISerivce is blocked on the FailedDiscoveryCheck condition:
```
status:
  conditions:
  - lastTransitionTime: "2023-10-15T16:35:28Z"
    message: 'failing or missing response from https://12.4.170.157:6443/apis/cluster.alauda.io/v1:
      bad status from https://12.4.170.157:6443/apis/cluster.alauda.io/v1: 503'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available
```

But we get the right response by directly accessing the endpoint:
```
# curl -ik https://12.4.170.157:6443/apis/cluster.alauda.io/v1 --cert /etc/kubernetes/pki/front-proxy-client.crt --key /etc/kubernetes/pki/front-proxy-client.key -H 'X-Remote-User: system:kube-aggregator' -H 'X-Remote-Group: system:masters'
HTTP/2 200
audit-id: 63a060ef-1e42-4fc0-8d00-d859cddd0763
cache-control: no-cache, private
content-type: application/json
content-length: 393
date: Mon, 16 Oct 2023 02:34:19 GMT

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "cluster.alauda.io/v1",
  "resources": [
    {
      "name": "clusterviews",
      "singularName": "",
      "namespaced": true,
      "kind": "ClusterView",
      "verbs": [
        "delete",
        "deletecollection",
        "get",
        "list",
        "patch",
        "update",
        "watch"
      ]
    }
  ]
}
```

We also checked the endpoints and the IPVS rules, and all of them looked well.

We have met this problem several times. Rebuilding the pods can not make the APIService become ready.
The only way we found that can resolve it temporally is to restart the kube-apiserver.




### What did you expect to happen?

The APIService works well.

### How can we reproduce it (as minimally and precisely as possible)?

It's hard to reproduce it. After reading the source code, we think reproducing the problem needs these conditions:
1. The endpoint of the APIService uses the 6443 port which is also the kube-apiserver's port.
2. The Service existed before but is without the APIService's port, Update the Service to add APIService's port and Create the APIService at almost the same time.
3. The kube-apiservice checks the APISerivice's endpoint when the service IP has been bound to the `kube-ipvs0` network card, but the IPVS rule has not been generated.

The following example may reproduce it:
`kubectl create -f  demo-apiservice-1.yaml`

```
# demo-apiservice-1.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: demo-apiservice
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  type: ClusterIP
  selector:
    app: demo-apiservice
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  generation: 1
  labels:
    app: demo-apiservice
  name: demo-apiservice
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-apiservice
  template:
    metadata:
      labels:
        app: demo-apiservice
    spec:
      containers:
      - image: nginx:alpine
        name: demo-apiservice
```
Wait several seconds to let the Service and Deployment work.

Then 
`kubectl apply -f demo-apiservice-2.yaml`

```
# demo-apiservice-1.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: demo-apiservice
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
    name: port-80
  - port: 6443
    protocol: TCP
    targetPort: 80
    name: port-6443
  type: ClusterIP
  selector:
    app: demo-apiservice
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1.example.com
spec:
  group: example.com
  groupPriorityMinimum: 1000
  insecureSkipTLSVerify: true
  service:
    name: demo-apiservice
    namespace: default
    port: 6443
  version: v1
  versionPriority: 1
```

Then look at the APIService's status.
` kubectl get apiservice v1.example.com -o yaml`

You may see:
```
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiregistration.k8s.io/v1","kind":"APIService","metadata":{"annotations":{},"name":"v1.example.com"},"spec":{"group":"example.com","groupPriorityMinimum":1000,"insecureSkipTLSVerify":true,"service":{"name":"demo-apiservice","namespace":"default","port":6443},"version":"v1","versionPriority":1}}
  creationTimestamp: "2023-10-16T03:41:33Z"
  name: v1.example.com
  resourceVersion: "721774"
  uid: 47c46244-1dc4-4af0-932a-dc78f2c47585
spec:
  group: example.com
  groupPriorityMinimum: 1000
  insecureSkipTLSVerify: true
  service:
    name: demo-apiservice
    namespace: default
    port: 6443
  version: v1
  versionPriority: 1
status:
  conditions:
  - lastTransitionTime: "2023-10-16T03:41:33Z"
    message: 'failing or missing response from https://172.108.103.208:6443/apis/example.com/v1:
      bad status from https://172.108.103.208:6443/apis/example.com/v1: 503'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available
```






### Anything else we need to know?

By digging into the source code we found these implementations work together may cause this problem.

1. The APIService's `AvailableConditionController` uses a cached transport by `transport.New(transportConfig)` which uses `rt, err = tlsCache.get(config)` to get a cached transport. This causes the HTTP clients to always use the same socket to access the APISerivce's service IP.
2. We use kube-proxy's  IPVS mod. The Service existed before but is without the APIService's port. So there is the Service's IP on the `kube-ipvs0` dummy network card. But there isn't the APISerivce's Port in the IPVS entries until the Service has been updated.  If we access the Service IP with the APIServer's port now, we actually access the endpoint on the host. Unfortunately, there is the same port 6443 on the host that can answer this access.
3. After the kube-apiserver has built the socket to the 6443 port with Service IP (actually accessing the 6443 port on the host), the cached transport lets it keep using the same socket. And the IPVS won't break the existing socket even if the rules have been updated.

So all of the above worked together caused my problem. We are planning to change to another port for the APIService to avoid this kind of wrong access.

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
```
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.6", GitCommit:"741c8db18a52787d734cbe4795f0b4ad860906d6", GitTreeState:"clean", BuildDate:"2023-09-13T09:21:34Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.6", GitCommit:"741c8db18a52787d734cbe4795f0b4ad860906d6", GitTreeState:"clean", BuildDate:"2023-09-13T09:14:09Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>