Repository: kubernetes
Issue: The "health handlers should contain necessary checks" test fails from sig-api-machinery TC suite when kube-apiserver not exposed directly
Labels: kind/bug,area/test,sig/api-machinery,sig/testing,needs-triage

### What happened?

We faced an issue when executing one of the CNTT test case.

In our environment, the kube-apiserver not exposed directly, it can be reached via an nginx POD within the cluster which proxies the requests.
Client (running CNTT tests) => POD (running in the cluster, as proxy) => kube-apiserver
The kube-config configured properly to reach that endpoint:

```
clusters:
- cluster:
    server: https://x.y.z.73:8082/proxy/apiserver
...
```

Executing the sig_api_machinery test case, the "health handlers should contain necessary checks" test case failed.


```
podman run -it --env-file ~/opnfv/env \
-v ~/opnfv/ca.pem:/home/opnfv/functest/ca.pem:Z \
-v ~/opnfv/config:/root/.kube/config:Z \
-v ~/opnfv/repositories.yml:/home/opnfv/functest/repositories.yml:Z \
-v ~/opnfv/results:/home/opnfv/functest/results:Z \
opnfv/functest-kubernetes-smoke:v1.23 run_tests -t sig_api_machinery
```

Results:
```
STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in health-9251
STEP: Waiting for a default service account to be provisioned in namespace
[It] should contain necessary checks
 /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/health_handlers.go:120
STEP: /health
Dec 15 13:54:27.529: FAIL: Unexpected error:
   <*errors.errorString | 0xc0037fcd80>: {
       s: "missing required /healthz?verbose=1 checks: map[[+]autoregister-completion ok:{} [+]etcd ok:{} [+]log ok:{} [+]ping ok:{} [+]poststarthook/apiservice-openapi-controller ok:{} [+]poststarthook/apiservice-registration-controller ok:{} [+]poststarthook/apiservice-status-available-controller ok:{} [+]poststarthook/bootstrap-controller ok:{} [+]poststarthook/crd-informer-synced ok:{} [+]poststarthook/generic-apiserver-start-informers ok:{} [+]poststarthook/kube-apiserver-autoregistration ok:{} [+]poststarthook/scheduling/bootstrap-system-priority-classes ok:{} [+]poststarthook/start-apiextensions-controllers ok:{} [+]poststarthook/start-apiextensions-informers ok:{} [+]poststarthook/start-cluster-authentication-info-controller ok:{} [+]poststarthook/start-kube-aggregator-informers ok:{} [+]poststarthook/start-kube-apiserver-admission-initializer ok:{}] in: ",
   }
   missing required /healthz?verbose=1 checks: map[[+]autoregister-completion ok:{} [+]etcd ok:{} [+]log ok:{} [+]ping ok:{} [+]poststarthook/apiservice-openapi-controller ok:{} [+]poststarthook/apiservice-registration-controller ok:{} [+]poststarthook/apiservice-status-available-controller ok:{} [+]poststarthook/bootstrap-controller ok:{} [+]poststarthook/crd-informer-synced ok:{} [+]poststarthook/generic-apiserver-start-informers ok:{} [+]poststarthook/kube-apiserver-autoregistration ok:{} [+]poststarthook/scheduling/bootstrap-system-priority-classes ok:{} [+]poststarthook/start-apiextensions-controllers ok:{} [+]poststarthook/start-apiextensions-informers ok:{} [+]poststarthook/start-cluster-authentication-info-controller ok:{} [+]poststarthook/start-kube-aggregator-informers ok:{} [+]poststarthook/start-kube-apiserver-admission-initializer ok:{}] in: 
occurred

Full Stack Trace
k8s.io/kubernetes/test/e2e/apimachinery.glob..func14.1()
       /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/health_handlers.go:123 +0xc5
k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000d8e300)
       _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c
k8s.io/kubernetes/test/e2e.TestE2E(0xc000d8e300)
       _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:144 +0x2b
testing.tRunner(0xc000d8e300, 0x70f99e8)
       /usr/local/go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
       /usr/local/go/src/testing/testing.go:1238 +0x2b3
[AfterEach] [sig-api-machinery] health handlers
 /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186
STEP: Collecting events from namespace "health-9251".
STEP: Found 0 events.

...

• Failure [31.712 seconds]
[sig-api-machinery] health handlers
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/framework.go:23
 should contain necessary checks [It]
 /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/health_handlers.go:120

 Dec 15 13:54:27.529: Unexpected error:
     <*errors.errorString | 0xc0037fcd80>: {
         s: "missing required /healthz?verbose=1 checks: map[[+]autoregister-completion ok:{} [+]etcd ok:{} [+]log ok:{} [+]ping ok:{} [+]poststarthook/apiservice-openapi-controller ok:{} [+]poststarthook/apiservice-registration-controller ok:{} [+]poststarthook/apiservice-status-available-controller ok:{} [+]poststarthook/bootstrap-controller ok:{} [+]poststarthook/crd-informer-synced ok:{} [+]poststarthook/generic-apiserver-start-informers ok:{} [+]poststarthook/kube-apiserver-autoregistration ok:{} [+]poststarthook/scheduling/bootstrap-system-priority-classes ok:{} [+]poststarthook/start-apiextensions-controllers ok:{} [+]poststarthook/start-apiextensions-informers ok:{} [+]poststarthook/start-cluster-authentication-info-controller ok:{} [+]poststarthook/start-kube-aggregator-informers ok:{} [+]poststarthook/start-kube-apiserver-admission-initializer ok:{}] in: ",
     }
     missing required /healthz?verbose=1 checks: map[[+]autoregister-completion ok:{} [+]etcd ok:{} [+]log ok:{} [+]ping ok:{} [+]poststarthook/apiservice-openapi-controller ok:{} [+]poststarthook/apiservice-registration-controller ok:{} [+]poststarthook/apiservice-status-available-controller ok:{} [+]poststarthook/bootstrap-controller ok:{} [+]poststarthook/crd-informer-synced ok:{} [+]poststarthook/generic-apiserver-start-informers ok:{} [+]poststarthook/kube-apiserver-autoregistration ok:{} [+]poststarthook/scheduling/bootstrap-system-priority-classes ok:{} [+]poststarthook/start-apiextensions-controllers ok:{} [+]poststarthook/start-apiextensions-informers ok:{} [+]poststarthook/start-cluster-authentication-info-controller ok:{} [+]poststarthook/start-kube-aggregator-informers ok:{} [+]poststarthook/start-kube-apiserver-admission-initializer ok:{}] in: 
 occurred

 /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/health_handlers.go:123
------------------------------
{"msg":"FAILED [sig-api-machinery] health handlers should contain necessary checks","total":1,"completed":0,"skipped":4622,"failed":1,"failures":["[sig-api-machinery] health handlers should contain necessary checks"]}
```


After checking the code, it seems it's the same issue as described here:
https://github.com/kubernetes/kubernetes/issues/115200

So, the test use the testPath which use RequestURI function. Therefore the query not use the full path configured in the kube-config, it keeps only the IP:PORT and then add the /health. Which of course will fail on our setup.

https://github.com/kubernetes/kubernetes/blob/release-1.23/test/e2e/apimachinery/health_handlers.go#L117
```
func testPath(client clientset.Interface, path string, requiredChecks sets.String) error {
	var result restclient.Result
	err := wait.Poll(100*time.Millisecond, 30*time.Second, func() (bool, error) {
		result = client.CoreV1().RESTClient().Get().RequestURI(path).Do(context.TODO())
		status := 0
		result.StatusCode(&status)
		return status == 200, nil
	})
	if err != nil {
		return err
	}
	body, err := result.Raw()
	if err != nil {
		return err
	}
	checks := sets.NewString(strings.Split(string(body), "\n")...)
	if missing := requiredChecks.Difference(checks); missing.Len() > 0 {
		return fmt.Errorf("missing required %s checks: %v in: %s", path, missing, string(body))
	}
	return nil
}

var _ = SIGDescribe("health handlers", func() {
	f := framework.NewDefaultFramework("health")

	ginkgo.It("should contain necessary checks", func() {
		ginkgo.By("/health")
		err := testPath(f.ClientSet, "/healthz?verbose=1", requiredHealthzChecks)
		framework.ExpectNoError(err)

		ginkgo.By("/livez")
		err = testPath(f.ClientSet, "/livez?verbose=1", requiredLivezChecks)
		framework.ExpectNoError(err)

		ginkgo.By("/readyz")
		err = testPath(f.ClientSet, "/readyz?verbose=1", requiredReadyzChecks)
		framework.ExpectNoError(err)
	})
})
```

https://github.com/kubernetes/kubernetes/blob/release-1.23/staging/src/k8s.io/client-go/rest/request.go#L332
```
// RequestURI overwrites existing path and parameters with the value of the provided server relative
// URI.
func (r *Request) RequestURI(uri string) *Request {
	if r.err != nil {
		return r
	}
	locator, err := url.Parse(uri)
	if err != nil {
		r.err = err
		return r
	}
	r.pathPrefix = locator.Path
	if len(locator.Query()) > 0 {
		if r.params == nil {
			r.params = make(url.Values)
		}
		for k, v := range locator.Query() {
			r.params[k] = v
		}
	}
	return r
}
```




### What did you expect to happen?

The test case should executed successfully.

### How can we reproduce it (as minimally and precisely as possible)?

You can rerun the TC.

```
podman run -it --env-file ~/opnfv/env -v ~/opnfv/ca.pem:/home/opnfv/functest/ca.pem:Z -v ~/opnfv/config:/root/.kube/config:Z -v ~/opnfv/repositories.yml:/home/opnfv/functest/repositories.yml:Z -v ~/opnfv/results:/home/opnfv/functest/results:Z opnfv/functest-kubernetes-smoke:v1.23 /bin/bash  
```

```
/usr/local/bin/e2e.test \
--stderrthreshold=0 \
--test.timeout=24h0m0s \
--ginkgo.seed=1669123393 \
--ginkgo.noColor \
--ginkgo.slowSpecThreshold=5.00000 \
-kubeconfig /root/.kube/config \
-provider local \
-report-dir /home/opnfv/functest/results/focus/ \
-ginkgo.focus "health handlers should contain necessary checks" \
-non-blocking-taints is_control,is_edge \
-disable-log-dump false
```





### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:49:13Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.15", GitCommit:"b84cb8ab29366daa1bba65bc67f54de2f6c34848", GitTreeState:"clean", BuildDate:"2022-12-08T10:42:57Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
Openstack
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.6 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.6 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8"
$ uname -a
Linux cbis-sut1-control-01 4.18.0-372.9.1.el8.x86_64 #1 SMP Tue May 10 14:48:47 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>