Repository: kubernetes
Issue: Requests proxied from API Server to APIService obfuscate original request hostname
Labels: kind/bug,sig/api-machinery,lifecycle/frozen,triage/accepted

### What happened?

Requests that are proxied by the APIServer to a custom APIService (aggregation) don't generate the `X-Forwarded-Host` header correctly. Each request will have **both** `Host` and `X-Forwarded-Host` set to ClusterIP IP address of their service.  `X-Forwarded-Host` should be set to the original `Host` value sent by the client. 

The result is that the APIService will not be able to determine the original host and scheme used to make the request. An APIService that wants to generate URLs that are routable for an external client cannot do so.

----

As an example here's some network psuedocode that reflects what I observe:

Client makes an HTTP request to the API Server using a public API Server endpoint. Let's say `mycluster.mycloud.net:6443`.

Initial request: 
```txt
GET /apis/api.example.com/v1alpha1
Host: mycluster.mycloud.net:6443
```

This is routed to aggregation layer of the API Server which then reverse-proxies the request. In this case the APIService is using a ClusterIP service, which happens to have Cluster IP `10.43.176.179`.

Request received by APIService:

```txt
GET /apis/api.example.com/v1alpha1
Host: 10.43.176.179:443
X-Forwarded-Host: 10.43.176.179:443
<other headers added by API Server>
```

The thing I am saying is the bug is that `X-Forwarded-Host` is `10.43.176.179:443` when it should be `mycluster.mycloud.net:6443`.



### What did you expect to happen?

I would expect `X-Forwarded-Host` to reflect the original hostname. 

**Expected:** (request sent to APIService)

```txt
GET /apis/api.example.com/v1alpha1
Host: 10.43.176.179:443
X-Forwarded-Host: mycluster.mycloud.net:6443
<other headers added by API Server>
```

**Actual:** (request sent to APIService)

```txt
GET /apis/api.example.com/v1alpha1
Host: 10.43.176.179:443
X-Forwarded-Host: 10.43.176.179:443
<other headers added by API Server>
```

### How can we reproduce it (as minimally and precisely as possible)?

I have put together a sample that demonstrates this by echoing the values it observes for the `Host` and `X-Forwarded-Host` headers:

1. Clone https://github.com/rynowak/apiservice-repro
2. (option) If you prefer, build a container from `src/` and edit `deploy.yaml` to use yours
3. (with a Kubernetes cluster you don't mind getting dirty)

run:

```
kubectl apply -f deploy.yaml
```

4. Make sure APIService starts successfully

run:

```
kubectl get apiservice v1alpha1.api.example.com
```

You should see:

```
NAME                       SERVICE                   AVAILABLE   AGE
v1alpha1.api.example.com   default/demo-apiservice   True        1m
```
5. Open a local proxy

run:

```
kubectl proxy
```

6. Make a request 

run:

```
curl http://127.0.0.1:8001/apis/api.example.com/v1alpha1
```

you should see something like (you will have different IP addresses):

```
{"host":"10.43.176.179:443","forwardedHost":"10.43.176.179:443"}
```

Based on this repro (`kubectl proxy`) I would expect `127.0.0.1:8001` to be the value of `X-Forwarded-Host`

7. Check your service

run:

```
kubectl get service demo-apiservice
```

you should see:

```
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
demo-apiservice   ClusterIP   10.43.176.179   <none>        443/TCP   5m
```

Note that your value for CLUSTER-IP should match the IP addresses from step 6. This confirms that the APIServer proxy is using the downstream value for `X-Forwarded-Host` instead of the original value.

Based on this repro (`kubectl proxy`) I would expect `127.0.0.1:8001` to be the value of `X-Forwarded-Host`

This issue reproduces with and without `kubectl proxy`. I chose `kubectl proxy` for the repro steps here because it simpler to set up a repro.


### Anything else we need to know?

I suspect this root cause with this code:

https://github.com/kubernetes/kube-aggregator/blob/578a094fa7fa8cdba22411e9ab008757f3ef1cb2/pkg/apiserver/handler_proxy.go#L156

At Line 156 the a new request is created with the *downstream* as its destination URL

At Line 167 a roundtripper is created that understands how to rewrite host headers to match the *downstream* for proxying. It seems like the request should use the original host value, so that it can be rewritten by the roundtripper, not the value of the *downstream*. 

### Kubernetes version

<details>

Using k3d (have also reproduced with Kind)

```console
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:33:37Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.7+k3s1", GitCommit:"ac70570999c566ac3507d2cc17369bb0629c1cc0", GitTreeState:"clean", BuildDate:"2021-11-29T16:40:13Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1
```

</details>


### Cloud provider

(n/a)

### OS version

(n/a)

### Install tools

(n/a)

### Container runtime (CRI) and and version (if applicable)

(n/a)

### Related plugins (CNI, CSI, ...) and versions (if applicable)

(n/a)